For vector element types of less than one byte in size, we would
generate incorrect scalar offsets and produce incorrect codegen.
This optimization could be supported in the future, e.g. by loading a
byte, then shifting and masking out the smaller vector element type.
However, without an upstream target to test against it's best to avoid
the bad codegen in the simplest possible way.
Related to this bug: