Rather than inspecting the element type of the pointer, look for the GEP step instruction and take it's element type. Also store that in the IVDescriptor and fetch it in LoopVectorize.
A problem of this approach is that it fails if there is no direct GEP step instruction. It's usually present, but one LoopVectorize test shows a case where it isn't. I'm not sure how important it is.
The main alternative I see here is to use different logic for opaque pointers and always base them off an i8 element type (aka "uglygep"). The disadvantage of that is that a switch to opaque pointers would have a non-trivial impact on how optimization works here, so I think it would be the worst option. Another variant is to always use i8 element types, i.e. make this purely offset based and force necessary pointer casts for non-opaque pointers.