[ARM][MVE] Enable masked gathers from base + vector of offsets

[anwel]

Summary

This patch extends the earlier patch which enabled masked v4i32 gathers from a vector of pointers (to be found here: https://reviews.llvm.org/D71743).

The pass now checks for the presence of a getelementpointer instruction and, if it finds one, builds a masked gather loading from a base and a vector of offsets. This is more efficient then loading from a vector of pointers. Also, the masked gather instruction generated is not restricted to v4i32 loads, but works for v8i16 and v16i8 as well.

[samparker]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

131–215

Doesn't look this needs to be a variable?

170

I think you can just call C->isNullValue().

178

How about moving this logic up to around line 155 so that all the offset stuff is together, it's quite hard to follow scrolling up and down!

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

131–215

No, it doesn't.

170

I didn't realise that function exists, thanks for the hint

178

That makes sense, I'll move it.

[anwel]

Some small changes to address Sam's comments

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

162

This can be ">"

176

Do we have and tests where this is false? Is it for something like getelementptr <4 x i32>, <4 x i32> *%base, i64 16? That might make sense to generate a QI gather load from (although I don't know if anything will generate gep's like that at the moment without some help).

For the RQ form, we need to make sure that base is a single pointer.

181

This code comes from the ISel gather lowering? I think that for the moment we can just check that the number of GEP operands is 2 (Base + offset) and remove this loop. I'm not exactly sure when it would be useful. We don't seem to have any tests for it, and can add it back in later if they come up in the future. (Otherwise the types don't sound like they would be correct, and the getOperand(1) should be getOperand(FinalIndex)?)

189

I think this can be something like GEP->getResultElementType()->getPrimitiveSizeInBits() == 32. Same below, so maybe make a variable for that size. (scaled_v8f16_half should be be matched too, I think).

Also Clang format!

198

Worth adding a LLVM_DEBUG message so we can tell what prevented the generation.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

176

Yes, I will change this to a check of whether we do have a vector of pointers and, if that rare case occurs, leave it to expansion for now. I'll add a test for that behaviour.

189

You are right, that is also getting rid of the context variable. I will refactor the code a bit.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

181

As as of now I cannot present a test that would show this is actually relevant I agree.

[anwel]

Fixing a typo.

[anwel]

Fixed regress in some lines introduced by rebase on llvm master where the parent patch now has been commited.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

173–187

These return false's are worrying me a little. There may be cases where we cannot select this as a GEP, but it would be fine to fall back to selecting the qi variant instead. Maybe something like this, for example:

define arm_aapcs_vfpcc <4 x i32> @qi4(<4 x i32*> %p) {                                                                   
entry:                                                                                                                   
  %g = getelementptr inbounds i32, <4 x i32*> %p, i32 4                                                                  
  %gather = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> %g, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)                                                                                                
  ret <4 x i32> %gather                                                                                                  
}

It can also be selected as a VLDRW.u32 Qd, [P, 4] I think, but we should be able to get a VLDRW.u32 Qd, [g] at least. I believe you mentioned something similar to this when talking about scatter lowering.

Also it probably makes sense to check these basic things about the GEP before the offset code above. (Although I think any GEP we see will always has at least 1 offset, it makes conceptual sense to check the number of operands before using them).

[anwel]

Moduralised and re-ordered the code to give it a clearer structure and make the fallback to constructing a "basic" gather easier.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

173–187

You are right in this case, and maybe there are others we did not think of. I've modularised the code now which makes it easier to fall back to QI if the construction of a more complex gathers fails for some reason. Hopefully this also makes the code more readable.

[dmgreen]

This I like. I like the structure. I think it's nice to split it out like this.

I'm just not a fan of the Ty, Ptr, Mask and Alignment being part of the class. This class will be lowering multiple gathers/scatters, it may be all too easy to accidentally mis-use them in a non-obvious way, accidentally using the Mask from the wrong gather (for example). Especially as things change in the future.

If the values we need to pass around are {Ty, Ptr, Mask}, then its probably simpler to just pass them to functions that need them. Or maybe Pass I around? I'm not sure. The other option if we expect that there will be many parameters would be to create a struct to pass them around in. Keep them encapsulated together, but not requiring too many parameters. It looks like many of them may just be able to be grabbed from I quite easily though (Ty, Mask, etc).

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

86

LowerGather -> lowerGather to keep it like the others.

127

These could be Value *MVEGatherScatterLowering::tryCreateMaskedGatherOffset and return nullptr on failure. It could then look something like:

Value *Load = tryCreateMaskedGatherOffset(Builder, Ty, Ptr, Mask);
if (!Load)
  Load = tryCreateMaskedGatherBase(Builder, Ty, Ptr, Mask);
if (!Load)
  return false;

139

This one could be Value *MVEGatherScatterLowering::lookThroughBitcast(Value *Ptr), and return either BitCast->getOperand(0) or the original Ptr.

[anwel]

@dmgreen You are absolutely right about the danger regarding the class variables, I had missed that aspect.
I managed to reduce the number of variables which need to be passed between functions to at most four, which I think is acceptable.
Please have a look and tell me what you think.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

127

Yes, that makes one less argument to the function

[dmgreen]

LGTM. Nice one.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

271

Nit: Add a space between the "Can't" and the "create".

[anwel]

Added a space in the debug message.

[anwel]

Rebase and fix a typo.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

 INITIALIZE_PASS(MVEGatherScatterLowering, DEBUG_TYPE,
                 "MVE gather/scattering lowering pass", false, false)
 
 Pass *llvm::createMVEGatherScatterLoweringPass() {
   return new MVEGatherScatterLowering();
 }
 
 static bool isLegalTypeAndAlignment(unsigned NumElements, unsigned ElemSize,
                                     unsigned Alignment) {
-  // Do only allow non-extending v4i32 gathers for now
-  return NumElements == 4 && ElemSize == 32 && Alignment >= 4;
+  // Do only allow non-extending gathers for now
+  return ((NumElements == 4 && ElemSize == 32) ||
+          (NumElements == 8 && ElemSize == 16) ||
+          (NumElements == 16 && ElemSize == 8)) &&
+         ElemSize / 8 <= Alignment;
 }
 
 static bool LowerGather(IntrinsicInst *I) {
   using namespace PatternMatch;
   LLVM_DEBUG(dbgs() << "masked gathers: checking transform preconditions\n");
 
   // @llvm.masked.gather.*(Ptrs, alignment, Mask, Src0)
   // Attempt to turn the masked gather in I into a MVE intrinsic
static bool LowerGather(IntrinsicInst *I) {
   Value *Load = nullptr;
   // Look through bitcast instruction if #elements is the same
   if (auto *BitCast = dyn_cast<BitCastInst>(Ptr)) {
     Type *BCTy = BitCast->getType();
     Type *BCSrcTy = BitCast->getOperand(0)->getType();
     if (BCTy->getVectorNumElements() == BCSrcTy->getVectorNumElements()) {
       LLVM_DEBUG(dbgs() << "masked gathers: looking through bitcast\n");
       Ptr = BitCast->getOperand(0);
     }
   }
   assert(Ptr->getType()->isVectorTy() && "Unexpected pointer type");
 
+  Value *BasePtr = Ptr;
+  GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Ptr);
+  if (!GEP) {
     if (Ty->getVectorNumElements() != 4)
       // Can't build an intrinsic for this
       return false;
+
+    LLVM_DEBUG(dbgs() << "masked gathers: no getelementpointer found. Loading"
+                      << " from vector of pointers\n");
     if (match(Mask, m_One()))
       Load = Builder.CreateIntrinsic(Intrinsic::arm_mve_vldr_gather_base,
                                      {Ty, Ptr->getType()},
                                      {Ptr, Builder.getInt32(0)});
     else
       Load = Builder.CreateIntrinsic(
           Intrinsic::arm_mve_vldr_gather_base_predicated,
           {Ty, Ptr->getType(), Mask->getType()},
           {Ptr, Builder.getInt32(0), Mask});
+  } else {
+    LLVM_DEBUG(dbgs() << "masked gathers: getelementpointer found. Loading"
+                      << " from base + vector of offsets\n");
+    // Load from base + vector of offsets
+    Value *Offsets = GEP->getOperand(1);
+
+    if (ZExtInst *ZextOffs = dyn_cast<ZExtInst>(Offsets))
+      Offsets = ZextOffs->getOperand(0);
+    Type *OffsType = VectorType::getInteger(cast<VectorType>(Ty));
+    // If the offset we found does not have the type the intrinsic expects,
+    // i.e., the same type as the gather itself, we need to convert it (only i
+    // types) or fall back to expanding the gather
+    if (OffsType != Offsets->getType()) {
+      if (OffsType->getScalarSizeInBits() >=
+          Offsets->getType()->getScalarSizeInBits()) {
+        LLVM_DEBUG(dbgs() << "masked gathers: extending offsets\n");
+        Offsets = Builder.CreateZExt(Offsets, OffsType, "");
+      } else {
+        LLVM_DEBUG(dbgs() << "masked gathers: no correct offset type. can't"
+                          << " create masked gather\n");
+        return false;
+      }
+    }
+
+    // SExt offsets inside masked gathers are not permitted by the architecture;
+    // we therefore can't fold them
+    Value *GEPPtr = GEP->getPointerOperand();
+    if (!GEPPtr->getType()->isVectorTy())
+      BasePtr = GEPPtr;
+
+    unsigned FinalIndex = GEP->getNumOperands() - 1;
+    // Ensure all the other indices are 0.
+    for (unsigned i = 1; i < FinalIndex; ++i) {
+      auto *C = dyn_cast<Constant>(GEP->getOperand(i));
+      if (!C || !C->isNullValue())
+        return false;
+    }
+
+    unsigned Scale;
+    LLVMContext &C = I->getContext();
+    if (BasePtr->getType() == Type::getInt32PtrTy(C)
+        && Ty->getScalarSizeInBits() == 32) {
+      Scale = 2;
+    } else if (BasePtr->getType() == Type::getInt16PtrTy(C)
+        && Ty->getScalarSizeInBits() == 16) {
+      Scale = 1;
+    } else if (BasePtr->getType() == Type::getInt8PtrTy(C)) {
+      Scale = 0;
+    } else {
+      return false;
+    }
+
+    if (!match(Mask, m_One()))
+      Load = Builder.CreateIntrinsic(
+          Intrinsic::arm_mve_vldr_gather_offset_predicated,
+          {Ty, BasePtr->getType(), Offsets->getType(), Mask->getType()},
+          {BasePtr, Offsets,
+           Builder.getInt32(Ty->getScalarSizeInBits()),
+           Builder.getInt32(Scale), Builder.getInt32(1), Mask});
+    else
+      Load = Builder.CreateIntrinsic(
+          Intrinsic::arm_mve_vldr_gather_offset,
+          {Ty, BasePtr->getType(), Offsets->getType()},
+          {BasePtr, Offsets,
+           Builder.getInt32(Ty->getScalarSizeInBits()),
+           Builder.getInt32(Scale), Builder.getInt32(1)});
+  }
 
   if (!isa<UndefValue>(PassThru) && !match(PassThru, m_Zero())) {
     LLVM_DEBUG(dbgs() << "masked gathers: found non-trivial passthru - "
                       << "creating select\n");
     Load = Builder.CreateSelect(Mask, Load, PassThru);
   }
 
   LLVM_DEBUG(dbgs() << "masked gathers: successfully built masked gather\n");
bool MVEGatherScatterLowering::runOnFunction(Function &F) {
 
   if (Gathers.empty())
     return false;
 
   for (IntrinsicInst *I : Gathers)
     LowerGather(I);
 
   return true;
 }

llvm/test/CodeGen/Thumb2/mve-gather-ind16-scaled.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -enable-arm-maskedgatscat %s -o - | FileCheck %s
 
 ; VLDRH.16 Qd, [base, offs, uxtw #1]
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r3, s3
-; CHECK-NEXT:    vmov r5, s1
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrh.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh r5, [r5]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    vmov.16 q0[3], lr
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1, uxtw #1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> undef)
   ret <8 x i16> %gather
 }
 
 ; VLDRH.16 Qd, [base, offs, uxtw #1]
 define arm_aapcs_vfpcc <8 x half> @scaled_v8f16_i16(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8f16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r2, s5
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov r3, s4
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vldr.16 s0, [r3]
-; CHECK-NEXT:    vmov r3, s0
-; CHECK-NEXT:    vmov.16 q0[0], r3
-; CHECK-NEXT:    vmov.16 q0[1], r2
-; CHECK-NEXT:    vmov r2, s6
-; CHECK-NEXT:    vldr.16 s8, [r2]
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov r2, s7
-; CHECK-NEXT:    vldr.16 s4, [r2]
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vmov.16 q0[3], r2
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr.16 s4, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1, uxtw #1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %i16_ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %ptrs = bitcast <8 x i16*> %i16_ptrs to <8 x half*>
   %gather = call <8 x half> @llvm.masked.gather.v8f16(<8 x half*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x half> undef)
   ret <8 x half> %gather
entry:
   %gather = call <8 x half> @llvm.masked.gather.v8f16(<8 x half*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x half> undef)
   ret <8 x half> %gather
 }
 
 ; VLDRH.16 Qd, [base, zext(offs), uxtw #1]
 define arm_aapcs_vfpcc <8 x i16> @unsigned_scaled_v8i16_i8(i16* %base, <8 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_v8i16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vldrb.u32 q1, [r1, #4]
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r3, s3
-; CHECK-NEXT:    vmov r5, s1
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrh.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh r5, [r5]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    vmov.16 q0[3], lr
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrb.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1, uxtw #1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i8>, <8 x i8>* %offptr, align 1
   %offs.zext = zext <8 x i8> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> undef)
   ret <8 x i16> %gather
 }
 
 ; VLDRH.16 Qd, [base, zext(offs), uxtw #1]
 define arm_aapcs_vfpcc <8 x half> @unsigned_scaled_v8f16_i8(i16* %base, <8 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_v8f16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r2, s5
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov r3, s4
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vldr.16 s0, [r3]
-; CHECK-NEXT:    vmov r3, s0
-; CHECK-NEXT:    vmov.16 q0[0], r3
-; CHECK-NEXT:    vmov.16 q0[1], r2
-; CHECK-NEXT:    vmov r2, s6
-; CHECK-NEXT:    vldr.16 s8, [r2]
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov r2, s7
-; CHECK-NEXT:    vldr.16 s4, [r2]
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldrb.u32 q1, [r1, #4]
-; CHECK-NEXT:    vmov.16 q0[3], r2
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr.16 s4, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
+; CHECK-NEXT:    vldrb.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1, uxtw #1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i8>, <8 x i8>* %offptr, align 1
   %offs.zext = zext <8 x i8> %offs to <8 x i32>
   %i16_ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %ptrs = bitcast <8 x i16*> %i16_ptrs to <8 x half*>
   %gather = call <8 x half> @llvm.masked.gather.v8f16(<8 x half*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x half> undef)
   ret <8 x half> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16_passthru0t(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16_passthru0t:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r3, s3
-; CHECK-NEXT:    vmov r5, s1
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrh.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh r5, [r5]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    vmov.16 q0[3], lr
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1, uxtw #1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> zeroinitializer)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16_passthru1t(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16_passthru1t:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r3, s3
-; CHECK-NEXT:    vmov r5, s1
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrh.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh r5, [r5]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    vmov.16 q0[3], lr
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1, uxtw #1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16_passthru1f(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16_passthru1f:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    movs r5, #1
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    vmov r4, s1
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh.w lr, [r0]
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s7
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    vmov.16 q0[0], r0
-; CHECK-NEXT:    vmov.16 q0[1], r4
-; CHECK-NEXT:    vmov.16 q0[2], r5
-; CHECK-NEXT:    vmov.16 q0[3], r12
-; CHECK-NEXT:    vmov.16 q0[4], lr
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r2
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    movw r2, #65487
+; CHECK-NEXT:    vmov.i16 q0, #0x1
+; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrht.u16 q2, [r0, q1, uxtw #1]
+; CHECK-NEXT:    vpsel q0, q2, q0
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 false, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16_passthru0f(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16_passthru0f:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    movs r5, #0
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r4, s0
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh.w lr, [r0]
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s7
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.16 q0[0], r4
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    vmov.16 q0[3], r0
-; CHECK-NEXT:    vmov.16 q0[4], lr
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r2
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    movw r2, #65523
+; CHECK-NEXT:    vmsr p0, r2
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrht.u16 q0, [r0, q1, uxtw #1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 false, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> <i16 0, i16 0, i16 0, i16 0, i16 0, i16 0, i16 0, i16 0>)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16_passthru_icmp0(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16_passthru_icmp0:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    mov.w r12, #0
-; CHECK-NEXT:    vcmp.s16 gt, q0, zr
-; CHECK-NEXT:    vmov.u16 r2, q0[4]
-; CHECK-NEXT:    vmrs r1, p0
-; CHECK-NEXT:    vmov.32 q1[0], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[5]
-; CHECK-NEXT:    vmov.u16 lr, q0[7]
-; CHECK-NEXT:    vmov.32 q1[1], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[6]
-; CHECK-NEXT:    vmov.32 q1[2], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[0]
-; CHECK-NEXT:    vmov.32 q2[0], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[1]
-; CHECK-NEXT:    vmov.32 q2[1], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[2]
-; CHECK-NEXT:    vmov.32 q2[2], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[3]
-; CHECK-NEXT:    vmov.32 q2[3], r2
-; CHECK-NEXT:    vmovlb.u16 q0, q2
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vadd.i32 q2, q0, r0
-; CHECK-NEXT:    and r3, r1, #1
-; CHECK-NEXT:    rsbs r2, r3, #0
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    bfi r3, r2, #0, #1
-; CHECK-NEXT:    ubfx r2, r1, #2, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #1, #1
-; CHECK-NEXT:    ubfx r2, r1, #4, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #2, #1
-; CHECK-NEXT:    ubfx r2, r1, #6, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #3, #1
-; CHECK-NEXT:    ubfx r2, r1, #8, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #4, #1
-; CHECK-NEXT:    ubfx r2, r1, #10, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #5, #1
-; CHECK-NEXT:    ubfx r2, r1, #12, #1
-; CHECK-NEXT:    ubfx r1, r1, #14, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r3
-; CHECK-NEXT:    lsls r2, r3, #31
-; CHECK-NEXT:    beq .LBB11_2
-; CHECK-NEXT:  @ %bb.1: @ %cond.load
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vdup.16 q0, r12
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    b .LBB11_3
-; CHECK-NEXT:  .LBB11_2:
-; CHECK-NEXT:    vmov.i32 q0, #0x0
-; CHECK-NEXT:  .LBB11_3: @ %else
-; CHECK-NEXT:    vmov.32 q1[3], lr
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r2, s9
-; CHECK-NEXT:    ldrhmi r2, [r2]
-; CHECK-NEXT:    vmovmi.16 q0[1], r2
-; CHECK-NEXT:    vmovlb.u16 q1, q1
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r2, s10
-; CHECK-NEXT:    ldrhmi r2, [r2]
-; CHECK-NEXT:    vmovmi.16 q0[2], r2
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    lsls r2, r1, #28
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r2, s11
-; CHECK-NEXT:    ldrhmi r2, [r2]
-; CHECK-NEXT:    vmovmi.16 q0[3], r2
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    lsls r0, r1, #27
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s4
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[4], r0
-; CHECK-NEXT:    lsls r0, r1, #26
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s5
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[5], r0
-; CHECK-NEXT:    lsls r0, r1, #25
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s6
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[6], r0
-; CHECK-NEXT:    lsls r0, r1, #24
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s7
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[7], r0
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-NEXT:    vldrht.u16 q0, [r0, q1, uxtw #1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %mask = icmp sgt <8 x i16> %offs, zeroinitializer
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> %mask, <8 x i16> <i16 0, i16 0, i16 0, i16 0, i16 0, i16 0, i16 0, i16 0>)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @scaled_v8i16_i16_passthru_icmp1(i16* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: scaled_v8i16_i16_passthru_icmp1:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r7, lr}
-; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #8
-; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    vldrh.u16 q0, [r1]
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    vcmp.s16 gt, q0, zr
-; CHECK-NEXT:    vmov.u16 r2, q0[4]
-; CHECK-NEXT:    vmrs r1, p0
-; CHECK-NEXT:    vmov.32 q1[0], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[5]
-; CHECK-NEXT:    vmov.u16 r12, q0[7]
-; CHECK-NEXT:    vmov.32 q1[1], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[6]
-; CHECK-NEXT:    vmov.32 q1[2], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[0]
-; CHECK-NEXT:    vmov.32 q2[0], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[1]
-; CHECK-NEXT:    vmov.32 q2[1], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[2]
-; CHECK-NEXT:    vmov.32 q2[2], r2
-; CHECK-NEXT:    vmov.u16 r2, q0[3]
-; CHECK-NEXT:    vmov.32 q2[3], r2
-; CHECK-NEXT:    vmovlb.u16 q0, q2
-; CHECK-NEXT:    vshl.i32 q0, q0, #1
-; CHECK-NEXT:    vadd.i32 q2, q0, r0
-; CHECK-NEXT:    and lr, r1, #1
-; CHECK-NEXT:    rsb.w r2, lr, #0
-; CHECK-NEXT:    bfi r3, r2, #0, #1
-; CHECK-NEXT:    ubfx r2, r1, #2, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #1, #1
-; CHECK-NEXT:    ubfx r2, r1, #4, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #2, #1
-; CHECK-NEXT:    ubfx r2, r1, #6, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #3, #1
-; CHECK-NEXT:    ubfx r2, r1, #8, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #4, #1
-; CHECK-NEXT:    ubfx r2, r1, #10, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #5, #1
-; CHECK-NEXT:    ubfx r2, r1, #12, #1
-; CHECK-NEXT:    ubfx r1, r1, #14, #1
-; CHECK-NEXT:    rsbs r2, r2, #0
-; CHECK-NEXT:    bfi r3, r2, #6, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r3, r1, #7, #1
-; CHECK-NEXT:    uxtb r1, r3
-; CHECK-NEXT:    lsls r2, r3, #31
-; CHECK-NEXT:    beq .LBB12_2
-; CHECK-NEXT:  @ %bb.1: @ %cond.load
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vdup.16 q0, r3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    b .LBB12_3
-; CHECK-NEXT:  .LBB12_2:
 ; CHECK-NEXT:    vmov.i16 q0, #0x1
-; CHECK-NEXT:  .LBB12_3: @ %else
-; CHECK-NEXT:    vmov.32 q1[3], r12
-; CHECK-NEXT:    lsls r2, r1, #30
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r2, s9
-; CHECK-NEXT:    ldrhmi r2, [r2]
-; CHECK-NEXT:    vmovmi.16 q0[1], r2
-; CHECK-NEXT:    vmovlb.u16 q1, q1
-; CHECK-NEXT:    lsls r2, r1, #29
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r2, s10
-; CHECK-NEXT:    ldrhmi r2, [r2]
-; CHECK-NEXT:    vmovmi.16 q0[2], r2
-; CHECK-NEXT:    vshl.i32 q1, q1, #1
-; CHECK-NEXT:    lsls r2, r1, #28
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r2, s11
-; CHECK-NEXT:    ldrhmi r2, [r2]
-; CHECK-NEXT:    vmovmi.16 q0[3], r2
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    lsls r0, r1, #27
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s4
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[4], r0
-; CHECK-NEXT:    lsls r0, r1, #26
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s5
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[5], r0
-; CHECK-NEXT:    lsls r0, r1, #25
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s6
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[6], r0
-; CHECK-NEXT:    lsls r0, r1, #24
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r0, s7
-; CHECK-NEXT:    ldrhmi r0, [r0]
-; CHECK-NEXT:    vmovmi.16 q0[7], r0
-; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    pop {r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vpt.s16 gt, q1, zr
+; CHECK-NEXT:    vldrht.u16 q2, [r0, q1, uxtw #1]
+; CHECK-NEXT:    vpsel q0, q2, q0
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i16, i16* %base, <8 x i32> %offs.zext
   %mask = icmp sgt <8 x i16> %offs, zeroinitializer
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> %mask, <8 x i16> <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>)
   ret <8 x i16> %gather
 }
 
 declare <8 x i8> @llvm.masked.gather.v8i8(<8 x i8*>, i32, <8 x i1>, <8 x i8>) #1
 declare <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*>, i32, <8 x i1>, <8 x i16>) #1
 declare <8 x half> @llvm.masked.gather.v8f16(<8 x half*>, i32, <8 x i1>, <8 x half>) #1

llvm/test/CodeGen/Thumb2/mve-gather-ind16-unscaled.ll

entry:
   %gather = call <8 x i8> @llvm.masked.gather.v8i8(<8 x i8*> %ptrs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %gather.sext = sext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %gather.sext
 }
 
 define arm_aapcs_vfpcc <8 x i16> @unscaled_i16_i16(i8* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: unscaled_i16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r3, s3
-; CHECK-NEXT:    vmov r5, s1
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrh.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh r5, [r5]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    vmov.16 q0[3], lr
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %ptrs = bitcast <8 x i8*> %byte_ptrs to <8 x i16*>
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> undef)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x half> @unscaled_f16_i16(i8* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: unscaled_f16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov r3, s5
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vldr.16 s0, [r3]
-; CHECK-NEXT:    vmov r3, s0
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov r2, s6
-; CHECK-NEXT:    vmov.16 q0[1], r3
-; CHECK-NEXT:    vldr.16 s8, [r2]
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov r2, s7
-; CHECK-NEXT:    vldr.16 s4, [r2]
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vmov.16 q0[3], r2
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr.16 s4, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %ptrs = bitcast <8 x i8*> %byte_ptrs to <8 x half*>
   %gather = call <8 x half> @llvm.masked.gather.v8f16(<8 x half*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x half> undef)
   ret <8 x half> %gather
entry:
   %gather = call <8 x i8> @llvm.masked.gather.v8i8(<8 x i8*> %ptrs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %gather.sext = sext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %gather.sext
 }
 
 define arm_aapcs_vfpcc <8 x i16> @unsigned_unscaled_i16_i8(i8* %base, <8 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_i16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vldrb.u32 q1, [r1, #4]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    vmov r3, s3
-; CHECK-NEXT:    vmov r5, s1
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrh.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrh.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrh r5, [r5]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    ldrh r4, [r4]
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov.16 q0[1], r5
-; CHECK-NEXT:    vmov.16 q0[2], r12
-; CHECK-NEXT:    vmov.16 q0[3], lr
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrb.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i8>, <8 x i8>* %offptr, align 1
   %offs.zext = zext <8 x i8> %offs to <8 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %ptrs = bitcast <8 x i8*> %byte_ptrs to <8 x i16*>
   %gather = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i16> undef)
   ret <8 x i16> %gather
 }
 
 define arm_aapcs_vfpcc <8 x half> @unsigned_unscaled_f16_i8(i8* %base, <8 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_f16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov r3, s5
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vldr.16 s0, [r3]
-; CHECK-NEXT:    vmov r3, s0
-; CHECK-NEXT:    vmov.16 q0[0], r2
-; CHECK-NEXT:    vmov r2, s6
-; CHECK-NEXT:    vmov.16 q0[1], r3
-; CHECK-NEXT:    vldr.16 s8, [r2]
-; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov r2, s7
-; CHECK-NEXT:    vldr.16 s4, [r2]
-; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vldrb.u32 q1, [r1, #4]
-; CHECK-NEXT:    vmov.16 q0[3], r2
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[5], r0
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr.16 s8, [r0]
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    vmov.16 q0[6], r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr.16 s4, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov.16 q0[7], r0
+; CHECK-NEXT:    vldrb.u16 q1, [r1]
+; CHECK-NEXT:    vldrh.u16 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i8>, <8 x i8>* %offptr, align 1
   %offs.zext = zext <8 x i8> %offs to <8 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %ptrs = bitcast <8 x i8*> %byte_ptrs to <8 x half*>
   %gather = call <8 x half> @llvm.masked.gather.v8f16(<8 x half*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x half> undef)
   ret <8 x half> %gather
 }
 
 declare <8 x i8> @llvm.masked.gather.v8i8(<8 x i8*>, i32, <8 x i1>, <8 x i8>) #1
 declare <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*>, i32, <8 x i1>, <8 x i16>) #1
 declare <8 x half> @llvm.masked.gather.v8f16(<8 x half*>, i32, <8 x i1>, <8 x half>) #1

llvm/test/CodeGen/Thumb2/mve-gather-ind32-scaled.ll

entry:
   %gather = call <4 x i16> @llvm.masked.gather.v4i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @scaled_i32_i32(i32* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: scaled_i32_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 ; TODO: scaled_f16_i32
 
 define arm_aapcs_vfpcc <4 x float> @scaled_f32_i32(i32* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: scaled_f32_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_scaled_b_i32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_b_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_scaled_i32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: signed_scaled_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_scaled_f32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: a_unsigned_scaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_scaled_f32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: b_signed_scaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
entry:
   %gather = call <4 x i16> @llvm.masked.gather.v4i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_scaled_b_i32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_b_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_scaled_i32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: signed_scaled_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_scaled_f32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: a_unsigned_scaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_scaled_f32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: b_signed_scaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1, uxtw #2]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather

llvm/test/CodeGen/Thumb2/mve-gather-ind32-unscaled.ll

entry:
   %gather = call <4 x i16> @llvm.masked.gather.v4i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unscaled_i32_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: unscaled_i32_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @unscaled_f32_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: unscaled_f32_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_unscaled_b_i32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_b_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_unscaled_i32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: signed_unscaled_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_unscaled_f32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: a_unsigned_unscaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_unscaled_f32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: b_signed_unscaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
entry:
   %gather = call <4 x i8> @llvm.masked.gather.v4i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_unscaled_b_i32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_b_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_unscaled_i32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: signed_unscaled_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_unscaled_f32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: a_unsigned_unscaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_unscaled_f32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: b_signed_unscaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrw.u32 q0, [q1]
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrw.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather

llvm/test/CodeGen/Thumb2/mve-gather-ind8-unscaled.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -enable-arm-maskedgatscat %s -o - | FileCheck %s
 
 ; VLDRB.16
 define arm_aapcs_vfpcc <16 x i8> @unscaled_v16i8_i8(i8* %base, <16 x i8>* %offptr) {
 ; CHECK-LABEL: unscaled_v16i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r6, lr}
-; CHECK-NEXT:    push {r4, r5, r6, lr}
-; CHECK-NEXT:    vldrb.u32 q0, [r1, #8]
-; CHECK-NEXT:    vldrb.u32 q2, [r1, #4]
-; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vldrb.u32 q0, [r1, #12]
-; CHECK-NEXT:    vmov r2, s6
-; CHECK-NEXT:    vadd.i32 q2, q2, r0
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r6, s4
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r4, s3
-; CHECK-NEXT:    vmov r5, s7
-; CHECK-NEXT:    ldrb.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    ldrb r6, [r6]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    ldrb r4, [r4]
-; CHECK-NEXT:    ldrb r5, [r5]
-; CHECK-NEXT:    ldrb.w lr, [r2]
-; CHECK-NEXT:    vmov r2, s1
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q3, q0, r0
-; CHECK-NEXT:    vmov r0, s12
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[0], r0
-; CHECK-NEXT:    vmov r0, s13
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[1], r0
-; CHECK-NEXT:    vmov r0, s14
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[2], r0
-; CHECK-NEXT:    vmov r0, s15
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[3], r0
-; CHECK-NEXT:    vmov r0, s8
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[4], r0
-; CHECK-NEXT:    vmov r0, s9
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[5], r0
-; CHECK-NEXT:    vmov r0, s10
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[6], r0
-; CHECK-NEXT:    vmov r0, s11
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[7], r0
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vmov.8 q0[8], r6
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.8 q0[9], r0
-; CHECK-NEXT:    vmov.8 q0[10], r12
-; CHECK-NEXT:    vmov.8 q0[11], r5
-; CHECK-NEXT:    vmov.8 q0[12], lr
-; CHECK-NEXT:    vmov.8 q0[13], r2
-; CHECK-NEXT:    vmov.8 q0[14], r3
-; CHECK-NEXT:    vmov.8 q0[15], r4
-; CHECK-NEXT:    pop {r4, r5, r6, pc}
+; CHECK-NEXT:    vldrb.u8 q1, [r1]
+; CHECK-NEXT:    vldrb.u8 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <16 x i8>, <16 x i8>* %offptr, align 1
   %offs.zext = zext <16 x i8> %offs to <16 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <16 x i32> %offs.zext
   %gather = call <16 x i8> @llvm.masked.gather.v16i8(<16 x i8*> %ptrs, i32 1, <16 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <16 x i8> undef)
   ret <16 x i8> %gather
 }