[ARM][MVE] Enable extending gathers

[anwel]

Summary

This patch is built on top of the patch introducing non-extending masked gathers from a base and a vector of offsets (to be found at https://reviews.llvm.org/D72330).

It extends the pass that transforms selected gathers into calls to MVE's masked gather intrinsic by extending gathers.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

140

Should this be Type::getInt32Ty ? Or Builder.getInt32() ?

146

u -> U. Or better yet just check that I->hasOneUse(). The Use of a Instruction will always be an Instruction AFAIU.

Also "Parent" makes me think of the BasicBlock/Function, not the user. Maybe just call it "User" or something like it?

230

There are some tests in mve-gather-ind32-scaled.ll like "zext_signed_scaled_i16_i16" that I think can be converted to a VLDRH.u32 Qd, [base, offs.sext, uxtw #1] if we can get the extend and this scale working together. It's the type of the memory being loaded that's important for the scale, not the type of the result. (Err, I think. Check that sounds right)

268

Maybe drop the "to avoid reevaluation". The rest is reason enough to remove the gather.

[anwel]

Rebase on parent revision which now takes a more moduralised approach

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

140

I assume Type::getInt32Ty(C) is the cleanest. I'll use that instead.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

146

Okay. I wasn't sure about whether users always have to be instructions. If they are, the Parent variable becomes superfluous.

230

You're right. The modularisation fixed whatever was keeping those tests from being converted to a gather, they look better now.

[anwel]

Simplified the search for the gather instruction's user

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

71–72

This is a little long. Clang-format would fix it up I think.

86–87

This can be left in I would guess? Or removed entirely if it's too noisy. Maybe reword it to "can't find gep" as opposed to "expanding".

265–271

If you moved this "removing" code into tryCreateMaskedGatherOffset or tryCreateMaskedGatherBase it would remove the need for a Root parameter. Not sure if it's cleaner, but it's only tryCreateMaskedGatherOffset that will have a Root, so tryCreateMaskedGatherBase's version will be simpler.

273–274

ResultElemSize -> MemoryElemSize I think would be a better name. The "Result" would be after the extend.

291

This can just be cast<..>

292

I think we have to check the type of the extend, otherwise it might be transforming to any odd thing.

Make sure there is some testcase for this too, I don't think we have maybe tests for stranger types. i64 are a good candidate. Even they are not legal (for the instruction versions that we are dealing with here), they can quite easily be created by the vectorizer.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

86–87

I think I will remove it completely, to get rid of some noise. With the current implementation, it is not clear anymore whether we will fail (i.e., expand) or not after this point. For types for which we can build masked gathers from a vector of pointers, we are now able to transform this, as demonstrated by the following test:

define arm_aapcs_vfpcc <4 x i32> @qi4(<4 x i32*> %p) {
; CHECK-LABEL: qi4:
; CHECK:       @ %bb.0: @ %entry
; CHECK-NEXT:    vmov.i32 q1, #0x10
; CHECK-NEXT:    vadd.i32 q1, q0, q1
; CHECK-NEXT:    vldrw.u32 q0, [q1]
; CHECK-NEXT:    bx lr
entry:
  %g = getelementptr inbounds i32, <4 x i32*> %p, i32 4
  %gather = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> %g, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
  ret <4 x i32> %gather
}

265–271

That would make these few lines a bit simpler for tryCreateMaskedGatherBase, right, but I don't think it's worth the code duplication?
Because both kinds of gathers could come with a passthru, thus the 5 lines above this dealing with that that build a select would have to be duplicated or made into a new function to be called in tryCreateMaskedGatherBase as well as in tryCreateMaskedGatherOffset.
That is, whether you just copy or use a new function, around 6 new lines to make it 2 lines simpler in one case?

292

Thanks for spotting that, I wrote a test and discovered that it crashed the compiler. I'll upload a new diff with this fixed.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

265–271

Ah I see. Because you are updating Load to point to the select. Fair enough. I had missed that.

In that case we should make sure that the Root passed to tryCreateMaskedGatherOffset doesn't change unless the function returns a valid load. At the moment it looks like the function could find an extend (so set Root) but then not find a valid GEP.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

265–271

Good point. I'll make sure that can't happen.

[anwel]

Small change to make sure that Root will never be changed if no Qd load is built

[dmgreen]

LGTM. Thanks

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

278

Can you add a comment saying something like "we have already checked the size of the gather in isLegalTypeAndAlignment, and need to check that the extend is to a full vector width."

287

This can just be a cast<Instruction>. The dynamic part should never fail (and the cast<..> will check that).

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

278

Yes, I'll rephrase that comment to be more informative

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

public:
   StringRef getPassName() const override {
     return "MVE gather/scatter lowering";
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     AU.addRequired<TargetPassConfig>();
     FunctionPass::getAnalysisUsage(AU);
   }
 };
 
 } // end anonymous namespace
 
 char MVEGatherScatterLowering::ID = 0;
 
 INITIALIZE_PASS(MVEGatherScatterLowering, DEBUG_TYPE,
                 "MVE gather/scattering lowering pass", false, false)
 
 Pass *llvm::createMVEGatherScatterLoweringPass() {
   return new MVEGatherScatterLowering();
 }
 
 static bool isLegalTypeAndAlignment(unsigned NumElements, unsigned ElemSize,
                                     unsigned Alignment) {
-  // Do only allow non-extending gathers for now
-  return ((NumElements == 4 && ElemSize == 32) ||
-          (NumElements == 8 && ElemSize == 16) ||
+  return ((NumElements == 4 &&
+           (ElemSize == 32 || ElemSize == 16 || ElemSize == 8)) ||
+          (NumElements == 8 && (ElemSize == 16 || ElemSize == 8)) ||
           (NumElements == 16 && ElemSize == 8)) &&
          ElemSize / 8 <= Alignment;
 }
 
 static bool LowerGather(IntrinsicInst *I) {
   using namespace PatternMatch;
   LLVM_DEBUG(dbgs() << "masked gathers: checking transform preconditions\n");
 
   // @llvm.masked.gather.*(Ptrs, alignment, Mask, Src0)
   // Attempt to turn the masked gather in I into a MVE intrinsic
   // Potentially optimising the addressing modes as we do so.
-  Type *Ty = I->getType();
+  Type *OriginalTy = I->getType();
+  Type *ResultTy = OriginalTy;
   Value *Ptr = I->getArgOperand(0);
   unsigned Alignment = cast<ConstantInt>(I->getArgOperand(1))->getZExtValue();
   Value *Mask = I->getArgOperand(2);
   Value *PassThru = I->getArgOperand(3);
+  Instruction *Root = I;
 
   // Check this is a valid gather with correct alignment
-  if (!isLegalTypeAndAlignment(Ty->getVectorNumElements(),
-                        Ty->getScalarSizeInBits(), Alignment)) {
+  if (!isLegalTypeAndAlignment(OriginalTy->getVectorNumElements(),
+                        OriginalTy->getScalarSizeInBits(), Alignment)) {
     LLVM_DEBUG(dbgs() << "masked gathers: instruction does not have valid "
                       << "alignment or vector type \n");
     return false;
   }
 
   IRBuilder<> Builder(I->getContext());
   Builder.SetInsertPoint(I);
   Builder.SetCurrentDebugLocation(I->getDebugLoc());
 
   Value *Load = nullptr;
   // Look through bitcast instruction if #elements is the same
   if (auto *BitCast = dyn_cast<BitCastInst>(Ptr)) {
     Type *BCTy = BitCast->getType();
     Type *BCSrcTy = BitCast->getOperand(0)->getType();
     if (BCTy->getVectorNumElements() == BCSrcTy->getVectorNumElements()) {
       LLVM_DEBUG(dbgs() << "masked gathers: looking through bitcast\n");
       Ptr = BitCast->getOperand(0);
     }
   }
   assert(Ptr->getType()->isVectorTy() && "Unexpected pointer type");
 
+  // If this is an extending gather, a SExt or ZExt must be following
+  LLVMContext &C = I->getContext();
+  unsigned Unsigned = 1;
+  if (OriginalTy->getScalarSizeInBits() * OriginalTy->getVectorNumElements() <
+      128) {
+    bool FoundExtend = false;
+    if (OriginalTy->getVectorNumElements() == 4)
+      ResultTy = VectorType::get(OriginalTy->getInt32Ty(C), 4);
+    else if (OriginalTy->getVectorNumElements() == 8)
+      ResultTy = VectorType::get(OriginalTy->getInt16Ty(C), 8);
+    // The correct root to replace is the not the CallInst itself, but the
+    // instruction which extends it
+    Instruction *Parent = nullptr;
+    for (User *u : I->users()) {
+      // Only do this to gathers with exactly one use
+      if (Parent || !(Parent = dyn_cast<Instruction>(u)))
+        return false;
+    }
+    if (Parent) {
+      LLVM_DEBUG(dbgs() << "masked gathers: found an extending gather\n");
+      if (isa<ZExtInst>(Parent)) {
+        Root = Parent;
+        FoundExtend = true;
+      } else if (isa<SExtInst>(Parent)) {
+        Root = Parent;
+        Unsigned = 0;
+        FoundExtend = true;
+      }
+    }
+    if (!FoundExtend) {
+      LLVM_DEBUG(dbgs() << "masked gathers: extend needed but not provided. "
+                        << "Expanding\n");
+      return false;
+    }
+  }
+
   Value *BasePtr = Ptr;
   GetElementPtrInst *GEP = dyn_cast<GetElementPtrInst>(Ptr);
   if (!GEP) {
-    if (Ty->getVectorNumElements() != 4)
+    if (ResultTy->getVectorNumElements() != 4)
       // Can't build an intrinsic for this
       return false;
 
     LLVM_DEBUG(dbgs() << "masked gathers: no getelementpointer found. Loading"
                       << " from vector of pointers\n");
     if (match(Mask, m_One()))
       Load = Builder.CreateIntrinsic(Intrinsic::arm_mve_vldr_gather_base,
-                                     {Ty, Ptr->getType()},
+                                     {ResultTy, Ptr->getType()},
                                      {Ptr, Builder.getInt32(0)});
     else
       Load = Builder.CreateIntrinsic(
           Intrinsic::arm_mve_vldr_gather_base_predicated,
-          {Ty, Ptr->getType(), Mask->getType()},
+          {ResultTy, Ptr->getType(), Mask->getType()},
           {Ptr, Builder.getInt32(0), Mask});
   } else {
     LLVM_DEBUG(dbgs() << "masked gathers: getelementpointer found. Loading"
                       << " from base + vector of offsets\n");
     // Load from base + vector of offsets
     Value *Offsets = GEP->getOperand(1);
 
     // SExt offsets inside masked gathers are not permitted by the architecture;
     // we therefore can't fold them
     if (ZExtInst *ZextOffs = dyn_cast<ZExtInst>(Offsets))
       Offsets = ZextOffs->getOperand(0);
-    Type *OffsType = VectorType::getInteger(cast<VectorType>(Ty));
+    Type *OffsType = VectorType::getInteger(cast<VectorType>(ResultTy));
     // If the offset we found does not have the type the intrinsic expects,
     // i.e., the same type as the gather itself, we need to convert it (only i
     // types) or fall back to expanding the gather
     if (OffsType != Offsets->getType()) {
       if (OffsType->getScalarSizeInBits() >
           Offsets->getType()->getScalarSizeInBits()) {
         LLVM_DEBUG(dbgs() << "masked gathers: extending offsets\n");
         Offsets = Builder.CreateZExt(Offsets, OffsType, "");
if (!GEP) {
     if (GEP->getNumOperands() != 2) {
       LLVM_DEBUG(dbgs() << "masked gathers: getelementptr with too many"
                         << " operands. Expanding.\n");
       return false;
     }
 
     unsigned Scale;
     int GEPElemSize = GEP->getResultElementType()->getPrimitiveSizeInBits();
-    int ResultElemSize = Ty->getScalarSizeInBits();
+    int ResultElemSize = ResultTy->getScalarSizeInBits();
     // This can be a 32bit load scaled by 4, a 16bit load scaled by 2, or a
     // 8bit, 16bit or 32bit load scaled by 1
     if (GEPElemSize == 32 && ResultElemSize == 32) {
       Scale = 2;
     } else if (GEPElemSize == 16 && ResultElemSize == 16) {
       Scale = 1;
     } else if (GEPElemSize == 8) {
       Scale = 0;
     } else {
       LLVM_DEBUG(dbgs() << "masked gathers: incorrect scale for load. Can't"
                         << "create masked gather\n");
       return false;
     }
 
     if (!match(Mask, m_One()))
       Load = Builder.CreateIntrinsic(
           Intrinsic::arm_mve_vldr_gather_offset_predicated,
-          {Ty, BasePtr->getType(), Offsets->getType(), Mask->getType()},
-          {BasePtr, Offsets, Builder.getInt32(Ty->getScalarSizeInBits()),
-           Builder.getInt32(Scale), Builder.getInt32(1), Mask});
+          {ResultTy, BasePtr->getType(), Offsets->getType(), Mask->getType()},
+          {BasePtr, Offsets, Builder.getInt32(OriginalTy->getScalarSizeInBits()),
+           Builder.getInt32(Scale), Builder.getInt32(Unsigned), Mask});
     else
       Load = Builder.CreateIntrinsic(
           Intrinsic::arm_mve_vldr_gather_offset,
-          {Ty, BasePtr->getType(), Offsets->getType()},
-          {BasePtr, Offsets, Builder.getInt32(Ty->getScalarSizeInBits()),
-           Builder.getInt32(Scale), Builder.getInt32(1)});
+          {ResultTy, BasePtr->getType(), Offsets->getType()},
+          {BasePtr, Offsets, Builder.getInt32(OriginalTy->getScalarSizeInBits()),
+           Builder.getInt32(Scale), Builder.getInt32(Unsigned)});
   }
 
   if (!isa<UndefValue>(PassThru) && !match(PassThru, m_Zero())) {
     LLVM_DEBUG(dbgs() << "masked gathers: found non-trivial passthru - "
                       << "creating select\n");
     Load = Builder.CreateSelect(Mask, Load, PassThru);
   }
-
-  LLVM_DEBUG(dbgs() << "masked gathers: successfully built masked gather\n");
-  I->replaceAllUsesWith(Load);
+  Root->replaceAllUsesWith(Load);
+  Root->eraseFromParent();
+  if (Root != I)
+    // If this was an extending gather, also get rid of the gather instruction
+    // itself, to avoid reevaluation
     I->eraseFromParent();
+  LLVM_DEBUG(dbgs() << "masked gathers: successfully built masked gather\n");
   return true;
 }
 
 bool MVEGatherScatterLowering::runOnFunction(Function &F) {
   if (!EnableMaskedGatherScatters)
     return false;
   auto &TPC = getAnalysis<TargetPassConfig>();
   auto &TM = TPC.getTM<TargetMachine>();
   auto *ST = &TM.getSubtarget<ARMSubtarget>(F);
   if (!ST->hasMVEIntegerOps())
     return false;
   SmallVector<IntrinsicInst *, 4> Gathers;
   for (BasicBlock &BB : F) {
     for (Instruction &I : BB) {
       IntrinsicInst *II = dyn_cast<IntrinsicInst>(&I);
       if (II && II->getIntrinsicID() == Intrinsic::masked_gather)
         Gathers.push_back(II);
     }
   }
 
   if (Gathers.empty())
     return false;
 
   for (IntrinsicInst *I : Gathers)
     LowerGather(I);
 
   return true;
 }

llvm/test/CodeGen/Thumb2/mve-gather-ind16-unscaled.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -enable-arm-maskedgatscat %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc <8 x i16> @zext_unscaled_i8_i16(i8* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: zext_unscaled_i8_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r5, s0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrb.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    ldrb.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrb r5, [r5]
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.16 q0[0], r5
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.16 q0[1], lr
-; CHECK-NEXT:    ldrb r4, [r4]
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov.16 q0[3], r12
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrb.u16 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %gather = call <8 x i8> @llvm.masked.gather.v8i8.v8p0i8(<8 x i8*> %ptrs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %gather.zext = zext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %gather.zext
 }
 
 define arm_aapcs_vfpcc <8 x i16> @sext_unscaled_i8_i16(i8* %base, <8 x i16>* %offptr) {
 ; CHECK-LABEL: sext_unscaled_i8_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vldrh.u32 q1, [r1, #8]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r5, s0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrb.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    ldrb.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrb r5, [r5]
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.16 q0[0], r5
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.16 q0[1], lr
-; CHECK-NEXT:    ldrb r4, [r4]
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov.16 q0[3], r12
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrh.u16 q1, [r1]
+; CHECK-NEXT:    vldrb.s16 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i16>, <8 x i16>* %offptr, align 2
   %offs.zext = zext <8 x i16> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %gather = call <8 x i8> @llvm.masked.gather.v8i8.v8p0i8(<8 x i8*> %ptrs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %gather.sext = sext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %gather.sext
 }
entry:
   %ptrs = bitcast <8 x i8*> %byte_ptrs to <8 x half*>
   %gather = call <8 x half> @llvm.masked.gather.v8f16.v8p0f16(<8 x half*> %ptrs, i32 2, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x half> undef)
   ret <8 x half> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i16> @zext_unsigned_unscaled_i8_i8(i8* %base, <8 x i8>* %offptr) {
 ; CHECK-LABEL: zext_unsigned_unscaled_i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vldrb.u32 q1, [r1, #4]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r5, s0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrb.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    ldrb.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrb r5, [r5]
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.16 q0[0], r5
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.16 q0[1], lr
-; CHECK-NEXT:    ldrb r4, [r4]
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov.16 q0[3], r12
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrb.u16 q1, [r1]
+; CHECK-NEXT:    vldrb.u16 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i8>, <8 x i8>* %offptr, align 1
   %offs.zext = zext <8 x i8> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %gather = call <8 x i8> @llvm.masked.gather.v8i8.v8p0i8(<8 x i8*> %ptrs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %gather.zext = zext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %gather.zext
 }
 
 define arm_aapcs_vfpcc <8 x i16> @sext_unsigned_unscaled_i8_i8(i8* %base, <8 x i8>* %offptr) {
 ; CHECK-LABEL: sext_unsigned_unscaled_i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vldrb.u32 q1, [r1, #4]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vadd.i32 q1, q1, r0
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r5, s0
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vmov r1, s5
-; CHECK-NEXT:    vmov r4, s7
-; CHECK-NEXT:    ldrb.w r12, [r2]
-; CHECK-NEXT:    vmov r2, s2
-; CHECK-NEXT:    ldrb.w lr, [r3]
-; CHECK-NEXT:    vmov r3, s6
-; CHECK-NEXT:    ldrb r5, [r5]
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.16 q0[0], r5
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.16 q0[1], lr
-; CHECK-NEXT:    ldrb r4, [r4]
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.16 q0[2], r2
-; CHECK-NEXT:    vmov.16 q0[3], r12
-; CHECK-NEXT:    vmov.16 q0[4], r0
-; CHECK-NEXT:    vmov.16 q0[5], r1
-; CHECK-NEXT:    vmov.16 q0[6], r3
-; CHECK-NEXT:    vmov.16 q0[7], r4
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vldrb.u16 q1, [r1]
+; CHECK-NEXT:    vldrb.s16 q0, [r0, q1]
+; CHECK-NEXT:    bx lr
 entry:
   %offs = load <8 x i8>, <8 x i8>* %offptr, align 1
   %offs.zext = zext <8 x i8> %offs to <8 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <8 x i32> %offs.zext
   %gather = call <8 x i8> @llvm.masked.gather.v8i8.v8p0i8(<8 x i8*> %ptrs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %gather.sext = sext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %gather.sext
 }

llvm/test/CodeGen/Thumb2/mve-gather-ind32-unscaled.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -enable-arm-maskedgatscat %s -o 2>/dev/null - | FileCheck %s
 
 define arm_aapcs_vfpcc <4 x i32> @zext_unscaled_i8_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: zext_unscaled_i8_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r1
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r0
-; CHECK-NEXT:    vand q0, q0, q1
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrb.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.zext = zext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_unscaled_i8_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: sext_unscaled_i8_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrb.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_unscaled_i16_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: zext_unscaled_i16_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrh.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.zext = zext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_unscaled_i16_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: sext_unscaled_i16_i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vldrh.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
entry:
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_signed_unscaled_i16_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: zext_signed_unscaled_i16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrh.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.zext = zext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_signed_unscaled_i16_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: sext_signed_unscaled_i16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrh.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_unsigned_unscaled_i16_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: zext_unsigned_unscaled_i16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrh.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.zext = zext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_unsigned_unscaled_i16_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: sext_unsigned_unscaled_i16_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrh.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_signed_unscaled_i8_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: zext_signed_unscaled_i8_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r1
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r0
-; CHECK-NEXT:    vand q0, q0, q1
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrb.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.zext = zext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_signed_unscaled_i8_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: sext_signed_unscaled_i8_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrh.s32 q1, [r1]
+; CHECK-NEXT:    vldrb.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_unsigned_unscaled_i8_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: zext_unsigned_unscaled_i8_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r1
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r0
-; CHECK-NEXT:    vand q0, q0, q1
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrb.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.zext = zext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_unsigned_unscaled_i8_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: sext_unsigned_unscaled_i8_i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrh.u32 q1, [r1]
+; CHECK-NEXT:    vldrb.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
entry:
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_signed_unscaled_i16_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: zext_signed_unscaled_i16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrh.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.zext = zext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_signed_unscaled_i16_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: sext_signed_unscaled_i16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrh.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_unsigned_unscaled_i16_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: zext_unsigned_unscaled_i16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrh.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.zext = zext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_unsigned_unscaled_i16_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: sext_unsigned_unscaled_i16_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrh.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i16*>
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %ptrs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_signed_unscaled_i8_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: zext_signed_unscaled_i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r1
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r0
-; CHECK-NEXT:    vand q0, q0, q1
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrb.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.zext = zext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_signed_unscaled_i8_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: sext_signed_unscaled_i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrb.s32 q1, [r1]
+; CHECK-NEXT:    vldrb.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @zext_unsigned_unscaled_i8_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: zext_unsigned_unscaled_i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r1
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r0
-; CHECK-NEXT:    vand q0, q0, q1
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrb.u32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.zext = zext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.zext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @sext_unsigned_unscaled_i8_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: sext_unsigned_unscaled_i8_i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrb.u32 q1, [r1]
+; CHECK-NEXT:    vldrb.s32 q0, [r0, q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %ptrs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext

llvm/test/CodeGen/Thumb2/mve-gather-ptrs.ll

entry:
   %gather = call <2 x i16> @llvm.masked.gather.v2i16.v2p0i16(<2 x i16*> %offs, i32 2, <2 x i1> <i1 true, i1 true>, <2 x i16> undef)
   %ext = zext <2 x i16> %gather to <2 x i32>
   ret <2 x i32> %ext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i16_sext(<4 x i16*>* %offptr) {
 ; CHECK-LABEL: ptr_v4i16_sext:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16*>, <4 x i16*>* %offptr, align 4
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %offs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %ext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %ext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i16_zext(<4 x i16*>* %offptr) {
 ; CHECK-LABEL: ptr_v4i16_zext:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r0, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r1, s3
-; CHECK-NEXT:    ldrh r2, [r2]
-; CHECK-NEXT:    ldrh r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrh r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r0
-; CHECK-NEXT:    ldrh r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r1
-; CHECK-NEXT:    vmovlb.u16 q0, q0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16*>, <4 x i16*>* %offptr, align 4
   %gather = call <4 x i16> @llvm.masked.gather.v4i16.v4p0i16(<4 x i16*> %offs, i32 2, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i16> undef)
   %ext = zext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %ext
 }
 
entry:
   %gather = call <8 x i8> @llvm.masked.gather.v8i8.v8p0i8(<8 x i8*> %offs, i32 1, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x i8> undef)
   %ext = zext <8 x i8> %gather to <8 x i16>
   ret <8 x i16> %ext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i8_sext32(<4 x i8*>* %offptr) {
 ; CHECK-LABEL: ptr_v4i8_sext32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q0, q0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8*>, <4 x i8*>* %offptr, align 4
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %offs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %ext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %ext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i8_zext32(<4 x i8*>* %offptr) {
 ; CHECK-LABEL: ptr_v4i8_zext32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    vmov.i32 q1, #0xff
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s1
-; CHECK-NEXT:    vmov r3, s2
-; CHECK-NEXT:    vmov r0, s3
-; CHECK-NEXT:    ldrb r2, [r2]
-; CHECK-NEXT:    ldrb r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[0], r2
-; CHECK-NEXT:    ldrb r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[1], r1
-; CHECK-NEXT:    ldrb r0, [r0]
-; CHECK-NEXT:    vmov.32 q0[2], r3
-; CHECK-NEXT:    vmov.32 q0[3], r0
-; CHECK-NEXT:    vand q0, q0, q1
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8*>, <4 x i8*>* %offptr, align 4
   %gather = call <4 x i8> @llvm.masked.gather.v4i8.v4p0i8(<4 x i8*> %offs, i32 1, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i8> undef)
   %ext = zext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %ext
 }