[ARM][MVE] Enable masked gathers from vector of pointers

[anwel]

Summary

Adds a pass to the ARM backend that takes a v4i32 gather and transforms it into a call of arm's masked gather intrinsic.

There are limited legal addressing forms available for MVE; whether a gather is legal depends on the address as well as the gather itself. This makes legalising in ISel difficult. We thus handle this in a FunctionPass, creating intrinsics. This allows us to deal with the addressing modes and at the same time optimise the code to use more efficient instructions where possible.

[dmgreen]

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

547

Alignment >= 4

552

We should add the "ST->hasMVEIntegerOps()" check here too.

568

I think it would be useful to add debug comments for what is happening, which will become more useful as we add more variants in here. Especially around why and where we have decided not to transform.

598

You can drop the brackets if there is only a single statement in the if block.

[dmgreen]

llvm/include/llvm/Analysis/TargetTransformInfo.h

610

Also, can you add some documentation for the ModifiedDT bool. It is a little bit of a odd name, but it essentially means "we might have modified anything else, not just the intrinsic, please start again."

[anwel]

Adopt the changes suggested in comments and give a more detailed summary of the patch and the motivation to do it this way.

[anwel]

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

568

Yes, indeed. I have added some debug information.

[RKSimon]

llvm/include/llvm/Analysis/TargetTransformInfo.h

611

Its unusual to put transformational methods inside TargetTransformInfo - that tends to be legality/cost checks, not IR transforms.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

535

isa<VectorType>(Ty)

[dmgreen]

llvm/include/llvm/Analysis/TargetTransformInfo.h

611

Oh yeah. We were thinking of the interleaved load lowering that we do, but now that I look again that happens in TLI. I had not realized.

Because of the amount of code this might turn into (and how long TLI is already), it might be better as a separate pass though.

We will look into this after the hols. Thanks for the comments.

[anwel]

llvm/include/llvm/Analysis/TargetTransformInfo.h

611

You are right, putting the transforming code into a pass looks a lot cleaner and should also make future additions easier and more localized. I'll update the diff with a new version.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

535

Yes :)

[anwel]

Moved the transformation of the gather from ARMTargetTransformInfo to a separate FunctionPass.

[dmgreen]

Nice, thanks!

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

1 ↗

(On Diff #236330)

This looks a touch too short, unless my eyes deceive me.

15 ↗

(On Diff #236330)

It's customary in llvm to put std headers below the llvm headers.

86 ↗

(On Diff #236330)

Any function that is only used in this file can be static.

178 ↗

(On Diff #236330)

This isn't used it seems.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

1 ↗

(On Diff #236330)

Yes, you're right

15 ↗

(On Diff #236330)

Okay in that case I'll of course follow the standard format

86 ↗

(On Diff #236330)

Do you mean private?

178 ↗

(On Diff #236330)

No, sorry. I'll remove it.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

86 ↗

(On Diff #236330)

This is global scope I think? So the function can be static. Same for LowerGather below.

[dmgreen]

Thanks. LGTM, if no-one else has any other comments.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

532

Maybe expand this a little to "For MVE, we have a custom lowering pass that will already have custom legalised any gathers that we can to mve intrinsics, and want to expand all the rest. The pass runs before the masked intrinsic lowering pass, so if we are here, we know we want to expand."

[SjoerdMeijer]

looks good to me too, other than the irrelevant nits, I have one question though, see inlined.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

527

nit: should it be called "at" or "in" 2 places?

532

nit: mve -> MVE

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

86 ↗

(On Diff #236363)

nit: indentation

96 ↗

(On Diff #236363)

nit: mve -> MVE

108 ↗

(On Diff #236363)

nit: are we checking types too? Just omit "or vector type"?

117 ↗

(On Diff #236363)

is same -> is the same

147 ↗

(On Diff #236363)

I might be wrong, but it looks like Load is always set here? Do we need to check it here?

153 ↗

(On Diff #236363)

and this is never reached? Which makes sense probably, because we should be able to lower it?

[anwel]

Changed the comment in ARMTTIImpl::isLegalMaskedGather to give a better description, and addressed the nits

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

108 ↗

(On Diff #236363)

Yes, we are actually checking both, so I changed the function name to reflect that.

147 ↗

(On Diff #236363)

You are right, the checks for Load are superfluous

86 ↗

(On Diff #236330)

Ah, ok.

[SjoerdMeijer]

A last question/request, I guess we need 2 more tests? For completeness probably best to have a test with -enable-arm-maskedgatscat=false and another one with -mattr=-mve?

[anwel]

Yes, that's a good idea. Better safe then sorry. I added a test that makes sure that the gathers are not constructed if the mve or enable-arm-maskedgatscat option is not set

[SjoerdMeijer]

Thanks, LGTM again

llvm/include/llvm/Analysis/TargetTransformInfo.h

public:
   /// Return true if the target supports masked gather.
   bool isLegalMaskedGather(Type *DataType, MaybeAlign Alignment) const;
 
   /// Return true if the target supports masked compress store.
   bool isLegalMaskedCompressStore(Type *DataType) const;
   /// Return true if the target supports masked expand load.
   bool isLegalMaskedExpandLoad(Type *DataType) const;
 
+  /// Give the target a chance to custom legalise a masked gather whilst it is
+  /// still IR. Return true if legalised.
+  bool customLegalizeMaskedGather(IntrinsicInst *I, bool &ModifiedDT) const;
+
   /// Return true if the target has a unified operation to calculate division
   /// and remainder. If so, the additional implicit multiplication and
   /// subtraction required to calculate a remainder from division are free. This
   /// can enable more aggressive transformations for division and remainder than
   /// would typically be allowed using throughput or size cost models.
   bool hasDivRemOp(Type *DataType, bool IsSigned) const;
 
   /// Return true if the given instruction (assumed to be a memory access
public:
   virtual bool isLegalMaskedStore(Type *DataType, MaybeAlign Alignment) = 0;
   virtual bool isLegalMaskedLoad(Type *DataType, MaybeAlign Alignment) = 0;
   virtual bool isLegalNTStore(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalNTLoad(Type *DataType, Align Alignment) = 0;
   virtual bool isLegalMaskedScatter(Type *DataType, MaybeAlign Alignment) = 0;
   virtual bool isLegalMaskedGather(Type *DataType, MaybeAlign Alignment) = 0;
   virtual bool isLegalMaskedCompressStore(Type *DataType) = 0;
   virtual bool isLegalMaskedExpandLoad(Type *DataType) = 0;
+  virtual bool customLegalizeMaskedGather(IntrinsicInst *I, bool &ModifiedDT) = 0;
   virtual bool hasDivRemOp(Type *DataType, bool IsSigned) = 0;
   virtual bool hasVolatileVariant(Instruction *I, unsigned AddrSpace) = 0;
   virtual bool prefersVectorizedAddressing() = 0;
   virtual int getScalingFactorCost(Type *Ty, GlobalValue *BaseGV,
                                    int64_t BaseOffset, bool HasBaseReg,
                                    int64_t Scale, unsigned AddrSpace) = 0;
   virtual bool LSRWithInstrQueries() = 0;
   virtual bool isTruncateFree(Type *Ty1, Type *Ty2) = 0;
bool isLegalMaskedGather(Type *DataType, MaybeAlign Alignment) override {
     return Impl.isLegalMaskedGather(DataType, Alignment);
   }
   bool isLegalMaskedCompressStore(Type *DataType) override {
     return Impl.isLegalMaskedCompressStore(DataType);
   }
   bool isLegalMaskedExpandLoad(Type *DataType) override {
     return Impl.isLegalMaskedExpandLoad(DataType);
   }
+  bool customLegalizeMaskedGather(IntrinsicInst *I, bool &ModifiedDT) override {
+    return Impl.customLegalizeMaskedGather(I, ModifiedDT);
+  }
   bool hasDivRemOp(Type *DataType, bool IsSigned) override {
     return Impl.hasDivRemOp(DataType, IsSigned);
   }
   bool hasVolatileVariant(Instruction *I, unsigned AddrSpace) override {
     return Impl.hasVolatileVariant(I, AddrSpace);
   }
   bool prefersVectorizedAddressing() override {
     return Impl.prefersVectorizedAddressing();

llvm/include/llvm/Analysis/TargetTransformInfoImpl.h

public:
   bool isLegalMaskedGather(Type *DataType, MaybeAlign Alignment) {
     return false;
   }
 
   bool isLegalMaskedCompressStore(Type *DataType) { return false; }
 
   bool isLegalMaskedExpandLoad(Type *DataType) { return false; }
 
+  bool customLegalizeMaskedGather(IntrinsicInst *I, bool &ModifiedDT) { return false; }
+
   bool hasDivRemOp(Type *DataType, bool IsSigned) { return false; }
 
   bool hasVolatileVariant(Instruction *I, unsigned AddrSpace) { return false; }
 
   bool prefersVectorizedAddressing() { return true; }
 
   int getScalingFactorCost(Type *Ty, GlobalValue *BaseGV, int64_t BaseOffset,
                            bool HasBaseReg, int64_t Scale, unsigned AddrSpace) {

llvm/lib/Analysis/TargetTransformInfo.cpp

 bool TargetTransformInfo::isLegalMaskedCompressStore(Type *DataType) const {
   return TTIImpl->isLegalMaskedCompressStore(DataType);
 }
 
 bool TargetTransformInfo::isLegalMaskedExpandLoad(Type *DataType) const {
   return TTIImpl->isLegalMaskedExpandLoad(DataType);
 }
 
+bool TargetTransformInfo::customLegalizeMaskedGather(IntrinsicInst *I,
+    bool &ModifiedDT) const {
+  return TTIImpl->customLegalizeMaskedGather(I, ModifiedDT);
+}
+
 bool TargetTransformInfo::hasDivRemOp(Type *DataType, bool IsSigned) const {
   return TTIImpl->hasDivRemOp(DataType, IsSigned);
 }
 
 bool TargetTransformInfo::hasVolatileVariant(Instruction *I,
                                              unsigned AddrSpace) const {
   return TTIImpl->hasVolatileVariant(I, AddrSpace);
 }

llvm/lib/CodeGen/ScalarizeMaskedMemIntrin.cpp

case Intrinsic::masked_store: {
       if (TTI->isLegalMaskedStore(CI->getArgOperand(0)->getType(),
                                   MaybeAlign(Alignment)))
         return false;
       scalarizeMaskedStore(CI, ModifiedDT);
       return true;
     }
     case Intrinsic::masked_gather:
       Alignment = cast<ConstantInt>(CI->getArgOperand(1))->getZExtValue();
+      if (TTI->customLegalizeMaskedGather(II, ModifiedDT))
+        return true;
       if (TTI->isLegalMaskedGather(CI->getType(), MaybeAlign(Alignment)))
         return false;
       scalarizeMaskedGather(CI, ModifiedDT);
       return true;
     case Intrinsic::masked_scatter:
       Alignment = cast<ConstantInt>(CI->getArgOperand(2))->getZExtValue();
       if (TTI->isLegalMaskedScatter(CI->getArgOperand(0)->getType(),
                                     MaybeAlign(Alignment)))

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

public:
   }
 
   bool isLegalMaskedLoad(Type *DataTy, MaybeAlign Alignment);
 
   bool isLegalMaskedStore(Type *DataTy, MaybeAlign Alignment) {
     return isLegalMaskedLoad(DataTy, Alignment);
   }
 
-  bool isLegalMaskedGather(Type *Ty, MaybeAlign Alignment) { return false; }
+  bool isLegalMaskedGather(Type *Ty, MaybeAlign Alignment);
 
   bool isLegalMaskedScatter(Type *Ty, MaybeAlign Alignment) { return false; }
 
+  bool customLegalizeMaskedGather(IntrinsicInst *I, bool &ModifiedDT) const;
+
   int getMemcpyCost(const Instruction *I);
 
   int getShuffleCost(TTI::ShuffleKind Kind, Type *Tp, int Index, Type *SubTp);
 
   bool useReductionIntrinsic(unsigned Opcode, Type *Ty,
                              TTI::ReductionFlags Flags) const;
 
   bool shouldExpandReduction(const IntrinsicInst *II) const {

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/IntrinsicsARM.h"
+#include "llvm/IR/PatternMatch.h"
 #include "llvm/IR/Type.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/MachineValueType.h"
 #include "llvm/Target/TargetMachine.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <utility>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "armtti"
 
 static cl::opt<bool> EnableMaskedLoadStores(
   "enable-arm-maskedldst", cl::Hidden, cl::init(true),
   cl::desc("Enable the generation of masked loads and stores"));
 
+static cl::opt<bool> EnableMaskedGatherScatters(
+    "enable-arm-maskedgatscat", cl::Hidden, cl::init(false),
+    cl::desc("Enable the generation of masked gathers and scatters"));
+
 static cl::opt<bool> DisableLowOverheadLoops(
   "disable-arm-loloops", cl::Hidden, cl::init(false),
   cl::desc("Disable the generation of low-overhead loops"));
 
 extern cl::opt<bool> DisableTailPredication;
 
 bool ARMTTIImpl::areInlineCompatible(const Function *Caller,
                                      const Function *Callee) const {
bool ARMTTIImpl::isLegalMaskedLoad(Type *DataTy, MaybeAlign Alignment) {
   }
 
   unsigned EltWidth = DataTy->getScalarSizeInBits();
   return (EltWidth == 32 && (!Alignment || Alignment >= 4)) ||
          (EltWidth == 16 && (!Alignment || Alignment >= 2)) ||
          (EltWidth == 8);
 }
 
+bool ARMTTIImpl::isLegalMaskedGather(Type *Ty, MaybeAlign Alignment) {
+  if (!EnableMaskedGatherScatters || !ST->hasMVEIntegerOps())
+    return false;
+
+  // This method is called to 2 places
+  //  - once from the vectorizer with a scalar type, in which case we need to
+  //  get this as good as we can with the limited info we have (and rely on
+  //  the cot model for the rest).
+  //  - and also from the masked intrinsic lowering pass with the actual type.
+  //  For MVE, we want to custom legalise any gathers that we can to mve
+  //  intrinsics, and expand all the rest. So if we are here, we know we want
+  //  to expand.
+  if (dyn_cast<VectorType>(Ty)) {
+    return false;
+  }
+
+  unsigned EltWidth = Ty->getScalarSizeInBits();
+  return ((EltWidth == 32 && (!Alignment || Alignment >= 4)) ||
+          (EltWidth == 16 && (!Alignment || Alignment >= 2)) || EltWidth == 8);
+}
+
+bool isLegalAlignment(unsigned NumElements, unsigned ElemSize,
+                      unsigned Alignment) {
+  // Do only allow non-extending v4i32 gathers for now
+  return NumElements == 4 && ElemSize == 32 && Alignment >= 4;
+}
+
+bool ARMTTIImpl::customLegalizeMaskedGather(IntrinsicInst *I,
+                                            bool &ModifiedDT) const {
+  if (!EnableMaskedGatherScatters || !ST->hasMVEIntegerOps())
+    return false;
+  using namespace PatternMatch;
+  LLVM_DEBUG(dbgs() << "masked gathers: checking transform preconditions\n");
+
+  // @llvm.masked.gather.*(Ptrs, alignment, Mask, Src0)
+  // Attempt to turn the masked gather in I into a mve intrinsic
+  // Potentially optimising the addressing modes as we do so.
+  Type *Ty = I->getType();
+  Value *Ptr = I->getArgOperand(0);
+  unsigned Alignment = cast<ConstantInt>(I->getArgOperand(1))->getZExtValue();
+  Value *Mask = I->getArgOperand(2);
+  Value *PassThru = I->getArgOperand(3);
+
+  // Check this is a valid gather with correct alignment
+  if (!isLegalAlignment(Ty->getVectorNumElements(),
+                        Ty->getScalarSizeInBits(), Alignment)) {
+    LLVM_DEBUG(dbgs() << "masked gathers: instruction does not have valid "
+                      << "alignment or vector type \n");
+    return false;
+  }
+
+  IRBuilder<> Builder(I->getContext());
+  Builder.SetInsertPoint(I);
+  Builder.SetCurrentDebugLocation(I->getDebugLoc());
+
+  Value *Load = nullptr;
+  // Look through bitcast instruction if #elements is same
+  if (auto *BitCast = dyn_cast<BitCastInst>(Ptr)) {
+    Type *BCTy = BitCast->getType();
+    Type *BCSrcTy = BitCast->getOperand(0)->getType();
+    if (BCTy->getVectorNumElements() == BCSrcTy->getVectorNumElements()) {
+      LLVM_DEBUG(dbgs() << "masked gathers: looking through bitcast\n");
+      Ptr = BitCast->getOperand(0);
+    }
+  }
+  assert(Ptr->getType()->isVectorTy() && "Unexpected pointer type");
+
+  if (Ty->getVectorNumElements() != 4)
+    // Can't build an intrinsic for this
+    return false;
+  if (match(Mask, m_One()))
+    Load = Builder.CreateIntrinsic(Intrinsic::arm_mve_vldr_gather_base,
+                                   {Ty, Ptr->getType()},
+                                   {Ptr, Builder.getInt32(0)});
+  else
+    Load = Builder.CreateIntrinsic(
+        Intrinsic::arm_mve_vldr_gather_base_predicated,
+        {Ty, Ptr->getType(), Mask->getType()},
+        {Ptr, Builder.getInt32(0), Mask});
+
+  if (Load && !isa<UndefValue>(PassThru) && !match(PassThru, m_Zero())) {
+    LLVM_DEBUG(dbgs() << "masked gathers: found non-trivial passthru - "
+                      << "creating select\n");
+    Load = Builder.CreateSelect(Mask, Load, PassThru);
+  }
+
+  if (Load) {
+    LLVM_DEBUG(dbgs() << "masked gathers: successfully built masked gather\n");
+    I->replaceAllUsesWith(Load);
+    I->eraseFromParent();
+    return true;
+  }
+  LLVM_DEBUG(dbgs() << "masked gathers: could not build masked gather\n");
+  return false;
+}
+
 int ARMTTIImpl::getMemcpyCost(const Instruction *I) {
   const MemCpyInst *MI = dyn_cast<MemCpyInst>(I);
   assert(MI && "MemcpyInst expected");
   ConstantInt *C = dyn_cast<ConstantInt>(MI->getLength());
 
   // To model the cost of a library call, we assume 1 for the call, and
   // 3 for the argument setup.
   const unsigned LibCallCost = 4;

llvm/test/CodeGen/Thumb2/mve-gather-ind32-scaled.ll

entry:
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @scaled_i32_i32(i32* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: scaled_i32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 ; TODO: scaled_f16_i32
 
 define arm_aapcs_vfpcc <4 x float> @scaled_f32_i32(i32* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: scaled_f32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_scaled_b_i32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_b_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_scaled_i32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: signed_scaled_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_scaled_f32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: a_unsigned_scaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_scaled_f32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: b_signed_scaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
entry:
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_scaled_b_i32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_b_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_scaled_i32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: signed_scaled_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_scaled_f32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: a_unsigned_scaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_scaled_f32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: b_signed_scaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather

llvm/test/CodeGen/Thumb2/mve-gather-ind32-unscaled.ll

entry:
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unscaled_i32_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: unscaled_i32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @unscaled_f32_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: unscaled_f32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_unscaled_b_i32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_b_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_unscaled_i32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: signed_unscaled_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_unscaled_f32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: a_unsigned_unscaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_unscaled_f32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: b_signed_unscaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
entry:
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_unscaled_b_i32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_b_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_unscaled_i32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: signed_unscaled_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_unscaled_f32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: a_unsigned_unscaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_unscaled_f32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: b_signed_unscaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather

llvm/test/CodeGen/Thumb2/mve-gather-ptrs.ll

entry:
   %offs = load <2 x i32*>, <2 x i32*>* %offptr, align 4
   %gather = call <2 x i32> @llvm.masked.gather.v2i32.v2p0i32(<2 x i32*> %offs, i32 4, <2 x i1> <i1 true, i1 true>, <2 x i32> undef)
   ret <2 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i32(<4 x i32*>* %offptr) {
 ; CHECK-LABEL: ptr_v4i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32*>, <4 x i32*>* %offptr, align 4
   %gather = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> %offs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i32> @ptr_v8i32(<8 x i32*>* %offptr) {
entry:
   %gather = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> %offs, i32 4, <2 x i1> <i1 true, i1 true>, <2 x float> undef)
   ret <2 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @ptr_v4f32(<4 x float*>* %offptr) {
 ; CHECK-LABEL: ptr_v4f32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x float*>, <4 x float*>* %offptr, align 4
   %gather = call <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*> %offs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <8 x float> @ptr_v8f32(<8 x float*>* %offptr) {

 
 ; loops
 
 define void @foo_ptr_p_int32_t(i32* %dest, i32** %src, i32 %n) {
 ; CHECK-LABEL: foo_ptr_p_int32_t:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    bic r2, r2, #15
 ; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB22_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.body.preheader
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    poplt {r7, pc}
 ; CHECK-NEXT:    subs r2, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    sub.w r12, r1, #16
+; CHECK-NEXT:    subs r1, #16
 ; CHECK-NEXT:    add.w lr, r3, r2, lsr #2
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB22_2: @ %vector.body
+; CHECK-NEXT:  .LBB22_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q0, [r12, #16]
-; CHECK-NEXT:    @ implicit-def: $q1
-; CHECK-NEXT:    add.w r12, r12, #16
-; CHECK-NEXT:    vcmp.i32 ne, q0, zr
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r1, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r1, #0, #1
-; CHECK-NEXT:    ubfx r1, r3, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, r3, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, r3, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #3, #1
-; CHECK-NEXT:    lsls r1, r2, #31
-; CHECK-NEXT:    ittt ne
-; CHECK-NEXT:    vmovne r1, s0
-; CHECK-NEXT:    ldrne r1, [r1]
-; CHECK-NEXT:    vmovne.32 q1[0], r1
-; CHECK-NEXT:    lsls r1, r2, #30
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s1
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[1], r1
-; CHECK-NEXT:    lsls r1, r2, #29
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s2
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[2], r1
-; CHECK-NEXT:    lsls r1, r2, #28
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[3], r1
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrw.u32 q0, [r1, #16]!
+; CHECK-NEXT:    vptt.i32 ne, q0, zr
+; CHECK-NEXT:    vldrwt.u32 q1, [q0]
 ; CHECK-NEXT:    vstrwt.32 q1, [r0], #16
-; CHECK-NEXT:    le lr, .LBB22_2
-; CHECK-NEXT:  .LBB22_3: @ %for.end
-; CHECK-NEXT:    add sp, #4
+; CHECK-NEXT:    le lr, .LBB22_1
+; CHECK-NEXT:  @ %bb.2: @ %for.end
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %and = and i32 %n, -16
   %cmp11 = icmp sgt i32 %and, 0
   br i1 %cmp11, label %vector.body, label %for.end
 
 vector.body:                                      ; preds = %entry, %vector.body
   %index = phi i32 [ %index.next, %vector.body ], [ 0, %entry ]
for.end:                                          ; preds = %vector.body, %entry
   ret void
 }
 
 define void @foo_ptr_p_float(float* %dest, float** %src, i32 %n) {
 ; CHECK-LABEL: foo_ptr_p_float:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    bic r2, r2, #15
 ; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB23_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.body.preheader
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    poplt {r7, pc}
 ; CHECK-NEXT:    subs r2, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    sub.w r12, r1, #16
+; CHECK-NEXT:    subs r1, #16
 ; CHECK-NEXT:    add.w lr, r3, r2, lsr #2
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB23_2: @ %vector.body
+; CHECK-NEXT:  .LBB23_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q0, [r12, #16]
-; CHECK-NEXT:    @ implicit-def: $q1
-; CHECK-NEXT:    add.w r12, r12, #16
-; CHECK-NEXT:    vcmp.i32 ne, q0, zr
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r1, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r1, #0, #1
-; CHECK-NEXT:    ubfx r1, r3, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, r3, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, r3, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #3, #1
-; CHECK-NEXT:    lsls r1, r2, #31
-; CHECK-NEXT:    ittt ne
-; CHECK-NEXT:    vmovne r1, s0
-; CHECK-NEXT:    ldrne r1, [r1]
-; CHECK-NEXT:    vmovne.32 q1[0], r1
-; CHECK-NEXT:    lsls r1, r2, #30
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s1
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[1], r1
-; CHECK-NEXT:    lsls r1, r2, #29
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s2
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[2], r1
-; CHECK-NEXT:    lsls r1, r2, #28
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[3], r1
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrw.u32 q0, [r1, #16]!
+; CHECK-NEXT:    vptt.i32 ne, q0, zr
+; CHECK-NEXT:    vldrwt.u32 q1, [q0]
 ; CHECK-NEXT:    vstrwt.32 q1, [r0], #16
-; CHECK-NEXT:    le lr, .LBB23_2
-; CHECK-NEXT:  .LBB23_3: @ %for.end
-; CHECK-NEXT:    add sp, #4
+; CHECK-NEXT:    le lr, .LBB23_1
+; CHECK-NEXT:  @ %bb.2: @ %for.end
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %and = and i32 %n, -16
   %cmp11 = icmp sgt i32 %and, 0
   br i1 %cmp11, label %vector.body, label %for.end
 
 vector.body:                                      ; preds = %entry, %vector.body
   %index = phi i32 [ %index.next, %vector.body ], [ 0, %entry ]