[ARM][MVE] Enable masked gathers from vector of pointers

[anwel]

Summary

Adds a pass to the ARM backend that takes a v4i32 gather and transforms it into a call of arm's masked gather intrinsic.

There are limited legal addressing forms available for MVE; whether a gather is legal depends on the address as well as the gather itself. This makes legalising in ISel difficult. We thus handle this in a FunctionPass, creating intrinsics. This allows us to deal with the addressing modes and at the same time optimise the code to use more efficient instructions where possible.

[dmgreen]

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

544

Alignment >= 4

549

We should add the "ST->hasMVEIntegerOps()" check here too.

565

I think it would be useful to add debug comments for what is happening, which will become more useful as we add more variants in here. Especially around why and where we have decided not to transform.

595

You can drop the brackets if there is only a single statement in the if block.

[dmgreen]

llvm/include/llvm/Analysis/TargetTransformInfo.h

610 ↗

(On Diff #234835)

Also, can you add some documentation for the ModifiedDT bool. It is a little bit of a odd name, but it essentially means "we might have modified anything else, not just the intrinsic, please start again."

[anwel]

Adopt the changes suggested in comments and give a more detailed summary of the patch and the motivation to do it this way.

[anwel]

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

565

Yes, indeed. I have added some debug information.

[RKSimon]

llvm/include/llvm/Analysis/TargetTransformInfo.h

611 ↗

(On Diff #234849)

Its unusual to put transformational methods inside TargetTransformInfo - that tends to be legality/cost checks, not IR transforms.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

532

isa<VectorType>(Ty)

[dmgreen]

llvm/include/llvm/Analysis/TargetTransformInfo.h

611 ↗

(On Diff #234849)

Oh yeah. We were thinking of the interleaved load lowering that we do, but now that I look again that happens in TLI. I had not realized.

Because of the amount of code this might turn into (and how long TLI is already), it might be better as a separate pass though.

We will look into this after the hols. Thanks for the comments.

[anwel]

llvm/include/llvm/Analysis/TargetTransformInfo.h

611 ↗

(On Diff #234849)

You are right, putting the transforming code into a pass looks a lot cleaner and should also make future additions easier and more localized. I'll update the diff with a new version.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

532

Yes :)

[anwel]

Moved the transformation of the gather from ARMTargetTransformInfo to a separate FunctionPass.

[dmgreen]

Nice, thanks!

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

2

This looks a touch too short, unless my eyes deceive me.

16

It's customary in llvm to put std headers below the llvm headers.

87

Any function that is only used in this file can be static.

179

This isn't used it seems.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

2

Yes, you're right

16

Okay in that case I'll of course follow the standard format

87

Do you mean private?

179

No, sorry. I'll remove it.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

87

This is global scope I think? So the function can be static. Same for LowerGather below.

[dmgreen]

Thanks. LGTM, if no-one else has any other comments.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

529

Maybe expand this a little to "For MVE, we have a custom lowering pass that will already have custom legalised any gathers that we can to mve intrinsics, and want to expand all the rest. The pass runs before the masked intrinsic lowering pass, so if we are here, we know we want to expand."

[SjoerdMeijer]

looks good to me too, other than the irrelevant nits, I have one question though, see inlined.

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

524

nit: should it be called "at" or "in" 2 places?

529

nit: mve -> MVE

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

86

nit: indentation

96

nit: mve -> MVE

108

nit: are we checking types too? Just omit "or vector type"?

117

is same -> is the same

147

I might be wrong, but it looks like Load is always set here? Do we need to check it here?

153

and this is never reached? Which makes sense probably, because we should be able to lower it?

[anwel]

Changed the comment in ARMTTIImpl::isLegalMaskedGather to give a better description, and addressed the nits

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

87

Ah, ok.

108

Yes, we are actually checking both, so I changed the function name to reflect that.

147

You are right, the checks for Load are superfluous

[SjoerdMeijer]

A last question/request, I guess we need 2 more tests? For completeness probably best to have a test with -enable-arm-maskedgatscat=false and another one with -mattr=-mve?

[anwel]

Yes, that's a good idea. Better safe then sorry. I added a test that makes sure that the gathers are not constructed if the mve or enable-arm-maskedgatscat option is not set

[SjoerdMeijer]

Thanks, LGTM again

llvm/lib/Target/ARM/ARM.h

 FunctionPass *createThumb2ITBlockPass();
 FunctionPass *createMVEVPTBlockPass();
 FunctionPass *createARMOptimizeBarriersPass();
 FunctionPass *createThumb2SizeReductionPass(
     std::function<bool(const Function &)> Ftor = nullptr);
 InstructionSelector *
 createARMInstructionSelector(const ARMBaseTargetMachine &TM, const ARMSubtarget &STI,
                              const ARMRegisterBankInfo &RBI);
+Pass *createMVEGatherScatterLoweringPass();
 
 void LowerARMMachineInstrToMCInst(const MachineInstr *MI, MCInst &OutMI,
                                   ARMAsmPrinter &AP);
 
 void initializeARMParallelDSPPass(PassRegistry &);
 void initializeARMLoadStoreOptPass(PassRegistry &);
 void initializeARMPreAllocLoadStoreOptPass(PassRegistry &);
 void initializeARMConstantIslandsPass(PassRegistry &);
 void initializeARMExpandPseudoPass(PassRegistry &);
 void initializeThumb2SizeReducePass(PassRegistry &);
 void initializeThumb2ITBlockPass(PassRegistry &);
 void initializeMVEVPTBlockPass(PassRegistry &);
 void initializeARMLowOverheadLoopsPass(PassRegistry &);
 void initializeMVETailPredicationPass(PassRegistry &);
+void initializeMVEGatherScatterLoweringPass(PassRegistry &);
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_ARM_ARM_H

llvm/lib/Target/ARM/ARMTargetMachine.cpp

extern "C" void LLVMInitializeARMTarget() {
   initializeARMParallelDSPPass(Registry);
   initializeARMConstantIslandsPass(Registry);
   initializeARMExecutionDomainFixPass(Registry);
   initializeARMExpandPseudoPass(Registry);
   initializeThumb2SizeReducePass(Registry);
   initializeMVEVPTBlockPass(Registry);
   initializeMVETailPredicationPass(Registry);
   initializeARMLowOverheadLoopsPass(Registry);
+  initializeMVEGatherScatterLoweringPass(Registry);
 }
 
 static std::unique_ptr<TargetLoweringObjectFile> createTLOF(const Triple &TT) {
   if (TT.isOSBinFormatMachO())
     return std::make_unique<TargetLoweringObjectFileMachO>();
   if (TT.isOSWindows())
     return std::make_unique<TargetLoweringObjectFileCOFF>();
   return std::make_unique<ARMElfTargetObjectFile>();
void ARMPassConfig::addIRPasses() {
   // ldrex/strex loops to simplify this, but it needs tidying up.
   if (TM->getOptLevel() != CodeGenOpt::None && EnableAtomicTidy)
     addPass(createCFGSimplificationPass(
         1, false, false, true, true, [this](const Function &F) {
           const auto &ST = this->TM->getSubtarget<ARMSubtarget>(F);
           return ST.hasAnyDataBarrier() && !ST.isThumb1Only();
         }));
 
+  addPass(createMVEGatherScatterLoweringPass());
+
   TargetPassConfig::addIRPasses();
 
   // Run the parallel DSP pass.
   if (getOptLevel() == CodeGenOpt::Aggressive) 
     addPass(createARMParallelDSPPass());
 
   // Match interleaved memory accesses to ldN/stN intrinsics.
   if (TM->getOptLevel() != CodeGenOpt::None)

llvm/lib/Target/ARM/ARMTargetTransformInfo.h

public:
   }
 
   bool isLegalMaskedLoad(Type *DataTy, MaybeAlign Alignment);
 
   bool isLegalMaskedStore(Type *DataTy, MaybeAlign Alignment) {
     return isLegalMaskedLoad(DataTy, Alignment);
   }
 
-  bool isLegalMaskedGather(Type *Ty, MaybeAlign Alignment) { return false; }
+  bool isLegalMaskedGather(Type *Ty, MaybeAlign Alignment);
 
   bool isLegalMaskedScatter(Type *Ty, MaybeAlign Alignment) { return false; }
 
   int getMemcpyCost(const Instruction *I);
 
   int getShuffleCost(TTI::ShuffleKind Kind, Type *Tp, int Index, Type *SubTp);
 
   bool useReductionIntrinsic(unsigned Opcode, Type *Ty,

llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp

 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/PatternMatch.h"
 #include "llvm/IR/Type.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/MachineValueType.h"
 #include "llvm/Target/TargetMachine.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <utility>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "armtti"
 
 static cl::opt<bool> EnableMaskedLoadStores(
   "enable-arm-maskedldst", cl::Hidden, cl::init(true),
   cl::desc("Enable the generation of masked loads and stores"));
 
 static cl::opt<bool> DisableLowOverheadLoops(
   "disable-arm-loloops", cl::Hidden, cl::init(false),
   cl::desc("Disable the generation of low-overhead loops"));
 
 extern cl::opt<bool> DisableTailPredication;
 
+extern cl::opt<bool> EnableMaskedGatherScatters;
+
 bool ARMTTIImpl::areInlineCompatible(const Function *Caller,
                                      const Function *Callee) const {
   const TargetMachine &TM = getTLI()->getTargetMachine();
   const FeatureBitset &CallerBits =
       TM.getSubtargetImpl(*Caller)->getFeatureBits();
   const FeatureBitset &CalleeBits =
       TM.getSubtargetImpl(*Callee)->getFeatureBits();
 
bool ARMTTIImpl::isLegalMaskedLoad(Type *DataTy, MaybeAlign Alignment) {
   }
 
   unsigned EltWidth = DataTy->getScalarSizeInBits();
   return (EltWidth == 32 && (!Alignment || Alignment >= 4)) ||
          (EltWidth == 16 && (!Alignment || Alignment >= 2)) ||
          (EltWidth == 8);
 }
 
+bool ARMTTIImpl::isLegalMaskedGather(Type *Ty, MaybeAlign Alignment) {
+  if (!EnableMaskedGatherScatters || !ST->hasMVEIntegerOps())
+    return false;
+
+  // This method is called to 2 places
+  //  - once from the vectorizer with a scalar type, in which case we need to
+  //  get this as good as we can with the limited info we have (and rely on
+  //  the cost model for the rest).
+  //  - and also from the masked intrinsic lowering pass with the actual vector
+  //  type. For MVE, we want to custom legalise any gathers that we can to mve
+  //  intrinsics, and expand all the rest. So if we are here, we know we want
+  //  to expand.
+  if (isa<VectorType>(Ty))
+    return false;
+
+  unsigned EltWidth = Ty->getScalarSizeInBits();
+  return ((EltWidth == 32 && (!Alignment || Alignment >= 4)) ||
+          (EltWidth == 16 && (!Alignment || Alignment >= 2)) || EltWidth == 8);
+}
+
 int ARMTTIImpl::getMemcpyCost(const Instruction *I) {
   const MemCpyInst *MI = dyn_cast<MemCpyInst>(I);
   assert(MI && "MemcpyInst expected");
   ConstantInt *C = dyn_cast<ConstantInt>(MI->getLength());
 
   // To model the cost of a library call, we assume 1 for the call, and
   // 3 for the argument setup.
   const unsigned LibCallCost = 4;
 
   // If 'size' is not a constant, a library call will be generated.
   if (!C)
     return LibCallCost;
 
   const unsigned Size = C->getValue().getZExtValue();
   const unsigned DstAlign = MI->getDestAlignment();
   const unsigned SrcAlign = MI->getSourceAlignment();
   const Function *F = I->getParent()->getParent();
   const unsigned Limit = TLI->getMaxStoresPerMemmove(F->hasMinSize());
   std::vector<EVT> MemOps;
 
   // MemOps will be poplulated with a list of data types that needs to be
   // loaded and stored. That's why we multiply the number of elements by 2 to
   // get the cost for this memcpy.
   if (getTLI()->findOptimalMemOpLowering(
           MemOps, Limit, Size, DstAlign, SrcAlign, false /*IsMemset*/,
           false /*ZeroMemset*/, false /*MemcpyStrSrc*/, false /*AllowOverlap*/,
           MI->getDestAddressSpace(), MI->getSourceAddressSpace(),
           F->getAttributes()))
     return MemOps.size() * 2;
 
   // If we can't find an optimal memop lowering, return the default cost
   return LibCallCost;
 }
 
if (Kind == TTI::SK_Broadcast) {
           {ISD::VECTOR_SHUFFLE, MVT::v4i32, 1},
           {ISD::VECTOR_SHUFFLE, MVT::v4f32, 1},
           {ISD::VECTOR_SHUFFLE, MVT::v8i16, 1},
           {ISD::VECTOR_SHUFFLE, MVT::v16i8, 1}};
 
       std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, Tp);
 
       if (const auto *Entry =
               CostTableLookup(NEONDupTbl, ISD::VECTOR_SHUFFLE, LT.second))
         return LT.first * Entry->Cost;
     }
     if (Kind == TTI::SK_Reverse) {
       static const CostTblEntry NEONShuffleTbl[] = {
           // Reverse shuffle cost one instruction if we are shuffling within a
           // double word (vrev) or two if we shuffle a quad word (vrev, vext).
           {ISD::VECTOR_SHUFFLE, MVT::v2i32, 1},
           {ISD::VECTOR_SHUFFLE, MVT::v2f32, 1},

llvm/lib/Target/ARM/CMakeLists.txt

add_llvm_target(ARMCodeGen
   ARMOptimizeBarriersPass.cpp
   ARMRegisterBankInfo.cpp
   ARMSelectionDAGInfo.cpp
   ARMSubtarget.cpp
   ARMTargetMachine.cpp
   ARMTargetObjectFile.cpp
   ARMTargetTransformInfo.cpp
   MLxExpansionPass.cpp
+  MVEGatherScatterLowering.cpp
   MVETailPredication.cpp
   MVEVPTBlockPass.cpp
   Thumb1FrameLowering.cpp
   Thumb1InstrInfo.cpp
   ThumbRegisterInfo.cpp
   Thumb2ITBlockPass.cpp
   Thumb2InstrInfo.cpp
   Thumb2SizeReduction.cpp
   )
 
 add_subdirectory(AsmParser)
 add_subdirectory(Disassembler)
 add_subdirectory(MCTargetDesc)
 add_subdirectory(TargetInfo)
 add_subdirectory(Utils)

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

+//===- ARMGatherScatterLowering.cpp - Gather/Scatter lowering -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// This pass custom lowers llvm.gather and llvm.scatter instructions to
+/// arm.mve.gather and arm.mve.scatter intrinsics, optimising the code to
+/// produce a better final result as we go.
+//
+//===----------------------------------------------------------------------===//
+
+#include "ARM.h"
+#include "ARMBaseInstrInfo.h"
+#include "ARMSubtarget.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/CodeGen/TargetLowering.h"
+#include "llvm/CodeGen/TargetPassConfig.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/IR/BasicBlock.h"
+#include "llvm/IR/Constant.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/DerivedTypes.h"
+#include "llvm/IR/Function.h"
+#include "llvm/IR/InstrTypes.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Intrinsics.h"
+#include "llvm/IR/IntrinsicsARM.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/PatternMatch.h"
+#include "llvm/IR/Type.h"
+#include "llvm/IR/Value.h"
+#include "llvm/Pass.h"
+#include "llvm/Support/Casting.h"
+#include <algorithm>
+#include <cassert>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "mve-gather-scatter-lowering"
+
+cl::opt<bool> EnableMaskedGatherScatters(
+    "enable-arm-maskedgatscat", cl::Hidden, cl::init(false),
+    cl::desc("Enable the generation of masked gathers and scatters"));
+
+namespace {
+
+class MVEGatherScatterLowering : public FunctionPass {
+public:
+  static char ID; // Pass identification, replacement for typeid
+
+  explicit MVEGatherScatterLowering() : FunctionPass(ID) {
+    initializeMVEGatherScatterLoweringPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnFunction(Function &F) override;
+
+  StringRef getPassName() const override {
+    return "MVE gather/scatter lowering";
+  }
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    AU.addRequired<TargetPassConfig>();
+    FunctionPass::getAnalysisUsage(AU);
+  }
+};
+
+} // end anonymous namespace
+
+char MVEGatherScatterLowering::ID = 0;
+
+INITIALIZE_PASS(MVEGatherScatterLowering, DEBUG_TYPE,
+                "MVE gather/scattering lowering pass", false, false)
+
+Pass *llvm::createMVEGatherScatterLoweringPass() {
+  return new MVEGatherScatterLowering();
+}
+
+static bool isLegalAlignment(unsigned NumElements, unsigned ElemSize,
+                      unsigned Alignment) {
+  // Do only allow non-extending v4i32 gathers for now
+  return NumElements == 4 && ElemSize == 32 && Alignment >= 4;
+}
+
+static bool LowerGather(IntrinsicInst *I) {
+  using namespace PatternMatch;
+  LLVM_DEBUG(dbgs() << "masked gathers: checking transform preconditions\n");
+
+  // @llvm.masked.gather.*(Ptrs, alignment, Mask, Src0)
+  // Attempt to turn the masked gather in I into a mve intrinsic
+  // Potentially optimising the addressing modes as we do so.
+  Type *Ty = I->getType();
+  Value *Ptr = I->getArgOperand(0);
+  unsigned Alignment = cast<ConstantInt>(I->getArgOperand(1))->getZExtValue();
+  Value *Mask = I->getArgOperand(2);
+  Value *PassThru = I->getArgOperand(3);
+
+  // Check this is a valid gather with correct alignment
+  if (!isLegalAlignment(Ty->getVectorNumElements(),
+                        Ty->getScalarSizeInBits(), Alignment)) {
+    LLVM_DEBUG(dbgs() << "masked gathers: instruction does not have valid "
+                      << "alignment or vector type \n");
+    return false;
+  }
+
+  IRBuilder<> Builder(I->getContext());
+  Builder.SetInsertPoint(I);
+  Builder.SetCurrentDebugLocation(I->getDebugLoc());
+
+  Value *Load = nullptr;
+  // Look through bitcast instruction if #elements is same
+  if (auto *BitCast = dyn_cast<BitCastInst>(Ptr)) {
+    Type *BCTy = BitCast->getType();
+    Type *BCSrcTy = BitCast->getOperand(0)->getType();
+    if (BCTy->getVectorNumElements() == BCSrcTy->getVectorNumElements()) {
+      LLVM_DEBUG(dbgs() << "masked gathers: looking through bitcast\n");
+      Ptr = BitCast->getOperand(0);
+    }
+  }
+  assert(Ptr->getType()->isVectorTy() && "Unexpected pointer type");
+
+  if (Ty->getVectorNumElements() != 4)
+    // Can't build an intrinsic for this
+    return false;
+  if (match(Mask, m_One()))
+    Load = Builder.CreateIntrinsic(Intrinsic::arm_mve_vldr_gather_base,
+                                   {Ty, Ptr->getType()},
+                                   {Ptr, Builder.getInt32(0)});
+  else
+    Load = Builder.CreateIntrinsic(
+        Intrinsic::arm_mve_vldr_gather_base_predicated,
+        {Ty, Ptr->getType(), Mask->getType()},
+        {Ptr, Builder.getInt32(0), Mask});
+
+  if (Load && !isa<UndefValue>(PassThru) && !match(PassThru, m_Zero())) {
+    LLVM_DEBUG(dbgs() << "masked gathers: found non-trivial passthru - "
+                      << "creating select\n");
+    Load = Builder.CreateSelect(Mask, Load, PassThru);
+  }
+
+  if (Load) {
+    LLVM_DEBUG(dbgs() << "masked gathers: successfully built masked gather\n");
+    I->replaceAllUsesWith(Load);
+    I->eraseFromParent();
+    return true;
+  }
+  LLVM_DEBUG(dbgs() << "masked gathers: could not build masked gather\n");
+  return false;
+}
+
+bool MVEGatherScatterLowering::runOnFunction(Function &F) {
+  if (!EnableMaskedGatherScatters)
+    return false;
+  auto &TPC = getAnalysis<TargetPassConfig>();
+  auto &TM = TPC.getTM<TargetMachine>();
+  auto *ST = &TM.getSubtarget<ARMSubtarget>(F);
+  if (!ST->hasMVEIntegerOps())
+    return false;
+  SmallVector<IntrinsicInst *, 4> Gathers;
+  for (BasicBlock &BB : F) {
+    for (Instruction &I : BB) {
+      IntrinsicInst *II = dyn_cast<IntrinsicInst>(&I);
+      if (II && II->getIntrinsicID() == Intrinsic::masked_gather)
+        Gathers.push_back(II);
+    }
+  }
+
+  if (Gathers.empty())
+    return false;
+
+  for (IntrinsicInst *I : Gathers)
+    LowerGather(I);
+
+  return true;
+}

llvm/test/CodeGen/ARM/O3-pipeline.ll

 ; RUN: llc -mtriple=arm -O3 -debug-pass=Structure < %s -o /dev/null 2>&1 | grep -v "Verify generated machine code" | FileCheck %s
 
 ; REQUIRES: asserts
 
 ; CHECK:       ModulePass Manager
 ; CHECK-NEXT:    Pre-ISel Intrinsic Lowering
 ; CHECK-NEXT:    FunctionPass Manager
 ; CHECK-NEXT:      Expand Atomic instructions
 ; CHECK-NEXT:      Simplify the CFG
+; CHECK-NEXT:      MVE gather/scatter lowering
 ; CHECK-NEXT:      Dominator Tree Construction
 ; CHECK-NEXT:      Basic Alias Analysis (stateless AA impl)
 ; CHECK-NEXT:      Module Verifier
 ; CHECK-NEXT:      Natural Loop Information
 ; CHECK-NEXT:      Canonicalize natural loops
 ; CHECK-NEXT:      Scalar Evolution Analysis
 ; CHECK-NEXT:      Loop Pass Manager
 ; CHECK-NEXT:        Induction Variable Users

llvm/test/CodeGen/Thumb2/mve-gather-ind32-scaled.ll

entry:
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @scaled_i32_i32(i32* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: scaled_i32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 ; TODO: scaled_f16_i32
 
 define arm_aapcs_vfpcc <4 x float> @scaled_f32_i32(i32* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: scaled_f32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_scaled_b_i32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_b_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_scaled_i32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: signed_scaled_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_scaled_f32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: a_unsigned_scaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_scaled_f32_i16(i32* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: b_signed_scaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
entry:
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_scaled_b_i32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_scaled_b_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_scaled_i32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: signed_scaled_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_scaled_f32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: a_unsigned_scaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_scaled_f32_i8(i32* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: b_signed_scaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
 ; CHECK-NEXT:    vshl.i32 q0, q0, #2
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %i32_ptrs = getelementptr inbounds i32, i32* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i32*> %i32_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather

llvm/test/CodeGen/Thumb2/mve-gather-ind32-unscaled.ll

entry:
   %gather.sext = sext <4 x i16> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unscaled_i32_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: unscaled_i32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @unscaled_f32_i32(i8* %base, <4 x i32>* %offptr) {
 ; CHECK-LABEL: unscaled_f32_i32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32>, <4 x i32>* %offptr, align 4
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_unscaled_b_i32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_b_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_unscaled_i32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: signed_unscaled_i32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_unscaled_f32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: a_unsigned_unscaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.u32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.zext = zext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_unscaled_f32_i16(i8* %base, <4 x i16>* %offptr) {
 ; CHECK-LABEL: b_signed_unscaled_f32_i16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrh.s32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i16>, <4 x i16>* %offptr, align 2
   %offs.sext = sext <4 x i16> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
entry:
   %gather.sext = sext <4 x i8> %gather to <4 x i32>
   ret <4 x i32> %gather.sext
 }
 
 define arm_aapcs_vfpcc <4 x i32> @unsigned_unscaled_b_i32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: unsigned_unscaled_b_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @signed_unscaled_i32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: signed_unscaled_i32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vadd.i32 q1, q0, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x i32*>
   %gather = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @a_unsigned_unscaled_f32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: a_unsigned_unscaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.u32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.zext = zext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.zext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @b_signed_unscaled_f32_i8(i8* %base, <4 x i8>* %offptr) {
 ; CHECK-LABEL: b_signed_unscaled_f32_i8:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrb.s32 q0, [r1]
 ; CHECK-NEXT:    vadd.i32 q1, q0, r0
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i8>, <4 x i8>* %offptr, align 1
   %offs.sext = sext <4 x i8> %offs to <4 x i32>
   %byte_ptrs = getelementptr inbounds i8, i8* %base, <4 x i32> %offs.sext
   %ptrs = bitcast <4 x i8*> %byte_ptrs to <4 x float*>
   %gather = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather

llvm/test/CodeGen/Thumb2/mve-gather-ptrs.ll

entry:
   %offs = load <2 x i32*>, <2 x i32*>* %offptr, align 4
   %gather = call <2 x i32> @llvm.masked.gather.v2i32.v2p0i32(<2 x i32*> %offs, i32 4, <2 x i1> <i1 true, i1 true>, <2 x i32> undef)
   ret <2 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <4 x i32> @ptr_v4i32(<4 x i32*>* %offptr) {
 ; CHECK-LABEL: ptr_v4i32:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vldrw.u32 q0, [r0]
-; CHECK-NEXT:    vmov r0, s0
-; CHECK-NEXT:    vmov r3, s1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vmov r2, s3
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    ldr r3, [r3]
-; CHECK-NEXT:    vmov.32 q0[0], r0
-; CHECK-NEXT:    ldr r1, [r1]
-; CHECK-NEXT:    vmov.32 q0[1], r3
-; CHECK-NEXT:    ldr r2, [r2]
-; CHECK-NEXT:    vmov.32 q0[2], r1
-; CHECK-NEXT:    vmov.32 q0[3], r2
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x i32*>, <4 x i32*>* %offptr, align 4
   %gather = call <4 x i32> @llvm.masked.gather.v4i32.v4p0i32(<4 x i32*> %offs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x i32> undef)
   ret <4 x i32> %gather
 }
 
 define arm_aapcs_vfpcc <8 x i32> @ptr_v8i32(<8 x i32*>* %offptr) {
entry:
   %gather = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> %offs, i32 4, <2 x i1> <i1 true, i1 true>, <2 x float> undef)
   ret <2 x float> %gather
 }
 
 define arm_aapcs_vfpcc <4 x float> @ptr_v4f32(<4 x float*>* %offptr) {
 ; CHECK-LABEL: ptr_v4f32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vmov r0, s7
-; CHECK-NEXT:    vldr s3, [r0]
-; CHECK-NEXT:    vmov r0, s6
-; CHECK-NEXT:    vldr s2, [r0]
-; CHECK-NEXT:    vmov r0, s5
-; CHECK-NEXT:    vldr s1, [r0]
-; CHECK-NEXT:    vmov r0, s4
-; CHECK-NEXT:    vldr s0, [r0]
+; CHECK-NEXT:    vldrw.u32 q0, [q1]
 ; CHECK-NEXT:    bx lr
 entry:
   %offs = load <4 x float*>, <4 x float*>* %offptr, align 4
   %gather = call <4 x float> @llvm.masked.gather.v4f32.v4p0f32(<4 x float*> %offs, i32 4, <4 x i1> <i1 true, i1 true, i1 true, i1 true>, <4 x float> undef)
   ret <4 x float> %gather
 }
 
 define arm_aapcs_vfpcc <8 x float> @ptr_v8f32(<8 x float*>* %offptr) {

 
 ; loops
 
 define void @foo_ptr_p_int32_t(i32* %dest, i32** %src, i32 %n) {
 ; CHECK-LABEL: foo_ptr_p_int32_t:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    bic r2, r2, #15
 ; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB22_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.body.preheader
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    poplt {r7, pc}
 ; CHECK-NEXT:    subs r2, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    sub.w r12, r1, #16
+; CHECK-NEXT:    subs r1, #16
 ; CHECK-NEXT:    add.w lr, r3, r2, lsr #2
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB22_2: @ %vector.body
+; CHECK-NEXT:  .LBB22_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q0, [r12, #16]
-; CHECK-NEXT:    @ implicit-def: $q1
-; CHECK-NEXT:    add.w r12, r12, #16
-; CHECK-NEXT:    vcmp.i32 ne, q0, zr
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r1, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r1, #0, #1
-; CHECK-NEXT:    ubfx r1, r3, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, r3, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, r3, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #3, #1
-; CHECK-NEXT:    lsls r1, r2, #31
-; CHECK-NEXT:    ittt ne
-; CHECK-NEXT:    vmovne r1, s0
-; CHECK-NEXT:    ldrne r1, [r1]
-; CHECK-NEXT:    vmovne.32 q1[0], r1
-; CHECK-NEXT:    lsls r1, r2, #30
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s1
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[1], r1
-; CHECK-NEXT:    lsls r1, r2, #29
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s2
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[2], r1
-; CHECK-NEXT:    lsls r1, r2, #28
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[3], r1
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrw.u32 q0, [r1, #16]!
+; CHECK-NEXT:    vptt.i32 ne, q0, zr
+; CHECK-NEXT:    vldrwt.u32 q1, [q0]
 ; CHECK-NEXT:    vstrwt.32 q1, [r0], #16
-; CHECK-NEXT:    le lr, .LBB22_2
-; CHECK-NEXT:  .LBB22_3: @ %for.end
-; CHECK-NEXT:    add sp, #4
+; CHECK-NEXT:    le lr, .LBB22_1
+; CHECK-NEXT:  @ %bb.2: @ %for.end
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %and = and i32 %n, -16
   %cmp11 = icmp sgt i32 %and, 0
   br i1 %cmp11, label %vector.body, label %for.end
 
 vector.body:                                      ; preds = %entry, %vector.body
   %index = phi i32 [ %index.next, %vector.body ], [ 0, %entry ]
for.end:                                          ; preds = %vector.body, %entry
   ret void
 }
 
 define void @foo_ptr_p_float(float* %dest, float** %src, i32 %n) {
 ; CHECK-LABEL: foo_ptr_p_float:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    bic r2, r2, #15
 ; CHECK-NEXT:    cmp r2, #1
-; CHECK-NEXT:    blt .LBB23_3
-; CHECK-NEXT:  @ %bb.1: @ %vector.body.preheader
+; CHECK-NEXT:    it lt
+; CHECK-NEXT:    poplt {r7, pc}
 ; CHECK-NEXT:    subs r2, #4
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    sub.w r12, r1, #16
+; CHECK-NEXT:    subs r1, #16
 ; CHECK-NEXT:    add.w lr, r3, r2, lsr #2
 ; CHECK-NEXT:    dls lr, lr
-; CHECK-NEXT:  .LBB23_2: @ %vector.body
+; CHECK-NEXT:  .LBB23_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q0, [r12, #16]
-; CHECK-NEXT:    @ implicit-def: $q1
-; CHECK-NEXT:    add.w r12, r12, #16
-; CHECK-NEXT:    vcmp.i32 ne, q0, zr
-; CHECK-NEXT:    vmrs r3, p0
-; CHECK-NEXT:    and r2, r3, #1
-; CHECK-NEXT:    rsbs r1, r2, #0
-; CHECK-NEXT:    movs r2, #0
-; CHECK-NEXT:    bfi r2, r1, #0, #1
-; CHECK-NEXT:    ubfx r1, r3, #4, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #1, #1
-; CHECK-NEXT:    ubfx r1, r3, #8, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #2, #1
-; CHECK-NEXT:    ubfx r1, r3, #12, #1
-; CHECK-NEXT:    rsbs r1, r1, #0
-; CHECK-NEXT:    bfi r2, r1, #3, #1
-; CHECK-NEXT:    lsls r1, r2, #31
-; CHECK-NEXT:    ittt ne
-; CHECK-NEXT:    vmovne r1, s0
-; CHECK-NEXT:    ldrne r1, [r1]
-; CHECK-NEXT:    vmovne.32 q1[0], r1
-; CHECK-NEXT:    lsls r1, r2, #30
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s1
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[1], r1
-; CHECK-NEXT:    lsls r1, r2, #29
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s2
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[2], r1
-; CHECK-NEXT:    lsls r1, r2, #28
-; CHECK-NEXT:    ittt mi
-; CHECK-NEXT:    vmovmi r1, s3
-; CHECK-NEXT:    ldrmi r1, [r1]
-; CHECK-NEXT:    vmovmi.32 q1[3], r1
-; CHECK-NEXT:    vpst
+; CHECK-NEXT:    vldrw.u32 q0, [r1, #16]!
+; CHECK-NEXT:    vptt.i32 ne, q0, zr
+; CHECK-NEXT:    vldrwt.u32 q1, [q0]
 ; CHECK-NEXT:    vstrwt.32 q1, [r0], #16
-; CHECK-NEXT:    le lr, .LBB23_2
-; CHECK-NEXT:  .LBB23_3: @ %for.end
-; CHECK-NEXT:    add sp, #4
+; CHECK-NEXT:    le lr, .LBB23_1
+; CHECK-NEXT:  @ %bb.2: @ %for.end
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %and = and i32 %n, -16
   %cmp11 = icmp sgt i32 %and, 0
   br i1 %cmp11, label %vector.body, label %for.end
 
 vector.body:                                      ; preds = %entry, %vector.body
   %index = phi i32 [ %index.next, %vector.body ], [ 0, %entry ]