[ARM][MVE] Add support for incrementing gathers

[anwel]

Summary

Enables the MVEGatherScatterLowering pass to build pre-incrementing gathers.
If the increment (i.e., add instruction) that is merged into the gather is the loop increment, an incrementing write-back gather can be built, which then assumes the role of the loop increment.

[samparker]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

787

You could use getIncomingValueForBlock instead.

804

Would be nicer to cast to a GEP just once.

857

Looking at the manual, it looks like it's a signed immediate and are you checking that it's a multiple of 4?

864

Could these checks be together with the other phi stuff around line 788?

[dmgreen]

Tests look nice on this one.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

306

Will this ever be false? If so, the place this is called might be casting a nullptr.

770

I think LI should already be available. It's common to store analyses like that as a member of the pass.

785

This can be merged into the previous if?

797

I don't think Add will be nullptr here. (cast cannot fail by returning nullptr. dyn_cast would, but Offset is already known to be an Instruction)

800

aligned -> scaled

911–916

Can you move the if into the argument?

Also is it worth having tryCreateMaskedGatherBaseWB just return an IntrinsicInst?

[anwel]

Changes addressing most of the comments - see below.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

306

With the current usage, this should never be false - tryCreateMergedGatherIncrement currently is the only user and will have checked this condition long before the gather is built. I'd still like to leave this check in here for the future, as we might add another use of this function where it might not be clear.

770

Alright, I made it a class member, so it'll only be queried once.

785

Yes, seems like it can.

797

You're right. I did change it to dyn_cast and merge the ifs though.

804

That's true. Done.

857

You are right, we should be checking the lower bound, too - thanks for catching that!
The value that is in the end handed over to the gather as increment will be a multiple of four, as we are shifting it left to replicate the change it would have experienced from the GEP we replaced.

864

These checks are to determine whether a non-incrementing gather should be built and immediately call that function - which requires the variable Const (calculated above) to be handed over. So I couldn't move this up to line 788, at least not as a block.

911–916

Yes, can do. I'd rather have all the functions creating gathers or scatters having the same return type. It is however no longer necessary to have an IntrinsicInst here, so I just removed the cast.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

796

How come this is testing multiple users? Don't we have to test the one that related to our gather? Otherwise if there were multiple and they had different scales, might we pick the wrong one?

831

uint64_t is better for constants values. Also if you are using them signed, then they should maybe be getSExtValue, and use int64_t instead?

850

I think it might be something like this maybe?

int64_t Immediate = Const << TypeScale;
if (Immediate > 512 || Immediate < -512 || Immediate  % 4 != 0)

As in it is +/- 7bit imm * 4 in the instruction.

875

aligned -> scaled

985

Is Ignore ever used in this code? If not, would it be simpler to just remove it?

992–993

Do these extends/truncates ever come up in the tests here?

[anwel]

Made some changes suggested in the comments, and also re-ordered the code such that it is easier to follow.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

796

You are right. We need information about the specific GEP used by the gather here. I have made some changes to make sure that information is present.

831

That is a good point, the constants we're looking at here could be negative so signed extension will be the right way to go. I'll make it an int64_t then.

985

No, the parameter is indeed not needed anymore.

985

Ignore has indeed become superfluent.

992–993

No, they don't. If they ever come up in the context in which this function is used, we will later decide to abort anyway.
I have decided to re-work this function from generally checking whether a value is a loop invariant or constant to checking whether it is a constant or arithmetic expression a constant can be derived from, and made it so that it returns that constant. This reduces code duplication at the location the function is called, and also makes it more clear what is happening.

992–993

No, they don't. If they ever come up in the context in which this function is used, we will later decide to abort anyway.
I have decided to re-work this function from generally checking whether a value is a loop invariant or constant to checking whether it is a constant or arithmetic expression a constant can be derived from, and made it so that it returns that constant. This reduces code duplication at the location the function is called, and also makes it more clear what is happening.

[anwel]

Fixed calculation of the constant in getIfConst, moved isGatherScatter to ARMBaseInstrInfo so it is accessible for other passes, too.

[dmgreen]

llvm/lib/Target/ARM/ARMBaseInstrInfo.h

359 ↗

(On Diff #258728)

This needn't be in the ARMBaseInstrInfo class specifically. There are some free functions at the end of this file. I think it can go with those.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

231–1

Give this a clang-format?

232

I think you may be able to drop all the llvm::'s from the Optional's

232

I think this might as well always set LI.

234

Can GEP ever be null here?

236

You don't need else's after returns.

254

This should probably be using LI->getLoopLatch(), in case there are multiple blocks in the loop.

273

Can be removed.

294–296

Can this be OffsetsIncoming != Phi? Or am I missing what this is checking?

309

= 1 - IncrementIndex;

319

This is overwritten below I think

350

It looks like Phi is always valid here.

[anwel]

llvm/lib/Target/ARM/ARMBaseInstrInfo.h

359 ↗

(On Diff #258728)

Alright, I moved it there

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

234

Now that you mention it, no it can't.

254

Okay.

294–296

Not exactly, because Phi can be nullptr in some cases where OffsetsIncoming still is a phi node. Using the latter instead of checking all operands of Add again does make this code more readable, though

319

Yes, you are right.

350

It is in fact.

[anwel]

Fixing a bug in the latest diff: We should always make sure to scale the offsets that the loop is initialised with.

[anwel]

Restructured the code such that the differentiation between writeback and non-writeback gathers is clearer

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

232

You don't need experimental/optional (we technically need llvm/ADT/Optional.h I think, as we are using the one in llvm. It is likely included transitively at the moment).

287

Maybe add a check that Phi->getBB == L->getHeader too.

292

Creating a variable for L sounds useful.

301

I think we have to check that OffsetsIncoming == Phi. To make sure it's the loop structure / IV we want it to be.

328

As we are modifying the Phi inplace, and it can produce different values, I think we unfortunately need to check it has no other uses.

[anwel]

Made optimisation of offsets independent of the creation of the individual gathers and scatters, such that when looking at transforming those we're not in an environment where some instructions have been optimised but others, which will be optimised too, haven't been yet.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

287

Has been added.

301

Yes, you are right.

328

You convinced me, though it is sad we have to restrict the usage of the writeback gathers that much. I'll leave it to future work to make it right and find a way to allow them being used more often again.

[dmgreen]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

271

I feel like this should be using BasePtr somewhere?

278

This looks OK. Can you add one extra check somewhere that gep has a single user? As the phi is changing and the gep uses the phi.

[anwel]

Applied the chances suggested in the comments.

[anwel]

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

271

You are right, the base pointer should be added to the offsets before handing them over to the intrinsic construction. I fixed that now.

278

You are right, unfortunately we cannot allow this as is with gep having more than one user. This means for now that in cases where multiple gathers and/or scatters use the same gep we cannot make any of them write the increment back, which we might want to fix in a future patch.

[dmgreen]

Thanks. I think in hindsight this might have been a couple of patches, we we managed to power through.

LGTM with one minor alteration

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

290

I think that in this case we needn't check for one use of Offsets. Because we are only folding it into the instruction, not altering it in any way, it should always be OK I think. And smaller. But, er, check that makes sense.

[anwel]

Last correction: For the incrementing non-writeback gathers, removed the check that the offset only has one use.

llvm/lib/Target/ARM/MVEGatherScatterLowering.cpp

private:
   int computeScale(unsigned GEPElemSize, unsigned MemoryElemSize);
 
   Value *lowerGather(IntrinsicInst *I);
   // Create a gather from a base + vector of offsets
   Value *tryCreateMaskedGatherOffset(IntrinsicInst *I, Value *Ptr,
                                      Instruction *&Root, IRBuilder<> &Builder);
   // Create a gather from a vector of pointers
   Value *tryCreateMaskedGatherBase(IntrinsicInst *I, Value *Ptr,
-                                   IRBuilder<> &Builder);
+                                   IRBuilder<> &Builder,
+                                   unsigned Increment = 0);
+  // Create a gather from a vector of pointers
+  Value *tryCreateMaskedGatherBaseWB(IntrinsicInst *I, Value *Ptr,
+                                     IRBuilder<> &Builder,
+                                     unsigned Increment = 0);
 
   Value *lowerScatter(IntrinsicInst *I);
   // Create a scatter to a base + vector of offsets
   Value *tryCreateMaskedScatterOffset(IntrinsicInst *I, Value *Offsets,
                                       IRBuilder<> &Builder);
   // Create a scatter to a vector of pointers
   Value *tryCreateMaskedScatterBase(IntrinsicInst *I, Value *Ptr,
                                     IRBuilder<> &Builder);
+  // QI gathers can increment their offsets on their own if the increment is
+  // a constant value (digit)
+  Value *tryCreateMergedGatherIncrement(IntrinsicInst *I, Value *BasePtr,
+                                        Value *Ptr, IRBuilder<> &Builder);
 
   // Check whether these offsets could be moved out of the loop they're in
   bool optimiseOffsets(Value *Offsets, BasicBlock *BB, LoopInfo *LI);
   // Pushes the given add out of the loop
   void pushOutAdd(PHINode *&Phi, Value *SecondOperand, unsigned StartIndex);
   // Pushes the given mul out of the loop
   void pushOutMul(PHINode *&Phi, Value *IncrementPerRound, Value *SecondOperand,
                   unsigned LoopIncrement, IRBuilder<> &Builder);
 
+  // Returns true if V is a constant or a loop invariant instruction
+  bool isConstOrLoopInvInst(const Value *V, const Loop *L, const Value *Ignore);
 };
 
 } // end anonymous namespace
 
 char MVEGatherScatterLowering::ID = 0;
 
 INITIALIZE_PASS(MVEGatherScatterLowering, DEBUG_TYPE,
                 "MVE gather/scattering lowering pass", false, false)
else if (GEPElemSize == 8)
     return 0;
   LLVM_DEBUG(dbgs() << "masked gathers/scatters: incorrect scale. Can't "
                     << "create intrinsic\n");
   return -1;
 }
 
 Value *MVEGatherScatterLowering::lowerGather(IntrinsicInst *I) {
   using namespace PatternMatch;
   LLVM_DEBUG(dbgs() << "masked gathers: checking transform preconditions\n");
 
   // @llvm.masked.gather.*(Ptrs, alignment, Mask, Src0)
   // Attempt to turn the masked gather in I into a MVE intrinsic
   // Potentially optimising the addressing modes as we do so.
   Type *Ty = I->getType();
   Value *Ptr = I->getArgOperand(0);
   unsigned Alignment = cast<ConstantInt>(I->getArgOperand(1))->getZExtValue();
   Value *Mask = I->getArgOperand(2);
   Value *PassThru = I->getArgOperand(3);
 
   if (!isLegalTypeAndAlignment(Ty->getVectorNumElements(),
                                Ty->getScalarSizeInBits(), Alignment))
     return nullptr;
   lookThroughBitcast(Ptr);
   assert(Ptr->getType()->isVectorTy() && "Unexpected pointer type");
 
   IRBuilder<> Builder(I->getContext());
   Builder.SetInsertPoint(I);
   Builder.SetCurrentDebugLocation(I->getDebugLoc());
 
   Instruction *Root = I;
   Value *Load = tryCreateMaskedGatherOffset(I, Ptr, Root, Builder);
   if (!Load)
     Load = tryCreateMaskedGatherBase(I, Ptr, Builder);
   if (!Load)
     return nullptr;
 
   if (!isa<UndefValue>(PassThru) && !match(PassThru, m_Zero())) {
     LLVM_DEBUG(dbgs() << "masked gathers: found non-trivial passthru - "
                       << "creating select\n");
     Load = Builder.CreateSelect(Mask, Load, PassThru);
   }
 
   Root->replaceAllUsesWith(Load);
   Root->eraseFromParent();
   if (Root != I)
     // If this was an extending gather, we need to get rid of the sext/zext
     // sext/zext as well as of the gather itself
     I->eraseFromParent();
 
   LLVM_DEBUG(dbgs() << "masked gathers: successfully built masked gather\n");
   return Load;
 }
 
 Value *MVEGatherScatterLowering::tryCreateMaskedGatherBase(IntrinsicInst *I,
                                                            Value *Ptr,
-                                                           IRBuilder<> &Builder) {
+                                                           IRBuilder<> &Builder,
+                                                           unsigned Increment) {
   using namespace PatternMatch;
   Type *Ty = I->getType();
   LLVM_DEBUG(dbgs() << "masked gathers: loading from vector of pointers\n");
   if (Ty->getVectorNumElements() != 4 || Ty->getScalarSizeInBits() != 32)
     // Can't build an intrinsic for this
     return nullptr;
   Value *Mask = I->getArgOperand(2);
   if (match(Mask, m_One()))
     return Builder.CreateIntrinsic(Intrinsic::arm_mve_vldr_gather_base,
                                    {Ty, Ptr->getType()},
-                                   {Ptr, Builder.getInt32(0)});
+                                   {Ptr, Builder.getInt32(Increment)});
   else
     return Builder.CreateIntrinsic(
         Intrinsic::arm_mve_vldr_gather_base_predicated,
         {Ty, Ptr->getType(), Mask->getType()},
-        {Ptr, Builder.getInt32(0), Mask});
+        {Ptr, Builder.getInt32(Increment), Mask});
+}
+
+Value *MVEGatherScatterLowering::tryCreateMaskedGatherBaseWB(
+    IntrinsicInst *I, Value *Ptr, IRBuilder<> &Builder, unsigned Increment) {
+  using namespace PatternMatch;
+  Type *Ty = I->getType();
+  LLVM_DEBUG(
+      dbgs()
+      << "masked gathers: loading from vector of pointers with writeback\n");
+  if (Ty->getVectorNumElements() != 4 || Ty->getScalarSizeInBits() != 32)
+    // Can't build an intrinsic for this
+    return nullptr;
+  Value *Mask = I->getArgOperand(2);
+  if (match(Mask, m_One()))
+    return Builder.CreateIntrinsic(Intrinsic::arm_mve_vldr_gather_base_wb,
+                                   {Ty, Ptr->getType()},
+                                   {Ptr, Builder.getInt32(Increment)});
+  else
+    return Builder.CreateIntrinsic(
+        Intrinsic::arm_mve_vldr_gather_base_wb_predicated,
+        {Ty, Ptr->getType(), Mask->getType()},
+        {Ptr, Builder.getInt32(Increment), Mask});
 }
 
 Value *MVEGatherScatterLowering::tryCreateMaskedGatherOffset(
     IntrinsicInst *I, Value *Ptr, Instruction *&Root, IRBuilder<> &Builder) {
   using namespace PatternMatch;
 
   Type *OriginalTy = I->getType();
   Type *ResultTy = OriginalTy;
 
   unsigned Unsigned = 1;
   // The size of the gather was already checked in isLegalTypeAndAlignment;
   // if it was not a full vector width an appropriate extend should follow.
   auto *Extend = Root;
   if (OriginalTy->getPrimitiveSizeInBits() < 128) {
     // Only transform gathers with exactly one use
     if (!I->hasOneUse())
       return nullptr;
 
     // The correct root to replace is not the CallInst itself, but the
     // instruction which extends it
     Extend = cast<Instruction>(*I->users().begin());
     if (isa<SExtInst>(Extend)) {
       Unsigned = 0;
     } else if (!isa<ZExtInst>(Extend)) {
       LLVM_DEBUG(dbgs() << "masked gathers: extend needed but not provided. "
                         << "Expanding\n");
       return nullptr;
     }
     LLVM_DEBUG(dbgs() << "masked gathers: found an extending gather\n");
     ResultTy = Extend->getType();
     // The final size of the gather must be a full vector width
     if (ResultTy->getPrimitiveSizeInBits() != 128) {
       LLVM_DEBUG(dbgs() << "masked gathers: extending from the wrong type. "
                         << "Expanding\n");
       return nullptr;
     }
   }
 
   Value *Offsets;
   Value *BasePtr = checkGEP(Offsets, ResultTy, Ptr, Builder);
   if (!BasePtr)
     return nullptr;
+  // Check whether the offset is a constant increment that could be merged into
+  // a QI gather
+  Value *Load = tryCreateMergedGatherIncrement(I, BasePtr, Offsets, Builder);
+  if (Load)
+    return Load;
 
   int Scale = computeScale(
       BasePtr->getType()->getPointerElementType()->getPrimitiveSizeInBits(),
       OriginalTy->getScalarSizeInBits());
   if (Scale == -1)
     return nullptr;
   Root = Extend;
 
bool MVEGatherScatterLowering::optimiseOffsets(Value *Offsets, BasicBlock *BB,
   // Clean up the old increment in case it's unused because we built a new
   // one
   if (IncInstruction->hasNUses(0))
     IncInstruction->eraseFromParent();
 
   return true;
 }
 
+Value *MVEGatherScatterLowering::tryCreateMergedGatherIncrement(
+    IntrinsicInst *I, Value *BasePtr, Value *Offsets, IRBuilder<> &Builder) {
+  // Check whether this gather's offset is incremented by a constant - if so,
+  // and the load is of the right type, we can merge this into a QI gather
+  Type *Ty = I->getType();
+  // Incrementing gathers only exist for v4i32
+  if (Ty->getVectorNumElements() != 4 || Ty->getScalarSizeInBits() != 32)
+    return nullptr;
+  LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
+  if (LI.getLoopFor(I->getParent()) == nullptr)
+    // Incrementing gathers are not beneficial outside of a loop
+    return nullptr;
+  LLVM_DEBUG(dbgs() << "masked gathers: try to merge increment and gather\n");
+  DataLayout DT = I->getParent()->getParent()->getParent()->getDataLayout();
+
+  // Offsets that are worth merging into this instruction will be incremented
+  // by a constant, thus we're looking for an add of something (most probably a
+  // phi) and a constant
+  PHINode *Phi = nullptr;
+  if (isa<PHINode>(Offsets))
+    Phi = cast<PHINode>(Offsets);
+  unsigned IncrementIndex;
+
+  if (Phi != nullptr) {
+    // Look through the phi to the phi increment
+    IncrementIndex = Phi->getIncomingBlock(0) == Phi->getParent() ? 0 : 1;
+    Offsets = Phi->getIncomingValue(IncrementIndex);
+  }
+  if (!isa<Instruction>(Offsets))
+    // Offsets as vector of constants - there is nothing to write back to
+    return nullptr;
+
+  // At this point, the instruction we're looking at must be an add or we
+  // bail out
+  Instruction *Add = cast<Instruction>(Offsets);
+  if ((Add == nullptr) || !(Add->getOpcode() == Instruction::Add))
+    return nullptr;
+
+  // The gep was in charge of making sure the offsets are aligned correctly
+  // - calculate that factor so it can be applied by hand
+  int TypeScale = 0;
+  for (User *U : (Phi == nullptr ? Add->users() : Phi->users())) {
+    if (isa<GetElementPtrInst>(U)) {
+      TypeScale = computeScale(
+          DT.getTypeSizeInBits(
+              cast<GetElementPtrInst>(U)->getOperand(0)->getType()),
+          DT.getTypeSizeInBits(cast<GetElementPtrInst>(U)->getType()) /
+              cast<GetElementPtrInst>(U)->getType()->getVectorNumElements());
+    }
+  }
+
+  if (TypeScale == -1)
+    return nullptr;
+
+  Value *C = nullptr;
+  Value *OffsetsIncoming;
+  // Find out which operand the value that is increased is
+  if (isConstOrLoopInvInst(Add->getOperand(0), LI.getLoopFor(I->getParent()),
+                      nullptr)) {
+    C = Add->getOperand(0);
+    OffsetsIncoming = Add->getOperand(1);
+  } else if (isConstOrLoopInvInst(Add->getOperand(1), LI.getLoopFor(I->getParent()),
+                             nullptr)) {
+    OffsetsIncoming = Add->getOperand(0);
+    C = Add->getOperand(1);
+  }
+
+  int Const;
+  // The increment can only be merged with the gather if C is a constant
+  // or a combination of two constants
+  if (!isa<Constant>(C)) {
+    if (!isa<Instruction>(C))
+      return nullptr;
+    Instruction *Inst = cast<Instruction>(C);
+    if (isa<Constant>(Inst->getOperand(0)) &&
+        isa<Constant>(Inst->getOperand(1))) {
+      int Left = cast<Constant>(Inst->getOperand(0))
+                     ->getUniqueInteger()
+                     .getZExtValue();
+      int Right = cast<Constant>(Inst->getOperand(1))
+                      ->getUniqueInteger()
+                      .getZExtValue();
+      // Only take the number, we don't need to construct an instruction
+      if (Inst->getOpcode() == Instruction::Add)
+        Const = Left + Right;
+      else if (Inst->getOpcode() == Instruction::Mul)
+        Const = Left * Right;
+      else
+        return nullptr;
+    } else
+      return nullptr;
+  } else {
+    Const = cast<Constant>(C)->getUniqueInteger().getZExtValue();
+  }
+  // Check that the constant is small enough for an incrementing gather
+  if (Const > 126)
+    return nullptr;
+
+  // If right or left operand of offsets is a phi node, then if Offsets
+  // itself is not used by that phi node (i.e., is no immediate increment)
+  // the gather doesn't need to be write-back
+  PHINode *P;
+  if ((P = dyn_cast<PHINode>(Add->getOperand(0))) != nullptr ||
+      ((P = dyn_cast<PHINode>(Add->getOperand(1))) != nullptr)) {
+    if (!(P->getIncomingValue(0) == Add || P->getIncomingValue(1) == Add)) {
+      // No loop increment, so no writeback
+      return cast<IntrinsicInst>(
+          tryCreateMaskedGatherBase(I, P, Builder, Const << TypeScale));
+    }
+  }
+  if (Phi == nullptr) {
+    // If no phi node has been found, there's nothing to write back too -
+    // so either the former condition was true and we built a normal
+    // incrementing gather, or we bail out
+    return nullptr;
+  }
+
+  unsigned Incoming = IncrementIndex == 0 ? 1 : 0;
+  Builder.SetInsertPoint(&Phi->getIncomingBlock(Incoming)->back());
+
+  // Make sure the offsets are aligned correctly
+  Instruction *AlignOffsets = BinaryOperator::Create(
+      Instruction::Shl, Phi->getIncomingValue(Incoming),
+      Builder.CreateVectorSplat(
+          OffsetsIncoming->getType()->getVectorNumElements(),
+          Builder.getInt32(TypeScale)),
+      "AlignedIndex", &Phi->getIncomingBlock(Incoming)->back());
+  Phi->setIncomingValue(Incoming, AlignOffsets);
+  // Add the base to the offsets
+  OffsetsIncoming = BinaryOperator::Create(
+      Instruction::Add, Phi->getIncomingValue(Incoming),
+      Builder.CreateVectorSplat(
+          AlignOffsets->getType()->getVectorNumElements(),
+          Builder.CreatePtrToInt(
+              BasePtr, AlignOffsets->getType()->getVectorElementType())),
+      "StartIndex", &Phi->getIncomingBlock(Incoming)->back());
+  // The gather is pre-incrementing
+  OffsetsIncoming = BinaryOperator::Create(
+      Instruction::Sub, OffsetsIncoming,
+      Builder.CreateVectorSplat(
+          OffsetsIncoming->getType()->getVectorNumElements(),
+          Builder.getInt32(Const << TypeScale)),
+      "PreIncrementStartIndex", &Phi->getIncomingBlock(Incoming)->back());
+  Phi->setIncomingValue(Incoming, OffsetsIncoming);
+
+  Builder.SetInsertPoint(I);
+
+  // Build the incrementing gather
+  IntrinsicInst *Load;
+  if (Phi == nullptr)
+    Load = cast<IntrinsicInst>(tryCreateMaskedGatherBaseWB(
+        I, OffsetsIncoming, Builder, Const << TypeScale));
+  else
+    Load = cast<IntrinsicInst>(
+        tryCreateMaskedGatherBaseWB(I, Phi, Builder, Const << TypeScale));
+
+  // One value to be handed to whoever uses the gather, one is the loop
+  // increment
+  Value *ExtractedLoad = Builder.CreateExtractValue(Load, 0, "Gather");
+  Value *Inc = Builder.CreateExtractValue(Load, 1, "GatherIncrement");
+  Add->replaceAllUsesWith(Inc);
+  Add->eraseFromParent();
+  if (Phi != nullptr)
+    Phi->setIncomingValue(IncrementIndex, Inc);
+
+  return ExtractedLoad;
+}
+
 bool MVEGatherScatterLowering::runOnFunction(Function &F) {
   if (!EnableMaskedGatherScatters)
     return false;
   auto &TPC = getAnalysis<TargetPassConfig>();
   auto &TM = TPC.getTM<TargetMachine>();
   auto *ST = &TM.getSubtarget<ARMSubtarget>(F);
   if (!ST->hasMVEIntegerOps())
     return false;
   SmallVector<IntrinsicInst *, 4> Gathers;
   SmallVector<IntrinsicInst *, 4> Scatters;
+
   LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
 
   for (BasicBlock &BB : F) {
     for (Instruction &I : BB) {
       IntrinsicInst *II = dyn_cast<IntrinsicInst>(&I);
       if (II && II->getIntrinsicID() == Intrinsic::masked_gather)
         Gathers.push_back(II);
       else if (II && II->getIntrinsicID() == Intrinsic::masked_scatter)
         Scatters.push_back(II);
     }
   }
 
   bool Changed = false;
   for (unsigned i = 0; i < Gathers.size(); i++) {
     IntrinsicInst *I = Gathers[i];
     if (isa<GetElementPtrInst>(I->getArgOperand(0)))
       optimiseOffsets(cast<Instruction>(I->getArgOperand(0))->getOperand(1),
                       I->getParent(), &LI);
     Value *L = lowerGather(I);
     if (L == nullptr)
       continue;
+
     // Get rid of any now dead instructions
     SimplifyInstructionsInBlock(cast<Instruction>(L)->getParent());
     Changed = true;
   }
 
   for (unsigned i = 0; i < Scatters.size(); i++) {
     IntrinsicInst *I = Scatters[i];
     if (isa<GetElementPtrInst>(I->getArgOperand(1)))
       optimiseOffsets(cast<Instruction>(I->getArgOperand(1))->getOperand(1),
                       I->getParent(), &LI);
     Value *S = lowerScatter(I);
     if (S == nullptr)
       continue;
+
     // Get rid of any now dead instructions
     SimplifyInstructionsInBlock(cast<Instruction>(S)->getParent());
     Changed = true;
   }
   return Changed;
 }
+
+bool MVEGatherScatterLowering::isConstOrLoopInvInst(const Value *V, const Loop *L,
+                                               const Value *Ignore) {
+  if (isa<Constant>(V) || V == Ignore) {
+    return true;
+  } else if (!isa<Instruction>(V)) {
+    return true;
+  }
+  const Instruction *I = cast<Instruction>(V);
+  if (isa<TruncInst>(I) || isa<ZExtInst>(I) || isa<SExtInst>(I)) {
+    return isConstOrLoopInvInst(I->getOperand(0), L, Ignore);
+  } else if (I->getOpcode() == Instruction::Add ||
+              I->getOpcode() == Instruction::Mul) {
+    for (Value *O : I->operands()) {
+      if (!isConstOrLoopInvInst(O, L, Ignore))
+        return false;
+    }
+    return true;
+  }
+  return !L->contains(I);
+}

llvm/test/CodeGen/Thumb2/mve-gather-increment.ll

; CHECK-NEXT:    pop {r4, r5, r7, pc}
   %wide.masked.gather = call <16 x i8> @llvm.masked.gather.v16i8.v16p0i8(<16 x i8*> %2, i32 2, <16 x i1> <i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false, i1 true, i1 false>, <16 x i8> undef)
   ret <16 x i8> %wide.masked.gather
 }
 
 define arm_aapcs_vfpcc void @gather_pre_inc(i32* noalias nocapture readonly %data, i32* noalias nocapture %dst, i32 %n.vec) {
 ; CHECK-LABEL: gather_pre_inc:
 ; CHECK:       @ %bb.0: @ %vector.ph
 ; CHECK-NEXT:    adr r3, .LCPI6_0
-; CHECK-NEXT:    vmov.i32 q1, #0x18
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0, q0, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q1, [q0, #96]!
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vstrb.8 q2, [r1], #16
+; CHECK-NEXT:    vstrb.8 q1, [r1], #16
 ; CHECK-NEXT:    bne .LBB6_1
 ; CHECK-NEXT:  @ %bb.2: @ %end
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI6_0:
-; CHECK-NEXT:    .long 6 @ 0x6
-; CHECK-NEXT:    .long 12 @ 0xc
-; CHECK-NEXT:    .long 18 @ 0x12
-; CHECK-NEXT:    .long 24 @ 0x18
+; CHECK-NEXT:    .long 4294967224 @ 0xffffffb8
+; CHECK-NEXT:    .long 4294967248 @ 0xffffffd0
+; CHECK-NEXT:    .long 4294967272 @ 0xffffffe8
+; CHECK-NEXT:    .long 0 @ 0x0
 vector.ph:                                        ; preds = %for.body.preheader
   %ind.end = shl i32 %n.vec, 1
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %vector.ph
   %index = phi i32 [ 0, %vector.ph ], [ %index.next, %vector.body ]
   %vec.ind = phi <4 x i32> [ <i32 0, i32 2, i32 4, i32 6>, %vector.ph ], [ %vec.ind.next, %vector.body ]
   %0 = mul <4 x i32> %vec.ind, <i32 3, i32 3, i32 3, i32 3>

 end:
   ret void;
 }
 
 define arm_aapcs_vfpcc void @gather_post_inc(i32* noalias nocapture readonly %data, i32* noalias nocapture %dst, i32 %n.vec43) {
 ; CHECK-LABEL: gather_post_inc:
 ; CHECK:       @ %bb.0: @ %vector.ph41
 ; CHECK-NEXT:    adr r3, .LCPI7_0
-; CHECK-NEXT:    vmov.i32 q1, #0x18
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB7_1: @ %vector.body39
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0, q0, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q1, [q0, #96]!
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vstrb.8 q2, [r1], #16
+; CHECK-NEXT:    vstrb.8 q1, [r1], #16
 ; CHECK-NEXT:    bne .LBB7_1
 ; CHECK-NEXT:  @ %bb.2: @ %end
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI7_0:
-; CHECK-NEXT:    .long 0 @ 0x0
-; CHECK-NEXT:    .long 6 @ 0x6
-; CHECK-NEXT:    .long 12 @ 0xc
-; CHECK-NEXT:    .long 18 @ 0x12
+; CHECK-NEXT:    .long 4294967200 @ 0xffffffa0
+; CHECK-NEXT:    .long 4294967224 @ 0xffffffb8
+; CHECK-NEXT:    .long 4294967248 @ 0xffffffd0
+; CHECK-NEXT:    .long 4294967272 @ 0xffffffe8
 vector.ph41:                                      ; preds = %for.body6.preheader
   %ind.end47 = shl i32 %n.vec43, 1
   br label %vector.body39
 
 vector.body39:                                    ; preds = %vector.body39, %vector.ph41
   %index44 = phi i32 [ 0, %vector.ph41 ], [ %index.next45, %vector.body39 ]
   %vec.ind50 = phi <4 x i32> [ <i32 0, i32 2, i32 4, i32 6>, %vector.ph41 ], [ %vec.ind.next51, %vector.body39 ]
   %0 = mul nuw nsw <4 x i32> %vec.ind50, <i32 3, i32 3, i32 3, i32 3>

 ; CHECK-LABEL: gather_inc_v4i32_simple:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, lr}
 ; CHECK-NEXT:    push {r4, lr}
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    poplt {r4, pc}
 ; CHECK-NEXT:    bic r12, r2, #3
-; CHECK-NEXT:    movs r4, #1
-; CHECK-NEXT:    sub.w r3, r12, #4
-; CHECK-NEXT:    vmov.i32 q1, #0x4
-; CHECK-NEXT:    add.w r4, r4, r3, lsr #2
+; CHECK-NEXT:    movs r3, #1
+; CHECK-NEXT:    sub.w lr, r12, #4
+; CHECK-NEXT:    add.w r4, r3, lr, lsr #2
 ; CHECK-NEXT:    adr r3, .LCPI8_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB8_1: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB8_2 Depth 2
-; CHECK-NEXT:    mov r3, r1
-; CHECK-NEXT:    vmov q2, q0
+; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:    dls lr, r4
 ; CHECK-NEXT:  .LBB8_2: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB8_1 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT:    vldrw.u32 q3, [r0, q2, uxtw #2]
-; CHECK-NEXT:    vadd.i32 q2, q2, q1
-; CHECK-NEXT:    vstrb.8 q3, [r3], #16
+; CHECK-NEXT:    vldrw.u32 q2, [q1, #16]!
+; CHECK-NEXT:    vstrb.8 q2, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB8_2
 ; CHECK-NEXT:  @ %bb.3: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB8_1 Depth=1
 ; CHECK-NEXT:    cmp r12, r2
 ; CHECK-NEXT:    bne .LBB8_1
 ; CHECK-NEXT:  @ %bb.4: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r4, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.5:
 ; CHECK-NEXT:  .LCPI8_0:
-; CHECK-NEXT:    .long 0 @ 0x0
-; CHECK-NEXT:    .long 1 @ 0x1
-; CHECK-NEXT:    .long 2 @ 0x2
-; CHECK-NEXT:    .long 3 @ 0x3
+; CHECK-NEXT:    .long 4294967280 @ 0xfffffff0
+; CHECK-NEXT:    .long 4294967284 @ 0xfffffff4
+; CHECK-NEXT:    .long 4294967288 @ 0xfffffff8
+; CHECK-NEXT:    .long 4294967292 @ 0xfffffffc
 entry:
   %cmp22 = icmp sgt i32 %n, 0
   br i1 %cmp22, label %vector.ph, label %for.cond.cleanup
 
 vector.ph:                                        ; preds = %for.body.preheader
   %n.vec = and i32 %n, -4
   br label %vector.body
 

 
 for.cond.cleanup:                                 ; preds = %for.body, %middle.block, %entry
   ret void
 }
 
 define arm_aapcs_vfpcc void @gather_inc_v4i32_complex(i32* noalias nocapture readonly %data, i32* noalias nocapture %dst, i32 %n) {
 ; CHECK-LABEL: gather_inc_v4i32_complex:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, r5, r7, lr}
-; CHECK-NEXT:    push {r4, r5, r7, lr}
-; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    .pad #48
-; CHECK-NEXT:    sub sp, #48
+; CHECK-NEXT:    .save {r4, lr}
+; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    .vsave {d8, d9}
+; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    blt .LBB9_5
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph.preheader
 ; CHECK-NEXT:    bic r12, r2, #3
-; CHECK-NEXT:    movs r4, #1
-; CHECK-NEXT:    sub.w r3, r12, #4
-; CHECK-NEXT:    adr r5, .LCPI9_2
-; CHECK-NEXT:    adr.w lr, .LCPI9_0
-; CHECK-NEXT:    vmov.i32 q3, #0xc
-; CHECK-NEXT:    add.w r4, r4, r3, lsr #2
-; CHECK-NEXT:    adr r3, .LCPI9_1
+; CHECK-NEXT:    movs r3, #1
+; CHECK-NEXT:    sub.w lr, r12, #4
+; CHECK-NEXT:    add.w r4, r3, lr, lsr #2
+; CHECK-NEXT:    adr r3, .LCPI9_0
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
-; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
-; CHECK-NEXT:    vldrw.u32 q0, [r5]
-; CHECK-NEXT:    vstrw.32 q0, [sp, #16] @ 16-byte Spill
-; CHECK-NEXT:    vldrw.u32 q0, [lr]
-; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB9_2: @ %vector.ph
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB9_3 Depth 2
-; CHECK-NEXT:    mov r3, r1
-; CHECK-NEXT:    vldrw.u32 q4, [sp, #16] @ 16-byte Reload
-; CHECK-NEXT:    vldrw.u32 q5, [sp, #32] @ 16-byte Reload
-; CHECK-NEXT:    vldrw.u32 q6, [sp] @ 16-byte Reload
+; CHECK-NEXT:    mov r0, r1
+; CHECK-NEXT:    vmov q2, q0
 ; CHECK-NEXT:    dls lr, r4
+; CHECK-NEXT:    vmov q1, q0
 ; CHECK-NEXT:  .LBB9_3: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB9_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT:    vadd.i32 q7, q6, q3
-; CHECK-NEXT:    vldrw.u32 q2, [r0, q6, uxtw #2]
-; CHECK-NEXT:    vldrw.u32 q6, [r0, q4, uxtw #2]
-; CHECK-NEXT:    vadd.i32 q1, q4, q3
-; CHECK-NEXT:    vldrw.u32 q4, [r0, q5, uxtw #2]
-; CHECK-NEXT:    vadd.i32 q0, q5, q3
-; CHECK-NEXT:    vadd.i32 q2, q6, q2
-; CHECK-NEXT:    vadd.i32 q2, q2, q4
-; CHECK-NEXT:    vmov q4, q1
-; CHECK-NEXT:    vstrb.8 q2, [r3], #16
-; CHECK-NEXT:    vmov q5, q0
-; CHECK-NEXT:    vmov q6, q7
+; CHECK-NEXT:    vldrw.u32 q3, [q1, #48]!
+; CHECK-NEXT:    vldrw.u32 q4, [q2, #4]
+; CHECK-NEXT:    vadd.i32 q3, q4, q3
+; CHECK-NEXT:    vldrw.u32 q4, [q2, #8]
+; CHECK-NEXT:    vadd.i32 q2, q3, q4
+; CHECK-NEXT:    vstrb.8 q2, [r0], #16
+; CHECK-NEXT:    vmov q2, q1
 ; CHECK-NEXT:    le lr, .LBB9_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB9_2 Depth=1
 ; CHECK-NEXT:    cmp r12, r2
 ; CHECK-NEXT:    bne .LBB9_2
 ; CHECK-NEXT:  .LBB9_5: @ %for.cond.cleanup
-; CHECK-NEXT:    add sp, #48
-; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    pop {r4, r5, r7, pc}
+; CHECK-NEXT:    vpop {d8, d9}
+; CHECK-NEXT:    pop {r4, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.6:
 ; CHECK-NEXT:  .LCPI9_0:
-; CHECK-NEXT:    .long 0 @ 0x0
-; CHECK-NEXT:    .long 3 @ 0x3
-; CHECK-NEXT:    .long 6 @ 0x6
-; CHECK-NEXT:    .long 9 @ 0x9
-; CHECK-NEXT:  .LCPI9_1:
-; CHECK-NEXT:    .long 2 @ 0x2
-; CHECK-NEXT:    .long 5 @ 0x5
-; CHECK-NEXT:    .long 8 @ 0x8
-; CHECK-NEXT:    .long 11 @ 0xb
-; CHECK-NEXT:  .LCPI9_2:
-; CHECK-NEXT:    .long 1 @ 0x1
-; CHECK-NEXT:    .long 4 @ 0x4
-; CHECK-NEXT:    .long 7 @ 0x7
-; CHECK-NEXT:    .long 10 @ 0xa
+; CHECK-NEXT:    .long 4294967248 @ 0xffffffd0
+; CHECK-NEXT:    .long 4294967260 @ 0xffffffdc
+; CHECK-NEXT:    .long 4294967272 @ 0xffffffe8
+; CHECK-NEXT:    .long 4294967284 @ 0xfffffff4
 entry:
   %cmp22 = icmp sgt i32 %n, 0
   br i1 %cmp22, label %vector.ph, label %for.cond.cleanup
 
 vector.ph:                                        ; preds = %for.body.preheader
   %n.vec = and i32 %n, -4
   br label %vector.body
 

llvm/test/CodeGen/Thumb2/mve-gather-optimisation.ll

 !10 = !{!"llvm.loop.unroll.runtime.disable"}
 
 
 
 define arm_aapcs_vfpcc void @push_out_mul(i32* noalias nocapture readonly %data, i32* noalias nocapture %dst, i32 %n.vec) {
 ; CHECK-LABEL: push_out_mul:
 ; CHECK:       @ %bb.0: @ %vector.ph
 ; CHECK-NEXT:    adr r3, .LCPI0_0
-; CHECK-NEXT:    vmov.i32 q1, #0x18
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0, q0, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q1, [q0, #96]!
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vstrb.8 q2, [r1], #16
+; CHECK-NEXT:    vstrb.8 q1, [r1], #16
 ; CHECK-NEXT:    bne .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %end
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI0_0:
-; CHECK-NEXT:    .long 0 @ 0x0
-; CHECK-NEXT:    .long 6 @ 0x6
-; CHECK-NEXT:    .long 12 @ 0xc
-; CHECK-NEXT:    .long 18 @ 0x12
+; CHECK-NEXT:    .long 4294967200 @ 0xffffffa0
+; CHECK-NEXT:    .long 4294967224 @ 0xffffffb8
+; CHECK-NEXT:    .long 4294967248 @ 0xffffffd0
+; CHECK-NEXT:    .long 4294967272 @ 0xffffffe8
 
 vector.ph:                                        ; preds = %for.body.preheader
   %ind.end = shl i32 %n.vec, 1
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %vector.ph
   %index = phi i32 [ 0, %vector.ph ], [ %index.next, %vector.body ]
   %vec.ind = phi <4 x i32> [ <i32 0, i32 2, i32 4, i32 6>, %vector.ph ], [ %vec.ind.next, %vector.body ]

 end:
   ret void;
 }
 
 define arm_aapcs_vfpcc void @push_out_add(i32* noalias nocapture readonly %data, i32* noalias nocapture %dst, i32 %n.vec) {
 ; CHECK-LABEL: push_out_add:
 ; CHECK:       @ %bb.0: @ %vector.ph
 ; CHECK-NEXT:    adr r3, .LCPI1_0
-; CHECK-NEXT:    vmov.i32 q1, #0x8
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0, q0, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q1, [q0, #32]!
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vstrb.8 q2, [r1], #16
+; CHECK-NEXT:    vstrb.8 q1, [r1], #16
 ; CHECK-NEXT:    bne .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %end
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI1_0:
-; CHECK-NEXT:    .long 6 @ 0x6
+; CHECK-NEXT:    .long 4294967288 @ 0xfffffff8
+; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 8 @ 0x8
-; CHECK-NEXT:    .long 10 @ 0xa
-; CHECK-NEXT:    .long 12 @ 0xc
+; CHECK-NEXT:    .long 16 @ 0x10
 
 vector.ph:                                        ; preds = %for.body.preheader
   %ind.end = shl i32 %n.vec, 1
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %vector.ph
   %index = phi i32 [ 0, %vector.ph ], [ %index.next, %vector.body ]
   %vec.ind = phi <4 x i32> [ <i32 0, i32 2, i32 4, i32 6>, %vector.ph ], [ %vec.ind.next, %vector.body ]

 end:
   ret void;
 }
 
 define arm_aapcs_vfpcc void @push_out_add_sub_block(i32* noalias nocapture readonly %data, i32* noalias nocapture %dst, i32 %n.vec) {
 ; CHECK-LABEL: push_out_add_sub_block:
 ; CHECK:       @ %bb.0: @ %vector.ph
 ; CHECK-NEXT:    adr r3, .LCPI2_0
-; CHECK-NEXT:    vmov.i32 q1, #0x8
 ; CHECK-NEXT:    vldrw.u32 q0, [r3]
+; CHECK-NEXT:    vadd.i32 q0, q0, r0
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0, q0, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q1, [q0, #32]!
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vstrb.8 q2, [r1], #16
+; CHECK-NEXT:    vstrb.8 q1, [r1], #16
 ; CHECK-NEXT:    bne .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %end
 ; CHECK-NEXT:    bx lr
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.3:
 ; CHECK-NEXT:  .LCPI2_0:
-; CHECK-NEXT:    .long 6 @ 0x6
+; CHECK-NEXT:    .long 4294967288 @ 0xfffffff8
+; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 8 @ 0x8
-; CHECK-NEXT:    .long 10 @ 0xa
-; CHECK-NEXT:    .long 12 @ 0xc
+; CHECK-NEXT:    .long 16 @ 0x10
 
 vector.ph:                                        ; preds = %for.body.preheader
   %ind.end = shl i32 %n.vec, 1
   br label %vector.body
 
 vector.body:                                      ; preds = %vector.body, %vector.ph
   %index = phi i32 [ 0, %vector.ph ], [ %index.next, %vector.body.end ]
   %vec.ind = phi <4 x i32> [ <i32 0, i32 2, i32 4, i32 6>, %vector.ph ], [ %vec.ind.next, %vector.body.end ]

 
 define dso_local void @arm_mat_mult_q31(i32* noalias nocapture readonly %A, i32* noalias nocapture readonly %B, i32* noalias nocapture %C, i32 %n, i32 %m, i32 %l) local_unnamed_addr #0 {
 ; CHECK-LABEL: arm_mat_mult_q31:
 ; CHECK:       @ %bb.0: @ %for.cond8.preheader.us.us.preheader.preheader
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    .pad #48
-; CHECK-NEXT:    sub sp, #48
+; CHECK-NEXT:    .pad #32
+; CHECK-NEXT:    sub sp, #32
 ; CHECK-NEXT:    adr r6, .LCPI5_0
-; CHECK-NEXT:    ldrd r9, r12, [sp, #144]
+; CHECK-NEXT:    ldrd r9, r12, [sp, #128]
 ; CHECK-NEXT:    vldrw.u32 q0, [r6]
 ; CHECK-NEXT:    sub.w r6, r12, #1
 ; CHECK-NEXT:    movs r7, #1
 ; CHECK-NEXT:    vdup.32 q2, r9
 ; CHECK-NEXT:    add.w r6, r7, r6, lsr #1
 ; CHECK-NEXT:    vstrw.32 q0, [sp] @ 16-byte Spill
 ; CHECK-NEXT:    bic r6, r6, #3
 ; CHECK-NEXT:    vmul.i32 q0, q0, r9
 ; CHECK-NEXT:    subs r6, #4
 ; CHECK-NEXT:    vshl.i32 q2, q2, #3
 ; CHECK-NEXT:    mov.w r8, #0
-; CHECK-NEXT:    vmov.i32 q3, #0x8
+; CHECK-NEXT:    vmov.i32 q3, #0x20
 ; CHECK-NEXT:    add.w r4, r7, r6, lsr #2
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #16] @ 16-byte Spill
 ; CHECK-NEXT:  .LBB5_1: @ %for.cond8.preheader.us.us.preheader
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB5_2 Depth 2
 ; CHECK-NEXT:    @ Child Loop BB5_3 Depth 3
 ; CHECK-NEXT:    mul r10, r8, r9
 ; CHECK-NEXT:    vldrw.u32 q0, [sp] @ 16-byte Reload
 ; CHECK-NEXT:    mul r7, r8, r12
-; CHECK-NEXT:    vadd.i32 q0, q0, r7
+; CHECK-NEXT:    vadd.i32 q4, q0, r7
 ; CHECK-NEXT:    movs r7, #0
-; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:  .LBB5_2: @ %vector.ph
 ; CHECK-NEXT:    @ Parent Loop BB5_1 Depth=1
 ; CHECK-NEXT:    @ => This Loop Header: Depth=2
 ; CHECK-NEXT:    @ Child Loop BB5_3 Depth 3
-; CHECK-NEXT:    vmov.i32 q5, #0x0
-; CHECK-NEXT:    vldrw.u32 q6, [sp, #32] @ 16-byte Reload
+; CHECK-NEXT:    vshl.i32 q6, q4, #2
+; CHECK-NEXT:    vadd.i32 q6, q6, r0
 ; CHECK-NEXT:    vldrw.u32 q7, [sp, #16] @ 16-byte Reload
+; CHECK-NEXT:    vmov.i32 q5, #0x0
+; CHECK-NEXT:    vsub.i32 q6, q6, q3
 ; CHECK-NEXT:    dls lr, r4
 ; CHECK-NEXT:  .LBB5_3: @ %vector.body
 ; CHECK-NEXT:    @ Parent Loop BB5_1 Depth=1
 ; CHECK-NEXT:    @ Parent Loop BB5_2 Depth=2
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=3
 ; CHECK-NEXT:    vadd.i32 q0, q7, q2
 ; CHECK-NEXT:    vadd.i32 q7, q7, r7
-; CHECK-NEXT:    vldrw.u32 q4, [r1, q7, uxtw #2]
-; CHECK-NEXT:    vldrw.u32 q7, [r0, q6, uxtw #2]
-; CHECK-NEXT:    vadd.i32 q1, q6, q3
-; CHECK-NEXT:    vmul.i32 q4, q4, q7
-; CHECK-NEXT:    vmov q6, q1
-; CHECK-NEXT:    vadd.i32 q5, q4, q5
+; CHECK-NEXT:    vldrw.u32 q1, [r1, q7, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q7, [q6, #32]!
+; CHECK-NEXT:    vmul.i32 q1, q1, q7
 ; CHECK-NEXT:    vmov q7, q0
+; CHECK-NEXT:    vadd.i32 q5, q1, q5
 ; CHECK-NEXT:    le lr, .LBB5_3
 ; CHECK-NEXT:  @ %bb.4: @ %middle.block
 ; CHECK-NEXT:    @ in Loop: Header=BB5_2 Depth=2
 ; CHECK-NEXT:    add.w r5, r7, r10
 ; CHECK-NEXT:    adds r7, #1
 ; CHECK-NEXT:    vaddv.u32 r6, q5
 ; CHECK-NEXT:    cmp r7, r9
 ; CHECK-NEXT:    str.w r6, [r2, r5, lsl #2]
 ; CHECK-NEXT:    bne .LBB5_2
 ; CHECK-NEXT:  @ %bb.5: @ %for.cond4.for.cond.cleanup6_crit_edge.us
 ; CHECK-NEXT:    @ in Loop: Header=BB5_1 Depth=1
 ; CHECK-NEXT:    add.w r8, r8, #1
 ; CHECK-NEXT:    cmp r8, r3
 ; CHECK-NEXT:    bne .LBB5_1
 ; CHECK-NEXT:  @ %bb.6: @ %for.end25
-; CHECK-NEXT:    add sp, #48
+; CHECK-NEXT:    add sp, #32
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.7:
 ; CHECK-NEXT:  .LCPI5_0:
 ; CHECK-NEXT:    .long 0 @ 0x0
 ; CHECK-NEXT:    .long 2 @ 0x2
 ; CHECK-NEXT:    .long 4 @ 0x4