[LSR] Generate formulae to enable more indexed accesses

[samparker]

Summary

Modify GenerateConstantOffsetsImpl to create offsets that can be used by indexed addressing modes. If formulae can be generated which result in the constant offset being the same size as the recurrence, we can generate an indexed access.

The resulting code, at least for Arm, is that usually pre-indexed loads are used as the last access, but sometimes the first.

@kparzysz Would you be able to provide feedback on how this effects Hexagon? It's a target that I don't build and I haven't looked at the tests, but I'm assuming this would interest you.

[gilr]

test/CodeGen/ARM/loop-align-cortex-m.ll

3

You're just fixing this by the way, right?

test/Transforms/LoopStrengthReduce/ARM/complexity.ll

10

Shouldn't -DEFAULT be removed here too?

[samparker]

test/CodeGen/ARM/loop-align-cortex-m.ll

3

yes

test/Transforms/LoopStrengthReduce/ARM/complexity.ll

10

thanks!

[kparzysz]

I've run some tests and the results are not great for us. On some tests we got up to 5.5% improvements, but there are a lot of severe degradations (15+% worse). If this patch goes in, we'd like to be able to opt out.

[samparker]

Okay, thanks. We're also seeing some regressions, so I know I've got some tuning to do. Do you have any idea of the characteristics of your regressions? At the moment I'm thinking:

• That the costs that I've added here are overly simplistic, for one I think I need to add a setup cost.
• It's also probably not worth doing when we know that the loop iteration count is low.
• In the current state, we also see code size regressions whereas your previous work helps us reduce code size. It may mean that I'll need a different flag to enable this change, but it also maybe a symptom of the performance regressions.

[gilr]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

3793

Is this relevant for non-Address kind formulae?

3793

Worth a comment here to describe the motivation and how adjusting the offset generates the post-inc opportunity.

3795

How does a non-constant step (e.g. 50 + %x) translate to post-inc? Could you add such a test case?

[samparker]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

3793

No. I had naively assumed that this function was only generating formulae for addresses... so if that's not the case, I'll make the change here.

3795

Ah, good catch. This should only trigger for constant steps.

[samparker]

I've moved the logic under the control of a new TTI flag as it seems that the current shouldFavourPostInc is trying to achieve different things. Hopefully I've also addressed Gil's comments.

[gilr]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

1258

• The existing {LI, +, C} pattern seems to already match this case (i.e. {(-C + %a + ...), +, C}) and the more general case {(Offset - C2) + %a + ...), +, C2}, where Offset =/= 0. So IIUC matching this pattern here is only needed if the new TTI flag is set but the existing one is reset, right?
• Won't this also match something like {3, +, 5, +, %x}?

1384

IIRC single-statement clauses shouldn't get curly braces.

3802

The new TTI API relates to the same HW feature, right? Why not use a cl::opt in LSR to turn this optimization off?

[samparker]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

1258

I'm going to need to replace this with something that compares the step with the base offset, aiming to have the post increment happen on the last, not the first, access. I need to make a few other changes to support this though as well.

[samparker]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

3802

When I've got the next patch ready, I will re-run my tests and see if this is viable. It may not be good for all targets, and if so, I think a target hook would be more useful.

[samparker]

• Added a command line option to 'EnableBackedgePostIncs'.
• Added a command line option to enable narrowing the search space by collapsing unrolled code.
• The TTI hook now accepts the loop so that the target can make a more informed decision on when it 'shouldFavorBackedgePostIncs'.
• LSRInstance contains a boolean 'FavorBackedgePostInc' which is equal to EnableBackedgePostIncs && shouldFavorBackedgePostIncs.
• When FavorBackedgePostIncs:
  • Generate the new constant offsets.
  • In RateRegister, the LoopCost is now set to 0 if the step recurrence is equal to the base offset of the parent formula.
  • IsProfitableChain has a higher limit
  • The last expression in the IVChain is not added to IVIncSet so it's a target for optimising.

[samparker]

some renaming, renamed post inc bits to 'index'.

[samparker]

Changed the default value for the command line option 'EnableBackedgeIndexing' to false.

[samparker]

Removed changes to the complexity.ll test.

[samparker]

ping

[hfinkel]

In D55373#1367469, @samparker wrote:

ping

Are you still seeing regressions? Can you provide performance data with the current patch?

[samparker]

Hi Hal,

I've attached a graph to show geomean performance results of a popular embedded benchmark suite running on Arm microcontroller. The suite is broken into five sub-suites and only three of which are affected by these changes. The large regression comes from a test which often exhibits bi-modal behaviour, so I could be getting unlucky. The whole suite comprises of several dozen tests and I only four of the benchmarks have minor regressions across the three runs. There's also some backend work that I need to do for Arm to get this optimised properly.

[gilr]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

159

False by default?

4466

The convention for flags controlling the narrowing heuristics seem to be to use them in NarrowSearchSpaceUsingHeuristics() rather than in the the functions they affect.

4466

I think you meant '||' here.

[samparker]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

159

This option has been introduced to force the collapsing, even if the complexity limit hasn't been reached. Which is why it's implemented differently to the other, similar, options.

[gilr]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

159

Aaargh, sorry, read it backwards ...
Is it needed since unrolled code doesn't initially go into the same LSRUse? If so, can FavorBackedgeIndex be used to have initial construction put them in the same LSRUse?

2904

Deserves a comment.

3121

I'm a bit confused here. IIUC a profitable complete chain prvides an efficient solution using post-increments. Why break the last increment out?

[samparker]

lib/Transforms/Scalar/LoopStrengthReduce.cpp

159

I will look into this, thanks.

3121

This was so that the last access could also be a target for optimisation. I'll remove the change and do some testing again to see the difference.

[samparker]

Made some simplifications:

• reset isProfitableChain.
• reset FinalizeChain so that the tail is added to the chain again.
• removed the CollapseUnrolled option because with the reset in changes, it wasn't really interesting.

I'll post the updated performance numbers.

[samparker]

[gilr]

include/llvm/Analysis/TargetTransformInfo.h

489 ↗

(On Diff #185268)

Please add a doxygen comment.

lib/Target/ARM/ARMTargetTransformInfo.h

98

Is this optimization inherently code-size unfriendly for ARM? (The patch actually reduces the instruction count in LSR's Cost when this optimization kicks in)

100

Is the single-block constraint due to CodeGen's single-block optimization scope? (If so, then IINM it's not target-specific)

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.h

98

There's two reasons really: the transform is most useful in 'unrolled' loops (which we disable when optimising for code size), and this transform will introduce instructions into the preheader and if the address can't be kept in the same register, we'll also produce moves. So this is mainly a defensive restriction, because I haven't been tracking code size, but I would hope that I can remove the restriction later.

100

No, its not because of ISel restrictions or anything like that. It's because the transform is only likely to be useful is the address can be kept in the same register - which becomes increasingly less likely once multiple blocks are considered.

[samparker]

Added doxygen comment.

[gilr]

lib/Target/ARM/ARMTargetTransformInfo.h

100

So both the code-size and single-block heuristics seem target-independent. Why not do this in LSR?

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.h

100

I'd argue because different backends would come to different conclusions to me. I've gone for a very simplistic heuristic, but it would be good to consider register pressure rather than just the number of blocks. Also, some targets may not support indexed accesses on certain types of memory operations and so it worthless generating formulae for them.

[gilr]

LGTM!
(One last nitpick: I'd consider letting EnableBackedgeIndexing default to true as TTI's default already disables it for Hexagon and all other targets)

[samparker]

Will do. Thanks for the review!

lib/Target/ARM/ARMTargetTransformInfo.h

explicit ARMTTIImpl(const ARMBaseTargetMachine *TM, const Function &F)
       : BaseT(TM, F.getParent()->getDataLayout()), ST(TM->getSubtargetImpl(F)),
         TLI(ST->getTargetLowering()) {}
 
   bool areInlineCompatible(const Function *Caller,
                            const Function *Callee) const;
 
   bool enableInterleavedAccessVectorization() { return true; }
 
+  bool shouldFavorPostInc() const { return ST->isMClass() && ST->isThumb2(); }
+
   /// Floating-point computation using ARMv8 AArch32 Advanced
   /// SIMD instructions remains unchanged from ARMv7. Only AArch64 SIMD
   /// is IEEE-754 compliant, but it's not covered in this target.
   bool isFPVectorizationPotentiallyUnsafe() {
     return !ST->isTargetDarwin();
   }
 
   /// \name Scalar TTI Implementations
   /// @{
 

lib/Transforms/Scalar/LoopStrengthReduce.cpp

 } // end anonymous namespace
 
 static bool isAMCompletelyFolded(const TargetTransformInfo &TTI,
                                  LSRUse::KindType Kind, MemAccessTy AccessTy,
                                  GlobalValue *BaseGV, int64_t BaseOffset,
                                  bool HasBaseReg, int64_t Scale,
                                  Instruction *Fixup = nullptr);
 
+static const SCEVConstant* GetConstantStart(const SCEV *S) {
+  if (auto *C = dyn_cast<SCEVConstant>(S))
+    return C;
+
+  if (auto *AddRec = dyn_cast<SCEVAddRecExpr>(S))
+    return GetConstantStart(AddRec->getStart());
+
+  if (auto *Add = dyn_cast<SCEVAddExpr>(S))
+    return GetConstantStart(Add->getOperand(0));
+
+  return nullptr;
+}
+
 /// Tally up interesting quantities from the given register.
 void Cost::RateRegister(const SCEV *Reg,
                         SmallPtrSetImpl<const SCEV *> &Regs,
                         const Loop *L,
                         ScalarEvolution &SE, DominatorTree &DT,
                         const TargetTransformInfo &TTI) {
+  unsigned NumRegs = 1;
   if (const SCEVAddRecExpr *AR = dyn_cast<SCEVAddRecExpr>(Reg)) {
     // If this is an addrec for another loop, it should be an invariant
     // with respect to L since L is the innermost loop (at least
     // for now LSR only handles innermost loops).
     if (AR->getLoop() != L) {
       // If the AddRec exists, consider it's register free and leave it alone.
       if (isExistingPhi(AR, SE))
         return;
 
       // It is bad to allow LSR for current loop to add induction variables
       // for its sibling loops.
       if (!AR->getLoop()->contains(L)) {
         Lose();
         return;
       }
 
       // Otherwise, it will be an invariant with respect to Loop L.
       ++C.NumRegs;
       return;
     }
 
     unsigned LoopCost = 1;
-    if (TTI.shouldFavorPostInc()) {
+    if (TTI.shouldFavorPostInc() &&
+        (TTI.isIndexedLoadLegal(TTI.MIM_PostInc, AR->getType()) ||
+         TTI.isIndexedStoreLegal(TTI.MIM_PostInc, AR->getType()))) {
+
+      if (isa<SCEVConstant>(AR->getOperand(1))) {
+        if (auto *Start = GetConstantStart(AR)) {
+          const APInt &StartInt = Start->getAPInt();
+          const APInt &ARInt = cast<SCEVConstant>(AR->getOperand(1))->getAPInt();
+          // We can turn this access into a post increment as the initial offset
+          // required matches the recurrence.
+          if ((StartInt.isNegative() && StartInt.abs() == ARInt) ||
+              (ARInt.isNegative() && ARInt.abs() == StartInt))
+            NumRegs = 0;
+        }
+      }
+
       const SCEV *LoopStep = AR->getStepRecurrence(SE);
       if (isa<SCEVConstant>(LoopStep)) {
-        // Check if a post-indexed load/store can be used.
-        if (TTI.isIndexedLoadLegal(TTI.MIM_PostInc, AR->getType()) ||
-            TTI.isIndexedStoreLegal(TTI.MIM_PostInc, AR->getType())) {
-          const SCEV *LoopStart = AR->getStart();
-          if (!isa<SCEVConstant>(LoopStart) &&
+        const SCEV *LoopStart = AR->getStart();
+        if (!isa<SCEVConstant>(LoopStart) &&
             SE.isLoopInvariant(LoopStart, L))
-              LoopCost = 0;
-        }
       }
     }
     C.AddRecCost += LoopCost;
 
     // Add the step value register, if it needs one.
     // TODO: The non-affine case isn't precisely modeled here.
     if (!AR->isAffine() || !isa<SCEVConstant>(AR->getOperand(1))) {
       if (!Regs.count(AR->getOperand(1))) {
         RateRegister(AR->getOperand(1), Regs, L, SE, DT, TTI);
         if (isLoser())
           return;
       }
     }
   }
-  ++C.NumRegs;
 
+  C.NumRegs += NumRegs;
   // Rough heuristic; favor registers which don't require extra setup
   // instructions in the preheader.
   if (!isa<SCEVUnknown>(Reg) &&
       !isa<SCEVConstant>(Reg) &&
       !(isa<SCEVAddRecExpr>(Reg) &&
         (isa<SCEVUnknown>(cast<SCEVAddRecExpr>(Reg)->getStart()) ||
          isa<SCEVConstant>(cast<SCEVAddRecExpr>(Reg)->getStart()))))
     ++C.SetupCost;

       C.ImmCost += APInt(64, Offset, true).getMinSignedBits();
 
     // Check with target if this offset with this instruction is
     // specifically not supported.
     if (LU.Kind == LSRUse::Address && Offset != 0 &&
         !isAMCompletelyFolded(TTI, LSRUse::Address, LU.AccessTy, F.BaseGV,
                               Offset, F.HasBaseReg, F.Scale, Fixup.UserInst))
       C.NumBaseAdds++;
   }
 
   // If we don't count instruction cost exit here.
   if (!InsnsCost) {
     assert(isValid() && "invalid cost");
     return;
   }
 
   // Treat every new register that exceeds TTI.getNumberOfRegisters() - 1 as

                     << "\n");
 
   return cost < 0;
 }
 
 /// Add this IV user to an existing chain or make it the head of a new chain.
 void LSRInstance::ChainInstruction(Instruction *UserInst, Instruction *IVOper,
                                    SmallVectorImpl<ChainUsers> &ChainUsersVec) {
   // When IVs are used as types of varying widths, they are generally converted
   // to a wider type with some uses remaining narrow under a (free) trunc.
   Value *const NextIV = getWideOperand(IVOper);
   const SCEV *const OperExpr = SE.getSCEV(NextIV);
   const SCEV *const OperExprBase = getExprBase(OperExpr);
 
   // Visit all existing chains. Check if its IVOper can be computed as a
   // profitable loop invariant increment from the last link in the Chain.
   unsigned ChainIdx = 0, NChains = IVChainVec.size();

 }
 
 /// Return true if the IVInc can be folded into an addressing mode.
 static bool canFoldIVIncExpr(const SCEV *IncExpr, Instruction *UserInst,
                              Value *Operand, const TargetTransformInfo &TTI) {
   const SCEVConstant *IncConst = dyn_cast<SCEVConstant>(IncExpr);
   if (!IncConst || !isAddressUse(TTI, UserInst, Operand))
     return false;
 
   if (IncConst->getAPInt().getMinSignedBits() > 64)
     return false;
 
   MemAccessTy AccessTy = getAccessType(TTI, UserInst, Operand);
   int64_t IncOffset = IncConst->getValue()->getSExtValue();
   if (!isAlwaysFoldable(TTI, LSRUse::Address, AccessTy, /*BaseGV=*/nullptr,
                         IncOffset, /*HaseBaseReg=*/false))
     return false;

     GenerateSymbolicOffsetsImpl(LU, LUIdx, Base, /* Idx */ -1,
                                 /* IsScaledReg */ true);
 }
 
 /// Helper function for LSRInstance::GenerateConstantOffsets.
 void LSRInstance::GenerateConstantOffsetsImpl(
     LSRUse &LU, unsigned LUIdx, const Formula &Base,
     const SmallVectorImpl<int64_t> &Worklist, size_t Idx, bool IsScaledReg) {
-  const SCEV *G = IsScaledReg ? Base.ScaledReg : Base.BaseRegs[Idx];
-  for (int64_t Offset : Worklist) {
+
+  auto GenerateOffset = [&](const SCEV *G, int64_t Offset) {
     Formula F = Base;
     F.BaseOffset = (uint64_t)Base.BaseOffset - Offset;
     if (isLegalUse(TTI, LU.MinOffset - Offset, LU.MaxOffset - Offset, LU.Kind,
                    LU.AccessTy, F)) {
       // Add the offset to the base register.
       const SCEV *NewG = SE.getAddExpr(SE.getConstant(G->getType(), Offset), G);
       // If it cancelled out, drop the base register, otherwise update it.
       if (NewG->isZero()) {
         if (IsScaledReg) {
           F.Scale = 0;
           F.ScaledReg = nullptr;
         } else
           F.deleteBaseReg(F.BaseRegs[Idx]);
         F.canonicalize(*L);
       } else if (IsScaledReg)
         F.ScaledReg = NewG;
       else
         F.BaseRegs[Idx] = NewG;
 
       (void)InsertFormula(LU, LUIdx, F);
     }
+  };
+
+  const SCEV *G = IsScaledReg ? Base.ScaledReg : Base.BaseRegs[Idx];
+
+  if (TTI.shouldFavorPostInc()) {
+    if (auto *GAddRec = dyn_cast<SCEVAddRecExpr>(G)) {
+      if (auto *StepSCEV = GetConstantStart(GAddRec->getStepRecurrence(SE))) {
+        const APInt &StepInt = StepSCEV->getAPInt();
+        int64_t Step = StepInt.isNegative() ?
+          StepInt.getSExtValue() : StepInt.getZExtValue();
+
+        for (int64_t Offset : Worklist) {
+          Offset -= Step;
+          GenerateOffset(G, Offset);
+        }
+      }
+    }
   }
 
+  for (int64_t Offset : Worklist)
+    GenerateOffset(G, Offset);
+
   int64_t Imm = ExtractImmediate(G, SE);
   if (G->isZero() || Imm == 0)
     return;
   Formula F = Base;
   F.BaseOffset = (uint64_t)F.BaseOffset + Imm;
   if (!isLegalUse(TTI, LU.MinOffset, LU.MaxOffset, LU.Kind, LU.AccessTy, F))
     return;
   if (IsScaledReg)

   for (size_t LUIdx = 0, NumUses = Uses.size(); LUIdx != NumUses; ++LUIdx) {
     LSRUse &LU = Uses[LUIdx];
     for (const Formula &F : LU.Formulae) {
       if (F.BaseOffset == 0 || (F.Scale != 0 && F.Scale != 1))
         continue;
 
       LSRUse *LUThatHas = FindUseWithSimilarFormula(F, LU);
       if (!LUThatHas)
         continue;
 
       if (!reconcileNewOffset(*LUThatHas, F.BaseOffset, /*HasBaseReg=*/ false,
                               LU.Kind, LU.AccessTy))
         continue;
 
       LLVM_DEBUG(dbgs() << "  Deleting use "; LU.print(dbgs()); dbgs() << '\n');
 
       LUThatHas->AllFixupsOutsideLoop &= LU.AllFixupsOutsideLoop;

test/CodeGen/ARM/dsp-post-incs.ll

+; RUN: llc -mtriple=thumbv8m.main -mattr=+dsp,+fp-armv8 %s -o - | FileCheck %s
+
+; Tests to check that post increment addressing modes are used instead of
+; updating base pointers with add instructions.
+
+; CHECK-LABEL: test_qadd_2
+; CHECK: sub{{.*}} [[A:r[0-9]+]], r0, #8
+; CHECK: subs [[B:r[0-9]+]], #8
+; CHECK: subs [[OUT:r[0-9]+]], #8
+
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #8]!
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #8]!
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #8]!
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #4]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #4]
+; CHECK: blo
+define void @test_qadd_2(i32* %a.array, i32* %b.array, i32* %out.array, i32 %N) {
+entry:
+  br label %loop
+
+loop:
+  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+  %idx.1 = phi i32 [ 0, %entry ], [ %idx.next, %loop ]
+  %gep.a.1 = getelementptr inbounds i32, i32* %a.array, i32 %idx.1
+  %a.1 = load i32, i32* %gep.a.1
+  %gep.b.1 = getelementptr inbounds i32, i32* %b.array, i32 %idx.1
+  %b.1 = load i32, i32* %gep.b.1
+  %qadd.1 = call i32 @llvm.arm.qadd(i32 %a.1, i32 %b.1)
+  %addr.1 = getelementptr inbounds i32, i32* %out.array, i32 %idx.1
+  store i32 %qadd.1, i32* %addr.1
+  %idx.2 = or i32 %idx.1, 1
+  %gep.a.2 = getelementptr inbounds i32, i32* %a.array, i32 %idx.2
+  %a.2 = load i32, i32* %gep.a.2
+  %gep.b.2 = getelementptr inbounds i32, i32* %b.array, i32 %idx.2
+  %b.2 = load i32, i32* %gep.b.2
+  %qadd.2 = call i32 @llvm.arm.qadd(i32 %a.2, i32 %b.2)
+  %addr.2 = getelementptr inbounds i32, i32* %out.array, i32 %idx.2
+  store i32 %qadd.2, i32* %addr.2
+  %i.next = add nsw nuw i32 %i, -2
+  %idx.next = add nsw nuw i32 %idx.1, 2
+  %cmp = icmp ult i32 %i.next, %N
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+; CHECK-LABEl: test_qadd_2_backwards
+
+; CHECK: [[shift:[rl0-9]+]], r3, #2
+; CHECK: add{{.*}} [[A:r[0-9]+]], r0, [[shift]], lsl #2
+; CHECK: add{{.*}} [[B:r[0-9]+]], r1, [[shift]], lsl #2
+; CHECK: add{{.*}} [[OUT:r[0-9]+]], r2, [[shift]], lsl #2
+
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #-8]!
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #-8]!
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #-8]!
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #-4]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #-4]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #-4]
+; CHECK: blo
+define void @test_qadd_2_backwards(i32* %a.array, i32* %b.array, i32* %out.array, i32 %N) {
+entry:
+  br label %loop
+
+loop:
+  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+  %idx.1 = phi i32 [ %N, %entry ], [ %idx.next, %loop ]
+  %gep.a.1 = getelementptr inbounds i32, i32* %a.array, i32 %idx.1
+  %a.1 = load i32, i32* %gep.a.1
+  %gep.b.1 = getelementptr inbounds i32, i32* %b.array, i32 %idx.1
+  %b.1 = load i32, i32* %gep.b.1
+  %qadd.1 = call i32 @llvm.arm.qadd(i32 %a.1, i32 %b.1)
+  %addr.1 = getelementptr inbounds i32, i32* %out.array, i32 %idx.1
+  store i32 %qadd.1, i32* %addr.1
+  %idx.2 = sub nsw nuw i32 %idx.1, 1
+  %gep.a.2 = getelementptr inbounds i32, i32* %a.array, i32 %idx.2
+  %a.2 = load i32, i32* %gep.a.2
+  %gep.b.2 = getelementptr inbounds i32, i32* %b.array, i32 %idx.2
+  %b.2 = load i32, i32* %gep.b.2
+  %qadd.2 = call i32 @llvm.arm.qadd(i32 %a.2, i32 %b.2)
+  %addr.2 = getelementptr inbounds i32, i32* %out.array, i32 %idx.2
+  store i32 %qadd.2, i32* %addr.2
+  %i.next = add nsw nuw i32 %i, -2
+  %idx.next = sub nsw nuw i32 %idx.1, 2
+  %cmp = icmp ult i32 %i.next, %N
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_qadd_3
+; CHECK: subs [[A:r[0-9]+]], #12
+; CHECK: subs [[B:r[0-9]+]], #12
+; CHECK: subs [[OUT:r[0-9]+]], #12
+
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #12]!
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #12]!
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #12]!
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #4]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #8]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #8]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #8]
+; CHECK: blo
+define void @test_qadd_3(i32* %a.array, i32* %b.array, i32* %out.array, i32 %N) {
+entry:
+  br label %loop
+
+loop:
+  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+  %idx.1 = phi i32 [ 0, %entry ], [ %idx.next, %loop ]
+  %gep.a.1 = getelementptr inbounds i32, i32* %a.array, i32 %idx.1
+  %a.1 = load i32, i32* %gep.a.1
+  %gep.b.1 = getelementptr inbounds i32, i32* %b.array, i32 %idx.1
+  %b.1 = load i32, i32* %gep.b.1
+  %qadd.1 = call i32 @llvm.arm.qadd(i32 %a.1, i32 %b.1)
+  %addr.1 = getelementptr inbounds i32, i32* %out.array, i32 %idx.1
+  store i32 %qadd.1, i32* %addr.1
+  %idx.2 = add nuw nsw i32 %idx.1, 1
+  %gep.a.2 = getelementptr inbounds i32, i32* %a.array, i32 %idx.2
+  %a.2 = load i32, i32* %gep.a.2
+  %gep.b.2 = getelementptr inbounds i32, i32* %b.array, i32 %idx.2
+  %b.2 = load i32, i32* %gep.b.2
+  %qadd.2 = call i32 @llvm.arm.qadd(i32 %a.2, i32 %b.2)
+  %addr.2 = getelementptr inbounds i32, i32* %out.array, i32 %idx.2
+  store i32 %qadd.2, i32* %addr.2
+  %idx.3 = add nuw nsw i32 %idx.1, 2
+  %gep.a.3 = getelementptr inbounds i32, i32* %a.array, i32 %idx.3
+  %a.3 = load i32, i32* %gep.a.3
+  %gep.b.3 = getelementptr inbounds i32, i32* %b.array, i32 %idx.3
+  %b.3 = load i32, i32* %gep.b.3
+  %qadd.3 = call i32 @llvm.arm.qadd(i32 %a.3, i32 %b.3)
+  %addr.3 = getelementptr inbounds i32, i32* %out.array, i32 %idx.3
+  store i32 %qadd.3, i32* %addr.3
+  %i.next = add nsw nuw i32 %i, -3
+  %idx.next = add nsw nuw i32 %idx.1, 3
+  %cmp = icmp ult i32 %i.next, %N
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_qadd_4
+; CHECK: subs [[A:r[0-9]+]], #16
+; CHECK: subs [[B:r[0-9]+]], #16
+; CHECK: subs [[OUT:r[0-9]+]], #16
+
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #16]!
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #16]!
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #16]!
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #4]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #8]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #8]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #8]
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #12]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #12]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #12]
+; CHECK: blo
+define void @test_qadd_4(i32* %a.array, i32* %b.array, i32* %out.array, i32 %N) {
+entry:
+  br label %loop
+
+loop:
+  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+  %idx.1 = phi i32 [ 0, %entry ], [ %idx.next, %loop ]
+  %gep.a.1 = getelementptr inbounds i32, i32* %a.array, i32 %idx.1
+  %a.1 = load i32, i32* %gep.a.1
+  %gep.b.1 = getelementptr inbounds i32, i32* %b.array, i32 %idx.1
+  %b.1 = load i32, i32* %gep.b.1
+  %qadd.1 = call i32 @llvm.arm.qadd(i32 %a.1, i32 %b.1)
+  %addr.1 = getelementptr inbounds i32, i32* %out.array, i32 %idx.1
+  store i32 %qadd.1, i32* %addr.1
+  %idx.2 = or i32 %idx.1, 1
+  %gep.a.2 = getelementptr inbounds i32, i32* %a.array, i32 %idx.2
+  %a.2 = load i32, i32* %gep.a.2
+  %gep.b.2 = getelementptr inbounds i32, i32* %b.array, i32 %idx.2
+  %b.2 = load i32, i32* %gep.b.2
+  %qadd.2 = call i32 @llvm.arm.qadd(i32 %a.2, i32 %b.2)
+  %addr.2 = getelementptr inbounds i32, i32* %out.array, i32 %idx.2
+  store i32 %qadd.2, i32* %addr.2
+  %idx.3 = or i32 %idx.1, 2
+  %gep.a.3 = getelementptr inbounds i32, i32* %a.array, i32 %idx.3
+  %a.3 = load i32, i32* %gep.a.3
+  %gep.b.3 = getelementptr inbounds i32, i32* %b.array, i32 %idx.3
+  %b.3 = load i32, i32* %gep.b.3
+  %qadd.3 = call i32 @llvm.arm.qadd(i32 %a.3, i32 %b.3)
+  %addr.3 = getelementptr inbounds i32, i32* %out.array, i32 %idx.3
+  store i32 %qadd.3, i32* %addr.3
+  %idx.4 = or i32 %idx.1, 3
+  %gep.a.4 = getelementptr inbounds i32, i32* %a.array, i32 %idx.4
+  %a.4 = load i32, i32* %gep.a.4
+  %gep.b.4 = getelementptr inbounds i32, i32* %b.array, i32 %idx.4
+  %b.4 = load i32, i32* %gep.b.4
+  %qadd.4 = call i32 @llvm.arm.qadd(i32 %a.4, i32 %b.4)
+  %addr.4 = getelementptr inbounds i32, i32* %out.array, i32 %idx.4
+  store i32 %qadd.4, i32* %addr.4
+  %i.next = add nsw nuw i32 %i, -4
+  %idx.next = add nsw nuw i32 %idx.1, 4
+  %cmp = icmp ult i32 %i.next, %N
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_qadd16_2
+; CHECK: sub.w [[A:r[0-9]+]], r0, #8
+; CHECK: subs [[B:r[0-9]+]], #8
+; CHECK: subs [[OUT:r[0-9]+]], #16
+
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #8]!
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #8]!
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #16]!
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #4]
+; CHECK: str{{.*}}, {{\[}}[[OUT]], #8]
+define void @test_qadd16_2(i16* %a.array, i16* %b.array, i32* %out.array, i32 %N) {
+entry:
+  br label %loop
+
+loop:
+  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+  %idx.1 = phi i32 [ 0, %entry ], [ %idx.next, %loop ]
+  %gep.a.1 = getelementptr inbounds i16, i16* %a.array, i32 %idx.1
+  %cast.a.1 = bitcast i16* %gep.a.1 to i32*
+  %a.1 = load i32, i32* %cast.a.1
+  %gep.b.1 = getelementptr inbounds i16, i16* %b.array, i32 %idx.1
+  %cast.b.1 = bitcast i16* %gep.b.1 to i32*
+  %b.1 = load i32, i32* %cast.b.1
+  %qadd.1 = call i32 @llvm.arm.qadd16(i32 %a.1, i32 %b.1)
+  %addr.1 = getelementptr inbounds i32, i32* %out.array, i32 %idx.1
+  store i32 %qadd.1, i32* %addr.1
+  %idx.2 = add nsw nuw i32 %idx.1, 2
+  %gep.a.2 = getelementptr inbounds i16, i16* %a.array, i32 %idx.2
+  %cast.a.2 = bitcast i16* %gep.a.2 to i32*
+  %a.2 = load i32, i32* %cast.a.2
+  %gep.b.2 = getelementptr inbounds i16, i16* %b.array, i32 %idx.2
+  %cast.b.2 = bitcast i16* %gep.b.2 to i32*
+  %b.2 = load i32, i32* %cast.b.2
+  %qadd.2 = call i32 @llvm.arm.qadd16(i32 %a.2, i32 %b.2)
+  %addr.2 = getelementptr inbounds i32, i32* %out.array, i32 %idx.2
+  store i32 %qadd.2, i32* %addr.2
+  %i.next = add nsw nuw i32 %i, -2
+  %idx.next = add nsw nuw i32 %idx.1, 4
+  %cmp = icmp ult i32 %i.next, %N
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret void
+}
+
+; TODO: I think we should be able to use post inc addressing with VLDM
+; instructions.
+; CHECK-LABEL: test_fma
+; CHECK: subs [[A:r[0-9]+]], #8
+; CHECK: subs [[B:r[0-9]+]], #8
+
+; CHECK: vldr s{{.*}}, {{\[}}[[B]], #8]
+; CHECK: vldr s{{.*}}, {{\[}}[[A]], #8]
+; CHECK: vldr s{{.*}}, {{\[}}[[B]], #12]
+; CHECK: vldr s{{.*}}, {{\[}}[[A]], #12]
+define float @test_fma(float* %a, float* %b, i32 %N) {
+entry:
+  br label %loop
+
+loop:
+  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
+  %idx.1 = phi i32 [ 0, %entry ], [ %idx.next, %loop ]
+  %res = phi float [ 0.0, %entry ], [ %fma.2, %loop ]
+  %gep.a.1 = getelementptr inbounds float, float* %a, i32 %idx.1
+  %a.1 = load float, float* %gep.a.1
+  %gep.b.1 = getelementptr inbounds float, float* %b, i32 %idx.1
+  %b.1 = load float, float* %gep.b.1
+  %fmul.1 = fmul float %a.1, %b.1
+  %fma.1 = fadd float %fmul.1, %res
+  %idx.2 = or i32 %idx.1, 1
+  %gep.a.2 = getelementptr inbounds float, float* %a, i32 %idx.2
+  %a.2 = load float, float* %gep.a.2
+  %gep.b.2 = getelementptr inbounds float, float* %b, i32 %idx.2
+  %b.2 = load float, float* %gep.b.2
+  %fmul.2 = fmul float %a.2, %b.2
+  %fma.2 = fadd float %fmul.2, %fma.1
+  %i.next = add nsw nuw i32 %i, -2
+  %idx.next = add nsw nuw i32 %idx.1, 2
+  %cmp = icmp ult i32 %i.next, %N
+  br i1 %cmp, label %loop, label %exit
+
+exit:
+  ret float %fma.2
+}
+
+; CHECK-LABEL: convolve_16bit
+
+; CHECK: ldr.w {{.*}}, [{{.*}}, lsl #2]
+; CHECK: ldr.w [[pA:r[rl0-9]+]], [{{.*}}, lsl #2]
+; CHECK: ldr.w [[pB:[rl0-9]+]], [{{.*}}, lsl #2]
+; CHECK: add{{.*}} [[A:[rl0-9]+]], [[pA]], {{.*}}, lsl #1
+; CHECK: sub{{.*}} [[B:[rl0-9]+]], [[pB]], #8
+
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #8]!
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #8]!
+; CHECK: ldr{{.*}}, {{\[}}[[B]], #4]
+; CHECK: ldr{{.*}}, {{\[}}[[A]], #4]
+define void @convolve_16bit(i16** nocapture readonly %input_image, i16** nocapture readonly %filter,
+                            i32 %filter_dim, i32 %out_width, i32 %out_height,
+                            i32** nocapture readonly %convolved) {
+entry:
+  %cmp92 = icmp eq i32 %out_height, 0
+  br i1 %cmp92, label %for.cond.cleanup, label %for.cond1.preheader.lr.ph
+
+for.cond1.preheader.lr.ph:                        ; preds = %entry
+  %xtraiter = and i32 %filter_dim, 3
+  %unroll_iter = sub i32 %filter_dim, %xtraiter
+  br label %for.cond1.preheader
+
+for.cond1.preheader:                              ; preds = %for.cond.cleanup3, %for.cond1.preheader.lr.ph
+  %res_y.093 = phi i32 [ 0, %for.cond1.preheader.lr.ph ], [ %add28, %for.cond.cleanup3 ]
+  %arrayidx22 = getelementptr inbounds i32*, i32** %convolved, i32 %res_y.093
+  %tmp3 = load i32*, i32** %arrayidx22, align 4
+  br label %for.cond9.preheader.us.us.preheader
+
+for.cond9.preheader.us.us.preheader:              ; preds = %for.cond5.for.cond.cleanup7_crit_edge.us, %for.cond5.preheader.lr.ph
+  %res_x.060.us = phi i32 [ %add25.us, %for.cond5.for.cond.cleanup7_crit_edge.us ], [ 0, %for.cond1.preheader ]
+  br label %for.cond9.preheader.us.us
+
+for.cond9.preheader.us.us:                        ; preds = %for.cond9.for.cond.cleanup11_crit_edge.us.us, %for.cond9.preheader.us.us.preheader
+  %filter_y.056.us.us = phi i32 [ %inc20.us.us, %for.cond9.for.cond.cleanup11_crit_edge.us.us.unr-lcssa ], [ 0, %for.cond9.preheader.us.us.preheader ]
+  %result_element.055.us.us = phi i32 [ %add18.us.us.3, %for.cond9.for.cond.cleanup11_crit_edge.us.us.unr-lcssa ], [ 0, %for.cond9.preheader.us.us.preheader ]
+  %add.us.us = add i32 %filter_y.056.us.us, %res_y.093
+  %arrayidx.us.us = getelementptr inbounds i16*, i16** %filter, i32 %filter_y.056.us.us
+  %tmp5 = load i16*, i16** %arrayidx.us.us, align 4
+  %arrayidx15.us.us = getelementptr inbounds i16*, i16** %input_image, i32 %add.us.us
+  %tmp6 = load i16*, i16** %arrayidx15.us.us, align 4
+  br label %for.body12.us.us
+
+for.body12.us.us:                                 ; preds = %for.body12.us.us, %for.cond9.preheader.us.us
+  %filter_x.053.us.us = phi i32 [ %inc.us.us.3, %for.body12.us.us ], [ 0, %for.cond9.preheader.us.us ]
+  %result_element.152.us.us = phi i32 [ %add18.us.us.3, %for.body12.us.us ], [ %result_element.055.us.us, %for.cond9.preheader.us.us ]
+  %niter = phi i32 [ %niter.nsub.3, %for.body12.us.us ], [ %unroll_iter, %for.cond9.preheader.us.us ]
+  %add13.us.us = add i32 %filter_x.053.us.us, %res_x.060.us
+  %arrayidx14.us.us = getelementptr inbounds i16, i16* %tmp5, i32 %filter_x.053.us.us
+  %tmp9 = load i16, i16* %arrayidx14.us.us, align 2
+  %conv.us.us = sext i16 %tmp9 to i32
+  %arrayidx16.us.us = getelementptr inbounds i16, i16* %tmp6, i32 %add13.us.us
+  %tmp10 = load i16, i16* %arrayidx16.us.us, align 2
+  %conv17.us.us = sext i16 %tmp10 to i32
+  %mul.us.us = mul nsw i32 %conv17.us.us, %conv.us.us
+  %add18.us.us = add nsw i32 %mul.us.us, %result_element.152.us.us
+  %inc.us.us = or i32 %filter_x.053.us.us, 1
+  %add13.us.us.1 = add i32 %inc.us.us, %res_x.060.us
+  %arrayidx14.us.us.1 = getelementptr inbounds i16, i16* %tmp5, i32 %inc.us.us
+  %tmp11 = load i16, i16* %arrayidx14.us.us.1, align 2
+  %conv.us.us.1 = sext i16 %tmp11 to i32
+  %arrayidx16.us.us.1 = getelementptr inbounds i16, i16* %tmp6, i32 %add13.us.us.1
+  %tmp12 = load i16, i16* %arrayidx16.us.us.1, align 2
+  %conv17.us.us.1 = sext i16 %tmp12 to i32
+  %mul.us.us.1 = mul nsw i32 %conv17.us.us.1, %conv.us.us.1
+  %add18.us.us.1 = add nsw i32 %mul.us.us.1, %add18.us.us
+  %inc.us.us.1 = or i32 %filter_x.053.us.us, 2
+  %add13.us.us.2 = add i32 %inc.us.us.1, %res_x.060.us
+  %arrayidx14.us.us.2 = getelementptr inbounds i16, i16* %tmp5, i32 %inc.us.us.1
+  %tmp13 = load i16, i16* %arrayidx14.us.us.2, align 2
+  %conv.us.us.2 = sext i16 %tmp13 to i32
+  %arrayidx16.us.us.2 = getelementptr inbounds i16, i16* %tmp6, i32 %add13.us.us.2
+  %tmp14 = load i16, i16* %arrayidx16.us.us.2, align 2
+  %conv17.us.us.2 = sext i16 %tmp14 to i32
+  %mul.us.us.2 = mul nsw i32 %conv17.us.us.2, %conv.us.us.2
+  %add18.us.us.2 = add nsw i32 %mul.us.us.2, %add18.us.us.1
+  %inc.us.us.2 = or i32 %filter_x.053.us.us, 3
+  %add13.us.us.3 = add i32 %inc.us.us.2, %res_x.060.us
+  %arrayidx14.us.us.3 = getelementptr inbounds i16, i16* %tmp5, i32 %inc.us.us.2
+  %tmp15 = load i16, i16* %arrayidx14.us.us.3, align 2
+  %conv.us.us.3 = sext i16 %tmp15 to i32
+  %arrayidx16.us.us.3 = getelementptr inbounds i16, i16* %tmp6, i32 %add13.us.us.3
+  %tmp16 = load i16, i16* %arrayidx16.us.us.3, align 2
+  %conv17.us.us.3 = sext i16 %tmp16 to i32
+  %mul.us.us.3 = mul nsw i32 %conv17.us.us.3, %conv.us.us.3
+  %add18.us.us.3 = add nsw i32 %mul.us.us.3, %add18.us.us.2
+  %inc.us.us.3 = add i32 %filter_x.053.us.us, 4
+  %niter.nsub.3 = add i32 %niter, -4
+  %niter.ncmp.3 = icmp eq i32 %niter.nsub.3, 0
+  br i1 %niter.ncmp.3, label %for.cond9.for.cond.cleanup11_crit_edge.us.us.unr-lcssa, label %for.body12.us.us
+
+for.cond9.for.cond.cleanup11_crit_edge.us.us.unr-lcssa: ; preds = %for.body12.us.us, %for.cond9.preheader.us.us
+  %inc20.us.us = add nuw i32 %filter_y.056.us.us, 1
+  %exitcond98 = icmp eq i32 %inc20.us.us, %filter_dim
+  br i1 %exitcond98, label %for.cond5.for.cond.cleanup7_crit_edge.us, label %for.cond9.preheader.us.us
+
+for.cond5.for.cond.cleanup7_crit_edge.us:         ; preds = %for.cond9.for.cond.cleanup11_crit_edge.us.us
+  %arrayidx23.us = getelementptr inbounds i32, i32* %tmp3, i32 %res_x.060.us
+  store i32 %add18.us.us.3, i32* %arrayidx23.us, align 4
+  %add25.us = add nuw i32 %res_x.060.us, 1
+  %exitcond99 = icmp eq i32 %add25.us, %out_width
+  br i1 %exitcond99, label %for.cond.cleanup3, label %for.cond9.preheader.us.us.preheader
+
+for.cond.cleanup3:                                ; preds = %for.cond5.for.cond.cleanup7_crit_edge.us, %for.cond5.preheader.preheader, %for.cond1.preheader
+  %add28 = add nuw i32 %res_y.093, 1
+  %exitcond100 = icmp eq i32 %add28, %out_height
+  br i1 %exitcond100, label %for.cond.cleanup, label %for.cond1.preheader
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup3, %entry
+  ret void
+}
+
+declare i32 @llvm.arm.qadd(i32, i32)
+declare i32 @llvm.arm.qadd16(i32, i32)
+

test/CodeGen/ARM/loop-align-cortex-m.ll

 ; RUN: llc -mtriple=thumbv7m-none-eabi %s -mcpu=cortex-m3 -o - | FileCheck %s
 ; RUN: llc -mtriple=thumbv7m-none-eabi %s -mcpu=cortex-m4 -o - | FileCheck %s
-; RUN: llc -mtriple=thumbv7m-none-eabi %s -mcpu=cortex-m33 -o - | FileCheck %s
+; RUN: llc -mtriple=thumbv8m-none-eabi %s -mcpu=cortex-m33 -o - | FileCheck %s
 
 define void @test_loop_alignment(i32* %in, i32*  %out) optsize {
 ; CHECK-LABEL: test_loop_alignment:
-; CHECK: movs {{r[0-9]+}}, #0
+; CHECK: mov.w {{r[0-9]+}}, #1024
 ; CHECK: .p2align 2
 
 entry:
   br label %loop
 
 loop:
   %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
   %in.addr = getelementptr inbounds i32, i32* %in, i32 %i
   %lhs = load i32, i32* %in.addr, align 4
   %res = mul nsw i32 %lhs, 5
   %out.addr = getelementptr inbounds i32, i32* %out, i32 %i
   store i32 %res, i32* %out.addr, align 4
   %i.next = add i32 %i, 1
   %done = icmp eq i32 %i.next, 1024
   br i1 %done, label %end, label %loop
 
 end:
   ret void
 }
 
 define void @test_loop_alignment_minsize(i32* %in, i32*  %out) minsize {
 ; CHECK-LABEL: test_loop_alignment_minsize:
-; CHECK: movs {{r[0-9]+}}, #0
+; CHECK: mov.w {{r[0-9]+}}, #1024
 ; CHECK-NOT: .p2align
 
 entry:
   br label %loop
 
 loop:
   %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
   %in.addr = getelementptr inbounds i32, i32* %in, i32 %i

test/CodeGen/ARM/t2-shrink-ldrpost.ll

 ; RUN: llc < %s | FileCheck %s
 
 target datalayout = "e-m:e-p:32:32-i1:8:32-i8:8:32-i16:16:32-f64:32:64-v64:32:64-v128:32:128-a:0:32-n32-S32"
 target triple = "thumbv7m--linux-gnu"
 
 ; CHECK-LABEL: f:
-; CHECK: ldm  r{{[0-9]}}!, {r[[x:[0-9]]]}
-; CHECK: add.w  r[[x]], r[[x]], #3
-; CHECK: stm  r{{[0-9]}}!, {r[[x]]}
+; CHECK: ldr{{.*}}, [{{.*}}, #4]!
+; CHECK: str{{.*}}, [{{.*}}, #4]!
 define void @f(i32 %n, i32* nocapture %a, i32* nocapture readonly %b) optsize minsize {
   %1 = icmp sgt i32 %n, 0
   br i1 %1, label %.lr.ph, label %._crit_edge
 
 .lr.ph:                                           ; preds = %.lr.ph, %0
   %i.04 = phi i32 [ %6, %.lr.ph ], [ 0, %0 ]
   %.03 = phi i32* [ %2, %.lr.ph ], [ %b, %0 ]
   %.012 = phi i32* [ %5, %.lr.ph ], [ %a, %0 ]
   %2 = getelementptr inbounds i32, i32* %.03, i32 1
   %3 = load i32, i32* %.03, align 4
   %4 = add nsw i32 %3, 3
   %5 = getelementptr inbounds i32, i32* %.012, i32 1
   store i32 %4, i32* %.012, align 4
   %6 = add nsw i32 %i.04, 1
   %exitcond = icmp eq i32 %6, %n
   br i1 %exitcond, label %._crit_edge, label %.lr.ph
 
 ._crit_edge:                                      ; preds = %.lr.ph, %0
   ret void
 }
 
 ; CHECK-LABEL: f_nominsize:
-; CHECK-NOT: ldm
+; CHECK: ldr{{.*}}, [{{.*}}, #4]!
+; CHECK: str{{.*}}, [{{.*}}, #4]!
 define void @f_nominsize(i32 %n, i32* nocapture %a, i32* nocapture readonly %b) optsize {
   %1 = icmp sgt i32 %n, 0
   br i1 %1, label %.lr.ph, label %._crit_edge
 
 .lr.ph:                                           ; preds = %.lr.ph, %0
   %i.04 = phi i32 [ %6, %.lr.ph ], [ 0, %0 ]
   %.03 = phi i32* [ %2, %.lr.ph ], [ %b, %0 ]
   %.012 = phi i32* [ %5, %.lr.ph ], [ %a, %0 ]

test/Transforms/LoopStrengthReduce/ARM/complexity.ll

 target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
 
-; RUN: opt -mtriple=thumbv7em %s -S -loop-reduce -lsr-complexity-limit=65536 -o - | FileCheck %s --check-prefix=CHECK-DEFAULT
-; RUN: opt -mtriple=thumbv7em %s -S -loop-reduce -lsr-complexity-limit=2147483647 -o - | FileCheck %s --check-prefix=CHECK-COMPLEX
+; RUN: opt -mtriple=thumbv7em %s -S -loop-reduce -lsr-complexity-limit=65536 -o - | FileCheck %s
+; RUN: opt -mtriple=thumbv7em %s -S -loop-reduce -lsr-complexity-limit=2147483647 -o - | FileCheck %s
+
+; CHECK-LABEL: for.cond9.preheader.us.us:
+; CHECK: [[SCEVGEP:%[^ ]+]] = getelementptr i16, i16* %tmp5, i32 -4 
+; CHECK: [[SCEVGEP9:%[^ ]+]] = getelementptr i16, i16* %tmp6, i32 %lsr.iv7
 
 ; CHECK-DEFAULT-LABEL: for.body12.us.us:
-; CHECK-DEFAULT: phi i32
-; CHECK-DEFAULT: [[LSR_IV:%[^ ]+]] = phi i32 [ [[LSR_IV_NEXT:%[^ ]+]], %for.body12.us.us ], [ 0, %for.cond9.preheader.us.us ]
-; CHECK-DEFAULT: phi i32
-; CHECK-DEFAULT: [[LSR_IV_NEXT]] = add i32 [[LSR_IV]], 8
-
-; CHECK-COMPLEX-LABEL: for.body12.us.us:
-; CHECK-COMPLEX: phi i32
-; CHECK-COMPLEX: [[LSR_IV6:%[^ ]+]] = phi i16* [ [[SCEVGEP7:%[^ ]+]], %for.body12.us.us ], [ [[SCEVGEP5:%[^ ]+]], %for.cond9.preheader.us.us ]
-; CHECK-COMPLEX: [[LSR_IV:%[^ ]+]] = phi i16* [ [[SCEVGEP1:%[^ ]+]], %for.body12.us.us ], [ [[SCEVGEP:%[^ ]+]], %for.cond9.preheader.us.us ]
-; CHECK-COMPLEX: phi i32
-; CHECK-COMPLEX: [[SCEVGEP1]] = getelementptr i16, i16* [[LSR_IV]], i32 4
-; CHECK-COMPLEX: [[SCEVGEP7]] = getelementptr i16, i16* [[LSR_IV6]], i32 4
+; CHECK: [[LSR_IV10:%[^ ]+]] = phi i16* [ [[SCEVGEP11:%[^ ]+]], %for.body12.us.us ], [ [[SCEVGEP9]], %for.cond9.preheader.us.us ]
+; CHECK: phi i32
+; CHECK: [[LSR_IV:%[^ ]+]] = phi i16* [ [[SCEVGEP1:%[^ ]+]], %for.body12.us.us ], [ [[SCEVGEP]], %for.cond9.preheader.us.us ]
+; CHECK: phi i32
+; CHECK: getelementptr i16, i16* [[LSR_IV]], i32 4
+; CHECK: getelementptr i16, i16* [[LSR_IV10]], i32 4
+; CHECK: getelementptr i16, i16* [[LSR_IV]], i32 5
+; CHECK: getelementptr i16, i16* [[LSR_IV10]], i32 5
+; CHECK: getelementptr i16, i16* [[LSR_IV]], i32 6
+; CHECK: getelementptr i16, i16* [[LSR_IV10]], i32 6
+; CHECK: getelementptr i16, i16* [[LSR_IV]], i32 7
+; CHECK: getelementptr i16, i16* [[LSR_IV10]], i32 7
+; CHECK: [[SCEVGEP1]] = getelementptr i16, i16* [[LSR_IV]], i32 4
+; CHECK: [[SCEVGEP11]] = getelementptr i16, i16* [[LSR_IV10]], i32 4
 
 define void @convolve(i16** nocapture readonly %input_image, i16** nocapture readonly %filter, i32 %filter_dim, i32 %out_width, i32 %out_height, i32** nocapture readonly %convolved) {
 entry:
   %cmp92 = icmp eq i32 %out_height, 0
   br i1 %cmp92, label %for.cond.cleanup, label %for.cond1.preheader.lr.ph
 
 for.cond1.preheader.lr.ph:                        ; preds = %entry
   %xtraiter = and i32 %filter_dim, 3