[ARM] Implement TTI::isHardwareLoopProfitable

[samparker]

Summary

Implement the backend target hook to drive the HardwareLoops pass. The low-overhead branch extension for Arm M-class cores is flexible enough that we don't have to ensure correctness at this point, except checking that the loop counter variable can be stored in LR - a 32-bit register. For it to be profitable, we want to avoid loops that contain function calls, or any other instruction that alters the PC.
This implementation uses TargetLoweringInfo, to query type and operation actions, looks at intrinsic calls and also performs some manual checks for remainder/division and FP operations.
I think this should be a good base to start and extra details can be filled out later.

[samparker]

Now implemented isLoweredToCall to look at intrinsics.

[dmgreen]

lib/Target/ARM/ARMTargetTransformInfo.cpp

646

There is a floating point sqrt instruction, at least for scalar fp. MVE does not have sqrt or div, but will expand into multiple scalar instructions.

688

I think most intrinsics like these will be expanded, so not end up as functions (even if they are not very efficient).

698

This is an extension to 8.1-m? Not just a base feature?

704

Will getBackedgeTakenCount ever be nullptr (as opposed to something like SCEVCouldNotCompute below)

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.cpp

646

cheers!

688

That would make sense, I'll double check.

698

Section A1.4 'Armv8-M' variants reads to me that the LOB extension is optional on 8.1-M. The main extension doesn't appear to include LOB, but MVE-I does. I've raised this with Simon for the instruction support too.

704

I'll check.

[samparker]

• Folded saturation into the overflow case.
• Allow scalar sqrt.

[dmgreen]

Looks like a good start for these loloops. I'm guessing we can always extend MaybeCall if we find more cases that end up as calls. (Along with the difficulties of tail predication!)

lib/Target/ARM/ARMTargetTransformInfo.cpp

656

I think that even a vector type should become scalar sqrts. Same for abs and friends. So they wouldn't remain as vector operations (which may effect tail prediction), but wouldn't become calls, unless the scalar version will be calls.

704

I also see the blurb for getBackedgeTakenCount talking about calling hasLoopInvariantBackedgeTakenCount, but many passes don't seem to check that. Maybe worth adding a check in place of !SE.getBackedgeTakenCount(L).

723

I would remove this line :)

775

Should this one be to return true?

[samparker]

lib/Target/ARM/ARMTargetTransformInfo.cpp

656

Okay, I'll query TLI here instead and check for Expand or LibCall.

704

I saw that getBackedgeTakenCount appears to always return some sort of value, but I had certainly missed the LoopInvariant restriction - which seems arbitrary and unnecessary to me! But I'll follow the good practise.

775

bah! yep, probably should add some double tests...

[samparker]

• Now checking for loop invariant trip count.
• Added tests for double and half precision floats.
• Now checking the new hasLOB target feature.

I didn't use TLI in isLoweredToCall because it requires an Instruction and it doesn't seem to give any meaningful results when an intrinsic call is passed either. Instead, I'm just checking the precision against the subtarget supported features.

[dmgreen]

LGTM. With one minor point

lib/Target/ARM/ARMTargetTransformInfo.cpp

706

Can we remove this !SE.getBackedgeTakenCount(L) check now?

[thakis]

All these tests seem to fail when the ARM target isn't enabled, with errors like this:

********************
FAIL: LLVM :: Transforms/HardwareLoops/ARM/counter.ll (25696 of 31777)
******************** TEST 'LLVM :: Transforms/HardwareLoops/ARM/counter.ll' FAILED ********************
Script:
--
: 'RUN: at line 1';   /Users/thakis/src/llvm-project/out/gn/bin/opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false /Users/thakis/src/llvm-project/llvm/test/Transforms/HardwareLoops/ARM/counter.ll -o - | /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/llvm/test/Transforms/HardwareLoops/ARM/counter.ll
--
Exit Code: 2

Command Output (stderr):
--
opt: Unknown command line argument '-disable-arm-loloops=false'.  Try: '/Users/thakis/src/llvm-project/out/gn/bin/opt --help'
opt: Did you mean '  --disable-x86-lea-opt=false'?
FileCheck error: '-' is empty.
FileCheck command line:  /Users/thakis/src/llvm-project/out/gn/bin/FileCheck /Users/thakis/src/llvm-project/llvm/test/Transforms/HardwareLoops/ARM/counter.ll

--

********************
Testing Time: 266.30s
********************
Failing Tests (6):
    LLVM :: Transforms/HardwareLoops/ARM/calls.ll
    LLVM :: Transforms/HardwareLoops/ARM/counter.ll
    LLVM :: Transforms/HardwareLoops/ARM/do-rem.ll
    LLVM :: Transforms/HardwareLoops/ARM/fp-emulation.ll
    LLVM :: Transforms/HardwareLoops/ARM/simple-do.ll
    LLVM :: Transforms/HardwareLoops/ARM/structure.ll

They probably all need a REQUIRES: arm-registered-target?

[dmgreen]

Sam: you can do the same thing as test/Transforms/LoopVectorize/ARM/lit.local.cfg

[samparker]

Thanks both. I went for the REQUIRES option, though slightly different, and I'll also add that config file.

[thakis]

Thanks! Having both is a bit belt-and-suspenders. https://reviews.llvm.org/rL363157 obsoletes https://reviews.llvm.org/rL363156 , so I think you could revert https://reviews.llvm.org/rL363156 again.

lib/Target/ARM/ARMTargetTransformInfo.h

int getMemoryOpCost(unsigned Opcode, Type *Src, unsigned Alignment,
                       unsigned AddressSpace, const Instruction *I = nullptr);
 
   int getInterleavedMemoryOpCost(unsigned Opcode, Type *VecTy, unsigned Factor,
                                  ArrayRef<unsigned> Indices, unsigned Alignment,
                                  unsigned AddressSpace,
                                  bool UseMaskForCond = false,
                                  bool UseMaskForGaps = false);
 
+  bool isLoweredToCall(const Function *F);
+  bool isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,
+                                AssumptionCache &AC,
+                                TargetLibraryInfo *LibInfo,
+                                TTI::HardwareLoopInfo &HWLoopInfo);
+
   void getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                TTI::UnrollingPreferences &UP);
 
   bool shouldBuildLookupTablesForConstant(Constant *C) const {
     // In the ROPI and RWPI relocation models we can't have pointers to global
     // variables or functions in constant data, so don't convert switches to
     // lookup tables if any of the values would need relocation.
     if (ST->isROPI() || ST->isRWPI())

lib/Target/ARM/ARMTargetTransformInfo.cpp

 #include <cassert>
 #include <cstdint>
 #include <utility>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "armtti"
 
+static cl::opt<bool> DisableLowOverheadLoops(
+  "disable-arm-loloops", cl::Hidden, cl::init(true),
+  cl::desc("Disable the generation of low-overhead loops"));
+
 bool ARMTTIImpl::areInlineCompatible(const Function *Caller,
                                      const Function *Callee) const {
   const TargetMachine &TM = getTLI()->getTargetMachine();
   const FeatureBitset &CallerBits =
       TM.getSubtargetImpl(*Caller)->getFeatureBits();
   const FeatureBitset &CalleeBits =
       TM.getSubtargetImpl(*Callee)->getFeatureBits();
 
if (NumElts % Factor == 0 &&
       return Factor * TLI->getNumInterleavedAccesses(SubVecTy, DL);
   }
 
   return BaseT::getInterleavedMemoryOpCost(Opcode, VecTy, Factor, Indices,
                                            Alignment, AddressSpace,
                                            UseMaskForCond, UseMaskForGaps);
 }
 
+bool ARMTTIImpl::isLoweredToCall(const Function *F) {
+  if (!F->isIntrinsic())
+    BaseT::isLoweredToCall(F);
+
+  // Assume all Arm-specific intrinsics map to an instruction.
+  if (F->getName().startswith("llvm.arm"))
+    return false;
+
+  switch (F->getIntrinsicID()) {
+  default: break;
+  case Intrinsic::powi:
+  case Intrinsic::sin:
+  case Intrinsic::cos:
+  case Intrinsic::pow:
+  case Intrinsic::log:
+  case Intrinsic::log10:
+  case Intrinsic::log2:
+  case Intrinsic::exp:
+  case Intrinsic::exp2:
+    return true;
+  case Intrinsic::sqrt:
+    if (F->getReturnType()->isVectorTy())
+      return true;
+    return !ST->hasFPARMv8Base() && !ST->hasVFP2Base();
+  case Intrinsic::fabs:
+  case Intrinsic::copysign:
+  case Intrinsic::floor:
+  case Intrinsic::ceil:
+  case Intrinsic::trunc:
+  case Intrinsic::rint:
+  case Intrinsic::nearbyint:
+  case Intrinsic::round:
+  case Intrinsic::canonicalize:
+  case Intrinsic::lround:
+  case Intrinsic::llround:
+  case Intrinsic::lrint:
+  case Intrinsic::llrint:
+    if (F->getReturnType()->isVectorTy() && !ST->hasNEON() && !ST->hasMVEFloatOps())
+      return true;
+    // TODO Handle scalar operations properly.
+    return !ST->hasFPARMv8Base() && !ST->hasVFP2Base();
+  case Intrinsic::masked_store:
+  case Intrinsic::masked_load:
+  case Intrinsic::masked_gather:
+  case Intrinsic::masked_scatter:
+    return !ST->hasMVEIntegerOps();
+  case Intrinsic::sadd_with_overflow:
+  case Intrinsic::uadd_with_overflow:
+  case Intrinsic::ssub_with_overflow:
+  case Intrinsic::usub_with_overflow:
+  case Intrinsic::sadd_sat:
+  case Intrinsic::uadd_sat:
+  case Intrinsic::ssub_sat:
+  case Intrinsic::usub_sat:
+    return false;
+  }
+
+  return BaseT::isLoweredToCall(F);
+}
+
+bool ARMTTIImpl::isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,
+                                          AssumptionCache &AC,
+                                          TargetLibraryInfo *LibInfo,
+                                          TTI::HardwareLoopInfo &HWLoopInfo) {
+  // FIXME: Low-overhead branches are only supported in the 'low-overhead branch'
+  // extension of v8.1-m.
+  if (!ST->hasV8_1MMainlineOps() || DisableLowOverheadLoops)
+    return false;
+
+  // For now, for simplicity, only support loops with one exit block.
+  if (!L->getExitBlock() || !SE.getBackedgeTakenCount(L))
+    return false;
+
+  const SCEV *BackedgeTakenCount = SE.getBackedgeTakenCount(L);
+  if (isa<SCEVCouldNotCompute>(BackedgeTakenCount))
+    return false;
+
+  const SCEV *TripCountSCEV =
+    SE.getAddExpr(BackedgeTakenCount,
+                  SE.getOne(BackedgeTakenCount->getType()));
+
+  // We need to store the trip count in LR, a 32-bit register.
+  if (SE.getUnsignedRangeMax(TripCountSCEV).getBitWidth() > 32)
+    return false;
+
+  // Making a call will trash LR and clear LO_BRANCH_INFO, so there's little
+  // point in generating a hardware loop if that's going to happen.
+  auto MaybeCall = [this](Instruction &I) {
+
+    // Check if an intrinsic will be lowered to a call and assume that any
+    // other CallInst will generate a bl.
+    if (auto *Call = dyn_cast<CallInst>(&I)) {
+      if (isa<IntrinsicInst>(Call)) {
+        if (const Function *F = Call->getCalledFunction())
+          return isLoweredToCall(F);
+      }
+      return true;
+    }
+
+    // Will an instruction get lowered to a lib call?
+    const ARMTargetLowering *TLI = getTLI();
+    unsigned ISD = TLI->InstructionOpcodeToISD(I.getOpcode());
+    EVT VT = TLI->getValueType(DL, I.getType(), true);
+    if (TLI->getOperationAction(ISD, VT) == TargetLowering::LibCall)
+      return true;
+
+    // FIXME: Unfortunately the approach of checking the Operation Action does
+    // not catch all cases of Legalization that use library calls. Our
+    // Legalization step categorizes some transformations into library calls as
+    // Custom, Expand or even Legal when doing type legalization. So for now
+    // we have to special case for instance the SDIV of 64bit integers and the
+    // use of floating point emulation.
+    if (VT.isInteger() && VT.getSizeInBits() >= 64) {
+      switch (ISD) {
+      default:
+        break;
+      case ISD::SDIV:
+      case ISD::UDIV:
+      case ISD::SREM:
+      case ISD::UREM:
+      case ISD::SDIVREM:
+      case ISD::UDIVREM:
+        return true;
+      }
+    }
+
+    // If the instruction is not of a floating point type or a conversion from
+    // a floating point type, then it is safe to assume no FP emulation will be
+    // used.
+    Type *Ty = (I.getOpcode() == Instruction::FPToSI ||
+                I.getOpcode() == Instruction::FPToUI) ?
+      I.getOperand(0)->getType() : I.getType();
+
+    // Assume any other integer operations are okay.
+    if (!Ty->isHalfTy() && !Ty->isFloatTy() && !Ty->isDoubleTy())
+      return false;
+
+    // We'll need a libcall to perform double precision operations on a single
+    // precision only FPU.
+    if (Ty->isDoubleTy() && !ST->hasFP64())
+      return false;
+
+    // We'll need a library call to handle most floats when using soft.
+    if (TLI->useSoftFloat()) {
+      switch (I.getOpcode()) {
+      default:
+        return true;
+      case Instruction::Alloca:
+      case Instruction::Load:
+      case Instruction::Store:
+      case Instruction::Select:
+      case Instruction::PHI:
+        break;
+      }
+    }
+    return false;
+  };
+
+  // Scan the instructions to see if there's any that we know will turn into a
+  // call.
+  for (auto *BB : L->getBlocks())
+    for (auto &I : *BB)
+      if (MaybeCall(I))
+        return false;
+
+  // TODO: Check whether the trip count calculation is expensive. If L is the
+  // inner loop but we know it has a low trip count, calculating that trip
+  // count (in the parent loop) may be detrimental.
+
+  LLVMContext &C = L->getHeader()->getContext();
+  HWLoopInfo.CounterInReg = true;
+  HWLoopInfo.CountType = Type::getInt32Ty(C);
+  HWLoopInfo.LoopDecrement = ConstantInt::get(HWLoopInfo.CountType, 1);
+  return true;
+}
+
 void ARMTTIImpl::getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                          TTI::UnrollingPreferences &UP) {
   // Only currently enable these preferences for M-Class cores.
   if (!ST->isMClass())
     return BasicTTIImplBase::getUnrollingPreferences(L, SE, UP);
 
   // Disable loop unrolling for Oz and Os.
   UP.OptSizeThreshold = 0;

test/Transforms/HardwareLoops/ARM/calls.ll

+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MAIN
+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+fp-armv8 -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-FP
+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVE
+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MVEFP
+
+
+; CHECK-LABEL: skip_call
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-NOT: call i32 @llvm.loop.decrement
+
+define i32 @skip_call(i32 %n) {
+entry:
+  %cmp6 = icmp eq i32 %n, 0
+  br i1 %cmp6, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.08 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.07 = phi i32 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %call = tail call i32 bitcast (i32 (...)* @bar to i32 ()*)() #2
+  %add = add nsw i32 %call, %res.07
+  %inc1 = add nuw i32 %i.08, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: test_target_specific
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 50)
+; CHECK: [[COUNT:%[^ ]+]] = phi i32 [ 50, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %loop, label %exit
+
+define i32 @test_target_specific(i32* %a, i32* %b) {
+entry:
+  br label %loop
+loop:
+  %acc = phi i32 [ 0, %entry ], [ %res, %loop ]
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr i32, i32* %a, i32 %count
+  %addr.b = getelementptr i32, i32* %b, i32 %count
+  %load.a = load i32, i32* %addr.a
+  %load.b = load i32, i32* %addr.b
+  %res = call i32 @llvm.arm.smlad(i32 %load.a, i32 %load.b, i32 %acc)
+  %count.next = add nuw i32 %count, 2
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret i32 %res
+}
+
+; CHECK-LABEL: test_fabs
+; CHECK-MAIN-NOT: call void @llvm.set.loop.iterations 
+; CHECK-MVE-NOT: call void @llvm.set.loop.iterations
+; CHECK-FP: call void @llvm.set.loop.iterations
+; CHECK-MVEFP: call void @llvm.set.loop.iterations.i32(i32 100)
+; CHECK-MVEFP: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK-MVEFP: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK-MVEFP: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-MVEFP: br i1 [[CMP]], label %loop, label %exit
+define float @test_fabs(float* %a) {
+entry:
+  br label %loop
+loop:
+  %acc = phi float [ 0.0, %entry ], [ %res, %loop ]
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr float, float* %a, i32 %count
+  %load.a = load float, float* %addr.a
+  %abs = call float @llvm.fabs.f32(float %load.a)
+  %res = fadd float %abs, %acc
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret float %res
+}
+
+; CHECK-LABEL: test_log
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-NOT: llvm.loop.decrement
+define float @test_log(float* %a) {
+entry:
+  br label %loop
+loop:
+  %acc = phi float [ 0.0, %entry ], [ %res, %loop ]
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr float, float* %a, i32 %count
+  %load.a = load float, float* %addr.a
+  %abs = call float @llvm.log.f32(float %load.a)
+  %res = fadd float %abs, %acc
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret float %res
+}
+
+; CHECK-LABEL: test_fabs_vec
+; CHECK-MVE-NOT: call void @llvm.set.loop.iterations
+; CHECK-MVEFP: call void @llvm.set.loop.iterations.i32(i32 100)
+; CHECK-MVEFP: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK-MVEFP: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK-MVEFP: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-MVEFP: br i1 [[CMP]], label %loop, label %exit
+define <4 x float> @test_fabs_vec(<4 x float>* %a) {
+entry:
+  br label %loop
+loop:
+  %acc = phi <4 x float> [ zeroinitializer, %entry ], [ %res, %loop ]
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr <4 x float>, <4 x float>* %a, i32 %count
+  %load.a = load <4 x float>, <4 x float>* %addr.a
+  %abs = call <4 x float> @llvm.fabs.v4f32(<4 x float> %load.a)
+  %res = fadd <4 x float> %abs, %acc
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret <4 x float> %res
+}
+
+; CHECK-LABEL: test_sqrt
+; CHECK-MAIN-NOT: call void @llvm.set.loop.iterations 
+; CHECK-MVE-NOT: call void @llvm.set.loop.iterations
+; CHECK-FP: call void @llvm.set.loop.iterations
+; CHECK-MVEFP: call void @llvm.set.loop.iterations.i32(i32 100)
+; CHECK-MVEFP: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK-MVEFP: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK-MVEFP: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-MVEFP: br i1 [[CMP]], label %loop, label %exit
+define void @test_sqrt(float* %a, float* %b) {
+entry:
+  br label %loop
+loop:
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr float, float* %a, i32 %count
+  %load.a = load float, float* %addr.a
+  %sqrt = call float @llvm.sqrt.f32(float %load.a)
+  %addr.b = getelementptr float, float* %b, i32 %count
+  store float %sqrt, float* %addr.b
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_sqrt_vec
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-NOT: call void @llvm.loop.decrement
+define void @test_sqrt_vec(<4 x float>* %a, <4 x float>* %b) {
+entry:
+  br label %loop
+loop:
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr <4 x float>, <4 x float>* %a, i32 %count
+  %load.a = load <4 x float>, <4 x float>* %addr.a
+  %sqrt = call <4 x float> @llvm.sqrt.v4f32(<4 x float> %load.a)
+  %addr.b = getelementptr <4 x float>, <4 x float>* %b, i32 %count
+  store <4 x float> %sqrt, <4 x float>* %addr.b
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_overflow
+; CHECK: call void @llvm.set.loop.iterations
+define i32 @test_overflow(i32* %a, i32* %b) {
+entry:
+  br label %loop
+loop:
+  %acc = phi i32 [ 0, %entry ], [ %res, %loop ]
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr i32, i32* %a, i32 %count
+  %addr.b = getelementptr i32, i32* %b, i32 %count
+  %load.a = load i32, i32* %addr.a
+  %load.b = load i32, i32* %addr.b
+  %sadd = call {i32, i1} @llvm.sadd.with.overflow.i32(i32 %load.a, i32 %load.b)
+  %res = extractvalue {i32, i1} %sadd, 0
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret i32 %res
+}
+
+; TODO: We should be able to generate a qadd/sub
+; CHECK-LABEL: test_sat
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 100)
+define i32 @test_sat(i32* %a, i32* %b) {
+entry:
+  br label %loop
+loop:
+  %acc = phi i32 [ 0, %entry ], [ %res, %loop ]
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr i32, i32* %a, i32 %count
+  %addr.b = getelementptr i32, i32* %b, i32 %count
+  %load.a = load i32, i32* %addr.a
+  %load.b = load i32, i32* %addr.b
+  %res = call i32 @llvm.sadd.sat.i32(i32 %load.a, i32 %load.b)
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret i32 %res
+}
+
+; CHECK-LABEL: test_masked_i32
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-MVEFP: call void @llvm.set.loop.iterations
+; CHECK-MVE: call void @llvm.set.loop.iterations.i32(i32 100)
+; CHECK-MVE: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK-MVE: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK-MVE: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-MVE: br i1 [[CMP]], label %loop, label %exit
+define void @test_masked_i32(<4 x i1> %mask, <4 x i32>* %a, <4 x i32>* %b, <4 x i32>* %c, <4 x i32> %passthru) {
+entry:
+  br label %loop
+loop:
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr <4 x i32>, <4 x i32>* %a, i32 %count
+  %addr.b = getelementptr <4 x i32>, <4 x i32>* %b, i32 %count
+  %addr.c = getelementptr <4 x i32>, <4 x i32>* %c, i32 %count
+  %load.a = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %addr.a, i32 4, <4 x i1> %mask, <4 x i32> %passthru)
+  %load.b = call <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>* %addr.b, i32 4, <4 x i1> %mask, <4 x i32> %passthru)
+  %res = add <4 x i32> %load.a, %load.b
+  call void @llvm.masked.store.v4i32.p0v4i32(<4 x i32> %res, <4 x i32>* %addr.c, i32 4, <4 x i1> %mask)
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_masked_f32
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-MVEFP: call void @llvm.set.loop.iterations
+; CHECK-MVE: call void @llvm.set.loop.iterations.i32(i32 100)
+; CHECK-MVE: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK-MVE: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK-MVE: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-MVE: br i1 [[CMP]], label %loop, label %exit
+define void @test_masked_f32(<4 x i1> %mask, <4 x float>* %a, <4 x float>* %b, <4 x float>* %c, <4 x float> %passthru) {
+entry:
+  br label %loop
+loop:
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %addr.a = getelementptr <4 x float>, <4 x float>* %a, i32 %count
+  %addr.b = getelementptr <4 x float>, <4 x float>* %b, i32 %count
+  %addr.c = getelementptr <4 x float>, <4 x float>* %c, i32 %count
+  %load.a = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %addr.a, i32 4, <4 x i1> %mask, <4 x float> %passthru)
+  %load.b = call <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>* %addr.b, i32 4, <4 x i1> %mask, <4 x float> %passthru)
+  %res = fadd <4 x float> %load.a, %load.b
+  call void @llvm.masked.store.v4f32.p0v4f32(<4 x float> %res, <4 x float>* %addr.c, i32 4, <4 x i1> %mask)
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret void
+}
+
+; CHECK-LABEL: test_gather_scatter
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-MVEFP: call void @llvm.set.loop.iterations
+; CHECK-MVE: call void @llvm.set.loop.iterations.i32(i32 100)
+; CHECK-MVE: [[COUNT:%[^ ]+]] = phi i32 [ 100, %entry ], [ [[LOOP_DEC:%[^ ]+]], %loop ]
+; CHECK-MVE: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[COUNT]], i32 1)
+; CHECK-MVE: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-MVE: br i1 [[CMP]], label %loop, label %exit
+define void @test_gather_scatter(<4 x i1> %mask, <4 x float*> %a, <4 x float*> %b, <4 x float*> %c, <4 x float> %passthru) {
+entry:
+  br label %loop
+loop:
+  %count = phi i32 [ 0, %entry ], [ %count.next, %loop ]
+  %load.a = call <4 x float> @llvm.masked.gather.v4f32.p0v4f32(<4 x float*> %a, i32 4, <4 x i1> %mask, <4 x float> %passthru)
+  %load.b = call <4 x float> @llvm.masked.gather.v4f32.p0v4f32(<4 x float*> %b, i32 4, <4 x i1> %mask, <4 x float> %passthru)
+  %res = fadd <4 x float> %load.a, %load.b
+  call void @llvm.masked.scatter.v4f32.p0v4f32(<4 x float> %res, <4 x float*> %c, i32 4, <4 x i1> %mask)
+  %count.next = add nuw i32 %count, 1
+  %cmp = icmp ne i32 %count.next, 100
+  br i1 %cmp, label %loop, label %exit
+exit:
+  ret void
+}
+
+declare i32 @bar(...) local_unnamed_addr #1
+declare i32 @llvm.arm.smlad(i32, i32, i32)
+declare float @llvm.fabs.f32(float)
+declare float @llvm.log.f32(float)
+declare <4 x float> @llvm.fabs.v4f32(<4 x float>)
+declare float @llvm.sqrt.f32(float)
+declare <4 x float> @llvm.sqrt.v4f32(<4 x float>)
+declare i32 @llvm.sadd.sat.i32(i32, i32)
+declare {i32, i1} @llvm.sadd.with.overflow.i32(i32, i32)
+declare <4 x i32> @llvm.masked.load.v4i32.p0v4i32(<4 x i32>*, i32, <4 x i1>, <4 x i32>)
+declare void @llvm.masked.store.v4i32.p0v4i32(<4 x i32>, <4 x i32>*, i32, <4 x i1>)
+declare <4 x float> @llvm.masked.load.v4f32.p0v4f32(<4 x float>*, i32, <4 x i1>, <4 x float>)
+declare void @llvm.masked.store.v4f32.p0v4f32(<4 x float>, <4 x float>*, i32, <4 x i1>)
+declare <4 x float> @llvm.masked.gather.v4f32.p0v4f32(<4 x float*>, i32, <4 x i1>, <4 x float>)
+declare void @llvm.masked.scatter.v4f32.p0v4f32(<4 x float>, <4 x float*>, i32, <4 x i1>)

test/Transforms/HardwareLoops/ARM/counter.ll

+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -o - | FileCheck %s
+
+@g = common local_unnamed_addr global i32* null, align 4
+
+; CHECK-LABEL: counter_too_large
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-NOT: call i32 @llvm.loop.decrement
+
+define i32 @counter_too_large(i64 %n) {
+entry:
+  %cmp7 = icmp eq i64 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.lr.ph
+
+while.body.lr.ph:
+  %0 = load i32*, i32** @g, align 4
+  br label %while.body
+
+while.body:
+  %i.09 = phi i64 [ 0, %while.body.lr.ph ], [ %inc1, %while.body ]
+  %res.08 = phi i32 [ 0, %while.body.lr.ph ], [ %add, %while.body ]
+  %idxprom = trunc i64 %i.09 to i32
+  %arrayidx = getelementptr inbounds i32, i32* %0, i32 %idxprom
+  %1 = load i32, i32* %arrayidx, align 4
+  %add = add nsw i32 %1, %res.08
+  %inc1 = add nuw i64 %i.09, 1
+  %cmp = icmp ult i64 %inc1, %n
+  br i1 %cmp, label %while.body, label %while.end.loopexit
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}

test/Transforms/HardwareLoops/ARM/do-rem.ll

+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s
+
+@g = common local_unnamed_addr global i32* null, align 4
+
+; CHECK-LABEL: do_with_i32_urem
+; CHECK: while.body.preheader:
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+
+define i32 @do_with_i32_urem(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i32 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %rem = urem i32 %i.09, 5
+  %add = add i32 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i32_srem
+; CHECK: while.body.preheader:
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+
+define i32 @do_with_i32_srem(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i32 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %rem = srem i32 %i.09, 5
+  %add = sub i32 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i32_udiv
+; CHECK: while.body.preheader:
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+
+define i32 @do_with_i32_udiv(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i32 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %rem = udiv i32 %i.09, 5
+  %add = add i32 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i32_sdiv
+; CHECK: while.body.preheader:
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+
+define i32 @do_with_i32_sdiv(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i32 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %rem = sdiv i32 %i.09, 5
+  %add = sub i32 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i64_urem
+; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.loop.decrement
+define i64 @do_with_i64_urem(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i64 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %conv = zext i32 %i.09 to i64
+  %rem = urem i64 %conv, 5
+  %add = add i64 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i64 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i64_srem
+; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.loop.decrement
+define i64 @do_with_i64_srem(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i64 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %conv = zext i32 %i.09 to i64
+  %rem = srem i64 %conv, 5
+  %add = sub i64 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i64 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i64_udiv
+; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.loop.decrement
+define i64 @do_with_i64_udiv(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i64 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %conv = zext i32 %i.09 to i64
+  %rem = udiv i64 %conv, 5
+  %add = add i64 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i64 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_with_i64_sdiv
+; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-NOT: call i32 @llvm.loop.decrement
+define i64 @do_with_i64_sdiv(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.preheader
+
+while.body.preheader:
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ %inc1, %while.body ], [ 0, %while.body.preheader ]
+  %res.08 = phi i64 [ %add, %while.body ], [ 0, %while.body.preheader ]
+  %conv = zext i32 %i.09 to i64
+  %rem = sdiv i64 %conv, 5
+  %add = sub i64 %rem, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i64 %res.0.lcssa
+}

test/Transforms/HardwareLoops/ARM/fp-emulation.ll

+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-HARD
+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+soft-float -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-SOFT
+
+; CHECK-LABEL: test_fptosi
+; CHECK: while.body.lr.ph:
+; CHECK: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
+; CHECK: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
+
+define void @test_fptosi(i32 %n, i32** %g, double** %d) {
+entry:
+  %n.off = add i32 %n, -1
+  %0 = icmp ult i32 %n.off, 500
+  br i1 %0, label %while.body.lr.ph, label %cleanup
+
+while.body.lr.ph:
+  %1 = load double*, double** %d, align 4
+  %2 = load i32*, i32** %g, align 4
+  br label %while.body
+
+while.body:
+  %i.012 = phi i32 [ 0, %while.body.lr.ph ], [ %inc, %if.end4 ]
+  %rem = urem i32 %i.012, 10
+  %tobool = icmp eq i32 %rem, 0
+  br i1 %tobool, label %if.end4, label %if.then2
+
+if.then2:
+  %arrayidx = getelementptr inbounds double, double* %1, i32 %i.012
+  %3 = load double, double* %arrayidx, align 8
+  %conv = fptosi double %3 to i32
+  %arrayidx3 = getelementptr inbounds i32, i32* %2, i32 %i.012
+  store i32 %conv, i32* %arrayidx3, align 4
+  br label %if.end4
+
+if.end4:
+  %inc = add nuw i32 %i.012, 1
+  %cmp1 = icmp ult i32 %inc, %n
+  br i1 %cmp1, label %while.body, label %cleanup.loopexit
+
+cleanup.loopexit:
+  br label %cleanup
+
+cleanup:
+  ret void
+}
+
+; CHECK-LABEL: test_fptoui
+; CHECK: while.body.lr.ph:
+; CHECK: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
+; CHECK: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
+
+define void @test_fptoui(i32 %n, i32** %g, double** %d) {
+entry:
+  %n.off = add i32 %n, -1
+  %0 = icmp ult i32 %n.off, 500
+  br i1 %0, label %while.body.lr.ph, label %cleanup
+
+while.body.lr.ph:
+  %1 = load double*, double** %d, align 4
+  %2 = load i32*, i32** %g, align 4
+  br label %while.body
+
+while.body:
+  %i.012 = phi i32 [ 0, %while.body.lr.ph ], [ %inc, %if.end4 ]
+  %rem = urem i32 %i.012, 10
+  %tobool = icmp eq i32 %rem, 0
+  br i1 %tobool, label %if.end4, label %if.then2
+
+if.then2:
+  %arrayidx = getelementptr inbounds double, double* %1, i32 %i.012
+  %3 = load double, double* %arrayidx, align 8
+  %conv = fptoui double %3 to i32
+  %arrayidx3 = getelementptr inbounds i32, i32* %2, i32 %i.012
+  store i32 %conv, i32* %arrayidx3, align 4
+  br label %if.end4
+
+if.end4:
+  %inc = add nuw i32 %i.012, 1
+  %cmp1 = icmp ult i32 %inc, %n
+  br i1 %cmp1, label %while.body, label %cleanup.loopexit
+
+cleanup.loopexit:
+  br label %cleanup
+
+cleanup:
+  ret void
+}
+
+; CHECK-LABEL: load_store_float
+; CHECK: while.body.lr.ph:
+; CHECK: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
+; CHECK: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
+
+define void @load_store_float(i32 %n, double** %d, double** %g) {
+entry:
+  %n.off = add i32 %n, -1
+  %0 = icmp ult i32 %n.off, 500
+  br i1 %0, label %while.body.lr.ph, label %cleanup
+
+while.body.lr.ph:
+  %1 = load double*, double** %d, align 4
+  %2 = load double*, double** %g, align 4
+  br label %while.body
+
+while.body:
+  %i.012 = phi i32 [ 0, %while.body.lr.ph ], [ %inc, %if.end4 ]
+  %rem = urem i32 %i.012, 10
+  %tobool = icmp eq i32 %rem, 0
+  br i1 %tobool, label %if.end4, label %if.then2
+
+if.then2:
+  %arrayidx = getelementptr inbounds double, double* %1, i32 %i.012
+  %3 = load double, double* %arrayidx, align 8
+  %arrayidx3 = getelementptr inbounds double, double* %2, i32 %i.012
+  store double %3, double* %arrayidx3, align 8
+  br label %if.end4
+
+if.end4:
+  %inc = add nuw i32 %i.012, 1
+  %cmp1 = icmp ult i32 %inc, %n
+  br i1 %cmp1, label %while.body, label %cleanup.loopexit
+
+cleanup.loopexit:
+  br label %cleanup
+
+cleanup:
+  ret void
+}
+
+; CHECK-LABEL: fp_add
+; CHECK: while.body.lr.ph:
+
+; CHECK-SOFT-NOT: call void @llvm.set.loop.iterations
+
+; CHECK-HARD: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
+; CHECK-HARD: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
+; CHECK-HARD: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK: br label %while.body
+
+; CHECK-SOFT-NOT: call i32 @llvm.loop.decrement
+
+; CHECK-HARD: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
+; CHECK-HARD: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK-HARD: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK-HARD: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
+
+define void @fp_add(i32 %n, float** %d, float** %g) {
+entry:
+  %n.off = add i32 %n, -1
+  %0 = icmp ult i32 %n.off, 500
+  br i1 %0, label %while.body.lr.ph, label %cleanup
+
+while.body.lr.ph:
+  %1 = load float*, float** %d, align 4
+  %2 = load float*, float** %g, align 4
+  br label %while.body
+
+while.body:
+  %i.012 = phi i32 [ 0, %while.body.lr.ph ], [ %inc, %if.end4 ]
+  %rem = urem i32 %i.012, 10
+  %tobool = icmp eq i32 %rem, 0
+  br i1 %tobool, label %if.end4, label %if.then2
+
+if.then2:
+  %arrayidx = getelementptr inbounds float, float* %1, i32 %i.012
+  %3 = load float, float* %arrayidx, align 4
+  %arrayidx3 = getelementptr inbounds float, float* %2, i32 %i.012
+  %4 = load float, float* %arrayidx3, align 4
+  %add = fadd float %3, %4
+  store float %add, float* %arrayidx3, align 4
+  br label %if.end4
+
+if.end4:
+  %inc = add nuw i32 %i.012, 1
+  %cmp1 = icmp ult i32 %inc, %n
+  br i1 %cmp1, label %while.body, label %cleanup.loopexit
+
+cleanup.loopexit:
+  br label %cleanup
+
+cleanup:
+  ret void
+}

test/Transforms/HardwareLoops/ARM/simple-do.ll

+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s
+
+; RUN: opt -mtriple=thumbv8m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=DISABLED
+
+; DISABLED-NOT: llvm.set.loop.iterations
+; DISABLED-NOT: llvm.loop.decrement
+
+@g = common local_unnamed_addr global i32* null, align 4
+
+; CHECK-LABEL: do_copy
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
+; CHECK: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %entry ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end
+define i32 @do_copy(i32 %n, i32* nocapture %p, i32* nocapture readonly %q) {
+entry:
+  br label %while.body
+
+while.body:
+  %q.addr.05 = phi i32* [ %incdec.ptr, %while.body ], [ %q, %entry ]
+  %p.addr.04 = phi i32* [ %incdec.ptr1, %while.body ], [ %p, %entry ]
+  %x.addr.03 = phi i32 [ %dec, %while.body ], [ %n, %entry ]
+  %dec = add nsw i32 %x.addr.03, -1
+  %incdec.ptr = getelementptr inbounds i32, i32* %q.addr.05, i32 1
+  %0 = load i32, i32* %q.addr.05, align 4
+  %incdec.ptr1 = getelementptr inbounds i32, i32* %p.addr.04, i32 1
+  store i32 %0, i32* %p.addr.04, align 4
+  %tobool = icmp eq i32 %dec, 0
+  br i1 %tobool, label %while.end, label %while.body
+
+while.end:
+  ret i32 0
+}
+
+; CHECK-LABEL: do_inc1
+; CHECK: while.body.lr.ph:
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+
+define i32 @do_inc1(i32 %n) {
+entry:
+  %cmp7 = icmp eq i32 %n, 0
+  br i1 %cmp7, label %while.end, label %while.body.lr.ph
+
+while.body.lr.ph:
+  %0 = load i32*, i32** @g, align 4
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ 0, %while.body.lr.ph ], [ %inc1, %while.body ]
+  %res.08 = phi i32 [ 0, %while.body.lr.ph ], [ %add, %while.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %0, i32 %i.09
+  %1 = load i32, i32* %arrayidx, align 4
+  %add = add nsw i32 %1, %res.08
+  %inc1 = add nuw i32 %i.09, 1
+  %exitcond = icmp eq i32 %inc1, %n
+  br i1 %exitcond, label %while.end.loopexit, label %while.body
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_inc2
+; CHECK: while.body.lr.ph:
+; CHECK: [[ROUND:%[^ ]+]] = add i32 %n, -1
+; CHECK: [[HALVE:%[^ ]+]] = lshr i32 [[ROUND]], 1
+; CHECK: [[COUNT:%[^ ]+]] = add i32 [[HALVE]], 1
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+define i32 @do_inc2(i32 %n) {
+entry:
+  %cmp7 = icmp sgt i32 %n, 0
+  br i1 %cmp7, label %while.body.lr.ph, label %while.end
+
+while.body.lr.ph:
+  %0 = load i32*, i32** @g, align 4
+  br label %while.body
+
+while.body:
+  %i.09 = phi i32 [ 0, %while.body.lr.ph ], [ %add1, %while.body ]
+  %res.08 = phi i32 [ 0, %while.body.lr.ph ], [ %add, %while.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %0, i32 %i.09
+  %1 = load i32, i32* %arrayidx, align 4
+  %add = add nsw i32 %1, %res.08
+  %add1 = add nuw nsw i32 %i.09, 2
+  %cmp = icmp slt i32 %add1, %n
+  br i1 %cmp, label %while.body, label %while.end.loopexit
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}
+
+; CHECK-LABEL: do_dec2
+
+; CHECK: while.body.lr.ph:
+; CHECK: [[ROUND:%[^ ]+]] = add i32 %n, 1
+; CHECK: [[CMP:%[^ ]+]] = icmp slt i32 %n, 2
+; CHECK: [[SMIN:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 2
+; CHECK: [[SUB:%[^ ]+]] = sub i32 [[ROUND]], [[SMIN]]
+; CHECK: [[HALVE:%[^ ]+]] = lshr i32 [[SUB]], 1
+; CHECK: [[COUNT:%[^ ]+]] = add i32 [[HALVE]], 1
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-NEXT: br label %while.body
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
+define i32 @do_dec2(i32 %n) {
+entry:
+  %cmp6 = icmp sgt i32 %n, 0
+  br i1 %cmp6, label %while.body.lr.ph, label %while.end
+
+while.body.lr.ph:
+  %0 = load i32*, i32** @g, align 4
+  br label %while.body
+
+while.body:
+  %i.08 = phi i32 [ %n, %while.body.lr.ph ], [ %sub, %while.body ]
+  %res.07 = phi i32 [ 0, %while.body.lr.ph ], [ %add, %while.body ]
+  %arrayidx = getelementptr inbounds i32, i32* %0, i32 %i.08
+  %1 = load i32, i32* %arrayidx, align 4
+  %add = add nsw i32 %1, %res.07
+  %sub = add nsw i32 %i.08, -2
+  %cmp = icmp sgt i32 %i.08, 2
+  br i1 %cmp, label %while.body, label %while.end.loopexit
+
+while.end.loopexit:
+  br label %while.end
+
+while.end:
+  %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
+  ret i32 %res.0.lcssa
+}

test/Transforms/HardwareLoops/ARM/structure.ll

+; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s
+
+; CHECK-LABEL: early_exit
+; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.loop.decrement
+define i32 @early_exit(i32* nocapture readonly %a, i32 %max, i32 %n) {
+entry:
+  br label %do.body
+
+do.body:
+  %i.0 = phi i32 [ 0, %entry ], [ %inc, %if.end ]
+  %arrayidx = getelementptr inbounds i32, i32* %a, i32 %i.0
+  %0 = load i32, i32* %arrayidx, align 4
+  %cmp = icmp sgt i32 %0, %max
+  br i1 %cmp, label %do.end, label %if.end
+
+if.end:
+  %inc = add nuw i32 %i.0, 1
+  %cmp1 = icmp ult i32 %inc, %n
+  br i1 %cmp1, label %do.body, label %if.end.do.end_crit_edge
+
+if.end.do.end_crit_edge:
+  %arrayidx2.phi.trans.insert = getelementptr inbounds i32, i32* %a, i32 %inc
+  %.pre = load i32, i32* %arrayidx2.phi.trans.insert, align 4
+  br label %do.end
+
+do.end:
+  %1 = phi i32 [ %.pre, %if.end.do.end_crit_edge ], [ %0, %do.body ]
+  ret i32 %1
+}
+
+; CHECK-LABEL: nested
+; CHECK-NOT: call void @llvm.set.loop.iterations.i32(i32 %N)
+; CHECK: br i1 %cmp20, label %while.end7, label %while.cond1.preheader.us
+
+; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
+; CHECK: br label %while.body3.us
+
+; CHECK: [[REM:%[^ ]+]] = phi i32 [ %N, %while.cond1.preheader.us ], [ [[LOOP_DEC:%[^ ]+]], %while.body3.us ]
+; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
+; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK: br i1 [[CMP]], label %while.body3.us, label %while.cond1.while.end_crit_edge.us
+
+; CHECK-NOT: [[LOOP_DEC1:%[^ ]+]] = call i1 @llvm.loop.decrement.i32(i32 1)
+; CHECK-NOT: br i1 [[LOOP_DEC1]], label %while.cond1.preheader.us, label %while.end7
+define void @nested(i32* nocapture %A, i32 %N) {
+entry:
+  %cmp20 = icmp eq i32 %N, 0
+  br i1 %cmp20, label %while.end7, label %while.cond1.preheader.us
+
+while.cond1.preheader.us:
+  %i.021.us = phi i32 [ %inc6.us, %while.cond1.while.end_crit_edge.us ], [ 0, %entry ]
+  %mul.us = mul i32 %i.021.us, %N
+  br label %while.body3.us
+
+while.body3.us:
+  %j.019.us = phi i32 [ 0, %while.cond1.preheader.us ], [ %inc.us, %while.body3.us ]
+  %add.us = add i32 %j.019.us, %mul.us
+  %arrayidx.us = getelementptr inbounds i32, i32* %A, i32 %add.us
+  store i32 %add.us, i32* %arrayidx.us, align 4
+  %inc.us = add nuw i32 %j.019.us, 1
+  %exitcond = icmp eq i32 %inc.us, %N
+  br i1 %exitcond, label %while.cond1.while.end_crit_edge.us, label %while.body3.us
+
+while.cond1.while.end_crit_edge.us:
+  %inc6.us = add nuw i32 %i.021.us, 1
+  %exitcond23 = icmp eq i32 %inc6.us, %N
+  br i1 %exitcond23, label %while.end7, label %while.cond1.preheader.us
+
+while.end7:
+  ret void
+}