[ARM] WLS/LE Code Generation

[samparker]

Summary

Backend changes to enable WLS/LE low-overhead loops for armv8.1-m:

1. Use TTI to communicate to the HardwareLoop pass that we should try to generate intrinsics that guard the loop entry, as well as setting the loop trip count.
2. Lower the BRCOND that uses said intrinsic to an Arm specific node: ARMWLS.
3. ISelDAGToDAG the node to a new pseudo instruction: t2WhileLoopStart.
4. Add support in ArmLowOverheadLoops to handle the new pseudo instruction.

[SjoerdMeijer]

lib/Target/ARM/ARMISelLowering.cpp

12736

Initially I was a bit confused by this, the xor in particular. It makes sense though, but you're not checking it here in the patterns?

[SjoerdMeijer]

lib/Target/ARM/ARMLowOverheadLoops.cpp

144

As soon as we found one, can we stop? Or, can we continue looking as a sanity check to see if we find more and bail with a diagnostic?

183

Who reviewed this? ;-)
These 2 if statements here, can it just be this:

if (!(Start && Dec && End))

[dmgreen]

lib/Target/ARM/ARMLowOverheadLoops.cpp

320

These is something called ARMBaseInstrInfo::analyzeBranch which can help optimise away these unconditional fallthroughs too, if it can be made to work with loloop instructions.

[samparker]

lib/Target/ARM/ARMLowOverheadLoops.cpp

144

We carry on searching just in case there's multiple predecessors with multiple loop starts. I'll add a TODO here because I think having multiple LoopStarts is fine and we need to support it, but currently this pass is not setup for it.

320

I'll add a TODO

[samparker]

• Added some TODOs and FIXMEs.
• Checking for xor/setcc in the backend while lowering the intrinsic.
• Added some more tests.

[SjoerdMeijer]

This looks fine as an initial commit to me. I.e., this can be followed up, addressing the TODOs, to produce even more efficient code and handle more cases.

lib/Target/ARM/ARMInstrInfo.td

103

nit:

// start supporting *TP versions

I think for clarity I would spell this out as the tail-predicated versions

test/CodeGen/Thumb2/LowOverheadLoops/loop-guards.ll

4 ↗

(On Diff #207050)

nit:

".. that changes the generic HardwareLoop .."

->

".. that changes in the generic HardwareLoop.. "?

[dmgreen]

lib/Target/ARM/ARMLowOverheadLoops.cpp

320

Sounds good. Cheers. I'm not sure if those functions will be a little awkward to use with hardware loops, but they may allow some extra BB layout if we teach them how to analyse the branches.

[RKSimon]

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

12911 ↗

(On Diff #207239)

@samparker This is problematic on several levels... - we have an unused variable warning in NDEBUG builds and the else case only occurs if Const == NULL

if (isa<ConstantSDNode>(CC->getOperand(1)))
  assert(cast<ConstantSDNode>(CC->getOperand(1)).isOne() && "Expected to compare against 1");

[samparker]

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

12911 ↗

(On Diff #207239)

Ah - good point! I'll push up a fix. Thanks.

[samparker]

@RKSimon Committed in rL365046

lib/Target/ARM/ARMISelDAGToDAG.cpp

     } else if (Subtarget->isThumb()) {
       if (tryT1IndexedLoad(N))
         return;
     } else if (tryARMIndexedLoad(N))
       return;
     // Other cases are autogenerated.
     break;
   }
+  case ARMISD::WLS: {
+    SDValue Ops[] = { N->getOperand(1),   // Loop count
+                      N->getOperand(2),   // Exit target
+                      N->getOperand(0) };
+    SDNode *LoopStart =
+      CurDAG->getMachineNode(ARM::t2WhileLoopStart, dl, MVT::Other, Ops);
+    ReplaceUses(N, LoopStart);
+    CurDAG->RemoveDeadNode(N);
+    return;
+  }
   case ARMISD::BRCOND: {
     // Pattern: (ARMbrcond:void (bb:Other):$dst, (imm:i32):$cc)
     // Emits: (Bcc:void (bb:Other):$dst, (imm:i32):$cc)
     // Pattern complexity = 6  cost = 1  size = 0
 
     // Pattern: (ARMbrcond:void (bb:Other):$dst, (imm:i32):$cc)
     // Emits: (tBcc:void (bb:Other):$dst, (imm:i32):$cc)
     // Pattern complexity = 6  cost = 1  size = 0

lib/Target/ARM/ARMISelLowering.h

enum NodeType : unsigned {
 
       MEMBARRIER_MCR, // Memory barrier (MCR)
 
       PRELOAD,      // Preload
 
       WIN__CHKSTK,  // Windows' __chkstk call to do stack probing.
       WIN__DBZCHK,  // Windows' divide by zero check
 
+      WLS,          // Low-overhead loops, While Loop Start
+
       VCEQ,         // Vector compare equal.
       VCEQZ,        // Vector compare equal to zero.
       VCGE,         // Vector compare greater than or equal.
       VCGEZ,        // Vector compare greater than or equal to zero.
       VCLEZ,        // Vector compare less than or equal to zero.
       VCGEU,        // Vector compare unsigned greater than or equal.
       VCGT,         // Vector compare greater than.
       VCGTZ,        // Vector compare greater than zero.

lib/Target/ARM/ARMISelLowering.cpp

   setOperationAction(ISD::ConstantFP, MVT::f64, Custom);
 
   setOperationAction(ISD::READ_REGISTER, MVT::i64, Custom);
   setOperationAction(ISD::WRITE_REGISTER, MVT::i64, Custom);
 
   if (Subtarget->hasMVEIntegerOps())
     addMVEVectorTypes();
 
+  // Combine low-overhead loop intrinsics so that we can lower i1 types.
+  if (Subtarget->hasLOB())
+    setTargetDAGCombine(ISD::BRCOND);
+
   if (Subtarget->hasNEON()) {
     addDRTypeForNEON(MVT::v2f32);
     addDRTypeForNEON(MVT::v8i8);
     addDRTypeForNEON(MVT::v4i16);
     addDRTypeForNEON(MVT::v2i32);
     addDRTypeForNEON(MVT::v1i64);
 
     addQRTypeForNEON(MVT::v4f32);

   case ARMISD::VLD4DUP_UPD:   return "ARMISD::VLD4DUP_UPD";
   case ARMISD::VST1_UPD:      return "ARMISD::VST1_UPD";
   case ARMISD::VST2_UPD:      return "ARMISD::VST2_UPD";
   case ARMISD::VST3_UPD:      return "ARMISD::VST3_UPD";
   case ARMISD::VST4_UPD:      return "ARMISD::VST4_UPD";
   case ARMISD::VST2LN_UPD:    return "ARMISD::VST2LN_UPD";
   case ARMISD::VST3LN_UPD:    return "ARMISD::VST3LN_UPD";
   case ARMISD::VST4LN_UPD:    return "ARMISD::VST4LN_UPD";
+  case ARMISD::WLS:           return "ARMISD::WLS";
   }
   return nullptr;
 }
 
 EVT ARMTargetLowering::getSetCCResultType(const DataLayout &DL, LLVMContext &,
                                           EVT VT) const {
   if (!VT.isVector())
     return getPointerTy(DL);

     V = DAG.getNode(ARMISD::BFI, dl, VT, V, X,
                     // Confusingly, the operand is an *inverted* mask.
                     DAG.getConstant(~Mask, dl, VT));
   }
 
   return V;
 }
 
+static SDValue PerformHWLoopCombine(SDNode *N,
+                                    TargetLowering::DAGCombinerInfo &DCI,
+                                    const ARMSubtarget *ST) {
+  // Look for (brcond (xor test.set.loop.iterations, -1)
+  SDValue CC = N->getOperand(1);
+  if (CC->getOperand(0)->getOpcode() != ISD::INTRINSIC_W_CHAIN)
+    return SDValue();
+
+  SDValue Int = CC->getOperand(0);
+  unsigned IntOp = cast<ConstantSDNode>(Int.getOperand(1))->getZExtValue();
+  if (IntOp != Intrinsic::test_set_loop_iterations)
+    return SDValue();
+
+  SDLoc dl(Int);
+  SDValue Chain = N->getOperand(0);
+  SDValue Elements = Int.getOperand(2);
+  SDValue ExitBlock = N->getOperand(2);
+
+  // TODO: Once we start supporting tail predication, we can add another
+  // operand to WLS for the number of elements processed in a vector loop.
+
+  SDValue Ops[] = { Chain, Elements, ExitBlock };
+  SDValue Res = DCI.DAG.getNode(ARMISD::WLS, dl, MVT::Other, Ops);
+  DCI.DAG.ReplaceAllUsesOfValueWith(Int.getValue(1), Int.getOperand(0));
+  return Res;
+}
+
 /// PerformBRCONDCombine - Target-specific DAG combining for ARMISD::BRCOND.
 SDValue
 ARMTargetLowering::PerformBRCONDCombine(SDNode *N, SelectionDAG &DAG) const {
   SDValue Cmp = N->getOperand(4);
   if (Cmp.getOpcode() != ARMISD::CMPZ)
     // Only looking at NE cases.
     return SDValue();
 

   case ARMISD::ADDE:    return PerformADDECombine(N, DCI, Subtarget);
   case ARMISD::UMLAL:   return PerformUMLALCombine(N, DCI.DAG, Subtarget);
   case ISD::ADD:        return PerformADDCombine(N, DCI, Subtarget);
   case ISD::SUB:        return PerformSUBCombine(N, DCI);
   case ISD::MUL:        return PerformMULCombine(N, DCI, Subtarget);
   case ISD::OR:         return PerformORCombine(N, DCI, Subtarget);
   case ISD::XOR:        return PerformXORCombine(N, DCI, Subtarget);
   case ISD::AND:        return PerformANDCombine(N, DCI, Subtarget);
+  case ISD::BRCOND:     return PerformHWLoopCombine(N, DCI, Subtarget);
   case ARMISD::ADDC:
   case ARMISD::SUBC:    return PerformAddcSubcCombine(N, DCI, Subtarget);
   case ARMISD::SUBE:    return PerformAddeSubeCombine(N, DCI, Subtarget);
   case ARMISD::BFI:     return PerformBFICombine(N, DCI);
   case ARMISD::VMOVRRD: return PerformVMOVRRDCombine(N, DCI, Subtarget);
   case ARMISD::VMOVDRR: return PerformVMOVDRRCombine(N, DCI.DAG);
   case ISD::STORE:      return PerformSTORECombine(N, DCI);
   case ISD::BUILD_VECTOR: return PerformBUILD_VECTORCombine(N, DCI, Subtarget);

lib/Target/ARM/ARMInstrInfo.td

 
 def SDT_LongMac  : SDTypeProfile<2, 4, [SDTCisVT<0, i32>,
                                         SDTCisSameAs<0, 1>,
                                         SDTCisSameAs<0, 2>,
                                         SDTCisSameAs<0, 3>,
                                         SDTCisSameAs<0, 4>,
                                         SDTCisSameAs<0, 5>]>;
 
+// TODO Add another operand for 'Size' so that we can re-use this node when we
+// start supporting *TP versions.
+def SDT_ARMWhileLoop : SDTypeProfile<0, 2, [SDTCisVT<0, i32>,
+                                            SDTCisVT<1, OtherVT>]>;
+
 def ARMSmlald        : SDNode<"ARMISD::SMLALD", SDT_LongMac>;
 def ARMSmlaldx       : SDNode<"ARMISD::SMLALDX", SDT_LongMac>;
 def ARMSmlsld        : SDNode<"ARMISD::SMLSLD", SDT_LongMac>;
 def ARMSmlsldx       : SDNode<"ARMISD::SMLSLDX", SDT_LongMac>;
 
 def SDT_MulHSR       : SDTypeProfile<1, 3, [SDTCisVT<0,i32>,
                                             SDTCisSameAs<0, 1>,
                                             SDTCisSameAs<0, 2>,

 
 def ARMsmulwb       : SDNode<"ARMISD::SMULWB", SDTIntBinOp, []>;
 def ARMsmulwt       : SDNode<"ARMISD::SMULWT", SDTIntBinOp, []>;
 def ARMsmlalbb      : SDNode<"ARMISD::SMLALBB", SDT_LongMac, []>;
 def ARMsmlalbt      : SDNode<"ARMISD::SMLALBT", SDT_LongMac, []>;
 def ARMsmlaltb      : SDNode<"ARMISD::SMLALTB", SDT_LongMac, []>;
 def ARMsmlaltt      : SDNode<"ARMISD::SMLALTT", SDT_LongMac, []>;
 
+def ARMWLS : SDNode<"ARMISD::WLS", SDT_ARMWhileLoop,
+                    [SDNPHasChain]>;
+
 //===----------------------------------------------------------------------===//
 // ARM Flag Definitions.
 
 class RegConstraint<string C> {
   string Constraints = C;
 }
 
 //===----------------------------------------------------------------------===//

lib/Target/ARM/ARMInstrThumb2.td

 def t2DoLoopStart :
   t2PseudoInst<(outs), (ins rGPR:$elts), 4, IIC_Br,
   [(int_set_loop_iterations rGPR:$elts)]>, Sched<[WriteBr]>;
 
 def t2LoopDec :
   t2PseudoInst<(outs GPRlr:$Rm), (ins GPRlr:$Rn, imm0_7:$size),
                4, IIC_Br, []>, Sched<[WriteBr]>;
 
-let isBranch = 1, isTerminator = 1, hasSideEffects = 1 in
+let isBranch = 1, isTerminator = 1, hasSideEffects = 1 in {
+def t2WhileLoopStart :
+    t2PseudoInst<(outs),
+                 (ins rGPR:$elts, brtarget:$target),
+                 4, IIC_Br, []>,
+                 Sched<[WriteBr]>;
+
 def t2LoopEnd :
   t2PseudoInst<(outs), (ins GPRlr:$elts, brtarget:$target),
   8, IIC_Br, []>, Sched<[WriteBr]>;
 
+} // end isBranch, isTerminator, hasSideEffects
+
 } // end isNotDuplicable
 
 class CS<string iname, bits<4> opcode, list<dag> pattern=[]>
   : V8_1MI<(outs rGPR:$Rd), (ins GPRwithZR:$Rn, GPRwithZR:$Rm, pred_noal:$fcond),
            AddrModeNone, NoItinerary, iname, "$Rd, $Rn, $Rm, $fcond", "", pattern> {
   bits<4> Rd;
   bits<4> Rm;
   bits<4> Rn;

lib/Target/ARM/ARMLowOverheadLoops.cpp

bool ARMLowOverheadLoops::ProcessLoop(MachineLoop *ML) {
 
   // Process inner loops first.
   for (auto I = ML->begin(), E = ML->end(); I != E; ++I)
     Changed |= ProcessLoop(*I);
 
   LLVM_DEBUG(dbgs() << "ARM Loops: Processing " << *ML);
 
   auto IsLoopStart = [](MachineInstr &MI) {
-    return MI.getOpcode() == ARM::t2DoLoopStart;
+    return MI.getOpcode() == ARM::t2DoLoopStart ||
+           MI.getOpcode() == ARM::t2WhileLoopStart;
   };
 
-  auto SearchForStart =
-    [&IsLoopStart](MachineBasicBlock *MBB) -> MachineInstr* {
+  // Search the given block for a loop start instruction. If one isn't found,
+  // and there's only one predecessor block, search that one too.
+  std::function<MachineInstr*(MachineBasicBlock*)> SearchForStart =
+    [&IsLoopStart, &SearchForStart](MachineBasicBlock *MBB) -> MachineInstr* {
     for (auto &MI : *MBB) {
       if (IsLoopStart(MI))
         return &MI;
     }
+    if (MBB->pred_size() == 1)
+      return SearchForStart(*MBB->pred_begin());
     return nullptr;
   };
 
   MachineInstr *Start = nullptr;
   MachineInstr *Dec = nullptr;
   MachineInstr *End = nullptr;
   bool Revert = false;
 
-  if (auto *Preheader = ML->getLoopPreheader())
+  if (auto *Preheader = ML->getLoopPreheader()) {
     Start = SearchForStart(Preheader);
+  } else {
+    LLVM_DEBUG(dbgs() << "ARM Loops: Failed to find loop preheader!\n"
+               << " - Performing manual predecessor search.\n");
+    MachineBasicBlock *Pred = nullptr;
+    for (auto *MBB : ML->getHeader()->predecessors()) {
+      if (!ML->contains(MBB)) {
+        if (Pred) {
+          LLVM_DEBUG(dbgs() << " - Found multiple out-of-loop preds.\n");
+          Start = nullptr;
+          break;
+        }
+        Pred = MBB;
+        Start = SearchForStart(MBB);
+      }
+    }
+  }
 
   // Find the low-overhead loop components and decide whether or not to fall
   // back to a normal loop.
   for (auto *MBB : reverse(ML->getBlocks())) {
     for (auto &MI : *MBB) {
       if (MI.getOpcode() == ARM::t2LoopDec)
         Dec = &MI;
       else if (MI.getOpcode() == ARM::t2LoopEnd)
for (auto &MI : *MBB) {
           MI.getOperand(0).isReg() && MI.getOperand(0).getReg() == ARM::LR;
     }
 
     if (Dec && End && Revert)
       break;
   }
 
   if (Start || Dec || End) {
     if (!Start || !Dec || !End)
       report_fatal_error("Failed to find all loop components");
   } else {
     LLVM_DEBUG(dbgs() << "ARM Loops: Not a low-overhead loop.\n");
     return Changed;
   }
 
   if (!End->getOperand(1).isMBB() ||
       End->getOperand(1).getMBB() != ML->getHeader())
for (auto &I : MRI->def_instructions(ARM::LR)) {
       // before the setup instruction anyway.
       if (!I.getDesc().isMoveReg() ||
           !I.getOperand(1).isIdenticalTo(Start->getOperand(0)))
         continue;
       InsertPt = &I;
       break;
     }
 
+    unsigned Opc = Start->getOpcode() == ARM::t2DoLoopStart ?
+      ARM::t2DLS : ARM::t2WLS;
     MachineInstrBuilder MIB =
-      BuildMI(*MBB, InsertPt, InsertPt->getDebugLoc(), TII->get(ARM::t2DLS));
-    if (InsertPt != Start)
-      InsertPt->eraseFromParent();
+      BuildMI(*MBB, InsertPt, InsertPt->getDebugLoc(), TII->get(Opc));
 
     MIB.addDef(ARM::LR);
     MIB.add(Start->getOperand(0));
-    LLVM_DEBUG(dbgs() << "ARM Loops: Inserted DLS: " << *MIB);
+    if (Opc == ARM::t2WLS)
+      MIB.add(Start->getOperand(1));
+
+    if (InsertPt != Start)
+      InsertPt->eraseFromParent();
     Start->eraseFromParent();
+    LLVM_DEBUG(dbgs() << "ARM Loops: Inserted start: " << *MIB);
+    return &*MIB;
   };
 
   // Combine the LoopDec and LoopEnd instructions into LE(TP).
   auto ExpandLoopEnd = [this](MachineLoop *ML, MachineInstr *Dec,
                               MachineInstr *End) {
     MachineBasicBlock *MBB = End->getParent();
     MachineInstrBuilder MIB = BuildMI(*MBB, End, End->getDebugLoc(),
                                       TII->get(ARM::t2LEUpdate));
     MIB.addDef(ARM::LR);
     MIB.add(End->getOperand(0));
     MIB.add(End->getOperand(1));
     LLVM_DEBUG(dbgs() << "ARM Loops: Inserted LE: " << *MIB);
 
-    // If there is a branch after loop end, which branches to the fallthrough
-    // block, remove the branch.
-    MachineBasicBlock *Latch = End->getParent();
-    MachineInstr *Terminator = &Latch->instr_back();
-    if (End != Terminator) {
-      MachineBasicBlock *Exit = ML->getExitBlock();
-      if (Latch->isLayoutSuccessor(Exit)) {
-        LLVM_DEBUG(dbgs() << "ARM Loops: Removing loop exit branch: "
-                   << *Terminator);
-        Terminator->eraseFromParent();
-      }
-    }
     End->eraseFromParent();
     Dec->eraseFromParent();
+    return &*MIB;
   };
 
   // Generate a subs, or sub and cmp, and a branch instead of an LE.
   // TODO: Check flags so that we can possibly generate a subs.
   auto ExpandBranch = [this](MachineInstr *Dec, MachineInstr *End) {
     LLVM_DEBUG(dbgs() << "ARM Loops: Reverting to sub, cmp, br.\n");
     // Create sub
     MachineBasicBlock *MBB = Dec->getParent();
auto ExpandBranch = [this](MachineInstr *Dec, MachineInstr *End) {
     MIB = BuildMI(*MBB, End, End->getDebugLoc(), TII->get(ARM::t2Bcc));
     MIB.add(End->getOperand(1));  // branch target
     MIB.addImm(ARMCC::NE);        // condition code
     MIB.addReg(ARM::CPSR);
     End->eraseFromParent();
     Dec->eraseFromParent();
   };
 
+  // If there is an unconditional branch, after I, that just branches to the
+  // next block, remove it.
+  auto RemoveDeadBranch = [](MachineInstr *I) {
+    MachineBasicBlock *BB = I->getParent();
+    MachineInstr *Terminator = &BB->instr_back();
+    if (Terminator->isUnconditionalBranch() && I != Terminator) {
+      MachineBasicBlock *Succ = Terminator->getOperand(0).getMBB();
+      if (BB->isLayoutSuccessor(Succ)) {
+        LLVM_DEBUG(dbgs() << "ARM Loops: Removing branch: " << *Terminator);
+        Terminator->eraseFromParent();
+      }
+    }
+  };
+
   if (Revert) {
     Start->eraseFromParent();
     ExpandBranch(Dec, End);
   } else {
-    ExpandLoopStart(ML, Start);
-    ExpandLoopEnd(ML, Dec, End);
+    Start = ExpandLoopStart(ML, Start);
+    RemoveDeadBranch(Start);
+    End = ExpandLoopEnd(ML, Dec, End);
+    RemoveDeadBranch(End);
   }
 }
 
 FunctionPass *llvm::createARMLowOverheadLoopsPass() {
   return new ARMLowOverheadLoops();
 }

lib/Target/ARM/ARMTargetTransformInfo.cpp

bool ARMTTIImpl::isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,
   };
 
   auto IsHardwareLoopIntrinsic = [](Instruction &I) {
     if (auto *Call = dyn_cast<IntrinsicInst>(&I)) {
       switch (Call->getIntrinsicID()) {
       default:
         break;
       case Intrinsic::set_loop_iterations:
+      case Intrinsic::test_set_loop_iterations:
       case Intrinsic::loop_decrement:
       case Intrinsic::loop_decrement_reg:
         return true;
       }
     }
     return false;
   };
 
bool ARMTTIImpl::isHardwareLoopProfitable(Loop *L, ScalarEvolution &SE,
 
   // TODO: Check whether the trip count calculation is expensive. If L is the
   // inner loop but we know it has a low trip count, calculating that trip
   // count (in the parent loop) may be detrimental.
 
   LLVMContext &C = L->getHeader()->getContext();
   HWLoopInfo.CounterInReg = true;
   HWLoopInfo.IsNestingLegal = false;
+  HWLoopInfo.PerformEntryTest = true;
   HWLoopInfo.CountType = Type::getInt32Ty(C);
   HWLoopInfo.LoopDecrement = ConstantInt::get(HWLoopInfo.CountType, 1);
   return true;
 }
 
 void ARMTTIImpl::getUnrollingPreferences(Loop *L, ScalarEvolution &SE,
                                          TTI::UnrollingPreferences &UP) {
   // Only currently enable these preferences for M-Class cores.

test/Transforms/HardwareLoops/ARM/do-rem.ll

 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s
 
 @g = common local_unnamed_addr global i32* null, align 4
 
 ; CHECK-LABEL: do_with_i32_urem
+; CHECK: entry:
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %n)
+; CHECK: br i1 [[TEST]], label %while.body.preheader, label %while.end
+
 ; CHECK: while.body.preheader:
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
 ; CHECK-NEXT: br label %while.body
 
+; CHECK: while.body:
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 define i32 @do_with_i32_urem(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i32 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i32_srem
+; CHECK: entry:
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %n)
+; CHECK: br i1 [[TEST]], label %while.body.preheader, label %while.end
+
 ; CHECK: while.body.preheader:
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
 ; CHECK-NEXT: br label %while.body
 
+; CHECK: while.body:
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 define i32 @do_with_i32_srem(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i32 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i32_udiv
+; CHECK: entry:
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %n)
+; CHECK: br i1 [[TEST]], label %while.body.preheader, label %while.end
+
 ; CHECK: while.body.preheader:
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
 ; CHECK-NEXT: br label %while.body
 
+; CHECK: while.body:
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 define i32 @do_with_i32_udiv(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i32 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i32_sdiv
+; CHECK: entry:
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %n)
+; CHECK: br i1 [[TEST]], label %while.body.preheader, label %while.end
+
 ; CHECK: while.body.preheader:
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
 ; CHECK-NEXT: br label %while.body
 
+; CHECK: while.body:
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 define i32 @do_with_i32_sdiv(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i32 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i64_urem
-; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.{{.*}}.loop.iterations
 ; CHECK-NOT: llvm.loop.decrement
 define i64 @do_with_i64_urem(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
   br i1 %cmp7, label %while.end, label %while.body.preheader
 
 while.body.preheader:
   br label %while.body
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i64 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i64_srem
-; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.{{.*}}.loop.iterations
 ; CHECK-NOT: llvm.loop.decrement
 define i64 @do_with_i64_srem(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
   br i1 %cmp7, label %while.end, label %while.body.preheader
 
 while.body.preheader:
   br label %while.body
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i64 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i64_udiv
-; CHECK-NOT: llvm.set.loop.iterations
+; CHECK-NOT: llvm.{{.*}}.loop.iterations
 ; CHECK-NOT: llvm.loop.decrement
 define i64 @do_with_i64_udiv(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
   br i1 %cmp7, label %while.end, label %while.body.preheader
 
 while.body.preheader:
   br label %while.body
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i64 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i64 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_with_i64_sdiv
-; CHECK-NOT: call void @llvm.set.loop.iterations
+; CHECK-NOT: call void @llvm.{{.*}}.loop.iterations
 ; CHECK-NOT: call i32 @llvm.loop.decrement
 define i64 @do_with_i64_sdiv(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
   br i1 %cmp7, label %while.end, label %while.body.preheader
 
 while.body.preheader:
   br label %while.body

test/Transforms/HardwareLoops/ARM/fp-emulation.ll

 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+fp-armv8 -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-FP
 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+soft-float -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-SOFT
 
 ; CHECK-LABEL: test_fptosi
-; CHECK: while.body.lr.ph:
+; CHECK-FP: entry:
 ; CHECK-FP: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
 ; CHECK-FP: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
-; CHECK-FP: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
-; CHECK-FP-NEXT: br label %while.body
+; CHECK-FP: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-FP: br i1 [[TEST]], label %while.body.lr.ph, label %cleanup
+
+; CHECK: while.body.lr.ph:
+; CHECK-FP: br label %while.body
 
+; CHECK: while.body:
 ; CHECK-FP: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
 ; CHECK-FP: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK-FP: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-FP: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
 
 ; CHECK-SOFT-NOT: call void @llvm.set.loop.iterations
 
 define void @test_fptosi(i32 %n, i32** %g, double** %d) {

 cleanup.loopexit:
   br label %cleanup
 
 cleanup:
   ret void
 }
 
 ; CHECK-LABEL: test_fptoui
-; CHECK-FP: while.body.lr.ph:
+; CHECK-FP: entry:
 ; CHECK-FP: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
 ; CHECK-FP: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
-; CHECK-FP: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
-; CHECK-FP-NEXT: br label %while.body
+; CHECK-FP: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-FP: br i1 [[TEST]], label %while.body.lr.ph, label %cleanup
+
+; CHECK: while.body.lr.ph:
+; CHECK-FP: br label %while.body
 
+; CHECK: while.body:
 ; CHECK-FP: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
 ; CHECK-FP: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK-FP: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-FP: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
 
 ; CHECK-SOFT-NOT: call void @llvm.set.loop.iterations
 
 define void @test_fptoui(i32 %n, i32** %g, double** %d) {

 cleanup.loopexit:
   br label %cleanup
 
 cleanup:
   ret void
 }
 
 ; CHECK-LABEL: load_store_float
-; CHECK: while.body.lr.ph:
+; CHECK: entry:
 ; CHECK: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
 ; CHECK: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
-; CHECK-NEXT: br label %while.body
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK: br i1 [[TEST]], label %while.body.lr.ph, label %cleanup
 
+; CHECK: while.body.lr.ph:
+; CHECK: br label %while.body
+
+; CHECK: while.body:
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
 
 define void @load_store_float(i32 %n, double** %d, double** %g) {
 entry:
   %n.off = add i32 %n, -1

 cleanup.loopexit:
   br label %cleanup
 
 cleanup:
   ret void
 }
 
 ; CHECK-LABEL: fp_add
-; CHECK: while.body.lr.ph:
-
 ; CHECK-SOFT-NOT: call void @llvm.set.loop.iterations
+; CHECK-SOFT-NOT: call i32 @llvm.loop.decrement
 
+; CHECK-FP: entry:
 ; CHECK-FP: [[CMP:%[^ ]+]] = icmp ugt i32 %n, 1
 ; CHECK-FP: [[COUNT:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 1
-; CHECK-FP: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
-; CHECK: br label %while.body
+; CHECK-FP: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-FP: br i1 [[TEST]], label %while.body.lr.ph, label %cleanup
 
-; CHECK-SOFT-NOT: call i32 @llvm.loop.decrement
+; CHECK: while.body.lr.ph:
+; CHECK-FP: br label %while.body
 
+; CHECK: while.body:
 ; CHECK-FP: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %if.end4 ]
 ; CHECK-FP: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK-FP: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-FP: br i1 [[CMP]], label %while.body, label %cleanup.loopexit
 
 define void @fp_add(i32 %n, float** %d, float** %g) {
 entry:
   %n.off = add i32 %n, -1

test/Transforms/HardwareLoops/ARM/simple-do.ll

 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=false %s -S -o - | FileCheck %s
 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -hardware-loops -disable-arm-loloops=true %s -S -o - | FileCheck %s --check-prefix=DISABLED
 ; RUN: opt -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=-lob -hardware-loops %s -S -o - | FileCheck %s --check-prefix=DISABLED
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -disable-arm-loloops=false %s -o - | FileCheck %s --check-prefix=CHECK-LLC
 
-; DISABLED-NOT: llvm.set.loop.iterations
+; DISABLED-NOT: llvm.{{.*}}.loop.iterations
 ; DISABLED-NOT: llvm.loop.decrement
 
 @g = common local_unnamed_addr global i32* null, align 4
 
 ; CHECK-LABEL: do_copy
 ; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
 ; CHECK: br label %while.body
 
while.body:
   %tobool = icmp eq i32 %dec, 0
   br i1 %tobool, label %while.end, label %while.body
 
 while.end:
   ret i32 0
 }
 
 ; CHECK-LABEL: do_inc1
+; CHECK: entry:
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %n)
+; CHECK: br i1 [[TEST]], label %while.body.lr.ph, label %while.end
+
 ; CHECK: while.body.lr.ph:
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %n)
-; CHECK-NEXT: br label %while.body
+; CHECK: br label %while.body
 
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ %n, %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 ; CHECK-LLC-LABEL:do_inc1:
-; CHECK-LLC:        dls lr,
+; CHECK-LLC:        wls lr, {{.*}}, [[LOOP_EXIT:.[LBB_0-3]+]]
 ; CHECK-LLC-NOT:    mov lr,
 ; CHECK-LLC:      [[LOOP_HEADER:\.LBB[0-9_]+]]:
 ; CHECK-LLC:        le lr, [[LOOP_HEADER]]
 ; CHECK-LLC-NOT:    b [[LOOP_EXIT:\.LBB[0-9_]+]]
-; CHECK-LLC:      [[LOOP_EXIT:\.LBB[0-9_]+]]:
+; CHECK-LLC:      [[LOOP_EXIT]]:
 
 define i32 @do_inc1(i32 %n) {
 entry:
   %cmp7 = icmp eq i32 %n, 0
   br i1 %cmp7, label %while.end, label %while.body.lr.ph
 
 while.body.lr.ph:
   %0 = load i32*, i32** @g, align 4
while.end.loopexit:
   br label %while.end
 
 while.end:
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i32 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_inc2
-; CHECK: while.body.lr.ph:
+; CHECK: entry:
 ; CHECK: [[ROUND:%[^ ]+]] = add i32 %n, -1
 ; CHECK: [[HALVE:%[^ ]+]] = lshr i32 [[ROUND]], 1
 ; CHECK: [[COUNT:%[^ ]+]] = add nuw i32 [[HALVE]], 1
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
-; CHECK-NEXT: br label %while.body
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK: br i1 [[TEST]], label %while.body.lr.ph, label %while.end
+
+; CHECK: while.body.lr.ph:
+; CHECK: br label %while.body
 
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 ; CHECK-LLC:      do_inc2:
 ; CHECK-LLC-NOT:    mov lr,
-; CHECK-LLC:        dls lr,
+; CHECK-LLC:        wls lr, {{.*}}, [[LOOP_EXIT:.LBB[0-9_]+]]
 ; CHECK-LLC-NOT:    mov lr,
 ; CHECK-LLC:      [[LOOP_HEADER:\.LBB[0-9._]+]]:
 ; CHECK-LLC:        le lr, [[LOOP_HEADER]]
 ; CHECK-LLC-NOT:    b [[LOOP_EXIT:\.LBB[0-9._]+]]
-; CHECK-LLC:      [[LOOP_EXIT:\.LBB[0-9_]+]]:
+; CHECK-LLC:      [[LOOP_EXIT]]:
 
 define i32 @do_inc2(i32 %n) {
 entry:
   %cmp7 = icmp sgt i32 %n, 0
   br i1 %cmp7, label %while.body.lr.ph, label %while.end
 
 while.body.lr.ph:
   %0 = load i32*, i32** @g, align 4

 
 while.end:
   %res.0.lcssa = phi i32 [ 0, %entry ], [ %add, %while.end.loopexit ]
   ret i32 %res.0.lcssa
 }
 
 ; CHECK-LABEL: do_dec2
 
-; CHECK: while.body.lr.ph:
+; CHECK: entry:
 ; CHECK: [[ROUND:%[^ ]+]] = add i32 %n, 1
 ; CHECK: [[CMP:%[^ ]+]] = icmp slt i32 %n, 2
 ; CHECK: [[SMIN:%[^ ]+]] = select i1 [[CMP]], i32 %n, i32 2
 ; CHECK: [[SUB:%[^ ]+]] = sub i32 [[ROUND]], [[SMIN]]
 ; CHECK: [[HALVE:%[^ ]+]] = lshr i32 [[SUB]], 1
 ; CHECK: [[COUNT:%[^ ]+]] = add nuw i32 [[HALVE]], 1
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
-; CHECK-NEXT: br label %while.body
+; CHECK: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK: br i1 [[TEST]], label %while.body.lr.ph, label %while.end
+
+; CHECK: while.body.lr.ph:
+; CHECK: br label %while.body
 
 ; CHECK: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.lr.ph ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK: br i1 [[CMP]], label %while.body, label %while.end.loopexit
 
 ; CHECK-LLC:      do_dec2
 ; CHECK-LLC-NOT:    mov lr,
-; CHECK-LLC:        dls lr,
+; CHECK-LLC:        wls lr, {{.*}}, [[LOOP_EXIT:.LBB[0-9_]+]]
 ; CHECK-LLC-NOT:    mov lr,
 ; CHECK-LLC:      [[LOOP_HEADER:\.LBB[0-9_]+]]:
 ; CHECK-LLC:        le lr, [[LOOP_HEADER]]
 ; CHECK-LLC-NOT:    b .
-; CHECK-LLC:      @ %while.end
+; CHECK-LLC:      [[LOOP_EXIT]]:
 define i32 @do_dec2(i32 %n) {
 entry:
   %cmp6 = icmp sgt i32 %n, 0
   br i1 %cmp6, label %while.body.lr.ph, label %while.end
 
 while.body.lr.ph:
   %0 = load i32*, i32** @g, align 4
   br label %while.body

test/Transforms/HardwareLoops/ARM/structure.ll

while.body:                                       ; preds = %while.body, %entry
   %2 = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 %0, i32 1)
   %3 = icmp ne i32 %2, 0
   br i1 %3, label %while.body, label %while.end
 
 while.end:                                        ; preds = %while.body
   ret i32 0
 }
 
+; CHECK-LABEL: pre_existing_test_set
+; CHECK: call i1 @llvm.test.set.loop.iterations
+; CHECK-NOT: llvm.set{{.*}}.loop.iterations
+; CHECK: call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 %0, i32 1)
+; CHECK-NOT: call i32 @llvm.loop.decrement.reg
+define i32 @pre_existing_test_set(i32 %n, i32* nocapture %p, i32* nocapture readonly %q) {
+entry:
+  %guard = call i1 @llvm.test.set.loop.iterations.i32(i32 %n)
+  br i1 %guard, label %while.preheader, label %while.end
+
+while.preheader:
+  br label %while.body
+
+while.body:                                       ; preds = %while.body, %entry
+  %q.addr.05 = phi i32* [ %incdec.ptr, %while.body ], [ %q, %while.preheader ]
+  %p.addr.04 = phi i32* [ %incdec.ptr1, %while.body ], [ %p, %while.preheader ]
+  %0 = phi i32 [ %n, %while.preheader ], [ %2, %while.body ]
+  %incdec.ptr = getelementptr inbounds i32, i32* %q.addr.05, i32 1
+  %1 = load i32, i32* %q.addr.05, align 4
+  %incdec.ptr1 = getelementptr inbounds i32, i32* %p.addr.04, i32 1
+  store i32 %1, i32* %p.addr.04, align 4
+  %2 = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 %0, i32 1)
+  %3 = icmp ne i32 %2, 0
+  br i1 %3, label %while.body, label %while.end
+
+while.end:                                        ; preds = %while.body
+  ret i32 0
+}
+
 ; CHECK-LABEL: pre_existing_inner
 ; CHECK-NOT: llvm.set.loop.iterations
 ; CHECK: while.cond1.preheader.us:
 ; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
 ; CHECK: call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 %0, i32 1)
 ; CHECK: br i1
 ; CHECK-NOT: call i32 @llvm.loop.decrement
 define void @pre_existing_inner(i32* nocapture %A, i32 %N) {
latch.1:
   %cmp.2 = icmp ult i32 %count.next, %N
   br i1 %cmp.2, label %header, label %latch.1
 
 exit:
   ret void
 }
 
 ; CHECK-LABEL: search
+; CHECK: entry:
+; CHECK:   [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
+; CHECK:   br i1 [[TEST]], label %for.body.preheader, label %for.cond.cleanup
 ; CHECK: for.body.preheader:
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
-; CHECK: br label %for.body
+; CHECK:   br label %for.body
 ; CHECK: for.body:
 ; CHECK: for.inc:
-; CHECK: [[LOOP_DEC:%[^ ]+]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32
-; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
-; CHECK: br i1 [[CMP]], label %for.body, label %for.cond.cleanup
+; CHECK:   [[LOOP_DEC:%[^ ]+]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32
+; CHECK:   [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
+; CHECK:   br i1 [[CMP]], label %for.body, label %for.cond.cleanup
 define i32 @search(i8* nocapture readonly %c, i32 %N) {
 entry:
   %cmp11 = icmp eq i32 %N, 0
   br i1 %cmp11, label %for.cond.cleanup, label %for.body
 
 for.cond.cleanup:
   %found.0.lcssa = phi i32 [ 0, %entry ], [ %found.1, %for.inc ]
   %spaces.0.lcssa = phi i32 [ 0, %entry ], [ %spaces.1, %for.inc ]
for.inc:                                          ; preds = %sw.bb, %sw.bb1, %for.body
   %found.1 = phi i32 [ %found.012, %for.body ], [ %found.012, %sw.bb1 ], [ %inc, %sw.bb ]
   %spaces.1 = phi i32 [ %spaces.013, %for.body ], [ %inc2, %sw.bb1 ], [ %spaces.013, %sw.bb ]
   %inc3 = add nuw i32 %i.014, 1
   %exitcond = icmp eq i32 %inc3, %N
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; CHECK-LABEL: unroll_inc_int
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
+; CHECK: call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
 ; CHECK: call i32 @llvm.loop.decrement.reg.i32.i32.i32(
 
 ; TODO: We should be able to support the unrolled loop body.
-; CHECK-UNROLL-LABEL: unroll_inc_int:
+; CHECK-UNROLL-LABEL: unroll_inc_int
 ; CHECK-UNROLL:     [[PREHEADER:.LBB[0-9_]+]]: @ %for.body.preheader
 ; CHECK-UNROLL-NOT: dls
 ; CHECK-UNROLL:     [[LOOP:.LBB[0-9_]+]]: @ %for.body
 ; CHECK-UNROLL-NOT: le lr, [[LOOP]]
 ; CHECK-UNROLL:     bne [[LOOP]]
-; CHECK-UNROLL:     %for.body.epil.preheader
-; CHECK-UNROLL:     dls
-; CHECK-UNROLL:     %for.body.epil
-; CHECK-UNROLL:     le
+; CHECK-UNROLL:     wls lr, lr, [[EXIT:.LBB[0-9_]+]]
+; CHECK-UNROLL:     [[EPIL:.LBB[0-9_]+]]:
+; CHECK-UNROLL:     le lr, [[EPIL]]
+; CHECK-UNROLL-NEXT: [[EXIT]]
 
 define void @unroll_inc_int(i32* nocapture %a, i32* nocapture readonly %b, i32* nocapture readonly %c, i32 %N) {
 entry:
   %cmp8 = icmp sgt i32 %N, 0
   br i1 %cmp8, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:
   ret void
 
 for.body:
   %i.09 = phi i32 [ %inc, %for.body ], [ 0, %entry ]
   %arrayidx = getelementptr inbounds i32, i32* %b, i32 %i.09
   %0 = load i32, i32* %arrayidx, align 4
   %arrayidx1 = getelementptr inbounds i32, i32* %c, i32 %i.09
   %1 = load i32, i32* %arrayidx1, align 4
   %mul = mul nsw i32 %1, %0
   %arrayidx2 = getelementptr inbounds i32, i32* %a, i32 %i.09
   store i32 %mul, i32* %arrayidx2, align 4
   %inc = add nuw nsw i32 %i.09, 1
   %exitcond = icmp eq i32 %inc, %N
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; CHECK-LABEL: unroll_inc_unsigned
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
+; CHECK: call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
 ; CHECK: call i32 @llvm.loop.decrement.reg.i32.i32.i32(
 
 ; CHECK-LLC-LABEL: unroll_inc_unsigned:
-; CHECK-LLC: dls lr, [[COUNT:r[0-9]+]]
-; CHECK-LLC: le  lr
+; CHECK-LLC: wls lr, r3, [[EXIT:.LBB[0-9_]+]]
+; CHECK-LLC: [[HEADER:.LBB[0-9_]+]]:
+; CHECK-LLC: le lr, [[HEADER]]
+; CHECK-LLC-NEXT: [[EXIT]]:
 
 ; TODO: We should be able to support the unrolled loop body.
-; CHECK-UNROLL-LABEL: unroll_inc_unsigned:
+; CHECK-UNROLL-LABEL: unroll_inc_unsigned
 ; CHECK-UNROLL:     [[PREHEADER:.LBB[0-9_]+]]: @ %for.body.preheader
 ; CHECK-UNROLL-NOT: dls
 ; CHECK-UNROLL:     [[LOOP:.LBB[0-9_]+]]: @ %for.body
 ; CHECK-UNROLL-NOT: le lr, [[LOOP]]
 ; CHECK-UNROLL:     bne [[LOOP]]
-; CHECK-UNROLL:     %for.body.epil.preheader
-; CHECK-UNROLL:     dls
-; CHECK-UNROLL:     %for.body.epil
-; CHECK-UNROLL:     le
+; CHECK-UNROLL:     wls lr, lr, [[EPIL_EXIT:.LBB[0-9_]+]]
+; CHECK-UNROLL: [[EPIL:.LBB[0-9_]+]]:
+; CHECK-UNROLL:     le lr, [[EPIL]]
+; CHECK-UNROLL: [[EPIL_EXIT]]:
+; CHECK-UNROLL:     pop
 define void @unroll_inc_unsigned(i32* nocapture %a, i32* nocapture readonly %b, i32* nocapture readonly %c, i32 %N) {
 entry:
   %cmp8 = icmp eq i32 %N, 0
   br i1 %cmp8, label %for.cond.cleanup, label %for.body
 
 for.cond.cleanup:
   ret void
 
 for.body:
   %i.09 = phi i32 [ %inc, %for.body ], [ 0, %entry ]
   %arrayidx = getelementptr inbounds i32, i32* %b, i32 %i.09
   %0 = load i32, i32* %arrayidx, align 4
   %arrayidx1 = getelementptr inbounds i32, i32* %c, i32 %i.09
   %1 = load i32, i32* %arrayidx1, align 4
   %mul = mul nsw i32 %1, %0
   %arrayidx2 = getelementptr inbounds i32, i32* %a, i32 %i.09
   store i32 %mul, i32* %arrayidx2, align 4
   %inc = add nuw i32 %i.09, 1
   %exitcond = icmp eq i32 %inc, %N
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; CHECK-LABEL: unroll_dec_int
-; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
+; CHECK: call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
 ; CHECK: call i32 @llvm.loop.decrement.reg.i32.i32.i32(
 
 ; TODO: An unnecessary register is being held to hold COUNT, lr should just
 ; be used instead.
 ; CHECK-LLC-LABEL: unroll_dec_int:
-; CHECK-LLC: dls lr, [[COUNT:r[0-9]+]]
-; CHECK-LLC: subs  [[COUNT]], #1
-; CHECK-LLC: le  lr
+; CHECK-LLC: wls lr, r3, [[EXIT:.LBB[0-9_]+]]
+; CHECK-LLC: [[HEADER:.LBB[0-9_]+]]:
+; CHECK-LLC: le lr, [[HEADER]]
+; CHECK-LLC-NEXT: [[EXIT]]:
 
 ; CHECK-UNROLL-LABEL: unroll_dec_int
-; CHECK-UNROLL: dls lr
-; CHECK-UNROLL: le lr
-; CHECK-UNROLL: dls lr
-; CHECK-UNROLL: le lr
+; CHECK-UNROLL:         wls lr, {{.*}}, [[PROLOGUE_EXIT:.LBB[0-9_]+]]
+; CHECK-UNROLL-NEXT: [[PROLOGUE:.LBB[0-9_]+]]:
+; CHECK-UNROLL:         le lr, [[PROLOGUE]]
+; CHECK-UNROLL-NEXT: [[PROLOGUE_EXIT:.LBB[0-9_]+]]:
+; CHECK-UNROLL:         wls lr, lr, [[EXIT:.LBB[0-9_]+]]
+; CHECK-UNROLL:      [[BODY:.LBB[0-9_]+]]:
+; CHECK-UNROLL:         le lr, [[BODY]]
+; CHECK-UNROLL-NEXT: [[EXIT]]:
 define void @unroll_dec_int(i32* nocapture %a, i32* nocapture readonly %b, i32* nocapture readonly %c, i32 %N) {
 entry:
   %cmp8 = icmp sgt i32 %N, 0
   br i1 %cmp8, label %for.body, label %for.cond.cleanup
 
 for.cond.cleanup:
   ret void
 
 for.body:
   %i.09 = phi i32 [ %dec, %for.body ], [ %N, %entry ]
   %arrayidx = getelementptr inbounds i32, i32* %b, i32 %i.09
   %0 = load i32, i32* %arrayidx, align 4
   %arrayidx1 = getelementptr inbounds i32, i32* %c, i32 %i.09
   %1 = load i32, i32* %arrayidx1, align 4
   %mul = mul nsw i32 %1, %0
   %arrayidx2 = getelementptr inbounds i32, i32* %a, i32 %i.09
   store i32 %mul, i32* %arrayidx2, align 4
   %dec = add nsw i32 %i.09, -1
   %cmp = icmp sgt i32 %dec, 0
   br i1 %cmp, label %for.body, label %for.cond.cleanup
 }
 
 declare void @llvm.set.loop.iterations.i32(i32) #0
+declare i1 @llvm.test.set.loop.iterations.i32(i32) #0
 declare i32 @llvm.loop.decrement.reg.i32.i32.i32(i32, i32) #0
 

test/Transforms/HardwareLoops/ARM/while.mir

+# RUN: llc -mtriple=thumbv8.1m.main -mattr=+lob %s -run-pass=arm-low-overhead-loops --verify-machineinstrs -o - | FileCheck %s
+# CHECK: $lr = t2WLS
+# CHECK: $lr = t2LEUpdate renamable $lr
+
+--- |
+  target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
+  target triple = "thumbv8.1m.main-arm-unknown"
+  
+  ; Function Attrs: norecurse nounwind optsize
+  define dso_local arm_aapcscc void @copy(i16* nocapture %a, i16* nocapture readonly %b, i32 %N) {
+  entry:
+    %cmp4 = icmp eq i32 %N, 0
+    %0 = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
+    br i1 %0, label %while.body.preheader, label %while.end
+  
+  while.body.preheader:                             ; preds = %entry
+    br label %while.body
+  
+  while.body:                                       ; preds = %while.body, %while.body.preheader
+    %a.addr.06 = phi i16* [ %incdec.ptr1, %while.body ], [ %a, %while.body.preheader ]
+    %b.addr.05 = phi i16* [ %incdec.ptr, %while.body ], [ %b, %while.body.preheader ]
+    %1 = phi i32 [ %N, %while.body.preheader ], [ %3, %while.body ]
+    %incdec.ptr = getelementptr inbounds i16, i16* %b.addr.05, i32 1
+    %2 = load i16, i16* %b.addr.05, align 2, !tbaa !3
+    %incdec.ptr1 = getelementptr inbounds i16, i16* %a.addr.06, i32 1
+    store i16 %2, i16* %a.addr.06, align 2, !tbaa !3
+    %3 = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 %1, i32 1)
+    %4 = icmp ne i32 %3, 0
+    br i1 %4, label %while.body, label %while.end
+  
+  while.end:                                        ; preds = %while.body, %entry
+    ret void
+  }
+  
+  declare i1 @llvm.test.set.loop.iterations.i32(i32) #1
+  declare i32 @llvm.loop.decrement.reg.i32.i32.i32(i32, i32) #1
+  
+  attributes #1 = { noduplicate nounwind }
+  attributes #2 = { nounwind }
+  
+  !llvm.module.flags = !{!0, !1}
+  
+  !0 = !{i32 1, !"wchar_size", i32 4}
+  !1 = !{i32 1, !"min_enum_size", i32 4}
+  !3 = !{!4, !4, i64 0}
+  !4 = !{!"short", !5, i64 0}
+  !5 = !{!"omnipotent char", !6, i64 0}
+  !6 = !{!"Simple C/C++ TBAA"}
+
+...
+---
+name:            copy
+alignment:       1
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: false
+hasWinCFI:       false
+registers:       []
+liveins:         
+  - { reg: '$r0', virtual-reg: '' }
+  - { reg: '$r1', virtual-reg: '' }
+  - { reg: '$r2', virtual-reg: '' }
+frameInfo:       
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       8
+  offsetAdjustment: 0
+  maxAlignment:    4
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           
+  - { id: 0, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4, 
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false, 
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4, 
+      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true, 
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x40000000), %bb.3(0x40000000)
+  
+    frame-setup tPUSH 14, $noreg, $r7, killed $lr, implicit-def $sp, implicit $sp
+    frame-setup CFI_INSTRUCTION def_cfa_offset 8
+    frame-setup CFI_INSTRUCTION offset $lr, -4
+    frame-setup CFI_INSTRUCTION offset $r7, -8
+    $r7 = frame-setup tMOVr $sp, 14, $noreg
+    frame-setup CFI_INSTRUCTION def_cfa_register $r7
+    t2WhileLoopStart $r2, %bb.3
+    tB %bb.1, 14, $noreg
+  
+  bb.1.while.body.preheader:
+    successors: %bb.2(0x80000000)
+  
+    $lr = tMOVr killed $r2, 14, $noreg
+  
+  bb.2.while.body:
+    successors: %bb.2(0x7c000000), %bb.3(0x04000000)
+  
+    renamable $r2, renamable $r1 = t2LDRH_POST killed renamable $r1, 2, 14, $noreg :: (load 2 from %ir.b.addr.05, !tbaa !3)
+    early-clobber renamable $r0 = t2STRH_POST killed renamable $r2, killed renamable $r0, 2, 14, $noreg :: (store 2 into %ir.a.addr.06, !tbaa !3)
+    renamable $lr = t2LoopDec killed renamable $lr, 1
+    t2LoopEnd renamable $lr, %bb.2
+    tB %bb.3, 14, $noreg
+  
+  bb.3.while.end:
+    tPOP_RET 14, $noreg, def $r7, def $pc
+
+...