[HardwareLoops] Loop counter guard Loop intrinsic

[samparker]

Summary

Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero.

[SjoerdMeijer]

Just a first look, some questions inline.

include/llvm/IR/Intrinsics.td

1195

Perhaps it's blindingly obvious, but maybe a comment that this is useful for while-do hardware kind of loops?

lib/CodeGen/HardwareLoops.cpp

269

Is there always a preheader? Can we dereference a nullptr here?

276

perhaps silly question, I guess the assumption here is that is if this is a conditional then it must be the loop check, but is that always true?

[samparker]

lib/CodeGen/HardwareLoops.cpp

269

Just before landing here we create a preheader if one doesn't exist.

276

Well it certainly is controlling whether the loop is entered or not... I had the same thought tbh but I reasoned that any other condition of entering a loop would be handled before the iteration condition - i will add some tests along these lines though.

[samparker]

lib/CodeGen/HardwareLoops.cpp

276

My assumption was totally wrong :) so I'll revert to what I did originally: check for an icmp ne 0 so that the semantics don't change. I think we should be able to handle other icmp later by transforming them, if it's profitable.

[SjoerdMeijer]

lib/CodeGen/HardwareLoops.cpp

276

cheers, sounds like a good plan

[samparker]

Added a check that we're replacing the conditional with an equivalent statement... The intrinsic tests for a non-zero value for the trip count, so we now ensure that the loop entry is guarded by icmp ne %trip.count, 0. This has meant the code has been moved around a bit from the previous patch.

[SjoerdMeijer]

lib/CodeGen/HardwareLoops.cpp

307

A nit, and more a style preference, so please ignore if you disagree!
The first thing in initLoopCount is a very big lambda definition. But I think CanGenerateTest would be a good candidate for a local helper function as it requires very few arguments and only returns a bool, so that InitLoopCount becomes smaller and easier to read.

326

Another nit: I think this could be nice candidate for a lambda to share code, i.e. the checks are the same in the if-clause and else-if clause, just the indexes are swapped.

352

I am now actually wondering if we are checking here if the loop is particular kind of loop, i.e. a Single-Entry Single-Exit loop? Or, do we e.g. get this 'guarantee' from SCEV, because otherwise we can't calculate an iteration count? But what happens here if there are multiple predecessors?

[samparker]

lib/CodeGen/HardwareLoops.cpp

352

There can be multiple exits, but the search for the given exit block is quite odd in my opinion... the exit block that forms the loop has to be executed on every iteration - dominating all other exit blocks. The trip count is given for that exit block.

Multiple predecessors of what...? For the loop preheader, we just don't try to guard the entry and place the normal set.loop.iteration counter in the preheader. I'm wondering whether we could actually use the test.set.loop.iterations in the case of multiple preheader predecessors by placing the intrinsic in all those predecessors.... Something to look at later.

[samparker]

• CanGenerateTest is its own function - also containing a lambda to check the icmp.
• Also using isLoopGuardedByCond to try to avoid the loop count being expanded before its required.

[SjoerdMeijer]

Last question I guess, I know it defaults to false, and hope I didn't miss it, but perhaps good to have a test too that explicitly sets -force-hardware-loop-guard=false?

lib/CodeGen/HardwareLoops.cpp

352

Ok, got it, thanks!

[samparker]

• Added a test line for -force-hardware-loop-guard=false.
• Also added a test line for -force-hardware-loop-phi=true for some extra coverage.

[SjoerdMeijer]

Looks good to me!
Perhaps wait a day with committing just in case someone has more comments.

include/llvm/Analysis/TargetTransformInfo.h

struct HardwareLoopInfo {
   const SCEV *ExitCount = nullptr;
   IntegerType *CountType = nullptr;
   Value *LoopDecrement = nullptr; // Decrement the loop counter by this
                                   // value in every iteration.
   bool IsNestingLegal = false;    // Can a hardware loop be a parent to
                                   // another hardware loop?
   bool CounterInReg = false;      // Should loop counter be updated in
                                   // the loop via a phi?
+  bool PerformEntryTest = false;  // Generate the intrinsic which also performs
+                                  // icmp ne zero on the loop counter value and
+                                  // produces an i1 to guard the loop entry.
   bool isHardwareLoopCandidate(ScalarEvolution &SE, LoopInfo &LI,
                                DominatorTree &DT, bool ForceNestedLoop = false,
                                bool ForceHardwareLoopPHI = false);
 };
 
 /// This pass provides access to the codegen interfaces that are needed
 /// for IR-level transformations.
 class TargetTransformInfo {

include/llvm/IR/Intrinsics.td

 
 //===---------- Intrinsics to control hardware supported loops ----------===//
 
 // Specify that the value given is the number of iterations that the next loop
 // will execute.
 def int_set_loop_iterations :
   Intrinsic<[], [llvm_anyint_ty], [IntrNoDuplicate]>;
 
+// Specify that the value given is the number of iterations that the next loop
+// will execute. Also test that the given count is not zero.
+def int_test_set_loop_iterations :
+  Intrinsic<[llvm_i1_ty], [llvm_anyint_ty], [IntrNoDuplicate]>;
+
 // Decrement loop counter by the given argument. Return false if the loop
 // should exit.
 def int_loop_decrement :
   Intrinsic<[llvm_i1_ty], [llvm_anyint_ty], [IntrNoDuplicate]>;
 
 // Decrement the first operand (the loop counter) by the second operand (the
 // maximum number of elements processed in an iteration). Return the remaining
 // number of iterations still to be executed. This is effectively a sub which

lib/CodeGen/HardwareLoops.cpp

 static cl::opt<unsigned>
 LoopDecrement("hardware-loop-decrement", cl::Hidden, cl::init(1),
             cl::desc("Set the loop decrement value"));
 
 static cl::opt<unsigned>
 CounterBitWidth("hardware-loop-counter-bitwidth", cl::Hidden, cl::init(32),
                 cl::desc("Set the loop counter bitwidth"));
 
+static cl::opt<bool>
+ForceGuardLoopEntry(
+  "force-hardware-loop-guard", cl::Hidden, cl::init(false),
+  cl::desc("Force generation of loop guard intrinsic"));
+
 STATISTIC(NumHWLoops, "Number of loops converted to hardware loops");
 
 namespace {
 
   using TTI = TargetTransformInfo;
 
   class HardwareLoops : public FunctionPass {
   public:
namespace {
   public:
     HardwareLoop(HardwareLoopInfo &Info, ScalarEvolution &SE,
                  const DataLayout &DL) :
       SE(SE), DL(DL), L(Info.L), M(L->getHeader()->getModule()),
       ExitCount(Info.ExitCount),
       CountType(Info.CountType),
       ExitBranch(Info.ExitBranch),
       LoopDecrement(Info.LoopDecrement),
-      UsePHICounter(Info.CounterInReg) { }
+      UsePHICounter(Info.CounterInReg),
+      UseLoopGuard(Info.PerformEntryTest) { }
 
     void Create();
 
   private:
     ScalarEvolution &SE;
     const DataLayout &DL;
     Loop *L                 = nullptr;
     Module *M               = nullptr;
     const SCEV *ExitCount   = nullptr;
     Type *CountType         = nullptr;
     BranchInst *ExitBranch  = nullptr;
-    Value *LoopDecrement      = nullptr;
+    Value *LoopDecrement    = nullptr;
     bool UsePHICounter      = false;
+    bool UseLoopGuard       = false;
   };
 }
 
 char HardwareLoops::ID = 0;
 
 bool HardwareLoops::runOnFunction(Function &F) {
   if (skipFunction(F))
     return false;
bool HardwareLoops::TryConvertLoop(HardwareLoopInfo &HWLoopInfo) {
   HardwareLoop HWLoop(HWLoopInfo, *SE, *DL);
   HWLoop.Create();
   ++NumHWLoops;
   return true;
 }
 
 void HardwareLoop::Create() {
   LLVM_DEBUG(dbgs() << "HWLoops: Converting loop..\n");
-  BasicBlock *BeginBB = L->getLoopPreheader();
+ 
+  // Can we replace a conditional branch with an intrinsic that sets the
+  // loop counter and tests that is not zero?
+  auto CanGenerateTest = [](Loop *L) {
+    BasicBlock *Preheader = L->getLoopPreheader();
+    if (!Preheader->getSinglePredecessor())
+      return false;
+
+    BasicBlock *Pred = Preheader->getSinglePredecessor();
+    if (!isa<BranchInst>(Pred->getTerminator()))
+      return false;
+
+    return cast<BranchInst>(Pred->getTerminator())->isConditional();
+  };
+
+  UseLoopGuard = (UseLoopGuard || ForceGuardLoopEntry) && CanGenerateTest(L);
+  BasicBlock *Preheader = L->getLoopPreheader();
+  BasicBlock *BeginBB = UseLoopGuard ?
+    Preheader->getSinglePredecessor() : Preheader;
+
   Value *LoopCountInit = InitLoopCount(BeginBB);
   if (!LoopCountInit)
     return;
 
   InsertIterationSetup(LoopCountInit, BeginBB);
 
   if (UsePHICounter || ForceHardwareLoopPHI) {
     Instruction *LoopDec = InsertLoopRegDec(LoopCountInit);
     Value *EltsRem = InsertPHICounter(LoopCountInit, LoopDec);
     LoopDec->setOperand(0, EltsRem);
     UpdateBranch(LoopDec);
   } else
     InsertLoopDec();
 
   // Run through the basic blocks of the loop and see if any of them have dead
   // PHIs that can be removed.
   for (auto I : L->blocks())
     DeleteDeadPHIs(I);
 }
 
 Value *HardwareLoop::InitLoopCount(BasicBlock *BB) {
   SCEVExpander SCEVE(SE, DL, "loopcnt");
   if (!ExitCount->getType()->isPointerTy() &&
       ExitCount->getType() != CountType)
     ExitCount = SE.getZeroExtendExpr(ExitCount, CountType);
 
   ExitCount = SE.getAddExpr(ExitCount, SE.getOne(CountType));
 
   if (!isSafeToExpandAt(ExitCount, BB->getTerminator(), SE)) {
     LLVM_DEBUG(dbgs() << "HWLoops: Bailing, unsafe to expand ExitCount "
                << *ExitCount << "\n");
     return nullptr;
   }
 
   Value *Count = SCEVE.expandCodeFor(ExitCount, CountType,
                                      BB->getTerminator());
   LLVM_DEBUG(dbgs() << "HWLoops: Loop Count: " << *Count << "\n");
   return Count;
 }
 
 void HardwareLoop::InsertIterationSetup(Value *LoopCountInit,
                                         BasicBlock *BB) {
   IRBuilder<> Builder(BB->getTerminator());
   Type *Ty = LoopCountInit->getType();
-  Function *LoopIter =
-    Intrinsic::getDeclaration(M, Intrinsic::set_loop_iterations, Ty);
-  Builder.CreateCall(LoopIter, LoopCountInit);
+  Intrinsic::ID ID = UseLoopGuard ?
+    Intrinsic::test_set_loop_iterations : Intrinsic::set_loop_iterations;
+  Function *LoopIter = Intrinsic::getDeclaration(M, ID, Ty);
+  Value *SetCount = Builder.CreateCall(LoopIter, LoopCountInit);
+
+  // Use the return value of the intrinsic to control the entry of the loop.
+  if (UseLoopGuard) {
+    assert((isa<BranchInst>(BB->getTerminator()) &&
+            cast<BranchInst>(BB->getTerminator())->isConditional()) &&
+           "Expected conditional branch");
+    auto *LoopGuard = cast<BranchInst>(BB->getTerminator());
+    LoopGuard->setCondition(SetCount);
+    if (LoopGuard->getSuccessor(0) != L->getLoopPreheader())
+      LoopGuard->swapSuccessors();
+  }
 }
 
 void HardwareLoop::InsertLoopDec() {
   IRBuilder<> CondBuilder(ExitBranch);
 
   Function *DecFunc =
     Intrinsic::getDeclaration(M, Intrinsic::loop_decrement,
                               LoopDecrement->getType());
   Value *Ops[] = { LoopDecrement };
   Value *NewCond = CondBuilder.CreateCall(DecFunc, Ops);
   Value *OldCond = ExitBranch->getCondition();
   ExitBranch->setCondition(NewCond);
 
   // The false branch must exit the loop.
   if (!L->contains(ExitBranch->getSuccessor(0)))
     ExitBranch->swapSuccessors();
 
   // The old condition may be dead now, and may have even created a dead PHI

test/Transforms/HardwareLoops/scalar-while.ll

 ; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-DEC
 ; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-phi=true -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-REGDEC
 ; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-nested-hardware-loop=true -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-DEC --check-prefix=CHECK-NESTED
+; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-guard=true -S %s -o - | FileCheck %s --check-prefix=CHECK-GUARD
+; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-phi=true -force-hardware-loop-guard=true -S %s -o - | FileCheck %s --check-prefix=CHECK-GUARD
 
 ; CHECK-LABEL: while_lt
 define void @while_lt(i32 %i, i32 %N, i32* nocapture %A) {
 entry:
   %cmp4 = icmp ult i32 %i, %N
   br i1 %cmp4, label %while.body, label %while.end
 
+; CHECK-GUARD-LABEL: while_lt
+; CHECK-GUARD: entry:
+; CHECK-GUARD:   [[COUNT:%[^ ]+]] = sub i32 %N, %i
+; CHECK-GUARD:   [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-GUARD:   br i1 [[TEST]], label %while.body.preheader, label %while.end
+; CHECK-GUARD: while.body.preheader:
+; CHECK-GUARD:   br label %while.body
+
 ; CHECK: while.body.preheader:
 ; CHECK: [[COUNT:%[^ ]+]] = sub i32 %N, %i
 ; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
 ; CHECK: br label %while.body
 
 ; CHECK-REGDEC: [[REM:%[^ ]+]] = phi i32 [ [[COUNT]], %while.body.preheader ], [ [[LOOP_DEC:%[^ ]+]], %while.body ]
 ; CHECK-REGDEC: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK-REGDEC: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
while.body:
   %dec = add nsw i32 %i.addr.05, -1
   %cmp = icmp sgt i32 %dec, %N
   br i1 %cmp, label %while.body, label %while.end
 
 while.end:
   ret void
 }
 
+; CHECK-GUARD-LABEL: while_gte
+; CHECK-GUARD: entry:
+; CHECK-GUARD:  [[ADD:%[^ ]+]] = add i32 %i, 1
+; CHECK-GUARD:  [[SEL:%[^ ]+]] = icmp slt i32 %N, %i
+; CHECK-GUARD:  [[MIN:%[^ ]+]] = select i1 [[SEL]], i32 %N, i32 %i
+; CHECK-GUARD:  [[COUNT:%[^ ]+]] = sub i32 [[ADD]], [[MIN]]
+; CHECK-GUARD:  [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-GUARD:  br i1 [[TEST]], label %while.body.preheader, label %while.end
+; CHECK-GUARD: while.body.preheader:
+; CHECK-GUARD:   br label %while.body
+
 ; CHECK-LABEL: while_gte
 ; CHECK: while.body.preheader:
 ; CHECK: [[ADD:%[^ ]+]] = add i32 %i, 1
 ; CHECK: [[SEL:%[^ ]+]] = icmp slt i32 %N, %i
 ; CHECK: [[MIN:%[^ ]+]] = select i1 [[SEL]], i32 %N, i32 %i
 ; CHECK: [[COUNT:%[^ ]+]] = sub i32 [[ADD]], [[MIN]]
 ; CHECK: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
 ; CHECK: br label %while.body
while.body:
   %dec = add nsw i32 %i.addr.05, -1
   %cmp = icmp sgt i32 %i.addr.05, %N
   br i1 %cmp, label %while.body, label %while.end
 
 while.end:
   ret void
 }
 
+; CHECK-GUARD-LABEL: while_ne
+; CHECK-GUARD: entry:
+; CHECK-GUARD:   [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
+; CHECK-GUARD:   br i1 [[TEST]], label %while.body.preheader, label %while.end
+; CHECK-GUARD: while.body.preheader:
+; CHECK-GUARD:   br label %while.body
+define void @while_ne(i32 %N, i32* nocapture %A) {
+entry:
+  %cmp = icmp ne i32 %N, 0
+  br i1 %cmp, label %while.body, label %while.end
+
+while.body:
+  %i.addr.05 = phi i32 [ %inc, %while.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i.addr.05
+  store i32 %i.addr.05, i32* %arrayidx, align 4
+  %inc = add nuw i32 %i.addr.05, 1
+  %exitcond = icmp eq i32 %inc, %N
+  br i1 %exitcond, label %while.end, label %while.body
+
+while.end:
+  ret void
+}
+
+; CHECK-GUARD-LABEL: while_eq
+; CHECK-GUARD: entry:
+; CHECK-GUARD:   [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
+; CHECK-GUARD:   br i1 [[TEST]], label %while.body.preheader, label %while.end
+; CHECK-GUARD: while.body.preheader:
+; CHECK-GUARD:   br label %while.body
+define void @while_eq(i32 %N, i32* nocapture %A) {
+entry:
+  %cmp = icmp eq i32 %N, 0
+  br i1 %cmp, label %while.end, label %while.body
+
+while.body:
+  %i.addr.05 = phi i32 [ %inc, %while.body ], [ 0, %entry ]
+  %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i.addr.05
+  store i32 %i.addr.05, i32* %arrayidx, align 4
+  %inc = add nuw i32 %i.addr.05, 1
+  %exitcond = icmp eq i32 %inc, %N
+  br i1 %exitcond, label %while.end, label %while.body
+
+while.end:
+  ret void
+}
+
+; CHECK-GUARD-LABEL: while_preheader_eq
+; CHECK-GUARD: entry:
+; CHECK-GUARD:   br label %preheader
+; CHECK-GUARD: preheader:
+; CHECK-GUARD:   [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
+; CHECK-GUARD:   br i1 [[TEST]], label %while.body.preheader, label %while.end
+; CHECK-GUARD: while.body.preheader:
+; CHECK-GUARD:   br label %while.body
+define void @while_preheader_eq(i32 %N, i32* nocapture %A) {
+entry:
+  br label %preheader
+
+preheader:
+  %cmp = icmp eq i32 %N, 0
+  br i1 %cmp, label %while.end, label %while.body
+
+while.body:
+  %i.addr.05 = phi i32 [ %inc, %while.body ], [ 0, %preheader ]
+  %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i.addr.05
+  store i32 %i.addr.05, i32* %arrayidx, align 4
+  %inc = add nuw i32 %i.addr.05, 1
+  %exitcond = icmp eq i32 %inc, %N
+  br i1 %exitcond, label %while.end, label %while.body
+
+while.end:
+  ret void
+}
+
 ; CHECK-LABEL: nested
 ; CHECK-NESTED: call void @llvm.set.loop.iterations.i32(i32 %N)
 ; CHECK-NESTED: br label %while.cond1.preheader.us
 
 ; CHECK: call void @llvm.set.loop.iterations.i32(i32 %N)
 ; CHECK: br label %while.body3.us
 
 ; CHECK-DEC: [[LOOP_DEC:%[^ ]+]] = call i1 @llvm.loop.decrement.i32(i32 1)
 
 ; CHECK-REGDEC: [[REM:%[^ ]+]] = phi i32 [ %N, %while.cond1.preheader.us ], [ [[LOOP_DEC:%[^ ]+]], %while.body3.us ]
 ; CHECK-REGDEC: [[LOOP_DEC]] = call i32 @llvm.loop.decrement.reg.i32.i32.i32(i32 [[REM]], i32 1)
 ; CHECK-REGDEC: [[CMP:%[^ ]+]] = icmp ne i32 [[LOOP_DEC]], 0
 ; CHECK-REGDEC: br i1 [[CMP]], label %while.body3.us, label %while.cond1.while.end_crit_edge.us
 
 ; CHECK-NESTED: [[LOOP_DEC1:%[^ ]+]] = call i1 @llvm.loop.decrement.i32(i32 1)
 ; CHECK-NESTED: br i1 [[LOOP_DEC1]], label %while.cond1.preheader.us, label %while.end7
 
+; CHECK-GUARD: while.cond1.preheader.us:
+; CHECK-GUARD:   call void @llvm.set.loop.iterations.i32(i32 %N)
+; CHECK-GUARD:   br label %while.body3.us
+
 define void @nested(i32* nocapture %A, i32 %N) {
 entry:
   %cmp20 = icmp eq i32 %N, 0
   br i1 %cmp20, label %while.end7, label %while.cond1.preheader.us
 
 while.cond1.preheader.us:
   %i.021.us = phi i32 [ %inc6.us, %while.cond1.while.end_crit_edge.us ], [ 0, %entry ]
   %mul.us = mul i32 %i.021.us, %N

test/Transforms/HardwareLoops/unconditional-latch.ll

 ; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -hardware-loops -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-ALLOW
+; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -hardware-loops -force-hardware-loop-guard=true -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-GUARD
 ; RUN: opt -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-phi=true -hardware-loops -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-LATCH
 
 ; CHECK-LABEL: not_rotated
 ; CHECK-LATCH-NOT: call void @llvm.set.loop.iterations
 ; CHECK-LATCH-NOT: call i1 @llvm.loop.decrement
 
-; CHECK-ALLOW: call void @llvm.set.loop.iterations.i32(i32 %4)
-; CHECK-ALLOW: br label %10
+; CHECK-ALLOW: bb:
+; CHECK-ALLOW:   [[COUNT:%[^ ]+]] = add i32 %arg, 1
+; CHECK-ALLOW:   br label %bb3
+; CHECK-ALLOW: bb5:
+; CHECK-ALLOW:   call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-ALLOW:   br label %bb7
+
+; CHECK-GUARD: bb:
+; CHECK-GUARD:   [[COUNT:%[^ ]+]] = add i32 %arg, 1
+; CHECK-GUARD:   br label %bb3
+; CHECK-GUARD: bb3:
+; CHECK-GUARD:   [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 [[COUNT]])
+; CHECK-GUARD:   br i1 [[TEST]], label %bb5, label %bb18
 
 ; CHECK-ALLOW: [[CMP:%[^ ]+]] = call i1 @llvm.loop.decrement.i32(i32 1)
-; CHECK-ALLOW: br i1 [[CMP]], label %13, label %19
+; CHECK-ALLOW: br i1 [[CMP]], label %bb10, label %bb16
+define void @not_rotated(i32 %arg, i16* nocapture %arg1, i16 signext %arg2) {
+bb:
+  br label %bb3
+
+bb3:                                              ; preds = %bb16, %bb
+  %tmp = phi i32 [ 0, %bb ], [ %tmp17, %bb16 ]
+  %tmp4 = icmp eq i32 %tmp, %arg
+  br i1 %tmp4, label %bb18, label %bb5
+
+bb5:                                              ; preds = %bb3
+  %tmp6 = mul i32 %tmp, %arg
+  br label %bb7
+
+bb7:                                              ; preds = %bb10, %bb5
+  %tmp8 = phi i32 [ %tmp15, %bb10 ], [ 0, %bb5 ]
+  %tmp9 = icmp eq i32 %tmp8, %arg
+  br i1 %tmp9, label %bb16, label %bb10
+
+bb10:                                             ; preds = %bb7
+  %tmp11 = add i32 %tmp8, %tmp6
+  %tmp12 = getelementptr inbounds i16, i16* %arg1, i32 %tmp11
+  %tmp13 = load i16, i16* %tmp12, align 2
+  %tmp14 = add i16 %tmp13, %arg2
+  store i16 %tmp14, i16* %tmp12, align 2
+  %tmp15 = add i32 %tmp8, 1
+  br label %bb7
+
+bb16:                                             ; preds = %bb7
+  %tmp17 = add i32 %tmp, 1
+  br label %bb3
 
-define void @not_rotated(i32, i16* nocapture, i16 signext) {
-  br label %4
-
-4:
-  %5 = phi i32 [ 0, %3 ], [ %19, %18 ]
-  %6 = icmp eq i32 %5, %0
-  br i1 %6, label %20, label %7
-
-7:
-  %8 = mul i32 %5, %0
-  br label %9
-
-9:
-  %10 = phi i32 [ %17, %12 ], [ 0, %7 ]
-  %11 = icmp eq i32 %10, %0
-  br i1 %11, label %18, label %12
-
-12:
-  %13 = add i32 %10, %8
-  %14 = getelementptr inbounds i16, i16* %1, i32 %13
-  %15 = load i16, i16* %14, align 2
-  %16 = add i16 %15, %2
-  store i16 %16, i16* %14, align 2
-  %17 = add i32 %10, 1
-  br label %9
-
-18:
-  %19 = add i32 %5, 1
-  br label %4
-
-20:
+bb18:                                             ; preds = %bb3
   ret void
 }