[LoopUnroll] Eliminate PreserveCondBr parameter and fix a bug in the process

[reames]

Summary

This builds on D103584. The change eliminates the coupling between unroll heuristic and implementation w.r.t. knowing when the passed in trip count is an exact trip count or a max trip count. In theory the new code is slightly less powerful (since it relies on exact computable trip counts), but in practice, it appears to cover all the same cases. It can also be extended if needed.

The test change shows what appears to be a bug in the existing code around the interaction of peeling and unrolling. The original loop only ran 8 iterations. The previous output had the loop peeled by 2, and then an exact unroll of 8. This meant the loop ran a total of 10 iterations which appears to have been a miscompile.

[nikic]

llvm/lib/Transforms/Utils/LoopUnroll.cpp

380

I don't like this. This not resulting in regressions just means we have bad test coverage. I've added an extra test in https://reviews.llvm.org/rG33e41eaecdd7 that should fail after this change.

It would be better to base this on the exact trip count of ExitingBlock (below) and then generalize from there.

772

I think it would be better to clamp ULO.Count to MaxTripCount upfront, and avoid these special cases for MaxTripCount and ExactTripCount. It both makes the code simpler, and the unrolling result simpler.

This is actually already done at Effectively "DCE" unrolled iterations above, but we should do it based on the MaxTripCount, not ULO.TripCount.

[reames]

address review comment

[reames]

llvm/lib/Transforms/Utils/LoopUnroll.cpp

380

Good suggestion, incorporated.

JFYI, ExitingBI is on my list to kill, but not in this patch. :)

772

I'm not quite sure what you're asking for. It sounds like maybe you want me to change how many iterations we unroll when ULO.Count > MaxTripCount?

If so, I definitely request that be a separate patch. I'm trying to avoid large test diffs with each incremental patch and changing too much at once makes the test diffs really hard to understand.

[nikic]

LGTM

llvm/lib/Transforms/Utils/LoopUnroll.cpp

415

nit: Space after comma.

760

nit: Stray semicolon.

770

It may make sense to move the if (j == 0) return false; case to the start, as it's always the same. With that done, you should be able to drop the separate check for ExactUnroll, as the ExactTripCount case below should cover it.

772

Fair enough, I'm happy to have that in a followup. I mainly suggested it here because we would not have to worry about j >= MaxTripCount situations.

[reames]

llvm/lib/Transforms/Utils/LoopUnroll.cpp

770

Landed as is, then did this in cddcc4cf. This code is complicated enough that I want something easy to revert if this final style change turns out be buggy. :)

772

I'm definitely going to hold on doing that until after we split out peeling. I don't believe the current logic is even correct when there's a non-zero peel count. I don't want to build on it yet.

llvm/include/llvm/Transforms/Utils/UnrollLoop.h

 };
 
 struct UnrollLoopOptions {
   unsigned Count;
   unsigned TripCount;
   bool Force;
   bool AllowRuntime;
   bool AllowExpensiveTripCount;
-  bool PreserveCondBr;
   unsigned TripMultiple;
   unsigned PeelCount;
   bool UnrollRemainder;
   bool ForgetAllSCEV;
 };
 
 LoopUnrollResult UnrollLoop(Loop *L, UnrollLoopOptions ULO, LoopInfo *LI,
                             ScalarEvolution *SE, DominatorTree *DT,

llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp

static LoopUnrollResult tryToUnrollLoop(
   // Save loop properties before it is transformed.
   MDNode *OrigLoopID = L->getLoopID();
 
   // Unroll the loop.
   Loop *RemainderLoop = nullptr;
   LoopUnrollResult UnrollResult = UnrollLoop(
       L,
       {UP.Count, TripCount, UP.Force, UP.Runtime, UP.AllowExpensiveTripCount,
-       UseUpperBound, TripMultiple, PP.PeelCount, UP.UnrollRemainder,
-       ForgetAllSCEV},
+       TripMultiple, PP.PeelCount, UP.UnrollRemainder, ForgetAllSCEV},
       LI, &SE, &DT, &AC, &TTI, &ORE, PreserveLCSSA, &RemainderLoop);
   if (UnrollResult == LoopUnrollResult::Unmodified)
     return LoopUnrollResult::Unmodified;
 
   if (RemainderLoop) {
     Optional<MDNode *> RemainderLoopID =
         makeFollowupLoopID(OrigLoopID, {LLVMLoopUnrollFollowupAll,
                                         LLVMLoopUnrollFollowupRemainder});

llvm/lib/Transforms/Utils/LoopUnroll.cpp

void llvm::simplifyLoopAfterUnroll(Loop *L, bool SimplifyIVs, LoopInfo *LI,
   }
 }
 
 /// Unroll the given loop by Count. The loop must be in LCSSA form.  Unrolling
 /// can only fail when the loop's latch block is not terminated by a conditional
 /// branch instruction. However, if the trip count (and multiple) are not known,
 /// loop unrolling will mostly produce more code that is no faster.
 ///
-/// TripCount is the upper bound of the iteration on which control exits
-/// LatchBlock. Control may exit the loop prior to TripCount iterations either
-/// via an early branch in other loop block or via LatchBlock terminator. This
-/// is relaxed from the general definition of trip count which is the number of
-/// times the loop header executes. Note that UnrollLoop assumes that the loop
-/// counter test is in LatchBlock in order to remove unnecesssary instances of
-/// the test.  If control can exit the loop from the LatchBlock's terminator
-/// prior to TripCount iterations, flag PreserveCondBr needs to be set.
-///
-/// PreserveCondBr indicates whether the conditional branch of the LatchBlock
-/// needs to be preserved.  It is needed when we use trip count upper bound to
-/// fully unroll the loop.
+/// TripCount is an upper bound on the number of times the loop header runs.
+/// Note that the trip count does not need to be exact, it can be any upper
+/// bound on the true trip count.
 ///
 /// Similarly, TripMultiple divides the number of times that the LatchBlock may
 /// execute without exiting the loop.
 ///
 /// If AllowRuntime is true then UnrollLoop will consider unrolling loops that
 /// have a runtime (i.e. not compile time constant) trip count.  Unrolling these
 /// loops require a unroll "prologue" that runs "RuntimeTripCount % Count"
 /// iterations before branching into the unrolled loop.  UnrollLoop will not
if (ULO.TripCount == 0 && ULO.Count < 2 && ULO.PeelCount == 0) {
     LLVM_DEBUG(dbgs() << "Won't unroll; almost nothing to do\n");
     return LoopUnrollResult::Unmodified;
   }
 
   assert(ULO.Count > 0);
   assert(ULO.TripMultiple > 0);
   assert(ULO.TripCount == 0 || ULO.TripCount % ULO.TripMultiple == 0);
 
-  // Are we eliminating the loop control altogether?
-  bool CompletelyUnroll = ULO.Count == ULO.TripCount;
-
-  // We assume a run-time trip count if the compiler cannot
-  // figure out the loop trip count and the unroll-runtime
-  // flag is specified.
-  bool RuntimeTripCount =
-      (ULO.TripCount == 0 && ULO.Count > 0 && ULO.AllowRuntime);
-
-  assert((!RuntimeTripCount || !ULO.PeelCount) &&
-         "Did not expect runtime trip-count unrolling "
-         "and peeling for the same loop");
 
   bool Peeled = false;
   if (ULO.PeelCount) {
     Peeled = peelLoop(L, ULO.PeelCount, LI, SE, DT, AC, PreserveLCSSA);
 
     // Successful peeling may result in a change in the loop preheader/trip
     // counts. If we later unroll the loop, we want these to be updated.
     if (Peeled) {
       // According to our guards and profitability checks the only
       // meaningful exit should be latch block. Other exits go to deopt,
       // so we do not worry about them.
       BasicBlock *ExitingBlock = L->getLoopLatch();
       assert(ExitingBlock && "Loop without exiting block?");
       assert(L->isLoopExiting(ExitingBlock) && "Latch is not exiting?");
       ULO.TripCount = SE->getSmallConstantTripCount(L, ExitingBlock);
       ULO.TripMultiple = SE->getSmallConstantTripMultiple(L, ExitingBlock);
     }
   }
 
+  // Are we eliminating the loop control altogether?  Note that we can know
+  // we're eliminating the backedge without knowing exactly which iteration
+  // of the unrolled body exits.
+  const bool CompletelyUnroll = ULO.Count == ULO.TripCount;
+
+  // We assume a run-time trip count if the compiler cannot
+  // figure out the loop trip count and the unroll-runtime
+  // flag is specified.
+  bool RuntimeTripCount =
+      (ULO.TripCount == 0 && ULO.Count > 0 && ULO.AllowRuntime);
+
+  assert((!RuntimeTripCount || !ULO.PeelCount) &&
+         "Did not expect runtime trip-count unrolling "
+         "and peeling for the same loop");
+
   // All these values should be taken only after peeling because they might have
   // changed.
   BasicBlock *Preheader = L->getLoopPreheader();
   BasicBlock *Header = L->getHeader();
   BasicBlock *LatchBlock = L->getLoopLatch();
   SmallVector<BasicBlock *, 4> ExitBlocks;
   L->getExitBlocks(ExitBlocks);
   std::vector<BasicBlock *> OriginalLoopBlocks = L->getBlocks();
 
   // Go through all exits of L and see if there are any phi-nodes there. We just
   // conservatively assume that they're inserted to preserve LCSSA form, which
   // means that complete unrolling might break this form. We need to either fix
   // it in-place after the transformation, or entirely rebuild LCSSA. TODO: For
   // now we just recompute LCSSA for the outer loop, but it should be possible
   // to fix it in-place.
   bool NeedToFixLCSSA =
       PreserveLCSSA && CompletelyUnroll &&
       any_of(ExitBlocks,
              [](const BasicBlock *BB) { return isa<PHINode>(BB->begin()); });
 
   const unsigned MaxTripCount = SE->getSmallConstantMaxTripCount(L);
   const bool MaxOrZero = SE->isBackedgeTakenCountMaxOrZero(L);
 
   const bool PreserveOnlyFirst = ULO.Count == MaxTripCount && MaxOrZero;
 
   // The current loop unroll pass can unroll loops that have
   // (1) single latch; and
   // (2a) latch is unconditional; or
   // (2b) latch is conditional and is an exiting block
   // FIXME: The implementation can be extended to work with more complicated
   // cases, e.g. loops with multiple latches.
   BranchInst *LatchBI = dyn_cast<BranchInst>(LatchBlock->getTerminator());
LoopUnrollResult llvm::UnrollLoop(Loop *L, UnrollLoopOptions ULO, LoopInfo *LI,
   LLVM_DEBUG({
     if (ExitingBI)
       dbgs() << "  Exiting Block = " << ExitingBI->getParent()->getName()
              << "\n";
     else
       dbgs() << "  No single exiting block\n";
   });
 
+  const unsigned ExactTripCount = ExitingBI ?
+    SE->getSmallConstantTripCount(L,ExitingBI->getParent()) : 0;
+  const bool ExactUnroll = (ExactTripCount && ExactTripCount == ULO.Count);
+
   // Loops containing convergent instructions must have a count that divides
   // their TripMultiple.
   LLVM_DEBUG(
       {
         bool HasConvergent = false;
         for (auto &BB : L->blocks())
           for (auto &I : *BB)
             if (auto *CB = dyn_cast<CallBase>(&I))
auto SetDest = [&](BasicBlock *Src, bool WillExit, bool ExitOnTrue) {
       BranchInst::Create(Dest, Term);
       Term->eraseFromParent();
 
       DTU.applyUpdates({{DominatorTree::Delete, Src, DeadSucc}});
     };
 
     auto WillExit = [&](unsigned i, unsigned j) -> Optional<bool> {
       if (CompletelyUnroll) {
-        if (ULO.PreserveCondBr && j && !(PreserveOnlyFirst && i != 0))
+        ;
+        if (PreserveOnlyFirst) {
+          if (i == 0)
             return None;
           return j == 0;
         }
+        if (ExactUnroll)
+          return j == 0;
+        // Full, but non-exact unrolling
+        if (j == 0)
+          return true;
+        if (MaxTripCount && j >= MaxTripCount)
+          return false;
+        if (ExactTripCount && j != ExactTripCount)
+          return false;
+        return None;
+      }
 
       if (RuntimeTripCount && j != 0)
         return false;
 
       if (j != BreakoutTrip &&
           (ULO.TripMultiple == 0 || j % ULO.TripMultiple != 0)) {
         // If we know the trip count or a multiple of it, we can safely use an
         // unconditional branch for some iterations.

llvm/lib/Transforms/Utils/LoopUnrollRuntime.cpp

#endif
 
   auto UnrollResult = LoopUnrollResult::Unmodified;
   if (remainderLoop && UnrollRemainder) {
     LLVM_DEBUG(dbgs() << "Unrolling remainder loop\n");
     UnrollResult =
         UnrollLoop(remainderLoop,
                    {/*Count*/ Count - 1, /*TripCount*/ Count - 1,
                     /*Force*/ false, /*AllowRuntime*/ false,
-                    /*AllowExpensiveTripCount*/ false, /*PreserveCondBr*/ true,
-                    /*TripMultiple*/ 1,
+                    /*AllowExpensiveTripCount*/ false, /*TripMultiple*/ 1,
                     /*PeelCount*/ 0, /*UnrollRemainder*/ false, ForgetAllSCEV},
                    LI, SE, DT, AC, TTI, /*ORE*/ nullptr, PreserveLCSSA);
   }
 
   if (ResultLoop && UnrollResult != LoopUnrollResult::FullyUnrolled)
     *ResultLoop = remainderLoop;
   NumRuntimeUnrolled++;
   return true;
 }

llvm/test/Transforms/LoopUnroll/pr45939-peel-count-and-complete-unroll.ll

 ; PEEL2-NEXT:    br i1 [[EXITCOND_PEEL5]], label [[FOR_BODY_PEEL_NEXT1:%.*]], label [[FOR_EXIT]]
 ; PEEL2:       for.body.peel.next1:
 ; PEEL2-NEXT:    br label [[FOR_BODY_PEEL_NEXT6:%.*]]
 ; PEEL2:       for.body.peel.next6:
 ; PEEL2-NEXT:    br label [[ENTRY_PEEL_NEWPH:%.*]]
 ; PEEL2:       entry.peel.newph:
 ; PEEL2-NEXT:    br label [[FOR_BODY:%.*]]
 ; PEEL2:       for.body:
-; PEEL2-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_PEEL4]]
-; PEEL2-NEXT:    [[TMP2:%.*]] = trunc i64 [[INDVARS_IV_NEXT_PEEL4]] to i32
+; PEEL2-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT_PEEL4]], [[ENTRY_PEEL_NEWPH]] ], [ [[INDVARS_IV_NEXT_7:%.*]], [[FOR_BODY_6:%.*]] ]
+; PEEL2-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV]]
+; PEEL2-NEXT:    [[TMP2:%.*]] = trunc i64 [[INDVARS_IV]] to i32
 ; PEEL2-NEXT:    store i32 [[TMP2]], i32* [[ARRAYIDX]], align 4
-; PEEL2-NEXT:    store i32 3, i32* getelementptr inbounds ([8 x i32], [8 x i32]* @a, i64 0, i64 3), align 4
-; PEEL2-NEXT:    store i32 4, i32* getelementptr inbounds ([8 x i32], [8 x i32]* @a, i64 0, i64 4), align 4
-; PEEL2-NEXT:    store i32 5, i32* getelementptr inbounds ([8 x i32], [8 x i32]* @a, i64 0, i64 5), align 4
-; PEEL2-NEXT:    store i32 6, i32* getelementptr inbounds ([8 x i32], [8 x i32]* @a, i64 0, i64 6), align 4
-; PEEL2-NEXT:    store i32 7, i32* getelementptr inbounds ([8 x i32], [8 x i32]* @a, i64 0, i64 7), align 4
-; PEEL2-NEXT:    store i32 8, i32* getelementptr inbounds ([8 x i32], [8 x i32]* @a, i64 1, i64 0), align 4
-; PEEL2-NEXT:    store i32 9, i32* getelementptr ([8 x i32], [8 x i32]* @a, i64 1, i64 1), align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; PEEL2-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT]]
+; PEEL2-NEXT:    [[TMP3:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; PEEL2-NEXT:    store i32 [[TMP3]], i32* [[ARRAYIDX_1]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_1:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT]], 1
+; PEEL2-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_1]]
+; PEEL2-NEXT:    [[TMP4:%.*]] = trunc i64 [[INDVARS_IV_NEXT_1]] to i32
+; PEEL2-NEXT:    store i32 [[TMP4]], i32* [[ARRAYIDX_2]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_2:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_1]], 1
+; PEEL2-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_2]]
+; PEEL2-NEXT:    [[TMP5:%.*]] = trunc i64 [[INDVARS_IV_NEXT_2]] to i32
+; PEEL2-NEXT:    store i32 [[TMP5]], i32* [[ARRAYIDX_3]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_3:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_2]], 1
+; PEEL2-NEXT:    [[ARRAYIDX_4:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_3]]
+; PEEL2-NEXT:    [[TMP6:%.*]] = trunc i64 [[INDVARS_IV_NEXT_3]] to i32
+; PEEL2-NEXT:    store i32 [[TMP6]], i32* [[ARRAYIDX_4]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_4:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_3]], 1
+; PEEL2-NEXT:    [[ARRAYIDX_5:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_4]]
+; PEEL2-NEXT:    [[TMP7:%.*]] = trunc i64 [[INDVARS_IV_NEXT_4]] to i32
+; PEEL2-NEXT:    store i32 [[TMP7]], i32* [[ARRAYIDX_5]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_5:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_4]], 1
+; PEEL2-NEXT:    [[EXITCOND_5:%.*]] = icmp ne i64 [[INDVARS_IV_NEXT_5]], 8
+; PEEL2-NEXT:    br i1 [[EXITCOND_5]], label [[FOR_BODY_6]], label [[FOR_EXIT_LOOPEXIT:%.*]], !llvm.loop [[LOOP0:![0-9]+]]
+; PEEL2:       for.exit.loopexit:
 ; PEEL2-NEXT:    br label [[FOR_EXIT]]
 ; PEEL2:       for.exit:
 ; PEEL2-NEXT:    ret void
+; PEEL2:       for.body.6:
+; PEEL2-NEXT:    [[ARRAYIDX_6:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_5]]
+; PEEL2-NEXT:    [[TMP8:%.*]] = trunc i64 [[INDVARS_IV_NEXT_5]] to i32
+; PEEL2-NEXT:    store i32 [[TMP8]], i32* [[ARRAYIDX_6]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_6:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_5]], 1
+; PEEL2-NEXT:    [[ARRAYIDX_7:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_6]]
+; PEEL2-NEXT:    [[TMP9:%.*]] = trunc i64 [[INDVARS_IV_NEXT_6]] to i32
+; PEEL2-NEXT:    store i32 [[TMP9]], i32* [[ARRAYIDX_7]], align 4
+; PEEL2-NEXT:    [[INDVARS_IV_NEXT_7]] = add nuw nsw i64 [[INDVARS_IV_NEXT_6]], 1
+; PEEL2-NEXT:    br label [[FOR_BODY]], !llvm.loop [[LOOP2:![0-9]+]]
 ;
 ; PEEL8-LABEL: @test1(
 ; PEEL8-NEXT:  entry:
 ; PEEL8-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; PEEL8:       for.body.peel.begin:
 ; PEEL8-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; PEEL8:       for.body.peel:
 ; PEEL8-NEXT:    [[ARRAYIDX_PEEL:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 0

 ; PEEL8-NEXT:    br i1 [[EXITCOND_PEEL35]], label [[FOR_BODY_PEEL_NEXT31:%.*]], label [[FOR_EXIT]]
 ; PEEL8:       for.body.peel.next31:
 ; PEEL8-NEXT:    br label [[FOR_BODY_PEEL_NEXT36:%.*]]
 ; PEEL8:       for.body.peel.next36:
 ; PEEL8-NEXT:    br label [[ENTRY_PEEL_NEWPH:%.*]]
 ; PEEL8:       entry.peel.newph:
 ; PEEL8-NEXT:    br label [[FOR_BODY:%.*]]
 ; PEEL8:       for.body:
-; PEEL8-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_PEEL34]]
-; PEEL8-NEXT:    [[TMP8:%.*]] = trunc i64 [[INDVARS_IV_NEXT_PEEL34]] to i32
+; PEEL8-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ [[INDVARS_IV_NEXT_PEEL34]], [[ENTRY_PEEL_NEWPH]] ], [ [[INDVARS_IV_NEXT_7:%.*]], [[FOR_BODY_7:%.*]] ]
+; PEEL8-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV]]
+; PEEL8-NEXT:    [[TMP8:%.*]] = trunc i64 [[INDVARS_IV]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP8]], i32* [[ARRAYIDX]], align 4
-; PEEL8-NEXT:    [[INDVARS_IV_NEXT:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_PEEL34]], 1
+; PEEL8-NEXT:    [[INDVARS_IV_NEXT:%.*]] = add nuw nsw i64 [[INDVARS_IV]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_1:%.*]], label [[FOR_EXIT_LOOPEXIT:%.*]], !llvm.loop [[LOOP0:![0-9]+]]
+; PEEL8:       for.exit.loopexit:
+; PEEL8-NEXT:    br label [[FOR_EXIT]]
+; PEEL8:       for.exit:
+; PEEL8-NEXT:    ret void
+; PEEL8:       for.body.1:
 ; PEEL8-NEXT:    [[ARRAYIDX_1:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT]]
 ; PEEL8-NEXT:    [[TMP9:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP9]], i32* [[ARRAYIDX_1]], align 4
 ; PEEL8-NEXT:    [[INDVARS_IV_NEXT_1:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_2:%.*]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP0]]
+; PEEL8:       for.body.2:
 ; PEEL8-NEXT:    [[ARRAYIDX_2:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_1]]
 ; PEEL8-NEXT:    [[TMP10:%.*]] = trunc i64 [[INDVARS_IV_NEXT_1]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP10]], i32* [[ARRAYIDX_2]], align 4
 ; PEEL8-NEXT:    [[INDVARS_IV_NEXT_2:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_1]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_3:%.*]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP0]]
+; PEEL8:       for.body.3:
 ; PEEL8-NEXT:    [[ARRAYIDX_3:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_2]]
 ; PEEL8-NEXT:    [[TMP11:%.*]] = trunc i64 [[INDVARS_IV_NEXT_2]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP11]], i32* [[ARRAYIDX_3]], align 4
 ; PEEL8-NEXT:    [[INDVARS_IV_NEXT_3:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_2]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_4:%.*]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP0]]
+; PEEL8:       for.body.4:
 ; PEEL8-NEXT:    [[ARRAYIDX_4:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_3]]
 ; PEEL8-NEXT:    [[TMP12:%.*]] = trunc i64 [[INDVARS_IV_NEXT_3]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP12]], i32* [[ARRAYIDX_4]], align 4
 ; PEEL8-NEXT:    [[INDVARS_IV_NEXT_4:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_3]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_5:%.*]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP0]]
+; PEEL8:       for.body.5:
 ; PEEL8-NEXT:    [[ARRAYIDX_5:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_4]]
 ; PEEL8-NEXT:    [[TMP13:%.*]] = trunc i64 [[INDVARS_IV_NEXT_4]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP13]], i32* [[ARRAYIDX_5]], align 4
 ; PEEL8-NEXT:    [[INDVARS_IV_NEXT_5:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_4]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_6:%.*]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP0]]
+; PEEL8:       for.body.6:
 ; PEEL8-NEXT:    [[ARRAYIDX_6:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_5]]
 ; PEEL8-NEXT:    [[TMP14:%.*]] = trunc i64 [[INDVARS_IV_NEXT_5]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP14]], i32* [[ARRAYIDX_6]], align 4
 ; PEEL8-NEXT:    [[INDVARS_IV_NEXT_6:%.*]] = add nuw nsw i64 [[INDVARS_IV_NEXT_5]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY_7]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP0]]
+; PEEL8:       for.body.7:
 ; PEEL8-NEXT:    [[ARRAYIDX_7:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 [[INDVARS_IV_NEXT_6]]
 ; PEEL8-NEXT:    [[TMP15:%.*]] = trunc i64 [[INDVARS_IV_NEXT_6]] to i32
 ; PEEL8-NEXT:    store i32 [[TMP15]], i32* [[ARRAYIDX_7]], align 4
-; PEEL8-NEXT:    br label [[FOR_EXIT]]
-; PEEL8:       for.exit:
-; PEEL8-NEXT:    ret void
+; PEEL8-NEXT:    [[INDVARS_IV_NEXT_7]] = add nuw nsw i64 [[INDVARS_IV_NEXT_6]], 1
+; PEEL8-NEXT:    br i1 true, label [[FOR_BODY]], label [[FOR_EXIT_LOOPEXIT]], !llvm.loop [[LOOP2:![0-9]+]]
 ;
 ; PEEL2UNROLL2-LABEL: @test1(
 ; PEEL2UNROLL2-NEXT:  entry:
 ; PEEL2UNROLL2-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; PEEL2UNROLL2:       for.body.peel.begin:
 ; PEEL2UNROLL2-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; PEEL2UNROLL2:       for.body.peel:
 ; PEEL2UNROLL2-NEXT:    [[ARRAYIDX_PEEL:%.*]] = getelementptr inbounds [8 x i32], [8 x i32]* @a, i64 0, i64 0