[Loop Peeling] Add possibility to enable peeling on loop nests.

[ashlykov]

Summary

Current peeling implementation bails out in case of loop nests.
The patch introduces the field in TargetTransformInfo structure that
certain targets can use to relax the constraints if it's
profitable (disabled by default).
Also additional option is added to enable peeling manually for
experimenting and testing purposes.

[lebedev.ri]

+1 to the general direction - we *should* be peeling more, as discussed in
https://bugs.llvm.org/show_bug.cgi?id=43910 https://bugs.llvm.org/show_bug.cgi?id=43924
but i'm not sure i'm qualified to review this.

[fhahn]

Out of interest, do you have any performance & size numbers on the impact of this change?

llvm/lib/Transforms/Utils/LoopUnrollPeel.cpp

291–292

Update comment.

llvm/test/Transforms/LoopUnroll/peel-loop-conditions.ll

3

I think it would be better to add tests for this in a separate file.

[ashlykov]

@fhahn thanks for your comments, addressed!

Regarding the impact - it highly depends on further simplifications that can be applied to a code after the loop peeling (branch folding, better dependency analysis, reducing of register pressure inside the loop). And since the max number of iterations to be peeled is restricted - the codesize impact is manageable.

As a good case I can point to the nested loops in ImageMagick project (line 3346, https://github.com/ImageMagick/ImageMagick6/blob/master/magick/accelerate-kernels-private.h), I observed a significant gain on the kernel here.

[xbolva00]

llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp

158

enable by default?

or atleast for x86?

[fhahn]

llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp

158

I don't think that would be good idea, without at least some analysis on the impact on code size on targets turning this on. The cost-model currently assumes innermost loops exclusively.

[ashlykov]

llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp

158

The change is intended to be enabled on per-target basis via TTI where it's profitable. If someone could supply enough benchmarking for e.g. x86 target we can proceed further with enabling it/adjusting cost-model.

[ashlykov]

Hi @fhahn, do you have any more comments? Is it OK for trunk?

[ashlykov]

@fhahn Gentle reminder

[xbolva00]

Makes sense and no regressions expected since off by default.

[RKSimon]

@ashlykov This is failing on EXPENSIVE_CHECKS builds - revert? http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-debian/builds/1493/

[ashlykov]

Hi @RKSimon!

I'm trying to figure that out, the change is supposed to be off by default except the added test case and shouldn't affect anyone functionally.
What's LLVM policy in such cases - revert?

[RKSimon]

In D70304#1824307, @ashlykov wrote:

Hi @RKSimon!

I'm trying to figure that out, the change is supposed to be off by default except the added test case and shouldn't affect anyone functionally.
What's LLVM policy in such cases - revert?

Yes please revert if you don't have a fix, if the build stays red then later committers are less likely to notice if their commits cause extra builds fails.

[ashlykov]

@RKSimon thanks, reverted. The added test is failed with LLVM_ENABLE_EXPENSIVE_CHECKS on. Will re-commit after the fix.

[ashlykov]

Hi @xbolva00!

Could you please take a look on slightly updated changes? The previous version didn't preserve LoopInfo in case of new loops are created during a peeling, that causes a fail on EXPENSIVE_CHECKS builds. It's fixed now by recursively creating corresponding Loop objects by cloneLoop() utility function (moved from LoopUnswitch.cpp to LoopUtils.cpp to reuse).

[xbolva00]

Thanks, +1.

@fhahn ?

[ashlykov]

@fhahn, @xbolva00 Gentle reminder, the only change from the previously accepted revision is a fix of EXPENSIVE_CHECKS build failure (missing LoopInfo), change is off by default, can we land it?

[ashlykov]

Hi @xbolva00, @fhahn! Gentle ping, can we land the change with the fix added?
For some reasons the revision is marked in Phabricator as accepted even after the modification, I would like to have explicit confirmation that I can commit it.

llvm/include/llvm/Analysis/TargetTransformInfo.h

struct UnrollingPreferences {
     bool AllowExpensiveTripCount;
     /// Apply loop unroll on any kind of loop
     /// (mainly to loops that fail runtime unrolling).
     bool Force;
     /// Allow using trip count upper bound to unroll loops.
     bool UpperBound;
     /// Allow peeling off loop iterations.
     bool AllowPeeling;
+    /// Allow peeling off loop iterations for loop nests.
+    bool AllowLoopNestsPeeling;
     /// Allow unrolling of all the iterations of the runtime loop remainder.
     bool UnrollRemainder;
     /// Allow unroll and jam. Used to enable unroll and jam for the target.
     bool UnrollAndJam;
     /// Allow peeling basing on profile. Uses to enable peeling off all
     /// iterations basing on provided profile.
     /// If the value is true the peeling cost model can decide to peel only
     /// some iterations and in this case it will set this to false.

llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp

cl::desc("If the runtime tripcount for the loop is lower than the "
              "threshold, the loop is considered as flat and will be less "
              "aggressively unrolled."));
 
 static cl::opt<bool>
     UnrollAllowPeeling("unroll-allow-peeling", cl::init(true), cl::Hidden,
                        cl::desc("Allows loops to be peeled when the dynamic "
                                 "trip count is known to be low."));
 
+static cl::opt<bool> UnrollAllowLoopNestsPeeling(
+    "unroll-allow-loop-nests-peeling", cl::init(false), cl::Hidden,
+    cl::desc("Allows loop nests to be peeled."));
+
 static cl::opt<bool> UnrollUnrollRemainder(
   "unroll-remainder", cl::Hidden,
   cl::desc("Allow the loop remainder to be unrolled."));
 
 // This option isn't ever intended to be enabled, it serves to allow
 // experiments to check the assumptions about when this kind of revisit is
 // necessary.
 static cl::opt<bool> UnrollRevisitChildLoops(
TargetTransformInfo::UnrollingPreferences llvm::gatherUnrollingPreferences(
   UP.Partial = false;
   UP.Runtime = false;
   UP.AllowRemainder = true;
   UP.UnrollRemainder = false;
   UP.AllowExpensiveTripCount = false;
   UP.Force = false;
   UP.UpperBound = false;
   UP.AllowPeeling = true;
+  UP.AllowLoopNestsPeeling = false;
   UP.UnrollAndJam = false;
   UP.PeelProfiledIterations = true;
   UP.UnrollAndJamInnerLoopThreshold = 60;
 
   // Override with any target specific settings
   TTI.getUnrollingPreferences(L, SE, UP);
 
   // Apply size attributes
TargetTransformInfo::UnrollingPreferences llvm::gatherUnrollingPreferences(
   if (UnrollAllowRemainder.getNumOccurrences() > 0)
     UP.AllowRemainder = UnrollAllowRemainder;
   if (UnrollRuntime.getNumOccurrences() > 0)
     UP.Runtime = UnrollRuntime;
   if (UnrollMaxUpperBound == 0)
     UP.UpperBound = false;
   if (UnrollAllowPeeling.getNumOccurrences() > 0)
     UP.AllowPeeling = UnrollAllowPeeling;
+  if (UnrollAllowLoopNestsPeeling.getNumOccurrences() > 0)
+    UP.AllowLoopNestsPeeling = UnrollAllowLoopNestsPeeling;
   if (UnrollUnrollRemainder.getNumOccurrences() > 0)
     UP.UnrollRemainder = UnrollUnrollRemainder;
 
   // Apply user values provided by argument
   if (UserThreshold.hasValue()) {
     UP.Threshold = *UserThreshold;
     UP.PartialThreshold = *UserThreshold;
   }

llvm/lib/Transforms/Utils/LoopUnrollPeel.cpp

void llvm::computePeelCount(Loop *L, unsigned LoopSize,
                             unsigned &TripCount, ScalarEvolution &SE) {
   assert(LoopSize > 0 && "Zero loop size is not allowed!");
   // Save the UP.PeelCount value set by the target in
   // TTI.getUnrollingPreferences or by the flag -unroll-peel-count.
   unsigned TargetPeelCount = UP.PeelCount;
   UP.PeelCount = 0;
   if (!canPeel(L))
     return;
 
-  // Only try to peel innermost loops.
-  if (!L->empty())
+  // Only try to peel innermost loops by default.
+  // The constraint can be relaxed by the target in TTI.getUnrollingPreferences
+  // or by the flag -unroll-allow-loop-nests-peeling.
+  if (!UP.AllowLoopNestsPeeling && !L->empty())
     return;
 
   // If the user provided a peel count, use that.
   bool UserPeelCount = UnrollForcePeelCount.getNumOccurrences() > 0;
   if (UserPeelCount) {
     LLVM_DEBUG(dbgs() << "Force-peeling first " << UnrollForcePeelCount
                       << " iterations.\n");
     UP.PeelCount = UnrollForcePeelCount;

llvm/test/Transforms/LoopUnroll/peel-loop-conditions.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -S -loop-unroll -unroll-peel-max-count=4 -verify-dom-info | FileCheck %s
 
 declare void @f1()
 declare void @f2()
 
 ; Check that we can peel off iterations that make conditions true.
 define void @test1(i32 %k) {
 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  for.body.lr.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
for.inc:
   %inc = add nsw i32 %i.05, 1
   %cmp = icmp slt i32 %inc, %k
   br i1 %cmp, label %for.body, label %for.end
 
 for.end:
   ret void
 }
 
-; In this case we cannot peel the inner loop, because the condition involves
-; the outer induction variable.
-define void @test5(i32 %k) {
-; CHECK-LABEL: @test5(
-; CHECK-NEXT:  for.body.lr.ph:
-; CHECK-NEXT:    br label [[OUTER_HEADER:%.*]]
-; CHECK:       outer.header:
-; CHECK-NEXT:    [[J:%.*]] = phi i32 [ 0, [[FOR_BODY_LR_PH:%.*]] ], [ [[J_INC:%.*]], [[OUTER_INC:%.*]] ]
-; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
-; CHECK:       for.body:
-; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[OUTER_HEADER]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
-; CHECK-NEXT:    [[CMP1:%.*]] = icmp ult i32 [[J]], 2
-; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[IF_ELSE:%.*]]
-; CHECK:       if.then:
-; CHECK-NEXT:    call void @f1()
-; CHECK-NEXT:    br label [[FOR_INC]]
-; CHECK:       if.else:
-; CHECK-NEXT:    call void @f2()
-; CHECK-NEXT:    br label [[FOR_INC]]
-; CHECK:       for.inc:
-; CHECK-NEXT:    [[INC]] = add nsw i32 [[I_05]], 1
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC]], [[K:%.*]]
-; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[OUTER_INC]]
-; CHECK:       outer.inc:
-; CHECK-NEXT:    [[J_INC]] = add nsw i32 [[J]], 1
-; CHECK-NEXT:    [[OUTER_CMP:%.*]] = icmp slt i32 [[J_INC]], [[K]]
-; CHECK-NEXT:    br i1 [[OUTER_CMP]], label [[OUTER_HEADER]], label [[FOR_END:%.*]]
-; CHECK:       for.end:
-; CHECK-NEXT:    ret void
-;
-for.body.lr.ph:
-  br label %outer.header
-
-outer.header:
-  %j = phi i32 [ 0, %for.body.lr.ph ], [ %j.inc, %outer.inc ]
-  br label %for.body
-
-for.body:
-  %i.05 = phi i32 [ 0, %outer.header ], [ %inc, %for.inc ]
-  %cmp1 = icmp ult i32 %j, 2
-  br i1 %cmp1, label %if.then, label %if.else
-
-if.then:
-  call void @f1()
-  br label %for.inc
-
-if.else:
-  call void @f2()
-  br label %for.inc
-
-for.inc:
-  %inc = add nsw i32 %i.05, 1
-  %cmp = icmp slt i32 %inc, %k
-  br i1 %cmp, label %for.body, label %outer.inc
-
-outer.inc:
-  %j.inc = add nsw i32 %j, 1
-  %outer.cmp = icmp slt i32 %j.inc, %k
-  br i1 %outer.cmp, label %outer.header, label %for.end
-
-
-for.end:
-  ret void
-}
-
 ; In this test, the condition involves 2 AddRecs. Without evaluating both
 ; AddRecs, we cannot prove that the condition becomes known in the loop body
 ; after peeling.
-define void @test6(i32 %k) {
-; CHECK-LABEL: @test6(
+define void @test5(i32 %k) {
+; CHECK-LABEL: @test5(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
 ; CHECK-NEXT:    [[J:%.*]] = phi i32 [ 4, [[ENTRY]] ], [ [[J_INC:%.*]], [[FOR_INC]] ]
 ; CHECK-NEXT:    [[CMP1:%.*]] = icmp ult i32 [[I_05]], [[J]]
 ; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[IF_ELSE:%.*]]
 ; CHECK:       if.then:
for.inc:
   %j.inc = add nsw i32 %j, 1
   %cmp = icmp slt i32 %inc, %k
   br i1 %cmp, label %for.body, label %for.end
 
 for.end:
   ret void
 }
 
-define void @test7(i32 %k) {
-; CHECK-LABEL: @test7(
+define void @test6(i32 %k) {
+; CHECK-LABEL: @test6(
 ; CHECK-NEXT:  for.body.lr.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; CHECK:       for.body.peel.begin:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; CHECK:       for.body.peel:
 ; CHECK-NEXT:    [[CMP1_PEEL:%.*]] = icmp ne i32 0, 3
 ; CHECK-NEXT:    br i1 [[CMP1_PEEL]], label [[IF_THEN_PEEL:%.*]], label [[FOR_INC_PEEL:%.*]]
 ; CHECK:       if.then.peel:
for.inc:
   %inc = add nsw i32 %i.05, 1
   %cmp = icmp slt i32 %inc, %k
   br i1 %cmp, label %for.body, label %for.end
 
 for.end:
   ret void
 }
 
-define void @test8(i32 %k) {
-; CHECK-LABEL: @test8(
+define void @test7(i32 %k) {
+; CHECK-LABEL: @test7(
 ; CHECK-NEXT:  for.body.lr.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; CHECK:       for.body.peel.begin:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; CHECK:       for.body.peel:
 ; CHECK-NEXT:    [[CMP1_PEEL:%.*]] = icmp eq i32 0, 3
 ; CHECK-NEXT:    br i1 [[CMP1_PEEL]], label [[IF_THEN_PEEL:%.*]], label [[FOR_INC_PEEL:%.*]]
 ; CHECK:       if.then.peel:
for.inc:
   br i1 %cmp, label %for.body, label %for.end
 
 for.end:
   ret void
 }
 
 ; Comparison with non-monotonic predicate due to possible wrapping, loop
 ; body cannot be simplified.
-define void @test9(i32 %k) {
-; CHECK-LABEL: @test9(
+define void @test8(i32 %k) {
+; CHECK-LABEL: @test8(
 ; CHECK-NEXT:  for.body.lr.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[FOR_BODY_LR_PH:%.*]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
 ; CHECK-NEXT:    [[CMP1:%.*]] = icmp slt i32 [[I_05]], 3
 ; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[FOR_INC]]
 ; CHECK:       if.then:
 ; CHECK-NEXT:    call void @f1()
for.inc:
   %cmp = icmp slt i32 %inc, %k
   br i1 %cmp, label %for.body, label %for.end
 
 for.end:
   ret void
 }
 ; CHECK-NOT: llvm.loop.unroll.disable
 
-define void @test_10__peel_first_iter_via_slt_pred(i32 %len) {
-; CHECK-LABEL: @test_10__peel_first_iter_via_slt_pred(
+define void @test_9__peel_first_iter_via_slt_pred(i32 %len) {
+; CHECK-LABEL: @test_9__peel_first_iter_via_slt_pred(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[LEN:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP5]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; CHECK:       for.body.peel.begin:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; CHECK:       for.body.peel:

 
 if.end:                                           ; preds = %if.then, %for.body
   call void @sink()
   %inc = add nuw nsw i32 %i.06, 1
   %exitcond = icmp eq i32 %inc, %len
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-define void @test_11__peel_first_iter_via_sgt_pred(i32 %len) {
-; CHECK-LABEL: @test_11__peel_first_iter_via_sgt_pred(
+define void @test_10__peel_first_iter_via_sgt_pred(i32 %len) {
+; CHECK-LABEL: @test_10__peel_first_iter_via_sgt_pred(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[LEN:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP5]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; CHECK:       for.body.peel.begin:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; CHECK:       for.body.peel:
if.end:                                           ; preds = %if.then, %for.body
   call void @sink()
   %inc = add nuw nsw i32 %i.06, 1
   %exitcond = icmp eq i32 %inc, %len
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; NOTE: here we should only peel the first iteration,
 ;       i.e. all calls to sink() must stay in loop.
-define void @test12__peel_first_iter_via_eq_pred(i32 %len) {
-; CHECK-LABEL: @test12__peel_first_iter_via_eq_pred(
+define void @test11__peel_first_iter_via_eq_pred(i32 %len) {
+; CHECK-LABEL: @test11__peel_first_iter_via_eq_pred(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[LEN:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP5]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; CHECK:       for.body.peel.begin:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; CHECK:       for.body.peel:
if.end:                                           ; preds = %if.then, %for.body
   call void @sink()
   %inc = add nuw nsw i32 %i.06, 1
   %exitcond = icmp eq i32 %inc, %len
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; NOTE: here we should only peel the first iteration,
 ;       i.e. all calls to sink() must stay in loop.
-define void @test13__peel_first_iter_via_ne_pred(i32 %len) {
-; CHECK-LABEL: @test13__peel_first_iter_via_ne_pred(
+define void @test12__peel_first_iter_via_ne_pred(i32 %len) {
+; CHECK-LABEL: @test12__peel_first_iter_via_ne_pred(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[LEN:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP5]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL_BEGIN:%.*]]
 ; CHECK:       for.body.peel.begin:
 ; CHECK-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
 ; CHECK:       for.body.peel:

 if.end:                                           ; preds = %if.then, %for.body
   call void @sink()
   %inc = add nuw nsw i32 %i.06, 1
   %exitcond = icmp eq i32 %inc, %len
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; No peeling is profitable here.
-define void @test14__ivar_mod2_is_1(i32 %len) {
-; CHECK-LABEL: @test14__ivar_mod2_is_1(
+define void @test13__ivar_mod2_is_1(i32 %len) {
+; CHECK-LABEL: @test13__ivar_mod2_is_1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[LEN:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP5]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup.loopexit:
 ; CHECK-NEXT:    br label [[FOR_COND_CLEANUP]]
 ; CHECK:       for.cond.cleanup:

 if.end:                                           ; preds = %if.then, %for.body
   call void @sink()
   %inc = add nuw nsw i32 %i.06, 1
   %exitcond = icmp eq i32 %inc, %len
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
 ; No peeling is profitable here.
-define void @test15__ivar_mod2_is_0(i32 %len) {
-; CHECK-LABEL: @test15__ivar_mod2_is_0(
+define void @test14__ivar_mod2_is_0(i32 %len) {
+; CHECK-LABEL: @test14__ivar_mod2_is_0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP5:%.*]] = icmp sgt i32 [[LEN:%.*]], 0
 ; CHECK-NEXT:    br i1 [[CMP5]], label [[FOR_BODY_PREHEADER:%.*]], label [[FOR_COND_CLEANUP:%.*]]
 ; CHECK:       for.body.preheader:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.cond.cleanup.loopexit:
 ; CHECK-NEXT:    br label [[FOR_COND_CLEANUP]]
 ; CHECK:       for.cond.cleanup:

 
 if.end:                                           ; preds = %if.then, %for.body
   call void @sink()
   %inc = add nuw nsw i32 %i.06, 1
   %exitcond = icmp eq i32 %inc, %len
   br i1 %exitcond, label %for.cond.cleanup, label %for.body
 }
 
-; Similar to @test7, we need to peel one extra iteration, and we can't do that
+; Similar to @test6, we need to peel one extra iteration, and we can't do that
 ; as per the -unroll-peel-max-count=4, so this shouldn't be peeled at all.
-define void @test16(i32 %k) {
-; CHECK-LABEL: @test16(
+define void @test15(i32 %k) {
+; CHECK-LABEL: @test15(
 ; CHECK-NEXT:  for.body.lr.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[FOR_BODY_LR_PH:%.*]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
 ; CHECK-NEXT:    [[CMP1:%.*]] = icmp ne i32 [[I_05]], 4
 ; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[FOR_INC]]
 ; CHECK:       if.then:
 ; CHECK-NEXT:    call void @f1()
for.inc:
   %inc = add nsw i32 %i.05, 1
   %cmp = icmp slt i32 %inc, %k
   br i1 %cmp, label %for.body, label %for.end
 
 for.end:
   ret void
 }
 
-; Similar to @test8, we need to peel one extra iteration, and we can't do that
+; Similar to @test7, we need to peel one extra iteration, and we can't do that
 ; as per the -unroll-peel-max-count=4, so this shouldn't be peeled at all.
-define void @test17(i32 %k) {
-; CHECK-LABEL: @test17(
+define void @test16(i32 %k) {
+; CHECK-LABEL: @test16(
 ; CHECK-NEXT:  for.body.lr.ph:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[FOR_BODY_LR_PH:%.*]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
 ; CHECK-NEXT:    [[CMP1:%.*]] = icmp eq i32 [[I_05]], 4
 ; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[FOR_INC]]
 ; CHECK:       if.then:
 ; CHECK-NEXT:    call void @f1()

llvm/test/Transforms/LoopUnroll/peel-loop-nests.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -S -loop-unroll -unroll-peel-max-count=4 -verify-dom-info | FileCheck %s
+; RUN: opt < %s -S -loop-unroll -unroll-peel-max-count=4 -unroll-allow-loop-nests-peeling -verify-dom-info | FileCheck %s --check-prefix PEELED
+
+declare void @f1()
+declare void @f2()
+
+; In this case we cannot peel the inner loop, because the condition involves
+; the outer induction variable.
+; Peel the loop nest if allowed by the flag -unroll-allow-loop-nests-peeling.
+define void @test1(i32 %k) {
+; CHECK-LABEL: @test1(
+; CHECK-NEXT:  for.body.lr.ph:
+; CHECK-NEXT:    br label [[OUTER_HEADER:%.*]]
+; CHECK:       outer.header:
+; CHECK-NEXT:    [[J:%.*]] = phi i32 [ 0, [[FOR_BODY_LR_PH:%.*]] ], [ [[J_INC:%.*]], [[OUTER_INC:%.*]] ]
+; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
+; CHECK:       for.body:
+; CHECK-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[OUTER_HEADER]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
+; CHECK-NEXT:    [[CMP1:%.*]] = icmp ult i32 [[J]], 2
+; CHECK-NEXT:    br i1 [[CMP1]], label [[IF_THEN:%.*]], label [[IF_ELSE:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    call void @f1()
+; CHECK-NEXT:    br label [[FOR_INC]]
+; CHECK:       if.else:
+; CHECK-NEXT:    call void @f2()
+; CHECK-NEXT:    br label [[FOR_INC]]
+; CHECK:       for.inc:
+; CHECK-NEXT:    [[INC]] = add nsw i32 [[I_05]], 1
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC]], [[K:%.*]]
+; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[OUTER_INC]]
+; CHECK:       outer.inc:
+; CHECK-NEXT:    [[J_INC]] = add nsw i32 [[J]], 1
+; CHECK-NEXT:    [[OUTER_CMP:%.*]] = icmp slt i32 [[J_INC]], [[K]]
+; CHECK-NEXT:    br i1 [[OUTER_CMP]], label [[OUTER_HEADER]], label [[FOR_END:%.*]], !llvm.loop !{{.*}}
+; CHECK:       for.end:
+; CHECK-NEXT:    ret void
+;
+; PEELED-LABEL: @test1(
+; PEELED-NEXT:  for.body.lr.ph:
+; PEELED-NEXT:    br label [[OUTER_HEADER_PEEL_BEGIN:%.*]]
+; PEELED:       outer.header.peel.begin:
+; PEELED-NEXT:    br label [[OUTER_HEADER_PEEL:%.*]]
+; PEELED:       outer.header.peel:
+; PEELED-NEXT:    br label [[FOR_BODY_PEEL:%.*]]
+; PEELED:       for.body.peel:
+; PEELED-NEXT:    [[I_05_PEEL:%.*]] = phi i32 [ 0, [[OUTER_HEADER_PEEL]] ], [ [[INC_PEEL:%.*]], [[FOR_INC_PEEL:%.*]] ]
+; PEELED-NEXT:    [[CMP1_PEEL:%.*]] = icmp ult i32 0, 2
+; PEELED-NEXT:    br i1 [[CMP1_PEEL]], label [[IF_THEN_PEEL:%.*]], label [[IF_ELSE_PEEL:%.*]]
+; PEELED:       if.else.peel:
+; PEELED-NEXT:    call void @f2()
+; PEELED-NEXT:    br label [[FOR_INC_PEEL]]
+; PEELED:       if.then.peel:
+; PEELED-NEXT:    call void @f1()
+; PEELED-NEXT:    br label [[FOR_INC_PEEL]]
+; PEELED:       for.inc.peel:
+; PEELED-NEXT:    [[INC_PEEL]] = add nsw i32 [[I_05_PEEL]], 1
+; PEELED-NEXT:    [[CMP_PEEL:%.*]] = icmp slt i32 [[INC_PEEL]], [[K:%.*]]
+; PEELED-NEXT:    br i1 [[CMP_PEEL]], label [[FOR_BODY_PEEL]], label [[OUTER_INC_PEEL:%.*]]
+; PEELED:       outer.inc.peel:
+; PEELED-NEXT:    [[J_INC_PEEL:%.*]] = add nsw i32 0, 1
+; PEELED-NEXT:    [[OUTER_CMP_PEEL:%.*]] = icmp slt i32 [[J_INC_PEEL]], [[K]]
+; PEELED-NEXT:    br i1 [[OUTER_CMP_PEEL]], label [[OUTER_HEADER_PEEL_NEXT:%.*]], label [[FOR_END:%[^,]*]]
+; Verify that MD_loop metadata is dropped.
+; PEELED-NOT:   , !llvm.loop !{{[0-9]*}}
+; PEELED:       outer.header.peel.next:
+; PEELED-NEXT:    br label [[OUTER_HEADER_PEEL2:%.*]]
+; PEELED:       outer.header.peel2:
+; PEELED-NEXT:    br label [[FOR_BODY_PEEL3:%.*]]
+; PEELED:       for.body.peel3:
+; PEELED-NEXT:    [[I_05_PEEL4:%.*]] = phi i32 [ 0, [[OUTER_HEADER_PEEL2]] ], [ [[INC_PEEL9:%.*]], [[FOR_INC_PEEL8:%.*]] ]
+; PEELED-NEXT:    [[CMP1_PEEL5:%.*]] = icmp ult i32 [[J_INC_PEEL]], 2
+; PEELED-NEXT:    br i1 [[CMP1_PEEL5]], label [[IF_THEN_PEEL7:%.*]], label [[IF_ELSE_PEEL6:%.*]]
+; PEELED:       if.else.peel6:
+; PEELED-NEXT:    call void @f2()
+; PEELED-NEXT:    br label [[FOR_INC_PEEL8]]
+; PEELED:       if.then.peel7:
+; PEELED-NEXT:    call void @f1()
+; PEELED-NEXT:    br label [[FOR_INC_PEEL8]]
+; PEELED:       for.inc.peel8:
+; PEELED-NEXT:    [[INC_PEEL9]] = add nsw i32 [[I_05_PEEL4]], 1
+; PEELED-NEXT:    [[CMP_PEEL10:%.*]] = icmp slt i32 [[INC_PEEL9]], [[K]]
+; PEELED-NEXT:    br i1 [[CMP_PEEL10]], label [[FOR_BODY_PEEL3]], label [[OUTER_INC_PEEL11:%.*]]
+; PEELED:       outer.inc.peel11:
+; PEELED-NEXT:    [[J_INC_PEEL12:%.*]] = add nsw i32 [[J_INC_PEEL]], 1
+; PEELED-NEXT:    [[OUTER_CMP_PEEL13:%.*]] = icmp slt i32 [[J_INC_PEEL12]], [[K]]
+; PEELED-NEXT:    br i1 [[OUTER_CMP_PEEL13]], label [[OUTER_HEADER_PEEL_NEXT1:%.*]], label [[FOR_END]]
+; Verify that MD_loop metadata is dropped.
+; PEELED-NOT:   , !llvm.loop !{{[0-9]*}}
+; PEELED:       outer.header.peel.next1:
+; PEELED-NEXT:    br label [[OUTER_HEADER_PEEL_NEXT14:%.*]]
+; PEELED:       outer.header.peel.next14:
+; PEELED-NEXT:    br label [[FOR_BODY_LR_PH_PEEL_NEWPH:%.*]]
+; PEELED:       for.body.lr.ph.peel.newph:
+; PEELED-NEXT:    br label [[OUTER_HEADER:%.*]]
+; PEELED:       outer.header:
+; PEELED-NEXT:    [[J:%.*]] = phi i32 [ [[J_INC_PEEL12]], [[FOR_BODY_LR_PH_PEEL_NEWPH]] ], [ [[J_INC:%.*]], [[OUTER_INC:%.*]] ]
+; PEELED-NEXT:    br label [[FOR_BODY:%.*]]
+; PEELED:       for.body:
+; PEELED-NEXT:    [[I_05:%.*]] = phi i32 [ 0, [[OUTER_HEADER]] ], [ [[INC:%.*]], [[FOR_INC:%.*]] ]
+; PEELED-NEXT:    br i1 false, label [[IF_THEN:%.*]], label [[IF_ELSE:%.*]]
+; PEELED:       if.then:
+; PEELED-NEXT:    call void @f1()
+; PEELED-NEXT:    br label [[FOR_INC]]
+; PEELED:       if.else:
+; PEELED-NEXT:    call void @f2()
+; PEELED-NEXT:    br label [[FOR_INC]]
+; PEELED:       for.inc:
+; PEELED-NEXT:    [[INC]] = add nsw i32 [[I_05]], 1
+; PEELED-NEXT:    [[CMP:%.*]] = icmp slt i32 [[INC]], [[K]]
+; PEELED-NEXT:    br i1 [[CMP]], label [[FOR_BODY]], label [[OUTER_INC]]
+; PEELED:       outer.inc:
+; PEELED-NEXT:    [[J_INC]] = add nuw nsw i32 [[J]], 1
+; PEELED-NEXT:    [[OUTER_CMP:%.*]] = icmp slt i32 [[J_INC]], [[K]]
+; PEELED-NEXT:    br i1 [[OUTER_CMP]], label [[OUTER_HEADER]], label [[FOR_END_LOOPEXIT:%.*]], !llvm.loop !{{.*}}
+; PEELED:       for.end.loopexit:
+; PEELED-NEXT:    br label [[FOR_END]]
+; PEELED:       for.end:
+; PEELED-NEXT:    ret void
+;
+for.body.lr.ph:
+  br label %outer.header
+
+outer.header:
+  %j = phi i32 [ 0, %for.body.lr.ph ], [ %j.inc, %outer.inc ]
+  br label %for.body
+
+for.body:
+  %i.05 = phi i32 [ 0, %outer.header ], [ %inc, %for.inc ]
+  %cmp1 = icmp ult i32 %j, 2
+  br i1 %cmp1, label %if.then, label %if.else
+
+if.then:
+  call void @f1()
+  br label %for.inc
+
+if.else:
+  call void @f2()
+  br label %for.inc
+
+for.inc:
+  %inc = add nsw i32 %i.05, 1
+  %cmp = icmp slt i32 %inc, %k
+  br i1 %cmp, label %for.body, label %outer.inc
+
+outer.inc:
+  %j.inc = add nsw i32 %j, 1
+  %outer.cmp = icmp slt i32 %j.inc, %k
+  br i1 %outer.cmp, label %outer.header, label %for.end, !llvm.loop !0
+
+for.end:
+  ret void
+}
+
+!0 = distinct !{!0}
 No newline at end of file