[NFCI][LoopUnrollAndJam] Changing LoopUnrollAndJamPass to a function pass.

[Whitney]

Summary

This patch changes LoopUnrollAndJamPass to a function pass, and keeps the loops traversal order same as defined in FunctionToLoopPassAdaptor LoopPassManager.h.

The next patch will change the loop traversal to outer to inner order, so more loops can be transform.

Discussion in llvm-dev mailing list: https://groups.google.com/forum/#!topic/llvm-dev/LF4rUjkVI2g

[dmgreen]

Thanks.

This is only the new pass manager, right? It's probably best to keep the old pass manager working in the same way too, to keep them in sync. That will be the one that we use downstream at the mo.

[Whitney]

Changing LoopUnrollAndJamPass to a function pass for old pass manager as well.

[dmgreen]

llvm/lib/Transforms/Scalar/LoopUnrollAndJamPass.cpp

440

What if the function has no loops?

[Whitney]

Addressed review comment.

[dmgreen]

If UnJ needs loops in LoopSimplify form then it should request (or enforce) that loops are in that form. I think it can be done with AU.addRequiredID(LoopSimplifyID); (and maybe a DEPENDENCY on LoopSimplify?) That should hopefully mean that the tests don't need these adjustments.

Do we have any tests for the new pass manager? I feel we would have added some, but I don't see any here.

llvm/lib/Transforms/Scalar/LoopUnrollAndJamPass.cpp

441

I feel like we may need LCSSA on the outermost loops before calling UnJ on the inner one, from the loop of the check on line 576 of LoopUnrollAndJam.cpp. I'm not sure though, I didn't see it crash anywhere, but we likely don't have a good testcase for this.

[Whitney]

In D72230#1808620, @dmgreen wrote:

If UnJ needs loops in LoopSimplify form then it should request (or enforce) that loops are in that form. I think it can be done with AU.addRequiredID(LoopSimplifyID); (and maybe a DEPENDENCY on LoopSimplify?) That should hopefully mean that the tests don't need these adjustments.

I am not familiar with addRequiredID, do you know if there is an equivalent in the new pass manager?

Do we have any tests for the new pass manager? I feel we would have added some, but I don't see any here.

I don't see any LIT tests using the new pass manager in llvm/test/Transforms/LoopUnrollAndJam, is it in some other folder I missed?

[Whitney]

llvm/lib/Transforms/Scalar/LoopUnrollAndJamPass.cpp

441

I am thinking that we can treat LCSSA the same as loop simplify, if the loop is not in LCSSA form, then give up. And we can run LoopSimplify pass and LCSSA pass before unroll and jam pass. What do you think?

[dmgreen]

In D72230#1808657, @Whitney wrote:

In D72230#1808620, @dmgreen wrote:

If UnJ needs loops in LoopSimplify form then it should request (or enforce) that loops are in that form. I think it can be done with AU.addRequiredID(LoopSimplifyID); (and maybe a DEPENDENCY on LoopSimplify?) That should hopefully mean that the tests don't need these adjustments.

I am not familiar with addRequiredID, do you know if there is an equivalent in the new pass manager?

Do we have any tests for the new pass manager? I feel we would have added some, but I don't see any here.

I don't see any LIT tests using the new pass manager in llvm/test/Transforms/LoopUnrollAndJam, is it in some other folder I missed?

Not sure I'm afraid. I'm not much of an expert on the new pass manager (whenever I've tried it here, it's made performance worse and codesize bigger! So haven't used it much)

Please add the tests if it's simple enough. If it looks like it might be trouble, feel free to leave that to another patch.

llvm/lib/Transforms/Scalar/LoopUnrollAndJamPass.cpp

441

Sounds good. I think (but may be mistaken) that LCSSA can't "fail" to be created in the same way as LoopSimplify can. It looks like we can request them and mark them as preserved in the same way, though.

[Whitney]

In D72230#1808620, @dmgreen wrote:

If UnJ needs loops in LoopSimplify form then it should request (or enforce) that loops are in that form. I think it can be done with AU.addRequiredID(LoopSimplifyID); (and maybe a DEPENDENCY on LoopSimplify?) That should hopefully mean that the tests don't need these adjustments.

In https://groups.google.com/forum/m/?fbclid=IwAR3jQc7xQaWv2ovAvFTNuZXFKvCBo5lRx-vt_kyomdmcBgYpcRy5YcaDa0s#!topic/llvm-dev/0K8gEZ22VOE,
You cannot reliably use the addRequired() method to force a transform pass to run before another pass
The addRequired method is designed to tell the PassManager which *analysis* passes to run before your pass.

The discussion in the link above sounds like using addRequiredID on a transformation pass is not a good idea. Beside that, I don't know a way to require transformations to run before a pass in the new pass manager.
I suggested to give up on loops that are not simplified or loop nests that are not in LCSSA. And change the LITs to be all in simplified and LCSSA form. In additional, add the two transformation passes before LoopUnrollAndJamPass in the pipeline. What do you think?

[dmgreen]

In D72230#1809519, @Whitney wrote:

The discussion in the link above sounds like using addRequiredID on a transformation pass is not a good idea. Beside that, I don't know a way to require transformations to run before a pass in the new pass manager.
I suggested to give up on loops that are not simplified or loop nests that are not in LCSSA. And change the LITs to be all in simplified and LCSSA form. In additional, add the two transformation passes before LoopUnrollAndJamPass in the pipeline. What do you think?

I would agree that depending on other transform passes is a little odd, but it seems to work elsewhere. How do other passes work? LoopLoadElimination seems to be a FunctionPass that requires loops be in LoopSimplifyForm. I see it using INITIALIZE_PASS_DEPENDENCY(LoopSimplify) and AU.addRequiredID(LoopSimplifyID);

The LoopUnrollPass in the new pass manager seems to be a function pass now, and has this (same for the vectorizer):

// The unroller requires loops to be in simplified form, and also needs LCSSA.
// Since simplification may add new inner loops, it has to run before the
// legality and profitability checks. This means running the loop unroller
// will simplify all loops, regardless of whether anything end up being
// unrolled.
for (auto &L : LI) {
  Changed |=
      simplifyLoop(L, &DT, &LI, &SE, &AC, nullptr, false /* PreserveLCSSA */);
  Changed |= formLCSSARecursively(*L, DT, &LI, &SE);
}

Can we steal that? I would prefer it if this was mostly automatic, not having to update the tests or be careful to run passes in the right order every time.

From poking around, you may also need -aa-pipeline=type-based-aa,basic-aa to some of the tests if you run them under the new pass manager. And the order of addRequiredID seems to matter to get it to work.

[Whitney]

1. Makes loops in simplified and LCSSA form in the beginning of LoopUnrollAndJamPass.
2. Add LITs for new pass manager.

[dmgreen]

LGTM. Thanks.

llvm/include/llvm/Transforms/Scalar/LoopUnrollAndJamPass.h

 #define LLVM_TRANSFORMS_SCALAR_LOOPUNROLLANDJAMPASS_H
 
 #include "llvm/Analysis/LoopAnalysisManager.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/IR/PassManager.h"
 
 namespace llvm {
 
-class Loop;
-struct LoopStandardAnalysisResults;
-class LPMUpdater;
+class Function;
 
 /// A simple loop rotation transformation.
 class LoopUnrollAndJamPass : public PassInfoMixin<LoopUnrollAndJamPass> {
   const int OptLevel;
 
 public:
   explicit LoopUnrollAndJamPass(int OptLevel = 2) : OptLevel(OptLevel) {}
-  PreservedAnalyses run(Loop &L, LoopAnalysisManager &AM,
-                        LoopStandardAnalysisResults &AR, LPMUpdater &U);
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_SCALAR_LOOPUNROLLANDJAMPASS_H

llvm/lib/Passes/PassBuilder.cpp

ModulePassManager PassBuilder::buildModuleOptimizationPipeline(
   // Unroll small loops to hide loop backedge latency and saturate any parallel
   // execution resources of an out-of-order processor. We also then need to
   // clean up redundancies and loop invariant code.
   // FIXME: It would be really good to use a loop-integrated instruction
   // combiner for cleanup here so that the unrolling and LICM can be pipelined
   // across the loop nests.
   // We do UnrollAndJam in a separate LPM to ensure it happens before unroll
   if (EnableUnrollAndJam && PTO.LoopUnrolling) {
-    OptimizePM.addPass(
-        createFunctionToLoopPassAdaptor(LoopUnrollAndJamPass(Level)));
+    OptimizePM.addPass(LoopUnrollAndJamPass(Level));
   }
   OptimizePM.addPass(LoopUnrollPass(
       LoopUnrollOptions(Level, /*OnlyWhenForced=*/!PTO.LoopUnrolling,
                         PTO.ForgetAllSCEVInLoopUnroll)));
   OptimizePM.addPass(WarnMissedTransformationsPass());
   OptimizePM.addPass(InstCombinePass());
   OptimizePM.addPass(RequireAnalysisPass<OptimizationRemarkEmitterAnalysis, Function>());
   OptimizePM.addPass(createFunctionToLoopPassAdaptor(

llvm/lib/Passes/PassRegistry.def

 FUNCTION_PASS("sccp", SCCPPass())
 FUNCTION_PASS("sink", SinkingPass())
 FUNCTION_PASS("slp-vectorizer", SLPVectorizerPass())
 FUNCTION_PASS("speculative-execution", SpeculativeExecutionPass())
 FUNCTION_PASS("spec-phis", SpeculateAroundPHIsPass())
 FUNCTION_PASS("sroa", SROA())
 FUNCTION_PASS("tailcallelim", TailCallElimPass())
 FUNCTION_PASS("unreachableblockelim", UnreachableBlockElimPass())
+FUNCTION_PASS("unroll-and-jam", LoopUnrollAndJamPass())
 FUNCTION_PASS("verify", VerifierPass())
 FUNCTION_PASS("verify<domtree>", DominatorTreeVerifierPass())
 FUNCTION_PASS("verify<loops>", LoopVerifierPass())
 FUNCTION_PASS("verify<memoryssa>", MemorySSAVerifierPass())
 FUNCTION_PASS("verify<regions>", RegionInfoVerifierPass())
 FUNCTION_PASS("verify<safepoint-ir>", SafepointIRVerifierPass())
 FUNCTION_PASS("verify<scalar-evolution>", ScalarEvolutionVerifierPass())
 FUNCTION_PASS("view-cfg", CFGViewerPass())

 LOOP_PASS("rotate", LoopRotatePass())
 LOOP_PASS("no-op-loop", NoOpLoopPass())
 LOOP_PASS("print", PrintLoopPass(dbgs()))
 LOOP_PASS("loop-deletion", LoopDeletionPass())
 LOOP_PASS("simplify-cfg", LoopSimplifyCFGPass())
 LOOP_PASS("strength-reduce", LoopStrengthReducePass())
 LOOP_PASS("indvars", IndVarSimplifyPass())
 LOOP_PASS("irce", IRCEPass())
-LOOP_PASS("unroll-and-jam", LoopUnrollAndJamPass())
 LOOP_PASS("unroll-full", LoopFullUnrollPass())
 LOOP_PASS("print-access-info", LoopAccessInfoPrinterPass(dbgs()))
 LOOP_PASS("print<ddg>", DDGAnalysisPrinterPass(dbgs()))
 LOOP_PASS("print<ivusers>", IVUsersPrinterPass(dbgs()))
 LOOP_PASS("print<loop-cache-cost>", LoopCachePrinterPass(dbgs()))
 LOOP_PASS("loop-predication", LoopPredicationPass())
 LOOP_PASS("guard-widening", GuardWideningPass())
 #undef LOOP_PASS

llvm/lib/Transforms/Scalar/LoopUnrollAndJamPass.cpp

tryToUnrollAndJamLoop(Loop *L, DominatorTree &DT, LoopInfo *LI,
   // If loop has an unroll count pragma or unrolled by explicitly set count
   // mark loop as unrolled to prevent unrolling beyond that requested.
   if (UnrollResult != LoopUnrollResult::FullyUnrolled && IsCountSetExplicitly)
     L->setLoopAlreadyUnrolled();
 
   return UnrollResult;
 }
 
+static bool tryToUnrollAndJamLoop(Function &F, DominatorTree &DT, LoopInfo &LI,
+                                  ScalarEvolution &SE,
+                                  const TargetTransformInfo &TTI,
+                                  AssumptionCache &AC, DependenceInfo &DI,
+                                  OptimizationRemarkEmitter &ORE,
+                                  int OptLevel) {
+  SmallPriorityWorklist<Loop *, 4> Worklist;
+  internal::appendLoopsToWorklist(reverse(LI), Worklist);
+  bool DidSomething = false;
+  while (!Worklist.empty()) {
+    Loop *L = Worklist.pop_back_val();
+    formLCSSA(*L, DT, &LI, &SE);
+    LoopUnrollResult Result =
+        tryToUnrollAndJamLoop(L, DT, &LI, SE, TTI, AC, DI, ORE, OptLevel);
+    if (Result != LoopUnrollResult::Unmodified)
+      DidSomething = true;
+  }
+
+  return DidSomething;
+}
+
 namespace {
 
-class LoopUnrollAndJam : public LoopPass {
+class LoopUnrollAndJam : public FunctionPass {
 public:
   static char ID; // Pass ID, replacement for typeid
   unsigned OptLevel;
 
-  LoopUnrollAndJam(int OptLevel = 2) : LoopPass(ID), OptLevel(OptLevel) {
+  LoopUnrollAndJam(int OptLevel = 2) : FunctionPass(ID), OptLevel(OptLevel) {
     initializeLoopUnrollAndJamPass(*PassRegistry::getPassRegistry());
   }
 
-  bool runOnLoop(Loop *L, LPPassManager &LPM) override {
-    if (skipLoop(L))
+  bool runOnFunction(Function &F) override {
+    if (skipFunction(F))
       return false;
 
-    Function &F = *L->getHeader()->getParent();
-
     auto &DT = getAnalysis<DominatorTreeWrapperPass>().getDomTree();
-    LoopInfo *LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
+    LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
     ScalarEvolution &SE = getAnalysis<ScalarEvolutionWrapperPass>().getSE();
     const TargetTransformInfo &TTI =
         getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
     auto &AC = getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
     auto &DI = getAnalysis<DependenceAnalysisWrapperPass>().getDI();
-    // For the old PM, we can't use OptimizationRemarkEmitter as an analysis
-    // pass.  Function analyses need to be preserved across loop transformations
-    // but ORE cannot be preserved (see comment before the pass definition).
-    OptimizationRemarkEmitter ORE(&F);
+    auto &ORE = getAnalysis<OptimizationRemarkEmitterWrapperPass>().getORE();
 
-    LoopUnrollResult Result =
-        tryToUnrollAndJamLoop(L, DT, LI, SE, TTI, AC, DI, ORE, OptLevel);
-
-    if (Result == LoopUnrollResult::FullyUnrolled)
-      LPM.markLoopAsDeleted(*L);
-
-    return Result != LoopUnrollResult::Unmodified;
+    return tryToUnrollAndJamLoop(F, DT, LI, SE, TTI, AC, DI, ORE, OptLevel);
   }
 
   /// This transformation requires natural loop information & requires that
   /// loop preheaders be inserted into the CFG...
   void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<AssumptionCacheTracker>();
+    AU.addRequired<DominatorTreeWrapperPass>();
+    AU.addRequired<LoopInfoWrapperPass>();
+    AU.addRequired<ScalarEvolutionWrapperPass>();
     AU.addRequired<TargetTransformInfoWrapperPass>();
+    AU.addRequired<AssumptionCacheTracker>();
     AU.addRequired<DependenceAnalysisWrapperPass>();
-    getLoopAnalysisUsage(AU);
+    AU.addRequired<OptimizationRemarkEmitterWrapperPass>();
   }
 };
 
 } // end anonymous namespace
 
 char LoopUnrollAndJam::ID = 0;
 
 INITIALIZE_PASS_BEGIN(LoopUnrollAndJam, "loop-unroll-and-jam",
                       "Unroll and Jam loops", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_DEPENDENCY(LoopPass)
 INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(DependenceAnalysisWrapperPass)
 INITIALIZE_PASS_END(LoopUnrollAndJam, "loop-unroll-and-jam",
                     "Unroll and Jam loops", false, false)
 
 Pass *llvm::createLoopUnrollAndJamPass(int OptLevel) {
   return new LoopUnrollAndJam(OptLevel);
 }
 
-PreservedAnalyses LoopUnrollAndJamPass::run(Loop &L, LoopAnalysisManager &AM,
-                                            LoopStandardAnalysisResults &AR,
-                                            LPMUpdater &) {
-  const auto &FAM =
-      AM.getResult<FunctionAnalysisManagerLoopProxy>(L, AR).getManager();
-  Function *F = L.getHeader()->getParent();
-
-  auto *ORE = FAM.getCachedResult<OptimizationRemarkEmitterAnalysis>(*F);
-  // FIXME: This should probably be optional rather than required.
-  if (!ORE)
-    report_fatal_error(
-        "LoopUnrollAndJamPass: OptimizationRemarkEmitterAnalysis not cached at "
-        "a higher level");
-
-  DependenceInfo DI(F, &AR.AA, &AR.SE, &AR.LI);
-
-  LoopUnrollResult Result = tryToUnrollAndJamLoop(
-      &L, AR.DT, &AR.LI, AR.SE, AR.TTI, AR.AC, DI, *ORE, OptLevel);
+PreservedAnalyses LoopUnrollAndJamPass::run(Function &F,
+                                            FunctionAnalysisManager &AM) {
+  ScalarEvolution &SE = AM.getResult<ScalarEvolutionAnalysis>(F);
+  LoopInfo &LI = AM.getResult<LoopAnalysis>(F);
+  TargetTransformInfo &TTI = AM.getResult<TargetIRAnalysis>(F);
+  AssumptionCache &AC = AM.getResult<AssumptionAnalysis>(F);
+  DominatorTree &DT = AM.getResult<DominatorTreeAnalysis>(F);
+  DependenceInfo &DI = AM.getResult<DependenceAnalysis>(F);
+  OptimizationRemarkEmitter &ORE =
+      AM.getResult<OptimizationRemarkEmitterAnalysis>(F);
 
-  if (Result == LoopUnrollResult::Unmodified)
+  if (!tryToUnrollAndJamLoop(F, DT, LI, SE, TTI, AC, DI, ORE, OptLevel))
     return PreservedAnalyses::all();
 
   return getLoopPassPreservedAnalyses();
 }

llvm/test/Transforms/LoopUnrollAndJam/dependencies.ll

 ; RUN: opt -basicaa -loop-unroll-and-jam -allow-unroll-and-jam -unroll-and-jam-count=4 < %s -S | FileCheck %s
 
 target datalayout = "e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64"
 
 ; CHECK-LABEL: fore_aft_less
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 ; CHECK: %j.2 = phi
 ; CHECK: %j.3 = phi
 define void @fore_aft_less(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %add72 = add nuw nsw i32 %i, -1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: fore_aft_eq
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 ; CHECK: %j.2 = phi
 ; CHECK: %j.3 = phi
 define void @fore_aft_eq(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %add72 = add nuw nsw i32 %i, 0
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: fore_aft_more
 ; CHECK: %j = phi
 ; CHECK-NOT: %j.1 = phi
 define void @fore_aft_more(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %add72 = add nuw nsw i32 %i, 1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: fore_sub_less
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 ; CHECK: %j.2 = phi
 ; CHECK: %j.3 = phi
 define void @fore_sub_less(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add72 = add nuw nsw i32 %i, -1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %add6 = add nuw nsw i32 %j, 1
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: fore_sub_eq
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 ; CHECK: %j.2 = phi
 ; CHECK: %j.3 = phi
 define void @fore_sub_eq(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add72 = add nuw nsw i32 %i, 0
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %add6 = add nuw nsw i32 %j, 1
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: fore_sub_more
 ; CHECK: %j = phi
 ; CHECK-NOT: %j.1 = phi
 define void @fore_sub_more(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add72 = add nuw nsw i32 %i, 1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %add6 = add nuw nsw i32 %j, 1
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: sub_aft_less
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 ; CHECK: %j.2 = phi
 ; CHECK: %j.3 = phi
 define void @sub_aft_less(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %add72 = add nuw nsw i32 %i, -1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: sub_aft_eq
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 ; CHECK: %j.2 = phi
 ; CHECK: %j.3 = phi
 define void @sub_aft_eq(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %add72 = add nuw nsw i32 %i, 0
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: sub_aft_more
 ; CHECK: %j = phi
 ; CHECK-NOT: %j.1 = phi
 define void @sub_aft_more(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %add72 = add nuw nsw i32 %i, 1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: sub_sub_less
 ; CHECK: %j = phi
 ; CHECK-NOT: %j.1 = phi
 define void @sub_sub_less(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   %add72 = add nuw nsw i32 %i, -1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: sub_sub_eq
 ; CHECK: %j = phi
 ; CHECK: %j.1 = phi
 define void @sub_sub_eq(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   %add72 = add nuw nsw i32 %i, 0
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }
 
 
 ; CHECK-LABEL: sub_sub_more
 ; CHECK: %j = phi
 ; CHECK-NOT: %j.1 = phi
 define void @sub_sub_more(i32* noalias nocapture %A, i32 %N, i32* noalias nocapture readonly %B) {
 entry:
   %cmp = icmp sgt i32 %N, 0
-  br i1 %cmp, label %for.outer, label %cleanup
+  br i1 %cmp, label %for.outer.preheader, label %cleanup
+
+for.outer.preheader:
+  br label %for.outer
 
 for.outer:
-  %i = phi i32 [ %add7, %for.latch ], [ 0, %entry ]
+  %i = phi i32 [ %add7, %for.latch ], [ 0, %for.outer.preheader ]
   br label %for.inner
 
 for.inner:
   %j = phi i32 [ %add6, %for.inner ], [ 0, %for.outer ]
   %sum = phi i32 [ %add, %for.inner ], [ 0, %for.outer ]
   %arrayidx5 = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx5, align 4
   %mul = mul nsw i32 %0, %i
   %add = add nsw i32 %mul, %sum
   %add6 = add nuw nsw i32 %j, 1
   %arrayidx = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 1, i32* %arrayidx, align 4
   %add72 = add nuw nsw i32 %i, 1
   %arrayidx8 = getelementptr inbounds i32, i32* %A, i32 %add72
   store i32 %add, i32* %arrayidx8, align 4
   %exitcond = icmp eq i32 %add6, %N
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %add7 = add nuw nsw i32 %i, 1
   %exitcond29 = icmp eq i32 %add7, %N
-  br i1 %exitcond29, label %cleanup, label %for.outer
+  br i1 %exitcond29, label %cleanup.loopexit, label %for.outer
+
+cleanup.loopexit:
+  br label %cleanup
 
 cleanup:
   ret void
 }

llvm/test/Transforms/LoopUnrollAndJam/disable.ll

for.inner:
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %arrayidx7 = getelementptr inbounds i32, i32* %A, i32 %b.028
   store i32 %add, i32* %arrayidx7, align 4
   %inc8 = add nuw nsw i32 %b.028, 1
   %add10 = add nuw nsw i32 %i.029, 1
   %exitcond30 = icmp eq i32 %add10, %I
-  br i1 %exitcond30, label %return, label %for.outer
+  br i1 %exitcond30, label %return.loopexit, label %for.outer
+
+return.loopexit:
+  br label %return
 
 return:
   ret void
 }
 
 
 ; CHECK-LABEL: disabled2
 ; Tests an incompatible block layout (for.outer jumps past for.inner)

 for.preheader:
   br label %for.outer
 
 for.outer:
   %i.032 = phi i32 [ %add13, %for.latch ], [ 0, %for.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %B, i32 %i.032
   %0 = load i32, i32* %arrayidx, align 4
   %tobool = icmp eq i32 %0, 0
-  br i1 %tobool, label %for.latch, label %for.inner
+  br i1 %tobool, label %for.latch, label %for.inner.preheader
+
+for.inner.preheader:
+  br label %for.inner
 
 for.inner:
-  %j.030 = phi i32 [ %inc, %for.inner ], [ 0, %for.outer ]
-  %sum1.029 = phi i32 [ %sum1.1, %for.inner ], [ 0, %for.outer ]
+  %j.030 = phi i32 [ %inc, %for.inner ], [ 0, %for.inner.preheader ]
+  %sum1.029 = phi i32 [ %sum1.1, %for.inner ], [ 0, %for.inner.preheader ]
   %arrayidx6 = getelementptr inbounds i32, i32* %B, i32 %j.030
   %1 = load i32, i32* %arrayidx6, align 4
   %tobool7 = icmp eq i32 %1, 0
   %sub = add i32 %sum1.029, 10
   %add = sub i32 %sub, %1
   %sum1.1 = select i1 %tobool7, i32 %sum1.029, i32 %add
   %inc = add nuw i32 %j.030, 1
   %exitcond = icmp eq i32 %inc, %J
-  br i1 %exitcond, label %for.latch, label %for.inner
+  br i1 %exitcond, label %for.latch.loopexit, label %for.inner
+
+for.latch.loopexit:
+  br label %for.latch
 
 for.latch:
-  %sum1.1.lcssa = phi i32 [ 0, %for.outer ], [ %sum1.1, %for.inner ]
+  %sum1.1.lcssa = phi i32 [ 0, %for.outer ], [ %sum1.1, %for.latch.loopexit ]
   %arrayidx11 = getelementptr inbounds i32, i32* %A, i32 %i.032
   store i32 %sum1.1.lcssa, i32* %arrayidx11, align 4
   %add13 = add nuw i32 %i.032, 1
   %exitcond33 = icmp eq i32 %add13, %I
-  br i1 %exitcond33, label %for.end14, label %for.outer
+  br i1 %exitcond33, label %for.end14.loopexit, label %for.outer
+
+for.end14.loopexit:
+  br label %for.end14
 
 for.end14:
   ret void
 }
 
 
 ; CHECK-LABEL: disabled3
 ; Tests loop carry dependencies in an array S

 
 for.latch:
   %arrayidx6 = getelementptr inbounds i32, i32* %A, i32 %indvars.iv
   store i32 %add, i32* %arrayidx6, align 4
   %indvars.iv.next = add nuw i32 %indvars.iv, 1
   %exitcond24 = icmp eq i32 %indvars.iv.next, %I
   br i1 %exitcond24, label %for.end9, label %for.outer
 
+for.end9.loopexit:
+  br label %for.end9
+
 for.end9:
   ret void
 }
 
 
 ; CHECK-LABEL: disabled5
 ; Test odd uses of phi nodes where the outer IV cannot be moved into Fore as it hits a PHI
 @f = hidden global i32 0, align 4
for.body6:
   %j.026 = phi i32 [ 0, %for.body ], [ %add9, %for.body6 ]
   %arrayidx7 = getelementptr inbounds i32, i32* %B, i32 %j.026
   %1 = load i32, i32* %arrayidx7, align 4
   %add = add i32 %1, %0
   %add9 = add nuw i32 %j.026, 1
   %exitcond = icmp eq i32 %add9, %J
   br i1 %exitcond, label %for.cond3.for.cond.cleanup5_crit_edge, label %for.body6
 
+for.end12.loopexit:
+  br label %for.end12
+
 for.end12:
   ret void
 }
 
 
 ; CHECK-LABEL: disabled8
 ; Same as above with an extra outer loop nest
 define void @disabled8(i32 %I, i32 %J, i32* noalias nocapture %A, i32* noalias nocapture readonly %B) #0 {
for.inner:
   %j.034 = phi i32 [ 0, %for.outer ], [ %add13, %for.inner ]
   %arrayidx11 = getelementptr inbounds i32, i32* %B, i32 %j.034
   %1 = load i32, i32* %arrayidx11, align 4
   %add = add i32 %1, %0
   %add13 = add nuw i32 %j.034, 1
   %exitcond = icmp eq i32 %add13, %J
   br i1 %exitcond, label %for.latch, label %for.inner
 
+for.end18.loopexit:
+  br label %for.end18
+
 for.end18:
   ret void
 }
 
 
 ; CHECK-LABEL: disabled9
 ; Can't prove alias between A and B
 define void @disabled9(i32 %I, i32 %J, i32* nocapture %A, i32* nocapture readonly %B) #0 {

 for.inner:
   %j = phi i32 [ 0, %for.outer ], [ %inc, %for.inner ]
   %sum1 = phi i32 [ 0, %for.outer ], [ %add, %for.inner ]
   %arrayidx = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx, align 4
   %add = add i32 %0, %sum1
   %inc = add nuw i32 %j, 1
   %exitcond = icmp eq i32 %inc, %J
-  br i1 %exitcond, label %for.inner2, label %for.inner
+  br i1 %exitcond, label %for.inner2.preheader, label %for.inner
+
+for.inner2.preheader:
+  br label %for.inner2
 
 for.inner2:
-  %j2 = phi i32 [ 0, %for.inner ], [ %inc2, %for.inner2 ]
-  %sum12 = phi i32 [ 0, %for.inner ], [ %add2, %for.inner2 ]
+  %j2 = phi i32 [ %inc2, %for.inner2 ], [ 0, %for.inner2.preheader ]
+  %sum12 = phi i32 [ %add2, %for.inner2 ], [ 0, %for.inner2.preheader ]
   %arrayidx2 = getelementptr inbounds i32, i32* %B, i32 %j2
   %l0 = load i32, i32* %arrayidx2, align 4
   %add2 = add i32 %l0, %sum12
   %inc2 = add nuw i32 %j2, 1
   %exitcond2 = icmp eq i32 %inc2, %J
   br i1 %exitcond2, label %for.latch, label %for.inner2
 
 for.latch:

 
 for.outer.preheader:
   br label %for.outer
 
 for.outer:
   %i = phi i32 [ %add8, %for.latch ], [ 0, %for.outer.preheader ]
   %add8 = add nuw i32 %i, 1
   %exitcond23 = icmp eq i32 %add8, %I
-  br i1 %exitcond23, label %for.end.loopexit, label %for.inner
+  br i1 %exitcond23, label %for.end.loopexit, label %for.inner.preheader
+
+for.inner.preheader:
+  br label %for.inner
 
 for.inner:
-  %j = phi i32 [ 0, %for.outer ], [ %inc, %for.inner ]
-  %sum1 = phi i32 [ 0, %for.outer ], [ %add, %for.inner ]
+  %j = phi i32 [ %inc, %for.inner ], [ 0, %for.inner.preheader ]
+  %sum1 = phi i32 [ %add, %for.inner ], [ 0, %for.inner.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx, align 4
   %add = add i32 %0, %sum1
   %inc = add nuw i32 %j, 1
   %exitcond = icmp eq i32 %inc, %J
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:

 
 for.outer.preheader:
   br label %for.outer
 
 for.outer:
   %i = phi i32 [ %add8, %for.latch ], [ 0, %for.outer.preheader ]
   %add8 = add nuw i32 %i, 1
   %exitcond25 = icmp eq i32 %add8, %I
-  br i1 %exitcond25, label %for.end.loopexit, label %for.inner
+  br i1 %exitcond25, label %for.end.loopexit, label %for.inner.preheader
+
+for.inner.preheader:
+  br label %for.inner
 
 for.inner:
-  %j = phi i32 [ 0, %for.outer ], [ %inc, %for.inner ]
-  %sum1 = phi i32 [ 0, %for.outer ], [ %add, %for.inner ]
+  %j = phi i32 [ %inc, %for.inner ], [ 0, %for.inner.preheader ]
+  %sum1 = phi i32 [ %add, %for.inner ], [ 0, %for.inner.preheader ]
   %arrayidx = getelementptr inbounds i32, i32* %B, i32 %j
   %0 = load i32, i32* %arrayidx, align 4
   %add = add i32 %0, %sum1
   %inc = add nuw i32 %j, 1
   %exitcond = icmp eq i32 %inc, %J
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:

llvm/test/Transforms/LoopUnrollAndJam/pragma-explicit.ll

for.body4.us:                                     ; preds = %for.body4.us, %for.cond1.preheader.us
   store i8 %conv24.us, i8* %arrayidx26.us, align 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond1.for.cond.cleanup3_crit_edge.us, label %for.body4.us
 
 for.cond1.for.cond.cleanup3_crit_edge.us:         ; preds = %for.body4.us
   %add.ptr.us = getelementptr inbounds i8, i8* %dst.addr.054.us, i64 %idx.ext
   %inc30.us = add nuw nsw i32 %y.055.us, 1
   %exitcond58 = icmp eq i32 %inc30.us, %height
-  br i1 %exitcond58, label %for.cond.cleanup, label %for.cond1.preheader.us, !llvm.loop !5
+  br i1 %exitcond58, label %for.cond.cleanup.loopexit, label %for.cond1.preheader.us, !llvm.loop !5
 
-for.cond.cleanup:                                 ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us, %for.cond1.preheader.lr.ph, %entry
+for.cond.cleanup.loopexit:                        ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us
+  br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup.loopexit, %for.cond1.preheader.lr.ph, %entry
   ret void
 }
 
 ; CHECK-LABEL: function2
 ; The explicit metadata here should force this to be unroll and jammed, but
 ; the count is left to thresholds. In this case 2 (hence %.pre60.1).
 ; CHECK: %.pre = phi i8 [ %.pre60.1, %for.cond1.for.cond.cleanup3_crit_edge.us ], [ %.pre.pre, %for.cond1.preheader.us.preheader.new ]
 ; CHECK: %indvars.iv.1 = phi i64 [ 0, %for.cond1.preheader.us ], [ %indvars.iv.next.1, %for.body4.us ]
for.body4.us:                                     ; preds = %for.body4.us, %for.cond1.preheader.us
   store i8 %conv24.us, i8* %arrayidx26.us, align 1
   %exitcond = icmp eq i64 %indvars.iv.next, %wide.trip.count
   br i1 %exitcond, label %for.cond1.for.cond.cleanup3_crit_edge.us, label %for.body4.us
 
 for.cond1.for.cond.cleanup3_crit_edge.us:         ; preds = %for.body4.us
   %add.ptr.us = getelementptr inbounds i8, i8* %dst.addr.054.us, i64 %idx.ext
   %inc30.us = add nuw nsw i32 %y.055.us, 1
   %exitcond58 = icmp eq i32 %inc30.us, %height
-  br i1 %exitcond58, label %for.cond.cleanup, label %for.cond1.preheader.us, !llvm.loop !7
+  br i1 %exitcond58, label %for.cond.cleanup.loopexit, label %for.cond1.preheader.us, !llvm.loop !7
+
+for.cond.cleanup.loopexit:                        ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us
+  br label %for.cond.cleanup
 
-for.cond.cleanup:                                 ; preds = %for.cond1.for.cond.cleanup3_crit_edge.us, %for.cond1.preheader.lr.ph, %entry
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup.loopexit, %for.cond1.preheader.lr.ph, %entry
   ret void
 }
 
 !5 = distinct !{!5, !6}
 !6 = !{!"llvm.loop.unroll_and_jam.count", i32 4}
 !7 = distinct !{!7, !8}
 !8 = !{!"llvm.loop.unroll_and_jam.enable"}

llvm/test/Transforms/LoopUnrollAndJam/unroll-and-jam.ll

for.inner:
   %exitcond = icmp eq i32 %inc, %J
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %arrayidx6 = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 %add, i32* %arrayidx6, align 4, !tbaa !5
   %add8 = add nuw nsw i32 %i, 1
   %exitcond23 = icmp eq i32 %add8, 4
-  br i1 %exitcond23, label %for.end, label %for.outer
+  br i1 %exitcond23, label %for.end.loopexit, label %for.outer
+
+for.end.loopexit:
+  br label %for.end
 
 for.end:
   ret void
 }
 
 
 ; CHECK-LABEL: test4
 ; Tests Complete unroll-and-jam with a trip count of 1
for.inner:
   %exitcond = icmp eq i32 %inc, %J
   br i1 %exitcond, label %for.latch, label %for.inner
 
 for.latch:
   %arrayidx6 = getelementptr inbounds i32, i32* %A, i32 %i
   store i32 %add, i32* %arrayidx6, align 4, !tbaa !5
   %add8 = add nuw nsw i32 %i, 1
   %exitcond23 = icmp eq i32 %add8, 1
-  br i1 %exitcond23, label %for.end, label %for.outer
+  br i1 %exitcond23, label %for.end.loopexit, label %for.outer
+
+for.end.loopexit:
+  br label %for.end
 
 for.end:
   ret void
 }
 
 
 ; CHECK-LABEL: test5
 ; Multiple SubLoopBlocks
for.outer:
   %add = add nuw i32 %i, 1
   %arrayidx2 = getelementptr inbounds i32, i32* %A, i32 %add
   store i32 2, i32* %arrayidx2, align 4, !tbaa !5
   br label %for.inner
 
 for.latch:
   store i32 %add9, i32* %arrayidx, align 4, !tbaa !5
   %exitcond30 = icmp eq i32 %add, %I
-  br i1 %exitcond30, label %for.end, label %for.outer
+  br i1 %exitcond30, label %for.end.loopexit, label %for.outer
 
 for.inner:
   %sum = phi i32 [ 0, %for.outer ], [ %add9, %for.inner ]
   %j = phi i32 [ 0, %for.outer ], [ %add10, %for.inner ]
   %arrayidx7 = getelementptr inbounds i32, i32* %B, i32 %j
   %l1 = load i32, i32* %arrayidx7, align 4, !tbaa !5
   %add9 = add i32 %l1, %sum
   %add10 = add nuw i32 %j, 1
   %exitcond = icmp eq i32 %add10, %J
   br i1 %exitcond, label %for.latch, label %for.inner
 
+for.end.loopexit:
+  br label %for.end
+
 for.end:
   ret void
 }
 
 
 ; CHECK-LABEL: test8
 ; Same as test7 with an extra outer loop nest
 ; CHECK: for.outest:

 for.latch:
   store i32 %add9, i32* %arrayidx, align 4, !tbaa !5
   %exitcond39 = icmp eq i32 %add, %I
   br i1 %exitcond39, label %for.cleanup, label %for.outer
 
 for.cleanup:
   %inc = add nuw nsw i32 %x.038, 1
   %exitcond41 = icmp eq i32 %inc, 5
-  br i1 %exitcond41, label %for.end, label %for.outest
+  br i1 %exitcond41, label %for.end.loopexit, label %for.outest
+
+for.end.loopexit:
+  br label %for.end
 
 for.end:
   ret void
 }
 
 
 ; CHECK-LABEL: test9
 ; Same as test1 with tbaa, not noalias