[PM/LoopUnswitch] Teach the new unswitch to handle nontrivial unswitching of switches.

[chandlerc]

Summary

This works much like trivial unswitching of switches in that it reliably
moves the switch out of the loop. Here we potentially clone the entire
loop into each successor of the switch and re-point the cases at these
clones.

Due to the complexity of actually doing nontrivial unswitching, this
patch doesn't create a dedicated routine for handling switches -- it
would duplicate far too much code. Instead, it generalizes the existing
routine to handle both branches and switches as it largely reduces to
looping in a few places instead of doing something once. This actually
improves the results in some cases with branches due to being much more
careful about how dead regions of code are managed. With branches,
because exactly one clone is created and there are exactly two edges
considered, somewhat sloppy handling of the dead regions of code was
sufficient in most cases. But with switches, there are much more
complicated patterns of dead code and so I've had to move to a more
robust model generally. We still do as much pruning of the dead code
early as possible because that allows us to avoid even cloning the code.

This also surfaced another problem with nontrivial unswitching before
which is that we weren't as precise in reconstructing loops as we could
have been. This seems to have been mostly harmless, but resulted in
pointless LCSSA PHI nodes and other unnecessary cruft. With switches, we
have to get this *right*, and everything benefits from it.

While the testing may seem a bit light here because we only have two
real cases with actual switches, they do a surprisingly good job of
exercising numerous edge cases here. Also, because we share the logic
with branches, most of the changes in this patch are reasonably well
covered by existing tests. Still, I'm going to look at some potentially
more interesting test coverage as well but didn't want to delay sending
this out.

The new unswitch now has all of the same fundamental power as the old
one with the exception of the single unsound case of *partial* switch
unswitching -- that really is just loop specialization and not
unswitching at all. It doesn't fit into the canonicalization model in
any way. We can add a loop specialization pass that runs late based on
profile data if important test cases ever come up here.

Depends on D47522.

[hfinkel]

Can you please update the comment at the top of include/llvm/Transforms/Scalar/SimpleLoopUnswitch.h to define "full vs. partial" unswitching and "trivial vs. non-trivial" unswitching?

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

1798 ↗

(On Diff #149630)

Dead code?

2016 ↗

(On Diff #149630)

This FIXME is now out of date (switches are handled above)?

[asbirlea]

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

731 ↗

(On Diff #149630)

Could you update the above comment to clarify who DominatingSucc is?
e.g. "It skips loop and exit blocks if their dominating successor (DominatingSucc) is not the block being unswitched." ?

1747 ↗

(On Diff #149630)

clang-format?

[chandlerc]

Update to address code review comments.

[chandlerc]

Another fixe.

[chandlerc]

Thanks for comments, should all be addressed

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

731 ↗

(On Diff #149630)

I used different phrasing, but let me know if it didn't work.

1747 ↗

(On Diff #149630)

Fixed.

1798 ↗

(On Diff #149630)

Doh! sorry i didn't delete this.

[chandlerc]

This time *actually* inculde the updated comments in the header file that Hal requested. Soryr this were missing in the prior update.

[fedor.sergeev]

Not sure where to put this into... but I ran an experiment by adding -passes=loop(unswitch) to the LoopUnswitch tests and here is the result
with the build of all the three dependent fixes:
Failing Tests (11):

  LLVM :: Transforms/LoopUnswitch/2011-11-18-TwoSwitches.ll
  LLVM :: Transforms/LoopUnswitch/basictest.ll
  LLVM :: Transforms/LoopUnswitch/copy-metadata.ll
  LLVM :: Transforms/LoopUnswitch/elseif-non-exponential-behavior.ll
  LLVM :: Transforms/LoopUnswitch/guards.ll
  LLVM :: Transforms/LoopUnswitch/infinite-loop.ll
  LLVM :: Transforms/LoopUnswitch/msan.ll
  LLVM :: Transforms/LoopUnswitch/simplify-with-nonvalness.ll
  LLVM :: Transforms/LoopUnswitch/trivial-unswitch.ll
  LLVM :: Transforms/LoopUnswitch/unswitch-equality-undef.ll
  LLVM :: Transforms/LoopUnswitch/unswitch-select.ll

Expected Passes    : 30
Unsupported Tests  : 1
Unexpected Failures: 11

---

Some tests involve selects and guards, which I gather are not yet supported, some involve multiple unswitches over the same loop etc.
IMO it makes sense to go through these results and see if there are any surprises there.
I can submit a diff (and/or log of the run) if needed.

[chandlerc]

In D47683#1126575, @fedor.sergeev wrote:

Not sure where to put this into... but I ran an experiment by adding -passes=loop(unswitch) to the LoopUnswitch tests and here is the result
with the build of all the three dependent fixes:
Failing Tests (11):

  LLVM :: Transforms/LoopUnswitch/2011-11-18-TwoSwitches.ll
  LLVM :: Transforms/LoopUnswitch/basictest.ll
  LLVM :: Transforms/LoopUnswitch/copy-metadata.ll
  LLVM :: Transforms/LoopUnswitch/elseif-non-exponential-behavior.ll
  LLVM :: Transforms/LoopUnswitch/guards.ll
  LLVM :: Transforms/LoopUnswitch/infinite-loop.ll
  LLVM :: Transforms/LoopUnswitch/msan.ll
  LLVM :: Transforms/LoopUnswitch/simplify-with-nonvalness.ll
  LLVM :: Transforms/LoopUnswitch/trivial-unswitch.ll
  LLVM :: Transforms/LoopUnswitch/unswitch-equality-undef.ll
  LLVM :: Transforms/LoopUnswitch/unswitch-select.ll

Expected Passes    : 30
Unsupported Tests  : 1
Unexpected Failures: 11

---

Some tests involve selects and guards, which I gather are not yet supported, some involve multiple unswitches over the same loop etc.
IMO it makes sense to go through these results and see if there are any surprises there.
I can submit a diff (and/or log of the run) if needed.

Most of these test files were ported to the new pass. Some of them aren't relevant any more.

I did in fact go through these and ported as many as I could when doing the initial work. But the old passes's testing is .... not great. I don't think there is much more value in re-analyzing these, but if you see specific things that you'd like to port over, go for it.

On the flip side, if you see specific logic you'd like better testing of, I'm more than happy to just directly add the testing. I think the tests we will get from that will be substantially higher quality.

ONe thing that makes it very expensive is that the results of unswitching are substantially different between the passes meaning that most of the CHECK lines have to be manually rewritten anyways.

[chandlerc]

Rebase now that all the prior patches are landed and ping.

[asbirlea]

LGTM.

llvm/trunk/include/llvm/Transforms/Scalar/SimpleLoopUnswitch.h

 
 #include "llvm/Analysis/LoopAnalysisManager.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Transforms/Scalar/LoopPassManager.h"
 
 namespace llvm {
 
-/// This pass transforms loops that contain branches on loop-invariant
-/// conditions to have multiple loops. For example, it turns the left into the
-/// right code:
+/// This pass transforms loops that contain branches or switches on loop-
+/// invariant conditions to have multiple loops. For example, it turns the left
+/// into the right code:
 ///
 ///  for (...)                  if (lic)
 ///    A                          for (...)
 ///    if (lic)                     A; B; C
 ///      B                      else
 ///    C                          for (...)
 ///                                 A; C
 ///
 /// This can increase the size of the code exponentially (doubling it every time
 /// a loop is unswitched) so we only unswitch if the resultant code will be
 /// smaller than a threshold.
 ///
 /// This pass expects LICM to be run before it to hoist invariant conditions out
 /// of the loop, to make the unswitching opportunity obvious.
 ///
+/// There is a taxonomy of unswitching that we use to classify different forms
+/// of this transformaiton:
+///
+/// - Trival unswitching: this is when the condition can be unswitched without
+///   cloning any code from inside the loop. A non-trivial unswitch requires
+///   code duplication.
+///
+/// - Full unswitching: this is when the branch or switch is completely moved
+///   from inside the loop to outside the loop. Partial unswitching removes the
+///   branch from the clone of the loop but must leave a (somewhat simplified)
+///   branch in the original loop. While theoretically partial unswitching can
+///   be done for switches, the requirements are extreme - we need the loop
+///   invariant input to the switch to be sufficient to collapse to a single
+///   successor in each clone.
+///
+/// This pass always does trivial, full unswitching for both branches and
+/// switches. For branches, it also always does trivial, partial unswitching.
+///
+/// If enabled (via the constructor's `NonTrivial` parameter), this pass will
+/// additionally do non-trivial, full unswitching for branches and switches, and
+/// will do non-trivial, partial unswitching for branches.
+///
+/// Because partial unswitching of switches is extremely unlikely to be possible
+/// in practice and significantly complicates the implementation, this pass does
+/// not currently implement that in any mode.
 class SimpleLoopUnswitchPass : public PassInfoMixin<SimpleLoopUnswitchPass> {
   bool NonTrivial;
 
 public:
   SimpleLoopUnswitchPass(bool NonTrivial = false) : NonTrivial(NonTrivial) {}
 
   PreservedAnalyses run(Loop &L, LoopAnalysisManager &AM,
                         LoopStandardAnalysisResults &AR, LPMUpdater &U);

llvm/trunk/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

 /// Build the cloned blocks for an unswitched copy of the given loop.
 ///
 /// The cloned blocks are inserted before the loop preheader (`LoopPH`) and
 /// after the split block (`SplitBB`) that will be used to select between the
 /// cloned and original loop.
 ///
 /// This routine handles cloning all of the necessary loop blocks and exit
 /// blocks including rewriting their instructions and the relevant PHI nodes.
-/// It skips loop and exit blocks that are not necessary based on the provided
-/// set. It also correctly creates the unconditional branch in the cloned
+/// Any loop blocks or exit blocks which are dominated by a different successor
+/// than the one for this clone of the loop blocks can be trivially skipped. We
+/// use the `DominatingSucc` map to determine whether a block satisfies that
+/// property with a simple map lookup.
+///
+/// It also correctly creates the unconditional branch in the cloned
 /// unswitched parent block to only point at the unswitched successor.
 ///
 /// This does not handle most of the necessary updates to `LoopInfo`. Only exit
 /// block splitting is correctly reflected in `LoopInfo`, essentially all of
 /// the cloned blocks (and their loops) are left without full `LoopInfo`
 /// updates. This also doesn't fully update `DominatorTree`. It adds the cloned
 /// blocks to them but doesn't create the cloned `DominatorTree` structure and
 /// instead the caller must recompute an accurate DT. It *does* correctly
 /// update the `AssumptionCache` provided in `AC`.
 static BasicBlock *buildClonedLoopBlocks(
     Loop &L, BasicBlock *LoopPH, BasicBlock *SplitBB,
     ArrayRef<BasicBlock *> ExitBlocks, BasicBlock *ParentBB,
     BasicBlock *UnswitchedSuccBB, BasicBlock *ContinueSuccBB,
-    const SmallPtrSetImpl<BasicBlock *> &SkippedLoopAndExitBlocks,
+    const SmallDenseMap<BasicBlock *, BasicBlock *, 16> &DominatingSucc,
     ValueToValueMapTy &VMap,
     SmallVectorImpl<DominatorTree::UpdateType> &DTUpdates, AssumptionCache &AC,
     DominatorTree &DT, LoopInfo &LI) {
   SmallVector<BasicBlock *, 4> NewBlocks;
   NewBlocks.reserve(L.getNumBlocks() + ExitBlocks.size());
 
   // We will need to clone a bunch of blocks, wrap up the clone operation in
   // a helper.
   auto CloneBlock = [&](BasicBlock *OldBB) {
     // Clone the basic block and insert it before the new preheader.
     BasicBlock *NewBB = CloneBasicBlock(OldBB, VMap, ".us", OldBB->getParent());
     NewBB->moveBefore(LoopPH);
 
     // Record this block and the mapping.
     NewBlocks.push_back(NewBB);
     VMap[OldBB] = NewBB;
 
     return NewBB;
   };
 
+  // We skip cloning blocks when they have a dominating succ that is not the
+  // succ we are cloning for.
+  auto SkipBlock = [&](BasicBlock *BB) {
+    auto It = DominatingSucc.find(BB);
+    return It != DominatingSucc.end() && It->second != UnswitchedSuccBB;
+  };
+
   // First, clone the preheader.
   auto *ClonedPH = CloneBlock(LoopPH);
 
   // Then clone all the loop blocks, skipping the ones that aren't necessary.
   for (auto *LoopBB : L.blocks())
-    if (!SkippedLoopAndExitBlocks.count(LoopBB))
+    if (!SkipBlock(LoopBB))
       CloneBlock(LoopBB);
 
   // Split all the loop exit edges so that when we clone the exit blocks, if
   // any of the exit blocks are *also* a preheader for some other loop, we
   // don't create multiple predecessors entering the loop header.
   for (auto *ExitBB : ExitBlocks) {
-    if (SkippedLoopAndExitBlocks.count(ExitBB))
+    if (SkipBlock(ExitBB))
       continue;
 
     // When we are going to clone an exit, we don't need to clone all the
     // instructions in the exit block and we want to ensure we have an easy
     // place to merge the CFG, so split the exit first. This is always safe to
     // do because there cannot be any non-loop predecessors of a loop exit in
     // loop simplified form.
     auto *MergeBB = SplitBlock(ExitBB, &ExitBB->front(), &DT, &LI);
static BasicBlock *buildClonedLoopBlocks(
   // unswitched successor.
   auto *ClonedSuccBB = cast<BasicBlock>(VMap.lookup(UnswitchedSuccBB));
   ClonedParentBB->getTerminator()->eraseFromParent();
   BranchInst::Create(ClonedSuccBB, ClonedParentBB);
 
   // Update any PHI nodes in the cloned successors of the skipped blocks to not
   // have spurious incoming values.
   for (auto *LoopBB : L.blocks())
-    if (SkippedLoopAndExitBlocks.count(LoopBB))
+    if (SkipBlock(LoopBB))
       for (auto *SuccBB : successors(LoopBB))
         if (auto *ClonedSuccBB = cast_or_null<BasicBlock>(VMap.lookup(SuccBB)))
           for (PHINode &PN : ClonedSuccBB->phis())
             PN.removeIncomingValue(LoopBB, /*DeletePHIIfEmpty*/ false);
 
   // Record the domtree updates for the new blocks.
   SmallPtrSet<BasicBlock *, 4> SuccSet;
   for (auto *ClonedBB : NewBlocks) {

 #endif
 
     NonChildClonedLoops.push_back(cloneLoopNest(
         *ChildL, ExitLoopMap.lookup(ClonedChildHeader), VMap, LI));
   }
 }
 
 static void
+deleteDeadClonedBlocks(Loop &L, ArrayRef<BasicBlock *> ExitBlocks,
+                       ArrayRef<std::unique_ptr<ValueToValueMapTy>> VMaps,
+                       DominatorTree &DT) {
+  // Find all the dead clones, and remove them from their successors.
+  SmallVector<BasicBlock *, 16> DeadBlocks;
+  for (BasicBlock *BB : llvm::concat<BasicBlock *const>(L.blocks(), ExitBlocks))
+    for (auto &VMap : VMaps)
+      if (BasicBlock *ClonedBB = cast_or_null<BasicBlock>(VMap->lookup(BB)))
+        if (!DT.isReachableFromEntry(ClonedBB)) {
+          for (BasicBlock *SuccBB : successors(ClonedBB))
+            SuccBB->removePredecessor(ClonedBB);
+          DeadBlocks.push_back(ClonedBB);
+        }
+
+  // Drop any remaining references to break cycles.
+  for (BasicBlock *BB : DeadBlocks)
+    BB->dropAllReferences();
+  // Erase them from the IR.
+  for (BasicBlock *BB : DeadBlocks)
+    BB->eraseFromParent();
+}
+
+static void
 deleteDeadBlocksFromLoop(Loop &L,
-                         const SmallVectorImpl<BasicBlock *> &DeadBlocks,
                          SmallVectorImpl<BasicBlock *> &ExitBlocks,
                          DominatorTree &DT, LoopInfo &LI) {
+  // Find all the dead blocks, and remove them from their successors.
+  SmallVector<BasicBlock *, 16> DeadBlocks;
+  for (BasicBlock *BB : llvm::concat<BasicBlock *const>(L.blocks(), ExitBlocks))
+    if (!DT.isReachableFromEntry(BB)) {
+      for (BasicBlock *SuccBB : successors(BB))
+        SuccBB->removePredecessor(BB);
+      DeadBlocks.push_back(BB);
+    }
+
   SmallPtrSet<BasicBlock *, 16> DeadBlockSet(DeadBlocks.begin(),
                                              DeadBlocks.end());
 
   // Filter out the dead blocks from the exit blocks list so that it can be
   // used in the caller.
   llvm::erase_if(ExitBlocks,
                  [&](BasicBlock *BB) { return DeadBlockSet.count(BB); });
 
-  // Remove these blocks from their successors.
-  for (auto *BB : DeadBlocks)
-    for (BasicBlock *SuccBB : successors(BB))
-      SuccBB->removePredecessor(BB, /*DontDeleteUselessPHIs*/ true);
-
   // Walk from this loop up through its parents removing all of the dead blocks.
   for (Loop *ParentL = &L; ParentL; ParentL = ParentL->getParentLoop()) {
     for (auto *BB : DeadBlocks)
       ParentL->getBlocksSet().erase(BB);
     llvm::erase_if(ParentL->getBlocksVector(),
                    [&](BasicBlock *BB) { return DeadBlockSet.count(BB); });
   }
 
do {
     for (DomTreeNode *ChildN : *N) {
       assert(Visited.insert(ChildN).second &&
              "Cannot visit a node twice when walking a tree!");
       DomWorklist.push_back(ChildN);
     }
   } while (!DomWorklist.empty());
 }
 
-/// Take an invariant branch that has been determined to be safe and worthwhile
-/// to unswitch despite being non-trivial to do so and perform the unswitch.
-///
-/// This directly updates the CFG to hoist the predicate out of the loop, and
-/// clone the necessary parts of the loop to maintain behavior.
-///
-/// It also updates both dominator tree and loopinfo based on the unswitching.
-///
-/// Once unswitching has been performed it runs the provided callback to report
-/// the new loops and no-longer valid loops to the caller.
-static bool unswitchInvariantBranch(
-    Loop &L, BranchInst &BI, ArrayRef<Value *> Invariants, DominatorTree &DT,
-    LoopInfo &LI, AssumptionCache &AC,
+static bool unswitchNontrivialInvariants(
+    Loop &L, TerminatorInst &TI, ArrayRef<Value *> Invariants,
+    DominatorTree &DT, LoopInfo &LI, AssumptionCache &AC,
     function_ref<void(bool, ArrayRef<Loop *>)> UnswitchCB) {
-  auto *ParentBB = BI.getParent();
+  auto *ParentBB = TI.getParent();
+  BranchInst *BI = dyn_cast<BranchInst>(&TI);
+  SwitchInst *SI = BI ? nullptr : cast<SwitchInst>(&TI);
 
-  // We can only unswitch conditional branches with an invariant condition or
-  // combining invariant conditions with an instruction.
-  assert(BI.isConditional() && "Can only unswitch a conditional branch!");
-  bool FullUnswitch = BI.getCondition() == Invariants[0];
+  // We can only unswitch switches, conditional branches with an invariant
+  // condition, or combining invariant conditions with an instruction.
+  assert((SI || BI->isConditional()) &&
+         "Can only unswitch switches and conditional branch!");
+  bool FullUnswitch = SI || BI->getCondition() == Invariants[0];
   if (FullUnswitch)
     assert(Invariants.size() == 1 &&
            "Cannot have other invariants with full unswitching!");
   else
-    assert(isa<Instruction>(BI.getCondition()) &&
+    assert(isa<Instruction>(BI->getCondition()) &&
            "Partial unswitching requires an instruction as the condition!");
 
   // Constant and BBs tracking the cloned and continuing successor. When we are
   // unswitching the entire condition, this can just be trivially chosen to
   // unswitch towards `true`. However, when we are unswitching a set of
   // invariants combined with `and` or `or`, the combining operation determines
   // the best direction to unswitch: we want to unswitch the direction that will
   // collapse the branch.
   bool Direction = true;
   int ClonedSucc = 0;
   if (!FullUnswitch) {
-    if (cast<Instruction>(BI.getCondition())->getOpcode() != Instruction::Or) {
-      assert(cast<Instruction>(BI.getCondition())->getOpcode() == Instruction::And &&
-        "Only `or` and `and` instructions can combine invariants being unswitched.");
+    if (cast<Instruction>(BI->getCondition())->getOpcode() != Instruction::Or) {
+      assert(cast<Instruction>(BI->getCondition())->getOpcode() ==
+                 Instruction::And &&
+             "Only `or` and `and` instructions can combine invariants being "
+             "unswitched.");
       Direction = false;
       ClonedSucc = 1;
     }
   }
-  auto *UnswitchedSuccBB = BI.getSuccessor(ClonedSucc);
-  auto *ContinueSuccBB = BI.getSuccessor(1 - ClonedSucc);
 
-  assert(UnswitchedSuccBB != ContinueSuccBB &&
-         "Should not unswitch a branch that always goes to the same place!");
+  BasicBlock *RetainedSuccBB =
+      BI ? BI->getSuccessor(1 - ClonedSucc) : SI->getDefaultDest();
+  SmallSetVector<BasicBlock *, 4> UnswitchedSuccBBs;
+  if (BI)
+    UnswitchedSuccBBs.insert(BI->getSuccessor(ClonedSucc));
+  else
+    for (auto Case : SI->cases())
+      UnswitchedSuccBBs.insert(Case.getCaseSuccessor());
+
+  assert(!UnswitchedSuccBBs.count(RetainedSuccBB) &&
+         "Should not unswitch the same successor we are retaining!");
 
   // The branch should be in this exact loop. Any inner loop's invariant branch
   // should be handled by unswitching that inner loop. The caller of this
   // routine should filter out any candidates that remain (but were skipped for
   // whatever reason).
   assert(LI.getLoopFor(ParentBB) == &L && "Branch in an inner loop!");
 
   SmallVector<BasicBlock *, 4> ExitBlocks;
   L.getUniqueExitBlocks(ExitBlocks);
 
   // We cannot unswitch if exit blocks contain a cleanuppad instruction as we
   // don't know how to split those exit blocks.
   // FIXME: We should teach SplitBlock to handle this and remove this
   // restriction.
   for (auto *ExitBB : ExitBlocks)
     if (isa<CleanupPadInst>(ExitBB->getFirstNonPHI()))
       return false;
 
-  SmallPtrSet<BasicBlock *, 4> ExitBlockSet(ExitBlocks.begin(),
-                                            ExitBlocks.end());
-
   // Compute the parent loop now before we start hacking on things.
   Loop *ParentL = L.getParentLoop();
 
   // Compute the outer-most loop containing one of our exit blocks. This is the
   // furthest up our loopnest which can be mutated, which we will use below to
   // update things.
   Loop *OuterExitL = &L;
   for (auto *ExitBB : ExitBlocks) {
     Loop *NewOuterExitL = LI.getLoopFor(ExitBB);
     if (!NewOuterExitL) {
       // We exited the entire nest with this block, so we're done.
       OuterExitL = nullptr;
       break;
     }
     if (NewOuterExitL != OuterExitL && NewOuterExitL->contains(OuterExitL))
       OuterExitL = NewOuterExitL;
   }
 
-  // If the edge we *aren't* cloning in the unswitch (the continuing edge)
-  // dominates its target, we can skip cloning the dominated region of the loop
-  // and its exits. We compute this as a set of nodes to be skipped.
-  SmallPtrSet<BasicBlock *, 4> SkippedLoopAndExitBlocks;
-  if (ContinueSuccBB->getUniquePredecessor() ||
-      llvm::all_of(predecessors(ContinueSuccBB), [&](BasicBlock *PredBB) {
-        return PredBB == ParentBB || DT.dominates(ContinueSuccBB, PredBB);
-      })) {
-    visitDomSubTree(DT, ContinueSuccBB, [&](BasicBlock *BB) {
-      SkippedLoopAndExitBlocks.insert(BB);
+  // If the edge from this terminator to a successor dominates that successor,
+  // store a map from each block in its dominator subtree to it. This lets us
+  // tell when cloning for a particular successor if a block is dominated by
+  // some *other* successor with a single data structure. We use this to
+  // significantly reduce cloning.
+  SmallDenseMap<BasicBlock *, BasicBlock *, 16> DominatingSucc;
+  for (auto *SuccBB : llvm::concat<BasicBlock *const>(
+           makeArrayRef(RetainedSuccBB), UnswitchedSuccBBs))
+    if (SuccBB->getUniquePredecessor() ||
+        llvm::all_of(predecessors(SuccBB), [&](BasicBlock *PredBB) {
+          return PredBB == ParentBB || DT.dominates(SuccBB, PredBB);
+        }))
+      visitDomSubTree(DT, SuccBB, [&](BasicBlock *BB) {
+        DominatingSucc[BB] = SuccBB;
         return true;
       });
-  }
-  // If we are doing full unswitching, then similarly to the above, the edge we
-  // *are* cloning in the unswitch (the unswitched edge) dominates its target,
-  // we will end up with dead nodes in the original loop and its exits that will
-  // need to be deleted. Here, we just retain that the property holds and will
-  // compute the deleted set later.
-  bool DeleteUnswitchedSucc =
-      FullUnswitch &&
-      (UnswitchedSuccBB->getUniquePredecessor() ||
-       llvm::all_of(predecessors(UnswitchedSuccBB), [&](BasicBlock *PredBB) {
-         return PredBB == ParentBB || DT.dominates(UnswitchedSuccBB, PredBB);
-       }));
 
   // Split the preheader, so that we know that there is a safe place to insert
   // the conditional branch. We will change the preheader to have a conditional
   // branch on LoopCond. The original preheader will become the split point
   // between the unswitched versions, and we will have a new preheader for the
   // original loop.
   BasicBlock *SplitBB = L.getLoopPreheader();
   BasicBlock *LoopPH = SplitEdge(SplitBB, L.getHeader(), &DT, &LI);
 
-  // Keep a mapping for the cloned values.
-  ValueToValueMapTy VMap;
-
   // Keep track of the dominator tree updates needed.
   SmallVector<DominatorTree::UpdateType, 4> DTUpdates;
 
-  // Build the cloned blocks from the loop.
-  auto *ClonedPH = buildClonedLoopBlocks(
-      L, LoopPH, SplitBB, ExitBlocks, ParentBB, UnswitchedSuccBB,
-      ContinueSuccBB, SkippedLoopAndExitBlocks, VMap, DTUpdates, AC, DT, LI);
+  // Clone the loop for each unswitched successor.
+  SmallVector<std::unique_ptr<ValueToValueMapTy>, 4> VMaps;
+  VMaps.reserve(UnswitchedSuccBBs.size());
+  SmallDenseMap<BasicBlock *, BasicBlock *, 4> ClonedPHs;
+  for (auto *SuccBB : UnswitchedSuccBBs) {
+    VMaps.emplace_back(new ValueToValueMapTy());
+    ClonedPHs[SuccBB] = buildClonedLoopBlocks(
+        L, LoopPH, SplitBB, ExitBlocks, ParentBB, SuccBB, RetainedSuccBB,
+        DominatingSucc, *VMaps.back(), DTUpdates, AC, DT, LI);
+  }
 
   // The stitching of the branched code back together depends on whether we're
   // doing full unswitching or not with the exception that we always want to
   // nuke the initial terminator placed in the split block.
   SplitBB->getTerminator()->eraseFromParent();
   if (FullUnswitch) {
-    // Remove the parent as a predecessor of the
-    // unswitched successor.
-    UnswitchedSuccBB->removePredecessor(ParentBB,
-                                        /*DontDeleteUselessPHIs*/ true);
-    DTUpdates.push_back({DominatorTree::Delete, ParentBB, UnswitchedSuccBB});
+    for (BasicBlock *SuccBB : UnswitchedSuccBBs) {
+      // Remove the parent as a predecessor of the unswitched successor.
+      SuccBB->removePredecessor(ParentBB,
+      DTUpdates.push_back({DominatorTree::Delete, ParentBB, SuccBB});
+    }
 
-    // Now splice the branch from the original loop and use it to select between
-    // the two loops.
-    SplitBB->getInstList().splice(SplitBB->end(), ParentBB->getInstList(), BI);
-    BI.setSuccessor(ClonedSucc, ClonedPH);
-    BI.setSuccessor(1 - ClonedSucc, LoopPH);
+    // Now splice the terminator from the original loop and rewrite its
+    // successors.
+    SplitBB->getInstList().splice(SplitBB->end(), ParentBB->getInstList(), TI);
+    if (BI) {
+      assert(UnswitchedSuccBBs.size() == 1 &&
+             "Only one possible unswitched block for a branch!");
+      BasicBlock *ClonedPH = ClonedPHs.begin()->second;
+      BI->setSuccessor(ClonedSucc, ClonedPH);
+      BI->setSuccessor(1 - ClonedSucc, LoopPH);
+      DTUpdates.push_back({DominatorTree::Insert, SplitBB, ClonedPH});
+    } else {
+      assert(SI && "Must either be a branch or switch!");
+
+      // Walk the cases and directly update their successors.
+      for (auto &Case : SI->cases())
+        Case.setSuccessor(ClonedPHs.find(Case.getCaseSuccessor())->second);
+      // We need to use the set to populate domtree updates as even when there
+      // are multiple cases pointing at the same successor we only want to
+      // insert one edge in the domtree.
+      for (BasicBlock *SuccBB : UnswitchedSuccBBs)
+        DTUpdates.push_back(
+            {DominatorTree::Insert, SplitBB, ClonedPHs.find(SuccBB)->second});
+
+      SI->setDefaultDest(LoopPH);
+    }
 
     // Create a new unconditional branch to the continuing block (as opposed to
     // the one cloned).
-    BranchInst::Create(ContinueSuccBB, ParentBB);
+    BranchInst::Create(RetainedSuccBB, ParentBB);
   } else {
+    assert(BI && "Only branches have partial unswitching.");
+    assert(UnswitchedSuccBBs.size() == 1 &&
+           "Only one possible unswitched block for a branch!");
+    BasicBlock *ClonedPH = ClonedPHs.begin()->second;
     // When doing a partial unswitch, we have to do a bit more work to build up
     // the branch in the split block.
     buildPartialUnswitchConditionalBranch(*SplitBB, Invariants, Direction,
                                           *ClonedPH, *LoopPH);
+    DTUpdates.push_back({DominatorTree::Insert, SplitBB, ClonedPH});
   }
 
-  // Before we update the dominator tree, collect the dead blocks if we're going
-  // to end up deleting the unswitched successor.
-  SmallVector<BasicBlock *, 16> DeadBlocks;
-  if (DeleteUnswitchedSucc) {
-    DeadBlocks.push_back(UnswitchedSuccBB);
-    for (int i = 0; i < (int)DeadBlocks.size(); ++i) {
-      // If we reach an exit block, stop recursing as the unswitched loop will
-      // end up reaching the merge block which we make the successor of the
-      // exit.
-      if (ExitBlockSet.count(DeadBlocks[i]))
-        continue;
-
-      // Insert the children that are within the loop or exit block set. Other
-      // children may reach out of the loop. While we don't expect these to be
-      // dead (as the unswitched clone should reach them) we don't try to prove
-      // that here.
-      for (DomTreeNode *ChildN : *DT[DeadBlocks[i]])
-        if (L.contains(ChildN->getBlock()) ||
-            ExitBlockSet.count(ChildN->getBlock()))
-          DeadBlocks.push_back(ChildN->getBlock());
-    }
-  }
-
-  // Add the remaining edge to our updates and apply them to get an up-to-date
-  // dominator tree. Note that this will cause the dead blocks above to be
-  // unreachable and no longer in the dominator tree.
-  DTUpdates.push_back({DominatorTree::Insert, SplitBB, ClonedPH});
+  // Apply the updates accumulated above to get an up-to-date dominator tree.
   DT.applyUpdates(DTUpdates);
 
+  // Now that we have an accurate dominator tree, first delete the dead cloned
+  // blocks so that we can accurately build any cloned loops. It is important to
+  // not delete the blocks from the original loop yet because we still want to
+  // reference the original loop to understand the cloned loop's structure.
+  deleteDeadClonedBlocks(L, ExitBlocks, VMaps, DT);
+
   // Build the cloned loop structure itself. This may be substantially
   // different from the original structure due to the simplified CFG. This also
   // handles inserting all the cloned blocks into the correct loops.
   SmallVector<Loop *, 4> NonChildClonedLoops;
-  buildClonedLoops(L, ExitBlocks, VMap, LI, NonChildClonedLoops);
-
-  // Delete anything that was made dead in the original loop due to
-  // unswitching.
-  if (!DeadBlocks.empty())
-    deleteDeadBlocksFromLoop(L, DeadBlocks, ExitBlocks, DT, LI);
+  for (std::unique_ptr<ValueToValueMapTy> &VMap : VMaps)
+    buildClonedLoops(L, ExitBlocks, *VMap, LI, NonChildClonedLoops);
 
+  // Now that our cloned loops have been built, we can update the original loop.
+  // First we delete the dead blocks from it and then we rebuild the loop
+  // structure taking these deletions into account.
+  deleteDeadBlocksFromLoop(L, ExitBlocks, DT, LI);
   SmallVector<Loop *, 4> HoistedLoops;
   bool IsStillLoop = rebuildLoopAfterUnswitch(L, ExitBlocks, LI, HoistedLoops);
 
   // This transformation has a high risk of corrupting the dominator tree, and
   // the below steps to rebuild loop structures will result in hard to debug
   // errors in that case so verify that the dominator tree is sane first.
   // FIXME: Remove this when the bugs stop showing up and rely on existing
   // verification steps.
   assert(DT.verify(DominatorTree::VerificationLevel::Fast));
 
-  // Now we want to replace all the uses of the invariants within both the
-  // original and cloned blocks. We do this here so that we can use the now
-  // updated dominator tree to identify which side the users are on.
+  if (BI) {
+    // If we unswitched a branch which collapses the condition to a known
+    // constant we want to replace all the uses of the invariants within both
+    // the original and cloned blocks. We do this here so that we can use the
+    // now updated dominator tree to identify which side the users are on.
+    assert(UnswitchedSuccBBs.size() == 1 &&
+           "Only one possible unswitched block for a branch!");
+    BasicBlock *ClonedPH = ClonedPHs.begin()->second;
     ConstantInt *UnswitchedReplacement =
-      Direction ? ConstantInt::getTrue(BI.getContext())
-                : ConstantInt::getFalse(BI.getContext());
+        Direction ? ConstantInt::getTrue(BI->getContext())
+                  : ConstantInt::getFalse(BI->getContext());
     ConstantInt *ContinueReplacement =
-      Direction ? ConstantInt::getFalse(BI.getContext())
-                : ConstantInt::getTrue(BI.getContext());
+        Direction ? ConstantInt::getFalse(BI->getContext())
+                  : ConstantInt::getTrue(BI->getContext());
     for (Value *Invariant : Invariants)
       for (auto UI = Invariant->use_begin(), UE = Invariant->use_end();
            UI != UE;) {
         // Grab the use and walk past it so we can clobber it in the use list.
         Use *U = &*UI++;
         Instruction *UserI = dyn_cast<Instruction>(U->getUser());
         if (!UserI)
           continue;
 
         // Replace it with the 'continue' side if in the main loop body, and the
         // unswitched if in the cloned blocks.
         if (DT.dominates(LoopPH, UserI->getParent()))
           U->set(ContinueReplacement);
         else if (DT.dominates(ClonedPH, UserI->getParent()))
           U->set(UnswitchedReplacement);
       }
+  }
 
   // We can change which blocks are exit blocks of all the cloned sibling
   // loops, the current loop, and any parent loops which shared exit blocks
   // with the current loop. As a consequence, we need to re-form LCSSA for
   // them. But we shouldn't need to re-form LCSSA for any child loops.
   // FIXME: This could be made more efficient by tracking which exit blocks are
   // new, and focusing on them, but that isn't likely to be necessary.
   //
static bool unswitchBestCondition(
   // Collect all invariant conditions within this loop (as opposed to an inner
   // loop which would be handled when visiting that inner loop).
   SmallVector<std::pair<TerminatorInst *, TinyPtrVector<Value *>>, 4>
       UnswitchCandidates;
   for (auto *BB : L.blocks()) {
     if (LI.getLoopFor(BB) != &L)
       continue;
 
+    if (auto *SI = dyn_cast<SwitchInst>(BB->getTerminator())) {
+      // We can only consider fully loop-invariant switch conditions as we need
+      // to completely eliminate the switch after unswitching.
+      if (!isa<Constant>(SI->getCondition()) &&
+          L.isLoopInvariant(SI->getCondition()))
+        UnswitchCandidates.push_back({SI, {SI->getCondition()}});
+      continue;
+    }
+
     auto *BI = dyn_cast<BranchInst>(BB->getTerminator());
-    // FIXME: Handle switches here!
     if (!BI || !BI->isConditional() || isa<Constant>(BI->getCondition()) ||
         BI->getSuccessor(0) == BI->getSuccessor(1))
       continue;
 
     if (L.isLoopInvariant(BI->getCondition())) {
       UnswitchCandidates.push_back({BI, {BI->getCondition()}});
       continue;
     }
static bool unswitchBestCondition(
   };
   TerminatorInst *BestUnswitchTI = nullptr;
   int BestUnswitchCost;
   ArrayRef<Value *> BestUnswitchInvariants;
   for (auto &TerminatorAndInvariants : UnswitchCandidates) {
     TerminatorInst &TI = *TerminatorAndInvariants.first;
     ArrayRef<Value *> Invariants = TerminatorAndInvariants.second;
     BranchInst *BI = dyn_cast<BranchInst>(&TI);
-    int CandidateCost =
-        ComputeUnswitchedCost(TI, /*FullUnswitch*/ Invariants.size() == 1 && BI &&
-                                      Invariants[0] == BI->getCondition());
+    int CandidateCost = ComputeUnswitchedCost(
+        TI, /*FullUnswitch*/ !BI || (Invariants.size() == 1 &&
+                                     Invariants[0] == BI->getCondition()));
     LLVM_DEBUG(dbgs() << "  Computed cost of " << CandidateCost
                       << " for unswitch candidate: " << TI << "\n");
     if (!BestUnswitchTI || CandidateCost < BestUnswitchCost) {
       BestUnswitchTI = &TI;
       BestUnswitchCost = CandidateCost;
       BestUnswitchInvariants = Invariants;
     }
   }
 
   if (BestUnswitchCost >= UnswitchThreshold) {
     LLVM_DEBUG(dbgs() << "Cannot unswitch, lowest cost found: "
                       << BestUnswitchCost << "\n");
     return false;
   }
 
-  auto *UnswitchBI = dyn_cast<BranchInst>(BestUnswitchTI);
-  if (!UnswitchBI) {
-    // FIXME: Add support for unswitching a switch here!
-    LLVM_DEBUG(dbgs() << "Cannot unswitch anything but a branch!\n");
-    return false;
-  }
-
   LLVM_DEBUG(dbgs() << "  Trying to unswitch non-trivial (cost = "
-                    << BestUnswitchCost << ") branch: " << *UnswitchBI << "\n");
-  return unswitchInvariantBranch(L, *UnswitchBI, BestUnswitchInvariants, DT, LI,
-                                 AC, UnswitchCB);
+                    << BestUnswitchCost << ") terminator: " << *BestUnswitchTI
+                    << "\n");
+  return unswitchNontrivialInvariants(
+      L, *BestUnswitchTI, BestUnswitchInvariants, DT, LI, AC, UnswitchCB);
 }
 
 /// Unswitch control flow predicated on loop invariant conditions.
 ///
 /// This first hoists all branches or switches which are trivial (IE, do not
 /// require duplicating any part of the loop) out of the loop body. It then
 /// looks at other loop invariant control flows and tries to unswitch those as
 /// well by cloning the loop if the result is small enough.

llvm/trunk/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll

 ;
 ; CHECK:       loop_exit.split.us:
 ; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B]], %loop_b.us ]
 ; CHECK-NEXT:    br label %loop_exit
 
 loop_b:
   %b = load i32, i32* %b.ptr
   br i1 %v, label %loop_begin, label %loop_exit
-; The 'loop_b' unswitched loop.
+; The original loop, now non-looping due to unswitching..
 ;
 ; CHECK:       entry.split:
 ; CHECK-NEXT:    br label %loop_begin
 ;
 ; CHECK:       loop_begin:
 ; CHECK-NEXT:    %[[V:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    %[[A:.*]] = load i32, i32* %a.ptr
 ; CHECK-NEXT:    br label %loop_exit.split
 ;
 ; CHECK:       loop_exit.split:
-; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A]], %loop_begin ]
 ; CHECK-NEXT:    br label %loop_exit
 
 loop_exit:
   %ab.phi = phi i32 [ %b, %loop_b ], [ %a, %loop_begin ]
   ret i32 %ab.phi
 ; CHECK:       loop_exit:
-; CHECK-NEXT:    %[[AB_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit.split ], [ %[[B_LCSSA]], %loop_exit.split.us ]
+; CHECK-NEXT:    %[[AB_PHI:.*]] = phi i32 [ %[[A]], %loop_exit.split ], [ %[[B_LCSSA]], %loop_exit.split.us ]
 ; CHECK-NEXT:    ret i32 %[[AB_PHI]]
 }
 
 ; Test a non-trivial unswitch of an exiting edge to an exit block with no other
 ; in-loop predecessors.
 define void @test4a(i1* %ptr, i1 %cond1, i32* %a.ptr, i32* %b.ptr) {
 ; CHECK-LABEL: @test4a(
 entry:

 ; CHECK-NEXT:    %[[B:.*]] = load i32, i32* %b.ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_begin, label %loop_exit2
 
 loop_exit1:
   %a.phi = phi i32 [ %a, %loop_begin ]
   call void @sink1(i32 %a.phi)
   ret void
 ; CHECK:       loop_exit1:
-; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit1.split.us ]
-; CHECK-NEXT:    call void @sink1(i32 %[[A_PHI]])
+; CHECK-NEXT:    call void @sink1(i32 %[[A_LCSSA]])
 ; CHECK-NEXT:    ret void
 
 loop_exit2:
   %b.phi = phi i32 [ %b, %loop_b ]
   call void @sink2(i32 %b.phi)
   ret void
 ; CHECK:       loop_exit2:
-; CHECK-NEXT:    %[[B_PHI:.*]] = phi i32 [ %[[B]], %loop_b ]
-; CHECK-NEXT:    call void @sink2(i32 %[[B_PHI]])
+; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B]], %loop_b ]
+; CHECK-NEXT:    call void @sink2(i32 %[[B_LCSSA]])
 ; CHECK-NEXT:    ret void
 }
 
 ; Test a non-trivial unswitch of an exiting edge to an exit block with no other
 ; in-loop predecessors. This is the same as @test4a but with the edges reversed
 ; so that the exiting edge is *not* the cloned edge.
 define void @test4b(i1* %ptr, i1 %cond1, i32* %a.ptr, i32* %b.ptr) {
 ; CHECK-LABEL: @test4b(

 ; CHECK-NEXT:    call void @sink1(i32 %[[A_PHI]])
 ; CHECK-NEXT:    ret void
 
 loop_exit2:
   %b.phi = phi i32 [ %b, %loop_b ]
   call void @sink2(i32 %b.phi)
   ret void
 ; CHECK:       loop_exit2:
-; CHECK-NEXT:    %[[B_PHI:.*]] = phi i32 [ %[[B_LCSSA]], %loop_exit2.split.us ]
-; CHECK-NEXT:    call void @sink2(i32 %[[B_PHI]])
+; CHECK-NEXT:    call void @sink2(i32 %[[B_LCSSA]])
 ; CHECK-NEXT:    ret void
 }
 
 ; Test a non-trivial unswitch of an exiting edge to an exit block with no other
 ; in-loop predecessors. This is the same as @test4a but with a common merge
 ; block after the independent loop exits. This requires a different structural
 ; update to the dominator tree.
 define void @test4c(i1* %ptr, i1 %cond1, i32* %a.ptr, i32* %b.ptr) {

 ; CHECK-NEXT:    %[[B:.*]] = load i32, i32* %b.ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_begin, label %loop_exit2
 
 loop_exit1:
   %a.phi = phi i32 [ %a, %loop_begin ]
   call void @sink1(i32 %a.phi)
   br label %exit
 ; CHECK:       loop_exit1:
-; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit1.split.us ]
-; CHECK-NEXT:    call void @sink1(i32 %[[A_PHI]])
+; CHECK-NEXT:    call void @sink1(i32 %[[A_LCSSA]])
 ; CHECK-NEXT:    br label %exit
 
 loop_exit2:
   %b.phi = phi i32 [ %b, %loop_b ]
   call void @sink2(i32 %b.phi)
   br label %exit
 ; CHECK:       loop_exit2:
-; CHECK-NEXT:    %[[B_PHI:.*]] = phi i32 [ %[[B]], %loop_b ]
-; CHECK-NEXT:    call void @sink2(i32 %[[B_PHI]])
+; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B]], %loop_b ]
+; CHECK-NEXT:    call void @sink2(i32 %[[B_LCSSA]])
 ; CHECK-NEXT:    br label %exit
 
 exit:
   ret void
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 }
 

 ; CHECK-NEXT:    %[[B:.*]] = load i32, i32* %b.ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %inner_loop_begin, label %loop_latch
 
 loop_latch:
   %b.phi = phi i32 [ %b, %inner_loop_b ]
   %v2 = load i1, i1* %ptr
   br i1 %v2, label %loop_begin, label %loop_exit
 ; CHECK:       loop_latch:
-; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B]], %inner_loop_b ]
+; CHECK-NEXT:    %[[B_INNER_LCSSA:.*]] = phi i32 [ %[[B]], %inner_loop_b ]
 ; CHECK-NEXT:    %[[V2:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    br i1 %[[V2]], label %loop_begin, label %loop_exit.loopexit1
 
 loop_exit:
   %ab.phi = phi i32 [ %a, %inner_loop_begin ], [ %b.phi, %loop_latch ]
   ret i32 %ab.phi
 ; CHECK:       loop_exit.loopexit:
-; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit.loopexit.split.us ]
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit.loopexit1:
-; CHECK-NEXT:    %[[B_PHI:.*]] = phi i32 [ %[[B_LCSSA]], %loop_latch ]
+; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B_INNER_LCSSA]], %loop_latch ]
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit:
-; CHECK-NEXT:    %[[AB_PHI:.*]] = phi i32 [ %[[A_PHI]], %loop_exit.loopexit ], [ %[[B_PHI]], %loop_exit.loopexit1 ]
+; CHECK-NEXT:    %[[AB_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit.loopexit ], [ %[[B_LCSSA]], %loop_exit.loopexit1 ]
 ; CHECK-NEXT:    ret i32 %[[AB_PHI]]
 }
 
 ; Test that we can unswitch a condition where we end up only cloning some of
 ; the nested loops and needing to delete some of the nested loops.
 define i32 @test6(i1* %ptr, i1 %cond1, i32* %a.ptr, i32* %b.ptr) {
 ; CHECK-LABEL: @test6(
 entry:

 ; CHECK-NEXT:    %[[B:.*]] = load i32, i32* %b.ptr
 ; CHECK-NEXT:    br i1 %[[VB]], label %loop_b_inner, label %loop_b_inner_exit
 ;
 ; CHECK:       loop_b_inner_exit:
 ; CHECK-NEXT:    %[[B_INNER_LCSSA:.*]] = phi i32 [ %[[B]], %loop_b_inner ]
 ; CHECK-NEXT:    br label %latch
 ;
 ; CHECK:       latch:
-; CHECK-NEXT:    %[[B_PHI:.*]] = phi i32 [ %[[B_INNER_LCSSA]], %loop_b_inner_exit ]
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_begin, label %loop_exit.split
 ;
 ; CHECK:       loop_exit.split:
-; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B_PHI]], %latch ]
+; CHECK-NEXT:    %[[B_LCSSA:.*]] = phi i32 [ %[[B_INNER_LCSSA]], %latch ]
 ; CHECK-NEXT:    br label %loop_exit
 
 loop_exit:
   %ab.lcssa = phi i32 [ %ab.phi, %latch ]
   ret i32 %ab.lcssa
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    %[[AB_PHI:.*]] = phi i32 [ %[[B_LCSSA]], %loop_exit.split ], [ %[[A_LCSSA]], %loop_exit.split.us ]
 ; CHECK-NEXT:    ret i32 %[[AB_PHI]]

 ; CHECK-NEXT:    call void @sink1(i32 %[[B_LCSSA]])
 ; CHECK-NEXT:    br label %inner_loop_begin
 
 inner_loop_exit:
   %a.inner_lcssa = phi i32 [ %a, %inner_loop_begin ]
   %v = load i1, i1* %ptr
   br i1 %v, label %loop_begin, label %loop_exit
 ; CHECK:       inner_loop_exit:
-; CHECK-NEXT:    %[[A_INNER_LCSSA:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %inner_loop_exit.split.us ]
 ; CHECK-NEXT:    %[[V:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_begin, label %loop_exit
 
 loop_exit:
   %a.lcssa = phi i32 [ %a.inner_lcssa, %inner_loop_exit ]
   ret i32 %a.lcssa
 ; CHECK:       loop_exit:
-; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A_INNER_LCSSA]], %inner_loop_exit ]
+; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %inner_loop_exit ]
 ; CHECK-NEXT:    ret i32 %[[A_LCSSA]]
 }
 
 ; Test that requires re-forming dedicated exits for the cloned loop.
 define i32 @test10a(i1* %ptr, i1 %cond, i32* %a.ptr) {
 ; CHECK-LABEL: @test10a(
 entry:
   br label %loop_begin

 ; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A]], %loop_a ]
 ; CHECK-NEXT:    br label %loop_exit
 
 loop_exit:
   %a.lcssa = phi i32 [ %a, %loop_a ], [ %a, %loop_b ]
   ret i32 %a.lcssa
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit.split ], [ %[[A_PHI_US]], %loop_exit.split.us ]
-; CHECK-NEXT:    ret i32 %[[AB_PHI]]
+; CHECK-NEXT:    ret i32 %[[A_PHI]]
 }
 
 ; Test that requires re-forming dedicated exits for the original loop.
 define i32 @test10b(i1* %ptr, i1 %cond, i32* %a.ptr) {
 ; CHECK-LABEL: @test10b(
 entry:
   br label %loop_begin
 ; CHECK-NEXT:  entry:

 ; CHECK-NEXT:    %[[A_PHI_SPLIT:.*]] = phi i32 [ %[[A_LCSSA_B]], %loop_b ], [ %[[A_LCSSA_A]], %loop_exit.split.loopexit ]
 ; CHECK-NEXT:    br label %loop_exit
 
 loop_exit:
   %a.lcssa = phi i32 [ %a, %loop_a ], [ %a, %loop_b ]
   ret i32 %a.lcssa
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_PHI_SPLIT]], %loop_exit.split ], [ %[[A_LCSSA_US]], %loop_exit.split.us ]
-; CHECK-NEXT:    ret i32 %[[AB_PHI]]
+; CHECK-NEXT:    ret i32 %[[A_PHI]]
 }
 
 ; Check that if a cloned inner loop after unswitching doesn't loop and directly
 ; exits even an outer loop, we don't add the cloned preheader to the outer
 ; loop and do add the needed LCSSA phi nodes for the new exit block from the
 ; outer loop.
 define i32 @test11a(i1* %ptr, i1* %cond.ptr, i32* %a.ptr, i32* %b.ptr) {
 ; CHECK-LABEL: @test11a(
loop_latch:
   br label %loop_begin
 ; CHECK:       loop_latch:
 ; CHECK-NEXT:    br label %loop_begin
 
 loop_exit:
   %a.lcssa = phi i32 [ %a, %inner_loop_begin ], [ %a.inner_lcssa, %inner_loop_exit ]
   ret i32 %a.lcssa
 ; CHECK:       loop_exit.loopexit:
-; CHECK-NEXT:    %[[A_LCSSA_US:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %loop_exit.loopexit.split.us ]
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit.loopexit1:
 ; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A_INNER_LCSSA]], %inner_loop_exit ]
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit:
-; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_LCSSA_US]], %loop_exit.loopexit ], [ %[[A_LCSSA]], %loop_exit.loopexit1 ]
+; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %loop_exit.loopexit ], [ %[[A_LCSSA]], %loop_exit.loopexit1 ]
 ; CHECK-NEXT:    ret i32 %[[A_PHI]]
 }
 
 ; Check that if the original inner loop after unswitching doesn't loop and
 ; directly exits even an outer loop, we remove the original preheader from the
 ; outer loop and add needed LCSSA phi nodes for the new exit block from the
 ; outer loop.
 define i32 @test11b(i1* %ptr, i1* %cond.ptr, i32* %a.ptr, i32* %b.ptr) {

 ; CHECK-NEXT:    %[[A:.*]] = load i32, i32* %a.ptr
 ; CHECK-NEXT:    br label %loop_exit.loopexit
 
 inner_loop_exit:
   %a.inner_lcssa = phi i32 [ %a, %inner_loop_a ]
   %v3 = load i1, i1* %ptr
   br i1 %v3, label %loop_latch, label %loop_exit
 ; CHECK:       inner_loop_exit:
-; CHECK-NEXT:    %[[A_INNER_PHI:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %inner_loop_exit.split.us ]
 ; CHECK-NEXT:    %[[V:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_latch, label %loop_exit.loopexit1
 
 loop_latch:
   br label %loop_begin
 ; CHECK:       loop_latch:
 ; CHECK-NEXT:    br label %loop_begin
 
 loop_exit:
   %a.lcssa = phi i32 [ %a, %inner_loop_begin ], [ %a.inner_lcssa, %inner_loop_exit ]
   ret i32 %a.lcssa
 ; CHECK:       loop_exit.loopexit:
 ; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A]], %inner_loop_begin ]
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit.loopexit1:
-; CHECK-NEXT:    %[[A_LCSSA_US:.*]] = phi i32 [ %[[A_INNER_PHI]], %inner_loop_exit ]
+; CHECK-NEXT:    %[[A_LCSSA_US:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %inner_loop_exit ]
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    %[[A_PHI:.*]] = phi i32 [ %[[A_LCSSA]], %loop_exit.loopexit ], [ %[[A_LCSSA_US]], %loop_exit.loopexit1 ]
 ; CHECK-NEXT:    ret i32 %[[A_PHI]]
 }
 
 ; Like test11a, but checking that when the whole thing is wrapped in yet

 ; CHECK:       inner_loop_latch:
 ; CHECK-NEXT:    br label %inner_loop_begin
 
 inner_loop_exit:
   %a.inner_lcssa = phi i32 [ %a, %inner_inner_loop_begin ], [ %a.inner_inner_lcssa, %inner_inner_loop_exit ]
   %v4 = load i1, i1* %ptr
   br i1 %v4, label %loop_begin, label %loop_exit
 ; CHECK:       inner_loop_exit.loopexit:
-; CHECK-NEXT:    %[[A_INNER_LCSSA_US:.*]] = phi i32 [ %[[A_INNER_INNER_LCSSA_US]], %inner_loop_exit.loopexit.split.us ]
 ; CHECK-NEXT:    br label %inner_loop_exit
 ;
 ; CHECK:       inner_loop_exit.loopexit1:
 ; CHECK-NEXT:    %[[A_INNER_LCSSA:.*]] = phi i32 [ %[[A_INNER_INNER_LCSSA]], %inner_inner_loop_exit ]
 ; CHECK-NEXT:    br label %inner_loop_exit
 ;
 ; CHECK:       inner_loop_exit:
-; CHECK-NEXT:    %[[A_INNER_PHI:.*]] = phi i32 [ %[[A_INNER_LCSSA_US]], %inner_loop_exit.loopexit ], [ %[[A_INNER_LCSSA]], %inner_loop_exit.loopexit1 ]
+; CHECK-NEXT:    %[[A_INNER_PHI:.*]] = phi i32 [ %[[A_INNER_INNER_LCSSA_US]], %inner_loop_exit.loopexit ], [ %[[A_INNER_LCSSA]], %inner_loop_exit.loopexit1 ]
 ; CHECK-NEXT:    %[[V:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_begin, label %loop_exit
 
 loop_exit:
   %a.lcssa = phi i32 [ %a.inner_lcssa, %inner_loop_exit ]
   ret i32 %a.lcssa
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    %[[A_LCSSA:.*]] = phi i32 [ %[[A_INNER_PHI]], %inner_loop_exit ]

 ; CHECK-NEXT:    %[[A:.*]] = load i32, i32* %a.ptr
 ; CHECK-NEXT:    br label %inner_loop_exit.loopexit
 
 inner_inner_loop_exit:
   %a.inner_inner_lcssa = phi i32 [ %a, %inner_inner_loop_a ]
   %v3 = load i1, i1* %ptr
   br i1 %v3, label %inner_loop_latch, label %inner_loop_exit
 ; CHECK:       inner_inner_loop_exit:
-; CHECK-NEXT:    %[[A_INNER_INNER_PHI:.*]] = phi i32 [ %[[A_INNER_INNER_LCSSA_US]], %inner_inner_loop_exit.split.us ]
 ; CHECK-NEXT:    %[[V:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %inner_loop_latch, label %inner_loop_exit.loopexit1
 
 inner_loop_latch:
   br label %inner_loop_begin
 ; CHECK:       inner_loop_latch:
 ; CHECK-NEXT:    br label %inner_loop_begin
 
 inner_loop_exit:
   %a.inner_lcssa = phi i32 [ %a, %inner_inner_loop_begin ], [ %a.inner_inner_lcssa, %inner_inner_loop_exit ]
   %v4 = load i1, i1* %ptr
   br i1 %v4, label %loop_begin, label %loop_exit
 ; CHECK:       inner_loop_exit.loopexit:
 ; CHECK-NEXT:    %[[A_INNER_LCSSA:.*]] = phi i32 [ %[[A]], %inner_inner_loop_begin ]
 ; CHECK-NEXT:    br label %inner_loop_exit
 ;
 ; CHECK:       inner_loop_exit.loopexit1:
-; CHECK-NEXT:    %[[A_INNER_LCSSA_US:.*]] = phi i32 [ %[[A_INNER_INNER_PHI]], %inner_inner_loop_exit ]
+; CHECK-NEXT:    %[[A_INNER_LCSSA_US:.*]] = phi i32 [ %[[A_INNER_INNER_LCSSA_US]], %inner_inner_loop_exit ]
 ; CHECK-NEXT:    br label %inner_loop_exit
 ;
 ; CHECK:       inner_loop_exit:
 ; CHECK-NEXT:    %[[A_INNER_PHI:.*]] = phi i32 [ %[[A_INNER_LCSSA]], %inner_loop_exit.loopexit ], [ %[[A_INNER_LCSSA_US]], %inner_loop_exit.loopexit1 ]
 ; CHECK-NEXT:    %[[V:.*]] = load i1, i1* %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_begin, label %loop_exit
 
 loop_exit:

 ; CHECK-NEXT:    ret i32 %[[AB_PHI]]
 }
 
 define i32 @test20(i32* %var, i32 %cond1, i32 %cond2) {
 ; CHECK-LABEL: @test20(
 entry:
   br label %loop_begin
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    br label %loop_begin
+; CHECK-NEXT:    switch i32 %cond2, label %[[ENTRY_SPLIT_EXIT:.*]] [
+; CHECK-NEXT:      i32 0, label %[[ENTRY_SPLIT_A:.*]]
+; CHECK-NEXT:      i32 1, label %[[ENTRY_SPLIT_A]]
+; CHECK-NEXT:      i32 13, label %[[ENTRY_SPLIT_B:.*]]
+; CHECK-NEXT:      i32 2, label %[[ENTRY_SPLIT_A]]
+; CHECK-NEXT:      i32 42, label %[[ENTRY_SPLIT_C:.*]]
+; CHECK-NEXT:    ]
 
 loop_begin:
   %var_val = load i32, i32* %var
-  switch i32 %cond2, label %loop_a [
-    i32 0, label %loop_b
-    i32 1, label %loop_b
-    i32 13, label %loop_c
-    i32 2, label %loop_b
-    i32 42, label %loop_exit
+  switch i32 %cond2, label %loop_exit [
+    i32 0, label %loop_a
+    i32 1, label %loop_a
+    i32 13, label %loop_b
+    i32 2, label %loop_a
+    i32 42, label %loop_c
   ]
-; CHECK:       loop_begin:
-; CHECK-NEXT:    %[[V:.*]] = load i32, i32* %var
-; CHECK-NEXT:    switch i32 %cond2, label %loop_a [
-; CHECK-NEXT:      i32 0, label %loop_b
-; CHECK-NEXT:      i32 1, label %loop_b
-; CHECK-NEXT:      i32 13, label %loop_c
-; CHECK-NEXT:      i32 2, label %loop_b
-; CHECK-NEXT:      i32 42, label %loop_exit
-; CHECK-NEXT:    ]
 
 loop_a:
   call void @a()
   br label %loop_latch
-; CHECK:       loop_a:
+; Unswitched 'a' loop.
+;
+; CHECK:       [[ENTRY_SPLIT_A]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_A:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_A]]:
+; CHECK-NEXT:    %{{.*}} = load i32, i32* %var
+; CHECK-NEXT:    br label %[[LOOP_A:.*]]
+;
+; CHECK:       [[LOOP_A]]:
 ; CHECK-NEXT:    call void @a()
-; CHECK-NEXT:    br label %loop_latch
+; CHECK-NEXT:    br label %[[LOOP_LATCH_A:.*]]
+;
+; CHECK:       [[LOOP_LATCH_A]]:
+; CHECK:         br label %[[LOOP_BEGIN_A]]
 
 loop_b:
   call void @b()
   br label %loop_latch
-; CHECK:       loop_b:
+; Unswitched 'b' loop.
+;
+; CHECK:       [[ENTRY_SPLIT_B]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_B:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_B]]:
+; CHECK-NEXT:    %{{.*}} = load i32, i32* %var
+; CHECK-NEXT:    br label %[[LOOP_B:.*]]
+;
+; CHECK:       [[LOOP_B]]:
 ; CHECK-NEXT:    call void @b()
-; CHECK-NEXT:    br label %loop_latch
+; CHECK-NEXT:    br label %[[LOOP_LATCH_B:.*]]
+;
+; CHECK:       [[LOOP_LATCH_B]]:
+; CHECK:         br label %[[LOOP_BEGIN_B]]
 
 loop_c:
   call void @c() noreturn nounwind
   br label %loop_latch
-; CHECK:       loop_c:
+; Unswitched 'c' loop.
+;
+; CHECK:       [[ENTRY_SPLIT_C]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_C:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_C]]:
+; CHECK-NEXT:    %{{.*}} = load i32, i32* %var
+; CHECK-NEXT:    br label %[[LOOP_C:.*]]
+;
+; CHECK:       [[LOOP_C]]:
 ; CHECK-NEXT:    call void @c()
-; CHECK-NEXT:    br label %loop_latch
+; CHECK-NEXT:    br label %[[LOOP_LATCH_C:.*]]
+;
+; CHECK:       [[LOOP_LATCH_C]]:
+; CHECK:         br label %[[LOOP_BEGIN_C]]
 
 loop_latch:
   br label %loop_begin
-; CHECK:       loop_latch:
-; CHECK-NEXT:    br label %loop_begin
 
 loop_exit:
   %lcssa = phi i32 [ %var_val, %loop_begin ]
   ret i32 %lcssa
+; Unswitched exit edge (no longer a loop).
+;
+; CHECK:       [[ENTRY_SPLIT_EXIT]]:
+; CHECK-NEXT:    br label %loop_begin
+;
+; CHECK:       loop_begin:
+; CHECK-NEXT:    %[[V:.*]] = load i32, i32* %var
+; CHECK-NEXT:    br label %loop_exit
+;
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    %[[LCSSA:.*]] = phi i32 [ %[[V]], %loop_begin ]
 ; CHECK-NEXT:    ret i32 %[[LCSSA]]
 }
 
 ; Negative test: we do not switch when the loop contains unstructured control
 ; flows as it would significantly complicate the process as novel loops might
 ; be formed, etc.

 loop_exit:
   ret i32 0
 ; CHECK:       loop_exit.split:
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    ret
 }
+
+; Non-trivial unswitching of a switch.
+define i32 @test27(i1* %ptr, i32 %cond) {
+; CHECK-LABEL: @test27(
+entry:
+  br label %loop_begin
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    switch i32 %cond, label %[[ENTRY_SPLIT_LATCH:.*]] [
+; CHECK-NEXT:      i32 0, label %[[ENTRY_SPLIT_A:.*]]
+; CHECK-NEXT:      i32 1, label %[[ENTRY_SPLIT_B:.*]]
+; CHECK-NEXT:      i32 2, label %[[ENTRY_SPLIT_C:.*]]
+; CHECK-NEXT:    ]
+
+loop_begin:
+  switch i32 %cond, label %latch [
+    i32 0, label %loop_a
+    i32 1, label %loop_b
+    i32 2, label %loop_c
+  ]
+
+loop_a:
+  call void @a()
+  br label %latch
+; Unswitched 'a' loop.
+;
+; CHECK:       [[ENTRY_SPLIT_A]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_A:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_A]]:
+; CHECK-NEXT:    br label %[[LOOP_A:.*]]
+;
+; CHECK:       [[LOOP_A]]:
+; CHECK-NEXT:    call void @a()
+; CHECK-NEXT:    br label %[[LOOP_LATCH_A:.*]]
+;
+; CHECK:       [[LOOP_LATCH_A]]:
+; CHECK-NEXT:    %[[V_A:.*]] = load i1, i1* %ptr
+; CHECK:         br i1 %[[V_A]], label %[[LOOP_BEGIN_A]], label %[[LOOP_EXIT_A:.*]]
+;
+; CHECK:       [[LOOP_EXIT_A]]:
+; CHECK-NEXT:    br label %loop_exit
+
+loop_b:
+  call void @b()
+  br label %latch
+; Unswitched 'b' loop.
+;
+; CHECK:       [[ENTRY_SPLIT_B]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_B:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_B]]:
+; CHECK-NEXT:    br label %[[LOOP_B:.*]]
+;
+; CHECK:       [[LOOP_B]]:
+; CHECK-NEXT:    call void @b()
+; CHECK-NEXT:    br label %[[LOOP_LATCH_B:.*]]
+;
+; CHECK:       [[LOOP_LATCH_B]]:
+; CHECK-NEXT:    %[[V_B:.*]] = load i1, i1* %ptr
+; CHECK:         br i1 %[[V_B]], label %[[LOOP_BEGIN_B]], label %[[LOOP_EXIT_B:.*]]
+;
+; CHECK:       [[LOOP_EXIT_B]]:
+; CHECK-NEXT:    br label %loop_exit
+
+loop_c:
+  call void @c()
+  br label %latch
+; Unswitched 'c' loop.
+;
+; CHECK:       [[ENTRY_SPLIT_C]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_C:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_C]]:
+; CHECK-NEXT:    br label %[[LOOP_C:.*]]
+;
+; CHECK:       [[LOOP_C]]:
+; CHECK-NEXT:    call void @c()
+; CHECK-NEXT:    br label %[[LOOP_LATCH_C:.*]]
+;
+; CHECK:       [[LOOP_LATCH_C]]:
+; CHECK-NEXT:    %[[V_C:.*]] = load i1, i1* %ptr
+; CHECK:         br i1 %[[V_C]], label %[[LOOP_BEGIN_C]], label %[[LOOP_EXIT_C:.*]]
+;
+; CHECK:       [[LOOP_EXIT_C]]:
+; CHECK-NEXT:    br label %loop_exit
+
+latch:
+  %v = load i1, i1* %ptr
+  br i1 %v, label %loop_begin, label %loop_exit
+; Unswitched the 'latch' only loop.
+;
+; CHECK:       [[ENTRY_SPLIT_LATCH]]:
+; CHECK-NEXT:    br label %[[LOOP_BEGIN_LATCH:.*]]
+;
+; CHECK:       [[LOOP_BEGIN_LATCH]]:
+; CHECK-NEXT:    br label %[[LOOP_LATCH_LATCH:.*]]
+;
+; CHECK:       [[LOOP_LATCH_LATCH]]:
+; CHECK-NEXT:    %[[V_LATCH:.*]] = load i1, i1* %ptr
+; CHECK:         br i1 %[[V_LATCH]], label %[[LOOP_BEGIN_LATCH]], label %[[LOOP_EXIT_LATCH:.*]]
+;
+; CHECK:       [[LOOP_EXIT_LATCH]]:
+; CHECK-NEXT:    br label %loop_exit
+
+loop_exit:
+  ret i32 0
+; CHECK:       loop_exit:
+; CHECK-NEXT:    ret i32 0
+}
 No newline at end of file