[SimplifyCFG] Tail-merging all blocks with `unreachable` terminator, final take

[lebedev.ri]

Summary

This implements the approach disscussed in D104870:
instead of simply alaways tail-merging all unreachable blocks,
first try to group the calls that precede unreachable,
and only merge the ones where grouping succeeded.

https://llvm-compile-time-tracker.com/compare.php?from=3564551400224cd24dd8650dc2ace19174833af7&to=a4fe8518811df84d14a21071f3516ec3841cd369&stat=instructions

[lebedev.ri]

Numbers are in:
D104870 was: (as of https://reviews.llvm.org/D104870#inline-998899)
https://llvm-compile-time-tracker.com/compare.php?from=1f169a774cb865659cefe085e70a56a884e3711e&to=fc54bb9a8ef85bd76dd9e934b2546f4beadc5b5e&stat=instructions
This now is: https://llvm-compile-time-tracker.com/compare.php?from=2353e1c87b09c20e75f0f3ceb05fa4a4261fe3dd&to=bed7b8df4565f4503889a19235e853b985ca3481&stat=instructions

So slightly better compile-time-wise, basically the same size impact.

[rnk]

I think I prefer this version over the previous. This may be slightly ugly, but IMO it is functionally better (fewer analysis reruns and pass updates). Maybe others have ideas for how to make it nicer.

What do other folks (@nikic @aeubanks @asbirlea) think about this approach?

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

It looks to me like tail merging unreachable blocks is preventing vectorization in this test case and the next, which seems like a blocking issue. The test was added here, if that helps understand why it no longer works:
https://reviews.llvm.org/rG39cc0b8c68b8d316954ecfac0d1f8498ea42866c
@fhahn

[lebedev.ri]

In D116692#3223679, @rnk wrote:

I think I prefer this version over the previous. This may be slightly ugly, but IMO it is functionally better (fewer analysis reruns and pass updates).

I think one big reason why this is indeed better is because now it will be trivial to introduce
(and pass here) bool SkipProfitabilityChecks option to SinkCommonCodeFromPredecessors().

Maybe others have ideas for how to make it nicer.

Looking at it, it's actually not *that* ugly as it seemed when i was writing the code.
I guess i could wrap this tail-merging/undo into a class, not sure if that would help.

What do other folks (@nikic @aeubanks @asbirlea) think about this approach?

[lebedev.ri]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

Filed https://github.com/llvm/llvm-project/issues/53020

[xbolva00]

llvm/lib/Transforms/Scalar/SimplifyCFGPass.cpp

238–239

Fix comment

[lebedev.ri]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

CC @aqjune @nikic

I'm not actually sure that we can solve this within LV itself,
but i would love to be proven wrong here.

I'm pretty sure LV does expand the backedge taken count,
so i suppose only not being allowed to expand BTC wouldn't help here.

As i see it, the options are:

• ignore this failure
• adjust the test to mask the failure (i would hope adding noundef's should help?), potentially coupled with:
• are there some missing reasoning bits in impliesPoison() and friends that could prevent this regression?
• Introduce UB-safe mode for SCEVExpander, lift backedge taken count poison-safety restriction
• Prevent simplifycfg from merging conditions like that (as in, iff plain and/or isn't going to be used)

[nikic]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

I think the proper way to address this (and other poison-safety issues in SCEV) is to add umin variants in both IR and SCEV that don't propagate op2 poison if op1 is zero.

[lebedev.ri]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

Posted D116766

[lebedev.ri]

Ok, the regression has been dealt with :)

[nikic]

It looks like the new version still has the large code size regressions (9% on mafft, 3% on 7zip). I understand that some code size increase is expected (and intended), but I don't think a particularly good case for the tradeoff has been made yet (in terms of where / how much performance this is buying for more code size). Though maybe I missed this in previous discussion threads.

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

[lebedev.ri]

In D116692#3232485, @nikic wrote:

It looks like the new version still has the large code size regressions (9% on mafft, 3% on 7zip). I understand that some code size increase is expected (and intended), but I don't think a particularly good case for the tradeoff has been made yet (in terms of where / how much performance this is buying for more code size). Though maybe I missed this in previous discussion threads.

How do we know that whatever compile time benchmark we see regresses is a reliable indicator in this regard?
I think this is yet another irresolvable clash between the optimization and compilation time/size.
-O3 does not mean "please quickly give me minimal code", there's -Os/-Oz for that.
IOW if you indent to block a patch, could you please actually do so, not just waguley imply so?

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

Were you able to to so?

[aeubanks]

In D116692#3249159, @lebedev.ri wrote:

In D116692#3232485, @nikic wrote:

It looks like the new version still has the large code size regressions (9% on mafft, 3% on 7zip). I understand that some code size increase is expected (and intended), but I don't think a particularly good case for the tradeoff has been made yet (in terms of where / how much performance this is buying for more code size). Though maybe I missed this in previous discussion threads.

How do we know that whatever compile time benchmark we see regresses is a reliable indicator in this regard?
I think this is yet another irresolvable clash between the optimization and compilation time/size.
-O3 does not mean "please quickly give me minimal code", there's -Os/-Oz for that.
IOW if you indent to block a patch, could you please actually do so, not just waguley imply so?

As somebody without much context for this patch reading the patch summary, there's no "why" for this patch. The llvm-compile-time-tracker numbers are all negative in various aspects. I think nikic's question is do you have metrics showing that this actually helps performance on any code?

[nikic]

In D116692#3249159, @lebedev.ri wrote:

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

Were you able to to so?

Sorry for the delay. I tested this together with your recent commit removing sinking limitations for unreachable blocks. The result was close to no change in either compile-time or run-time (where "run-time" here is non-LLVM compile-time, so a fairly narrow workload). All sub-1% and mostly below the significance threshold. Unfortunately I wasn't able to get code size information because the necessary infrastructure was broken at the time. I did look at some rustc artifacts as a sanity check and the only larger increase I spotted was rustdoc by +0.4%.

I played with the patch a bit, and found that this approach has one major limitation as far as rust code is concerned: It only works if you have a single assert/panic/whatever function. All unreachable terminators are merged together and we can then only sink if the predecessors have the same call. Rust has a bunch of different panic functions depending on the situation. So if you have a bunch of array accesses that all call panic_bounds_check and then add a single assert, then the tail merging stops working. I expect that significantly limits the cases where the optimization applies.

[lebedev.ri]

In D116692#3265576, @nikic wrote:

In D116692#3249159, @lebedev.ri wrote:

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

Were you able to to so?

Sorry for the delay. I tested this together with your recent commit removing sinking limitations for unreachable blocks.
The result was close to no change in either compile-time or run-time (where "run-time" here is non-LLVM compile-time,
so a fairly narrow workload). All sub-1% and mostly below the significance threshold. Unfortunately I wasn't able to
get code size information because the necessary infrastructure was broken at the time.
I did look at some rustc artifacts as a sanity check and the only larger increase I spotted was rustdoc by +0.4%.

I played with the patch a bit, and found that this approach has one major limitation as far as rust code is concerned:
It only works if you have a single assert/panic/whatever function.
All unreachable terminators are merged together and we can then only sink if the predecessors have the same call.
Rust has a bunch of different panic functions depending on the situation.
So if you have a bunch of array accesses that all call panic_bounds_check and then add a single assert,
then the tail merging stops working. I expect that significantly limits the cases where the optimization applies.

Since asking that back then, i've come up with D117805, which indeed handles said more generic case,
and after that patch is accepted i'll rewrite this patch to use that new infra.

[lebedev.ri]

Now that the 'merge compatible invokes' is effectively done, let's revisit this.

I've reimplemented this using the approach innovated/invented there,
so now not only we don't just tail-merge everything,
not only do we only do that when sinking succeeds,
we now also handle multiple sets of mergeable calls.

[nikic]

I've tested the newest version against rust, and unfortunately it came back as a universal regression, both in terms of compile-time and run-time. Compile-time regressions up to 4% and run-time regressions in the ~1% range (not by much, but very consistently regressing).

llvm/include/llvm/Transforms/Utils/Local.h

 
 /// BB is known to contain an unconditional branch, and contains no instructions
 /// other than PHI nodes, potential debug intrinsics and the branch. If
 /// possible, eliminate BB by rewriting all the predecessors to branch to the
 /// successor block and return true. If we can't transform, return false.
 bool TryToSimplifyUncondBranchFromEmptyBlock(BasicBlock *BB,
                                              DomTreeUpdater *DTU = nullptr);
 
+/// Check whether BB's predecessors end with unconditional branches. If it is
+/// true, sink any common code from the predecessors to BB.
+/// Returns true if any changes were made.
+bool SinkCommonCodeFromPredecessors(BasicBlock *BB,
+                                    DomTreeUpdater *DTU = nullptr);
+
 /// Check for and eliminate duplicate PHI nodes in this block. This doesn't try
 /// to be clever about PHI nodes which differ only in the order of the incoming
 /// values, but instcombine orders them so it usually won't matter.
 bool EliminateDuplicatePHINodes(BasicBlock *BB);
 
 /// This function is used to do simplification of a CFG.  For example, it
 /// adjusts branches to branches to eliminate the extra hop, it eliminates
 /// unreachable basic blocks, and does other peephole optimization of the CFG.

llvm/lib/Transforms/Scalar/SimplifyCFGPass.cpp

 //   * Eliminates a basic block that only contains an unconditional branch.
 //   * Changes invoke instructions to nounwind functions to be calls.
 //   * Change things like "if (x) if (y)" into "if (x&y)".
 //   * etc..
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/MapVector.h"
+#include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/CFG.h"
 #include "llvm/Analysis/DomTreeUpdater.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/TargetTransformInfo.h"

 
 static cl::opt<bool> UserSinkCommonInsts(
     "sink-common-insts", cl::Hidden, cl::init(false),
     cl::desc("Sink common instructions (default = false)"));
 
 
 STATISTIC(NumSimpl, "Number of blocks simplified");
 
-static bool
-performBlockTailMerging(Function &F, ArrayRef<BasicBlock *> BBs,
+static BasicBlock * /*CanonicalBB*/
+performBlockTailMergingImpl(Function &F, ArrayRef<BasicBlock *> BBs,
+                            SmallVectorImpl<Instruction *> &OrigTerminators,
                             std::vector<DominatorTree::UpdateType> *Updates) {
   SmallVector<PHINode *, 1> NewOps;
 
   // We don't want to change IR just because we can.
   // Only do that if there are at least two blocks we'll tail-merge.
   if (BBs.size() < 2)
-    return false;
+    return nullptr;
 
   if (Updates)
     Updates->reserve(Updates->size() + BBs.size());
 
   BasicBlock *CanonicalBB;
   Instruction *CanonicalTerm;
   {
     auto *Term = BBs[0]->getTerminator();
for (BasicBlock *BB : BBs) {
     if (!CommonDebugLoc)
       CommonDebugLoc = Term->getDebugLoc();
     else
       CommonDebugLoc =
           DILocation::getMergedLocation(CommonDebugLoc, Term->getDebugLoc());
 
     // And turn BB into a block that just unconditionally branches
     // to the canonical block.
-    Term->eraseFromParent();
+    OrigTerminators.emplace_back(Term);
+    Term->removeFromParent();
     BranchInst::Create(CanonicalBB, BB);
     if (Updates)
       Updates->push_back({DominatorTree::Insert, BB, CanonicalBB});
   }
 
   CanonicalTerm->setDebugLoc(CommonDebugLoc);
 
-  return true;
+  return CanonicalBB;
+}
+
+static bool /*Changed*/
+performBlockTailMerging(Function &F, ArrayRef<BasicBlock *> BBs,
+                        std::vector<DominatorTree::UpdateType> *Updates) {
+  unsigned TermOpc = BBs[0]->getTerminator()->getOpcode();
+  const size_t OrigUpdatesSize = Updates ? Updates->size() : -1;
+
+  SmallVector<Instruction *> OrigTerminators;
+  auto _ = make_scope_exit([&OrigTerminators]() {
+    for (Instruction *Term : OrigTerminators)
+      Term->deleteValue();
+  });
+  OrigTerminators.reserve(BBs.size());
+
+  BasicBlock *CanonicalBB =
+      performBlockTailMergingImpl(F, BBs, OrigTerminators, Updates);
+  if (!CanonicalBB) // Did we fail to tail-merge?
+    return CanonicalBB;
+
+  assert(CanonicalBB->getTerminator()->getOpcode() == TermOpc &&
+         "Tail-folding does not change the terminator type.");
+
+  // If we aren't dealing with the `unreachable` terminator, then that's it!
+  if (TermOpc != Instruction::Unreachable)
+    return CanonicalBB;
+
+  // For `unreachable`, however, we have more to do. We don't want to just merge
+  // all `unreachable` terminators, we want to do so if that allows us to sink
+  // some common code from them. So now we need to manually run
+  // common code sinking, and if that fails, undo tail merging.
+
+  // We intentionally do not pass DomTreeUpdater here, because it is only needed
+  // there to split conditional edges to CanonicalBB, but we know there are none
+  // and we haven't applied Updates yet so DomTreeUpdater is not up to date.
+  if (SinkCommonCodeFromPredecessors(CanonicalBB, /*DTU=*/nullptr))
+    return CanonicalBB; // Awesome, sinking succeeded, we are all good!
+
+  // Nope, nothing sunk. Need to backtrack.
+
+  assert(&*CanonicalBB->begin() == CanonicalBB->getTerminator() &&
+         "CanonicalBB only contains the terminator.");
+
+  // First, drop all the DomTree updates related to this tail-folding.
+  if (Updates) {
+    assert(Updates->size() >= OrigUpdatesSize);
+    Updates->resize(OrigUpdatesSize, {DominatorTree::Insert, nullptr, nullptr});
+  }
+
+  for (auto I : zip(BBs, OrigTerminators)) {
+    BasicBlock *PredBB = std::get<0>(I);
+    Instruction *OrigTerm = std::get<1>(I);
+    auto *CurrBr = dyn_cast<BranchInst>(PredBB->getTerminator());
+    assert(CurrBr && CurrBr->isUnconditional() &&
+           CurrBr->getSuccessor(0) == CanonicalBB &&
+           "All of BBs now unconditionally branch to CanonicalBB.");
+    OrigTerm->insertBefore(CurrBr);
+    CurrBr->eraseFromParent();
+  }
+  OrigTerminators.clear(); // Defuse scope-exit.
+  assert(pred_empty(CanonicalBB) && "CanonicalBB is now unreachable.");
+  CanonicalBB->eraseFromParent();
+
+  return false; // Did not tail-merge after all.
+  // Note that we indeed are allowed to return false here,
+  // because we've made sure that all IR and CFG changes were undone.
 }
 
-static bool tailMergeBlocksWithSimilarFunctionTerminators(Function &F,
-                                                          DomTreeUpdater *DTU) {
+static bool tailMergeBlocksWithSimilarFunctionTerminators(
+    Function &F, DomTreeUpdater *DTU, const SimplifyCFGOptions &Options) {
   SmallMapVector<unsigned /*TerminatorOpcode*/, SmallVector<BasicBlock *, 2>, 4>
       Structure;
 
   // Scan all the blocks in the function, record the interesting-ones.
   for (BasicBlock &BB : F) {
     if (DTU && DTU->isBBPendingDeletion(&BB))
       continue;
 
     // We are only interested in function-terminating blocks.
     if (!succ_empty(&BB))
       continue;
 
     auto *Term = BB.getTerminator();
 
-    // Fow now only support `ret`/`resume` function terminators.
-    // FIXME: lift this restriction.
+    // We currently only support `ret`/`resume` function terminators,
+    // and seldomly handle `unreachable` iff that allows sinking instructions.
     switch (Term->getOpcode()) {
     case Instruction::Ret:
     case Instruction::Resume:
       break;
+    case Instruction::Unreachable:
+      if (Options.SinkCommonInsts)
+        break;
+      continue;
     default:
       continue;
     }
 
     // We can't tail-merge block that contains a musttail call.
     if (BB.getTerminatingMustTailCall())
       continue;
 

 }
 
 static bool simplifyFunctionCFGImpl(Function &F, const TargetTransformInfo &TTI,
                                     DominatorTree *DT,
                                     const SimplifyCFGOptions &Options) {
   DomTreeUpdater DTU(DT, DomTreeUpdater::UpdateStrategy::Eager);
 
   bool EverChanged = removeUnreachableBlocks(F, DT ? &DTU : nullptr);
-  EverChanged |=
-      tailMergeBlocksWithSimilarFunctionTerminators(F, DT ? &DTU : nullptr);
+  EverChanged |= tailMergeBlocksWithSimilarFunctionTerminators(
+      F, DT ? &DTU : nullptr, Options);
   EverChanged |= iterativelySimplifyCFG(F, TTI, DT ? &DTU : nullptr, Options);
 
   // If neither pass changed anything, we're done.
   if (!EverChanged) return false;
 
   // iterativelySimplifyCFG can (rarely) make some loops dead.  If this happens,
   // removeUnreachableBlocks is needed to nuke them, which means we should
   // iterate between the two optimizations.  We structure the code like this to

llvm/lib/Transforms/Utils/SimplifyCFG.cpp

ArrayRef<Instruction*> operator * () const {
       return Insts;
     }
   };
 
 } // end anonymous namespace
 
 /// Check whether BB's predecessors end with unconditional branches. If it is
 /// true, sink any common code from the predecessors to BB.
-static bool SinkCommonCodeFromPredecessors(BasicBlock *BB,
-                                           DomTreeUpdater *DTU) {
+bool llvm::SinkCommonCodeFromPredecessors(BasicBlock *BB, DomTreeUpdater *DTU) {
   // We support two situations:
   //   (1) all incoming arcs are unconditional
   //   (2) there are non-unconditional incoming arcs
   //
   // (2) is very common in switch defaults and
   // else-if patterns;
   //
   //   if (a) f(1);

llvm/test/CodeGen/Thumb2/setjmp_longjmp.ll

 ; CHECK-NEXT:    str.w sp, [sp, #12]
 ; CHECK-NEXT:    mov r1, pc @ eh_setjmp begin
 ; CHECK-NEXT:    adds r1, r1, #7
 ; CHECK-NEXT:    str r1, [r0, #4]
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    b LSJLJEH0
 ; CHECK-NEXT:    movs r0, #1 @ eh_setjmp end
 ; CHECK-NEXT:  LSJLJEH0:
-; CHECK-NEXT:    cbz r0, LBB0_3
-; CHECK-NEXT:  @ %bb.1: @ %if.then
-; CHECK-NEXT:    movw r0, :lower16:(L_g$non_lazy_ptr-(LPC0_0+4))
-; CHECK-NEXT:    movt r0, :upper16:(L_g$non_lazy_ptr-(LPC0_0+4))
+; CHECK-NEXT:    movw r1, :lower16:(L_g$non_lazy_ptr-(LPC0_0+4))
+; CHECK-NEXT:    movt r1, :upper16:(L_g$non_lazy_ptr-(LPC0_0+4))
 ; CHECK-NEXT:  LPC0_0:
-; CHECK-NEXT:    add r0, pc
-; CHECK-NEXT:    ldr r1, [r0]
+; CHECK-NEXT:    add r1, pc
+; CHECK-NEXT:    cbz r0, LBB0_4
+; CHECK-NEXT:  @ %bb.1: @ %if.then
+; CHECK-NEXT:    ldr r2, [r1]
 ; CHECK-NEXT:    movs r0, #1
-; CHECK-NEXT:    str r1, [sp] @ 4-byte Spill
-; CHECK-NEXT:    str r0, [r1]
-; CHECK-NEXT:    add r0, sp, #4
+; CHECK-NEXT:    str r2, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    movs r1, #0
+; CHECK-NEXT:    str r0, [r2]
+; CHECK-NEXT:    add r0, sp, #4
 ; CHECK-NEXT:    str r7, [sp, #4]
 ; CHECK-NEXT:    str.w sp, [sp, #12]
 ; CHECK-NEXT:    mov r1, pc @ eh_setjmp begin
 ; CHECK-NEXT:    adds r1, r1, #7
 ; CHECK-NEXT:    str r1, [r0, #4]
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    b LSJLJEH1
 ; CHECK-NEXT:    movs r0, #1 @ eh_setjmp end
 ; CHECK-NEXT:  LSJLJEH1:
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    itttt ne
 ; CHECK-NEXT:    movne r0, #3
 ; CHECK-NEXT:    ldrne r1, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    strne r0, [r1]
 ; CHECK-NEXT:    addne sp, #24
 ; CHECK-NEXT:    it ne
 ; CHECK-NEXT:    popne.w {r4, r5, r6, r7, r8, r10, r11, pc}
-; CHECK-NEXT:  LBB0_2: @ %if2.else
-; CHECK-NEXT:    ldr r1, [sp] @ 4-byte Reload
+; CHECK-NEXT:  LBB0_2:
+; CHECK-NEXT:    movw r1, :lower16:(L_g$non_lazy_ptr-(LPC0_1+4))
+; CHECK-NEXT:    add r2, sp, #4
+; CHECK-NEXT:    movt r1, :upper16:(L_g$non_lazy_ptr-(LPC0_1+4))
 ; CHECK-NEXT:    movs r0, #2
+; CHECK-NEXT:  LPC0_1:
+; CHECK-NEXT:    add r1, pc
+; CHECK-NEXT:  LBB0_3: @ %common.unreachable
+; CHECK-NEXT:    ldr r1, [r1]
 ; CHECK-NEXT:    str r0, [r1]
-; CHECK-NEXT:    add r1, sp, #4
 ; CHECK-NEXT:    movs r0, #0
-; CHECK-NEXT:    ldr r0, [r1, #8]
+; CHECK-NEXT:    ldr r0, [r2, #8]
 ; CHECK-NEXT:    mov sp, r0
-; CHECK-NEXT:    ldr r0, [r1, #4]
-; CHECK-NEXT:    ldr r7, [r1]
+; CHECK-NEXT:    ldr r0, [r2, #4]
+; CHECK-NEXT:    ldr r7, [r2]
 ; CHECK-NEXT:    bx r0
-; CHECK-NEXT:  LBB0_3: @ %if.else
-; CHECK-NEXT:    movw r0, :lower16:(L_g$non_lazy_ptr-(LPC0_1+4))
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    movt r0, :upper16:(L_g$non_lazy_ptr-(LPC0_1+4))
-; CHECK-NEXT:  LPC0_1:
-; CHECK-NEXT:    add r0, pc
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    str r1, [r0]
-; CHECK-NEXT:    add r0, sp, #4
-; CHECK-NEXT:    ldr r1, [r0, #8]
-; CHECK-NEXT:    mov sp, r1
-; CHECK-NEXT:    ldr r1, [r0, #4]
-; CHECK-NEXT:    ldr r7, [r0]
-; CHECK-NEXT:    bx r1
+; CHECK-NEXT:  LBB0_4:
+; CHECK-NEXT:    add r2, sp, #4
+; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:    b LBB0_3
 entry:
   %buf = alloca [5 x i8*], align 4
   %bufptr = bitcast [5 x i8*]* %buf to i8*
   %arraydecay = getelementptr inbounds [5 x i8*], [5 x i8*]* %buf, i32 0, i32 0
 
   %fa = tail call i8* @llvm.frameaddress(i32 0)
   store i8* %fa, i8** %arraydecay, align 4
   %ss = tail call i8* @llvm.stacksave()

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

 ; CHECK-NEXT:    [[END_INT_I6_PEEL:%.*]] = ptrtoint i64* [[END_I4_PEEL]] to i64
 ; CHECK-NEXT:    [[SUB_I7_PEEL:%.*]] = sub i64 [[END_INT_I6_PEEL]], [[START_INT_I5_PEEL]]
 ; CHECK-NEXT:    [[LV_I_PEEL:%.*]] = load i64, i64* [[START_I]], align 4
 ; CHECK-NEXT:    [[LV_I10_PEEL:%.*]] = load i64, i64* [[START_I2_PEEL]], align 4
 ; CHECK-NEXT:    [[SUM_NEXT_PEEL:%.*]] = add i64 [[LV_I_PEEL]], [[LV_I10_PEEL]]
 ; CHECK-NEXT:    [[C_PEEL:%.*]] = icmp sgt i64 [[N:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C_PEEL]], label [[LOOP_PREHEADER:%.*]], label [[EXIT:%.*]]
 ; CHECK:       loop.preheader:
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp eq i64 [[SUB_I]], 0
 ; CHECK-NEXT:    [[UMIN:%.*]] = call i64 @llvm.umin.i64(i64 [[SUB_I7_PEEL]], i64 [[SUB_I]])
-; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[N]], -1
-; CHECK-NEXT:    [[UMIN16:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN]], i64 [[TMP0]])
-; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[UMIN16]], 1
-; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP1]], 5
-; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER22:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[TMP0]], i64 0, i64 [[UMIN]]
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[N]], -1
+; CHECK-NEXT:    [[UMIN16:%.*]] = call i64 @llvm.umin.i64(i64 [[TMP1]], i64 [[TMP2]])
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[UMIN16]], 1
+; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP3]], 5
+; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER28:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
-; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP1]], 3
-; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
-; CHECK-NEXT:    [[TMP3:%.*]] = select i1 [[TMP2]], i64 4, i64 [[N_MOD_VF]]
-; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP3]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
+; CHECK-NEXT:    [[TMP5:%.*]] = select i1 [[TMP4]], i64 4, i64 [[N_MOD_VF]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP3]], [[TMP5]]
 ; CHECK-NEXT:    [[IND_END:%.*]] = add i64 [[N_VEC]], 1
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP4]], [[VECTOR_PH]] ], [ [[TMP15:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI18:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP16:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP6]], [[VECTOR_PH]] ], [ [[TMP17:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI18:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP18:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = or i64 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
-; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i64* [[TMP5]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP6]], align 4
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[TMP5]], i64 2
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = bitcast i64* [[TMP7]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD19:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[TMP7]], i64 2
 ; CHECK-NEXT:    [[TMP10:%.*]] = bitcast i64* [[TMP9]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD20:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
-; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[TMP9]], i64 2
+; CHECK-NEXT:    [[WIDE_LOAD25:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = bitcast i64* [[TMP11]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD21:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
-; CHECK-NEXT:    [[TMP13:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
-; CHECK-NEXT:    [[TMP14:%.*]] = add <2 x i64> [[WIDE_LOAD19]], [[VEC_PHI18]]
-; CHECK-NEXT:    [[TMP15]] = add <2 x i64> [[TMP13]], [[WIDE_LOAD20]]
-; CHECK-NEXT:    [[TMP16]] = add <2 x i64> [[TMP14]], [[WIDE_LOAD21]]
+; CHECK-NEXT:    [[WIDE_LOAD26:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
+; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr i64, i64* [[TMP11]], i64 2
+; CHECK-NEXT:    [[TMP14:%.*]] = bitcast i64* [[TMP13]] to <2 x i64>*
+; CHECK-NEXT:    [[WIDE_LOAD27:%.*]] = load <2 x i64>, <2 x i64>* [[TMP14]], align 4
+; CHECK-NEXT:    [[TMP15:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
+; CHECK-NEXT:    [[TMP16:%.*]] = add <2 x i64> [[WIDE_LOAD25]], [[VEC_PHI18]]
+; CHECK-NEXT:    [[TMP17]] = add <2 x i64> [[TMP15]], [[WIDE_LOAD26]]
+; CHECK-NEXT:    [[TMP18]] = add <2 x i64> [[TMP16]], [[WIDE_LOAD27]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP17:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP17]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-NEXT:    [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP19]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP16]], [[TMP15]]
-; CHECK-NEXT:    [[TMP18:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
-; CHECK-NEXT:    br label [[LOOP_PREHEADER22]]
-; CHECK:       loop.preheader22:
+; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP18]], [[TMP17]]
+; CHECK-NEXT:    [[TMP20:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
+; CHECK-NEXT:    br label [[LOOP_PREHEADER28]]
+; CHECK:       loop.preheader28:
 ; CHECK-NEXT:    [[IV_PH:%.*]] = phi i64 [ 1, [[LOOP_PREHEADER]] ], [ [[IND_END]], [[MIDDLE_BLOCK]] ]
-; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP18]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP20]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER22]] ]
-; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12]] ], [ [[SUM_PH]], [[LOOP_PREHEADER22]] ]
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER28]] ]
+; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12]] ], [ [[SUM_PH]], [[LOOP_PREHEADER28]] ]
 ; CHECK-NEXT:    [[INRANGE_I:%.*]] = icmp ult i64 [[SUB_I]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I]], label [[ERROR_I:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT:%.*]]
-; CHECK:       error.i:
-; CHECK-NEXT:    tail call void @error()
-; CHECK-NEXT:    unreachable
-; CHECK:       at_with_int_conversion.exit:
 ; CHECK-NEXT:    [[INRANGE_I8:%.*]] = icmp ult i64 [[SUB_I7_PEEL]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I8]], label [[ERROR_I11:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT12]]
-; CHECK:       error.i11:
+; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[INRANGE_I]], i1 true, i1 [[INRANGE_I8]]
+; CHECK-NEXT:    br i1 [[OR_COND]], label [[COMMON_UNREACHABLE:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT12]]
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    tail call void @error()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       at_with_int_conversion.exit12:
 ; CHECK-NEXT:    [[GEP_IDX_I:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I:%.*]] = load i64, i64* [[GEP_IDX_I]], align 4
 ; CHECK-NEXT:    [[GEP_IDX_I9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I10:%.*]] = load i64, i64* [[GEP_IDX_I9]], align 4
 ; CHECK-NEXT:    [[ADD:%.*]] = add i64 [[LV_I]], [[SUM]]

 ; CHECK-NEXT:    [[SUB_I19_PEEL:%.*]] = sub i64 [[END_INT_I18_PEEL]], [[START_INT_I17_PEEL]]
 ; CHECK-NEXT:    [[LV_I10_PEEL:%.*]] = load i64, i64* [[START_I2_PEEL]], align 4
 ; CHECK-NEXT:    [[LV_I22_PEEL:%.*]] = load i64, i64* [[START_I14_PEEL]], align 4
 ; CHECK-NEXT:    [[ADD_2_PEEL:%.*]] = add i64 [[LV_I_PEEL]], [[LV_I10_PEEL]]
 ; CHECK-NEXT:    [[SUM_NEXT_PEEL:%.*]] = add i64 [[ADD_2_PEEL]], [[LV_I22_PEEL]]
 ; CHECK-NEXT:    [[COND_PEEL:%.*]] = icmp sgt i64 [[N:%.*]], 0
 ; CHECK-NEXT:    br i1 [[COND_PEEL]], label [[LOOP_PREHEADER:%.*]], label [[EXIT:%.*]]
 ; CHECK:       loop.preheader:
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp eq i64 [[SUB_I7_PEEL]], 0
 ; CHECK-NEXT:    [[UMIN:%.*]] = call i64 @llvm.umin.i64(i64 [[SUB_I19_PEEL]], i64 [[SUB_I7_PEEL]])
-; CHECK-NEXT:    [[UMIN28:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN]], i64 [[SUB_I]])
-; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[N]], -1
-; CHECK-NEXT:    [[UMIN29:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN28]], i64 [[TMP0]])
-; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[UMIN29]], 1
-; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP1]], 5
-; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER37:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[TMP0]], i64 0, i64 [[UMIN]]
+; CHECK-NEXT:    [[UMIN28:%.*]] = call i64 @llvm.umin.i64(i64 [[TMP1]], i64 [[SUB_I]])
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[N]], -1
+; CHECK-NEXT:    [[UMIN29:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN28]], i64 [[TMP2]])
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[UMIN29]], 1
+; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP3]], 5
+; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER43:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
-; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP1]], 3
-; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
-; CHECK-NEXT:    [[TMP3:%.*]] = select i1 [[TMP2]], i64 4, i64 [[N_MOD_VF]]
-; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP3]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
+; CHECK-NEXT:    [[TMP5:%.*]] = select i1 [[TMP4]], i64 4, i64 [[N_MOD_VF]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP3]], [[TMP5]]
 ; CHECK-NEXT:    [[IND_END:%.*]] = add i64 [[N_VEC]], 1
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP4]], [[VECTOR_PH]] ], [ [[TMP21:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI31:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP22:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP6]], [[VECTOR_PH]] ], [ [[TMP23:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI31:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP24:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = or i64 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
-; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i64* [[TMP5]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP6]], align 4
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[TMP5]], i64 2
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = bitcast i64* [[TMP7]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD32:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[TMP7]], i64 2
 ; CHECK-NEXT:    [[TMP10:%.*]] = bitcast i64* [[TMP9]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD33:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
-; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[TMP9]], i64 2
+; CHECK-NEXT:    [[WIDE_LOAD38:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = bitcast i64* [[TMP11]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD34:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
-; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr i64, i64* [[START_I14_PEEL]], i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[WIDE_LOAD39:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
+; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr i64, i64* [[TMP11]], i64 2
 ; CHECK-NEXT:    [[TMP14:%.*]] = bitcast i64* [[TMP13]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD35:%.*]] = load <2 x i64>, <2 x i64>* [[TMP14]], align 4
-; CHECK-NEXT:    [[TMP15:%.*]] = getelementptr i64, i64* [[TMP13]], i64 2
+; CHECK-NEXT:    [[WIDE_LOAD40:%.*]] = load <2 x i64>, <2 x i64>* [[TMP14]], align 4
+; CHECK-NEXT:    [[TMP15:%.*]] = getelementptr i64, i64* [[START_I14_PEEL]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP16:%.*]] = bitcast i64* [[TMP15]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD36:%.*]] = load <2 x i64>, <2 x i64>* [[TMP16]], align 4
-; CHECK-NEXT:    [[TMP17:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
-; CHECK-NEXT:    [[TMP18:%.*]] = add <2 x i64> [[WIDE_LOAD32]], [[VEC_PHI31]]
-; CHECK-NEXT:    [[TMP19:%.*]] = add <2 x i64> [[TMP17]], [[WIDE_LOAD33]]
-; CHECK-NEXT:    [[TMP20:%.*]] = add <2 x i64> [[TMP18]], [[WIDE_LOAD34]]
-; CHECK-NEXT:    [[TMP21]] = add <2 x i64> [[TMP19]], [[WIDE_LOAD35]]
-; CHECK-NEXT:    [[TMP22]] = add <2 x i64> [[TMP20]], [[WIDE_LOAD36]]
+; CHECK-NEXT:    [[WIDE_LOAD41:%.*]] = load <2 x i64>, <2 x i64>* [[TMP16]], align 4
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr i64, i64* [[TMP15]], i64 2
+; CHECK-NEXT:    [[TMP18:%.*]] = bitcast i64* [[TMP17]] to <2 x i64>*
+; CHECK-NEXT:    [[WIDE_LOAD42:%.*]] = load <2 x i64>, <2 x i64>* [[TMP18]], align 4
+; CHECK-NEXT:    [[TMP19:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
+; CHECK-NEXT:    [[TMP20:%.*]] = add <2 x i64> [[WIDE_LOAD38]], [[VEC_PHI31]]
+; CHECK-NEXT:    [[TMP21:%.*]] = add <2 x i64> [[TMP19]], [[WIDE_LOAD39]]
+; CHECK-NEXT:    [[TMP22:%.*]] = add <2 x i64> [[TMP20]], [[WIDE_LOAD40]]
+; CHECK-NEXT:    [[TMP23]] = add <2 x i64> [[TMP21]], [[WIDE_LOAD41]]
+; CHECK-NEXT:    [[TMP24]] = add <2 x i64> [[TMP22]], [[WIDE_LOAD42]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP23:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP23]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP25]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP22]], [[TMP21]]
-; CHECK-NEXT:    [[TMP24:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
-; CHECK-NEXT:    br label [[LOOP_PREHEADER37]]
-; CHECK:       loop.preheader37:
+; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP24]], [[TMP23]]
+; CHECK-NEXT:    [[TMP26:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
+; CHECK-NEXT:    br label [[LOOP_PREHEADER43]]
+; CHECK:       loop.preheader43:
 ; CHECK-NEXT:    [[IV_PH:%.*]] = phi i64 [ 1, [[LOOP_PREHEADER]] ], [ [[IND_END]], [[MIDDLE_BLOCK]] ]
-; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP24]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP26]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER37]] ]
-; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24]] ], [ [[SUM_PH]], [[LOOP_PREHEADER37]] ]
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER43]] ]
+; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24]] ], [ [[SUM_PH]], [[LOOP_PREHEADER43]] ]
 ; CHECK-NEXT:    [[INRANGE_I:%.*]] = icmp ult i64 [[SUB_I]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I]], label [[ERROR_I:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT:%.*]]
-; CHECK:       error.i:
+; CHECK-NEXT:    br i1 [[INRANGE_I]], label [[COMMON_UNREACHABLE:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT:%.*]]
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    tail call void @error()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       at_with_int_conversion.exit:
-; CHECK-NEXT:    [[GEP_IDX_I:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[IV]]
-; CHECK-NEXT:    [[LV_I:%.*]] = load i64, i64* [[GEP_IDX_I]], align 4
 ; CHECK-NEXT:    [[INRANGE_I8:%.*]] = icmp ult i64 [[SUB_I7_PEEL]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I8]], label [[ERROR_I11:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT12:%.*]]
-; CHECK:       error.i11:
-; CHECK-NEXT:    tail call void @error()
-; CHECK-NEXT:    unreachable
-; CHECK:       at_with_int_conversion.exit12:
 ; CHECK-NEXT:    [[INRANGE_I20:%.*]] = icmp ult i64 [[SUB_I19_PEEL]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I20]], label [[ERROR_I23:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT24]]
-; CHECK:       error.i23:
-; CHECK-NEXT:    tail call void @error()
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[INRANGE_I8]], i1 true, i1 [[INRANGE_I20]]
+; CHECK-NEXT:    br i1 [[OR_COND]], label [[COMMON_UNREACHABLE]], label [[AT_WITH_INT_CONVERSION_EXIT24]]
 ; CHECK:       at_with_int_conversion.exit24:
+; CHECK-NEXT:    [[GEP_IDX_I:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[IV]]
+; CHECK-NEXT:    [[LV_I:%.*]] = load i64, i64* [[GEP_IDX_I]], align 4
 ; CHECK-NEXT:    [[GEP_IDX_I9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I10:%.*]] = load i64, i64* [[GEP_IDX_I9]], align 4
 ; CHECK-NEXT:    [[GEP_IDX_I21:%.*]] = getelementptr i64, i64* [[START_I14_PEEL]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I22:%.*]] = load i64, i64* [[GEP_IDX_I21]], align 4
 ; CHECK-NEXT:    [[ADD_1:%.*]] = add i64 [[LV_I]], [[SUM]]
 ; CHECK-NEXT:    [[ADD_2:%.*]] = add i64 [[ADD_1]], [[LV_I10]]
 ; CHECK-NEXT:    [[SUM_NEXT]] = add i64 [[ADD_2]], [[LV_I22]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1

llvm/test/Transforms/SimplifyCFG/tail-merge-noreturn.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -simplifycfg -simplifycfg-require-and-preserve-domtree=1 -sink-common-insts  -S < %s | FileCheck %s
 
 ; Test that we tail merge noreturn call blocks and phi constants properly.
 
 declare void @abort()
 declare void @assert_fail_1(i32)
 declare void @assert_fail_1_alt(i32)
 
 define void @merge_simple() {
 ; CHECK-LABEL: @merge_simple(
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
-; CHECK:       a1:
+; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[COMMON_UNREACHABLE:%.*]]
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    call void @assert_fail_1(i32 0)
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A3:%.*]]
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 0)
   unreachable

 cont3:
   ret void
 }
 
 define void @phi_three_constants() {
 ; CHECK-LABEL: @phi_three_constants(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
-; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
+; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[COMMON_UNREACHABLE:%.*]]
+; CHECK:       common.unreachable:
+; CHECK-NEXT:    [[DOTSINK:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ 1, [[CONT1]] ], [ 2, [[CONT2:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[DOTSINK]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 1)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A3:%.*]]
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 2)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 0)
a3:
   unreachable
 cont3:
   ret void
 }
 
 define void @dont_phi_values(i32 %x, i32 %y) {
 ; CHECK-LABEL: @dont_phi_values(
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
-; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[X:%.*]])
+; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[COMMON_UNREACHABLE:%.*]]
+; CHECK:       common.unreachable:
+; CHECK-NEXT:    [[Y_SINK:%.*]] = phi i32 [ [[X:%.*]], [[TMP0:%.*]] ], [ [[Y:%.*]], [[CONT1]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[Y_SINK]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[Y:%.*]])
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    ret void
 ;
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 %x)
   unreachable

 declare i1 @bar()
 
 define void @unmergeable_phis(i32 %v, i1 %c) {
 ; CHECK-LABEL: @unmergeable_phis(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[C:%.*]], label [[S1:%.*]], label [[S2:%.*]]
 ; CHECK:       s1:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C1]], label [[A1:%.*]], label [[A2:%.*]]
+; CHECK-NEXT:    br i1 [[C1]], label [[COMMON_UNREACHABLE:%.*]], label [[A2:%.*]]
 ; CHECK:       s2:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @bar()
-; CHECK-NEXT:    br i1 [[C2]], label [[A1]], label [[A2]]
-; CHECK:       a1:
-; CHECK-NEXT:    [[L1:%.*]] = phi i32 [ 0, [[S1]] ], [ 1, [[S2]] ]
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[L1]])
+; CHECK-NEXT:    br i1 [[C2]], label [[COMMON_UNREACHABLE]], label [[A2]]
+; CHECK:       common.unreachable:
+; CHECK-NEXT:    [[L2_SINK:%.*]] = phi i32 [ [[L2:%.*]], [[A2]] ], [ 0, [[S1]] ], [ 1, [[S2]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[L2_SINK]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       a2:
-; CHECK-NEXT:    [[L2:%.*]] = phi i32 [ 2, [[S1]] ], [ 3, [[S2]] ]
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[L2]])
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    [[L2]] = phi i32 [ 2, [[S1]] ], [ 3, [[S2]] ]
+; CHECK-NEXT:    br label [[COMMON_UNREACHABLE]]
 ;
 entry:
   br i1 %c, label %s1, label %s2
 s1:
   %c1 = call i1 @foo()
   br i1 %c1, label %a1, label %a2
 s2:
   %c2 = call i1 @bar()
   br i1 %c2, label %a1, label %a2
 a1:
   %l1 = phi i32 [ 0, %s1 ], [ 1, %s2 ]
   call void @assert_fail_1(i32 %l1)
   unreachable
 a2:
   %l2 = phi i32 [ 2, %s1 ], [ 3, %s2 ]
   call void @assert_fail_1(i32 %l2)
   unreachable
 }
 
 define void @tail_merge_switch(i32 %v) {
 ; CHECK-LABEL: @tail_merge_switch(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    switch i32 [[V:%.*]], label [[RET:%.*]] [
-; CHECK-NEXT:    i32 0, label [[A1:%.*]]
+; CHECK-NEXT:    i32 0, label [[COMMON_UNREACHABLE:%.*]]
 ; CHECK-NEXT:    i32 13, label [[A2:%.*]]
 ; CHECK-NEXT:    i32 42, label [[A3:%.*]]
 ; CHECK-NEXT:    ]
-; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
+; CHECK:       common.unreachable:
+; CHECK-NEXT:    [[DOTSINK:%.*]] = phi i32 [ 2, [[A3]] ], [ 1, [[A2]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[DOTSINK]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 1)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br label [[COMMON_UNREACHABLE]]
 ; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 2)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br label [[COMMON_UNREACHABLE]]
 ; CHECK:       ret:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   switch i32 %v, label %ret [
   i32 0, label %a1
   i32 13, label %a2
   i32 42, label %a3
a3:
   unreachable
 ret:
   ret void
 }
 
 define void @need_to_add_bb2_preds(i1 %c1) {
 ; CHECK-LABEL: @need_to_add_bb2_preds(
 ; CHECK-NEXT:  bb1:
-; CHECK-NEXT:    br i1 [[C1:%.*]], label [[BB2:%.*]], label [[A1:%.*]]
+; CHECK-NEXT:    br i1 [[C1:%.*]], label [[BB2:%.*]], label [[COMMON_UNREACHABLE:%.*]]
 ; CHECK:       bb2:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @bar()
-; CHECK-NEXT:    br i1 [[C2]], label [[A2:%.*]], label [[A3:%.*]]
-; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 1)
-; CHECK-NEXT:    unreachable
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 2)
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[C2]], i32 1, i32 2
+; CHECK-NEXT:    br label [[COMMON_UNREACHABLE]]
+; CHECK:       common.unreachable:
+; CHECK-NEXT:    [[DOTSINK:%.*]] = phi i32 [ [[DOT]], [[BB2]] ], [ 0, [[BB1:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[DOTSINK]])
 ; CHECK-NEXT:    unreachable
 ;
 bb1:
   br i1 %c1, label %bb2, label %a1
 bb2:
   %c2 = call i1 @bar()
   br i1 %c2, label %a2, label %a3
 
 a1:
   call void @assert_fail_1(i32 0)
   unreachable
 a2:
   call void @assert_fail_1(i32 1)
   unreachable
 a3:
   call void @assert_fail_1(i32 2)
   unreachable
 }
 
 define void @phi_in_bb2() {
 ; CHECK-LABEL: @phi_in_bb2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
-; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
+; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[COMMON_UNREACHABLE:%.*]]
+; CHECK:       common.unreachable:
+; CHECK-NEXT:    [[P2_SINK:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ 1, [[CONT1]] ], [ 2, [[CONT2:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[P2_SINK]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    [[P2:%.*]] = phi i32 [ 1, [[CONT1]] ], [ 2, [[CONT2]] ]
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[P2]])
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A2]]
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 0)

 ; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[X]] to i8*
 ; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 4, i8* nonnull [[TMP0]])
 ; CHECK-NEXT:    store i32 0, i32* [[X]], align 4
 ; CHECK-NEXT:    [[TOBOOL:%.*]] = icmp eq i32 [[C2:%.*]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL]], label [[IF_END:%.*]], label [[IF_THEN1:%.*]]
 ; CHECK:       if.then1:
 ; CHECK-NEXT:    call void @escape_i32_ptr(i32* nonnull [[X]])
 ; CHECK-NEXT:    br label [[IF_END]]
-; CHECK:       if.end:
-; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 4, i8* nonnull [[TMP0]])
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    call void @abort()
 ; CHECK-NEXT:    unreachable
+; CHECK:       if.end:
+; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 4, i8* nonnull [[TMP0]])
+; CHECK-NEXT:    br label [[COMMON_UNREACHABLE:%.*]]
 ; CHECK:       if.then3:
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[Y]] to i8*
 ; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 4, i8* nonnull [[TMP1]])
 ; CHECK-NEXT:    store i32 0, i32* [[Y]], align 4
 ; CHECK-NEXT:    [[TOBOOL5:%.*]] = icmp eq i32 [[C2]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL5]], label [[IF_END7:%.*]], label [[IF_THEN6:%.*]]
 ; CHECK:       if.then6:
 ; CHECK-NEXT:    call void @escape_i32_ptr(i32* nonnull [[Y]])
 ; CHECK-NEXT:    br label [[IF_END7]]
 ; CHECK:       if.end7:
 ; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 4, i8* nonnull [[TMP1]])
-; CHECK-NEXT:    call void @abort()
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br label [[COMMON_UNREACHABLE]]
 ; CHECK:       if.end9:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %x = alloca i32, align 4
   %y = alloca i32, align 4
   switch i32 %c1, label %if.end9 [
   i32 13, label %if.then

 ; Dead phis in the block need to be handled.
 
 declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 
 define void @dead_phi() {
 ; CHECK-LABEL: @dead_phi(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
-; CHECK:       a1:
-; CHECK-NEXT:    [[DEAD:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ 1, [[CONT1]] ]
+; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[COMMON_UNREACHABLE:%.*]]
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    call void @assert_fail_1(i32 0)
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A1]]
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A3:%.*]]
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[COMMON_UNREACHABLE]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   %dead = phi i32 [ 0, %entry ], [ 1, %cont1 ]
cont3:
   ret void
 }
 
 define void @strip_dbg_value(i32 %c) {
 ; CHECK-LABEL: @strip_dbg_value(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 [[C:%.*]], metadata [[META5:![0-9]+]], metadata !DIExpression()), !dbg [[DBG7:![0-9]+]]
 ; CHECK-NEXT:    switch i32 [[C]], label [[SW_EPILOG:%.*]] [
-; CHECK-NEXT:    i32 13, label [[SW_BB:%.*]]
-; CHECK-NEXT:    i32 42, label [[SW_BB1:%.*]]
+; CHECK-NEXT:    i32 13, label [[COMMON_UNREACHABLE:%.*]]
+; CHECK-NEXT:    i32 42, label [[COMMON_UNREACHABLE]]
 ; CHECK-NEXT:    ]
-; CHECK:       sw.bb:
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 55, metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
-; CHECK-NEXT:    tail call void @abort()
-; CHECK-NEXT:    unreachable
-; CHECK:       sw.bb1:
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 67, metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    tail call void @abort()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       sw.epilog:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   call void @llvm.dbg.value(metadata i32 %c, i64 0, metadata !12, metadata !13), !dbg !14
   switch i32 %c, label %sw.epilog [
sw.epilog:                                        ; preds = %entry
   ret void
 }
 
 define void @dead_phi_and_dbg(i32 %c) {
 ; CHECK-LABEL: @dead_phi_and_dbg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 [[C:%.*]], metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
 ; CHECK-NEXT:    switch i32 [[C]], label [[SW_EPILOG:%.*]] [
-; CHECK-NEXT:    i32 13, label [[SW_BB:%.*]]
-; CHECK-NEXT:    i32 42, label [[SW_BB1:%.*]]
-; CHECK-NEXT:    i32 53, label [[SW_BB2:%.*]]
+; CHECK-NEXT:    i32 13, label [[COMMON_UNREACHABLE:%.*]]
+; CHECK-NEXT:    i32 42, label [[COMMON_UNREACHABLE]]
+; CHECK-NEXT:    i32 53, label [[COMMON_UNREACHABLE]]
 ; CHECK-NEXT:    ]
-; CHECK:       sw.bb:
-; CHECK-NEXT:    [[C_1:%.*]] = phi i32 [ 55, [[ENTRY:%.*]] ], [ 67, [[SW_BB1]] ]
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 [[C_1]], metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
-; CHECK-NEXT:    tail call void @abort()
-; CHECK-NEXT:    unreachable
-; CHECK:       sw.bb1:
-; CHECK-NEXT:    br label [[SW_BB]]
-; CHECK:       sw.bb2:
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 84, metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
+; CHECK:       common.unreachable:
 ; CHECK-NEXT:    tail call void @abort()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       sw.epilog:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   call void @llvm.dbg.value(metadata i32 %c, i64 0, metadata !12, metadata !13), !dbg !14
   switch i32 %c, label %sw.epilog [