[SimplifyCFG] Tail-merging all blocks with `unreachable` terminator, final take

[lebedev.ri]

Summary

This implements the approach disscussed in D104870:
instead of simply alaways tail-merging all unreachable blocks,
first try to group the calls that precede unreachable,
and only merge the ones where grouping succeeded.

https://llvm-compile-time-tracker.com/compare.php?from=3564551400224cd24dd8650dc2ace19174833af7&to=a4fe8518811df84d14a21071f3516ec3841cd369&stat=instructions

[lebedev.ri]

Numbers are in:
D104870 was: (as of https://reviews.llvm.org/D104870#inline-998899)
https://llvm-compile-time-tracker.com/compare.php?from=1f169a774cb865659cefe085e70a56a884e3711e&to=fc54bb9a8ef85bd76dd9e934b2546f4beadc5b5e&stat=instructions
This now is: https://llvm-compile-time-tracker.com/compare.php?from=2353e1c87b09c20e75f0f3ceb05fa4a4261fe3dd&to=bed7b8df4565f4503889a19235e853b985ca3481&stat=instructions

So slightly better compile-time-wise, basically the same size impact.

[rnk]

I think I prefer this version over the previous. This may be slightly ugly, but IMO it is functionally better (fewer analysis reruns and pass updates). Maybe others have ideas for how to make it nicer.

What do other folks (@nikic @aeubanks @asbirlea) think about this approach?

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

It looks to me like tail merging unreachable blocks is preventing vectorization in this test case and the next, which seems like a blocking issue. The test was added here, if that helps understand why it no longer works:
https://reviews.llvm.org/rG39cc0b8c68b8d316954ecfac0d1f8498ea42866c
@fhahn

[lebedev.ri]

In D116692#3223679, @rnk wrote:

I think I prefer this version over the previous. This may be slightly ugly, but IMO it is functionally better (fewer analysis reruns and pass updates).

I think one big reason why this is indeed better is because now it will be trivial to introduce
(and pass here) bool SkipProfitabilityChecks option to SinkCommonCodeFromPredecessors().

Maybe others have ideas for how to make it nicer.

Looking at it, it's actually not *that* ugly as it seemed when i was writing the code.
I guess i could wrap this tail-merging/undo into a class, not sure if that would help.

What do other folks (@nikic @aeubanks @asbirlea) think about this approach?

[lebedev.ri]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

Filed https://github.com/llvm/llvm-project/issues/53020

[xbolva00]

llvm/lib/Transforms/Scalar/SimplifyCFGPass.cpp

373

Fix comment

[lebedev.ri]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

CC @aqjune @nikic

I'm not actually sure that we can solve this within LV itself,
but i would love to be proven wrong here.

I'm pretty sure LV does expand the backedge taken count,
so i suppose only not being allowed to expand BTC wouldn't help here.

As i see it, the options are:

• ignore this failure
• adjust the test to mask the failure (i would hope adding noundef's should help?), potentially coupled with:
• are there some missing reasoning bits in impliesPoison() and friends that could prevent this regression?
• Introduce UB-safe mode for SCEVExpander, lift backedge taken count poison-safety restriction
• Prevent simplifycfg from merging conditions like that (as in, iff plain and/or isn't going to be used)

[nikic]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

I think the proper way to address this (and other poison-safety issues in SCEV) is to add umin variants in both IR and SCEV that don't propagate op2 poison if op1 is zero.

[lebedev.ri]

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

39

Posted D116766

[lebedev.ri]

Ok, the regression has been dealt with :)

[nikic]

It looks like the new version still has the large code size regressions (9% on mafft, 3% on 7zip). I understand that some code size increase is expected (and intended), but I don't think a particularly good case for the tradeoff has been made yet (in terms of where / how much performance this is buying for more code size). Though maybe I missed this in previous discussion threads.

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

[lebedev.ri]

In D116692#3232485, @nikic wrote:

It looks like the new version still has the large code size regressions (9% on mafft, 3% on 7zip). I understand that some code size increase is expected (and intended), but I don't think a particularly good case for the tradeoff has been made yet (in terms of where / how much performance this is buying for more code size). Though maybe I missed this in previous discussion threads.

How do we know that whatever compile time benchmark we see regresses is a reliable indicator in this regard?
I think this is yet another irresolvable clash between the optimization and compilation time/size.
-O3 does not mean "please quickly give me minimal code", there's -Os/-Oz for that.
IOW if you indent to block a patch, could you please actually do so, not just waguley imply so?

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

Were you able to to so?

[aeubanks]

In D116692#3249159, @lebedev.ri wrote:

In D116692#3232485, @nikic wrote:

It looks like the new version still has the large code size regressions (9% on mafft, 3% on 7zip). I understand that some code size increase is expected (and intended), but I don't think a particularly good case for the tradeoff has been made yet (in terms of where / how much performance this is buying for more code size). Though maybe I missed this in previous discussion threads.

How do we know that whatever compile time benchmark we see regresses is a reliable indicator in this regard?
I think this is yet another irresolvable clash between the optimization and compilation time/size.
-O3 does not mean "please quickly give me minimal code", there's -Os/-Oz for that.
IOW if you indent to block a patch, could you please actually do so, not just waguley imply so?

As somebody without much context for this patch reading the patch summary, there's no "why" for this patch. The llvm-compile-time-tracker numbers are all negative in various aspects. I think nikic's question is do you have metrics showing that this actually helps performance on any code?

[nikic]

In D116692#3249159, @lebedev.ri wrote:

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

Were you able to to so?

Sorry for the delay. I tested this together with your recent commit removing sinking limitations for unreachable blocks. The result was close to no change in either compile-time or run-time (where "run-time" here is non-LLVM compile-time, so a fairly narrow workload). All sub-1% and mostly below the significance threshold. Unfortunately I wasn't able to get code size information because the necessary infrastructure was broken at the time. I did look at some rustc artifacts as a sanity check and the only larger increase I spotted was rustdoc by +0.4%.

I played with the patch a bit, and found that this approach has one major limitation as far as rust code is concerned: It only works if you have a single assert/panic/whatever function. All unreachable terminators are merged together and we can then only sink if the predecessors have the same call. Rust has a bunch of different panic functions depending on the situation. So if you have a bunch of array accesses that all call panic_bounds_check and then add a single assert, then the tail merging stops working. I expect that significantly limits the cases where the optimization applies.

[lebedev.ri]

In D116692#3265576, @nikic wrote:

In D116692#3249159, @lebedev.ri wrote:

I should probably test how this impacts rust code (which has a lot of unreachable terminators in release builds due to bounds checks), though that requires applying this patch on top of LLVM 13.

Were you able to to so?

Sorry for the delay. I tested this together with your recent commit removing sinking limitations for unreachable blocks.
The result was close to no change in either compile-time or run-time (where "run-time" here is non-LLVM compile-time,
so a fairly narrow workload). All sub-1% and mostly below the significance threshold. Unfortunately I wasn't able to
get code size information because the necessary infrastructure was broken at the time.
I did look at some rustc artifacts as a sanity check and the only larger increase I spotted was rustdoc by +0.4%.

I played with the patch a bit, and found that this approach has one major limitation as far as rust code is concerned:
It only works if you have a single assert/panic/whatever function.
All unreachable terminators are merged together and we can then only sink if the predecessors have the same call.
Rust has a bunch of different panic functions depending on the situation.
So if you have a bunch of array accesses that all call panic_bounds_check and then add a single assert,
then the tail merging stops working. I expect that significantly limits the cases where the optimization applies.

Since asking that back then, i've come up with D117805, which indeed handles said more generic case,
and after that patch is accepted i'll rewrite this patch to use that new infra.

[lebedev.ri]

Now that the 'merge compatible invokes' is effectively done, let's revisit this.

I've reimplemented this using the approach innovated/invented there,
so now not only we don't just tail-merge everything,
not only do we only do that when sinking succeeds,
we now also handle multiple sets of mergeable calls.

[nikic]

I've tested the newest version against rust, and unfortunately it came back as a universal regression, both in terms of compile-time and run-time. Compile-time regressions up to 4% and run-time regressions in the ~1% range (not by much, but very consistently regressing).

llvm/lib/Transforms/Scalar/SimplifyCFGPass.cpp

 
 static cl::opt<bool> UserSinkCommonInsts(
     "sink-common-insts", cl::Hidden, cl::init(false),
     cl::desc("Sink common instructions (default = false)"));
 
 
 STATISTIC(NumSimpl, "Number of blocks simplified");
 
+namespace {
+
+struct CompatibleSets {
+  using SetTy = SmallVector<CallInst *, 2>;
+
+  SmallVector<SetTy, 1> Sets;
+
+  static bool shouldBelongToSameSet(ArrayRef<CallInst *> Calls);
+
+  SetTy &getCompatibleSet(CallInst *II);
+
+  void insert(CallInst *II);
+};
+
+CompatibleSets::SetTy &CompatibleSets::getCompatibleSet(CallInst *II) {
+  // Perform a linear scan over all the existing sets, see if the new `call`
+  // is compatible with any particular set. Since we know that all the `calls`
+  // within a set are compatible, only check the first `call` in each set.
+  // WARNING: at worst, this has quadratic complexity.
+  for (CompatibleSets::SetTy &Set : Sets) {
+    if (CompatibleSets::shouldBelongToSameSet({Set.front(), II}))
+      return Set;
+  }
+
+  // Otherwise, we either had no sets yet, or this call forms a new set.
+  return Sets.emplace_back();
+}
+
+void CompatibleSets::insert(CallInst *II) {
+  getCompatibleSet(II).emplace_back(II);
+}
+
+bool CompatibleSets::shouldBelongToSameSet(ArrayRef<CallInst *> Calls) {
+  assert(Calls.size() == 2 && "Always called with exactly two candidates.");
+
+  // Can we theoretically merge these `call`s?
+  auto IsIllegalToMerge = [](CallInst *II) {
+    return II->cannotMerge() || II->isInlineAsm();
+  };
+  if (any_of(Calls, IsIllegalToMerge))
+    return false;
+
+  // Either both `call`s must be   direct,
+  // or     both `call`s must be indirect.
+  auto IsIndirectCall = [](CallInst *II) { return II->isIndirectCall(); };
+  bool HaveIndirectCalls = any_of(Calls, IsIndirectCall);
+  bool AllCallsAreIndirect = all_of(Calls, IsIndirectCall);
+  if (HaveIndirectCalls) {
+    if (!AllCallsAreIndirect)
+      return false;
+  } else {
+    // All callees must be identical.
+    Value *Callee = nullptr;
+    for (CallInst *II : Calls) {
+      Value *CurrCallee = II->getCalledOperand();
+      assert(CurrCallee && "There is always a called operand.");
+      if (!Callee)
+        Callee = CurrCallee;
+      else if (Callee != CurrCallee)
+        return false;
+    }
+  }
+
+  // Ignoring arguments, these `call`s must be identical,
+  // including operand bundles.
+  const CallInst *II0 = Calls.front();
+  for (auto *II : Calls.drop_front())
+    if (!II->isSameOperationAs(II0))
+      return false;
+
+  // Can we theoretically form the data operands for the merged `call`?
+  auto IsIllegalToMergeArguments = [](auto Ops) {
+    Type *Ty = std::get<0>(Ops)->getType();
+    assert(Ty == std::get<1>(Ops)->getType() && "Incompatible types?");
+    return Ty->isTokenTy() && std::get<0>(Ops) != std::get<1>(Ops);
+  };
+  assert(Calls.size() == 2 && "Always called with exactly two candidates.");
+  if (any_of(zip(Calls[0]->data_ops(), Calls[1]->data_ops()),
+             IsIllegalToMergeArguments))
+    return false;
+
+  return true;
+}
+
+} // namespace
+
+// Merge all calls in the provided set, all of which are compatible
+// as per the `CompatibleSets::shouldBelongToSameSet()`.
+static void MergeCompatibleUnreachableTerminatedCallsImpl(
+    ArrayRef<CallInst *> Calls,
+    std::vector<DominatorTree::UpdateType> *Updates) {
+  assert(Calls.size() >= 2 && "Must have at least two calls to merge.");
+
+  if (Updates)
+    Updates->reserve(Updates->size() + Calls.size());
+
+  // Clone one of the calls into a new basic block.
+  // Since they are all compatible, it doesn't matter which call is cloned.
+  CallInst *MergedCall = [&Calls]() {
+    CallInst *II0 = Calls.front();
+    BasicBlock *II0BB = II0->getParent();
+    BasicBlock *InsertBeforeBlock =
+        II0->getParent()->getIterator()->getNextNode();
+    Function *Func = II0BB->getParent();
+    LLVMContext &Ctx = II0->getContext();
+
+    BasicBlock *MergedCallBB =
+        BasicBlock::Create(Ctx, "", Func, InsertBeforeBlock);
+
+    auto *MergedCall = cast<CallInst>(II0->clone());
+    // NOTE: all calls have the same attributes, so no handling needed.
+    MergedCallBB->getInstList().push_back(MergedCall);
+    new UnreachableInst(Ctx, MergedCallBB);
+
+    return MergedCall;
+  }();
+
+  if (Updates) {
+    // Blocks that contained these calls will now branch to
+    // the new block that contains the merged call.
+    for (CallInst *CI : Calls)
+      Updates->push_back(
+          {DominatorTree::Insert, CI->getParent(), MergedCall->getParent()});
+  }
+
+  bool IsIndirectCall = Calls[0]->isIndirectCall();
+
+  // Form the merged operands for the merged call.
+  for (Use &U : MergedCall->operands()) {
+    // Only PHI together the indirect callees and data operands.
+    if (MergedCall->isCallee(&U)) {
+      if (!IsIndirectCall)
+        continue;
+    } else if (!MergedCall->isDataOperand(&U))
+      continue;
+
+    // Don't create trivial PHI's with all-identical incoming values.
+    bool NeedPHI = any_of(Calls, [&U](CallInst *CI) {
+      return CI->getOperand(U.getOperandNo()) != U.get();
+    });
+    if (!NeedPHI)
+      continue;
+
+    // Form a PHI out of all the data ops under this index.
+    PHINode *PN = PHINode::Create(
+        U->getType(), /*NumReservedValues=*/Calls.size(), "", MergedCall);
+    for (CallInst *CI : Calls)
+      PN->addIncoming(CI->getOperand(U.getOperandNo()), CI->getParent());
+
+    U.set(PN);
+  }
+
+  // And finally, replace the original `call`s with an unconditional branch
+  // to the block with the merged `call`. Also, give that merged `call`
+  // the merged debugloc of all the original `call`s.
+  const DILocation *MergedDebugLoc = nullptr;
+  for (CallInst *CI : Calls) {
+    // Compute the debug location common to all the original `call`s.
+    if (!MergedDebugLoc)
+      MergedDebugLoc = CI->getDebugLoc();
+    else
+      MergedDebugLoc =
+          DILocation::getMergedLocation(MergedDebugLoc, CI->getDebugLoc());
+
+    // And replace the old `call`+`unreachable` with an unconditional branch
+    // to the block with the merged `call`.
+    BranchInst::Create(MergedCall->getParent(), CI->getParent());
+    cast<UnreachableInst>(CI->getNextNode())->eraseFromParent();
+    CI->eraseFromParent();
+  }
+  MergedCall->setDebugLoc(MergedDebugLoc);
+}
+
+static bool MergeCompatibleUnreachableTerminatedCalls(
+    ArrayRef<BasicBlock *> BBs,
+    std::vector<DominatorTree::UpdateType> *Updates) {
+  bool Changed = false;
+
+  CompatibleSets Grouper;
+
+  for (BasicBlock *BB : BBs) {
+    auto *Term = BB->getTerminator();
+    assert(isa<UnreachableInst>(Term) &&
+           "Only for blocks with `unreachable` terminator.");
+    // Only deal with blocks where `unreachable` is preceeded by a `call`.
+    if (auto *CI = dyn_cast_or_null<CallInst>(Term->getPrevNode()))
+      Grouper.insert(CI);
+  }
+
+  for (ArrayRef<CallInst *> Calls : Grouper.Sets) {
+    if (Calls.size() < 2)
+      continue;
+    Changed = true;
+    MergeCompatibleUnreachableTerminatedCallsImpl(Calls, Updates);
+  }
+
+  return Changed;
+}
+
 static bool
 performBlockTailMerging(Function &F, ArrayRef<BasicBlock *> BBs,
                         std::vector<DominatorTree::UpdateType> *Updates) {
   SmallVector<PHINode *, 1> NewOps;
 
   // We don't want to change IR just because we can.
   // Only do that if there are at least two blocks we'll tail-merge.
   if (BBs.size() < 2)
     return false;
 
+  // Defer handling of `unreachable` blocks to the specialized utility.
+  if (isa<UnreachableInst>(BBs[0]->getTerminator()))
+    return MergeCompatibleUnreachableTerminatedCalls(BBs, Updates);
+
   if (Updates)
     Updates->reserve(Updates->size() + BBs.size());
 
   BasicBlock *CanonicalBB;
   Instruction *CanonicalTerm;
   {
     auto *Term = BBs[0]->getTerminator();
 
if (DTU && DTU->isBBPendingDeletion(&BB))
       continue;
 
     // We are only interested in function-terminating blocks.
     if (!succ_empty(&BB))
       continue;
 
     auto *Term = BB.getTerminator();
 
     // Fow now only support `ret`/`resume` function terminators.
     // FIXME: lift this restriction.
     switch (Term->getOpcode()) {
     case Instruction::Ret:
     case Instruction::Resume:
+    case Instruction::Unreachable:
       break;
     default:
       continue;
     }
 
     // We can't tail-merge block that contains a musttail call.
     if (BB.getTerminatingMustTailCall())
       continue;

llvm/test/CodeGen/Thumb2/setjmp_longjmp.ll

 ; CHECK-NEXT:    str.w sp, [sp, #12]
 ; CHECK-NEXT:    mov r1, pc @ eh_setjmp begin
 ; CHECK-NEXT:    adds r1, r1, #7
 ; CHECK-NEXT:    str r1, [r0, #4]
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    b LSJLJEH0
 ; CHECK-NEXT:    movs r0, #1 @ eh_setjmp end
 ; CHECK-NEXT:  LSJLJEH0:
-; CHECK-NEXT:    cbz r0, LBB0_3
-; CHECK-NEXT:  @ %bb.1: @ %if.then
-; CHECK-NEXT:    movw r0, :lower16:(L_g$non_lazy_ptr-(LPC0_0+4))
-; CHECK-NEXT:    movt r0, :upper16:(L_g$non_lazy_ptr-(LPC0_0+4))
+; CHECK-NEXT:    movw r1, :lower16:(L_g$non_lazy_ptr-(LPC0_0+4))
+; CHECK-NEXT:    movt r1, :upper16:(L_g$non_lazy_ptr-(LPC0_0+4))
 ; CHECK-NEXT:  LPC0_0:
-; CHECK-NEXT:    add r0, pc
-; CHECK-NEXT:    ldr r1, [r0]
+; CHECK-NEXT:    add r1, pc
+; CHECK-NEXT:    cbz r0, LBB0_4
+; CHECK-NEXT:  @ %bb.1: @ %if.then
+; CHECK-NEXT:    ldr r2, [r1]
 ; CHECK-NEXT:    movs r0, #1
-; CHECK-NEXT:    str r1, [sp] @ 4-byte Spill
-; CHECK-NEXT:    str r0, [r1]
-; CHECK-NEXT:    add r0, sp, #4
+; CHECK-NEXT:    str r2, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    movs r1, #0
+; CHECK-NEXT:    str r0, [r2]
+; CHECK-NEXT:    add r0, sp, #4
 ; CHECK-NEXT:    str r7, [sp, #4]
 ; CHECK-NEXT:    str.w sp, [sp, #12]
 ; CHECK-NEXT:    mov r1, pc @ eh_setjmp begin
 ; CHECK-NEXT:    adds r1, r1, #7
 ; CHECK-NEXT:    str r1, [r0, #4]
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    b LSJLJEH1
 ; CHECK-NEXT:    movs r0, #1 @ eh_setjmp end
 ; CHECK-NEXT:  LSJLJEH1:
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    itttt ne
 ; CHECK-NEXT:    movne r0, #3
 ; CHECK-NEXT:    ldrne r1, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    strne r0, [r1]
 ; CHECK-NEXT:    addne sp, #24
 ; CHECK-NEXT:    it ne
 ; CHECK-NEXT:    popne.w {r4, r5, r6, r7, r8, r10, r11, pc}
-; CHECK-NEXT:  LBB0_2: @ %if2.else
-; CHECK-NEXT:    ldr r1, [sp] @ 4-byte Reload
+; CHECK-NEXT:  LBB0_2:
+; CHECK-NEXT:    movw r1, :lower16:(L_g$non_lazy_ptr-(LPC0_1+4))
+; CHECK-NEXT:    add r2, sp, #4
+; CHECK-NEXT:    movt r1, :upper16:(L_g$non_lazy_ptr-(LPC0_1+4))
 ; CHECK-NEXT:    movs r0, #2
+; CHECK-NEXT:  LPC0_1:
+; CHECK-NEXT:    add r1, pc
+; CHECK-NEXT:  LBB0_3: @ %if2.else
+; CHECK-NEXT:    ldr r1, [r1]
 ; CHECK-NEXT:    str r0, [r1]
-; CHECK-NEXT:    add r1, sp, #4
 ; CHECK-NEXT:    movs r0, #0
-; CHECK-NEXT:    ldr r0, [r1, #8]
+; CHECK-NEXT:    ldr r0, [r2, #8]
 ; CHECK-NEXT:    mov sp, r0
-; CHECK-NEXT:    ldr r0, [r1, #4]
-; CHECK-NEXT:    ldr r7, [r1]
+; CHECK-NEXT:    ldr r0, [r2, #4]
+; CHECK-NEXT:    ldr r7, [r2]
 ; CHECK-NEXT:    bx r0
-; CHECK-NEXT:  LBB0_3: @ %if.else
-; CHECK-NEXT:    movw r0, :lower16:(L_g$non_lazy_ptr-(LPC0_1+4))
-; CHECK-NEXT:    movs r1, #0
-; CHECK-NEXT:    movt r0, :upper16:(L_g$non_lazy_ptr-(LPC0_1+4))
-; CHECK-NEXT:  LPC0_1:
-; CHECK-NEXT:    add r0, pc
-; CHECK-NEXT:    ldr r0, [r0]
-; CHECK-NEXT:    str r1, [r0]
-; CHECK-NEXT:    add r0, sp, #4
-; CHECK-NEXT:    ldr r1, [r0, #8]
-; CHECK-NEXT:    mov sp, r1
-; CHECK-NEXT:    ldr r1, [r0, #4]
-; CHECK-NEXT:    ldr r7, [r0]
-; CHECK-NEXT:    bx r1
+; CHECK-NEXT:  LBB0_4:
+; CHECK-NEXT:    add r2, sp, #4
+; CHECK-NEXT:    movs r0, #0
+; CHECK-NEXT:    b LBB0_3
 entry:
   %buf = alloca [5 x i8*], align 4
   %bufptr = bitcast [5 x i8*]* %buf to i8*
   %arraydecay = getelementptr inbounds [5 x i8*], [5 x i8*]* %buf, i32 0, i32 0
 
   %fa = tail call i8* @llvm.frameaddress(i32 0)
   store i8* %fa, i8** %arraydecay, align 4
   %ss = tail call i8* @llvm.stacksave()

llvm/test/Transforms/PhaseOrdering/AArch64/peel-multiple-unreachable-exits-for-vectorization.ll

 ; CHECK-NEXT:    [[END_INT_I6_PEEL:%.*]] = ptrtoint i64* [[END_I4_PEEL]] to i64
 ; CHECK-NEXT:    [[SUB_I7_PEEL:%.*]] = sub i64 [[END_INT_I6_PEEL]], [[START_INT_I5_PEEL]]
 ; CHECK-NEXT:    [[LV_I_PEEL:%.*]] = load i64, i64* [[START_I]], align 4
 ; CHECK-NEXT:    [[LV_I10_PEEL:%.*]] = load i64, i64* [[START_I2_PEEL]], align 4
 ; CHECK-NEXT:    [[SUM_NEXT_PEEL:%.*]] = add i64 [[LV_I_PEEL]], [[LV_I10_PEEL]]
 ; CHECK-NEXT:    [[C_PEEL:%.*]] = icmp sgt i64 [[N:%.*]], 0
 ; CHECK-NEXT:    br i1 [[C_PEEL]], label [[LOOP_PREHEADER:%.*]], label [[EXIT:%.*]]
 ; CHECK:       loop.preheader:
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp eq i64 [[SUB_I]], 0
 ; CHECK-NEXT:    [[UMIN:%.*]] = call i64 @llvm.umin.i64(i64 [[SUB_I7_PEEL]], i64 [[SUB_I]])
-; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[N]], -1
-; CHECK-NEXT:    [[UMIN16:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN]], i64 [[TMP0]])
-; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[UMIN16]], 1
-; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP1]], 5
-; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER22:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[TMP0]], i64 0, i64 [[UMIN]]
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[N]], -1
+; CHECK-NEXT:    [[UMIN16:%.*]] = call i64 @llvm.umin.i64(i64 [[TMP1]], i64 [[TMP2]])
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[UMIN16]], 1
+; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP3]], 5
+; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER28:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
-; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP1]], 3
-; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
-; CHECK-NEXT:    [[TMP3:%.*]] = select i1 [[TMP2]], i64 4, i64 [[N_MOD_VF]]
-; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP3]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
+; CHECK-NEXT:    [[TMP5:%.*]] = select i1 [[TMP4]], i64 4, i64 [[N_MOD_VF]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP3]], [[TMP5]]
 ; CHECK-NEXT:    [[IND_END:%.*]] = add i64 [[N_VEC]], 1
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP4]], [[VECTOR_PH]] ], [ [[TMP15:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI18:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP16:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP6]], [[VECTOR_PH]] ], [ [[TMP17:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI18:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP18:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = or i64 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
-; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i64* [[TMP5]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP6]], align 4
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[TMP5]], i64 2
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = bitcast i64* [[TMP7]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD19:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[TMP7]], i64 2
 ; CHECK-NEXT:    [[TMP10:%.*]] = bitcast i64* [[TMP9]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD20:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
-; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[TMP9]], i64 2
+; CHECK-NEXT:    [[WIDE_LOAD25:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = bitcast i64* [[TMP11]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD21:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
-; CHECK-NEXT:    [[TMP13:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
-; CHECK-NEXT:    [[TMP14:%.*]] = add <2 x i64> [[WIDE_LOAD19]], [[VEC_PHI18]]
-; CHECK-NEXT:    [[TMP15]] = add <2 x i64> [[TMP13]], [[WIDE_LOAD20]]
-; CHECK-NEXT:    [[TMP16]] = add <2 x i64> [[TMP14]], [[WIDE_LOAD21]]
+; CHECK-NEXT:    [[WIDE_LOAD26:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
+; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr i64, i64* [[TMP11]], i64 2
+; CHECK-NEXT:    [[TMP14:%.*]] = bitcast i64* [[TMP13]] to <2 x i64>*
+; CHECK-NEXT:    [[WIDE_LOAD27:%.*]] = load <2 x i64>, <2 x i64>* [[TMP14]], align 4
+; CHECK-NEXT:    [[TMP15:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
+; CHECK-NEXT:    [[TMP16:%.*]] = add <2 x i64> [[WIDE_LOAD25]], [[VEC_PHI18]]
+; CHECK-NEXT:    [[TMP17]] = add <2 x i64> [[TMP15]], [[WIDE_LOAD26]]
+; CHECK-NEXT:    [[TMP18]] = add <2 x i64> [[TMP16]], [[WIDE_LOAD27]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP17:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP17]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-NEXT:    [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP19]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP16]], [[TMP15]]
-; CHECK-NEXT:    [[TMP18:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
-; CHECK-NEXT:    br label [[LOOP_PREHEADER22]]
-; CHECK:       loop.preheader22:
+; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP18]], [[TMP17]]
+; CHECK-NEXT:    [[TMP20:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
+; CHECK-NEXT:    br label [[LOOP_PREHEADER28]]
+; CHECK:       loop.preheader28:
 ; CHECK-NEXT:    [[IV_PH:%.*]] = phi i64 [ 1, [[LOOP_PREHEADER]] ], [ [[IND_END]], [[MIDDLE_BLOCK]] ]
-; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP18]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP20]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER22]] ]
-; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12]] ], [ [[SUM_PH]], [[LOOP_PREHEADER22]] ]
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER28]] ]
+; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT12]] ], [ [[SUM_PH]], [[LOOP_PREHEADER28]] ]
 ; CHECK-NEXT:    [[INRANGE_I:%.*]] = icmp ult i64 [[SUB_I]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I]], label [[ERROR_I:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT:%.*]]
-; CHECK:       error.i:
-; CHECK-NEXT:    tail call void @error()
-; CHECK-NEXT:    unreachable
-; CHECK:       at_with_int_conversion.exit:
 ; CHECK-NEXT:    [[INRANGE_I8:%.*]] = icmp ult i64 [[SUB_I7_PEEL]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I8]], label [[ERROR_I11:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT12]]
-; CHECK:       error.i11:
+; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[INRANGE_I]], i1 true, i1 [[INRANGE_I8]]
+; CHECK-NEXT:    br i1 [[OR_COND]], label [[ERROR_I:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT12]]
+; CHECK:       error.i:
 ; CHECK-NEXT:    tail call void @error()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       at_with_int_conversion.exit12:
 ; CHECK-NEXT:    [[GEP_IDX_I:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I:%.*]] = load i64, i64* [[GEP_IDX_I]], align 4
 ; CHECK-NEXT:    [[GEP_IDX_I9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I10:%.*]] = load i64, i64* [[GEP_IDX_I9]], align 4
 ; CHECK-NEXT:    [[ADD:%.*]] = add i64 [[LV_I]], [[SUM]]

 ; CHECK-NEXT:    [[SUB_I19_PEEL:%.*]] = sub i64 [[END_INT_I18_PEEL]], [[START_INT_I17_PEEL]]
 ; CHECK-NEXT:    [[LV_I10_PEEL:%.*]] = load i64, i64* [[START_I2_PEEL]], align 4
 ; CHECK-NEXT:    [[LV_I22_PEEL:%.*]] = load i64, i64* [[START_I14_PEEL]], align 4
 ; CHECK-NEXT:    [[ADD_2_PEEL:%.*]] = add i64 [[LV_I_PEEL]], [[LV_I10_PEEL]]
 ; CHECK-NEXT:    [[SUM_NEXT_PEEL:%.*]] = add i64 [[ADD_2_PEEL]], [[LV_I22_PEEL]]
 ; CHECK-NEXT:    [[COND_PEEL:%.*]] = icmp sgt i64 [[N:%.*]], 0
 ; CHECK-NEXT:    br i1 [[COND_PEEL]], label [[LOOP_PREHEADER:%.*]], label [[EXIT:%.*]]
 ; CHECK:       loop.preheader:
+; CHECK-NEXT:    [[TMP0:%.*]] = icmp eq i64 [[SUB_I7_PEEL]], 0
 ; CHECK-NEXT:    [[UMIN:%.*]] = call i64 @llvm.umin.i64(i64 [[SUB_I19_PEEL]], i64 [[SUB_I7_PEEL]])
-; CHECK-NEXT:    [[UMIN28:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN]], i64 [[SUB_I]])
-; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[N]], -1
-; CHECK-NEXT:    [[UMIN29:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN28]], i64 [[TMP0]])
-; CHECK-NEXT:    [[TMP1:%.*]] = add i64 [[UMIN29]], 1
-; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP1]], 5
-; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER37:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = select i1 [[TMP0]], i64 0, i64 [[UMIN]]
+; CHECK-NEXT:    [[UMIN28:%.*]] = call i64 @llvm.umin.i64(i64 [[TMP1]], i64 [[SUB_I]])
+; CHECK-NEXT:    [[TMP2:%.*]] = add i64 [[N]], -1
+; CHECK-NEXT:    [[UMIN29:%.*]] = call i64 @llvm.umin.i64(i64 [[UMIN28]], i64 [[TMP2]])
+; CHECK-NEXT:    [[TMP3:%.*]] = add i64 [[UMIN29]], 1
+; CHECK-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[TMP3]], 5
+; CHECK-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[LOOP_PREHEADER43:%.*]], label [[VECTOR_PH:%.*]]
 ; CHECK:       vector.ph:
-; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP1]], 3
-; CHECK-NEXT:    [[TMP2:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
-; CHECK-NEXT:    [[TMP3:%.*]] = select i1 [[TMP2]], i64 4, i64 [[N_MOD_VF]]
-; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP1]], [[TMP3]]
+; CHECK-NEXT:    [[N_MOD_VF:%.*]] = and i64 [[TMP3]], 3
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i64 [[N_MOD_VF]], 0
+; CHECK-NEXT:    [[TMP5:%.*]] = select i1 [[TMP4]], i64 4, i64 [[N_MOD_VF]]
+; CHECK-NEXT:    [[N_VEC:%.*]] = sub i64 [[TMP3]], [[TMP5]]
 ; CHECK-NEXT:    [[IND_END:%.*]] = add i64 [[N_VEC]], 1
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x i64> <i64 poison, i64 0>, i64 [[SUM_NEXT_PEEL]], i64 0
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP4]], [[VECTOR_PH]] ], [ [[TMP21:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT:    [[VEC_PHI31:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP22:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI:%.*]] = phi <2 x i64> [ [[TMP6]], [[VECTOR_PH]] ], [ [[TMP23:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT:    [[VEC_PHI31:%.*]] = phi <2 x i64> [ zeroinitializer, [[VECTOR_PH]] ], [ [[TMP24:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[OFFSET_IDX:%.*]] = or i64 [[INDEX]], 1
-; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
-; CHECK-NEXT:    [[TMP6:%.*]] = bitcast i64* [[TMP5]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP6]], align 4
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[TMP5]], i64 2
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP8:%.*]] = bitcast i64* [[TMP7]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD32:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
-; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <2 x i64>, <2 x i64>* [[TMP8]], align 4
+; CHECK-NEXT:    [[TMP9:%.*]] = getelementptr i64, i64* [[TMP7]], i64 2
 ; CHECK-NEXT:    [[TMP10:%.*]] = bitcast i64* [[TMP9]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD33:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
-; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[TMP9]], i64 2
+; CHECK-NEXT:    [[WIDE_LOAD38:%.*]] = load <2 x i64>, <2 x i64>* [[TMP10]], align 4
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP12:%.*]] = bitcast i64* [[TMP11]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD34:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
-; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr i64, i64* [[START_I14_PEEL]], i64 [[OFFSET_IDX]]
+; CHECK-NEXT:    [[WIDE_LOAD39:%.*]] = load <2 x i64>, <2 x i64>* [[TMP12]], align 4
+; CHECK-NEXT:    [[TMP13:%.*]] = getelementptr i64, i64* [[TMP11]], i64 2
 ; CHECK-NEXT:    [[TMP14:%.*]] = bitcast i64* [[TMP13]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD35:%.*]] = load <2 x i64>, <2 x i64>* [[TMP14]], align 4
-; CHECK-NEXT:    [[TMP15:%.*]] = getelementptr i64, i64* [[TMP13]], i64 2
+; CHECK-NEXT:    [[WIDE_LOAD40:%.*]] = load <2 x i64>, <2 x i64>* [[TMP14]], align 4
+; CHECK-NEXT:    [[TMP15:%.*]] = getelementptr i64, i64* [[START_I14_PEEL]], i64 [[OFFSET_IDX]]
 ; CHECK-NEXT:    [[TMP16:%.*]] = bitcast i64* [[TMP15]] to <2 x i64>*
-; CHECK-NEXT:    [[WIDE_LOAD36:%.*]] = load <2 x i64>, <2 x i64>* [[TMP16]], align 4
-; CHECK-NEXT:    [[TMP17:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
-; CHECK-NEXT:    [[TMP18:%.*]] = add <2 x i64> [[WIDE_LOAD32]], [[VEC_PHI31]]
-; CHECK-NEXT:    [[TMP19:%.*]] = add <2 x i64> [[TMP17]], [[WIDE_LOAD33]]
-; CHECK-NEXT:    [[TMP20:%.*]] = add <2 x i64> [[TMP18]], [[WIDE_LOAD34]]
-; CHECK-NEXT:    [[TMP21]] = add <2 x i64> [[TMP19]], [[WIDE_LOAD35]]
-; CHECK-NEXT:    [[TMP22]] = add <2 x i64> [[TMP20]], [[WIDE_LOAD36]]
+; CHECK-NEXT:    [[WIDE_LOAD41:%.*]] = load <2 x i64>, <2 x i64>* [[TMP16]], align 4
+; CHECK-NEXT:    [[TMP17:%.*]] = getelementptr i64, i64* [[TMP15]], i64 2
+; CHECK-NEXT:    [[TMP18:%.*]] = bitcast i64* [[TMP17]] to <2 x i64>*
+; CHECK-NEXT:    [[WIDE_LOAD42:%.*]] = load <2 x i64>, <2 x i64>* [[TMP18]], align 4
+; CHECK-NEXT:    [[TMP19:%.*]] = add <2 x i64> [[WIDE_LOAD]], [[VEC_PHI]]
+; CHECK-NEXT:    [[TMP20:%.*]] = add <2 x i64> [[WIDE_LOAD38]], [[VEC_PHI31]]
+; CHECK-NEXT:    [[TMP21:%.*]] = add <2 x i64> [[TMP19]], [[WIDE_LOAD39]]
+; CHECK-NEXT:    [[TMP22:%.*]] = add <2 x i64> [[TMP20]], [[WIDE_LOAD40]]
+; CHECK-NEXT:    [[TMP23]] = add <2 x i64> [[TMP21]], [[WIDE_LOAD41]]
+; CHECK-NEXT:    [[TMP24]] = add <2 x i64> [[TMP22]], [[WIDE_LOAD42]]
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT:    [[TMP23:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT:    br i1 [[TMP23]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; CHECK-NEXT:    [[TMP25:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT:    br i1 [[TMP25]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
 ; CHECK:       middle.block:
-; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP22]], [[TMP21]]
-; CHECK-NEXT:    [[TMP24:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
-; CHECK-NEXT:    br label [[LOOP_PREHEADER37]]
-; CHECK:       loop.preheader37:
+; CHECK-NEXT:    [[BIN_RDX:%.*]] = add <2 x i64> [[TMP24]], [[TMP23]]
+; CHECK-NEXT:    [[TMP26:%.*]] = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> [[BIN_RDX]])
+; CHECK-NEXT:    br label [[LOOP_PREHEADER43]]
+; CHECK:       loop.preheader43:
 ; CHECK-NEXT:    [[IV_PH:%.*]] = phi i64 [ 1, [[LOOP_PREHEADER]] ], [ [[IND_END]], [[MIDDLE_BLOCK]] ]
-; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP24]], [[MIDDLE_BLOCK]] ]
+; CHECK-NEXT:    [[SUM_PH:%.*]] = phi i64 [ [[SUM_NEXT_PEEL]], [[LOOP_PREHEADER]] ], [ [[TMP26]], [[MIDDLE_BLOCK]] ]
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER37]] ]
-; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24]] ], [ [[SUM_PH]], [[LOOP_PREHEADER37]] ]
+; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ [[IV_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24:%.*]] ], [ [[IV_PH]], [[LOOP_PREHEADER43]] ]
+; CHECK-NEXT:    [[SUM:%.*]] = phi i64 [ [[SUM_NEXT:%.*]], [[AT_WITH_INT_CONVERSION_EXIT24]] ], [ [[SUM_PH]], [[LOOP_PREHEADER43]] ]
 ; CHECK-NEXT:    [[INRANGE_I:%.*]] = icmp ult i64 [[SUB_I]], [[IV]]
 ; CHECK-NEXT:    br i1 [[INRANGE_I]], label [[ERROR_I:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT:%.*]]
 ; CHECK:       error.i:
 ; CHECK-NEXT:    tail call void @error()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       at_with_int_conversion.exit:
-; CHECK-NEXT:    [[GEP_IDX_I:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[IV]]
-; CHECK-NEXT:    [[LV_I:%.*]] = load i64, i64* [[GEP_IDX_I]], align 4
 ; CHECK-NEXT:    [[INRANGE_I8:%.*]] = icmp ult i64 [[SUB_I7_PEEL]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I8]], label [[ERROR_I11:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT12:%.*]]
-; CHECK:       error.i11:
-; CHECK-NEXT:    tail call void @error()
-; CHECK-NEXT:    unreachable
-; CHECK:       at_with_int_conversion.exit12:
 ; CHECK-NEXT:    [[INRANGE_I20:%.*]] = icmp ult i64 [[SUB_I19_PEEL]], [[IV]]
-; CHECK-NEXT:    br i1 [[INRANGE_I20]], label [[ERROR_I23:%.*]], label [[AT_WITH_INT_CONVERSION_EXIT24]]
-; CHECK:       error.i23:
-; CHECK-NEXT:    tail call void @error()
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    [[OR_COND:%.*]] = select i1 [[INRANGE_I8]], i1 true, i1 [[INRANGE_I20]]
+; CHECK-NEXT:    br i1 [[OR_COND]], label [[ERROR_I]], label [[AT_WITH_INT_CONVERSION_EXIT24]]
 ; CHECK:       at_with_int_conversion.exit24:
+; CHECK-NEXT:    [[GEP_IDX_I:%.*]] = getelementptr i64, i64* [[START_I]], i64 [[IV]]
+; CHECK-NEXT:    [[LV_I:%.*]] = load i64, i64* [[GEP_IDX_I]], align 4
 ; CHECK-NEXT:    [[GEP_IDX_I9:%.*]] = getelementptr i64, i64* [[START_I2_PEEL]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I10:%.*]] = load i64, i64* [[GEP_IDX_I9]], align 4
 ; CHECK-NEXT:    [[GEP_IDX_I21:%.*]] = getelementptr i64, i64* [[START_I14_PEEL]], i64 [[IV]]
 ; CHECK-NEXT:    [[LV_I22:%.*]] = load i64, i64* [[GEP_IDX_I21]], align 4
 ; CHECK-NEXT:    [[ADD_1:%.*]] = add i64 [[LV_I]], [[SUM]]
 ; CHECK-NEXT:    [[ADD_2:%.*]] = add i64 [[ADD_1]], [[LV_I10]]
 ; CHECK-NEXT:    [[SUM_NEXT]] = add i64 [[ADD_2]], [[LV_I22]]
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw nsw i64 [[IV]], 1

llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll

if.end:
   ret void
 }
 
 ; normal destinations are not unreachable and different but could be merged
 define void @t30_completely_different_normal_dests() personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 ; CHECK-LABEL: @t30_completely_different_normal_dests(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
-; CHECK-NEXT:    br i1 [[C0]], label [[IF_THEN0:%.*]], label [[IF_ELSE:%.*]]
-; CHECK:       if.then0:
-; CHECK-NEXT:    invoke void @maybe_throw()
-; CHECK-NEXT:    to label [[INVOKE_CONT0:%.*]] unwind label [[LPAD:%.*]]
+; CHECK-NEXT:    br i1 [[C0]], label [[IF_THEN1_INVOKE:%.*]], label [[IF_ELSE:%.*]]
 ; CHECK:       invoke.cont0:
 ; CHECK-NEXT:    call void @sideeffect()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       lpad:
 ; CHECK-NEXT:    [[EH:%.*]] = landingpad { i8*, i32 }
 ; CHECK-NEXT:    cleanup
 ; CHECK-NEXT:    call void @destructor()
 ; CHECK-NEXT:    resume { i8*, i32 } [[EH]]
 ; CHECK:       if.else:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
-; CHECK-NEXT:    br i1 [[C1]], label [[IF_THEN1:%.*]], label [[IF_END:%.*]]
-; CHECK:       if.then1:
+; CHECK-NEXT:    br i1 [[C1]], label [[IF_THEN1_INVOKE]], label [[IF_END:%.*]]
+; CHECK:       if.then1.invoke:
 ; CHECK-NEXT:    invoke void @maybe_throw()
-; CHECK-NEXT:    to label [[INVOKE_CONT2:%.*]] unwind label [[LPAD]]
-; CHECK:       invoke.cont2:
-; CHECK-NEXT:    call void @sideeffect()
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    to label [[INVOKE_CONT0:%.*]] unwind label [[LPAD:%.*]]
 ; CHECK:       if.end:
 ; CHECK-NEXT:    call void @sideeffect()
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c0 = call i1 @cond()
   br i1 %c0, label %if.then0, label %if.else
 

llvm/test/Transforms/SimplifyCFG/implied-cond.ll

 ; i condition since it could have overflowed.
 define void @test_neg(i32 %length.i, i32 %i) {
 ; CHECK-LABEL: @test_neg(
 ; CHECK-NEXT:    [[IPLUS1:%.*]] = add i32 [[I:%.*]], 1
 ; CHECK-NEXT:    [[VAR29:%.*]] = icmp slt i32 [[IPLUS1]], [[LENGTH_I:%.*]]
 ; CHECK-NEXT:    br i1 [[VAR29]], label [[NEXT:%.*]], label [[OUT_OF_BOUNDS:%.*]]
 ; CHECK:       next:
 ; CHECK-NEXT:    [[VAR30:%.*]] = icmp slt i32 [[I]], [[LENGTH_I]]
-; CHECK-NEXT:    br i1 [[VAR30]], label [[IN_BOUNDS:%.*]], label [[OUT_OF_BOUNDS2:%.*]]
+; CHECK-NEXT:    br i1 [[VAR30]], label [[IN_BOUNDS:%.*]], label [[OUT_OF_BOUNDS]]
 ; CHECK:       in_bounds:
 ; CHECK-NEXT:    ret void
 ; CHECK:       out_of_bounds:
-; CHECK-NEXT:    call void @foo(i64 0)
-; CHECK-NEXT:    unreachable
-; CHECK:       out_of_bounds2:
-; CHECK-NEXT:    call void @foo(i64 1)
+; CHECK-NEXT:    [[TMP1:%.*]] = phi i64 [ 0, [[TMP0:%.*]] ], [ 1, [[NEXT]] ]
+; CHECK-NEXT:    call void @foo(i64 [[TMP1]])
 ; CHECK-NEXT:    unreachable
 ;
   %iplus1 = add i32 %i, 1
   %var29 = icmp slt i32 %iplus1, %length.i
   br i1 %var29, label %next, label %out_of_bounds
 
 next:
   %var30 = icmp slt i32 %i, %length.i

llvm/test/Transforms/SimplifyCFG/iterative-simplify.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -simplifycfg -simplifycfg-require-and-preserve-domtree=1 -S | FileCheck %s
 ; PR1786
 
 define i32 @main() {
 ; CHECK-LABEL: @main(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[RETVAL:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[I:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[Z:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    [[Z16:%.*]] = alloca i32, align 4
 ; CHECK-NEXT:    %"alloca point" = bitcast i32 0 to i32
 ; CHECK-NEXT:    store i32 0, i32* [[I]], align 4
 ; CHECK-NEXT:    [[TOBOOL:%.*]] = icmp ne i8 1, 0
-; CHECK-NEXT:    br i1 [[TOBOOL]], label [[COND_TRUE:%.*]], label [[COND_FALSE33:%.*]]
+; CHECK-NEXT:    br i1 [[TOBOOL]], label [[COND_TRUE:%.*]], label [[BB13:%.*]]
 ; CHECK:       cond_true:
 ; CHECK-NEXT:    store i32 0, i32* [[Z]], align 4
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
 ; CHECK-NEXT:    [[TMP:%.*]] = load i32, i32* [[Z]], align 4
 ; CHECK-NEXT:    [[TMP1:%.*]] = sub i32 [[TMP]], 16384
 ; CHECK-NEXT:    store i32 [[TMP1]], i32* [[Z]], align 4
 ; CHECK-NEXT:    [[TMP2:%.*]] = load i32, i32* [[I]], align 4
 ; CHECK-NEXT:    [[TMP3:%.*]] = add i32 [[TMP2]], 1
 ; CHECK-NEXT:    store i32 [[TMP3]], i32* [[I]], align 4
 ; CHECK-NEXT:    [[TMP4:%.*]] = load i32, i32* [[I]], align 4
 ; CHECK-NEXT:    [[TMP5:%.*]] = icmp sgt i32 [[TMP4]], 262144
 ; CHECK-NEXT:    [[TMP56:%.*]] = zext i1 [[TMP5]] to i8
 ; CHECK-NEXT:    [[TOBOOL7:%.*]] = icmp ne i8 [[TMP56]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL7]], label [[COND_TRUE8:%.*]], label [[COND_NEXT:%.*]]
 ; CHECK:       cond_true8:
 ; CHECK-NEXT:    call void @abort()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cond_next:
 ; CHECK-NEXT:    [[TMP9:%.*]] = load i32, i32* [[Z]], align 4
 ; CHECK-NEXT:    [[TMP10:%.*]] = icmp ne i32 [[TMP9]], 0
 ; CHECK-NEXT:    [[TMP1011:%.*]] = zext i1 [[TMP10]] to i8
 ; CHECK-NEXT:    [[TOBOOL12:%.*]] = icmp ne i8 [[TMP1011]], 0
-; CHECK-NEXT:    br i1 [[TOBOOL12]], label [[BB]], label [[BB13:%.*]]
+; CHECK-NEXT:    br i1 [[TOBOOL12]], label [[BB]], label [[BB13]]
 ; CHECK:       bb13:
 ; CHECK-NEXT:    call void @exit(i32 0)
 ; CHECK-NEXT:    unreachable
-; CHECK:       cond_false33:
-; CHECK-NEXT:    call void @exit(i32 0)
-; CHECK-NEXT:    unreachable
 ;
 entry:
   %retval = alloca i32, align 4   ; <i32*> [#uses=1]
   %i = alloca i32, align 4    ; <i32*> [#uses=7]
   %z = alloca i32, align 4    ; <i32*> [#uses=4]
   %z16 = alloca i32, align 4    ; <i32*> [#uses=4]
   %"alloca point" = bitcast i32 0 to i32    ; <i32> [#uses=0]
   store i32 0, i32* %i

llvm/test/Transforms/SimplifyCFG/tail-merge-assert.ll

 declare void @glibc_assert_fail(i8*, i8*, i32, i8*)
 
 define void @merge_glibc_asserts(i32 %x, i32 %y) {
 ; CHECK-LABEL: @merge_glibc_asserts(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp ugt i32 [[Y:%.*]], [[X:%.*]]
 ; CHECK-NEXT:    br i1 [[CMP]], label [[COND_END:%.*]], label [[COND_FALSE:%.*]]
 ; CHECK:       cond.false:
-; CHECK-NEXT:    tail call void @glibc_assert_fail(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str, i64 0, i64 0), i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str.1, i64 0, i64 0), i32 3, i8* getelementptr inbounds ([35 x i8], [35 x i8]* @__PRETTY_FUNCTION__._Z1fjj, i64 0, i64 0))
+; CHECK-NEXT:    [[TMP0:%.*]] = phi i8* [ getelementptr inbounds ([6 x i8], [6 x i8]* @.str, i64 0, i64 0), [[ENTRY:%.*]] ], [ getelementptr inbounds ([10 x i8], [10 x i8]* @.str.2, i64 0, i64 0), [[COND_END]] ], [ getelementptr inbounds ([11 x i8], [11 x i8]* @.str.3, i64 0, i64 0), [[COND_END4:%.*]] ]
+; CHECK-NEXT:    [[TMP1:%.*]] = phi i32 [ 3, [[ENTRY]] ], [ 4, [[COND_END]] ], [ 5, [[COND_END4]] ]
+; CHECK-NEXT:    tail call void @glibc_assert_fail(i8* [[TMP0]], i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str.1, i64 0, i64 0), i32 [[TMP1]], i8* getelementptr inbounds ([35 x i8], [35 x i8]* @__PRETTY_FUNCTION__._Z1fjj, i64 0, i64 0))
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cond.end:
 ; CHECK-NEXT:    [[SUB:%.*]] = sub i32 [[Y]], [[X]]
 ; CHECK-NEXT:    [[CMP1:%.*]] = icmp ugt i32 [[SUB]], 7
-; CHECK-NEXT:    br i1 [[CMP1]], label [[COND_END4:%.*]], label [[COND_FALSE3:%.*]]
-; CHECK:       cond.false3:
-; CHECK-NEXT:    tail call void @glibc_assert_fail(i8* getelementptr inbounds ([10 x i8], [10 x i8]* @.str.2, i64 0, i64 0), i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str.1, i64 0, i64 0), i32 4, i8* getelementptr inbounds ([35 x i8], [35 x i8]* @__PRETTY_FUNCTION__._Z1fjj, i64 0, i64 0))
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[CMP1]], label [[COND_END4]], label [[COND_FALSE]]
 ; CHECK:       cond.end4:
 ; CHECK-NEXT:    [[CMP6:%.*]] = icmp ult i32 [[SUB]], 40
-; CHECK-NEXT:    br i1 [[CMP6]], label [[COND_END9:%.*]], label [[COND_FALSE8:%.*]]
-; CHECK:       cond.false8:
-; CHECK-NEXT:    tail call void @glibc_assert_fail(i8* getelementptr inbounds ([11 x i8], [11 x i8]* @.str.3, i64 0, i64 0), i8* getelementptr inbounds ([6 x i8], [6 x i8]* @.str.1, i64 0, i64 0), i32 5, i8* getelementptr inbounds ([35 x i8], [35 x i8]* @__PRETTY_FUNCTION__._Z1fjj, i64 0, i64 0))
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[CMP6]], label [[COND_END9:%.*]], label [[COND_FALSE]]
 ; CHECK:       cond.end9:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %cmp = icmp ugt i32 %y, %x
   br i1 %cmp, label %cond.end, label %cond.false
 
 cond.false:                                       ; preds = %entry

llvm/test/Transforms/SimplifyCFG/tail-merge-noreturn.ll

 ; CHECK-LABEL: @merge_simple(
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
 ; CHECK:       a1:
 ; CHECK-NEXT:    call void @assert_fail_1(i32 0)
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A1]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A3:%.*]]
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A1]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 0)
   unreachable

 }
 
 define void @phi_three_constants() {
 ; CHECK-LABEL: @phi_three_constants(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
 ; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
+; CHECK-NEXT:    [[TMP0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ 1, [[CONT1]] ], [ 2, [[CONT2:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[TMP0]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 1)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2]], label [[A1]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A3:%.*]]
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 2)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A1]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 0)
cont3:
   ret void
 }
 
 define void @dont_phi_values(i32 %x, i32 %y) {
 ; CHECK-LABEL: @dont_phi_values(
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
 ; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[X:%.*]])
+; CHECK-NEXT:    [[TMP1:%.*]] = phi i32 [ [[X:%.*]], [[TMP0:%.*]] ], [ [[Y:%.*]], [[CONT1]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[TMP1]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[Y:%.*]])
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A1]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    ret void
 ;
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 %x)
   unreachable

 ; CHECK-NEXT:    br i1 [[C:%.*]], label [[S1:%.*]], label [[S2:%.*]]
 ; CHECK:       s1:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C1]], label [[A1:%.*]], label [[A2:%.*]]
 ; CHECK:       s2:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @bar()
 ; CHECK-NEXT:    br i1 [[C2]], label [[A1]], label [[A2]]
 ; CHECK:       a1:
-; CHECK-NEXT:    [[L1:%.*]] = phi i32 [ 0, [[S1]] ], [ 1, [[S2]] ]
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[L1]])
+; CHECK-NEXT:    [[TMP0:%.*]] = phi i32 [ [[L2:%.*]], [[A2]] ], [ 0, [[S1]] ], [ 1, [[S2]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[TMP0]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       a2:
-; CHECK-NEXT:    [[L2:%.*]] = phi i32 [ 2, [[S1]] ], [ 3, [[S2]] ]
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[L2]])
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    [[L2]] = phi i32 [ 2, [[S1]] ], [ 3, [[S2]] ]
+; CHECK-NEXT:    br label [[A1]]
 ;
 entry:
   br i1 %c, label %s1, label %s2
 s1:
   %c1 = call i1 @foo()
   br i1 %c1, label %a1, label %a2
 s2:
   %c2 = call i1 @bar()

 ; CHECK-LABEL: @tail_merge_switch(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    switch i32 [[V:%.*]], label [[RET:%.*]] [
 ; CHECK-NEXT:    i32 0, label [[A1:%.*]]
 ; CHECK-NEXT:    i32 13, label [[A2:%.*]]
 ; CHECK-NEXT:    i32 42, label [[A3:%.*]]
 ; CHECK-NEXT:    ]
 ; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
+; CHECK-NEXT:    [[TMP0:%.*]] = phi i32 [ 1, [[A2]] ], [ 2, [[A3]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[TMP0]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 1)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br label [[A1]]
 ; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 2)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br label [[A1]]
 ; CHECK:       ret:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   switch i32 %v, label %ret [
   i32 0, label %a1
   i32 13, label %a2
   i32 42, label %a3

 }
 
 define void @need_to_add_bb2_preds(i1 %c1) {
 ; CHECK-LABEL: @need_to_add_bb2_preds(
 ; CHECK-NEXT:  bb1:
 ; CHECK-NEXT:    br i1 [[C1:%.*]], label [[BB2:%.*]], label [[A1:%.*]]
 ; CHECK:       bb2:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @bar()
-; CHECK-NEXT:    br i1 [[C2]], label [[A2:%.*]], label [[A3:%.*]]
+; CHECK-NEXT:    [[DOT:%.*]] = select i1 [[C2]], i32 1, i32 2
+; CHECK-NEXT:    br label [[A1]]
 ; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
-; CHECK:       a2:
-; CHECK-NEXT:    call void @assert_fail_1(i32 1)
-; CHECK-NEXT:    unreachable
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 2)
+; CHECK-NEXT:    [[TMP0:%.*]] = phi i32 [ [[DOT]], [[BB2]] ], [ 0, [[BB1:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[TMP0]])
 ; CHECK-NEXT:    unreachable
 ;
 bb1:
   br i1 %c1, label %bb2, label %a1
 bb2:
   %c2 = call i1 @bar()
   br i1 %c2, label %a2, label %a3
 

 }
 
 define void @phi_in_bb2() {
 ; CHECK-LABEL: @phi_in_bb2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
 ; CHECK:       a1:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
+; CHECK-NEXT:    [[TMP0:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ 1, [[CONT1]] ], [ 2, [[CONT2:%.*]] ]
+; CHECK-NEXT:    call void @assert_fail_1(i32 [[TMP0]])
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A2:%.*]]
-; CHECK:       a2:
-; CHECK-NEXT:    [[P2:%.*]] = phi i32 [ 1, [[CONT1]] ], [ 2, [[CONT2]] ]
-; CHECK-NEXT:    call void @assert_fail_1(i32 [[P2]])
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C2]], label [[CONT2]], label [[A1]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A2]]
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A1]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   call void @assert_fail_1(i32 0)

 ; CHECK-NEXT:    store i32 0, i32* [[X]], align 4
 ; CHECK-NEXT:    [[TOBOOL:%.*]] = icmp eq i32 [[C2:%.*]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL]], label [[IF_END:%.*]], label [[IF_THEN1:%.*]]
 ; CHECK:       if.then1:
 ; CHECK-NEXT:    call void @escape_i32_ptr(i32* nonnull [[X]])
 ; CHECK-NEXT:    br label [[IF_END]]
 ; CHECK:       if.end:
 ; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 4, i8* nonnull [[TMP0]])
+; CHECK-NEXT:    br label [[TMP1:%.*]]
+; CHECK:       1:
 ; CHECK-NEXT:    call void @abort()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       if.then3:
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[Y]] to i8*
-; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 4, i8* nonnull [[TMP1]])
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i32* [[Y]] to i8*
+; CHECK-NEXT:    call void @llvm.lifetime.start.p0i8(i64 4, i8* nonnull [[TMP2]])
 ; CHECK-NEXT:    store i32 0, i32* [[Y]], align 4
 ; CHECK-NEXT:    [[TOBOOL5:%.*]] = icmp eq i32 [[C2]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL5]], label [[IF_END7:%.*]], label [[IF_THEN6:%.*]]
 ; CHECK:       if.then6:
 ; CHECK-NEXT:    call void @escape_i32_ptr(i32* nonnull [[Y]])
 ; CHECK-NEXT:    br label [[IF_END7]]
 ; CHECK:       if.end7:
-; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 4, i8* nonnull [[TMP1]])
-; CHECK-NEXT:    call void @abort()
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    call void @llvm.lifetime.end.p0i8(i64 4, i8* nonnull [[TMP2]])
+; CHECK-NEXT:    br label [[TMP1]]
 ; CHECK:       if.end9:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %x = alloca i32, align 4
   %y = alloca i32, align 4
   switch i32 %c1, label %if.end9 [
   i32 13, label %if.then

 declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 
 define void @dead_phi() {
 ; CHECK-LABEL: @dead_phi(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C1]], label [[CONT1:%.*]], label [[A1:%.*]]
 ; CHECK:       a1:
-; CHECK-NEXT:    [[DEAD:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ 1, [[CONT1]] ]
 ; CHECK-NEXT:    call void @assert_fail_1(i32 0)
 ; CHECK-NEXT:    unreachable
 ; CHECK:       cont1:
 ; CHECK-NEXT:    [[C2:%.*]] = call i1 @foo()
 ; CHECK-NEXT:    br i1 [[C2]], label [[CONT2:%.*]], label [[A1]]
 ; CHECK:       cont2:
 ; CHECK-NEXT:    [[C3:%.*]] = call i1 @foo()
-; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A3:%.*]]
-; CHECK:       a3:
-; CHECK-NEXT:    call void @assert_fail_1(i32 0)
-; CHECK-NEXT:    unreachable
+; CHECK-NEXT:    br i1 [[C3]], label [[CONT3:%.*]], label [[A1]]
 ; CHECK:       cont3:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %c1 = call i1 @foo()
   br i1 %c1, label %cont1, label %a1
 a1:
   %dead = phi i32 [ 0, %entry ], [ 1, %cont1 ]

 }
 
 define void @strip_dbg_value(i32 %c) {
 ; CHECK-LABEL: @strip_dbg_value(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 [[C:%.*]], metadata [[META5:![0-9]+]], metadata !DIExpression()), !dbg [[DBG7:![0-9]+]]
 ; CHECK-NEXT:    switch i32 [[C]], label [[SW_EPILOG:%.*]] [
 ; CHECK-NEXT:    i32 13, label [[SW_BB:%.*]]
-; CHECK-NEXT:    i32 42, label [[SW_BB1:%.*]]
+; CHECK-NEXT:    i32 42, label [[SW_BB]]
 ; CHECK-NEXT:    ]
 ; CHECK:       sw.bb:
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 55, metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
-; CHECK-NEXT:    tail call void @abort()
-; CHECK-NEXT:    unreachable
-; CHECK:       sw.bb1:
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 67, metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
 ; CHECK-NEXT:    tail call void @abort()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       sw.epilog:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   call void @llvm.dbg.value(metadata i32 %c, i64 0, metadata !12, metadata !13), !dbg !14
   switch i32 %c, label %sw.epilog [

 }
 
 define void @dead_phi_and_dbg(i32 %c) {
 ; CHECK-LABEL: @dead_phi_and_dbg(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 [[C:%.*]], metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
 ; CHECK-NEXT:    switch i32 [[C]], label [[SW_EPILOG:%.*]] [
 ; CHECK-NEXT:    i32 13, label [[SW_BB:%.*]]
-; CHECK-NEXT:    i32 42, label [[SW_BB1:%.*]]
-; CHECK-NEXT:    i32 53, label [[SW_BB2:%.*]]
+; CHECK-NEXT:    i32 42, label [[SW_BB]]
+; CHECK-NEXT:    i32 53, label [[SW_BB]]
 ; CHECK-NEXT:    ]
 ; CHECK:       sw.bb:
-; CHECK-NEXT:    [[C_1:%.*]] = phi i32 [ 55, [[ENTRY:%.*]] ], [ 67, [[SW_BB1]] ]
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 [[C_1]], metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
-; CHECK-NEXT:    tail call void @abort()
-; CHECK-NEXT:    unreachable
-; CHECK:       sw.bb1:
-; CHECK-NEXT:    br label [[SW_BB]]
-; CHECK:       sw.bb2:
-; CHECK-NEXT:    call void @llvm.dbg.value(metadata i32 84, metadata [[META5]], metadata !DIExpression()), !dbg [[DBG7]]
 ; CHECK-NEXT:    tail call void @abort()
 ; CHECK-NEXT:    unreachable
 ; CHECK:       sw.epilog:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   call void @llvm.dbg.value(metadata i32 %c, i64 0, metadata !12, metadata !13), !dbg !14
   switch i32 %c, label %sw.epilog [