[LoopUnswitch] Perform loop unswitching on select instructions

[kachkov98]

Summary

Try to convert select instruction with loop-invariant condition into
control-flow equivalent (if-then branch and phi node) and unswitch
this branch. This unswitching is always non-trivial since the successors
of created branch are not loop exits, so it requires cloning of loop.

[kachkov98]

test-suite statistics:

Program                                                                             simple-loop-unswitch.NumSelects
                                                                                    selects                        
           test-suite :: External/SPEC/CFP2017rate/526.blender_r/526.blender_r.test  52.00                         
                 test-suite :: External/SPEC/CINT2017speed/602.gcc_s/602.gcc_s.test  16.00                         
                  test-suite :: External/SPEC/CINT2017rate/502.gcc_r/502.gcc_r.test  16.00                         
                        test-suite :: MultiSource/Applications/ClamAV/clamscan.test   7.00                         
                      test-suite :: MultiSource/Benchmarks/7zip/7zip-benchmark.test   6.00                         
                       test-suite :: MultiSource/Applications/JM/ldecod/ldecod.test   6.00                         
test-suite :: MultiSource/Benchmarks/MiBench/consumer-typeset/consumer-typeset.test   4.00                         
                   test-suite :: External/SPEC/CINT2017speed/657.xz_s/657.xz_s.test   3.00                         
                          test-suite :: MultiSource/Applications/oggenc/oggenc.test   3.00                         
      test-suite :: External/SPEC/CINT2017rate/500.perlbench_r/500.perlbench_r.test   3.00                         
     test-suite :: External/SPEC/CINT2017speed/600.perlbench_s/600.perlbench_s.test   3.00                         
                    test-suite :: External/SPEC/CINT2017rate/557.xz_r/557.xz_r.test   3.00                         
             test-suite :: External/SPEC/CFP2017rate/511.povray_r/511.povray_r.test   3.00                         
                       test-suite :: MultiSource/Applications/JM/lencod/lencod.test   2.00                         
            test-suite :: MultiSource/Benchmarks/mediabench/jpeg/jpeg-6a/cjpeg.test   2.00                         
      test-suite :: MultiSource/Benchmarks/MiBench/consumer-jpeg/consumer-jpeg.test   2.00                         
      test-suite :: External/SPEC/CINT2017rate/523.xalancbmk_r/523.xalancbmk_r.test   2.00                         
     test-suite :: External/SPEC/CINT2017speed/623.xalancbmk_s/623.xalancbmk_s.test   2.00                         
                            test-suite :: MultiSource/Benchmarks/Bullet/bullet.test   2.00                         
                            test-suite :: MultiSource/Applications/spiff/spiff.test   2.00                         
                            test-suite :: MultiSource/Benchmarks/nbench/nbench.test   1.00                         
        test-suite :: MultiSource/Benchmarks/mediabench/g721/g721encode/encode.test   1.00                         
          test-suite :: MultiSource/Benchmarks/FreeBench/pcompress2/pcompress2.test   1.00                         
                        test-suite :: MultiSource/Applications/sqlite3/sqlite3.test   1.00                         
test-suite :: MultiSource/Benchmarks/MiBench/automotive-susan/automotive-susan.test   1.00                         
          test-suite :: External/SPEC/CFP2017speed/638.imagick_s/638.imagick_s.test   1.00                         
                  test-suite :: MultiSource/Benchmarks/ASC_Sequoia/AMGmk/AMGmk.test   1.00                         
      test-suite :: MultiSource/Benchmarks/MiBench/consumer-lame/consumer-lame.test   1.00                         
                     test-suite :: MultiSource/Benchmarks/mafft/pairlocalalign.test   1.00                         
      test-suite :: MultiSource/Benchmarks/MiBench/office-ispell/office-ispell.test   1.00                         
           test-suite :: External/SPEC/CFP2017rate/538.imagick_r/538.imagick_r.test   1.00                         
             test-suite :: External/SPEC/CFP2017rate/510.parest_r/510.parest_r.test   1.00                         
                                  test-suite :: MultiSource/Benchmarks/sim/sim.test   1.00

I don't observe any measurable execution time change on SPEC2017 with this change, but in some cases unswitching of select looks desirable (like https://godbolt.org/z/1rPTPan7G)

[mkazantsev]

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

3104

It is not an equivalent transform in sense of UB. If the condition was posion, then select by this condition is also poison. If you turn it into a branch, branch by poisoned condition is immediate UB. How do you account for that?

[kachkov98]

Insert freeze of select condition

[kachkov98]

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

3104

Accidentionally it was correct in some tests, because unswitchNontrivialInvariants() inserts freeze for branch conditions if the branch is not guaranteed to execute at least once in loop body; for other loops it will not work, so now we insert freeze of select condition explicitly.

[mkazantsev]

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

2619

Can we change the insertion point to right after the cond? Presumeably it's outside of loop.

2628

This is wrong, you can't insert a Phi before SI if there is another instruction before it. Your insertion point should be smth like block.begin().

2720–2721

This condition is growing weird, can we rewrite it to make it explicit that we choose between guard, select, terminator etc? Maybe factor out into a lambda.

2806–2820

Just for record: this will cost us compile time, turning the algorithm from O(number of blocks) to O(number of instructions) (if guards are off). Keep that in mind if you see compile time regressions.

2812

I would also filter out such obviously useless cases as:

• Select is not used
• Select is only used outside of the loop (and therefore can be sunk)

maybe smth else, give it some more thought.

llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll

4393

To see what your tests are doing, I propose the following steps:

• Regenerate all checks in this file using update_test_checks.py script;
• Check in your new tests with checks as is (without your change in code)
• Then rebase your patch on top of it, to see what exactly your patch has changed.

Without this, it's hard to tell what's right or wrong here.

[mkazantsev]

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

2628

I also propose to give this Phi a name.

[fhahn]

It looks like this tries to address the same issue as D138526. Might be goo to sync up with @caojoshua

[caojoshua]

In D139109#3978212, @fhahn wrote:

It looks like this tries to address the same issue as D138526. Might be goo to sync up with @caojoshua

Can confirm this is addressing the same issue. Our implementations are quite a bit different. It seems you are converting select's into branch's and mostly using existing logic. In my review, I try to directly unswitch the select. Not sure which approach is preferred. I'll let people who have been working on LLVM longer give their thoughts.

Either way, I think we can share some test cases. In my review, I pulled in select tests from the old PM LoopUnswitch pass, and added some checks to existing tests.

[caojoshua]

i had previously attempted to solve the same problem in https://reviews.llvm.org/D138526. @kachkov98 are you still working on this? If not, can I take over?

[kachkov98]

@caojoshua Sure, I prefer your approach without select -> branch convertion more (the weak point here is that unswitching can fail after this conversion, and subsequent passes should fold branch to select back). You can share LIT test if it will be helpful.

[caojoshua]

In D139109#4255024, @kachkov98 wrote:

@caojoshua Sure, I prefer your approach without select -> branch convertion more (the weak point here is that unswitching can fail after this conversion, and subsequent passes should fold branch to select back). You can share LIT test if it will be helpful.

Ha, that's funny. I was actually going to say I prefer your approach and would have liked to work on top of this patch. It does not look possible that a select is converted, and then unswitching fails. By the time you convert a select, it has already been determined a legal unswitch target with the best cost.

This patch uses already existing unswitching logic, while my patch adds a whole bunch of new logic. This patch is much easier to understand.

llvm/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp

 
 #define DEBUG_TYPE "simple-loop-unswitch"
 
 using namespace llvm;
 using namespace llvm::PatternMatch;
 
 STATISTIC(NumBranches, "Number of branches unswitched");
 STATISTIC(NumSwitches, "Number of switches unswitched");
+STATISTIC(NumSelects, "Number of selects turned into branches for unswitching");
 STATISTIC(NumGuards, "Number of guards turned into branches for unswitching");
 STATISTIC(NumTrivial, "Number of unswitches that are trivial");
 STATISTIC(
     NumCostMultiplierSkipped,
     "Number of unswitch candidates that had their cost multiplier skipped");
 
 static cl::opt<bool> EnableNonTrivialUnswitch(
     "enable-nontrivial-unswitch", cl::init(false), cl::Hidden,
InstructionCost Cost = std::accumulate(
         return Sum + computeDomSubtreeCost(*ChildN, BBCostMap, DTCostMap);
       });
   bool Inserted = DTCostMap.insert({&N, Cost}).second;
   (void)Inserted;
   assert(Inserted && "Should not insert a node while visiting children!");
   return Cost;
 }
 
+/// Turns a select instruction into implicit control flow branch,
+/// making the following replacement:
+///
+/// head:
+///   --code before select--
+///   select %cond, %trueval, %falseval
+///   --code after select--
+///
+/// into
+///
+/// head:
+///   --code before select--
+///   br i1 %cond, label %then, label %tail
+///
+/// then:
+///   br %tail
+///
+/// tail:
+///   phi [ %trueval, %then ], [ %falseval, %head]
+///   unreachable
+///
+/// It also makes all relevant DT and LI updates, so that all structures are in
+/// valid state after this transform.
+static BranchInst *turnSelectIntoBranch(SelectInst *SI, DominatorTree &DT,
+                                        LoopInfo &LI, MemorySSAUpdater *MSSAU) {
+  LLVM_DEBUG(dbgs() << "Turning " << *SI << " into a branch.\n");
+  BasicBlock *HeadBB = SI->getParent();
+
+  SplitBlockAndInsertIfThen(SI->getCondition(), SI, false,
+                            SI->getMetadata(LLVMContext::MD_prof), &DT, &LI);
+  auto *CondBr = cast<BranchInst>(HeadBB->getTerminator());
+  BasicBlock *ThenBB = CondBr->getSuccessor(0),
+             *TailBB = CondBr->getSuccessor(1);
+  if (MSSAU)
+    MSSAU->moveAllAfterSpliceBlocks(HeadBB, TailBB, SI);
+
+  PHINode *Phi = PHINode::Create(SI->getType(), 2, "", SI);
+  Phi->addIncoming(SI->getTrueValue(), ThenBB);
+  Phi->addIncoming(SI->getFalseValue(), HeadBB);
+  SI->replaceAllUsesWith(Phi);
+  SI->eraseFromParent();
+
+  if (MSSAU && VerifyMemorySSA)
+    MSSAU->getMemorySSA()->verifyMemorySSA();
+
+  ++NumSelects;
+  return CondBr;
+}
+
 /// Turns a llvm.experimental.guard intrinsic into implicit control flow branch,
 /// making the following replacement:
 ///
 ///   --code before guard--
 ///   call void (i1, ...) @llvm.experimental.guard(i1 %cond) [ "deopt"() ]
 ///   --code after guard--
 ///
 /// into
 ///
 ///   --code before guard--
 ///   br i1 %cond, label %guarded, label %deopt
 ///
 /// guarded:
 ///   --code after guard--
 ///
 /// deopt:
 ///   call void (i1, ...) @llvm.experimental.guard(i1 false) [ "deopt"() ]
 ///   unreachable
 ///
 /// It also makes all relevant DT and LI updates, so that all structures are in
 /// valid state after this transform.
-static BranchInst *turnGuardIntoBranch(IntrinsicInst *GI, Loop &L,
-                                       DominatorTree &DT, LoopInfo &LI,
-                                       MemorySSAUpdater *MSSAU) {
-  SmallVector<DominatorTree::UpdateType, 4> DTUpdates;
+static BranchInst *turnGuardIntoBranch(IntrinsicInst *GI, DominatorTree &DT,
+                                       LoopInfo &LI, MemorySSAUpdater *MSSAU) {
   LLVM_DEBUG(dbgs() << "Turning " << *GI << " into a branch.\n");
   BasicBlock *CheckBB = GI->getParent();
 
   if (MSSAU && VerifyMemorySSA)
      MSSAU->getMemorySSA()->verifyMemorySSA();
 
-  // Remove all CheckBB's successors from DomTree. A block can be seen among
-  // successors more than once, but for DomTree it should be added only once.
-  SmallPtrSet<BasicBlock *, 4> Successors;
-  for (auto *Succ : successors(CheckBB))
-    if (Successors.insert(Succ).second)
-      DTUpdates.push_back({DominatorTree::Delete, CheckBB, Succ});
-
-  Instruction *DeoptBlockTerm =
-      SplitBlockAndInsertIfThen(GI->getArgOperand(0), GI, true);
+  Instruction *DeoptBlockTerm = SplitBlockAndInsertIfThen(
+      GI->getArgOperand(0), GI, true, GI->getMetadata(LLVMContext::MD_prof),
+      &DT, &LI);
   BranchInst *CheckBI = cast<BranchInst>(CheckBB->getTerminator());
   // SplitBlockAndInsertIfThen inserts control flow that branches to
   // DeoptBlockTerm if the condition is true.  We want the opposite.
   CheckBI->swapSuccessors();
 
   BasicBlock *GuardedBlock = CheckBI->getSuccessor(0);
   GuardedBlock->setName("guarded");
   CheckBI->getSuccessor(1)->setName("deopt");
   BasicBlock *DeoptBlock = CheckBI->getSuccessor(1);
 
   if (MSSAU)
     MSSAU->moveAllAfterSpliceBlocks(CheckBB, GuardedBlock, GI);
 
   GI->moveBefore(DeoptBlockTerm);
   GI->setArgOperand(0, ConstantInt::getFalse(GI->getContext()));
 
-  // Add new successors of CheckBB into DomTree.
-  for (auto *Succ : successors(CheckBB))
-    DTUpdates.push_back({DominatorTree::Insert, CheckBB, Succ});
-
-  // Now the blocks that used to be CheckBB's successors are GuardedBlock's
-  // successors.
-  for (auto *Succ : Successors)
-    DTUpdates.push_back({DominatorTree::Insert, GuardedBlock, Succ});
-
-  // Make proper changes to DT.
-  DT.applyUpdates(DTUpdates);
-  // Inform LI of a new loop block.
-  L.addBasicBlockToLoop(GuardedBlock, LI);
-
   if (MSSAU) {
     MemoryDef *MD = cast<MemoryDef>(MSSAU->getMemorySSA()->getMemoryAccess(GI));
     MSSAU->moveToPlace(MD, DeoptBlock, MemorySSA::BeforeTerminator);
     if (VerifyMemorySSA)
       MSSAU->getMemorySSA()->verifyMemorySSA();
   }
 
   ++NumGuards;

 /// TODO: Make a proper accounting of "explosion" effect for all kinds of
 /// unswitch candidates, making adequate predictions instead of wild guesses.
 /// That requires knowing not just the number of "remaining" candidates but
 /// also costs of unswitching for each of these candidates.
 static int CalculateUnswitchCostMultiplier(
     const Instruction &TI, const Loop &L, const LoopInfo &LI,
     const DominatorTree &DT,
     ArrayRef<NonTrivialUnswitchCandidate> UnswitchCandidates) {
-
   // Guards and other exiting conditions do not contribute to exponential
   // explosion as soon as they dominate the latch (otherwise there might be
   // another path to the latch remaining that does not allow to eliminate the
   // loop copy on unswitch).
   const BasicBlock *Latch = L.getLoopLatch();
   const BasicBlock *CondBlock = TI.getParent();
   if (DT.dominates(CondBlock, Latch) &&
       (isGuard(&TI) ||
+       (TI.isTerminator() &&
         llvm::count_if(successors(&TI), [&L](const BasicBlock *SuccBB) {
           return L.contains(SuccBB);
-       }) <= 1)) {
+        }) <= 1))) {
     NumCostMultiplierSkipped++;
     return 1;
   }
 
   auto *ParentL = L.getParentLoop();
   int SiblingsCount = (ParentL ? ParentL->getSubLoopsVector().size()
                                : std::distance(LI.begin(), LI.end()));
   // Count amount of clones that all the candidates might cause during
-  // unswitching. Branch/guard counts as 1, switch counts as log2 of its cases.
+  // unswitching. Branch/guard/select counts as 1, switch counts as log2 of its
+  // cases.
   int UnswitchedClones = 0;
   for (auto Candidate : UnswitchCandidates) {
     const Instruction *CI = Candidate.TI;
     const BasicBlock *CondBlock = CI->getParent();
     bool SkipExitingSuccessors = DT.dominates(CondBlock, Latch);
+    if (isa<SelectInst>(CI)) {
+      UnswitchedClones++;
+      continue;
+    }
     if (isGuard(CI)) {
       if (!SkipExitingSuccessors)
         UnswitchedClones++;
       continue;
     }
     int NonExitingSuccessors =
         llvm::count_if(successors(CondBlock),
                        [SkipExitingSuccessors, &L](const BasicBlock *SuccBB) {
if (UnswitchGuards) {
     if (GuardDecl && !GuardDecl->use_empty())
       CollectGuards = true;
   }
 
   for (auto *BB : L.blocks()) {
     if (LI.getLoopFor(BB) != &L)
       continue;
 
-    if (CollectGuards)
-      for (auto &I : *BB)
-        if (isGuard(&I)) {
-          auto *Cond =
-              skipTrivialSelect(cast<IntrinsicInst>(&I)->getArgOperand(0));
-          // TODO: Support AND, OR conditions and partial unswitching.
-          if (!isa<Constant>(Cond) && L.isLoopInvariant(Cond))
-            UnswitchCandidates.push_back({&I, {Cond}});
-        }
+    for (auto &I : *BB) {
+      if (auto *SI = dyn_cast<SelectInst>(&I)) {
+        if (SI->getType()->isIntegerTy(1))
+          continue;
+        auto *Cond = SI->getCondition();
+        if (!isa<Constant>(Cond) && L.isLoopInvariant(Cond))
+          UnswitchCandidates.push_back({&I, {Cond}});
+      } else if (CollectGuards && isGuard(&I)) {
+    }
 
     if (auto *SI = dyn_cast<SwitchInst>(BB->getTerminator())) {
       // We can only consider fully loop-invariant switch conditions as we need
       // to completely eliminate the switch after unswitching.
       if (!isa<Constant>(SI->getCondition()) &&
           L.isLoopInvariant(SI->getCondition()) && !BB->getUniqueSuccessor())
         UnswitchCandidates.push_back({SI, {SI->getCondition()}});
       continue;
for (BasicBlock *SuccBB : successors(&BB)) {
       }
     }
 
     // Now scale the cost by the number of unique successors minus one. We
     // subtract one because there is already at least one copy of the entire
     // loop. This is computing the new cost of unswitching a condition.
     // Note that guards always have 2 unique successors that are implicit and
     // will be materialized if we decide to unswitch it.
-    int SuccessorsCount = isGuard(&TI) ? 2 : Visited.size();
+    int SuccessorsCount =
+        isGuard(&TI) || isa<SelectInst>(TI) ? 2 : Visited.size();
     assert(SuccessorsCount > 1 &&
            "Cannot unswitch a condition without multiple distinct successors!");
     return (LoopCost - Cost) * (SuccessorsCount - 1);
   };
 
   std::optional<NonTrivialUnswitchCandidate> Best;
   for (auto &Candidate : UnswitchCandidates) {
     Instruction &TI = *Candidate.TI;
if (*Best.Cost >= UnswitchThreshold) {
     LLVM_DEBUG(dbgs() << "Cannot unswitch, lowest cost found: " << *Best.Cost
                       << "\n");
     return false;
   }
 
   if (Best.TI != PartialIVCondBranch)
     PartialIVInfo.InstToDuplicate.clear();
 
-  // If the best candidate is a guard, turn it into a branch.
-  if (isGuard(Best.TI))
-    Best.TI =
-        turnGuardIntoBranch(cast<IntrinsicInst>(Best.TI), L, DT, LI, MSSAU);
+  // If the best candidate is a select or a guard, turn it into a branch.
+  if (auto *SI = dyn_cast<SelectInst>(Best.TI))
+    Best.TI = turnSelectIntoBranch(SI, DT, LI, MSSAU);
+  else if (isGuard(Best.TI))
+    Best.TI = turnGuardIntoBranch(cast<IntrinsicInst>(Best.TI), DT, LI, MSSAU);
 
   LLVM_DEBUG(dbgs() << "  Unswitching non-trivial (cost = " << Best.Cost
                     << ") terminator: " << *Best.TI << "\n");
   unswitchNontrivialInvariants(L, *Best.TI, Best.Invariants, PartialIVInfo, DT,
                                LI, AC, UnswitchCB, SE, MSSAU, DestroyLoopCB);
   return true;
 }
 

llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll

 loop_exit:
   ret i32 0
 ; CHECK:       loop_exit.split:
 ; CHECK-NEXT:    br label %loop_exit
 ;
 ; CHECK:       loop_exit:
 ; CHECK-NEXT:    ret
 }
+
+; Non-trivial loop unswitching of select instruction.
+define i32 @test34(i32 %N, i1 %cond) {
+; CHECK-LABEL: @test34(
+entry:
+  br label %for.cond
+; CHECK-NEXT: entry:
+; CHECK-NEXT:   %cond.fr = freeze i1 %cond
+; CHECK-NEXT:   br i1 %cond.fr, label %entry.split.us, label %entry.split
+
+for.cond:                                         ; preds = %for.body, %entry
+  %res = phi i32 [ 0, %entry ], [ %add, %for.body ]
+  %i = phi i32 [ 0, %entry ], [ %inc, %for.body ]
+  %cmp = icmp slt i32 %i, %N
+  br i1 %cmp, label %for.body, label %for.cond.cleanup
+
+for.body:                                         ; preds = %for.cond
+  %cond1 = select i1 %cond, i32 %i, i32 42
+  %add = add nuw nsw i32 %cond1, %res
+  %inc = add nuw nsw i32 %i, 1
+  br label %for.cond
+
+; CHECK:       entry.split.us:
+; CHECK-NEXT:    br label %for.cond.us
+
+; CHECK:       for.cond.us:
+; CHECK-NEXT:    %[[RES_US_PHI:.*]] = phi i32 [ 0, %entry.split.us ], [ %[[ADD_US:.*]], %[[TAIL_US:.*]] ]
+; CHECK-NEXT:    %[[I_US:.*]] = phi i32 [ 0, %entry.split.us ], [ %[[INC_US:.*]], %[[TAIL_US]] ]
+; CHECK-NEXT:    %[[CMP_US:.*]] = icmp slt i32 %[[I_US]], %N
+; CHECK-NEXT:    br i1 %[[CMP_US]], label %for.body.us, label %for.cond.cleanup.split.us
+
+; CHECK:       for.body.us:
+; CHECK-NEXT:    br label %[[THEN:.*]]
+
+; CHECK:       [[THEN]]:
+; CHECK-NEXT:    br label %[[TAIL_US]]
+
+; CHECK:       [[TAIL_US]]:
+; CHECK-NEXT:    %[[SEL_PHI:.*]] = phi i32 [ %[[I_US]], %[[THEN]] ]
+; CHECK-NEXT:    %[[ADD_US]] = add nuw nsw i32 %[[SEL_PHI]], %[[RES_US_PHI]]
+; CHECK-NEXT:    %[[INC_US]] = add nuw nsw i32 %[[I_US]], 1
+; CHECK-NEXT:    br label %for.cond.us
+
+; CHECK:       for.cond.cleanup.split.us:
+; CHECK-NEXT:    %[[RES_US:.*]] = phi i32 [ %[[RES_US_PHI]], %for.cond.us ]
+; CHECK-NEXT:    br label %for.cond.cleanup
+
+; CHECK:       entry.split:
+; CHECK-NEXT:    br label %for.cond
+
+; CHECK:       for.cond:
+; CHECK-NEXT:    %[[RES_PHI:.*]] = phi i32 [ 0, %entry.split ], [ %[[ADD:.*]], %[[TAIL:.*]] ]
+; CHECK-NEXT:    %[[I:.*]] = phi i32 [ 0, %entry.split ], [ %[[INC:.*]], %[[TAIL]] ]
+; CHECK-NEXT:    %[[CMP:.*]] = icmp slt i32 %[[I]], %N
+; CHECK-NEXT:    br i1 %[[CMP]], label %for.body, label %for.cond.cleanup.split
+
+; CHECK:       for.body:
+; CHECK-NEXT:    br label %[[TAIL]]
+
+; CHECK:       [[TAIL]]:
+; CHECK-NEXT:    %[[ADD]] = add nuw nsw i32 42, %[[RES_PHI]]
+; CHECK-NEXT:    %[[INC]] = add nuw nsw i32 %[[I]], 1
+; CHECK-NEXT:  br label %for.cond
+
+; CHECK:       for.cond.cleanup.split:
+; CHECK-NEXT:    %[[RES:.*]] = phi i32 [ %[[RES_PHI]], %for.cond ]
+; CHECK-NEXT:    br label %for.cond.cleanup
+
+for.cond.cleanup:                                 ; preds = %for.cond
+  ret i32 %res
+; CHECK:       for.cond.cleanup:
+; CHECK-NEXT:    %[[US_PHI:.*]] = phi i32 [ %[[RES]], %for.cond.cleanup.split ], [ %[[RES_US]], %for.cond.cleanup.split.us ]
+; CHECK-NEXT:    ret i32 %[[US_PHI]]
+}