[LSR] Preserve LCSSA when rewriting instruction with PHI user

[dmakogon]

Summary

Fixes https://github.com/llvm/llvm-project/issues/61182.
LoopStrengthReduce may sometimes break LCSSA form when applying a rewrite for an instruction used in a PHI.
It happens if:

1. The PHI is in a loop exit block,
2. The edge from the corresponding exiting block to that exit is critical,
3. The PHI has at least two inputs coming from loop blocks,
4. The rewritten instruction is inserted in the loop.

An example of initial CFG:

LoopBlock  <--  ExitingBlock1   ExitingBlock2  
                    |                 |
                    v                 |
                ExitBlock  <----------+

In such case we split the critical edge:

LoopBlock  <--  ExitingBlock1   ExitingBlock2  
                    |                |
                    v                v
                 CritEdge          split
                    |                |
                    v                |
                ExitBlock  <---------+
           (not exit anymore)

and then replace PHI inputs with the rewritten instruction. However ExitBlock is no longer a loop exit, so LCSSA form is broken.

This patch fixes it by collecting all inserted instructions for PHIs whose parent block is not a loop exit and then forming LCSSA for them.

[dmakogon]

Accidentally changed one test's run line to use new PM, reverted back now.

[mkazantsev]

llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp

5623

I propose to form InsertedNonLCSSAInsts here, because here the order is deterministic.

5668

The order of iteration here is not deterministic. I don't know if it's a source of any real problem, but we may end up doing slightly different things in different order. If we are especially unlucky, it will be different resulting code. See my other comment where I think it's appropriate to form this vector.

[mkazantsev]

llvm/test/Transforms/LoopStrengthReduce/depth-limit-overrun.ll

77

That's weird.

[mkazantsev]

llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp

5540–5547

I'm curious, is it possible that second value here is *not* an instruction? Might be a good NFC change beforehand.

[dmakogon]

llvm/test/Transforms/LoopStrengthReduce/depth-limit-overrun.ll

77

I guess it's ok because we get exactly the same IR as with DEFAULT checks above which are generated by running the pass through new pass manager.

[dmakogon]

llvm/test/Transforms/LoopStrengthReduce/depth-limit-overrun.ll

77

Ok, I found out why this happens. LSR here calculates the exit value of the inner loop IV (%i33) using SCEV, and it is able to do so because its only use is in loop exit block (in LCSSA PHI which is inserted in outer_tail.loopexit block because of my changes). Previously we wouldn't create this LCSSA PHI and the IV's use would be in a inner loop exit's successor.
LSR checks for LCSSA form of the loop explicitly, and if the check fails, we don't even try this exit value calculation.
The corresponding code for that:

if (L->isRecursivelyLCSSAForm(DT, LI) && L->getExitBlock()) {
    ...
    int Rewrites = rewriteLoopExitValues(L, &LI, &TLI, &SE, &TTI, Rewriter, &DT,
                                             UnusedIndVarInLoop, DeadInsts);

[dmakogon]

llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp

5540–5547

Yes, it's possible. LSR uses SCEVExpander here. There are case where it expands the expression to constants.

[dmakogon]

Address comment: collect instructions right where they are inserted.

[mkazantsev]

llvm/test/Transforms/LoopStrengthReduce/depth-limit-overrun.ll

77

So it's an improvement then. Allright, let it be.

[thakis]

This seems to break tests: http://45.33.8.238/linux/103038/step_12.txt

Please take a look and revert for now if it takes a while to fix.

[thakis]

Reverted in efd34ba60f3839b0a68b2e32ff9011b6823bc16f for now.

[dmakogon]

In D146811#4232993, @thakis wrote:

Reverted in efd34ba60f3839b0a68b2e32ff9011b6823bc16f for now.

Thanks for reverting @thakis! Should be fixed in https://github.com/llvm/llvm-project/commit/3d7242f05e6ebae394fe47039abb0568083209a0

[samtebbs]

The mve-float-loops.ll change that was added after the review is a performance regression because of the extra mov instruction. The mov comes from a copy that can't be merged because of "interference" in RegisterCoalescer.cpp.

Considering merging to rGPR with %73 in %106
RHS = %73 [384r,448r:0) 0@384r  weight:0.000000e+00
LHS = %106 [192r,240B:0)[240B,432r:3)[448r,896B:4)[896B,1008r:5)[1312r,1504B:1)[1616r,2096B:2) 0@192r 1@1312r 2@1616r 3@240B-phi 4@448r 5@896B-phi  weight:0.000000e+00
merge %106:4@448r into %73:0@384r --> @384r
interference at %73:0@384r
Interference!

Do you think you'd be able to fix that in this patch or a follow-up?

llvm/lib/Transforms/Scalar/LoopStrengthReduce.cpp

 }
 
 /// Helper for Rewrite. PHI nodes are special because the use of their operands
 /// effectively happens in their predecessor blocks, so the expression may need
 /// to be expanded in multiple places.
 void LSRInstance::RewriteForPHI(
     PHINode *PN, const LSRUse &LU, const LSRFixup &LF, const Formula &F,
     SmallVectorImpl<WeakTrackingVH> &DeadInsts) const {
   DenseMap<BasicBlock *, Value *> Inserted;
+
+  // Inserting instructions in the loop and using them as PHI's input could
+  // break LCSSA in case if PHI's parent block is not a loop exit (i.e. the
+  // corresponding incoming block is not loop exiting). So collect all such
+  // instructions to form LCSSA for them later.
+  SmallVector<Instruction *, 4> InsertedNonLCSSAInsts;
+
   for (unsigned i = 0, e = PN->getNumIncomingValues(); i != e; ++i)
     if (PN->getIncomingValue(i) == LF.OperandValToReplace) {
       bool needUpdateFixups = false;
       BasicBlock *BB = PN->getIncomingBlock(i);
 
       // If this is a critical edge, split the edge so that we do not insert
       // the code on all predecessor/successor paths.  We do this unless this
       // is the canonical backedge for this loop, which complicates post-inc
if (PN->getIncomingValue(i) == LF.OperandValToReplace) {
         Type *OpTy = LF.OperandValToReplace->getType();
         if (FullV->getType() != OpTy)
           FullV =
             CastInst::Create(CastInst::getCastOpcode(FullV, false,
                                                      OpTy, false),
                              FullV, LF.OperandValToReplace->getType(),
                              "tmp", BB->getTerminator());
 
+        // If the incoming block for this value is not in the loop, it means the
+        // current PHI is not in a loop exit, so we must create a LCSSA PHI for
+        // the inserted value.
+        if (auto *I = dyn_cast<Instruction>(FullV))
+          if (L->contains(I) && !L->contains(BB))
+            InsertedNonLCSSAInsts.push_back(I);
+
         PN->setIncomingValue(i, FullV);
         Pair.first->second = FullV;
       }
 
       // If LSR splits critical edge and phi node has other pending
       // fixup operands, we need to update those pending fixups. Otherwise
       // formulae will not be implemented completely and some instructions
       // will not be eliminated.
       if (needUpdateFixups) {
         for (size_t LUIdx = 0, NumUses = Uses.size(); LUIdx != NumUses; ++LUIdx)
           for (LSRFixup &Fixup : Uses[LUIdx].Fixups)
             // If fixup is supposed to rewrite some operand in the phi
if (PN->getIncomingValue(i) == LF.OperandValToReplace) {
                   PHINode *NewPN = cast<PHINode>(I);
                   for (const auto &val : NewPN->incoming_values())
                     if (val == Fixup.OperandValToReplace)
                       Fixup.UserInst = NewPN;
                 }
             }
       }
     }
+
+  IRBuilder<> Builder(L->getHeader()->getContext());
+  formLCSSAForInstructions(InsertedNonLCSSAInsts, DT, LI, &SE, Builder);
 }
 
 /// Emit instructions for the leading candidate expression for this LSRUse (this
 /// is called "expanding"), and update the UserInst to reference the newly
 /// expanded value.
 void LSRInstance::Rewrite(const LSRUse &LU, const LSRFixup &LF,
                           const Formula &F,
                           SmallVectorImpl<WeakTrackingVH> &DeadInsts) const {
   // First, find an insertion point that dominates UserInst. For PHI nodes,
   // find the nearest block which dominates all the relevant uses.
   if (PHINode *PN = dyn_cast<PHINode>(LF.UserInst)) {
     RewriteForPHI(PN, LU, LF, F, DeadInsts);

llvm/test/Transforms/LoopStrengthReduce/2011-10-03-CritEdgeMerge.ll

 ; CHECK-LABEL: @test1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[LSR_IV:%.*]] = phi ptr [ [[SCEVGEP:%.*]], [[LOOP]] ], [ null, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[SCEVGEP]] = getelementptr i8, ptr [[LSR_IV]], i64 1
 ; CHECK-NEXT:    br i1 false, label [[LOOP]], label [[LOOPEXIT:%.*]]
 ; CHECK:       loopexit:
+; CHECK-NEXT:    [[SCEVGEP_LCSSA:%.*]] = phi ptr [ [[SCEVGEP]], [[LOOP]] ]
 ; CHECK-NEXT:    br i1 false, label [[BBA:%.*]], label [[BBB:%.*]]
 ; CHECK:       bbA:
 ; CHECK-NEXT:    switch i32 0, label [[BBA_BB89_CRIT_EDGE:%.*]] [
 ; CHECK-NEXT:    i32 47, label [[BBA_BB89_CRIT_EDGE]]
 ; CHECK-NEXT:    i32 58, label [[BBA_BB89_CRIT_EDGE]]
 ; CHECK-NEXT:    ]
 ; CHECK:       bbA.bb89_crit_edge:
 ; CHECK-NEXT:    br label [[BB89:%.*]]
 ; CHECK:       bbB:
 ; CHECK-NEXT:    switch i8 0, label [[BBB_BB89_CRIT_EDGE:%.*]] [
 ; CHECK-NEXT:    i8 47, label [[BBB_BB89_CRIT_EDGE]]
 ; CHECK-NEXT:    i8 58, label [[BBB_BB89_CRIT_EDGE]]
 ; CHECK-NEXT:    ]
 ; CHECK:       bbB.bb89_crit_edge:
 ; CHECK-NEXT:    br label [[BB89]]
 ; CHECK:       bb89:
-; CHECK-NEXT:    [[TMP75PHI:%.*]] = phi ptr [ [[SCEVGEP]], [[BBA_BB89_CRIT_EDGE]] ], [ [[SCEVGEP]], [[BBB_BB89_CRIT_EDGE]] ]
+; CHECK-NEXT:    [[TMP75PHI:%.*]] = phi ptr [ [[SCEVGEP_LCSSA]], [[BBA_BB89_CRIT_EDGE]] ], [ [[SCEVGEP_LCSSA]], [[BBB_BB89_CRIT_EDGE]] ]
 ; CHECK-NEXT:    br label [[EXIT:%.*]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret ptr [[TMP75PHI]]
 ;
 entry:
   br label %loop
 
 loop:

 ; CHECK-LABEL: @test2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[LSR_IV:%.*]] = phi ptr [ [[SCEVGEP:%.*]], [[LOOP]] ], [ null, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[SCEVGEP]] = getelementptr i8, ptr [[LSR_IV]], i64 1
 ; CHECK-NEXT:    br i1 false, label [[LOOP]], label [[LOOPEXIT:%.*]]
 ; CHECK:       loopexit:
+; CHECK-NEXT:    [[SCEVGEP_LCSSA1:%.*]] = phi ptr [ [[SCEVGEP]], [[LOOP]] ]
+; CHECK-NEXT:    [[SCEVGEP_LCSSA:%.*]] = phi ptr [ [[SCEVGEP]], [[LOOP]] ]
 ; CHECK-NEXT:    br i1 false, label [[BBA:%.*]], label [[BBB:%.*]]
 ; CHECK:       bbA:
 ; CHECK-NEXT:    switch i32 0, label [[BB89:%.*]] [
 ; CHECK-NEXT:    i32 47, label [[BB89]]
 ; CHECK-NEXT:    i32 58, label [[BB89]]
 ; CHECK-NEXT:    ]
 ; CHECK:       bbB:
 ; CHECK-NEXT:    switch i8 0, label [[BBB_EXIT_CRIT_EDGE:%.*]] [
 ; CHECK-NEXT:    i8 47, label [[BBB_EXIT_CRIT_EDGE]]
 ; CHECK-NEXT:    i8 58, label [[BBB_EXIT_CRIT_EDGE]]
 ; CHECK-NEXT:    ]
 ; CHECK:       bbB.exit_crit_edge:
 ; CHECK-NEXT:    br label [[EXIT:%.*]]
 ; CHECK:       bb89:
-; CHECK-NEXT:    [[TMP75PHI:%.*]] = phi ptr [ [[SCEVGEP]], [[BBA]] ], [ [[SCEVGEP]], [[BBA]] ], [ [[SCEVGEP]], [[BBA]] ]
+; CHECK-NEXT:    [[TMP75PHI:%.*]] = phi ptr [ [[SCEVGEP_LCSSA1]], [[BBA]] ], [ [[SCEVGEP_LCSSA1]], [[BBA]] ], [ [[SCEVGEP_LCSSA1]], [[BBA]] ]
 ; CHECK-NEXT:    br label [[EXIT]]
 ; CHECK:       exit:
-; CHECK-NEXT:    [[RESULT:%.*]] = phi ptr [ [[TMP75PHI]], [[BB89]] ], [ [[SCEVGEP]], [[BBB_EXIT_CRIT_EDGE]] ]
+; CHECK-NEXT:    [[RESULT:%.*]] = phi ptr [ [[TMP75PHI]], [[BB89]] ], [ [[SCEVGEP_LCSSA]], [[BBB_EXIT_CRIT_EDGE]] ]
 ; CHECK-NEXT:    ret ptr [[RESULT]]
 ;
 entry:
   br label %loop
 
 loop:
   %rec = phi i32 [ %next, %loop ], [ 0, %entry ]
   %next = add i32 %rec, 1

llvm/test/Transforms/LoopStrengthReduce/AMDGPU/lsr-invalid-ptr-extend.ll

 
 @gVar = external hidden local_unnamed_addr addrspace(3) global [1024 x double], align 16
 
 define amdgpu_kernel void @scaledregtest() local_unnamed_addr {
 ; CHECK-LABEL: @scaledregtest(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       loopexit:
+; CHECK-NEXT:    [[SCEVGEP13_LCSSA:%.*]] = phi ptr [ [[SCEVGEP13:%.*]], [[FOR_BODY]] ]
+; CHECK-NEXT:    [[SCEVGEP11_LCSSA:%.*]] = phi ptr addrspace(5) [ [[SCEVGEP11:%.*]], [[FOR_BODY]] ]
 ; CHECK-NEXT:    br label [[FOR_BODY_1:%.*]]
 ; CHECK:       for.body.1:
-; CHECK-NEXT:    [[LSR_IV5:%.*]] = phi ptr addrspace(5) [ [[SCEVGEP6:%.*]], [[FOR_BODY_1]] ], [ [[SCEVGEP11:%.*]], [[LOOPEXIT:%.*]] ]
-; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[SCEVGEP2:%.*]], [[FOR_BODY_1]] ], [ [[SCEVGEP13:%.*]], [[LOOPEXIT]] ]
+; CHECK-NEXT:    [[LSR_IV5:%.*]] = phi ptr addrspace(5) [ [[SCEVGEP6:%.*]], [[FOR_BODY_1]] ], [ [[SCEVGEP11_LCSSA]], [[LOOPEXIT:%.*]] ]
+; CHECK-NEXT:    [[LSR_IV1:%.*]] = phi ptr [ [[SCEVGEP2:%.*]], [[FOR_BODY_1]] ], [ [[SCEVGEP13_LCSSA]], [[LOOPEXIT]] ]
 ; CHECK-NEXT:    [[TMP0:%.*]] = load ptr, ptr addrspace(5) [[LSR_IV5]], align 8
 ; CHECK-NEXT:    store ptr [[TMP0]], ptr [[LSR_IV1]], align 8
 ; CHECK-NEXT:    [[SCEVGEP2]] = getelementptr i8, ptr [[LSR_IV1]], i64 8
 ; CHECK-NEXT:    [[SCEVGEP6]] = getelementptr i8, ptr addrspace(5) [[LSR_IV5]], i32 8
 ; CHECK-NEXT:    br label [[FOR_BODY_1]]
 ; CHECK:       for.body:
 ; CHECK-NEXT:    [[LSR_IV12:%.*]] = phi ptr [ [[SCEVGEP13]], [[FOR_BODY]] ], [ null, [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[LSR_IV10:%.*]] = phi ptr addrspace(5) [ [[SCEVGEP11]], [[FOR_BODY]] ], [ null, [[ENTRY]] ]

llvm/test/Transforms/LoopStrengthReduce/X86/2011-11-29-postincphi.ll

 ; CHECK-LABEL: sqlite3DropTriggerPtr:
 ; CHECK:       # %bb.0: # %bb
 ; CHECK-NEXT:    pushq %rbx
 ; CHECK-NEXT:    movl $1, %ebx
 ; CHECK-NEXT:    callq check@PLT
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_1: # %bb1
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movq %rbx, %rcx
 ; CHECK-NEXT:    testb %al, %al
-; CHECK-NEXT:    je .LBB0_4
+; CHECK-NEXT:    je .LBB0_3
 ; CHECK-NEXT:  # %bb.2: # %bb4
 ; CHECK-NEXT:    # in Loop: Header=BB0_1 Depth=1
-; CHECK-NEXT:    incq %rbx
+; CHECK-NEXT:    leaq 1(%rcx), %rbx
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    jne .LBB0_1
-; CHECK-NEXT:  # %bb.3: # %bb8split
-; CHECK-NEXT:    decq %rbx
-; CHECK-NEXT:  .LBB0_4: # %bb8
-; CHECK-NEXT:    movq %rbx, %rax
+; CHECK-NEXT:  .LBB0_3: # %bb8
+; CHECK-NEXT:    movq %rcx, %rax
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    retq
 bb:
   %cmp = call zeroext i1 @check()
   br label %bb1
 
 bb1:                                              ; preds = %bb4, %bb
   %t0 = phi i64 [ 0, %bb ], [ %t3, %bb4 ]

llvm/test/Transforms/LoopStrengthReduce/X86/expander-crashes.ll

 ; CHECK:       loop.1.header:
 ; CHECK-NEXT:    [[LSR_IV5:%.*]] = phi i64 [ [[LSR_IV_NEXT6:%.*]], [[LOOP_1_HEADER]] ], [ [[START1]], [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[IV:%.*]] = phi %struct.hoge* [ [[IV_NEXT:%.*]], [[LOOP_1_HEADER]] ], [ [[START]], [[ENTRY]] ]
 ; CHECK-NEXT:    [[IV_NEXT]] = getelementptr inbounds [[STRUCT_HOGE:%.*]], %struct.hoge* [[IV]], i64 1
 ; CHECK-NEXT:    [[LSR_IV_NEXT6]] = add nuw i64 [[LSR_IV5]], 16
 ; CHECK-NEXT:    [[EC:%.*]] = icmp eq %struct.hoge* [[IV_NEXT]], [[END:%.*]]
 ; CHECK-NEXT:    br i1 [[EC]], label [[LOOP_2_PH:%.*]], label [[LOOP_1_HEADER]]
 ; CHECK:       loop.2.ph:
+; CHECK-NEXT:    [[IV_NEXT_LCSSA:%.*]] = phi %struct.hoge* [ [[IV_NEXT]], [[LOOP_1_HEADER]] ]
+; CHECK-NEXT:    [[LSR_IV_NEXT6_LCSSA:%.*]] = phi i64 [ [[LSR_IV_NEXT6]], [[LOOP_1_HEADER]] ]
 ; CHECK-NEXT:    br label [[LOOP_2_HEADER:%.*]]
 ; CHECK:       loop.2.header:
-; CHECK-NEXT:    [[LSR_IV2:%.*]] = phi i64 [ [[LSR_IV_NEXT3:%.*]], [[LOOP_2_LATCH:%.*]] ], [ [[LSR_IV_NEXT6]], [[LOOP_2_PH]] ]
-; CHECK-NEXT:    [[IV2:%.*]] = phi %struct.hoge* [ [[IV2_NEXT:%.*]], [[LOOP_2_LATCH]] ], [ [[IV_NEXT]], [[LOOP_2_PH]] ]
+; CHECK-NEXT:    [[LSR_IV2:%.*]] = phi i64 [ [[LSR_IV_NEXT3:%.*]], [[LOOP_2_LATCH:%.*]] ], [ [[LSR_IV_NEXT6_LCSSA]], [[LOOP_2_PH]] ]
+; CHECK-NEXT:    [[IV2:%.*]] = phi %struct.hoge* [ [[IV2_NEXT:%.*]], [[LOOP_2_LATCH]] ], [ [[IV_NEXT_LCSSA]], [[LOOP_2_PH]] ]
 ; CHECK-NEXT:    [[IV24:%.*]] = bitcast %struct.hoge* [[IV2]] to i32*
 ; CHECK-NEXT:    [[TMP0:%.*]] = add i64 [[LSR_IV2]], 12
 ; CHECK-NEXT:    call void @use.i64(i64 [[TMP0]])
 ; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i32, i32* [[IV24]], i64 2
 ; CHECK-NEXT:    store i32 10, i32* [[SCEVGEP]], align 8
 ; CHECK-NEXT:    [[EC_2:%.*]] = icmp ugt %struct.hoge* [[IV2]], [[PTR_2:%.*]]
 ; CHECK-NEXT:    br i1 [[EC_2]], label [[LOOP_2_EXIT:%.*]], label [[LOOP_2_LATCH]]
 ; CHECK:       loop.2.latch:

llvm/test/Transforms/LoopStrengthReduce/X86/expander-reused-value-insert-point.ll

 ; CHECK-NEXT:    [[P0R_0335:%.*]] = phi ptr [ [[ADD_PTR94:%.*]], [[FOR_BODY37]] ], [ [[START:%.*]], [[FOR_BODY15]] ]
 ; CHECK-NEXT:    [[ADD_PTR94]] = getelementptr inbounds double, ptr [[P0R_0335]], i64 1
 ; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, ptr [[P0R_0335]], i64 [[TMP1]]
 ; CHECK-NEXT:    [[F1I_0:%.*]] = load double, ptr [[SCEVGEP]], align 8
 ; CHECK-NEXT:    call void @use(double [[F1I_0]])
 ; CHECK-NEXT:    [[EC0:%.*]] = icmp eq ptr [[ADD_PTR94]], [[END:%.*]]
 ; CHECK-NEXT:    br i1 [[EC0]], label [[FOR_BODY37]], label [[FOR_END_LOOPEXIT:%.*]]
 ; CHECK:       for.end.loopexit:
+; CHECK-NEXT:    [[ADD_PTR94_LCSSA:%.*]] = phi ptr [ [[ADD_PTR94]], [[FOR_BODY37]] ]
 ; CHECK-NEXT:    br label [[FOR_END:%.*]]
 ; CHECK:       for.end:
-; CHECK-NEXT:    [[P0R_0_LCSSA:%.*]] = phi ptr [ [[ADD_PTR94]], [[FOR_END_LOOPEXIT]] ]
+; CHECK-NEXT:    [[P0R_0_LCSSA:%.*]] = phi ptr [ [[ADD_PTR94_LCSSA]], [[FOR_END_LOOPEXIT]] ]
 ; CHECK-NEXT:    [[EC1:%.*]] = icmp eq ptr [[P0R_0_LCSSA]], [[END]]
 ; CHECK-NEXT:    br i1 [[EC1]], label [[FOR_BODY15]], label [[FOR_INC133:%.*]]
 ; CHECK:       for.inc133:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %idx.ext32 = sext i32 %X to i64
   br label %for.body15

llvm/test/Transforms/LoopStrengthReduce/X86/sibling-loops.ll

 ; CHECK-NEXT:    [[J_2:%.*]] = phi i64 [ [[INC10:%.*]], [[DO_BODY8]] ], [ [[TMP1]], [[DO_BODY8_PREHEADER]] ]
 ; CHECK-NEXT:    tail call void @goo(i64 [[I_2]], i64 [[J_2]])
 ; CHECK-NEXT:    [[INC9]] = add nuw nsw i64 [[I_2]], 1
 ; CHECK-NEXT:    [[INC10]] = add i64 [[J_2]], 1
 ; CHECK-NEXT:    [[T2:%.*]] = load i64, ptr @cond, align 8
 ; CHECK-NEXT:    [[TOBOOL12:%.*]] = icmp eq i64 [[T2]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL12]], label [[DO_BODY14_PREHEADER:%.*]], label [[DO_BODY8]]
 ; CHECK:       do.body14.preheader:
+; CHECK-NEXT:    [[INC10_LCSSA:%.*]] = phi i64 [ [[INC10]], [[DO_BODY8]] ]
 ; CHECK-NEXT:    br label [[DO_BODY14:%.*]]
 ; CHECK:       do.body14:
 ; CHECK-NEXT:    [[I_3:%.*]] = phi i64 [ [[INC15:%.*]], [[DO_BODY14]] ], [ 0, [[DO_BODY14_PREHEADER]] ]
-; CHECK-NEXT:    [[J_3:%.*]] = phi i64 [ [[INC16:%.*]], [[DO_BODY14]] ], [ [[INC10]], [[DO_BODY14_PREHEADER]] ]
+; CHECK-NEXT:    [[J_3:%.*]] = phi i64 [ [[INC16:%.*]], [[DO_BODY14]] ], [ [[INC10_LCSSA]], [[DO_BODY14_PREHEADER]] ]
 ; CHECK-NEXT:    tail call void @goo(i64 [[I_3]], i64 [[J_3]])
 ; CHECK-NEXT:    [[INC15]] = add nuw nsw i64 [[I_3]], 1
 ; CHECK-NEXT:    [[INC16]] = add i64 [[J_3]], 1
 ; CHECK-NEXT:    [[T3:%.*]] = load i64, ptr @cond, align 8
 ; CHECK-NEXT:    [[TOBOOL18:%.*]] = icmp eq i64 [[T3]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL18]], label [[DO_BODY20_PREHEADER:%.*]], label [[DO_BODY14]]
 ; CHECK:       do.body20.preheader:
+; CHECK-NEXT:    [[INC16_LCSSA:%.*]] = phi i64 [ [[INC16]], [[DO_BODY14]] ]
 ; CHECK-NEXT:    br label [[DO_BODY20:%.*]]
 ; CHECK:       do.body20:
 ; CHECK-NEXT:    [[I_4:%.*]] = phi i64 [ [[INC21:%.*]], [[DO_BODY20]] ], [ 0, [[DO_BODY20_PREHEADER]] ]
-; CHECK-NEXT:    [[J_4:%.*]] = phi i64 [ [[INC22:%.*]], [[DO_BODY20]] ], [ [[INC16]], [[DO_BODY20_PREHEADER]] ]
+; CHECK-NEXT:    [[J_4:%.*]] = phi i64 [ [[INC22:%.*]], [[DO_BODY20]] ], [ [[INC16_LCSSA]], [[DO_BODY20_PREHEADER]] ]
 ; CHECK-NEXT:    tail call void @goo(i64 [[I_4]], i64 [[J_4]])
 ; CHECK-NEXT:    [[INC21]] = add nuw nsw i64 [[I_4]], 1
 ; CHECK-NEXT:    [[INC22]] = add i64 [[J_4]], 1
 ; CHECK-NEXT:    [[T4:%.*]] = load i64, ptr @cond, align 8
 ; CHECK-NEXT:    [[TOBOOL24:%.*]] = icmp eq i64 [[T4]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL24]], label [[DO_BODY26_PREHEADER:%.*]], label [[DO_BODY20]]
 ; CHECK:       do.body26.preheader:
+; CHECK-NEXT:    [[INC22_LCSSA:%.*]] = phi i64 [ [[INC22]], [[DO_BODY20]] ]
 ; CHECK-NEXT:    br label [[DO_BODY26:%.*]]
 ; CHECK:       do.body26:
 ; CHECK-NEXT:    [[I_5:%.*]] = phi i64 [ [[INC27:%.*]], [[DO_BODY26]] ], [ 0, [[DO_BODY26_PREHEADER]] ]
-; CHECK-NEXT:    [[J_5:%.*]] = phi i64 [ [[INC28:%.*]], [[DO_BODY26]] ], [ [[INC22]], [[DO_BODY26_PREHEADER]] ]
+; CHECK-NEXT:    [[J_5:%.*]] = phi i64 [ [[INC28:%.*]], [[DO_BODY26]] ], [ [[INC22_LCSSA]], [[DO_BODY26_PREHEADER]] ]
 ; CHECK-NEXT:    tail call void @goo(i64 [[I_5]], i64 [[J_5]])
 ; CHECK-NEXT:    [[INC27]] = add nuw nsw i64 [[I_5]], 1
 ; CHECK-NEXT:    [[INC28]] = add nsw i64 [[J_5]], 1
 ; CHECK-NEXT:    [[T5:%.*]] = load i64, ptr @cond, align 8
 ; CHECK-NEXT:    [[TOBOOL30:%.*]] = icmp eq i64 [[T5]], 0
 ; CHECK-NEXT:    br i1 [[TOBOOL30]], label [[DO_END31:%.*]], label [[DO_BODY26]]
 ; CHECK:       do.end31:
 ; CHECK-NEXT:    ret void

llvm/test/Transforms/LoopStrengthReduce/callbr-critical-edge-splitting.ll

 ; LEGACYPM:       for.cond:
 ; LEGACYPM-NEXT:    callbr void asm sideeffect "", "!i,!i,~{dirflag},~{fpsr},~{flags}"()
 ; LEGACYPM-NEXT:    to label [[ASM_FALLTHROUGH_I_I:%.*]] [label [[COND_TRUE_I:%.*]], label %for.endsplit]
 ; LEGACYPM:       asm.fallthrough.i.i:
 ; LEGACYPM-NEXT:    unreachable
 ; LEGACYPM:       cond.true.i:
 ; LEGACYPM-NEXT:    br label [[DO_BODY_I_I_DO_BODY_I_I_CRIT_EDGE:%.*]]
 ; LEGACYPM:       do.body.i.i.do.body.i.i_crit_edge:
-; LEGACYPM-NEXT:    [[LSR_IV:%.*]] = phi i64 [ [[LSR_IV_NEXT:%.*]], [[DO_BODY_I_I_DO_BODY_I_I_CRIT_EDGE]] ], [ undef, [[COND_TRUE_I]] ]
-; LEGACYPM-NEXT:    [[LSR_IV_NEXT]] = add i64 [[LSR_IV]], 1
 ; LEGACYPM-NEXT:    br i1 true, label [[DO_BODY_I_I_RDRAND_INT_EXIT_I_CRIT_EDGE:%.*]], label [[DO_BODY_I_I_DO_BODY_I_I_CRIT_EDGE]]
 ; LEGACYPM:       do.body.i.i.rdrand_int.exit.i_crit_edge:
 ; LEGACYPM-NEXT:    br i1 true, label [[DO_BODY_I_I_RDRAND_INT_EXIT_I_CRIT_EDGE_FOR_END_CRIT_EDGE:%.*]], label [[FOR_INC:%.*]]
 ; LEGACYPM:       do.body.i.i.rdrand_int.exit.i_crit_edge.for.end_crit_edge:
+; LEGACYPM-NEXT:    [[LSR_IV_NEXT_LCSSA_LCSSA:%.*]] = phi i64 [ undef, [[DO_BODY_I_I_RDRAND_INT_EXIT_I_CRIT_EDGE]] ]
 ; LEGACYPM-NEXT:    br label [[FOR_END:%.*]]
 ; LEGACYPM:       for.inc:
 ; LEGACYPM-NEXT:    br label [[FOR_COND]]
 ; LEGACYPM:       for.endsplit:
 ; LEGACYPM-NEXT:    br label [[FOR_END]]
 ; LEGACYPM:       for.end:
-; LEGACYPM-NEXT:    [[PGOCOUNT_PROMOTED24:%.*]] = phi i64 [ [[LSR_IV_NEXT]], [[DO_BODY_I_I_RDRAND_INT_EXIT_I_CRIT_EDGE_FOR_END_CRIT_EDGE]] ], [ undef, [[FOR_ENDSPLIT:%.*]] ]
+; LEGACYPM-NEXT:    [[PGOCOUNT_PROMOTED24:%.*]] = phi i64 [ [[LSR_IV_NEXT_LCSSA_LCSSA]], [[DO_BODY_I_I_RDRAND_INT_EXIT_I_CRIT_EDGE_FOR_END_CRIT_EDGE]] ], [ undef, [[FOR_ENDSPLIT:%.*]] ]
 ; LEGACYPM-NEXT:    ret i32 undef
 ;
 ; NEWPM-LABEL: @test1(
 ; NEWPM-NEXT:  entry:
 ; NEWPM-NEXT:    br label [[FOR_COND:%.*]]
 ; NEWPM:       for.cond:
 ; NEWPM-NEXT:    callbr void asm sideeffect "", "!i,!i,~{dirflag},~{fpsr},~{flags}"()
 ; NEWPM-NEXT:    to label [[ASM_FALLTHROUGH_I_I:%.*]] [label [[COND_TRUE_I:%.*]], label %for.end]

llvm/test/Transforms/LoopStrengthReduce/depth-limit-overrun.ll

 ; DEFAULT-NEXT:    [[I35]] = sub i32 [[A]], [[PHI7]]
 ; DEFAULT-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[I35]], 9876
 ; DEFAULT-NEXT:    br i1 [[CMP]], label [[EXIT:%.*]], label [[OUTER_LOOP]]
 ; DEFAULT:       exit:
 ; DEFAULT-NEXT:    ret void
 ;
 ; LIMIT-LABEL: @test(
 ; LIMIT-NEXT:  entry:
+; LIMIT-NEXT:    [[TMP0:%.*]] = mul i32 [[C:%.*]], -3
 ; LIMIT-NEXT:    br label [[OUTER_LOOP:%.*]]
 ; LIMIT:       outer_loop:
 ; LIMIT-NEXT:    [[PHI2:%.*]] = phi i32 [ [[A:%.*]], [[ENTRY:%.*]] ], [ 204, [[OUTER_TAIL:%.*]] ]
 ; LIMIT-NEXT:    [[PHI3:%.*]] = phi i32 [ [[A]], [[ENTRY]] ], [ 243, [[OUTER_TAIL]] ]
 ; LIMIT-NEXT:    [[PHI4:%.*]] = phi i32 [ [[B:%.*]], [[ENTRY]] ], [ [[I35:%.*]], [[OUTER_TAIL]] ]
 ; LIMIT-NEXT:    br label [[GUARD:%.*]]
 ; LIMIT:       guard:
-; LIMIT-NEXT:    [[LCMP_MOD:%.*]] = icmp eq i32 [[C:%.*]], 0
+; LIMIT-NEXT:    [[LCMP_MOD:%.*]] = icmp eq i32 [[C]], 0
 ; LIMIT-NEXT:    br i1 [[LCMP_MOD]], label [[OUTER_TAIL]], label [[PREHEADER:%.*]]
 ; LIMIT:       preheader:
 ; LIMIT-NEXT:    [[I15:%.*]] = shl i32 [[B]], 1
-; LIMIT-NEXT:    [[TMP0:%.*]] = mul i32 [[PHI2]], -1
-; LIMIT-NEXT:    [[TMP1:%.*]] = mul i32 [[TMP0]], -1
-; LIMIT-NEXT:    [[TMP2:%.*]] = sub i32 [[PHI4]], [[TMP1]]
-; LIMIT-NEXT:    [[TMP3:%.*]] = add i32 [[B]], [[PHI4]]
-; LIMIT-NEXT:    [[TMP4:%.*]] = sub i32 [[TMP3]], [[TMP1]]
-; LIMIT-NEXT:    [[TMP5:%.*]] = sub i32 14, [[TMP4]]
+; LIMIT-NEXT:    [[TMP1:%.*]] = mul i32 [[PHI2]], -1
+; LIMIT-NEXT:    [[TMP2:%.*]] = mul i32 [[TMP1]], -1
+; LIMIT-NEXT:    [[TMP3:%.*]] = sub i32 [[PHI4]], [[TMP2]]
+; LIMIT-NEXT:    [[TMP4:%.*]] = add i32 [[B]], [[PHI4]]
+; LIMIT-NEXT:    [[TMP5:%.*]] = sub i32 [[TMP4]], [[TMP2]]
+; LIMIT-NEXT:    [[TMP6:%.*]] = sub i32 14, [[TMP5]]
+; LIMIT-NEXT:    [[TMP7:%.*]] = add i32 [[TMP0]], [[PHI2]]
 ; LIMIT-NEXT:    br label [[INNER_LOOP:%.*]]
 ; LIMIT:       inner_loop:
-; LIMIT-NEXT:    [[LSR_IV3:%.*]] = phi i32 [ [[LSR_IV_NEXT4:%.*]], [[INNER_LOOP]] ], [ [[TMP5]], [[PREHEADER]] ]
-; LIMIT-NEXT:    [[LSR_IV1:%.*]] = phi i32 [ [[LSR_IV_NEXT2:%.*]], [[INNER_LOOP]] ], [ [[TMP4]], [[PREHEADER]] ]
-; LIMIT-NEXT:    [[LSR_IV:%.*]] = phi i32 [ [[LSR_IV_NEXT:%.*]], [[INNER_LOOP]] ], [ [[TMP2]], [[PREHEADER]] ]
+; LIMIT-NEXT:    [[LSR_IV3:%.*]] = phi i32 [ [[LSR_IV_NEXT4:%.*]], [[INNER_LOOP]] ], [ [[TMP6]], [[PREHEADER]] ]
+; LIMIT-NEXT:    [[LSR_IV1:%.*]] = phi i32 [ [[LSR_IV_NEXT2:%.*]], [[INNER_LOOP]] ], [ [[TMP5]], [[PREHEADER]] ]
+; LIMIT-NEXT:    [[LSR_IV:%.*]] = phi i32 [ [[LSR_IV_NEXT:%.*]], [[INNER_LOOP]] ], [ [[TMP3]], [[PREHEADER]] ]
 ; LIMIT-NEXT:    [[PHI5:%.*]] = phi i32 [ [[PHI3]], [[PREHEADER]] ], [ [[I30:%.*]], [[INNER_LOOP]] ]
 ; LIMIT-NEXT:    [[PHI6:%.*]] = phi i32 [ [[PHI2]], [[PREHEADER]] ], [ [[I33:%.*]], [[INNER_LOOP]] ]
 ; LIMIT-NEXT:    [[ITER:%.*]] = phi i32 [ [[C]], [[PREHEADER]] ], [ [[ITER_SUB:%.*]], [[INNER_LOOP]] ]
 ; LIMIT-NEXT:    [[I17:%.*]] = sub i32 [[PHI4]], [[PHI6]]
 ; LIMIT-NEXT:    [[I18:%.*]] = sub i32 14, [[PHI5]]
 ; LIMIT-NEXT:    [[I19:%.*]] = mul i32 [[I18]], [[C]]
 ; LIMIT-NEXT:    [[FACTOR_PROL:%.*]] = shl i32 [[PHI5]], 1
-; LIMIT-NEXT:    [[TMP6:%.*]] = add i32 [[LSR_IV1]], [[I19]]
-; LIMIT-NEXT:    [[TMP7:%.*]] = add i32 [[TMP6]], [[FACTOR_PROL]]
-; LIMIT-NEXT:    [[TMP8:%.*]] = shl i32 [[TMP7]], 1
-; LIMIT-NEXT:    [[TMP9:%.*]] = add i32 [[LSR_IV]], [[TMP8]]
-; LIMIT-NEXT:    [[TMP10:%.*]] = sub i32 [[LSR_IV3]], [[I19]]
-; LIMIT-NEXT:    [[TMP11:%.*]] = sub i32 [[TMP10]], [[FACTOR_PROL]]
-; LIMIT-NEXT:    [[TMP12:%.*]] = mul i32 [[C]], [[TMP11]]
-; LIMIT-NEXT:    [[TMP13:%.*]] = add i32 [[LSR_IV1]], [[I19]]
-; LIMIT-NEXT:    [[TMP14:%.*]] = add i32 [[TMP13]], [[FACTOR_PROL]]
-; LIMIT-NEXT:    [[TMP15:%.*]] = shl i32 [[TMP14]], 1
-; LIMIT-NEXT:    [[TMP16:%.*]] = add i32 [[TMP12]], [[TMP15]]
-; LIMIT-NEXT:    [[TMP17:%.*]] = add i32 [[LSR_IV]], [[TMP16]]
-; LIMIT-NEXT:    [[I29:%.*]] = mul i32 [[TMP9]], [[C]]
-; LIMIT-NEXT:    [[FACTOR_2_PROL:%.*]] = shl i32 [[TMP17]], 1
+; LIMIT-NEXT:    [[TMP8:%.*]] = add i32 [[LSR_IV1]], [[I19]]
+; LIMIT-NEXT:    [[TMP9:%.*]] = add i32 [[TMP8]], [[FACTOR_PROL]]
+; LIMIT-NEXT:    [[TMP10:%.*]] = shl i32 [[TMP9]], 1
+; LIMIT-NEXT:    [[TMP11:%.*]] = add i32 [[LSR_IV]], [[TMP10]]
+; LIMIT-NEXT:    [[TMP12:%.*]] = sub i32 [[LSR_IV3]], [[I19]]
+; LIMIT-NEXT:    [[TMP13:%.*]] = sub i32 [[TMP12]], [[FACTOR_PROL]]
+; LIMIT-NEXT:    [[TMP14:%.*]] = mul i32 [[C]], [[TMP13]]
+; LIMIT-NEXT:    [[TMP15:%.*]] = add i32 [[LSR_IV1]], [[I19]]
+; LIMIT-NEXT:    [[TMP16:%.*]] = add i32 [[TMP15]], [[FACTOR_PROL]]
+; LIMIT-NEXT:    [[TMP17:%.*]] = shl i32 [[TMP16]], 1
+; LIMIT-NEXT:    [[TMP18:%.*]] = add i32 [[TMP14]], [[TMP17]]
+; LIMIT-NEXT:    [[TMP19:%.*]] = add i32 [[LSR_IV]], [[TMP18]]
+; LIMIT-NEXT:    [[I29:%.*]] = mul i32 [[TMP11]], [[C]]
+; LIMIT-NEXT:    [[FACTOR_2_PROL:%.*]] = shl i32 [[TMP19]], 1
 ; LIMIT-NEXT:    [[I30]] = add i32 [[I17]], [[FACTOR_2_PROL]]
 ; LIMIT-NEXT:    [[I33]] = add i32 [[PHI6]], -3
 ; LIMIT-NEXT:    [[ITER_SUB]] = add i32 [[ITER]], -1
 ; LIMIT-NEXT:    [[LSR_IV_NEXT]] = add i32 [[LSR_IV]], 3
 ; LIMIT-NEXT:    [[LSR_IV_NEXT2]] = add i32 [[LSR_IV1]], 3
 ; LIMIT-NEXT:    [[LSR_IV_NEXT4]] = add i32 [[LSR_IV3]], -3
 ; LIMIT-NEXT:    [[ITER_CMP:%.*]] = icmp eq i32 [[ITER_SUB]], 0
 ; LIMIT-NEXT:    br i1 [[ITER_CMP]], label [[OUTER_TAIL_LOOPEXIT:%.*]], label [[INNER_LOOP]]
 ; LIMIT:       outer_tail.loopexit:
 ; LIMIT-NEXT:    br label [[OUTER_TAIL]]
 ; LIMIT:       outer_tail:
-; LIMIT-NEXT:    [[PHI7:%.*]] = phi i32 [ [[PHI2]], [[GUARD]] ], [ [[I33]], [[OUTER_TAIL_LOOPEXIT]] ]
+; LIMIT-NEXT:    [[PHI7:%.*]] = phi i32 [ [[PHI2]], [[GUARD]] ], [ [[TMP7]], [[OUTER_TAIL_LOOPEXIT]] ]
 ; LIMIT-NEXT:    [[I35]] = sub i32 [[A]], [[PHI7]]
 ; LIMIT-NEXT:    [[CMP:%.*]] = icmp sgt i32 [[I35]], 9876
 ; LIMIT-NEXT:    br i1 [[CMP]], label [[EXIT:%.*]], label [[OUTER_LOOP]]
 ; LIMIT:       exit:
 ; LIMIT-NEXT:    ret void
 ;
 entry:
   br label %outer_loop

llvm/test/Transforms/LoopStrengthReduce/post-inc-icmpzero.ll

 ; CHECK-NEXT:    [[TMP0:%.*]] = icmp ugt i32 [[I_ADDR_0]], 9
 ; CHECK-NEXT:    [[SCEVGEP8:%.*]] = getelementptr [33 x i16], [33 x i16]* [[LSR_IV7]], i64 0, i64 -1
 ; CHECK-NEXT:    [[TMP1]] = bitcast i16* [[SCEVGEP8]] to [33 x i16]*
 ; CHECK-NEXT:    [[SCEVGEP15:%.*]] = getelementptr [33 x i16], [33 x i16]* [[LSR_IV14]], i64 0, i64 -1
 ; CHECK-NEXT:    [[TMP2]] = bitcast i16* [[SCEVGEP15]] to [33 x i16]*
 ; CHECK-NEXT:    [[LSR_IV_NEXT17]] = add i64 [[LSR_IV16]], 1
 ; CHECK-NEXT:    br i1 [[TMP0]], label [[DO_BODY]], label [[DO_END:%.*]]
 ; CHECK:       do.end:
+; CHECK-NEXT:    [[DOTLCSSA:%.*]] = phi [33 x i16]* [ [[TMP2]], [[DO_BODY]] ]
 ; CHECK-NEXT:    [[XAP_0:%.*]] = inttoptr i64 [[LSR_IV_NEXT17]] to i1*
 ; CHECK-NEXT:    [[CAP_0:%.*]] = ptrtoint i1* [[XAP_0]] to i64
 ; CHECK-NEXT:    [[SUB_PTR_SUB:%.*]] = sub i64 [[SUB_PTR_LHS_CAST]], [[SUB_PTR_RHS_CAST]]
 ; CHECK-NEXT:    [[SUB_PTR_DIV39:%.*]] = lshr exact i64 [[SUB_PTR_SUB]], 1
 ; CHECK-NEXT:    [[CONV11:%.*]] = trunc i64 [[SUB_PTR_DIV39]] to i32
 ; CHECK-NEXT:    [[MLENGTH:%.*]] = getelementptr inbounds [[STRUCT_VECTOR2:%.*]], %struct.Vector2* [[RESULT:%.*]], i64 0, i32 2
 ; CHECK-NEXT:    [[IDX_EXT21:%.*]] = bitcast i64 [[SUB_PTR_DIV39]] to i64
 ; CHECK-NEXT:    [[CMP2740:%.*]] = icmp eq i64 [[IDX_EXT21]], 0
 ; CHECK-NEXT:    br i1 [[CMP2740]], label [[FOR_END:%.*]], label [[FOR_BODY_LR_PH:%.*]]
 ; CHECK:       for.body.lr.ph:
 ; CHECK-NEXT:    [[INST16:%.*]] = load i32, i32* [[MLENGTH]], align 4
 ; CHECK-NEXT:    [[MBEGIN:%.*]] = getelementptr inbounds [[STRUCT_VECTOR2]], %struct.Vector2* [[RESULT]], i64 0, i32 0
 ; CHECK-NEXT:    [[INST14:%.*]] = load i16*, i16** [[MBEGIN]], align 8
 ; CHECK-NEXT:    [[INST48:%.*]] = zext i32 [[INST16]] to i64
 ; CHECK-NEXT:    [[SCEVGEP:%.*]] = getelementptr i16, i16* [[INST14]], i64 [[INST48]]
 ; CHECK-NEXT:    [[SCEVGEP1:%.*]] = bitcast i16* [[SCEVGEP]] to i8*
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
-; CHECK-NEXT:    [[LSR_IV9:%.*]] = phi [33 x i16]* [ [[TMP3:%.*]], [[FOR_BODY]] ], [ [[TMP2]], [[FOR_BODY_LR_PH]] ]
+; CHECK-NEXT:    [[LSR_IV9:%.*]] = phi [33 x i16]* [ [[TMP3:%.*]], [[FOR_BODY]] ], [ [[DOTLCSSA]], [[FOR_BODY_LR_PH]] ]
 ; CHECK-NEXT:    [[LSR_IV:%.*]] = phi i64 [ [[LSR_IV_NEXT:%.*]], [[FOR_BODY]] ], [ 0, [[FOR_BODY_LR_PH]] ]
 ; CHECK-NEXT:    [[LSR_IV911:%.*]] = bitcast [33 x i16]* [[LSR_IV9]] to i16*
 ; CHECK-NEXT:    [[SCEVGEP2:%.*]] = getelementptr i8, i8* [[SCEVGEP1]], i64 [[LSR_IV]]
 ; CHECK-NEXT:    [[SCEVGEP23:%.*]] = bitcast i8* [[SCEVGEP2]] to i16*
 ; CHECK-NEXT:    [[INST29:%.*]] = load i16, i16* [[LSR_IV911]], align 2
 ; CHECK-NEXT:    store i16 [[INST29]], i16* [[SCEVGEP23]], align 2
 ; CHECK-NEXT:    [[LSR_IV_NEXT]] = add i64 [[LSR_IV]], 2
 ; CHECK-NEXT:    [[LSR_IV_NEXT4:%.*]] = inttoptr i64 [[LSR_IV_NEXT]] to i16*

llvm/test/Transforms/LoopStrengthReduce/preserve-lcssa.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 2
 ; RUN: opt -S -loop-reduce %s | FileCheck --check-prefixes=LEGACYPM %s
 ; RUN: opt -S -passes=loop-reduce %s | FileCheck --check-prefixes=NEWPM %s
 
-; REQUIRES: asserts
-; XFAIL: *
-
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define void @function_0(i32 %val_i32_8, i32 %val_i32_9) {
 ; LEGACYPM-LABEL: define void @function_0
 ; LEGACYPM-SAME: (i32 [[VAL_I32_8:%.*]], i32 [[VAL_I32_9:%.*]]) {
 ; LEGACYPM-NEXT:    [[VAL_I1_22:%.*]] = trunc i8 -66 to i1
 ; LEGACYPM-NEXT:    br i1 [[VAL_I1_22]], label [[BB_2_PREHEADER:%.*]], label [[BB_2_PREHEADER]]
 ; LEGACYPM:       bb_2.preheader:
 ; LEGACYPM-NEXT:    br label [[BB_2:%.*]]
 ; LEGACYPM:       bb_2:
 ; LEGACYPM-NEXT:    br label [[PRHDR_LOOP_3:%.*]]
 ; LEGACYPM:       prhdr_loop_3:
 ; LEGACYPM-NEXT:    br label [[LOOP_4:%.*]]
 ; LEGACYPM:       loop_4:
 ; LEGACYPM-NEXT:    [[LSR_IV:%.*]] = phi i32 [ [[LSR_IV_NEXT:%.*]], [[BE_6:%.*]] ], [ 7851, [[PRHDR_LOOP_3]] ]
 ; LEGACYPM-NEXT:    br i1 [[VAL_I1_22]], label [[BE_6]], label [[LOOP_EXIT_7SPLIT:%.*]]
 ; LEGACYPM:       bb_5:
 ; LEGACYPM-NEXT:    [[VAL_I32_40:%.*]] = mul i32 [[VAL_I32_9]], [[VAL_I32_24_LCSSA:%.*]]
 ; LEGACYPM-NEXT:    br label [[BB_2]]
 ; LEGACYPM:       be_6:
 ; LEGACYPM-NEXT:    [[LSR_IV_NEXT]] = add i32 [[LSR_IV]], 1
 ; LEGACYPM-NEXT:    br i1 [[VAL_I1_22]], label [[LOOP_4]], label [[BE_6_LOOP_EXIT_7_CRIT_EDGE:%.*]]
 ; LEGACYPM:       loop_exit_7split:
+; LEGACYPM-NEXT:    [[LSR_IV_LCSSA:%.*]] = phi i32 [ [[LSR_IV]], [[LOOP_4]] ]
 ; LEGACYPM-NEXT:    br label [[LOOP_EXIT_7:%.*]]
 ; LEGACYPM:       be_6.loop_exit_7_crit_edge:
+; LEGACYPM-NEXT:    [[LSR_IV_LCSSA1:%.*]] = phi i32 [ [[LSR_IV]], [[BE_6]] ]
 ; LEGACYPM-NEXT:    br label [[LOOP_EXIT_7]]
 ; LEGACYPM:       loop_exit_7:
-; LEGACYPM-NEXT:    [[VAL_I32_24_LCSSA]] = phi i32 [ [[LSR_IV]], [[BE_6_LOOP_EXIT_7_CRIT_EDGE]] ], [ [[LSR_IV]], [[LOOP_EXIT_7SPLIT]] ]
+; LEGACYPM-NEXT:    [[VAL_I32_24_LCSSA]] = phi i32 [ [[LSR_IV_LCSSA1]], [[BE_6_LOOP_EXIT_7_CRIT_EDGE]] ], [ [[LSR_IV_LCSSA]], [[LOOP_EXIT_7SPLIT]] ]
 ; LEGACYPM-NEXT:    br label [[BB_5:%.*]]
 ;
+; NEWPM-LABEL: define void @function_0
+; NEWPM-SAME: (i32 [[VAL_I32_8:%.*]], i32 [[VAL_I32_9:%.*]]) {
+; NEWPM-NEXT:    [[VAL_I1_22:%.*]] = trunc i8 -66 to i1
+; NEWPM-NEXT:    br i1 [[VAL_I1_22]], label [[BB_2_PREHEADER:%.*]], label [[BB_2_PREHEADER]]
+; NEWPM:       bb_2.preheader:
+; NEWPM-NEXT:    br label [[BB_2:%.*]]
+; NEWPM:       bb_2:
+; NEWPM-NEXT:    br label [[PRHDR_LOOP_3:%.*]]
+; NEWPM:       prhdr_loop_3:
+; NEWPM-NEXT:    br label [[LOOP_4:%.*]]
+; NEWPM:       loop_4:
+; NEWPM-NEXT:    [[LSR_IV:%.*]] = phi i32 [ [[LSR_IV_NEXT:%.*]], [[BE_6:%.*]] ], [ 7851, [[PRHDR_LOOP_3]] ]
+; NEWPM-NEXT:    br i1 [[VAL_I1_22]], label [[BE_6]], label [[LOOP_EXIT_7SPLIT:%.*]]
+; NEWPM:       bb_5:
+; NEWPM-NEXT:    [[VAL_I32_40:%.*]] = mul i32 [[VAL_I32_9]], [[VAL_I32_24_LCSSA:%.*]]
+; NEWPM-NEXT:    br label [[BB_2]]
+; NEWPM:       be_6:
+; NEWPM-NEXT:    [[LSR_IV_NEXT]] = add i32 [[LSR_IV]], 1
+; NEWPM-NEXT:    br i1 [[VAL_I1_22]], label [[LOOP_4]], label [[BE_6_LOOP_EXIT_7_CRIT_EDGE:%.*]]
+; NEWPM:       loop_exit_7split:
+; NEWPM-NEXT:    [[LSR_IV_LCSSA:%.*]] = phi i32 [ [[LSR_IV]], [[LOOP_4]] ]
+; NEWPM-NEXT:    br label [[LOOP_EXIT_7:%.*]]
+; NEWPM:       be_6.loop_exit_7_crit_edge:
+; NEWPM-NEXT:    [[LSR_IV_LCSSA1:%.*]] = phi i32 [ [[LSR_IV]], [[BE_6]] ]
+; NEWPM-NEXT:    br label [[LOOP_EXIT_7]]
+; NEWPM:       loop_exit_7:
+; NEWPM-NEXT:    [[VAL_I32_24_LCSSA]] = phi i32 [ [[LSR_IV_LCSSA1]], [[BE_6_LOOP_EXIT_7_CRIT_EDGE]] ], [ [[LSR_IV_LCSSA]], [[LOOP_EXIT_7SPLIT]] ]
+; NEWPM-NEXT:    br label [[BB_5:%.*]]
+;
   %val_i1_22 = trunc i8 -66 to i1
   br i1 %val_i1_22, label %bb_2, label %bb_2
 
 bb_2:                                             ; preds = %bb_5, %entry_1, %entry_1
   br label %prhdr_loop_3
 
 prhdr_loop_3:                                     ; preds = %bb_2
   br label %loop_4

llvm/test/Transforms/LoopStrengthReduce/uglygep.ll

 ; CHECK-NEXT:    br label [[BB6SPLIT:%.*]]
 ; CHECK:       bb6splitsplit:
 ; CHECK-NEXT:    br label [[BB6SPLIT]]
 ; CHECK:       bb6split:
 ; CHECK-NEXT:    [[P8_PH:%.*]] = phi i32 [ [[TMP0]], [[BB5_BB6SPLIT_CRIT_EDGE]] ], [ undef, [[BB6SPLITSPLIT]] ]
 ; CHECK-NEXT:    [[P9_PH:%.*]] = phi i32 [ undef, [[BB5_BB6SPLIT_CRIT_EDGE]] ], [ [[I1]], [[BB6SPLITSPLIT]] ]
 ; CHECK-NEXT:    br label [[BB6:%.*]]
 ; CHECK:       loop1.bb6_crit_edge:
+; CHECK-NEXT:    [[I1_LCSSA:%.*]] = phi i32 [ [[I1]], [[LOOP1]] ]
 ; CHECK-NEXT:    br label [[BB6]]
 ; CHECK:       bb6:
 ; CHECK-NEXT:    [[P8:%.*]] = phi i32 [ undef, [[LOOP1_BB6_CRIT_EDGE]] ], [ [[P8_PH]], [[BB6SPLIT]] ]
-; CHECK-NEXT:    [[P9:%.*]] = phi i32 [ [[I1]], [[LOOP1_BB6_CRIT_EDGE]] ], [ [[P9_PH]], [[BB6SPLIT]] ]
+; CHECK-NEXT:    [[P9:%.*]] = phi i32 [ [[I1_LCSSA]], [[LOOP1_BB6_CRIT_EDGE]] ], [ [[P9_PH]], [[BB6SPLIT]] ]
 ; CHECK-NEXT:    unreachable
 ;
 bb:
   br label %loop0
 
 loop0:                                            ; preds = %loop0, %bb
   %i0 = phi i32 [ %i0.next, %loop0 ], [ 0, %bb ]  ; <i32> [#uses=2]
   %i0.next = add i32 %i0, 1                       ; <i32> [#uses=1]