[PassManager] Run Induction Variable Simplification pass *after* Recognize loop idioms pass, not before

[lebedev.ri]

Summary

Currently, -indvars runs first, and then immediately after -loop-idiom does.
I'm not really sure if -loop-idiom requires -indvars to run beforehand,
but i'm *very* sure that -indvars requires -loop-idiom to run afterwards,
as it can be seen in the phase-ordering test.

LoopIdiom runs on two types of loops: countable ones, and uncountable ones.
For uncountable ones, IndVars obviously didn't make any change to them,
since they are uncountable, so for them the order should be irrelevant.
For countable ones, well, they should have been countable before IndVars
for IndVars to make any change to them, and since SCEV is used on them,
it shouldn't matter if IndVars have already canonicalized them.
So i don't really see why we'd want the current ordering.

While this is quite likely beneficial in-the-wild already,
it's a required part for the full motivational pattern
behind left-shift-until-bittest loop idiom (D91038).

[lebedev.ri]

Fix one more pipeline structure test.

[lebedev.ri]

(adding some more people potentially-relevant people)

TBN i'm personally not *very* sure whether or not loop-idiom benefits from being run after indvars.
If it does, we either need to run indvars twice i guess?

[dmgreen]

Do we have phase ordering tests for various memset/memcpy loops?

I have certainly seen places where we have managed to recognize a loop as a memcpy/memset, but not remove the remaining now empty loop. Here is some more fuel for your fire if you need it: https://godbolt.org/z/eW4rsY

[lebedev.ri]

In D91800#2408151, @dmgreen wrote:

Do we have phase ordering tests for various memset/memcpy loops?

We clearly do not.

I have certainly seen places where we have managed to recognize a loop as a memcpy/memset, but not remove the remaining now empty loop. Here is some more fuel for your fire if you need it: https://godbolt.org/z/eW4rsY

[mkazantsev]

The change looks very reasonable, though I don't know what motivation was behind the current ordering. There might have been some (arcane) reasons behind it. Fine by me, but please have someone else to take a look.

[lebedev.ri]

In D91800#2408151, @dmgreen wrote:

I have certainly seen places where we have managed to recognize a loop as a memcpy/memset, but not remove the remaining now empty loop. Here is some more fuel for your fire if you need it: https://godbolt.org/z/eW4rsY

Thank you for the yet another motivational example, test added!

In D91800#2415347, @mkazantsev wrote:

The change looks very reasonable, though I don't know what motivation was behind the current ordering. There might have been some (arcane) reasons behind it. Fine by me, but please have someone else to take a look.

Yep. LoopIdiom runs on two types of loops: countable ones, and uncountable ones.
For uncountable ones, IndVars obviously didn't make any change to them, since they are uncountable, so for them the order should be irrelevant.
For countable ones, well, they should have been countable before IndVars for IndVars to make any change to them,
and since SCEV is used on them, it shouldn't matter if IndVars have already canonicalized them.
So i don't really see why we'd want the current ordering.

With that, does anyone feel confident accepting? :)

[dmgreen]

I ran some more benchmarks, and they look good. Lets give this a try and if someone finds an issue we can always adjust it.

LGTM

[lebedev.ri]

In D91800#2416416, @dmgreen wrote:

I ran some more benchmarks, and they look good. Lets give this a try and if someone finds an issue we can always adjust it.

LGTM

Thank you.
I'm going to land this now; should this regress some things, we can indeed revert this.

llvm/lib/Passes/PassBuilder.cpp

PassBuilder::buildO1FunctionSimplificationPipeline(OptimizationLevel Level,
 
   LPM1.addPass(LoopRotatePass(/* Disable header duplication */ true));
   // TODO: Investigate promotion cap for O1.
   LPM1.addPass(LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap));
   LPM1.addPass(SimpleLoopUnswitchPass());
 
   if (EnableLoopFlatten)
     FPM.addPass(LoopFlattenPass());
-  LPM2.addPass(IndVarSimplifyPass());
   LPM2.addPass(LoopIdiomRecognizePass());
+  LPM2.addPass(IndVarSimplifyPass());
 
   for (auto &C : LateLoopOptimizationsEPCallbacks)
     C(LPM2, Level);
 
   LPM2.addPass(LoopDeletionPass());
   // Do not enable unrolling in PreLinkThinLTO phase during sample PGO
   // because it changes IR to makes profile annotation in back compile
   // inaccurate. The normal unroller doesn't pay attention to forced full unroll
PassBuilder::buildFunctionSimplificationPipeline(OptimizationLevel Level,
   LPM1.addPass(LoopInstSimplifyPass());
   LPM1.addPass(LoopSimplifyCFGPass());
 
   // Disable header duplication in loop rotation at -Oz.
   LPM1.addPass(LoopRotatePass(Level != OptimizationLevel::Oz));
   // TODO: Investigate promotion cap for O1.
   LPM1.addPass(LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap));
   LPM1.addPass(SimpleLoopUnswitchPass());
-  LPM2.addPass(IndVarSimplifyPass());
   LPM2.addPass(LoopIdiomRecognizePass());
+  LPM2.addPass(IndVarSimplifyPass());
 
   for (auto &C : LateLoopOptimizationsEPCallbacks)
     C(LPM2, Level);
 
   LPM2.addPass(LoopDeletionPass());
   // Do not enable unrolling in PreLinkThinLTO phase during sample PGO
   // because it changes IR to makes profile annotation in back compile
   // inaccurate. The normal unroller doesn't pay attention to forced full unroll
PassBuilder::buildFunctionSimplificationPipeline(OptimizationLevel Level,
   // this once as it is immutable.
   FPM.addPass(
       RequireAnalysisPass<OptimizationRemarkEmitterAnalysis, Function>());
   FPM.addPass(createFunctionToLoopPassAdaptor(
       std::move(LPM1), EnableMSSALoopDependency, /*UseBlockFrequencyInfo=*/true,
       DebugLogging));
   FPM.addPass(SimplifyCFGPass());
   FPM.addPass(InstCombinePass());
-  // The loop passes in LPM2 (IndVarSimplifyPass, LoopIdiomRecognizePass,
+  // The loop passes in LPM2 (LoopIdiomRecognizePass, IndVarSimplifyPass,
   // LoopDeletionPass and LoopFullUnrollPass) do not preserve MemorySSA.
   // *All* loop passes must preserve it, in order to be able to use it.
   FPM.addPass(createFunctionToLoopPassAdaptor(
       std::move(LPM2), /*UseMemorySSA=*/false, /*UseBlockFrequencyInfo=*/false,
       DebugLogging));
 
   // Delete small array after loop unroll.
   FPM.addPass(SROA());

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

void PassManagerBuilder::addFunctionSimplificationPasses(
   // need for this.
   MPM.add(createCFGSimplificationPass());
   MPM.add(createInstructionCombiningPass());
   // We resume loop passes creating a second loop pipeline here.
   if (EnableLoopFlatten) {
     MPM.add(createLoopFlattenPass()); // Flatten loops
     MPM.add(createLoopSimplifyCFGPass());
   }
-  MPM.add(createIndVarSimplifyPass());        // Canonicalize indvars
   MPM.add(createLoopIdiomPass());             // Recognize idioms like memset.
+  MPM.add(createIndVarSimplifyPass());        // Canonicalize indvars
   addExtensionsToPM(EP_LateLoopOptimizations, MPM);
   MPM.add(createLoopDeletionPass());          // Delete dead loops
 
   if (EnableLoopInterchange)
     MPM.add(createLoopInterchangePass()); // Interchange loops
 
   // Unroll small loops
   MPM.add(createSimpleLoopUnrollPass(OptLevel, DisableUnrollLoops,

llvm/test/CodeGen/AMDGPU/opt-pipeline.ll

 ; GCN-O1-NEXT:       Lazy Block Frequency Analysis
 ; GCN-O1-NEXT:       Optimization Remark Emitter
 ; GCN-O1-NEXT:       Combine redundant instructions
 ; GCN-O1-NEXT:       Canonicalize natural loops
 ; GCN-O1-NEXT:       LCSSA Verifier
 ; GCN-O1-NEXT:       Loop-Closed SSA Form Pass
 ; GCN-O1-NEXT:       Scalar Evolution Analysis
 ; GCN-O1-NEXT:       Loop Pass Manager
-; GCN-O1-NEXT:         Induction Variable Simplification
 ; GCN-O1-NEXT:         Recognize loop idioms
+; GCN-O1-NEXT:         Induction Variable Simplification
 ; GCN-O1-NEXT:         Delete dead loops
 ; GCN-O1-NEXT:         Unroll loops
 ; GCN-O1-NEXT:       SROA
 ; GCN-O1-NEXT:       Phi Values Analysis
 ; GCN-O1-NEXT:       Function Alias Analysis Results
 ; GCN-O1-NEXT:       Memory Dependence Analysis
 ; GCN-O1-NEXT:       MemCpy Optimization
 ; GCN-O1-NEXT:       Sparse Conditional Constant Propagation

llvm/test/Other/new-pm-defaults.ll

 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished llvm::Function pass manager run.
 ; CHECK-O-NEXT: Starting Loop pass manager run.
-; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopIdiomRecognizePass
+; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-EP-LOOP-LATE-NEXT: Running pass: NoOpLoopPass
 ; CHECK-O-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O-NEXT: Running pass: LoopFullUnrollPass
 ; CHECK-EP-LOOP-END-NEXT: Running pass: NoOpLoopPass
 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SROA on foo
 ; CHECK-O23SZ-NEXT: Running pass: MergedLoadStoreMotionPass
 ; CHECK-O23SZ-NEXT: Running pass: GVN

llvm/test/Other/new-pm-thinlto-defaults.ll

 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Starting llvm::Function pass manager run
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished llvm::Function pass manager run
 ; CHECK-O-NEXT: Starting Loop pass manager run.
-; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopIdiomRecognizePass
+; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O-NEXT: Running pass: LoopFullUnrollPass
 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SROA on foo
 ; CHECK-Os-NEXT: Running pass: MergedLoadStoreMotionPass
 ; CHECK-Os-NEXT: Running pass: GVN
 ; CHECK-Os-NEXT: Running analysis: MemoryDependenceAnalysis
 ; CHECK-Os-NEXT: Running analysis: PhiValuesAnalysis

llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll

 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Starting {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Starting Loop pass manager run.
-; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopIdiomRecognizePass
+; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O-NEXT: Running pass: LoopFullUnrollPass
 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SROA on foo
 ; CHECK-Os-NEXT: Running pass: MergedLoadStoreMotionPass
 ; CHECK-Os-NEXT: Running pass: GVN
 ; CHECK-Os-NEXT: Running analysis: MemoryDependenceAnalysis
 ; CHECK-Os-NEXT: Running analysis: PhiValuesAnalysis

llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll

 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Starting {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Starting Loop pass manager run.
-; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopIdiomRecognizePass
+; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O-NEXT: Running pass: LoopFullUnrollPass
 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SROA on foo
 ; CHECK-Os-NEXT: Running pass: MergedLoadStoreMotionPass
 ; CHECK-Os-NEXT: Running pass: GVN
 ; CHECK-Os-NEXT: Running analysis: MemoryDependenceAnalysis
 ; CHECK-Os-NEXT: Running analysis: PhiValuesAnalysis

llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll

 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Starting {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Starting Loop pass manager run.
-; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopIdiomRecognizePass
+; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O-NEXT: Running pass: LoopFullUnrollPass
 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SROA on foo
 ; CHECK-Os-NEXT: Running pass: MergedLoadStoreMotionPass
 ; CHECK-Os-NEXT: Running pass: GVN
 ; CHECK-Os-NEXT: Running analysis: MemoryDependenceAnalysis
 ; CHECK-Os-NEXT: Running analysis: PhiValuesAnalysis

llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll

 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Starting {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O-NEXT: Running pass: LCSSAPass
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Starting Loop pass manager run.
-; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopIdiomRecognizePass
+; CHECK-O-NEXT: Running pass: IndVarSimplifyPass
 ; CHECK-O-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O-NEXT: Finished Loop pass manager run.
 ; CHECK-O-NEXT: Running pass: SROA on foo
 ; CHECK-Os-NEXT: Running pass: MergedLoadStoreMotionPass
 ; CHECK-Os-NEXT: Running pass: GVN
 ; CHECK-Os-NEXT: Running analysis: MemoryDependenceAnalysis
 ; CHECK-Os-NEXT: Running analysis: PhiValuesAnalysis
 ; CHECK-Oz-NEXT: Running pass: MergedLoadStoreMotionPass

llvm/test/Other/opt-O2-pipeline.ll

 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:         Canonicalize natural loops
 ; CHECK-NEXT:         LCSSA Verifier
 ; CHECK-NEXT:         Loop-Closed SSA Form Pass
 ; CHECK-NEXT:         Scalar Evolution Analysis
 ; CHECK-NEXT:         Loop Pass Manager
-; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Recognize loop idioms
+; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Delete dead loops
 ; CHECK-NEXT:           Unroll loops
 ; CHECK-NEXT:         SROA
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         MergedLoadStoreMotion
 ; CHECK-NEXT:         Phi Values Analysis
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Memory Dependence Analysis

llvm/test/Other/opt-O3-pipeline-enable-matrix.ll

 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:         Canonicalize natural loops
 ; CHECK-NEXT:         LCSSA Verifier
 ; CHECK-NEXT:         Loop-Closed SSA Form Pass
 ; CHECK-NEXT:         Scalar Evolution Analysis
 ; CHECK-NEXT:         Loop Pass Manager
-; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Recognize loop idioms
+; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Delete dead loops
 ; CHECK-NEXT:           Unroll loops
 ; CHECK-NEXT:         SROA
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         MergedLoadStoreMotion
 ; CHECK-NEXT:         Phi Values Analysis
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Memory Dependence Analysis

llvm/test/Other/opt-O3-pipeline.ll

 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:         Canonicalize natural loops
 ; CHECK-NEXT:         LCSSA Verifier
 ; CHECK-NEXT:         Loop-Closed SSA Form Pass
 ; CHECK-NEXT:         Scalar Evolution Analysis
 ; CHECK-NEXT:         Loop Pass Manager
-; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Recognize loop idioms
+; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Delete dead loops
 ; CHECK-NEXT:           Unroll loops
 ; CHECK-NEXT:         SROA
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         MergedLoadStoreMotion
 ; CHECK-NEXT:         Phi Values Analysis
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Memory Dependence Analysis

llvm/test/Other/opt-Os-pipeline.ll

 ; CHECK-NEXT:         Lazy Block Frequency Analysis
 ; CHECK-NEXT:         Optimization Remark Emitter
 ; CHECK-NEXT:         Combine redundant instructions
 ; CHECK-NEXT:         Canonicalize natural loops
 ; CHECK-NEXT:         LCSSA Verifier
 ; CHECK-NEXT:         Loop-Closed SSA Form Pass
 ; CHECK-NEXT:         Scalar Evolution Analysis
 ; CHECK-NEXT:         Loop Pass Manager
-; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Recognize loop idioms
+; CHECK-NEXT:           Induction Variable Simplification
 ; CHECK-NEXT:           Delete dead loops
 ; CHECK-NEXT:           Unroll loops
 ; CHECK-NEXT:         SROA
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         MergedLoadStoreMotion
 ; CHECK-NEXT:         Phi Values Analysis
 ; CHECK-NEXT:         Function Alias Analysis Results
 ; CHECK-NEXT:         Memory Dependence Analysis

llvm/test/Transforms/PhaseOrdering/ARM/arm_fill_q7.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -O3 -S                                        | FileCheck -check-prefixes=ALL,OLDPM %s
 ; RUN: opt < %s -passes='default<O3>' -aa-pipeline=default -S | FileCheck -check-prefixes=ALL,NEWPM %s
 
 target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
 target triple = "thumbv6m-none-none-eabi"
 
 ; Not only should we be able to to form memsets here, the original loops
 ; should be deleted, too.
 
 ; Function Attrs: nounwind
 define dso_local void @arm_fill_q7(i8 signext %value, i8* %pDst, i32 %blockSize) #0 {
 ; OLDPM-LABEL: @arm_fill_q7(
 ; OLDPM-NEXT:  entry:
-; OLDPM-NEXT:    [[SHR:%.*]] = lshr i32 [[BLOCKSIZE:%.*]], 2
-; OLDPM-NEXT:    [[CMP_NOT20:%.*]] = icmp eq i32 [[SHR]], 0
+; OLDPM-NEXT:    [[CMP_NOT20:%.*]] = icmp ult i32 [[BLOCKSIZE:%.*]], 4
 ; OLDPM-NEXT:    br i1 [[CMP_NOT20]], label [[WHILE_END:%.*]], label [[WHILE_BODY_PREHEADER:%.*]]
 ; OLDPM:       while.body.preheader:
 ; OLDPM-NEXT:    [[TMP0:%.*]] = and i32 [[BLOCKSIZE]], -4
 ; OLDPM-NEXT:    call void @llvm.memset.p0i8.i32(i8* align 1 [[PDST:%.*]], i8 [[VALUE:%.*]], i32 [[TMP0]], i1 false)
-; OLDPM-NEXT:    br label [[WHILE_BODY:%.*]]
-; OLDPM:       while.body:
-; OLDPM-NEXT:    [[BLKCNT_022:%.*]] = phi i32 [ [[DEC:%.*]], [[WHILE_BODY]] ], [ [[SHR]], [[WHILE_BODY_PREHEADER]] ]
-; OLDPM-NEXT:    [[PDST_ADDR_021:%.*]] = phi i8* [ [[ADD_PTR_I:%.*]], [[WHILE_BODY]] ], [ [[PDST]], [[WHILE_BODY_PREHEADER]] ]
-; OLDPM-NEXT:    [[ADD_PTR_I]] = getelementptr inbounds i8, i8* [[PDST_ADDR_021]], i32 4
-; OLDPM-NEXT:    [[DEC]] = add nsw i32 [[BLKCNT_022]], -1
-; OLDPM-NEXT:    [[CMP_NOT:%.*]] = icmp eq i32 [[DEC]], 0
-; OLDPM-NEXT:    br i1 [[CMP_NOT]], label [[WHILE_END]], label [[WHILE_BODY]], [[LOOP3:!llvm.loop !.*]]
+; OLDPM-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, i8* [[PDST]], i32 [[TMP0]]
+; OLDPM-NEXT:    br label [[WHILE_END]]
 ; OLDPM:       while.end:
-; OLDPM-NEXT:    [[PDST_ADDR_0_LCSSA:%.*]] = phi i8* [ [[PDST]], [[ENTRY:%.*]] ], [ [[ADD_PTR_I]], [[WHILE_BODY]] ]
+; OLDPM-NEXT:    [[PDST_ADDR_0_LCSSA:%.*]] = phi i8* [ [[PDST]], [[ENTRY:%.*]] ], [ [[SCEVGEP]], [[WHILE_BODY_PREHEADER]] ]
 ; OLDPM-NEXT:    [[REM:%.*]] = and i32 [[BLOCKSIZE]], 3
 ; OLDPM-NEXT:    [[CMP14_NOT17:%.*]] = icmp eq i32 [[REM]], 0
 ; OLDPM-NEXT:    br i1 [[CMP14_NOT17]], label [[WHILE_END18:%.*]], label [[WHILE_BODY16_PREHEADER:%.*]]
 ; OLDPM:       while.body16.preheader:
 ; OLDPM-NEXT:    call void @llvm.memset.p0i8.i32(i8* align 1 [[PDST_ADDR_0_LCSSA]], i8 [[VALUE]], i32 [[REM]], i1 false)
 ; OLDPM-NEXT:    br label [[WHILE_END18]]
 ; OLDPM:       while.end18:
 ; OLDPM-NEXT:    ret void
 ;
 ; NEWPM-LABEL: @arm_fill_q7(
 ; NEWPM-NEXT:  entry:
-; NEWPM-NEXT:    [[SHR:%.*]] = lshr i32 [[BLOCKSIZE:%.*]], 2
-; NEWPM-NEXT:    [[CMP_NOT17:%.*]] = icmp eq i32 [[SHR]], 0
+; NEWPM-NEXT:    [[CMP_NOT17:%.*]] = icmp ult i32 [[BLOCKSIZE:%.*]], 4
 ; NEWPM-NEXT:    br i1 [[CMP_NOT17]], label [[WHILE_END:%.*]], label [[WHILE_BODY_PREHEADER:%.*]]
 ; NEWPM:       while.body.preheader:
 ; NEWPM-NEXT:    [[TMP0:%.*]] = and i32 [[BLOCKSIZE]], -4
 ; NEWPM-NEXT:    call void @llvm.memset.p0i8.i32(i8* align 1 [[PDST:%.*]], i8 [[VALUE:%.*]], i32 [[TMP0]], i1 false)
-; NEWPM-NEXT:    br label [[WHILE_BODY:%.*]]
-; NEWPM:       while.body:
-; NEWPM-NEXT:    [[BLKCNT_019:%.*]] = phi i32 [ [[DEC:%.*]], [[WHILE_BODY]] ], [ [[SHR]], [[WHILE_BODY_PREHEADER]] ]
-; NEWPM-NEXT:    [[PDST_ADDR_018:%.*]] = phi i8* [ [[ADD_PTR_I:%.*]], [[WHILE_BODY]] ], [ [[PDST]], [[WHILE_BODY_PREHEADER]] ]
-; NEWPM-NEXT:    [[ADD_PTR_I]] = getelementptr inbounds i8, i8* [[PDST_ADDR_018]], i32 4
-; NEWPM-NEXT:    [[DEC]] = add nsw i32 [[BLKCNT_019]], -1
-; NEWPM-NEXT:    [[CMP_NOT:%.*]] = icmp eq i32 [[DEC]], 0
-; NEWPM-NEXT:    br i1 [[CMP_NOT]], label [[WHILE_END]], label [[WHILE_BODY]], [[LOOP3:!llvm.loop !.*]]
+; NEWPM-NEXT:    [[SCEVGEP:%.*]] = getelementptr i8, i8* [[PDST]], i32 [[TMP0]]
+; NEWPM-NEXT:    br label [[WHILE_END]]
 ; NEWPM:       while.end:
-; NEWPM-NEXT:    [[PDST_ADDR_0_LCSSA:%.*]] = phi i8* [ [[PDST]], [[ENTRY:%.*]] ], [ [[ADD_PTR_I]], [[WHILE_BODY]] ]
+; NEWPM-NEXT:    [[PDST_ADDR_0_LCSSA:%.*]] = phi i8* [ [[PDST]], [[ENTRY:%.*]] ], [ [[SCEVGEP]], [[WHILE_BODY_PREHEADER]] ]
 ; NEWPM-NEXT:    [[REM:%.*]] = and i32 [[BLOCKSIZE]], 3
 ; NEWPM-NEXT:    [[CMP14_NOT20:%.*]] = icmp eq i32 [[REM]], 0
 ; NEWPM-NEXT:    br i1 [[CMP14_NOT20]], label [[WHILE_END18:%.*]], label [[WHILE_BODY16_PREHEADER:%.*]]
 ; NEWPM:       while.body16.preheader:
 ; NEWPM-NEXT:    call void @llvm.memset.p0i8.i32(i8* align 1 [[PDST_ADDR_0_LCSSA]], i8 [[VALUE]], i32 [[REM]], i1 false)
 ; NEWPM-NEXT:    br label [[WHILE_END18]]
 ; NEWPM:       while.end18:
 ; NEWPM-NEXT:    ret void

llvm/test/Transforms/PhaseOrdering/X86/loop-idiom-vs-indvars.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -mcpu=core-avx2 < %s -O3 -S                                        | FileCheck -check-prefixes=ALL,OLDPM %s
 ; RUN: opt -mcpu=core-avx2 < %s -passes='default<O3>' -aa-pipeline=default -S | FileCheck -check-prefixes=ALL,NEWPM %s
 
 ; Not only should we be able to make the loop countable,
 ; %whatever.next recurrence should be rewritten, making loop dead.
 
 target triple = "x86_64--"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
 define i32 @cttz(i32 %n, i32* %p1) {
 ; ALL-LABEL: @cttz(
 ; ALL-NEXT:  entry:
 ; ALL-NEXT:    [[TMP0:%.*]] = shl i32 [[N:%.*]], 1
 ; ALL-NEXT:    [[TMP1:%.*]] = call i32 @llvm.cttz.i32(i32 [[TMP0]], i1 false), [[RNG0:!range !.*]]
-; ALL-NEXT:    [[TMP2:%.*]] = sub nuw nsw i32 33, [[TMP1]]
-; ALL-NEXT:    [[TMP3:%.*]] = sub nuw nsw i32 33, [[TMP1]]
-; ALL-NEXT:    [[MIN_ITERS_CHECK:%.*]] = icmp ugt i32 [[TMP1]], 25
-; ALL-NEXT:    br i1 [[MIN_ITERS_CHECK]], label [[WHILE_COND_PREHEADER:%.*]], label [[VECTOR_PH:%.*]]
-; ALL:       vector.ph:
-; ALL-NEXT:    [[N_VEC:%.*]] = and i32 [[TMP3]], -8
-; ALL-NEXT:    [[IND_END:%.*]] = sub nsw i32 [[TMP2]], [[N_VEC]]
-; ALL-NEXT:    [[TMP4:%.*]] = add nsw i32 [[N_VEC]], -8
-; ALL-NEXT:    [[TMP5:%.*]] = lshr exact i32 [[TMP4]], 3
-; ALL-NEXT:    [[TMP6:%.*]] = add nuw nsw i32 [[TMP5]], 1
-; ALL-NEXT:    [[XTRAITER:%.*]] = and i32 [[TMP6]], 7
-; ALL-NEXT:    [[TMP7:%.*]] = icmp ult i32 [[TMP4]], 56
-; ALL-NEXT:    br i1 [[TMP7]], label [[MIDDLE_BLOCK_UNR_LCSSA:%.*]], label [[VECTOR_PH_NEW:%.*]]
-; ALL:       vector.ph.new:
-; ALL-NEXT:    [[UNROLL_ITER:%.*]] = and i32 [[TMP6]], 1073741816
-; ALL-NEXT:    br label [[VECTOR_BODY:%.*]]
-; ALL:       vector.body:
-; ALL-NEXT:    [[VEC_PHI:%.*]] = phi <8 x i32> [ <i32 42, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>, [[VECTOR_PH_NEW]] ], [ [[TMP8:%.*]], [[VECTOR_BODY]] ]
-; ALL-NEXT:    [[NITER:%.*]] = phi i32 [ [[UNROLL_ITER]], [[VECTOR_PH_NEW]] ], [ [[NITER_NSUB_7:%.*]], [[VECTOR_BODY]] ]
-; ALL-NEXT:    [[TMP8]] = add <8 x i32> [[VEC_PHI]], <i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8, i32 8>
-; ALL-NEXT:    [[NITER_NSUB_7]] = add i32 [[NITER]], -8
-; ALL-NEXT:    [[NITER_NCMP_7:%.*]] = icmp eq i32 [[NITER_NSUB_7]], 0
-; ALL-NEXT:    br i1 [[NITER_NCMP_7]], label [[MIDDLE_BLOCK_UNR_LCSSA]], label [[VECTOR_BODY]], [[LOOP1:!llvm.loop !.*]]
-; ALL:       middle.block.unr-lcssa:
-; ALL-NEXT:    [[DOTLCSSA_PH:%.*]] = phi <8 x i32> [ undef, [[VECTOR_PH]] ], [ [[TMP8]], [[VECTOR_BODY]] ]
-; ALL-NEXT:    [[VEC_PHI_UNR:%.*]] = phi <8 x i32> [ <i32 42, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>, [[VECTOR_PH]] ], [ [[TMP8]], [[VECTOR_BODY]] ]
-; ALL-NEXT:    [[LCMP_MOD_NOT:%.*]] = icmp eq i32 [[XTRAITER]], 0
-; ALL-NEXT:    br i1 [[LCMP_MOD_NOT]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY_EPIL:%.*]]
-; ALL:       vector.body.epil:
-; ALL-NEXT:    [[VEC_PHI_EPIL:%.*]] = phi <8 x i32> [ [[TMP9:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[VEC_PHI_UNR]], [[MIDDLE_BLOCK_UNR_LCSSA]] ]
-; ALL-NEXT:    [[EPIL_ITER:%.*]] = phi i32 [ [[EPIL_ITER_SUB:%.*]], [[VECTOR_BODY_EPIL]] ], [ [[XTRAITER]], [[MIDDLE_BLOCK_UNR_LCSSA]] ]
-; ALL-NEXT:    [[TMP9]] = add <8 x i32> [[VEC_PHI_EPIL]], <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1>
-; ALL-NEXT:    [[EPIL_ITER_SUB]] = add i32 [[EPIL_ITER]], -1
-; ALL-NEXT:    [[EPIL_ITER_CMP_NOT:%.*]] = icmp eq i32 [[EPIL_ITER_SUB]], 0
-; ALL-NEXT:    br i1 [[EPIL_ITER_CMP_NOT]], label [[MIDDLE_BLOCK]], label [[VECTOR_BODY_EPIL]], [[LOOP3:!llvm.loop !.*]]
-; ALL:       middle.block:
-; ALL-NEXT:    [[DOTLCSSA:%.*]] = phi <8 x i32> [ [[DOTLCSSA_PH]], [[MIDDLE_BLOCK_UNR_LCSSA]] ], [ [[TMP9]], [[VECTOR_BODY_EPIL]] ]
-; ALL-NEXT:    [[TMP10:%.*]] = call i32 @llvm.vector.reduce.add.v8i32(<8 x i32> [[DOTLCSSA]])
-; ALL-NEXT:    [[CMP_N:%.*]] = icmp eq i32 [[TMP3]], [[N_VEC]]
-; ALL-NEXT:    br i1 [[CMP_N]], label [[WHILE_END:%.*]], label [[WHILE_COND_PREHEADER]]
-; ALL:       while.cond.preheader:
-; ALL-NEXT:    [[TCPHI_PH:%.*]] = phi i32 [ [[TMP2]], [[ENTRY:%.*]] ], [ [[IND_END]], [[MIDDLE_BLOCK]] ]
-; ALL-NEXT:    [[WHATEVER_PH:%.*]] = phi i32 [ 42, [[ENTRY]] ], [ [[TMP10]], [[MIDDLE_BLOCK]] ]
-; ALL-NEXT:    br label [[WHILE_COND:%.*]]
-; ALL:       while.cond:
-; ALL-NEXT:    [[TCPHI:%.*]] = phi i32 [ [[TCDEC:%.*]], [[WHILE_COND]] ], [ [[TCPHI_PH]], [[WHILE_COND_PREHEADER]] ]
-; ALL-NEXT:    [[WHATEVER:%.*]] = phi i32 [ [[WHATEVER_NEXT:%.*]], [[WHILE_COND]] ], [ [[WHATEVER_PH]], [[WHILE_COND_PREHEADER]] ]
-; ALL-NEXT:    [[TCDEC]] = add nsw i32 [[TCPHI]], -1
-; ALL-NEXT:    [[TOBOOL:%.*]] = icmp eq i32 [[TCDEC]], 0
-; ALL-NEXT:    [[WHATEVER_NEXT]] = add nuw nsw i32 [[WHATEVER]], 1
-; ALL-NEXT:    br i1 [[TOBOOL]], label [[WHILE_END]], label [[WHILE_COND]], [[LOOP5:!llvm.loop !.*]]
-; ALL:       while.end:
-; ALL-NEXT:    [[WHATEVER_NEXT_LCSSA:%.*]] = phi i32 [ [[TMP10]], [[MIDDLE_BLOCK]] ], [ [[WHATEVER_NEXT]], [[WHILE_COND]] ]
-; ALL-NEXT:    [[TMP11:%.*]] = sub nuw nsw i32 32, [[TMP1]]
-; ALL-NEXT:    store i32 [[WHATEVER_NEXT_LCSSA]], i32* [[P1:%.*]], align 4
-; ALL-NEXT:    ret i32 [[TMP11]]
+; ALL-NEXT:    [[TMP2:%.*]] = sub nuw nsw i32 32, [[TMP1]]
+; ALL-NEXT:    [[TMP3:%.*]] = sub nuw nsw i32 75, [[TMP1]]
+; ALL-NEXT:    store i32 [[TMP3]], i32* [[P1:%.*]], align 4
+; ALL-NEXT:    ret i32 [[TMP2]]
 ;
 entry:
   br label %while.cond
 
 while.cond:                                       ; preds = %while.cond, %entry
   %n.addr.0 = phi i32 [ %n, %entry ], [ %shl, %while.cond ]
   %whatever = phi i32 [ 42, %entry ], [ %whatever.next, %while.cond ]
   %i.0 = phi i32 [ 0, %entry ], [ %inc, %while.cond ]