[PassManager] `buildFunctionSimplificationPipeline()`: schedule another `LoopDeletion` pass run before last `LICM`

[lebedev.ri]

Summary

Test thanks to Michael Kuklinski from #llvm: https://godbolt.org/z/bdrah5Goo
originally inspired by Daniel Lemire's https://lemire.me/blog/2021/10/26/in-c-is-empty-faster-than-comparing-the-size-with-zero/

We manage to deduce that the answer does not require looping,
but we do that after the last LoopDeletion pass run,
so we end up being stuck with a dead loop.

While perhaps we are (clearly) also missing LoopDeletion
in buildModuleOptimizationPipeline() part of the pipeline,
we *should* add this one here, to reduce the function size
before inlining. I believe we want to have a new loop pass manager,
so that LoopSimplifyPass/LCSSAPass are guaranteed to be computed once?

Now, as with all things SCEV, this has a very expected ~+0.5% compile time performance regression:
https://llvm-compile-time-tracker.com/compare.php?from=e5df0a5a6f412965eb1be495d2672b4164c6c3d5&to=4af31d7accb368597bc8b4fc8f125d7c2fdbae6a&stat=instructions

Looking at the transformation stats over vanilla test-suite, i think it's rather expected:

| statistic name                                   |  baseline |  proposed |      Δ |      % |    |%| |
|--------------------------------------------------|----------:|----------:|-------:|-------:|-------:|
| scalar-evolution.NumTripCountsNotComputed        |    105592 |    137458 |  31866 | 30.18% | 30.18% |
| scalar-evolution.NumBruteForceTripCountsComputed |       789 |       955 |    166 | 21.04% | 21.04% |
| scalar-evolution.NumTripCountsComputed           |    299759 |    348901 |  49142 | 16.39% | 16.39% |
| loop-delete.NumBackedgesBroken                   |       542 |       557 |     15 |  2.77% |  2.77% |
| regalloc.numExtends                              |        81 |        79 |     -2 | -2.47% |  2.47% |
| licm.NumSunk                                     |     12167 |     11875 |   -292 | -2.40% |  2.40% |
| indvars.NumFoldedUser                            |       408 |       400 |     -8 | -1.96% |  1.96% |
| indvars.NumElimCmp                               |      3831 |      3757 |    -74 | -1.93% |  1.93% |
| loop-delete.NumDeleted                           |      8055 |      8096 |     41 |  0.51% |  0.51% |
| codegenprepare.NumSelectsExpanded                |       277 |       278 |      1 |  0.36% |  0.36% |
| assume-queries.NumAssumeQueries                  | 120910554 | 121324983 | 414429 |  0.34% |  0.34% |
| loop-unroll.NumRuntimeUnrolled                   |     13841 |     13796 |    -45 | -0.33% |  0.33% |
| phi-node-elimination.NumCriticalEdgesSplit       |     83054 |     82804 |   -250 | -0.30% |  0.30% |
| branch-folder.NumBranchOpts                      |    108122 |    107827 |   -295 | -0.27% |  0.27% |
| correlated-value-propagation.NumCmps             |      1461 |      1465 |      4 |  0.27% |  0.27% |
| branch-folder.NumDeadBlocks                      |    130818 |    130490 |   -328 | -0.25% |  0.25% |
| branch-folder.NumTailMerge                       |     72501 |     72347 |   -154 | -0.21% |  0.21% |
| correlated-value-propagation.NumAShrs            |       514 |       515 |      1 |  0.19% |  0.19% |
| loop-unroll.NumUnrolled                          |     40136 |     40069 |    -67 | -0.17% |  0.17% |
| licm.NumHoisted                                  |    388827 |    388221 |   -606 | -0.16% |  0.16% |
| machine-cse.NumPREs                              |      3085 |      3080 |     -5 | -0.16% |  0.16% |
| loop-unroll.NumCompletelyUnrolled                |      9236 |      9222 |    -14 | -0.15% |  0.15% |

Note that i'm only changing current PM, and not touching obsolete PM.

[xbolva00]

Also partially solves PR52354?

Maybe loop unrolling + some cleanup pass could also solve your motivating case and also all motivation cases from PR52354?

[lebedev.ri]

In D112840#3097462, @xbolva00 wrote:

Also partially solves PR52354?

I don't believe so.

Maybe loop unrolling + some cleanup pass could also solve your motivating case and also all motivation cases from PR52354?

I don't have an answer to that question.

[aeubanks]

any numbers for actual performance gains? not sure that it's worth addressing some specifically bad code in a blog post for 0.5% compile time, but I'll run this patch on some internal benchmarks

do you know after which pass the loop becomes deletable? presumably some pass between LPM2 and LPM3 causes the loop to be deletable since LPM2 contains a LoopDeletionPass

[lebedev.ri]

In D112840#3097772, @aeubanks wrote:

any numbers for actual performance gains? not sure that it's worth addressing some specifically bad code in a blog post for 0.5% compile time, but I'll run this patch on some internal benchmarks

do you know after which pass the loop becomes deletable? presumably some pass between LPM2 and LPM3 causes the loop to be deletable since LPM2 contains a LoopDeletionPass

Answered inline.

llvm/lib/Passes/PassBuilderPipelines.cpp

527

do you know after which pass the loop becomes deletable? presumably some pass between LPM2 and LPM3 causes the loop to be deletable since LPM2 contains a LoopDeletionPass

In that particular case after this one.

[lebedev.ri]

Hmm, actually, i looked, and we catch a lot more cases if we do this in buildModuleOptimizationPipeline(), so let me do that instead. BRB.

[aeubanks]

In D112840#3097842, @lebedev.ri wrote:

Hmm, actually, i looked, and we catch a lot more cases if we do this in buildModuleOptimizationPipeline(), so let me do that instead. BRB.

it'd be nice to make this work in the function simplification pipeline so that more passes have a chance to take advantage of deleted loops, e.g. the inliner
putting passes at the end of the pipeline doesn't seem ideal unless there's a specific reason for it

[lebedev.ri]

In D112840#3097844, @aeubanks wrote:

In D112840#3097842, @lebedev.ri wrote:

Hmm, actually, i looked, and we catch a lot more cases if we do this in buildModuleOptimizationPipeline(), so let me do that instead. BRB.

it'd be nice to make this work in the function simplification pipeline so that more passes have a chance to take advantage of deleted loops, e.g. the inliner
putting passes at the end of the pipeline doesn't seem ideal unless there's a specific reason for it

From where i stand, we roughly have 3 obvious choices:

1. keep dead loops :)
2. the current diff, delete dead loops at the end of function simplification pipeline
3. upcoming diff, delete dead loops before vectorization in module optimization pipeline. catches more cases, but perhaps pessimizes inliner.
4. 2 & 3, slowest of them all.

What do you have in mind instead?

[aeubanks]

In D112840#3097852, @lebedev.ri wrote:

In D112840#3097844, @aeubanks wrote:

In D112840#3097842, @lebedev.ri wrote:

Hmm, actually, i looked, and we catch a lot more cases if we do this in buildModuleOptimizationPipeline(), so let me do that instead. BRB.

it'd be nice to make this work in the function simplification pipeline so that more passes have a chance to take advantage of deleted loops, e.g. the inliner
putting passes at the end of the pipeline doesn't seem ideal unless there's a specific reason for it

From where i stand, we roughly have 3 obvious choices:

1. keep dead loops :)
2. the current diff, delete dead loops at the end of function simplification pipeline
3. upcoming diff, delete dead loops before vectorization in module optimization pipeline. catches more cases, but perhaps pessimizes inliner.
4. 2 & 3, slowest of them all.

What do you have in mind instead?

I looked into @is_not_empty_variant1(). It seems like GVN + an instcombine cleanup is what's allowing the loop to be deleted. Currently GVN runs after the loop passes which is the issue.
I tried replacing the EarlyCSEPass at the beginning of the function simplification pipeline with a NewGVNPass and that fixes @is_not_empty_variant1. I think something along those lines is more principled than adding an LoopDeletionPass somewhere.
(Of course, if we want to turn on NewGVN specifically we need to iron out remaining issues)

[lebedev.ri]

Tentatively abandoning in favor of buildModuleOptimizationPipeline() variant D112851.
It is both more powerful, and TBD compile time wise.

[lebedev.ri]

In D112840#3097907, @lebedev.ri wrote:

Tentatively abandoning in favor of buildModuleOptimizationPipeline() variant D112851.
It is both more powerful, and TBD compile time wise.

And https://llvm-compile-time-tracker.com/compare.php?from=0ae7bf124a9bca76dd9a91b2f7379168ff13f562&to=c2ae57c9b961aeb4a28c747266949340613a6d84&stat=instructions is in,
it's only a ~+0.12% geomean regression.

llvm/lib/Passes/PassBuilderPipelines.cpp

PassBuilder::buildFunctionSimplificationPipeline(OptimizationLevel Level,
 
   // Delete dead bit computations (instcombine runs after to fold away the dead
   // computations, and then ADCE will run later to exploit any new DCE
   // opportunities that creates).
   FPM.addPass(BDCEPass());
 
   // Run instcombine after redundancy and dead bit elimination to exploit
   // opportunities opened up by them.
   FPM.addPass(InstCombinePass());
   invokePeepholeEPCallbacks(FPM, Level);
 
   // Re-consider control flow based optimizations after redundancy elimination,
   // redo DCE, etc.
   if (EnableDFAJumpThreading && Level.getSizeLevel() == 0)
     FPM.addPass(DFAJumpThreadingPass());
 
   FPM.addPass(JumpThreadingPass());
   FPM.addPass(CorrelatedValuePropagationPass());
 
   // Finally, do an expensive DCE pass to catch all the dead code exposed by
   // the simplifications and basic cleanup after all the simplifications.
   // TODO: Investigate if this is too expensive.
   FPM.addPass(ADCEPass());
 
   // Specially optimize memory movement as it doesn't look like dataflow in SSA.
   FPM.addPass(MemCpyOptPass());
 
   FPM.addPass(DSEPass());
+
+  LoopPassManager LPM3;
+  // Some loops may have become dead by now. Try to delete them first.
+  LPM3.addPass(LoopDeletionPass());
+  // If they are still around, perform loop-invariant code motion on them.
+  LPM3.addPass(LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap));
   FPM.addPass(createFunctionToLoopPassAdaptor(
-      LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap),
-      /*UseMemorySSA=*/true, /*UseBlockFrequencyInfo=*/true));
+      std::move(LPM3), /*UseMemorySSA=*/true, /*UseBlockFrequencyInfo=*/true));
 
   FPM.addPass(CoroElidePass());
 
   for (auto &C : ScalarOptimizerLateEPCallbacks)
     C(FPM, Level);
 
   FPM.addPass(SimplifyCFGPass(
       SimplifyCFGOptions().hoistCommonInsts(true).sinkCommonInsts(true)));

llvm/test/Other/new-pm-defaults.ll

 ; CHECK-O23SZ-NEXT: Invalidating analysis: LazyValueAnalysis
 ; CHECK-O1-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: ADCEPass
 ; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis
 ; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass
 ; CHECK-O23SZ-NEXT: Running pass: DSEPass
 ; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O23SZ-NEXT: Running pass: LCSSAPass
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O23SZ-NEXT: Running pass: LICMPass
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-EP-SCALAR-LATE-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-PEEPHOLE-NEXT: Running pass: NoOpFunctionPass
-; CHECK-O-NEXT: Running pass: 
+; CHECK-O-NEXT: Running pass:
 ; CHECK-O-NEXT: Invalidating analysis: InlineAdvisorAnalysis
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: GlobalDCEPass
 ; CHECK-DEFAULT-NEXT: Running pass: EliminateAvailableExternallyPass
 ; CHECK-LTO-NOT: Running pass: EliminateAvailableExternallyPass
 ; CHECK-O-NEXT: Running pass: ReversePostOrderFunctionAttrsPass
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running pass: Float2IntPass

llvm/test/Other/new-pm-thinlto-defaults.ll

 ; CHECK-O23SZ-NEXT: Invalidating analysis: LazyValueAnalysis
 ; CHECK-O1-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: ADCEPass
 ; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis
 ; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass
 ; CHECK-O23SZ-NEXT: Running pass: DSEPass
 ; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O23SZ-NEXT: Running pass: LCSSAPass
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass on Loop at depth 1 containing: %loop
 ; CHECK-O23SZ-NEXT: Running pass: LICMPass on Loop at depth 1 containing: %loop
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Invalidating analysis: InlineAdvisorAnalysis
 ; CHECK-PRELINK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-POSTLINK-O-NEXT: Running pass: GlobalOptPass

llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll

 ; CHECK-O23SZ-NEXT: Running pass: CorrelatedValuePropagationPass
 ; CHECK-O23SZ-NEXT: Invalidating analysis: LazyValueAnalysis
 ; CHECK-O1-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: ADCEPass
 ; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass
 ; CHECK-O23SZ-NEXT: Running pass: DSEPass
 ; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O23SZ-NEXT: Running pass: LCSSAPass
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O23SZ-NEXT: Running pass: LICMPass
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Running pass: CoroSplitPass
 ; CHECK-O-NEXT: Invalidating analysis: InlineAdvisorAnalysis
 ; CHECK-O-NEXT: Running pass: GlobalOptPass
 ; CHECK-O-NEXT: Running pass: GlobalDCEPass

llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll

 ; CHECK-O23SZ-NEXT: Running pass: CorrelatedValuePropagationPass
 ; CHECK-O23SZ-NEXT: Invalidating analysis: LazyValueAnalysis
 ; CHECK-O1-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: ADCEPass
 ; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass
 ; CHECK-O23SZ-NEXT: Running pass: DSEPass
 ; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O23SZ-NEXT: Running pass: LCSSAPass
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O23SZ-NEXT: Running pass: LICMPass
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O3-NEXT: Running pass: ControlHeightReductionPass on foo
 ; CHECK-O3-NEXT: Running analysis: RegionInfoAnalysis on foo
 ; CHECK-O3-NEXT: Running analysis: DominanceFrontierAnalysis on foo
 ; CHECK-O-NEXT: Running pass: CoroSplitPass

llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll

 ; CHECK-O23SZ-NEXT: Running pass: CorrelatedValuePropagationPass
 ; CHECK-O23SZ-NEXT: Invalidating analysis: LazyValueAnalysis
 ; CHECK-O1-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: ADCEPass
 ; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass
 ; CHECK-O23SZ-NEXT: Running pass: DSEPass
 ; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O23SZ-NEXT: Running pass: LCSSAPass
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O23SZ-NEXT: Running pass: LICMPass
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O3-NEXT: Running pass: ControlHeightReductionPass on foo
 ; CHECK-O3-NEXT: Running analysis: RegionInfoAnalysis on foo
 ; CHECK-O3-NEXT: Running analysis: DominanceFrontierAnalysis on foo
 ; CHECK-O-NEXT: Running pass: CoroSplitPass

llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll

 ; CHECK-O23SZ-NEXT: Running pass: CorrelatedValuePropagationPass
 ; CHECK-O23SZ-NEXT: Invalidating analysis: LazyValueAnalysis
 ; CHECK-O1-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: ADCEPass
 ; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass
 ; CHECK-O23SZ-NEXT: Running pass: DSEPass
 ; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass
 ; CHECK-O23SZ-NEXT: Running pass: LCSSAPass
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass
 ; CHECK-O23SZ-NEXT: Running pass: LICMPass
 ; CHECK-O23SZ-NEXT: Running pass: CoroElidePass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O3-NEXT: Running pass: ControlHeightReductionPass on foo
 ; CHECK-O3-NEXT: Running analysis: RegionInfoAnalysis on foo
 ; CHECK-O3-NEXT: Running analysis: DominanceFrontierAnalysis on foo
 ; CHECK-O-NEXT: Running pass: CoroSplitPass

llvm/test/Transforms/PhaseOrdering/deletion-of-loops-that-became-side-effect-free.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -passes='default<O3>' -S < %s  | FileCheck %s --check-prefixes=ALL,O3
 ; RUN: opt -passes='default<O2>' -S < %s  | FileCheck %s --check-prefixes=ALL,O2
 ; RUN: opt -passes='default<O1>' -S < %s  | FileCheck %s --check-prefixes=ALL,O1
 
 ; All these tests should optimize to a single comparison
 ; of the original argument with null. There should be no loops.
 
 %struct.node = type { %struct.node*, i32 }
 
 define dso_local zeroext i1 @is_not_empty_variant1(%struct.node* %p) {
-; ALL-LABEL: @is_not_empty_variant1(
-; ALL-NEXT:  entry:
-; ALL-NEXT:    [[TOBOOL_NOT3_I:%.*]] = icmp eq %struct.node* [[P:%.*]], null
-; ALL-NEXT:    br i1 [[TOBOOL_NOT3_I]], label [[COUNT_NODES_VARIANT1_EXIT:%.*]], label [[WHILE_BODY_I:%.*]]
-; ALL:       while.body.i:
-; ALL-NEXT:    [[P_ADDR_04_I:%.*]] = phi %struct.node* [ [[TMP0:%.*]], [[WHILE_BODY_I]] ], [ [[P]], [[ENTRY:%.*]] ]
-; ALL-NEXT:    [[NEXT_I:%.*]] = getelementptr inbounds [[STRUCT_NODE:%.*]], %struct.node* [[P_ADDR_04_I]], i64 0, i32 0
-; ALL-NEXT:    [[TMP0]] = load %struct.node*, %struct.node** [[NEXT_I]], align 8
-; ALL-NEXT:    [[TOBOOL_NOT_I:%.*]] = icmp eq %struct.node* [[TMP0]], null
-; ALL-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[COUNT_NODES_VARIANT1_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP0:![0-9]+]]
-; ALL:       count_nodes_variant1.exit:
-; ALL-NEXT:    [[TMP1:%.*]] = xor i1 [[TOBOOL_NOT3_I]], true
-; ALL-NEXT:    ret i1 [[TMP1]]
+; O3-LABEL: @is_not_empty_variant1(
+; O3-NEXT:  entry:
+; O3-NEXT:    [[TOBOOL_NOT3_I:%.*]] = icmp ne %struct.node* [[P:%.*]], null
+; O3-NEXT:    ret i1 [[TOBOOL_NOT3_I]]
+;
+; O2-LABEL: @is_not_empty_variant1(
+; O2-NEXT:  entry:
+; O2-NEXT:    [[TOBOOL_NOT3_I:%.*]] = icmp ne %struct.node* [[P:%.*]], null
+; O2-NEXT:    ret i1 [[TOBOOL_NOT3_I]]
+;
+; O1-LABEL: @is_not_empty_variant1(
+; O1-NEXT:  entry:
+; O1-NEXT:    [[TOBOOL_NOT3_I:%.*]] = icmp eq %struct.node* [[P:%.*]], null
+; O1-NEXT:    br i1 [[TOBOOL_NOT3_I]], label [[COUNT_NODES_VARIANT1_EXIT:%.*]], label [[WHILE_BODY_I:%.*]]
+; O1:       while.body.i:
+; O1-NEXT:    [[P_ADDR_04_I:%.*]] = phi %struct.node* [ [[TMP0:%.*]], [[WHILE_BODY_I]] ], [ [[P]], [[ENTRY:%.*]] ]
+; O1-NEXT:    [[NEXT_I:%.*]] = getelementptr inbounds [[STRUCT_NODE:%.*]], %struct.node* [[P_ADDR_04_I]], i64 0, i32 0
+; O1-NEXT:    [[TMP0]] = load %struct.node*, %struct.node** [[NEXT_I]], align 8
+; O1-NEXT:    [[TOBOOL_NOT_I:%.*]] = icmp eq %struct.node* [[TMP0]], null
+; O1-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[COUNT_NODES_VARIANT1_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP0:![0-9]+]]
+; O1:       count_nodes_variant1.exit:
+; O1-NEXT:    [[TMP1:%.*]] = xor i1 [[TOBOOL_NOT3_I]], true
+; O1-NEXT:    ret i1 [[TMP1]]
 ;
 entry:
   %p.addr = alloca %struct.node*, align 8
   store %struct.node* %p, %struct.node** %p.addr, align 8
   %0 = load %struct.node*, %struct.node** %p.addr, align 8
   %call = call i32 @count_nodes_variant1(%struct.node* %0)
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp
while.end:
   %6 = load i64, i64* %size, align 8
   %7 = bitcast i64* %size to i8*
   ret i64 %6
 }
 
 define dso_local zeroext i1 @is_not_empty_variant3(%struct.node* %p) {
 ; O3-LABEL: @is_not_empty_variant3(
 ; O3-NEXT:  entry:
-; O3-NEXT:    [[TOBOOL_NOT4_I:%.*]] = icmp eq %struct.node* [[P:%.*]], null
-; O3-NEXT:    br i1 [[TOBOOL_NOT4_I]], label [[COUNT_NODES_VARIANT3_EXIT:%.*]], label [[WHILE_BODY_I:%.*]]
-; O3:       while.body.i:
-; O3-NEXT:    [[SIZE_06_I:%.*]] = phi i64 [ [[INC_I:%.*]], [[WHILE_BODY_I]] ], [ 0, [[ENTRY:%.*]] ]
-; O3-NEXT:    [[P_ADDR_05_I:%.*]] = phi %struct.node* [ [[TMP0:%.*]], [[WHILE_BODY_I]] ], [ [[P]], [[ENTRY]] ]
-; O3-NEXT:    [[CMP_I:%.*]] = icmp ne i64 [[SIZE_06_I]], -1
-; O3-NEXT:    tail call void @llvm.assume(i1 [[CMP_I]]) #[[ATTR3:[0-9]+]]
-; O3-NEXT:    [[NEXT_I:%.*]] = getelementptr inbounds [[STRUCT_NODE:%.*]], %struct.node* [[P_ADDR_05_I]], i64 0, i32 0
-; O3-NEXT:    [[TMP0]] = load %struct.node*, %struct.node** [[NEXT_I]], align 8
-; O3-NEXT:    [[INC_I]] = add nuw i64 [[SIZE_06_I]], 1
-; O3-NEXT:    [[TOBOOL_NOT_I:%.*]] = icmp eq %struct.node* [[TMP0]], null
-; O3-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[COUNT_NODES_VARIANT3_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP2:![0-9]+]]
-; O3:       count_nodes_variant3.exit:
-; O3-NEXT:    [[TMP1:%.*]] = xor i1 [[TOBOOL_NOT4_I]], true
-; O3-NEXT:    ret i1 [[TMP1]]
+; O3-NEXT:    [[TOBOOL_NOT4_I:%.*]] = icmp ne %struct.node* [[P:%.*]], null
+; O3-NEXT:    ret i1 [[TOBOOL_NOT4_I]]
 ;
 ; O2-LABEL: @is_not_empty_variant3(
 ; O2-NEXT:  entry:
-; O2-NEXT:    [[TOBOOL_NOT4_I:%.*]] = icmp eq %struct.node* [[P:%.*]], null
-; O2-NEXT:    br i1 [[TOBOOL_NOT4_I]], label [[COUNT_NODES_VARIANT3_EXIT:%.*]], label [[WHILE_BODY_I:%.*]]
-; O2:       while.body.i:
-; O2-NEXT:    [[SIZE_06_I:%.*]] = phi i64 [ [[INC_I:%.*]], [[WHILE_BODY_I]] ], [ 0, [[ENTRY:%.*]] ]
-; O2-NEXT:    [[P_ADDR_05_I:%.*]] = phi %struct.node* [ [[TMP0:%.*]], [[WHILE_BODY_I]] ], [ [[P]], [[ENTRY]] ]
-; O2-NEXT:    [[CMP_I:%.*]] = icmp ne i64 [[SIZE_06_I]], -1
-; O2-NEXT:    tail call void @llvm.assume(i1 [[CMP_I]]) #[[ATTR3:[0-9]+]]
-; O2-NEXT:    [[NEXT_I:%.*]] = getelementptr inbounds [[STRUCT_NODE:%.*]], %struct.node* [[P_ADDR_05_I]], i64 0, i32 0
-; O2-NEXT:    [[TMP0]] = load %struct.node*, %struct.node** [[NEXT_I]], align 8
-; O2-NEXT:    [[INC_I]] = add nuw i64 [[SIZE_06_I]], 1
-; O2-NEXT:    [[TOBOOL_NOT_I:%.*]] = icmp eq %struct.node* [[TMP0]], null
-; O2-NEXT:    br i1 [[TOBOOL_NOT_I]], label [[COUNT_NODES_VARIANT3_EXIT]], label [[WHILE_BODY_I]], !llvm.loop [[LOOP2:![0-9]+]]
-; O2:       count_nodes_variant3.exit:
-; O2-NEXT:    [[TMP1:%.*]] = xor i1 [[TOBOOL_NOT4_I]], true
-; O2-NEXT:    ret i1 [[TMP1]]
+; O2-NEXT:    [[TOBOOL_NOT4_I:%.*]] = icmp ne %struct.node* [[P:%.*]], null
+; O2-NEXT:    ret i1 [[TOBOOL_NOT4_I]]
 ;
 ; O1-LABEL: @is_not_empty_variant3(
 ; O1-NEXT:  entry:
 ; O1-NEXT:    [[TOBOOL_NOT4_I:%.*]] = icmp eq %struct.node* [[P:%.*]], null
 ; O1-NEXT:    br i1 [[TOBOOL_NOT4_I]], label [[COUNT_NODES_VARIANT3_EXIT:%.*]], label [[WHILE_BODY_I:%.*]]
 ; O1:       while.body.i:
 ; O1-NEXT:    [[SIZE_06_I:%.*]] = phi i64 [ [[INC_I:%.*]], [[WHILE_BODY_I]] ], [ 0, [[ENTRY:%.*]] ]
 ; O1-NEXT:    [[P_ADDR_05_I:%.*]] = phi %struct.node* [ [[TMP0:%.*]], [[WHILE_BODY_I]] ], [ [[P]], [[ENTRY]] ]