[NPM][LTO] Update buildLTODefaultPipeline to be more in-line with the old pass manager

[dmgreen]

Summary

The NPM LTO pipeline has a lot of fixme's and missing passes, causing a lot of regressions after the switch in c70737b. Notably unrolling and vectorization were both disabled, but many other passes are missing compared to the old pass manager. This attempt to enable the most obvious missing passes like the unroller, vectorization and other loop passes, fixing the existing FIXME comments.

[dmgreen]

Added some passes to a LPM, to get the same indvar->unroll->indvar behavior as before.

[fhahn]

Thanks for putting up the patch! I am a bit surprised that all those optimizations were not run during LTO and that this did not pop up before the switch.

llvm/lib/Passes/PassBuilder.cpp

1740–1741

It looks like there are a few other passes missing here, like LoopDeletion and MergedLoadStoreMotionPass, and conditionally LoopInterchange/ConstraintElimination. Should they be added as well?

1762

Should we run WarnMissedTransformationsPass here, like in the legacy PM?

[fhahn]

(now that c70737ba1dea landed, I should be able to provide some performance numbers on Darwin platforms soonish)

[dmgreen]

In D96780#2565858, @fhahn wrote:

Thanks for putting up the patch! I am a bit surprised that all those optimizations were not run during LTO and that this did not pop up before the switch.

We didn't see any change in SPEC, surprisingly. I think it's much more important for certain code than others. We have some DSP routines where with LTO the loop range can be constant-propagated, allowing the whole loop to potentially be unrolled and greatly simplified. The largest increase I saw from this patch was a 1350% speed increase, with others in the 100-500% range. This didn't get us exactly back to the old pass manager though, which is probably expected and what remained was some much smaller positives, some negatives.

llvm/lib/Passes/PassBuilder.cpp

1740–1741

I was trying to keep this simple, to fix the regressions sooner rather than later. I'll see what I can do to add a few others.

[dmgreen]

Added a few extra passes.

llvm/lib/Passes/PassBuilder.cpp

1740–1741

LoopInterchange I've not added because it's not anywhere in the NPM yet. It can be added when it's added to the main optimization passes. The others should be here now.

[ychen]

It would be great to have some compile-time impact data before landing (https://llvm-compile-time-tracker.com/, clang self-host, etc.). This also depends on the potential performance gain on average. Just want to make sure we make the appropriate tradeoff.

[dmgreen]

In D96780#2566399, @ychen wrote:

It would be great to have some compile-time impact data before landing (https://llvm-compile-time-tracker.com/, clang self-host, etc.). This also depends on the potential performance gain on average. Just want to make sure we make the appropriate tradeoff.

"1350%" :)

This is only trying to get us back to where we were before, before all the regressions from the new pass manager. Whilst I can agree that compile time is important, this was a pretty significant set of regressions for us. Even if it is a compile time increase, the results I'm seeing would easily make it worth-while! The only real alternative is that we revert c70737b again until this can be fixed (which I was hoping not to have to do).

[fhahn]

In D96780#2566421, @dmgreen wrote:

In D96780#2566399, @ychen wrote:

It would be great to have some compile-time impact data before landing (https://llvm-compile-time-tracker.com/, clang self-host, etc.). This also depends on the potential performance gain on average. Just want to make sure we make the appropriate tradeoff.

"1350%" :)

This is only trying to get us back to where we were before, before all the regressions from the new pass manager. Whilst I can agree that compile time is important, this was a pretty significant set of regressions for us.

+1. From the FIXMEs it appears as if the omission of LoopVectorize. & co was not intentional, but because of initial bugs/limitations, so enabling them even if it impacts compile-time seems appropriate to me. This will help making the transition as smooth as possible and will avoid us having to track down/fix obvious regressions caused by big differences in the pipeline.

I think if we want to drastically adjust the pipeline, that should be done well after we ironed out the issues caused alone by the NewPM switch.

llvm/lib/Passes/PassBuilder.cpp

1740–1741

Oh right, we don't have to add them all in one go.

[fhahn]

In D96780#2566648, @fhahn wrote:

In D96780#2566421, @dmgreen wrote:

In D96780#2566399, @ychen wrote:

It would be great to have some compile-time impact data before landing (https://llvm-compile-time-tracker.com/, clang self-host, etc.). This also depends on the potential performance gain on average. Just want to make sure we make the appropriate tradeoff.

(I kicked off a run on https://llvm-compile-time-tracker.com/ with the patch)

[ychen]

:-) I should have mentioned that I do think we should do this even with compile-time regression. It is just useful to know what the regression/perf gain ratio is so we have the proper expectations.

[aeubanks]

thanks for doing this. I think nobody was using monolithic LTO with the new PM before so nobody had looked into this

[dmgreen]

(I kicked off a run on https://llvm-compile-time-tracker.com/ with the patch)

Thanks I don't think I have access.

Is this https://llvm-compile-time-tracker.com/compare.php?from=ba2aa5f49ebbe28ad2dbf0c5bea451f0ebf436c6&to=351ad9bca5822d58b1aef519eea069135406613a&stat=instructions? It's an increase for NewPM-ReleaseLTO-g. But with the number of extra passes run, I would say that is in line with expectation.

[nikic]

In D96780#2566850, @dmgreen wrote:

(I kicked off a run on https://llvm-compile-time-tracker.com/ with the patch)

Thanks I don't think I have access.

Is this https://llvm-compile-time-tracker.com/compare.php?from=ba2aa5f49ebbe28ad2dbf0c5bea451f0ebf436c6&to=351ad9bca5822d58b1aef519eea069135406613a&stat=instructions? It's an increase for NewPM-ReleaseLTO-g. But with the number of extra passes run, I would say that is in line with expectation.

Yes, that's the one. The impact looks fine to me for what this change does. Worth noting that even with this change NewPM-ReleaseLTO-g is still faster than LegacyPM-ReleaseLTO-g on average -- it just caught up a bit :)

[fhahn]

LGTM, thanks! This brings the new PM LTO pipeline more in line with the legacy PM. Compile-time increases a bit, but as discussed in earlier comments, it's in line with the expectations.

[dmgreen]

Thanks folks.

[haowei]

Hi, it looks like this change broke our (Fuchsia's) toolchain builder. We are seeing linker errors when we do stage2 builds:

FAILED: lib/libLTO.so.13git 
: && /b/s/w/ir/x/w/staging/llvm_build/./bin/clang++ --sysroot=/b/s/w/ir/x/w/cipd/linux-amd64 -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -flto -ffile-prefix-map=/b/s/w/ir/x/w/staging/llvm_build/tools/clang/stage2-bins=../staging/llvm_build/tools/clang/stage2-bins -ffile-prefix-map=/b/s/w/ir/x/w/llvm-project/= -no-canonical-prefixes -O3 -DNDEBUG  -static-libstdc++ -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics -flto   -Wl,-O3 -Wl,--gc-sections  -Wl,--version-script,"/b/s/w/ir/x/w/staging/llvm_build/tools/clang/stage2-bins/tools/lto/LTO.exports" -shared -Wl,-soname,libLTO.so.13git -o lib/libLTO.so.13git tools/lto/CMakeFiles/LTO.dir/LTODisassembler.cpp.o tools/lto/CMakeFiles/LTO.dir/lto.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMX86AsmParser.a  lib/libLLVMARMAsmParser.a  lib/libLLVMAArch64AsmParser.a  lib/libLLVMRISCVAsmParser.a  lib/libLLVMX86CodeGen.a  lib/libLLVMARMCodeGen.a  lib/libLLVMAArch64CodeGen.a  lib/libLLVMRISCVCodeGen.a  lib/libLLVMX86Desc.a  lib/libLLVMARMDesc.a  lib/libLLVMAArch64Desc.a  lib/libLLVMRISCVDesc.a  lib/libLLVMX86Disassembler.a  lib/libLLVMARMDisassembler.a  lib/libLLVMAArch64Disassembler.a  lib/libLLVMRISCVDisassembler.a  lib/libLLVMX86Info.a  lib/libLLVMARMInfo.a  lib/libLLVMAArch64Info.a  lib/libLLVMRISCVInfo.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMCodeGen.a  lib/libLLVMLTO.a  lib/libLLVMMC.a  lib/libLLVMMCDisassembler.a  lib/libLLVMSupport.a  lib/libLLVMTarget.a  lib/libLLVMCFGuard.a  lib/libLLVMAsmPrinter.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMARMDesc.a  lib/libLLVMARMInfo.a  lib/libLLVMARMUtils.a  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a  lib/libLLVMAArch64Utils.a  lib/libLLVMRISCVDesc.a  lib/libLLVMRISCVInfo.a  lib/libLLVMMCDisassembler.a  lib/libLLVMCodeGen.a  lib/libLLVMExtensions.a  lib/libLLVMPasses.a  lib/libLLVMTarget.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a  lib/libLLVMScalarOpts.a  lib/libLLVMBitWriter.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMLinker.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMVectorize.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMObject.a  lib/libLLVMBitReader.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMTextAPI.a  lib/libLLVMBinaryFormat.a  lib/libLLVMSupport.a  -lrt  -ldl  -lpthread  -lm  /b/s/w/ir/x/w/staging/zlib_install/lib/libz.a  lib/libLLVMDemangle.a && :
ld.lld: /b/s/w/ir/x/w/llvm-project/llvm/lib/Analysis/MemorySSAUpdater.cpp:1275: void llvm::MemorySSAUpdater::wireOldPredecessorsToNewImmediatePredecessor(llvm::BasicBlock *, llvm::BasicBlock *, ArrayRef<llvm::BasicBlock *>, bool): Assertion `!MSSA->getWritableBlockAccesses(New) && "Access list should be null for a new block."' failed.
clang++: error: unable to execute command: Aborted
clang++: error: linker command failed due to signal (use -v to see invocation)

We tried to revert this change locally and this error goes away. Could you revert this change please?

[dmgreen]

Hello. Yeah I've been looking at a similar thing. Let me try and fix it, after verifying it works. The LPM is not preserving MemorySSA.

Thanks for the report.

[dmgreen]

I've committed c141c6551be64f220b71786d24e98f6de906e6de in an attempt to fix the similar problem I was seeing here. Please let me know if it helps or doesn't in the Fuchsia build.

[leonardchan]

In D96780#2574007, @dmgreen wrote:

I've committed c141c6551be64f220b71786d24e98f6de906e6de in an attempt to fix the similar problem I was seeing here. Please let me know if it helps or doesn't in the Fuchsia build.

Thanks. This looks like it fixed our builders.

llvm/lib/Passes/PassBuilder.cpp

PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level,
 
   // LTO provides additional opportunities for tailcall elimination due to
   // link-time inlining, and visibility of nocapture attribute.
   FPM.addPass(TailCallElimPass());
 
   // Run a few AA driver optimizations here and now to cleanup the code.
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(FPM)));
 
-  MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(
-              PostOrderFunctionAttrsPass()));
+  MPM.addPass(
+      createModuleToPostOrderCGSCCPassAdaptor(PostOrderFunctionAttrsPass()));
   // FIXME: here we run IP alias analysis in the legacy PM.
 
   FunctionPassManager MainFPM;
 
-  // FIXME: once we fix LoopPass Manager, add LICM here.
-  // FIXME: once we provide support for enabling MLSM, add it here.
+  MainFPM.addPass(createFunctionToLoopPassAdaptor(
+      LICMPass(PTO.LicmMssaOptCap, PTO.LicmMssaNoAccForPromotionCap)));
+
   if (RunNewGVN)
     MainFPM.addPass(NewGVNPass());
   else
     MainFPM.addPass(GVN());
 
   // Remove dead memcpy()'s.
   MainFPM.addPass(MemCpyOptPass());
 
   // Nuke dead stores.
   MainFPM.addPass(DSEPass());
+  // FIXME: once we provide support for enabling MLSM, add it here.
 
-  // FIXME: at this point, we run a bunch of loop passes:
-  // indVarSimplify, loopDeletion, loopInterchange, loopUnroll,
-  // loopVectorize. Enable them once the remaining issue with LPM
-  // are sorted out.
+  // More loops are countable; try to optimize them.
+  if (EnableLoopFlatten && Level.getSpeedupLevel() > 1)
+    MainFPM.addPass(LoopFlattenPass());
+  MainFPM.addPass(createFunctionToLoopPassAdaptor(IndVarSimplifyPass()));
+  MainFPM.addPass(createFunctionToLoopPassAdaptor(LoopDeletionPass()));
+
+  // Unroll small loops and perform peeling.
+  MainFPM.addPass(createFunctionToLoopPassAdaptor(
+      LoopFullUnrollPass(Level.getSpeedupLevel(),
+                         /* OnlyWhenForced= */ !PTO.LoopUnrolling,
+                         PTO.ForgetAllSCEVInLoopUnroll)));
+  MainFPM.addPass(LoopDistributePass());
+  MainFPM.addPass(LoopVectorizePass(
+      LoopVectorizeOptions(!PTO.LoopInterleaving, !PTO.LoopVectorization)));
+  // The vectorizer may have significantly shortened a loop body; unroll again.
+  MainFPM.addPass(LoopUnrollPass(LoopUnrollOptions(
+      Level.getSpeedupLevel(), /*OnlyWhenForced=*/!PTO.LoopUnrolling,
+      PTO.ForgetAllSCEVInLoopUnroll)));
 
   MainFPM.addPass(InstCombinePass());
   MainFPM.addPass(SimplifyCFGPass(SimplifyCFGOptions().hoistCommonInsts(true)));
   MainFPM.addPass(SCCPPass());
   MainFPM.addPass(InstCombinePass());
   MainFPM.addPass(BDCEPass());
 
-  // FIXME: We may want to run SLPVectorizer here.
-  // After vectorization, assume intrinsics may tell us more
-  // about pointer alignments.
-#if 0
-  MainFPM.add(AlignmentFromAssumptionsPass());
-#endif
+  // More scalar chains could be vectorized due to more alias information
+  if (PTO.SLPVectorization) {
+    MainFPM.addPass(SLPVectorizerPass());
+    if (Level.getSpeedupLevel() > 1 && ExtraVectorizerPasses) {
+      MainFPM.addPass(EarlyCSEPass());
+    }
+  }
+
+  MainFPM.addPass(VectorCombinePass()); // Clean up partial vectorization.
+
+  // After vectorization, assume intrinsics may tell us more about pointer
+  // alignments.
+  MainFPM.addPass(AlignmentFromAssumptionsPass());
 
   // FIXME: Conditionally run LoadCombine here, after it's ported
   // (in case we still have this pass, given its questionable usefulness).
 
   MainFPM.addPass(InstCombinePass());
   invokePeepholeEPCallbacks(MainFPM, Level);
   MainFPM.addPass(JumpThreadingPass(/*InsertFreezeWhenUnfoldingSelect*/ true));
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(MainFPM)));

llvm/test/Other/new-pm-lto-defaults.ll

 ; Basic test for the new LTO pipeline.
 ; For now the only difference is between -O1 and everything else, so
 ; -O2, -O3, -Os, -Oz are the same.
 
 ; RUN: opt -disable-verify -debug-pass-manager \
 ; RUN:     -passes='lto<O1>' -S %s 2>&1 \
 ; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O1
 ; RUN: opt -disable-verify -debug-pass-manager \
 ; RUN:     -passes='lto<O2>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O2
+; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O23SZ \
+; RUN:     --check-prefix=CHECK-O2
 ; RUN: opt -disable-verify -debug-pass-manager \
 ; RUN:     -passes='lto<O3>' -S  %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O2 \
+; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O23SZ \
 ; RUN:     --check-prefix=CHECK-O3
 ; RUN: opt -disable-verify -debug-pass-manager \
 ; RUN:     -passes='lto<Os>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O2
+; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O23SZ \
+; RUN:     --check-prefix=CHECK-OS
 ; RUN: opt -disable-verify -debug-pass-manager \
 ; RUN:     -passes='lto<Oz>' -S %s 2>&1 \
-; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O2
+; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O23SZ
 ; RUN: opt -disable-verify -debug-pass-manager \
 ; RUN:     -passes='lto<O3>' -S  %s -passes-ep-peephole='no-op-function' 2>&1 \
-; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O2 \
+; RUN:     | FileCheck %s --check-prefix=CHECK-O --check-prefix=CHECK-O23SZ \
 ; RUN:     --check-prefix=CHECK-O3 --check-prefix=CHECK-EP-Peephole
 
 ; CHECK-O: Starting llvm::Module pass manager run.
 ; CHECK-O-NEXT: Running pass: Annotation2Metadata
 ; CHECK-O-NEXT: Running pass: GlobalDCEPass
 ; CHECK-O-NEXT: Running pass: ForceFunctionAttrsPass
 ; CHECK-O-NEXT: Running pass: InferFunctionAttrsPass
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}Module
 ; CHECK-O-NEXT: Running analysis: TargetLibraryAnalysis
-; CHECK-O2-NEXT: Starting llvm::Function pass manager run.
-; CHECK-O2-NEXT: Running pass: CallSiteSplittingPass on foo
-; CHECK-O2-NEXT: Running analysis: TargetLibraryAnalysis on foo
-; CHECK-O2-NEXT: Running analysis: TargetIRAnalysis on foo
-; CHECK-O2-NEXT: Running analysis: DominatorTreeAnalysis on foo
-; CHECK-O2-NEXT: Finished llvm::Function pass manager run.
-; CHECK-O2-NEXT: PGOIndirectCallPromotion
-; CHECK-O2-NEXT: Running analysis: ProfileSummaryAnalysis
-; CHECK-O2-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis
-; CHECK-O2-NEXT: Running pass: IPSCCPPass
-; CHECK-O2-NEXT: Running analysis: AssumptionAnalysis on foo
-; CHECK-O2-NEXT: Running pass: CalledValuePropagationPass
+; CHECK-O23SZ-NEXT: Starting llvm::Function pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: CallSiteSplittingPass on foo
+; CHECK-O23SZ-NEXT: Running analysis: TargetLibraryAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: TargetIRAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: DominatorTreeAnalysis on foo
+; CHECK-O23SZ-NEXT: Finished llvm::Function pass manager run.
+; CHECK-O23SZ-NEXT: PGOIndirectCallPromotion
+; CHECK-O23SZ-NEXT: Running analysis: ProfileSummaryAnalysis
+; CHECK-O23SZ-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis
+; CHECK-O23SZ-NEXT: Running pass: IPSCCPPass
+; CHECK-O23SZ-NEXT: Running analysis: AssumptionAnalysis on foo
+; CHECK-O23SZ-NEXT: Running pass: CalledValuePropagationPass
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy<{{.*}}SCC
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O1-NEXT: Running analysis: TargetLibraryAnalysis
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running pass: PostOrderFunctionAttrsPass
 ; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O-NEXT: Running analysis: BasicAA
 ; CHECK-O1-NEXT: Running analysis: AssumptionAnalysis on foo
 ; CHECK-O1-NEXT: Running analysis: DominatorTreeAnalysis
 ; CHECK-O-NEXT: Running analysis: ScopedNoAliasAA
 ; CHECK-O-NEXT: Running analysis: TypeBasedAA
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running pass: ReversePostOrderFunctionAttrsPass
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: GlobalSplitPass
 ; CHECK-O-NEXT: Running pass: WholeProgramDevirtPass
 ; CHECK-O1-NEXT: Running pass: LowerTypeTestsPass
-; CHECK-O2-NEXT: Running pass: GlobalOptPass
-; CHECK-O2-NEXT: Running pass: PromotePass
-; CHECK-O2-NEXT: Running pass: ConstantMergePass
-; CHECK-O2-NEXT: Running pass: DeadArgumentEliminationPass
-; CHECK-O2-NEXT: Starting llvm::Function pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: GlobalOptPass
+; CHECK-O23SZ-NEXT: Running pass: PromotePass
+; CHECK-O23SZ-NEXT: Running pass: ConstantMergePass
+; CHECK-O23SZ-NEXT: Running pass: DeadArgumentEliminationPass
+; CHECK-O23SZ-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O3-NEXT: Running pass: AggressiveInstCombinePass
-; CHECK-O2-NEXT: Running pass: InstCombinePass
+; CHECK-O23SZ-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
-; CHECK-O2-NEXT: Finished llvm::Function pass manager run.
-; CHECK-O2-NEXT: Running pass: ModuleInlinerWrapperPass
-; CHECK-O2-NEXT: Running analysis: InlineAdvisorAnalysis
-; CHECK-O2-NEXT: Starting llvm::Module pass manager run.
-; CHECK-O2-NEXT: Starting CGSCC pass manager run.
-; CHECK-O2-NEXT: Running pass: InlinerPass
-; CHECK-O2-NEXT: Running pass: InlinerPass
-; CHECK-O2-NEXT: Finished CGSCC pass manager run.
-; CHECK-O2-NEXT: Finished llvm::Module pass manager run.
-; CHECK-O2-NEXT: Running pass: GlobalOptPass
-; CHECK-O2-NEXT: Running pass: GlobalDCEPass
-; CHECK-O2-NEXT: Starting llvm::Function pass manager run.
-; CHECK-O2-NEXT: Running pass: InstCombinePass
+; CHECK-O23SZ-NEXT: Finished llvm::Function pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O23SZ-NEXT: Running analysis: InlineAdvisorAnalysis
+; CHECK-O23SZ-NEXT: Starting llvm::Module pass manager run.
+; CHECK-O23SZ-NEXT: Starting CGSCC pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: InlinerPass
+; CHECK-O23SZ-NEXT: Running pass: InlinerPass
+; CHECK-O23SZ-NEXT: Finished CGSCC pass manager run.
+; CHECK-O23SZ-NEXT: Finished llvm::Module pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: GlobalOptPass
+; CHECK-O23SZ-NEXT: Running pass: GlobalDCEPass
+; CHECK-O23SZ-NEXT: Starting llvm::Function pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
-; CHECK-O2-NEXT: Running pass: JumpThreadingPass
-; CHECK-O2-NEXT: Running analysis: LazyValueAnalysis
-; CHECK-O2-NEXT: Running pass: SROA on foo
-; CHECK-O2-NEXT: Running pass: TailCallElimPass on foo
-; CHECK-O2-NEXT: Finished llvm::Function pass manager run.
-; CHECK-O2-NEXT: Running pass: PostOrderFunctionAttrsPass
-; CHECK-O2-NEXT: Running pass: GVN on foo
-; CHECK-O2-NEXT: Running analysis: MemoryDependenceAnalysis
-; CHECK-O2-NEXT: Running analysis: PhiValuesAnalysis
-; CHECK-O2-NEXT: Running pass: MemCpyOptPass on foo
-; CHECK-O2-NEXT: Running pass: DSEPass on foo
-; CHECK-O2-NEXT: Running analysis: MemorySSAAnalysis on foo
-; CHECK-O2-NEXT: Running analysis: PostDominatorTreeAnalysis on foo
-; CHECK-O2-NEXT: Running pass: InstCombinePass on foo
-; CHECK-O2-NEXT: Running pass: SimplifyCFGPass on foo
-; CHECK-O2-NEXT: Running pass: SCCPPass on foo
-; CHECK-O2-NEXT: Running pass: InstCombinePass on foo
-; CHECK-O2-NEXT: Running pass: BDCEPass on foo
-; CHECK-O2-NEXT: Running analysis: DemandedBitsAnalysis
-; CHECK-O2-NEXT: Running pass: InstCombinePass
-; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
-; CHECK-O2-NEXT: Running pass: JumpThreadingPass
-; CHECK-O2-NEXT: Running pass: CrossDSOCFIPass
-; CHECK-O2-NEXT: Running pass: LowerTypeTestsPass
+; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass
+; CHECK-O23SZ-NEXT: Running analysis: LazyValueAnalysis
+; CHECK-O23SZ-NEXT: Running pass: SROA on foo
+; CHECK-O23SZ-NEXT: Running pass: TailCallElimPass on foo
+; CHECK-O23SZ-NEXT: Finished llvm::Function pass manager run.
+; CHECK-O23SZ-NEXT: Running pass: PostOrderFunctionAttrsPass on (foo)
+; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass on foo
+; CHECK-O23SZ-NEXT: Running analysis: LoopAnalysis on foo
+; CHECK-O23SZ-NEXT: Running pass: LCSSAPass on foo
+; CHECK-O23SZ-NEXT: Running analysis: ScalarEvolutionAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: InnerAnalysisManagerProxy
+; CHECK-O23SZ-NEXT: Running pass: LICMPass on Loop
+; CHECK-O23SZ-NEXT: Running pass: GVN on foo
+; CHECK-O23SZ-NEXT: Running analysis: MemoryDependenceAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: PhiValuesAnalysis on foo
+; CHECK-O23SZ-NEXT: Running pass: MemCpyOptPass on foo
+; CHECK-O23SZ-NEXT: Running pass: DSEPass on foo
+; CHECK-O23SZ-NEXT: Running analysis: MemorySSAAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: PostDominatorTreeAnalysis on foo
+; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass on foo
+; CHECK-O23SZ-NEXT: Running pass: LCSSAPass on foo
+; CHECK-O23SZ-NEXT: Running pass: IndVarSimplifyPass on Loop
+; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass on foo
+; CHECK-O23SZ-NEXT: Running pass: LCSSAPass on foo
+; CHECK-O23SZ-NEXT: Running pass: LoopDeletionPass on Loop
+; CHECK-O23SZ-NEXT: Running pass: LoopSimplifyPass on foo
+; CHECK-O23SZ-NEXT: Running pass: LCSSAPass on foo
+; CHECK-O23SZ-NEXT: Running pass: LoopFullUnrollPass on Loop
+; CHECK-O23SZ-NEXT: Running pass: LoopDistributePass on foo
+; CHECK-O23SZ-NEXT: Running pass: LoopVectorizePass on foo
+; CHECK-O23SZ-NEXT: Running analysis: BlockFrequencyAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: BranchProbabilityAnalysis on foo
+; CHECK-O23SZ-NEXT: Running analysis: DemandedBitsAnalysis on foo
+; CHECK-O23SZ-NEXT: Running pass: LoopUnrollPass on foo
+; CHECK-O23SZ-NEXT: Running pass: InstCombinePass on foo
+; CHECK-O23SZ-NEXT: Running pass: SimplifyCFGPass on foo
+; CHECK-O23SZ-NEXT: Running pass: SCCPPass on foo
+; CHECK-O23SZ-NEXT: Running pass: InstCombinePass on foo
+; CHECK-O23SZ-NEXT: Running pass: BDCEPass on foo
+; CHECK-O2-NEXT: Running pass: SLPVectorizerPass on foo
+; CHECK-O3-NEXT: Running pass: SLPVectorizerPass on foo
+; CHECK-OS-NEXT: Running pass: SLPVectorizerPass on foo
+; CHECK-O23SZ-NEXT: Running pass: VectorCombinePass on foo
+; CHECK-O23SZ-NEXT: Running pass: AlignmentFromAssumptionsPass on foo
+; CHECK-O23SZ-NEXT: Running pass: InstCombinePass on foo
+; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass on foo
+; CHECK-O23SZ-NEXT: Running pass: JumpThreadingPass on foo
+; CHECK-O23SZ-NEXT: Running pass: CrossDSOCFIPass
+; CHECK-O23SZ-NEXT: Running pass: LowerTypeTestsPass
 ; CHECK-O-NEXT: Running pass: LowerTypeTestsPass
-; CHECK-O2-NEXT: Running pass: SimplifyCFGPass
-; CHECK-O2-NEXT: Running pass: EliminateAvailableExternallyPass
-; CHECK-O2-NEXT: Running pass: GlobalDCEPass
+; CHECK-O23SZ-NEXT: Running pass: SimplifyCFGPass
+; CHECK-O23SZ-NEXT: Running pass: EliminateAvailableExternallyPass
+; CHECK-O23SZ-NEXT: Running pass: GlobalDCEPass
 ; CHECK-O-NEXT: Running pass: AnnotationRemarksPass on foo
 ; CHECK-O-NEXT: Running pass: PrintModulePass
 
 ; Make sure we get the IR back out without changes when we print the module.
 ; CHECK-O-LABEL: define void @foo(i32 %n) local_unnamed_addr {
 ; CHECK-O-NEXT: entry:
 ; CHECK-O-NEXT:   br label %loop
 ; CHECK-O:      loop: