[ThinLTO/FullLTO] Support Os and Oz

[ychen]

Summary

• When optnone is not present, add optsize for Os and minsize for Oz.
• Pass Os/Oz to (old/new PM) LTO pipeline setup.
• Ran SPEC2017 & test-suite MultiSource to confirm the size reduction are comparable with NonLTO.

[pcc]

This is going in the wrong direction IMO. We already have a way of specifying the size optimization level, which is to specify -Os or -Oz at compile time. Why is that not sufficient?

[ychen]

In D72404#1810489, @pcc wrote:

This is going in the wrong direction IMO. We already have a way of specifying the size optimization level, which is to specify -Os or -Oz at compile time. Why is that not sufficient?

Thanks for the quick feedback, @pcc. :-) I saw your comments somewhere on this.
I don't know why is this changing directions. It only gives the user choice to optimize for size at link stage.
They still have the freedom the do this at compile stage.

From user perspective, if O1, O2 etc. are valid options for linker stage,
I don't see a reason Os and Oz can not be used or not do the right thing.

With ThinLTO, many decisions are postponed til linker stage to have more flexibility. I guess this is one of them?

[merge_guards_bot]

Unit tests: fail. 60542 tests passed, 1 failed and 726 were skipped.

failed: lld.wasm/lto/opt-level.ll

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[ychen]

• fix test

[ychen]

fix test

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

[merge_guards_bot]

Unit tests: fail. 60542 tests passed, 1 failed and 726 were skipped.

failed: lld.ELF/note.s

clang-tidy: fail. Please fix clang-tidy findings.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[ychen]

• when optnone is not present, add optsize for Os and minsize for Oz
• add Os Oz function attribute test.
• add Os Oz pipeline test

[ychen]

• fix a typo

[ychen]

• Rebase

[merge_guards_bot]

Unit tests: fail. 60543 tests passed, 1 failed and 726 were skipped.

failed: Clang.CodeGen/thinlto-debug-pm.c

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[merge_guards_bot]

Unit tests: fail. 60543 tests passed, 1 failed and 726 were skipped.

failed: Clang.CodeGen/thinlto-debug-pm.c

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[merge_guards_bot]

Unit tests: fail. 61849 tests passed, 1 failed and 781 were skipped.

failed: Clang.CodeGen/thinlto-debug-pm.c

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[ychen]

• rebase & clang-format

[merge_guards_bot]

Unit tests: fail. 61858 tests passed, 1 failed and 781 were skipped.

failed: Clang.CodeGen/thinlto-debug-pm.c

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[tycho]

In D72404#1820461, @merge_guards_bot wrote:

Unit tests: fail. 61858 tests passed, 1 failed and 781 were skipped.

failed: Clang.CodeGen/thinlto-debug-pm.c

I think that failure can be fixed with something like this:

diff --git a/clang/test/CodeGen/thinlto-debug-pm.c b/clang/test/CodeGen/thinlto-debug-pm.c
index 5f449d493af..185a8c8fb8b 100644
--- a/clang/test/CodeGen/thinlto-debug-pm.c
+++ b/clang/test/CodeGen/thinlto-debug-pm.c
@@ -13,17 +13,17 @@
 // O0123sz-NEWPM: Running analysis: PassInstrumentationAnalysis
 // O0123sz-NEWPM: Starting llvm::Module pass manager run.
 // O0123sz-NEWPM: Running pass: WholeProgramDevirtPass
-// O0123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O0123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O0123sz-NEWPM: Running pass: LowerTypeTestsPass
 // O0123sz-NEWPM: Invalidating all non-preserved analyses for:
-// O0123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O0123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running pass: ForceFunctionAttrsPass
 // O123sz-NEWPM: Running pass: PassManager<llvm::Module>
 // O123sz-NEWPM: Starting llvm::Module pass manager run.
 // O123sz-NEWPM: Running pass: PGOIndirectCallPromotion
 // O123sz-NEWPM: Running analysis: ProfileSummaryAnalysis
 // O123sz-NEWPM: Running pass: InferFunctionAttrsPass
-// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running pass: ModuleToFunctionPassAdaptor<llvm::PassManager<llvm::Function> >
 // O123sz-NEWPM: Running analysis: PassInstrumentationAnalysis on foo
 // O123sz-NEWPM: Starting llvm::Function pass manager run.
@@ -41,9 +41,9 @@
 // O123sz-NEWPM: Running pass: CalledValuePropagationPass
 // O123sz-NEWPM: Running pass: GlobalOptPass
 // O123sz-NEWPM: Invalidating all non-preserved analyses for:
-// O123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running pass: ModuleToFunctionPassAdaptor<llvm::PromotePass>
-// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running analysis: DominatorTreeAnalysis on foo
 // O123sz-NEWPM: Running analysis: PassInstrumentationAnalysis on foo
 // O123sz-NEWPM: Running analysis: AssumptionAnalysis on foo
@@ -57,20 +57,20 @@
 // O123sz-NEWPM: Running analysis: BasicAA on foo
 // O123sz-NEWPM: Running analysis: ScopedNoAliasAA on foo
 // O123sz-NEWPM: Running analysis: TypeBasedAA on foo
-// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::AnalysisManager<llvm::Module>, llvm::Function> on foo
+// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::ModuleAnalysisManager, llvm::Function> on foo
 // O123sz-NEWPM: Running pass: SimplifyCFGPass on foo
 // O123sz-NEWPM: Running analysis: TargetIRAnalysis on foo
 // O123sz-NEWPM: Finished llvm::Function pass manager run.
-// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::GlobalsAA, llvm::Module>
+// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::GlobalsAA, llvm::Module, llvm::AnalysisManager<llvm::Module>>
 // O123sz-NEWPM: Running analysis: GlobalsAA
 // O123sz-NEWPM: Running analysis: CallGraphAnalysis
-// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::ProfileSummaryAnalysis, llvm::Module>
-// O123sz-NEWPM: Running pass: ModuleToPostOrderCGSCCPassAdaptor<llvm::DevirtSCCRepeatedPass<llvm::PassManager<llvm::LazyCallGraph::SCC, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&> > >
-// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::Module>
+// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::ProfileSummaryAnalysis, llvm::Module, llvm::AnalysisManager<llvm::Module>>
+// O123sz-NEWPM: Running pass: ModuleToPostOrderCGSCCPassAdaptor<llvm::DevirtSCCRepeatedPass<llvm::PassManager<LazyCallGraph::SCC, llvm::CGSCCAnalysisManager, llvm::LazyCallGraph &, llvm::CGSCCUpdateResult &> > >
+// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::CGSCCAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running analysis: LazyCallGraphAnalysis
 // O123sz-NEWPM: Running analysis: FunctionAnalysisManagerCGSCCProxy on (foo)
 // O123sz-NEWPM: Running analysis: PassInstrumentationAnalysis on (foo)
-// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::AnalysisManager<llvm::Module>, llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&> on (foo)
+// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::ModuleAnalysisManager, LazyCallGraph::SCC, llvm::LazyCallGraph &>
 // O123sz-NEWPM: Starting CGSCC pass manager run.
 // O123sz-NEWPM: Running pass: InlinerPass on (foo)
 // O123sz-NEWPM: Running pass: PostOrderFunctionAttrsPass on (foo)
@@ -91,8 +91,8 @@
 // O23sz-NEWPM: Running pass: TailCallElimPass on foo
 // O123sz-NEWPM: Running pass: SimplifyCFGPass on foo
 // O123sz-NEWPM: Running pass: ReassociatePass on foo
-// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::OptimizationRemarkEmitterAnalysis, llvm::Function> on foo
-// O123sz-NEWPM: Running pass: FunctionToLoopPassAdaptor<llvm::PassManager<llvm::Loop, llvm::AnalysisManager<llvm::Loop, llvm::LoopStandardAnalysisResults&>, llvm::LoopStandardAnalysisResults&, llvm::LPMUpdater&> > on foo
+// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::OptimizationRemarkEmitterAnalysis, llvm::Function, llvm::AnalysisManager<llvm::Function>>
+// O123sz-NEWPM: Running pass: FunctionToLoopPassAdaptor<llvm::PassManager<llvm::Loop, llvm::LoopAnalysisManager, llvm::LoopStandardAnalysisResults &, llvm::LPMUpdater &> >
 // O123sz-NEWPM: Starting llvm::Function pass manager run.
 // O123sz-NEWPM: Running pass: LoopSimplifyPass on foo
 // O123sz-NEWPM: Running analysis: LoopAnalysis on foo
@@ -100,7 +100,7 @@
 // O123sz-NEWPM: Finished llvm::Function pass manager run.
 // O123sz-NEWPM: Running pass: SimplifyCFGPass on foo
 // O123sz-NEWPM: Running pass: InstCombinePass on foo
-// O123sz-NEWPM: Running pass: FunctionToLoopPassAdaptor<llvm::PassManager<llvm::Loop, llvm::AnalysisManager<llvm::Loop, llvm::LoopStandardAnalysisResults&>, llvm::LoopStandardAnalysisResults&, llvm::LPMUpdater&> > on foo
+// O123sz-NEWPM: Running pass: FunctionToLoopPassAdaptor<llvm::PassManager<llvm::Loop, llvm::LoopAnalysisManager, llvm::LoopStandardAnalysisResults &, llvm::LPMUpdater &> >
 // O123sz-NEWPM: Starting llvm::Function pass manager run.
 // O123sz-NEWPM: Running pass: LoopSimplifyPass on foo
 // O123sz-NEWPM: Running pass: LCSSAPass on foo
@@ -137,7 +137,7 @@
 // O123sz-NEWPM: Running pass: GlobalDCEPass
 // O123sz-NEWPM: Running pass: EliminateAvailableExternallyPass
 // O123sz-NEWPM: Running pass: ReversePostOrderFunctionAttrsPass
-// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::GlobalsAA, llvm::Module>
+// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::GlobalsAA, llvm::Module, llvm::AnalysisManager<llvm::Module>>
 // O123sz-NEWPM: Running pass: ModuleToFunctionPassAdaptor<llvm::PassManager<llvm::Function> >
 // O123sz-NEWPM: Starting llvm::Function pass manager run.
 // O123sz-NEWPM: Running pass: Float2IntPass on foo
@@ -149,7 +149,7 @@
 // O123sz-NEWPM: Finished llvm::Function pass manager run.
 // O123sz-NEWPM: Running pass: LoopDistributePass on foo
 // O123sz-NEWPM: Running analysis: ScalarEvolutionAnalysis on foo
-// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Loop, llvm::LoopStandardAnalysisResults&>, llvm::Function> on foo
+// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::LoopAnalysisManager, llvm::Function>
 // O123sz-NEWPM: Running pass: LoopVectorizePass on foo
 // O123sz-NEWPM: Running analysis: BlockFrequencyAnalysis on foo
 // O123sz-NEWPM: Running analysis: BranchProbabilityAnalysis on foo
@@ -160,7 +160,7 @@
 // O123sz-NEWPM: Running pass: LoopUnrollPass on foo
 // O123sz-NEWPM: Running pass: WarnMissedTransformationsPass on foo
 // O123sz-NEWPM: Running pass: InstCombinePass on foo
-// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::OptimizationRemarkEmitterAnalysis, llvm::Function> on foo
+// O123sz-NEWPM: Running pass: RequireAnalysisPass<llvm::OptimizationRemarkEmitterAnalysis, llvm::Function, llvm::AnalysisManager<llvm::Function>> on foo
 // O123sz-NEWPM: Running pass: FunctionToLoopPassAdaptor<llvm::LICMPass> on foo
 // O123sz-NEWPM: Starting llvm::Function pass manager run.
 // O123sz-NEWPM: Running pass: LoopSimplifyPass on foo

EDIT: Had to make more changes than I thought. Above passes the test.

[ychen]

In D72404#1820628, @tycho wrote:

In D72404#1820461, @merge_guards_bot wrote:

Unit tests: fail. 61858 tests passed, 1 failed and 781 were skipped.

failed: Clang.CodeGen/thinlto-debug-pm.c

I think that failure can be fixed with something like this:

diff --git a/clang/test/CodeGen/thinlto-debug-pm.c b/clang/test/CodeGen/thinlto-debug-pm.c
index 5f449d493af..9d6d69afd13 100644
--- a/clang/test/CodeGen/thinlto-debug-pm.c
+++ b/clang/test/CodeGen/thinlto-debug-pm.c
@@ -13,17 +13,17 @@
 // O0123sz-NEWPM: Running analysis: PassInstrumentationAnalysis
 // O0123sz-NEWPM: Starting llvm::Module pass manager run.
 // O0123sz-NEWPM: Running pass: WholeProgramDevirtPass
-// O0123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O0123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O0123sz-NEWPM: Running pass: LowerTypeTestsPass
 // O0123sz-NEWPM: Invalidating all non-preserved analyses for:
-// O0123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O0123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running pass: ForceFunctionAttrsPass
 // O123sz-NEWPM: Running pass: PassManager<llvm::Module>
 // O123sz-NEWPM: Starting llvm::Module pass manager run.
 // O123sz-NEWPM: Running pass: PGOIndirectCallPromotion
 // O123sz-NEWPM: Running analysis: ProfileSummaryAnalysis
 // O123sz-NEWPM: Running pass: InferFunctionAttrsPass
-// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running pass: ModuleToFunctionPassAdaptor<llvm::PassManager<llvm::Function> >
 // O123sz-NEWPM: Running analysis: PassInstrumentationAnalysis on foo
 // O123sz-NEWPM: Starting llvm::Function pass manager run.
@@ -41,9 +41,9 @@
 // O123sz-NEWPM: Running pass: CalledValuePropagationPass
 // O123sz-NEWPM: Running pass: GlobalOptPass
 // O123sz-NEWPM: Invalidating all non-preserved analyses for:
-// O123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O123sz-NEWPM: Invalidating analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running pass: ModuleToFunctionPassAdaptor<llvm::PromotePass>
-// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::AnalysisManager<llvm::Function>, llvm::Module>
+// O123sz-NEWPM: Running analysis: InnerAnalysisManagerProxy<llvm::FunctionAnalysisManager, llvm::Module>
 // O123sz-NEWPM: Running analysis: DominatorTreeAnalysis on foo
 // O123sz-NEWPM: Running analysis: PassInstrumentationAnalysis on foo
 // O123sz-NEWPM: Running analysis: AssumptionAnalysis on foo
@@ -57,7 +57,7 @@
 // O123sz-NEWPM: Running analysis: BasicAA on foo
 // O123sz-NEWPM: Running analysis: ScopedNoAliasAA on foo
 // O123sz-NEWPM: Running analysis: TypeBasedAA on foo
-// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::AnalysisManager<llvm::Module>, llvm::Function> on foo
+// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::ModuleAnalysisManager, llvm::Function> on foo
 // O123sz-NEWPM: Running pass: SimplifyCFGPass on foo
 // O123sz-NEWPM: Running analysis: TargetIRAnalysis on foo
 // O123sz-NEWPM: Finished llvm::Function pass manager run.
@@ -70,7 +70,7 @@
 // O123sz-NEWPM: Running analysis: LazyCallGraphAnalysis
 // O123sz-NEWPM: Running analysis: FunctionAnalysisManagerCGSCCProxy on (foo)
 // O123sz-NEWPM: Running analysis: PassInstrumentationAnalysis on (foo)
-// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::AnalysisManager<llvm::Module>, llvm::LazyCallGraph::SCC, llvm::LazyCallG>
+// O123sz-NEWPM: Running analysis: OuterAnalysisManagerProxy<llvm::ModuleAnalysisManager, llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&> o>
 // O123sz-NEWPM: Starting CGSCC pass manager run.
 // O123sz-NEWPM: Running pass: InlinerPass on (foo)
 // O123sz-NEWPM: Running pass: PostOrderFunctionAttrsPass on (foo)

Thank you. It turns out that GCC&Clang disagree on __PRETTY_FUNCTION__ for type alias (using). will fix those.

[ychen]

• fix test when clang is the host compiler

[merge_guards_bot]

Unit tests: pass. 61859 tests passed, 0 failed and 781 were skipped.

clang-tidy: unknown.

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

[aykevl]

I would be very interested in this patch, because for me to use ThinLTO without size regressions I need to set the optimization size level in the linker (PassManagerBuilder.SizeLevel etc).
This patch seems mostly correct to me, except for the function attributes. These function attributes (optsize, minsize) should IMHO be set in the frontend, not in the linker.

Apart from that, would it be reasonable to implement this in the form of -plugin-opt=Os and -plugin-opt=Oz? That is perhaps a cleaner UI (doesn't need a new flag!) and more cleanly maps to PassBuilder::Oz and the like.

[steven_wu]

I just saw this. I know it is a good idea to have choice during link time for the pipeline configuration and from your benchmark it also has size impact on the output. I also feel like this is going in the wrong direction as if I have part of the object files built with -O3 and part built with -Oz, I need to make a choice of unoptimized part of the program.

Before I say yes or no to this patch, have you figured out what are the passes that causes the most size regression? Ideally, with function attributes on the function, it shouldn't be much size impact on the output.

[mehdi_amini]

In D72404#3304205, @aykevl wrote:

I would be very interested in this patch, because for me to use ThinLTO without size regressions I need to set the optimization size level in the linker (PassManagerBuilder.SizeLevel etc).

Why can't you build the object files with -Os in the first place instead of changing the optimization level between the compilation and the linking phases?

[aykevl]

Apparently there is also another patch that tries to do something very similar: D81223.

In D72404#3305275, @mehdi_amini wrote:

Why can't you build the object files with -Os in the first place instead of changing the optimization level between the compilation and the linking phases?

I'm not changing the optimization level. The bitcode files are built with an equivalent of -Oz and have the optsize and minsize attributes. It's the optimization passes in the linker (ThinLTO) that don't respect these attributes.

In D72404#3305267, @steven_wu wrote:

Before I say yes or no to this patch, have you figured out what are the passes that causes the most size regression? Ideally, with function attributes on the function, it shouldn't be much size impact on the output.

Unfortunately, there is an impact. I did a quick test with a small program (around 4.7kB compiled code) and it looks like the LoopRotate pass is the main culprit. If MaxHeaderSize is set to 0 instead of -1, the code size regression is avoided. The following hacky patch avoids the code size increase:

diff --git a/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp b/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
index aa916345954d..99be1926cf34 100644
--- a/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
+++ b/llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
@@ -450,7 +450,7 @@ void PassManagerBuilder::addFunctionSimplificationPasses(
   // TODO: Investigate promotion cap for O1.
   MPM.add(createLICMPass(LicmMssaOptCap, LicmMssaNoAccForPromotionCap));
   // Rotate Loop - disable header duplication at -Oz
-  MPM.add(createLoopRotatePass(SizeLevel == 2 ? 0 : -1, PrepareForLTO));
+  MPM.add(createLoopRotatePass(0/*SizeLevel == 2 ? 0 : -1*/, PrepareForLTO));
   // TODO: Investigate promotion cap for O1.
   MPM.add(createLICMPass(LicmMssaOptCap, LicmMssaNoAccForPromotionCap));
   if (EnableSimpleLoopUnswitch)
@@ -917,7 +917,7 @@ void PassManagerBuilder::populateModulePassManager(
   // Re-rotate loops in all our loop nests. These may have fallout out of
   // rotated form due to GVN or other transformations, and the vectorizer relies
   // on the rotated form. Disable header duplication at -Oz.
-  MPM.add(createLoopRotatePass(SizeLevel == 2 ? 0 : -1, PrepareForLTO));
+  MPM.add(createLoopRotatePass(0 /*SizeLevel == 2 ? 0 : -1*/, PrepareForLTO));
 
   // Distribute loops to allow partial vectorization.  I.e. isolate dependences
   // into separate loop that would otherwise inhibit vectorization.  This is

So, should all passes just look at the optsize and minsize attributes instead of the SizeLevel? In other words, should PassManagerBuilder.SizeLevel be removed and should passes only look at function attributes instead of SizeLevel? Because at the moment, it's a weird mix of both. IMHO size level should either all go via function attributes or via a flag, not something in between as it is now.
Also, if size level is done via function attributes, why not optimization level? There is already optnone. I'm not saying that's better, but right now I don't see the logic in this whole system.

[aykevl]

After some more testing on a larger amount of code (many small programs, together over 1MB in binary size), LoopRotate indeed seems to be the culprit. I'm now looking into a patch to LoopRotate to respect the optsize function attribute.

EDIT: see D119342.

[mehdi_amini]

In D72404#3307623, @aykevl wrote:

So, should all passes just look at the optsize and minsize attributes instead of the SizeLevel? In other words, should PassManagerBuilder.SizeLevel be removed and should passes only look at function attributes instead of SizeLevel? Because at the moment, it's a weird mix of both. IMHO size level should either all go via function attributes or via a flag, not something in between as it is now.

I agree, I don't know (other than history) why we couldn't move towards removing PassManagerBuilder.SizeLevel?

Also, if size level is done via function attributes, why not optimization level? There is already optnone. I'm not saying that's better, but right now I don't see the logic in this whole system.

Despite what gcc and clang exposes to their users, at the LLVM level we don't have a single dimension on which to put O1/O2/O3 compared to Os/Oz. These also may not make sense for every single compiler out-there: O1/O2/O3 for clang may not be the right pass pipeline for my proprietary shader compiler.
Another reason why O1/O2/O3 are not making much such to be in the IR is that the IR is intended to be stored and reloaded any time in the middle of pipeline. LTO is an example of this, so we don't really want to store the "list of pass to run" in the IR.
Finally, we don't want to (or we can't really...) teach passes about whether they should execute during O1/O2/O3, while optnone is just a "fuse" to disable them all. It is also convenient to have optnone as a function attribute because it allows to selectively disable the optimizer on a per function basis. On the other hand because of the nature of the pass pipeline, it can't be tweaked on a per function basis (what about Module passes?).

So O1/O2/O3 is somehow just like an alias for an arbitrary list of passes (not really arbitrary, it is a good "default" one for clang-style IR), on the other hand the optsize/minsize are driving heuristic and can be orthogonal to the pass pipeline / controlled on a per-function basis and made available to every pass: they convey an "optimization goal" that applies to every pass individually.

[aykevl]

@mehdi_amini thanks for explaining! D119342 moves slightly closer to removing SizeLevel from the pass pipeline setup.

---

In other news, I found a workaround that can be used to avoid the size increase due to LoopRotate (until D119342 is merged). Basically, just pass the flags -mllvm --rotation-max-header-size=0 to ld.lld when compiling with -Oz.

[tejohnson]

In D72404#3310704, @aykevl wrote:

@mehdi_amini thanks for explaining! D119342 moves slightly closer to removing SizeLevel from the pass pipeline setup.

I left a comment on D119342 - I think that is the right way to go. As mentioned there, there is still some legacy handling of options passed down from the driver, but over time we've been trying to move things to use function attributes, for the reasons @mehdi_amini mentioned. To expand on one of the reasons Mehdi mentioned: using function attributes naturally handles LTO linking in the case where one file is compiled -flto -Os and another is compiled -flto -O2 the same way as if you compiled the two files with those different flags all the way down to native code without LTO. Using the approach in this pass you would be forced to pick either -Os or -O2 for both files at LTO link time.

lld/COFF/Config.h

struct Configuration {
   GuardCFLevel guardCF = GuardCFLevel::Off;
 
   // Used for SafeSEH.
   bool safeSEH = false;
   Symbol *sehTable = nullptr;
   Symbol *sehCount = nullptr;
 
   // Used for /opt:lldlto=N
   unsigned ltoo = 2;
+  unsigned ltoos = 0;
 
   // Used for /opt:lldltojobs=N
   unsigned thinLTOJobs = 0;
   // Used for /opt:lldltopartitions=N
   unsigned ltoPartitions = 1;
 
   // Used for /opt:lldltocache=path
   StringRef ltoCache;
   // Used for /opt:lldltocachepolicy=policy
   llvm::CachePruningPolicy ltoCachePolicy;
 

lld/COFF/Driver.cpp

for (StringRef s : vec) {
         icfLevel = 2;
       } else if (s == "noicf") {
         icfLevel = 0;
       } else if (s == "lldtailmerge") {
         tailMerge = 2;
       } else if (s == "nolldtailmerge") {
         tailMerge = 0;
       } else if (s.startswith("lldlto=")) {
         StringRef optLevel = s.substr(7);
-        if (optLevel.getAsInteger(10, config->ltoo) || config->ltoo > 3)
-          error("/opt:lldlto: invalid optimization level: " + optLevel);
+        config->ltoo = CHECK(llvm::lto::getOptLevel(optLevel), "/opt:lldlto: ");
+        config->ltoos = check(llvm::lto::getSizeLevel(optLevel));
       } else if (s.startswith("lldltojobs=")) {
         StringRef jobs = s.substr(11);
         if (jobs.getAsInteger(10, config->thinLTOJobs) ||
             config->thinLTOJobs == 0)
           error("/opt:lldltojobs: invalid job count: " + jobs);
       } else if (s.startswith("lldltopartitions=")) {
         StringRef n = s.substr(17);
         if (n.getAsInteger(10, config->ltoPartitions) ||
             config->ltoPartitions == 0)
           error("/opt:lldltopartitions: invalid partition count: " + n);

lld/COFF/LTO.cpp

static lto::Config createConfig() {
   // using the PIC model (see PR34306).
   if (config->machine == COFF::IMAGE_FILE_MACHINE_I386)
     c.RelocModel = Reloc::Static;
   else
     c.RelocModel = Reloc::PIC_;
   c.DisableVerify = true;
   c.DiagHandler = diagnosticHandler;
   c.OptLevel = config->ltoo;
+  c.SizeLevel = config->ltoos;
   c.CPU = getCPUStr();
   c.MAttrs = getMAttrs();
   c.CGOptLevel = args::getCGOptLevel(config->ltoo);
 
   if (config->saveTemps)
     checkError(c.addSaveTemps(std::string(config->outputFile) + ".",
                               /*UseInputModulePath*/ true));
   return c;

lld/ELF/Config.h

struct Configuration {
   BuildIdKind buildId = BuildIdKind::None;
   SeparateSegmentKind zSeparate;
   ELFKind ekind = ELFNoneKind;
   uint16_t emachine = llvm::ELF::EM_NONE;
   llvm::Optional<uint64_t> imageBase;
   uint64_t commonPageSize;
   uint64_t maxPageSize;
   uint64_t mipsGotSize;
   uint64_t zStackSize;
   unsigned ltoPartitions;
   unsigned ltoo;
+  unsigned ltoos;
   unsigned optimize;
   unsigned thinLTOJobs;
   int32_t splitStackAdjustSize;
 
   // The following config options do not directly correspond to any
   // particular command line options.
 
   // True if we need to pass through relocations in input files to the
   // output file. Usually false because we consume relocations.
   bool copyRelocs;
 

lld/ELF/Driver.cpp

config->ignoreFunctionAddressEquality =
       args.hasArg(OPT_ignore_function_address_equality);
   config->init = args.getLastArgValue(OPT_init, "_init");
   config->ltoAAPipeline = args.getLastArgValue(OPT_lto_aa_pipeline);
   config->ltoCSProfileGenerate = args.hasArg(OPT_lto_cs_profile_generate);
   config->ltoCSProfileFile = args.getLastArgValue(OPT_lto_cs_profile_file);
   config->ltoDebugPassManager = args.hasArg(OPT_lto_debug_pass_manager);
   config->ltoNewPassManager = args.hasArg(OPT_lto_new_pass_manager);
   config->ltoNewPmPasses = args.getLastArgValue(OPT_lto_newpm_passes);
-  config->ltoo = args::getInteger(args, OPT_lto_O, 2);
+  config->ltoo = check(llvm::lto::getOptLevel(args.getLastArg(OPT_lto_O), 2));
+  config->ltoos = check(llvm::lto::getSizeLevel(args.getLastArg(OPT_lto_O), 0));
   config->ltoObjPath = args.getLastArgValue(OPT_lto_obj_path_eq);
   config->ltoPartitions = args::getInteger(args, OPT_lto_partitions, 1);
   config->ltoSampleProfile = args.getLastArgValue(OPT_lto_sample_profile);
   config->mapFile = args.getLastArgValue(OPT_Map);
   config->mipsGotSize = args::getInteger(args, OPT_mips_got_size, 0xfff0);
   config->mergeArmExidx =
       args.hasFlag(OPT_merge_exidx_entries, OPT_no_merge_exidx_entries, true);
   config->mmapOutputFile =
static void readConfigs(opt::InputArgList &args) {
   if (auto *arg = args.getLastArg(OPT_plugin_opt_mcpu_eq))
     parseClangOption(saver.save("-mcpu=" + StringRef(arg->getValue())),
                      arg->getSpelling());
 
   for (auto *arg : args.filtered(OPT_plugin_opt))
     parseClangOption(arg->getValue(), arg->getSpelling());
 
   // Parse -mllvm options.
   for (auto *arg : args.filtered(OPT_mllvm))
     parseClangOption(arg->getValue(), arg->getSpelling());
 
-  if (config->ltoo > 3)
-    error("invalid optimization level for LTO: " + Twine(config->ltoo));
   if (config->ltoPartitions == 0)
     error("--lto-partitions: number of threads must be > 0");
   if (config->thinLTOJobs == 0)
     error("--thinlto-jobs: number of threads must be > 0");
 
   if (config->splitStackAdjustSize < 0)
     error("--split-stack-adjust-size: size must be >= 0");
 

lld/ELF/LTO.cpp

else if (config->isPic)
     c.RelocModel = Reloc::PIC_;
   else
     c.RelocModel = Reloc::Static;
 
   c.CodeModel = getCodeModelFromCMModel();
   c.DisableVerify = config->disableVerify;
   c.DiagHandler = diagnosticHandler;
   c.OptLevel = config->ltoo;
+  c.SizeLevel = config->ltoos;
   c.CPU = getCPUStr();
   c.MAttrs = getMAttrs();
   c.CGOptLevel = args::getCGOptLevel(config->ltoo);
 
   // Set up a custom pipeline if we've been asked to.
   c.OptPipeline = config->ltoNewPmPasses;
   c.AAPipeline = config->ltoAAPipeline;
 

lld/test/COFF/lto-opt-level.ll

 ; REQUIRES: x86
 ; RUN: llvm-as -o %t.obj %s
 ; RUN: lld-link /out:%t0.exe /entry:main /subsystem:console /opt:lldlto=0 /lldmap:%t0.map %t.obj
 ; RUN: FileCheck --check-prefix=CHECK-O0 %s < %t0.map
 ; RUN: lld-link /out:%t2.exe /entry:main /subsystem:console /opt:lldlto=2 /lldmap:%t2.map %t.obj
 ; RUN: FileCheck --check-prefix=CHECK-O2 %s < %t2.map
+; RUN: lld-link /out:%ts.exe /entry:main /subsystem:console /opt:lldlto=s /lldmap:%ts.map %t.obj
+; RUN: FileCheck --check-prefix=CHECK-O2 %s < %ts.map
+; RUN: lld-link /out:%tz.exe /entry:main /subsystem:console /opt:lldlto=z /lldmap:%tz.map %t.obj
+; RUN: FileCheck --check-prefix=CHECK-O2 %s < %tz.map
 ; RUN: lld-link /out:%t2a.exe /entry:main /subsystem:console /lldmap:%t2a.map %t.obj
 ; RUN: FileCheck --check-prefix=CHECK-O2 %s < %t2a.map
 
 target datalayout = "e-m:w-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-windows-msvc"
 
 ; CHECK-O0: foo
 ; CHECK-O2-NOT: foo
 define internal void @foo() {
   ret void
 }
 
 define void @main() {
   call void @foo()
   ret void
 }

lld/test/ELF/lto/opt-level.ll

 ; REQUIRES: x86
 ; RUN: llvm-as -o %t.o %s
 ; RUN: ld.lld -o %t0 -e main --lto-O0 %t.o
 ; RUN: llvm-nm %t0 | FileCheck --check-prefix=CHECK-O0 %s
 ; RUN: ld.lld -o %t0 -e main --plugin-opt=O0 %t.o
 ; RUN: llvm-nm %t0 | FileCheck --check-prefix=CHECK-O0 %s
 ; RUN: ld.lld -o %t2 -e main --lto-O2 %t.o
 ; RUN: llvm-nm %t2 | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: ld.lld -o %ts -e main --lto-Os %t.o
+; RUN: llvm-nm %ts | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: ld.lld -o %tz -e main --lto-Oz %t.o
+; RUN: llvm-nm %tz | FileCheck --check-prefix=CHECK-O2 %s
 ; RUN: ld.lld -o %t2a -e main %t.o
 ; RUN: llvm-nm %t2a | FileCheck --check-prefix=CHECK-O2 %s
 ; RUN: ld.lld -o %t2 -e main %t.o --plugin-opt O2
 ; RUN: llvm-nm %t2 | FileCheck --check-prefix=CHECK-O2 %s
 
 ; Reject invalid optimization levels.
 ; RUN: not ld.lld -o %t3 -e main --lto-O6 %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALID1 %s
 ; INVALID1: invalid optimization level for LTO: 6
 ; RUN: not ld.lld -o %t3 -e main --plugin-opt=O6 %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALID1 %s
 ; RUN: not ld.lld -o %t3 -e main --plugin-opt=Ofoo %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALID2 %s
-; INVALID2: --plugin-opt=Ofoo: number expected, but got 'foo'
+; INVALID2: invalid optimization level for LTO: foo
 
 ; RUN: not ld.lld -o %t3 -e main --lto-O-1 %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALIDNEGATIVE1 %s
-; INVALIDNEGATIVE1: invalid optimization level for LTO: 4294967295
+; INVALIDNEGATIVE1: invalid optimization level for LTO: -1
 ; RUN: not ld.lld -o %t3 -e main --plugin-opt=O-1 %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALIDNEGATIVE2 %s
-; INVALIDNEGATIVE2: invalid optimization level for LTO: 4294967295
+; INVALIDNEGATIVE2: invalid optimization level for LTO: -1
 
 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; CHECK-O0: foo
 ; CHECK-O2-NOT: foo
 define internal void @foo() {
   ret void
 }
 
 define void @main() {
   call void @foo()
   ret void
 }

lld/test/wasm/lto/opt-level.ll

 ; RUN: llvm-as -o %t.o %s
 ; RUN: wasm-ld -o %t0 -e main --lto-O0 %t.o
 ; RUN: obj2yaml %t0 | FileCheck --check-prefix=CHECK-O0 %s
 ; RUN: wasm-ld -o %t2 -e main --lto-O2 %t.o
 ; RUN: obj2yaml %t2 | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: wasm-ld -o %ts -e main --lto-Os %t.o
+; RUN: obj2yaml %ts | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: wasm-ld -o %tz -e main --lto-Oz %t.o
+; RUN: obj2yaml %tz | FileCheck --check-prefix=CHECK-O2 %s
 ; RUN: wasm-ld -o %t2a -e main %t.o
 ; RUN: obj2yaml %t2a | FileCheck --check-prefix=CHECK-O2 %s
 
 ; Reject invalid optimization levels.
 ; RUN: not wasm-ld -o %t3 -e main --lto-O6 %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALID %s
 ; INVALID: invalid optimization level for LTO: 6
 
 ; RUN: not wasm-ld -o %t3 -m elf_x86_64 -e main --lto-O-1 %t.o 2>&1 | \
 ; RUN:   FileCheck --check-prefix=INVALIDNEGATIVE %s
-; INVALIDNEGATIVE: invalid optimization level for LTO: 4294967295
+; INVALIDNEGATIVE: invalid optimization level for LTO: -1
 
 target datalayout = "e-m:e-p:32:32-i64:64-n32:64-S128"
 target triple = "wasm32-unknown-unknown-wasm"
 
 ; CHECK-O0: Name: foo
 ; CHECK-O2-NOT: Name: foo
 define internal void @foo() {
   ret void
 }
 
 define void @main() {
   call void @foo()
   ret void
 }

lld/wasm/Config.h

struct Configuration {
   bool shared;
   bool stripAll;
   bool stripDebug;
   bool stackFirst;
   bool trace;
   uint32_t globalBase;
   uint32_t initialMemory;
   uint32_t maxMemory;
   uint32_t zStackSize;
   unsigned ltoPartitions;
   unsigned ltoo;
+  unsigned ltoos;
   unsigned optimize;
   unsigned thinLTOJobs;
 
   llvm::StringRef entry;
   llvm::StringRef outputFile;
   llvm::StringRef thinLTOCacheDir;
 
   llvm::StringSet<> allowUndefinedSymbols;
   llvm::StringSet<> exportedSymbols;
   std::vector<llvm::StringRef> searchPaths;

lld/wasm/Driver.cpp

static void readConfigs(opt::InputArgList &args) {
   config->exportAll = args.hasArg(OPT_export_all);
   config->exportTable = args.hasArg(OPT_export_table);
   config->growableTable = args.hasArg(OPT_growable_table);
   errorHandler().fatalWarnings =
       args.hasFlag(OPT_fatal_warnings, OPT_no_fatal_warnings, false);
   config->importMemory = args.hasArg(OPT_import_memory);
   config->sharedMemory = args.hasArg(OPT_shared_memory);
   config->importTable = args.hasArg(OPT_import_table);
-  config->ltoo = args::getInteger(args, OPT_lto_O, 2);
+  config->ltoo = check(llvm::lto::getOptLevel(args.getLastArg(OPT_lto_O), 2));
+  config->ltoos = check(llvm::lto::getSizeLevel(args.getLastArg(OPT_lto_O), 0));
   config->ltoPartitions = args::getInteger(args, OPT_lto_partitions, 1);
   config->optimize = args::getInteger(args, OPT_O, 0);
   config->outputFile = args.getLastArgValue(OPT_o);
   config->relocatable = args.hasArg(OPT_relocatable);
   config->gcSections =
       args.hasFlag(OPT_gc_sections, OPT_no_gc_sections, !config->relocatable);
   config->mergeDataSegments =
       args.hasFlag(OPT_merge_data_segments, OPT_no_merge_data_segments,

 
 // Some command line options or some combinations of them are not allowed.
 // This function checks for such errors.
 static void checkOptions(opt::InputArgList &args) {
   if (!config->stripDebug && !config->stripAll && config->compressRelocations)
     error("--compress-relocations is incompatible with output debug"
           " information. Please pass --strip-debug or --strip-all");
 
-  if (config->ltoo > 3)
-    error("invalid optimization level for LTO: " + Twine(config->ltoo));
   if (config->ltoPartitions == 0)
     error("--lto-partitions: number of threads must be > 0");
   if (config->thinLTOJobs == 0)
     error("--thinlto-jobs: number of threads must be > 0");
 
   if (config->pie && config->shared)
     error("-shared and -pie may not be used together");
 

lld/wasm/LTO.cpp

static std::unique_ptr<lto::LTO> createLTO() {
 
   // Always emit a section per function/data with LTO.
   c.Options.FunctionSections = true;
   c.Options.DataSections = true;
 
   c.DisableVerify = config->disableVerify;
   c.DiagHandler = diagnosticHandler;
   c.OptLevel = config->ltoo;
+  c.SizeLevel = config->ltoos;
   c.MAttrs = getMAttrs();
   c.CGOptLevel = args::getCGOptLevel(config->ltoo);
 
   if (config->relocatable)
     c.RelocModel = None;
   else if (config->isPic)
     c.RelocModel = Reloc::PIC_;
   else

llvm/include/llvm/LTO/Config.h

struct Config {
   std::string CPU;
   TargetOptions Options;
   std::vector<std::string> MAttrs;
   Optional<Reloc::Model> RelocModel = Reloc::PIC_;
   Optional<CodeModel::Model> CodeModel = None;
   CodeGenOpt::Level CGOptLevel = CodeGenOpt::Default;
   CodeGenFileType CGFileType = CGFT_ObjectFile;
   unsigned OptLevel = 2;
+  unsigned SizeLevel = 0;
   bool DisableVerify = false;
 
   /// Use the new pass manager
   bool UseNewPM = false;
 
   /// Flag to indicate that the optimizer should not assume builtins are present
   /// on the target.
   bool Freestanding = false;

llvm/include/llvm/LTO/LTO.h

 namespace llvm {
 
 class BitcodeModule;
 class Error;
 class LLVMContext;
 class MemoryBufferRef;
 class Module;
 class Target;
 class raw_pwrite_stream;
+namespace opt {
+class Arg;
+}
 
 /// Resolve linkage for prevailing symbols in the \p Index. Linkage changes
 /// recorded in the index and the ThinLTO backends must apply the changes to
 /// the module via thinLTOResolvePrevailingInModule.
 ///
 /// This is done for correctness (if value exported, ensure we always
 /// emit a copy), and compile-time optimization (allow drop of duplicates).
 void thinLTOResolvePrevailingInIndex(
struct SymbolResolution {
   /// The definition of this symbol is visible outside of the LTO unit.
   unsigned VisibleToRegularObj : 1;
 
   /// Linker redefined version of the symbol which appeared in -wrap or -defsym
   /// linker option.
   unsigned LinkerRedefined : 1;
 };
 
+class LTOOLevelError : public ErrorInfo<LTOOLevelError> {
+public:
+  static char ID;
+  StringRef Level;
+
+  LTOOLevelError(StringRef Level) : Level(Level) {}
+
+  void log(raw_ostream &OS) const override {
+    OS << "invalid optimization level for LTO: " << Level;
+  }
+
+  std::error_code convertToErrorCode() const override {
+    return llvm::inconvertibleErrorCode();
+  }
+};
+
+Expected<unsigned> getOptLevel(StringRef OArg);
+Expected<unsigned> getOptLevel(llvm::opt::Arg *OArg, unsigned Default);
+Expected<unsigned> getSizeLevel(StringRef OArg);
+Expected<unsigned> getSizeLevel(llvm::opt::Arg *OArg, unsigned Default);
+
 } // namespace lto
 } // namespace llvm
 
 #endif

llvm/lib/LTO/LTO.cpp

 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Mangler.h"
 #include "llvm/IR/Metadata.h"
 #include "llvm/IR/RemarkStreamer.h"
 #include "llvm/LTO/LTOBackend.h"
 #include "llvm/LTO/SummaryBasedOptimizations.h"
 #include "llvm/Linker/IRMover.h"
 #include "llvm/Object/IRObjectFile.h"
+#include "llvm/Option/Arg.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SHA1.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Support/TargetRegistry.h"
else
     AddUnsigned(-1);
   if (Conf.CodeModel)
     AddUnsigned(*Conf.CodeModel);
   else
     AddUnsigned(-1);
   AddUnsigned(Conf.CGOptLevel);
   AddUnsigned(Conf.CGFileType);
   AddUnsigned(Conf.OptLevel);
+  AddUnsigned(Conf.SizeLevel);
   AddUnsigned(Conf.UseNewPM);
   AddUnsigned(Conf.Freestanding);
   AddString(Conf.OptPipeline);
   AddString(Conf.AAPipeline);
   AddString(Conf.OverrideTriple);
   AddString(Conf.DefaultTriple);
   AddString(Conf.DwoDir);
 
lto::setupStatsFile(StringRef StatsFilename) {
   auto StatsFile =
       std::make_unique<ToolOutputFile>(StatsFilename, EC, sys::fs::OF_None);
   if (EC)
     return errorCodeToError(EC);
 
   StatsFile->keep();
   return std::move(StatsFile);
 }
+
+char LTOOLevelError::ID;
+
+Expected<unsigned> lto::getOptLevel(StringRef S) {
+  if (S == "s" || S == "z" || S.empty())
+    return 2;
+
+  if (S == "g")
+    return 1;
+
+  int Res;
+  if (S.getAsInteger(10, Res) || Res < 0 || Res > 3)
+    return make_error<LTOOLevelError>(S);
+  return Res;
+}
+
+Expected<unsigned> lto::getOptLevel(opt::Arg *OArg, unsigned Default) {
+  if (!OArg)
+    return Default;
+  return lto::getOptLevel(OArg->getValue());
+}
+
+Expected<unsigned> lto::getSizeLevel(StringRef S) {
+  if (S[0] == 's')
+    return 1;
+  if (S[0] == 'z')
+    return 2;
+  if (S[0] >= '0' || S[0] <= '3' )
+    return 0;
+  return make_error<LTOOLevelError>(S);
+}
+
+Expected<unsigned> lto::getSizeLevel(opt::Arg *OArg, unsigned Default) {
+  if (!OArg)
+    return Default;
+  return lto::getSizeLevel(OArg->getValue());
+}

llvm/lib/LTO/LTOBackend.cpp

createTargetMachine(Config &Conf, const Target *TheTarget, Module &M) {
   else
     CodeModel = M.getCodeModel();
 
   return std::unique_ptr<TargetMachine>(TheTarget->createTargetMachine(
       TheTriple, Conf.CPU, Features.getString(), Conf.Options, RelocModel,
       CodeModel, Conf.CGOptLevel));
 }
 
+static PassBuilder::OptimizationLevel mapToLevel(const Config &Conf) {
+  switch (Conf.OptLevel) {
+  default:
+    llvm_unreachable("Invalid optimization level!");
+
+  case 0:
+    return PassBuilder::O0;
+
+  case 1:
+    return PassBuilder::O1;
+
+  case 2:
+    switch (Conf.SizeLevel) {
+    default:
+      llvm_unreachable("Invalid optimization level for size!");
+
+    case 0:
+      return PassBuilder::O2;
+
+    case 1:
+      return PassBuilder::Os;
+
+    case 2:
+      return PassBuilder::Oz;
+    }
+
+  case 3:
+    return PassBuilder::O3;
+  }
+}
+
 static void runNewPMPasses(Config &Conf, Module &Mod, TargetMachine *TM,
-                           unsigned OptLevel, bool IsThinLTO,
-                           ModuleSummaryIndex *ExportSummary,
+                           bool IsThinLTO, ModuleSummaryIndex *ExportSummary,
                            const ModuleSummaryIndex *ImportSummary) {
   Optional<PGOOptions> PGOOpt;
   if (!Conf.SampleProfile.empty())
     PGOOpt = PGOOptions(Conf.SampleProfile, "", Conf.ProfileRemapping,
                         PGOOptions::SampleUse, PGOOptions::NoCSAction, true);
   else if (Conf.RunCSIRInstr) {
     PGOOpt = PGOOptions("", Conf.CSIRProfile, Conf.ProfileRemapping,
                         PGOOptions::IRUse, PGOOptions::CSIRInstr);
static void runNewPMPasses(Config &Conf, Module &Mod, TargetMachine *TM,
   PB.registerCGSCCAnalyses(CGAM);
   PB.registerFunctionAnalyses(FAM);
   PB.registerLoopAnalyses(LAM);
   PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
 
   ModulePassManager MPM(Conf.DebugPassManager);
   // FIXME (davide): verify the input.
 
-  PassBuilder::OptimizationLevel OL;
-
-  switch (OptLevel) {
-  default:
-    llvm_unreachable("Invalid optimization level");
-  case 0:
-    OL = PassBuilder::O0;
-    break;
-  case 1:
-    OL = PassBuilder::O1;
-    break;
-  case 2:
-    OL = PassBuilder::O2;
-    break;
-  case 3:
-    OL = PassBuilder::O3;
-    break;
-  }
+  PassBuilder::OptimizationLevel OL = mapToLevel(Conf);
 
   if (IsThinLTO)
     MPM = PB.buildThinLTODefaultPipeline(OL, Conf.DebugPassManager,
                                          ImportSummary);
   else
     MPM = PB.buildLTODefaultPipeline(OL, Conf.DebugPassManager, ExportSummary);
   MPM.run(Mod, MAM);
 
static void runOldPMPasses(Config &Conf, Module &Mod, TargetMachine *TM,
   PMB.ImportSummary = ImportSummary;
   // Unconditionally verify input since it is not verified before this
   // point and has unknown origin.
   PMB.VerifyInput = true;
   PMB.VerifyOutput = !Conf.DisableVerify;
   PMB.LoopVectorize = true;
   PMB.SLPVectorize = true;
   PMB.OptLevel = Conf.OptLevel;
+  PMB.SizeLevel = Conf.SizeLevel;
   PMB.PGOSampleUse = Conf.SampleProfile;
   PMB.EnablePGOCSInstrGen = Conf.RunCSIRInstr;
   if (!Conf.RunCSIRInstr && !Conf.CSIRProfile.empty()) {
     PMB.EnablePGOCSInstrUse = true;
     PMB.PGOInstrUse = Conf.CSIRProfile;
   }
   if (IsThinLTO)
     PMB.populateThinLTOPassManager(passes);
   else
     PMB.populateLTOPassManager(passes);
   passes.run(Mod);
 }
 
-bool opt(Config &Conf, TargetMachine *TM, unsigned Task, Module &Mod,
+bool optimize(Config &Conf, TargetMachine *TM, unsigned Task, Module &Mod,
          bool IsThinLTO, ModuleSummaryIndex *ExportSummary,
          const ModuleSummaryIndex *ImportSummary) {
   // FIXME: Plumb the combined index into the new pass manager.
   if (!Conf.OptPipeline.empty())
     runNewPMCustomPasses(Mod, TM, Conf.OptPipeline, Conf.AAPipeline,
                          Conf.DisableVerify);
-  else if (Conf.UseNewPM)
-    runNewPMPasses(Conf, Mod, TM, Conf.OptLevel, IsThinLTO, ExportSummary,
-                   ImportSummary);
-  else
-    runOldPMPasses(Conf, Mod, TM, IsThinLTO, ExportSummary, ImportSummary);
+  else {
+    auto runPasses = Conf.UseNewPM ? runNewPMPasses : runOldPMPasses;
+    runPasses(Conf, Mod, TM, IsThinLTO, ExportSummary, ImportSummary);
+  }
   return !Conf.PostOptModuleHook || Conf.PostOptModuleHook(Task, Mod);
 }
 
 void codegen(Config &Conf, TargetMachine *TM, AddStreamFn AddStream,
              unsigned Task, Module &Mod) {
   if (Conf.PreCodeGenModuleHook && !Conf.PreCodeGenModuleHook(Task, Mod))
     return;
 
Error lto::backend(Config &C, AddStreamFn AddStream,
   auto DiagFileOrErr = lto::setupOptimizationRemarks(
       Mod->getContext(), C.RemarksFilename, C.RemarksPasses, C.RemarksFormat,
       C.RemarksWithHotness);
   if (!DiagFileOrErr)
     return DiagFileOrErr.takeError();
   auto DiagnosticOutputFile = std::move(*DiagFileOrErr);
 
   if (!C.CodeGenOnly) {
-    if (!opt(C, TM.get(), 0, *Mod, /*IsThinLTO=*/false,
+    if (!optimize(C, TM.get(), 0, *Mod, /*IsThinLTO=*/false,
              /*ExportSummary=*/&CombinedIndex, /*ImportSummary=*/nullptr))
       return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
   }
 
   if (ParallelCodeGenParallelismLevel == 1) {
     codegen(C, TM.get(), AddStream, 0, *Mod);
   } else {
     splitCodeGen(C, TM.get(), AddStream, ParallelCodeGenParallelismLevel,
                  std::move(Mod));
Error lto::thinBackend(Config &Conf, unsigned Task, AddStreamFn AddStream,
 
   FunctionImporter Importer(CombinedIndex, ModuleLoader);
   if (Error Err = Importer.importFunctions(Mod, ImportList).takeError())
     return Err;
 
   if (Conf.PostImportModuleHook && !Conf.PostImportModuleHook(Task, Mod))
     return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
 
-  if (!opt(Conf, TM.get(), Task, Mod, /*IsThinLTO=*/true,
+  if (!optimize(Conf, TM.get(), Task, Mod, /*IsThinLTO=*/true,
            /*ExportSummary=*/nullptr, /*ImportSummary=*/&CombinedIndex))
     return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
 
   codegen(Conf, TM.get(), AddStream, Task, Mod);
   return finalizeOptimizationRemarks(std::move(DiagnosticOutputFile));
 }

llvm/test/tools/gold/X86/opt-level.ll

 ; RUN: llvm-as -o %t.bc %s
 ; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
 ; RUN:    -m elf_x86_64 \
 ; RUN:    -plugin-opt=O0 -r -o %t.o %t.bc
 ; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O0 %s
 ; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
 ; RUN:    -m elf_x86_64 \
 ; RUN:    -plugin-opt=O1 -r -o %t.o %t.bc
 ; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O1 --check-prefix=CHECK-O1-OLDPM %s
 ; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
 ; RUN:    -m elf_x86_64 \
 ; RUN:    -plugin-opt=O2 -r -o %t.o %t.bc
 ; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
+; RUN:    -m elf_x86_64 \
+; RUN:    -plugin-opt=Os -r -o %t.o %t.bc
+; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
+; RUN:    -m elf_x86_64 \
+; RUN:    -plugin-opt=Oz -r -o %t.o %t.bc
+; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O2 %s
 
 ; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
 ; RUN:    -m elf_x86_64 --plugin-opt=new-pass-manager \
 ; RUN:    -plugin-opt=O0 -r -o %t.o %t.bc
 ; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O0 %s
 ; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
 ; RUN:    -m elf_x86_64 --plugin-opt=new-pass-manager \
 ; RUN:    -plugin-opt=O1 -r -o %t.o %t.bc
 ; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O1 --check-prefix=CHECK-O1-NEWPM %s
 ; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
 ; RUN:    -m elf_x86_64 --plugin-opt=new-pass-manager \
 ; RUN:    -plugin-opt=O2 -r -o %t.o %t.bc
 ; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
+; RUN:    -m elf_x86_64 --plugin-opt=new-pass-manager \
+; RUN:    -plugin-opt=Os -r -o %t.o %t.bc
+; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: %gold -plugin %llvmshlibdir/LLVMgold%shlibext -plugin-opt=save-temps \
+; RUN:    -m elf_x86_64 --plugin-opt=new-pass-manager \
+; RUN:    -plugin-opt=Oz -r -o %t.o %t.bc
+; RUN: llvm-dis < %t.o.0.4.opt.bc -o - | FileCheck --check-prefix=CHECK-O2 %s
 
 ; CHECK-O0: define internal void @foo(
 ; CHECK-O1: define internal void @foo(
 ; CHECK-O2-NOT: define internal void @foo(
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 

llvm/test/tools/lto/opt-level.ll

 ; RUN: llvm-as %s -o %t.o
 ; RUN: %ld64 -lto_library %llvmshlibdir/libLTO.dylib -arch x86_64 -dylib -mllvm -O0 -o %t.dylib %t.o
 ; RUN: llvm-nm --no-llvm-bc %t.dylib | FileCheck --check-prefix=CHECK-O0 %s
 ; RUN: %ld64 -lto_library %llvmshlibdir/libLTO.dylib -arch x86_64 -dylib -mllvm -O2 -o %t.dylib %t.o
 ; RUN: llvm-nm --no-llvm-bc %t.dylib | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: %ld64 -lto_library %llvmshlibdir/libLTO.dylib -arch x86_64 -dylib -mllvm -Os -o %t.dylib %t.o
+; RUN: llvm-nm --no-llvm-bc %t.dylib | FileCheck --check-prefix=CHECK-O2 %s
+; RUN: %ld64 -lto_library %llvmshlibdir/libLTO.dylib -arch x86_64 -dylib -mllvm -Oz -o %t.dylib %t.o
+; RUN: llvm-nm --no-llvm-bc %t.dylib | FileCheck --check-prefix=CHECK-O2 %s
 
 target triple = "x86_64-apple-macosx10.8.0"
 
 ; CHECK-O0: t _f1
 ; CHECK-O2-NOT: _f1
 define internal void @f1() {
   ret void
 }
 
 ; CHECK-O0: T _f2
 ; CHECK-O2: T _f2
 define void @f2() {
   call void @f1()
   ret void
 }

llvm/tools/gold/gold-plugin.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/raw_ostream.h"
 #include <list>
 #include <map>
 #include <plugin-api.h>
 #include <string>
 #include <system_error>
 #include <utility>
 #include <vector>
 
 // FIXME: remove this declaration when we stop maintaining Ubuntu Quantal and
 // Precise and Debian Wheezy (binutils 2.23 is required)
 #define LDPO_PIE 3
enum OutputType {
     OT_NORMAL,
     OT_DISABLE,
     OT_BC_ONLY,
     OT_ASM_ONLY,
     OT_SAVE_TEMPS
   };
   static OutputType TheOutputType = OT_NORMAL;
   static unsigned OptLevel = 2;
+  static unsigned SizeLevel = 0;
   // Default parallelism of 0 used to indicate that user did not specify.
   // Actual parallelism default value depends on implementation.
   // Currently only affects ThinLTO, where the default is
   // llvm::heavyweight_hardware_concurrency.
   static unsigned Parallelism = 0;
   // Default regular LTO codegen parallelism (number of partitions).
   static unsigned ParallelCodeGenParallelismLevel = 1;
 #ifdef NDEBUG
if (opt.startswith("mcpu=")) {
       if (thinlto_object_suffix_replace.find(';') == std::string::npos)
         message(LDPL_FATAL,
                 "thinlto-object-suffix-replace expects 'old;new' format");
     } else if (opt.startswith("cache-dir=")) {
       cache_dir = opt.substr(strlen("cache-dir="));
     } else if (opt.startswith("cache-policy=")) {
       cache_policy = opt.substr(strlen("cache-policy="));
     } else if (opt.size() == 2 && opt[0] == 'O') {
-      if (opt[1] < '0' || opt[1] > '3')
-        message(LDPL_FATAL, "Optimization level must be between 0 and 3");
-      OptLevel = opt[1] - '0';
+      if (auto LevelOrErr = llvm::lto::getOptLevel(opt.substr(1, 1)))
+        OptLevel = *LevelOrErr;
+      else
+        message(LDPL_FATAL, toString(LevelOrErr.takeError()).c_str());
+
+      if (auto LevelOrErr = llvm::lto::getSizeLevel(opt.substr(1, 1)))
+        SizeLevel = *LevelOrErr;
+      else
+        message(LDPL_FATAL, toString(LevelOrErr.takeError()).c_str());
     } else if (opt.startswith("jobs=")) {
       if (StringRef(opt_ + 5).getAsInteger(10, Parallelism))
         message(LDPL_FATAL, "Invalid parallelism level: %s", opt_ + 5);
     } else if (opt.startswith("lto-partitions=")) {
       if (opt.substr(strlen("lto-partitions="))
               .getAsInteger(10, ParallelCodeGenParallelismLevel))
         message(LDPL_FATAL, "Invalid codegen partition level: %s", opt_ + 5);
     } else if (opt == "disable-verify") {
if (DataSections.getNumOccurrences() == 0)
     Conf.Options.DataSections = SplitSections;
 
   Conf.MAttrs = MAttrs;
   Conf.RelocModel = RelocationModel;
   Conf.CodeModel = getCodeModel();
   Conf.CGOptLevel = getCGOptLevel();
   Conf.DisableVerify = options::DisableVerify;
   Conf.OptLevel = options::OptLevel;
+  Conf.SizeLevel = options::SizeLevel;
   if (options::Parallelism)
     Backend = createInProcessThinBackend(options::Parallelism);
   if (options::thinlto_index_only) {
     std::string OldPrefix, NewPrefix;
     getThinLTOOldAndNewPrefix(OldPrefix, NewPrefix);
     Backend = createWriteIndexesThinBackend(OldPrefix, NewPrefix,
                                             options::thinlto_emit_imports_files,
                                             LinkedObjectsFile, OnIndexWrite);

llvm/tools/llvm-lto2/llvm-lto2.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/Threading.h"
 
 using namespace llvm;
 using namespace lto;
 
-static cl::opt<char>
-    OptLevel("O", cl::desc("Optimization level. [-O0, -O1, -O2, or -O3] "
-                           "(default = '-O2')"),
-             cl::Prefix, cl::ZeroOrMore, cl::init('2'));
+static cl::opt<std::string>
+    OptLevel("O",
+             cl::desc("Optimization level. [-O0, -O1, -O2, -Os, -Oz, or "
+                      "-O3] (default = '-O2')"),
+             cl::Prefix, cl::ZeroOrMore, cl::init("2"));
 
 static cl::opt<char> CGOptLevel(
     "cg-opt-level",
     cl::desc("Codegen optimization level (0, 1, 2 or 3, default = '2')"),
     cl::init('2'));
 
 static cl::list<std::string> InputFilenames(cl::Positional, cl::OneOrMore,
                                             cl::desc("<input bitcode files>"));
static int run(int argc, char **argv) {
   Conf.SampleProfile = SamplePGOFile;
   Conf.CSIRProfile = CSPGOFile;
   Conf.RunCSIRInstr = RunCSIRInstr;
 
   // Run a custom pipeline, if asked for.
   Conf.OptPipeline = OptPipeline;
   Conf.AAPipeline = AAPipeline;
 
-  Conf.OptLevel = OptLevel - '0';
+  Conf.OptLevel =
+      check(llvm::lto::getOptLevel(OptLevel), "invalid optimization level");
+  Conf.SizeLevel =
+      check(llvm::lto::getSizeLevel(OptLevel), "invalid optimization level");
   Conf.UseNewPM = UseNewPM;
   switch (CGOptLevel) {
   case '0':
     Conf.CGOptLevel = CodeGenOpt::None;
     break;
   case '1':
     Conf.CGOptLevel = CodeGenOpt::Less;
     break;