Add `-f[no-]split-cold-code` toggling outlining & enable in -O

[compnerd]

Summary

In order for the hot/cold splitting pass to graduate out of
experimental status, users need some way to safely enable it.

The current method of passing -mllvm -hot-cold-split=true to clang is
not safe, because directly setting a cl::opt cannot interact well with
other CC1 options (e.g. -O0, or -disable-llvm-passes).

This patch adds -f[no-]split-cold-code CC1 options to clang so that the
splitting pass can be toggled/combined with other options without issue.
This makes it possible to deploy the new pass on a large scale.

Patch by Vedant Kumar <[e-mail]>!

This is extracted from the patches from the Swift contributed changes to
LLVM at https://github.com/llvm/llvm-project-staging/.

The few test changes here are to allow the tests to continue working
unperturbed.

This has been shipping with the Apple toolchain and has been tested.
The known deficiencies of the outlining here are the incorrect handling
for Windows EH unwind pads which are sometimes duplicately outlined
incorrectly.

[vsk]

@compnerd thank you for working on upstreaming this patch! Would you be open to commandeering D57265 instead? This is my (unfortunately stalled) attempt to upstream the same patch, and it has some review concerns from Fedor Sergeev, Philip Pfaffe, and others addressed.

[compnerd]

Sure, I think I can do that instead. Thanks for pointing me to that!

clang/include/clang/Basic/CodeGenOptions.def

CODEGENOPT(EnableSplitLTOUnit, 1, 0) ///< Enable LTO unit splitting to support
              /// CFI and traditional whole program
              /// devirtualization that require whole
              /// program IR support.
 CODEGENOPT(IncrementalLinkerCompatible, 1, 0) ///< Emit an object file which can
                                               ///< be used with an incremental
                                               ///< linker.
 CODEGENOPT(MergeAllConstants , 1, 1) ///< Merge identical constants.
 CODEGENOPT(MergeFunctions    , 1, 0) ///< Set when -fmerge-functions is enabled.
+CODEGENOPT(SplitColdCode     , 1, 0) ///< Set when -fsplit-cold-code is enabled.
 CODEGENOPT(MemProf           , 1, 0) ///< Set when -fmemory-profile is enabled.
 CODEGENOPT(MSVolatile        , 1, 0) ///< Set when /volatile:ms is enabled.
 CODEGENOPT(NoCommon          , 1, 0) ///< Set when -fno-common or C++ is enabled.
 CODEGENOPT(NoDwarfDirectoryAsm , 1, 0) ///< Set when -fno-dwarf-directory-asm is
                                        ///< enabled.
 CODEGENOPT(NoExecStack       , 1, 0) ///< Set when -Wa,--noexecstack is enabled.
 CODEGENOPT(FatalWarnings     , 1, 0) ///< Set when -Wa,--fatal-warnings is
                                      ///< enabled.

clang/include/clang/Driver/Options.td

 def fforbid_guard_variables : Flag<["-"], "fforbid-guard-variables">,
   HelpText<"Emit an error if a C++ static local initializer would need a guard variable">;
 def no_implicit_float : Flag<["-"], "no-implicit-float">,
   HelpText<"Don't generate implicit floating point instructions">;
 def fdump_vtable_layouts : Flag<["-"], "fdump-vtable-layouts">,
   HelpText<"Dump the layouts of all vtables that will be emitted in a translation unit">;
 def fmerge_functions : Flag<["-"], "fmerge-functions">,
   HelpText<"Permit merging of identical functions when optimizing.">;
+def fsplit_cold_code : Flag<["-"], "fsplit-cold-code">,
+  HelpText<"Permit splitting of cold code when optimizing (off by default).">;
+def fno_split_cold_code : Flag<["-"], "fno-split-cold-code">,
+  HelpText<"Disable splitting of cold code when optimizing.">;
 def femit_coverage_notes : Flag<["-"], "femit-coverage-notes">,
   HelpText<"Emit a gcov coverage notes file when compiling.">;
 def femit_coverage_data: Flag<["-"], "femit-coverage-data">,
   HelpText<"Instrument the program to emit gcov coverage data when run.">;
 def coverage_data_file : Separate<["-"], "coverage-data-file">,
   HelpText<"Emit coverage data to this filename.">;
 def coverage_data_file_EQ : Joined<["-"], "coverage-data-file=">,
   Alias<coverage_data_file>;

clang/lib/CodeGen/BackendUtil.cpp

void EmitAssemblyHelper::CreatePasses(legacy::PassManager &MPM,
   // non-integrated assemblers don't recognize .cgprofile section.
   PMBuilder.CallGraphProfile = !CodeGenOpts.DisableIntegratedAS;
 
   PMBuilder.DisableUnrollLoops = !CodeGenOpts.UnrollLoops;
   // Loop interleaving in the loop vectorizer has historically been set to be
   // enabled when loop unrolling is enabled.
   PMBuilder.LoopsInterleaved = CodeGenOpts.UnrollLoops;
   PMBuilder.MergeFunctions = CodeGenOpts.MergeFunctions;
+  PMBuilder.SplitColdCode = CodeGenOpts.SplitColdCode;
   PMBuilder.PrepareForThinLTO = CodeGenOpts.PrepareForThinLTO;
   PMBuilder.PrepareForLTO = CodeGenOpts.PrepareForLTO;
   PMBuilder.RerollLoops = CodeGenOpts.RerollLoops;
 
   MPM.add(new TargetLibraryInfoWrapperPass(*TLII));
 
   if (TM)
     TM->adjustPassManager(PMBuilder);
if (CodeGenOpts.OptimizationLevel == 0) {
         MPM.addPass(CanonicalizeAliasesPass());
         MPM.addPass(NameAnonGlobalPass());
       }
     } else {
       // Map our optimization levels into one of the distinct levels used to
       // configure the pipeline.
       PassBuilder::OptimizationLevel Level = mapToLevel(CodeGenOpts);
 
+      // -f[no-]split-cold-code
+      PB.setEnableHotColdSplitting(CodeGenOpts.SplitColdCode);
+
       // If we reached here with a non-empty index file name, then the index
       // file was empty and we are not performing ThinLTO backend compilation
       // (used in testing in a distributed build environment). Drop any the type
       // test assume sequences inserted for whole program vtables so that
       // codegen doesn't complain.
       if (!CodeGenOpts.ThinLTOIndexFile.empty())
         PB.registerPipelineStartEPCallback([](ModulePassManager &MPM) {
           MPM.addPass(LowerTypeTestsPass(/*ExportSummary=*/nullptr,

clang/lib/Frontend/CompilerInvocation.cpp

   Opts.Addrsig = Args.hasArg(OPT_faddrsig);
 
   Opts.KeepStaticConsts = Args.hasArg(OPT_fkeep_static_consts);
 
   Opts.SpeculativeLoadHardening = Args.hasArg(OPT_mspeculative_load_hardening);
 
   Opts.DefaultFunctionAttrs = Args.getAllArgValues(OPT_default_function_attr);
 
+  // -f[no-]split-cold-code
+  // This may only be enabled when optimizing, and when small code size
+  // increases are tolerable.
+  Opts.SplitColdCode =
+      (Opts.OptimizationLevel > 0) && (Opts.OptimizeSize != 2) &&
+      Args.hasFlag(OPT_fsplit_cold_code, OPT_fno_split_cold_code, true);
+
   Opts.PassPlugins = Args.getAllArgValues(OPT_fpass_plugin_EQ);
 
   Opts.SymbolPartition =
       std::string(Args.getLastArgValue(OPT_fsymbol_partition_EQ));
 
   Opts.ForceAAPCSBitfieldLoad = Args.hasArg(OPT_ForceAAPCSBitfieldLoad);
 
   Opts.PassByValueIsNoAlias = Args.hasArg(OPT_fpass_by_value_is_noalias);

clang/test/CodeGen/split-cold-code.c

+// === Old PM ===
+// No splitting at -O0.
+// RUN: %clang_cc1 -O0 -fsplit-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-NO-SPLIT %s
+//
+// No splitting at -Oz.
+// RUN: %clang_cc1 -Oz -fsplit-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-NO-SPLIT %s
+//
+// Split by default.
+// RUN: %clang_cc1 -O3 -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-SPLIT %s
+//
+// No splitting when it's explicitly disabled.
+// RUN: %clang_cc1 -O3 -fno-split-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-NO-SPLIT %s
+//
+// No splitting when LLVM passes are disabled.
+// RUN: %clang_cc1 -O3 -fsplit-cold-code -disable-llvm-passes -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-NO-SPLIT %s
+//
+// Split at -O1.
+// RUN: %clang_cc1 -O1 -fsplit-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-SPLIT %s
+//
+// Split at -Os.
+// RUN: %clang_cc1 -Os -fsplit-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-SPLIT %s
+//
+// Split at -O2.
+// RUN: %clang_cc1 -O2 -fsplit-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-SPLIT %s
+//
+// Split at -O3.
+// RUN: %clang_cc1 -O3 -fsplit-cold-code -mllvm -debug-pass=Structure \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=OLDPM-SPLIT %s
+
+// OLDPM-NO-SPLIT-NOT: Hot Cold Split
+
+// OLDPM-SPLIT: Hot Cold Split
+
+// === New PM (ditto) ===
+// No splitting at -O0.
+// RUN: %clang_cc1 -O0 -fsplit-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-NO-SPLIT %s
+//
+// No splitting at -Oz.
+// RUN: %clang_cc1 -Oz -fsplit-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-NO-SPLIT %s
+//
+// Split by default.
+// RUN: %clang_cc1 -O3 -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-SPLIT %s
+//
+// No splitting when it's explicitly disabled.
+// RUN: %clang_cc1 -O3 -fno-split-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-NO-SPLIT %s
+//
+// No splitting when LLVM passes are disabled.
+// RUN: %clang_cc1 -O3 -fsplit-cold-code -disable-llvm-passes -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-NO-SPLIT %s
+//
+// Split at -O1.
+// RUN: %clang_cc1 -O1 -fsplit-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-SPLIT %s
+//
+// Split at -Os.
+// RUN: %clang_cc1 -Os -fsplit-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-SPLIT %s
+//
+// Split at -O2.
+// RUN: %clang_cc1 -O2 -fsplit-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-SPLIT %s
+//
+// Split at -O3.
+// RUN: %clang_cc1 -O3 -fsplit-cold-code -fexperimental-new-pass-manager -fdebug-pass-manager \
+// RUN:   -emit-llvm -o /dev/null %s 2>&1 | FileCheck --check-prefix=NEWPM-SPLIT %s
+
+// NEWPM-NO-SPLIT-NOT: HotColdSplit
+
+// NEWPM-SPLIT: HotColdSplit

clang/test/CodeGenCXX/microsoft-abi-typeid.cpp

-// RUN: %clang_cc1 -emit-llvm -O1 -o - -triple=i386-pc-win32 %s -fexceptions -fcxx-exceptions | FileCheck %s
+// RUN: %clang_cc1 -emit-llvm -O1 -o - -triple=i386-pc-win32 %s -fno-split-cold-code -fexceptions -fcxx-exceptions | FileCheck %s
 
 struct type_info;
 namespace std { using ::type_info; }
 
 struct V { virtual void f(); };
 struct A : virtual V { A(); };
 
 extern A a;

clang/test/CodeGenCXX/nrvo.cpp

-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm -O1 -fno-experimental-new-pass-manager -o - %s | FileCheck %s
-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm -O1 -fno-experimental-new-pass-manager -fcxx-exceptions -fexceptions -std=c++03 -o - %s | FileCheck --check-prefixes=CHECK-EH,CHECK-EH-03 %s
-// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm -O1 -fno-experimental-new-pass-manager -fcxx-exceptions -fexceptions -std=c++11 -o - %s | FileCheck --check-prefixes=CHECK-EH,CHECK-EH-11 %s
+// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm -O1 -fno-split-cold-code -fno-experimental-new-pass-manager -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm -O1 -fno-split-cold-code -fno-experimental-new-pass-manager -fcxx-exceptions -fexceptions -std=c++03 -o - %s | FileCheck --check-prefixes=CHECK-EH,CHECK-EH-03 %s
+// RUN: %clang_cc1 -triple i386-unknown-unknown -emit-llvm -O1 -fno-split-cold-code -fno-experimental-new-pass-manager -fcxx-exceptions -fexceptions -std=c++11 -o - %s | FileCheck --check-prefixes=CHECK-EH,CHECK-EH-11 %s
 
 // Test code generation for the named return value optimization.
 class X {
 public:
   X();
   X(const X&);
   ~X();
 };

clang/test/CodeGenObjC/synchronized.m

-// RUN: %clang_cc1 -emit-llvm -triple i686-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5 -o - %s -O2 | FileCheck %s
+// RUN: %clang_cc1 -emit-llvm -triple i686-apple-darwin9 -fno-split-cold-code -fobjc-runtime=macosx-fragile-10.5 -o - %s -O2 | FileCheck %s
 
 @interface MyClass
 {
 }
 - (void)method;
 @end
 
 @implementation MyClass

clang/test/CodeGenObjCXX/exceptions-legacy.mm

-// RUN: %clang_cc1 -triple i386-apple-darwin10 -fobjc-runtime=macosx-fragile-10.5 -emit-llvm -fexceptions -fobjc-exceptions -O2 -o - %s | FileCheck %s
+// RUN: %clang_cc1 -triple i386-apple-darwin10 -fno-split-cold-code -fobjc-runtime=macosx-fragile-10.5 -emit-llvm -fexceptions -fobjc-exceptions -O2 -o - %s | FileCheck %s
 
 // Test we maintain at least a basic amount of interoperation between
 // ObjC and C++ exceptions in the legacy runtime.
 
 // rdar://12364847
 
 void foo(void);
 

llvm/include/llvm/Passes/PassBuilder.h

public:
   ///
   /// If the PassManager type is not given at the top level of the pipeline
   /// text, this Callback should be used to determine the appropriate stack of
   /// PassManagers and populate the passed ModulePassManager.
   void registerParseTopLevelPipelineCallback(
       const std::function<bool(ModulePassManager &, ArrayRef<PipelineElement>,
                                bool VerifyEachPass, bool DebugLogging)> &C);
 
+  /// Enable or disable the hot/cold splitting optimization. By default, it is
+  /// disabled.
+  void setEnableHotColdSplitting(bool Enabled);
+
   /// Add PGOInstrumenation passes for O0 only.
   void addPGOInstrPassesForO0(ModulePassManager &MPM, bool DebugLogging,
                               bool RunProfileGen, bool IsCS,
                               std::string ProfileFile,
                               std::string ProfileRemappingFile);
 
 
   /// Returns PIC. External libraries can use this to register pass
SmallVector<std::function<void(LoopAnalysisManager &)>, 2>
       LoopAnalysisRegistrationCallbacks;
   SmallVector<std::function<bool(StringRef, LoopPassManager &,
                                  ArrayRef<PipelineElement>)>,
               2>
       LoopPipelineParsingCallbacks;
   // AA callbacks
   SmallVector<std::function<bool(StringRef Name, AAManager &AA)>, 2>
       AAParsingCallbacks;
+
+  // Tunable passes
+  bool SplitColdCode = false;
 };
 
 /// This utility template takes care of adding require<> and invalidate<>
 /// passes for an analysis to a given \c PassManager. It is intended to be used
 /// during parsing of a pass pipeline when parsing a single PipelineName.
 /// When registering a new function analysis FancyAnalysis with the pass
 /// pipeline name "fancy-analysis", a matching ParsePipelineCallback could look
 /// like this:

llvm/include/llvm/Transforms/IPO/PassManagerBuilder.h

public:
   bool LoopsInterleaved;
   bool RerollLoops;
   bool NewGVN;
   bool DisableGVNLoadPRE;
   bool ForgetAllSCEVInLoopUnroll;
   bool VerifyInput;
   bool VerifyOutput;
   bool MergeFunctions;
+  bool SplitColdCode;
   bool PrepareForLTO;
   bool PrepareForThinLTO;
   bool PerformThinLTO;
   bool DivergentTarget;
   unsigned LicmMssaOptCap;
   unsigned LicmMssaNoAccForPromotionCap;
 
   /// Enable profile instrumentation pass.

llvm/lib/Passes/PassBuilder.cpp

 
   // Now that we've vectorized and unrolled loops, we may have more refined
   // alignment information, try to re-derive it here.
   OptimizePM.addPass(AlignmentFromAssumptionsPass());
 
   // Split out cold code. Splitting is done late to avoid hiding context from
   // other optimizations and inadvertently regressing performance. The tradeoff
   // is that this has a higher code size cost than splitting early.
-  if (EnableHotColdSplit && !LTOPreLink)
+  if ((EnableHotColdSplit || SplitColdCode) && !LTOPreLink)
     MPM.addPass(HotColdSplittingPass());
 
   // LoopSink pass sinks instructions hoisted by LICM, which serves as a
   // canonicalization pass that enables other optimizations. As a result,
   // LoopSink pass needs to be a very late IR pass to avoid undoing LICM
   // result too early.
   OptimizePM.addPass(LoopSinkPass());
 

   // CFI is disabled.
   MPM.addPass(LowerTypeTestsPass(ExportSummary, nullptr));
   // Run a second time to clean up any type tests left behind by WPD for use
   // in ICP (which is performed earlier than this in the regular LTO pipeline).
   MPM.addPass(LowerTypeTestsPass(nullptr, nullptr, true));
 
   // Enable splitting late in the FullLTO post-link pipeline. This is done in
   // the same stage in the old pass manager (\ref addLateLTOOptimizationPasses).
-  if (EnableHotColdSplit)
+  if (EnableHotColdSplit || SplitColdCode)
     MPM.addPass(HotColdSplittingPass());
 
   // Add late LTO optimization passes.
   // Delete basic blocks, which optimization passes may have killed.
   MPM.addPass(createModuleToFunctionPassAdaptor(SimplifyCFGPass()));
 
   // Drop bodies of available eternally objects to improve GlobalDCE.
   MPM.addPass(EliminateAvailableExternallyPass());

       return make_error<StringError>(
           formatv("unknown alias analysis name '{0}'", Name).str(),
           inconvertibleErrorCode());
   }
 
   return Error::success();
 }
 
+void PassBuilder::setEnableHotColdSplitting(bool Enabled) {
+  SplitColdCode = Enabled;
+}
+
 bool PassBuilder::isAAPassName(StringRef PassName) {
 #define MODULE_ALIAS_ANALYSIS(NAME, CREATE_PASS)                               \
   if (PassName == NAME)                                                        \
     return true;
 #define FUNCTION_ALIAS_ANALYSIS(NAME, CREATE_PASS)                             \
   if (PassName == NAME)                                                        \
     return true;
 #include "PassRegistry.def"

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

PassManagerBuilder::PassManagerBuilder() {
     NewGVN = RunNewGVN;
     LicmMssaOptCap = SetLicmMssaOptCap;
     LicmMssaNoAccForPromotionCap = SetLicmMssaNoAccForPromotionCap;
     DisableGVNLoadPRE = false;
     ForgetAllSCEVInLoopUnroll = ForgetSCEVInLoopUnroll;
     VerifyInput = false;
     VerifyOutput = false;
     MergeFunctions = false;
+    SplitColdCode = false;
     PrepareForLTO = false;
     EnablePGOInstrGen = false;
     EnablePGOCSInstrGen = false;
     EnablePGOCSInstrUse = false;
     PGOInstrGen = "";
     PGOInstrUse = "";
     PGOSampleUse = "";
     PrepareForThinLTO = EnablePrepareForThinLTO;
void PassManagerBuilder::populateModulePassManager(
   // late pass of GlobalDCE.  It is capable of deleting dead cycles.
   if (OptLevel > 1) {
     MPM.add(createGlobalDCEPass());         // Remove dead fns and globals.
     MPM.add(createConstantMergePass());     // Merge dup global constants
   }
 
   // See comment in the new PM for justification of scheduling splitting at
   // this stage (\ref buildModuleSimplificationPipeline).
-  if (EnableHotColdSplit && !(PrepareForLTO || PrepareForThinLTO))
+  if ((EnableHotColdSplit || SplitColdCode) &&
+      !(PrepareForLTO || PrepareForThinLTO))
     MPM.add(createHotColdSplittingPass());
 
   if (MergeFunctions)
     MPM.add(createMergeFunctionsPass());
 
   // Add Module flag "CG Profile" based on Branch Frequency Information.
   if (CallGraphProfile)
     MPM.add(createCGProfileLegacyPass());
void PassManagerBuilder::addLTOOptimizationPasses(legacy::PassManagerBase &PM) {
 
   PM.add(createJumpThreadingPass(/*FreezeSelectCond*/ true));
 }
 
 void PassManagerBuilder::addLateLTOOptimizationPasses(
     legacy::PassManagerBase &PM) {
   // See comment in the new PM for justification of scheduling splitting at
   // this stage (\ref buildLTODefaultPipeline).
-  if (EnableHotColdSplit)
+  if (EnableHotColdSplit || SplitColdCode)
     PM.add(createHotColdSplittingPass());
 
   // Delete basic blocks, which optimization passes may have killed.
   PM.add(createCFGSimplificationPass());
 
   // Drop bodies of available externally objects to improve GlobalDCE.
   PM.add(createEliminateAvailableExternallyPass());