[LTO] Support new PM in ThinLTOCodeGenerator.

[fhahn]

Summary

This patch adds initial support for using the new pass manager when
doing ThinLTO via libLTO.

[steven_wu]

That is easier than I think. Is there still any missing function in terms of NewPM after this patch?

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

529

There is another instance of optimizeModule. Maybe update that as well?

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

Why this test case needs to be changed for switch PM?

[fhahn]

clang-format, also update second use or optimizeModule.

[fhahn]

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

The new PM does inlining a bit different, that's where the difference comes from I think. Perhaps @thegameg has any additional insight in the test case?

[thegameg]

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

Looks like we went from foo inlined into main to bar inlined into foo. Maybe we now have two remarks: bar inlined into foo + foo inlined into main?

[fhahn]

Updated to also add the -debug-pass-manager option, so it is easier to check that -use-new-pm works as expected. I also pinned the remarks tests to use the legacy PM and updated llvm/test/ThinLTO/X86/newpm-basic.ll to check that the new PM is run via the debug output.

In D102627#2763674, @steven_wu wrote:

That is easier than I think. Is there still any missing function in terms of NewPM after this patch?

It's still not unified with the new LTO API, so if anything in the pipeline setup changes in the new LTO API we have to also update the code here. But other than that, ThinLTO should use the NewPM same as with the new API.

[fhahn]

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

Yep, but I don't think this change is really relevant for the switch to the new PM, as different inlining decisions are just a side-effect of using the new PM. In the latest version I pinned the remarks tests to use the legacy PM and updated llvm/test/ThinLTO/X86/newpm-basic.ll to check that -use-new-pm works as expected.

[steven_wu]

LGTM

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

void setCodeGenOptLevel(CodeGenOpt::Level CGOptLevel) {
     TMBuilder.CGOptLevel = CGOptLevel;
   }
 
   /// IR optimization level: from 0 to 3.
   void setOptLevel(unsigned NewOptLevel) {
     OptLevel = (NewOptLevel > 3) ? 3 : NewOptLevel;
   }
 
+  // Enable or disable the new pass manager.
+  void setUseNewPM(unsigned Enabled) { UseNewPM = Enabled; }
+
   /// Disable CodeGen, only run the stages till codegen and stop. The output
   /// will be bitcode.
   void disableCodeGen(bool Disable) { DisableCodeGen = Disable; }
 
   /// Perform CodeGen only: disable all other stages.
   void setCodeGenOnly(bool CGOnly) { CodeGenOnly = CGOnly; }
 
   /**@}*/
private:
   bool CodeGenOnly = false;
 
   /// Flag to indicate that the optimizer should not assume builtins are present
   /// on the target.
   bool Freestanding = false;
 
   /// IR Optimization Level [0-3].
   unsigned OptLevel = 3;
+
+  /// Flag to indicate whether the new pass manager should be used for IR
+  /// optimizations.
+  bool UseNewPM = LLVM_ENABLE_NEW_PASS_MANAGER;
 };
 }
 #endif

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/LTO/legacy/ThinLTOCodeGenerator.h"
 #include "llvm/Support/CommandLine.h"
 
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/ModuleSummaryAnalysis.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Bitcode/BitcodeReader.h"
 #include "llvm/Bitcode/BitcodeWriter.h"
 #include "llvm/Bitcode/BitcodeWriterPass.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/LLVMRemarkStreamer.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Mangler.h"
 #include "llvm/IR/PassTimingInfo.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/IRReader/IRReader.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/LTO/SummaryBasedOptimizations.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/Object/IRObjectFile.h"
+#include "llvm/Passes/PassBuilder.h"
 #include "llvm/Remarks/HotnessThresholdParser.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileUtilities.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SHA1.h"
 #include "llvm/Support/SmallVectorMemoryBuffer.h"
static void optimizeModule(Module &TheModule, TargetMachine &TM,
   PM.add(createTargetTransformInfoWrapperPass(TM.getTargetIRAnalysis()));
 
   // Add optimizations
   PMB.populateThinLTOPassManager(PM);
 
   PM.run(TheModule);
 }
 
+static void optimizeModuleNewPM(Module &TheModule, TargetMachine &TM,
+                                unsigned OptLevel, bool Freestanding,
+                                ModuleSummaryIndex *Index) {
+  Optional<PGOOptions> PGOOpt;
+  LoopAnalysisManager LAM;
+  FunctionAnalysisManager FAM;
+  CGSCCAnalysisManager CGAM;
+  ModuleAnalysisManager MAM;
+
+  PassInstrumentationCallbacks PIC;
+  PipelineTuningOptions PTO;
+  PTO.LoopVectorization = true;
+  PTO.SLPVectorization = true;
+  PassBuilder PB(&TM, PTO, PGOOpt, &PIC);
+
+  std::unique_ptr<TargetLibraryInfoImpl> TLII(
+      new TargetLibraryInfoImpl(Triple(TM.getTargetTriple())));
+  if (Freestanding)
+    TLII->disableAllFunctions();
+  FAM.registerPass([&] { return TargetLibraryAnalysis(*TLII); });
+
+  AAManager AA = PB.buildDefaultAAPipeline();
+  ;
+  // Register the AA manager first so that our version is the one used.
+  FAM.registerPass([&] { return std::move(AA); });
+
+  // Register all the basic analyses with the managers.
+  PB.registerModuleAnalyses(MAM);
+  PB.registerCGSCCAnalyses(CGAM);
+  PB.registerFunctionAnalyses(FAM);
+  PB.registerLoopAnalyses(LAM);
+  PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
+
+  ModulePassManager MPM;
+
+  PassBuilder::OptimizationLevel OL;
+
+  switch (OptLevel) {
+  default:
+    llvm_unreachable("Invalid optimization level");
+  case 0:
+    OL = PassBuilder::OptimizationLevel::O0;
+    break;
+  case 1:
+    OL = PassBuilder::OptimizationLevel::O1;
+    break;
+  case 2:
+    OL = PassBuilder::OptimizationLevel::O2;
+    break;
+  case 3:
+    OL = PassBuilder::OptimizationLevel::O3;
+    break;
+  }
+
+  MPM.addPass(PB.buildThinLTODefaultPipeline(OL, Index));
+
+  MPM.run(TheModule, MAM);
+}
+
 static void
 addUsedSymbolToPreservedGUID(const lto::InputFile &File,
                              DenseSet<GlobalValue::GUID> &PreservedGUID) {
   for (const auto &Sym : File.symbols()) {
     if (Sym.isUsed())
       PreservedGUID.insert(GlobalValue::getGUID(Sym.getIRName()));
   }
 }

 ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
                      StringMap<lto::InputFile *> &ModuleMap, TargetMachine &TM,
                      const FunctionImporter::ImportMapTy &ImportList,
                      const FunctionImporter::ExportSetTy &ExportList,
                      const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
                      const GVSummaryMapTy &DefinedGlobals,
                      const ThinLTOCodeGenerator::CachingOptions &CacheOptions,
                      bool DisableCodeGen, StringRef SaveTempsDir,
-                     bool Freestanding, unsigned OptLevel, unsigned count) {
+                     bool Freestanding, unsigned OptLevel, unsigned count,
+                     bool UseNewPM) {
 
   // "Benchmark"-like optimization: single-source case
   bool SingleModule = (ModuleMap.size() == 1);
 
   // When linking an ELF shared object, dso_local should be dropped. We
   // conservatively do this for -fpic.
   bool ClearDSOLocalOnDeclarations =
       TM.getTargetTriple().isOSBinFormatELF() &&
ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
   if (!SingleModule) {
     crossImportIntoModule(TheModule, Index, ModuleMap, ImportList,
                           ClearDSOLocalOnDeclarations);
 
     // Save temps: after cross-module import.
     saveTempBitcode(TheModule, SaveTempsDir, count, ".3.imported.bc");
   }
 
+  if (UseNewPM)
+    optimizeModuleNewPM(TheModule, TM, OptLevel, Freestanding, &Index);
+  else
     optimizeModule(TheModule, TM, OptLevel, Freestanding, &Index);
 
   saveTempBitcode(TheModule, SaveTempsDir, count, ".4.opt.bc");
 
   if (DisableCodeGen) {
     // Configured to stop before CodeGen, serialize the bitcode and return.
     SmallVector<char, 128> OutputBuffer;
     {
       raw_svector_ostream OS(OutputBuffer);

 
 /**
  * Perform post-importing ThinLTO optimizations.
  */
 void ThinLTOCodeGenerator::optimize(Module &TheModule) {
   initTMBuilder(TMBuilder, Triple(TheModule.getTargetTriple()));
 
   // Optimize now
+  if (UseNewPM)
+    optimizeModuleNewPM(TheModule, *TMBuilder.create(), OptLevel, Freestanding,
+                        nullptr);
+  else
     optimizeModule(TheModule, *TMBuilder.create(), OptLevel, Freestanding,
                    nullptr);
 }
 
 /// Write out the generated object file, either from CacheEntryPath or from
 /// OutputBuffer, preferring hard-link when possible.
 /// Returns the path to the generated file in SavedObjectsDirectoryPath.
 std::string
 ThinLTOCodeGenerator::writeGeneratedObject(int count, StringRef CacheEntryPath,
                                            const MemoryBuffer &OutputBuffer) {
for (auto IndexCount : ModulesOrdering) {
         saveTempBitcode(*TheModule, SaveTempsDir, count, ".0.original.bc");
 
         auto &ImportList = ImportLists[ModuleIdentifier];
         // Run the main process now, and generates a binary
         auto OutputBuffer = ProcessThinLTOModule(
             *TheModule, *Index, ModuleMap, *TMBuilder.create(), ImportList,
             ExportList, GUIDPreservedSymbols,
             ModuleToDefinedGVSummaries[ModuleIdentifier], CacheOptions,
-            DisableCodeGen, SaveTempsDir, Freestanding, OptLevel, count);
+            DisableCodeGen, SaveTempsDir, Freestanding, OptLevel, count,
+            UseNewPM);
 
         // Commit to the cache (if enabled)
         CacheEntry.write(*OutputBuffer);
 
         if (SavedObjectsDirectoryPath.empty()) {
           // We need to generated a memory buffer for the linker.
           if (!CacheEntryPath.empty()) {
             // When cache is enabled, reload from the cache if possible.

llvm/test/ThinLTO/X86/diagnostic-handler-remarks-with-hotness.ll

 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: opt -module-summary %p/Inputs/diagnostic-handler-remarks.ll -o %t2.bc
 
 ; Check that the hotness attribute is included in the optimization record file
 ; with -lto-pass-remarks-with-hotness.
 
 ; RUN: llvm-lto -thinlto-action=run \
+; RUN           -use-new-pm \
 ; RUN:          -lto-pass-remarks-output=%t.yaml \
 ; RUN:          -lto-pass-remarks-with-hotness \
 ; RUN:          -exported-symbol _func2 \
 ; RUN:          -exported-symbol _main %t1.bc %t2.bc 2>&1 | \
 ; RUN:     FileCheck %s -allow-empty
 ; CHECK-NOT: remark:
 ; CHECK-NOT: llvm-lto:
 
 
 ; Verify that bar is imported and inlined into foo
 ; RUN: cat %t.yaml.thin.0.yaml | FileCheck %s -check-prefix=YAML1
 ; YAML1:      --- !Passed
 ; YAML1-NEXT: Pass:            inline
 ; YAML1-NEXT: Name:            Inlined
+; YAML1-NEXT: Function:        foo
+; YAML1-NEXT: Args:
+; YAML1-NEXT:   - Callee:          bar
+; YAML1-NEXT:   - String:          ' inlined into '
+; YAML1-NEXT:   - Caller:          foo
+; YAML1-NEXT:   - String:          ' with '
+; YAML1-NEXT:   - String:          '(cost='
+; YAML1-NEXT:   - Cost:            '-30'
+; YAML1-NEXT:   - String:          ', threshold='
+; YAML1-NEXT:   - Threshold:       '375'
+; YAML1-NEXT:   - String:          ')'
+; YAML1-NEXT: ...
+; YAML1-NEXT: --- !Passed
+; YAML1-NEXT: Pass:            inline
+; YAML1-NEXT: Name:            Inlined
 ; YAML1-NEXT: Function:        main
 ; YAML1-NEXT: Hotness:         50
 ; YAML1-NEXT: Args:
 ; YAML1-NEXT:   - Callee:          foo
 ; YAML1-NEXT:   - String:          ' inlined into '
 ; YAML1-NEXT:   - Caller:          main
 ; YAML1-NEXT:   - String:          ' with '
 ; YAML1-NEXT:   - String:          '(cost='
 ; YAML1-NEXT:   - Cost:            '-30'
 ; YAML1-NEXT:   - String:          ', threshold='
-; YAML1-NEXT:   - Threshold:       '337'
+; YAML1-NEXT:   - Threshold:       '375'
 ; YAML1-NEXT:   - String:          ')'
 ; YAML1-NEXT: ...
 
 
 ; Verify that bar is imported and inlined into foo
 ; RUN: cat %t.yaml.thin.1.yaml | FileCheck %s -check-prefix=YAML2
 ; YAML2:      --- !Passed
 ; YAML2-NEXT: Pass:            inline
 ; YAML2-NEXT: Name:            Inlined
 ; YAML2-NEXT: Function:        foo
 ; YAML2-NEXT: Args:
 ; YAML2-NEXT:   - Callee:          bar
 ; YAML2-NEXT:   - String:          ' inlined into '
 ; YAML2-NEXT:   - Caller:          foo
 ; YAML2-NEXT:   - String:          ' with '
 ; YAML2-NEXT:   - String:          '(cost='
 ; YAML2-NEXT:   - Cost:            '-30'
 ; YAML2-NEXT:   - String:          ', threshold='
-; YAML2-NEXT:   - Threshold:       '337'
+; YAML2-NEXT:   - Threshold:       '375'
 ; YAML2-NEXT:   - String:          ')'
 ; YAML2-NEXT: ...
 
 
 target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 define i32 @bar() {

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: opt -module-summary %p/Inputs/diagnostic-handler-remarks.ll -o %t2.bc
 
 ; Optimization records are collected regardless of the diagnostic handler
 ; RUN: rm -f %t.yaml.thin.0.yaml %t.yaml.thin.1.yaml
 ; RUN: llvm-lto -thinlto-action=run \
+; RUN:          -use-new-pm \
 ; RUN:          -lto-pass-remarks-output=%t.yaml \
 ; RUN:          -lto-pass-remarks-filter=inline \
 ; RUN:          -lto-pass-remarks-format=yaml \
 ; RUN:          -exported-symbol _func2 \
 ; RUN:          -exported-symbol _main %t1.bc %t2.bc 2>&1 | \
 ; RUN:     FileCheck %s -allow-empty
 ; CHECK-NOT: remark:
 ; CHECK-NOT: llvm-lto:
 
 
 ; Verify that bar is imported and inlined into foo
 ; RUN: cat %t.yaml.thin.0.yaml | FileCheck %s -check-prefix=YAML1
-; YAML1:      --- !Passed
+; YAML1: --- !Passed
 ; YAML1-NEXT: Pass:            inline
 ; YAML1-NEXT: Name:            Inlined
-; YAML1-NEXT: Function:        main
+; YAML1-NEXT: Function:        foo
 ; YAML1-NEXT: Args:
-; YAML1-NEXT:   - Callee:          foo
+; YAML1-NEXT:   - Callee:          bar
 ; YAML1-NEXT:   - String:          ' inlined into '
-; YAML1-NEXT:   - Caller:          main
+; YAML1-NEXT:   - Caller:          foo
 ; YAML1-NEXT:   - String:          ' with '
 ; YAML1-NEXT:   - String:          '(cost='
 ; YAML1-NEXT:   - Cost:            '-30'
 ; YAML1-NEXT:   - String:          ', threshold='
-; YAML1-NEXT:   - Threshold:       '337'
+; YAML1-NEXT:   - Threshold:       '375'
 ; YAML1-NEXT:   - String:          ')'
 ; YAML1-NEXT: ...
 
 
 ; Verify that bar is imported and inlined into foo
 ; RUN: cat %t.yaml.thin.1.yaml | FileCheck %s -check-prefix=YAML2
 ; YAML2: --- !Passed
 ; YAML2-NEXT: Pass:            inline
 ; YAML2-NEXT: Name:            Inlined
 ; YAML2-NEXT: Function:        foo
 ; YAML2-NEXT: Args:
 ; YAML2-NEXT:   - Callee:          bar
 ; YAML2-NEXT:   - String:          ' inlined into '
 ; YAML2-NEXT:   - Caller:          foo
 ; YAML2-NEXT:   - String:          ' with '
 ; YAML2-NEXT:   - String:          '(cost='
 ; YAML2-NEXT:   - Cost:            '-30'
 ; YAML2-NEXT:   - String:          ', threshold='
-; YAML2-NEXT:   - Threshold:       '337'
+; YAML2-NEXT:   - Threshold:       '375'
 ; YAML2-NEXT:   - String:          ')'
 ; YAML2-NEXT: ...
 
 ; The file extension depends on the format of the remarks
 ; RUN: rm -f %t.bitstream.thin.0.bitstream %t.bitstream.thin.1.bitstream
 ; RUN: llvm-lto -thinlto-action=run \
+; RUN:          -use-new-pm \
 ; RUN:          -lto-pass-remarks-output=%t.bitstream \
 ; RUN:          -lto-pass-remarks-filter=inline \
 ; RUN:          -lto-pass-remarks-format=bitstream \
 ; RUN:          -exported-symbol _func2 \
 ; RUN:          -exported-symbol _main %t1.bc %t2.bc 2>&1 | \
 ; RUN:     FileCheck %s -allow-empty
 ; RUN: llvm-bcanalyzer %t.bitstream.thin.0.bitstream
 ; RUN: llvm-bcanalyzer %t.bitstream.thin.1.bitstream

llvm/tools/llvm-lto/llvm-lto.cpp

ThinLTOProcessing(const TargetOptions &Options) {
     ThinGenerator.setCodePICModel(codegen::getExplicitRelocModel());
     ThinGenerator.setTargetOptions(Options);
     ThinGenerator.setCacheDir(ThinLTOCacheDir);
     ThinGenerator.setCachePruningInterval(ThinLTOCachePruningInterval);
     ThinGenerator.setCacheEntryExpiration(ThinLTOCacheEntryExpiration);
     ThinGenerator.setCacheMaxSizeFiles(ThinLTOCacheMaxSizeFiles);
     ThinGenerator.setCacheMaxSizeBytes(ThinLTOCacheMaxSizeBytes);
     ThinGenerator.setFreestanding(EnableFreestanding);
+    ThinGenerator.setUseNewPM(UseNewPM);
 
     // Add all the exported symbols to the table of symbols to preserve.
     for (unsigned i = 0; i < ExportedSymbols.size(); ++i)
       ThinGenerator.preserveSymbol(ExportedSymbols[i]);
   }
 
   void run() {
     switch (ThinLTOMode) {