[LTO] Support new PM in ThinLTOCodeGenerator.

[fhahn]

Summary

This patch adds initial support for using the new pass manager when
doing ThinLTO via libLTO.

[steven_wu]

That is easier than I think. Is there still any missing function in terms of NewPM after this patch?

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

533

There is another instance of optimizeModule. Maybe update that as well?

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

Why this test case needs to be changed for switch PM?

[fhahn]

clang-format, also update second use or optimizeModule.

[fhahn]

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

The new PM does inlining a bit different, that's where the difference comes from I think. Perhaps @thegameg has any additional insight in the test case?

[thegameg]

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

Looks like we went from foo inlined into main to bar inlined into foo. Maybe we now have two remarks: bar inlined into foo + foo inlined into main?

[fhahn]

Updated to also add the -debug-pass-manager option, so it is easier to check that -use-new-pm works as expected. I also pinned the remarks tests to use the legacy PM and updated llvm/test/ThinLTO/X86/newpm-basic.ll to check that the new PM is run via the debug output.

In D102627#2763674, @steven_wu wrote:

That is easier than I think. Is there still any missing function in terms of NewPM after this patch?

It's still not unified with the new LTO API, so if anything in the pipeline setup changes in the new LTO API we have to also update the code here. But other than that, ThinLTO should use the NewPM same as with the new API.

[fhahn]

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

23

Yep, but I don't think this change is really relevant for the switch to the new PM, as different inlining decisions are just a side-effect of using the new PM. In the latest version I pinned the remarks tests to use the legacy PM and updated llvm/test/ThinLTO/X86/newpm-basic.ll to check that -use-new-pm works as expected.

[steven_wu]

LGTM

llvm/include/llvm/LTO/legacy/ThinLTOCodeGenerator.h

void setCodeGenOptLevel(CodeGenOpt::Level CGOptLevel) {
     TMBuilder.CGOptLevel = CGOptLevel;
   }
 
   /// IR optimization level: from 0 to 3.
   void setOptLevel(unsigned NewOptLevel) {
     OptLevel = (NewOptLevel > 3) ? 3 : NewOptLevel;
   }
 
+  /// Enable or disable the new pass manager.
+  void setUseNewPM(unsigned Enabled) { UseNewPM = Enabled; }
+
+  /// Enable or disable debug output for the new pass manager.
+  void setDebugPassManager(unsigned Enabled) { DebugPassManager = Enabled; }
+
   /// Disable CodeGen, only run the stages till codegen and stop. The output
   /// will be bitcode.
   void disableCodeGen(bool Disable) { DisableCodeGen = Disable; }
 
   /// Perform CodeGen only: disable all other stages.
   void setCodeGenOnly(bool CGOnly) { CodeGenOnly = CGOnly; }
 
   /**@}*/
private:
   bool CodeGenOnly = false;
 
   /// Flag to indicate that the optimizer should not assume builtins are present
   /// on the target.
   bool Freestanding = false;
 
   /// IR Optimization Level [0-3].
   unsigned OptLevel = 3;
+
+  /// Flag to indicate whether the new pass manager should be used for IR
+  /// optimizations.
+  bool UseNewPM = LLVM_ENABLE_NEW_PASS_MANAGER;
+
+  /// Flag to indicate whether debug output should be enabled for the new pass
+  /// manager.
+  bool DebugPassManager = false;
 };
 }
 #endif

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/LTO/legacy/ThinLTOCodeGenerator.h"
 #include "llvm/Support/CommandLine.h"
 
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/ModuleSummaryAnalysis.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Bitcode/BitcodeReader.h"
 #include "llvm/Bitcode/BitcodeWriter.h"
 #include "llvm/Bitcode/BitcodeWriterPass.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/DiagnosticPrinter.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/LLVMRemarkStreamer.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/IR/Mangler.h"
 #include "llvm/IR/PassTimingInfo.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/IRReader/IRReader.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/LTO/SummaryBasedOptimizations.h"
 #include "llvm/MC/SubtargetFeature.h"
 #include "llvm/Object/IRObjectFile.h"
+#include "llvm/Passes/PassBuilder.h"
+#include "llvm/Passes/StandardInstrumentations.h"
 #include "llvm/Remarks/HotnessThresholdParser.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileUtilities.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SHA1.h"
 #include "llvm/Support/SmallVectorMemoryBuffer.h"
static void optimizeModule(Module &TheModule, TargetMachine &TM,
   PM.add(createTargetTransformInfoWrapperPass(TM.getTargetIRAnalysis()));
 
   // Add optimizations
   PMB.populateThinLTOPassManager(PM);
 
   PM.run(TheModule);
 }
 
+static void optimizeModuleNewPM(Module &TheModule, TargetMachine &TM,
+                                unsigned OptLevel, bool Freestanding,
+                                bool DebugPassManager,
+                                ModuleSummaryIndex *Index) {
+  Optional<PGOOptions> PGOOpt;
+  LoopAnalysisManager LAM;
+  FunctionAnalysisManager FAM;
+  CGSCCAnalysisManager CGAM;
+  ModuleAnalysisManager MAM;
+
+  PassInstrumentationCallbacks PIC;
+  StandardInstrumentations SI(DebugPassManager);
+  SI.registerCallbacks(PIC, &FAM);
+  PipelineTuningOptions PTO;
+  PTO.LoopVectorization = true;
+  PTO.SLPVectorization = true;
+  PassBuilder PB(&TM, PTO, PGOOpt, &PIC);
+
+  std::unique_ptr<TargetLibraryInfoImpl> TLII(
+      new TargetLibraryInfoImpl(Triple(TM.getTargetTriple())));
+  if (Freestanding)
+    TLII->disableAllFunctions();
+  FAM.registerPass([&] { return TargetLibraryAnalysis(*TLII); });
+
+  AAManager AA = PB.buildDefaultAAPipeline();
+
+  // Register the AA manager first so that our version is the one used.
+  FAM.registerPass([&] { return std::move(AA); });
+
+  // Register all the basic analyses with the managers.
+  PB.registerModuleAnalyses(MAM);
+  PB.registerCGSCCAnalyses(CGAM);
+  PB.registerFunctionAnalyses(FAM);
+  PB.registerLoopAnalyses(LAM);
+  PB.crossRegisterProxies(LAM, FAM, CGAM, MAM);
+
+  ModulePassManager MPM;
+
+  PassBuilder::OptimizationLevel OL;
+
+  switch (OptLevel) {
+  default:
+    llvm_unreachable("Invalid optimization level");
+  case 0:
+    OL = PassBuilder::OptimizationLevel::O0;
+    break;
+  case 1:
+    OL = PassBuilder::OptimizationLevel::O1;
+    break;
+  case 2:
+    OL = PassBuilder::OptimizationLevel::O2;
+    break;
+  case 3:
+    OL = PassBuilder::OptimizationLevel::O3;
+    break;
+  }
+
+  MPM.addPass(PB.buildThinLTODefaultPipeline(OL, Index));
+
+  MPM.run(TheModule, MAM);
+}
+
 static void
 addUsedSymbolToPreservedGUID(const lto::InputFile &File,
                              DenseSet<GlobalValue::GUID> &PreservedGUID) {
   for (const auto &Sym : File.symbols()) {
     if (Sym.isUsed())
       PreservedGUID.insert(GlobalValue::getGUID(Sym.getIRName()));
   }
 }

 ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
                      StringMap<lto::InputFile *> &ModuleMap, TargetMachine &TM,
                      const FunctionImporter::ImportMapTy &ImportList,
                      const FunctionImporter::ExportSetTy &ExportList,
                      const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols,
                      const GVSummaryMapTy &DefinedGlobals,
                      const ThinLTOCodeGenerator::CachingOptions &CacheOptions,
                      bool DisableCodeGen, StringRef SaveTempsDir,
-                     bool Freestanding, unsigned OptLevel, unsigned count) {
+                     bool Freestanding, unsigned OptLevel, unsigned count,
+                     bool UseNewPM, bool DebugPassManager) {
 
   // "Benchmark"-like optimization: single-source case
   bool SingleModule = (ModuleMap.size() == 1);
 
   // When linking an ELF shared object, dso_local should be dropped. We
   // conservatively do this for -fpic.
   bool ClearDSOLocalOnDeclarations =
       TM.getTargetTriple().isOSBinFormatELF() &&
ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
   if (!SingleModule) {
     crossImportIntoModule(TheModule, Index, ModuleMap, ImportList,
                           ClearDSOLocalOnDeclarations);
 
     // Save temps: after cross-module import.
     saveTempBitcode(TheModule, SaveTempsDir, count, ".3.imported.bc");
   }
 
+  if (UseNewPM)
+    optimizeModuleNewPM(TheModule, TM, OptLevel, Freestanding, DebugPassManager,
+                        &Index);
+  else
     optimizeModule(TheModule, TM, OptLevel, Freestanding, &Index);
 
   saveTempBitcode(TheModule, SaveTempsDir, count, ".4.opt.bc");
 
   if (DisableCodeGen) {
     // Configured to stop before CodeGen, serialize the bitcode and return.
     SmallVector<char, 128> OutputBuffer;
     {
       raw_svector_ostream OS(OutputBuffer);
for (auto IndexCount : ModulesOrdering) {
         saveTempBitcode(*TheModule, SaveTempsDir, count, ".0.original.bc");
 
         auto &ImportList = ImportLists[ModuleIdentifier];
         // Run the main process now, and generates a binary
         auto OutputBuffer = ProcessThinLTOModule(
             *TheModule, *Index, ModuleMap, *TMBuilder.create(), ImportList,
             ExportList, GUIDPreservedSymbols,
             ModuleToDefinedGVSummaries[ModuleIdentifier], CacheOptions,
-            DisableCodeGen, SaveTempsDir, Freestanding, OptLevel, count);
+            DisableCodeGen, SaveTempsDir, Freestanding, OptLevel, count,
+            UseNewPM, DebugPassManager);
 
         // Commit to the cache (if enabled)
         CacheEntry.write(*OutputBuffer);
 
         if (SavedObjectsDirectoryPath.empty()) {
           // We need to generated a memory buffer for the linker.
           if (!CacheEntryPath.empty()) {
             // When cache is enabled, reload from the cache if possible.

llvm/test/ThinLTO/X86/diagnostic-handler-remarks-with-hotness.ll

 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: opt -module-summary %p/Inputs/diagnostic-handler-remarks.ll -o %t2.bc
 
 ; Check that the hotness attribute is included in the optimization record file
 ; with -lto-pass-remarks-with-hotness.
 
 ; RUN: llvm-lto -thinlto-action=run \
+; RUN:          -use-new-pm=false \
 ; RUN:          -lto-pass-remarks-output=%t.yaml \
 ; RUN:          -lto-pass-remarks-with-hotness \
 ; RUN:          -exported-symbol _func2 \
 ; RUN:          -exported-symbol _main %t1.bc %t2.bc 2>&1 | \
 ; RUN:     FileCheck %s -allow-empty
 ; CHECK-NOT: remark:
 ; CHECK-NOT: llvm-lto:
 

llvm/test/ThinLTO/X86/diagnostic-handler-remarks.ll

 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: opt -module-summary %p/Inputs/diagnostic-handler-remarks.ll -o %t2.bc
 
 ; Optimization records are collected regardless of the diagnostic handler
 ; RUN: rm -f %t.yaml.thin.0.yaml %t.yaml.thin.1.yaml
 ; RUN: llvm-lto -thinlto-action=run \
+; RUN:          -use-new-pm=false \
 ; RUN:          -lto-pass-remarks-output=%t.yaml \
 ; RUN:          -lto-pass-remarks-filter=inline \
 ; RUN:          -lto-pass-remarks-format=yaml \
 ; RUN:          -exported-symbol _func2 \
 ; RUN:          -exported-symbol _main %t1.bc %t2.bc 2>&1 | \
 ; RUN:     FileCheck %s -allow-empty
 ; CHECK-NOT: remark:
 ; CHECK-NOT: llvm-lto:
 
 
 ; Verify that bar is imported and inlined into foo
 ; RUN: cat %t.yaml.thin.0.yaml | FileCheck %s -check-prefix=YAML1
 ; YAML1:      --- !Passed
 ; YAML1-NEXT: Pass:            inline
 ; YAML1-NEXT: Name:            Inlined
 ; YAML1-NEXT: Function:        main
 ; YAML1-NEXT: Args:
 ; YAML1-NEXT:   - Callee:          foo
 ; YAML1-NEXT:   - String:          ' inlined into '
 ; YAML1-NEXT:   - Caller:          main
 ; YAML1-NEXT:   - String:          ' with '
 ; YAML1-NEXT:   - String:          '(cost='
 ; YAML1-NEXT:   - Cost:            '-30'
 ; YAML1-NEXT:   - String:          ', threshold='

llvm/test/ThinLTO/X86/newpm-basic.ll

 ; RUN: opt -module-summary %s -o %t1.bc
 ; RUN: llvm-lto2 run %t1.bc -o %t.o \
 ; RUN:     -r=%t1.bc,_tinkywinky,pxl \
-; RUN:     -use-new-pm
+; RUN:     -debug-pass-manager \
+; RUN:     -use-new-pm 2>&1 | FileCheck --check-prefix=NEWPM %s
+
+; RUN: llvm-lto -thinlto-action=run -exported-symbol tinkywinky \
+; RUN:          -use-new-pm \
+; RUN:          -debug-pass-manager \
+; RUN:          %t1.bc 2>&1 | FileCheck --check-prefix=NEWPM %s
+
+; RUN: llvm-lto -thinlto-action=run -exported-symbol tinkywinky \
+; RUN:          -use-new-pm=false \
+; RUN:          -debug-pass-manager \
+; RUN:          -debug-pass=Structure \
+; RUN:          %t1.bc 2>&1 | FileCheck --check-prefix=LEGACYPM %s
+
+; Check that the selected pass manager is used for middle-end optimizations by
+; checking the debug output for IPSCCP.
+
+; NEWPM-NOT: Interprocedural Sparse Conditional Constant Propagation
+; NEWPM: Running pass: IPSCCPPass on [module]
+; NEWPM-NOT: Interprocedural Sparse Conditional Constant Propagation
+
+; LEGACYPM-NOT: IPSCCPPass
+; LEGACYPM:  Interprocedural Sparse Conditional Constant Propagation
+; LEGACYPM-NOT: IPSCCPPass
 
 target datalayout = "e-m:o-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.11.0"
 
 define void @tinkywinky() {
     ret void
 }

llvm/tools/llvm-lto/llvm-lto.cpp

static cl::opt<bool> PrintMachOCPUOnly(
     "print-macho-cpu-only", cl::init(false),
     cl::desc("Instead of running LTO, print the mach-o cpu in each IR file"));
 
 static cl::opt<bool>
     UseNewPM("use-new-pm",
              cl::desc("Run LTO passes using the new pass manager"),
              cl::init(LLVM_ENABLE_NEW_PASS_MANAGER), cl::Hidden);
 
+static cl::opt<bool>
+    DebugPassManager("debug-pass-manager", cl::init(false), cl::Hidden,
+                     cl::desc("Print pass management debugging information"));
+
 namespace {
 
 struct ModuleInfo {
   std::vector<bool> CanBeHidden;
 };
 
 } // end anonymous namespace
 
ThinLTOProcessing(const TargetOptions &Options) {
     ThinGenerator.setCodePICModel(codegen::getExplicitRelocModel());
     ThinGenerator.setTargetOptions(Options);
     ThinGenerator.setCacheDir(ThinLTOCacheDir);
     ThinGenerator.setCachePruningInterval(ThinLTOCachePruningInterval);
     ThinGenerator.setCacheEntryExpiration(ThinLTOCacheEntryExpiration);
     ThinGenerator.setCacheMaxSizeFiles(ThinLTOCacheMaxSizeFiles);
     ThinGenerator.setCacheMaxSizeBytes(ThinLTOCacheMaxSizeBytes);
     ThinGenerator.setFreestanding(EnableFreestanding);
+    ThinGenerator.setUseNewPM(UseNewPM);
+    ThinGenerator.setDebugPassManager(DebugPassManager);
 
     // Add all the exported symbols to the table of symbols to preserve.
     for (unsigned i = 0; i < ExportedSymbols.size(); ++i)
       ThinGenerator.preserveSymbol(ExportedSymbols[i]);
   }
 
   void run() {
     switch (ThinLTOMode) {