Add support for LTO options

[rdhindsa]

Summary

Add support for LTO options: sample_profile, new_pass_manager and debug_pass_manager

[ruiu]

Generally looking good.

lld/ELF/Driver.cpp

660–663 ↗

(On Diff #141002)

Please sort alphabetically.

lld/ELF/LTO.cpp

108 ↗

(On Diff #141002)

Do you need this if? I thought you could copy a string even if it is empty.

[ruiu]

By the way, is there any way to write a test for the new flags?

[pcc]

In D45275#1057316, @ruiu wrote:

By the way, is there any way to write a test for the new flags?

It should be possible. For new-pass-manger see llvm/test/tools/llvm-lto2/X86/pipeline.ll and for sample-profile see llvm/test/tools/gold/X86/thinlto_afdo.ll

[rdhindsa]

Added tests for options: sample-profile and new-pass-manager.

[ruiu]

lld/ELF/Config.h

135 ↗

(On Diff #141209)

Remove the default value because it is always overwritten.

lld/ELF/Driver.cpp

661 ↗

(On Diff #141209)

hasFlag is for a boolean flag such as --foo and --no-foo, and you are supposed to use this as

Args.hasFlag(OPT_foo, OPT_no_foo, default-value-which-is-true-or-false)

So this is a misuse of the function. The first and the second should be different. You should use Args.hasArg instead.

lld/ELF/LTO.cpp

110 ↗

(On Diff #141209)

I wouldn't add this comment because this is obvious from the code.

lld/ELF/Options.td

398 ↗

(On Diff #141209)

HelpText is displayed as part of the --help message,

$ ld.lld --help
...
--as-needed             Only set DT_NEEDED for shared libraries if used
--auxiliary <value>     Set DT_AUXILIARY field to the specified name
--Bdynamic              Link against shared libraries
--Bshareable            Build a shared object
...

"New pass manager" doesn't start with a verb, and it is not appropriate as a help message. "Use new pass manager" is better.

lld/test/ELF/lto/Inputs/sample.prof

1 ↗

(On Diff #141209)

What is this file? Is there any way to avoid checking in a binary file?

[rdhindsa]

lld/test/ELF/lto/Inputs/sample.prof

1 ↗

(On Diff #141209)

This is the file for input profile to be used with sample-profile option. Here, it is an input to sample-profile.ll test. For now, it is empty as we just want to test the option.

[ruiu]

lld/test/ELF/lto/Inputs/sample.prof

1 ↗

(On Diff #141209)

If an empty file suffices, please use /dev/null instead of a real empty file.

[ruiu]

Peter, are you fine with this change?

lld/test/ELF/lto/new-pass-manager.ll

4–5 ↗

(On Diff #141222)

Don't break this line.

[pcc]

lld/test/ELF/lto/new-pass-manager.ll

7 ↗

(On Diff #141222)

You don't actually need to pass --plugin-opt=thinlto here. The flag has no effect.

I would also do something like D45293 to make sure that we are actually using the new pass manager.

[rdhindsa]

lld/test/ELF/lto/new-pass-manager.ll

7 ↗

(On Diff #141222)

Added new-pass-manager option to check if Module Pass manager is run.

[pcc]

LGTM

lld/ELF/LTO.cpp

108–110 ↗

(On Diff #141414)

Nit: I'd move these lines to above line 104 where the rest of the Conf.foo = bar; lines are.

lld/trunk/ELF/Config.h

struct Configuration {
   llvm::StringRef Chroot;
   llvm::StringRef DynamicLinker;
   llvm::StringRef Entry;
   llvm::StringRef Emulation;
   llvm::StringRef Fini;
   llvm::StringRef Init;
   llvm::StringRef LTOAAPipeline;
   llvm::StringRef LTONewPmPasses;
+  llvm::StringRef LTOSampleProfile;
   llvm::StringRef MapFile;
   llvm::StringRef OutputFile;
   llvm::StringRef OptRemarksFilename;
   llvm::StringRef ProgName;
   llvm::StringRef SoName;
   llvm::StringRef Sysroot;
   llvm::StringRef ThinLTOCacheDir;
   std::string Rpath;
struct Configuration {
   bool GdbIndex;
   bool GnuHash = false;
   bool GnuUnique;
   bool HasDynamicList = false;
   bool HasDynSymTab;
   bool ICF;
   bool IgnoreDataAddressEquality;
   bool IgnoreFunctionAddressEquality;
+  bool LTODebugPassManager;
+  bool LTONewPassManager;
   bool MergeArmExidx;
   bool MipsN32Abi = false;
   bool NoinhibitExec;
   bool Nostdlib;
   bool OFormatBinary;
   bool Omagic;
   bool OptRemarksWithHotness;
   bool Pie;

lld/trunk/ELF/Driver.cpp

void LinkerDriver::readConfigs(opt::InputArgList &Args) {
   Config->GdbIndex = Args.hasFlag(OPT_gdb_index, OPT_no_gdb_index, false);
   Config->ICF = Args.hasFlag(OPT_icf_all, OPT_icf_none, false);
   Config->IgnoreDataAddressEquality =
       Args.hasArg(OPT_ignore_data_address_equality);
   Config->IgnoreFunctionAddressEquality =
       Args.hasArg(OPT_ignore_function_address_equality);
   Config->Init = Args.getLastArgValue(OPT_init, "_init");
   Config->LTOAAPipeline = Args.getLastArgValue(OPT_lto_aa_pipeline);
+  Config->LTODebugPassManager = Args.hasArg(OPT_lto_debug_pass_manager);
+  Config->LTONewPassManager = Args.hasArg(OPT_lto_new_pass_manager);
   Config->LTONewPmPasses = Args.getLastArgValue(OPT_lto_newpm_passes);
   Config->LTOO = args::getInteger(Args, OPT_lto_O, 2);
   Config->LTOPartitions = args::getInteger(Args, OPT_lto_partitions, 1);
+  Config->LTOSampleProfile = Args.getLastArgValue(OPT_lto_sample_profile);
   Config->MapFile = Args.getLastArgValue(OPT_Map);
   Config->MergeArmExidx =
       Args.hasFlag(OPT_merge_exidx_entries, OPT_no_merge_exidx_entries, true);
   Config->NoinhibitExec = Args.hasArg(OPT_noinhibit_exec);
   Config->Nostdlib = Args.hasArg(OPT_nostdlib);
   Config->OFormatBinary = isOutputFormatBinary(Args);
   Config->Omagic = Args.hasFlag(OPT_omagic, OPT_no_omagic, false);
   Config->OptRemarksFilename = Args.getLastArgValue(OPT_opt_remarks_filename);
for (auto *Arg : Args.filtered(OPT_plugin_opt)) {
     else if (S.startswith("O"))
       Config->LTOO = parseInt(S.substr(1), Arg);
     else if (S.startswith("lto-partitions="))
       Config->LTOPartitions = parseInt(S.substr(15), Arg);
     else if (S.startswith("jobs="))
       Config->ThinLTOJobs = parseInt(S.substr(5), Arg);
     else if (S.startswith("mcpu="))
       parseClangOption(Saver.save("-" + S), Arg->getSpelling());
+    else if (S == "new-pass-manager")
+      Config->LTONewPassManager = true;
+    else if (S == "debug-pass-manager")
+      Config->LTODebugPassManager = true;
+    else if (S.startswith("sample-profile="))
+      Config->LTOSampleProfile = S.substr(strlen("sample-profile="));
     else if (!S.startswith("/") && !S.startswith("-fresolution=") &&
              !S.startswith("-pass-through=") && !S.startswith("thinlto"))
       parseClangOption(S, Arg->getSpelling());
   }
 
   // Parse -mllvm options.
   for (auto *Arg : Args.filtered(OPT_mllvm))
     parseClangOption(Arg->getValue(), Arg->getSpelling());

lld/trunk/ELF/LTO.cpp

static std::unique_ptr<lto::LTO> createLTO() {
 
   if (Config->SaveTemps)
     checkError(Conf.addSaveTemps(std::string(Config->OutputFile) + ".",
                                  /*UseInputModulePath*/ true));
 
   lto::ThinBackend Backend;
   if (Config->ThinLTOJobs != -1u)
     Backend = lto::createInProcessThinBackend(Config->ThinLTOJobs);
+
+  Conf.SampleProfile = Config->LTOSampleProfile;
+  Conf.UseNewPM = Config->LTONewPassManager;
+  Conf.DebugPassManager = Config->LTODebugPassManager;
+
   return llvm::make_unique<lto::LTO>(std::move(Conf), Backend,
                                      Config->LTOPartitions);
 }
 
 BitcodeCompiler::BitcodeCompiler() : LTOObj(createLTO()) {
   for (Symbol *Sym : Symtab->getSymbols()) {
     StringRef Name = Sym->getName();
     for (StringRef Prefix : {"__start_", "__stop_"})

lld/trunk/ELF/Options.td

 def end_group: F<"end-group">;
 def end_group_paren: Flag<["-"], ")">;
 def start_group: F<"start-group">;
 def start_group_paren: Flag<["-"], "(">;
 
 // LTO-related options.
 def lto_aa_pipeline: J<"lto-aa-pipeline=">,
   HelpText<"AA pipeline to run during LTO. Used in conjunction with -lto-newpm-passes">;
+def lto_debug_pass_manager: F<"lto-debug-pass-manager">,
+  HelpText<"Debug new pass manager">;
+def lto_new_pass_manager: F<"lto-new-pass-manager">,
+  HelpText<"Use new pass manager">;
 def lto_newpm_passes: J<"lto-newpm-passes=">,
   HelpText<"Passes to run during LTO">;
 def lto_partitions: J<"lto-partitions=">,
   HelpText<"Number of LTO codegen partitions">;
+def lto_sample_profile: J<"lto-sample-profile=">,
+  HelpText<"Sample profile file path">;
 def disable_verify: F<"disable-verify">;
 defm mllvm: Eq<"mllvm">;
 def opt_remarks_filename: Separate<["--"], "opt-remarks-filename">,
   HelpText<"YAML output file for optimization remarks">;
 def opt_remarks_with_hotness: Flag<["--"], "opt-remarks-with-hotness">,
   HelpText<"Include hotness information in the optimization remarks file">;
 defm plugin_opt: Eq<"plugin-opt">,
   HelpText<"specifies LTO options for compatibility with GNU linkers">;

lld/trunk/test/ELF/lto/new-pass-manager.ll

+; REQUIRES: x86
+; RUN: opt -module-summary %s -o %t.o
+
+; Test new-pass-manager and debug-pass-manager option
+; RUN: ld.lld --plugin-opt=new-pass-manager --plugin-opt=debug-pass-manager -o %t2.o %t.o 2>&1 | FileCheck %s
+; RUN: ld.lld --plugin-opt=new-pass-manager --lto-debug-pass-manager -o %t2.o %t.o 2>&1 | FileCheck %s
+; RUN: ld.lld --lto-new-pass-manager --plugin-opt=debug-pass-manager -o %t2.o %t.o 2>&1 | FileCheck %s
+; RUN: ld.lld --lto-new-pass-manager --lto-debug-pass-manager -o %t2.o %t.o 2>&1 | FileCheck %s
+
+; CHECK: Starting llvm::Module pass manager run
+; CHECK: Finished llvm::Module pass manager run
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"

lld/trunk/test/ELF/lto/sample-profile.ll

+; REQUIRES: x86
+; RUN: opt -module-summary %s -o %t1.o
+; RUN: opt -module-summary %p/Inputs/thinlto.ll -o %t2.o
+
+; RUN: rm -f %t1.lto.o %t2.lto.o
+; RUN: ld.lld --lto-sample-profile=/dev/null %t1.o %t2.o -o %t3
+; RUN  opt -S %t3.lto.o | FileCheck %s
+
+; RUN: rm -f %t1.lto.o %t2.lto.o
+; RUN: ld.lld --plugin-opt=sample-profile=/dev/null %t1.o %t2.o -o %t3
+; RUN  opt -S %t3.lto.o | FileCheck %s
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; CHECK: ProfileSummary
+declare void @g(...)
+declare void @h(...)
+
+define void @f() {
+entry:
+  call void (...) @g()
+  call void (...) @h()
+  ret void
+}