[LTO] Add -flto-jobs=N to control backend parallelism

[tejohnson]

Summary

Currently, a linker option must be used to control the backend
parallelism of ThinLTO or full LTO (where it only applies to
parallel code gen). The linker option varies depending on the
linker (e.g. gold vs ld64). Add a new clang option -flto-jobs=N
to control this.

I've added in the wiring to pass this to the gold plugin. I also
added in the logic to pass this down in the form I understand that
ld64 uses on MacOS, for the darwin target.

[mehdi_amini]

The Gold path looks fine.
On OSX, we would have the clang driver relying on a LLVM cl::opt, for which I don't think there is any precedent. CC Duncan for advice.

[mehdi_amini]

Also I don't think the same option should be used for the parallel LTO codegen: it actually does not generate the same binary, which should deserve a dedicated opt-in (What if I mix ThinLTO and LTO, and I don't want // codegen?)

[tejohnson]

In D24826#549788, @mehdi_amini wrote:

The Gold path looks fine.
On OSX, we would have the clang driver relying on a LLVM cl::opt, for which I don't think there is any precedent. CC Duncan for advice.

I do see other uses of -mllvm in lib/Driver/Tools.cpp, but are you talking about something else?

Also I don't think the same option should be used for the parallel LTO codegen: it actually does not generate the same binary, which should deserve a dedicated opt-in (What if I mix ThinLTO and LTO, and I don't want // codegen?)

Ok good point. I can change this to -fthinlto_jobs. However, while the two parallel settings are separate in the LTO API, currently the gold-plugin jobs option controls both, so I will need to do a preparatory gold-plugin patch to split this into thinlto_jobs and lto_jobs. On the libLTO/ld64 path, looks like the current -mllvm -threads only affects ThinLTO so there is no work to do there.

[tejohnson]

Update option description as per decision to split from parallel code gen.

[tejohnson]

I do see other uses of -mllvm in lib/Driver/Tools.cpp, but are you talking about something else?

I think this is okay, since clang is talking to the same version of libLTO.dylib. I feel like there might be another case where
clang talks to libLTO.dylib through ld64 using -mllvm... perhaps, -O0?

Let's ask around though to be sure.

Ok, let me know what you find out.

Ok good point. I can change this to -fthinlto_jobs. However, while the two parallel settings are separate in the LTO API, currently the gold-plugin jobs option controls both, so I will need to do a preparatory gold-plugin patch to split this into thinlto_jobs and lto_jobs. On the libLTO/ld64 path, looks like the current -mllvm -threads only affects ThinLTO so there is no work to do there.

I actually like -flto-jobs=N better for this. I expect "jobs" not to affect output at all.

I think the current parallel FullLTO CodeGen (where it *does* affect output) should have a special name that calls this out, perhaps -flto-partitions=N? -flto-slices=N? -flto-random-partitions=N? Is it urgent to add that flag now though?

Note that I imagine someone will parallelizing FullLTO the hard way in the future, which won't affect output. That implementation should use -flto-jobs=N.

Ok, sure that seems reasonable. I changed the option documentation to note that this is currently just for ThinLTO. See also D24873 where I split the gold-plugin options.

[mehdi_amini]

include/clang/Driver/Options.td

818

std::thread::hardware_concurrency is a bit implementation specific, can we find a better formulation?
Something like (default of 0 means the number of thread will be derived from the number of CPUs detected)?

[tejohnson]

Update option help message per Mehdi's suggestion

[mehdi_amini]

LGTM, Thanks!

[andrewford]

This is breaking our android lldb build, because it uses std::to_string. Looks like there is llvm::to_string, which should be preferred. Would someone mind changing it? I don't have commit access or I would myself. :)

Thanks in advance

include/clang/Driver/Options.td

 def flax_vector_conversions : Flag<["-"], "flax-vector-conversions">, Group<f_Group>;
 def flimited_precision_EQ : Joined<["-"], "flimited-precision=">, Group<f_Group>;
 def flto_EQ : Joined<["-"], "flto=">, Flags<[CC1Option]>, Group<f_Group>,
   HelpText<"Set LTO mode to either 'full' or 'thin'">;
 def flto : Flag<["-"], "flto">, Flags<[CC1Option]>, Group<f_Group>,
   HelpText<"Enable LTO in 'full' mode">;
 def fno_lto : Flag<["-"], "fno-lto">, Group<f_Group>,
   HelpText<"Disable LTO mode (default)">;
+def flto_jobs_EQ : Joined<["-"], "flto-jobs=">,
+  Flags<[CC1Option]>, Group<f_Group>,
+  HelpText<"Controls the backend parallelism of -flto=thin (default "
+           "std::thread::hardware_concurrency) or the code generation "
+           "parallelism of -flto=full (default 1)">;
 def fthinlto_index_EQ : Joined<["-"], "fthinlto-index=">,
   Flags<[CC1Option]>, Group<f_Group>,
   HelpText<"Perform ThinLTO importing using provided function summary index">;
 def fmacro_backtrace_limit_EQ : Joined<["-"], "fmacro-backtrace-limit=">,
                                 Group<f_Group>, Flags<[DriverOption, CoreOption]>;
 def fmerge_all_constants : Flag<["-"], "fmerge-all-constants">, Group<f_Group>;
 def fmessage_length_EQ : Joined<["-"], "fmessage-length=">, Group<f_Group>;
 def fms_extensions : Flag<["-"], "fms-extensions">, Group<f_Group>, Flags<[CC1Option, CoreOption]>,

lib/Driver/Tools.cpp

case llvm::Triple::amdgcn:
     return getR600TargetGPU(Args);
 
   case llvm::Triple::wasm32:
   case llvm::Triple::wasm64:
     return getWebAssemblyTargetCPU(Args);
   }
 }
 
+static unsigned getLTOParallelism(const ArgList &Args, const Driver &D) {
+  unsigned Parallelism = 0;
+  Arg *LtoJobsArg = Args.getLastArg(options::OPT_flto_jobs_EQ);
+  if (LtoJobsArg &&
+      StringRef(LtoJobsArg->getValue()).getAsInteger(10, Parallelism))
+    D.Diag(diag::err_drv_invalid_int_value) << LtoJobsArg->getAsString(Args)
+                                            << LtoJobsArg->getValue();
+  return Parallelism;
+}
+
 static void AddGoldPlugin(const ToolChain &ToolChain, const ArgList &Args,
-                          ArgStringList &CmdArgs, bool IsThinLTO) {
+                          ArgStringList &CmdArgs, bool IsThinLTO,
+                          const Driver &D) {
   // Tell the linker to load the plugin. This has to come before AddLinkerInputs
   // as gold requires -plugin to come before any -plugin-opt that -Wl might
   // forward.
   CmdArgs.push_back("-plugin");
   std::string Plugin =
       ToolChain.getDriver().Dir + "/../lib" CLANG_LIBDIR_SUFFIX "/LLVMgold.so";
   CmdArgs.push_back(Args.MakeArgString(Plugin));
 
else if (A->getOption().matches(options::OPT_O0))
       OOpt = "0";
     if (!OOpt.empty())
       CmdArgs.push_back(Args.MakeArgString(Twine("-plugin-opt=O") + OOpt));
   }
 
   if (IsThinLTO)
     CmdArgs.push_back("-plugin-opt=thinlto");
 
+  if (unsigned Parallelism = getLTOParallelism(Args, D))
+    CmdArgs.push_back(Args.MakeArgString(Twine("-plugin-opt=jobs=") +
+                                         std::to_string(Parallelism)));
+
   // If an explicit debugger tuning argument appeared, pass it along.
   if (Arg *A = Args.getLastArg(options::OPT_gTune_Group,
                                options::OPT_ggdbN_Group)) {
     if (A->getOption().matches(options::OPT_glldb))
       CmdArgs.push_back("-plugin-opt=-debugger-tune=lldb");
     else if (A->getOption().matches(options::OPT_gsce))
       CmdArgs.push_back("-plugin-opt=-debugger-tune=sce");
     else
void cloudabi::Linker::ConstructJob(Compilation &C, const JobAction &JA,
 
   Args.AddAllArgs(CmdArgs, options::OPT_L);
   ToolChain.AddFilePathLibArgs(Args, CmdArgs);
   Args.AddAllArgs(CmdArgs,
                   {options::OPT_T_Group, options::OPT_e, options::OPT_s,
                    options::OPT_t, options::OPT_Z_Flag, options::OPT_r});
 
   if (D.isUsingLTO())
-    AddGoldPlugin(ToolChain, Args, CmdArgs, D.getLTOMode() == LTOK_Thin);
+    AddGoldPlugin(ToolChain, Args, CmdArgs, D.getLTOMode() == LTOK_Thin, D);
 
   AddLinkerInputs(ToolChain, Inputs, Args, CmdArgs);
 
   if (!Args.hasArg(options::OPT_nostdlib, options::OPT_nodefaultlibs)) {
     if (D.CCCIsCXX())
       ToolChain.AddCXXStdlibLibArgs(Args, CmdArgs);
     CmdArgs.push_back("-lc");
     CmdArgs.push_back("-lcompiler_rt");
if (LinkingOutput) {
     CmdArgs.push_back(LinkingOutput);
   }
 
   if (Args.hasArg(options::OPT_fnested_functions))
     CmdArgs.push_back("-allow_stack_execute");
 
   getMachOToolChain().addProfileRTLibs(Args, CmdArgs);
 
+  if (unsigned Parallelism =
+          getLTOParallelism(Args, getToolChain().getDriver())) {
+    CmdArgs.push_back("-mllvm");
+    CmdArgs.push_back(
+        Args.MakeArgString(Twine("-threads=") + std::to_string(Parallelism)));
+  }
+
   if (!Args.hasArg(options::OPT_nostdlib, options::OPT_nodefaultlibs)) {
     if (getToolChain().getDriver().CCCIsCXX())
       getToolChain().AddCXXStdlibLibArgs(Args, CmdArgs);
 
     // link_ssp spec is empty.
 
     // Let the tool chain choose which runtime library to link.
     getMachOToolChain().AddLinkRuntimeLibArgs(Args, CmdArgs);
void freebsd::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   Args.AddAllArgs(CmdArgs, options::OPT_T_Group);
   Args.AddAllArgs(CmdArgs, options::OPT_e);
   Args.AddAllArgs(CmdArgs, options::OPT_s);
   Args.AddAllArgs(CmdArgs, options::OPT_t);
   Args.AddAllArgs(CmdArgs, options::OPT_Z_Flag);
   Args.AddAllArgs(CmdArgs, options::OPT_r);
 
   if (D.isUsingLTO())
-    AddGoldPlugin(ToolChain, Args, CmdArgs, D.getLTOMode() == LTOK_Thin);
+    AddGoldPlugin(ToolChain, Args, CmdArgs, D.getLTOMode() == LTOK_Thin, D);
 
   bool NeedsSanitizerDeps = addSanitizerRuntimes(ToolChain, Args, CmdArgs);
   AddLinkerInputs(ToolChain, Inputs, Args, CmdArgs);
 
   if (!Args.hasArg(options::OPT_nostdlib, options::OPT_nodefaultlibs)) {
     addOpenMPRuntime(CmdArgs, ToolChain, Args);
     if (D.CCCIsCXX()) {
       ToolChain.AddCXXStdlibLibArgs(Args, CmdArgs);
void gnutools::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   }
 
   Args.AddAllArgs(CmdArgs, options::OPT_L);
   Args.AddAllArgs(CmdArgs, options::OPT_u);
 
   ToolChain.AddFilePathLibArgs(Args, CmdArgs);
 
   if (D.isUsingLTO())
-    AddGoldPlugin(ToolChain, Args, CmdArgs, D.getLTOMode() == LTOK_Thin);
+    AddGoldPlugin(ToolChain, Args, CmdArgs, D.getLTOMode() == LTOK_Thin, D);
 
   if (Args.hasArg(options::OPT_Z_Xlinker__no_demangle))
     CmdArgs.push_back("--no-demangle");
 
   bool NeedsSanitizerDeps = addSanitizerRuntimes(ToolChain, Args, CmdArgs);
   bool NeedsXRayDeps = addXRayRuntime(ToolChain, Args, CmdArgs);
   AddLinkerInputs(ToolChain, Inputs, Args, CmdArgs);
   // The profile runtime also needs access to system libraries.

test/Driver/lto-jobs.c

+// Confirm that -flto-jobs=N is passed to linker
+
+// RUN: %clang -target x86_64-unknown-linux -### %s -flto=thin -flto-jobs=5 2> %t
+// RUN: FileCheck -check-prefix=CHECK-LINK-THIN-JOBS-ACTION < %t %s
+//
+// CHECK-LINK-THIN-JOBS-ACTION: "-plugin-opt=jobs=5"
+
+// RUN: %clang -target x86_64-apple-darwin13.3.0 -### %s -flto=thin -flto-jobs=5 2> %t
+// RUN: FileCheck -check-prefix=CHECK-LINK-THIN-JOBS2-ACTION < %t %s
+//
+// CHECK-LINK-THIN-JOBS2-ACTION: "-mllvm" "-threads=5"