Clang support for -fthinlto.

[tejohnson]

Summary

I am going to update this patch in a moment to reflect recent changes needed for the new LLVM/gold patch (D13107), and add a couple of tests.

Add clang support for -fthinlto option, which is used to set the EmitThinLTOIndex code gen option on compiles.

Dependent on LLVM/gold patch http://reviews.llvm.org/D13107.

[tejohnson]

Removed gold plugin option.

[davidxl]

include/clang/Driver/Options.td

685

spell it like "fthin-lto" seems better than one word "fthinlto"

lib/CodeGen/BackendUtil.cpp

621–622

The change to this function is missing in the patch.

[tejohnson]

include/clang/Driver/Options.td

685

I considered that too, but wanted to distinguish from regular LTO. I could go either way.

lib/CodeGen/BackendUtil.cpp

621–622

Right, that is in LLVM and needs to be in a different patch (goes into a different repository). As mentioned in the patch summary this is dependent on LLVM patch http://reviews.llvm.org/D11907.

[tejohnson]

Updated the patch for the new LLVM/gold patch (D13107).

Two changes:

• I put back the original change to pass a new plugin option to gold. Since the function index design/name has been generalized to not be ThinLTO-specific, it makes sense to have the ThinLTO-related gold behavior for modules with those sections to be dependent on an option.
• Added 2 tests.

[tejohnson]

Address review comments.

Instead of -fthinlto use -flto=thin, and add in -flto=full as an alias to the existing -flto. Add tests to check for proper overriding of -flto variants on the command line, and convert grep tests to FileCheck.

[alexr]

I realize that we've all been referring to this as ThinLTO, but is that necessarily descriptive for users? What about inlineonly or something similar?

[davidxl]

ThinLTO is a perfectly good term to indicate what it tries to accomplish. InlineOnly will narrow its scope. For instance, there is no reason why thinLTO can not do fast summary based WPA in the future.

[klimek]

Perhaps "sharded" would fit what it is?

[tejohnson]

Added echristo, rafael, pcc officially as subscribers as Duncan's cc
of them via email didn't stick.

[tejohnson]

Sorry for the duplicate - my previous response didn't go to Duncan or Mehdi for some reason. Trying again...

In D11908#258540, @klimek wrote:

Perhaps "sharded" would fit what it is?

You could have a sharded mode for full FDO (like gcc's partitioned LTO). And we aren't really making any explicit sharding decisions, since the backends do importing on demand.

As David mentioned, "inlineonly" is much too restrictive for what is possible. I prefer to stick with "thin" since it refers to this new model of keeping the whole program part very thin.

Does anyone have an opinion on "full" vs "monolithic" vs something else for the traditional full/monolithic LTO?

[klimek]

In D11908#258570, @tejohnson wrote:

Sorry for the duplicate - my previous response didn't go to Duncan or Mehdi for some reason. Trying again...

In D11908#258540, @klimek wrote:

Perhaps "sharded" would fit what it is?

You could have a sharded mode for full FDO (like gcc's partitioned LTO). And we aren't really making any explicit sharding decisions, since the backends do importing on demand.

As David mentioned, "inlineonly" is much too restrictive for what is possible. I prefer to stick with "thin" since it refers to this new model of keeping the whole program part very thin.

Does anyone have an opinion on "full" vs "monolithic" vs something else for the traditional full/monolithic LTO?

If "sharded" is not the right term, than "monolithic" doesn't seem like the right term, either, right?

If "thin" basically refers to how much information is given to the lto steps (for which "thin" seems to be a good name actually), then "full" seems to be a good term for the, well, full information.

[tejohnson]

• Rename code gen option to EmitFunctionSummary as per review for D13107.
• Clang format the latest changes.
  
  Let me know if it looks good now.

[tejohnson]

Ping.
Thanks,
Teresa

[mehdi_amini]

LGTM

include/clang/Driver/Options.td

 def finstrument_functions : Flag<["-"], "finstrument-functions">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Generate calls to instrument function entry and exit">;
 def flat__namespace : Flag<["-"], "flat_namespace">;
 def flax_vector_conversions : Flag<["-"], "flax-vector-conversions">, Group<f_Group>;
 def flimited_precision_EQ : Joined<["-"], "flimited-precision=">, Group<f_Group>;
 def flto_EQ : Joined<["-"], "flto=">, Group<clang_ignored_gcc_optimization_f_Group>;
 def flto : Flag<["-"], "flto">, Flags<[CC1Option]>, Group<f_Group>;
 def fno_lto : Flag<["-"], "fno-lto">, Group<f_Group>;
+def fthinlto : Flag<["-"], "fthinlto">, Flags<[CC1Option]>, Group<f_Group>;
+def fno_thinlto : Flag<["-"], "fno-thinlto">, Group<f_Group>;
 def fmacro_backtrace_limit_EQ : Joined<["-"], "fmacro-backtrace-limit=">,
                                 Group<f_Group>, Flags<[DriverOption, CoreOption]>;
 def fmerge_all_constants : Flag<["-"], "fmerge-all-constants">, Group<f_Group>;
 def fmessage_length_EQ : Joined<["-"], "fmessage-length=">, Group<f_Group>;
 def fms_extensions : Flag<["-"], "fms-extensions">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Accept some non-standard constructs supported by the Microsoft compiler">;
 def fms_compatibility : Flag<["-"], "fms-compatibility">, Group<f_Group>, Flags<[CC1Option]>,
   HelpText<"Enable full Microsoft Visual C++ compatibility">;

include/clang/Frontend/CodeGenOptions.def

 CODEGENOPT(ForbidGuardVariables , 1, 0) ///< Issue errors if C++ guard variables
                                         ///< are required.
 CODEGENOPT(FunctionSections  , 1, 0) ///< Set when -ffunction-sections is enabled.
 CODEGENOPT(InstrumentFunctions , 1, 0) ///< Set when -finstrument-functions is
                                        ///< enabled.
 CODEGENOPT(InstrumentForProfiling , 1, 0) ///< Set when -pg is enabled.
 CODEGENOPT(LessPreciseFPMAD  , 1, 0) ///< Enable less precise MAD instructions to
                                      ///< be generated.
-CODEGENOPT(PrepareForLTO     , 1, 0) ///< Set when -flto is enabled on the
+CODEGENOPT(PrepareForLTO     , 1, 0) ///< Set when -flto or -fthinlto is enabled
+                                     ///< on the compile step.
+CODEGENOPT(EmitThinLTOIndex  , 1, 0) ///< Set when -fthinlto is enabled on the
                                      ///< compile step.
 CODEGENOPT(MergeAllConstants , 1, 1) ///< Merge identical constants.
 CODEGENOPT(MergeFunctions    , 1, 0) ///< Set when -fmerge-functions is enabled.
 CODEGENOPT(MSVolatile        , 1, 0) ///< Set when /volatile:ms is enabled.
 CODEGENOPT(NoCommon          , 1, 0) ///< Set when -fno-common or C++ is enabled.
 CODEGENOPT(NoDwarfDirectoryAsm , 1, 0) ///< Set when -fno-dwarf-directory-asm is
                                        ///< enabled.
 CODEGENOPT(NoExecStack       , 1, 0) ///< Set when -Wa,--noexecstack is enabled.

lib/CodeGen/BackendUtil.cpp

void EmitAssemblyHelper::EmitAssembly(BackendAction Action,
   CreatePasses();
 
   switch (Action) {
   case Backend_EmitNothing:
     break;
 
   case Backend_EmitBC:
     getPerModulePasses()->add(
-        createBitcodeWriterPass(*OS, CodeGenOpts.EmitLLVMUseLists));
+        createBitcodeWriterPass(*OS, CodeGenOpts.EmitLLVMUseLists,
+                                CodeGenOpts.EmitThinLTOIndex));
     break;
 
   case Backend_EmitLL:
     getPerModulePasses()->add(
         createPrintModulePass(*OS, "", CodeGenOpts.EmitLLVMUseLists));
     break;
 
   default:

lib/Driver/Driver.cpp

case phases::Assemble:
     return llvm::make_unique<AssembleJobAction>(std::move(Input),
                                                 types::TY_Object);
   }
 
   llvm_unreachable("invalid phase in ConstructPhaseAction");
 }
 
 bool Driver::IsUsingLTO(const ArgList &Args) const {
-  return Args.hasFlag(options::OPT_flto, options::OPT_fno_lto, false);
+  return Args.hasFlag(options::OPT_flto, options::OPT_fno_lto, false) ||
+      Args.hasFlag(options::OPT_fthinlto, options::OPT_fno_thinlto, false);
 }
 
 void Driver::BuildJobs(Compilation &C) const {
   llvm::PrettyStackTraceString CrashInfo("Building compilation jobs");
 
   Arg *FinalOutput = C.getArgs().getLastArg(options::OPT_o);
 
   // It is an error to provide a -o option if we are making multiple output

lib/Driver/Tools.cpp

static void AddGoldPlugin(const ToolChain &ToolChain, const ArgList &Args,
 
   // Try to pass driver level flags relevant to LTO code generation down to
   // the plugin.
 
   // Handle flags for selecting CPU variants.
   std::string CPU = getCPUName(Args, ToolChain.getTriple());
   if (!CPU.empty())
     CmdArgs.push_back(Args.MakeArgString(Twine("-plugin-opt=mcpu=") + CPU));
+
+  if (Args.hasArg(options::OPT_fthinlto))
+    CmdArgs.push_back("-plugin-opt=thinlto");
 }
 
 /// This is a helper function for validating the optional refinement step
 /// parameter in reciprocal argument strings. Return false if there is an error
 /// parsing the refinement step. Otherwise, return true and set the Position
 /// of the refinement step in the input string.
 static bool getRefinementStep(const StringRef &In, const Driver &D,
                               const Arg &A, size_t &Position) {

 // Claim options we don't want to warn if they are unused. We do this for
 // options that build systems might add but are unused when assembling or only
 // running the preprocessor for example.
 static void claimNoWarnArgs(const ArgList &Args) {
   // Don't warn about unused -f(no-)?lto.  This can happen when we're
   // preprocessing, precompiling or assembling.
   Args.ClaimAllArgs(options::OPT_flto);
   Args.ClaimAllArgs(options::OPT_fno_lto);
+  Args.ClaimAllArgs(options::OPT_fthinlto);
+  Args.ClaimAllArgs(options::OPT_fno_thinlto);
 }
 
 static void appendUserToPath(SmallVectorImpl<char> &Result) {
 #ifdef LLVM_ON_UNIX
   const char *Username = getenv("LOGNAME");
 #else
   const char *Username = getenv("USERNAME");
 #endif
else
       CmdArgs.push_back("-emit-pth");
   } else if (isa<VerifyPCHJobAction>(JA)) {
     CmdArgs.push_back("-verify-pch");
   } else {
     assert((isa<CompileJobAction>(JA) || isa<BackendJobAction>(JA)) &&
            "Invalid action for clang tool.");
 
     if (JA.getType() == types::TY_LTO_IR || JA.getType() == types::TY_LTO_BC) {
+      // Note that both -flto and -fthinlto use the TY_LTO_* types.
+      // Setting -flto here will enable the PrepareForLTO codegen option,
+      // which we do want for -fthinlto compiles as well.
       CmdArgs.push_back("-flto");
     }
     if (JA.getType() == types::TY_Nothing) {
       CmdArgs.push_back("-fsyntax-only");
     } else if (JA.getType() == types::TY_LLVM_IR ||
                JA.getType() == types::TY_LTO_IR) {
       CmdArgs.push_back("-emit-llvm");
     } else if (JA.getType() == types::TY_LLVM_BC ||
if (isa<AnalyzeJobAction>(JA)) {
     }
 
     // Preserve use-list order by default when emitting bitcode, so that
     // loading the bitcode up in 'opt' or 'llc' and running passes gives the
     // same result as running passes here.  For LTO, we don't need to preserve
     // the use-list order, since serialization to bitcode is part of the flow.
     if (JA.getType() == types::TY_LLVM_BC)
       CmdArgs.push_back("-emit-llvm-uselists");
+
+    if (Args.hasArg(options::OPT_fthinlto)) {
+      CmdArgs.push_back("-fthinlto");
+    }
   }
 
   // We normally speed up the clang process a bit by skipping destructors at
   // exit, but when we're generating diagnostics we can rely on some of the
   // cleanup.
   if (!C.isForDiagnostics())
     CmdArgs.push_back("-disable-free");
 

lib/Frontend/CompilerInvocation.cpp

static bool ParseCodeGenArgs(CodeGenOptions &Opts, ArgList &Args, InputKind IK,
   Opts.DataSections = Args.hasFlag(OPT_fdata_sections,
                                    OPT_fno_data_sections, false);
   Opts.UniqueSectionNames = Args.hasFlag(OPT_funique_section_names,
                                          OPT_fno_unique_section_names, true);
 
   Opts.MergeFunctions = Args.hasArg(OPT_fmerge_functions);
 
   Opts.PrepareForLTO = Args.hasArg(OPT_flto);
+  Opts.EmitThinLTOIndex = Args.hasArg(OPT_fthinlto);
 
   Opts.MSVolatile = Args.hasArg(OPT_fms_volatile);
 
   Opts.VectorizeBB = Args.hasArg(OPT_vectorize_slp_aggressive);
   Opts.VectorizeLoop = Args.hasArg(OPT_vectorize_loops);
   Opts.VectorizeSLP = Args.hasArg(OPT_vectorize_slp);
 
   Opts.MainFileName = Args.getLastArgValue(OPT_main_file_name);