[WIP] First draft of -ffat-lto-object related changes: front-end additions, duplication and repurposing -fembed-bitcode related code.

[arda]

Summary

LLVM users currently have to choose at build-time whether to generate
object code or bitcode. Only having these options can be limiting since
translation units can often be shared between many targets. These different
targets may have different requirements in terms of compile time or
performance. For instance, the benefits for LTO might be worth the overhead for
main targets whereas this might not be the case for the tests.

GCC allows GIMPLE bytecode to be saved alongside object code if the
-ffat-lto-objects option is passed. This “fat” object format allows one to
build one set of fat objects which could be used for targets with different
requirements since the decision to use LTO can be made at link time.

Our goal is to implement support for emitting bitcode alongside object code in
LLVM and use that support to implement the -ffat-lto-objects option in Clang.
Doing so would bring Clang to parity with GCC. We will initially focus on ELF,
but this support could be extended to other formats later.

[arda]

Remove the unrelated format changes.

[paulkirth]

Can we get the proposed LLD changes as well. Separate patches are probably best.

clang/lib/CodeGen/BackendUtil.cpp

196

I'd drop this, since its unrelated. If you still want to fix it, you can submit a separate change.

clang/lib/Driver/ToolChains/Clang.cpp

7120

I think these can go away, right?

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

4959

Can you make the error a bit more specific? Like let them know this is a "Fat LTO" problem? I'm not sure how well the issue will be surfaced as is.

There's been some discussion re: the use of unreachable. Not all of it applies here, especially in a prototype, but its a point to consider as work develops.

see https://discourse.llvm.org/t/llvm-unreachable-is-widely-misused/60587 for more context.

I know this is a copy paste from existing code, but a fatal error might be the better choice. What do you think?

5080

Does this just do what embed bitcode does? My understanding was that LTO generated IR is emitted into object files differently than the embed bitcode scenario. I'm basing that on the discussion at https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977

[paulkirth]

I'd also like to see tests for this.

[arda]

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

4959

From that discussion, it seems like unreachable should not be used in this case. I agree that fatal error might be a better choice here. Thanks!

5080

From my research, -fembed-bitcode was originally used for recompilation purposes, not for LTO. Since our scope is limited to ThinLTO, this prototype currently is used to embed ThinLTO bitcode into the FatLTO object as follows:

clang -c -ffat-lto-objects -o fatLTO.o -x ir thinLTO.bc

[arda]

Addressed Paul's first comments.

[paulkirth]

clang/include/clang/Driver/ToolChain.h

572

This file mostly uses toolchain, so let's try to be consistent. We don't need to worry about other uses in this patch.

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

5080

Looking at this a second time, I'm not sure I follow your meaning here.

What I meant was that the way EmbedBitcode worked was not TheRightThing according to that discussion. It appears as though your change just copies its logic.

So if I want to compile a C file and get a fat-LTO object? When do the LTO passes get added to the pipeline? I don't mean the big whole program optimization stuff that gets invoked at link-time, I mean the module level passes and summary generation? It's not clear to me when those will run after invoking clang on a source file.

can you elaborate on how this is intended to work? I seem to be missing something.

[phosek]

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

4965

Since this section is LLVM specific, I think it should use an .llvm prefix, so the name could be .llvm.lto. This is similar to GCC which uses .gcc.lto_ prefixed section names.

[arda]

The changes in this patch have been moved to here: https://reviews.llvm.org/D131618

clang/include/clang/Basic/CodeGenOptions.def

 CODEGENOPT(LessPreciseFPMAD  , 1, 0) ///< Enable less precise MAD instructions to
                                      ///< be generated.
 CODEGENOPT(PrepareForLTO     , 1, 0) ///< Set when -flto is enabled on the
                                      ///< compile step.
 CODEGENOPT(PrepareForThinLTO , 1, 0) ///< Set when -flto=thin is enabled on the
                                      ///< compile step.
 CODEGENOPT(LTOUnit, 1, 0) ///< Emit IR to support LTO unit features (CFI, whole
                           ///< program vtable opt).
+CODEGENOPT(FatLTO, 1, 0) ///< Set when -ffat-lto-objects is enabled.
 CODEGENOPT(EnableSplitLTOUnit, 1, 0) ///< Enable LTO unit splitting to support
              /// CFI and traditional whole program
              /// devirtualization that require whole
              /// program IR support.
 CODEGENOPT(IncrementalLinkerCompatible, 1, 0) ///< Emit an object file which can
                                               ///< be used with an incremental
                                               ///< linker.
 CODEGENOPT(MergeAllConstants , 1, 1) ///< Merge identical constants.

clang/include/clang/CodeGen/BackendUtil.h

void EmitBackendOutput(DiagnosticsEngine &Diags, const HeaderSearchOptions &,
                          const CodeGenOptions &CGOpts,
                          const TargetOptions &TOpts, const LangOptions &LOpts,
                          StringRef TDesc, llvm::Module *M, BackendAction Action,
                          std::unique_ptr<raw_pwrite_stream> OS);
 
   void EmbedBitcode(llvm::Module *M, const CodeGenOptions &CGOpts,
                     llvm::MemoryBufferRef Buf);
 
+  void EmbedFatLTO(llvm::Module *M, const CodeGenOptions &CGOpts,
+                   llvm::MemoryBufferRef Buf);
+
   void EmbedObject(llvm::Module *M, const CodeGenOptions &CGOpts,
                    DiagnosticsEngine &Diags);
 }
 
 #endif

clang/include/clang/Driver/Options.td

 def flto_EQ : Joined<["-"], "flto=">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   HelpText<"Set LTO mode">, Values<"thin,full">;
 def flto_EQ_jobserver : Flag<["-"], "flto=jobserver">, Group<f_Group>,
   Alias<flto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode">;
 def flto_EQ_auto : Flag<["-"], "flto=auto">, Group<f_Group>,
   Alias<flto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode">;
 def flto : Flag<["-"], "flto">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   Alias<flto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode">;
+defm fat_lto_objects : BoolFOption<"fat-lto-objects",
+  CodeGenOpts<"FatLTO">, DefaultFalse,
+  PosFlag<SetTrue, [], "Enable fat LTO object support">,
+  NegFlag<SetFalse, [], "Disable fat LTO object support">,
+  BothFlags<[CC1Option], "">>;
 def fno_lto : Flag<["-"], "fno-lto">, Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   HelpText<"Disable LTO mode (default)">;
 def foffload_lto_EQ : Joined<["-"], "foffload-lto=">, Flags<[CoreOption]>, Group<f_Group>,
   HelpText<"Set LTO mode for offload compilation">, Values<"thin,full">;
 def foffload_lto : Flag<["-"], "foffload-lto">, Flags<[CoreOption]>, Group<f_Group>,
   Alias<foffload_lto_EQ>, AliasArgs<["full"]>, HelpText<"Enable LTO in 'full' mode for offload compilation">;
 def fno_offload_lto : Flag<["-"], "fno-offload-lto">, Flags<[CoreOption]>, Group<f_Group>,
   HelpText<"Disable LTO mode (default) for offload compilation">;

 // ignore it for now to avoid breaking builds that use it.
 def fdiagnostics_show_location_EQ : Joined<["-"], "fdiagnostics-show-location=">, Group<clang_ignored_f_Group>;
 
 defm fcheck_new : BooleanFFlag<"check-new">, Group<clang_ignored_f_Group>;
 defm caller_saves : BooleanFFlag<"caller-saves">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm reorder_blocks : BooleanFFlag<"reorder-blocks">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm branch_count_reg : BooleanFFlag<"branch-count-reg">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm default_inline : BooleanFFlag<"default-inline">, Group<clang_ignored_gcc_optimization_f_Group>;
-defm fat_lto_objects : BooleanFFlag<"fat-lto-objects">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm float_store : BooleanFFlag<"float-store">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm friend_injection : BooleanFFlag<"friend-injection">, Group<clang_ignored_f_Group>;
 defm function_attribute_list : BooleanFFlag<"function-attribute-list">, Group<clang_ignored_f_Group>;
 defm gcse : BooleanFFlag<"gcse">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gcse_after_reload: BooleanFFlag<"gcse-after-reload">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gcse_las: BooleanFFlag<"gcse-las">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gcse_sm: BooleanFFlag<"gcse-sm">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gnu : BooleanFFlag<"gnu">, Group<clang_ignored_f_Group>;

clang/include/clang/Driver/ToolChain.h

public:
 
   /// GetExceptionModel - Return the tool chain exception model.
   virtual llvm::ExceptionHandling
   GetExceptionModel(const llvm::opt::ArgList &Args) const;
 
   /// SupportsEmbeddedBitcode - Does this tool chain support embedded bitcode.
   virtual bool SupportsEmbeddedBitcode() const { return false; }
 
+  /// SupportsFatLTO - Does this tool chain support fat LTO objects.
+  virtual bool SupportsFatLTO() const { return true; }
+
   /// getThreadModel() - Which thread model does this target use?
   virtual std::string getThreadModel() const { return "posix"; }
 
   /// isThreadModelSupported() - Does this target support a thread model?
   virtual bool isThreadModelSupported(const StringRef Model) const;
 
   virtual std::string getMultiarchTriple(const Driver &D,
                                          const llvm::Triple &TargetTriple,

clang/lib/CodeGen/BackendUtil.cpp

public:
   }
 
   std::unique_ptr<TargetMachine> TM;
 
   // Emit output using the new pass manager for the optimization pipeline.
   void EmitAssembly(BackendAction Action,
                     std::unique_ptr<raw_pwrite_stream> OS);
 };
 }
 
 static SanitizerCoverageOptions
 getSancovOptsFromCGOpts(const CodeGenOptions &CGOpts) {
   SanitizerCoverageOptions Opts;
   Opts.CoverageType =
       static_cast<SanitizerCoverageOptions::Type>(CGOpts.SanitizeCoverageType);
   Opts.IndirectCalls = CGOpts.SanitizeCoverageIndirectCalls;
   Opts.TraceBB = CGOpts.SanitizeCoverageTraceBB;
if (AsmHelper.TM) {
     }
   }
 }
 
 // With -fembed-bitcode, save a copy of the llvm IR as data in the
 // __LLVM,__bitcode section.
 void clang::EmbedBitcode(llvm::Module *M, const CodeGenOptions &CGOpts,
                          llvm::MemoryBufferRef Buf) {
+  if (CGOpts.FatLTO)
+    return;
   if (CGOpts.getEmbedBitcode() == CodeGenOptions::Embed_Off)
     return;
   llvm::embedBitcodeInModule(
       *M, Buf, CGOpts.getEmbedBitcode() != CodeGenOptions::Embed_Marker,
       CGOpts.getEmbedBitcode() != CodeGenOptions::Embed_Bitcode,
       CGOpts.CmdArgs);
 }
 
+// With -ffat-lto-objects, save a copy of the llvm IR as data in the
+// .fatlto section.
+void clang::EmbedFatLTO(llvm::Module *M, const CodeGenOptions &CGOpts,
+                        llvm::MemoryBufferRef Buf) {
+  if (!CGOpts.FatLTO)
+    return;
+  llvm::embedBitcodeInFatObject(*M, Buf);
+}
+
 void clang::EmbedObject(llvm::Module *M, const CodeGenOptions &CGOpts,
                         DiagnosticsEngine &Diags) {
   if (CGOpts.OffloadObjects.empty())
     return;
 
   for (StringRef OffloadObject : CGOpts.OffloadObjects) {
     llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> ObjectOrErr =
         llvm::MemoryBuffer::getFileOrSTDIN(OffloadObject);

clang/lib/CodeGen/CodeGenAction.cpp

void HandleTranslationUnit(ASTContext &C) override {
         // things in ASTContext used after this point and null out the
         // ASTContext, but too many various parts of the ASTContext are still
         // used in various parts.
         C.cleanup();
         C.getAllocator().Reset();
       }
 
       EmbedBitcode(getModule(), CodeGenOpts, llvm::MemoryBufferRef());
+      EmbedFatLTO(getModule(), CodeGenOpts, llvm::MemoryBufferRef());
 
       EmitBackendOutput(Diags, HeaderSearchOpts, CodeGenOpts, TargetOpts,
                         LangOpts, C.getTargetInfo().getDataLayoutString(),
                         getModule(), Action, std::move(AsmOutStream));
 
       Ctx.setDiagnosticHandler(std::move(OldDiagnosticHandler));
 
       if (OptRecordFile)
void CodeGenAction::ExecuteAction() {
   if (TheModule->getTargetTriple() != TargetOpts.Triple) {
     Diagnostics.Report(SourceLocation(), diag::warn_fe_override_module)
         << TargetOpts.Triple;
     TheModule->setTargetTriple(TargetOpts.Triple);
   }
 
   EmbedObject(TheModule.get(), CodeGenOpts, Diagnostics);
   EmbedBitcode(TheModule.get(), CodeGenOpts, *MainFile);
+  EmbedFatLTO(TheModule.get(), CodeGenOpts, *MainFile);
+
 
   LLVMContext &Ctx = TheModule->getContext();
 
   // Restore any diagnostic handler previously set before returning from this
   // function.
   struct RAII {
     LLVMContext &Ctx;
     std::unique_ptr<DiagnosticHandler> PrevHandler = Ctx.getDiagnosticHandler();

clang/lib/Driver/Driver.cpp

unsigned Model = llvm::StringSwitch<unsigned>(Name)
         .Default(~0U);
     if (Model == ~0U) {
       Diags.Report(diag::err_drv_invalid_value) << A->getAsString(Args)
                                                 << Name;
     } else
       BitcodeEmbed = static_cast<BitcodeEmbedMode>(Model);
   }
 
+  // Process -ffat-lto-objects flag.
+  if (Arg *A = Args.getLastArg(options::OPT_ffat_lto_objects)) {
+    BitcodeEmbed = EmbedBitcode;
+  }
+
   // Remove existing compilation database so that each job can append to it.
   if (Arg *A = Args.getLastArg(options::OPT_MJ))
     llvm::sys::fs::remove(A->getValue());
 
   // Setting up the jobs for some precompile cases depends on whether we are
   // treating them as PCH, implicit modules or C++20 ones.
   // TODO: inferring the mode like this seems fragile (it meets the objective
   // of not requiring anything new for operation, however).

clang/lib/Driver/ToolChains/Clang.cpp

if (A->getOption().matches(options::OPT_fglobal_isel)) {
         else
           D.Diag(diag::warn_drv_global_isel_incomplete_opt);
       }
     } else {
       CmdArgs.push_back("-global-isel=0");
     }
   }
 
+  if (Args.hasArg(options::OPT_ffat_lto_objects))
+    CmdArgs.push_back("-ffat-lto-objects");
+
   if (Args.hasArg(options::OPT_forder_file_instrumentation)) {
      CmdArgs.push_back("-forder-file-instrumentation");
      // Enable order file instrumentation when ThinLTO is not on. When ThinLTO is
      // on, we need to pass these flags as linker flags and that will be handled
      // outside of the compiler.
      if (!IsUsingLTO) {
        CmdArgs.push_back("-mllvm");
        CmdArgs.push_back("-enable-order-file-instrumentation");

llvm/include/llvm/Bitcode/BitcodeWriter.h

class raw_ostream;
   /// is not bitcode (i.e. if it's invalid data or even textual LLVM assembly).
   /// If EmbedCmdline is set, the command line is also exported in
   /// the corresponding section (__LLVM,_cmdline / .llvmcmd) - even if CmdArgs
   /// were empty.
   void embedBitcodeInModule(Module &M, MemoryBufferRef Buf, bool EmbedBitcode,
                             bool EmbedCmdline,
                             const std::vector<uint8_t> &CmdArgs);
 
+  void embedBitcodeInFatObject(Module &M, MemoryBufferRef Buf);
 } // end namespace llvm
 
 #endif // LLVM_BITCODE_BITCODEWRITER_H

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

case Triple::XCOFF:
     break;
   case Triple::DXContainer:
     llvm_unreachable("DXC is not yet implemented");
     break;
   }
   llvm_unreachable("Unimplemented ObjectFormatType");
 }
 
+static const char *getSectionNameForBitcodeForFatLTO(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+    llvm::report_fatal_error("MachO is not yet implemented for FatLTO");
+    break;
+  case Triple::COFF:
+    llvm::report_fatal_error("COFF is not yet implemented for FatLTO");
+    break;
+  case Triple::ELF:
+    return ".fatlto";
+  case Triple::Wasm:
+    llvm::report_fatal_error("Wasm is not yet implemented for FatLTO");
+    break;
+  case Triple::UnknownObjectFormat:
+    return ".fatlto";
+  case Triple::GOFF:
+    llvm::report_fatal_error("GOFF is not yet implemented for FatLTO");
+    break;
+  case Triple::SPIRV:
+    llvm::report_fatal_error("SPIRV is not yet implemented for FatLTO");
+    break;
+  case Triple::XCOFF:
+    llvm::report_fatal_error("XCOFF is not yet implemented for FatLTO");
+    break;
+  case Triple::DXContainer:
+    llvm::report_fatal_error("DXContainer is not yet implemented for FatLTO");
+    break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
 void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
                                 bool EmbedBitcode, bool EmbedCmdline,
                                 const std::vector<uint8_t> &CmdArgs) {
   // Save llvm.compiler.used and remove it.
   SmallVector<Constant *, 2> UsedArray;
   SmallVector<GlobalValue *, 4> UsedGlobals;
   Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
   GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
 
   // Recreate llvm.compiler.used.
   ArrayType *ATy = ArrayType::get(UsedElementType, UsedArray.size());
   auto *NewUsed = new GlobalVariable(
       M, ATy, false, llvm::GlobalValue::AppendingLinkage,
       llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
   NewUsed->setSection("llvm.metadata");
 }
+
+void llvm::embedBitcodeInFatObject(llvm::Module &M, llvm::MemoryBufferRef Buf) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector<Constant *, 2> UsedArray;
+  SmallVector<GlobalValue *, 4> UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+    if (GV->getName() != "llvm.embedded.module" &&
+        GV->getName() != "llvm.cmdline")
+      UsedArray.push_back(
+          ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+    Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef<uint8_t> ModuleData;
+  Triple T(M.getTargetTriple());
+
+  if (Buf.getBufferSize() == 0 ||
+      !isBitcode((const unsigned char *)Buf.getBufferStart(),
+                 (const unsigned char *)Buf.getBufferEnd())) {
+    // If the input is LLVM Assembly, bitcode is produced by serializing
+    // the module. Use-lists order need to be preserved in this case.
+    llvm::raw_string_ostream OS(Data);
+    llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+    ModuleData =
+        ArrayRef<uint8_t>((const uint8_t *)OS.str().data(), OS.str().size());
+  } else
+    // If the input is LLVM bitcode, write the input byte stream directly.
+    ModuleData = ArrayRef<uint8_t>((const uint8_t *)Buf.getBufferStart(),
+                                   Buf.getBufferSize());
+  llvm::Constant *ModuleConstant =
+      llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new llvm::GlobalVariable(
+      M, ModuleConstant->getType(), true, llvm::GlobalValue::PrivateLinkage,
+      ModuleConstant);
+  GV->setSection(getSectionNameForBitcodeForFatLTO(T));
+  // Set alignment to 1 to prevent padding between two contributions from input
+  // sections after linking.
+  GV->setAlignment(Align(1));
+  UsedArray.push_back(
+      ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  if (llvm::GlobalVariable *Old =
+          M.getGlobalVariable("llvm.embedded.module", true)) {
+    assert(Old->hasZeroLiveUses() &&
+           "llvm.embedded.module can only be used once in llvm.compiler.used");
+    GV->takeName(Old);
+    Old->eraseFromParent();
+  } else {
+    GV->setName("llvm.embedded.module");
+  }
+
+  if (UsedArray.empty())
+    return;
+
+  // Recreate llvm.compiler.used.
+  ArrayType *ATy = ArrayType::get(UsedElementType, UsedArray.size());
+  auto *NewUsed = new GlobalVariable(
+      M, ATy, false, llvm::GlobalValue::AppendingLinkage,
+      llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
+  NewUsed->setSection("llvm.metadata");
+}