diff --git a/llvm/docs/FatLTO.rst b/llvm/docs/FatLTO.rst
new file mode 100644
--- /dev/null
+++ b/llvm/docs/FatLTO.rst
@@ -0,0 +1,75 @@
+===================
+FatLTO Support
+===================
+.. contents::
+
+.. toctree::
+ :maxdepth: 1
+
+Introduction
+============
+
+FatLTO objects are a special type of `fat object file
+<https://en.wikipedia.org/wiki/Fat_binary>`_ that contain LTO compatible IR in
+addition to generated object code, instead of containing object code for
+multiple target architectures. This allows users to defer the choice of whether
+to use LTO or not to link-time, and has been a feature available in other
+compilers, like `GCC
+<https://gcc.gnu.org/onlinedocs/gccint/LTO-Overview.html>`_ for some time.
+
+Under FatLTO the compiler can emit standard object files which contain both the
+machine code in the ``.text`` section and LLVM bitcode in the ``.llvm.lto``
+section.
+
+Overview
+========
+
+Within LLVM, FatLTO is supported by choosing the ``FatLTODefaultPipeline``.
+This pipeline will:
+
+#) Clone the IR module.
+#) Run the pre-link (Thin)LTO pipeline using the cloned module.
+#) Embed the pre-link bitcode in a special ``.llvm.lto`` section.
+#) Optimize the unmodified copy of the module using the normal compilation pipeline.
+#) Emit the object file, including the new ``.llvm.lto`` section.
+
+.. NOTE
+
+ At the time of writing, we conservatively run independent pipelines to
+ generate the bitcode section and the object code, which happen to be
+ identical to those used outside of FatLTO. This results in compiled
+ artifacts that are identical to those produced by the default and (Thin)LTO
+ pipelines. However, this is not a guarantee, and we reserve the right to
+ change this at any time. Explicitly, users should not rely on the produced
+ bitcode or object code to mach their non-LTO counterparts precisely. They
+ will exhibit similar performance characteristics, but may not be bit-for-bit
+ the same.
+
+Internally, the ``.llvm.lto`` section is created by running the
+``EmbedBitcodePass`` at the start of the ``PerModuleDefaultPipeline``. This
+pass is responsible for cloning and optimizing the module with the appropriate
+LTO pipeline and emitting the ``.llvm.lto`` section. Afterwards, the
+``PerModuleDefaultPipeline`` runs normally and the compiler can emit the fat
+object file.
+
+Limitations
+===========
+
+Linkers
+-------
+
+Currently, using LTO with LLVM fat lto objects is supported by LLD and by the
+GNU linkers via the LLVMGold plugin. This may change in the future, but
+extending support to other linkers isn't planned for now.
+
+.. NOTE
+ For standard linking the fat object files should be usable by any
+ linker capable of using ELF objects, since the ``.llvm.lto`` section is
+ marked ``SHF_EXLUDE``.
+
+Supported File Formats
+----------------------
+
+The current implementation only supports ELF files. At time of writing, it is
+unclear if it will be useful to support other object file formats like ``COFF``
+or ``Mach-O``.
diff --git a/llvm/docs/ReleaseNotes.rst b/llvm/docs/ReleaseNotes.rst
--- a/llvm/docs/ReleaseNotes.rst
+++ b/llvm/docs/ReleaseNotes.rst
@@ -80,6 +80,11 @@
* InstructionSimplify APIs now require instructions be inserted into a
parent function.
+* A new FatLTO pipeline was added to support generating object files that have
+ both machine code and LTO compatible bitcode. See the `Documentation <FatLTOSupport.html>`_
+ and original `RFC <https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977>`_
+ for more details.
+
Changes to building LLVM
------------------------
diff --git a/llvm/docs/UserGuides.rst b/llvm/docs/UserGuides.rst
--- a/llvm/docs/UserGuides.rst
+++ b/llvm/docs/UserGuides.rst
@@ -32,6 +32,7 @@
DebuggingJITedCode
DirectXUsage
Docker
+ FatLTOSupport
ExtendingLLVM
GoldPlugin
HowToBuildOnARM
diff --git a/llvm/include/llvm/Bitcode/EmbedBitcodePass.h b/llvm/include/llvm/Bitcode/EmbedBitcodePass.h
new file mode 100644
--- /dev/null
+++ b/llvm/include/llvm/Bitcode/EmbedBitcodePass.h
@@ -0,0 +1,53 @@
+//===-- EmbedBitcodePass.h - Embeds bitcode into global ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// This file provides a pass which embeds the bitcode into a global variable.
+///
+//===----------------------------------------------------------------------===//
+//
+#ifndef LLVM_BITCODE_EMBEDBITCODEPASS_H
+#define LLVM_BITCODE_EMBEDBITCODEPASS_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+class Module;
+class ModulePass;
+class Pass;
+
+struct EmbedBitcodeOptions {
+ EmbedBitcodeOptions() : EmbedBitcodeOptions(false, false) {}
+ EmbedBitcodeOptions(bool IsThinLTO, bool EmitLTOSummary)
+ : IsThinLTO(IsThinLTO), EmitLTOSummary(EmitLTOSummary) {}
+ bool IsThinLTO;
+ bool EmitLTOSummary;
+};
+
+/// Pass embeds the current module into a global variable.
+class EmbedBitcodePass : public PassInfoMixin<EmbedBitcodePass> {
+ bool IsThinLTO;
+ bool EmitLTOSummary;
+ ModulePassManager MPM;
+
+public:
+ EmbedBitcodePass(EmbedBitcodeOptions Opts)
+ : EmbedBitcodePass(Opts.IsThinLTO, Opts.EmitLTOSummary,
+ ModulePassManager()) {}
+ EmbedBitcodePass(bool IsThinLTO, bool EmitLTOSummary, ModulePassManager &&MPM)
+ : IsThinLTO(IsThinLTO), EmitLTOSummary(EmitLTOSummary),
+ MPM(std::move(MPM)) {}
+
+ PreservedAnalyses run(Module &M, ModuleAnalysisManager &);
+
+ static bool isRequired() { return true; }
+};
+
+} // end namespace llvm.
+
+#endif
diff --git a/llvm/include/llvm/Passes/PassBuilder.h b/llvm/include/llvm/Passes/PassBuilder.h
--- a/llvm/include/llvm/Passes/PassBuilder.h
+++ b/llvm/include/llvm/Passes/PassBuilder.h
@@ -234,6 +234,18 @@
ModulePassManager buildPerModuleDefaultPipeline(OptimizationLevel Level,
bool LTOPreLink = false);
+ /// Build a fat object default optimization pipeline.
+ ///
+ /// This builds a pipeline that runs the LTO/ThinLTO pre-link pipeline, and
+ /// emits a section containing the pre-link bitcode along side the object code
+ /// generated by running the PerModuleDefaultPipeline, used when compiling
+ /// without LTO. It clones the module and runs the LTO/non-LTO pipelines
+ /// separately to avoid any inconsistencies with an ad-hoc pipeline that tries
+ /// to approximate the PerModuleDefaultPipeline from the pre-link LTO
+ /// pipelines.
+ ModulePassManager buildFatLTODefaultPipeline(OptimizationLevel Level,
+ bool ThinLTO, bool EmitSummary);
+
/// Build a pre-link, ThinLTO-targeting default optimization pipeline to
/// a pass manager.
///
diff --git a/llvm/lib/Bitcode/Writer/CMakeLists.txt b/llvm/lib/Bitcode/Writer/CMakeLists.txt
--- a/llvm/lib/Bitcode/Writer/CMakeLists.txt
+++ b/llvm/lib/Bitcode/Writer/CMakeLists.txt
@@ -2,6 +2,7 @@
BitWriter.cpp
BitcodeWriter.cpp
BitcodeWriterPass.cpp
+ EmbedBitcodePass.cpp
ValueEnumerator.cpp
DEPENDS
diff --git a/llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp b/llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp
new file mode 100644
--- /dev/null
+++ b/llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp
@@ -0,0 +1,59 @@
+//===- EmbedBitcodePass.cpp - Pass that embeds the bitcode into a global---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// EmbedBitcodePass implementation.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Bitcode/EmbedBitcodePass.h"
+#include "llvm/Bitcode/BitcodeWriter.h"
+#include "llvm/Bitcode/BitcodeWriterPass.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Passes/PassBuilder.h"
+#include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/TargetParser/Triple.h"
+#include "llvm/Transforms/IPO/ThinLTOBitcodeWriter.h"
+#include "llvm/Transforms/Utils/Cloning.h"
+#include "llvm/Transforms/Utils/ModuleUtils.h"
+
+using namespace llvm;
+
+PreservedAnalyses EmbedBitcodePass::run(Module &M, ModuleAnalysisManager &AM) {
+ if (M.getGlobalVariable("llvm.embedded.module", /*AllowInternal=*/true))
+ report_fatal_error("Can only embed the module once.",
+ /*gen_crash_diag=*/false);
+
+ Triple T(M.getTargetTriple());
+ if (T.getObjectFormat() != Triple::ELF)
+ report_fatal_error(
+ "EmbedBitcode pass currently only supports ELF object format.",
+ /*gen_crash_diag=*/false);
+
+ std::unique_ptr<Module> NewModule = CloneModule(M);
+
+ MPM.run(*NewModule, AM);
+
+ std::string Data;
+ raw_string_ostream OS(Data);
+
+ if (IsThinLTO)
+ ThinLTOBitcodeWriterPass(OS, /*ThinLinkOS=*/nullptr).run(*NewModule, AM);
+ else
+ BitcodeWriterPass(OS, /*ShouldPreserveUseListOrder=*/false, EmitLTOSummary)
+ .run(*NewModule, AM);
+
+ StringRef ModuleData(OS.str().data(), OS.str().size());
+ MemoryBufferRef Buf(ModuleData, "ModuleData");
+ embedBufferInModule(M, Buf, ".llvm.lto");
+
+ return PreservedAnalyses::all();
+}
diff --git a/llvm/lib/Object/ObjectFile.cpp b/llvm/lib/Object/ObjectFile.cpp
--- a/llvm/lib/Object/ObjectFile.cpp
+++ b/llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
Expected<StringRef> NameOrErr = getSectionName(Sec);
if (NameOrErr)
- return *NameOrErr == ".llvmbc";
+ return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
consumeError(NameOrErr.takeError());
return false;
}
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -71,6 +71,7 @@
#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Analysis/TypeBasedAliasAnalysis.h"
#include "llvm/Analysis/UniformityAnalysis.h"
+#include "llvm/Bitcode/EmbedBitcodePass.h"
#include "llvm/CodeGen/HardwareLoops.h"
#include "llvm/CodeGen/TypePromotion.h"
#include "llvm/IR/DebugInfo.h"
@@ -738,6 +739,26 @@
return Result;
}
+Expected<EmbedBitcodeOptions> parseEmbedBitcodePassOptions(StringRef Params) {
+ EmbedBitcodeOptions Result;
+ while (!Params.empty()) {
+ StringRef ParamName;
+ std::tie(ParamName, Params) = Params.split(';');
+
+ if (ParamName == "thinlto") {
+ Result.IsThinLTO = true;
+ } else if (ParamName == "emit-summary") {
+ Result.EmitLTOSummary = true;
+ } else {
+ return make_error<StringError>(
+ formatv("invalid EmbedBitcode pass parameter '{0}' ", ParamName)
+ .str(),
+ inconvertibleErrorCode());
+ }
+ }
+ return Result;
+}
+
Expected<MemorySanitizerOptions> parseMSanPassOptions(StringRef Params) {
MemorySanitizerOptions Result;
while (!Params.empty()) {
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp b/llvm/lib/Passes/PassBuilderPipelines.cpp
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -23,6 +23,7 @@
#include "llvm/Analysis/ProfileSummaryInfo.h"
#include "llvm/Analysis/ScopedNoAliasAA.h"
#include "llvm/Analysis/TypeBasedAliasAnalysis.h"
+#include "llvm/Bitcode/EmbedBitcodePass.h"
#include "llvm/IR/PassManager.h"
#include "llvm/Passes/OptimizationLevel.h"
#include "llvm/Passes/PassBuilder.h"
@@ -1464,7 +1465,18 @@
if (LTOPreLink)
addRequiredLTOPreLinkPasses(MPM);
+ return MPM;
+}
+ModulePassManager
+PassBuilder::buildFatLTODefaultPipeline(OptimizationLevel Level, bool ThinLTO,
+ bool EmitSummary) {
+ ModulePassManager MPM;
+ MPM.addPass(EmbedBitcodePass(ThinLTO, EmitSummary,
+ ThinLTO
+ ? buildThinLTOPreLinkDefaultPipeline(Level)
+ : buildLTOPreLinkDefaultPipeline(Level)));
+ MPM.addPass(buildPerModuleDefaultPipeline(Level));
return MPM;
}
diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def
--- a/llvm/lib/Passes/PassRegistry.def
+++ b/llvm/lib/Passes/PassRegistry.def
@@ -170,6 +170,13 @@
},
parseIPSCCPOptions,
"no-func-spec;func-spec")
+MODULE_PASS_WITH_PARAMS("embed-bitcode",
+ "EmbedBitcodePass",
+ [](EmbedBitcodeOptions Opts){
+ return EmbedBitcodePass(Opts);
+ },
+ parseEmbedBitcodePassOptions,
+ "thinlto;emit-summary")
#undef MODULE_PASS_WITH_PARAMS
#ifndef CGSCC_ANALYSIS
diff --git a/llvm/test/Bitcode/embed-multiple.ll b/llvm/test/Bitcode/embed-multiple.ll
new file mode 100644
--- /dev/null
+++ b/llvm/test/Bitcode/embed-multiple.ll
@@ -0,0 +1,6 @@
+; RUN: not opt --mtriple x86_64-unknown-linux-gnu < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s
+
+@a = global i32 1
+@llvm.embedded.module = private constant [4 x i8] c"BC\C0\DE"
+
+; CHECK: LLVM ERROR: Can only embed the module once.
diff --git a/llvm/test/Bitcode/embed-unsupported-object-format.ll b/llvm/test/Bitcode/embed-unsupported-object-format.ll
new file mode 100644
--- /dev/null
+++ b/llvm/test/Bitcode/embed-unsupported-object-format.ll
@@ -0,0 +1,5 @@
+; RUN: not opt --mtriple powerpc64-unknown-aix < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s
+
+@a = global i32 1
+
+; CHECK: LLVM ERROR: EmbedBitcode pass currently only supports ELF object format
diff --git a/llvm/test/Bitcode/embed.ll b/llvm/test/Bitcode/embed.ll
new file mode 100644
--- /dev/null
+++ b/llvm/test/Bitcode/embed.ll
@@ -0,0 +1,18 @@
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S | FileCheck %s
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode<thinlto>" -S | FileCheck %s
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode<emit-summary>" -S | FileCheck %s
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode<thinlto;emit-summary>" -S | FileCheck %s
+
+@a = global i32 1
+
+; CHECK: @a = global i32 1
+;; Make sure the module is in the correct section
+; CHECK: @llvm.embedded.object = private constant {{.*}}, section ".llvm.lto", align 1
+
+;; Ensure that the metadata is in llvm.compiler.used
+; CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr @llvm.embedded.object], section "llvm.metadata"
+
+;; Make sure the metadata correlates to the .llvm.lto section
+; CHECK: !llvm.embedded.objects = !{!1}
+; CHECK: !0 = !{}
+; CHECK: !{ptr @llvm.embedded.object, !".llvm.lto"}