[LTO] Support for embedding bitcode section during LTO

[zapster]

Summary

This adds support for embedding bitcode in a binary during LTO. The libLTO gains supports the -lto-embed-bitcode flag. The option allows users of the LTO library to embed a bitcode section. For example, LLD can pass the option via ld.lld -mllvm=-lto-embed-bitcode.

This feature allows doing something comparable to clang -c -fembed-bitcode, but on the (LTO) linker level. Having bitcode alongside native code has many use-cases. To give an example, the MacOS linker can create a -bitcode_bundle section containing bitcode. Also, having this feature built into LLVM is an alternative to 3rd party tools such as wllvm or gllvm. As with these tools, this feature simplifies creating "whole-program" llvm bitcode files, but in contrast to wllvm/gllvm it does not rely on a specific llvm frontend/driver.

[zapster]

re-ran clang format

[zapster]

Added inline remarks.

clang/lib/CodeGen/BackendUtil.cpp

1547

moved to llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

clang/test/Frontend/x86-embed-bitcode.ll

2

This duplicates the embed-bitcode.ll test (which only runs on ARM) for x86.

llvm/include/llvm/Bitcode/BitcodeWriter.h

158

BitcodeWriter.h seems like a natural place for this functionality. However, suggestions for a better location are more than appreciated.

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

4670

moved from clang/lib/CodeGen/BackendUtil.cpp

llvm/lib/LTO/LTOBackend.cpp

327

This options parsing logic is duplicated from clang. We might want move this to a shared place, but I failed to find a good candidate. include/llvm/Support/CodeGen.h came to mind, but it currently only contains types, no code. Any suggestions?

[zapster]

(ping)

[zapster]

(Ping)

Hi! I am still looking for reviews/reviewers for this change. Please let me know what I can do to make progress with this.
Many thanks in advance!

[tejohnson]

It seems reasonable to support this from LTO since it provides analogous support to what is available via clang without LTO. @pcc what do you think?

I only just skimmed it for now, one question below.

llvm/lib/LTO/LTOBackend.cpp

335

Does this value make any sense since CmdArgs is always empty below?

[zapster]

Thanks for your comment. I replied inline.

llvm/lib/LTO/LTOBackend.cpp

335

You are right. Currently, this option value is not utterly useful, but I kept it for compatibility with clangs -fembed-bitcode. It seems the marker section is needed (even if empty), otherwise certain tools will complain (notably the MacOS linker). Ideally, we would have something sensible to write into the section but I am not sure about reasonable values. I was not able to find any information about consumers of these commands. Pointers in that regard would be highly appreciated.

Anyhow, I lean toward keeping the options for compatibility with clang and for making it simple to properly implement the marker case. That said, I am perfectly fine with removing the option. In that case, I guess I should get rid of EmbedBitcodeKind altogether and always insert the bitcode and the marker, right?

On the other hand, we should maybe just consolidate the option (parsing) with the clang option. Some thoughts about that in my inline comment above.

[tejohnson]

llvm/lib/LTO/LTOBackend.cpp

335

It's unclear to me whether it is better to not provide the marker, and get a clear error, or put in an empty marker which may cause these tools to do the wrong thing. @steven_wu who might be able to provide some guidance about how this is used by the MacOS linker. Note that if someone is using MacOS to do their LTO linking, they won't even trigger this code, as that toolchain uses the old legacy LTO, which doesn't come here.

Presumably your tool doesn't use the marker section?

[steven_wu]

llvm/lib/LTO/LTOBackend.cpp

335

I don't think marker is useful for this use case. I will suggest not to have the marker LTO option unless we have the need. It is being tested by clang and that is probably enough coverage for testing.

I don't think any users of the legacy interface express interests in using a unified bitcode section so it is fine just to leave this out of the legacy C interface.

[zapster]

llvm/lib/LTO/LTOBackend.cpp

335

[...] if someone is using MacOS to do their LTO linking, they won't even trigger this code, as that toolchain uses the old legacy LTO

Right. I mixed up the case where the embedding logic was called from clang and from LTO. And of course, the MacOS linker does (unfortunately) not use this code.

Presumably your tool doesn't use the marker section?

Nope, I don't need the marker section.

I don't think marker is useful for this use case.

Right, makes sense. I'll update the review and make -lto-embed-bitcode a boolean option.

Thanks to both of you for the clarifications.

[zapster]

Made -lto-embed-bitcode a boolean option.

[tejohnson]

LGTM with a few small suggestions before you commit.

llvm/lib/LTO/LTOBackend.cpp

317

Expand description a bit to indicate that it will embed LLVM bitcode in object files produced by LTO.

llvm/test/LTO/X86/embed-bitcode.ll

9

I don't think it is necessary to check anything other than "-lto-embed-bitcode=false" and "-lto-embed-bitcode", and the default. Otherwise you are just testing the standard behavior of cl::opt, and that isn't necessary for a new internal option.

[zapster]

Addressed suggestions from @tejohnson

[zapster]

Thanks again for you reviews! Since I do not have commit rights, I would be grateful if someone could push it for me.

[tejohnson]

In D68213#1781436, @zapster wrote:

Thanks again for you reviews! Since I do not have commit rights, I would be grateful if someone could push it for me.

The new LTO/X86/embed-bitcode.ll test is failing for me:

llvm-lto2: llvm/lib/LTO/LTO.cpp:514: void llvm::lto::LTO::addModuleToGlobalRes(ArrayRef<InputFile::Symbol>, ArrayRef<llvm::lto::SymbolResolution>, unsigned int, bool): Assertion `!GlobalRes.Prevailing && "Multiple prevailing defs are not allowed"' failed.

[zapster]

The new LTO/X86/embed-bitcode.ll test is failing for me

Should be fixed now, thanks.

clang/lib/CodeGen/BackendUtil.cpp

if (AsmHelper.TM) {
     if (DLDesc != TDesc.getStringRepresentation()) {
       unsigned DiagID = Diags.getCustomDiagID(
           DiagnosticsEngine::Error, "backend data layout '%0' does not match "
                                     "expected target description '%1'");
       Diags.Report(DiagID) << DLDesc << TDesc.getStringRepresentation();
     }
   }
 }
 
-static const char* getSectionNameForBitcode(const Triple &T) {
-  switch (T.getObjectFormat()) {
-  case Triple::MachO:
-    return "__LLVM,__bitcode";
-  case Triple::COFF:
-  case Triple::ELF:
-  case Triple::Wasm:
-  case Triple::UnknownObjectFormat:
-    return ".llvmbc";
-  case Triple::XCOFF:
-    llvm_unreachable("XCOFF is not yet implemented");
-    break;
-  }
-  llvm_unreachable("Unimplemented ObjectFormatType");
-}
-
-static const char* getSectionNameForCommandline(const Triple &T) {
-  switch (T.getObjectFormat()) {
-  case Triple::MachO:
-    return "__LLVM,__cmdline";
-  case Triple::COFF:
-  case Triple::ELF:
-  case Triple::Wasm:
-  case Triple::UnknownObjectFormat:
-    return ".llvmcmd";
-  case Triple::XCOFF:
-    llvm_unreachable("XCOFF is not yet implemented");
-    break;
-  }
-  llvm_unreachable("Unimplemented ObjectFormatType");
-}
-
 // With -fembed-bitcode, save a copy of the llvm IR as data in the
 // __LLVM,__bitcode section.
 void clang::EmbedBitcode(llvm::Module *M, const CodeGenOptions &CGOpts,
                          llvm::MemoryBufferRef Buf) {
   if (CGOpts.getEmbedBitcode() == CodeGenOptions::Embed_Off)
     return;
-
-  // Save llvm.compiler.used and remote it.
-  SmallVector<Constant*, 2> UsedArray;
-  SmallPtrSet<GlobalValue*, 4> UsedGlobals;
-  Type *UsedElementType = Type::getInt8Ty(M->getContext())->getPointerTo(0);
-  GlobalVariable *Used = collectUsedGlobalVariables(*M, UsedGlobals, true);
-  for (auto *GV : UsedGlobals) {
-    if (GV->getName() != "llvm.embedded.module" &&
-        GV->getName() != "llvm.cmdline")
-      UsedArray.push_back(
-          ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
-  }
-  if (Used)
-    Used->eraseFromParent();
-
-  // Embed the bitcode for the llvm module.
-  std::string Data;
-  ArrayRef<uint8_t> ModuleData;
-  Triple T(M->getTargetTriple());
-  // Create a constant that contains the bitcode.
-  // In case of embedding a marker, ignore the input Buf and use the empty
-  // ArrayRef. It is also legal to create a bitcode marker even Buf is empty.
-  if (CGOpts.getEmbedBitcode() != CodeGenOptions::Embed_Marker) {
-    if (!isBitcode((const unsigned char *)Buf.getBufferStart(),
-                   (const unsigned char *)Buf.getBufferEnd())) {
-      // If the input is LLVM Assembly, bitcode is produced by serializing
-      // the module. Use-lists order need to be perserved in this case.
-      llvm::raw_string_ostream OS(Data);
-      llvm::WriteBitcodeToFile(*M, OS, /* ShouldPreserveUseListOrder */ true);
-      ModuleData =
-          ArrayRef<uint8_t>((const uint8_t *)OS.str().data(), OS.str().size());
-    } else
-      // If the input is LLVM bitcode, write the input byte stream directly.
-      ModuleData = ArrayRef<uint8_t>((const uint8_t *)Buf.getBufferStart(),
-                                     Buf.getBufferSize());
-  }
-  llvm::Constant *ModuleConstant =
-      llvm::ConstantDataArray::get(M->getContext(), ModuleData);
-  llvm::GlobalVariable *GV = new llvm::GlobalVariable(
-      *M, ModuleConstant->getType(), true, llvm::GlobalValue::PrivateLinkage,
-      ModuleConstant);
-  GV->setSection(getSectionNameForBitcode(T));
-  UsedArray.push_back(
-      ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
-  if (llvm::GlobalVariable *Old =
-          M->getGlobalVariable("llvm.embedded.module", true)) {
-    assert(Old->hasOneUse() &&
-           "llvm.embedded.module can only be used once in llvm.compiler.used");
-    GV->takeName(Old);
-    Old->eraseFromParent();
-  } else {
-    GV->setName("llvm.embedded.module");
-  }
-
-  // Skip if only bitcode needs to be embedded.
-  if (CGOpts.getEmbedBitcode() != CodeGenOptions::Embed_Bitcode) {
-    // Embed command-line options.
-    ArrayRef<uint8_t> CmdData(const_cast<uint8_t *>(CGOpts.CmdArgs.data()),
-                              CGOpts.CmdArgs.size());
-    llvm::Constant *CmdConstant =
-      llvm::ConstantDataArray::get(M->getContext(), CmdData);
-    GV = new llvm::GlobalVariable(*M, CmdConstant->getType(), true,
-                                  llvm::GlobalValue::PrivateLinkage,
-                                  CmdConstant);
-    GV->setSection(getSectionNameForCommandline(T));
-    UsedArray.push_back(
-        ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
-    if (llvm::GlobalVariable *Old =
-            M->getGlobalVariable("llvm.cmdline", true)) {
-      assert(Old->hasOneUse() &&
-             "llvm.cmdline can only be used once in llvm.compiler.used");
-      GV->takeName(Old);
-      Old->eraseFromParent();
-    } else {
-      GV->setName("llvm.cmdline");
-    }
-  }
-
-  if (UsedArray.empty())
-    return;
-
-  // Recreate llvm.compiler.used.
-  ArrayType *ATy = ArrayType::get(UsedElementType, UsedArray.size());
-  auto *NewUsed = new GlobalVariable(
-      *M, ATy, false, llvm::GlobalValue::AppendingLinkage,
-      llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
-  NewUsed->setSection("llvm.metadata");
+  llvm::EmbedBitcodeInModule(
+      *M, Buf, CGOpts.getEmbedBitcode() != CodeGenOptions::Embed_Marker,
+      CGOpts.getEmbedBitcode() != CodeGenOptions::Embed_Bitcode,
+      &CGOpts.CmdArgs);
 }

clang/test/Frontend/x86-embed-bitcode.ll

+; REQUIRES: x86-registered-target
+; check .ll input
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm \
+; RUN:    -fembed-bitcode=all -x ir %s -o - \
+; RUN:    | FileCheck %s
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm \
+; RUN:    -fembed-bitcode=marker -x ir %s -o - \
+; RUN:    | FileCheck %s -check-prefix=CHECK-MARKER
+; RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm \
+; RUN:    -fembed-bitcode=all -x ir %s -o - \
+; RUN:    | FileCheck %s -check-prefix=CHECK-ELF
+; RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm \
+; RUN:    -fembed-bitcode=marker -x ir %s -o - \
+; RUN:    | FileCheck %s -check-prefix=CHECK-ELF-MARKER
+; RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -emit-llvm \
+; RUN:    -fembed-bitcode=bitcode -x ir %s -o - \
+; RUN:    | FileCheck %s -check-prefix=CHECK-ELF-ONLY-BITCODE
+
+; check .bc input
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm-bc \
+; RUN:    -x ir %s -o %t.bc
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm \
+; RUN:    -fembed-bitcode=all -x ir %t.bc -o - \
+; RUN:    | FileCheck %s
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm \
+; RUN:    -fembed-bitcode=bitcode -x ir %t.bc -o - \
+; RUN:    | FileCheck %s -check-prefix=CHECK-ONLY-BITCODE
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm \
+; RUN:    -fembed-bitcode=marker -x ir %t.bc -o - \
+; RUN:    | FileCheck %s -check-prefix=CHECK-MARKER
+
+; run through -fembed-bitcode twice and make sure it doesn't crash
+; RUN: %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm-bc \
+; RUN:    -fembed-bitcode=all -x ir %s -o - \
+; RUN: | %clang_cc1 -triple x86_64-apple-macosx10.10 -emit-llvm \
+; RUN:    -fembed-bitcode=all -x ir - -o /dev/null
+
+; check the magic number of bitcode at the beginning of the string
+; CHECK: @llvm.embedded.module = private constant
+; CHECK: c"\DE\C0\17\0B
+; CHECK: section "__LLVM,__bitcode"
+; CHECK: @llvm.cmdline = private constant
+; CHECK: section "__LLVM,__cmdline"
+
+; CHECK-ELF: @llvm.embedded.module
+; CHECK-ELF: section ".llvmbc"
+; CHECK-ELF: @llvm.cmdline
+; CHECK-ELF: section ".llvmcmd"
+
+; CHECK-ELF-MARKER: @llvm.embedded.module
+; CHECK-ELF-MARKER: constant [0 x i8] zeroinitializer
+; CHECK-ELF-MARKER: @llvm.cmdline
+; CHECK-ELF-MARKER: section ".llvmcmd"
+
+; CHECK-ELF-ONLY-BITCODE: @llvm.embedded.module
+; CHECK-ELF-ONLY-BITCODE: section ".llvmbc"
+; CHECK-ELF-ONLY-BITCODE-NOT: @llvm.cmdline
+; CHECK-ELF-ONLY-BITCODE-NOT: section ".llvmcmd"
+
+; CHECK-ONLY-BITCODE: @llvm.embedded.module = private constant
+; CHECK-ONLY-BITCODE: c"\DE\C0\17\0B
+; CHECK-ONLY-BITCODE: section "__LLVM,__bitcode"
+; CHECK-ONLY-BITCODE-NOT: @llvm.cmdline = private constant
+; CHECK-ONLY-BITCODE-NOT: section "__LLVM,__cmdline"
+
+; CHECK-MARKER: @llvm.embedded.module
+; CHECK-MARKER: constant [0 x i8] zeroinitializer
+; CHECK-MARKER: section "__LLVM,__bitcode"
+; CHECK-MARKER: @llvm.cmdline
+; CHECK-MARKER: section "__LLVM,__cmdline"
+
+define i32 @f0() {
+  ret i32 0
+}

llvm/include/llvm/Bitcode/BitcodeWriter.h

 
 #ifndef LLVM_BITCODE_BITCODEWRITER_H
 #define LLVM_BITCODE_BITCODEWRITER_H
 
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
 #include "llvm/MC/StringTableBuilder.h"
 #include "llvm/Support/Allocator.h"
+#include "llvm/Support/MemoryBuffer.h"
 #include <map>
 #include <memory>
 #include <string>
 #include <vector>
 
 namespace llvm {
 
 class BitstreamWriter;
class raw_ostream;
   /// where it will be written in a new bitcode block. This is used when
   /// writing the combined index file for ThinLTO. When writing a subset of the
   /// index for a distributed backend, provide the \p ModuleToSummariesForIndex
   /// map.
   void WriteIndexToFile(const ModuleSummaryIndex &Index, raw_ostream &Out,
                         const std::map<std::string, GVSummaryMapTy>
                             *ModuleToSummariesForIndex = nullptr);
 
+  /// Save a copy of the llvm IR as data in the __LLVM,__bitcode section.
+  void EmbedBitcodeInModule(Module &M, MemoryBufferRef Buf, bool EmbedBitcode,
+                            bool EmbedMarker,
+                            const std::vector<uint8_t> *CmdArgs);
+
 } // end namespace llvm
 
 #endif // LLVM_BITCODE_BITCODEWRITER_H

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
+#include "llvm/Bitcode/BitcodeReader.h"
+#include "llvm/Bitcode/LLVMBitCodes.h"
 #include "llvm/Bitstream/BitCodes.h"
 #include "llvm/Bitstream/BitstreamWriter.h"
-#include "llvm/Bitcode/LLVMBitCodes.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/Comdat.h"
 #include "llvm/IR/Constant.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"
void llvm::WriteThinLinkBitcodeToFile(const Module &M, raw_ostream &Out,
 
   BitcodeWriter Writer(Buffer);
   Writer.writeThinLinkBitcode(M, Index, ModHash);
   Writer.writeSymtab();
   Writer.writeStrtab();
 
   Out.write((char *)&Buffer.front(), Buffer.size());
 }
+
+static const char *getSectionNameForBitcode(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+    return "__LLVM,__bitcode";
+  case Triple::COFF:
+  case Triple::ELF:
+  case Triple::Wasm:
+  case Triple::UnknownObjectFormat:
+    return ".llvmbc";
+  case Triple::XCOFF:
+    llvm_unreachable("XCOFF is not yet implemented");
+    break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
+static const char *getSectionNameForCommandline(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+    return "__LLVM,__cmdline";
+  case Triple::COFF:
+  case Triple::ELF:
+  case Triple::Wasm:
+  case Triple::UnknownObjectFormat:
+    return ".llvmcmd";
+  case Triple::XCOFF:
+    llvm_unreachable("XCOFF is not yet implemented");
+    break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
+void llvm::EmbedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
+                                bool EmbedBitcode, bool EmbedMarker,
+                                const std::vector<uint8_t> *CmdArgs) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector<Constant *, 2> UsedArray;
+  SmallPtrSet<GlobalValue *, 4> UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+    if (GV->getName() != "llvm.embedded.module" &&
+        GV->getName() != "llvm.cmdline")
+      UsedArray.push_back(
+          ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+    Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef<uint8_t> ModuleData;
+  Triple T(M.getTargetTriple());
+  // Create a constant that contains the bitcode.
+  // In case of embedding a marker, ignore the input Buf and use the empty
+  // ArrayRef. It is also legal to create a bitcode marker even Buf is empty.
+  if (EmbedBitcode) {
+    if (!isBitcode((const unsigned char *)Buf.getBufferStart(),
+                   (const unsigned char *)Buf.getBufferEnd())) {
+      // If the input is LLVM Assembly, bitcode is produced by serializing
+      // the module. Use-lists order need to be preserved in this case.
+      llvm::raw_string_ostream OS(Data);
+      llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+      ModuleData =
+          ArrayRef<uint8_t>((const uint8_t *)OS.str().data(), OS.str().size());
+    } else
+      // If the input is LLVM bitcode, write the input byte stream directly.
+      ModuleData = ArrayRef<uint8_t>((const uint8_t *)Buf.getBufferStart(),
+                                     Buf.getBufferSize());
+  }
+  llvm::Constant *ModuleConstant =
+      llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new llvm::GlobalVariable(
+      M, ModuleConstant->getType(), true, llvm::GlobalValue::PrivateLinkage,
+      ModuleConstant);
+  GV->setSection(getSectionNameForBitcode(T));
+  UsedArray.push_back(
+      ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  if (llvm::GlobalVariable *Old =
+          M.getGlobalVariable("llvm.embedded.module", true)) {
+    assert(Old->hasOneUse() &&
+           "llvm.embedded.module can only be used once in llvm.compiler.used");
+    GV->takeName(Old);
+    Old->eraseFromParent();
+  } else {
+    GV->setName("llvm.embedded.module");
+  }
+
+  // Skip if only bitcode needs to be embedded.
+  if (EmbedMarker) {
+    // Embed command-line options.
+    ArrayRef<uint8_t> CmdData(const_cast<uint8_t *>(CmdArgs->data()),
+                              CmdArgs->size());
+    llvm::Constant *CmdConstant =
+        llvm::ConstantDataArray::get(M.getContext(), CmdData);
+    GV = new llvm::GlobalVariable(M, CmdConstant->getType(), true,
+                                  llvm::GlobalValue::PrivateLinkage,
+                                  CmdConstant);
+    GV->setSection(getSectionNameForCommandline(T));
+    UsedArray.push_back(
+        ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+    if (llvm::GlobalVariable *Old = M.getGlobalVariable("llvm.cmdline", true)) {
+      assert(Old->hasOneUse() &&
+             "llvm.cmdline can only be used once in llvm.compiler.used");
+      GV->takeName(Old);
+      Old->eraseFromParent();
+    } else {
+      GV->setName("llvm.cmdline");
+    }
+  }
+
+  if (UsedArray.empty())
+    return;
+
+  // Recreate llvm.compiler.used.
+  ArrayType *ATy = ArrayType::get(UsedElementType, UsedArray.size());
+  auto *NewUsed = new GlobalVariable(
+      M, ATy, false, llvm::GlobalValue::AppendingLinkage,
+      llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
+  NewUsed->setSection("llvm.metadata");
+}

llvm/lib/LTO/LTOBackend.cpp

 #include "llvm/Object/ModuleSymbolTable.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Passes/StandardInstrumentations.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
+#include "llvm/Support/SmallVectorMemoryBuffer.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 #include "llvm/Transforms/Scalar/LoopPassManager.h"
 #include "llvm/Transforms/Utils/FunctionImportUtils.h"
bool opt(Config &Conf, TargetMachine *TM, unsigned Task, Module &Mod,
   else if (Conf.UseNewPM)
     runNewPMPasses(Conf, Mod, TM, Conf.OptLevel, IsThinLTO, ExportSummary,
                    ImportSummary);
   else
     runOldPMPasses(Conf, Mod, TM, IsThinLTO, ExportSummary, ImportSummary);
   return !Conf.PostOptModuleHook || Conf.PostOptModuleHook(Task, Mod);
 }
 
+static cl::opt<bool> EmbedBitcode(
+    "lto-embed-bitcode", cl::init(false),
+    cl::desc("Embed LLVM bitcode in object files produced by LTO"));
+
+static void EmitBitcodeSection(Module &M, Config &Conf) {
+  if (!EmbedBitcode)
+    return;
+  SmallVector<char, 0> Buffer;
+  raw_svector_ostream OS(Buffer);
+  WriteBitcodeToFile(M, OS);
+
+  std::unique_ptr<MemoryBuffer> Buf(
+      new SmallVectorMemoryBuffer(std::move(Buffer)));
+  llvm::EmbedBitcodeInModule(M, Buf->getMemBufferRef(), /*EmbedBitcode*/ true,
+                             /*EmbedMarker*/ false, /*CmdArgs*/ nullptr);
+}
+
 void codegen(Config &Conf, TargetMachine *TM, AddStreamFn AddStream,
              unsigned Task, Module &Mod) {
   if (Conf.PreCodeGenModuleHook && !Conf.PreCodeGenModuleHook(Task, Mod))
     return;
 
+  EmitBitcodeSection(Mod, Conf);
+
   std::unique_ptr<ToolOutputFile> DwoOut;
   SmallString<1024> DwoFile(Conf.SplitDwarfOutput);
   if (!Conf.DwoDir.empty()) {
     std::error_code EC;
     if (auto EC = llvm::sys::fs::create_directories(Conf.DwoDir))
       report_fatal_error("Failed to create directory " + Conf.DwoDir + ": " +
                          EC.message());
 

llvm/test/LTO/X86/Inputs/start-lib1.ll

+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @bar()
+
+define void @foo() {
+  ret void
+}

llvm/test/LTO/X86/Inputs/start-lib2.ll

+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @bar() {
+  ret void
+}

llvm/test/LTO/X86/embed-bitcode.ll

+; RUN: llvm-as %s -o %t1.o
+; RUN: llvm-as %p/Inputs/start-lib1.ll -o %t2.o
+; RUN: llvm-as %p/Inputs/start-lib2.ll -o %t3.o
+
+; RUN: llvm-lto2 run -r %t1.o,_start,px -r %t2.o,foo,px -r %t3.o,bar,px -r %t2.o,bar,px -o %t3 %t1.o %t2.o %t3.o
+; RUN: llvm-readelf -S %t3.0 | FileCheck %s --implicit-check-not=.llvmbc
+
+; RUN: llvm-lto2 run -r %t1.o,_start,px -r %t2.o,foo,px -r %t3.o,bar,px -r %t2.o,bar,px -lto-embed-bitcode=false -o %t3 %t1.o %t2.o %t3.o
+; RUN: llvm-readelf -S %t3.0 | FileCheck %s --implicit-check-not=.llvmbc
+
+; RUN: llvm-lto2 run -r %t1.o,_start,px -r %t2.o,foo,px -r %t3.o,bar,px -r %t2.o,bar,px -lto-embed-bitcode -o %t3 %t1.o %t2.o %t3.o
+; RUN: llvm-readelf -S %t3.0 | FileCheck %s --check-prefix=CHECK-ELF
+; RUN: llvm-objcopy -O binary -j .llvmbc %t3.0 %t-embedded.bc
+; RUN: llvm-dis %t-embedded.bc -o - | FileCheck %s --check-prefix=CHECK-LL
+
+; CHECK-ELF: .text
+; CHECK-ELF: .llvmbc
+
+; CHECK-LL: @_start
+; CHECK-LL: @foo
+; CHECK-LL: @bar
+
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @_start() {
+  ret void
+}