Reland [llvm] Preliminary fat-lto-objects support

[paulkirth]

Summary

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not to link-time. This is a feature available in GCC for some time,
and makes the existing -ffat-lto-objects flag functional in the same
way as GCC's.

Within LLVM, we add a new EmbedBitcodePass that serializes the module to
the object file, and expose a new pass pipeline for compiling fat
objects. The new pipeline initially clones the module and runs the
selected (Thin)LTOPrelink pipeline, after which it will serialize the
module into a .llvm.lto section of an ELF file. When compiling for
(Thin)LTO, this normally the point at which the compiler would emit a
object file containing the bitcode and metadata.

After that point we compile the original module using the
PerModuleDefaultPipeline used for non-LTO compilation. We generate
standard object files at the end of this pipeline, which contain machine
code and the new .llvm.lto section containing bitcode.

Since the two pipelines operate on different copies of the module, we
can be sure that the bitcode in the .llvm.lto section and object code
in .text are congruent with the existing output produced by the
default and LTO pipelines.

Original RFC: https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977

Earlier versions of this patch were missing REQUIRES lines for llc
related tests in Transforms/EmbedBitcode. Those tests are now under
CodeGen/X86, which should avoid running the check on unsupported
platforms.

The EmbedbBitcodePass also returned PreservedAnalyses::all when adding a
metadata section, which failed expensive checks, since it modified the
module. This is now corrected.

[paulkirth]

llvm/lib/Passes/PassBuilderPipelines.cpp

1485

I don't think the prior code mimic'ed the non-LTO pipeline, however. My understanding is that the prior version did the following for the non embedded code on which it would generate a native object:

(Thin)LTOPreLink + ModuleSimplification(ThinOrFullLTOPhase::None) + ModuleOptimization(ThinOrFullLTOPhase::None)

So you would have been essentially duplicating ModuleSimplification (since it is invoked in both the *LTO pre-link and again with LTO Phase = None). That can cause some issues because there are passes that are only meant to be invoked once that you could duplicate. A good example is PGO instrumentation/annotation (see the conditions under which addPGOInstrPasses is called from buildModuleSimplificationPipeline).

With your new version, my understanding is that you are calling:

1. buildPerModuleDefaultPipeline on original Module -> generate native object
2. (Thin)LTOPreLink on Module clone -> embed IR

And buildPerModuleDefaultPipeline is largely just ModuleSimplification + ModuleOptimization. So I'm not sure about why it would be more wasteful (other than cloning the Module) - you are not running more pipelines total than in the prior code. Unless I am misunderstanding the former or current code, which is possible!

I think that's an accurate summary. While my intent originally was to closely mimic the default pipeline, I don't think it was very close in reality. Also, as you pointed out the new approach isn't nearly as wasteful as I felt initially.

[aeubanks]

In D146776#4302238, @tejohnson wrote:

I agree ensuring full performance with FatLTO's non-LTO native object isn't terribly important. But my concern is whether this results in some unexpected behavior with certain passes not being run at all - e.g. ICP won't run at all if you are doing SamplePGO in this configuration.

...

As mentioned earlier, I think the question is whether it will be a surprise when using SamplePGO that certain things simply don't get run when using the non-LTO native objects. I don't know enough about when these will get used to know if this is a configuration anyone will care about. If it is, I suppose one approach would be to invoke the longer pipelines only under SamplePGO. Either way, this probably needs some clear documentation.

These are all performance concerns for the non-LTO native objects, not correctness right? I think if we clearly document that performance might be very meh for fat LTO non-LTO native objects this would be fine.

llvm/include/llvm/Bitcode/EmbedBitcodePass.h

33

is this default param necessary?

llvm/lib/Passes/PassBuilderPipelines.cpp

1265

do we still need IsFatLTO with the latest approach?

1485

IIUC the previous version was ThinLTOPreLink + ModuleOptimization(ThinOrFullLTOPhase::None), where we'd write out the module after pre-link and before module optimization to some global. So we'd avoid running the simplification pipeline twice with that. (the problem is that the simplification pipeline is slightly different between a normal build and a ThinLTO build)

the current version will run the simplification pipeline twice now, once for ThinLTO and once for the default pipeline

llvm/lib/Passes/PassRegistry.def

61

these should be parameterizable, see MODULE_PASS_WITH_PARAMS

[paulkirth]

In D146776#4374005, @aeubanks wrote:

These are all performance concerns for the non-LTO native objects, not correctness right? I think if we clearly document that performance might be very meh for fat LTO non-LTO native objects this would be fine.

In the prior version, I think its too hard to say if running pipelines, like ModuleSimplification, multiple times is safe/correct without some fairly detailed analysis. I can easily imagine any transform that is only expected to run once may introduce a subtle bug if run again on the same module. Nothing obvious here jumps out at me, but I'm also not confident enough to say that it can't happen.

That said, since FatLTO no longer use a modified/custom pipeline but uses the (Thin)LTO and Module piplines on independed modules, there should no longer be any correctness nor performance issues over a normal compilation for the non-LTO object code.

llvm/include/llvm/Bitcode/EmbedBitcodePass.h

33

IIRC it was required for some reason, but let me double check that.

llvm/lib/Passes/PassBuilderPipelines.cpp

1265

That's a good point. I don't think so, so I can simplify this futher. Thanks for pointing this out.

llvm/lib/Passes/PassRegistry.def

61

That is a much better way to handle this. Thank you.

[paulkirth]

Update summary, documentation, and address comments.

[aeubanks]

llvm/include/llvm/Bitcode/EmbedBitcodePass.h

27

extraneous ;

29

this is never used

44

if you really wanted to hook this up to the pipeline parsing (so you could specify a sub-passmanager) you'd have to thread through all the parsing code. probably not worth it

llvm/include/llvm/Passes/PassBuilder.h

242

comment out of date

llvm/lib/Passes/PassBuilderPipelines.cpp

1265

still not removed?

1471

this comment is obsolete now

[paulkirth]

Address remaining comments

• Fix Doc string
• Remove obsolete comment
• Remove uses of IsThinkLTO
• Remove some dead code
• Fix some typos
• Fix test looking for wrong pass name in the error string
• git clang-format

llvm/lib/Passes/PassBuilderPipelines.cpp

1471

I was thinking once that patch lands we can take the ThinOrFullLTOPhase direcly and avoid the branch, but maybe it still makes sense to leave this as is.

[aeubanks]

this basically lgtm

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

31–38

these should be unnecessary with the newly added pass params, e.g. -passes=embed-bitcode<emit-summary>

[paulkirth]

Rebase

• Fix typo in docs
• Remove unneeded cl::opts

[paulkirth]

@tejohnson @nikic any lingering concerns?

[tejohnson]

LGTM with a couple of minor nits and a question back to @nikic about one of this comments.

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

31

true seems to be the default, did you mean false? The same could probably be said for the later call to report_fatal_error (ELF format). I actually think this one seems less like a user error and more like a compiler error (if this pass gets run twice somehow).

49

Document constant parameter (nullptr)

llvm/test/Bitcode/embed.ll

15

s/sue/sure

[paulkirth]

Thanks for the feedback @tejohnson. I'm about to take off for the weekend, but I'll address these on Monday.

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

31

Oh, yeah, probably. I do see your point though. I guess maybe user error encompasses adding a pass multiple times?

IMO they're all kind of hard errors. I'm happy to do this either way, but if we want to suppress this one, then you're right that it should be false here.

49

ah, good catch. Thanks.

llvm/test/Bitcode/embed.ll

15

ty

[nikic]

I'm okay with the new approach -- I think it makes the feature somewhat less compelling, but it's the path of least resistance for now. However, I'd like to have it documented that we're explicitly not guaranteeing that the results are bit-for-bit identical, and this is just an artifact of the current implementation.

Two notes on the larger feature:

• I'm concerned about the lld-only limitation. Unless there is some technical limitation here, this should be supported in LLVMgold as well.
• It looks like this requires passing -fat-lto-objects to lld to actually get LTO -- and it looks like the clang doesn't do this itself even if -ffat-lto-objects is passed to the driver? Can you please check how these flags work in GCC? I'm not entirely confident on this, but I expect that if you have fat objects they will use LTO by default unless you pass -fno-lto to the driver. We should try to match whatever the GCC behavior is here.

llvm/docs/FatLTOSupport.rst

28

29

34

38–40

43

The pre-link pipeline is run as part of EmbedBitcode, not before it.

50

This seems like the important part for end users, so maybe put that in the introduction.

60

This is a very annoying limitation -- is it particularly hard to implementation support for this in the LLVMgold plugin?

We won't be able to use fat objects if it's only supported by LLD.

llvm/docs/ReleaseNotes.rst

85

Maybe link the docs instead / as well here?

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

31

Sorry, I meant false here.

My general rule of thumb is that if you can write a test for it, then it should not crash.

[tejohnson]

In D146776#4410967, @nikic wrote:

I'm okay with the new approach -- I think it makes the feature somewhat less compelling, but it's the path of least resistance for now. However, I'd like to have it documented that we're explicitly not guaranteeing that the results are bit-for-bit identical, and this is just an artifact of the current implementation.

Two notes on the larger feature:

• I'm concerned about the lld-only limitation. Unless there is some technical limitation here, this should be supported in LLVMgold as well.

Great point. I see that the linker changes aren't in this patch, but in another related one. I agree it should be supported by the LLVM gold plugin as well, which should be straightforward. This can be another follow on patch. It looks like you would want to modify this change made to ignore the -fembed-bitcode sections so that we do in fact claim the fat object files:
https://github.com/llvm/llvm-project/blob/main/llvm/tools/gold/gold-plugin.cpp#L541-L547

• It looks like this requires passing -fat-lto-objects to lld to actually get LTO -- and it looks like the clang doesn't do this itself even if -ffat-lto-objects is passed to the driver? Can you please check how these flags work in GCC? I'm not entirely confident on this, but I expect that if you have fat objects they will use LTO by default unless you pass -fno-lto to the driver. We should try to match whatever the GCC behavior is here.

agree

[paulkirth]

Rebase and address comments

[paulkirth]

llvm/test/Bitcode/embed-multiple.ll

2

looks like I failed to update the tests after making the errors non-fatal.

[paulkirth]

Rebase and fix test cases.

[nikic]

LGTM

llvm/docs/FatLTOSupport.rst

38

unmodified

61

I feel like this information has been repeated (with minor variations) three times: In the bullet points above, the paragraph following it, and then here...

74

Despite the name, the gold plugin is also supported by the bfd linker, so you actually support all common linkers (on ELF based systems).

[MaskRay]

Apologies for not commenting soon. I'll take a close look shortly.

[paulkirth]

In D146776#4429120, @MaskRay wrote:

Apologies for not commenting soon. I'll take a close look shortly.

TY!

llvm/docs/FatLTOSupport.rst

61

Yeah, it's definitely a bit redundant. I'll go through this one more time and try to streamline the text.

74

Ah, right. I completely forgot bfd could use the plugin.

[paulkirth]

Revise text in documentation.

[paulkirth]

Document constant parameter.

[paulkirth]

llvm/docs/FatLTOSupport.rst

61

I feel like the current version reads a bit better. WDYT?

[nikic]

llvm/docs/FatLTOSupport.rst

50

61

Looks good!

62

[MaskRay]

Started looking...

-DBUILD_SHARED_LIBS=on uses -Wl,-z,defs and can detect some library layering (https://llvm.org/docs/CodingStandards.html#library-layering https://maskray.me/blog/2021-06-13-dependency-related-linker-options) problems. ninja LLVMBitWriter doesn't build due to missing dependencies.

One issue can be addressed with the following change but the dependency may be a bit weird.

diff --git i/llvm/lib/Bitcode/Writer/CMakeLists.txt w/llvm/lib/Bitcode/Writer/CMakeLists.txt
index 2b17aa912016..51849d2d0fc0 100644
--- i/llvm/lib/Bitcode/Writer/CMakeLists.txt
+++ w/llvm/lib/Bitcode/Writer/CMakeLists.txt
@@ -17,2 +17,4 @@ add_llvm_component_library(LLVMBitWriter
   TargetParser
+  TransformUtils
   )

The other issue (error: undefined symbol: llvm::ThinLTOBitcodeWriterPass::run) is more tricky: LLVMipo (llvm/lib/Transforms/IPO/CMakeLists.txt) depends on LLVMBitWriter, so LLVMBitWriter cannot depend on LLVMipo.

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

25

Including llvm/Transforms headers seems problematic.

[MaskRay]

llvm/docs/FatLTOSupport.rst

2

Maybe just FatLTO.rst. For other features, we don't use the Support suffix.

19

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

33

Nit: diagnostics do not need capitalization (https://llvm.org/docs/CodingStandards.html#error-and-warning-messages) but report_fatal_error messages often do capitalization, so I think capitalization is fine, but the period is often omitted.

56

You can delete ModuleData and even Buf.

llvm/lib/Passes/PassRegistry.def

175

llvm/test/Bitcode/embed.ll

10

Nit: complete sentences in comments end with a period.

[MaskRay]

I think we need a new section type (say, SHT_LLVM_LTO) in llvm/include/llvm/BinaryFormat/ELF.h. Then add a test llvm/test/MC/ELF to use llvm-readobj -S to test this new section type.

[MaskRay]

.llvm.lto should be added beside .llvmcmd in llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp so that .llvm.lto has the SHF_EXCLUDE flag on ELF.
We need a llvm/test/CodeGen/ test using llvm-readelf -S to test the section flags (E).

[paulkirth]

In D146776#4430608, @MaskRay wrote:

Started looking...

-DBUILD_SHARED_LIBS=on uses -Wl,-z,defs and can detect some library layering (https://llvm.org/docs/CodingStandards.html#library-layering https://maskray.me/blog/2021-06-13-dependency-related-linker-options) problems. ninja LLVMBitWriter doesn't build due to missing dependencies.

One issue can be addressed with the following change but the dependency may be a bit weird.

diff --git i/llvm/lib/Bitcode/Writer/CMakeLists.txt w/llvm/lib/Bitcode/Writer/CMakeLists.txt
index 2b17aa912016..51849d2d0fc0 100644
--- i/llvm/lib/Bitcode/Writer/CMakeLists.txt
+++ w/llvm/lib/Bitcode/Writer/CMakeLists.txt
@@ -17,2 +17,4 @@ add_llvm_component_library(LLVMBitWriter
   TargetParser
+  TransformUtils
   )

The other issue (error: undefined symbol: llvm::ThinLTOBitcodeWriterPass::run) is more tricky: LLVMipo (llvm/lib/Transforms/IPO/CMakeLists.txt) depends on LLVMBitWriter, so LLVMBitWriter cannot depend on LLVMipo.

What if we move the pass into transforms? Yes, its fairly correlated to bitcode writing, but the pass is really just creating a new section using existing utilities, which could just be considered a transform. There's also an argument that this pass should just live wherever the ThinLTOBitcodeWritterPass does. I think that should fix both the issues you've mentioned here.

Do you think that would be a better solution?

[paulkirth]

Address comments about documentation.

I still need to address more of @MaskRay's comments.

[aeubanks]

In D146776#4435430, @paulkirth wrote:

In D146776#4430608, @MaskRay wrote:

Started looking...

-DBUILD_SHARED_LIBS=on uses -Wl,-z,defs and can detect some library layering (https://llvm.org/docs/CodingStandards.html#library-layering https://maskray.me/blog/2021-06-13-dependency-related-linker-options) problems. ninja LLVMBitWriter doesn't build due to missing dependencies.

One issue can be addressed with the following change but the dependency may be a bit weird.

diff --git i/llvm/lib/Bitcode/Writer/CMakeLists.txt w/llvm/lib/Bitcode/Writer/CMakeLists.txt
index 2b17aa912016..51849d2d0fc0 100644
--- i/llvm/lib/Bitcode/Writer/CMakeLists.txt
+++ w/llvm/lib/Bitcode/Writer/CMakeLists.txt
@@ -17,2 +17,4 @@ add_llvm_component_library(LLVMBitWriter
   TargetParser
+  TransformUtils
   )

The other issue (error: undefined symbol: llvm::ThinLTOBitcodeWriterPass::run) is more tricky: LLVMipo (llvm/lib/Transforms/IPO/CMakeLists.txt) depends on LLVMBitWriter, so LLVMBitWriter cannot depend on LLVMipo.

What if we move the pass into transforms? Yes, its fairly correlated to bitcode writing, but the pass is really just creating a new section using existing utilities, which could just be considered a transform. There's also an argument that this pass should just live wherever the ThinLTOBitcodeWritterPass does. I think that should fix both the issues you've mentioned here.

Do you think that would be a better solution?

yes living alongside ThinLTOBitcodeWritterPass makes sense

[paulkirth]

Address some comments

• Move EmbedBitcodePass into Transforms/IPO
• Fix misc typos + formatting
• Simplify some code in EmbedBitcodePass

I have not addressed anything related to the SHT_LLVM_LTO flag or handling of .llvm.lto yet.

[paulkirth]

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

54–57 ↗

(On Diff #532989)

Looks like I forgot to remove these lines.

[paulkirth]

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

58 ↗

(On Diff #532989)

Also, while I'm not arguing that we shouldn't add SHF_LLVM_LTO, embedBufferInModule does mark it as MD_exclude https://github.com/llvm/llvm-project/blob/f1a040298381cdfc0657d5ecba231ebe6bbef61a/llvm/lib/Transforms/Utils/ModuleUtils.cpp#L375, which will eventually become SHF_EXCLUDE.

[MaskRay]

Thanks for the update. This fixes -DBUILD_SHARED_LIBS=on build of mine.

Perhaps llvm/test/Bitcode/ tests (which don't normally test -passes=) should be moved to elsewhere, e.g. llvm/test/Transforms/EmbedBitcodePass/ ?

Another strange thing to fix:

llc a.ll => .type .Lllvm.embedded.object,@object

We don't normally set attributes on temporary symbols (.L), which do not normally have symbol table entries anyway.

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

9 ↗

(On Diff #532989)

If EmbedBitcodePass.h provides a descriptive comment, this comment can probably be removed. It does not give additional information to users:)

42 ↗

(On Diff #532989)

delete unneeded blank line

47 ↗

(On Diff #532989)

delete unneeded blank line

[MaskRay]

In D146776#4435885, @MaskRay wrote:

Thanks for the update. This fixes -DBUILD_SHARED_LIBS=on build of mine.

Perhaps llvm/test/Bitcode/ tests (which don't normally test -passes=) should be moved to elsewhere, e.g. llvm/test/Transforms/EmbedBitcodePass/ ?

Another strange thing to fix:

llc a.ll => .type .Lllvm.embedded.object,@object

We don't normally set attributes on temporary symbols (.L), which do not normally have symbol table entries anyway.

This is due to private but I think this is acceptable. Both GNU assembler and LLVM integrated assembler do not create a symbol table entry, so this is fine. This likely does not justify adding a special case for private lowering to the underlying ELF section.

Feel free to land this one with a readelf -S test to check SHF_EXCLUDE. Maybe give some time for the Clang patch. Landing this one shall make reviewing the Clang part easier.

llvm/docs/FatLTO.rst

62 ↗

(On Diff #532989)

LLVMgold.so plugin.

Consider using hypertext to link this to llvm/docs/GoldPlugin.rst. I don't recall immediately the reST syntax...

[paulkirth]

In D146776#4435885, @MaskRay wrote:

Thanks for the update. This fixes -DBUILD_SHARED_LIBS=on build of mine.

Perhaps llvm/test/Bitcode/ tests (which don't normally test -passes=) should be moved to elsewhere, e.g. llvm/test/Transforms/EmbedBitcodePass/ ?

Oh, yes. I should have moved those too. I'll take care of that in the next version

In D146776#4435915, @MaskRay wrote:

In D146776#4435885, @MaskRay wrote:

Thanks for the update. This fixes -DBUILD_SHARED_LIBS=on build of mine.

Perhaps llvm/test/Bitcode/ tests (which don't normally test -passes=) should be moved to elsewhere, e.g. llvm/test/Transforms/EmbedBitcodePass/ ?

Another strange thing to fix:

llc a.ll => .type .Lllvm.embedded.object,@object

We don't normally set attributes on temporary symbols (.L), which do not normally have symbol table entries anyway.

This is due to private but I think this is acceptable. Both GNU assembler and LLVM integrated assembler do not create a symbol table entry, so this is fine. This likely does not justify adding a special case for private lowering to the underlying ELF section.

I'm not sure I follow the conversation here. Is this an issue w/ the defaults used by embedBufferInModule()? or is the oddity just related to the symol being private?

Feel free to land this one with a readelf -S test to check SHF_EXCLUDE. Maybe give some time for the Clang patch. Landing this one shall make reviewing the Clang part easier.

Will do. Thanks for all the feedback.

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

9 ↗

(On Diff #532989)

noted. I'll expand the description in the header and remove this.

[paulkirth]

Address comments

• Add test for SHT_EXCLUDE
• Fix some formatting
• Improve documentation/description in header
• Update documentation links

[paulkirth]

Reopening until I can investigate the buildbot failures.

[MaskRay]

llvm/test/Transforms/EmbedBitcode/embed.ll

23 ↗

(On Diff #534016)

We normally indent continuation lines with 2 spaces

43 ↗

(On Diff #534016)

delete trailing blank line

[MaskRay]

llvm/test/Transforms/EmbedBitcode/embed.ll

1 ↗

(On Diff #534016)

llc needs REQUIRES: x86-registered-target

[MaskRay]

llvm/test/Transforms/EmbedBitcode/embed.ll

24 ↗

(On Diff #534016)

--elf-output-style=JSON --pretty-print is not normal. llvm-readelf -S output is much conciser.

[paulkirth]

Fix test code

• move llc check for SHF_EXCLUDE under CodeGen/X86
• fix formatting

[paulkirth]

Update summary and title for reland

[paulkirth]

Remove some unused headers & fix formatting of the include guards

[MaskRay]

Thanks! One nit in the test.

llvm/test/CodeGen/X86/fat-lto-section.ll

8 ↗

(On Diff #534074)

I typically just drop [Nr] and [ 4] from the output so that adding a new section will not cause the section index to shuffle and cause the test to fail.

[paulkirth]

Make test less brittle to ordering changes

[paulkirth]

This seems to have failed on expensive checks because we marked the pass as PreservedAnalyses::all(). I guess the simplest thing to do is to change that to none, but I think there should be something else we can do here to avoid invalidating analyses that aren't affected by inserting the metadata section.

[paulkirth]

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

51 ↗

(On Diff #534116)

Is there another option we can use here. none seems like overkill. Since this is basically the first thing to run on the module, I'm not sure if there is much point in trying to do anything smarter, but it certainly doesn't feel right.

[aeubanks]

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

51 ↗

(On Diff #534116)

perhaps something like https://reviews.llvm.org/D153855?

[paulkirth]

Don't preserve analysis to satisfy expensive checks

[paulkirth]

Modify the structural hash since we don't modify the module except to add a
metadata section.

[paulkirth]

llvm/lib/Transforms/IPO/EmbedBitcodePass.cpp

51 ↗

(On Diff #534116)

That seems like it will work. Thanks for the suggestion. We can go back to none if we're unhappy about amending the structural hash.

[aeubanks]

llvm/lib/IR/StructuralHash.cpp

62 ↗

(On Diff #535511)

maybe we should exclude anything that begins with llvm., but that should be a separate patch, so this lgtm

[paulkirth]

Update comment.

[paulkirth]

llvm/lib/IR/StructuralHash.cpp

62 ↗

(On Diff #535511)

Filed https://github.com/llvm/llvm-project/issues/63590. I can make the follow up patch, but I may not get a chance till tomorrow.

llvm/docs/FatLTOSupport.rst

+===================
+FatLTO Support
+===================
+.. contents::
+
+.. toctree::
+   :maxdepth: 1
+
+Introduction
+============
+
+FatLTO objects are a special type of `fat object file
+<https://en.wikipedia.org/wiki/Fat_binary>`_ that contain LTO compatible IR in
+addition to generated object code, instead of containing object code for
+multiple target architectures. This allows users to defer the choice of whether
+to use LTO or not to link-time, and has been a feature available in other
+compilers, like `GCC
+<https://gcc.gnu.org/onlinedocs/gccint/LTO-Overview.html>`_ for some time.
+
+Overview
+========
+
+Within LLVM, FatLTO is supported by choosing the ``FatLTODefaultPipeline``.
+This pipeline needs to:
+
+#) Clone the IR module.
+#) Run the pre-link (Thin)LTO pipeline using the cloned module.
+#) Embed the pre-link bitcode a special `.llvm.lto` section.
+#) Optimize and the unmodified copy of the module using the normal compilation pipeline.
+#) Emit the object file, including the new `.llvm.lto` section.
+
+The ``FatLTODefaultPipeline`` does this by running the pre-link (Thin)LTO
+pipeline, saving the module in a new ``.llvm.lto`` section, and then running
+the `ModuleSimplificationPipeline` and the `ModuleOptimizationPipeline` to
+generate optimized object code for the module.
+
+.. NOTE
+   We conservatively run independent pipelines that are identical to those used
+   outside of FatLTO, so that we can guarantee that the compiled artifacts are
+   identical to those produced by the default and (Thin)LTO pipelines.
+
+Internally, the ``.llvm.lto`` section is created by running the
+``EmbedBitcodePass`` directly after the (Thin)LTO pre-link pipeline. When
+compiling for (Thin)LTO, this is normally the point at which the compiler would
+emit a object file containing the bitcode and metadata. However, for FatLTO, we
+still want to emit optimized object code, so after the section is created, we
+then run the normal compilation pipeline on an unmodified copy of the module.
+
+At the end of this process the compiler can emit standard object files which
+contain both the machine code in ``.text`` and the bitcode in ``.llvm.lto``.
+
+Limitations
+===========
+
+Linkers
+-------
+
+Currently, using LTO with LLVM fat lto objects is only supported with LLD. This
+may change in the future, but extending support to other linkers isn't planned
+for now.
+
+.. NOTE
+   For standard linking the fat object files should be usable by any
+   linker capable of using ELF objects, since the ``.llvm.lto`` section is
+   marked ``SHF_EXLUDE``.
+
+Supported File Formats
+----------------------
+
+The current implementation only supports ELF files. At time of writing, it is
+unclear if it will be useful to support other object file formats like ``COFF``
+or ``Mach-O``.

llvm/docs/ReleaseNotes.rst

 
 * Alloca merging in the inliner has been removed, since it only worked with the
   legacy inliner pass. Backend stack coloring should handle cases alloca
   merging initially set out to handle.
 
 * InstructionSimplify APIs now require instructions be inserted into a
   parent function.
 
+* A new FatLTO pipeline was added to support generating object files that have
+  both machine code and LTO compatible bitcode. See the `RFC
+  <https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977>`_ for more
+  details.
+
 Changes to building LLVM
 ------------------------
 
 Changes to TableGen
 -------------------
 
 Changes to Interprocedural Optimizations
 ----------------------------------------

llvm/docs/UserGuides.rst

.. toctree::
    CodeOfConduct
    CommandLine
    CompileCudaWithLLVM
    CoverageMappingFormat
    CycleTerminology
    DebuggingJITedCode
    DirectXUsage
    Docker
+   FatLTOSupport
    ExtendingLLVM
    GoldPlugin
    HowToBuildOnARM
    HowToBuildWithPGO
    HowToBuildWindowsItaniumPrograms
    HowToCrossCompileBuiltinsOnArm
    HowToCrossCompileLLVM
    HowToUpdateDebugInfo

llvm/include/llvm/Bitcode/EmbedBitcodePass.h

+//===-- EmbedBitcodePass.h - Embeds bitcode into global ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// This file provides a pass which embeds the bitcode into a global variable.
+///
+//===----------------------------------------------------------------------===//
+//
+#ifndef LLVM_BITCODE_EMBEDBITCODEPASS_H
+#define LLVM_BITCODE_EMBEDBITCODEPASS_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+class Module;
+class ModulePass;
+class Pass;
+
+struct EmbedBitcodeOptions {
+  EmbedBitcodeOptions() : EmbedBitcodeOptions(false, false) {}
+  EmbedBitcodeOptions(bool IsThinLTO, bool EmitLTOSummary)
+      : IsThinLTO(IsThinLTO), EmitLTOSummary(EmitLTOSummary) {}
+  bool IsThinLTO;
+  bool EmitLTOSummary;
+};
+
+/// Pass embeds the current module into a global variable.
+class EmbedBitcodePass : public PassInfoMixin<EmbedBitcodePass> {
+  bool IsThinLTO;
+  bool EmitLTOSummary;
+  ModulePassManager MPM;
+
+public:
+  EmbedBitcodePass(EmbedBitcodeOptions Opts)
+      : EmbedBitcodePass(Opts.IsThinLTO, Opts.EmitLTOSummary,
+                         ModulePassManager()) {}
+  EmbedBitcodePass(bool IsThinLTO, bool EmitLTOSummary, ModulePassManager &&MPM)
+      : IsThinLTO(IsThinLTO), EmitLTOSummary(EmitLTOSummary),
+        MPM(std::move(MPM)) {}
+
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &);
+
+  static bool isRequired() { return true; }
+};
+
+} // end namespace llvm.
+
+#endif

llvm/include/llvm/Passes/PassBuilder.h

public:
   ///
   /// This provides a good default optimization pipeline for per-module
   /// optimization and code generation without any link-time optimization. It
   /// typically correspond to frontend "-O[123]" options for optimization
   /// levels \c O1, \c O2 and \c O3 resp.
   ModulePassManager buildPerModuleDefaultPipeline(OptimizationLevel Level,
                                                   bool LTOPreLink = false);
 
+  /// Build a fat object default optimization pipeline.
+  ///
+  /// This builds a pipeline that runs the LTO/ThinLTO  pre-link pipeline, and
+  /// emits a section containing the pre-link bitcode along side the object code
+  /// generated by running the PerModuleDefaultPipeline, used when compiling
+  /// without LTO. It clones the module and runs the LTO/non-LTO pipelines
+  /// separately to avoid any inconsistencies with an ad-hoc pipeline that tries
+  /// to approximate the PerModuleDefaultPipeline from the pre-link LTO
+  /// pipelines.
+  ModulePassManager buildFatLTODefaultPipeline(OptimizationLevel Level,
+                                               bool ThinLTO, bool EmitSummary);
+
   /// Build a pre-link, ThinLTO-targeting default optimization pipeline to
   /// a pass manager.
   ///
   /// This adds the pre-link optimizations tuned to prepare a module for
   /// a ThinLTO run. It works to minimize the IR which needs to be analyzed
   /// without making irreversible decisions which could be made better during
   /// the LTO run.
   ModulePassManager buildThinLTOPreLinkDefaultPipeline(OptimizationLevel Level);

llvm/lib/Bitcode/Writer/CMakeLists.txt

 add_llvm_component_library(LLVMBitWriter
   BitWriter.cpp
   BitcodeWriter.cpp
   BitcodeWriterPass.cpp
+  EmbedBitcodePass.cpp
   ValueEnumerator.cpp
 
   DEPENDS
   intrinsics_gen
 
   LINK_COMPONENTS
   Analysis
   Core
   MC
   Object
   Support
   TargetParser
   )

llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp

+//===- EmbedBitcodePass.cpp - Pass that embeds the bitcode into a global---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// EmbedBitcodePass implementation.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Bitcode/EmbedBitcodePass.h"
+#include "llvm/Bitcode/BitcodeWriter.h"
+#include "llvm/Bitcode/BitcodeWriterPass.h"
+#include "llvm/IR/Constants.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+#include "llvm/Passes/PassBuilder.h"
+#include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/TargetParser/Triple.h"
+#include "llvm/Transforms/IPO/ThinLTOBitcodeWriter.h"
+#include "llvm/Transforms/Utils/Cloning.h"
+#include "llvm/Transforms/Utils/ModuleUtils.h"
+
+using namespace llvm;
+
+PreservedAnalyses EmbedBitcodePass::run(Module &M, ModuleAnalysisManager &AM) {
+  if (M.getGlobalVariable("llvm.embedded.module", /*AllowInternal=*/true))
+    report_fatal_error("Can only embed the module once.",
+                       /*gen_crash_diag=*/true);
+
+  Triple T(M.getTargetTriple());
+  if (T.getObjectFormat() != Triple::ELF)
+    report_fatal_error(
+        "EmbedBitcode pass currently only supports ELF object format.");
+
+  // Clone the module
+  std::unique_ptr<Module> NewModule = CloneModule(M);
+
+  MPM.run(*NewModule, AM);
+
+  std::string Data;
+  raw_string_ostream OS(Data);
+
+  if (IsThinLTO)
+    ThinLTOBitcodeWriterPass(OS, nullptr).run(*NewModule, AM);
+  else
+    BitcodeWriterPass(OS, /*ShouldPreserveUseListOrder=*/false, EmitLTOSummary)
+        .run(*NewModule, AM);
+
+  StringRef ModuleData(OS.str().data(), OS.str().size());
+  MemoryBufferRef Buf(ModuleData, "ModuleData");
+  embedBufferInModule(M, Buf, ".llvm.lto");
+
+  return PreservedAnalyses::all();
+}

llvm/lib/Object/ObjectFile.cpp

Error ObjectFile::printSymbolName(raw_ostream &OS, DataRefImpl Symb) const {
   return Error::success();
 }
 
 uint32_t ObjectFile::getSymbolAlignment(DataRefImpl DRI) const { return 0; }
 
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected<StringRef> NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-    return *NameOrErr == ".llvmbc";
+    return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
   consumeError(NameOrErr.takeError());
   return false;
 }
 
 bool ObjectFile::isSectionStripped(DataRefImpl Sec) const { return false; }
 
 bool ObjectFile::isBerkeleyText(DataRefImpl Sec) const {
   return isSectionText(Sec);

llvm/lib/Passes/PassBuilder.cpp

 #include "llvm/Analysis/ScalarEvolutionAliasAnalysis.h"
 #include "llvm/Analysis/ScopedNoAliasAA.h"
 #include "llvm/Analysis/StackLifetime.h"
 #include "llvm/Analysis/StackSafetyAnalysis.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Analysis/TypeBasedAliasAnalysis.h"
 #include "llvm/Analysis/UniformityAnalysis.h"
+#include "llvm/Bitcode/EmbedBitcodePass.h"
 #include "llvm/CodeGen/HardwareLoops.h"
 #include "llvm/CodeGen/TypePromotion.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/PrintPasses.h"
 #include "llvm/IR/SafepointIRVerifier.h"
 #include "llvm/IR/Verifier.h"
if (ParamName == "recover") {
           formatv("invalid HWAddressSanitizer pass parameter '{0}' ", ParamName)
               .str(),
           inconvertibleErrorCode());
     }
   }
   return Result;
 }
 
+Expected<EmbedBitcodeOptions> parseEmbedBitcodePassOptions(StringRef Params) {
+  EmbedBitcodeOptions Result;
+  while (!Params.empty()) {
+    StringRef ParamName;
+    std::tie(ParamName, Params) = Params.split(';');
+
+    if (ParamName == "thinlto") {
+      Result.IsThinLTO = true;
+    } else if (ParamName == "emit-summary") {
+      Result.EmitLTOSummary = true;
+    } else {
+      return make_error<StringError>(
+          formatv("invalid EmbedBitcode pass parameter '{0}' ", ParamName)
+              .str(),
+          inconvertibleErrorCode());
+    }
+  }
+  return Result;
+}
+
 Expected<MemorySanitizerOptions> parseMSanPassOptions(StringRef Params) {
   MemorySanitizerOptions Result;
   while (!Params.empty()) {
     StringRef ParamName;
     std::tie(ParamName, Params) = Params.split(';');
 
     if (ParamName == "recover") {
       Result.Recover = true;

llvm/lib/Passes/PassBuilderPipelines.cpp

 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/BasicAliasAnalysis.h"
 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/Analysis/ScopedNoAliasAA.h"
 #include "llvm/Analysis/TypeBasedAliasAnalysis.h"
+#include "llvm/Bitcode/EmbedBitcodePass.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Passes/OptimizationLevel.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/PGOOptions.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/Target/TargetMachine.h"
void PassBuilder::addVectorPasses(OptimizationLevel Level,
 
   if (IsFullLTO)
     FPM.addPass(InstCombinePass());
 }
 
 ModulePassManager
 PassBuilder::buildModuleOptimizationPipeline(OptimizationLevel Level,
                                              ThinOrFullLTOPhase LTOPhase) {
   const bool LTOPreLink = isLTOPreLink(LTOPhase);
   ModulePassManager MPM;
 
   // Run partial inlining pass to partially inline functions that have
   // large bodies.
   if (RunPartialInlining)
     MPM.addPass(PartialInlinerPass());
 
   // Remove avail extern fns and globals definitions since we aren't compiling
PassBuilder::buildPerModuleDefaultPipeline(OptimizationLevel Level,
 
   ModulePassManager MPM;
 
   // Convert @llvm.global.annotations to !annotation metadata.
   MPM.addPass(Annotation2MetadataPass());
 
   // Force any function attributes we want the rest of the pipeline to observe.
   MPM.addPass(ForceFunctionAttrsPass());
 
   if (PGOOpt && PGOOpt->DebugInfoForProfiling)
     MPM.addPass(createModuleToFunctionPassAdaptor(AddDiscriminatorsPass()));
 
   // Apply module pipeline start EP callback.
   invokePipelineStartEPCallbacks(MPM, Level);
 
   const ThinOrFullLTOPhase LTOPhase = LTOPreLink
                                           ? ThinOrFullLTOPhase::FullLTOPreLink
                                           : ThinOrFullLTOPhase::None;
   // Add the core simplification pipeline.
   MPM.addPass(buildModuleSimplificationPipeline(Level, LTOPhase));
 
   // Now add the optimization pipeline.
   MPM.addPass(buildModuleOptimizationPipeline(Level, LTOPhase));
 
   if (PGOOpt && PGOOpt->PseudoProbeForProfiling &&
       PGOOpt->Action == PGOOptions::SampleUse)
     MPM.addPass(PseudoProbeUpdatePass());
 
   // Emit annotation remarks.
   addAnnotationRemarksPass(MPM);
 
   if (LTOPreLink)
     addRequiredLTOPreLinkPasses(MPM);
+  return MPM;
+}
 
+ModulePassManager
+PassBuilder::buildFatLTODefaultPipeline(OptimizationLevel Level, bool ThinLTO,
+                                        bool EmitSummary) {
+  ModulePassManager MPM;
+  MPM.addPass(EmbedBitcodePass(ThinLTO, EmitSummary,
+                               ThinLTO
+                                   ? buildThinLTOPreLinkDefaultPipeline(Level)
+                                   : buildLTOPreLinkDefaultPipeline(Level)));
+  MPM.addPass(buildPerModuleDefaultPipeline(Level));
   return MPM;
 }
 
 ModulePassManager
 PassBuilder::buildThinLTOPreLinkDefaultPipeline(OptimizationLevel Level) {
   if (Level == OptimizationLevel::O0)
     return buildO0DefaultPipeline(Level, /*LTOPreLink*/true);
 
   ModulePassManager MPM;
 
   // Convert @llvm.global.annotations to !annotation metadata.
   MPM.addPass(Annotation2MetadataPass());
 
   // Force any function attributes we want the rest of the pipeline to observe.

llvm/lib/Passes/PassRegistry.def

 MODULE_PASS("coro-early", CoroEarlyPass())
 MODULE_PASS("coro-cleanup", CoroCleanupPass())
 MODULE_PASS("cross-dso-cfi", CrossDSOCFIPass())
 MODULE_PASS("deadargelim", DeadArgumentEliminationPass())
 MODULE_PASS("debugify", NewPMDebugifyPass())
 MODULE_PASS("dot-callgraph", CallGraphDOTPrinterPass())
 MODULE_PASS("elim-avail-extern", EliminateAvailableExternallyPass())
 MODULE_PASS("extract-blocks", BlockExtractorPass({}, false))
 MODULE_PASS("forceattrs", ForceFunctionAttrsPass())
 MODULE_PASS("function-import", FunctionImportPass())
 MODULE_PASS("globaldce", GlobalDCEPass())
 MODULE_PASS("globalopt", GlobalOptPass())
 MODULE_PASS("globalsplit", GlobalSplitPass())
 MODULE_PASS("hotcoldsplit", HotColdSplittingPass())
 MODULE_PASS("inferattrs", InferFunctionAttrsPass())
 MODULE_PASS("inliner-wrapper", ModuleInlinerWrapperPass())
 MODULE_PASS("inliner-ml-advisor-release", ModuleInlinerWrapperPass(getInlineParams(), true, {}, InliningAdvisorMode::Release, 0))
MODULE_PASS_WITH_PARAMS("msan",
                         "recover;kernel;eager-checks;track-origins=N")
 MODULE_PASS_WITH_PARAMS("ipsccp",
                         "IPSCCPPass",
                         [](IPSCCPOptions Opts) {
                           return IPSCCPPass(Opts);
                         },
                         parseIPSCCPOptions,
                         "no-func-spec;func-spec")
+MODULE_PASS_WITH_PARAMS("embed-bitcode",
+                         "EmbedBitcodePass",
+                        [](EmbedBitcodeOptions Opts){
+                          return EmbedBitcodePass(Opts);
+                        },
+                        parseEmbedBitcodePassOptions,
+                        "thinlto;emit-summary")
 #undef MODULE_PASS_WITH_PARAMS
 
 #ifndef CGSCC_ANALYSIS
 #define CGSCC_ANALYSIS(NAME, CREATE_PASS)
 #endif
 CGSCC_ANALYSIS("no-op-cgscc", NoOpCGSCCAnalysis())
 CGSCC_ANALYSIS("fam-proxy", FunctionAnalysisManagerCGSCCProxy())
 CGSCC_ANALYSIS("pass-instrumentation", PassInstrumentationAnalysis(PIC))

llvm/test/Bitcode/embed-multiple.ll

+; RUN: not --crash opt --mtriple x86_64-unknown-linux-gnu < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s
+
+@a = global i32 1
[e-mail] = private constant [4 x i8] c"BC\C0\DE"
+
+; CHECK: LLVM ERROR: Can only embed the module once.

llvm/test/Bitcode/embed-unsupported-object-format.ll

+; RUN: not --crash opt --mtriple powerpc64-unknown-aix < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s
+
+@a = global i32 1
+
+; CHECK: LLVM ERROR: EmbedBitcode pass currently only supports ELF object format

llvm/test/Bitcode/embed.ll

+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode" -S | FileCheck %s
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode<thinlto>" -S | FileCheck %s
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode<emit-summary>" -S | FileCheck %s
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes="embed-bitcode<thinlto;emit-summary>" -S | FileCheck %s
+
+@a = global i32 1
+
+; CHECK: @a = global i32 1
+;; Make sure the module is in the correct section
+; CHECK: @llvm.embedded.object = private constant {{.*}}, section ".llvm.lto", align 1
+
+;; Ensure that the metadata is in llvm.compiler.used
+; CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr @llvm.embedded.object], section "llvm.metadata"
+
+;; Make sue the metadata correlates to the .llvm.lto section
+; CHECK: !llvm.embedded.objects = !{!1}
+; CHECK: !0 = !{}
+; CHECK: !{ptr @llvm.embedded.object, !".llvm.lto"}