[clang] Preliminary fat-lto-object support

[paulkirth]

Summary

Fat LTO objects contain both LTO compatible IR, as well as generated
object code. This allows users to defer the choice of whether to use LTO
or not to link-time. This is a feature available in GCC for some time,
and makes the existing -ffat-lto-objects flag functional in the same
way as GCC's.

This patch adds support for that flag in the driver, as well as setting the
necessary codegen options for the backend. Largely, this means we select
the newly added pass pipeline for generating fat objects.

Users are expected to pass -ffat-lto-objects to clang in addition to one
of the -flto variants. Without the -flto flag, -ffat-lto-objects has no
effect.

// Compile and link. Use the object code from the fat object w/o LTO.
clang -fno-lto -ffat-lto-objects -fuse-ld=lld foo.c

// Compile and link. Select full LTO at link time.
clang -flto -ffat-lto-objects -fuse-ld=lld foo.c

// Compile and link. Select ThinLTO at link time.
clang -flto=thin -ffat-lto-objects -fuse-ld=lld foo.c

// Compile and link. Use ThinLTO with the UnifiedLTO pipeline.
clang -flto=thin -ffat-lto-objects -funified-lto -fuse-ld=lld foo.c

// Compile and link. Use full LTO with the UnifiedLTO pipeline.
clang -flto -ffat-lto-objects -funified-lto -fuse-ld=lld foo.c

// Link separately, using ThinLTO.
clang -c -flto=thin -ffat-lto-objects foo.c
clang -flto=thin -fuse-ld=lld foo.o -ffat-lto-objects # pass --lto=thin --fat-lto-objects to ld.lld

// Link separately, using full LTO.
clang -c -flto -ffat-lto-objects foo.c
clang -flto -fuse-ld=lld foo.o # pass --lto=full --fat-lto-objects to ld.lld

Original RFC: https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977

Depends on D146776

[paulkirth]

Fix typo in title

[paulkirth]

Rebase.

[paulkirth]

Rebase.

[paulkirth]

Rebase.

[paulkirth]

Rebase.

[paulkirth]

Add relese note.

[paulkirth]

Rebase

[paulkirth]

Rebase.

[paulkirth]

Rebase

[paulkirth]

Rebase

[paulkirth]

Rebase.

[paulkirth]

Rebase and pass fat-lto-objects to LLD from clang.

[tejohnson]

clang/lib/CodeGen/BackendUtil.cpp

1097

Can you expand the comment a bit - specifically why it won't end up in the above handling? I assume the Action type is different for FatLTO?

clang/lib/Driver/ToolChains/CommonArgs.cpp

620

nit: missing spaces

623

Needs a test

clang/test/CodeGen/embed-lto-fatlto.c

2 ↗

(On Diff #531534)

Can you also test -flto=thin, with and without LTO splitting enabled?

[paulkirth]

clang/lib/CodeGen/BackendUtil.cpp

1097

yes, we're not either of the above Action types. I'll expand that description to include more context, but maybe it should also be an else if(...)to make this code's relationship with the above more clear?

clang/lib/Driver/ToolChains/CommonArgs.cpp

620

TY. Looks like it needs a git clang-format. I'll be sure to run that again when I address the other comments.

623

Noted.

clang/test/CodeGen/embed-lto-fatlto.c

2 ↗

(On Diff #531534)

will do.

[paulkirth]

Rebase, address comments, and add missing test cases.

[tejohnson]

lgtm

clang/test/Driver/fatlto.c

1 ↗

(On Diff #532212)

Is this empty file meant to contain something?

[paulkirth]

Rebase and fix run line in test

[paulkirth]

clang/test/Driver/fatlto.c

1 ↗

(On Diff #532212)

No, this was just a mistake in my commit. I had the linker checks in another empty file, following the example from some of the lld/gold plugin tests and realized those should have just been in the existing file. I forgot to remove the empty file from git when updating the patch. I've removed it now.

[MaskRay]

There is a -fno-fat-lto-objects issue, but otherwise looks good after some nits are addressed. Thanks!

clang/docs/ReleaseNotes.rst

270

clang/include/clang/Driver/Options.td

2373

We just need the pos flag for CC1Option, so use OptInCC1FFlag

clang/lib/Driver/ToolChains/Clang.cpp

7404

This does not consider the negative option.

if (IsUsingLTO) {
  if (Arg *A = Args.getLastArg(options::OPT_ffat_lto_objects, options::OPT_fno_fat_lto_objects)) {
    if (A->getOption().matches(options::OPT_ffat_lto_objects)) {
      CmdArgs.push_back(...)
      A->render(Args, CmdArgs);
    }
  }
}

or move getLastArg outside of if (IsUsingLTO) to avoid -Wunused-command-line-argument in the absence of -flto=.

This code block probably should be moved before fglobal_isel to be closer to other LTO code.

clang/test/CodeGen/embed-lto-fatlto.c

1 ↗

(On Diff #532272)

Remove -S. In CC1, -emit-llvm wins.

8 ↗

(On Diff #532272)

11 ↗

(On Diff #532272)

clang/test/Driver/fatlto-objects.c

1 ↗

(On Diff #532272)

Delete the REQUIRES. The driver code should be portable as long as we only use -###.

1 ↗

(On Diff #532272)

fat-lto-objects.c or ffat-lto-objects.c

2 ↗

(On Diff #532272)

Delete -fintegrated-as. It's the default.

4 ↗

(On Diff #532272)

Use -SAME: whenever applicable.

7 ↗

(On Diff #532272)

You may test that -ffat-lto-objects in the absence of -flto= gives a warning: argument unused during compilation: '-ffat-lto-objects' warning

18 ↗

(On Diff #532272)

[MaskRay]

clang/docs/ReleaseNotes.rst

270

bitcode => LLVM bitcode

[MaskRay]

clang/lib/Driver/ToolChains/Clang.cpp

7405

We also need a err_drv_unsupported_opt_for_target error for non-ELF

[paulkirth]

Rebase and address comments

• Update & rename tests
• Fix typos
• Handle options

clang/include/clang/Driver/Options.td

2373

Wouldn't that prevent me from setting CodeGenOpts<"FatLTO">? AFAICT OptInCC1Flag doesn't take a CodeGenOpts since it isn't an OptionsFlag.

clang/test/CodeGen/embed-lto-fatlto.c

8 ↗

(On Diff #532272)

Extremely useful to know. Not sure how I've missed that until now. Thanks!

clang/test/Driver/fatlto-objects.c

7 ↗

(On Diff #532272)

Since I've slightly refactored the driver code based on your comments, we shouldn't get that warning anymore. I've tested that instead, but we can also drop it, since I'm not sure how much value that adds.

[paulkirth]

Rebase

[MaskRay]

As mentioned, you may consider landing llvm patch then wait a bit so that (a) people can experiment with the clang patch better (b) prevent the llvm/clang patches to be both reverted, if some issue has been identified with the llvm patch.
(Don't worry that a feature doesn't have contiguous commits. It's common.)

I do not accept the lld part now, just to have more time to ensure the llvm and clang part is mature :)

[paulkirth]

In D146777#4457209, @MaskRay wrote:

As mentioned, you may consider landing llvm patch then wait a bit so that (a) people can experiment with the clang patch better (b) prevent the llvm/clang patches to be both reverted, if some issue has been identified with the llvm patch.

Well, any later than Friday morning and it will have to wait until I get back from vacation. :) That isn't a big deal, though. and yeah, given the llvm patch had to be reverted twice, I'm fine w/ being a bit cautious.

(Don't worry that a feature doesn't have contiguous commits. It's common.)

Yeah, I'm just trying to keep this stack up to date w/ the rebases. When I've let them get very much out of sync it's caused my headaches in the past w/ phabricator not being able to apply the patch or because I hadn't kept them up to date and downloaded the patch from phabricator. I really miss Gerrits ability to upload the whole stack at once and have all the correct relationships. IDK if things will be better w/ github PRs, but I guess we'll all find out soon.

I do not accept the lld part now, just to have more time to ensure the llvm and clang part is mature :)

No problem. There's still some non trivial test improvements to implement on those anyway. Plus I need to figure out how to handle the archive case, which I don't think will be very hard, but I haven't looked at that code yet either.

[MaskRay]

In D146777#4457839, @paulkirth wrote:

In D146777#4457209, @MaskRay wrote:

As mentioned, you may consider landing llvm patch then wait a bit so that (a) people can experiment with the clang patch better (b) prevent the llvm/clang patches to be both reverted, if some issue has been identified with the llvm patch.

Well, any later than Friday morning and it will have to wait until I get back from vacation. :) That isn't a big deal, though. and yeah, given the llvm patch had to be reverted twice, I'm fine w/ being a bit cautious.

Sounds good!

(Don't worry that a feature doesn't have contiguous commits. It's common.)

Yeah, I'm just trying to keep this stack up to date w/ the rebases. When I've let them get very much out of sync it's caused my headaches in the past w/ phabricator not being able to apply the patch or because I hadn't kept them up to date and downloaded the patch from phabricator. I really miss Gerrits ability to upload the whole stack at once and have all the correct relationships. IDK if things will be better w/ github PRs, but I guess we'll all find out soon.

I hope that this will be useful: git rebase -i origin/main -x 'arc diff "HEAD^"'

I do not accept the lld part now, just to have more time to ensure the llvm and clang part is mature :)

No problem. There's still some non trivial test improvements to implement on those anyway. Plus I need to figure out how to handle the archive case, which I don't think will be very hard, but I haven't looked at that code yet either.

Thanks. I forgot that I mentioned archives:(

[paulkirth]

Rebase and try to accomodate Unified LTO changes.

Based on the Unified LTO patches, I think this is the correct handling for
FatLTO, but I'd like to get a second opinion before landing this.

CC: @ormris

[MaskRay]

clang/lib/Driver/ToolChains/CommonArgs.cpp

623

New lld long options only allow -- form. So use two dashes.

[paulkirth]

Use -- for LLD options

[MaskRay]

In D146777#4491842, @paulkirth wrote:

Rebase and try to accomodate Unified LTO changes.

Based on the Unified LTO patches, I think this is the correct handling for
FatLTO, but I'd like to get a second opinion before landing this.

CC: @ormris

Would you mind testing the behavior when both -funified-lto -ffat-lto-objects are specified and mentioning it in the summary? Some concrete instructions for end users will be very useful. See https://reviews.llvm.org/D123804#4491107 for what I suggested for -funified-lto.

[MaskRay]

clang/docs/ReleaseNotes.rst

271

You can insert the https://reviews.llvm.org/D146777 link here. It seems that libc++ and lld use a link more frequently.

[paulkirth]

Update summary w/ instructions for using -ffat-lto-objects

[MaskRay]

Thanks for the update. Can you add a comment for the -funified-lto combination? It's unclear what it does...

clang -flto=thin -ffat-lto-objects -funified-lto -fuse-ld=lld foo.c

I've left some comments about missing test coverage.

clang/lib/Driver/Driver.cpp

4736

This part is not tested.

clang/lib/Driver/ToolChains/CommonArgs.cpp

621

Tell LLD to find and use .llvm.lto section in regular relocatable object files.

clang/test/CodeGen/embed-fat-lto-objects.c

1 ↗

(On Diff #540192)

embed- in the filename seems unneeded? Perhaps just fat-lto-objects.c

3 ↗

(On Diff #540192)

We need a -emit-obj test with // REQUIRES: x86-registered-target.
Use llvm-readelf to check that .llvm.lto section is present.

We also need a -S test, otherwise the behavior of driver clang -S -ffat-lto-objects is untested.

clang/test/Driver/fat-lto-objects.c

11

This NOT pattern has no effect as warnings are usually emitted before -cc1.

You can use --implicit-check-not=warning:

[paulkirth]

In D146777#4499608, @MaskRay wrote:

Thanks for the update. Can you add a comment for the -funified-lto combination? It's unclear what it does...

ugh, I had put comments above each compiler invocation, but forgot that # makes lines ignored in the commit message. Thanks for pointing that out, as I hadn't noticed after updating.

clang -flto=thin -ffat-lto-objects -funified-lto -fuse-ld=lld foo.c

I've left some comments about missing test coverage.

ah, sorry, for some reason I read your last comment as test locally to confirm it works instead of the obvious write a proper test. I will correct that shortly.

clang/test/CodeGen/embed-fat-lto-objects.c

3 ↗

(On Diff #540192)

Sure, I can do both of those, but as a general question, isn't the generation of assembly the responsibility of the backend? I believe we have some tests that use llc on the the modules that have gone through the FatLTO pipeline. Is that insufficient?

Also, as a best practice, do you know if there are specific types of changes to clang that imply that we'd need to test -S? Sorry for all the questions, but I'm trying to get a bit better in how I structure my patches and approach testing.

clang/test/Driver/fat-lto-objects.c

11

I'm happy to try that suggestion, but isn't this testing the generated cc1 command from the driver(e.g., because we're using %clang and not %clang_cc1)? I think that should still produce the warning, shouldn't it?

It's been a while since I made the patch, but I recall writing this test, having it fail, and then applying the change to stop it from being an unused argument...

Also, thanks for all the great feedback on testing. Despite writing many(well, more than a few at least) lit tests, I'm still surprised by lots of behaviors, and your suggestions have been very helpful.

[paulkirth]

clang/lib/Driver/Driver.cpp

4736

oh, feel free to disregard my Q about -S. I missed this comment, and why you've asked for that is now obvious.

[MaskRay]

clang/test/CodeGen/embed-fat-lto-objects.c

3 ↗

(On Diff #540192)

Ah, sorry for my inaccurate suggestion. You are right.

To test clang/lib/Driver/Driver.cpp, we need to test the cc1 options of these driver options:

• -S -ffat-lto-objects -flto
• -S -emit-llvm -ffat-lto-objects -flto
• -c -ffat-lto-objects -flto (already tested)

test/CodeGen doesn't need more tests.

clang/test/Driver/fat-lto-objects.c

11

Yes, test/Driver just tests how Clang Driver generates warnings/errors and cc1 options. If there is a warning, it will be before the cc1 line, so // CHECK-CC-NOLTO-NOT: warning: argument unused during compilation: '-ffat-lto-objects' after cc1 will have no effect.

% myclang -flto -ffat-lto-objects -pie -c a.cc '-###'
clang version 17.0.0
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /tmp/Debug/bin
clang: warning: argument unused during compilation: '-pie' [-Wunused-command-line-argument]
 (in-process)
 "/tmp/Debug/bin/clang-17" "-cc1" ...

Actually, we usually don't want to check that a certain warning doesn't occur. Rather, we want to test no warning occurs, as it's easy to adjust a warning message and get stale tests.

In this case, I think I find a better test strategy: -fdriver-only -v -Werror. -fdriver-only ensures that an error changes the exit code and lit does check the exit code, fulfilling our testing requirement.

% myclang -fdriver-only -v -Werror -flto -ffat-lto-objects -pie -fdriver-only -c a.cc
clang: error: argument unused during compilation: '-pie' [-Werror,-Wunused-command-line-argument]
% echo $?
1

Also, thanks for all the great feedback on testing. Despite writing many(well, more than a few at least) lit tests, I'm still surprised by lots of behaviors, and your suggestions have been very helpful.

NP! Glad to help. I have spent too much time fiddling with driver options...

[paulkirth]

Revise tests + update summary

• Add driver tests for -S
• Add codegen tests for -S
• Test Unified LTO codegen
• Use -fdriver-only to test that -ffat-lto-objects isn't unused
• Test that the embedded bitcode has the correct module options set in object files
• Check that the .llvm.lto section exists when we generate ELF
• Make sure we set any FatLTO related module flags, even for other BackendActions(e.g., -emit-llvm), so that the IR emitted will match the IR embedded in the object file

I realized when writing the UnifiedLTO test that the existing implementation
wouldn't actually emit the correct IR using -emit-llvm for FatLTO. The flags
were set correctly in the .llvm.lto section, so I slightly adjusted the way we
set flags for the FatLTO case, so that -ffat-lto-objects -emit-llvm will have
the same module flags. The issue was that FatLTO isn't mutually exclusive w/
the other BackendActions, so my use of if else wasn't correct, and stopped us
from adding any flags when we used -emit-llvm. I missed this, since we set the
flags the same way.

The new tests in CodeGen may be a bit overkill, but they were already helpful in
finding this subtle bug.

[MaskRay]

Thank you for the update. I think this Clang patch is good enough to land even if the lld part is still in review.

[paulkirth]

Small forward fix for the documentation in D155490. If something else crops up I can revert both and reland.

clang/docs/ReleaseNotes.rst

 Modified Compiler Flags
 -----------------------
 
 - ``clang -g -gsplit-dwarf a.c -o obj/x`` (compile and link) now generates the
   ``.dwo`` file at ``obj/x-a.dwo``, instead of a file in the temporary
   directory (``/tmp`` on \*NIX systems, if none of the environment variables
   TMPDIR, TMP, and TEMP are specified).
 
+- ``-ffat-lto-objects`` can now be used to emit object files with both object
+  code and LLVM bitcode. Previously this flag was ignored for GCC compatibility.
+  (`See patch <https://reviews.llvm.org/D146777>`_).
+
 Removed Compiler Flags
 -------------------------
 - The deprecated flag `-fmodules-ts` is removed. Please use ``-std=c++20``
   or higher to use standard C++ modules instead.
 - The deprecated flag `-fcoroutines-ts` is removed. Please use ``-std=c++20``
   or higher to use standard C++ coroutines instead.
 - The CodeGen flag `-lower-global-dtors-via-cxa-atexit` which affects how global
   destructors are lowered for MachO is removed without replacement. The default

clang/include/clang/Basic/CodeGenOptions.def

 CODEGENOPT(LessPreciseFPMAD  , 1, 0) ///< Enable less precise MAD instructions to
                                      ///< be generated.
 CODEGENOPT(PrepareForLTO     , 1, 0) ///< Set when -flto is enabled on the
                                      ///< compile step.
 CODEGENOPT(PrepareForThinLTO , 1, 0) ///< Set when -flto=thin is enabled on the
                                      ///< compile step.
 CODEGENOPT(LTOUnit, 1, 0) ///< Emit IR to support LTO unit features (CFI, whole
                           ///< program vtable opt).
+CODEGENOPT(FatLTO, 1, 0) ///< Set when -ffat-lto-objects is enabled.
 CODEGENOPT(EnableSplitLTOUnit, 1, 0) ///< Enable LTO unit splitting to support
              /// CFI and traditional whole program
              /// devirtualization that require whole
              /// program IR support.
 CODEGENOPT(UnifiedLTO, 1, 0) ///< Use the unified LTO pipeline.
 CODEGENOPT(IncrementalLinkerCompatible, 1, 0) ///< Emit an object file which can
                                               ///< be used with an incremental
                                               ///< linker.

clang/include/clang/Driver/Options.td

HelpText<"Controls the backend parallelism of -flto=thin (default "
            "the number of CPUs detected)">;
 def fthinlto_index_EQ : Joined<["-"], "fthinlto-index=">,
   Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   HelpText<"Perform ThinLTO importing using provided function summary index">;
 def fthin_link_bitcode_EQ : Joined<["-"], "fthin-link-bitcode=">,
   Flags<[CoreOption, CC1Option]>, Group<f_Group>,
   HelpText<"Write minimized bitcode to <file> for the ThinLTO thin link only">,
   MarshallingInfoString<CodeGenOpts<"ThinLinkBitcodeFile">>;
+defm fat_lto_objects : BoolFOption<"fat-lto-objects",
+  CodeGenOpts<"FatLTO">, DefaultFalse,
+  PosFlag<SetTrue, [CC1Option], "Enable">,
+  NegFlag<SetFalse, [CC1Option], "Disable">,
+  BothFlags<[CC1Option], " fat LTO object support">>;
 def fmacro_backtrace_limit_EQ : Joined<["-"], "fmacro-backtrace-limit=">,
   Group<f_Group>, Flags<[NoXarchOption, CC1Option, CoreOption]>,
   HelpText<"Set the maximum number of entries to print in a macro expansion backtrace (0 = no limit)">,
   MarshallingInfoInt<DiagnosticOpts<"MacroBacktraceLimit">, "DiagnosticOptions::DefaultMacroBacktraceLimit">;
 def fcaret_diagnostics_max_lines_EQ :
   Joined<["-"], "fcaret-diagnostics-max-lines=">,
   Group<f_Group>, Flags<[CC1Option, CoreOption]>,
   HelpText<"Set the maximum number of source lines to show in a caret diagnostic (0 = no limit).">,
defm check_new : BoolOption<"f", "check-new",
   LangOpts<"CheckNew">, DefaultFalse,
   PosFlag<SetTrue, [], "Do not assume C++ operator new may not return NULL">,
   NegFlag<SetFalse>, BothFlags<[CC1Option]>>;
 
 defm caller_saves : BooleanFFlag<"caller-saves">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm reorder_blocks : BooleanFFlag<"reorder-blocks">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm branch_count_reg : BooleanFFlag<"branch-count-reg">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm default_inline : BooleanFFlag<"default-inline">, Group<clang_ignored_gcc_optimization_f_Group>;
-defm fat_lto_objects : BooleanFFlag<"fat-lto-objects">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm float_store : BooleanFFlag<"float-store">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm friend_injection : BooleanFFlag<"friend-injection">, Group<clang_ignored_f_Group>;
 defm function_attribute_list : BooleanFFlag<"function-attribute-list">, Group<clang_ignored_f_Group>;
 defm gcse : BooleanFFlag<"gcse">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gcse_after_reload: BooleanFFlag<"gcse-after-reload">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gcse_las: BooleanFFlag<"gcse-las">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gcse_sm: BooleanFFlag<"gcse-sm">, Group<clang_ignored_gcc_optimization_f_Group>;
 defm gnu : BooleanFFlag<"gnu">, Group<clang_ignored_f_Group>;

clang/lib/CodeGen/BackendUtil.cpp

 #include "llvm/Support/Timer.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/TargetParser/SubtargetFeature.h"
 #include "llvm/TargetParser/Triple.h"
+#include "llvm/Transforms/IPO/EmbedBitcodePass.h"
 #include "llvm/Transforms/IPO/LowerTypeTests.h"
 #include "llvm/Transforms/IPO/ThinLTOBitcodeWriter.h"
 #include "llvm/Transforms/InstCombine/InstCombine.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/Instrumentation/AddressSanitizer.h"
 #include "llvm/Transforms/Instrumentation/AddressSanitizerOptions.h"
 #include "llvm/Transforms/Instrumentation/BoundsChecking.h"
 #include "llvm/Transforms/Instrumentation/DataFlowSanitizer.h"
if (!CodeGenOpts.DisableLLVMPasses) {
     if (!CodeGenOpts.MemoryProfileOutput.empty()) {
       PB.registerOptimizerLastEPCallback(
           [](ModulePassManager &MPM, OptimizationLevel Level) {
             MPM.addPass(createModuleToFunctionPassAdaptor(MemProfilerPass()));
             MPM.addPass(ModuleMemProfilerPass());
           });
     }
 
-    if (IsThinLTO || (IsLTO && CodeGenOpts.UnifiedLTO)) {
+    bool IsThinOrUnifiedLTO = IsThinLTO || (IsLTO && CodeGenOpts.UnifiedLTO);
+    if (CodeGenOpts.FatLTO) {
+      MPM = PB.buildFatLTODefaultPipeline(Level, IsThinOrUnifiedLTO,
+                                          IsThinOrUnifiedLTO ||
+                                              shouldEmitRegularLTOSummary());
+    } else if (IsThinOrUnifiedLTO) {
       MPM = PB.buildThinLTOPreLinkDefaultPipeline(Level);
     } else if (IsLTO) {
       MPM = PB.buildLTOPreLinkDefaultPipeline(Level);
     } else {
       MPM = PB.buildPerModuleDefaultPipeline(Level);
     }
   }
 
if (CodeGenOpts.PrepareForThinLTO && !CodeGenOpts.DisableLLVMPasses) {
       if (Action == Backend_EmitBC)
         MPM.addPass(BitcodeWriterPass(*OS, CodeGenOpts.EmitLLVMUseLists,
                                       EmitLTOSummary));
       else
         MPM.addPass(PrintModulePass(*OS, "", CodeGenOpts.EmitLLVMUseLists,
                                     EmitLTOSummary));
     }
   }
+  if (CodeGenOpts.FatLTO) {
+    // Set module flags, like EnableSplitLTOUnit and UnifiedLTO, since FatLTO
+    // uses a different action than Backend_EmitBC or Backend_EmitLL.
+    bool IsThinOrUnifiedLTO =
+        CodeGenOpts.PrepareForThinLTO ||
+        (CodeGenOpts.PrepareForLTO && CodeGenOpts.UnifiedLTO);
+    if (!TheModule->getModuleFlag("ThinLTO"))
+      TheModule->addModuleFlag(Module::Error, "ThinLTO",
+                               uint32_t(IsThinOrUnifiedLTO));
+    if (!TheModule->getModuleFlag("EnableSplitLTOUnit"))
+      TheModule->addModuleFlag(Module::Error, "EnableSplitLTOUnit",
+                               uint32_t(CodeGenOpts.EnableSplitLTOUnit));
+    if (CodeGenOpts.UnifiedLTO && !TheModule->getModuleFlag("UnifiedLTO"))
+      TheModule->addModuleFlag(Module::Error, "UnifiedLTO", uint32_t(1));
+  }
 
   // Now that we have all of the passes ready, run them.
   {
     PrettyStackTraceString CrashInfo("Optimizer");
     llvm::TimeTraceScope TimeScope("Optimizer");
     MPM.run(*TheModule, MAM);
   }
 }
 
 void EmitAssemblyHelper::RunCodegenPipeline(
     BackendAction Action, std::unique_ptr<raw_pwrite_stream> &OS,

clang/lib/Driver/Driver.cpp

case phases::Compile: {
     if (Args.hasArg(options::OPT_verify_pch))
       return C.MakeAction<VerifyPCHJobAction>(Input, types::TY_Nothing);
     if (Args.hasArg(options::OPT_extract_api))
       return C.MakeAction<ExtractAPIJobAction>(Input, types::TY_API_INFO);
     return C.MakeAction<CompileJobAction>(Input, types::TY_LLVM_BC);
   }
   case phases::Backend: {
     if (isUsingLTO() && TargetDeviceOffloadKind == Action::OFK_None) {
-      types::ID Output =
-          Args.hasArg(options::OPT_S) ? types::TY_LTO_IR : types::TY_LTO_BC;
+      types::ID Output;
+      if (Args.hasArg(options::OPT_S))
+        Output = types::TY_LTO_IR;
+      else if (Args.hasArg(options::OPT_ffat_lto_objects))
+        Output = types::TY_PP_Asm;
+      else
+        Output = types::TY_LTO_BC;
       return C.MakeAction<BackendJobAction>(Input, Output);
     }
     if (isUsingLTO(/* IsOffload */ true) &&
         TargetDeviceOffloadKind != Action::OFK_None) {
       types::ID Output =
           Args.hasArg(options::OPT_S) ? types::TY_LTO_IR : types::TY_LTO_BC;
       return C.MakeAction<BackendJobAction>(Input, Output);
     }

clang/lib/Driver/ToolChains/Clang.cpp

bool SplitLTOUnit =
       Args.hasFlag(options::OPT_fsplit_lto_unit,
                    options::OPT_fno_split_lto_unit, DefaultsSplitLTOUnit);
   if (SanitizeArgs.needsLTO() && !SplitLTOUnit)
     D.Diag(diag::err_drv_argument_not_allowed_with) << "-fno-split-lto-unit"
                                                     << "-fsanitize=cfi";
   if (SplitLTOUnit)
     CmdArgs.push_back("-fsplit-lto-unit");
 
+  if (Arg *A = Args.getLastArg(options::OPT_ffat_lto_objects,
+                               options::OPT_fno_fat_lto_objects)) {
+    if (IsUsingLTO && A->getOption().matches(options::OPT_ffat_lto_objects)) {
+      assert(LTOMode == LTOK_Full || LTOMode == LTOK_Thin);
+      if (!Triple.isOSBinFormatELF()) {
+        D.Diag(diag::err_drv_unsupported_opt_for_target)
+            << A->getAsString(Args) << TC.getTripleString();
+      }
+      CmdArgs.push_back(Args.MakeArgString(
+          Twine("-flto=") + (LTOMode == LTOK_Thin ? "thin" : "full")));
+      CmdArgs.push_back("-flto-unit");
+      CmdArgs.push_back("-ffat-lto-objects");
+      A->render(Args, CmdArgs);
+    }
+  }
+
   if (Arg *A = Args.getLastArg(options::OPT_fglobal_isel,
                                options::OPT_fno_global_isel)) {
     CmdArgs.push_back("-mllvm");
     if (A->getOption().matches(options::OPT_fglobal_isel)) {
       CmdArgs.push_back("-global-isel=1");
 
       // GISel is on by default on AArch64 -O0, so don't bother adding
       // the fallback remarks for it. Other combinations will add a warning of
if (A->getOption().matches(options::OPT_fglobal_isel)) {
         else
           D.Diag(diag::warn_drv_global_isel_incomplete_opt);
       }
     } else {
       CmdArgs.push_back("-global-isel=0");
     }
   }
 
   if (Args.hasArg(options::OPT_forder_file_instrumentation)) {
      CmdArgs.push_back("-forder-file-instrumentation");
      // Enable order file instrumentation when ThinLTO is not on. When ThinLTO is
      // on, we need to pass these flags as linker flags and that will be handled
      // outside of the compiler.
      if (!IsUsingLTO) {
        CmdArgs.push_back("-mllvm");
        CmdArgs.push_back("-enable-order-file-instrumentation");
      }
   }

clang/lib/Driver/ToolChains/CommonArgs.cpp

 #endif
 
     SmallString<1024> Plugin;
     llvm::sys::path::native(Twine(D.Dir) +
                                 "/../" CLANG_INSTALL_LIBDIR_BASENAME +
                                 PluginName + Suffix,
                             Plugin);
     CmdArgs.push_back(Args.MakeArgString(Twine(PluginPrefix) + Plugin));
+  } else {
+    // Tell LLD to find and use .llvm.lto section in regular relocatable object
+    // files
+    if (Args.hasArg(options::OPT_ffat_lto_objects))
+      CmdArgs.push_back("--fat-lto-objects");
   }
 
   const char *PluginOptPrefix = IsOSAIX ? "-bplugin_opt:" : "-plugin-opt=";
   const char *ExtraDash = IsOSAIX ? "-" : "";
 
   // Note, this solution is far from perfect, better to encode it into IR
   // metadata, but this may not be worth it, since it looks like aranges is on
   // the way out.

clang/test/CodeGen/fat-lto-objects.c

+// REQUIRES: x86-registered-target
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=full -ffat-lto-objects -fsplit-lto-unit -emit-llvm < %s  | FileCheck %s --check-prefixes=FULL,SPLIT
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=full -ffat-lto-objects -emit-llvm < %s  | FileCheck %s --check-prefixes=FULL,SPLIT
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=thin -fsplit-lto-unit -ffat-lto-objects -emit-llvm < %s  | FileCheck %s --check-prefixes=THIN,SPLIT
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=thin -ffat-lto-objects -emit-llvm < %s  | FileCheck %s --check-prefixes=THIN,NOSPLIT
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=full -ffat-lto-objects -fsplit-lto-unit -emit-obj < %s -o %t.full.split.o
+// RUN: llvm-readelf -S %t.full.split.o | FileCheck %s --check-prefixes=ELF
+// RUN: llvm-objcopy --dump-section=.llvm.lto=%t.full.split.bc %t.full.split.o
+// RUN: llvm-dis %t.full.split.bc -o - | FileCheck %s --check-prefixes=FULL,SPLIT,NOUNIFIED
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=full -ffat-lto-objects -emit-obj < %s -o %t.full.nosplit.o
+// RUN: llvm-readelf -S %t.full.nosplit.o | FileCheck %s --check-prefixes=ELF
+// RUN: llvm-objcopy --dump-section=.llvm.lto=%t.full.nosplit.bc %t.full.nosplit.o
+// RUN: llvm-dis %t.full.nosplit.bc -o - | FileCheck %s --check-prefixes=FULL,NOSPLIT,NOUNIFIED
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=thin -fsplit-lto-unit -ffat-lto-objects -emit-obj < %s -o %t.thin.split.o
+// RUN: llvm-readelf -S %t.thin.split.o | FileCheck %s --check-prefixes=ELF
+// RUN: llvm-objcopy --dump-section=.llvm.lto=%t.thin.split.bc %t.thin.split.o
+// RUN: llvm-dis %t.thin.split.bc -o - | FileCheck %s --check-prefixes=THIN,SPLIT,NOUNIFIED
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=thin -ffat-lto-objects -emit-obj < %s -o %t.thin.nosplit.o
+// RUN: llvm-readelf -S %t.thin.nosplit.o | FileCheck %s --check-prefixes=ELF
+// RUN: llvm-objcopy --dump-section=.llvm.lto=%t.thin.nosplit.bc %t.thin.nosplit.o
+// RUN: llvm-dis %t.thin.nosplit.bc -o -  | FileCheck %s --check-prefixes=THIN,NOSPLIT,NOUNIFIED
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=thin -funified-lto -ffat-lto-objects -emit-obj < %s -o %t.unified.o
+// RUN: llvm-readelf -S %t.unified.o | FileCheck %s --check-prefixes=ELF
+// RUN: llvm-objcopy --dump-section=.llvm.lto=%t.unified.bc %t.unified.o
+// RUN: llvm-dis %t.unified.bc -o - | FileCheck %s --check-prefixes=THIN,NOSPLIT,UNIFIED
+
+// RUN: %clang -cc1 -triple x86_64-unknown-linux-gnu -flto=full -ffat-lto-objects -fsplit-lto-unit -S < %s -o - \
+// RUN: | FileCheck %s --check-prefixes=ASM
+
+/// Check that the ThinLTO metadata is only set false for full LTO.
+// FULL: ![[#]] = !{i32 1, !"ThinLTO", i32 0}
+// THIN-NOT: ![[#]] = !{i32 1, !"ThinLTO", i32 0}
+
+/// Be sure we enable split LTO units correctly under -ffat-lto-objects.
+// SPLIT: ![[#]] = !{i32 1, !"EnableSplitLTOUnit", i32 1}
+// NOSPLIT: ![[#]] = !{i32 1, !"EnableSplitLTOUnit", i32 0}
+
+// UNIFIED: ![[#]] = !{i32 1, !"UnifiedLTO", i32 1}
+// NOUNIFIED-NOT: ![[#]] = !{i32 1, !"UnifiedLTO", i32 1}
+
+// ELF: .llvm.lto
+
+//      ASM: .section        .llvm.lto,"e",@progbits
+// ASM-NEXT: .Lllvm.embedded.object:
+// ASM-NEXT:        .asciz  "BC
+// ASM-NEXT: .size   .Lllvm.embedded.object
+
+int test(void) {
+  return 0xabcd;
+}

clang/test/Driver/clang_f_opts.c

 // CHECK-WARNING-DAG: optimization flag '-funsafe-loop-optimizations' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fuse-linker-plugin' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fvect-cost-model' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fvariable-expansion-in-unroller' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fweb' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fwhole-program' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fcaller-saves' is not supported
 // CHECK-WARNING-DAG: optimization flag '-freorder-blocks' is not supported
-// CHECK-WARNING-DAG: optimization flag '-ffat-lto-objects' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fmerge-constants' is not supported
 // CHECK-WARNING-DAG: optimization flag '-finline-small-functions' is not supported
 // CHECK-WARNING-DAG: optimization flag '-ftree-dce' is not supported
 // CHECK-WARNING-DAG: optimization flag '-ftree-ter' is not supported
 // CHECK-WARNING-DAG: optimization flag '-ftree-vrp' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fno-devirtualize' is not supported
 // CHECK-WARNING-DAG: optimization flag '-fno-devirtualize-speculatively' is not supported
 // CHECK-WARNING-DAG: the flag '-fslp-vectorize-aggressive' has been deprecated and will be ignored

clang/test/Driver/fat-lto-objects.c

+// RUN: %clang --target=x86_64-unknown-linux-gnu -flto -ffat-lto-objects -### %s -c 2>&1 | FileCheck %s -check-prefix=CHECK-CC
+// CHECK-CC: -cc1
+// CHECK-CC-SAME: -emit-obj
+// CHECK-CC-SAME: -ffat-lto-objects
+
+/// Without -flto -S will just emit normal ASM, so we don't expect -emit-{llvm,obj} or -ffat-lto-objects to be passed to cc1.
+// RUN: %clang --target=x86_64-unknown-linux-gnu -ffat-lto-objects -### %s -S 2>&1 | FileCheck %s -check-prefix=CHECK-CC-S
+// CHECK-CC-S: -cc1
+// CHECK-CC-S: -S
+// CHECK-CC-S-NOT: -emit-obj
+// CHECK-CC-S-NOT: -emit-llvm
+// CHECK-CC-S-NOT: -ffat-lto-objects
+
+/// When LTO is enabled, we expect LLVM IR output and -ffat-lto-objects to be passed to cc1.
+// RUN: %clang --target=x86_64-unknown-linux-gnu -flto -ffat-lto-objects -### %s -S 2>&1 | FileCheck %s -check-prefix=CHECK-CC-S-LTO
+// RUN: %clang --target=x86_64-unknown-linux-gnu -flto -ffat-lto-objects -### %s -S -emit-llvm 2>&1 | FileCheck %s -check-prefix=CHECK-CC-S-LTO
+// CHECK-CC-S-LTO: -cc1
+// CHECK-CC-S-LTO-SAME: -emit-llvm
+// CHECK-CC-S-LTO-SAME: -ffat-lto-objects
+
+/// Make sure we don't have a warning for -ffat-lto-objects being unused
+// RUN: %clang --target=x86_64-unknown-linux-gnu -ffat-lto-objects -fdriver-only -Werror -v %s -c 2>&1 | FileCheck %s -check-prefix=CHECK-CC-NOLTO
+// CHECK-CC-NOLTO: -cc1
+// CHECK-CC-NOLTO-SAME: -emit-obj
+// CHECK-CC-NOLTO-NOT: -ffat-lto-objects
+
+/// We need to pass an additional flag (--fat-lto-objects) to lld when linking w/ -flto -ffat-lto-objects
+/// But it should not be there when LTO is disabled w/ -fno-lto
+// RUN: %clang --target=x86_64-unknown-linux-gnu --sysroot=%S/Inputs/basic_cross_linux_tree %s \
+// RUN:   -fuse-ld=lld -flto -ffat-lto-objects -### 2>&1 | FileCheck --check-prefix=LTO %s
+// RUN: %clang --target=x86_64-unknown-linux-gnu --sysroot=%S/Inputs/basic_cross_linux_tree %s \
+// RUN:   -fuse-ld=lld -fno-lto -ffat-lto-objects -### 2>&1 | FileCheck --check-prefix=NOLTO %s
+// LTO: "--fat-lto-objects"
+// NOLTO-NOT: "--fat-lto-objects"