[LTO][ELF] Add selective --save-temps= option

[Northbadge]

Summary

Allows specific “temps” to be saved, instead of the current all-or-nothing nature of --save-temps. Multiple of these “temps” can be saved by specifying the argument multiple times.

[MaskRay]

lld/ELF/Config.h

220

use 0.

this isn't a common option. having a large size makes accessing other bool variables slow.

[MaskRay]

lld/test/ELF/lto/save-temps-eq.ll

3

Using glob with rm is dangerous.

14

You can use ls with glob. find in lit is very unusual.

101

use a normal x86_64 triple.

llvm/lib/LTO/LTOBackend.cpp

130

Avoid a reserved identifier. https://en.cppreference.com/w/cpp/language/identifiers

[Northbadge]

Resolved inline comments. Thanks for the review!

[tejohnson]

lld/test/ELF/lto/save-temps-eq.ll

20

Probably want to test to ensure which temp files were emitted by each of these? Edit: I do see now how you are removing them one by one and check for an empty %t.all at the end, but I think it would be clearer to check which file is emitted more directly.

Also, can you test that the options are composable? I.e. that you can specify multiple and get the union of the requested temps files?

99

It's really hard to follow what the checking below is doing, with the various for loops and diffs. Can the output file checks be made explicit, similar to what I suggested above?

[Northbadge]

Reworked the test case following feedback, thanks!

[MaskRay]

From a layering perspective, the feature is better tested on the llvm/ project, not lld/. You may need to change llvm-lto2 to check the values. Having tests in lld/test/ELF is fine too as that reflects how people use the functionality, but they should not be replacement for llvm/ side tests.
(llvm/test/ThinLTO/X86 and llvm/test/LTO/X86)

[MaskRay]

lld/ELF/Driver.cpp

1160

1161

Since there are only very few allowed values, SmallVector<StringRef, 0> may work better.
You may deduplicate the values in lld/ELF/Driver.cpp

How about this: remove bool config->saveTemps. If --save-temps is specified, populate the array with all allowed values.

1162

The comment is unnecessary.

lld/test/ELF/lto/save-temps-eq.ll

10

With mkdir %t && cd %t, all other %t references can be removed.
If you want to test a subdirectory behavior, create a directory without the %t prefix.

[Northbadge]

lld/ELF/Driver.cpp

1161

Wouldn't deduplicating the vector necessitate using a set (or sorting)? Also, if it were switched to being a vector there'd need to be a loop to check whether to save temps at a given stage, specifically at "prelink" (I placed a self-executing lambda there earlier on, it's not great for readability)

I agree with removing the bool though

[Northbadge]

Removed the saveTemps bool, updated test to use subdirectories

[tejohnson]

In D127778#3603536, @MaskRay wrote:

From a layering perspective, the feature is better tested on the llvm/ project, not lld/. You may need to change llvm-lto2 to check the values. Having tests in lld/test/ELF is fine too as that reflects how people use the functionality, but they should not be replacement for llvm/ side tests.
(llvm/test/ThinLTO/X86 and llvm/test/LTO/X86)

This is a good point, and for D127779 as well.

[Northbadge]

In D127778#3608725, @tejohnson wrote:

In D127778#3603536, @MaskRay wrote:

From a layering perspective, the feature is better tested on the llvm/ project, not lld/. You may need to change llvm-lto2 to check the values. Having tests in lld/test/ELF is fine too as that reflects how people use the functionality, but they should not be replacement for llvm/ side tests.
(llvm/test/ThinLTO/X86 and llvm/test/LTO/X86)

This is a good point, and for D127779 as well.

I defer to the both of you on that point. However, this patch and D127779 are more of an extension of --save-temps (in my opinion), and as such the tests are predicated on --save-temps being correct. I just assumed --save-temps was properly tested to begin with, but it seems to have only been tested within ELF/lto on a basic level. Would it be sufficient (for these 2 patches) to write llvm side tests for --save-temps as a separate patch?

[tejohnson]

In D127778#3609617, @Northbadge wrote:

In D127778#3608725, @tejohnson wrote:

In D127778#3603536, @MaskRay wrote:

From a layering perspective, the feature is better tested on the llvm/ project, not lld/. You may need to change llvm-lto2 to check the values. Having tests in lld/test/ELF is fine too as that reflects how people use the functionality, but they should not be replacement for llvm/ side tests.
(llvm/test/ThinLTO/X86 and llvm/test/LTO/X86)

This is a good point, and for D127779 as well.

I defer to the both of you on that point. However, this patch and D127779 are more of an extension of --save-temps (in my opinion), and as such the tests are predicated on --save-temps being correct. I just assumed --save-temps was properly tested to begin with, but it seems to have only been tested within ELF/lto on a basic level. Would it be sufficient (for these 2 patches) to write llvm side tests for --save-temps as a separate patch?

It's not the --save-temps option that should be tested by llvm-lto2, but rather the core LTO changes (which are linker independent) being introduced in the patches. Typically when we add functionality to LTO we test it independently of any linker, via llvm-lto2, in one or both of the test directories @MaskRay mentioned above, then also add tests for whatever linker is utilizing the LTO support.

I missed this in D127777, which should ideally also have some llvm-lto2 based testing. E.g. see the -thinlto-distributed-indexes option in llvm/tools/llvm-lto2/llvm-lto2.cpp which is testing the thinlto-index-only functionality (llvm-lto2.cpp also has a -save-temps option). There are a bunch of tests under llvm/test/ThinLTO/X86/ that utilize these options (and end up testing the related LTO functionality in the process).

[Northbadge]

In D127778#3609766, @tejohnson wrote:

In D127778#3609617, @Northbadge wrote:

In D127778#3608725, @tejohnson wrote:

In D127778#3603536, @MaskRay wrote:

From a layering perspective, the feature is better tested on the llvm/ project, not lld/. You may need to change llvm-lto2 to check the values. Having tests in lld/test/ELF is fine too as that reflects how people use the functionality, but they should not be replacement for llvm/ side tests.
(llvm/test/ThinLTO/X86 and llvm/test/LTO/X86)

This is a good point, and for D127779 as well.

I defer to the both of you on that point. However, this patch and D127779 are more of an extension of --save-temps (in my opinion), and as such the tests are predicated on --save-temps being correct. I just assumed --save-temps was properly tested to begin with, but it seems to have only been tested within ELF/lto on a basic level. Would it be sufficient (for these 2 patches) to write llvm side tests for --save-temps as a separate patch?

It's not the --save-temps option that should be tested by llvm-lto2, but rather the core LTO changes (which are linker independent) being introduced in the patches. Typically when we add functionality to LTO we test it independently of any linker, via llvm-lto2, in one or both of the test directories @MaskRay mentioned above, then also add tests for whatever linker is utilizing the LTO support.

I missed this in D127777, which should ideally also have some llvm-lto2 based testing. E.g. see the -thinlto-distributed-indexes option in llvm/tools/llvm-lto2/llvm-lto2.cpp which is testing the thinlto-index-only functionality (llvm-lto2.cpp also has a -save-temps option). There are a bunch of tests under llvm/test/ThinLTO/X86/ that utilize these options (and end up testing the related LTO functionality in the process).

Ah ok, thanks for clarifying that! I'll get on those right away along with a separate patch for D127777 's missing tests

[MaskRay]

Ping on the llvm-lto2 change.

lld/ELF/Config.h

219

Use llvm::DenseSet<StringRef>.

SmallSet performs worse due to std::set.

lld/ELF/Driver.cpp

1161

Add static const char *saveTempsValues[] = {...};

Then append the values in if (args.hasArg(OPT_save_temps)) {

Below change s == "resolution" || s == "preopt" || s == "postpromote" to llvm::is_contained(saveTempsValues, ...)

llvm/include/llvm/LTO/Config.h

18

remove

275

Use llvm::DenseSet<StringRef>.

llvm/lib/LTO/LTOBackend.cpp

152

Shall we omit post from postinternalize,postimport,postopt so that the names match the temporary file names?

[Northbadge]

Added LLVM-side tests, modified llvm-lto2, fixed some capitalization and typos in tests.

[Northbadge]

In D127778#3620784, @MaskRay wrote:

Ping on the llvm-lto2 change.

Oh, didn't see this, was prepping the patch, i'll get one the comments now

[Northbadge]

Removed post* prefix, use DenseSet<StringRef> to store args

lld/ELF/Driver.cpp

1161

Thanks! didn't know about those. I changed it to use llvm::find since I needed the static string's address, not sure if there's a better way

[MaskRay]

lld/ELF/Driver.cpp

1164

Use args.filtered(OPT_save_temps_eq) which avoids a temporary std::vector. Then you can just use the value since its backing memory has sufficient lifetime. And you can avoid adl_end below.

1166

std::end

1169

lld/test/ELF/lto/save-temps-eq.ll

20

Prefer cmp for binary comparison.

llvm/include/llvm/LTO/Config.h

275

Omit llvm:: prefix since the delcaration is in namespace llvm { ... }

llvm/lib/LTO/LTOBackend.cpp

86

omit llvm:: prefix

llvm/test/ThinLTO/X86/selective-save-temps.ll

6

If you use ;; for non-RUN non-CHECK lines, change this ; too. Be consistent.

15

-r per line may be too verbose. You can pack multiple -r on one line.

llvm/tools/llvm-lto2/llvm-lto2.cpp

75

selected

[Northbadge]

Update to use DenseSet, cleaned up tests and arg parsing. Thanks for the feedback!

[MaskRay]

\opt caused a Windows failure: https://buildkite.com/llvm-project/premerge-checks/builds/100428#0181b59c-9093-413f-bee2-39f2a6364181

Use --save-temps=opt

POSIX.1-2008 says:

A <backslash> that is not quoted shall preserve the literal value of the following character, with the exception of a <newline>.

therefore \opt is interpreted as opt. But Windows doesn't seem to do this.

Please fix them.

lld/ELF/Driver.cpp

1161

See https://llvm.org/docs/CodingStandards.html#use-auto-type-deduction-to-make-code-more-readable

lld/test/ELF/lto/save-temps-eq.ll

19

\opt caused a Windows failure: https://buildkite.com/llvm-project/premerge-checks/builds/100428#0181b59c-9093-413f-bee2-39f2a6364181

--save-temps=opt

POSIX.1-2008 says:

A <backslash> that is not quoted shall preserve the literal value of the following character, with the exception of a <newline>.

therefore \opt is interpreted as opt. But Windows doesn't seem to do this.

61

llvm/test/ThinLTO/X86/selective-save-temps.ll

2

This comment may be inverse (due to layering). The lld/ELF test can say that it is similar to this file.

llvm/tools/llvm-lto2/llvm-lto2.cpp

277

remove else since the early return pattern is used.

283

[Northbadge]

lld/test/ELF/lto/save-temps-eq.ll

19

Aha, was running into an issue with plain opt being translated/preprocessed to the full path of opt (the executable) , so it called --save-temps=/path/to/opt. Will see what I can do about it

[Northbadge]

Resolved inline comments; Also having buildbots test a potential fix for escaping characters on Windows (will split it into a separate patch if it works)

[MaskRay]

llvm/utils/lit/lit/TestRunner.py

1163 ↗

(On Diff #441760)

Don't introduce this unrelated change in this patch.

[MaskRay]

lld/test/ELF/lto/save-temps-eq.ll

61

I am confused. What does this %\opt do?

[MaskRay]

lld/test/ELF/lto/save-temps-eq.ll

61

Ah, ok. If this is to avoid lit substitution, does 'opt' work?

[Northbadge]

lld/test/ELF/lto/save-temps-eq.ll

61

Nope

llvm/utils/lit/lit/TestRunner.py

1163 ↗

(On Diff #441760)

Yep, got that! I don't have access to a windows machine to test so I'm having the buildbots do it before splitting it off to a separate patch

[Northbadge]

Walked back the lit change for cross-platform escape char (didn't work). Disabled the tests on Windows instead, let me know if I should avoid doing that. I'll probably land this after the long weekend. Thanks for all the feedback!

[MaskRay]

Unsupporting Windows is fine. I am surprised that lit substitution happens even with xxx='opt' and xxx="opt". I'd expect that it only triggers for xxx opt yyy.

lld/test/ELF/lto/save-temps-eq.ll

6

[Northbadge]

Missed semicolons.

lld/ELF/Config.h

 //===- Config.h -------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_CONFIG_H
 #define LLD_ELF_CONFIG_H
 
 #include "lld/Common/ErrorHandler.h"
 #include "llvm/ADT/CachedHashString.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Endian.h"
struct Configuration {
   bool picThunk;
   bool pie;
   bool printGcSections;
   bool printIcfSections;
   bool relax;
   bool relocatable;
   bool relrGlibc = false;
   bool relrPackDynRelocs = false;
-  bool saveTemps;
+  llvm::DenseSet<llvm::StringRef> saveTempsArgs;
   std::vector<std::pair<llvm::GlobPattern, uint32_t>> shuffleSections;
   bool singleRoRx;
   bool shared;
   bool symbolic;
   bool isStatic = false;
   bool sysvHash = false;
   bool target1Rel;
   bool trace;
   bool thinLTOEmitImportsFiles;

lld/ELF/Driver.cpp

 
 // Report a warning for an unknown -z option.
 static void checkZOptions(opt::InputArgList &args) {
   for (auto *arg : args.filtered(OPT_z))
     if (!isKnownZFlag(arg->getValue()))
       warn("unknown -z value: " + StringRef(arg->getValue()));
 }
 
+constexpr const char *saveTempsValues[] = {
+    "resolution", "preopt",     "promote", "internalize",  "import",
+    "opt",        "precodegen", "prelink", "combinedindex"};
+
 void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   ELFOptTable parser;
   opt::InputArgList args = parser.parse(argsArr.slice(1));
 
   // Interpret these flags early because error()/warn() depend on them.
   errorHandler().errorLimit = args::getInteger(args, OPT_error_limit, 20);
   errorHandler().fatalWarnings =
       args.hasFlag(OPT_fatal_warnings, OPT_no_fatal_warnings, false);
static void readConfigs(opt::InputArgList &args) {
   config->printGcSections =
       args.hasFlag(OPT_print_gc_sections, OPT_no_print_gc_sections, false);
   config->printArchiveStats = args.getLastArgValue(OPT_print_archive_stats);
   config->printSymbolOrder =
       args.getLastArgValue(OPT_print_symbol_order);
   config->relax = args.hasFlag(OPT_relax, OPT_no_relax, true);
   config->rpath = getRpath(args);
   config->relocatable = args.hasArg(OPT_relocatable);
-  config->saveTemps = args.hasArg(OPT_save_temps);
+
+  if (args.hasArg(OPT_save_temps)) {
+    // --save-temps implies saving all temps.
+    for (const char *s : saveTempsValues)
+      config->saveTempsArgs.insert(s);
+  } else {
+    for (auto *arg : args.filtered(OPT_save_temps_eq)) {
+      StringRef s = arg->getValue();
+      if (llvm::is_contained(saveTempsValues, s))
+        config->saveTempsArgs.insert(s);
+      else
+        error("unknown --save-temps value: " + s);
+    }
+  }
+
   config->searchPaths = args::getStrings(args, OPT_library_path);
   config->sectionStartMap = getSectionStartMap(args);
   config->shared = args.hasArg(OPT_shared);
   config->singleRoRx = !args.hasFlag(OPT_rosegment, OPT_no_rosegment, true);
   config->soName = args.getLastArgValue(OPT_soname);
   config->sortSection = getSortSection(args);
   config->splitStackAdjustSize = args::getInteger(args, OPT_split_stack_adjust_size, 16384);
   config->strip = getStrip(args);

lld/ELF/LTO.cpp

c.PostInternalizeModuleHook = [](size_t task, const Module &m) {
         WriteBitcodeToFile(m, *os, false);
       return false;
     };
   }
 
   if (config->ltoEmitAsm)
     c.CGFileType = CGFT_AssemblyFile;
 
-  if (config->saveTemps)
+  if (!config->saveTempsArgs.empty())
     checkError(c.addSaveTemps(config->outputFile.str() + ".",
-                              /*UseInputModulePath*/ true));
+                              /*UseInputModulePath*/ true,
+                              config->saveTempsArgs));
   return c;
 }
 
 BitcodeCompiler::BitcodeCompiler() {
   // Initialize indexFile.
   if (!config->thinLTOIndexOnlyArg.empty())
     indexFile = openFile(config->thinLTOIndexOnlyArg);
 
if (!config->thinLTOCacheDir.empty())
     pruneCache(config->thinLTOCacheDir, config->thinLTOCachePolicy);
 
   if (!config->ltoObjPath.empty()) {
     saveBuffer(buf[0], config->ltoObjPath);
     for (unsigned i = 1; i != maxTasks; ++i)
       saveBuffer(buf[i], config->ltoObjPath + Twine(i));
   }
 
-  if (config->saveTemps) {
+  if (config->saveTempsArgs.contains("prelink")) {
     if (!buf[0].empty())
       saveBuffer(buf[0], config->outputFile + ".lto.o");
     for (unsigned i = 1; i != maxTasks; ++i)
       saveBuffer(buf[i], config->outputFile + Twine(i) + ".lto.o");
   }
 
   if (config->ltoEmitAsm) {
     saveBuffer(buf[0], config->outputFile);

lld/ELF/Options.td

defm opt_remarks_hotness_threshold: EEq<"opt-remarks-hotness-threshold",
   MetaVarName<"<value>">;
 def opt_remarks_passes: Separate<["--"], "opt-remarks-passes">,
   HelpText<"Regex for the passes that need to be serialized to the output file">;
 def opt_remarks_with_hotness: FF<"opt-remarks-with-hotness">,
   HelpText<"Include hotness information in the optimization remarks file">;
 def opt_remarks_format: Separate<["--"], "opt-remarks-format">,
   HelpText<"The format used for serializing remarks (default: YAML)">;
 def save_temps: F<"save-temps">, HelpText<"Save intermediate LTO compilation results">;
+def save_temps_eq: JJ<"save-temps=">, HelpText<"Save select intermediate LTO compilation results">,
+  Values<"resolution,preopt,promote,internalize,import,opt,precodegen,prelink,combinedindex">;
 def lto_basic_block_sections: JJ<"lto-basic-block-sections=">,
   HelpText<"Enable basic block sections for LTO">;
 defm lto_unique_basic_block_section_names: BB<"lto-unique-basic-block-section-names",
     "Give unique names to every basic block section for LTO",
     "Do not give unique names to every basic block section for LTO (default)">;
 defm shuffle_sections: EEq<"shuffle-sections",
   "Shuffle matched sections using the given seed before mapping them to the output sections. "
   "If -1, reverse the section order. If 0, use a random seed">,

lld/test/ELF/lto/save-temps-eq.ll

+;; This test is similar to llvm/test/ThinLTO/X86/selective-save-temps.ll
+
+; REQUIRES: x86
+; UNSUPPORTED: system-windows
+;; Unsupported on Windows due to difficulty with escaping "opt" across platforms.
+;; lit substitutes 'opt' with /path/to/opt.
+
+; RUN: rm -rf %t && mkdir %t && cd %t
+; RUN: mkdir all all2 all3 build subset subset2 && cd build
+
+; RUN: opt -thinlto-bc -o main.o %s
+; RUN: opt -thinlto-bc -o thin1.o %S/Inputs/thinlto.ll
+
+;; Create the .all dir with save-temps saving everything, this will be used to compare
+;; with the output from individualized save-temps later
+; RUN: ld.lld main.o thin1.o --save-temps -o %t/all/a.out
+; RUN: mv *.o.* %t/all
+;; Sanity check that everything got moved
+; RUN: ls | count 2
+
+;; Check precedence if both --save-temps and --save-temps= are present
+; RUN: ld.lld main.o thin1.o --save-temps=preopt --save-temps --save-temps=\opt -o %t/all2/a.out
+; RUN: cmp %t/all2/a.out %t/all/a.out
+; RUN: mv *.o.* %t/all2
+; RUN: ls | count 2
+; RUN: diff -r %t/all %t/all2
+
+;; The next 9 blocks follow this structure:
+;; for each option of save-temps=
+;;   Run linker and generate files
+;;   Make sure a.out exists and is correct (by diff-ing)
+;;     this is the only file that should recur between runs
+;;   (Also, for some stages, copy the generated files to %t/subset2 to check composability later)
+;;   Move files that were expected to be generated to %t/all3
+;;   Make sure there's no unexpected extra files
+;; After that, we'll diff %t/all and %t/all3 to make sure all contents are identical
+
+;; Check preopt
+; RUN: ld.lld main.o thin1.o --save-temps=preopt
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: cp *.0.preopt.* %t/subset2
+; RUN: mv *.0.preopt.* %t/all3
+; RUN: ls | count 2
+
+;; Check promote
+; RUN: ld.lld main.o thin1.o --save-temps=promote
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: mv *.1.promote* %t/all3
+; RUN: ls | count 2
+
+;; Check internalize
+; RUN: ld.lld main.o thin1.o --save-temps=internalize
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: mv *.2.internalize* %t/all3
+; RUN: ls | count 2
+
+;; Check import
+; RUN: ld.lld main.o thin1.o --save-temps=import
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: mv *.3.import* %t/all3
+; RUN: ls | count 2
+
+;; Check opt
+; RUN: ld.lld main.o thin1.o --save-temps=\opt
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: cp *.4.opt* %t/subset2
+; RUN: mv *.4.opt* %t/all3
+; RUN: ls | count 2
+
+;; Check precodegen
+; RUN: ld.lld main.o thin1.o --save-temps=precodegen
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: mv *.5.precodegen* %t/all3
+; RUN: ls | count 2
+
+;; Check combinedindex
+; RUN: ld.lld main.o thin1.o --save-temps=combinedindex
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: mv *.index.bc %t/all3
+; RUN: mv *.index.dot %t/all3
+; RUN: ls | count 2
+
+;; Check prelink
+; RUN: ld.lld main.o thin1.o --save-temps=prelink
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: cp *.lto.o %t/subset2
+; RUN: mv *.lto.o %t/all3
+; RUN: ls | count 2
+
+;; Check resolution
+; RUN: ld.lld main.o thin1.o --save-temps=resolution
+;; %t/all3 needs at least 1 copy of a.out, move it over now since its the last block
+; RUN: mv a.out %t/all3
+; RUN: mv *.resolution.txt %t/all3
+; RUN: ls | count 2
+
+;; If no files were left out from individual stages, the .all3 dir should be identical to .all
+; RUN: diff -r %t/all %t/all3
+
+;; Check multi-stage composability
+;; Similar to the above, but do it with a subset instead.
+;; .all -> .subset, .all3 -> .subset2
+; RUN: ld.lld main.o thin1.o --save-temps=preopt --save-temps=prelink --save-temps=\opt
+; RUN: cmp %t/all/a.out a.out && rm -f a.out
+; RUN: mv *.0.preopt.* %t/subset
+; RUN: mv *.4.opt* %t/subset
+; RUN: mv *.lto.o %t/subset
+; RUN: ls | count 2
+; RUN: diff -r %t/subset2 %t/subset
+
+;; Check error message
+; RUN: not ld.lld --save-temps=prelink --save-temps=\opt --save-temps=notastage 2>&1 \
+; RUN: | FileCheck %s
+; CHECK: unknown --save-temps value: notastage
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare void @g()
+
+define i32 @_start() {
+  call void @g()
+  ret i32 0
+}

llvm/include/llvm/LTO/Config.h

 // configure LTO.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LTO_CONFIG_H
 #define LLVM_LTO_CONFIG_H
 
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/Passes/PassBuilder.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Target/TargetOptions.h"
 
struct Config {
   ///
   /// Specifically, it (1) sets each of the above module hooks and the combined
   /// index hook to a function that calls the hook function (if any) that was
   /// present in the appropriate field when the addSaveTemps function was
   /// called, and writes the module to a bitcode file with a name prefixed by
   /// the given output file name, and (2) creates a resolution file whose name
   /// is prefixed by the given output file name and sets ResolutionFile to its
   /// file handle.
+  ///
+  /// SaveTempsArgs can be specified to select which temps to save.
+  /// If SaveTempsArgs is not provided, all temps are saved.
   Error addSaveTemps(std::string OutputFileName,
-                     bool UseInputModulePath = false);
+                     bool UseInputModulePath = false,
+                     const DenseSet<StringRef> &SaveTempsArgs = {});
 };
 
 struct LTOLLVMDiagnosticHandler : public DiagnosticHandler {
   DiagnosticHandlerFunction *Fn;
   LTOLLVMDiagnosticHandler(DiagnosticHandlerFunction *DiagHandlerFn)
       : Fn(DiagHandlerFn) {}
   bool handleDiagnostics(const DiagnosticInfo &DI) override {
     (*Fn)(DI);

llvm/lib/LTO/LTOBackend.cpp

 }
 
 [[noreturn]] static void reportOpenError(StringRef Path, Twine Msg) {
   errs() << "failed to open " << Path << ": " << Msg << '\n';
   errs().flush();
   exit(1);
 }
 
-Error Config::addSaveTemps(std::string OutputFileName,
-                           bool UseInputModulePath) {
+Error Config::addSaveTemps(std::string OutputFileName, bool UseInputModulePath,
+                           const DenseSet<StringRef> &SaveTempsArgs) {
   ShouldDiscardValueNames = false;
 
   std::error_code EC;
+  if (SaveTempsArgs.empty() || SaveTempsArgs.contains("resolution")) {
     ResolutionFile =
         std::make_unique<raw_fd_ostream>(OutputFileName + "resolution.txt", EC,
                                          sys::fs::OpenFlags::OF_TextWithCRLF);
     if (EC) {
       ResolutionFile.reset();
       return errorCodeToError(EC);
     }
+  }
 
   auto setHook = [&](std::string PathSuffix, ModuleHookFn &Hook) {
     // Keep track of the hook provided by the linker, which also needs to run.
     ModuleHookFn LinkerHook = Hook;
     Hook = [=](unsigned Task, const Module &M) {
       // If the linker's hook returned false, we need to pass that result
       // through.
       if (LinkerHook && !LinkerHook(Task, M))
Hook = [=](unsigned Task, const Module &M) {
       // directly and exit.
       if (EC)
         reportOpenError(Path, EC.message());
       WriteBitcodeToFile(M, OS, /*ShouldPreserveUseListOrder=*/false);
       return true;
     };
   };
 
-  setHook("0.preopt", PreOptModuleHook);
-  setHook("1.promote", PostPromoteModuleHook);
-  setHook("2.internalize", PostInternalizeModuleHook);
-  setHook("3.import", PostImportModuleHook);
-  setHook("4.opt", PostOptModuleHook);
-  setHook("5.precodegen", PreCodeGenModuleHook);
-
-  CombinedIndexHook =
+  auto SaveCombinedIndex =
       [=](const ModuleSummaryIndex &Index,
           const DenseSet<GlobalValue::GUID> &GUIDPreservedSymbols) {
         std::string Path = OutputFileName + "index.bc";
         std::error_code EC;
         raw_fd_ostream OS(Path, EC, sys::fs::OpenFlags::OF_None);
         // Because -save-temps is a debugging feature, we report the error
         // directly and exit.
         if (EC)
           reportOpenError(Path, EC.message());
         writeIndexToFile(Index, OS);
 
         Path = OutputFileName + "index.dot";
         raw_fd_ostream OSDot(Path, EC, sys::fs::OpenFlags::OF_None);
         if (EC)
           reportOpenError(Path, EC.message());
         Index.exportToDot(OSDot, GUIDPreservedSymbols);
         return true;
       };
 
+  if (SaveTempsArgs.empty()) {
+    setHook("0.preopt", PreOptModuleHook);
+    setHook("1.promote", PostPromoteModuleHook);
+    setHook("2.internalize", PostInternalizeModuleHook);
+    setHook("3.import", PostImportModuleHook);
+    setHook("4.opt", PostOptModuleHook);
+    setHook("5.precodegen", PreCodeGenModuleHook);
+    CombinedIndexHook = SaveCombinedIndex;
+  } else {
+    if (SaveTempsArgs.contains("preopt"))
+      setHook("0.preopt", PreOptModuleHook);
+    if (SaveTempsArgs.contains("promote"))
+      setHook("1.promote", PostPromoteModuleHook);
+    if (SaveTempsArgs.contains("internalize"))
+      setHook("2.internalize", PostInternalizeModuleHook);
+    if (SaveTempsArgs.contains("import"))
+      setHook("3.import", PostImportModuleHook);
+    if (SaveTempsArgs.contains("opt"))
+      setHook("4.opt", PostOptModuleHook);
+    if (SaveTempsArgs.contains("precodegen"))
+      setHook("5.precodegen", PreCodeGenModuleHook);
+    if (SaveTempsArgs.contains("combinedindex"))
+      CombinedIndexHook = SaveCombinedIndex;
+  }
+
   return Error::success();
 }
 
 #define HANDLE_EXTENSION(Ext)                                                  \
   llvm::PassPluginLibraryInfo get##Ext##PluginInfo();
 #include "llvm/Support/Extension.def"
 
 static void RegisterPassPlugins(ArrayRef<std::string> PassPlugins,

llvm/test/ThinLTO/X86/selective-save-temps.ll

+; UNSUPPORTED: system-windows
+;; Unsupported on Windows due to difficulty with escaping "opt" across platforms.
+;; lit substitutes 'opt' with /path/to/opt.
+
+; RUN: rm -rf %t && mkdir %t && cd %t
+
+;; Copy IR from import-constant.ll since it generates all the temps
+; RUN: opt -thinlto-bc %s -o 1.bc
+; RUN: opt -thinlto-bc %p/Inputs/import-constant.ll -o 2.bc
+
+;; Create the .all dir with save-temps saving everything, this will be used to compare
+;; with the output from individualized save-temps later
+; RUN: mkdir all all2 build subset subset2
+; RUN: llvm-lto2 run 1.bc 2.bc -o all/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -save-temps
+
+;; The next 8 blocks follow this structure:
+;; for each option of save-temps=
+;;   Run lto and generate files
+;;   Make sure a.out exists and is correct (by diff-ing)
+;;     this is the only file that should recur between runs
+;;   (Also, for some stages, copy the generated files to subset2 to check composability later)
+;;   Move files that were expected to be generated to all2
+;;   Make sure there's no unexpected extra files
+;; After that, we'll diff all and all2 to make sure all contents are identical
+
+;; Check preopt
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=preopt
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: cp build/*.0.preopt.* subset2
+; RUN: mv build/*.0.preopt.* all2
+; RUN: ls build | count 0
+
+;; Check promote
+; RUN: rm -f all2/*.1.promote*
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=promote
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: mv build/*.1.promote* all2
+; RUN: ls build | count 0
+
+;; Check internalize
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=internalize
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: mv build/*.2.internalize* all2
+; RUN: ls build | count 0
+
+;; Check import
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=import
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: mv build/*.3.import* all2
+; RUN: ls build | count 0
+
+;; Check opt
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=\opt
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: cp build/*.4.opt* subset2
+; RUN: mv build/*.4.opt* all2
+; RUN: ls build | count 0
+
+;; Check precodegen
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=precodegen
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: mv build/*.5.precodegen* all2
+; RUN: ls build | count 0
+
+;; Check combinedindex
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=combinedindex
+; RUN: cmp all/a.out.1 build/a.out.1 && rm -f build/a.out.1
+; RUN: cmp all/a.out.2 build/a.out.2 && rm -f build/a.out.2
+; RUN: cp build/*.index.bc subset2
+; RUN: cp build/*.index.dot subset2
+; RUN: mv build/*.index.bc all2
+; RUN: mv build/*.index.dot all2
+; RUN: ls build | count 0
+
+;; Check resolution
+; RUN: llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=resolution
+;; all2 needs at least 1 copy of a.out, move it over now since its the last block
+; RUN: mv build/a.out.1 build/a.out.2 all2
+; RUN: mv build/*.resolution.txt all2
+; RUN: ls build | count 0
+
+;; If no files were left out from individual stages, the .all2 dir should be identical to .all
+; RUN: diff -r all all2
+
+;; Check multi-stage composability
+;; Similar to the above, but do it with a subset instead.
+;; .all -> .subset, .all2 -> .subset2
+; RUN: llvm-lto2 run 1.bc 2.bc -o subset/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=preopt,combinedindex,\opt
+; RUN: cmp all/a.out.1 subset/a.out.1 && rm -f subset/a.out.1
+; RUN: cmp all/a.out.2 subset/a.out.2 && rm -f subset/a.out.2
+; RUN: diff -r subset subset2
+
+;; Check error messages
+; RUN: not llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=prelink 2>&1 \
+; RUN: | FileCheck %s --check-prefix=ERR1
+; ERR1: invalid -select-save-temps argument: prelink
+
+; RUN: not llvm-lto2 run 1.bc 2.bc -o build/a.out \
+; RUN:    -import-constants-with-refs -r=1.bc,main,plx -r=1.bc,_Z6getObjv,l \
+; RUN:    -r=2.bc,_Z6getObjv,pl -r=2.bc,val,pl -r=2.bc,outer,pl \
+; RUN:    -select-save-temps=preopt -save-temps 2>&1 \
+; RUN: | FileCheck %s --check-prefix=ERR2
+; ERR2: -save-temps cannot be specified with -select-save-temps
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+%struct.S = type { i32, i32, i32* }
+
+define dso_local i32 @main() local_unnamed_addr {
+entry:
+  %call = tail call %struct.S* @_Z6getObjv()
+  %d = getelementptr inbounds %struct.S, %struct.S* %call, i64 0, i32 0
+  %0 = load i32, i32* %d, align 8
+  %v = getelementptr inbounds %struct.S, %struct.S* %call, i64 0, i32 1
+  %1 = load i32, i32* %v, align 4
+  %add = add nsw i32 %1, %0
+  ret i32 %add
+}
+
+declare dso_local %struct.S* @_Z6getObjv() local_unnamed_addr

llvm/tools/llvm-lto2/llvm-lto2.cpp

static cl::opt<std::string> OptPipeline("opt-pipeline",
                                         cl::value_desc("pipeline"));
 
 static cl::opt<std::string> AAPipeline("aa-pipeline",
                                        cl::desc("Alias Analysis Pipeline"),
                                        cl::value_desc("aapipeline"));
 
 static cl::opt<bool> SaveTemps("save-temps", cl::desc("Save temporary files"));
 
+static cl::list<std::string> SelectSaveTemps(
+    "select-save-temps",
+    cl::value_desc("One, or multiple of: "
+                   "resolution,preopt,promote,internalize,import,opt,precodegen"
+                   ",combinedindex"),
+    cl::desc("Save selected temporary files. Cannot be specified together with "
+             "-save-temps"),
+    cl::CommaSeparated);
+
+constexpr const char *SaveTempsValues[] = {
+    "resolution", "preopt", "promote",    "internalize",
+    "import",     "opt",    "precodegen", "combinedindex"};
+
 static cl::opt<bool>
     ThinLTODistributedIndexes("thinlto-distributed-indexes",
                               cl::desc("Write out individual index and "
                                        "import files for the "
                                        "distributed backend case"));
 
 static cl::opt<bool>
     ThinLTOEmitIndexes("thinlto-emit-indexes",
static int run(int argc, char **argv) {
   Conf.Options = codegen::InitTargetOptionsFromCodeGenFlags(Triple());
   Conf.MAttrs = codegen::getMAttrs();
   if (auto RM = codegen::getExplicitRelocModel())
     Conf.RelocModel = *RM;
   Conf.CodeModel = codegen::getExplicitCodeModel();
 
   Conf.DebugPassManager = DebugPassManager;
 
-  if (SaveTemps)
-    check(Conf.addSaveTemps(OutputFilename + "."),
+  if (SaveTemps && !SelectSaveTemps.empty()) {
+    llvm::errs() << "-save-temps cannot be specified with -select-save-temps\n";
+    return 1;
+  }
+  if (SaveTemps || !SelectSaveTemps.empty()) {
+    DenseSet<StringRef> SaveTempsArgs;
+    for (auto &S : SelectSaveTemps)
+      if (is_contained(SaveTempsValues, S))
+        SaveTempsArgs.insert(S);
+      else {
+        llvm::errs() << ("invalid -select-save-temps argument: " + S) << '\n';
+        return 1;
+      }
+    check(Conf.addSaveTemps(OutputFilename + ".", false, SaveTempsArgs),
           "Config::addSaveTemps failed");
+  }
 
   // Optimization remarks.
   Conf.RemarksFilename = RemarksFilename;
   Conf.RemarksPasses = RemarksPasses;
   Conf.RemarksWithHotness = RemarksWithHotness;
   Conf.RemarksHotnessThreshold = RemarksHotnessThreshold;
   Conf.RemarksFormat = RemarksFormat;