[ELF] Add --remap-inputs= and --remap-inputs-file=

[MaskRay]

Summary

--remap-inputs-file= can be specified multiple times, each naming a
remap file that contains from-glob=to-file lines or #-led comments.
('=' is used a separator a la -fdebug-prefix-map=)
--remap-inputs-file= can be used to:

• replace an input file. E.g. "*/libz.so=exp/libz.so" can replace a resolved -lz without updating the input file list or (if used) a response file. When debugging an application where a bug is isolated to one single input file, this option gives a convenient way to test fixes.
• remove an input file with /dev/null (changed to NUL on Windows), e.g. "a.o=/dev/null". A build system may add unneeded dependencies. This option gives a convenient way to test the result removing some inputs.

--remap-inputs=a.o=aa.o can be specified to provide one pattern without using
an extra file.
(bash/zsh process substitution is handy for specifying a pattern without using
a remap file, e.g. --remap-inputs-file=<(printf 'a.o=aa.o'), but it may be
unavailable in some systems. An extra file can be inconvenient for a build
system.)

Exact patterns are tested before wildcard patterns. In case of a tie, the first
patterns wins. This is an implementation detail that users should not rely on.

Co-authored-by: Marco Elver <[e-mail]>
Link: https://discourse.llvm.org/t/rfc-support-exclude-inputs/70070

[melver]

We need --remap-inputs=. Relying on <(...) being available just does not always work. Especially when the build system does not use bash/zsh to dispatch the link command.

Please add the test for the non-file usage from https://reviews.llvm.org/D148795.

[melver]

lld/ELF/Options.td

621

Why is this here?

[peter.smith]

Code that is here looks good to me. I agree with melver that we shouldn't assume the presence of bash/zsh, particularly for Windows users. I can't comment on how necessary the command line version is, to me writing a file is an extra step that I may have to do if I just want a quick experiment, but that is more of an inconvenience. However a build system may be so difficult that adding an extra file without it being checked in somewhere might exist.

[MaskRay]

Add --remap-inputs=.
Add release note.
Use = as a separator.

[peter.smith]

I'm happy with these changes, I prefer the <from-glob>=<to-file> as it is more obvious than the tab separated option. I think this addresses melver's requirement for a command line option too.

[melver]

Looks good, thanks!

[melver]

Commit message: "--remap-inputs=a.o,aa.o can be specified to provide one pattern without using an extra file." <-- ',' is no longer the separator, update this line in case it's not already.

[melver]

LGTM, thanks!

[MaskRay]

Thank you! I want to wait a bit before landing to give others (especially the binutils folks) a chance to comment, but I'll not wait beyond Thursday :)

lld/ELF/Config.h

struct Config {
   // Signal to the dynamic loader to enable heap MTE.
   bool androidMemtagHeap;
   // Signal to the dynamic loader that this binary expects stack MTE. Generally,
   // this means to map the primary and thread stacks as PROT_MTE. Note: This is
   // not supported on Android 11 & 12.
   bool androidMemtagStack;
 
   unsigned threadCount;
+
+  // If an input file equals a key, remap it to the value.
+  llvm::DenseMap<llvm::StringRef, llvm::StringRef> remapInputs;
+  // If an input file matches a wildcard pattern, remap it to the value.
+  llvm::SmallVector<std::pair<llvm::GlobPattern, llvm::StringRef>, 0>
+      remapInputsWildcards;
 };
 struct ConfigWrapper {
   Config c;
   Config *operator->() { return &c; }
 };
 
 LLVM_LIBRARY_VISIBILITY extern ConfigWrapper config;
 

lld/ELF/Driver.cpp

static void parseClangOption(StringRef opt, const Twine &msg) {
   error(msg + ": " + StringRef(err).trim());
 }
 
 // Checks the parameter of the bti-report and cet-report options.
 static bool isValidReportString(StringRef arg) {
   return arg == "none" || arg == "warning" || arg == "error";
 }
 
+// Process a remap pattern 'from-glob=to-file'.
+static bool remapInputs(StringRef line, const Twine &location) {
+  SmallVector<StringRef, 0> fields;
+  line.split(fields, '=');
+  if (fields.size() != 2 || fields[1].empty()) {
+    error(location + ": parse error, not 'from-glob=to-file'");
+    return true;
+  }
+  if (!hasWildcard(fields[0]))
+    config->remapInputs[fields[0]] = fields[1];
+  else if (Expected<GlobPattern> pat = GlobPattern::create(fields[0]))
+    config->remapInputsWildcards.emplace_back(std::move(*pat), fields[1]);
+  else {
+    error(location + ": " + toString(pat.takeError()));
+    return true;
+  }
+  return false;
+}
+
 // Initializes Config members by the command line options.
 static void readConfigs(opt::InputArgList &args) {
   errorHandler().verbose = args.hasArg(OPT_verbose);
   errorHandler().vsDiagnostics =
       args.hasArg(OPT_visual_studio_diagnostics_format, false);
 
   config->allowMultipleDefinition =
       args.hasFlag(OPT_allow_multiple_definition,
static void readConfigs(opt::InputArgList &args) {
 
   if (opt::Arg *arg = args.getLastArg(OPT_eb, OPT_el)) {
     if (arg->getOption().matches(OPT_eb))
       config->optEB = true;
     else
       config->optEL = true;
   }
 
+  for (opt::Arg *arg : args.filtered(OPT_remap_inputs)) {
+    StringRef value(arg->getValue());
+    remapInputs(value, arg->getSpelling());
+  }
+  for (opt::Arg *arg : args.filtered(OPT_remap_inputs_file)) {
+    StringRef filename(arg->getValue());
+    std::optional<MemoryBufferRef> buffer = readFile(filename);
+    if (!buffer)
+      continue;
+    // Parse 'from-glob=to-file' lines, ignoring #-led comments.
+    for (auto [lineno, line] : llvm::enumerate(args::getLines(*buffer)))
+      if (remapInputs(line, filename + ":" + Twine(lineno + 1)))
+        break;
+  }
+
   for (opt::Arg *arg : args.filtered(OPT_shuffle_sections)) {
     constexpr StringRef errPrefix = "--shuffle-sections=: ";
     std::pair<StringRef, StringRef> kv = StringRef(arg->getValue()).split('=');
     if (kv.first.empty() || kv.second.empty()) {
       error(errPrefix + "expected <section_glob>=<seed>, but got '" +
             arg->getValue() + "'");
       continue;
     }

lld/ELF/DriverUtils.cpp

for (auto *arg : args) {
     case OPT_lto_sample_profile:
       os << arg->getSpelling() << quote(rewritePath(arg->getValue())) << "\n";
       break;
     case OPT_call_graph_ordering_file:
     case OPT_dynamic_list:
     case OPT_export_dynamic_symbol_list:
     case OPT_just_symbols:
     case OPT_library_path:
+    case OPT_remap_inputs_file:
     case OPT_retain_symbols_file:
     case OPT_rpath:
     case OPT_script:
     case OPT_symbol_ordering_file:
     case OPT_sysroot:
     case OPT_version_script:
       os << arg->getSpelling() << " " << quote(rewritePath(arg->getValue()))
          << "\n";

lld/ELF/InputFiles.cpp

 std::optional<MemoryBufferRef> elf::readFile(StringRef path) {
   llvm::TimeTraceScope timeScope("Load input files", path);
 
   // The --chroot option changes our virtual root directory.
   // This is useful when you are dealing with files created by --reproduce.
   if (!config->chroot.empty() && path.startswith("/"))
     path = saver().save(config->chroot + path);
 
+  bool remapped = false;
+  auto it = config->remapInputs.find(path);
+  if (it != config->remapInputs.end()) {
+    path = it->second;
+    remapped = true;
+  } else {
+    for (const auto &[pat, toFile] : config->remapInputsWildcards) {
+      if (pat.match(path)) {
+        path = toFile;
+        remapped = true;
+        break;
+      }
+    }
+  }
+  if (remapped) {
+    // Use /dev/null to indicate an input file that should be ignored. Change
+    // the path to NUL on Windows.
+#ifdef _WIN32
+    if (path == "/dev/null")
+      path = "NUL";
+#endif
+  }
+
   log(path);
   config->dependencyFiles.insert(llvm::CachedHashString(path));
 
   auto mbOrErr = MemoryBuffer::getFile(path, /*IsText=*/false,
                                        /*RequiresNullTerminator=*/false);
   if (auto ec = mbOrErr.getError()) {
     error("cannot open " + path + ": " + ec.message());
     return std::nullopt;

lld/ELF/Options.td

 defm relax: BB<"relax",
   "Enable target-specific relaxations if supported (default)",
   "Disable target-specific relaxations">;
 
 defm relax_gp: BB<"relax-gp",
   "Enable global pointer relaxation",
   "Disable global pointer relaxation (default)">;
 
+defm remap_inputs: EEq<"remap-inputs",
+  "Remap input files matching <from-glob> to <to-file>">,
+  MetaVarName<"<from-glob>=<to-file>">;
+
+def remap_inputs_file: JJ<"remap-inputs-file=">,
+  HelpText<"Each line contains 'from-glob=to-file'. An input file matching <from-glob> is remapped to <to-file>">,
+  MetaVarName<"<file>">;
+
 defm reproduce:
   EEq<"reproduce",
      "Write tar file containing inputs and command to reproduce link">;
 
 defm rosegment: BB<"rosegment",
   "Put read-only non-executable sections in their own segment (default)",
   "Do not put read-only non-executable sections in their own segment">;
 
defm shuffle_sections: EEq<"shuffle-sections",
   "Shuffle matched sections using the given seed before mapping them to the output sections. "
   "If -1, reverse the section order. If 0, use a random seed">,
   MetaVarName<"<section-glob>=<seed>">;
 def thinlto_cache_dir: JJ<"thinlto-cache-dir=">,
   HelpText<"Path to ThinLTO cached object file directory">;
 defm thinlto_cache_policy: EEq<"thinlto-cache-policy", "Pruning policy for the ThinLTO cache">;
 def thinlto_emit_imports_files: FF<"thinlto-emit-imports-files">;
 def thinlto_emit_index_files: FF<"thinlto-emit-index-files">;
 def thinlto_index_only: FF<"thinlto-index-only">;
 def thinlto_index_only_eq: JJ<"thinlto-index-only=">;
 def thinlto_jobs_eq: JJ<"thinlto-jobs=">,
   HelpText<"Number of ThinLTO jobs. Default to --threads=">;
 def thinlto_object_suffix_replace_eq: JJ<"thinlto-object-suffix-replace=">;
 def thinlto_prefix_replace_eq: JJ<"thinlto-prefix-replace=">;
 def thinlto_single_module_eq: JJ<"thinlto-single-module=">,
   HelpText<"Specify a single module to compile in ThinLTO mode, for debugging only">;
 

lld/docs/ReleaseNotes.rst

 from the `LLVM releases web site <https://llvm.org/releases/>`_.
 
 Non-comprehensive list of changes in this release
 =================================================
 
 ELF Improvements
 ----------------
 
+* ``--remap-inputs=`` and ``--remap-inputs-file=`` are added to remap input files.
+  (`D148859 <https://reviews.llvm.org/D148859>`_)
+
 Breaking changes
 ----------------
 
 COFF Improvements
 -----------------
 
 MinGW Improvements
 ------------------

lld/docs/ld.lld.1

 .Fl -whole-archive.
 .It Fl -pop-state
 Restore the states saved by
 .Fl -push-state.
 .It Fl --relax-gp
 Enable global pointer relaxation for RISC-V.
 .It Fl -relocatable , Fl r
 Create relocatable object file.
+.It Fl -remap-inputs Ns = Ns Ar from-glob=to-file
+Input files matching
+.Cm from-glob
+are mapped to
+.Cm to-file.
+Use
+.Cm /dev/null
+to ignore an input file.
+.It Fl -remap-inputs-file Ns = Ns Ar file
+Remap input files based on patterns in
+.Ar file .
+Each line in the remap file is of the format
+.Cm from-glob=to-file
+or a comment starting with
+.Cm # .
 .It Fl -reproduce Ns = Ns Ar path
 Write a tar file to
 .Ar path,
 containing all the input files needed to reproduce the link, a text file called
 response.txt containing the command line options and a text file called
 version.txt containing the output of ld.lld --version.
 The archive when
 unpacked can be used to re-run the linker with the same options and input files.

lld/test/ELF/remap-inputs.test

+# REQUIRES: x86
+## --remap-inputs and --remap-inputs-file= remap input files.
+
+# RUN: rm -rf %t && split-file %s %t && cd %t
+# RUN: llvm-mc -filetype=obj -triple=x86_64 a.s -o a.o
+# RUN: llvm-as b.ll -o b.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64 c.s -o c.o && llvm-ar rc c.a c.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64 d.s -o d.o && ld.lld -shared -soname=d d.o -o d.so
+# RUN: ld.lld --remap-inputs-file=1.map --remap-inputs-file=2.map --remap-inputs='d*.so=d.so' --reproduce=repro.tar aa.o bb.bc cc.a dd.so empty -o out
+# RUN: tar tf repro.tar | FileCheck %s --check-prefix=REPRO
+
+# REPRO:      1.map
+# REPRO-NEXT: 2.map
+# REPRO-NEXT: a.o
+# REPRO-NEXT: b.o
+# REPRO-NEXT: c.a
+# REPRO-NEXT: d.so
+
+## --remap-inputs can also be specified multiple times.
+# RUN: ld.lld --remap-inputs 'aa.o=a.o' --remap-inputs='d[d].so=d.so' aa.o b.o c.a d.so -o /dev/null
+
+## A multiple-to-one pattern may easily cause issues. Users should be careful.
+# RUN: not ld.lld --remap-inputs-file=3.map aa.o bb.bc -o /dev/null 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=DUPLICATE --implicit-check-not=error:
+# DUPLICATE: error: duplicate symbol: _start
+
+# RUN: not ld.lld --remap-inputs-file=err1.map aa.o bb.bc -o /dev/null 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERR1 --implicit-check-not=error:
+# ERR1:      error: err1.map:2: parse error, not 'from-glob=to-file'
+# ERR1-NEXT: error: cannot open bb.bc: {{.*}}
+
+# RUN: not ld.lld --remap-inputs-file=err2.map aa.o -o /dev/null 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERR2 --implicit-check-not=error:
+# ERR2:      error: err2.map:1: invalid glob pattern: aa.[o
+# ERR2-NEXT: error: cannot open aa.o: {{.*}}
+
+# RUN: not ld.lld --remap-inputs=aa.o aa.o -o /dev/null 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERR3 --implicit-check-not=error:
+# RUN: not ld.lld --remap-inputs=aa.o= aa.o -o /dev/null 2>&1 | \
+# RUN:   FileCheck %s --check-prefix=ERR3 --implicit-check-not=error:
+# ERR3:      error: --remap-inputs: parse error, not 'from-glob=to-file'
+# ERR3-NEXT: error: cannot open aa.o: {{.*}}
+
+#--- a.s
+.globl _start
+_start:
+  call b
+  call c
+  call d
+
+#--- b.ll
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @b() {
+  ret void
+}
+
+#--- c.s
+.globl c
+c:
+
+#--- d.s
+.globl d
+d:
+
+#--- 1.map
+aa.o=a.o
+b?.[b]c=b.o
+
+#--- 2.map
+cc.a=c.a
+## Use /dev/null to indicate an input file which should be ignored.
+empty=/dev/null
+
+#--- 3.map
+*=a.o
+
+#--- err1.map
+aa.o=a.o
+bb.bc
+cc.a
+
+#--- err2.map
+aa.[o=a.o