[ELF] Support --remap-inputs and --remap-inputs-file

[melver]

Summary

Linking of binaries in large projects often results in processing of
thousands of linker inputs. In such projects, controlling all transitive
dependencies' inputs and libraries to be added to the final linker
command is (unfortunately) often too complex.

In theory the build system should provide a convenient way to simply
exclude (or replace) a given library, but the reality is that few build
systems do.

The simplest possible design, that will work across all build systems,
is therefore to pass an option to the linker to exclude or replace
inputs matching a given pattern. The possible use cases are:

1. Removing inputs added by dependencies (e.g. via pkg-config) where modification of the build scripts (or build systems such as Bazel) is infeasible.
2. Replacing inputs added by dependencies where modification of the build scripts is infeasible.
3. Replacing default inputs with custom implementations.

Introduce --remap-inputs and --remap-inputs-file, which accomplish just
that. See added man page documentation for more details.

This version is the unified feature first proposed in
https://reviews.llvm.org/D130229 and https://reviews.llvm.org/D148726.

Link: https://discourse.llvm.org/t/rfc-support-exclude-inputs
Co-developed-by: Fangrui Song <[e-mail]>

[melver]

• Don't use @ trickery to pass file, just provide --remap-inputs-file, which is more like other options that have a non-file and file based option.

[melver]

• Help text tweaks

[peter.smith]

I don't have a strong opinion on this one. I prefer the more general approachof this patch than the previous patch of filtering out files.

One thing we may want to be careful of is the use of wildcards. IIUC the only valid use is in the pattern and not the replacement, so we end up with many to one mappings. I can see these being OK for /dev/null and libraries but many to one for an object file would likely cause multiply defined symbol errors. Not entirely sure what we can do about that given we're looking at names.

[MaskRay]

In D148795#4283846, @peter.smith wrote:

I don't have a strong opinion on this one. I prefer the more general approachof this patch than the previous patch of filtering out files.

One thing we may want to be careful of is the use of wildcards. IIUC the only valid use is in the pattern and not the replacement, so we end up with many to one mappings. I can see these being OK for /dev/null and libraries but many to one for an object file would likely cause multiply defined symbol errors. Not entirely sure what we can do about that given we're looking at names.

Yes, and the performance is important. I have fixed some performance issue by checking whether wildcard is used 49279ca160183c1567e2637f88f3e92eb458c3e7.

I think patterns without a wildcard need to optimized when we add --thinlto-index= and can use an unordered map. Patterns with a wildcard can use a SmallVector<xxx, 0>.

I am inclined that we add just one option, not two. shell process substitution may be handy, e.g. --remap-inputs-file=<(printf 'NO-MATCH\t/dev/null')

See D148859 :)

lld/ELF/Config.h

struct Config {
   bool pie;
   bool printGcSections;
   bool printIcfSections;
   bool relax;
   bool relaxGP;
   bool relocatable;
   bool relrGlibc = false;
   bool relrPackDynRelocs = false;
+  llvm::SmallVector<std::pair<llvm::GlobPattern, llvm::StringRef>, 0>
+      remapInputs;
   llvm::DenseSet<llvm::StringRef> saveTempsArgs;
   llvm::SmallVector<std::pair<llvm::GlobPattern, uint32_t>, 0> shuffleSections;
   bool singleRoRx;
   bool shared;
   bool symbolic;
   bool isStatic = false;
   bool sysvHash = false;
   bool target1Rel;

lld/ELF/Driver.cpp

static void parseClangOption(StringRef opt, const Twine &msg) {
   error(msg + ": " + StringRef(err).trim());
 }
 
 // Checks the parameter of the bti-report and cet-report options.
 static bool isValidReportString(StringRef arg) {
   return arg == "none" || arg == "warning" || arg == "error";
 }
 
+// A remapping file contains a tab-separated pair: from, to.
+static void readRemapInputsFile(StringRef filename) {
+  std::optional<MemoryBufferRef> buffer = readFile(filename);
+  if (!buffer)
+    return;
+  SmallVector<StringRef, 0> fields;
+  size_t lineno = 0;
+  for (StringRef line : args::getLines(*buffer)) {
+    fields.clear();
+    line.split(fields, '\t');
+    ++lineno;
+    if (fields.size() != 2) {
+      error(filename + ":" + Twine(lineno) + ": parse error");
+      return;
+    }
+    // Try to parse pattern.
+    if (Expected<GlobPattern> pat = GlobPattern::create(fields[0]))
+      config->remapInputs.emplace_back(std::move(*pat), fields[1]);
+    else {
+      error(filename + ":" + Twine(lineno) + ": " + toString(pat.takeError()));
+      return;
+    }
+  }
+}
+
 // Initializes Config members by the command line options.
 static void readConfigs(opt::InputArgList &args) {
   errorHandler().verbose = args.hasArg(OPT_verbose);
   errorHandler().vsDiagnostics =
       args.hasArg(OPT_visual_studio_diagnostics_format, false);
 
   config->allowMultipleDefinition =
       args.hasFlag(OPT_allow_multiple_definition,
static void readConfigs(opt::InputArgList &args) {
 
   for (auto *arg : args.filtered(OPT_version_script))
     if (std::optional<std::string> path = searchScript(arg->getValue())) {
       if (std::optional<MemoryBufferRef> buffer = readFile(*path))
         readVersionScript(*buffer);
     } else {
       error(Twine("cannot find version script ") + arg->getValue());
     }
+
+  for (opt::Arg *arg : args.filtered(OPT_remap_inputs)) {
+    StringRef value(arg->getValue());
+    auto [pattern, toFile] = value.split(',');
+    if (toFile.empty())
+      error(arg->getSpelling() + ": invalid argument '" + value + "'");
+    else if (Expected<GlobPattern> pat = GlobPattern::create(pattern))
+      config->remapInputs.emplace_back(std::move(*pat), toFile);
+    else
+      error(arg->getSpelling() + ": " + toString(pat.takeError()));
+  }
+  for (opt::Arg *arg : args.filtered(OPT_remap_inputs_file))
+    readRemapInputsFile(arg->getValue());
 }
 
 // Some Config members do not directly correspond to any particular
 // command line options, but computed based on other Config values.
 // This function initialize such members. See Config.h for the details
 // of these values.
 static void setConfigs(opt::InputArgList &args) {
   ELFKind k = config->ekind;

lld/ELF/InputFiles.cpp

InputFile::InputFile(Kind k, MemoryBufferRef m)
   // Otherwise, a new file will get a new group ID.
   if (!isInGroup)
     ++nextGroupId;
 }
 
 std::optional<MemoryBufferRef> elf::readFile(StringRef path) {
   llvm::TimeTraceScope timeScope("Load input files", path);
 
+  for (const auto &[pat, toFile] : config->remapInputs)
+    if (pat.match(path)) {
+      log("mapping " + Twine(path) + " to " + toFile);
+      path = toFile;
+#ifdef _WIN32
+      if (path == "/dev/null")
+        path = "NUL";
+#endif
+    }
+
   // The --chroot option changes our virtual root directory.
   // This is useful when you are dealing with files created by --reproduce.
   if (!config->chroot.empty() && path.startswith("/"))
     path = saver().save(config->chroot + path);
 
   log(path);
   config->dependencyFiles.insert(llvm::CachedHashString(path));
 

lld/ELF/Options.td

 defm relax: BB<"relax",
   "Enable target-specific relaxations if supported (default)",
   "Disable target-specific relaxations">;
 
 defm relax_gp: BB<"relax-gp",
   "Enable global pointer relaxation",
   "Disable global pointer relaxation (default)">;
 
+defm remap_inputs: EEq<"remap-inputs", "Remap inputs matching given pattern to another input">;
+
+defm remap_inputs_file:
+  EEq<"remap-inputs-file", "Remap inputs matching given pattern to another input given rules in file">,
+  MetaVarName<"<file>">;
+
 defm reproduce:
   EEq<"reproduce",
      "Write tar file containing inputs and command to reproduce link">;
 
 defm rosegment: BB<"rosegment",
   "Put read-only non-executable sections in their own segment (default)",
   "Do not put read-only non-executable sections in their own segment">;
 

lld/docs/ld.lld.1

 .Fl -static ,
 and
 .Fl -whole-archive.
 .It Fl -pop-state
 Restore the states saved by
 .Fl -push-state.
 .It Fl --relax-gp
 Enable global pointer relaxation for RISC-V.
+.It Fl -remap-inputs Ns = Ns Ar value
+Remap inputs matching given pattern to another input.
+The provided
+.Ar value
+is of the format
+.Cm from-pattern,to
+where inputs matching the
+.Cm from-pattern
+are mapped to the
+.Cm to
+file.
+.It Fl -remap-inputs-file Ns = Ns Ar file
+Remap inputs based on rules in
+.Ar file .
+Each line in the mapping file is of the format
+.Cm from-pattern<tab>to .
 .It Fl -relocatable , Fl r
 Create relocatable object file.
 .It Fl -reproduce Ns = Ns Ar path
 Write a tar file to
 .Ar path,
 containing all the input files needed to reproduce the link, a text file called
 response.txt containing the command line options and a text file called
 version.txt containing the output of ld.lld --version.

lld/test/ELF/remap-inputs-file.s

+# REQUIRES: x86
+
+# RUN: rm -rf %t && split-file %s %t && cd %t
+# RUN: llvm-mc -filetype=obj -triple=x86_64 a.s -o a.o
+# RUN: llvm-as b.ll -o b.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64 c.s -o c.o && llvm-ar rc c.a c.o
+# RUN: llvm-mc -filetype=obj -triple=x86_64 d.s -o d.o && ld.lld -shared -soname=d d.o -o d.so
+# RUN: ld.lld --remap-inputs-file=1.map --reproduce=repro.tar aa.o bb.bc cc.a dd.so empty -o 1
+# RUN: tar tf repro.tar | FileCheck %s --check-prefix=REPRO
+
+# REPRO:      1.map
+# REPRO-NEXT: a.o
+# REPRO-NEXT: b.o
+# REPRO-NEXT: c.a
+# REPRO-NEXT: d.so
+
+# RUN: not ld.lld --remap-inputs-file=err.map --reproduce=repro.tar aa.o bb.bc -o /dev/null 2>&1 | FileCheck %s --check-prefix=ERR
+
+# ERR:      error: err.map:2: parse error
+# ERR-NEXT: error: cannot open bb.bc: {{.*}}
+
+#--- a.s
+.globl _start
+_start:
+  call b
+  call c
+  call d
+
+#--- b.ll
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @b() {
+  ret void
+}
+
+#--- c.s
+.globl c
+c:
+
+#--- d.s
+.globl d
+d:
+
+#--- 1.map
+aa.o  a.o
+bb.bc b.o
+cc.a  c.a
+*dd*  d.so
+empty /dev/null
+
+#--- err.map
+aa.o  a.o
+bb.bc
+cc.a

lld/test/ELF/remap-inputs.s

+// REQUIRES: x86
+
+// RUN: rm -rf %t && split-file %s %t && cd %t
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux foo.s -o foo.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux bar.s -o bar.o
+// RUN: llvm-mc -filetype=obj -triple=x86_64-unknown-linux baz.s -o baz.o
+// RUN: llvm-ar rcs bar.a bar.o
+
+// RUN: ld.lld --remap-inputs='NO-MATCH,/dev/null' --remap-inputs='*foo*,/dev/null' --whole-archive foo.o bar.a -o exe
+// RUN: llvm-readelf -s exe | FileCheck --check-prefix=OBJECT %s
+// OBJECT-NOT: foo
+// OBJECT:     bar
+// OBJECT-NOT: baz
+
+// RUN: ld.lld --remap-inputs='*bar*,/dev/null' --whole-archive foo.o bar.a -o exe
+// RUN: llvm-readelf -s exe | FileCheck --check-prefix=ARCHIVE %s
+// ARCHIVE:     foo
+// ARCHIVE-NOT: bar
+// ARCHIVE-NOT: baz
+
+// RUN: ld.lld --remap-inputs='*bar*,baz.o' --whole-archive foo.o bar.a -o exe
+// RUN: llvm-readelf -s exe | FileCheck --check-prefix=REMAP %s
+// REMAP:     foo
+// REMAP-NOT: bar
+// REMAP:     baz
+
+// RUN: not ld.lld --remap-inputs='abc' 2>&1 | FileCheck --check-prefix=ERR %s
+// ERR: error: --remap-inputs: invalid argument 'abc'
+
+#--- foo.s
+.globl foo
+foo:
+  nop
+
+#--- bar.s
+.globl bar
+bar:
+  nop
+
+#--- baz.s
+.globl baz
+baz:
+  nop