[lld] [ELF] Support customizing behavior on target triple

[mgorny]

Summary

Support customizing LLD behavior on target triple; in particular,
having defaults depend on target system. The triple can be either
passed via --target= option, inferred from filename (${triple}-ld.lld)
or stored at build time as LLVM default.

[arichardson]

I'm happy with this approach since the triple can be set on the commandline.
If this gets merged I will update CHERI lld to use the triple instead of the new emulation that we added.

ELF/Driver.cpp

757

I think failing on an unknown triple would be good since it can avoid user error where the wrong defaults are chosen just because of a small typo.

[mgorny]

ELF/Driver.cpp

757

Somewhat done. Note that LLD apparently doesn't fail on errors immediately, so it behaves a bit weird.

[mgorny]

Fixed leaving triple unset on invalid --target.

[peter.smith]

I think we need to be very careful here. If I've understood this correctly then if this functionality is used for anything critical then a manually supplied target will be needed when doing cross-linking. For example my host LLD is x86_64, is just called ld.lld and will have an inferred x86_64 target triple. If someone customises the behaviour of LLD on the triple in a way that doesn't get caught by the test suite then we could get some strange breakages when doing cross-linking. I personally would prefer to see any option like this not try and auto-infer the target unless it can do it reliably and accurately from the input objects and I don't know if that is possible for all supported targets.

I think this might be better raised on lllvm-dev as I suspect that we need to give this wider visibility. I'm not totally opposed to this as I can see that --target has some advantages over adding extra emulations, or relying on the --target in the compiler driver but I think we need to be careful to define how this will interact with the emulation, and what the bounds of what we can customise with the option are?

[ruiu]

Let me make it clear again that I'm *not* okay with this approach. Please explicitly pass command line arguments from the compiler driver to the linker.

[krytarowski]

@ruiu we need some resolution here. We have stronger feelings from the community to customize the linker directly based on detected triple.

At least other !GNU platforms will benefit from it too (at least FreeBSD, OpenBSD and other BSD derivations like mentioned CheriBSD).
FreeBSD already adds such target customization based on an emulation name hack, we cannot repeat it on NetBSD in the same way.

In the worst case we can even produce a new flavor of lld target that resembles/duplicates original ELF and targets (Net)BSD.

It does allow us better customization with keeping compat with other toolchain components, especially from the the GNU world.

Ideally we would get something for LLVM 8.0.. as it could be still backportable at least for downstream purposes. RC2 is planned for Feburary 6th and we need unpatched lld functional out of the box.

Under what circumstances and what model will you let to support this approach?

[ruiu]

Kamil,

I understand how adding --target option to the linker would make target-specific customization easier, but that's as I said not acceptable by design in lld. The code we have for FreeBSD just sets one bit in the output file, and that's fundamentally different from adding a "master switch" to turn on all customization for some specific target. We've been making changes to lld to make it work for many targets, but the features we've added are on by default (such as the code for Android TLS layout or supporting .openbsd.randomdata section for OpenBSD) or enabled by per-feature basis by a command line flag (such as *-freebsd emulation name). There's no flag like "turning lld into FreeBSD mode" or something like that. It seems to be working pretty well, and I'm happy about that design. So, I'm sorry, but I don't think I can support this patch's approach at all.

Honestly what's special about NetBSD from the lld's perspective seems to be the code ownership of the compiler driver. I can help you solve technical issues, but I don't think I should accept a patch that's against one of the lld's fundamental design choices to solve non-technical problems.

[mgorny]

I think the main concern is that if we link ld -> ld.lld, gcc will no longer work out of the box. So we would have to repeat all the logic in gcc and in any other compiler that calls ld directly.

[ruiu]

But that's only adding missing -L and perhaps a few more, no? That doesn't seem too hard to do in gcc. I don't want to repeat what compiler drivers do in the linker. Also, even with this patch, you need to make a change to gcc to pass --target parameter to the linker, right?

[mgorny]

In D56650#1377494, @ruiu wrote:

But that's only adding missing -L and perhaps a few more, no? That doesn't seem too hard to do in gcc.

Probably not, provided nobody will block it. I'm not the one opposed to it.

I don't want to repeat what compiler drivers do in the linker. Also, even with this patch, you need to make a change to gcc to pass --target parameter to the linker, right?

Well, no, since gcc doesn't really support cross-compiling the way clang does, so it will just be happy with LLVM's default triple. We will probably want to patch GNU ld to accept/ignore --target though.

[ruiu]

If you still need to patch GNU ld, it doesn't seems like this patch makes things easier for you. (But even if this would make it easier for you, this patch's approach is not okay by design though.)

[krytarowski]

If we pass flags from clang, we sacrifice:

• lld usable as a standalone executable
• gcc capable to use lld as a functional linker
• clang driver not capable to call unpatched gnu ld/gold as we grow local flags to workaroud the customization (such as -z nognustack)
• compat with a number of programs programs that call linker directly and we customize the passed flags (gcc itself is one of them)

We just gain clang+lld tandem to work and we must retain GCC/LD interoperability (which as I understand isn't a priority/interesting for other OSes).

In theory we can patch:

• GCC to pass lld flags
• GNU LD (/gold) to accept new flags
• all the programs using linker to stop doing so (like the GCC project)

We need to tune in lld at least for start (out of my head, likely not a finished list):

• tune the DT_NEEDED behavior,
• rpath behavior,
• GNU stack generation
• specify default target/emulation search paths
• disable 3 segments (disable ro one)

We can address all the issues either in lld or in all the other projects around which certainly doesn't scale (and I expect in particular problems to push e.g. -z nognustack to GNU ld(1)).

OpenBSD indeed patches downstream their linkers for additional flags and accidentally it works better in the lld design. In NetBSD we struggle to get sane defaults rather than a set of additional flags. GNU stack is unwanted also on at least OpenBSD(/Fuchsia..) and there is no way to disable it in lld as of now (a proposal is to grow a custom flag). All OSes need to pass default paths for standalone linker usage (I've listed a dozen of examples from FreeBSD ports in other review).

The current feeling in the project is that if we cannot set the defaults we should fork the lld driver downstream and patch it there, which is a lose as the upstream version will keep to be unusable.

[krytarowski]

In D56650#1377546, @ruiu wrote:

If you still need to patch GNU ld, it doesn't seems like this patch makes things easier for you. (But even if this would make it easier for you, this patch's approach is not okay by design though.)

If we can get standalone lld functional out of the box and respecting target, we no longer need to patch GCC/LD/distribution/3rd-party-software.

If there is a strong feeling to retain lld ELF target as it is, we can grow a custom target like Darwin/Windows and specify customization there. The mentioned Darwin adds /usr/lib in the linker directly. We need analogous opportunity on our end.

[krytarowski]

chandlerc added a comment.

There was a long discussion between two NetBSD maintainers about this (both already in the reviewers list of this patch). I'm not sure if there is an existing thread that would be better to follow up on as opposed to starting a fresh thread.

The discussion is ongoing since 2017 and we have an impasse.

I think the question was: should the path search logic be handled in the clang driver or in LLD itself. FWIW, I prefer that NetBSD follow the same design as every other platform here with the (somewhat C and C++ specific) search logic provided by the clang driver.

Path logic is unfortunately just the tip of iceberg of customization we need, but it's true that its the first problem to encounter when attempting to use lld/NetBSD. Depending on the approach of solving it we can follow up with other changes.

Also to be clear, the current design of LLD eliminates usa-cases of using ld that we find important to keep supporting (and they are in real-world usage) - upstream doesn't want to handle them for (as far as I understand) the sake of simplicity -- the result is that LLD deliverables are highly biased with a specific OS target, which can (and actually do) produce dysfunctional binaries for others. So in other words this patch is more like adding a missing feature that can be turned into benefit of every ELF target, adding skeleton of standalone support and we can follow up with other OS-specific changes.

For now we need to use GCC with lld with these patches applied, but this is not a desirable solution (I call it a failure).

[krytarowski]

@ruiu ping?

LLVM 9 is branching soon and we would like to unearth from local patches.

From the internal discussions it was decided that we want to maintain our current behavior that differs to Linux.

[MaskRay]

krytarowski: The discussion is ongoing since 2017 and we have an impasse.

peter.smith: I think this might be better raised on lllvm-dev as I suspect that we need to give this wider visibility. I'm not totally opposed to this as I can see that --target has some advantages over adding extra emulations, or relying on the --target in the compiler driver but I think we need to be careful to define how this will interact with the emulation, and what the bounds of what we can customise with the option are?

Peter suggested raising your problem on the llvm-dev mailing list. Have you done that?

Personally I am opposed to this change. The compiler driver (gcc,clang) has a set of arch/OS dependent defaults. It seems weird/redundant to add another sets of defaults on the linker side.

A few replies to some bullet points you raised in a previous comment.

GCC to pass lld flags

Compiler drivers have more information (OS version, toolchain path, linker name, etc) to decide if some linker flags are needed. Some -f options can generate linker options as well. Your statistics on the other revision suggested there are very few users of ld. For them, they usually want to accomplish a very specific goal by dancing with a linker script.

Have you thought that the arch/os/environment dependent behaviors will actually make them harder to port?

rpath behavior

From my reading of https://blog.netbsd.org/tnf/entry/the_first_report_on_lld , I don't see why NetBSD must maintain the --disable-new-dtags local patch.

GNU stack generation

-z nognustack makes sense to me. I commented on D56554.

specify default target/emulation search paths

This job should be done in the compiler drivers.

disable 3 segments

Personally I don't think it is really necessary to change the default.. But you can add --no-rosegment if you are really bothered by the layout.

I think your requirement can be simply accomplished with a shell script wrapper called ld:

#!/bin/sh
# customization
exec ld.lld <options>

the result is that LLD deliverables are highly biased with a specific OS target, which can (and actually do) produce dysfunctional binaries for others

I don't agree. You might have this belief because many mainstream OSes are dominated by GNU toolchain for a decade (there is an interesting topic on llvm-dev/2019-June that may catch your eyes).
I'm glad this doesn't happen on lld land.

If you read my recent test changes in various components of llvm and lld, you'll find I avoid -unknown-linux-gnu as long as it makes sense :) I don't want my changes geared toward Linux "quirks" (as you might say).

[krytarowski]

Personally I am opposed to this change. The compiler driver (gcc,clang) has a set of arch/OS dependent defaults. It seems weird/redundant to add another sets of defaults on the linker side.

The NetBSD toolchain relies on internal knowledge in a linker that can be used as a standalone program. This is our design choice and this was decided internally to be kept. You might not like it from some point of view, but it is our real-world use-case, we wrote set of patches for it, we maintain a buildbot building and running lld tests... we try to upstream it.

A similar customization is actually done for (at least) Darwin (as the only OS using MachO) and FreeBSD (by an accident it's possible to differentiate FreeBSD on emul name). Please see that Darwin appends default paths (last time I checked) and for FreeBSD there are some fixups based on detecting FreeBSD. For OpenBSD there is an accident that it works thanks to embedding 'openbsd' in customization parts. For NetBSD we don't have such escape detection viable to include fixups and we shall detect triple target based on lld executable name.

If you don't like to see this code by defaut in lld, we can include it under #ifdef or overload the 'elf' namespace into 'netbsdelf'.

We need to unearth from our local patches and get upstream version of lld functional.

ELF/Config.h

 //===- Config.h -------------------------------------------------*- C++ -*-===//
 //
 //                             The LLVM Linker
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_ELF_CONFIG_H
 #define LLD_ELF_CONFIG_H
 
 #include "lld/Common/ErrorHandler.h"
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSet.h"
+#include "llvm/ADT/Triple.h"
 #include "llvm/BinaryFormat/ELF.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Endian.h"
 #include <vector>
 
 namespace lld {
 namespace elf {
struct Configuration {
   // few 32-bit ABIs are using RELA too.
   bool IsRela;
 
   // True if we are creating position-independent code.
   bool Pic;
 
   // 4 for ELF32, 8 for ELF64.
   int Wordsize;
+
+  // Target triple, inferred from program name or defaulted to LLVM
+  // default target.
+  llvm::Triple TargetTriple;
 };
 
 // The only instance of Configuration struct.
 extern Configuration *Config;
 
 static inline void errorOrWarn(const Twine &Msg) {
   if (!Config->NoinhibitExec)
     error(Msg);
   else
     warn(Msg);
 }
 } // namespace elf
 } // namespace lld
 
 #endif

ELF/Driver.h

 
 class LinkerDriver {
 public:
   void main(ArrayRef<const char *> Args);
   void addFile(StringRef Path, bool WithLOption);
   void addLibrary(StringRef Name);
 
 private:
+  void setTargetTriple(StringRef argv0, llvm::opt::InputArgList &Args);
   void readConfigs(llvm::opt::InputArgList &Args);
   void createFiles(llvm::opt::InputArgList &Args);
   void inferMachineType();
   template <class ELFT> void link(llvm::opt::InputArgList &Args);
 
   // True if we are in --whole-archive and --no-whole-archive.
   bool InWholeArchive = false;
 

ELF/Driver.cpp

 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compression.h"
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/TarWriter.h"
+#include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cstdlib>
 #include <utility>
 
 using namespace llvm;
 using namespace llvm::ELF;
 using namespace llvm::object;
void LinkerDriver::main(ArrayRef<const char *> ArgsArr) {
   errorHandler().ErrorLimit = args::getInteger(Args, OPT_error_limit, 20);
 
   // Handle -help
   if (Args.hasArg(OPT_help)) {
     printHelp();
     return;
   }
 
+  if (const char *Path = getReproduceOption(Args)) {
+    // Note that --reproduce is a debug option so you can ignore it
+    // if you are trying to understand the whole picture of the code.
+    Expected<std::unique_ptr<TarWriter>> ErrOrWriter =
+        TarWriter::create(Path, path::stem(Path));
+    if (ErrOrWriter) {
+      Tar = std::move(*ErrOrWriter);
+      Tar->append("response.txt", createResponseFile(Args));
+      Tar->append("version.txt", getLLDVersion() + "\n");
+    } else {
+      error("--reproduce: " + toString(ErrOrWriter.takeError()));
+    }
+  }
+
+  initLLVM();
+  setTargetTriple(ArgsArr[0], Args);
+  readConfigs(Args);
+  checkZOptions(Args);
+
   // Handle -v or -version.
   //
   // A note about "compatible with GNU linkers" message: this is a hack for
   // scripts generated by GNU Libtool 2.4.6 (released in February 2014 and
   // still the newest version in March 2017) or earlier to recognize LLD as
   // a GNU compatible linker. As long as an output for the -v option
   // contains "GNU" or "with BFD", they recognize us as GNU-compatible.
   //
   // This is somewhat ugly hack, but in reality, we had no choice other
   // than doing this. Considering the very long release cycle of Libtool,
   // it is not easy to improve it to recognize LLD as a GNU compatible
   // linker in a timely manner. Even if we can make it, there are still a
   // lot of "configure" scripts out there that are generated by old version
   // of Libtool. We cannot convince every software developer to migrate to
   // the latest version and re-generate scripts. So we have this hack.
-  if (Args.hasArg(OPT_v) || Args.hasArg(OPT_version))
+  if (Args.hasArg(OPT_v) || Args.hasArg(OPT_version)) {
     message(getLLDVersion() + " (compatible with GNU linkers)");
-
-  if (const char *Path = getReproduceOption(Args)) {
-    // Note that --reproduce is a debug option so you can ignore it
-    // if you are trying to understand the whole picture of the code.
-    Expected<std::unique_ptr<TarWriter>> ErrOrWriter =
-        TarWriter::create(Path, path::stem(Path));
-    if (ErrOrWriter) {
-      Tar = std::move(*ErrOrWriter);
-      Tar->append("response.txt", createResponseFile(Args));
-      Tar->append("version.txt", getLLDVersion() + "\n");
-    } else {
-      error("--reproduce: " + toString(ErrOrWriter.takeError()));
-    }
+    message("Target: " + Config->TargetTriple.str());
   }
 
-  readConfigs(Args);
-  checkZOptions(Args);
-
   // The behavior of -v or --version is a bit strange, but this is
   // needed for compatibility with GNU linkers.
   if (Args.hasArg(OPT_v) && !Args.hasArg(OPT_INPUT))
     return;
   if (Args.hasArg(OPT_version))
     return;
 
-  initLLVM();
   createFiles(Args);
   if (errorCount())
     return;
 
   inferMachineType();
   setConfigs(Args);
   checkOptions();
   if (errorCount())
static void parseClangOption(StringRef Opt, const Twine &Msg) {
 
   const char *Argv[] = {Config->ProgName.data(), Opt.data()};
   if (cl::ParseCommandLineOptions(2, Argv, "", &OS))
     return;
   OS.flush();
   error(Msg + ": " + StringRef(Err).trim());
 }
 
+void LinkerDriver::setTargetTriple(StringRef argv0, opt::InputArgList &Args) {
+  std::string TargetError;
+
+  // Firstly, see if user specified explicit --target
+  StringRef TargetOpt = Args.getLastArgValue(OPT_target);
+  if (!TargetOpt.empty()) {
+    if (llvm::TargetRegistry::lookupTarget(TargetOpt, TargetError)) {
+      Config->TargetTriple = llvm::Triple(TargetOpt);
+      return;
+    } else
+      error("Unsupported --target=" + TargetOpt + ": " + TargetError);
+  }
+
+  // Secondly, try to get it from program name prefix
+  std::string ProgName = llvm::sys::path::stem(argv0);
+  size_t LastComponent = ProgName.rfind('-');
+  if (LastComponent != std::string::npos) {
+    std::string Prefix = ProgName.substr(0, LastComponent);
+    if (llvm::TargetRegistry::lookupTarget(Prefix, TargetError)) {
+      Config->TargetTriple = llvm::Triple(Prefix);
+      return;
+    }
+  }
+
+  // Finally, use the default target triple
+  Config->TargetTriple = llvm::Triple(getDefaultTargetTriple());
+}
+
 // Initializes Config members by the command line options.
 void LinkerDriver::readConfigs(opt::InputArgList &Args) {
   errorHandler().Verbose = Args.hasArg(OPT_verbose);
   errorHandler().FatalWarnings =
       Args.hasFlag(OPT_fatal_warnings, OPT_no_fatal_warnings, false);
   ThreadsEnabled = Args.hasFlag(OPT_threads, OPT_no_threads, true);
 
   Config->AllowMultipleDefinition =
void LinkerDriver::inferMachineType() {
   }
   error("target emulation unknown: -m or at least one .o file required");
 }
 
 // Parse -z max-page-size=<value>. The default value is defined by
 // each target.
 static uint64_t getMaxPageSize(opt::InputArgList &Args) {
   uint64_t Val = args::getZOptionValue(Args, OPT_z, "max-page-size",
-                                       Target->DefaultMaxPageSize);
+                                       lld::elf::Target->DefaultMaxPageSize);
   if (!isPowerOf2_64(Val))
     error("max-page-size: value isn't a power of 2");
   return Val;
 }
 
 // Parses -image-base option.
 static Optional<uint64_t> getImageBase(opt::InputArgList &Args) {
   // Because we are using "Config->MaxPageSize" here, this function has to be

ELF/Options.td

 
 def strip_debug: F<"strip-debug">, HelpText<"Strip debugging information">;
 
 defm symbol_ordering_file:
   Eq<"symbol-ordering-file", "Layout sections to place symbols in the order specified by symbol ordering file">;
 
 defm sysroot: Eq<"sysroot", "Set the system root">;
 
+defm target: Eq<"target", "Apply configuration defaults for a given target">;
+
 def target1_rel: F<"target1-rel">, HelpText<"Interpret R_ARM_TARGET1 as R_ARM_REL32">;
 
 def target1_abs: F<"target1-abs">, HelpText<"Interpret R_ARM_TARGET1 as R_ARM_ABS32 (default)">;
 
 defm target2:
   Eq<"target2", "Interpret R_ARM_TARGET2 as <type>, where <type> is one of rel, abs, or got-rel">,
   MetaVarName<"<type>">;