[LLD] Add NetBSD support as a new flavor of LLD (nb.lld)

[krytarowski]

Summary

The NetBSD target wraps the default Linux/ELF target with OS specific
customization. It is implemented as a light nb.lld program that
mutates arguments in argv[] and spawns ld.lld.

This flavor detects the native/current and target Triple based on
argv[0] parsing. This is prerequisite for cross-compilation, in
particular the NetBSD distribution is cross-built always.

The default configuration of the ELF target is tuned for Linux and
there is no way to costomize in-place for the NetBSD target in the
same way as FreeBSD/OpenBSD. FreeBSD whenever needed can check
emulation name ("*_fbsd") and OpenBSD calls its extensions
"PT_OPENBSD_*".

This distinct flavor is needed for NetBSD as:

• the linker MUST work in the standalone mode
• it must be useful with gcc/pcc/others out of the box
• clang NetBSD driver shall not hardcode LLD specific options
• the linker must have support for cross-building
• LLD shall be a drop-in replacement for (NetBSD-patched) GNU ld

nb.lld calls internally ld.lld and there is no code-duplication
between nb.lld and ld.lld. There is no functional or code
maintenance change for other Operating Systems, especially the ELF
ones.

Equivalent customization is already done for the Darwin mode. For
instance there are hardcoded default search paths such as "/usr/lib"
and "/Library/Frameworks" in DarwinLdDriver.cpp.

This change is a starting point for development of NetBSD support
in LLD. This is also another approach to achieve the same goal as
mutating the default LLD/ELF target based on Triple, but in a
less invasive way as everything is moved out of LLD to nb.lld now.

[krytarowski]

• upload diff with wider context

[MaskRay]

I still have the feeling that such configurations should be added to clangDriver/gcc specs or a shell script wrapper of lld.

How do you decide to handle "Handling of indirect shared library dependencies"? It does not seem to be favored by lld contributors.

lld/tools/nb.lld/nb.lld.cpp

94

You can't use DT_RUNPATH probably because the change Add support for DT_RUNPATH - it is just the same as we always have treated DT_RPATH. Approved by core. (2018-12-30) is still considered new. If you do this in clangDriver, you need a comment with a TODO about the migration process.

[krytarowski]

lld/tools/nb.lld/nb.lld.cpp

94

I am pretty sure that the only reason to pick it was rust (prebuilt snapshot) and its hard-coding of RUNPATH. We still want to disable it everywhere always.

[krytarowski]

In D69755#1731394, @MaskRay wrote:

I still have the feeling that such configurations should be added to clangDriver/gcc specs or a shell script wrapper of lld.

OK, as you allow now a shell wrapper of lld, this is a progress. nb.lld is a C++ wrapper on steroids doing the right job and part of the upstream repo (this is required).

We do not allow downstream LLVM/Clang/... patches in our basesystem and keeping this code upstream is required. Also from maintenance burden as LLVM APIs evolve quickly.

How do you decide to handle "Handling of indirect shared library dependencies"? It does not seem to be favored by lld contributors.

The modern Linux/GNU behavior is rejected in NetBSD and we keep the traditional behavior that is considered by our community as desired.

This is another topic and indeed probably our next major target to get to work within LLD.

[MaskRay]

In D69755#1731420, @krytarowski wrote:

In D69755#1731394, @MaskRay wrote:

I still have the feeling that such configurations should be added to clangDriver/gcc specs or a shell script wrapper of lld.

OK, as you allow now a shell wrapper of lld, this is a progress. nb.lld is a C++ wrapper on steroids doing the right job and part of the upstream repo (this is required).

We do not allow downstream LLVM/Clang/... patches in our basesystem and keeping this code upstream is required. Also from maintenance burden as LLVM APIs evolve quickly.

Yes, by upstreaming you will share maintenance burden with the upstream.. but do we really need it? LLVM and Clang APIs and unstable and evolve quickly, but that is unrelated to the linker. I have reviewed the options added in this patch. These GNU ld options (with the exception or --no-rosegment and --image-base) have been so ingrained that I don't think the lld upstream can suddenly stop supporting some options or make disruptive changes that break the behaviors.

How do you decide to handle "Handling of indirect shared library dependencies"? It does not seem to be favored by lld contributors.

The modern Linux/GNU behavior is rejected in NetBSD and we keep the traditional behavior that is considered by our community as desired.

Can you give links about the (1) rejection and (2) community needs? I am under the impression that some of the decisions are simply GNU hatred. I've mentioned in previous review comments that I don't like many GNU ELF things (STB_GNU_UNIQUE, STT_GNU_IFUNC, 4-byte Elf64_Nhdr fields, strip/objcopy --strip-all not stripping non-SHF_ALLOC, etc), some are bearable, because diverging the behavior may just cost more. However, for this particular DT_RUNPATH case, I don't really understand what it will cost if you follow the GNU behavior.

// force-disable RO segment on NetBSD due to ld.elf_so limitations

args.push_back("--no-rosegment");

I hope you can fix the limitation soon. I just implemented the --only-keep-debug feature for llvm-objcopy. My observation is that --no-rosegment can make the debug file produced by --only-keep-debug larger (larger readonly sections such as .dynsym, .dynstr, .rodata and .eh_frame will occupy much space that cannot be eliminated). Mixing sections of different permission in the same RX segment does not look nice, either.

This is another topic and indeed probably our next major target to get to work within LLD.

I'm very concerned about this...... This traditional behavior will bring much complexity to the linker, while its usefulness is still very unclear to me.

[krytarowski]

I don't want to diverge this patch on offtopic or general discussion.

Does it look good for landing? We need this support to be functional out of the box and as a part of LLVM. E.g. clang -fuse-ld= currently tries to call ld.lld, with this patch it will go through nb.lld as expected.

--no-rosegment is fixed with our WIP patches, but we NEED any support in LLD as a starting point so we can later just detach it (or add a check for version of NetBSD as we need Triple). Later we will the set the options on our own. I actually forgot to e.g. disable GNU stack and keep delaying it until the patch will land the sources.

We e.g. miss certain -switches in LLD, but hard to do anything in LLD without starting point. They will be proposed in future.

[krytarowski]

burden with the upstream

We are the part of upstream and have 4 developers in LLVM + few others contributing. We run the NetBSD buildbot that builds and runs test of LLD.

http://lab.llvm.org:8014/builders/netbsd-amd64

[krytarowski]

Better patch in https://reviews.llvm.org/D70048

clang/lib/Driver/ToolChain.cpp

std::string ToolChain::GetLinkerPath() const {
   } else if (UseLinker.empty() || UseLinker == "ld") {
     // If we're passed -fuse-ld= with no argument, or with the argument ld,
     // then use whatever the default system linker is.
     return GetProgramPath(getDefaultLinker());
   } else {
     llvm::SmallString<8> LinkerName;
     if (Triple.isOSDarwin())
       LinkerName.append("ld64.");
+    else if (Triple.isOSNetBSD())
+      LinkerName.append("nb.");
     else
       LinkerName.append("ld.");
     LinkerName.append(UseLinker);
 
     std::string LinkerPath(GetProgramPath(LinkerName.c_str()));
     if (llvm::sys::fs::can_execute(LinkerPath))
       return LinkerPath;
   }

lld/CMakeLists.txt

install(DIRECTORY include/
     PATTERN "*.h"
     PATTERN ".svn" EXCLUDE
     )
 endif()
 
 add_subdirectory(Common)
 add_subdirectory(lib)
 add_subdirectory(tools/lld)
+add_subdirectory(tools/nb.lld)
 
 if (LLVM_INCLUDE_TESTS)
   add_subdirectory(test)
   add_subdirectory(unittests)
 endif()
 
 add_subdirectory(docs)
 add_subdirectory(COFF)
 add_subdirectory(ELF)
 add_subdirectory(MinGW)
 add_subdirectory(wasm)

lld/tools/lld/lld.cpp

 //===- tools/lld/lld.cpp - Linker Driver Dispatcher -----------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains the main function of the lld executable. The main
 // function is a thin wrapper which dispatches to the platform specific
 // driver.
 //
-// lld is a single executable that contains four different linkers for ELF,
-// COFF, WebAssembly and Mach-O. The main function dispatches according to
-// argv[0] (i.e. command name). The most common name for each target is shown
+// lld is a single executable that contains five different linkers for ELF,
+// NetBSD, COFF, WebAssembly and Mach-O. The main function dispatches according
+// to argv[0] (i.e. command name). The most common name for each target is shown
 // below:
 //
 //  - ld.lld:    ELF (Unix)
+//  - nb.lld:    ELF (NetBSD)
 //  - ld64:      Mach-O (macOS)
 //  - lld-link:  COFF (Windows)
 //  - ld-wasm:   WebAssembly
 //
 // lld can be invoked as "lld" along with "-flavor" option. This is for
 // backward compatibility and not recommended.
 //
 //===----------------------------------------------------------------------===//
 
 #include "lld/Common/Driver.h"
 #include "lld/Common/Memory.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Host.h"
 #include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/WithColor.h"
+#include "llvm/Support/raw_ostream.h"
 #include <cstdlib>
 
 using namespace lld;
 using namespace llvm;
 using namespace llvm::sys;
 
 enum Flavor {
   Invalid,
   Gnu,     // -flavor gnu
+  NetBSD,  // -flavor netbsd
   WinLink, // -flavor link
   Darwin,  // -flavor darwin
   Wasm,    // -flavor wasm
 };
 
 LLVM_ATTRIBUTE_NORETURN static void die(const Twine &s) {
   errs() << s << "\n";
   exit(1);
 }
 
 static Flavor getFlavor(StringRef s) {
   return StringSwitch<Flavor>(s)
       .CasesLower("ld", "ld.lld", "gnu", Gnu)
+      .CasesLower("nb.lld", "netbsd", NetBSD)
       .CasesLower("wasm", "ld-wasm", Wasm)
       .CaseLower("link", WinLink)
       .CasesLower("ld64", "ld64.lld", "darwin", Darwin)
       .Default(Invalid);
 }
 
 static cl::TokenizerCallback getDefaultQuotingStyle() {
   if (Triple(sys::getProcessTriple()).getOS() == Triple::Win32)

 static Flavor parseProgname(StringRef progname) {
 #if __APPLE__
   // Use Darwin driver for "ld" on Darwin.
   if (progname == "ld")
     return Darwin;
 #endif
 
 #if LLVM_ON_UNIX
-  // Use GNU driver for "ld" on other Unix-like system.
-  if (progname == "ld")
+  // Use GNU or NetBSD driver for "ld" on other Unix-like system.
+  if (progname == "ld") {
+#if defined(__NetBSD__)
+    return NetBSD;
+#else
     return Gnu;
 #endif
+  }
+#endif
 
   // Progname may be something like "lld-gnu". Parse it.
   SmallVector<StringRef, 3> v;
   progname.split(v, "-");
   for (StringRef s : v)
     if (Flavor f = getFlavor(s))
       return f;
   return Invalid;
static Flavor parseFlavor(std::vector<const char *> &v) {
 
   // Deduct the flavor from argv[0].
   StringRef arg0 = path::filename(v[0]);
   if (arg0.endswith_lower(".exe"))
     arg0 = arg0.drop_back(4);
   return parseProgname(arg0);
 }
 
+// The NetBSD linker flavor will mutate arguments and call the GNU linker.
+static int exec_nb_lld(int argc, const char **argv) {
+  auto Program = sys::findProgramByName("nb.lld");
+  if (!Program) {
+    WithColor::error() << "unable to find `nb.lld' in PATH: "
+                       << Program.getError().message() << "\n";
+    return 1;
+  }
+
+  std::vector<StringRef> Argv;
+  Argv.push_back("nb.lld");
+
+  // Trim -flavor option.
+  if (argc > 1 && argv[0] == StringRef("-flavor")) {
+    if (argc <= 2)
+      die("missing arg value for '-flavor'");
+    argc -= 2;
+    argv += 2;
+  }
+
+  for (int i = 0; i < argc; ++i)
+    Argv.push_back(argv[i]);
+
+  std::string ErrMsg;
+  int Result = sys::ExecuteAndWait(*Program, Argv, None, {}, 0, 0, &ErrMsg);
+  if (Result < 0) {
+    WithColor::error() << ErrMsg << "\n";
+    return 1;
+  }
+  return Result;
+}
+
 // If this function returns true, lld calls _exit() so that it quickly
 // exits without invoking destructors of globally allocated objects.
 //
 // We don't want to do that if we are running tests though, because
 // doing that breaks leak sanitizer. So, lit sets this environment variable,
 // and we use it to detect whether we are running tests or not.
 static bool canExitEarly() { return StringRef(getenv("LLD_IN_TEST")) != "1"; }
 
-/// Universal linker main(). This linker emulates the gnu, darwin, or
+/// Universal linker main(). This linker emulates the gnu, netbsd, darwin, or
 /// windows linker based on the argv[0] or -flavor option.
 int main(int argc, const char **argv) {
   InitLLVM x(argc, argv);
 
   std::vector<const char *> args(argv, argv + argc);
   switch (parseFlavor(args)) {
   case Gnu:
     if (isPETarget(args))
       return !mingw::link(args);
     return !elf::link(args, canExitEarly());
+  case NetBSD:
+    return exec_nb_lld(argc - 1, argv + 1);
   case WinLink:
     return !coff::link(args, canExitEarly());
   case Darwin:
     return !mach_o::link(args, canExitEarly());
   case Wasm:
     return !wasm::link(args, canExitEarly());
   default:
     die("lld is a generic driver.\n"
-        "Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld"
+        "Invoke ld.lld (Unix), nb.lld (NetBSD), ld64.lld (macOS), lld-link "
+        "(Windows), wasm-ld"
         " (WebAssembly) instead");
   }
 }

lld/tools/nb.lld/CMakeLists.txt

+set(LLVM_TARGET_DEFINITIONS Options.td)
+tablegen(LLVM Options.inc -gen-opt-parser-defs)
+add_public_tablegen_target(SLLDOptionsTableGen)
+
+set(LLVM_LINK_COMPONENTS
+  Support
+  Option
+  Target
+  ${LLVM_TARGETS_TO_BUILD}
+  )
+
+add_lld_tool(nb.lld
+  nb.lld.cpp
+
+  DEPENDS
+  SLLDOptionsTableGen
+  )
+
+install(TARGETS nb.lld
+  RUNTIME DESTINATION bin
+  )

lld/tools/nb.lld/Options.td

+include "llvm/Option/OptParser.td"
+
+class F<string name>: Flag<["--", "-"], name>;
+
+def version: F<"version">, HelpText<"Display the version number and exit">;
+def v: Flag<["-"], "v">, HelpText<"Display the version number">;
+def: Flag<["-"], "V">, Alias<version>, HelpText<"Alias for --version">;

lld/tools/nb.lld/nb.lld.cpp

+//===- nb.lld.cpp - NetBSD LLD standalone linker --------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// The NetBSD flavor of LLD.
+//
+// This code wraps the default ELF/UNIX lld variation with NetBSD specific
+// customization.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/Triple.h"
+#include "llvm/Option/ArgList.h"
+#include "llvm/Option/Option.h"
+#include "llvm/Support/Host.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
+#include "llvm/Support/TargetRegistry.h"
+#include "llvm/Support/TargetSelect.h"
+#include "llvm/Support/WithColor.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+using namespace llvm::sys;
+
+#define LLD_PROGNAME "ld.lld"
+
+namespace {
+enum ID {
+  OPT_INVALID = 0, // This is not an option ID.
+#define OPTION(PREFIX, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,  \
+               HELPTEXT, METAVAR, VALUES)                                      \
+  OPT_##ID,
+#include "Options.inc"
+#undef OPTION
+};
+
+#define PREFIX(NAME, VALUE) const char *const NAME[] = VALUE;
+#include "Options.inc"
+#undef PREFIX
+
+const opt::OptTable::Info InfoTable[] = {
+#define OPTION(PREFIX, NAME, ID, KIND, GROUP, ALIAS, ALIASARGS, FLAGS, PARAM,  \
+               HELPTEXT, METAVAR, VALUES)                                      \
+  {                                                                            \
+      PREFIX,      NAME,      HELPTEXT,                                        \
+      METAVAR,     OPT_##ID,  opt::Option::KIND##Class,                        \
+      PARAM,       FLAGS,     OPT_##GROUP,                                     \
+      OPT_##ALIAS, ALIASARGS, VALUES},
+#include "Options.inc"
+#undef OPTION
+};
+
+class SLLDOptTable : public opt::OptTable {
+public:
+  SLLDOptTable() : OptTable(InfoTable) {}
+};
+} // namespace
+
+static Triple targetTriple;
+
+static void setTargetTriple(StringRef argv0, opt::InputArgList &args) {
+  std::string targetError;
+
+  // Firstly, try to get it from program name prefix
+  std::string ProgName = llvm::sys::path::stem(argv0);
+  size_t lastComponent = ProgName.rfind('-');
+  if (lastComponent != std::string::npos) {
+    std::string prefix = ProgName.substr(0, lastComponent);
+    if (llvm::TargetRegistry::lookupTarget(prefix, targetError)) {
+      targetTriple = llvm::Triple(prefix);
+      return;
+    }
+  }
+
+  // Secondly, use the default target triple
+  targetTriple = llvm::Triple(getDefaultTargetTriple());
+}
+
+static void appendSearchPath(std::vector<StringRef> &args, const char *path) {
+  args.push_back("--library-path");
+  args.push_back(path);
+}
+
+static void appendTargetCustomization(std::vector<StringRef> &args) {
+  // force-disable RO segment on NetBSD due to ld.elf_so limitations
+  args.push_back("--no-rosegment");
+
+  // disable superfluous RUNPATH on NetBSD
+  args.push_back("--disable-new-dtags");
+
+  // set default image base address
+  switch (targetTriple.getArch()) {
+  case llvm::Triple::aarch64:
+  case llvm::Triple::aarch64_be:
+    args.push_back("--image-base=0x200100000");
+    break;
+  default:
+    break;
+  }
+
+  // NetBSD driver relies on the linker knowing the default search paths.
+  // Please keep this in sync with clang/lib/Driver/ToolChains/NetBSD.cpp
+  // (NetBSD::NetBSD constructor)
+  switch (targetTriple.getArch()) {
+  case llvm::Triple::x86:
+    appendSearchPath(args, "=/usr/lib/i386");
+    break;
+  case llvm::Triple::arm:
+  case llvm::Triple::armeb:
+  case llvm::Triple::thumb:
+  case llvm::Triple::thumbeb:
+    switch (targetTriple.getEnvironment()) {
+    case llvm::Triple::EABI:
+    case llvm::Triple::GNUEABI:
+      appendSearchPath(args, "=/usr/lib/eabi");
+      break;
+    case llvm::Triple::EABIHF:
+    case llvm::Triple::GNUEABIHF:
+      appendSearchPath(args, "=/usr/lib/eabihf");
+      break;
+    default:
+      appendSearchPath(args, "=/usr/lib/oabi");
+      break;
+    }
+    break;
+#if 0 // TODO
+  case llvm::Triple::mips64:
+  case llvm::Triple::mips64el:
+    if (tools::mips::hasMipsAbiArg(Args, "o32")) {
+      appendSearchPath(args, "=/usr/lib/o32");
+    } else if (tools::mips::hasMipsAbiArg(Args, "64")) {
+      appendSearchPath(args, "=/usr/lib/64");
+    }
+    break;
+#endif
+  case llvm::Triple::ppc:
+    appendSearchPath(args, "=/usr/lib/powerpc");
+    break;
+  case llvm::Triple::sparc:
+    appendSearchPath(args, "=/usr/lib/sparc");
+    break;
+  default:
+    break;
+  }
+
+  appendSearchPath(args, "=/usr/lib");
+}
+
+int main(int argc, const char **argv) {
+  bool printTarget = false;
+
+  InitLLVM X(argc, argv);
+
+  auto Program = sys::findProgramByName(LLD_PROGNAME);
+  if (!Program) {
+    WithColor::error() << "unable to find `" << LLD_PROGNAME
+                       << "' in PATH: " << Program.getError().message() << "\n";
+    return 1;
+  }
+
+  ArrayRef<const char *> argsArr = makeArrayRef(argv, argc);
+
+  SLLDOptTable parser;
+  unsigned MAI;
+  unsigned MAC;
+  opt::InputArgList args = parser.ParseArgs(argsArr.slice(1), MAI, MAC);
+
+  // Append to -v or -version the target information from slld.
+  if (args.hasArg(OPT_v) || args.hasArg(OPT_version))
+    printTarget = true;
+
+  InitializeAllTargets();
+  setTargetTriple(argsArr[0], args);
+
+  if (!targetTriple.isOSNetBSD()) {
+    WithColor::error() << "invalid NetBSD target: " << targetTriple.str()
+                       << "\n";
+  }
+
+  argc--;
+  argv++;
+
+  std::vector<StringRef> Argv;
+  Argv.push_back(*Program);
+
+  // Prepend original arguments with the target options.
+  appendTargetCustomization(Argv);
+
+  // Append original options.
+  // Trim -flavor option.
+  if (argc > 1 && argv[0] == StringRef("-flavor")) {
+    if (argc <= 2)
+      WithColor::error() << "missing arg value for '-flavor'\n";
+    argc -= 2;
+    argv += 2;
+  }
+
+  for (int i = 0; i < argc; ++i)
+    Argv.push_back(argv[i]);
+
+  std::string ErrMsg;
+  int Result = sys::ExecuteAndWait(*Program, Argv, None, {}, 0, 0, &ErrMsg);
+  if (Result < 0) {
+    WithColor::error() << ErrMsg << "\n";
+    return 1;
+  }
+
+  if (printTarget)
+    WithColor::note() << "Target: " << targetTriple.str() << "\n";
+
+  return Result;
+}