[WIP][lld] Implement crash reproducer for ELF

[haowei]

Summary

This is still work in progress.

Compared to D102304, this implementation does not invoke lldMain twice. Instead, it tries to record the file names when lld touches any file and repackage them when lld recovered from CrashRecoveryContext. In test it generate the same reproducer tarball when compared to D102304.

The change still need more work. It currently require environment variable to make lld to invoke safeLldMain (ideally, lld should use safeLldMain when "--reproduce-on-crash") and the path of the crash reproducer can be customized. Let me know if this is the direction we want to go.

[phosek]

I left some comments, but after reading the change, I'd suggest a slightly different structure: I think it'd be nice if we could avoid duplicating the logic between the --reproduce and --reproduce-on-crash case.

What I'm thinking is we could introduce a new function such as LinkerDriver::generateReproducer. This function would be responsible for generating the reproducer in both cases: we would always collect reproducer files and then call generateReproducer rather than writing the reproducer on demand as is the case now. That also has the advantage of not needing the std::unique_ptr<llvm::TarWriter> tar global since the TarWriter can be only created inside generateReproducer as needed.

I think we should also consider moving the llvm::CrashRecoveryContext setup from safeLldMain into each driver, for example elf::link, that's going to lead to some duplication, but the recovery context setup is fairly minimal and we could avoid having to repeat the flavor during normal execution and recovery and lldMain could remain completely agnostic to whether each driver supports crash recovery or not, that detail is fully encapsulated inside the *::link function.

lld/ELF/Driver.cpp

503–504

Do this check before instantiating the option parser which is a potentially expensive operation.

507

I'd do the early return on error. It might be also useful to signal the error back to the caller to avoid printing the output in the case of an error.

lld/tools/lld/lld.cpp

184

We might consider moving this logic into the ELF implementation so we can use ELFOptTable for option parsing.

197

I think it'd be also useful to allow explicitly setting the directory where to write the reproducer akin to Clang's -fcrash-diagnostics-dir. You could still use llvm::sys::fs::createTemporaryFile to create the file and then just move it to the final location at the end which is what Clang does.

You could either introduce another flag, for example --reproduce-dir or allow --reproduce-on-crash to take an optional value which would be the directory, either option is fine with me.

202

written

202

I think this output should be only printed after elf::createCrashReproduceTar so you don't print an output in the case that function fails for some reason.

lld/ELF/Driver.cpp

std::vector<std::pair<MemoryBufferRef, uint64_t>> static getArchiveMembers(
 
   return v;
 }
 
 // Opens a file and create a file object. Path has to be resolved already.
 void LinkerDriver::addFile(StringRef path, bool withLOption) {
   using namespace sys::fs;
 
+  if (!tar)
+    reproducerObjectFiles.push_back(std::string(path));
+
   Optional<MemoryBufferRef> buffer = readFile(path);
   if (!buffer.hasValue())
     return;
   MemoryBufferRef mbref = *buffer;
 
   if (config->formatBinary) {
     files.push_back(make<BinaryFile>(mbref));
     return;

 
 // Report an error for an unknown -z option.
 static void checkZOptions(opt::InputArgList &args) {
   for (auto *arg : args.filtered(OPT_z))
     if (!isKnownZFlag(arg->getValue()))
       error("unknown -z value: " + StringRef(arg->getValue()));
 }
 
+static void addFileToTar(StringRef path) {
+  auto mbOrErr = MemoryBuffer::getFile(path, false, false);
+  if (auto ec = mbOrErr.getError()) {
+    error("cannot open " + path + ": " + ec.message());
+    return;
+  }
+  std::unique_ptr<MemoryBuffer> &mb = *mbOrErr;
+  if (tar)
+    tar->append(relativeToRoot(path), mb->getBuffer());
+}
+
+static void addObjectFileToTar(StringRef path) {
+  Optional<MemoryBufferRef> buffer = readFile(path);
+  if (!buffer.hasValue())
+    return;
+  MemoryBufferRef mbref = *buffer;
+  if (identify_magic(mbref.getBuffer()) == file_magic::archive)
+    getArchiveMembers(mbref);
+}
+
+void elf::createCrashReproduceTar(const char *path,
+                                  ArrayRef<const char *> argsArr) {
+  ELFOptTable parser;
+  opt::InputArgList args = parser.parse(argsArr.slice(1));
+
+  if (!path)
+    return;
+  Expected<std::unique_ptr<TarWriter>> errOrWriter =
+      TarWriter::create(path, path::stem(path));
+  if (errOrWriter) {
+    tar = std::move(*errOrWriter);
+    tar->append("response.txt", createResponseFile(args));
+    tar->append("version.txt", getLLDVersion() + "\n");
+    StringRef ltoSampleProfile = args.getLastArgValue(OPT_lto_sample_profile);
+    if (!ltoSampleProfile.empty())
+      addFileToTar(ltoSampleProfile);
+  }
+  for (const std::string &file : reproducerFiles)
+    addFileToTar(file);
+
+  for (const std::string &file : reproducerObjectFiles)
+    addObjectFileToTar(file);
+}
+
 void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
   ELFOptTable parser;
   opt::InputArgList args = parser.parse(argsArr.slice(1));
 
   // Interpret this flag early because error() depends on them.
   errorHandler().errorLimit = args::getInteger(args, OPT_error_limit, 20);
   checkZOptions(args);
 
if (auto E = timeTraceProfilerWrite(args.getLastArgValue(OPT_time_trace_file_eq).str(),
       handleAllErrors(std::move(E), [&](const StringError &SE) {
         error(SE.getMessage());
       });
       return;
     }
 
     timeTraceProfilerCleanup();
   }
+
+  if (args.getLastArg(OPT_debug_crash))
+    LLVM_BUILTIN_TRAP;
 }
 
 static std::string getRpath(opt::InputArgList &args) {
   std::vector<StringRef> v = args::getStrings(args, OPT_rpath);
   return llvm::join(v.begin(), v.end(), ":");
 }
 
 // Determines what we should do if there are remaining unresolved

lld/ELF/InputFiles.h

 using llvm::object::Archive;
 
 class Symbol;
 
 // If -reproduce option is given, all input files are written
 // to this tar archive.
 extern std::unique_ptr<llvm::TarWriter> tar;
 
+// Store file names to the files required to construct a reproducer, added by
+// readFile().
+extern std::vector<std::string> reproducerFiles;
+
+// Store file names to the files required to construct a reproducer, added by
+// addFile().
+extern std::vector<std::string> reproducerObjectFiles;
+
 // Opens a given file.
 llvm::Optional<MemoryBufferRef> readFile(StringRef path);
 
 // Add symbols in File to the symbol table.
 void parseFile(InputFile *file);
 
 // The root class of input files.
 class InputFile {

lld/ELF/InputFiles.cpp

 std::vector<ArchiveFile *> elf::archiveFiles;
 std::vector<BinaryFile *> elf::binaryFiles;
 std::vector<BitcodeFile *> elf::bitcodeFiles;
 std::vector<LazyObjFile *> elf::lazyObjFiles;
 std::vector<InputFile *> elf::objectFiles;
 std::vector<SharedFile *> elf::sharedFiles;
 
 std::unique_ptr<TarWriter> elf::tar;
+std::vector<std::string> elf::reproducerFiles;
+std::vector<std::string> elf::reproducerObjectFiles;
 
 // Returns "<internal>", "foo.a(bar.o)" or "baz.o".
 std::string lld::toString(const InputFile *f) {
   if (!f)
     return "<internal>";
 
   if (f->toStringCache.empty()) {
     if (f->archiveName.empty())
Optional<MemoryBufferRef> elf::readFile(StringRef path) {
   }
 
   std::unique_ptr<MemoryBuffer> &mb = *mbOrErr;
   MemoryBufferRef mbref = mb->getMemBufferRef();
   make<std::unique_ptr<MemoryBuffer>>(std::move(mb)); // take MB ownership
 
   if (tar)
     tar->append(relativeToRoot(path), mbref.getBuffer());
+  else
+    reproducerFiles.push_back(std::string(path));
   return mbref;
 }
 
 // All input object files must be for the same architecture
 // (e.g. it does not make sense to link x86 object files with
 // MIPS object files.) This function checks for that error.
 static bool isCompatible(InputFile *file) {
   if (!file->isElf() && !isa<BitcodeFile>(file))

lld/ELF/Options.td

 
 def print_map: F<"print-map">,
   HelpText<"Print a link map to the standard output">;
 
 defm reproduce:
   Eq<"reproduce",
      "Write tar file containing inputs and command to reproduce link">;
 
+def reproduce_on_crash: F<"reproduce-on-crash">,
+     HelpText<"Generate tar file containing inputs and command when program crashes">;
+
+def debug_crash: F<"debug-crash">,
+    HelpText<"Debug flag to crash lld on purpose">;
+
 defm rosegment: BB<"rosegment",
   "Put read-only non-executable sections in their own segment (default)",
   "Do not put read-only non-executable sections in their own segment">;
 
 defm rpath: Eq<"rpath", "Add a DT_RUNPATH to the output">;
 
 def relocatable: F<"relocatable">, HelpText<"Create relocatable object file">;
 

lld/include/lld/Common/Driver.h

 namespace mingw {
 bool link(llvm::ArrayRef<const char *> args, bool canExitEarly,
           llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS);
 }
 
 namespace elf {
 bool link(llvm::ArrayRef<const char *> args, bool canExitEarly,
           llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS);
+
+void createCrashReproduceTar(const char *path,
+                             llvm::ArrayRef<const char *> argsArr);
 }
 
 namespace mach_o {
 bool link(llvm::ArrayRef<const char *> args, bool canExitEarly,
           llvm::raw_ostream &stdoutOS, llvm::raw_ostream &stderrOS);
 }
 
 namespace macho {

lld/include/lld/Common/Reproduce.h

 //===- Reproduce.h - Utilities for creating reproducers ---------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLD_COMMON_REPRODUCE_H
 #define LLD_COMMON_REPRODUCE_H
 
+#include "lld/Common/Args.h"
 #include "lld/Common/LLVM.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace opt { class Arg; }
 }
 

lld/tools/lld/lld.cpp

 #include "lld/Common/Memory.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Host.h"
 #include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PluginLoader.h"
 #include <cstdlib>
 
 using namespace lld;
 using namespace llvm;
SafeReturn lld::safeLldMain(int argc, const char **argv,
   int r = 0;
   {
     // The crash recovery is here only to be able to recover from arbitrary
     // control flow when fatal() is called (through setjmp/longjmp or
     // __try/__except).
     llvm::CrashRecoveryContext crc;
     if (!crc.RunSafely([&]() {
           r = lldMain(argc, argv, stdoutOS, stderrOS, /*exitEarly=*/false);
-        }))
-      return {crc.RetCode, /*canRunAgain=*/false};
+        })) {
+      // lld crashed, save return code.
+      SafeReturn ret{crc.RetCode, /*canRunAgain=*/false};
+      // Generate reproducer if related flag is specified.
+      std::vector<const char *> args(argv, argv + argc);
+      bool hasReproducer = false;
+      bool hasReproduceOnCrash = false;
+      for (const char *arg : args) {
+        std::string arg_str(arg);
+        if (arg_str == "--reproduce-on-crash")
+          hasReproduceOnCrash = true;
+        if (StringRef(arg_str).startswith("--reproduce="))
+          hasReproducer = true;
+      }
+      // For now only ELF is supported.
+      if (parseFlavor(args) == Gnu && !isPETarget(args) &&
+          hasReproduceOnCrash && !hasReproducer) {
+        // Creat temp file for reproducer.
+        SmallString<128> reproducerPath;
+        std::error_code EC =
+            llvm::sys::fs::createTemporaryFile("lld", "tar", reproducerPath);
+        if (EC) {
+          stderrOS << "failed to create temporary file for reproducer\n";
+          return ret;
+        }
+        stdoutOS << "reproducer writes to \"" << reproducerPath << "\"\n";
+        elf::createCrashReproduceTar(std::string(reproducerPath).data(), args);
+      }
+      return ret;
+    }
   }
 
   // Cleanup memory and reset everything back in pristine condition. This path
   // is only taken when LLD is in test, or when it is used as a library.
   llvm::CrashRecoveryContext crc;
   if (!crc.RunSafely([&]() { errorHandler().reset(); })) {
     // The memory is corrupted beyond any possible recovery.
     return {r, /*canRunAgain=*/false};