[WIP][lld] Implement crash reproducer

[haowei]

Summary

This is still work in progress to test my ideas. Not ready for review yet.

In this patch, I tried to use CrashRecoveryContext to re-run lldMain with '--reproduce' when the first run crashes. Flag 'reproduce-on-crash' is added to optionally turn on this feature. Flag '--debug-crash' to allow lld crash on purpose to test crash reproducer.

The reason I choose this approach is that it has a few advantages:

• It is quite simple.
• It works with every lld driver which already support '--reproduce'.

However, during testing, I saw a few issues:

• If the crash happens before the driver initialize the TarWriter for reproducer, this implementation will not catch it.
• lldMain is not designed to run twice internally. It will write unexpected errors to stderr. (e.g. I saw 'lld/ELF/InputSection.cpp:1402: void lld::elf::MergeInputSection::splitIntoPieces(): Assertion `pieces.empty()' failed.' error in tests)

Therefore, I am wondering if I should consider another approach: reimplement the reproducer each driver (start with ELF driver) to make it independent from the lld driver. In this case the reproducer can be directly invoked from CrashRecoveryContext in the lld/tools/lld/lld.cpp instead of being invoked from the rerun lld driver.

Discussion thread: https://lists.llvm.org/pipermail/llvm-dev/2021-April/149853.html

[MaskRay]

I don't understand the purpose. Didn't we agree that implementing this on a different process is a correct approach?

[dblaikie]

(it'd be good to link the description to the discussion thread on the issues, whatever way this goes)

[haowei]

In D102304#2762298, @MaskRay wrote:

I don't understand the purpose. Didn't we agree that implementing this on a different process is a correct approach?

Sorry for the confusion, I created this change so I can share the code with other people. It's not in a reviewable state and I mainly use it as a baseline for testing.

If I understood correctly, we agreed to implement a lld crash reproducer in a similar way like clang with "-fintegrated-cc1" as the first step, which relies on CrashRecoveryContext instead of relying on a second process. The crash reproducer can be further improved by implementing a reproducer that works on a different process if the first step is not good enough.

D102304 added a createCrashReproduceTar to the lld elf driver, which generates a reproducer tarball without re-invoke the lldMain and will be used by CrashRecoveryContext in case lld recovered from a crash. The principle is to record the file names used by lld and use them to generate the reproducer in case lld crashes. I tried the other approach which makes the reproducer to determine the file needs to be included but it turns out very easy to miss some files and hard to maintain. So I just record the filenames, which also seems to be easier to be extended to other drivers. Please let me know your opinions on this.

lld/ELF/Driver.cpp

if (auto E = timeTraceProfilerWrite(args.getLastArgValue(OPT_time_trace_file_eq).str(),
       handleAllErrors(std::move(E), [&](const StringError &SE) {
         error(SE.getMessage());
       });
       return;
     }
 
     timeTraceProfilerCleanup();
   }
+
+  if (args.getLastArg(OPT_debug_crash)) {
+    LLVM_BUILTIN_TRAP;
+  }
 }
 
 static std::string getRpath(opt::InputArgList &args) {
   std::vector<StringRef> v = args::getStrings(args, OPT_rpath);
   return llvm::join(v.begin(), v.end(), ":");
 }
 
 // Determines what we should do if there are remaining unresolved

lld/ELF/Options.td

 
 def print_map: F<"print-map">,
   HelpText<"Print a link map to the standard output">;
 
 defm reproduce:
   Eq<"reproduce",
      "Write tar file containing inputs and command to reproduce link">;
 
+def reproduce_on_crash: F<"reproduce-on-crash">,
+     HelpText<"Generate tar file containing inputs and command when program crashes">;
+
+def debug_crash: F<"debug-crash">,
+    HelpText<"Debug flag to crash lld on purpose">;
+
 defm rosegment: BB<"rosegment",
   "Put read-only non-executable sections in their own segment (default)",
   "Do not put read-only non-executable sections in their own segment">;
 
 defm rpath: Eq<"rpath", "Add a DT_RUNPATH to the output">;
 
 def relocatable: F<"relocatable">, HelpText<"Create relocatable object file">;
 

lld/tools/lld/lld.cpp

SafeReturn lld::safeLldMain(int argc, const char **argv,
   int r = 0;
   {
     // The crash recovery is here only to be able to recover from arbitrary
     // control flow when fatal() is called (through setjmp/longjmp or
     // __try/__except).
     llvm::CrashRecoveryContext crc;
     if (!crc.RunSafely([&]() {
           r = lldMain(argc, argv, stdoutOS, stderrOS, /*exitEarly=*/false);
-        }))
-      return {crc.RetCode, /*canRunAgain=*/false};
+        })) {
+      // lld crashed, save return code.
+      SafeReturn ret{crc.RetCode, /*canRunAgain=*/false};
+      // Generate reproducer if related flag is specified.
+      std::vector<const char *> args(argv, argv + argc);
+      bool hasReproducer = false;
+      bool hasReproduceOnCrash = false;
+      for (const char *arg : args) {
+        std::string arg_str(arg);
+        if (arg_str == "--reproduce-on-crash")
+          hasReproduceOnCrash = true;
+        if (StringRef(arg_str).startswith("--reproduce="))
+          hasReproducer = true;
+      }
+      if (hasReproduceOnCrash && !hasReproducer) {
+        // Creat temp file for reproducer.
+        SmallString<128> reproducerPath;
+        std::error_code EC =
+            llvm::sys::fs::createTemporaryFile("lld", "tar", reproducerPath);
+        if (EC) {
+          stderrOS << "failed to create temporary file for reproducer\n";
+          return ret;
+        }
+        stdoutOS << "reproducer writes to \"" << reproducerPath << "\"\n";
+        std::string reproduceArg = "--reproduce=" + std::string(reproducerPath);
+        args.push_back(reproduceArg.c_str());
+        llvm::CrashRecoveryContext crcReproducer;
+        crcReproducer.RunSafely([&]() {
+          lldMain(args.size(), args.data(), stdoutOS, stderrOS,
+                  /*exitEarly=*/false);
+        });
+      }
+      return ret;
+    }
   }
 
   // Cleanup memory and reset everything back in pristine condition. This path
   // is only taken when LLD is in test, or when it is used as a library.
   llvm::CrashRecoveryContext crc;
   if (!crc.RunSafely([&]() { errorHandler().reset(); })) {
     // The memory is corrupted beyond any possible recovery.
     return {r, /*canRunAgain=*/false};