[Support] Make llvm-symbolizer runs for stack traces work on Apple platforms

[luismarques]

Summary

This is a follow-up patch to D142282 (but can be applied independently) to achieve fully symbolized stack traces with line information on Apple platforms (basically macOS, 64-bit for now). This seems to be necessary for the llvm-symbolizer executions to work properly for Mach-O, and thus for fully symbolized stack traces on Darwin. I suppose we could, alternatively, somehow make llvm-symbolizer smarter in how it finds the debug information.

[luismarques]

llvm/lib/Support/Signals.cpp

207

The __LP64__ check is not strictly necessary, but running this code is pointless until the findModulesAndOffsets implementation also runs on 32-bit Apple platforms.

[luismarques]

Improve LLVM directory search.

[dblaikie]

Huh, this seems like an unfortunate workaround/thing to have to implement.

When run on the command line/directly, is llvm-symbolizer unable to find debug info for the binaries? (I guess, alternatively: why is this workaround/hint needed? and is it only needed when doing the crash symbolizing/not symbolizing from the command line? (& if so, why only then?))

[luismarques]

In D142283#4091407, @dblaikie wrote:

Huh, this seems like an unfortunate workaround/thing to have to implement.

Indeed, which is also a further reason why I separated it from D142282. Still, it works, so it's provided as a "better than nothing" and as a way to test/use D142282.

When run on the command line/directly, is llvm-symbolizer unable to find debug info for the binaries? (I guess, alternatively: why is this workaround/hint needed? and is it only needed when doing the crash symbolizing/not symbolizing from the command line? (& if so, why only then?))

The behaviour on the command line is the same. I'll edit the patch summary to make that clearer. What I was trying to communicate in the patch summary was that (with the current llvm-symbolizer behaviour) this was necessary for fully symbolized stack traces, not that it was specific to that scenario.

[dblaikie]

When run on the command line/directly, is llvm-symbolizer unable to find debug info for the binaries? (I guess, alternatively: why is this workaround/hint needed? and is it only needed when doing the crash symbolizing/not symbolizing from the command line? (& if so, why only then?))

The behaviour on the command line is the same.

Ah, so how does a debugger find debug info without these sort of hints? Could that logic be reused in llvm-symbolizer? (are there other symbolizers (addr2line, etc) that work on MacOS? How do they find the debug info?)

[luismarques]

In D142283#4092092, @dblaikie wrote:

Ah, so how does a debugger find debug info without these sort of hints? Could that logic be reused in llvm-symbolizer? (are there other symbolizers (addr2line, etc) that work on MacOS? How do they find the debug info?)

I provided this more expedient patch exactly because I suspected that finding out the answers to those questions and implementing the missing logic was going to be non-trivial. For me, this was initially just a weekend patch to make stack traces work properly on macOS, as contributing to LLVM's Darwin support is a bit outside of my current professional scope. Still, if no one else steps up in the meantime I expect I'll eventually get around to doing that investigation and proper implementation, but it might take a while. I'll reference any follow-up patches or investigation results here.

[dblaikie]

In D142283#4114093, @luismarques wrote:

In D142283#4092092, @dblaikie wrote:

Ah, so how does a debugger find debug info without these sort of hints? Could that logic be reused in llvm-symbolizer? (are there other symbolizers (addr2line, etc) that work on MacOS? How do they find the debug info?)

I provided this more expedient patch exactly because I suspected that finding out the answers to those questions and implementing the missing logic was going to be non-trivial. For me, this was initially just a weekend patch to make stack traces work properly on macOS, as contributing to LLVM's Darwin support is a bit outside of my current professional scope. Still, if no one else steps up in the meantime I expect I'll eventually get around to doing that investigation and proper implementation, but it might take a while. I'll reference any follow-up patches or investigation results here.

Fiar enough, for sure. Though in this case we might be able to skip the research and ask the people who work on lldb - @aprantl @JDevlieghere do either of you happen to know how the debugger finds .o or dsyms for debugging? Could that be reasonably implemented in llvm-symbolizer so it just "does the right thing" and finds debug info itself without the need for extra hints? (hmm, I guess then there's another question, whether llvm-symbolizer can read the non-dsym debug info in .o files - that'd be ideal, but maybe that's a bunch more work, I'm not sure)

[aprantl]

llvm/lib/Support/Signals.cpp

227

The correct way of locating dSYMs on macOS is to run mdfind with the UUID of binary (e.g., https://github.com/llvm/llvm-project/blob/b40bfc1b9e389c154f6c60550dfc010ff8b3658f/lldb/examples/python/crashlog.py#L303). The UUID can be extracted with dwarfdump --uuid for example.

[dblaikie]

llvm/lib/Support/Signals.cpp

227

Does that cover the non-dsym case (debugging/symbolizing from raw .o files) too?

[aprantl]

llvm/lib/Support/Signals.cpp

227

No, this is strictly for finding .dSYM bundles. The official way of symbolicating on macOS is to use the CoreSymbolication framework which hides all these details and is exposed through tools like atos. (That's how the sanitizer runtime does it for example). The other option is to call into LLDB.framework using its public API, but that needs developer tools to be installed on the system.

[aprantl]

llvm/lib/Support/Signals.cpp

227

https://lldb.llvm.org/use/symbolication.html
https://lldb.llvm.org/use/symbols.html

So technically in addition to running mdfind you'd then also need to run any script in com.apple.DebugSymbols.DBGShellCommands ...

[dblaikie]

llvm/lib/Support/Signals.cpp

227

ah, should we just not make llvm-symbolizer work on macos, then? Should llvm use atos for symbolizing on macoS?

[aprantl]

llvm/lib/Support/Signals.cpp

227

I'm not familiar with llvm-symbolizer and how it's meant to be used. For symbolizing addresses in other Darwin process, I think just having llvm-symbolizer forward to atos would be the most future-proof and compatible implementation. But if you use it to e.g., symbolize macOS crashlogs on a Linux server then that would not be an option.

[dblaikie]

llvm/lib/Support/Signals.cpp

227

*nod* If the goal is cross-platform, then we'd need some more explicit contract about how to find debug info (both .o and dsym) than "ask atos".

Yeah, guess it depends what folks goals are

[mehdi_amini]

llvm/lib/Support/Signals.cpp

227

While llvm-symbolizer is likely intended to work in a cross-platform way, the signal handler here is in-process and not concerned by this right?

If so, we should be fine with just invoking atos instead of llvm-symbolizer here, shouldn't we?

llvm/lib/Support/Signals.cpp

FileRemover OutputRemover(OutputFile.c_str());
     for (int i = 0; i < Depth; i++) {
       if (Modules[i])
         Input << Modules[i] << " " << (void*)Offsets[i] << "\n";
     }
   }
 
   std::optional<StringRef> Redirects[] = {InputFile.str(), OutputFile.str(),
                                           StringRef("")};
-  StringRef Args[] = {"llvm-symbolizer", "--functions=linkage", "--inlining",
+  std::vector<StringRef> Args = {
+      "llvm-symbolizer", "--functions=linkage", "--inlining",
 #ifdef _WIN32
-                      // Pass --relative-address on Windows so that we don't
-                      // have to add ImageBase from PE file.
-                      // FIXME: Make this the default for llvm-symbolizer.
-                      "--relative-address",
 #endif
-                      "--demangle"};
+
+#if defined(__APPLE__) && defined(__LP64__)
+  // Find all of the .dSYM bundles in the LLVM build directory and pass each of
+  // them to llvm-symbolizer as a --dsym-hint argument.
+  std::vector<std::string> DSymHintArgs;
+  StringRef LLVMDir = Argv0;
+  while (!LLVMDir.empty()) {
+    LLVMDir = llvm::sys::path::parent_path(LLVMDir);
+    if (sys::fs::exists(LLVMDir + "/bin/llvm-config"))
+      break;
+  }
+  if (!LLVMDir.empty()) {
+    std::error_code ErrorCode;
+    for (llvm::sys::fs::recursive_directory_iterator I(LLVMDir, ErrorCode), E;
+         I != E && !ErrorCode; I.increment(ErrorCode)) {
+      if (!llvm::sys::fs::is_directory(I->path()))
+        continue;
+      if (StringRef(I->path()).ends_with(".dSYM")) {
+        I.no_push();
+        DSymHintArgs.push_back(I->path());
+        Args.push_back("--dsym-hint");
+        Args.push_back(DSymHintArgs.back());
+      }
+    }
+  }
+#endif
+
   int RunResult =
       sys::ExecuteAndWait(LLVMSymbolizerPath, Args, std::nullopt, Redirects);
   if (RunResult != 0)
     return false;
 
   // This report format is based on the sanitizer stack trace printer.  See
   // sanitizer_stacktrace_printer.cc in compiler-rt.
   auto OutputBuf = MemoryBuffer::getFile(OutputFile.c_str());