[Support] Implement findModulesAndOffsets on Apple 64-bit platforms

[luismarques]

Summary

To have file name and line information in stack traces (per stack frame) it's necessary to implement findModulesAndOffsets.

With this implementation, stack traces can show source location information on macOS (64-bit, for now). To make it work I used the cmake option LLVM_EXTERNALIZE_DEBUGINFO=True to generate the .dSYM directories and changed the calls to llvm-symbolizer to pass --dsym-hint arguments with the generated .dSYM directories (see follow-up patch D142283). Tested on macOS 13.1.

[mehdi_amini]

Is this something we can test for?

[luismarques]

In D142282#4081511, @mehdi_amini wrote:

Is this something we can test for?

It seems it is.

I'll submit a separate patch to add a unit test for symbolized stack traces. To enable that new test for Darwin/macOS will require both this patch and D142283 (or another alternative), so this patch can't be tested in isolation by that test.

[luismarques]

In D142282#4088945, @luismarques wrote:

I'll submit a separate patch to add a unit test for symbolized stack traces.

D143607.

[jloser]

I can't speak to the correctness of the implementation of findModulesAndOffsets here, but I'd love to see this make progress.

@luismarques @mehdi_amini @dblaikie are there any updates here?

[dblaikie]

I have some reservations about this, given the reservations about https://reviews.llvm.org/D142283 - but I guess this part of the work's relatively neutral, even if a more robust implementation of debug-info finding for MachO is implemented rather than the dsym hints direction.

Though some of the discussion there sounds like the more robust direction might be too much work/invasive/not very standalone, and maybe folks would be better off using the apple tool atos insetad?

[mehdi_amini]

Oh I forgot I reviewed this, and got frustrated by this just a couple of days ago and implemented https://reviews.llvm.org/D144839

Note that you're only supporting 64 bits platforms here while I added support for 32 bits as well.

[mehdi_amini]

LG to me!

In D142282#4142831, @dblaikie wrote:

I have some reservations about this, given the reservations about https://reviews.llvm.org/D142283 - but I guess this part of the work's relatively neutral, even if a more robust implementation of debug-info finding for MachO is implemented rather than the dsym hints direction.

This patch is just the part about finding the offset, it is orthogonal to D142283 in that with just the patch we have here we will correctly symbolize traces if we run separately dsymutil on the binary.

Having llvm-symbolizer be able to do the work of dsymutil automatically would be nice as well of course :)

Though some of the discussion there sounds like the more robust direction might be too much work/invasive/not very standalone, and maybe folks would be better off using the apple tool atos insetad?

Any direction would be based on top of this patch anyway I think?

llvm/lib/Support/Unix/Signals.inc

544

The variable style isn't LLVM right now it seems, can you fix it?

[luismarques]

Yes, this can land independently of the other work. Since D144839 was abandoned I'll update and land this.
I'll also work on the unit test and llvm-symbolizer improvement, I've just been busy with childcare and other work.

llvm/lib/Support/Unix/Signals.inc

544

I'll fix the style before landing it.

[dblaikie]

Though some of the discussion there sounds like the more robust direction might be too much work/invasive/not very standalone, and maybe folks would be better off using the apple tool atos insetad?

Any direction would be based on top of this patch anyway I think?

Ah, right, fair enough.

[stefanp]

@luismarques
Hi,
It looks like your patch broke our build bot on AIX:
https://lab.llvm.org/buildbot/#/builders/214/builds/6115

On AIX we end up without a definition for findModulesAndOffsets so very likely the combination of #if, #elif, #else, #endif is excluding AIX from all of the definitions and it did not do that before..

Could you please take a look?

Thanks!

[hubert.reinterpretcast]

llvm/lib/Support/Unix/Signals.inc

511–512

This comment was wrong: the fallback definition was also provided when backtraces are not requested.

[luismarques]

In D142282#4164562, @stefanp wrote:

On AIX we end up without a definition for findModulesAndOffsets so very likely the combination of #if, #elif, #else, #endif is excluding AIX from all of the definitions and it did not do that before..

Sorry, I only noticed your message now. I just reverted this instead of fixing forward. I'll take the time tomorrow to check how to best fix this, potentially not just messing with the preprocessor conditions. Thanks @hubert.reinterpretcast for the review tip about the old code comment being wrong.

[luismarques]

Update with the re-landed patch.

llvm/lib/Support/Unix/Signals.inc

 #include <sys/stat.h>
 #endif
 #if HAVE_DLFCN_H
 #include <dlfcn.h>
 #endif
 #if HAVE_MACH_MACH_H
 #include <mach/mach.h>
 #endif
+#ifdef __APPLE__
+#include <mach-o/dyld.h>
+#endif
 #if HAVE_LINK_H
 #include <link.h>
 #endif
 #ifdef HAVE__UNWIND_BACKTRACE
 // FIXME: We should be able to use <unwind.h> for any target that has an
 // _Unwind_Backtrace function, but on FreeBSD the configure test passes
 // despite the function not existing, and on Android, <unwind.h> conflicts
 // with <link.h>.

 /// handler can have a cookie passed to it to identify what instance of the
 /// handler it is.
 void llvm::sys::AddSignalHandler(sys::SignalHandlerCallback FnPtr,
                                  void *Cookie) { // Signal-safe.
   insertSignalHandler(FnPtr, Cookie);
   RegisterHandlers();
 }
 
-#if defined(HAVE_BACKTRACE) && ENABLE_BACKTRACES && HAVE_LINK_H &&             \
-    (defined(__linux__) || defined(__FreeBSD__) ||                             \
-     defined(__FreeBSD_kernel__) || defined(__NetBSD__))
+#if defined(HAVE_BACKTRACE) && ENABLE_BACKTRACES
+#if HAVE_LINK_H && (defined(__linux__) || defined(__FreeBSD__) ||              \
 struct DlIteratePhdrData {
   void **StackTrace;
   int depth;
   bool first;
   const char **modules;
   intptr_t *offsets;
   const char *main_exec_name;
 };
static bool findModulesAndOffsets(void **StackTrace, int Depth,
                                   const char **Modules, intptr_t *Offsets,
                                   const char *MainExecutableName,
                                   StringSaver &StrPool) {
   DlIteratePhdrData data = {StackTrace, Depth,   true,
                             Modules,    Offsets, MainExecutableName};
   dl_iterate_phdr(dl_iterate_phdr_cb, &data);
   return true;
 }
+#elif defined(__APPLE__) && defined(__LP64__)
+static bool findModulesAndOffsets(void **StackTrace, int Depth,
+                                  const char **Modules, intptr_t *Offsets,
+                                  const char *MainExecutableName,
+                                  StringSaver &StrPool) {
+  uint32_t NumImgs = _dyld_image_count();
+  for (uint32_t ImageIndex = 0; ImageIndex < NumImgs; ImageIndex++) {
+    const char *Name = _dyld_get_image_name(ImageIndex);
+    intptr_t Slide = _dyld_get_image_vmaddr_slide(ImageIndex);
+    auto *Header =
+        (const struct mach_header_64 *)_dyld_get_image_header(ImageIndex);
+    if (Header == NULL)
+      continue;
+    auto Cmd = (const struct load_command *)(&Header[1]);
+    for (uint32_t CmdNum = 0; CmdNum < Header->ncmds; ++CmdNum) {
+      uint32_t BaseCmd = Cmd->cmd & ~LC_REQ_DYLD;
+      if (BaseCmd == LC_SEGMENT_64) {
+        auto CmdSeg64 = (const struct segment_command_64 *)Cmd;
+        for (int j = 0; j < Depth; j++) {
+          if (Modules[j])
+            continue;
+          intptr_t Addr = (intptr_t)StackTrace[j];
+          if ((intptr_t)CmdSeg64->vmaddr + Slide <= Addr &&
+              Addr < intptr_t(CmdSeg64->vmaddr + CmdSeg64->vmsize + Slide)) {
+            Modules[j] = Name;
+            Offsets[j] = Addr - Slide;
+          }
+        }
+      }
+      Cmd = (const load_command *)(((const char *)Cmd) + (Cmd->cmdsize));
+    }
+  }
+  return true;
+}
 #else
-/// This platform does not have dl_iterate_phdr, so we do not yet know how to
-/// find all loaded DSOs.
+/// We don't yet know how to find all loaded DSOs on this platform.
 static bool findModulesAndOffsets(void **StackTrace, int Depth,
                                   const char **Modules, intptr_t *Offsets,
                                   const char *MainExecutableName,
                                   StringSaver &StrPool) {
   return false;
 }
-#endif // defined(HAVE_BACKTRACE) && ENABLE_BACKTRACES && ...
+#endif // per-platform findModulesAndOffsets implementations
+#endif // defined(HAVE_BACKTRACE) && ENABLE_BACKTRACES
 
 #if ENABLE_BACKTRACES && defined(HAVE__UNWIND_BACKTRACE)
 static int unwindBacktrace(void **StackTrace, int MaxEntries) {
   if (MaxEntries < 0)
     return 0;
 
   // Skip the first frame ('unwindBacktrace' itself).
   int Entries = -1;