[libunwind] On Darwin, add a callback-based lookup scheme for JIT'd unwind info.

[lhames]

Summary

We currently register JIT'd unwind-info with libunwind on Darwin by calling
libunwind's register_frame and deregister_frame functions (or
unw_add_dynamic_eh_frame_section and unw_remove_dynamic_eh_frame_section
where available), however this registration scheme has some drawbacks:

(1) It only supports registration of DWARF eh-frames and not Darwin's preferred

compact-unwind format. This problem becomes acute on Darwin/arm64 machines
(e.g. Apple Silicon Macs) where the compiler only emits compact-unwind by
default (falling back to DWARF only if the frame info cannot be encoded in
compact-unwind). This currently prevents the JIT from handling exceptions
(and from unwinding in general) in any object files not specifically
compiled to target the JIT on Darwin/arm64.

(2) It provides no way to describe the sub-architecture for a given frame,

which is necessary in order to properly handle arm64e pointer
authentication. This issue made it impossible to catch exceptions thrown
in JIT'd code on Apple Silicon Macs until recently, even when they had been
compiled to target the JIT. (See
https://github.com/llvm/llvm-project/issues/49036).

(3) The data structures used to hold the registered unwind info are hand-rolled

(presumably to avoid introducing a dependency on libcxx) and quite simple,
so not particularly efficient for lookup.

This commit adds support for a new callback-based lookup scheme for unwind
info that was inspired by the _dyld_find_unwind_info_sections SPI that
libunwind uses to find unwind-info in non-JIT'd frames. From
llvm-project/libunwind/src/AddressSpace.hpp:

struct dyld_unwind_sections {
  const struct mach_header*   mh;
  const void*                 dwarf_section;
  uintptr_t                   dwarf_section_length;
  const void*                 compact_unwind_section;
  uintptr_t                   compact_unwind_section_length;
};

extern bool _dyld_find_unwind_sections(void *, dyld_unwind_sections *);

During unwinding libunwind calls _dyld_find_unwind_sections to both find
unwind section addresses and identify the subarchitecture for frames (via the
MachO-header pointed to by the mh field).

This commit introduces two new libunwind SPI functions:

struct unw_dynamic_unwind_sections {
  unw_word_t dso_base;
  unw_word_t dwarf_section;
  size_t     dwarf_section_length;
  unw_word_t compact_unwind_section;
  size_t     compact_unwind_section_length;
};

typedef int (*unw_find_dynamic_unwind_sections)(
    unw_word_t addr, struct unw_dynamic_unwind_sections *info);

// Returns 0 if successfully registered, -1 to indicate too many
// registrations.
extern int __unw_add_find_dynamic_unwind_sections(
    unw_find_dynamic_unwind_sections find_dynamic_unwind_sections);

// Returns 0 if successfully deregistered, -1 to indicate no such
// registration.
extern int __unw_remove_find_dynamic_unwind_sections(
    unw_find_dynamic_unwind_sections find_dynamic_unwind_sections);

These can be used to register (and deregister) callbacks that have a similar
signature to _dyld_find_unwind_sections. During unwinding, if
_dyld_find_unwind_sections returns false (indicating that no frame info
was found by dyld) then registered callbacks are run in registration order until
either the unwind info is found or the end of the list is reached.

With this commit, and by implementing the find-unwind-info callback in the ORC
runtime in LLVM, we can address all three issues above: (1) compact unwind
ranges for JIT'd objects can be constructed and communicated to libunwind via
the new callback, (2) the subarchitecture for the given frame can be identified
by including a pointer to a JIT'd MachO header, and (3) the ORC runtime can use
STL or other complex data structures to enable efficient address-based lookup
for unwind info (a custom IntervalMap data type was recently introduced in
ab59185fbfb as a first step in this direction).

Explicitly out of scope for this commit is any change to registration on
non-Darwin platforms. While a callback-based lookup scheme may be useful on
other platforms it is not currently necessary for basic functionality the way
that it is on Darwin/arm64 (at least not to my knowledge).

[lhames]

I've committed the LLVM ORC and ORC runtime sides of this in 3507df9c20a4. If you check out that commit or later and then build libunwind with this patch you should be see the new registration pathway kick in on Darwin:

% ORC_RT_DEBUG=1 llvm-jitlink -orc-runtime=/path/to/built-llvm/lib/clang/16/lib/darwin/liborc_rt_osx.a  <object file(s)>
__unw_add/remove_find_dynamic_unwind_sections available. Using callback-based frame info lookup.
...

Sadly this will only work on x86-64 Macs for now as the ptrauth changes for arm64e haven't been upstreamed yet. I'm working on that, but don't have a timeline for it yet. This commit should flow into libunwind releases however, addressing https://github.com/llvm/llvm-project/issues/49036 properly (the existing released fix is unreliable) and opening a path to compact-unwind support.

[benlangmuir]

I'm not that familiar with development practices for libunwind, but is possible to write a test for this?

libunwind/src/libunwind_ext.h

71

Should document the return value, particularly since success/failure is not the same as other functions returning int in this file.

76

Would be nice to have doc comments for this; maybe just a lightly edited version of this paragraph from your description?

These can be used to register (and deregister) callbacks that have a similar
signature to _dyld_find_unwind_sections. During unwinding, if
_dyld_find_unwind_sections returns false (indicating that no frame info
was found by dyld) then registered callbacks are run in registration order until
either the unwind info is found or the end of the list is reached.

[lhames]

Expand comments for new SPI.

[lhames]

Rebase to include the original work in the updated diff.

[pete]

LGTM

[lhames]

In D142176#4069579, @benlangmuir wrote:

I'm not that familiar with development practices for libunwind, but is possible to write a test for this?

libunwind doesn't have tests for the JIT registration paths -- to exercise them it would need a way to produce executable code with valid headers and unwind info that was outside any range known to the system loader.

I think we're better off testing this in the ORC runtime -- we could have tests that look for the new registration paths and then verify that they're used.

libunwind/src/libunwind_ext.h

76

Yep -- good idea. I've included more detailed comments in the latest diff.

[lhames]

Here's a potential testcase, for anyone who wants to try this out. You'll need to build LLVM, compiler-rt, and libunwind:

C++ source for testcase:

int main(int argc, char *argv[]) {
  try {
    throw 42;
  } catch (int X) {
    return X;
  }
  return 0;
}

This can be compiled and run under the llvm-jitlink test tool on Darwin with:

% clang++ -c -o eh.o eh.cpp
% DYLD_LIBRARY_PATH=/path/to/llvm-build/lib ORC_RT_DEBUG=1 llvm-jitlink -orc-runtime=/path/to/llvm-build/lib/clang/16/lib/darwin/liborc_rt_osx.a eh.o
__unw_add/remove_find_dynamic_unwind_sections available. Using callback-based frame info lookup.
...

You can verify that the exception was caught and handled correctly by printing the return code:

% echo $?
42

[lhames]

Are there any further comments on this patch? If not I'll go ahead and land it based on Pete's approval.

[v.g.vassilev]

In D142176#4075822, @lhames wrote:

Here's a potential testcase, for anyone who wants to try this out. You'll need to build LLVM, compiler-rt, and libunwind:

C++ source for testcase:

int main(int argc, char *argv[]) {
  try {
    throw 42;
  } catch (int X) {
    return X;
  }
  return 0;
}

This can be compiled and run under the llvm-jitlink test tool on Darwin with:

% clang++ -c -o eh.o eh.cpp
% DYLD_LIBRARY_PATH=/path/to/llvm-build/lib ORC_RT_DEBUG=1 llvm-jitlink -orc-runtime=/path/to/llvm-build/lib/clang/16/lib/darwin/liborc_rt_osx.a eh.o
__unw_add/remove_find_dynamic_unwind_sections available. Using callback-based frame info lookup.
...

You can verify that the exception was caught and handled correctly by printing the return code:

% echo $?
42

I was wondering if it is feasible to add a regression test in clang-repl, possibly now with the right callback?

libunwind/src/AddressSpace.hpp

struct dyld_unwind_sections
     uintptr_t                   dwarf_section_length;
     const void*                 compact_unwind_section;
     uintptr_t                   compact_unwind_section_length;
   };
 
   // In 10.7.0 or later, libSystem.dylib implements this function.
   extern "C" bool _dyld_find_unwind_sections(void *, dyld_unwind_sections *);
 
+namespace libunwind {
+  bool findDynamicUnwindSections(void *, unw_dynamic_unwind_sections *);
+}
+
 #elif defined(_LIBUNWIND_SUPPORT_DWARF_UNWIND) && defined(_LIBUNWIND_IS_BAREMETAL)
 
 // When statically linked on bare-metal, the symbols for the EH table are looked
 // up without going through the dynamic loader.
 
 // The following linker script may be used to produce the necessary sections and symbols.
 // Unless the --eh-frame-hdr linker option is provided, the section is not generated
 // and does not take space in the output file.

  #if defined(_LIBUNWIND_SUPPORT_DWARF_UNWIND)
     info.dwarf_section                 = (uintptr_t)dyldInfo.dwarf_section;
     info.dwarf_section_length          = (size_t)dyldInfo.dwarf_section_length;
  #endif
     info.compact_unwind_section        = (uintptr_t)dyldInfo.compact_unwind_section;
     info.compact_unwind_section_length = (size_t)dyldInfo.compact_unwind_section_length;
     return true;
   }
+
+  unw_dynamic_unwind_sections dynamicUnwindSectionInfo;
+  if (findDynamicUnwindSections((void *)targetAddr,
+                                &dynamicUnwindSectionInfo)) {
+    info.dso_base = dynamicUnwindSectionInfo.dso_base;
+#if defined(_LIBUNWIND_SUPPORT_DWARF_UNWIND)
+    info.dwarf_section = (uintptr_t)dynamicUnwindSectionInfo.dwarf_section;
+    info.dwarf_section_length = dynamicUnwindSectionInfo.dwarf_section_length;
+#endif
+    info.compact_unwind_section =
+        (uintptr_t)dynamicUnwindSectionInfo.compact_unwind_section;
+    info.compact_unwind_section_length =
+        dynamicUnwindSectionInfo.compact_unwind_section_length;
+    return true;
+  }
+
 #elif defined(_LIBUNWIND_SUPPORT_DWARF_UNWIND) && defined(_LIBUNWIND_IS_BAREMETAL)
   info.dso_base = 0;
   // Bare metal is statically linked, so no need to ask the dynamic loader
   info.dwarf_section_length = (size_t)(&__eh_frame_end - &__eh_frame_start);
   info.dwarf_section =        (uintptr_t)(&__eh_frame_start);
   _LIBUNWIND_TRACE_UNWINDING("findUnwindSections: section %p length %p",
                              (void *)info.dwarf_section, (void *)info.dwarf_section_length);
 #if defined(_LIBUNWIND_SUPPORT_DWARF_INDEX)

libunwind/src/libunwind.cpp

void __unw_remove_dynamic_eh_frame_section(unw_word_t eh_frame_start) {
   // The eh_frame section start serves as the mh_group
   DwarfFDECache<LocalAddressSpace>::removeAllIn(
       (LocalAddressSpace::pint_t)eh_frame_start);
 }
 
 #endif // defined(_LIBUNWIND_SUPPORT_DWARF_UNWIND)
 #endif // !defined(__USING_SJLJ_EXCEPTIONS__)
 
+#ifdef __APPLE__
 
+namespace libunwind {
+
+static constexpr size_t MAX_DYNAMIC_UNWIND_SECTIONS_FINDERS = 8;
+
+static RWMutex findDynamicUnwindSectionsLock;
+static size_t numDynamicUnwindSectionsFinders = 0;
+static unw_find_dynamic_unwind_sections
+    dynamicUnwindSectionsFinders[MAX_DYNAMIC_UNWIND_SECTIONS_FINDERS] = {0};
+
+bool findDynamicUnwindSections(void *addr, unw_dynamic_unwind_sections *info) {
+  bool found = false;
+  findDynamicUnwindSectionsLock.lock_shared();
+  for (size_t i = 0; i != numDynamicUnwindSectionsFinders; ++i) {
+    if (dynamicUnwindSectionsFinders[i]((unw_word_t)addr, info)) {
+      found = true;
+      break;
+    }
+  }
+  findDynamicUnwindSectionsLock.unlock_shared();
+  return found;
+}
+
+} // namespace libunwind
+
+int __unw_add_find_dynamic_unwind_sections(
+    unw_find_dynamic_unwind_sections find_dynamic_unwind_sections) {
+  findDynamicUnwindSectionsLock.lock();
+
+  // Check that we have enough space...
+  if (numDynamicUnwindSectionsFinders == MAX_DYNAMIC_UNWIND_SECTIONS_FINDERS) {
+    findDynamicUnwindSectionsLock.unlock();
+    return UNW_ENOMEM;
+  }
+
+  // Check for value already present...
+  for (size_t i = 0; i != numDynamicUnwindSectionsFinders; ++i) {
+    if (dynamicUnwindSectionsFinders[i] == find_dynamic_unwind_sections) {
+      findDynamicUnwindSectionsLock.unlock();
+      return UNW_EINVAL;
+    }
+  }
+
+  // Success -- add callback entry.
+  dynamicUnwindSectionsFinders[numDynamicUnwindSectionsFinders++] =
+    find_dynamic_unwind_sections;
+  findDynamicUnwindSectionsLock.unlock();
+
+  return UNW_ESUCCESS;
+}
+
+int __unw_remove_find_dynamic_unwind_sections(
+    unw_find_dynamic_unwind_sections find_dynamic_unwind_sections) {
+  findDynamicUnwindSectionsLock.lock();
+
+  // Find index to remove.
+  size_t finderIdx = numDynamicUnwindSectionsFinders;
+  for (size_t i = 0; i != numDynamicUnwindSectionsFinders; ++i) {
+    if (dynamicUnwindSectionsFinders[i] == find_dynamic_unwind_sections) {
+      finderIdx = i;
+      break;
+    }
+  }
+
+  // If no such registration is present then error out.
+  if (finderIdx == numDynamicUnwindSectionsFinders) {
+    findDynamicUnwindSectionsLock.unlock();
+    return UNW_EINVAL;
+  }
+
+  // Remove entry.
+  for (size_t i = finderIdx; i != numDynamicUnwindSectionsFinders - 1; ++i)
+    dynamicUnwindSectionsFinders[i] = dynamicUnwindSectionsFinders[i + 1];
+  dynamicUnwindSectionsFinders[--numDynamicUnwindSectionsFinders] = nullptr;
+
+  findDynamicUnwindSectionsLock.unlock();
+  return UNW_ESUCCESS;
+}
+
+#endif // __APPLE__
 
 // Add logging hooks in Debug builds only
 #ifndef NDEBUG
 #include <stdlib.h>
 
 _LIBUNWIND_HIDDEN
 bool logAPIs() {
   // do manual lock to avoid use of _cxa_guard_acquire or initializers

libunwind/src/libunwind_ext.h

 
 // IPI
 extern void __unw_add_dynamic_fde(unw_word_t fde);
 extern void __unw_remove_dynamic_fde(unw_word_t fde);
 
 extern void __unw_add_dynamic_eh_frame_section(unw_word_t eh_frame_start);
 extern void __unw_remove_dynamic_eh_frame_section(unw_word_t eh_frame_start);
 
+#ifdef __APPLE__
+
+// Holds a description of the object-format-header (if any) and unwind info
+// sections for a given address:
+//
+// * dso_base should point to a header for the JIT'd object containing the
+//   given address. The header's type should match the format type that
+//   libunwind was compiled for (so a mach_header or mach_header_64 on Darwin).
+//   A value of zero indicates that no such header exists.
+//
+// * dwarf_section and dwarf_section_length hold the address range of a DWARF
+//   eh-frame section associated with the given address, if any. If the
+//   dwarf_section_length field is zero it indicates that no such section
+//   exists (and in this case dwarf_section should also be set to zero).
+//
+// * compact_unwind_section and compact_unwind_section_length hold the address
+//   range of a compact-unwind info section associated with the given address,
+//   if any. If the compact_unwind_section_length field is zero it indicates
+//   that no such section exists (and in this case compact_unwind_section
+//   should also be set to zero).
+//
+// See the unw_find_dynamic_unwind_sections type below for more details.
+struct unw_dynamic_unwind_sections {
+  unw_word_t dso_base;
+  unw_word_t dwarf_section;
+  size_t     dwarf_section_length;
+  unw_word_t compact_unwind_section;
+  size_t     compact_unwind_section_length;
+};
+
+// Typedef for unwind-info lookup callbacks. Functions of this type can be
+// registered and deregistered using __unw_add_find_dynamic_unwind_sections
+// and __unw_remove_find_dynamic_unwind_sections respectively.
+//
+// An unwind-info lookup callback should return 1 to indicate that it found
+// unwind-info for the given address, or 0 to indicate that it did not find
+// unwind-info for the given address. If found, the callback should populate
+// some or all of the fields of the info argument (which is guaranteed to be
+// non-null with all fields zero-initialized):
+typedef int (*unw_find_dynamic_unwind_sections)(
+    unw_word_t addr, struct unw_dynamic_unwind_sections *info);
+
+// Register a dynamic unwind-info lookup callback. If libunwind does not find
+// unwind info for a given frame in the executable program or normal dynamic
+// shared objects then it will call all registered dynamic lookup functions
+// in registration order until either one of them returns true, or the end
+// of the list is reached. This lookup will happen before libunwind searches
+// any eh-frames registered via __register_frame or
+// __unw_add_dynamic_eh_frame_section.
+//
+// Returns UNW_ESUCCESS for successful registrations. If the given callback
+// has already been registered then UNW_EINVAL will be returned. If all
+// available callback entries are in use then UNW_ENOMEM will be returned.
+extern int __unw_add_find_dynamic_unwind_sections(
+    unw_find_dynamic_unwind_sections find_dynamic_unwind_sections);
+
+// Deregister a dynacim unwind-info lookup callback.
+//
+// Returns UNW_ESUCCESS for successful deregistrations. If the given callback
+// has already been registered then UNW_EINVAL will be returned.
+extern int __unw_remove_find_dynamic_unwind_sections(
+    unw_find_dynamic_unwind_sections find_dynamic_unwind_sections);
+
+#endif
+
 #if defined(_LIBUNWIND_ARM_EHABI)
 extern const uint32_t* decode_eht_entry(const uint32_t*, size_t*, size_t*);
 extern _Unwind_Reason_Code _Unwind_VRS_Interpret(_Unwind_Context *context,
                                                  const uint32_t *data,
                                                  size_t offset, size_t len);
 #endif
 
 #ifdef __cplusplus
 }
 #endif
 
 #endif // __LIBUNWIND_EXT__