[lld/mac] Don't fold UNWIND_X86_64_MODE_STACK_IND unwind entries

[thakis]

Summary

libunwind uses unwind info to find the function address belonging
to the current instruction pointer. libunwind/src/CompactUnwinder.hpp's
step functions read functionStart for UNWIND_X86_64_MODE_STACK_IND
(and for nothing else), so these encodings need a dedicated entry
per function, so that the runtime can get the stacksize off the
subq instrunction in the function's prologue.

This matches ld64.

(CompactUnwinder.hpp from https://opensource.apple.com/source/libunwind/
also reads functionStart in a few more cases if SUPPORT_OLD_BINARIES is set,
but it defaults to 0, and ld64 seems to not worry about these additional
cases.)

Related upstream bug: https://crbug.com/1220175

[thakis]

(While working on this, I realized that xcrun unwinddump stack.o prints incorrect stack sizes for 0x03 encodings because it (understandably) doesn't process relocations in the object file, and then it interprets the wrong bytes as subq instruction. On linked images it works. Maybe it should just say "frame size unknown" instead of printing something wrong.)

[thakis]

whitespace adjustments

[int3]

Thanks!

lld/test/MachO/compact-unwind-stack-ind.s

10–11

[uabelho]

lld/MachO/UnwindInfoSection.cpp

332–333

gcc warns on the comparisons in the two new static_asserts:

lld/MachO/UnwindInfoSection.cpp:345:44: warning: comparison between 'enum<unnamed>' and 'enum<unnamed>' [-Wenum-compare]
353 |   static_assert(UNWIND_X86_64_MODE_MASK == UNWIND_X86_MODE_MASK, "");
    |                                            ^~~~~~~~~~~~~~~~~~~~
lld/MachO/UnwindInfoSection.cpp:346:49: warning: comparison between 'enum<unnamed>' and 'enum<unnamed>' [-Wenum-compare]
354 |   static_assert(UNWIND_X86_64_MODE_STACK_IND == UNWIND_X86_MODE_STACK_IND, "");
    |                                                 ^~~~~~~~~~~~~~~~~~~~~~~~~

Perhaps cast the enum values to some integer type to silence the warnings if we really do want to compare values from different enums? I'm not sure what type to cast it to make it sufficiently large on all platforms though, perhaps intmax_t?

lld/MachO/UnwindInfoSection.cpp

static void addEntriesForFunctionsWithoutUnwindInfo(
   for (const InputFile *file : inputFiles)
     if (auto *objFile = dyn_cast<ObjFile>(file))
       for (Symbol *sym : objFile->symbols)
         if (auto *d = dyn_cast_or_null<Defined>(sym))
           if (!d->isExternal())
             markNoUnwindInfo(d);
 }
 
+static bool canFoldEncoding(compact_unwind_encoding_t encoding) {
+  // From compact_unwind_encoding.h:
+  //  UNWIND_X86_64_MODE_STACK_IND:
+  //  A "frameless" (RBP not used as frame pointer) function large constant
+  //  stack size.  This case is like the previous, except the stack size is too
+  //  large to encode in the compact unwind encoding.  Instead it requires that
+  //  the function contains "subq $nnnnnnnn,RSP" in its prolog.  The compact
+  //  encoding contains the offset to the nnnnnnnn value in the function in
+  //  UNWIND_X86_64_FRAMELESS_STACK_SIZE.
+  // Since this means the unwinder has to look at the `subq` in the function
+  // of the unwind info's unwind address, two functions that have identical
+  // unwind info can't be folded if it's using this encoding since both
+  // entries need unique addresses.
+  static_assert(UNWIND_X86_64_MODE_MASK == UNWIND_X86_MODE_MASK, "");
+  static_assert(UNWIND_X86_64_MODE_STACK_IND == UNWIND_X86_MODE_STACK_IND, "");
+  if ((target->cpuType == CPU_TYPE_X86_64 || target->cpuType == CPU_TYPE_X86) &&
+      (encoding & UNWIND_X86_64_MODE_MASK) == UNWIND_X86_64_MODE_STACK_IND) {
+    // FIXME: Consider passing in the two function addresses and getting
+    // their two stack sizes off the `subq` and only returning false if they're
+    // actually different.
+    return false;
+  }
+  return true;
+}
+
 // Scan the __LD,__compact_unwind entries and compute the space needs of
 // __TEXT,__unwind_info and __TEXT,__eh_frame
 template <class Ptr> void UnwindInfoSectionImpl<Ptr>::finalize() {
   if (compactUnwindSection == nullptr)
     return;
 
   // At this point, the address space for __TEXT,__text has been
   // assigned, so we can relocate the __LD,__compact_unwind entries
template <class Ptr> void UnwindInfoSectionImpl<Ptr>::finalize() {
   // entries that can be folded into a single entry and written to ...
   // (3) `foldWrite`
   auto foldWrite = cuPtrVector.begin();
   for (auto foldBegin = cuPtrVector.begin(); foldBegin < cuPtrVector.end();) {
     auto foldEnd = foldBegin;
     while (++foldEnd < cuPtrVector.end() &&
            (*foldBegin)->encoding == (*foldEnd)->encoding &&
            (*foldBegin)->personality == (*foldEnd)->personality &&
-           (*foldBegin)->lsda == (*foldEnd)->lsda)
+           (*foldBegin)->lsda == (*foldEnd)->lsda &&
+           canFoldEncoding((*foldEnd)->encoding))
       ;
     *foldWrite++ = *foldBegin;
     foldBegin = foldEnd;
   }
   cuPtrVector.erase(foldWrite, cuPtrVector.end());
 
   encodePersonalities(cuPtrVector, personalities);
 

lld/test/MachO/compact-unwind-stack-ind.s

+# REQUIRES: x86
+# RUN: llvm-mc -filetype=obj -triple=x86_64-apple-darwin19.0.0 %s -o %t.o
+# RUN: %lld -arch x86_64 -dylib %t.o -o %t.dylib
+# RUN: llvm-objdump --macho --syms --unwind-info %t.dylib | FileCheck %s
+
+## Both _f and _g have the same compact unwind encoding,
+## but different stack sizes. So their compact unwindings
+## can't be merged.
+# CHECK:  SYMBOL TABLE:
+# CHECK:  [[#%x,F:]] g  F __TEXT,__text _f
+# CHECK:  [[#%x,G:]] g  F __TEXT,__text _g
+# CHECK: Number of common encodings in array:       0x1
+# CHECK:  Common encodings: (count = 1)
+# CHECK:    encoding[0]: 0x03032000
+# CHECK:  Second level indices:
+# CHECK:    Second level index[0]:
+# CHECK:      [0]: function offset=0x[[#%.8x,F]], encoding[0]=0x03032000
+# CHECK:      [1]: function offset=0x[[#%.8x,G]], encoding[0]=0x03032000
+
+## Based on compiling
+##     int f() {
+##       char alloca[3260] = { 0 };
+##       return alloca[0];
+##     }
+##
+##     int g() {
+##       char alloca[2560] = { 0 };
+##       return alloca[0];
+##     }
+## with `-fomit-frame-pointer -fno-stack-protector -S`.
+.section __TEXT,__text,regular,pure_instructions
+.build_version macos, 10, 15 sdk_version 10, 15, 6
+
+.globl _f
+.p2align 4, 0x90
+_f:
+  .cfi_startproc
+  subq $3272, %rsp
+  .cfi_def_cfa_offset 3280
+  addq $3272, %rsp
+  retq
+  .cfi_endproc
+
+.globl _g
+.p2align 4, 0x90
+_g:
+  .cfi_startproc
+  subq $2568, %rsp
+  .cfi_def_cfa_offset 2576
+  addq $2568, %rsp
+  retq
+  .cfi_endproc
+
+.subsections_via_symbols