[libFuzzer] Use crash stack for fuchsia.

[charco]

Summary

This commit adds a temporary stack to handle libfuzzer crashes in
Fuchsia.

Crashes in fuchsia are handled via exception channels: an exception
handler thread waits for an exception, and when one happens, it will try
to "resurrect" the crashed thread by writing the registers onto the
stack and changing the pc to a crash trampoline, which then calls
libfuzzer's static crash handler.

If the crashed thread has an invalid stack, writing the registers onto
the stack will fail. The end result is that the fuzzer would hang and
the error would be reported as a time out.

To solve it, we set up a temporary stack of a few pages so the crash
handler can run. This crash handler will end up the application, so we
are not expected to go back to normal.

Related to that change, now that we only have one crash stack, we can't
allow other threads to use the same crash stack, and we aren't supposed
to let multiple threads call the crash handler concurrently anyways, so
now we only let the first thread to crash to go through.

Finally, there's also a change in the asan code for fuchsia, to check
whether we are in the default stack or in a different one. Upon calls to
noreturn functions, asan unpoisons the thread stacks, so we check
whether the current stack is the default one, and if so, do everything
as normal. Otherwise, we unpoison the whole default stack, as well as
the current stack page (we don't know how big the current stack is).

[charco]

clang-format

[vitalybuka]

compiler-rt/lib/asan/asan_fuchsia.cpp

74

local_var unused?

and better just use __builtin_frame_address()

82

why exactly GetPageSize?

[charco]

Fix cfi directives for the crash trampoline.

[eep]

compiler-rt/lib/fuzzer/FuzzerUtilFuchsia.cpp

204

Do we need to ensure this is aligned on ARM?

[aarongreen]

compiler-rt/lib/asan/asan_fuchsia.cpp

65–87

Does this affect more than libFuzzer? If so, it probably belongs in its own change, with the libFuzzer changes depending on it.

compiler-rt/lib/fuzzer/FuzzerFuchsiaCrashTrampoline.S

1 ↗

(On Diff #324825)

See my note on the next file. I think this should be FuzzerUnwindFuchsia.S.

compiler-rt/lib/fuzzer/FuzzerFuchsiaRegisters.h

1 ↗

(On Diff #324825)

I don't love this file, as it seems to be squashing together several orthogonal concerns (os, arch, elf/dwarf).

I didn't love how it was before (all hidden in FuzzerUtilFuchsia) but at least it was mostly abstracted away.

I don't love the idea of splitting everything up into separate files, as it leads to quite a few files added for what's by and large a corner case (e.g. FuzzerDwarf.h, FuzzerDwarfX86_64.h, FuzzerDwarfAarch64.h, FuzzerUnwindFuchsia.h, FuzzerUnwindFuchsiax86_64.h, FuzzerUnwindFuchsiaAarch64.h).

At minimum, let's at least name this consistently with the rest of libFuzzer, i.e. make this FuzzerUnwindFuchsia.h.

compiler-rt/lib/fuzzer/FuzzerUtilFuchsia.cpp

42

Pretty sure you mean the zx_thread_state_general_regs_t struct, but it would be good to say so.

42

Also, these look like they can be moved to FuzzerFuchsiaRegisters.h (or FuzzerUnwindFuchsia.h), where they would have more context (but may need a guard like #ifdef __cplusplus... I'm not sure about their inclusion in the .S file).

52

Any particular reason for this size?

125–143

Tackling a few suggestions with one comment:

• The constructor is redundant with the default value for Handle.
• An explicit constructor might be nice.
• If you define a move constructor, its good practice to define a move assignment operator as well, so all future changes work as expected, e.g. unhandled_exceptions.back() = std::move(replacement);

Taken together, I think this is close enough to add unique_ptr-style semantics and put all the logic into get/release/reset methods, much like zx::object in Fuchsia's //zircon/system/ulib/zx/include/zx/object.h.

183–185

This needs a test, but since we're cross compiling and not self-hosted, it probably needs to be on the Fuchsia side. I've opened https://bugs.fuchsia.dev/p/fuchsia/issues/detail?id=71106.

Alternately, you may be able to add some general, OS-agnostic unit tests for the concurrent and double fault cases. We'll still need to address the shortcomings of testing on Fuchsia, though, so the above bug would still be valid.

204

So close to extracting all the arch bits from this file... I wonder if it's worth abstracting this too.

213

Hmm. I feel it might be better to invoke one of the error callbacks directly in this case rather than relying on the std::at_exit handler to land us in Fuzzer::ExitCallback. If we're really concerned with the process being hosed, Fuzzer::DeathCallback might be the best option, as it skips trying to print the stack, but you'd need to call _Exit yourself.

[aarongreen]

compiler-rt/lib/fuzzer/FuzzerFuchsiaCrashTrampoline.S

121 ↗

(On Diff #324825)

Can this just be pc, as with rip above?

[charco]

Code review comments.

[charco]

Thanks for the review. I addressed all the comments. PTAL when you have some time!

compiler-rt/lib/asan/asan_fuchsia.cpp

65–87

libfuzzer is the only one using a different stack than the default, that I am aware of. But this would affect any program that uses asan and a non-default stack.

82

If we are not in the default stack, we have no way of knowing the size of the current stack. In that case, the best thing we can do is to unpoison the current stack page (as it is the one that we are using). An alternative would be to do nothing.

compiler-rt/lib/fuzzer/FuzzerFuchsiaCrashTrampoline.S

1 ↗

(On Diff #324825)

FuzzerUnwindFuchsia is a great name for this file. Thanks for the suggestion!

121 ↗

(On Diff #324825)

I am not sure I follow what you mean here. Above we use the rdi register to hold the address of were we want to jump. Here we use x0.

compiler-rt/lib/fuzzer/FuzzerFuchsiaRegisters.h

1 ↗

(On Diff #324825)

This is consistent to how they handle this stuff in libunwind (one file with ifdefs per arch). It is true that the DWARF Reg Numbers are not specific for fuchsia, but we are the only ones who care about them.

compiler-rt/lib/fuzzer/FuzzerUtilFuchsia.cpp

52

Nope. I just needed to pick a size. This is only used for "printing the backtrace and exiting", so 16KiB seemed more than enough.

125–143

Thanks! The code looks better now.
I decided to leave Handle public because we need its address to be used in the system calls down below. (See for example the usage of Channel.Handle).

183–185

Ack.

204

Technically yes. It just works because sizeof() the struct is already multiple of 16. I added back the rounding formula, but I think a static check would also do, as we don't expect that struct to change.

204

Hm. We will need the arch ifdefs one way or the other. I moved most of this code into a SetupCrashTrampoline function, at least that way we constrain the ifdef to a small function.

213

We had agreed to keep it as is, right? An alternative would be to somehow generate a crash artifact and then let next handler handle the crash (finishing the program)

[charco]

Comments on SetupTrampoline

[charco]

• Uncomment static assert

[charco]

• Enforce alignment & fix array type.

[mcgrathr]

The stated goal is very sensible. But this change is doing too many things at once to get there. Please separate the refactorings into smaller changes that come with some discussion of why that refactoring is needed.

compiler-rt/lib/asan/asan_fuchsia.cpp

65–87

I would prefer to see this as a separate change and with more comments in the code right here explaining the rationale for what it's doing. It's a subtle change in semantics that needs it own discussion separate from the libfuzzer context.

compiler-rt/lib/fuzzer/FuzzerUnwindFuchsia.S

33–34

That's not at all what I think the CFA ought to be. That's something like what the CFA would have been if the faulting instruction has instead been a normal call. That's not what the contract about the CFA is. The CFA is about *your* frame, not your caller's frame. For unwinding purposes, the CFA is whatever you say it is in the CFI. Some ABIs like the generic aarch64 ABI says that the CFA should match the SP on entry to the function. The ABI says that but it's not really part of the unwinding contract, it's just something the ABI says.

Anyway, even to match the letter of the aarch64 ABI, the CFA for the trampoline has nothing whatsoever to do with the interrupted state. The CFA for the trampoline can be the SP on entry *to the trampoline*. Since the protocol of the trampoline as a "function" is actually private between that code and the setup code, you can construe that however you like, e.g. either at the top of the trap frame or the bottom.

What I'd do is make it as if the trampoline were function called by the normal convention, so its SP on entry is below the trap frame. This means the trap frame itself is kept at the stack's alignment requirement (16). On x86 this means an extra stub return address word is pushed first to correctly misalign the SP on entry as per the ABI. This is the CFA definition that plain .cfi_startproc gives you (in the CIE). So if that's the trap frame, it's your CFA. And then all you need is a .cfi_offset R, offsetof(genregs, R) for each one, plus .cfi_signal_frame, and you're unwound.

This way nicely works the same whether you've pushed the trap frame below the interrupted stack or you've started a fresh signal stack.

compiler-rt/lib/fuzzer/FuzzerUnwindFuchsia.h

46

If you're going to take this approach then you need to pepper this file with static_assert(offsetof(...) == REG_...); to validate each and every constant here. This is the big downside of putting the assembly into a .S file instead of using inline asm as before.

You haven't given any rationale for the change to define the assembly in a .S file instead of using the previous method.
That alone is a big refactoring that ought to be a change of its own before the semantic changes. It needs to be explained why that's a better way to organize the code than what we were doing before, which IMHO is substantially less fragile for ongoing maintenance.

compiler-rt/lib/fuzzer/FuzzerUtilFuchsia.cpp

78

If it's important to make this size fixed at compile time, it should be at least 64k.
But it would be better to choose at at runtime with zx_system_get_page_size() * 4 or suchlike, along with some comments giving rationale for the size chosen.

79

Is it important that this be statically allocated? Why not dynamically allocate it during setup?

The alignment requirement is a machine ABI detail and should have a symbolic constant and comment about that, though it happens to be 16 on machines currently supported.

111

Reusing the crashing thread's shadow call stack is no safer than reusing its machine stack.
The same is true of the safe-stack unsafe stack, which is only used by default on x86 but is fully supported by the Fuchsia libc ABI on both machines.

I think you need to have separate unsafe stack (everywhere) and shadow call stack (aarch64 only) for the crash handler too.

126

This seems like a good standalone refactoring with a simple goal: make the type move-friendly (and add the reset method, though since it's not used elsewhere that doesn't seem to have been a goal).
Make that its own separate change, with commentary explaining its goal. I only surmised what the motivation for this change must have been. A log message or code comment should explain that.

[charco]

Thanks for the review, Roland.

I will be OOO next week, but will address the comments when I get back.

• smime.p7s3 KBDownload

compiler-rt/lib/asan/asan_fuchsia.cpp

 
 void *AsanDoesNotSupportStaticLinkage() { return nullptr; }
 
 void InitializePlatformExceptionHandlers() {}
 void AsanOnDeadlySignal(int signo, void *siginfo, void *context) {
   UNIMPLEMENTED();
 }
 
-bool PlatformUnpoisonStacks() { return false; }
+bool PlatformUnpoisonStacks() {
+  // Is the current stack the default one? If not,
+  // we are in a different stack (might be a crash stack from fuzzing).
+  // Unpoison the original stack and the current stack page.
+  AsanThread *curr_thread = GetCurrentThread();
+  CHECK(curr_thread != nullptr);
+  uptr top = curr_thread->stack_top();
+  uptr bottom = curr_thread->stack_bottom();
+
+  uptr local_stack = reinterpret_cast<uptr>(__builtin_frame_address(0));
+
+  if (local_stack < bottom || local_stack > top) {
+    // The current stack is not the default stack.
+    // Unpoison the entire default stack and the current stack page.
+    UnpoisonStack(bottom, top, "default");
+    bottom = RoundDownTo(local_stack, GetPageSize());
+    top = bottom + GetPageSize();
+    UnpoisonStack(bottom, top, "unknown");
+    return true;
+  }
+
+  return false;
+}
 
 // We can use a plain thread_local variable for TSD.
 static thread_local void *per_thread;
 
 void *AsanTSDGet() { return per_thread; }
 
 void AsanTSDSet(void *tsd) { per_thread = tsd; }
 

compiler-rt/lib/fuzzer/CMakeLists.txt

set(LIBFUZZER_SOURCES
   FuzzerIO.cpp
   FuzzerIOPosix.cpp
   FuzzerIOWindows.cpp
   FuzzerLoop.cpp
   FuzzerMerge.cpp
   FuzzerMutate.cpp
   FuzzerSHA1.cpp
   FuzzerTracePC.cpp
+  FuzzerUnwindFuchsia.S
   FuzzerUtil.cpp
   FuzzerUtilDarwin.cpp
   FuzzerUtilFuchsia.cpp
   FuzzerUtilLinux.cpp
   FuzzerUtilPosix.cpp
   FuzzerUtilWindows.cpp)
 
 set(LIBFUZZER_HEADERS
set(LIBFUZZER_HEADERS
   FuzzerInterface.h
   FuzzerInternal.h
   FuzzerMerge.h
   FuzzerMutate.h
   FuzzerOptions.h
   FuzzerRandom.h
   FuzzerSHA1.h
   FuzzerTracePC.h
+  FuzzerUnwindFuchsia.h
   FuzzerUtil.h
   FuzzerValueBitMap.h)
 
 include_directories(../../include)
 
 CHECK_CXX_SOURCE_COMPILES("
   static thread_local int blah;
   int main() {

compiler-rt/lib/fuzzer/FuzzerUnwindFuchsia.h

+//===- FuzzerUnwindFuchsia.h - Unwind information for Fuchsia.      -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// This file contains dwarf registers numbering and the offsets of each
+// register into the zx_thread_state_general_regs_t struct. This is to be
+// consumed from assembly.
+//===----------------------------------------------------------------------===//
+#include "FuzzerPlatform.h"
+
+#if LIBFUZZER_FUCHSIA
+
+#if defined(__x86_64__)
+// DWARF Register Number Mappings for x86-64
+// Extracted from:
+// System V Application Binary Interface
+// AMD64 Architecture Processor Supplement
+// Draft Version 0.99.7
+// https://www.uclibc.org/docs/psABI-x86_64.pdf
+
+#define REG_DWARF_NUMBER_rax 0
+#define REG_DWARF_NUMBER_rbx 3
+#define REG_DWARF_NUMBER_rcx 2
+#define REG_DWARF_NUMBER_rdx 1
+#define REG_DWARF_NUMBER_rsi 4
+#define REG_DWARF_NUMBER_rdi 5
+#define REG_DWARF_NUMBER_rbp 6
+#define REG_DWARF_NUMBER_rsp 7
+#define REG_DWARF_NUMBER_r8 8
+#define REG_DWARF_NUMBER_r9 9
+#define REG_DWARF_NUMBER_r10 10
+#define REG_DWARF_NUMBER_r11 11
+#define REG_DWARF_NUMBER_r12 12
+#define REG_DWARF_NUMBER_r13 13
+#define REG_DWARF_NUMBER_r14 14
+#define REG_DWARF_NUMBER_r15 15
+#define REG_DWARF_NUMBER_rip 16
+
+#define REG_DWARF_NUMBER(reg) (REG_DWARF_NUMBER_##reg)
+
+// Offsets for different x86-64 registers into the
+// zx_thread_state_general_regs_t structure.
+#define REG_OFFSET_rax 0
+#define REG_OFFSET_rbx 8
+#define REG_OFFSET_rcx 16
+#define REG_OFFSET_rdx 24
+#define REG_OFFSET_rsi 32
+#define REG_OFFSET_rdi 40
+#define REG_OFFSET_rbp 48
+#define REG_OFFSET_rsp 56
+#define REG_OFFSET_r8 64
+#define REG_OFFSET_r9 72
+#define REG_OFFSET_r10 80
+#define REG_OFFSET_r11 88
+#define REG_OFFSET_r12 96
+#define REG_OFFSET_r13 104
+#define REG_OFFSET_r14 112
+#define REG_OFFSET_r15 120
+#define REG_OFFSET_rip 128
+#define REG_OFFSET(reg) (REG_OFFSET_##reg)
+
+#define REG_FOREACH(OP_REG, OP_NUM)                                            \
+  OP_REG(rax)                                                                  \
+  OP_REG(rbx)                                                                  \
+  OP_REG(rcx)                                                                  \
+  OP_REG(rdx)                                                                  \
+  OP_REG(rsi)                                                                  \
+  OP_REG(rdi)                                                                  \
+  OP_REG(rbp)                                                                  \
+  OP_REG(rsp)                                                                  \
+  OP_REG(r8)                                                                   \
+  OP_REG(r9)                                                                   \
+  OP_REG(r10)                                                                  \
+  OP_REG(r11)                                                                  \
+  OP_REG(r12)                                                                  \
+  OP_REG(r13)                                                                  \
+  OP_REG(r14)                                                                  \
+  OP_REG(r15)                                                                  \
+  OP_REG(rip)
+
+#elif defined(__aarch64__)
+
+// DWARF Register Number Mappings for ARM64
+// Extracted from:
+// DWARF for the ARM 64-bit Architecture
+// 2020Q4
+// https://github.com/ARM-software/abi-aa/blob/f52e1ad3f81254497a83578dc102f6aac89e52d0/aadwarf64/aadwarf64.rst
+
+#define REG_DWARF_NUMBER_x(num) (num)
+#define REG_DWARF_NUMBER_lr (30)
+#define REG_DWARF_NUMBER_sp (31)
+#define REG_DWARF_NUMBER_pc (32)
+#define REG_DWARF_NUMBER(reg) (REG_DWARF_NUMBER_##reg)
+
+// Offsets for different arm64 registers into the
+// zx_thread_state_general_regs_t structure.
+#define REG_OFFSET_lr (30 * 8)
+#define REG_OFFSET_sp (31 * 8)
+#define REG_OFFSET_pc (32 * 8)
+#define REG_OFFSET_x(num) (num * 8)
+#define REG_OFFSET(reg) (REG_OFFSET_##reg)
+
+#define REG_FOREACH(OP_REG, OP_NUM)                                            \
+  OP_NUM(0)                                                                    \
+  OP_NUM(1)                                                                    \
+  OP_NUM(2)                                                                    \
+  OP_NUM(3)                                                                    \
+  OP_NUM(4)                                                                    \
+  OP_NUM(5)                                                                    \
+  OP_NUM(6)                                                                    \
+  OP_NUM(7)                                                                    \
+  OP_NUM(8)                                                                    \
+  OP_NUM(9)                                                                    \
+  OP_NUM(10)                                                                   \
+  OP_NUM(11)                                                                   \
+  OP_NUM(12)                                                                   \
+  OP_NUM(13)                                                                   \
+  OP_NUM(14)                                                                   \
+  OP_NUM(15)                                                                   \
+  OP_NUM(16)                                                                   \
+  OP_NUM(17)                                                                   \
+  OP_NUM(18)                                                                   \
+  OP_NUM(19)                                                                   \
+  OP_NUM(20)                                                                   \
+  OP_NUM(21)                                                                   \
+  OP_NUM(22)                                                                   \
+  OP_NUM(23)                                                                   \
+  OP_NUM(24)                                                                   \
+  OP_NUM(25)                                                                   \
+  OP_NUM(26)                                                                   \
+  OP_NUM(27)                                                                   \
+  OP_NUM(28)                                                                   \
+  OP_NUM(29)                                                                   \
+  OP_REG(lr)                                                                   \
+  OP_REG(sp)                                                                   \
+  OP_REG(pc)
+
+#else
+#error "Unsupported architecture for fuzzing on Fuchsia"
+#endif
+
+#ifdef __cplusplus
+
+#include <zircon/syscalls/debug.h>
+
+extern "C" void CrashTrampoline(void);
+
+// The register offsets have to match the zx_thread_state_general_regs_t layout.
+#define CHECK_OFFSET(reg)                                                      \
+  static_assert(offsetof(zx_thread_state_general_regs_t, reg) ==               \
+                    REG_OFFSET(reg),                                           \
+                "invalid offset");
+#define CHECK_OFFSET_NUM(num)                                                  \
+  static_assert(offsetof(zx_thread_state_general_regs_t, r[num]) ==            \
+                    REG_OFFSET_x(num),                                         \
+                "invalid offset");
+
+REG_FOREACH(CHECK_OFFSET, CHECK_OFFSET_NUM)
+#endif // __cplusplus
+
+#endif // LIBFUZZER_FUCHSIA

compiler-rt/lib/fuzzer/FuzzerUnwindFuchsia.S

+//===- FuzzerFuchsiaCrashTrampoline.S - Crash Trampoline function for Fuchsia.//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// CrashTrampoline function to be used by libfuzzer in Fuchsia while catching
+// crashes. The CrashTrampoline function just calls into a function it receives
+// as parameter. It has the appropriate CFI directives for libunwind to unwind
+// the full backtrace.
+//===----------------------------------------------------------------------===//
+#include "FuzzerPlatform.h"
+#include "FuzzerUnwindFuchsia.h"
+
+#if LIBFUZZER_FUCHSIA
+
+// These macros expand into the low and high bytes of the 2-byte
+// ULEB128 encoding of the number X.
+// The low byte is bits [0, 7) of X, with a 1 in bit 7.
+// The high byte is bits [7, 14) of X, with a 0 in bit 15. 
+// It is only valid to call these macros with 0 <= X < 0x3FFF
+#define ULEB128_2_LO(X) (((X) & 0x7F) | 0x80)
+#define ULEB128_2_HI(X) (((X) >> 7) & 0x7F)
+
+#define DW_CFA_def_cfa_expression 0x0f
+#define DW_CFA_expression 0x10
+#define DW_OP_breg7 0x77
+#define DW_OP_breg31 0x8f
+#define DW_OP_deref 0x06
+
+// The trampoline function gets called with a new stack that contains the
+// registers state at the point of the crash. In order to unwind it correctly,
+// we need to set the CFA to the previous stack pointer (which is stored in
+// the stack), and add information to restore the register values, so
+// unwinding can continue.
+//
+// Sadly, all the cfi macros to specify register values are relative to
+// the CFA, which in this case is far away from where the registers are
+// stored. Thus, we have to use the .cfi_escape macro to add custom
+// directives. These directives are just dwarf expressions that tells where
+// each register is located relative to the current stack pointer.
+#if defined(__x86_64__)
+
+#define cfi_cfa_deref_sp(offset) .cfi_escape         \
+  DW_CFA_def_cfa_expression,                   \
+  0x04,                             /* size */ \
+  DW_OP_breg7,                                 \
+        ULEB128_2_LO(offset),            /* offset*/ \
+        ULEB128_2_HI(offset),            /* offset*/ \
+  DW_OP_deref
+
+#define cfi_reg_from_sp(regno, offset) .cfi_escape   \
+  DW_CFA_expression,                           \
+  regno,                 /* register number */ \
+  0x03,                             /* size */ \
+  DW_OP_breg7,                                 \
+  ULEB128_2_LO(offset),            /* offset */\
+  ULEB128_2_HI(offset)             /* offset */
+
+#define cfi_num(num) 
+#define cfi_reg(reg) cfi_reg_from_sp(REG_DWARF_NUMBER(reg), REG_OFFSET(reg))
+
+#elif defined(__aarch64__)
+
+#define cfi_cfa_deref_sp(offset) .cfi_escape         \
+  DW_CFA_def_cfa_expression,                   \
+  0x04,                             /* size */ \
+  DW_OP_breg31,                                \
+        ULEB128_2_LO(offset),            /* offset*/ \
+        ULEB128_2_HI(offset),            /* offset*/ \
+  DW_OP_deref
+
+#define cfi_reg_from_sp(regno, offset) .cfi_escape   \
+  DW_CFA_expression,                           \
+  regno,                 /* register number */ \
+  0x03,                             /* size */ \
+  DW_OP_breg31,                                \
+  ULEB128_2_LO(offset),            /* offset */\
+  ULEB128_2_HI(offset)             /* offset */
+
+#define cfi_num(num) cfi_reg_from_sp(REG_DWARF_NUMBER_x(num), REG_OFFSET_x(num))
+#define cfi_reg(reg) cfi_reg_from_sp(REG_DWARF_NUMBER(reg), REG_OFFSET(reg))
+
+#else
+#error "Unsupported architecture for fuzzing on Fuchsia"
+#endif
+
+// Trampoline function with the necessary CFI information to unwind through
+// to the crashing call stack. This function gets called with a new stack, that
+// contains the registers at the point of the crash, and also receives a
+// function pointer to the libfuzzer crash handler so it can call it.
+//
+// The function contains the following CFI directives for the unwinder:
+//
+//  * cfi_cfa_deref_sp: defines the CFA so that it points to the stack pointer
+//    at the point of crash.
+//  * cfi_return_column: when unwinding, the return address will be stored in
+//    the pc register.
+//  * cfi_reg/cfi_num: tells the unwinder how to restore each of the registers
+//    from the stack.
+
+  .text
+  .global CrashTrampoline
+  .type CrashTrampoline, %function
+CrashTrampoline:
+  .cfi_startproc simple
+  .cfi_signal_frame
+
+#if defined(__x86_64__)
+  cfi_cfa_deref_sp(REG_OFFSET(rsp))
+  .cfi_return_column rip
+  REG_FOREACH(cfi_reg, cfi_num)
+  callq *%rdi
+  ud2
+
+#elif defined(__aarch64__)
+  cfi_cfa_deref_sp(REG_OFFSET(sp))
+  .cfi_return_column REG_DWARF_NUMBER(pc)
+  REG_FOREACH(cfi_reg, cfi_num)
+  blr x0
+  brk 1
+
+#else
+#error "Unsupported architecture for fuzzing on Fuchsia"
+#endif
+  .cfi_endproc
+  .size CrashTrampoline, . - CrashTrampoline
+
+#endif  //  LIBFUZZER_FUCHSIA

compiler-rt/lib/fuzzer/FuzzerUtilFuchsia.cpp

 //===- FuzzerUtilFuchsia.cpp - Misc utils for Fuchsia. --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 // Misc utils implementation using Fuchsia/Zircon APIs.
 //===----------------------------------------------------------------------===//
 #include "FuzzerPlatform.h"
 
 #if LIBFUZZER_FUCHSIA
 
 #include "FuzzerInternal.h"
+#include "FuzzerUnwindFuchsia.h"
 #include "FuzzerUtil.h"
 #include <cassert>
 #include <cerrno>
 #include <cinttypes>
 #include <cstdint>
 #include <fcntl.h>
 #include <lib/fdio/fdio.h>
 #include <lib/fdio/spawn.h>

 #include <zircon/syscalls/exception.h>
 #include <zircon/syscalls/object.h>
 #include <zircon/types.h>
 
 #include <vector>
 
 namespace fuzzer {
 
 // Given that Fuchsia doesn't have the POSIX signals that libFuzzer was written
 // around, the general approach is to spin up dedicated threads to watch for
 // each requested condition (alarm, interrupt, crash).  Of these, the crash
 // handler is the most involved, as it requires resuming the crashed thread in
 // order to invoke the sanitizers to get the needed state.
 
-// Forward declaration of assembly trampoline needed to resume crashed threads.
-// This appears to have external linkage to  C++, which is why it's not in the
-// anonymous namespace.  The assembly definition inside MakeTrampoline()
-// actually defines the symbol with internal linkage only.
-void CrashTrampolineAsm() __asm__("CrashTrampolineAsm");
-
 namespace {
 
 // Helper function to handle Zircon syscall failures.
 void ExitOnErr(zx_status_t Status, const char *Syscall) {
   if (Status != ZX_OK) {
     Printf("libFuzzer: %s failed: %s\n", Syscall,
            _zx_status_get_string(Status));
     exit(1);
   }
 }
 
 void AlarmHandler(int Seconds) {
   while (true) {
     SleepSeconds(Seconds);
     Fuzzer::StaticAlarmCallback();
   }
 }
 
-// CFAOffset is used to reference the stack pointer before entering the
-// trampoline (Stack Pointer + CFAOffset = prev Stack Pointer). Before jumping
-// to the trampoline we copy all the registers onto the stack. We need to make
-// sure that the new stack has enough space to store all the registers.
-//
-// The trampoline holds CFI information regarding the registers stored in the
-// stack, which is then used by the unwinder to restore them.
-#if defined(__x86_64__)
-// In x86_64 the crashing function might also be using the red zone (128 bytes
-// on top of their rsp).
-constexpr size_t CFAOffset = 128 + sizeof(zx_thread_state_general_regs_t);
-#elif defined(__aarch64__)
-// In aarch64 we need to always have the stack pointer aligned to 16 bytes, so we
-// make sure that we are keeping that same alignment.
-constexpr size_t CFAOffset = (sizeof(zx_thread_state_general_regs_t) + 15) & -(uintptr_t)16;
-#endif
-
-// For the crash handler, we need to call Fuzzer::StaticCrashSignalCallback
-// without POSIX signal handlers.  To achieve this, we use an assembly function
-// to add the necessary CFI unwinding information and a C function to bridge
-// from that back into C++.
-
-// FIXME: This works as a short-term solution, but this code really shouldn't be
-// architecture dependent. A better long term solution is to implement remote
-// unwinding and expose the necessary APIs through sanitizer_common and/or ASAN
-// to allow the exception handling thread to gather the crash state directly.
-//
-// Alternatively, Fuchsia may in future actually implement basic signal
-// handling for the machine trap signals.
-#if defined(__x86_64__)
-#define FOREACH_REGISTER(OP_REG, OP_NUM) \
-  OP_REG(rax)                            \
-  OP_REG(rbx)                            \
-  OP_REG(rcx)                            \
-  OP_REG(rdx)                            \
-  OP_REG(rsi)                            \
-  OP_REG(rdi)                            \
-  OP_REG(rbp)                            \
-  OP_REG(rsp)                            \
-  OP_REG(r8)                             \
-  OP_REG(r9)                             \
-  OP_REG(r10)                            \
-  OP_REG(r11)                            \
-  OP_REG(r12)                            \
-  OP_REG(r13)                            \
-  OP_REG(r14)                            \
-  OP_REG(r15)                            \
-  OP_REG(rip)
-
-#elif defined(__aarch64__)
-#define FOREACH_REGISTER(OP_REG, OP_NUM) \
-  OP_NUM(0)                              \
-  OP_NUM(1)                              \
-  OP_NUM(2)                              \
-  OP_NUM(3)                              \
-  OP_NUM(4)                              \
-  OP_NUM(5)                              \
-  OP_NUM(6)                              \
-  OP_NUM(7)                              \
-  OP_NUM(8)                              \
-  OP_NUM(9)                              \
-  OP_NUM(10)                             \
-  OP_NUM(11)                             \
-  OP_NUM(12)                             \
-  OP_NUM(13)                             \
-  OP_NUM(14)                             \
-  OP_NUM(15)                             \
-  OP_NUM(16)                             \
-  OP_NUM(17)                             \
-  OP_NUM(18)                             \
-  OP_NUM(19)                             \
-  OP_NUM(20)                             \
-  OP_NUM(21)                             \
-  OP_NUM(22)                             \
-  OP_NUM(23)                             \
-  OP_NUM(24)                             \
-  OP_NUM(25)                             \
-  OP_NUM(26)                             \
-  OP_NUM(27)                             \
-  OP_NUM(28)                             \
-  OP_NUM(29)                             \
-  OP_REG(sp)
-
-#else
-#error "Unsupported architecture for fuzzing on Fuchsia"
-#endif
-
-// Produces a CFI directive for the named or numbered register.
-// The value used refers to an assembler immediate operand with the same name
-// as the register (see ASM_OPERAND_REG).
-#define CFI_OFFSET_REG(reg) ".cfi_offset " #reg ", %c[" #reg "]\n"
-#define CFI_OFFSET_NUM(num) CFI_OFFSET_REG(x##num)
-
-// Produces an assembler immediate operand for the named or numbered register.
-// This operand contains the offset of the register relative to the CFA.
-#define ASM_OPERAND_REG(reg) \
-  [reg] "i"(offsetof(zx_thread_state_general_regs_t, reg) - CFAOffset),
-#define ASM_OPERAND_NUM(num)                                 \
-  [x##num] "i"(offsetof(zx_thread_state_general_regs_t, r[num]) - CFAOffset),
-
 // Trampoline to bridge from the assembly below to the static C++ crash
 // callback.
 __attribute__((noreturn))
 static void StaticCrashHandler() {
   Fuzzer::StaticCrashSignalCallback();
   for (;;) {
     _Exit(1);
   }
 }
 
-// Creates the trampoline with the necessary CFI information to unwind through
-// to the crashing call stack:
-//  * Defining the CFA so that it points to the stack pointer at the point
-//    of crash.
-//  * Storing all registers at the point of crash in the stack and refer to them
-//    via CFI information (relative to the CFA).
-//  * Setting the return column so the unwinder knows how to continue unwinding.
-//  * (x86_64) making sure rsp is aligned before calling StaticCrashHandler.
-//  * Calling StaticCrashHandler that will trigger the unwinder.
-//
-// The __attribute__((used)) is necessary because the function
-// is never called; it's just a container around the assembly to allow it to
-// use operands for compile-time computed constants.
-__attribute__((used))
-void MakeTrampoline() {
-  __asm__(".cfi_endproc\n"
-    ".pushsection .text.CrashTrampolineAsm\n"
-    ".type CrashTrampolineAsm,STT_FUNC\n"
-"CrashTrampolineAsm:\n"
-    ".cfi_startproc simple\n"
-    ".cfi_signal_frame\n"
+// The crashing thread might not have a valid stack.
+// Before executing the crash trampoline, we set up a new stack for them.
+constexpr size_t kCrashStackSize = 4096 * 4;
+uint8_t CrashStack[kCrashStackSize] __attribute__((aligned(16)));
+
+zx_thread_state_general_regs_t
+SetupCrashTrampoline(const zx_thread_state_general_regs_t &registers) {
+  // To unwind properly, we need to push the crashing thread's register state
+  // onto the stack and jump into a trampoline with CFI instructions on how
+  // to restore it. Make sure that the size is aligned to 16 bytes, as in arm
+  // the stack needs to follow that alignment.
+  uintptr_t StackPtr =
+      reinterpret_cast<uintptr_t>(&CrashStack[kCrashStackSize]) -
+      ((sizeof(registers) + 15) & ~(uintptr_t)15);
+
+  __unsanitized_memcpy(reinterpret_cast<void *>(StackPtr), &registers,
+                       sizeof(registers));
+
+  // Minimal registers needed for continuing execution:
+  // + A new stack that holds the register values at point of crash.
+  // + Resume execution from the CrashTrampoline which contains
+  //   CFI directives for restoring registers from the stack and unwind.
+  // + A pointer to the static crash handler.
+  // + Platform reserved registers (fs/tpidr for TLS, r18 for ShadowCallStack)
 #if defined(__x86_64__)
-    ".cfi_return_column rip\n"
-    ".cfi_def_cfa rsp, %c[CFAOffset]\n"
-    FOREACH_REGISTER(CFI_OFFSET_REG, CFI_OFFSET_NUM)
-    "mov %%rsp, %%rbp\n"
-    ".cfi_def_cfa_register rbp\n"
-    "andq $-16, %%rsp\n"
-    "call %c[StaticCrashHandler]\n"
-    "ud2\n"
+  zx_thread_state_general_regs_t res = {
+      .rdi = reinterpret_cast<uint64_t>(StaticCrashHandler),
+      .rsp = StackPtr,
+      .rip = reinterpret_cast<uint64_t>(CrashTrampoline),
+      .fs_base = registers.fs_base,
+      .rflags = registers.rflags,
+  };
 #elif defined(__aarch64__)
-    ".cfi_return_column 33\n"
-    ".cfi_def_cfa sp, %c[CFAOffset]\n"
-    FOREACH_REGISTER(CFI_OFFSET_REG, CFI_OFFSET_NUM)
-    ".cfi_offset 33, %c[pc]\n"
-    ".cfi_offset 30, %c[lr]\n"
-    "bl %c[StaticCrashHandler]\n"
-    "brk 1\n"
+  zx_thread_state_general_regs_t res = {
+      .r[0] = reinterpret_cast<uint64_t>(StaticCrashHandler),
+      .r[18] = registers.r[18],
+      .sp = StackPtr,
+      .pc = reinterpret_cast<uint64_t>(CrashTrampoline),
+      .cpsr = registers.cpsr,
+      .tpidr = registers.tpidr,
+  };
 #else
 #error "Unsupported architecture for fuzzing on Fuchsia"
 #endif
-    ".cfi_endproc\n"
-    ".size CrashTrampolineAsm, . - CrashTrampolineAsm\n"
-    ".popsection\n"
-    ".cfi_startproc\n"
-    : // No outputs
-    : FOREACH_REGISTER(ASM_OPERAND_REG, ASM_OPERAND_NUM)
-#if defined(__aarch64__)
-      ASM_OPERAND_REG(pc)
-      ASM_OPERAND_REG(lr)
-#endif
-      [StaticCrashHandler] "i" (StaticCrashHandler),
-      [CFAOffset] "i" (CFAOffset));
+  return res;
 }
 
 void CrashHandler(zx_handle_t *Event) {
   // This structure is used to ensure we close handles to objects we create in
   // this handler.
-  struct ScopedHandle {
-    ~ScopedHandle() { _zx_handle_close(Handle); }
+  class ScopedHandle {
+  public:
+    ScopedHandle() { reset(); }
+    explicit ScopedHandle(zx_handle_t H) { reset(H); }
+    ScopedHandle(ScopedHandle &&Other) : Handle(Other.release()) {}
+    ScopedHandle &operator=(ScopedHandle &&Other) {
+      Handle = Other.release();
+      return *this;
+    }
+    ScopedHandle(const ScopedHandle &Other) = delete;
+    ScopedHandle &operator=(const ScopedHandle &) = delete;
+    ~ScopedHandle() { reset(Handle); }
+    zx_handle_t release() { return std::exchange(Handle, ZX_HANDLE_INVALID); }
+    void reset(zx_handle_t H = ZX_HANDLE_INVALID) {
+      _zx_handle_close(std::exchange(Handle, H));
+    }
     zx_handle_t Handle = ZX_HANDLE_INVALID;
   };
 
   // Create the exception channel.  We need to claim to be a "debugger" so the
   // kernel will allow us to modify and resume dying threads (see below). Once
   // the channel is set, we can signal the main thread to continue and wait
   // for the exception to arrive.
   ScopedHandle Channel;
   zx_handle_t Self = _zx_process_self();
   ExitOnErr(_zx_task_create_exception_channel(
                 Self, ZX_EXCEPTION_CHANNEL_DEBUGGER, &Channel.Handle),
             "_zx_task_create_exception_channel");
 
   ExitOnErr(_zx_object_signal(*Event, 0, ZX_USER_SIGNAL_0),
             "_zx_object_signal");
 
+  zx_koid_t crashed_tid = ZX_KOID_INVALID;
+  std::vector<ScopedHandle> unhandled_exceptions;
+
   // This thread lives as long as the process in order to keep handling
   // crashes.  In practice, the first crashed thread to reach the end of the
   // StaticCrashHandler will end the process.
   while (true) {
     ExitOnErr(_zx_object_wait_one(Channel.Handle, ZX_CHANNEL_READABLE,
                                   ZX_TIME_INFINITE, nullptr),
               "_zx_object_wait_one");
 
     zx_exception_info_t ExceptionInfo;
     ScopedHandle Exception;
     ExitOnErr(_zx_channel_read(Channel.Handle, 0, &ExceptionInfo,
                                &Exception.Handle, sizeof(ExceptionInfo), 1,
                                nullptr, nullptr),
               "_zx_channel_read");
 
     // Ignore informational synthetic exceptions.
     if (ZX_EXCP_THREAD_STARTING == ExceptionInfo.type ||
         ZX_EXCP_THREAD_EXITING == ExceptionInfo.type ||
         ZX_EXCP_PROCESS_STARTING == ExceptionInfo.type) {
       continue;
     }
 
+    // For the crash handler, we need to call Fuzzer::StaticCrashSignalCallback
+    // without POSIX signal handlers.  To achieve this, we use an assembly
+    // function to add the necessary CFI unwinding information.
+
     // At this point, we want to get the state of the crashing thread, but
     // libFuzzer and the sanitizers assume this will happen from that same
     // thread via a POSIX signal handler. "Resurrecting" the thread in the
     // middle of the appropriate callback is as simple as forcibly setting the
     // instruction pointer/program counter, provided we NEVER EVER return from
     // that function (since otherwise our stack will not be valid).
     ScopedHandle Thread;
     ExitOnErr(_zx_exception_get_thread(Exception.Handle, &Thread.Handle),
               "_zx_exception_get_thread");
 
     zx_thread_state_general_regs_t GeneralRegisters;
     ExitOnErr(_zx_thread_read_state(Thread.Handle, ZX_THREAD_STATE_GENERAL_REGS,
                                     &GeneralRegisters,
                                     sizeof(GeneralRegisters)),
               "_zx_thread_read_state");
 
-    // To unwind properly, we need to push the crashing thread's register state
-    // onto the stack and jump into a trampoline with CFI instructions on how
-    // to restore it.
-#if defined(__x86_64__)
-    uintptr_t StackPtr = GeneralRegisters.rsp - CFAOffset;
-    __unsanitized_memcpy(reinterpret_cast<void *>(StackPtr), &GeneralRegisters,
-                         sizeof(GeneralRegisters));
-    GeneralRegisters.rsp = StackPtr;
-    GeneralRegisters.rip = reinterpret_cast<zx_vaddr_t>(CrashTrampolineAsm);
-
-#elif defined(__aarch64__)
-    uintptr_t StackPtr = GeneralRegisters.sp - CFAOffset;
-    __unsanitized_memcpy(reinterpret_cast<void *>(StackPtr), &GeneralRegisters,
-                         sizeof(GeneralRegisters));
-    GeneralRegisters.sp = StackPtr;
-    GeneralRegisters.pc = reinterpret_cast<zx_vaddr_t>(CrashTrampolineAsm);
+    if (crashed_tid != ZX_KOID_INVALID) {
+      // If we reach this point it means that we received more than
+      // one exception. This new exception could either be from the
+      // same thread as the first exception or from another thread.
+      Printf("libFuzzer: An exception occurred while handling a previous "
+             "crash.\n");
+      if (crashed_tid == ExceptionInfo.tid) {
+        // Exception came from the same thread. This means that the exception
+        // handling code crashed. This is bad. Generate a crashing artifact and
+        // exit.
+        exit(1);
+      } else {
+        // We received a crash from a different thread. Leave it suspended and
+        // wait for the next exception.
+        unhandled_exceptions.push_back(std::move(Exception));
+        continue;
+      }
+    }
 
-#else
-#error "Unsupported architecture for fuzzing on Fuchsia"
-#endif
+    crashed_tid = ExceptionInfo.tid;
+    GeneralRegisters = SetupCrashTrampoline(GeneralRegisters);
 
-    // Now force the crashing thread's state.
     ExitOnErr(
         _zx_thread_write_state(Thread.Handle, ZX_THREAD_STATE_GENERAL_REGS,
                                &GeneralRegisters, sizeof(GeneralRegisters)),
         "_zx_thread_write_state");
 
     // Set the exception to HANDLED so it resumes the thread on close.
     uint32_t ExceptionState = ZX_EXCEPTION_STATE_HANDLED;
     ExitOnErr(_zx_object_set_property(Exception.Handle, ZX_PROP_EXCEPTION_STATE,