[compiler-rt][hwasan] Setup hwasan thread handling on Fuchsia

[leonardchan]

Summary

This patch splits up hwasan thread creation between __sanitizer_before_thread_create_hook, __sanitizer_thread_create_hook, and __sanitizer_thread_start_hook. The linux implementation creates the hwasan thread object inside the new thread. On Fuchsia, we know the stack bounds before thread creation, so we can initialize part of the thread object in __sanitizer_before_thread_create_hook, then initialize the stack ring buffer in __sanitizer_thread_start_hook once we enter the thread.

[leonardchan]

The non-fuchsia stuff can be split up in a separate patch when ready, but I'll keep it here for now until we're set on the thread handling.

[mcgrathr]

Usually we like to split changes up into separate small changes for the pure refactoring, and then changes that purely add new Fuchsia-specific code.
I'll do an initial review round of the Fuchsia code here since you've sent it. But then I think this review should be tightened to just the pure refactoring, and we should have a separate review thread for the Fuchsia parts.

compiler-rt/lib/hwasan/hwasan_fuchsia.cpp

26

Specifically call out here that the compiler generates direct TLS references to this and that's why it has a public C ABI name.

For Fuchsia the hwasan will always be in a DSO that is a dependency of libc.so and so always loaded at startup. Hence we can use [[gnu::tls_model("initial-exec")]] here.

32–33

department of redundancy department

34

Nit: I'd call it something more like InitState.

41

What depends on this alignment? I'm not aware of anything that does in the compiler ABI we're using on Fuchsia.
If it's needed, it should have comments.

84–88

Actually we could do all of the allocation and setup except for storing the uptr in __hwasan_tls.
But that would take more refactoring of the common code and it's not clear there's a particular benefit.

137

Most of what's happening in this function is common with the Linux code.
I think a shared function can be factored out and put into hwasan_thread.cpp.

compiler-rt/lib/hwasan/hwasan_thread.cpp

42–45

This is stuff that could be done either in the before-create hook or in the on-start hook. But it's probably not especially worthwhile to move it to before-create as long as we have on-start doing any allocation at all. So to avoid a whole lot more refactoring to move the ringbuffer setup and such, might as well just leave this in on-start too.

58–63

This is probably the only part that actually needs to be different between Linux and Fuchsia.
So this code could just move into an InitStackAndTls(options) that looks like this on Linux and on Fuchsia is just copying values out of options.

compiler-rt/lib/hwasan/hwasan_thread.h

85

I think these need to be left uniformly uninitialized to guarantee they can be "linker-initialized".
At any rate, there's no reason one pointer member here should be initialized and the others not.

[leonardchan]

Rebased against recent refactoring commits and addressed some comments.

I think I can omit some functions like GetCurrentThread or GetRingBufferSize which are small enough to merit their own changes .

[leonardchan]

compiler-rt/lib/hwasan/hwasan_fuchsia.cpp

41

Added.

137

Updated such that this just calls InitStackRingBuffer which handles the stack ring buffer initialization.

compiler-rt/lib/hwasan/hwasan_thread.cpp

42–45

With the refactoring from D104248, this bit will be done in the before-create hook without any changes to this code.

58–63

This is done in D104248.

[leonardchan]

Ok. Ready for reviews.

[leonardchan]

Let me know if this should be split up also. I think this might be small and and simple enough for one patch.

[mcgrathr]

lgtm

compiler-rt/lib/hwasan/CMakeLists.txt

 include_directories(..)
 
 # Runtime library sources and build flags.
 set(HWASAN_RTL_SOURCES
   hwasan.cpp
   hwasan_allocator.cpp
   hwasan_allocation_functions.cpp
   hwasan_dynamic_shadow.cpp
   hwasan_exceptions.cpp
+  hwasan_fuchsia.cpp
   hwasan_globals.cpp
   hwasan_interceptors.cpp
   hwasan_interceptors_vfork.S
   hwasan_linux.cpp
   hwasan_memintrinsics.cpp
   hwasan_poisoning.cpp
   hwasan_report.cpp
   hwasan_setjmp.S

compiler-rt/lib/hwasan/hwasan.cpp

static void InitInstrumentation() {
 
   if (!InitShadow()) {
     Printf("FATAL: HWAddressSanitizer cannot mmap the shadow memory.\n");
     DumpProcessMap();
     Die();
   }
 
   InitThreads();
-  hwasanThreadList().CreateCurrentThread();
 
   hwasan_instrumentation_inited = 1;
 }
 
 // Interface.
 
 uptr __hwasan_shadow_memory_dynamic_address;  // Global interface symbol.
 

compiler-rt/lib/hwasan/hwasan_fuchsia.cpp

+//===-- hwasan_fuchsia.cpp --------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file is a part of HWAddressSanitizer and contains Fuchsia-specific
+/// code.
+///
+//===----------------------------------------------------------------------===//
+
+#include "sanitizer_common/sanitizer_fuchsia.h"
+#if SANITIZER_FUCHSIA
+
+#include "hwasan.h"
+#include "hwasan_interface_internal.h"
+#include "hwasan_report.h"
+#include "hwasan_thread.h"
+#include "hwasan_thread_list.h"
+
+// This TLS variable contains the location of the stack ring buffer and can be
+// used to always find the hwasan thread object associated with the current
+// running thread.
+SANITIZER_INTERFACE_ATTRIBUTE
+THREADLOCAL uptr __hwasan_tls;
+
+namespace __hwasan {
+
+// These are known parameters passed to the hwasan runtime on thread creation
+// when creating a thread.
+struct Thread::InitOptions {
+  uptr stack_bottom, stack_top;
+};
+
+static void FinishThreadInitialization(Thread *thread);
+
+void InitThreads() {
+  uptr alloc_size = UINT64_C(1) << kShadowBaseAlignment;
+  uptr thread_start = reinterpret_cast<uptr>(
+      MmapAlignedOrDieOnFatalError(alloc_size, alloc_size, __func__));
+
+  InitThreadList(thread_start, alloc_size);
+
+  // Create the hwasan thread object for the current (main) thread. Stack info
+  // for this thread is known from information passed via
+  // __sanitizer_startup_hook.
+  const Thread::InitOptions options = {
+      .stack_bottom = __sanitizer::MainThreadStackBase,
+      .stack_top =
+          __sanitizer::MainThreadStackBase + __sanitizer::MainThreadStackSize,
+  };
+  FinishThreadInitialization(hwasanThreadList().CreateCurrentThread(&options));
+}
+
+uptr *GetCurrentThreadLongPtr() { return &__hwasan_tls; }
+
+Thread *GetCurrentThread() {
+  uptr *ThreadLongPtr = GetCurrentThreadLongPtr();
+  if (UNLIKELY(*ThreadLongPtr == 0))
+    return nullptr;
+  auto *R = (StackAllocationsRingBuffer *)ThreadLongPtr;
+  return hwasanThreadList().GetThreadByBufferAddress((uptr)R->Next());
+}
+
+// This is called from the parent thread before the new thread is created. Here
+// we can propagate known info like the stack bounds to Thread::Init before
+// jumping into the thread. We cannot initialize the stack ring buffer yet since
+// we have not entered the new thread.
+static void *BeforeThreadCreateHook(uptr user_id, bool detached,
+                                    const char *name, uptr stack_bottom,
+                                    uptr stack_size) {
+  const Thread::InitOptions options = {
+      .stack_bottom = stack_bottom,
+      .stack_top = stack_bottom + stack_size,
+  };
+  return hwasanThreadList().CreateCurrentThread(&options);
+}
+
+// Unlike the linux implementation which requires Thread::Init to be called
+// while in the newly created thread, we can set the stack bounds because they
+// are known before thread creation. However, we cannot setup the stack ring
+// buffer just yet because it is stored in the global __hwasan_tls, which we can
+// only correctly access once in the new thread. This will be set up in the
+// ThreadStartHook where we can safely reference __hwasan_tls while in the new
+// thread.
+//
+// This function is called through HwasanThreadList::CreateCurrentThread.
+void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size,
+                  const InitOptions *options) {
+  CHECK_EQ(0, unique_id_);  // try to catch bad stack reuse
+  CHECK_EQ(0, stack_top_);
+  CHECK_EQ(0, stack_bottom_);
+
+  static u64 unique_id;
+  unique_id_ = unique_id++;
+  if (auto sz = flags()->heap_history_size)
+    heap_allocations_ = HeapAllocationsRingBuffer::New(sz);
+
+  CHECK_NE(options->stack_bottom, 0);
+  CHECK_NE(options->stack_top, 0);
+  stack_bottom_ = options->stack_bottom;
+  stack_top_ = options->stack_top;
+  tls_end_ = tls_begin_ = 0;
+}
+
+// This is called after creating a new thread with the pointer returned by
+// BeforeThreadCreateHook. We are still in the creating thread and should check
+// if it was actually created correctly.
+static void ThreadCreateHook(void *hook, bool aborted) {
+  Thread *thread = static_cast<Thread *>(hook);
+  if (!aborted) {
+    // The thread was created successfully.
+    // ThreadStartHook can already be running in the new thread.
+  } else {
+    // The thread wasn't created after all.
+    // Clean up everything we set up in BeforeThreadCreateHook.
+    atomic_signal_fence(memory_order_seq_cst);
+    hwasanThreadList().ReleaseThread(thread);
+  }
+}
+
+// This is called in the newly-created thread before it runs anything else,
+// with the pointer returned by BeforeThreadCreateHook (above). Here we can
+// setup the stack ring buffer.
+static void ThreadStartHook(void *hook, thrd_t self) {
+  Thread *thread = static_cast<Thread *>(hook);
+  FinishThreadInitialization(thread);
+}
+
+// This is the actual function that (1) sets up the stack ring buffer and (2)
+// initializes the random state of the thread. This function should only be
+// called while IN the thread of the passed hwasan thread object so the right
+// __hwasan_tls can be referenced.
+static void FinishThreadInitialization(Thread *thread) {
+  CHECK_NE(thread, nullptr);
+
+  // The following implicitly sets up the thread object as the current thread.
+  // That is, we can always get the appropriate thread object from the ring
+  // buffer using __hwasan_tls. The ring buffer is located immediately before
+  // the thread object.
+  uptr stack_buffer_size = hwasanThreadList().GetRingBufferSize();
+  uptr stack_buffer_start = reinterpret_cast<uptr>(thread) - stack_buffer_size;
+  uptr *ThreadLong = GetCurrentThreadLongPtr();
+  thread->InitStackAllocations(new (ThreadLong) StackAllocationsRingBuffer(
+      (void *)stack_buffer_start, stack_buffer_size));
+  CHECK_EQ(GetCurrentThread(), thread);
+
+  // ScopedTaggingDisable needs GetCurrentThread to be set up. This disables
+  // random tag generation until we exit this function. We do this because we
+  // still need to initialize the random state of the thread. This is only
+  // relevant if tags are generated with calls to the runtime.
+  ScopedTaggingDisabler disabler;
+
+  // Sanity checks to make sure we have the correct stack bounds.
+  CHECK_NE(thread->stack_bottom(), 0);
+  CHECK_NE(thread->stack_top(), 0);
+  int local;
+  CHECK(thread->AddrIsInStack((uptr)&local));
+
+  // TODO(leonardchan): Uncomment these once MemIsApp is implemented.
+  // CHECK(MemIsApp(thread->stack_bottom()));
+  // CHECK(MemIsApp(thread->stack_top() - 1));
+
+  if (flags()->verbose_threads) {
+    if (thread->IsMainThread()) {
+      Printf("sizeof(Thread): %zd sizeof(HeapRB): %zd sizeof(StackRB): %zd\n",
+             sizeof(Thread), thread->heap_allocations()->SizeInBytes(),
+             thread->stack_allocations()->size() * sizeof(uptr));
+    }
+    Printf("Creating  : ");
+  }
+
+  // Initialize a random state. This sets a random starting position in the
+  // stack ring buffer and is used for making random tags if tags are generated
+  // by calls to the runtime.
+  thread->InitRandomState();
+}
+
+static void ThreadExitHook(void *hook, thrd_t self) {
+  Thread *thread = static_cast<Thread *>(hook);
+  atomic_signal_fence(memory_order_seq_cst);
+  hwasanThreadList().ReleaseThread(thread);
+}
+
+}  // namespace __hwasan
+
+extern "C" {
+
+void *__sanitizer_before_thread_create_hook(thrd_t thread, bool detached,
+                                            const char *name, void *stack_base,
+                                            size_t stack_size) {
+  return __hwasan::BeforeThreadCreateHook(
+      reinterpret_cast<uptr>(thread), detached, name,
+      reinterpret_cast<uptr>(stack_base), stack_size);
+}
+
+void __sanitizer_thread_create_hook(void *hook, thrd_t thread, int error) {
+  __hwasan::ThreadCreateHook(hook, error != thrd_success);
+}
+
+void __sanitizer_thread_start_hook(void *hook, thrd_t self) {
+  __hwasan::ThreadStartHook(hook, reinterpret_cast<uptr>(self));
+}
+
+void __sanitizer_thread_exit_hook(void *hook, thrd_t self) {
+  __hwasan::ThreadExitHook(hook, self);
+}
+
+}  // extern "C"
+
+#endif  // SANITIZER_FUCHSIA

compiler-rt/lib/hwasan/hwasan_linux.cpp

uptr thread_space_start =
       __hwasan_shadow_memory_dynamic_address - (1ULL << kShadowBaseAlignment);
   uptr thread_space_end =
       __hwasan_shadow_memory_dynamic_address - guard_page_size;
   ReserveShadowMemoryRange(thread_space_start, thread_space_end - 1,
                            "hwasan threads", /*madvise_shadow*/ false);
   ProtectGap(thread_space_end,
              __hwasan_shadow_memory_dynamic_address - thread_space_end);
   InitThreadList(thread_space_start, thread_space_end - thread_space_start);
+
+  hwasanThreadList().CreateCurrentThread();
 }
 
 bool MemIsApp(uptr p) {
 // Memory outside the alias range has non-zero tags.
 #  if !defined(HWASAN_ALIASING_MODE)
   CHECK(GetTagFromPointer(p) == 0);
 #  endif
 
void HwasanOnDeadlySignal(int signo, void *info, void *context) {
   // Probably a tag mismatch.
   if (signo == SIGTRAP)
     if (HwasanOnSIGTRAP(signo, (siginfo_t *)info, (ucontext_t*)context))
       return;
 
   HandleDeadlySignal(info, context, GetTid(), &OnStackUnwind, nullptr);
 }
 
+// This must be called from the thread itself.
+void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size,
+                  const InitOptions *options) {
+  CHECK_EQ(0, unique_id_);  // try to catch bad stack reuse
+  CHECK_EQ(0, stack_top_);
+  CHECK_EQ(0, stack_bottom_);
+
+  static u64 unique_id;
+  unique_id_ = unique_id++;
+  if (auto sz = flags()->heap_history_size)
+    heap_allocations_ = HeapAllocationsRingBuffer::New(sz);
+
+  HwasanTSDThreadInit();  // Only needed with interceptors.
+  uptr *ThreadLong = GetCurrentThreadLongPtr();
+  // The following implicitly sets (this) as the current thread.
+  stack_allocations_ = new (ThreadLong)
+      StackAllocationsRingBuffer((void *)stack_buffer_start, stack_buffer_size);
+  // Check that it worked.
+  CHECK_EQ(GetCurrentThread(), this);
+
+  // ScopedTaggingDisable needs GetCurrentThread to be set up.
+  ScopedTaggingDisabler disabler;
+
+  // The options should not be passed in this implementation. Stack and TLS info
+  // are retrieved via GetThreadStackAndTls.
+  DCHECK_EQ(options, nullptr);
+  uptr tls_size;
+  uptr stack_size;
+  GetThreadStackAndTls(IsMainThread(), &stack_bottom_, &stack_size, &tls_begin_,
+                       &tls_size);
+  stack_top_ = stack_bottom_ + stack_size;
+  tls_end_ = tls_begin_ + tls_size;
+
+  if (stack_bottom_) {
+    int local;
+    CHECK(AddrIsInStack((uptr)&local));
+    CHECK(MemIsApp(stack_bottom_));
+    CHECK(MemIsApp(stack_top_ - 1));
+  }
+
+  if (flags()->verbose_threads) {
+    if (IsMainThread()) {
+      Printf("sizeof(Thread): %zd sizeof(HeapRB): %zd sizeof(StackRB): %zd\n",
+             sizeof(Thread), heap_allocations_->SizeInBytes(),
+             stack_allocations_->size() * sizeof(uptr));
+    }
+    Print("Creating  : ");
+  }
+}
 
 } // namespace __hwasan
 
 // Entry point for interoperability between __hwasan_tag_mismatch (ASM) and the
 // rest of the mismatch handling code (C++).
 void __hwasan_tag_mismatch4(uptr addr, uptr access_info, uptr *registers_frame,
                             size_t outsize) {
   __hwasan::HwasanTagMismatch(addr, access_info, registers_frame, outsize);
 }
 
 #endif // SANITIZER_FREEBSD || SANITIZER_LINUX || SANITIZER_NETBSD

compiler-rt/lib/hwasan/hwasan_thread.h

 #include "sanitizer_common/sanitizer_ring_buffer.h"
 
 namespace __hwasan {
 
 typedef __sanitizer::CompactRingBuffer<uptr> StackAllocationsRingBuffer;
 
 class Thread {
  public:
-  void Init(uptr stack_buffer_start, uptr stack_buffer_size);  // Must be called from the thread itself.
+  // These are optional parameters that can be passed to Init.
+  struct InitOptions;
+
+  void Init(uptr stack_buffer_start, uptr stack_buffer_size,
+            const InitOptions *options = nullptr);
+
+  void InitStackAllocations(StackAllocationsRingBuffer *allocations) {
+    CHECK_EQ(stack_allocations_, nullptr);
+    stack_allocations_ = allocations;
+  }
+
   void InitRandomState();
   void Destroy();
 
   uptr stack_top() { return stack_top_; }
   uptr stack_bottom() { return stack_bottom_; }
   uptr stack_size() { return stack_top() - stack_bottom(); }
   uptr tls_begin() { return tls_begin_; }
   uptr tls_end() { return tls_end_; }
private:
   uptr tls_begin_;
   uptr tls_end_;
 
   u32 random_state_;
   u32 random_buffer_;
 
   AllocatorCache allocator_cache_;
   HeapAllocationsRingBuffer *heap_allocations_;
-  StackAllocationsRingBuffer *stack_allocations_;
+  StackAllocationsRingBuffer *stack_allocations_ = nullptr;
 
   u64 unique_id_;  // counting from zero.
 
   u32 tagging_disabled_;  // if non-zero, malloc uses zero tag in this thread.
 
   bool announced_;
 
   friend struct ThreadListHead;

compiler-rt/lib/hwasan/hwasan_thread.cpp

void Thread::InitRandomState() {
   random_state_ = flags()->random_tags ? RandomSeed() : unique_id_;
 
   // Push a random number of zeros onto the ring buffer so that the first stack
   // tag base will be random.
   for (tag_t i = 0, e = GenerateRandomTag(); i != e; ++i)
     stack_allocations_->push(0);
 }
 
-void Thread::Init(uptr stack_buffer_start, uptr stack_buffer_size) {
-  CHECK_EQ(0, unique_id_);  // try to catch bad stack reuse
-  CHECK_EQ(0, stack_top_);
-  CHECK_EQ(0, stack_bottom_);
-
-  static u64 unique_id;
-  unique_id_ = unique_id++;
-  if (auto sz = flags()->heap_history_size)
-    heap_allocations_ = HeapAllocationsRingBuffer::New(sz);
-
-  HwasanTSDThreadInit();  // Only needed with interceptors.
-  uptr *ThreadLong = GetCurrentThreadLongPtr();
-  // The following implicitly sets (this) as the current thread.
-  stack_allocations_ = new (ThreadLong)
-      StackAllocationsRingBuffer((void *)stack_buffer_start, stack_buffer_size);
-  // Check that it worked.
-  CHECK_EQ(GetCurrentThread(), this);
-
-  // ScopedTaggingDisable needs GetCurrentThread to be set up.
-  ScopedTaggingDisabler disabler;
-
-  uptr tls_size;
-  uptr stack_size;
-  GetThreadStackAndTls(IsMainThread(), &stack_bottom_, &stack_size, &tls_begin_,
-                       &tls_size);
-  stack_top_ = stack_bottom_ + stack_size;
-  tls_end_ = tls_begin_ + tls_size;
-
-  if (stack_bottom_) {
-    int local;
-    CHECK(AddrIsInStack((uptr)&local));
-    CHECK(MemIsApp(stack_bottom_));
-    CHECK(MemIsApp(stack_top_ - 1));
-  }
-
-  if (flags()->verbose_threads) {
-    if (IsMainThread()) {
-      Printf("sizeof(Thread): %zd sizeof(HeapRB): %zd sizeof(StackRB): %zd\n",
-             sizeof(Thread), heap_allocations_->SizeInBytes(),
-             stack_allocations_->size() * sizeof(uptr));
-    }
-    Print("Creating  : ");
-  }
-}
-
 void Thread::ClearShadowForThreadStackAndTLS() {
   if (stack_top_ != stack_bottom_)
     TagMemory(stack_bottom_, stack_top_ - stack_bottom_, 0);
   if (tls_begin_ != tls_end_)
     TagMemory(tls_begin_, tls_end_ - tls_begin_, 0);
 }
 
 void Thread::Destroy() {

compiler-rt/lib/hwasan/hwasan_thread_list.h

HwasanThreadList(uptr storage, uptr size)
     // Each element contains
     // * a ring buffer at offset 0,
     // * a Thread object at offset ring_buffer_size_.
     ring_buffer_size_ = RingBufferSize();
     thread_alloc_size_ =
         RoundUpTo(ring_buffer_size_ + sizeof(Thread), ring_buffer_size_ * 2);
   }
 
-  Thread *CreateCurrentThread() {
+  Thread *CreateCurrentThread(const Thread::InitOptions *options = nullptr) {
     Thread *t = nullptr;
     {
       SpinMutexLock l(&free_list_mutex_);
       if (!free_list_.empty()) {
         t = free_list_.back();
         free_list_.pop_back();
       }
     }
     if (t) {
       uptr start = (uptr)t - ring_buffer_size_;
       internal_memset((void *)start, 0, ring_buffer_size_ + sizeof(Thread));
     } else {
       t = AllocThread();
     }
     {
       SpinMutexLock l(&live_list_mutex_);
       live_list_.push_back(t);
     }
-    t->Init((uptr)t - ring_buffer_size_, ring_buffer_size_);
+    t->Init((uptr)t - ring_buffer_size_, ring_buffer_size_, options);
     AddThreadStats(t);
     return t;
   }
 
   void DontNeedThread(Thread *t) {
     uptr start = (uptr)t - ring_buffer_size_;
     ReleaseMemoryPagesToOS(start, start + thread_alloc_size_);
   }
void RemoveThreadStats(Thread *t) {
     stats_.total_stack_size -= t->stack_size();
   }
 
   ThreadStats GetThreadStats() {
     SpinMutexLock l(&stats_mutex_);
     return stats_;
   }
 
+  // Reuse this value so external users don't need to call RingBufferSize().
+  uptr GetRingBufferSize() const { return ring_buffer_size_; }
+
  private:
   Thread *AllocThread() {
     SpinMutexLock l(&free_space_mutex_);
     uptr align = ring_buffer_size_ * 2;
     CHECK(IsAligned(free_space_, align));
     Thread *t = (Thread *)(free_space_ + ring_buffer_size_);
     free_space_ += thread_alloc_size_;
     CHECK(free_space_ <= free_space_end_ && "out of thread memory");