[asan] Improved support for swapcontext()

[pdziepak]

Summary

These changess improve ASan support for swapcontext() and friends mainly
focusing on eliminating false positives caused by throwing and catching
exceptions when on a custom user stack.

This is achieved by:

1. Making swapcontext() and getcontext() fill ucontext_t::uc_stack with proper

values.

2. Adding ability to AsanThread to change current stack (in an async-safe way).

Some of the things that are still not fully supported:

• swapcontext() and fake stacks
• changing context from signal handlers
• changing context using longjmp()
• final context change via ucontext_t::uc_link

[kcc]

Thanks for this work, but, just curious, what is your motivation?
What kind of software are you trying to apply asan to?
I haven't heard about swapcontext for a loooong time.

The change in AsanThread is quite intrusive and potentially risky.
While I don't see what may go wrong, I'd like to have more tests, e.g. with threads, leaks, signals, etc.

Besides, I'd like to have another pair of eyes here. Anyone?

lib/asan/asan_interceptors.cc

350

initialize these just in case (with =0); same below.

351

do you know a situation where curr_thread is nullptr? same below

test/asan/TestCases/Linux/swapcontext_test.cc

32

shouldn't we call Func from Throw() or some such?

test/lsan/TestCases/swapcontext.cc

5

don't we still want to test the case with heap memory?

[samsonov]

lib/asan/asan_interceptors.cc

351

Yeah, I believe this happens during thread destruction.

354

Won't these be overwritten anyway by real swapcontext(), when it will be copying ucp to oucp?

392

Why do you need this?

lib/asan/asan_thread.cc

204

Maybe, just add a constructor for StackDescriptor that would fill everything with zeroes?

lib/asan/asan_thread.h

170

Note: you can replace three pointers with a single "int current_stack_;", where
next_stack_ = &stacks_[current_stack_];
previous_stack_ = &stacks_[(current_stack_ + 1) % 3];
temp_stack_ = &stacks_[(current_stack_ + 2) % 3];

test/lsan/TestCases/swapcontext.cc

5

We definitely still want to run it. Can we predict if LSan will or won't report leak in this case?

[pdziepak]

The main motivation is to avoid false positives we have been getting in SeaStar ( http://www.seastar-project.org/ ). It is a future/promise/continuation based framework that also implements coroutines using swapcontext() and friends, so that we get something similar to resumable functions (actually, it's using mostly longjmp() but that's more difficult to support properly in ASan). With these changes the false positives we've been seeing so far are gone.

lib/asan/asan_interceptors.cc

351

Every use of GetCurrentThread() I've seen in ASan code has guard against the function returning nullptr. I admit I haven't dug deeper but just wanted to be safe and followed what is done in the other places.

354

swapcontext() doesn't copy ucp to oucp. It saves the current context to oucp (so that when it is restored it behaves as if swapcontext() has returned 0) and switches to ucp.
However, glibc implementations of swapcontext() and getcontext() (at least their x86 versions) don't write anything to ucontext_t::uc_stack and that's why that WriteContextStack() is needed – to preserve stack information across setcontext/getcontext/swapcontext calls. If swapcontext() actually writes something to uc_stack it is ok as long it is valid data and not zeroes or some garbage.

392

Like I mentioned in earlier reply, getcontext() (or at least some its implementations) doesn't write anything to ucontext_t::uc_stack. Relying in setcontext() and swapcontext() on the fact that uc_stack contains valid information makes thing much easier so I decided to "fix" getcontext() and swapcontext().

lib/asan/asan_thread.h

170

Is it worth it? We save no more than 24 bytes per thread which usually will be much less than 1% percent of memory used by the thread stack, but make code both harder to read (another layer of indirection since logic needed to get stack pointer will have to be hidden in some function) and longer to execute (mod non-power-of-two).

test/asan/TestCases/Linux/swapcontext_test.cc

32

What these changes are trying to fix is mainly the false positive like this:

1. An exception is thrown on custom stack.
2. __asan_handle_no_return() is called but doesn't have correct stack information so just prints a warning and bails out, as a result stack shadow memory is not cleared
3. Exception is handled, stack is unwind. Execution continues.
4. Function called from exception handler or after exception was handled touches stack, but the shadow memory contains stale information (that was supposed to be cleared by __asan_handle_no_return()). Result: false positive.

So that's what I am testing here. Throw() has some data on stack, exception is thrown and handled and then another function touches the stack in places that Throw() has used before.

test/lsan/TestCases/swapcontext.cc

5

LSan will retain its original behaviour (i.e. not reporting leaks on custom stack). The solution would be to either add similar changes to LSan (or perhaps reduce code duplication between LSan and ASan) or write these testing directives so that different behaviour is expected from LSan than from ASan. I am not sure if the latter is possible.

lib/asan/asan_interceptors.cc

Report("WARNING: ASan doesn't fully support makecontext/swapcontext "
            "functions and may produce false positives in some cases!\n");
     reported_warning = true;
   }
   // Clear shadow memory for new context (it may share stack
   // with current context).
   uptr stack, ssize;
   ReadContextStack(ucp, &stack, &ssize);
   ClearShadowMemoryForContextStack(stack, ssize);
+  AsanThread *curr_thread = GetCurrentThread();
+  uptr old_stack, old_ssize;
+  if (curr_thread) {
+    old_stack = curr_thread->stack_bottom();
+    old_ssize = curr_thread->stack_size();
+    WriteContextStack(oucp, old_stack, old_ssize);
+    curr_thread->SetUserStack(stack, ssize);
+  }
   int res = REAL(swapcontext)(oucp, ucp);
+  if (curr_thread) {
+    curr_thread->SetUserStack(old_stack, old_ssize);
+  }
   // swapcontext technically does not return, but program may swap context to
   // "oucp" later, that would look as if swapcontext() returned 0.
   // We need to clear shadow for ucp once again, as it may be in arbitrary
   // state.
   ClearShadowMemoryForContextStack(stack, ssize);
   return res;
 }
+
+INTERCEPTOR(int, setcontext, struct ucontext_t *ucp) {
+  uptr stack, ssize;
+  ReadContextStack(ucp, &stack, &ssize);
+  ClearShadowMemoryForContextStack(stack, ssize);
+
+  AsanThread *curr_thread = GetCurrentThread();
+  uptr old_stack, old_ssize;
+  if (curr_thread) {
+      old_stack = curr_thread->stack_bottom();
+      old_ssize = curr_thread->stack_size();
+      curr_thread->SetUserStack(stack, ssize);
+  }
+  int res = REAL(setcontext)(ucp);
+  if (curr_thread) {
+    curr_thread->SetUserStack(old_stack, old_ssize);
+  }
+  return res;
+}
+
+INTERCEPTOR(int, getcontext, struct ucontext_t *ucp) {
+  int res = REAL(getcontext)(ucp);
+  AsanThread *curr_thread = GetCurrentThread();
+  if (!res && curr_thread) {
+    WriteContextStack(ucp, curr_thread->stack_bottom(),
+                      curr_thread->stack_size());
+  }
+  return res;
+}
 #endif  // ASAN_INTERCEPT_SWAPCONTEXT
 
 INTERCEPTOR(void, longjmp, void *env, int val) {
   __asan_handle_no_return();
   REAL(longjmp)(env, val);
 }
 
 #if ASAN_INTERCEPT__LONGJMP
#if ASAN_INTERCEPT_SIGNAL_AND_SIGACTION
   ASAN_INTERCEPT_FUNC(sigaction);
 #if SANITIZER_ANDROID
   ASAN_INTERCEPT_FUNC(bsd_signal);
 #endif
   ASAN_INTERCEPT_FUNC(signal);
 #endif
 #if ASAN_INTERCEPT_SWAPCONTEXT
   ASAN_INTERCEPT_FUNC(swapcontext);
+  ASAN_INTERCEPT_FUNC(setcontext);
+  ASAN_INTERCEPT_FUNC(getcontext);
 #endif
 #if ASAN_INTERCEPT__LONGJMP
   ASAN_INTERCEPT_FUNC(_longjmp);
 #endif
 #if ASAN_INTERCEPT_SIGLONGJMP
   ASAN_INTERCEPT_FUNC(siglongjmp);
 #endif
 

lib/asan/asan_internal.h

 // asan_linux.cc / asan_mac.cc / asan_win.cc
 void *AsanDoesNotSupportStaticLinkage();
 void AsanCheckDynamicRTPrereqs();
 void AsanCheckIncompatibleRT();
 
 void AsanOnDeadlySignal(int, void *siginfo, void *context);
 
 void ReadContextStack(void *context, uptr *stack, uptr *ssize);
+void WriteContextStack(void *context, uptr stack, uptr ssize);
 void StopInitOrderChecking();
 
 // Wrapper for TLS/TSD.
 void AsanTSDInit(void (*destructor)(void *tsd));
 void *AsanTSDGet();
 void AsanTSDSet(void *tsd);
 void PlatformTSDDtor(void *tsd);
 

lib/asan/asan_linux.cc

 #endif // SANITIZER_ANDROID
 
 #if !SANITIZER_ANDROID
 void ReadContextStack(void *context, uptr *stack, uptr *ssize) {
   ucontext_t *ucp = (ucontext_t*)context;
   *stack = (uptr)ucp->uc_stack.ss_sp;
   *ssize = ucp->uc_stack.ss_size;
 }
+
+void WriteContextStack(void *context, uptr stack, uptr ssize) {
+  ucontext_t *ucp = (ucontext_t*)context;
+  ucp->uc_stack.ss_sp = (void*)stack;
+  ucp->uc_stack.ss_size = ssize;
+}
 #else
 void ReadContextStack(void *context, uptr *stack, uptr *ssize) {
   UNIMPLEMENTED();
 }
+
+void WriteContextStack(void *context, uptr stack, uptr ssize) {
+  UNIMPLEMENTED();
+}
 #endif
 
 void *AsanDlSymNext(const char *sym) {
   return dlsym(RTLD_NEXT, sym);
 }
 
 } // namespace __asan
 
 #endif // SANITIZER_FREEBSD || SANITIZER_LINUX

lib/asan/asan_mac.cc

 
 // No-op. Mac does not support static linkage anyway.
 void AsanCheckIncompatibleRT() {}
 
 void ReadContextStack(void *context, uptr *stack, uptr *ssize) {
   UNIMPLEMENTED();
 }
 
+void WriteContextStack(void *context, uptr stack, uptr ssize) {
+  UNIMPLEMENTED();
+}
+
 // Support for the following functions from libdispatch on Mac OS:
 //   dispatch_async_f()
 //   dispatch_async()
 //   dispatch_sync_f()
 //   dispatch_sync()
 //   dispatch_after_f()
 //   dispatch_after()
 //   dispatch_group_async_f()

lib/asan/asan_thread.h

static AsanThread *Create(thread_callback_t start_routine, void *arg,
                             u32 parent_tid, StackTrace *stack, bool detached);
   static void TSDDtor(void *tsd);
   void Destroy();
 
   void Init();  // Should be called from the thread itself.
   thread_return_t ThreadStart(uptr os_id,
                               atomic_uintptr_t *signal_thread_is_registered);
 
-  uptr stack_top() { return stack_top_; }
-  uptr stack_bottom() { return stack_bottom_; }
-  uptr stack_size() { return stack_size_; }
+  struct StackDescriptor {
+    uptr stack_top;
+    uptr stack_bottom;
+    uptr stack_size;
+  };
+
+  uptr stack_top() { return CurrentStack()->stack_top; }
+  uptr stack_bottom() { return CurrentStack()->stack_bottom; }
+  uptr stack_size() { return CurrentStack()->stack_size; }
   uptr tls_begin() { return tls_begin_; }
   uptr tls_end() { return tls_end_; }
   DTLS *dtls() { return dtls_; }
   u32 tid() { return context_->tid; }
   AsanThreadContext *context() { return context_; }
   void set_context(AsanThreadContext *context) { context_ = context; }
 
+  void SetUserStack(uptr base, uptr size) {
+    temp_stack_->stack_bottom = base;
+    temp_stack_->stack_top = base + size;
+    temp_stack_->stack_size = size;
+
+    StackDescriptor* oprev = previous_stack_;
+    previous_stack_ = next_stack_;
+    next_stack_ = temp_stack_;
+    temp_stack_ = oprev;
+  }
+
   struct StackFrameAccess {
     uptr offset;
     uptr frame_pc;
     const char *frame_descr;
   };
   bool GetStackFrameAccessByAddr(uptr addr, StackFrameAccess *access);
 
+  bool AddrIsInStack(StackDescriptor *stack, uptr addr) {
+    return addr >= stack->stack_bottom && addr < stack->stack_top;
+  }
+
+  StackDescriptor *CurrentStack() {
+    int local;
+    if (AddrIsInStack(previous_stack_, (uptr)&local)) {
+        return previous_stack_;
+    } else {
+        return next_stack_;
+    }
+  }
+
   bool AddrIsInStack(uptr addr) {
-    return addr >= stack_bottom_ && addr < stack_top_;
+    return AddrIsInStack(CurrentStack(), addr);
   }
 
   void DeleteFakeStack(int tid) {
     if (!fake_stack_) return;
     FakeStack *t = fake_stack_;
     fake_stack_ = nullptr;
     SetTLSFakeStack(nullptr);
     t->Destroy(tid);
private:
   // via mmap() and *must* be valid in zero-initialized state.
   void SetThreadStackAndTls();
   void ClearShadowForThreadStackAndTLS();
   FakeStack *AsyncSignalSafeLazyInitFakeStack();
 
   AsanThreadContext *context_;
   thread_callback_t start_routine_;
   void *arg_;
-  uptr stack_top_;
-  uptr stack_bottom_;
-  // stack_size_ == stack_top_ - stack_bottom_;
-  // It needs to be set in a async-signal-safe manner.
-  uptr stack_size_;
+
+  // We have three stack descriptors for async-safe stack change. New stack
+  // information is written to temp_stack_. Then previous_stack_ is made to
+  // point to the same descriptor that next_stack_ does. Finally, temp_stack_
+  // is assigned to next_stack_. The result is that at any time either
+  // previous_stack_ or next_stack_ contain the correct stack information.
+  StackDescriptor stacks_[3];
+  StackDescriptor* temp_stack_;
+  StackDescriptor* next_stack_;
+  StackDescriptor* previous_stack_;
+
   uptr tls_begin_;
   uptr tls_end_;
   DTLS *dtls_;
 
   FakeStack *fake_stack_;
   AsanThreadLocalMallocStorage malloc_storage_;
   AsanStats stats_;
   bool unwinding_;

lib/asan/asan_thread.cc

if (atomic_compare_exchange_strong(
     fake_stack_ = FakeStack::Create(stack_size_log);
     SetTLSFakeStack(fake_stack_);
     return fake_stack_;
   }
   return nullptr;
 }
 
 void AsanThread::Init() {
+  temp_stack_ = &stacks_[0];
+  next_stack_ = &stacks_[1];
+  previous_stack_ = &stacks_[2];
   fake_stack_ = nullptr;  // Will be initialized lazily if needed.
   CHECK_EQ(this->stack_size(), 0U);
   SetThreadStackAndTls();
   CHECK_GT(this->stack_size(), 0U);
-  CHECK(AddrIsInMem(stack_bottom_));
-  CHECK(AddrIsInMem(stack_top_ - 1));
+  CHECK(AddrIsInMem(stack_bottom()));
+  CHECK(AddrIsInMem(stack_top() - 1));
   ClearShadowForThreadStackAndTLS();
   int local = 0;
   VReport(1, "T%d: stack [%p,%p) size 0x%zx; local=%p\n", tid(),
-          (void *)stack_bottom_, (void *)stack_top_, stack_top_ - stack_bottom_,
-          &local);
+          (void *)stack_bottom(), (void *)stack_top(),
+          stack_top() - stack_bottom(), &local);
 }
 
 thread_return_t AsanThread::ThreadStart(
     uptr os_id, atomic_uintptr_t *signal_thread_is_registered) {
   Init();
   asanThreadRegistry().StartThread(tid(), os_id, nullptr);
   if (signal_thread_is_registered)
     atomic_store(signal_thread_is_registered, 1, memory_order_release);
thread_return_t AsanThread::ThreadStart(
   if (!SANITIZER_POSIX)
     this->Destroy();
 
   return res;
 }
 
 void AsanThread::SetThreadStackAndTls() {
   uptr tls_size = 0;
-  GetThreadStackAndTls(tid() == 0, &stack_bottom_, &stack_size_, &tls_begin_,
-                       &tls_size);
-  stack_top_ = stack_bottom_ + stack_size_;
+  GetThreadStackAndTls(tid() == 0, &next_stack_->stack_bottom,
+                       &next_stack_->stack_size, &tls_begin_, &tls_size);
+  next_stack_->stack_top = next_stack_->stack_bottom + next_stack_->stack_size;
+  previous_stack_->stack_top = previous_stack_->stack_bottom = 0;
+  previous_stack_->stack_size = 0;
   tls_end_ = tls_begin_ + tls_size;
   dtls_ = DTLS_Get();
 
   int local;
   CHECK(AddrIsInStack((uptr)&local));
 }
 
 void AsanThread::ClearShadowForThreadStackAndTLS() {
-  PoisonShadow(stack_bottom_, stack_top_ - stack_bottom_, 0);
+  PoisonShadow(stack_bottom(), stack_top() - stack_bottom(), 0);
   if (tls_begin_ != tls_end_)
     PoisonShadow(tls_begin_, tls_end_ - tls_begin_, 0);
 }
 
 bool AsanThread::GetStackFrameAccessByAddr(uptr addr,
                                            StackFrameAccess *access) {
   uptr bottom = 0;
   if (AddrIsInStack(addr)) {

lib/asan/asan_win.cc

 void AsanCheckDynamicRTPrereqs() {}
 
 void AsanCheckIncompatibleRT() {}
 
 void ReadContextStack(void *context, uptr *stack, uptr *ssize) {
   UNIMPLEMENTED();
 }
 
+void WriteContextStack(void *context, uptr stack, uptr ssize) {
+  UNIMPLEMENTED();
+}
+
 void AsanOnDeadlySignal(int, void *siginfo, void *context) {
   UNIMPLEMENTED();
 }
 
 static LPTOP_LEVEL_EXCEPTION_FILTER default_seh_handler;
 
 static long WINAPI SEHHandler(EXCEPTION_POINTERS *info) {
   EXCEPTION_RECORD *exception_record = info->ExceptionRecord;

test/asan/TestCases/Linux/swapcontext_test.cc

 #include <ucontext.h>
 #include <unistd.h>
 
 ucontext_t orig_context;
 ucontext_t child_context;
 
 const int kStackSize = 1 << 20;
 
+volatile int force_write;
+
 __attribute__((noinline))
 void Throw() {
+  char buf[1024];
+  for (int i = 0; i < 1024; i++) {
+    buf[i] = force_write;
+  }
   throw 1;
 }
 
 __attribute__((noinline))
+void Func() {
+  int buf[4 * 1024];
+  for (int i = 0; i < 4 * 1024; i++) {
+    buf[i] = force_write;
+  }
+}
+
+__attribute__((noinline))
 void ThrowAndCatch() {
   try {
     Throw();
   } catch(int a) {
+    Func();
     printf("ThrowAndCatch: %d\n", a);
   }
 }
 
 void Child(int mode) {
   char x[32] = {0};  // Stack gets poisoned.
   printf("Child: %p\n", x);
   ThrowAndCatch();  // Simulate __asan_handle_no_return().

test/lsan/TestCases/swapcontext.cc

-// We can't unwind stack if we're running coroutines on heap-allocated
-// memory. Make sure we don't report these leaks.
-
 // RUN: %clangxx_lsan %s -o %t
-// RUN: %run %t 2>&1
 // RUN: not %run %t foo 2>&1 | FileCheck %s
 
 #include <stdio.h>
 #if defined(__APPLE__)
 // Note: ucontext.h is deprecated on OSX, so this test may stop working
 // someday. We define _XOPEN_SOURCE to keep using ucontext.h for now.
 #define _XOPEN_SOURCE 1
 #endif