asan: handle SIGABRT

[tlipcon]

Summary

commit 0eb8067bfe9ee4848df03469ccd1a8f9364df4a5
Author: Todd Lipcon <[e-mail]>
Date: Wed Jul 10 15:34:15 2013 -0700

asan: handle SIGABRT

On some kernel versions, if /proc/sys/vm/core_pattern is set to
a pipe, then the setrlimit(RLIMIT_CORE) call to disable core
dumps has no effect. In that case, if the user application calls
abort(), then the uncaught SIGABRT triggers a core dump, resulting
in a multi-TB pipe into the core dump handler (blocking the process
for many hours).

This patch adds a handler for SIGABRT which logs the error, then
unmaps the shadow memory before propagating the signal. This allows
a core dump to be processed reasonably even on 64-bit systems.

Manually tested with an application which calls "abort()". On systems
set up as described above, this used to hang for hours as it tried
to send the large core dump through the pipe. With the patch, it
outputs:

$ ./coredump
ASAN:SIGABRT
==28640==ERROR: AddressSanitizer: ABRT (pc 0x7f8632666425 sp 0x7fff0cae8f08 bp 0x7fff0cae91d0 T0)
    #0 0x7f8632666424 (/lib/x86_64-linux-gnu/libc-2.15.so+0x36424)
    #1 0x7f8632669b8a (/lib/x86_64-linux-gnu/libc-2.15.so+0x39b8a)
    #2 0x42b11e (/tmp/coredump+0x42b11e)
    #3 0x7f863265176c (/lib/x86_64-linux-gnu/libc-2.15.so+0x2176c)
    #4 0x42ae7c (/tmp/coredump+0x42ae7c)
Aborted (core dumped)

[glider]

lib/asan/asan_posix.cc

90

Once you unmap the shadow mappings, it's unsafe to invoke the default signal handler, because the instrumented code will try to access the shadow memory.

[samsonov]

I'd rather avoid dealing with SIGABRT in ASan. Is there really no way to limit the size of the core dump on the systems you mention?

lib/asan/asan_report.cc

526

No, generally ASan should fail the program immediately after it prints the error report - that's why all "ReportFoo" functions are noreturn and we have ScopedInErrorReport object. Among the other things, it makes sure that no two error reports are printed simultaneously, otherwise bad things may happen - the code in "ReportFoo" functions is thread-unsafe.

lib/asan/asan_rtl.cc

151

Why? I don't think SIGABRT handler should be on by default.

[tlipcon]

lib/asan/asan_posix.cc

90

The default signal handler, though, is in the kernel -- not user code. So, it won't run instrumented code, it'll just generate a core dump and exit with the right return code. Right?

I'm explicitly setting to SIG_DFL, not restoring whatever signal handler the user might have installed.

lib/asan/asan_report.cc

526

The issue is that if you die with "Die()" then you don't get the core dump at all. I'd like to actually get core dumps on SEGV/ABRT so I can debug things, but I want the core dumps to be _without_ the shadow mapping.

With this patch, you can get this behavior by turning on abort_on_error -- the flow is:

• user code accesses bad memory
  • ReportSEGV generates the report
  • Die() calls abort(), generating a SIGABRT
    • HandleSIGABRT gets called, reports SIGABRT, unmaps memory
    • HandleSIGABRT re-kills self with SIGABRT after setting to SIG_DFL
      • kernel generates core dump (without shadow mappings)
• user happily debugs their application from the corefile

As for ensuring thread-safety, I put that check in the signal handler -- it uses an atomic swap to make sure that only one thread will proceed into this code

lib/asan/asan_rtl.cc

151

Why not? Unmapping the shadow memory on abort makes corefiles possible on 64-bit (and could enable us to then get rid of the other code which sets RLIMIT_CORE->0 on those systems)

My general use case is that I have an integration test build which runs all of my tests with ASAN enabled. Sometimes my tests fail due to a race which is difficult to reproduce, but a corefile would allow us to debug it the next morning. Currently it's impossible to get corefiles out of this build, but with this patch we're able to.

[samsonov]

Sorry for the delayed response.

lib/asan/asan_posix.cc

62

Looks like you don't "invoke" default handler, you just install it here.

90

Ok... So you mean that after ASAN_OnSIGABRT exits, default signal handler will be called, and no more user code will be executed?

lib/asan/asan_report.cc

526

Hm... but Die() already unmaps shadow at exit, doesn't it? Why do you need to do this in SIGABRT handler? Will the following strategy work?

• make SIGABRT and SIGSEGV handlers identical: print an error report, optionally set the handler to SIG_DFL, and call Die()
• Die() optionally unmaps the shadow memory
• Die() optionally calls abort(), which goes to kernel and generates a core dump.

Your atomic swap doesn't prevent two threads from entering ReportSIGABRT and ReportFreeNotMalloced simultaneously. The code in these functions is not thread-safe, for example they print symbolized stack traces, and symbolization is single-threaded.

lib/asan/asan_rtl.cc

151

Your use case is legitimate, but we generally introduce new options under a flag. We need to make sure it won't break the existing users, especially if they were able to live w/o this capability.

See my comment above about generating core dumps.

lib/asan/asan_flags.h

struct Flags {
   // If set, user may manually mark memory regions as poisoned or unpoisoned.
   bool allow_user_poisoning;
   // Number of seconds to sleep between printing an error report and
   // terminating application. Useful for debug purposes (when one needs
   // to attach gdb, for example).
   int  sleep_before_dying;
   // If set, registers ASan custom segv handler.
   bool handle_segv;
+  // If set, registers ASan custom abrt handler.
+  bool handle_abrt;
   // If set, allows user register segv handler even if ASan registers one.
   bool allow_user_segv_handler;
   // If set, uses alternate stack for signal handling.
   bool use_sigaltstack;
   // Allow the users to work around the bug in Nvidia drivers prior to 295.*.
   bool check_malloc_usable_size;
   // If set, explicitly unmaps (huge) shadow at exit.
   bool unmap_shadow_on_exit;

lib/asan/asan_linux.cc

# else
   *bp = stk_ptr[15];
 # endif
 #else
 # error "Unsupported arch"
 #endif
 }
 
 bool AsanInterceptsSignal(int signum) {
-  return signum == SIGSEGV && flags()->handle_segv;
+  return (signum == SIGSEGV && flags()->handle_segv) ||
+         (signum == SIGABRT && flags()->handle_abrt);
 }
 
 void AsanPlatformThreadInit() {
   // Nothing here for now.
 }
 
 void GetStackTrace(StackTrace *stack, uptr max_s, uptr pc, uptr bp, bool fast) {
 #if defined(__arm__) || \

lib/asan/asan_mapping.h

static inline bool AddressIsPoisoned(uptr a) {
   if (shadow_value) {
     u8 last_accessed_byte = (a & (SHADOW_GRANULARITY - 1))
                                  + kAccessSize - 1;
     return (last_accessed_byte >= shadow_value);
   }
   return false;
 }
 
+static inline void UnmapShadowMappings() {
+  if (kMidMemBeg) {
+    UnmapOrDie((void*)kLowShadowBeg, kMidMemBeg - kLowShadowBeg);
+    UnmapOrDie((void*)kMidMemEnd, kHighShadowEnd - kMidMemEnd);
+  } else {
+    UnmapOrDie((void*)kLowShadowBeg, kHighShadowEnd - kLowShadowBeg);
+  }
+}
+
 // Must be after all calls to PROFILE_ASAN_MAPPING().
 static const uptr kAsanMappingProfileSize = __LINE__;
 
 }  // namespace __asan
 
 #endif  // ASAN_MAPPING_H

lib/asan/asan_posix.cc

static void     ASAN_OnSIGSEGV(int, siginfo_t *siginfo, void *context) {
   uptr addr = (uptr)siginfo->si_addr;
   // Write the first message using the bullet-proof write.
   if (13 != internal_write(2, "ASAN:SIGSEGV\n", 13)) Die();
   uptr pc, sp, bp;
   GetPcSpBp(context, &pc, &sp, &bp);
   ReportSIGSEGV(pc, sp, bp, addr);
 }
 
+// Invoke the default signal handler.
+void InvokeDefaultSignalHandler(int signal_number) {
+  struct sigaction sig_action;
+  REAL(memset)(&sig_action, 0, sizeof(sig_action));
+  sigemptyset(&sig_action.sa_mask);
+  sig_action.sa_handler = SIG_DFL;
+  if (flags()->use_sigaltstack) sig_action.sa_flags |= SA_ONSTACK;
+  CHECK_EQ(0, REAL(sigaction)(signal_number, &sig_action, NULL));
+}
+
+static void ASAN_OnSIGABRT(int, siginfo_t *siginfo, void *context) {
+  static atomic_uint32_t num_calls;
+  if (atomic_fetch_add(&num_calls, 1, memory_order_relaxed) != 0) {
+    Report("Reentrant call into SIGABRT handler");
+    // we may end up with a reentrant call if the munmap() calls
+    // below fail. In that case, we should ignore the reentrant
+    // failure since we're in the process of exiting anyway.
+    return;
+  }
+
+  // Write the first message using the bullet-proof write.
+  if (13 != internal_write(2, "ASAN:SIGABRT\n", 13)) Die();
+  uptr pc, sp, bp;
+  GetPcSpBp(context, &pc, &sp, &bp);
+  ReportSIGABRT(pc, sp, bp);
+
+  // Unmap the shadow pages so that a core dump is reasonable,
+  // then call through to the original SIGABRT handler
+  UnmapShadowMappings();
+  InvokeDefaultSignalHandler(SIGABRT);
+}
+
 void SetAlternateSignalStack() {
   stack_t altstack, oldstack;
   CHECK_EQ(0, sigaltstack(0, &oldstack));
   // If the alternate stack is already in place, do nothing.
   if ((oldstack.ss_flags & SS_DISABLE) == 0) return;
   // TODO(glider): the mapped stack should have the MAP_STACK flag in the
   // future. It is not required by man 2 sigaltstack now (they're using
   // malloc()).

 
 void InstallSignalHandlers() {
   // Set the alternate signal stack for the main thread.
   // This will cause SetAlternateSignalStack to be called twice, but the stack
   // will be actually set only once.
   if (flags()->use_sigaltstack) SetAlternateSignalStack();
   MaybeInstallSigaction(SIGSEGV, ASAN_OnSIGSEGV);
   MaybeInstallSigaction(SIGBUS, ASAN_OnSIGSEGV);
+  MaybeInstallSigaction(SIGABRT, ASAN_OnSIGABRT);
 }
 
 // ---------------------- TSD ---------------- {{{1
 
 static pthread_key_t tsd_key;
 static bool tsd_key_inited = false;
 void AsanTSDInit(void (*destructor)(void *tsd)) {
   CHECK(!tsd_key_inited);

lib/asan/asan_report.h

 bool DescribeAddressIfStack(uptr addr, uptr access_size);
 // Determines memory type on its own.
 void DescribeAddress(uptr addr, uptr access_size);
 
 void DescribeThread(AsanThreadContext *context);
 
 // Different kinds of error reports.
 void NORETURN ReportSIGSEGV(uptr pc, uptr sp, uptr bp, uptr addr);
+void ReportSIGABRT(uptr pc, uptr sp, uptr bp);
 void NORETURN ReportDoubleFree(uptr addr, StackTrace *stack);
 void NORETURN ReportFreeNotMalloced(uptr addr, StackTrace *stack);
 void NORETURN ReportAllocTypeMismatch(uptr addr, StackTrace *stack,
                                       AllocType alloc_type,
                                       AllocType dealloc_type);
 void NORETURN ReportMallocUsableSizeNotOwned(uptr addr,
                                              StackTrace *stack);
 void NORETURN ReportAsanGetAllocatedSizeNotOwned(uptr addr,

lib/asan/asan_report.cc

Report("ERROR: AddressSanitizer: SEGV on unknown address %p"
              GetCurrentTidOrInvalid());
   Printf("%s", d.EndWarning());
   Printf("AddressSanitizer can not provide additional info.\n");
   GET_STACK_TRACE_FATAL(pc, bp);
   PrintStack(&stack);
   ReportSummary("SEGV", &stack);
 }
 
+void ReportSIGABRT(uptr pc, uptr sp, uptr bp) {
+  Decorator d;
+  Printf("%s", d.Warning());
+  Report("ERROR: AddressSanitizer: ABRT"
+             " (pc %p sp %p bp %p T%d)\n",
+             (void*)pc, (void*)sp, (void*)bp,
+             GetCurrentTidOrInvalid());
+  Printf("%s", d.EndWarning());
+  GET_STACK_TRACE_FATAL(pc, bp);
+  PrintStack(&stack);
+  ReportSummary("ABRT", &stack);
+}
+
 void ReportDoubleFree(uptr addr, StackTrace *stack) {
   ScopedInErrorReport in_report;
   Decorator d;
   Printf("%s", d.Warning());
   char tname[128];
   u32 curr_tid = GetCurrentTidOrInvalid();
   Report("ERROR: AddressSanitizer: attempting double-free on %p in "
          "thread T%d%s:\n",

lib/asan/asan_rtl.cc

if (atomic_fetch_add(&num_calls, 1, memory_order_relaxed) != 0) {
     // Don't die twice - run a busy loop.
     while (1) { }
   }
   if (flags()->sleep_before_dying) {
     Report("Sleeping for %d second(s)\n", flags()->sleep_before_dying);
     SleepForSeconds(flags()->sleep_before_dying);
   }
   if (flags()->unmap_shadow_on_exit) {
-    if (kMidMemBeg) {
-      UnmapOrDie((void*)kLowShadowBeg, kMidMemBeg - kLowShadowBeg);
-      UnmapOrDie((void*)kMidMemEnd, kHighShadowEnd - kMidMemEnd);
-    } else {
-      UnmapOrDie((void*)kLowShadowBeg, kHighShadowEnd - kLowShadowBeg);
-    }
+    UnmapShadowMappings();
   }
   if (death_callback)
     death_callback();
   if (flags()->abort_on_error)
     Abort();
   internal__exit(flags()->exitcode);
 }
 
static void ParseFlagsFromString(Flags *f, const char *str) {
   ParseFlag(str, &f->mac_ignore_invalid_free, "mac_ignore_invalid_free");
   ParseFlag(str, &f->use_fake_stack, "use_fake_stack");
   ParseFlag(str, &f->max_malloc_fill_size, "max_malloc_fill_size");
   ParseFlag(str, &f->malloc_fill_byte, "malloc_fill_byte");
   ParseFlag(str, &f->exitcode, "exitcode");
   ParseFlag(str, &f->allow_user_poisoning, "allow_user_poisoning");
   ParseFlag(str, &f->sleep_before_dying, "sleep_before_dying");
   ParseFlag(str, &f->handle_segv, "handle_segv");
+  ParseFlag(str, &f->handle_abrt, "handle_abrt");
   ParseFlag(str, &f->allow_user_segv_handler, "allow_user_segv_handler");
   ParseFlag(str, &f->use_sigaltstack, "use_sigaltstack");
   ParseFlag(str, &f->check_malloc_usable_size, "check_malloc_usable_size");
   ParseFlag(str, &f->unmap_shadow_on_exit, "unmap_shadow_on_exit");
   ParseFlag(str, &f->abort_on_error, "abort_on_error");
   ParseFlag(str, &f->print_stats, "print_stats");
   ParseFlag(str, &f->print_legend, "print_legend");
   ParseFlag(str, &f->atexit, "atexit");
void InitializeFlags(Flags *f, const char *env) {
   f->mac_ignore_invalid_free = false;
   f->use_fake_stack = true;
   f->max_malloc_fill_size = 0x1000;  // By default, fill only the first 4K.
   f->malloc_fill_byte = 0xbe;
   f->exitcode = ASAN_DEFAULT_FAILURE_EXITCODE;
   f->allow_user_poisoning = true;
   f->sleep_before_dying = 0;
   f->handle_segv = ASAN_NEEDS_SEGV;
+  f->handle_abrt = ASAN_NEEDS_SEGV;
   f->allow_user_segv_handler = false;
   f->use_sigaltstack = false;
   f->check_malloc_usable_size = true;
   f->unmap_shadow_on_exit = false;
   f->abort_on_error = false;
   f->print_stats = false;
   f->print_legend = true;
   f->atexit = false;