[esan] Intercept calloc to avoid deadlocks with tcmalloc

[bruening]

Summary

When tcmalloc initializes before esan, esan's initialization ends up
calling back into tcmalloc due to the calloc done by dlsym. This results
in a deadlock. We avoid this by special-casing this single allocation.

Intercepting calloc also gives us the opportunity to act on its zeroing as
stores by the application.

[aizatsky]

lib/esan/esan_interceptors.cpp

431

Is there a multi-threading concern here? I assume tcmalloc initializes in one thread only?

450

Do you want to clear used_early_alloc_buf here?

[bruening]

lib/esan/esan_interceptors.cpp

431

This will only happen at process init prior to __esan_init, but I suppose it's possible for some shared library to create early threads that all race here. Given the fragility of sanitizer libc interceptors in general I'd have to say that's not high on my list of corner cases to worry about. Are you ok with a comment saying we simply don't handle it?

450

Sure, for cleanliness, though we do not expect it to matter (we expect a single call to calloc during dlsym prior to REAL(calloc) being set and after that to never need the buffer again).

[bruening]

Clear the used flag, and comment on races.

lib/esan/esan_interceptors.cpp

 } // namespace __sanitizer
 
 #define ESAN_MAYBE_INTERCEPT_SIGACTION INTERCEPT_FUNCTION(sigaction)
 #else
 #error Platform not supported
 #define ESAN_MAYBE_INTERCEPT_SIGACTION
 #endif
 
+//===----------------------------------------------------------------------===//
+// Malloc interceptors
+//===----------------------------------------------------------------------===//
+
+static char early_alloc_buf[128];
+static bool used_early_alloc_buf;
+
+static void *handleEarlyAlloc(uptr size) {
+  // If esan is initialized during an interceptor (which happens with some
+  // tcmalloc implementations that call pthread_mutex_lock), the call from
+  // dlsym to calloc will deadlock.  There is only one such calloc (dlsym
+  // allocates a single pthread key), so we work around it by using a
+  // static buffer for the calloc request.  The loader currently needs
+  // 32 bytes but we size at 128 to allow for future changes.
+  // This solution will also allow us to deliberately intercept malloc & family
+  // in the future (to perform tool actions on each allocation, without
+  // replacing the allocator), as it also solves the problem of intercepting
+  // calloc when it will itself be called before its REAL pointer is
+  // initialized.
+  CHECK(!used_early_alloc_buf && size < sizeof(early_alloc_buf));
+  used_early_alloc_buf = true;
+  return (void *)early_alloc_buf;
+}
+
+INTERCEPTOR(void*, calloc, uptr size, uptr n) {
+  if (EsanDuringInit && REAL(calloc) == nullptr)
+    return handleEarlyAlloc(size * n);
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, calloc, size, n);
+  void *res = REAL(calloc)(size, n);
+  // The memory is zeroed and thus is all written.
+  COMMON_INTERCEPTOR_WRITE_RANGE(nullptr, (uptr)res, size * n);
+  return res;
+}
+
+INTERCEPTOR(void, free, void *p) {
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, free, p);
+  if (p == (void *)early_alloc_buf)
+    return;
+  REAL(free)(p);
+}
+
 namespace __esan {
 
 void initializeInterceptors() {
   InitializeCommonInterceptors();
 
   INTERCEPT_FUNCTION(strcpy); // NOLINT
   INTERCEPT_FUNCTION(strncpy);
 
   INTERCEPT_FUNCTION(open);
   ESAN_MAYBE_INTERCEPT_OPEN64;
   INTERCEPT_FUNCTION(creat);
   ESAN_MAYBE_INTERCEPT_CREAT64;
   INTERCEPT_FUNCTION(unlink);
   INTERCEPT_FUNCTION(fread);
   INTERCEPT_FUNCTION(fwrite);
   INTERCEPT_FUNCTION(puts);
   INTERCEPT_FUNCTION(rmdir);
 
   INTERCEPT_FUNCTION(mmap);
   ESAN_MAYBE_INTERCEPT_MMAP64;
 
   ESAN_MAYBE_INTERCEPT_SIGNAL;
   ESAN_MAYBE_INTERCEPT_SIGACTION;
 
-  // TODO(bruening): we should intercept calloc() and other memory allocation
-  // routines that zero memory and update our shadow memory appropriately.
+  INTERCEPT_FUNCTION(calloc);
+  INTERCEPT_FUNCTION(free);
 
   // TODO(bruening): intercept routines that other sanitizers intercept that
   // are not in the common pool or here yet, ideally by adding to the common
   // pool.  Examples include wcslen and bcopy.
 
   // TODO(bruening): there are many more libc routines that read or write data
   // structures that no sanitizer is intercepting: sigaction, strtol, etc.
 }
 
 } // namespace __esan