[scudo] Android support groundwork

[cryptoad]

Summary

This makes the required changes to the Scudo code to be able to work on
Android, both for ARM and AArch64. Note that Android is not enabled yet in
the cmake configs, it will be the object of a later commit.

The following constraints had to be taken into account:

• Android doesn't support thread_local, so things had to be changed to take this into account, using pthread_{set,get}specific as the alternative;
• Android can't necessarily afford a cache per thread, due to memory limitations (there are incentives to low memort devices). The current Android allocator, jemalloc, uses a fixed set of arenas, which is the solution I adopted as well, assigning them to threads in a round-robin fashion.

As a result, the thread specific data was moved into a context structure, that
can either be thread_local or global (in this case, allocated during
initialization and lasting the lifetime of the program). This accounts for the
new files: scudo_thread.h, scudo_thread_linux.cpp, scudo_thread_android.cpp.
The number of contexts for Android is modifiable as a define. This also means
that we must be able to tell if a thread cache requires locking now, as for
Android it can be shared between multiple threads. Hence the addition of a
mutex per context, that is locked only when dealing with non thread_local
scenarios.

Another change is the ability to bypass the Quarantine if the cache size is
set to 0. Android watches closely its PSS, and 1MB of Quarantine ends up in
about 6MB of extra PSS for Android 32-bit. So the Quarantine might not be an
option. With no quarantine, the old execution flow ended up with the Scudo
header being checked twice, once before Put, once in Recyle, without any added
value to it.

Other changes involve a bit of reordering of lines here and there, and some
variable renaming to make things a bit more explicit.

For this review, I would also appreciate potential optimization feedback, if
someone sees a way to do things better, that would be great! Thanks for reading
that long blob of text.

[alekseyshl]

lib/scudo/scudo_allocator.cpp

176

Does UNLIKELY really improve the generated code?

[cryptoad]

lib/scudo/scudo_allocator.cpp

176

I do like it as it will make the error condition the target of the conditional jump, and the rest of the code follows the regular flow.
For example for that particular check, on x64:

.text:0000000000053B72 E8 99 9A 00 00                    call    _ZN7__scudo20computeHardwareCRC32Ejm ; __scudo::computeHardwareCRC32(uint,ulong)
.text:0000000000053B77 4C 8B 44 24 08                    mov     r8, [rsp+78h+var_70]
.text:0000000000053B7C 66 41 39 C0                       cmp     r8w, ax
.text:0000000000053B80 0F 85 CA 02 00 00                 jnz     loc_53E50       ; error
.text:0000000000053B86 49 C1 EC 20                       shr     r12, 20h

The error is the target of the long jump.
I changed all the die conditions to be UNLIKELY as if they are hit, we terminate anyway.
The performance gain is not necessarily significant, but the assembly does look cleaner.

[cryptoad]

I could use some more eyes and comments on this please!

[kcc]

Android doesn't support thread_local

eugenis@, Is that still true?
Anyway, would you like to look at this from the Android POV?

[srhines]

In D31947#728470, @kcc wrote:

Android doesn't support thread_local

eugenis@, Is that still true?
Anyway, would you like to look at this from the Android POV?

That isn't true. Android does support thread_local. This patch should not have a different Android path.

[cryptoad]

In D31947#728499, @srhines wrote:

In D31947#728470, @kcc wrote:

Android doesn't support thread_local

eugenis@, Is that still true?
Anyway, would you like to look at this from the Android POV?

That isn't true. Android does support thread_local. This patch should not have a different Android path.

To be more specific, thread_local is supported via emutls (llvm/projects/compiler-rt/lib/builtins/emutls.c), which itself calls malloc to allocate the TLS.
Which unfortunately doesn't work well for an allocator that replaces malloc.

[eugenis]

lib/scudo/scudo_allocator.cpp

317

Calling initThreadMaybe() from all entry points looks error-prone. Is it ok (performance-wise) to do lazy initialization in getCurrentThreadContext?

lib/scudo/scudo_thread.h

10

an actual description would be nice

[cryptoad]

lib/scudo/scudo_allocator.cpp

317

Things like BackendAllocator.ReturnNullOrDieOnBadRequest will require initialization, but are called prior to getCurrentThreadContext.
I am also actually not super happy about the way this turned out. The alternative would be a preinit or equivalent?

lib/scudo/scudo_thread.h

10

Sorry, user failure. I will address that.

[eugenis]

lib/scudo/scudo_allocator.cpp

317

Ideally access to the allocator cache should go through ThreadLocalContext as well. Maybe.

What do you mean by preinit?

[cryptoad]

lib/scudo/scudo_allocator.cpp

317

Access to the local cache & quarantine cache go through the thread context.
The issue I think is more about the global init, which is currently called via a pthread_once in the thread init.
So realistically, the global init doesn't happen until a thread has been initialized.
Unfortunately, that means all the options for the backend haven't been parsed yet if we wait to get the thread context (unless I move it up the code).
I was thinking like in ASan, where it could be initialized once and for all via the preinit_array?

[eugenis]

preinit_array can not be used in a shared library. A library can use -z initfirst, but we probably don't want to go there, and it's not supported on Android.

Let's leave it like this.

[dvyukov]

Android doesn't support thread_local

We have a (real) TLS slot for tsan (TLS_SLOT_TSAN). I wonder if we can re-use it for other sanitizers and scudo. Sanitizers are mutually exclusive, so there should be no problems using TLS_SLOT_TSAN is asan/msan. Scudo is not supposed to be used with sanitizers, right? So it seems we can just (ab)use TLS_SLOT_TSAN as well. Accessing it is just few machine instructions (see TLS_SLOT_TSAN in tsan_platform_linux.cc).

[rengolin]

In D31947#728503, @cryptoad wrote:

To be more specific, thread_local is supported via emutls (llvm/projects/compiler-rt/lib/builtins/emutls.c), which itself calls malloc to allocate the TLS.
Which unfortunately doesn't work well for an allocator that replaces malloc.

I thought we had fixed this one by asking the Android devs to implement proper TLS on Android. They said it was simple and wasn't there because no one needed it.

@srhines, were you at the meeting with @jasonk, @kcc and some Android devs at Google, maybe 2 years ago? We discussed how emutls could be completely avoided.

Or maybe I remember it wrong...

--renato

[dvyukov]

First of all, do you have any benchmarks? Single-threaded, multi-threaded with/without contention. How do profiles look like?

lib/scudo/scudo_allocator.cpp

317

Agree, this does not look good if we care about performance. We could use preinit_array (or some other android alternative), but we also need to init thread (do we?). So we could piggy-back on thread init check. Namely:

ThreadContext* GetThreadContext() {
   ThreadContext* ctx = __get_tls()[TLS_TSAN_SLOT];
   if (LIKELY(ctx))
     return ctx; // implies that thread and global context were initialized
   return GetThreadContextSlow();
}

ThreadContext* GetThreadContextSlow() {
   init global context once;
   create and init thread context;
   store it into TLS_TSAN_SLOT and return;
}

If we call GetThreadContext() once in each interface function and then explicitly propagate the object pointer, we don't do any excessive work at all.
This needs special care if some interface functions can be called from signal handlers.

lib/scudo/scudo_thread_android.cpp

35

Humm... is the idea that you build scudo for each particular device knowing number of cores?
We can get severe performance drop if 2 or more active threads share the same context. We need to avoid contention at all costs. I am thinking of the following scheme.

• obtain number of cores at runtime
• create NCORES (or 2*NCORES) contexts
• allow threads to dynamically re-associate with contexts to resolve contention

The re-association algorithm can be along the following lines. Remember the last used contenxt. Try to TryLock it. If TryLock succeeds, use it. If TryLock fails, try to find another unlocked context. If all contexts are locked, find the one that was unused for the longest time. The usage accounting can be done with only 1 additional check/store on fast path. Namely:

struct Context {
  Mutex mu;
  atomic_uint64_t unused_since;
  ...
};

bool Context::TryLock() {
  if (mu.TryLock()) {
    atomic_store_relaxed(&unused_since, 0);
    return true;
  }
  if (atomic_load_relaxed(&unused_since) == 0)
    atomic_store_relaxed(&unused_since, now());
  return false;
}

I think this should provide pretty good contention resolution with minimal cost and reasonable complexity.

[cryptoad]

In D31947#728975, @rengolin wrote:

I thought we had fixed this one by asking the Android devs to implement proper TLS on Android. They said it was simple and wasn't there because no one needed it.

I had a short talk yesterday with some Android Bionic people, and they said they would put it back on the to-do list, but without a commitment as to when it would land.

[cryptoad]

In D31947#728956, @dvyukov wrote:

We have a (real) TLS slot for tsan (TLS_SLOT_TSAN). I wonder if we can re-use it for other sanitizers and scudo. Sanitizers are mutually exclusive, so there should be no problems using TLS_SLOT_TSAN is asan/msan. Scudo is not supposed to be used with sanitizers, right? So it seems we can just (ab)use TLS_SLOT_TSAN as well. Accessing it is just few machine instructions (see TLS_SLOT_TSAN in tsan_platform_linux.cc).

Scudo indeed is mutually exclusive with the other Sanitizers, so this option sounds like it could work!

[cryptoad]

In D31947#729047, @dvyukov wrote:

First of all, do you have any benchmarks? Single-threaded, multi-threaded with/without contention. How do profiles look like?

I have shared with Aleksey some Flame profiles, I can send that to you as well. This ended up with the couple of optimizations he recently did.

[dvyukov]

In D31947#729227, @cryptoad wrote:

In D31947#729047, @dvyukov wrote:

First of all, do you have any benchmarks? Single-threaded, multi-threaded with/without contention. How do profiles look like?

I have shared with Aleksey some Flame profiles, I can send that to you as well. This ended up with the couple of optimizations he recently did.

Yes, please.

[srhines]

In D31947#728975, @rengolin wrote:

In D31947#728503, @cryptoad wrote:

To be more specific, thread_local is supported via emutls (llvm/projects/compiler-rt/lib/builtins/emutls.c), which itself calls malloc to allocate the TLS.
Which unfortunately doesn't work well for an allocator that replaces malloc.

I thought we had fixed this one by asking the Android devs to implement proper TLS on Android. They said it was simple and wasn't there because no one needed it.

@srhines, were you at the meeting with @jasonk, @kcc and some Android devs at Google, maybe 2 years ago? We discussed how emutls could be completely avoided.

It's unfortunately been too long. There was no progress there IIRC, so we didn't continue pursuing anything related to it. The bionic folks were also concerned about the direction that most of the patches were taking. Yabin added TSan support, but it isn't widely used in Android (it remains harder to set up than ASan).

Or maybe I remember it wrong...

--renato

[cryptoad]

Added descriptions to scudo_thread*.
One of the additional change that didn't make it to the initial commit was
to move the software CRC32 into scudo_utils.h for inlining purposes.

[cryptoad]

@dvyukov would it be acceptable to get this patch out in its current form and use this as a base for the further improvement you suggested?
Namely: improve contention, potential use of the tsan TLS slot, rework thread/global initialization.

[dvyukov]

In D31947#730656, @cryptoad wrote:

@dvyukov would it be acceptable to get this patch out in its current form and use this as a base for the further improvement you suggested?
Namely: improve contention, potential use of the tsan TLS slot, rework thread/global initialization.

Is there a pressing need?
I think the main problem with this change is that it's very large. You can get both if you split it. For example:

1. Minor, unrelated changes/refactorings.
2. Preparation for Android support, but no actual Android code (only Linux as now).
3. Rework initialization (but again no Android code).
4. Actual Android support (will not need to touch any common code, just provide platform-specific parts).

[cryptoad]

In D31947#730825, @dvyukov wrote:

In D31947#730656, @cryptoad wrote:

@dvyukov would it be acceptable to get this patch out in its current form and use this as a base for the further improvement you suggested?
Namely: improve contention, potential use of the tsan TLS slot, rework thread/global initialization.

Is there a pressing need?
I think the main problem with this change is that it's very large. You can get both if you split it. For example:

1. Minor, unrelated changes/refactorings.
2. Preparation for Android support, but no actual Android code (only Linux as now).
3. Rework initialization (but again no Android code).
4. Actual Android support (will not need to touch any common code, just provide platform-specific parts).

Sounds good.

lib/scudo/CMakeLists.txt

 add_compiler_rt_component(scudo)
 
 include_directories(..)
 
 set(SCUDO_CFLAGS ${SANITIZER_COMMON_CFLAGS})
 # SANITIZER_COMMON_CFLAGS include -fno-builtin, but we actually want builtins!
 list(APPEND SCUDO_CFLAGS -fbuiltin)
 append_rtti_flag(OFF SCUDO_CFLAGS)
 
 set(SCUDO_SOURCES
   scudo_allocator.cpp
   scudo_flags.cpp
   scudo_crc32.cpp
   scudo_interceptors.cpp
   scudo_new_delete.cpp
   scudo_termination.cpp
+  scudo_thread_android.cpp
+  scudo_thread_linux.cpp
   scudo_utils.cpp)
 
 # Enable the SSE 4.2 instruction set for scudo_crc32.cpp, if available.
 if (COMPILER_RT_HAS_MSSE4_2_FLAG)
   set_source_files_properties(scudo_crc32.cpp PROPERTIES COMPILE_FLAGS -msse4.2)
 endif()
 
 # Enable the AArch64 CRC32 feature for scudo_crc32.cpp, if available.

lib/scudo/scudo_allocator.h

struct AllocatorOptions {
   bool DeallocationTypeMismatch;
   bool DeleteSizeMismatch;
   bool ZeroContents;
 
   void setFrom(const Flags *f, const CommonFlags *cf);
   void copyTo(Flags *f, CommonFlags *cf) const;
 };
 
+#if SANITIZER_CAN_USE_ALLOCATOR64
+const uptr AllocatorSpace = ~0ULL;
+# if defined(__aarch64__) && SANITIZER_ANDROID
+const uptr AllocatorSize = 0x4000000000ULL;  // 256G.
+typedef DefaultSizeClassMap SizeClassMap;
+# elif defined(__aarch64__)
+// AArch64/SANITIZER_CAN_USER_ALLOCATOR64 is only for 42-bit VMA
+// so no need to different values for different VMA.
+const uptr kAllocatorSize  =  0x10000000000ULL;  // 1T.
+typedef DefaultSizeClassMap SizeClassMap;
+# else
+const uptr AllocatorSize = 0x40000000000ULL;  // 4T.
+typedef DefaultSizeClassMap SizeClassMap;
+# endif
+struct AP {
+  static const uptr kSpaceBeg = AllocatorSpace;
+  static const uptr kSpaceSize = AllocatorSize;
+  static const uptr kMetadataSize = 0;
+  typedef __scudo::SizeClassMap SizeClassMap;
+  typedef NoOpMapUnmapCallback MapUnmapCallback;
+  static const uptr kFlags =
+      SizeClassAllocator64FlagMasks::kRandomShuffleChunks;
+};
+typedef SizeClassAllocator64<AP> PrimaryAllocator;
+#else
+// Currently, the 32-bit Sanitizer allocator has not yet benefited from all the
+// security improvements brought to the 64-bit one. This makes the 32-bit
+// version of Scudo slightly less toughened.
+static const uptr RegionSizeLog = 20;
+static const uptr NumRegions = SANITIZER_MMAP_RANGE_SIZE >> RegionSizeLog;
+# if SANITIZER_WORDSIZE == 32
+typedef FlatByteMap<NumRegions> ByteMap;
+# elif SANITIZER_WORDSIZE == 64
+typedef TwoLevelByteMap<(NumRegions >> 12), 1 << 12> ByteMap;
+# endif  // SANITIZER_WORDSIZE
+typedef DefaultSizeClassMap SizeClassMap;
+typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, 0, SizeClassMap,
+    RegionSizeLog, ByteMap> PrimaryAllocator;
+#endif  // SANITIZER_CAN_USE_ALLOCATOR64
+
+#include "scudo_allocator_secondary.h"
+
+typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache;
+typedef ScudoLargeMmapAllocator SecondaryAllocator;
+typedef CombinedAllocator<PrimaryAllocator, AllocatorCache, SecondaryAllocator>
+  ScudoBackendAllocator;
+
+ScudoBackendAllocator &getBackendAllocator();
+
+void initScudo();
 void initAllocator(const AllocatorOptions &options);
-void drainQuarantine();
 
 void *scudoMalloc(uptr Size, AllocType Type);
 void scudoFree(void *Ptr, AllocType Type);
 void scudoSizedFree(void *Ptr, uptr Size, AllocType Type);
 void *scudoRealloc(void *Ptr, uptr Size);
 void *scudoCalloc(uptr NMemB, uptr Size);
 void *scudoMemalign(uptr Alignment, uptr Size);
 void *scudoValloc(uptr Size);
 void *scudoPvalloc(uptr Size);
 int scudoPosixMemalign(void **MemPtr, uptr Alignment, uptr Size);
 void *scudoAlignedAlloc(uptr Alignment, uptr Size);
 uptr scudoMallocUsableSize(void *Ptr);
 
-#include "scudo_allocator_secondary.h"
-
 }  // namespace __scudo
 
 #endif  // SCUDO_ALLOCATOR_H_

lib/scudo/scudo_allocator.cpp

 /// Scudo Hardened Allocator implementation.
 /// It uses the sanitizer_common allocator as a base and aims at mitigating
 /// heap corruption vulnerabilities. It provides a checksum-guarded chunk
 /// header, a delayed free list, and additional sanity checks.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "scudo_allocator.h"
+#include "scudo_thread.h"
 #include "scudo_utils.h"
 
 #include "sanitizer_common/sanitizer_allocator_interface.h"
 #include "sanitizer_common/sanitizer_quarantine.h"
 
-#include <limits.h>
-#include <pthread.h>
-
-#include <cstring>
+#include <string.h>
 
 namespace __scudo {
 
-#if SANITIZER_CAN_USE_ALLOCATOR64
-const uptr AllocatorSpace = ~0ULL;
-const uptr AllocatorSize = 0x40000000000ULL;
-typedef DefaultSizeClassMap SizeClassMap;
-struct AP {
-  static const uptr kSpaceBeg = AllocatorSpace;
-  static const uptr kSpaceSize = AllocatorSize;
-  static const uptr kMetadataSize = 0;
-  typedef __scudo::SizeClassMap SizeClassMap;
-  typedef NoOpMapUnmapCallback MapUnmapCallback;
-  static const uptr kFlags =
-      SizeClassAllocator64FlagMasks::kRandomShuffleChunks;
-};
-typedef SizeClassAllocator64<AP> PrimaryAllocator;
-#else
-// Currently, the 32-bit Sanitizer allocator has not yet benefited from all the
-// security improvements brought to the 64-bit one. This makes the 32-bit
-// version of Scudo slightly less toughened.
-static const uptr RegionSizeLog = 20;
-static const uptr NumRegions = SANITIZER_MMAP_RANGE_SIZE >> RegionSizeLog;
-# if SANITIZER_WORDSIZE == 32
-typedef FlatByteMap<NumRegions> ByteMap;
-# elif SANITIZER_WORDSIZE == 64
-typedef TwoLevelByteMap<(NumRegions >> 12), 1 << 12> ByteMap;
-# endif  // SANITIZER_WORDSIZE
-typedef DefaultSizeClassMap SizeClassMap;
-typedef SizeClassAllocator32<0, SANITIZER_MMAP_RANGE_SIZE, 0, SizeClassMap,
-    RegionSizeLog, ByteMap> PrimaryAllocator;
-#endif  // SANITIZER_CAN_USE_ALLOCATOR64
-
-typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache;
-typedef ScudoLargeMmapAllocator SecondaryAllocator;
-typedef CombinedAllocator<PrimaryAllocator, AllocatorCache, SecondaryAllocator>
-  ScudoAllocator;
-
-static ScudoAllocator &getAllocator();
-
-static thread_local Xorshift128Plus Prng;
 // Global static cookie, initialized at start-up.
 static uptr Cookie;
 
 // We default to software CRC32 if the alternatives are not supported, either
 // at compilation or at runtime.
 static atomic_uint8_t HashAlgorithm = { CRC32Software };
 
 SANITIZER_WEAK_ATTRIBUTE u32 computeHardwareCRC32(u32 Crc, uptr Data);
struct ScudoChunk : UnpackedHeader {
   void *getAllocBeg(UnpackedHeader *Header) {
     return reinterpret_cast<void *>(
         reinterpret_cast<uptr>(this) - (Header->Offset << MinAlignmentLog));
   }
 
   // Returns the usable size for a chunk, meaning the amount of bytes from the
   // beginning of the user data to the end of the backend allocated chunk.
   uptr getUsableSize(UnpackedHeader *Header) {
-    uptr Size = getAllocator().GetActuallyAllocatedSize(getAllocBeg(Header));
+    uptr Size =
+        getBackendAllocator().GetActuallyAllocatedSize(getAllocBeg(Header));
     if (Size == 0)
-      return Size;
+      return 0;
     return Size - AlignedChunkHeaderSize - (Header->Offset << MinAlignmentLog);
   }
 
   // Compute the checksum of the Chunk pointer and its ChunkHeader.
   u16 computeChecksum(UnpackedHeader *Header) const {
     UnpackedHeader ZeroChecksumHeader = *Header;
     ZeroChecksumHeader.Checksum = 0;
     uptr HeaderHolder[sizeof(UnpackedHeader) / sizeof(uptr)];
     memcpy(&HeaderHolder, &ZeroChecksumHeader, sizeof(HeaderHolder));
     u8 HashType = atomic_load_relaxed(&HashAlgorithm);
     u32 Crc = computeCRC32(Cookie, reinterpret_cast<uptr>(this), HashType);
     for (uptr i = 0; i < ARRAY_SIZE(HeaderHolder); i++)
       Crc = computeCRC32(Crc, HeaderHolder[i], HashType);
     return static_cast<u16>(Crc);
   }
 
-  // Checks the validity of a chunk by verifying its checksum.
+  // Checks the validity of a chunk by verifying its checksum. This check
+  // doesn't incur termination in case of an invalid chunk.
   bool isValid() {
     UnpackedHeader NewUnpackedHeader;
     const AtomicPackedHeader *AtomicHeader =
         reinterpret_cast<const AtomicPackedHeader *>(this);
     PackedHeader NewPackedHeader = atomic_load_relaxed(AtomicHeader);
     NewUnpackedHeader = bit_cast<UnpackedHeader>(NewPackedHeader);
     return (NewUnpackedHeader.Checksum == computeChecksum(&NewUnpackedHeader));
   }
 
+  // Nulls out a chunk header. When returning the chunk to the backend, there
+  // is no need to store a valid ChunkAvailable header, as this would be
+  // computationally expensive. Zero-ing out serves the same purpose by making
+  // the header invalid. In the extremely rare event where 0 would be a valid
+  // checksum for the chunk, the state of the chunk is ChunkAvailable anyway.
+  COMPILER_CHECK(ChunkAvailable == 0);
+  void eraseHeader() {
+    PackedHeader NullPackedHeader = 0;
+    AtomicPackedHeader *AtomicHeader =
+        reinterpret_cast<AtomicPackedHeader *>(this);
+    atomic_store_relaxed(AtomicHeader, NullPackedHeader);
+  }
+
   // Loads and unpacks the header, verifying the checksum in the process.
   void loadHeader(UnpackedHeader *NewUnpackedHeader) const {
     const AtomicPackedHeader *AtomicHeader =
         reinterpret_cast<const AtomicPackedHeader *>(this);
     PackedHeader NewPackedHeader = atomic_load_relaxed(AtomicHeader);
     *NewUnpackedHeader = bit_cast<UnpackedHeader>(NewPackedHeader);
-    if (NewUnpackedHeader->Checksum != computeChecksum(NewUnpackedHeader)) {
+    if (UNLIKELY(NewUnpackedHeader->Checksum
+                 != computeChecksum(NewUnpackedHeader))) {
       dieWithMessage("ERROR: corrupted chunk header at address %p\n", this);
     }
   }
 
   // Packs and stores the header, computing the checksum in the process.
   void storeHeader(UnpackedHeader *NewUnpackedHeader) {
     NewUnpackedHeader->Checksum = computeChecksum(NewUnpackedHeader);
     PackedHeader NewPackedHeader = bit_cast<PackedHeader>(*NewUnpackedHeader);
     AtomicPackedHeader *AtomicHeader =
         reinterpret_cast<AtomicPackedHeader *>(this);
     atomic_store_relaxed(AtomicHeader, NewPackedHeader);
   }
 
   // Packs and stores the header, computing the checksum in the process. We
   // compare the current header with the expected provided one to ensure that
   // we are not being raced by a corruption occurring in another thread.
   void compareExchangeHeader(UnpackedHeader *NewUnpackedHeader,
                              UnpackedHeader *OldUnpackedHeader) {
     NewUnpackedHeader->Checksum = computeChecksum(NewUnpackedHeader);
     PackedHeader NewPackedHeader = bit_cast<PackedHeader>(*NewUnpackedHeader);
     PackedHeader OldPackedHeader = bit_cast<PackedHeader>(*OldUnpackedHeader);
     AtomicPackedHeader *AtomicHeader =
         reinterpret_cast<AtomicPackedHeader *>(this);
-    if (!atomic_compare_exchange_strong(AtomicHeader,
+    if (UNLIKELY(!atomic_compare_exchange_strong(AtomicHeader,
                                                  &OldPackedHeader,
                                                  NewPackedHeader,
-                                        memory_order_relaxed)) {
+                                                 memory_order_relaxed))) {
       dieWithMessage("ERROR: race on chunk header at address %p\n", this);
     }
   }
 };
 
 static bool ScudoInitIsRunning = false;
 
-static pthread_once_t GlobalInited = PTHREAD_ONCE_INIT;
-static pthread_key_t PThreadKey;
-
-static thread_local bool ThreadInited = false;
-static thread_local bool ThreadTornDown = false;
-static thread_local AllocatorCache Cache;
-
-static void teardownThread(void *p) {
-  uptr v = reinterpret_cast<uptr>(p);
-  // The glibc POSIX thread-local-storage deallocation routine calls user
-  // provided destructors in a loop of PTHREAD_DESTRUCTOR_ITERATIONS.
-  // We want to be called last since other destructors might call free and the
-  // like, so we wait until PTHREAD_DESTRUCTOR_ITERATIONS before draining the
-  // quarantine and swallowing the cache.
-  if (v < PTHREAD_DESTRUCTOR_ITERATIONS) {
-    pthread_setspecific(PThreadKey, reinterpret_cast<void *>(v + 1));
-    return;
-  }
-  drainQuarantine();
-  getAllocator().DestroyCache(&Cache);
-  ThreadTornDown = true;
-}
-
-static void initInternal() {
+void initScudo() {
   SanitizerToolName = "Scudo";
   CHECK(!ScudoInitIsRunning && "Scudo init calls itself!");
   ScudoInitIsRunning = true;
 
   // Check is SSE4.2 is supported, if so, opt for the CRC32 hardware version.
   if (testCPUFeature(CRC32CPUFeature)) {
     atomic_store_relaxed(&HashAlgorithm, CRC32Hardware);
   }
 
   initFlags();
-
   AllocatorOptions Options;
   Options.setFrom(getFlags(), common_flags());
   initAllocator(Options);
 
-  MaybeStartBackgroudThread();
-
   ScudoInitIsRunning = false;
 }
 
-static void initGlobal() {
-  pthread_key_create(&PThreadKey, teardownThread);
-  initInternal();
-}
-
-static void NOINLINE initThread() {
-  pthread_once(&GlobalInited, initGlobal);
-  pthread_setspecific(PThreadKey, reinterpret_cast<void *>(1));
-  getAllocator().InitCache(&Cache);
-  ThreadInited = true;
-}
-
 struct QuarantineCallback {
   explicit QuarantineCallback(AllocatorCache *Cache)
     : Cache_(Cache) {}
 
   // Chunk recycling function, returns a quarantined chunk to the backend.
   void Recycle(ScudoChunk *Chunk) {
     UnpackedHeader Header;
     Chunk->loadHeader(&Header);
-    if (Header.State != ChunkQuarantine) {
+    if (UNLIKELY(Header.State != ChunkQuarantine)) {
       dieWithMessage("ERROR: invalid chunk state when recycling address %p\n",
                      Chunk);
     }
+    Chunk->eraseHeader();
     void *Ptr = Chunk->getAllocBeg(&Header);
-    getAllocator().Deallocate(Cache_, Ptr);
+    getBackendAllocator().Deallocate(Cache_, Ptr);
   }
 
   /// Internal quarantine allocation and deallocation functions.
   void *Allocate(uptr Size) {
-    // The internal quarantine memory cannot be protected by us. But the only
-    // structures allocated are QuarantineBatch, that are 8KB for x64. So we
-    // will use mmap for those, and given that Deallocate doesn't pass a size
-    // in, we enforce the size of the allocation to be sizeof(QuarantineBatch).
-    // TODO(kostyak): switching to mmap impacts greatly performances, we have
-    //                to find another solution
-    // CHECK_EQ(Size, sizeof(QuarantineBatch));
-    // return MmapOrDie(Size, "QuarantineBatch");
-    return getAllocator().Allocate(Cache_, Size, 1, false);
+    // TODO(kostyak): figure out the best way to protect the batches.
+    return getBackendAllocator().Allocate(Cache_, Size, MinAlignment);
   }
 
   void Deallocate(void *Ptr) {
-    // UnmapOrDie(Ptr, sizeof(QuarantineBatch));
-    getAllocator().Deallocate(Cache_, Ptr);
+    getBackendAllocator().Deallocate(Cache_, Ptr);
   }
 
   AllocatorCache *Cache_;
 };
 
 typedef Quarantine<QuarantineCallback, ScudoChunk> ScudoQuarantine;
-typedef ScudoQuarantine::Cache QuarantineCache;
-static thread_local QuarantineCache ThreadQuarantineCache;
+typedef ScudoQuarantine::Cache ScudoQuarantineCache;
+COMPILER_CHECK(sizeof(ScudoQuarantineCache)
+               <= sizeof(ScudoThreadContext::QuarantineCachePlaceHolder));
+
+AllocatorCache *getAllocatorCache(ScudoThreadContext *ThreadContext) {
+  return &ThreadContext->Cache;
+}
+
+ScudoQuarantineCache *getQuarantineCache(ScudoThreadContext *ThreadContext) {
+  return reinterpret_cast<
+      ScudoQuarantineCache *>(ThreadContext->QuarantineCachePlaceHolder);
+}
+
+Xorshift128Plus *getPrng(ScudoThreadContext *ThreadContext) {
+  return &ThreadContext->Prng;
+}
 
 void AllocatorOptions::setFrom(const Flags *f, const CommonFlags *cf) {
   MayReturnNull = cf->allocator_may_return_null;
   ReleaseToOSIntervalMs = cf->allocator_release_to_os_interval_ms;
   QuarantineSizeMb = f->QuarantineSizeMb;
   ThreadLocalQuarantineSizeKb = f->ThreadLocalQuarantineSizeKb;
   DeallocationTypeMismatch = f->DeallocationTypeMismatch;
   DeleteSizeMismatch = f->DeleteSizeMismatch;
   ZeroContents = f->ZeroContents;
 }
 
 void AllocatorOptions::copyTo(Flags *f, CommonFlags *cf) const {
   cf->allocator_may_return_null = MayReturnNull;
   cf->allocator_release_to_os_interval_ms = ReleaseToOSIntervalMs;
   f->QuarantineSizeMb = QuarantineSizeMb;
   f->ThreadLocalQuarantineSizeKb = ThreadLocalQuarantineSizeKb;
   f->DeallocationTypeMismatch = DeallocationTypeMismatch;
   f->DeleteSizeMismatch = DeleteSizeMismatch;
   f->ZeroContents = ZeroContents;
 }
 
-struct Allocator {
+struct ScudoAllocator {
   static const uptr MaxAllowedMallocSize =
       FIRST_32_SECOND_64(2UL << 30, 1ULL << 40);
 
-  ScudoAllocator BackendAllocator;
+  ScudoBackendAllocator BackendAllocator;
   ScudoQuarantine AllocatorQuarantine;
 
   // The fallback caches are used when the thread local caches have been
   // 'detroyed' on thread tear-down. They are protected by a Mutex as they can
   // be accessed by different threads.
   StaticSpinMutex FallbackMutex;
   AllocatorCache FallbackAllocatorCache;
-  QuarantineCache FallbackQuarantineCache;
+  ScudoQuarantineCache FallbackQuarantineCache;
+  Xorshift128Plus GlobalPrng;
 
   bool DeallocationTypeMismatch;
   bool ZeroContents;
   bool DeleteSizeMismatch;
 
-  explicit Allocator(LinkerInitialized)
+  explicit ScudoAllocator(LinkerInitialized)
     : AllocatorQuarantine(LINKER_INITIALIZED),
       FallbackQuarantineCache(LINKER_INITIALIZED) {}
 
   void init(const AllocatorOptions &Options) {
     // Verify that the header offset field can hold the maximum offset. In the
     // case of the Secondary allocator, it takes care of alignment and the
     // offset will always be 0. In the case of the Primary, the worst case
     // scenario happens in the last size class, when the backend allocation
void init(const AllocatorOptions &Options) {
 
     DeallocationTypeMismatch = Options.DeallocationTypeMismatch;
     DeleteSizeMismatch = Options.DeleteSizeMismatch;
     ZeroContents = Options.ZeroContents;
     BackendAllocator.Init(Options.MayReturnNull, Options.ReleaseToOSIntervalMs);
     AllocatorQuarantine.Init(
         static_cast<uptr>(Options.QuarantineSizeMb) << 20,
         static_cast<uptr>(Options.ThreadLocalQuarantineSizeKb) << 10);
+    FallbackMutex.Init();
     BackendAllocator.InitCache(&FallbackAllocatorCache);
-    Cookie = Prng.Next();
+    GlobalPrng.initFromURandom();
+    Cookie = GlobalPrng.getNext();
   }
 
-  // Helper function that checks for a valid Scudo chunk.
+  // Helper function that checks for a valid Scudo chunk. nullptr isn't.
   bool isValidPointer(const void *UserPtr) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
-    uptr ChunkBeg = reinterpret_cast<uptr>(UserPtr);
-    if (!IsAligned(ChunkBeg, MinAlignment)) {
+    initThreadMaybe();
+    if (!UserPtr)
       return false;
-    }
-    ScudoChunk *Chunk =
-        reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize);
-    return Chunk->isValid();
+    uptr UserBeg = reinterpret_cast<uptr>(UserPtr);
+    if (!IsAligned(UserBeg, MinAlignment))
+      return false;
+    return getScudoChunk(UserBeg)->isValid();
   }
 
   // Allocates a chunk.
-  void *allocate(uptr Size, uptr Alignment, AllocType Type) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
-    if (!IsPowerOfTwo(Alignment)) {
+  void *allocate(uptr Size, uptr Alignment, AllocType Type,
+                 bool ForceZeroContents = false) {
+    initThreadMaybe();
+    if (UNLIKELY(!IsPowerOfTwo(Alignment))) {
       dieWithMessage("ERROR: alignment is not a power of 2\n");
     }
     if (Alignment > MaxAlignment)
       return BackendAllocator.ReturnNullOrDieOnBadRequest();
     if (Alignment < MinAlignment)
       Alignment = MinAlignment;
-    if (Size == 0)
-      Size = 1;
     if (Size >= MaxAllowedMallocSize)
       return BackendAllocator.ReturnNullOrDieOnBadRequest();
+    if (Size == 0)
+      Size = 1;
 
     uptr NeededSize = RoundUpTo(Size, MinAlignment) + AlignedChunkHeaderSize;
     if (Alignment > MinAlignment)
       NeededSize += Alignment;
     if (NeededSize >= MaxAllowedMallocSize)
       return BackendAllocator.ReturnNullOrDieOnBadRequest();
 
     // Primary backed and Secondary backed allocations have a different
     // treatment. We deal with alignment requirements of Primary serviced
     // allocations here, but the Secondary will take care of its own alignment
     // needs, which means we also have to work around some limitations of the
     // combined allocator to accommodate the situation.
     bool FromPrimary = PrimaryAllocator::CanAllocate(NeededSize, MinAlignment);
 
+    uptr Salt;
     void *Ptr;
-    if (LIKELY(!ThreadTornDown)) {
-      Ptr = BackendAllocator.Allocate(&Cache, NeededSize,
-                                      FromPrimary ? MinAlignment : Alignment);
+    uptr AllocationAlignment = FromPrimary ? MinAlignment : Alignment;
+    ScudoThreadContext *ThreadContext = getCurrentThreadContext();
+    // Based on what backs the ThreadContext, the locking mechanism differ (it
+    // might not be locked at all for thread_local variables).
+    if (ThreadContext) {
+      ThreadContext->Lock();
+      Salt = getPrng(ThreadContext)->getNext();
+      Ptr = BackendAllocator.Allocate(getAllocatorCache(ThreadContext),
+                                      NeededSize, AllocationAlignment);
+      ThreadContext->Unlock();
     } else {
-      SpinMutexLock l(&FallbackMutex);
-      Ptr = BackendAllocator.Allocate(&FallbackAllocatorCache, NeededSize,
-                                      FromPrimary ? MinAlignment : Alignment);
+      SpinMutexLock Lock(&FallbackMutex);
+      Salt = GlobalPrng.getNext();
+      Ptr = BackendAllocator.Allocate(&FallbackAllocatorCache,
+                                      NeededSize, AllocationAlignment);
     }
     if (!Ptr)
       return BackendAllocator.ReturnNullOrDieOnOOM();
 
     uptr AllocBeg = reinterpret_cast<uptr>(Ptr);
     // If the allocation was serviced by the secondary, the returned pointer
     // accounts for ChunkHeaderSize to pass the alignment check of the combined
     // allocator. Adjust it here.
     if (!FromPrimary) {
       AllocBeg -= AlignedChunkHeaderSize;
       if (Alignment > MinAlignment)
         NeededSize -= Alignment;
     }
 
     uptr ActuallyAllocatedSize = BackendAllocator.GetActuallyAllocatedSize(
         reinterpret_cast<void *>(AllocBeg));
     // If requested, we will zero out the entire contents of the returned chunk.
-    if (ZeroContents && FromPrimary)
+    if ((ForceZeroContents || ZeroContents) && FromPrimary)
        memset(Ptr, 0, ActuallyAllocatedSize);
 
-    uptr ChunkBeg = AllocBeg + AlignedChunkHeaderSize;
-    if (!IsAligned(ChunkBeg, Alignment))
-      ChunkBeg = RoundUpTo(ChunkBeg, Alignment);
-    CHECK_LE(ChunkBeg + Size, AllocBeg + NeededSize);
-    ScudoChunk *Chunk =
-        reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize);
+    uptr UserBeg = AllocBeg + AlignedChunkHeaderSize;
+    if (!IsAligned(UserBeg, Alignment))
+      UserBeg = RoundUpTo(UserBeg, Alignment);
+    CHECK_LE(UserBeg + Size, AllocBeg + NeededSize);
     UnpackedHeader Header = {};
     Header.State = ChunkAllocated;
-    uptr Offset = ChunkBeg - AlignedChunkHeaderSize - AllocBeg;
+    uptr Offset = UserBeg - AlignedChunkHeaderSize - AllocBeg;
     Header.Offset = Offset >> MinAlignmentLog;
     Header.AllocType = Type;
     Header.UnusedBytes = ActuallyAllocatedSize - Offset -
         AlignedChunkHeaderSize - Size;
-    Header.Salt = static_cast<u8>(Prng.Next());
-    Chunk->storeHeader(&Header);
-    void *UserPtr = reinterpret_cast<void *>(ChunkBeg);
-    // TODO(kostyak): hooks sound like a terrible idea security wise but might
-    //                be needed for things to work properly?
+    Header.Salt = static_cast<u8>(Salt);
+    getScudoChunk(UserBeg)->storeHeader(&Header);
+    void *UserPtr = reinterpret_cast<void *>(UserBeg);
     // if (&__sanitizer_malloc_hook) __sanitizer_malloc_hook(UserPtr, Size);
     return UserPtr;
   }
 
+  // Place a chunk in the quarantine. In the event of a zero-sized quarantine,
+  // we directly deallocate the chunk, otherwise the flow would lead to the
+  // chunk being checksummed twice, once before Put and once in Recycle, with
+  // no additional security value.
+  void quarantineOrDeallocateChunk(ScudoChunk *Chunk, UnpackedHeader *Header,
+                                   uptr Size) {
+    bool BypassQuarantine = (AllocatorQuarantine.GetCacheSize() == 0);
+    ScudoThreadContext *ThreadContext = getCurrentThreadContext();
+    if (BypassQuarantine) {
+      Chunk->eraseHeader();
+      void *Ptr = Chunk->getAllocBeg(Header);
+      if (ThreadContext) {
+        ThreadContext->Lock();
+        getBackendAllocator().Deallocate(getAllocatorCache(ThreadContext), Ptr);
+        ThreadContext->Unlock();
+      } else {
+        SpinMutexLock Lock(&FallbackMutex);
+        getBackendAllocator().Deallocate(&FallbackAllocatorCache, Ptr);
+      }
+    } else {
+      UnpackedHeader NewHeader = *Header;
+      NewHeader.State = ChunkQuarantine;
+      Chunk->compareExchangeHeader(&NewHeader, Header);
+      if (ThreadContext) {
+        ThreadContext->Lock();
+        AllocatorQuarantine.Put(getQuarantineCache(ThreadContext),
+                                QuarantineCallback(
+                                    getAllocatorCache(ThreadContext)),
+                                Chunk, Size);
+        ThreadContext->Unlock();
+      } else {
+        SpinMutexLock Lock(&FallbackMutex);
+        AllocatorQuarantine.Put(&FallbackQuarantineCache,
+                                QuarantineCallback(&FallbackAllocatorCache),
+                                Chunk, Size);
+      }
+    }
+  }
+
   // Deallocates a Chunk, which means adding it to the delayed free list (or
-  // Quarantine).
+  // Quarantine) or returning it to the backend.
   void deallocate(void *UserPtr, uptr DeleteSize, AllocType Type) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
-    // TODO(kostyak): see hook comment above
+    initThreadMaybe();
     // if (&__sanitizer_free_hook) __sanitizer_free_hook(UserPtr);
     if (!UserPtr)
       return;
-    uptr ChunkBeg = reinterpret_cast<uptr>(UserPtr);
-    if (!IsAligned(ChunkBeg, MinAlignment)) {
+    uptr UserBeg = reinterpret_cast<uptr>(UserPtr);
+    if (UNLIKELY(!IsAligned(UserBeg, MinAlignment))) {
       dieWithMessage("ERROR: attempted to deallocate a chunk not properly "
                      "aligned at address %p\n", UserPtr);
     }
-    ScudoChunk *Chunk =
-        reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize);
+    ScudoChunk *Chunk = getScudoChunk(UserBeg);
     UnpackedHeader OldHeader;
     Chunk->loadHeader(&OldHeader);
-    if (OldHeader.State != ChunkAllocated) {
+    if (UNLIKELY(OldHeader.State != ChunkAllocated)) {
       dieWithMessage("ERROR: invalid chunk state when deallocating address "
                      "%p\n", UserPtr);
     }
-    uptr UsableSize = Chunk->getUsableSize(&OldHeader);
-    UnpackedHeader NewHeader = OldHeader;
-    NewHeader.State = ChunkQuarantine;
-    Chunk->compareExchangeHeader(&NewHeader, &OldHeader);
     if (DeallocationTypeMismatch) {
       // The deallocation type has to match the allocation one.
-      if (NewHeader.AllocType != Type) {
+      if (OldHeader.AllocType != Type) {
         // With the exception of memalign'd Chunks, that can be still be free'd.
-        if (NewHeader.AllocType != FromMemalign || Type != FromMalloc) {
+        if (OldHeader.AllocType != FromMemalign || Type != FromMalloc) {
           dieWithMessage("ERROR: allocation type mismatch on address %p\n",
-                         Chunk);
+                         UserPtr);
         }
       }
     }
-    uptr Size = UsableSize - OldHeader.UnusedBytes;
+    uptr UsableSize = Chunk->getUsableSize(&OldHeader);
     if (DeleteSizeMismatch) {
+      uptr Size = UsableSize - OldHeader.UnusedBytes;
       if (DeleteSize && DeleteSize != Size) {
         dieWithMessage("ERROR: invalid sized delete on chunk at address %p\n",
-                       Chunk);
+                       UserPtr);
       }
     }
-
-    if (LIKELY(!ThreadTornDown)) {
-      AllocatorQuarantine.Put(&ThreadQuarantineCache,
-                              QuarantineCallback(&Cache), Chunk, UsableSize);
-    } else {
-      SpinMutexLock l(&FallbackMutex);
-      AllocatorQuarantine.Put(&FallbackQuarantineCache,
-                              QuarantineCallback(&FallbackAllocatorCache),
-                              Chunk, UsableSize);
-    }
+    quarantineOrDeallocateChunk(Chunk, &OldHeader, UsableSize);
   }
 
   // Reallocates a chunk. We can save on a new allocation if the new requested
   // size still fits in the chunk.
   void *reallocate(void *OldPtr, uptr NewSize) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
-    uptr ChunkBeg = reinterpret_cast<uptr>(OldPtr);
-    ScudoChunk *Chunk =
-        reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize);
+    initThreadMaybe();
+    uptr UserBeg = reinterpret_cast<uptr>(OldPtr);
+    if (UNLIKELY(!IsAligned(UserBeg, MinAlignment))) {
+      dieWithMessage("ERROR: attempted to reallocate a chunk not properly "
+                     "aligned at address %p\n", OldPtr);
+    }
+    ScudoChunk *Chunk = getScudoChunk(UserBeg);
     UnpackedHeader OldHeader;
     Chunk->loadHeader(&OldHeader);
-    if (OldHeader.State != ChunkAllocated) {
+    if (UNLIKELY(OldHeader.State != ChunkAllocated)) {
       dieWithMessage("ERROR: invalid chunk state when reallocating address "
                      "%p\n", OldPtr);
     }
-    uptr Size = Chunk->getUsableSize(&OldHeader);
-    if (OldHeader.AllocType != FromMalloc) {
+    if (UNLIKELY(OldHeader.AllocType != FromMalloc)) {
       dieWithMessage("ERROR: invalid chunk type when reallocating address %p\n",
-                     Chunk);
+                     OldPtr);
     }
-    UnpackedHeader NewHeader = OldHeader;
+    uptr UsableSize = Chunk->getUsableSize(&OldHeader);
     // The new size still fits in the current chunk.
-    if (NewSize <= Size) {
-      NewHeader.UnusedBytes = Size - NewSize;
+    if (NewSize <= UsableSize) {
+      UnpackedHeader NewHeader = OldHeader;
+      NewHeader.UnusedBytes = UsableSize - NewSize;
       Chunk->compareExchangeHeader(&NewHeader, &OldHeader);
       return OldPtr;
     }
     // Otherwise, we have to allocate a new chunk and copy the contents of the
     // old one.
     void *NewPtr = allocate(NewSize, MinAlignment, FromMalloc);
     if (NewPtr) {
-      uptr OldSize = Size - OldHeader.UnusedBytes;
+      uptr OldSize = UsableSize - OldHeader.UnusedBytes;
       memcpy(NewPtr, OldPtr, Min(NewSize, OldSize));
-      NewHeader.State = ChunkQuarantine;
-      Chunk->compareExchangeHeader(&NewHeader, &OldHeader);
-      if (LIKELY(!ThreadTornDown)) {
-        AllocatorQuarantine.Put(&ThreadQuarantineCache,
-                                QuarantineCallback(&Cache), Chunk, Size);
-      } else {
-        SpinMutexLock l(&FallbackMutex);
-        AllocatorQuarantine.Put(&FallbackQuarantineCache,
-                                QuarantineCallback(&FallbackAllocatorCache),
-                                Chunk, Size);
-      }
+      quarantineOrDeallocateChunk(Chunk, &OldHeader, UsableSize);
     }
     return NewPtr;
   }
 
+  ScudoChunk *getScudoChunk(uptr UserBeg) {
+    return reinterpret_cast<ScudoChunk *>(UserBeg - AlignedChunkHeaderSize);
+  }
+
   // Helper function that returns the actual usable size of a chunk.
   uptr getUsableSize(const void *Ptr) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
+    initThreadMaybe();
     if (!Ptr)
       return 0;
-    uptr ChunkBeg = reinterpret_cast<uptr>(Ptr);
-    ScudoChunk *Chunk =
-        reinterpret_cast<ScudoChunk *>(ChunkBeg - AlignedChunkHeaderSize);
+    uptr UserBeg = reinterpret_cast<uptr>(Ptr);
+    ScudoChunk *Chunk = getScudoChunk(UserBeg);
     UnpackedHeader Header;
     Chunk->loadHeader(&Header);
     // Getting the usable size of a chunk only makes sense if it's allocated.
-    if (Header.State != ChunkAllocated) {
+    if (UNLIKELY(Header.State != ChunkAllocated)) {
       dieWithMessage("ERROR: invalid chunk state when sizing address %p\n",
                      Ptr);
     }
     return Chunk->getUsableSize(&Header);
   }
 
   void *calloc(uptr NMemB, uptr Size) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
+    initThreadMaybe();
     uptr Total = NMemB * Size;
-    if (Size != 0 && Total / Size != NMemB) // Overflow check
+    if (Size != 0 && Total / Size != NMemB)  // Overflow check
       return BackendAllocator.ReturnNullOrDieOnBadRequest();
-    void *Ptr = allocate(Total, MinAlignment, FromMalloc);
-    // If ZeroContents, the content of the chunk has already been zero'd out.
-    if (!ZeroContents && Ptr && BackendAllocator.FromPrimary(Ptr))
-      memset(Ptr, 0, getUsableSize(Ptr));
-    return Ptr;
+    return allocate(Total, MinAlignment, FromMalloc, true);
   }
 
-  void drainQuarantine() {
-    AllocatorQuarantine.Drain(&ThreadQuarantineCache,
-                              QuarantineCallback(&Cache));
+  void commitBack(ScudoThreadContext *ThreadContext) {
+    AllocatorCache *Cache = getAllocatorCache(ThreadContext);
+    AllocatorQuarantine.Drain(getQuarantineCache(ThreadContext),
+                              QuarantineCallback(Cache));
+    BackendAllocator.DestroyCache(Cache);
   }
 
   uptr getStats(AllocatorStat StatType) {
-    if (UNLIKELY(!ThreadInited))
-      initThread();
+    initThreadMaybe();
     uptr stats[AllocatorStatCount];
     BackendAllocator.GetStats(stats);
     return stats[StatType];
   }
 };
 
-static Allocator Instance(LINKER_INITIALIZED);
+static ScudoAllocator Instance(LINKER_INITIALIZED);
 
-static ScudoAllocator &getAllocator() {
+ScudoBackendAllocator &getBackendAllocator() {
   return Instance.BackendAllocator;
 }
 
 void initAllocator(const AllocatorOptions &Options) {
   Instance.init(Options);
 }
 
-void drainQuarantine() {
-  Instance.drainQuarantine();
+void ScudoThreadContext::commitBack() {
+  Instance.commitBack(this);
 }
 
 void *scudoMalloc(uptr Size, AllocType Type) {
   return Instance.allocate(Size, MinAlignment, Type);
 }
 
 void scudoFree(void *Ptr, AllocType Type) {
   Instance.deallocate(Ptr, 0, Type);

lib/scudo/scudo_allocator_secondary.h

CHECK_EQ(MapBeg + PageSize, reinterpret_cast<uptr>(
         MmapFixedOrDie(MapBeg + PageSize, MapSize - 2 * PageSize)));
     uptr Ptr = UserBeg - AlignedChunkHeaderSize;
     SecondaryHeader *Header = getHeader(Ptr);
     Header->MapBeg = MapBeg;
     Header->MapSize = MapSize;
     // The primary adds the whole class size to the stats when allocating a
     // chunk, so we will do something similar here. But we will not account for
     // the guard pages.
+    {
+      SpinMutexLock l(&StatsMutex);
       Stats->Add(AllocatorStatAllocated, MapSize - 2 * PageSize);
       Stats->Add(AllocatorStatMapped, MapSize - 2 * PageSize);
+    }
 
     return reinterpret_cast<void *>(UserBeg);
   }
 
   void *ReturnNullOrDieOnBadRequest() {
     if (atomic_load(&MayReturnNull, memory_order_acquire))
       return nullptr;
     ReportAllocatorCannotReturnNull(false);
   }
 
   void *ReturnNullOrDieOnOOM() {
     if (atomic_load(&MayReturnNull, memory_order_acquire))
       return nullptr;
     ReportAllocatorCannotReturnNull(true);
   }
 
   void SetMayReturnNull(bool AllocatorMayReturnNull) {
     atomic_store(&MayReturnNull, AllocatorMayReturnNull, memory_order_release);
   }
 
   void Deallocate(AllocatorStats *Stats, void *Ptr) {
     SecondaryHeader *Header = getHeader(Ptr);
+    {
+      SpinMutexLock l(&StatsMutex);
       Stats->Sub(AllocatorStatAllocated, Header->MapSize - 2 * PageSize);
       Stats->Sub(AllocatorStatMapped, Header->MapSize - 2 * PageSize);
+    }
     UnmapOrDie(reinterpret_cast<void *>(Header->MapBeg), Header->MapSize);
   }
 
   uptr TotalMemoryUsed() {
     UNIMPLEMENTED();
   }
 
   bool PointerIsMine(const void *Ptr) {
     UNIMPLEMENTED();
   }
 
   uptr GetActuallyAllocatedSize(void *Ptr) {
     SecondaryHeader *Header = getHeader(Ptr);
-    // Deduct PageSize as MapEnd includes the trailing guard page.
+    // Deduct PageSize as MapSize includes the trailing guard page.
     uptr MapEnd = Header->MapBeg + Header->MapSize - PageSize;
     return MapEnd - reinterpret_cast<uptr>(Ptr);
   }
 
   void *GetMetaData(const void *Ptr) {
     UNIMPLEMENTED();
   }
 
private:
   }
   SecondaryHeader *getHeader(const void *Ptr) {
     return getHeader(reinterpret_cast<uptr>(Ptr));
   }
 
   const uptr SecondaryHeaderSize = sizeof(SecondaryHeader);
   const uptr HeadersSize = SecondaryHeaderSize + AlignedChunkHeaderSize;
   uptr PageSize;
+  SpinMutex StatsMutex;
   atomic_uint8_t MayReturnNull;
 };
 
 #endif  // SCUDO_ALLOCATOR_SECONDARY_H_

lib/scudo/scudo_thread.h

+//===-- scudo_thread.h ------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// TODO(kostyak): description.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef SCUDO_THREAD_H_
+#define SCUDO_THREAD_H_
+
+#include "scudo_allocator.h"
+#include "scudo_utils.h"
+
+#include "sanitizer_common/sanitizer_platform.h"
+
+namespace __scudo {
+
+struct ScudoThreadContext {
+ public:
+  StaticSpinMutex Mutex;
+  AllocatorCache Cache;
+  Xorshift128Plus Prng;
+  uptr QuarantineCachePlaceHolder[4];
+  void init() {
+    Mutex.Init();
+    getBackendAllocator().InitCache(&Cache);
+    Prng.initFromURandom();
+    memset(QuarantineCachePlaceHolder, 0, sizeof(QuarantineCachePlaceHolder));
+  }
+  void commitBack();
+  void Lock();
+  void Unlock();
+};
+
+void initThreadMaybe();
+ScudoThreadContext *getCurrentThreadContext();
+
+}  // namespace __scudo
+
+#endif  // SCUDO_THREAD_H_

lib/scudo/scudo_thread_android.cpp

+//===-- scudo_thread_android.cpp --------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// TODO(kostyak): description.
+///
+//===----------------------------------------------------------------------===//
+
+#include "sanitizer_common/sanitizer_platform.h"
+
+#if SANITIZER_LINUX && SANITIZER_ANDROID
+
+#include "scudo_thread.h"
+
+#include "sanitizer_common/sanitizer_atomic.h"
+
+#include <pthread.h>
+
+namespace __scudo {
+
+static pthread_once_t GlobalInited = PTHREAD_ONCE_INIT;
+static pthread_key_t PThreadKey;
+
+// For Android, and possibly other platforms with stronger memory constraints,
+// we can't necessarily afford a context per thread. The alternative adopted
+// here is a fixed set of contexts (the number being defined at compile time),
+// where contexts are assigned to new threads in a round-robin fashion.
+
+#ifndef SCUDO_N_CONTEXTS
+# define SCUDO_N_CONTEXTS 4
+#endif
+
+static atomic_uint32_t ThreadContextCurrentIndex;
+static ScudoThreadContext *ThreadContexts;
+
+static void initOnce() {
+  CHECK_EQ(pthread_key_create(&PThreadKey, NULL), 0);
+  initScudo();
+  ThreadContexts = reinterpret_cast<ScudoThreadContext *>(
+      MmapOrDie(sizeof(ScudoThreadContext) * SCUDO_N_CONTEXTS, __func__));
+  for (int i = 0; i < SCUDO_N_CONTEXTS; i++)
+    ThreadContexts[i].init();
+}
+
+void initThreadMaybe() {
+  pthread_once(&GlobalInited, initOnce);
+  if (pthread_getspecific(PThreadKey) == NULL) {
+    u32 Index = atomic_fetch_add(&ThreadContextCurrentIndex, 1,
+                                 memory_order_relaxed);
+    ScudoThreadContext *ThreadContext =
+        &ThreadContexts[Index % SCUDO_N_CONTEXTS];
+    pthread_setspecific(PThreadKey, ThreadContext);
+  }
+}
+
+ScudoThreadContext *getCurrentThreadContext() {
+  ScudoThreadContext *ThreadContext =
+      reinterpret_cast<ScudoThreadContext *>(pthread_getspecific(PThreadKey));
+  CHECK(ThreadContext);
+  return ThreadContext;
+}
+
+void ScudoThreadContext::Lock() { Mutex.Lock(); }
+void ScudoThreadContext::Unlock() { Mutex.Unlock(); }
+
+}  // namespace __scudo
+
+#endif

lib/scudo/scudo_thread_linux.cpp

+//===-- scudo_thread_linux.cpp ----------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// TODO(kostyak): description.
+///
+//===----------------------------------------------------------------------===//
+
+#include "sanitizer_common/sanitizer_platform.h"
+
+#if SANITIZER_LINUX && !SANITIZER_ANDROID
+
+#include "scudo_thread.h"
+
+#include <limits.h>
+#include <pthread.h>
+
+namespace __scudo {
+
+static pthread_once_t GlobalInited = PTHREAD_ONCE_INIT;
+static pthread_key_t PThreadKey;
+
+static thread_local bool ThreadInited = false;
+static thread_local bool ThreadTornDown = false;
+static thread_local ScudoThreadContext ThreadLocalContext;
+
+static void teardownThread(void *Ptr) {
+  uptr Iteration = reinterpret_cast<uptr>(Ptr);
+  // The glibc POSIX thread-local-storage deallocation routine calls user
+  // provided destructors in a loop of PTHREAD_DESTRUCTOR_ITERATIONS.
+  // We want to be called last since other destructors might call free and the
+  // like, so we wait until PTHREAD_DESTRUCTOR_ITERATIONS before draining the
+  // quarantine and swallowing the cache.
+  if (Iteration < PTHREAD_DESTRUCTOR_ITERATIONS) {
+    pthread_setspecific(PThreadKey, reinterpret_cast<void *>(Iteration + 1));
+    return;
+  }
+  ThreadLocalContext.commitBack();
+  ThreadTornDown = true;
+}
+
+
+static void initOnce() {
+  CHECK_EQ(pthread_key_create(&PThreadKey, teardownThread), 0);
+  initScudo();
+}
+
+void initThreadMaybe() {
+  if (LIKELY(ThreadInited))
+    return;
+  pthread_once(&GlobalInited, initOnce);
+  pthread_setspecific(PThreadKey, reinterpret_cast<void *>(1));
+  ThreadLocalContext.init();
+  ThreadInited = true;
+}
+
+ScudoThreadContext *getCurrentThreadContext() {
+  if (ThreadTornDown)
+    return nullptr;
+  return &ThreadLocalContext;
+}
+
+void ScudoThreadContext::Lock() {}
+void ScudoThreadContext::Unlock() {}
+
+}  // namespace __scudo
+
+#endif  // SANITIZER_LINUX && !SANITIZER_ANDROID

lib/scudo/scudo_utils.h

 //===-- scudo_utils.h -------------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 ///
 /// Header for scudo_utils.cpp.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef SCUDO_UTILS_H_
 #define SCUDO_UTILS_H_
 
-#include <string.h>
-
 #include "sanitizer_common/sanitizer_common.h"
 
+#include <string.h>
+
 namespace __scudo {
 
 template <class Dest, class Source>
 inline Dest bit_cast(const Source& source) {
   static_assert(sizeof(Dest) == sizeof(Source), "Sizes are not equal!");
   Dest dest;
   memcpy(&dest, &source, sizeof(dest));
   return dest;
 }
 
 void NORETURN dieWithMessage(const char *Format, ...);
 
 enum CPUFeature {
   CRC32CPUFeature = 0,
   MaxCPUFeature,
 };
 bool testCPUFeature(CPUFeature feature);
 
 // Tiny PRNG based on https://en.wikipedia.org/wiki/Xorshift#xorshift.2B
 // The state (128 bits) will be stored in thread local storage.
 struct Xorshift128Plus {
  public:
-  Xorshift128Plus();
-  u64 Next() {
+  void initFromURandom();
+  u64 getNext() {
     u64 x = State[0];
     const u64 y = State[1];
     State[0] = y;
     x ^= x << 23;
     State[1] = x ^ y ^ (x >> 17) ^ (y >> 26);
     return State[1] + y;
   }
  private:

lib/scudo/scudo_utils.cpp

static void fillRandom(u8 *Data, ssize_t Size) {
   }
   bool Success = readRetry(Fd, Data, Size) == Size;
   close(Fd);
   if (!Success) {
     dieWithMessage("ERROR: failed to read enough data from /dev/urandom.\n");
   }
 }
 
-// Default constructor for Xorshift128Plus seeds the state with /dev/urandom.
 // TODO(kostyak): investigate using getrandom() if available.
-Xorshift128Plus::Xorshift128Plus() {
+void Xorshift128Plus::initFromURandom() {
   fillRandom(reinterpret_cast<u8 *>(State), sizeof(State));
 }
 
 const static u32 CRC32Table[] = {
   0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f,
   0xe963a535, 0x9e6495a3, 0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988,
   0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, 0x1db71064, 0x6ab020f2,
   0xf3b97148, 0x84be41de, 0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7,