[scudo][standalone] Merge Spin & Blocking mutex into a Hybrid one

[cryptoad]

Summary

We ran into a problem on Fuchsia where yielding threads would never
be deboosted, ultimately resulting in several threads spinning on the
same TSD, and no possibility for another thread to be scheduled,
dead-locking the process.

While this was fixed in Zircon, this lead to discussions about if
spinning without a break condition was a good decision, and settled on
a new hybrid model that would spin for a while then block.

Currently we are using a number of iterations for spinning that is
mostly arbitrary (based on sanitizer_common values), but this can
be tuned in the future.

Since we are touching common.h, we also use this change as a vehicle
for an Android optimization (the page size is fixed in Bionic, so use
a fixed value too).

[hctim]

lib/scudo/standalone/common.h

122 ↗

(On Diff #208471)

Could we potentially add an android-specific test to make sure this assumption doesn't change?

Also, could this potentially be a compile-time resolution? getPageSizeCached() is called for every allocation to the secondary allocator and seems like an unneccessary branch that can be eliminated.

lib/scudo/standalone/mutex.h

21 ↗

(On Diff #208471)

Instead of memset-init here, can we simply have a constexpr constructor to guarantee zero-initialisation? Would potentially prevent forgotten init() in future.

27 ↗

(On Diff #208471)

Why?

29 ↗

(On Diff #208471)

Could we have some sort of class-level constexpr here instead of just hardcoding the number 10?

30 ↗

(On Diff #208471)

Same as above.

40 ↗

(On Diff #208471)

Why is this a uptr[1]? It looks like the underlying atomic_compare_exchange_strong supports any atomic type, including char.

From my reading of the spec, the underlying type for an unscoped enum is implementation defined. If you want a concrete type, you could always use enum State : char { ... }. If you're worried about alignment, you could always use the uptr as the backing type or alignas. I feel like State Storage; would be much more easy to understand here. You could always also use enum class to get a guaranteed integer-backed enum.

[cryptoad]

lib/scudo/standalone/common.h

122 ↗

(On Diff #208471)

Good point, I will add a test.
This will be optimized by the compiler, as SCUDO_ANDROID is a compile time constant.
From previous sanitizer_common reviews, this construct is preferred to an #if as it is less cumbersome, and more maintainable.

lib/scudo/standalone/mutex.h

21 ↗

(On Diff #208471)

The benefit here is to not do unnecessary zeroing of memory when we are dealing with TLS or static variable that are already zero'd out.
That allows us to only have to use init in tests and for stack variables.
I'd rather keep it that way since it's conformant with other similar code constructs.

27 ↗

(On Diff #208471)

We do not need speed here, a tighter function is preferable.
From g3 & Android disassembly, the compiler unrolls that leading to a giant chunk that isn't ideal.
I tried to figure out a way that would work for all compilers, but loop optimization (or disabling of) is fairly specific afaict.
I settled for this.

40 ↗

(On Diff #208471)

This is variable that stores the underlying platform-specific data.
For Linux, we want a u32 (atomic), but for Fuchsia it's a sync_mutex_t, which should be opaque to us (Fuchsia doesn't use the State enum).
A single uptr is enough to store either (with CHECKs in their respective implementation to verify size and alignment), but using an array would make it easier to add other platforms.

[cryptoad]

• Adding constexpr for the number of yields & trys in the mutex class;
• Adding a test to ensure that scudo::getPageSizeCached is consitent with getpagesize;

[cryptoad]

Move the mutex state enum into linux.cc since it's only used there.

[hctim]

lib/scudo/standalone/mutex.h

21 ↗

(On Diff #208471)

Constant initialization should generally be done at compile-time for statics, and non-IE TLS will have to zero-init either in the c'tor or init() anyway, right?

Currently, it's quite hard to keep track of whether the mutex is initialised or not. For example, SizeClassInfo::Mutex is only safe as it's used in Allocator::Primary which is (as far as I can tell) statically allocated. There shouldn't be a perf overhead for this, and it would make it a strong guarantee that the mutex is always initialised.

Up to you though - I personally find it very hard to keep track of what needs to be initialised. Also, if there is ever a change that makes the class unusable if zero-initialised, the author may mistakenly assume init() is called every time.

27 ↗

(On Diff #208471)

Sounds good. Could you add this as a comment for future readers?

40 ↗

(On Diff #208471)

For GWP-ASan, we decided that ifdefs around the member and the header include were okay, which is also what jemalloc does. I was initially very hesitent to use opaque platform-specific bytes here, as I didn't know the internals of zircon's sync_mutex_t. Looks like it basically typedefs down to an int through zx_futex_t, which means it doesn't violate the standard.

On this note, why in the linux implementation was the choice made to use SYS_futex instead of pthread_mutex_t?

[cryptoad]

Addressing review comments:

• commenting the use of the #pragma nounroll
• using the platform specific types in mutex.h

[cryptoad]

lib/scudo/standalone/mutex.h

40 ↗

(On Diff #208471)

Replaced the opaque uptr with platform specific types.
Discussed offline the reason for not using pthread_mutex_t, to make a long story short: performance on some platforms.

[cryptoad]

lib/scudo/standalone/mutex.h

21 ↗

(On Diff #208471)

I'd like to keep it that way. I might revisit that at some point.

[hctim]

LGTM.

compiler-rt/trunk/lib/scudo/standalone/atomic_helpers.h

INLINE typename T::Type atomic_load_relaxed(const volatile T *A) {
   return atomic_load(A, memory_order_relaxed);
 }
 
 template <typename T>
 INLINE void atomic_store_relaxed(volatile T *A, typename T::Type V) {
   atomic_store(A, V, memory_order_relaxed);
 }
 
+template <typename T>
+INLINE typename T::Type atomic_compare_exchange(volatile T *A,
+                                                typename T::Type Cmp,
+                                                typename T::Type Xchg) {
+  atomic_compare_exchange_strong(A, &Cmp, Xchg, memory_order_acquire);
+  return Cmp;
+}
+
 } // namespace scudo
 
 #endif // SCUDO_ATOMIC_H_

compiler-rt/trunk/lib/scudo/standalone/bytemap.h

 
 template <uptr Level1Size, uptr Level2Size> class TwoLevelByteMap {
 public:
   void initLinkerInitialized() {
     Level1Map = reinterpret_cast<atomic_uptr *>(
         map(nullptr, sizeof(atomic_uptr) * Level1Size, "scudo:bytemap"));
   }
   void init() {
-    initLinkerInitialized();
     Mutex.init();
+    initLinkerInitialized();
   }
 
   void reset() {
     for (uptr I = 0; I < Level1Size; I++) {
       u8 *P = get(I);
       if (!P)
         continue;
       unmap(P, Level2Size);
u8 *get(uptr Index) const {
     DCHECK_LT(Index, Level1Size);
     return reinterpret_cast<u8 *>(
         atomic_load(&Level1Map[Index], memory_order_acquire));
   }
 
   u8 *getOrCreate(uptr Index) {
     u8 *Res = get(Index);
     if (!Res) {
-      SpinMutexLock L(&Mutex);
+      ScopedLock L(Mutex);
       if (!(Res = get(Index))) {
         Res = reinterpret_cast<u8 *>(map(nullptr, Level2Size, "scudo:bytemap"));
         atomic_store(&Level1Map[Index], reinterpret_cast<uptr>(Res),
                      memory_order_release);
       }
     }
     return Res;
   }
 
   atomic_uptr *Level1Map;
-  StaticSpinMutex Mutex;
+  HybridMutex Mutex;
 };
 
 } // namespace scudo
 
 #endif // SCUDO_BYTEMAP_H_

compiler-rt/trunk/lib/scudo/standalone/common.h

#elif defined(__aarch64__) || defined(__arm__)
   for (u8 I = 0; I < Count; I++)
     __asm__ __volatile__("yield");
 #endif
   __asm__ __volatile__("" ::: "memory");
 }
 
 // Platform specific functions.
 
-void yieldPlatform();
-
 extern uptr PageSizeCached;
 uptr getPageSizeSlow();
 INLINE uptr getPageSizeCached() {
+  // Bionic uses a hardcoded value.
+  if (SCUDO_ANDROID)
+    return 4096U;
   if (LIKELY(PageSizeCached))
     return PageSizeCached;
   return getPageSizeSlow();
 }
 
 u32 getNumberOfCPUs();
 
 const char *getEnv(const char *Name);

compiler-rt/trunk/lib/scudo/standalone/fuchsia.cc

 #include <limits.h>         // for PAGE_SIZE
 #include <stdlib.h>         // for getenv()
 #include <zircon/compiler.h>
 #include <zircon/sanitizer.h>
 #include <zircon/syscalls.h>
 
 namespace scudo {
 
-void yieldPlatform() {
-  const zx_status_t Status = _zx_nanosleep(0);
-  CHECK_EQ(Status, ZX_OK);
-}
-
 uptr getPageSize() { return PAGE_SIZE; }
 
 void NORETURN die() { __builtin_trap(); }
 
 // We zero-initialize the Extra parameter of map(), make sure this is consistent
 // with ZX_HANDLE_INVALID.
 COMPILER_CHECK(ZX_HANDLE_INVALID == 0);
 
void releasePagesToOS(UNUSED uptr BaseAddress, uptr Offset, uptr Size,
   CHECK_EQ(Status, ZX_OK);
 }
 
 const char *getEnv(const char *Name) { return getenv(Name); }
 
 // Note: we need to flag these methods with __TA_NO_THREAD_SAFETY_ANALYSIS
 // because the Fuchsia implementation of sync_mutex_t has clang thread safety
 // annotations. Were we to apply proper capability annotations to the top level
-// BlockingMutex class itself, they would not be needed. As it stands, the
+// HybridMutex class itself, they would not be needed. As it stands, the
 // thread analysis thinks that we are locking the mutex and accidentally leaving
 // it locked on the way out.
-void BlockingMutex::lock() __TA_NO_THREAD_SAFETY_ANALYSIS {
+bool HybridMutex::tryLock() __TA_NO_THREAD_SAFETY_ANALYSIS {
   // Size and alignment must be compatible between both types.
-  COMPILER_CHECK(sizeof(sync_mutex_t) <= sizeof(OpaqueStorage));
-  COMPILER_CHECK(!(alignof(decltype(OpaqueStorage)) % alignof(sync_mutex_t)));
-  sync_mutex_lock(reinterpret_cast<sync_mutex_t *>(OpaqueStorage));
+  return sync_mutex_trylock(&M) == ZX_OK;
+}
+
+void HybridMutex::lockSlow() __TA_NO_THREAD_SAFETY_ANALYSIS {
+  sync_mutex_lock(&M);
 }
 
-void BlockingMutex::unlock() __TA_NO_THREAD_SAFETY_ANALYSIS {
-  sync_mutex_unlock(reinterpret_cast<sync_mutex_t *>(OpaqueStorage));
+void HybridMutex::unlock() __TA_NO_THREAD_SAFETY_ANALYSIS {
+  sync_mutex_unlock(&M);
 }
 
 u64 getMonotonicTime() { return _zx_clock_get_monotonic(); }
 
 u32 getNumberOfCPUs() { return _zx_system_get_num_cpus(); }
 
 bool getRandom(void *Buffer, uptr Length, bool Blocking) {
   COMPILER_CHECK(MaxRandomLength <= ZX_CPRNG_DRAW_MAX_LEN);

compiler-rt/trunk/lib/scudo/standalone/linux.cc

 #include <sys/prctl.h>
 // Definitions of prctl arguments to set a vma name in Android kernels.
 #define ANDROID_PR_SET_VMA 0x53564d41
 #define ANDROID_PR_SET_VMA_ANON_NAME 0
 #endif
 
 namespace scudo {
 
-void yieldPlatform() { sched_yield(); }
-
 uptr getPageSize() { return static_cast<uptr>(sysconf(_SC_PAGESIZE)); }
 
 void NORETURN die() { abort(); }
 
 void *map(void *Addr, uptr Size, UNUSED const char *Name, uptr Flags,
           UNUSED MapPlatformData *Data) {
   int MmapFlags = MAP_PRIVATE | MAP_ANON;
-  if (Flags & MAP_NOACCESS)
+  int MmapProt;
+  if (Flags & MAP_NOACCESS) {
     MmapFlags |= MAP_NORESERVE;
+    MmapProt = PROT_NONE;
+  } else {
+    MmapProt = PROT_READ | PROT_WRITE;
+  }
   if (Addr) {
     // Currently no scenario for a noaccess mapping with a fixed address.
     DCHECK_EQ(Flags & MAP_NOACCESS, 0);
     MmapFlags |= MAP_FIXED;
   }
-  const int MmapProt =
-      (Flags & MAP_NOACCESS) ? PROT_NONE : PROT_READ | PROT_WRITE;
   void *P = mmap(Addr, Size, MmapProt, MmapFlags, -1, 0);
   if (P == MAP_FAILED) {
     if (!(Flags & MAP_ALLOWNOMEM) || errno != ENOMEM)
       dieOnMapUnmapError(errno == ENOMEM);
     return nullptr;
   }
 #if SCUDO_ANDROID
   if (!(Flags & MAP_NOACCESS))
void releasePagesToOS(uptr BaseAddress, uptr Offset, uptr Size,
   void *Addr = reinterpret_cast<void *>(BaseAddress + Offset);
   while (madvise(Addr, Size, MADV_DONTNEED) == -1 && errno == EAGAIN) {
   }
 }
 
 // Calling getenv should be fine (c)(tm) at any time.
 const char *getEnv(const char *Name) { return getenv(Name); }
 
-void BlockingMutex::lock() {
-  atomic_u32 *M = reinterpret_cast<atomic_u32 *>(&OpaqueStorage);
-  if (atomic_exchange(M, MtxLocked, memory_order_acquire) == MtxUnlocked)
+namespace {
+enum State : u32 { Unlocked = 0, Locked = 1, Sleeping = 2 };
+}
+
+bool HybridMutex::tryLock() {
+  return atomic_compare_exchange(&M, Unlocked, Locked) == Unlocked;
+}
+
+// The following is based on https://akkadia.org/drepper/futex.pdf.
+void HybridMutex::lockSlow() {
+  u32 V = atomic_compare_exchange(&M, Unlocked, Locked);
+  if (V == Unlocked)
     return;
-  while (atomic_exchange(M, MtxSleeping, memory_order_acquire) != MtxUnlocked)
-    syscall(SYS_futex, reinterpret_cast<uptr>(OpaqueStorage),
-            FUTEX_WAIT_PRIVATE, MtxSleeping, nullptr, nullptr, 0);
+  if (V != Sleeping)
+    V = atomic_exchange(&M, Sleeping, memory_order_acquire);
+  while (V != Unlocked) {
+    syscall(SYS_futex, reinterpret_cast<uptr>(&M), FUTEX_WAIT_PRIVATE, Sleeping,
+            nullptr, nullptr, 0);
+    V = atomic_exchange(&M, Sleeping, memory_order_acquire);
+  }
 }
 
-void BlockingMutex::unlock() {
-  atomic_u32 *M = reinterpret_cast<atomic_u32 *>(&OpaqueStorage);
-  const u32 V = atomic_exchange(M, MtxUnlocked, memory_order_release);
-  DCHECK_NE(V, MtxUnlocked);
-  if (V == MtxSleeping)
-    syscall(SYS_futex, reinterpret_cast<uptr>(OpaqueStorage),
-            FUTEX_WAKE_PRIVATE, 1, nullptr, nullptr, 0);
+void HybridMutex::unlock() {
+  if (atomic_fetch_sub(&M, 1U, memory_order_release) != Locked) {
+    atomic_store(&M, Unlocked, memory_order_release);
+    syscall(SYS_futex, reinterpret_cast<uptr>(&M), FUTEX_WAKE_PRIVATE, 1,
+            nullptr, nullptr, 0);
+  }
 }
 
 u64 getMonotonicTime() {
   timespec TS;
   clock_gettime(CLOCK_MONOTONIC, &TS);
   return static_cast<u64>(TS.tv_sec) * (1000ULL * 1000 * 1000) +
          static_cast<u64>(TS.tv_nsec);
 }
#endif // defined(SYS_getrandom)
   if (FileDesc == -1)
     return false;
   ReadBytes = read(FileDesc, Buffer, Length);
   close(FileDesc);
   return (ReadBytes == static_cast<ssize_t>(Length));
 }
 
 void outputRaw(const char *Buffer) {
-  static StaticSpinMutex Mutex;
-  SpinMutexLock L(&Mutex);
+  static HybridMutex Mutex;
+  ScopedLock L(Mutex);
   write(2, Buffer, strlen(Buffer));
 }
 
 extern "C" WEAK void android_set_abort_message(const char *);
 
 void setAbortMessage(const char *Message) {
   if (&android_set_abort_message)
     android_set_abort_message(Message);
 }
 
 } // namespace scudo
 
 #endif // SCUDO_LINUX

compiler-rt/trunk/lib/scudo/standalone/mutex.h

 //===-- mutex.h -------------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef SCUDO_MUTEX_H_
 #define SCUDO_MUTEX_H_
 
 #include "atomic_helpers.h"
 #include "common.h"
 
+#include <string.h>
+
+#if SCUDO_FUCHSIA
+#include <lib/sync/mutex.h> // for sync_mutex_t
+#endif
+
 namespace scudo {
 
-class StaticSpinMutex {
+class HybridMutex {
 public:
-  void init() { atomic_store_relaxed(&State, 0); }
-
-  void lock() {
+  void init() { memset(this, 0, sizeof(*this)); }
+  bool tryLock();
+  NOINLINE void lock() {
     if (tryLock())
       return;
-    lockSlow();
-  }
-
-  bool tryLock() {
-    return atomic_exchange(&State, 1, memory_order_acquire) == 0;
-  }
-
-  void unlock() { atomic_store(&State, 0, memory_order_release); }
-
-  void checkLocked() { CHECK_EQ(atomic_load_relaxed(&State), 1); }
-
-private:
-  atomic_u8 State;
-
-  void NOINLINE lockSlow() {
-    for (u32 I = 0;; I++) {
-      if (I < 10)
-        yieldProcessor(10);
-      else
-        yieldPlatform();
-      if (atomic_load_relaxed(&State) == 0 &&
-          atomic_exchange(&State, 1, memory_order_acquire) == 0)
+      // The compiler may try to fully unroll the loop, ending up in a
+      // NumberOfTries*NumberOfYields block of pauses mixed with tryLocks. This
+      // is large, ugly and unneeded, a compact loop is better for our purpose
+      // here. Use a pragma to tell the compiler not to unroll the loop.
+#ifdef __clang__
+#pragma nounroll
+#endif
+    for (u8 I = 0U; I < NumberOfTries; I++) {
+      yieldProcessor(NumberOfYields);
+      if (tryLock())
         return;
     }
+    lockSlow();
   }
-};
-
-class SpinMutex : public StaticSpinMutex {
-public:
-  SpinMutex() { init(); }
+  void unlock();
 
 private:
-  SpinMutex(const SpinMutex &) = delete;
-  void operator=(const SpinMutex &) = delete;
-};
+  static constexpr u8 NumberOfTries = 10U;
+  static constexpr u8 NumberOfYields = 10U;
 
-class BlockingMutex {
-public:
-  explicit constexpr BlockingMutex(LinkerInitialized) : OpaqueStorage{} {}
-  BlockingMutex() { memset(this, 0, sizeof(*this)); }
-  void lock();
-  void unlock();
-  void checkLocked() {
-    atomic_u32 *M = reinterpret_cast<atomic_u32 *>(&OpaqueStorage);
-    CHECK_NE(MtxUnlocked, atomic_load_relaxed(M));
-  }
+#if SCUDO_LINUX
+  atomic_u32 M;
+#elif SCUDO_FUCHSIA
+  sync_mutex_t M;
+#endif
 
-private:
-  enum MutexState { MtxUnlocked = 0, MtxLocked = 1, MtxSleeping = 2 };
-  uptr OpaqueStorage[1];
+  void lockSlow();
 };
 
-template <typename MutexType> class GenericScopedLock {
+class ScopedLock {
 public:
-  explicit GenericScopedLock(MutexType *M) : Mutex(M) { Mutex->lock(); }
-  ~GenericScopedLock() { Mutex->unlock(); }
+  explicit ScopedLock(HybridMutex &M) : Mutex(M) { Mutex.lock(); }
+  ~ScopedLock() { Mutex.unlock(); }
 
 private:
-  MutexType *Mutex;
+  HybridMutex &Mutex;
 
-  GenericScopedLock(const GenericScopedLock &) = delete;
-  void operator=(const GenericScopedLock &) = delete;
+  ScopedLock(const ScopedLock &) = delete;
+  void operator=(const ScopedLock &) = delete;
 };
 
-typedef GenericScopedLock<StaticSpinMutex> SpinMutexLock;
-typedef GenericScopedLock<BlockingMutex> BlockingMutexLock;
-
 } // namespace scudo
 
 #endif // SCUDO_MUTEX_H_

compiler-rt/trunk/lib/scudo/standalone/primary32.h

for (uptr I = 0; I < NumRegions; I++)
       if (PossibleRegions[I])
         unmap(reinterpret_cast<void *>(I * RegionSize), RegionSize);
     PossibleRegions.unmapTestOnly();
   }
 
   TransferBatch *popBatch(CacheT *C, uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     SizeClassInfo *Sci = getSizeClassInfo(ClassId);
-    BlockingMutexLock L(&Sci->Mutex);
+    ScopedLock L(Sci->Mutex);
     TransferBatch *B = Sci->FreeList.front();
     if (B)
       Sci->FreeList.pop_front();
     else {
       B = populateFreeList(C, ClassId, Sci);
       if (UNLIKELY(!B))
         return nullptr;
     }
     DCHECK_GT(B->getCount(), 0);
     Sci->Stats.PoppedBlocks += B->getCount();
     return B;
   }
 
   void pushBatch(uptr ClassId, TransferBatch *B) {
     DCHECK_LT(ClassId, NumClasses);
     DCHECK_GT(B->getCount(), 0);
     SizeClassInfo *Sci = getSizeClassInfo(ClassId);
-    BlockingMutexLock L(&Sci->Mutex);
+    ScopedLock L(Sci->Mutex);
     Sci->FreeList.push_front(B);
     Sci->Stats.PushedBlocks += B->getCount();
     if (Sci->CanRelease)
       releaseToOSMaybe(Sci, ClassId);
   }
 
   void disable() {
     for (uptr I = 0; I < NumClasses; I++)
Printf("Stats: SizeClassAllocator32: %zuM mapped in %zu allocations; "
            TotalMapped >> 20, PoppedBlocks, PoppedBlocks - PushedBlocks);
     for (uptr I = 0; I < NumClasses; I++)
       printStats(I, 0);
   }
 
   void releaseToOS() {
     for (uptr I = 1; I < NumClasses; I++) {
       SizeClassInfo *Sci = getSizeClassInfo(I);
-      BlockingMutexLock L(&Sci->Mutex);
+      ScopedLock L(Sci->Mutex);
       releaseToOSMaybe(Sci, I, /*Force=*/true);
     }
   }
 
 private:
   static const uptr NumClasses = SizeClassMap::NumClasses;
   static const uptr RegionSize = 1UL << RegionSizeLog;
   static const uptr NumRegions = SCUDO_MMAP_RANGE_SIZE >> RegionSizeLog;
#endif
   struct ReleaseToOsInfo {
     uptr PushedBlocksAtLastRelease;
     uptr RangesReleased;
     uptr LastReleasedBytes;
     u64 LastReleaseAtNs;
   };
 
   struct ALIGNED(SCUDO_CACHE_LINE_SIZE) SizeClassInfo {
-    BlockingMutex Mutex;
+    HybridMutex Mutex;
     IntrusiveList<TransferBatch> FreeList;
     SizeClassStats Stats;
     bool CanRelease;
     u32 RandState;
     uptr AllocatedUser;
     ReleaseToOsInfo ReleaseInfo;
   };
   COMPILER_CHECK(sizeof(SizeClassInfo) % SCUDO_CACHE_LINE_SIZE == 0);
 
   uptr computeRegionId(uptr Mem) {
     const uptr Id = Mem >> RegionSizeLog;
     CHECK_LT(Id, NumRegions);
     return Id;
   }
 
   uptr allocateRegionSlow() {
     uptr MapSize = 2 * RegionSize;
     const uptr MapBase = reinterpret_cast<uptr>(
         map(nullptr, MapSize, "scudo:primary", MAP_ALLOWNOMEM));
     if (UNLIKELY(!MapBase))
       return 0;
     const uptr MapEnd = MapBase + MapSize;
     uptr Region = MapBase;
     if (isAligned(Region, RegionSize)) {
-      SpinMutexLock L(&RegionsStashMutex);
+      ScopedLock L(RegionsStashMutex);
       if (NumberOfStashedRegions < MaxStashedRegions)
         RegionsStash[NumberOfStashedRegions++] = MapBase + RegionSize;
       else
         MapSize = RegionSize;
     } else {
       Region = roundUpTo(MapBase, RegionSize);
       unmap(reinterpret_cast<void *>(MapBase), Region - MapBase);
       MapSize = RegionSize;
     }
     const uptr End = Region + MapSize;
     if (End != MapEnd)
       unmap(reinterpret_cast<void *>(End), MapEnd - End);
     return Region;
   }
 
   uptr allocateRegion(uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     uptr Region = 0;
     {
-      SpinMutexLock L(&RegionsStashMutex);
+      ScopedLock L(RegionsStashMutex);
       if (NumberOfStashedRegions > 0)
         Region = RegionsStash[--NumberOfStashedRegions];
     }
     if (!Region)
       Region = allocateRegionSlow();
     if (LIKELY(Region)) {
       if (ClassId) {
         const uptr RegionIndex = computeRegionId(Region);
#endif
   // Keep track of the lowest & highest regions allocated to avoid looping
   // through the whole NumRegions.
   uptr MinRegionIndex;
   uptr MaxRegionIndex;
   s32 ReleaseToOsIntervalMs;
   // Unless several threads request regions simultaneously from different size
   // classes, the stash rarely contains more than 1 entry.
   static constexpr uptr MaxStashedRegions = 4;
-  StaticSpinMutex RegionsStashMutex;
+  HybridMutex RegionsStashMutex;
   uptr NumberOfStashedRegions;
   uptr RegionsStash[MaxStashedRegions];
 };
 
 } // namespace scudo
 
 #endif // SCUDO_PRIMARY32_H_

compiler-rt/trunk/lib/scudo/standalone/primary64.h

void unmapTestOnly() {
     unmap(reinterpret_cast<void *>(PrimaryBase), PrimarySize, UNMAP_ALL, &Data);
     unmap(reinterpret_cast<void *>(RegionInfoArray),
           sizeof(RegionInfo) * NumClasses);
   }
 
   TransferBatch *popBatch(CacheT *C, uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     RegionInfo *Region = getRegionInfo(ClassId);
-    BlockingMutexLock L(&Region->Mutex);
+    ScopedLock L(Region->Mutex);
     TransferBatch *B = Region->FreeList.front();
     if (B)
       Region->FreeList.pop_front();
     else {
       B = populateFreeList(C, ClassId, Region);
       if (UNLIKELY(!B))
         return nullptr;
     }
     DCHECK_GT(B->getCount(), 0);
     Region->Stats.PoppedBlocks += B->getCount();
     return B;
   }
 
   void pushBatch(uptr ClassId, TransferBatch *B) {
     DCHECK_GT(B->getCount(), 0);
     RegionInfo *Region = getRegionInfo(ClassId);
-    BlockingMutexLock L(&Region->Mutex);
+    ScopedLock L(Region->Mutex);
     Region->FreeList.push_front(B);
     Region->Stats.PushedBlocks += B->getCount();
     if (Region->CanRelease)
       releaseToOSMaybe(Region, ClassId);
   }
 
   void disable() {
     for (uptr I = 0; I < NumClasses; I++)
void printStats() const {
 
     for (uptr I = 0; I < NumClasses; I++)
       printStats(I, 0);
   }
 
   void releaseToOS() {
     for (uptr I = 1; I < NumClasses; I++) {
       RegionInfo *Region = getRegionInfo(I);
-      BlockingMutexLock L(&Region->Mutex);
+      ScopedLock L(Region->Mutex);
       releaseToOSMaybe(Region, I, /*Force=*/true);
     }
   }
 
 private:
   static const uptr RegionSize = 1UL << RegionSizeLog;
   static const uptr NumClasses = SizeClassMap::NumClasses;
   static const uptr PrimarySize = RegionSize * NumClasses;
private:
   struct ReleaseToOsInfo {
     uptr PushedBlocksAtLastRelease;
     uptr RangesReleased;
     uptr LastReleasedBytes;
     u64 LastReleaseAtNs;
   };
 
   struct ALIGNED(SCUDO_CACHE_LINE_SIZE) RegionInfo {
-    BlockingMutex Mutex;
+    HybridMutex Mutex;
     IntrusiveList<TransferBatch> FreeList;
     RegionStats Stats;
     bool CanRelease;
     bool Exhausted;
     u32 RandState;
     uptr RegionBeg;
     uptr MappedUser;    // Bytes mapped for user memory.
     uptr AllocatedUser; // Bytes allocated for user memory.

compiler-rt/trunk/lib/scudo/standalone/quarantine.h

public:
   void put(CacheT *C, Callback Cb, Node *Ptr, uptr Size) {
     C->enqueue(Cb, Ptr, Size);
     if (C->getSize() > getCacheSize())
       drain(C, Cb);
   }
 
   void NOINLINE drain(CacheT *C, Callback Cb) {
     {
-      SpinMutexLock L(&CacheMutex);
+      ScopedLock L(CacheMutex);
       Cache.transfer(C);
     }
     if (Cache.getSize() > getMaxSize() && RecyleMutex.tryLock())
       recycle(atomic_load_relaxed(&MinSize), Cb);
   }
 
   void NOINLINE drainAndRecycle(CacheT *C, Callback Cb) {
     {
-      SpinMutexLock L(&CacheMutex);
+      ScopedLock L(CacheMutex);
       Cache.transfer(C);
     }
     RecyleMutex.lock();
     recycle(0, Cb);
   }
 
   void printStats() const {
     // It assumes that the world is stopped, just as the allocator's printStats.
     Printf("Quarantine limits: global: %zdM; thread local: %zdK\n",
            getMaxSize() >> 20, getCacheSize() >> 10);
     Cache.printStats();
   }
 
 private:
   // Read-only data.
-  alignas(SCUDO_CACHE_LINE_SIZE) StaticSpinMutex CacheMutex;
+  alignas(SCUDO_CACHE_LINE_SIZE) HybridMutex CacheMutex;
   CacheT Cache;
-  alignas(SCUDO_CACHE_LINE_SIZE) StaticSpinMutex RecyleMutex;
+  alignas(SCUDO_CACHE_LINE_SIZE) HybridMutex RecyleMutex;
   atomic_uptr MinSize;
   atomic_uptr MaxSize;
   alignas(SCUDO_CACHE_LINE_SIZE) atomic_uptr MaxCacheSize;
 
   void NOINLINE recycle(uptr MinSize, Callback Cb) {
     CacheT Tmp;
     Tmp.init();
     {
-      SpinMutexLock L(&CacheMutex);
+      ScopedLock L(CacheMutex);
       // Go over the batches and merge partially filled ones to
       // save some memory, otherwise batches themselves (since the memory used
       // by them is counted against quarantine limit) can overcome the actual
       // user's quarantined chunks, which diminishes the purpose of the
       // quarantine.
       const uptr CacheSize = Cache.getSize();
       const uptr OverheadSize = Cache.getOverheadSize();
       DCHECK_GE(CacheSize, OverheadSize);

compiler-rt/trunk/lib/scudo/standalone/secondary.h

public:
   void enable() { Mutex.unlock(); }
 
   template <typename F> void iterateOverBlocks(F Callback) const {
     for (LargeBlock::Header *H = Tail; H != nullptr; H = H->Prev)
       Callback(reinterpret_cast<uptr>(H) + LargeBlock::getHeaderSize());
   }
 
 private:
-  StaticSpinMutex Mutex;
+  HybridMutex Mutex;
   LargeBlock::Header *Tail;
   uptr AllocatedBytes;
   uptr FreedBytes;
   uptr LargestSize;
   u32 NumberOfAllocs;
   u32 NumberOfFrees;
   LocalStats Stats;
 };
 
 } // namespace scudo
 
 #endif // SCUDO_SECONDARY_H_

compiler-rt/trunk/lib/scudo/standalone/secondary.cc

const uptr Ptr =
       reinterpret_cast<uptr>(map(reinterpret_cast<void *>(CommitBase),
                                  CommitSize, "scudo:secondary", 0, &Data));
   LargeBlock::Header *H = reinterpret_cast<LargeBlock::Header *>(Ptr);
   H->MapBase = MapBase;
   H->MapSize = MapEnd - MapBase;
   H->BlockEnd = CommitBase + CommitSize;
   H->Data = Data;
   {
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     if (!Tail) {
       Tail = H;
     } else {
       Tail->Next = H;
       H->Prev = Tail;
       Tail = H;
     }
     AllocatedBytes += CommitSize;
     if (LargestSize < CommitSize)
       LargestSize = CommitSize;
     NumberOfAllocs++;
     Stats.add(StatAllocated, CommitSize);
     Stats.add(StatMapped, H->MapSize);
   }
   if (BlockEnd)
     *BlockEnd = CommitBase + CommitSize;
   return reinterpret_cast<void *>(Ptr + LargeBlock::getHeaderSize());
 }
 
 void MapAllocator::deallocate(void *Ptr) {
   LargeBlock::Header *H = LargeBlock::getHeader(Ptr);
   {
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     LargeBlock::Header *Prev = H->Prev;
     LargeBlock::Header *Next = H->Next;
     if (Prev) {
       CHECK_EQ(Prev->Next, H);
       Prev->Next = Next;
     }
     if (Next) {
       CHECK_EQ(Next->Prev, H);

compiler-rt/trunk/lib/scudo/standalone/stats.h

void initLinkerInitialized() {
     Prev = this;
   }
   void init() {
     memset(this, 0, sizeof(*this));
     initLinkerInitialized();
   }
 
   void link(LocalStats *S) {
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     S->Next = Next;
     S->Prev = this;
     Next->Prev = S;
     Next = S;
   }
 
   void unlink(LocalStats *S) {
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     S->Prev->Next = S->Next;
     S->Next->Prev = S->Prev;
     for (uptr I = 0; I < StatCount; I++)
       add(static_cast<StatType>(I), S->get(static_cast<StatType>(I)));
   }
 
   void get(uptr *S) const {
     memset(S, 0, StatCount * sizeof(uptr));
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     const LocalStats *Stats = this;
     for (;;) {
       for (uptr I = 0; I < StatCount; I++)
         S[I] += Stats->get(static_cast<StatType>(I));
       Stats = Stats->Next;
       if (Stats == this)
         break;
     }
     // All stats must be non-negative.
     for (uptr I = 0; I < StatCount; I++)
       S[I] = static_cast<sptr>(S[I]) >= 0 ? S[I] : 0;
   }
 
 private:
-  mutable StaticSpinMutex Mutex;
+  mutable HybridMutex Mutex;
 };
 
 } // namespace scudo
 
 #endif // SCUDO_STATS_H_

compiler-rt/trunk/lib/scudo/standalone/tests/map_test.cc

 //===-- map_test.cc ---------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "common.h"
 
 #include "gtest/gtest.h"
 
 #include <string.h>
+#include <unistd.h>
 
 static const char *MappingName = "scudo:test";
 
+TEST(ScudoMapTest, PageSize) {
+  EXPECT_EQ(scudo::getPageSizeCached(),
+            static_cast<scudo::uptr>(getpagesize()));
+}
+
 TEST(ScudoMapTest, MapNoAccessUnmap) {
   const scudo::uptr Size = 4 * scudo::getPageSizeCached();
   scudo::MapPlatformData Data = {};
   void *P = scudo::map(nullptr, Size, MappingName, MAP_NOACCESS, &Data);
   EXPECT_NE(P, nullptr);
   EXPECT_DEATH(memset(P, 0xaa, Size), "");
   scudo::unmap(P, Size, UNMAP_ALL, &Data);
 }

compiler-rt/trunk/lib/scudo/standalone/tests/mutex_test.cc

 //===-- mutex_test.cc -------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "mutex.h"
 
 #include "gtest/gtest.h"
 
 #include <string.h>
 
-template <typename MutexType> class TestData {
+class TestData {
 public:
-  explicit TestData(MutexType *M) : Mutex(M) {
+  explicit TestData(scudo::HybridMutex &M) : Mutex(M) {
     for (scudo::u32 I = 0; I < Size; I++)
       Data[I] = 0;
   }
 
   void write() {
-    Lock L(Mutex);
+    scudo::ScopedLock L(Mutex);
     T V0 = Data[0];
     for (scudo::u32 I = 0; I < Size; I++) {
       EXPECT_EQ(Data[I], V0);
       Data[I]++;
     }
   }
 
   void tryWrite() {
-    if (!Mutex->tryLock())
+    if (!Mutex.tryLock())
       return;
     T V0 = Data[0];
     for (scudo::u32 I = 0; I < Size; I++) {
       EXPECT_EQ(Data[I], V0);
       Data[I]++;
     }
-    Mutex->unlock();
+    Mutex.unlock();
   }
 
   void backoff() {
     volatile T LocalData[Size] = {};
     for (scudo::u32 I = 0; I < Size; I++) {
       LocalData[I]++;
       EXPECT_EQ(LocalData[I], 1U);
     }
   }
 
 private:
-  typedef scudo::GenericScopedLock<MutexType> Lock;
   static const scudo::u32 Size = 64U;
   typedef scudo::u64 T;
-  MutexType *Mutex;
+  scudo::HybridMutex &Mutex;
   ALIGNED(SCUDO_CACHE_LINE_SIZE) T Data[Size];
 };
 
 const scudo::u32 NumberOfThreads = 8;
 #if SCUDO_DEBUG
 const scudo::u32 NumberOfIterations = 4 * 1024;
 #else
 const scudo::u32 NumberOfIterations = 16 * 1024;
 #endif
 
-template <typename MutexType> static void *lockThread(void *Param) {
-  TestData<MutexType> *Data = reinterpret_cast<TestData<MutexType> *>(Param);
+static void *lockThread(void *Param) {
+  TestData *Data = reinterpret_cast<TestData *>(Param);
   for (scudo::u32 I = 0; I < NumberOfIterations; I++) {
     Data->write();
     Data->backoff();
   }
   return 0;
 }
 
-template <typename MutexType> static void *tryThread(void *Param) {
-  TestData<MutexType> *Data = reinterpret_cast<TestData<MutexType> *>(Param);
+static void *tryThread(void *Param) {
+  TestData *Data = reinterpret_cast<TestData *>(Param);
   for (scudo::u32 I = 0; I < NumberOfIterations; I++) {
     Data->tryWrite();
     Data->backoff();
   }
   return 0;
 }
 
-template <typename MutexType> static void checkLocked(MutexType *M) {
-  scudo::GenericScopedLock<MutexType> L(M);
-  M->checkLocked();
-}
-
-TEST(ScudoMutexTest, SpinMutex) {
-  scudo::SpinMutex M;
+TEST(ScudoMutexTest, Mutex) {
+  scudo::HybridMutex M;
   M.init();
-  TestData<scudo::SpinMutex> Data(&M);
+  TestData Data(M);
   pthread_t Threads[NumberOfThreads];
   for (scudo::u32 I = 0; I < NumberOfThreads; I++)
-    pthread_create(&Threads[I], 0, lockThread<scudo::SpinMutex>, &Data);
+    pthread_create(&Threads[I], 0, lockThread, &Data);
   for (scudo::u32 I = 0; I < NumberOfThreads; I++)
     pthread_join(Threads[I], 0);
 }
 
-TEST(ScudoMutexTest, SpinMutexTry) {
-  scudo::SpinMutex M;
+TEST(ScudoMutexTest, MutexTry) {
+  scudo::HybridMutex M;
   M.init();
-  TestData<scudo::SpinMutex> Data(&M);
-  pthread_t Threads[NumberOfThreads];
-  for (scudo::u32 I = 0; I < NumberOfThreads; I++)
-    pthread_create(&Threads[I], 0, tryThread<scudo::SpinMutex>, &Data);
-  for (scudo::u32 I = 0; I < NumberOfThreads; I++)
-    pthread_join(Threads[I], 0);
-}
-
-TEST(ScudoMutexTest, BlockingMutex) {
-  scudo::u64 MutexMemory[1024] = {};
-  scudo::BlockingMutex *M =
-      new (MutexMemory) scudo::BlockingMutex(scudo::LINKER_INITIALIZED);
-  TestData<scudo::BlockingMutex> Data(M);
+  TestData Data(M);
   pthread_t Threads[NumberOfThreads];
   for (scudo::u32 I = 0; I < NumberOfThreads; I++)
-    pthread_create(&Threads[I], 0, lockThread<scudo::BlockingMutex>, &Data);
+    pthread_create(&Threads[I], 0, tryThread, &Data);
   for (scudo::u32 I = 0; I < NumberOfThreads; I++)
     pthread_join(Threads[I], 0);
-  checkLocked(M);
 }

compiler-rt/trunk/lib/scudo/standalone/tsd.h

template <class Allocator> struct ALIGNED(SCUDO_CACHE_LINE_SIZE) TSD {
   INLINE void lock() {
     atomic_store_relaxed(&Precedence, 0);
     Mutex.lock();
   }
   INLINE void unlock() { Mutex.unlock(); }
   INLINE uptr getPrecedence() { return atomic_load_relaxed(&Precedence); }
 
 private:
-  StaticSpinMutex Mutex;
+  HybridMutex Mutex;
   atomic_uptr Precedence;
 };
 
 } // namespace scudo
 
 #endif // SCUDO_TSD_H_

compiler-rt/trunk/lib/scudo/standalone/tsd_exclusive.h

ALWAYS_INLINE TSD<Allocator> *getTSDAndLock(bool *UnlockRequired) {
     DCHECK(FallbackTSD);
     FallbackTSD->lock();
     *UnlockRequired = true;
     return FallbackTSD;
   }
 
 private:
   void initOnceMaybe(Allocator *Instance) {
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     if (Initialized)
       return;
     initLinkerInitialized(Instance); // Sets Initialized.
   }
 
   // Using minimal initialization allows for global initialization while keeping
   // the thread specific structure untouched. The fallback structure will be
   // used instead.
   NOINLINE void initThread(Allocator *Instance, bool MinimalInit) {
     initOnceMaybe(Instance);
     if (MinimalInit)
       return;
     CHECK_EQ(
         pthread_setspecific(PThreadKey, reinterpret_cast<void *>(Instance)), 0);
     ThreadTSD.initLinkerInitialized(Instance);
     State = ThreadState::Initialized;
   }
 
   pthread_key_t PThreadKey;
   bool Initialized;
   TSD<Allocator> *FallbackTSD;
-  StaticSpinMutex Mutex;
+  HybridMutex Mutex;
   static THREADLOCAL ThreadState State;
   static THREADLOCAL TSD<Allocator> ThreadTSD;
 
   friend void teardownThread<Allocator>(void *Ptr);
 };
 
 template <class Allocator>
 THREADLOCAL TSD<Allocator> TSDRegistryExT<Allocator>::ThreadTSD;

compiler-rt/trunk/lib/scudo/standalone/tsd_shared.h

 #elif SCUDO_LINUX
     return ThreadTSD;
 #else
     return reinterpret_cast<TSD<Allocator> *>(pthread_getspecific(PThreadKey));
 #endif
   }
 
   void initOnceMaybe(Allocator *Instance) {
-    SpinMutexLock L(&Mutex);
+    ScopedLock L(Mutex);
     if (Initialized)
       return;
     initLinkerInitialized(Instance); // Sets Initialized.
   }
 
   NOINLINE void initThread(Allocator *Instance) {
     initOnceMaybe(Instance);
     // Initial context assignment is done in a plain round-robin fashion.
#endif
 
   pthread_key_t PThreadKey;
   atomic_u32 CurrentIndex;
   u32 NumberOfTSDs;
   TSD<Allocator> *TSDs;
   u32 NumberOfCoPrimes;
   u32 CoPrimes[MaxTSDCount];
   bool Initialized;
-  StaticSpinMutex Mutex;
+  HybridMutex Mutex;
 #if SCUDO_LINUX && !SCUDO_ANDROID
   static THREADLOCAL TSD<Allocator> *ThreadTSD;
 #endif
 };
 
 #if SCUDO_LINUX && !SCUDO_ANDROID
 template <class Allocator, u32 MaxTSDCount>
 THREADLOCAL TSD<Allocator>
     *TSDRegistrySharedT<Allocator, MaxTSDCount>::ThreadTSD;
 #endif
 
 } // namespace scudo
 
 #endif // SCUDO_TSD_SHARED_H_