[sanitizer] Re-introduce kUseSeparateSizeClassForBatch for the 32-bit Primary

[cryptoad]

Summary

Currently TransferBatch are located within the same memory regions as
"regular" chunks. This is not ideal for security: they make for an interesting
target to overwrite, and are not protected by the frontend (namely, Scudo).

To solve this, we re-introduce kUseSeparateSizeClassForBatch for the 32-bit
Primary allowing for TransferBatch to end up in their own memory region.
Currently only Scudo would use this new feature, the default behavior remains
unchanged. The separate kBatchClassID was used for a brief period of time
previously but removed when the 64-bit ended up using the "free array".

[alekseyshl]

lib/sanitizer_common/sanitizer_allocator_local_cache.h

132

Do we really need all these types and consts to be public?

232

Add {} around in and else parts.

lib/sanitizer_common/sanitizer_allocator_primary32.h

103–104

Shouldn't it be also guarded by kUseSeparateSizeClassForBatch flag?

lib/sanitizer_common/sanitizer_allocator_size_class_map.h

137

So now we have two unused classes at the top? Why kLargestClassID was not adjusted then? Many other places depend on kNumClasses, including asan_allocator, I'm feeling not that easy about this change, how these places are affected?

150

Wouldn't these checks affect all other allocator's performance? From your earlier measurements, I recall that Size, ClassID and MaxCachedHint are pretty hot functions.

206

Why this changed? Everywhere else iterating classes we use kNumClasses.

[cryptoad]

lib/sanitizer_common/sanitizer_allocator_local_cache.h

132

That's an interesting point.
You can see below that the private: was commented out. I am not sure when or why, but it's all public.
Do you want me to try and reintroduce it an see what we can put in there?

232

Will do.

lib/sanitizer_common/sanitizer_allocator_primary32.h

103–104

In my opinion, even when not using a separate size class for batches, this should return kBatchSize for consistency.

lib/sanitizer_common/sanitizer_allocator_size_class_map.h

137

kLargestClassID was indeed not adjusted after the removal of kBatchClassID. It should have been kNumClasses - 1, but was left - 2.
It ended up not having any impact as it was only used in a test (SizeClassAllocatorGetBlockBeginStress), which means we didn't stress the last class.
In the end there are 2 "unused" classes when not using separate batches: 0 and kBatchClassID.
I tried to repurpose class 0, but ended up hitting some snags (for example in a ByteMap 0 means not-ours as opposed to being region 0), so I went back to getting an extra class past the end.

Here is a sample output of a SizeClassMap::Print():

c00 => s: 0 diff: +0 00% l 0 cached: 0 0; id 0
c01 => s: 16 diff: +16 00% l 4 cached: 64 1024; id 1
c02 => s: 32 diff: +16 100% l 5 cached: 64 2048; id 2
c03 => s: 48 diff: +16 50% l 5 cached: 64 3072; id 3
...
c50 => s: 98304 diff: +16384 20% l 16 cached: 1 98304; id 50
c51 => s: 114688 diff: +16384 16% l 16 cached: 1 114688; id 51

c52 => s: 131072 diff: +16384 14% l 17 cached: 1 131072; id 52

c53 => s: 512 diff: +0 00% l 0 cached: 16 8192; id 20

I didn't hit any snag anywhere, all tests pass for check-asan, check-sanitizer, etc.

It's still fair game to loop on kNumClasses anywhere, as long as there is no assumption that kNumClasses - 1 is the largest class.
As far as I can tell, no code anywhere is making that assumption.

150

This is indeed the case, but we cached the hottest ones in the per-class arrays (in class_size and max_count), which is done on cache initialization (once per thread). The overhead of the other calls shouldn't be a problem.

206

The issue is that kBatchClassID doesn't follow the same logic as the other class: its size is not larger than the one of the previous class, and it is meant to allocate only a single size. Meaning the test checking that size minus 1 still belongs to that class wouldn't pass.
In that sense, the validation code doesn't make sense for that particular class.

[alekseyshl]

Ok, overall, I have no better idea how to implement it, so let's iron out minor details and get it in.

lib/sanitizer_common/sanitizer_allocator_local_cache.h

132

Yes, if it is not too much of a burden, it would be great to make all non-API stuff private.

lib/sanitizer_common/sanitizer_allocator_primary32.h

103–104

Maybe yes, it does not matter that much. Now I am more concerned with ClassID(ClassIdToSize(kBatchClassID)) != kBatchClassID, although I cannot point out where exactly it might be a problem.

lib/sanitizer_common/sanitizer_allocator_size_class_map.h

150

It would be great if you could run that perf test with these changes, I'm still a bit concerned.

[cryptoad]

lib/sanitizer_common/sanitizer_allocator_local_cache.h

132

Working on it.

lib/sanitizer_common/sanitizer_allocator_primary32.h

103–104

It does show in the SizeClassMap::Print:

c53 => s: 512 diff: +0 00% l 0 cached: 16 8192; id 20

53 is the kBatchClassID, and the ClassID(ClassIdToSize(kBatchClassID)) is 20.
I didn't think much of it, it could be solved with a special case in ClassID.

lib/sanitizer_common/sanitizer_allocator_size_class_map.h

150

Will do.

206

Revisiting this: I can loop on kNumClasses and make a special case for kBatchClassID. Validate is only a test/debug functionality anyway.

[cryptoad]

Re-introducing private: in SizeClassAllocator*LocalCache.
Adding {} where requested.

[cryptoad]

Making kNumClasses more consistent between the SizeClassAllocator*LocalCache.

[alekseyshl]

lib/sanitizer_common/sanitizer_allocator_primary32.h

103–104

How exactly can it be solved there? It does not know what these 8KB of memory are going to be used for, for the transfer batch or is it a regular user allocation. Anyway, it's more of a hypothetical problem at this point.

lib/sanitizer_common/sanitizer_allocator_size_class_map.h

206

That would be a better way to handle it, please do.

[cryptoad]

Changing SizeClassMap::Validate to loop on kNumClasses.
Performance testing is the last item remaining.

[cryptoad]

Latest piece of information: this patch doesn't seem to have a meaningful impact on performances, whether it be single or multi threaded for the 64-bit allocator and 32-bit allocator without batch separation.
For the 32-bit allocator with batch separation enabled (eg: Scudo), this translates to a performance loss of about 2% for a single threaded pure allocation benchmark. A bit less on multi-threaded benchmarks.
This is expected as we are now allocating batches instead of re-using a chunk, but is required to keep the 32-bit allocator safe.
This also translate in a 1MB RSS increase due to the new region allocated for the batches. This is not expected to grow much higher as a single 1MB region can hold 2048 batches (for 512bytes batches).

lib/sanitizer_common/sanitizer_allocator_local_cache.h

 struct SizeClassAllocatorLocalCache
     : SizeClassAllocator::AllocatorCache {
 };
 
 // Cache used by SizeClassAllocator64.
 template <class SizeClassAllocator>
 struct SizeClassAllocator64LocalCache {
   typedef SizeClassAllocator Allocator;
-  static const uptr kNumClasses = SizeClassAllocator::kNumClasses;
-  typedef typename Allocator::SizeClassMapT SizeClassMap;
-  typedef typename Allocator::CompactPtrT CompactPtrT;
 
   void Init(AllocatorGlobalStats *s) {
     stats_.Init();
     if (s)
       s->Register(&stats_);
   }
 
   void Destroy(SizeClassAllocator *allocator, AllocatorGlobalStats *s) {
if (UNLIKELY(c->count == c->max_count))
       Drain(c, allocator, class_id, c->max_count / 2);
     CompactPtrT chunk = allocator->PointerToCompactPtr(
         allocator->GetRegionBeginBySizeClass(class_id),
         reinterpret_cast<uptr>(p));
     c->chunks[c->count++] = chunk;
   }
 
   void Drain(SizeClassAllocator *allocator) {
-    for (uptr class_id = 0; class_id < kNumClasses; class_id++) {
-      PerClass *c = &per_class_[class_id];
+    for (uptr i = 0; i < kNumClasses; i++) {
+      PerClass *c = &per_class_[i];
       while (c->count > 0)
-        Drain(c, allocator, class_id, c->count);
+        Drain(c, allocator, i, c->count);
     }
   }
 
-  // private:
+ private:
+  typedef typename Allocator::SizeClassMapT SizeClassMap;
+  static const uptr kNumClasses = SizeClassMap::kNumClasses;
+  typedef typename Allocator::CompactPtrT CompactPtrT;
+
   struct PerClass {
     u32 count;
     u32 max_count;
     uptr class_size;
     CompactPtrT chunks[2 * SizeClassMap::kMaxNumCachedHint];
   };
   PerClass per_class_[kNumClasses];
   AllocatorStats stats_;
 
   void InitCache() {
-    if (per_class_[1].max_count)
+    if (LIKELY(per_class_[1].max_count))
       return;
     for (uptr i = 0; i < kNumClasses; i++) {
       PerClass *c = &per_class_[i];
       c->max_count = 2 * SizeClassMap::MaxCachedHint(i);
       c->class_size = Allocator::ClassIdToSize(i);
     }
   }
 
NOINLINE void Drain(PerClass *c, SizeClassAllocator *allocator, uptr class_id,
     allocator->ReturnToAllocator(&stats_, class_id,
                                  &c->chunks[first_idx_to_drain], count);
   }
 };
 
 // Cache used by SizeClassAllocator32.
 template <class SizeClassAllocator>
 struct SizeClassAllocator32LocalCache {
   typedef SizeClassAllocator Allocator;
   typedef typename Allocator::TransferBatch TransferBatch;
-  static const uptr kNumClasses = SizeClassAllocator::kNumClasses;
 
   void Init(AllocatorGlobalStats *s) {
     stats_.Init();
     if (s)
       s->Register(&stats_);
   }
 
+  // Returns a TransferBatch suitable for class_id.
+  TransferBatch *CreateBatch(uptr class_id, SizeClassAllocator *allocator,
+                             TransferBatch *b) {
+    if (uptr batch_class_id = per_class_[class_id].batch_class_id)
+      return (TransferBatch*)Allocate(allocator, batch_class_id);
+    return b;
+  }
+
+  // Destroys TransferBatch b.
+  void DestroyBatch(uptr class_id, SizeClassAllocator *allocator,
+                    TransferBatch *b) {
+    if (uptr batch_class_id = per_class_[class_id].batch_class_id)
+      Deallocate(allocator, batch_class_id, b);
+  }
+
   void Destroy(SizeClassAllocator *allocator, AllocatorGlobalStats *s) {
     Drain(allocator);
     if (s)
       s->Unregister(&stats_);
   }
 
   void *Allocate(SizeClassAllocator *allocator, uptr class_id) {
     CHECK_NE(class_id, 0UL);
void Deallocate(SizeClassAllocator *allocator, uptr class_id, void *p) {
     stats_.Sub(AllocatorStatAllocated, c->class_size);
     CHECK_NE(c->max_count, 0UL);
     if (UNLIKELY(c->count == c->max_count))
       Drain(allocator, class_id);
     c->batch[c->count++] = p;
   }
 
   void Drain(SizeClassAllocator *allocator) {
-    for (uptr class_id = 0; class_id < kNumClasses; class_id++) {
-      PerClass *c = &per_class_[class_id];
+    for (uptr i = 0; i < kNumClasses; i++) {
+      PerClass *c = &per_class_[i];
       while (c->count > 0)
-        Drain(allocator, class_id);
+        Drain(allocator, i);
     }
   }
 
-  // private:
-  typedef typename SizeClassAllocator::SizeClassMapT SizeClassMap;
+ private:
+  typedef typename Allocator::SizeClassMapT SizeClassMap;
+  static const uptr kBatchClassID = SizeClassMap::kBatchClassID;
+  static const uptr kNumClasses = SizeClassMap::kNumClasses;
+  // If kUseSeparateSizeClassForBatch is true, all TransferBatch objects are
+  // allocated from kBatchClassID size class (except for those that are needed
+  // for kBatchClassID itself). The goal is to have TransferBatches in a totally
+  // different region of RAM to improve security.
+  static const bool kUseSeparateSizeClassForBatch =
+      Allocator::kUseSeparateSizeClassForBatch;
+
   struct PerClass {
     uptr count;
     uptr max_count;
     uptr class_size;
-    uptr class_id_for_transfer_batch;
+    uptr batch_class_id;
     void *batch[2 * TransferBatch::kMaxNumCached];
   };
   PerClass per_class_[kNumClasses];
   AllocatorStats stats_;
 
   void InitCache() {
-    if (per_class_[1].max_count)
+    if (LIKELY(per_class_[1].max_count))
       return;
-    // TransferBatch class is declared in SizeClassAllocator.
-    uptr class_id_for_transfer_batch =
-        SizeClassMap::ClassID(sizeof(TransferBatch));
+    const uptr batch_class_id = SizeClassMap::ClassID(sizeof(TransferBatch));
     for (uptr i = 0; i < kNumClasses; i++) {
       PerClass *c = &per_class_[i];
       uptr max_cached = TransferBatch::MaxCached(i);
       c->max_count = 2 * max_cached;
       c->class_size = Allocator::ClassIdToSize(i);
-      // We transfer chunks between central and thread-local free lists in
-      // batches. For small size classes we allocate batches separately. For
-      // large size classes we may use one of the chunks to store the batch.
-      // sizeof(TransferBatch) must be a power of 2 for more efficient
-      // allocation.
-      c->class_id_for_transfer_batch = (c->class_size <
+      // Precompute the class id to use to store batches for the current class
+      // id. 0 means the class size is large enough to store a batch within one
+      // of the chunks. If using a separate size class, it will always be
+      // kBatchClassID, except for kBatchClassID itself.
+      if (kUseSeparateSizeClassForBatch) {
+        c->batch_class_id = (i == kBatchClassID) ? 0 : kBatchClassID;
+      } else {
+        c->batch_class_id = (c->class_size <
           TransferBatch::AllocationSizeRequiredForNElements(max_cached)) ?
-              class_id_for_transfer_batch : 0;
+              batch_class_id : 0;
       }
     }
-
-  // Returns a TransferBatch suitable for class_id.
-  // For small size classes allocates the batch from the allocator.
-  // For large size classes simply returns b.
-  TransferBatch *CreateBatch(uptr class_id, SizeClassAllocator *allocator,
-                             TransferBatch *b) {
-    if (uptr batch_class_id = per_class_[class_id].class_id_for_transfer_batch)
-      return (TransferBatch*)Allocate(allocator, batch_class_id);
-    return b;
-  }
-
-  // Destroys TransferBatch b.
-  // For small size classes deallocates b to the allocator.
-  // Does notthing for large size classes.
-  void DestroyBatch(uptr class_id, SizeClassAllocator *allocator,
-                    TransferBatch *b) {
-    if (uptr batch_class_id = per_class_[class_id].class_id_for_transfer_batch)
-      Deallocate(allocator, batch_class_id, b);
   }
 
   NOINLINE bool Refill(SizeClassAllocator *allocator, uptr class_id) {
     InitCache();
     PerClass *c = &per_class_[class_id];
     TransferBatch *b = allocator->AllocateBatch(&stats_, this, class_id);
     if (UNLIKELY(!b))
       return false;

lib/sanitizer_common/sanitizer_allocator_primary32.h

 // UserChunk1 .. UserChunkN <gap> MetaChunkN .. MetaChunk1
 //
 // In order to avoid false sharing the objects of this class should be
 // chache-line aligned.
 
 struct SizeClassAllocator32FlagMasks {  //  Bit masks.
   enum {
     kRandomShuffleChunks = 1,
+    kUseSeparateSizeClassForBatch = 2,
   };
 };
 
 template <class Params>
 class SizeClassAllocator32 {
  public:
   static const uptr kSpaceBeg = Params::kSpaceBeg;
   static const u64 kSpaceSize = Params::kSpaceSize;
   static const uptr kMetadataSize = Params::kMetadataSize;
   typedef typename Params::SizeClassMap SizeClassMap;
   static const uptr kRegionSizeLog = Params::kRegionSizeLog;
   typedef typename Params::ByteMap ByteMap;
   typedef typename Params::MapUnmapCallback MapUnmapCallback;
 
-  static const bool kRandomShuffleChunks =
-      Params::kFlags & SizeClassAllocator32FlagMasks::kRandomShuffleChunks;
+  static const bool kRandomShuffleChunks = Params::kFlags &
+      SizeClassAllocator32FlagMasks::kRandomShuffleChunks;
+  static const bool kUseSeparateSizeClassForBatch = Params::kFlags &
+      SizeClassAllocator32FlagMasks::kUseSeparateSizeClassForBatch;
 
   struct TransferBatch {
     static const uptr kMaxNumCached = SizeClassMap::kMaxNumCachedHint - 2;
     void SetFromArray(uptr region_beg_unused, void *batch[], uptr count) {
       count_ = count;
       CHECK_LE(count_, kMaxNumCached);
       for (uptr i = 0; i < count; i++)
         batch_[i] = batch[i];
public:
 
    private:
     uptr count_;
     void *batch_[kMaxNumCached];
   };
 
   static const uptr kBatchSize = sizeof(TransferBatch);
   COMPILER_CHECK((kBatchSize & (kBatchSize - 1)) == 0);
-  COMPILER_CHECK(sizeof(TransferBatch) ==
-                 SizeClassMap::kMaxNumCachedHint * sizeof(uptr));
+  COMPILER_CHECK(kBatchSize == SizeClassMap::kMaxNumCachedHint * sizeof(uptr));
 
   static uptr ClassIdToSize(uptr class_id) {
-    return SizeClassMap::Size(class_id);
+    return (class_id == SizeClassMap::kBatchClassID) ?
+        kBatchSize : SizeClassMap::Size(class_id);
   }
 
   typedef SizeClassAllocator32<Params> ThisT;
   typedef SizeClassAllocator32LocalCache<ThisT> AllocatorCache;
 
   void Init(s32 release_to_os_interval_ms) {
     possible_regions.TestOnlyInit();
     internal_memset(size_class_info_array, 0, sizeof(size_class_info_array));
NOINLINE TransferBatch *AllocateBatch(AllocatorStats *stat, AllocatorCache *c,
     TransferBatch *b = sci->free_list.front();
     sci->free_list.pop_front();
     return b;
   }
 
   NOINLINE void DeallocateBatch(AllocatorStats *stat, uptr class_id,
                                 TransferBatch *b) {
     CHECK_LT(class_id, kNumClasses);
+    CHECK_GT(b->Count(), 0);
     SizeClassInfo *sci = GetSizeClassInfo(class_id);
     SpinMutexLock l(&sci->mutex);
-    CHECK_GT(b->Count(), 0);
     sci->free_list.push_front(b);
   }
 
   uptr GetRegionBeginBySizeClass(uptr class_id) { return 0; }
 
   bool PointerIsMine(const void *p) {
     uptr mem = reinterpret_cast<uptr>(p);
     if (mem < kSpaceBeg || mem >= kSpaceBeg + kSpaceSize)

lib/sanitizer_common/sanitizer_allocator_size_class_map.h

  public:
   // kMaxNumCachedHintT is a power of two. It serves as a hint
   // for the size of TransferBatch, the actual size could be a bit smaller.
   static const uptr kMaxNumCachedHint = kMaxNumCachedHintT;
   COMPILER_CHECK((kMaxNumCachedHint & (kMaxNumCachedHint - 1)) == 0);
 
   static const uptr kMaxSize = 1UL << kMaxSizeLog;
   static const uptr kNumClasses =
-      kMidClass + ((kMaxSizeLog - kMidSizeLog) << S) + 1;
+      kMidClass + ((kMaxSizeLog - kMidSizeLog) << S) + 1 + 1;
   static const uptr kLargestClassID = kNumClasses - 2;
+  static const uptr kBatchClassID = kNumClasses - 1;
   COMPILER_CHECK(kNumClasses >= 16 && kNumClasses <= 256);
   static const uptr kNumClassesRounded =
       kNumClasses <= 32  ? 32 :
       kNumClasses <= 64  ? 64 :
       kNumClasses <= 128 ? 128 : 256;
 
   static uptr Size(uptr class_id) {
+    // Estimate the result for kBatchClassID because this class does not know
+    // the exact size of TransferBatch. It's OK since we are using the actual
+    // sizeof(TransferBatch) where it matters.
+    if (UNLIKELY(class_id == kBatchClassID))
+      return kMaxNumCachedHint * sizeof(uptr);
     if (class_id <= kMidClass)
       return kMinSize * class_id;
     class_id -= kMidClass;
     uptr t = kMidSize << (class_id >> S);
     return t + (t >> S) * (class_id & M);
   }
 
   static uptr ClassID(uptr size) {
+    if (UNLIKELY(size > kMaxSize))
+      return 0;
     if (size <= kMidSize)
       return (size + kMinSize - 1) >> kMinSizeLog;
-    if (size > kMaxSize) return 0;
     uptr l = MostSignificantSetBitIndex(size);
     uptr hbits = (size >> (l - S)) & M;
     uptr lbits = size & ((1 << (l - S)) - 1);
     uptr l1 = l - kMidSizeLog;
     return kMidClass + (l1 << S) + hbits + (lbits > 0);
   }
 
   static uptr MaxCachedHint(uptr class_id) {
-    if (class_id == 0) return 0;
+    // Estimate the result for kBatchClassID because this class does not know
+    // the exact size of TransferBatch. We need to cache fewer batches than user
+    // chunks, so this number can be small.
+    if (UNLIKELY(class_id == kBatchClassID))
+      return 16;
+    if (UNLIKELY(class_id == 0))
+      return 0;
     uptr n = (1UL << kMaxBytesCachedLog) / Size(class_id);
     return Max<uptr>(1, Min(kMaxNumCachedHint, n));
   }
 
   static void Print() {
     uptr prev_s = 0;
     uptr total_cached = 0;
     for (uptr i = 0; i < kNumClasses; i++) {
       uptr s = Size(i);
       if (s >= kMidSize / 2 && (s & (s - 1)) == 0)
         Printf("\n");
       uptr d = s - prev_s;
       uptr p = prev_s ? (d * 100 / prev_s) : 0;
       uptr l = s ? MostSignificantSetBitIndex(s) : 0;
       uptr cached = MaxCachedHint(i) * s;
+      if (i == kBatchClassID)
+        d = p = l = 0;
       Printf("c%02zd => s: %zd diff: +%zd %02zd%% l %zd "
              "cached: %zd %zd; id %zd\n",
              i, Size(i), d, p, l, MaxCachedHint(i), cached, ClassID(s));
       total_cached += cached;
       prev_s = s;
     }
     Printf("Total cached: %zd\n", total_cached);
   }
 
   static void Validate() {
     for (uptr c = 1; c < kNumClasses; c++) {
       // Printf("Validate: c%zd\n", c);
       uptr s = Size(c);
       CHECK_NE(s, 0U);
+      if (c == kBatchClassID)
+        continue;
       CHECK_EQ(ClassID(s), c);
-      if (c != kNumClasses - 1)
+      if (c < kLargestClassID)
         CHECK_EQ(ClassID(s + 1), c + 1);
       CHECK_EQ(ClassID(s - 1), c);
-      if (c)
-        CHECK_GT(Size(c), Size(c-1));
+      CHECK_GT(Size(c), Size(c - 1));
     }
     CHECK_EQ(ClassID(kMaxSize + 1), 0);
 
     for (uptr s = 1; s <= kMaxSize; s++) {
       uptr c = ClassID(s);
       // Printf("s%zd => c%zd\n", s, c);
       CHECK_LT(c, kNumClasses);
       CHECK_GE(Size(c), s);
       if (c > 0)
-        CHECK_LT(Size(c-1), s);
+        CHECK_LT(Size(c - 1), s);
     }
   }
 };
 
 typedef SizeClassMap<3, 4, 8, 17, 128, 16> DefaultSizeClassMap;
 typedef SizeClassMap<3, 4, 8, 17, 64, 14> CompactSizeClassMap;
 typedef SizeClassMap<2, 5, 9, 16, 64, 14> VeryCompactSizeClassMap;

lib/sanitizer_common/tests/sanitizer_allocator_test.cc

 }
 #endif
 #endif
 
 TEST(SanitizerCommon, SizeClassAllocator32Compact) {
   TestSizeClassAllocator<Allocator32Compact>();
 }
 
+struct AP32SeparateBatches {
+  static const uptr kSpaceBeg = 0;
+  static const u64 kSpaceSize = kAddressSpaceSize;
+  static const uptr kMetadataSize = 16;
+  typedef DefaultSizeClassMap SizeClassMap;
+  static const uptr kRegionSizeLog = ::kRegionSizeLog;
+  typedef FlatByteMap<kFlatByteMapSize> ByteMap;
+  typedef NoOpMapUnmapCallback MapUnmapCallback;
+  static const uptr kFlags =
+      SizeClassAllocator32FlagMasks::kUseSeparateSizeClassForBatch;
+};
+typedef SizeClassAllocator32<AP32SeparateBatches> Allocator32SeparateBatches;
+
+TEST(SanitizerCommon, SizeClassAllocator32SeparateBatches) {
+  TestSizeClassAllocator<Allocator32SeparateBatches>();
+}
+
 template <class Allocator>
 void SizeClassAllocatorMetadataStress() {
   Allocator *a = new Allocator;
   a->Init(kReleaseToOSIntervalNever);
   SizeClassAllocatorLocalCache<Allocator> cache;
   memset(&cache, 0, sizeof(cache));
   cache.Init(0);
 

lib/scudo/scudo_allocator.h

 
 #if !SANITIZER_LINUX
 # error "The Scudo hardened allocator is currently only supported on Linux."
 #endif
 
 namespace __scudo {
 
 enum AllocType : u8 {
-  FromMalloc    = 0, // Memory block came from malloc, realloc, calloc, etc.
-  FromNew       = 1, // Memory block came from operator new.
-  FromNewArray  = 2, // Memory block came from operator new [].
-  FromMemalign  = 3, // Memory block came from memalign, posix_memalign, etc.
+  FromMalloc    = 0,  // Memory block came from malloc, realloc, calloc, etc.
+  FromNew       = 1,  // Memory block came from operator new.
+  FromNewArray  = 2,  // Memory block came from operator new [].
+  FromMemalign  = 3,  // Memory block came from memalign, posix_memalign, etc.
 };
 
 enum ChunkState : u8 {
   ChunkAvailable  = 0,
   ChunkAllocated  = 1,
   ChunkQuarantine = 2
 };
 
 // Our header requires 64 bits of storage. Having the offset saves us from
 // using functions such as GetBlockBegin, that is fairly costly. Our first
 // implementation used the MetaData as well, which offers the advantage of
 // being stored away from the chunk itself, but accessing it was costly as
 // well. The header will be atomically loaded and stored.
 typedef u64 PackedHeader;
 struct UnpackedHeader {
   u64 Checksum          : 16;
-  u64 SizeOrUnusedBytes : 19; // Size for Primary backed allocations, amount of
+  u64 SizeOrUnusedBytes : 19;  // Size for Primary backed allocations, amount of
                                // unused bytes in the chunk for Secondary ones.
   u64 FromPrimary       : 1;
-  u64 State             : 2;  // available, allocated, or quarantined
-  u64 AllocType         : 2;  // malloc, new, new[], or memalign
-  u64 Offset            : 16; // Offset from the beginning of the backend
+  u64 State             : 2;   // available, allocated, or quarantined
+  u64 AllocType         : 2;   // malloc, new, new[], or memalign
+  u64 Offset            : 16;  // Offset from the beginning of the backend
                                // allocation to the beginning of the chunk
                                // itself, in multiples of MinAlignment. See
                                // comment about its maximum value and in init().
   u64 Salt              : 8;
 };
 
 typedef atomic_uint64_t AtomicPackedHeader;
 COMPILER_CHECK(sizeof(UnpackedHeader) == sizeof(PackedHeader));
 
 // Minimum alignment of 8 bytes for 32-bit, 16 for 64-bit
 const uptr MinAlignmentLog = FIRST_32_SECOND_64(3, 4);
struct AP32 {
   static const uptr kSpaceBeg = 0;
   static const u64 kSpaceSize = SANITIZER_MMAP_RANGE_SIZE;
   static const uptr kMetadataSize = 0;
   typedef __scudo::SizeClassMap SizeClassMap;
   static const uptr kRegionSizeLog = RegionSizeLog;
   typedef __scudo::ByteMap ByteMap;
   typedef NoOpMapUnmapCallback MapUnmapCallback;
   static const uptr kFlags =
-      SizeClassAllocator32FlagMasks::kRandomShuffleChunks;
+      SizeClassAllocator32FlagMasks::kRandomShuffleChunks |
+      SizeClassAllocator32FlagMasks::kUseSeparateSizeClassForBatch;
 };
 typedef SizeClassAllocator32<AP32> PrimaryAllocator;
 #endif  // SANITIZER_CAN_USE_ALLOCATOR64
 
 // __sanitizer::RoundUp has a CHECK that is extraneous for us. Use our own.
 INLINE uptr RoundUpTo(uptr Size, uptr Boundary) {
   return (Size + Boundary - 1) & ~(Boundary - 1);
 }