sanitizer_common: optimize memory drain

[dvyukov]

Summary

Currently we allocate MemoryMapper per size class.
MemoryMapper mmap's and munmap's internal buffer.
This results in 50 mmap/munmap calls under the global
allocator mutex. Reuse MemoryMapper and the buffer
for all size classes. This radically reduces number of
mmap/munmap calls. Smaller size classes tend to have
more objects allocated, so it's highly likely that
the buffer allocated for the first size class will
be enough for all subsequent size classes.

[melver]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

46

I think this can just be 'MemoryMapper', and the typedef later would just shadow the template class.

Because there's still that 'MemoryMapperT' template argument elsewhere[1], and on a whole it looks like 'MemoryMapperT' would indicate a template arg, whereas here it no longer is.

50

explicit?

490

[1]

[dvyukov]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

490

I fail to decipher this comment.

[dvyukov]

• renamed MemoryMapperT to MemoryMapper
• added explicit to MemoryMapper ctor

[melver]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

490

See other comment:

[...]
Because there's still that 'MemoryMapperT' template argument elsewhere[1]
[...]

(... cross referencing code in comments is complicated.)

[nikic]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

114

It looks like this kind of shadowing is not accepted by GCC:

error: declaration of ‘typedef class sanitizer::MemoryMapper<sanitizer::SizeClassAllocator64<Params> > __sanitizer::SizeClassAllocator64<Params>::MemoryMapper’ changes meaning of ‘MemoryMapper’ [-fpermissive]

I've reverted the change for now.

[dvyukov]

resubmit after revert with gcc warning fixed

[dvyukov]

PTAL

I've re-uploaded with the gcc warning fixed, you can look at the new diff with:
https://reviews.llvm.org/D105778?vs=357981&id=358180#toc

[vitalybuka]

Breaks Windows https://lab.llvm.org/buildbot/#/builders/127/builds/13818/steps/8/logs/stdio

# command output:
   Creating library C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\asan\X86_64WindowsConfig\TestCases\Windows\Output\sanitizer_purge.cpp.lib and object C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\asan\X86_64WindowsConfig\TestCases\Windows\Output\sanitizer_purge.cpp.exp
$ ":" "RUN: at line 7"
$ "C:\b\slave\sanitizer-windows\build\stage1\projects\compiler-rt\test\asan\X86_64WindowsConfig\TestCases\Windows\Output\sanitizer_purge.cpp.tmp"
# command stderr:
AddressSanitizer: CHECK failed: sanitizer_win.cpp:139 "((VirtualQuery(addr, &mbi, sizeof(mbi)))) != (0)" (0x0, 0x0) (tid=8988)
    #0 0x7ff7650e4f6d in __asan::CheckUnwind C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\asan\asan_rtl.cpp:67
    #1 0x7ff7650d0273 in __sanitizer::CheckFailed(char const *, int, char const *, unsigned __int64, unsigned __int64) C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_termination.cpp:86
    #2 0x7ff7650d2e69 in __sanitizer::UnmapOrDie(void *, unsigned __int64) C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_win.cpp:139
    #3 0x7ff7650d2b6a in __sanitizer::ReleaseMemoryPagesToOS(unsigned __int64, unsigned __int64) C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_win.cpp:342
    #4 0x7ff765100f12 in __sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView> >::FreePagesRangeTracker<__sanitizer::MemoryMapper<__sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView> > > >::CloseOpenedRange C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_allocator_primary64.h:523
    #5 0x7ff765100f12 in __sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView> >::FreePagesRangeTracker<__sanitizer::MemoryMapper<__sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView> > > >::NextPage C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_allocator_primary64.h:508
    #6 0x7ff765100f12 in __sanitizer::SizeClassAllocator64<struct __asan::AP64<struct __sanitizer::LocalAddressSpaceView>>::ReleaseFreeMemoryToOS<class __sanitizer::MemoryMapper<class __sanitizer::SizeClassAllocator64<struct __asan::AP64<struct __sanitizer::LocalAddressSpaceView>>>>(unsigned int *, unsigned __int64, unsigned __int64, unsigned __int64, class __sanitizer::MemoryMapper<class __sanitizer::SizeClassAllocator64<struct __asan::AP64<struct __sanitizer::LocalAddressSpaceView>>> *, unsigned __int64) C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_allocator_primary64.h:610
    #7 0x7ff7651031c8 in __sanitizer::SizeClassAllocator64<struct __asan::AP64<struct __sanitizer::LocalAddressSpaceView>>::ForceReleaseToOS(void) C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_allocator_primary64.h:181
    #8 0x7ff7651071ae in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__asan::AP64<__sanitizer::LocalAddressSpaceView> >,__sanitizer::LargeMmapAllocatorPtrArrayDynamic>::ForceReleaseToOS C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\sanitizer_common\sanitizer_allocator_combined.h:86
    #9 0x7ff7651071ae in __asan::Allocator::Purge C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\asan\asan_allocator.cpp:836
    #10 0x7ff7651071ae in __sanitizer_purge_allocator C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\asan\asan_allocator.cpp:1242
    #11 0x7ff7650c129e in main C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\test\asan\TestCases\Windows\sanitizer_purge.cpp:23
    #12 0x7ff76510a247 in invoke_main d:\A01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
    #13 0x7ff76510a247 in __scrt_common_main_seh d:\A01\_work\12\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #14 0x7ffb5c4213f1  (C:\windows\system32\KERNEL32.DLL+0x1800013f1)
    #15 0x7ffb5c9154f3  (C:\windows\SYSTEM32\ntdll.dll+0x1800154f3)

FYI @hctim

[vitalybuka]

rebase

[vitalybuka]

another rebase

[vitalybuka]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

82

Maybe problem is that class_id everywhere else is the first arg.

[vitalybuka]

rebase

[dvyukov]

Maybe problem is that class_id everywhere else is the first arg.

Thanks for fixing this.
I very much confused how all linux tests passes with the swapped args... and there is a number of tests for this...

compiler-rt/lib/sanitizer_common/sanitizer_allocator_local_cache.h

 //===-- sanitizer_allocator_local_cache.h -----------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Part of the Sanitizer Allocator.
 //
 //===----------------------------------------------------------------------===//
 #ifndef SANITIZER_ALLOCATOR_H
 #error This file must be included inside sanitizer_allocator.h
 #endif
 
 // Cache used by SizeClassAllocator64.
 template <class SizeClassAllocator>
 struct SizeClassAllocator64LocalCache {
   typedef SizeClassAllocator Allocator;
   typedef MemoryMapper<Allocator> MemoryMapperT;
 
   void Init(AllocatorGlobalStats *s) {
     stats_.Init();
     if (s)
       s->Register(&stats_);
   }
 
   void Destroy(SizeClassAllocator *allocator, AllocatorGlobalStats *s) {
     Drain(allocator);
     if (s)
       s->Unregister(&stats_);
   }
 
   void *Allocate(SizeClassAllocator *allocator, uptr class_id) {
     CHECK_NE(class_id, 0UL);
     CHECK_LT(class_id, kNumClasses);
     PerClass *c = &per_class_[class_id];
     if (UNLIKELY(c->count == 0)) {
       if (UNLIKELY(!Refill(c, allocator, class_id)))
         return nullptr;
       DCHECK_GT(c->count, 0);
     }
     CompactPtrT chunk = c->chunks[--c->count];
     stats_.Add(AllocatorStatAllocated, c->class_size);
     return reinterpret_cast<void *>(allocator->CompactPtrToPointer(
         allocator->GetRegionBeginBySizeClass(class_id), chunk));
   }
 
   void Deallocate(SizeClassAllocator *allocator, uptr class_id, void *p) {
     CHECK_NE(class_id, 0UL);
     CHECK_LT(class_id, kNumClasses);
     // If the first allocator call on a new thread is a deallocation, then
     // max_count will be zero, leading to check failure.
     PerClass *c = &per_class_[class_id];
     InitCache(c);
     if (UNLIKELY(c->count == c->max_count))
-      Drain(c, allocator, class_id, c->max_count / 2);
+      Drain(c, allocator, class_id);
     CompactPtrT chunk = allocator->PointerToCompactPtr(
         allocator->GetRegionBeginBySizeClass(class_id),
         reinterpret_cast<uptr>(p));
     c->chunks[c->count++] = chunk;
     stats_.Sub(AllocatorStatAllocated, c->class_size);
   }
 
   void Drain(SizeClassAllocator *allocator) {
+    MemoryMapperT memory_mapper(*allocator);
     for (uptr i = 1; i < kNumClasses; i++) {
       PerClass *c = &per_class_[i];
-      while (c->count > 0) Drain(c, allocator, i, c->count);
+      while (c->count > 0) Drain(&memory_mapper, c, allocator, i, c->count);
     }
   }
 
  private:
   typedef typename Allocator::SizeClassMapT SizeClassMap;
   static const uptr kNumClasses = SizeClassMap::kNumClasses;
   typedef typename Allocator::CompactPtrT CompactPtrT;
 
   struct PerClass {
     u32 count;
     u32 max_count;
     uptr class_size;
     CompactPtrT chunks[2 * SizeClassMap::kMaxNumCachedHint];
   };
   PerClass per_class_[kNumClasses];
   AllocatorStats stats_;
 
   void InitCache(PerClass *c) {
     if (LIKELY(c->max_count))
       return;
     for (uptr i = 1; i < kNumClasses; i++) {
       PerClass *c = &per_class_[i];
       const uptr size = Allocator::ClassIdToSize(i);
       c->max_count = 2 * SizeClassMap::MaxCachedHint(size);
       c->class_size = size;
     }
     DCHECK_NE(c->max_count, 0UL);
   }
 
   NOINLINE bool Refill(PerClass *c, SizeClassAllocator *allocator,
                        uptr class_id) {
     InitCache(c);
     const uptr num_requested_chunks = c->max_count / 2;
     if (UNLIKELY(!allocator->GetFromAllocator(&stats_, class_id, c->chunks,
                                               num_requested_chunks)))
       return false;
     c->count = num_requested_chunks;
     return true;
   }
 
-  NOINLINE void Drain(PerClass *c, SizeClassAllocator *allocator, uptr class_id,
-                      uptr count) {
+  NOINLINE void Drain(PerClass *c, SizeClassAllocator *allocator,
+                      uptr class_id) {
+    MemoryMapperT memory_mapper(*allocator);
+    Drain(&memory_mapper, c, allocator, class_id, c->max_count / 2);
+  }
+
+  void Drain(MemoryMapperT *memory_mapper, PerClass *c,
+             SizeClassAllocator *allocator, uptr class_id, uptr count) {
     CHECK_GE(c->count, count);
     const uptr first_idx_to_drain = c->count - count;
     c->count -= count;
-    allocator->ReturnToAllocator(&stats_, class_id,
+    allocator->ReturnToAllocator(memory_mapper, &stats_, class_id,
                                  &c->chunks[first_idx_to_drain], count);
   }
 };
 
 // Cache used by SizeClassAllocator32.
 template <class SizeClassAllocator>
 struct SizeClassAllocator32LocalCache {
   typedef SizeClassAllocator Allocator;

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

 //===-- sanitizer_allocator_primary64.h -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Part of the Sanitizer Allocator.
 //
 //===----------------------------------------------------------------------===//
 #ifndef SANITIZER_ALLOCATOR_H
 #error This file must be included inside sanitizer_allocator.h
 #endif
 
 template<class SizeClassAllocator> struct SizeClassAllocator64LocalCache;
 
 // SizeClassAllocator64 -- allocator for 64-bit address space.
 // The template parameter Params is a class containing the actual parameters.
 //
 // Space: a portion of address space of kSpaceSize bytes starting at SpaceBeg.

 
 struct SizeClassAllocator64FlagMasks {  //  Bit masks.
   enum {
     kRandomShuffleChunks = 1,
   };
 };
 
 template <typename Allocator>
 class MemoryMapper {
  public:
   typedef typename Allocator::CompactPtrT CompactPtrT;
 
-  MemoryMapper(const Allocator &base_allocator, uptr class_id)
-      : allocator(base_allocator),
-        region_base(base_allocator.GetRegionBeginBySizeClass(class_id)) {}
+  explicit MemoryMapper(const Allocator &allocator) : allocator_(allocator) {}
 
-  uptr GetReleasedRangesCount() const { return released_ranges_count; }
+  ~MemoryMapper() {
+    if (buffer_)
+      UnmapOrDie(buffer_, buffer_size_);
+  }
 
-  uptr GetReleasedBytes() const { return released_bytes; }
+  bool GetAndResetStats(uptr &ranges, uptr &bytes) {
+    ranges = released_ranges_count_;
+    released_ranges_count_ = 0;
+    bytes = released_bytes_;
+    released_bytes_ = 0;
+    return ranges != 0;
+  }
 
   void *MapPackedCounterArrayBuffer(uptr buffer_size) {
     // TODO(alekseyshl): The idea to explore is to check if we have enough
     // space between num_freed_chunks*sizeof(CompactPtrT) and
     // mapped_free_array to fit buffer_size bytes and use that space instead
     // of mapping a temporary one.
-    return MmapOrDieOnFatalError(buffer_size, "ReleaseToOSPageCounters");
+    if (buffer_size_ < buffer_size) {
+      if (buffer_)
+        UnmapOrDie(buffer_, buffer_size_);
+      buffer_ = MmapOrDieOnFatalError(buffer_size, "ReleaseToOSPageCounters");
+      buffer_size_ = buffer_size;
+    } else {
+      internal_memset(buffer_, 0, buffer_size);
     }
-
-  void UnmapPackedCounterArrayBuffer(void *buffer, uptr buffer_size) {
-    UnmapOrDie(buffer, buffer_size);
+    return buffer_;
   }
 
   // Releases [from, to) range of pages back to OS.
-  void ReleasePageRangeToOS(CompactPtrT from, CompactPtrT to) {
-    const uptr from_page = allocator.CompactPtrToPointer(region_base, from);
-    const uptr to_page = allocator.CompactPtrToPointer(region_base, to);
+  void ReleasePageRangeToOS(CompactPtrT from, CompactPtrT to, uptr class_id) {
+    const uptr region_base = allocator_.GetRegionBeginBySizeClass(class_id);
+    const uptr from_page = allocator_.CompactPtrToPointer(region_base, from);
+    const uptr to_page = allocator_.CompactPtrToPointer(region_base, to);
     ReleaseMemoryPagesToOS(from_page, to_page);
-    released_ranges_count++;
-    released_bytes += to_page - from_page;
+    released_ranges_count_++;
+    released_bytes_ += to_page - from_page;
   }
 
  private:
-  const Allocator &allocator;
-  const uptr region_base = 0;
-  uptr released_ranges_count = 0;
-  uptr released_bytes = 0;
+  const Allocator &allocator_;
+  uptr released_ranges_count_ = 0;
+  uptr released_bytes_ = 0;
+  void *buffer_ = nullptr;
+  uptr buffer_size_ = 0;
 };
 
 template <class Params>
 class SizeClassAllocator64 {
  public:
   using AddressSpaceView = typename Params::AddressSpaceView;
   static const uptr kSpaceBeg = Params::kSpaceBeg;
   static const uptr kSpaceSize = Params::kSpaceSize;
   static const uptr kMetadataSize = Params::kMetadataSize;
   typedef typename Params::SizeClassMap SizeClassMap;
   typedef typename Params::MapUnmapCallback MapUnmapCallback;
 
   static const bool kRandomShuffleChunks =
       Params::kFlags & SizeClassAllocator64FlagMasks::kRandomShuffleChunks;
 
   typedef SizeClassAllocator64<Params> ThisT;
   typedef SizeClassAllocator64LocalCache<ThisT> AllocatorCache;
   typedef MemoryMapper<ThisT> MemoryMapperT;
 
   // When we know the size class (the region base) we can represent a pointer
   // as a 4-byte integer (offset from the region start shifted right by 4).
   typedef u32 CompactPtrT;
   static const uptr kCompactPtrScale = 4;
   CompactPtrT PointerToCompactPtr(uptr base, uptr ptr) const {
     return static_cast<CompactPtrT>((ptr - base) >> kCompactPtrScale);
   }
   uptr CompactPtrToPointer(uptr base, CompactPtrT ptr32) const {
     return base + (static_cast<uptr>(ptr32) << kCompactPtrScale);
   }
 
   // If heap_start is nonzero, assumes kSpaceSize bytes are already mapped R/W
   // at heap_start and places the heap there.  This mode requires kSpaceBeg ==
   // ~(uptr)0.
   void Init(s32 release_to_os_interval_ms, uptr heap_start = 0) {
     uptr TotalSpaceSize = kSpaceSize + AdditionalSize();
public:
   }
 
   void SetReleaseToOSIntervalMs(s32 release_to_os_interval_ms) {
     atomic_store(&release_to_os_interval_ms_, release_to_os_interval_ms,
                  memory_order_relaxed);
   }
 
   void ForceReleaseToOS() {
+    MemoryMapperT memory_mapper(*this);
     for (uptr class_id = 1; class_id < kNumClasses; class_id++) {
       BlockingMutexLock l(&GetRegionInfo(class_id)->mutex);
-      MaybeReleaseToOS(class_id, true /*force*/);
+      MaybeReleaseToOS(&memory_mapper, class_id, true /*force*/);
     }
   }
 
   static bool CanAllocate(uptr size, uptr alignment) {
     return size <= SizeClassMap::kMaxSize &&
       alignment <= SizeClassMap::kMaxSize;
   }
 
-  NOINLINE void ReturnToAllocator(AllocatorStats *stat, uptr class_id,
+  NOINLINE void ReturnToAllocator(MemoryMapperT *memory_mapper,
+                                  AllocatorStats *stat, uptr class_id,
                                   const CompactPtrT *chunks, uptr n_chunks) {
     RegionInfo *region = GetRegionInfo(class_id);
     uptr region_beg = GetRegionBeginBySizeClass(class_id);
     CompactPtrT *free_array = GetFreeArray(region_beg);
 
     BlockingMutexLock l(&region->mutex);
     uptr old_num_chunks = region->num_freed_chunks;
     uptr new_num_freed_chunks = old_num_chunks + n_chunks;
     // Failure to allocate free array space while releasing memory is non
     // recoverable.
     if (UNLIKELY(!EnsureFreeArraySpace(region, region_beg,
                                        new_num_freed_chunks))) {
       Report("FATAL: Internal error: %s's allocator exhausted the free list "
              "space for size class %zd (%zd bytes).\n", SanitizerToolName,
              class_id, ClassIdToSize(class_id));
       Die();
     }
     for (uptr i = 0; i < n_chunks; i++)
       free_array[old_num_chunks + i] = chunks[i];
     region->num_freed_chunks = new_num_freed_chunks;
     region->stats.n_freed += n_chunks;
 
-    MaybeReleaseToOS(class_id, false /*force*/);
+    MaybeReleaseToOS(memory_mapper, class_id, false /*force*/);
   }
 
   NOINLINE bool GetFromAllocator(AllocatorStats *stat, uptr class_id,
                                  CompactPtrT *chunks, uptr n_chunks) {
     RegionInfo *region = GetRegionInfo(class_id);
     uptr region_beg = GetRegionBeginBySizeClass(class_id);
     CompactPtrT *free_array = GetFreeArray(region_beg);
 
PackedCounterArray(u64 num_counters, u64 max_value, MemoryMapper *mapper)
       bit_offset_mask = packing_ratio - 1;
 
       buffer_size =
           (RoundUpTo(n, 1ULL << packing_ratio_log) >> packing_ratio_log) *
           sizeof(*buffer);
       buffer = reinterpret_cast<u64*>(
           memory_mapper->MapPackedCounterArrayBuffer(buffer_size));
     }
-    ~PackedCounterArray() {
-      if (buffer) {
-        memory_mapper->UnmapPackedCounterArrayBuffer(buffer, buffer_size);
-      }
-    }
 
     bool IsAllocated() const {
       return !!buffer;
     }
 
     u64 GetCount() const {
       return n;
     }
private:
     u64 packing_ratio_log;
     u64 bit_offset_mask;
 
     MemoryMapper *const memory_mapper;
     u64 buffer_size;
     u64* buffer;
   };
 
   template <class MemoryMapperT>
   class FreePagesRangeTracker {
    public:
-    explicit FreePagesRangeTracker(MemoryMapperT *mapper)
+    explicit FreePagesRangeTracker(MemoryMapperT *mapper, uptr class_id)
         : memory_mapper(mapper),
+          class_id(class_id),
           page_size_scaled_log(Log2(GetPageSizeCached() >> kCompactPtrScale)),
           in_the_range(false),
           current_page(0),
           current_range_start_page(0) {}
 
     void NextPage(bool freed) {
       if (freed) {
         if (!in_the_range) {
public:
     void Done() {
       CloseOpenedRange();
     }
 
    private:
     void CloseOpenedRange() {
       if (in_the_range) {
         memory_mapper->ReleasePageRangeToOS(
-            current_range_start_page << page_size_scaled_log,
+            class_id, current_range_start_page << page_size_scaled_log,
             current_page << page_size_scaled_log);
         in_the_range = false;
       }
     }
 
     MemoryMapperT *const memory_mapper;
+    const uptr class_id;
     const uptr page_size_scaled_log;
     bool in_the_range;
     uptr current_page;
     uptr current_range_start_page;
   };
 
   // Iterates over the free_array to identify memory pages containing freed
   // chunks only and returns these pages back to OS.
   // allocated_pages_count is the total number of pages allocated for the
   // current bucket.
   template <typename MemoryMapper>
   static void ReleaseFreeMemoryToOS(CompactPtrT *free_array,
                                     uptr free_array_count, uptr chunk_size,
                                     uptr allocated_pages_count,
-                                    MemoryMapper *memory_mapper) {
+                                    MemoryMapper *memory_mapper,
+                                    uptr class_id) {
     const uptr page_size = GetPageSizeCached();
 
     // Figure out the number of chunks per page and whether we can take a fast
     // path (the number of chunks per page is the same for all pages).
     uptr full_pages_chunk_count_max;
     bool same_chunk_count_per_page;
     if (chunk_size <= page_size && page_size % chunk_size == 0) {
       // Same number of chunks per page, no cross overs.
if (chunk_size <= page_size && page_size % chunk_size == 0) {
         counters.IncRange(
             free_array[i] >> page_size_scaled_log,
             (free_array[i] + chunk_size_scaled - 1) >> page_size_scaled_log);
       }
     }
 
     // Iterate over pages detecting ranges of pages with chunk counters equal
     // to the expected number of chunks for the particular page.
-    FreePagesRangeTracker<MemoryMapper> range_tracker(memory_mapper);
+    FreePagesRangeTracker<MemoryMapper> range_tracker(memory_mapper, class_id);
     if (same_chunk_count_per_page) {
       // Fast path, every page has the same number of chunks affecting it.
       for (uptr i = 0; i < counters.GetCount(); i++)
         range_tracker.NextPage(counters.Get(i) == full_pages_chunk_count_max);
     } else {
       // Show path, go through the pages keeping count how many chunks affect
       // each page.
       const uptr pn =
NOINLINE bool PopulateFreeArray(AllocatorStats *stat, uptr class_id,
     return true;
   }
 
   // Attempts to release RAM occupied by freed chunks back to OS. The region is
   // expected to be locked.
   //
   // TODO(morehouse): Support a callback on memory release so HWASan can release
   // aliases as well.
-  void MaybeReleaseToOS(uptr class_id, bool force) {
+  void MaybeReleaseToOS(MemoryMapperT *memory_mapper, uptr class_id,
+                        bool force) {
     RegionInfo *region = GetRegionInfo(class_id);
     const uptr chunk_size = ClassIdToSize(class_id);
     const uptr page_size = GetPageSizeCached();
 
     uptr n = region->num_freed_chunks;
     if (n * chunk_size < page_size)
       return;  // No chance to release anything.
     if ((region->stats.n_freed -
          region->rtoi.n_freed_at_last_release) * chunk_size < page_size) {
       return;  // Nothing new to release.
     }
 
     if (!force) {
       s32 interval_ms = ReleaseToOSIntervalMs();
       if (interval_ms < 0)
         return;
 
       if (region->rtoi.last_release_at_ns + interval_ms * 1000000ULL >
           MonotonicNanoTime()) {
         return;  // Memory was returned recently.
       }
     }
 
-    MemoryMapper<ThisT> memory_mapper(*this, class_id);
-
     ReleaseFreeMemoryToOS(
         GetFreeArray(GetRegionBeginBySizeClass(class_id)), n, chunk_size,
-        RoundUpTo(region->allocated_user, page_size) / page_size,
-        &memory_mapper);
+        RoundUpTo(region->allocated_user, page_size) / page_size, memory_mapper,
+        class_id);
 
-    if (memory_mapper.GetReleasedRangesCount() > 0) {
+    uptr ranges, bytes;
+    if (memory_mapper->GetAndResetStats(ranges, bytes)) {
       region->rtoi.n_freed_at_last_release = region->stats.n_freed;
-      region->rtoi.num_releases += memory_mapper.GetReleasedRangesCount();
-      region->rtoi.last_released_bytes = memory_mapper.GetReleasedBytes();
+      region->rtoi.num_releases += ranges;
+      region->rtoi.last_released_bytes = bytes;
     }
     region->rtoi.last_release_at_ns = MonotonicNanoTime();
   }
 };

compiler-rt/lib/sanitizer_common/tests/sanitizer_allocator_test.cpp

  public:
   std::string reported_pages;
 
   RangeRecorder()
       : page_size_scaled_log(
             Log2(GetPageSizeCached() >> Allocator64::kCompactPtrScale)),
         last_page_reported(0) {}
 
-  void ReleasePageRangeToOS(u32 from, u32 to) {
+  void ReleasePageRangeToOS(u32 class_id, u32 from, u32 to) {
     from >>= page_size_scaled_log;
     to >>= page_size_scaled_log;
     ASSERT_LT(from, to);
     if (!reported_pages.empty())
       ASSERT_LT(last_page_reported, from);
     reported_pages.append(from - last_page_reported, '.');
     reported_pages.append(to - from, 'x');
     last_page_reported = to;
const char* test_cases[] = {
       "x.x.x.x.x.x.x.x.x.x.x.x.",
       ".x.x.x.x.x.x.x.x.x.x.x.x",
       ".x.x.x.x.x.x.x.x.x.x.x.x.",
       "x.x.x.x.x.x.x.x.x.x.x.x.x",
   };
 
   for (auto test_case : test_cases) {
     RangeRecorder range_recorder;
-    RangeTracker tracker(&range_recorder);
+    RangeTracker tracker(&range_recorder, 1);
     for (int i = 0; test_case[i] != 0; i++)
       tracker.NextPage(test_case[i] == 'x');
     tracker.Done();
     // Strip trailing '.'-pages before comparing the results as they are not
     // going to be reported to range_recorder anyway.
     const char* last_x = strrchr(test_case, 'x');
     std::string expected(
         test_case,
public:
   void *MapPackedCounterArrayBuffer(uptr buffer_size) {
     reported_pages.clear();
     return calloc(1, buffer_size);
   }
   void UnmapPackedCounterArrayBuffer(void *buffer, uptr buffer_size) {
     free(buffer);
   }
 
-  void ReleasePageRangeToOS(u32 from, u32 to) {
+  void ReleasePageRangeToOS(u32 class_id, u32 from, u32 to) {
     uptr page_size_scaled =
         GetPageSizeCached() >> Allocator64::kCompactPtrScale;
     for (u32 i = from; i < to; i += page_size_scaled)
       reported_pages.insert(i);
   }
 };
 
 template <class Allocator>
for (uptr class_id = 1; class_id <= Allocator::SizeClassMapT::kLargestClassID;
     if (free_array.empty())
       continue;
     // Shuffle free_list to verify that ReleaseFreeMemoryToOS does not depend on
     // the list ordering.
     std::shuffle(free_array.begin(), free_array.end(), r);
 
     Allocator::ReleaseFreeMemoryToOS(&free_array[0], free_array.size(),
                                      chunk_size, kAllocatedPagesCount,
-                                     &memory_mapper);
+                                     &memory_mapper, class_id);
 
     // Verify that there are no released pages touched by used chunks and all
     // ranges of free chunks big enough to contain the entire memory pages had
     // these pages released.
     uptr verified_released_pages = 0;
     std::set<u32> free_chunks(free_array.begin(), free_array.end());
 
     u32 current_chunk = 0;