[sanitizer] Support dynamic premapped R/W range in primary allocator.

[morehouse]

Summary

The main use case for this change is HWASan aliasing mode, which premaps
the alias space adjacent to the dynamic shadow. With this change, the
primary allocator can allocate from the alias space instead of a
separate region.

[eugenis]

LGTM

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

I wonder if MADVISE_DONTNEED works correctly on aliased memory? You probably need to run it on each alias for it to have an effect on RSS.

[morehouse]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

Yes, just ran an experiment. Each alias occupies RSS independently. So we can fault in one alias and the others won't take up RSS until we touch them. If we touch multiple aliases, they each take up a chunk of RSS.

Probably we need a callback that hwasan_allocator.cpp can register in order to release alias pages as well.

[pcc]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

I don't think it would. Since the original mapping was created with MAP_SHARED, MADV_DONTNEED will not affect the contents of the pages. I confirmed this by running this program:

#define _GNU_SOURCE
#include <sys/mman.h>
#include <stdio.h>

int main() {
  void *p = mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANON, -1, 0); 
  void *p2 = mremap(p, 0, 4096, MREMAP_MAYMOVE);
  *(char *)p = 1;
  printf("%d\n", *(volatile char *)p2);
  madvise(p, 4096, MADV_DONTNEED);
  printf("%d\n", *(volatile char *)p2);
  madvise(p2, 4096, MADV_DONTNEED);
  printf("%d\n", *(volatile char *)p2);
}

The output is

1
1
1

[morehouse]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

Right, the memory is not cleared, but RSS does go down by the expected amount after the MADV_DONTNEED.

So it seems worth adding a callback?

[pcc]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

The goal isn't just to make RSS go down though, right? Presumably if the data is still accessible then it's still in memory. I'm not sure if there's a syscall sequence that we can use to release this memory.

This is also a bit more concerning:

#define _GNU_SOURCE
#include <sys/mman.h>
#include <stdio.h>

int main() {
  void *p = mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANON, -1, 0);
  void *p2 = mremap(p, 0, 4096, MREMAP_MAYMOVE);
  *(char *)p = 1;
  printf("%d\n", *(volatile char *)p2);
  madvise(p, 4096, MADV_DONTNEED);
  printf("%d\n", *(volatile char *)p2);
  madvise(p2, 4096, MADV_DONTNEED);
  printf("%d\n", *(volatile char *)p2);
  if (fork() == 0) {
    *(char *)p = 2;
    exit(0);
  }
  sleep(1);
  printf("%d\n", *(volatile char *)p);
}

The output is

1
1
1
2

[vitalybuka]

LGTM if "release" will be addressed in another patch

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

678

do we need to check that beg/size are in mapped range?

[morehouse]

• Check bounds on mmaps.

[morehouse]

In D98293#2618636, @vitalybuka wrote:

LGTM if "release" will be addressed in another patch

Yes, I will address it in a different patch.

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

I think memory sharing across fork is the price we pay for MAP_SHARED. At least within Google I expect it to not be much problem since uses of fork are 99%+ fork+exec. Maybe we can duplicate the alias region with a fork interceptor, but this is complexity I'd like to avoid unless necessary.

For actually releasing the physical memory, the only thing I think *might* work is tmpfs + fallocate(FALLOCATE_FL_PUNCH_HOLE). I'd need to experiment to be sure. But that also involves more complexity that I'd like to avoid (at least initially). Let's only deal with it if it becomes an issue?

[morehouse]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

Unfortunately some of the build infra (python, flex) within Google calls fork. I tried changing TCMalloc to use MAP_SHARED, and got crashes when building things:

Fatal Python error: GC object already tracked

(Segmentation fault) flex failed: error executing command...

So I am less confident we can deploy this now...

With a fork interceptor we might be able to duplicate and recreate aliases in the child, but copying TBs of memory will be very slow.

[eugenis]

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

108–109

Ouch. Even in fork + exec code there may be a heap memory write between fork and exec that would affect the parent process.

We do not need to copy TBs of memory - only the actual physical pages. We could do this by scanning /proc/$PID/pagemap. Or maybe mlock + mincore? Even such limited copy might be prohibitively expensive.

[morehouse]

• Add TODO for release callback support.

[eugenis]

LGTM

[vitalybuka]

Reverted, it fails on Windows https://lab.llvm.org/buildbot/#/builders/127/builds/7999 and Android https://lab.llvm.org/buildbot/#/builders/77/builds/4839

[morehouse]

• Disable tests on Android and Windows due to OOM.

compiler-rt/lib/sanitizer_common/sanitizer_allocator_combined.h

 //===-- sanitizer_allocator_combined.h --------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Part of the Sanitizer Allocator.
 //
 //===----------------------------------------------------------------------===//
 #ifndef SANITIZER_ALLOCATOR_H
 #error This file must be included inside sanitizer_allocator.h
 #endif
 
 // This class implements a complete memory allocator by using two
 // internal allocators:
 // PrimaryAllocator is efficient, but may not allocate some sizes (alignments).
 //  When allocating 2^x bytes it should return 2^x aligned chunk.
 // PrimaryAllocator is used via a local AllocatorCache.
 // SecondaryAllocator can allocate anything, but is not efficient.
 template <class PrimaryAllocator,
           class LargeMmapAllocatorPtrArray = DefaultLargeMmapAllocatorPtrArray>
 class CombinedAllocator {
  public:
   using AllocatorCache = typename PrimaryAllocator::AllocatorCache;
   using SecondaryAllocator =
       LargeMmapAllocator<typename PrimaryAllocator::MapUnmapCallback,
                          LargeMmapAllocatorPtrArray,
                          typename PrimaryAllocator::AddressSpaceView>;
 
   void InitLinkerInitialized(s32 release_to_os_interval_ms) {
     stats_.InitLinkerInitialized();
     primary_.Init(release_to_os_interval_ms);
     secondary_.InitLinkerInitialized();
   }
 
-  void Init(s32 release_to_os_interval_ms) {
+  void Init(s32 release_to_os_interval_ms, uptr heap_start = 0) {
     stats_.Init();
-    primary_.Init(release_to_os_interval_ms);
+    primary_.Init(release_to_os_interval_ms, heap_start);
     secondary_.Init();
   }
 
   void *Allocate(AllocatorCache *cache, uptr size, uptr alignment) {
     // Returning 0 on malloc(0) may break a lot of code.
     if (size == 0)
       size = 1;
     if (size + alignment < size) {
       Report("WARNING: %s: CombinedAllocator allocation overflow: "
              "0x%zx bytes with 0x%zx alignment requested\n",
              SanitizerToolName, size, alignment);
       return nullptr;
     }
     uptr original_size = size;
     // If alignment requirements are to be fulfilled by the frontend allocator
     // rather than by the primary or secondary, passing an alignment lower than
     // or equal to 8 will prevent any further rounding up, as well as the later
     // alignment check.
     if (alignment > 8)
if (primary_.CanAllocate(size, alignment))
       res = cache->Allocate(&primary_, primary_.ClassID(size));
     else
       res = secondary_.Allocate(&stats_, original_size, alignment);
     if (alignment > 8)
       CHECK_EQ(reinterpret_cast<uptr>(res) & (alignment - 1), 0);
     return res;
   }
 
   s32 ReleaseToOSIntervalMs() const {
     return primary_.ReleaseToOSIntervalMs();
   }
 
   void SetReleaseToOSIntervalMs(s32 release_to_os_interval_ms) {
     primary_.SetReleaseToOSIntervalMs(release_to_os_interval_ms);
   }
 
   void ForceReleaseToOS() {
     primary_.ForceReleaseToOS();
   }
 
   void Deallocate(AllocatorCache *cache, void *p) {
     if (!p) return;
     if (primary_.PointerIsMine(p))
       cache->Deallocate(&primary_, primary_.GetSizeClass(p), p);
     else
       secondary_.Deallocate(&stats_, p);
   }
 
   void *Reallocate(AllocatorCache *cache, void *p, uptr new_size,
                    uptr alignment) {
     if (!p)
       return Allocate(cache, new_size, alignment);
     if (!new_size) {
       Deallocate(cache, p);
       return nullptr;
     }
     CHECK(PointerIsMine(p));
     uptr old_size = GetActuallyAllocatedSize(p);
public:
   // This function does the same as GetBlockBegin, but is much faster.
   // Must be called with the allocator locked.
   void *GetBlockBeginFastLocked(void *p) {
     if (primary_.PointerIsMine(p))
       return primary_.GetBlockBegin(p);
     return secondary_.GetBlockBeginFastLocked(p);
   }
 
   uptr GetActuallyAllocatedSize(void *p) {
     if (primary_.PointerIsMine(p))
       return primary_.GetActuallyAllocatedSize(p);
     return secondary_.GetActuallyAllocatedSize(p);
   }
 
   uptr TotalMemoryUsed() {
     return primary_.TotalMemoryUsed() + secondary_.TotalMemoryUsed();
   }
 
   void TestOnlyUnmap() { primary_.TestOnlyUnmap(); }
 
   void InitCache(AllocatorCache *cache) {
     cache->Init(&stats_);
   }
 
   void DestroyCache(AllocatorCache *cache) {
     cache->Destroy(&primary_, &stats_);
   }
 
   void SwallowCache(AllocatorCache *cache) {
     cache->Drain(&primary_);
   }
 
   void GetStats(AllocatorStatCounters s) const {
     stats_.Get(s);
   }
 
   void PrintStats() {
     primary_.PrintStats();
     secondary_.PrintStats();
   }
 
   // ForceLock() and ForceUnlock() are needed to implement Darwin malloc zone
   // introspection API.
   void ForceLock() {
     primary_.ForceLock();
     secondary_.ForceLock();
   }
 
   void ForceUnlock() {
     secondary_.ForceUnlock();
     primary_.ForceUnlock();
   }
 
   // Iterate over all existing chunks.
   // The allocator must be locked when calling this function.
   void ForEachChunk(ForEachChunkCallback callback, void *arg) {
     primary_.ForEachChunk(callback, arg);
     secondary_.ForEachChunk(callback, arg);
   }
 
  private:
   PrimaryAllocator primary_;
   SecondaryAllocator secondary_;
   AllocatorGlobalStats stats_;
 };

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary32.h

 //===-- sanitizer_allocator_primary32.h -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Part of the Sanitizer Allocator.
 //
 //===----------------------------------------------------------------------===//
 #ifndef SANITIZER_ALLOCATOR_H
 #error This file must be included inside sanitizer_allocator.h
 #endif
 
 template<class SizeClassAllocator> struct SizeClassAllocator32LocalCache;
 
 // SizeClassAllocator32 -- allocator for 32-bit address space.
 // This allocator can theoretically be used on 64-bit arch, but there it is less
 // efficient than SizeClassAllocator64.
 //
enum {
     kRandomShuffleChunks = 1,
     kUseSeparateSizeClassForBatch = 2,
   };
 };
 
 template <class Params>
 class SizeClassAllocator32 {
  private:
   static const u64 kTwoLevelByteMapSize1 =
       (Params::kSpaceSize >> Params::kRegionSizeLog) >> 12;
   static const u64 kMinFirstMapSizeTwoLevelByteMap = 4;
 
  public:
   using AddressSpaceView = typename Params::AddressSpaceView;
   static const uptr kSpaceBeg = Params::kSpaceBeg;
   static const u64 kSpaceSize = Params::kSpaceSize;
   static const uptr kMetadataSize = Params::kMetadataSize;
   typedef typename Params::SizeClassMap SizeClassMap;
   static const uptr kRegionSizeLog = Params::kRegionSizeLog;
   typedef typename Params::MapUnmapCallback MapUnmapCallback;
   using ByteMap = typename conditional<
       (kTwoLevelByteMapSize1 < kMinFirstMapSizeTwoLevelByteMap),
       FlatByteMap<(Params::kSpaceSize >> Params::kRegionSizeLog),
                   AddressSpaceView>,
       TwoLevelByteMap<kTwoLevelByteMapSize1, 1 << 12, AddressSpaceView>>::type;
 
   COMPILER_CHECK(!SANITIZER_SIGN_EXTENDED_ADDRESSES ||
                  (kSpaceSize & (kSpaceSize - 1)) == 0);
 
   static const bool kRandomShuffleChunks = Params::kFlags &
       SizeClassAllocator32FlagMasks::kRandomShuffleChunks;
   static const bool kUseSeparateSizeClassForBatch = Params::kFlags &
       SizeClassAllocator32FlagMasks::kUseSeparateSizeClassForBatch;
 
   struct TransferBatch {
     static const uptr kMaxNumCached = SizeClassMap::kMaxNumCachedHint - 2;
     void SetFromArray(void *batch[], uptr count) {
       DCHECK_LE(count, kMaxNumCached);
       count_ = count;
       for (uptr i = 0; i < count; i++)
         batch_[i] = batch[i];
     }
     uptr Count() const { return count_; }
     void Clear() { count_ = 0; }
     void Add(void *ptr) {
       batch_[count_++] = ptr;
       DCHECK_LE(count_, kMaxNumCached);
     }
     void CopyToArray(void *to_batch[]) const {
       for (uptr i = 0, n = Count(); i < n; i++)
         to_batch[i] = batch_[i];
     }
 
     // How much memory do we need for a batch containing n elements.
     static uptr AllocationSizeRequiredForNElements(uptr n) {
       return sizeof(uptr) * 2 + sizeof(void *) * n;
     }
     static uptr MaxCached(uptr size) {
       return Min(kMaxNumCached, SizeClassMap::MaxCachedHint(size));
     }
 
     TransferBatch *next;
 
    private:
     uptr count_;
     void *batch_[kMaxNumCached];
   };
 
   static const uptr kBatchSize = sizeof(TransferBatch);
   COMPILER_CHECK((kBatchSize & (kBatchSize - 1)) == 0);
   COMPILER_CHECK(kBatchSize == SizeClassMap::kMaxNumCachedHint * sizeof(uptr));
 
   static uptr ClassIdToSize(uptr class_id) {
     return (class_id == SizeClassMap::kBatchClassID) ?
         kBatchSize : SizeClassMap::Size(class_id);
   }
 
   typedef SizeClassAllocator32<Params> ThisT;
   typedef SizeClassAllocator32LocalCache<ThisT> AllocatorCache;
 
-  void Init(s32 release_to_os_interval_ms) {
+  void Init(s32 release_to_os_interval_ms, uptr heap_start = 0) {
+    CHECK(!heap_start);
     possible_regions.Init();
     internal_memset(size_class_info_array, 0, sizeof(size_class_info_array));
   }
 
   s32 ReleaseToOSIntervalMs() const {
     return kReleaseToOSIntervalNever;
   }
 

compiler-rt/lib/sanitizer_common/sanitizer_allocator_primary64.h

 //===-- sanitizer_allocator_primary64.h -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Part of the Sanitizer Allocator.
 //
 //===----------------------------------------------------------------------===//
 #ifndef SANITIZER_ALLOCATOR_H
 #error This file must be included inside sanitizer_allocator.h
 #endif
 
 template<class SizeClassAllocator> struct SizeClassAllocator64LocalCache;
 
 // SizeClassAllocator64 -- allocator for 64-bit address space.
 // The template parameter Params is a class containing the actual parameters.
 //
 // Space: a portion of address space of kSpaceSize bytes starting at SpaceBeg.
enum {
     kRandomShuffleChunks = 1,
   };
 };
 
 template <class Params>
 class SizeClassAllocator64 {
  public:
   using AddressSpaceView = typename Params::AddressSpaceView;
   static const uptr kSpaceBeg = Params::kSpaceBeg;
   static const uptr kSpaceSize = Params::kSpaceSize;
   static const uptr kMetadataSize = Params::kMetadataSize;
   typedef typename Params::SizeClassMap SizeClassMap;
   typedef typename Params::MapUnmapCallback MapUnmapCallback;
 
   static const bool kRandomShuffleChunks =
       Params::kFlags & SizeClassAllocator64FlagMasks::kRandomShuffleChunks;
 
   typedef SizeClassAllocator64<Params> ThisT;
   typedef SizeClassAllocator64LocalCache<ThisT> AllocatorCache;
 
   // When we know the size class (the region base) we can represent a pointer
   // as a 4-byte integer (offset from the region start shifted right by 4).
   typedef u32 CompactPtrT;
   static const uptr kCompactPtrScale = 4;
   CompactPtrT PointerToCompactPtr(uptr base, uptr ptr) const {
     return static_cast<CompactPtrT>((ptr - base) >> kCompactPtrScale);
   }
   uptr CompactPtrToPointer(uptr base, CompactPtrT ptr32) const {
     return base + (static_cast<uptr>(ptr32) << kCompactPtrScale);
   }
 
-  void Init(s32 release_to_os_interval_ms) {
+  // If heap_start is nonzero, assumes kSpaceSize bytes are already mapped R/W
+  // at heap_start and places the heap there.  This mode requires kSpaceBeg ==
+  // ~(uptr)0.
+  void Init(s32 release_to_os_interval_ms, uptr heap_start = 0) {
     uptr TotalSpaceSize = kSpaceSize + AdditionalSize();
+    PremappedHeap = heap_start != 0;
+    if (PremappedHeap) {
+      CHECK(!kUsingConstantSpaceBeg);
+      NonConstSpaceBeg = heap_start;
+      uptr RegionInfoSize = AdditionalSize();
+      RegionInfoSpace =
+          address_range.Init(RegionInfoSize, PrimaryAllocatorName);
+      CHECK_NE(RegionInfoSpace, ~(uptr)0);
+      CHECK_EQ(RegionInfoSpace,
+               address_range.MapOrDie(RegionInfoSpace, RegionInfoSize,
+                                      "SizeClassAllocator: region info"));
+      MapUnmapCallback().OnMap(RegionInfoSpace, RegionInfoSize);
+    } else {
       if (kUsingConstantSpaceBeg) {
         CHECK(IsAligned(kSpaceBeg, SizeClassMap::kMaxSize));
-      CHECK_EQ(kSpaceBeg, address_range.Init(TotalSpaceSize,
-                                             PrimaryAllocatorName, kSpaceBeg));
+        CHECK_EQ(kSpaceBeg,
+                 address_range.Init(TotalSpaceSize, PrimaryAllocatorName,
+                                    kSpaceBeg));
       } else {
-      // Combined allocator expects that an 2^N allocation is always aligned to
-      // 2^N. For this to work, the start of the space needs to be aligned as
-      // high as the largest size class (which also needs to be a power of 2).
+        // Combined allocator expects that an 2^N allocation is always aligned
+        // to 2^N. For this to work, the start of the space needs to be aligned
+        // as high as the largest size class (which also needs to be a power of
+        // 2).
         NonConstSpaceBeg = address_range.InitAligned(
             TotalSpaceSize, SizeClassMap::kMaxSize, PrimaryAllocatorName);
         CHECK_NE(NonConstSpaceBeg, ~(uptr)0);
       }
-    SetReleaseToOSIntervalMs(release_to_os_interval_ms);
-    MapWithCallbackOrDie(SpaceEnd(), AdditionalSize(),
+      RegionInfoSpace = SpaceEnd();
+      MapWithCallbackOrDie(RegionInfoSpace, AdditionalSize(),
                            "SizeClassAllocator: region info");
+    }
+    SetReleaseToOSIntervalMs(release_to_os_interval_ms);
     // Check that the RegionInfo array is aligned on the CacheLine size.
-    DCHECK_EQ(SpaceEnd() % kCacheLineSize, 0);
+    DCHECK_EQ(RegionInfoSpace % kCacheLineSize, 0);
   }
 
   s32 ReleaseToOSIntervalMs() const {
     return atomic_load(&release_to_os_interval_ms_, memory_order_relaxed);
   }
 
   void SetReleaseToOSIntervalMs(s32 release_to_os_interval_ms) {
     atomic_store(&release_to_os_interval_ms_, release_to_os_interval_ms,
                  memory_order_relaxed);
   }
 
   void ForceReleaseToOS() {
     for (uptr class_id = 1; class_id < kNumClasses; class_id++) {
       BlockingMutexLock l(&GetRegionInfo(class_id)->mutex);
       MaybeReleaseToOS(class_id, true /*force*/);
     }
   }
 
   static bool CanAllocate(uptr size, uptr alignment) {
     return size <= SizeClassMap::kMaxSize &&
       alignment <= SizeClassMap::kMaxSize;
   }
 
   NOINLINE void ReturnToAllocator(AllocatorStats *stat, uptr class_id,
                                   const CompactPtrT *chunks, uptr n_chunks) {
     RegionInfo *region = GetRegionInfo(class_id);
     uptr region_beg = GetRegionBeginBySizeClass(class_id);
     CompactPtrT *free_array = GetFreeArray(region_beg);
 
     BlockingMutexLock l(&region->mutex);
     uptr old_num_chunks = region->num_freed_chunks;
     uptr new_num_freed_chunks = old_num_chunks + n_chunks;
private:
   static const uptr kUserMapSize = 1 << 16;
   // Call mmap for metadata memory with at least this size.
   static const uptr kMetaMapSize = 1 << 16;
   // Call mmap for free array memory with at least this size.
   static const uptr kFreeArrayMapSize = 1 << 16;
 
   atomic_sint32_t release_to_os_interval_ms_;
 
+  uptr RegionInfoSpace;
+
+  // True if the user has already mapped the entire heap R/W.
+  bool PremappedHeap;
+
   struct Stats {
     uptr n_allocated;
     uptr n_freed;
   };
 
   struct ReleaseToOsInfo {
     uptr n_freed_at_last_release;
     uptr num_releases;
struct ALIGNED(SANITIZER_CACHE_LINE_SIZE) RegionInfo {
     bool exhausted;  // Whether region is out of space for new chunks.
     Stats stats;
     ReleaseToOsInfo rtoi;
   };
   COMPILER_CHECK(sizeof(RegionInfo) % kCacheLineSize == 0);
 
   RegionInfo *GetRegionInfo(uptr class_id) const {
     DCHECK_LT(class_id, kNumClasses);
-    RegionInfo *regions = reinterpret_cast<RegionInfo *>(SpaceEnd());
+    RegionInfo *regions = reinterpret_cast<RegionInfo *>(RegionInfoSpace);
     return &regions[class_id];
   }
 
   uptr GetMetadataEnd(uptr region_beg) const {
     return region_beg + kRegionSize - kFreeArraySize;
   }
 
   uptr GetChunkIdx(uptr chunk, uptr size) const {
     if (!kUsingConstantSpaceBeg)
       chunk -= SpaceBeg();
 
     uptr offset = chunk % kRegionSize;
     // Here we divide by a non-constant. This is costly.
     // size always fits into 32-bits. If the offset fits too, use 32-bit div.
     if (offset >> (SANITIZER_WORDSIZE / 2))
       return offset / size;
     return (u32)offset / (u32)size;
   }
 
   CompactPtrT *GetFreeArray(uptr region_beg) const {
     return reinterpret_cast<CompactPtrT *>(GetMetadataEnd(region_beg));
   }
 
   bool MapWithCallback(uptr beg, uptr size, const char *name) {
+    if (PremappedHeap)
+      return beg >= NonConstSpaceBeg &&
+             beg + size <= NonConstSpaceBeg + kSpaceSize;
     uptr mapped = address_range.Map(beg, size, name);
     if (UNLIKELY(!mapped))
       return false;
     CHECK_EQ(beg, mapped);
     MapUnmapCallback().OnMap(beg, size);
     return true;
   }
 
   void MapWithCallbackOrDie(uptr beg, uptr size, const char *name) {
+    if (PremappedHeap) {
+      CHECK_GE(beg, NonConstSpaceBeg);
+      CHECK_LE(beg + size, NonConstSpaceBeg + kSpaceSize);
+      return;
+    }
     CHECK_EQ(beg, address_range.MapOrDie(beg, size, name));
     MapUnmapCallback().OnMap(beg, size);
   }
 
   void UnmapWithCallbackOrDie(uptr beg, uptr size) {
+    if (PremappedHeap)
+      return;
     MapUnmapCallback().OnUnmap(beg, size);
     address_range.Unmap(beg, size);
   }
 
   bool EnsureFreeArraySpace(RegionInfo *region, uptr region_beg,
                             uptr num_freed_chunks) {
     uptr needed_space = num_freed_chunks * sizeof(CompactPtrT);
     if (region->mapped_free_array < needed_space) {
private:
     const ThisT& allocator;
     const uptr region_base;
     uptr released_ranges_count;
     uptr released_bytes;
   };
 
   // Attempts to release RAM occupied by freed chunks back to OS. The region is
   // expected to be locked.
+  //
+  // TODO(morehouse): Support a callback on memory release so HWASan can release
+  // aliases as well.
   void MaybeReleaseToOS(uptr class_id, bool force) {
     RegionInfo *region = GetRegionInfo(class_id);
     const uptr chunk_size = ClassIdToSize(class_id);
     const uptr page_size = GetPageSizeCached();
 
     uptr n = region->num_freed_chunks;
     if (n * chunk_size < page_size)
       return;  // No chance to release anything.

compiler-rt/lib/sanitizer_common/tests/sanitizer_allocator_test.cpp

TEST(SanitizerCommon, InternalSizeClassMap) {
   TestSizeClassMap<InternalSizeClassMap>();
 }
 
 TEST(SanitizerCommon, DenseSizeClassMap) {
   TestSizeClassMap<VeryCompactSizeClassMap>();
 }
 
 template <class Allocator>
-void TestSizeClassAllocator() {
+void TestSizeClassAllocator(uptr premapped_heap = 0) {
   Allocator *a = new Allocator;
-  a->Init(kReleaseToOSIntervalNever);
+  a->Init(kReleaseToOSIntervalNever, premapped_heap);
   typename Allocator::AllocatorCache cache;
   memset(&cache, 0, sizeof(cache));
   cache.Init(0);
 
   static const uptr sizes[] = {
     1, 16,  30, 40, 100, 1000, 10000,
     50000, 60000, 100000, 120000, 300000, 500000, 1000000, 2000000
   };
for (uptr x = 0, step = kAddressSpaceSize / 100000;
     if (a->PointerIsMine(reinterpret_cast<void *>(x)))
       Ident(a->GetBlockBegin(reinterpret_cast<void *>(x)));
 
   a->TestOnlyUnmap();
   delete a;
 }
 
 #if SANITIZER_CAN_USE_ALLOCATOR64
+
+// Allocates kAllocatorSize aligned bytes on construction and frees it on
+// destruction.
+class ScopedPremappedHeap {
+ public:
+  ScopedPremappedHeap() {
+    BasePtr = MmapNoReserveOrDie(2 * kAllocatorSize, "preallocated heap");
+    AlignedAddr = RoundUpTo(reinterpret_cast<uptr>(BasePtr), kAllocatorSize);
+  }
+
+  ~ScopedPremappedHeap() { UnmapOrDie(BasePtr, kAllocatorSize); }
+
+  uptr Addr() { return AlignedAddr; }
+
+ private:
+  void *BasePtr;
+  uptr AlignedAddr;
+};
+
 // These tests can fail on Windows if memory is somewhat full and lit happens
 // to run them all at the same time. FIXME: Make them not flaky and reenable.
 #if !SANITIZER_WINDOWS
 TEST(SanitizerCommon, SizeClassAllocator64) {
   TestSizeClassAllocator<Allocator64>();
 }
 
 TEST(SanitizerCommon, SizeClassAllocator64Dynamic) {
   TestSizeClassAllocator<Allocator64Dynamic>();
 }
 
+TEST(SanitizerCommon, SizeClassAllocator64DynamicPremapped) {
+  ScopedPremappedHeap h;
+  TestSizeClassAllocator<Allocator64Dynamic>(h.Addr());
+}
+
 #if !SANITIZER_ANDROID
 //FIXME(kostyak): find values so that those work on Android as well.
 TEST(SanitizerCommon, SizeClassAllocator64Compact) {
   TestSizeClassAllocator<Allocator64Compact>();
 }
 
 TEST(SanitizerCommon, SizeClassAllocator64Dense) {
   TestSizeClassAllocator<Allocator64Dense>();

 using Allocator32SeparateBatches =
     Allocator32SeparateBatchesASVT<LocalAddressSpaceView>;
 
 TEST(SanitizerCommon, SizeClassAllocator32SeparateBatches) {
   TestSizeClassAllocator<Allocator32SeparateBatches>();
 }
 
 template <class Allocator>
-void SizeClassAllocatorMetadataStress() {
+void SizeClassAllocatorMetadataStress(uptr premapped_heap = 0) {
   Allocator *a = new Allocator;
-  a->Init(kReleaseToOSIntervalNever);
+  a->Init(kReleaseToOSIntervalNever, premapped_heap);
   typename Allocator::AllocatorCache cache;
   memset(&cache, 0, sizeof(cache));
   cache.Init(0);
 
   const uptr kNumAllocs = 1 << 13;
   void *allocated[kNumAllocs];
   void *meta[kNumAllocs];
   for (uptr i = 0; i < kNumAllocs; i++) {

 TEST(SanitizerCommon, SizeClassAllocator64MetadataStress) {
   SizeClassAllocatorMetadataStress<Allocator64>();
 }
 
 TEST(SanitizerCommon, SizeClassAllocator64DynamicMetadataStress) {
   SizeClassAllocatorMetadataStress<Allocator64Dynamic>();
 }
 
+TEST(SanitizerCommon, SizeClassAllocator64DynamicPremappedMetadataStress) {
+  ScopedPremappedHeap h;
+  SizeClassAllocatorMetadataStress<Allocator64Dynamic>(h.Addr());
+}
+
 #if !SANITIZER_ANDROID
 TEST(SanitizerCommon, SizeClassAllocator64CompactMetadataStress) {
   SizeClassAllocatorMetadataStress<Allocator64Compact>();
 }
 #endif
 
 #endif
 #endif  // SANITIZER_CAN_USE_ALLOCATOR64
 TEST(SanitizerCommon, SizeClassAllocator32CompactMetadataStress) {
   SizeClassAllocatorMetadataStress<Allocator32Compact>();
 }
 
 template <class Allocator>
-void SizeClassAllocatorGetBlockBeginStress(u64 TotalSize) {
+void SizeClassAllocatorGetBlockBeginStress(u64 TotalSize,
+                                           uptr premapped_heap = 0) {
   Allocator *a = new Allocator;
-  a->Init(kReleaseToOSIntervalNever);
+  a->Init(kReleaseToOSIntervalNever, premapped_heap);
   typename Allocator::AllocatorCache cache;
   memset(&cache, 0, sizeof(cache));
   cache.Init(0);
 
   uptr max_size_class = Allocator::SizeClassMapT::kLargestClassID;
   uptr size = Allocator::SizeClassMapT::Size(max_size_class);
   // Make sure we correctly compute GetBlockBegin() w/o overflow.
   for (size_t i = 0; i <= TotalSize / size; i++) {

 TEST(SanitizerCommon, SizeClassAllocator64GetBlockBegin) {
   SizeClassAllocatorGetBlockBeginStress<Allocator64>(
       1ULL << (SANITIZER_ANDROID ? 31 : 33));
 }
 TEST(SanitizerCommon, SizeClassAllocator64DynamicGetBlockBegin) {
   SizeClassAllocatorGetBlockBeginStress<Allocator64Dynamic>(
       1ULL << (SANITIZER_ANDROID ? 31 : 33));
 }
+TEST(SanitizerCommon, SizeClassAllocator64DynamicPremappedGetBlockBegin) {
+  ScopedPremappedHeap h;
+  SizeClassAllocatorGetBlockBeginStress<Allocator64Dynamic>(
+      1ULL << (SANITIZER_ANDROID ? 31 : 33), h.Addr());
+}
 #if !SANITIZER_ANDROID
 TEST(SanitizerCommon, SizeClassAllocator64CompactGetBlockBegin) {
   SizeClassAllocatorGetBlockBeginStress<Allocator64Compact>(1ULL << 33);
 }
 #endif
 TEST(SanitizerCommon, SizeClassAllocator64VeryCompactGetBlockBegin) {
   // Does not have > 4Gb for each class.
   SizeClassAllocatorGetBlockBeginStress<Allocator64VeryCompact>(1ULL << 31);
TEST(SanitizerCommon, LargeMmapAllocator) {
   char *p = (char *)a.Allocate(&stats, page_size, 1);
   CHECK_EQ(p, a.GetBlockBegin(p));
   CHECK_EQ(p, (char *)a.GetBlockBegin(p + page_size - 1));
   CHECK_NE(p, (char *)a.GetBlockBegin(p + page_size));
   a.Deallocate(&stats, p);
 }
 
 template <class PrimaryAllocator>
-void TestCombinedAllocator() {
+void TestCombinedAllocator(uptr premapped_heap = 0) {
   typedef CombinedAllocator<PrimaryAllocator> Allocator;
   Allocator *a = new Allocator;
-  a->Init(kReleaseToOSIntervalNever);
+  a->Init(kReleaseToOSIntervalNever, premapped_heap);
   std::mt19937 r;
 
   typename Allocator::AllocatorCache cache;
   memset(&cache, 0, sizeof(cache));
   a->InitCache(&cache);
 
   EXPECT_EQ(a->Allocate(&cache, -1, 1), (void*)0);
   EXPECT_EQ(a->Allocate(&cache, -1, 1024), (void*)0);

 TEST(SanitizerCommon, CombinedAllocator64) {
   TestCombinedAllocator<Allocator64>();
 }
 
 TEST(SanitizerCommon, CombinedAllocator64Dynamic) {
   TestCombinedAllocator<Allocator64Dynamic>();
 }
 
+TEST(SanitizerCommon, CombinedAllocator64DynamicPremapped) {
+  ScopedPremappedHeap h;
+  TestCombinedAllocator<Allocator64Dynamic>(h.Addr());
+}
+
 #if !SANITIZER_ANDROID
 TEST(SanitizerCommon, CombinedAllocator64Compact) {
   TestCombinedAllocator<Allocator64Compact>();
 }
 #endif
 
 TEST(SanitizerCommon, CombinedAllocator64VeryCompact) {
   TestCombinedAllocator<Allocator64VeryCompact>();
 }
 #endif
 
 TEST(SanitizerCommon, SKIP_ON_SOLARIS_SPARCV9(CombinedAllocator32Compact)) {
   TestCombinedAllocator<Allocator32Compact>();
 }
 
 template <class Allocator>
-void TestSizeClassAllocatorLocalCache() {
+void TestSizeClassAllocatorLocalCache(uptr premapped_heap = 0) {
   using AllocatorCache = typename Allocator::AllocatorCache;
   AllocatorCache cache;
   Allocator *a = new Allocator();
 
-  a->Init(kReleaseToOSIntervalNever);
+  a->Init(kReleaseToOSIntervalNever, premapped_heap);
   memset(&cache, 0, sizeof(cache));
   cache.Init(0);
 
   const uptr kNumAllocs = 10000;
   const int kNumIter = 100;
   uptr saved_total = 0;
   for (int class_id = 1; class_id <= 5; class_id++) {
     for (int it = 0; it < kNumIter; it++) {

 TEST(SanitizerCommon, SizeClassAllocator64LocalCache) {
   TestSizeClassAllocatorLocalCache<Allocator64>();
 }
 
 TEST(SanitizerCommon, SizeClassAllocator64DynamicLocalCache) {
   TestSizeClassAllocatorLocalCache<Allocator64Dynamic>();
 }
 
+TEST(SanitizerCommon, SizeClassAllocator64DynamicPremappedLocalCache) {
+  ScopedPremappedHeap h;
+  TestSizeClassAllocatorLocalCache<Allocator64Dynamic>(h.Addr());
+}
+
 #if !SANITIZER_ANDROID
 TEST(SanitizerCommon, SizeClassAllocator64CompactLocalCache) {
   TestSizeClassAllocatorLocalCache<Allocator64Compact>();
 }
 #endif
 TEST(SanitizerCommon, SizeClassAllocator64VeryCompactLocalCache) {
   TestSizeClassAllocatorLocalCache<Allocator64VeryCompact>();
 }
TEST(Allocator, ScopedBuffer) {
   }
 }
 
 void IterationTestCallback(uptr chunk, void *arg) {
   reinterpret_cast<std::set<uptr> *>(arg)->insert(chunk);
 }
 
 template <class Allocator>
-void TestSizeClassAllocatorIteration() {
+void TestSizeClassAllocatorIteration(uptr premapped_heap = 0) {
   Allocator *a = new Allocator;
-  a->Init(kReleaseToOSIntervalNever);
+  a->Init(kReleaseToOSIntervalNever, premapped_heap);
   typename Allocator::AllocatorCache cache;
   memset(&cache, 0, sizeof(cache));
   cache.Init(0);
 
   static const uptr sizes[] = {1, 16, 30, 40, 100, 1000, 10000,
     50000, 60000, 100000, 120000, 300000, 500000, 1000000, 2000000};
 
   std::vector<void *> allocated;

 // to run them all at the same time. FIXME: Make them not flaky and reenable.
 #if !SANITIZER_WINDOWS
 TEST(SanitizerCommon, SizeClassAllocator64Iteration) {
   TestSizeClassAllocatorIteration<Allocator64>();
 }
 TEST(SanitizerCommon, SizeClassAllocator64DynamicIteration) {
   TestSizeClassAllocatorIteration<Allocator64Dynamic>();
 }
+TEST(SanitizerCommon, SizeClassAllocator64DynamicPremappedIteration) {
+  ScopedPremappedHeap h;
+  TestSizeClassAllocatorIteration<Allocator64Dynamic>(h.Addr());
+}
 #endif
 #endif
 
 TEST(SanitizerCommon, SKIP_ON_SOLARIS_SPARCV9(SizeClassAllocator32Iteration)) {
   TestSizeClassAllocatorIteration<Allocator32Compact>();
 }
 
 TEST(SanitizerCommon, LargeMmapAllocatorIteration) {