[scudo] Manage pages with MemMap in SizeClassAllocator64

[Chia-hungDuan]

Summary

Introduce a new data structure to manage the allocated pages from the
system. This is meant to deprecate certain memory system call wrappers
in Scudo, e.g., map()/unmap(). Besides, we would like to make
MapPlatformData to be appeared in platform specific data structure only.
Given that there are several allocators in Scudo and each of them has
different way of page management. The deprecation will be done in
several CLs. In this commit, we start from SizeClassAllocator64.

[Chia-hungDuan]

So far I call it RFC but it's a formal CL as well. There are still many places we can simplify. In order to make the review process easier, I'll do the simplification in coming CLs.

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/allocator_config.h

104 ↗

(On Diff #504942)

As @fabio-d suggested, I'll move this out from the config. So that we can still easily mix different configs on different platforms

[Chia-hungDuan]

Don't define MapReserver in allocator config

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/release.h

50

Note:This will be deprecated as well.

[Chia-hungDuan]

Make getCapacity as required and add few more comments

[Chia-hungDuan]

Add unmap() and remove the use of MapPlatformData in primary64

[Chia-hungDuan]

Remove the defining of MapReserver in the tests

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map.h

76

Per discussion, this may not be a good name. It may lead some confusion. I'm considering to make it either ReservedMemory or change it to has-a relation.

[Chia-hungDuan]

1. Rename MapReserver to ReservedMap.
2. Add more comments on why ReservedMap still derives from MemMapBase
3. Leave the unused functions in ReservedMap unimplemented (which were having default implementations which does nothing)

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map.h

81

Will be rephrased as

`ReservedMap` is a special MemMap which still manages a contiguous pages

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map.h

68

We may consider having two versions of this operation,

1. releaseAndZeroPagesToOS will ensure the later accessing the released pages will get it zero-filled
2. releasePagesToOS is optional to zero-fill

The reason for this is these two operations have performance difference on some platforms and in Scudo, not all releasePagesToOS calls reply on the zero-fill property.

Will update the CL with this change

[Chia-hungDuan]

Add releaseAndZeroPagesToOS and add more DCHECK in the base classes

[Chia-hungDuan]

Move some stuff to mem_map_base.h so that it will make the use of platform MemMap easier.

Also add more comments and checks

[Chia-hungDuan]

Slightly improve the semantic of DefaultMemMap::remap()

[cferris]

compiler-rt/lib/scudo/standalone/mem_map.cpp

41

Should you verify that Addr is within the actual map?

46

Similar to above, should you verify that this is within the actual map?

compiler-rt/lib/scudo/standalone/mem_map.h

57

This is slightly confusing. Perhaps reworded something like:

The following functions correspond to the Impl functions from MemMapBase.

compiler-rt/lib/scudo/standalone/mem_map_base.h

18

this

19–20

Do you mean before the operation is executed? The wording is a bit confusing.

22

A class derived from

22

handle contiguous pages

23

derived class uses.

28–29

it has to implement all of the 'Impl' named functions.

34

set of pages

35

any

58

set of physical

85

a set of contiguous pages

86

a set of contiguous pages

97

functions are not used and should not be defined in

98

following

99

that a derived class has to implement.

[Chia-hungDuan]

Address review comment

compiler-rt/lib/scudo/standalone/mem_map.cpp

41

The basic invariant of the API is verified in the base class (mem_map_base.h), For example,

bool remap(uptr Addr, uptr Size, const char *Name, uptr Flags = 0) {              
  DCHECK(isAllocated());                                                          
  DCHECK((Addr >= getMapBase()) ||                                                
         (Addr + Size <= getMapBase() + getMapCapacity()));                       
  return static_cast<Derived *>(this)->remapImpl(Addr, Size, Name, Flags);        
}

46

Same as above.

compiler-rt/lib/scudo/standalone/mem_map_base.h

19–20

Yes, reworded a little bit.

86

The suggestion is a little bit different from above. I follow the suggestion here (suppose the above didn't mark the right range to update)

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map_base.h

46

I'm wrapping this ugly cast as Fabio suggested

[Chia-hungDuan]

wrap static_cast<Derived *>(this)->...Impl(...)

[Chia-hungDuan]

Rename ReservedMap to ReservedMemory and untangle it with MemMap

Change the names of some APIs

[fabio-d]

Hi @Chia-hungDuan, thank you for bringing this new design forward!

I've left some notes on specific APIs, and I also have some other general suggestions:

• It would be useful to document what flags are acceptable in a comment next to each method.
• As you know, Fuchsia doesn't accept addresses and sizes that are not page-aligned, so I suggest to add page-alignment checks to all the arguments.
• I am slightly confused by the fact that some arguments are called Size but the corresponding property is called Capacity. Is there any specific reason why you didn't use Size everywhere?

compiler-rt/lib/scudo/standalone/mem_map.h

35–37

Since these methods are only called by the wrappers in MemMapBase and they always pass a Flags value, I suggest to remove the default values here

60–61

Same suggestion as above on the Flags.

compiler-rt/lib/scudo/standalone/mem_map_base.h

23

I assume that the contract is that the ctor creates an object such that !isAllocated(). Should we also have a dtor that automatically unmaps if still isAllocated()?

25–26

If Addr is only a suggestion, I suggest to remove it completely from this API. For carefully-placed memory ranges, clients will use ReservedMemory and dispatch

27–28

Are you referring to Trusty? I would move this comment to unmap

30

Should this also check that !isAllocated() as a precondition?

35

I think we do not need Flags for unmap in this new design. Is there any specific flag that you have in mind? From a quick look, it seems to me that we can safely get rid of the UNMAP_ALL flag if we add explicit alignment control to cover primary32.h's special case.

Also, Fuchsia does not have a syscall to shrink a VMO from the beginning. Would it be a problem if we only allowed to shrink from the end?

41–43

I would separate resizing and changing permissions into different methods (even on Linux, mremap and mprotect are separate syscalls).
Also, I'm not sure we actually need Addr at all:

• If this is being called with the intent of setting new permissions, I would assume that the upper layers would always want to apply them to the entire range covered by the MemMap. Unless we already have use case for this level flexibility, I would suggest to leave it out of the new API and we cann add it back later as needed.
• If this is being called with the intent of resizing, I also would assume that it applies to the entire range too (in fact, I cannot think of a good formal definition of "resizing a subrange", especially if the resized range ends before the end of the MemMap).

50–52

What do "physical" and "virtual" mean in this context? (maybe this a stale comment from the previous revision of this CL)

60

The distinction between releasePagesToOS and releaseAndZeroPagesToOS does not currently exist in Scudo, right? Can we add it as a new feature in a later CL?

87

Similar to above, I assume that the ctor creates an object such that !isCreated(). Should we also have a dtor that automatically releases if still isCreated()?

92

Should this also check that !isCreated() as a precondition?

96

Similarly, I think we do not need Flags for release either.

[Chia-hungDuan]

Address review comment

compiler-rt/lib/scudo/standalone/mem_map_base.h

23

These two are supposed to be lazy initialized so the creation needs to be done explicitly.

The order of destructing allocator fields is tricky. Especially we have the thread local cache. Given that tearing down the allocator only happens at program exit, we just make them being released naturally. Therefore, the creation/deletion are done explicitly.

We have been thinking to make it more elegantly but that depends on too many stuff (e.g., libc). Besides, this CL is supposed to do the abstraction transition only, I prefer to do further logic changes in the later CLs.

25–26

I prefer to keep this suggestion because it can still benefit some contexts (like some cache management stuff we will have in the future). It's a different case than ReservedMemory supposed to cover.

35

Like the release one, the only case I was thinking is the UNMAP_ALL for Fuchsia, if you think it's fine to remove it. I'm happy to do that :)

is there any alternative way to achieve this? Like create a new VMO ?

41–43

The MemMap is a handle of range of virtual address and it's supposed to do the supported operations in the range. So it can remap part of the virtual address. For example, the RegionBeg is unlikely to be equal to Region base address. That's why we have this Addr argument here (For future function it'll be useful as well). If a platform doesn't allow this, then we just do the valid operation in that platform to fulfill their constraint.

BTW, we have setMemoryPermission (which is added in the follow up CL)

50–52

It means virtual memory and physical memory.

60

In general, yes, we may consider adding this in a separate CL.

The reason I do it here is that we are revising API contract and try to reduce ambiguity. For example, the arguments of releasePagesToOS was confusing. And then we found that the zero-page stuff is ambiguous as well. That's why I bring it in this CL.

87

Same as above.

92

This is intended to make it implementation defined behavior. In some case we may want few reserved spaces.

96

I was hesitating this before and the only case is UNMAP_ALL used by Fuchsia. If you agree that we can get rid of it. I'm happy with that :) Done.

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map_base.h

92

Oops, please ignore this wrong reply.

[cferris]

LGTM.

[Chia-hungDuan]

Make ReservedMemory::create return boolean to indicate if the creation succeeds.
This makes the page allocation have the consistent style of API

[cferris]

Still looks good.

[fabio-d]

compiler-rt/lib/scudo/standalone/primary64.h

735–747

This used to be a grow operation for a memory region that always started at RegionBeg. Now we are dispatching at getRegionBaseByClassId(ClassId) and then resizing starting from RegionBeg. Can we directly dispatch at RegionBeg instead?
This results in MemMapDefault::releaseAndZeroPagesToOSImpl passing an offset relative to getRegionBaseByClassId(ClassId) instead of RegionBeg to the ::releasePagesToOS function, which makes Scudo release in-use pages on Fuchsia.

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/primary64.h

735–747

Because it's a region's base, I tended to think it's still part of a region. To make it work before transition to each platform specific implementation. I agree that we can dispatch the reserved memory from RegionBeg and fix it back later

[fabio-d]

Closing because the issue on Fuchsia was fixed in D147792

compiler-rt/lib/scudo/standalone/CMakeLists.txt

set(SCUDO_HEADERS
   flags_parser.h
   flags.h
   fuchsia.h
   internal_defs.h
   linux.h
   list.h
   local_cache.h
   memtag.h
+  mem_map.h
+  mem_map_base.h
   mutex.h
   options.h
   platform.h
   primary32.h
   primary64.h
   quarantine.h
   release.h
   report.h

 set(SCUDO_SOURCES
   checksum.cpp
   common.cpp
   crc32_hw.cpp
   flags_parser.cpp
   flags.cpp
   fuchsia.cpp
   linux.cpp
+  mem_map.cpp
   release.cpp
   report.cpp
   rss_limit_checker.cpp
   string_utils.cpp
   )
 
 # Enable the necessary instruction set for scudo_crc32.cpp, if available.
 # Newer compiler versions use -mcrc32 rather than -msse4.2.

compiler-rt/lib/scudo/standalone/mem_map.h

+//===-- mem_map.h -----------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SCUDO_MEM_MAP_H_
+#define SCUDO_MEM_MAP_H_
+
+#include "mem_map_base.h"
+
+#include "common.h"
+#include "internal_defs.h"
+
+// TODO: This is only used for `MapPlatformData`. Remove these includes when we
+// have all three platform specific `MemMap` and `ReservedMap` implementations.
+#include "fuchsia.h"
+#include "linux.h"
+#include "trusty.h"
+
+namespace scudo {
+
+// This will be deprecated when every allocator has been supported by each
+// platform's `MemMap` implementation.
+class DefaultMemMap final : public MemMapBase<DefaultMemMap> {
+public:
+  constexpr DefaultMemMap() = default;
+  DefaultMemMap(uptr Base, uptr Capacity)
+      : MapBase(Base), MapCapacity(Capacity) {}
+
+  // Impls for base functions.
+  bool mapImpl(uptr Addr, uptr Size, const char *Name, uptr Flags = 0);
+  void unmapImpl(uptr Addr, uptr Size, uptr Flags = 0);
+  bool remapImpl(uptr Addr, uptr Size, const char *Name, uptr Flags = 0);
+  void releasePagesToOSImpl(uptr From, uptr Size) {
+    return releaseAndZeroPagesToOSImpl(From, Size);
+  }
+  void releaseAndZeroPagesToOSImpl(uptr From, uptr Size);
+  uptr getMapBaseImpl() { return MapBase; }
+  uptr getMapCapacityImpl() { return MapCapacity; }
+
+  void setMapPlatformData(MapPlatformData &NewData) { Data = NewData; }
+
+private:
+  uptr MapBase = 0;
+  uptr MapCapacity = 0;
+  MapPlatformData Data = {};
+};
+
+// This will be deprecated when every allocator has been supported by each
+// platform's `MemMap` implementation.
+class DefaultReservedMap final
+    : public ReservedMap<DefaultReservedMap, DefaultMemMap> {
+public:
+  // The following threes are the Impls for function in `MemMapBase`.
+  uptr getMapBaseImpl() { return MapBase; }
+  uptr getMapCapacityImpl() { return MapCapacity; }
+  void unmapImpl(uptr Addr, uptr Size, uptr Flags = 0);
+
+  // These two are specific to `ReservedMap`.
+  void reserveImpl(uptr Addr, uptr Size, const char *Name, uptr Flags = 0);
+  MemMapT allocateImpl(uptr Addr, uptr Size);
+
+private:
+  uptr MapBase = 0;
+  uptr MapCapacity = 0;
+  MapPlatformData Data = {};
+};
+
+#if SCUDO_LINUX
+using ReservedMapT = DefaultReservedMap;
+using MemMapT = ReservedMapT::MemMapT;
+#elif SCUDO_FUCHSIA
+using ReservedMapT = DefaultReservedMap;
+using MemMapT = ReservedMapT::MemMapT;
+#elif SCUDO_TRUSTY
+using ReservedMapT = DefaultReservedMap;
+using MemMapT = ReservedMapT::MemMapT;
+#else
+#error                                                                         \
+    "Unsupported platform, please implement the ReservedMap for your platform!"
+#endif
+
+} // namespace scudo
+
+#endif // SCUDO_MEM_MAP_H_

compiler-rt/lib/scudo/standalone/mem_map.cpp

+//===-- mem_map.cpp ---------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mem_map.h"
+
+#include "common.h"
+
+namespace scudo {
+
+bool DefaultMemMap::mapImpl(uptr Addr, uptr Size, const char *Name,
+                            uptr Flags) {
+  void *MappedAddr =
+      ::scudo::map(reinterpret_cast<void *>(Addr), Size, Name, Flags, &Data);
+  if (MappedAddr == nullptr)
+    return false;
+  MapBase = reinterpret_cast<uptr>(MappedAddr);
+  MapCapacity = Size;
+  return true;
+}
+
+void DefaultMemMap::unmapImpl(uptr Addr, uptr Size, uptr Flags) {
+  if (Size == MapCapacity) {
+    MapBase = MapCapacity = 0;
+  } else {
+    if (MapBase == Addr)
+      MapBase = Addr + Size;
+    MapCapacity -= Size;
+  }
+
+  ::scudo::unmap(reinterpret_cast<void *>(Addr), Size, Flags, &Data);
+}
+
+bool DefaultMemMap::remapImpl(uptr Addr, uptr Size, const char *Name,
+                              uptr Flags) {
+  void *RemappedAddr =
+      ::scudo::map(reinterpret_cast<void *>(Addr), Size, Name, Flags, &Data);
+  return reinterpret_cast<uptr>(RemappedAddr) == Addr;
+}
+
+void DefaultMemMap::releaseAndZeroPagesToOSImpl(uptr From, uptr Size) {
+  return ::scudo::releasePagesToOS(MapBase, From - MapBase, Size, &Data);
+}
+
+void DefaultReservedMap::unmapImpl(uptr Addr, uptr Size, uptr Flags) {
+  ::scudo::unmap(reinterpret_cast<void *>(Addr), Size, Flags, &Data);
+  if (MapBase == Addr)
+    MapBase = Addr + Size;
+  MapCapacity -= Size;
+}
+
+void DefaultReservedMap::reserveImpl(uptr Addr, uptr Size, const char *Name,
+                                     uptr Flags) {
+  void *Reserved = ::scudo::map(reinterpret_cast<void *>(Addr), Size, Name,
+                                Flags | MAP_NOACCESS, &Data);
+  if (Reserved == nullptr)
+    return;
+
+  MapBase = reinterpret_cast<uptr>(Reserved);
+  MapCapacity = Size;
+}
+
+DefaultReservedMap::MemMapT DefaultReservedMap::allocateImpl(uptr Addr,
+                                                             uptr Size) {
+  DefaultReservedMap::MemMapT NewMap(Addr, Size);
+  NewMap.setMapPlatformData(Data);
+  return NewMap;
+}
+
+} // namespace scudo

compiler-rt/lib/scudo/standalone/mem_map_base.h

+//===-- mem_map_base.h ------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SCUDO_MEM_MAP_BASE_H_
+#define SCUDO_MEM_MAP_BASE_H_
+
+#include "common.h"
+
+namespace scudo {
+
+// In Scudo, each page request from the system will be tracked by a
+// platform-specific `MemMap` implementation. For example, `DefaultMemMap`
+// records the base of the contiguous pages and its size. With these
+// information, we are able to validate various page operations before it goes
+// beyond the first place.
+
+// A derived of `MemMapBase` is supposed to be a handle of a contiguous pages
+// and `MemMapBase` contains the interfaces that the derived usually uses. Note
+// that this class doesn't have any data member even for the map base and its
+// size. The reason for that is, different platforms may have its own data
+// structure to store those things. As a result, we simply list the common
+// interfaces for all platforms. This is implemented in CRTP, so for each
+// implementation, they have to implement the following functions with name
+// suffixing with `Impl`.
+template <class Derived> class MemMapBase {
+public:
+  constexpr MemMapBase() {}
+
+  // This is used to map a new contiguous pages. Whether the holding pages
+  // (if nay) will be released is implementation defined.
+  bool map(uptr Addr, uptr Size, const char *Name, uptr Flags = 0) {
+    return static_cast<Derived *>(this)->mapImpl(Addr, Size, Name, Flags);
+  }
+
+  // This is used to unmap partial/full pages from the beginning or the end.
+  // I.e., the result pages are expected to be still contiguous.
+  void unmap(uptr Addr, uptr Size, uptr Flags = 0) {
+    DCHECK(isAllocated());
+    DCHECK((Addr == getMapBase()) ||
+           (Addr + Size == getMapBase() + getMapCapacity()));
+    static_cast<Derived *>(this)->unmapImpl(Addr, Size, Flags);
+  }
+
+  // This is used to remap a mapped range. For example, we have reserved several
+  // pages and then we want to remap them with different flags.
+  bool remap(uptr Addr, uptr Size, const char *Name, uptr Flags = 0) {
+    DCHECK(isAllocated());
+    DCHECK((Addr >= getMapBase()) ||
+           (Addr + Size <= getMapBase() + getMapCapacity()));
+    return static_cast<Derived *>(this)->remapImpl(Addr, Size, Name, Flags);
+  }
+
+  // Release a contiguous physical pages back to the OS. Note that only physical
+  // pages are supposed to be released. Any release of virtual pages may lead
+  // to undefined behavior.
+  void releasePagesToOS(uptr From, uptr Size) {
+    DCHECK(isAllocated());
+    DCHECK((From >= getMapBase()) ||
+           (From + Size <= getMapBase() + getMapCapacity()));
+    return static_cast<Derived *>(this)->releasePagesToOSImpl(From, Size);
+  }
+  // This is similar to the above one except that any subsequent access to the
+  // released pages will return zero-filled pages.
+  void releaseAndZeroPagesToOS(uptr From, uptr Size) {
+    DCHECK(isAllocated());
+    DCHECK((From >= getMapBase()) ||
+           (From + Size <= getMapBase() + getMapCapacity()));
+    return static_cast<Derived *>(this)->releaseAndZeroPagesToOSImpl(From,
+                                                                     Size);
+  }
+
+  uptr getMapBase() { return static_cast<Derived *>(this)->getMapBaseImpl(); }
+  uptr getMapCapacity() {
+    return static_cast<Derived *>(this)->getMapCapacityImpl();
+  }
+
+  bool isAllocated() { return getMapBase() != 0U; }
+};
+
+// `ReservedMap` is a special MemMap which still manages a contiguous pages. The
+// difference is that it reserves a contiguous pages for later dispatch. On some
+// platforms, this may be a cheaper operation than a general map() call.
+// Therefore, here we add two functions, `reserve` and `allocate` to fulfill the
+// use cases. This is useful if you want to ensure the MemMaps are managed in a
+// given range. It can be viewed as a sub-page-allocator.
+template <class Derived, typename GeneralMapT>
+class ReservedMap : public MemMapBase<Derived> {
+public:
+  using Base = MemMapBase<Derived>;
+  using MemMapT = GeneralMapT;
+
+  // Note that `map`/`remap`/`releasePagesToOs` are supposed not to be used in
+  // `ReservedMap`. Leave them unimplemented intentionally here. The followings
+  // are the required functions and the derived has to implement them.
+  using Base::getMapBase;
+  using Base::getMapCapacity;
+  using Base::isAllocated;
+  // This is only used for test.
+  using Base::unmap;
+
+  // Reserve a chunk of memory. Note that if there's already a reserved range,
+  // the behavior is undefined.
+  void reserve(uptr Addr, uptr Size, const char *Name, uptr Flags = 0) {
+    static_cast<Derived *>(this)->reserveImpl(Addr, Size, Name, Flags);
+  }
+
+  // Prepare a MemMap from the reserved range. Note that any fragmentation of
+  // the reserved pages is managed by each implementation.
+  MemMapT allocate(uptr Addr, uptr Size) {
+    DCHECK(isAllocated());
+    DCHECK((Addr >= getMapBase()) ||
+           (Addr + Size <= getMapBase() + getMapCapacity()));
+    return static_cast<Derived *>(this)->allocateImpl(Addr, Size);
+  }
+};
+
+} // namespace scudo
+
+#endif // SCUDO_MEM_MAP_BASE_H_

compiler-rt/lib/scudo/standalone/primary64.h

 //===-- primary64.h ---------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef SCUDO_PRIMARY64_H_
 #define SCUDO_PRIMARY64_H_
 
 #include "bytemap.h"
 #include "common.h"
 #include "list.h"
 #include "local_cache.h"
+#include "mem_map.h"
 #include "memtag.h"
 #include "options.h"
 #include "release.h"
 #include "stats.h"
 #include "string_utils.h"
 #include "thread_annotations.h"
 
 namespace scudo {
public:
 
   static bool canAllocate(uptr Size) { return Size <= SizeClassMap::MaxSize; }
 
   void init(s32 ReleaseToOsInterval) NO_THREAD_SAFETY_ANALYSIS {
     DCHECK(isAligned(reinterpret_cast<uptr>(this), alignof(ThisT)));
     DCHECK_EQ(PrimaryBase, 0U);
 
     // Reserve the space required for the Primary.
-    PrimaryBase = reinterpret_cast<uptr>(map(
-        nullptr, PrimarySize, "scudo:primary_reserve", MAP_NOACCESS, &Data));
+    PrimaryMaps.reserve(/*Addr=*/0U, PrimarySize, "scudo:primary_reserve");
+    PrimaryBase = PrimaryMaps.getMapBase();
+    DCHECK_NE(PrimaryBase, 0U);
 
     u32 Seed;
     const u64 Time = getMonotonicTimeFast();
     if (!getRandom(reinterpret_cast<void *>(&Seed), sizeof(Seed)))
       Seed = static_cast<u32>(Time ^ (PrimaryBase >> 12));
     const uptr PageSize = getPageSizeCached();
     for (uptr I = 0; I < NumClasses; I++) {
       RegionInfo *Region = getRegionInfo(I);
public:
   }
 
   void unmapTestOnly() NO_THREAD_SAFETY_ANALYSIS {
     for (uptr I = 0; I < NumClasses; I++) {
       RegionInfo *Region = getRegionInfo(I);
       *Region = {};
     }
     if (PrimaryBase)
-      unmap(reinterpret_cast<void *>(PrimaryBase), PrimarySize, UNMAP_ALL,
-            &Data);
+      PrimaryMaps.unmap(PrimaryBase, PrimarySize, UNMAP_ALL);
     PrimaryBase = 0U;
   }
 
   TransferBatch *popBatch(CacheT *C, uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     RegionInfo *Region = getRegionInfo(ClassId);
     bool PrintStats = false;
     {
struct UnpaddedRegionInfo {
     // This is initialized before thread creation.
     uptr RegionBeg = 0;
     RegionStats Stats GUARDED_BY(Mutex) = {};
     u32 RandState GUARDED_BY(Mutex) = 0;
     // Bytes mapped for user memory.
     uptr MappedUser GUARDED_BY(Mutex) = 0;
     // Bytes allocated for user memory.
     uptr AllocatedUser GUARDED_BY(Mutex) = 0;
-    MapPlatformData Data GUARDED_BY(Mutex) = {};
+    MemMapT MemMap = {};
     ReleaseToOsInfo ReleaseInfo GUARDED_BY(Mutex) = {};
     bool Exhausted GUARDED_BY(Mutex) = false;
   };
   struct RegionInfo : UnpaddedRegionInfo {
     char Padding[SCUDO_CACHE_LINE_SIZE -
                  (sizeof(UnpaddedRegionInfo) % SCUDO_CACHE_LINE_SIZE)] = {};
   };
   static_assert(sizeof(RegionInfo) % SCUDO_CACHE_LINE_SIZE == 0, "");
 
+  // TODO: `PrimaryBase` can be obtained from PrimaryMaps. This needs to be
+  // deprecated.
   uptr PrimaryBase = 0;
+  ReservedMapT PrimaryMaps = {};
   // The minimum size of pushed blocks that we will try to release the pages in
   // that size class.
   uptr SmallerBlockReleasePageDelta = 0;
-  MapPlatformData Data = {};
   atomic_s32 ReleaseToOsIntervalMs = {};
   alignas(SCUDO_CACHE_LINE_SIZE) RegionInfo RegionInfoArray[NumClasses];
 
   RegionInfo *getRegionInfo(uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     return &RegionInfoArray[ClassId];
   }
 
if (TotalUserBytes > MappedUser) {
       // Do the mmap for the user memory.
       const uptr MapSize =
           roundUp(TotalUserBytes - MappedUser, MapSizeIncrement);
       const uptr RegionBase = RegionBeg - getRegionBaseByClassId(ClassId);
       if (UNLIKELY(RegionBase + MappedUser + MapSize > RegionSize)) {
         Region->Exhausted = true;
         return false;
       }
-      if (MappedUser == 0)
-        Region->Data = Data;
-      if (UNLIKELY(!map(
-              reinterpret_cast<void *>(RegionBeg + MappedUser), MapSize,
-              "scudo:primary",
+      // TODO: Consider allocating MemMap in init().
+      if (!Region->MemMap.isAllocated()) {
+        Region->MemMap =
+            PrimaryMaps.allocate(getRegionBaseByClassId(ClassId), RegionSize);
+      }
+      DCHECK(Region->MemMap.isAllocated());
+
+      if (UNLIKELY(!Region->MemMap.remap(
+              RegionBeg + MappedUser, MapSize, "scudo:primary",
               MAP_ALLOWNOMEM | MAP_RESIZABLE |
-                  (useMemoryTagging<Config>(Options.load()) ? MAP_MEMTAG : 0),
-              &Region->Data))) {
+                  (useMemoryTagging<Config>(Options.load()) ? MAP_MEMTAG
+                                                            : 0)))) {
         return false;
       }
       Region->MappedUser += MapSize;
       C->getStats().add(StatMapped, MapSize);
     }
 
     const u32 NumberOfBlocks = Min(
         MaxNumBatches * MaxCount,
         static_cast<u32>((Region->MappedUser - Region->AllocatedUser) / Size));
     DCHECK_GT(NumberOfBlocks, 0);
NOINLINE uptr releaseToOSMaybe(RegionInfo *Region, uptr ClassId,
     // The last block may straddle the group boundary. Rounding up to BlockSize
     // to get the exact range.
     const uptr ReleaseEnd =
         roundUpSlow(LastGroupEnd - Region->RegionBeg, BlockSize) +
         Region->RegionBeg;
     const uptr ReleaseRangeSize = ReleaseEnd - ReleaseBase;
     const uptr ReleaseOffset = ReleaseBase - Region->RegionBeg;
 
-    ReleaseRecorder Recorder(Region->RegionBeg, ReleaseOffset, &Region->Data);
+    RegionReleaseRecorder<MemMapT> Recorder(&Region->MemMap, Region->RegionBeg,
+                                            ReleaseOffset);
     PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
                                ReleaseRangeSize, ReleaseOffset);
 
     for (BatchGroup &BG : GroupToRelease) {
       BG.PushedBlocksAtLastCheckpoint = BG.PushedBlocks;
 
       const uptr BatchGroupBase =
           decompactGroupBase(CompactPtrBase, BG.CompactPtrGroupBase);

compiler-rt/lib/scudo/standalone/release.h

 //===-- release.h -----------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef SCUDO_RELEASE_H_
 #define SCUDO_RELEASE_H_
 
 #include "common.h"
 #include "list.h"
+#include "mem_map.h"
 #include "mutex.h"
 #include "thread_annotations.h"
 
 namespace scudo {
 
+template <typename MemMapT> class RegionReleaseRecorder {
+public:
+  RegionReleaseRecorder(MemMapT *RegionMemMap, uptr Base, uptr Offset = 0)
+      : RegionMemMap(RegionMemMap), Base(Base), Offset(Offset) {}
+
+  uptr getReleasedRangesCount() const { return ReleasedRangesCount; }
+
+  uptr getReleasedBytes() const { return ReleasedBytes; }
+
+  uptr getBase() const { return Base; }
+
+  // Releases [From, To) range of pages back to OS. Note that `From` and `To`
+  // are offseted from `Base` + Offset.
+  void releasePageRangeToOS(uptr From, uptr To) {
+    const uptr Size = To - From;
+    RegionMemMap->releasePagesToOS(getBase() + Offset + From, Size);
+    ReleasedRangesCount++;
+    ReleasedBytes += Size;
+  }
+
+private:
+  uptr ReleasedRangesCount = 0;
+  uptr ReleasedBytes = 0;
+  MemMapT *RegionMemMap = nullptr;
+  uptr Base = 0;
+  // The release offset from Base. This is used when we know a given range after
+  // Base will not be released.
+  uptr Offset = 0;
+};
+
 class ReleaseRecorder {
 public:
   ReleaseRecorder(uptr Base, uptr Offset = 0, MapPlatformData *Data = nullptr)
       : Base(Base), Offset(Offset), Data(Data) {}
 
   uptr getReleasedRangesCount() const { return ReleasedRangesCount; }
 
   uptr getReleasedBytes() const { return ReleasedBytes; }