[scudo] Mark all blocks in a range without visiting each of them

[Chia-hungDuan]

Summary

When all the blocks in the group are known to be used, we should just
mark the pages in the range as all counted instead of visiting each of
them. This will reduce the time of marking free blocks especially for
smaller size class.

[Chia-hungDuan]

Rebase to a utility change and add more tests

[cferris]

Do you have any numbers that show how many times the code chooses the mark all range path? Also, what is the difference in performance using the mark all range path? I assume the other path is variable so it maybe some kind of average savings.

compiler-rt/lib/scudo/standalone/primary32.h

786

from the same

787

tricky case

788

while

Don't need the doing.

794

This check looks odd. Don't the blocks have to be in the same group id to do range marking? I'm probably misunderstanding what this is actually checking though.

796

It's kind of confusing to do it this way. Can you reverse the if and else so that you are not checking nots and ors? I think it makes the code easier to understand when you make the check CandDoRangeMark && NumBlocks == MaxContainedBlocks.

compiler-rt/lib/scudo/standalone/release.h

348

to the

348

that

349

sitting across

compiler-rt/lib/scudo/standalone/tests/release_test.cpp

335

not supposed

350

sitting

355

straddling

[Chia-hungDuan]

Rebase

[Chia-hungDuan]

Address review commnet

[Chia-hungDuan]

In D141958#4101464, @cferris wrote:

Do you have any numbers that show how many times the code chooses the mark all range path? Also, what is the difference in performance using the mark all range path? I assume the other path is variable so it maybe some kind of average savings.

Sure, I will bring some numbers later soon.

compiler-rt/lib/scudo/standalone/primary32.h

794

The trick thing is the current region may not have used up all region space. It only uses AllocatedUser. However, in PageReleaseContext, we have the logic of marking the last in-use page for each Region but the (RegionSize == AllocatedUser) introduces a different assumption which we need special handling for it. This condition will be removed in D143303 which simplify the block marking logic.

BTW, we don't release the last page in current region before D143303 either.

796

Yep, Agree! I thought the CanDoRangeMark would be removed in the later CL soon so I left it as it is. Also change the order in primary64 to make them consistent

[Chia-hungDuan]

Just a quick update,

Size class 32 w/o range marking

-- Average Operation Time -- -- Name (# of Calls) --
       1987080.0(ns)            DoPageRelease (1)
       1849800.0(ns)              PageMarking (1)
        135860.0(ns)              ReleasePage (1)

Size class 32 w/ range marking

-- Average Operation Time -- -- Name (# of Calls) --
       1435780.0(ns)            DoPageRelease (1)
       1283760.0(ns)              PageMarking (1)
        150330.0(ns)              ReleasePage (1)

The PageMarking difference is 566040 (ns). Total allocated blocks is 65536. This aims to simply measure the worst case only so I disable proactive releasing, i.e., only M_PURGE will release pages. Will collect more data later.

[cferris]

Assuming all of the benchmarking keeps showing positive results, LGTM.

[Chia-hungDuan]

The followings are measured by stressing certain size class, allocate up to 16 MB then randomly release them

size class 32
-- Average Operation Time -- -- Name (# of Calls) --
        987335.9(ns)            markFreeBlocks (39)
          2447.4(ns)            markRangeAsAllCounted (47)

size class 512

-- Average Operation Time -- -- Name (# of Calls) --
         26269.0(ns)            markFreeBlocks (51)
           495.8(ns)            markRangeAsAllCounted (16)

size class 8192
-- Average Operation Time -- -- Name (# of Calls) --
          9530.6(ns)            markFreeBlocks (49)
          1427.0(ns)            markRangeAsAllCounted (14)

Even the randomness, we can see that it still has some cases fall into range-marking. For real use cases on the phone, the frequency of falling into the range-marking is lower, usually happens when there's some heavy computation like scrolling the launcher. As a result, I think this still give good performance improvement in real cases.

compiler-rt/lib/scudo/standalone/primary32.h

public:
   void *decompactPtr(UNUSED uptr ClassId, CompactPtrT CompactPtr) const {
     return reinterpret_cast<void *>(static_cast<uptr>(CompactPtr));
   }
 
   uptr compactPtrGroup(CompactPtrT CompactPtr) {
     return CompactPtr >> GroupSizeLog;
   }
 
+  uptr batchGroupBase(uptr GroupId) { return GroupId << GroupSizeLog; }
+
   TransferBatch *popBatch(CacheT *C, uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     SizeClassInfo *Sci = getSizeClassInfo(ClassId);
     ScopedLock L(Sci->Mutex);
     TransferBatch *B = popBatchImpl(C, ClassId, Sci);
     if (UNLIKELY(!B)) {
       if (UNLIKELY(!populateFreeList(C, ClassId, Sci)))
         return nullptr;
for (BatchGroup &BG : Sci->FreeList) {
       // bytes used by free blocks exceed certain proportion of allocated
       // spaces.
       if (CheckDensity && (BytesInBG * 100U) / AllocatedGroupSize <
                               (100U - 1U - BlockSize / 16U)) {
         continue;
       }
 
       BG.PushedBlocksAtLastCheckpoint = BG.PushedBlocks;
+
+      const uptr MaxContainedBlocks = AllocatedGroupSize / BlockSize;
+      // The first condition to do range marking is that all the blocks in the
+      // range need to be from same region. In SizeClassAllocator32, this is
+      // true when GroupSize and RegionSize are the same. Another tirck case is,
+      // while doing range marking, the last block in a region needs the logic
+      // to mark the last page. However, in SizeClassAllocator32, the RegionSize
+      // recorded in PageReleaseContext may be different from
+      // `CurrentRegionAllocated` of the current region. This exception excludes
+      // the chance of doing range marking for the current region.
+      const bool CanDoRangeMark =
+          GroupSize == RegionSize && BG.GroupId != CurRegionGroupId;
+
+      if (!CanDoRangeMark || NumBlocks != MaxContainedBlocks) {
+        if (CanDoRangeMark)
+          DCHECK_LT(NumBlocks, MaxContainedBlocks);
+
         // Note that we don't always visit blocks in each BatchGroup so that we
-      // may miss the chance of releasing certain pages that cross BatchGroups.
+        // may miss the chance of releasing certain pages that cross
+        // BatchGroups.
         Context.markFreeBlocks(BG.Batches, DecompactPtr, Base);
+      } else {
+        for (const auto &It : BG.Batches)
+          for (u16 I = 0; I < It.getCount(); ++I)
+            DCHECK_EQ(compactPtrGroup(It.get(I)), BG.GroupId);
+
+        const uptr From = batchGroupBase(BG.GroupId);
+        const uptr To = batchGroupBase(BG.GroupId) + AllocatedGroupSize;
+        Context.markRangeAsAllCounted(From, To, Base);
+      }
     }
 
     if (!Context.hasBlockMarked())
       return 0;
 
     auto SkipRegion = [this, First, ClassId](uptr RegionIndex) {
       return (PossibleRegions[First + RegionIndex] - 1U) != ClassId;
     };

compiler-rt/lib/scudo/standalone/primary64.h

for (BatchGroup &BG : Region->FreeList) {
       // that this heuristic only applies when all the spaces in a BatchGroup
       // are allocated.
       if (CheckDensity && (BytesInBG * 100U) / AllocatedGroupSize <
                               (100U - 1U - BlockSize / 16U)) {
         continue;
       }
 
       BG.PushedBlocksAtLastCheckpoint = BG.PushedBlocks;
+
+      const uptr BatchGroupUsedEnd = BatchGroupBeg + AllocatedGroupSize;
+      const bool BlockAlignedWithUsedEnd =
+          (BatchGroupUsedEnd - Region->RegionBeg) % BlockSize == 0;
+
+      uptr MaxContainedBlocks = AllocatedGroupSize / BlockSize;
+      if (!BlockAlignedWithUsedEnd)
+        ++MaxContainedBlocks;
+
+      if (NumBlocks != MaxContainedBlocks) {
+        DCHECK_LT(NumBlocks, MaxContainedBlocks);
         // Note that we don't always visit blocks in each BatchGroup so that we
-      // may miss the chance of releasing certain pages that cross BatchGroups.
+        // may miss the chance of releasing certain pages that cross
+        // BatchGroups.
         Context.markFreeBlocks(BG.Batches, DecompactPtr, Region->RegionBeg);
+      } else {
+        for (const auto &It : BG.Batches)
+          for (u16 I = 0; I < It.getCount(); ++I)
+            DCHECK_EQ(compactPtrGroup(It.get(I)), BG.GroupId);
+
+        Context.markRangeAsAllCounted(BatchGroupBeg, BatchGroupUsedEnd,
+                                      Region->RegionBeg);
+      }
     }
 
     if (!Context.hasBlockMarked())
       return 0;
 
     auto SkipRegion = [](UNUSED uptr RegionIndex) { return false; };
     releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
 

compiler-rt/lib/scudo/standalone/release.h

void inc(uptr Region, uptr I) const {
     const uptr Index = I >> PackingRatioLog;
     const uptr BitOffset = (I & BitOffsetMask) << CounterSizeBitsLog;
     DCHECK_LT(BitOffset, SCUDO_WORDSIZE);
     DCHECK_EQ(isAllCounted(Region, I), false);
     Buffer[Region * SizePerRegion + Index] += static_cast<uptr>(1U)
                                               << BitOffset;
   }
 
+  void incN(uptr Region, uptr I, uptr N) const {
+    DCHECK_GT(N, 0U);
+    DCHECK_LE(N, CounterMask);
+    DCHECK_LE(get(Region, I), CounterMask - N);
+    const uptr Index = I >> PackingRatioLog;
+    const uptr BitOffset = (I & BitOffsetMask) << CounterSizeBitsLog;
+    DCHECK_LT(BitOffset, SCUDO_WORDSIZE);
+    DCHECK_EQ(isAllCounted(Region, I), false);
+    Buffer[Region * SizePerRegion + Index] += N << BitOffset;
+  }
+
   void incRange(uptr Region, uptr From, uptr To) const {
     DCHECK_LE(From, To);
     const uptr Top = Min(To + 1, NumCounters);
     for (uptr I = From; I < Top; I++)
       inc(Region, I);
   }
 
   // Set the counter to the max value. Note that the max number of blocks in a
   // page may vary. To provide an easier way to tell if all the blocks are
   // counted for different pages, set to the same max value to denote the
   // all-counted status.
   void setAsAllCounted(uptr Region, uptr I) const {
     DCHECK_LE(get(Region, I), CounterMask);
     const uptr Index = I >> PackingRatioLog;
     const uptr BitOffset = (I & BitOffsetMask) << CounterSizeBitsLog;
     DCHECK_LT(BitOffset, SCUDO_WORDSIZE);
     Buffer[Region * SizePerRegion + Index] |= CounterMask << BitOffset;
   }
+  void setAsAllCountedRange(uptr Region, uptr From, uptr To) const {
+    DCHECK_LE(From, To);
+    const uptr Top = Min(To + 1, NumCounters);
+    for (uptr I = From; I < Top; I++)
+      setAsAllCounted(Region, I);
+  }
+
+  bool updateAsAllCountedIf(uptr Region, uptr I, uptr MaxCount) {
+    const uptr Count = get(Region, I);
+    if (Count == CounterMask)
+      return true;
+    if (Count == MaxCount) {
+      setAsAllCounted(Region, I);
+      return true;
+    }
+    return false;
+  }
   bool isAllCounted(uptr Region, uptr I) const {
     return get(Region, I) == CounterMask;
   }
 
   uptr getBufferSize() const { return BufferSize; }
 
   static const uptr StaticBufferCount = 2048U;
 
struct PageReleaseContext {
 
   void ensurePageMapAllocated() {
     if (PageMap.isAllocated())
       return;
     PageMap.reset(NumberOfRegions, PagesCount, FullPagesBlockCountMax);
     DCHECK(PageMap.isAllocated());
   }
 
+  // Mark all the blocks in the given range [From, to). Instead of visiting all
+  // the blocks, we will just mark the page as all counted. Note the `From` and
+  // `To` has to be page aligned but with one exception, if `To` is equal to the
+  // RegionSize, it's not necessary to be aligned with page size.
+  void markRangeAsAllCounted(uptr From, uptr To, uptr Base) {
+    DCHECK_LT(From, To);
+    DCHECK_EQ(From % PageSize, 0U);
+
+    ensurePageMapAllocated();
+
+    const uptr FromOffset = From - Base;
+    const uptr ToOffset = To - Base;
+
+    const uptr RegionIndex =
+        NumberOfRegions == 1U ? 0 : FromOffset / RegionSize;
+    if (SCUDO_DEBUG) {
+      const uptr ToRegionIndex =
+          NumberOfRegions == 1U ? 0 : (ToOffset - 1) / RegionSize;
+      CHECK_EQ(RegionIndex, ToRegionIndex);
+    }
+
+    uptr FromInRegion = FromOffset - RegionIndex * RegionSize;
+    uptr ToInRegion = ToOffset - RegionIndex * RegionSize;
+    uptr FirstBlockInRange = roundUpSlow(FromInRegion, BlockSize);
+
+    // The straddling block sits across entire range.
+    if (FirstBlockInRange >= ToInRegion)
+      return;
+
+    // First block may not sit at the first pape in the range, move
+    // `FromInRegion` to the first block page.
+    FromInRegion = roundDown(FirstBlockInRange, PageSize);
+
+    // When The first block is not aligned the range boundary, which means there
+    // is a block sits acorss `From`, which looks like,
+    //
+    //   From                                             To
+    //     V                                               V
+    //     +-----------------------------------------------+
+    //  +-----+-----+-----+-----+
+    //  |     |     |     |     | ...
+    //  +-----+-----+-----+-----+
+    //     |-    first page     -||-    second page    -||- ...
+    //
+    // Therefore, we can't just mark the first page as all counted. Instead, we
+    // increment the number of blocks in the first page in the page map and
+    // then round up the `From` to the next page.
+    if (FirstBlockInRange != FromInRegion) {
+      DCHECK_GT(FromInRegion + PageSize, FirstBlockInRange);
+      uptr NumBlocksInFirstPage =
+          (FromInRegion + PageSize - FirstBlockInRange + BlockSize - 1) /
+          BlockSize;
+      PageMap.incN(RegionIndex, FromInRegion >> PageSizeLog,
+                   NumBlocksInFirstPage);
+      FromInRegion = roundUp(FromInRegion + 1, PageSize);
+    }
+
+    uptr LastBlockInRange = roundDownSlow(ToInRegion - 1, BlockSize);
+    if (LastBlockInRange < FromInRegion)
+      return;
+
+    // When the last block sits across `To`, we can't just mark the pages
+    // occupied by the last block as all counted. Instead, we increment the
+    // counters of those pages by 1. The exception is that if it's the last
+    // block in the region, it's fine to mark those pages as all counted.
+    if (LastBlockInRange + BlockSize != RegionSize) {
+      DCHECK_EQ(ToInRegion % PageSize, 0U);
+      // The case below is like,
+      //
+      //   From                                      To
+      //     V                                        V
+      //     +----------------------------------------+
+      //                          +-----+-----+-----+-----+
+      //                          |     |     |     |     | ...
+      //                          +-----+-----+-----+-----+
+      //                    ... -||-    last page    -||-    next page    -|
+      //
+      // The last block is not aligned to `To`, we need to increment the
+      // counter of `next page` by 1.
+      if (LastBlockInRange + BlockSize != ToInRegion) {
+        PageMap.incRange(RegionIndex, ToInRegion >> PageSizeLog,
+                         (LastBlockInRange + BlockSize - 1) >> PageSizeLog);
+      }
+    } else {
+      ToInRegion = RegionSize;
+    }
+
+    // After handling the first page and the last block, it's safe to mark any
+    // page in between the range [From, To).
+    if (FromInRegion < ToInRegion) {
+      PageMap.setAsAllCountedRange(RegionIndex, FromInRegion >> PageSizeLog,
+                                   (ToInRegion - 1) >> PageSizeLog);
+    }
+  }
+
   template<class TransferBatchT, typename DecompactPtrT>
   void markFreeBlocks(const IntrusiveList<TransferBatchT> &FreeList,
                       DecompactPtrT DecompactPtr, uptr Base) {
     ensurePageMapAllocated();
 
     // Iterate over free chunks and count how many free chunks affect each
     // allocated page.
     if (BlockSize <= PageSize && PageSize % BlockSize == 0) {
releaseFreeMemoryToOS(PageReleaseContext &Context,
   if (SameBlockCountPerPage) {
     // Fast path, every page has the same number of chunks affecting it.
     for (uptr I = 0; I < NumberOfRegions; I++) {
       if (SkipRegion(I)) {
         RangeTracker.skipPages(PagesCount);
         continue;
       }
       for (uptr J = 0; J < PagesCount; J++) {
-        const bool CanRelease = PageMap.get(I, J) == FullPagesBlockCountMax;
-        if (CanRelease)
-          PageMap.setAsAllCounted(I, J);
+        const bool CanRelease =
+            PageMap.updateAsAllCountedIf(I, J, FullPagesBlockCountMax);
         RangeTracker.processNextPage(CanRelease);
       }
     }
   } else {
     // Slow path, go through the pages keeping count how many chunks affect
     // each page.
     const uptr Pn = BlockSize < PageSize ? PageSize / BlockSize : 1;
     const uptr Pnc = Pn * BlockSize;
for (uptr I = 0; I < NumberOfRegions; I++) {
             BlocksPerPage++;
           CurrentBoundary += Pnc;
           if (CurrentBoundary < PageBoundary) {
             BlocksPerPage++;
             CurrentBoundary += BlockSize;
           }
         }
         PrevPageBoundary = PageBoundary;
-        const bool CanRelease = PageMap.get(I, J) == BlocksPerPage;
-        if (CanRelease)
-          PageMap.setAsAllCounted(I, J);
+        const bool CanRelease =
+            PageMap.updateAsAllCountedIf(I, J, BlocksPerPage);
         RangeTracker.processNextPage(CanRelease);
       }
     }
   }
   RangeTracker.finish();
 }
 
 // An overload releaseFreeMemoryToOS which doesn't require the page usage

compiler-rt/lib/scudo/standalone/tests/release_test.cpp

for (scudo::uptr I = 0; (SCUDO_WORDSIZE >> I) != 0; I++) {
     EXPECT_EQ(0UL, PageMap.get(0U, NumCounters - 1));
     PageMap.inc(0U, NumCounters - 1);
     if (I > 0) {
       PageMap.incRange(0u, 0U, NumCounters - 1);
       for (scudo::uptr C = 0; C < NumCounters; C++)
         EXPECT_EQ(2UL, PageMap.get(0U, C));
     }
   }
+
+  // Similar to the above except that we are using incN().
+  for (scudo::uptr I = 0; (SCUDO_WORDSIZE >> I) != 0; I++) {
+    // Make sure counters request one memory page for the buffer.
+    const scudo::uptr NumCounters =
+        (scudo::getPageSizeCached() / 8) * (SCUDO_WORDSIZE >> I);
+    scudo::uptr MaxValue = 1UL << ((1UL << I) - 1);
+    if (MaxValue <= 1U)
+      continue;
+
+    scudo::RegionPageMap PageMap(1U, NumCounters, MaxValue);
+
+    scudo::uptr N = MaxValue / 2;
+    PageMap.incN(0U, 0, N);
+    for (scudo::uptr C = 1; C < NumCounters; C++) {
+      EXPECT_EQ(0UL, PageMap.get(0U, C));
+      PageMap.incN(0U, C, N);
+      EXPECT_EQ(N, PageMap.get(0U, C - 1));
+    }
+    EXPECT_EQ(N, PageMap.get(0U, NumCounters - 1));
+  }
 }
 
 class StringRangeRecorder {
 public:
   std::string ReportedPages;
 
   StringRangeRecorder()
       : PageSizeScaledLog(scudo::getLog2(scudo::getPageSizeCached())) {}
for (scudo::uptr I = 1; I <= SizeClassMap::LargestClassId; I++) {
     while (!FreeList.empty()) {
       CurrentBatch = FreeList.front();
       FreeList.pop_front();
       delete CurrentBatch;
     }
   }
 }
 
+template <class SizeClassMap> void testPageMapMarkRange() {
+  const scudo::uptr PageSize = scudo::getPageSizeCached();
+
+  for (scudo::uptr I = 1; I <= SizeClassMap::LargestClassId; I++) {
+    const scudo::uptr BlockSize = SizeClassMap::getSizeByClassId(I);
+
+    const scudo::uptr GroupNum = 2;
+    const scudo::uptr GroupSize = scudo::roundUp(BlockSize, PageSize) * 2;
+    const scudo::uptr RegionSize =
+        scudo::roundUpSlow(GroupSize * GroupNum, BlockSize);
+    const scudo::uptr RoundedRegionSize = scudo::roundUp(RegionSize, PageSize);
+
+    std::vector<scudo::uptr> Pages(RoundedRegionSize / PageSize, 0);
+    for (scudo::uptr Block = 0; Block + BlockSize <= RoundedRegionSize;
+         Block += BlockSize) {
+      for (scudo::uptr page = Block / PageSize;
+           page <= (Block + BlockSize - 1) / PageSize; ++page) {
+        ASSERT_LT(page, Pages.size());
+        ++Pages[page];
+      }
+    }
+
+    for (scudo::uptr GroupId = 0; GroupId < GroupNum; ++GroupId) {
+      const scudo::uptr GroupBeg = GroupId * GroupSize;
+      const scudo::uptr GroupEnd = GroupBeg + GroupSize;
+
+      scudo::PageReleaseContext Context(BlockSize, RegionSize,
+                                        /*NumberOfRegions=*/1U);
+      Context.markRangeAsAllCounted(GroupBeg, GroupEnd, /*Base=*/0);
+
+      scudo::uptr FirstBlock =
+          ((GroupBeg + BlockSize - 1) / BlockSize) * BlockSize;
+
+      // All the pages before first block page are supposed not to be marked.
+      if (FirstBlock / PageSize > 0) {
+        for (scudo::uptr Page = 0; Page <= FirstBlock / PageSize - 1; ++Page)
+          EXPECT_EQ(Context.PageMap.get(/*Region=*/0, Page), 0U);
+      }
+
+      // Verify the pages used by the blocks in the group except that if the
+      // end of the last block is not aligned with `GroupEnd`, it'll be verified
+      // later.
+      scudo::uptr Block;
+      for (Block = FirstBlock; Block + BlockSize <= GroupEnd;
+           Block += BlockSize) {
+        for (scudo::uptr Page = Block / PageSize;
+             Page <= (Block + BlockSize - 1) / PageSize; ++Page) {
+          // First used page in the group has two cases, which are w/ and w/o
+          // block sits across the boundary.
+          if (Page == FirstBlock / PageSize) {
+            if (FirstBlock % PageSize == 0) {
+              EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0U, Page));
+            } else {
+              // There's a block straddles `GroupBeg`, it's supposed to only
+              // increment the counter and we expect it should be 1 less
+              // (exclude the straddling one) than the total blocks on the page.
+              EXPECT_EQ(Context.PageMap.get(/*Region=*/0U, Page),
+                        Pages[Page] - 1);
+            }
+          } else {
+            EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0, Page));
+          }
+        }
+      }
+
+      if (Block == GroupEnd)
+        continue;
+
+      // Examine the last block which sits across the group boundary.
+      if (Block + BlockSize == RegionSize) {
+        // This is the last block in the region, it's supposed to mark all the
+        // pages as all counted.
+        for (scudo::uptr Page = Block / PageSize;
+             Page <= (Block + BlockSize - 1) / PageSize; ++Page) {
+          EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0, Page));
+        }
+      } else {
+        for (scudo::uptr Page = Block / PageSize;
+             Page <= (Block + BlockSize - 1) / PageSize; ++Page) {
+          if (Page <= (GroupEnd - 1) / PageSize)
+            EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0, Page));
+          else
+            EXPECT_EQ(Context.PageMap.get(/*Region=*/0U, Page), 1U);
+        }
+      }
+
+      const scudo::uptr FirstUncountedPage =
+          scudo::roundUp(Block + BlockSize, PageSize);
+      for (scudo::uptr Page = FirstUncountedPage;
+           Page <= RoundedRegionSize / PageSize; ++Page) {
+        EXPECT_EQ(Context.PageMap.get(/*Region=*/0U, Page), 0U);
+      }
+    } // Iterate each Group
+
+    // Release the entire region. This is to ensure the last page is counted.
+    scudo::PageReleaseContext Context(BlockSize, RegionSize,
+                                      /*NumberOfRegions=*/1U);
+    Context.markRangeAsAllCounted(/*From=*/0U, /*To=*/RegionSize, /*Base=*/0);
+    for (scudo::uptr Page = 0; Page < RoundedRegionSize / PageSize; ++Page)
+      EXPECT_TRUE(Context.PageMap.isAllCounted(/*Region=*/0, Page));
+  } // Iterate each size class
+}
+
 TEST(ScudoReleaseTest, ReleaseFreeMemoryToOSDefault) {
   testReleaseFreeMemoryToOS<scudo::DefaultSizeClassMap>();
 }
 
 TEST(ScudoReleaseTest, ReleaseFreeMemoryToOSAndroid) {
   testReleaseFreeMemoryToOS<scudo::AndroidSizeClassMap>();
 }
 
 TEST(ScudoReleaseTest, ReleaseFreeMemoryToOSSvelte) {
   testReleaseFreeMemoryToOS<scudo::SvelteSizeClassMap>();
 }
+
+TEST(ScudoReleaseTest, PageMapMarkRange) {
+  testPageMapMarkRange<scudo::DefaultSizeClassMap>();
+  testPageMapMarkRange<scudo::AndroidSizeClassMap>();
+  testPageMapMarkRange<scudo::FuchsiaSizeClassMap>();
+  testPageMapMarkRange<scudo::SvelteSizeClassMap>();
+}