[scudo] Extract steps releaseToOSMaybe into functions in

[Chia-hungDuan]

Summary

This refactor helps us identify which steps need FLLock so that we can
reduce the holding time of FLLock in SizeClassAllocator64.

Also move the data members to the end of class to align the style in
SizeClassAllocator32.

[Chia-hungDuan]

Update comment

[cferris]

compiler-rt/lib/scudo/standalone/primary64.h

912–913

Should maybe say:

in step 3 & 4 described in the comment below.

922

worth doing a page

925

It feels like there is a subtle difference between this and the previous code. If BytesInFreeList is zero, we return no matter the value of ReleaseType. Now, if ReleaseType is Force, then we will continue even if BytesInFreeList is zero. Is that expected?

This check feels like it's not quite right, it feels like it should be checking ForceAll instead.

963

can release pages.

964

I think you mean that the groups have pages that can be released?

The second sentence might be better written as:

Use a heuristic to gather groups that are candidates for doing a release.

Or maybe something similar.

968

It's not clear what the difference is between Normal, Force and ForceAll. Why is using the Force option doing this? It probably warrants a comment.

[Chia-hungDuan]

Address review comments

compiler-rt/lib/scudo/standalone/primary64.h

925

You're right, the Force here and below should be ForceAll instead.

About the BytesInFreeList == 0, you're right, it should be checked for all ReleaseType.

After the fixes above, there is one slightly difference is

if (BytesInFreeList <=
    Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint) {
  Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint = BytesInFreeList;
}

It seems that we bypass the update for ForceAll, however, this variable will be updated after releaseFreeMemoryToOS()

releaseFreeMemoryToOS(Context, Recorder, SkipRegion);

if (Recorder.getReleasedRangesCount() > 0) {
  Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint = BytesInFreeList;
  Region->ReleaseInfo.RangesReleased += Recorder.getReleasedRangesCount();
  Region->ReleaseInfo.LastReleasedBytes = Recorder.getReleasedBytes();
}

Given that ForceAll is likely to release some memory, I just lift the ForceAll check out the hasChanceToReleasePages and think it will align the ForceAll concept more.
"ForceAll doesn't take any heuristic, just release the pages (except the zero free byte)"

Same reason for the GroupsToRelease below.

968

This should be ForceAll. I guess using ForceAll will be self-explained here. Let me know if you think a comment will make it more clear.

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/primary64.h

968

Add one more explanation here,
In the past, even if ForceAll, we still do the CheckDensity part in collecting GroupsToRelease,

if (CheckDensity) {
  const uptr ReleaseThreshold =
      (AllocatedGroupSize * (100 - 1U - BlockSize / 16U)) / 100U;
  const bool HighDensity = BytesInBG >= ReleaseThreshold;
  const bool MayHaveReleasedAll = NumBlocks >= (GroupSize / BlockSize);

I think this was against our expectation for ForceAll so I also fix it here.

1091–1092

This is another stuff that ForceAll won't check after. I think this is a dead condition (if the AllocatedGroupSize is zero, it shouldn't have any BatchGroup created. I feel like this is a legacy check.

Will remove it along with next round review

[cferris]

compiler-rt/lib/scudo/standalone/primary64.h

969

I presume that if force all, then doing this make sense since you are going to operate on all of the blocks. Is there any chance that you could call this but not do a release? For example, would it ever be legal for hasBlockMarked to return false when BlockList is not empty?

It seems at least one potential issue is the page map cannot be created in markFreeBlocks. But maybe that causes problems in both cases.

1063

Isn't this clear unneeded? The class initializes Size, First and Last to the same values as clear does.

[Chia-hungDuan]

Address review comment

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/primary64.h

969

Nice Catch! This is a bug that has existed since we allow failure of markFreeBlocks(). Will fix that in a separate CL (will be the parent CL of this) and also restore the freelist in this CL

1063

hmm, I do have the impression that this won't clear. Maybe I messed it up with TransferBatch which doesn't do clear().

Removed.

[cferris]

LGTM.

compiler-rt/lib/scudo/standalone/primary64.h

struct UnpaddedRegionInfo {
     bool Exhausted GUARDED_BY(MMLock) = false;
   };
   struct RegionInfo : UnpaddedRegionInfo {
     char Padding[SCUDO_CACHE_LINE_SIZE -
                  (sizeof(UnpaddedRegionInfo) % SCUDO_CACHE_LINE_SIZE)] = {};
   };
   static_assert(sizeof(RegionInfo) % SCUDO_CACHE_LINE_SIZE == 0, "");
 
-  // TODO: `PrimaryBase` can be obtained from ReservedMemory. This needs to be
-  // deprecated.
-  uptr PrimaryBase = 0;
-  ReservedMemoryT ReservedMemory = {};
-  // The minimum size of pushed blocks that we will try to release the pages in
-  // that size class.
-  uptr SmallerBlockReleasePageDelta = 0;
-  atomic_s32 ReleaseToOsIntervalMs = {};
-  alignas(SCUDO_CACHE_LINE_SIZE) RegionInfo RegionInfoArray[NumClasses];
-
   RegionInfo *getRegionInfo(uptr ClassId) {
     DCHECK_LT(ClassId, NumClasses);
     return &RegionInfoArray[ClassId];
   }
 
   uptr getRegionBaseByClassId(uptr ClassId) {
     return roundDown(getRegionInfo(ClassId)->RegionBeg - PrimaryBase,
                      RegionSize) +
Str->append("%s %02zu (%6zu): mapped: %6zuK popped: %7zu pushed: %7zu "
                 Rss >> 10, Region->ReleaseInfo.RangesReleased,
                 Region->ReleaseInfo.LastReleasedBytes >> 10, Region->RegionBeg,
                 getRegionBaseByClassId(ClassId));
   }
 
   NOINLINE uptr releaseToOSMaybe(RegionInfo *Region, uptr ClassId,
                                  ReleaseToOS ReleaseType = ReleaseToOS::Normal)
       REQUIRES(Region->MMLock) EXCLUDES(Region->FLLock) {
-    // TODO(chiahungduan): Release `FLLock` when doing releaseFreeMemoryToOS().
+    // TODO(chiahungduan): Release `FLLock` in step 3 & 4 described in the
+    // comment below.
     ScopedLock L(Region->FLLock);
 
     const uptr BlockSize = getSizeByClassId(ClassId);
-    const uptr PageSize = getPageSizeCached();
-
-    DCHECK_GE(Region->FreeListInfo.PoppedBlocks,
-              Region->FreeListInfo.PushedBlocks);
     const uptr BytesInFreeList =
         Region->MemMapInfo.AllocatedUser - (Region->FreeListInfo.PoppedBlocks -
                                             Region->FreeListInfo.PushedBlocks) *
                                                BlockSize;
-
     if (UNLIKELY(BytesInFreeList == 0))
+      return false;
+
+    const uptr AllocatedUserEnd =
+        Region->MemMapInfo.AllocatedUser + Region->RegionBeg;
+    const uptr CompactPtrBase = getCompactPtrBaseByClassId(ClassId);
+
+    // ====================================================================== //
+    // 1. Check if we have enough free blocks and if it's worth doing a page
+    //    release.
+    // ====================================================================== //
+    if (ReleaseType != ReleaseToOS::ForceAll &&
+        !hasChanceToReleasePages(Region, BlockSize, BytesInFreeList,
+                                 ReleaseType)) {
+      return 0;
+    }
+
+    // This is only used for debugging to ensure the consistency of the number
+    // of groups.
+    uptr NumberOfBatchGroups = Region->FreeListInfo.BlockList.size();
+
+    // ====================================================================== //
+    // 2. Determine which groups can release the pages. Use a heuristic to
+    //    gather groups that are candidates for doing a release.
+    // ====================================================================== //
+    SinglyLinkedList<BatchGroup> GroupsToRelease;
+    if (ReleaseType == ReleaseToOS::ForceAll) {
+      GroupsToRelease = Region->FreeListInfo.BlockList;
+      Region->FreeListInfo.BlockList.clear();
+    } else {
+      GroupsToRelease = collectGroupsToRelease(
+          Region, BlockSize, AllocatedUserEnd, CompactPtrBase);
+    }
+    if (GroupsToRelease.empty())
       return 0;
 
+    // ====================================================================== //
+    // 3. Mark the free blocks in `GroupsToRelease` in the `PageReleaseContext`.
+    //    Then we can tell which pages are in-use by querying
+    //    `PageReleaseContext`.
+    // ====================================================================== //
+    PageReleaseContext Context = markFreeBlocks(
+        Region, BlockSize, AllocatedUserEnd, CompactPtrBase, GroupsToRelease);
+    if (UNLIKELY(!Context.hasBlockMarked())) {
+      mergeGroupsToReleaseBack(Region, GroupsToRelease, NumberOfBatchGroups);
+      return 0;
+    }
+
+    // ====================================================================== //
+    // 4. Release the unused physical pages back to the OS.
+    // ====================================================================== //
+    RegionReleaseRecorder<MemMapT> Recorder(&Region->MemMapInfo.MemMap,
+                                            Region->RegionBeg,
+                                            Context.getReleaseOffset());
+    auto SkipRegion = [](UNUSED uptr RegionIndex) { return false; };
+    releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
+    if (Recorder.getReleasedRangesCount() > 0) {
+      Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint = BytesInFreeList;
+      Region->ReleaseInfo.RangesReleased += Recorder.getReleasedRangesCount();
+      Region->ReleaseInfo.LastReleasedBytes = Recorder.getReleasedBytes();
+    }
+    Region->ReleaseInfo.LastReleaseAtNs = getMonotonicTimeFast();
+
+    // ====================================================================== //
+    // 5. Merge the `GroupsToRelease` back to the freelist.
+    // ====================================================================== //
+    mergeGroupsToReleaseBack(Region, GroupsToRelease, NumberOfBatchGroups);
+
+    return Recorder.getReleasedBytes();
+  }
+
+  bool hasChanceToReleasePages(RegionInfo *Region, uptr BlockSize,
+                               uptr BytesInFreeList, ReleaseToOS ReleaseType)
+      REQUIRES(Region->MMLock, Region->FLLock) {
+    DCHECK_GE(Region->FreeListInfo.PoppedBlocks,
+              Region->FreeListInfo.PushedBlocks);
+    const uptr PageSize = getPageSizeCached();
+
     // Always update `BytesInFreeListAtLastCheckpoint` with the smallest value
     // so that we won't underestimate the releasable pages. For example, the
     // following is the region usage,
     //
     //  BytesInFreeListAtLastCheckpoint   AllocatedUser
     //                v                         v
     //  |--------------------------------------->
     //         ^                   ^
     //  BytesInFreeList     ReleaseThreshold
     //
     // In general, if we have collected enough bytes and the amount of free
     // bytes meets the ReleaseThreshold, we will try to do page release. If we
     // don't update `BytesInFreeListAtLastCheckpoint` when the current
     // `BytesInFreeList` is smaller, we may take longer time to wait for enough
     // freed blocks because we miss the bytes between
     // (BytesInFreeListAtLastCheckpoint - BytesInFreeList).
     if (BytesInFreeList <=
         Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint) {
       Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint = BytesInFreeList;
     }
 
     const uptr RegionPushedBytesDelta =
         BytesInFreeList - Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint;
-    if (RegionPushedBytesDelta < PageSize &&
-        ReleaseType != ReleaseToOS::ForceAll) {
-      return 0;
-    }
+    if (RegionPushedBytesDelta < PageSize)
+      return false;
 
-    const bool CheckDensity = isSmallBlock(BlockSize);
     // Releasing smaller blocks is expensive, so we want to make sure that a
     // significant amount of bytes are free, and that there has been a good
     // amount of batches pushed to the freelist before attempting to release.
-    if (CheckDensity && ReleaseType == ReleaseToOS::Normal)
+    if (isSmallBlock(BlockSize) && ReleaseType == ReleaseToOS::Normal)
       if (RegionPushedBytesDelta < Region->TryReleaseThreshold)
-        return 0;
+        return false;
 
     if (ReleaseType == ReleaseToOS::Normal) {
       const s32 IntervalMs = atomic_load_relaxed(&ReleaseToOsIntervalMs);
       if (IntervalMs < 0)
-        return 0;
+        return false;
 
       // The constant 8 here is selected from profiling some apps and the number
       // of unreleased pages in the large size classes is around 16 pages or
       // more. Choose half of it as a heuristic and which also avoids page
       // release every time for every pushBlocks() attempt by large blocks.
       const bool ByPassReleaseInterval =
           isLargeBlock(BlockSize) && RegionPushedBytesDelta > 8 * PageSize;
       if (!ByPassReleaseInterval) {
         if (Region->ReleaseInfo.LastReleaseAtNs +
                 static_cast<u64>(IntervalMs) * 1000000 >
             getMonotonicTimeFast()) {
           // Memory was returned recently.
-          return 0;
+          return false;
         }
       }
     } // if (ReleaseType == ReleaseToOS::Normal)
 
-    const uptr GroupSize = (1U << GroupSizeLog);
-    const uptr AllocatedUserEnd =
-        Region->MemMapInfo.AllocatedUser + Region->RegionBeg;
-    const uptr CompactPtrBase = getCompactPtrBaseByClassId(ClassId);
-    auto DecompactPtr = [CompactPtrBase](CompactPtrT CompactPtr) {
-      return decompactPtrInternal(CompactPtrBase, CompactPtr);
-    };
-
-    // Instead of always preparing PageMap for the entire region, we only do it
-    // for the range of releasing groups. To do that, the free-block marking
-    // process includes visiting BlockGroups twice.
-
-    // The first visit is to determine the range of BatchGroups we are going to
-    // release. And we will extract those BatchGroups out and push into
-    // `GroupToRelease`.
-    SinglyLinkedList<BatchGroup> GroupToRelease;
-    GroupToRelease.clear();
+    return true;
+  }
 
-    // This is only used for debugging to ensure the consistency of the number
-    // of groups.
-    uptr NumberOfBatchGroups = Region->FreeListInfo.BlockList.size();
+  SinglyLinkedList<BatchGroup>
+  collectGroupsToRelease(RegionInfo *Region, const uptr BlockSize,
+                         const uptr AllocatedUserEnd, const uptr CompactPtrBase)
+      REQUIRES(Region->MMLock, Region->FLLock) {
+    const uptr GroupSize = (1U << GroupSizeLog);
+    const uptr PageSize = getPageSizeCached();
+    SinglyLinkedList<BatchGroup> GroupsToRelease;
 
     // We are examining each group and will take the minimum distance to the
     // release threshold as the next Region::TryReleaseThreshold(). Note that if
     // the size of free blocks has reached the release threshold, the distance
     // to the next release will be PageSize * SmallerBlockReleasePageDelta. See
     // the comment on `SmallerBlockReleasePageDelta` for more details.
     uptr MinDistToThreshold = GroupSize;
 
     for (BatchGroup *BG = Region->FreeListInfo.BlockList.front(),
for (BatchGroup *BG = Region->FreeListInfo.BlockList.front(),
       //             ---   GroupSize   ---   ---   GroupSize   ---
       //
       // After decompacting the CompactPtrGroupBase, we expect the alignment
       // property is held as well.
       const uptr BatchGroupBase =
           decompactGroupBase(CompactPtrBase, BG->CompactPtrGroupBase);
       DCHECK_LE(Region->RegionBeg, BatchGroupBase);
       DCHECK_GE(AllocatedUserEnd, BatchGroupBase);
       DCHECK_EQ((Region->RegionBeg - BatchGroupBase) % GroupSize, 0U);
-      const uptr BatchGroupEnd = BatchGroupBase + GroupSize;
-      const uptr AllocatedGroupSize = AllocatedUserEnd >= BatchGroupEnd
-                                          ? GroupSize
-                                          : AllocatedUserEnd - BatchGroupBase;
-      if (AllocatedGroupSize == 0) {
-        Prev = BG;
-        BG = BG->Next;
-        continue;
-      }
-
       // TransferBatches are pushed in front of BG.Batches. The first one may
       // not have all caches used.
       const uptr NumBlocks = (BG->Batches.size() - 1) * BG->MaxCachedPerBatch +
                              BG->Batches.front()->getCount();
       const uptr BytesInBG = NumBlocks * BlockSize;
 
-      if (ReleaseType != ReleaseToOS::ForceAll &&
-          BytesInBG <= BG->BytesInBGAtLastCheckpoint) {
+      if (BytesInBG <= BG->BytesInBGAtLastCheckpoint) {
         BG->BytesInBGAtLastCheckpoint = BytesInBG;
         Prev = BG;
         BG = BG->Next;
         continue;
       }
 
       const uptr PushedBytesDelta = BG->BytesInBGAtLastCheckpoint - BytesInBG;
 
       // Given the randomness property, we try to release the pages only if the
       // bytes used by free blocks exceed certain proportion of group size. Note
       // that this heuristic only applies when all the spaces in a BatchGroup
       // are allocated.
-      if (CheckDensity) {
+      if (isSmallBlock(BlockSize)) {
+        const uptr BatchGroupEnd = BatchGroupBase + GroupSize;
+        const uptr AllocatedGroupSize = AllocatedUserEnd >= BatchGroupEnd
+                                            ? GroupSize
+                                            : AllocatedUserEnd - BatchGroupBase;
         const uptr ReleaseThreshold =
             (AllocatedGroupSize * (100 - 1U - BlockSize / 16U)) / 100U;
         const bool HighDensity = BytesInBG >= ReleaseThreshold;
         const bool MayHaveReleasedAll = NumBlocks >= (GroupSize / BlockSize);
         // If all blocks in the group are released, we will do range marking
         // which is fast. Otherwise, we will wait until we have accumulated
         // a certain amount of free memory.
         const bool ReachReleaseDelta =
for (BatchGroup *BG = Region->FreeListInfo.BlockList.front(),
       //   |       |        |
       //   v       v        v
       //  +--+    +--+     +--+
       //  |  |-+  |X |  +->|  | -> ...
       //  +--+ |  +--+  |  +--+
       //       +--------+
       //
       // Note that we need to advance before pushing this BatchGroup to
-      // GroupToRelease because it's a destructive operation.
+      // GroupsToRelease because it's a destructive operation.
 
       BatchGroup *Cur = BG;
       BG = BG->Next;
 
       // Ideally, we may want to update this only after successful release.
       // However, for smaller blocks, each block marking is a costly operation.
       // Therefore, we update it earlier.
-      // TODO: Consider updating this after page release if `ReleaseRecorder`
-      // can tell the releasd bytes in each group.
+      // TODO: Consider updating this after releasing pages if `ReleaseRecorder`
+      // can tell the released bytes in each group.
       Cur->BytesInBGAtLastCheckpoint = BytesInBG;
 
       if (Prev != nullptr)
         Region->FreeListInfo.BlockList.extract(Prev, Cur);
       else
         Region->FreeListInfo.BlockList.pop_front();
-      GroupToRelease.push_back(Cur);
+      GroupsToRelease.push_back(Cur);
     }
 
     // Only small blocks have the adaptive `TryReleaseThreshold`.
     if (isSmallBlock(BlockSize)) {
       // If the MinDistToThreshold is not updated, that means each memory group
       // may have only pushed less than a page size. In that case, just set it
       // back to normal.
       if (MinDistToThreshold == GroupSize)
         MinDistToThreshold = PageSize * SmallerBlockReleasePageDelta;
       Region->TryReleaseThreshold = MinDistToThreshold;
     }
 
-    if (GroupToRelease.empty())
-      return 0;
+    return GroupsToRelease;
+  }
+
+  PageReleaseContext
+  markFreeBlocks(RegionInfo *Region, const uptr BlockSize,
+                 const uptr AllocatedUserEnd, const uptr CompactPtrBase,
+                 SinglyLinkedList<BatchGroup> &GroupsToRelease)
+      REQUIRES(Region->MMLock, Region->FLLock) {
+    const uptr GroupSize = (1U << GroupSizeLog);
+    auto DecompactPtr = [CompactPtrBase](CompactPtrT CompactPtr) {
+      return decompactPtrInternal(CompactPtrBase, CompactPtr);
+    };
 
     const uptr ReleaseBase = decompactGroupBase(
-        CompactPtrBase, GroupToRelease.front()->CompactPtrGroupBase);
+        CompactPtrBase, GroupsToRelease.front()->CompactPtrGroupBase);
     const uptr LastGroupEnd =
         Min(decompactGroupBase(CompactPtrBase,
-                               GroupToRelease.back()->CompactPtrGroupBase) +
+                               GroupsToRelease.back()->CompactPtrGroupBase) +
                 GroupSize,
             AllocatedUserEnd);
     // The last block may straddle the group boundary. Rounding up to BlockSize
     // to get the exact range.
     const uptr ReleaseEnd =
         roundUpSlow(LastGroupEnd - Region->RegionBeg, BlockSize) +
         Region->RegionBeg;
     const uptr ReleaseRangeSize = ReleaseEnd - ReleaseBase;
     const uptr ReleaseOffset = ReleaseBase - Region->RegionBeg;
 
-    RegionReleaseRecorder<MemMapT> Recorder(&Region->MemMapInfo.MemMap,
-                                            Region->RegionBeg, ReleaseOffset);
     PageReleaseContext Context(BlockSize, /*NumberOfRegions=*/1U,
                                ReleaseRangeSize, ReleaseOffset);
     // We may not be able to do the page release in a rare case that we may
     // fail on PageMap allocation.
-    if (Context.ensurePageMapAllocated()) {
-      for (BatchGroup &BG : GroupToRelease) {
-        const uptr BatchGroupBase =
-            decompactGroupBase(CompactPtrBase, BG.CompactPtrGroupBase);
-        const uptr BatchGroupEnd = BatchGroupBase + GroupSize;
-        const uptr AllocatedGroupSize = AllocatedUserEnd >= BatchGroupEnd
-                                            ? GroupSize
-                                            : AllocatedUserEnd - BatchGroupBase;
-        const uptr BatchGroupUsedEnd = BatchGroupBase + AllocatedGroupSize;
-        const bool MayContainLastBlockInRegion =
-            BatchGroupUsedEnd == AllocatedUserEnd;
-        const bool BlockAlignedWithUsedEnd =
-            (BatchGroupUsedEnd - Region->RegionBeg) % BlockSize == 0;
+    if (UNLIKELY(!Context.ensurePageMapAllocated()))
+      return Context;
+
+    for (BatchGroup &BG : GroupsToRelease) {
 
-        uptr MaxContainedBlocks = AllocatedGroupSize / BlockSize;
-        if (!BlockAlignedWithUsedEnd)
-          ++MaxContainedBlocks;
 
-        const uptr NumBlocks = (BG.Batches.size() - 1) * BG.MaxCachedPerBatch +
-                               BG.Batches.front()->getCount();
 
-        if (NumBlocks == MaxContainedBlocks) {
-          for (const auto &It : BG.Batches)
-            for (u16 I = 0; I < It.getCount(); ++I)
-              DCHECK_EQ(compactPtrGroup(It.get(I)), BG.CompactPtrGroupBase);
 
-          Context.markRangeAsAllCounted(BatchGroupBase, BatchGroupUsedEnd,
-                                        Region->RegionBeg, /*RegionIndex=*/0,
-                                        Region->MemMapInfo.AllocatedUser);
-        } else {
-          DCHECK_LT(NumBlocks, MaxContainedBlocks);
-          // Note that we don't always visit blocks in each BatchGroup so that
-          // we may miss the chance of releasing certain pages that cross
-          // BatchGroups.
-          Context.markFreeBlocksInRegion(
-              BG.Batches, DecompactPtr, Region->RegionBeg, /*RegionIndex=*/0,
-              Region->MemMapInfo.AllocatedUser, MayContainLastBlockInRegion);
-        }
-      }
+        // Note that we don't always visit blocks in each BatchGroup so that we
+        // may miss the chance of releasing certain pages that cross
 
-      DCHECK(Context.hasBlockMarked());
 
-      auto SkipRegion = [](UNUSED uptr RegionIndex) { return false; };
-      releaseFreeMemoryToOS(Context, Recorder, SkipRegion);
-
-      if (Recorder.getReleasedRangesCount() > 0) {
-        Region->ReleaseInfo.BytesInFreeListAtLastCheckpoint = BytesInFreeList;
-        Region->ReleaseInfo.RangesReleased += Recorder.getReleasedRangesCount();
-        Region->ReleaseInfo.LastReleasedBytes = Recorder.getReleasedBytes();
-      }
-      Region->ReleaseInfo.LastReleaseAtNs = getMonotonicTimeFast();
-    } // if (Context.ensurePageMapAllocated())
+    return Context;
 
-    // Merge GroupToRelease back to the Region::FreeListInfo.BlockList. Note
-    // that both `Region->FreeListInfo.BlockList` and `GroupToRelease` are
+  void mergeGroupsToReleaseBack(RegionInfo *Region,
+                                SinglyLinkedList<BatchGroup> &GroupsToRelease,
+                                const uptr NumberOfBatchGroups)
+      REQUIRES(Region->MMLock, Region->FLLock) {
+    // Merge GroupsToRelease back to the Region::FreeListInfo.BlockList. Note
+    // that both `Region->FreeListInfo.BlockList` and `GroupsToRelease` are
     // sorted.
     for (BatchGroup *BG = Region->FreeListInfo.BlockList.front(),
                     *Prev = nullptr;
          ;) {
-      if (BG == nullptr || GroupToRelease.empty()) {
-        if (!GroupToRelease.empty())
-          Region->FreeListInfo.BlockList.append_back(&GroupToRelease);
+      if (BG == nullptr || GroupsToRelease.empty()) {
+        if (!GroupsToRelease.empty())
+          Region->FreeListInfo.BlockList.append_back(&GroupsToRelease);
         break;
       }
 
       DCHECK_NE(BG->CompactPtrGroupBase,
-                GroupToRelease.front()->CompactPtrGroupBase);
+                GroupsToRelease.front()->CompactPtrGroupBase);
 
       if (BG->CompactPtrGroupBase <
-          GroupToRelease.front()->CompactPtrGroupBase) {
+          GroupsToRelease.front()->CompactPtrGroupBase) {
         Prev = BG;
         BG = BG->Next;
         continue;
       }
 
       // At here, the `BG` is the first BatchGroup with CompactPtrGroupBase
-      // larger than the first element in `GroupToRelease`. We need to insert
-      // `GroupToRelease::front()` (which is `Cur` below)  before `BG`.
+      // larger than the first element in `GroupsToRelease`. We need to insert
+      // `GroupsToRelease::front()` (which is `Cur` below)  before `BG`.
       //
       //   1. If `Prev` is nullptr, we simply push `Cur` to the front of
       //      FreeListInfo.BlockList.
       //   2. Otherwise, use `insert()` which inserts an element next to `Prev`.
       //
       // Afterwards, we don't need to advance `BG` because the order between
-      // `BG` and the new `GroupToRelease::front()` hasn't been checked.
-      BatchGroup *Cur = GroupToRelease.front();
-      GroupToRelease.pop_front();
+      // `BG` and the new `GroupsToRelease::front()` hasn't been checked.
+      BatchGroup *Cur = GroupsToRelease.front();
+      GroupsToRelease.pop_front();
       if (Prev == nullptr)
         Region->FreeListInfo.BlockList.push_front(Cur);
       else
         Region->FreeListInfo.BlockList.insert(Prev, Cur);
       DCHECK_EQ(Cur->Next, BG);
       Prev = Cur;
     }
 
     DCHECK_EQ(Region->FreeListInfo.BlockList.size(), NumberOfBatchGroups);
     (void)NumberOfBatchGroups;
 
     if (SCUDO_DEBUG) {
       BatchGroup *Prev = Region->FreeListInfo.BlockList.front();
       for (BatchGroup *Cur = Prev->Next; Cur != nullptr;
            Prev = Cur, Cur = Cur->Next) {
         CHECK_LT(Prev->CompactPtrGroupBase, Cur->CompactPtrGroupBase);
       }
     }
-
-    return Recorder.getReleasedBytes();
   }
+
+  // TODO: `PrimaryBase` can be obtained from ReservedMemory. This needs to be
+  // deprecated.
+  uptr PrimaryBase = 0;
+  ReservedMemoryT ReservedMemory = {};
+  // The minimum size of pushed blocks that we will try to release the pages in
+  // that size class.
+  uptr SmallerBlockReleasePageDelta = 0;
+  atomic_s32 ReleaseToOsIntervalMs = {};
+  alignas(SCUDO_CACHE_LINE_SIZE) RegionInfo RegionInfoArray[NumClasses];
 };
 
 } // namespace scudo
 
 #endif // SCUDO_PRIMARY64_H_

compiler-rt/lib/scudo/standalone/release.h

if (BlockSize <= PageSize && PageSize % BlockSize == 0) {
         }
       }
     }
 
     return true;
   }
 
   uptr getPageIndex(uptr P) { return (P >> PageSizeLog) - ReleasePageOffset; }
+  uptr getReleaseOffset() { return ReleasePageOffset << PageSizeLog; }
 
   uptr BlockSize;
   uptr NumberOfRegions;
   // For partial region marking, some pages in front are not needed to be
   // counted.
   uptr ReleasePageOffset;
   uptr PageSize;
   uptr PagesCount;