scudo: Support memory tagging in the secondary allocator.

[pcc]

Summary

This patch enhances the secondary allocator to be able to detect
use-after-free, buffer overflow and (on hardware supporting
memory tagging) buffer underflow.

Use-after-free detection is implemented by setting memory page
protection to PROT_NONE on free. Because this must be done immediately
rather than after the memory has been quarantined, we no longer use the
combined allocator quarantine for secondary allocations. Instead, a
quarantine has been added to the secondary allocator cache.

Buffer overflow detection is implemented by aligning the allocation
to the right of the writable pages, so that any overflows will
spill into the guard page to the right of the allocation, which
will have PROT_NONE page protection. Because this would require the
secondary allocator to produce a header at the correct position,
the responsibility for ensuring chunk alignment has been moved to
the secondary allocator.

Buffer underflow detection has been implemented on hardware supporting
memory tagging by tagging the memory region between the start of the
mapping and the start of the allocation with a non-zero tag. Due to
the cost of pre-tagging secondary allocations, the allocation itself
uses a tag of 0.

[eugenis]

compiler-rt/lib/scudo/standalone/combined.h

255

Do you think it will be too expensive to make the tag random and use LDG to read it in the allocator?

1083

Rename the argument to "headerTaggedPtr" or something like that

compiler-rt/lib/scudo/standalone/secondary.h

133

Do you expect this to be faster than the other branch? If we are doing an mm syscall anyway, why not release the memory immediately?

523

is this an unrelated bugfix?

533

I don't know if it is worth the cost of the extra syscall, but we only need MAP_MEMTAG on the first page of the mapping.

[pcc]

compiler-rt/lib/scudo/standalone/combined.h

255

I didn't see a huge advantage in making it random. I think we need to be careful about loads which follow the pattern of LDG followed by a load as that effectively makes MTE ineffective for those loads. It seemed better to use a fixed tag since that provides at least a little more protection than none at all.

1083

Will do.

compiler-rt/lib/scudo/standalone/secondary.h

133

The advantage of not releasing the memory is that we don't have to pay the cost of faulting the memory once it is made writable again. (Or at least that's the theory. It turns out that the Linux kernel currently faults the accesses anyway, and I have a patch to get rid of the faults.) With that patch in place I did continue to measure a significant performance improvement for decay=1 versus decay=0.

523

No, this is a consequence of the fact that we are now passing the original size into this function rather than the size padded for alignment.

533

It's the first four pages I think, because the cache may end up reusing an allocation and throwing away the first four pages. I'm inclined to wait for data before trying to optimize this.

That being said, one simple thing that we can do here is to make the flag conditional on whether memory tagging is enabled, similar to what we're already doing in the primary allocator. I'll do that.

[pcc]

• Set MAP_MEMTAG only if tagging enabled

[eugenis]

LGTM

[cryptoad]

Is this still WIP like the title says?

[pcc]

It's ready to go in if you don't mind me breaking Fuchsia. Otherwise I can wait for a Fuchsia implementation of setMemoryPermission() from you.

[cryptoad]

In D93731#2482877, @pcc wrote:

It's ready to go in if you don't mind me breaking Fuchsia. Otherwise I can wait for a Fuchsia implementation of setMemoryPermission() from you.

Can we double check with cferris@ about any potential perf impact? I am going to have a look at the Fuchsia part.

[cryptoad]

In D93731#2482877, @pcc wrote:

It's ready to go in if you don't mind me breaking Fuchsia. Otherwise I can wait for a Fuchsia implementation of setMemoryPermission() from you.

Trying the following:

void setMemoryPermission(uptr Addr, uptr Size, uptr Flags,
                         MapPlatformData *Data) {
  const zx_vm_option_t Prot =
      (Flags & MAP_NOACCESS) ? 0 : (ZX_VM_PERM_READ | ZX_VM_PERM_WRITE);
  DCHECK(Data);
  DCHECK_NE(Data->Vmar, ZX_HANDLE_INVALID);
  if (_zx_vmar_protect(Data->Vmar, Prot, Addr, Size) != ZX_OK)
    dieOnMapUnmapError();
}

[cryptoad]

There seems to be something wrong on Fuchsia with it, regular processes are crashing. Looking into this.

[cryptoad]

compiler-rt/lib/scudo/standalone/secondary.h

60

The Fuchsia error is here: Fuchsia unmap tries to write to &H->Data post unmapping, so it results in an access violation.
The solution is to use a local Data variable that would store the header's Data.

[pcc]

Fixes for Fuchsia

[cryptoad]

compiler-rt/lib/scudo/standalone/fuchsia.cpp

139

So this ended up not working for me, because the protection must have at least 1 flag (https://fuchsia.dev/fuchsia-src/reference/syscalls/vmar_protect#description)
Did it work for you?

[pcc]

compiler-rt/lib/scudo/standalone/fuchsia.cpp

139

It seems to work for me. I don't think that documentation is accurate. I verified that this function is being called like so:

diff --git a/compiler-rt/lib/scudo/standalone/fuchsia.cpp b/compiler-rt/lib/scudo/standalone/fuchsia.cpp
index bf063e11bb3d..5e90373a7fc3 100644
--- a/compiler-rt/lib/scudo/standalone/fuchsia.cpp
+++ b/compiler-rt/lib/scudo/standalone/fuchsia.cpp
@@ -137,6 +137,8 @@ void unmap(void *Addr, uptr Size, uptr Flags, MapPlatformData *Data) {
 
 void setMemoryPermission(UNUSED uptr Addr, UNUSED uptr Size, UNUSED uptr Flags,
                          UNUSED MapPlatformData *Data) {
+  if (Flags & MAP_NOACCESS)
+    dieOnMapUnmapError();
   const zx_vm_option_t Prot =
       (Flags & MAP_NOACCESS) ? 0 : (ZX_VM_PERM_READ | ZX_VM_PERM_WRITE);
   DCHECK(Data);

With that I observed a crash in the unit tests (I don't think this code is ever actually called in production on Fuchsia because the Fuchsia allocator does not enable the cache).

I also made this change to verify that the permissions were being applied properly:

diff --git a/compiler-rt/lib/scudo/standalone/fuchsia.cpp b/compiler-rt/lib/scudo/standalone/fuchsia.cpp
index bf063e11bb3d..0d4f8d9bcddd 100644
--- a/compiler-rt/lib/scudo/standalone/fuchsia.cpp
+++ b/compiler-rt/lib/scudo/standalone/fuchsia.cpp
@@ -143,6 +143,10 @@ void setMemoryPermission(UNUSED uptr Addr, UNUSED uptr Size, UNUSED uptr Flags,
   DCHECK_NE(Data->Vmar, ZX_HANDLE_INVALID);
   if (_zx_vmar_protect(Data->Vmar, Prot, Addr, Size) != ZX_OK)
     dieOnMapUnmapError();
+  if (Flags & MAP_NOACCESS) {
+    volatile char C = *(char *)Addr;
+    (void)C;
+  }
 }
 
 void releasePagesToOS(UNUSED uptr BaseAddress, uptr Offset, uptr Size,

And again I observed a crash.

[cryptoad]

It all seems to be working.
I'll have to work on the EXPECT_DEATH aspect as a follow up.

[pcc]

• Disable memory tagging in the test allocator if disabled via prctl()

[cferris]

Sorry it took a long time to get back to this. Unfortunately, not only does this cause relatively large performance regressions, it also causes test failures. The test failures are related to malloc_iterate, but I haven't looked too closely at them yet. Is it possible that something needs to be updated in the code related to tracking the total allocated memory?

Benchmarks show as high as 10%, but many closer to 5% regression in other benchmarks.

As you mentioned, there are some large size calloc/mallocs that are 10x slower, for the 128KB sizes. These are all of the microbenchmarks, but even the trace benchmarks show large performance issues. I see 15% to 20% regressions on a few of these.

Also, this increases the RSS significantly on some of the traces. For example, the 32 bit camera trace that does a lot of large allocations went from ~50MB of RSS to ~55MB. It also increased the VA space on a few of the traces. For example, the maps 32 bit trace goes from ~38MB to ~46.5MB. We already have issues in 32 bit land with taking too much VA space, this would likely cause additional issues.

If there is a way to enable/disable this that would be great, otherwise, the regression is too large on almost every axis to take. Even with a possible kernel patch, I don't know if that will fix the RSS and VA space issues.

[pcc]

• Disable secondary quarantine/UAF detection if MTE is disabled; fix tests

[pcc]

In D93731#2516638, @cferris wrote:

Sorry it took a long time to get back to this. Unfortunately, not only does this cause relatively large performance regressions, it also causes test failures. The test failures are related to malloc_iterate, but I haven't looked too closely at them yet. Is it possible that something needs to be updated in the code related to tracking the total allocated memory?

Benchmarks show as high as 10%, but many closer to 5% regression in other benchmarks.

As you mentioned, there are some large size calloc/mallocs that are 10x slower, for the 128KB sizes. These are all of the microbenchmarks, but even the trace benchmarks show large performance issues. I see 15% to 20% regressions on a few of these.

Also, this increases the RSS significantly on some of the traces. For example, the 32 bit camera trace that does a lot of large allocations went from ~50MB of RSS to ~55MB. It also increased the VA space on a few of the traces. For example, the maps 32 bit trace goes from ~38MB to ~46.5MB. We already have issues in 32 bit land with taking too much VA space, this would likely cause additional issues.

If there is a way to enable/disable this that would be great, otherwise, the regression is too large on almost every axis to take. Even with a possible kernel patch, I don't know if that will fix the RSS and VA space issues.

I've changed the patch so that the secondary quarantine and UAF detection (the two main sources of overhead) are only enabled when MTE is enabled. I used the memory replay and performance trace benchmarks listed at https://source.corp.google.com/android/bionic/docs/native_allocator.md;bpv=0 and verified no regression on camera trace. PTAL.

[cferris]

Unfortunately, 32 bit tests are failing with this abort:

[ RUN ] ScudoCombinedTest.BasicCombined
Scudo ERROR: CHECK failed @ external/scudo/standalone/memtag.h:235 ((0 && "memory tagging not supported")) != (0) (0, 0)
Aborted

It looks like a number of the ScudoCombined tests all fail with this abort, but only on 32 bit. The 64 bit versions pass without any errors.

It looks like the test is trying to do something memtag related that isn't supported on 32 bit. Should the tests be disabled on 32 bit?

[pcc]

• Fix tests on non-arm64

[pcc]

In D93731#2537982, @cferris wrote:

Unfortunately, 32 bit tests are failing with this abort:

[ RUN ] ScudoCombinedTest.BasicCombined
Scudo ERROR: CHECK failed @ external/scudo/standalone/memtag.h:235 ((0 && "memory tagging not supported")) != (0) (0, 0)
Aborted

It looks like a number of the ScudoCombined tests all fail with this abort, but only on 32 bit. The 64 bit versions pass without any errors.

It looks like the test is trying to do something memtag related that isn't supported on 32 bit. Should the tests be disabled on 32 bit?

Sorry, I forgot to test on 32-bit. The problem with the tests should be fixed now.

[pcc]

• Bug fix

[hctim]

compiler-rt/lib/scudo/standalone/combined.h

828–836

Yeah, this unfortunately means that some of the sizeclasses get mapped as PROT_MTE even when MTE is disabled. Maybe we should pull this change out of this patch and follow up separately, as it sounds like we might need to pull the MTE options out of the primary options bitset so it doesn't get clobbered at init-time.

$ adb shell process/with/no/mte
$ adb shell cat /proc/<pid>/smaps | egrep 'VmFlags.*mt' -B23
7ca588f000-7ca58cf000 rw-p 00000000 00:00 0                              [anon:scudo:primary]
VmFlags: rd wr mr mw me ac mt
7cd588c000-7cd58cc000 rw-p 00000000 00:00 0                              [anon:scudo:primary]
VmFlags: rd wr mr mw me ac mt

[pcc]

compiler-rt/lib/scudo/standalone/combined.h

828–836

Pulling this change out of the patch would lead to MTE never being disabled (from the allocator's perspective), and consequent crashes due to an incorrect include mask. I'll see what can be done about these mappings.

[pcc]

• Avoid thread init, which may allocate via post-init callback

[pcc]

compiler-rt/lib/scudo/standalone/combined.h

828–836

It turned out that the mappings were created because of the post-init callback. I've changed the code to avoid calling the post-init callback from disableMemoryTagging().

[cferris]

I reran the benchmarks and everything looks much better. There were only a few micro-benchmarks that showed decreases that are worrying. For example:

32 bit:

BM_stdlib_calloc_free_decay1/131072 decreased by about 18%
BM_stdlib_calloc_free_default/131072 decreased by about 10%

Various multi-alloc (malloc_forty, malloc_multiple_8192 ones) benchmarks decreased by about 10%.

64 bit:

BM_stdlib_malloc_free_decay1/16384 decreased by about 7%.
BM_stdlib_malloc_free_default/16384 decreased by about 7%.
BM_stdlib_calloc_free_decay1/16384 decreased by about 7%.
BM_stdlib_calloc_free_default/131072 decreased by about 5%.

And then a few of malloc_forty ones decreased by about 12%.

The malloc_forty and malloc_multiple do not really ever correlate with real world code, so I don't think they matter. The larger allocations performance decreases are worrying. Does it make sense that we would see this decrease?

Nothing else showed any really big issues, which probably means that there aren't that many traces that do large allocations that would cause large performance degradation.

I'm inclined to say this is fine if the degradation makes sense.

[pcc]

• Keep track of the oldest cache entry's time in order to reduce the number of times we need to call releaseOlderThan()

[pcc]

In D93731#2558250, @cferris wrote:

I reran the benchmarks and everything looks much better. There were only a few micro-benchmarks that showed decreases that are worrying. For example:

32 bit:

BM_stdlib_calloc_free_decay1/131072 decreased by about 18%
BM_stdlib_calloc_free_default/131072 decreased by about 10%

Various multi-alloc (malloc_forty, malloc_multiple_8192 ones) benchmarks decreased by about 10%.

64 bit:

BM_stdlib_malloc_free_decay1/16384 decreased by about 7%.
BM_stdlib_malloc_free_default/16384 decreased by about 7%.
BM_stdlib_calloc_free_decay1/16384 decreased by about 7%.
BM_stdlib_calloc_free_default/131072 decreased by about 5%.

And then a few of malloc_forty ones decreased by about 12%.

The malloc_forty and malloc_multiple do not really ever correlate with real world code, so I don't think they matter. The larger allocations performance decreases are worrying. Does it make sense that we would see this decrease?

Nothing else showed any really big issues, which probably means that there aren't that many traces that do large allocations that would cause large performance degradation.

I'm inclined to say this is fine if the degradation makes sense.

I think I've managed to fix the performance regressions with the latest update.

[pcc]

• Create two maps for each MTE enabled secondary allocation in order to improve memory bandwidth utilization

[cferris]

Everything is better than before. There is still some benchmarks that were around 6% worse:

32
BM_stdlib_calloc_free_decay1/131072 about 6% worse
BM_stdlib_calloc_free_decay1/65536 about 6% worse
BM_stdlib_calloc_free_default/65536 about 6.6% worse

64 bit
BM_stdlib_calloc_free_default/131072 about 5% worse

However, some larger numbers also improved with this change, and all trace based were about the same. I think the calloc issue is related to zeroing out the memory (which is on by default on Android) and might not be related to this change, or made worse by this change.

Given this, I think this is good enough, but I'm still a bit worried about a gradual degradation of performance if it happens a bit at a time.

[pcc]

• Compute the correct value for BlockSize

[pcc]

The perf regression on Android appears to have been caused by an incorrect computation of BlockSize in the secondary allocator, which made its way into the statistics returned by mallinfo, which caused ART to enter a slow path. With this fixed I was unable to measure a regression.

[cryptoad]

compiler-rt/lib/scudo/standalone/secondary.h

371

Since this is up again for review, could you please find a workaround for the 0-sized array warning?

[pcc]

• Add -Wno-pedantic to silence GCC warning

[pcc]

compiler-rt/lib/scudo/standalone/secondary.h

371

Done; I just silenced the warning.

[pcc]

• Optionally add some slack at the end of secondary allocations

[eugenis]

LGTM

compiler-rt/lib/scudo/standalone/CMakeLists.txt

 
 # Remove -stdlib= which is unused when passing -nostdinc++.
 string(REGEX REPLACE "-stdlib=[a-zA-Z+]*" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
 
 append_list_if(COMPILER_RT_HAS_FFREESTANDING_FLAG -ffreestanding SCUDO_CFLAGS)
 
 append_list_if(COMPILER_RT_HAS_FVISIBILITY_HIDDEN_FLAG -fvisibility=hidden SCUDO_CFLAGS)
 
+append_list_if(COMPILER_RT_HAS_WNO_PEDANTIC -Wno-pedantic SCUDO_CFLAGS)
+
 # FIXME: find cleaner way to agree with GWPAsan flags
 append_list_if(COMPILER_RT_HAS_FNO_LTO_FLAG -fno-lto SCUDO_CFLAGS)
 
 if (COMPILER_RT_HAS_GWP_ASAN)
   append_list_if(COMPILER_RT_HAS_OMIT_FRAME_POINTER_FLAG -fno-omit-frame-pointer
                 SCUDO_CFLAGS)
   append_list_if(COMPILER_RT_HAS_OMIT_FRAME_POINTER_FLAG
                 -mno-omit-leaf-frame-pointer SCUDO_CFLAGS)

compiler-rt/lib/scudo/standalone/allocator_config.h

#else
   static const uptr PrimaryRegionSizeLog = 19U;
   typedef uptr PrimaryCompactPtrT;
 #endif
   static const s32 PrimaryMinReleaseToOsIntervalMs = INT32_MIN;
   static const s32 PrimaryMaxReleaseToOsIntervalMs = INT32_MAX;
 
   typedef MapAllocatorCache<DefaultConfig> SecondaryCache;
   static const u32 SecondaryCacheEntriesArraySize = 32U;
+  static const u32 SecondaryCacheQuarantineSize = 0U;
   static const u32 SecondaryCacheDefaultMaxEntriesCount = 32U;
   static const uptr SecondaryCacheDefaultMaxEntrySize = 1UL << 19;
   static const s32 SecondaryCacheMinReleaseToOsIntervalMs = INT32_MIN;
   static const s32 SecondaryCacheMaxReleaseToOsIntervalMs = INT32_MAX;
 
   template <class A> using TSDRegistryT = TSDRegistryExT<A>; // Exclusive
 };
 
#else
   static const uptr PrimaryRegionSizeLog = 18U;
   typedef uptr PrimaryCompactPtrT;
 #endif
   static const s32 PrimaryMinReleaseToOsIntervalMs = 1000;
   static const s32 PrimaryMaxReleaseToOsIntervalMs = 1000;
 
   typedef MapAllocatorCache<AndroidConfig> SecondaryCache;
   static const u32 SecondaryCacheEntriesArraySize = 256U;
+  static const u32 SecondaryCacheQuarantineSize = 32U;
   static const u32 SecondaryCacheDefaultMaxEntriesCount = 32U;
   static const uptr SecondaryCacheDefaultMaxEntrySize = 2UL << 20;
   static const s32 SecondaryCacheMinReleaseToOsIntervalMs = 0;
   static const s32 SecondaryCacheMaxReleaseToOsIntervalMs = 1000;
 
   template <class A>
   using TSDRegistryT = TSDRegistrySharedT<A, 8U, 2U>; // Shared, max 8 TSDs.
 };
#else
   static const uptr PrimaryRegionSizeLog = 16U;
   typedef uptr PrimaryCompactPtrT;
 #endif
   static const s32 PrimaryMinReleaseToOsIntervalMs = 1000;
   static const s32 PrimaryMaxReleaseToOsIntervalMs = 1000;
 
   typedef MapAllocatorCache<AndroidSvelteConfig> SecondaryCache;
   static const u32 SecondaryCacheEntriesArraySize = 16U;
+  static const u32 SecondaryCacheQuarantineSize = 32U;
   static const u32 SecondaryCacheDefaultMaxEntriesCount = 4U;
   static const uptr SecondaryCacheDefaultMaxEntrySize = 1UL << 18;
   static const s32 SecondaryCacheMinReleaseToOsIntervalMs = 0;
   static const s32 SecondaryCacheMaxReleaseToOsIntervalMs = 0;
 
   template <class A>
   using TSDRegistryT = TSDRegistrySharedT<A, 2U, 1U>; // Shared, max 2 TSDs.
 };

compiler-rt/lib/scudo/standalone/combined.h

void recycle(void *Ptr) {
       Chunk::loadHeader(Allocator.Cookie, Ptr, &Header);
       if (UNLIKELY(Header.State != Chunk::State::Quarantined))
         reportInvalidChunkState(AllocatorAction::Recycling, Ptr);
 
       Chunk::UnpackedHeader NewHeader = Header;
       NewHeader.State = Chunk::State::Available;
       Chunk::compareExchangeHeader(Allocator.Cookie, Ptr, &NewHeader, &Header);
 
+      if (allocatorSupportsMemoryTagging<Params>())
+        Ptr = untagPointer(Ptr);
       void *BlockBegin = Allocator::getBlockBegin(Ptr, &NewHeader);
-      const uptr ClassId = NewHeader.ClassId;
-      if (LIKELY(ClassId))
-        Cache.deallocate(ClassId, BlockBegin);
-      else
-        Allocator.Secondary.deallocate(BlockBegin);
+      Cache.deallocate(NewHeader.ClassId, BlockBegin);
     }
 
     // We take a shortcut when allocating a quarantine batch by working with the
     // appropriate class ID instead of using Size. The compiler should optimize
     // the class ID computation and work with the associated cache directly.
     void *allocate(UNUSED uptr Size) {
       const uptr QuarantineClassId = SizeClassMap::getClassIdBySize(
           sizeof(QuarantineBatch) + Chunk::getHeaderSize());
#endif // GWP_ASAN_HOOKS
   // - unlinking the local stats from the global ones (destroying the cache does
   //   the last two items).
   void commitBack(TSD<ThisT> *TSD) {
     Quarantine.drain(&TSD->QuarantineCache,
                      QuarantineCallback(*this, TSD->Cache));
     TSD->Cache.destroy(&Stats);
   }
 
-  ALWAYS_INLINE void *untagPointerMaybe(void *Ptr) {
-    if (allocatorSupportsMemoryTagging<Params>())
-      return reinterpret_cast<void *>(
-          untagPointer(reinterpret_cast<uptr>(Ptr)));
+  ALWAYS_INLINE void *getHeaderTaggedPointer(void *Ptr) {
+    if (!allocatorSupportsMemoryTagging<Params>())
+      return Ptr;
+    auto UntaggedPtr = untagPointer(Ptr);
+    if (UntaggedPtr != Ptr)
+      return UntaggedPtr;
+    // Secondary, or pointer allocated while memory tagging is unsupported or
+    // disabled. The tag mismatch is okay in the latter case because tags will
+    // not be checked.
+    return addHeaderTag(Ptr);
+  }
+
+  ALWAYS_INLINE uptr addHeaderTag(uptr Ptr) {
+    if (!allocatorSupportsMemoryTagging<Params>())
       return Ptr;
+    return addFixedTag(Ptr, 2);
+  }
+
+  ALWAYS_INLINE void *addHeaderTag(void *Ptr) {
+    return reinterpret_cast<void *>(addHeaderTag(reinterpret_cast<uptr>(Ptr)));
   }
 
   NOINLINE u32 collectStackTrace() {
 #ifdef HAVE_ANDROID_UNSAFE_FRAME_POINTER_CHASE
     // Discard collectStackTrace() frame and allocator function frame.
     constexpr uptr DiscardFrames = 2;
     uptr Stack[MaxTraceSize + DiscardFrames];
     uptr Size =
if (LIKELY(PrimaryT::canAllocate(NeededSize))) {
           Block = TSD->Cache.allocate(++ClassId);
         if (!Block)
           ClassId = 0;
       }
       if (UnlockRequired)
         TSD->unlock();
     }
     if (UNLIKELY(ClassId == 0))
-      Block = Secondary.allocate(NeededSize, Alignment, &SecondaryBlockEnd,
+      Block = Secondary.allocate(Options, Size, Alignment, &SecondaryBlockEnd,
                                  FillContents);
 
     if (UNLIKELY(!Block)) {
       if (Options.get(OptionBit::MayReturnNull))
         return nullptr;
       reportOutOfMemory(NeededSize);
     }
 
if (LIKELY(ClassId)) {
             memset(TaggedPtr, 0, archMemoryTagGranuleSize());
           }
         } else {
           const uptr OddEvenMask =
               computeOddEvenMaskForPointerMaybe(Options, BlockUptr, BlockSize);
           TaggedPtr = prepareTaggedChunk(Ptr, Size, OddEvenMask, BlockEnd);
         }
         storeAllocationStackMaybe(Options, Ptr);
-      } else if (UNLIKELY(FillContents != NoFill)) {
+      } else {
+        Block = addHeaderTag(Block);
+        Ptr = addHeaderTag(Ptr);
+        if (UNLIKELY(FillContents != NoFill)) {
           // This condition is not necessarily unlikely, but since memset is
           // costly, we might as well mark it as such.
           memset(Block, FillContents == ZeroFill ? 0 : PatternFillByte,
                  PrimaryT::getSizeByClassId(ClassId));
         }
       }
+    } else {
+      Block = addHeaderTag(Block);
+      Ptr = addHeaderTag(Ptr);
+      if (UNLIKELY(useMemoryTagging<Params>(Options)))
+        storeTags(reinterpret_cast<uptr>(Block), reinterpret_cast<uptr>(Ptr));
+    }
 
     Chunk::UnpackedHeader Header = {};
     if (UNLIKELY(UnalignedUserPtr != UserPtr)) {
       const uptr Offset = UserPtr - UnalignedUserPtr;
       DCHECK_GE(Offset, 2 * sizeof(u32));
       // The BlockMarker has no security purpose, but is specifically meant for
       // the chunk iteration function that can be used in debugging situations.
       // It is the only situation where we have to locate the start of a chunk
#endif // GWP_ASAN_HOOKS
     if (UNLIKELY(&__scudo_deallocate_hook))
       __scudo_deallocate_hook(Ptr);
 
     if (UNLIKELY(!Ptr))
       return;
     if (UNLIKELY(!isAligned(reinterpret_cast<uptr>(Ptr), MinAlignment)))
       reportMisalignedPointer(AllocatorAction::Deallocating, Ptr);
 
-    Ptr = untagPointerMaybe(Ptr);
+    Ptr = getHeaderTaggedPointer(Ptr);
 
     Chunk::UnpackedHeader Header;
     Chunk::loadHeader(Cookie, Ptr, &Header);
 
     if (UNLIKELY(Header.State != Chunk::State::Allocated))
       reportInvalidChunkState(AllocatorAction::Deallocating, Ptr);
 
     const Options Options = Primary.Options.load();
void *reallocate(void *OldPtr, uptr NewSize, uptr Alignment = MinAlignment) {
     const Options Options = Primary.Options.load();
     if (UNLIKELY(NewSize >= MaxAllowedMallocSize)) {
       if (Options.get(OptionBit::MayReturnNull))
         return nullptr;
       reportAllocationSizeTooBig(NewSize, 0, MaxAllowedMallocSize);
     }
 
     void *OldTaggedPtr = OldPtr;
-    OldPtr = untagPointerMaybe(OldPtr);
+    OldPtr = getHeaderTaggedPointer(OldPtr);
 
     // The following cases are handled by the C wrappers.
     DCHECK_NE(OldPtr, nullptr);
     DCHECK_NE(NewSize, 0);
 
 #ifdef GWP_ASAN_HOOKS
     if (UNLIKELY(GuardedAlloc.pointerIsMine(OldPtr))) {
       uptr OldSize = GuardedAlloc.getSize(OldPtr);
#endif // GWP_ASAN_HOOKS
     // will trigger this check. It really isn't.
     if (Options.get(OptionBit::DeallocTypeMismatch)) {
       if (UNLIKELY(OldHeader.OriginOrWasZeroed != Chunk::Origin::Malloc))
         reportDeallocTypeMismatch(AllocatorAction::Reallocating, OldPtr,
                                   OldHeader.OriginOrWasZeroed,
                                   Chunk::Origin::Malloc);
     }
 
-    void *BlockBegin = getBlockBegin(OldPtr, &OldHeader);
+    void *BlockBegin = getBlockBegin(OldTaggedPtr, &OldHeader);
     uptr BlockEnd;
     uptr OldSize;
     const uptr ClassId = OldHeader.ClassId;
     if (LIKELY(ClassId)) {
       BlockEnd = reinterpret_cast<uptr>(BlockBegin) +
                  SizeClassMap::getSizeByClassId(ClassId);
       OldSize = OldHeader.SizeOrUnusedBytes;
     } else {
       BlockEnd = SecondaryT::getBlockEnd(BlockBegin);
-      OldSize = BlockEnd -
-                (reinterpret_cast<uptr>(OldPtr) + OldHeader.SizeOrUnusedBytes);
+      OldSize = BlockEnd - (reinterpret_cast<uptr>(OldTaggedPtr) +
+                            OldHeader.SizeOrUnusedBytes);
     }
     // If the new chunk still fits in the previously allocated block (with a
     // reasonable delta), we just keep the old block, and update the chunk
     // header to reflect the size change.
-    if (reinterpret_cast<uptr>(OldPtr) + NewSize <= BlockEnd) {
+    if (reinterpret_cast<uptr>(OldTaggedPtr) + NewSize <= BlockEnd) {
       if (NewSize > OldSize || (OldSize - NewSize) < getPageSizeCached()) {
         Chunk::UnpackedHeader NewHeader = OldHeader;
         NewHeader.SizeOrUnusedBytes =
             (ClassId ? NewSize
-                     : BlockEnd - (reinterpret_cast<uptr>(OldPtr) + NewSize)) &
+                     : BlockEnd -
+                           (reinterpret_cast<uptr>(OldTaggedPtr) + NewSize)) &
             Chunk::SizeOrUnusedBytesMask;
         Chunk::compareExchangeHeader(Cookie, OldPtr, &NewHeader, &OldHeader);
         if (UNLIKELY(ClassId && useMemoryTagging<Params>(Options))) {
           resizeTaggedChunk(reinterpret_cast<uptr>(OldTaggedPtr) + OldSize,
                             reinterpret_cast<uptr>(OldTaggedPtr) + NewSize,
                             BlockEnd);
           storeAllocationStackMaybe(Options, OldPtr);
         }
#endif
   // Iterate over all chunks and call a callback for all busy chunks located
   // within the provided memory range. Said callback must not use this allocator
   // or a deadlock can ensue. This fits Android's malloc_iterate() needs.
   void iterateOverChunks(uptr Base, uptr Size, iterate_callback Callback,
                          void *Arg) {
     initThreadMaybe();
     const uptr From = Base;
     const uptr To = Base + Size;
-    auto Lambda = [this, From, To, Callback, Arg](uptr Block) {
+    bool MayHaveTaggedPrimary = allocatorSupportsMemoryTagging<Params>() &&
+                                systemSupportsMemoryTagging();
+    auto Lambda = [this, From, To, MayHaveTaggedPrimary, Callback,
+                   Arg](uptr Block) {
       if (Block < From || Block >= To)
         return;
       uptr Chunk;
       Chunk::UnpackedHeader Header;
-      if (getChunkFromBlock(Block, &Chunk, &Header) &&
-          Header.State == Chunk::State::Allocated) {
+      if (MayHaveTaggedPrimary) {
+        // A chunk header can either have a zero tag (tagged primary) or the
+        // header tag (secondary, or untagged primary). We don't know which so
+        // try both.
+        ScopedDisableMemoryTagChecks x;
+        if (!getChunkFromBlock(Block, &Chunk, &Header) &&
+            !getChunkFromBlock(addHeaderTag(Block), &Chunk, &Header))
+          return;
+      } else {
+        if (!getChunkFromBlock(addHeaderTag(Block), &Chunk, &Header))
+          return;
+      }
+      if (Header.State == Chunk::State::Allocated) {
         uptr TaggedChunk = Chunk;
+        if (allocatorSupportsMemoryTagging<Params>())
+          TaggedChunk = untagPointer(TaggedChunk);
         if (useMemoryTagging<Params>(Primary.Options.load()))
           TaggedChunk = loadTag(Chunk);
         Callback(TaggedChunk, getSize(reinterpret_cast<void *>(Chunk), &Header),
                  Arg);
       }
     };
     Primary.iterateOverBlocks(Lambda);
     Secondary.iterateOverBlocks(Lambda);
uptr getUsableSize(const void *Ptr) {
     if (UNLIKELY(!Ptr))
       return 0;
 
 #ifdef GWP_ASAN_HOOKS
     if (UNLIKELY(GuardedAlloc.pointerIsMine(Ptr)))
       return GuardedAlloc.getSize(Ptr);
 #endif // GWP_ASAN_HOOKS
 
-    Ptr = untagPointerMaybe(const_cast<void *>(Ptr));
+    Ptr = getHeaderTaggedPointer(const_cast<void *>(Ptr));
     Chunk::UnpackedHeader Header;
     Chunk::loadHeader(Cookie, Ptr, &Header);
     // Getting the usable size of a chunk only makes sense if it's allocated.
     if (UNLIKELY(Header.State != Chunk::State::Allocated))
       reportInvalidChunkState(AllocatorAction::Sizing, const_cast<void *>(Ptr));
     return getSize(Ptr, &Header);
   }
 
   void getStats(StatCounters S) {
     initThreadMaybe();
     Stats.get(S);
   }
 
   // Returns true if the pointer provided was allocated by the current
   // allocator instance, which is compliant with tcmalloc's ownership concept.
   // A corrupted chunk will not be reported as owned, which is WAI.
   bool isOwned(const void *Ptr) {
     initThreadMaybe();
 #ifdef GWP_ASAN_HOOKS
     if (GuardedAlloc.pointerIsMine(Ptr))
       return true;
 #endif // GWP_ASAN_HOOKS
     if (!Ptr || !isAligned(reinterpret_cast<uptr>(Ptr), MinAlignment))
       return false;
-    Ptr = untagPointerMaybe(const_cast<void *>(Ptr));
+    Ptr = getHeaderTaggedPointer(const_cast<void *>(Ptr));
     Chunk::UnpackedHeader Header;
     return Chunk::isValid(Cookie, Ptr, &Header) &&
            Header.State == Chunk::State::Allocated;
   }
 
   bool useMemoryTaggingTestOnly() const {
     return useMemoryTagging<Params>(Primary.Options.load());
   }
   void disableMemoryTagging() {
-    if (allocatorSupportsMemoryTagging<Params>())
+    // If we haven't been initialized yet, we need to initialize now in order to
+    // prevent a future call to initThreadMaybe() from enabling memory tagging
+    // based on feature detection. But don't call initThreadMaybe() because it
+    // may end up calling the allocator (via pthread_atfork, via the post-init
+    // callback), which may cause mappings to be created with memory tagging
+    // enabled.
+    TSDRegistry.initOnceMaybe(this);
+    if (allocatorSupportsMemoryTagging<Params>()) {
+      Secondary.disableMemoryTagging();
       Primary.Options.clear(OptionBit::UseMemoryTagging);
     }
+  }
 
   void setTrackAllocationStacks(bool Track) {
     initThreadMaybe();
     if (Track)
       Primary.Options.set(OptionBit::TrackAllocationStacks);
     else
       Primary.Options.clear(OptionBit::TrackAllocationStacks);
   }
 
   void setFillContents(FillContentsMode FillContents) {
     initThreadMaybe();
     Primary.Options.setFillContentsMode(FillContents);
   }
 
+  void setAddLargeAllocationSlack(bool AddSlack) {
+    initThreadMaybe();
+    if (AddSlack)
+      Primary.Options.set(OptionBit::AddLargeAllocationSlack);
+    else
+      Primary.Options.clear(OptionBit::AddLargeAllocationSlack);
+  }
+
   const char *getStackDepotAddress() const {
     return reinterpret_cast<const char *>(&Depot);
   }
 
   const char *getRegionInfoArrayAddress() const {
     return Primary.getRegionInfoArrayAddress();
   }
 
#endif // GWP_ASAN_HOOKS
   static inline void *getBlockBegin(const void *Ptr,
                                     Chunk::UnpackedHeader *Header) {
     return reinterpret_cast<void *>(
         reinterpret_cast<uptr>(Ptr) - Chunk::getHeaderSize() -
         (static_cast<uptr>(Header->Offset) << MinAlignmentLog));
   }
 
   // Return the size of a chunk as requested during its allocation.
   inline uptr getSize(const void *Ptr, Chunk::UnpackedHeader *Header) {
     const uptr SizeOrUnusedBytes = Header->SizeOrUnusedBytes;
     if (LIKELY(Header->ClassId))
       return SizeOrUnusedBytes;
+    if (allocatorSupportsMemoryTagging<Params>())
+      Ptr = untagPointer(const_cast<void *>(Ptr));
     return SecondaryT::getBlockEnd(getBlockBegin(Ptr, Header)) -
            reinterpret_cast<uptr>(Ptr) - SizeOrUnusedBytes;
   }
 
   void quarantineOrDeallocateChunk(Options Options, void *Ptr,
                                    Chunk::UnpackedHeader *Header, uptr Size) {
     Chunk::UnpackedHeader NewHeader = *Header;
     if (UNLIKELY(NewHeader.ClassId && useMemoryTagging<Params>(Options))) {
if (UNLIKELY(NewHeader.ClassId && useMemoryTagging<Params>(Options))) {
                      &TaggedEnd);
       }
       NewHeader.OriginOrWasZeroed = !TSDRegistry.getDisableMemInit();
       storeDeallocationStackMaybe(Options, Ptr, PrevTag);
     }
     // If the quarantine is disabled, the actual size of a chunk is 0 or larger
     // than the maximum allowed, we return a chunk directly to the backend.
     // This purposefully underflows for Size == 0.
-    const bool BypassQuarantine =
-        !Quarantine.getCacheSize() || ((Size - 1) >= QuarantineMaxChunkSize);
+    const bool BypassQuarantine = !Quarantine.getCacheSize() ||
+                                  ((Size - 1) >= QuarantineMaxChunkSize) ||
+                                  !NewHeader.ClassId;
     if (BypassQuarantine) {
       NewHeader.State = Chunk::State::Available;
       Chunk::compareExchangeHeader(Cookie, Ptr, &NewHeader, Header);
+      if (allocatorSupportsMemoryTagging<Params>())
+        Ptr = untagPointer(Ptr);
       void *BlockBegin = getBlockBegin(Ptr, &NewHeader);
       const uptr ClassId = NewHeader.ClassId;
       if (LIKELY(ClassId)) {
         bool UnlockRequired;
         auto *TSD = TSDRegistry.getTSDAndLock(&UnlockRequired);
         TSD->Cache.deallocate(ClassId, BlockBegin);
         if (UnlockRequired)
           TSD->unlock();
       } else {
-        Secondary.deallocate(BlockBegin);
+        if (UNLIKELY(useMemoryTagging<Params>(Options)))
+          storeTags(reinterpret_cast<uptr>(BlockBegin),
+                    reinterpret_cast<uptr>(Ptr));
+        Secondary.deallocate(Options, BlockBegin);
       }
     } else {
       NewHeader.State = Chunk::State::Quarantined;
       Chunk::compareExchangeHeader(Cookie, Ptr, &NewHeader, Header);
       bool UnlockRequired;
       auto *TSD = TSDRegistry.getTSDAndLock(&UnlockRequired);
       Quarantine.put(&TSD->QuarantineCache,
                      QuarantineCallback(*this, TSD->Cache), Ptr, Size);

compiler-rt/lib/scudo/standalone/common.h

 
 // Indicates that we are getting rid of the whole mapping, which might have
 // further consequences on Data, depending on the platform.
 #define UNMAP_ALL (1U << 0)
 
 void unmap(void *Addr, uptr Size, uptr Flags = 0,
            MapPlatformData *Data = nullptr);
 
+void setMemoryPermission(uptr Addr, uptr Size, uptr Flags,
+                         MapPlatformData *Data = nullptr);
+
 void releasePagesToOS(uptr BaseAddress, uptr Offset, uptr Size,
                       MapPlatformData *Data = nullptr);
 
 // Internal map & unmap fatal error. This must not call map().
 void NORETURN dieOnMapUnmapError(bool OutOfMemory = false);
 
 // Logging related functions.
 

compiler-rt/lib/scudo/standalone/fuchsia.cpp

void unmap(void *Addr, uptr Size, uptr Flags, MapPlatformData *Data) {
   }
   if (Data) {
     if (Data->Vmo != ZX_HANDLE_INVALID)
       CHECK_EQ(_zx_handle_close(Data->Vmo), ZX_OK);
     memset(Data, 0, sizeof(*Data));
   }
 }
 
+void setMemoryPermission(UNUSED uptr Addr, UNUSED uptr Size, UNUSED uptr Flags,
+                         UNUSED MapPlatformData *Data) {
+  const zx_vm_option_t Prot =
+      (Flags & MAP_NOACCESS) ? 0 : (ZX_VM_PERM_READ | ZX_VM_PERM_WRITE);
+  DCHECK(Data);
+  DCHECK_NE(Data->Vmar, ZX_HANDLE_INVALID);
+  if (_zx_vmar_protect(Data->Vmar, Prot, Addr, Size) != ZX_OK)
+    dieOnMapUnmapError();
+}
+
 void releasePagesToOS(UNUSED uptr BaseAddress, uptr Offset, uptr Size,
                       MapPlatformData *Data) {
   DCHECK(Data);
   DCHECK_NE(Data->Vmar, ZX_HANDLE_INVALID);
   DCHECK_NE(Data->Vmo, ZX_HANDLE_INVALID);
   const zx_status_t Status =
       _zx_vmo_op_range(Data->Vmo, ZX_VMO_OP_DECOMMIT, Offset, Size, NULL, 0);
   CHECK_EQ(Status, ZX_OK);

compiler-rt/lib/scudo/standalone/linux.cpp

void *map(void *Addr, uptr Size, UNUSED const char *Name, uptr Flags,
           UNUSED MapPlatformData *Data) {
   int MmapFlags = MAP_PRIVATE | MAP_ANONYMOUS;
   int MmapProt;
   if (Flags & MAP_NOACCESS) {
     MmapFlags |= MAP_NORESERVE;
     MmapProt = PROT_NONE;
   } else {
     MmapProt = PROT_READ | PROT_WRITE;
+  }
 #if defined(__aarch64__)
 #ifndef PROT_MTE
 #define PROT_MTE 0x20
 #endif
-    if (Flags & MAP_MEMTAG)
-      MmapProt |= PROT_MTE;
 #endif
-  }
   if (Addr) {
     // Currently no scenario for a noaccess mapping with a fixed address.
     DCHECK_EQ(Flags & MAP_NOACCESS, 0);
     MmapFlags |= MAP_FIXED;
   }
   void *P = mmap(Addr, Size, MmapProt, MmapFlags, -1, 0);
   if (P == MAP_FAILED) {
     if (!(Flags & MAP_ALLOWNOMEM) || errno != ENOMEM)
       dieOnMapUnmapError(errno == ENOMEM);
     return nullptr;
   }
 #if SCUDO_ANDROID
-  if (!(Flags & MAP_NOACCESS))
+  if (Name)
     prctl(ANDROID_PR_SET_VMA, ANDROID_PR_SET_VMA_ANON_NAME, P, Size, Name);
 #endif
   return P;
 }
 
 void unmap(void *Addr, uptr Size, UNUSED uptr Flags,
            UNUSED MapPlatformData *Data) {
   if (munmap(Addr, Size) != 0)
     dieOnMapUnmapError();
 }
 
+void setMemoryPermission(uptr Addr, uptr Size, uptr Flags,
+                         UNUSED MapPlatformData *Data) {
+  int Prot = (Flags & MAP_NOACCESS) ? PROT_NONE : (PROT_READ | PROT_WRITE);
+  if (mprotect(reinterpret_cast<void *>(Addr), Size, Prot) != 0)
+    dieOnMapUnmapError();
+}
+
 void releasePagesToOS(uptr BaseAddress, uptr Offset, uptr Size,
                       UNUSED MapPlatformData *Data) {
   void *Addr = reinterpret_cast<void *>(BaseAddress + Offset);
   while (madvise(Addr, Size, MADV_DONTNEED) == -1 && errno == EAGAIN) {
   }
 }
 
 // Calling getenv should be fine (c)(tm) at any time.

compiler-rt/lib/scudo/standalone/memtag.h

 
 namespace scudo {
 
 void setRandomTag(void *Ptr, uptr Size, uptr ExcludeMask, uptr *TaggedBegin,
                   uptr *TaggedEnd);
 
 #if defined(__aarch64__) || defined(SCUDO_FUZZ)
 
+// We assume that Top-Byte Ignore is enabled if the architecture supports memory
+// tagging. Not all operating systems enable TBI, so we only claim architectural
+// support for memory tagging if the operating system enables TBI.
+#if SCUDO_LINUX
 inline constexpr bool archSupportsMemoryTagging() { return true; }
+#else
+inline constexpr bool archSupportsMemoryTagging() { return false; }
+#endif
+
 inline constexpr uptr archMemoryTagGranuleSize() { return 16; }
 
 inline uptr untagPointer(uptr Ptr) { return Ptr & ((1ULL << 56) - 1); }
 
 inline uint8_t extractTag(uptr Ptr) { return (Ptr >> 56) & 0xf; }
 
 #else
 
inline uptr selectRandomTag(uptr Ptr, uptr ExcludeMask) {
   uptr TaggedPtr;
   __asm__ __volatile__(
       ".arch_extension memtag; irg %[TaggedPtr], %[Ptr], %[ExcludeMask]"
       : [TaggedPtr] "=r"(TaggedPtr)
       : [Ptr] "r"(Ptr), [ExcludeMask] "r"(ExcludeMask));
   return TaggedPtr;
 }
 
+inline uptr addFixedTag(uptr Ptr, uptr Tag) { return Ptr | (Tag << 56); }
+
 inline uptr storeTags(uptr Begin, uptr End) {
   DCHECK(Begin % 16 == 0);
   if (Begin != End) {
     __asm__ __volatile__(
         R"(
       .arch_extension memtag
 
     1:

 };
 
 inline uptr selectRandomTag(uptr Ptr, uptr ExcludeMask) {
   (void)Ptr;
   (void)ExcludeMask;
   UNREACHABLE("memory tagging not supported");
 }
 
+inline uptr addFixedTag(uptr Ptr, uptr Tag) {
+  (void)Ptr;
+  (void)Tag;
+  UNREACHABLE("memory tagging not supported");
+}
+
 inline uptr storeTags(uptr Begin, uptr End) {
   (void)Begin;
   (void)End;
   UNREACHABLE("memory tagging not supported");
 }
 
 inline void *prepareTaggedChunk(void *Ptr, uptr Size, uptr ExcludeMask,
                                 uptr BlockEnd) {

 #endif
 
 inline void setRandomTag(void *Ptr, uptr Size, uptr ExcludeMask,
                          uptr *TaggedBegin, uptr *TaggedEnd) {
   *TaggedBegin = selectRandomTag(reinterpret_cast<uptr>(Ptr), ExcludeMask);
   *TaggedEnd = storeTags(*TaggedBegin, *TaggedBegin + Size);
 }
 
+inline void *untagPointer(void *Ptr) {
+  return reinterpret_cast<void *>(untagPointer(reinterpret_cast<uptr>(Ptr)));
+}
+
 template <typename Config>
 inline constexpr bool allocatorSupportsMemoryTagging() {
   return archSupportsMemoryTagging() && Config::MaySupportMemoryTagging;
 }
 
 } // namespace scudo
 
 #endif

compiler-rt/lib/scudo/standalone/options.h

enum class OptionBit {
   MayReturnNull,
   FillContents0of2,
   FillContents1of2,
   DeallocTypeMismatch,
   DeleteSizeMismatch,
   TrackAllocationStacks,
   UseOddEvenTags,
   UseMemoryTagging,
+  AddLargeAllocationSlack,
 };
 
 struct Options {
   u32 Val;
 
   bool get(OptionBit Opt) const { return Val & (1U << static_cast<u32>(Opt)); }
 
   FillContentsMode getFillContentsMode() const {

compiler-rt/lib/scudo/standalone/primary64.h

return (ClassId == SizeClassMap::BatchClassId)
                : SizeClassMap::getSizeByClassId(ClassId);
   }
 
   static bool canAllocate(uptr Size) { return Size <= SizeClassMap::MaxSize; }
 
   void initLinkerInitialized(s32 ReleaseToOsInterval) {
     // Reserve the space required for the Primary.
     PrimaryBase = reinterpret_cast<uptr>(
-        map(nullptr, PrimarySize, "scudo:primary", MAP_NOACCESS, &Data));
+        map(nullptr, PrimarySize, nullptr, MAP_NOACCESS, &Data));
 
     u32 Seed;
     const u64 Time = getMonotonicTime();
     if (!getRandom(reinterpret_cast<void *>(&Seed), sizeof(Seed)))
       Seed = static_cast<u32>(Time ^ (PrimaryBase >> 12));
     const uptr PageSize = getPageSizeCached();
     for (uptr I = 0; I < NumClasses; I++) {
       RegionInfo *Region = getRegionInfo(I);

compiler-rt/lib/scudo/standalone/secondary.h

 // their own header that is not checksummed: the guard pages and the Combined
 // header should be enough for our purpose.
 
 namespace LargeBlock {
 
 struct Header {
   LargeBlock::Header *Prev;
   LargeBlock::Header *Next;
-  uptr BlockEnd;
+  uptr CommitBase;
+  uptr CommitSize;
   uptr MapBase;
   uptr MapSize;
   [[no_unique_address]] MapPlatformData Data;
 };
 
 constexpr uptr getHeaderSize() {
   return roundUpTo(sizeof(Header), 1U << SCUDO_MIN_ALIGNMENT_LOG);
 }
 
-static Header *getHeader(uptr Ptr) {
-  return reinterpret_cast<Header *>(Ptr - getHeaderSize());
+template <typename Config> static uptr addHeaderTag(uptr Ptr) {
+  if (allocatorSupportsMemoryTagging<Config>())
+    return addFixedTag(Ptr, 1);
+  return Ptr;
+}
+
+template <typename Config> static Header *getHeader(uptr Ptr) {
+  return reinterpret_cast<Header *>(addHeaderTag<Config>(Ptr) -
+                                    getHeaderSize());
 }
 
-static Header *getHeader(const void *Ptr) {
-  return getHeader(reinterpret_cast<uptr>(Ptr));
+template <typename Config> static Header *getHeader(const void *Ptr) {
+  return getHeader<Config>(reinterpret_cast<uptr>(Ptr));
 }
 
 } // namespace LargeBlock
 
+static void unmap(LargeBlock::Header *H) {
+  MapPlatformData Data = H->Data;
+  unmap(reinterpret_cast<void *>(H->MapBase), H->MapSize, UNMAP_ALL, &Data);
+}
+
 class MapAllocatorNoCache {
 public:
   void initLinkerInitialized(UNUSED s32 ReleaseToOsInterval) {}
   void init(UNUSED s32 ReleaseToOsInterval) {}
-  bool retrieve(UNUSED uptr Size, UNUSED LargeBlock::Header **H,
-                UNUSED bool *Zeroed) {
+  bool retrieve(UNUSED Options Options, UNUSED uptr Size, UNUSED uptr Alignment,
+                UNUSED LargeBlock::Header **H, UNUSED bool *Zeroed) {
     return false;
   }
-  bool store(UNUSED LargeBlock::Header *H) { return false; }
+  void store(UNUSED Options Options, UNUSED LargeBlock::Header *H) { unmap(H); }
   bool canCache(UNUSED uptr Size) { return false; }
   void disable() {}
   void enable() {}
   void releaseToOS() {}
+  void disableMemoryTagging() {}
   bool setOption(Option O, UNUSED sptr Value) {
     if (O == Option::ReleaseInterval || O == Option::MaxCacheEntriesCount ||
         O == Option::MaxCacheEntrySize)
       return false;
     // Not supported by the Secondary Cache, but not an error either.
     return true;
   }
 };
 
+static const uptr MaxUnusedCachePages = 4U;
+
 template <typename Config> class MapAllocatorCache {
 public:
   // Ensure the default maximum specified fits the array.
   static_assert(Config::SecondaryCacheDefaultMaxEntriesCount <=
                     Config::SecondaryCacheEntriesArraySize,
                 "");
 
   void initLinkerInitialized(s32 ReleaseToOsInterval) {
     setOption(Option::MaxCacheEntriesCount,
               static_cast<sptr>(Config::SecondaryCacheDefaultMaxEntriesCount));
     setOption(Option::MaxCacheEntrySize,
               static_cast<sptr>(Config::SecondaryCacheDefaultMaxEntrySize));
     setOption(Option::ReleaseInterval, static_cast<sptr>(ReleaseToOsInterval));
   }
   void init(s32 ReleaseToOsInterval) {
     memset(this, 0, sizeof(*this));
     initLinkerInitialized(ReleaseToOsInterval);
   }
 
-  bool store(LargeBlock::Header *H) {
+  void store(Options Options, LargeBlock::Header *H) {
+    if (!canCache(H->CommitSize))
+      return unmap(H);
+
     bool EntryCached = false;
     bool EmptyCache = false;
+    const s32 Interval = atomic_load_relaxed(&ReleaseToOsIntervalMs);
     const u64 Time = getMonotonicTime();
     const u32 MaxCount = atomic_load_relaxed(&MaxEntriesCount);
-    {
+    CachedBlock Entry;
+    Entry.CommitBase = H->CommitBase;
+    Entry.CommitSize = H->CommitSize;
+    Entry.MapBase = H->MapBase;
+    Entry.MapSize = H->MapSize;
+    Entry.BlockBegin = reinterpret_cast<uptr>(H + 1);
+    Entry.Data = H->Data;
+    Entry.Time = Time;
+    if (useMemoryTagging<Config>(Options)) {
+      if (Interval == 0 && !SCUDO_FUCHSIA) {
+        // Release the memory and make it inaccessible at the same time by
+        // creating a new MAP_NOACCESS mapping on top of the existing mapping.
+        // Fuchsia does not support replacing mappings by creating a new mapping
+        // on top so we just do the two syscalls there.
+        Entry.Time = 0;
+        map(reinterpret_cast<void *>(Entry.CommitBase), Entry.CommitSize,
+            "scudo:secondary", MAP_RESIZABLE | MAP_NOACCESS | MAP_MEMTAG,
+            &Entry.Data);
+      } else {
+        setMemoryPermission(Entry.CommitBase, Entry.CommitSize, MAP_NOACCESS,
+                            &Entry.Data);
+      }
+    } else if (Interval == 0) {
+      releasePagesToOS(Entry.CommitBase, 0, Entry.CommitSize, &Entry.Data);
+      Entry.Time = 0;
+    }
+    do {
       ScopedLock L(Mutex);
+      if (useMemoryTagging<Config>(Options) && QuarantinePos == -1U) {
+        // If we get here then memory tagging was disabled in between when we
+        // read Options and when we locked Mutex. We can't insert our entry into
+        // the quarantine or the cache because the permissions would be wrong so
+        // just unmap it.
+        break;
+      }
+      if (Config::SecondaryCacheQuarantineSize &&
+          useMemoryTagging<Config>(Options)) {
+        QuarantinePos =
+            (QuarantinePos + 1) % Config::SecondaryCacheQuarantineSize;
+        if (!Quarantine[QuarantinePos].CommitBase) {
+          Quarantine[QuarantinePos] = Entry;
+          return;
+        }
+        CachedBlock PrevEntry = Quarantine[QuarantinePos];
+        Quarantine[QuarantinePos] = Entry;
+        if (OldestTime == 0)
+          OldestTime = Entry.Time;
+        Entry = PrevEntry;
+      }
       if (EntriesCount >= MaxCount) {
         if (IsFullEvents++ == 4U)
           EmptyCache = true;
       } else {
         for (u32 I = 0; I < MaxCount; I++) {
-          if (Entries[I].Block)
+          if (Entries[I].CommitBase)
             continue;
           if (I != 0)
             Entries[I] = Entries[0];
-          Entries[0].Block = reinterpret_cast<uptr>(H);
-          Entries[0].BlockEnd = H->BlockEnd;
-          Entries[0].MapBase = H->MapBase;
-          Entries[0].MapSize = H->MapSize;
-          Entries[0].Data = H->Data;
-          Entries[0].Time = Time;
+          Entries[0] = Entry;
           EntriesCount++;
+          if (OldestTime == 0)
+            OldestTime = Entry.Time;
           EntryCached = true;
           break;
         }
       }
-    }
-    s32 Interval;
+    } while (0);
     if (EmptyCache)
       empty();
-    else if ((Interval = atomic_load_relaxed(&ReleaseToOsIntervalMs)) >= 0)
+    else if (Interval >= 0)
       releaseOlderThan(Time - static_cast<u64>(Interval) * 1000000);
-    return EntryCached;
+    if (!EntryCached)
+      unmap(reinterpret_cast<void *>(Entry.MapBase), Entry.MapSize, UNMAP_ALL,
+            &Entry.Data);
   }
 
-  bool retrieve(uptr Size, LargeBlock::Header **H, bool *Zeroed) {
+  bool retrieve(Options Options, uptr Size, uptr Alignment,
+                LargeBlock::Header **H, bool *Zeroed) {
     const uptr PageSize = getPageSizeCached();
     const u32 MaxCount = atomic_load_relaxed(&MaxEntriesCount);
+    bool Found = false;
+    CachedBlock Entry;
+    uptr HeaderPos;
+    {
       ScopedLock L(Mutex);
       if (EntriesCount == 0)
         return false;
       for (u32 I = 0; I < MaxCount; I++) {
-      if (!Entries[I].Block)
+        const uptr CommitBase = Entries[I].CommitBase;
+        if (!CommitBase)
           continue;
-      const uptr BlockSize = Entries[I].BlockEnd - Entries[I].Block;
-      if (Size > BlockSize)
+        const uptr CommitSize = Entries[I].CommitSize;
+        const uptr AllocPos =
+            roundDownTo(CommitBase + CommitSize - Size, Alignment);
+        HeaderPos =
+            AllocPos - Chunk::getHeaderSize() - LargeBlock::getHeaderSize();
+        if (HeaderPos > CommitBase + CommitSize)
           continue;
-      if (Size < BlockSize - PageSize * 4U)
+        if (HeaderPos < CommitBase ||
+            AllocPos > CommitBase + PageSize * MaxUnusedCachePages)
           continue;
-      *H = reinterpret_cast<LargeBlock::Header *>(Entries[I].Block);
-      *Zeroed = Entries[I].Time == 0;
-      Entries[I].Block = 0;
-      (*H)->BlockEnd = Entries[I].BlockEnd;
-      (*H)->MapBase = Entries[I].MapBase;
-      (*H)->MapSize = Entries[I].MapSize;
-      (*H)->Data = Entries[I].Data;
+        Found = true;
+        Entry = Entries[I];
+        Entries[I].CommitBase = 0;
+        break;
+      }
+    }
+    if (Found) {
+      *H = reinterpret_cast<LargeBlock::Header *>(
+          LargeBlock::addHeaderTag<Config>(HeaderPos));
+      *Zeroed = Entry.Time == 0;
+      if (useMemoryTagging<Config>(Options))
+        setMemoryPermission(Entry.CommitBase, Entry.CommitSize, 0, &Entry.Data);
+      uptr NewBlockBegin = reinterpret_cast<uptr>(*H + 1);
+      if (useMemoryTagging<Config>(Options)) {
+        if (*Zeroed)
+          storeTags(LargeBlock::addHeaderTag<Config>(Entry.CommitBase),
+                    NewBlockBegin);
+        else if (Entry.BlockBegin < NewBlockBegin)
+          storeTags(Entry.BlockBegin, NewBlockBegin);
+        else
+          storeTags(untagPointer(NewBlockBegin),
+                    untagPointer(Entry.BlockBegin));
+      }
+      (*H)->CommitBase = Entry.CommitBase;
+      (*H)->CommitSize = Entry.CommitSize;
+      (*H)->MapBase = Entry.MapBase;
+      (*H)->MapSize = Entry.MapSize;
+      (*H)->Data = Entry.Data;
       EntriesCount--;
-      return true;
     }
-    return false;
+    return Found;
   }
 
   bool canCache(uptr Size) {
     return atomic_load_relaxed(&MaxEntriesCount) != 0U &&
            Size <= atomic_load_relaxed(&MaxEntrySize);
   }
 
   bool setOption(Option O, sptr Value) {
if (O == Option::ReleaseInterval) {
       return true;
     }
     // Not supported by the Secondary Cache, but not an error either.
     return true;
   }
 
   void releaseToOS() { releaseOlderThan(UINT64_MAX); }
 
+  void disableMemoryTagging() {
+    ScopedLock L(Mutex);
+    for (u32 I = 0; I != Config::SecondaryCacheQuarantineSize; ++I) {
+      if (Quarantine[I].CommitBase) {
+        unmap(reinterpret_cast<void *>(Quarantine[I].MapBase),
+              Quarantine[I].MapSize, UNMAP_ALL, &Quarantine[I].Data);
+        Quarantine[I].CommitBase = 0;
+      }
+    }
+    const u32 MaxCount = atomic_load_relaxed(&MaxEntriesCount);
+    for (u32 I = 0; I < MaxCount; I++)
+      if (Entries[I].CommitBase)
+        setMemoryPermission(Entries[I].CommitBase, Entries[I].CommitSize, 0,
+                            &Entries[I].Data);
+    QuarantinePos = -1U;
+  }
+
   void disable() { Mutex.lock(); }
 
   void enable() { Mutex.unlock(); }
 
 private:
   void empty() {
     struct {
       void *MapBase;
       uptr MapSize;
       MapPlatformData Data;
     } MapInfo[Config::SecondaryCacheEntriesArraySize];
     uptr N = 0;
     {
       ScopedLock L(Mutex);
       for (uptr I = 0; I < Config::SecondaryCacheEntriesArraySize; I++) {
-        if (!Entries[I].Block)
+        if (!Entries[I].CommitBase)
           continue;
         MapInfo[N].MapBase = reinterpret_cast<void *>(Entries[I].MapBase);
         MapInfo[N].MapSize = Entries[I].MapSize;
         MapInfo[N].Data = Entries[I].Data;
-        Entries[I].Block = 0;
+        Entries[I].CommitBase = 0;
         N++;
       }
       EntriesCount = 0;
       IsFullEvents = 0;
     }
     for (uptr I = 0; I < N; I++)
       unmap(MapInfo[I].MapBase, MapInfo[I].MapSize, UNMAP_ALL,
             &MapInfo[I].Data);
   }
 
-  void releaseOlderThan(u64 Time) {
-    ScopedLock L(Mutex);
-    if (!EntriesCount)
-      return;
-    for (uptr I = 0; I < Config::SecondaryCacheEntriesArraySize; I++) {
-      if (!Entries[I].Block || !Entries[I].Time || Entries[I].Time > Time)
-        continue;
-      releasePagesToOS(Entries[I].Block, 0,
-                       Entries[I].BlockEnd - Entries[I].Block,
-                       &Entries[I].Data);
-      Entries[I].Time = 0;
-    }
-  }
-
   struct CachedBlock {
-    uptr Block;
-    uptr BlockEnd;
+    uptr CommitBase;
+    uptr CommitSize;
     uptr MapBase;
     uptr MapSize;
+    uptr BlockBegin;
     [[no_unique_address]] MapPlatformData Data;
     u64 Time;
   };
 
+  void releaseIfOlderThan(CachedBlock &Entry, u64 Time) {
+    if (!Entry.CommitBase || !Entry.Time)
+      return;
+    if (Entry.Time > Time) {
+      if (OldestTime == 0 || Entry.Time < OldestTime)
+        OldestTime = Entry.Time;
+      return;
+    }
+    releasePagesToOS(Entry.CommitBase, 0, Entry.CommitSize, &Entry.Data);
+    Entry.Time = 0;
+  }
+
+  void releaseOlderThan(u64 Time) {
+    ScopedLock L(Mutex);
+    if (!EntriesCount || OldestTime == 0 || OldestTime > Time)
+      return;
+    OldestTime = 0;
+    for (uptr I = 0; I < Config::SecondaryCacheQuarantineSize; I++)
+      releaseIfOlderThan(Quarantine[I], Time);
+    for (uptr I = 0; I < Config::SecondaryCacheEntriesArraySize; I++)
+      releaseIfOlderThan(Entries[I], Time);
+  }
+
   HybridMutex Mutex;
-  CachedBlock Entries[Config::SecondaryCacheEntriesArraySize];
   u32 EntriesCount;
+  u32 QuarantinePos;
   atomic_u32 MaxEntriesCount;
   atomic_uptr MaxEntrySize;
-  uptr LargestSize;
+  u64 OldestTime;
   u32 IsFullEvents;
   atomic_s32 ReleaseToOsIntervalMs;
+
+  CachedBlock Entries[Config::SecondaryCacheEntriesArraySize];
+  CachedBlock Quarantine[Config::SecondaryCacheQuarantineSize];
 };
 
 template <typename Config> class MapAllocator {
 public:
   void initLinkerInitialized(GlobalStats *S, s32 ReleaseToOsInterval = -1) {
     Cache.initLinkerInitialized(ReleaseToOsInterval);
     Stats.initLinkerInitialized();
     if (LIKELY(S))
       S->link(&Stats);
   }
   void init(GlobalStats *S, s32 ReleaseToOsInterval = -1) {
     memset(this, 0, sizeof(*this));
     initLinkerInitialized(S, ReleaseToOsInterval);
   }
 
-  void *allocate(uptr Size, uptr AlignmentHint = 0, uptr *BlockEnd = nullptr,
+  void *allocate(Options Options, uptr Size, uptr AlignmentHint = 0,
+                 uptr *BlockEnd = nullptr,
                  FillContentsMode FillContents = NoFill);
 
-  void deallocate(void *Ptr);
+  void deallocate(Options Options, void *Ptr);
 
   static uptr getBlockEnd(void *Ptr) {
-    return LargeBlock::getHeader(Ptr)->BlockEnd;
+    auto *B = LargeBlock::getHeader<Config>(Ptr);
+    return B->CommitBase + B->CommitSize;
   }
 
   static uptr getBlockSize(void *Ptr) {
     return getBlockEnd(Ptr) - reinterpret_cast<uptr>(Ptr);
   }
 
   void getStats(ScopedString *Str) const;
 
   void disable() {
     Mutex.lock();
     Cache.disable();
   }
 
   void enable() {
     Cache.enable();
     Mutex.unlock();
   }
 
   template <typename F> void iterateOverBlocks(F Callback) const {
-    for (const auto &H : InUseBlocks)
-      Callback(reinterpret_cast<uptr>(&H) + LargeBlock::getHeaderSize());
+    for (const auto &H : InUseBlocks) {
+      uptr Ptr = reinterpret_cast<uptr>(&H) + LargeBlock::getHeaderSize();
+      if (allocatorSupportsMemoryTagging<Config>())
+        Ptr = untagPointer(Ptr);
+      Callback(Ptr);
+    }
   }
 
   uptr canCache(uptr Size) { return Cache.canCache(Size); }
 
   bool setOption(Option O, sptr Value) { return Cache.setOption(O, Value); }
 
   void releaseToOS() { Cache.releaseToOS(); }
 
+  void disableMemoryTagging() { Cache.disableMemoryTagging(); }
+
 private:
   typename Config::SecondaryCache Cache;
 
   HybridMutex Mutex;
   DoublyLinkedList<LargeBlock::Header> InUseBlocks;
   uptr AllocatedBytes;
   uptr FreedBytes;
   uptr LargestSize;

 // Due to the sparsity of address space available on those platforms, requesting
 // an allocation from the Secondary with a large alignment would end up wasting
 // VA space (even though we are not committing the whole thing), hence the need
 // to trim off some of the reserved space.
 // For allocations requested with an alignment greater than or equal to a page,
 // the committed memory will amount to something close to Size - AlignmentHint
 // (pending rounding and headers).
 template <typename Config>
-void *MapAllocator<Config>::allocate(uptr Size, uptr AlignmentHint,
-                                     uptr *BlockEnd,
+void *MapAllocator<Config>::allocate(Options Options, uptr Size, uptr Alignment,
+                                     uptr *BlockEndPtr,
                                      FillContentsMode FillContents) {
-  DCHECK_GE(Size, AlignmentHint);
+  if (Options.get(OptionBit::AddLargeAllocationSlack))
+    Size += 1UL << SCUDO_MIN_ALIGNMENT_LOG;
+  Alignment = Max(Alignment, 1UL << SCUDO_MIN_ALIGNMENT_LOG);
   const uptr PageSize = getPageSizeCached();
-  const uptr RoundedSize =
-      roundUpTo(Size + LargeBlock::getHeaderSize(), PageSize);
+  uptr RoundedSize =
+      roundUpTo(roundUpTo(Size, Alignment) + LargeBlock::getHeaderSize() +
+                    Chunk::getHeaderSize(),
+                PageSize);
+  if (Alignment > PageSize)
+    RoundedSize += Alignment - PageSize;
 
-  if (AlignmentHint < PageSize && Cache.canCache(RoundedSize)) {
+  if (Alignment < PageSize && Cache.canCache(RoundedSize)) {
     LargeBlock::Header *H;
     bool Zeroed;
-    if (Cache.retrieve(RoundedSize, &H, &Zeroed)) {
-      if (BlockEnd)
-        *BlockEnd = H->BlockEnd;
-      void *Ptr = reinterpret_cast<void *>(reinterpret_cast<uptr>(H) +
-                                           LargeBlock::getHeaderSize());
+    if (Cache.retrieve(Options, Size, Alignment, &H, &Zeroed)) {
+      const uptr BlockEnd = H->CommitBase + H->CommitSize;
+      if (BlockEndPtr)
+        *BlockEndPtr = BlockEnd;
+      uptr HInt = reinterpret_cast<uptr>(H);
+      if (allocatorSupportsMemoryTagging<Config>())
+        HInt = untagPointer(HInt);
+      const uptr PtrInt = HInt + LargeBlock::getHeaderSize();
+      void *Ptr = reinterpret_cast<void *>(PtrInt);
       if (FillContents && !Zeroed)
         memset(Ptr, FillContents == ZeroFill ? 0 : PatternFillByte,
-               H->BlockEnd - reinterpret_cast<uptr>(Ptr));
-      const uptr BlockSize = H->BlockEnd - reinterpret_cast<uptr>(H);
+               BlockEnd - PtrInt);
+      const uptr BlockSize = BlockEnd - HInt;
       {
         ScopedLock L(Mutex);
         InUseBlocks.push_back(H);
         AllocatedBytes += BlockSize;
         NumberOfAllocs++;
         Stats.add(StatAllocated, BlockSize);
         Stats.add(StatMapped, H->MapSize);
       }
       return Ptr;
     }
   }
 
   MapPlatformData Data = {};
   const uptr MapSize = RoundedSize + 2 * PageSize;
-  uptr MapBase =
-      reinterpret_cast<uptr>(map(nullptr, MapSize, "scudo:secondary",
-                                 MAP_NOACCESS | MAP_ALLOWNOMEM, &Data));
+  uptr MapBase = reinterpret_cast<uptr>(
+      map(nullptr, MapSize, nullptr, MAP_NOACCESS | MAP_ALLOWNOMEM, &Data));
   if (UNLIKELY(!MapBase))
     return nullptr;
   uptr CommitBase = MapBase + PageSize;
   uptr MapEnd = MapBase + MapSize;
 
   // In the unlikely event of alignments larger than a page, adjust the amount
   // of memory we want to commit, and trim the extra memory.
-  if (UNLIKELY(AlignmentHint >= PageSize)) {
+  if (UNLIKELY(Alignment >= PageSize)) {
     // For alignments greater than or equal to a page, the user pointer (eg: the
     // pointer that is returned by the C or C++ allocation APIs) ends up on a
     // page boundary , and our headers will live in the preceding page.
-    CommitBase = roundUpTo(MapBase + PageSize + 1, AlignmentHint) - PageSize;
+    CommitBase = roundUpTo(MapBase + PageSize + 1, Alignment) - PageSize;
     const uptr NewMapBase = CommitBase - PageSize;
     DCHECK_GE(NewMapBase, MapBase);
     // We only trim the extra memory on 32-bit platforms: 64-bit platforms
     // are less constrained memory wise, and that saves us two syscalls.
     if (SCUDO_WORDSIZE == 32U && NewMapBase != MapBase) {
       unmap(reinterpret_cast<void *>(MapBase), NewMapBase - MapBase, 0, &Data);
       MapBase = NewMapBase;
     }
-    const uptr NewMapEnd = CommitBase + PageSize +
-                           roundUpTo((Size - AlignmentHint), PageSize) +
-                           PageSize;
+    const uptr NewMapEnd =
+        CommitBase + PageSize + roundUpTo(Size, PageSize) + PageSize;
     DCHECK_LE(NewMapEnd, MapEnd);
     if (SCUDO_WORDSIZE == 32U && NewMapEnd != MapEnd) {
       unmap(reinterpret_cast<void *>(NewMapEnd), MapEnd - NewMapEnd, 0, &Data);
       MapEnd = NewMapEnd;
     }
   }
 
   const uptr CommitSize = MapEnd - PageSize - CommitBase;
-  const uptr Ptr = reinterpret_cast<uptr>(
-      map(reinterpret_cast<void *>(CommitBase), CommitSize, "scudo:secondary",
-          MAP_RESIZABLE, &Data));
-  LargeBlock::Header *H = reinterpret_cast<LargeBlock::Header *>(Ptr);
+  const uptr AllocPos = roundDownTo(CommitBase + CommitSize - Size, Alignment);
+  const uptr MaxUnusedCacheBytes = MaxUnusedCachePages * getPageSizeCached();
+  if (useMemoryTagging<Config>(Options) && CommitSize > MaxUnusedCacheBytes) {
+    const uptr UntaggedPos = Max(AllocPos, CommitBase + MaxUnusedCacheBytes);
+    map(reinterpret_cast<void *>(CommitBase), UntaggedPos - CommitBase,
+        "scudo:secondary", MAP_RESIZABLE | MAP_MEMTAG, &Data);
+    map(reinterpret_cast<void *>(UntaggedPos),
+        CommitBase + CommitSize - UntaggedPos, "scudo:secondary", MAP_RESIZABLE,
+        &Data);
+  } else {
+        MAP_RESIZABLE | (useMemoryTagging<Config>(Options) ? MAP_MEMTAG : 0),
+        &Data);
+  }
+  const uptr HeaderPos =
+      AllocPos - Chunk::getHeaderSize() - LargeBlock::getHeaderSize();
+  LargeBlock::Header *H = reinterpret_cast<LargeBlock::Header *>(
+      LargeBlock::addHeaderTag<Config>(HeaderPos));
+  if (useMemoryTagging<Config>(Options))
+    storeTags(LargeBlock::addHeaderTag<Config>(CommitBase),
+              reinterpret_cast<uptr>(H + 1));
   H->MapBase = MapBase;
   H->MapSize = MapEnd - MapBase;
-  H->BlockEnd = CommitBase + CommitSize;
+  H->CommitBase = CommitBase;
+  H->CommitSize = CommitSize;
   H->Data = Data;
-  if (BlockEnd)
-    *BlockEnd = CommitBase + CommitSize;
+  if (BlockEndPtr)
+    *BlockEndPtr = CommitBase + CommitSize;
   {
     ScopedLock L(Mutex);
     InUseBlocks.push_back(H);
     AllocatedBytes += CommitSize;
     if (LargestSize < CommitSize)
       LargestSize = CommitSize;
     NumberOfAllocs++;
     Stats.add(StatAllocated, CommitSize);
     Stats.add(StatMapped, H->MapSize);
   }
-  return reinterpret_cast<void *>(Ptr + LargeBlock::getHeaderSize());
+  return reinterpret_cast<void *>(HeaderPos + LargeBlock::getHeaderSize());
 }
 
-template <typename Config> void MapAllocator<Config>::deallocate(void *Ptr) {
-  LargeBlock::Header *H = LargeBlock::getHeader(Ptr);
-  const uptr Block = reinterpret_cast<uptr>(H);
-  const uptr CommitSize = H->BlockEnd - Block;
+template <typename Config>
+void MapAllocator<Config>::deallocate(Options Options, void *Ptr) {
+  LargeBlock::Header *H = LargeBlock::getHeader<Config>(Ptr);
+  const uptr CommitSize = H->CommitSize;
   {
     ScopedLock L(Mutex);
     InUseBlocks.remove(H);
     FreedBytes += CommitSize;
     NumberOfFrees++;
     Stats.sub(StatAllocated, CommitSize);
     Stats.sub(StatMapped, H->MapSize);
   }
-  if (Cache.canCache(CommitSize) && Cache.store(H))
-    return;
-  void *Addr = reinterpret_cast<void *>(H->MapBase);
-  const uptr Size = H->MapSize;
-  MapPlatformData Data = H->Data;
-  unmap(Addr, Size, UNMAP_ALL, &Data);
+  Cache.store(Options, H);
 }
 
 template <typename Config>
 void MapAllocator<Config>::getStats(ScopedString *Str) const {
   Str->append(
       "Stats: MapAllocator: allocated %zu times (%zuK), freed %zu times "
       "(%zuK), remains %zu (%zuK) max %zuM\n",
       NumberOfAllocs, AllocatedBytes >> 10, NumberOfFrees, FreedBytes >> 10,
       NumberOfAllocs - NumberOfFrees, (AllocatedBytes - FreedBytes) >> 10,
       LargestSize >> 20);
 }
 
 } // namespace scudo
 
 #endif // SCUDO_SECONDARY_H_

compiler-rt/lib/scudo/standalone/tests/combined_test.cpp

if (Alignment < MinAlignment)
     Alignment = MinAlignment;
   const scudo::uptr NeededSize =
       scudo::roundUpTo(Size, MinAlignment) +
       ((Alignment > MinAlignment) ? Alignment : scudo::Chunk::getHeaderSize());
   return AllocatorT::PrimaryT::canAllocate(NeededSize);
 }
 
 template <class AllocatorT>
-bool isTaggedAllocation(AllocatorT *Allocator, scudo::uptr Size,
-                        scudo::uptr Alignment) {
-  return Allocator->useMemoryTaggingTestOnly() &&
-         scudo::systemDetectsMemoryTagFaultsTestOnly() &&
-         isPrimaryAllocation<AllocatorT>(Size, Alignment);
-}
-
-template <class AllocatorT>
 void checkMemoryTaggingMaybe(AllocatorT *Allocator, void *P, scudo::uptr Size,
                              scudo::uptr Alignment) {
-  if (!isTaggedAllocation(Allocator, Size, Alignment))
-    return;
-
-  Size = scudo::roundUpTo(Size, scudo::archMemoryTagGranuleSize());
+  const scudo::uptr MinAlignment = 1UL << SCUDO_MIN_ALIGNMENT_LOG;
+  Size = scudo::roundUpTo(Size, MinAlignment);
+  if (Allocator->useMemoryTaggingTestOnly())
     EXPECT_DEATH(
         {
           disableDebuggerdMaybe();
           reinterpret_cast<char *>(P)[-1] = 0xaa;
         },
         "");
+  if (isPrimaryAllocation<AllocatorT>(Size, Alignment)
+          ? Allocator->useMemoryTaggingTestOnly()
+          : Alignment == MinAlignment) {
     EXPECT_DEATH(
         {
           disableDebuggerdMaybe();
           reinterpret_cast<char *>(P)[Size] = 0xaa;
         },
         "");
   }
+}
 
 template <typename Config> struct TestAllocator : scudo::Allocator<Config> {
   TestAllocator() {
     this->reset();
     this->initThreadMaybe();
+    if (scudo::archSupportsMemoryTagging() &&
+        !scudo::systemDetectsMemoryTagFaultsTestOnly())
+      this->disableMemoryTagging();
   }
   ~TestAllocator() { this->unmapTestOnly(); }
 };
 
 template <class Config> static void testAllocator() {
   using AllocatorT = TestAllocator<Config>;
   auto Allocator = std::unique_ptr<AllocatorT>(new AllocatorT());
 
template <class Config> static void testAllocator() {
 
   // Verify that a chunk will end up being reused, at some point.
   const scudo::uptr NeedleSize = 1024U;
   void *NeedleP = Allocator->allocate(NeedleSize, Origin);
   Allocator->deallocate(NeedleP, Origin);
   bool Found = false;
   for (scudo::uptr I = 0; I < 1024U && !Found; I++) {
     void *P = Allocator->allocate(NeedleSize, Origin);
-    if (Allocator->untagPointerMaybe(P) ==
-        Allocator->untagPointerMaybe(NeedleP))
+    if (Allocator->getHeaderTaggedPointer(P) ==
+        Allocator->getHeaderTaggedPointer(NeedleP))
       Found = true;
     Allocator->deallocate(P, Origin);
   }
   EXPECT_TRUE(Found);
 
   constexpr scudo::uptr MaxSize = Config::Primary::SizeClassMap::MaxSize;
 
   // Reallocate a large chunk all the way down to a byte, verifying that we
if (!UseQuarantine) {
     while (!V.empty()) {
       Allocator->deallocate(V.back(), Origin);
       V.pop_back();
     }
   }
 
   Allocator->releaseToOS();
 
-  if (Allocator->useMemoryTaggingTestOnly() &&
-      scudo::systemDetectsMemoryTagFaultsTestOnly()) {
-    // Check that use-after-free is detected.
-    for (scudo::uptr SizeLog = 0U; SizeLog <= 20U; SizeLog++) {
-      const scudo::uptr Size = 1U << SizeLog;
-      if (!isTaggedAllocation(Allocator.get(), Size, 1))
-        continue;
-      // UAF detection is probabilistic, so we repeat the test up to 256 times
-      // if necessary. With 15 possible tags this means a 1 in 15^256 chance of
-      // a false positive.
-      EXPECT_DEATH(
-          {
-            disableDebuggerdMaybe();
-            for (unsigned I = 0; I != 256; ++I) {
-              void *P = Allocator->allocate(Size, Origin);
-              Allocator->deallocate(P, Origin);
-              reinterpret_cast<char *>(P)[0] = 0xaa;
-            }
-          },
-          "");
-      EXPECT_DEATH(
-          {
-            disableDebuggerdMaybe();
-            for (unsigned I = 0; I != 256; ++I) {
-              void *P = Allocator->allocate(Size, Origin);
-              Allocator->deallocate(P, Origin);
-              reinterpret_cast<char *>(P)[Size - 1] = 0xaa;
-            }
-          },
-          "");
-    }
+    if (!Allocator->useMemoryTaggingTestOnly())
 
+  if (Allocator->useMemoryTaggingTestOnly()) {
     // Check that disabling memory tagging works correctly.
     void *P = Allocator->allocate(2048, Origin);
     EXPECT_DEATH(reinterpret_cast<char *>(P)[2048] = 0xaa, "");
     scudo::disableMemoryTagChecksTestOnly();
     Allocator->disableMemoryTagging();
     reinterpret_cast<char *>(P)[2048] = 0xaa;
     Allocator->deallocate(P, Origin);
 
     P = Allocator->allocate(2048, Origin);
-    EXPECT_EQ(Allocator->untagPointerMaybe(P), P);
+    EXPECT_EQ(scudo::untagPointer(P), P);
     reinterpret_cast<char *>(P)[2048] = 0xaa;
     Allocator->deallocate(P, Origin);
 
     Allocator->releaseToOS();
 
     // Disabling memory tag checks may interfere with subsequent tests.
     // Re-enable them now.
     scudo::enableMemoryTagChecksTestOnly();

compiler-rt/lib/scudo/standalone/tests/secondary_test.cpp

 template <typename Config> static void testSecondaryBasic(void) {
   using SecondaryT = scudo::MapAllocator<Config>;
 
   scudo::GlobalStats S;
   S.init();
   std::unique_ptr<SecondaryT> L(new SecondaryT);
   L->init(&S);
   const scudo::uptr Size = 1U << 16;
-  void *P = L->allocate(Size);
+  void *P = L->allocate(scudo::Options{}, Size);
   EXPECT_NE(P, nullptr);
   memset(P, 'A', Size);
   EXPECT_GE(SecondaryT::getBlockSize(P), Size);
-  L->deallocate(P);
+  L->deallocate(scudo::Options{}, P);
   // If the Secondary can't cache that pointer, it will be unmapped.
   if (!L->canCache(Size))
     EXPECT_DEATH(memset(P, 'A', Size), "");
 
   const scudo::uptr Align = 1U << 16;
-  P = L->allocate(Size + Align, Align);
+  P = L->allocate(scudo::Options{}, Size + Align, Align);
   EXPECT_NE(P, nullptr);
   void *AlignedP = reinterpret_cast<void *>(
       scudo::roundUpTo(reinterpret_cast<scudo::uptr>(P), Align));
   memset(AlignedP, 'A', Size);
-  L->deallocate(P);
+  L->deallocate(scudo::Options{}, P);
 
   std::vector<void *> V;
   for (scudo::uptr I = 0; I < 32U; I++)
-    V.push_back(L->allocate(Size));
+    V.push_back(L->allocate(scudo::Options{}, Size));
   std::shuffle(V.begin(), V.end(), std::mt19937(std::random_device()()));
   while (!V.empty()) {
-    L->deallocate(V.back());
+    L->deallocate(scudo::Options{}, V.back());
     V.pop_back();
   }
   scudo::ScopedString Str(1024);
   L->getStats(&Str);
   Str.output();
 }
 
 struct NoCacheConfig {
   typedef scudo::MapAllocatorNoCache SecondaryCache;
+  static const bool MaySupportMemoryTagging = false;
 };
 
 struct TestConfig {
   typedef scudo::MapAllocatorCache<TestConfig> SecondaryCache;
+  static const bool MaySupportMemoryTagging = false;
   static const scudo::u32 SecondaryCacheEntriesArraySize = 128U;
+  static const scudo::u32 SecondaryCacheQuarantineSize = 0U;
   static const scudo::u32 SecondaryCacheDefaultMaxEntriesCount = 64U;
   static const scudo::uptr SecondaryCacheDefaultMaxEntrySize = 1UL << 20;
   static const scudo::s32 SecondaryCacheMinReleaseToOsIntervalMs = INT32_MIN;
   static const scudo::s32 SecondaryCacheMaxReleaseToOsIntervalMs = INT32_MAX;
 };
 
 TEST(ScudoSecondaryTest, SecondaryBasic) {
   testSecondaryBasic<NoCacheConfig>();
for (scudo::uptr AlignLog = FIRST_32_SECOND_64(3, 4); AlignLog <= 16;
       const scudo::uptr Align = 1U << AlignLog;
       for (scudo::sptr Delta = -128; Delta <= 128; Delta += 8) {
         if (static_cast<scudo::sptr>(1U << SizeLog) + Delta <= 0)
           continue;
         const scudo::uptr UserSize =
             scudo::roundUpTo((1U << SizeLog) + Delta, MinAlign);
         const scudo::uptr Size =
             HeaderSize + UserSize + (Align > MinAlign ? Align - HeaderSize : 0);
-        void *P = L->allocate(Size, Align);
+        void *P = L->allocate(scudo::Options{}, Size, Align);
         EXPECT_NE(P, nullptr);
         void *AlignedP = reinterpret_cast<void *>(
             scudo::roundUpTo(reinterpret_cast<scudo::uptr>(P), Align));
         memset(AlignedP, 0xff, UserSize);
-        L->deallocate(P);
+        L->deallocate(scudo::Options{}, P);
       }
     }
   }
   scudo::ScopedString Str(1024);
   L->getStats(&Str);
   Str.output();
 }
 
 TEST(ScudoSecondaryTest, SecondaryIterate) {
   std::unique_ptr<LargeAllocator> L(new LargeAllocator);
   L->init(nullptr);
   std::vector<void *> V;
   const scudo::uptr PageSize = scudo::getPageSizeCached();
   for (scudo::uptr I = 0; I < 32U; I++)
-    V.push_back(L->allocate((std::rand() % 16) * PageSize));
+    V.push_back(L->allocate(scudo::Options{}, (std::rand() % 16) * PageSize));
   auto Lambda = [V](scudo::uptr Block) {
     EXPECT_NE(std::find(V.begin(), V.end(), reinterpret_cast<void *>(Block)),
               V.end());
   };
   L->disable();
   L->iterateOverBlocks(Lambda);
   L->enable();
   while (!V.empty()) {
-    L->deallocate(V.back());
+    L->deallocate(scudo::Options{}, V.back());
     V.pop_back();
   }
   scudo::ScopedString Str(1024);
   L->getStats(&Str);
   Str.output();
 }
 
 TEST(ScudoSecondaryTest, SecondaryOptions) {
static void performAllocations(LargeAllocator *L) {
   {
     std::unique_lock<std::mutex> Lock(Mutex);
     while (!Ready)
       Cv.wait(Lock);
   }
   for (scudo::uptr I = 0; I < 128U; I++) {
     // Deallocate 75% of the blocks.
     const bool Deallocate = (rand() & 3) != 0;
-    void *P = L->allocate((std::rand() % 16) * PageSize);
+    void *P = L->allocate(scudo::Options{}, (std::rand() % 16) * PageSize);
     if (Deallocate)
-      L->deallocate(P);
+      L->deallocate(scudo::Options{}, P);
     else
       V.push_back(P);
   }
   while (!V.empty()) {
-    L->deallocate(V.back());
+    L->deallocate(scudo::Options{}, V.back());
     V.pop_back();
   }
 }
 
 TEST(ScudoSecondaryTest, SecondaryThreadsRace) {
   Ready = false;
   std::unique_ptr<LargeAllocator> L(new LargeAllocator);
   L->init(nullptr, /*ReleaseToOsInterval=*/0);

compiler-rt/lib/scudo/standalone/tsd_exclusive.h

void initLinkerInitialized(Allocator *Instance) {
     FallbackTSD.initLinkerInitialized(Instance);
     Initialized = true;
   }
   void init(Allocator *Instance) {
     memset(this, 0, sizeof(*this));
     initLinkerInitialized(Instance);
   }
 
+  void initOnceMaybe(Allocator *Instance) {
+    ScopedLock L(Mutex);
+    if (LIKELY(Initialized))
+      return;
+    initLinkerInitialized(Instance); // Sets Initialized.
+  }
+
   void unmapTestOnly() {}
 
   ALWAYS_INLINE void initThreadMaybe(Allocator *Instance, bool MinimalInit) {
     if (LIKELY(State.InitState != ThreadState::NotInitialized))
       return;
     initThread(Instance, MinimalInit);
   }
 
bool setOption(Option O, UNUSED sptr Value) {
     if (O == Option::MaxTSDsCount)
       return false;
     return true;
   }
 
   bool getDisableMemInit() { return State.DisableMemInit; }
 
 private:
-  void initOnceMaybe(Allocator *Instance) {
-    ScopedLock L(Mutex);
-    if (LIKELY(Initialized))
-      return;
-    initLinkerInitialized(Instance); // Sets Initialized.
-  }
-
   // Using minimal initialization allows for global initialization while keeping
   // the thread specific structure untouched. The fallback structure will be
   // used instead.
   NOINLINE void initThread(Allocator *Instance, bool MinimalInit) {
     initOnceMaybe(Instance);
     if (UNLIKELY(MinimalInit))
       return;
     CHECK_EQ(

compiler-rt/lib/scudo/standalone/tsd_shared.h

setNumberOfTSDs((NumberOfCPUs == 0) ? DefaultTSDCount
                                         : Min(NumberOfCPUs, DefaultTSDCount));
     Initialized = true;
   }
   void init(Allocator *Instance) {
     memset(this, 0, sizeof(*this));
     initLinkerInitialized(Instance);
   }
 
+  void initOnceMaybe(Allocator *Instance) {
+    ScopedLock L(Mutex);
+    if (LIKELY(Initialized))
+      return;
+    initLinkerInitialized(Instance); // Sets Initialized.
+  }
+
   void unmapTestOnly() { setCurrentTSD(nullptr); }
 
   ALWAYS_INLINE void initThreadMaybe(Allocator *Instance,
                                      UNUSED bool MinimalInit) {
     if (LIKELY(getCurrentTSD()))
       return;
     initThread(Instance);
   }
bool setNumberOfTSDs(u32 N) {
     return true;
   }
 
   void setDisableMemInit(bool B) {
     *getTlsPtr() &= ~1ULL;
     *getTlsPtr() |= B;
   }
 
-  void initOnceMaybe(Allocator *Instance) {
-    ScopedLock L(Mutex);
-    if (LIKELY(Initialized))
-      return;
-    initLinkerInitialized(Instance); // Sets Initialized.
-  }
-
   NOINLINE void initThread(Allocator *Instance) {
     initOnceMaybe(Instance);
     // Initial context assignment is done in a plain round-robin fashion.
     const u32 Index = atomic_fetch_add(&CurrentIndex, 1U, memory_order_relaxed);
     setCurrentTSD(&TSDs[Index % NumberOfTSDs]);
     Instance->callPostInitCallback();
   }
 

compiler-rt/lib/scudo/standalone/wrappers_c.inc

 
 // Sets whether scudo pattern-initializes all allocated memory.
 INTERFACE WEAK void
 SCUDO_PREFIX(malloc_set_pattern_fill_contents)(int pattern_fill_contents) {
   SCUDO_ALLOCATOR.setFillContents(
       pattern_fill_contents ? scudo::PatternOrZeroFill : scudo::NoFill);
 }
 
+// Sets whether scudo adds a small amount of slack at the end of large
+// allocations, before the guard page. This can be enabled to work around buggy
+// applications that read a few bytes past the end of their allocation.
+INTERFACE WEAK void
+SCUDO_PREFIX(malloc_set_add_large_allocation_slack)(int add_slack) {
+  SCUDO_ALLOCATOR.setAddLargeAllocationSlack(add_slack);
+}
+
 } // extern "C"