[asan] Don't double poison secondary allocations

[vitalybuka]

Summary

Sanitizers allocate shadow and memory as MAP_NORESERVE.

User memory can stay this way and do not increase RSS as long as we
don't store there.

The shadow unpoisoning also can avoid RSS increase for zeroed pages.
However as soon we poison the shadow, we need the page in RSS.

To avoid unnececary RSS increase we should not poison memory just before
unpoisoning them.

Depends on D153497.

[ro]

This test causes massive problems on the Solaris/amd64 builder:

load averages:  2.96,  3.29,  4.21;               up 4+11:19:19        09:22:13
65 processes: 63 sleeping, 2 on cpu
CPU states: 87.4% idle,  0.0% user, 12.4% kernel,  0.2% stolen,  0.0% swap
Kernel: 3107 ctxsw, 5994 trap, 2564 intr, 881 syscall, 5990 flt, 1596 pgout
Memory: 256G phys mem, 320M free mem, 64G total swap, 60G free swap

   PID USERNAME NLWP PRI NICE  SIZE   RES STATE     TIME    CPU COMMAND
  7299 llvm-bui    1  40    0  241M  173M cpu/2    13:26  4.79% Sanitizer-x86_6
  8958 llvm-bui    1  60    0  256G   67G sleep     9:01  0.14% huge_malloc.c.t

The system has 256 GB RAM and 64 GB swap. However, unlike Linux, Solaris doesn't do lazy allocation, so the test needs 256 GB of backing store, which causes the host to almost go to a halt, badly impacting another builder on the same system.

[ro]

I've now temporarily disabled the Solaris/amd64 buildbot: this test caused the other builder on the same host to fail due to lack of memory several times! How should we handle this? Just mark the test as UNSUPPORTED on Solaris, or restrict it to Linux and Android instead? Keeping it as is is unacceptable.

[vitalybuka]

In D153500#4451868, @ro wrote:

I've now temporarily disabled the Solaris/amd64 buildbot: this test caused the other builder on the same host to fail due to lack of memory several times! How should we handle this? Just mark the test as UNSUPPORTED on Solaris, or restrict it to Linux and Android instead? Keeping it as is is unacceptable.

Sorry of that.

Please mark UNSUPPORTED with comment.

Is this a different implementation of MAP_NORESERVE or just Solaris sanitizer implementation does not uses MAP_NORESERVE?
If there is a way to make use of MAP_NORESERVE, it's worse of investigation. Sanitizers heavily rely on that.

[vitalybuka]

In D153500#4453008, @vitalybuka wrote:

In D153500#4451868, @ro wrote:

I've now temporarily disabled the Solaris/amd64 buildbot: this test caused the other builder on the same host to fail due to lack of memory several times! How should we handle this? Just mark the test as UNSUPPORTED on Solaris, or restrict it to Linux and Android instead? Keeping it as is is unacceptable.

Sorry of that.

Please mark UNSUPPORTED with comment.

Is this a different implementation of MAP_NORESERVE or just Solaris sanitizer implementation does not uses MAP_NORESERVE?
If there is a way to make use of MAP_NORESERVE, it's worse of investigation. Sanitizers heavily rely on that.

Also there is hard_rss_limit_mb, why it does not kill the process like on linux.

[ro]

In D153500#4453008, @vitalybuka wrote:

Also there is hard_rss_limit_mb, why it does not kill the process like on linux.

This is implemented in sanitizer_common_libcdep.cpp, but guarded by (SANITIZER_LINUX || SANITIZER_NETBSD) && !SANITIZER_GO

[ro]

In D153500#4453008, @vitalybuka wrote:

Please mark UNSUPPORTED with comment.

Thanks for taking care of this already.

Is this a different implementation of MAP_NORESERVE or just Solaris sanitizer implementation does not uses MAP_NORESERVE?
If there is a way to make use of MAP_NORESERVE, it's worse of investigation. Sanitizers heavily rely on that.

I've meanwhile run the test under truss (Solaris) resp. strace (Linux), with unexpected results:

• On Solaris, only brk is used:

5246:	brk(0x8000000000E955C0)				Err#12 ENOMEM

• while on Linux, mmap is used indeed:

879621 mmap(NULL, 4611686018427523072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

however no MAP_NORESERVE in sight.

[ro]

In D153500#4455548, @ro wrote:

In D153500#4453008, @vitalybuka wrote:

Please mark UNSUPPORTED with comment.

Thanks for taking care of this already.

Unfortunately, the

UNSUPPORTED: solaris

syntax doesn't work any longer (i.e. it's silently ignored). Judging from other compiler-rt testcases, one now needs to match the target triple explicitly, like

UNSUPPORTED: target={{.*solaris.*}}

At least that's what worked for me locally.

compiler-rt/lib/asan/asan_allocator.cpp

void AsanMapUnmapCallback::OnMap(uptr p, uptr size) const {
   // Statistics.
   AsanStats &thread_stats = GetCurrentThreadStats();
   thread_stats.mmaps++;
   thread_stats.mmaped += size;
 }
 
 void AsanMapUnmapCallback::OnMapSecondary(uptr p, uptr size, uptr user_begin,
                                           uptr user_size) const {
-  PoisonShadow(p, size, kAsanHeapLeftRedzoneMagic);
+  uptr user_end = RoundDownTo(user_begin + user_size, ASAN_SHADOW_GRANULARITY);
+  user_begin = RoundUpTo(user_begin, ASAN_SHADOW_GRANULARITY);
+  // The secondary mapping will be immediately returned to user, no value
+  // poisoning that with non-zero just before unpoisoning by Allocate(). So just
+  // poison head/tail invisible to Allocate().
+  PoisonShadow(p, user_begin - p, kAsanHeapLeftRedzoneMagic);
+  PoisonShadow(user_end, size - (user_end - p), kAsanHeapLeftRedzoneMagic);
   // Statistics.
   AsanStats &thread_stats = GetCurrentThreadStats();
   thread_stats.mmaps++;
   thread_stats.mmaped += size;
 }
 
 void AsanMapUnmapCallback::OnUnmap(uptr p, uptr size) const {
   PoisonShadow(p, size, 0);
void *Allocate(uptr size, uptr alignment, BufferedStackTrace *stack,
     }
     CHECK(IsPowerOfTwo(alignment));
     uptr rz_log = ComputeRZLog(size);
     uptr rz_size = RZLog2Size(rz_log);
     uptr rounded_size = RoundUpTo(Max(size, kChunkHeader2Size), alignment);
     uptr needed_size = rounded_size + rz_size;
     if (alignment > min_alignment)
       needed_size += alignment;
+    bool from_primary = PrimaryAllocator::CanAllocate(needed_size, alignment);
     // If we are allocating from the secondary allocator, there will be no
     // automatic right redzone, so add the right redzone manually.
-    if (!PrimaryAllocator::CanAllocate(needed_size, alignment))
+    if (!from_primary)
       needed_size += rz_size;
     CHECK(IsAligned(needed_size, min_alignment));
     if (size > kMaxAllowedMallocSize || needed_size > kMaxAllowedMallocSize ||
         size > max_user_defined_malloc_size) {
       if (AllocatorMayReturnNull()) {
         Report("WARNING: AddressSanitizer failed to allocate 0x%zx bytes\n",
                size);
         return nullptr;
void *Allocate(uptr size, uptr alignment, BufferedStackTrace *stack,
     }
     if (UNLIKELY(!allocated)) {
       SetAllocatorOutOfMemory();
       if (AllocatorMayReturnNull())
         return nullptr;
       ReportOutOfMemory(size, stack);
     }
 
-    if (*(u8 *)MEM_TO_SHADOW((uptr)allocated) == 0 && CanPoisonMemory()) {
-      // Heap poisoning is enabled, but the allocator provides an unpoisoned
-      // chunk. This is possible if CanPoisonMemory() was false for some
-      // time, for example, due to flags()->start_disabled.
-      // Anyway, poison the block before using it for anything else.
-      uptr allocated_size = allocator.GetActuallyAllocatedSize(allocated);
-      PoisonShadow((uptr)allocated, allocated_size, kAsanHeapLeftRedzoneMagic);
-    }
-
     uptr alloc_beg = reinterpret_cast<uptr>(allocated);
     uptr alloc_end = alloc_beg + needed_size;
     uptr user_beg = alloc_beg + rz_size;
     if (!IsAligned(user_beg, alignment))
       user_beg = RoundUpTo(user_beg, alignment);
     uptr user_end = user_beg + size;
     CHECK_LE(user_end, alloc_end);
     uptr chunk_beg = user_beg - kChunkHeaderSize;
     AsanChunk *m = reinterpret_cast<AsanChunk *>(chunk_beg);
     m->alloc_type = alloc_type;
     CHECK(size);
     m->SetUsedSize(size);
     m->user_requested_alignment_log = user_requested_alignment_log;
 
     m->SetAllocContext(t ? t->tid() : kMainTid, StackDepotPut(*stack));
 
+    if (!from_primary || *(u8 *)MEM_TO_SHADOW((uptr)allocated) == 0) {
+      // The allocator provides an unpoisoned chunk. This is possible for the
+      // secondary allocator, or if CanPoisonMemory() was false for some time,
+      // for example, due to flags()->start_disabled. Anyway, poison left and
+      // right of the block before using it for anything else.
+      uptr tail_beg = RoundUpTo(user_end, ASAN_SHADOW_GRANULARITY);
+      uptr tail_end = alloc_beg + allocator.GetActuallyAllocatedSize(allocated);
+      PoisonShadow(alloc_beg, user_beg - alloc_beg, kAsanHeapLeftRedzoneMagic);
+      PoisonShadow(tail_beg, tail_end - tail_beg, kAsanHeapLeftRedzoneMagic);
+    }
+
     uptr size_rounded_down_to_granularity =
         RoundDownTo(size, ASAN_SHADOW_GRANULARITY);
     // Unpoison the bulk of the memory region.
     if (size_rounded_down_to_granularity)
       PoisonShadow(user_beg, size_rounded_down_to_granularity, 0);
     // Deal with the end of the region if size is not aligned to granularity.
     if (size != size_rounded_down_to_granularity && CanPoisonMemory()) {
       u8 *shadow =

compiler-rt/test/sanitizer_common/TestCases/Posix/huge_malloc.c

-// RUN: %clang -O0 %s -o %t && %env_tool_opts=allocator_may_return_null=1:hard_rss_limit_mb=50 %run %t
+// RUN: %clang -O0 %s -o %t && %env_tool_opts=allocator_may_return_null=1:hard_rss_limit_mb=50:quarantine_size_mb=1 %run %t
 
 #include <stdlib.h>
 #include <stdio.h>
 
 // FIXME: Hangs.
 // UNSUPPORTED: tsan
 
 // https://github.com/google/sanitizers/issues/981
 // UNSUPPORTED: android-26
 
 // FIXME: Make it work. Don't xfail to avoid excessive memory usage.
-// UNSUPPORTED: asan, msan, hwasan
+// UNSUPPORTED: msan, hwasan
 
 void *p;
 
 int main(int argc, char **argv) {
   for (int i = 0; i < sizeof(void *) * 8; ++i) {
     p = malloc(1ull << i);
     fprintf(stderr, "%llu: %p\n", (1ull << i), p);
     free(p);
   }
   return 0;
 }