[asan] Speed up __asan_unpoison_stack_memory on linux

[loladiro]

Summary

The __asan_unpoison_stack_memory implementation attempts to zero pages more
quickly by asking the kernel to mmap new zero pages over the region to be
zerod. To do this, it performs three system calls, the mmap and then two
madvise calls to restore the madvise flags for the region. For very large
regions, this is generally better than attempting to memset the entire thing,
however, we can do even better (at least on linux), by not touching the memory
map at all and just asking the kernel to replace the memory in question by zero
pages using MADV_DONTNEED. With the current implementation, the kernel needs to
do a fair bit of work to split the original shadow region on the first mmap and
then merge it back together again after the second madvise. By using
MADV_DONTNEED, we can skip this work, while obtaining the same result (replacing
the memory range in question by demand-paged zero pages). In extreme cases,
this can be significantly faster. Consider for example, the following benchmark:

extern void __asan_unpoison_stack_memory(void *, size_t);

int main(void) {
    char *stacks[NSTACKS];
    for (int i = 0; i < NSTACKS; ++i) {
        char *stack = mmap(NULL, STACK_SIZE, PROT_READ | PROT_WRITE,
                            MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
        assert(stack != MAP_FAILED);
        stacks[i] = stack;
    }
    for (int i = 0; i < NROUNDS; i++) {
        // This is meant to simulate the repeated and rapid allocation of
        // fiber stacks. Obviously everything but the allocation is cut out,
        // but it's still a reasonable benchmark of the performance of
        // __asan_unpoison_stack_memory.
        __asan_unpoison_stack_memory(stacks[i % NSTACKS], STACK_SIZE);
        /* Stack would be used here by some task, the task finishes and the
           stack gets recycled for the next task. */
    }
    return 0;
}

On my machine, current master takes 1.4s on this benchmark, while this patch
is about 7x faster. Obviously, this is a best-case estimate. Replacing the
mmap by malloc gives timings that are closer together, but nevertheless with
the version in this patch being somewhat faster.

The other benefit, although my primary motivation for this is somewhat more
subtle: In the current implementation, as mentioned above, the kernel needs
to figure out that it can stitch the memory regions back together after
the second madvise. However, in an application that does lots of forking,
we have observed that the kernel is sometimes unable to stich the shadow
memory region back together, shattering it into many small pieces over the
lifetime of the process and eventually exceeding the system maximum map limit.
My understanding is that this is because the kernel will refuse to merge
memory regions that do not share the same anon_vma, which gets established
by fork and even if the fork later concludes, the kernel does not go back
to re-coalesce any mappings it might have missed. Because this patch does
not split the original shadow region, it can avoid this particular issue.

As an addendum to this, I think a similar fracturing would occur on systems
that have decorate_proc_maps enabled, as the kernel will refuse to merge
mappings with different names and the current implementation does not provide
a name for the replacement pages (but does name the shadow regions in general).
However, because ithis is not enabled by default on non-Android Linux,
I have not actually observed this.

[loladiro]

clang-format patch

[vitalybuka]

compiler-rt/lib/asan/asan_poisoning.h

79

can you move ReserveShadowMemoryRange into fallback of ReleaseMemoryPagesToOSAndZeroFill
and ReleaseMemoryPagesToOSAndZeroFill into sanitizer_common?

compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

2376

Just call ReleaseMemoryPagesToOS and CHECK is redundant.

compiler-rt/lib/asan/asan_poisoning.h

 //
 // Shadow memory poisoning by ASan RTL and by user application.
 //===----------------------------------------------------------------------===//
 
 #include "asan_interceptors.h"
 #include "asan_internal.h"
 #include "asan_mapping.h"
 #include "sanitizer_common/sanitizer_flags.h"
+#include "sanitizer_common/sanitizer_linux.h"
 #include "sanitizer_common/sanitizer_platform.h"
 
 namespace __asan {
 
 // Enable/disable memory poisoning.
 void SetCanPoisonMemory(bool value);
 bool CanPoisonMemory();
 
if (page_beg >= page_end) {
       REAL(memset)((void *)shadow_beg, 0, shadow_end - shadow_beg);
     } else {
       if (page_beg != shadow_beg) {
         REAL(memset)((void *)shadow_beg, 0, page_beg - shadow_beg);
       }
       if (page_end != shadow_end) {
         REAL(memset)((void *)page_end, 0, shadow_end - page_end);
       }
+#  ifdef SANITIZER_LINUX
+      // MADV_DONTNEED is not guaranteed to succeed. In particular it fails
+      // on locked pages. We never lock any shadow pages ourselves, but let's
+      // not assume that the user would never attempt to lock the entire process
+      // (e.g. using mlockall). However, if it does succeed, we are guaranteed
+      // that future accesses will be zero'd.
+      if (ReleaseMemoryPagesToOSAndZeroFill(page_beg, page_end) != 0)
+#  endif
         ReserveShadowMemoryRange(page_beg, page_end - 1, nullptr);
     }
   }
 #endif // SANITIZER_FUCHSIA
 }
 
 ALWAYS_INLINE void FastPoisonShadowPartialRightRedzone(
     uptr aligned_addr, uptr size, uptr redzone_size, u8 value) {
   DCHECK(CanPoisonMemory());

compiler-rt/lib/sanitizer_common/sanitizer_linux.h

 //===-- sanitizer_linux.h ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Linux-specific syscall wrappers and classes.

 bool LibraryNameIs(const char *full_name, const char *base_name);
 
 // Call cb for each region mapped by map.
 void ForEachMappedRegion(link_map *map, void (*cb)(const void *, uptr));
 
 // Releases memory pages entirely within the [beg, end] address range.
 // The pages no longer count toward RSS; reads are guaranteed to return 0.
 // Requires (but does not verify!) that pages are MAP_PRIVATE.
-inline void ReleaseMemoryPagesToOSAndZeroFill(uptr beg, uptr end) {
-  // man madvise on Linux promises zero-fill for anonymous private pages.
-  // Testing shows the same behaviour for private (but not anonymous) mappings
-  // of shm_open() files, as long as the underlying file is untouched.
-  CHECK(SANITIZER_LINUX);
-  ReleaseMemoryPagesToOS(beg, end);
-}
+// Returns 0 on success, -1 on error.
+int ReleaseMemoryPagesToOSAndZeroFill(uptr beg, uptr end);
 
 #if SANITIZER_ANDROID
 
 #if defined(__aarch64__)
 # define __get_tls() \
     ({ void** __v; __asm__("mrs %0, tpidr_el0" : "=r"(__v)); __v; })
 #elif defined(__arm__)
 # define __get_tls() \

compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

#endif // SANITIZER_USE_GETRANDOM
   if (internal_iserror(fd))
     return false;
   uptr res = internal_read(fd, buffer, length);
   if (internal_iserror(res))
     return false;
   internal_close(fd);
   return true;
 }
 
+int ReleaseMemoryPagesToOSAndZeroFill(uptr beg, uptr end) {
+  // man madvise on Linux promises zero-fill for anonymous private pages.
+  // Testing shows the same behaviour for private (but not anonymous) mappings
+  // of shm_open() files, as long as the underlying file is untouched.
+  CHECK(SANITIZER_LINUX);
+  // This is the same as ReleaseMemoryPagesToOS, but with additional error
+  // handling.
+  uptr page_size = GetPageSizeCached();
+  uptr beg_aligned = RoundUpTo(beg, page_size);
+  uptr end_aligned = RoundDownTo(end, page_size);
+  if (beg_aligned < end_aligned)
+    return internal_madvise(beg_aligned, end_aligned - beg_aligned,
+                            SANITIZER_MADVISE_DONTNEED);
+  return 0;
+}
+
 } // namespace __sanitizer
 
 #endif