Set Huge Page mode on shadow regions based on no_huge_pages_for_shadow

[stephan.yichao.zhao]

Summary

It turned out that at dynamic shared library mode, the memory access

pattern can increase memory footprint significantly on OS when transparent
hugepages (THP) are enabled. This could cause >70x memory overhead than
running a static linked binary. For example, a static binary with RSS
overhead 300M can use > 23G RSS if it is built dynamically.
/proc/../smaps shows in 6204552 kB RSS 6141952 kB relates to
AnonHugePages.

Also such a high RSS happens in some rate: around 25% runs may use > 23G RSS, the
rest uses in between 6-23G. I guess this may relate to how user memory
is allocated and distributted across huge pages.

THP is a trade-off between time and space. We have a flag
no_huge_pages_for_shadow for sanitizer. It is true by default but DFSan
did not follow this. Depending on if a target is built statically or
dynamically, maybe Clang can set no_huge_pages_for_shadow accordingly
after this change. But it still seems fine to follow the default setting of
no_huge_pages_for_shadow. If time is an issue, and users are fine with
high RSS, this flag can be set to false selectively.

[morehouse]

compiler-rt/lib/dfsan/dfsan.cpp

440

Nit: Let's prefer SetShadowRegionFlags style for new code.

[morehouse]

compiler-rt/lib/dfsan/dfsan.cpp

440

On second thought, let's eliminate this static function entirely since we only need to call it from one place.

447

Let's reuse MmapFixedSuperNoReserve and then to the use_madv_dontdump check inline here.

458

I think this is unnecessary here since the memory is marked PROT_NONE anyway.

[eugenis]

But it still seems fine to follow the default setting of
no_huge_pages_for_shadow. If time is an issue, and users are fine with
high RSS, this flag can be set to false selectively.

I agree 100%

[stephan.yichao.zhao]

addressed comments

[stephan.yichao.zhao]

update

[morehouse]

LGTM, thanks!

[stephan.yichao.zhao]

submitted by https://github.com/llvm/llvm-project/commit/91dc545bf24daa60c21c93039408061194dd0ab3

compiler-rt/lib/dfsan/dfsan.cpp

static void dfsan_fini() {
   }
 }
 
 extern "C" void dfsan_flush() {
   if (!MmapFixedNoReserve(ShadowAddr(), UnusedAddr() - ShadowAddr()))
     Die();
 }
 
 static void dfsan_init(int argc, char **argv, char **envp) {
   InitializeFlags();
 
   ::InitializePlatformEarly();
 
-  if (!MmapFixedNoReserve(ShadowAddr(), UnusedAddr() - ShadowAddr()))
+  if (!MmapFixedSuperNoReserve(ShadowAddr(), UnusedAddr() - ShadowAddr()))
     Die();
+  if (common_flags()->use_madv_dontdump)
+    DontDumpShadowMemory(ShadowAddr(), UnusedAddr() - ShadowAddr());
 
   // Protect the region of memory we don't use, to preserve the one-to-one
   // mapping from application to shadow memory. But if ASLR is disabled, Linux
   // will load our executable in the middle of our unused region. This mostly
   // works so long as the program doesn't use too much memory. We support this
   // case by disabling memory protection when ASLR is disabled.
   uptr init_addr = (uptr)&dfsan_init;
   if (!(init_addr >= UnusedAddr() && init_addr < AppAddr()))
     MmapFixedNoAccess(UnusedAddr(), AppAddr() - UnusedAddr());
 
   InitializeInterceptors();
 
   // Register the fini callback to run when the program terminates successfully
   // or it is killed by the runtime.
   Atexit(dfsan_fini);
   AddDieCallback(dfsan_fini);
 
   __dfsan_label_info[kInitializingLabel].desc = "<init label>";
 }
 
 #if SANITIZER_CAN_USE_PREINIT_ARRAY
 __attribute__((section(".preinit_array"), used))
 static void (*dfsan_init_ptr)(int, char **, char **) = dfsan_init;
 #endif

compiler-rt/test/dfsan/release_shadow_space.c

-// RUN: %clang_dfsan %s -o %t && %run %t
+// DFSAN_OPTIONS=no_huge_pages_for_shadow=false RUN: %clang_dfsan %s -o %t && %run %t
+// DFSAN_OPTIONS=no_huge_pages_for_shadow=true RUN: %clang_dfsan %s -o %t && %run %t
 
 #include <assert.h>
 #include <sanitizer/dfsan_interface.h>
 #include <stdbool.h>
 #include <stdio.h>
 #include <string.h>
 #include <sys/mman.h>
 #include <unistd.h>