[sanitizer] [asan] Use same shadow offset for aarch64

[zatrazz]

Summary

Following a discussion in gcc maillist for the libsanitizer merge I am now
trying to remove the SANITIZER_AARCH64_VMA flags for aarch64 port.

This patch makes ASAN for aarch64 use the same shadow offset for all
currently supported VMAs (39 and 42 bits). The shadow offset is the
same of 39-bit (36). To make it work on 42-bit llvm instrumentation will
require a change [1] on how to generate the memory address.

No regression tests found in 39 and 42-bit VMA. I have not tested on
48-bit VMA due lack of available platforms (but it is on my radar).

[1] http://reviews.llvm.org/D13781

[zatrazz]

Updated patch correcting the segments placement for 42-bit VMA.

[eugenis]

lib/sanitizer_common/sanitizer_platform.h

107–108

I'm pretty sure this won't work on 48 bit VMA.
Could you evaluate the cost of lifting the upper limit to 48? Last time I looked, this should only affect .bss size (so, no real RAM cost), and not really that much.

[zatrazz]

lib/sanitizer_common/sanitizer_platform.h

107–108

I won't, but I am also not sure if kAArch64_ShadowOffset64 = 1ULL << 36 will work correctly on 48-bit kernels. But it will check this out.

[zatrazz]

lib/sanitizer_common/sanitizer_platform.h

107–108

I did some analysis of the memory consumption difference about using 47 instead of 42 and the results look reasonable: with test/asan/TestCases/mmap_limit_mb.cc as base with some code to dump '/proc/self/statm' I see:

$ ./test_mem-{42,48} 2000 16

MMAP          42       48      Diff
Resident   21970    22110       140
          186924   187035       111
         1476712  1476841       129

So it seems an overhead not really dependent of total memory utilization for about 129-140 kb.

[eugenis]

Why is there a difference in resident memory?
I'd expect it to go to .bss.
This could be related to http://llvm.org/viewvc/llvm-project?rev=251717&view=rev

[zatrazz]

In D13782#279444, @eugenis wrote:

Why is there a difference in resident memory?
I'd expect it to go to .bss.
This could be related to http://llvm.org/viewvc/llvm-project?rev=251717&view=rev

I will recheck against master.

[zatrazz]

Remove SANITIZER_AARCH64_VMA on LSAN.

[zatrazz]

In D13782#279444, @eugenis wrote:

Why is there a difference in resident memory?
I'd expect it to go to .bss.
This could be related to http://llvm.org/viewvc/llvm-project?rev=251717&view=rev

Master (r252155) looks better:

MMAP          42       48      Diff
Resident   12989    12988        -1
          128228   128227        -1
         1280210  1280224       +14

I did not dig into to find exactly from where the difference came from when total usage is 1280210.

[eugenis]

LGTM, thanks!

lib/asan/asan_mapping.h

 static const u64 kDefaultShadowScale = 3;
 static const u64 kDefaultShadowOffset32 = 1ULL << 29;  // 0x20000000
 static const u64 kDefaultShadowOffset64 = 1ULL << 44;
 static const u64 kDefaultShort64bitShadowOffset = 0x7FFF8000;  // < 2G.
 static const u64 kIosShadowOffset32 = 1ULL << 30;  // 0x40000000
 static const u64 kIosShadowOffset64 = 0x130000000;
 static const u64 kIosSimShadowOffset32 = 1ULL << 30;
 static const u64 kIosSimShadowOffset64 = kDefaultShadowOffset64;
-#if SANITIZER_AARCH64_VMA == 39
 static const u64 kAArch64_ShadowOffset64 = 1ULL << 36;
-#elif SANITIZER_AARCH64_VMA == 42
-static const u64 kAArch64_ShadowOffset64 = 1ULL << 39;
-#endif
 static const u64 kMIPS32_ShadowOffset32 = 0x0aaa0000;
 static const u64 kMIPS64_ShadowOffset64 = 1ULL << 37;
 static const u64 kPPC64_ShadowOffset64 = 1ULL << 41;
 static const u64 kFreeBSD_ShadowOffset32 = 1ULL << 30;  // 0x40000000
 static const u64 kFreeBSD_ShadowOffset64 = 1ULL << 46;  // 0x400000000000
 static const u64 kWindowsShadowOffset32 = 3ULL << 28;  // 0x30000000
 
 #define SHADOW_SCALE kDefaultShadowScale

lib/asan/asan_rtl.cc

static void AsanInitInternal() {
   asan_init_is_running = true;
 
   CacheBinaryName();
 
   // Initialize flags. This must be done early, because most of the
   // initialization steps look at flags().
   InitializeFlags();
 
-  CheckVMASize();
-
   AsanCheckIncompatibleRT();
   AsanCheckDynamicRTPrereqs();
 
   SetCanPoisonMemory(flags()->poison_heap);
   SetMallocContextSize(common_flags()->malloc_context_size);
 
   InitializeHighMemEnd();
 

lib/lsan/lsan_common.cc

static inline bool CanBeAHeapPointer(uptr p) {
   const uptr kMinAddress = 4 * 4096;
   if (p < kMinAddress) return false;
 #if defined(__x86_64__)
   // Accept only canonical form user-space addresses.
   return ((p >> 47) == 0);
 #elif defined(__mips64)
   return ((p >> 40) == 0);
 #elif defined(__aarch64__)
-  return ((p >> SANITIZER_AARCH64_VMA) == 0);
+  unsigned runtimeVMA =
+    (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1);
+  return ((p >> runtimeVMA) == 0);
 #else
   return true;
 #endif
 }
 
 // Scans the memory range, looking for byte patterns that point into allocator
 // chunks. Marks those chunks with |tag| and adds them to |frontier|.
 // There are two usage modes for this function: finding reachable chunks

lib/sanitizer_common/sanitizer_platform.h

 #endif
 
 // By default we allow to use SizeClassAllocator64 on 64-bit platform.
 // But in some cases (e.g. AArch64's 39-bit address space) SizeClassAllocator64
 // does not work well and we need to fallback to SizeClassAllocator32.
 // For such platforms build this code with -DSANITIZER_CAN_USE_ALLOCATOR64=0 or
 // change the definition of SANITIZER_CAN_USE_ALLOCATOR64 here.
 #ifndef SANITIZER_CAN_USE_ALLOCATOR64
-# if defined(__mips64) || (defined(__aarch64__) && SANITIZER_AARCH64_VMA == 39)
+# if defined(__mips64) || defined(__aarch64__)
 #  define SANITIZER_CAN_USE_ALLOCATOR64 0
 # else
 #  define SANITIZER_CAN_USE_ALLOCATOR64 (SANITIZER_WORDSIZE == 64)
 # endif
 #endif
 
 // The range of addresses which can be returned my mmap.
-// FIXME: this value should be different on different platforms,
-// e.g. on AArch64 it is most likely (1ULL << 39). Larger values will still work
-// but will consume more memory for TwoLevelByteMap.
-#if defined(__aarch64__)
-# if SANITIZER_AARCH64_VMA == 39
-#  define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 39)
-# elif SANITIZER_AARCH64_VMA == 42
-#  define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 42)
-# endif
-#elif defined(__mips__)
+// FIXME: this value should be different on different platforms.  Larger values
+// will still work but will consume more memory for TwoLevelByteMap.
+#if defined(__mips__)
 # define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 40)
 #else
 # define SANITIZER_MMAP_RANGE_SIZE FIRST_32_SECOND_64(1ULL << 32, 1ULL << 47)
 #endif
 
 // The AArch64 linux port uses the canonical syscall set as mandated by
 // the upstream linux community for all new ports. Other ports may still
 // use legacy syscalls.