Add NetBSD ASAN shadow mapping for x86-64

[krytarowski]

Summary

The maximal virtual address on NetBSD/amd64 is 0x7f7ffffff000.
Define shadow offset 0x400000000000 (1ULL << 46).

Sponsored by <The NetBSD Foundation>

[krytarowski]

I don't really get why the shadow offset differs between systems, I copied it from FreeBSD.
Please help to optimize it.

Also there is a problem with X86 (32-bit). There are two possible options: 32-bit application on 32-bit kernel or on 64-bit kernel.

For 32-bit kernel the maximal user address:
0xfffff000
For 64-bit kernel the maximal user address of 32-bit application:
0xbfbff000

What would be the proper offset here?

[krytarowski]

Adding more reviewers to get feedback.

[joerg]

Just assume that the full 32bit address space is available.

[krytarowski]

ping x2

[fjricci]

I'm not familiar with this part of the codebase.

[krytarowski]

ping x3

This is the last functional patch for asan/NetBSD support in compiler-rt.

[kcc]

LGTM

[krytarowski]

I kindly ask for explanation of this shadow offset value. (My questions were on top, I see them hidden by Phabricator now)

But anyway, I will commit it now and keep it optimizing in future.

[eugenis]

It's just an address where a shadow-sized hole is most like to be found.
But there are other considerations. For example, 0x7FFF8000 is the highest (or close to highest) address that allows shorter instruction encoding on x86_64. Zero offset is typically not used (even though it is usually the fastest option) because it conflicts with MAP_32BIT.

[krytarowski]

In D36587#854657, @eugenis wrote:

It's just an address where a shadow-sized hole is most like to be found.
But there are other considerations. For example, 0x7FFF8000 is the highest (or close to highest) address that allows shorter instruction encoding on x86_64. Zero offset is typically not used (even though it is usually the fastest option) because it conflicts with MAP_32BIT.

MAP_32BIT is Linux/FreeBSD specific, it's absent in NetBSD... does it mean that the shadowoffset should be 0?

How about i386?

For 32-bit kernel the maximal user address:
0xfffff000
For 64-bit kernel the maximal user address of 32-bit application:
0xbfbff000

[eugenis]

Does NetBSD support non-PIE executables? Will they work with shadow at 0?

[krytarowski]

In D36587#854723, @eugenis wrote:

Does NetBSD support non-PIE executables? Will they work with shadow at 0?

NetBSD supports PIE and non-PIE executables. The non-PIE ones are the default option on x86_64.

I'm going to execute tests with shadow at 0.

While there waiting for build.. is PIE required for tsan? If so, why?

https://github.com/llvm-mirror/compiler-rt/blob/master/lib/tsan/rtl/tsan_platform_posix.cc#L34
https://github.com/llvm-mirror/compiler-rt/blob/master/lib/tsan/rtl/tsan_platform_posix.cc#L119

I need also enlightening how to translate struct Mapping to NetBSD

https://github.com/llvm-mirror/compiler-rt/blob/master/lib/tsan/rtl/tsan_platform_posix.cc#L119

[eugenis]

I'm not sure if that is the real reason, but TSan shadow is 8-to-1. It's hard to find an address that can fit such a big allocation in both PIE and non-PIE layout.

[krytarowski]

Unless I'm looking wrongly Clang in SanitizerArgs.cpp, does not mandate PIE for TSAN.

What determined the defined Linux/FreeBSD struct Mapping?

I will follow it analogously with NetBSD specific bits like MAP_32BIT not needed.

[eugenis]

Looking at the comment,
0000 0000 1000 - 0080 0000 0000: main binary and/or MAP_32BIT mappings (512GB)
5500 0000 0000 - 5680 0000 0000: pie binaries without ASLR or on 4.1+ kernels
7e80 0000 0000 - 8000 0000 0000: modules and main thread stack

^ these are the OS constraints, the rest can be moved.

Another constraint is MemToShadowImpl() which, for any address in, I assume, writable memory (so any of the above + heap) needs to provide an distinct address in "shadow". AFAIK, "metainfo" and "traces" are completely arbitrary.

[krytarowski]

I've checked that ShadowOffset 0 breaks ASAN.

In D36587#854807, @eugenis wrote:

Looking at the comment,
0000 0000 1000 - 0080 0000 0000: main binary and/or MAP_32BIT mappings (512GB)
5500 0000 0000 - 5680 0000 0000: pie binaries without ASLR or on 4.1+ kernels
7e80 0000 0000 - 8000 0000 0000: modules and main thread stack

^ these are the OS constraints, the rest can be moved.

I'm having now trouble to translate these values to NetBSD.

Another constraint is MemToShadowImpl() which, for any address in, I assume, writable memory (so any of the above + heap) needs to provide an distinct address in "shadow". AFAIK, "metainfo" and "traces" are completely arbitrary.

Another thread: If I understand correctly there is an assumption that PIE code is mapped on Linux/FreeBSD into adjacent memory regions. This is not true on NetBSD (@joerg can correct me).

I was trying to reuse Linux/FreeBSD process memory mapping and all tests fail. It breaks at PIE detection (I disabled it locally to check what happens).

[krytarowski]

After turning off PIE verification (Linux/FreeBSD specific?), I was getting random corruption and SIGSEGV.

lib/asan/asan_mapping.h

 //
 // Shadow mapping on FreeBSD/i386 with SHADOW_OFFSET == 0x40000000:
 // || `[0x60000000, 0xffffffff]` || HighMem    ||
 // || `[0x4c000000, 0x5fffffff]` || HighShadow ||
 // || `[0x48000000, 0x4bffffff]` || ShadowGap  ||
 // || `[0x40000000, 0x47ffffff]` || LowShadow  ||
 // || `[0x00000000, 0x3fffffff]` || LowMem     ||
 //
+// Shadow mapping on NetBSD/x86-64 with SHADOW_OFFSET == 0x400000000000:
+// || `[0x4feffffffe01, 0x7f7ffffff000]` || HighMem    ||
+// || `[0x49fdffffffc0, 0x4feffffffe00]` || HighShadow ||
+// || `[0x480000000000, 0x49fdffffffbf]` || ShadowGap  ||
+// || `[0x400000000000, 0x47ffffffffff]` || LowShadow  ||
+// || `[0x000000000000, 0x3fffffffffff]` || LowMem     ||
+//
 // Default Windows/i386 mapping:
 // (the exact location of HighShadow/HighMem may vary depending
 //  on WoW64, /LARGEADDRESSAWARE, etc).
 // || `[0x50000000, 0xffffffff]` || HighMem    ||
 // || `[0x3a000000, 0x4fffffff]` || HighShadow ||
 // || `[0x36000000, 0x39ffffff]` || ShadowGap  ||
 // || `[0x30000000, 0x35ffffff]` || LowShadow  ||
 // || `[0x00000000, 0x2fffffff]` || LowMem     ||

 static const u64 kIosSimShadowOffset64 = kDefaultShadowOffset64;
 static const u64 kAArch64_ShadowOffset64 = 1ULL << 36;
 static const u64 kMIPS32_ShadowOffset32 = 0x0aaa0000;
 static const u64 kMIPS64_ShadowOffset64 = 1ULL << 37;
 static const u64 kPPC64_ShadowOffset64 = 1ULL << 41;
 static const u64 kSystemZ_ShadowOffset64 = 1ULL << 52;
 static const u64 kFreeBSD_ShadowOffset32 = 1ULL << 30;  // 0x40000000
 static const u64 kFreeBSD_ShadowOffset64 = 1ULL << 46;  // 0x400000000000
+static const u64 kNetBSD_ShadowOffset64 = 1ULL << 46;  // 0x400000000000
 static const u64 kWindowsShadowOffset32 = 3ULL << 28;  // 0x30000000
 
 #define SHADOW_SCALE kDefaultShadowScale
 
 #if SANITIZER_FUCHSIA
 #  define SHADOW_OFFSET (0)
 #elif SANITIZER_WORDSIZE == 32
 #  if SANITIZER_ANDROID

 #  elif defined(__aarch64__)
 #    define SHADOW_OFFSET kAArch64_ShadowOffset64
 #  elif defined(__powerpc64__)
 #    define SHADOW_OFFSET kPPC64_ShadowOffset64
 #  elif defined(__s390x__)
 #    define SHADOW_OFFSET kSystemZ_ShadowOffset64
 #  elif SANITIZER_FREEBSD
 #    define SHADOW_OFFSET kFreeBSD_ShadowOffset64
+#  elif SANITIZER_NETBSD
+#    define SHADOW_OFFSET kNetBSD_ShadowOffset64
 #  elif SANITIZER_MAC
 #   define SHADOW_OFFSET kDefaultShadowOffset64
 #  elif defined(__mips64)
 #   define SHADOW_OFFSET kMIPS64_ShadowOffset64
 #  elif SANITIZER_WINDOWS64
 #   define SHADOW_OFFSET __asan_shadow_memory_dynamic_address
 #  else
 #   define SHADOW_OFFSET kDefaultShort64bitShadowOffset