[compiler-rt] [tsan] Unify aarch64 mapping

[zatrazz]

Summary

This patch unify the 39 and 42-bit VMA support for AArch64 by using indirect
calls instead of inline the address function transformation. The already
defined maps for both VMA are used. Although slower, this leads to same
instrumented binary to be independent of the kernel. It also has the advantage
to be easier to enable the remaining VMA for AArch64 (48-bits).

Along with this change this patch also fix some 42-bit failures with
ALSR disable by increasing the upper high app memory threshold and also
the 42-bit madvise value for non large page set.

Regarding performance, I did a run with speccpu2006 using 'test' set (mainly
to evaluate faster) and the results shows the new indirect calls are roughly
4.6% slower than the inline version:

                 DIFF
401.bzip2        5.12 
403.gcc          5.24 
429.mcf          1.00 
445.gobmk        3.36 
456.hmmer        6.72 
458.sjeng        6.15 
462.libquantum   4.55 
464.h264ref      6.82 
473.astar        5.68 
483.xalancbmk    6.77 
GEOMEAN          4.62

PS: 400.perlbench fails due missing longjmp family instrumentation for aarch64 and 471.omnetpp
the know issue with user defined new/delete operators.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

153

You introduce both indirect functions and these non-const variables. This looks excessive. Let's do one or another. I would prefer global variables as it looks less intrusive.

You can do something along the lines of:

Here declare the consts:

extern uptr kLoAppMemBeg;
...

And then in InitializePlatformEarly initialize them:

kLoAppMemEnd = vma_39_42(0x0000400000ull, 0x00000400000ull);

And that's it.

169

All identifiers with double underscores are reserved for language implementation. TSAN_ is a fine prefix, and also consistent with all macros in the codebase.

383

Please rename this to InitializePlatformEarly.
I don't understand what is "Modules" (dynamic libraries?). And "Specific" is excessive, "Platform" already implies platform-specific stuff.

[zatrazz]

Changes from previous version:

• Increase the VMA range for application on both 39-bit and 42-bit VMA. Now 39-bit ranges from 0x7D00000000-0x7FFFFFFFFF (previously 0x7F00000000-0x7FFFFFFFFF) and 42-bits from 0x3f000000000-0x3FFFFFFFFFF (previously 0x3FF00000000-0x3FFFFFFFFFF);
• Fixed a printf format specifier in test/tsan/test.h.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

153

I declared them in the header because I used the constants in the function definitions at lib/tsan/rtl/tsan_platform_linux.cc. I can move them to the cc file if it desirable, but I think by removing the definitions will require to hardcode its value on the function definition somehow (and I think current approach in simple than replicate the hexadecimals values in some places).

Also, I use them so to add another VMA (48-bits) would not require much work. Instead of adding now 'vma_39_42' and then moving it to 'vma_39_42_48', the patch will just require to check and add the 48-bits definitions.

169

I will change it.

383

I will change it (I used 'modules' and 'specific' because I could not came up with a better naming).

[zatrazz]

This is an updated version based on comments:

• Moved aarch64 const VMA definition from platform header to platform file
• Rename InitializePlatformInitModules to InitializePlatformEarly

[dvyukov]

lib/tsan/rtl/tsan_platform.h

185

But can we have just global vars? It seems to be enough. The same generic functions will just operate with different values stores in the vars.

I am just frighten by the size of this change, amount of logical indirections and macros.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

185

We can, but it will be slower: every function transformation will require not only an indirection but also potentially memory reads to actually get the global variables value. The idea of the patch to create multiple functions is to try to minimize the performance hit this same indirection occurs by creating functions is fast as possible.

[zatrazz]

Small update on previous patch to initialize kMadviseRangeBeg and kMadviseRangeSize.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

187

Won't it then be faster to inline all 3 variants?

E.g.

struct Mapping39 {
  static const uptr kLoAppMemBeg = 0x...;
  static const uptr kLoAppMemEnd = 0x...;
  static const uptr kShadowBeg = 0x...;
  ...
};

struct Mapping42 {
  static const uptr kLoAppMemBeg = 0x...;
  static const uptr kLoAppMemEnd = 0x...;
  static const uptr kShadowBeg = 0x...;
  ...
};

template<typename Mapping>
uptr MemToShadowImpl(uptr x) {
  return (((x) & ~(Mapping::kAppMemMsk | (kShadowCell - 1)))
      ^ Mapping::kAppMemXor) * kShadowCnt;
}

uptr MemToShadow(uptr x) {
  if (vma_size == 39)
    return MemToShadowImpl<Mapping39>(x);
  else
    return MemToShadowImpl<Mapping42>(x);
  DCHECK(0);
}

I suspect it can be faster as it is inlinable. And probably compiler will be able to do some common expressions merging, so the code won't be too large.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

187

I will check this out, thanks!

[zatrazz]

The mapping using inline function is indeed slight faster than the previous attempt (I used the same benchmark as before, speccpu2006 test size). Changes from previous version:

• Use a different strategy for mapping functions: instead of indirect functions inline version based on an external VMA detected at runtime.
• Fixed the 39-bit shadow memory upper bound.

[dvyukov]

Much nicer now. Thanks.

Do we still need these extern vars in header
extern uptr kLoAppMemBeg
?
I would prefer if we get rid of any remaining usages of these and introduce more functions instead if needed.
They introduce significant amount of boilerplate. And also they are not necessary consts now, so it can be dangerous (performance-wise) to use them occasionally on some performance-critical path.

Also we now have duplication of complex code between MemToShadow for x86 and MemToShadowImpl for power.
Please move x86 consts to a Mapping struct as well. Then we can have a single MemToShadowImpl implementation and then define MemToShadow as:

uptr MemToShadowImpl(uptr x) {
#ifdef power

if (vmaSize == 39)
  return MemToShadowImpl<Mapping39>(x);
else
  return MemToShadowImpl<Mapping42>(x);

#else

return MemToShadowImpl<Mapping>(x);

#endif
}

[zatrazz]

Changes from previous patch:

• Unify all implementations using the Mapping scheme.
• Remove redundant code for go implementation by unify windows and non-windows transformation.
• All accesses to mapping const definition are done through functions now. For architecture with only one mapping this won't change code generation.

For architecture with static mapping I had to still define kHeapMemBeg and kHeapMemEnd because they are used on SizeClassAllocator64 definition. I think a possible cleanup is possible (prob when I adjust tsan for aarch64 48-bit VMA).

[zatrazz]

Ping.

[dvyukov]

Sorry for long delays.
I've looked at it several times and it still feels somewhat awkward, but at the same time I cannot come up with something better (we could remove lots of boilerplate and replication with advanced C++ templates, but I doubt it worth it, advanced C++ templates is an issue in itself). So probably some amount of duplication and boilerplate here is the way to go.
I will take a final look today.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

285

UserRegions are used only once during startup, there is no point in having both const and dynamic versions of it. Also you initialize the dynamic variable only on linux, but it actually needs to be initialized when TSAN_RUNTIME_VMA (e.g. on mac/aarch64 in future).

Please leave only one version and define it in this header.
Something along the lines of:

bool GetUserRegion(int i, uptr *start, uptr *end) {

switch (i) {
default:
  return false;
case 0:
  *start = LoAppMemBeg();
  *end = LoAppMemEnd();
case 1:
  ...
case 2:
  ...
}

}

And loop in CheckShadowMapping until this function returns false.

333

Why !SANITIZER_WINDOWS?
There is also no windows-Go-specific code, so these definitions must be necessary either on all Go platforms or on no Go platforms.

lib/tsan/rtl/tsan_platform_posix.cc

50 ↗

(On Diff #39903)

vmaSize should be already set by now, so use it.
Add

} else {
  CHECK(0);
}

branch below. It will catch both case when vmaSize is somehow not initialized by now and case when support for a new vma size is added.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

285

I will change that.

333

This is wrong, I will fix it.

lib/tsan/rtl/tsan_platform_posix.cc

50 ↗

(On Diff #39903)

I will change that.

[zatrazz]

Changes from previous version:

• Remove UserRegions struct and change the interface to call GetUserRegion.
• Remove a wrong windows SANITIZER_GO define
• Use vmaSize on InitializeShadowMemory for aarch64.

[dvyukov]

LGTM with nits

sorry for delays and thanks for bearing with me

lib/tsan/rtl/tsan_platform.h

276

drop !SANITIZER_WINDOWS part

309

#ifndef SANITIZER_GO
that's more common throughout tsan

317

#else

lib/tsan/rtl/tsan_rtl.cc

293

please replace this with:

uptr beg, end;
for (int i = 0; GetUserRegion(i, &beg, &end); i++) {

it is shorter and more idiomatic for loops.

[eugenis]

lib/tsan/rtl/tsan_platform.h

123

I've noticed that this mapping does not have an application region at 0x55...
That's where PIE executables are mapped by the newer kernels.
Did you test this on an older kernel (I don't know exactly when the transition happened; it was around 4.1 for X86 and AFAIK a bit earlier for AArch 64)?

See http://reviews.llvm.org/D15308 for more context.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

123

I did check the PIE executable with a full TSAN test built with '-pie -fpic' and I saw no issue. Analyzing the trace for 39-bits using the mutexset1.cc (I picked it at random) I see:

• Non-pie build:

00400000-004d1000 r-xp 00000000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
004e0000-004e3000 r--p 000d0000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
004e3000-004e6000 rw-p 000d3000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1

• PIE build:

7fae119000-7fae1ee000 r-xp 00000000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
7fae1ee000-7fae1fb000 rw-p 00000000 00:00 0
7fae1fd000-7fae200000 r--p 000d4000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
7fae200000-7fae203000 rw-p 000d7000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1

So at least for 39-bit PIE addresses are being covered. I will check again on 42-bit VMA.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

123

I and I checked on a fairly recent kernel, 3.19. I am not aware if the mappings for PIE has changed in recent kernels, neither I checked on recent ones. Do you have more information?

[eugenis]

lib/tsan/rtl/tsan_platform.h

123

See this bug for the linux 4.1.2 problem with MSan:
https://llvm.org/bugs/show_bug.cgi?id=24155

I see 0x55... mappings with 3.10.40 kernel on Android/AAarch64.

git tag --contains in the torvalds repo tells me that this change appears in v4.1 and newer. Android must have cherry-picked it in 3.10.

What happens if you disable ASLR? We used to get the main executable at 0x55... under gdb (i.e. with disabled randomization) long before 3.9.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

123

My previous analysis was in fact wrong, even for the kernels I am currently using (3.19 for 39-bit and 3.17 for 42-bit I am seeing):

• 0x00000000000-0x00010000000: 39/42-bits program own segments
• 0x05500000000-0x05600000000: 39-bits PIE program segments
• 0x07f80000000-0x07fffffffff: 39-bits libraries segments
• 0x2aa00000000-0x2ab00000000: 42-bits PIE program segments
• 0x3ff00000000-0x3ffffffffff: 42-bits libraries segments

Fortunately I could adjust aarch64 mapping to include these segments using the current mapping. I am preparing a patch.

lib/tsan/rtl/tsan_platform.h

 const uptr kLoAppMemBeg   = 0x0100000000ull;
 const uptr kLoAppMemEnd   = 0x0200000000ull;
 const uptr kHiAppMemBeg   = 0xff80000000ull;
 const uptr kHiAppMemEnd   = 0xffffffffffull;
 const uptr kAppMemMsk     = 0xfc00000000ull;
 const uptr kAppMemXor     = 0x0400000000ull;
 const uptr kVdsoBeg       = 0xfffff00000ull;
 #elif defined(__aarch64__)
-# if SANITIZER_AARCH64_VMA == 39
+// AArch64 supports multiple VMA which leads to multiple address transformation
+// functions.  To support these multiple transformations and mappings the TSAN
+// runtime for AArch64 uses indirect calls for the memory to/from shadow,
+// traces, etc segments.  Although slower, it make a same instrumented binary
+// run on multiple kernels.
+
 /*
 C/C++ on linux/aarch64 (39-bit VMA)
 0000 4000 00 - 0200 0000 00: main binary
 2000 0000 00 - 4000 0000 00: shadow memory
 4000 0000 00 - 5000 0000 00: metainfo
 5000 0000 00 - 6000 0000 00: -
 6000 0000 00 - 6200 0000 00: traces
 6200 0000 00 - 7d00 0000 00: -
 7d00 0000 00 - 7e00 0000 00: heap
 7e00 0000 00 - 7fff ffff ff: modules and main thread stack
 */
-const uptr kLoAppMemBeg   = 0x0000400000ull;
-const uptr kLoAppMemEnd   = 0x0200000000ull;
-const uptr kShadowBeg     = 0x2000000000ull;
-const uptr kShadowEnd     = 0x4000000000ull;
-const uptr kMetaShadowBeg = 0x4000000000ull;
-const uptr kMetaShadowEnd = 0x5000000000ull;
-const uptr kTraceMemBeg   = 0x6000000000ull;
-const uptr kTraceMemEnd   = 0x6200000000ull;
-const uptr kHeapMemBeg    = 0x7d00000000ull;
-const uptr kHeapMemEnd    = 0x7e00000000ull;
-const uptr kHiAppMemBeg   = 0x7e00000000ull;
-const uptr kHiAppMemEnd   = 0x7fffffffffull;
-const uptr kAppMemMsk     = 0x7800000000ull;
-const uptr kAppMemXor     = 0x0800000000ull;
-const uptr kVdsoBeg       = 0x7f00000000ull;
-# elif SANITIZER_AARCH64_VMA == 42
+const uptr kLoAppMemBeg39   = 0x0000400000ull;
+const uptr kLoAppMemEnd39   = 0x0200000000ull;
+const uptr kShadowBeg39     = 0x2000000000ull;
+const uptr kShadowEnd39     = 0x4000000000ull;
+const uptr kMetaShadowBeg39 = 0x4000000000ull;
+const uptr kMetaShadowEnd39 = 0x5000000000ull;
+const uptr kTraceMemBeg39   = 0x6000000000ull;
+const uptr kTraceMemEnd39   = 0x6200000000ull;
+const uptr kHeapMemBeg39    = 0x7d00000000ull;
+const uptr kHeapMemEnd39    = 0x7e00000000ull;
+const uptr kHiAppMemBeg39   = 0x7e00000000ull;
+const uptr kHiAppMemEnd39   = 0x7fffffffffull;
+const uptr kAppMemMsk39     = 0x7800000000ull;
+const uptr kAppMemXor39     = 0x0800000000ull;
+const uptr kVdsoBeg39       = 0x7f00000000ull;
+
 /*
 C/C++ on linux/aarch64 (42-bit VMA)
 00000 4000 00 - 01000 0000 00: main binary
 01000 0000 00 - 10000 0000 00: -
 10000 0000 00 - 20000 0000 00: shadow memory
 20000 0000 00 - 26000 0000 00: -
 26000 0000 00 - 28000 0000 00: metainfo
 28000 0000 00 - 36200 0000 00: -
 36200 0000 00 - 36240 0000 00: traces
 36240 0000 00 - 3e000 0000 00: -
 3e000 0000 00 - 3f000 0000 00: heap
 3c000 0000 00 - 3ff00 0000 00: -
-3ff00 0000 00 - 3ffff f000 00: modules and main thread stack
+3ff00 0000 00 - 3ffff ffff ff: modules and main thread stack
 */
-const uptr kLoAppMemBeg   = 0x00000400000ull;
-const uptr kLoAppMemEnd   = 0x01000000000ull;
-const uptr kShadowBeg     = 0x10000000000ull;
-const uptr kShadowEnd     = 0x20000000000ull;
-const uptr kMetaShadowBeg = 0x26000000000ull;
-const uptr kMetaShadowEnd = 0x28000000000ull;
-const uptr kTraceMemBeg   = 0x36200000000ull;
-const uptr kTraceMemEnd   = 0x36400000000ull;
-const uptr kHeapMemBeg    = 0x3e000000000ull;
-const uptr kHeapMemEnd    = 0x3f000000000ull;
-const uptr kHiAppMemBeg   = 0x3ff00000000ull;
-const uptr kHiAppMemEnd   = 0x3fffff00000ull;
-const uptr kAppMemMsk     = 0x3c000000000ull;
-const uptr kAppMemXor     = 0x04000000000ull;
-const uptr kVdsoBeg       = 0x37f00000000ull;
-# endif
+const uptr kLoAppMemBeg42   = 0x00000400000ull;
+const uptr kLoAppMemEnd42   = 0x01000000000ull;
+const uptr kShadowBeg42     = 0x10000000000ull;
+const uptr kShadowEnd42     = 0x20000000000ull;
+const uptr kMetaShadowBeg42 = 0x26000000000ull;
+const uptr kMetaShadowEnd42 = 0x28000000000ull;
+const uptr kTraceMemBeg42   = 0x36200000000ull;
+const uptr kTraceMemEnd42   = 0x36400000000ull;
+const uptr kHeapMemBeg42    = 0x3e000000000ull;
+const uptr kHeapMemEnd42    = 0x3f000000000ull;
+const uptr kHiAppMemBeg42   = 0x3ff00000000ull;
+const uptr kHiAppMemEnd42   = 0x3ffffffffffull;
+const uptr kAppMemMsk42     = 0x3c000000000ull;
+const uptr kAppMemXor42     = 0x04000000000ull;
+const uptr kVdsoBeg42       = 0x37f00000000ull;
+
+extern uptr kLoAppMemBeg;
+extern uptr kLoAppMemEnd;
+extern uptr kShadowBeg;
+extern uptr kShadowEnd;
+extern uptr kMetaShadowBeg;
+extern uptr kMetaShadowEnd;
+extern uptr kTraceMemBeg;
+extern uptr kTraceMemEnd;
+extern uptr kHeapMemBeg;
+extern uptr kHeapMemEnd;
+extern uptr kHiAppMemBeg;
+extern uptr kHiAppMemEnd;
+extern uptr kAppMemMsk;
+extern uptr kAppMemXor;
+extern uptr kVdsoBeg;
+
+#define __TSAN_NOINLINE_FUNCTIONS 1
 #endif
 
+#ifdef __TSAN_NOINLINE_FUNCTIONS
+// For non-inline function indirect calls are used to provide diferent
+// mappings strategies depending of runtime (for instance, on AArch64
+// with multiple VMA different mappings are used).
+extern bool (*IsAppMem)(uptr mem);
+extern bool (*IsShadowMem)(uptr mem);
+extern bool (*IsMetaMem)(uptr mem);
+extern uptr (*MemToShadow)(uptr x);
+extern u32* (*MemToMeta)(uptr x);
+extern uptr (*ShadowToMem)(uptr s);
+extern uptr *UserRegions;
+#else
 ALWAYS_INLINE
 bool IsAppMem(uptr mem) {
   return (mem >= kHeapMemBeg && mem < kHeapMemEnd) ||
          (mem >= kLoAppMemBeg && mem < kLoAppMemEnd) ||
          (mem >= kHiAppMemBeg && mem < kHiAppMemEnd);
 }
 
 ALWAYS_INLINE
 bool IsShadowMem(uptr mem) {
   return mem >= kShadowBeg && mem <= kShadowEnd;
 }
 
else
     return ((s / kShadowCnt) ^ kAppMemXor) | kAppMemMsk;
 }
 
 static USED uptr UserRegions[] = {
   kLoAppMemBeg, kLoAppMemEnd,
   kHiAppMemBeg, kHiAppMemEnd,
   kHeapMemBeg,  kHeapMemEnd,
 };
+#endif
 
 #elif defined(SANITIZER_GO) && !SANITIZER_WINDOWS
 
 /* Go on linux, darwin and freebsd
 0000 0000 1000 - 0000 1000 0000: executable
 0000 1000 0000 - 00c0 0000 0000: -
 00c0 0000 0000 - 00e0 0000 0000: heap
 00e0 0000 0000 - 2000 0000 0000: -

 }
 
 ALWAYS_INLINE
 uptr MemToShadow(uptr x) {
   DCHECK(IsAppMem(x));
   return ((x & ~(kShadowCell - 1)) * kShadowCnt) | kShadowBeg;
 }
 
 ALWAYS_INLINE
 u32 *MemToMeta(uptr x) {
   DCHECK(IsAppMem(x));
   return (u32*)(((x & ~(kMetaShadowCell - 1)) / \
       kMetaShadowCell * kMetaShadowSize) | kMetaShadowBeg);
 }
 
 ALWAYS_INLINE
 uptr ShadowToMem(uptr s) {
   CHECK(IsShadowMem(s));
   return (s & ~kShadowBeg) / kShadowCnt;
 }
 
 static USED uptr UserRegions[] = {
   kAppMemBeg, kAppMemEnd,
 };
 
 #elif defined(SANITIZER_GO) && SANITIZER_WINDOWS
 
 /* Go on windows
 0000 0000 1000 - 0000 1000 0000: executable
 0000 1000 0000 - 00f8 0000 0000: -
 00c0 0000 0000 - 00e0 0000 0000: heap
 00e0 0000 0000 - 0100 0000 0000: -
 0100 0000 0000 - 0500 0000 0000: shadow
 0500 0000 0000 - 0560 0000 0000: -
 0560 0000 0000 - 0760 0000 0000: traces
 0760 0000 0000 - 07d0 0000 0000: metainfo (memory blocks and sync objects)
 07d0 0000 0000 - 8000 0000 0000: -
 */
 
 const uptr kMetaShadowBeg = 0x076000000000ull;
 const uptr kMetaShadowEnd = 0x07d000000000ull;
 const uptr kTraceMemBeg   = 0x056000000000ull;
 const uptr kTraceMemEnd   = 0x076000000000ull;
 const uptr kShadowBeg     = 0x010000000000ull;
 const uptr kShadowEnd     = 0x050000000000ull;
 const uptr kAppMemBeg     = 0x000000001000ull;
 const uptr kAppMemEnd     = 0x00e000000000ull;
 
 ALWAYS_INLINE
 bool IsAppMem(uptr mem) {
   return mem >= kAppMemBeg && mem < kAppMemEnd;
 }
 
 ALWAYS_INLINE
 bool IsShadowMem(uptr mem) {
   return mem >= kShadowBeg && mem <= kShadowEnd;
 }
 
 ALWAYS_INLINE
 bool IsMetaMem(uptr mem) {
   return mem >= kMetaShadowBeg && mem <= kMetaShadowEnd;
 }
 
 ALWAYS_INLINE
 uptr MemToShadow(uptr x) {
   DCHECK(IsAppMem(x));
   return ((x & ~(kShadowCell - 1)) * kShadowCnt) + kShadowBeg;
 }
 
 ALWAYS_INLINE
 u32 *MemToMeta(uptr x) {
   DCHECK(IsAppMem(x));
   return (u32*)(((x & ~(kMetaShadowCell - 1)) / \
       kMetaShadowCell * kMetaShadowSize) | kMetaShadowBeg);

 # error "Unknown platform"
 #endif
 
 // The additional page is to catch shadow stack overflow as paging fault.
 // Windows wants 64K alignment for mmaps.
 const uptr kTotalTraceSize = (kTraceSize * sizeof(Event) + sizeof(Trace)
     + (64 << 10) + (64 << 10) - 1) & ~((64 << 10) - 1);
 
+#ifdef __TSAN_NOINLINE_FUNCTIONS
+extern uptr (*GetThreadTrace)(int tid);
+extern uptr (*GetThreadTraceHeader)(int tid);
+#else
 uptr ALWAYS_INLINE GetThreadTrace(int tid) {
   uptr p = kTraceMemBeg + (uptr)tid * kTotalTraceSize;
   DCHECK_LT(p, kTraceMemEnd);
   return p;
 }
 
 uptr ALWAYS_INLINE GetThreadTraceHeader(int tid) {
   uptr p = kTraceMemBeg + (uptr)tid * kTotalTraceSize
       + kTraceSize * sizeof(Event);
   DCHECK_LT(p, kTraceMemEnd);
   return p;
 }
+#endif
 
 void InitializePlatform();
+void InitializePlatformSpecificModules();
 void FlushShadowMemory();
 void WriteMemoryProfile(char *buf, uptr buf_size, uptr nthread, uptr nlive);
 
 // Says whether the addr relates to a global var.
 // Guesses with high probability, may yield both false positives and negatives.
 bool IsGlobalVar(uptr addr);
 int ExtractResolvFDs(void *state, int *fds, int nfd);
 int ExtractRecvmsgFDs(void *msg, int *fds, int nfd);
 
 int call_pthread_cancel_with_cleanup(int(*fn)(void *c, void *m,
     void *abstime), void *c, void *m, void *abstime,
     void(*cleanup)(void *arg), void *arg);
 
 }  // namespace __tsan
 
 #endif  // TSAN_PLATFORM_H

lib/tsan/rtl/tsan_platform_linux.cc

 void *__libc_stack_end = 0;
 #endif
 
 namespace __tsan {
 
 static uptr g_data_start;
 static uptr g_data_end;
 
+#if !defined(SANITIZER_GO) && defined(__aarch64__)
+uptr kLoAppMemBeg;
+uptr kLoAppMemEnd;
+uptr kShadowBeg;
+uptr kShadowEnd;
+uptr kMetaShadowBeg;
+uptr kMetaShadowEnd;
+uptr kTraceMemBeg;
+uptr kTraceMemEnd;
+uptr kHeapMemBeg;
+uptr kHeapMemEnd;
+uptr kHiAppMemBeg;
+uptr kHiAppMemEnd;
+uptr kAppMemMsk;
+uptr kAppMemXor;
+uptr kVdsoBeg;
+
+bool (*IsAppMem)(uptr mem);
+bool (*IsShadowMem)(uptr mem);
+bool (*IsMetaMem)(uptr mem);
+uptr (*MemToShadow)(uptr x);
+u32* (*MemToMeta)(uptr x);
+uptr (*ShadowToMem)(uptr s);
+uptr *UserRegions;
+
+#define __DECLARE_IS_APP_MEM(__vma) \
+bool IsAppMem##__vma(uptr mem) { \
+  return (mem >= kHeapMemBeg##__vma && mem < kHeapMemEnd##__vma) || \
+         (mem >= kLoAppMemBeg##__vma && mem < kLoAppMemEnd##__vma) || \
+         (mem >= kHiAppMemBeg##__vma && mem < kHiAppMemEnd##__vma); \
+}
+
+__DECLARE_IS_APP_MEM(39)
+__DECLARE_IS_APP_MEM(42)
+
+#define __DECLARE_IS_SHADOW_MEM(__vma) \
+bool IsShadowMem##__vma(uptr mem) { \
+  return mem >= kShadowBeg##__vma && mem <= kShadowEnd##__vma; \
+}
+
+__DECLARE_IS_SHADOW_MEM(39)
+__DECLARE_IS_SHADOW_MEM(42)
+
+#define __DECLARE_IS_META_MEM(__vma) \
+bool IsMetaMem##__vma(uptr mem) { \
+  return mem >= kMetaShadowBeg##__vma && mem <= kMetaShadowEnd##__vma; \
+}
+
+__DECLARE_IS_META_MEM(39)
+__DECLARE_IS_META_MEM(42)
+
+#define __DECLARE_MEM_TO_SHADOW(__vma) \
+uptr MemToShadow##__vma(uptr x) { \
+  DCHECK(IsAppMem##__vma(x)); \
+  return (((x) & ~(kAppMemMsk##__vma | (kShadowCell - 1))) \
+      ^ kAppMemXor##__vma) * kShadowCnt; \
+}
+
+__DECLARE_MEM_TO_SHADOW(39)
+__DECLARE_MEM_TO_SHADOW(42)
+
+#define __DECLARE_MEM_TO_META(__vma) \
+u32 *MemToMeta##__vma(uptr x) { \
+  DCHECK(IsAppMem##__vma(x)); \
+  return (u32*)(((((x) & ~(kAppMemMsk##__vma | (kMetaShadowCell - 1))) \
+      ^ kAppMemXor##__vma) / kMetaShadowCell * kMetaShadowSize) \
+      | kMetaShadowBeg##__vma); \
+}
+
+__DECLARE_MEM_TO_META(39)
+__DECLARE_MEM_TO_META(42)
+
+#define __DECLARE_SHADOW_TO_MEM(__vma) \
+uptr ShadowToMem##__vma(uptr s) { \
+  CHECK(IsShadowMem##__vma(s)); \
+  if (s >= MemToShadow##__vma(kLoAppMemBeg##__vma) \
+      && s <= MemToShadow##__vma(kLoAppMemEnd##__vma - 1)) \
+    return (s / kShadowCnt) ^ kAppMemXor##__vma; \
+  return ((s / kShadowCnt) ^ kAppMemXor##__vma) | kAppMemMsk##__vma; \
+}
+
+__DECLARE_SHADOW_TO_MEM(39)
+__DECLARE_SHADOW_TO_MEM(42)
+
+#define __DECLARE_USER_REGIONS(__vma) \
+static USED uptr UserRegions##__vma[] = { \
+  kLoAppMemBeg##__vma, kLoAppMemEnd##__vma, \
+  kHiAppMemBeg##__vma, kHiAppMemEnd##__vma, \
+  kHeapMemBeg##__vma,  kHeapMemEnd##__vma, \
+};
+
+__DECLARE_USER_REGIONS(39)
+__DECLARE_USER_REGIONS(42)
+
+uptr (*GetThreadTrace)(int tid);
+uptr (*GetThreadTraceHeader)(int tid);
+
+#define __DECLARE_GET_THREAD_TRACE(__vma) \
+uptr GetThreadTrace##__vma(int tid) { \
+  uptr p = kTraceMemBeg##__vma + (uptr)tid * kTotalTraceSize; \
+  DCHECK_LT(p, kTraceMemEnd##__vma); \
+  return p; \
+}
+
+__DECLARE_GET_THREAD_TRACE(39)
+__DECLARE_GET_THREAD_TRACE(42)
+
+#define __DECLARE_GET_THREAD_TRACE_HEADER(__vma) \
+uptr GetThreadTraceHeader##__vma(int tid) { \
+  uptr p = kTraceMemBeg##__vma + (uptr)tid * kTotalTraceSize \
+      + kTraceSize * sizeof(Event); \
+  DCHECK_LT(p, kTraceMemEnd##__vma); \
+  return p; \
+}
+
+__DECLARE_GET_THREAD_TRACE_HEADER(39)
+__DECLARE_GET_THREAD_TRACE_HEADER(42)
+#endif
+
 enum {
   MemTotal  = 0,
   MemShadow = 1,
   MemMeta   = 2,
   MemFile   = 3,
   MemMmap   = 4,
   MemTrace  = 5,
   MemHeap   = 6,
void InitializeShadowMemory() {
   // FIXME: don't use constants here.
 #if defined(__x86_64__)
   const uptr kMadviseRangeBeg  = 0x7f0000000000ull;
   const uptr kMadviseRangeSize = 0x010000000000ull;
 #elif defined(__mips64)
   const uptr kMadviseRangeBeg  = 0xff00000000ull;
   const uptr kMadviseRangeSize = 0x0100000000ull;
 #elif defined(__aarch64__)
-  const uptr kMadviseRangeBeg  = 0x7e00000000ull;
-  const uptr kMadviseRangeSize = 0x0100000000ull;
+  unsigned runtimeVMA =
+    (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1);
+  uptr kMadviseRangeBeg;
+  uptr kMadviseRangeSize;
+  if (runtimeVMA == 39) {
+    kMadviseRangeBeg  = 0x7e00000000ull;
+    kMadviseRangeSize = 0x0100000000ull;
+  } else if (runtimeVMA == 42) {
+    kMadviseRangeBeg  = 0x3ff00000000ull;
+    kMadviseRangeSize = 0x0100000000ull;
+  }
 #endif
   NoHugePagesInRegion(MemToShadow(kMadviseRangeBeg),
                       kMadviseRangeSize * kShadowMultiplier);
   // Meta shadow is compressing and we don't flush it,
   // so it makes sense to mark it as NOHUGEPAGE to not over-allocate memory.
   // On one program it reduces memory consumption from 5GB to 2.5GB.
   NoHugePagesInRegion(kMetaShadowBeg, kMetaShadowEnd - kMetaShadowBeg);
   if (common_flags()->use_madv_dontdump)
static void CheckAndProtect() {
   // Memory for traces is mapped lazily in MapThreadTrace.
   // Protect the whole range for now, so that user does not map something here.
   ProtectRange(kTraceMemBeg, kTraceMemEnd);
   ProtectRange(kTraceMemEnd, kHeapMemBeg);
   ProtectRange(HeapEnd(), kHiAppMemBeg);
 }
 #endif  // #ifndef SANITIZER_GO
 
+void InitializePlatformSpecificModules() {
+#if !defined(SANITIZER_GO) && defined(__aarch64__)
+
+# define __INITIALIZE_MAPPING_VALUES(__vma) \
+    kLoAppMemBeg = kLoAppMemBeg##__vma;     \
+    kLoAppMemEnd = kLoAppMemEnd##__vma;     \
+    kShadowBeg = kShadowBeg##__vma;         \
+    kShadowEnd = kShadowEnd##__vma;         \
+    kMetaShadowBeg = kMetaShadowBeg##__vma; \
+    kMetaShadowEnd = kMetaShadowEnd##__vma; \
+    kTraceMemBeg = kTraceMemBeg##__vma;     \
+    kTraceMemEnd = kTraceMemEnd##__vma;     \
+    kHeapMemBeg = kHeapMemEnd##__vma;       \
+    kHeapMemEnd = kHeapMemEnd##__vma;       \
+    kHiAppMemBeg = kHiAppMemBeg##__vma;     \
+    kHiAppMemEnd = kHiAppMemEnd##__vma;     \
+    kAppMemMsk = kAppMemMsk##__vma;         \
+    kAppMemXor = kAppMemXor##__vma;         \
+    kVdsoBeg = kVdsoBeg##__vma;
+
+# define __INITIALIZE_MAPPING_FUNCTIONS(__vma) \
+    IsAppMem = IsAppMem##__vma;                \
+    IsShadowMem = IsShadowMem##__vma;          \
+    IsMetaMem = IsMetaMem##__vma;              \
+    MemToShadow = MemToShadow##__vma;          \
+    MemToMeta = MemToMeta##__vma;              \
+    ShadowToMem = ShadowToMem##__vma;          \
+    UserRegions = UserRegions##__vma;          \
+    GetThreadTrace = GetThreadTrace##__vma;    \
+    GetThreadTraceHeader = GetThreadTraceHeader##__vma;
+
+  unsigned runtimeVMA =
+    (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1);
+  if (runtimeVMA == 39) {
+    __INITIALIZE_MAPPING_VALUES(39);
+    __INITIALIZE_MAPPING_FUNCTIONS(39);
+  } else if (runtimeVMA == 42) {
+    __INITIALIZE_MAPPING_VALUES(42);
+    __INITIALIZE_MAPPING_FUNCTIONS(42);
+  } else {
+    Printf("FATAL: ThreadSanitizer: unsupported VMA range\n");
+    Printf("FATAL: Found %d - Supported 39 and 42\n", runtimeVMA);
+    Die();
+  }
+#endif
+}
+
 void InitializePlatform() {
   DisableCoreDumperIfNecessary();
 
   // Go maps shadow memory lazily and works fine with limited address space.
   // Unlimited stack is not a problem as well, because the executable
   // is not compiled with -pie.
   if (kCppMode) {
     bool reexec = false;

lib/tsan/rtl/tsan_platform_mac.cc

DPrintf("kShadow %zx-%zx (%zuGB)\n",
       kShadowBeg, kShadowEnd,
       (kShadowEnd - kShadowBeg) >> 30);
   DPrintf("kAppMem %zx-%zx (%zuGB)\n",
       kAppMemBeg, kAppMemEnd,
       (kAppMemEnd - kAppMemBeg) >> 30);
 }
 #endif
 
+void InitializePlatformSpecificModules() {
+}
+
 void InitializePlatform() {
   DisableCoreDumperIfNecessary();
 }
 
 #ifndef SANITIZER_GO
 // Note: this function runs with async signals enabled,
 // so it must not touch any tsan state.
 int call_pthread_cancel_with_cleanup(int(*fn)(void *c, void *m,

lib/tsan/rtl/tsan_platform_windows.cc

 }
 
 void FlushShadowMemory() {
 }
 
 void WriteMemoryProfile(char *buf, uptr buf_size, uptr nthread, uptr nlive) {
 }
 
+void InitializePlatformSpecificModules() {
+}
+
 void InitializePlatform() {
 }
 
 }  // namespace __tsan
 
 #endif  // SANITIZER_WINDOWS

lib/tsan/rtl/tsan_rtl.cc

 }
 
 static void CheckShadowMapping() {
   for (uptr i = 0; i < ARRAY_SIZE(UserRegions); i += 2) {
     const uptr beg = UserRegions[i];
     const uptr end = UserRegions[i + 1];
     VPrintf(3, "checking shadow region %p-%p\n", beg, end);
     for (uptr p0 = beg; p0 <= end; p0 += (end - beg) / 4) {
       for (int x = -1; x <= 1; x++) {
         const uptr p = p0 + x;
         if (p < beg || p >= end)
           continue;
         const uptr s = MemToShadow(p);
         const uptr m = (uptr)MemToMeta(p);
         VPrintf(3, "  checking pointer %p: shadow=%p meta=%p\n", p, s, m);
         CHECK(IsAppMem(p));
         CHECK(IsShadowMem(s));
void Initialize(ThreadState *thr) {
   SanitizerToolName = "ThreadSanitizer";
   // Install tool-specific callbacks in sanitizer_common.
   SetCheckFailedCallback(TsanCheckFailed);
 
   ctx = new(ctx_placeholder) Context;
   const char *options = GetEnv(kTsanOptionsEnv);
   CacheBinaryName();
   InitializeFlags(&ctx->flags, options);
-  CheckVMASize();
+  InitializePlatformSpecificModules();
 #ifndef SANITIZER_GO
   InitializeAllocator();
 #endif
   InitializeInterceptors();
   CheckShadowMapping();
   InitializePlatform();
   InitializeMutex();
   InitializeDynamicAnnotations();