[compiler-rt] [tsan] Unify aarch64 mapping

[zatrazz]

Summary

This patch unify the 39 and 42-bit VMA support for AArch64 by using indirect
calls instead of inline the address function transformation. The already
defined maps for both VMA are used. Although slower, this leads to same
instrumented binary to be independent of the kernel. It also has the advantage
to be easier to enable the remaining VMA for AArch64 (48-bits).

Along with this change this patch also fix some 42-bit failures with
ALSR disable by increasing the upper high app memory threshold and also
the 42-bit madvise value for non large page set.

Regarding performance, I did a run with speccpu2006 using 'test' set (mainly
to evaluate faster) and the results shows the new indirect calls are roughly
4.6% slower than the inline version:

                 DIFF
401.bzip2        5.12 
403.gcc          5.24 
429.mcf          1.00 
445.gobmk        3.36 
456.hmmer        6.72 
458.sjeng        6.15 
462.libquantum   4.55 
464.h264ref      6.82 
473.astar        5.68 
483.xalancbmk    6.77 
GEOMEAN          4.62

PS: 400.perlbench fails due missing longjmp family instrumentation for aarch64 and 471.omnetpp
the know issue with user defined new/delete operators.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

159

You introduce both indirect functions and these non-const variables. This looks excessive. Let's do one or another. I would prefer global variables as it looks less intrusive.

You can do something along the lines of:

Here declare the consts:

extern uptr kLoAppMemBeg;
...

And then in InitializePlatformEarly initialize them:

kLoAppMemEnd = vma_39_42(0x0000400000ull, 0x00000400000ull);

And that's it.

175

All identifiers with double underscores are reserved for language implementation. TSAN_ is a fine prefix, and also consistent with all macros in the codebase.

580

Please rename this to InitializePlatformEarly.
I don't understand what is "Modules" (dynamic libraries?). And "Specific" is excessive, "Platform" already implies platform-specific stuff.

[zatrazz]

Changes from previous version:

• Increase the VMA range for application on both 39-bit and 42-bit VMA. Now 39-bit ranges from 0x7D00000000-0x7FFFFFFFFF (previously 0x7F00000000-0x7FFFFFFFFF) and 42-bits from 0x3f000000000-0x3FFFFFFFFFF (previously 0x3FF00000000-0x3FFFFFFFFFF);
• Fixed a printf format specifier in test/tsan/test.h.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

159

I declared them in the header because I used the constants in the function definitions at lib/tsan/rtl/tsan_platform_linux.cc. I can move them to the cc file if it desirable, but I think by removing the definitions will require to hardcode its value on the function definition somehow (and I think current approach in simple than replicate the hexadecimals values in some places).

Also, I use them so to add another VMA (48-bits) would not require much work. Instead of adding now 'vma_39_42' and then moving it to 'vma_39_42_48', the patch will just require to check and add the 48-bits definitions.

175

I will change it.

580

I will change it (I used 'modules' and 'specific' because I could not came up with a better naming).

[zatrazz]

This is an updated version based on comments:

• Moved aarch64 const VMA definition from platform header to platform file
• Rename InitializePlatformInitModules to InitializePlatformEarly

[dvyukov]

lib/tsan/rtl/tsan_platform.h

157

But can we have just global vars? It seems to be enough. The same generic functions will just operate with different values stores in the vars.

I am just frighten by the size of this change, amount of logical indirections and macros.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

157

We can, but it will be slower: every function transformation will require not only an indirection but also potentially memory reads to actually get the global variables value. The idea of the patch to create multiple functions is to try to minimize the performance hit this same indirection occurs by creating functions is fast as possible.

[zatrazz]

Small update on previous patch to initialize kMadviseRangeBeg and kMadviseRangeSize.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

143–144

Won't it then be faster to inline all 3 variants?

E.g.

struct Mapping39 {
  static const uptr kLoAppMemBeg = 0x...;
  static const uptr kLoAppMemEnd = 0x...;
  static const uptr kShadowBeg = 0x...;
  ...
};

struct Mapping42 {
  static const uptr kLoAppMemBeg = 0x...;
  static const uptr kLoAppMemEnd = 0x...;
  static const uptr kShadowBeg = 0x...;
  ...
};

template<typename Mapping>
uptr MemToShadowImpl(uptr x) {
  return (((x) & ~(Mapping::kAppMemMsk | (kShadowCell - 1)))
      ^ Mapping::kAppMemXor) * kShadowCnt;
}

uptr MemToShadow(uptr x) {
  if (vma_size == 39)
    return MemToShadowImpl<Mapping39>(x);
  else
    return MemToShadowImpl<Mapping42>(x);
  DCHECK(0);
}

I suspect it can be faster as it is inlinable. And probably compiler will be able to do some common expressions merging, so the code won't be too large.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

143–144

I will check this out, thanks!

[zatrazz]

The mapping using inline function is indeed slight faster than the previous attempt (I used the same benchmark as before, speccpu2006 test size). Changes from previous version:

• Use a different strategy for mapping functions: instead of indirect functions inline version based on an external VMA detected at runtime.
• Fixed the 39-bit shadow memory upper bound.

[dvyukov]

Much nicer now. Thanks.

Do we still need these extern vars in header
extern uptr kLoAppMemBeg
?
I would prefer if we get rid of any remaining usages of these and introduce more functions instead if needed.
They introduce significant amount of boilerplate. And also they are not necessary consts now, so it can be dangerous (performance-wise) to use them occasionally on some performance-critical path.

Also we now have duplication of complex code between MemToShadow for x86 and MemToShadowImpl for power.
Please move x86 consts to a Mapping struct as well. Then we can have a single MemToShadowImpl implementation and then define MemToShadow as:

uptr MemToShadowImpl(uptr x) {
#ifdef power

if (vmaSize == 39)
  return MemToShadowImpl<Mapping39>(x);
else
  return MemToShadowImpl<Mapping42>(x);

#else

return MemToShadowImpl<Mapping>(x);

#endif
}

[zatrazz]

Changes from previous patch:

• Unify all implementations using the Mapping scheme.
• Remove redundant code for go implementation by unify windows and non-windows transformation.
• All accesses to mapping const definition are done through functions now. For architecture with only one mapping this won't change code generation.

For architecture with static mapping I had to still define kHeapMemBeg and kHeapMemEnd because they are used on SizeClassAllocator64 definition. I think a possible cleanup is possible (prob when I adjust tsan for aarch64 48-bit VMA).

[zatrazz]

Ping.

[dvyukov]

Sorry for long delays.
I've looked at it several times and it still feels somewhat awkward, but at the same time I cannot come up with something better (we could remove lots of boilerplate and replication with advanced C++ templates, but I doubt it worth it, advanced C++ templates is an issue in itself). So probably some amount of duplication and boilerplate here is the way to go.
I will take a final look today.

[dvyukov]

lib/tsan/rtl/tsan_platform.h

224

UserRegions are used only once during startup, there is no point in having both const and dynamic versions of it. Also you initialize the dynamic variable only on linux, but it actually needs to be initialized when TSAN_RUNTIME_VMA (e.g. on mac/aarch64 in future).

Please leave only one version and define it in this header.
Something along the lines of:

bool GetUserRegion(int i, uptr *start, uptr *end) {

switch (i) {
default:
  return false;
case 0:
  *start = LoAppMemBeg();
  *end = LoAppMemEnd();
case 1:
  ...
case 2:
  ...
}

}

And loop in CheckShadowMapping until this function returns false.

272

Why !SANITIZER_WINDOWS?
There is also no windows-Go-specific code, so these definitions must be necessary either on all Go platforms or on no Go platforms.

lib/tsan/rtl/tsan_platform_posix.cc

50–60

vmaSize should be already set by now, so use it.
Add

} else {
  CHECK(0);
}

branch below. It will catch both case when vmaSize is somehow not initialized by now and case when support for a new vma size is added.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

224

I will change that.

272

This is wrong, I will fix it.

lib/tsan/rtl/tsan_platform_posix.cc

50–60

I will change that.

[zatrazz]

Changes from previous version:

• Remove UserRegions struct and change the interface to call GetUserRegion.
• Remove a wrong windows SANITIZER_GO define
• Use vmaSize on InitializeShadowMemory for aarch64.

[dvyukov]

LGTM with nits

sorry for delays and thanks for bearing with me

lib/tsan/rtl/tsan_platform.h

248

#ifndef SANITIZER_GO
that's more common throughout tsan

256

#else

315

drop !SANITIZER_WINDOWS part

lib/tsan/rtl/tsan_rtl.cc

291

please replace this with:

uptr beg, end;
for (int i = 0; GetUserRegion(i, &beg, &end); i++) {

it is shorter and more idiomatic for loops.

[eugenis]

lib/tsan/rtl/tsan_platform.h

129

I've noticed that this mapping does not have an application region at 0x55...
That's where PIE executables are mapped by the newer kernels.
Did you test this on an older kernel (I don't know exactly when the transition happened; it was around 4.1 for X86 and AFAIK a bit earlier for AArch 64)?

See http://reviews.llvm.org/D15308 for more context.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

129

I did check the PIE executable with a full TSAN test built with '-pie -fpic' and I saw no issue. Analyzing the trace for 39-bits using the mutexset1.cc (I picked it at random) I see:

• Non-pie build:

00400000-004d1000 r-xp 00000000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
004e0000-004e3000 r--p 000d0000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
004e3000-004e6000 rw-p 000d3000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1

• PIE build:

7fae119000-7fae1ee000 r-xp 00000000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
7fae1ee000-7fae1fb000 rw-p 00000000 00:00 0
7fae1fd000-7fae200000 r--p 000d4000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1
7fae200000-7fae203000 rw-p 000d7000 08:02 21008174 /home/adhemerval.zanella/llvm/llvm-git-aarch64-build-release/mutexset1

So at least for 39-bit PIE addresses are being covered. I will check again on 42-bit VMA.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

129

I and I checked on a fairly recent kernel, 3.19. I am not aware if the mappings for PIE has changed in recent kernels, neither I checked on recent ones. Do you have more information?

[eugenis]

lib/tsan/rtl/tsan_platform.h

129

See this bug for the linux 4.1.2 problem with MSan:
https://llvm.org/bugs/show_bug.cgi?id=24155

I see 0x55... mappings with 3.10.40 kernel on Android/AAarch64.

git tag --contains in the torvalds repo tells me that this change appears in v4.1 and newer. Android must have cherry-picked it in 3.10.

What happens if you disable ASLR? We used to get the main executable at 0x55... under gdb (i.e. with disabled randomization) long before 3.9.

[zatrazz]

lib/tsan/rtl/tsan_platform.h

129

My previous analysis was in fact wrong, even for the kernels I am currently using (3.19 for 39-bit and 3.17 for 42-bit I am seeing):

• 0x00000000000-0x00010000000: 39/42-bits program own segments
• 0x05500000000-0x05600000000: 39-bits PIE program segments
• 0x07f80000000-0x07fffffffff: 39-bits libraries segments
• 0x2aa00000000-0x2ab00000000: 42-bits PIE program segments
• 0x3ff00000000-0x3ffffffffff: 42-bits libraries segments

Fortunately I could adjust aarch64 mapping to include these segments using the current mapping. I am preparing a patch.

lib/tsan/rtl/tsan_platform.h

 3000 0000 0000 - 4000 0000 0000: metainfo (memory blocks and sync objects)
 4000 0000 0000 - 6000 0000 0000: -
 6000 0000 0000 - 6200 0000 0000: traces
 6200 0000 0000 - 7d00 0000 0000: -
 7d00 0000 0000 - 7e00 0000 0000: heap
 7e00 0000 0000 - 7e80 0000 0000: -
 7e80 0000 0000 - 8000 0000 0000: modules and main thread stack
 */
-const uptr kMetaShadowBeg = 0x300000000000ull;
-const uptr kMetaShadowEnd = 0x400000000000ull;
-const uptr kTraceMemBeg   = 0x600000000000ull;
-const uptr kTraceMemEnd   = 0x620000000000ull;
-const uptr kShadowBeg     = 0x020000000000ull;
-const uptr kShadowEnd     = 0x100000000000ull;
-const uptr kHeapMemBeg    = 0x7d0000000000ull;
-const uptr kHeapMemEnd    = 0x7e0000000000ull;
-const uptr kLoAppMemBeg   = 0x000000001000ull;
-const uptr kLoAppMemEnd   = 0x010000000000ull;
-const uptr kHiAppMemBeg   = 0x7e8000000000ull;
-const uptr kHiAppMemEnd   = 0x800000000000ull;
-const uptr kAppMemMsk     = 0x7c0000000000ull;
-const uptr kAppMemXor     = 0x020000000000ull;
-const uptr kVdsoBeg       = 0xf000000000000000ull;
+struct Mapping {
+  static const uptr kMetaShadowBeg = 0x300000000000ull;
+  static const uptr kMetaShadowEnd = 0x400000000000ull;
+  static const uptr kTraceMemBeg   = 0x600000000000ull;
+  static const uptr kTraceMemEnd   = 0x620000000000ull;
+  static const uptr kShadowBeg     = 0x020000000000ull;
+  static const uptr kShadowEnd     = 0x100000000000ull;
+  static const uptr kHeapMemBeg    = 0x7d0000000000ull;
+  static const uptr kHeapMemEnd    = 0x7e0000000000ull;
+  static const uptr kLoAppMemBeg   = 0x000000001000ull;
+  static const uptr kLoAppMemEnd   = 0x010000000000ull;
+  static const uptr kHiAppMemBeg   = 0x7e8000000000ull;
+  static const uptr kHiAppMemEnd   = 0x800000000000ull;
+  static const uptr kAppMemMsk     = 0x7c0000000000ull;
+  static const uptr kAppMemXor     = 0x020000000000ull;
+  static const uptr kVdsoBeg       = 0xf000000000000000ull;
+};
 #elif defined(__mips64)
 /*
 C/C++ on linux/mips64
 0100 0000 00 - 0200 0000 00: main binary
 0200 0000 00 - 1400 0000 00: -
 1400 0000 00 - 2400 0000 00: shadow
 2400 0000 00 - 3000 0000 00: -
 3000 0000 00 - 4000 0000 00: metainfo (memory blocks and sync objects)
 4000 0000 00 - 6000 0000 00: -
 6000 0000 00 - 6200 0000 00: traces
 6200 0000 00 - fe00 0000 00: -
 fe00 0000 00 - ff00 0000 00: heap
 ff00 0000 00 - ff80 0000 00: -
 ff80 0000 00 - ffff ffff ff: modules and main thread stack
 */
-const uptr kMetaShadowBeg = 0x3000000000ull;
-const uptr kMetaShadowEnd = 0x4000000000ull;
-const uptr kTraceMemBeg   = 0x6000000000ull;
-const uptr kTraceMemEnd   = 0x6200000000ull;
-const uptr kShadowBeg     = 0x1400000000ull;
-const uptr kShadowEnd     = 0x2400000000ull;
-const uptr kHeapMemBeg    = 0xfe00000000ull;
-const uptr kHeapMemEnd    = 0xff00000000ull;
-const uptr kLoAppMemBeg   = 0x0100000000ull;
-const uptr kLoAppMemEnd   = 0x0200000000ull;
-const uptr kHiAppMemBeg   = 0xff80000000ull;
-const uptr kHiAppMemEnd   = 0xffffffffffull;
-const uptr kAppMemMsk     = 0xfc00000000ull;
-const uptr kAppMemXor     = 0x0400000000ull;
-const uptr kVdsoBeg       = 0xfffff00000ull;
+struct Mapping {
+  static const uptr kMetaShadowBeg = 0x3000000000ull;
+  static const uptr kMetaShadowEnd = 0x4000000000ull;
+  static const uptr kTraceMemBeg   = 0x6000000000ull;
+  static const uptr kTraceMemEnd   = 0x6200000000ull;
+  static const uptr kShadowBeg     = 0x1400000000ull;
+  static const uptr kShadowEnd     = 0x2400000000ull;
+  static const uptr kHeapMemBeg    = 0xfe00000000ull;
+  static const uptr kHeapMemEnd    = 0xff00000000ull;
+  static const uptr kLoAppMemBeg   = 0x0100000000ull;
+  static const uptr kLoAppMemEnd   = 0x0200000000ull;
+  static const uptr kHiAppMemBeg   = 0xff80000000ull;
+  static const uptr kHiAppMemEnd   = 0xffffffffffull;
+  static const uptr kAppMemMsk     = 0xfc00000000ull;
+  static const uptr kAppMemXor     = 0x0400000000ull;
+  static const uptr kVdsoBeg       = 0xfffff00000ull;
+};
 #elif defined(__aarch64__)
-# if SANITIZER_AARCH64_VMA == 39
+// AArch64 supports multiple VMA which leads to multiple address transformation
+// functions.  To support these multiple VMAS transformations and mappings TSAN
+// runtime for AArch64 uses an external memory read (vmaSize) to select which
+// mapping to use.  Although slower, it make a same instrumented binary run on
+// multiple kernels.
+
 /*
 C/C++ on linux/aarch64 (39-bit VMA)
-0000 4000 00 - 0200 0000 00: main binary
-2000 0000 00 - 4000 0000 00: shadow memory
-4000 0000 00 - 5000 0000 00: metainfo
-5000 0000 00 - 6000 0000 00: -
+0000 0010 00 - 0100 0000 00: main binary
+0100 0000 00 - 0800 0000 00: -
+0800 0000 00 - 1F00 0000 00: shadow memory
+1C00 0000 00 - 3100 0000 00: -
[phone] - 3400 0000 00: metainfo
[phone] - 6000 0000 00: -
 6000 0000 00 - 6200 0000 00: traces
 6200 0000 00 - 7d00 0000 00: -
-7d00 0000 00 - 7e00 0000 00: heap
-7e00 0000 00 - 7fff ffff ff: modules and main thread stack
+7c00 0000 00 - 7d00 0000 00: heap
+7d00 0000 00 - 7fff ffff ff: modules and main thread stack
 */
-const uptr kLoAppMemBeg   = 0x0000400000ull;
-const uptr kLoAppMemEnd   = 0x0200000000ull;
-const uptr kShadowBeg     = 0x2000000000ull;
-const uptr kShadowEnd     = 0x4000000000ull;
-const uptr kMetaShadowBeg = 0x4000000000ull;
-const uptr kMetaShadowEnd = 0x5000000000ull;
-const uptr kTraceMemBeg   = 0x6000000000ull;
-const uptr kTraceMemEnd   = 0x6200000000ull;
-const uptr kHeapMemBeg    = 0x7d00000000ull;
-const uptr kHeapMemEnd    = 0x7e00000000ull;
-const uptr kHiAppMemBeg   = 0x7e00000000ull;
-const uptr kHiAppMemEnd   = 0x7fffffffffull;
-const uptr kAppMemMsk     = 0x7800000000ull;
-const uptr kAppMemXor     = 0x0800000000ull;
-const uptr kVdsoBeg       = 0x7f00000000ull;
-# elif SANITIZER_AARCH64_VMA == 42
+struct Mapping39 {
+  static const uptr kLoAppMemBeg   = 0x0000001000ull;
+  static const uptr kLoAppMemEnd   = 0x0100000000ull;
+  static const uptr kShadowBeg     = 0x0800000000ull;
+  static const uptr kShadowEnd     = 0x1F00000000ull;
+  static const uptr kMetaShadowBeg = 0x3100000000ull;
+  static const uptr kMetaShadowEnd = 0x3400000000ull;
+  static const uptr kTraceMemBeg   = 0x6000000000ull;
+  static const uptr kTraceMemEnd   = 0x6200000000ull;
+  static const uptr kHeapMemBeg    = 0x7c00000000ull;
+  static const uptr kHeapMemEnd    = 0x7d00000000ull;
+  static const uptr kHiAppMemBeg   = 0x7d00000000ull;
+  static const uptr kHiAppMemEnd   = 0x7fffffffffull;
+  static const uptr kAppMemMsk     = 0x7800000000ull;
+  static const uptr kAppMemXor     = 0x0200000000ull;
+  static const uptr kVdsoBeg       = 0x7f00000000ull;
+};
+
 /*
 C/C++ on linux/aarch64 (42-bit VMA)
-00000 4000 00 - 01000 0000 00: main binary
+00000 0010 00 - 01000 0000 00: main binary
 01000 0000 00 - 10000 0000 00: -
 10000 0000 00 - 20000 0000 00: shadow memory
 20000 0000 00 - 26000 0000 00: -
 26000 0000 00 - 28000 0000 00: metainfo
 28000 0000 00 - 36200 0000 00: -
 36200 0000 00 - 36240 0000 00: traces
 36240 0000 00 - 3e000 0000 00: -
 3e000 0000 00 - 3f000 0000 00: heap
-3c000 0000 00 - 3ff00 0000 00: -
-3ff00 0000 00 - 3ffff f000 00: modules and main thread stack
+3f000 0000 00 - 3ffff ffff ff: modules and main thread stack
 */
-const uptr kLoAppMemBeg   = 0x00000400000ull;
-const uptr kLoAppMemEnd   = 0x01000000000ull;
-const uptr kShadowBeg     = 0x10000000000ull;
-const uptr kShadowEnd     = 0x20000000000ull;
-const uptr kMetaShadowBeg = 0x26000000000ull;
-const uptr kMetaShadowEnd = 0x28000000000ull;
-const uptr kTraceMemBeg   = 0x36200000000ull;
-const uptr kTraceMemEnd   = 0x36400000000ull;
-const uptr kHeapMemBeg    = 0x3e000000000ull;
-const uptr kHeapMemEnd    = 0x3f000000000ull;
-const uptr kHiAppMemBeg   = 0x3ff00000000ull;
-const uptr kHiAppMemEnd   = 0x3fffff00000ull;
-const uptr kAppMemMsk     = 0x3c000000000ull;
-const uptr kAppMemXor     = 0x04000000000ull;
-const uptr kVdsoBeg       = 0x37f00000000ull;
-# endif
-#endif
-
-ALWAYS_INLINE
-bool IsAppMem(uptr mem) {
-  return (mem >= kHeapMemBeg && mem < kHeapMemEnd) ||
-         (mem >= kLoAppMemBeg && mem < kLoAppMemEnd) ||
-         (mem >= kHiAppMemBeg && mem < kHiAppMemEnd);
-}
-
-ALWAYS_INLINE
-bool IsShadowMem(uptr mem) {
-  return mem >= kShadowBeg && mem <= kShadowEnd;
-}
-
-ALWAYS_INLINE
-bool IsMetaMem(uptr mem) {
-  return mem >= kMetaShadowBeg && mem <= kMetaShadowEnd;
-}
-
-ALWAYS_INLINE
-uptr MemToShadow(uptr x) {
-  DCHECK(IsAppMem(x));
-  return (((x) & ~(kAppMemMsk | (kShadowCell - 1)))
-      ^ kAppMemXor) * kShadowCnt;
-}
-
-ALWAYS_INLINE
-u32 *MemToMeta(uptr x) {
-  DCHECK(IsAppMem(x));
-  return (u32*)(((((x) & ~(kAppMemMsk | (kMetaShadowCell - 1)))
-      ^ kAppMemXor) / kMetaShadowCell * kMetaShadowSize) | kMetaShadowBeg);
-}
-
-ALWAYS_INLINE
-uptr ShadowToMem(uptr s) {
-  CHECK(IsShadowMem(s));
-  if (s >= MemToShadow(kLoAppMemBeg) && s <= MemToShadow(kLoAppMemEnd - 1))
-    return (s / kShadowCnt) ^ kAppMemXor;
-  else
-    return ((s / kShadowCnt) ^ kAppMemXor) | kAppMemMsk;
-}
-
-static USED uptr UserRegions[] = {
-  kLoAppMemBeg, kLoAppMemEnd,
-  kHiAppMemBeg, kHiAppMemEnd,
-  kHeapMemBeg,  kHeapMemEnd,
+struct Mapping42 {
+  static const uptr kLoAppMemBeg   = 0x00000001000ull;
+  static const uptr kLoAppMemEnd   = 0x01000000000ull;
+  static const uptr kShadowBeg     = 0x10000000000ull;
+  static const uptr kShadowEnd     = 0x20000000000ull;
+  static const uptr kMetaShadowBeg = 0x26000000000ull;
+  static const uptr kMetaShadowEnd = 0x28000000000ull;
+  static const uptr kTraceMemBeg   = 0x36200000000ull;
+  static const uptr kTraceMemEnd   = 0x36400000000ull;
+  static const uptr kHeapMemBeg    = 0x3e000000000ull;
+  static const uptr kHeapMemEnd    = 0x3f000000000ull;
+  static const uptr kHiAppMemBeg   = 0x3f000000000ull;
+  static const uptr kHiAppMemEnd   = 0x3ffffffffffull;
+  static const uptr kAppMemMsk     = 0x3c000000000ull;
+  static const uptr kAppMemXor     = 0x04000000000ull;
+  static const uptr kVdsoBeg       = 0x37f00000000ull;
 };
 
+// Indicates the runtime will define the memory regions at runtime.
+#define TSAN_RUNTIME_VMA 1
+#endif
+
 #elif defined(SANITIZER_GO) && !SANITIZER_WINDOWS
 
 /* Go on linux, darwin and freebsd
 0000 0000 1000 - 0000 1000 0000: executable
 0000 1000 0000 - 00c0 0000 0000: -
 00c0 0000 0000 - 00e0 0000 0000: heap
 00e0 0000 0000 - 2000 0000 0000: -
 2000 0000 0000 - 2380 0000 0000: shadow
 2380 0000 0000 - 3000 0000 0000: -
 3000 0000 0000 - 4000 0000 0000: metainfo (memory blocks and sync objects)
 4000 0000 0000 - 6000 0000 0000: -
 6000 0000 0000 - 6200 0000 0000: traces
 6200 0000 0000 - 8000 0000 0000: -
 */
 
-const uptr kMetaShadowBeg = 0x300000000000ull;
-const uptr kMetaShadowEnd = 0x400000000000ull;
-const uptr kTraceMemBeg   = 0x600000000000ull;
-const uptr kTraceMemEnd   = 0x620000000000ull;
-const uptr kShadowBeg     = 0x200000000000ull;
-const uptr kShadowEnd     = 0x238000000000ull;
-const uptr kAppMemBeg     = 0x000000001000ull;
-const uptr kAppMemEnd     = 0x00e000000000ull;
+struct Mapping {
+  static const uptr kMetaShadowBeg = 0x300000000000ull;
+  static const uptr kMetaShadowEnd = 0x400000000000ull;
+  static const uptr kTraceMemBeg   = 0x600000000000ull;
+  static const uptr kTraceMemEnd   = 0x620000000000ull;
+  static const uptr kShadowBeg     = 0x200000000000ull;
+  static const uptr kShadowEnd     = 0x238000000000ull;
+  static const uptr kAppMemBeg     = 0x000000001000ull;
+  static const uptr kAppMemEnd     = 0x00e000000000ull;
+};
+
+#elif defined(SANITIZER_GO) && SANITIZER_WINDOWS
 
+/* Go on windows
+0000 0000 1000 - 0000 1000 0000: executable
+0000 1000 0000 - 00f8 0000 0000: -
+00c0 0000 0000 - 00e0 0000 0000: heap
+00e0 0000 0000 - 0100 0000 0000: -
+0100 0000 0000 - 0500 0000 0000: shadow
+0500 0000 0000 - 0560 0000 0000: -
+0560 0000 0000 - 0760 0000 0000: traces
+0760 0000 0000 - 07d0 0000 0000: metainfo (memory blocks and sync objects)
+07d0 0000 0000 - 8000 0000 0000: -
+*/
+
+struct Mapping {
+  static const uptr kMetaShadowBeg = 0x076000000000ull;
+  static const uptr kMetaShadowEnd = 0x07d000000000ull;
+  static const uptr kTraceMemBeg   = 0x056000000000ull;
+  static const uptr kTraceMemEnd   = 0x076000000000ull;
+  static const uptr kShadowBeg     = 0x010000000000ull;
+  static const uptr kShadowEnd     = 0x050000000000ull;
+  static const uptr kAppMemBeg     = 0x000000001000ull;
+  static const uptr kAppMemEnd     = 0x00e000000000ull;
+}
+
+#else
+# error "Unknown platform"
+#endif
+
+
+#ifdef TSAN_RUNTIME_VMA
+extern uptr vmaSize;
+#endif
+
+
+enum MappingType {
+  MAPPING_LO_APP_BEG,
+  MAPPING_LO_APP_END,
+  MAPPING_HI_APP_BEG,
+  MAPPING_HI_APP_END,
+  MAPPING_HEAP_BEG,
+  MAPPING_HEAP_END,
+  MAPPING_APP_BEG,
+  MAPPING_APP_END,
+  MAPPING_SHADOW_BEG,
+  MAPPING_SHADOW_END,
+  MAPPING_META_SHADOW_BEG,
+  MAPPING_META_SHADOW_END,
+  MAPPING_TRACE_BEG,
+  MAPPING_TRACE_END,
+  MAPPING_VDSO_BEG,
+};
+
+template<typename Mapping, int Type>
+uptr MappingImpl(void) {
+  switch (Type) {
+#if !defined(SANITIZER_GO)
+    case MAPPING_LO_APP_BEG: return Mapping::kLoAppMemBeg;
+    case MAPPING_LO_APP_END: return Mapping::kLoAppMemEnd;
+    case MAPPING_HI_APP_BEG: return Mapping::kHiAppMemBeg;
+    case MAPPING_HI_APP_END: return Mapping::kHiAppMemEnd;
+    case MAPPING_HEAP_BEG: return Mapping::kHeapMemBeg;
+    case MAPPING_HEAP_END: return Mapping::kHeapMemEnd;
+    case MAPPING_VDSO_BEG: return Mapping::kVdsoBeg;
+#elif defined(SANITIZER_GO)
+    case MAPPING_APP_BEG: return Mapping::kAppMemBeg;
+    case MAPPING_APP_END: return Mapping::kAppMemEnd;
+#endif
+    case MAPPING_SHADOW_BEG: return Mapping::kShadowBeg;
+    case MAPPING_SHADOW_END: return Mapping::kShadowEnd;
+    case MAPPING_META_SHADOW_BEG: return Mapping::kMetaShadowBeg;
+    case MAPPING_META_SHADOW_END: return Mapping::kMetaShadowEnd;
+    case MAPPING_TRACE_BEG: return Mapping::kTraceMemBeg;
+    case MAPPING_TRACE_END: return Mapping::kTraceMemEnd;
+  }
+}
+
+template<int Type>
+uptr MappingArchImpl(void) {
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return MappingImpl<Mapping39, Type>();
+  else
+    return MappingImpl<Mapping42, Type>();
+  DCHECK(0);
+#else
+  return MappingImpl<Mapping, Type>();
+#endif
+}
+
+#if !defined(SANITIZER_GO)
 ALWAYS_INLINE
-bool IsAppMem(uptr mem) {
-  return mem >= kAppMemBeg && mem < kAppMemEnd;
+uptr LoAppMemBeg(void) {
+  return MappingArchImpl<MAPPING_LO_APP_BEG>();
+}
+ALWAYS_INLINE
+uptr LoAppMemEnd(void) {
+  return MappingArchImpl<MAPPING_LO_APP_END>();
 }
 
 ALWAYS_INLINE
-bool IsShadowMem(uptr mem) {
-  return mem >= kShadowBeg && mem <= kShadowEnd;
+uptr HeapMemBeg(void) {
+  return MappingArchImpl<MAPPING_HEAP_BEG>();
+}
+ALWAYS_INLINE
+uptr HeapMemEnd(void) {
+  return MappingArchImpl<MAPPING_HEAP_END>();
 }
 
 ALWAYS_INLINE
-bool IsMetaMem(uptr mem) {
-  return mem >= kMetaShadowBeg && mem <= kMetaShadowEnd;
+uptr HiAppMemBeg(void) {
+  return MappingArchImpl<MAPPING_HI_APP_BEG>();
+}
+ALWAYS_INLINE
+uptr HiAppMemEnd(void) {
+  return MappingArchImpl<MAPPING_HI_APP_END>();
 }
 
 ALWAYS_INLINE
-uptr MemToShadow(uptr x) {
-  DCHECK(IsAppMem(x));
-  return ((x & ~(kShadowCell - 1)) * kShadowCnt) | kShadowBeg;
+uptr VdsoBeg(void) {
+  return MappingArchImpl<MAPPING_VDSO_BEG>();
 }
 
+#elif defined(SANITIZER_GO) && !SANITIZER_WINDOWS
+
 ALWAYS_INLINE
-u32 *MemToMeta(uptr x) {
-  DCHECK(IsAppMem(x));
-  return (u32*)(((x & ~(kMetaShadowCell - 1)) / \
-      kMetaShadowCell * kMetaShadowSize) | kMetaShadowBeg);
+uptr AppMemBeg(void) {
+  return MappingArchImpl<MAPPING_APP_BEG>();
+}
+ALWAYS_INLINE
+uptr AppMemEnd(void) {
+  return MappingArchImpl<MAPPING_APP_END>();
+}
+
+#endif
+
+static inline
+bool GetUserRegion(int i, uptr *start, uptr *end) {
+  switch (i) {
+  default:
+    return false;
+#ifndef SANITIZER_GO
+  case 0:
+    *start = LoAppMemBeg();
+    *end = LoAppMemEnd();
+    return true;
+  case 1:
+    *start = HiAppMemBeg();
+    *end = HiAppMemEnd();
+    return true;
+  case 2:
+    *start = HeapMemBeg();
+    *end = HeapMemEnd();
+    return true;
+#else
+  case 0:
+    *start = AppMemBeg();
+    *end = AppMemEnd();
+    return true;
+#endif
+  }
 }
 
 ALWAYS_INLINE
-uptr ShadowToMem(uptr s) {
-  CHECK(IsShadowMem(s));
-  return (s & ~kShadowBeg) / kShadowCnt;
+uptr ShadowBeg(void) {
+  return MappingArchImpl<MAPPING_SHADOW_BEG>();
+}
+ALWAYS_INLINE
+uptr ShadowEnd(void) {
+  return MappingArchImpl<MAPPING_SHADOW_END>();
 }
 
-static USED uptr UserRegions[] = {
-  kAppMemBeg, kAppMemEnd,
-};
+ALWAYS_INLINE
+uptr MetaShadowBeg(void) {
+  return MappingArchImpl<MAPPING_META_SHADOW_BEG>();
+}
+ALWAYS_INLINE
+uptr MetaShadowEnd(void) {
+  return MappingArchImpl<MAPPING_META_SHADOW_END>();
+}
 
-#elif defined(SANITIZER_GO) && SANITIZER_WINDOWS
+ALWAYS_INLINE
+uptr TraceMemBeg(void) {
+  return MappingArchImpl<MAPPING_TRACE_BEG>();
+}
+ALWAYS_INLINE
+uptr TraceMemEnd(void) {
+  return MappingArchImpl<MAPPING_TRACE_END>();
+}
 
-/* Go on windows
-0000 0000 1000 - 0000 1000 0000: executable
-0000 1000 0000 - 00f8 0000 0000: -
-00c0 0000 0000 - 00e0 0000 0000: heap
-00e0 0000 0000 - 0100 0000 0000: -
-0100 0000 0000 - 0500 0000 0000: shadow
-0500 0000 0000 - 0560 0000 0000: -
-0560 0000 0000 - 0760 0000 0000: traces
-0760 0000 0000 - 07d0 0000 0000: metainfo (memory blocks and sync objects)
-07d0 0000 0000 - 8000 0000 0000: -
-*/
 
-const uptr kMetaShadowBeg = 0x076000000000ull;
-const uptr kMetaShadowEnd = 0x07d000000000ull;
-const uptr kTraceMemBeg   = 0x056000000000ull;
-const uptr kTraceMemEnd   = 0x076000000000ull;
-const uptr kShadowBeg     = 0x010000000000ull;
-const uptr kShadowEnd     = 0x050000000000ull;
-const uptr kAppMemBeg     = 0x000000001000ull;
-const uptr kAppMemEnd     = 0x00e000000000ull;
+template<typename Mapping>
+bool IsAppMemImpl(uptr mem) {
+#if !defined(SANITIZER_GO)
+  return (mem >= Mapping::kHeapMemBeg && mem < Mapping::kHeapMemEnd) ||
+         (mem >= Mapping::kLoAppMemBeg && mem < Mapping::kLoAppMemEnd) ||
+         (mem >= Mapping::kHiAppMemBeg && mem < Mapping::kHiAppMemEnd);
+#else
+  return mem >= Mapping::kAppMemBeg && mem < Mapping::kAppMemEnd;
+#endif
+}
 
 ALWAYS_INLINE
 bool IsAppMem(uptr mem) {
-  return mem >= kAppMemBeg && mem < kAppMemEnd;
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return IsAppMemImpl<Mapping39>(mem);
+  else
+    return IsAppMemImpl<Mapping42>(mem);
+  DCHECK(0);
+#else
+  return IsAppMemImpl<Mapping>(mem);
+#endif
+}
+
+
+template<typename Mapping>
+bool IsShadowMemImpl(uptr mem) {
+  return mem >= Mapping::kShadowBeg && mem <= Mapping::kShadowEnd;
 }
 
 ALWAYS_INLINE
 bool IsShadowMem(uptr mem) {
-  return mem >= kShadowBeg && mem <= kShadowEnd;
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return IsShadowMemImpl<Mapping39>(mem);
+  else
+    return IsShadowMemImpl<Mapping42>(mem);
+  DCHECK(0);
+#else
+  return IsShadowMemImpl<Mapping>(mem);
+#endif
+}
+
+
+template<typename Mapping>
+bool IsMetaMemImpl(uptr mem) {
+  return mem >= Mapping::kMetaShadowBeg && mem <= Mapping::kMetaShadowEnd;
 }
 
 ALWAYS_INLINE
 bool IsMetaMem(uptr mem) {
-  return mem >= kMetaShadowBeg && mem <= kMetaShadowEnd;
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return IsMetaMemImpl<Mapping39>(mem);
+  else
+    return IsMetaMemImpl<Mapping42>(mem);
+  DCHECK(0);
+#else
+  return IsMetaMemImpl<Mapping>(mem);
+#endif
 }
 
-ALWAYS_INLINE
-uptr MemToShadow(uptr x) {
+
+template<typename Mapping>
+uptr MemToShadowImpl(uptr x) {
   DCHECK(IsAppMem(x));
-  return ((x & ~(kShadowCell - 1)) * kShadowCnt) + kShadowBeg;
+#if !defined(SANITIZER_GO)
+  return (((x) & ~(Mapping::kAppMemMsk | (kShadowCell - 1)))
+      ^ Mapping::kAppMemXor) * kShadowCnt;
+#else
+  return ((x & ~(kShadowCell - 1)) * kShadowCnt) | Mapping::kShadowBeg;
+#endif
 }
 
 ALWAYS_INLINE
-u32 *MemToMeta(uptr x) {
+uptr MemToShadow(uptr x) {
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return MemToShadowImpl<Mapping39>(x);
+  else
+    return MemToShadowImpl<Mapping42>(x);
+  DCHECK(0);
+#else
+  return MemToShadowImpl<Mapping>(x);
+#endif
+}
+
+
+template<typename Mapping>
+u32 *MemToMetaImpl(uptr x) {
   DCHECK(IsAppMem(x));
+#if !defined(SANITIZER_GO)
+  return (u32*)(((((x) & ~(Mapping::kAppMemMsk | (kMetaShadowCell - 1)))
+        ^ Mapping::kAppMemXor) / kMetaShadowCell * kMetaShadowSize)
+          | Mapping::kMetaShadowBeg);
+#else
   return (u32*)(((x & ~(kMetaShadowCell - 1)) / \
-      kMetaShadowCell * kMetaShadowSize) | kMetaShadowBeg);
+      kMetaShadowCell * kMetaShadowSize) | Mapping::kMetaShadowBeg);
+#endif
 }
 
 ALWAYS_INLINE
-uptr ShadowToMem(uptr s) {
-  CHECK(IsShadowMem(s));
-  // FIXME(dvyukov): this is most likely wrong as the mapping is not bijection.
-  return (s - kShadowBeg) / kShadowCnt;
+u32 *MemToMeta(uptr x) {
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return MemToMetaImpl<Mapping39>(x);
+  else
+    return MemToMetaImpl<Mapping42>(x);
+  DCHECK(0);
+#else
+  return MemToMetaImpl<Mapping>(x);
+#endif
 }
 
-static USED uptr UserRegions[] = {
-  kAppMemBeg, kAppMemEnd,
-};
 
-#else
-# error "Unknown platform"
+template<typename Mapping>
+uptr ShadowToMemImpl(uptr s) {
+  DCHECK(IsShadowMem(s));
+#if !defined(SANITIZER_GO)
+  if (s >= MemToShadow(Mapping::kLoAppMemBeg)
+      && s <= MemToShadow(Mapping::kLoAppMemEnd - 1))
+    return (s / kShadowCnt) ^ Mapping::kAppMemXor;
+  else
+    return ((s / kShadowCnt) ^ Mapping::kAppMemXor) | Mapping::kAppMemMsk;
+#elif defined(SANITIZER_GO)
+# ifndef SANITIZER_WINDOWS
+  return (s & ~Mapping::kShadowBeg) / kShadowCnt;
+# else
+  // FIXME(dvyukov): this is most likely wrong as the mapping is not bijection.
+  return (s - Mapping::kShadowBeg) / kShadowCnt;
+# endif // SANITIZER_WINDOWS
+#endif
+}
+
+ALWAYS_INLINE
+uptr ShadowToMem(uptr s) {
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return ShadowToMemImpl<Mapping39>(s);
+  else
+    return ShadowToMemImpl<Mapping42>(s);
+  DCHECK(0);
+#else
+  return ShadowToMemImpl<Mapping>(s);
 #endif
+}
+
+
 
 // The additional page is to catch shadow stack overflow as paging fault.
 // Windows wants 64K alignment for mmaps.
 const uptr kTotalTraceSize = (kTraceSize * sizeof(Event) + sizeof(Trace)
     + (64 << 10) + (64 << 10) - 1) & ~((64 << 10) - 1);
 
-uptr ALWAYS_INLINE GetThreadTrace(int tid) {
-  uptr p = kTraceMemBeg + (uptr)tid * kTotalTraceSize;
-  DCHECK_LT(p, kTraceMemEnd);
+template<typename Mapping>
+uptr GetThreadTraceImpl(int tid) {
+  uptr p = Mapping::kTraceMemBeg + (uptr)tid * kTotalTraceSize;
+  DCHECK_LT(p, Mapping::kTraceMemEnd);
   return p;
 }
 
-uptr ALWAYS_INLINE GetThreadTraceHeader(int tid) {
-  uptr p = kTraceMemBeg + (uptr)tid * kTotalTraceSize
+ALWAYS_INLINE
+uptr GetThreadTrace(int tid) {
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return GetThreadTraceImpl<Mapping39>(tid);
+  else
+    return GetThreadTraceImpl<Mapping42>(tid);
+  DCHECK(0);
+#else
+  return GetThreadTraceImpl<Mapping>(tid);
+#endif
+}
+
+
+template<typename Mapping>
+uptr GetThreadTraceHeaderImpl(int tid) {
+  uptr p = Mapping::kTraceMemBeg + (uptr)tid * kTotalTraceSize
       + kTraceSize * sizeof(Event);
-  DCHECK_LT(p, kTraceMemEnd);
+  DCHECK_LT(p, Mapping::kTraceMemEnd);
   return p;
 }
 
+ALWAYS_INLINE
+uptr GetThreadTraceHeader(int tid) {
+#ifdef __aarch64__
+  if (vmaSize == 39)
+    return GetThreadTraceHeaderImpl<Mapping39>(tid);
+  else
+    return GetThreadTraceHeaderImpl<Mapping42>(tid);
+  DCHECK(0);
+#else
+  return GetThreadTraceHeaderImpl<Mapping>(tid);
+#endif
+}
+
 void InitializePlatform();
+void InitializePlatformEarly();
 void CheckAndProtect();
 void InitializeShadowMemoryPlatform();
 void FlushShadowMemory();
 void WriteMemoryProfile(char *buf, uptr buf_size, uptr nthread, uptr nlive);
 
 // Says whether the addr relates to a global var.
 // Guesses with high probability, may yield both false positives and negatives.
 bool IsGlobalVar(uptr addr);

lib/tsan/rtl/tsan_platform_linux.cc

 void *__libc_stack_end = 0;
 #endif
 
 namespace __tsan {
 
 static uptr g_data_start;
 static uptr g_data_end;
 
+#ifdef TSAN_RUNTIME_VMA
+// Runtime detected VMA size.
+uptr vmaSize;
+#endif
+
 enum {
   MemTotal  = 0,
   MemShadow = 1,
   MemMeta   = 2,
   MemFile   = 3,
   MemMmap   = 4,
   MemTrace  = 5,
   MemHeap   = 6,
   MemOther  = 7,
   MemCount  = 8,
 };
 
 void FillProfileCallback(uptr p, uptr rss, bool file,
                          uptr *mem, uptr stats_size) {
   mem[MemTotal] += rss;
-  if (p >= kShadowBeg && p < kShadowEnd)
+  if (p >= ShadowBeg() && p < ShadowEnd())
     mem[MemShadow] += rss;
-  else if (p >= kMetaShadowBeg && p < kMetaShadowEnd)
+  else if (p >= MetaShadowBeg() && p < MetaShadowEnd())
     mem[MemMeta] += rss;
 #ifndef SANITIZER_GO
-  else if (p >= kHeapMemBeg && p < kHeapMemEnd)
+  else if (p >= HeapMemBeg() && p < HeapMemEnd())
     mem[MemHeap] += rss;
-  else if (p >= kLoAppMemBeg && p < kLoAppMemEnd)
+  else if (p >= LoAppMemBeg() && p < LoAppMemEnd())
     mem[file ? MemFile : MemMmap] += rss;
-  else if (p >= kHiAppMemBeg && p < kHiAppMemEnd)
+  else if (p >= HiAppMemBeg() && p < HiAppMemEnd())
     mem[file ? MemFile : MemMmap] += rss;
 #else
-  else if (p >= kAppMemBeg && p < kAppMemEnd)
+  else if (p >= AppMemBeg() && p < AppMemEnd())
     mem[file ? MemFile : MemMmap] += rss;
 #endif
-  else if (p >= kTraceMemBeg && p < kTraceMemEnd)
+  else if (p >= TraceMemBeg() && p < TraceMemEnd())
     mem[MemTrace] += rss;
   else
     mem[MemOther] += rss;
 }
 
 void WriteMemoryProfile(char *buf, uptr buf_size, uptr nthread, uptr nlive) {
   uptr mem[MemCount] = {};
   __sanitizer::GetMemoryProfile(FillProfileCallback, mem, 7);
   StackDepotStats *stacks = StackDepotGetStats();
   internal_snprintf(buf, buf_size,
       "RSS %zd MB: shadow:%zd meta:%zd file:%zd mmap:%zd"
       " trace:%zd heap:%zd other:%zd stacks=%zd[%zd] nthr=%zd/%zd\n",
       mem[MemTotal] >> 20, mem[MemShadow] >> 20, mem[MemMeta] >> 20,
       mem[MemFile] >> 20, mem[MemMmap] >> 20, mem[MemTrace] >> 20,
       mem[MemHeap] >> 20, mem[MemOther] >> 20,
       stacks->allocated >> 20, stacks->n_uniq_ids,
       nlive, nthread);
 }
 
 #if SANITIZER_LINUX
 void FlushShadowMemoryCallback(
     const SuspendedThreadsList &suspended_threads_list,
     void *argument) {
-  FlushUnneededShadowMemory(kShadowBeg, kShadowEnd - kShadowBeg);
+  FlushUnneededShadowMemory(ShadowBeg(), ShadowEnd() - ShadowBeg());
 }
 #endif
 
 void FlushShadowMemory() {
 #if SANITIZER_LINUX
   StopTheWorld(FlushShadowMemoryCallback, 0);
 #endif
 }
#endif
   DPrintf("guessed data_start=%p data_end=%p\n",  g_data_start, g_data_end);
   CHECK_LT(g_data_start, g_data_end);
   CHECK_GE((uptr)&g_data_start, g_data_start);
   CHECK_LT((uptr)&g_data_start, g_data_end);
 }
 
 #endif  // #ifndef SANITIZER_GO
 
+void InitializePlatformEarly() {
+#ifdef TSAN_RUNTIME_VMA
+  vmaSize =
+    (MostSignificantSetBitIndex(GET_CURRENT_FRAME()) + 1);
+  if (vmaSize != 39 && vmaSize != 42) {
+    Printf("FATAL: ThreadSanitizer: unsupported VMA range\n");
+    Printf("FATAL: Found %d - Supported 39 and 42\n", vmaSize);
+    Die();
+  }
+#endif
+}
+
 void InitializePlatform() {
   DisableCoreDumperIfNecessary();
 
   // Go maps shadow memory lazily and works fine with limited address space.
   // Unlimited stack is not a problem as well, because the executable
   // is not compiled with -pie.
   if (kCppMode) {
     bool reexec = false;

lib/tsan/rtl/tsan_platform_mac.cc

if (event == PTHREAD_INTROSPECTION_THREAD_CREATE) {
     }
   }
 
   if (prev_pthread_introspection_hook != nullptr)
     prev_pthread_introspection_hook(event, thread, addr, size);
 }
 #endif
 
+void InitializePlatformEarly() {
+}
+
 void InitializePlatform() {
   DisableCoreDumperIfNecessary();
 #ifndef SANITIZER_GO
   CheckAndProtect();
 
   CHECK_EQ(main_thread_identity, 0);
   main_thread_identity = (uptr)pthread_self();
 

lib/tsan/rtl/tsan_platform_posix.cc

 #include "tsan_rtl.h"
 
 namespace __tsan {
 
 #ifndef SANITIZER_GO
 void InitializeShadowMemory() {
   // Map memory shadow.
   uptr shadow =
-      (uptr)MmapFixedNoReserve(kShadowBeg, kShadowEnd - kShadowBeg, "shadow");
-  if (shadow != kShadowBeg) {
+      (uptr)MmapFixedNoReserve(ShadowBeg(), ShadowEnd() - ShadowBeg(),
+                               "shadow");
+  if (shadow != ShadowBeg()) {
     Printf("FATAL: ThreadSanitizer can not mmap the shadow memory\n");
     Printf("FATAL: Make sure to compile with -fPIE and "
-               "to link with -pie (%p, %p).\n", shadow, kShadowBeg);
+               "to link with -pie (%p, %p).\n", shadow, ShadowBeg());
     Die();
   }
   // This memory range is used for thread stacks and large user mmaps.
   // Frequently a thread uses only a small part of stack and similarly
   // a program uses a small part of large mmap. On some programs
   // we see 20% memory usage reduction without huge pages for this range.
   // FIXME: don't use constants here.
 #if defined(__x86_64__)
   const uptr kMadviseRangeBeg  = 0x7f0000000000ull;
   const uptr kMadviseRangeSize = 0x010000000000ull;
 #elif defined(__mips64)
   const uptr kMadviseRangeBeg  = 0xff00000000ull;
   const uptr kMadviseRangeSize = 0x0100000000ull;
 #elif defined(__aarch64__)
-  const uptr kMadviseRangeBeg  = 0x7e00000000ull;
-  const uptr kMadviseRangeSize = 0x0100000000ull;
+  uptr kMadviseRangeBeg = 0;
+  uptr kMadviseRangeSize = 0;
+  if (vmaSize == 39) {
+    kMadviseRangeBeg  = 0x7d00000000ull;
+    kMadviseRangeSize = 0x0300000000ull;
+  } else if (vmaSize == 42) {
+    kMadviseRangeBeg  = 0x3f000000000ull;
+    kMadviseRangeSize = 0x01000000000ull;
+  } else {
+    DCHECK(0);
+  }
 #endif
   NoHugePagesInRegion(MemToShadow(kMadviseRangeBeg),
                       kMadviseRangeSize * kShadowMultiplier);
   // Meta shadow is compressing and we don't flush it,
   // so it makes sense to mark it as NOHUGEPAGE to not over-allocate memory.
   // On one program it reduces memory consumption from 5GB to 2.5GB.
-  NoHugePagesInRegion(kMetaShadowBeg, kMetaShadowEnd - kMetaShadowBeg);
+  NoHugePagesInRegion(MetaShadowBeg(), MetaShadowEnd() - MetaShadowBeg());
   if (common_flags()->use_madv_dontdump)
-    DontDumpShadowMemory(kShadowBeg, kShadowEnd - kShadowBeg);
+    DontDumpShadowMemory(ShadowBeg(), ShadowEnd() - ShadowBeg());
   DPrintf("memory shadow: %zx-%zx (%zuGB)\n",
-      kShadowBeg, kShadowEnd,
-      (kShadowEnd - kShadowBeg) >> 30);
+      ShadowBeg(), ShadowEnd(),
+      (ShadowEnd() - ShadowBeg()) >> 30);
 
   // Map meta shadow.
-  uptr meta_size = kMetaShadowEnd - kMetaShadowBeg;
+  uptr meta_size = MetaShadowEnd() - MetaShadowBeg();
   uptr meta =
-      (uptr)MmapFixedNoReserve(kMetaShadowBeg, meta_size, "meta shadow");
-  if (meta != kMetaShadowBeg) {
+      (uptr)MmapFixedNoReserve(MetaShadowBeg(), meta_size, "meta shadow");
+  if (meta != MetaShadowBeg()) {
     Printf("FATAL: ThreadSanitizer can not mmap the shadow memory\n");
     Printf("FATAL: Make sure to compile with -fPIE and "
-               "to link with -pie (%p, %p).\n", meta, kMetaShadowBeg);
+               "to link with -pie (%p, %p).\n", meta, MetaShadowBeg());
     Die();
   }
   if (common_flags()->use_madv_dontdump)
     DontDumpShadowMemory(meta, meta_size);
   DPrintf("meta shadow: %zx-%zx (%zuGB)\n",
       meta, meta + meta_size, meta_size >> 30);
 
   InitializeShadowMemoryPlatform();

 
 void CheckAndProtect() {
   // Ensure that the binary is indeed compiled with -pie.
   MemoryMappingLayout proc_maps(true);
   uptr p, end, prot;
   while (proc_maps.Next(&p, &end, 0, 0, 0, &prot)) {
     if (IsAppMem(p))
       continue;
-    if (p >= kHeapMemEnd &&
+    if (p >= HeapMemEnd() &&
         p < HeapEnd())
       continue;
     if (prot == 0)  // Zero page or mprotected.
       continue;
-    if (p >= kVdsoBeg)  // vdso
+    if (p >= VdsoBeg())  // vdso
       break;
     Printf("FATAL: ThreadSanitizer: unexpected memory mapping %p-%p\n", p, end);
     Die();
   }
 
-  ProtectRange(kLoAppMemEnd, kShadowBeg);
-  ProtectRange(kShadowEnd, kMetaShadowBeg);
-  ProtectRange(kMetaShadowEnd, kTraceMemBeg);
+  ProtectRange(LoAppMemEnd(), ShadowBeg());
+  ProtectRange(ShadowEnd(), MetaShadowBeg());
+  ProtectRange(MetaShadowEnd(), TraceMemBeg());
   // Memory for traces is mapped lazily in MapThreadTrace.
   // Protect the whole range for now, so that user does not map something here.
-  ProtectRange(kTraceMemBeg, kTraceMemEnd);
-  ProtectRange(kTraceMemEnd, kHeapMemBeg);
-  ProtectRange(HeapEnd(), kHiAppMemBeg);
+  ProtectRange(TraceMemBeg(), TraceMemEnd());
+  ProtectRange(TraceMemEnd(), HeapMemBeg());
+  ProtectRange(HeapEnd(), HiAppMemBeg());
 }
 #endif
 
 }  // namespace __tsan
 
 #endif  // SANITIZER_POSIX

lib/tsan/rtl/tsan_platform_windows.cc

 }
 
 void FlushShadowMemory() {
 }
 
 void WriteMemoryProfile(char *buf, uptr buf_size, uptr nthread, uptr nlive) {
 }
 
+void InitializePlatformEarly() {
+}
+
 void InitializePlatform() {
 }
 
 }  // namespace __tsan
 
 #endif  // SANITIZER_WINDOWS

lib/tsan/rtl/tsan_rtl.h

 static const uptr kAllocatorNumRegions =
     kAllocatorSize >> kAllocatorRegionSizeLog;
 typedef TwoLevelByteMap<(kAllocatorNumRegions >> 12), 1 << 12,
     MapUnmapCallback> ByteMap;
 typedef SizeClassAllocator32<kAllocatorSpace, kAllocatorSize, 0,
     CompactSizeClassMap, kAllocatorRegionSizeLog, ByteMap,
     MapUnmapCallback> PrimaryAllocator;
 #else
-typedef SizeClassAllocator64<kHeapMemBeg, kHeapMemEnd - kHeapMemBeg, 0,
+typedef SizeClassAllocator64<Mapping::kHeapMemBeg,
+    Mapping::kHeapMemEnd - Mapping::kHeapMemBeg, 0,
     DefaultSizeClassMap, MapUnmapCallback> PrimaryAllocator;
 #endif
 typedef SizeClassAllocatorLocalCache<PrimaryAllocator> AllocatorCache;
 typedef LargeMmapAllocator<MapUnmapCallback> SecondaryAllocator;
 typedef CombinedAllocator<PrimaryAllocator, AllocatorCache,
     SecondaryAllocator> Allocator;
 Allocator *allocator();
 #endif
#endif
   Event *trace = (Event*)GetThreadTrace(fs.tid());
   Event *evp = &trace[pos];
   Event ev = (u64)addr | ((u64)typ << 61);
   *evp = ev;
 }
 
 #ifndef SANITIZER_GO
 uptr ALWAYS_INLINE HeapEnd() {
-  return kHeapMemEnd + PrimaryAllocator::AdditionalSize();
+  return HeapMemEnd() + PrimaryAllocator::AdditionalSize();
 }
 #endif
 
 }  // namespace __tsan
 
 #endif  // TSAN_RTL_H

lib/tsan/rtl/tsan_rtl.cc

if (!data_mapped) {
     mapped_meta_end = meta_end;
   }
   VPrintf(2, "mapped meta shadow for (%p-%p) at (%p-%p)\n",
       addr, addr+size, meta_begin, meta_end);
 }
 
 void MapThreadTrace(uptr addr, uptr size, const char *name) {
   DPrintf("#0: Mapping trace at %p-%p(0x%zx)\n", addr, addr + size, size);
-  CHECK_GE(addr, kTraceMemBeg);
-  CHECK_LE(addr + size, kTraceMemEnd);
+  CHECK_GE(addr, TraceMemBeg());
+  CHECK_LE(addr + size, TraceMemEnd());
   CHECK_EQ(addr, addr & ~((64 << 10) - 1));  // windows wants 64K alignment
   uptr addr1 = (uptr)MmapFixedNoReserve(addr, size, name);
   if (addr1 != addr) {
     Printf("FATAL: ThreadSanitizer can not mmap thread trace (%p/%p->%p)\n",
         addr, size, addr1);
     Die();
   }
 }
 
 static void CheckShadowMapping() {
-  for (uptr i = 0; i < ARRAY_SIZE(UserRegions); i += 2) {
-    const uptr beg = UserRegions[i];
-    const uptr end = UserRegions[i + 1];
+  int i = 0;
+  while (1) {
+    uptr beg, end;
+    if (!GetUserRegion (i, &beg, &end))
+      break;
+    i++;
+
     VPrintf(3, "checking shadow region %p-%p\n", beg, end);
     for (uptr p0 = beg; p0 <= end; p0 += (end - beg) / 4) {
       for (int x = -1; x <= 1; x++) {
         const uptr p = p0 + x;
         if (p < beg || p >= end)
           continue;
         const uptr s = MemToShadow(p);
         const uptr m = (uptr)MemToMeta(p);
void Initialize(ThreadState *thr) {
   SanitizerToolName = "ThreadSanitizer";
   // Install tool-specific callbacks in sanitizer_common.
   SetCheckFailedCallback(TsanCheckFailed);
 
   ctx = new(ctx_placeholder) Context;
   const char *options = GetEnv(kTsanOptionsEnv);
   CacheBinaryName();
   InitializeFlags(&ctx->flags, options);
-  CheckVMASize();
+  InitializePlatformEarly();
 #ifndef SANITIZER_GO
   InitializeAllocator();
   ReplaceSystemMalloc();
 #endif
   InitializeInterceptors();
   CheckShadowMapping();
   InitializePlatform();
   InitializeMutex();

test/tsan/test.h

#elif defined(__aarch64__)
   const char *format = 0;
   unsigned long vma = (unsigned long)__builtin_frame_address(0);
   vma = 64 - __builtin_clzll(vma);
   if (vma == 39)
     format = "0x%010lx";
   else if (vma == 42)
     format = "0x%011lx";
   else {
-    fprintf(stderr, "unsupported vma: %ul\n", vma);
+    fprintf(stderr, "unsupported vma: %lu\n", vma);
     exit(1);
   }
 
   fprintf(stderr, format, (unsigned long) address);
 #endif
 }