[tsan] Disable randomized address space on linux aarch64.

[yabinc]

Summary

After patch https://lkml.org/lkml/2015/12/21/340 is introduced in
linux kernel, the random gap between stack and heap is increased
from 128M to 36G on 39-bit aarch64. And it is almost impossible
to cover this big range. So I think we need to disable randomized
virtual space on aarch64 linux.

[rengolin]

Is this Android specific?

[zatrazz]

lib/tsan/rtl/tsan_platform_linux.cc

310

Is this correct? This test is checking if system is *not* using ASLR, not the contrary. I think the correct test should be

(old_personality & ADDR_NO_RANDOMIZE) == ADDR_NO_RANDOMIZE

312

TSAN do work on aarch64 with ASLR, at least with kernel 3.19. I have not tested with a newer kernel, but I noticed it has now some VMA changes. Which is the environment that you seeing this failing?

[dvyukov]

FWIW we could also check whether modules are loaded at suitable addresses or not. And disable randomization/reexec only if it's really necessary. I don't know why reexec is a thing to avoid, though. So maybe it's good as is.

[yabinc]

Check if real heap space matches expectation before reexec.

[yabinc]

Limit columns to < 80.

[yabinc]

not android specific, I have updated in SUMMARY.
Done checking real heap space.

lib/tsan/rtl/tsan_platform_linux.cc

312

This is correct. I want to set ADDR_NO_RANDOMIZE instead of removing it. even it works with kernel 3.19, but doesn't work with latest kernel. And https://lkml.org/lkml/2015/12/21/340 has been cherry picked back to android kernels < 3.19. I mainly use N5X android device to test, which uses kernel 3.10.

[dvyukov]

lib/tsan/rtl/tsan_platform_linux.cc

267

Please call it somehow differently from "Heap". Heap commonly means "malloc heap" throughout the runtime, as is HeapMemEnd below.
I think we are mainly looking for mmaped modules here, so "ModulesEnd" would be fine. "MmapEnd" would be fine as well.

272

Please explain why we do this.

311

I am trying to convince myself that it won't break any of linux setups. We can have pie/non-pie binary. COMPAT mapping (setarch -L) is not supported now on x86 (modules mapped at 0x2a). Also disabled randomization is _not_ supported on x86 (modules mapped at 0x55).
So you have a good explanation as to why it all will continue to work?
I guess that we always have at least 1 dynamic library (static libc linking is not supported), so on x86 glibc should be mapped at 0x7f in supported configuration (randomization + no compat mapping). So GetHeapEnd should return 0x7f, and that's larger than HeapMemEnd.

But I think it will make failure mode for at least x86+COMPAT mapping much worse (infinite exec recursion instead of a readable message). Potentially there are other cases on other platforms (e.g. power/mips) when that will happen as well. Or maybe they legally have modules below heap?

I think it makes sense to tread more gradually and enable it just for aarch64 with an explanation as to why we are doing this. Other platforms will be enable to enable this by just altering ifdef condition later if necessary.

Or am I missing something?

[yabinc]

lib/tsan/rtl/tsan_platform_linux.cc

267

I am confused what is kHeapMem is tsan_platform.h for? In my current understanding, if the malloc map is allocated using brk(), the space is just above bss; if the malloc map is allocated using mmap(), the space should have no difference than other mmaped areas. like the one in my x86_64 linux pc:
006f0000-006f9000 rw-p 000f0000 fc:00 1838842 /bin/bash
006f9000-006ff000 rw-p 00000000 00:00 0
00f8e000-01401000 rw-p 00000000 00:00 0 [heap]

To make me more confusing, there is a gap between kHeapMemEnd and kHiAppMemBeg in x86_64, how is it decided?

For the code here, I am looking for mmaped modules, but I think it is probably represented by kHeapMem in tsan_platform.h, that's why I compare GetHeapEnd() < HeapMemEnd() below.

311

Frankly I don't understand what is COMPAT mapping and try setarch -L fails on my x86_64 pc. I'd like to learn if you can give me more contexts or instructions about how to test.
As I can see, on x86_64, mmap base is always at HiAppMem range, thus larger than HeapMemEnd.
x86+COMPAT mapping, how to experiment that? setarch -L seems don't work.
I don't think there will a infinite loop of reexec, because if CHECK(personality()) fails, it will exit. In consideration of not breaking existing behavior, I remove the CHECK() and don't reexec if failed to set personality. I am not sure if it is proper.

I just realize that the check of GetHeapEnd() < HeapMemEnd() doesn't work on aarch64 39-bit. Because there is a hole between kHeapMemEnd and kHiAppMemBeg, whose shadow mem space is taken by kMidAppMem. It means even if GetHeapEnd() is bigger than HeapMemEnd(), it can be like 0x7d20000000, and takes the same shadow mem as kMidAppMem. Currently I think we have two choices, one is making it android specific and reexec without checking real mmap space, another is keep the checking and change Mapping39 of aarch64 (but I am not confident as I don't know what kHeapMem stands for).

[dvyukov]

lib/tsan/rtl/tsan_platform_linux.cc

267

Tsan substitutes malloc, so the libc [heap] mapping is pretty much irrelevant. Tsan malloc manually mmaps heap at kHeapMem-kHeapMemEnd.

To make me more confusing, there is a gap between kHeapMemEnd and kHiAppMemBeg in x86_64, how is it decided?

These holes should be mprotected to prevent app from mapping anything there. See CheckAndProtect in tsan_platform_posix.cc.

For the code here, I am looking for mmaped modules, but I think it is probably represented by kHeapMem in tsan_platform.h, that's why I compare GetHeapEnd() < HeapMemEnd() below.

No, kHeapMem is tsan mapping for heap. mmap for heap uses fixed address, so kHeapMem can be arbitrary is not related to any other mappings.

311

I don't think there will a infinite loop of reexec, because if CHECK(personality()) fails, it will exit.

I mean the following situation.
Randomization is already disabled, but we still have a bad mapping. We try to disable randomization and it succeeds (as randomization is already disable). Then we reexec, but nothing is changed, we still have the same bad mapping. And here we get into infinite loop.

It probably would be safer to never try to reexec more than once. If first reexec does not help, we can just continue and CheckAndProtect later will catch and properly report the bad mapping.

Re COMPAT mapping, I can't find any definitive docs. It probably requires some kernel CONFIG. What it does is it causes modules to be mapped at 0x2a range on x86_64 linux. It is not supported now, but we must not go into infinite reexec loop, if user enables it.

[yabinc]

Disable randomized address space only for aarch64.

[yabinc]

lib/tsan/rtl/tsan_platform_linux.cc

311

`Thanks for explanation of kHeapMem, that makes much sense. And now I believe it is wrong to compare ModuleEnd() with HeapEnd().
For COMPAT mode, I run following commands with this patch enabled on x86_64:
$./thread_sanitize
old_personality = 0, ADDR_NO_RANDOMIZE = 40000

16085==WARNING: Program is run with randomized virtual address space, and uses mmap base out of that assumed by ThreadSanitizer.

16085==Re-execing with fixed virtual address space.

old_personality = 40000, ADDR_NO_RANDOMIZE = 40000

$setarch i686 -L ./thread_sanitize
old_personality = 200008, ADDR_NO_RANDOMIZE = 40000

16186==WARNING: Program is run with randomized virtual address space, and uses mmap base out of that assumed by ThreadSanitizer.

16186==Re-execing with fixed virtual address space.

old_personality = 240008, ADDR_NO_RANDOMIZE = 40000
FATAL: ThreadSanitizer: unexpected memory mapping 0x2aaaaaaab000-0x2aaaaaace000

So it won't cause infinite loop. Because personality() takes effect accross execv(), so the code won't try to set it again if it has been set before. But I think it is fine to limit this patch to aarch64 if other architectures don't need this.`

[dvyukov]

LGMT with a nit

lib/tsan/rtl/tsan_platform_linux.cc

310

Please leave a short comment here for next generations as to why we are doing this and name/hash of the kernel commit.

[yabinc]

Add comment.

[rengolin]

lib/tsan/rtl/tsan_platform_linux.cc

312

Just making sure I got it right: this should work on all kernels, right?

[yabinc]

lib/tsan/rtl/tsan_platform_linux.cc

312

yes. personality(ADDR_NO_RANDOMIZE) exists before the git history of linux kernel, and this option is not architecture specific.

[dvyukov]

LGTM

[rengolin]

LGTM too, thanks!

lib/tsan/rtl/tsan_platform_linux.cc

 #include <fcntl.h>
 #include <pthread.h>
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <stdarg.h>
 #include <sys/mman.h>
+#include <sys/personality.h>
 #include <sys/syscall.h>
 #include <sys/socket.h>
 #include <sys/time.h>
 #include <sys/types.h>
 #include <sys/resource.h>
 #include <sys/stat.h>
 #include <unistd.h>
 #include <errno.h>
if (vmaSize != 44 && vmaSize != 46) {
     Printf("FATAL: ThreadSanitizer: unsupported VMA range\n");
     Printf("FATAL: Found %d - Supported 44 and 46\n", vmaSize);
     Die();
   }
 #endif
 #endif
 }
 
+static uptr GetHeapEnd() {
+  MemoryMappingLayout proc_maps(true);
+  uptr heap_end = 0;
+  uptr start, end, prot;
+  while (proc_maps.Next(&start, &end, 0, 0, 0, &prot)) {
+    // Omit stack
+    if (start <= (uptr)&heap_end && (uptr)&heap_end < end)
+      continue;
+    if (end > heap_end)
+      heap_end = end;
+  }
+  return heap_end;
+}
+
 void InitializePlatform() {
   DisableCoreDumperIfNecessary();
 
   // Go maps shadow memory lazily and works fine with limited address space.
   // Unlimited stack is not a problem as well, because the executable
   // is not compiled with -pie.
   if (kCppMode) {
     bool reexec = false;
if (kCppMode) {
 
     if (!AddressSpaceIsUnlimited()) {
       Report("WARNING: Program is run with limited virtual address space,"
              " which wouldn't work with ThreadSanitizer.\n");
       Report("Re-execing with unlimited virtual address space.\n");
       SetAddressSpaceUnlimited();
       reexec = true;
     }
+    int old_personality = personality(0xffffffff);
+    if (old_personality != -1 && (old_personality & ADDR_NO_RANDOMIZE) == 0) {
+      if (GetHeapEnd() < HeapMemEnd()) {
+        Report("WARNING: Program is run with randomized virtual address space,"
+               " and uses heap space out of that assumed by ThreadSanitizer."
+               "\n");
+        Report("Re-execing with fixed virtual address space.\n");
+        CHECK(personality(old_personality | ADDR_NO_RANDOMIZE) != -1);
+        reexec = true;
+      }
+    }
     if (reexec)
       ReExec();
   }
 
 #ifndef SANITIZER_GO
   CheckAndProtect();
   InitTlsSize();
   InitDataSeg();