tsan: add debugging code for ptrace test failures

[dvyukov]

Summary

Debugging of crashes on powerpc after commit:
c80604f7a3 ("tsan: remove real func check from interceptors")
Somehow replacing if with DCHECK leads to strange failures in:
SanitizerCommon-tsan-powerpc64le-Linux :: Linux/ptrace.cpp
https://lab.llvm.org/buildbot/#/builders/105
https://lab.llvm.org/buildbot/#/builders/121
https://lab.llvm.org/buildbot/#/builders/57

The hypothesis is that something writes out-of-bounds
into pt_regs on stack and that corrupts internal tsan state.

[melver]

This is very strange.

@amyk
Just in case, is there a way to get ssh access to a powerpc environment to repro these kinds of failures? There have been a number of powerpc-related issues, and we just can't repro them. :-/

[dvyukov]

Yes, this is very strange :)

The failure looks like the ScopedInterceptor object on stack was corrupted when returning from the interceptor (after the real syscall). So I don't know if it's the case or not, but if we assume that syscall assumes that pt_regs are larger and overwrites more memory, then it would explain the failure mode perfectly.

[amyk]

Hi @melver @dvyukov,

I've checked and we do have a Power environment that can be accessed. I've sent an e-mail to you both regarding this.

[amyk]

Hi @melver @dvyukov,

Just wanted to check in - I believe I've granted access to a PPC machine. Were we able to determine if this patch resolves the issues seen on the buildbot, and would this patch be ready to commit?

[dvyukov]

Hi Any,

I only get to this point:

# to get newer cmake
export PATH=/usr/sbin:$PATH
CC=/home/fedora/gcc/install/gcc-7.1.0/bin/gcc CXX=/home/fedora/gcc/install/gcc-7.1.0/bin/g++ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;libcxx;libcxxabi;lld" -GNinja ../llvm
...
CMake Error in /home/llvmguest/dvyukov/llvm-project/libcxx/benchmarks/CMakeLists.txt:
  The compiler feature "cxx_std_20" is not known to CXX compiler
  version 7.1.0.

I can't find any other gcc/clang on the machine.

[amyk]

In D112703#3096790, @dvyukov wrote:

Hi Any,

I only get to this point:

# to get newer cmake
export PATH=/usr/sbin:$PATH
CC=/home/fedora/gcc/install/gcc-7.1.0/bin/gcc CXX=/home/fedora/gcc/install/gcc-7.1.0/bin/g++ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;libcxx;libcxxabi;lld" -GNinja ../llvm
...
CMake Error in /home/llvmguest/dvyukov/llvm-project/libcxx/benchmarks/CMakeLists.txt:
  The compiler feature "cxx_std_20" is not known to CXX compiler
  version 7.1.0.

I can't find any other gcc/clang on the machine.

I'm sorry that you were having troubles building on the machine. I can look into that further.
In any case, I just tested this patch on the machine I am on right now and this resolves the SanitizerCommon-tsan-powerpc64le-Linux :: Linux/ptrace.cpp failure.
Since this patch resolves the failure, can this be committed? Or were you planning on investigating the problem further on the machine (If this is the case and if this patch cannot be committed, perhaps we can revert the offending patch first prior to doing more investigation)?

[dvyukov]

Thanks for testing. I've landed this change.

Marco helped me to figure out the command that works. For the record:

export PATH=/usr/sbin:$PATH
cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;" -GNinja ../llvm
ninja && ninja check-sanitizer

It's now building...

[amyk]

Great, thank you very much Dmitry. Really appreciate it!

[dvyukov]

I've reproduced the failure and reproduced "fixing" by this change.
I see some stack corruption and it's affected by small unrelated changes like adding Printf's. I've tried dumping corrupted memory, etc, but it did not give me any glues.
At this point I suspect gcc/glibc/kernel bug. I don't think it makes sense to spend more time debugging this with gcc 6.1 and kernel 4.11. Thousands of bugs were fixed since then.

compiler-rt/lib/tsan/rtl/tsan_interceptors.h

 
 }  // namespace __tsan
 
 #define SCOPED_INTERCEPTOR_RAW(func, ...)            \
   ThreadState *thr = cur_thread_init();              \
   ScopedInterceptor si(thr, #func, GET_CALLER_PC()); \
   UNUSED const uptr pc = GET_CURRENT_PC();
 
+#ifdef __powerpc64__
+// Debugging of crashes on powerpc after commit:
+// c80604f7a3 ("tsan: remove real func check from interceptors")
+// Somehow replacing if with DCHECK leads to strange failures in:
+// SanitizerCommon-tsan-powerpc64le-Linux :: Linux/ptrace.cpp
+// https://lab.llvm.org/buildbot/#/builders/105
+// https://lab.llvm.org/buildbot/#/builders/121
+// https://lab.llvm.org/buildbot/#/builders/57
+#  define CHECK_REAL_FUNC(func)                                          \
+    if (REAL(func) == 0) {                                               \
+      Report("FATAL: ThreadSanitizer: failed to intercept %s\n", #func); \
+      Die();                                                             \
+    }
+#else
+#  define CHECK_REAL_FUNC(func) DCHECK(REAL(func))
+#endif
+
 #define SCOPED_TSAN_INTERCEPTOR(func, ...)                                \
   SCOPED_INTERCEPTOR_RAW(func, __VA_ARGS__);                              \
-  DCHECK(REAL(func));                                                     \
+  CHECK_REAL_FUNC(func);                                                  \
   if (!thr->is_inited || thr->ignore_interceptors || thr->in_ignored_lib) \
     return REAL(func)(__VA_ARGS__);
 
 #define SCOPED_TSAN_INTERCEPTOR_USER_CALLBACK_START() \
     si.DisableIgnores();
 
 #define SCOPED_TSAN_INTERCEPTOR_USER_CALLBACK_END() \
     si.EnableIgnores();

compiler-rt/test/sanitizer_common/TestCases/Linux/ptrace.cpp

 // RUN: %clangxx -O0 %s -o %t && %run %t
 
 // UNSUPPORTED: android
 
 #include <assert.h>
+#include <elf.h>
 #include <signal.h>
 #include <stdio.h>
+#include <string.h>
 #include <sys/ptrace.h>
 #include <sys/types.h>
+#include <sys/uio.h>
 #include <sys/user.h>
 #include <sys/wait.h>
-#include <sys/uio.h>
 #include <unistd.h>
-#include <elf.h>
 #if __mips64 || __arm__
  #include <asm/ptrace.h>
  #include <sys/procfs.h>
 #endif
 #ifdef __aarch64__
 // GLIBC 2.20+ sys/user does not include asm/ptrace.h
  #include <asm/ptrace.h>
 #endif
#if __x86_64__
     user_fpregs_struct fpregs;
     res = ptrace(PTRACE_GETFPREGS, pid, NULL, &fpregs);
     assert(!res);
     if (fpregs.mxcsr)
       printf("%x\n", fpregs.mxcsr);
 #endif // __x86_64__
 
 #if (__powerpc64__ || __mips64 || __arm__)
-    struct pt_regs regs;
+    // Check that nothing writes out-of-bounds.
+    struct pt_regs regs_buf[4];
+    memset(&regs_buf, 0xcd, sizeof(regs_buf));
+    struct pt_regs &regs = regs_buf[1];
     res = ptrace((enum __ptrace_request)PTRACE_GETREGS, pid, NULL, &regs);
     assert(!res);
+    assert(memcmp(&regs_buf[0], &regs_buf[3], sizeof(regs_buf[3])) == 0);
+    assert(memcmp(&regs_buf[2], &regs_buf[3], sizeof(regs_buf[3])) == 0);
 #if (__powerpc64__)
     if (regs.nip)
       printf("%lx\n", regs.nip);
 #elif (__mips64)
     if (regs.cp0_epc)
     printf("%lx\n", regs.cp0_epc);
 #elif (__arm__)
     if (regs.ARM_pc)