Implement pthread_exit() interceptor for Thread sanitizer

[yuri]

Summary

This change is preparation for fiber support.

[dvyukov]

Committed in 353385
http://llvm.org/viewvc/llvm-project?view=revision&revision=353385

[vitalybuka]

http://lab.llvm.org:8011/builders/sanitizer-ppc64be-linux/builds/11338

: 'RUN: at line 1';      /home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/clang_build/./bin/clang  -fsanitize=thread -Wall  -m64  -gline-tables-only -I/home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/llvm/projects/compiler-rt/test/tsan/../ -O1 /home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/llvm/projects/compiler-rt/test/tsan/thread_exit.c -o /home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/clang_build/projects/compiler-rt/test/tsan/POWERPC64Config/Output/thread_exit.c.tmp &&  /home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/clang_build/projects/compiler-rt/test/tsan/POWERPC64Config/Output/thread_exit.c.tmp 2>&1 | FileCheck /home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/llvm/projects/compiler-rt/test/tsan/thread_exit.c
--
Exit Code: 1

Command Output (stderr):
--
/home/buildbots/ppc64be-sanitizer/sanitizer-ppc64be/build/llvm/projects/compiler-rt/test/tsan/thread_exit.c:25:11: error: CHECK: expected string not found in input
// CHECK: PASS
          ^
<stdin>:1:1: note: scanning from here
ThreadSanitizer:DEADLYSIGNAL
^
<stdin>:1:4: note: possible intended match here
ThreadSanitizer:DEADLYSIGNAL
   ^

[yuri]

Can someone with access to ppc64be system help to investigate the issue?

1. Check if the test works without any sanitizers.
2. What versions of gcc and glibc are installed on the host?
3. Run test under gdb and collect stack trace at crash point.

#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>

int var;

void *Thread(void *x) {
  pthread_exit(&var);
  return 0;
}

int main() {
  pthread_t t;
  pthread_create(&t, 0, Thread, 0);
  void *retval = 0;
  pthread_join(t, &retval);
  if (retval != &var) {
    fprintf(stderr, "Unexpected return value\n");
    exit(1);
  }
  fprintf(stderr, "PASS\n");
  return 0;
}

[dvyukov]

• Forwarded message ---------

From: Alex L
Date: Fri, Feb 8, 2019 at 1:17 AM
Subject: Re: [compiler-rt] r353385 - tsan: Implement pthread_exit() interceptor for Thread sanitizer

Hi Dmitry,

It looks like your change has caused a failure of the Darwin/dispatch_main.mm TSAN test on our CI:

• TEST 'ThreadSanitizer-x86_64-iossim :: Darwin/dispatch_main.mm' FAILED ****

Script:

: 'RUN: at line 4'; /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/llvm/projects/compiler-rt/test/sanitizer_common/ios_commands/iossim_compile.py /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/clang-build/./bin/clang -fsanitize=thread -Wall -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator11.4.sdk -gline-tables-only -I/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/llvm/projects/compiler-rt/test/tsan/../ /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/llvm/projects/compiler-rt/test/tsan/Darwin/dispatch_main.mm -o /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/clang-build/tools/clang/runtime/compiler-rt-bins/test/tsan/IOSSimX86_64Config/Darwin/Output/dispatch_main.mm.tmp -framework Foundation

: 'RUN: at line 5'; /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/llvm/projects/compiler-rt/test/sanitizer_common/ios_commands/iossim_run.py /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/clang-build/tools/clang/runtime/compiler-rt-bins/test/tsan/IOSSimX86_64Config/Darwin/Output/dispatch_main.mm.tmp 2>&1 | FileCheck /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-expensive/llvm/projects/compiler-rt/test/tsan/Darwin/dispatch_main.mm

Exit Code: 66

---

Here's the full log:

http://lab.llvm.org:8080/green/job/clang-stage1-cmake-RA-expensive/13539/console

I narrowed it down to your commit.
You can reproduce it by running 'check-tsan' on macOS with the following CMake options "-DCMAKE_BUILD_TYPE=Release" "-DLLVM_ENABLE_ASSERTIONS=On" "-DLLVM_ENABLE_EXPENSIVE_CHECK=ON".

Could you please take a look?

Thanks,
Alex

[dvyukov]

One thing that I noticed is that pthread_exit is a noreturn function and SCOPED_TSAN_INTERCEPTOR creates a local object with destructor. At the very least this leads to confusing code where readers assumption is that the destructor will run, but it will not.
Perhaps we should try to remove the SCOPED_TSAN_INTERCEPTOR entirely as the CHECK that we want to add here does not need it.

[yuri]

In D57876#1390146, @dvyukov wrote:

It looks like your change has caused a failure of the Darwin/dispatch_main.mm TSAN test on our CI:

I will try to reproduce it myself and report my findings later

[yuri]

In D57876#1390150, @dvyukov wrote:

One thing that I noticed is that pthread_exit is a noreturn function and SCOPED_TSAN_INTERCEPTOR creates a local object with destructor. At the very least this leads to confusing code where readers assumption is that the destructor will run, but it will not.
Perhaps we should try to remove the SCOPED_TSAN_INTERCEPTOR entirely as the CHECK that we want to add here does not need it.

Let's fix after resolution of test failures

[yuri]

Fix for macOS is in D57963

[vitalybuka]

In D57876#1390843, @yuri wrote:

Fix for macOS is in D57963

Removing SCOPED_TSAN_INTERCEPTOR does not fix PPC
http://lab.llvm.org:8011/builders/sanitizer-ppc64be-linux/builds/11358/steps/check-tsan%20in%20gcc%20build/logs/stdio

[vitalybuka]

Removed interceptor in r353604

[vitalybuka]

It still crashes even without interceptor

[yuri]

In D57876#1391757, @vitalybuka wrote:

It still crashes even without interceptor

Looks like stack unwinding (pthread_exit depends on it) is completly broken on this platform.

I suggest to restore interceptor code and disable this test or mark it as expected failure on ppc64be.

[vitalybuka]

https://reviews.llvm.org/rCRT353627

[dvyukov]

I've seen some fixes has landed over the weekend. Is this fixed now? Or some breakages still remain?

[yuri]

In D57876#1392414, @dvyukov wrote:

I've seen some fixes has landed over the weekend. Is this fixed now? Or some breakages still remain?

macOS is fixed.
Test on ppc64 is disabled — pthread_exit does not work there without any interceptors.

[dvyukov]

In D57876#1392448, @yuri wrote:

In D57876#1392414, @dvyukov wrote:

I've seen some fixes has landed over the weekend. Is this fixed now? Or some breakages still remain?

macOS is fixed.
Test on ppc64 is disabled — pthread_exit does not work there without any interceptors.

Great! Thanks for taking care of this.

lib/tsan/rtl/tsan_interceptors.cc

TSAN_INTERCEPTOR(int, pthread_detach, void *th) {
   int tid = ThreadTid(thr, pc, (uptr)th);
   int res = REAL(pthread_detach)(th);
   if (res == 0) {
     ThreadDetach(thr, pc, tid);
   }
   return res;
 }
 
+TSAN_INTERCEPTOR(void, pthread_exit, void *retval) {
+  SCOPED_TSAN_INTERCEPTOR(pthread_exit, retval);
+  REAL(pthread_exit)(retval);
+}
+
 #if SANITIZER_LINUX
 TSAN_INTERCEPTOR(int, pthread_tryjoin_np, void *th, void **ret) {
   SCOPED_TSAN_INTERCEPTOR(pthread_tryjoin_np, th, ret);
   int tid = ThreadTid(thr, pc, (uptr)th);
   ThreadIgnoreBegin(thr, pc);
   int res = REAL(pthread_tryjoin_np)(th, ret);
   ThreadIgnoreEnd(thr, pc);
   if (res == 0)
#endif
 
   TSAN_INTERCEPT(strcpy);  // NOLINT
   TSAN_INTERCEPT(strncpy);
   TSAN_INTERCEPT(strdup);
 
   TSAN_INTERCEPT(pthread_create);
   TSAN_INTERCEPT(pthread_join);
   TSAN_INTERCEPT(pthread_detach);
+  TSAN_INTERCEPT(pthread_exit);
   #if SANITIZER_LINUX
   TSAN_INTERCEPT(pthread_tryjoin_np);
   TSAN_INTERCEPT(pthread_timedjoin_np);
   #endif
 
   TSAN_INTERCEPT_VER(pthread_cond_init, PTHREAD_ABI_BASE);
   TSAN_INTERCEPT_VER(pthread_cond_signal, PTHREAD_ABI_BASE);
   TSAN_INTERCEPT_VER(pthread_cond_broadcast, PTHREAD_ABI_BASE);

test/tsan/thread_exit.c

+// RUN: %clang_tsan -O1 %s -o %t && %run %t 2>&1 | FileCheck %s
+#include "test.h"
+
+int var;
+
+void *Thread(void *x) {
+  pthread_exit(&var);
+  return 0;
+}
+
+int main() {
+  pthread_t t;
+  pthread_create(&t, 0, Thread, 0);
+  void *retval = 0;
+  pthread_join(t, &retval);
+  if (retval != &var) {
+    fprintf(stderr, "Unexpected return value\n");
+    exit(1);
+  }
+  fprintf(stderr, "PASS\n");
+  return 0;
+}
+
+// CHECK-NOT: WARNING: ThreadSanitizer:
+// CHECK: PASS