tsan: automatically deflake flaky tests

[dvyukov]

Summary

Add a script that is used to deflake inherently flaky tsan tests.
It is invoked from lit tests as:
$(dirname %s)/deflake.bash %run %t %s
The script runs the target program up to 10 times,
until it produces the necessary output.

If/when it LGTM, I will add it to other tests as well.

[kcc]

test/tsan/deflake.bash

11

some tests runs FileCheck with more than one parameter

[dvyukov]

allow custom FileCheck args

[dvyukov]

PTAL

[samsonov]

test/tsan/deflake.bash

9

What about the %run argument? If it is expanded to anything non-empty, this script won't work.

test/tsan/free_race.c

2

You can add a lit substitution to test/tsan/lit.cfg and instead invoke this "tool" as smth. like %tsan_deflake

[dvyukov]

test/tsan/deflake.bash

9

Can it be expanded to anything non-empty?
I guess it's not possible to pass some abstract way of running a program into a shell script.

test/tsan/free_race.c

2

will try to figure out how to do it

[samsonov]

test/tsan/deflake.bash

9

Well, it can, otherwise %run wouldn't be there. Although, I never tried it (by setting config.emulator). Adding Greg for comments on this.

[garious]

Why is the test flakey?

[dvyukov]

The tests test that ThreadSanitizer finds the data race in particular conditions. However, ThreadSanitizer core algorithm can miss a data race when the racy memory access attempts happen very close to each other (literally simultaneously). This was done intentionally, fixing this would impose significant slowdown and this is not a problem for programs other than unit tests.
So the unit tests suffer from this aspect.

[kcc]

More info if you are curious:
https://groups.google.com/forum/#!topic/thread-sanitizer/KIok3F_b1oI

[garious]

I see, thanks. How about something like:

%try 10 not %run %t | FileCheck %s

Meaning, try $* up to 10 times, capturing stdout/stderr each time. When $* returns zero, write its output to stdout/stderr.

[dvyukov]

Looks good to me.

It does not handle the case when a single test expects several reports. I've found 2 such existing tests:
test/tsan/global_race.cc
test/tsan/inlined_memcpy_race.cc
But both of them just checking several independent things. So I guess if it becomes a problem, we can split them into several tests.

Do you have any idea how to implement %try? Because I don't.

[garious]

Do you have any idea how to implement %try? Because I don't.

For portability, I'd write it in Python using the subprocess module, ensuring shell=False.

[dvyukov]

What? What does it mean to "write %try"? How do I do it and where?

[garious]

You replace "%try" with a call to your script using config.substitutions in "<compiler-rt>/test/tsan/lit.cfg".

[dvyukov]

introduce %deflake macro

[dvyukov]

I've created %deflake macro.
I've named it so that it's purpose is clear, while for %try it's not obvious.
It's still bash. I don't know Python. Can write in Go or C, if you wish :)

[samsonov]

LGTM with a nit

test/tsan/lit.cfg

60

os.path.join(os.path.dirname(__file__), "deflake.bash")

[garious]

How about: %S/deflake.bash %run %t | FileCheck %s

[dvyukov]

All others equal I would prefer to stick with already implemented suggestion about lit.cfg macro.

[dvyukov]

os.path.join(os.path.dirname(file), "deflake.bash")

Done

Committed in rev 209898.

test/tsan/deflake.bash

+#!/usr/bin/env bash
+# This script is used to deflake inherently flaky tsan tests.
+# It is invoked from lit tests as:
+# %deflake mybinary
+# which is then substituted by lit to:
+# $(dirname %s)/deflake.bash mybinary
+# The script runs the target program up to 10 times,
+# until it fails (i.e. produces a race report).
+
+for i in $(seq 1 10); do
+  OUT=`$@ 2>&1`
+  if [[ $? != 0 ]]; then
+    echo "$OUT"
+    exit 0
+  fi
+done
+exit 1

test/tsan/free_race.c

 // RUN: %clang_tsan -O1 %s -o %t
-// RUN: not %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-NOZUPP
+// RUN: %deflake %run %t | FileCheck %s --check-prefix=CHECK-NOZUPP
 // RUN: TSAN_OPTIONS="suppressions=%s.supp print_suppressions=1" %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-SUPP
 
 #include <pthread.h>
 #include <stdlib.h>
 #include <stdio.h>
 #include <stddef.h>
 #include <unistd.h>
 

test/tsan/lit.cfg

def build_invocation(compile_flags):
   return " " + " ".join([config.clang] + compile_flags) + " "
 
 config.substitutions.append( ("%clang_tsan ", build_invocation(clang_tsan_cflags)) )
 config.substitutions.append( ("%clangxx_tsan ", build_invocation(clang_tsan_cxxflags)) )
 
 # Define CHECK-%os to check for OS-dependent output.
 config.substitutions.append( ('CHECK-%os', ("CHECK-" + config.host_os)))
 
+config.substitutions.append( ("%deflake ", "$(dirname %s)/deflake.bash ") )
+
 # Default test suffixes.
 config.suffixes = ['.c', '.cc', '.cpp']
 
 # ThreadSanitizer tests are currently supported on Linux only.
 if config.host_os not in ['Linux']:
   config.unsupported = True

test/tsan/simple_race.c

-// RUN: %clang_tsan -O1 %s -o %t && not %run %t 2>&1 | FileCheck %s
+// RUN: %clang_tsan -O1 %s -o %t && %deflake %run %t | FileCheck %s
 #include <pthread.h>
 #include <stdio.h>
 #include <unistd.h>
 
 int Global;
 
 void *Thread1(void *x) {
   sleep(1);
int main() {
   pthread_create(&t[0], NULL, Thread1, NULL);
   pthread_create(&t[1], NULL, Thread2, NULL);
   pthread_join(t[0], NULL);
   pthread_join(t[1], NULL);
   return 0;
 }
 
 // CHECK: WARNING: ThreadSanitizer: data race
+

test/tsan/simple_race.cc

-// RUN: %clangxx_tsan -O1 %s -o %t && not %run %t 2>&1 | FileCheck %s
+// RUN: %clangxx_tsan -O1 %s -o %t && %deflake %run %t 2>&1 | FileCheck %s
 #include <pthread.h>
 #include <stdio.h>
 
 int Global;
 
 void *Thread1(void *x) {
   Global++;
   return NULL;