[sanitizer] On OS X, verify that interceptors work and abort if not

[kubamracek]

Summary

On OS X 10.11+, we have "automatic interceptors", so we don't need to use DYLD_INSERT_LIBRARIES when launching instrumented programs. However, non-instrumented programs that load TSan late (e.g. via dlopen) are currently broken, as TSan will still try to initialize, but the program will crash/hang at random places (because the interceptors don't work). This patch adds an explicit check that interceptors are working, and if not, it aborts and prints out an error message suggesting to explicitly use DYLD_INSERT_LIBRARIES.

[zaks.anna]

Thanks for adding the check this is very useful!

Looks like this would work for all sanitizers, including ASan. Is that correct?

[kubamracek]

In D18121#373925, @zaks.anna wrote:

Thanks for adding the check this is very useful!

Looks like this would work for all sanitizers, including ASan. Is that correct?

Yes, this should work for all sanitizers.

[filcab]

lib/sanitizer_common/sanitizer_mac.cc

636

Wouldn't if (info.dli_fbase != info_pthread_create.dli_fbase) work as well? Seems easier to read.

test/tsan/Darwin/dlopen.cc

13

I really think this would be better in the lit.cfg files (and then dealt with via REQUIRES/UNSUPPORTED), instead of having tests depend on more and more available executables/specific shells.
Having said that, I don't see it as a very big problem because it's a Darwin-specific test (Still hinders to do that, anyway :-) )

[kubamracek]

lib/sanitizer_common/sanitizer_mac.cc

636

It seems to work, but I don't think it's guaranteed by the API. Since we're doing this once only, it's not a big deal performance-wise, and I think readability is fine here – we *are* actually comparing two strings.

[glider]

LGTM

test/tsan/Darwin/dlopen.cc

30

I'm curious how does the dynamic linker behave in the presence of another main().
Maybe rename it to foo() just to be sure?

[kubamracek]

Changed the test to detect OS X version in lit.cfg instead, and to produce a regular shared library instead of two executables with two main functions.

I'm curious how does the dynamic linker behave in the presence of another main().
Maybe rename it to foo() just to be sure?

I believe this is supposed to work fine because of OS X two-level namespaces, and because I'm performing the dlopen/dladdr lookup in a single module (so there is no clash between the symbols). Anyway, I changed the test to generate a regular shared library with a foo symbol.

[samsonov]

LGTM

[kubamracek]

I had to revert this and make more changes, but I can't reopen this revision, so I instead submitted a new patch to http://reviews.llvm.org/D18212.

lib/sanitizer_common/sanitizer_mac.cc

if (DyldNeedsEnvVariable() && !lib_is_in_env) {
     Report("ERROR: The process is launched without DYLD_INSERT_LIBRARIES, "
            "which is required for the sanitizer to work. We tried to set the "
            "environment variable and re-execute itself, but execv() failed, "
            "possibly because of sandbox restrictions. Make sure to launch the "
            "executable with:\n%s=%s\n", kDyldInsertLibraries, new_env);
     CHECK("execv failed" && 0);
   }
 
+  // Verify that interceptors really work.  We'll use dlsym to locate
+  // "pthread_create", if interceptors are working, it should really point to
+  // "wrap_pthread_create" within our own dylib.
+  Dl_info info_pthread_create;
+  void *dlopen_addr = dlsym(RTLD_DEFAULT, "pthread_create");
+  CHECK(dladdr(dlopen_addr, &info_pthread_create));
+  if (internal_strcmp(info.dli_fname, info_pthread_create.dli_fname) != 0) {
+    Report(
+        "ERROR: Interceptors are not working. This may be because %s is "
+        "loaded too late (e.g. via dlopen). Please launch the executable "
+        "with:\n%s=%s\n",
+        SanitizerToolName, kDyldInsertLibraries, info.dli_fname);
+    CHECK("interceptors not installed" && 0);
+  }
+
   if (!lib_is_in_env)
     return;
 
   // DYLD_INSERT_LIBRARIES is set and contains the runtime library. Let's remove
   // the dylib from the environment variable, because interceptors are installed
   // and we don't want our children to inherit the variable.
 
   uptr env_name_len = internal_strlen(kDyldInsertLibraries);

test/lit.common.cfg

 if config.has_lld:
   config.available_features.add('lld')
 
 if config.can_symbolize:
   config.available_features.add('can-symbolize')
 
 lit.util.usePlatformSdkOnDarwin(config, lit_config)
 
+if config.host_os == 'Darwin':
+  ld_cmd = subprocess.Popen(["bash", "-c", "sw_vers -productVersion | awk -F '.' '{print $1 \".\" $2}'"], stdout = subprocess.PIPE)
+  ld_out = ld_cmd.stdout.read().decode()
+  ld_cmd.wait()
+  osx_version = float(ld_out)
+  if osx_version >= 10.11:
+    config.available_features.add('osx-autointerception')
+
 sancovcc_path = os.path.join(llvm_tools_dir, "sancov") 
 if os.path.exists(sancovcc_path):
   config.available_features.add("has_sancovcc")
   config.substitutions.append( ("%sancovcc ", sancovcc_path) )
 
 def is_darwin_lto_supported():
   return os.path.exists(os.path.join(config.llvm_shlib_dir, 'libLTO.dylib'))
 

test/tsan/Darwin/dlopen.cc

+// Checks that on OS X 10.11+ (where we do not re-exec anymore, because
+// interceptors work automatically), dlopen'ing a TSanified library from a
+// non-instrumented program exits with a user-friendly message.
+
+// REQUIRES: osx-autointerception
+
+// RUN: %clangxx_tsan %s -o %t.so -shared -DSHARED_LIB
+// RUN: %clangxx_tsan -fno-sanitize=thread %s -o %t
+
+// RUN: TSAN_DYLIB_PATH=`%clangxx_tsan %s -### 2>&1 \
+// RUN:   | grep "libclang_rt.tsan_osx_dynamic.dylib" \
+// RUN:   | sed -e 's/.*"\(.*libclang_rt.tsan_osx_dynamic.dylib\)".*/\1/'`
+
+// Launching a non-instrumented binary that dlopen's an instrumented library should fail.
+// RUN: not %run %t-noninstr %t.so 2>&1 | FileCheck %s --check-prefix=CHECK-FAIL
+// Launching a non-instrumented binary with an explicit DYLD_INSERT_LIBRARIES should work.
+// RUN: DYLD_INSERT_LIBRARIES=$TSAN_DYLIB_PATH %run %t-noninstr %t.so 2>&1 | FileCheck %s
+
+#include <dlfcn.h>
+#include <pthread.h>
+#include <stdio.h>
+
+#if defined(SHARED_LIB)
+void foo() {
+  fprintf(stderr, "Hello world.\n");
+}
+#else  // defined(SHARED_LIB)
+int main(int argc, char *argv[]) {
+  void *handle = dlopen(argv[1], RTLD_NOW);
+  fprintf(stderr, "handle = %p\n", handle);
+  void (*foo)() = (void (*)())dlsym(handle, "foo");
+  fprintf(stderr, "foo = %p\n", foo);
+  foo();
+}
+#endif  // defined(SHARED_LIB)
+
+// CHECK: Hello world.
+// CHECK-NOT: ERROR: Interceptors are not working.
+
+// CHECK-FAIL-NOT: Hello world.
+// CHECK-FAIL: ERROR: Interceptors are not working.