sanitizer_common: initialize sanitizer runtimes lazily from signal interceptors

[dvyukov]

Summary

Currently if a program calls sigaction very early (before non-lazy sanitizer
initialization, in particular if .preinit_array initialization is not enabled),
then sigaction will wrongly fail since the interceptor is not initialized yet.

In all other interceptors we do lazy runtime initialization for this reason,
but we don't do it in the signal interceptors.
Do lazy runtime initialization in signal interceptors as well.

[dvyukov]

compiler-rt/test/lit.common.cfg.py

932

Needed to move it for -shared-libsan in the test to work.

[dvyukov]

Vitaly, I've landed this to catch bot failures earlier. But if you have any comments, I will fix them in a separate commit.

[Caslyn]

compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp

1

Hello - this test is failing on clang-linux builders in Fuchsia. For ex: https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-linux-arm64/b8775675662976612977/overview.

Below is the error output, it appears the issue is that %run doesn't expand to anything when running this script on Linux builds:

Script:
--
: 'RUN: at line 1';      /b/s/w/ir/x/w/llvm_build/./bin/clang  --driver-mode=g++ -fsanitize=thread   -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta   -fsanitize=undefined -shared-libsan /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp -o /b/s/w/ir/x/w/llvm_build/runtimes/runtimes-aarch64-unknown-linux-gnu-bins/compiler-rt/test/ubsan/ThreadSanitizer-aarch64/TestCases/Misc/Linux/Output/sigaction.cpp.tmp &&  /b/s/w/ir/x/w/llvm_build/runtimes/runtimes-aarch64-unknown-linux-gnu-bins/compiler-rt/test/ubsan/ThreadSanitizer-aarch64/TestCases/Misc/Linux/Output/sigaction.cpp.tmp 2>&1 | FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp
--
Exit Code: 2

Command Output (stderr):
--
FileCheck error: '<stdin>' is empty.
FileCheck command line:  FileCheck /b/s/w/ir/x/w/llvm-llvm-project/compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp

[vitalybuka]

In D155188#4497741, @dvyukov wrote:

Vitaly, I've landed this to catch bot failures earlier. But if you have any comments, I will fix them in a separate commit.

LGTM

[Caslyn]

compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp

1

it appears the issue is that...

Sorry, I don't think that's it. The invocation only fails when running with UBSan-ThreadSanitizer-aarch64 :: TestCases/Misc/Linux/sigaction.cpp.

[dvyukov]

Thanks for reporting.
I've tested similar command for x86 (my host) and it worked.
I've sent a change with XFAIL for aarch64 to make bots green: https://reviews.llvm.org/D155272

[mysterymath]

compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp

1

I dug into this a bit more, and found that there's a segfault in the sanitizer code leading up to the fprintf in ctor(). It dereferences some value that it picks out of the GOT, but that isn't mapped in the binary. I'm not entirely sure what it's doing, or why this selectively doesn't work on the Fuchsia builders.

[pgousseau]

Hi @vitalybuka @dvyukov

This seems to have flagged an issue with some test configurations, it seems if tsan's so is built with ld.bfd instead of ld.lld then the tsan shared library gets corrupted, causing this test to fail.
I am wondering if it is a know issue using tsan + gnu ld ?

cmake -G Ninja -DLLVM_INCLUDE_EXAMPLES=OFF -DCLANG_ENABLE_ARCMT=OFF -DPYTHON_EXECUTABLE=/usr/bin/python3 -DCLANG_ENABLE_CLANGD=OFF -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCOMPILER_RT_TEST_USE_LLD=0 -DLLVM_BUILD_RUNTIME=ON -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;cross-project-tests;lld" -DCMAKE_BUILD_TYPE=Release -S ./llvm -B build && ninja -C build -j8 check-ubsan

******************** TEST 'UBSan-ThreadSanitizer-x86_64 :: TestCases/Misc/Linux/sigaction.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';      /home/pierre/llvm-project/build/./bin/clang  --driver-mode=g++ -fsanitize=thread  -m64  -fsanitize=undefined -shared-libsan /home/pierre/llvm-project/compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp -o /home/pierre/llvm-project/build/projects/compiler-rt/test/ubsan/ThreadSanitizer-x86_64/TestCases/Misc/Linux/Output/sigaction.cpp.tmp &&  /home/pierre/llvm-project/build/projects/compiler-rt/test/ubsan/ThreadSanitizer-x86_64/TestCases/Misc/Linux/Output/sigaction.cpp.tmp 2>&1 | FileCheck /home/pierre/llvm-project/compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp
--
Exit Code: 1

Command Output (stderr):
--
/usr/bin/ld: /home/pierre/llvm-project/build/lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so: siglongjmp: invalid version 21 (max 0)
/usr/bin/ld: /home/pierre/llvm-project/build/lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so: error adding symbols: bad value
clang: error: linker command failed with exit code 1 (use -v to see invocation)

llvm-readelf -a lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so > /dev/null
llvm-readelf: warning: 'lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so': SHT_GNU_versym section refers to a version index 21 which is missing
llvm-readelf: warning: 'lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so': unable to get a version for entry 754 of SHT_GNU_versym section with index 4: SHT_GNU_versym section refers to a version index 21 which is missing
llvm-readelf: warning: 'lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so': unable to get a version for entry 2617 of SHT_GNU_versym section with index 4: SHT_GNU_versym section refers to a version index 21 which is missing
llvm-readelf: warning: 'lib/clang/17/lib/x86_64-unknown-linux-gnu/libclang_rt.tsan.so': unable to get a version for entry 3154 of SHT_GNU_versym section with index 4: SHT_GNU_versym section refers to a version index 21 which is missing

Many thanks!

Pierre

[dvyukov]

I am wondering if it is a know issue using tsan + gnu ld ?

No, I wasn't aware.

These looks like existing bugs that were exposed by the test due to use of shared runtime.
It looks like the shared runtime is not well tested in general.

[arichardson]

compiler-rt/test/lit.common.cfg.py

939

This breaks testing only the builtins library since config.bits is only set in memprof/asan/asan_abi.
AttributeError: 'TestingConfig' object has no attribute 'bits'.

If I understand this correctly, libasan_dir will expand to <GCC>/../lib(64), i.e. /usr/lib. Would it be possible to use gcc -print-file-name=libasan.so instead?

❯ gcc -print-file-name=libasan.so      
/usr/lib/gcc/x86_64-linux-gnu/12/libasan.so
❯ gcc -print-file-name=libasan.so -m32
/usr/lib/gcc/x86_64-linux-gnu/12/32/libasan.so

[dvyukov]

Would it be possible to use gcc -print-file-name=libasan.so instead?

In clang the library is called libclang_rt.asan.so.
Also if the compiler is host, won't this give path to the host asan, while we need the local just built library.

[arichardson]

In D155188#4530886, @dvyukov wrote:

Would it be possible to use gcc -print-file-name=libasan.so instead?

In clang the library is called libclang_rt.asan.so.
Also if the compiler is host, won't this give path to the host asan, while we need the local just built library.

It currently gives the path to the libraries in the same tree as the GCC installation since the branch that triggers the error is guarded by compiler==gcc. Do we actually need this logic if the compiler rt libpath was already added in the line above?

[arichardson]

In D155188#4532036, @arichardson wrote:

In D155188#4530886, @dvyukov wrote:

Would it be possible to use gcc -print-file-name=libasan.so instead?

In clang the library is called libclang_rt.asan.so.
Also if the compiler is host, won't this give path to the host asan, while we need the local just built library.

It currently gives the path to the libraries in the same tree as the GCC installation since the branch that triggers the error is guarded by compiler==gcc. Do we actually need this logic if the compiler rt libpath was already added in the line above?

I just tried building compiler-rt with GCC again and am still seeing this error. Would be nice if we could fix this, even if it's just getattr(config, "bits", "") rather than some more correct fix.

    libasan_dir = os.path.join(gcc_dir, "..", "lib" + config.bits)
                                                      ^^^^^^^^^^^
AttributeError: 'TestingConfig' object has no attribute 'bits'

compiler-rt/lib/asan/asan_interceptors.cpp

do {                                                                         \
     return munmap_interceptor(REAL(munmap), addr, sz);                         \
   } while (false)
 
 #if CAN_SANITIZE_LEAKS
 #define COMMON_INTERCEPTOR_STRERROR()                       \
   __lsan::ScopedInterceptorDisabler disabler
 #endif
 
+#define SIGNAL_INTERCEPTOR_ENTER() ENSURE_ASAN_INITED()
+
 #include "sanitizer_common/sanitizer_common_interceptors.inc"
 #include "sanitizer_common/sanitizer_signal_interceptors.inc"
 
 // Syscall interceptors don't have contexts, we don't support suppressions
 // for them.
 #define COMMON_SYSCALL_PRE_READ_RANGE(p, s) ASAN_READ_RANGE(nullptr, p, s)
 #define COMMON_SYSCALL_PRE_WRITE_RANGE(p, s) ASAN_WRITE_RANGE(nullptr, p, s)
 #define COMMON_SYSCALL_POST_READ_RANGE(p, s) \

compiler-rt/lib/lsan/lsan_interceptors.cpp

 DEFINE_REAL_PTHREAD_FUNCTIONS
 
 INTERCEPTOR(void, _exit, int status) {
   if (status == 0 && HasReportedLeaks()) status = common_flags()->exitcode;
   REAL(_exit)(status);
 }
 
 #define COMMON_INTERCEPT_FUNCTION(name) INTERCEPT_FUNCTION(name)
+#define SIGNAL_INTERCEPTOR_ENTER() ENSURE_LSAN_INITED
 #include "sanitizer_common/sanitizer_signal_interceptors.inc"
 
 #endif  // SANITIZER_POSIX
 
 namespace __lsan {
 
 void InitializeInterceptors() {
   // Fuchsia doesn't use interceptors that require any setup.

compiler-rt/lib/msan/msan_interceptors.cpp

 
 #define SIGNAL_INTERCEPTOR_SIGNAL_IMPL(func, signo, handler) \
   {                                                          \
     handler = signal_impl(signo, handler);                   \
     InterceptorScope interceptor_scope;                      \
     return REAL(func)(signo, handler);                       \
   }
 
+#define SIGNAL_INTERCEPTOR_ENTER() ENSURE_MSAN_INITED()
+
 #include "sanitizer_common/sanitizer_signal_interceptors.inc"
 
 static int sigaction_impl(int signo, const __sanitizer_sigaction *act,
                           __sanitizer_sigaction *oldact) {
   ENSURE_MSAN_INITED();
   if (signo <= 0 || signo >= kMaxSignals) {
     errno = errno_EINVAL;
     return -1;

compiler-rt/lib/sanitizer_common/sanitizer_signal_interceptors.inc

#  define SIGNAL_INTERCEPTOR_SIGACTION_IMPL(signum, act, oldact)              \
         return -1;                                                            \
       }                                                                       \
       return REAL(sigaction_symname)(signum, act, oldact);                    \
     }
 #endif
 
 #if SANITIZER_INTERCEPT_BSD_SIGNAL
 INTERCEPTOR(uptr, bsd_signal, int signum, uptr handler) {
+  SIGNAL_INTERCEPTOR_ENTER();
   if (GetHandleSignalMode(signum) == kHandleSignalExclusive) return 0;
   SIGNAL_INTERCEPTOR_SIGNAL_IMPL(bsd_signal, signum, handler);
 }
 #define INIT_BSD_SIGNAL COMMON_INTERCEPT_FUNCTION(bsd_signal)
 #else  // SANITIZER_INTERCEPT_BSD_SIGNAL
 #define INIT_BSD_SIGNAL
 #endif  // SANITIZER_INTERCEPT_BSD_SIGNAL
 
 #if SANITIZER_INTERCEPT_SIGNAL_AND_SIGACTION
 INTERCEPTOR(uptr, signal, int signum, uptr handler) {
+  SIGNAL_INTERCEPTOR_ENTER();
   if (GetHandleSignalMode(signum) == kHandleSignalExclusive)
     return (uptr) nullptr;
   SIGNAL_INTERCEPTOR_SIGNAL_IMPL(signal, signum, handler);
 }
 #define INIT_SIGNAL COMMON_INTERCEPT_FUNCTION(signal)
 
 INTERCEPTOR(int, sigaction_symname, int signum,
             const __sanitizer_sigaction *act, __sanitizer_sigaction *oldact) {
+  SIGNAL_INTERCEPTOR_ENTER();
   if (GetHandleSignalMode(signum) == kHandleSignalExclusive) {
     if (!oldact) return 0;
     act = nullptr;
   }
   SIGNAL_INTERCEPTOR_SIGACTION_IMPL(signum, act, oldact);
 }
 #define INIT_SIGACTION COMMON_INTERCEPT_FUNCTION(sigaction_symname)
 

compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp

static __sanitizer_sighandler_ptr signal_impl(int sig,
                                               __sanitizer_sighandler_ptr h);
 
 #define SIGNAL_INTERCEPTOR_SIGACTION_IMPL(signo, act, oldact) \
   { return sigaction_impl(signo, act, oldact); }
 
 #define SIGNAL_INTERCEPTOR_SIGNAL_IMPL(func, signo, handler) \
   { return (uptr)signal_impl(signo, (__sanitizer_sighandler_ptr)handler); }
 
+#define SIGNAL_INTERCEPTOR_ENTER() LazyInitialize(cur_thread_init())
+
 #include "sanitizer_common/sanitizer_signal_interceptors.inc"
 
 int sigaction_impl(int sig, const __sanitizer_sigaction *act,
                    __sanitizer_sigaction *old) {
   // Note: if we call REAL(sigaction) directly for any reason without proxying
   // the signal handler through sighandler, very bad things will happen.
   // The handler will run synchronously and corrupt tsan per-thread state.
   SCOPED_INTERCEPTOR_RAW(sigaction, sig, act, old);

compiler-rt/lib/ubsan/ubsan_signals_standalone.cpp

 #if SANITIZER_FUCHSIA || SANITIZER_ANDROID
 
 namespace __ubsan {
 void InitializeDeadlySignals() {}
 }
 
 #else
 
+namespace __ubsan {
+void InitializeDeadlySignals();
+} // namespace __ubsan
+
 #define COMMON_INTERCEPT_FUNCTION(name) INTERCEPT_FUNCTION(name)
+#define SIGNAL_INTERCEPTOR_ENTER() __ubsan::InitializeDeadlySignals()
 #include "sanitizer_common/sanitizer_signal_interceptors.inc"
 
 // TODO(yln): Temporary workaround. Will be removed.
 void ubsan_GetStackTrace(BufferedStackTrace *stack, uptr max_depth,
                          uptr pc, uptr bp, void *context, bool fast);
 
 namespace __ubsan {
 

compiler-rt/test/asan/lit.cfg.py

def get_required_attr(config, attr_name):
     if attr_value == None:
         lit_config.fatal(
             "No attribute %r in test configuration! You may need to run "
             "tests from your build directory or add this attribute "
             "to lit.site.cfg.py " % attr_name
         )
     return attr_value
 
-
-def push_dynamic_library_lookup_path(config, new_path):
-    if platform.system() == "Windows":
-        dynamic_library_lookup_var = "PATH"
-    elif platform.system() == "Darwin":
-        dynamic_library_lookup_var = "DYLD_LIBRARY_PATH"
-    else:
-        dynamic_library_lookup_var = "LD_LIBRARY_PATH"
-
-    new_ld_library_path = os.path.pathsep.join(
-        (new_path, config.environment.get(dynamic_library_lookup_var, ""))
-    )
-    config.environment[dynamic_library_lookup_var] = new_ld_library_path
-
-    if platform.system() == "FreeBSD":
-        dynamic_library_lookup_var = "LD_32_LIBRARY_PATH"
-        new_ld_32_library_path = os.path.pathsep.join(
-            (new_path, config.environment.get(dynamic_library_lookup_var, ""))
-        )
-        config.environment[dynamic_library_lookup_var] = new_ld_32_library_path
-
-    if platform.system() == "SunOS":
-        dynamic_library_lookup_var = "LD_LIBRARY_PATH_32"
-        new_ld_library_path_32 = os.path.pathsep.join(
-            (new_path, config.environment.get(dynamic_library_lookup_var, ""))
-        )
-        config.environment[dynamic_library_lookup_var] = new_ld_library_path_32
-
-        dynamic_library_lookup_var = "LD_LIBRARY_PATH_64"
-        new_ld_library_path_64 = os.path.pathsep.join(
-            (new_path, config.environment.get(dynamic_library_lookup_var, ""))
-        )
-        config.environment[dynamic_library_lookup_var] = new_ld_library_path_64
-
-
 # Setup config name.
 config.name = "AddressSanitizer" + config.name_suffix
 
 # Platform-specific default ASAN_OPTIONS for lit tests.
 default_asan_opts = list(config.default_sanitizer_opts)
 
 # On Darwin, leak checking is not enabled by default. Enable on macOS
 # tests to prevent regressions

 if (
     leak_detection_android
     or leak_detection_linux
     or leak_detection_mac
     or leak_detection_netbsd
 ):
     config.available_features.add("leak-detection")
 
-# Set LD_LIBRARY_PATH to pick dynamic runtime up properly.
-push_dynamic_library_lookup_path(config, config.compiler_rt_libdir)
-
-# GCC-ASan uses dynamic runtime by default.
-if config.compiler_id == "GNU":
-    gcc_dir = os.path.dirname(config.clang)
-    libasan_dir = os.path.join(gcc_dir, "..", "lib" + config.bits)
-    push_dynamic_library_lookup_path(config, libasan_dir)
-
 # Add the RT libdir to PATH directly so that we can successfully run the gtest
 # binary to list its tests.
 if config.host_os == "Windows" and config.asan_dynamic:
     os.environ["PATH"] = os.path.pathsep.join(
         [config.compiler_rt_libdir, os.environ.get("PATH", "")]
     )
 
 # Default test suffixes.

compiler-rt/test/lit.common.cfg.py

if config.host_os == "Darwin":
         if not os.path.exists(runtime_dir):
             lit_config.fatal(f"Path reported by clang does not exist: {runtime_dir}")
         return os.path.realpath(runtime_dir)
 
     lit_config.warning("Failed to determine compiler's runtime directory")
     return None
 
 
+def push_dynamic_library_lookup_path(config, new_path):
+    if platform.system() == "Windows":
+        dynamic_library_lookup_var = "PATH"
+    elif platform.system() == "Darwin":
+        dynamic_library_lookup_var = "DYLD_LIBRARY_PATH"
+    else:
+        dynamic_library_lookup_var = "LD_LIBRARY_PATH"
+
+    new_ld_library_path = os.path.pathsep.join(
+        (new_path, config.environment.get(dynamic_library_lookup_var, ""))
+    )
+    config.environment[dynamic_library_lookup_var] = new_ld_library_path
+
+    if platform.system() == "FreeBSD":
+        dynamic_library_lookup_var = "LD_32_LIBRARY_PATH"
+        new_ld_32_library_path = os.path.pathsep.join(
+            (new_path, config.environment.get(dynamic_library_lookup_var, ""))
+        )
+        config.environment[dynamic_library_lookup_var] = new_ld_32_library_path
+
+    if platform.system() == "SunOS":
+        dynamic_library_lookup_var = "LD_LIBRARY_PATH_32"
+        new_ld_library_path_32 = os.path.pathsep.join(
+            (new_path, config.environment.get(dynamic_library_lookup_var, ""))
+        )
+        config.environment[dynamic_library_lookup_var] = new_ld_library_path_32
+
+        dynamic_library_lookup_var = "LD_LIBRARY_PATH_64"
+        new_ld_library_path_64 = os.path.pathsep.join(
+            (new_path, config.environment.get(dynamic_library_lookup_var, ""))
+        )
+        config.environment[dynamic_library_lookup_var] = new_ld_library_path_64
+
+
 # Choose between lit's internal shell pipeline runner and a real shell.  If
 # LIT_USE_INTERNAL_SHELL is in the environment, we use that as an override.
 use_lit_shell = os.environ.get("LIT_USE_INTERNAL_SHELL")
 if use_lit_shell:
     # 0 is external, "" is default, and everything else is internal.
     execute_external = use_lit_shell == "0"
 else:
     # Otherwise we default to internal on Windows and external elsewhere, as
if config.host_os == "Darwin":
     )
 
 # It is not realistically possible to account for all options that could
 # possibly be present in system and user configuration files, so disable
 # default configs for the test runs. In particular, anything hardening
 # related is likely to cause issues with sanitizer tests, because it may
 # preempt something we're looking to trap (e.g. _FORTIFY_SOURCE vs our ASAN).
 config.environment["CLANG_NO_DEFAULT_CONFIG"] = "1"
+
+# Set LD_LIBRARY_PATH to pick dynamic runtime up properly.
+push_dynamic_library_lookup_path(config, config.compiler_rt_libdir)
+
+# GCC-ASan uses dynamic runtime by default.
+if config.compiler_id == "GNU":
+    gcc_dir = os.path.dirname(config.clang)
+    libasan_dir = os.path.join(gcc_dir, "..", "lib" + config.bits)
+    push_dynamic_library_lookup_path(config, libasan_dir)

compiler-rt/test/ubsan/TestCases/Misc/Linux/sigaction.cpp

+// RUN: %clangxx -fsanitize=undefined -shared-libsan %s -o %t && %run %t 2>&1 | FileCheck %s
+
+// Ensure ubsan runtime/interceptors are lazily initialized if called early.
+
+#include <assert.h>
+#include <signal.h>
+#include <stdio.h>
+
+__attribute__((constructor(0))) void ctor() {
+  fprintf(stderr, "INIT\n");
+  struct sigaction old;
+  assert(!sigaction(SIGSEGV, nullptr, &old));
+};
+
+int main() {
+  fprintf(stderr, "DONE\n");
+  return 0;
+}
+
+// CHECK: INIT
+// CHECK: DONE