On FreeBSD, add -pthread to the flags for dynamic ASan tests

[dim]

Summary

When the dynamic ASan tests are linked on FreeBSD, -pthread must be
added to the link flags (so that libthr.so is put in the NEEDED section
of the executable), otherwise the initial call to pthread_key_create()
in AsanTSDInit() will return ENOSYS, and ASan will fail to initialize.

I'm not completely sure if this is the correct way to add the flag, but
it worked for me...

[eugenis]

Does asan runtime library have libthr.so in DT_NEEDED? Would adding that help?

If FreeBSD requires that everything built with ASan uses -pthread, then perhaps it is best done in the driver.

[krytarowski]

What exactly breaks? What symbols are missing? How are you testing it?

check-asan-dynamic on NetBSD/amd64 works fine. Modulo around 3 tests that are unresearched.

[dim]

In D39254#905625, @eugenis wrote:

Does asan runtime library have libthr.so in DT_NEEDED? Would adding that help?

Yes it has, but that doesn't help. It looks like the AsanTSDInit() function is called before the pthread_key_create() stub which returns ENOSYS is replace by the 'real' function from libthr.so.

If FreeBSD requires that everything built with ASan uses -pthread, then perhaps it is best done in the driver.

Well, not everything, I only encountered this problem when using the dynamic ASan library. Though it seems that some of the static ASan test cases are linked with -pthread already, for example:

/home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/./bin/clang --driver-mode=g++ -fsanitize=address -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls -gline-tables-only -m32  -O0 /share/dim/src/llvm/trunk/runtimes/compiler-rt/test/asan/TestCases/Posix/stack-use-after-return.cc -pthread -o /home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/runtimes/runtimes-bins/compiler-rt/test/asan/I386FreeBSDConfig/TestCases/Posix/Output/stack-use-after-return.cc.tmp && env ASAN_OPTIONS=detect_stack_use_after_return=1 not  /home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/runtimes/runtimes-bins/compiler-rt/test/asan/I386FreeBSDConfig/TestCases/Posix/Output/stack-use-after-return.cc.tmp 2>&1 | FileCheck /share/dim/src/llvm/trunk/runtimes/compiler-rt/test/asan/TestCases/Posix/stack-use-after-return.cc

but this does not apply to all of them, for reasons that are unclear to me.

In D39254#905639, @krytarowski wrote:

What exactly breaks?

All dynamic ASan tests fail with:

==7913==AddressSanitizer CHECK failed: /share/dim/src/llvm/trunk/runtimes/compiler-rt/lib/asan/asan_posix.cc:49 "((0)) == ((pthread_key_create(&tsd_key, destructor)))" (0x0, 0x4e)

This is because AsanTSDInit() calls pthread_key_create() while libthr.so is not loaded, and that causes pthread_key_create() to always return ENOSYS.

What symbols are missing?

No symbols are missing, as far as I can see.

Now are you testing it?

I run ninja check-all.

check-asan-dynamic on NetBSD/amd64 works fine. Modulo around 3 tests that are unresearched.

I guess NetBSD doesn't use libthr.so? Are all pthread functions folded into libc.so?

[eugenis]

What changes when the binary starts depending on libthr? Before that, we had
binary -> asan -> libthr
Now we have
binary -> asan -> libthr

-> libthr

This should affect neither symbol lookup order nor constructor order.

[krytarowski]

In D39254#905671, @dim wrote:

In D39254#905639, @krytarowski wrote:

What exactly breaks?

All dynamic ASan tests fail with:

==7913==AddressSanitizer CHECK failed: /share/dim/src/llvm/trunk/runtimes/compiler-rt/lib/asan/asan_posix.cc:49 "((0)) == ((pthread_key_create(&tsd_key, destructor)))" (0x0, 0x4e)

This is because AsanTSDInit() calls pthread_key_create() while libthr.so is not loaded, and that causes pthread_key_create() to always return ENOSYS.

OK, I will put this in TODO to research for later.

What symbols are missing?

No symbols are missing, as far as I can see.

Now are you testing it?

I run ninja check-all.

It would be nice to narrow it down to a more specific check target. I could verify it locally.

Can you reproduce it with check-asan-dynamic?

check-asan-dynamic on NetBSD/amd64 works fine. Modulo around 3 tests that are unresearched.

I guess NetBSD doesn't use libthr.so?

libthr.so does not exist on NetBSD.

Are all pthread functions folded into libc.so?

No. There are stubs as weak references.

[dim]

In D39254#905688, @eugenis wrote:

What changes when the binary starts depending on libthr? Before that, we had
binary -> asan -> libthr
Now we have
binary -> asan -> libthr

-> libthr

This should affect neither symbol lookup order nor constructor order.

I agree, and I would think so too, but in practice it apparently does... The way the dynamic linker fixes this up is rather magic to me, and I don't know the internal details. @emaste any idea?

In D39254#905707, @krytarowski wrote:

...

Can you reproduce it with check-asan-dynamic?

Any of the dynamic asan tests should trigger it. For example the runtimes/compiler-rt/test/asan/TestCases/Posix/asprintf.cc, which is the first one in the list.

check-asan-dynamic on NetBSD/amd64 works fine. Modulo around 3 tests that are unresearched.

I guess NetBSD doesn't use libthr.so?

libthr.so does not exist on NetBSD.

Are all pthread functions folded into libc.so?

No. There are stubs as weak references.

FreeBSD's libc.so also has these stubs, but where does the actual pthread implementation live in NetBSD then?

[dim]

Btw for me, ninja check-asan-dynamic gives ninja: error: unknown target 'check-asan-dynamic'. I guess that must be run from a specific location?

[krytarowski]

In D39254#905727, @dim wrote:

FreeBSD's libc.so also has these stubs, but where does the actual pthread implementation live in NetBSD then?

In libpthread.

[krytarowski]

In D39254#905728, @dim wrote:

Btw for me, ninja check-asan-dynamic gives ninja: error: unknown target 'check-asan-dynamic'. I guess that must be run from a specific location?

$ pwd
/public/llvm-build
$ make help|grep check-asan-dynamic
... check-asan-dynamic

[dim]

In D39254#905730, @krytarowski wrote:

In D39254#905728, @dim wrote:

Btw for me, ninja check-asan-dynamic gives ninja: error: unknown target 'check-asan-dynamic'. I guess that must be run from a specific location?

$ pwd
/public/llvm-build
$ make help|grep check-asan-dynamic
... check-asan-dynamic

Strange, I don't use the make generator, but I also can't find the string "check-asan-dynamic" in the full checkout (which contains llvm, clang and compiler-rt). For me, these special targets never seem to work :)

[eugenis]

$ grep check-asan-dynamic * -RI
test/asan/CMakeLists.txt: add_lit_testsuite(check-asan-dynamic
test/asan/CMakeLists.txt: set_target_properties(check-asan-dynamic

[dim]

So if I *don't* add the -pthread flag, the test case executables have the following needed entries:

$ readelf -dW /home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/runtimes/runtimes-bins/compiler-rt/test/asan/I386FreeBSDDynamicConfig/TestCases/Posix/Output/deep_call_stack.cc.tmp

Dynamic section at offset 0xef0 contains 29 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libclang_rt.asan-i386.so]
 0x00000001 (NEEDED)                     Shared library: [libc++.so.1]
 0x00000001 (NEEDED)                     Shared library: [libcxxrt.so.1]
 0x00000001 (NEEDED)                     Shared library: [libm.so.5]
 0x00000001 (NEEDED)                     Shared library: [libc.so.7]

while the libclang_rt.asan-i386.so has:

$ readelf -dW ~/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/lib/clang/6.0.0/lib/freebsd/libclang_rt.asan-i386.so

Dynamic section at offset 0xb995c contains 30 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.7]
 0x00000001 (NEEDED)                     Shared library: [librt.so.1]
 0x00000001 (NEEDED)                     Shared library: [libm.so.5]
 0x00000001 (NEEDED)                     Shared library: [libthr.so.3]
 0x0000000e (SONAME)                     Library soname: [libclang_rt.asan-i386.so]

When I run ldd on such a test case, the order shown is:

libclang_rt.asan-i386.so => /home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/lib/clang/6.0.0/lib/freebsd/libclang_rt.asan-i386.so (0x28071000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x285c7000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x28689000)
libm.so.5 => /lib/libm.so.5 (0x286a3000)
libc.so.7 => /lib/libc.so.7 (0x286ce000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x28884000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2896f000)
librt.so.1 => /usr/lib/librt.so.1 (0x28984000)
libthr.so.3 => /lib/libthr.so.3 (0x2898a000)

E.g. libthr.so.3 is at the end of the list.

However, If I do add the -pthread flag, the order in the test case executables becomes a little different:

readelf -dW /home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/runtimes/runtimes-bins/compiler-rt/test/asan/I386FreeBSDDynamicConfig/TestCases/Posix/Output/deep_call_stack.cc.tmp

Dynamic section at offset 0xee8 contains 30 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libclang_rt.asan-i386.so]
 0x00000001 (NEEDED)                     Shared library: [libc++.so.1]
 0x00000001 (NEEDED)                     Shared library: [libcxxrt.so.1]
 0x00000001 (NEEDED)                     Shared library: [libm.so.5]
 0x00000001 (NEEDED)                     Shared library: [libthr.so.3]
 0x00000001 (NEEDED)                     Shared library: [libc.so.7]

and then the ldd order becomes:

libclang_rt.asan-i386.so => /home/dim/obj/llvm-316264-trunk-freebsd12-i386-ninja-rel-1/lib/clang/6.0.0/lib/freebsd/libclang_rt.asan-i386.so (0x28071000)
libc++.so.1 => /usr/lib/libc++.so.1 (0x285c7000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x28689000)
libm.so.5 => /lib/libm.so.5 (0x286a3000)
libthr.so.3 => /lib/libthr.so.3 (0x286ce000)
libc.so.7 => /lib/libc.so.7 (0x286f3000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x288a9000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x28994000)
librt.so.1 => /usr/lib/librt.so.1 (0x289a9000)

Here, libthr.so.3 is loaded before libc.so.3, and apparently that works better, somehow.

[eugenis]

Then the problem must be in the linking of asan-rt: libthr should go before libc. We use -nodefaultlibs, so this order is explicitly defined in CMakeLists.txt.

[krytarowski]

$ ldd ./lib/clang/6.0.0/lib/netbsd/libclang_rt.asan-x86_64.so                                                                
./lib/clang/6.0.0/lib/netbsd/libclang_rt.asan-x86_64.so:
        -lstdc++.8 => /usr/lib/libstdc++.so.8
        -lm.0 => /usr/lib/libm.so.0
        -lc.12 => /usr/lib/libc.so.12
        -lgcc_s.1 => /usr/lib/libgcc_s.so.1
        -lrt.1 => /usr/lib/librt.so.1
        -lpthread.1 => /usr/lib/libpthread.so.1

[dim]

In D39254#905769, @eugenis wrote:

Then the problem must be in the linking of asan-rt: libthr should go before libc. We use -nodefaultlibs, so this order is explicitly defined in CMakeLists.txt.

Aha, you are most likely right. If I compile a small test case .so file (with just int foo(void) { return pthread_key_create(&key, NULL); } in it), the resulting .so indeed has libthr before libc:

0x0000000000000001 (NEEDED)             Shared library: [libthr.so.3]
0x0000000000000001 (NEEDED)             Shared library: [libc.so.7]

[krytarowski]

$ readelf -dW ./lib/clang/6.0.0/lib/netbsd/libclang_rt.asan-x86_64.so |grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.12]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.1]

test/asan/lit.cfg

clang_asan_static_cflags = (["-fsanitize=address",
                             config.debug_info_flags + target_cflags)
 if config.target_arch == 's390x':
   clang_asan_static_cflags.append("-mbackchain")
 clang_asan_static_cxxflags = config.cxx_mode_flags + clang_asan_static_cflags
 
 asan_dynamic_flags = []
 if config.asan_dynamic:
   asan_dynamic_flags = ["-shared-libasan"]
+  # On FreeBSD, we need to add -pthread to ensure pthread functions are available.
+  if platform.system() == 'FreeBSD':
+    asan_dynamic_flags += ['-pthread']
   # On Windows, we need to simulate "clang-cl /MD" on the clang driver side.
   if platform.system() == 'Windows':
     asan_dynamic_flags += ["-D_MT", "-D_DLL", "-Wl,-nodefaultlib:libcmt,-defaultlib:msvcrt,-defaultlib:oldnames"]
   config.available_features.add("asan-dynamic-runtime")
 else:
   config.available_features.add("asan-static-runtime")
 clang_asan_cflags = clang_asan_static_cflags + asan_dynamic_flags
 clang_asan_cxxflags = clang_asan_static_cxxflags + asan_dynamic_flags