Add libFuzzer shared object build output

[IanPudney]

Summary

This change adds a CMake rule to produce shared object versions of libFuzzer (no-main). Unlike the static library versions, these shared libraries do not have a copy of libc++ statically linked in. Doing so is theoretically possible for x86-64 (via hacks); however, it is not possible for i386 because i386 does not suppot mixing position-independent and non-position-independent code in the same library.

[kcc]

Do we need a version for 32-bit at all?
Not having a private version of libc++ is likely to cause subtle stability issues.

(Matt, please help me with the review here)

[IanPudney]

Sticking just with x86_64 is possible; I actually have the code for that here, but it's a bit ugly:
https://reviews.llvm.org/differential/diff/281467/

However, what stability issues could dynamic linking cause? The change to privately link libc++ was only added in 2019, so things must have worked well before then:
https://github.com/llvm/llvm-project/commit/66c60d9d714bed88765c1575fbf57e7582fd27a3

I'm fine with whichever version you prefer.

[kcc]

In D84947#2186106, @IanPudney wrote:

Sticking just with x86_64 is possible; I actually have the code for that here, but it's a bit ugly:
https://reviews.llvm.org/differential/diff/281467/

If you don't need 32-bit .so, let's just have it in 64-bit version (and this fact documented)

However, what stability issues could dynamic linking cause? The change to privately link libc++ was only added in 2019, so things must have worked well before then:
https://github.com/llvm/llvm-project/commit/66c60d9d714bed88765c1575fbf57e7582fd27a3

Things didn't work well and we had a few workarounds, some were since reverted.
The problem with a non-private libc++ is that the common libc++ is instrumented for coverage.
Since libFuzzer itself uses libc++, the coverage instrumentation code is executed while inside libFuzzer,
screwing everything up.

I'm fine with whichever version you prefer.

[IanPudney]

Updated to skip building i386 output, but to statically link libc++.

[morehouse]

LGTM

[morehouse]

When I patch this in locally and run ninja check-fuzzer, I get a linking error for both i386 and x86_64:

FAILED: lib/clang/12.0.0/lib/linux/libclang_rt.fuzzer_no_main-i386.so                                                          
...
/usr/bin/ld: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerDriver.cpp.o: in function `fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned int))':
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:130: undefined reference to `pthread_create'           
/usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:130: undefined reference to `pthread_create'
/usr/bin/ld: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o: in function `fuzzer::FuzzWithFork(fuzzer::Random&, fuzzer::FuzzingOptions const&, std::vector<std::__cxx11::basic_stri
ng<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_stri
ng<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, int)':
/usr/local/google/home/mascasa/code/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp:(.text+0xd6d): undefined reference to `pthread_create'
clang: error: linker command failed with exit code 1 (use -v to see invocation)                                                
[655/3462] Linking CXX shared library lib/clang/12.0.0/lib/linux/libclang_rt.fuzzer_no_main-x86_64.so                          


FAILED: lib/clang/12.0.0/lib/linux/libclang_rt.fuzzer_no_main-x86_64.so                                                        
...                                                                           
/usr/bin/ld: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerDriver.cpp.o: in function `fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long))':
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:130: undefined reference to `pthread_create'           
/usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:130: undefined reference to `pthread_create'
/usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:130: undefined reference to `pthread_create'
/usr/bin/ld: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerFork.cpp.o: in function `fuzzer::FuzzWithFork(fuzzer::Random&, fuzzer::FuzzingOptions const&, std::vector<std::__cxx11::basic_st
ring<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_st
ring<char, std::char_traits<char>, std::allocator<char> >, fuzzer::fuzzer_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, int)':
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/thread:130: undefined reference to `pthread_create'

Ian, can you please take a look?

[IanPudney]

So it looks like the bug in question was pre-existing, and this change just exposes it. Specifically, when building with Ninja instead of Makefiles, the code path for statically linking libc++ isn't actually hit. Rather, the fallthrough "normal" code path is hit. I've confirmed this by verifying that the assemblies produced before my change with Ninja still want regular libc++.

[IanPudney]

Adjusted rule to skip -z,defs when building under Ninja.

[morehouse]

compiler-rt/lib/fuzzer/CMakeLists.txt

61

Nit: Can we rename this to LIBFUZZER_SHARED_LINK_FLAGS and move it down next to the case where it is used?

[morehouse]

I'll land this tomorrow.

[IanPudney]

While this now builds with Ninja, there are some rtti-related issues with linking.

[IanPudney]

Issue fixed.

[jfb]

compiler-rt/lib/fuzzer/CMakeLists.txt

192

This is broken on non-Linux and non-Fuchsia architectures which don't have libstdc++, for example Darwin. Can you please fix or revert?

[morehouse]

compiler-rt/lib/fuzzer/CMakeLists.txt

192

Would it be sufficient to add the following?

if (DARWIN)
  list(APPEND LIBFUZZER_SHARED_LINK_LIBS "-lc++")
else()
  list(APPEND LIBFUZZER_SHARED_LINK_LIBS "-lstdc++")
endif()

Or are there more platforms that are missing libstdc++?

[lebedev.ri]

compiler-rt/lib/fuzzer/CMakeLists.txt

192

Why is this particular line hardcoding some particular C++ STD implementation in the first place?
Ignoring distro defaults, one can always pass -stdlib=libc++ to clang.

[morehouse]

compiler-rt/lib/fuzzer/CMakeLists.txt

194

This line was also failing on Android, where there is no pthreads library.

[IanPudney]

compiler-rt/lib/fuzzer/CMakeLists.txt

192

Passing -stdlib=libc++ into the CFLAGS of the rule doesn't solve the problem. The .so links, but at runtime it complains about missing symbols like _ZTVN10cxxabiv120si_class_type_infoE.

192

Added this in https://reviews.llvm.org/D85339. However, I don't have a Darwin machine to test with.

[IanPudney]

Updated to use -lc++ instead of lstdc++ on Darwin.

[lebedev.ri]

compiler-rt/lib/fuzzer/CMakeLists.txt

195

This is too fragile, basically duplicates defaults from clang/lib/Driver/ToolChains
Is there no way to make compiler do the right thing?

[IanPudney]

compiler-rt/lib/fuzzer/CMakeLists.txt

195

I don't suppose you have any suggestions? I don't even know why libstdc++ isn't being linked in automatically.

compiler-rt/lib/fuzzer/CMakeLists.txt

CHECK_CXX_SOURCE_COMPILES("
   int main() {
   return 0;
   }
   " HAS_THREAD_LOCAL)
 
 set(LIBFUZZER_CFLAGS ${SANITIZER_COMMON_CFLAGS})
 
 if(OS_NAME MATCHES "Linux|Fuchsia" AND
    COMPILER_RT_LIBCXX_PATH AND
    COMPILER_RT_LIBCXXABI_PATH)
   list(APPEND LIBFUZZER_CFLAGS -nostdinc++ -D_LIBCPP_ABI_VERSION=Fuzzer)
   # Remove -stdlib= which is unused when passing -nostdinc++.
   string(REGEX REPLACE "-stdlib=[a-zA-Z+]*" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
 elseif(TARGET cxx-headers OR HAVE_LIBCXX)
   set(LIBFUZZER_DEPS cxx-headers)
 endif()
 
foreach(arch ${FUZZER_SUPPORTED_ARCH})
     add_dependencies(RTfuzzer.${arch} libcxx_fuzzer_${arch}-build)
     target_compile_options(RTfuzzer_main.${arch} PRIVATE -isystem ${LIBCXX_${arch}_PREFIX}/include/c++/v1)
     add_dependencies(RTfuzzer_main.${arch} libcxx_fuzzer_${arch}-build)
     target_compile_options(RTfuzzer_interceptors.${arch} PRIVATE -isystem ${LIBCXX_${arch}_PREFIX}/include/c++/v1)
     add_dependencies(RTfuzzer_interceptors.${arch} libcxx_fuzzer_${arch}-build)
     partially_link_libcxx(fuzzer_no_main ${LIBCXX_${arch}_PREFIX} ${arch})
     partially_link_libcxx(fuzzer_interceptors ${LIBCXX_${arch}_PREFIX} ${arch})
     partially_link_libcxx(fuzzer ${LIBCXX_${arch}_PREFIX} ${arch})
+    if(NOT ${arch} MATCHES "i386") # i386 unsupported for .so version.
+      add_custom_command(
+        OUTPUT clang_rt.fuzzer_no_main-${arch}.so
+        DEPENDS clang_rt.fuzzer_no_main-${arch}
+       COMMAND ${CMAKE_CXX_COMPILER} ${EMULATION_ARGUMENT} -Wl,--whole-archive -rdynamic "$<TARGET_LINKER_FILE:clang_rt.fuzzer_no_main-${arch}>" -Wl,--no-whole-archive -shared -fPIC -o "$<TARGET_FILE_DIR:clang_rt.fuzzer_no_main-${arch}>/libclang_rt.fuzzer_no_main-${arch}.so"
+        COMMENT "Building clang_rt.fuzzer_no_main-${arch}.so"
+      )
+      get_compiler_rt_install_dir(${arch} install_dir)
+      install(FILES "$<TARGET_FILE_DIR:clang_rt.fuzzer_no_main-${arch}>/libclang_rt.fuzzer_no_main-${arch}.so"
+        DESTINATION ${install_dir}
+      )
+      add_custom_target(
+        clang_rt.fuzzer_no_main-${arch}-so ALL
+        DEPENDS clang_rt.fuzzer_no_main-${arch}.so
+      )
+    endif()
   endforeach()
+else()
+
+set(LIBFUZZER_SHARED_LINK_FLAGS ${SANITIZER_COMMON_LINK_FLAGS})
+
+set(LIBFUZZER_SHARED_LINK_LIBS ${SANITIZER_COMMON_LINK_LIBS})
+list(APPEND LIBFUZZER_SHARED_LINK_LIBS "-lstdc++")
+list(APPEND LIBFUZZER_SHARED_LINK_LIBS "-lm")
+list(APPEND LIBFUZZER_SHARED_LINK_LIBS "-lpthread")
+
+# If we aren't statically linking libc++ into the fuzzer, we can build the shared object directly
+add_compiler_rt_runtime(clang_rt.fuzzer_no_main
+  SHARED
+  OS ${FUZZER_SUPPORTED_OS}
+  ARCHS ${FUZZER_SUPPORTED_ARCH}
+  OBJECT_LIBS RTfuzzer
+  CFLAGS ${LIBFUZZER_CFLAGS}
+  LINK_FLAGS ${LIBFUZZER_SHARED_LINK_FLAGS}
+  LINK_LIBS ${LIBFUZZER_SHARED_LINK_LIBS}
+  PARENT_TARGET fuzzer)
+
 endif()
 
 if(COMPILER_RT_INCLUDE_TESTS)
   add_subdirectory(tests)
 endif()

llvm/docs/LibFuzzer.rst

.. code-block:: c++
    extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv) {
     ReadAndMaybeModify(argc, argv);
     return 0;
    }
 
 Using libFuzzer as a library
 ----------------------------
 If the code being fuzzed must provide its own `main`, it's possible to
-invoke libFuzzer as a library. Be sure to pass ``-fsanitize=fuzzer-no-link``
+invoke libFuzzer as a library. Static linking is available on all platforms
+supported by libFuzzer; however, dynamic linking is not available on
+certain platforms (notably 32-bit x86 Linux).
+
+When using libFuzzer as a library, be sure to pass ``-fsanitize=fuzzer-no-link``
 during compilation, and link your binary against the no-main version of
 libFuzzer. On Linux installations, this is typically located at:
 
 .. code-block:: bash
 
   /usr/lib/<llvm-version>/lib/clang/<clang-version>/lib/linux/libclang_rt.fuzzer_no_main-<architecture>.a
+  /usr/lib/<llvm-version>/lib/clang/<clang-version>/lib/linux/libclang_rt.fuzzer_no_main-<architecture>.so
 
 If building libFuzzer from source, this is located at the following path
 in the build output directory:
 
 .. code-block:: bash
 
   lib/linux/libclang_rt.fuzzer_no_main-<architecture>.a
+  lib/linux/libclang_rt.fuzzer_no_main-<architecture>.so
 
 From here, the code can do whatever setup it requires, and when it's ready
 to start fuzzing, it can call `LLVMFuzzerRunDriver`, passing in the program
 arguments and a callback. This callback is invoked just like
 `LLVMFuzzerTestOneInput`, and has the same signature.
 
 .. code-block:: c++
 
   extern "C" int LLVMFuzzerRunDriver(int *argc, char ***argv,
                     int (*UserCb)(const uint8_t *Data, size_t Size));
 
-
-
 Leaks
 -----
 
 Binaries built with AddressSanitizer_ or LeakSanitizer_ will try to detect
 memory leaks at the process shutdown.
 For in-process fuzzing this is inconvenient
 since the fuzzer needs to report a leak with a reproducer as soon as the leaky
 mutation is found. However, running full leak detection after every mutation