This is an archive of the discontinued LLVM Phabricator instance.

[GWP-ASan] Build stack_trace_compressor_fuzzer.
ClosedPublic

Authored by hctim on Aug 20 2019, 1:35 PM.

Details

Summary

Flips the switch to build stack_trace_compressor_fuzzer. This was recently
temporarily disabled in rL369079 as it was breaking the sanitizer buildbots.

My diagnosis of the problem is that on clang-only bootstrap builds, we build
gwp_asan before libfuzzer. This causes a discrepancy when the clang driver
attempts to link libclang_rt.fuzzer* as CMake doesn't see a dependency there.

I've (hopefully) fixed the issue by adding a direct dependency for the fuzz
target so CMake can resolve the build order properly. As part of this, the
libFuzzer 'fuzzer' target has to be discovered before the declaration of the
fuzz target.

pcc@ for mild review + notification as buildcop.

Event Timeline

hctim created this revision.Aug 20 2019, 1:35 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptAug 20 2019, 1:35 PM
Herald added subscribers: llvm-commits, Restricted Project, mgorny. · View Herald Transcript
pcc added inline comments.Aug 20 2019, 5:43 PM
compiler-rt/lib/gwp_asan/CMakeLists.txt
115–116

Shouldn't the order of these be reversed, i.e.

add_dependencies(stack_trace_compressor_fuzzer gwp_asan fuzzer)

https://cmake.org/cmake/help/latest/command/add_dependencies.html

I think that means that you don't need to move the target ordering around.

hctim marked 2 inline comments as done.Aug 20 2019, 5:58 PM
hctim added inline comments.
compiler-rt/lib/gwp_asan/CMakeLists.txt
115–116

Thanks for the catch - looks like this resolves the issue. Slight change is that the desire is for the fuzzer to be built with ninja gwp_asan, so the add_dependencies(gwp_asan stack_trace_compressor_fuzzer) is WAI :)

hctim updated this revision to Diff 216300.Aug 20 2019, 5:58 PM
hctim marked an inline comment as done.
  • Reverse order of fuzzer dependencies.
pcc added inline comments.Aug 20 2019, 6:12 PM
compiler-rt/lib/gwp_asan/CMakeLists.txt
115–116

That dependency edge seems mildly surprising to me, but it's your project.

That aside though, how is this linking successfully without a dependency from the fuzzer to gwp_asan?

pcc accepted this revision.Aug 20 2019, 6:13 PM

LGTM

compiler-rt/lib/gwp_asan/CMakeLists.txt
115–116

Oh, I see the ${GWP_ASAN_SOURCES} now. Never mind.

This revision is now accepted and ready to land.Aug 20 2019, 6:13 PM
This revision was automatically updated to reflect the committed changes.
phosek added a subscriber: phosek.Aug 21 2019, 4:16 PM

This seems to have broke our toolchain build, specifically the stage that (cross-)compiles runtimes for i386-linux-gnu:

[697/697] Linking CXX executable compiler-rt/lib/gwp_asan/stack_trace_compressor_fuzzer
FAILED: compiler-rt/lib/gwp_asan/stack_trace_compressor_fuzzer 
: && /b/s/w/ir/k/recipe_cleanup/clangB3DKrD/llvm_build_dir/./bin/clang++ --target=i386-unknown-linux-gnu --sysroot=/b/s/w/ir/k/cipd/linux-i386  -resource-dir=/b/s/w/ir/k/recipe_cleanup/clangB3DKrD/llvm_build_dir/tools/clang/stage2-bins/./lib/clang/10.0.0 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++14 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fdebug-prefix-map=/b/s/w/ir/k/recipe_cleanup/clangB3DKrD/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-i386-unknown-linux-gnu-bins=../recipe_cleanup/clangB3DKrD/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-i386-unknown-linux-gnu-bins -fdebug-prefix-map=/b/s/w/ir/k/llvm-project/= -no-canonical-prefixes -Wall -std=c++14 -Wno-unused-parameter -O2 -g -DNDEBUG  -resource-dir=/b/s/w/ir/k/recipe_cleanup/clangB3DKrD/llvm_build_dir/tools/clang/stage2-bins/./lib/clang/10.0.0 -fuse-ld=lld -fuse-ld=lld -Wl,--color-diagnostics   -fsanitize=fuzzer compiler-rt/lib/gwp_asan/CMakeFiles/stack_trace_compressor_fuzzer.dir/stack_trace_compressor_fuzzer.cpp.o compiler-rt/lib/gwp_asan/CMakeFiles/stack_trace_compressor_fuzzer.dir/platform_specific/guarded_pool_allocator_posix.cpp.o compiler-rt/lib/gwp_asan/CMakeFiles/stack_trace_compressor_fuzzer.dir/platform_specific/mutex_posix.cpp.o compiler-rt/lib/gwp_asan/CMakeFiles/stack_trace_compressor_fuzzer.dir/guarded_pool_allocator.cpp.o compiler-rt/lib/gwp_asan/CMakeFiles/stack_trace_compressor_fuzzer.dir/random.cpp.o compiler-rt/lib/gwp_asan/CMakeFiles/stack_trace_compressor_fuzzer.dir/stack_trace_compressor.cpp.o  -o compiler-rt/lib/gwp_asan/stack_trace_compressor_fuzzer   && :
ld.lld: error: unable to find library -lc++
ld.lld: error: unable to find library -lc++
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
FAILED: runtimes/runtimes-i386-unknown-linux-gnu-stamps/runtimes-i386-unknown-linux-gnu-build

Is this supposed to be linking libc++?

vitalybuka added inline comments.
compiler-rt/trunk/lib/gwp_asan/CMakeLists.txt
104 ↗(On Diff #216430)

We should not build these fuzzers if host compiler has no libFuzzer.

vitalybuka added inline comments.Aug 23 2019, 6:10 PM
compiler-rt/trunk/lib/gwp_asan/CMakeLists.txt
104 ↗(On Diff #216430)

maybe do the same as clang/tools/clang-format/CMakeLists.txt

if( LLVM_LIB_FUZZING_ENGINE OR LLVM_USE_SANITIZE_COVERAGE )

I think this breaks compiler-rt build if the host compiler does not have compiler-rt present, cc @pree-jackie

@hctim This breaks the debug build of llvm when the host compiler doesn't build with compiler-rt. I build llvm/clang (release build) as a host compiler, to build llvm,compiler-rt (debug) build. Unfortunately, while building the debug build, the process tries to find the already installed compiler-rt of the host compiler (which is not present). Do you have any guidelines?

You can also check the build process error log : https://pastebin.com/Q76H9FW9

Thanks

@pree-jackie i'll take a look, thanks for letting me know

hctim added a comment.Aug 26 2019, 3:04 PM

Update: there's a slight problem with the dependency chain that I encountered when whipping up a quick fix:

stack_trace_compressor_fuzzer requires:

  • libFuzzer
  • asan/ubsan/msan (to be useful)

libFuzzer needs COMPILER_RT_HAS_SANITIZER_COMMON=True
COMPILER_RT_HAS_SANITIZER_COMMON needs LLVM_USE_SANITIZER=False

... so basically the fuzz target requires sanitization, but providing sanitization would cancel the build of the parent gwp_asan target...

Let me have a think. If I can't hack together a solution soon I'll just comment out the target to unbreak you.