This is an archive of the discontinued LLVM Phabricator instance.

Implement `-fsanitize-coverage-whitelist` and `-fsanitize-coverage-blacklist` for clang
ClosedPublic

Authored by tuktuk on Jun 20 2019, 11:36 AM.

Details

Summary

This commit adds two command-line options to clang.
These options let the user decide which functions will receive SanitizerCoverage instrumentation.
This is most useful in the libFuzzer use case, where it enables targeted coverage-guided fuzzing.

Patch by Yannis Juglaret of DGA-MI, Rennes, France

libFuzzer tests its target against an evolving corpus, and relies on SanitizerCoverage instrumentation to collect the code coverage information that drives corpus evolution. Currently, libFuzzer collects such information for all functions of the target under test, and adds to the corpus every mutated sample that finds a new code coverage path in any function of the target. We propose instead to let the user specify which functions' code coverage information is relevant for building the upcoming fuzzing campaign's corpus. To this end, we add two new command line options for clang, enabling targeted coverage-guided fuzzing with libFuzzer. We see targeted coverage guided fuzzing as a simple way to leverage libFuzzer for big targets with thousands of functions or multiple dependencies. We publish this patch as work from DGA-MI of Rennes, France, with proper authorization from the hierarchy.

Targeted coverage-guided fuzzing can accelerate bug finding for two reasons. First, the compiler will avoid costly instrumentation for non-relevant functions, accelerating fuzzer execution for each call to any of these functions. Second, the built fuzzer will produce and use a more accurate corpus, because it will not keep the samples that find new coverage paths in non-relevant functions.

The two new command line options are -fsanitize-coverage-whitelist and -fsanitize-coverage-blacklist. They accept files in the same format as the existing -fsanitize-blacklist option https://clang.llvm.org/docs/SanitizerSpecialCaseList.html#format. The new options influence SanitizerCoverage so that it will only instrument a subset of the functions in the target. We explain these options in detail in clang/docs/SanitizerCoverage.rst.

Consider now the woff2 fuzzing example from the libFuzzer tutorial https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md. We are aware that we cannot conclude much from this example because mutating compressed data is generally a bad idea, but let us use it anyway as an illustration for its simplicity. Let us use an empty blacklist together with one of the three following whitelists:

# (a)
src:*
fun:*

# (b)
src:SRC/*
fun:*

# (c)
src:SRC/src/woff2_dec.cc
fun:*

Running the built fuzzers shows how many instrumentation points the compiler adds, the fuzzer will output XXX PCs. Whitelist (a) is the instrument-everything whitelist, it produces 11912 instrumentation points. Whitelist (b) focuses coverage to instrument woff2 source code only, ignoring the dependency code for brotli (de)compression; it produces 3984 instrumented instrumentation points. Whitelist (c) focuses coverage to only instrument functions in the main file that deals with WOFF2 to TTF conversion, resulting in 1056 instrumentation points.

For experimentation purposes, we ran each fuzzer approximately 100 times, single process, with the initial corpus provided in the tutorial. We let the fuzzer run until it either found the heap buffer overflow or went out of memory. On this simple example, whitelists (b) and (c) found the heap buffer overflow more reliably and 5x faster than whitelist (a). The average execution times when finding the heap buffer overflow were as follows: (a) 904 s, (b) 156 s, and (c) 176 s.

We explain these results by the fact that WOFF2 to TTF conversion calls the brotli decompression algorithm's functions, which are mostly irrelevant for finding bugs in WOFF2 font reconstruction but nevertheless instrumented and used by whitelist (a) to guide fuzzing. This results in longer execution time for these functions and a partially irrelevant corpus. Contrary to whitelist (a), whitelists (b) and (c) will execute brotli-related functions without instrumentation overhead, and ignore new code paths found in them. This results in faster bug finding for WOFF2 font reconstruction.

The results for whitelist (b) are similar to the ones for whitelist (c). Indeed, WOFF2 to TTF conversion calls functions that are mostly located in SRC/src/woff2_dec.cc. The 2892 extra instrumentation points allowed by whitelist (b) do not tamper with bug finding, even though they are mostly irrelevant, simply because most of these functions do not get called. We get a slightly faster average time for bug finding with whitelist (b), which might indicate that some of the extra instrumentation points are actually relevant, or might just be random noise.

Diff Detail

Event Timeline

tuktuk created this revision.Jun 20 2019, 11:36 AM

Test coverage missing.

tuktuk updated this revision to Diff 206006.Jun 21 2019, 8:26 AM

I followed Roman Lebedev's advice and adapted the sanitizer_coverage_no_prune.cc test to create a sanitizer_coverage_whitelist_blacklist.cc test under make check_sanitizer. I can only try the test on a Linux machine, and it passes on that machine.

Herald added subscribers: Restricted Project, kubamracek, srhines. · View Herald TranscriptJun 21 2019, 8:26 AM

Thanks for the patch! Seems like a useful feature for targeted fuzzing.

clang/docs/SanitizerCoverage.rst
310

The wording makes it sound like there may be exceptions to the expected whitelist/blacklist behavior. But IIUC the paragraph is meant to explain the typical use case. Can we make this more explicit?

e.g.,

A common use case is to have the whitelist list folders and source files ... while the blacklist ...

Or maybe we don't need this paragraph at all...

clang/include/clang/Driver/Options.td
978

For fsanitize_blacklist we only support -fsanitize-blacklist=. Let's do the same for these lists to keep things simple.

clang/lib/CodeGen/BackendUtil.cpp
218

Please run clang-format --style=LLVM on the patch.

clang/lib/Driver/SanitizerArgs.cpp
743

The two cases have lots of overlapping code. Let's try to coalesce.

759

Let's try to coalesce here too. Maybe a helper function? Then we could also use it for the sanitizer blacklist.

compiler-rt/test/sanitizer_common/TestCases/sanitizer_coverage_whitelist_blacklist.cc
60 ↗(On Diff #206006)

Can we also test with -fsanitize=inline-8bit-counters, since that is what libFuzzer uses by default?

llvm/lib/Transforms/Instrumentation/SanitizerCoverage.cpp
192

Nit: Preferred style is no curly braces for one-statement ifs.

tuktuk updated this revision to Diff 207086.Jun 28 2019, 10:07 AM
tuktuk edited the summary of this revision. (Show Details)

I followed Matt Morehouse's advice: mainly, I adapted the test so that it uses libFuzzer's default SanitizerCoverage options instead of trace-pc, and I rewrote some parts of the code to make it less redundant.

This revision is now accepted and ready to land.Jun 28 2019, 11:36 AM
tuktuk edited the summary of this revision. (Show Details)Jun 28 2019, 12:25 PM

Thanks for the reviews.

This is a really useful feature for fuzzing bigger software projects and the review was accepted. Any plans to merge this?

Think so, and when can this feature be merge to the mainline? look forward it.

This is a really useful feature for fuzzing bigger software projects and the review was accepted. Any plans to merge this?

Thanks for the reviews.

Hi, @tuktuk. I want to use this function on the latest llvm, but now the code has been modified, this patch is no longer valid, can you take a look, thank you very much. : )

tuktuk updated this revision to Diff 255032.Apr 4 2020, 7:02 AM

Thank you for your interest in this feature! It is unfortunate indeed that the patch was not merged when accepted, so here is an update that matches the current status of the code base.

Dear reviewers, can you please make sure that this time, the patch gets merged if accepted?
Also, this time I was unable to run the new test that this patch adds from within the test system, because of my limited understanding; so I gave up and followed the test myself, command by command.
Am I missing an additional step for test integration, after adding the test file?
Thanks a lot.

vitalybuka added inline comments.
clang/lib/CodeGen/BackendUtil.cpp
218

it should be -style=file
"arc diff" should also invoke clang-format, check utils/arcanist/clang-format.sh

vitalybuka accepted this revision.Apr 7 2020, 1:43 AM
pratyai added a subscriber: pratyai.Apr 7 2020, 7:44 AM
morehouse accepted this revision.Apr 9 2020, 12:42 PM

Thanks again for the patch. Sorry about the delay in landing it; I'll work on it today.

morehouse requested changes to this revision.Apr 9 2020, 3:38 PM

Am I missing an additional step for test integration, after adding the test file?

Just tried locally and I think you need to rename to file from *.cc to *.cpp. Then it will run during ninja check-sanitizer.

I did that and the test fails for me.

--
Exit Code: 1 

Command Output (stderr): 
--
Expected 14 lines, got 28.

--

Please take a look and fix it.

compiler-rt/test/sanitizer_common/TestCases/sanitizer_coverage_whitelist_blacklist.cc
8 ↗(On Diff #255032)

Please use %t or %T in temp file names so that they get cleaned up properly.

This revision now requires changes to proceed.Apr 9 2020, 3:38 PM
tuktuk updated this revision to Diff 256546.Apr 10 2020, 4:12 AM

I was indeed able to run the test again after changing its name to .cpp, thank you for your help. I have restored XFAIL lines from the original sanitizer_coverage_no_prune.cpp that I should not have deleted. Now the test passes again. Also the test now uses %t to work in a subdirectory like sanitizer_coverage_symbolize.cpp does.

tuktuk updated this revision to Diff 256556.EditedApr 10 2020, 6:13 AM

I forgot to apply clang-format on clang/lib/Driver/SanitizerArgs.cpp, now it should be OK.

This revision is now accepted and ready to land.Apr 10 2020, 10:41 AM
This revision was automatically updated to reflect the committed changes.