Page MenuHomePhabricator

Implement `-fsanitize-coverage-whitelist` and `-fsanitize-coverage-blacklist` for clang
AcceptedPublic

Authored by tuktuk on Jun 20 2019, 11:36 AM.

Details

Reviewers
kcc
morehouse
Summary

This commit adds two command-line options to clang.
These options let the user decide which functions will receive SanitizerCoverage instrumentation.
This is most useful in the libFuzzer use case, where it enables targeted coverage-guided fuzzing.

Patch by Yannis Juglaret of DGA-MI, Rennes, France

libFuzzer tests its target against an evolving corpus, and relies on SanitizerCoverage instrumentation to collect the code coverage information that drives corpus evolution. Currently, libFuzzer collects such information for all functions of the target under test, and adds to the corpus every mutated sample that finds a new code coverage path in any function of the target. We propose instead to let the user specify which functions' code coverage information is relevant for building the upcoming fuzzing campaign's corpus. To this end, we add two new command line options for clang, enabling targeted coverage-guided fuzzing with libFuzzer. We see targeted coverage guided fuzzing as a simple way to leverage libFuzzer for big targets with thousands of functions or multiple dependencies. We publish this patch as work from DGA-MI of Rennes, France, with proper authorization from the hierarchy.

Targeted coverage-guided fuzzing can accelerate bug finding for two reasons. First, the compiler will avoid costly instrumentation for non-relevant functions, accelerating fuzzer execution for each call to any of these functions. Second, the built fuzzer will produce and use a more accurate corpus, because it will not keep the samples that find new coverage paths in non-relevant functions.

The two new command line options are -fsanitize-coverage-whitelist and -fsanitize-coverage-blacklist. They accept files in the same format as the existing -fsanitize-blacklist option https://clang.llvm.org/docs/SanitizerSpecialCaseList.html#format. The new options influence SanitizerCoverage so that it will only instrument a subset of the functions in the target. We explain these options in detail in clang/docs/SanitizerCoverage.rst.

Consider now the woff2 fuzzing example from the libFuzzer tutorial https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md. We are aware that we cannot conclude much from this example because mutating compressed data is generally a bad idea, but let us use it anyway as an illustration for its simplicity. Let us use an empty blacklist together with one of the three following whitelists:

# (a)
src:*
fun:*

# (b)
src:SRC/*
fun:*

# (c)
src:SRC/src/woff2_dec.cc
fun:*

Running the built fuzzers shows how many instrumentation points the compiler adds, the fuzzer will output XXX PCs. Whitelist (a) is the instrument-everything whitelist, it produces 11912 instrumentation points. Whitelist (b) focuses coverage to instrument woff2 source code only, ignoring the dependency code for brotli (de)compression; it produces 3984 instrumented instrumentation points. Whitelist (c) focuses coverage to only instrument functions in the main file that deals with WOFF2 to TTF conversion, resulting in 1056 instrumentation points.

For experimentation purposes, we ran each fuzzer approximately 100 times, single process, with the initial corpus provided in the tutorial. We let the fuzzer run until it either found the heap buffer overflow or went out of memory. On this simple example, whitelists (b) and (c) found the heap buffer overflow more reliably and 5x faster than whitelist (a). The average execution times when finding the heap buffer overflow were as follows: (a) 904 s, (b) 156 s, and (c) 176 s.

We explain these results by the fact that WOFF2 to TTF conversion calls the brotli decompression algorithm's functions, which are mostly irrelevant for finding bugs in WOFF2 font reconstruction but nevertheless instrumented and used by whitelist (a) to guide fuzzing. This results in longer execution time for these functions and a partially irrelevant corpus. Contrary to whitelist (a), whitelists (b) and (c) will execute brotli-related functions without instrumentation overhead, and ignore new code paths found in them. This results in faster bug finding for WOFF2 font reconstruction.

The results for whitelist (b) are similar to the ones for whitelist (c). Indeed, WOFF2 to TTF conversion calls functions that are mostly located in SRC/src/woff2_dec.cc. The 2892 extra instrumentation points allowed by whitelist (b) do not tamper with bug finding, even though they are mostly irrelevant, simply because most of these functions do not get called. We get a slightly faster average time for bug finding with whitelist (b), which might indicate that some of the extra instrumentation points are actually relevant, or might just be random noise.

Diff Detail

Event Timeline

tuktuk created this revision.Jun 20 2019, 11:36 AM

Test coverage missing.

tuktuk updated this revision to Diff 206006.Jun 21 2019, 8:26 AM

I followed Roman Lebedev's advice and adapted the sanitizer_coverage_no_prune.cc test to create a sanitizer_coverage_whitelist_blacklist.cc test under make check_sanitizer. I can only try the test on a Linux machine, and it passes on that machine.

Herald added subscribers: Restricted Project, kubamracek, srhines. · View Herald TranscriptJun 21 2019, 8:26 AM

Thanks for the patch! Seems like a useful feature for targeted fuzzing.

clang/docs/SanitizerCoverage.rst
310

The wording makes it sound like there may be exceptions to the expected whitelist/blacklist behavior. But IIUC the paragraph is meant to explain the typical use case. Can we make this more explicit?

e.g.,

A common use case is to have the whitelist list folders and source files ... while the blacklist ...

Or maybe we don't need this paragraph at all...

clang/include/clang/Driver/Options.td
978

For fsanitize_blacklist we only support -fsanitize-blacklist=. Let's do the same for these lists to keep things simple.

clang/lib/CodeGen/BackendUtil.cpp
218–220

Please run clang-format --style=LLVM on the patch.

clang/lib/Driver/SanitizerArgs.cpp
771

The two cases have lots of overlapping code. Let's try to coalesce.

787

Let's try to coalesce here too. Maybe a helper function? Then we could also use it for the sanitizer blacklist.

compiler-rt/test/sanitizer_common/TestCases/sanitizer_coverage_whitelist_blacklist.cc
61

Can we also test with -fsanitize=inline-8bit-counters, since that is what libFuzzer uses by default?

llvm/lib/Transforms/Instrumentation/SanitizerCoverage.cpp
194

Nit: Preferred style is no curly braces for one-statement ifs.

tuktuk updated this revision to Diff 207086.Jun 28 2019, 10:07 AM
tuktuk edited the summary of this revision. (Show Details)

I followed Matt Morehouse's advice: mainly, I adapted the test so that it uses libFuzzer's default SanitizerCoverage options instead of trace-pc, and I rewrote some parts of the code to make it less redundant.

This revision is now accepted and ready to land.Jun 28 2019, 11:36 AM
tuktuk edited the summary of this revision. (Show Details)Jun 28 2019, 12:25 PM

Thanks for the reviews.