[libFuzzer] Create Unstable Edge Check

[kevinwkt]

Summary

Created Flags, Options to run 2 more times if UniqFeature is found to check for unstable edges.
Option 1 in flag is to compare everyone to everyone, 2 is to compare run 2 and 3 taking into account that run 1 is initialization run.
Must not use dump_coverage flag or "_Exit(1);" will be triggered.

[Dor1s]

I've taken a quick look just to find what can be breaking the tests.

lib/fuzzer/FuzzerOptions.h

58

I think you need 0 here as well.

[metzman]

Did a first pass. Looks pretty good!

I'll take another look and go over the actual algorithm soon.

lib/fuzzer/FuzzerDriver.cpp

618

Please print some kind of explanation here.

619

I don't think ErrorExitCode should be used here since the docs seem to say it is the return code when a bug is found:
(see https://llvm.org/docs/LibFuzzer.html#options)

621

While it's fine for now, I think we should use "deterministic" instead of "det". Other flags don't really shorten things and deterministic isn't that long.

lib/fuzzer/FuzzerFlags.def

113

You should explain what 1 and 2 do and that this flag is incompatible with dump_coverage.

lib/fuzzer/FuzzerOptions.h

58

Yep this a bug: -1 evaluates to true.

lib/fuzzer/FuzzerTracePC.cpp

34

The __ in the variable name as well as this attribute are unnecessary, it isn't being used by the compiler unlike sancov_trace_pc_guard_8bit_counters.
Same for sancov_trace_pc_nondeterministic_counters_initialized

[kevinwkt]

Changed Flag name from non_det_check to non_deterministic_check.
Corrected the bug in FuzzerOptions from -1 to 0.
Changed Exit code to _Exit(0);
Removed unnecessary lines in declaring arrays.

[kevinwkt]

Removed __ as commented in first version.

[metzman]

Thanks for the changes.
Left some cosmetic suggestions, I don't think this needs any important changes to functionality.

mmoroz@ Is the summary used for the commit message? That probably should be updated since it still mentions using Options.ErrorExitCode

lib/fuzzer/FuzzerDriver.cpp

618

Tomorrow let's discuss a name that is better than non_deterministic_check, since that isn't really descriptive of what happens (I like DumpUnstable but we should discuss).

622

I guess this check isn't necessary because if !non_deterministic_check then Options.NonDetCheck gets set to zero (which it already is)

lib/fuzzer/FuzzerFlags.def

115

nit: "a unique feature"
Also, let's explain that this option dumps the unstable edges, right now only the comparison is clear.

lib/fuzzer/FuzzerLoop.cpp

354

Given that we may rename the flag, you don't make this change yet:
It would probably better for this (as well as the function DumpNonDetCoverage) to be consistent with the option name (ie: NonDeterministicCheck and DumpNonDeterministicCoverage

473

I think -1 should also be an enum value.

476

Please put spaces around the equals sign

481

I think instead of comparing NonDetCheck to 1 and 2, we should use an enum. eg:

enum {

AllInstability = 1,
NonInitializationInstability = 2,

};

and then this if statement would be if (Options.NonDetCheck == AllInstability)

481

We can reduce the nesting required here by doing:

if (Options.NonDetCheck == 1 && TPC.CountersNonDet()[idx] != TPC.Counters()[idx])
  TPC.CountersNonDet()[idx] = -1;

484

Let's reduce nesting here

else if (run == 0 && TPC.CountersNonDetInitialized()[idx] != -1) {
  TPC.CountersNonDetInitialized()[idx] = TPC.Counters()[idx];
}

488

Let's do this

else if (TPC.CountersNonDetInitialized()[idx] != TPC.Counters()[idx])
  TPC.CountersNonDetInitialized()[idx] = -1;

to reduce nesting

lib/fuzzer/FuzzerTracePC.cpp

323

2 thoughts on this line:

1. If I were to keep this check in this function, I would do:

if (!EF->__sanitizer_dump_coverage)
  return;

So that we don't need to nest the entire rest of this function.
This would be inconsistent with DumpCoverage so I could be wrong

2. The behavior of this function (even with the change I just suggested) is consistent with DumpCoverage, so I believe it is fine for now.

However, IMO the behavior of DumpCoverage (and this function) is unfriendly right now.

The way it works, if a user tells libFuzzer to fuzz and dump_coverage, libFuzzer waits until the end of the process and then if __sanitizer_dump_coverage is unavailable, (something libFuzzer knew at the start) libFuzzer ignores the option and doesn't explain why.

I think checks for __sanitizer_dump_coverage should be done before fuzzing but that is not for this patch.
For now I think we can just print some kind of explanation (I think there is no sanitizer_dump_coverage if -fsanitize-coverage=trace-pc-guard is not used).

327

Add a space before the question mark please

329

else if (check == 2)

To reduce nesting please

330

Add a space before the question mark please

lib/fuzzer/FuzzerTracePC.h

165

Is this necessary? I don't think we are using PCs() anywhere outside of FuzzerTracePC

[Dor1s]

Don't have much to add on top of Jonathan's comments, will take another look later when those get resolved.

[kevinwkt]

Changed Flag and Option names.
Fixed nested ifs.
Created enum and constexpr for magic numbers.
Fixed Flag explanations.
Fixed format for dump_nondeterministic_coverage by printing explanation and getting rid of the if statement.
PCs() back to private.

[Dor1s]

@kevinwkt, is this ready for the next review round, or are you still doing more changes locally?

[kevinwkt]

PTAL

Moved things to TracePC as suggested along with previous changes.

[kevinwkt]

PTAL

Moved all Counters() and PCs() to private again.

[Dor1s]

Nice! Left some nits, will take another look in the morning.

lib/fuzzer/FuzzerFlags.def

117

Maybe let's say "execute every input 3 times in total in a unique feature is found during the first execution". I'm just trying to get rid of word "run", as it sounds a bit confusing given that there is another -runs= flag.

lib/fuzzer/FuzzerTracePC.cpp

80

make the arguments const

83

nit: && in the beginning of the line looks inconsistent with conditions on lines 85, 89.

88

add a space after if

342

make the argument const here and in the header

[Dor1s]

lib/fuzzer/FuzzerDriver.cpp

619

very minor thing, but let's maybe say dump_unstable_coverage and dump_coverage to be clear that those are names of flags

lib/fuzzer/FuzzerLoop.cpp

355

do we want to rename this function to DumpUnstableCoverage as per previous Jonathan's comment, to keep names of option and function consistent?

472

nit: unnecessary empty line, but I think we should put a comment here. Something short like "Run the same input two more times to detect non-deterministic coverage"

lib/fuzzer/FuzzerTracePC.cpp

34

I'm slightly worried that we always allocate these two arrays, even when dump_unstable_coverage is 0. However, TracePC doesn't seem to have an explicit constructor, so I'm not sure whether we have implement it because of these or leave it as is. Let's leave as is for now, I guess.

77

Counters() points to values of uint8_t type, while CountersNonDet() points to a signed int8_t. Are we sure it's going to work fine? Why do we use signed type ourselves, just to be able to put -1 as a magic value for UnstableCounter?

349

nit: in the code above you're using size_t idx, and here just i. Either way is fine (I'd prefer just i though), but let's keep it consistent.

351

use UnstableCounter instead of -1 here and on line 353?

lib/fuzzer/FuzzerTracePC.h

144

I'm trying to think of better names for these two, but can't generate anything... Maybe AllUnstableCoverage and NonInitUnstableCoverage, but not sure. Up to you, feel free to ignore.

[Dor1s]

lib/fuzzer/FuzzerTracePC.cpp

60

Are you sure you need two arrays? I think one would be enough. If we do AllInstability:

1. InitializeNonDetCounters will put Counters into our array, this will be our baseline to compare against

2. during the 2nd and the 3rd executions we will be comparing new Counters against our array and assign UnstableCounter when needed

If we do NonInitializationInstability:

1. We don't need to call InitializeNonDetCounters after the first run.

2. After the 2nd run we call InitializeNonDetCounters in order to set up a baseline to compare against

3. After the 3rd run we compare new Counters against our baseline and assign UnstableCounter when needed

That would also simplify TracePC::DumpNonDetCoverage function and other code.

74

Is there any value in calling this function when we use NonInitializationInstability? I think no.

81

I think we should check UnstableOption and run values outside of the loop. Checking that on every iteration is not efficient. It's fine to have three loops inside three different IF branches.

82

should we also skip counters that are already known as unstable? e.g. add && CountersNonDet()[idx] != UnstableCounter. Otherwise, we'll be re-assigning UnstableCounter to the same counters again and again.

89

same problem here, if we already put UnstableCounter into CountersNonDetInitialized()[idx], there is no way for Counters()[idx] to match it. That means, we'll be overwriting those counters with UnstableCounter again and again.

[kevinwkt]

PTAL

[metzman]

lib/fuzzer/FuzzerFlags.def

113

Please add a space after the comma.

lib/fuzzer/FuzzerLoop.cpp

476

Please put a space in between for and the paren.

483

nit: use just one newline

lib/fuzzer/FuzzerTracePC.cpp

88

Space after ==.

351

This line is over 80 chars right? lines should be less than 80 chars in width.
(https://llvm.org/docs/CodingStandards.html#source-code-width)

I think it would be better to change this to a traditional if-else since I think multiline ternary ifs are not allowed.

Please also fix any other lines above 80 chars in width in this CL.

lib/fuzzer/FuzzerTracePC.h

141–142

nit: remove the semi colon in the comment.

[kevinwkt]

I will use clang-tidy or something else soon. I promise.

[metzman]

lib/fuzzer/FuzzerDriver.cpp

619

nit: "Can not use dump_unstable_coverage and dump_coverage options together\n"

[Dor1s]

Looks great now! I'll take another look Monday morning, but I guess we're almost done here :)

lib/fuzzer/FuzzerTracePC.h

142

actually, the comment is not needed at all, please remove it. Plus, rename UnstableCounter to kUnstableCounter to match the style used in this file / project, e.g. https://cs.chromium.org/chromium/src/third_party/libFuzzer/src/FuzzerTracePC.h?l=72

[metzman]

I have two comments that I didn't think should be put in any particular place.

1. The naming in this patch is inconsistent, we talk about instability, unstable, and nondet (which, as I mentioned earlier, isn't great since its shortened) and non-deterministic. Some of this is due to inconsistent suggestions. Regardless, I think it is best to use as few terms as possible when describing one thing. Thus, I think we should just use the terms stable and unstable in the code. I don't think these are the best terms since it isn't obvious what they mean, but I think we should be consistent with AFL. In comments and messages we can use these other terms to explain what stable and unstable mean.

2. It's hard to do this review with "Context not available." everywhere. As Kode pointed out yesterday, there are instructions on how to upload as much context as possible with each diff, please do this in the future.

lib/fuzzer/FuzzerLoop.cpp

470

nit: Add a comment here explaining that if requested by the user, we will look for unstable edges.

476

Why is run a size_t? It's not a size and isn't being used in another operation with any other size_t.
I think it should be an int. Please do the same to UpdateNonDetCounters

476

nit: The initial value of run is somewhat confusing given its name
The zeroth run already happened (and maybe the 1st as well thanks to leak detection?)
This is more like the zeroth unstable check run, maybe change name to unstable_rerun.

lib/fuzzer/FuzzerOptions.h

58

I think the type of DumpUnstableCoverage should be the name of the enum we use (see the enum for my comments on that).
Also, instead of setting it to zero we should set it to the value in the enum signifiying not to dump unstable coverage (maybe DontDumpUnstable).

lib/fuzzer/FuzzerTracePC.cpp

70

Please add a comment explaining what this function does and when it should be run (I think it's a bit unclear why there are two calls to a function with Initialize in the title)

72

I think a comment should be added about what would set a value in CountersNondet to UnstableCounter.

87

Let's get rid of uneeded nesting and reduce the complexity of this function. You can move the if (run == 0) and the else that comes after it outside of the outer else so that we have:

if (UnstableOption == AllInstability)
  UpdateNonDetCountersImpl();
else if (run == 0)  
  InitializeNonDetCounters();
else 
  UpdateNonDetCountersImpl();

We should also get rid of the curly braces to be consistent with the rest of libFuzzer (which seems not to use them with one line if bodies.

Now that we've done that, it is easier to see another simplification we can make. We only need an if-else since there are just two possible actions taken, let's change this:

if (UnstableOption != AllInstability && run == 0)  
  InitializeNonDetCounters();
else 
  UpdateNonDetCountersImpl();

At this point it's clearer that we don't even need the else, just the if and a return in its body, followed by UpdateNonDetCountersImpl() after the body, but I don't really prefer one way or the other.

lib/fuzzer/FuzzerTracePC.h

140

Let's use the type of the enum that defines the possibilities for UnstableOption, instead of int8_t

144

3 comments on this enum:

1. +1 to Max's comments. The names I suggested were not as good. And I suggest a big change to naming in this PR which Max's suggestion fits with.
2. Let's give this enum a name, such as DumpUnstableCoverageOptions. That way we can set the type of DumpUnstableCoverage to it.
3. Let's add an enum value set to zero, called something like DontDumpUnstableso that we don't have to use 0 as a magic value.

[Dor1s]

+1 to Jonathan's comments, those are great suggestions.

lib/fuzzer/FuzzerLoop.cpp

476

s/unstable_rerun/UnstableRerun/

[kevinwkt]

lib/fuzzer/FuzzerOptions.h

58

Will complicate things given that the enum would have to be in a file usable by both FuzzerOptions.h and also by FuzzerTracePC.h/cpp,

[kevinwkt]

lib/fuzzer/FuzzerTracePC.h

144

Changed all names to *unstable*, if definite alternative is chosen, will change later on.

[kevinwkt]

Once again, minor syntax and coding standard changes. Will improve over time.

[metzman]

let's update the language in the commit message too, when you get a chance.

[Dor1s]

In D48150#1136744, @metzman wrote:

let's update the language in the commit message too, when you get a chance.

+1, take a look at other commit messages for an example: https://github.com/llvm-mirror/compiler-rt/commits/master/lib/fuzzer

[Dor1s]

I don't seem to have any other comments except of the naming (e.g. Idx) and the CL description. Jonathan, once you're also satisfied, I think we should ask @morehouse to take a look later this week. WDYT?

[metzman]

I don't seem to have any other comments except of the naming (e.g. Idx) and the CL description. Jonathan, once you're also satisfied, I think we should ask @morehouse to take a look later this week. WDYT?

+1, after the language in the commit is updated and [libFuzzer] is added to the title.

[metzman]

LGTM

[kevinwkt]

Changed idx to i.

[Dor1s]

Matt, Kevin is an intern on our team who is fighting against non-deterministic coverage. Could you please take a look?

There also will be a follow-up CL that uses this approach in order to ignore "unstable" edges when making a decision whether a particular input is NEW or now.

[Dor1s]

Oh, I completely forgot. We also have to add a test. Kevin, please check out the section regarding the tests in LLVM Dev Starting Kit. Then, take a look at the existing tests: https://github.com/llvm-mirror/compiler-rt/tree/master/test/fuzzer, especially at those that have "coverage" in its name. You can either update some existing .testfile or add a new one, whichever is more convenient for you.

[morehouse]

lib/fuzzer/FuzzerLoop.cpp

506

Can we run this loop three times, move the initialization above into UpdateUnstableCounters, and make InitializeUnstableCounters private?

508

It might make sense to set/unset RunningCB = true around the callback so the exit handler works correctly. (An unstable edge could potentially lead to an exit()).

However, if you do this you'll also need to set UnitStartTime and UnitStopTime so the timeout handler works correctly.

lib/fuzzer/FuzzerTracePC.cpp

36

If this isn't weakly defined anywhere else, what is the reason for this being a global?

66

This effectively poisons a counter if it has ever previously been marked unstable. Do we want this?

Suppose we have something like:

void foo() {
  if (rand()) {
    doSomething();
  }
}
void bar() {
  doSomething();
}

If the first run calls foo(), we would mark the PCs in doSomething as unstable. Then if a future run calls bar(), doSomething will still be marked unstable even though it isn't unstable in the context of bar().

82

This comment is confusing. What is the "zeroth rerun"?

85

Nit: const is unnecessarily verbose since the parameters are passed by value anyway.

lib/fuzzer/FuzzerTracePC.h

80

Nit: enum DumpUnstableCoverageOptions { is simpler.

147

Does this need to be public? Also, please add a comment describing the purpose of this constant.

[metzman]

lib/fuzzer/FuzzerTracePC.cpp

66

AFL does the same thing (see: https://cs.chromium.org/chromium/src/third_party/afl/src/afl-fuzz.c?q=afl-fuzz.c&sq=package:chromium&dr=C&l=2603)
I agree it isn't ideal, when we looked at the coverage reports they were hard to interpret for this reason.
Do you think there is a reasonable way to store info about paths so we only poison the first unstable edge in an unstable path (interested in other alternatives too)?

82

I think I suggested this language. Do you think "first rerun" makes more sense? I didn't like it because it means when UnstableRerun == 0 but I don't feel strongly about it.

[morehouse]

lib/fuzzer/FuzzerLoop.cpp

506

Scratch that, we don't want to rerun the callback 3 times. But maybe we could still move the initialization above into UpdateUnstableCounters to make that implementation detail private.

lib/fuzzer/FuzzerTracePC.cpp

66

We could probably store path info for the latest run only (in a large array), then discard/overwrite it on the next run. Then iterate through the path and only mark the first PC in each unstable sequence.

But that should probably be a future CL, since the change would be non-trivial, and I'd like Kostya to comment on such an approach first.

82

Maybe moving some of these comments inline would make it clearer. Something like:

if (UnstableOption != AllUnstable && UnstableRerun == 0) {
  // We are only considering the last 2 runs for determining stability.
  // Therefore we re-initialize counters on the first rerun.
...
}  else {
  // We are considering all 3 runs for determining stability.
...

[kevinwkt]

lib/fuzzer/FuzzerLoop.cpp

506

So the reason I did it this way was due to the fact that I need the counters for the original run copied into my unstable_counters which are erased at the beginning of every TPC.ResetMaps. The only way I can think of at the moment to do the initialization on the TPC side is to not ResetMaps() and CB() before the TPC.Update. However, this would unnecessarily complicate things on the TPC side given counters we want are updated as the last thing in the for loop.
If there are any better way to solve this, please let me know as I may have overlooked things.

[morehouse]

lib/fuzzer/FuzzerLoop.cpp

506

Yes, I see that now.

After looking at this again, I don't think this loop is helping much. I'd suggest inlining the logic from UpdateUnstableCounters since it's awkward to be passing loop indexes to it anyway.

E.g.,

if (option == AllUnstable)
  Initialize();
doCallback();
if (option == AllUnstable)
  UpdateUnstableCounters();
else
  Initialize();
doCallback();
UpdateUnstableCounters();

This would also allow you to eliminate UpdateUnstableCountersImpl.

[kevinwkt]

PTAL,

Made changes as requested taking out the for loop.
However, given that I need to call Initialize() and Update() from FuzzerLoop, both were declared public in FuzzerTracePC.

[morehouse]

lib/fuzzer/FuzzerLoop.cpp

527

Might be nice to move this new code into a helper method Fuzzer::CheckForUnstableCounters.

lib/fuzzer/FuzzerTracePC.cpp

54

Can we make this a member of TracePC and shorten the name?

63

Don't we always run this whether -dump_unstable_coverage is 1 or 2?

82

Sorry to contradict @Dor1s, but I think I'd actually prefer if we removed the check for kUnstableCounter. The code would be simpler, and in the typical case might be marginally faster (I would expect unstable edges to be the exception rather than the rule).

371

Typically warnings for missing external functions are handled by setting the last argument to true in FuzzerExtFunctions.def. In the case of __sanitizer_dump_coverage, we don't warn currently.

Is there a good reason to warn anyway? If so, maybe we should flip the flag in FuzzerExtFunctions.def instead. If not, let's remove the Printf.

lib/fuzzer/FuzzerTracePC.h

78

Nit: NonInitializationUnstable is a slightly confusing name to me. Maybe IgnoreInitUnstable or IgnoreFirstRunUnstable would be clearer?

195

Nit: Please move kUnstableCounter to very top of private section.

[Dor1s]

lib/fuzzer/FuzzerTracePC.cpp

82

That's a good point. Sounds reasonable to me. We may consider making this to be the second condition rather than the first one, so that it would be executed only when the edge is indeed unstable. I feel like doing another CMP on a value which we already have loaded (in a register or whatever) would be faster than doing 'store' operation, given that with unstable targets we likely to hit the same non-deterministic edges again and again. But if you insist, I'm fine with removing that.

371

The existing dump_coverage doesn't complain about missing __sanitizer_dump_coverage and thus provides not great UX in such cases. +1 to make it warn via FuzzerExtFunctions.def

[kevinwkt]

ptal

Fixed comments.

[morehouse]

Code looks good. Please add a test.

lib/fuzzer/FuzzerTracePC.h

153

Please follow the surrounding naming conventions. i.e. UnstableCounters.

[kevinwkt]

ptal
Added tests

[morehouse]

lib/fuzzer/FuzzerTracePC.h

152

uint8_t

test/fuzzer/DumpUnstableCoverageTest.cpp

7

Why do we need a pointer to x?

47

A switch would be more compact.

test/fuzzer/dump_unstable_coverage.test

2

Why do we need -g -fsanitize-coverage=0 -fsanitize-coverage=trace-pc-guard?

12

We need to look at the actual sancov file to make sure it has the PCs we expect to be unstable.

[kevinwkt]

test/fuzzer/dump_unstable_coverage.test

2

Deleted "-g -fsanitize-coverage=0", we only need "-fsanitize-coverage=trace-pc-guard" since we use the counters I believe.

12

None of the tests so far handles sancov files except for dump_coverage, and dump_coverage does the same thing I do except they only check for "LLVMFuzzerTestOneInput" in the sancov file. What would be they right way to approach this?

[morehouse]

lib/fuzzer/FuzzerTracePC.h

152

uint8_t is still misspelled.

test/fuzzer/dump_unstable_coverage.test

12

Can you do something similar to dump_coverage.test, except check that the unstable functions are there and not the stable ones?

[kevinwkt]

test/fuzzer/DumpUnstableCoverageTest.cpp

47

Did not use a switch since it seemed to produce only 1 edge since it is a lookup table. Used ifs to make more edges.

[kevinwkt]

test/fuzzer/DumpUnstableCoverageTest.cpp

47

I think unlike if-else which uses multiple edges switches uses a jump-table.

[morehouse]

test/fuzzer/DumpUnstableCoverageTest.cpp

47

Right, but I don't see how a jump table would be an issue here. Especially since the test only checks coverage at the function level.

test/fuzzer/dump_unstable_coverage.test

16

Could you also check the sancov file the -dump_unstable_coverage=2 case? You'll have to make the fuzz target save some state the first time it sees a certain input, and then skip some code if that state has been set.

For example, I think this would be enough:

static bool SkipFoo = false;
if (Size == 1 && Data[0] == 'Z' && !SkipFoo)  {
  SkipFoo = true;
  Foo();
}

You'll probably also have to bump up the -runs parameter enough for libFuzzer to always find the particular input that triggers this case.

[morehouse]

test/fuzzer/dump_unstable_coverage.test

13

I think it makes sense to also check that the ini functions are marked unstable for SANCOV1.

[kevinwkt]

test/fuzzer/dump_unstable_coverage.test

13

Then I would have to bump up the runs for this test to guarantee that there is at least 1 case where ini function is triggered. How many runs do you think is ok? Would 100000 runs be adequate?

[morehouse]

test/fuzzer/dump_unstable_coverage.test

13

That should be enough. I think 100000 is pretty common for other fuzzer tests.

[kcc]

lib/fuzzer/FuzzerTracePC.h

78

Do you really need these two choices?
I understand the difference, but maybe we can have just one of these?

test/fuzzer/dump_unstable_coverage.test

2

Why do you use the old -fsanitize-coverage=trace-pc-guard instead of the new fsanitize=fuzzer?
I'd prefer if you used the new instrumentation instead of the old one because I was planing to
completely remove the old one.

[kevinwkt]

This seems to detect the unstable edges, however I am having errors in TracePC::DumpUnstableCoverage() in these lines.

if (UnstableCounters[UnstableIdx] == kUnstableCounter)

PCsCopy[UnstableIdx] = ModulePCTable[i].Start[j].PC;

"ERROR: Coverage points in binary and .sancov file do not match." is what I am getting which I am assuming I am sending the wrong PC. But then which PC should be sent?

[morehouse]

In D48150#1148532, @kevinwkt wrote:

"ERROR: Coverage points in binary and .sancov file do not match." is what I am getting which I am assuming I am sending the wrong PC. But then which PC should be sent?

Is this coming from sancov? I think it might only support trace-pc-guard...

@kcc: How can we test this with inline counters?

[kcc]

In D48150#1148584, @morehouse wrote:

In D48150#1148532, @kevinwkt wrote:

"ERROR: Coverage points in binary and .sancov file do not match." is what I am getting which I am assuming I am sending the wrong PC. But then which PC should be sent?

Is this coming from sancov? I think it might only support trace-pc-guard...

@kcc: How can we test this with inline counters?

We don't develop sancov any further and it does not support instrumentation other than trace-pc-guard.
Do you guys use it, or you use something else?
I guess the simplest way here is to not use sancov in this test and instead use llvm-symbolizer.
You can also try to relax the check in sancov to make it work (should be simple).

[kevinwkt]

We don't develop sancov any further and it does not support instrumentation other than trace-pc-guard.
Do you guys use it, or you use something else?
I guess the simplest way here is to not use sancov in this test and instead use llvm-symbolizer.
You can also try to relax the check in sancov to make it work (should be simple).

Would we be able to get a coverage report through llvm-symbolizer? Because I think one of the things we want here is to be able to use this as a report as well as handling instability.

[kcc]

You may try projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py instead of sancov, this tool is less restrictive.
pipe the output of sancov.py through llvm-symbolizer

[kevinwkt]

In D48150#1149859, @kcc wrote:

You may try projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py instead of sancov, this tool is less restrictive.
pipe the output of sancov.py through llvm-symbolizer

I'm trying /usr/.../llvm/projects/compiler-rt/lib/sanitizer_common/scripts/sancov.py merge %t-DumpUnstableCoverageTest* %t_workdir/*.sancov > llvm-symbolizer and it's still not working. I tried unpacking and rawunpacking the sancov files and I get the error:
"UnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 12: invalid start byte"

[metzman]

@kcc @morehouse It doesn't look like a coverage report with new instrumentation will be feasible. What do you think of changing this CL to report stability as a stat, like AFL does, instead?

[kcc]

In D48150#1158238, @metzman wrote:

@kcc @morehouse It doesn't look like a coverage report with new instrumentation will be feasible.

I don't understand why.

What do you think of changing this CL to report stability as a stat, like AFL does, instead?

I also don't understand what you mean by that. Maybe create a separate CL?

[metzman]

In D48150#1158252, @kcc wrote:

In D48150#1158238, @metzman wrote:

@kcc @morehouse It doesn't look like a coverage report with new instrumentation will be feasible.

I don't understand why.

I don't think sancov reports are possible using the new instrumentation. Not 100% about this, but it doesn't seem to work like it did for the old instrumentation.

What do you think of changing this CL to report stability as a stat, like AFL does, instead?

I also don't understand what you mean by that. Maybe create a separate CL?

I mean that instead of implementing coverage reports for unstable edges, add a final_stat (or a stat that gets printed to stdout) that lists the percentage of unstable edges.
This is the stability stat that AFL reports.
A separate CL works too.

[kcc]

In D48150#1158253, @metzman wrote:

In D48150#1158252, @kcc wrote:

In D48150#1158238, @metzman wrote:

@kcc @morehouse It doesn't look like a coverage report with new instrumentation will be feasible.

I don't understand why.

I don't think sancov reports are possible using the new instrumentation. Not 100% about this, but it doesn't seem to work like it did for the old instrumentation.

I am 100% confident it can work.
Just either fix sancov (a bit) or use sancov.py + llvm-symbolizer

What do you think of changing this CL to report stability as a stat, like AFL does, instead?

I also don't understand what you mean by that. Maybe create a separate CL?

I mean that instead of implementing coverage reports for unstable edges, add a final_stat (or a stat that gets printed to stdout) that lists the percentage of unstable edges.
This is the stability stat that AFL reports.
A separate CL works too.

That will work too. Up to you.
"coverage reports for unstable edges" is kind of fun though.
You can also generate this report from within libFuzzer, no need to involve external files, sancov, etc.

See how we implement printing newly discovered functions or PCs, it's all there.

[metzman]

Removing myself as reviewer since this won't be landed and I don't like having clutter on my reviews page.

lib/fuzzer/FuzzerDriver.cpp

int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
   bool DoPlainRun = AllInputsAreFiles();
   Options.SaveArtifacts =
       !DoPlainRun || Flags.minimize_crash_internal_step;
   Options.PrintNewCovPcs = Flags.print_pcs;
   Options.PrintNewCovFuncs = Flags.print_funcs;
   Options.PrintFinalStats = Flags.print_final_stats;
   Options.PrintCorpusStats = Flags.print_corpus_stats;
   Options.PrintCoverage = Flags.print_coverage;
+  if (Flags.dump_unstable_coverage && Flags.dump_coverage) {
+    Printf("Can not use dump_unstable_coverage and dump_coverage options together\n");
+    _Exit(1);
+  }
+  if (Flags.dump_unstable_coverage == TracePC::AllUnstable ||
+      Flags.dump_unstable_coverage == TracePC::IgnoreInitUnstable)
+    Options.DumpUnstableCoverage = Flags.dump_unstable_coverage;
   Options.DumpCoverage = Flags.dump_coverage;
   if (Flags.exit_on_src_pos)
     Options.ExitOnSrcPos = Flags.exit_on_src_pos;
   if (Flags.exit_on_item)
     Options.ExitOnItem = Flags.exit_on_item;
   if (Flags.focus_function)
     Options.FocusFunction = Flags.focus_function;
   if (Flags.data_flow_trace)

lib/fuzzer/FuzzerExtFunctions.def

 EXT_FUNC(__sanitizer_symbolize_pc, void,
          (void *, const char *fmt, char *out_buf, size_t out_buf_size), false);
 EXT_FUNC(__sanitizer_get_module_and_offset_for_pc, int,
          (void *pc, char *module_path,
          size_t module_path_len,void **pc_offset), false);
 EXT_FUNC(__sanitizer_set_death_callback, void, (void (*)(void)), true);
 EXT_FUNC(__sanitizer_set_report_fd, void, (void*), false);
 EXT_FUNC(__sanitizer_dump_coverage, void, (const uintptr_t *, uintptr_t),
-         false);
+         true);

lib/fuzzer/FuzzerFlags.def

 FUZZER_FLAG_INT(print_final_stats, 0, "If 1, print statistics at exit.")
 FUZZER_FLAG_INT(print_corpus_stats, 0,
   "If 1, print statistics on corpus elements at exit.")
 FUZZER_FLAG_INT(print_coverage, 0, "If 1, print coverage information as text"
                                    " at exit.")
 FUZZER_FLAG_INT(dump_coverage, 0, "Deprecated."
                                   " If 1, dump coverage information as a"
                                   " .sancov file at exit.")
+FUZZER_FLAG_INT(dump_unstable_coverage, 0, "Experimental."
+          " Is not compatible with dump_coverage flag."
+          " Dumps unstable edge coverage information"
+          " as a .sancov file at exit."
+          " Executes every input 3 times in total if a unique feature"
+          " is found during the first execution."
+          " If 1, compares all 3 runs for unstable edges"
+          " If 2, compares run 2 and 3 only taking into account"
+          " that run 1 is initialization run.")
 FUZZER_FLAG_INT(handle_segv, 1, "If 1, try to intercept SIGSEGV.")
 FUZZER_FLAG_INT(handle_bus, 1, "If 1, try to intercept SIGBUS.")
 FUZZER_FLAG_INT(handle_abrt, 1, "If 1, try to intercept SIGABRT.")
 FUZZER_FLAG_INT(handle_ill, 1, "If 1, try to intercept SIGILL.")
 FUZZER_FLAG_INT(handle_fpe, 1, "If 1, try to intercept SIGFPE.")
 FUZZER_FLAG_INT(handle_int, 1, "If 1, try to intercept SIGINT.")
 FUZZER_FLAG_INT(handle_term, 1, "If 1, try to intercept SIGTERM.")
 FUZZER_FLAG_INT(handle_xfsz, 1, "If 1, try to intercept SIGXFSZ.")

lib/fuzzer/FuzzerInternal.h

public:
   static void StaticAlarmCallback();
   static void StaticCrashSignalCallback();
   static void StaticExitCallback();
   static void StaticInterruptCallback();
   static void StaticFileSizeExceedCallback();
   static void StaticGracefulExitCallback();
 
   void ExecuteCallback(const uint8_t *Data, size_t Size);
+  void CheckForUnstableCounters(const uint8_t *Data, size_t Size);
   bool RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile = false,
               InputInfo *II = nullptr, bool *FoundUniqFeatures = nullptr);
 
   // Merge Corpora[1:] into Corpora[0].
   void Merge(const Vector<std::string> &Corpora);
   void CrashResistantMerge(const Vector<std::string> &Args,
                            const Vector<std::string> &Corpora,
                            const char *CoverageSummaryInputPathOrNull,

lib/fuzzer/FuzzerLoop.cpp

void Fuzzer::PrintStats(const char *Where, const char *End, size_t Units) {
   Printf(" exec/s: %zd", ExecPerSec);
   Printf(" rss: %zdMb", GetPeakRSSMb());
   Printf("%s", End);
 }
 
 void Fuzzer::PrintFinalStats() {
   if (Options.PrintCoverage)
     TPC.PrintCoverage();
+  if (Options.DumpUnstableCoverage)
+    TPC.DumpUnstableCoverage();
   if (Options.DumpCoverage)
     TPC.DumpCoverage();
   if (Options.PrintCorpusStats)
     Corpus.PrintStats();
   if (!Options.PrintFinalStats)
     return;
   size_t ExecPerSec = execPerSec();
   Printf("stat::number_of_executed_units: %zd\n", TotalNumberOfRuns);
void Fuzzer::PrintPulseAndReportSlowInput(const uint8_t *Data, size_t Size) {
   if (TimeOfUnit > TimeOfLongestUnitInSeconds * 1.1 &&
       TimeOfUnit >= Options.ReportSlowUnits) {
     TimeOfLongestUnitInSeconds = TimeOfUnit;
     Printf("Slowest unit: %zd s:\n", TimeOfLongestUnitInSeconds);
     WriteUnitToFileWithPrefix({Data, Data + Size}, "slow-unit-");
   }
 }
 
+void Fuzzer::CheckForUnstableCounters(const uint8_t *Data, size_t Size) {
+  auto CBSetupAndRun = [&]() {
+    UnitStartTime = system_clock::now();
+    TPC.ResetMaps();
+    RunningCB = true;
+    CB(Data, Size);
+    RunningCB = false;
+    UnitStopTime = system_clock::now();
+  };
+
+  // Copy original run counters into our unstable counters
+  if (Options.DumpUnstableCoverage == TPC.AllUnstable)
+    TPC.InitializeUnstableCounters();
+
+  // First Rerun
+  CBSetupAndRun();
+
+  if (Options.DumpUnstableCoverage == TPC.AllUnstable)
+    TPC.UpdateUnstableCounters();
+  else
+    TPC.InitializeUnstableCounters();
+
+  // Second Rerun
+  CBSetupAndRun();
+
+  TPC.UpdateUnstableCounters();
+
+}
+
 bool Fuzzer::RunOne(const uint8_t *Data, size_t Size, bool MayDeleteFile,
                     InputInfo *II, bool *FoundUniqFeatures) {
   if (!Size)
     return false;
 
   ExecuteCallback(Data, Size);
 
   UniqFeatureSetTmp.clear();
   size_t FoundUniqFeaturesOfII = 0;
   size_t NumUpdatesBefore = Corpus.NumFeatureUpdates();
   TPC.CollectFeatures([&](size_t Feature) {
     if (Corpus.AddFeature(Feature, Size, Options.Shrink))
       UniqFeatureSetTmp.push_back(Feature);
     if (Options.ReduceInputs && II)
       if (std::binary_search(II->UniqFeatureSet.begin(),
                              II->UniqFeatureSet.end(), Feature))
         FoundUniqFeaturesOfII++;
   });
   if (FoundUniqFeatures)
     *FoundUniqFeatures = FoundUniqFeaturesOfII;
   PrintPulseAndReportSlowInput(Data, Size);
   size_t NumNewFeatures = Corpus.NumFeatureUpdates() - NumUpdatesBefore;
+
+  // If dump_unstable_coverage, execute the same input two more times to detect
+  // unstable edges.
+  if (NumNewFeatures && Options.DumpUnstableCoverage)
+    CheckForUnstableCounters(Data, Size);
+
   if (NumNewFeatures) {
     TPC.UpdateObservedPCs();
     Corpus.AddToCorpus({Data, Data + Size}, NumNewFeatures, MayDeleteFile,
                        TPC.ObservedFocusFunction(),
                        UniqFeatureSetTmp, DFT);
     return true;
   }
   if (II && FoundUniqFeaturesOfII &&
       II->DataFlowTraceForFocusFunction.empty() &&
       FoundUniqFeaturesOfII == II->UniqFeatureSet.size() &&
       II->U.size() > Size) {
     Corpus.Replace(II, {Data, Data + Size});
     return true;
   }
   return false;
 }
 
 size_t Fuzzer::GetCurrentUnitInFuzzingThead(const uint8_t **Data) const {
   assert(InFuzzingThread());
   *Data = CurrentUnitData;
   return CurrentUnitSize;
 }
 
 void Fuzzer::CrashOnOverwrittenData() {
   Printf("==%d== ERROR: libFuzzer: fuzz target overwrites it's const input\n",
          GetPid());
   DumpCurrentUnit("crash-");
   Printf("SUMMARY: libFuzzer: out-of-memory\n");
   _Exit(Options.ErrorExitCode); // Stop right now.
 }
 

lib/fuzzer/FuzzerOptions.h

struct FuzzingOptions {
   bool SaveArtifacts = true;
   bool PrintNEW = true; // Print a status line when new units are found;
   bool PrintNewCovPcs = false;
   int PrintNewCovFuncs = 0;
   bool PrintFinalStats = false;
   bool PrintCorpusStats = false;
   bool PrintCoverage = false;
   bool DumpCoverage = false;
+  int DumpUnstableCoverage = 0;
   bool DetectLeaks = true;
   int PurgeAllocatorIntervalSec = 1;
   int  TraceMalloc = 0;
   bool HandleAbrt = false;
   bool HandleBus = false;
   bool HandleFpe = false;
   bool HandleIll = false;
   bool HandleInt = false;

lib/fuzzer/FuzzerTracePC.h

 };
 
 class TracePC {
  public:
   static const size_t kNumPCs = 1 << 21;
   // How many bits of PC are used from __sanitizer_cov_trace_pc.
   static const size_t kTracePcBits = 18;
 
+  enum DumpUnstableCoverageOptions {
+    AllUnstable = 1,
+    IgnoreInitUnstable = 2,
+  };
+
   void HandleInit(uint32_t *Start, uint32_t *Stop);
   void HandleInline8bitCountersInit(uint8_t *Start, uint8_t *Stop);
   void HandlePCsInit(const uintptr_t *Start, const uintptr_t *Stop);
   void HandleCallerCallee(uintptr_t Caller, uintptr_t Callee);
   template <class T> void HandleCmp(uintptr_t PC, T Arg1, T Arg2);
   size_t GetTotalPCCoverage();
   void SetUseCounters(bool UC) { UseCounters = UC; }
   void SetUseValueProfile(bool VP) { UseValueProfile = VP; }
public:
 
   void UpdateFeatureSet(size_t CurrentElementIdx, size_t CurrentElementSize);
   void PrintFeatureSet();
 
   void PrintModuleInfo();
 
   void PrintCoverage();
   void DumpCoverage();
+  void DumpUnstableCoverage();
 
   template<class CallBack>
   void IterateCoveredFunctions(CallBack CB);
 
   void AddValueForMemcmp(void *caller_pc, const void *s1, const void *s2,
                          size_t n, bool StopAtZero);
 
   TableOfRecentCompares<uint32_t, 32> TORC4;
public:
   void RecordInitialStack();
   uintptr_t GetMaxStackOffset() const;
 
   template<class CallBack>
   void ForEachObservedPC(CallBack CB) {
     for (auto PC : ObservedPCs)
       CB(PC);
   }
 
   void SetFocusFunction(const std::string &FuncName);
   bool ObservedFocusFunction();
 
+  void InitializeUnstableCounters();
+  void UpdateUnstableCounters();
+
 private:
+  // Value used to represent unstable edge.
+  static constexpr int16_t kUnstableCounter = -1;
+
+  // Uses 16-bit signed type to be able to accommodate any possible value from
+  // uint8_t counter and -1 constant as well.
+  int16_t UnstableCounters[kNumPCs];
+
   bool UseCounters = false;
   bool UseValueProfile = false;
   bool DoPrintNewPCs = false;
   size_t NumPrintNewFuncs = 0;
 
   struct Module {
     uint32_t *Start, *Stop;
   };
private:
   struct PCTableEntry {
     uintptr_t PC, PCFlags;
   };
 
   struct { const PCTableEntry *Start, *Stop; } ModulePCTable[4096];
   size_t NumPCTables;
   size_t NumPCsInPCTables;
 
-  uint8_t *Counters() const;
-  uintptr_t *PCs() const;
-
   Set<uintptr_t> ObservedPCs;
   Set<uintptr_t> ObservedFuncs;
 
   std::pair<size_t, size_t> FocusFunction = {-1, -1};  // Module and PC IDs.
 
 
   ValueBitMap ValueProfileMap;
   uintptr_t InitialStack;
+
+  uintptr_t *PCs() const;
+  uint8_t *Counters() const;
 };
 
 template <class Callback>
 // void Callback(size_t FirstFeature, size_t Idx, uint8_t Value);
 ATTRIBUTE_NO_SANITIZE_ALL
 void ForEachNonZeroByte(const uint8_t *Begin, const uint8_t *End,
                         size_t FirstFeature, Callback Handle8bitCounter) {
   typedef uintptr_t LargeType;
   const size_t Step = sizeof(LargeType) / sizeof(uint8_t);
   const size_t StepMask = Step - 1;
   auto P = Begin;
   // Iterate by 1 byte until either the alignment boundary or the end.
   for (; reinterpret_cast<uintptr_t>(P) & StepMask && P < End; P++)

lib/fuzzer/FuzzerTracePC.cpp

 // These are declared as global variables named "__sancov_*" to simplify
 // experiments with inlined instrumentation.
 alignas(64) ATTRIBUTE_INTERFACE
 uint8_t __sancov_trace_pc_guard_8bit_counters[fuzzer::TracePC::kNumPCs];
 
 ATTRIBUTE_INTERFACE
 uintptr_t __sancov_trace_pc_pcs[fuzzer::TracePC::kNumPCs];
 
 // Used by -fsanitize-coverage=stack-depth to track stack depth
 ATTRIBUTE_INTERFACE __attribute__((tls_model("initial-exec")))
 thread_local uintptr_t __sancov_lowest_stack;
 
 namespace fuzzer {
 
 TracePC TPC;
 
 int ScopedDoingMyOwnMemOrStr::DoingMyOwnMemOrStr;
 
 uint8_t *TracePC::Counters() const {
   return __sancov_trace_pc_guard_8bit_counters;
 }
 
 uintptr_t *TracePC::PCs() const {
   return __sancov_trace_pc_pcs;
 }
 
 size_t TracePC::GetTotalPCCoverage() {
   if (ObservedPCs.size())
     return ObservedPCs.size();
   size_t Res = 0;
   for (size_t i = 1, N = GetNumPCs(); i < N; i++)
     if (PCs()[i])
       Res++;
   return Res;
 }
 
+// Initializes unstable counters by copying Inline8bitCounters to unstable counters.
+void TracePC::InitializeUnstableCounters() {
+  if (NumInline8bitCounters == NumPCsInPCTables) {
+    size_t UnstableIdx = 1;
+    for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {
+      uint8_t *Beg = ModuleCounters[i].Start;
+      size_t Size = ModuleCounters[i].Stop - Beg;
+      assert(Size ==
+              (size_t)(ModulePCTable[i].Stop - ModulePCTable[i].Start));
+      for (size_t j = 0; j < Size; j++, UnstableIdx++)
+        if (UnstableCounters[UnstableIdx] != kUnstableCounter)
+          UnstableCounters[UnstableIdx] = Beg[j];
+    }
+  }
+}
+
+// Compares the current counters with counters from previous runs
+// and records differences as unstable edges.
+void TracePC::UpdateUnstableCounters() {
+  if (NumInline8bitCounters == NumPCsInPCTables) {
+    size_t UnstableIdx = 1;
+    for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {
+      uint8_t *Beg = ModuleCounters[i].Start;
+      size_t Size = ModuleCounters[i].Stop - Beg;
+      assert(Size ==
+              (size_t)(ModulePCTable[i].Stop - ModulePCTable[i].Start));
+      for (size_t j = 0; j < Size; j++, UnstableIdx++)
+        if (Beg[j] != UnstableCounters[UnstableIdx] &&
+            UnstableCounters[UnstableIdx] != kUnstableCounter)
+          UnstableCounters[UnstableIdx] = kUnstableCounter;
+    }
+  }
+}
 
 void TracePC::HandleInline8bitCountersInit(uint8_t *Start, uint8_t *Stop) {
   if (Start == Stop) return;
   if (NumModulesWithInline8bitCounters &&
       ModuleCounters[NumModulesWithInline8bitCounters-1].Start == Start) return;
   assert(NumModulesWithInline8bitCounters <
          sizeof(ModuleCounters) / sizeof(ModuleCounters[0]));
   ModuleCounters[NumModulesWithInline8bitCounters++] = {Start, Stop};
auto CoveredFunctionCallback = [&](const PCTableEntry *First, const PCTableEntry *Last) {
     std::vector<uintptr_t> UncoveredPCs;
     for (auto TE = First; TE < Last; TE++)
       if (!ObservedPCs.count(TE->PC))
         UncoveredPCs.push_back(TE->PC);
     Printf("COVERED_FUNC: ");
     UncoveredPCs.empty()
         ? Printf("all")
         : Printf("%zd/%zd", (Last - First) - UncoveredPCs.size(), Last - First);
     Printf(" PCs covered %s %s:%zd\n", FunctionStr.c_str(), FileStr.c_str(),
            Line);
     for (auto PC: UncoveredPCs) {
       Printf("  UNCOVERED_PC: %s\n",
              DescribePC("%s:%l", GetNextInstructionPc(PC)).c_str());
     }
   };
 
   IterateCoveredFunctions(CoveredFunctionCallback);
 }
 
 void TracePC::DumpCoverage() {
-  if (EF->__sanitizer_dump_coverage) {
-    Vector<uintptr_t> PCsCopy(GetNumPCs());
-    for (size_t i = 0; i < GetNumPCs(); i++)
-      PCsCopy[i] = PCs()[i] ? GetPreviousInstructionPc(PCs()[i]) : 0;
-    EF->__sanitizer_dump_coverage(PCsCopy.data(), PCsCopy.size());
-  }
+  if (!EF->__sanitizer_dump_coverage)
+    return;
+
+
+void TracePC::DumpUnstableCoverage() {
+  if (!EF->__sanitizer_dump_coverage)
+    return;
+
+  Vector<uintptr_t> PCsCopy(GetNumPCs());
+  for (size_t i = 0; i < GetNumPCs(); i++)
+    PCsCopy[i] = 0;
+
+  size_t UnstableIdx = 1;
+  if (NumInline8bitCounters == NumPCsInPCTables) {
+    for (size_t i = 0; i < NumModulesWithInline8bitCounters; i++) {
+      uint8_t *Beg = ModuleCounters[i].Start;
+      size_t Size = ModuleCounters[i].Stop - Beg;
+      assert(Size ==
+             (size_t)(ModulePCTable[i].Stop - ModulePCTable[i].Start));
+      for (size_t j = 0; j < Size; j++, UnstableIdx++)
+        if (UnstableCounters[UnstableIdx] == kUnstableCounter)
+          PCsCopy[UnstableIdx] = ModulePCTable[i].Start[j].PC;
+    }
+  }
+  EF->__sanitizer_dump_coverage(PCsCopy.data(), PCsCopy.size());
 }
 
 // Value profile.
 // We keep track of various values that affect control flow.
 // These values are inserted into a bit-set-based hash map.
 // Every new bit in the map is treated as a new coverage.
 //
 // For memcmp/strcmp/etc the interesting value is the length of the common
 // prefix of the parameters.
 // For cmp instructions the interesting value is a XOR of the parameters.
 // The interesting value is mixed up with the PC and is then added to the map.
 
 ATTRIBUTE_NO_SANITIZE_ALL
 void TracePC::AddValueForMemcmp(void *caller_pc, const void *s1, const void *s2,
                                 size_t n, bool StopAtZero) {

test/fuzzer/DumpUnstableCoverageTest.cpp

+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+
+int x = 0;
+bool skip0 = false;
+bool skip1 = false;
+bool skip2 = false;
+
+__attribute__((noinline))
+  void det0() { x++; }
+__attribute__((noinline))
+  void det1() { x++; }
+__attribute__((noinline))
+  void det2() { x++; }
+__attribute__((noinline))
+  void det3() { x++; }
+__attribute__((noinline))
+  void det4() { x++; }
+
+__attribute__((noinline))
+  void ini0() { x++; }
+__attribute__((noinline))
+  void ini1() { x++; }
+__attribute__((noinline))
+  void ini2() { x++; }
+
+__attribute__((noinline))
+  void t0() { x++; }
+__attribute__((noinline))
+  void t1() { x++; }
+__attribute__((noinline))
+  void t2() { x++; }
+__attribute__((noinline))
+  void t3() { x++; }
+__attribute__((noinline))
+  void t4() { x++; }
+
+  extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) {
+
+    if (Size == 1 && Data[0] == 'A' && !skip0) {
+      skip0 = true;
+      ini0();
+    }
+    if (Size == 1 && Data[0] == 'B' && !skip1) {
+      skip1 = true;
+      ini1();
+    }
+    if (Size == 1 && Data[0] == 'C' && !skip2) {
+      skip2 = true;
+      ini2();
+    }
+
+    det0();
+    det1();
+
+    int a = rand();
+
+    det2();
+
+    switch(a % 5) {
+      case 0 :
+        t0();
+        break;
+      case 1:
+        t1();
+        break;
+      case 2:
+        t2();
+        break;
+      case 3:
+        t3();
+        break;
+      default:
+        t4();
+        break;
+    }
+
+    det3();
+    det4();
+
+    return 0;
+  }

test/fuzzer/dump_unstable_coverage.test

+RUN: %cpp_compiler -fsanitize=fuzzer %S/DumpUnstableCoverageTest.cpp -o %t-DumpUnstableCoverageTest
+
+RUN: rm -rf %t_workdir && mkdir -p %t_workdir
+
+RUN: env ASAN_OPTIONS=coverage_dir='"%t_workdir"' %run %t-DumpUnstableCoverageTest -dump_unstable_coverage=1 -runs=100000 2>&1 | FileCheck %s --check-prefix=ALLCOV
+ALLCOV: SanitizerCoverage: {{.*}}DumpUnstableCoverageTest.{{.*}}.sancov: {{.*}} PCs written
+
+RUN: sancov -covered-functions %t-DumpUnstableCoverageTest* %t_workdir/*.sancov | FileCheck %s --check-prefix=SANCOV1
+SANCOV1: LLVMFuzzerTestOneInput
+SANCOV1-DAG: t{{[0-4]}}
+SANCOV1-DAG: ini{{[0-2]}}
+SANCOV1-NOT: det{{[0-4]}}
+SANCOV1-NOT: rand
+
+RUN: rm %t_workdir/*.sancov
+
+RUN: env ASAN_OPTIONS=coverage_dir='"%t_workdir"' %run %t-DumpUnstableCoverageTest -dump_unstable_coverage=2 -runs=100000 2>&1 | FileCheck %s --check-prefix=INICOV
+INICOV: SanitizerCoverage: {{.*}}DumpUnstableCoverageTest.{{.*}}.sancov: {{.*}} PCs written
+
+RUN: sancov -covered-functions %t-DumpUnstableCoverageTest* %t_workdir/*.sancov | FileCheck %s --check-prefix=SANCOV2
+SANCOV2: LLVMFuzzerTestOneInput
+SANCOV2-DAG: t{{[0-4]}}
+SANCOV2-NOT: ini{{[0-2]}}
+SANCOV2-NOT: det{{[0-4]}}
+SANCOV2-NOT: rand
+
+RUN: env ASAN_OPTIONS=coverage_dir='"%t_workdir"' %run %t-DumpUnstableCoverageTest -dump_unstable_coverage=0 -runs=10 2>&1 | FileCheck %s --check-prefix=NOCOV
+NOCOV-NOT: SanitizerCoverage: {{.*}} PCs written