[libFuzzer] Mutation tracking and logging implemented.

[kodewilliams]

Summary

Code now exists to track number of mutations that are used in fuzzing in total
and ones that produce new coverage. The stats are currently being dumped to the
command line.

Patch by Kodé Williams (@kodewilliams).

[metzman]

lib/fuzzer/FuzzerMutate.cpp

72 ↗

(On Diff #151532)

Does anyone else find it cleaner to increment in MutateImpl (in FuzzerMutate.cpp), so that there are just 2 calls to IncrementCount?
I think there would be a negligible performance hit, but I think it would be worth it.

lib/fuzzer/FuzzerMutationStats.cpp

16 ↗

(On Diff #151532)

Let's change this to NAME_OF_STAT like the other C Macros in this progam.

[Dor1s]

lib/fuzzer/FuzzerMutate.cpp

72 ↗

(On Diff #151532)

Good point! Also, that code needs to be under if (Options.PrintMutationStats). So, having it in a single place instead of 10+ would be better.

[kodewilliams]

Fixed nits and made code more efficient.

[kodewilliams]

Fixing context not available problem as per LLVM documentation.

[kodewilliams]

More efficient code.

[kodewilliams]

Removed a macro.

[metzman]

Is the whole patch uploaded? I can only see one file.

[kodewilliams]

Encompasses all changes.

[metzman]

@Dor1s I still think using a map that contains string -> count makes most sense.
What is the advantage of using an array? I don't consider the performance benefits worth it since that cost is paid instead when printing the name of a mutation (ie: when this option is not being used).
Additionally this approach is simpler, we only need 2 data structures instead of 3.

lib/fuzzer/FuzzerMutate.cpp

41 ↗

(On Diff #151940)

I think the names here should match the mutations exactly (minus the Mutate_ prefix I guess). So please change this to ChangeAsciiInteger.

44 ↗

(On Diff #151940)

Same as above, please change to CrossOver

52 ↗

(On Diff #151940)

Same as above, please change to AddWordFromTORC

55 ↗

(On Diff #151940)

Same as above, please change to Custom (I don't love the fact that its less descriptive than the name you used)

490 ↗

(On Diff #151940)

Even though the lookuptime is constant, I don't like that we are slowing down fuzzing even when this option is not used.
We wouldn't need to do this if the map was string -> enum Value.

lib/fuzzer/FuzzerMutate.h

115 ↗

(On Diff #151940)

Please remove this blank line so that there is just one blank line between the end of Muatator and private

lib/fuzzer/FuzzerMutationStats.cpp

23 ↗

(On Diff #151940)

Probably nicer to use the iterator as done in FuzzerMutate.cpp:540

29 ↗

(On Diff #151940)

Same as before: Probably nicer to use the iterator as done in FuzzerMutate.cpp:540

38 ↗

(On Diff #151940)

Please remove the paren around the first condition, i don't think it helps readability and is inconsistent with the second condition.

40 ↗

(On Diff #151940)

We can get rid of the else here.

45 ↗

(On Diff #151940)

Please remove the paren around the first condition, i don't think it helps readability and is inconsistent with the second condition.

47 ↗

(On Diff #151940)

We can get rid of the else here.

lib/fuzzer/FuzzerMutationStats.h

25 ↗

(On Diff #151940)

I'm a bit conflicted about using size_t here. On the one hand it is a nice way of saying the largest int type. On the other, this isn't really a size.

[metzman]

I still think using a map that contains string -> count makes most sense.

I meant unordered_map

[Dor1s]

Kode, you also need to write a test for the functionality your're adding. Please check out the section regarding the tests in LLVM Dev Starting Kit. Then, take a look at the existing tests: https://github.com/llvm-mirror/compiler-rt/tree/master/test/fuzzer, especially at "fuzzer-finalstats.test".

While thinking about the test, I've realized the following. Since we eventually plan to remove the detailed statistics output (with mutation names), we will have to update the tests after that. The fact that we have that detailed output right now would also slightly complicate writing a test for you.

So, my proposal would be to remove the detailed output at all, just print those useful / total float numbers in a single line, as we discussed today. We still be using MutationType enum type for tracking mutation stats, but we can get rid of kMutationNames and the code that uses it. Struct Mutator would have both name and identifier. That would simplify the code and also automatically address some of Jonathan's comments.

lib/fuzzer/FuzzerMutate.h

112 ↗

(On Diff #151940)

Please see my main comment to this CL, and let's have both const char *Name; and MutationType Identifier; in this struct.

170 ↗

(On Diff #151940)

I've just realized that this lookup thing is unnecessary complicated which we'll eventually have to remove anyway. Let's do this now, please see my main comment to the CL.

lib/fuzzer/FuzzerMutationStats.cpp

38 ↗

(On Diff #151940)

Actually, inverse the condition and put it inside an assert:

assert(MType >= 0 && MType < MaxNumberOfMutationTypes);

45 ↗

(On Diff #151940)

Actually, inverse the condition and put it inside an assert:

assert(MType >= 0 && MType < MaxNumberOfMutationTypes);

lib/fuzzer/FuzzerMutationStats.h

25 ↗

(On Diff #151940)

yeah, uint64_t would be a better choice

[kodewilliams]

lib/fuzzer/FuzzerMutate.cpp

44 ↗

(On Diff #151940)

I had to change it because there is a function that exists called CrossOver and it was the cause of those template errors. Same goes for ChangeBinInt instead of using ChangeBinaryInteger.

490 ↗

(On Diff #151940)

The Mutator struct doesnt have a name anymore, it just has the function pointer and an identifier of type MutationType, and as a result, there has to be some lookup in the unordered_map for the name regardless

[kodewilliams]

Wrote tests for new feature and updated enums.

[Dor1s]

Did you forget to do git add for the test file you've written?

lib/fuzzer/FuzzerMutate.cpp

34 ↗

(On Diff #152396)

Please use original string names, like the one on line 42, those look better. Plus, we shouldn't change any existing code without a strong reason.
Also, please make sure that every line is no longer than 80 characters. You can do it manually, or use clang-format utility for that, it should be in your `build/bin' directory, but be careful, as it can do some unnecessary changes which again we shouldn't do.

lib/fuzzer/FuzzerMutate.h

27 ↗

(On Diff #152396)

I think we don't need this comment and the one on line 31 anymore, please remove.

38 ↗

(On Diff #152396)

just call it NUMBER_OF_MUTATION_TYPESor maybe MUTATION_TYPES_NUMBER

lib/fuzzer/FuzzerMutationStats.cpp

25 ↗

(On Diff #152396)

It doesn't look like you've uploaded your change properly. All the code of MutationStats::PrintMutationCounts() should be highlighted as new, not as a context. Please upload a diff made against origin/master or origin/HEAD.

[kodewilliams]

Made requested changes.

[kodewilliams]

Removed PrintMutationCounts function

[Dor1s]

Left some minor comments, looks good otherwise! Once Jonathan is also happy with that, we should add Matt (@morehouse) as the next review step.

lib/fuzzer/FuzzerLoop.cpp

667 ↗

(On Diff #152501)

I think we should put it under if (Options.PrintMutationUsefulness) for now, as the stats aren't used otherwise.

lib/fuzzer/FuzzerMutate.cpp

529 ↗

(On Diff #152501)

Let's put it under if (Options.PrintMutationUsefulness) for now, as the stats aren't used otherwise.

lib/fuzzer/FuzzerMutate.h

22 ↗

(On Diff #152501)

It would be simpler to do:

enum MutationType {
  ADD_WORD_FROM_MANUAL_DICTIONARY,
  ...
  etc
};

lib/fuzzer/FuzzerMutationStats.cpp

23 ↗

(On Diff #152501)

I think the current version looks unnecessary messy, let's just do something like:

// +1 is not significant, but prevents us from dividing by zero.
double Ratio = static_cast<double>(100 * UsefulMutations.at(i)) / (1 + TotalMutations.at(i));

lib/fuzzer/FuzzerMutationStats.h

1 ↗

(On Diff #152501)

please fix this first line to be the same length and format as it is in other files

test/fuzzer/fuzzer-mutationstats.test

3 ↗

(On Diff #152501)

add a comment before that line, something like "# Make sure there are some non-zero values in the usefulness percentages printed."

[Dor1s]

Since Matt got automatically added to the subscribers, here is some context. Kode is an intern on our team who is looking into leveraging mutation usefulness statistics somehow to improve fuzzing efficiency. This step only adds stats computation under an experimental flag. The next step would be to enable those stats on ClusterFuzz and collect usefulness distribution from various fuzz targets running in various circumstances. While stats will be collected, Kode will be trying different approaches for mutation selection algorithm (instead of existing Rand()) and, as a result, expected to add that new algorithm under this or another experimental flag.

[kodewilliams]

Just addressed host comments and suggestions.

[metzman]

lib/fuzzer/FuzzerMutationStats.cpp

23 ↗

(On Diff #152501)

+1

29 ↗

(On Diff #152501)

I think every arithmetic operator in libFuzzer has spaces around it. Please do that here by adding spaces around the "-"

30 ↗

(On Diff #152501)

This if isn't needed. else is enough.
Please remove it

30 ↗

(On Diff #152501)

This if isn't needed, else is enough

test/fuzzer/fuzzer-mutationstats.test

2 ↗

(On Diff #152501)

We don't need -max_total_time=360 -print_final_stats=1 -print_mutation_stats=1
Remove them please

[metzman]

It looks like the line numbers on my last comments were messed up, trying again.

lib/fuzzer/FuzzerMutationStats.cpp

26 ↗

(On Diff #152553)

I think every arithmetic operator in libFuzzer has spaces around it. Please do that here by adding spaces around the "-"

27 ↗

(On Diff #152553)

This if isn't needed. else is enough.
Please remove it

29 ↗

(On Diff #152501)

I messed up with this comment please ignore

30 ↗

(On Diff #152501)

I messed up with this comment please ignore

[kodewilliams]

Still addressing comments.

[kodewilliams]

PTAL

[metzman]

lib/fuzzer/FuzzerMutationStats.cpp

24 ↗

(On Diff #152710)

Please rename Ratio since it is a percent not a ratio.
Also, what if you do 100. * UsefulMutations.at(i), do you still need to static_cast this?

test/fuzzer/fuzzer-mutationstats.test

3 ↗

(On Diff #152501)

Sorry, I think comments in this format begin with ; not #.
See afl-driver-stderr.test as an example.

[kodewilliams]

More comments :)

[Dor1s]

lib/fuzzer/FuzzerMutate.cpp

544 ↗

(On Diff #152731)

nit: for const auto& M

test/fuzzer/fuzzer-mutationstats.test

3 ↗

(On Diff #152501)

Not a big deal, but looks like only AFL tests have that syntax, others use # for comments (search for ^# file:\.test file:^src/third_party/llvm/compiler-rt/test/fuzzer/ in on the internal CS)

[kodewilliams]

Had to fix a test.

[morehouse]

lib/fuzzer/FuzzerDefs.h

15 ↗

(On Diff #152774)

Can we include this only in the files that need it?

lib/fuzzer/FuzzerFlags.def

156 ↗

(On Diff #152774)

Maybe print_mutation_stats would be a more accurate name?

lib/fuzzer/FuzzerLoop.cpp

369 ↗

(On Diff #152774)

Should this go before the short-circuit above? Otherwise these stats won't be printed if PrintFinalStats is false.

lib/fuzzer/FuzzerMutate.h

38 ↗

(On Diff #152774)

Maybe call this kNumMutationTypes or similar to distinguish it from the other enum values. That might help prevent future contributors from adding new values after it.

lib/fuzzer/FuzzerMutationStats.h

18 ↗

(On Diff #152774)

Can we just omit the destructor since it has an empty implementation anyway?

[morehouse]

lib/fuzzer/FuzzerMutationStats.cpp

25 ↗

(On Diff #152798)

Can we just do:

double UsefulPercentage = TotalMutations.at(i) ? ... : 0

26 ↗

(On Diff #152798)

The output will be easier to understand if you also print the mutation names.

18 ↗

(On Diff #152774)

Can we just do MutationStats MStats instead of making this on the heap?

test/fuzzer/fuzzer-mutationstats.test

5 ↗

(On Diff #152798)

I think this regex can be simplified to {{[0-9]+\.[0-9]+}}

[kodewilliams]

lib/fuzzer/FuzzerLoop.cpp

369 ↗

(On Diff #152774)

Mutation stats relies on final stats. This way, the same line doesnt have to be put into the code 10 times (like PrintFinalStats() is).

[morehouse]

lib/fuzzer/FuzzerLoop.cpp

369 ↗

(On Diff #152774)

I don't think moving this above the short circuit should affect that. It will just make your flag have the same behavior as -print_coverage, -dump_coverage, etc. which don't require -print_final_stats=1 to work.

[kodewilliams]

lib/fuzzer/FuzzerMutationStats.cpp

26 ↗

(On Diff #152798)

I was told to remove the names from the output for parsing purposes, as it will make it easier to use in the latter phase of the project. Does this still stand?

[morehouse]

lib/fuzzer/FuzzerMutationStats.cpp

25 ↗

(On Diff #152923)

Please remove dead code.

27 ↗

(On Diff #152923)

This formatting looks odd. Please use clang-format on your changes.

26 ↗

(On Diff #152798)

I was interpreting this patch as an option to print human-readable stats similar to -print_final_stats.

@kcc, @Dor1s: Do we care if this information is human readable or not?

[metzman]

lib/fuzzer/FuzzerMutationStats.cpp

26 ↗

(On Diff #152798)

@Dor1s is OOO for 2 weeks.
But I think he was more concerned that the output be easy for libFuzzer to consume than human readable.
I believe he wanted it output as a CSV without any column names.

[morehouse]

test/fuzzer/fuzzer-mutationstats.test

2 ↗

(On Diff #152956)

You should be able to remove -print_final_stats=1 now.

[kcc]

lib/fuzzer/FuzzerLoop.cpp

667 ↗

(On Diff #152956)

Shouldn't we call this near the call to RecordSuccessfulMutationSequence, or even inside it?
Also, I think it's find to collect the stats even when printing stats is not requested,
as soon as the stats collection is cheap. This way will have less code.

lib/fuzzer/FuzzerMutate.h

22 ↗

(On Diff #152956)

ouch. please find a way w/o introducing this enum

lib/fuzzer/FuzzerMutationStats.h

1 ↗

(On Diff #152956)

this is a bit over-engineered.
Just put this into FuzzerMutate.h and please make it simpler.

I guess it could be two extra Vector<uint64_t> objects inside MutationDispatcher
which you resize in the MutationDispatcher CTOR.

[kodewilliams]

Removed FuzzerMutationStats.* files and simplified implementation inside FuzzerMutate.

[kcc]

lib/fuzzer/FuzzerMutate.cpp

33 ↗

(On Diff #153230)

This is still pretty gross. You don't need to assign numbers to mutations.
When you add a mutation to DefaultMutators that becomes the mutation's number.
And so you also should not need "int Identifier;" above.

66 ↗

(On Diff #153230)

you should use Mutators.size() here in stead of kNumMutationTypes

531 ↗

(On Diff #153230)

instead of having M.Identifier, change the line above

auto M = Mutators[Rand(Mutators.size())];

to something like

size_t MutatorIdx = Rand(Mutators.size());   
auto M = Mutators[MutatorIdx];

and then use MutatorIdx

lib/fuzzer/FuzzerMutate.h

22 ↗

(On Diff #153230)

you should not need this.

154 ↗

(On Diff #153230)

check text alignment.

lib/fuzzer/FuzzerOptions.h

56 ↗

(On Diff #153230)

stale?

[kodewilliams]

lib/fuzzer/FuzzerMutate.cpp

531 ↗

(On Diff #153230)

@kcc I had been using the mutation identifier to increment useful counts but now since it is gone, the only way I can think of getting the correct index inside CountCurrentMutatorSequence is by looking up each element in the vector to get it's index then using that, which is expensive. Since this is not done a very large number of times, is it acceptable?

[kcc]

the current patch is hard to review as it contains tons of formatting (or otherwise unrelated) fixed.
Plz don't mix formatting/refactoring with meaningful changes in one patch.

FuzzerDefs.h

19

plz don't include unrelated changes, they make the review harder.

FuzzerMutate.cpp

68

again, don;t include unrelated changes.
here and below.

[kodewilliams]

@kcc Insignificant changes are undone. PTAL.

[kcc]

add a .lit test, please

FuzzerMutate.cpp

586–587

still an unrelated change?

FuzzerMutate.h

98

do you still need to move "private:"

134

Now that you have CurrentMutatorIdxSequence you probably can get rid of CurrentMutatorSequence

[kodewilliams]

Re-added test and removed other insignificant changes.

[kcc]

a couple of nits, then good to go.

lib/fuzzer/FuzzerMutate.cpp

549 ↗

(On Diff #153564)

the loop bound is a size_t, so please use size_t

lib/fuzzer/FuzzerMutate.h

91 ↗

(On Diff #153564)

Maybe try to find a better name.
(Nitpicking, feel free to ignore)

[kodewilliams]

Fixed final nits, good to go @kcc

[kcc]

LGTM

[kodewilliams]

@kcc @morehouse requesting these changes be landed.

[kcc]

Matt, please land it

[morehouse]

Landed in r336597.

[morehouse]

Reverted due to bot breakage in r336616.

FAIL: libFuzzer :: fuzzer-custommutator.test (6330 of 6439)
******************** TEST 'libFuzzer :: fuzzer-custommutator.test' FAILED ********************
Script:
--
: 'RUN: at line 1';     /b/sanitizer-x86_64-linux/build/llvm_build64/bin/clang  --driver-mode=g++ -std=c++11 -lstdc++ -O2 -gline-tables-only -fsanitize=address,fuzzer -I/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer -m64 /b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/test/fuzzer/CustomMutatorTest.cpp -o /b/sanitizer-x86_64-linux/build/compiler_rt_build/test/fuzzer/Output/fuzzer-custommutator.test.tmp-CustomMutatorTest
: 'RUN: at line 2';   not  /b/sanitizer-x86_64-linux/build/compiler_rt_build/test/fuzzer/Output/fuzzer-custommutator.test.tmp-CustomMutatorTest 2>&1 | FileCheck /b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/test/fuzzer/fuzzer-custommutator.test --check-prefix=LLVMFuzzerCustomMutator
--
Exit Code: 1

Command Output (stderr):
--
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/test/fuzzer/fuzzer-custommutator.test:4:26: error: expected string not found in input
LLVMFuzzerCustomMutator: BINGO
                         ^
<stdin>:8:1: note: scanning from here
=================================================================
^
<stdin>:9:9: note: possible intended match here
==57316==ERROR: AddressSanitizer: attempting double-free on 0x602000000050 in thread T0:
        ^

See http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/16996.

[kodewilliams]

@morehouse I have been looking over the code and I cannot seem to find a reason why that single test is failing. PTAL.

[metzman]

@kodewilliams have you rerun the tests to find the failing one?

[kubamracek]

I have seen the same failure when the commit was landed and I debugged it a little bit. The problem was that CustomMutatorTest.cpp's LLVMFuzzerCustomMutator implementation calls LLVMFuzzerMutate, which uses the default set of mutators. This causes a discrepancy: MutateImpl will log into CurrentMutatorIdxSequence that a mutator with an index between 0..12 (from the default set) was used, but when printing the sequence (in PrintMutationSequence), we only expect to find indexes from MutationDispatcher::Mutators. When using a custom mutator (like in this test), MutationDispatcher::Mutators only has one element. So PrintMutationSequence overflows the vector when it tries to use an index from 0..12. Add a if (M >= Mutators.size()) abort(); line into PrintMutationSequence and run the test to reproduce this.

[morehouse]

In D48054#1156815, @kodewilliams wrote:

@morehouse I have been looking over the code and I cannot seem to find a reason why that single test is failing. PTAL.

I suspect this has something to do with LLVMFuzzerMutate being called from CustomMutatorTest.cpp, which uses DefaultMutators instead of Mutators. So your CurrentMutatorIdxSequence vector gets indexes from both DefaultMutators and Mutators.

[kodewilliams]

@morehouse tests are passing now.

[morehouse]

lib/fuzzer/FuzzerMutate.cpp

530 ↗

(On Diff #155036)

I don't like this solution. If we don't want to support mutation stats for custom mutators, we should print an error and exit before any fuzzing happens.

Otherwise, we should make stats work with custom mutators too.

[metzman]

lib/fuzzer/FuzzerMutate.cpp

530 ↗

(On Diff #155036)

+1 for exit.

[kodewilliams]

Tests now all pass and custom mutations can be tracked.

[morehouse]

LGTM

[Dor1s]

Rebase and getting ready to commit.

[Dor1s]

I'm seeing a build failure on one of the bots http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/13625/steps/build%20with%20ninja/logs/stdio :

[3726/4950] Building CXX object projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerMutate.cpp.o
FAILED: projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerMutate.cpp.o 
/b/sanitizer-x86_64-linux/build/clang_build/bin/clang++   -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iprojects/compiler-rt/lib/fuzzer -I/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer -I/usr/include/libxml2 -Iinclude -I/b/sanitizer-x86_64-linux/build/llvm/include -gmlt -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -Werror -std=c++11 -Wno-unused-parameter -O3    -UNDEBUG  -gmlt -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -Werror -std=c++11 -Wno-unused-parameter -m64 -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-non-virtual-dtor -nostdinc++ -D_LIBCPP_ABI_VERSION=Fuzzer -fno-omit-frame-pointer -isystem /b/sanitizer-x86_64-linux/build/llvm_build_ninja/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1 -MD -MT projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerMutate.cpp.o -MF projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerMutate.cpp.o.d -o projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerMutate.cpp.o -c /b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:33:64: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_EraseBytes, "EraseBytes"},
                                                               ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:34:64: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_InsertByte, "InsertByte"},
                                                               ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:36:33: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
           "InsertRepeatedBytes"},
                                ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:37:64: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_ChangeByte, "ChangeByte"},
                                                               ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:38:62: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_ChangeBit, "ChangeBit"},
                                                             ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:39:68: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_ShuffleBytes, "ShuffleBytes"},
                                                                   ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:40:76: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_ChangeASCIIInteger, "ChangeASCIIInt"},
                                                                           ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:41:75: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_ChangeBinaryInteger, "ChangeBinInt"},
                                                                          ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:42:60: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_CopyPart, "CopyPart"},
                                                           ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:43:62: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
          {&MutationDispatcher::Mutate_CrossOver, "CrossOver"},
                                                             ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:45:24: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
           "ManualDict"},
                       ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:47:26: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
           "PersAutoDict"},
                         ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:51:60: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
        {&MutationDispatcher::Mutate_AddWordFromTORC, "CMP"});
                                                           ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:54:69: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
    Mutators.push_back({&MutationDispatcher::Mutate_Custom, "Custom"});
                                                                    ^
/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/fuzzer/FuzzerMutate.cpp:60:72: error: missing field 'UsefulCount' initializer [-Werror,-Wmissing-field-initializers]
        {&MutationDispatcher::Mutate_CustomCrossOver, "CustomCrossOver"});
                                                                       ^
15 errors generated.

[Dor1s]

Reverted in https://reviews.llvm.org/rCRT337206.

Kode, please add -DLLVM_ENABLE_WERROR=ON flag to your cmake command to reproduce the failure locally.

[Dor1s]

Actually, I think you might've already used that flag (it is used in LLVM Dev Starting Kit). In that case, you would need to add -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++, I think.

[kodewilliams]

Had to initialize the counters to silence build bot warnings and errors.

[Dor1s]

Hey Matt, do you think we can land this or should also wait for Kostya's approval?

[morehouse]

I think it's fine to land. Kostya approved the previous version, and he probably won't mind the extra 0's in the initialization lists.

It isn't as clean looking, but it silences the warning, and it's probably good practice to specify all fields in the list anyway.

FuzzerDefs.h

 
 #ifndef LLVM_FUZZER_DEFS_H
 #define LLVM_FUZZER_DEFS_H
 
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <cstring>
+#include <memory>
+#include <set>
 #include <string>
 #include <vector>
-#include <set>
-#include <memory>
 
 // Platform detection.
 #ifdef __linux__
 #define LIBFUZZER_APPLE 0
 #define LIBFUZZER_FUCHSIA 0
 #define LIBFUZZER_LINUX 1
 #define LIBFUZZER_NETBSD 0
 #define LIBFUZZER_FREEBSD 0

FuzzerDriver.cpp

int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
   if (Flags.verbosity > 0 && !Dictionary.empty())
     Printf("Dictionary: %zd entries\n", Dictionary.size());
   bool DoPlainRun = AllInputsAreFiles();
   Options.SaveArtifacts =
       !DoPlainRun || Flags.minimize_crash_internal_step;
   Options.PrintNewCovPcs = Flags.print_pcs;
   Options.PrintNewCovFuncs = Flags.print_funcs;
   Options.PrintFinalStats = Flags.print_final_stats;
+  Options.PrintMutationStats = Flags.print_mutation_stats;
   Options.PrintCorpusStats = Flags.print_corpus_stats;
   Options.PrintCoverage = Flags.print_coverage;
   Options.DumpCoverage = Flags.dump_coverage;
   if (Flags.exit_on_src_pos)
     Options.ExitOnSrcPos = Flags.exit_on_src_pos;
   if (Flags.exit_on_item)
     Options.ExitOnItem = Flags.exit_on_item;
   if (Flags.focus_function)

FuzzerFlags.def

FUZZER_FLAG_INT(ignore_remaining_args, 0, "If 1, ignore all arguments passed "
                 "argument parsing.")
 FUZZER_FLAG_STRING(focus_function, "Experimental. "
      "Fuzzing will focus on inputs that trigger calls to this function")
 
 FUZZER_DEPRECATED_FLAG(run_equivalence_server)
 FUZZER_DEPRECATED_FLAG(use_equivalence_server)
 FUZZER_FLAG_INT(analyze_dict, 0, "Experimental")
 FUZZER_DEPRECATED_FLAG(use_clang_coverage)
 FUZZER_FLAG_STRING(data_flow_trace, "Experimental: use the data flow trace")
+FUZZER_FLAG_INT(print_mutation_stats, 0, "Experimental")

FuzzerLoop.cpp

 
 void Fuzzer::PrintFinalStats() {
   if (Options.PrintCoverage)
     TPC.PrintCoverage();
   if (Options.DumpCoverage)
     TPC.DumpCoverage();
   if (Options.PrintCorpusStats)
     Corpus.PrintStats();
+  if (Options.PrintMutationStats)
+    MD.PrintMutationStats();
   if (!Options.PrintFinalStats)
     return;
   size_t ExecPerSec = execPerSec();
   Printf("stat::number_of_executed_units: %zd\n", TotalNumberOfRuns);
   Printf("stat::average_exec_per_sec:     %zd\n", ExecPerSec);
   Printf("stat::new_units_added:          %zd\n", NumberOfNewUnitsAdded);
   Printf("stat::slowest_unit_time_sec:    %zd\n", TimeOfLongestUnitInSeconds);
   Printf("stat::peak_rss_mb:              %zd\n", GetPeakRSSMb());

FuzzerMutate.h

public:
   void AddWordToManualDictionary(const Word &W);
 
   void PrintRecommendedDictionary();
 
   void SetCorpus(const InputCorpus *Corpus) { this->Corpus = Corpus; }
 
   Random &GetRand() { return Rand; }
 
-private:
+  void PrintMutationStats();
+
+  void CountCurrentMutatorSequence();
 
   struct Mutator {
     size_t (MutationDispatcher::*Fn)(uint8_t *Data, size_t Size, size_t Max);
     const char *Name;
   };
 
+private:
   size_t AddWordFromDictionary(Dictionary &D, uint8_t *Data, size_t Size,
                                size_t MaxSize);
   size_t MutateImpl(uint8_t *Data, size_t Size, size_t MaxSize,
                     Vector<Mutator> &Mutators);
 
   size_t InsertPartOf(const uint8_t *From, size_t FromSize, uint8_t *To,
                       size_t ToSize, size_t MaxToSize);
   size_t CopyPartOf(const uint8_t *From, size_t FromSize, uint8_t *To,
private:
   Dictionary ManualDictionary;
   // Temporary dictionary modified by the fuzzer itself,
   // recreated periodically.
   Dictionary TempAutoDictionary;
   // Persistent dictionary modified by the fuzzer, consists of
   // entries that led to successful discoveries in the past mutations.
   Dictionary PersistentAutoDictionary;
 
   Vector<Mutator> CurrentMutatorSequence;
   Vector<DictionaryEntry *> CurrentDictionaryEntrySequence;
+  Vector<size_t> CurrentMutatorIdxSequence;
 
   static const size_t kCmpDictionaryEntriesDequeSize = 16;
   DictionaryEntry CmpDictionaryEntriesDeque[kCmpDictionaryEntriesDequeSize];
   size_t CmpDictionaryEntriesDequeIdx = 0;
 
   const InputCorpus *Corpus = nullptr;
   Vector<uint8_t> MutateInPlaceHere;
   // CustomCrossOver needs its own buffer as a custom implementation may call
   // LLVMFuzzerMutate, which in turn may resize MutateInPlaceHere.
   Vector<uint8_t> CustomCrossOverInPlaceHere;
 
   Vector<Mutator> Mutators;
   Vector<Mutator> DefaultMutators;
+
+  // A total count of each mutation used in the fuzzing process.
+  Vector<uint64_t> TotalMutations;
+  // The number of each mutation that resulted in new coverage.
+  Vector<uint64_t> UsefulMutations;
 };
 
 }  // namespace fuzzer
 
 #endif  // LLVM_FUZZER_MUTATE_H

FuzzerMutate.cpp

DefaultMutators.insert(
           {&MutationDispatcher::Mutate_ChangeBinaryInteger, "ChangeBinInt"},
           {&MutationDispatcher::Mutate_CopyPart, "CopyPart"},
           {&MutationDispatcher::Mutate_CrossOver, "CrossOver"},
           {&MutationDispatcher::Mutate_AddWordFromManualDictionary,
            "ManualDict"},
           {&MutationDispatcher::Mutate_AddWordFromPersistentAutoDictionary,
            "PersAutoDict"},
       });
-  if(Options.UseCmp)
+  if (Options.UseCmp)
     DefaultMutators.push_back(
         {&MutationDispatcher::Mutate_AddWordFromTORC, "CMP"});
 
   if (EF->LLVMFuzzerCustomMutator)
     Mutators.push_back({&MutationDispatcher::Mutate_Custom, "Custom"});
   else
     Mutators = DefaultMutators;
 
   if (EF->LLVMFuzzerCustomCrossOver)
     Mutators.push_back(
         {&MutationDispatcher::Mutate_CustomCrossOver, "CustomCrossOver"});
+
+  // Initialize mutation statistic counters.
+  TotalMutations.resize(Mutators.size(), 0);
+  UsefulMutations.resize(Mutators.size(), 0);
 }
 
 static char RandCh(Random &Rand) {
-  if (Rand.RandBool()) return Rand(256);
+  if (Rand.RandBool())
+    return Rand(256);
   const char Special[] = "!*'();:@&=+$,/?%#[]012Az-`~.\xff\x00";
   return Special[Rand(sizeof(Special) - 1)];
 }
 
 size_t MutationDispatcher::Mutate_Custom(uint8_t *Data, size_t Size,
                                          size_t MaxSize) {
   return EF->LLVMFuzzerCustomMutator(Data, Size, MaxSize, Rand.Rand());
 }
if (!NewSize)
     return 0;
   assert(NewSize <= MaxSize && "CustomCrossOver returned overisized unit");
   memcpy(Data, U.data(), NewSize);
   return NewSize;
 }
 
 size_t MutationDispatcher::Mutate_ShuffleBytes(uint8_t *Data, size_t Size,
                                                size_t MaxSize) {
-  if (Size > MaxSize || Size == 0) return 0;
+  if (Size > MaxSize || Size == 0)
+    return 0;
   size_t ShuffleAmount =
       Rand(std::min(Size, (size_t)8)) + 1; // [1,8] and <= Size.
   size_t ShuffleStart = Rand(Size - ShuffleAmount);
   assert(ShuffleStart + ShuffleAmount <= Size);
   std::shuffle(Data + ShuffleStart, Data + ShuffleStart + ShuffleAmount, Rand);
   return Size;
 }
 
 size_t MutationDispatcher::Mutate_EraseBytes(uint8_t *Data, size_t Size,
                                              size_t MaxSize) {
-  if (Size <= 1) return 0;
+  if (Size <= 1)
+    return 0;
   size_t N = Rand(Size / 2) + 1;
   assert(N < Size);
   size_t Idx = Rand(Size - N + 1);
   // Erase Data[Idx:Idx+N].
   memmove(Data + Idx, Data + Idx + N, Size - Idx - N);
   // Printf("Erase: %zd %zd => %zd; Idx %zd\n", N, Size, Size - N, Idx);
   return Size - N;
 }
 
 size_t MutationDispatcher::Mutate_InsertByte(uint8_t *Data, size_t Size,
                                              size_t MaxSize) {
-  if (Size >= MaxSize) return 0;
+  if (Size >= MaxSize)
+    return 0;
   size_t Idx = Rand(Size + 1);
   // Insert new value at Data[Idx].
   memmove(Data + Idx + 1, Data + Idx, Size - Idx);
   Data[Idx] = RandCh(Rand);
   return Size + 1;
 }
 
 size_t MutationDispatcher::Mutate_InsertRepeatedBytes(uint8_t *Data,
                                                       size_t Size,
                                                       size_t MaxSize) {
   const size_t kMinBytesToInsert = 3;
-  if (Size + kMinBytesToInsert >= MaxSize) return 0;
+  if (Size + kMinBytesToInsert >= MaxSize)
+    return 0;
   size_t MaxBytesToInsert = std::min(MaxSize - Size, (size_t)128);
   size_t N = Rand(MaxBytesToInsert - kMinBytesToInsert + 1) + kMinBytesToInsert;
   assert(Size + N <= MaxSize && N);
   size_t Idx = Rand(Size + 1);
   // Insert new values at Data[Idx].
   memmove(Data + Idx + N, Data + Idx, Size - Idx);
   // Give preference to 0x00 and 0xff.
   uint8_t Byte = Rand.RandBool() ? Rand(256) : (Rand.RandBool() ? 0 : 255);
   for (size_t i = 0; i < N; i++)
     Data[Idx + i] = Byte;
   return Size + N;
 }
 
 size_t MutationDispatcher::Mutate_ChangeByte(uint8_t *Data, size_t Size,
                                              size_t MaxSize) {
-  if (Size > MaxSize) return 0;
+  if (Size > MaxSize)
+    return 0;
   size_t Idx = Rand(Size);
   Data[Idx] = RandCh(Rand);
   return Size;
 }
 
 size_t MutationDispatcher::Mutate_ChangeBit(uint8_t *Data, size_t Size,
                                             size_t MaxSize) {
-  if (Size > MaxSize) return 0;
+  if (Size > MaxSize)
+    return 0;
   size_t Idx = Rand(Size);
   Data[Idx] ^= 1 << Rand(8);
   return Size;
 }
 
 size_t MutationDispatcher::Mutate_AddWordFromManualDictionary(uint8_t *Data,
                                                               size_t Size,
                                                               size_t MaxSize) {
   return AddWordFromDictionary(ManualDictionary, Data, Size, MaxSize);
 }
 
 size_t MutationDispatcher::ApplyDictionaryEntry(uint8_t *Data, size_t Size,
                                                 size_t MaxSize,
                                                 DictionaryEntry &DE) {
   const Word &W = DE.GetW();
   bool UsePositionHint = DE.HasPositionHint() &&
                          DE.GetPositionHint() + W.size() < Size &&
                          Rand.RandBool();
-  if (Rand.RandBool()) {  // Insert W.
-    if (Size + W.size() > MaxSize) return 0;
+  if (Rand.RandBool()) { // Insert W.
+    if (Size + W.size() > MaxSize)
+      return 0;
     size_t Idx = UsePositionHint ? DE.GetPositionHint() : Rand(Size + 1);
     memmove(Data + Idx + W.size(), Data + Idx, Size - Idx);
     memcpy(Data + Idx, W.data(), W.size());
     Size += W.size();
-  } else {  // Overwrite some bytes with W.
-    if (W.size() > Size) return 0;
+  } else { // Overwrite some bytes with W.
+    if (W.size() > Size)
+      return 0;
     size_t Idx = UsePositionHint ? DE.GetPositionHint() : Rand(Size - W.size());
     memcpy(Data + Idx, W.data(), W.size());
   }
   return Size;
 }
 
 // Somewhere in the past we have observed a comparison instructions
 // with arguments Arg1 Arg2. This function tries to guess a dictionary
 // entry that will satisfy that comparison.
 // It first tries to find one of the arguments (possibly swapped) in the
 // input and if it succeeds it creates a DE with a position hint.
 // Otherwise it creates a DE with one of the arguments w/o a position hint.
 DictionaryEntry MutationDispatcher::MakeDictionaryEntryFromCMP(
-    const void *Arg1, const void *Arg2,
-    const void *Arg1Mutation, const void *Arg2Mutation,
-    size_t ArgSize, const uint8_t *Data,
+    const void *Arg1, const void *Arg2, const void *Arg1Mutation,
+    const void *Arg2Mutation, size_t ArgSize, const uint8_t *Data,
     size_t Size) {
   ScopedDoingMyOwnMemOrStr scoped_doing_my_own_mem_os_str;
   bool HandleFirst = Rand.RandBool();
   const void *ExistingBytes, *DesiredBytes;
   Word W;
   const uint8_t *End = Data + Size;
   for (int Arg = 0; Arg < 2; Arg++) {
     ExistingBytes = HandleFirst ? Arg1 : Arg2;
     DesiredBytes = HandleFirst ? Arg2Mutation : Arg1Mutation;
     HandleFirst = !HandleFirst;
-    W.Set(reinterpret_cast<const uint8_t*>(DesiredBytes), ArgSize);
+    W.Set(reinterpret_cast<const uint8_t *>(DesiredBytes), ArgSize);
     const size_t kMaxNumPositions = 8;
     size_t Positions[kMaxNumPositions];
     size_t NumPositions = 0;
     for (const uint8_t *Cur = Data;
          Cur < End && NumPositions < kMaxNumPositions; Cur++) {
       Cur =
           (const uint8_t *)SearchMemory(Cur, End - Cur, ExistingBytes, ArgSize);
-      if (!Cur) break;
+      if (!Cur)
+        break;
       Positions[NumPositions++] = Cur - Data;
     }
-    if (!NumPositions) continue;
+    if (!NumPositions)
+      continue;
     return DictionaryEntry(W, Positions[Rand(NumPositions)]);
   }
   DictionaryEntry DE(W);
   return DE;
 }
 
-
 template <class T>
 DictionaryEntry MutationDispatcher::MakeDictionaryEntryFromCMP(
     T Arg1, T Arg2, const uint8_t *Data, size_t Size) {
-  if (Rand.RandBool()) Arg1 = Bswap(Arg1);
-  if (Rand.RandBool()) Arg2 = Bswap(Arg2);
+  if (Rand.RandBool())
+    Arg1 = Bswap(Arg1);
+  if (Rand.RandBool())
+    Arg2 = Bswap(Arg2);
   T Arg1Mutation = Arg1 + Rand(-1, 1);
   T Arg2Mutation = Arg2 + Rand(-1, 1);
   return MakeDictionaryEntryFromCMP(&Arg1, &Arg2, &Arg1Mutation, &Arg2Mutation,
                                     sizeof(Arg1), Data, Size);
 }
 
 DictionaryEntry MutationDispatcher::MakeDictionaryEntryFromCMP(
     const Word &Arg1, const Word &Arg2, const uint8_t *Data, size_t Size) {
   return MakeDictionaryEntryFromCMP(Arg1.data(), Arg2.data(), Arg1.data(),
                                     Arg2.data(), Arg1.size(), Data, Size);
 }
 
-size_t MutationDispatcher::Mutate_AddWordFromTORC(
-    uint8_t *Data, size_t Size, size_t MaxSize) {
+size_t MutationDispatcher::Mutate_AddWordFromTORC(uint8_t *Data, size_t Size,
+                                                  size_t MaxSize) {
   Word W;
   DictionaryEntry DE;
   switch (Rand(4)) {
   case 0: {
     auto X = TPC.TORC8.Get(Rand.Rand());
     DE = MakeDictionaryEntryFromCMP(X.A, X.B, Data, Size);
   } break;
   case 1: {
     auto X = TPC.TORC4.Get(Rand.Rand());
     if ((X.A >> 16) == 0 && (X.B >> 16) == 0 && Rand.RandBool())
       DE = MakeDictionaryEntryFromCMP((uint16_t)X.A, (uint16_t)X.B, Data, Size);
     else
       DE = MakeDictionaryEntryFromCMP(X.A, X.B, Data, Size);
   } break;
   case 2: {
     auto X = TPC.TORCW.Get(Rand.Rand());
     DE = MakeDictionaryEntryFromCMP(X.A, X.B, Data, Size);
   } break;
-  case 3: if (Options.UseMemmem) {
+  case 3:
+    if (Options.UseMemmem) {
       auto X = TPC.MMT.Get(Rand.Rand());
       DE = DictionaryEntry(X);
-  } break;
+    }
+    break;
   default:
     assert(0);
   }
-  if (!DE.GetW().size()) return 0;
+  if (!DE.GetW().size())
+    return 0;
   Size = ApplyDictionaryEntry(Data, Size, MaxSize, DE);
-  if (!Size) return 0;
+  if (!Size)
+    return 0;
   DictionaryEntry &DERef =
       CmpDictionaryEntriesDeque[CmpDictionaryEntriesDequeIdx++ %
                                 kCmpDictionaryEntriesDequeSize];
   DERef = DE;
   CurrentDictionaryEntrySequence.push_back(&DERef);
   return Size;
 }
 
 size_t MutationDispatcher::Mutate_AddWordFromPersistentAutoDictionary(
     uint8_t *Data, size_t Size, size_t MaxSize) {
   return AddWordFromDictionary(PersistentAutoDictionary, Data, Size, MaxSize);
 }
 
 size_t MutationDispatcher::AddWordFromDictionary(Dictionary &D, uint8_t *Data,
                                                  size_t Size, size_t MaxSize) {
-  if (Size > MaxSize) return 0;
-  if (D.empty()) return 0;
+  if (Size > MaxSize)
+    return 0;
+  if (D.empty())
+    return 0;
   DictionaryEntry &DE = D[Rand(D.size())];
   Size = ApplyDictionaryEntry(Data, Size, MaxSize, DE);
-  if (!Size) return 0;
+  if (!Size)
+    return 0;
   DE.IncUseCount();
   CurrentDictionaryEntrySequence.push_back(&DE);
   return Size;
 }
 
 // Overwrites part of To[0,ToSize) with a part of From[0,FromSize).
 // Returns ToSize.
 size_t MutationDispatcher::CopyPartOf(const uint8_t *From, size_t FromSize,
size_t MutationDispatcher::CopyPartOf(const uint8_t *From, size_t FromSize,
   return ToSize;
 }
 
 // Inserts part of From[0,ToSize) into To.
 // Returns new size of To on success or 0 on failure.
 size_t MutationDispatcher::InsertPartOf(const uint8_t *From, size_t FromSize,
                                         uint8_t *To, size_t ToSize,
                                         size_t MaxToSize) {
-  if (ToSize >= MaxToSize) return 0;
+  if (ToSize >= MaxToSize)
+    return 0;
   size_t AvailableSpace = MaxToSize - ToSize;
   size_t MaxCopySize = std::min(AvailableSpace, FromSize);
   size_t CopySize = Rand(MaxCopySize) + 1;
   size_t FromBeg = Rand(FromSize - CopySize + 1);
   assert(FromBeg + CopySize <= FromSize);
   size_t ToInsertPos = Rand(ToSize + 1);
   assert(ToInsertPos + CopySize <= MaxToSize);
   size_t TailSize = ToSize - ToInsertPos;
   if (To == From) {
     MutateInPlaceHere.resize(MaxToSize);
     memcpy(MutateInPlaceHere.data(), From + FromBeg, CopySize);
     memmove(To + ToInsertPos + CopySize, To + ToInsertPos, TailSize);
     memmove(To + ToInsertPos, MutateInPlaceHere.data(), CopySize);
   } else {
     memmove(To + ToInsertPos + CopySize, To + ToInsertPos, TailSize);
     memmove(To + ToInsertPos, From + FromBeg, CopySize);
   }
   return ToSize + CopySize;
 }
 
 size_t MutationDispatcher::Mutate_CopyPart(uint8_t *Data, size_t Size,
                                            size_t MaxSize) {
-  if (Size > MaxSize || Size == 0) return 0;
+  if (Size > MaxSize || Size == 0)
+    return 0;
   // If Size == MaxSize, `InsertPartOf(...)` will
   // fail so there's no point using it in this case.
   if (Size == MaxSize || Rand.RandBool())
     return CopyPartOf(Data, Size, Data, Size);
   else
     return InsertPartOf(Data, Size, Data, Size, MaxSize);
 }
 
 size_t MutationDispatcher::Mutate_ChangeASCIIInteger(uint8_t *Data, size_t Size,
                                                      size_t MaxSize) {
-  if (Size > MaxSize) return 0;
+  if (Size > MaxSize)
+    return 0;
   size_t B = Rand(Size);
-  while (B < Size && !isdigit(Data[B])) B++;
-  if (B == Size) return 0;
+  while (B < Size && !isdigit(Data[B]))
+    B++;
+  if (B == Size)
+    return 0;
   size_t E = B;
-  while (E < Size && isdigit(Data[E])) E++;
+  while (E < Size && isdigit(Data[E]))
+    E++;
   assert(B < E);
   // now we have digits in [B, E).
   // strtol and friends don't accept non-zero-teminated data, parse it manually.
   uint64_t Val = Data[B] - '0';
   for (size_t i = B + 1; i < E; i++)
     Val = Val * 10 + Data[i] - '0';
 
   // Mutate the integer value.
-  switch(Rand(5)) {
-    case 0: Val++; break;
-    case 1: Val--; break;
-    case 2: Val /= 2; break;
-    case 3: Val *= 2; break;
-    case 4: Val = Rand(Val * Val); break;
-    default: assert(0);
+  switch (Rand(5)) {
+  case 0:
+    Val++;
+    break;
+  case 1:
+    Val--;
+    break;
+  case 2:
+    Val /= 2;
+    break;
+  case 3:
+    Val *= 2;
+    break;
+  case 4:
+    Val = Rand(Val * Val);
+    break;
+  default:
+    assert(0);
   }
   // Just replace the bytes with the new ones, don't bother moving bytes.
   for (size_t i = B; i < E; i++) {
     size_t Idx = E + B - i - 1;
     assert(Idx >= B && Idx < E);
     Data[Idx] = (Val % 10) + '0';
     Val /= 10;
   }
   return Size;
 }
 
-template<class T>
+template <class T>
 size_t ChangeBinaryInteger(uint8_t *Data, size_t Size, Random &Rand) {
-  if (Size < sizeof(T)) return 0;
+  if (Size < sizeof(T))
+    return 0;
   size_t Off = Rand(Size - sizeof(T) + 1);
   assert(Off + sizeof(T) <= Size);
   T Val;
   if (Off < 64 && !Rand(4)) {
     Val = Size;
     if (Rand.RandBool())
       Val = Bswap(Val);
   } else {
size_t ChangeBinaryInteger(uint8_t *Data, size_t Size, Random &Rand) {
   }
   memcpy(Data + Off, &Val, sizeof(Val));
   return Size;
 }
 
 size_t MutationDispatcher::Mutate_ChangeBinaryInteger(uint8_t *Data,
                                                       size_t Size,
                                                       size_t MaxSize) {
-  if (Size > MaxSize) return 0;
+  if (Size > MaxSize)
+    return 0;
   switch (Rand(4)) {
-    case 3: return ChangeBinaryInteger<uint64_t>(Data, Size, Rand);
-    case 2: return ChangeBinaryInteger<uint32_t>(Data, Size, Rand);
-    case 1: return ChangeBinaryInteger<uint16_t>(Data, Size, Rand);
-    case 0: return ChangeBinaryInteger<uint8_t>(Data, Size, Rand);
-    default: assert(0);
+  case 3:
+    return ChangeBinaryInteger<uint64_t>(Data, Size, Rand);
+  case 2:
+    return ChangeBinaryInteger<uint32_t>(Data, Size, Rand);
+  case 1:
+    return ChangeBinaryInteger<uint16_t>(Data, Size, Rand);
+  case 0:
+    return ChangeBinaryInteger<uint8_t>(Data, Size, Rand);
+  default:
+    assert(0);
   }
   return 0;
 }
 
 size_t MutationDispatcher::Mutate_CrossOver(uint8_t *Data, size_t Size,
                                             size_t MaxSize) {
-  if (Size > MaxSize) return 0;
-  if (!Corpus || Corpus->size() < 2 || Size == 0) return 0;
+  if (Size > MaxSize)
+    return 0;
+  if (!Corpus || Corpus->size() < 2 || Size == 0)
+    return 0;
   size_t Idx = Rand(Corpus->size());
   const Unit &O = (*Corpus)[Idx];
-  if (O.empty()) return 0;
+  if (O.empty())
+    return 0;
   MutateInPlaceHere.resize(MaxSize);
   auto &U = MutateInPlaceHere;
   size_t NewSize = 0;
-  switch(Rand(3)) {
-    case 0:
-      NewSize = CrossOver(Data, Size, O.data(), O.size(), U.data(), U.size());
-      break;
-    case 1:
-      NewSize = InsertPartOf(O.data(), O.size(), U.data(), U.size(), MaxSize);
-      if (!NewSize)
-        NewSize = CopyPartOf(O.data(), O.size(), U.data(), U.size());
-      break;
-    case 2:
-      NewSize = CopyPartOf(O.data(), O.size(), U.data(), U.size());
-      break;
-    default: assert(0);
+  switch (Rand(3)) {
+  default:
+    assert(0);
   }
   assert(NewSize > 0 && "CrossOver returned empty unit");
   assert(NewSize <= MaxSize && "CrossOver returned overisized unit");
   memcpy(Data, U.data(), NewSize);
   return NewSize;
 }
 
 void MutationDispatcher::StartMutationSequence() {
   CurrentMutatorSequence.clear();
+  CurrentMutatorIdxSequence.clear();
   CurrentDictionaryEntrySequence.clear();
 }
 
 // Copy successful dictionary entries to PersistentAutoDictionary.
 void MutationDispatcher::RecordSuccessfulMutationSequence() {
   for (auto DE : CurrentDictionaryEntrySequence) {
     // PersistentAutoDictionary.AddWithSuccessCountOne(DE);
     DE->IncSuccessCount();
     assert(DE->GetW().size());
     // Linear search is fine here as this happens seldom.
     if (!PersistentAutoDictionary.ContainsWord(DE->GetW()))
       PersistentAutoDictionary.push_back({DE->GetW(), 1});
   }
+  CountCurrentMutatorSequence();
 }
 
 void MutationDispatcher::PrintRecommendedDictionary() {
   Vector<DictionaryEntry> V;
   for (auto &DE : PersistentAutoDictionary)
     if (!ManualDictionary.ContainsWord(DE.GetW()))
       V.push_back(DE);
-  if (V.empty()) return;
+  if (V.empty())
+    return;
   Printf("###### Recommended dictionary. ######\n");
-  for (auto &DE: V) {
+  for (auto &DE : V) {
     assert(DE.GetW().size());
     Printf("\"");
     PrintASCII(DE.GetW(), "\"");
     Printf(" # Uses: %zd\n", DE.GetUseCount());
   }
   Printf("###### End of recommended dictionary. ######\n");
 }
 
 void MutationDispatcher::PrintMutationSequence() {
   Printf("MS: %zd ", CurrentMutatorSequence.size());
   for (auto M : CurrentMutatorSequence)
     Printf("%s-", M.Name);
   if (!CurrentDictionaryEntrySequence.empty()) {
     Printf(" DE: ");
     for (auto DE : CurrentDictionaryEntrySequence) {
       Printf("\"");
       PrintASCII(DE->GetW(), "\"-");
     }
   }
 }
 
 size_t MutationDispatcher::Mutate(uint8_t *Data, size_t Size, size_t MaxSize) {
   return MutateImpl(Data, Size, MaxSize, Mutators);
 }
 
 size_t MutationDispatcher::DefaultMutate(uint8_t *Data, size_t Size,
                                          size_t MaxSize) {
   return MutateImpl(Data, Size, MaxSize, DefaultMutators);
 }
 
 // Mutates Data in place, returns new size.
 size_t MutationDispatcher::MutateImpl(uint8_t *Data, size_t Size,
                                       size_t MaxSize,
                                       Vector<Mutator> &Mutators) {
   assert(MaxSize > 0);
   // Some mutations may fail (e.g. can't insert more bytes if Size == MaxSize),
   // in which case they will return 0.
   // Try several times before returning un-mutated data.
   for (int Iter = 0; Iter < 100; Iter++) {
-    auto M = Mutators[Rand(Mutators.size())];
+    size_t MutatorIdx = Rand(Mutators.size());
+    auto M = Mutators[MutatorIdx];
     size_t NewSize = (this->*(M.Fn))(Data, Size, MaxSize);
     if (NewSize && NewSize <= MaxSize) {
       if (Options.OnlyASCII)
         ToASCII(Data, NewSize);
       CurrentMutatorSequence.push_back(M);
+      CurrentMutatorIdxSequence.push_back(MutatorIdx);
+      TotalMutations[MutatorIdx]++;
       return NewSize;
     }
   }
   *Data = ' ';
-  return 1;   // Fallback, should not happen frequently.
+  return 1; // Fallback, should not happen frequently.
 }
 
 void MutationDispatcher::AddWordToManualDictionary(const Word &W) {
-  ManualDictionary.push_back(
-      {W, std::numeric_limits<size_t>::max()});
+  ManualDictionary.push_back({W, std::numeric_limits<size_t>::max()});
+}
+
+void MutationDispatcher::CountCurrentMutatorSequence() {
+  for (const size_t M : CurrentMutatorIdxSequence)
+    UsefulMutations[M]++;
+}
+
+void MutationDispatcher::PrintMutationStats() {
+  Printf("\nstat::mutation_usefulness:      ");
+  for (unsigned int i = 0; i < Mutators.size(); i++) {
+    double UsefulPercentage =
+        TotalMutations[i]
+            ? (100.0 * UsefulMutations[i]) / TotalMutations[i]
+            : 0;
+    Printf("%.3f", UsefulPercentage);
+    if (i < Mutators.size() - 1)
+      Printf(",");
+  }
+  Printf("\n");
 }
 
-}  // namespace fuzzer
+} // namespace fuzzer

FuzzerOptions.h

struct FuzzingOptions {
   std::string ExitOnItem;
   std::string FocusFunction;
   std::string DataFlowTrace;
   bool SaveArtifacts = true;
   bool PrintNEW = true; // Print a status line when new units are found;
   bool PrintNewCovPcs = false;
   int PrintNewCovFuncs = 0;
   bool PrintFinalStats = false;
+  bool PrintMutationStats = false;
   bool PrintCorpusStats = false;
   bool PrintCoverage = false;
   bool DumpCoverage = false;
   bool DetectLeaks = true;
   int PurgeAllocatorIntervalSec = 1;
   int  TraceMalloc = 0;
   bool HandleAbrt = false;
   bool HandleBus = false;