This is an archive of the discontinued LLVM Phabricator instance.

[libfuzzer] Reduce default verbosity when printing large mutation sequences
ClosedPublic

Authored by mhlakhani on Aug 26 2020, 1:29 PM.

Details

Summary

When using a custom mutator (e.g. thrift mutator, similar to LPM)
that calls back into libfuzzer's mutations via LLVMFuzzerMutate, the mutation
sequences needed to achieve new coverage can get prohibitively large.

Printing these large sequences has two downsides:

  1. It makes the logs hard to understand for a human.
  2. The performance cost slows down fuzzing.

In this patch I change the PrintMutationSequence function to take a max
number of entries, to achieve this goal. I also update PrintStatusForNewUnit
to default to printing only 10 entries, in the default verbosity level (1),
requiring the user to set verbosity to 2 if they want the full mutation
sequence.

I would like feedback on whether this behaviour change is OK, or if there's
another preferred value for the constant (10).

For our use case, turning off verbosity is not an option, as that would also
disable PrintStats() which is very useful for infrastructure that analyzes
the logs in realtime. I imagine most users of libfuzzer always want those logs
in the default.

Test Plan:

I did not see automated tests for this and I am not sure how to add them. I
would appreciate feedback on how to do so.

I built a fuzzer locally with this patch applied to libfuzzer.

When running with the default verbosity, I see logs like this:

#65 NEW    cov: 4799 ft: 10443 corp: 41/1447Kb lim: 64000 exec/s: 1 rss: 575Mb L: 28658/62542 MS: 196 Custom-CrossOver-ChangeBit-EraseBytes-ChangeBit-ChangeBit-ChangeBit-CrossOver-ChangeBit-CrossOver- DE: "\xff\xff\xff\x0e"-"\xfe\xff\xff\x7f"-"\xfe\xff\xff\x7f"-"\x17\x00\x00\x00\x00\x00\x00\x00"-"\x00\x00\x00\xf9"-"\xff\xff\xff\xff"-"\xfa\xff\xff\xff"-"\xf7\xff\xff\xff"-"@\xff\xff\xff\xff\xff\xff\xff"-"E\x00"-
#67 NEW    cov: 4810 ft: 10462 corp: 42/1486Kb lim: 64000 exec/s: 1 rss: 577Mb L: 39823/62542 MS: 135 Custom-CopyPart-ShuffleBytes-ShuffleBytes-ChangeBit-ChangeBinInt-EraseBytes-ChangeBit-ChangeBinInt-ChangeBit- DE: "\x01\x00\x00\x00\x00\x00\x01\xf1"-"\x00\x00\x00\x07"-"\x00\x0d"-"\xfd\xff\xff\xff"-"\xfe\xff\xff\xf4"-"\xe3\xff\xff\xff"-"\xff\xff\xff\xf1"-"\xea\xff\xff\xff"-"\x00\x00\x00\xfd"-"\x01\x00\x00\x05"-

Staring hard at the logs it's clear that the cap of 10 is applied.

When running with verbosity level 2, the logs look like the below:

#66    NEW    cov: 4700 ft: 10188 corp: 37/1186Kb lim: 64000 exec/s: 2 rss: 509Mb L: 47616/61231 MS: 520 Custom-CopyPart-ChangeBinInt-ChangeBit-ChangeByte-EraseBytes-PersAutoDict-CopyPart-ShuffleBytes-ChangeBit-ShuffleBytes-CopyPart-EraseBytes-CopyPart-ChangeBinInt-CopyPart-ChangeByte-ShuffleBytes-ChangeBinInt-ShuffleBytes-ChangeBit-CMP-ShuffleBytes-ChangeBit-CrossOver-ChangeBinInt-ChangeByte-ShuffleBytes-CrossOver-EraseBytes-ChangeBinInt-InsertRepeatedBytes-PersAutoDict-InsertRepeatedBytes-InsertRepeatedBytes-CrossOver-ChangeByte-ShuffleBytes-CopyPart-ShuffleBytes-CopyPart-CrossOver-ChangeBit-ShuffleBytes-CrossOver-PersAutoDict-ChangeByte-ChangeBit-ShuffleBytes-CrossOver-ChangeByte-EraseBytes-CopyPart-ChangeBinInt-PersAutoDict-CrossOver-ShuffleBytes-CrossOver-CrossOver-EraseBytes-CrossOver-EraseBytes-CrossOver-ChangeBit-ChangeBinInt-ChangeByte-EraseBytes-ShuffleBytes-ShuffleBytes-ChangeBit-EraseBytes-ChangeBinInt-ChangeBit-ChangeBinInt-CopyPart-EraseBytes-PersAutoDict-EraseBytes-CopyPart-ChangeBinInt-ChangeByte-CrossOver-ChangeBinInt-ShuffleBytes-PersAutoDict-PersAutoDict-ChangeBinInt-CopyPart-ChangeBinInt-CrossOver-ChangeBit-ChangeBinInt-CopyPart-ChangeByte-ChangeBit-CopyPart-CrossOver-ChangeByte-ChangeBit-ChangeByte-ShuffleBytes-CMP-ChangeBit-CopyPart-ChangeBit-ChangeByte-ChangeBinInt-PersAutoDict-ChangeBinInt-CrossOver-ChangeBinInt-ChangeBit-ChangeBinInt-ChangeBinInt-PersAutoDict-ChangeBinInt-ChangeBinInt-ChangeByte-CopyPart-ShuffleBytes-ChangeByte-ChangeBit-ChangeByte-ChangeByte-EraseBytes-CrossOver-ChangeByte-ChangeByte-EraseBytes-EraseBytes-InsertRepeatedBytes-ShuffleBytes-CopyPart-CopyPart-ChangeBit-ShuffleBytes-PersAutoDict-ShuffleBytes-ChangeBit-ChangeByte-ChangeBit-ShuffleBytes-ChangeByte-ChangeBinInt-CrossOver-ChangeBinInt-ChangeBit-EraseBytes-CopyPart-ChangeByte-CrossOver-EraseBytes-CrossOver-ChangeByte-ShuffleBytes-ChangeByte-ChangeBinInt-CrossOver-ChangeByte-InsertRepeatedBytes-InsertByte-ShuffleBytes-PersAutoDict-ChangeBit-ChangeByte-ChangeBit-ShuffleBytes-ShuffleBytes-CopyPart-ShuffleBytes-EraseBytes-ShuffleBytes-ShuffleBytes-CrossOver-ChangeBinInt-CopyPart-CopyPart-CopyPart-EraseBytes-EraseBytes-ChangeByte-ChangeBinInt-ShuffleBytes-CMP-InsertByte-EraseBytes-ShuffleBytes-CopyPart-ChangeBit-CrossOver-CopyPart-CopyPart-ShuffleBytes-ChangeByte-ChangeByte-ChangeBinInt-EraseBytes-ChangeByte-ChangeBinInt-ChangeBit-ChangeBit-ChangeByte-ShuffleBytes-PersAutoDict-PersAutoDict-CMP-ChangeBit-ShuffleBytes-PersAutoDict-ChangeBinInt-EraseBytes-EraseBytes-ShuffleBytes-ChangeByte-ShuffleBytes-ChangeBit-EraseBytes-CMP-ShuffleBytes-ChangeByte-ChangeBinInt-EraseBytes-ChangeBinInt-ChangeByte-EraseBytes-ChangeByte-CrossOver-ShuffleBytes-EraseBytes-EraseBytes-ShuffleBytes-ChangeBit-EraseBytes-CopyPart-ShuffleBytes-ShuffleBytes-CrossOver-CopyPart-ChangeBinInt-ShuffleBytes-CrossOver-InsertByte-InsertByte-ChangeBinInt-ChangeBinInt-CopyPart-EraseBytes-ShuffleBytes-ChangeBit-ChangeBit-EraseBytes-ChangeByte-ChangeByte-ChangeBinInt-CrossOver-ChangeBinInt-ChangeBinInt-ShuffleBytes-ShuffleBytes-ChangeByte-ChangeByte-ChangeBinInt-ShuffleBytes-CrossOver-EraseBytes-CopyPart-CopyPart-CopyPart-ChangeBit-ShuffleBytes-ChangeByte-EraseBytes-ChangeByte-InsertRepeatedBytes-InsertByte-InsertRepeatedBytes-PersAutoDict-EraseBytes-ShuffleBytes-ChangeByte-ShuffleBytes-ChangeBinInt-ShuffleBytes-ChangeBinInt-ChangeBit-CrossOver-CrossOver-ShuffleBytes-CrossOver-CopyPart-CrossOver-CrossOver-CopyPart-ChangeByte-ChangeByte-CrossOver-ChangeBit-ChangeBinInt-EraseBytes-ShuffleBytes-EraseBytes-CMP-PersAutoDict-PersAutoDict-InsertByte-ChangeBit-ChangeByte-CopyPart-CrossOver-ChangeByte-ChangeBit-ChangeByte-CopyPart-ChangeBinInt-EraseBytes-CrossOver-ChangeBit-CrossOver-PersAutoDict-CrossOver-ChangeByte-CrossOver-ChangeByte-ChangeByte-CrossOver-ShuffleBytes-CopyPart-CopyPart-ShuffleBytes-ChangeByte-ChangeByte-ChangeBinInt-ChangeBinInt-ChangeBinInt-ChangeBinInt-ShuffleBytes-CrossOver-ChangeBinInt-ShuffleBytes-ChangeBit-PersAutoDict-ChangeBinInt-ShuffleBytes-ChangeBinInt-ChangeByte-CrossOver-ChangeBit-CopyPart-ChangeBit-ChangeBit-CopyPart-ChangeByte-PersAutoDict-ChangeBit-ShuffleBytes-ChangeByte-ChangeBit-CrossOver-ChangeByte-CrossOver-ChangeByte-CrossOver-ChangeBit-ChangeByte-ChangeBinInt-PersAutoDict-CopyPart-ChangeBinInt-ChangeBit-CrossOver-ChangeBit-PersAutoDict-ShuffleBytes-EraseBytes-CrossOver-ChangeByte-ChangeBinInt-ShuffleBytes-ChangeBinInt-InsertRepeatedBytes-PersAutoDict-CrossOver-ChangeByte-Custom-PersAutoDict-CopyPart-CopyPart-ChangeBinInt-ShuffleBytes-ChangeBinInt-ChangeBit-ShuffleBytes-CrossOver-CMP-ChangeByte-CopyPart-ShuffleBytes-CopyPart-CopyPart-CrossOver-CrossOver-CrossOver-ShuffleBytes-ChangeByte-ChangeBinInt-ChangeBit-ChangeBit-ChangeBit-ChangeByte-EraseBytes-ChangeByte-ChangeBit-ChangeByte-ChangeByte-CopyPart-PersAutoDict-ChangeBinInt-PersAutoDict-PersAutoDict-PersAutoDict-CopyPart-CopyPart-CrossOver-ChangeByte-ChangeBinInt-ShuffleBytes-ChangeBit-CopyPart-EraseBytes-CopyPart-CopyPart-CrossOver-ChangeByte-EraseBytes-ShuffleBytes-ChangeByte-CopyPart-EraseBytes-CopyPart-CrossOver-ChangeBinInt-ChangeBinInt-InsertByte-ChangeBinInt-ChangeBit-ChangeByte-CopyPart-ChangeByte-EraseBytes-ChangeByte-ChangeBit-ChangeByte-ShuffleBytes-CopyPart-ChangeBinInt-EraseBytes-CrossOver-ChangeBit-ChangeBit-CrossOver-EraseBytes-ChangeBinInt-CopyPart-CopyPart-ChangeBinInt-ChangeBit-EraseBytes-InsertRepeatedBytes-EraseBytes-ChangeBit-CrossOver-CrossOver-EraseBytes-EraseBytes-ChangeByte-CopyPart-CopyPart-ShuffleBytes-ChangeByte-ChangeBit-ChangeByte-EraseBytes-ChangeBit-ChangeByte-ChangeByte-CrossOver-CopyPart-EraseBytes-ChangeByte-EraseBytes-ChangeByte-ShuffleBytes-ShuffleBytes-ChangeByte-CopyPart-ChangeByte-ChangeByte-ChangeBit-CopyPart-ChangeBit-ChangeBinInt-CopyPart-ShuffleBytes-ChangeBit-ChangeBinInt-ChangeBit-EraseBytes-CMP-CrossOver-CopyPart-ChangeBinInt-CrossOver-CrossOver-CopyPart-CrossOver-CrossOver-InsertByte-InsertByte-CopyPart-Custom- DE: "warn"-"\x00\x00\x00\x80"-"\xfe\xff\xff\xfb"-"\xff\xff"-"\x10\x00\x00\x00"-"\xfe\xff\xff\xff"-"\xff\xff\xff\xf6"-"U\x01\x00\x00\x00\x00\x00\x00"-"\xd9\xff\xff\xff"-"\xfe\xff\xff\xea"-"\xf0\xff\xff\xff"-"\xfc\xff\xff\xff"-"warn"-"\xff\xff\xff\xff"-"\xfe\xff\xff\xfb"-"\x00\x00\x00\x80"-"\xfe\xff\xff\xf1"-"\xfe\xff\xff\xea"-"\x00\x00\x00\x00\x00\x00\x012"-"\xe2\x00"-"\xfb\xff\xff\xff"-"\x00\x00\x00\x00"-"\xe9\xff\xff\xff"-"\xff\xff"-"\x00\x00\x00\x80"-"\x01\x00\x04\xc9"-"\xf0\xff\xff\xff"-"\xf9\xff\xff\xff"-"\xff\xff\xff\xff\xff\xff\xff\x12"-"\xe2\x00"-"\xfe\xff\xff\xff"-"\xfe\xff\xff\xea"-"\xff\xff\xff\xff"-"\xf4\xff\xff\xff"-"\xe9\xff\xff\xff"-"\xf1\xff\xff\xff"-
#48    NEW    cov: 4502 ft: 9151 corp: 27/750Kb lim: 64000 exec/s: 2 rss: 458Mb L: 50772/50772 MS: 259 ChangeByte-ShuffleBytes-ChangeBinInt-ChangeByte-ChangeByte-ChangeByte-ChangeByte-ChangeBit-CopyPart-CrossOver-CopyPart-ChangeByte-CrossOver-CopyPart-ChangeBit-ChangeByte-EraseBytes-ChangeByte-CopyPart-CopyPart-CopyPart-ChangeBit-EraseBytes-ChangeBinInt-CrossOver-CopyPart-CrossOver-CopyPart-ChangeBit-ChangeByte-ChangeBit-InsertByte-CrossOver-InsertRepeatedBytes-InsertRepeatedBytes-InsertRepeatedBytes-ChangeBinInt-EraseBytes-InsertRepeatedBytes-InsertByte-ChangeBit-ShuffleBytes-ChangeBit-ChangeBit-CopyPart-ChangeBit-ChangeByte-CrossOver-ChangeBinInt-ChangeByte-CrossOver-CMP-ChangeByte-CrossOver-ChangeByte-ShuffleBytes-ShuffleBytes-ChangeByte-ChangeBinInt-CopyPart-EraseBytes-CrossOver-ChangeBit-ChangeBinInt-InsertByte-ChangeBit-CopyPart-ChangeBinInt-ChangeByte-CrossOver-ChangeBit-EraseBytes-CopyPart-ChangeBinInt-ChangeBit-ChangeBit-ChangeByte-CopyPart-ChangeBinInt-CrossOver-PersAutoDict-ChangeByte-ChangeBit-ChangeByte-ChangeBinInt-ChangeBinInt-EraseBytes-CopyPart-CopyPart-ChangeByte-ChangeByte-EraseBytes-PersAutoDict-CopyPart-ChangeByte-ChangeByte-EraseBytes-CrossOver-CopyPart-CopyPart-CopyPart-ChangeByte-ChangeBit-CMP-CopyPart-ChangeBinInt-ChangeBinInt-CrossOver-ChangeBit-ChangeBit-EraseBytes-ChangeByte-ShuffleBytes-ChangeBit-ChangeBinInt-CMP-InsertRepeatedBytes-CopyPart-Custom-ChangeByte-CrossOver-EraseBytes-ChangeBit-CopyPart-CrossOver-CMP-ShuffleBytes-EraseBytes-CrossOver-PersAutoDict-ChangeByte-CrossOver-CopyPart-CrossOver-CrossOver-ShuffleBytes-ChangeBinInt-CrossOver-ChangeBinInt-ShuffleBytes-PersAutoDict-ChangeByte-EraseBytes-ChangeBit-CrossOver-EraseBytes-CrossOver-ChangeBit-ChangeBinInt-EraseBytes-InsertByte-InsertRepeatedBytes-InsertByte-InsertByte-ChangeByte-ChangeBinInt-ChangeBit-CrossOver-ChangeByte-CrossOver-EraseBytes-ChangeByte-ShuffleBytes-ChangeBit-ChangeBit-ShuffleBytes-CopyPart-ChangeByte-PersAutoDict-ChangeBit-ChangeByte-InsertRepeatedBytes-CMP-CrossOver-ChangeByte-EraseBytes-ShuffleBytes-CrossOver-ShuffleBytes-ChangeBinInt-ChangeBinInt-CopyPart-PersAutoDict-ShuffleBytes-ChangeBit-CopyPart-ShuffleBytes-CopyPart-EraseBytes-ChangeByte-ChangeBit-ChangeBit-ChangeBinInt-ChangeByte-CopyPart-EraseBytes-ChangeBinInt-EraseBytes-EraseBytes-PersAutoDict-CMP-PersAutoDict-CrossOver-CrossOver-ChangeBit-CrossOver-PersAutoDict-CrossOver-CopyPart-ChangeByte-EraseBytes-ChangeByte-ShuffleBytes-ChangeByte-ChangeByte-CrossOver-ChangeBit-EraseBytes-ChangeByte-EraseBytes-ChangeBinInt-CrossOver-CrossOver-EraseBytes-ChangeBinInt-CrossOver-ChangeBit-ShuffleBytes-ChangeBit-ChangeByte-EraseBytes-ChangeBit-CrossOver-CrossOver-CrossOver-ChangeByte-ChangeBit-ShuffleBytes-ChangeBit-ChangeBit-EraseBytes-CrossOver-CrossOver-CopyPart-ShuffleBytes-ChangeByte-ChangeByte-CopyPart-CrossOver-CopyPart-CrossOver-CrossOver-EraseBytes-EraseBytes-ShuffleBytes-InsertRepeatedBytes-ChangeBit-CopyPart-Custom- DE: "\xfe\xff\xff\xfc"-"\x00\x00\x00\x00"-"F\x00"-"\xf3\xff\xff\xff"-"St9exception"-"_\x00\x00\x00"-"\xf6\xff\xff\xff"-"\xfe\xff\xff\xff"-"\x00\x00\x00\x00"-"p\x02\x00\x00\x00\x00\x00\x00"-"\xfe\xff\xff\xfb"-"\xff\xff"-"\xff\xff\xff\xff"-"\x01\x00\x00\x07"-"\xfe\xff\xff\xfe"-

These are prohibitively large and of limited value in the default case (when
someone is running the fuzzer, not debugging it), in my opinion.

Diff Detail

Event Timeline

mhlakhani created this revision.Aug 26 2020, 1:29 PM
Herald added a project: Restricted Project. · View Herald TranscriptAug 26 2020, 1:29 PM
Herald added a subscriber: Restricted Project. · View Herald Transcript
mhlakhani requested review of this revision.Aug 26 2020, 1:29 PM
mhlakhani edited the summary of this revision. (Show Details)Aug 26 2020, 1:29 PM

I am happy to move discussion on whether this patch is useful to the community to the mailing list, given the size of the patch I figured I would put it up directly for review over here.

Note that I don't have commit access to the repo, so I would need some help in getting this committed if the patch is acceptable.

plotfi added a subscriber: plotfi.Aug 26 2020, 1:53 PM
plotfi added inline comments.
compiler-rt/lib/fuzzer/FuzzerLoop.cpp
40

Why 10?

604

const int maxLength = Options.Verbosity >= 2 ? -1 : kMaxMutationsToPrint;

608

Why not give PrintMutationSequence a default parameter?

compiler-rt/lib/fuzzer/FuzzerMutate.cpp
484

Any reason its int32_t and not just an int or a size_t? Seems like it should be a size_t.

488

How about

size_t entriesToPrint = maxLength ? std::min(maxLength, CurrentDictionaryEntrySequence.size()) : CurrentDictionaryEntrySequence.size();

494

How about

entriesToPrint = maxLength ? std::min(maxLength, CurrentDictionaryEntrySequence.size()) : CurrentDictionaryEntrySequence.size();
498

Why not keep the modern syntax and instead do:

for (auto DE : CurrentDictionaryEntrySequence) {
  if (!(--entriesToPrint))
    break;
  Printf("\"");
  PrintASCII(DE->GetW(), "\"-");
}
mhlakhani added inline comments.Aug 26 2020, 2:01 PM
compiler-rt/lib/fuzzer/FuzzerLoop.cpp
40

I'm looking for feedback on what a good value is. The resulting lines didn't seem too long with a value of 10, but this is a guess and not a perfect value.

608

This is done below, unless I am missing something?

compiler-rt/lib/fuzzer/FuzzerMutate.cpp
484

The only reason this is int32_t here is so we could use a default of -1 to indicate unlimited.

An alternative way to design this is to potentially move kMaxMutationsToPrint to be inside this file, and instead take a boolean argument that indicates whether we apply this limit or not. I'm happy to try out that version if preferable.

Seems like a reasonable patch to me. Please address @plotfi's comments, then I'll take a closer look.

mhlakhani updated this revision to Diff 288483.Aug 27 2020, 4:39 PM

address review comments

most of the clang-tidy checks are pointing out issues in the existing code, or places where I am matching the style within libfuzzer's code. I hope it's OK to leave those unfixed.

A test would be nice, but maybe tricky.

Maybe we could contrive a test like this:

  • custom mutator that does nothing until the Xth run (X > 10), at which point it returns a hard-coded input that finds the bug
  • run with -mutate_depth=100
  • use grep to count the number of mutations output by the fuzzer
compiler-rt/lib/fuzzer/FuzzerMutate.cpp
486–491

Nit: Please capitalize local vars for consistency with existing style

498

I prefer the original syntax (straightforward index incrementing loop) since it is slightly more compact and requires slightly less thought to understand what its doing.

mhlakhani updated this revision to Diff 289033.Aug 31 2020, 3:18 PM

Address review comments:

  • Revert back to index based for loop
  • Add a test for the behaviour

I couldn't figure out a way to grep exactly for the number of mutations but I think this is close enough.
The test sets up a mutator which calls LLVMFuzzerMutate a large number of times, which replicates
the behavior in my actual mutator.

I then added two test cases:

One that runs this with -verbosity=2 and verifies we see very long mutation sequences -
this identifies a mutation sequence by the presence of NEW: and checks via grep that a line
that is longer than 1kb exists. (i.e. producing the old behaviour, and verifying that verbose
data is still available)

The fix is verified by repeating the above test but without the -verbosity option, and verifying
that long lines are no longer seen (all lines are < 1kb).

I inspected the log files on my local machine as well and verified they look good

morehouse accepted this revision.Sep 1 2020, 8:09 AM
morehouse added inline comments.
compiler-rt/test/fuzzer/fuzzer-custommutator.test
22 ↗(On Diff #289033)

No need to rebuild the test here.

This revision is now accepted and ready to land.Sep 1 2020, 8:09 AM
mhlakhani updated this revision to Diff 289203.Sep 1 2020, 9:36 AM

don't rebuild test binary twice

mhlakhani marked 10 inline comments as done.Sep 1 2020, 9:38 AM

thanks for all the feedback, @morehouse and @plotfi! With the latest change I believe all the comments are addressed.

I don't have commit privileges, so I would appreciate some help from either of you to commit this patch.

This revision was landed with ongoing or failed builds.Sep 1 2020, 11:14 AM
This revision was automatically updated to reflect the committed changes.
This revision is now accepted and ready to land.Sep 1 2020, 12:51 PM

apologies for the thrash here, the build bot failure is indeed confusing.

This is unfortunately one of those "works on my machine" cases -- looking at the logs it's not immediately obvious to me what the failure is. @morehouse , do you have suggestions on how I could go about debugging it?

The log from the darwin bot has:

+ : 'RUN: at line 18'
+ grep NEW /Users/buildslave/jenkins/workspace/clang-stage1-RA/clang-build/tools/clang/runtime/compiler-rt-bins/test/fuzzer/X86_64DefaultDarwinConfig/Output/fuzzer-custommutator.test.tmp-mutate-verbose-log
+ grep '.\{1024\}'
grep: invalid repetition count(s)

So at least one issue seems to be from different grep feature support on different machines.

Thanks! Will find a more portable solution for that. I'll also continue investigating the linux failure (http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/38581/steps/check-fuzzer/logs/stdio) which seems to be a different issue. Is there a way to get access to the contents of /b/sanitizer-x86_64-linux-fuzzer/build/llvm_build0/projects/compiler-rt/test/fuzzer/X86_64DefaultLinuxConfig/Output/fuzzer-custommutator.test.tmp-mutate-log ?

I do have access to that bot, but the log is gone already. Sorry.

I think I've managed to reproduce that failure locally, working on a fix. Is there a way to kick off the failing tests before commit, so we can verify that they are all fixed before attempt to land this again?

The libFuzzer tests automatically run on harbormaster when you update the patch, but that's often a different environment than the bots.

Unfortunately we have to push a commit before the bots will will catch any issues on their configs.

mhlakhani updated this revision to Diff 289311.Sep 1 2020, 4:23 PM

Attempt to fix test failures.
Rewrite the test to be more portable by using awk for the line length check
and updating the grep statement as a result.

Tested on a linux machine using ninja check-fuzzer and on a mac
by copying over the log files and rerunning the awk+grep commands and
verifying exit codes

These test failures are legitimate, will see if I can somehow get them to reproduce on my local environment (not sure how just yet ...)

@morehouse I spent some more time digging and it's hard to get a portable test that works across machines, because I don't know how to express the condition I'm trying to check for within the test framework. Do you have any suggestions here?

Alternatively, how bad is it if we land this without a test?

I'd like to keep the test, but we can separate it into its own file and use UNSUPPORTED or REQUIRES to avoid running on systems that don't have the tools we need.

Another idea: if we can express the length check using FileCheck regexes, we could use that instead of grep or awk and get portability automatically.

mhlakhani updated this revision to Diff 291769.Sep 14 2020, 9:03 PM

Rewrite test

I'm hopeful this should work. I wasn't able to find a portable way to assert on the full line length, but I found a seed where the verbose value is not prohibitively long, and the trimmed value is indeed a subset. It was possible to assert against the whole sequence this way.

This test should be stable across platforms assuming the RNG is portable (I hope...)

mhlakhani updated this revision to Diff 291770.Sep 14 2020, 9:09 PM

Make test a bit more robust by also verifying the start of the range

morehouse added inline comments.Sep 15 2020, 9:48 AM
compiler-rt/test/fuzzer/fuzzer-custommutator.test
19 ↗(On Diff #291770)

Unfortunately, RNG is not identical across platforms.

Rather than matching specific mutations, can you use a regex to match any 1024 characters on the same line?

mhlakhani updated this revision to Diff 292082.Sep 15 2020, 8:56 PM

Rewrite test to be independent of the seed

I think I've finally got the syntax right after fighting with the escaping and the FileCheck docs.
This asserts that when verbosity=2, we get a line with >= 11 mutation sequences (letters separated by a dash), and with lower verbosity no such line is found

mhlakhani updated this revision to Diff 292085.Sep 15 2020, 9:09 PM

fix comment

morehouse accepted this revision.Sep 16 2020, 9:19 AM

Test looks good, thanks for figuring out the regex stuff.

This revision was landed with ongoing or failed builds.Sep 16 2020, 9:24 AM
This revision was automatically updated to reflect the committed changes.

That's one hell of a commit message :-)