This is an archive of the discontinued LLVM Phabricator instance.

[libFuzzer] Scale energy assigned to each input based on input execution time.
ClosedPublic

Authored by dokyungs on Aug 17 2020, 10:27 AM.

Details

Summary

This patch scales the energy computed by the Entropic schedule based on the
execution time of each input. The input execution time is compared with the
average execution time of inputs in the corpus, and, based on the amount by
which they differ, the energy is scaled from 0.1x (for inputs executing slow) to
3x (for inputs executing fast). Note that the exact formula is borrowed from
AFL.

On FuzzBench, this gives a sizeable throughput increase, which in turn leads to
more coverage on several benchmarks. For details, see the following report.

https://storage.googleapis.com/fuzzer-test-suite-public/exectime-report/index.html

Diff Detail

Event Timeline

dokyungs created this revision.Aug 17 2020, 10:27 AM
Herald added a project: Restricted Project. · View Herald TranscriptAug 17 2020, 10:27 AM
Herald added a subscriber: Restricted Project. · View Herald Transcript
dokyungs requested review of this revision.Aug 17 2020, 10:27 AM

LGTM for an experiment!

hctim added inline comments.Aug 17 2020, 3:25 PM
compiler-rt/lib/fuzzer/FuzzerFlags.def
164

(prior to real submission - please describe this feature)

dokyungs updated this revision to Diff 289207.Sep 1 2020, 10:07 AM

Fixed compiler warnings by using signed int type for the average time variable (AverageTimeOfUnit); also fixed the unit tests.

Please add a test.

compiler-rt/lib/fuzzer/FuzzerFlags.def
164

Yes, please document this feature.

compiler-rt/lib/fuzzer/tests/FuzzerUnittest.cpp
599–600

Please fix this clang-tidy.

dokyungs updated this revision to Diff 289234.Sep 1 2020, 11:30 AM

Addressed comments.

dokyungs edited the summary of this revision. (Show Details)Sep 1 2020, 11:30 AM
dokyungs marked 3 inline comments as done.

Please add a test.

I will do this in the next upload.

dokyungs updated this revision to Diff 289302.Sep 1 2020, 3:57 PM

Add a test.

dokyungs added inline comments.Sep 1 2020, 4:02 PM
compiler-rt/test/fuzzer/entropic-scale-per-exec-time.test
3

This test is not deterministic, but mostly falls in the same bucket of scaling; so it mostly takes around 44,557 (and around 1.3-1.5 sec in my machine) to find the crash.

4

This test is deterministic, it takes 126,633 executions (and around 5.4-5.6 sec) to find the crash if I give more runs.

hctim added inline comments.Sep 1 2020, 6:26 PM
compiler-rt/lib/fuzzer/FuzzerCorpus.h
69

Nit: please update comment to describe the strategy behind ScalePerExecTime

71

std::chrono::microseconds AverageUnitExecutionTime

98–111

I suspect that these multiplications are going to be really expensive.

Can you instead use a shift-based approach here?

Like:

if (TimeOfUnit.count() < (AverageTimeOfUnit >> 1)) // 2x faster.
  PerfScore = 150;
else if (TimeOfUnit.count() < (AverageTimeOfUnit >> 2)) // 4x faster.
  PerfScore = 200;

etc...

490–493

rewrite as:

std::chrono::microseconds AverageUnitExecutionTime(0);
for (auto II : Inputs) {
  AverageUnitExecutionTime += II->TimeOfUnit;
}
AverageUnitExecutionTime /= N;

Also I was going to say something about overflow check, but 1 << 55 microseconds (minimum precision defined by the spec) is 1141 years, so I'm not particularly concerned here...

501

pass AverageUnitExecutionTime

compiler-rt/test/fuzzer/entropic-scale-per-exec-time.test
4

Thanks for the heads up. Instead of running this test (I'd prefer not to introduce tests with sleeps(), apart from the one were we're deliberately trying to avoid the sleeps() by scheduling those inputs less), could you just add the results as a comment?

dokyungs updated this revision to Diff 289477.Sep 2 2020, 8:55 AM

Addressed comments.

dokyungs updated this revision to Diff 289480.Sep 2 2020, 8:57 AM
dokyungs marked 2 inline comments as done.

Variable name change, as suggested.

dokyungs marked 3 inline comments as done.Sep 2 2020, 8:58 AM
dokyungs added inline comments.
compiler-rt/lib/fuzzer/FuzzerCorpus.h
98–111

I changed multiplication to shift where I can. I am not sure how to handle non-power-of-two cases. Do you have any recommendation for those?

dokyungs marked an inline comment as not done.Sep 2 2020, 8:58 AM
dokyungs edited the summary of this revision. (Show Details)
Harbormaster completed remote builds in B70410: Diff 289480.
morehouse added inline comments.Sep 2 2020, 1:12 PM
compiler-rt/lib/fuzzer/FuzzerCorpus.h
98–111

These shifts make the code harder to understand.

I might prefer we keep the multiplications for readability, but we can probably avoid the slow floating point operations with some algebra:

TimeOfUnit.count() * 0.1 > AverageUnitExecutionTime.count()

becomes

TimeOfUnit.count() > 10 * AverageUnitExecutionTime.count()

For powers of 2, the compiler will optimize to shifts under the hood. And I'd also expect it to optimize things like 10 * x into x << 3 + x << 1 if that's faster than the multiplication.

hctim added inline comments.Sep 2 2020, 3:02 PM
compiler-rt/lib/fuzzer/FuzzerCorpus.h
98–111

https://quick-bench.com/q/z8GU7H0hFkmai-AEFvSWQiOEzno

Yep - let's just remove the floating points here - as long as the multiplicand can be shift-optimised as you mention we should be good.

dokyungs updated this revision to Diff 289601.Sep 2 2020, 4:13 PM

Remove floting point by turning them into integer multiplication.

dokyungs marked 3 inline comments as done.Sep 2 2020, 4:14 PM
dokyungs added inline comments.
compiler-rt/lib/fuzzer/FuzzerCorpus.h
98–111

Thanks for benchmarking! I turned them into integer multiplication.

dokyungs edited the summary of this revision. (Show Details)Sep 2 2020, 4:16 PM
morehouse added inline comments.Sep 2 2020, 5:25 PM
compiler-rt/lib/fuzzer/FuzzerCorpus.h
102

I think this line should have * 2

dokyungs updated this revision to Diff 289609.Sep 2 2020, 5:51 PM
dokyungs marked an inline comment as done.
dokyungs edited the summary of this revision. (Show Details)

Addressed comment.

dokyungs marked an inline comment as done.Sep 2 2020, 5:52 PM
This revision is now accepted and ready to land.Sep 2 2020, 5:54 PM
This revision was landed with ongoing or failed builds.Sep 3 2020, 1:47 PM
This revision was automatically updated to reflect the committed changes.