Redistribute energy for Corpus

[gtt1995]

Summary

I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method can give more energy to the small seeds in the corpus and
increase the throughput of the test.

Fuzzbench data (glibfuzzer is -fork_corpus_groups=1):
https://www.fuzzbench.com/reports/experimental/2021-08-05-parallel/index.html

[gtt1995]

I used this cmd : clang-format -i xxx.cpp -style=LLVM.

[gtt1995]

clang-format -i xxx.cpp -style=LLVM

Many irrelevant files have been modified.
I don't know what to do to pass the inspection. Who will tell me what to do?

[gtt1995]

This is a lastet CHANGE!

Don't care about all the diffs submitted before.

Please refer to the latest diff！

[gtt1995]

Add the corpus merge stage.

[morehouse]

Thanks for the patch. I see you have https://github.com/google/clusterfuzz/issues/2373 about testing it on FuzzBench. If the FuzzBench results are better than the current libFuzzer configuration, I'll do a proper review of this patch.

[gtt1995]

I already applied for PR at https://github.com/google/fuzzbench/pull/1197
But no response ....

[gtt1995]

Hello,Master morehouse , you finally replied to me!
I don’t know if the experiment has been deployed. No one responded to my latest pr? Can you help me confirm it on Fuzzbench.
https://github.com/google/fuzzbench/pull/1197

[gtt1995]

Fix an error in CreateOneJob() ,it happens when use a empty corpus to start fuzzing.

[gtt1995]

In D105084#2866163, @gtt1995 wrote:

Hello,Master morehouse , you finally replied to me!
I don’t know if the experiment has been deployed. No one responded to my latest pr? Can you help me confirm it on Fuzzbench.
https://github.com/google/fuzzbench/pull/1197

@metzman has replied to me, he thinks this experiment will take two weeks to deploy.Then I give you the experiment data.

[gtt1995]

@morehouse
Hello. Parallel fuzzing experiments have not been deployed so far. PR believes that some special customizations are needed for parallel fuzzing, because the previous experiments were deployed on a single-core single-core virtual machine. So I want you to help me ask your friend about the progress of the experiment.
According to the plan, it should be completed last week, I left a message to @metzman , and he has not yet responded to me.
Thanks
https://github.com/google/fuzzbench/pull/1197#issuecomment-880775133
https://github.com/google/fuzzbench/pull/1197#issuecomment-880795471

[gtt1995]

In D105084#2864647, @morehouse wrote:

Thanks for the patch. I see you have https://github.com/google/clusterfuzz/issues/2373 about testing it on FuzzBench. If the FuzzBench results are better than the current libFuzzer configuration, I'll do a proper review of this patch.

Hello,Fuzzbench has not replied to me so far, can you ask me for it?
Thanks.

[gtt1995]

Fuzzbench experiment data
Hello, @morehouse the results of the parallel experiment deployed some time ago have come out. Here I explain the results to you. Among them, glibfuzzer performs better in large programs, especially in fuzzers that generate a large number of seeds (generating more than 10,000 seeds) ), its scheduling strategy can always choose better seeds. In the end, the average size of each seed is 50% of the baseline. On the other hand, the glibfuzzer test results are very stable with very small variance.
Since the large targets such as ffmpeg, php-executed... are not included in the benchmark, I deployed these large programs on my own machine.name ,glibfuzzer is more effective on large programs.
Finally, do you have time to help me review these patches?

[morehouse]

Thanks for running the experiment; the FuzzBench results look quite good overall!

Any idea what happened on the lcms and vorbis benchmarks? Why does glibfuzzer have 0 coverage there?

This patch will need some work to get it in a submittable state, but we can start reviewing today and continue next week.

[morehouse]

Some high level comments:

• Please run clang-format to fix all the lint errors. This will make it easier to review.

• I'd like to guard the new functionality with a flag, so that we can further experiment with this mode without losing the current defaults. If we see good results in production over time, we can turn the flag on by default.

[gtt1995]

Hello, hello, hello
@morehouse @morehouse @morehouse

[gtt1995]

In D105084#2944034, @morehouse wrote:

Thanks for running the experiment; the FuzzBench results look quite good overall!

Any idea what happened on the lcms and vorbis benchmarks? Why does glibfuzzer have 0 coverage there?

This patch will need some work to get it in a submittable state, but we can start reviewing today and continue next week.

Their problem is that the fuzzer uses 1 seeds, which causes grouping errors. This is a simple problem and I can fix it.

[gtt1995]

In D105084#2944071, @morehouse wrote:

Some high level comments:

• Please run clang-format to fix all the lint errors. This will make it easier to review.

• I'd like to guard the new functionality with a flag, so that we can further experiment with this mode without losing the current defaults. If we see good results in production over time, we can turn the flag on by default.

I think you are right, I can add this function.

[gtt1995]

Hello,

1. We can turn on the new function by setting group-flags.
2. I have fixed the error( one seed).
3. It's like my clang-format doesn't work anymore.

[gtt1995]

compiler-rt/lib/fuzzer/FuzzerFork.cpp

138

group flags

[morehouse]

Please take a look at the variable naming style for LLVM and follow it in this patch. I noticed some variable names that started with a lower-case letter.

Also, please try to figure out clang-format for this patch. I won't submit it with inconsistent indentation, etc.

Finally, I noticed the patch doesn't apply cleanly, so the pre-merge tests aren't running properly. Could you rebase onto HEAD for the next diff?

compiler-rt/lib/fuzzer/FuzzerFlags.def

59

Please use the same style convention as other flags: i.e. num_corpuses

60

Can we get the same behavior with a single flag? For example, if num_corpuses is 0, then we can avoid the new strategy.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

141

Why the + 1?

141

AverageSize makes me think of file sizes, but really it's referencing the average number of files in each seed corpus. AverageCorpusSize might be less confusing.

148

I understand this might be a net-positive change, but is there a situation where we would expect it to perform worse?

For example, what if generating certain coverage requires us to do a CrossOver mutation between two inputs in different groups. IIUC, this new method may make it impossible for those two inputs to end up in the same seed corpus.

166–169

This chunk of code is the same as before. Let's share it between the two cases.

220

NumCorpuses appears currently unused in this function.

265

Let's guard this sorted insertion so we only do it in the new mode, and keep doing push_back in the old mode.

271

Please remove this commented-out code.

346

s/JobExected/JobExecuted

347

Please move these definitions closer to where they are used.

381

This looks like it will break in KeepSeed mode, since the above code already pushes the file sizes.

431

Do we actually need to duplicate the MainCorpusDir logic here? IIUC, we only need to do this once at startup.

447

Please remove this commented-out code.

450

What is the motivation for linearly increasing MergeCycle? Does it make a noticeable difference in fuzzer performance?

453

Does this printf cause spam in libFuzzer's logs?

Either way, we should use libFuzzer's Printf instead.

[gtt1995]

compiler-rt/lib/fuzzer/FuzzerFlags.def

59

OK！

60

Of course yes.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

141

Because the number of seeds is sometimes not divisible by numcorpues, the remainder will be discarded, and the largest few seeds in the corpus will not be selected.

141

Yes，AverageCorpusSize is better.

148

I don't know if I understand it correctly. This new method only restricts different sub-instances to select seeds in different smaller ranges. Similar to the old method, the old method is that each instance selects seeds within the newly inserted seed range. As you said, if the sizes of the two inputs that need to be corossover mutations differ greatly, they are indeed likely to not appear in the same corpus, but the old method can definitely be satisfied

166–169

OK！

220

OK, i will remove it.

265

It's Ok.

271

ok.

346

hhh

347

It's ok.

381

This is to initialize File Sizes, record and store the size of the seed provided in the initial corpus in FilesSizes, which can be controlled by NumCorpuses later.

431

I think merge is necessary, libfuzzer will generate more streamlined test cases during the testing process, and we choose seeds from the entire corpus. We need to use the Merge function to update Files. In actual tests, it will indeed effectively reduce the number of seeds in the main corpus.

447

It's ok.

450

Because the growth rate of corpus seeds slows down over time, the number of corpus seeds in the early stage grows rapidly, and frequent merges are required. In the later stage of the test, the merge cycle is increased, which can reduce the overhead of merge. For some programs, the merge operation is more expensive time.
But linear growth may not be the best choice, what is your suggestion？

453

You are right, i will remove it later.

[gtt1995]

I used git diff -U0 HEAD^ ./
This is a no-clang-format diff.
Can you help me adjust the format using clang-format? I always have problems here.

[gtt1995]

This is the diff after clang-format --style=LLVM. Maybe it modified something irrelevant, but I don’t know how to undo the operation, and the git apply --check operation always fails.

To make it easier to review the new code, you can refer to Diff 366867

[morehouse]

Yeah, I don't want the whole file clang-formatted, just the changes you made. Also the "Context not available" makes it tricky to review.

I'm not sure how you're uploading patches, maybe you could try using Arcanist? My local workflow is like this:

$ git checkout main
$ git pull -r origin main
$ git checkout -b fuzzer-fork-group-mode
... (apply patch) ...
$ git add -A
$ git clang-format
$ git add -A
$ git commit
$ arc diff main --update D105084

When I do it this way, only the changed lines are clang-formatted, and the uploaded diff has full context.

[gtt1995]

In D105084#2949582, @morehouse wrote:

Yeah, I don't want the whole file clang-formatted, just the changes you made. Also the "Context not available" makes it tricky to review.

I'm not sure how you're uploading patches, maybe you could try using Arcanist? My local workflow is like this:

$ git checkout main
$ git pull -r origin main
$ git checkout -b fuzzer-fork-group-mode
... (apply patch) ...
$ git add -A
$ git clang-format
$ git add -A
$ git commit
$ arc diff main --update D105084

When I do it this way, only the changed lines are clang-formatted, and the uploaded diff has full context.

Ok, thanks , i will try it !

[gtt1995]

OK!!!
It seems that some irrelevant things have also been added, you can ignore it

[morehouse]

Please remove 11.diff, libFuzzer.a, and lib/fuzzer/test from the diff.

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

Typo.

61

Let's name it something more descriptive. Maybe fork_corpus_groups?

compiler-rt/lib/fuzzer/FuzzerFork.cpp

145

I think technically we don't need to check this case. If Files.size() == 0, then CorpusSubsetSize will also be 0, and we won't execute this code at all.

148

Right, my concern is that this change essentially partitions the corpus, so that different partitions rarely (if ever) cross-pollinate.

It still may be a useful feature to have and do further experimentation on, but for now I want it off by default.

148–151

Can we use (*Rand)(AverageCorpusSize) instead of creating another RNG?

152–161

This can be simplified.

240–244

Let's be consistent with surrounding style, and we can use size_t instead of long.

381

Right, but it looks like the new code pushes the sizes twice in KeepSeed mode.

compiler-rt/lib/fuzzer/FuzzerFork.h

22 ↗

(On Diff #367082)

Instead of passing Group as a parameter everywhere, could we add it to FuzzerOptions and just use Options.Group?

[morehouse]

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

Also this flag is currently used as a boolean, the actual value is ignored. Can we either fix this or change the description here.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

244

Also, since we regenerate Files and FilesSizes on each merge, do we still need to insert in sorted order here? What benefit to do we get from it?

373

Should NumCorpuses be part of Env, so we don't need to pass it around?

396–397

Can simplify.

402–405

I think we can at least remove these lines, since I think they only need to be executed once.

427–442

Can simplify.

428

Since we dynamically adjust NumCorpuses, should we even allow the user to set the initial number of corpuses?

[gtt1995]

compiler-rt/lib/fuzzer/FuzzerFork.cpp

145

it seems to be like this.

148–151

You mean to use Rand->SkewTowardsLast()？ or else?

244

Merge operation is very expensive, so choose to merge periodically. Before each merge, Files are out of order, so they need to be inserted according to size all the time.

373

It is not meaningful to pass the initial NumCorpuses value, because it will automatically adjust with the number of seeds in the corpus.

428

The presence or absence of the initial value has little effect on performance, and the number of seeds will change rapidly.

compiler-rt/lib/fuzzer/FuzzerFork.h

22 ↗

(On Diff #367082)

OK.

[gtt1995]

• 0820
• delete something

1. Updating D105084: Redistribute energy for Corpus #
2. Enter a brief description of the changes included in this update.
3. The first line is used as subject, next lines as comment. #
4. If you intended to create a new revision, use:
5. $ arc diff --create

Some modifications.

[gtt1995]

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

[gtt1995]

I have corrected most of the problems.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

148

All right.

166–169

Sorry, Where is the same?

381

OK, i deleted that line.

447

hhh, this comment is not added by me.

[gtt1995]

• some fix

[gtt1995]

• simplify

[morehouse]

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

Please change the default value to 0.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

148–151

No, I mean using Rand to choose a random value between 0 and AverageCorpusSize, instead of creating a new std::uniform_int_distribution.

340–342

To match surrounding style.

353–355

410–412

421–436

We need the elses for it to work right. Otherwise, we'll always get NumCorpuses = 60 or NumCorpuses = 80.

[gtt1995]

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

It's ok.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

148–151

All right.

340–342

It's ok.

421–436

Maye not. When fuzzing different programs, the usually generated the number of interesting seeds are different, hundreds, thousands, tens of thousands of exist. Therefore, when fixed set to numcorpuses=80, such as 1600 seeds, subset to pick sqrt (1600) = 40 at a time, but only 20 per group, and eventually repeat each time, and then more groups 80, it needs more job to test a round of the corpus.

Maybe i can set an interval of coarse graininess.

[gtt1995]

• ok

[gtt1995]

• a little fix

[gtt1995]

• change the NumCorpuses

[gtt1995]

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

[morehouse]

In D105084#2969098, @gtt1995 wrote:

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

I'm not sure whether it's a problem. I think sometimes cov and ft can be the same. But in general ft >= cov. And as long as the default behavior is unchanged I'm not too worried.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

421–436

Ok, but we still need else if instead of if or it won't work.

[morehouse]

I'll be gone until late next week. Then I'll apply it locally and experiment. If all looks good, we can land it after fixing my last comment

[gtt1995]

In D105084#2969374, @morehouse wrote:

I'll be gone until late next week. Then I'll apply it locally and experiment. If all looks good, we can land it after fixing my last comment

It's OK ! Thanks a lot !

[gtt1995]

• if -> else if

[gtt1995]

In D105084#2969369, @morehouse wrote:

In D105084#2969098, @gtt1995 wrote:

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

I'm not sure whether it's a problem. I think sometimes cov and ft can be the same. But in general ft >= cov. And as long as the default behavior is unchanged I'm not too worried.

OK, look forward to your local experiment result, it may have the problem of inaccurate COV , the COV on the panel will be quite different from the result after -merge (dry run) .

[gtt1995]

try to fix Unit test

[morehouse]

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

[gtt1995]

In D105084#2982975, @morehouse wrote:

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

Ok!

[gtt1995]

In D105084#2982975, @morehouse wrote:

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

Sorry, Could you tell me how to do this? I have no idea.

[gtt1995]

In D105084#2982975, @morehouse wrote:

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

I applied the latest changes locally, and -fork_corpus_groups=1 is actually more efficient.

[gtt1995]

• add a fork_corpus_groups.test

[gtt1995]

In D105084#2983993, @gtt1995 wrote:

• add a fork_corpus_groups.test

Hello, Does this work?

[morehouse]

LGTM

[gtt1995]

In D105084#2987018, @morehouse wrote:

LGTM

Thank you so much for your work! thanks matt！

[gtt1995]

In D105084#2987018, @morehouse wrote:

LGTM

In D105084#2969369, @morehouse wrote:

In D105084#2969098, @gtt1995 wrote:

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

I'm not sure whether it's a problem. I think sometimes cov and ft can be the same. But in general ft >= cov. And as long as the default behavior is unchanged I'm not too worried.

Hi matt ,the aforementioned problem does not seem to have been solved.

[gtt1995]

The cov on the panel and the cov result after -merge=1 are very different, and ft is accurate.

[gtt1995]

#19820674: cov: 869 ft: 2528 corp: 687 exec/s 46456 oom/timeout/crash: 3/0/0 time: 219s job: 35 dft_time: 0
#20722822: cov: 870 ft: 2543 corp: 698 exec/s 24382 oom/timeout/crash: 3/0/0 time: 225s job: 36 dft_time: 0
#20891453: cov: 870 ft: 2549 corp: 704 exec/s 4437 oom/timeout/crash: 3/0/0 time: 250s job: 37 dft_time: 0
^Z
[5]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1 -keep_seed=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ mkdir test
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2915063534
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 840 files, 0 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2946813034
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268439.txt'
MERGE-INNER: 840 total files; 0 processed earlier; will process 840 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#4 pulse cov: 17 ft: 18 exec/s: 0 rss: 31Mb
#8 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#16 pulse cov: 23 ft: 24 exec/s: 0 rss: 31Mb
#32 pulse cov: 301 ft: 315 exec/s: 0 rss: 315Mb
#64 pulse cov: 648 ft: 1141 exec/s: 0 rss: 321Mb
#128 pulse cov: 724 ft: 1575 exec/s: 0 rss: 327Mb
#256 pulse cov: 795 ft: 1976 exec/s: 0 rss: 327Mb
#512 pulse cov: 843 ft: 2317 exec/s: 0 rss: 364Mb
#840 DONE cov: 870 ft: 2550 exec/s: 0 rss: 430Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 81658 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 542 new files with 2550 new features added; 870 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

---

#2725439: cov: 2924 ft: 2562 corp: 549 exec/s 726 oom/timeout/crash: 0/0/0 time: 22s job: 9 dft_time: 0
#3332558: cov: 2924 ft: 2562 corp: 549 exec/s 55192 oom/timeout/crash: 0/0/0 time: 27s job: 10 dft_time: 0
#3978193: cov: 2928 ft: 2565 corp: 550 exec/s 53802 oom/timeout/crash: 0/0/0 time: 31s job: 11 dft_time: 0
^Z
[6]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 908055613
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 1392 files, 542 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 938266106
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268550.txt'
MERGE-INNER: 1392 total files; 0 processed earlier; will process 1392 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 208 ft: 210 exec/s: 0 rss: 32Mb
#32 pulse cov: 535 ft: 773 exec/s: 0 rss: 35Mb
#64 pulse cov: 654 ft: 1273 exec/s: 0 rss: 322Mb
#128 pulse cov: 747 ft: 1693 exec/s: 0 rss: 322Mb
#256 pulse cov: 814 ft: 2074 exec/s: 0 rss: 322Mb
#512 pulse cov: 868 ft: 2498 exec/s: 0 rss: 322Mb
#542 LOADED cov: 870 ft: 2550 exec/s: 0 rss: 322Mb
#1024 pulse cov: 870 ft: 2557 exec/s: 0 rss: 399Mb
#1392 DONE cov: 871 ft: 2567 exec/s: 1392 rss: 439Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 125428 bytes
MERGE-OUTER: consumed 0Mb (33Mb rss) to parse the control file
MERGE-OUTER: 10 new files with 17 new features added; 1 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

[gtt1995]

In D105084#2989377, @gtt1995 wrote:

#19820674: cov: 869 ft: 2528 corp: 687 exec/s 46456 oom/timeout/crash: 3/0/0 time: 219s job: 35 dft_time: 0
#20722822: cov: 870 ft: 2543 corp: 698 exec/s 24382 oom/timeout/crash: 3/0/0 time: 225s job: 36 dft_time: 0
#20891453: cov: 870 ft: 2549 corp: 704 exec/s 4437 oom/timeout/crash: 3/0/0 time: 250s job: 37 dft_time: 0
^Z
[5]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1 -keep_seed=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ mkdir test
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2915063534
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 840 files, 0 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2946813034
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268439.txt'
MERGE-INNER: 840 total files; 0 processed earlier; will process 840 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#4 pulse cov: 17 ft: 18 exec/s: 0 rss: 31Mb
#8 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#16 pulse cov: 23 ft: 24 exec/s: 0 rss: 31Mb
#32 pulse cov: 301 ft: 315 exec/s: 0 rss: 315Mb
#64 pulse cov: 648 ft: 1141 exec/s: 0 rss: 321Mb
#128 pulse cov: 724 ft: 1575 exec/s: 0 rss: 327Mb
#256 pulse cov: 795 ft: 1976 exec/s: 0 rss: 327Mb
#512 pulse cov: 843 ft: 2317 exec/s: 0 rss: 364Mb
#840 DONE cov: 870 ft: 2550 exec/s: 0 rss: 430Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 81658 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 542 new files with 2550 new features added; 870 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

---

#2725439: cov: 2924 ft: 2562 corp: 549 exec/s 726 oom/timeout/crash: 0/0/0 time: 22s job: 9 dft_time: 0
#3332558: cov: 2924 ft: 2562 corp: 549 exec/s 55192 oom/timeout/crash: 0/0/0 time: 27s job: 10 dft_time: 0
#3978193: cov: 2928 ft: 2565 corp: 550 exec/s 53802 oom/timeout/crash: 0/0/0 time: 31s job: 11 dft_time: 0
^Z
[6]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 908055613
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 1392 files, 542 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 938266106
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268550.txt'
MERGE-INNER: 1392 total files; 0 processed earlier; will process 1392 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 208 ft: 210 exec/s: 0 rss: 32Mb
#32 pulse cov: 535 ft: 773 exec/s: 0 rss: 35Mb
#64 pulse cov: 654 ft: 1273 exec/s: 0 rss: 322Mb
#128 pulse cov: 747 ft: 1693 exec/s: 0 rss: 322Mb
#256 pulse cov: 814 ft: 2074 exec/s: 0 rss: 322Mb
#512 pulse cov: 868 ft: 2498 exec/s: 0 rss: 322Mb
#542 LOADED cov: 870 ft: 2550 exec/s: 0 rss: 322Mb
#1024 pulse cov: 870 ft: 2557 exec/s: 0 rss: 399Mb
#1392 DONE cov: 871 ft: 2567 exec/s: 1392 rss: 439Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 125428 bytes
MERGE-OUTER: consumed 0Mb (33Mb rss) to parse the control file
MERGE-OUTER: 10 new files with 17 new features added; 1 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

A further discovery was made. When -keep_seed=1, as shown in the first part, panel data is consistent with cov after -merge=1. The problem occurs when -keep_seed=0, as shown in Part two.

[morehouse]

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

[gtt1995]

In D105084#2989419, @morehouse wrote:

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

NEW_FUNC: 0x5dee50 in woff2::WOFF2StringOut::WOFF2StringOut(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:24
NEW_FUNC: 0x5deed0 in woff2::WOFF2StringOut::Write(void const*, unsigned long) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:26
NEW_FUNC: 0x5def80 in woff2::WOFF2StringOut::Write(void const*, unsigned long, unsigned long) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:30
NEW_FUNC: 0x5df230 in woff2::WOFF2StringOut::SetMaxSize(unsigned long) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:47
NEW_FUNC: 0x612130 in LLVMFuzzerTestOneInput /workspace/fuzzer-test-suite/woff2-2016-05-06/target.cc:9

#866938: cov: 2723 ft: 2568 corp: 549 exec/s 71217 oom/timeout/crash: 0/0/0 time: 12s job: 6 dft_time: 0
^Z
[7]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=0 -max_total_time=2000 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3965717123
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 1403 files, 552 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 4006378744
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3270041.txt'
MERGE-INNER: 1403 total files; 0 processed earlier; will process 1403 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 30Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 228 ft: 230 exec/s: 0 rss: 31Mb
#32 pulse cov: 545 ft: 775 exec/s: 0 rss: 35Mb
#64 pulse cov: 659 ft: 1289 exec/s: 0 rss: 320Mb
#128 pulse cov: 745 ft: 1675 exec/s: 128 rss: 320Mb
#256 pulse cov: 814 ft: 2077 exec/s: 256 rss: 320Mb
#512 pulse cov: 868 ft: 2504 exec/s: 512 rss: 320Mb
#552 LOADED cov: 871 ft: 2567 exec/s: 552 rss: 320Mb
#1024 pulse cov: 871 ft: 2568 exec/s: 512 rss: 393Mb
#1403 DONE cov: 871 ft: 2568 exec/s: 701 rss: 436Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 126311 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 1 new files with 1 new features added; 0 new coverage edges

[gtt1995]

In D105084#2989419, @morehouse wrote:

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

Yes, It still exists.
I guess it has something to do with how I build?
LIB_FUZZING_ENGINE="/workspace/llvm-project/compiler-rt/lib/fuzzer/libFuzzer.a"

[gtt1995]

In D105084#2989419, @morehouse wrote:

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

I'm guessing it's fork mode, but we haven't made any changes to the rest of FuzzerFork.cpp.

[morehouse]

I think the problem was probably there before this patch. Can you reproduce the issue without this patch?

[gtt1995]

gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -max_total_time=100 -fork_corpus_groups=0 -ignore_crashes=1

WARNING: unrecognized flag '-fork_corpus_groups=0'; use -help=1 to list all flags

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3207799496
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72aff3, 0x72db60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6ca4c0,0x6f5b90),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 551 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork3271042.dir
#21325: cov: 2568 ft: 2568 corp: 551 exec/s 10662 oom/timeout/crash: 0/0/0 time: 3s job: 1 dft_time: 0
#74315: cov: 2568 ft: 2568 corp: 551 exec/s 17663 oom/timeout/crash: 0/0/0 time: 4s job: 2 dft_time: 0
#158045: cov: 2568 ft: 2568 corp: 551 exec/s 20932 oom/timeout/crash: 0/0/0 time: 5s job: 3 dft_time: 0
^Z
[8]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -max_total_time=100 -fork_corpus_groups=0 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1298798922
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72aff3, 0x72db60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6ca4c0,0x6f5b90),
MERGE-OUTER: 1404 files, 553 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1358052956
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72aff3, 0x72db60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6ca4c0,0x6f5b90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3271077.txt'
MERGE-INNER: 1404 total files; 0 processed earlier; will process 1404 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 142 ft: 144 exec/s: 0 rss: 31Mb
#32 pulse cov: 530 ft: 721 exec/s: 0 rss: 35Mb
#64 pulse cov: 664 ft: 1288 exec/s: 32 rss: 316Mb
#128 pulse cov: 747 ft: 1713 exec/s: 64 rss: 317Mb
#256 pulse cov: 814 ft: 2077 exec/s: 128 rss: 317Mb
#512 pulse cov: 868 ft: 2506 exec/s: 256 rss: 317Mb
#553 LOADED cov: 871 ft: 2568 exec/s: 276 rss: 317Mb
#1024 pulse cov: 871 ft: 2568 exec/s: 341 rss: 394Mb
#1404 DONE cov: 871 ft: 2568 exec/s: 468 rss: 433Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 126391 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 0 new files with 0 new features added; 0 new coverage edges

[gtt1995]

In D105084#2989463, @morehouse wrote:

I think the problem was probably there before this patch. Can you reproduce the issue without this patch?

This makes me confused. There are still problems with this version without Patch. Is this my own problem? This build flow is fuzzer-test-suite flow, libEngine=/path/to/ libfuzzer.a is custom.

gutaotao@x1-1:/workspace/llvm-project/compiler-rt/lib/fuzzer$ ./build.sh
ar: u' 修饰符被忽略，因为 D' 为默认（参见 `U'）
ar: 正在创建 libFuzzer.a
gutaotao@x1-1:/workspace/llvm-project/compiler-rt/lib/fuzzer$ git branch
23:25:37.898721 git.c:439 trace: built-in: git branch
23:25:37.903940 run-command.c:663 trace: run_command: unset GIT_PAGER_IN_USE; LESS=FRX LV=-c pager

fuzzer-fork-group-mode

• main

[morehouse]

Looks like an issue with fork mode. I don't think this patch caused it, so you don't need to fix it here.

I'll go ahead and land this patch today.

[gtt1995]

In D105084#2989645, @morehouse wrote:

Looks like an issue with fork mode. I don't think this patch caused it, so you don't need to fix it here.

I'll go ahead and land this patch today.

Ok. Thank you matt.

[leonardchan]

Hi. I suspect this patch is leading to the build failure we're seeing (https://luci-milo.appspot.com/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8836640518656750849?):

FAILED: compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o 
/b/s/w/ir/x/w/staging/llvm_build/./bin/clang++ --target=i386-unknown-linux-gnu --sysroot=/b/s/w/ir/x/w/cipd/linux -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/../../include --target=i386-unknown-linux-gnu -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -ffile-prefix-map=/b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins=../staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins -ffile-prefix-map=/b/s/w/ir/x/w/llvm-project/= -no-canonical-prefixes -Wall -std=c++14 -Wno-unused-parameter -O2 -g -DNDEBUG  -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-format-pedantic -D_LIBCPP_ABI_VERSION=Fuzzer -nostdinc++ -UNDEBUG -isystem /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_i386/include/c++/v1 -MD -MT compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -MF compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o.d -o compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -c /b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp:411:7: error: no matching function for call to 'CrashResistantMerge'
      CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files,
      ^~~~~~~~~~~~~~~~~~~
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerMerge.h:81:6: note: candidate function not viable: requires 11 arguments, but 10 were provided
void CrashResistantMerge(const std::vector<std::string> &Args,
     ^
1 error generated.

Would be able to send out a fix or revert? Thanks.

[morehouse]

In D105084#2990055, @leonardchan wrote:

Hi. I suspect this patch is leading to the build failure we're seeing (https://luci-milo.appspot.com/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8836640518656750849?):

FAILED: compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o 
/b/s/w/ir/x/w/staging/llvm_build/./bin/clang++ --target=i386-unknown-linux-gnu --sysroot=/b/s/w/ir/x/w/cipd/linux -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/../../include --target=i386-unknown-linux-gnu -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -ffile-prefix-map=/b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins=../staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins -ffile-prefix-map=/b/s/w/ir/x/w/llvm-project/= -no-canonical-prefixes -Wall -std=c++14 -Wno-unused-parameter -O2 -g -DNDEBUG  -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-format-pedantic -D_LIBCPP_ABI_VERSION=Fuzzer -nostdinc++ -UNDEBUG -isystem /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_i386/include/c++/v1 -MD -MT compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -MF compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o.d -o compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -c /b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp:411:7: error: no matching function for call to 'CrashResistantMerge'
      CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files,
      ^~~~~~~~~~~~~~~~~~~
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerMerge.h:81:6: note: candidate function not viable: requires 11 arguments, but 10 were provided
void CrashResistantMerge(const std::vector<std::string> &Args,
     ^
1 error generated.

Would be able to send out a fix or revert? Thanks.

Sorry, I missed this when resolving a merge conflict. Submitted https://reviews.llvm.org/rGff77c4eac79c to fix.

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

if (RunIndividualFiles) {
     Printf("***\n"
            "*** NOTE: fuzzing was not performed, you have only\n"
            "***       executed the target code on a fixed set of inputs.\n"
            "***\n");
     F->PrintFinalStats();
     exit(0);
   }
 
+  Options.ForkCorpusGroups = Flags.fork_corpus_groups;
   if (Flags.fork)
     FuzzWithFork(F->GetMD().GetRand(), Options, Args, *Inputs, Flags.fork);
 
   if (Flags.merge)
     Merge(F, Options, Args, *Inputs, Flags.merge_control_file);
 
   if (Flags.merge_inner) {
     const size_t kDefaultMaxMergeLen = 1 << 20;

compiler-rt/lib/fuzzer/FuzzerFlags.def

FUZZER_FLAG_INT(
     "If one unit runs more than this number of seconds the process will abort.")
 FUZZER_FLAG_INT(error_exitcode, 77, "When libFuzzer itself reports a bug "
   "this exit code will be used.")
 FUZZER_FLAG_INT(timeout_exitcode, 70, "When libFuzzer reports a timeout "
   "this exit code will be used.")
 FUZZER_FLAG_INT(max_total_time, 0, "If positive, indicates the maximal total "
                                    "time in seconds to run the fuzzer.")
 FUZZER_FLAG_INT(help, 0, "Print help.")
 FUZZER_FLAG_INT(fork, 0, "Experimental mode where fuzzing happens "
                 "in a subprocess")
+FUZZER_FLAG_INT(fork_corpus_groups, 0, "For fork mode, enable the corpus-group "
+    "strategy, The main corpus will be grouped according to size, "
+    "and each sub-process will randomly select seeds from different "
+    "groups as the sub-corpus.")
 FUZZER_FLAG_INT(ignore_timeouts, 1, "Ignore timeouts in fork mode")
 FUZZER_FLAG_INT(ignore_ooms, 1, "Ignore OOMs in fork mode")
 FUZZER_FLAG_INT(ignore_crashes, 0, "Ignore crashes in fork mode")
 FUZZER_FLAG_INT(merge, 0, "If 1, the 2-nd, 3-rd, etc corpora will be "
   "merged into the 1-st corpus. Only interesting units will be taken. "
   "This flag can be used to minimize a corpus.")
 FUZZER_FLAG_STRING(stop_file, "Stop fuzzing ASAP if this file exists")
 FUZZER_FLAG_STRING(merge_inner, "internal flag")

compiler-rt/lib/fuzzer/FuzzerFork.cpp

struct GlobalEnv {
   std::vector<std::string> CorpusDirs;
   std::string MainCorpusDir;
   std::string TempDir;
   std::string DFTDir;
   std::string DataFlowBinary;
   std::set<uint32_t> Features, Cov;
   std::set<std::string> FilesWithDFT;
   std::vector<std::string> Files;
+  std::vector<std::size_t> FilesSizes;
   Random *Rand;
   std::chrono::system_clock::time_point ProcessStartTime;
   int Verbosity = 0;
+  int Group = 0;
+  int NumCorpuses = 8;
 
   size_t NumTimeouts = 0;
   size_t NumOOMs = 0;
   size_t NumCrashes = 0;
 
 
   size_t NumRuns = 0;
 
FuzzJob *CreateNewJob(size_t JobId) {
     Cmd.addFlag("max_total_time", std::to_string(std::min((size_t)300, JobId)));
     Cmd.addFlag("stop_file", StopFile());
     if (!DataFlowBinary.empty()) {
       Cmd.addFlag("data_flow_trace", DFTDir);
       if (!Cmd.hasFlag("focus_function"))
         Cmd.addFlag("focus_function", "auto");
     }
     auto Job = new FuzzJob;
     std::string Seeds;
     if (size_t CorpusSubsetSize =
             std::min(Files.size(), (size_t)sqrt(Files.size() + 2))) {
       auto Time1 = std::chrono::system_clock::now();
+      if (Group) { // whether to group the corpus.
+        size_t AverageCorpusSize = Files.size() / NumCorpuses + 1;
+        size_t StartIndex = ((JobId - 1) % NumCorpuses) * AverageCorpusSize;
+        for (size_t i = 0; i < CorpusSubsetSize; i++) {
+          size_t RandNum = (*Rand)(AverageCorpusSize);
+          size_t Index = RandNum + StartIndex;
+          Index = Index < Files.size() ? Index
+                                       : Rand->SkewTowardsLast(Files.size());
+          auto &SF = Files[Index];
+          Seeds += (Seeds.empty() ? "" : ",") + SF;
+          CollectDFT(SF);
+        }
+      } else {
         for (size_t i = 0; i < CorpusSubsetSize; i++) {
           auto &SF = Files[Rand->SkewTowardsLast(Files.size())];
           Seeds += (Seeds.empty() ? "" : ",") + SF;
           CollectDFT(SF);
         }
+      }
       auto Time2 = std::chrono::system_clock::now();
       auto DftTimeInSeconds = duration_cast<seconds>(Time2 - Time1).count();
       assert(DftTimeInSeconds < std::numeric_limits<int>::max());
       Job->DftTimeInSeconds = static_cast<int>(DftTimeInSeconds);
     }
     if (!Seeds.empty()) {
       Job->SeedListPath =
           DirPlusFile(TempDir, std::to_string(JobId) + ".seeds");
       WriteToFile(Seeds, Job->SeedListPath);
       Cmd.addFlag("seed_inputs", "@" + Job->SeedListPath);
     }
     Job->LogPath = DirPlusFile(TempDir, std::to_string(JobId) + ".log");
     Job->CorpusDir = DirPlusFile(TempDir, "C" + std::to_string(JobId));
     Job->FeaturesDir = DirPlusFile(TempDir, "F" + std::to_string(JobId));
     Job->CFPath = DirPlusFile(TempDir, std::to_string(JobId) + ".merge");
     Job->JobId = JobId;
 
for (auto &F : TempFiles) {
       assert((FeatureBytes.size() % sizeof(uint32_t)) == 0);
       std::vector<uint32_t> NewFeatures(FeatureBytes.size() / sizeof(uint32_t));
       memcpy(NewFeatures.data(), FeatureBytes.data(), FeatureBytes.size());
       for (auto Ft : NewFeatures) {
         if (!Features.count(Ft)) {
           MergeCandidates.push_back(F);
           break;
         }
       }
     }
     // if (!FilesToAdd.empty() || Job->ExitCode != 0)
     Printf("#%zd: cov: %zd ft: %zd corp: %zd exec/s %zd "
            "oom/timeout/crash: %zd/%zd/%zd time: %zds job: %zd dft_time: %d\n",
            NumRuns, Cov.size(), Features.size(), Files.size(),
            Stats.average_exec_per_sec, NumOOMs, NumTimeouts, NumCrashes,
            secondsSinceProcessStartUp(), Job->JobId, Job->DftTimeInSeconds);
 
     if (MergeCandidates.empty()) return;
 
     std::vector<std::string> FilesToAdd;
     std::set<uint32_t> NewFeatures, NewCov;
     CrashResistantMerge(Args, {}, MergeCandidates, &FilesToAdd, Features,
                         &NewFeatures, Cov, &NewCov, Job->CFPath, false);
     for (auto &Path : FilesToAdd) {
       auto U = FileToVector(Path);
       auto NewPath = DirPlusFile(MainCorpusDir, Hash(U));
       WriteToFile(U, NewPath);
+      if (Group) { // Insert the queue according to the size of the seed.
+        size_t UnitSize = U.size();
+        auto Idx =
+            std::upper_bound(FilesSizes.begin(), FilesSizes.end(), UnitSize) -
+            FilesSizes.begin();
+        FilesSizes.insert(FilesSizes.begin() + Idx, UnitSize);
+        Files.insert(Files.begin() + Idx, NewPath);
+      } else {
         Files.push_back(NewPath);
       }
+    }
     Features.insert(NewFeatures.begin(), NewFeatures.end());
     Cov.insert(NewCov.begin(), NewCov.end());
     for (auto Idx : NewCov)
       if (auto *TE = TPC.PCTableEntryByIdx(Idx))
         if (TPC.PcIsFuncEntry(TE))
           PrintPC("  NEW_FUNC: %p %F %L\n", "",
                   TPC.GetNextInstructionPc(TE->PC));
-
   }
 
-
   void CollectDFT(const std::string &InputPath) {
     if (DataFlowBinary.empty()) return;
     if (!FilesWithDFT.insert(InputPath).second) return;
     Command Cmd(Args);
     Cmd.removeFlag("fork");
     Cmd.removeFlag("runs");
     Cmd.addFlag("data_flow_trace", DFTDir);
     Cmd.addArgument(InputPath);
     for (auto &C : CorpusDirs) // Remove all corpora from the args.
       Cmd.removeArgument(C);
     Cmd.setOutputFile(DirPlusFile(TempDir, "dft.log"));
     Cmd.combineOutAndErr();
     // Printf("CollectDFT: %s\n", Cmd.toString().c_str());
     ExecuteCommand(Cmd);
   }
 
 };
 
 struct JobQueue {
   std::queue<FuzzJob *> Qu;
   std::mutex Mu;
void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
 
   GlobalEnv Env;
   Env.Args = Args;
   Env.CorpusDirs = CorpusDirs;
   Env.Rand = &Rand;
   Env.Verbosity = Options.Verbosity;
   Env.ProcessStartTime = std::chrono::system_clock::now();
   Env.DataFlowBinary = Options.CollectDataFlow;
+  Env.Group = Options.ForkCorpusGroups;
 
   std::vector<SizedFile> SeedFiles;
   for (auto &Dir : CorpusDirs)
     GetSizedFilesFromDir(Dir, &SeedFiles);
   std::sort(SeedFiles.begin(), SeedFiles.end());
   Env.TempDir = TempPath("FuzzWithFork", ".dir");
   Env.DFTDir = DirPlusFile(Env.TempDir, "DFT");
   RmDirRecursive(Env.TempDir);  // in case there is a leftover from old runs.
   MkDir(Env.TempDir);
   MkDir(Env.DFTDir);
 
 
   if (CorpusDirs.empty())
     MkDir(Env.MainCorpusDir = DirPlusFile(Env.TempDir, "C"));
   else
     Env.MainCorpusDir = CorpusDirs[0];
 
   if (Options.KeepSeed) {
     for (auto &File : SeedFiles)
       Env.Files.push_back(File.File);
   } else {
     auto CFPath = DirPlusFile(Env.TempDir, "merge.txt");
     std::set<uint32_t> NewFeatures, NewCov;
     CrashResistantMerge(Env.Args, {}, SeedFiles, &Env.Files, Env.Features,
                         &NewFeatures, Env.Cov, &NewCov, CFPath, false);
     Env.Features.insert(NewFeatures.begin(), NewFeatures.end());
     Env.Cov.insert(NewFeatures.begin(), NewFeatures.end());
     RemoveFile(CFPath);
   }
+
+  if (Env.Group) {
+    for (auto &path : Env.Files)
+      Env.FilesSizes.push_back(FileSize(path));
+  }
+
   Printf("INFO: -fork=%d: %zd seed inputs, starting to fuzz in %s\n", NumJobs,
          Env.Files.size(), Env.TempDir.c_str());
 
   int ExitCode = 0;
 
   JobQueue FuzzQ, MergeQ;
 
   auto StopJobs = [&]() {
     for (int i = 0; i < NumJobs; i++)
       FuzzQ.Push(nullptr);
     MergeQ.Push(nullptr);
     WriteToFile(Unit({1}), Env.StopFile());
   };
 
+  size_t MergeCycle = 20;
+  size_t JobExecuted = 0;
   size_t JobId = 1;
   std::vector<std::thread> Threads;
   for (int t = 0; t < NumJobs; t++) {
     Threads.push_back(std::thread(WorkerThread, &FuzzQ, &MergeQ));
     FuzzQ.Push(Env.CreateNewJob(JobId++));
   }
 
   while (true) {
     std::unique_ptr<FuzzJob> Job(MergeQ.Pop());
     if (!Job)
       break;
     ExitCode = Job->ExitCode;
     if (ExitCode == Options.InterruptExitCode) {
       Printf("==%lu== libFuzzer: a child was interrupted; exiting\n", GetPid());
       StopJobs();
       break;
     }
     Fuzzer::MaybeExitGracefully();
 
     Env.RunOneMergeJob(Job.get());
 
+    // merge the corpus .
+    JobExecuted++;
+    if (Env.Group && JobExecuted >= MergeCycle) {
+      std::vector<SizedFile> CurrentSeedFiles;
+      for (auto &Dir : CorpusDirs)
+        GetSizedFilesFromDir(Dir, &CurrentSeedFiles);
+      std::sort(CurrentSeedFiles.begin(), CurrentSeedFiles.end());
+
+      auto CFPath = DirPlusFile(Env.TempDir, "merge.txt");
+      std::set<uint32_t> TmpNewFeatures, TmpNewCov;
+      std::set<uint32_t> TmpFeatures, TmpCov;
+      Env.Files.clear();
+      Env.FilesSizes.clear();
+      CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files,
+                          TmpFeatures, &TmpNewFeatures, TmpCov, &TmpNewCov,
+                          CFPath, false);
+      for (auto &path : Env.Files)
+        Env.FilesSizes.push_back(FileSize(path));
+      RemoveFile(CFPath);
+      JobExecuted = 0;
+      MergeCycle += 5;
+    }
+
+    // Since the number of corpus seeds will gradually increase, in order to
+    // control the number in each group to be about three times the number of
+    // seeds selected each time, the number of groups is dynamically adjusted.
+    if (Env.Files.size() < 2000)
+      Env.NumCorpuses = 12;
+    else if (Env.Files.size() < 6000)
+      Env.NumCorpuses = 20;
+    else if (Env.Files.size() < 12000)
+      Env.NumCorpuses = 32;
+    else if (Env.Files.size() < 16000)
+      Env.NumCorpuses = 40;
+    else if (Env.Files.size() < 24000)
+      Env.NumCorpuses = 60;
+    else
+      Env.NumCorpuses = 80;
+
     // Continue if our crash is one of the ignorred ones.
     if (Options.IgnoreTimeouts && ExitCode == Options.TimeoutExitCode)
       Env.NumTimeouts++;
     else if (Options.IgnoreOOMs && ExitCode == Options.OOMExitCode)
       Env.NumOOMs++;
     else if (ExitCode != 0) {
       Env.NumCrashes++;
       if (Options.IgnoreCrashes) {
         std::ifstream In(Job->LogPath);
         std::string Line;
         while (std::getline(In, Line, '\n'))
           if (Line.find("ERROR:") != Line.npos ||
               Line.find("runtime error:") != Line.npos)
             Printf("%s\n", Line.c_str());
       } else {
         // And exit if we don't ignore this crash.
         Printf("INFO: log from the inner process:\n%s",
                FileToString(Job->LogPath).c_str());
         StopJobs();
         break;
       }
     }
 
     // Stop if we are over the time budget.
     // This is not precise, since other threads are still running
     // and we will wait while joining them.
     // We also don't stop instantly: other jobs need to finish.
     if (Options.MaxTotalTimeSec > 0 &&
         Env.secondsSinceProcessStartUp() >= (size_t)Options.MaxTotalTimeSec) {

compiler-rt/lib/fuzzer/FuzzerOptions.h

struct FuzzingOptions {
   bool ReduceInputs = false;
   int ReloadIntervalSec = 1;
   bool ShuffleAtStartUp = true;
   bool PreferSmall = true;
   size_t MaxNumberOfRuns = -1L;
   int ReportSlowUnits = 10;
   bool OnlyASCII = false;
   bool Entropic = true;
+  bool ForkCorpusGroups = true;
   size_t EntropicFeatureFrequencyThreshold = 0xFF;
   size_t EntropicNumberOfRarestFeatures = 100;
   bool EntropicScalePerExecTime = false;
   std::string OutputCorpus;
   std::string ArtifactPrefix = "./";
   std::string ExactArtifactPath;
   std::string ExitOnSrcPos;
   std::string ExitOnItem;

compiler-rt/test/fuzzer/fork_corpus_groups.test

+# UNSUPPORTED: darwin, freebsd, aarch64
+BINGO: BINGO
+RUN: %cpp_compiler %S/SimpleTest.cpp -o %t-SimpleTest
+RUN: not %run %t-SimpleTest -fork=1 -fork_corpus_groups=1 2>&1 | FileCheck %s --check-prefix=BINGO
+
+TIMEOUT: ERROR: libFuzzer: timeout
+RUN: %cpp_compiler %S/TimeoutTest.cpp -o %t-TimeoutTest
+RUN: not %run %t-TimeoutTest -fork=1 -fork_corpus_groups=1 -timeout=1 -ignore_timeouts=0 2>&1 | FileCheck %s --check-prefix=TIMEOUT
+
+OOM: ERROR: libFuzzer: out-of-memory
+RUN: %cpp_compiler %S/OutOfMemoryTest.cpp -o %t-OutOfMemoryTest
+RUN: not %run %t-OutOfMemoryTest -fork=1 -fork_corpus_groups=1 -ignore_ooms=0  -rss_limit_mb=128 2>&1 | FileCheck %s --check-prefix=OOM
+
+# access-violation is the error thrown on Windows. Address will be smaller on i386.
+CRASH: {{SEGV|access-violation}} on unknown address 0x00000000
+RUN: %cpp_compiler %S/ShallowOOMDeepCrash.cpp -o %t-ShallowOOMDeepCrash
+RUN: not %run %t-ShallowOOMDeepCrash -fork=1 -fork_corpus_groups=1 -rss_limit_mb=128 2>&1 | FileCheck %s --check-prefix=CRASH
+
+MAX_TOTAL_TIME: INFO: fuzzed for {{.*}} seconds, wrapping up soon
+MAX_TOTAL_TIME: INFO: exiting: {{.*}} time:
+RUN: not %run %t-ShallowOOMDeepCrash -fork=1 -fork_corpus_groups=1 -rss_limit_mb=128 -ignore_crashes=1 -max_total_time=10 2>&1 | FileCheck %s  --check-prefix=MAX_TOTAL_TIME