Redistribute energy for Corpus

[gtt1995]

Summary

I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method can give more energy to the small seeds in the corpus and
increase the throughput of the test.

Fuzzbench data (glibfuzzer is -fork_corpus_groups=1):
https://www.fuzzbench.com/reports/experimental/2021-08-05-parallel/index.html

[morehouse]

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

Also this flag is currently used as a boolean, the actual value is ignored. Can we either fix this or change the description here.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

251

Also, since we regenerate Files and FilesSizes on each merge, do we still need to insert in sorted order here? What benefit to do we get from it?

382

Should NumCorpuses be part of Env, so we don't need to pass it around?

406–407

Can simplify.

412–415

I think we can at least remove these lines, since I think they only need to be executed once.

437–452

Can simplify.

438

Since we dynamically adjust NumCorpuses, should we even allow the user to set the initial number of corpuses?

[gtt1995]

compiler-rt/lib/fuzzer/FuzzerFork.cpp

143

it seems to be like this.

146–149

You mean to use Rand->SkewTowardsLast()？ or else?

251

Merge operation is very expensive, so choose to merge periodically. Before each merge, Files are out of order, so they need to be inserted according to size all the time.

382

It is not meaningful to pass the initial NumCorpuses value, because it will automatically adjust with the number of seeds in the corpus.

438

The presence or absence of the initial value has little effect on performance, and the number of seeds will change rapidly.

compiler-rt/lib/fuzzer/FuzzerFork.h

22

OK.

[gtt1995]

• 0820
• delete something

1. Updating D105084: Redistribute energy for Corpus #
2. Enter a brief description of the changes included in this update.
3. The first line is used as subject, next lines as comment. #
4. If you intended to create a new revision, use:
5. $ arc diff --create

Some modifications.

[gtt1995]

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

[gtt1995]

I have corrected most of the problems.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

148

All right.

166–169

Sorry, Where is the same?

381

OK, i deleted that line.

447

hhh, this comment is not added by me.

[gtt1995]

• some fix

[gtt1995]

• simplify

[morehouse]

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

Please change the default value to 0.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

146–149

No, I mean using Rand to choose a random value between 0 and AverageCorpusSize, instead of creating a new std::uniform_int_distribution.

346–350

To match surrounding style.

361–363

470–472

481–496

We need the elses for it to work right. Otherwise, we'll always get NumCorpuses = 60 or NumCorpuses = 80.

[gtt1995]

compiler-rt/lib/fuzzer/FuzzerFlags.def

61

It's ok.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

146–149

All right.

346–350

It's ok.

481–496

Maye not. When fuzzing different programs, the usually generated the number of interesting seeds are different, hundreds, thousands, tens of thousands of exist. Therefore, when fixed set to numcorpuses=80, such as 1600 seeds, subset to pick sqrt (1600) = 40 at a time, but only 20 per group, and eventually repeat each time, and then more groups 80, it needs more job to test a round of the corpus.

Maybe i can set an interval of coarse graininess.

[gtt1995]

• ok

[gtt1995]

• a little fix

[gtt1995]

• change the NumCorpuses

[gtt1995]

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

[morehouse]

In D105084#2969098, @gtt1995 wrote:

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

I'm not sure whether it's a problem. I think sometimes cov and ft can be the same. But in general ft >= cov. And as long as the default behavior is unchanged I'm not too worried.

compiler-rt/lib/fuzzer/FuzzerFork.cpp

481–496

Ok, but we still need else if instead of if or it won't work.

[morehouse]

I'll be gone until late next week. Then I'll apply it locally and experiment. If all looks good, we can land it after fixing my last comment

[gtt1995]

In D105084#2969374, @morehouse wrote:

I'll be gone until late next week. Then I'll apply it locally and experiment. If all looks good, we can land it after fixing my last comment

It's OK ! Thanks a lot !

[gtt1995]

• if -> else if

[gtt1995]

In D105084#2969369, @morehouse wrote:

In D105084#2969098, @gtt1995 wrote:

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

I'm not sure whether it's a problem. I think sometimes cov and ft can be the same. But in general ft >= cov. And as long as the default behavior is unchanged I'm not too worried.

OK, look forward to your local experiment result, it may have the problem of inaccurate COV , the COV on the panel will be quite different from the result after -merge (dry run) .

[gtt1995]

try to fix Unit test

[morehouse]

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

[gtt1995]

In D105084#2982975, @morehouse wrote:

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

Ok!

[gtt1995]

In D105084#2982975, @morehouse wrote:

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

Sorry, Could you tell me how to do this? I have no idea.

[gtt1995]

In D105084#2982975, @morehouse wrote:

@gtt1995 Can you add a simple test that verifies -fork_corpus_groups=1 works? Something like https://github.com/llvm/llvm-project/blob/main/compiler-rt/test/fuzzer/fork.test.

Other than that, LGTM. All existing tests pass for me locally.

I applied the latest changes locally, and -fork_corpus_groups=1 is actually more efficient.

[gtt1995]

• add a fork_corpus_groups.test

[gtt1995]

In D105084#2983993, @gtt1995 wrote:

• add a fork_corpus_groups.test

Hello, Does this work?

[morehouse]

LGTM

[gtt1995]

In D105084#2987018, @morehouse wrote:

LGTM

Thank you so much for your work! thanks matt！

[gtt1995]

In D105084#2987018, @morehouse wrote:

LGTM

In D105084#2969369, @morehouse wrote:

In D105084#2969098, @gtt1995 wrote:

In D105084#2954998, @gtt1995 wrote:

I found a serious problem. After adding this new method, libfuzzer's panel data ”cov“ becomes inaccurate, and its initial value is the same as "ft". I haven't been able to find the problem, so I didn't solve it, can you help me?

This is glibfuzzer:
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12$ ./boringssl-2016-02-12-fsanitize_fuzzer 2 seeds/ -fork=3 -fork_corpus_groups=1 -max_total_time=600
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1768398068
INFO: Loaded 1 modules (16551 inline 8-bit counters): 16551 [0x7c8fc3, 0x7cd06a),
INFO: Loaded 1 PC tables (16551 PCs): 16551 [0x714b10,0x755580),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 100 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork1195053.dir
#11250: cov: 2285 ft: 2285 corp: 100 exec/s 5625 oom/timeout/crash: 0/0/0 time: 2s job: 1 dft_time: 0

NEW_FUNC: 0x4c93a0 in EVP_PKEY_new /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-boringssl-2016-02-12/BUILD/crypto/evp/evp.c:75

The initial values ​​of cov and ft should be different.

However, this is only a display error and has no effect on the actual performance. Fuzzbench directly measures the performance by evaluating the actual seeds in the corpus.

Hello, how should this issue be handled?

I'm not sure whether it's a problem. I think sometimes cov and ft can be the same. But in general ft >= cov. And as long as the default behavior is unchanged I'm not too worried.

Hi matt ,the aforementioned problem does not seem to have been solved.

[gtt1995]

The cov on the panel and the cov result after -merge=1 are very different, and ft is accurate.

[gtt1995]

#19820674: cov: 869 ft: 2528 corp: 687 exec/s 46456 oom/timeout/crash: 3/0/0 time: 219s job: 35 dft_time: 0
#20722822: cov: 870 ft: 2543 corp: 698 exec/s 24382 oom/timeout/crash: 3/0/0 time: 225s job: 36 dft_time: 0
#20891453: cov: 870 ft: 2549 corp: 704 exec/s 4437 oom/timeout/crash: 3/0/0 time: 250s job: 37 dft_time: 0
^Z
[5]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1 -keep_seed=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ mkdir test
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2915063534
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 840 files, 0 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2946813034
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268439.txt'
MERGE-INNER: 840 total files; 0 processed earlier; will process 840 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#4 pulse cov: 17 ft: 18 exec/s: 0 rss: 31Mb
#8 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#16 pulse cov: 23 ft: 24 exec/s: 0 rss: 31Mb
#32 pulse cov: 301 ft: 315 exec/s: 0 rss: 315Mb
#64 pulse cov: 648 ft: 1141 exec/s: 0 rss: 321Mb
#128 pulse cov: 724 ft: 1575 exec/s: 0 rss: 327Mb
#256 pulse cov: 795 ft: 1976 exec/s: 0 rss: 327Mb
#512 pulse cov: 843 ft: 2317 exec/s: 0 rss: 364Mb
#840 DONE cov: 870 ft: 2550 exec/s: 0 rss: 430Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 81658 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 542 new files with 2550 new features added; 870 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

---

#2725439: cov: 2924 ft: 2562 corp: 549 exec/s 726 oom/timeout/crash: 0/0/0 time: 22s job: 9 dft_time: 0
#3332558: cov: 2924 ft: 2562 corp: 549 exec/s 55192 oom/timeout/crash: 0/0/0 time: 27s job: 10 dft_time: 0
#3978193: cov: 2928 ft: 2565 corp: 550 exec/s 53802 oom/timeout/crash: 0/0/0 time: 31s job: 11 dft_time: 0
^Z
[6]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 908055613
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 1392 files, 542 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 938266106
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268550.txt'
MERGE-INNER: 1392 total files; 0 processed earlier; will process 1392 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 208 ft: 210 exec/s: 0 rss: 32Mb
#32 pulse cov: 535 ft: 773 exec/s: 0 rss: 35Mb
#64 pulse cov: 654 ft: 1273 exec/s: 0 rss: 322Mb
#128 pulse cov: 747 ft: 1693 exec/s: 0 rss: 322Mb
#256 pulse cov: 814 ft: 2074 exec/s: 0 rss: 322Mb
#512 pulse cov: 868 ft: 2498 exec/s: 0 rss: 322Mb
#542 LOADED cov: 870 ft: 2550 exec/s: 0 rss: 322Mb
#1024 pulse cov: 870 ft: 2557 exec/s: 0 rss: 399Mb
#1392 DONE cov: 871 ft: 2567 exec/s: 1392 rss: 439Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 125428 bytes
MERGE-OUTER: consumed 0Mb (33Mb rss) to parse the control file
MERGE-OUTER: 10 new files with 17 new features added; 1 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

[gtt1995]

In D105084#2989377, @gtt1995 wrote:

#19820674: cov: 869 ft: 2528 corp: 687 exec/s 46456 oom/timeout/crash: 3/0/0 time: 219s job: 35 dft_time: 0
#20722822: cov: 870 ft: 2543 corp: 698 exec/s 24382 oom/timeout/crash: 3/0/0 time: 225s job: 36 dft_time: 0
#20891453: cov: 870 ft: 2549 corp: 704 exec/s 4437 oom/timeout/crash: 3/0/0 time: 250s job: 37 dft_time: 0
^Z
[5]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1 -keep_seed=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ mkdir test
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2915063534
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 840 files, 0 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 2946813034
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268439.txt'
MERGE-INNER: 840 total files; 0 processed earlier; will process 840 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#4 pulse cov: 17 ft: 18 exec/s: 0 rss: 31Mb
#8 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#16 pulse cov: 23 ft: 24 exec/s: 0 rss: 31Mb
#32 pulse cov: 301 ft: 315 exec/s: 0 rss: 315Mb
#64 pulse cov: 648 ft: 1141 exec/s: 0 rss: 321Mb
#128 pulse cov: 724 ft: 1575 exec/s: 0 rss: 327Mb
#256 pulse cov: 795 ft: 1976 exec/s: 0 rss: 327Mb
#512 pulse cov: 843 ft: 2317 exec/s: 0 rss: 364Mb
#840 DONE cov: 870 ft: 2550 exec/s: 0 rss: 430Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 81658 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 542 new files with 2550 new features added; 870 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

---

#2725439: cov: 2924 ft: 2562 corp: 549 exec/s 726 oom/timeout/crash: 0/0/0 time: 22s job: 9 dft_time: 0
#3332558: cov: 2924 ft: 2562 corp: 549 exec/s 55192 oom/timeout/crash: 0/0/0 time: 27s job: 10 dft_time: 0
#3978193: cov: 2928 ft: 2565 corp: 550 exec/s 53802 oom/timeout/crash: 0/0/0 time: 31s job: 11 dft_time: 0
^Z
[6]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=1 -max_total_time=2000 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 908055613
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 1392 files, 542 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 938266106
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3268550.txt'
MERGE-INNER: 1392 total files; 0 processed earlier; will process 1392 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 208 ft: 210 exec/s: 0 rss: 32Mb
#32 pulse cov: 535 ft: 773 exec/s: 0 rss: 35Mb
#64 pulse cov: 654 ft: 1273 exec/s: 0 rss: 322Mb
#128 pulse cov: 747 ft: 1693 exec/s: 0 rss: 322Mb
#256 pulse cov: 814 ft: 2074 exec/s: 0 rss: 322Mb
#512 pulse cov: 868 ft: 2498 exec/s: 0 rss: 322Mb
#542 LOADED cov: 870 ft: 2550 exec/s: 0 rss: 322Mb
#1024 pulse cov: 870 ft: 2557 exec/s: 0 rss: 399Mb
#1392 DONE cov: 871 ft: 2567 exec/s: 1392 rss: 439Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 125428 bytes
MERGE-OUTER: consumed 0Mb (33Mb rss) to parse the control file
MERGE-OUTER: 10 new files with 17 new features added; 1 new coverage edges
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$

A further discovery was made. When -keep_seed=1, as shown in the first part, panel data is consistent with cov after -merge=1. The problem occurs when -keep_seed=0, as shown in Part two.

[morehouse]

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

[gtt1995]

In D105084#2989419, @morehouse wrote:

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

NEW_FUNC: 0x5dee50 in woff2::WOFF2StringOut::WOFF2StringOut(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:24
NEW_FUNC: 0x5deed0 in woff2::WOFF2StringOut::Write(void const*, unsigned long) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:26
NEW_FUNC: 0x5def80 in woff2::WOFF2StringOut::Write(void const*, unsigned long, unsigned long) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:30
NEW_FUNC: 0x5df230 in woff2::WOFF2StringOut::SetMaxSize(unsigned long) /workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06/SRC/src/woff2_out.cc:47
NEW_FUNC: 0x612130 in LLVMFuzzerTestOneInput /workspace/fuzzer-test-suite/woff2-2016-05-06/target.cc:9

#866938: cov: 2723 ft: 2568 corp: 549 exec/s 71217 oom/timeout/crash: 0/0/0 time: 12s job: 6 dft_time: 0
^Z
[7]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -fork_corpus_groups=0 -max_total_time=2000 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3965717123
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
MERGE-OUTER: 1403 files, 552 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 4006378744
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72cff3, 0x72fb60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6cb5c0,0x6f6c90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3270041.txt'
MERGE-INNER: 1403 total files; 0 processed earlier; will process 1403 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 30Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 228 ft: 230 exec/s: 0 rss: 31Mb
#32 pulse cov: 545 ft: 775 exec/s: 0 rss: 35Mb
#64 pulse cov: 659 ft: 1289 exec/s: 0 rss: 320Mb
#128 pulse cov: 745 ft: 1675 exec/s: 128 rss: 320Mb
#256 pulse cov: 814 ft: 2077 exec/s: 256 rss: 320Mb
#512 pulse cov: 868 ft: 2504 exec/s: 512 rss: 320Mb
#552 LOADED cov: 871 ft: 2567 exec/s: 552 rss: 320Mb
#1024 pulse cov: 871 ft: 2568 exec/s: 512 rss: 393Mb
#1403 DONE cov: 871 ft: 2568 exec/s: 701 rss: 436Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 126311 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 1 new files with 1 new features added; 0 new coverage edges

[gtt1995]

In D105084#2989419, @morehouse wrote:

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

Yes, It still exists.
I guess it has something to do with how I build?
LIB_FUZZING_ENGINE="/workspace/llvm-project/compiler-rt/lib/fuzzer/libFuzzer.a"

[gtt1995]

In D105084#2989419, @morehouse wrote:

Does it still happen with -fork=3 -fork_corpus_groups=0? From the code I'm guessing the issue is with fork mode, not the new fork_corpus_groups mode.

I'm guessing it's fork mode, but we haven't made any changes to the rest of FuzzerFork.cpp.

[morehouse]

I think the problem was probably there before this patch. Can you reproduce the issue without this patch?

[gtt1995]

gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -max_total_time=100 -fork_corpus_groups=0 -ignore_crashes=1

WARNING: unrecognized flag '-fork_corpus_groups=0'; use -help=1 to list all flags

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3207799496
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72aff3, 0x72db60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6ca4c0,0x6f5b90),
INFO: -fork=3: fuzzing in separate process(s)
INFO: -fork=3: 551 seed inputs, starting to fuzz in /tmp/libFuzzerTemp.FuzzWithFork3271042.dir
#21325: cov: 2568 ft: 2568 corp: 551 exec/s 10662 oom/timeout/crash: 0/0/0 time: 3s job: 1 dft_time: 0
#74315: cov: 2568 ft: 2568 corp: 551 exec/s 17663 oom/timeout/crash: 0/0/0 time: 4s job: 2 dft_time: 0
#158045: cov: 2568 ft: 2568 corp: 551 exec/s 20932 oom/timeout/crash: 0/0/0 time: 5s job: 3 dft_time: 0
^Z
[8]+ 已停止 ./woff2-2016-05-06-fsanitize_fuzzer 1/ -fork=3 -max_total_time=100 -fork_corpus_groups=0 -ignore_crashes=1
gutaotao@x1-1:/workspace/fuzzer-test-suite/build-libfuzzer/RUNDIR-woff2-2016-05-06$ ./woff2-2016-05-06-fsanitize_fuzzer test/ 1/ -merge=1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1298798922
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72aff3, 0x72db60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6ca4c0,0x6f5b90),
MERGE-OUTER: 1404 files, 553 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1358052956
INFO: Loaded 1 modules (11117 inline 8-bit counters): 11117 [0x72aff3, 0x72db60),
INFO: Loaded 1 PC tables (11117 PCs): 11117 [0x6ca4c0,0x6f5b90),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge3271077.txt'
MERGE-INNER: 1404 total files; 0 processed earlier; will process 1404 files now
#1 pulse cov: 15 ft: 16 exec/s: 0 rss: 31Mb
#2 pulse cov: 16 ft: 17 exec/s: 0 rss: 31Mb
#4 pulse cov: 18 ft: 19 exec/s: 0 rss: 31Mb
#8 pulse cov: 22 ft: 23 exec/s: 0 rss: 31Mb
#16 pulse cov: 142 ft: 144 exec/s: 0 rss: 31Mb
#32 pulse cov: 530 ft: 721 exec/s: 0 rss: 35Mb
#64 pulse cov: 664 ft: 1288 exec/s: 32 rss: 316Mb
#128 pulse cov: 747 ft: 1713 exec/s: 64 rss: 317Mb
#256 pulse cov: 814 ft: 2077 exec/s: 128 rss: 317Mb
#512 pulse cov: 868 ft: 2506 exec/s: 256 rss: 317Mb
#553 LOADED cov: 871 ft: 2568 exec/s: 276 rss: 317Mb
#1024 pulse cov: 871 ft: 2568 exec/s: 341 rss: 394Mb
#1404 DONE cov: 871 ft: 2568 exec/s: 468 rss: 433Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 126391 bytes
MERGE-OUTER: consumed 0Mb (32Mb rss) to parse the control file
MERGE-OUTER: 0 new files with 0 new features added; 0 new coverage edges

[gtt1995]

In D105084#2989463, @morehouse wrote:

I think the problem was probably there before this patch. Can you reproduce the issue without this patch?

This makes me confused. There are still problems with this version without Patch. Is this my own problem? This build flow is fuzzer-test-suite flow, libEngine=/path/to/ libfuzzer.a is custom.

gutaotao@x1-1:/workspace/llvm-project/compiler-rt/lib/fuzzer$ ./build.sh
ar: u' 修饰符被忽略，因为 D' 为默认（参见 `U'）
ar: 正在创建 libFuzzer.a
gutaotao@x1-1:/workspace/llvm-project/compiler-rt/lib/fuzzer$ git branch
23:25:37.898721 git.c:439 trace: built-in: git branch
23:25:37.903940 run-command.c:663 trace: run_command: unset GIT_PAGER_IN_USE; LESS=FRX LV=-c pager

fuzzer-fork-group-mode

• main

[morehouse]

Looks like an issue with fork mode. I don't think this patch caused it, so you don't need to fix it here.

I'll go ahead and land this patch today.

[gtt1995]

In D105084#2989645, @morehouse wrote:

Looks like an issue with fork mode. I don't think this patch caused it, so you don't need to fix it here.

I'll go ahead and land this patch today.

Ok. Thank you matt.

[leonardchan]

Hi. I suspect this patch is leading to the build failure we're seeing (https://luci-milo.appspot.com/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8836640518656750849?):

FAILED: compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o 
/b/s/w/ir/x/w/staging/llvm_build/./bin/clang++ --target=i386-unknown-linux-gnu --sysroot=/b/s/w/ir/x/w/cipd/linux -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/../../include --target=i386-unknown-linux-gnu -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -ffile-prefix-map=/b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins=../staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins -ffile-prefix-map=/b/s/w/ir/x/w/llvm-project/= -no-canonical-prefixes -Wall -std=c++14 -Wno-unused-parameter -O2 -g -DNDEBUG  -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-format-pedantic -D_LIBCPP_ABI_VERSION=Fuzzer -nostdinc++ -UNDEBUG -isystem /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_i386/include/c++/v1 -MD -MT compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -MF compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o.d -o compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -c /b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp:411:7: error: no matching function for call to 'CrashResistantMerge'
      CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files,
      ^~~~~~~~~~~~~~~~~~~
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerMerge.h:81:6: note: candidate function not viable: requires 11 arguments, but 10 were provided
void CrashResistantMerge(const std::vector<std::string> &Args,
     ^
1 error generated.

Would be able to send out a fix or revert? Thanks.

[morehouse]

In D105084#2990055, @leonardchan wrote:

Hi. I suspect this patch is leading to the build failure we're seeing (https://luci-milo.appspot.com/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8836640518656750849?):

FAILED: compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o 
/b/s/w/ir/x/w/staging/llvm_build/./bin/clang++ --target=i386-unknown-linux-gnu --sysroot=/b/s/w/ir/x/w/cipd/linux -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/../../include --target=i386-unknown-linux-gnu -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -ffile-prefix-map=/b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins=../staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins -ffile-prefix-map=/b/s/w/ir/x/w/llvm-project/= -no-canonical-prefixes -Wall -std=c++14 -Wno-unused-parameter -O2 -g -DNDEBUG  -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-format-pedantic -D_LIBCPP_ABI_VERSION=Fuzzer -nostdinc++ -UNDEBUG -isystem /b/s/w/ir/x/w/staging/llvm_build/runtimes/runtimes-i386-unknown-linux-gnu-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_i386/include/c++/v1 -MD -MT compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -MF compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o.d -o compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.i386.dir/FuzzerFork.cpp.o -c /b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerFork.cpp:411:7: error: no matching function for call to 'CrashResistantMerge'
      CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files,
      ^~~~~~~~~~~~~~~~~~~
/b/s/w/ir/x/w/llvm-project/compiler-rt/lib/fuzzer/FuzzerMerge.h:81:6: note: candidate function not viable: requires 11 arguments, but 10 were provided
void CrashResistantMerge(const std::vector<std::string> &Args,
     ^
1 error generated.

Would be able to send out a fix or revert? Thanks.

Sorry, I missed this when resolving a merge conflict. Submitted https://reviews.llvm.org/rGff77c4eac79c to fix.

compiler-rt/lib/fuzzer/11.diff

+diff --git a/compiler-rt/lib/fuzzer/FuzzerDriver.cpp b/compiler-rt/lib/fuzzer/FuzzerDriver.cpp
+index ceaa9070512f..37d9df13af80 100644
+--- a/compiler-rt/lib/fuzzer/FuzzerDriver.cpp
++++ b/compiler-rt/lib/fuzzer/FuzzerDriver.cpp
+@@ -870 +870,2 @@ int FuzzerDriver(int *argc, char ***argv, UserCallback Callback) {
+-    FuzzWithFork(F->GetMD().GetRand(), Options, Args, *Inputs, Flags.fork);
++    FuzzWithFork(F->GetMD().GetRand(), Options, Args, *Inputs, Flags.fork,
++    Flags.group);
+diff --git a/compiler-rt/lib/fuzzer/FuzzerFlags.def b/compiler-rt/lib/fuzzer/FuzzerFlags.def
+index ab31da0ae5d6..2c9d94951fe1 100644
+--- a/compiler-rt/lib/fuzzer/FuzzerFlags.def
++++ b/compiler-rt/lib/fuzzer/FuzzerFlags.def
+@@ -58,0 +59,2 @@ FUZZER_FLAG_INT(help, 0, "Print help.")
++FUZZER_FLAG_INT(group, 1, "FOR fork mode. Divide the main corpus into N groups "
++   "according to size.")
+diff --git a/compiler-rt/lib/fuzzer/FuzzerFork.cpp b/compiler-rt/lib/fuzzer/FuzzerFork.cpp
+index 5134a5d979e6..1f19134a79e2 100644
+--- a/compiler-rt/lib/fuzzer/FuzzerFork.cpp
++++ b/compiler-rt/lib/fuzzer/FuzzerFork.cpp
+@@ -72 +72 @@ struct FuzzJob {
+-  size_t      JobId;
++  size_t JobId;
+@@ -74 +74 @@ struct FuzzJob {
+-  int         DftTimeInSeconds = 0;
++  int DftTimeInSeconds = 0;
+@@ -97,0 +98,2 @@ struct GlobalEnv {
++  // Declare a variable to store the seed size.
++  Vector<std::size_t> FilesSizes;
+@@ -117 +119 @@ struct GlobalEnv {
+-  FuzzJob *CreateNewJob(size_t JobId) {
++  FuzzJob *CreateNewJob(size_t JobId, int NumCorpuses, int Group) {
+@@ -135,0 +138 @@ struct GlobalEnv {
++    auto Time1 = std::chrono::system_clock::now();
+@@ -137,11 +140,31 @@ struct GlobalEnv {
+-            std::min(Files.size(), (size_t)sqrt(Files.size() + 2))) {
+-      auto Time1 = std::chrono::system_clock::now();
+-      for (size_t i = 0; i < CorpusSubsetSize; i++) {
+-        auto &SF = Files[Rand->SkewTowardsLast(Files.size())];
+-        Seeds += (Seeds.empty() ? "" : ",") + SF;
+-        CollectDFT(SF);
+-      }
+-      auto Time2 = std::chrono::system_clock::now();
+-      auto DftTimeInSeconds = duration_cast<seconds>(Time2 - Time1).count();
+-      assert(DftTimeInSeconds < std::numeric_limits<int>::max());
+-      Job->DftTimeInSeconds = static_cast<int>(DftTimeInSeconds);
++                std::min(Files.size(), (size_t)sqrt(Files.size() + 2))){
++       if (Group) { // flag: whether to group the corpus.
++             size_t AverageCorpusSize = Files.size() / NumCorpuses + 1;
++             if (Files.size() == 0)
++                     AverageCorpusSize = 0;
++             size_t StartIndex = ((JobId - 1) % NumCorpuses) * AverageCorpusSize;
++             for (size_t i = 0; i < CorpusSubsetSize; i++) {
++                std::random_device rd;
++                std::mt19937 RandomSeed(rd());
++                std::uniform_int_distribution<> rand(0,AverageCorpusSize);
++                size_t RandNum = rand(RandomSeed);
++                size_t Index = RandNum + StartIndex;
++                if (Index < Files.size()) {
++                       auto &SF = Files[Index];
++                     Seeds += (Seeds.empty() ? "" : ",") + SF;
++                     CollectDFT(SF);
++                }
++                else  {
++                       auto &SF = Files[Rand->SkewTowardsLast(Files.size())];
++                     Seeds += (Seeds.empty() ? "" : ",") + SF;
++                     CollectDFT(SF);
++                }
++             }
++       }
++       else{
++               for (size_t i = 0; i < CorpusSubsetSize; i++) {
++               auto &SF = Files[Rand->SkewTowardsLast(Files.size())];
++               Seeds += (Seeds.empty() ? "" : ",") + SF;
++               CollectDFT(SF);
++               }
++       }
+@@ -148,0 +172,5 @@ struct GlobalEnv {
++    auto Time2 = std::chrono::system_clock::now();
++    auto DftTimeInSeconds = duration_cast<seconds>(Time2 - Time1).count();
++    assert(DftTimeInSeconds < std::numeric_limits<int>::max());
++    Job->DftTimeInSeconds = static_cast<int>(DftTimeInSeconds);
++
+@@ -182 +210 @@ struct GlobalEnv {
+-  void RunOneMergeJob(FuzzJob *Job) {
++  void RunOneMergeJob(FuzzJob *Job, int Group) {
+@@ -222 +250,10 @@ struct GlobalEnv {
+-      Files.push_back(NewPath);
++      if (Group) {// Insert the queue according to the size of the seed.
++       long usz = U.size();
++       auto idx = std::upper_bound(FilesSizes.begin(), FilesSizes.end(), usz) - 
++         FilesSizes.begin();
++       FilesSizes.insert(FilesSizes.begin() + idx, usz);
++       Files.insert(Files.begin() + idx, NewPath);
++      }
++      else {
++       Files.push_back(NewPath);
++      }
+@@ -287 +324,2 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-                  const Vector<std::string> &CorpusDirs, int NumJobs) {
++                  const Vector<std::string> &CorpusDirs, int NumJobs,
++     int Group) {
+@@ -315 +353 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-    for (auto &File : SeedFiles)
++    for (auto &File : SeedFiles){
+@@ -316,0 +355,2 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
++      Env.FilesSizes.push_back(File.Size);
++    }
+@@ -325,0 +366,7 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
++
++  if (Group) {
++   for (auto &path : Env.Files) {
++     Env.FilesSizes.push_back(FileSize(path));
++   }
++  }
++
+@@ -328,0 +376 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
++
+@@ -339 +387,4 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-
++  
++  size_t MergeCycle = 20;
++  size_t JobExecuted = 0;
++  int NumCorpuses = 8;
+@@ -344 +395 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-    FuzzQ.Push(Env.CreateNewJob(JobId++));
++    FuzzQ.Push(Env.CreateNewJob(JobId++, NumCorpuses, Group));
+@@ -359 +410,29 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-    Env.RunOneMergeJob(Job.get());
++    Env.RunOneMergeJob(Job.get(), Group);
++    // merge the corpus .
++    JobExecuted ++;
++    if (Group) {
++        if(JobExecuted >= MergeCycle){
++               Vector<SizedFile> CurrentSeedFiles;
++               for (auto &Dir : CorpusDirs)
++                   GetSizedFilesFromDir(Dir, &CurrentSeedFiles);
++               std::sort(CurrentSeedFiles.begin(), CurrentSeedFiles.end());
++               if (CorpusDirs.empty())
++                   MkDir(Env.MainCorpusDir = DirPlusFile(Env.TempDir, "C"));
++               else
++                   Env.MainCorpusDir = CorpusDirs[0];
++
++               auto CFPath = DirPlusFile(Env.TempDir, "merge.txt");
++            Set<uint32_t> TmpNewFeatures, TmpNewCov;
++            Set<uint32_t> TmpFeatures, TmpCov;
++            Env.Files.clear();
++            Env.FilesSizes.clear();
++            CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files, TmpFeatures,
++                                  &TmpNewFeatures, TmpCov, &TmpNewCov, CFPath, false);
++               for (auto &path : Env.Files) {
++                Env.FilesSizes.push_back(FileSize(path));
++               }
++               RemoveFile(CFPath);
++               JobExecuted = 0;
++               MergeCycle += 5;
++ }    
++    }
+@@ -402 +481,20 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-    FuzzQ.Push(Env.CreateNewJob(JobId++));
++    // Since the number of corpus seeds will gradually increase, in order to
++    // control the number in each group to be about three times the number of
++    // seeds selected each time, the number of groups is dynamically adjusted.
++    if (Env.Files.size() >= 1 && Env.Files.size() < 1600)
++     NumCorpuses = 16;
++    if (Env.Files.size() >= 1600 && Env.Files.size() < 3600)
++     NumCorpuses = 20;
++    if (Env.Files.size() >= 3600 && Env.Files.size() < 6400)
++     NumCorpuses = 24;
++    if (Env.Files.size() >= 6400 && Env.Files.size() < 8100)
++     NumCorpuses = 32;
++    if (Env.Files.size() >= 8100 && Env.Files.size() < 12000)
++     NumCorpuses = 36;
++    if (Env.Files.size() >= 12000 && Env.Files.size() < 16000)
++     NumCorpuses = 40;
++    if (Env.Files.size() >= 16000 && Env.Files.size() < 30000)
++     NumCorpuses = 60;
++    if(Env.Files.size() >= 30000)
++     NumCorpuses = 80;
++    FuzzQ.Push(Env.CreateNewJob(JobId++, NumCorpuses, Group));
+@@ -404 +501,0 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-
+diff --git a/compiler-rt/lib/fuzzer/FuzzerFork.h b/compiler-rt/lib/fuzzer/FuzzerFork.h
+index b29a43e13fbc..0b24baf5222a 100644
+--- a/compiler-rt/lib/fuzzer/FuzzerFork.h
++++ b/compiler-rt/lib/fuzzer/FuzzerFork.h
+@@ -21 +21,2 @@ void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
+-                  const Vector<std::string> &CorpusDirs, int NumJobs);
++                  const Vector<std::string> &CorpusDirs, int NumJobs,
++     int Group);

compiler-rt/lib/fuzzer/FuzzerDriver.cpp

Printf("***\n"
            "*** NOTE: fuzzing was not performed, you have only\n"
            "***       executed the target code on a fixed set of inputs.\n"
            "***\n");
     F->PrintFinalStats();
     exit(0);
   }
 
   if (Flags.fork)
-    FuzzWithFork(F->GetMD().GetRand(), Options, Args, *Inputs, Flags.fork);
+    FuzzWithFork(F->GetMD().GetRand(), Options, Args, *Inputs, Flags.fork,
+                 Flags.group);
 
   if (Flags.merge)
     Merge(F, Options, Args, *Inputs, Flags.merge_control_file);
 
   if (Flags.merge_inner) {
     const size_t kDefaultMaxMergeLen = 1 << 20;
     if (Options.MaxLen == 0)
       F->SetMaxInputLen(kDefaultMaxMergeLen);

compiler-rt/lib/fuzzer/FuzzerFlags.def

FUZZER_FLAG_INT(
     "If one unit runs more than this number of seconds the process will abort.")
 FUZZER_FLAG_INT(error_exitcode, 77, "When libFuzzer itself reports a bug "
   "this exit code will be used.")
 FUZZER_FLAG_INT(timeout_exitcode, 70, "When libFuzzer reports a timeout "
   "this exit code will be used.")
 FUZZER_FLAG_INT(max_total_time, 0, "If positive, indicates the maximal total "
                                    "time in seconds to run the fuzzer.")
 FUZZER_FLAG_INT(help, 0, "Print help.")
 FUZZER_FLAG_INT(fork, 0, "Experimental mode where fuzzing happens "
                 "in a subprocess")
+FUZZER_FLAG_INT(group, 1, "For fork mode. Devide the main corpus into N groups "
+               " according to size.")
 FUZZER_FLAG_INT(ignore_timeouts, 1, "Ignore timeouts in fork mode")
 FUZZER_FLAG_INT(ignore_ooms, 1, "Ignore OOMs in fork mode")
 FUZZER_FLAG_INT(ignore_crashes, 0, "Ignore crashes in fork mode")
 FUZZER_FLAG_INT(merge, 0, "If 1, the 2-nd, 3-rd, etc corpora will be "
   "merged into the 1-st corpus. Only interesting units will be taken. "
   "This flag can be used to minimize a corpus.")
 FUZZER_FLAG_STRING(stop_file, "Stop fuzzing ASAP if this file exists")
 FUZZER_FLAG_STRING(merge_inner, "internal flag")

compiler-rt/lib/fuzzer/FuzzerFork.h

 #include "FuzzerOptions.h"
 #include "FuzzerRandom.h"
 
 #include <string>
 
 namespace fuzzer {
 void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
                   const std::vector<std::string> &Args,
-                  const std::vector<std::string> &CorpusDirs, int NumJobs);
+                  const std::vector<std::string> &CorpusDirs, int NumJobs,
+                  int Group);
 } // namespace fuzzer
 
 #endif // LLVM_FUZZER_FORK_H

compiler-rt/lib/fuzzer/FuzzerFork.cpp

struct GlobalEnv {
   std::vector<std::string> CorpusDirs;
   std::string MainCorpusDir;
   std::string TempDir;
   std::string DFTDir;
   std::string DataFlowBinary;
   std::set<uint32_t> Features, Cov;
   std::set<std::string> FilesWithDFT;
   std::vector<std::string> Files;
+  std::vector<std::size_t> FilesSizes;
   Random *Rand;
   std::chrono::system_clock::time_point ProcessStartTime;
   int Verbosity = 0;
 
   size_t NumTimeouts = 0;
   size_t NumOOMs = 0;
   size_t NumCrashes = 0;
 
 
   size_t NumRuns = 0;
 
   std::string StopFile() { return DirPlusFile(TempDir, "STOP"); }
 
   size_t secondsSinceProcessStartUp() const {
     return std::chrono::duration_cast<std::chrono::seconds>(
                std::chrono::system_clock::now() - ProcessStartTime)
         .count();
   }
 
-  FuzzJob *CreateNewJob(size_t JobId) {
+  FuzzJob *CreateNewJob(size_t JobId, int NumCorpuses, int Group) {
     Command Cmd(Args);
     Cmd.removeFlag("fork");
     Cmd.removeFlag("runs");
     Cmd.removeFlag("collect_data_flow");
     for (auto &C : CorpusDirs) // Remove all corpora from the args.
       Cmd.removeArgument(C);
     Cmd.addFlag("reload", "0");  // working in an isolated dir, no reload.
     Cmd.addFlag("print_final_stats", "1");
     Cmd.addFlag("print_funcs", "0");  // no need to spend time symbolizing.
     Cmd.addFlag("max_total_time", std::to_string(std::min((size_t)300, JobId)));
     Cmd.addFlag("stop_file", StopFile());
     if (!DataFlowBinary.empty()) {
       Cmd.addFlag("data_flow_trace", DFTDir);
       if (!Cmd.hasFlag("focus_function"))
         Cmd.addFlag("focus_function", "auto");
     }
     auto Job = new FuzzJob;
     std::string Seeds;
     if (size_t CorpusSubsetSize =
             std::min(Files.size(), (size_t)sqrt(Files.size() + 2))) {
       auto Time1 = std::chrono::system_clock::now();
+      if (Group) { // flag: whether to group the corpus.
+        size_t AverageCorpusSize = Files.size() / NumCorpuses + 1;
+        if (Files.size() == 0)
+          AverageCorpusSize = 0;
+        size_t StartIndex = ((JobId - 1) % NumCorpuses) * AverageCorpusSize;
         for (size_t i = 0; i < CorpusSubsetSize; i++) {
+          std::random_device rd;
+          std::mt19937 RandomSeed(rd());
+          std::uniform_int_distribution<> rand(0, AverageCorpusSize);
+          size_t RandNum = rand(RandomSeed);
+          size_t Index = RandNum + StartIndex;
+          if (Index < Files.size()) {
+            auto &SF = Files[Index];
+            Seeds += (Seeds.empty() ? "" : ",") + SF;
+            CollectDFT(SF);
+          } else {
             auto &SF = Files[Rand->SkewTowardsLast(Files.size())];
             Seeds += (Seeds.empty() ? "" : ",") + SF;
             CollectDFT(SF);
           }
+        }
+      } else {
+        for (size_t i = 0; i < CorpusSubsetSize; i++) {
+          auto &SF = Files[Rand->SkewTowardsLast(Files.size())];
+          Seeds += (Seeds.empty() ? "" : ",") + SF;
+          CollectDFT(SF);
+        }
+      }
       auto Time2 = std::chrono::system_clock::now();
       auto DftTimeInSeconds = duration_cast<seconds>(Time2 - Time1).count();
       assert(DftTimeInSeconds < std::numeric_limits<int>::max());
       Job->DftTimeInSeconds = static_cast<int>(DftTimeInSeconds);
     }
     if (!Seeds.empty()) {
       Job->SeedListPath =
           DirPlusFile(TempDir, std::to_string(JobId) + ".seeds");
       WriteToFile(Seeds, Job->SeedListPath);
       Cmd.addFlag("seed_inputs", "@" + Job->SeedListPath);
FuzzJob *CreateNewJob(size_t JobId, int NumCorpuses, int Group) {
 
     if (Verbosity >= 2)
       Printf("Job %zd/%p Created: %s\n", JobId, Job,
              Job->Cmd.toString().c_str());
     // Start from very short runs and gradually increase them.
     return Job;
   }
 
-  void RunOneMergeJob(FuzzJob *Job) {
+  void RunOneMergeJob(FuzzJob *Job, int Group) {
     auto Stats = ParseFinalStatsFromLog(Job->LogPath);
     NumRuns += Stats.number_of_executed_units;
 
     std::vector<SizedFile> TempFiles, MergeCandidates;
     // Read all newly created inputs and their feature sets.
     // Choose only those inputs that have new features.
     GetSizedFilesFromDir(Job->CorpusDir, &TempFiles);
     std::sort(TempFiles.begin(), TempFiles.end());
     for (auto &F : TempFiles) {
       auto FeatureFile = F.File;
       FeatureFile.replace(0, Job->CorpusDir.size(), Job->FeaturesDir);
       auto FeatureBytes = FileToVector(FeatureFile, 0, false);
       assert((FeatureBytes.size() % sizeof(uint32_t)) == 0);
       std::vector<uint32_t> NewFeatures(FeatureBytes.size() / sizeof(uint32_t));
       memcpy(NewFeatures.data(), FeatureBytes.data(), FeatureBytes.size());
       for (auto Ft : NewFeatures) {
         if (!Features.count(Ft)) {
           MergeCandidates.push_back(F);
           break;
         }
       }
     }
void RunOneMergeJob(FuzzJob *Job, int Group) {
     std::vector<std::string> FilesToAdd;
     std::set<uint32_t> NewFeatures, NewCov;
     CrashResistantMerge(Args, {}, MergeCandidates, &FilesToAdd, Features,
                         &NewFeatures, Cov, &NewCov, Job->CFPath, false);
     for (auto &Path : FilesToAdd) {
       auto U = FileToVector(Path);
       auto NewPath = DirPlusFile(MainCorpusDir, Hash(U));
       WriteToFile(U, NewPath);
+      if (Group) { // Insert the queue according to the size of the seed.
+        long usz = U.size();
+        auto idx = std::upper_bound(FilesSizes.begin(), FilesSizes.end(), usz) -
+                   FilesSizes.begin();
+        FilesSizes.insert(FilesSizes.begin() + idx, usz);
+        Files.insert(Files.begin() + idx, NewPath);
+      } else {
         Files.push_back(NewPath);
       }
+    }
     Features.insert(NewFeatures.begin(), NewFeatures.end());
     Cov.insert(NewCov.begin(), NewCov.end());
     for (auto Idx : NewCov)
       if (auto *TE = TPC.PCTableEntryByIdx(Idx))
         if (TPC.PcIsFuncEntry(TE))
           PrintPC("  NEW_FUNC: %p %F %L\n", "",
                   TPC.GetNextInstructionPc(TE->PC));
-
   }
 
-
   void CollectDFT(const std::string &InputPath) {
     if (DataFlowBinary.empty()) return;
     if (!FilesWithDFT.insert(InputPath).second) return;
     Command Cmd(Args);
     Cmd.removeFlag("fork");
     Cmd.removeFlag("runs");
     Cmd.addFlag("data_flow_trace", DFTDir);
     Cmd.addArgument(InputPath);
     for (auto &C : CorpusDirs) // Remove all corpora from the args.
       Cmd.removeArgument(C);
     Cmd.setOutputFile(DirPlusFile(TempDir, "dft.log"));
     Cmd.combineOutAndErr();
     // Printf("CollectDFT: %s\n", Cmd.toString().c_str());
     ExecuteCommand(Cmd);
   }
while (auto Job = FuzzQ->Pop()) {
     Job->ExitCode = ExecuteCommand(Job->Cmd);
     MergeQ->Push(Job);
   }
 }
 
 // This is just a skeleton of an experimental -fork=1 feature.
 void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
                   const std::vector<std::string> &Args,
-                  const std::vector<std::string> &CorpusDirs, int NumJobs) {
+                  const std::vector<std::string> &CorpusDirs, int NumJobs,
+                  int Group) {
   Printf("INFO: -fork=%d: fuzzing in separate process(s)\n", NumJobs);
 
   GlobalEnv Env;
   Env.Args = Args;
   Env.CorpusDirs = CorpusDirs;
   Env.Rand = &Rand;
   Env.Verbosity = Options.Verbosity;
   Env.ProcessStartTime = std::chrono::system_clock::now();
void FuzzWithFork(Random &Rand, const FuzzingOptions &Options,
 
 
   if (CorpusDirs.empty())
     MkDir(Env.MainCorpusDir = DirPlusFile(Env.TempDir, "C"));
   else
     Env.MainCorpusDir = CorpusDirs[0];
 
   if (Options.KeepSeed) {
-    for (auto &File : SeedFiles)
+    for (auto &File : SeedFiles) {
       Env.Files.push_back(File.File);
+      Env.FilesSizes.push_back(File.Size);
+    }
   } else {
     auto CFPath = DirPlusFile(Env.TempDir, "merge.txt");
     std::set<uint32_t> NewFeatures, NewCov;
     CrashResistantMerge(Env.Args, {}, SeedFiles, &Env.Files, Env.Features,
                         &NewFeatures, Env.Cov, &NewCov, CFPath, false);
     Env.Features.insert(NewFeatures.begin(), NewFeatures.end());
     Env.Cov.insert(NewFeatures.begin(), NewFeatures.end());
     RemoveFile(CFPath);
   }
+
+  if (Group) {
+    for (auto &path : Env.Files) {
+      Env.FilesSizes.push_back(FileSize(path));
+    }
+  }
+
   Printf("INFO: -fork=%d: %zd seed inputs, starting to fuzz in %s\n", NumJobs,
          Env.Files.size(), Env.TempDir.c_str());
 
   int ExitCode = 0;
 
   JobQueue FuzzQ, MergeQ;
 
   auto StopJobs = [&]() {
     for (int i = 0; i < NumJobs; i++)
       FuzzQ.Push(nullptr);
     MergeQ.Push(nullptr);
     WriteToFile(Unit({1}), Env.StopFile());
   };
 
+  size_t MergeCycle = 20;
+  size_t JobExecuted = 0;
+  int NumCorpuses = 8;
   size_t JobId = 1;
   std::vector<std::thread> Threads;
   for (int t = 0; t < NumJobs; t++) {
     Threads.push_back(std::thread(WorkerThread, &FuzzQ, &MergeQ));
-    FuzzQ.Push(Env.CreateNewJob(JobId++));
+    FuzzQ.Push(Env.CreateNewJob(JobId++, NumCorpuses, Group));
   }
 
   while (true) {
     std::unique_ptr<FuzzJob> Job(MergeQ.Pop());
     if (!Job)
       break;
     ExitCode = Job->ExitCode;
     if (ExitCode == Options.InterruptExitCode) {
       Printf("==%lu== libFuzzer: a child was interrupted; exiting\n", GetPid());
       StopJobs();
       break;
     }
     Fuzzer::MaybeExitGracefully();
 
-    Env.RunOneMergeJob(Job.get());
+    Env.RunOneMergeJob(Job.get(), Group);
+
+    // merge the corpus .
+    JobExecuted++;
+    if (Group) {
+      if (JobExecuted >= MergeCycle) {
+        std::vector<SizedFile> CurrentSeedFiles;
+        for (auto &Dir : CorpusDirs)
+          GetSizedFilesFromDir(Dir, &CurrentSeedFiles);
+        std::sort(CurrentSeedFiles.begin(), CurrentSeedFiles.end());
+        if (CorpusDirs.empty())
+          MkDir(Env.MainCorpusDir = DirPlusFile(Env.TempDir, "C"));
+        else
+          Env.MainCorpusDir = CorpusDirs[0];
+
+        auto CFPath = DirPlusFile(Env.TempDir, "merge.txt");
+        std::set<uint32_t> TmpNewFeatures, TmpNewCov;
+        std::set<uint32_t> TmpFeatures, TmpCov;
+        Env.Files.clear();
+        Env.FilesSizes.clear();
+        CrashResistantMerge(Env.Args, {}, CurrentSeedFiles, &Env.Files,
+                            TmpFeatures, &TmpNewFeatures, TmpCov, &TmpNewCov,
+                            CFPath, false);
+        for (auto &path : Env.Files) {
+          Env.FilesSizes.push_back(FileSize(path));
+        }
+        RemoveFile(CFPath);
+        JobExecuted = 0;
+        MergeCycle += 5;
+      }
+    }
+
+    // Since the number of corpus seeds will gradually increase, in order to
+    // control the number in each group to be about three times the number of
+    // seeds selected each time, the number of groups is dynamically adjusted.
+    if (Env.Files.size() >= 1 && Env.Files.size() < 1600)
+      NumCorpuses = 16;
+    if (Env.Files.size() >= 1600 && Env.Files.size() < 3600)
+      NumCorpuses = 20;
+    if (Env.Files.size() >= 3600 && Env.Files.size() < 6400)
+      NumCorpuses = 24;
+    if (Env.Files.size() >= 6400 && Env.Files.size() < 8100)
+      NumCorpuses = 32;
+    if (Env.Files.size() >= 8100 && Env.Files.size() < 12000)
+      NumCorpuses = 36;
+    if (Env.Files.size() >= 12000 && Env.Files.size() < 16000)
+      NumCorpuses = 40;
+    if (Env.Files.size() >= 16000 && Env.Files.size() < 30000)
+      NumCorpuses = 60;
+    if (Env.Files.size() >= 30000)
+      NumCorpuses = 80;
 
     // Continue if our crash is one of the ignorred ones.
     if (Options.IgnoreTimeouts && ExitCode == Options.TimeoutExitCode)
       Env.NumTimeouts++;
     else if (Options.IgnoreOOMs && ExitCode == Options.OOMExitCode)
       Env.NumOOMs++;
     else if (ExitCode != 0) {
       Env.NumCrashes++;
       if (Options.IgnoreCrashes) {
         std::ifstream In(Job->LogPath);
         std::string Line;
         while (std::getline(In, Line, '\n'))
           if (Line.find("ERROR:") != Line.npos ||
               Line.find("runtime error:") != Line.npos)
             Printf("%s\n", Line.c_str());
       } else {
         // And exit if we don't ignore this crash.
         Printf("INFO: log from the inner process:\n%s",
                FileToString(Job->LogPath).c_str());
         StopJobs();
         break;
       }
     }
 
     // Stop if we are over the time budget.
     // This is not precise, since other threads are still running
     // and we will wait while joining them.
     // We also don't stop instantly: other jobs need to finish.
     if (Options.MaxTotalTimeSec > 0 &&
         Env.secondsSinceProcessStartUp() >= (size_t)Options.MaxTotalTimeSec) {
       Printf("INFO: fuzzed for %zd seconds, wrapping up soon\n",
              Env.secondsSinceProcessStartUp());
       StopJobs();
       break;
     }
     if (Env.NumRuns >= Options.MaxNumberOfRuns) {
       Printf("INFO: fuzzed for %zd iterations, wrapping up soon\n",
              Env.NumRuns);
       StopJobs();
       break;
     }
 
-    FuzzQ.Push(Env.CreateNewJob(JobId++));
+    FuzzQ.Push(Env.CreateNewJob(JobId++, NumCorpuses, Group));
   }
 
   for (auto &T : Threads)
     T.join();
 
   // The workers have terminated. Don't try to remove the directory before they
   // terminate to avoid a race condition preventing cleanup on Windows.
   RmDirRecursive(Env.TempDir);
 
   // Use the exit code from the last child process.
   Printf("INFO: exiting: %d time: %zds\n", ExitCode,
          Env.secondsSinceProcessStartUp());
   exit(ExitCode);
 }
 
 } // namespace fuzzer

compiler-rt/lib/fuzzer/test

+Subproject commit 130ccbc909edb9f0e7026bdfeb8a6eb4b56ef60b