[libFuzzer] Don't use same seed for all parallel jobs

[vitalybuka]

Summary

If booth -seed and -jobs are set we could set deterministic but unique seed for
each jobs by combining input seed with a job number.

[kcc]

rationale?

(BTW, I was going to delete this test, it's flaky and not very useful)

[vitalybuka]

Rationlaly it exactly the same a for -seed. It's used to make execution deterministic (e.g. for debugging).
Currently it is deterministic, but -seed=1 and -jobs=24 pointlessly runs 24 exactly same fuzzers.

I don't care much about the test, I can undo this part, but

[kcc]

You can not have deterministic execution with -jobs=N when running on the same corpus because the corpus is reloaded (i.e. you have concurrent execution).
If you want to run N deterministic executions in parallel, simply do a shell loop.

[vitalybuka]

if you don't provide corpus, it's determenistic
i am using this just for benchmarks and just to look at coverage

[vitalybuka]

Behavior with change makes more sense to me.
It's not something that make code complicated.

[kcc]

imho this is a completely unnecessary complication.
To get what you need simply do this:

for((i=1;i<=20;i++)); do ./fuzz -seed=$i & done; wait

[vitalybuka]

Then why do we need -jobs, it can be also emulated with some shell script

[kcc]

it's not easy to emulate -jobs in a script, because -jobs=N works together with -workers=M and if N > M new jos start after some old die.

[vitalybuka]

In D30684#694594, @kcc wrote:

it's not easy to emulate -jobs in a script, because -jobs=N works together with -workers=M and if N > M new jos start after some old die.

And user has to go this inconvenient way if jobs and seed both are desired.

I agree that
convenience(jobs+seed) < convenience(jobs)
complexity(jobs) < convenience(jobs)
but also
complexity(jobs+seed) < complexity(jobs)

It's still not clear to my why complexity(jobs+seed) > convenience(jobs+seed)

[kcc]

In D30684#694651, @vitalybuka wrote:

In D30684#694594, @kcc wrote:

it's not easy to emulate -jobs in a script, because -jobs=N works together with -workers=M and if N > M new jos start after some old die.

And user has to go this inconvenient way if jobs and seed both are desired.

You are the first user to request this :)
And you are also not really a user, but a developer :)

I agree that
convenience(jobs+seed) < convenience(jobs)
complexity(jobs) < convenience(jobs)
but also
complexity(jobs+seed) < complexity(jobs)

It's still not clear to my why complexity(jobs+seed) > convenience(jobs+seed)

With you change, you will also need to document the non-intuitive behavior.
In all such cases it's easier to not have non-intuitive behavior

lib/Fuzzer/FuzzerDriver.cpp

static void PulseThread() {
   }
 }
 
 static void WorkerThread(const std::string &Cmd, std::atomic<unsigned> *Counter,
                          unsigned NumJobs, std::atomic<bool> *HasErrors) {
   while (true) {
     unsigned C = (*Counter)++;
     if (C >= NumJobs) break;
+    std::string Seed;
+    if (Flags.seed)
+      Seed = "-seed=" + std::to_string(C + Flags.seed);
     std::string Log = "fuzz-" + std::to_string(C) + ".log";
-    std::string ToRun = Cmd + " > " + Log + " 2>&1\n";
+    std::string ToRun = Cmd + " " + Seed + " > " + Log + " 2>&1\n";
     if (Flags.verbosity)
       Printf("%s", ToRun.c_str());
     int ExitCode = ExecuteCommand(ToRun);
     if (ExitCode != 0)
       *HasErrors = true;
     std::lock_guard<std::mutex> Lock(Mu);
     Printf("================== Job %u exited with exit code %d ============\n",
            C, ExitCode);

lib/Fuzzer/test/fuzzer-jobs.test

 REQUIRES: posix
 
 RUN: rm -rf %tmp
 RUN: mkdir %tmp && cd %tmp
 # Create a shared corpus directory
 RUN: rm -rf FuzzerJobsTestCORPUS
 RUN: mkdir FuzzerJobsTestCORPUS
 RUN: rm -f fuzz-{0,1}.log
 # Start fuzzer and in parallel check that the output files
 # that should be created exist.
-RUN: LLVMFuzzer-EmptyTest -max_total_time=4 -jobs=2 -workers=2 FuzzerJobsTestCORPUS > %t-fuzzer-jobs-test.log 2>&1 & export FUZZER_PID=$!
+RUN: LLVMFuzzer-EmptyTest -seed=127 -max_total_time=4 -jobs=2 -workers=2 FuzzerJobsTestCORPUS > %t-fuzzer-jobs-test.log 2>&1 & export FUZZER_PID=$!
 # Wait a short while to give time for the child processes
 # to start fuzzing
 RUN: sleep 2
 # If the instances are running in parallel they should have created their log
 # files by now.
 RUN: ls fuzz-0.log
 RUN: ls fuzz-1.log
 # Wait for libfuzzer to finish.
 # This probably isn't portable but we need a way to block until
 # the fuzzer is done otherwise we might remove the files while
 # they are being used.
 RUN: while kill -0 ${FUZZER_PID}; do : ; done
 RUN: rm -f fuzz-{0,1}.log
 RUN: rm -rf FuzzerJobsTestCORPUS
 RUN: FileCheck -input-file=%t-fuzzer-jobs-test.log %s
 RUN: rm %t-fuzzer-jobs-test.log
 RUN: cd ../
 
+CHECK-DAG: Seed: 127
+CHECK-DAG: Seed: 128
+
 CHECK-DAG: Job 0 exited with exit code 0
 CHECK-DAG: Job 1 exited with exit code 0