[libFuzzer] [Tests] [NFC] Change seed for reduce_inputs.test

[george.karpenkov]

Summary

On one of our bot machines, -seed=1 gets unlucky and does not finish in time.
While it makes for a great question why (from what I gather, std::mt19937 should be identical on both platforms) for now I would just like to keep the bots green.

[george.karpenkov]

@kcc @morehouse Any ideas on what could cause discrepancies between platforms? I've manually checked the output of the random generator with the same seed on all machines in question, and it's the same. libFuzzer performs it's own hash computation, so that should be stable as well. Yet we have a machine which behaves differently with this seed.

I'm also surprised how does the first line run, since it's not seeded at all -- shouldn't it not find the input every now and then?

[morehouse]

Does that machine build libFuzzer differently? Maybe the version of libc++ it uses is different.

[george.karpenkov]

Maybe the version of libc++ it uses is different

Yeah that's what I've thought, but I've tried using the prime generator directly with the freshly built clang on that machine, and got the same sequence as elsewhere.
They all use fresh libcxx.

[kcc]

oh, anything could be different. I wouldn't expect seed=1 to behave the same on different platforms.
Need to check why this test is flaky (too few iterations?)

[george.karpenkov]

@kcc Any sources of non-determinism you suspect? 10^6 iterations already take ~20 seconds, would be hesitant to bump it more.

Also it consistently finds the issue if (even empty) corpus is supplied. That's expected due to the limit of corpus data which would be stored in RAM?

[kcc]

Any sources of non-determinism you suspect?

RNG is expected to provide the same values, but if e.g. the code is compiled slightly differently because
the system headers are different, then the coverage feedback will be different and hence the RNG will be called in different order.

This is very strange.
I've just ran the test manually for a few times with different seeds and it always finds the bug quickly.

clang -g -std=c++11 -fsanitize=address,fuzzer ~/llvm/projects/compiler-rt/test/fuzzer/ShrinkControlFlowSimpleTest.cpp
./a.out -exit_on_item=0eb8e4ed029b774d80f2b66408203801cb982a60 -runs=1000000 -jobs=100

[kcc]

also:

>> 10^6 iterations already take ~20 seconds, would be hesitant to bump it more.

what?

for me this test completes in < 1 second even if I let it execute 10^6 iterations.

[george.karpenkov]

@kcc

I've just ran the test manually for a few times with different seeds and it always finds the bug quickly.

That's the simple one, the problematic one for me is ShrinkControlFlowTest.

[george.karpenkov]

@kcc Are you saying the behavior should be the same whether or not the (empty) coverage dir is used?

[george.karpenkov]

@kcc

for me this test completes in < 1 second even if I let it execute 10^6 iterations.

For me it runs in 1 second on my machine, in 12 seconds on the bot in question. Weird.

[kcc]

same with ShrinkControlFlowTest. I've run it 10000 times.

Are you saying the behavior should be the same whether or not the (empty) coverage dir is used?

Hm... I would expect so...
But I don't think we provide any kind of hard guarantee for that.

For me it runs in 1 second on my machine, in 12 seconds on the bot in question. Weird.

The bot might be hugely overloaded, which in turn could be causing the troubles with the SIGUSR tests.

[george.karpenkov]

@kcc @morehouse
OK seems I have figured out the issue, the bot was running an earlier branch of LLVM, which did not have the OPT_FOR_FUZZING LLVM attribute,
which allowed efficient fuzzing under -O2. Seems like most problems went from there.

[morehouse]

Does the bot config also explain the SIGUSR test flakiness?

[george.karpenkov]

@morehouse Probably, though test is still inherently flaky.

compiler-rt/trunk/test/fuzzer/reduce_inputs.test

 # Test -reduce_inputs=1
 
 RUN: rm -rf %t/C
 RUN: mkdir -p %t/C
 RUN: %cpp_compiler %S/ShrinkControlFlowSimpleTest.cpp -o %t-ShrinkControlFlowSimpleTest
 RUN: %cpp_compiler %S/ShrinkControlFlowTest.cpp -o %t-ShrinkControlFlowTest
 RUN: %run %t-ShrinkControlFlowSimpleTest  -exit_on_item=0eb8e4ed029b774d80f2b66408203801cb982a60   -runs=1000000 %t/C 2>&1 | FileCheck %s
 CHECK: INFO: found item with checksum '0eb8e4ed029b774d80f2b66408203801cb982a60'
 
 # Test that reduce_inputs deletes redundant files in the corpus.
 RUN: %run %t-ShrinkControlFlowSimpleTest -runs=0 %t/C 2>&1 | FileCheck %s --check-prefix=COUNT
 COUNT: seed corpus: files: 4
 
 # a bit longer test
-RUN: %run %t-ShrinkControlFlowTest  -exit_on_item=0eb8e4ed029b774d80f2b66408203801cb982a60  -seed=1 -runs=1000000  2>&1 | FileCheck %s
+RUN: %run %t-ShrinkControlFlowTest  -exit_on_item=0eb8e4ed029b774d80f2b66408203801cb982a60  -seed=42 -runs=1000000  2>&1 | FileCheck %s