This is an archive of the discontinued LLVM Phabricator instance.

[llvm-stress] Switch to a FuzzMutate driver
Needs ReviewPublic

Authored by oakrc on Apr 20 2023, 3:55 PM.

Details

Summary

This revision of llvm-stress replaces the old module generation code with a FuzzMutate driver, which supports random module generation and mutation.

  • When invoked without any options, the new llvm-stress generates a new module.
  • Since one round of mutation does little to the module, llvm-stress defaults to 100 mutations. This behavior can be adjusted with the repeat option.
  • Optionally, the user can supply an existing module (textual IR or bitcode) to mutate from.
  • File output also switches between both textual IR and bitcode based on file extension (textual IR by default).
  • When no seed is given, llvm-stress will create one instead of using a default of 0.
  • The old size option is replaced with max-size, which limits the maximum bitcode size of the generated/mutated module.

Related: Add a mutation fuzzing mode

Diff Detail

Event Timeline

oakrc created this revision.Apr 20 2023, 3:55 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 20 2023, 3:55 PM
Herald added a subscriber: StephenFan. · View Herald Transcript
oakrc requested review of this revision.Apr 20 2023, 3:55 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 20 2023, 3:55 PM
fhahn added a comment.Apr 21 2023, 1:39 AM

I think this makes sense in general, but I think @bjope is using llvm-stress heavily IIRC, so it would be good for them to try the new version to see if there are any obvious issues.

Also, it would be good to have at least some basic tests.

bjope added a subscriber: uabelho.Apr 21 2023, 1:48 AM

I think this makes sense in general, but I think @bjope is using llvm-stress heavily IIRC, so it would be good for them to try the new version to see if there are any obvious issues.

Also, it would be good to have at least some basic tests.

I think "heavily" is a bit of an overstatement. But we do use llvm-stress as part of validation of "main" before updating our downstream branches.
I'd be happy to try using this patch, but not sure I'll find time doing it already today.

I think this makes sense in general, but I think @bjope is using llvm-stress heavily IIRC, so it would be good for them to try the new version to see if there are any obvious issues.

Also, it would be good to have at least some basic tests.

I think "heavily" is a bit of an overstatement. But we do use llvm-stress as part of validation of "main" before updating our downstream branches.
I'd be happy to try using this patch, but not sure I'll find time doing it already today.

Hi @bjope, we are developing FuzzMutate here and are very interested in how you use llvm-stress. Can you elaborate more on " validation of "main" "?

P.S> We are using FuzzMutate to validate backend of LLVM (llc)

I think this makes sense in general, but I think @bjope is using llvm-stress heavily IIRC, so it would be good for them to try the new version to see if there are any obvious issues.

Also, it would be good to have at least some basic tests.

I think "heavily" is a bit of an overstatement. But we do use llvm-stress as part of validation of "main" before updating our downstream branches.
I'd be happy to try using this patch, but not sure I'll find time doing it already today.

Hi @bjope, we are developing FuzzMutate here and are very interested in how you use llvm-stress. Can you elaborate more on " validation of "main" "?

P.S> We are using FuzzMutate to validate backend of LLVM (llc)

Ok, "validation of main" was a bit vague (and I realize that it wasn't the full story either).

I'm involved in a project where we use LLVM for an out-of-tree target. We update out downstream branches on a regular basis, by merging new commits from the llvm-project main branch (that is why I referred to "main").

Before we merge a new commit from upstream (main) we run some tests using in-tree targets on llvm-project's main branch. This is done as a kind of quick validation that the commit we want to merge isn't total crap, and one part of that validation is using llvm-stress. Partially that can be seen as a to verify that llvm-stress is working, rather than validating that for example the backend isn't broken.
More practically about what we do is that we have a script that generates a number of LLVM IR programs using llvm-stress. And then we run llc on those programs to see that there are no crashes (we enable some backend verifiers such as the machine verifier when doing these tests). So nothing fancy really. We've been doing this for a long time just to get some extra testing, and we still run those tests even if we rarely see any problems in these kinds of tests.

What I didn't mention earlier is that we also use llvm-stress as a fuzzy test program generator when validating our out-of-tree target. So that is one reason why we want to see that llvm-stress amd llc work nicely together already when running same kinds of tests on main branch with in-tree targets.
The tests for the out-of-tree target are also just simple "do-not-crash" kinds of tests, but the difference is that we use the out-of-tree target in llc.

I think these tests were more important when the out-of-tree target was more immature. In those early stages llvm-stress helped out with getting some code coverage for the backend. Quickly providing lots of different IR to use as input to llc.

oakrc updated this revision to Diff 515914.Apr 21 2023, 2:16 PM

Generated a new patch file based on llvm-project:main.

Peter added a comment.Apr 21 2023, 3:09 PM

I think this makes sense in general, but I think @bjope is using llvm-stress heavily IIRC, so it would be good for them to try the new version to see if there are any obvious issues.

Also, it would be good to have at least some basic tests.

I think "heavily" is a bit of an overstatement. But we do use llvm-stress as part of validation of "main" before updating our downstream branches.
I'd be happy to try using this patch, but not sure I'll find time doing it already today.

Hi @bjope, we are developing FuzzMutate here and are very interested in how you use llvm-stress. Can you elaborate more on " validation of "main" "?

P.S> We are using FuzzMutate to validate backend of LLVM (llc)

Ok, "validation of main" was a bit vague (and I realize that it wasn't the full story either).

I'm involved in a project where we use LLVM for an out-of-tree target. We update out downstream branches on a regular basis, by merging new commits from the llvm-project main branch (that is why I referred to "main").

Before we merge a new commit from upstream (main) we run some tests using in-tree targets on llvm-project's main branch. This is done as a kind of quick validation that the commit we want to merge isn't total crap, and one part of that validation is using llvm-stress. Partially that can be seen as a to verify that llvm-stress is working, rather than validating that for example the backend isn't broken.
More practically about what we do is that we have a script that generates a number of LLVM IR programs using llvm-stress. And then we run llc on those programs to see that there are no crashes (we enable some backend verifiers such as the machine verifier when doing these tests). So nothing fancy really. We've been doing this for a long time just to get some extra testing, and we still run those tests even if we rarely see any problems in these kinds of tests.

What I didn't mention earlier is that we also use llvm-stress as a fuzzy test program generator when validating our out-of-tree target. So that is one reason why we want to see that llvm-stress amd llc work nicely together already when running same kinds of tests on main branch with in-tree targets.
The tests for the out-of-tree target are also just simple "do-not-crash" kinds of tests, but the difference is that we use the out-of-tree target in llc.

I think these tests were more important when the out-of-tree target was more immature. In those early stages llvm-stress helped out with getting some code coverage for the backend. Quickly providing lots of different IR to use as input to llc.

Thanks for the explanation, that makes a lot more sense.

oakrc added a comment.Apr 21 2023, 4:09 PM

Also, it would be good to have at least some basic tests.

Hi @fhahn , I'm not sure how to make tests for a patch. Should I create some tests in llvm/unittests/tools/llvm-stress?

bjope added a comment.Apr 24 2023, 2:35 AM

One thing I've noticed when testing this is that I do not get the same program every time, even when using the same seed.

So for example running llvm-stress -seed 20379 multiple times gives lots of different programs. That seems bad.
Also, quite often it result in crashes such as:

llvm-stress -seed 20379
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: bin/llvm-stress -seed 20379
 #0 0x00000000005fb9f8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (bin/llvm-stress+0x5fb9f8)
 #1 0x00000000005f96ee llvm::sys::RunSignalHandlers() (bin/llvm-stress+0x5f96ee)
 #2 0x00000000005fc1c6 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f7fa59c5630 __restore_rt sigaction.c:0:0
 #4 0x000000000047f070 llvm::AttributeList::hasAttributeAtIndex(unsigned int, llvm::Attribute::AttrKind) const (bin/llvm-stress+0x47f070)
 #5 0x00000000005958fe llvm::RandomIRBuilder::connectToSink(llvm::BasicBlock&, llvm::ArrayRef<llvm::Instruction*>, llvm::Value*)::$_2::operator()(llvm::ArrayRef<llvm::Instruction*>) const RandomIRBuilder.cpp:0:0
 #6 0x0000000000594dab llvm::RandomIRBuilder::connectToSink(llvm::BasicBlock&, llvm::ArrayRef<llvm::Instruction*>, llvm::Value*) (bin/llvm-stress+0x594dab)
 #7 0x000000000058449e llvm::InjectorIRStrategy::mutate(llvm::BasicBlock&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x58449e)
 #8 0x00000000005833bb llvm::IRMutationStrategy::mutate(llvm::Function&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x5833bb)
 #9 0x0000000000583999 llvm::InjectorIRStrategy::mutate(llvm::Function&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x583999)
#10 0x00000000005832a6 llvm::IRMutationStrategy::mutate(llvm::Module&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x5832a6)
#11 0x00000000005838c8 llvm::IRMutator::mutateModule(llvm::Module&, int, unsigned long, unsigned long) (bin/llvm-stress+0x5838c8)
#12 0x000000000040e820 main (bin/llvm-stress+0x40e820)
#13 0x00007f7fa30f8555 __libc_start_main (/lib64/libc.so.6+0x22555)
#14 0x000000000040c86b _start (bin/llvm-stress+0x40c86b)
Segmentation fault (core dumped)

One thing I've noticed when testing this is that I do not get the same program every time, even when using the same seed.

So for example running llvm-stress -seed 20379 multiple times gives lots of different programs. That seems bad.
Also, quite often it result in crashes such as:

llvm-stress -seed 20379
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: bin/llvm-stress -seed 20379
 #0 0x00000000005fb9f8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (bin/llvm-stress+0x5fb9f8)
 #1 0x00000000005f96ee llvm::sys::RunSignalHandlers() (bin/llvm-stress+0x5f96ee)
 #2 0x00000000005fc1c6 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f7fa59c5630 __restore_rt sigaction.c:0:0
 #4 0x000000000047f070 llvm::AttributeList::hasAttributeAtIndex(unsigned int, llvm::Attribute::AttrKind) const (bin/llvm-stress+0x47f070)
 #5 0x00000000005958fe llvm::RandomIRBuilder::connectToSink(llvm::BasicBlock&, llvm::ArrayRef<llvm::Instruction*>, llvm::Value*)::$_2::operator()(llvm::ArrayRef<llvm::Instruction*>) const RandomIRBuilder.cpp:0:0
 #6 0x0000000000594dab llvm::RandomIRBuilder::connectToSink(llvm::BasicBlock&, llvm::ArrayRef<llvm::Instruction*>, llvm::Value*) (bin/llvm-stress+0x594dab)
 #7 0x000000000058449e llvm::InjectorIRStrategy::mutate(llvm::BasicBlock&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x58449e)
 #8 0x00000000005833bb llvm::IRMutationStrategy::mutate(llvm::Function&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x5833bb)
 #9 0x0000000000583999 llvm::InjectorIRStrategy::mutate(llvm::Function&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x583999)
#10 0x00000000005832a6 llvm::IRMutationStrategy::mutate(llvm::Module&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x5832a6)
#11 0x00000000005838c8 llvm::IRMutator::mutateModule(llvm::Module&, int, unsigned long, unsigned long) (bin/llvm-stress+0x5838c8)
#12 0x000000000040e820 main (bin/llvm-stress+0x40e820)
#13 0x00007f7fa30f8555 __libc_start_main (/lib64/libc.so.6+0x22555)
#14 0x000000000040c86b _start (bin/llvm-stress+0x40c86b)
Segmentation fault (core dumped)

Thanks, I also noticed some bugs in FuzzMutate since this patch, we are working on fixing it. Let us ping you when we finished debugging.

oakrc added a comment.Apr 24 2023, 3:33 PM

One thing I've noticed when testing this is that I do not get the same program every time, even when using the same seed.

So for example running llvm-stress -seed 20379 multiple times gives lots of different programs. That seems bad.
Also, quite often it result in crashes such as:

llvm-stress -seed 20379
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: bin/llvm-stress -seed 20379
 #0 0x00000000005fb9f8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (bin/llvm-stress+0x5fb9f8)
 #1 0x00000000005f96ee llvm::sys::RunSignalHandlers() (bin/llvm-stress+0x5f96ee)
 #2 0x00000000005fc1c6 SignalHandler(int) Signals.cpp:0:0
 #3 0x00007f7fa59c5630 __restore_rt sigaction.c:0:0
 #4 0x000000000047f070 llvm::AttributeList::hasAttributeAtIndex(unsigned int, llvm::Attribute::AttrKind) const (bin/llvm-stress+0x47f070)
 #5 0x00000000005958fe llvm::RandomIRBuilder::connectToSink(llvm::BasicBlock&, llvm::ArrayRef<llvm::Instruction*>, llvm::Value*)::$_2::operator()(llvm::ArrayRef<llvm::Instruction*>) const RandomIRBuilder.cpp:0:0
 #6 0x0000000000594dab llvm::RandomIRBuilder::connectToSink(llvm::BasicBlock&, llvm::ArrayRef<llvm::Instruction*>, llvm::Value*) (bin/llvm-stress+0x594dab)
 #7 0x000000000058449e llvm::InjectorIRStrategy::mutate(llvm::BasicBlock&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x58449e)
 #8 0x00000000005833bb llvm::IRMutationStrategy::mutate(llvm::Function&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x5833bb)
 #9 0x0000000000583999 llvm::InjectorIRStrategy::mutate(llvm::Function&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x583999)
#10 0x00000000005832a6 llvm::IRMutationStrategy::mutate(llvm::Module&, llvm::RandomIRBuilder&) (bin/llvm-stress+0x5832a6)
#11 0x00000000005838c8 llvm::IRMutator::mutateModule(llvm::Module&, int, unsigned long, unsigned long) (bin/llvm-stress+0x5838c8)
#12 0x000000000040e820 main (bin/llvm-stress+0x40e820)
#13 0x00007f7fa30f8555 __libc_start_main (/lib64/libc.so.6+0x22555)
#14 0x000000000040c86b _start (bin/llvm-stress+0x40c86b)
Segmentation fault (core dumped)

Thanks for letting me know. I'm working on it.

nikic resigned from this revision.Apr 26 2023, 1:13 AM
oakrc updated this revision to Diff 543235.Jul 22 2023, 4:28 PM

FuzzMutate shouldn't be causing issues for llvm-stress now.
Also changed it to use std::random_device when a seed is not specified.

oakrc updated this revision to Diff 544975.Jul 27 2023, 4:02 PM

Rebase main to get rid of the build check failures.

bjope added a comment.Aug 4 2023, 3:22 PM

Just for info:
Downstream we have some hacks (extra options) to avoid certain vector types, and also to avoid some variants of InsertElement and ExtractElement with non-constant indices.
One idea would be to add support in our backend for those things to get rid of the restrictions, or I guess we need to hack into FuzzMutate somehow.

IIUC the allowed vector types are specified in addVectorTypeGetters so at least that might not be that complicated to adjust that downstream.
I also think llvm::fuzzerop::insertElementDescriptor , llvm::fuzzerop::extractElementDescriptor is the place to edit if I want to add an option forcing the index to be a constant, right?

Not saying that our downstream adjustments really should block something like this from being committed upstream. I'm just trying to figure out how we eventually would deal with it downstream.

Peter added a comment.Aug 4 2023, 3:42 PM
  1. Yes, you can modify TypeGetter to enable or disable certain types.
  2. Yes, we use a lambda expression to describe the requirement for each operand. For example, extractElementDescriptor says that extract element needs {anyVectorType(), anyIntType()}, i.e., the second operand (index) needs to be an integer. For your use case, you can change it to something like anyConstInt.