This is an archive of the discontinued LLVM Phabricator instance.

[Docs] Add best practices for regression tests
ClosedPublic

Authored by nikic on Jan 24 2023, 1:46 AM.

Details

Summary

There are a lot of conventions for writing tests that don't seem to be documented anywhere right now, so this takes a stab at writing down some "best practices".

Diff Detail

Event Timeline

nikic created this revision.Jan 24 2023, 1:46 AM
Herald added a project: Restricted Project. · View Herald TranscriptJan 24 2023, 1:46 AM
Herald added a subscriber: StephenFan. · View Herald Transcript
nikic requested review of this revision.Jan 24 2023, 1:46 AM
Herald added a project: Restricted Project. · View Herald TranscriptJan 24 2023, 1:46 AM
nikic updated this revision to Diff 491669.Jan 24 2023, 1:47 AM

Fix typo

Thanks, this is really helpful. One question:

> Avoid unnamed instructions/blocks (such as `%0 or 1:`).
Why so? (It would be nice to clarify this in the doc.)

nikic updated this revision to Diff 491678.Jan 24 2023, 2:23 AM

Mention why not to use unnamed values.

nikic added a comment.Jan 24 2023, 2:26 AM

Thanks, this is really helpful. One question:

> Avoid unnamed instructions/blocks (such as `%0 or 1:`).
Why so? (It would be nice to clarify this in the doc.)

Added a note. Basically, if you remove an unnamed instruction, you need to renumber all following instructions. Also, generated check lines don't use wildcards for basic block names, so while check lines don't care about var names, they care about block names, which are less stable if you use unnamed blocks in the input.

I have a handy script that I use whenever I need to touch a test with unnamed instructions: https://gist.github.com/nikic/145581a29ce951c31e2650ad322359e4

barannikov88 accepted this revision.Jan 24 2023, 2:51 AM

Thanks, LGTM

This revision is now accepted and ready to land.Jan 24 2023, 2:51 AM

I could think of a number of additional notes for mir tests (although many of those are really because we have poor defaults)

llvm/docs/TestingGuide.rst
340

Maybe also mention minimal pass sets (e.g avoid using opt -O3 in most tests)

LGTM - see inline for minor edits.

Nice! I was wondering if we could adapt/copy:
https://developers.redhat.com/articles/2022/12/20/how-contribute-llvm#
...to the LLVM doc pages directly. This is a great start.

llvm/docs/TestingGuide.rst
321–325
- If the test does not crash, assert, or infinite loop, commit the test
with baseline check-lines. That is, the test will show a miscompile 
or missing optimization. Add a "TODO" or "FIXME" comment to
indicate that something is expected to change in a test. 

- A follow-up patch with code changes to the compiler will then 
show check-line differences to the tests, so it is easier to see the 
effect of the patch. Remove TODO/FIXME comments added in the
previous step if a problem is solved.

- Baseline tests (no-functional-change or NFC patch) may be pushed 
to main without pre-commit review if you have commit access 
(link to getting started page?).
328–329

No need to repeat clause about regression tests - this section only applies to those.

- Include comments about what is tested/expected in a particular test. If there are
relevant issues in the bug tracker add references to those bug reports (for example, 
"See PR999 for more details").
330–332
- Avoid undefined behavior or values unless necessary.
338–339

Remove "SSA" - this is true in general for source-level tests too?

nikic updated this revision to Diff 491793.Jan 24 2023, 7:38 AM

Incorporate review feedback. I've split off a separate section for the precommit workflow, as it's longer than a single bullet point.

This revision was landed with ongoing or failed builds.Jan 26 2023, 5:28 AM
This revision was automatically updated to reflect the committed changes.