This is an archive of the discontinued LLVM Phabricator instance.

Differential D154383

[UTC] Add do-not-autogenerate capability
ClosedPublic

Authored by probinson on Jul 3 2023, 11:44 AM.

Details

Reviewers
eli.friedman
efriedma
Commits
rGc2bed2a1703a: [UTC] Add do-not-autogenerate capability
Summary

A file with manual checks that should be preserved can now be
protected against accidental updates by a script.

Diff Detail

Event Timeline

probinson created this revision.Jul 3 2023, 11:44 AM
Herald added a project: Restricted Project. · View Herald TranscriptJul 3 2023, 11:44 AM
Herald added a subscriber: arichardson. · View Herald Transcript
probinson requested review of this revision.Jul 3 2023, 11:44 AM
Herald added a project: Restricted Project. · View Herald TranscriptJul 3 2023, 11:44 AM
Herald added a subscriber: llvm-commits. · View Herald Transcript
probinson added a comment.Jul 3 2023, 11:47 AM

If there is someplace where tests and/or documentation live that should be updated, let me know, a casual look didn't turn up anything.

efriedma added a subscriber: efriedma.Jul 3 2023, 12:04 PM

We have a small section in the documentation describing generated assertions: https://llvm.org/docs/TestingGuide.html#generating-assertions-in-regression-tests . If you want to try to expand it a bit, that could be useful.

llvm/utils/UpdateTestChecks/common.py
473–475

Maybe delete the trailing space?

Harbormaster completed remote builds in B242866: Diff 536856.Jul 3 2023, 2:38 PM
probinson updated this revision to Diff 537752.Jul 6 2023, 8:54 AM

Update docs; remove trailing space from the key string.

probinson marked an inline comment as done.Jul 6 2023, 8:54 AM
efriedma accepted this revision.Jul 6 2023, 9:48 AM

LGTM

This revision is now accepted and ready to land.Jul 6 2023, 9:48 AM
Harbormaster completed remote builds in B243488: Diff 537752.Jul 6 2023, 9:53 AM
Closed by commit rGc2bed2a1703a: [UTC] Add do-not-autogenerate capability (authored by probinson). · Explain WhyJul 6 2023, 10:56 AM
This revision was automatically updated to reflect the committed changes.
probinson added a commit: rGc2bed2a1703a: [UTC] Add do-not-autogenerate capability.
jdoerfert added a subscriber: jdoerfert.Jul 14 2023, 12:20 PM

cool!

Revision Contents

PathSize
llvm/
docs/
15 lines
utils/
UpdateTestChecks/
4 lines

Diff 537795

llvm/docs/TestingGuide.rst

Show First 20 Lines • Show All 277 Lines • ▼ Show 20 Lines
Generating assertions in regression tests Generating assertions in regression tests
----------------------------------------- -----------------------------------------
Some regression test cases are very large and complex to write/update by hand. Some regression test cases are very large and complex to write/update by hand.
In that case to reduce the human work we can use the scripts available in In that case to reduce the human work we can use the scripts available in
llvm/utils/ to generate the assertions. llvm/utils/ to generate the assertions.
For example to generate assertions in an :program:`llc`-based test, after For example to generate assertions in an :program:`llc`-based test, after
adding RUN line use: adding one or more RUN lines use:
.. code-block:: bash .. code-block:: bash
% llvm/utils/update_llc_test_checks.py --llc-binary build/bin/llc test.ll % llvm/utils/update_llc_test_checks.py --llc-binary build/bin/llc test.ll
And if you want to update assertions in an existing test case, pass `-u` option This will generate FileCheck assertions, and insert a ``NOTE:`` line at the
which first check the ``NOTE:`` line exists and matches the script name. top to indicate that assertions were automatically generated.
If you want to update assertions in an existing test case, pass the `-u` option
which first checks the ``NOTE:`` line exists and matches the script name.
Sometimes a test absolutely depends on hand-written assertions and should not
have assertions automatically generated. In that case, add the text ``NOTE: Do
not autogenerate`` to the first line, and the scripts will skip that test. It
is a good idea to explain why generated assertions will not work for the test
so future developers will understand what is going on.
These are the most common scripts and their purposes/applications in generating These are the most common scripts and their purposes/applications in generating
assertions: assertions:
.. code-block:: none .. code-block:: none
update_analyze_test_checks.py update_analyze_test_checks.py
opt -passes='print<cost-model>' opt -passes='print<cost-model>'
▲ Show 20 Lines • Show All 696 Lines • Show Last 20 Lines

llvm/utils/UpdateTestChecks/common.py

Show First 20 Lines • Show All 321 Lines • ▼ Show 20 Lines for pattern in test_patterns:
tests_list = glob.glob(pattern) tests_list = glob.glob(pattern)
if not tests_list: if not tests_list:
warn("Test file pattern '%s' was not found. Ignoring it." % (pattern,)) warn("Test file pattern '%s' was not found. Ignoring it." % (pattern,))
continue continue
for test in tests_list: for test in tests_list:
with open(test) as f: with open(test) as f:
input_lines = [l.rstrip() for l in f] input_lines = [l.rstrip() for l in f]
first_line = input_lines[0] if input_lines else "" first_line = input_lines[0] if input_lines else ""
if UTC_AVOID in first_line:
warn("Skipping test that must not be autogenerated: " + test)
continue
is_regenerate = UTC_ADVERT in first_line is_regenerate = UTC_ADVERT in first_line
# If we're generating a new test, set the default version to the latest. # If we're generating a new test, set the default version to the latest.
argv = sys.argv[:] argv = sys.argv[:]
if not is_regenerate: if not is_regenerate:
argv.insert(1, "--version=" + str(DEFAULT_VERSION)) argv.insert(1, "--version=" + str(DEFAULT_VERSION))
args = parse_args(parser, argv[1:]) args = parse_args(parser, argv[1:])
▲ Show 20 Lines • Show All 124 Lines • ▼ Show 20 Lines
CHECK_PREFIX_RE = re.compile(r"--?check-prefix(?:es)?[= ](\S+)") CHECK_PREFIX_RE = re.compile(r"--?check-prefix(?:es)?[= ](\S+)")
PREFIX_RE = re.compile("^[a-zA-Z0-9_-]+$") PREFIX_RE = re.compile("^[a-zA-Z0-9_-]+$")
CHECK_RE = re.compile( CHECK_RE = re.compile(
r"^\s*(?://|[;#])\s*([^:]+?)(?:-NEXT|-NOT|-DAG|-LABEL|-SAME|-EMPTY)?:" r"^\s*(?://|[;#])\s*([^:]+?)(?:-NEXT|-NOT|-DAG|-LABEL|-SAME|-EMPTY)?:"
) )
UTC_ARGS_KEY = "UTC_ARGS:" UTC_ARGS_KEY = "UTC_ARGS:"
UTC_ARGS_CMD = re.compile(r".*" + UTC_ARGS_KEY + "\s*(?P<cmd>.*)\s*$") UTC_ARGS_CMD = re.compile(r".*" + UTC_ARGS_KEY + "\s*(?P<cmd>.*)\s*$")
UTC_ADVERT = "NOTE: Assertions have been autogenerated by " UTC_ADVERT = "NOTE: Assertions have been autogenerated by "
UTC_AVOID = "NOTE: Do not autogenerate"
UNUSED_NOTE = "NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:" UNUSED_NOTE = "NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:"
efriedmaUnsubmitted
Done
ReplyInline Actions

Maybe delete the trailing space?

efriedma: Maybe delete the trailing space?
OPT_FUNCTION_RE = re.compile( OPT_FUNCTION_RE = re.compile(
r"^(\s*;\s*Function\sAttrs:\s(?P<attrs>[\w\s():,]+?))?\s*define\s+(?P<funcdef_attrs_and_ret>[^@]*)@(?P<func>[\w.$-]+?)\s*" r"^(\s*;\s*Function\sAttrs:\s(?P<attrs>[\w\s():,]+?))?\s*define\s+(?P<funcdef_attrs_and_ret>[^@]*)@(?P<func>[\w.$-]+?)\s*"
r"(?P<args_and_sig>\((\)|(.*?[\w.-]+?)\))[^{]*\{)\n(?P<body>.*?)^\}$", r"(?P<args_and_sig>\((\)|(.*?[\w.-]+?)\))[^{]*\{)\n(?P<body>.*?)^\}$",
flags=(re.M | re.S), flags=(re.M | re.S),
) )
ANALYZE_FUNCTION_RE = re.compile( ANALYZE_FUNCTION_RE = re.compile(
▲ Show 20 Lines • Show All 1,381 Lines • Show Last 20 Lines