This is an archive of the discontinued LLVM Phabricator instance.

[Utils] Allow "on-the-fly" argument changes for update_test_check scripts
ClosedPublic

Authored by jdoerfert on Oct 31 2019, 8:53 PM.

Details

Summary

Update test scripts were limited because they performed a single action
on the entire file and if that action was controlled by arguments, like
the one introduced in D68819, there was no record of it.

This patch introduces the capability of changing the arguments passed to
the script "on-the-fly" while processing a test file. In addition, an
"on/off" switch was added so that processing can be disabled for parts
of the file where the content is simply copied. The last extension is a
record of the invocation arguments in the auto generated NOTE. These
arguments are also picked up in a subsequent invocation, allowing
updates with special options enabled without user interaction.

To change the arguments the string UTC_ARGS: has to be present in a
line, followed by "additional command line arguments". That is
everything that follows UTC_ARGS: will be added to a growing list
of "command line arguments" which is reparsed after every update.

Diff Detail

Event Timeline

jdoerfert created this revision.Oct 31 2019, 8:53 PM
Herald added a project: Restricted Project. · View Herald TranscriptOct 31 2019, 8:53 PM
Herald added a subscriber: bollu. · View Herald Transcript

This seems useful, thanks! Another case that would be nice is if there was a way to keep comments inside the autogenerated CHECK lines.

For example:

; CHECK-NEXT: ...
;; We generate a foo instruction and not a bar instruction here because...
; CHECK-NEXT: foo a, b, c
; CHECK-NEXT: ...

Right now if you re-run the update script the comment will be moved after all the CHECK lines. I wonder if there is a way to keep inline comments (maybe only if the CHECK: lines before and after didn't change?). Would this work with the UTC_ARGS: --turn off method?

llvm/utils/UpdateTestChecks/common.py
66

I think you need a r for the second string as well since otherwise newer python will complain that \s is not a valid escape sequence.

This seems useful, thanks! Another case that would be nice is if there was a way to keep comments inside the autogenerated CHECK lines.

For example:

; CHECK-NEXT: ...
;; We generate a foo instruction and not a bar instruction here because...
; CHECK-NEXT: foo a, b, c
; CHECK-NEXT: ...

Right now if you re-run the update script the comment will be moved after all the CHECK lines. I wonder if there is a way to keep inline comments (maybe only if the CHECK: lines before and after didn't change?). Would this work with the UTC_ARGS: --turn off method?

IMHO, that kind of comment belongs with the code that *generates* the output, rather than buried in the middle of a random test file. Separably, if it isn't clear what the test is testing, then perhaps some discussion at the top of the file/function seems appropriate, perhaps with links back to the relevant files/functions that are being tested. For example: "Please see the discussion inside of function foo in Bar.cpp for what is being tested below."

jdoerfert marked an inline comment as done.Nov 1 2019, 9:06 AM

This seems useful, thanks! Another case that would be nice is if there was a way to keep comments inside the autogenerated CHECK lines.

For example:

; CHECK-NEXT: ...
;; We generate a foo instruction and not a bar instruction here because...
; CHECK-NEXT: foo a, b, c
; CHECK-NEXT: ...

Right now if you re-run the update script the comment will be moved after all the CHECK lines. I wonder if there is a way to keep inline comments (maybe only if the CHECK: lines before and after didn't change?). Would this work with the UTC_ARGS: --turn off method?

It would not. The generation still happens in bulk. See below.

IMHO, that kind of comment belongs with the code that *generates* the output, rather than buried in the middle of a random test file. Separably, if it isn't clear what the test is testing, then perhaps some discussion at the top of the file/function seems appropriate, perhaps with links back to the relevant files/functions that are being tested. For example: "Please see the discussion inside of function foo in Bar.cpp for what is being tested below."

There are multiple use cases here for sure and the script addresses only one:
You generate check lines with the script, then you modify some of them for some reason. (Or you insert a test with manual check in lines into a file that has automatically generated ones.)
With this change you can "protect" the manual check lines in a reasonable way so we can rerun the script on files with the NOTE line.

llvm/utils/UpdateTestChecks/common.py
66

agreed

jdoerfert updated this revision to Diff 227488.Nov 1 2019, 11:03 AM

Lessons learned from actually using it (D68766)

Any comments (on this and the other [utils] patches)?

lebedev.ri requested changes to this revision.Dec 6 2019, 6:05 AM

Now that i believe there's some testing infra for these utils, add/update tests for this?

This revision now requires changes to proceed.Dec 6 2019, 6:05 AM
jdoerfert updated this revision to Diff 235673.Dec 30 2019, 8:51 PM

Add and adjust regression tests

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

This seems ok, but i don't feel confident accepting.
Any takers?

arichardson added inline comments.Dec 31 2019, 12:54 AM
llvm/utils/update_test_checks.py
69

I find the equals in the flag slightly confusing/non-intuitive.

Maybe better to add a action=store_true and a action=store_false booleanoption with the same dest=. Maybe --turn-on/--turn-off or simply --enable/--disable?

174

Use a bool argparse option?

If I understand this correctly existing check lines between off/on directives should be copied verbatim. Could you add one (that does not match the auto-generated content) to the disabled part of the test?

If I understand this correctly existing check lines between off/on directives should be copied verbatim. Could you add one (that does not match the auto-generated content) to the disabled part of the test?

Will do.

Maybe better to add a action=store_true and a action=store_false booleanoption with the same dest=. Maybe --turn-on/--turn-off or simply --enable/--disable?

I'll look into that and update the test.

davezarzycki resigned from this revision.Jan 5 2020, 6:20 AM

Needs rebasing

jdoerfert updated this revision to Diff 241197.Jan 29 2020, 9:32 AM

Rebase, make the arguments -enable and -disable

jdoerfert updated this revision to Diff 241198.Jan 29 2020, 9:33 AM

Add check lines in the "disabled" part to show how they "survive"

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

lebedev.ri added a subscriber: MaskRay.

@MaskRay thoughts?

lebedev.ri resigned from this revision.Feb 8 2020, 12:32 PM
arichardson accepted this revision.Feb 11 2020, 6:57 AM

Looks good to me. Since this is in common.py, it would be nice to see it also applied to update_cc_test checks or update_llc_test_checks.

llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test
11–14

Maybe expand this comment to include "since the generated file will have --function-signature in an UTC_ARGS: comment" or similar

This revision is now accepted and ready to land.Feb 11 2020, 6:57 AM
jdoerfert marked an inline comment as done.Feb 11 2020, 7:46 AM

Looks good to me. Since this is in common.py, it would be nice to see it also applied to update_cc_test checks or update_llc_test_checks.

I agree but will leave that as a future improvement ;)

llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test
11–14

Will do.

This revision was automatically updated to reflect the committed changes.