[Utils] Allow "on-the-fly" argument changes for update_test_check scripts

[jdoerfert]

Summary

Update test scripts were limited because they performed a single action
on the entire file and if that action was controlled by arguments, like
the one introduced in D68819, there was no record of it.

This patch introduces the capability of changing the arguments passed to
the script "on-the-fly" while processing a test file. In addition, an
"on/off" switch was added so that processing can be disabled for parts
of the file where the content is simply copied. The last extension is a
record of the invocation arguments in the auto generated NOTE. These
arguments are also picked up in a subsequent invocation, allowing
updates with special options enabled without user interaction.

To change the arguments the string UTC_ARGS: has to be present in a
line, followed by "additional command line arguments". That is
everything that follows UTC_ARGS: will be added to a growing list
of "command line arguments" which is reparsed after every update.

[arichardson]

This seems useful, thanks! Another case that would be nice is if there was a way to keep comments inside the autogenerated CHECK lines.

For example:

; CHECK-NEXT: ...
;; We generate a foo instruction and not a bar instruction here because...
; CHECK-NEXT: foo a, b, c
; CHECK-NEXT: ...

Right now if you re-run the update script the comment will be moved after all the CHECK lines. I wonder if there is a way to keep inline comments (maybe only if the CHECK: lines before and after didn't change?). Would this work with the UTC_ARGS: --turn off method?

llvm/utils/UpdateTestChecks/common.py

66

I think you need a r for the second string as well since otherwise newer python will complain that \s is not a valid escape sequence.

[davezarzycki]

In D69701#1729829, @arichardson wrote:

This seems useful, thanks! Another case that would be nice is if there was a way to keep comments inside the autogenerated CHECK lines.

For example:

; CHECK-NEXT: ...
;; We generate a foo instruction and not a bar instruction here because...
; CHECK-NEXT: foo a, b, c
; CHECK-NEXT: ...

Right now if you re-run the update script the comment will be moved after all the CHECK lines. I wonder if there is a way to keep inline comments (maybe only if the CHECK: lines before and after didn't change?). Would this work with the UTC_ARGS: --turn off method?

IMHO, that kind of comment belongs with the code that *generates* the output, rather than buried in the middle of a random test file. Separably, if it isn't clear what the test is testing, then perhaps some discussion at the top of the file/function seems appropriate, perhaps with links back to the relevant files/functions that are being tested. For example: "Please see the discussion inside of function foo in Bar.cpp for what is being tested below."

[jdoerfert]

In D69701#1729845, @davezarzycki wrote:

In D69701#1729829, @arichardson wrote:

This seems useful, thanks! Another case that would be nice is if there was a way to keep comments inside the autogenerated CHECK lines.

For example:

; CHECK-NEXT: ...
;; We generate a foo instruction and not a bar instruction here because...
; CHECK-NEXT: foo a, b, c
; CHECK-NEXT: ...

Right now if you re-run the update script the comment will be moved after all the CHECK lines. I wonder if there is a way to keep inline comments (maybe only if the CHECK: lines before and after didn't change?). Would this work with the UTC_ARGS: --turn off method?

It would not. The generation still happens in bulk. See below.

IMHO, that kind of comment belongs with the code that *generates* the output, rather than buried in the middle of a random test file. Separably, if it isn't clear what the test is testing, then perhaps some discussion at the top of the file/function seems appropriate, perhaps with links back to the relevant files/functions that are being tested. For example: "Please see the discussion inside of function foo in Bar.cpp for what is being tested below."

There are multiple use cases here for sure and the script addresses only one:
You generate check lines with the script, then you modify some of them for some reason. (Or you insert a test with manual check in lines into a file that has automatically generated ones.)
With this change you can "protect" the manual check lines in a reasonable way so we can rerun the script on files with the NOTE line.

llvm/utils/UpdateTestChecks/common.py

66

agreed

[jdoerfert]

Lessons learned from actually using it (D68766)

[jdoerfert]

Any comments (on this and the other [utils] patches)?

[lebedev.ri]

Now that i believe there's some testing infra for these utils, add/update tests for this?

[jdoerfert]

Add and adjust regression tests

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

[lebedev.ri]

This seems ok, but i don't feel confident accepting.
Any takers?

[arichardson]

llvm/utils/update_test_checks.py

69

I find the equals in the flag slightly confusing/non-intuitive.

Maybe better to add a action=store_true and a action=store_false booleanoption with the same dest=. Maybe --turn-on/--turn-off or simply --enable/--disable?

174

Use a bool argparse option?

[arichardson]

If I understand this correctly existing check lines between off/on directives should be copied verbatim. Could you add one (that does not match the auto-generated content) to the disabled part of the test?

[jdoerfert]

In D69701#1799763, @arichardson wrote:

If I understand this correctly existing check lines between off/on directives should be copied verbatim. Could you add one (that does not match the auto-generated content) to the disabled part of the test?

Will do.

Maybe better to add a action=store_true and a action=store_false booleanoption with the same dest=. Maybe --turn-on/--turn-off or simply --enable/--disable?

I'll look into that and update the test.

[lebedev.ri]

Needs rebasing

[jdoerfert]

Rebase, make the arguments -enable and -disable

[jdoerfert]

Add check lines in the "disabled" part to show how they "survive"

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: unknown.

clang-format: unknown.

Build artifacts: diff.json, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[jdoerfert]

ping

[lebedev.ri]

@MaskRay thoughts?

[arichardson]

Looks good to me. Since this is in common.py, it would be nice to see it also applied to update_cc_test checks or update_llc_test_checks.

llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test

11–14

Maybe expand this comment to include "since the generated file will have --function-signature in an UTC_ARGS: comment" or similar

[jdoerfert]

In D69701#1869559, @arichardson wrote:

Looks good to me. Since this is in common.py, it would be nice to see it also applied to update_cc_test checks or update_llc_test_checks.

I agree but will leave that as a future improvement ;)

llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test

11–14

Will do.

llvm/test/Transforms/Attributor/align.ll

; <label>:3:                                      ; preds = %1
   %4 = tail call i8* @f1(i8* nonnull @a2)
   br label %5
 
 ; <label>:5:                                      ; preds = %1, %3
   %6 = phi i8* [ %4, %3 ], [ @a1, %1 ]
   ret i8* %6
 }
 
-; UTC_ARGS: --turn=on
+; UTC_ARGS: --enable
 
 ; TEST 7
 ; Better than IR information
 define align 4 i32* @test7(i32* align 32 %p) #0 {
 ; ATTRIBUTOR_MODULE-LABEL: define {{[^@]+}}@test7
 ; ATTRIBUTOR_MODULE-SAME: (i32* nofree readnone returned align 32 "no-capture-maybe-returned" [[P:%.*]])
 ; ATTRIBUTOR_MODULE-NEXT:    ret i32* [[P]]
 ;

 ; ATTRIBUTOR-SAME: (i32* nofree readnone returned align 32 "no-capture-maybe-returned" [[P:%.*]])
 ; ATTRIBUTOR-NEXT:    [[TMP1:%.*]] = tail call i8* @f1b(i8* noalias nofree nonnull readnone align 8 dereferenceable(1) @a1)
 ; ATTRIBUTOR-NEXT:    ret i32* [[P]]
 ;
   tail call i8* @f1b(i8* align 8 dereferenceable(1) @a1)
   ret i32* %p
 }
 
-; UTC_ARGS: --turn=off
+; UTC_ARGS: --disable
 
 ; TEST 8
 define void @test8_helper() {
   %ptr0 = tail call i32* @unknown()
   %ptr1 = tail call align 4 i32* @unknown()
   %ptr2 = tail call align 8 i32* @unknown()
 
   tail call void @test8(i32* %ptr1, i32* %ptr1, i32* %ptr0)

llvm/test/Transforms/Attributor/liveness.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --scrub-attributes
 ; RUN: opt -attributor --attributor-disable=false -attributor-max-iterations-verify -attributor-annotate-decl-cs -attributor-max-iterations=6 -S < %s | FileCheck %s --check-prefixes=CHECK,MODULE,ALL_BUT_OLD_CGSCCC
 ; RUN: opt -attributor-cgscc --attributor-disable=false -attributor-annotate-decl-cs -attributor-max-iterations=6 -S < %s | FileCheck %s --check-prefixes=CHECK,CGSCC
 ; RUN: opt -passes=attributor --attributor-disable=false -attributor-max-iterations-verify -attributor-annotate-decl-cs -attributor-max-iterations=6 -S < %s | FileCheck %s --check-prefixes=CHECK,MODULE,ALL_BUT_OLD_CGSCCC
 ; RUN: opt -passes='attributor-cgscc' --attributor-disable=false -attributor-annotate-decl-cs -attributor-max-iterations=6 -S < %s | FileCheck %s --check-prefixes=CHECK,CGSCC,ALL_BUT_OLD_CGSCCC
-; UTC_ARGS: --turn off
+; UTC_ARGS: --disable
 
 ; ALL_BUT_OLD_CGSCCC: @dead_with_blockaddress_users.l = constant [2 x i8*] [i8* inttoptr (i32 1 to i8*), i8* inttoptr (i32 1 to i8*)]
 @dead_with_blockaddress_users.l = constant [2 x i8*] [i8* blockaddress(@dead_with_blockaddress_users, %lab0), i8* blockaddress(@dead_with_blockaddress_users, %end)]
 
 declare void @no_return_call() nofree noreturn nounwind readnone
 
 declare void @normal_call() readnone
 
continue:
   br label %cond.end
 
 cleanup:
   %res = landingpad { i8*, i32 }
   catch i8* null
   ret i32 0
 }
 
-; UTC_ARGS: --turn on
+; UTC_ARGS: --enable
 
 ; TEST 5.4 unounwind invoke instruction replaced by a call and a branch instruction put after it.
 define i32 @invoke_nounwind_phi(i32 %a) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*) {
 ; CHECK-LABEL: define {{[^@]+}}@invoke_nounwind_phi
 ; CHECK-SAME: (i32 [[A:%.*]]) personality i8* bitcast (i32 (...)* @__gxx_personality_v0 to i8*)
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[CMP:%.*]] = icmp eq i32 [[A]], 0
 ; CHECK-NEXT:    br i1 [[CMP]], label [[COND_TRUE:%.*]], label [[COND_FALSE:%.*]]
continue:
   %p = phi i32 [ %call, %cond.true ], [ %call1, %cond.false ]
   ret i32 %p
 
 cleanup:
   %res = landingpad { i8*, i32 } catch i8* null
   ret i32 0
 }
 
-; UTC_ARGS: --turn off
+; UTC_ARGS: --disable
 
 ; TEST 6: Undefined behvior, taken from LangRef.
 ; FIXME: Should be able to detect undefined behavior.
 
 ; CHECK: define void @ub(i32* nocapture nofree writeonly %0)
 define void @ub(i32* %0) {
   %poison = sub nuw i32 0, 1           ; Results in a poison value.
   %still_poison = and i32 %poison, 0   ; 0, but also poison.

 }
 
 define void @useless_arg_ext_int_ext(i32* %a) {
 ; CHECK: call void @useless_arg_ext_int(i32* %a)
   call void @useless_arg_ext_int(i32* %a)
   ret void
 }
 
-; UTC_ARGS: --turn on
+; UTC_ARGS: --enable
 
 ; FIXME: We should fold terminators.
 
 define internal i32 @switch_default(i64 %i) nounwind {
 ; CHECK-LABEL: define {{[^@]+}}@switch_default
 ; CHECK-SAME: (i64 [[I:%.*]])
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    switch i64 0, label [[SW_DEFAULT:%.*]] [

 
 define i32 @switch_default_dead_caller() {
 ; CHECK-LABEL: define {{[^@]+}}@switch_default_dead_caller()
 ; CHECK-NEXT:    ret i32 123
 ;
   %call2 = tail call i32 @switch_default_dead(i64 0)
   ret i32 %call2
 }
-; UTC_ARGS: --turn off
+
+; UTC_ARGS: --disable
 
 ; Allow blockaddress users
 ; ALL_BUT_OLD_CGSCCC-NOT @dead_with_blockaddress_users
 define internal void @dead_with_blockaddress_users(i32* nocapture %pc) nounwind readonly {
 entry:
   br label %indirectgoto
 
 lab0:                                             ; preds = %indirectgoto

llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/argument_name_reuse.ll.expected

-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature
 ; RUN: opt < %s -S | FileCheck %s
 
 define i32 @reuse_arg_names(i32 %X, i32 %Y) {
 ; CHECK-LABEL: define {{[^@]+}}@reuse_arg_names
 ; CHECK-SAME: (i32 [[X:%.*]], i32 [[Y:%.*]])
 ; CHECK-NEXT:    [[Z:%.*]] = sub i32 [[X]], [[Y]]
 ; CHECK-NEXT:    [[Q:%.*]] = add i32 [[Z]], [[Y]]
 ; CHECK-NEXT:    ret i32 [[Q]]
 ;
   %Z = sub i32 %X, %Y
   %Q = add i32 %Z, %Y
   ret i32 %Q
 }

llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/basic.ll.funcsig.expected

-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature
 ; Example input for update_llc_test_checks (taken from test/Transforms/InstSimplify/add.ll)
 ; RUN: opt < %s -instsimplify -S | FileCheck %s
 
 define i32 @common_sub_operand(i32 %X, i32 %Y) {
 ; CHECK-LABEL: define {{[^@]+}}@common_sub_operand
 ; CHECK-SAME: (i32 [[X:%.*]], i32 [[Y:%.*]])
 ; CHECK-NEXT:    ret i32 [[X]]
 ;

llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/on_the_fly_arg_change.ll

+; RUN: opt -S < %s | FileCheck %s
+
+declare void @foo()
+
+define void @check_lines_1() {
+  ret void
+}
+
+; UTC_ARGS: --disable
+
+; A check line that would not be auto generated.
+; CHECK: define void @no_check_lines() {
+define void @no_check_lines() {
+  ret void
+}
+
+; UTC_ARGS: --enable
+
+define void @check_lines_2() {
+  ret void
+}
+
+define void @scrub() {
+  call void @foo() readnone
+  ret void
+}
+
+define i32 @signature(i32 %arg) {
+  ret i32 %arg
+}

llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/on_the_fly_arg_change.ll.expected

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --function-signature --scrub-attributes
+; RUN: opt -S < %s | FileCheck %s
+
+declare void @foo()
+
+define void @check_lines_1() {
+; CHECK-LABEL: define {{[^@]+}}@check_lines_1()
+; CHECK-NEXT:    ret void
+;
+  ret void
+}
+
+; UTC_ARGS: --disable
+
+; A check line that would not be auto generated.
+; CHECK: define void @no_check_lines() {
+define void @no_check_lines() {
+  ret void
+}
+
+; UTC_ARGS: --enable
+
+define void @check_lines_2() {
+; CHECK-LABEL: define {{[^@]+}}@check_lines_2()
+; CHECK-NEXT:    ret void
+;
+  ret void
+}
+
+define void @scrub() {
+; CHECK-LABEL: define {{[^@]+}}@scrub()
+; CHECK-NEXT:    call void @foo()
+; CHECK-NEXT:    ret void
+;
+  call void @foo() readnone
+  ret void
+}
+
+define i32 @signature(i32 %arg) {
+; CHECK-LABEL: define {{[^@]+}}@signature
+; CHECK-SAME: (i32 [[ARG:%.*]])
+; CHECK-NEXT:    ret i32 [[ARG]]
+;
+  ret i32 %arg
+}

llvm/test/tools/UpdateTestChecks/update_test_checks/Inputs/scrub_attrs.ll.scrub.expected

-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --scrub-attributes
 ; RUN: opt -S < %s | FileCheck %s
 
 declare void @foo()
 
 define internal void @bar() {
 ; CHECK-LABEL: @bar(
 ; CHECK-NEXT:    call void @foo()
 ; CHECK-NEXT:    ret void
 ;
   call void @foo() readnone
   ret void
 }

llvm/test/tools/UpdateTestChecks/update_test_checks/basic.test

 # REQUIRES: x86-registered-target
 ## Basic test checking that update_test_checks.py works correctly
 # RUN: cp -f %S/Inputs/basic.ll %t.ll && %update_test_checks %t.ll
 # RUN: diff -u %t.ll %S/Inputs/basic.ll.expected
 ## Check that running the script again does not change the result:
 # RUN: %update_test_checks %t.ll
 # RUN: diff -u %t.ll %S/Inputs/basic.ll.expected
 ## Also try the --function-signature flag
 # RUN: %update_test_checks %t.ll --function-signature
 # RUN: diff -u %t.ll %S/Inputs/basic.ll.funcsig.expected
-## Verify that running without the --function-signature flag removes the -SAME: lines:
+## Verify that running without the --function-signature flag does not removes
+## the -SAME: lines since the generated file will have --function-signature in
+## an UTC_ARGS: comment in the first line (from the invocation above) which is
+## added to the update invocation below.
 # RUN: %update_test_checks %t.ll
-# RUN: diff -u %t.ll %S/Inputs/basic.ll.expected
+# RUN: diff -u %t.ll %S/Inputs/basic.ll.funcsig.expected

llvm/test/tools/UpdateTestChecks/update_test_checks/on_the_fly_arg_change.test

+# RUN: cp -f %S/Inputs/on_the_fly_arg_change.ll %t.ll
+# RUN: %update_test_checks %t.ll --function-signature --scrub-attributes
+# RUN: diff -u %t.ll %S/Inputs/on_the_fly_arg_change.ll.expected
+## Check that running the script again does not change the result:
+# RUN: %update_test_checks %t.ll
+# RUN: diff -u %t.ll %S/Inputs/on_the_fly_arg_change.ll.expected

llvm/test/tools/UpdateTestChecks/update_test_checks/scrub_attrs.test

 ## scrub_attrs test checking that update_test_checks.py works correctly
 # RUN: cp -f %S/Inputs/scrub_attrs.ll %t.ll && %update_test_checks %t.ll
 # RUN: diff -u %t.ll %S/Inputs/scrub_attrs.ll.plain.expected
-## Check that running the script again does not change the result:
-# RUN: %update_test_checks %t.ll
-# RUN: diff -u %t.ll %S/Inputs/scrub_attrs.ll.plain.expected
 ## Also try the --scrub-attributes flag
 # RUN: %update_test_checks %t.ll --scrub-attributes
 # RUN: diff -u %t.ll %S/Inputs/scrub_attrs.ll.scrub.expected
+## Check that running the script again does not change the result:
+# RUN: %update_test_checks %t.ll
+# RUN: diff -u %t.ll %S/Inputs/scrub_attrs.ll.scrub.expected

llvm/utils/UpdateTestChecks/common.py

 
 ##### LLVM IR parser
 
 RUN_LINE_RE = re.compile(r'^\s*(?://|[;#])\s*RUN:\s*(.*)$')
 CHECK_PREFIX_RE = re.compile(r'--?check-prefix(?:es)?[= ](\S+)')
 PREFIX_RE = re.compile('^[a-zA-Z0-9_-]+$')
 CHECK_RE = re.compile(r'^\s*(?://|[;#])\s*([^:]+?)(?:-NEXT|-NOT|-DAG|-LABEL|-SAME|-EMPTY)?:')
 
+UTC_ARGS_KEY = 'UTC_ARGS:'
+UTC_ARGS_CMD = re.compile(r'.*' + UTC_ARGS_KEY + '\s*(?P<cmd>.*)\s*$')
+
 OPT_FUNCTION_RE = re.compile(
     r'^\s*define\s+(?:internal\s+)?[^@]*@(?P<func>[\w-]+?)\s*'
     r'(?P<args_and_sig>\((\)|(.*?[\w\.\-]+?)\))[^{]*)\{\n(?P<body>.*?)^\}$',
     flags=(re.M | re.S))
 
 ANALYZE_FUNCTION_RE = re.compile(
     r'^\s*\'(?P<analysis>[\w\s-]+?)\'\s+for\s+function\s+\'(?P<func>[\w-]+?)\':'
     r'\s*\n(?P<body>.*)$',
if "check-prefix=" in part:
       prefix = part.split('=', 1)[1]
       check_prefix(prefix)
     elif "check-prefixes=" in part:
       prefixes = part.split('=', 1)[1].split(',')
       for prefix in prefixes:
         check_prefix(prefix)
         if prefixes.count(prefix) > 1:
           warn("Supplied prefix '%s' is not unique in the prefix list." % (prefix,))
+
+def get_autogennote_suffix(parser, args):
+    autogenerated_note_args = ''
+    for k, v in args._get_kwargs():
+        if parser.get_default(k) == v or k == 'tests' or k == 'update_only' or k == 'opt_binary':
+            continue
+        k = k.replace('_', '-')
+        if type(v) is bool:
+            autogenerated_note_args += '--%s ' % (k)
+        else:
+            autogenerated_note_args += '--%s %s ' % (k, v)
+    if autogenerated_note_args:
+        autogenerated_note_args = ' %s %s' % (UTC_ARGS_KEY, autogenerated_note_args[:-1])
+    return autogenerated_note_args
+
+
+def check_for_command(line, parser, args, argv):
+    cmd_m = UTC_ARGS_CMD.match(line)
+    if cmd_m:
+        cmd = cmd_m.group('cmd').strip().split(' ')
+        argv = argv + cmd
+        args = parser.parse_args(filter(lambda arg: arg not in args.tests, argv))
+    return args, argv

llvm/utils/update_test_checks.py

def main():
   parser.add_argument(
       '--function', help='The function in the test file to update')
   parser.add_argument('-p', '--preserve-names', action='store_true',
                       help='Do not scrub IR names')
   parser.add_argument('--function-signature', action='store_true',
                       help='Keep function signature information around for the check line')
   parser.add_argument('--scrub-attributes', action='store_true',
                       help='Remove attribute annotations (#0) from the end of check line')
+  parser.add_argument('--enable', action='store_true', dest='enabled', default=True,
+                      help='Activate CHECK line generation from this point forward')
+  parser.add_argument('--disable', action='store_false', dest='enabled',
+                      help='Deativate CHECK line generation from this point forward')
   parser.add_argument('tests', nargs='+')
   args = common.parse_commandline_args(parser)
 
   script_name = os.path.basename(__file__)
   autogenerated_note = (ADVERT + 'utils/' + script_name)
 
   opt_basename = os.path.basename(args.opt_binary)
   if not re.match(r'^opt(-\d+)?$', opt_basename):
     common.error('Unexpected opt name: ' + opt_basename)
     sys.exit(1)
   opt_basename = 'opt'
 
   for test in args.tests:
     if not glob.glob(test):
       common.warn("Test file pattern '%s' was not found. Ignoring it." % (test,))
       continue
 
   # On Windows we must expand the patterns ourselves.
   test_paths = [test for pattern in args.tests for test in glob.glob(pattern)]
   for test in test_paths:
+    argv = sys.argv[:]
+    args = parser.parse_args()
     with open(test) as f:
       input_lines = [l.rstrip() for l in f]
 
     first_line = input_lines[0] if input_lines else ""
     if 'autogenerated' in first_line and script_name not in first_line:
       common.warn("Skipping test which wasn't autogenerated by " + script_name, test)
       continue
+    if first_line and 'autogenerated' in first_line:
+      args, argv = common.check_for_command(first_line, parser, args, argv)
+    test_autogenerated_note = autogenerated_note + common.get_autogennote_suffix(parser, args)
 
     if args.update_only:
       if not first_line or 'autogenerated' not in first_line:
         common.warn("Skipping test which isn't autogenerated: " + test)
         continue
 
     run_lines = common.find_run_lines(test, input_lines)
 
for prefixes, opt_args in prefix_list:
               raw_tool_output, prefixes, func_dict, args.verbose,
               args.function_signature)
 
     is_in_function = False
     is_in_function_start = False
     prefix_set = set([prefix for prefixes, _ in prefix_list for prefix in prefixes])
     common.debug('Rewriting FileCheck prefixes:', str(prefix_set))
     output_lines = []
-    output_lines.append(autogenerated_note)
+    output_lines.append(test_autogenerated_note)
 
     for input_line in input_lines:
+      # Discard any previous script advertising.
+      if input_line.startswith(ADVERT):
+        continue
+
+      args, argv = common.check_for_command(input_line, parser, args, argv)
+      if not args.enabled:
+          output_lines.append(input_line)
+          continue
       if is_in_function_start:
         if input_line == '':
           continue
         if input_line.lstrip().startswith(';'):
           m = common.CHECK_RE.match(input_line)
           if not m or m.group(1) not in prefix_set:
             output_lines.append(input_line)
             continue
for input_line in input_lines:
           input_line = common.SCRUB_LEADING_WHITESPACE_RE.sub(r'  ', input_line)
           output_lines.append(input_line)
         else:
           continue
         if input_line.strip() == '}':
           is_in_function = False
         continue
 
-      # Discard any previous script advertising.
-      if input_line.startswith(ADVERT):
-        continue
-
       # If it's outside a function, it just gets copied to the output.
       output_lines.append(input_line)
 
       m = IR_FUNCTION_RE.match(input_line)
       if not m:
         continue
       func_name = m.group(1)
       if args.function is not None and func_name != args.function: