[update_cc_test_checks] Support 'clang | opt | FileCheck'

[simon_tatham]

Summary

Some clang lit tests use a pipeline of the form

// RUN: %clang [args] -O0 %s | opt [specific optimizations] | FileCheck %s

to make the expected test output depend on as few optimization phases
as possible, for stability. But when you write a RUN line of this
form, you lose the ability to use update_cc_test_checks.py to
automatically generate the expected output, because it only supports
two-stage pipelines consisting of '%clang | FileCheck' (or %clang_cc1).

This change extends the set of supported RUN lines so that pipelines
with an invocation of opt in the middle can still be automatically
handled.

To implement it, I've adjusted get_function_body() so that it can
cope with an arbitrary sequence of intermediate pipeline commands. But
the code that decides which RUN lines to consider is more
conservative: it only adds clang | opt | FileCheck to the set of
supported lines, because I didn't want to accidentally include some
other kind of line that doesn't output IR at all.

(Also in this commit is the minimal change to make this script work at
all, after r373912 added an extra parameter to add_ir_checks.)

[lebedev.ri]

For the record, is this intended to be used in upstream clang tests?

[MaskRay]

There are 30 existing clang CodeGen tests using %clang ... | opt ... | FileCheck, but many (mem2reg) are used to increase readability while some are probably misuses of opt. +@RKSimon who has expressed concerns of maintenance problems on D68230.

[e-mail] and @spatel who have added opt to clang tests.

llvm/utils/update_cc_test_checks.py

115

If a RUN line uses opt, you'll need a similar option for opt here.

This is named --opt-binary in update_test_checks.py, but --opt might be more convenient.

[simon_tatham]

@lebedev.ri, yes, I was planning to use it for the tests I'm adding in D67161, after a review comment in there suggested I should be using -O0 | opt -mem2reg in place of -O3.

[spatel]

In D68406#1694002, @MaskRay wrote:

[e-mail] and @spatel who have added opt to clang tests.

I don't remember that...but I'll trust that's correct over my memory. :)
For reference, we discussed the compromise solution of using "opt -mem2reg" in:
D17999
(and there's a rough script there that might have been used to help implement update_cc_test_checks.py)

[simon_tatham]

Added an option as requested to point at the opt binary. Unlike the
existing --clang, I've made it only lazily give an error: in the
common case where the test file doesn't try to call opt in the first
place, it shouldn't be necessary to have it available.

Also, I've added a fix to the add_ir_checks call, which stopped
working recently due to r373912.

[dmgreen]

This looks very useful to me (I've often wanted something similar with opt -> llc). It would make all the testing MVE intrinsics a lot more readable (to the extent that I'm not sure what we would do without it!)

[MaskRay]

llvm/utils/update_cc_test_checks.py

225

'opt'

llvm/utils/update_cc_test_checks.py

def config():
   parser.add_argument('-v', '--verbose', action='store_true')
   parser.add_argument('--llvm-bin', help='llvm $prefix/bin path')
   parser.add_argument('--clang',
                       help='"clang" executable, defaults to $llvm_bin/clang')
   parser.add_argument('--clang-args',
                       help='Space-separated extra args to clang, e.g. --clang-args=-v')
   parser.add_argument('--c-index-test',
                       help='"c-index-test" executable, defaults to $llvm_bin/c-index-test')
+  parser.add_argument('--opt',
+                      help='"opt" executable, defaults to $llvm_bin/opt')
   parser.add_argument(
       '--functions', nargs='+', help='A list of function name regexes. '
       'If specified, update CHECK lines for functions matching at least one regex')
   parser.add_argument(
       '--x86_extra_scrub', action='store_true',
       help='Use more regex for x86 matching to reduce diffs between various subtargets')
   parser.add_argument('-u', '--update-only', action='store_true',
                       help='Only update test if it was already autogened')
   parser.add_argument('tests', nargs='+')
   args = parser.parse_args()
   args.clang_args = shlex.split(args.clang_args or '')
 
   if args.clang is None:
     if args.llvm_bin is None:
       args.clang = 'clang'
     else:
       args.clang = os.path.join(args.llvm_bin, 'clang')
   if not distutils.spawn.find_executable(args.clang):
     print('Please specify --llvm-bin or --clang', file=sys.stderr)
     sys.exit(1)
+
+  if args.opt is None:
+    if args.llvm_bin is None:
+      args.opt = 'opt'
+    else:
+      args.opt = os.path.join(args.llvm_bin, 'opt')
+  if not distutils.spawn.find_executable(args.opt):
+    # Many uses of this tool will not need an opt binary, because it's only
+    # needed for updating a test that runs clang | opt | FileCheck. So we
+    # defer this error message until we find that opt is actually needed.
+    args.opt = None
+
   if args.c_index_test is None:
     if args.llvm_bin is None:
       args.c_index_test = 'c-index-test'
     else:
       args.c_index_test = os.path.join(args.llvm_bin, 'c-index-test')
   if not distutils.spawn.find_executable(args.c_index_test):
     print('Please specify --llvm-bin or --c-index-test', file=sys.stderr)
     sys.exit(1)
 
   return args
 
 
-def get_function_body(args, filename, clang_args, prefixes, triple_in_cmd, func_dict):
+def get_function_body(args, filename, clang_args, extra_commands, prefixes, triple_in_cmd, func_dict):
   # TODO Clean up duplication of asm/common build_function_body_dictionary
   # Invoke external tool and extract function bodies.
   raw_tool_output = common.invoke_tool(args.clang, clang_args, filename)
+  for extra_command in extra_commands:
+    extra_args = shlex.split(extra_command)
+    with tempfile.NamedTemporaryFile() as f:
+      f.write(raw_tool_output.encode())
+      f.flush()
+      if extra_args[0] == "opt":
+        if args.opt is None:
+          print(filename, 'needs to run opt. '
+                'Please specify --llvm-bin or --opt', file=sys.stderr)
+          sys.exit(1)
+        extra_args[0] = args.opt
+      raw_tool_output = common.invoke_tool(extra_args[0],
+                                           extra_args[1:], f.name)
   if '-emit-llvm' in clang_args:
     common.build_function_body_dictionary(
             common.OPT_FUNCTION_RE, common.scrub_body, [],
             raw_tool_output, prefixes, func_dict, args.verbose)
   else:
     print('The clang command line should include -emit-llvm as asm tests '
           'are discouraged in Clang testsuite.', file=sys.stderr)
     sys.exit(1)
if args.verbose:
       print('Found {} RUN lines:'.format(len(run_lines)), file=sys.stderr)
       for l in run_lines:
         print('  RUN: ' + l, file=sys.stderr)
 
     # Build a list of clang command lines and check prefixes from RUN lines.
     run_list = []
     line2spell_and_mangled_list = collections.defaultdict(list)
     for l in run_lines:
-      commands = [cmd.strip() for cmd in l.split('|', 1)]
+      commands = [cmd.strip() for cmd in l.split('|')]
 
       triple_in_cmd = None
       m = common.TRIPLE_ARG_RE.search(commands[0])
       if m:
         triple_in_cmd = m.groups()[0]
 
       # Apply %clang substitution rule, replace %s by `filename`, and append args.clang_args
       clang_args = shlex.split(commands[0])
       if clang_args[0] not in SUBST:
         print('WARNING: Skipping non-clang RUN line: ' + l, file=sys.stderr)
         continue
       clang_args[0:1] = SUBST[clang_args[0]]
       clang_args = [filename if i == '%s' else i for i in clang_args] + args.clang_args
 
+      # Permit piping the output through opt
+      if not (len(commands) == 2 or
+              (len(commands) == 3 and commands[1].startswith("opt"))):
+        print('WARNING: Skipping non-clang RUN line: ' + l, file=sys.stderr)
+
       # Extract -check-prefix in FileCheck args
       filecheck_cmd = commands[-1]
       common.verify_filecheck_prefixes(filecheck_cmd)
       if not filecheck_cmd.startswith('FileCheck '):
         print('WARNING: Skipping non-FileChecked RUN line: ' + l, file=sys.stderr)
         continue
       check_prefixes = [item for m in common.CHECK_PREFIX_RE.finditer(filecheck_cmd)
                                for item in m.group(1).split(',')]
       if not check_prefixes:
         check_prefixes = ['CHECK']
-      run_list.append((check_prefixes, clang_args, triple_in_cmd))
+      run_list.append((check_prefixes, clang_args, commands[1:-1], triple_in_cmd))
 
     # Strip CHECK lines which are in `prefix_set`, update test file.
     prefix_set = set([prefix for p in run_list for prefix in p[0]])
     input_lines = []
     with open(filename, 'r+') as f:
       for line in f:
         m = CHECK_RE.match(line)
         if not (m and m.group(1) in prefix_set) and line != '//\n':
           input_lines.append(line)
       f.seek(0)
       f.writelines(input_lines)
       f.truncate()
 
     # Execute clang, generate LLVM IR, and extract functions.
     func_dict = {}
     for p in run_list:
       prefixes = p[0]
       for prefix in prefixes:
         func_dict.update({prefix: dict()})
-    for prefixes, clang_args, triple_in_cmd in run_list:
+    for prefixes, clang_args, extra_commands, triple_in_cmd in run_list:
       if args.verbose:
         print('Extracted clang cmd: clang {}'.format(clang_args), file=sys.stderr)
         print('Extracted FileCheck prefixes: {}'.format(prefixes), file=sys.stderr)
 
-      get_function_body(args, filename, clang_args, prefixes, triple_in_cmd, func_dict)
+      get_function_body(args, filename, clang_args, extra_commands, prefixes, triple_in_cmd, func_dict)
 
       # Invoke c-index-test to get mapping from start lines to mangled names.
       # Forward all clang args for now.
       for k, v in get_line2spell_and_mangled(args, clang_args).items():
         line2spell_and_mangled_list[k].append(v)
 
     output_lines = [autogenerated_note]
     for idx, line in enumerate(input_lines):
for idx, line in enumerate(input_lines):
           # we simply require the spelling name to appear on the line
           # to exclude functions from other files.
           if mangled in added or spell not in line:
             continue
           if args.functions is None or any(re.search(regex, spell) for regex in args.functions):
             if added:
               output_lines.append('//')
             added.add(mangled)
-            common.add_ir_checks(output_lines, '//', run_list, func_dict, mangled)
+            common.add_ir_checks(output_lines, '//', run_list, func_dict, mangled, False)
       output_lines.append(line.rstrip('\n'))
 
     # Update the test file.
     with open(filename, 'w') as f:
       for line in output_lines:
         f.write(line + '\n')
 
   return 0
 
 
 if __name__ == '__main__':
   sys.exit(main())