[utils] Add utils/update_cc_test_checks.py

[MaskRay]

Summary

A utility to update LLVM IR in C/C++ FileCheck test files.

Example RUN lines in .c/.cc test files:

// RUN: %clang -S -Os -DXX %s -o - | FileCheck %s
// RUN: %clangxx -S -Os %s -o - | FileCheck -check-prefix=IR %s

Usage:

% utils/update_cc_test_checks.py --llvm-bin=release/bin test/a.cc
% utils/update_cc_test_checks.py --c-index-test=release/bin/c-index-test --clang=release/bin/clang /tmp/c/a.cc

Example input:

// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
// RUN: %clang -emit-llvm -S -Os -DXX %s -o - | FileCheck -check-prefix=AA %s
// RUN: %clangxx -emit-llvm -S -Os %s -o - | FileCheck -check-prefix=BB %s
using T =
#ifdef XX
    int __attribute__((vector_size(16)))
#else
    short __attribute__((vector_size(16)))
#endif
    ;

T foo(T a) {
  return a + a;
}

After utils/update_cc_test_checks.py --llvm-bin=release/bin a.cc

// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
// RUN: %clang -emit-llvm -S -Os -DXX %s -o - | FileCheck -check-prefix=AA %s
// RUN: %clangxx -emit-llvm -S -Os %s -o - | FileCheck -check-prefix=BB %s
using T =
#ifdef XX
    int __attribute__((vector_size(16)))
#else
    short __attribute__((vector_size(16)))
#endif
    ;

// AA-LABEL: _Z3fooDv4_i:
// AA:       entry:
// AA-NEXT:    %add = shl <4 x i32> %a, <i32 1, i32 1, i32 1, i32 1>
// AA-NEXT:    ret <4 x i32> %add
//
// BB-LABEL: _Z3fooDv8_s:
// BB:       entry:
// BB-NEXT:    %add = shl <8 x i16> %a, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
// BB-NEXT:    ret <8 x i16> %add
T foo(T a) {
  return a + a;
}

[MaskRay]

Multiple CHECK prefixes

[MaskRay]

Handle %clang_cc1

[MaskRay]

Update ADVERT and document --functions

[MaskRay]

Remove unused add_checks

[spatel]

Adding others who might be interested in this functionality, and must be more qualified to review python changes (since I don't know it at all).

We may be creating a moral hazard by making it easy to check asm from c/c++ (we're generally not supposed to have end-to-end unit/regression tests), but it's nice to have that functionality. :)

[RKSimon]

Ideally I think the refactoring for update_llc_test_checks.py/update_test_checks.py should be in a seperate patch that lands first and a patch adding update_cc_test_checks.py is added dependening on it.

[asb]

I agree with @RKSimon that splitting the refactoring to a separate patch would make sense. I've previously missed having an update_cc_test_checks.py for IR tests, so think this is a great improvement. End-to-end clang->assembly tests may not be welcome upstream, but given they're trivial to support after this refactoring it would seem mean not to expose that functionality to out-of-tree users who want it.

[MaskRay]

@RKSimon @asb Thanks for review. I have git rm utils/update_cc_test_checks.py and moved the refactoring part to https://reviews.llvm.org/D42805

[MaskRay]

Ignore '//\n' lines

[MaskRay]

rebase on D42805

[MaskRay]

Add --clang-args for debugging

[MaskRay]

Forward clang command line options to c-index-test (e.g. -std=c++11 -D which may change the AST)

[MaskRay]

Cleanup

[MaskRay]

Thanks everyone for review! https://reviews.llvm.org/D42805 has landed (refactoring utils/update* scripts so that this new script can re-use the CHECK line parsing and adding functionality).
should we push forward this one?

c-index-test is awesome. Different name mangling is now supported:

// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
// RUN: %clang -emit-llvm -S -Os -DXX %s -o - | FileCheck %s
// RUN: %clang -emit-llvm -S -Os %s -o - | FileCheck %s
struct A {};
struct B {};

using T =
#ifdef XX
A
#else
B
#endif
;

// CHECK-LABEL: _Z3foo1A:
// CHECK:       entry:
// CHECK-NEXT:    ret void
// CHECK-LABEL: _Z3foo1B:
// CHECK:       entry:
// CHECK-NEXT:    ret void
T foo(T x) {
  return x;
}

OT: It seems I can use arc diff refactor-update_test_checks to make this revision dependent on another git branch (refactor-update_test_checks).

[MaskRay]

Remove --c-index-test-args and use --clang-args for c-index-test

[MaskRay]

trivial

[MaskRay]

update

[MaskRay]

Support different mangled names on the same line

[MaskRay]

Clean up

[MaskRay]

Support %clangxx

[MaskRay]

Ping

[echristo]

Leaving it as an IR only check seems like the right thing to do. We definitely don't want to encourage people to have .cc -> .s checks.

utils/update_cc_test_checks.py

68

Perhaps you could split this into some functions?

[MaskRay]

Add --wont-emit-asm-in-clang-testsuite notice to discourage asm tests

[MaskRay]

utils/update_cc_test_checks.py

68

% utils/update_cc_test_checks.py --llvm-bin ~/Dev/llvm/release/bin /tmp/d/a.cc                                   
Enable --wont-emit-asm-in-clang-testsuite to acknowledge that asm tests are discouraged in Clang testsuite, and you won't use this script to check in them.

I was inspired by GNU parallel and added this option --wont-emit-asm-in-clang-testsuite

[MaskRay]

Change the notice. Time when I wish I was a lawyer.

[MaskRay]

Remove asm tests

[MaskRay]

Rebase

[echristo]

Start of some comment requests.

utils/UpdateTestChecks/asm.py

97 ↗

(On Diff #135710)

Hmm?

utils/UpdateTestChecks/common.py

32–36

Can you comment this change?

utils/update_cc_test_checks.py

190

Update comment.

[MaskRay]

Update comments

[MaskRay]

utils/UpdateTestChecks/asm.py

97 ↗

(On Diff #135710)

So that it will not warn that args (parsed command line options) does not have the attribute.

[arichardson]

Thanks for working on this!

It almost worked for me but it generated the following LABEL assertions for me
// CHECK-LABEL: test: which I then had to manually change to // CHECK-LABEL: define i32 @test(

[MaskRay]

In D42712#1022527, @arichardson wrote:

Thanks for working on this!

It almost worked for me but it generated the following LABEL assertions for me
// CHECK-LABEL: test: which I then had to manually change to // CHECK-LABEL: define i32 @test(

Thanks. The label is not kept as is because it has been scrubbed in the same way as tests are scrubbed by utils/update_test_checks.py

% utils/update_test_checks.py --opt-binary=~/Dev/llvm/release/bin/opt test/CodeGen/Generic/expand-experimental-reductions.ll
...............................
define i64 @or_i64(<2 x i64> %vec) {
; CHECK-LABEL: @or_i64(
; CHECK-NEXT:  entry:
; CHECK-NEXT:    [[RDX_SHUF:%.*]] = shufflevector <2 x i64> [[VEC:%.*]], <2 x i64> undef, <2 x i32> <i32 1, i32 undef>
; CHECK-NEXT:    [[BIN_RDX:%.*]] = or <2 x i64> [[VEC]], [[RDX_SHUF]]
; CHECK-NEXT:    [[TMP0:%.*]] = extractelement <2 x i64> [[BIN_RDX]], i32 0
; CHECK-NEXT:    ret i64 [[TMP0]]
;

[arichardson]

If I have the following:

// RUN: %clang_cc1 -emit-llvm -o - %s -O2 | FileCheck %s

int test(int a, int b) {
  return a + b;
}

After running update_cc1_checks.py I get the following test (which will not pass because of the broken label):

// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
// RUN: %clang_cc1 -emit-llvm -o - %s -O2 | FileCheck %s

// CHECK-LABEL: tests:
// CHECK:       entry:
// CHECK-NEXT:    %add = add nsw i32 %b, %a
// CHECK-NEXT:    ret i32 %add
int tests(int a, int b) {
  return a + b;
}

The label assertions doesn't work:

/Users/alex/devel/llvm/tools/clang/test/CodeGen/avr/test.c:7:17: error: expected string not found in input
// CHECK-LABEL: tests:
                ^
<stdin>:1:1: note: scanning from here
; ModuleID = '/Users/alex/devel/llvm/tools/clang/test/CodeGen/avr/test.c'
^
<stdin>:7:13: note: possible intended match here
define i32 @tests(i32 signext %a, i32 signext %b) local_unnamed_addr #0 {

utils/update_cc_test_checks.py

51

I noticed this does not work on MacOS because the mangled_name will be _test on OSX but the IR still uses @test, so the script does not add any IR checks

61

If I add the following lines here the script works on MacOS:

if mangled == "_" + spell:
  # HACK for MacOS (where the mangled name includes an _ for C but the IR won't):
  mangled = spell

[MaskRay]

Add arichardson's hack for Mac OS

[MaskRay]

utils/update_cc_test_checks.py

61

Thanks! Applied the hack

[arichardson]

LGTM, I guess the CHECK-LABEL thing can be fixed later.

[MaskRay]

In D42712#1025541, @arichardson wrote:

LGTM, I guess the CHECK-LABEL thing can be fixed later.

Sorry for committing without the revision being in the "approved" status , I did not know I should press "Add Action..." to address your comment and wait for you to remove the "Request Changes" status (the cross icon). Would you like me to revert this revision or send another revision after to fix this? I shall learn more about update_test_checks.py update_llc_test_checks.py

[arichardson]

No need to revert, I think it is fine to improve the script in follow-up revisions. Thanks again for working on this!

[MaskRay]

Created https://reviews.llvm.org/D44400

I had both asm and IR check lines in mind but broke it after making it IR only.

utils/UpdateTestChecks/asm.py:add_asm_checks used by utils/update_llc_test_checks.py, has some duplication and someone should fix it.

[vsk]

llvm/trunk/utils/update_cc_test_checks.py

100 ↗

(On Diff #136778)

Just a heads-up: find_executable doesn't seem to do the right thing here when clang is found within 'llvm_bin'. It rejects the full path to clang as it isn't technically in PATH.

108 ↗

(On Diff #136778)

Ditto.

[MaskRay]

llvm/trunk/utils/update_cc_test_checks.py

100 ↗

(On Diff #136778)

I think find_executable works as intended.

def find_executable(executable, path=None):
    ...
    if path is None:
        path = os.environ['PATH']

    paths = path.split(os.pathsep)
    base, ext = os.path.splitext(executable)

    if (sys.platform == 'win32') and (ext != '.exe'):
        executable = executable + '.exe'

    if not os.path.isfile(executable):
        for p in paths:
            f = os.path.join(p, executable)
            if os.path.isfile(f):
                # the file exists, we have a shot at spawn working
                return f
        return None
    else:
        return executable # if --clang is specified and is an absolute path to a regular file (or a symlink which can be resolved to a regular file)

utils/UpdateTestChecks/common.py

def should_add_line_to_output(input_line, prefix_set):
   if m and m.group(1) in prefix_set:
     return False
 
   return True
 
 # Invoke the tool that is being tested.
 def invoke_tool(exe, cmd_args, ir):
   with open(ir) as ir_file:
+    if isinstance(cmd_args, list):
+      stdout = subprocess.check_output([exe] + cmd_args, stdin=ir_file)
+    else:
       stdout = subprocess.check_output(exe + ' ' + cmd_args,
                                        shell=True, stdin=ir_file)
     if sys.version_info[0] > 2:
       stdout = stdout.decode()
   # Fix line endings to unix CR style.
   return stdout.replace('\r\n', '\n')
 
 ##### LLVM IR parser
 
 RUN_LINE_RE = re.compile('^\s*;\s*RUN:\s*(.*)$')

utils/update_cc_test_checks.py

+#!/usr/bin/env python3
+'''A utility to update LLVM IR or assembly CHECK lines in C/C++ FileCheck test files.
+
+Example RUN lines in .c/.cc test files:
+
+// RUN: %clang -S %s -o - -O2 | FileCheck %s
+// RUN: %clangxx -emit-llvm -S %s -o - -O2 | FileCheck -check-prefix=CHECK-A %s
+
+Usage:
+
+% utils/update_cc_test_checks.py --llvm-bin=release/bin test/a.cc
+% utils/update_cc_test_checks.py --c-index-test=release/bin/c-index-test \
+  --clang=release/bin/clang /tmp/c/a.cc
+'''
+
+import argparse
+import collections
+import distutils.spawn
+import os         # Used to advertise this file's name ("autogenerated_note").
+import shlex
+import string
+import subprocess
+import sys
+import re
+import tempfile
+
+from UpdateTestChecks import asm, common
+
+ADVERT = '// NOTE: Assertions have been autogenerated by '
+
+CHECK_RE = re.compile(r'^\s*//\s*([^:]+?)(?:-NEXT|-NOT|-DAG|-LABEL)?:')
+RUN_LINE_RE = re.compile('^//\s*RUN:\s*(.*)$')
+
+SUBST = {
+    '%clang': [],
+    '%clang_cc1': ['-cc1'],
+    '%clangxx': ['--driver-mode=g++'],
+}
+
+def get_line2spell_and_mangled(args, clang_args):
+  ret = {}
+  with tempfile.NamedTemporaryFile() as f:
+    # TODO Make c-index-test print mangled names without circumventing through precompiled headers
+    status = subprocess.run([args.c_index_test, '-write-pch', f.name, *clang_args],
+                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+    if status.returncode:
+      sys.stderr.write(status.stdout.decode())
+      sys.exit(2)
+    output = subprocess.check_output([args.c_index_test,
+        '-test-print-mangle', f.name])
+    if sys.version_info[0] > 2:
+      output = output.decode()
+
+  RE = re.compile(r'^FunctionDecl=(\w+):(\d+):\d+ \(Definition\) \[mangled=([^]]+)\]')
+  for line in output.splitlines():
+    m = RE.match(line)
+    if not m: continue
+    spell, line, mangled = m.groups()
+    # Note -test-print-mangle does not print file names so if #include is used,
+    # the line number may come from an included file.
+    ret[int(line)-1] = (spell, mangled)
+  if args.verbose:
+    for line, func_name in sorted(ret.items()):
+      print('line {}: found function {}'.format(line+1, func_name), file=sys.stderr)
+  return ret
+
+
+def main():
+  parser = argparse.ArgumentParser(
+      description=__doc__,
+      formatter_class=argparse.RawTextHelpFormatter)
+  parser.add_argument('-v', '--verbose', action='store_true')
+  parser.add_argument('--llvm-bin', help='llvm $prefix/bin path')
+  parser.add_argument('--clang',
+                      help='"clang" executable, defaults to $llvm_bin/clang')
+  parser.add_argument('--clang-args',
+                      help='Space-separated extra args to clang, e.g. --clang-args=-v')
+  parser.add_argument('--c-index-test',
+                      help='"c-index-test" executable, defaults to $llvm_bin/c-index-test')
+  parser.add_argument(
+      '--functions', nargs='+', help='A list of function name regexes. '
+      'If specified, update CHECK lines for functions matching at least one regex')
+  parser.add_argument(
+      '--x86_extra_scrub', action='store_true',
+      help='Use more regex for x86 matching to reduce diffs between various subtargets')
+  parser.add_argument('tests', nargs='+')
+  args = parser.parse_args()
+  args.clang_args = shlex.split(args.clang_args or '')
+
+  if args.clang is None:
+    if args.llvm_bin is None:
+      args.clang = 'clang'
+    else:
+      args.clang = os.path.join(args.llvm_bin, 'clang')
+  if not distutils.spawn.find_executable(args.clang):
+    print('Please specify --llvm-bin or --clang', file=sys.stderr)
+    return 1
+  if args.c_index_test is None:
+    if args.llvm_bin is None:
+      args.c_index_test = 'c-index-test'
+    else:
+      args.c_index_test = os.path.join(args.llvm_bin, 'c-index-test')
+  if not distutils.spawn.find_executable(args.c_index_test):
+    print('Please specify --llvm-bin or --c-index-test', file=sys.stderr)
+    return 1
+
+  autogenerated_note = (ADVERT + 'utils/' + os.path.basename(__file__))
+
+  for filename in args.tests:
+    with open(filename) as f:
+      input_lines = [l.rstrip() for l in f]
+
+    # Parse RUN lines and get `prefix_set`.
+    raw_lines = [m.group(1)
+                 for m in [RUN_LINE_RE.match(l) for l in input_lines] if m]
+    run_lines = [raw_lines[0]] if len(raw_lines) > 0 else []
+    for l in raw_lines[1:]:
+      if run_lines[-1].endswith("\\"):
+        run_lines[-1] = run_lines[-1].rstrip("\\") + " " + l
+      else:
+        run_lines.append(l)
+
+    if args.verbose:
+      print('Found {} RUN lines:'.format(len(run_lines)), file=sys.stderr)
+      for l in run_lines:
+        print('  RUN: ' + l, file=sys.stderr)
+
+    # Build a list of clang command lines from RUN lines.
+    run_list = []
+    line2spell_and_mangled_list = collections.defaultdict(list)
+    for l in run_lines:
+      commands = [cmd.strip() for cmd in l.split('|', 1)]
+
+      triple_in_cmd = None
+      m = common.TRIPLE_ARG_RE.search(commands[0])
+      if m:
+        triple_in_cmd = m.groups()[0]
+
+      # Apply %clang substitution rule, replace %s by `filename`, and append args.clang_args
+      clang_args = shlex.split(commands[0])
+      if clang_args[0] not in SUBST:
+        print('WARNING: Skipping non-clang RUN line: ' + l, file=sys.stderr)
+        continue
+      clang_args[0:1] = SUBST[clang_args[0]]
+      clang_args = [filename if i == '%s' else i for i in clang_args] + args.clang_args
+
+      # Extract -check-prefix in FileCheck args
+      filecheck_cmd = commands[-1]
+      if not filecheck_cmd.startswith('FileCheck '):
+        print('WARNING: Skipping non-FileChecked RUN line: ' + l, file=sys.stderr)
+        continue
+      check_prefixes = [item for m in common.CHECK_PREFIX_RE.finditer(filecheck_cmd)
+                               for item in m.group(1).split(',')]
+      if not check_prefixes:
+        check_prefixes = ['CHECK']
+      run_list.append((check_prefixes, clang_args, triple_in_cmd))
+
+    # Strip CHECK lines which are in `prefix_set`, update test file.
+    prefix_set = set([prefix for p in run_list for prefix in p[0]])
+    input_lines = []
+    with open(filename, 'r+') as f:
+      for line in f:
+        m = CHECK_RE.match(line)
+        if not (m and m.group(1) in prefix_set) and line != '//\n':
+          input_lines.append(line)
+      f.seek(0)
+      f.writelines(input_lines)
+      f.truncate()
+
+    # Execute clang, generate assembly, and extract functions.
+    func_dict = {}
+    for p in run_list:
+      prefixes = p[0]
+      for prefix in prefixes:
+        func_dict.update({prefix: dict()})
+    for prefixes, clang_args, triple_in_cmd in run_list:
+      if args.verbose:
+        print('Extracted clang cmd: clang {}'.format(clang_args), file=sys.stderr)
+        print('Extracted FileCheck prefixes: {}'.format(prefixes), file=sys.stderr)
+
+      raw_tool_output = common.invoke_tool(args.clang,
+                                           clang_args,
+                                           filename)
+
+      if '-emit-llvm' in clang_args:
+        common.build_function_body_dictionary(
+                common.OPT_FUNCTION_RE, common.scrub_body, [],
+                raw_tool_output, prefixes, func_dict, args.verbose)
+      else:
+        asm.build_function_body_dictionary_for_triple(args, raw_tool_output,
+            triple_in_cmd or 'x86', prefixes, func_dict)
+
+      # Invoke c-index-test to get mapping from start lines to mangled names.
+      # Forward all clang args for now.
+      for k, v in get_line2spell_and_mangled(args, clang_args).items():
+        line2spell_and_mangled_list[k].append(v)
+
+    output_lines = [autogenerated_note]
+    for idx, line in enumerate(input_lines):
+      # Discard any previous script advertising.
+      if line.startswith(ADVERT):
+        continue
+      if idx in line2spell_and_mangled_list:
+        added = set()
+        for spell, mangled in line2spell_and_mangled_list[idx]:
+          # One line main contain multiple function declarations.
+          # Skip if the mangled name has been added before.
+          # The line number may come from an included file,
+          # we simply require the spelling name to appear on the line
+          # to exclude functions from other files.
+          if mangled in added or spell not in line:
+            continue
+          added.add(mangled)
+          if args.functions is None or any(re.search(regex, spell) for regex in args.functions):
+            asm.add_asm_checks(output_lines, '//', run_list, func_dict, mangled)
+      output_lines.append(line.rstrip('\n'))
+
+    # Update the test file.
+    with open(filename, 'w') as f:
+      for line in output_lines:
+        f.write(line + '\n')
+
+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main())