Run non-filechecked commands in update_cc_test_checks.py

[ggeorgakoudis]

Summary

Some tests in clang require running non-filechecked commands to generate the actual filecheck input. For example, tests for openmp offloading require generating the host bc without any checking, before running the clang command to actually generate the filechecked IR of the target device. This patch enables update_cc_test_checks.py to run non-filechecked run lines in-place.

[ggeorgakoudis]

Update implementation to support running non-clang runlines, add tests

[jdoerfert]

LG, thx!

[thakis]

Looks like this breaks tests on mac: http://45.33.8.238/macm1/5075/step_6.txt

Please take a look, and revert for now if it takes a while to fix.

[ggeorgakoudis]

In D97068#2611269, @thakis wrote:

Looks like this breaks tests on mac: http://45.33.8.238/macm1/5075/step_6.txt

Please take a look, and revert for now if it takes a while to fix.

Hey @thakis, it should be an easy fix. I see no check lines are generated and I suspect something may be going wrong with the cp runline, but it's hard to know without replicating it. Is there any way you can give me temporary access to the mac machine to test things out?

Otherwise, I will simplify the test to exclude the cp runline.

[arichardson]

IMO this should also handle the %t substitution and create a temporary directory. Otherwise we end up writing files to whatever the current cwd is.

[jroelofs]

In D97068#2611696, @ggeorgakoudis wrote:

In D97068#2611269, @thakis wrote:

Looks like this breaks tests on mac: http://45.33.8.238/macm1/5075/step_6.txt

Please take a look, and revert for now if it takes a while to fix.

Hey @thakis, it should be an easy fix. I see no check lines are generated and I suspect something may be going wrong with the cp runline, but it's hard to know without replicating it. Is there any way you can give me temporary access to the mac machine to test things out?

Otherwise, I will simplify the test to exclude the cp runline.

The issue has something to do with Darwin's USER_LABEL_PREFIX mangling inserting an _ prefix. This "fixes" the test, but doesn't solve whatever underlying problem is there:

diff --git a/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c b/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c
index bf18179392dc..a0a826c3d0b4 100644
--- a/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c
+++ b/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c
@@ -1,7 +1,7 @@
 // Check that the non-clang/non-filechecked runlines execute
 // RUN: cp %s %s.copy.c
-// RUN: %clang_cc1 -fopenmp %s.copy.c -emit-llvm-bc -o %t-host.bc
-// RUN: %clang_cc1 -fopenmp -fopenmp-host-ir-file-path %t-host.bc %s.copy.c -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fopenmp %s.copy.c -emit-llvm-bc -o %t-host.bc
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fopenmp -fopenmp-host-ir-file-path %t-host.bc %s.copy.c -emit-llvm -o - | FileCheck %s
 
 void use(int);
 
diff --git a/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c.expected b/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c.expected
index ffc8caac0f23..3ad33fc683bd 100644
--- a/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c.expected
+++ b/clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c.expected
@@ -1,8 +1,8 @@
 // NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
 // Check that the non-clang/non-filechecked runlines execute
 // RUN: cp %s %s.copy.c
-// RUN: %clang_cc1 -fopenmp %s.copy.c -emit-llvm-bc -o %t-host.bc
-// RUN: %clang_cc1 -fopenmp -fopenmp-host-ir-file-path %t-host.bc %s.copy.c -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fopenmp %s.copy.c -emit-llvm-bc -o %t-host.bc
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fopenmp -fopenmp-host-ir-file-path %t-host.bc %s.copy.c -emit-llvm -o - | FileCheck %s
 
 void use(int);

You should be able to replicate the failure locally by setting an apple triple instead, i.e. -triple x86_64-apple-macosx.

[jroelofs]

Reverted in a24644bb1ce09b40c2d751569dd5bb37ea9c995d to get the bots green again. Feel free to re-land with a fix.

[ggeorgakoudis]

@jroelofs, thank you very much for the much useful information. I will try to replicate the problem with the apple triple and fix it, if possible, or otherwise use your patch to avoid the issue.

[ggeorgakoudis]

Add triple to avoid spurious failures in tests

[jroelofs]

In D97068#2617642, @ggeorgakoudis wrote:

Add triple to avoid spurious failures in tests

They're not really spurious, right... Doesn't this mean that utils/update_cc_test_checks.py is now broken for people who develop on a mac?

[ggeorgakoudis]

In D97068#2617751, @jroelofs wrote:

In D97068#2617642, @ggeorgakoudis wrote:

Add triple to avoid spurious failures in tests

They're not really spurious, right... Doesn't this mean that utils/update_cc_test_checks.py is now broken for people who develop on a mac?

It's spurious in the sense that it's not this commit that breaks mac tests. Testing on mac is already broken. By investigating the issue, you are right that the USER_LABEL_PREFIX _ is skipping generating the checks because function names mismatch in IR output. I suppose the issue should be fixed by identifying a mac triple and removing the _ prefix?

clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c

+// Check that the non-clang/non-filechecked runlines execute
+// RUN: cp %s %s.copy.c
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fopenmp %s.copy.c -emit-llvm-bc -o %t-host.bc
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fopenmp -fopenmp-host-ir-file-path %t-host.bc %s.copy.c -emit-llvm -o - | FileCheck %s
+
+void use(int);
+
+void test(int a)
+{
+}

clang/test/utils/update_cc_test_checks/Inputs/exec-all-runlines.c.expected

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// Check that the non-clang/non-filechecked runlines execute
+// RUN: cp %s %s.copy.c
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fopenmp %s.copy.c -emit-llvm-bc -o %t-host.bc
+// RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fopenmp -fopenmp-host-ir-file-path %t-host.bc %s.copy.c -emit-llvm -o - | FileCheck %s
+
+void use(int);
+
+// CHECK-LABEL: @test(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[A_ADDR:%.*]] = alloca i32, align 4
+// CHECK-NEXT:    store i32 [[A:%.*]], i32* [[A_ADDR]], align 4
+// CHECK-NEXT:    ret void
+//
+void test(int a)
+{
+}

clang/test/utils/update_cc_test_checks/exec-all-runlines.test

+## Test that non-clang/non-filechecked runlines execute
+
+# RUN: cp %S/Inputs/exec-all-runlines.c %t-generated.c && %update_cc_test_checks %t-generated.c
+# RUN: diff -u %S/Inputs/exec-all-runlines.c.expected %t-generated.c
+
+## Check that re-running update_cc_test_checks doesn't change the output
+# RUN: %update_cc_test_checks %t-generated.c
+# RUN: diff -u %S/Inputs/exec-all-runlines.c.expected %t-generated.c

llvm/utils/update_cc_test_checks.py

if '-emit-llvm' in clang_args:
     builder.process_run_line(
             common.OPT_FUNCTION_RE, common.scrub_body, raw_tool_output,
             prefixes)
   else:
     print('The clang command line should include -emit-llvm as asm tests '
           'are discouraged in Clang testsuite.', file=sys.stderr)
     sys.exit(1)
 
+def exec_run_line(exe):
+  popen = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
+  stdout, stderr = popen.communicate()
+  if popen.returncode != 0:
+    sys.stderr.write('Failed to run ' + ' '.join(exe) + '\n')
+    sys.stderr.write(stderr)
+    sys.stderr.write(stdout)
+    sys.exit(3)
 
 def main():
   initial_args, parser = config()
   script_name = os.path.basename(__file__)
 
   for ti in common.itertests(initial_args.tests, parser, 'utils/' + script_name,
                              comment_prefix='//', argparse_callback=infer_dependent_args):
     # Build a list of clang command lines and check prefixes from RUN lines.
     run_list = []
     line2spell_and_mangled_list = collections.defaultdict(list)
     for l in ti.run_lines:
       commands = [cmd.strip() for cmd in l.split('|')]
 
       triple_in_cmd = None
       m = common.TRIPLE_ARG_RE.search(commands[0])
       if m:
         triple_in_cmd = m.groups()[0]
 
-      # Apply %clang substitution rule, replace %s by `filename`, and append args.clang_args
-      clang_args = shlex.split(commands[0])
-      if clang_args[0] not in SUBST:
-        print('WARNING: Skipping non-clang RUN line: ' + l, file=sys.stderr)
+      # Parse executable args.
+      exec_args = shlex.split(commands[0])
+      # Execute non-clang runline.
+      if exec_args[0] not in SUBST:
+        print('NOTE: Executing non-clang RUN line: ' + l, file=sys.stderr)
+        # Replace %s by `filename`.
+        exec_args = [i.replace('%s', ti.path) if '%s' in i else i for i in exec_args]
+        exec_run_line(exec_args)
         continue
+      # This is a clang runline, apply %clang substitution rule, replace %s by `filename`,
+      # and append args.clang_args
+      clang_args = exec_args
       clang_args[0:1] = SUBST[clang_args[0]]
-      clang_args = [ti.path if i == '%s' else i for i in clang_args] + ti.args.clang_args
-
-      # Permit piping the output through opt
-      if not (len(commands) == 2 or
-              (len(commands) == 3 and commands[1].startswith('opt'))):
-        print('WARNING: Skipping non-clang RUN line: ' + l, file=sys.stderr)
+      clang_args = [i.replace('%s', ti.path) if '%s' in i else i for i in clang_args] + ti.args.clang_args
 
       # Extract -check-prefix in FileCheck args
       filecheck_cmd = commands[-1]
       common.verify_filecheck_prefixes(filecheck_cmd)
       if not filecheck_cmd.startswith('FileCheck '):
-        print('WARNING: Skipping non-FileChecked RUN line: ' + l, file=sys.stderr)
+        print('NOTE: Executing non-FileChecked clang RUN line: ' + l, file=sys.stderr)
+        # Execute non-filechecked clang runline.
+        exe = [ti.args.clang] + clang_args
+        exec_run_line(exe)
         continue
+
       check_prefixes = [item for m in common.CHECK_PREFIX_RE.finditer(filecheck_cmd)
                                for item in m.group(1).split(',')]
       if not check_prefixes:
         check_prefixes = ['CHECK']
       run_list.append((check_prefixes, clang_args, commands[1:-1], triple_in_cmd))
 
     # Execute clang, generate LLVM IR, and extract functions.