[lit] Print full googletest commad line

[vitalybuka]

Summary

Similar to regular output of LIT tests:
https://github.com/llvm/llvm-project/blob/c162f086ba632ffaedfe92d63bf21571bc8ae4da/llvm/utils/lit/lit/TestRunner.py#L1569

[delcypher]

The patch seems reasonable. I have a minor nit about the use of the % operator but other than that it seems fine.

llvm/utils/lit/lit/formats/googletest.py

125

Minor nit. This patch is mixing the old python string interpolation (% operator) and the format function.

Given that Python 3.6 is the minimum maybe we should just use format strings in both places e.g.

header = f"Script:\n--\n{' '.join(cmd)}\n--\n"
...
return (lit.test.TIMEOUT, f"{header}Reached timeout of {litConfig.maxIndividualTestTime} seconds")

[vitalybuka]

upgrade formating

[vitalybuka]

Add UNRESOLVED test

[vitalybuka]

rebase

[rnk]

I see test failures on Windows that appear to be related to CRLF handling. I think the CHECK-NEXT checks aren't working because something is doubling up the newlines in the test output. That seems unrelated, so I suggest dropping the -NEXTs as necessary to get it past the presubmit testing.

[vitalybuka]

rebase

[vitalybuka]

it's just rebase so far, windows is not fixed
it does not reproduce for me so I will wait for another pre-merge

llvm/utils/lit/lit/formats/googletest.py

140

I assume we don't need it here

[vitalybuka]

fix test on Windows

[vitalybuka]

Issue is in the path separator.
lines like FAIL: googletest-upstream-format :: [[PATH:[Dd]ummy[Ss]ub[Dd]ir/]][[FILE:OneTest\.py]]/[[TEST:FirstTest\.subTestB]]
always have /
and command line is \

[vitalybuka]

Reuse vars in tests

[vitalybuka]

"x64 windows passed" now

[rnk]

Classic.

lgtm

llvm/utils/lit/lit/formats/googletest.py

def execute(self, test, litConfig):
         cmd = [testPath, '--gtest_filter=' + testName]
         cmd = self.prepareCmd(cmd)
         if litConfig.useValgrind:
             cmd = litConfig.valgrindArgs + cmd
 
         if litConfig.noExecute:
             return lit.Test.PASS, ''
 
+        header = f"Script:\n--\n{' '.join(cmd)}\n--\n"
+
         try:
             out, err, exitCode = lit.util.executeCommand(
                 cmd, env=test.config.environment,
                 timeout=litConfig.maxIndividualTestTime)
         except lit.util.ExecuteCommandTimeoutException:
             return (lit.Test.TIMEOUT,
-                    'Reached timeout of {} seconds'.format(
-                        litConfig.maxIndividualTestTime)
-                   )
+                    f'{header}Reached timeout of '
+                    f'{litConfig.maxIndividualTestTime} seconds')
 
         if exitCode:
-            return lit.Test.FAIL, out + err
+            return lit.Test.FAIL, header + out + err
 
         if '[  SKIPPED ] 1 test,' in out:
             return lit.Test.SKIPPED, ''
 
         passing_test_line = '[  PASSED  ] 1 test.'
         if passing_test_line not in out:
-            msg = ('Unable to find %r in gtest output:\n\n%s%s' %
-                   (passing_test_line, out, err))
-            return lit.Test.UNRESOLVED, msg
+            return (lit.Test.UNRESOLVED,
+                    f'{header}Unable to find {passing_test_line} '
+                    f'in gtest output:\n\n{out}{err}')
 
         return lit.Test.PASS,''
 
     def prepareCmd(self, cmd):
         """Insert interpreter if needed.
 
         It inserts the python exe into the command if cmd[0] ends in .py or caller
         specified run_under.

llvm/utils/lit/tests/googletest-format.py

 # Check the various features of the GoogleTest format.
 
 # FIXME: this test depends on order of tests
 # RUN: rm -f %{inputs}/googletest-format/.lit_test_times.txt
 
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-format > %t.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
-# CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestA
-# CHECK: FAIL: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB
-# CHECK-NEXT: *** TEST 'googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB' FAILED ***
+# CHECK: PASS: googletest-format :: [[PATH:[Dd]ummy[Ss]ub[Dd]ir/]][[FILE:OneTest\.py]]/FirstTest.subTestA
+# CHECK: FAIL: googletest-format :: [[PATH]][[FILE]]/[[TEST:FirstTest\.subTestB]]
+# CHECK-NEXT: *** TEST 'googletest-format :: [[PATH]][[FILE]]/[[TEST]]' FAILED ***
+# CHECK-NEXT: Script:
+# CHECK-NEXT: --
+# CHECK-NEXT: [[FILE]] --gtest_filter=[[TEST]]
+# CHECK-NEXT: --
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
 # CHECK: ***
-# CHECK: SKIPPED: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestC
-# CHECK: UNRESOLVED: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestD
-# CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/0.subTest
-# CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/1.subTest
+# CHECK: SKIPPED: googletest-format :: [[PATH]][[FILE]]/FirstTest.subTestC
+# CHECK: UNRESOLVED: googletest-format :: [[PATH]][[FILE]]/[[TEST:FirstTest\.subTestD]]
+# CHECK-NEXT: *** TEST 'googletest-format :: [[PATH]][[FILE]]/[[TEST]]' FAILED ***
+# CHECK-NEXT: Script:
+# CHECK-NEXT: --
+# CHECK-NEXT: [[FILE]] --gtest_filter=[[TEST]]
+# CHECK-NEXT: --
+# CHECK-NEXT: Unable to find [ PASSED ] 1 test. in gtest output
+# CHECK: I am subTest D, I am UNRESOLVED
+# CHECK: PASS: googletest-format :: [[PATH]][[FILE]]/ParameterizedTest/0.subTest
+# CHECK: PASS: googletest-format :: [[PATH]][[FILE]]/ParameterizedTest/1.subTest
 # CHECK: Failed Tests (1)
 # CHECK: Skipped{{ *}}: 1
 # CHECK: Passed{{ *}}: 3
 # CHECK: Unresolved{{ *}}: 1
 # CHECK: Failed{{ *}}: 1

llvm/utils/lit/tests/googletest-timeout.py

 # the configuration file
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
 # RUN:  --filter=InfiniteLoopSubTest  --param set_timeout=1 \
 # RUN:  > %t.cfgset.out
 # RUN: FileCheck --check-prefix=CHECK-INF < %t.cfgset.out %s
 
 # CHECK-INF: -- Testing:
-# CHECK-INF: TIMEOUT: googletest-timeout :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/T.InfiniteLoopSubTest
+# CHECK-INF: TIMEOUT: googletest-timeout :: [[PATH:[Dd]ummy[Ss]ub[Dd]ir/]][[FILE:OneTest\.py]]/[[TEST:T\.InfiniteLoopSubTest]]
+# CHECK-INF-NEXT: ******************** TEST 'googletest-timeout :: [[PATH]][[FILE]]/[[TEST]]' FAILED ********************
+# CHECK-INF-NEXT: Script:
+# CHECK-INF-NEXT: --
+# CHECK-INF-NEXT: [[FILE]] --gtest_filter=[[TEST]]
+# CHECK-INF-NEXT: --
 # CHECK-INF: Timed Out: 1
 
 ###############################################################################
 # Check tests can complete with a timeout set
 #
 # `QuickSubTest` should execute quickly so we shouldn't wait anywhere near the
 # 3600 second timeout.
 ###############################################################################

llvm/utils/lit/tests/googletest-upstream-format.py

 # Check the various features of the GoogleTest format.
 
 # FIXME: this test depends on order of tests
 # RUN: rm -f %{inputs}/googletest-upstream-format/.lit_test_times.txt
 
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-upstream-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
-# CHECK: PASS: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestA
-# CHECK: FAIL: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB
-# CHECK-NEXT: *** TEST 'googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB' FAILED ***
+# CHECK: PASS: googletest-upstream-format :: [[PATH:[Dd]ummy[Ss]ub[Dd]ir/]][[FILE:OneTest\.py]]/FirstTest.subTestA
+# CHECK: FAIL: googletest-upstream-format :: [[PATH]][[FILE]]/[[TEST:FirstTest\.subTestB]]
+# CHECK-NEXT: *** TEST 'googletest-upstream-format :: [[PATH]][[FILE]]/[[TEST]]' FAILED ***
+# CHECK-NEXT: Script:
+# CHECK-NEXT: --
+# CHECK-NEXT: [[FILE]] --gtest_filter=[[TEST]]
+# CHECK-NEXT: --
 # CHECK-NEXT: Running main() from gtest_main.cc
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
+# CHECK: SKIPPED: googletest-upstream-format :: [[PATH]][[FILE]]/FirstTest.subTestC
+# CHECK: UNRESOLVED: googletest-upstream-format :: [[PATH]][[FILE]]/[[TEST:FirstTest\.subTestD]]
+# CHECK-NEXT: *** TEST 'googletest-upstream-format :: [[PATH]][[FILE]]/[[TEST]]' FAILED ***
+# CHECK-NEXT: Script:
+# CHECK-NEXT: --
+# CHECK-NEXT: [[FILE]] --gtest_filter=[[TEST]]
+# CHECK-NEXT: --
+# CHECK-NEXT: Unable to find [ PASSED ] 1 test. in gtest output
+# CHECK: I am subTest D, I am UNRESOLVED
 # CHECK: ***
-# CHECK: SKIPPED: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestC
-# CHECK: UNRESOLVED: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestD
-# CHECK: PASS: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/0.subTest
-# CHECK: PASS: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/1.subTest
+# CHECK: PASS: googletest-upstream-format :: [[PATH]][[FILE]]/ParameterizedTest/0.subTest
+# CHECK: PASS: googletest-upstream-format :: [[PATH]][[FILE]]/ParameterizedTest/1.subTest
 # CHECK: Failed Tests (1)
 # CHECK: Skipped{{ *}}: 1
 # CHECK: Passed{{ *}}: 3
 # CHECK: Unresolved{{ *}}: 1
 # CHECK: Failed{{ *}}: 1