[lit][gtest] Support SKIPPED tests

[ikudrin]

Summary

This updates the googletest format to support tests that use GTEST_SKIP(), which is now available with the updated googletest framework.

[lhames]

FWIW this looks reasonable to me, but I'm not a lit expert.

[ikudrin]

I am not really sure who can review this. If anyone knows the right person, please extend the list.

[thopre]

llvm/utils/lit/tests/googletest-format.py

25–27

Is this ordered guaranteed? If not, we should use CHECK-DAG.

[ikudrin]

llvm/utils/lit/tests/googletest-format.py

25–27

print_summary() in lit/main.py prints them in the exact order as ResultCodes are defined in lit/Test.py, so the sequence is stable.

[thopre]

LGTM then

[delcypher]

llvm/utils/lit/lit/formats/googletest.py

138

Looking for this exact string seems fragile. Maybe use a regex that can cope with a variable amount of spaces?

[ikudrin]

llvm/utils/lit/lit/formats/googletest.py

138

This corresponds to how the framework prints the summary, see PrettyUnitTestResultPrinter::OnTestIterationEnd(). If that is broken in some future release of the framework, we will fix the condition, but before that, we cannot predict in which way the report may be changed, so there is no need to overcomplicate the code.

llvm/utils/lit/lit/formats/googletest.py

def execute(self, test, litConfig):
             return (lit.Test.TIMEOUT,
                     'Reached timeout of {} seconds'.format(
                         litConfig.maxIndividualTestTime)
                    )
 
         if exitCode:
             return lit.Test.FAIL, out + err
 
+        if '[  SKIPPED ] 1 test,' in out:
+            return lit.Test.SKIPPED, ''
+
         passing_test_line = '[  PASSED  ] 1 test.'
         if passing_test_line not in out:
             msg = ('Unable to find %r in gtest output:\n\n%s%s' %
                    (passing_test_line, out, err))
             return lit.Test.UNRESOLVED, msg
 
         return lit.Test.PASS,''
 

llvm/utils/lit/tests/Inputs/googletest-format/DummySubDir/OneTest.py

 #!/usr/bin/env python
 
 import sys
 
 if len(sys.argv) != 2:
     raise ValueError("unexpected number of args")
 
 if sys.argv[1] == "--gtest_list_tests":
     print("""\
 FirstTest.
   subTestA
   subTestB
+  subTestC
 ParameterizedTest/0.
   subTest
 ParameterizedTest/1.
   subTest""")
     sys.exit(0)
 elif not sys.argv[1].startswith("--gtest_filter="):
     raise ValueError("unexpected argument: %r" % (sys.argv[1]))
 
 test_name = sys.argv[1].split('=',1)[1]
 if test_name == 'FirstTest.subTestA':
     print('I am subTest A, I PASS')
     print('[  PASSED  ] 1 test.')
     sys.exit(0)
 elif test_name == 'FirstTest.subTestB':
     print('I am subTest B, I FAIL')
     print('And I have two lines of output')
     sys.exit(1)
+elif test_name == 'FirstTest.subTestC':
+    print('I am subTest C, I am SKIPPED')
+    print('[  PASSED  ] 0 tests.')
+    print('[  SKIPPED ] 1 test, listed below:')
+    print('[  SKIPPED ] FirstTest.subTestC')
+    sys.exit(0)
 elif test_name in ('ParameterizedTest/0.subTest',
                    'ParameterizedTest/1.subTest'):
     print('I am a parameterized test, I also PASS')
     print('[  PASSED  ] 1 test.')
     sys.exit(0)
 else:
     raise SystemExit("error: invalid test name: %r" % (test_name,))

llvm/utils/lit/tests/googletest-format.py

 
 # CHECK: -- Testing:
 # CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestA
 # CHECK: FAIL: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB
 # CHECK-NEXT: *** TEST 'googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB' FAILED ***
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
 # CHECK: ***
+# CHECK: SKIPPED: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestC
 # CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/0.subTest
 # CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/1.subTest
 # CHECK: Failed Tests (1)
-# CHECK: Passed: 3
-# CHECK: Failed: 1
-
+# CHECK: Skipped: 1
+# CHECK: Passed : 3
+# CHECK: Failed : 1