[LIT] Fix failing LIT tests

[EricWF]

Summary

I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine.

I made the following changes for the following reasons.

1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes.
2. discovery.py: The output appears in a different order on my machine than it did in the test.
3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test.
4. The classname is now formed differently in getJUnitXML(...).

I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts?

[jroelofs]

LGTM

utils/lit/tests/unittest-adaptor.py

7

I don't know what controls the order, but I'm not sure we care? Maybe should make these CHECK-DAGs...

[jroelofs]

One more note: why didn't the buildbots catch this?

[EricWF]

In D9864#179372, @jroelofs wrote:

One more note: why didn't the buildbots catch this?

I don't see anything that creates a check-lit rule in CMake or something similar for autoconf. My guess is that they never get run.

[jroelofs]

ping: @EricWF, you should commit this....

[EricWF]

• fix tests

utils/lit/tests/Inputs/googletest-format/DummySubDir/OneTest

ParameterizedTest/1.
   subTest""")
     sys.exit(0)
 elif not sys.argv[1].startswith("--gtest_filter="):
     raise ValueError("unexpected argument: %r" % (sys.argv[1]))
 
 test_name = sys.argv[1].split('=',1)[1]
 if test_name == 'FirstTest.subTestA':
     print('I am subTest A, I PASS')
+    print('[  PASSED  ] 1 test.')
     sys.exit(0)
 elif test_name == 'FirstTest.subTestB':
     print('I am subTest B, I FAIL')
     print('And I have two lines of output')
     sys.exit(1)
 elif test_name in ('ParameterizedTest/0.subTest',
                    'ParameterizedTest/1.subTest'):
     print('I am a parameterized test, I also PASS')
+    print('[  PASSED  ] 1 test.')
     sys.exit(0)
 else:
     raise SystemExit("error: invalid test name: %r" % (test_name,))

utils/lit/tests/discovery.py

 # Check the basic discovery process, including a sub-suite.
 #
 # RUN: %{lit} %{inputs}/discovery \
 # RUN:   -j 1 --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-BASIC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-BASIC-ERR < %t.err %s
 #
 # CHECK-BASIC-ERR: loading suite config '{{.*}}/discovery/lit.cfg'
-# CHECK-BASIC-ERR: loading local config '{{.*}}/discovery/subdir/lit.local.cfg'
 # CHECK-BASIC-ERR: loading suite config '{{.*}}/discovery/subsuite/lit.cfg'
+# CHECK-BASIC-ERR: loading local config '{{.*}}/discovery/subdir/lit.local.cfg'
 #
 # CHECK-BASIC-OUT: -- Test Suites --
 # CHECK-BASIC-OUT:   sub-suite - 2 tests
 # CHECK-BASIC-OUT:     Source Root: {{.*/discovery/subsuite$}}
 # CHECK-BASIC-OUT:     Exec Root  : {{.*/discovery/subsuite$}}
 # CHECK-BASIC-OUT:   top-level-suite - 3 tests
 # CHECK-BASIC-OUT:     Source Root: {{.*/discovery$}}
 # CHECK-BASIC-OUT:     Exec Root  : {{.*/discovery$}}

 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-ERR < %t.err %s
 #
 # CHECK-ASEXEC-ERR: loading suite config '{{.*}}/exec-discovery/lit.site.cfg'
 # CHECK-ASEXEC-ERR: load_config from '{{.*}}/discovery/lit.cfg'
 # CHECK-ASEXEC-ERR: loaded config '{{.*}}/discovery/lit.cfg'
 # CHECK-ASEXEC-ERR: loaded config '{{.*}}/exec-discovery/lit.site.cfg'
-# CHECK-ASEXEC-ERR: loading local config '{{.*}}/discovery/subdir/lit.local.cfg'
 # CHECK-ASEXEC-ERR: loading suite config '{{.*}}/discovery/subsuite/lit.cfg'
+# CHECK-ASEXEC-ERR: loading local config '{{.*}}/discovery/subdir/lit.local.cfg'
 #
 # CHECK-ASEXEC-OUT: -- Test Suites --
 # CHECK-ASEXEC-OUT:   sub-suite - 2 tests
 # CHECK-ASEXEC-OUT:     Source Root: {{.*/discovery/subsuite$}}
 # CHECK-ASEXEC-OUT:     Exec Root  : {{.*/discovery/subsuite$}}
 # CHECK-ASEXEC-OUT:   top-level-suite - 3 tests
 # CHECK-ASEXEC-OUT:     Source Root: {{.*/discovery$}}
 # CHECK-ASEXEC-OUT:     Exec Root  : {{.*/exec-discovery$}}

utils/lit/tests/unittest-adaptor.py

 # Check the lit adaption to run under unittest.
 #
 # RUN: %{python} %s %{inputs}/unittest-adaptor 2> %t.err
 # RUN: FileCheck < %t.err %s
 #
-# CHECK: unittest-adaptor :: test-one.txt ... ok
-# CHECK: unittest-adaptor :: test-two.txt ... FAIL
+# CHECK-DAG: unittest-adaptor :: test-two.txt ... FAIL
+# CHECK-DAG: unittest-adaptor :: test-one.txt ... ok
 
 import unittest
 import sys
 
 import lit
 import lit.discovery
 
 input_path = sys.argv[1]
 unittest_suite = lit.discovery.load_test_suite([input_path])
 runner = unittest.TextTestRunner(verbosity=2)
 runner.run(unittest_suite)

utils/lit/tests/xunit-output.py

 # Check xunit output
 # RUN: %{lit} --xunit-xml-output %t.xunit.xml %{inputs}/test-data
 # RUN: FileCheck < %t.xunit.xml %s
 
 # CHECK: <?xml version="1.0" encoding="UTF-8" ?>
 # CHECK: <testsuites>
 # CHECK: <testsuite name='test-data' tests='1' failures='0'>
-# CHECK: <testcase classname='test-data.' name='metrics.ini' time='0.00'/>
+# CHECK: <testcase classname='test-data.test-data' name='metrics.ini' time='0.{{[0-9]+}}'/>
 # CHECK: </testsuite>
-# CHECK: </testsuites>
+# CHECK: </testsuites>