[lit] Include unexecuted tests in xUnit report

[yln]

Summary

Pass in all discovered tests to report generators.

The XunitReport generator now creates testcase items for unexecuted
tests and documents why they have been skipped. This makes it easier
to compare test runs with different filters or configurations, or across
platforms.

I don't know who is using the JsonReport generator and what the
expectations there are (it doesn't have tests), so decided to preserve
the old behavior by filtering out the unexecuted tests.

[jdenny]

For xunit support, LGTM.

I also don't know whether junit support is worth testing. It might be good to wait a few days in case someone has an opinion on that.

llvm/utils/lit/tests/Inputs/xunit-output/pass.ini

6

Do you know why this results is never tested?

[yln]

Thanks for the prompt review! I will wait for a week until landing this.

llvm/utils/lit/tests/Inputs/xunit-output/pass.ini

6

The only reason this is here is because the class DummyFormat from dummy_format.py (see above) expects this. This file has been copy&pasted :/ for a couple of tests and I wanted to avoid diverging this copy further. (I already added the required_feature property.)

The [results] part is used by one of the copies of the file in another test:
https://github.com/llvm/llvm-project/blob/master/llvm/utils/lit/tests/Inputs/test-data/metrics.ini#L6
https://github.com/llvm/llvm-project/blob/master/llvm/utils/lit/tests/test-data.py#L10

[jdenny]

llvm/utils/lit/tests/Inputs/xunit-output/pass.ini

6

Ah, it seems I misgrepped and overlooked that. Thanks!

[yln]

This causes "unresolved tests" on one bot:
http://lab.llvm.org:8011/builders/clang-ppc64be-linux-multistage/builds/26333/steps/ninja%20check%202/logs/stdio

Exception during script execution:
  Traceback (most recent call last):
  File "/home/buildbots/ppc64be-clang-multistage-test/clang-ppc64be-multistage/llvm/llvm/utils/lit/lit/worker.py", line 73, in _execute_test_handle_errors
  result = test.config.test_format.execute(test, lit_config)
  File "/home/buildbots/ppc64be-clang-multistage-test/clang-ppc64be-multistage/stage2/utils/lit/tests/Inputs/xunit-output/dummy_format.py", line 26, in execute
  required_feature = cfg.get('global', 'required_feature', fallback=None)
  TypeError: get() got an unexpected keyword argument 'fallback'

[yln]

Should be fixed in follow-up: 853e61142d3cf8f54f3c85363b8cdb0622bd308b

llvm/utils/lit/lit/main.py

def main(builtin_params={}):
     selected_tests = selected_tests[:opts.max_tests]
 
     mark_excluded(discovered_tests, selected_tests)
 
     start = time.time()
     run_tests(selected_tests, lit_config, opts, len(discovered_tests))
     elapsed = time.time() - start
 
-    # TODO(yln): eventually, all functions below should act on discovered_tests
-    executed_tests = [
-        t for t in selected_tests if t.result.code != lit.Test.SKIPPED]
-
     if opts.time_tests:
         print_histogram(discovered_tests)
 
     print_results(discovered_tests, elapsed, opts)
 
     for report in opts.reports:
-        report.write_results(executed_tests, elapsed)
+        report.write_results(discovered_tests, elapsed)
 
     if lit_config.numErrors:
         sys.stderr.write('\n%d error(s) in tests\n' % lit_config.numErrors)
         sys.exit(2)
 
     if lit_config.numWarnings:
         sys.stderr.write('\n%d warning(s) in tests\n' % lit_config.numWarnings)
 

llvm/utils/lit/lit/reports.py

def by_suite_and_test_path(test):
     return (test.suite.name, id(test.suite), test.path_in_suite)
 
 
 class JsonReport(object):
     def __init__(self, output_file):
         self.output_file = output_file
 
     def write_results(self, tests, elapsed):
-        assert not any(t.result.code in {lit.Test.EXCLUDED, lit.Test.SKIPPED} for t in tests)
+        unexecuted_codes = {lit.Test.EXCLUDED, lit.Test.SKIPPED}
+        tests = [t for t in tests if t.result.code not in unexecuted_codes]
         # Construct the data we will write.
         data = {}
         # Encode the current lit version as a schema version.
         data['__version__'] = lit.__versioninfo__
         data['elapsed'] = elapsed
         # FIXME: Record some information on the lit configuration used?
         # FIXME: Record information from the individual test suites?
 

 class XunitReport(object):
     def __init__(self, output_file):
         self.output_file = output_file
         self.skipped_codes = {lit.Test.EXCLUDED,
                               lit.Test.SKIPPED, lit.Test.UNSUPPORTED}
 
     # TODO(yln): elapsed unused, put it somewhere?
     def write_results(self, tests, elapsed):
-        assert not any(t.result.code in {lit.Test.EXCLUDED, lit.Test.SKIPPED} for t in tests)
         tests.sort(key=by_suite_and_test_path)
         tests_by_suite = itertools.groupby(tests, lambda t: t.suite)
 
         with open(self.output_file, 'w') as file:
             file.write('<?xml version="1.0" encoding="UTF-8"?>\n')
             file.write('<testsuites>\n')
             for suite, test_iter in tests_by_suite:
                 self._write_testsuite(file, suite, list(test_iter))
def _get_skip_reason(self, test):
             return 'Test not selected (--filter, --max-tests, --run-shard)'
         if code == lit.Test.SKIPPED:
             return 'User interrupt'
 
         assert code == lit.Test.UNSUPPORTED
         features = test.getMissingRequiredFeatures()
         if features:
             return 'Missing required feature(s): ' + ', '.join(features)
-        return 'Skipping because of configuration'
+        return 'Unsupported configuration'

llvm/utils/lit/tests/Inputs/xunit-output/dummy_format.py

def execute(self, test, lit_config):
         cfg.read(source_path)
 
         # Create the basic test result.
         result_code = cfg.get('global', 'result_code')
         result_output = cfg.get('global', 'result_output')
         result = lit.Test.Result(getattr(lit.Test, result_code),
                                  result_output)
 
+        required_feature = cfg.get('global', 'required_feature', fallback=None)
+        if required_feature:
+            test.requires.append(required_feature)
+
         # Load additional metrics.
         for key,value_str in cfg.items('results'):
             value = eval(value_str)
             if isinstance(value, int):
                 metric = lit.Test.IntMetricValue(value)
             elif isinstance(value, float):
                 metric = lit.Test.RealMetricValue(value)
             else:
                 raise RuntimeError("unsupported result type")
             result.addMetric(key, metric)
 
         return result
 

llvm/utils/lit/tests/Inputs/xunit-output/excluded.ini

+[global]
+result_code = EXCLUDED
+result_output = not shown
+
+[results]

llvm/utils/lit/tests/Inputs/xunit-output/missing_feature.ini

+[global]
+result_code = UNSUPPORTED
+result_output = not shown
+
+required_feature = dummy_feature
+
+[results]

llvm/utils/lit/tests/Inputs/xunit-output/pass.ini

+[global]
+result_code = PASS
+result_output = not shown
+
+[results]

llvm/utils/lit/tests/Inputs/xunit-output/unsupported.ini

+[global]
+result_code = UNSUPPORTED
+result_output = not shown
+
+[results]

llvm/utils/lit/tests/shtest-format.py

 
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-triple.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT:<skipped message="Missing required feature(s): x86_64"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-expr-false.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-expr-true.txt" time="{{[0-9]+\.[0-9]+}}">
-# XUNIT-NEXT:<skipped message="Skipping because of configuration"/>
+# XUNIT-NEXT:<skipped message="Unsupported configuration"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-star.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
 # XUNIT: </failure>
 # XUNIT-NEXT: </testcase>
 
 # XUNIT: <testcase classname="shtest-format.unsupported_dir" name="some-test.txt" time="{{[0-9]+\.[0-9]+}}">
-# XUNIT-NEXT:<skipped message="Skipping because of configuration"/>
+# XUNIT-NEXT:<skipped message="Unsupported configuration"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-expr-false.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-expr-true.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-feature.txt" time="{{[0-9]+\.[0-9]+}}"/>
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-target.txt" time="{{[0-9]+\.[0-9]+}}"/>

llvm/utils/lit/tests/xunit-output.py

 # REQUIRES: shell
 
 # Check xunit output
 # RUN: rm -rf %t.xunit.xml
 # RUN: not %{lit} --xunit-xml-output %t.xunit.xml %{inputs}/xunit-output
 # If xmllint is installed verify that the generated xml is well-formed
 # RUN: sh -c 'if command -v xmllint 2>/dev/null; then xmllint --noout %t.xunit.xml; fi'
 # RUN: FileCheck < %t.xunit.xml %s
 
-# CHECK: <?xml version="1.0" encoding="UTF-8"?>
-# CHECK: <testsuites>
-# CHECK: <testsuite name="test-data" tests="1" failures="1" skipped="0">
-# CHECK: <testcase classname="test-data.test-data" name="bad&amp;name.ini" time="{{[0-1]}}.{{[0-9]+}}">
-# CHECK-NEXT: <failure><![CDATA[& < > ]]]]><![CDATA[> &"]]></failure>
-# CHECK: </testsuite>
-# CHECK: </testsuites>
+# CHECK:      <?xml version="1.0" encoding="UTF-8"?>
+# CHECK-NEXT: <testsuites>
+# CHECK-NEXT: <testsuite name="test-data" tests="5" failures="1" skipped="3">
+# CHECK-NEXT: <testcase classname="test-data.test-data" name="bad&amp;name.ini" time="{{[0-1]\.[0-9]+}}">
+# CHECK-NEXT:   <failure><![CDATA[& < > ]]]]><![CDATA[> &"]]></failure>
+# CHECK-NEXT: </testcase>
+# CHECK-NEXT: <testcase classname="test-data.test-data" name="excluded.ini" time="{{[0-1]\.[0-9]+}}">
+# CHECK-NEXT:   <skipped message="Test not selected (--filter, --max-tests, --run-shard)"/>
+# CHECK-NEXT: </testcase>
+# CHECK-NEXT: <testcase classname="test-data.test-data" name="missing_feature.ini" time="{{[0-1]\.[0-9]+}}">
+# CHECK-NEXT:   <skipped message="Missing required feature(s): dummy_feature"/>
+# CHECK-NEXT: </testcase>
+# CHECK-NEXT: <testcase classname="test-data.test-data" name="pass.ini" time="{{[0-1]\.[0-9]+}}"/>
+# CHECK-NEXT: <testcase classname="test-data.test-data" name="unsupported.ini" time="{{[0-1]\.[0-9]+}}">
+# CHECK-NEXT:   <skipped message="Unsupported configuration"/>
+# CHECK-NEXT: </testcase>
+# CHECK-NEXT: </testsuite>
+# CHECK-NEXT: </testsuites>