[lit] Include total elapsed time in xunit output

[arichardson]

Summary

The time= attribute can also be used for the testsuites tag. While this
attribute appears to be ignored by Jenkins (https://github.com/jenkinsci/junit-plugin/blob/bab34bcc96154a494f8c371953efe06d45813f67/src/main/java/hudson/tasks/junit/SuiteResult.java#L202),
it is still useful if you manually inspect the xml file

[arichardson]

fix tests

[yln]

LGTM, thanks!

llvm/utils/lit/lit/reports.py

83

Note: I initially didn't include the overall time here because it is measured by simply doing the following in main.py:

start = time.time()
run_tests(...)
elapsed = time.time() - start

So it's not necessarily exactly equal to the sum of all the individual test times.

[arichardson]

llvm/utils/lit/lit/reports.py

83

Yeah, but I find it quite useful that it shows the real elapsed time when running multiple jobs rather than adding up all individual tests.

llvm/utils/lit/lit/reports.py

 
 
 class XunitReport(object):
     def __init__(self, output_file):
         self.output_file = output_file
         self.skipped_codes = {lit.Test.EXCLUDED,
                               lit.Test.SKIPPED, lit.Test.UNSUPPORTED}
 
-    # TODO(yln): elapsed unused, put it somewhere?
     def write_results(self, tests, elapsed):
         tests.sort(key=by_suite_and_test_path)
         tests_by_suite = itertools.groupby(tests, lambda t: t.suite)
 
         with open(self.output_file, 'w') as file:
             file.write('<?xml version="1.0" encoding="UTF-8"?>\n')
-            file.write('<testsuites>\n')
+            file.write('<testsuites time="{time:.2f}">\n'.format(time=elapsed))
             for suite, test_iter in tests_by_suite:
                 self._write_testsuite(file, suite, list(test_iter))
             file.write('</testsuites>\n')
 
     def _write_testsuite(self, file, suite, tests):
         skipped = sum(1 for t in tests if t.result.code in self.skipped_codes)
         failures = sum(1 for t in tests if t.isFailure())
 

llvm/utils/lit/tests/shtest-format.py

 # CHECK: Passed             : 6
 # CHECK: Expectedly Failed  : 4
 # CHECK: Unresolved         : 3
 # CHECK: Failed             : 3
 # CHECK: Unexpectedly Passed: 1
 
 
 # XUNIT: <?xml version="1.0" encoding="UTF-8"?>
-# XUNIT-NEXT: <testsuites>
+# XUNIT-NEXT: <testsuites time="{{[0-9.]+}}">
 # XUNIT-NEXT: <testsuite name="shtest-format" tests="21" failures="7" skipped="4">
 
 # XUNIT: <testcase classname="shtest-format.external_shell" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
 # XUNIT-NEXT: <failure{{[ ]*}}>
 # XUNIT: </failure>
 # XUNIT-NEXT: </testcase>
 
 

llvm/utils/lit/tests/xunit-output.py

 # REQUIRES: shell
 
 # Check xunit output
 # RUN: rm -rf %t.xunit.xml
 # RUN: not %{lit} --xunit-xml-output %t.xunit.xml %{inputs}/xunit-output
 # If xmllint is installed verify that the generated xml is well-formed
 # RUN: sh -c 'if command -v xmllint 2>/dev/null; then xmllint --noout %t.xunit.xml; fi'
 # RUN: FileCheck < %t.xunit.xml %s
 
 # CHECK:      <?xml version="1.0" encoding="UTF-8"?>
-# CHECK-NEXT: <testsuites>
+# CHECK-NEXT: <testsuites time="{{[0-9.]+}}">
 # CHECK-NEXT: <testsuite name="test-data" tests="5" failures="1" skipped="3">
 # CHECK-NEXT: <testcase classname="test-data.test-data" name="bad&amp;name.ini" time="{{[0-1]\.[0-9]+}}">
 # CHECK-NEXT:   <failure><![CDATA[& < > ]]]]><![CDATA[> &"]]></failure>
 # CHECK-NEXT: </testcase>
 # CHECK-NEXT: <testcase classname="test-data.test-data" name="excluded.ini" time="{{[0-1]\.[0-9]+}}">
 # CHECK-NEXT:   <skipped message="Test not selected (--filter, --max-tests, --run-shard)"/>
 # CHECK-NEXT: </testcase>
 # CHECK-NEXT: <testcase classname="test-data.test-data" name="missing_feature.ini" time="{{[0-1]\.[0-9]+}}">
 # CHECK-NEXT:   <skipped message="Missing required feature(s): dummy_feature"/>
 # CHECK-NEXT: </testcase>
 # CHECK-NEXT: <testcase classname="test-data.test-data" name="pass.ini" time="{{[0-1]\.[0-9]+}}"/>
 # CHECK-NEXT: <testcase classname="test-data.test-data" name="unsupported.ini" time="{{[0-1]\.[0-9]+}}">
 # CHECK-NEXT:   <skipped message="Unsupported configuration"/>
 # CHECK-NEXT: </testcase>
 # CHECK-NEXT: </testsuite>
 # CHECK-NEXT: </testsuites>