Basic JUnit XML support for llvm-lit

[theraven]

Summary

Export JUnit XML that Jenkins can understand. We've been using this for about 18 months in our internal Jenkins. It's not a perfect mapping, but hopefully it's adequate for other people to build on.

[russell.gallop]

Hi,

This would be very useful to us. I tried this patch and it basically works with our CI server too.

I would suggest a couple of minor changes to improve the output format, see inline comments. FWIW the change looks good to me.

Regards
Russ

utils/lit/lit/Test.py

200

Omitting the test name from the class name seems like a good idea to me. It allows a junit consumer to group tests by folder. (This is rather a moot point as our tests are files and folders rather than classes and packages).

-        xml = "<testcase classname='" + ".".join(self.path_in_suite)
+        xml = "<testcase classname='" + ".".join(self.path_in_suite[:-1])

200–201

Formatting.

     def getJUnitXML(self):
-        xml = "<testcase classname='" + ".".join(self.path_in_suite[:-1])
-        xml += "' name='" + '/'.join(self.path_in_suite) + "'"
+        xml = "<testcase classname='" + ".".join(self.path_in_suite[:-1]) + "'"
+        xml += " name='" + '/'.join(self.path_in_suite) + "'"
         if self.result.code.isFailure:

201–202

Add elapsed time.

         xml += " name='" + '/'.join(self.path_in_suite) + "'"
+        xml += " time='%.2f'" % (self.result.elapsed,)
         if self.result.code.isFailure:

utils/lit/lit/main.py

446

Some consumers use the errors attribute to determine the number of passes (tests - failures - errors).

            xmlFile.write(" tests='" + str(s['passes'] + s['failures']) + "'")
+            xmlFile.write(" errors='0'")
            xmlFile.write(" failures='" + str(s['failures']) + "'>\n")

[Eugene.Zelenko]

Committed in rL223163.

utils/lit/lit/Test.py

 import os
+from xml.sax.saxutils import escape
 
 # Test result codes.
 

                 return True
 
         return False
+
+    def getJUnitXML(self):
+        xml = "<testcase classname='" + ".".join(self.path_in_suite)
+        xml += "' name='" + '/'.join(self.path_in_suite) + "'"
+        if self.result.code.isFailure:
+          xml += ">\n\t<failure >\n" + escape(self.result.output)
+          xml += "\n\t</failure>\n</testcase>"
+        else:
+          xml += "/>"
+        return xml
+

utils/lit/lit/main.py

     group.add_option("", "--use-threads", dest="useProcesses",
                       help="Run tests in parallel with threads (not processes)",
                       action="store_false", default=useProcessesIsDefault)
+    group.add_option("", "--junit-xml-output", dest="xmlFile",
+                      help=("Write JUnit-compatible XML test reports to the"
+                            " specified file"), default=None)
     parser.add_option_group(group)
 
     (opts, args) = parser.parse_args()

         N = len(byCode.get(code,[]))
         if N:
             print('  %s: %d' % (name,N))
+    if opts.xmlFile:
+        # Collect the tests, indexed by test suite
+        bySuite = {}
+        for t in run.tests:
+            suite = t.suite.config.name
+            if suite not in bySuite:
+                bySuite[suite] = {
+                                   'passes'   : 0,
+                                   'failures' : 0,
+                                   'tests'    : [] }
+            bySuite[suite]['tests'].append(t)
+            if t.result.code.isFailure:
+                bySuite[suite]['failures'] += 1
+            else:
+                bySuite[suite]['passes'] += 1
+        xmlFile = open(opts.xmlFile, "w")
+        xmlFile.write("<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n")
+        xmlFile.write("<testsuites>\n")
+        for suiteName in bySuite:
+            s = bySuite[suiteName]
+            xmlFile.write("<testsuite name='" + suiteName + "'")
+            xmlFile.write(" tests='" + str(s['passes'] + s['failures']) + "'")
+            xmlFile.write(" failures='" + str(s['failures']) + "'>\n")
+            for t in s['tests']:
+                xmlFile.write(t.getJUnitXML() + "\n")
+            xmlFile.write("</testsuite>\n")
+        xmlFile.write("</testsuites>")
+        xmlFile.close()
+
 
     # If we encountered any additional errors, exit abnormally.
     if litConfig.numErrors: