[lldb] Parse the dotest output to determine the most appropriate result code

[JDevlieghere]

Summary

Currently we look for keywords in the dotest.py output to determine the result code. By parsing the dotest output and looking at the number of tests that passed/failed/were skipped etc we can be more structured in determining the appropriate lit result code. We're still mapping multiple tests to one result code, so some loss of information is inevitable, but I think the new approach is much more sane.

[mib]

Makes more sense. LGTM!

[JDevlieghere]

I was in the process of writing a comment saying that we could be even more accurate by parsing the number of tests with a particular result code from the dotest output but that that seemed overkill before I changed my mind mid-sentence and implemented that instead.

[mib]

Even better! LGTM!

[JDevlieghere]

Here are the results:

Before:

  Unsupported: 145
  Passed     : 873
  Failed     :   7

After:

  Unsupported      : 167
  Passed           : 834
  Expectedly Failed:  16
  Unresolved       :   1
  Failed           :   7

Note that this patch does change the behavior where passes and failures are no longer prioritized. That means that if you have a test that has 3 passes and 4 skips, it will be reported as unsupported (while currently it's considered a pass). We could special case PASS and FAIL so that if there's at least on PASS or FAIL the test is marked as such. That would be more in line with the current state of things.

[kastiglione]

That means that if you have a test that has 3 passes and 4 skips, it will be reported as unsupported (while currently it's considered a pass).

My initial reaction to this is that I don't love it. I think N passes and M skips is a pass, even if M>N.

lldb/test/API/lldbtest.py

91

previously out was also being searched, I take it that's not actually needed?

115

you could use operator.itemgetter(0) instead of the lambda.

[kastiglione]

instead of reducing and picking returning a single result, can we return the raw counts and then report the totals of the counts?

[JDevlieghere]

Mark the test as FAIL or PASS if there's at least one test that respectively failed or passed.

[JDevlieghere]

In D127258#3565279, @kastiglione wrote:

instead of reducing and picking returning a single result, can we return the raw counts and then report the totals of the counts?

No, the lit test format does not support that.

[kastiglione]

lldb/test/API/lldbtest.py

122

you could alternatively use max(lit_results, key=...)

lldb/test/API/lldbtest.py

 from __future__ import absolute_import
 import os
-import tempfile
-import subprocess
-import sys
-import platform
+import re
+import operator
 
 import lit.Test
 import lit.TestRunner
 import lit.util
 from lit.formats.base import TestFormat
 
 
 class LLDBTest(TestFormat):
def execute(self, test, litConfig):
         if out:
             output += """Command Output (stdout):\n--\n%s\n--\n""" % (out,)
         if err:
             output += """Command Output (stderr):\n--\n%s\n--\n""" % (err,)
 
         if timeoutInfo:
             return lit.Test.TIMEOUT, output
 
-        if exitCode:
-            if 'XPASS:' in out or 'XPASS:' in err:
-                return lit.Test.XPASS, output
-
-            # Otherwise this is just a failure.
-            return lit.Test.FAIL, output
-
-        has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
-        has_passing_tests = 'PASS:' in out or 'PASS:' in err
-        if has_unsupported_tests and not has_passing_tests:
-            return lit.Test.UNSUPPORTED, output
+        # Parse the dotest output from stderr.
+        result_regex = r"\((\d+) passes, (\d+) failures, (\d+) errors, (\d+) skipped, (\d+) expected failures, (\d+) unexpected successes\)"
+        results = re.search(result_regex, err)
 
-        passing_test_line = 'RESULT: PASSED'
-        if passing_test_line not in out and passing_test_line not in err:
+        # If parsing fails mark this test as unresolved.
+        if not results:
             return lit.Test.UNRESOLVED, output
 
+        passes = int(results.group(1))
+        failures = int(results.group(2))
+        errors = int(results.group(3))
+        skipped = int(results.group(4))
+        expected_failures = int(results.group(5))
+        unexpected_successes = int(results.group(6))
+
+        if exitCode:
+            # Mark this test as FAIL if at least one test failed.
+            if failures > 0:
+                return lit.Test.FAIL, output
+            lit_results = [(failures, lit.Test.FAIL),
+                           (errors, lit.Test.UNRESOLVED),
+                           (unexpected_successes, lit.Test.XPASS)]
+        else:
+            # Mark this test as PASS if at least one test passed.
+            if passes > 0:
                 return lit.Test.PASS, output
+            lit_results = [(passes, lit.Test.PASS),
+                           (skipped, lit.Test.UNSUPPORTED),
+                           (expected_failures, lit.Test.XFAIL)]
+
+        # Return the lit result code with the maximum occurrence. Only look at
+        # the first element and rely on the original order to break ties.
+        return max(lit_results, key=operator.itemgetter(0))[1], output