[dotest] Consider unexpected passes as failures.

[JDevlieghere]

Summary

Unexpected successes should be considered failures because they can hide regressions when not addressed. When a test is fixed and not re-enabled, it can easily regress without us noticing.

I couldn't find a good way to make this change other than changing it in the unittest2 framework. I know this is less than optimal but since we have the dependency checked in and the change is pretty fundamental to the framework I think it's not unreasonable.

[labath]

Looks good to me. Modifying this code is not ideal, but this is a pretty small but fundamental change in how unittest works. Changing the treatment of unexpected successes anywhere else would be a hack. It also looks like this is not the first change we've made in this piece of code.

lldb/trunk/lit/Suite/lldbtest.py

def execute(self, test, litConfig):
                 cmd,
                 env=test.config.environment,
                 timeout=litConfig.maxIndividualTestTime)
         except lit.util.ExecuteCommandTimeoutException:
             return (lit.Test.TIMEOUT, 'Reached timeout of {} seconds'.format(
                 litConfig.maxIndividualTestTime))
 
         if exitCode:
+            if 'FAIL:' in out or 'FAIL:' in err:
                 return lit.Test.FAIL, out + err
-
-        unexpected_test_line = 'XPASS'
-        if unexpected_test_line in out or unexpected_test_line in err:
-            return lit.Test.XPASS, ''
+            if 'XPASS:' in out or 'XPASS:' in err:
+                return lit.Test.XPASS, out + err
 
         passing_test_line = 'RESULT: PASSED'
         if passing_test_line not in out and passing_test_line not in err:
             msg = ('Unable to find %r in dotest output:\n\n%s%s' %
                    (passing_test_line, out, err))
             return lit.Test.UNRESOLVED, msg
 
         return lit.Test.PASS, ''

lldb/trunk/third_party/Python/module/unittest2/unittest2/result.py

class TestResult(unittest.TestResult):
 
     @failfast
     def addUnexpectedSuccess(self, test, bugnumber):
         """Called when a test was expected to fail, but succeed."""
         self.unexpectedSuccesses.append(test)
 
     def wasSuccessful(self):
         "Tells whether or not this result was a success"
-        return (len(self.failures) + len(self.errors) == 0)
+        return (len(self.failures) +
+                len(self.errors) +
+                len(self.unexpectedSuccesses) == 0)
 
     def stop(self):
         "Indicates that the tests should be aborted"
         self.shouldStop = True
 
     def _exc_info_to_string(self, err, test):
         """Converts a sys.exc_info()-style tuple of values into a string."""
         exctype, value, tb = err