[zorg] Fix LitTestCommand unexpected test result when TIMEOUT is returned.

[rfoos]

Summary

Support for LIT tests returning TIMEOUT was added a while back.

If a LIT test returns TIMEOUT, LitTestCommand reports an Unexpected test result error.

lldb.tests test lldb 28 expected failures Unexpected test result output TIMEOUT 28 expected passes
    stdio
    TIMEOUT: LLDB::15-breakpoint-by-symbol-disable-no-hit

With this fix, a count of timeouts and timeout reason are displayed in the title.

lldb.tests test lldb 6 unexpected failures 28 expected failures 2 timeout waiting for results 21 expected passes

I'm open to changing the timeout description "waiting for results". I can't think of anything better atm.

My use case is a Pexpect exception on "read non-blocking error" on expect_exact waiting for an LLDB prompt.

[jroelofs]

+ddunbar

[jroelofs]

Grr. Phab totally didn't do the right thing there.... Here's the body of the reply that it ate:

@rfoos: Do you have a reference to the commit where the TIMEOUT test result was added? @ddunbar asked me not to add that in D6584: "I would prefer that it not introduce a new failure code though, and just
use FAIL with some extra message output".

In other news, this is a good reminder that I need to finish http://reviews.llvm.org/D6584

Cheers,

Jon

[rfoos]

svn r230575 via git...

TIMEOUT currently works with verbose log output. This patch just
finishes the job.

We use it for exceptions from Pexpect that should not occur.

Not quite a FAIL, not quite a seg fault.

UNRESOLVED might be the DejaGnu equivalent failure, but that is cryptic
as well.

Rick

git blame:

6b2822b2 (Daniel Dunbar 2013-02-01 01:39:53 +0000 20) # These are
the codes for which we will inc
6b2822b2 (Daniel Dunbar 2013-02-01 01:39:53 +0000 21) # step results.
3dcd6abe (Ying Chen 2015-02-25 23:41:23 +0000 22) failingCodes =
set(['FAIL', 'XPASS', 'KPASS...TIMEOUT
3dcd6abe (Ying Chen 2015-02-25 23:41:23 +0000 23) # Regular
expressions for start of summary
3dcd6abe (Ying Chen 2015-02-25 23:41:23 +0000 24)
kStartSummaryRE = re.compile(r'^Failing Tes
6b2822b2 (Daniel Dunbar 2013-02-01 01:39:53 +0000 25)
3dcd6abe (Ying Chen 2015-02-25 23:41:23 +0000 26) def
init(self, maxLogs=None, parseSumma

git log:

commit 3dcd6abe8d85e186a2b74ecf544e4ec387efc7a8
Author: Ying Chen <[e-mail]>
Date: Wed Feb 25 23:41:23 2015 +0000

Modify LitLogObserver class to accept optional argument to parse

summary lines only

Summary:
Modify LitLogObserver class to accept optional argument to parse

summary lines only

• This patch is to address the problem that regular stdio output

messages be parsed as result steps

• Add flag to start parser only after summary marker has been matched
  
  Test Plan: Create bulidbot master on local workstation Create slave and connect to local master Monitor the build status with check-ins
  
  Reviewers: sivachandra, ovyalov, chaoren
  
  Subscribers: llvm-commits
  
  Differential Revision: http://reviews.llvm.org/D7877
  
  git-svn-id: https://llvm.org/svn/llvm-project/zorg/trunk@230575

91177308-0d34-0410-b5e6-96231b3b80d8

[jroelofs]

Ah, I hadn't caught that this was a zorg-specific thing... never mind.

Thanks Rick!

Jon

[rfoos]

So this should be clear to merge now.

[jroelofs]

yeah. LGTM.

zorg/buildbot/commands/LitTestCommand.py

resultNames = {'FAIL':'unexpected failures',
                  'XFAIL':'expected failures',
                  'XPASS':'unexpected passes',
                  'KFAIL':'known failures',
                  'KPASS':'unknown passes',
                  'UNRESOLVED':'unresolved testcases',
                  'UNTESTED':'untested testcases',
                  'REGRESSED':'runtime performance regression',
                  'IMPROVED':'runtime performance improvement',
-                 'UNSUPPORTED':'unsupported tests'}
+                 'UNSUPPORTED':'unsupported tests',
+                 'TIMEOUT':'timeout waiting for results'}
 
   def __init__(self, ignore=[], flaky=[], max_logs=20, parseSummaryOnly=False,
                *args, **kwargs):
     Test.__init__(self, *args, **kwargs)
     self.maxLogs = int(max_logs)
     self.logObserver = LitLogObserver(self.maxLogs, parseSummaryOnly)
     self.addFactoryArguments(max_logs=max_logs)
     self.addFactoryArguments(parseSummaryOnly=parseSummaryOnly)
def parse_log(self, text):
     observer = LitLogObserver()
     observer.step = StepProxy()
     for ln in text.split('\n'):
       observer.outLineReceived(ln)
     return observer
 
   def test_basic(self):
     obs = self.parse_log("""
-PASS: test-one (1 of 3)
-FAIL: test-two (2 of 3)
-PASS: test-three (3 of 3)
+PASS: test-one (1 of 4)
+FAIL: test-two (2 of 4)
+PASS: test-three (3 of 4)
+TIMEOUT: test-four (4 of 4)
 """)
 
-    self.assertEqual(obs.resultCounts, { 'FAIL' : 1, 'PASS' : 2 })
-    self.assertEqual(obs.step.logs, [('FAIL: test-two', 'FAIL: test-two')])
+    self.assertEqual(obs.resultCounts, { 'FAIL' : 1, 'TIMEOUT' : 1, 'PASS' : 2 })
+    self.assertEqual(obs.step.logs, [('FAIL: test-two', 'FAIL: test-two'), ('TIMEOUT: test-four', 'TIMEOUT: test-four')])
 
   def test_verbose_logs(self):
     obs = self.parse_log("""
 FAIL: test-one (1 of 3)
 FAIL: test-two (2 of 3)
 **** TEST 'test-two' FAILED ****
 bla bla bla
 **********
class TestCommand(unittest.TestCase):
 
   def test_command_status(self):
     # If the command failed, the status should always be error.
     cmd = self.parse_log("")
     self.assertEqual(cmd.evaluateCommand(RemoteCommandProxy(1)), FAILURE)
 
     # If there were failing tests, the status should be an error (even if the
     # test command didn't report as such).
-    for failing_code in ('FAIL', 'XPASS', 'KPASS', 'UNRESOLVED'):
+    for failing_code in set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED', 'TIMEOUT']):
       cmd = self.parse_log("""%s: test-one (1 of 1)""" % (failing_code,))
       self.assertEqual(cmd.evaluateCommand(RemoteCommandProxy(0)), FAILURE)
 
   def test_max_logs(self):
     cmd = self.parse_log("""
 FAIL: test-one (1 of 2)
 FAIL: test-two (2 of 2)
 """, max_logs=1)
     self.assertEqual(cmd.logObserver.step.logs, [('FAIL: test-one', 'FAIL: test-one')])
 
 if __name__ == '__main__':
   unittest.main()