This is an archive of the discontinued LLVM Phabricator instance.

Print unexpected successes at the end of dosep.
ClosedPublic

Authored by zturner on Aug 13 2015, 3:45 PM.

Details

Reviewers
chaoren
Summary

New output looks like this:

FAIL: LLDB (suite) :: TestThreadStepOut.py (Windows zturner-win81 8 6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel)
FAIL: LLDB (suite) :: TestValueAPI.py (Windows zturner-win81 8 6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel)
FAIL: LLDB (suite) :: TestValueMD5Crash.py (Windows zturner-win81 8 6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel)
FAIL: LLDB (suite) :: TestVectorTypesFormatting.py (Windows zturner-win81 8 6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel)
Unexpected Successes (1)
UNEXPECTED SUCCESS: LLDB (suite) :: TestSettings.py (Windows zturner-win81 8 6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel)
[50856 refs]
ninja: build stopped: subcommand failed.

It is not smart enough to dig into the individual file (TestSettings.py in this example) and find out which test passed that was not expected. That can be done as future work if anyone is interested. The important thing though is that unexpected successes currently have zero visibility, so it is hard to tell when a change fixes existing broken stuff.

The hope is that surfacing this will lead to more tests being re-enabled.

Diff Detail

Event Timeline

zturner updated this revision to Diff 32101.Aug 13 2015, 3:45 PM
zturner retitled this revision from to Print unexpected successes at the end of dosep..
zturner updated this object.
zturner added a reviewer: chaoren.
zturner added a subscriber: lldb-commits.
chaoren added inline comments.Aug 13 2015, 4:17 PM
test/dosep.py
202

Don't need status here.

Nit: could you please wrap the if like the above?
Nit: xpasses reads like "expected passes" to me.

266

There's a line above this:

# result = (timed_out, failed, passed, fail_count, pass_count)

that needs to be updated.

(could you please include all context in the future? I recommend using arcanist)

500

I wonder if it'll be worthwhile to include the count (i.e., x unexpected successes out of y expected failures)

zturner added inline comments.Aug 13 2015, 4:31 PM
test/dosep.py
266

Arcanist didn't work well on windows last time I tried.

zturner updated this revision to Diff 32110.Aug 13 2015, 4:35 PM
zturner added inline comments.
test/dosep.py
500

I thought about that. But if we're going to do that, might as well go all the way and print each one inside the test. Seems like a worthwhile thing to do, just didn't want to take that on quite yet.

chaoren accepted this revision.Aug 13 2015, 4:53 PM
chaoren edited edge metadata.

LGTM

This revision is now accepted and ready to land.Aug 13 2015, 4:53 PM
chaoren closed this revision.Aug 20 2015, 12:16 PM