Page MenuHomePhabricator

[lldbsuite, windows] Mark tests as XFAIL on Windows or skip them
ClosedPublic

Authored by stella.stamenova on Aug 2 2018, 12:45 PM.

Details

Summary
  1. Several tests that are flakey on windows fail the run even if they are marked as expected to be flakey. This is because they fail frequently enough that even a retry won't help
  2. Skip several tests on Windows that will occasionally hang rather than failing or exiting. This is causing the entire test suite to hang

Diff Detail

Repository
rLLDB LLDB

Event Timeline

zturner accepted this revision.Aug 2 2018, 1:06 PM
This revision is now accepted and ready to land.Aug 2 2018, 1:06 PM
This revision was automatically updated to reflect the committed changes.
labath added a comment.Aug 3 2018, 1:15 AM

BTW, since we switched to the lit test runner, the expectedFlakey decorators don't actually do anything.

BTW, since we switched to the lit test runner, the expectedFlakey decorators don't actually do anything.

Oh, haha, that explains why it made no difference. Should we remove the expectedFlakey decorators entirely, so that they don't confuse anyone else (and replace them with a comment) or is there a plan to make them work again in the future?

labath added a comment.Aug 7 2018, 6:20 AM

BTW, since we switched to the lit test runner, the expectedFlakey decorators don't actually do anything.

Oh, haha, that explains why it made no difference. Should we remove the expectedFlakey decorators entirely, so that they don't confuse anyone else (and replace them with a comment) or is there a plan to make them work again in the future?

I'm not sure. lit has a concept of a flakey test, and it would be nice to hook this up to that. However, that is going to be tricky as long as we have this disconnect between what lit considers a "test" and what our decorators do (see the "run each test separately" discussion).

The decorators also still serve a purpose when you run the test suite via dotest directly (our existing bots still do that, as the lit method does not cover all use cases). However, those bots may be turned down soon too, so I'm not sure how much we should optimize for that.