HomePhabricator

Revert "Fixup XFAIL marking on TestConstVariables for clang version"

Description

Revert "Fixup XFAIL marking on TestConstVariables for clang version"

Summary:
Linux with ToT clang (3.8) fails.

This reverts commit r247633.

Reviewers: tfiala

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D12868

Details

Committed
chaorenSep 14 2015, 5:52 PM
Differential Revision
D12868: Revert "Fixup XFAIL marking on TestConstVariables for clang version"
Parents
rL247647: Fix error on windows:
Branches
Unknown
Tags
Unknown

Event Timeline

tfiala added a subscriber: tfiala.Sep 14 2015, 6:03 PM

Hmm, looks like we need to pinpoint clang version 3.6, because it succeeds there.

I'm curious why we use an unreleased compiler for lldb build bots. That seems like we're adding instability to what should be a stable test environment.

We use both clang 3.5 and ToT clang. It'd be good to know if any bugs get introduced by recent clang changes.

This is a great illustration of why we should consider moving unexpected
successes to the status of a build break (i.e. equivalent to a failure).
It's something we expect to fail, and if it doesn't, that warrants
investigation. We'll also need the true flaky (aka flakey) category so
tests that are too hard to have work 100% of the time but exercise critical
elements are at least run with some ability to reason over the results over
time.

I just did a set of tests on TestConstVariables on Ubuntu 14.04 x86_64.

With clang-3.5, it passes. It was marked XFAIL, and so unexpectedly
succeeded. We missed this because we don't consider that a build break.

With clang-3.6, it passes. It was marked XFAIL (until earlier today, and
now again). We missed this.

(I don't have an easy way to try clang-3.7 on the box I'm using, so not
sure of the 3.7 status).

With TOT clang, it fails. Had the unexpected successes in the clang 3.5+
timeframe caused the actionable item of forcing us to mark it as passing,
then the moment it started failing as it does against TOT clang, we would
have been able to more readily associated the breakage.

So the real timeline for this test is it was passing (and should have been
marked as such) for some time, and has more recently started failing.

I'll figure out how to mark that up in the test system, and extend it if I
cannot. I can reproduce the pass/fail scenarios so should be relatively
easy to verify.

So the real timeline for this test is it was passing (and should have been marked as such) for some time, and has more recently started failing.

And by more recently started failing, I mean more recent versions of clang are needed to expose the failure, regardless of where exactly the root cause lies.

(I don't have an easy way to try clang-3.7 on the box I'm using, so not sure of the 3.7 status).

I have since built a clang 3.7 on that machine. It is busted there as well.

So:
clang-3.5 - pass
clang-3.6 - pass
clang-3.7 - fail
clang-TOT - fail

This change is take 2, with all clang versions properly marked. I tested against clang 3.4, 3.5, 3.6, 3.7 and TOT (3.8) on Ubuntu 14.04 x86_64.

Sending test/lang/c/const_variables/TestConstVariables.py
Transmitting file data .
Committed revision 247664.