Skip lldb-mi tests on FreeBSD for now

[emaste]

Summary

Thread race conditions in lldb-mi cause regular failures. Skip rather than xfail as otherwise each one has to wait for timeout.

[ki.stfu]

I think it will be easier to control if you skip each lldb-mi test using @ skipIfFreeBSD annotation.

[emaste]

In D7727#125574, @ki.stfu wrote:

I think it will be easier to control if you skip each lldb-mi test using @ skipIfFreeBSD annotation.

I tried adding @skipIfFreeBSD on each one as it failed, but the failures are inconsistent and different ones come up on subsequent runs. It seems like it will just be a lot of churn for no real benefit to go that way.

This approach makes it easy to run the lldb-mi tests with +m when we think it's worthwhile without having to back out all of the individual annotations each time.

[ki.stfu]

In D7727#125597, @emaste wrote:

In D7727#125574, @ki.stfu wrote:

I think it will be easier to control if you skip each lldb-mi test using @ skipIfFreeBSD annotation.

I tried adding @skipIfFreeBSD on each one as it failed, but the failures are inconsistent and different ones come up on subsequent runs. It seems like it will just be a lot of churn for no real benefit to go that way.

This approach makes it easy to run the lldb-mi tests with +m when we think it's worthwhile without having to back out all of the individual annotations each time.

If it works unstable on FreeBSD you should skip it and later we can re-enable test on FreeBSD when it will be fixed. I don't see a reason to modify dotest.py file.

FYI: @zturner used the same approach in r226614 when he was solving pexpect issue.

[emaste]

In D7727#125628, @ki.stfu wrote:

If it works unstable on FreeBSD you should skip it and later we can re-enable test on FreeBSD when it will be fixed. I don't see a reason to modify dotest.py file.

I think it's better to make this change here than add 43 decorators across 12 lldb-mi test files

[ki.stfu]

In D7727#125642, @emaste wrote:

In D7727#125628, @ki.stfu wrote:

If it works unstable on FreeBSD you should skip it and later we can re-enable test on FreeBSD when it will be fixed. I don't see a reason to modify dotest.py file.

I think it's better to make this change here than add 43 decorators across 12 lldb-mi test files

It's 'sad' that lldb-mi has grown to such an extent, but the designed @ skipIfFreeBSD is better than a hack in dotest.py.

[zturner]

Fwiw I don't feel strongly either way. Right now it seems there's issues
with lldb-mi such that even disabling them individually now, a new test
that gets added might have the same issue, because the issue is not in the
implementation of a particular test or feature, but of the larger lldb-mi
library. So this would still lead to the possibility that any future test
may also have to be disabled, creating more work than necessary.

Of course, the correct way to deal with this is to prioritize finding and
fixing the underlying race conditions. Presumably they affect not just
FreeBSD.

[emaste]

In D7727#125670, @zturner wrote:

Of course, the correct way to deal with this is to prioritize finding and
fixing the underlying race conditions. Presumably they affect not just
FreeBSD.

Indeed. On FreeBSD we often encounter threading issues earlier or more often than other platforms, for two reasons.

First, our thread library places stricter demands on POSIX compliance of consumers. For example, pthread_rwlock_wrlock and pthread_rwlock_unlock must be called from the same thread, but this was originally violated by LLDB. In practice it seemed to work on OS X and Linux, but did not on FreeBSD (which returns EINVAL from the unlock).

Second, we seem to have wider race windows in certain cases due to differences in scheduling or preemption. Issues that show up under these conditions will also happen on other platforms, perhaps just less frequently.

[emaste]

Added individual decorators instead and work is in progress to fix the threading issues.

[vharron]

Lldb-mi team,

How stable are your tests on OSX? If you run them 100 times, what are the
results?

Vince

test/dotest.py

def parseOptionsAndInitTestdirs():
 
     if do_help == True:
         usage(parser)
 
     # Do not specify both '-a' and '+a' at the same time.
     if dont_do_python_api_test and just_do_python_api_test:
         usage(parser)
 
+    # Skip lldb-mi tests by default on FreeBSD as lldb-mi has thread race
+    # conditions that regularly cause large numbers of failures. Skip rather
+    # than XFAIL the tests otherwise each test has to wait for the timeout.
+    if sys.platform.startswith('freebsd') and not just_do_lldbmi_test:
+        dont_do_lldbmi_test = True
+
     # Do not specify both '-m' and '+m' at the same time.
     if dont_do_lldbmi_test and just_do_lldbmi_test:
         usage(parser)
 
     if args.lldb_platform_name:
         lldb_platform_name = args.lldb_platform_name
     if args.lldb_platform_url:
         lldb_platform_url = args.lldb_platform_url