[lldbsuite, windows] Mark tests as XFAIL on Windows or skip them

[stella.stamenova]

Summary

1. Several tests that are flakey on windows fail the run even if they are marked as expected to be flakey. This is because they fail frequently enough that even a retry won't help
2. Skip several tests on Windows that will occasionally hang rather than failing or exiting. This is causing the entire test suite to hang

[labath]

BTW, since we switched to the lit test runner, the expectedFlakey decorators don't actually do anything.

[stella.stamenova]

In D50198#1186890, @labath wrote:

BTW, since we switched to the lit test runner, the expectedFlakey decorators don't actually do anything.

Oh, haha, that explains why it made no difference. Should we remove the expectedFlakey decorators entirely, so that they don't confuse anyone else (and replace them with a comment) or is there a plan to make them work again in the future?

[labath]

In D50198#1187653, @stella.stamenova wrote:

In D50198#1186890, @labath wrote:

BTW, since we switched to the lit test runner, the expectedFlakey decorators don't actually do anything.

Oh, haha, that explains why it made no difference. Should we remove the expectedFlakey decorators entirely, so that they don't confuse anyone else (and replace them with a comment) or is there a plan to make them work again in the future?

I'm not sure. lit has a concept of a flakey test, and it would be nice to hook this up to that. However, that is going to be tricky as long as we have this disconnect between what lit considers a "test" and what our decorators do (see the "run each test separately" discussion).

The decorators also still serve a purpose when you run the test suite via dotest directly (our existing bots still do that, as the lit method does not cover all use cases). However, those bots may be turned down soon too, so I'm not sure how much we should optimize for that.

packages/Python/lldbsuite/test/functionalities/attach_resume/TestAttachResume.py

 
 
 class AttachResumeTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfRemote
     @expectedFailureAll(oslist=['freebsd'], bugnumber='llvm.org/pr19310')
-    @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr24778")
+    @skipIfWindows # llvm.org/pr24778, llvm.org/pr21753
     def test_attach_continue_interrupt_detach(self):
         """Test attach/continue/interrupt/detach"""
         self.build()
         self.process_attach_continue_interrupt_detach()
 
     def process_attach_continue_interrupt_detach(self):
         """Test attach/continue/interrupt/detach"""
 

packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_ignore_count/TestBreakpointIgnoreCount.py

 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 class BreakpointIgnoreCountTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     def test_with_run_command(self):
         """Exercise breakpoint ignore count with 'breakpoint set -i <count>'."""
         self.build()
         self.breakpoint_ignore_count()
 
     @add_test_categories(['pyapi'])
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     def test_with_python_api(self):
         """Use Python APIs to set breakpoint ignore count."""
         self.build()
         self.breakpoint_ignore_count_python()
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)

packages/Python/lldbsuite/test/functionalities/memory/cache/TestMemoryCache.py

class MemoryCacheTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number to break inside main().
         self.line = line_number('main.cpp', '// Set break point at this line.')
 
-    @expectedFlakeyOS(oslist=["windows"])
+    # The test is actually flakey on Windows, failing every dozen or so runs, but even with the flakey
+    # decorator it still fails
+    @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr38373")
     def test_memory_cache(self):
         """Test the MemoryCache class with a sequence of 'memory read' and 'memory write' operations."""
         self.build()
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # Break in main() after the variables are assigned values.
         lldbutil.run_break_set_by_file_and_line(

packages/Python/lldbsuite/test/functionalities/thread/exit_during_step/TestExitDuringStep.py

 from lldbsuite.test import lldbutil
 
 
 class ExitDuringStepTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # llvm.org/pr21411: test is hanging
+    # The test is actually flakey on Windows, failing every dozen or so runs, but even with the flakey
+    # decorator it still fails
+    @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr38373")
     def test(self):
         """Test thread exit during step handling."""
         self.build(dictionary=self.getBuildFlags())
         self.exit_during_step_base(
             "thread step-inst -m all-threads",
             'stop reason = instruction step',
             True)
 
     @skipIfFreeBSD  # llvm.org/pr21411: test is hanging
+    # The test is actually flakey on Windows, failing every dozen or so runs, but even with the flakey
+    # decorator it still fails
+    @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr38373")
     def test_step_over(self):
         """Test thread exit during step-over handling."""
         self.build(dictionary=self.getBuildFlags())
         self.exit_during_step_base(
             "thread step-over -m all-threads",
             'stop reason = step over',
             False)
 
     @skipIfFreeBSD  # llvm.org/pr21411: test is hanging
+    # The test is actually flakey on Windows, failing every dozen or so runs, but even with the flakey
+    # decorator it still fails
+    @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr38373")
     def test_step_in(self):
         """Test thread exit during step-in handling."""
         self.build(dictionary=self.getBuildFlags())
         self.exit_during_step_base(
             "thread step-in -m all-threads",
             'stop reason = step in',
             False)
 

packages/Python/lldbsuite/test/functionalities/thread/step_out/TestThreadStepOut.py

class ThreadStepOutTestCase(TestBase):
     # Test occasionally times out on the Linux build bot
     @skipIfLinux
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23477 Test occasionally times out on the Linux build bot")
     @expectedFailureAll(
         oslist=["freebsd"],
         bugnumber="llvm.org/pr18066 inferior does not exit")
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     @expectedFailureAll(oslist=["windows"])
     def test_step_single_thread(self):
         """Test thread step out on one thread via command interpreter. """
         self.build(dictionary=self.getBuildFlags())
         self.step_out_test(self.step_out_single_thread_with_cmd)
 
     # Test occasionally times out on the Linux build bot
     @skipIfLinux
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23477 Test occasionally times out on the Linux build bot")
     @expectedFailureAll(
         oslist=["freebsd"],
         bugnumber="llvm.org/pr19347 2nd thread stops at breakpoint")
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     @expectedFailureAll(oslist=["windows"])
     @expectedFailureAll(oslist=["watchos"], archs=['armv7k'], bugnumber="rdar://problem/34674488") # stop reason is trace when it should be step-out
     def test_step_all_threads(self):
         """Test thread step out on all threads via command interpreter. """
         self.build(dictionary=self.getBuildFlags())
         self.step_out_test(self.step_out_all_threads_with_cmd)
 
     # Test occasionally times out on the Linux build bot
     @skipIfLinux
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23477 Test occasionally times out on the Linux build bot")
     @expectedFailureAll(
         oslist=["freebsd"],
         bugnumber="llvm.org/pr19347 2nd thread stops at breakpoint")
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr24681")
     def test_python(self):
         """Test thread step out on one thread via Python API (dwarf)."""
         self.build(dictionary=self.getBuildFlags())
         self.step_out_test(self.step_out_with_python)
 
     def setUp(self):
         # Call super's setUp().

packages/Python/lldbsuite/test/functionalities/thread/thread_exit/TestThreadExit.py

 """
 Test number of threads.
 """
 
 from __future__ import print_function
 
 
 import os
 import time
 import lldb
+from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 import lldbsuite.test.lldbutil as lldbutil
 
 
 class ThreadExitTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line numbers for our breakpoints.
         self.break_1 = line_number('main.cpp', '// Set first breakpoint here')
         self.break_2 = line_number('main.cpp', '// Set second breakpoint here')
         self.break_3 = line_number('main.cpp', '// Set third breakpoint here')
         self.break_4 = line_number('main.cpp', '// Set fourth breakpoint here')
 
+    # The test is actually flakey on Windows, failing every dozen or so runs, but even with the flakey
+    # decorator it still fails
+    @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr38373")
     def test(self):
         """Test thread exit handling."""
         self.build(dictionary=self.getBuildFlags())
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # This should create a breakpoint with 1 location.
         bp1_id = lldbutil.run_break_set_by_file_and_line(

packages/Python/lldbsuite/test/lang/cpp/virtual/TestVirtual.py

def setUp(self):
         TestBase.setUp(self)
         # Find the line number to break for main.cpp.
         self.source = 'main.cpp'
         self.line = line_number(self.source, '// Set first breakpoint here.')
 
     @expectedFailureAll(
         compiler="icc",
         bugnumber="llvm.org/pr16808 lldb does not call the correct virtual function with icc.")
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     def test_virtual_madness(self):
         """Test that expression works correctly with virtual inheritance as well as virtual function."""
         self.build()
 
         # Bring the program to the point where we can issue a series of
         # 'expression' command to compare against the golden output.
         self.dbg.SetAsync(False)
 

packages/Python/lldbsuite/test/python_api/event/TestEvents.py

def setUp(self):
         # Find the line number to of function 'c'.
         self.line = line_number(
             'main.c', '// Find the line number of function "c" here.')
 
     @add_test_categories(['pyapi'])
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23730 Flaky, fails ~1/10 cases")
+    @skipIfWindows # This test will hang on windows llvm.org/pr21753
     def test_listen_for_and_print_event(self):
         """Exercise SBEvent API."""
         self.build()
         exe = self.getBuildArtifact("a.out")
 
         self.dbg.SetAsync(True)
 
         # Create a target by the debugger.