[vscode] Skip some of the vscode tests on Linux and fix one

[stella.stamenova]

Summary

These are already skipped on Darwin because they cause build bot failures. Both on the build bots as well as in our testing we have seen a number of these tests fail and hang. This change skips the failing/hanging tests on Linux and also fixes one of the test - the test needs the thread library to build.

[teemperor]

packages/Python/lldbsuite/test/tools/lldb-vscode/attach/TestVSCode_attach.py

50

"is well as" -> "as well as" (same for the other comments).

[stella.stamenova]

packages/Python/lldbsuite/test/tools/lldb-vscode/attach/TestVSCode_attach.py

50

Apparently I can't spell today. Thanks for catching this!

[aprantl]

Looks like we *really* need to figure out what the underlying bug is here, because there's not many platforms left after this patch.
This patch is fine as a stopgap, but we can't have a feature like this be in the source code while being virtually untested, that's just asking for trouble.

Greg, were you able to reproduce this?
Stella, do you have any logging available that could help Greg in tracking down the root cause?

[stella.stamenova]

In D51227#1212838, @aprantl wrote:

Looks like we *really* need to figure out what the underlying bug is here, because there's not many platforms left after this patch.
This patch is fine as a stopgap, but we can't have a feature like this be in the source code while being virtually untested, that's just asking for trouble.

Greg, were you able to reproduce this?
Stella, do you have any logging available that could help Greg in tracking down the root cause?

It's fairly easy for me to reproduce, so if Greg has time to investigate, I can collect the logs he needs.

[stella.stamenova]

I spent some more time on this to isolate the failing tests (and reduce the number of disabled tests). I was also able to figure out what one of the failures is - the test needs the thread library to build. The update change reflects that.

packages/Python/lldbsuite/test/tools/lldb-vscode/attach/TestVSCode_attach.py

def set_and_hit_breakpoint(self, continueToExit=True):
                          "expect correct number of breakpoints")
         self.continue_to_breakpoints(breakpoint_ids)
         if continueToExit:
             self.continue_to_exit()
 
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_by_pid(self):
         '''
             Tests attaching to a process by process ID.
         '''
         self.build_and_create_debug_adaptor()
         program = self.getBuildArtifact("a.out")
         self.process = subprocess.Popen([program],
                                         stdin=subprocess.PIPE,
                                         stdout=subprocess.PIPE,
                                         stderr=subprocess.PIPE)
         self.attach(pid=self.process.pid)
         self.set_and_hit_breakpoint(continueToExit=True)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_by_name(self):
         '''
             Tests attaching to a process by process name.
         '''
         self.build_and_create_debug_adaptor()
         orig_program = self.getBuildArtifact("a.out")
         # Since we are going to attach by process name, we need a unique
def test_by_name(self):
                                         stderr=subprocess.PIPE)
         # Wait for a bit to ensure the process is launched
         time.sleep(5)
         self.attach(program=program)
         self.set_and_hit_breakpoint(continueToExit=True)
 
     @skipUnlessDarwin
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_by_name_waitFor(self):
         '''
             Tests attaching to a process by process name and waiting for the
             next instance of a process to be launched, ingoring all current
             ones.
         '''
         self.build_and_create_debug_adaptor()
         program = self.getBuildArtifact("a.out")
         self.spawn_thread = threading.Thread(target=spawn_and_wait,
                                              args=(program, 1.0,))
         self.spawn_thread.start()
         self.attach(program=program, waitFor=True)
         self.set_and_hit_breakpoint(continueToExit=True)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_commands(self):
         '''
             Tests the "initCommands", "preRunCommands", "stopCommands",
             "exitCommands", and "attachCommands" that can be passed during
             attach.
 
             "initCommands" are a list of LLDB commands that get executed

packages/Python/lldbsuite/test/tools/lldb-vscode/breakpoint/TestVSCode_setBreakpoints.py

 
 
 class TestVSCode_setBreakpoints(lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_set_and_clear(self):
         '''Tests setting and clearing source file and line breakpoints.
            This packet is a bit tricky on the debug adaptor side since there
            is no "clearBreakpoints" packet. Source file and line breakpoints
            are set by sending a "setBreakpoints" packet with a source file
            specified and zero or more source lines. If breakpoints have been
            set in the source file before, any exising breakpoints must remain
def test_set_and_clear(self):
             for breakpoint in breakpoints:
                 line = breakpoint['line']
                 self.assertTrue(line in lines, "line expected in lines array")
                 self.assertTrue(breakpoint['verified'],
                                 "expect breakpoint still verified")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_functionality(self):
         '''Tests hitting breakpoints and the functionality of a single
            breakpoint, like 'conditions' and 'hitCondition' settings.'''
         source_basename = 'main.cpp'
         source_path = os.path.join(os.getcwd(), source_basename)
         loop_line = line_number('main.cpp', '// break loop')
 

packages/Python/lldbsuite/test/tools/lldb-vscode/breakpoint/TestVSCode_setExceptionBreakpoints.py

 
 class TestVSCode_setExceptionBreakpoints(
         lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_functionality(self):
         '''Tests setting and clearing exception breakpoints.
            This packet is a bit tricky on the debug adaptor side since there
            is no "clear exception breakpoints" packet. Exception breakpoints
            are set by sending a "setExceptionBreakpoints" packet with zero or
            more exception filters. If exception breakpoints have been set
            before, any exising breakpoints must remain set, and any new

packages/Python/lldbsuite/test/tools/lldb-vscode/breakpoint/TestVSCode_setFunctionBreakpoints.py

 
 class TestVSCode_setFunctionBreakpoints(
         lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_set_and_clear(self):
         '''Tests setting and clearing function breakpoints.
            This packet is a bit tricky on the debug adaptor side since there
            is no "clearFunction Breakpoints" packet. Function breakpoints
            are set by sending a "setFunctionBreakpoints" packet with zero or
            more function names. If function breakpoints have been set before,
            any exising breakpoints must remain set, and any new breakpoints
def test_set_and_clear(self):
         response = self.vscode.request_testGetTargetBreakpoints()
         if response:
             breakpoints = response['body']['breakpoints']
             self.assertTrue(len(breakpoints) == len(functions),
                             "expect %u source breakpoints" % (len(functions)))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_functionality(self):
         '''Tests hitting breakpoints and the functionality of a single
            breakpoint, like 'conditions' and 'hitCondition' settings.'''
 
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)
         # Set a breakpoint on "twelve" with no condition and no hitCondition

packages/Python/lldbsuite/test/tools/lldb-vscode/launch/TestVSCode_launch.py

 
 
 class TestVSCode_launch(lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_default(self):
         '''
             Tests the default launch of a simple program. No arguments,
             environment, or anything else is specified.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)
         self.continue_to_exit()
         # Now get the STDOUT and verify our program argument is correct
         output = self.get_stdout()
         self.assertTrue(output and len(output) > 0,
                         "expect program output")
         lines = output.splitlines()
         self.assertTrue(program in lines[0],
                         "make sure program path is in first argument")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_stopOnEntry(self):
         '''
             Tests the default launch of a simple program that stops at the
             entry point instead of continuing.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program, stopOnEntry=True)
         self.set_function_breakpoints(['main'])
         stopped_events = self.continue_to_next_stop()
         for stopped_event in stopped_events:
             if 'body' in stopped_event:
                 body = stopped_event['body']
                 if 'reason' in body:
                     reason = body['reason']
                     self.assertTrue(
                         reason != 'breakpoint',
                         'verify stop isn\'t "main" breakpoint')
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_cwd(self):
         '''
             Tests the default launch of a simple program with a current working
             directory.
         '''
         program = self.getBuildArtifact("a.out")
         program_parent_dir = os.path.split(os.path.split(program)[0])[0]
def test_cwd(self):
                 found = True
                 self.assertTrue(quote_path in line,
                                 "working directory '%s' not in '%s'" % (
                                     program_parent_dir, line))
         self.assertTrue(found, "verified program working directory")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_debuggerRoot(self):
         '''
             Tests the "debuggerRoot" will change the working directory of
             the lldb-vscode debug adaptor.
         '''
         program = self.getBuildArtifact("a.out")
         program_parent_dir = os.path.split(os.path.split(program)[0])[0]
def test_debuggerRoot(self):
                 self.assertTrue(program_parent_dir == line[len(prefix):],
                                 "lldb-vscode working dir '%s' == '%s'" % (
                                     program_parent_dir, line[6:]))
         self.assertTrue(found, "verified lldb-vscode working directory")
         self.continue_to_exit()
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_sourcePath(self):
         '''
             Tests the "sourcePath" will set the target.source-map.
         '''
         program = self.getBuildArtifact("a.out")
         program_dir = os.path.split(program)[0]
         self.build_and_launch(program,
def test_sourcePath(self):
                 self.assertTrue(quoted_path == line[len(prefix):],
                                 "lldb-vscode working dir %s == %s" % (
                                     quoted_path, line[6:]))
         self.assertTrue(found, 'found "sourcePath" in console output')
         self.continue_to_exit()
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_disableSTDIO(self):
         '''
             Tests the default launch of a simple program with STDIO disabled.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program,
                               disableSTDIO=True)
         self.continue_to_exit()
         # Now get the STDOUT and verify our program argument is correct
         output = self.get_stdout()
         self.assertTrue(output is None or len(output) == 0,
                         "expect no program output")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_shellExpandArguments_enabled(self):
         '''
             Tests the default launch of a simple program with shell expansion
             enabled.
         '''
         program = self.getBuildArtifact("a.out")
         program_dir = os.path.split(program)[0]
def test_shellExpandArguments_enabled(self):
             quote_path = '"%s"' % (program)
             if line.startswith("arg[1] ="):
                 self.assertTrue(quote_path in line,
                                 'verify "%s" expanded to "%s"' % (
                                     glob, program))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_shellExpandArguments_disabled(self):
         '''
             Tests the default launch of a simple program with shell expansion
             disabled.
         '''
         program = self.getBuildArtifact("a.out")
         program_dir = os.path.split(program)[0]
def test_shellExpandArguments_disabled(self):
             quote_path = '"%s"' % (glob)
             if line.startswith("arg[1] ="):
                 self.assertTrue(quote_path in line,
                                 'verify "%s" stayed to "%s"' % (
                                     glob, glob))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_args(self):
         '''
             Tests launch of a simple program with arguments
         '''
         program = self.getBuildArtifact("a.out")
         args = ["one", "with space", "'with single quotes'",
                 '"with double quotes"']
def test_args(self):
         # Make sure arguments we specified are correct
         for (i, arg) in enumerate(args):
             quoted_arg = '"%s"' % (arg)
             self.assertTrue(quoted_arg in lines[i],
                             'arg[%i] "%s" not in "%s"' % (i+1, quoted_arg, lines[i]))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_environment(self):
         '''
             Tests launch of a simple program with environment variables
         '''
         program = self.getBuildArtifact("a.out")
         env = ["NO_VALUE", "WITH_VALUE=BAR", "EMPTY_VALUE=",
                "SPACE=Hello World"]
def test_environment(self):
                     found = True
                     break
             self.assertTrue(found,
                             '"%s" must exist in program environment (%s)' % (
                                 var, lines))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_commands(self):
         '''
             Tests the "initCommands", "preRunCommands", "stopCommands" and
             "exitCommands" that can be passed during launch.
 
             "initCommands" are a list of LLDB commands that get executed
             before the targt is created.

packages/Python/lldbsuite/test/tools/lldb-vscode/stackTrace/TestVSCode_stackTrace.py

def verify_stackFrame(self, frame_idx, stackFrame):
                         'frame #%i source "%s" == "%s"' % (
                             frame_idx, frame_source, self.source_path))
         self.assertTrue(frame_line == expected_line,
                         'frame #%i line %i == %i' % (frame_idx, frame_line,
                                                      expected_line))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_stackTrace(self):
         '''
             Tests the 'stackTrace' packet and all its variants.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)
         source = 'main.c'

packages/Python/lldbsuite/test/tools/lldb-vscode/step/TestVSCode_step.py

 
 
 class TestVSCode_step(lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_step(self):
         '''
             Tests the stepping in/out/over in threads.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)
         source = 'main.cpp'

packages/Python/lldbsuite/test/tools/lldb-vscode/variables/TestVSCode_variables.py

def verify_variables(self, verify_dict, variables, varref_dict=None):
         for variable in variables:
             name = variable['name']
             self.assertTrue(name in verify_dict,
                             'variable "%s" in verify dictionary' % (name))
             self.verify_values(verify_dict[name], variable, varref_dict)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
+    @skipIfLinux # This test is timing out and/or failing on Linux is well as Darwin
     @no_debug_info_test
     def test_scopes_variables_setVariable_evaluate(self):
         '''
             Tests the "scopes", "variables", "setVariable", and "evaluate"
             packets.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)