Re-enable most lldb-vscode tests on Linux.

[jgorbe]

Summary

After https://reviews.llvm.org/D59828 and https://reviews.llvm.org/D59849,
I believe the problems with these tests hanging have been solved.

I tried enabling all of them on my machine, and got two failures:

• One of them was spawning a child process that lives for 5 seconds, waited for 5 seconds to attach to the child, and failed because the child wasn't there.

• The other one was a legit failure because shell expansion of arguments doesn't work on Linux.

This tests enables all lldb-vscode tests on Linux except for "launch process
with shell expansion of args" (which doesn't work), and fixes the other broken
test by reducing the time it waits before attaching to its child process.

[labath]

I think we can give this a shot. We can always re-disable tests that are still flaky and iterate. Thank you for working on this.

lldb/packages/Python/lldbsuite/test/tools/lldb-vscode/attach/TestVSCode_attach.py

95–97 ↗

(On Diff #193365)

I am not sure this will be enough. I've seen tests fail sporadically (like 1 time out of a 1000 or so) even with bigger margins than this. I'd increase the wait in the test to at least 10 seconds. If you don't want to wait that long in the common case when things finish quickly, you can implement a debugger_flag pattern like some of the other tests do: https://github.com/llvm-mirror/lldb/blob/master/packages/Python/lldbsuite/test/functionalities/attach_resume/main.cpp.

[clayborg]

Thanks for getting this stuff reliably working. I debug using VS Code every day using lldb-vscode and it is my favorite LLDB based debugger! I look forward to seeing support for Windows and linux being tested and available.

[jgorbe]

Increased wait time in test binary for TestVSCode_attach.py from 5 to 10 seconds.

If this is still unreliable, or it makes the test take too long, I'll switch it
to use the debugger_flag pattern as suggested.

[labath]

You might be interested to know that I've just seen TestVSCode_step flake (so far just once out of ~dozen runs) locally. I've committed r357747 to get a better error message if it happens again.

[jgorbe]

Thanks! Please let me know if it happens again and I'll try my best to debug it.

[labath]

In D60153#1469397, @jgorbe wrote:

Thanks! Please let me know if it happens again and I'll try my best to debug it.

It is still happening (e.g. http://lab.llvm.org:8014/builders/lldb-x86_64-debian/builds/141), but I already have a fix for that: <D60608>.

[stella.stamenova]

A couple of the tests from TestVSCode_attach.py (test_by_pid and test_by_name) are failing for us on Ubuntu because they are failing to attach: AssertionError: False is not True : attach failed (Operation not permitted). It looks like attaching by pid or by name requires elevation - if I rerun the same tests with sudo, they pass reliably. How did you run the tests when they passed for you?

Incidentally, AFAIK there is no Ubuntu lldb bot...

[labath]

In D60153#1495725, @stella.stamenova wrote:

A couple of the tests from TestVSCode_attach.py (test_by_pid and test_by_name) are failing for us on Ubuntu because they are failing to attach: AssertionError: False is not True : attach failed (Operation not permitted). It looks like attaching by pid or by name requires elevation - if I rerun the same tests with sudo, they pass reliably. How did you run the tests when they passed for you?

Incidentally, AFAIK there is no Ubuntu lldb bot...

You probably have YAMA enabled on those machines. In the rest of our attach tests, we have the inferiors disable YAMA so that the debugger can attach, but it looks like the vscode tests are not making use of that feature. The fix should be as simple as adding lldb_enable_attach() at the start of the inferior main function.

[stella.stamenova]

Thanks, Pavel! I tested out the proposed change and all of our Ubuntu bots now work. I've committed it as well.

[labath]

BTW, it looks like the attach test is still flaky http://lab.llvm.org:8014/builders/lldb-x86_64-debian/builds/1541/steps/test/logs/stdio, though it looks like the failures are super-rare (this is the first time I've seen it do that). Judging by the error message alone, it's not clear to me whether this is due to the sleep being too small, or if there's something else going on.

lldb/trunk/packages/Python/lldbsuite/test/tools/lldb-vscode/attach/TestVSCode_attach.py

def set_and_hit_breakpoint(self, continueToExit=True):
                          "expect correct number of breakpoints")
         self.continue_to_breakpoints(breakpoint_ids)
         if continueToExit:
             self.continue_to_exit()
 
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @skipIfNetBSD # Hangs on NetBSD as well
     @no_debug_info_test
     def test_by_pid(self):
         '''
             Tests attaching to a process by process ID.
         '''
         self.build_and_create_debug_adaptor()
         program = self.getBuildArtifact("a.out")
         self.process = subprocess.Popen([program],
                                         stdin=subprocess.PIPE,
                                         stdout=subprocess.PIPE,
                                         stderr=subprocess.PIPE)
         self.attach(pid=self.process.pid)
         self.set_and_hit_breakpoint(continueToExit=True)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @skipIfNetBSD # Hangs on NetBSD as well
     @no_debug_info_test
     def test_by_name(self):
         '''
             Tests attaching to a process by process name.
         '''
         self.build_and_create_debug_adaptor()
         orig_program = self.getBuildArtifact("a.out")
def test_by_name(self):
                 os.unlink(program)
         # Execute the cleanup function during test case tear down.
         self.addTearDownHook(cleanup)
 
         self.process = subprocess.Popen([program],
                                         stdin=subprocess.PIPE,
                                         stdout=subprocess.PIPE,
                                         stderr=subprocess.PIPE)
-        # Wait for a bit to ensure the process is launched
-        time.sleep(5)
+        # Wait for a bit to ensure the process is launched, but not for so long
+        # that the process has already finished by the time we attach.
+        time.sleep(3)
         self.attach(program=program)
         self.set_and_hit_breakpoint(continueToExit=True)
 
     @skipUnlessDarwin
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @skipIfNetBSD # Hangs on NetBSD as well
     @no_debug_info_test
     def test_by_name_waitFor(self):
         '''
             Tests attaching to a process by process name and waiting for the
             next instance of a process to be launched, ingoring all current
             ones.
         '''
         self.build_and_create_debug_adaptor()
         program = self.getBuildArtifact("a.out")
         self.spawn_thread = threading.Thread(target=spawn_and_wait,
                                              args=(program, 1.0,))
         self.spawn_thread.start()
         self.attach(program=program, waitFor=True)
         self.set_and_hit_breakpoint(continueToExit=True)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @skipIfNetBSD # Hangs on NetBSD as well
     @no_debug_info_test
     def test_commands(self):
         '''
             Tests the "initCommands", "preRunCommands", "stopCommands",
             "exitCommands", and "attachCommands" that can be passed during
             attach.
 

lldb/trunk/packages/Python/lldbsuite/test/tools/lldb-vscode/attach/main.c

 #include <stdio.h>
 #include <unistd.h>
 
 int main(int argc, char const *argv[]) {
   printf("pid = %i\n", getpid());
-  sleep(5);
+  sleep(10);
   return 0; // breakpoint 1
 }

lldb/trunk/packages/Python/lldbsuite/test/tools/lldb-vscode/breakpoint/TestVSCode_setFunctionBreakpoints.py

 
 class TestVSCode_setFunctionBreakpoints(
         lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # test hangs on linux under heavy load
     @no_debug_info_test
     def test_set_and_clear(self):
         '''Tests setting and clearing function breakpoints.
            This packet is a bit tricky on the debug adaptor side since there
            is no "clearFunction Breakpoints" packet. Function breakpoints
            are set by sending a "setFunctionBreakpoints" packet with zero or
            more function names. If function breakpoints have been set before,
            any exising breakpoints must remain set, and any new breakpoints
def test_set_and_clear(self):
         response = self.vscode.request_testGetTargetBreakpoints()
         if response:
             breakpoints = response['body']['breakpoints']
             self.assertTrue(len(breakpoints) == len(functions),
                             "expect %u source breakpoints" % (len(functions)))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # test hangs on linux under heavy load
     @no_debug_info_test
     def test_functionality(self):
         '''Tests hitting breakpoints and the functionality of a single
            breakpoint, like 'conditions' and 'hitCondition' settings.'''
 
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)
         # Set a breakpoint on "twelve" with no condition and no hitCondition

lldb/trunk/packages/Python/lldbsuite/test/tools/lldb-vscode/launch/TestVSCode_launch.py

 
 
 class TestVSCode_launch(lldbvscode_testcase.VSCodeTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_default(self):
         '''
             Tests the default launch of a simple program. No arguments,
             environment, or anything else is specified.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program)
         self.continue_to_exit()
         # Now get the STDOUT and verify our program argument is correct
         output = self.get_stdout()
         self.assertTrue(output and len(output) > 0,
                         "expect program output")
         lines = output.splitlines()
         self.assertTrue(program in lines[0],
                         "make sure program path is in first argument")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_stopOnEntry(self):
         '''
             Tests the default launch of a simple program that stops at the
             entry point instead of continuing.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program, stopOnEntry=True)
         self.set_function_breakpoints(['main'])
         stopped_events = self.continue_to_next_stop()
         for stopped_event in stopped_events:
             if 'body' in stopped_event:
                 body = stopped_event['body']
                 if 'reason' in body:
                     reason = body['reason']
                     self.assertTrue(
                         reason != 'breakpoint',
                         'verify stop isn\'t "main" breakpoint')
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_cwd(self):
         '''
             Tests the default launch of a simple program with a current working
             directory.
         '''
         program = self.getBuildArtifact("a.out")
         program_parent_dir = os.path.split(os.path.split(program)[0])[0]
def test_cwd(self):
                 found = True
                 self.assertTrue(quote_path in line,
                                 "working directory '%s' not in '%s'" % (
                                     program_parent_dir, line))
         self.assertTrue(found, "verified program working directory")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_debuggerRoot(self):
         '''
             Tests the "debuggerRoot" will change the working directory of
             the lldb-vscode debug adaptor.
         '''
         program = self.getBuildArtifact("a.out")
         program_parent_dir = os.path.split(os.path.split(program)[0])[0]
def test_debuggerRoot(self):
                 self.assertTrue(program_parent_dir == line[len(prefix):],
                                 "lldb-vscode working dir '%s' == '%s'" % (
                                     program_parent_dir, line[6:]))
         self.assertTrue(found, "verified lldb-vscode working directory")
         self.continue_to_exit()
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_sourcePath(self):
         '''
             Tests the "sourcePath" will set the target.source-map.
         '''
         program = self.getBuildArtifact("a.out")
         program_dir = os.path.split(program)[0]
         self.build_and_launch(program,
def test_sourcePath(self):
                 self.assertTrue(quoted_path == line[len(prefix):],
                                 "lldb-vscode working dir %s == %s" % (
                                     quoted_path, line[6:]))
         self.assertTrue(found, 'found "sourcePath" in console output')
         self.continue_to_exit()
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_disableSTDIO(self):
         '''
             Tests the default launch of a simple program with STDIO disabled.
         '''
         program = self.getBuildArtifact("a.out")
         self.build_and_launch(program,
                               disableSTDIO=True)
         self.continue_to_exit()
         # Now get the STDOUT and verify our program argument is correct
         output = self.get_stdout()
         self.assertTrue(output is None or len(output) == 0,
                         "expect no program output")
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
+    @skipIfLinux # shell argument expansion doesn't seem to work on Linux
     @expectedFailureNetBSD
     @no_debug_info_test
     def test_shellExpandArguments_enabled(self):
         '''
             Tests the default launch of a simple program with shell expansion
             enabled.
         '''
         program = self.getBuildArtifact("a.out")
def test_shellExpandArguments_enabled(self):
             quote_path = '"%s"' % (program)
             if line.startswith("arg[1] ="):
                 self.assertTrue(quote_path in line,
                                 'verify "%s" expanded to "%s"' % (
                                     glob, program))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_shellExpandArguments_disabled(self):
         '''
             Tests the default launch of a simple program with shell expansion
             disabled.
         '''
         program = self.getBuildArtifact("a.out")
         program_dir = os.path.split(program)[0]
def test_shellExpandArguments_disabled(self):
             quote_path = '"%s"' % (glob)
             if line.startswith("arg[1] ="):
                 self.assertTrue(quote_path in line,
                                 'verify "%s" stayed to "%s"' % (
                                     glob, glob))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_args(self):
         '''
             Tests launch of a simple program with arguments
         '''
         program = self.getBuildArtifact("a.out")
         args = ["one", "with space", "'with single quotes'",
                 '"with double quotes"']
def test_args(self):
         # Make sure arguments we specified are correct
         for (i, arg) in enumerate(args):
             quoted_arg = '"%s"' % (arg)
             self.assertTrue(quoted_arg in lines[i],
                             'arg[%i] "%s" not in "%s"' % (i+1, quoted_arg, lines[i]))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_environment(self):
         '''
             Tests launch of a simple program with environment variables
         '''
         program = self.getBuildArtifact("a.out")
         env = ["NO_VALUE", "WITH_VALUE=BAR", "EMPTY_VALUE=",
                "SPACE=Hello World"]
def test_environment(self):
                     found = True
                     break
             self.assertTrue(found,
                             '"%s" must exist in program environment (%s)' % (
                                 var, lines))
 
     @skipIfWindows
     @skipIfDarwin # Skip this test for now until we can figure out why tings aren't working on build bots
-    @skipIfLinux # This test is timing out and/or failing on Linux as well as Darwin
     @no_debug_info_test
     def test_commands(self):
         '''
             Tests the "initCommands", "preRunCommands", "stopCommands" and
             "exitCommands" that can be passed during launch.
 
             "initCommands" are a list of LLDB commands that get executed
             before the targt is created.