[lldb] [llgs] Add base nonstop fork/vfork tests

[mgorny]

Summary

Extend the most of baseline fork tests to run in nonstop mode as well.
For more cases, we're just testing one example scenario to save time.
This patch does not cover tests that rely on correct exit handling,
as fixing that is addressed in a followup patch.

Sponsored by: The FreeBSD Foundation

[mgorny]

BTW this test is getting a bit long as well but I don't have a good idea how to split it, except for just moving some functions into a second file and making some reusable base class.

[labath]

In D128638#3611932, @mgorny wrote:

BTW this test is getting a bit long as well but I don't have a good idea how to split it, except for just moving some functions into a second file and making some reusable base class.

I don't know if that's necessary right now, but the general approach seems fine to me.

[mgorny]

Ok, I'll commit as-is then and will revisit if we start seeing timeouts.

[mib]

Hi @mgorny,

There are some random test failures following this patch on lldb-debian-x86_64: https://lab.llvm.org/buildbot/#/builders/68/builds/34883

********************
Unresolved Tests (1):
  lldb-api :: tools/lldb-server/TestGdbRemoteFork.py

Would you mind taking a look at this ? Thanks!

[mgorny]

I am trying to but unfortunately the server seems to be down. Unfortunately, I can't reproduce this on my aarch64 test machine but maybe the test is taking too long now (we've been suspecting that might be the case). I'm going to split it and see if that helps.

[labath]

This looks like a genuine problem, see https://lab.llvm.org/buildbot/#/builders/68/builds/34903. If I had to guess, I'd say the problem is you're overwriting the context object holding the first pid (here: https://github.com/llvm/llvm-project/blob/main/lldb/test/API/tools/lldb-server/TestGdbRemoteForkNonStop.py#L87)

[mgorny]

In D128638#3618039, @labath wrote:

This looks like a genuine problem, see https://lab.llvm.org/buildbot/#/builders/68/builds/34903. If I had to guess, I'd say the problem is you're overwriting the context object holding the first pid (here: https://github.com/llvm/llvm-project/blob/main/lldb/test/API/tools/lldb-server/TestGdbRemoteForkNonStop.py#L87)

Good catch, I obviously meant to use pid1 and pid2 vars there. Thanks!

[mgorny]

Ok, it seems that all the relevant bots are green now. Thanks again!

[mib]

Not sure if this is related to this patch but TestGdbRemote_vContThreads.py is still timing out on the macOS bot:

https://green.lab.llvm.org/green/job/lldb-cmake/44958/console

[mgorny]

In D128638#3620751, @mib wrote:

Not sure if this is related to this patch but TestGdbRemote_vContThreads.py is still timing out on the macOS bot:

https://green.lab.llvm.org/green/job/lldb-cmake/44958/console

I see Jonas has already marked them "skip". Since I don't really have an environment to investigate this, I suppose this is as far as it gets.

lldb/test/API/tools/lldb-server/TestGdbRemoteFork.py

 import random
 
 import gdbremote_testcase
 from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 class TestGdbRemoteFork(gdbremote_testcase.GdbRemoteTestCaseBase):
 
     fork_regex = ("[$]T05thread:p([0-9a-f]+)[.]([0-9a-f]+);.*"
                   "{}:p([0-9a-f]+)[.]([0-9a-f]+).*")
+    fork_regex_nonstop = ("%Stop:T05thread:p([0-9a-f]+)[.]([0-9a-f]+);.*"
+                          "{}:p([0-9a-f]+)[.]([0-9a-f]+).*")
     fork_capture = {1: "parent_pid", 2: "parent_tid",
                     3: "child_pid", 4: "child_tid"}
 
-    def start_fork_test(self, args, variant="fork"):
+    def start_fork_test(self, args, variant="fork", nonstop=False):
         self.build()
         self.prep_debug_monitor_and_inferior(inferior_args=args)
         self.add_qSupported_packets(["multiprocess+",
                                      "{}-events+".format(variant)])
         ret = self.expect_gdbremote_sequence()
         self.assertIn("{}-events+".format(variant), ret["qSupported_response"])
         self.reset_test_sequence()
 
         # continue and expect fork
+        if nonstop:
+            self.test_sequence.add_log_lines([
+                "read packet: $QNonStop:1#00",
+                "send packet: $OK#00",
+                "read packet: $c#00",
+                "send packet: $OK#00",
+                {"direction": "send",
+                 "regex": self.fork_regex_nonstop.format(variant),
+                 "capture": self.fork_capture},
+                "read packet: $vStopped#00",
+                "send packet: $OK#00",
+            ], True)
+        else:
             self.test_sequence.add_log_lines([
                 "read packet: $c#00",
                 {"direction": "send", "regex": self.fork_regex.format(variant),
                  "capture": self.fork_capture},
             ], True)
         ret = self.expect_gdbremote_sequence()
         self.reset_test_sequence()
 
         return tuple(ret[x] for x in ("parent_pid", "parent_tid",
                                       "child_pid", "child_tid"))
 
     @add_test_categories(["fork"])
     def test_fork_multithreaded(self):
         _, _, child_pid, _ = self.start_fork_test(["thread:new"]*2 + ["fork"])
 
         # detach the forked child
         self.test_sequence.add_log_lines([
             "read packet: $D;{}#00".format(child_pid),
             "send packet: $OK#00",
             "read packet: $k#00",
         ], True)
         self.expect_gdbremote_sequence()
 
-    def fork_and_detach_test(self, variant):
+    def fork_and_detach_test(self, variant, nonstop=False):
         parent_pid, parent_tid, child_pid, child_tid = (
-            self.start_fork_test([variant], variant))
+            self.start_fork_test([variant], variant, nonstop=nonstop))
 
         # detach the forked child
         self.test_sequence.add_log_lines([
             "read packet: $D;{}#00".format(child_pid),
             "send packet: $OK#00",
             # verify that the current process is correct
             "read packet: $qC#00",
             "send packet: $QCp{}.{}#00".format(parent_pid, parent_tid),
def test_fork(self):
         # resume the parent
         self.test_sequence.add_log_lines([
             "read packet: $c#00",
             "send packet: $W00;process:{}#00".format(parent_pid),
         ], True)
         self.expect_gdbremote_sequence()
 
     @add_test_categories(["fork"])
+    def test_fork_nonstop(self):
+        parent_pid, _ = self.fork_and_detach_test("fork", nonstop=True)
+
+        # resume the parent
+        self.test_sequence.add_log_lines([
+            "read packet: $c#00",
+            "send packet: $OK#00",
+            "send packet: %Stop:W00;process:{}#00".format(parent_pid),
+            "read packet: $vStopped#00",
+            "send packet: $OK#00",
+        ], True)
+        self.expect_gdbremote_sequence()
+
+    @add_test_categories(["fork"])
     def test_vfork(self):
         parent_pid, parent_tid = self.fork_and_detach_test("vfork")
 
         # resume the parent
         self.test_sequence.add_log_lines([
             "read packet: $c#00",
             {"direction": "send",
              "regex": r"[$]T05thread:p{}[.]{}.*vforkdone.*".format(parent_pid,
                                                                    parent_tid),
              },
             "read packet: $c#00",
             "send packet: $W00;process:{}#00".format(parent_pid),
         ], True)
         self.expect_gdbremote_sequence()
 
-    def fork_and_follow_test(self, variant):
+    @add_test_categories(["fork"])
+    def test_vfork_nonstop(self):
+        parent_pid, parent_tid = self.fork_and_detach_test("vfork",
+                                                           nonstop=True)
+
+        # resume the parent
+        self.test_sequence.add_log_lines([
+            "read packet: $c#00",
+            "send packet: $OK#00",
+            {"direction": "send",
+             "regex": r"%Stop:T05thread:p{}[.]{}.*vforkdone.*".format(
+                 parent_pid, parent_tid),
+             },
+            "read packet: $vStopped#00",
+            "send packet: $OK#00",
+            "read packet: $c#00",
+            "send packet: $OK#00",
+            "send packet: %Stop:W00;process:{}#00".format(parent_pid),
+            "read packet: $vStopped#00",
+            "send packet: $OK#00",
+        ], True)
+        self.expect_gdbremote_sequence()
+
+    def fork_and_follow_test(self, variant, nonstop=False):
         parent_pid, parent_tid, child_pid, child_tid = (
-            self.start_fork_test([variant], variant))
+            self.start_fork_test([variant], variant, nonstop=nonstop))
 
         # switch to the forked child
         self.test_sequence.add_log_lines([
             "read packet: $Hgp{}.{}#00".format(child_pid, child_tid),
             "send packet: $OK#00",
             "read packet: $Hcp{}.{}#00".format(child_pid, child_tid),
             "send packet: $OK#00",
             # detach the parent
             "read packet: $D;{}#00".format(parent_pid),
             "send packet: $OK#00",
             # verify that the correct processes are detached/available
             "read packet: $Hgp{}.{}#00".format(parent_pid, parent_tid),
             "send packet: $Eff#00",
             "read packet: $Hgp{}.{}#00".format(child_pid, child_tid),
             "send packet: $OK#00",
             # then resume the child
             "read packet: $c#00",
+        ], True)
+
+        if nonstop:
+            self.test_sequence.add_log_lines([
+                "send packet: $OK#00",
+                "send packet: %Stop:W00;process:{}#00".format(child_pid),
+                "read packet: $vStopped#00",
+                "send packet: $OK#00",
+            ], True)
+        else:
+            self.test_sequence.add_log_lines([
                 "send packet: $W00;process:{}#00".format(child_pid),
             ], True)
         self.expect_gdbremote_sequence()
 
     @add_test_categories(["fork"])
     def test_fork_follow(self):
         self.fork_and_follow_test("fork")
 
     @add_test_categories(["fork"])
+    def test_fork_follow_nonstop(self):
+        self.fork_and_follow_test("fork", nonstop=True)
+
+    @add_test_categories(["fork"])
     def test_vfork_follow(self):
         self.fork_and_follow_test("vfork")
 
     @add_test_categories(["fork"])
+    def test_vfork_follow_nonstop(self):
+        self.fork_and_follow_test("vfork", nonstop=True)
+
+    @add_test_categories(["fork"])
     def test_select_wrong_pid(self):
         self.build()
         self.prep_debug_monitor_and_inferior()
         self.add_qSupported_packets(["multiprocess+"])
         ret = self.expect_gdbremote_sequence()
         self.assertIn("multiprocess+", ret["qSupported_response"])
         self.reset_test_sequence()
 
def test_detach_current(self):
         self.test_sequence.add_log_lines([
             "read packet: $D;{}#00".format(pid),
             "send packet: $OK#00",
             "read packet: $qC#00",
             "send packet: $E44#00",
         ], True)
         self.expect_gdbremote_sequence()
 
-    @add_test_categories(["fork"])
-    def test_detach_all(self):
+    def detach_all_test(self, nonstop=False):
         parent_pid, parent_tid, child_pid, child_tid = (
-            self.start_fork_test(["fork"]))
+            self.start_fork_test(["fork"], nonstop=nonstop))
 
         self.test_sequence.add_log_lines([
             # double-check our PIDs
             "read packet: $Hgp{}.{}#00".format(parent_pid, parent_tid),
             "send packet: $OK#00",
             "read packet: $Hgp{}.{}#00".format(child_pid, child_tid),
             "send packet: $OK#00",
             # detach all processes
             "read packet: $D#00",
             "send packet: $OK#00",
             # verify that both PIDs are invalid now
             "read packet: $Hgp{}.{}#00".format(parent_pid, parent_tid),
             "send packet: $Eff#00",
             "read packet: $Hgp{}.{}#00".format(child_pid, child_tid),
             "send packet: $Eff#00",
         ], True)
         self.expect_gdbremote_sequence()
 
     @add_test_categories(["fork"])
+    def test_detach_all(self):
+        self.detach_all_test()
+
+    @add_test_categories(["fork"])
+    def test_detach_all_nonstop(self):
+        self.detach_all_test(nonstop=True)
+
+    @add_test_categories(["fork"])
     def test_kill_all(self):
         parent_pid, _, child_pid, _ = self.start_fork_test(["fork"])
 
         exit_regex = "[$]X09;process:([0-9a-f]+)#.*"
         self.test_sequence.add_log_lines([
             # kill all processes
             "read packet: $k#00",
             {"direction": "send", "regex": exit_regex,
              "capture": {1: "pid1"}},
             {"direction": "send", "regex": exit_regex,
              "capture": {1: "pid2"}},
         ], True)
         ret = self.expect_gdbremote_sequence()
         self.assertEqual(set([ret["pid1"], ret["pid2"]]),
                          set([parent_pid, child_pid]))
 
-    def vkill_test(self, kill_parent=False, kill_child=False):
+    @add_test_categories(["fork"])
+    def test_kill_all_nonstop(self):
+        parent_pid, _, child_pid, _ = self.start_fork_test(["fork"],
+                                                           nonstop=True)
+
+        exit_regex = "X09;process:([0-9a-f]+)"
+        # Depending on a potential race, the second kill may make it into
+        # the async queue before we issue vStopped or after.  In the former
+        # case, we should expect the exit status in reply to vStopped.
+        # In the latter, we should expect an OK response (queue empty),
+        # followed by another async notification.
+        vstop_regex = "[$](OK|{})#.*".format(exit_regex)
+        self.test_sequence.add_log_lines([
+            # kill all processes
+            "read packet: $k#00",
+            "send packet: $OK#00",
+            {"direction": "send", "regex": "%Stop:{}#.*".format(exit_regex),
+             "capture": {1: "pid1"}},
+            "read packet: $vStopped#00",
+            {"direction": "send", "regex": vstop_regex,
+             "capture": {1: "vstop_reply", 2: "pid2"}},
+        ], True)
+        ret = self.expect_gdbremote_sequence()
+        pid1 = ret["pid1"]
+        if ret["vstop_reply"] == "OK":
+            self.reset_test_sequence()
+            self.test_sequence.add_log_lines([
+                {"direction": "send", "regex": "%Stop:{}#.*".format(exit_regex),
+                 "capture": {1: "pid2"}},
+            ], True)
+            ret = self.expect_gdbremote_sequence()
+        pid2 = ret["pid2"]
+        self.reset_test_sequence()
+        self.test_sequence.add_log_lines([
+            "read packet: $vStopped#00",
+            "send packet: $OK#00",
+        ], True)
+        self.expect_gdbremote_sequence()
+        self.assertEqual(set([ret["pid1"], ret["pid2"]]),
+                         set([parent_pid, child_pid]))
+
+    def vkill_test(self, kill_parent=False, kill_child=False, nonstop=False):
         assert kill_parent or kill_child
         parent_pid, parent_tid, child_pid, child_tid = (
-            self.start_fork_test(["fork"]))
+            self.start_fork_test(["fork"], nonstop=nonstop))
 
         if kill_parent:
             self.test_sequence.add_log_lines([
                 # kill the process
                 "read packet: $vKill;{}#00".format(parent_pid),
                 "send packet: $OK#00",
             ], True)
         if kill_child:
class TestGdbRemoteFork(gdbremote_testcase.GdbRemoteTestCaseBase):
     @add_test_categories(["fork"])
     def test_vkill_parent(self):
         self.vkill_test(kill_parent=True)
 
     @add_test_categories(["fork"])
     def test_vkill_both(self):
         self.vkill_test(kill_parent=True, kill_child=True)
 
+    @add_test_categories(["fork"])
+    def test_vkill_both_nonstop(self):
+        self.vkill_test(kill_parent=True, kill_child=True, nonstop=True)
+
     def resume_one_test(self, run_order, use_vCont=False):
         parent_pid, parent_tid, child_pid, child_tid = (
             self.start_fork_test(["fork", "trap"]))
 
         parent_expect = [
             "[$]T05thread:p{}.{};.*".format(parent_pid, parent_tid),
             "[$]W00;process:{}#.*".format(parent_pid),
         ]