[lldb] [gdb-remote] Client support for using the non-stop protocol

[mgorny]

Summary

Add a client support for using the non-stop protocol to run processes
in all-stop mode. The operations are still done synchronously, so this
does not enable client to asynchronously resume threads or send
additional requests while the process is running.

If plugin.process.gdb-remote.use-non-stop-protocol is set to true prior
to connecting and the server reports support for QNonStop extension,
the client attempts to enable non-stop mode. Using non-stop mode has
the following implications:

1. When a continue packet is sent, the client receives the synchronous "OK" notification and blocks until %Stop notification arrives. Then it drains the notification queue.

2. When a "?" packet is sent, the client issues "vStopped" command series in order to drain the notification queue.

3. When the process is about to be stopped, the client issues a "vCtrlC" command instead of the break sequence.

Sponsored by: The FreeBSD Foundation

[mgorny]

Split the main tests into two variants: one that assumes that server runs in all-stop mode (like lldb-server), the other assuming it runs in non-stop mode (like gdbserver on Linux).

[mgorny]

Use vCont;t instead of vCtrlC to stop other threads for better gdbserver compatibility. Defer enabling non-stop until after the ? status query, to workaround a bug in gdbserver.

[labath]

(I think Jim would be interested in following this.)

I'm not sure I understand what this does. Is it that, when the setting is enabled, lldb uses non-stop communication at the protocol level, but then forces an all-stop whenever one thread stops (so that the upper layers are happy)?

API tests required changing the Process::Halt() API to inspect process'
private state rather than the public state, that remains eStateUnloaded
in gdb_remote_client tests.

I don't think that checking the private state is correct there. Doesn't that just mean that there's a bug in the tests that it's not simulating a real gdbserver sufficiently well? Or maybe (if the simulated responses are reasonable) lldb should be made to work (properly set the state) for the simulated responses?

lldb/packages/Python/lldbsuite/test/gdbclientutils.py

26

I guess this comment needs updating. I've been also thinking whether this is a sufficiently obvious way to send notifications, but I kinda like the brevity of it.

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

128

not vCont;t ? Also, I think these log lines are unnecessarily verbose, and are obscuring the real code. I think a single log line (indicating failure) should be more than enough as more information (the kind of failure, for instance) can be determined by looking at the packet log.

147

This is unfortunate. Isn't there a better way to do this? Why aren't you using vCtrlC as all the comments say? Did you find any parallel to this in gdb?

450

Isn't the vCtrlC packet supposed to have a response (OK or Exx according to https://sourceware.org/gdb/onlinedocs/gdb/Packets.html#Packets) ?

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.h

62

Maybe call this GetNonStopEnabled or GetNonStopInUse ?

lldb/test/API/functionalities/gdb_remote_client/TestGDBRemoteClient.py

686

This is not a good assertion. Maybe you could replace check that after-the-fact with assertPacketLogContains?

717

I think here you'll need to manually extract (and verify) the running/stopped events from the appropriate listener. In async mode, Continue and Stop return immediately, so there's no guarantee that anything has actually happened at this point.

[mgorny]

Thanks for the review and welcome back!

In D126614#3572845, @labath wrote:

(I think Jim would be interested in following this.)

Ok, I've added him to all the related diffs.

I'm not sure I understand what this does. Is it that, when the setting is enabled, lldb uses non-stop communication at the protocol level, but then forces an all-stop whenever one thread stops (so that the upper layers are happy)?

Yes. This way, it preserves compatibility with genuine gdbserver that actually implements non-stop mode.

API tests required changing the Process::Halt() API to inspect process'
private state rather than the public state, that remains eStateUnloaded
in gdb_remote_client tests.

I don't think that checking the private state is correct there. Doesn't that just mean that there's a bug in the tests that it's not simulating a real gdbserver sufficiently well? Or maybe (if the simulated responses are reasonable) lldb should be made to work (properly set the state) for the simulated responses?

Yes, I think it's limitation of the test logic that public state isn't being updated. I haven't really managed to get past figuring that part out. Do you have any suggestion where to look for the problematic code?

That said, I'm currently focused on multiprocess support in lldb-server. While this is relevant, it isn't blocking server and I don't want to distract myself switching contexts right now but I'll look into implementing your suggestions later on.

lldb/packages/Python/lldbsuite/test/gdbclientutils.py

26

Yeah, I can't say I'm very proud of it but I can't think of a way that would both be really clean and wouldn't involve major changes all around the place.

I suppose one option would be to wrap these packets in a minimal Notification class, and then do something like:

if isinstance(message, Notification):
  ...

and then pass Notification("foo") instead of `"%foo" but I'm not convinced it'd be actually more obvious.

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

128

Good catch.

Just to be clear, do I understand that you're talking of having a single LLDB_LOG that covers all non-successful responses?

147

Yes, it is unfortunate. I'm going to think more and try to figure out a better way. I was originally thinking of simply not waiting for the response and instead letting it come after and then taking care of any pending notifications before sending the next continue packet (this would require merging D126655 first) but I'm somewhat afraid of race conditions. Though maybe unnecessarily.

Though OTOH I guess 500 ms may be insufficient for debugging over the Internet, and this would lead to even worse results.

As for vCtrlC, I've changed the code to use vCont;t as the former didn't work well with gdbserver (and the latter is also more correct wrt the protocol). I've forgotten to update the comments.

450

Indeed you're correct. I guess I've missed it because stop reason handler took care of eating it.

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.h

62

I kinda wanted to emphasize 'protocol' here, to make it clear we aren't introducing full-scale non-stop support.

lldb/test/API/functionalities/gdb_remote_client/TestGDBRemoteClient.py

686

Do you mean because the client could issue an irrelevant QNonStop:0 at some point? I suppose that makes sense then.

[mgorny]

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

147

Ok, I've been thinking and thinking about this and I have a better idea. However, since this is not needed for LLGS and only for true gdbserver, I'm going to split it into a separate patch.

My idea is basically to, in a loop:

1. issue vCont;t to request stopping all threads.
2. issue ? to get the list of all stopped threads.
3. issue qfThreadInfo to get list of all threads.

and basically do this until both lists are the same. That should take care of pretty much every corner case and race condition I can think of.

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.h

62

Thinking about it again, I suppose you're right.

[mgorny]

Partial update. Implemented most of the requests, except what noted below. Also implemented support for %Stdio: stdout forwarding style as was eventually implemented in LLGS.

I haven't been able to figure out how to make the test suite work without the public-private state change.

As for the OK response to vCtrlC… it's hard because the incoming packet causes the pending read in GDBRemoteClientBase::SendContinuePacketAndWaitForResponse() to succeed, so the OK is swallowed there. I can't think of any way to do this that wouldn't be super ugly.

[mgorny]

Good news, everyone! It turns out that the private/public state is not needed anymore — I guess fixing the races in tests resolved that.

Did some final cleanup of tests as well. I guess this covers everything except for grabbing the "OK" response to "vCtrlC" interrupt.

[mgorny]

Move common parts of the mock responder into a base class. Include PIDs in responses, and add trailing ; to T packets.

[mgorny]

@labath, gentle ping. We appreciate all the work you've done reviewing my patches already, and we realize this is a lot of extra work for you. However, could you please try to review this one as well? It's starting to block our further work.

[labath]

Sorry about the delay. I've been putting this off because it's complicated, although that's a lousy excuse. The gist of it is that I don't think that the existing setup of the "async threads" and "continue locks" is going to work in this new world. The reason it has worked until now is because the async thread has been exclusively reading packets, and the only thing that the other threads could do is to write a special single-character "packet" which has no response. With this patch neither of those things is true anymore -- the async thread sends a bunch of notification packets, and vCtrlC packet sent by other threads is a regular packet that we should get a response to. Currently, I don't see anything preventing the two threads from talking over each other.

I think we need to do something about this. *One* way to do it (I'm sure there are others) is to move all the (interruption-related) communication into the async thread. Instead of sending the interrupt (both ^C and vCtrlC) packet from the interrupting thread directly, we would instead signal the async thread in some way, and let it do the interruption for us. The communication will be single threaded then, and it should be easier to maintain the protocol integrity. We might be able to reuse/repurpose the InterruptRead functionality to achieve this. (It's possible that won't work on windows -- I should be able to help if that's the case.)

lldb/test/API/functionalities/gdb_remote_client/TestNonStop.py

80–82 ↗

(On Diff #443428)

I don't think this is necessary anymore. These days we should be reseting all settings before each test.

[mgorny]

Thanks for the suggestions. I have to admit I don't necessarily like the InterruptRead() approach but given that the alternative involves major changes to how we read packets, I suppose it's a good enough solution (and big thanks for that suggestion!). All should be done now.

[labath]

Ok, I think I like this, but since this is pretty finicky code, could I ask you to split it up into smaller patches? The first could be just the interruption refactor. The second could contain all the m_notification_packet_queue and ReadPacket stuff (I think this one could come with the a c++ unit test in unittests/Process/gdb-remote/. And the last one could be the vCtrlC/vCont;t stuff and whatever is left (?)

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

105

Could we just send vCont;t to begin with? (I know this was discussed already, but that was before we switched to the new interruption mechanism)

[mgorny]

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

105

I'm sorry but I don't understand what you're asking about. The vCont;t part is split into D129554 since it involves some finicky logic because we can't predict if we're going to get any notifications or not.

[labath]

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

105

Nevermind, it's me who is not understanding things. I forgot about what's the problem with vCont;t (serves me right for not reviewing on time).

I guess my point was that we may be able to ignore the vCtrlC part, and immediately start off with a vCont;t (and then do the wait-until-everything-stops dance), but we can leave that for the other patch.

[mgorny]

Rebase on top of the split patches.

lldb/packages/Python/lldbsuite/test/gdbclientutils.py

def checksum(message):
     return check % 256
 
 
 def frame_packet(message):
     """
     Create a framed packet that's ready to send over the GDB connection
     channel.
 
     Framing includes surrounding the message between $ and #, and appending
     a two character hex checksum.
     """
-    return "$%s#%02x" % (message, checksum(message))
+    prefix = "$"
+    if message.startswith("%"):
+        prefix = "%"
+        message = message[1:]
+    return "%s%s#%02x" % (prefix, message, checksum(message))
 
 
 def escape_binary(message):
     """
     Escape the binary message using the process described in the GDB server
     protocol documentation.
 
     Most bytes are sent through as-is, but $, #, and { are escaped by writing
def respond(self, packet):
             return self.qLaunchSuccess()
         if packet.startswith("QEnvironment:"):
             return self.QEnvironment(packet)
         if packet.startswith("QEnvironmentHexEncoded:"):
             return self.QEnvironmentHexEncoded(packet)
         if packet.startswith("qRegisterInfo"):
             regnum = int(packet[len("qRegisterInfo"):], 16)
             return self.qRegisterInfo(regnum)
+        if packet.startswith("QNonStop:"):
+            return self.QNonStop(int(packet.split(":", 1)[1]))
+        if packet == "vStopped":
+            return self.vStopped()
+        if packet == "vCtrlC":
+            return self.vCtrlC()
         if packet == "k":
             return self.k()
 
         return self.other(packet)
 
     def qsProcessInfo(self):
         return "E04"
 
def QEnvironment(self, packet):
         return "OK"
 
     def QEnvironmentHexEncoded(self, packet):
         return "OK"
 
     def qRegisterInfo(self, num):
         return ""
 
+    def QNonStop(self, enabled):
+        return ""
+
+    def vStopped(self):
+        return ""
+
+    def vCtrlC(self):
+        return ""
+
     def k(self):
         return ["W01", self.RESPONSE_DISCONNECT]
 
     """
     Raised when we receive a packet for which there is no default action.
     Override the responder class to implement behavior suitable for the test at
     hand.
     """

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.h

PacketResult SendPacketAndWaitForResponse(
       llvm::StringRef payload, StringExtractorGDBRemote &response,
       std::chrono::seconds interrupt_timeout = std::chrono::seconds(0));
 
   PacketResult SendPacketAndReceiveResponseWithOutputSupport(
       llvm::StringRef payload, StringExtractorGDBRemote &response,
       std::chrono::seconds interrupt_timeout,
       llvm::function_ref<void(llvm::StringRef)> output_callback);
 
+  bool GetNonStopProtocol() { return m_non_stop_protocol; }
+
+  void SetNonStopProtocol(bool enabled) { m_non_stop_protocol = enabled; }
+
+  bool DrainNotificationQueue(llvm::StringRef command);
+
   class Lock {
   public:
     // If interrupt_timeout == 0 seconds, only take the lock if the target is
     // not running. If using this option, use the fact that the
     // interrupt_timeout is defaulted so it will be obvious at the call site
     // that you are choosing this mode. If it is non-zero, interrupt the target
     // if it is running, waiting for the given timeout for the interrupt to
     // succeed.
private:
   /// Number of threads interested in sending.
   uint32_t m_async_count;
 
   /// Whether the continue thread has control.
   bool m_is_running;
 
   /// Whether we should resume after a stop.
   bool m_should_stop;
+
+  /// Whether non-stop protocol should be used.
+  bool m_non_stop_protocol = false;
   /// @}
 
   /// This handles the synchronization between individual async threads. For
   /// now they just use a simple mutex.
   std::recursive_mutex m_async_mutex;
 
   bool ShouldStop(const UnixSignals &signals,
                   StringExtractorGDBRemote &response);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteClientBase.cpp

response.Clear();
     std::lock_guard<std::mutex> lock(m_mutex);
     m_continue_packet = std::string(payload);
     m_should_stop = false;
   }
   ContinueLock cont_lock(*this);
   if (!cont_lock)
     return eStateInvalid;
   OnRunPacketSent(true);
-  // The main ReadPacket loop wakes up at computed_timeout intervals, just to 
+  // The main ReadPacket loop wakes up at computed_timeout intervals, just to
   // check that the connection hasn't dropped.  When we wake up we also check
   // whether there is an interrupt request that has reached its endpoint.
-  // If we want a shorter interrupt timeout that kWakeupInterval, we need to 
+  // If we want a shorter interrupt timeout that kWakeupInterval, we need to
   // choose the shorter interval for the wake up as well.
-  std::chrono::seconds computed_timeout = std::min(interrupt_timeout, 
-                                                   kWakeupInterval);
+  std::chrono::seconds computed_timeout =
+      std::min(interrupt_timeout, kWakeupInterval);
   for (;;) {
     PacketResult read_result = ReadPacket(response, computed_timeout, false);
     // Reset the computed_timeout to the default value in case we are going
     // round again.
     computed_timeout = std::min(interrupt_timeout, kWakeupInterval);
     switch (read_result) {
     case PacketResult::ErrorReplyTimeout: {
       std::lock_guard<std::mutex> lock(m_mutex);
       if (m_async_count == 0) {
         continue;
       }
       auto cur_time = steady_clock::now();
       if (cur_time >= m_interrupt_endpoint)
         return eStateInvalid;
       else {
         // We woke up and found an interrupt is in flight, but we haven't
         // exceeded the interrupt wait time.  So reset the wait time to the
         // time left till the interrupt timeout.  But don't wait longer
         // than our wakeup timeout.
         auto new_wait = m_interrupt_endpoint - cur_time;
-        computed_timeout = std::min(kWakeupInterval,
+        computed_timeout = std::min(
+            kWakeupInterval,
             std::chrono::duration_cast<std::chrono::seconds>(new_wait));
         continue;
       }
       break;
     }
     case PacketResult::Success:
       break;
     default:
       LLDB_LOGF(log, "GDBRemoteClientBase::%s () ReadPacket(...) => false",
                 __FUNCTION__);
       return eStateInvalid;
     }
     if (response.Empty())
       return eStateInvalid;
 
-    const char stop_type = response.GetChar();
     LLDB_LOGF(log, "GDBRemoteClientBase::%s () got packet: %s", __FUNCTION__,
               response.GetStringRef().data());
 
+    // We do not currently support full asynchronous communication.  Instead,
+    // when in non-stop mode, we:
+    // 1) get the "OK" response to the continue packet
+    // 2) wait for the asynchronous %Stop notification
+    // 3) drain the notification queue
+    // 4) issue a "vCtrlC" command to ensure that all threads stop
+    // 5) repeat 1-3 for the response to this packet
+
+    if (response.IsOKResponse())
+      continue;
+    // TODO: get a proper mechanism for async notifications
+    if (response.GetStringRef().startswith("Stop:")) {
+      response.Reset(response.GetStringRef().substr(5));
+
+      if (!DrainNotificationQueue("vStopped"))
+        return eStateInvalid;
+
+      switch (response.GetStringRef()[0]) {
+      case 'W':
+      case 'X':
+        // Do not attempt to stop the process if it exited already.
+        break;
+      default:
+        StringExtractorGDBRemote stop_response;
+        if (SendPacketAndWaitForResponseNoLock("vCtrlC", stop_response) !=
+            PacketResult::Success) {
+          LLDB_LOGF(log, "GDBRemoteClientBase::%s () vCtrlC failed",
+                    __FUNCTION__);
+          return eStateInvalid;
+        }
+
+        LLDB_LOGF(log, "GDBRemoteClientBase::%s () vCtrlC response: %s",
+                  __FUNCTION__, stop_response.GetStringRef().data());
+        if (stop_response.IsUnsupportedResponse()) {
+          LLDB_LOGF(log, "GDBRemoteClientBase::%s () vCtrlC unsupported",
+                    __FUNCTION__);
+          return eStateInvalid;
+        }
+        if (!stop_response.IsOKResponse()) {
+          LLDB_LOGF(log, "GDBRemoteClientBase::%s () vCtrlC failed",
+                    __FUNCTION__);
+          return eStateInvalid;
+        }
+
+        // vCtrlC may not do anything, so timeout if we don't get notification
+        if (ReadPacket(stop_response, milliseconds(500), false) ==
+            PacketResult::Success) {
+          if (!stop_response.GetStringRef().startswith("Stop:")) {
+            LLDB_LOGF(log,
+                      "GDBRemoteClientBase::%s () unexpected response "
+                      "after vCtrlC",
+                      __FUNCTION__);
+            return eStateInvalid;
+          }
+
+          if (!DrainNotificationQueue("vStopped"))
+            return eStateInvalid;
+        }
+      }
+    }
+
+    const char stop_type = response.GetChar();
+
     switch (stop_type) {
     case 'W':
     case 'X':
       return eStateExited;
     case 'E':
       // ERROR
       return eStateInvalid;
     default:
bool GDBRemoteClientBase::ShouldStop(const UnixSignals &signals,
   return false;
 }
 
 void GDBRemoteClientBase::OnRunPacketSent(bool first) {
   if (first)
     BroadcastEvent(eBroadcastBitRunPacketSent, nullptr);
 }
 
+bool GDBRemoteClientBase::DrainNotificationQueue(llvm::StringRef command) {
+  Log *log = GetLog(GDBRLog::Process);
+  for (;;) {
+    StringExtractorGDBRemote response;
+    if (SendPacketAndWaitForResponseNoLock(command, response) ==
+        PacketResult::Success) {
+      if (response.IsUnsupportedResponse()) {
+        LLDB_LOGF(log, "GDBRemoteClientBase::%s () %s unsupported",
+                  __FUNCTION__, command.data());
+        return false;
+      }
+
+      LLDB_LOGF(log, "GDBRemoteClientBase::%s () %s response: %s", __FUNCTION__,
+                command.data(), response.GetStringRef().data());
+      if (response.IsOKResponse())
+        return true;
+    } else {
+      LLDB_LOGF(log, "GDBRemoteClientBase::%s () sending %s failed",
+                __FUNCTION__, command.data());
+      return false;
+    }
+  }
+}
+
 ///////////////////////////////////////
 // GDBRemoteClientBase::ContinueLock //
 ///////////////////////////////////////
 
 GDBRemoteClientBase::ContinueLock::ContinueLock(GDBRemoteClientBase &comm)
     : m_comm(comm), m_acquired(false) {
   lock();
 }
if (m_comm.m_is_running && m_interrupt_timeout == std::chrono::seconds(0))
     return; // We were asked to avoid interrupting the sender. Lock is not
             // acquired.
 
   ++m_comm.m_async_count;
   if (m_comm.m_is_running) {
     if (m_comm.m_async_count == 1) {
       // The sender has sent the continue packet and we are the first async
       // packet. Let's interrupt it.
+      bool success;
+      if (m_comm.GetNonStopProtocol())
+        success = m_comm.SendPacketNoLock("vCtrlC") == PacketResult::Success;
+      else {
         const char ctrl_c = '\x03';
         ConnectionStatus status = eConnectionStatusSuccess;
-      size_t bytes_written = m_comm.Write(&ctrl_c, 1, status, nullptr);
-      if (bytes_written == 0) {
+        success = m_comm.Write(&ctrl_c, 1, status, nullptr) != 0;
+      }
+      if (!success) {
         --m_comm.m_async_count;
         LLDB_LOGF(log, "GDBRemoteClientBase::Lock::Lock failed to send "
                        "interrupt packet");
         return;
       }
       m_comm.m_interrupt_endpoint = steady_clock::now() + m_interrupt_timeout;
-      if (log)
+      if (log && !m_comm.GetNonStopProtocol())
           log->PutCString("GDBRemoteClientBase::Lock::Lock sent packet: \\x03");
     }
     m_comm.m_cv.wait(lock, [this] { return !m_comm.m_is_running; });
     m_did_interrupt = true;
   }
   m_acquired = true;
 }
 
 GDBRemoteClientBase::Lock::~Lock() {
   if (!m_acquired)
     return;
   {
     std::unique_lock<std::mutex> lock(m_comm.m_mutex);
     --m_comm.m_async_count;
   }
   m_comm.m_cv.notify_one();
 }

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.h

public:
 
   bool SetCurrentThreadForRun(uint64_t tid,
                               lldb::pid_t pid = LLDB_INVALID_PROCESS_ID);
 
   bool GetQXferAuxvReadSupported();
 
   void EnableErrorStringInPacket();
 
+  void EnableNonStop();
+
   bool GetQXferLibrariesReadSupported();
 
   bool GetQXferLibrariesSVR4ReadSupported();
 
   uint64_t GetRemoteMaxPacketSize();
 
   bool GetEchoSupported();
 
SendTraceGetState(llvm::StringRef type,
                     std::chrono::seconds interrupt_timeout);
 
   llvm::Expected<std::vector<uint8_t>>
   SendTraceGetBinaryData(const TraceGetBinaryDataRequest &request,
                          std::chrono::seconds interrupt_timeout);
 
   bool GetSaveCoreSupported() const;
 
+  bool GetNonStopSupported() const;
+
 protected:
   LazyBool m_supports_not_sending_acks = eLazyBoolCalculate;
   LazyBool m_supports_thread_suffix = eLazyBoolCalculate;
   LazyBool m_supports_threads_in_stop_reply = eLazyBoolCalculate;
   LazyBool m_supports_vCont_all = eLazyBoolCalculate;
   LazyBool m_supports_vCont_any = eLazyBoolCalculate;
   LazyBool m_supports_vCont_c = eLazyBoolCalculate;
   LazyBool m_supports_vCont_C = eLazyBoolCalculate;
protected:
   LazyBool m_supports_jThreadExtendedInfo = eLazyBoolCalculate;
   LazyBool m_supports_jLoadedDynamicLibrariesInfos = eLazyBoolCalculate;
   LazyBool m_supports_jGetSharedCacheInfo = eLazyBoolCalculate;
   LazyBool m_supports_QPassSignals = eLazyBoolCalculate;
   LazyBool m_supports_error_string_reply = eLazyBoolCalculate;
   LazyBool m_supports_multiprocess = eLazyBoolCalculate;
   LazyBool m_supports_memory_tagging = eLazyBoolCalculate;
   LazyBool m_supports_qSaveCore = eLazyBoolCalculate;
+  LazyBool m_supports_QNonStop = eLazyBoolCalculate;
   LazyBool m_uses_native_signals = eLazyBoolCalculate;
 
   bool m_supports_qProcessInfoPID : 1, m_supports_qfProcessInfo : 1,
       m_supports_qUserName : 1, m_supports_qGroupName : 1,
       m_supports_qThreadStopInfo : 1, m_supports_z0 : 1, m_supports_z1 : 1,
       m_supports_z2 : 1, m_supports_z3 : 1, m_supports_z4 : 1,
       m_supports_QEnvironment : 1, m_supports_QEnvironmentHexEncoded : 1,
       m_supports_qSymbol : 1, m_qSymbol_requests_done : 1,

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

if (!did_exec) {
     m_qGDBServerVersion_is_valid = eLazyBoolCalculate;
     m_supports_alloc_dealloc_memory = eLazyBoolCalculate;
     m_supports_memory_region_info = eLazyBoolCalculate;
     m_prepare_for_reg_writing_reply = eLazyBoolCalculate;
     m_attach_or_wait_reply = eLazyBoolCalculate;
     m_avoid_g_packets = eLazyBoolCalculate;
     m_supports_multiprocess = eLazyBoolCalculate;
     m_supports_qSaveCore = eLazyBoolCalculate;
+    m_supports_QNonStop = eLazyBoolCalculate;
     m_supports_qXfer_auxv_read = eLazyBoolCalculate;
     m_supports_qXfer_libraries_read = eLazyBoolCalculate;
     m_supports_qXfer_libraries_svr4_read = eLazyBoolCalculate;
     m_supports_qXfer_features_read = eLazyBoolCalculate;
     m_supports_qXfer_memory_map_read = eLazyBoolCalculate;
     m_supports_qXfer_siginfo_read = eLazyBoolCalculate;
     m_supports_augmented_libraries_svr4_read = eLazyBoolCalculate;
     m_uses_native_signals = eLazyBoolCalculate;
void GDBRemoteCommunicationClient::GetRemoteQSupported() {
   m_supports_qXfer_features_read = eLazyBoolNo;
   m_supports_qXfer_memory_map_read = eLazyBoolNo;
   m_supports_qXfer_siginfo_read = eLazyBoolNo;
   m_supports_multiprocess = eLazyBoolNo;
   m_supports_qEcho = eLazyBoolNo;
   m_supports_QPassSignals = eLazyBoolNo;
   m_supports_memory_tagging = eLazyBoolNo;
   m_supports_qSaveCore = eLazyBoolNo;
+  m_supports_QNonStop = eLazyBoolNo;
   m_uses_native_signals = eLazyBoolNo;
 
   m_max_packet_size = UINT64_MAX; // It's supposed to always be there, but if
                                   // not, we assume no limit
 
   // build the qSupported packet
   std::vector<std::string> features = {"xmlRegisters=i386,arm,mips,arc",
                                        "multiprocess+", "fork-events+",
for (llvm::StringRef x : llvm::split(response.GetStringRef(), ';')) {
       else if (x == "QPassSignals+")
         m_supports_QPassSignals = eLazyBoolYes;
       else if (x == "multiprocess+")
         m_supports_multiprocess = eLazyBoolYes;
       else if (x == "memory-tagging+")
         m_supports_memory_tagging = eLazyBoolYes;
       else if (x == "qSaveCore+")
         m_supports_qSaveCore = eLazyBoolYes;
+      else if (x == "QNonStop+")
+        m_supports_QNonStop = eLazyBoolYes;
       else if (x == "native-signals+")
         m_uses_native_signals = eLazyBoolYes;
       // Look for a list of compressions in the features list e.g.
       // qXfer:features:read+;PacketSize=20000;qEcho+;SupportedCompressions=zlib-
       // deflate,lzma
       else if (x.consume_front("SupportedCompressions=")) {
         llvm::SmallVector<llvm::StringRef, 4> compressions;
         x.split(compressions, ',');
LazyBool GDBRemoteCommunicationClient::GetThreadPacketSupported(
   }
   return eLazyBoolNo;
 }
 
 bool GDBRemoteCommunicationClient::GetSaveCoreSupported() const {
   return m_supports_qSaveCore == eLazyBoolYes;
 }
 
+bool GDBRemoteCommunicationClient::GetNonStopSupported() const {
+  return m_supports_QNonStop == eLazyBoolYes;
+}
+
 StructuredData::ObjectSP GDBRemoteCommunicationClient::GetThreadsInfo() {
   // Get information on all threads at one using the "jThreadsInfo" packet
   StructuredData::ObjectSP object_sp;
 
   if (m_supports_jThreadsInfo) {
     StringExtractorGDBRemote response;
     response.SetResponseValidatorToJSON();
     if (SendPacketAndWaitForResponse("jThreadsInfo", response) ==
if (SendPacketAndWaitForResponse("QEnableErrorStrings", response) ==
         PacketResult::Success) {
       if (response.IsOKResponse()) {
         m_supports_error_string_reply = eLazyBoolYes;
       }
     }
   }
 }
 
+void GDBRemoteCommunicationClient::EnableNonStop() {
+  if (!GetNonStopSupported())
+    return;
+
+  StringExtractorGDBRemote response;
+  if (SendPacketAndWaitForResponse("QNonStop:1", response) ==
+      PacketResult::Success) {
+    if (response.IsOKResponse())
+      SetNonStopProtocol(true);
+  }
+}
+
 bool GDBRemoteCommunicationClient::GetLoadedDynamicLibrariesInfosSupported() {
   if (m_supports_jLoadedDynamicLibrariesInfos == eLazyBoolCalculate) {
     StringExtractorGDBRemote response;
     m_supports_jLoadedDynamicLibrariesInfos = eLazyBoolNo;
     if (SendPacketAndWaitForResponse("jGetLoadedDynamicLibrariesInfos:",
                                      response) == PacketResult::Success) {
       if (response.IsOKResponse()) {
         m_supports_jLoadedDynamicLibrariesInfos = eLazyBoolYes;
if (ret->pid != LLDB_INVALID_PROCESS_ID)
       m_curr_pid_run = ret->pid;
     m_curr_tid_run = ret->tid;
   }
   return ret.hasValue();
 }
 
 bool GDBRemoteCommunicationClient::GetStopReply(
     StringExtractorGDBRemote &response) {
-  if (SendPacketAndWaitForResponse("?", response) == PacketResult::Success)
+  if (SendPacketAndWaitForResponse("?", response) == PacketResult::Success) {
+    if (GetNonStopProtocol()) {
+      if (!DrainNotificationQueue("vStopped"))
+        return false;  // TODO: better error handling
+    }
     return response.IsNormalResponse();
+  }
   return false;
 }
 
 bool GDBRemoteCommunicationClient::GetThreadStopInfo(
     lldb::tid_t tid, StringExtractorGDBRemote &response) {
   if (m_supports_qThreadStopInfo) {
     char packet[256];
     int packet_len =

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemote.h

uint64_t m_remote_stub_max_memory_size; // The maximum memory size the remote
                                           // gdb stub can handle
   MMapMap m_addr_to_mmap_size;
   lldb::BreakpointSP m_thread_create_bp_sp;
   bool m_waiting_for_attach;
   lldb::CommandObjectSP m_command_sp;
   int64_t m_breakpoint_pc_offset;
   lldb::tid_t m_initial_tid; // The initial thread ID, given by stub on attach
   bool m_use_g_packet_for_reading;
+  bool m_use_non_stop_protocol;
 
   bool m_allow_flash_writes;
   using FlashRangeVector = lldb_private::RangeVector<lldb::addr_t, size_t>;
   using FlashRange = FlashRangeVector::Entry;
   FlashRangeVector m_erased_flash_ranges;
 
   bool m_vfork_in_progress;
 

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp

return m_collection_sp->GetPropertyAtIndexAsBoolean(
         nullptr, idx,
         g_processgdbremote_properties[idx].default_uint_value != 0);
   }
 
   bool GetUseGPacketForReading() const {
     const uint32_t idx = ePropertyUseGPacketForReading;
     return m_collection_sp->GetPropertyAtIndexAsBoolean(nullptr, idx, true);
   }
+
+  bool GetUseNonStopProtocol() const {
+    const uint32_t idx = ePropertyUseNonStopProtocol;
+    return m_collection_sp->GetPropertyAtIndexAsBoolean(nullptr, idx, true);
+  }
 };
 
 static PluginProperties &GetGlobalPluginProperties() {
   static PluginProperties g_settings;
   return g_settings;
 }
 
 } // namespace
ProcessGDBRemote::ProcessGDBRemote(lldb::TargetSP target_sp,
 
   const uint64_t timeout_seconds =
       GetGlobalPluginProperties().GetPacketTimeout();
   if (timeout_seconds > 0)
     m_gdb_comm.SetPacketTimeout(std::chrono::seconds(timeout_seconds));
 
   m_use_g_packet_for_reading =
       GetGlobalPluginProperties().GetUseGPacketForReading();
+  m_use_non_stop_protocol =
+      GetGlobalPluginProperties().GetUseNonStopProtocol();
 }
 
 // Destructor
 ProcessGDBRemote::~ProcessGDBRemote() {
   //  m_mach_process.UnregisterNotificationCallbacks (this);
   Clear();
   // We need to call finalize on the process before destroying ourselves to
   // make sure all of the broadcaster cleanup goes as planned. If we destruct
Status ProcessGDBRemote::ConnectToDebugserver(llvm::StringRef connect_url) {
 
   m_gdb_comm.GetEchoSupported();
   m_gdb_comm.GetThreadSuffixSupported();
   m_gdb_comm.GetListThreadsInStopReplySupported();
   m_gdb_comm.GetHostInfo();
   m_gdb_comm.GetVContSupported('c');
   m_gdb_comm.GetVAttachOrWaitSupported();
   m_gdb_comm.EnableErrorStringInPacket();
+  if (m_use_non_stop_protocol)
+    m_gdb_comm.EnableNonStop();
 
   // First dispatch any commands from the platform:
   auto handle_cmds = [&] (const Args &args) ->  void {
     for (const Args::ArgEntry &entry : args) {
       StringExtractorGDBRemote response;
       m_gdb_comm.SendPacketAndWaitForResponse(
           entry.c_str(), response);
     }

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemoteProperties.td

 include "../../../../include/lldb/Core/PropertiesBase.td"
 
 let Definition = "processgdbremote" in {
   def PacketTimeout: Property<"packet-timeout", "UInt64">,
     Global,
     DefaultUnsignedValue<5>,
     Desc<"Specify the default packet timeout in seconds.">;
   def TargetDefinitionFile: Property<"target-definition-file", "FileSpec">,
     Global,
     DefaultStringValue<"">,
     Desc<"The file that provides the description for remote target registers.">;
+  def UseNonStopProtocol: Property<"use-non-stop-protocol", "Boolean">,
+    Global,
+    DefaultFalse,
+    Desc<"If true, LLDB will enable non-stop mode on the server and expect non-stop-style notifications from it. Note that LLDB currently requires that all threads are stopped anyway in non-stop mode.">;
   def UseSVR4: Property<"use-libraries-svr4", "Boolean">,
     Global,
     DefaultTrue,
     Desc<"If true, the libraries-svr4 feature will be used to get a hold of the process's loaded modules. This setting is only effective if lldb was build with xml support.">;
   def UseGPacketForReading: Property<"use-g-packet-for-reading", "Boolean">,
     Global,
     DefaultFalse,
     Desc<"Specify if the server should use 'g' packets to read registers.">;
 }

lldb/source/Target/Process.cpp

if (error.Success()) {
     }
   } else
     LLDB_LOGF(log, "Process::PrivateResume() got an error \"%s\".",
               error.AsCString("<unknown error>"));
   return error;
 }
 
 Status Process::Halt(bool clear_thread_plans, bool use_run_lock) {
-  if (!StateIsRunningState(m_public_state.GetValue()))
+  if (!StateIsRunningState(m_private_state.GetValue()))
     return Status("Process is not running.");
 
   // Don't clear the m_clear_thread_plans_on_stop, only set it to true if in
   // case it was already set and some thread plan logic calls halt on its own.
   m_clear_thread_plans_on_stop |= clear_thread_plans;
 
   ListenerSP halt_listener_sp(
       Listener::MakeListener("lldb.process.halt_listener"));

lldb/test/API/functionalities/gdb_remote_client/TestGDBRemoteClient.py

def test_siginfo_freebsd_amd64(self):
             "si_signo": 11,  # SIGSEGV
             "si_errno": 0,
             "si_code": 1,  # SEGV_MAPERR
             "si_addr": 0xfedcba9876,
             "_reason._fault._trapno": 12,
         }
         self.do_siginfo_test("remote-freebsd", "basic_eh_frame.yaml",
                              data, expected)
+
+    def test_QNonStop_query(self):
+        class MyResponder(MockGDBServerResponder):
+            vStopped_counter = 0
+
+            def qSupported(self, client_supported):
+                return "QNonStop+;" + super().qSupported(client_supported)
+
+            def QNonStop(self, val):
+                assert val == 1
+                return "OK"
+
+            def qfThreadInfo(self):
+                return "m10,12"
+
+            def qsThreadInfo(self):
+                return "l"
+
+            def vStopped(self):
+                self.vStopped_counter += 1
+                return ("OK" if self.vStopped_counter % 2 == 0
+                        else "T00;thread:10")
+
+        self.dbg.HandleCommand(
+            "settings set plugin.process.gdb-remote.use-non-stop-protocol true")
+        self.addTearDownHook(lambda:
+            self.runCmd(
+                "settings set plugin.process.gdb-remote.use-non-stop-protocol "
+                "false"))
+        self.server.responder = MyResponder()
+        target = self.dbg.CreateTarget("")
+        process = self.connect(target)
+        self.assertPacketLogContains(["QNonStop:1", "vStopped"])
+
+    def test_QNonStop_run(self):
+        class MyResponder(MockGDBServerResponder):
+            vStopped_counter = 0
+
+            def qSupported(self, client_supported):
+                return "QNonStop+;" + super().qSupported(client_supported)
+
+            def QNonStop(self, val):
+                assert val == 1
+                return "OK"
+
+            def qfThreadInfo(self):
+                return "m10,12"
+
+            def qsThreadInfo(self):
+                return "l"
+
+            def vStopped(self):
+                self.vStopped_counter += 1
+                return ("OK" if self.vStopped_counter > 1
+                        else "T00;thread:10")
+
+            def cont(self):
+                self.vStopped_counter = 0
+                return ["OK", "%Stop:T02;thread:12"]
+
+            def vCtrlC(self):
+                return "OK"
+
+        self.dbg.HandleCommand(
+            "settings set plugin.process.gdb-remote.use-non-stop-protocol true")
+        self.addTearDownHook(lambda:
+            self.runCmd(
+                "settings set plugin.process.gdb-remote.use-non-stop-protocol "
+                "false"))
+        self.server.responder = MyResponder()
+        target = self.dbg.CreateTarget("")
+        process = self.connect(target)
+        self.assertPacketLogContains(["QNonStop:1"])
+
+        process.Continue()
+        self.assertPacketLogContains(["vStopped", "vCtrlC"])
+        self.assertEqual(process.GetSelectedThread().GetStopReason(),
+                         lldb.eStopReasonSignal)
+        self.assertEqual(process.GetSelectedThread().GetStopDescription(100),
+                         "signal SIGINT")
+
+    def test_QNonStop_run_non_stop_server(self):
+        class MyResponder(MockGDBServerResponder):
+            def qSupported(self, client_supported):
+                return "QNonStop+;" + super().qSupported(client_supported)
+
+            def QNonStop(self, val):
+                assert val == 1
+                return "OK"
+
+            def qfThreadInfo(self):
+                return "m10,12"
+
+            def qsThreadInfo(self):
+                return "l"
+
+            def vStopped(self):
+                return "OK"
+
+            def cont(self):
+                return ["OK", "%Stop:T02;thread:12"]
+
+            def vCtrlC(self):
+                return ["OK", "%Stop:T00;thread:10"]
+
+        self.dbg.HandleCommand(
+            "settings set plugin.process.gdb-remote.use-non-stop-protocol true")
+        self.addTearDownHook(lambda:
+            self.runCmd(
+                "settings set plugin.process.gdb-remote.use-non-stop-protocol "
+                "false"))
+        self.server.responder = MyResponder()
+        target = self.dbg.CreateTarget("")
+        process = self.connect(target)
+        self.assertPacketLogContains(["QNonStop:1"])
+
+        process.Continue()
+        self.assertPacketLogContains(["vStopped", "vCtrlC"])
+        self.assertEqual(process.GetSelectedThread().GetStopReason(),
+                         lldb.eStopReasonSignal)
+        self.assertEqual(process.GetSelectedThread().GetStopDescription(100),
+                         "signal SIGINT")
+
+    def test_QNonStop_vCtrlC(self):
+        class MyResponder(MockGDBServerResponder):
+            vStopped_counter = 0
+
+            def qSupported(self, client_supported):
+                return "QNonStop+;" + super().qSupported(client_supported)
+
+            def QNonStop(self, val):
+                assert val == 1
+                return "OK"
+
+            def qfThreadInfo(self):
+                return "m10,12"
+
+            def vStopped(self):
+                self.vStopped_counter += 1
+                return ("OK" if self.vStopped_counter % 2 == 0
+                        else "T00;thread:10")
+
+            def cont(self):
+                return ["OK"]
+
+            def vCtrlC(self):
+                return ["OK", "%Stop:T00;thread:10"]
+
+        self.dbg.HandleCommand(
+            "settings set plugin.process.gdb-remote.use-non-stop-protocol true")
+        self.addTearDownHook(lambda:
+            self.runCmd(
+                "settings set plugin.process.gdb-remote.use-non-stop-protocol "
+                "false"))
+        self.server.responder = MyResponder()
+        target = self.dbg.CreateTarget("")
+        process = self.connect(target)
+        self.assertPacketLogContains(["QNonStop:1"])
+
+        self.dbg.SetAsync(True)
+        process.Continue()
+        process.Stop()
+        self.assertPacketLogContains(["vStopped", "vCtrlC"])
+        self.assertEqual(process.GetSelectedThread().GetStopReason(),
+                         lldb.eStopReasonNone)