[lldb] [llgs] Implement non-stop style stop notification packets

[mgorny]

Summary

Implement the support for %Stop asynchronous notification packet format
in LLGS. This does not implement full support for non-stop mode for
threaded programs -- process plugins continue stopping all threads
on every event. However, it will be used to implement asynchronous
events in multiprocess debugging.

The non-stop protocol is enabled using QNonStop packet. When it is
enabled, the server uses notification protocol instead of regular stop
replies. Since all threads are always stopped, notifications are always
generated for all active threads and copied into stop notification
queue.

If the queue was empty, the initial asynchronous %Stop notification
is sent to the client immediately. The client needs to (eventually)
acknowledge the notification by sending the vStopped packet, in which
case it is popped from the queue and the stop reason for the next thread
is reported. This continues until notification queue is empty again,
in which case an OK reply is sent.

Asychronous notifications are also used for vAttach results and program
exits. The ? packet uses a hybrid approach -- it returns the first
stop reason synchronously, and exposes the stop reasons for remaining
threads via vStopped queue.

The change includes a test case for a program generating a segfault
on 3 threads. The server is expected to generate a stop notification
for the segfaulting thread, along with the notifications for the other
running threads (with "no stop reason"). This verifies that the stop
reasons are correctly reported for all threads, and that notification
queue works.

Sponsored by: The FreeBSD Foundation

[mgorny]

Rebase.

[mgorny]

Fix missing OK response for c/C/vCont/… packets.

[mgorny]

Implemented vCtrlC packet and added a test for it.

[mgorny]

Improve protocol conformance:

• use non-stop notifications for vAttach packets
• clear notification queue and use hybrid approach for ? packets
• ensure that the server doesn't exit before the client gets a chance to ACK the exit notification

[mgorny]

Send OK before the asynchronous stop reason to k packet. Support vCont;t (as a special case) in non-stop mode.

[labath]

I think this is a pretty good start. The main thing bugging me is the two bool flags on the stop reply packet functions. I wonder if it would be more readable to split this into two functions. Something like:

SendStopReplyForThread(bool force_synchronous /*seems better than allow_async, but that could just be me*/, bool enqueue /*enqueues just this thread*/);
EnqueueStopReplyPackets(NativeThreadProtocol &thread_to_skip);

perhaps?

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

3713

Is this correct? I would expect to this to first read (front()) the packet before popping it.

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1414

Could you put these into a new file? TestLldbGdbServer is already one of the longest running tests...

1443

Is this necessary? (I get that it's not useful, but it seems weird...)

1525

Are you sure this is not racy (like, can the inferior terminate before we get a chance to interrupt it)?

[mgorny]

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

3713

Yes, it is. I suppose it could be a bit misleading. The first message is sent as asynchronous notification but we keep it on the queue (and technically, the protocol assumes we may resend the asynchronous notification if we think client may have not gotten it). vStopped means client has processed it and is ready for the next one, so we pop the one that's been sent already and send the next one, wait for the next ACK and so on.

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1443

It's convenient because it lets us compare the matches from threads and threads_verify.

1525

Well, if I'm reading the code right, the new thread waits 60 seconds, so yes, technically it's racy ;-). Do you have a better idea? Some new argument that puts the process in an infinite sleep loop?

[mgorny]

In D125575#3586534, @labath wrote:

I think this is a pretty good start. The main thing bugging me is the two bool flags on the stop reply packet functions. I wonder if it would be more readable to split this into two functions. Something like:

SendStopReplyForThread(bool force_synchronous /*seems better than allow_async, but that could just be me*/, bool enqueue /*enqueues just this thread*/);
EnqueueStopReplyPackets(NativeThreadProtocol &thread_to_skip);

perhaps?

force_synchronous makes sense. I guess moving the loop into another function does as well. However, I think the bool enqueue change is going to be more confusing than helpful (note that the notification is always enqueued through SendNotificationPacketNoLock() unless force_synchronous, so bool enqueue only applies for that case), and I don't really see how to avoid bool queue_all_threads other than repeating the calls to EnqueueStopReplyPackets() all over the place — which ofc is possible but seems like a lot of duplication.

The thing is, we generally have three cases to handle:

1. A function that uses asynchronous replies in non-stop mode — i.e. we always enqueue, and send async notification if one wasn't sent already.
2. A function that always uses synchronous replies — i.e. we send plain old reply and don't queue anything.
3. ? — which in non-stop mode enqueues for all threads but sends the first one synchronous rather than asynchronous.

Perhaps the best approach would be to eliminate bool enqueue entirely and call PrepareStopReplyPacketForThread() from ? handler directly. I need to think about it more.

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1414

Yes, I agree. TestNonStop.py coming once!

[mgorny]

Thinking about it more, ? needs to be special-cased anyway to handle multiprocess, so that's probably the best way forward.

[mgorny]

Move tests to a separate file. Rename and reverse allow_async into force_synchronous. Remove queue_all_threads — we now always queue all in async mode. Rework Handle_stop_reason not to rely on SendStopReasonForState(), and fix reporting exiting processes via ? when not acked by client yet. Add tests for correct exit reporting.

[labath]

Ok, looks good. Some extra suggestions inline.

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

487

no else after return

1565–1576

Could we have a helper function for this, and make all the continue-like actions go through that?

1723

llvm::erase_if(m_stop_notification_queue, []{ return !begins_with_W_or_X;});

3207

add /*force_synchronous=*/ here and elsewhere.

3713

Yeah, it is confusing. Having a comment (with the explanation you just made) would help a lot. However, if you don't see any use case for the resending, we could also delete it, and use the standard queue pattern. Up to you...

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1443

Ok, I suppose I can live with that.

1525

That's ok, a 60s wait is fine. I forgot there was a default sleep action -- I though the process would quit immediately...

[mib]

Hi @mgorny ! TestNonStop.py is also failing on the macOS bots: https://green.lab.llvm.org/green/job/lldb-cmake/44743/

Let me know if you need help reproducing the issue, otherwise skip the test on Darwin systems @skipIfDarwin

Thanks!

[mgorny]

In D125575#3600640, @mib wrote:

Hi @mgorny ! TestNonStop.py is also failing on the macOS bots: https://green.lab.llvm.org/green/job/lldb-cmake/44743/

Let me know if you need help reproducing the issue, otherwise skip the test on Darwin systems @skipIfDarwin

Sorry about that. Actually, I need to mark them as LLGS-specific. Thanks for reporting!

lldb/include/lldb/Utility/StringExtractorGDBRemote.h

enum ServerPacketType {
     eServerPacketType_jLLDBTraceStop,
     eServerPacketType_jLLDBTraceGetState,
     eServerPacketType_jLLDBTraceGetBinaryData,
 
     eServerPacketType_qMemTags, // read memory tags
     eServerPacketType_QMemTags, // write memory tags
 
     eServerPacketType_qLLDBSaveCore,
-    eServerPacketType_QSetIgnoredExceptions
+    eServerPacketType_QSetIgnoredExceptions,
+    eServerPacketType_QNonStop,
+    eServerPacketType_vStopped,
   };
 
   ServerPacketType GetServerPacketType() const;
 
   enum ResponseType { eUnsupported = 0, eAck, eNack, eError, eOK, eResponse };
 
   ResponseType GetResponseType() const;
 

lldb/packages/Python/lldbsuite/test/tools/lldb-server/gdbremote_testcase.py

_KNOWN_QSUPPORTED_STUB_FEATURES = [
         "qEcho",
         "QPassSignals",
         "multiprocess",
         "fork-events",
         "vfork-events",
         "memory-tagging",
         "qSaveCore",
         "native-signals",
+        "QNonStop",
     ]
 
     def parse_qSupported_response(self, context):
         self.assertIsNotNone(context)
 
         raw_response = context.get("qSupported_response")
         self.assertIsNotNone(raw_response)
 

lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py

elif packet_contents[0:1] != b"O":
         return None
     elif packet_contents == b"OK":
         return None
     else:
         return binascii.unhexlify(packet_contents[1:])
 
 class Server(object):
 
-    _GDB_REMOTE_PACKET_REGEX = re.compile(br'^\$([^\#]*)#[0-9a-fA-F]{2}')
+    _GDB_REMOTE_PACKET_REGEX = re.compile(br'^[\$%]([^\#]*)#[0-9a-fA-F]{2}')
 
     class ChecksumMismatch(Exception):
         pass
 
     def __init__(self, sock, proc = None):
         self._accumulated_output = b""
         self._receive_buffer = b""
         self._normal_queue = []

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.h

protected:
   bool m_send_acks;
   bool m_is_platform; // Set to true if this class represents a platform,
                       // false if this class represents a debug session for
                       // a single process
 
   CompressionType m_compression_type;
 
   PacketResult SendPacketNoLock(llvm::StringRef payload);
+  PacketResult SendNotificationPacketNoLock(llvm::StringRef notify_type,
+                                            std::deque<std::string>& queue,
+                                            llvm::StringRef payload);
   PacketResult SendRawPacketNoLock(llvm::StringRef payload,
                                    bool skip_ack = false);
 
   PacketResult ReadPacket(StringExtractorGDBRemote &response,
                           Timeout<std::micro> timeout, bool sync_on_timeout);
 
   PacketResult ReadPacketWithOutputSupport(
       StringExtractorGDBRemote &response, Timeout<std::micro> timeout,

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp

GDBRemoteCommunication::SendPacketNoLock(llvm::StringRef payload) {
   packet.PutChar('#');
   packet.PutHex8(CalculcateChecksum(payload));
   std::string packet_str = std::string(packet.GetString());
 
   return SendRawPacketNoLock(packet_str);
 }
 
 GDBRemoteCommunication::PacketResult
+GDBRemoteCommunication::SendNotificationPacketNoLock(
+    llvm::StringRef notify_type, std::deque<std::string> &queue,
+    llvm::StringRef payload) {
+  PacketResult ret = PacketResult::Success;
+
+  // If there are no notification in the queue, send the notification
+  // packet.
+  if (queue.empty()) {
+    StreamString packet(0, 4, eByteOrderBig);
+    packet.PutChar('%');
+    packet.Write(notify_type.data(), notify_type.size());
+    packet.PutChar(':');
+    packet.Write(payload.data(), payload.size());
+    packet.PutChar('#');
+    packet.PutHex8(CalculcateChecksum(payload));
+    ret = SendRawPacketNoLock(packet.GetString(), true);
+  }
+
+  queue.push_back(payload.str());
+  return ret;
+}
+
+GDBRemoteCommunication::PacketResult
 GDBRemoteCommunication::SendRawPacketNoLock(llvm::StringRef packet,
                                             bool skip_ack) {
   if (IsConnected()) {
     Log *log = GetLog(GDBRLog::Packets);
     ConnectionStatus status = eConnectionStatusSuccess;
     const char *packet_data = packet.data();
     const size_t packet_length = packet.size();
     size_t bytes_written = WriteAll(packet_data, packet_length, status, nullptr);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.h

protected:
 
   lldb::StateType m_inferior_prev_state = lldb::StateType::eStateInvalid;
   llvm::StringMap<std::unique_ptr<llvm::MemoryBuffer>> m_xfer_buffer_map;
   std::mutex m_saved_registers_mutex;
   std::unordered_map<uint32_t, lldb::DataBufferSP> m_saved_registers_map;
   uint32_t m_next_saved_registers_id = 1;
   bool m_thread_suffix_supported = false;
   bool m_list_threads_in_stop_reply = false;
+  bool m_non_stop = false;
+  std::deque<std::string> m_stop_notification_queue;
 
   NativeProcessProtocol::Extension m_extensions_supported = {};
 
   PacketResult SendONotification(const char *buffer, uint32_t len);
 
   PacketResult SendWResponse(NativeProcessProtocol *process);
 
   PacketResult SendStopReplyPacketForThread(lldb::tid_t tid);
 
+  PacketResult SendStopReplyPacketForThread(NativeThreadProtocol &thread,
+                                            bool send_notify = true);
+
   PacketResult SendStopReasonForState(lldb::StateType process_state);
 
   PacketResult Handle_k(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qProcessInfo(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qC(StringExtractorGDBRemote &packet);
 
protected:
   PacketResult Handle_qWatchpointSupportInfo(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qFileLoadAddress(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_QPassSignals(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qSaveCore(StringExtractorGDBRemote &packet);
 
+  PacketResult Handle_QNonStop(StringExtractorGDBRemote &packet);
+
+  PacketResult Handle_vStopped(StringExtractorGDBRemote &packet);
+
   PacketResult Handle_g(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qMemTags(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_QMemTags(StringExtractorGDBRemote &packet);
 
   void SetCurrentThreadID(lldb::tid_t tid);
 

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

RegisterPacketHandler(StringExtractorGDBRemote::eServerPacketType_k,
                                bool &interrupt, bool &quit) {
                           quit = true;
                           return this->Handle_k(packet);
                         });
 
   RegisterMemberFunctionHandler(
       StringExtractorGDBRemote::eServerPacketType_qLLDBSaveCore,
       &GDBRemoteCommunicationServerLLGS::Handle_qSaveCore);
+
+  RegisterMemberFunctionHandler(
+      StringExtractorGDBRemote::eServerPacketType_QNonStop,
+      &GDBRemoteCommunicationServerLLGS::Handle_QNonStop);
+  RegisterMemberFunctionHandler(
+      StringExtractorGDBRemote::eServerPacketType_vStopped,
+      &GDBRemoteCommunicationServerLLGS::Handle_vStopped);
 }
 
 void GDBRemoteCommunicationServerLLGS::SetLaunchInfo(const ProcessLaunchInfo &info) {
   m_process_launch_info = info;
 }
 
 Status GDBRemoteCommunicationServerLLGS::LaunchProcess() {
   Log *log = GetLog(LLDBLog::Process);
GDBRemoteCommunicationServerLLGS::SendWResponse(
   LLDB_LOG(log, "pid = {0}, returning exit type {1}", process->GetID(),
            *wait_status);
 
   StreamGDBRemote response;
   response.Format("{0:g}", *wait_status);
   return SendPacketNoLock(response.GetString());
 }
 
 static void AppendHexValue(StreamString &response, const uint8_t *buf,
                            uint32_t buf_size, bool swap) {
   int64_t i;
   if (swap) {
     for (i = buf_size - 1; i >= 0; i--)
       response.PutHex8(buf[i]);
   } else {
     for (i = 0; i < buf_size; i++)
       response.PutHex8(buf[i]);
for (NativeThreadProtocol *thread;
     threads_array.push_back(std::move(thread_obj));
   }
   return threads_array;
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::SendStopReplyPacketForThread(
     lldb::tid_t tid) {
-  Log *log = GetLog(LLDBLog::Process | LLDBLog::Thread);
-
   // Ensure we have a debugged process.
   if (!m_current_process ||
       (m_current_process->GetID() == LLDB_INVALID_PROCESS_ID))
     return SendErrorResponse(50);
 
-  LLDB_LOG(log, "preparing packet for pid {0} tid {1}",
-           m_current_process->GetID(), tid);
-
   // Ensure we can get info on the given thread.
   NativeThreadProtocol *thread = m_current_process->GetThreadByID(tid);
   if (!thread)
     return SendErrorResponse(51);
 
+  return SendStopReplyPacketForThread(*thread);
+}
+
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::SendStopReplyPacketForThread(
+    NativeThreadProtocol &thread, bool send_notify) {
+  Log *log = GetLog(LLDBLog::Process | LLDBLog::Thread);
+
+  LLDB_LOG(log, "preparing packet for pid {0} tid {1}",
+           m_current_process->GetID(), thread.GetID());
+
   // Grab the reason this thread stopped.
   struct ThreadStopInfo tid_stop_info;
   std::string description;
-  if (!thread->GetStopReason(tid_stop_info, description))
+  if (!thread.GetStopReason(tid_stop_info, description))
     return SendErrorResponse(52);
 
   // FIXME implement register handling for exec'd inferiors.
   // if (tid_stop_info.reason == eStopReasonExec) {
   //     const bool force = true;
   //     InitializeRegisters(force);
   // }
 
   StreamString response;
   // Output the T packet with the thread
   response.PutChar('T');
   int signum = tid_stop_info.details.signal.signo;
   LLDB_LOG(
       log,
       "pid {0}, tid {1}, got signal signo = {2}, reason = {3}, exc_type = {4}",
-      m_current_process->GetID(), tid, signum, int(tid_stop_info.reason),
-      tid_stop_info.details.exception.type);
+      m_current_process->GetID(), thread.GetID(), signum,
+      int(tid_stop_info.reason), tid_stop_info.details.exception.type);
 
   // Print the signal number.
   response.PutHex8(signum & 0xff);
 
   // Include the tid.
-  response.Printf("thread:%" PRIx64 ";", tid);
+  response.Printf("thread:%" PRIx64 ";", thread.GetID());
 
   // Include the thread name if there is one.
-  const std::string thread_name = thread->GetName();
+  const std::string thread_name = thread.GetName();
   if (!thread_name.empty()) {
     size_t thread_name_len = thread_name.length();
 
     if (::strcspn(thread_name.c_str(), "$#+-;:") == thread_name_len) {
       response.PutCString("name:");
       response.PutCString(thread_name);
     } else {
       // The thread name contains special chars, send as hex bytes.
if (thread_index > 1) {
       }
     }
 
     uint32_t i = 0;
     response.PutCString("thread-pcs");
     char delimiter = ':';
     for (NativeThreadProtocol *thread;
          (thread = m_current_process->GetThreadAtIndex(i)) != nullptr; ++i) {
-      NativeRegisterContext& reg_ctx = thread->GetRegisterContext();
+      NativeRegisterContext &reg_ctx = thread->GetRegisterContext();
 
       uint32_t reg_to_read = reg_ctx.ConvertRegisterKindToRegisterNumber(
           eRegisterKindGeneric, LLDB_REGNUM_GENERIC_PC);
       const RegisterInfo *const reg_info_p =
           reg_ctx.GetRegisterInfoAtIndex(reg_to_read);
 
       RegisterValue reg_value;
       Status error = reg_ctx.ReadRegister(reg_info_p, reg_value);
if (m_list_threads_in_stop_reply) {
     response.PutChar(';');
   }
 
   //
   // Expedite registers.
   //
 
   // Grab the register context.
-  NativeRegisterContext& reg_ctx = thread->GetRegisterContext();
+  NativeRegisterContext &reg_ctx = thread.GetRegisterContext();
   const auto expedited_regs =
       reg_ctx.GetExpeditedRegisters(ExpeditedRegs::Full);
 
   for (auto &reg_num : expedited_regs) {
     const RegisterInfo *const reg_info_p =
         reg_ctx.GetRegisterInfoAtIndex(reg_num);
     // Only expediate registers that are not contained in other registers.
     if (reg_info_p != nullptr && reg_info_p->value_regs == nullptr) {
       RegisterValue reg_value;
       Status error = reg_ctx.ReadRegister(reg_info_p, reg_value);
       if (error.Success()) {
         response.Printf("%.02x:", reg_num);
         WriteRegisterValueInHexFixedWidth(response, reg_ctx, *reg_info_p,
                                           &reg_value, lldb::eByteOrderBig);
         response.PutChar(';');
       } else {
-        LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s failed to read "
-                       "register '%s' index %" PRIu32 ": %s",
+        LLDB_LOGF(log,
+                  "GDBRemoteCommunicationServerLLGS::%s failed to read "
                   __FUNCTION__,
                   reg_info_p->name ? reg_info_p->name : "<unnamed-register>",
                   reg_num, error.AsCString());
       }
     }
   }
 
   const char *reason_str = GetStopReasonString(tid_stop_info.reason);
if (tid_stop_info.reason == eStopReasonFork ||
     if (tid_stop_info.reason == eStopReasonVFork)
       assert(bool(m_extensions_supported &
                   NativeProcessProtocol::Extension::vfork));
     response.Printf("%s:p%" PRIx64 ".%" PRIx64 ";", reason_str,
                     tid_stop_info.details.fork.child_pid,
                     tid_stop_info.details.fork.child_tid);
   }
 
+  if (m_non_stop) {
+    PacketResult ret = SendNotificationPacketNoLock(
+        "Stop", m_stop_notification_queue, response.GetString());
+    if (send_notify) {
+      // Queue notification events for the remaining threads.
+      uint32_t thread_index = 0;
+      while (NativeThreadProtocol *listed_thread =
+                 m_current_process->GetThreadAtIndex(thread_index++)) {
+        if (listed_thread->GetID() != thread.GetID())
+          SendStopReplyPacketForThread(*listed_thread, false);
+      }
+    }
+    return ret;
+  } else
     return SendPacketNoLock(response.GetString());
 }
 
 void GDBRemoteCommunicationServerLLGS::HandleInferiorState_Exited(
     NativeProcessProtocol *process) {
   assert(process && "process cannot be NULL");
 
   Log *log = GetLog(LLDBLog::Process);
   LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s called", __FUNCTION__);
GDBRemoteCommunicationServerLLGS::Handle_C(StringExtractorGDBRemote &packet) {
   error = m_continue_process->Resume(resume_actions);
   if (error.Fail()) {
     LLDB_LOG(log, "failed to resume threads for process {0}: {1}",
              m_continue_process->GetID(), error);
 
     return SendErrorResponse(0x38);
   }
 
-  // Don't send an "OK" packet; response is the stopped/exited message.
-  return PacketResult::Success;
+  // Don't send an "OK" packet, except in non-stop mode;
+  // otherwise, the response is the stopped/exited message.
+  return m_non_stop ? SendOKResponse() : PacketResult::Success;
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_c(StringExtractorGDBRemote &packet) {
   Log *log = GetLog(LLDBLog::Process | LLDBLog::Thread);
   LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s called", __FUNCTION__);
 
   packet.SetFilePos(packet.GetFilePos() + ::strlen("c"));
LLDB_LOGF(log,
               __FUNCTION__);
     return SendErrorResponse(0x36);
   }
 
   // Build the ResumeActionList
   ResumeActionList actions(StateType::eStateRunning,
                            LLDB_INVALID_SIGNAL_NUMBER);
 
   Status error = m_continue_process->Resume(actions);
   if (error.Fail()) {
     LLDB_LOG(log, "c failed for process {0}: {1}", m_continue_process->GetID(),
              error);
     return SendErrorResponse(GDBRemoteServerError::eErrorResume);
   }
 
   LLDB_LOG(log, "continued process {0}", m_continue_process->GetID());
-  // No response required from continue.
-  return PacketResult::Success;
+
+  // No response required from continue, unless in non-stop mode.
+  return m_non_stop ? SendOKResponse() : PacketResult::Success;
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_vCont_actions(
     StringExtractorGDBRemote &packet) {
   StreamString response;
   response.Printf("vCont;c;C;s;S");
 
   return SendPacketNoLock(response.GetString());
GDBRemoteCommunicationServerLLGS::Handle_vCont(
   Status error = m_continue_process->Resume(thread_actions);
   if (error.Fail()) {
     LLDB_LOG(log, "vCont failed for process {0}: {1}",
              m_continue_process->GetID(), error);
     return SendErrorResponse(GDBRemoteServerError::eErrorResume);
   }
 
   LLDB_LOG(log, "continued process {0}", m_continue_process->GetID());
-  // No response required from vCont.
-  return PacketResult::Success;
+
+  // No response required from vCont, unless in non-stop mode.
+  return m_non_stop ? SendOKResponse() : PacketResult::Success;
 }
 
 void GDBRemoteCommunicationServerLLGS::SetCurrentThreadID(lldb::tid_t tid) {
   Log *log = GetLog(LLDBLog::Thread);
   LLDB_LOG(log, "setting current thread id to {0}", tid);
 
   m_current_tid = tid;
   if (m_current_process)
GDBRemoteCommunicationServerLLGS::Handle_stop_reason(
 
   // If no process, indicate error
   if (!m_current_process)
     return SendErrorResponse(02);
 
   return SendStopReasonForState(m_current_process->GetState());
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::SendStopReasonForState(
     lldb::StateType process_state) {
   Log *log = GetLog(LLDBLog::Process);
 
   switch (process_state) {
   case eStateAttaching:
   case eStateLaunching:
   case eStateRunning:
if (error.Fail()) {
     LLDB_LOGF(log,
               "GDBRemoteCommunicationServerLLGS::%s pid %" PRIu64
               " tid %" PRIu64 " Resume() failed with error: %s",
               __FUNCTION__, m_continue_process->GetID(), tid,
               error.AsCString());
     return SendErrorResponse(0x49);
   }
 
-  // No response here - the stop or exit will come from the resulting action.
-  return PacketResult::Success;
+  // No response here, unless in non-stop mode.
+  // Otherwise, the stop or exit will come from the resulting action.
+  return m_non_stop ? SendOKResponse() : PacketResult::Success;
 }
 
 llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
 GDBRemoteCommunicationServerLLGS::BuildTargetXml() {
   // Ensure we have a thread.
   NativeThreadProtocol *thread = m_current_process->GetThreadAtIndex(0);
   if (!thread)
     return llvm::createStringError(llvm::inconvertibleErrorCode(),
if (error.Fail()) {
     LLDB_LOGF(log,
               "GDBRemoteCommunicationServerLLGS::%s failed to attach to "
               "pid %" PRIu64 ": %s\n",
               __FUNCTION__, pid, error.AsCString());
     return SendErrorResponse(error);
   }
 
   // Notify we attached by sending a stop packet.
   return SendStopReasonForState(m_current_process->GetState());
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_vAttachWait(
     StringExtractorGDBRemote &packet) {
   Log *log = GetLog(LLDBLog::Process);
 
   // Consume the ';' after the identifier.
if (!ret)
     return SendErrorResponse(ret.takeError());
 
   StreamString response;
   response.PutCString("core-path:");
   response.PutStringAsRawHex8(ret.get());
   return SendPacketNoLock(response.GetString());
 }
 
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::Handle_QNonStop(
+    StringExtractorGDBRemote &packet) {
+  StringRef packet_str{packet.GetStringRef()};
+  assert(packet_str.startswith("QNonStop:"));
+  packet_str.consume_front("QNonStop:");
+  if (packet_str == "0") {
+    m_non_stop = false;
+    // TODO: stop all threads
+  } else if (packet_str == "1") {
+    m_non_stop = true;
+  } else
+    return SendErrorResponse(Status("Invalid QNonStop packet"));
+  return SendOKResponse();
+}
+
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::Handle_vStopped(
+    StringExtractorGDBRemote &packet) {
+  if (m_stop_notification_queue.empty())
+    return SendErrorResponse(Status("No pending notification to ack"));
+  m_stop_notification_queue.pop_front();
+  if (!m_stop_notification_queue.empty())
+    return SendPacketNoLock(m_stop_notification_queue.front());
+  return SendOKResponse();
+}
+
 void GDBRemoteCommunicationServerLLGS::MaybeCloseInferiorTerminalConnection() {
   Log *log = GetLog(LLDBLog::Process);
 
   // Tell the stdio connection to shut down.
   if (m_stdio_communication.IsConnected()) {
     auto connection = m_stdio_communication.GetConnection();
     if (connection) {
       Status error;

 std::vector<std::string> GDBRemoteCommunicationServerLLGS::HandleFeatures(
     const llvm::ArrayRef<llvm::StringRef> client_features) {
   std::vector<std::string> ret =
       GDBRemoteCommunicationServerCommon::HandleFeatures(client_features);
   ret.insert(ret.end(), {
                             "QThreadSuffixSupported+",
                             "QListThreadsInStopReply+",
                             "qXfer:features:read+",
+                            "QNonStop+",
                         });
 
   // report server-only features
   using Extension = NativeProcessProtocol::Extension;
   Extension plugin_features = m_process_factory.GetSupportedExtensions();
   if (bool(plugin_features & Extension::pass_signals))
     ret.push_back("QPassSignals+");
   if (bool(plugin_features & Extension::auxv))

lldb/source/Utility/StringExtractorGDBRemote.cpp

case 'L':
         return eServerPacketType_QListThreadsInStopReply;
       break;
 
     case 'M':
       if (PACKET_STARTS_WITH("QMemTags"))
         return eServerPacketType_QMemTags;
       break;
 
+    case 'N':
+      if (PACKET_STARTS_WITH("QNonStop:"))
+        return eServerPacketType_QNonStop;
+      break;
+
     case 'R':
       if (PACKET_STARTS_WITH("QRestoreRegisterState:"))
         return eServerPacketType_QRestoreRegisterState;
       break;
 
     case 'T':
       if (PACKET_MATCHES("QThreadSuffixSupported"))
         return eServerPacketType_QThreadSuffixSupported;
if (PACKET_STARTS_WITH("vFile:")) {
       if (PACKET_STARTS_WITH("vAttachName;"))
         return eServerPacketType_vAttachName;
       if (PACKET_STARTS_WITH("vCont;"))
         return eServerPacketType_vCont;
       if (PACKET_MATCHES("vCont?"))
         return eServerPacketType_vCont_actions;
       if (PACKET_STARTS_WITH("vRun;"))
         return eServerPacketType_vRun;
+      if (PACKET_MATCHES("vStopped"))
+        return eServerPacketType_vStopped;
+      break;
+
     }
     break;
   case '_':
     switch (packet_cstr[1]) {
     case 'M':
       return eServerPacketType__M;
 
     case 'm':

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

def test_qXfer_siginfo_read(self):
 
         decoder = struct.Struct(format_str)
         decoded = decoder.unpack(content[:decoder.size])
         self.assertEqual(decoded[signo_idx],
                          lldbutil.get_signal_number('SIGSEGV'))
         self.assertEqual(decoded[errno_idx], 0)  # si_errno
         self.assertEqual(decoded[code_idx], SEGV_MAPERR)  # si_code
         self.assertEqual(decoded[addr_idx], 0)  # si_addr
+
+    def test_QNonStop(self):
+        self.build()
+        self.set_inferior_startup_launch()
+        thread_num = 3
+        procs = self.prep_debug_monitor_and_inferior(
+                inferior_args=["thread:segfault"] + thread_num * ["thread:new"])
+        self.test_sequence.add_log_lines(
+            ["read packet: $QNonStop:1#00",
+             "send packet: $OK#00",
+             "read packet: $c#63",
+             "send packet: $OK#00",
+             ], True)
+        self.expect_gdbremote_sequence()
+
+        segv_signo = lldbutil.get_signal_number('SIGSEGV')
+        all_threads = set()
+        all_segv_threads = []
+
+        # we should get segfaults from all the threads
+        for segv_no in range(thread_num):
+            # first wait for the notification event
+            self.reset_test_sequence()
+            self.test_sequence.add_log_lines(
+                [{"direction": "send",
+                  "regex": r"^%Stop:T([0-9a-fA-F]{2})thread:([0-9a-fA-F]+);",
+                  "capture": {1: "signo", 2: "thread_id"},
+                  },
+                 ], True)
+            threads = [self.expect_gdbremote_sequence()]
+
+            # then we may get events for the remaining threads
+            # (but note that not all threads may have been started yet)
+            while True:
+                self.reset_test_sequence()
+                self.test_sequence.add_log_lines(
+                    ["read packet: $vStopped#00",
+                     {"direction": "send",
+                      "regex": r"^\$(OK|T([0-9a-fA-F]{2})thread:([0-9a-fA-F]+);)",
+                      "capture": {1: "packet", 2: "signo", 3: "thread_id"},
+                      },
+                     ], True)
+                m = self.expect_gdbremote_sequence()
+                if m["packet"] == "OK":
+                    break
+                threads.append(m)
+
+            segv_threads = []
+            other_threads = []
+            for t in threads:
+                signo = int(t["signo"], 16)
+                if signo == segv_signo:
+                    segv_threads.append(t["thread_id"])
+                else:
+                    self.assertEqual(signo, 0)
+                    other_threads.append(t["thread_id"])
+
+            # verify that exactly one thread segfaulted
+            self.assertEqual(len(segv_threads), 1)
+            # we should get only one segv from every thread
+            self.assertNotIn(segv_threads[0], all_segv_threads)
+            all_segv_threads.extend(segv_threads)
+            # segv_threads + other_threads should always be a superset
+            # of all_threads, i.e. we should get states for all threads
+            # already started
+            self.assertFalse(
+                    all_threads.difference(other_threads + segv_threads))
+            all_threads.update(other_threads + segv_threads)
+
+            self.reset_test_sequence()
+            self.test_sequence.add_log_lines(
+                ["read packet: $vCont;C{:02x}:{};c#00"
+                 .format(segv_signo, segv_threads[0]),
+                 "send packet: $OK#00",
+                 ], True)
+            self.expect_gdbremote_sequence()
+
+        # finally, verify that all threads have started
+        self.assertEqual(len(all_threads), thread_num + 1)