[lldb] [llgs] Implement non-stop style stop notification packets

[mgorny]

Summary

Implement the support for %Stop asynchronous notification packet format
in LLGS. This does not implement full support for non-stop mode for
threaded programs -- process plugins continue stopping all threads
on every event. However, it will be used to implement asynchronous
events in multiprocess debugging.

The non-stop protocol is enabled using QNonStop packet. When it is
enabled, the server uses notification protocol instead of regular stop
replies. Since all threads are always stopped, notifications are always
generated for all active threads and copied into stop notification
queue.

If the queue was empty, the initial asynchronous %Stop notification
is sent to the client immediately. The client needs to (eventually)
acknowledge the notification by sending the vStopped packet, in which
case it is popped from the queue and the stop reason for the next thread
is reported. This continues until notification queue is empty again,
in which case an OK reply is sent.

Asychronous notifications are also used for vAttach results and program
exits. The ? packet uses a hybrid approach -- it returns the first
stop reason synchronously, and exposes the stop reasons for remaining
threads via vStopped queue.

The change includes a test case for a program generating a segfault
on 3 threads. The server is expected to generate a stop notification
for the segfaulting thread, along with the notifications for the other
running threads (with "no stop reason"). This verifies that the stop
reasons are correctly reported for all threads, and that notification
queue works.

Sponsored by: The FreeBSD Foundation

[mgorny]

Rebase.

[mgorny]

Fix missing OK response for c/C/vCont/… packets.

[mgorny]

Implemented vCtrlC packet and added a test for it.

[mgorny]

Improve protocol conformance:

• use non-stop notifications for vAttach packets
• clear notification queue and use hybrid approach for ? packets
• ensure that the server doesn't exit before the client gets a chance to ACK the exit notification

[mgorny]

Send OK before the asynchronous stop reason to k packet. Support vCont;t (as a special case) in non-stop mode.

[labath]

I think this is a pretty good start. The main thing bugging me is the two bool flags on the stop reply packet functions. I wonder if it would be more readable to split this into two functions. Something like:

SendStopReplyForThread(bool force_synchronous /*seems better than allow_async, but that could just be me*/, bool enqueue /*enqueues just this thread*/);
EnqueueStopReplyPackets(NativeThreadProtocol &thread_to_skip);

perhaps?

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

3786

Is this correct? I would expect to this to first read (front()) the packet before popping it.

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1414 ↗

(On Diff #434006)

Could you put these into a new file? TestLldbGdbServer is already one of the longest running tests...

1443 ↗

(On Diff #434006)

Is this necessary? (I get that it's not useful, but it seems weird...)

1525 ↗

(On Diff #434006)

Are you sure this is not racy (like, can the inferior terminate before we get a chance to interrupt it)?

[mgorny]

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

3786

Yes, it is. I suppose it could be a bit misleading. The first message is sent as asynchronous notification but we keep it on the queue (and technically, the protocol assumes we may resend the asynchronous notification if we think client may have not gotten it). vStopped means client has processed it and is ready for the next one, so we pop the one that's been sent already and send the next one, wait for the next ACK and so on.

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1443 ↗

(On Diff #434006)

It's convenient because it lets us compare the matches from threads and threads_verify.

1525 ↗

(On Diff #434006)

Well, if I'm reading the code right, the new thread waits 60 seconds, so yes, technically it's racy ;-). Do you have a better idea? Some new argument that puts the process in an infinite sleep loop?

[mgorny]

In D125575#3586534, @labath wrote:

I think this is a pretty good start. The main thing bugging me is the two bool flags on the stop reply packet functions. I wonder if it would be more readable to split this into two functions. Something like:

SendStopReplyForThread(bool force_synchronous /*seems better than allow_async, but that could just be me*/, bool enqueue /*enqueues just this thread*/);
EnqueueStopReplyPackets(NativeThreadProtocol &thread_to_skip);

perhaps?

force_synchronous makes sense. I guess moving the loop into another function does as well. However, I think the bool enqueue change is going to be more confusing than helpful (note that the notification is always enqueued through SendNotificationPacketNoLock() unless force_synchronous, so bool enqueue only applies for that case), and I don't really see how to avoid bool queue_all_threads other than repeating the calls to EnqueueStopReplyPackets() all over the place — which ofc is possible but seems like a lot of duplication.

The thing is, we generally have three cases to handle:

1. A function that uses asynchronous replies in non-stop mode — i.e. we always enqueue, and send async notification if one wasn't sent already.
2. A function that always uses synchronous replies — i.e. we send plain old reply and don't queue anything.
3. ? — which in non-stop mode enqueues for all threads but sends the first one synchronous rather than asynchronous.

Perhaps the best approach would be to eliminate bool enqueue entirely and call PrepareStopReplyPacketForThread() from ? handler directly. I need to think about it more.

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1414 ↗

(On Diff #434006)

Yes, I agree. TestNonStop.py coming once!

[mgorny]

Thinking about it more, ? needs to be special-cased anyway to handle multiprocess, so that's probably the best way forward.

[mgorny]

Move tests to a separate file. Rename and reverse allow_async into force_synchronous. Remove queue_all_threads — we now always queue all in async mode. Rework Handle_stop_reason not to rely on SendStopReasonForState(), and fix reporting exiting processes via ? when not acked by client yet. Add tests for correct exit reporting.

[labath]

Ok, looks good. Some extra suggestions inline.

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

495

no else after return

1594–1604

Could we have a helper function for this, and make all the continue-like actions go through that?

1776

llvm::erase_if(m_stop_notification_queue, []{ return !begins_with_W_or_X;});

3262–3265

add /*force_synchronous=*/ here and elsewhere.

3786

Yeah, it is confusing. Having a comment (with the explanation you just made) would help a lot. However, if you don't see any use case for the resending, we could also delete it, and use the standard queue pattern. Up to you...

lldb/test/API/tools/lldb-server/TestLldbGdbServer.py

1443 ↗

(On Diff #434006)

Ok, I suppose I can live with that.

1525 ↗

(On Diff #434006)

That's ok, a 60s wait is fine. I forgot there was a default sleep action -- I though the process would quit immediately...

[mib]

Hi @mgorny ! TestNonStop.py is also failing on the macOS bots: https://green.lab.llvm.org/green/job/lldb-cmake/44743/

Let me know if you need help reproducing the issue, otherwise skip the test on Darwin systems @skipIfDarwin

Thanks!

[mgorny]

In D125575#3600640, @mib wrote:

Hi @mgorny ! TestNonStop.py is also failing on the macOS bots: https://green.lab.llvm.org/green/job/lldb-cmake/44743/

Let me know if you need help reproducing the issue, otherwise skip the test on Darwin systems @skipIfDarwin

Sorry about that. Actually, I need to mark them as LLGS-specific. Thanks for reporting!

lldb/include/lldb/Utility/StringExtractorGDBRemote.h

enum ServerPacketType {
     eServerPacketType_jLLDBTraceStop,
     eServerPacketType_jLLDBTraceGetState,
     eServerPacketType_jLLDBTraceGetBinaryData,
 
     eServerPacketType_qMemTags, // read memory tags
     eServerPacketType_QMemTags, // write memory tags
 
     eServerPacketType_qLLDBSaveCore,
-    eServerPacketType_QSetIgnoredExceptions
+    eServerPacketType_QSetIgnoredExceptions,
+    eServerPacketType_QNonStop,
+    eServerPacketType_vStopped,
+    eServerPacketType_vCtrlC,
   };
 
   ServerPacketType GetServerPacketType() const;
 
   enum ResponseType { eUnsupported = 0, eAck, eNack, eError, eOK, eResponse };
 
   ResponseType GetResponseType() const;
 

lldb/packages/Python/lldbsuite/test/tools/lldb-server/gdbremote_testcase.py

_KNOWN_QSUPPORTED_STUB_FEATURES = [
         "qEcho",
         "QPassSignals",
         "multiprocess",
         "fork-events",
         "vfork-events",
         "memory-tagging",
         "qSaveCore",
         "native-signals",
+        "QNonStop",
     ]
 
     def parse_qSupported_response(self, context):
         self.assertIsNotNone(context)
 
         raw_response = context.get("qSupported_response")
         self.assertIsNotNone(raw_response)
 

lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py

elif packet_contents[0:1] != b"O":
         return None
     elif packet_contents == b"OK":
         return None
     else:
         return binascii.unhexlify(packet_contents[1:])
 
 class Server(object):
 
-    _GDB_REMOTE_PACKET_REGEX = re.compile(br'^\$([^\#]*)#[0-9a-fA-F]{2}')
+    _GDB_REMOTE_PACKET_REGEX = re.compile(br'^[\$%]([^\#]*)#[0-9a-fA-F]{2}')
 
     class ChecksumMismatch(Exception):
         pass
 
     def __init__(self, sock, proc = None):
         self._accumulated_output = b""
         self._receive_buffer = b""
         self._normal_queue = []

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.h

protected:
   bool m_send_acks;
   bool m_is_platform; // Set to true if this class represents a platform,
                       // false if this class represents a debug session for
                       // a single process
 
   CompressionType m_compression_type;
 
   PacketResult SendPacketNoLock(llvm::StringRef payload);
+  PacketResult SendNotificationPacketNoLock(llvm::StringRef notify_type,
+                                            std::deque<std::string>& queue,
+                                            llvm::StringRef payload);
   PacketResult SendRawPacketNoLock(llvm::StringRef payload,
                                    bool skip_ack = false);
 
   PacketResult ReadPacket(StringExtractorGDBRemote &response,
                           Timeout<std::micro> timeout, bool sync_on_timeout);
 
   PacketResult ReadPacketWithOutputSupport(
       StringExtractorGDBRemote &response, Timeout<std::micro> timeout,

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp

GDBRemoteCommunication::SendPacketNoLock(llvm::StringRef payload) {
   packet.PutChar('#');
   packet.PutHex8(CalculcateChecksum(payload));
   std::string packet_str = std::string(packet.GetString());
 
   return SendRawPacketNoLock(packet_str);
 }
 
 GDBRemoteCommunication::PacketResult
+GDBRemoteCommunication::SendNotificationPacketNoLock(
+    llvm::StringRef notify_type, std::deque<std::string> &queue,
+    llvm::StringRef payload) {
+  PacketResult ret = PacketResult::Success;
+
+  // If there are no notification in the queue, send the notification
+  // packet.
+  if (queue.empty()) {
+    StreamString packet(0, 4, eByteOrderBig);
+    packet.PutChar('%');
+    packet.Write(notify_type.data(), notify_type.size());
+    packet.PutChar(':');
+    packet.Write(payload.data(), payload.size());
+    packet.PutChar('#');
+    packet.PutHex8(CalculcateChecksum(payload));
+    ret = SendRawPacketNoLock(packet.GetString(), true);
+  }
+
+  queue.push_back(payload.str());
+  return ret;
+}
+
+GDBRemoteCommunication::PacketResult
 GDBRemoteCommunication::SendRawPacketNoLock(llvm::StringRef packet,
                                             bool skip_ack) {
   if (IsConnected()) {
     Log *log = GetLog(GDBRLog::Packets);
     ConnectionStatus status = eConnectionStatusSuccess;
     const char *packet_data = packet.data();
     const size_t packet_length = packet.size();
     size_t bytes_written = WriteAll(packet_data, packet_length, status, nullptr);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.h

protected:
 
   lldb::StateType m_inferior_prev_state = lldb::StateType::eStateInvalid;
   llvm::StringMap<std::unique_ptr<llvm::MemoryBuffer>> m_xfer_buffer_map;
   std::mutex m_saved_registers_mutex;
   std::unordered_map<uint32_t, lldb::DataBufferSP> m_saved_registers_map;
   uint32_t m_next_saved_registers_id = 1;
   bool m_thread_suffix_supported = false;
   bool m_list_threads_in_stop_reply = false;
+  bool m_non_stop = false;
+  std::deque<std::string> m_stop_notification_queue;
 
   NativeProcessProtocol::Extension m_extensions_supported = {};
 
   PacketResult SendONotification(const char *buffer, uint32_t len);
 
   PacketResult SendWResponse(NativeProcessProtocol *process);
 
+  StreamString PrepareStopReplyPacketForThread(NativeThreadProtocol &thread);
+
   PacketResult SendStopReplyPacketForThread(NativeProcessProtocol &process,
-                                            lldb::tid_t tid);
+                                            lldb::tid_t tid,
+                                            bool force_synchronous);
 
   PacketResult SendStopReasonForState(NativeProcessProtocol &process,
-                                      lldb::StateType process_state);
+                                      lldb::StateType process_state,
+                                      bool force_synchronous);
+
+  void EnqueueStopReplyPackets(lldb::tid_t thread_to_skip);
 
   PacketResult Handle_k(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qProcessInfo(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qC(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_QSetDisableASLR(StringExtractorGDBRemote &packet);
protected:
   PacketResult Handle_qWatchpointSupportInfo(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qFileLoadAddress(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_QPassSignals(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qSaveCore(StringExtractorGDBRemote &packet);
 
+  PacketResult Handle_QNonStop(StringExtractorGDBRemote &packet);
+
+  PacketResult Handle_vStopped(StringExtractorGDBRemote &packet);
+
+  PacketResult Handle_vCtrlC(StringExtractorGDBRemote &packet);
+
   PacketResult Handle_g(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_qMemTags(StringExtractorGDBRemote &packet);
 
   PacketResult Handle_QMemTags(StringExtractorGDBRemote &packet);
 
   void SetCurrentThreadID(lldb::tid_t tid);
 
protected:
   llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
   ReadXferObject(llvm::StringRef object, llvm::StringRef annex);
 
   static std::string XMLEncodeAttributeValue(llvm::StringRef value);
 
   std::vector<std::string> HandleFeatures(
       const llvm::ArrayRef<llvm::StringRef> client_features) override;
 
+  // Provide a response for successful continue action, i.e. send "OK"
+  // in non-stop mode, no response otherwise.
+  PacketResult SendContinueSuccessResponse();
+
 private:
   llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>> BuildTargetXml();
 
   void HandleInferiorState_Exited(NativeProcessProtocol *process);
 
   void HandleInferiorState_Stopped(NativeProcessProtocol *process);
 
   NativeThreadProtocol *GetThreadFromSuffix(StringExtractorGDBRemote &packet);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServerLLGS.cpp

RegisterPacketHandler(StringExtractorGDBRemote::eServerPacketType_k,
                                bool &interrupt, bool &quit) {
                           quit = true;
                           return this->Handle_k(packet);
                         });
 
   RegisterMemberFunctionHandler(
       StringExtractorGDBRemote::eServerPacketType_qLLDBSaveCore,
       &GDBRemoteCommunicationServerLLGS::Handle_qSaveCore);
+
+  RegisterMemberFunctionHandler(
+      StringExtractorGDBRemote::eServerPacketType_QNonStop,
+      &GDBRemoteCommunicationServerLLGS::Handle_QNonStop);
+  RegisterMemberFunctionHandler(
+      StringExtractorGDBRemote::eServerPacketType_vStopped,
+      &GDBRemoteCommunicationServerLLGS::Handle_vStopped);
+  RegisterMemberFunctionHandler(
+      StringExtractorGDBRemote::eServerPacketType_vCtrlC,
+      &GDBRemoteCommunicationServerLLGS::Handle_vCtrlC);
 }
 
 void GDBRemoteCommunicationServerLLGS::SetLaunchInfo(const ProcessLaunchInfo &info) {
   m_process_launch_info = info;
 }
 
 Status GDBRemoteCommunicationServerLLGS::LaunchProcess() {
   Log *log = GetLog(LLDBLog::Process);
GDBRemoteCommunicationServerLLGS::SendWResponse(
 
   LLDB_LOG(log, "pid = {0}, returning exit type {1}", process->GetID(),
            *wait_status);
 
   StreamGDBRemote response;
   response.Format("{0:g}", *wait_status);
   if (bool(m_extensions_supported & NativeProcessProtocol::Extension::multiprocess))
     response.Format(";process:{0:x-}", process->GetID());
+  if (m_non_stop)
+    return SendNotificationPacketNoLock("Stop", m_stop_notification_queue,
+                                        response.GetString());
   return SendPacketNoLock(response.GetString());
 }
 
 static void AppendHexValue(StreamString &response, const uint8_t *buf,
                            uint32_t buf_size, bool swap) {
   int64_t i;
   if (swap) {
     for (i = buf_size - 1; i >= 0; i--)
       response.PutHex8(buf[i]);
   } else {
     for (i = 0; i < buf_size; i++)
       response.PutHex8(buf[i]);
if ((tid_stop_info.reason == eStopReasonException) &&
       }
       thread_obj.try_emplace("medata", std::move(medata_array));
     }
     threads_array.push_back(std::move(thread_obj));
   }
   return threads_array;
 }
 
-GDBRemoteCommunication::PacketResult
-GDBRemoteCommunicationServerLLGS::SendStopReplyPacketForThread(
-    NativeProcessProtocol &process, lldb::tid_t tid) {
+StreamString
+GDBRemoteCommunicationServerLLGS::PrepareStopReplyPacketForThread(
+    NativeThreadProtocol &thread) {
   Log *log = GetLog(LLDBLog::Process | LLDBLog::Thread);
 
-  LLDB_LOG(log, "preparing packet for pid {0} tid {1}", process.GetID(), tid);
+  NativeProcessProtocol &process = thread.GetProcess();
 
-  // Ensure we can get info on the given thread.
-  NativeThreadProtocol *thread = process.GetThreadByID(tid);
-  if (!thread)
-    return SendErrorResponse(51);
+  LLDB_LOG(log, "preparing packet for pid {0} tid {1}", process.GetID(),
+           thread.GetID());
 
   // Grab the reason this thread stopped.
+  StreamString response;
   struct ThreadStopInfo tid_stop_info;
   std::string description;
-  if (!thread->GetStopReason(tid_stop_info, description))
-    return SendErrorResponse(52);
+  if (!thread.GetStopReason(tid_stop_info, description))
+    return response;
 
   // FIXME implement register handling for exec'd inferiors.
   // if (tid_stop_info.reason == eStopReasonExec) {
   //     const bool force = true;
   //     InitializeRegisters(force);
   // }
 
-  StreamString response;
   // Output the T packet with the thread
   response.PutChar('T');
   int signum = tid_stop_info.details.signal.signo;
   LLDB_LOG(
       log,
       "pid {0}, tid {1}, got signal signo = {2}, reason = {3}, exc_type = {4}",
-      process.GetID(), tid, signum, int(tid_stop_info.reason),
+      process.GetID(), thread.GetID(), signum, int(tid_stop_info.reason),
       tid_stop_info.details.exception.type);
 
   // Print the signal number.
   response.PutHex8(signum & 0xff);
 
   // Include the (pid and) tid.
   response.PutCString("thread:");
   if (bool(m_extensions_supported &
            NativeProcessProtocol::Extension::multiprocess))
     response.Format("p{0:x-}.", process.GetID());
-  response.Format("{0:x-};", tid);
+  response.Format("{0:x-};", thread.GetID());
 
   // Include the thread name if there is one.
-  const std::string thread_name = thread->GetName();
+  const std::string thread_name = thread.GetName();
   if (!thread_name.empty()) {
     size_t thread_name_len = thread_name.length();
 
     if (::strcspn(thread_name.c_str(), "$#+-;:") == thread_name_len) {
       response.PutCString("name:");
       response.PutCString(thread_name);
     } else {
       // The thread name contains special chars, send as hex bytes.
if (thread_index > 1) {
       }
     }
 
     uint32_t i = 0;
     response.PutCString("thread-pcs");
     char delimiter = ':';
     for (NativeThreadProtocol *thread;
          (thread = process.GetThreadAtIndex(i)) != nullptr; ++i) {
-      NativeRegisterContext& reg_ctx = thread->GetRegisterContext();
+      NativeRegisterContext &reg_ctx = thread->GetRegisterContext();
 
       uint32_t reg_to_read = reg_ctx.ConvertRegisterKindToRegisterNumber(
           eRegisterKindGeneric, LLDB_REGNUM_GENERIC_PC);
       const RegisterInfo *const reg_info_p =
           reg_ctx.GetRegisterInfoAtIndex(reg_to_read);
 
       RegisterValue reg_value;
       Status error = reg_ctx.ReadRegister(reg_info_p, reg_value);
if (m_list_threads_in_stop_reply) {
     response.PutChar(';');
   }
 
   //
   // Expedite registers.
   //
 
   // Grab the register context.
-  NativeRegisterContext& reg_ctx = thread->GetRegisterContext();
+  NativeRegisterContext &reg_ctx = thread.GetRegisterContext();
   const auto expedited_regs =
       reg_ctx.GetExpeditedRegisters(ExpeditedRegs::Full);
 
   for (auto &reg_num : expedited_regs) {
     const RegisterInfo *const reg_info_p =
         reg_ctx.GetRegisterInfoAtIndex(reg_num);
     // Only expediate registers that are not contained in other registers.
     if (reg_info_p != nullptr && reg_info_p->value_regs == nullptr) {
       RegisterValue reg_value;
       Status error = reg_ctx.ReadRegister(reg_info_p, reg_value);
       if (error.Success()) {
         response.Printf("%.02x:", reg_num);
         WriteRegisterValueInHexFixedWidth(response, reg_ctx, *reg_info_p,
                                           &reg_value, lldb::eByteOrderBig);
         response.PutChar(';');
       } else {
-        LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s failed to read "
-                       "register '%s' index %" PRIu32 ": %s",
+        LLDB_LOGF(log,
+                  "GDBRemoteCommunicationServerLLGS::%s failed to read "
                   __FUNCTION__,
                   reg_info_p->name ? reg_info_p->name : "<unnamed-register>",
                   reg_num, error.AsCString());
       }
     }
   }
 
   const char *reason_str = GetStopReasonString(tid_stop_info.reason);
if (tid_stop_info.reason == eStopReasonFork ||
     if (tid_stop_info.reason == eStopReasonVFork)
       assert(bool(m_extensions_supported &
                   NativeProcessProtocol::Extension::vfork));
     response.Printf("%s:p%" PRIx64 ".%" PRIx64 ";", reason_str,
                     tid_stop_info.details.fork.child_pid,
                     tid_stop_info.details.fork.child_tid);
   }
 
+  return response;
+}
+
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::SendStopReplyPacketForThread(
+    NativeProcessProtocol &process, lldb::tid_t tid, bool force_synchronous) {
+  // Ensure we can get info on the given thread.
+  NativeThreadProtocol *thread = process.GetThreadByID(tid);
+  if (!thread)
+    return SendErrorResponse(51);
+
+  StreamString response = PrepareStopReplyPacketForThread(*thread);
+  if (response.Empty())
+    return SendErrorResponse(42);
+
+  if (m_non_stop && !force_synchronous) {
+    PacketResult ret = SendNotificationPacketNoLock(
+        "Stop", m_stop_notification_queue, response.GetString());
+    // Queue notification events for the remaining threads.
+    EnqueueStopReplyPackets(tid);
+    return ret;
+  }
+
   return SendPacketNoLock(response.GetString());
 }
 
+void GDBRemoteCommunicationServerLLGS::EnqueueStopReplyPackets(
+    lldb::tid_t thread_to_skip) {
+  if (!m_non_stop)
+    return;
+
+  uint32_t thread_index = 0;
+  while (NativeThreadProtocol *listed_thread =
+             m_current_process->GetThreadAtIndex(thread_index++)) {
+    if (listed_thread->GetID() != thread_to_skip)
+      m_stop_notification_queue.push_back(
+          PrepareStopReplyPacketForThread(*listed_thread).GetString().str());
+  }
+}
+
 void GDBRemoteCommunicationServerLLGS::HandleInferiorState_Exited(
     NativeProcessProtocol *process) {
   assert(process && "process cannot be NULL");
 
   Log *log = GetLog(LLDBLog::Process);
   LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s called", __FUNCTION__);
 
-  PacketResult result =
-      SendStopReasonForState(*process, StateType::eStateExited);
+  PacketResult result = SendStopReasonForState(
+      *process, StateType::eStateExited, /*force_synchronous=*/false);
   if (result != PacketResult::Success) {
     LLDB_LOGF(log,
               "GDBRemoteCommunicationServerLLGS::%s failed to send stop "
               "notification for PID %" PRIu64 ", state: eStateExited",
               __FUNCTION__, process->GetID());
   }
 
   // Close the pipe to the inferior terminal i/o if we launched it and set one
   // up.
   MaybeCloseInferiorTerminalConnection();
 
+  // When running in non-stop mode, wait for the vStopped to clear
+  // the notification queue.
+  if (!m_non_stop) {
     // We are ready to exit the debug monitor.
     m_exit_now = true;
     m_mainloop.RequestTermination();
   }
+}
 
 void GDBRemoteCommunicationServerLLGS::HandleInferiorState_Stopped(
     NativeProcessProtocol *process) {
   assert(process && "process cannot be NULL");
 
   Log *log = GetLog(LLDBLog::Process);
   LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s called", __FUNCTION__);
 
   // Send the stop reason unless this is the stop after the launch or attach.
   switch (m_inferior_prev_state) {
   case eStateLaunching:
   case eStateAttaching:
     // Don't send anything per debugserver behavior.
     break;
   default:
     // In all other cases, send the stop reason.
-    PacketResult result =
-        SendStopReasonForState(*process, StateType::eStateStopped);
+    PacketResult result = SendStopReasonForState(
+        *process, StateType::eStateStopped, /*force_synchronous=*/false);
     if (result != PacketResult::Success) {
       LLDB_LOGF(log,
                 "GDBRemoteCommunicationServerLLGS::%s failed to send stop "
                 "notification for PID %" PRIu64 ", state: eStateExited",
                 __FUNCTION__, process->GetID());
     }
     break;
   }
if (!m_current_process) {
     return PacketResult::Success;
   }
 
   Status error = m_current_process->Kill();
   if (error.Fail())
     LLDB_LOG(log, "Failed to kill debugged process {0}: {1}",
              m_current_process->GetID(), error);
 
-  // No OK response for kill packet.
-  // return SendOKResponse ();
-  return PacketResult::Success;
+  // The response to kill packet is undefined per the spec.  LLDB
+  // follows the same rules as for continue packets, i.e. no response
+  // in all-stop mode, and "OK" in non-stop mode; in both cases this
+  // is followed by the actual stop reason.
+  return SendContinueSuccessResponse();
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_QSetDisableASLR(
     StringExtractorGDBRemote &packet) {
   packet.SetFilePos(::strlen("QSetDisableASLR:"));
   if (packet.GetU32(0))
     m_process_launch_info.GetFlags().Set(eLaunchFlagDisableASLR);
GDBRemoteCommunicationServerLLGS::Handle_C(StringExtractorGDBRemote &packet) {
   error = m_continue_process->Resume(resume_actions);
   if (error.Fail()) {
     LLDB_LOG(log, "failed to resume threads for process {0}: {1}",
              m_continue_process->GetID(), error);
 
     return SendErrorResponse(0x38);
   }
 
-  // Don't send an "OK" packet; response is the stopped/exited message.
-  return PacketResult::Success;
+  // Don't send an "OK" packet, except in non-stop mode;
+  // otherwise, the response is the stopped/exited message.
+  return SendContinueSuccessResponse();
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_c(StringExtractorGDBRemote &packet) {
   Log *log = GetLog(LLDBLog::Process | LLDBLog::Thread);
   LLDB_LOGF(log, "GDBRemoteCommunicationServerLLGS::%s called", __FUNCTION__);
 
   packet.SetFilePos(packet.GetFilePos() + ::strlen("c"));
LLDB_LOGF(log,
               __FUNCTION__);
     return SendErrorResponse(0x36);
   }
 
   // Build the ResumeActionList
   ResumeActionList actions(StateType::eStateRunning,
                            LLDB_INVALID_SIGNAL_NUMBER);
 
   Status error = m_continue_process->Resume(actions);
   if (error.Fail()) {
     LLDB_LOG(log, "c failed for process {0}: {1}", m_continue_process->GetID(),
              error);
     return SendErrorResponse(GDBRemoteServerError::eErrorResume);
   }
 
   LLDB_LOG(log, "continued process {0}", m_continue_process->GetID());
-  // No response required from continue.
-  return PacketResult::Success;
+
+  return SendContinueSuccessResponse();
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_vCont_actions(
     StringExtractorGDBRemote &packet) {
   StreamString response;
   response.Printf("vCont;c;C;s;S");
 
   return SendPacketNoLock(response.GetString());
GDBRemoteCommunicationServerLLGS::Handle_vCont(
   if (::strcmp(packet.Peek(), ";c") == 0) {
     // Move past the ';', then do a simple 'c'.
     packet.SetFilePos(packet.GetFilePos() + 1);
     return Handle_c(packet);
   } else if (::strcmp(packet.Peek(), ";s") == 0) {
     // Move past the ';', then do a simple 's'.
     packet.SetFilePos(packet.GetFilePos() + 1);
     return Handle_s(packet);
+  } else if (m_non_stop && ::strcmp(packet.Peek(), ";t") == 0) {
+    // TODO: add full support for "t" action
+    return SendOKResponse();
   }
 
   // Ensure we have a native process.
   if (!m_continue_process) {
     LLDB_LOG(log, "no debugged process");
     return SendErrorResponse(0x36);
   }
 
GDBRemoteCommunicationServerLLGS::Handle_vCont(
   Status error = m_continue_process->Resume(thread_actions);
   if (error.Fail()) {
     LLDB_LOG(log, "vCont failed for process {0}: {1}",
              m_continue_process->GetID(), error);
     return SendErrorResponse(GDBRemoteServerError::eErrorResume);
   }
 
   LLDB_LOG(log, "continued process {0}", m_continue_process->GetID());
-  // No response required from vCont.
-  return PacketResult::Success;
+
+  return SendContinueSuccessResponse();
 }
 
 void GDBRemoteCommunicationServerLLGS::SetCurrentThreadID(lldb::tid_t tid) {
   Log *log = GetLog(LLDBLog::Thread);
   LLDB_LOG(log, "setting current thread id to {0}", tid);
 
   m_current_tid = tid;
   if (m_current_process)

 GDBRemoteCommunicationServerLLGS::Handle_stop_reason(
     StringExtractorGDBRemote &packet) {
   // Handle the $? gdbremote command.
 
   // If no process, indicate error
   if (!m_current_process)
     return SendErrorResponse(02);
 
+  if (m_non_stop) {
+    // Clear the notification queue first, except for pending exit
+    // notifications.
+    llvm::erase_if(m_stop_notification_queue, [](const std::string &x) {
+      return x.front() != 'W' && x.front() != 'X';
+    });
+
+    // Queue stop reply packets for all active threads.  Start with the current
+    // thread (for clients that don't actually support multiple stop reasons).
+    NativeThreadProtocol *thread = m_current_process->GetCurrentThread();
+    if (thread)
+      m_stop_notification_queue.push_back(
+          PrepareStopReplyPacketForThread(*thread).GetString().str());
+    EnqueueStopReplyPackets(thread ? thread->GetID() : LLDB_INVALID_THREAD_ID);
+
+    // If the notification queue is empty (i.e. everything is running), send OK.
+    if (m_stop_notification_queue.empty())
+      return SendOKResponse();
+
+    // Send the first item from the new notification queue synchronously.
+    return SendPacketNoLock(m_stop_notification_queue.front());
+  }
+
   return SendStopReasonForState(*m_current_process,
-                                m_current_process->GetState());
+                                m_current_process->GetState(),
+                                /*force_synchronous=*/true);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::SendStopReasonForState(
-    NativeProcessProtocol &process, lldb::StateType process_state) {
+    NativeProcessProtocol &process, lldb::StateType process_state,
+    bool force_synchronous) {
   Log *log = GetLog(LLDBLog::Process);
 
   switch (process_state) {
   case eStateAttaching:
   case eStateLaunching:
   case eStateRunning:
   case eStateStepping:
   case eStateDetached:
     // NOTE: gdb protocol doc looks like it should return $OK
     // when everything is running (i.e. no stopped result).
     return PacketResult::Success; // Ignore
 
   case eStateSuspended:
   case eStateStopped:
   case eStateCrashed: {
     lldb::tid_t tid = process.GetCurrentThreadID();
     // Make sure we set the current thread so g and p packets return the data
     // the gdb will expect.
     SetCurrentThreadID(tid);
-    return SendStopReplyPacketForThread(process, tid);
+    return SendStopReplyPacketForThread(process, tid, force_synchronous);
   }
 
   case eStateInvalid:
   case eStateUnloaded:
   case eStateExited:
     return SendWResponse(&process);
 
   default:
if (error.Fail()) {
     LLDB_LOGF(log,
               "GDBRemoteCommunicationServerLLGS::%s pid %" PRIu64
               " tid %" PRIu64 " Resume() failed with error: %s",
               __FUNCTION__, m_continue_process->GetID(), tid,
               error.AsCString());
     return SendErrorResponse(0x49);
   }
 
-  // No response here - the stop or exit will come from the resulting action.
-  return PacketResult::Success;
+  // No response here, unless in non-stop mode.
+  // Otherwise, the stop or exit will come from the resulting action.
+  return SendContinueSuccessResponse();
 }
 
 llvm::Expected<std::unique_ptr<llvm::MemoryBuffer>>
 GDBRemoteCommunicationServerLLGS::BuildTargetXml() {
   // Ensure we have a thread.
   NativeThreadProtocol *thread = m_current_process->GetThreadAtIndex(0);
   if (!thread)
     return llvm::createStringError(llvm::inconvertibleErrorCode(),
if (error.Fail()) {
     LLDB_LOGF(log,
               "GDBRemoteCommunicationServerLLGS::%s failed to attach to "
               "pid %" PRIu64 ": %s\n",
               __FUNCTION__, pid, error.AsCString());
     return SendErrorResponse(error);
   }
 
   // Notify we attached by sending a stop packet.
   assert(m_current_process);
   return SendStopReasonForState(*m_current_process,
-                                m_current_process->GetState());
+                                m_current_process->GetState(),
+                                /*force_synchronous=*/false);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_vAttachWait(
     StringExtractorGDBRemote &packet) {
   Log *log = GetLog(LLDBLog::Process);
 
   // Consume the ';' after the identifier.
if (error.Fail()) {
     LLDB_LOG(log, "failed to attach to process named '{0}': {1}", process_name,
              error);
     return SendErrorResponse(error);
   }
 
   // Notify we attached by sending a stop packet.
   assert(m_current_process);
   return SendStopReasonForState(*m_current_process,
-                                m_current_process->GetState());
+                                m_current_process->GetState(),
+                                /*force_synchronous=*/false);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_qVAttachOrWaitSupported(
     StringExtractorGDBRemote &packet) {
   return SendOKResponse();
 }
 
if (error.Fail()) {
     LLDB_LOG(log, "failed to attach to process named '{0}': {1}", process_name,
              error);
     return SendErrorResponse(error);
   }
 
   // Notify we attached by sending a stop packet.
   assert(m_current_process);
   return SendStopReasonForState(*m_current_process,
-                                m_current_process->GetState());
+                                m_current_process->GetState(),
+                                /*force_synchronous=*/false);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_vRun(
     StringExtractorGDBRemote &packet) {
   Log *log = GetLog(LLDBLog::Process);
 
   llvm::StringRef s = packet.GetStringRef();
GDBRemoteCommunicationServerLLGS::Handle_vRun(
 
   if (!argv.empty()) {
     m_process_launch_info.GetExecutableFile().SetFile(
         m_process_launch_info.GetArguments()[0].ref(), FileSpec::Style::native);
     m_process_launch_error = LaunchProcess();
     if (m_process_launch_error.Success()) {
       assert(m_current_process);
       return SendStopReasonForState(*m_current_process,
-                                    m_current_process->GetState());
+                                    m_current_process->GetState(),
+                                    /*force_synchronous=*/true);
     }
     LLDB_LOG(log, "failed to launch exe: {0}", m_process_launch_error);
   }
   return SendErrorResponse(8);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_D(StringExtractorGDBRemote &packet) {
GDBRemoteCommunicationServerLLGS::Handle_qThreadStopInfo(
   const lldb::tid_t tid = packet.GetHexMaxU64(false, LLDB_INVALID_THREAD_ID);
   if (tid == LLDB_INVALID_THREAD_ID) {
     LLDB_LOGF(log,
               "GDBRemoteCommunicationServerLLGS::%s failed, could not "
               "parse thread id from request \"%s\"",
               __FUNCTION__, packet.GetStringRef().data());
     return SendErrorResponse(0x15);
   }
-  return SendStopReplyPacketForThread(*m_current_process, tid);
+  return SendStopReplyPacketForThread(*m_current_process, tid,
+                                      /*force_synchronous=*/true);
 }
 
 GDBRemoteCommunication::PacketResult
 GDBRemoteCommunicationServerLLGS::Handle_jThreadsInfo(
     StringExtractorGDBRemote &) {
   Log *log = GetLog(LLDBLog::Process | LLDBLog::Thread);
 
   // Ensure we have a debugged process.
if (!ret)
     return SendErrorResponse(ret.takeError());
 
   StreamString response;
   response.PutCString("core-path:");
   response.PutStringAsRawHex8(ret.get());
   return SendPacketNoLock(response.GetString());
 }
 
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::Handle_QNonStop(
+    StringExtractorGDBRemote &packet) {
+  StringRef packet_str{packet.GetStringRef()};
+  assert(packet_str.startswith("QNonStop:"));
+  packet_str.consume_front("QNonStop:");
+  if (packet_str == "0") {
+    m_non_stop = false;
+    // TODO: stop all threads
+  } else if (packet_str == "1") {
+    m_non_stop = true;
+  } else
+    return SendErrorResponse(Status("Invalid QNonStop packet"));
+  return SendOKResponse();
+}
+
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::Handle_vStopped(
+    StringExtractorGDBRemote &packet) {
+  // Per the protocol, the first message put into the queue is sent
+  // immediately.  However, it remains the queue until the client ACKs
+  // it via vStopped -- then we pop it and send the next message.
+  // The process repeats until the last message in the queue is ACK-ed,
+  // in which case the vStopped packet sends an OK response.
+
+  if (m_stop_notification_queue.empty())
+    return SendErrorResponse(Status("No pending notification to ack"));
+  m_stop_notification_queue.pop_front();
+  if (!m_stop_notification_queue.empty())
+    return SendPacketNoLock(m_stop_notification_queue.front());
+  // If this was the last notification and the process exited, terminate
+  // the server.
+  if (m_inferior_prev_state == eStateExited) {
+    m_exit_now = true;
+    m_mainloop.RequestTermination();
+  }
+  return SendOKResponse();
+}
+
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::Handle_vCtrlC(
+    StringExtractorGDBRemote &packet) {
+  if (!m_non_stop)
+    return SendErrorResponse(Status("vCtrl is only valid in non-stop mode"));
+
+  PacketResult interrupt_res = Handle_interrupt(packet);
+  // If interrupting the process failed, pass the result through.
+  if (interrupt_res != PacketResult::Success)
+    return interrupt_res;
+  // Otherwise, vCtrlC should issue an OK response (normal interrupts do not).
+  return SendOKResponse();
+}
+
 void GDBRemoteCommunicationServerLLGS::MaybeCloseInferiorTerminalConnection() {
   Log *log = GetLog(LLDBLog::Process);
 
   // Tell the stdio connection to shut down.
   if (m_stdio_communication.IsConnected()) {
     auto connection = m_stdio_communication.GetConnection();
     if (connection) {
       Status error;

 std::vector<std::string> GDBRemoteCommunicationServerLLGS::HandleFeatures(
     const llvm::ArrayRef<llvm::StringRef> client_features) {
   std::vector<std::string> ret =
       GDBRemoteCommunicationServerCommon::HandleFeatures(client_features);
   ret.insert(ret.end(), {
                             "QThreadSuffixSupported+",
                             "QListThreadsInStopReply+",
                             "qXfer:features:read+",
+                            "QNonStop+",
                         });
 
   // report server-only features
   using Extension = NativeProcessProtocol::Extension;
   Extension plugin_features = m_process_factory.GetSupportedExtensions();
   if (bool(plugin_features & Extension::pass_signals))
     ret.push_back("QPassSignals+");
   if (bool(plugin_features & Extension::auxv))

 
 void GDBRemoteCommunicationServerLLGS::SetEnabledExtensions(
     NativeProcessProtocol &process) {
   NativeProcessProtocol::Extension flags = m_extensions_supported;
   assert(!bool(flags & ~m_process_factory.GetSupportedExtensions()));
   process.SetEnabledExtensions(flags);
 }
 
+GDBRemoteCommunication::PacketResult
+GDBRemoteCommunicationServerLLGS::SendContinueSuccessResponse() {
+  return m_non_stop ? SendOKResponse() : PacketResult::Success;
+}
+
 std::string
 lldb_private::process_gdb_remote::LLGSArgToURL(llvm::StringRef url_arg,
                                                bool reverse_connect) {
   // Try parsing the argument as URL.
   if (llvm::Optional<URI> url = URI::Parse(url_arg)) {
     if (reverse_connect)
       return url_arg.str();
 

lldb/source/Utility/StringExtractorGDBRemote.cpp

case 'L':
         return eServerPacketType_QListThreadsInStopReply;
       break;
 
     case 'M':
       if (PACKET_STARTS_WITH("QMemTags"))
         return eServerPacketType_QMemTags;
       break;
 
+    case 'N':
+      if (PACKET_STARTS_WITH("QNonStop:"))
+        return eServerPacketType_QNonStop;
+      break;
+
     case 'R':
       if (PACKET_STARTS_WITH("QRestoreRegisterState:"))
         return eServerPacketType_QRestoreRegisterState;
       break;
 
     case 'T':
       if (PACKET_MATCHES("QThreadSuffixSupported"))
         return eServerPacketType_QThreadSuffixSupported;
if (PACKET_STARTS_WITH("vFile:")) {
       if (PACKET_STARTS_WITH("vAttachName;"))
         return eServerPacketType_vAttachName;
       if (PACKET_STARTS_WITH("vCont;"))
         return eServerPacketType_vCont;
       if (PACKET_MATCHES("vCont?"))
         return eServerPacketType_vCont_actions;
       if (PACKET_STARTS_WITH("vRun;"))
         return eServerPacketType_vRun;
+      if (PACKET_MATCHES("vStopped"))
+        return eServerPacketType_vStopped;
+      if (PACKET_MATCHES("vCtrlC"))
+        return eServerPacketType_vCtrlC;
+      break;
+
     }
     break;
   case '_':
     switch (packet_cstr[1]) {
     case 'M':
       return eServerPacketType__M;
 
     case 'm':

lldb/test/API/tools/lldb-server/TestNonStop.py

+from lldbsuite.test.lldbtest import *
+
+import gdbremote_testcase
+
+
+class LldbGdbServerTestCase(gdbremote_testcase.GdbRemoteTestCaseBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    def test_run(self):
+        self.build()
+        self.set_inferior_startup_launch()
+        thread_num = 3
+        procs = self.prep_debug_monitor_and_inferior(
+                inferior_args=["thread:segfault"] + thread_num * ["thread:new"])
+        self.test_sequence.add_log_lines(
+            ["read packet: $QNonStop:1#00",
+             "send packet: $OK#00",
+             "read packet: $c#63",
+             "send packet: $OK#00",
+             ], True)
+        self.expect_gdbremote_sequence()
+
+        segv_signo = lldbutil.get_signal_number('SIGSEGV')
+        all_threads = set()
+        all_segv_threads = []
+
+        # we should get segfaults from all the threads
+        for segv_no in range(thread_num):
+            # first wait for the notification event
+            self.reset_test_sequence()
+            self.test_sequence.add_log_lines(
+                [{"direction": "send",
+                  "regex": r"^%Stop:(T([0-9a-fA-F]{2})thread:([0-9a-fA-F]+);)",
+                  "capture": {1: "packet", 2: "signo", 3: "thread_id"},
+                  },
+                 ], True)
+            m = self.expect_gdbremote_sequence()
+            del m["O_content"]
+            threads = [m]
+
+            # then we may get events for the remaining threads
+            # (but note that not all threads may have been started yet)
+            while True:
+                self.reset_test_sequence()
+                self.test_sequence.add_log_lines(
+                    ["read packet: $vStopped#00",
+                     {"direction": "send",
+                      "regex": r"^\$(OK|T([0-9a-fA-F]{2})thread:([0-9a-fA-F]+);)",
+                      "capture": {1: "packet", 2: "signo", 3: "thread_id"},
+                      },
+                     ], True)
+                m = self.expect_gdbremote_sequence()
+                if m["packet"] == "OK":
+                    break
+                del m["O_content"]
+                threads.append(m)
+
+            segv_threads = []
+            other_threads = []
+            for t in threads:
+                signo = int(t["signo"], 16)
+                if signo == segv_signo:
+                    segv_threads.append(t["thread_id"])
+                else:
+                    self.assertEqual(signo, 0)
+                    other_threads.append(t["thread_id"])
+
+            # verify that exactly one thread segfaulted
+            self.assertEqual(len(segv_threads), 1)
+            # we should get only one segv from every thread
+            self.assertNotIn(segv_threads[0], all_segv_threads)
+            all_segv_threads.extend(segv_threads)
+            # segv_threads + other_threads should always be a superset
+            # of all_threads, i.e. we should get states for all threads
+            # already started
+            self.assertFalse(
+                    all_threads.difference(other_threads + segv_threads))
+            all_threads.update(other_threads + segv_threads)
+
+            # verify that `?` returns the same result
+            self.reset_test_sequence()
+            self.test_sequence.add_log_lines(
+                ["read packet: $?#00",
+                 ], True)
+            threads_verify = []
+            while True:
+                self.test_sequence.add_log_lines(
+                    [{"direction": "send",
+                      "regex": r"^\$(OK|T([0-9a-fA-F]{2})thread:([0-9a-fA-F]+);)",
+                      "capture": {1: "packet", 2: "signo", 3: "thread_id"},
+                      },
+                     ], True)
+                m = self.expect_gdbremote_sequence()
+                if m["packet"] == "OK":
+                    break
+                del m["O_content"]
+                threads_verify.append(m)
+                self.reset_test_sequence()
+                self.test_sequence.add_log_lines(
+                    ["read packet: $vStopped#00",
+                     ], True)
+
+            self.assertEqual(threads, threads_verify)
+
+            self.reset_test_sequence()
+            self.test_sequence.add_log_lines(
+                ["read packet: $vCont;C{:02x}:{};c#00"
+                 .format(segv_signo, segv_threads[0]),
+                 "send packet: $OK#00",
+                 ], True)
+            self.expect_gdbremote_sequence()
+
+        # finally, verify that all threads have started
+        self.assertEqual(len(all_threads), thread_num + 1)
+
+    def test_vCtrlC(self):
+        self.build()
+        self.set_inferior_startup_launch()
+        procs = self.prep_debug_monitor_and_inferior(
+                inferior_args=["thread:new"])
+        self.test_sequence.add_log_lines(
+            ["read packet: $QNonStop:1#00",
+             "send packet: $OK#00",
+             "read packet: $c#63",
+             "send packet: $OK#00",
+             "read packet: $vCtrlC#00",
+             "send packet: $OK#00",
+             {"direction": "send",
+              "regex": r"^%Stop:T13",
+              },
+             ], True)
+        self.expect_gdbremote_sequence()
+
+    def test_exit(self):
+        self.build()
+        self.set_inferior_startup_launch()
+        procs = self.prep_debug_monitor_and_inferior()
+        self.test_sequence.add_log_lines(
+            ["read packet: $QNonStop:1#00",
+             "send packet: $OK#00",
+             "read packet: $c#63",
+             "send packet: $OK#00",
+             "send packet: %Stop:W00#00",
+             "read packet: $vStopped#00",
+             "send packet: $OK#00",
+             ], True)
+        self.expect_gdbremote_sequence()
+
+    def test_exit_query(self):
+        self.build()
+        self.set_inferior_startup_launch()
+        procs = self.prep_debug_monitor_and_inferior()
+        self.test_sequence.add_log_lines(
+            ["read packet: $QNonStop:1#00",
+             "send packet: $OK#00",
+             "read packet: $c#63",
+             "send packet: $OK#00",
+             "send packet: %Stop:W00#00",
+             "read packet: $?#00",
+             "send packet: $W00#00",
+             "read packet: $vStopped#00",
+             "send packet: $OK#00",
+             ], True)
+        self.expect_gdbremote_sequence()