<WIP> [lldb] [Process/NetBSD] Multithread support

[mgorny]

Summary

This is my WIP on fixing multithreaded process support.

So far includes support for reporting new and exited threads, and resuming/stepping individual threads.

[krytarowski]

I think it looks OK. there are some nits that could be optimized in future or handled additionally.. but for now it should be fine.

[labath]

Shouldn't there be some tests that come along with this patch?

lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp

56

Is this really meant to be negative?

[mgorny]

lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp

56

Good catch, it's leftover from PT_STEP usage…

[mgorny]

In D64647#1583125, @labath wrote:

Shouldn't there be some tests that come along with this patch?

I was actually hoping that the test suite already covers what needs to be covered, and buildbot would tell me which tests were fixed ;-).

[labath]

In D64647#1583178, @mgorny wrote:

In D64647#1583125, @labath wrote:

Shouldn't there be some tests that come along with this patch?

I was actually hoping that the test suite already covers what needs to be covered, and buildbot would tell me which tests were fixed ;-).

That is possible, though it would be good to know that before landing the patch. But I would expect that at least some aspect of this functionality is not covered by existing tests (e.g. the handling of multiple concurrent signals).

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

440

Is this "passing multiple signals simultaneously", or "passing multiple *distinct* signals simultaneously". (E.g,. thread 1 gets a SIGALRM, thread 2 gets SIGIO, etc.).

[mgorny]

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

440

The former. Basically there's one siginfo slot, so you can either put a signal for whole process, or for one LWP.

[krytarowski]

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

440

Once we emit a single signal to all threads, it's still technically a single signal that hits the process.

[krytarowski]

Something that we do not cover here is that once a tracee reports a signal (like someone poked it with SIGUSR1) and we want to pass it over to the tracee, we will reset siginfo.

This scenario should be covered by a test and we should handle it properly..

The solution in NetBSD for passing over signals without changing siginfo, is to not calling PT_SET_SIGINFO as the default one will be kept by the kernel. Optionally pick old siginfo with PT_GET_SIGINFO and optionally change destination lwp.

[labath]

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

440

Ok, that makes sense. But I don't think that's what the code does right now (IIUC, this line will only fire if the current signal is different that the signal sent to the previous thread).

[mgorny]

In D64647#1583429, @krytarowski wrote:

Something that we do not cover here is that once a tracee reports a signal (like someone poked it with SIGUSR1) and we want to pass it over to the tracee, we will reset siginfo.

This scenario should be covered by a test and we should handle it properly..

The solution in NetBSD for passing over signals without changing siginfo, is to not calling PT_SET_SIGINFO as the default one will be kept by the kernel. Optionally pick old siginfo with PT_GET_SIGINFO and optionally change destination lwp.

How is 'properly'? Should we reject resuming with a signal when there's already another signal pending?

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

440

There's a second if on line 400 that verifies the number of threads signaled.

The idea is that we have either:

• exactly one thread signaled,
• all threads signaled with the same signal.

Here we check for the 'same signal' condition.

[krytarowski]

In D64647#1584340, @mgorny wrote:

In D64647#1583429, @krytarowski wrote:

Something that we do not cover here is that once a tracee reports a signal (like someone poked it with SIGUSR1) and we want to pass it over to the tracee, we will reset siginfo.

This scenario should be covered by a test and we should handle it properly..

The solution in NetBSD for passing over signals without changing siginfo, is to not calling PT_SET_SIGINFO as the default one will be kept by the kernel. Optionally pick old siginfo with PT_GET_SIGINFO and optionally change destination lwp.

How is 'properly'? Should we reject resuming with a signal when there's already another signal pending?

We need to pass the same signal (with unchanged siginfo) to the tracee.

The easiest way is to not changing it, however as we can emit a signal over to tracee specifying LWP. we can go for the sequence of PT_GET_SIGINFO, change lwp, PT_SET_SIGINFO, PT_CONTINUE with a signal.

[krytarowski]

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

492

For extra completeness, all basic signals (without specified siginfo) from debugger (that weren't intercepted by a debugger as they were routed into debuggee) that are emitted should be of type SI_USER with filled si_pid and si_uid of the debugger.

We thought have the power to set it to whatever value we want, but LLDB probably doesn't allow to set defailed siginfo.

[labath]

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

440

Ah, I see. That makes sense.

It might be more readable to move all the checks up front, perhaps into some function like Expected< ptrace_siginfo_t> ComputeSignalInfo(const ResumeActionList&). Apart from readability, doing this will make sure you don't issue ptrace commands before you before realize that the resume packet was bogus (and leave the inferior in a weird state in the process).

[mgorny]

It seems that the patch is wrong with the current logic, and this is somehow blurry because of Linux implementation limitations. FWIU, eStateSuspended will never happen and instead we should keep threads without explicit action suspsended. It's unclear yet whether this is desirable long-term, or if we should change the logic to explicitly indicate state for each thread.

[labath]

In D64647#1609600, @mgorny wrote:

It seems that the patch is wrong with the current logic, and this is somehow blurry because of Linux implementation limitations. FWIU, eStateSuspended will never happen and instead we should keep threads without explicit action suspsended. It's unclear yet whether this is desirable long-term, or if we should change the logic to explicitly indicate state for each thread.

Yes, eStateSuspended will never happen on linux. It wouldn't be too hard to make use of that state -- we could set unresumed threads to "suspened" in NPL::Resume, and then when we go to stop all threads (NPL::StopRunningThreads), we would just avoid sending signals to suspended threads and just silently set them back to "stopped". However, I don't see a reason to do that now, as it would just add code for jumping between the states.

However, if it makes your life easier, I think you should be able to use eStateSuspended inside NetBSD code for anything you want. Nobody will inspect the state of threads while the process is running, so that could just be an implementation detail.

In fact I suspect the nobody will inspect the thread state whatsoever because once the process is stopped, all threads are assumed to be stopped too.

[mgorny]

Not sure if we're taking about the same thing. I meant the status in resume actions. I think it'd be logical if 'no action' meant 'resume in same state as prior to stopping the process', and explicit eStateS* meant 'keep this thread stopped after resuming'.

[labath]

In D64647#1609712, @mgorny wrote:

Not sure if we're taking about the same thing. I meant the status in resume actions. I think it'd be logical if 'no action' meant 'resume in same state as prior to stopping the process', and explicit eStateS* meant 'keep this thread stopped after resuming'.

Ah, you're right, we weren't talking about the same thing. I see what you mean now...

I can see how explicit modelling of suspension in resume actions would make sense. And it would definitely map better to how netbsd controls threads. OTOH, I think the current semantics (no action => keep stopped) makes sense too, and it maps better to how the gdb protocol represents resume actions. So either way, you have to do the conversion somewhere...

[mgorny]

So I've made some progress on this. However, I'm stuck and I'd use some help going forward.

I've made a trivial two-threaded program:

#include <stdio.h>
#include <assert.h>
#include <unistd.h>
#include <pthread.h>

void* thread_func(void* x) {
	printf("thread 1\n");
	int i;
	for (i = 0; i < 100; i++)
	{
		printf("t2 %d\n", i);
		sleep(3);
	}
	return NULL;
}

int main() {
	pthread_t t;

	printf("step 1\n");
	assert(!pthread_create(&t, NULL, thread_func, NULL));
	printf("step 2\n");
	int i;
	for (i = 0; i < 100; i++)
	{
		printf("t1 %d\n", i);
		sleep(3);
	}
	assert(!pthread_join(t, NULL));
	printf("step 3\n");

	return 0;
}

Now, if I start it, then issue process interrupt, then thread continue 1, everything works as expected. However, if I interrupt it again and then thread continue 2 (i.e. try to change the thread running), nothing happens.

According to the gdb-remote log, vCont is received but nothing is sent back (though normally stdout should happen). If I send any other command, lldb-server dies:

1568906517.952658892 <  16> read packet: $vCont;c:0002#a4
1568906874.978140354 <   1> read packet: 
lldb-server exiting...

According to the posix log, Resume is working as expected.

1568906976.934143305 NativeProcessNetBSD.cpp:Resume                               no action specified for pid 12124 tid 1
1568906976.934377670 NativeProcessNetBSD.cpp:Resume                               processing resume action state suspended signal 2147483647 for pid 12124 tid 1
1568906976.934622765 NativeProcessNetBSD.cpp:Resume                               processing resume action state running signal 2147483647 for pid 12124 tid 2

I suspect I'm not setting some state correctly but I so far haven't managed to figure out which.

[mgorny]

Nevermind that long report, it actually turned out to be a bug in NetBSD kernel that resulted in the thread not being resumed.

[mgorny]

Another WIP update.

[mgorny]

I'm going to upload a new series shortly.

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentBreakpointDelayBreakpointOneSignal.py

 @skipIfWindows
 class ConcurrentBreakpointDelayBreakpointOneSignal(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test two threads that trigger a breakpoint (one with a 1 second delay) and one signal thread. """
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_breakpoint_threads=1,
                                num_delay_breakpoint_threads=1,
                                num_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentBreakpointOneDelayBreakpointThreads.py

 @skipIfWindows
 class ConcurrentBreakpointOneDelayBreakpointThreads(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test threads that trigger a breakpoint where one thread has a 1 second delay. """
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_breakpoint_threads=1,
                                num_delay_breakpoint_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentCrashWithBreak.py

 @skipIfWindows
 class ConcurrentCrashWithBreak(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """ Test a thread that crashes while another thread hits a breakpoint."""
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_crash_threads=1, num_breakpoint_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentCrashWithSignal.py

 @skipIfWindows
 class ConcurrentCrashWithSignal(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """ Test a thread that crashes while another thread generates a signal."""
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_crash_threads=1, num_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentDelaySignalBreak.py

 @skipIfWindows
 class ConcurrentDelaySignalBreak(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test (1-second delay) signal and a breakpoint in multiple threads."""
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(
             num_breakpoint_threads=1,
             num_delay_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentDelayedCrashWithBreakpointSignal.py

 @skipIfWindows
 class ConcurrentDelayedCrashWithBreakpointSignal(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """ Test a thread with a delayed crash while other threads generate a signal and hit a breakpoint. """
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_delay_crash_threads=1,
                                num_breakpoint_threads=1,
                                num_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentManyCrash.py

 from __future__ import print_function
 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentManyCrash(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     @skipIfOutOfTreeDebugserver
     def test(self):
         """Test 100 threads that cause a segfault."""
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_crash_threads=100)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentSignalBreak.py

 @skipIfWindows
 class ConcurrentSignalBreak(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test signal and a breakpoint in multiple threads."""
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_breakpoint_threads=1, num_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentTwoBreakpointThreads.py

 @skipIfWindows
 class ConcurrentTwoBreakpointThreads(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test two threads that trigger a breakpoint. """
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_breakpoint_threads=2)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentTwoBreakpointsOneDelaySignal.py

 @skipIfWindows
 class ConcurrentTwoBreakpointsOneDelaySignal(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test two threads that trigger a breakpoint and one (1 second delay) signal thread. """
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(
             num_breakpoint_threads=2,
             num_delay_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/concurrent_events/TestConcurrentTwoBreakpointsOneSignal.py

 @skipIfWindows
 class ConcurrentTwoBreakpointsOneSignal(ConcurrentEventsBase):
 
     mydir = ConcurrentEventsBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # timing out on buildbot
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @expectedFailureNetBSD
     def test(self):
         """Test two threads that trigger a breakpoint and one signal thread. """
         self.build(dictionary=self.getBuildFlags())
         self.do_thread_actions(num_breakpoint_threads=2, num_signal_threads=1)

lldb/packages/Python/lldbsuite/test/functionalities/thread/exit_during_break/TestExitDuringBreak.py

class ExitDuringBreakpointTestCase(TestBase):
     mydir = TestBase.compute_mydir(__file__)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line number for our breakpoint.
         self.breakpoint = line_number('main.cpp', '// Set breakpoint here')
 
-    @expectedFailureNetBSD
     def test(self):
         """Test thread exit during breakpoint handling."""
         self.build(dictionary=self.getBuildFlags())
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # This should create a breakpoint in the main thread.
         lldbutil.run_break_set_by_file_and_line(

lldb/packages/Python/lldbsuite/test/functionalities/thread/exit_during_step/TestExitDuringStep.py

 
 
 class ExitDuringStepTestCase(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
 
     @skipIfFreeBSD  # llvm.org/pr21411: test is hanging
     @skipIfWindows # This is flakey on Windows: llvm.org/pr38373
-    @expectedFailureNetBSD
     def test(self):
         """Test thread exit during step handling."""
         self.build(dictionary=self.getBuildFlags())
         self.exit_during_step_base(
             "thread step-inst -m all-threads",
             'stop reason = instruction step',
             True)
 

lldb/packages/Python/lldbsuite/test/functionalities/thread/num_threads/TestNumThreads.py

class NumberOfThreadsTestCase(TestBase):
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)
         # Find the line numbers for our break points.
         self.thread3_notify_all_line = line_number('main.cpp', '// Set thread3 break point on notify_all at this line.')
         self.thread3_before_lock_line = line_number('main.cpp', '// thread3-before-lock')
 
-    @expectedFailureNetBSD
     def test_number_of_threads(self):
         """Test number of threads."""
         self.build()
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # This should create a breakpoint with 1 location.
         lldbutil.run_break_set_by_file_and_line(

lldb/packages/Python/lldbsuite/test/functionalities/thread/step_out/TestThreadStepOut.py

class ThreadStepOutTestCase(TestBase):
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23477 Test occasionally times out on the Linux build bot")
     @expectedFailureAll(
         oslist=["freebsd"],
         bugnumber="llvm.org/pr18066 inferior does not exit")
     @skipIfWindows # This test will hang on windows llvm.org/pr21753
     @expectedFailureAll(oslist=["windows"])
-    @expectedFailureNetBSD
     def test_step_single_thread(self):
         """Test thread step out on one thread via command interpreter. """
         self.build(dictionary=self.getBuildFlags())
         self.step_out_test(self.step_out_single_thread_with_cmd)
 
     # Test occasionally times out on the Linux build bot
     @skipIfLinux
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23477 Test occasionally times out on the Linux build bot")
     @expectedFailureAll(
         oslist=["freebsd"],
         bugnumber="llvm.org/pr19347 2nd thread stops at breakpoint")
     @skipIfWindows # This test will hang on windows llvm.org/pr21753
     @expectedFailureAll(oslist=["windows"])
     @expectedFailureAll(oslist=["watchos"], archs=['armv7k'], bugnumber="rdar://problem/34674488") # stop reason is trace when it should be step-out
-    @expectedFailureNetBSD
     def test_step_all_threads(self):
         """Test thread step out on all threads via command interpreter. """
         self.build(dictionary=self.getBuildFlags())
         self.step_out_test(self.step_out_all_threads_with_cmd)
 
     # Test occasionally times out on the Linux build bot
     @skipIfLinux
     @expectedFailureAll(
         oslist=["linux"],
         bugnumber="llvm.org/pr23477 Test occasionally times out on the Linux build bot")
     @expectedFailureAll(
         oslist=["freebsd"],
         bugnumber="llvm.org/pr19347 2nd thread stops at breakpoint")
     @skipIfWindows # This test will hang on windows llvm.org/pr21753
     @expectedFailureAll(oslist=["windows"], bugnumber="llvm.org/pr24681")
-    @expectedFailureNetBSD
     def test_python(self):
         """Test thread step out on one thread via Python API (dwarf)."""
         self.build(dictionary=self.getBuildFlags())
         self.step_out_test(self.step_out_with_python)
 
     def setUp(self):
         # Call super's setUp().
         TestBase.setUp(self)

lldb/packages/Python/lldbsuite/test/functionalities/thread/thread_exit/TestThreadExit.py

def setUp(self):
         TestBase.setUp(self)
         # Find the line numbers for our breakpoints.
         self.break_1 = line_number('main.cpp', '// Set first breakpoint here')
         self.break_2 = line_number('main.cpp', '// Set second breakpoint here')
         self.break_3 = line_number('main.cpp', '// Set third breakpoint here')
         self.break_4 = line_number('main.cpp', '// Set fourth breakpoint here')
 
     @skipIfWindows # This is flakey on Windows: llvm.org/pr38373
-    @expectedFailureNetBSD
     def test(self):
         """Test thread exit handling."""
         self.build(dictionary=self.getBuildFlags())
         exe = self.getBuildArtifact("a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
 
         # This should create a breakpoint with 1 location.
         bp1_id = lldbutil.run_break_set_by_file_and_line(

lldb/packages/Python/lldbsuite/test/python_api/lldbutil/iter/TestLLDBIterator.py

def test_lldb_iter_breakpoint(self):
         self.assertTrue(len(yours) == len(mine))
         for i in range(len(yours)):
             if self.TraceOn():
                 print("yours[%d]='%s'" % (i, get_description(yours[i])))
                 print("mine[%d]='%s'" % (i, get_description(mine[i])))
             self.assertTrue(yours[i] == mine[i],
                             "ID of yours[{0}] and mine[{0}] matches".format(i))
 
-    @expectedFailureNetBSD
     @add_test_categories(['pyapi'])
     def test_lldb_iter_frame(self):
         """Test iterator works correctly for SBProcess->SBThread->SBFrame."""
         self.build()
         exe = self.getBuildArtifact("a.out")
 
         target = self.dbg.CreateTarget(exe)
         self.assertTrue(target, VALID_TARGET)

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.h

private:
 
   // Private Instance Methods
   NativeProcessNetBSD(::pid_t pid, int terminal_fd, NativeDelegate &delegate,
                       const ArchSpec &arch, MainLoop &mainloop);
 
   bool HasThreadNoLock(lldb::tid_t thread_id);
 
   NativeThreadNetBSD &AddThread(lldb::tid_t thread_id);
+  void RemoveThread(lldb::tid_t thread_id);
 
   void MonitorCallback(lldb::pid_t pid, int signal);
   void MonitorExited(lldb::pid_t pid, WaitStatus status);
   void MonitorSIGSTOP(lldb::pid_t pid);
   void MonitorSIGTRAP(lldb::pid_t pid);
   void MonitorSignal(lldb::pid_t pid, int signal);
 
   Status PopulateMemoryRegionCache();

lldb/source/Plugins/Process/NetBSD/NativeProcessNetBSD.cpp

 //===-- NativeProcessNetBSD.cpp ------------------------------- -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "NativeProcessNetBSD.h"
 
 
-
 #include "Plugins/Process/POSIX/ProcessPOSIXLog.h"
 #include "lldb/Host/HostProcess.h"
 #include "lldb/Host/common/NativeRegisterContext.h"
 #include "lldb/Host/posix/ProcessLauncherPosixFork.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Utility/State.h"
 #include "llvm/Support/Errno.h"
 
NativeProcessNetBSD::Factory::Launch(ProcessLaunchInfo &launch_info,
   // Set the architecture to the exe architecture.
   LLDB_LOG(log, "pid = {0:x}, detected architecture {1}", pid,
            Info.GetArchitecture().GetArchitectureName());
 
   std::unique_ptr<NativeProcessNetBSD> process_up(new NativeProcessNetBSD(
       pid, launch_info.GetPTY().ReleaseMasterFileDescriptor(), native_delegate,
       Info.GetArchitecture(), mainloop));
 
+  // Enable event reporting
+  ptrace_event_t events;
+  status = PtraceWrapper(PT_GET_EVENT_MASK, pid, &events, sizeof(events));
+  if (status.Fail())
+    return status.ToError();
+  // TODO: PTRACE_FORK | PTRACE_VFORK | PTRACE_POSIX_SPAWN?
+  events.pe_set_event |= PTRACE_LWP_CREATE | PTRACE_LWP_EXIT;
+  status = PtraceWrapper(PT_SET_EVENT_MASK, pid, &events, sizeof(events));
+  if (status.Fail())
+    return status.ToError();
+
   status = process_up->ReinitializeThreads();
   if (status.Fail())
     return status.ToError();
 
   for (const auto &thread : process_up->m_threads)
     static_cast<NativeThreadNetBSD &>(*thread).SetStoppedBySignal(SIGSTOP);
   process_up->SetState(StateType::eStateStopped, false);
 
case TRAP_EXEC: {
 
     // Let our delegate know we have just exec'd.
     NotifyDidExec();
 
     for (const auto &thread : m_threads)
       static_cast<NativeThreadNetBSD &>(*thread).SetStoppedByExec();
     SetState(StateType::eStateStopped, true);
   } break;
+  case TRAP_LWP: {
+    ptrace_state_t pst;
+    Status error = PtraceWrapper(PT_GET_PROCESS_STATE, GetID(), &pst,
+                                 sizeof(pst));
+    if (error.Fail()) {
+      SetState(StateType::eStateInvalid);
+      return;
+    }
+
+    switch (pst.pe_report_event) {
+      case PTRACE_LWP_CREATE:
+        LLDB_LOG(log,
+                 "monitoring new thread, pid = {0}, LWP = {1}",
+                 GetID(), pst.pe_lwp);
+        AddThread(pst.pe_lwp);
+        break;
+      case PTRACE_LWP_EXIT:
+        LLDB_LOG(log,
+                 "removing exited thread, pid = {0}, LWP = {1}",
+                 GetID(), pst.pe_lwp);
+        RemoveThread(pst.pe_lwp);
+        break;
+    }
+
+    error = PtraceWrapper(PT_CONTINUE, GetID(), reinterpret_cast<void*>(1), 0);
+    if (error.Fail()) {
+      SetState(StateType::eStateInvalid);
+      return;
+    }
+  } break;
   case TRAP_DBREG: {
     // Find the thread.
     NativeThreadNetBSD* thread = nullptr;
     for (const auto &t : m_threads) {
-      if (t->GetID() == info.psi_lwpid) {
+      assert(info.psi_lwpid >= 1);
+      if (t->GetID() == static_cast<lldb::tid_t>(info.psi_lwpid)) {
         thread = static_cast<NativeThreadNetBSD *>(t.get());
         break;
       }
     }
     if (!thread) {
       LLDB_LOG(log,
                "thread not found in m_threads, pid = {0}, LWP = {1}",
                GetID(), info.psi_lwpid);
void NativeProcessNetBSD::MonitorSIGTRAP(lldb::pid_t pid) {
   }
 }
 
 void NativeProcessNetBSD::MonitorSignal(lldb::pid_t pid, int signal) {
   ptrace_siginfo_t info;
   const auto siginfo_err =
       PtraceWrapper(PT_GET_SIGINFO, pid, &info, sizeof(info));
 
-  for (const auto &thread : m_threads) {
-    static_cast<NativeThreadNetBSD &>(*thread).SetStoppedBySignal(
-        info.psi_siginfo.si_signo, &info.psi_siginfo);
+  for (const auto &abs_thread : m_threads) {
+    NativeThreadNetBSD &thread = static_cast<NativeThreadNetBSD &>(*abs_thread);
+    assert(info.psi_lwpid >= 0);
+    if (info.psi_lwpid == 0 ||
+        static_cast<lldb::tid_t>(info.psi_lwpid) == thread.GetID())
+      thread.SetStoppedBySignal(info.psi_siginfo.si_signo, &info.psi_siginfo);
+    else
+      thread.SetStoppedWithNoReason();
   }
   SetState(StateType::eStateStopped, true);
 }
 
 Status NativeProcessNetBSD::PtraceWrapper(int req, lldb::pid_t pid, void *addr,
                                           int data, int *result) {
   Log *log(ProcessPOSIXLog::GetLogIfAllCategoriesSet(POSIX_LOG_PTRACE));
   Status error;
Status NativeProcessNetBSD::PtraceWrapper(int req, lldb::pid_t pid, void *addr,
   LLDB_LOG(log, "ptrace({0}, {1}, {2}, {3})={4:x}", req, pid, addr, data, ret);
 
   if (error.Fail())
     LLDB_LOG(log, "ptrace() failed: {0}", error);
 
   return error;
 }
 
+static llvm::Expected<ptrace_siginfo_t> ComputeSignalInfo(
+    const std::vector<std::unique_ptr<NativeThreadProtocol>> &threads,
+    const ResumeActionList &resume_actions) {
+  // We need to account for three possible scenarios:
+  // 1. no signal being sent.
+  // 2. a signal being sent to one thread.
+  // 3. a signal being sent to the whole process.
+
+  // Count signaled threads.  While at it, determine which signal is being sent
+  // and ensure there's only one.
+  size_t signaled_threads = 0;
+  int signal = LLDB_INVALID_SIGNAL_NUMBER;
+  lldb::tid_t signaled_lwp;
+  for (const auto &thread : threads) {
+    assert(thread && "thread list should not contain NULL threads");
+    const ResumeAction *action =
+        resume_actions.GetActionForThread(thread->GetID(), true);
+    if (action) {
+      if (action->signal != LLDB_INVALID_SIGNAL_NUMBER) {
+        signaled_threads++;
+        if (action->signal != signal) {
+          if (signal != LLDB_INVALID_SIGNAL_NUMBER)
+            return Status("NetBSD does not support passing multiple signals "
+                          "simultaneously")
+                .ToError();
+          signal = action->signal;
+          signaled_lwp = thread->GetID();
+        }
+      }
+    }
+  }
+
+  if (signaled_threads == 0) {
+    ptrace_siginfo_t siginfo;
+    siginfo.psi_siginfo.si_signo = LLDB_INVALID_SIGNAL_NUMBER;
+    return siginfo;
+  }
+
+  if (signaled_threads > 1 && signaled_threads < threads.size())
+    return Status("NetBSD does not support passing signal to 1<i<all threads")
+        .ToError();
+
+  ptrace_siginfo_t siginfo;
+  siginfo.psi_siginfo.si_signo = signal;
+  siginfo.psi_siginfo.si_code = SI_USER;
+  siginfo.psi_siginfo.si_pid = getpid();
+  siginfo.psi_siginfo.si_uid = getuid();
+  if (signaled_threads == 1)
+    siginfo.psi_lwpid = signaled_lwp;
+  else // signal for the whole process
+    siginfo.psi_lwpid = 0;
+  return siginfo;
+}
+
 Status NativeProcessNetBSD::Resume(const ResumeActionList &resume_actions) {
   Log *log(ProcessPOSIXLog::GetLogIfAllCategoriesSet(POSIX_LOG_PROCESS));
   LLDB_LOG(log, "pid {0}", GetID());
 
-  const auto &thread = m_threads[0];
-  const ResumeAction *const action =
-      resume_actions.GetActionForThread(thread->GetID(), true);
+  Status ret;
+
+  Expected<ptrace_siginfo_t> siginfo =
+      ComputeSignalInfo(m_threads, resume_actions);
+  if (!siginfo)
+    return Status(siginfo.takeError());
+
+  for (const auto &abs_thread : m_threads) {
+    assert(abs_thread && "thread list should not contain NULL threads");
+    NativeThreadNetBSD &thread = static_cast<NativeThreadNetBSD &>(*abs_thread);
+
+    const ResumeAction *action =
+        resume_actions.GetActionForThread(thread.GetID(), true);
+    // we need to explicit issue suspend requests, so it is simpler to map it
+    // into proper action
+    ResumeAction suspend_action{thread.GetID(), eStateSuspended,
+                                LLDB_INVALID_SIGNAL_NUMBER};
 
     if (action == nullptr) {
       LLDB_LOG(log, "no action specified for pid {0} tid {1}", GetID(),
-             thread->GetID());
-    return Status();
+               thread.GetID());
+      action = &suspend_action;
     }
 
-  Status error;
-  int signal =
-      action->signal != LLDB_INVALID_SIGNAL_NUMBER ? action->signal : 0;
+    LLDB_LOG(
+        log,
+        "processing resume action state {0} signal {1} for pid {2} tid {3}",
+        action->state, action->signal, GetID(), thread.GetID());
 
     switch (action->state) {
-  case eStateRunning: {
-    // Run the thread, possibly feeding it the signal.
-    error = NativeProcessNetBSD::PtraceWrapper(PT_CONTINUE, GetID(), (void *)1,
-                                               signal);
-    if (!error.Success())
-      return error;
-    for (const auto &thread : m_threads)
-      static_cast<NativeThreadNetBSD &>(*thread).SetRunning();
-    SetState(eStateRunning, true);
+    case eStateRunning:
+      ret = thread.Resume();
       break;
-  }
     case eStateStepping:
-    // Run the thread, possibly feeding it the signal.
-    error = NativeProcessNetBSD::PtraceWrapper(PT_STEP, GetID(), (void *)1,
-                                               signal);
-    if (!error.Success())
-      return error;
-    for (const auto &thread : m_threads)
-      static_cast<NativeThreadNetBSD &>(*thread).SetStepping();
-    SetState(eStateStepping, true);
+      ret = thread.SingleStep();
       break;
-
     case eStateSuspended:
     case eStateStopped:
-    llvm_unreachable("Unexpected state");
+      if (action->signal != LLDB_INVALID_SIGNAL_NUMBER)
+        return Status("Passing signal to suspended thread unsupported");
+
+      ret = thread.Suspend();
+      break;
 
     default:
       return Status("NativeProcessNetBSD::%s (): unexpected state %s specified "
                     "for pid %" PRIu64 ", tid %" PRIu64,
                     __FUNCTION__, StateAsCString(action->state), GetID(),
-                  thread->GetID());
+                    thread.GetID());
     }
 
-  return Status();
+    if (!ret.Success())
+      return ret;
+  }
+
+  int signal = 0;
+  if (siginfo->psi_siginfo.si_signo != LLDB_INVALID_SIGNAL_NUMBER) {
+    ret = PtraceWrapper(PT_SET_SIGINFO, GetID(), &siginfo.get(),
+                        sizeof(*siginfo));
+    if (!ret.Success())
+      return ret;
+    signal = siginfo->psi_siginfo.si_signo;
+  }
+
+  ret = PtraceWrapper(PT_CONTINUE, GetID(), reinterpret_cast<void *>(1),
+                      signal);
+  if (ret.Success())
+    SetState(eStateRunning, true);
+  return ret;
 }
 
 Status NativeProcessNetBSD::Halt() {
   Status error;
 
   if (kill(GetID(), SIGSTOP) != 0)
     error.SetErrorToErrno();
 
NativeThreadNetBSD &NativeProcessNetBSD::AddThread(lldb::tid_t thread_id) {
   // If this is the first thread, save it as the current thread
   if (m_threads.empty())
     SetCurrentThreadID(thread_id);
 
   m_threads.push_back(std::make_unique<NativeThreadNetBSD>(*this, thread_id));
   return static_cast<NativeThreadNetBSD &>(*m_threads.back());
 }
 
+void NativeProcessNetBSD::RemoveThread(lldb::tid_t thread_id) {
+  Log *log(ProcessPOSIXLog::GetLogIfAllCategoriesSet(POSIX_LOG_THREAD));
+  LLDB_LOG(log, "pid {0} removing thread with tid {1}", GetID(), thread_id);
+
+  assert(HasThreadNoLock(thread_id) &&
+         "attempted to remove a thread that does not exist");
+
+  for (auto it = m_threads.begin(); it != m_threads.end(); ++it) {
+    if ((*it)->GetID() == thread_id) {
+      m_threads.erase(it);
+      break;
+    }
+  }
+}
+
 Status NativeProcessNetBSD::Attach() {
   // Attach to the requested process.
   // An attach will cause the thread to stop with a SIGSTOP.
   Status status = PtraceWrapper(PT_ATTACH, m_pid);
   if (status.Fail())
     return status;
 
   int wstatus;

lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.h

public:
 
   Status SetHardwareBreakpoint(lldb::addr_t addr, size_t size) override;
 
   Status RemoveHardwareBreakpoint(lldb::addr_t addr) override;
 
 private:
   // Interface for friend classes
 
+  Status Resume();
+  Status SingleStep();
+  Status Suspend();
+
   void SetStoppedBySignal(uint32_t signo, const siginfo_t *info = nullptr);
   void SetStoppedByBreakpoint();
   void SetStoppedByTrace();
   void SetStoppedByExec();
   void SetStoppedByWatchpoint(uint32_t wp_index);
+  void SetStoppedWithNoReason();
   void SetStopped();
   void SetRunning();
   void SetStepping();
 
   // Member Variables
   lldb::StateType m_state;
   ThreadStopInfo m_stop_info;
   std::unique_ptr<NativeRegisterContext> m_reg_context_up;

lldb/source/Plugins/Process/NetBSD/NativeThreadNetBSD.cpp

 #include "NativeProcessNetBSD.h"
 
 #include "Plugins/Process/POSIX/CrashReason.h"
 #include "Plugins/Process/POSIX/ProcessPOSIXLog.h"
 #include "lldb/Utility/LLDBAssert.h"
 #include "lldb/Utility/RegisterValue.h"
 #include "lldb/Utility/State.h"
 
+// clang-format off
+#include <sys/types.h>
+#include <sys/ptrace.h>
+// clang-format on
+
 #include <sstream>
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::process_netbsd;
 
 NativeThreadNetBSD::NativeThreadNetBSD(NativeProcessNetBSD &process,
                                        lldb::tid_t tid)
     : NativeThreadProtocol(process, tid), m_state(StateType::eStateInvalid),
       m_stop_info(), m_reg_context_up(
 NativeRegisterContextNetBSD::CreateHostNativeRegisterContextNetBSD(process.GetArchitecture(), *this)
 ), m_stop_description() {}
 
+Status NativeThreadNetBSD::Resume() {
+  Status ret = NativeProcessNetBSD::PtraceWrapper(PT_RESUME, m_process.GetID(),
+                                                  nullptr, GetID());
+  if (!ret.Success())
+    return ret;
+  ret = NativeProcessNetBSD::PtraceWrapper(PT_CLEARSTEP, m_process.GetID(),
+                                           nullptr, GetID());
+  if (ret.Success())
+    SetRunning();
+  return ret;
+}
+
+Status NativeThreadNetBSD::SingleStep() {
+  Status ret = NativeProcessNetBSD::PtraceWrapper(PT_RESUME, m_process.GetID(),
+                                                  nullptr, GetID());
+  if (!ret.Success())
+    return ret;
+  ret = NativeProcessNetBSD::PtraceWrapper(PT_SETSTEP, m_process.GetID(),
+                                           nullptr, GetID());
+  if (ret.Success())
+    SetStepping();
+  return ret;
+}
+
+Status NativeThreadNetBSD::Suspend() {
+  Status ret = NativeProcessNetBSD::PtraceWrapper(PT_SUSPEND, m_process.GetID(),
+                                                  nullptr, GetID());
+  if (ret.Success())
+    SetStopped();
+  return ret;
+}
+
 void NativeThreadNetBSD::SetStoppedBySignal(uint32_t signo,
                                             const siginfo_t *info) {
   Log *log(ProcessPOSIXLog::GetLogIfAllCategoriesSet(POSIX_LOG_THREAD));
   LLDB_LOG(log, "tid = {0} in called with signal {1}", GetID(), signo);
 
   SetStopped();
 
   m_stop_info.reason = StopReason::eStopReasonSignal;
void NativeThreadNetBSD::SetStoppedByWatchpoint(uint32_t wp_index) {
   ostr << " " << GetRegisterContext().GetWatchpointHitAddress(wp_index);
 
   m_stop_description = ostr.str();
 
   m_stop_info.reason = StopReason::eStopReasonWatchpoint;
   m_stop_info.details.signal.signo = SIGTRAP;
 }
 
+void NativeThreadNetBSD::SetStoppedWithNoReason() {
+  SetStopped();
+
+  m_stop_info.reason = StopReason::eStopReasonNone;
+  m_stop_info.details.signal.signo = 0;
+}
+
 void NativeThreadNetBSD::SetStopped() {
   const StateType new_state = StateType::eStateStopped;
   m_state = new_state;
   m_stop_description.clear();
 }
 
 void NativeThreadNetBSD::SetRunning() {
   m_state = StateType::eStateRunning;