Accurate watchpoint hit counts redux

[jingham]

Summary

This is a reworking of the previous patch to get the watchpoint hit counts to be accurate on systems where you have to walk the PC past the faulting instruction before reporting the stop. The previous patch worked reliably on macOS, but failed on the Linux aarch64 bots. As I suspected from the discussion of the previous patch, the problem was that the threads that don't get to step over the faulting instruction see a new StopInfoWatchpoint on the next stop, and so they schedule a new step-over plan, and things go poorly after that...

I fixed that issue by checking whether the thread had gotten a chance to run in StopInfoWatchpoint::ShouldStopSynchronous and not doing any work in that case. In the answer to Pavel's question, I managed to get both modes to work but it is much easier for lldb to deal with debugserver's clearing of stop info's on each step, than lldb-server's preserving them. The former is easy for lldb to reason about, since it is in control of what stop-info's it decides to preserve, the latter requires a little guessing...

However, even with that fix, the "threads/concurrent_events" Linux test runs that had several watchpoints and a breakpoint or signal were flaky. The reason for that is that depending on how events arrived, a signal or breakpoint event could force a stop before the threads stopped at watchpoints all had a chance to move past their faulting instructions. Then on Linux, we would get all the unprocessed watchpoint hits reported anew when the breakpoint stopped so that it looked like the watchpoint threads had hit the watchpoints too many times.

I fixed that by adding an explicit mechanism for a thread & it's thread plans to say "I have work that needs to be done before a public stop can be declared". That way all the threads that stop on the watchpoint can force their step-instructions before the final stop is reported, and we won't end up with some having run the faulting instruction and others not, causing us to overcount the hits. That's the ShouldRunBeforePublicStop machinery.

The only other substantial change over the previous patch is that there were a handful of places where we weren't being rigorous about our rule that "LLDB preserves the StopInfo for thread that doesn't get to run". In one place, we were using GetResumeState not GetTemporaryResumeState to check whether the thread had run. GetResumeState only returns true if the User suspended the thread, not if lldb has suspended it for this one resume for its own purposes, so it wasn't the right test. And in another place, if we get thread stop info in the JSON form, we would unconditionally reset all the threads w/o checking if any of them had run or not.

With this I'm getting clean test suite runs on macOS and on a Linux guest running Ubuntu22.04 on my AS Mac.

[jingham]

Note, just so it isn't confusing, there are some other bits of the ThreadPlan machinery where a ThreadPlan or the Thread itself can say "Even though the thread plan computation says I should stop, I'm going to force a run". However, those controls were all just about the one thread's vote. They didn't force the overall ThreadList::ShouldStop decision. To get this to work, however, I needed threads to be able say "You have to keep resuming till I say I'm done". So there needed to be a different mechanism, checked higher up in the should stop computation, for that behavior.

[labath]

This makes sense to me. Thanks for doing this.

lldb/include/lldb/Breakpoint/Watchpoint.h

public:
   // This doesn't really enable/disable the watchpoint.   It is currently just
   // for use in the Process plugin's {Enable,Disable}Watchpoint, which should
   // be used instead.
   void SetEnabled(bool enabled, bool notify = true);
 
   bool IsHardware() const override;
 
   bool ShouldStop(StoppointCallbackContext *context) override;
   
   bool WatchpointRead() const;
   bool WatchpointWrite() const;
   uint32_t GetIgnoreCount() const;
   void SetIgnoreCount(uint32_t n);
   void SetWatchpointType(uint32_t type, bool notify = true);
   void SetDeclInfo(const std::string &str);
   std::string GetWatchSpec();
   void SetWatchSpec(const std::string &str);
public:
 
   bool IsDisabledDuringEphemeralMode();
 
   const CompilerType &GetCompilerType() { return m_type; }
 
 private:
   friend class Target;
   friend class WatchpointList;
+  friend class StopInfoWatchpoint; // This needs to call UndoHitCount()
 
   void ResetHistoricValues() {
     m_old_value_sp.reset();
     m_new_value_sp.reset();
   }
 
+  void UndoHitCount() { m_hit_counter.Decrement(); }
+
   Target &m_target;
   bool m_enabled;           // Is this watchpoint enabled
   bool m_is_hardware;       // Is this a hardware watchpoint
   bool m_is_watch_variable; // True if set via 'watchpoint set variable'.
   bool m_is_ephemeral;      // True if the watchpoint is in the ephemeral mode,
                             // meaning that it is
   // undergoing a pair of temporary disable/enable actions to avoid recursively
   // triggering further watchpoint events.

lldb/include/lldb/Target/StopInfo.h

 #include <string>
 
 #include "lldb/Target/Process.h"
 #include "lldb/Utility/StructuredData.h"
 #include "lldb/lldb-public.h"
 
 namespace lldb_private {
 
-class StopInfo {
+class StopInfo : public std::enable_shared_from_this<StopInfo> {
   friend class Process::ProcessEventData;
   friend class ThreadPlanBase;
 
 public:
   // Constructors and Destructors
   StopInfo(Thread &thread, uint64_t value);
 
   virtual ~StopInfo() = default;

lldb/include/lldb/Target/Thread.h

public:
 
   // Return whether this thread matches the specification in ThreadSpec.  This
   // is a virtual method because at some point we may extend the thread spec
   // with a platform specific dictionary of attributes, which then only the
   // platform specific Thread implementation would know how to match.  For now,
   // this just calls through to the ThreadSpec's ThreadPassesBasicTests method.
   virtual bool MatchesSpec(const ThreadSpec *spec);
 
+  // Get the current public stop info, calculating it if necessary.
   lldb::StopInfoSP GetStopInfo();
 
   lldb::StopReason GetStopReason();
 
   bool StopInfoIsUpToDate() const;
 
   // This sets the stop reason to a "blank" stop reason, so you can call
   // functions on the thread without having the called function run with

   // NOTE: This function must be called before the current thread plan is
   // moved to the completed plan stack (in Thread::ShouldStop()).
   //
   // NOTE: If subclasses override this function, ensure they do not overwrite
   // the m_actual_stop_info if it is valid.  The stop info may be a
   // "checkpointed and restored" stop info, so if it is still around it is
   // right even if you have not calculated this yourself, or if it disagrees
   // with what you might have calculated.
-  virtual lldb::StopInfoSP GetPrivateStopInfo();
+  virtual lldb::StopInfoSP GetPrivateStopInfo(bool calculate = true);
 
   // Calculate the stop info that will be shown to lldb clients.  For instance,
   // a "step out" is implemented by running to a breakpoint on the function
   // return PC, so the process plugin initially sets the stop info to a
   // StopInfoBreakpoint. But once we've run the ShouldStop machinery, we
   // discover that there's a completed ThreadPlanStepOut, and that's really
   // the StopInfo we want to show.  That will happen naturally the next
   // time GetStopInfo is called, but if you want to force the replacement,
lldb::StateType GetTemporaryResumeState() const {
     return m_temporary_resume_state;
   }
 
   void SetStopInfo(const lldb::StopInfoSP &stop_info_sp);
 
   void ResetStopInfo();
 
   void SetShouldReportStop(Vote vote);
   
+  void SetShouldRunBeforePublicStop(bool newval) { 
+      m_should_run_before_public_stop = newval; 
+  }
+  
+  bool ShouldRunBeforePublicStop() {
+      return m_should_run_before_public_stop;
+  }
+
   /// Sets the extended backtrace token for this thread
   ///
   /// Some Thread subclasses may maintain a token to help with providing
   /// an extended backtrace.  The SystemRuntime plugin will set/request this.
   ///
   /// \param [in] token The extended backtrace token.
   virtual void SetExtendedBacktraceToken(uint64_t token) {}
 
protected:
   lldb::StopInfoSP m_stop_info_sp; ///< The private stop reason for this thread
   uint32_t m_stop_info_stop_id; // This is the stop id for which the StopInfo is
                                 // valid.  Can use this so you know that
   // the thread's m_stop_info_sp is current and you don't have to fetch it
   // again
   uint32_t m_stop_info_override_stop_id; // The stop ID containing the last time
                                          // the stop info was checked against
                                          // the stop info override
+  bool m_should_run_before_public_stop;  // If this thread has "stop others" 
+                                         // private work to do, then it will
+                                         // set this.
   const uint32_t m_index_id; ///< A unique 1 based index assigned to each thread
                              /// for easy UI/command line access.
   lldb::RegisterContextSP m_reg_context_sp; ///< The register context for this
                                             ///thread's current register state.
   lldb::StateType m_state;                  ///< The state of our process.
   mutable std::recursive_mutex
       m_state_mutex;       ///< Multithreaded protection for m_state.
   mutable std::recursive_mutex

lldb/include/lldb/Target/ThreadPlan.h

public:
 
   virtual Vote ShouldReportStop(Event *event_ptr);
 
   Vote ShouldReportRun(Event *event_ptr);
 
   virtual void SetStopOthers(bool new_value);
 
   virtual bool StopOthers();
   
+  virtual bool ShouldRunBeforePublicStop() { return false; }
+
   // This is the wrapper for DoWillResume that does generic ThreadPlan logic,
   // then calls DoWillResume.
   bool WillResume(lldb::StateType resume_state, bool current_plan);
 
   virtual bool WillStop() = 0;
 
   bool IsControllingPlan() { return m_is_controlling_plan; }
 

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp

if (thread_sp) {
         gdb_thread->ClearQueueInfo();
 
       gdb_thread->SetAssociatedWithLibdispatchQueue(
           associated_with_dispatch_queue);
 
       if (dispatch_queue_t != LLDB_INVALID_ADDRESS)
         gdb_thread->SetQueueLibdispatchQueueAddress(dispatch_queue_t);
 
-      // Make sure we update our thread stop reason just once
+      // Make sure we update our thread stop reason just once, but don't 
+      // overwrite the stop info for threads that haven't moved:
+      StopInfoSP current_stop_info_sp = thread_sp->GetPrivateStopInfo(false);
+      if (thread_sp->GetTemporaryResumeState() == eStateSuspended &&
+          current_stop_info_sp) {
+          thread_sp->SetStopInfo(current_stop_info_sp);
+          return thread_sp;
+      }
+
       if (!thread_sp->StopInfoIsUpToDate()) {
         thread_sp->SetStopInfo(StopInfoSP());
         // If there's a memory thread backed by this thread, we need to use it
         // to calculate StopInfo.
         if (ThreadSP memory_thread_sp =
                 m_thread_list.GetBackingThread(thread_sp))
           thread_sp = memory_thread_sp;
 

lldb/source/Target/StopInfo.cpp

 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/ValueObject.h"
 #include "lldb/Expression/UserExpression.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Target/StopInfo.h"
 #include "lldb/Target/Target.h"
 #include "lldb/Target/Thread.h"
 #include "lldb/Target/ThreadPlan.h"
+#include "lldb/Target/ThreadPlanStepInstruction.h"
 #include "lldb/Target/UnixSignals.h"
 #include "lldb/Utility/LLDBLog.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/StreamString.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
if (m_description.empty()) {
       StreamString strm;
       strm.Printf("watchpoint %" PRIi64, m_value);
       m_description = std::string(strm.GetString());
     }
     return m_description.c_str();
   }
 
 protected:
+  using StopInfoWatchpointSP = std::shared_ptr<StopInfoWatchpoint>;
+  // This plan is used to orchestrate stepping over the watchpoint for
+  // architectures (e.g. ARM) that report the watch before running the watched
+  // access.  This is the sort of job you have to defer to the thread plans,
+  // if you try to do it directly in the stop info and there are other threads
+  // that needed to process this stop you will have yanked control away from
+  // them and they won't behave correctly.
+  class ThreadPlanStepOverWatchpoint : public ThreadPlanStepInstruction {
+  public:
+    ThreadPlanStepOverWatchpoint(Thread &thread, 
+                                 StopInfoWatchpointSP stop_info_sp,
+                                 WatchpointSP watch_sp)
+        : ThreadPlanStepInstruction(thread, false, true, eVoteNoOpinion,
+                                    eVoteNoOpinion),
+          m_stop_info_sp(stop_info_sp), m_watch_sp(watch_sp) {
+      assert(watch_sp);
+      m_watch_index = watch_sp->GetHardwareIndex();
+    }
+
+    bool DoWillResume(lldb::StateType resume_state,
+                      bool current_plan) override {
+      if (resume_state == eStateSuspended)
+        return true;
+
+      if (!m_did_disable_wp) {
+        GetThread().GetProcess()->DisableWatchpoint(m_watch_sp.get(), false);
+        m_did_disable_wp = true;
+      }
+      return true;
+    }
+    
+    bool DoPlanExplainsStop(Event *event_ptr) override {
+      if (ThreadPlanStepInstruction::DoPlanExplainsStop(event_ptr))
+        return true;
+      StopInfoSP stop_info_sp = GetThread().GetPrivateStopInfo();
+      // lldb-server resets the stop info for threads that didn't get to run,
+      // so we might have not gotten to run, but still have a watchpoint stop
+      // reason, in which case this will indeed be for us.
+      if (stop_info_sp 
+          && stop_info_sp->GetStopReason() == eStopReasonWatchpoint)
+        return true;
+      return false;
+    }
+
+    void DidPop() override {
+      // Don't artifically keep the watchpoint alive.
+      m_watch_sp.reset();
+    }
+    
+    bool ShouldStop(Event *event_ptr) override {
+      bool should_stop = ThreadPlanStepInstruction::ShouldStop(event_ptr);
+      bool plan_done = MischiefManaged();
+      if (plan_done) {
+        m_stop_info_sp->SetStepOverPlanComplete();
+        GetThread().SetStopInfo(m_stop_info_sp);
+        ResetWatchpoint();
+      }
+      return should_stop;
+    }
+    
+    bool ShouldRunBeforePublicStop() override {
+        return true;
+    }
+
+  protected:
+    void ResetWatchpoint() {
+      if (!m_did_disable_wp)
+        return;
+      m_did_disable_wp = true;
+      GetThread().GetProcess()->EnableWatchpoint(m_watch_sp.get(), true);
+      m_watch_sp->SetHardwareIndex(m_watch_index);
+    }
+
+  private:
+    StopInfoWatchpointSP m_stop_info_sp;
+    WatchpointSP m_watch_sp;
+    uint32_t m_watch_index = LLDB_INVALID_INDEX32;
+    bool m_did_disable_wp = false;
+  };
+
   bool ShouldStopSynchronous(Event *event_ptr) override {
-    // ShouldStop() method is idempotent and should not affect hit count. See
-    // Process::RunPrivateStateThread()->Process()->HandlePrivateEvent()
-    // -->Process()::ShouldBroadcastEvent()->ThreadList::ShouldStop()->
-    // Thread::ShouldStop()->ThreadPlanBase::ShouldStop()->
-    // StopInfoWatchpoint::ShouldStop() and
-    // Event::DoOnRemoval()->Process::ProcessEventData::DoOnRemoval()->
-    // StopInfoWatchpoint::PerformAction().
+    // If we are running our step-over the watchpoint plan, stop if it's done
+    // and continue if it's not:
     if (m_should_stop_is_valid)
       return m_should_stop;
 
+    // If we are running our step over plan, then stop here and let the regular
+    // ShouldStop figure out what we should do:  Otherwise, give our plan
+    // more time to get run:
+    if (m_using_step_over_plan)
+      return m_step_over_plan_complete;
+
+    Log *log = GetLog(LLDBLog::Process);
     ThreadSP thread_sp(m_thread_wp.lock());
-    if (thread_sp) {
-      WatchpointSP wp_sp(
-          thread_sp->CalculateTarget()->GetWatchpointList().FindByID(
-              GetValue()));
-      if (wp_sp) {
-        // Check if we should stop at a watchpoint.
-        ExecutionContext exe_ctx(thread_sp->GetStackFrameAtIndex(0));
-        StoppointCallbackContext context(event_ptr, exe_ctx, true);
-        m_should_stop = wp_sp->ShouldStop(&context);
-      } else {
-        Log *log = GetLog(LLDBLog::Process);
+    assert(thread_sp);
+    
+    if (thread_sp->GetTemporaryResumeState() == eStateSuspended) {
+      // This is the second firing of a watchpoint so don't process it again.
+      LLDB_LOG(log, "We didn't run but stopped with a StopInfoWatchpoint, we "
+               "have already handled this one, don't do it again.");
+      m_should_stop = false;
+      m_should_stop_is_valid = true;
+      return m_should_stop;
+    }
+    
+        thread_sp->CalculateTarget()->GetWatchpointList().FindByID(GetValue()));
+    // If we can no longer find the watchpoint, we just have to stop:
+    if (!wp_sp) {
 
-        LLDB_LOGF(log,
-                  "Process::%s could not find watchpoint location id: %" PRId64
-                  "...",
-                  __FUNCTION__, GetValue());
 
-        m_should_stop = true;
-      }
+      m_should_stop_is_valid = true;
+      return true;
+
+    ExecutionContext exe_ctx(thread_sp->GetStackFrameAtIndex(0));
+    StoppointCallbackContext context(event_ptr, exe_ctx, true);
+    m_should_stop = wp_sp->ShouldStop(&context);
+    if (!m_should_stop) {
+      // This won't happen at present because we only allow one watchpoint per
+      // watched range.  So we won't stop at a watched address with a disabled
+      // watchpoint.  If we start allowing overlapping watchpoints, then we
+      // will have to make watchpoints be real "WatchpointSite" and delegate to
+      // all the watchpoints sharing the site.  In that case, the code below
+      // would be the right thing to do.
+      m_should_stop_is_valid = true;
+      return m_should_stop;
     }
+    // If this is a system where we need to execute the watchpoint by hand
+    // after the hit, queue a thread plan to do that, and then say not to stop.
+    // Otherwise, let the async action figure out whether the watchpoint should
+    // stop
+
+    ProcessSP process_sp = exe_ctx.GetProcessSP();
+    uint32_t num;
+    bool wp_triggers_after;
+
+    if (!process_sp->GetWatchpointSupportInfo(num, wp_triggers_after)
+            .Success()) {
+      m_should_stop_is_valid = true;
+      m_should_stop = true;
+      return m_should_stop;
+    }
+            
+    if (!wp_triggers_after) {
+      // We have to step over the watchpoint before we know what to do:   
+      StopInfoWatchpointSP me_as_siwp_sp 
+          = std::static_pointer_cast<StopInfoWatchpoint>(shared_from_this());
+      ThreadPlanSP step_over_wp_sp(new ThreadPlanStepOverWatchpoint(
+          *(thread_sp.get()), me_as_siwp_sp, wp_sp));
+      Status error;
+      error = thread_sp->QueueThreadPlan(step_over_wp_sp, false);
+      // If we couldn't push the thread plan, just stop here:
+      if (!error.Success()) {
+        LLDB_LOGF(log, "Could not push our step over watchpoint plan: %s", 
+            error.AsCString());
+
+        m_should_stop = true;
+        m_should_stop_is_valid = true;
+        return true;
+      } else {
+      // Otherwise, don't set m_should_stop, we don't know that yet.  Just 
+      // say we should continue, and tell the thread we really should do so:
+        thread_sp->SetShouldRunBeforePublicStop(true);
+        m_using_step_over_plan = true;
+        return false;
+      }
+    } else {
+      // We didn't have to do anything special
       m_should_stop_is_valid = true;
       return m_should_stop;
     }
     
+    return m_should_stop;
+  }
+
   bool ShouldStop(Event *event_ptr) override {
     // This just reports the work done by PerformAction or the synchronous
     // stop. It should only ever get called after they have had a chance to
     // run.
     assert(m_should_stop_is_valid);
     return m_should_stop;
   }
 
   void PerformAction(Event *event_ptr) override {
     Log *log = GetLog(LLDBLog::Watchpoints);
     // We're going to calculate if we should stop or not in some way during the
     // course of this code.  Also by default we're going to stop, so set that
     // here.
     m_should_stop = true;
 
 
     ThreadSP thread_sp(m_thread_wp.lock());
     if (thread_sp) {
 
       WatchpointSP wp_sp(
           thread_sp->CalculateTarget()->GetWatchpointList().FindByID(
               GetValue()));
       if (wp_sp) {
-        ExecutionContext exe_ctx(thread_sp->GetStackFrameAtIndex(0));
-        ProcessSP process_sp = exe_ctx.GetProcessSP();
-
-        {
-          // check if this process is running on an architecture where
-          // watchpoints trigger before the associated instruction runs. if so,
-          // disable the WP, single-step and then re-enable the watchpoint
-          if (process_sp) {
-            uint32_t num;
-            bool wp_triggers_after;
-
-            if (process_sp->GetWatchpointSupportInfo(num, wp_triggers_after)
-                    .Success()) {
-              if (!wp_triggers_after) {
-                // We need to preserve the watch_index before watchpoint  is
-                // disable. Since Watchpoint::SetEnabled will clear the watch
-                // index. This will fix TestWatchpointIter failure
-                Watchpoint *wp = wp_sp.get();
-                uint32_t watch_index = wp->GetHardwareIndex();
-                process_sp->DisableWatchpoint(wp, false);
-                StopInfoSP stored_stop_info_sp = thread_sp->GetStopInfo();
-                assert(stored_stop_info_sp.get() == this);
-
-                Status new_plan_status;
-                ThreadPlanSP new_plan_sp(
-                    thread_sp->QueueThreadPlanForStepSingleInstruction(
-                        false, // step-over
-                        false, // abort_other_plans
-                        true,  // stop_other_threads
-                        new_plan_status));
-                if (new_plan_sp && new_plan_status.Success()) {
-                  new_plan_sp->SetIsControllingPlan(true);
-                  new_plan_sp->SetOkayToDiscard(false);
-                  new_plan_sp->SetPrivate(true);
-                }
-                process_sp->GetThreadList().SetSelectedThreadByID(
-                    thread_sp->GetID());
-                process_sp->ResumeSynchronous(nullptr);
-                process_sp->GetThreadList().SetSelectedThreadByID(
-                    thread_sp->GetID());
-                thread_sp->SetStopInfo(stored_stop_info_sp);
-                process_sp->EnableWatchpoint(wp, false);
-                wp->SetHardwareIndex(watch_index);
-              }
-            }
-          }
-        }
-
         // This sentry object makes sure the current watchpoint is disabled
         // while performing watchpoint actions, and it is then enabled after we
         // are finished.
+        ExecutionContext exe_ctx(thread_sp->GetStackFrameAtIndex(0));
+        ProcessSP process_sp = exe_ctx.GetProcessSP();
+
         WatchpointSentry sentry(process_sp, wp_sp);
 
         /*
          * MIPS: Last 3bits of the watchpoint address are masked by the kernel.
          * For example:
          * 'n' is at 0x120010d00 and 'm' is 0x120010d04. When a watchpoint is
          * set at 'm', then
          * watch exception is generated even when 'n' is read/written. To handle
if (thread_sp) {
               thread_sp->CalculateTarget()->GetWatchpointList().FindByAddress(
                   m_watch_hit_addr);
           if (!wp_hit_sp) {
             m_should_stop = false;
             wp_sp->IncrementFalseAlarmsAndReviseHitCount();
           }
         }
 
-        // TODO: This condition should be checked in the synchronous part of the
-        // watchpoint code
-        // (Watchpoint::ShouldStop), so that we avoid pulling an event even if
-        // the watchpoint fails the ignore count condition. It is moved here
-        // temporarily, because for archs with
-        // watchpoint_exceptions_received=before, the code in the previous
-        // lines takes care of moving the inferior to next PC. We have to check
-        // the ignore count condition after this is done, otherwise we will hit
-        // same watchpoint multiple times until we pass ignore condition, but
-        // we won't actually be ignoring them.
-        if (wp_sp->GetHitCount() <= wp_sp->GetIgnoreCount())
+        if (wp_sp->GetHitCount() <= wp_sp->GetIgnoreCount()) {
           m_should_stop = false;
+          m_should_stop_is_valid = true;
+        }
 
         Debugger &debugger = exe_ctx.GetTargetRef().GetDebugger();
 
         if (m_should_stop && wp_sp->GetConditionText() != nullptr) {
           // We need to make sure the user sees any parse errors in their
           // condition, so we'll hook the constructor errors up to the
           // debugger's Async I/O.
           ExpressionResults result_code;
           EvaluateExpressionOptions expr_options;
           expr_options.SetUnwindOnError(true);
           expr_options.SetIgnoreBreakpoints(true);
           ValueObjectSP result_value_sp;
           Status error;
           result_code = UserExpression::Evaluate(
               exe_ctx, expr_options, wp_sp->GetConditionText(),
               llvm::StringRef(), result_value_sp, error);
 
           if (result_code == eExpressionCompleted) {
             if (result_value_sp) {
               Scalar scalar_value;
               if (result_value_sp->ResolveValue(scalar_value)) {
                 if (scalar_value.ULongLong(1) == 0) {
-                  // We have been vetoed.  This takes precedence over querying
-                  // the watchpoint whether it should stop (aka ignore count
-                  // and friends).  See also StopInfoWatchpoint::ShouldStop()
-                  // as well as Process::ProcessEventData::DoOnRemoval().
+                  // The condition failed, which we consider "not having hit
+                  // the watchpoint" so undo the hit count here.
+                  wp_sp->UndoHitCount();
                   m_should_stop = false;
                 } else
                   m_should_stop = true;
                 LLDB_LOGF(log,
                           "Condition successfully evaluated, result is %s.\n",
                           m_should_stop ? "true" : "false");
               } else {
                 m_should_stop = true;
if (thread_sp) {
                 "Process::%s returning from action with m_should_stop: %d.",
                 __FUNCTION__, m_should_stop);
 
       m_should_stop_is_valid = true;
     }
   }
 
 private:
+  void SetStepOverPlanComplete() {
+    assert(m_using_step_over_plan);
+    m_step_over_plan_complete = true;
+  }
+  
   bool m_should_stop = false;
   bool m_should_stop_is_valid = false;
   lldb::addr_t m_watch_hit_addr;
+  bool m_step_over_plan_complete = false;
+  bool m_using_step_over_plan = false;
 };
 
 // StopInfoUnixSignal
 
 class StopInfoUnixSignal : public StopInfo {
 public:
   StopInfoUnixSignal(Thread &thread, int signo, const char *description)
       : StopInfo(thread, signo) {

lldb/source/Target/Thread.cpp

 }
 
 Thread::Thread(Process &process, lldb::tid_t tid, bool use_invalid_index_id)
     : ThreadProperties(false), UserID(tid),
       Broadcaster(process.GetTarget().GetDebugger().GetBroadcasterManager(),
                   Thread::GetStaticBroadcasterClass().AsCString()),
       m_process_wp(process.shared_from_this()), m_stop_info_sp(),
       m_stop_info_stop_id(0), m_stop_info_override_stop_id(0),
+      m_should_run_before_public_stop(false),
       m_index_id(use_invalid_index_id ? LLDB_INVALID_INDEX32
                                       : process.GetNextThreadIndexID(tid)),
       m_reg_context_sp(), m_state(eStateUnloaded), m_state_mutex(),
       m_frame_mutex(), m_curr_frames_sp(), m_prev_frames_sp(),
       m_resume_signal(LLDB_INVALID_SIGNAL_NUMBER),
       m_resume_state(eStateRunning), m_temporary_resume_state(eStateRunning),
       m_unwinder_up(), m_destroy_called(false),
       m_override_should_notify(eLazyBoolCalculate),
lldb::StopInfoSP Thread::GetStopInfo() {
   }
 }
 
 void Thread::CalculatePublicStopInfo() {
   ResetStopInfo();
   SetStopInfo(GetStopInfo());
 }
 
-lldb::StopInfoSP Thread::GetPrivateStopInfo() {
+lldb::StopInfoSP Thread::GetPrivateStopInfo(bool calculate) {
+  if (!calculate)
+    return m_stop_info_sp;
+
   if (m_destroy_called)
     return m_stop_info_sp;
 
   ProcessSP process_sp(GetProcess());
   if (process_sp) {
     const uint32_t process_stop_id = process_sp->GetStopID();
     if (m_stop_info_stop_id != process_stop_id) {
+      // We preserve the old stop info for a variety of reasons:
+      // 1) Someone has already updated it by the time we get here
+      // 2) We didn't get to execute the breakpoint instruction we stopped at
+      // 3) This is a virtual step so we didn't actually run
+      // 4) If this thread wasn't allowed to run the last time round.
       if (m_stop_info_sp) {
         if (m_stop_info_sp->IsValid() || IsStillAtLastBreakpointHit() ||
-            GetCurrentPlan()->IsVirtualStep())
+            GetCurrentPlan()->IsVirtualStep()
+            || GetTemporaryResumeState() == eStateSuspended)
           SetStopInfo(m_stop_info_sp);
         else
           m_stop_info_sp.reset();
       }
 
       if (!m_stop_info_sp) {
         if (!CalculateStopInfo())
           SetStopInfo(StopInfoSP());
if (need_to_resume) {
     ClearStackFrames();
     // Let Thread subclasses do any special work they need to prior to resuming
     WillResume(resume_state);
   }
 
   return need_to_resume;
 }
 
-void Thread::DidResume() { SetResumeSignal(LLDB_INVALID_SIGNAL_NUMBER); }
+void Thread::DidResume() { 
+  SetResumeSignal(LLDB_INVALID_SIGNAL_NUMBER);
+  // This will get recomputed each time when we stop.
+  SetShouldRunBeforePublicStop(false);
+}
 
 void Thread::DidStop() { SetState(eStateStopped); }
 
 bool Thread::ShouldStop(Event *event_ptr) {
   ThreadPlan *current_plan = GetCurrentPlan();
 
   bool should_stop = true;
 
LLDB_LOGF(log,
               "Thread::%s for tid = 0x%4.4" PRIx64 " 0x%4.4" PRIx64
               ", pc = 0x%16.16" PRIx64
               ", should_stop = 0 (ignore since no stop reason)",
               __FUNCTION__, GetID(), GetProtocolID(),
               GetRegisterContext() ? GetRegisterContext()->GetPC()
                                    : LLDB_INVALID_ADDRESS);
     return false;
   }
   
+  // Clear the "must run me before stop" if it was set:
+  SetShouldRunBeforePublicStop(false);
+
   if (log) {
     LLDB_LOGF(log,
               "Thread::%s(%p) for tid = 0x%4.4" PRIx64 " 0x%4.4" PRIx64
               ", pc = 0x%16.16" PRIx64,
               __FUNCTION__, static_cast<void *>(this), GetID(), GetProtocolID(),
               GetRegisterContext() ? GetRegisterContext()->GetPC()
                                    : LLDB_INVALID_ADDRESS);
     LLDB_LOGF(log, "^^^^^^^^ Thread::ShouldStop Begin ^^^^^^^^");
if (current_plan->TracerExplainsStop()) {
                 current_plan->WillStop();
               PopPlan();
             } while ((current_plan = GetCurrentPlan()) != prev_plan_ptr);
             // Now, if the responsible plan was not "Okay to discard" then
             // we're done, otherwise we forward this to the next plan in the
             // stack below.
             done_processing_current_plan =
                 (plan_ptr->IsControllingPlan() && !plan_ptr->OkayToDiscard());
-          } else
+          } else {
+            bool should_force_run = plan_ptr->ShouldRunBeforePublicStop();
+            if (should_force_run) {
+              SetShouldRunBeforePublicStop(true);
+              should_stop = false;
+            }
             done_processing_current_plan = true;
-
+          }
           break;
         }
       }
     }
   }
 
   if (!done_processing_current_plan) {
     bool override_stop = false;

lldb/source/Target/ThreadList.cpp

bool ThreadList::ShouldStop(Event *event_ptr) {
   {
     // Scope for locker
     std::lock_guard<std::recursive_mutex> guard(GetMutex());
 
     m_process->UpdateThreadListIfNeeded();
     for (lldb::ThreadSP thread_sp : m_threads) {
       // This is an optimization...  If we didn't let a thread run in between
       // the previous stop and this one, we shouldn't have to consult it for
-      // ShouldStop.  So just leave it off the list we are going to inspect. On
-      // Linux, if a thread-specific conditional breakpoint was hit, it won't
+      // ShouldStop.  So just leave it off the list we are going to inspect.
+      // If the thread didn't run but had work to do before declaring a public
+      // stop, then also include it.
+      // On Linux, if a thread-specific conditional breakpoint was hit, it won't
       // necessarily be the thread that hit the breakpoint itself that
       // evaluates the conditional expression, so the thread that hit the
       // breakpoint could still be asked to stop, even though it hasn't been
       // allowed to run since the previous stop.
       if (thread_sp->GetTemporaryResumeState() != eStateSuspended ||
-          thread_sp->IsStillAtLastBreakpointHit())
+          thread_sp->IsStillAtLastBreakpointHit()
+          || thread_sp->ShouldRunBeforePublicStop())
         threads_copy.push_back(thread_sp);
     }
 
     // It is possible the threads we were allowing to run all exited and then
     // maybe the user interrupted or something, then fall back on looking at
     // all threads:
 
     if (threads_copy.size() == 0)
bool ThreadList::ShouldStop(Event *event_ptr) {
   // compute the StopInfo incorrectly. We don't need to use it here, we just
   // want to make sure it gets computed.
 
   for (pos = threads_copy.begin(); pos != end; ++pos) {
     ThreadSP thread_sp(*pos);
     thread_sp->GetStopInfo();
   }
 
+  // If a thread needs to finish some job that can be done just on this thread
+  // before broadcastion the stop, it will signal that by returning true for
+  // ShouldRunBeforePublicStop.  This variable gathers the results from that.
+  bool a_thread_needs_to_run = false;
   for (pos = threads_copy.begin(); pos != end; ++pos) {
     ThreadSP thread_sp(*pos);
 
     // We should never get a stop for which no thread had a stop reason, but
     // sometimes we do see this - for instance when we first connect to a
     // remote stub.  In that case we should stop, since we can't figure out the
     // right thing to do and stopping gives the user control over what to do in
     // this instance.
for (pos = threads_copy.begin(); pos != end; ++pos) {
     // "StopInfoHidden" which will look invalid everywhere but at this check.
 
     if (thread_sp->GetProcess()->GetStopID() > 1)
       did_anybody_stop_for_a_reason = true;
     else
       did_anybody_stop_for_a_reason |= thread_sp->ThreadStoppedForAReason();
 
     const bool thread_should_stop = thread_sp->ShouldStop(event_ptr);
+
     if (thread_should_stop)
       should_stop |= true;
+    else {
+      bool this_thread_forces_run = thread_sp->ShouldRunBeforePublicStop();
+      a_thread_needs_to_run |= this_thread_forces_run;
+      if (this_thread_forces_run) 
+        LLDB_LOG(log,
+                 "ThreadList::{0} thread: {1:x}, "
+                 "says it needs to run before public stop.",
+                 __FUNCTION__, thread_sp->GetID());
+    }
   }
 
-  if (!should_stop && !did_anybody_stop_for_a_reason) {
+  if (a_thread_needs_to_run) {
+    should_stop = false;
+  } else if (!should_stop && !did_anybody_stop_for_a_reason) {
     should_stop = true;
     LLDB_LOGF(log,
               "ThreadList::%s we stopped but no threads had a stop reason, "
               "overriding should_stop and stopping.",
               __FUNCTION__);
   }
 
   LLDB_LOGF(log, "ThreadList::%s overall should_stop = %i", __FUNCTION__,
Vote ThreadList::ShouldReportStop(Event *event_ptr) {
 
   Log *log = GetLog(LLDBLog::Step);
 
   LLDB_LOGF(log, "ThreadList::%s %" PRIu64 " threads", __FUNCTION__,
             (uint64_t)m_threads.size());
 
   // Run through the threads and ask whether we should report this event. For
   // stopping, a YES vote wins over everything.  A NO vote wins over NO
-  // opinion.
+  // opinion.  The exception is if a thread has work it needs to force before
+  // a public stop, which overrides everyone else's opinion:
   for (pos = m_threads.begin(); pos != end; ++pos) {
     ThreadSP thread_sp(*pos);
+    if (thread_sp->ShouldRunBeforePublicStop()) {
+      LLDB_LOG(log, "Thread {0:x} has private business to complete, overrode "
+               "the should report stop.", thread_sp->GetID());
+      result = eVoteNo;
+      break;
+    }
+
     const Vote vote = thread_sp->ShouldReportStop(event_ptr);
     switch (vote) {
     case eVoteNoOpinion:
       continue;
 
     case eVoteYes:
       result = eVoteYes;
       break;
bool ThreadList::WillResume() {
 
   // Now go through the threads and see if any thread wants to run just itself.
   // if so then pick one and run it.
 
   ThreadList run_me_only_list(m_process);
 
   run_me_only_list.SetStopID(m_process->GetStopID());
 
-  bool run_only_current_thread = false;
+  // One or more threads might want to "Stop Others".  We want to handle all
+  // those requests first.  And if there is a thread that wanted to "resume
+  // before a public stop", let it get the first crack:
+  // There are two special kinds of thread that have priority for "StopOthers":
+  // a "ShouldRunBeforePublicStop thread, or the currently selected thread.  If
+  // we find one satisfying that critereon, put it here.
+  ThreadSP stop_others_thread_sp;
 
   for (pos = m_threads.begin(); pos != end; ++pos) {
     ThreadSP thread_sp(*pos);
     if (thread_sp->GetResumeState() != eStateSuspended &&
         thread_sp->GetCurrentPlan()->StopOthers()) {
       if ((*pos)->IsOperatingSystemPluginThread() &&
           !(*pos)->GetBackingThread())
         continue;
 
       // You can't say "stop others" and also want yourself to be suspended.
       assert(thread_sp->GetCurrentPlan()->RunState() != eStateSuspended);
-
-      if (thread_sp == GetSelectedThread()) {
-        // If the currently selected thread wants to run on its own, always let
-        // it.
-        run_only_current_thread = true;
-        run_me_only_list.Clear();
-        run_me_only_list.AddThread(thread_sp);
+
+      if (thread_sp == GetSelectedThread())
+        stop_others_thread_sp = thread_sp;
+        
+      if (thread_sp->ShouldRunBeforePublicStop()) {
+        // This takes precedence, so if we find one of these, service it:
+        stop_others_thread_sp = thread_sp;
         break;
       }
-
-      run_me_only_list.AddThread(thread_sp);
     }
   }
 
   bool need_to_resume = true;
 
   if (run_me_only_list.GetSize(false) == 0) {
     // Everybody runs as they wish:
     for (pos = m_threads.begin(); pos != end; ++pos) {
       ThreadSP thread_sp(*pos);
       StateType run_state;
       if (thread_sp->GetResumeState() != eStateSuspended)
         run_state = thread_sp->GetCurrentPlan()->RunState();
       else
         run_state = eStateSuspended;
       if (!thread_sp->ShouldResume(run_state))
         need_to_resume = false;
     }
   } else {
     ThreadSP thread_to_run;
 
-    if (run_only_current_thread) {
-      thread_to_run = GetSelectedThread();
+    if (stop_others_thread_sp) {
+      thread_to_run = stop_others_thread_sp;
     } else if (run_me_only_list.GetSize(false) == 1) {
       thread_to_run = run_me_only_list.GetThreadAtIndex(0);
     } else {
       int random_thread =
           (int)((run_me_only_list.GetSize(false) * (double)rand()) /
                 (RAND_MAX + 1.0));
       thread_to_run = run_me_only_list.GetThreadAtIndex(random_thread);
     }
 
     for (pos = m_threads.begin(); pos != end; ++pos) {
       ThreadSP thread_sp(*pos);
       if (thread_sp == thread_to_run) {
+        // Note, a thread might be able to fulfil it's plan w/o actually
+        // resuming.  An example of this is a step that changes the current
+        // inlined function depth w/o moving the PC.  Check that here:
         if (!thread_sp->ShouldResume(thread_sp->GetCurrentPlan()->RunState()))
           need_to_resume = false;
       } else
         thread_sp->ShouldResume(eStateSuspended);
     }
   }
 
   return need_to_resume;
 }
 
 void ThreadList::DidResume() {
   std::lock_guard<std::recursive_mutex> guard(GetMutex());
   collection::iterator pos, end = m_threads.end();
   for (pos = m_threads.begin(); pos != end; ++pos) {
     // Don't clear out threads that aren't going to get a chance to run, rather
     // leave their state for the next time around.
     ThreadSP thread_sp(*pos);
-    if (thread_sp->GetResumeState() != eStateSuspended)
+    if (thread_sp->GetTemporaryResumeState() != eStateSuspended)
       thread_sp->DidResume();
   }
 }
 
 void ThreadList::DidStop() {
   std::lock_guard<std::recursive_mutex> guard(GetMutex());
   collection::iterator pos, end = m_threads.end();
   for (pos = m_threads.begin(); pos != end; ++pos) {

lldb/test/API/commands/watchpoints/watchpoint_commands/condition/TestWatchpointConditionCmd.py

def test_watchpoint_cond(self):
         self.expect("thread backtrace", STOPPED_DUE_TO_WATCHPOINT,
                     substrs=['stop reason = watchpoint'])
         self.expect("frame variable --show-globals global",
                     substrs=['(int32_t)', 'global = 5'])
 
         # Use the '-v' option to do verbose listing of the watchpoint.
         # The hit count should now be 2.
         self.expect("watchpoint list -v",
-                    substrs=['hit_count = 5'])
+                    substrs=['hit_count = 1'])

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentDelayWatchBreak.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentDelayWatchBreak(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://81811539")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test (1-second delay) watchpoint and a breakpoint in multiple threads."""
         self.build()
         self.do_thread_actions(
             num_breakpoint_threads=1,
             num_delay_watchpoint_threads=1)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentManyWatchpoints.py

 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentManyWatchpoints(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     @skipIfOutOfTreeDebugserver
     def test(self):
         """Test 100 watchpoints from 100 threads."""
         self.build()
         self.do_thread_actions(num_watchpoint_threads=100)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentNWatchNBreak.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentNWatchNBreak(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
     @expectedFailureAll(archs=["aarch64"], oslist=["freebsd"],
                         bugnumber="llvm.org/pr49433")
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test with 5 watchpoint and breakpoint threads."""
         self.build()
         self.do_thread_actions(num_watchpoint_threads=5,
                                num_breakpoint_threads=5)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentSignalNWatchNBreak.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentSignalNWatchNBreak(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
     @expectedFailureNetBSD
     @expectedFailureAll(archs=["aarch64"], oslist=["freebsd"],
                         bugnumber="llvm.org/pr49433")
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test one signal thread with 5 watchpoint and breakpoint threads."""
         self.build()
         self.do_thread_actions(num_signal_threads=1,
                                num_watchpoint_threads=5,
                                num_breakpoint_threads=5)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentSignalWatch.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentSignalWatch(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test a watchpoint and a signal in multiple threads."""
         self.build()
         self.do_thread_actions(num_signal_threads=1, num_watchpoint_threads=1)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentSignalWatchBreak.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentSignalWatchBreak(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
     @expectedFailureNetBSD
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test a signal/watchpoint/breakpoint in multiple threads."""
         self.build()
         self.do_thread_actions(num_signal_threads=1,
                                num_watchpoint_threads=1,
                                num_breakpoint_threads=1)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentTwoWatchpointThreads.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentTwoWatchpointThreads(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test two threads that trigger a watchpoint. """
         self.build()
         self.do_thread_actions(num_watchpoint_threads=2)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentTwoWatchpointsOneBreakpoint.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentTwoWatchpointsOneBreakpoint(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test two threads that trigger a watchpoint and one breakpoint thread. """
         self.build()
         self.do_thread_actions(
             num_watchpoint_threads=2,
             num_breakpoint_threads=1)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentTwoWatchpointsOneDelayBreakpoint.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentTwoWatchpointsOneDelayBreakpoint(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test two threads that trigger a watchpoint and one (1 second delay) breakpoint thread. """
         self.build()
         self.do_thread_actions(
             num_watchpoint_threads=2,
             num_delay_breakpoint_threads=1)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentTwoWatchpointsOneSignal.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentTwoWatchpointsOneSignal(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
     @expectedFailureNetBSD
-    @skipIf(
-        oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-        archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-        bugnumber="rdar://93863107")
     @add_test_categories(["watchpoint"])
     def test(self):
         """Test two threads that trigger a watchpoint and one signal thread. """
         self.build()
         self.do_thread_actions(num_watchpoint_threads=2, num_signal_threads=1)

lldb/test/API/functionalities/thread/concurrent_events/TestConcurrentWatchBreak.py

 
 import unittest2
 
 from lldbsuite.test.decorators import *
 from lldbsuite.test.concurrent_base import ConcurrentEventsBase
 from lldbsuite.test.lldbtest import TestBase
 
 
 @skipIfWindows
 class ConcurrentWatchBreak(ConcurrentEventsBase):
 
     # Atomic sequences are not supported yet for MIPS in LLDB.
     @skipIf(triple='^mips')
     @add_test_categories(["watchpoint"])
-    @skipIf(
-    oslist=["ios", "watchos", "tvos", "bridgeos", "macosx"],
-    archs=['arm64', 'arm64e', 'arm64_32', 'arm'],
-    bugnumber="rdar://93863107")
 
     def test(self):
         """Test watchpoint and a breakpoint in multiple threads."""
         self.build()
         self.do_thread_actions(
             num_breakpoint_threads=1,
             num_watchpoint_threads=1)