[trace][intelpt] Support system-wide tracing [9] - Collect and return context switch traces

[wallace]

Summary

• Add collection of context switches per cpu grouped with the per-cpu intel pt traces.
• Move the state handling from the interl pt trace class to the PerfEvent one.
• Add support for stopping and enabling perf event groups.
• Return context switch entries as part of the jLLDBTraceGetState response.
• Move the triggers of whenever the process stopped or resumed. Now the will-resume notification is in a better location, which will ensure that we'll capture the instructions that will be executed.
• Remove IntelPTSingleBufferTraceUP. The unique pointer was useless.
• Add unit tests

[wallace]

nit

[jj10306]

Submitting my comments so far, will continue my review in a bit

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

26–29

nit: these are just pointers to a buffer of bytes right? if so, maybe we could change the types to be const uint8_t *.
Obviously it doesn't make a difference since they are the same size, but when I see const char * my brain kinda immediately thinks STRING!
maybe it's just me though 🤩, wdyt?

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

41–45

Why do we need to separate these out into two separate functions?

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

36

thoughts on moving this to be a "utility" method of the Perf.cpp and then call that utility method from here? Moving it there would make it more accessible to other callers.

47

what does this do?

51–54

In addition to this context switching event needing to be part of the same group, you also must ensure that some other config bits (exclude_kernel, exclude_hv) are the same.
Also, what would happen if the disabled bit is set here but then the enabled bit of the intel pt event was set?

All of these considerations related to keeping the two events "in sync", are beginning to make me lean towards what I mentioned above about using the same perf event, because that would naturally remove any opportunities for the two events to be "out of sync"

56

101–116

to reduce opportunity for things to get out of sync if this is ever touched in the future?

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.h

94

perhaps you could make this it's own structure? that would make things a bit less verbose here and in the ForEachCore callback?
also, this may make it clearer that they must be part of the same perf event and potentially would allow enforcing this easier?
Instead of creating an entirely new structure you also could make the ContextSwitchTrace an optional member of IntelPTSingleBuffer trace, but not sure if that's the best place to do the coupling.
Another option would be to just use the same underlying perf_event of IntelPTSingleBufferTrace. That would naturally enforce the constraint of the events being part of the same group and wouldn't require adding this new ContextSwitchTrace notion.

lots of options lol, I think things are fine as is but wanted to share my thoughts.
wdyt?

[jj10306]

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

122–154

Is there a way to consolidate these two methods?

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.h

47

This wording is a bit confusing - maybe provide some additional context here about this flag controlling the whether the perf event is enabled/disabled at perf_event_open time

lldb/source/Plugins/Process/Linux/Perf.cpp

161

curious what adding the PROT_READ does here, given that this code was working with just PROT_WRITE it seems like PROT_WRITE implies PROT_READ

226

It's worth noting that this function should only be called when the aux buffer is configured as a ring buffer (ie the aux buffer was mmaped with PROT_READ). if the aux buffer mmaped with PROT_WRITE then the behavior of the head is different and the consumer is required to update the aux_tail when reading.
Perhaps we should check this before doing the reading or make it very clear in the docs that this should only be called on perf events with that specific configuration

lldb/source/Plugins/Process/Linux/Perf.h

101

is an enum needed for this? Since the perf event will only ever be in two states, feels like a bool would suffice

238

nit: This is wordy. imo the name needs not mention that it's flushed out, just put that in the docs

lldb/test/API/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.py

226–229

nice test!

[wallace]

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

26–29

oh no, these are just string enums, but it happens that c++ doesn't support natively string enums

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

41–45

The "state" thing is an implementation detail, the most important thing for callers is to notify the collector what's happening with the target.

So I'm also going with this approach for clarity, following the standard that LLDB uses everywhere. DidStop and WillResume appear so in the LLDB code often that it's better to keep them here for consistency.

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

36

good idea

47

this is super redundant because this field is already 0 after the memset

51–54

I've moved this function to Perf.h, so given this:

• disabled is dominated by the group perf event state, so I'll use that state instead of asking for the disabled value.
• exclude_hv is needed all the time
• exclude_kernel as well

so, besides the change to disabled, the only two fields that we should really synchronize are exclude_hv and exclude_kernel, but I don't see us creating more perf events, so I don't think if it's worth creating more abstractions just for these two fields. We could do more in the future when we need to trace kernel. You can leave more suggestions in my newer diff

56

+1

101–116

+1

122–154

not really :(

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.h

94

This map is barely used, so I don't want to create more structs for this. The idea of having an optional context switch trace is not very nice to me because that makes the code less composable. In any case, I'm also not a far of my gigantic map, but I don't find it bad enough to change it.
Something I'm doing now is to use the alias ContextSwitchTrace more ubiquitously to make other parts of the code more readable.

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.h

47

+1

lldb/source/Plugins/Process/Linux/Perf.cpp

161

it was needed for context switches. I don't remember why though

226

i'll leave a comment for when we add a way to read the buffer in a non-cyclic manner

lldb/source/Plugins/Process/Linux/Perf.h

101

+1

238

I want to be super explicit, because this means that while we read, we are pausing the state of the event, so that's non-trivial side effect that should be clear by any caller.

[wallace]

This was fixed in https://reviews.llvm.org/D127456

lldb/include/lldb/Host/common/NativeProcessProtocol.h

public:
 
     /// Get the bitmask of extensions supported by this process plugin.
     ///
     /// \return
     ///     A NativeProcessProtocol::Extension bitmask.
     virtual Extension GetSupportedExtensions() const { return {}; }
   };
 
+  /// Notify tracers that the target process will resume
+  virtual void NotifyTracersProcessWillResume() {}
+
+  /// Notify tracers that the target process just stopped
+  virtual void NotifyTracersProcessDidStop() {}
+
   /// Start tracing a process or its threads.
   ///
   /// \param[in] json_params
   ///     JSON object with the information of what and how to trace.
   ///     In the case of gdb-remote, this object should conform to the
   ///     jLLDBTraceStart packet.
   ///
   ///     This object should have a string entry called "type", which is the
protected:
   /// Notify the delegate that an exec occurred.
   ///
   /// Provide a mechanism for a delegate to clear out any exec-
   /// sensitive data.
   void NotifyDidExec();
 
   NativeThreadProtocol *GetThreadByIDUnlocked(lldb::tid_t tid);
 
-  /// Notify tracers that the state of the target process has changed.
-  virtual void NotifyTracersProcessStateChanged(lldb::StateType state) {}
-
 private:
   void SynchronouslyNotifyProcessStateChanged(lldb::StateType state);
   llvm::Expected<SoftwareBreakpoint>
   EnableSoftwareBreakpoint(lldb::addr_t addr, uint32_t size_hint);
 };
 } // namespace lldb_private
 
 #endif // LLDB_HOST_COMMON_NATIVEPROCESSPROTOCOL_H

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

 /// See docs/lldb-gdb-remote.txt for more information.
 ///
 /// Do not use system-dependent types, like size_t, because they might cause
 /// issues when compiling on arm.
 namespace lldb_private {
 
 // List of data kinds used by jLLDBGetState and jLLDBGetBinaryData.
 struct IntelPTDataKinds {
   static const char *kProcFsCpuInfo;
   static const char *kTraceBuffer;
+  static const char *kPerfContextSwitchTrace;
 };
 
 /// jLLDBTraceStart gdb-remote packet
 /// \{
 struct TraceIntelPTStartRequest : TraceStartRequest {
   /// Size in bytes to use for each thread's trace buffer.
   uint64_t trace_buffer_size;
 
   /// Whether to enable TSC

lldb/source/Host/common/NativeProcessProtocol.cpp

Status NativeProcessProtocol::RemoveHardwareBreakpoint(lldb::addr_t addr) {
   return error;
 }
 
 void NativeProcessProtocol::SynchronouslyNotifyProcessStateChanged(
     lldb::StateType state) {
   Log *log = GetLog(LLDBLog::Process);
 
   m_delegate.ProcessStateChanged(this, state);
-  NotifyTracersProcessStateChanged(state);
+
+  switch (state) {
+  case eStateStopped:
+  case eStateExited:
+  case eStateCrashed:
+    NotifyTracersProcessDidStop();
+    break;
+  default:
+    break;
+  }
 
   LLDB_LOG(log, "sent state notification [{0}] from process {1}", state,
            GetID());
 }
 
 void NativeProcessProtocol::NotifyDidExec() {
   Log *log = GetLog(LLDBLog::Process);
   LLDB_LOG(log, "process {0} exec()ed", GetID());

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

 class IntelPTCollector {
 public:
   /// \param[in] process
   ///     Process to be traced.
   IntelPTCollector(NativeProcessProtocol &process);
 
   static bool IsSupported();
 
-  /// To be invoked whenever the state of the target process has changed.
-  void OnProcessStateChanged(lldb::StateType state);
+  /// To be invoked as soon as we know the process stopped.
+  void ProcessDidStop();
+
+  /// To be invoked before the process will resume, so that we can capture all
+  /// the instructions.
+  void ProcessWillResume();
 
   /// If "process tracing" is enabled, then trace the given thread.
   llvm::Error OnThreadCreated(lldb::tid_t tid);
 
   /// Stops tracing a tracing upon a destroy event.
   llvm::Error OnThreadDestroyed(lldb::tid_t tid);
 
   /// Implementation of the jLLDBTraceStop packet

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

for (int64_t tid : *request.tids) {
       else
         error = joinErrors(std::move(error),
                            m_thread_traces.TraceStart(tid, request));
     }
     return error;
   }
 }
 
-void IntelPTCollector::OnProcessStateChanged(lldb::StateType state) {
+void IntelPTCollector::ProcessWillResume() {
   if (m_process_trace_up)
-    m_process_trace_up->OnProcessStateChanged(state);
+    m_process_trace_up->ProcessWillResume();
+}
+
+void IntelPTCollector::ProcessDidStop() {
+  if (m_process_trace_up)
+    m_process_trace_up->ProcessDidStop();
 }
 
 Error IntelPTCollector::OnThreadCreated(lldb::tid_t tid) {
   if (m_process_trace_up)
     return m_process_trace_up->TraceStart(tid);
 
   return Error::success();
 }

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.h

public:
   ///   The core id that is being traced.
   ///
   /// \param[in] callback.core_trace
   ///   The single-buffer trace instance for the given core.
   void ForEachCore(std::function<void(lldb::core_id_t core_id,
                                       IntelPTSingleBufferTrace &core_trace)>
                        callback);
 
-  void OnProcessStateChanged(lldb::StateType state) override;
+  /// Execute the provided callback on each core that is being traced.
+  ///
+  /// \param[in] callback.core_id
+  ///   The core id that is being traced.
+  ///
+  /// \param[in] callback.intelpt_trace
+  ///   The single-buffer intel pt trace instance for the given core.
+  ///
+  /// \param[in] callback.context_switch_trace
+  ///   The perf event collecting context switches for the given core.
+  void ForEachCore(std::function<void(lldb::core_id_t core_id,
+                                      IntelPTSingleBufferTrace &intelpt_trace,
+                                      PerfEvent &context_switch_trace)>
+                       callback);
+
+  void ProcessDidStop() override;
+
+  void ProcessWillResume() override;
 
   TraceGetStateResponse GetState() override;
 
   bool TracesThread(lldb::tid_t tid) const override;
 
   llvm::Error TraceStart(lldb::tid_t tid) override;
 
   llvm::Error TraceStop(lldb::tid_t tid) override;
 
   llvm::Expected<std::vector<uint8_t>>
   GetBinaryData(const TraceGetBinaryDataRequest &request) override;
 
 private:
+  /// This assumes that all underlying perf_events for each core are part of the
+  /// same perf event group.
   IntelPTMultiCoreTrace(
       llvm::DenseMap<lldb::core_id_t, IntelPTSingleBufferTraceUP>
-          &&traces_per_core,
+          &&intelpt_traces_per_core,
+      llvm::DenseMap<lldb::core_id_t, PerfEvent>
+          &&context_switch_traces_per_core,
       NativeProcessProtocol &process)
-      : m_traces_per_core(std::move(traces_per_core)), m_process(process) {}
-
-  llvm::DenseMap<lldb::core_id_t, IntelPTSingleBufferTraceUP> m_traces_per_core;
+      : m_intelpt_traces_per_core(std::move(intelpt_traces_per_core)),
+        m_context_switch_traces_per_core(
+            std::move(context_switch_traces_per_core)),
+        m_process(process) {}
 
-  /// The initial state is stopped because tracing can only start when the
-  /// process is paused.
-  lldb::StateType m_process_state = lldb::StateType::eStateStopped;
+  llvm::DenseMap<lldb::core_id_t, IntelPTSingleBufferTraceUP>
+      m_intelpt_traces_per_core;
+  llvm::DenseMap<lldb::core_id_t, PerfEvent> m_context_switch_traces_per_core;
 
   /// The target process.
   NativeProcessProtocol &m_process;
 };
 
 } // namespace process_linux
 } // namespace lldb_private
 
 #endif // liblldb_IntelPTMultiCoreTrace_H_

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

 static Error IncludePerfEventParanoidMessageInError(Error &&error) {
   return createStringError(
       inconvertibleErrorCode(),
       "%s\nYou might need to rerun as sudo or to set "
       "/proc/sys/kernel/perf_event_paranoid to a value of 0 or -1.",
       toString(std::move(error)).c_str());
 }
 
+static Expected<PerfEvent> CreateContextSwitchTracePerfEvent(
+    bool disabled, lldb::core_id_t core_id,
+    IntelPTSingleBufferTrace &intelpt_core_trace) {
+  Log *log = GetLog(POSIXLog::Trace);
+#ifndef PERF_ATTR_SIZE_VER5
+  return createStringError(inconvertibleErrorCode(),
+                           "Intel PT Linux perf event not supported");
+#else
+  perf_event_attr attr;
+  memset(&attr, 0, sizeof(attr));
+  attr.size = sizeof(attr);
+  attr.sample_period = 0;
+  attr.sample_type = PERF_SAMPLE_TID | PERF_SAMPLE_TIME;
+  attr.type = PERF_TYPE_SOFTWARE;
+  attr.context_switch = 1;
+  attr.exclude_kernel = 1;
+  attr.sample_id_all = 1;
+  attr.exclude_hv = 1;
+  attr.disabled = disabled;
+
+  // The given perf configuration will product context switch records of 32
+  // bytes each. Assuming that every context switch will be emitted twice (one
+  // for context switch ins and another one for context switch outs), and that a
+  // context switch will happen at least every half a millisecond per core, we
+  // need 500 * 32 bytes (~16 KB) for a trace of one second, which is much more
+  // than what a regular intel pt trace can get. Pessimistically we pick as
+  // 32KiB for the size of our context switch trace.
+
+  uint64_t data_buffer_size = 32768;
+  uint64_t data_buffer_numpages = data_buffer_size / getpagesize();
+
+  LLDB_LOG(log, "Will create context switch trace buffer of size {0}",
+           data_buffer_size);
+
+  if (Expected<PerfEvent> perf_event = PerfEvent::Init(
+          attr, /*pid=*/None, core_id,
+          intelpt_core_trace.GetPerfEvent().GetFd(), /*flags=*/0)) {
+    if (Error mmap_err =
+            perf_event->MmapMetadataAndBuffers(data_buffer_numpages, 0)) {
+      return std::move(mmap_err);
+    }
+    return perf_event;
+  } else {
+    return perf_event.takeError();
+  }
+#endif
+}
+
 Expected<IntelPTProcessTraceUP>
 IntelPTMultiCoreTrace::StartOnAllCores(const TraceIntelPTStartRequest &request,
                                        NativeProcessProtocol &process) {
   Expected<ArrayRef<core_id_t>> core_ids = GetAvailableLogicalCoreIDs();
   if (!core_ids)
     return core_ids.takeError();
 
   if (IsTotalBufferLimitReached(*core_ids, request))
     return createStringError(
         inconvertibleErrorCode(),
         "The process can't be traced because the process trace size limit "
         "has been reached. Consider retracing with a higher limit.");
 
-  llvm::DenseMap<core_id_t, IntelPTSingleBufferTraceUP> buffers;
+  DenseMap<core_id_t, IntelPTSingleBufferTraceUP> intelpt_traces;
+  DenseMap<core_id_t, PerfEvent> context_switch_traces;
+
   for (core_id_t core_id : *core_ids) {
-    if (Expected<IntelPTSingleBufferTraceUP> core_trace =
-            IntelPTSingleBufferTrace::Start(request, /*tid=*/None, core_id,
-                                            TraceCollectionState::Paused))
-      buffers.try_emplace(core_id, std::move(*core_trace));
-    else
+    Expected<IntelPTSingleBufferTraceUP> core_trace =
+                                        /*disabled=*/true);
+    if (!core_trace)
       return IncludePerfEventParanoidMessageInError(core_trace.takeError());
+
+    if (Expected<PerfEvent> context_switch_trace =
+            CreateContextSwitchTracePerfEvent(/*disabled=*/true, core_id,
+                                              *core_trace.get())) {
+      intelpt_traces.try_emplace(core_id, std::move(*core_trace));
+      context_switch_traces.try_emplace(core_id,
+                                        std::move(*context_switch_trace));
+    } else {
+      return context_switch_trace.takeError();
+    }
   }
 
-  return IntelPTProcessTraceUP(
-      new IntelPTMultiCoreTrace(std::move(buffers), process));
+  return IntelPTProcessTraceUP(new IntelPTMultiCoreTrace(
+      std::move(intelpt_traces), std::move(context_switch_traces), process));
 }
 
 void IntelPTMultiCoreTrace::ForEachCore(
     std::function<void(core_id_t core_id, IntelPTSingleBufferTrace &core_trace)>
         callback) {
-  for (auto &it : m_traces_per_core)
+  for (auto &it : m_intelpt_traces_per_core)
     callback(it.first, *it.second);
 }
 
-void IntelPTMultiCoreTrace::OnProcessStateChanged(lldb::StateType state) {
-  if (m_process_state == state)
-    return;
-  switch (state) {
-  case eStateStopped:
-  case eStateExited: {
-    ForEachCore([](core_id_t core_id, IntelPTSingleBufferTrace &core_trace) {
-      if (Error err =
-              core_trace.ChangeCollectionState(TraceCollectionState::Paused)) {
-        LLDB_LOG_ERROR(GetLog(POSIXLog::Trace), std::move(err),
-                       "Unable to pause the core trace for core {0}", core_id);
-      }
-    });
-    break;
-  }
-  case eStateRunning: {
-    ForEachCore([](core_id_t core_id, IntelPTSingleBufferTrace &core_trace) {
-      if (Error err =
-              core_trace.ChangeCollectionState(TraceCollectionState::Running)) {
-        LLDB_LOG_ERROR(GetLog(POSIXLog::Trace), std::move(err),
-                       "Unable to resume the core trace for core {0}", core_id);
-      }
-    });
-    break;
-  }
-  default:
-    break;
-  }
+void IntelPTMultiCoreTrace::ForEachCore(
+    std::function<void(core_id_t core_id,
+                       IntelPTSingleBufferTrace &intelpt_trace,
+                       PerfEvent &context_switch_trace)>
+        callback) {
+  for (auto &it : m_intelpt_traces_per_core)
+    callback(it.first, *it.second,
+             m_context_switch_traces_per_core.find(it.first)->second);
+}
+
+void IntelPTMultiCoreTrace::ProcessDidStop() {
+    if (Error err = core_trace.Pause()) {
+
+void IntelPTMultiCoreTrace::ProcessWillResume() {
+    if (Error err = core_trace.Resume()) {
 }
 
 TraceGetStateResponse IntelPTMultiCoreTrace::GetState() {
   TraceGetStateResponse state;
 
   for (size_t i = 0; m_process.GetThreadAtIndex(i); i++)
     state.traced_threads.push_back(
         TraceThreadState{m_process.GetThreadAtIndex(i)->GetID(), {}});
 
   state.cores.emplace();
   ForEachCore([&](lldb::core_id_t core_id,
-                  const IntelPTSingleBufferTrace &core_trace) {
+                  const IntelPTSingleBufferTrace &core_trace,
+                  const PerfEvent &context_switch_trace) {
     state.cores->push_back(
         {core_id,
-         {{IntelPTDataKinds::kTraceBuffer, core_trace.GetTraceBufferSize()}}});
+         {{IntelPTDataKinds::kTraceBuffer, core_trace.GetTraceBufferSize()},
+          {IntelPTDataKinds::kPerfContextSwitchTrace,
+           context_switch_trace.GetEffectiveDataBufferSize()}}});
   });
 
   return state;
 }
 
 bool IntelPTMultiCoreTrace::TracesThread(lldb::tid_t tid) const {
   // All the process' threads are being traced automatically.
   return (bool)m_process.GetThreadByID(tid);

lldb/source/Plugins/Process/Linux/IntelPTProcessTrace.h

 namespace lldb_private {
 namespace process_linux {
 
 // Abstract class to be inherited by all the process tracing strategies.
 class IntelPTProcessTrace {
 public:
   virtual ~IntelPTProcessTrace() = default;
 
-  /// This method should be invoked as early as possible whenever the process
-  /// resumes or stops so that intel-pt collection is not enabled when
-  /// the process is not running. A case in which this is useful in when
-  /// tracing is done per-core. In this case we want to prevent polluting the
-  /// core traces with executions of unrelated processes, which increases the
-  /// data loss of the target process, given that core traces don't filter by
-  /// process.
-  /// A possible way to avoid this is to use CR3 filtering, which is equivalent
-  /// to process filtering, but the perf_event API doesn't support it.
-  ///
-  /// \param[in] state
-  ///     The new state of the target process.
-  virtual void OnProcessStateChanged(lldb::StateType state){};
+  virtual void ProcessDidStop() {}
+
+  virtual void ProcessWillResume() {}
 
   /// Construct a minimal jLLDBTraceGetState response for this process trace.
   virtual TraceGetStateResponse GetState() = 0;
 
   virtual bool TracesThread(lldb::tid_t tid) const = 0;
 
   /// \copydoc IntelPTThreadTraceCollection::TraceStart()
   virtual llvm::Error TraceStart(lldb::tid_t tid) = 0;

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.h

 namespace process_linux {
 
 llvm::Expected<uint32_t> GetIntelPTOSEventType();
 
 class IntelPTSingleBufferTrace;
 
 using IntelPTSingleBufferTraceUP = std::unique_ptr<IntelPTSingleBufferTrace>;
 
-enum class TraceCollectionState {
-  Running,
-  Paused,
-};
-
 /// This class wraps a single perf event collecting intel pt data in a single
 /// buffer.
 class IntelPTSingleBufferTrace {
 public:
   /// Start tracing using a single Intel PT trace buffer.
   ///
   /// \param[in] request
   ///     Intel PT configuration parameters.
   ///
   /// \param[in] tid
   ///     The tid of the thread to be traced. If \b None, then this traces all
   ///     threads of all processes.
   ///
   /// \param[in] core_id
   ///     The CPU core id where to trace. If \b None, then this traces all CPUs.
   ///
-  /// \param[in] initial_state
-  ///     The initial trace collection state.
+  /// \param[in] disabled
+  ///     Whether to start the tracing paused.
   ///
   /// \return
   ///   A \a IntelPTSingleBufferTrace instance if tracing was successful, or
   ///   an \a llvm::Error otherwise.
   static llvm::Expected<IntelPTSingleBufferTraceUP>
   Start(const TraceIntelPTStartRequest &request,
         llvm::Optional<lldb::tid_t> tid,
-        llvm::Optional<lldb::core_id_t> core_id,
-        TraceCollectionState initial_state);
+        llvm::Optional<lldb::core_id_t> core_id = llvm::None,
+        bool disabled = false);
 
   /// \return
   ///    The bytes requested by a jLLDBTraceGetBinaryData packet that was routed
   ///    to this trace instace.
   llvm::Expected<std::vector<uint8_t>>
   GetBinaryData(const TraceGetBinaryDataRequest &request) const;
 
   /// Read the trace buffer managed by this trace instance. To ensure that the
public:
   llvm::Expected<std::vector<uint8_t>> GetTraceBuffer(size_t offset,
                                                       size_t size);
 
   /// \return
   ///     The total the size in bytes used by the trace buffer managed by this
   ///     trace instance.
   size_t GetTraceBufferSize() const;
 
-  /// Change the collection state for this trace.
+  /// Resume the collection of this trace.
   ///
-  /// This is a no-op if \p state is the same as the current state.
-  ///
-  /// \param[in] state
-  ///     The new state.
+  /// \return
+  ///     An error if the trace couldn't be resumed. If the trace is already
+  ///     running, this returns \a Error::success().
+  llvm::Error Resume();
+
+  /// Pause the collection of this trace.
   ///
   /// \return
-  ///     An error if the state couldn't be changed.
-  llvm::Error ChangeCollectionState(TraceCollectionState state);
+  ///     An error if the trace couldn't be paused. If the trace is already
+  ///     paused, this returns \a Error::success().
+  llvm::Error Pause();
+
+  /// \return
+  ///     The underlying PerfEvent for this trace.
+  const PerfEvent &GetPerfEvent() const;
 
 private:
   /// Construct new \a IntelPTSingleBufferThreadTrace. Users are supposed to
   /// create instances of this class via the \a Start() method and not invoke
   /// this one directly.
   ///
   /// \param[in] perf_event
   ///   perf event configured for IntelPT.
   ///
   /// \param[in] collection_state
   ///   The initial collection state for the provided perf_event.
-  IntelPTSingleBufferTrace(PerfEvent &&perf_event,
-                           TraceCollectionState collection_state)
-      : m_perf_event(std::move(perf_event)),
-        m_collection_state(collection_state) {}
+  IntelPTSingleBufferTrace(PerfEvent &&perf_event)
+      : m_perf_event(std::move(perf_event)) {}
 
   /// perf event configured for IntelPT.
   PerfEvent m_perf_event;
-
-  /// The initial state is stopped because tracing can only start when the
-  /// process is paused.
-  TraceCollectionState m_collection_state;
 };
 
 } // namespace process_linux
 } // namespace lldb_private
 
 #endif // liblldb_IntelPTSingleBufferTrace_H_

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.cpp

CreateIntelPTPerfEventConfiguration(bool enable_tsc,
   return attr;
 }
 #endif
 
 size_t IntelPTSingleBufferTrace::GetTraceBufferSize() const {
   return m_perf_event.GetAuxBuffer().size();
 }
 
-Error IntelPTSingleBufferTrace::ChangeCollectionState(
-    TraceCollectionState new_state) {
-  if (new_state == m_collection_state)
-    return Error::success();
-
-  switch (new_state) {
-  case TraceCollectionState::Paused:
-    if (Error err = m_perf_event.DisableWithIoctl())
-      return err;
-    break;
-  case TraceCollectionState::Running:
-    if (Error err = m_perf_event.EnableWithIoctl())
-      return err;
-    break;
-  }
-  m_collection_state = new_state;
-  return Error::success();
+Error IntelPTSingleBufferTrace::Pause() {
+  return m_perf_event.DisableWithIoctl();
+
+Error IntelPTSingleBufferTrace::Resume() {
+  return m_perf_event.EnableWithIoctl();
 }
 
 Expected<std::vector<uint8_t>>
 IntelPTSingleBufferTrace::GetTraceBuffer(size_t offset, size_t size) {
   // Disable the perf event to force a flush out of the CPU's internal buffer.
   // Besides, we can guarantee that the CPU won't override any data as we are
   // reading the buffer.
   // The Intel documentation says:
   //
   // Packets are first buffered internally and then written out
   // asynchronously. To collect packet output for postprocessing, a collector
   // needs first to ensure that all packet data has been flushed from internal
   // buffers. Software can ensure this by stopping packet generation by
   // clearing IA32_RTIT_CTL.TraceEn (see “Disabling Packet Generation” in
   // Section 35.2.7.2).
   //
   // This is achieved by the PERF_EVENT_IOC_DISABLE ioctl request, as
   // mentioned in the man page of perf_event_open.
-  TraceCollectionState previous_state = m_collection_state;
-  if (Error err = ChangeCollectionState(TraceCollectionState::Paused))
-    return std::move(err);
-
-  std::vector<uint8_t> data(size, 0);
-  perf_event_mmap_page &mmap_metadata = m_perf_event.GetMetadataPage();
-  Log *log = GetLog(POSIXLog::Trace);
-  uint64_t head = mmap_metadata.aux_head;
-
-  LLDB_LOG(log, "Aux size -{0} , Head - {1}", mmap_metadata.aux_size, head);
+  return m_perf_event.ReadFlushedOutAuxCyclicBuffer(offset, size);
+}
 
-  /**
-   * When configured as ring buffer, the aux buffer keeps wrapping around
-   * the buffer and its not possible to detect how many times the buffer
-   * wrapped. Initially the buffer is filled with zeros,as shown below
-   * so in order to get complete buffer we first copy firstpartsize, followed
-   * by any left over part from beginning to aux_head
-   *
-   * aux_offset [d,d,d,d,d,d,d,d,0,0,0,0,0,0,0,0,0,0,0] aux_size
-   *                 aux_head->||<- firstpartsize  ->|
-   *
-   * */
-
-  MutableArrayRef<uint8_t> buffer(data);
-  ReadCyclicBuffer(buffer, m_perf_event.GetAuxBuffer(),
-                   static_cast<size_t>(head), offset);
-
-  if (Error err = ChangeCollectionState(previous_state))
-    return std::move(err);
-
-  return data;
-}
-
-Expected<IntelPTSingleBufferTraceUP> IntelPTSingleBufferTrace::Start(
-    const TraceIntelPTStartRequest &request, Optional<lldb::tid_t> tid,
-    Optional<core_id_t> core_id, TraceCollectionState initial_state) {
+Expected<IntelPTSingleBufferTraceUP>
+IntelPTSingleBufferTrace::Start(const TraceIntelPTStartRequest &request,
+                                Optional<lldb::tid_t> tid,
+                                Optional<core_id_t> core_id, bool disabled) {
 #ifndef PERF_ATTR_SIZE_VER5
   return createStringError(inconvertibleErrorCode(),
                            "Intel PT Linux perf event not supported");
 #else
   Log *log = GetLog(POSIXLog::Trace);
 
   LLDB_LOG(log, "Will start tracing thread id {0} and cpu id {1}", tid,
            core_id);
uint64_t aux_buffer_numpages = static_cast<uint64_t>(llvm::PowerOf2Floor(
       (request.trace_buffer_size + page_size - 1) / page_size));
 
   Expected<perf_event_attr> attr = CreateIntelPTPerfEventConfiguration(
       request.enable_tsc, request.psb_period.map([](int value) {
         return static_cast<uint64_t>(value);
       }));
   if (!attr)
     return attr.takeError();
-  attr->disabled = initial_state == TraceCollectionState::Paused;
+  attr->disabled = disabled;
 
   LLDB_LOG(log, "Will create trace buffer of size {0}",
            request.trace_buffer_size);
 
   if (Expected<PerfEvent> perf_event = PerfEvent::Init(*attr, tid, core_id)) {
     if (Error mmap_err = perf_event->MmapMetadataAndBuffers(
             /*num_data_pages=*/0, aux_buffer_numpages)) {
       return std::move(mmap_err);
     }
     IntelPTSingleBufferTraceUP trace_up(
-        new IntelPTSingleBufferTrace(std::move(*perf_event), initial_state));
+        new IntelPTSingleBufferTrace(std::move(*perf_event)));
     return trace_up;
   } else {
     return perf_event.takeError();
   }
 #endif
 }
+
+const PerfEvent &IntelPTSingleBufferTrace::GetPerfEvent() const {
+  return m_perf_event;
+}

lldb/source/Plugins/Process/Linux/IntelPTThreadTraceCollection.cpp

 
 Error IntelPTThreadTraceCollection::TraceStart(
     lldb::tid_t tid, const TraceIntelPTStartRequest &request) {
   if (TracesThread(tid))
     return createStringError(inconvertibleErrorCode(),
                              "Thread %" PRIu64 " already traced", tid);
 
   Expected<IntelPTSingleBufferTraceUP> trace_up =
-      IntelPTSingleBufferTrace::Start(request, tid, /*core_id=*/None,
-                                      TraceCollectionState::Running);
+      IntelPTSingleBufferTrace::Start(request, tid);
   if (!trace_up)
     return trace_up.takeError();
 
   m_total_buffer_size += (*trace_up)->GetTraceBufferSize();
   m_thread_traces.try_emplace(tid, std::move(*trace_up));
   return Error::success();
 }
 

lldb/source/Plugins/Process/Linux/NativeProcessLinux.h

private:
   Status NotifyTracersOfNewThread(lldb::tid_t tid);
 
   /// Stop tracing threads upon a destroy event.
   ///
   /// Trace mechanisms should modify this method to provide automatic trace
   /// stopping for threads being destroyed.
   Status NotifyTracersOfThreadDestroyed(lldb::tid_t tid);
 
-  void NotifyTracersProcessStateChanged(lldb::StateType state) override;
+  void NotifyTracersProcessWillResume() override;
+
+  void NotifyTracersProcessDidStop() override;
 
   /// Writes the raw event message code (vis-a-vis PTRACE_GETEVENTMSG)
   /// corresponding to the given thread ID to the memory pointed to by @p
   /// message.
   Status GetEventMessage(lldb::tid_t tid, unsigned long *message);
 
   void NotifyThreadDeath(lldb::tid_t tid);
 

lldb/source/Plugins/Process/Linux/NativeProcessLinux.cpp

if (m_arch.GetMachine() == llvm::Triple::arm || m_arch.IsMIPS())
     return false;
   return true;
 }
 
 Status NativeProcessLinux::Resume(const ResumeActionList &resume_actions) {
   Log *log = GetLog(POSIXLog::Process);
   LLDB_LOG(log, "pid {0}", GetID());
 
+  NotifyTracersProcessWillResume();
+
   bool software_single_step = !SupportHardwareSingleStepping();
 
   if (software_single_step) {
     for (const auto &thread : m_threads) {
       assert(thread && "thread list should not contain NULL threads");
 
       const ResumeAction *const action =
           resume_actions.GetActionForThread(thread->GetID(), true);
void NativeProcessLinux::StopTrackingThread(NativeThreadLinux &thread) {
   });
   assert(it != m_threads.end());
   m_threads.erase(it);
 
   NotifyTracersOfThreadDestroyed(thread_id);
   SignalIfAllThreadsStopped();
 }
 
-void NativeProcessLinux::NotifyTracersProcessStateChanged(
-    lldb::StateType state) {
-  m_intel_pt_collector.OnProcessStateChanged(state);
+void NativeProcessLinux::NotifyTracersProcessDidStop() {
+  m_intel_pt_collector.ProcessDidStop();
+}
+
+void NativeProcessLinux::NotifyTracersProcessWillResume() {
+  m_intel_pt_collector.ProcessWillResume();
 }
 
 Status NativeProcessLinux::NotifyTracersOfNewThread(lldb::tid_t tid) {
   Log *log = GetLog(POSIXLog::Thread);
   Status error(m_intel_pt_collector.OnThreadCreated(tid));
   if (error.Fail())
     LLDB_LOG(log, "Failed to trace a new thread with intel-pt, tid = {0}. {1}",
              tid, error.AsCString());

lldb/source/Plugins/Process/Linux/Perf.h

void ReadCyclicBuffer(llvm::MutableArrayRef<uint8_t> &dst,
                       size_t offset);
 
 /// Thin wrapper of the perf_event_open API.
 ///
 /// Exposes the metadata page and data and aux buffers of a perf event.
 /// Handles the management of the event's file descriptor and mmap'ed
 /// regions.
 class PerfEvent {
+  enum class CollectionState {
+    Enabled,
+    Disabled,
+  };
+
 public:
   /// Create a new performance monitoring event via the perf_event_open syscall.
   ///
   /// The parameters are directly forwarded to a perf_event_open syscall,
   /// for additional information on the parameters visit
   /// https://man7.org/linux/man-pages/man2/perf_event_open.2.html.
   ///
   /// \param[in] attr
public:
   ///
   /// This should be called only after \a PerfEvent::MmapMetadataAndBuffers,
   /// otherwise a failure might happen.
   ///
   /// \return
   ///   \a ArrayRef<uint8_t> extending \a aux_size bytes from \a aux_offset.
   llvm::ArrayRef<uint8_t> GetAuxBuffer() const;
 
-  /// Use the ioctl API to disable the perf event. This doesn't terminate the
-  /// perf event.
+  /// Read the aux buffer managed by this perf event. To ensure that the
+  /// data is up-to-date and is not corrupted by read-write race conditions, the
+  /// underlying perf_event is paused during read, and later it's returned to
+  /// its initial state. The returned data will be linear, i.e. it will fix the
+  /// circular wrapping the might exist int he buffer.
+  ///
+  /// \param[in] offset
+  ///     Offset of the data to read.
+  ///
+  /// \param[in] size
+  ///     Number of bytes to read.
+  ///
+  /// \return
+  ///     A vector with the requested binary data. The vector will have the
+  ///     size of the requested \a size. Non-available positions will be
+  ///     filled with zeroes.
+  llvm::Expected<std::vector<uint8_t>>
+  ReadFlushedOutAuxCyclicBuffer(size_t offset, size_t size);
+
+  /// Use the ioctl API to disable the perf event and all the events in its
+  /// group. This doesn't terminate the perf event.
+  ///
+  /// This is no-op if the perf event is already disabled.
   ///
   /// \return
   ///   An Error if the perf event couldn't be disabled.
-  llvm::Error DisableWithIoctl() const;
+  llvm::Error DisableWithIoctl();
 
-  /// Use the ioctl API to enable the perf event.
+  /// Use the ioctl API to enable the perf event and all the events in its
+  /// group.
+  ///
+  /// This is no-op if the perf event is already enabled.
   ///
   /// \return
   ///   An Error if the perf event couldn't be enabled.
-  llvm::Error EnableWithIoctl() const;
+  llvm::Error EnableWithIoctl();
+
+  /// \return
+  ///   The size in bytes of the section of the data buffer that has effective
+  ///   data.
+  size_t GetEffectiveDataBufferSize() const;
 
 private:
   /// Create new \a PerfEvent.
   ///
   /// \param[in] fd
   ///   File descriptor of the perf event.
-  PerfEvent(long fd)
+  ///
+  /// \param[in] initial_state
+  ///   Initial collection state configured for this perf_event.
+  PerfEvent(long fd, CollectionState initial_state)
       : m_fd(new long(fd), resource_handle::FileDescriptorDeleter()),
-        m_metadata_data_base(), m_aux_base() {}
+        m_collection_state(initial_state) {}
 
   /// Wrapper for \a mmap to provide custom error messages.
   ///
   /// The parameters are directly forwarded to a \a mmap syscall,
   /// for information on the parameters visit
   /// https://man7.org/linux/man-pages/man2/mmap.2.html.
   ///
   /// The value of \a GetFd() is passed as the \a fd argument to \a mmap.
private:
 
   /// The file descriptor representing the perf event.
   resource_handle::FileDescriptorUP m_fd;
   /// Metadata page and data section where perf samples are stored.
   resource_handle::MmapUP m_metadata_data_base;
   /// AUX buffer is a separate region for high-bandwidth data streams
   /// such as IntelPT.
   resource_handle::MmapUP m_aux_base;
+  /// The state of the underlying perf_event.
+  CollectionState m_collection_state;
 };
 
 /// Load \a PerfTscConversionParameters from \a perf_event_mmap_page, if
 /// available.
 llvm::Expected<LinuxPerfZeroTscConversion> LoadPerfTscConversionParameters();
 
 } // namespace process_linux
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_PROCESS_LINUX_PERF_H

lldb/source/Plugins/Process/Linux/Perf.cpp

llvm::Expected<PerfEvent> PerfEvent::Init(perf_event_attr &attr,
   errno = 0;
   long fd = syscall(SYS_perf_event_open, &attr, pid.getValueOr(-1),
                     cpu.getValueOr(-1), group_fd.getValueOr(-1), flags);
   if (fd == -1) {
     std::string err_msg =
         llvm::formatv("perf event syscall failed: {0}", std::strerror(errno));
     return llvm::createStringError(llvm::inconvertibleErrorCode(), err_msg);
   }
-  return PerfEvent{fd};
+  return PerfEvent(fd, attr.disabled ? CollectionState::Disabled
+                                     : CollectionState::Enabled);
 }
 
 llvm::Expected<PerfEvent> PerfEvent::Init(perf_event_attr &attr,
                                           Optional<lldb::pid_t> pid,
                                           Optional<lldb::core_id_t> cpu) {
   return Init(attr, pid, cpu, -1, 0);
 }
 
 llvm::Expected<resource_handle::MmapUP>
 PerfEvent::DoMmap(void *addr, size_t length, int prot, int flags,
                   long int offset, llvm::StringRef buffer_name) {
   errno = 0;
-  auto mmap_result = ::mmap(nullptr, length, prot, flags, GetFd(), offset);
+  auto mmap_result = ::mmap(addr, length, prot, flags, GetFd(), offset);
 
   if (mmap_result == MAP_FAILED) {
     std::string err_msg =
         llvm::formatv("perf event mmap allocation failed for {0}: {1}",
                       buffer_name, std::strerror(errno));
     return createStringError(inconvertibleErrorCode(), err_msg);
   }
   return resource_handle::MmapUP(mmap_result, length);
 }
 
 llvm::Error PerfEvent::MmapMetadataAndDataBuffer(size_t num_data_pages) {
   size_t mmap_size = (num_data_pages + 1) * getpagesize();
   if (Expected<resource_handle::MmapUP> mmap_metadata_data =
-          DoMmap(nullptr, mmap_size, PROT_WRITE, MAP_SHARED, 0,
+          DoMmap(nullptr, mmap_size, PROT_READ | PROT_WRITE, MAP_SHARED, 0,
                  "metadata and data buffer")) {
     m_metadata_data_base = std::move(mmap_metadata_data.get());
     return Error::success();
   } else
     return mmap_metadata_data.takeError();
 }
 
 llvm::Error PerfEvent::MmapAuxBuffer(size_t num_aux_pages) {

 }
 
 ArrayRef<uint8_t> PerfEvent::GetAuxBuffer() const {
   perf_event_mmap_page &mmap_metadata = GetMetadataPage();
   return {reinterpret_cast<uint8_t *>(m_aux_base.get()),
            static_cast<size_t>(mmap_metadata.aux_size)};
 }
 
-Error PerfEvent::DisableWithIoctl() const {
-  if (ioctl(*m_fd, PERF_EVENT_IOC_DISABLE) < 0)
+Expected<std::vector<uint8_t>>
+PerfEvent::ReadFlushedOutAuxCyclicBuffer(size_t offset, size_t size) {
+  CollectionState previous_state = m_collection_state;
+  if (Error err = DisableWithIoctl())
+    return std::move(err);
+
+  std::vector<uint8_t> data(size, 0);
+  perf_event_mmap_page &mmap_metadata = GetMetadataPage();
+  Log *log = GetLog(POSIXLog::Trace);
+  uint64_t head = mmap_metadata.aux_head;
+
+  LLDB_LOG(log, "Aux size -{0} , Head - {1}", mmap_metadata.aux_size, head);
+
+  /**
+   * When configured as ring buffer, the aux buffer keeps wrapping around
+   * the buffer and its not possible to detect how many times the buffer
+   * wrapped. Initially the buffer is filled with zeros,as shown below
+   * so in order to get complete buffer we first copy firstpartsize, followed
+   * by any left over part from beginning to aux_head
+   *
+   * aux_offset [d,d,d,d,d,d,d,d,0,0,0,0,0,0,0,0,0,0,0] aux_size
+   *                 aux_head->||<- firstpartsize  ->|
+   *
+   * */
+
+  MutableArrayRef<uint8_t> buffer(data);
+  ReadCyclicBuffer(buffer, GetAuxBuffer(), static_cast<size_t>(head), offset);
+
+  if (previous_state == CollectionState::Enabled) {
+    if (Error err = EnableWithIoctl())
+      return std::move(err);
+  }
+
+  return data;
+}
+
+Error PerfEvent::DisableWithIoctl() {
+  if (m_collection_state == CollectionState::Disabled)
+    return Error::success();
+
+  if (ioctl(*m_fd, PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP) < 0)
     return createStringError(inconvertibleErrorCode(),
                              "Can't disable perf event. %s",
                              std::strerror(errno));
+
+  m_collection_state = CollectionState::Disabled;
   return Error::success();
 }
 
-Error PerfEvent::EnableWithIoctl() const {
-  if (ioctl(*m_fd, PERF_EVENT_IOC_ENABLE) < 0)
+Error PerfEvent::EnableWithIoctl() {
+  if (m_collection_state == CollectionState::Enabled)
+    return Error::success();
+
+  if (ioctl(*m_fd, PERF_EVENT_IOC_ENABLE, PERF_IOC_FLAG_GROUP) < 0)
     return createStringError(inconvertibleErrorCode(),
-                             "Can't disable perf event. %s",
+                             "Can't enable perf event. %s",
                              std::strerror(errno));
+
+  m_collection_state = CollectionState::Enabled;
   return Error::success();
 }
+
+size_t PerfEvent::GetEffectiveDataBufferSize() const {
+  perf_event_mmap_page &mmap_metadata = GetMetadataPage();
+  if (mmap_metadata.data_head < mmap_metadata.data_size)
+    return mmap_metadata.data_head;
+  else
+    return mmap_metadata.data_size; // The buffer has wrapped.
+}

lldb/source/Utility/TraceIntelPTGDBRemotePackets.cpp

 
 using namespace llvm;
 using namespace llvm::json;
 
 namespace lldb_private {
 
 const char *IntelPTDataKinds::kProcFsCpuInfo = "procfsCpuInfo";
 const char *IntelPTDataKinds::kTraceBuffer = "traceBuffer";
+const char *IntelPTDataKinds::kPerfContextSwitchTrace =
+    "perfContextSwitchTrace";
 
 bool TraceIntelPTStartRequest::IsPerCoreTracing() const {
   return per_core_tracing.getValueOr(false);
 }
 
 bool fromJSON(const json::Value &value, TraceIntelPTStartRequest &packet,
               Path path) {
   ObjectMapper o(value, path);

lldb/test/API/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.py

 import lldb
+import json
 from intelpt_testcase import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 from lldbsuite.test.decorators import *
 
 class TestTraceStartStopMultipleThreads(TraceIntelPTTestCaseBase):
 
     mydir = TestBase.compute_mydir(__file__)
def testStartPerCoreSession(self):
         self.traceStartThread(
             error="True",
             substrs=["Thread with tid ", "is currently traced"])
 
         # We can't stop individual thread when per core is enabled.
         self.traceStopThread(error="True",
             substrs=["Can't stop tracing an individual thread when per-core process tracing is enabled"])
 
-        # The GetState packet should return trace buffers per core and at least one traced thread
-        self.expect("""process plugin packet send 'jLLDBTraceGetState:{"type":"intel-pt"}]'""",
-            substrs=['''[{"kind":"traceBuffer","size":4096}],"coreId":''', '"tid":'])
+        # We move forward a little bit to collect some data
+        self.expect("b 19")
+        self.expect("c")
+
+        # We will assert that the trace state will contain valid context switch and trace buffer entries
+
+        # We first parse the json response from the custom packet
+        self.runCmd("""process plugin packet send 'jLLDBTraceGetState:{"type":"intel-pt"}]'""")
+        response_header = 'response: '
+        output = None
+        for line in self.res.GetOutput().splitlines():
+            if line.find(response_header) != -1:
+                response = line[line.find(response_header) + len(response_header):].strip()
+                output = json.loads(response)
+
+        self.assertTrue(output is not None)
+        self.assertIn("cores", output)
+        found_non_empty_context_switch = False
+
+        for core in output["cores"]:
+            context_switch_size = None
+            trace_buffer_size = None
+            for binary_data in core["binaryData"]:
+                if binary_data["kind"] == "traceBuffer":
+                    trace_buffer_size = binary_data["size"]
+                elif binary_data["kind"] == "perfContextSwitchTrace":
+                    context_switch_size = binary_data["size"]
+            self.assertTrue(context_switch_size is not None)
+            self.assertTrue(trace_buffer_size is not None)
+            if context_switch_size > 0:
+                found_non_empty_context_switch = True
+
+        # We must have captured the context switch of when the target resumed
+        self.assertTrue(found_non_empty_context_switch)
+
 
         self.traceStopProcess()