[trace][intelpt] Support system-wide tracing [4] - Support per core tracing on lldb-server

[wallace]

Summary

This diffs implements per-core tracing on lldb-server. It also includes tests that ensure that tracing can be initiated from the client and that the jLLDBGetState ppacket returns the list of trace buffers per core.

This doesn't include any decoder changes.

Finally, this makes some little changes here and there improving the existing code.

A specific piece of code that can't reliably be tested is when tracing
per core fails due to permissions. In this case we add a
troubleshooting message and this is the manual test:

/proc/sys/kernel/perf_event_paranoid set to 1

(lldb) process trace start --per-core-tracing                                         error: perf event syscall failed: Permission denied
 You might need that /proc/sys/kernel/perf_event_paranoid has a value of 0 or -1.
``

[wallace]

nits

[wallace]

add one test

[wallace]

improve error message

[wallace]

lldb/packages/Python/lldbsuite/test/tools/intelpt/intelpt_testcase.py

47

or use sudo

[wallace]

nits

[jj10306]

lldb/docs/lldb-gdb-remote.txt

372–375

Not sure what is trying to be communicated hear, so maybe my suggestion doesn't make sense, but I was confused by the wording here

465–474

Should the thread info be optional now that we have an optional cores? If not, can you explain how this output works in the case that you are doing per core tracing? Will both the tid binary data and the cores binary data section be populated?

lldb/include/lldb/Utility/TraceGDBRemotePackets.h

157–160

Similar question here as above on the GetState documentation - how do the cores and traced_threads options play together?

lldb/include/lldb/lldb-types.h

92

nit: this should be an unsigned int of some sort?

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

228–238

Do we not need to check if m_per_thread_process_trace_up is null here (aka per core tracing is enabled)?

263

Should this be an "else if" if these two are mutually exclusive?

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

64–65

Why is this named "PerThread" if it handles both the per thread and per core cases for "process trace"? Perhaps I'm missing something

152–156

So these are mutually exclusive tight?

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

21–23

nit: why are we using signed integers here?

lldb/source/Plugins/Process/Linux/Perf.h

111–117

Maybe add explanation of what is passed to the syscall for pid and cpu when they are None here?

[wallace]

lldb/docs/lldb-gdb-remote.txt

372–375

you are right!

465–474

I'll make this field optional and include more documentation below

lldb/include/lldb/lldb-types.h

92

I wasn't sure if int would be correct for all platforms, but in any case, I should go for unsigned, as you suggest, which is a more restrictive type, and if in the future someone needs a signed int, they could change this type

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

228–238

I need to check that m_per_thread_process_trace_up exists

263

good idea

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

64–65

this doesn't handle the per core case. This is handling exclusively the case of "process trace" without the "per-cpu" option. This class effectively creates a perf event per thread. Even its Start method asks for tids

152–156

yes, I'll leave a comment about that here

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

21–23

yes, this is bad, i'm changing to unsigned

lldb/source/Plugins/Process/Linux/Perf.h

111–117

makes sense

[jj10306]

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

228–238

Yes, I meant "is *not* null here", my bad (-:

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

64–65

ahhh yes I got messed up by the diff view and thought that the fields of IntelPTCollector were being stored on this per thread structure, thanks!

[wallace]

address comments

[wallace]

Now the tracedThreads field returned by the GetState request must have the list of all the threads of the process if per-core mode is enabled. This will avoid having to make that field optional.

[wallace]

lldb/docs/lldb-gdb-remote.txt

248

this should be "Embedded Trace Macrocells" (ETM) instead of coresight

465–474

in the end i didn't make this optional but instead forced the per-core case to return all threads

lldb/include/lldb/Utility/TraceGDBRemotePackets.h

157–160

traced_threads will contain all the threads in the per core case but without any binary data. The cores field will then be returned with per-core trace buffers

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.cpp

149

i'm the location of this ifdef so that perf_event_attr is used only if PERF_ATTR_SIZE_VER5 is set

[jj10306]

couple minor things, but looks good overall

lldb/docs/lldb-gdb-remote.txt

465–474

so the tid field of a tracedThread object will be populated but the binaryData will not be populated in the tracedThread object but instead in the cores section?

lldb/source/Plugins/Process/Linux/Perf.h

119

nit: we could make this optional as well and just default to passing in -1 to be consistent with cpu and pid

lldb/source/Utility/TraceGDBRemotePackets.cpp

124

isn't uint64_t serialization and deserialization now supported?

[jj10306]

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

152–156

this could be overkill, but potentially we could use a union here to enforce this invariant and a boolean flag/enum to determine which process tracing "handler" is being used

bool is_per_core_process_tracing_enabled; // used to determine how to interpret the union
union {
  IntelPTPerThreadProcessTraceUP m_per_thread_process_trace_up;
  IntelPTMultiCoreTraceUP m_per_core_process_trace_up;

};

imo this is much more clearly expresses the invariant of only one of the process tracing "handles" being non null at a time.
wdyt?

[wallace]

lldb/docs/lldb-gdb-remote.txt

465–474

yes, exactly

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

152–156

I tried this, but using unions with complex types requires some non-trivial work. std::variant would be a good choice but is not available in the c++ version used by llvm. So in this case I think the current implementation is the simpler. However, I'd rather create an interface and have one single instance based on that interface. I'll try that

lldb/source/Plugins/Process/Linux/Perf.h

119

good idea

lldb/source/Utility/TraceGDBRemotePackets.cpp

124

good catch

[wallace]

address comments

In a newer diff I'll create a base class for all "process tracing" strategies, so that we don't need to use two unique pointers for both process tracing classes in the collector. I'm not doing it here becaues rebasing would be a pain.

lldb/docs/lldb-gdb-remote.txt

 // BRIEF
 //  Get the processor tracing type supported by the gdb-server for the current
 //  inferior. Responses might be different depending on the architecture and
 //  capabilities of the underlying OS.
 //
 //  OUTPUT SCHEMA
 //   {
 //     "name": <string>,
-//         Tracing technology name, e.g. intel-pt, arm-coresight.
+//         Tracing technology name, e.g. intel-pt, arm-etm.
 //     "description": <string>,
 //         Description for this technology.
 //   }
 //
 //   If no tracing technology is supported for the inferior, or no process is
 //   running, then an error message is returned.
 //
 // NOTE

 //  returned if the process is already being traced.
 //
 // THREAD TRACING
 //  This traces specific threads.
 //
 // INPUT SCHEMA
 //  {
 //    "type": <string>,
-//        Tracing technology name, e.g. intel-pt, arm-coresight.
+//        Tracing technology name, e.g. intel-pt, arm-etm.
 //
 //    /* thread tracing only */
 //    "tids"?: [<decimal integer>],
 //        Individual threads to trace.
 //
 //    ... other parameters specific to the provided tracing type
 //  }
 //

 //         This option forces the capture of TSC timestamps (see "enableTsc").
 //
 //         Note: This option can't be used simulatenously with any other trace
 //         sessions because of its system-wide nature.
 //
 //     /* process tracing only */
 //     "processBufferSizeLimit": <decimal integer>,
 //         Maximum total buffer size per process in bytes.
-//         This limit applies to the sum of the sizes of all trace buffers for
-//         the current process, excluding the ones started with "thread tracing".
+//         This limit applies to the sum of the sizes of all thread or core
+//         buffers for the current process, excluding the ones started with
+//         "thread tracing".
 //
 //         If "perCoreTracing" is false, whenever a thread is attempted to be
 //         traced due to "process tracing" and the limit would be reached, the
 //         process is stopped with a "tracing" reason along with a meaningful
 //         description, so that the user can retrace the process if needed.
 //
 //         If "perCoreTracing" is true, then starting the system-wide trace
 //         session fails if all the individual per-core trace buffers require
 //         in total more memory that the limit impossed by this parameter.

 //  This is a best effort request, which tries to stop as many traces as
 //  possible.
 //
 // INPUT SCHEMA
 //  The schema for the input is
 //
 //  {
 //    "type": <string>
-//       Tracing technology name, e.g. intel-pt, arm-coresight.
+//       Tracing technology name, e.g. intel-pt, arm-etm.
 //
 //    /* thread trace stopping only */
 //    "tids":  [<decimal integer>]
 //       Individual thread traces to stop.
 //  }
 //
 // NOTES
 //  - If "tids" is not provided, then the operation is "process trace stopping".

 //  Get the current state of the process and its threads being traced by
 //  a given trace technology. The response is a JSON object with custom
 //  information depending on the trace technology. In case of errors, an
 //  error message is returned.
 //
 // INPUT SCHEMA
 //  {
 //     "type": <string>
-//        Tracing technology name, e.g. intel-pt, arm-coresight.
+//        Tracing technology name, e.g. intel-pt, arm-etm.
 //  }
 //
 // OUTPUT SCHEMA
 //  {
 //    "tracedThreads": [{
 //      "tid": <decimal integer>,
 //      "binaryData": [
 //        {
 //          "kind": <string>,
 //              Identifier for some binary data related to this thread to
 //              fetch with the jLLDBTraceGetBinaryData packet.
 //          "size": <decimal integer>,
 //              Size in bytes of this thread data.
 //        },
 //      ]
 //    }],
 //    "processBinaryData": [
 //      {
 //        "kind": <string>,
 //            Identifier for some binary data related to this process to
 //            fetch with the jLLDBTraceGetBinaryData packet.
 //        "size": <decimal integer>,
 //            Size in bytes of this thread data.
 //      },
 //    ],
+//    "cores"?: [
+//      "id": <decimal integer>,
+//          Identifier for this CPU logical core.
+//      "binaryData": [
+//        {
+//          "kind": <string>,
+//              Identifier for some binary data related to this thread to
+//              fetch with the jLLDBTraceGetBinaryData packet.
+//          "size": <decimal integer>,
+//              Size in bytes of this cpu core data.
+//        },
+//      ]
+//    ],
 //    "counters"?: {
 //      "info_kind": {...parameters specific to the provided counter info kind},
-//          Each entry includes information related to counters associated with the trace.
-//          They are described below.
+//          Each entry includes information related to counters associated with
+//          the trace. They are described below.
 //    }
 //  }
 //
 // NOTES
 //   - "traceThreads" includes all thread traced by both "process tracing" and
 //     "thread tracing".
 //
 // INTEL PT
 //
+//  If per-core process tracing is enabled, "tracedThreads" will contain all
+//  the threads of the process without any trace buffers. Besides that, the
+//  "cores" field will also be returned with per core trace buffers.
+//  A side effect of per-core tracing is that all the threads of unrelated
+//  processes will also be traced, thus polluting the tracing data.
+//
 //  Binary data kinds:
 //    - traceBuffer: trace buffer for a thread or a core.
 //    - procfsCpuInfo: contents of the /proc/cpuinfo file.
 //
 //  Counter info kinds:
 //    tsc-perf-zero-conversion:
 //
 //    This field allows converting Intel processor's TSC values to a wall time.

 //  as the "binary memory read" (aka "x") packet. In case of failures, an error
 //  message is returned.
 //
 // SCHEMA
 //  The schema for the input is
 //
 //  {
 //   "type": <string>,
-//       Tracing technology name, e.g. intel-pt, arm-coresight.
+//       Tracing technology name, e.g. intel-pt, arm-etm.
 //   "kind": <string>,
 //       Identifier for the data.
+//   "coreId": <Optional decimal>,
+//       Core id in decimal if the data belongs to a CPU core.
 //   "tid"?: <Optional decimal>,
 //       Tid in decimal if the data belongs to a thread.
 //   "offset": <decimal>,
 //       Offset of the data in bytes.
 //   "size": <decimal>,
 //      Number of bytes in to read starting from the offset.
 //  }
 //

lldb/include/lldb/Utility/TraceGDBRemotePackets.h

 bool fromJSON(const llvm::json::Value &value, TraceBinaryData &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceBinaryData &packet);
 
 struct TraceThreadState {
   lldb::tid_t tid;
   /// List of binary data objects for this thread.
-  std::vector<TraceBinaryData> binaryData;
+  std::vector<TraceBinaryData> binary_data;
 };
 
 bool fromJSON(const llvm::json::Value &value, TraceThreadState &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceThreadState &packet);
 
+struct TraceCoreState {
+  lldb::core_id_t core_id;
+  /// List of binary data objects for this core.
+  std::vector<TraceBinaryData> binary_data;
+};
+
+bool fromJSON(const llvm::json::Value &value, TraceCoreState &packet,
+              llvm::json::Path path);
+
+llvm::json::Value toJSON(const TraceCoreState &packet);
+
 /// Interface for different algorithms used to convert trace
 /// counters into different units.
 template <typename ToType> class TraceCounterConversion {
 public:
   virtual ~TraceCounterConversion() = default;
 
   /// Convert from raw counter value to the target type.
   ///
public:
   ///   \a llvm::json::Value representing the trace counter conversion object.
   virtual llvm::json::Value toJSON() = 0;
 };
 
 using TraceTscConversionUP =
     std::unique_ptr<TraceCounterConversion<std::chrono::nanoseconds>>;
 
 struct TraceGetStateResponse {
-  std::vector<TraceThreadState> tracedThreads;
-  std::vector<TraceBinaryData> processBinaryData;
+  std::vector<TraceThreadState> traced_threads;
+  std::vector<TraceBinaryData> process_binary_data;
+  llvm::Optional<std::vector<TraceCoreState>> cores;
 };
 
 bool fromJSON(const llvm::json::Value &value, TraceGetStateResponse &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceGetStateResponse &packet);
 /// \}
 
 /// jLLDBTraceGetBinaryData gdb-remote packet

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

struct TraceIntelPTStartRequest : TraceStartRequest {
   ///
   /// Limit in bytes on all the thread traces started by this "process trace"
   /// instance. When a thread is about to be traced and the limit would be hit,
   /// then a "tracing" stop event is triggered.
   llvm::Optional<uint64_t> process_buffer_size_limit;
 
   /// Whether to have a trace buffer per thread or per cpu core.
   llvm::Optional<bool> per_core_tracing;
+
+  bool IsPerCoreTracing() const;
 };
 
 bool fromJSON(const llvm::json::Value &value, TraceIntelPTStartRequest &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceIntelPTStartRequest &packet);
 /// \}
 

lldb/include/lldb/lldb-types.h

 typedef uint64_t user_id_t;
 typedef uint64_t pid_t;
 typedef uint64_t tid_t;
 typedef uint64_t offset_t;
 typedef int32_t break_id_t;
 typedef int32_t watch_id_t;
 typedef void *opaque_compiler_type_t;
 typedef uint64_t queue_id_t;
+typedef uint32_t core_id_t; // CPU core id
 } // namespace lldb
 
 #endif // LLDB_LLDB_TYPES_H

lldb/packages/Python/lldbsuite/test/tools/intelpt/intelpt_testcase.py

class TraceIntelPTTestCaseBase(TestBase):
     # If True, the trace test methods will use the SB API, otherwise they'll use raw commands.
     USE_SB_API = False
 
     def setUp(self):
         TestBase.setUp(self)
         if 'intel-pt' not in configuration.enabled_plugins:
             self.skipTest("The intel-pt test plugin is not enabled")
 
+    def skipIfPerCoreTracingIsNotSupported(self):
+        def is_supported():
+            try:
+                with open("/proc/sys/kernel/perf_event_paranoid", "r") as permissions:
+                    value = int(permissions.readlines()[0])
+                    if value <= 0:
+                        return True
+            except:
+                return False
+        if not is_supported():
+            self.skipTest("Per core tracing is not supported. You need "
+                "/proc/sys/kernel/perf_event_paranoid to be 0 or -1.")
+
     def getTraceOrCreate(self):
         if not self.target().GetTrace().IsValid():
             error = lldb.SBError()
             self.target().CreateTrace(error)
         return self.target().GetTrace()
 
     def assertSBError(self, sberror, error=False):
         if error:
def traceStartProcess(self, processBufferSizeLimit=None, error=False,
             self.expect(command, error=error, substrs=substrs)
 
     def traceStopProcess(self):
         if self.USE_SB_API:
             self.assertSuccess(self.target().GetTrace().Stop())
         else:
             self.expect("process trace stop")
 
-    def traceStopThread(self, thread=None, error=False):
+    def traceStopThread(self, thread=None, error=False, substrs=None):
         if self.USE_SB_API:
             thread = thread if thread is not None else self.thread()
             self.assertSBError(self.target().GetTrace().Stop(thread), error)
 
         else:
             command = "thread trace stop"
             if thread is not None:
                 command += " " + str(thread.GetIndexID())
-            self.expect(command, error=error)
+            self.expect(command, error=error, substrs=substrs)

lldb/source/Plugins/Process/Linux/CMakeLists.txt

 add_lldb_library(lldbPluginProcessLinux
   IntelPTCollector.cpp
   IntelPTSingleBufferTrace.cpp
+  IntelPTMultiCoreTrace.cpp
   NativeProcessLinux.cpp
   NativeRegisterContextLinux.cpp
   NativeRegisterContextLinux_arm.cpp
   NativeRegisterContextLinux_arm64.cpp
   NativeRegisterContextLinux_ppc64le.cpp
   NativeRegisterContextLinux_s390x.cpp
   NativeRegisterContextLinux_x86_64.cpp
   NativeThreadLinux.cpp

lldb/source/Plugins/Process/Linux/IntelPTCollector.h

-//===-- IntelPTCollector.h -------------------------------------- -*- C++ -*-===//
+//===-- IntelPTCollector.h ------------------------------------ -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_IntelPTCollector_H_
 #define liblldb_IntelPTCollector_H_
 
 #include "Perf.h"
 
+#include "IntelPTMultiCoreTrace.h"
 #include "IntelPTSingleBufferTrace.h"
 
+#include "lldb/Host/common/NativeProcessProtocol.h"
 #include "lldb/Utility/Status.h"
 #include "lldb/Utility/TraceIntelPTGDBRemotePackets.h"
 #include "lldb/lldb-types.h"
 
 #include <linux/perf_event.h>
 #include <sys/mman.h>
 #include <unistd.h>
 
public:
   llvm::Expected<const IntelPTSingleBufferTrace &>
   GetTracedThread(lldb::tid_t tid) const;
 
   llvm::Error TraceStart(lldb::tid_t tid,
                          const TraceIntelPTStartRequest &request);
 
   llvm::Error TraceStop(lldb::tid_t tid);
 
+  size_t GetTracedThreadsCount() const;
+
 private:
   llvm::DenseMap<lldb::tid_t, IntelPTSingleBufferTraceUP> m_thread_traces;
   /// Total actual thread buffer size in bytes
   size_t m_total_buffer_size = 0;
 };
 
-/// Manages a "process trace" instance.
-class IntelPTProcessTrace {
+class IntelPTPerThreadProcessTrace;
+using IntelPTPerThreadProcessTraceUP =
+    std::unique_ptr<IntelPTPerThreadProcessTrace>;
+
+/// Manages a "process trace" instance by tracing each thread individually.
+class IntelPTPerThreadProcessTrace {
 public:
-  IntelPTProcessTrace(const TraceIntelPTStartRequest &request)
-      : m_tracing_params(request) {}
+  /// Start tracing the current process by tracing each of its tids
+  /// individually.
+  ///
+  /// \param[in] request
+  ///   Intel PT configuration parameters.
+  ///
+  /// \param[in] current_tids
+  ///   List of tids currently alive. In the future, whenever a new thread is
+  ///   spawned, they should be traced by calling the \a TraceStart(tid) method.
+  ///
+  /// \return
+  ///   An \a IntelPTMultiCoreTrace instance if tracing was successful, or
+  ///   an \a llvm::Error otherwise.
+  static llvm::Expected<IntelPTPerThreadProcessTraceUP>
+  Start(const TraceIntelPTStartRequest &request,
+        llvm::ArrayRef<lldb::tid_t> current_tids);
 
   bool TracesThread(lldb::tid_t tid) const;
 
   const IntelPTThreadTraceCollection &GetThreadTraces() const;
 
   llvm::Error TraceStart(lldb::tid_t tid);
 
   llvm::Error TraceStop(lldb::tid_t tid);
 
 private:
+  IntelPTPerThreadProcessTrace(const TraceIntelPTStartRequest &request)
+      : m_tracing_params(request) {}
+
   IntelPTThreadTraceCollection m_thread_traces;
   /// Params used to trace threads when the user started "process tracing".
   TraceIntelPTStartRequest m_tracing_params;
 };
 
 /// Main class that manages intel-pt process and thread tracing.
 class IntelPTCollector {
 public:
-  IntelPTCollector();
+  /// \param[in] process
+  ///     Process to be traced.
+  IntelPTCollector(NativeProcessProtocol &process);
 
   static bool IsSupported();
 
   /// If "process tracing" is enabled, then trace the given thread.
   llvm::Error OnThreadCreated(lldb::tid_t tid);
 
   /// Stops tracing a tracing upon a destroy event.
   llvm::Error OnThreadDestroyed(lldb::tid_t tid);
 
   /// Implementation of the jLLDBTraceStop packet
   llvm::Error TraceStop(const TraceStopRequest &request);
 
   /// Implementation of the jLLDBTraceStart packet
-  ///
-  /// \param[in] process_threads
-  ///     A list of all threads owned by the process.
-  llvm::Error TraceStart(const TraceIntelPTStartRequest &request,
-                         const std::vector<lldb::tid_t> &process_threads);
+  llvm::Error TraceStart(const TraceIntelPTStartRequest &request);
 
   /// Implementation of the jLLDBTraceGetState packet
   llvm::Expected<llvm::json::Value> GetState() const;
 
   /// Implementation of the jLLDBTraceGetBinaryData packet
   llvm::Expected<std::vector<uint8_t>>
   GetBinaryData(const TraceGetBinaryDataRequest &request) const;
 
private:
 
   llvm::Expected<const IntelPTSingleBufferTrace &>
   GetTracedThread(lldb::tid_t tid) const;
 
   bool IsProcessTracingEnabled() const;
 
   void ClearProcessTracing();
 
+  NativeProcessProtocol &m_process;
   /// Threads traced due to "thread tracing"
   IntelPTThreadTraceCollection m_thread_traces;
-  /// Threads traced due to "process tracing". Only one active "process tracing"
-  /// instance is assumed for a single process.
-  llvm::Optional<IntelPTProcessTrace> m_process_trace;
+
+  /// Only one of the following "process tracing" handlers can be active at a
+  /// given time.
+  ///
+  /// \{
+  /// Threads traced due to per-thread "process tracing".  This might be \b
+  /// nullptr.
+  IntelPTPerThreadProcessTraceUP m_per_thread_process_trace_up;
+  /// Cores traced due to per-core "process tracing".  This might be \b nullptr.
+  IntelPTMultiCoreTraceUP m_per_core_process_trace_up;
+  /// \}
+
   /// TSC to wall time conversion.
   TraceTscConversionUP m_tsc_conversion;
 };
 
 } // namespace process_linux
 } // namespace lldb_private
 
 #endif // liblldb_IntelPTCollector_H_

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

IntelPTThreadTraceCollection::GetTracedThread(lldb::tid_t tid) const {
   return *it->second.get();
 }
 
 void IntelPTThreadTraceCollection::Clear() {
   m_thread_traces.clear();
   m_total_buffer_size = 0;
 }
 
-/// IntelPTProcessTrace
+size_t IntelPTThreadTraceCollection::GetTracedThreadsCount() const {
+  return m_thread_traces.size();
+}
+
+/// IntelPTPerThreadProcessTrace
 
-bool IntelPTProcessTrace::TracesThread(lldb::tid_t tid) const {
+bool IntelPTPerThreadProcessTrace::TracesThread(lldb::tid_t tid) const {
   return m_thread_traces.TracesThread(tid);
 }
 
-Error IntelPTProcessTrace::TraceStop(lldb::tid_t tid) {
+Error IntelPTPerThreadProcessTrace::TraceStop(lldb::tid_t tid) {
   return m_thread_traces.TraceStop(tid);
 }
 
-Error IntelPTProcessTrace::TraceStart(lldb::tid_t tid) {
+Error IntelPTPerThreadProcessTrace::TraceStart(lldb::tid_t tid) {
   if (m_thread_traces.GetTotalBufferSize() +
           m_tracing_params.trace_buffer_size >
       static_cast<size_t>(*m_tracing_params.process_buffer_size_limit))
     return createStringError(
         inconvertibleErrorCode(),
         "Thread %" PRIu64 " can't be traced as the process trace size limit "
         "has been reached. Consider retracing with a higher "
         "limit.",
         tid);
 
   return m_thread_traces.TraceStart(tid, m_tracing_params);
 }
 
 const IntelPTThreadTraceCollection &
-IntelPTProcessTrace::GetThreadTraces() const {
+IntelPTPerThreadProcessTrace::GetThreadTraces() const {
   return m_thread_traces;
 }
 
 /// IntelPTCollector
 
-IntelPTCollector::IntelPTCollector() {
+IntelPTCollector::IntelPTCollector(NativeProcessProtocol &process)
+    : m_process(process) {
   if (Expected<LinuxPerfZeroTscConversion> tsc_conversion =
           LoadPerfTscConversionParameters())
     m_tsc_conversion =
         std::make_unique<LinuxPerfZeroTscConversion>(*tsc_conversion);
   else
     LLDB_LOG_ERROR(GetLog(POSIXLog::Trace), tsc_conversion.takeError(),
                    "unable to load TSC to wall time conversion: {0}");
 }
 
 Error IntelPTCollector::TraceStop(lldb::tid_t tid) {
-  if (IsProcessTracingEnabled() && m_process_trace->TracesThread(tid))
-    return m_process_trace->TraceStop(tid);
+  if (m_per_thread_process_trace_up &&
+      m_per_thread_process_trace_up->TracesThread(tid))
+    return m_per_thread_process_trace_up->TraceStop(tid);
+  else if (m_per_core_process_trace_up)
+    return createStringError(inconvertibleErrorCode(),
+                             "Can't stop tracing an individual thread when "
+                             "per-core process tracing is enabled.");
   return m_thread_traces.TraceStop(tid);
 }
 
 Error IntelPTCollector::TraceStop(const TraceStopRequest &request) {
   if (request.IsProcessTracing()) {
     Clear();
     return Error::success();
   } else {
     Error error = Error::success();
     for (int64_t tid : *request.tids)
       error = joinErrors(std::move(error),
                          TraceStop(static_cast<lldb::tid_t>(tid)));
     return error;
   }
 }
 
-Error IntelPTCollector::TraceStart(
-    const TraceIntelPTStartRequest &request,
-    const std::vector<lldb::tid_t> &process_threads) {
+Expected<IntelPTPerThreadProcessTraceUP>
+IntelPTPerThreadProcessTrace::Start(const TraceIntelPTStartRequest &request,
+                                    ArrayRef<lldb::tid_t> current_tids) {
+  IntelPTPerThreadProcessTraceUP trace(
+      new IntelPTPerThreadProcessTrace(request));
+
+  Error error = Error::success();
+  for (lldb::tid_t tid : current_tids)
+    error = joinErrors(std::move(error), trace->TraceStart(tid));
+  if (error)
+    return std::move(error);
+  return trace;
+}
+
+Error IntelPTCollector::TraceStart(const TraceIntelPTStartRequest &request) {
   if (request.IsProcessTracing()) {
     if (IsProcessTracingEnabled()) {
       return createStringError(
           inconvertibleErrorCode(),
           "Process currently traced. Stop process tracing first");
     }
-    if (request.per_core_tracing.getValueOr(false)) {
-      return createStringError(inconvertibleErrorCode(),
-                               "Per-core tracing is not supported.");
+    if (request.IsPerCoreTracing()) {
+      if (m_thread_traces.GetTracedThreadsCount() > 0)
+        return createStringError(
+            inconvertibleErrorCode(),
+            "Threads currently traced. Stop tracing them first.");
+      if (Expected<IntelPTMultiCoreTraceUP> trace =
+              IntelPTMultiCoreTrace::StartOnAllCores(request)) {
+        m_per_core_process_trace_up = std::move(*trace);
+        return Error::success();
+      } else {
+        return trace.takeError();
+      }
+    } else {
+      std::vector<lldb::tid_t> process_threads;
+      for (size_t i = 0; m_process.GetThreadAtIndex(i); i++)
+        process_threads.push_back(m_process.GetThreadAtIndex(i)->GetID());
+
+      // per-thread process tracing
+      if (Expected<IntelPTPerThreadProcessTraceUP> trace =
+              IntelPTPerThreadProcessTrace::Start(request, process_threads)) {
+        m_per_thread_process_trace_up = std::move(trace.get());
+        return Error::success();
+      } else {
+        return trace.takeError();
+      }
     }
-    m_process_trace = IntelPTProcessTrace(request);
-
-    Error error = Error::success();
-    for (lldb::tid_t tid : process_threads)
-      error = joinErrors(std::move(error), m_process_trace->TraceStart(tid));
-    return error;
   } else {
+    // individual thread tracing
+    if (m_per_core_process_trace_up)
+      return createStringError(inconvertibleErrorCode(),
+                               "Process currently traced with per-core "
+                               "tracing. Stop process tracing first");
+
     Error error = Error::success();
     for (int64_t tid : *request.tids)
       error = joinErrors(std::move(error),
                          m_thread_traces.TraceStart(tid, request));
     return error;
   }
 }
 
 Error IntelPTCollector::OnThreadCreated(lldb::tid_t tid) {
-  if (!IsProcessTracingEnabled())
-    return Error::success();
-  return m_process_trace->TraceStart(tid);
+  if (m_per_thread_process_trace_up)
+    return m_per_thread_process_trace_up->TraceStart(tid);
+
 }
 
 Error IntelPTCollector::OnThreadDestroyed(lldb::tid_t tid) {
-  if (IsProcessTracingEnabled() && m_process_trace->TracesThread(tid))
-    return m_process_trace->TraceStop(tid);
+  if (m_per_thread_process_trace_up &&
+      m_per_thread_process_trace_up->TracesThread(tid))
+    return m_per_thread_process_trace_up->TraceStop(tid);
   else if (m_thread_traces.TracesThread(tid))
     return m_thread_traces.TraceStop(tid);
   return Error::success();
 }
 
 Expected<json::Value> IntelPTCollector::GetState() const {
   Expected<ArrayRef<uint8_t>> cpu_info = GetProcfsCpuInfo();
   if (!cpu_info)
     return cpu_info.takeError();
 
   TraceGetStateResponse state;
-  state.processBinaryData.push_back(
+  state.process_binary_data.push_back(
       {IntelPTDataKinds::kProcFsCpuInfo, cpu_info->size()});
 
   std::vector<TraceThreadState> thread_states =
       m_thread_traces.GetThreadStates();
-  state.tracedThreads.insert(state.tracedThreads.end(), thread_states.begin(),
+  state.traced_threads.insert(state.traced_threads.end(), thread_states.begin(),
                               thread_states.end());
 
-  if (IsProcessTracingEnabled()) {
-    thread_states = m_process_trace->GetThreadTraces().GetThreadStates();
-    state.tracedThreads.insert(state.tracedThreads.end(), thread_states.begin(),
-                               thread_states.end());
+  if (m_per_thread_process_trace_up) {
+    thread_states =
+        m_per_thread_process_trace_up->GetThreadTraces().GetThreadStates();
+    state.traced_threads.insert(state.traced_threads.end(),
+                                thread_states.begin(), thread_states.end());
+  } else if (m_per_core_process_trace_up) {
+    for (size_t i = 0; m_process.GetThreadAtIndex(i); i++)
+      state.traced_threads.push_back(
+          TraceThreadState{m_process.GetThreadAtIndex(i)->GetID(), {}});
+
+    state.cores.emplace();
+    m_per_core_process_trace_up->ForEachCore(
+        [&](lldb::core_id_t core_id,
+            const IntelPTSingleBufferTrace &core_trace) {
+          state.cores->push_back({core_id,
+                                  {{IntelPTDataKinds::kTraceBuffer,
+                                    core_trace.GetTraceBufferSize()}}});
+        });
   }
   return toJSON(state);
 }
 
 Expected<const IntelPTSingleBufferTrace &>
 IntelPTCollector::GetTracedThread(lldb::tid_t tid) const {
-  if (IsProcessTracingEnabled() && m_process_trace->TracesThread(tid))
-    return m_process_trace->GetThreadTraces().GetTracedThread(tid);
+  if (m_per_thread_process_trace_up &&
+      m_per_thread_process_trace_up->TracesThread(tid))
+    return m_per_thread_process_trace_up->GetThreadTraces().GetTracedThread(
+        tid);
   return m_thread_traces.GetTracedThread(tid);
 }
 
 Expected<std::vector<uint8_t>>
 IntelPTCollector::GetBinaryData(const TraceGetBinaryDataRequest &request) const {
   if (request.kind == IntelPTDataKinds::kTraceBuffer) {
     if (Expected<const IntelPTSingleBufferTrace &> trace =
             GetTracedThread(*request.tid))
       return trace->GetTraceBuffer(request.offset, request.size);
     else
       return trace.takeError();
   } else if (request.kind == IntelPTDataKinds::kProcFsCpuInfo) {
     return GetProcfsCpuInfo();
   }
   return createStringError(inconvertibleErrorCode(),
                            "Unsuported trace binary data kind: %s",
                            request.kind.c_str());
 }
 
-void IntelPTCollector::ClearProcessTracing() { m_process_trace = None; }
+void IntelPTCollector::ClearProcessTracing() {
+  m_per_thread_process_trace_up.reset();
+  m_per_core_process_trace_up.reset();
+}
 
 bool IntelPTCollector::IsSupported() {
   if (Expected<uint32_t> intel_pt_type = GetIntelPTOSEventType()) {
     return true;
   } else {
     llvm::consumeError(intel_pt_type.takeError());
     return false;
   }
 }
 
 bool IntelPTCollector::IsProcessTracingEnabled() const {
-  return (bool)m_process_trace;
+  return (bool)m_per_thread_process_trace_up ||
+         (bool)m_per_core_process_trace_up;
 }
 
 void IntelPTCollector::Clear() {
   ClearProcessTracing();
   m_thread_traces.Clear();
 }

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.h

+//===-- IntelPTMultiCoreTrace.h ------------------------------- -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef liblldb_IntelPTMultiCoreTrace_H_
+#define liblldb_IntelPTMultiCoreTrace_H_
+
+#include "IntelPTSingleBufferTrace.h"
+
+#include "lldb/Utility/TraceIntelPTGDBRemotePackets.h"
+#include "lldb/lldb-types.h"
+
+#include "llvm/Support/Error.h"
+
+#include <memory>
+
+namespace lldb_private {
+namespace process_linux {
+
+class IntelPTMultiCoreTrace;
+using IntelPTMultiCoreTraceUP = std::unique_ptr<IntelPTMultiCoreTrace>;
+
+class IntelPTMultiCoreTrace {
+public:
+  /// Start tracing all CPU cores.
+  ///
+  /// \param[in] request
+  ///   Intel PT configuration parameters.
+  ///
+  /// \return
+  ///   An \a IntelPTMultiCoreTrace instance if tracing was successful, or
+  ///   an \a llvm::Error otherwise.
+  static llvm::Expected<IntelPTMultiCoreTraceUP>
+  StartOnAllCores(const TraceIntelPTStartRequest &request);
+
+  /// Execute the provided callback on each core that is being traced.
+  ///
+  /// \param[in] callback.core_id
+  ///   The core id that is being traced.
+  ///
+  /// \param[in] callback.core_trace
+  ///   The single-buffer trace instance for the given core.
+  void
+  ForEachCore(std::function<void(lldb::core_id_t core_id,
+                                 const IntelPTSingleBufferTrace &core_trace)>
+                  callback);
+
+private:
+  IntelPTMultiCoreTrace(
+      llvm::DenseMap<lldb::core_id_t, IntelPTSingleBufferTraceUP>
+          &&traces_per_core)
+      : m_traces_per_core(std::move(traces_per_core)) {}
+
+  llvm::DenseMap<lldb::core_id_t, IntelPTSingleBufferTraceUP> m_traces_per_core;
+};
+
+} // namespace process_linux
+} // namespace lldb_private
+
+#endif // liblldb_IntelPTMultiCoreTrace_H_

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

+//===-- IntelPTMultiCoreTrace.cpp -----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "IntelPTMultiCoreTrace.h"
+
+#include "Procfs.h"
+
+using namespace lldb;
+using namespace lldb_private;
+using namespace process_linux;
+using namespace llvm;
+
+static bool IsTotalBufferLimitReached(ArrayRef<core_id_t> cores,
+                                      const TraceIntelPTStartRequest &request) {
+  uint64_t required = cores.size() * request.trace_buffer_size;
+  uint64_t limit = request.process_buffer_size_limit.getValueOr(
+      std::numeric_limits<uint64_t>::max());
+  return required > limit;
+}
+
+static Error IncludePerfEventParanoidMessageInError(Error &&error) {
+  return createStringError(
+      inconvertibleErrorCode(),
+      "%s\nYou might need to rerun as sudo or to set "
+      "/proc/sys/kernel/perf_event_paranoid to a value of 0 or -1.",
+      toString(std::move(error)).c_str());
+}
+
+Expected<IntelPTMultiCoreTraceUP> IntelPTMultiCoreTrace::StartOnAllCores(
+    const TraceIntelPTStartRequest &request) {
+  Expected<ArrayRef<core_id_t>> core_ids = GetAvailableLogicalCoreIDs();
+  if (!core_ids)
+    return core_ids.takeError();
+
+  if (IsTotalBufferLimitReached(*core_ids, request))
+    return createStringError(
+        inconvertibleErrorCode(),
+        "The process can't be traced because the process trace size limit "
+        "has been reached. Consider retracing with a higher limit.");
+
+  llvm::DenseMap<core_id_t, IntelPTSingleBufferTraceUP> buffers;
+  for (core_id_t core_id : *core_ids) {
+    if (Expected<IntelPTSingleBufferTraceUP> core_trace =
+            IntelPTSingleBufferTrace::Start(request, /*tid=*/None, core_id))
+      buffers.try_emplace(core_id, std::move(*core_trace));
+    else
+      return IncludePerfEventParanoidMessageInError(core_trace.takeError());
+  }
+
+  return IntelPTMultiCoreTraceUP(new IntelPTMultiCoreTrace(std::move(buffers)));
+}
+
+void IntelPTMultiCoreTrace::ForEachCore(
+    std::function<void(core_id_t core_id,
+                       const IntelPTSingleBufferTrace &core_trace)>
+        callback) {
+  for (auto &it : m_traces_per_core)
+    callback(it.first, *it.second);
+}

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.h

 
 #include <memory>
 
 namespace lldb_private {
 namespace process_linux {
 
 llvm::Expected<uint32_t> GetIntelPTOSEventType();
 
-class IntelPTTrace;
 class IntelPTSingleBufferTrace;
 
-using IntelPTThreadTraceUP = std::unique_ptr<IntelPTTrace>;
 using IntelPTSingleBufferTraceUP = std::unique_ptr<IntelPTSingleBufferTrace>;
 
 /// This class wraps a single perf event collecting intel pt data in a single
 /// buffer.
 class IntelPTSingleBufferTrace {
 public:
   /// Start tracing using a single Intel PT trace buffer.
   ///
   /// \param[in] request
   ///     Intel PT configuration parameters.
   ///
   /// \param[in] tid
-  ///     The tid of the thread to be traced.
+  ///     The tid of the thread to be traced. If \b None, then this traces all
+  ///     threads of all processes.
+  ///
+  /// \param[in] core_id
+  ///     The CPU core id where to trace. If \b None, then this traces all CPUs.
   ///
   /// \return
   ///   A \a IntelPTSingleBufferTrace instance if tracing was successful, or
   ///   an \a llvm::Error otherwise.
   static llvm::Expected<IntelPTSingleBufferTraceUP>
-  Start(const TraceIntelPTStartRequest &request, lldb::tid_t tid);
+  Start(const TraceIntelPTStartRequest &request,
+        llvm::Optional<lldb::tid_t> tid,
+        llvm::Optional<lldb::core_id_t> core_id = llvm::None);
 
   /// \return
   ///    The bytes requested by a jLLDBTraceGetBinaryData packet that was routed
   ///    to this trace instace.
   llvm::Expected<std::vector<uint8_t>>
   GetBinaryData(const TraceGetBinaryDataRequest &request) const;
 
   /// Read the trace buffer managed by this trace instance.

 
 private:
   /// Construct new \a IntelPTSingleBufferThreadTrace. Users are supposed to
   /// create instances of this class via the \a Start() method and not invoke
   /// this one directly.
   ///
   /// \param[in] perf_event
   ///   perf event configured for IntelPT.
-  ///
-  /// \param[in] tid
-  ///   The thread being traced.
-  IntelPTSingleBufferTrace(PerfEvent &&perf_event, lldb::tid_t tid)
+  IntelPTSingleBufferTrace(PerfEvent &&perf_event)
       : m_perf_event(std::move(perf_event)) {}
 
   /// perf event configured for IntelPT.
   PerfEvent m_perf_event;
 };
 
 } // namespace process_linux
 } // namespace lldb_private
 
 #endif // liblldb_IntelPTSingleBufferTrace_H_

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.cpp

if (index > 0)
       error << ", " << index;
     // clear the lowest bit
     mask &= mask - 1;
   }
   error << ".";
   return createStringError(inconvertibleErrorCode(), error.str().c_str());
 }
 
 #ifdef PERF_ATTR_SIZE_VER5
 static Expected<uint64_t>
 GeneratePerfEventConfigValue(bool enable_tsc, Optional<uint64_t> psb_period) {
   uint64_t config = 0;
   // tsc is always supported
   if (enable_tsc) {
     if (Expected<uint32_t> offset = ReadIntelPTConfigFile(
             kTSCBitOffsetFile, IntelPTConfigFileType::BitOffset))
       config |= 1 << *offset;
IntelPTSingleBufferTrace::GetTraceBuffer(size_t offset, size_t size) const {
 
   // Reenable tracing now we have read the buffer
   ioctl(fd, PERF_EVENT_IOC_ENABLE);
   return data;
 }
 
 Expected<IntelPTSingleBufferTraceUP>
 IntelPTSingleBufferTrace::Start(const TraceIntelPTStartRequest &request,
-                                lldb::tid_t tid) {
+                                Optional<lldb::tid_t> tid,
+                                Optional<core_id_t> core_id) {
 #ifndef PERF_ATTR_SIZE_VER5
   return createStringError(inconvertibleErrorCode(),
                            "Intel PT Linux perf event not supported");
 #else
   Log *log = GetLog(POSIXLog::Trace);
 
-  LLDB_LOG(log, "Will start tracing thread id {0}", tid);
+  LLDB_LOG(log, "Will start tracing thread id {0} and cpu id {1}", tid,
+           core_id);
 
   if (__builtin_popcount(request.trace_buffer_size) != 1 ||
       request.trace_buffer_size < 4096) {
     return createStringError(
         inconvertibleErrorCode(),
         "The trace buffer size must be a power of 2 greater than or equal to "
         "4096 (2^12) bytes. It was %" PRIu64 ".",
         request.trace_buffer_size);
   }
   uint64_t page_size = getpagesize();
   uint64_t buffer_numpages = static_cast<uint64_t>(llvm::PowerOf2Floor(
       (request.trace_buffer_size + page_size - 1) / page_size));
 
   Expected<perf_event_attr> attr = CreateIntelPTPerfEventConfiguration(
       request.enable_tsc, request.psb_period.map([](int value) {
         return static_cast<uint64_t>(value);
       }));
   if (!attr)
     return attr.takeError();
 
   LLDB_LOG(log, "Will create trace buffer of size {0}",
            request.trace_buffer_size);
 
-  if (Expected<PerfEvent> perf_event = PerfEvent::Init(*attr, tid)) {
+  if (Expected<PerfEvent> perf_event = PerfEvent::Init(*attr, tid, core_id)) {
     if (Error mmap_err = perf_event->MmapMetadataAndBuffers(buffer_numpages,
                                                             buffer_numpages)) {
       return std::move(mmap_err);
     }
     return IntelPTSingleBufferTraceUP(
-        new IntelPTSingleBufferTrace(std::move(*perf_event), tid));
+        new IntelPTSingleBufferTrace(std::move(*perf_event)));
   } else {
     return perf_event.takeError();
   }
 #endif
 }

lldb/source/Plugins/Process/Linux/NativeProcessLinux.cpp

 
 // Public Instance Methods
 
 NativeProcessLinux::NativeProcessLinux(::pid_t pid, int terminal_fd,
                                        NativeDelegate &delegate,
                                        const ArchSpec &arch, MainLoop &mainloop,
                                        llvm::ArrayRef<::pid_t> tids)
     : NativeProcessELF(pid, terminal_fd, delegate), m_arch(arch),
-      m_main_loop(mainloop) {
+      m_main_loop(mainloop), m_intel_pt_collector(*this) {
   if (m_terminal_fd != -1) {
     Status status = EnsureFDFlags(m_terminal_fd, O_NONBLOCK);
     assert(status.Success());
   }
 
   Status status;
   m_sigchld_handle = mainloop.RegisterSignal(
       SIGCHLD, [this](MainLoopBase &) { SigchldHandler(); }, status);
llvm::Expected<TraceSupportedResponse> NativeProcessLinux::TraceSupported() {
   return NativeProcessProtocol::TraceSupported();
 }
 
 Error NativeProcessLinux::TraceStart(StringRef json_request, StringRef type) {
   if (type == "intel-pt") {
     if (Expected<TraceIntelPTStartRequest> request =
             json::parse<TraceIntelPTStartRequest>(json_request,
                                                   "TraceIntelPTStartRequest")) {
-      std::vector<lldb::tid_t> process_threads;
-      for (auto &thread : m_threads)
-        process_threads.push_back(thread->GetID());
-      return m_intel_pt_collector.TraceStart(*request, process_threads);
+      return m_intel_pt_collector.TraceStart(*request);
     } else
       return request.takeError();
   }
 
   return NativeProcessProtocol::TraceStart(json_request, type);
 }
 
 Error NativeProcessLinux::TraceStop(const TraceStopRequest &request) {

lldb/source/Plugins/Process/Linux/Perf.h

public:
   ///
   /// The parameters are directly forwarded to a perf_event_open syscall,
   /// for additional information on the parameters visit
   /// https://man7.org/linux/man-pages/man2/perf_event_open.2.html.
   ///
   /// \param[in] attr
   ///     Configuration information for the event.
   ///
   /// \param[in] pid
-  ///     The process to be monitored by the event.
+  ///     The process or thread to be monitored by the event. If \b None, then
+  ///     all processes and threads are monitored.
   ///
   /// \param[in] cpu
-  ///     The cpu to be monitored by the event.
+  ///     The cpu to be monitored by the event. If \b None, then all cpus are
+  ///     monitored.
   ///
   /// \param[in] group_fd
-  ///     File descriptor of the group leader.
+  ///     File descriptor of the group leader. If \b None, then this perf_event
+  ///     doesn't belong to a preexisting group.
   ///
   /// \param[in] flags
   ///     Bitmask of additional configuration flags.
   ///
   /// \return
   ///     If the perf_event_open syscall was successful, a minimal \a PerfEvent
   ///     instance, or an \a llvm::Error otherwise.
-  static llvm::Expected<PerfEvent> Init(perf_event_attr &attr, lldb::pid_t pid,
-                                        int cpu, int group_fd,
+  static llvm::Expected<PerfEvent> Init(perf_event_attr &attr,
+                                        llvm::Optional<lldb::pid_t> pid,
+                                        llvm::Optional<lldb::core_id_t> cpu,
+                                        llvm::Optional<int> group_fd,
                                         unsigned long flags);
 
   /// Create a new performance monitoring event via the perf_event_open syscall
   /// with "default" values for the cpu, group_fd and flags arguments.
   ///
   /// Convenience method to be used when the perf event requires minimal
   /// configuration. It handles the default values of all other arguments.
   ///
   /// \param[in] attr
   ///     Configuration information for the event.
   ///
   /// \param[in] pid
-  ///     The process to be monitored by the event.
-  static llvm::Expected<PerfEvent> Init(perf_event_attr &attr, lldb::pid_t pid);
+  ///     The process or thread to be monitored by the event. If \b None, then
+  ///     all threads and processes are monitored.
+  static llvm::Expected<PerfEvent>
+  Init(perf_event_attr &attr, llvm::Optional<lldb::pid_t> pid,
+       llvm::Optional<lldb::core_id_t> core = llvm::None);
 
   /// Mmap the metadata page and the data and aux buffers of the perf event and
   /// expose them through \a PerfEvent::GetMetadataPage() , \a
   /// PerfEvent::GetDataBuffer() and \a PerfEvent::GetAuxBuffer().
   ///
   /// This uses mmap underneath, which means that the number of pages mmap'ed
   /// must be less than the actual data available by the kernel. The metadata
   /// page is always mmap'ed.

lldb/source/Plugins/Process/Linux/Perf.cpp

if (ptr == nullptr)
     return;
   if (*ptr == -1)
     return;
   close(*ptr);
   std::default_delete<long>()(ptr);
 }
 
 llvm::Expected<PerfEvent> PerfEvent::Init(perf_event_attr &attr,
-                                          lldb::pid_t pid, int cpu,
-                                          int group_fd, unsigned long flags) {
+                                          Optional<lldb::pid_t> pid,
+                                          Optional<lldb::core_id_t> cpu,
+                                          Optional<int> group_fd,
+                                          unsigned long flags) {
   errno = 0;
-  long fd = syscall(SYS_perf_event_open, &attr, pid, cpu, group_fd, flags);
+  long fd = syscall(SYS_perf_event_open, &attr, pid.getValueOr(-1),
+                    cpu.getValueOr(-1), group_fd.getValueOr(-1), flags);
   if (fd == -1) {
     std::string err_msg =
         llvm::formatv("perf event syscall failed: {0}", std::strerror(errno));
     return llvm::createStringError(llvm::inconvertibleErrorCode(), err_msg);
   }
   return PerfEvent{fd};
 }
 
 llvm::Expected<PerfEvent> PerfEvent::Init(perf_event_attr &attr,
-                                          lldb::pid_t pid) {
-  return Init(attr, pid, -1, -1, 0);
+                                          Optional<lldb::pid_t> pid,
+                                          Optional<lldb::core_id_t> cpu) {
+  return Init(attr, pid, cpu, -1, 0);
 }
 
 llvm::Expected<resource_handle::MmapUP>
 PerfEvent::DoMmap(void *addr, size_t length, int prot, int flags,
                   long int offset, llvm::StringRef buffer_name) {
   errno = 0;
   auto mmap_result = ::mmap(nullptr, length, prot, flags, GetFd(), offset);
 

lldb/source/Plugins/Process/Linux/Procfs.h

 //===-- Procfs.h ---------------------------------------------- -*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 // source/Plugins/Process/Linux/Procfs.h defines the symbols we need from
 // sys/procfs.h on Android/Linux for all supported architectures.
 
 #include <sys/ptrace.h>
 
+#include "lldb/lldb-types.h"
+
 #include "llvm/Support/Error.h"
 
 #include <vector>
 
 #ifdef __ANDROID__
 #if defined(__arm64__) || defined(__aarch64__)
 typedef unsigned long elf_greg_t;
 typedef elf_greg_t

 
 /// \return
 ///     The content of /proc/cpuinfo and cache it if errors didn't happen.
 llvm::Expected<llvm::ArrayRef<uint8_t>> GetProcfsCpuInfo();
 
 /// \return
 ///     A list of available logical core ids given the contents of
 ///     /proc/cpuinfo.
-llvm::Expected<std::vector<int>>
+llvm::Expected<std::vector<lldb::core_id_t>>
 GetAvailableLogicalCoreIDs(llvm::StringRef cpuinfo);
 
 /// \return
 ///     A list with all the logical cores available in the system and cache it
 ///     if errors didn't happen.
-llvm::Expected<llvm::ArrayRef<int>> GetAvailableLogicalCoreIDs();
+llvm::Expected<llvm::ArrayRef<lldb::core_id_t>> GetAvailableLogicalCoreIDs();
 
 } // namespace process_linux
 } // namespace lldb_private

lldb/source/Plugins/Process/Linux/Procfs.cpp

 //===-- Procfs.cpp --------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Procfs.h"
 
 #include "lldb/Host/linux/Support.h"
 #include "llvm/Support/MemoryBuffer.h"
 
+using namespace lldb;
 using namespace lldb_private;
 using namespace process_linux;
 using namespace llvm;
 
 Expected<ArrayRef<uint8_t>> lldb_private::process_linux::GetProcfsCpuInfo() {
   static Optional<std::vector<uint8_t>> cpu_info;
   if (!cpu_info) {
     auto buffer_or_error = errorOrToExpected(getProcFile("cpuinfo"));
     if (!buffer_or_error)
       return buffer_or_error.takeError();
     MemoryBuffer &buffer = **buffer_or_error;
     cpu_info = std::vector<uint8_t>(
         reinterpret_cast<const uint8_t *>(buffer.getBufferStart()),
         reinterpret_cast<const uint8_t *>(buffer.getBufferEnd()));
   }
   return *cpu_info;
 }
 
-Expected<std::vector<int>>
+Expected<std::vector<core_id_t>>
 lldb_private::process_linux::GetAvailableLogicalCoreIDs(StringRef cpuinfo) {
   SmallVector<StringRef, 8> lines;
   cpuinfo.split(lines, "\n", /*MaxSplit=*/-1, /*KeepEmpty=*/false);
-  std::vector<int> logical_cores;
+  std::vector<core_id_t> logical_cores;
 
   for (StringRef line : lines) {
     std::pair<StringRef, StringRef> key_value = line.split(':');
     auto key = key_value.first.trim();
     auto val = key_value.second.trim();
     if (key == "processor") {
-      int processor;
+      core_id_t processor;
       if (val.getAsInteger(10, processor))
         return createStringError(
             inconvertibleErrorCode(),
             "Failed parsing the /proc/cpuinfo line entry: %s", line.data());
       logical_cores.push_back(processor);
     }
   }
   return logical_cores;
 }
 
-llvm::Expected<llvm::ArrayRef<int>>
+llvm::Expected<llvm::ArrayRef<core_id_t>>
 lldb_private::process_linux::GetAvailableLogicalCoreIDs() {
-  static Optional<std::vector<int>> logical_cores_ids;
+  static Optional<std::vector<core_id_t>> logical_cores_ids;
   if (!logical_cores_ids) {
     // We find the actual list of core ids by parsing /proc/cpuinfo
     Expected<ArrayRef<uint8_t>> cpuinfo = GetProcfsCpuInfo();
     if (!cpuinfo)
       return cpuinfo.takeError();
 
-    Expected<std::vector<int>> core_ids = GetAvailableLogicalCoreIDs(StringRef(
-        reinterpret_cast<const char *>(cpuinfo->data()), cpuinfo->size()));
+    Expected<std::vector<core_id_t>> core_ids = GetAvailableLogicalCoreIDs(
+        StringRef(reinterpret_cast<const char *>(cpuinfo->data())));
     if (!core_ids)
       return core_ids.takeError();
 
     logical_cores_ids.emplace(std::move(*core_ids));
   }
   return *logical_cores_ids;
 }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

void TraceIntelPT::DoRefreshLiveProcessState(
     Expected<TraceGetStateResponse> state) {
   m_thread_decoders.clear();
 
   if (!state) {
     m_live_refresh_error = toString(state.takeError());
     return;
   }
 
-  for (const TraceThreadState &thread_state : state->tracedThreads) {
+  for (const TraceThreadState &thread_state : state->traced_threads) {
     ThreadSP thread_sp =
         m_live_process->GetThreadList().FindThreadByID(thread_state.tid);
     m_thread_decoders.emplace(
         thread_state.tid, std::make_unique<ThreadDecoder>(thread_sp, *this));
   }
 }
 
 bool TraceIntelPT::IsTraced(lldb::tid_t tid) {
llvm::Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,
   request.tids.emplace();
   for (lldb::tid_t tid : tids)
     request.tids->push_back(tid);
   return Trace::Start(toJSON(request));
 }
 
 Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,
                           StructuredData::ObjectSP configuration) {
-  size_t trace_buffer_size = kDefaultTraceBufferSize;
+  uint64_t trace_buffer_size = kDefaultTraceBufferSize;
   bool enable_tsc = kDefaultEnableTscValue;
-  Optional<size_t> psb_period = kDefaultPsbPeriod;
+  Optional<uint64_t> psb_period = kDefaultPsbPeriod;
 
   if (configuration) {
     if (StructuredData::Dictionary *dict = configuration->GetAsDictionary()) {
       dict->GetValueForKeyAsInteger("traceBufferSize", trace_buffer_size);
       dict->GetValueForKeyAsBoolean("enableTsc", enable_tsc);
       dict->GetValueForKeyAsInteger("psbPeriod", psb_period);
     } else {
       return createStringError(inconvertibleErrorCode(),

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTOptions.td

Desc<"Instead of having an individual trace buffer per thread, which uses "
          "threads. This might cause data loss for less frequent threads. This "
          "option forces the capture of TSC timestamps (see --tsc). Also, this "
          "option can't be used simulatenously with any other trace sessions "
          "because of its system-wide nature.">;
   def process_trace_start_intel_pt_process_size_limit: Option<"total-size-limit", "l">,
     Group<1>,
     Arg<"Value">,
     Desc<"Maximum total trace size per process in bytes. This limit applies to "
-         "the sum of the sizes of all thread traces of this process, excluding "
-         "the ones created with the \"thread trace start\" command. "
+         "the sum of the sizes of all thread and core traces of this process, "
+         "excluding the ones created with the \"thread trace start\" command. "
          "Whenever a thread is attempted to be traced due to this command and "
          "the limit would be reached, the process is stopped with a "
          "\"processor trace\" reason, so that the user can retrace the process "
          "if needed. Defaults to 500MB.">;
   def process_trace_start_intel_pt_tsc: Option<"tsc", "t">,
     Group<1>,
     Desc<"Enable the use of TSC timestamps. This is supported on all devices "
          "that support intel-pt.">;

lldb/source/Target/Trace.cpp

void Trace::RefreshLiveProcessState() {
   Expected<TraceGetStateResponse> live_process_state =
       json::parse<TraceGetStateResponse>(*json_string, "TraceGetStateResponse");
   if (!live_process_state) {
     DoRefreshLiveProcessState(live_process_state.takeError());
     return;
   }
 
   for (const TraceThreadState &thread_state :
-       live_process_state->tracedThreads) {
-    for (const TraceBinaryData &item : thread_state.binaryData)
+       live_process_state->traced_threads) {
+    for (const TraceBinaryData &item : thread_state.binary_data)
       m_live_thread_data[thread_state.tid][item.kind] = item.size;
   }
 
-  for (const TraceBinaryData &item : live_process_state->processBinaryData)
+  for (const TraceBinaryData &item : live_process_state->process_binary_data)
     m_live_process_data[item.kind] = item.size;
 
   DoRefreshLiveProcessState(std::move(live_process_state));
 }
 
 uint32_t Trace::GetStopID() {
   RefreshLiveProcessState();
   return m_stop_id;

lldb/source/Utility/TraceGDBRemotePackets.cpp

 
 json::Value toJSON(const TraceBinaryData &packet) {
   return json::Value(Object{{"kind", packet.kind}, {"size", packet.size}});
 }
 
 bool fromJSON(const json::Value &value, TraceThreadState &packet, Path path) {
   ObjectMapper o(value, path);
   return o && o.map("tid", packet.tid) &&
-         o.map("binaryData", packet.binaryData);
+         o.map("binaryData", packet.binary_data);
 }
 
 json::Value toJSON(const TraceThreadState &packet) {
   return json::Value(
-      Object{{"tid", packet.tid}, {"binaryData", packet.binaryData}});
+      Object{{"tid", packet.tid}, {"binaryData", packet.binary_data}});
 }
 
 bool fromJSON(const json::Value &value, TraceGetStateResponse &packet,
               Path path) {
   ObjectMapper o(value, path);
-  return o && o.map("tracedThreads", packet.tracedThreads) &&
-         o.map("processBinaryData", packet.processBinaryData);
+  return o && o.map("tracedThreads", packet.traced_threads) &&
+         o.map("processBinaryData", packet.process_binary_data) &&
+         o.map("cores", packet.cores);
 }
 
 json::Value toJSON(const TraceGetStateResponse &packet) {
-  return json::Value(Object{{"tracedThreads", packet.tracedThreads},
-                            {"processBinaryData", packet.processBinaryData}});
+  return json::Value(Object{{"tracedThreads", packet.traced_threads},
+                            {"processBinaryData", packet.process_binary_data},
+                            {"cores", packet.cores}});
+}
+
+bool fromJSON(const json::Value &value, TraceCoreState &packet,
+              json::Path path) {
+  ObjectMapper o(value, path);
+  int64_t core_id;
+  if (!o || !o.map("coreId", core_id) ||
+      !o.map("binaryData", packet.binary_data))
+    return false;
+  packet.core_id = static_cast<lldb::core_id_t>(core_id);
+  return true;
+}
+
+json::Value toJSON(const TraceCoreState &packet) {
+  return json::Value(
+      Object{{"coreId", packet.core_id}, {"binaryData", packet.binary_data}});
 }
 /// \}
 
 /// jLLDBTraceGetBinaryData
 /// \{
 json::Value toJSON(const TraceGetBinaryDataRequest &packet) {
   return json::Value(Object{{"type", packet.type},
                             {"kind", packet.kind},

lldb/source/Utility/TraceIntelPTGDBRemotePackets.cpp

 using namespace llvm;
 using namespace llvm::json;
 
 namespace lldb_private {
 
 const char *IntelPTDataKinds::kProcFsCpuInfo = "procfsCpuInfo";
 const char *IntelPTDataKinds::kTraceBuffer = "traceBuffer";
 
+bool TraceIntelPTStartRequest::IsPerCoreTracing() const {
+  return per_core_tracing.getValueOr(false);
+}
+
 bool fromJSON(const json::Value &value, TraceIntelPTStartRequest &packet,
               Path path) {
   ObjectMapper o(value, path);
   if (!o || !fromJSON(value, (TraceStartRequest &)packet, path) ||
       !o.map("enableTsc", packet.enable_tsc) ||
       !o.map("psbPeriod", packet.psb_period) ||
       !o.map("traceBufferSize", packet.trace_buffer_size))
     return false;

lldb/test/API/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.py

def testStartMultipleLiveThreads(self):
 
         self.expect("continue")
         self.expect("thread trace dump instructions", substrs=['main.cpp:9'])
 
         # We'll see here the second thread
         self.expect("continue")
         self.expect("thread trace dump instructions", substrs=['main.cpp:4'])
 
+        self.traceStopProcess()
+
     @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
     @testSBAPIAndCommands
     def testStartMultipleLiveThreadsWithStops(self):
         self.build()
         exe = self.getBuildArtifact("a.out")
 
         self.dbg.CreateTarget(exe)
 
def testStartMultipleLiveThreadsWithStops(self):
 
         # We'll stop at the next breakpoint, thread 2 will be still alive, but not traced. Thread 3 will be traced
         self.expect("continue")
         self.expect("thread trace dump instructions", substrs=['main.cpp:4'])
         self.expect("thread trace dump instructions 3", substrs=['main.cpp:4'])
 
         self.expect("thread trace dump instructions 2", substrs=['not traced'])
 
+        self.traceStopProcess()
+
     @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
     @testSBAPIAndCommands
     def testStartMultipleLiveThreadsWithStops(self):
         self.build()
         exe = self.getBuildArtifact("a.out")
         self.dbg.CreateTarget(exe)
 
         self.expect("b main")
def testStartMultipleLiveThreadsWithStops(self):
 
         # We'll stop at the next breakpoint in thread 3, thread 2 and 3 will be alive, but only 3 traced.
         self.expect("continue")
         self.expect("thread trace dump instructions", substrs=['main.cpp:4'])
         self.expect("thread trace dump instructions 3", substrs=['main.cpp:4'])
         self.expect("thread trace dump instructions 1", substrs=['not traced'])
         self.expect("thread trace dump instructions 2", substrs=['not traced'])
 
+        self.traceStopProcess()
+
     @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
     def testStartMultipleLiveThreadsWithThreadStartAll(self):
         self.build()
         exe = self.getBuildArtifact("a.out")
         target = self.dbg.CreateTarget(exe)
 
         self.expect("b main")
         self.expect("b 6")
def testStartMultipleLiveThreadsWithSmallTotalLimit(self):
         # we get the stop event when trace 3 appears and can't be traced
         self.expect("c", substrs=['Thread', "can't be traced"])
 
         self.traceStopProcess()
 
     @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
     @testSBAPIAndCommands
     def testStartPerCoreSession(self):
+        self.skipIfPerCoreTracingIsNotSupported()
+
         self.build()
         exe = self.getBuildArtifact("a.out")
         self.dbg.CreateTarget(exe)
 
         self.expect("b main")
         self.expect("r")
 
-        self.traceStartProcess(
-            error=True, perCoreTracing=True,
-            substrs=["Per-core tracing is not supported"])
+        # We should fail if we hit the total buffer limit. Useful if the number
+        # of cores is huge.
+        self.traceStartProcess(error="True", processBufferSizeLimit=100,
+            perCoreTracing=True,
+            substrs=["The process can't be traced because the process trace size "
+            "limit has been reached. Consider retracing with a higher limit."])
+
+        self.traceStartProcess(perCoreTracing=True)
+        self.traceStopProcess()
+
+        self.traceStartProcess(perCoreTracing=True)
+        # We can't support multiple per-core tracing sessions.
+        self.traceStartProcess(error=True, perCoreTracing=True,
+            substrs=["Process currently traced. Stop process tracing first"])
+
+        # We can't support tracing per thread is per core is enabled.
+        self.traceStartThread(
+            error="True",
+            substrs=["Process currently traced with per-core tracing. Stop process tracing first"])
+
+        # We can't stop individual thread when per core is enabled.
+        self.traceStopThread(error="True",
+            substrs=["Can't stop tracing an individual thread when per-core process tracing is enabled"])
+
+        # The GetState packet should return trace buffers per core and at least one traced thread
+        self.expect("""process plugin packet send 'jLLDBTraceGetState:{"type":"intel-pt"}]'""",
+            substrs=['''[{"kind":"traceBuffer","size":4096}],"coreId":''', '"tid":'])
+
+        self.traceStopProcess()

lldb/unittests/Process/Linux/ProcfsTests.cpp

 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 using namespace lldb_private;
 using namespace process_linux;
 using namespace llvm;
 
 TEST(Perf, HardcodedLogicalCoreIDs) {
-  Expected<std::vector<int>> core_ids =
+  Expected<std::vector<lldb::core_id_t>> core_ids =
       GetAvailableLogicalCoreIDs(R"(processor       : 13
 vendor_id       : GenuineIntel
 cpu family      : 6
 model           : 85
 model name      : Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
 stepping        : 4
 microcode       : 0x2000065
 cpu MHz         : 2886.370

 
 TEST(Perf, RealLogicalCoreIDs) {
   // We first check we can read /proc/cpuinfo
   auto buffer_or_error = errorOrToExpected(getProcFile("cpuinfo"));
   if (!buffer_or_error)
     GTEST_SKIP() << toString(buffer_or_error.takeError());
 
   // At this point we shouldn't fail parsing the core ids
-  Expected<ArrayRef<int>> core_ids = GetAvailableLogicalCoreIDs();
+  Expected<ArrayRef<lldb::core_id_t>> core_ids = GetAvailableLogicalCoreIDs();
   ASSERT_TRUE((bool)core_ids);
   ASSERT_GT((int)core_ids->size(), 0) << "We must see at least one core";
 }