[trace][intelpt] Support system-wide tracing [2] - Add a dummy --per-core-tracing option

[wallace]

Summary

This updates the documentation of the gdb-remote protocol, as well as the help messages, to include the new --per-core-tracing option.
I made some renames to account for this new option

[wallace]

nit

[jj10306]

Before doing a complete review can you provide clarity on the decision to only support perCore for process wide (see my inline comment with my thoughts/questions)? Understanding this will potentially provide the answers to many other questions I was going to leave, so figured I'd ask it ahead of time (:

lldb/docs/lldb-gdb-remote.txt

369

Why do we only want this option for process tracing?
Per cpu tracing collects all trace data agnostic to a user specified process/thread, so why should this only be exposed for process wide? I think it makes more sense to decouple the perCoreTracing option from process/threads specific options entirely so it is its own option all together and cannot be used in conjunction with process/thread options.
If there is reason to not go down that route, we then should also add support for perCoreTracing with the thread tracing option, not just the process tracing option as I feel it doesn't make since to only expose this for process tracing since it's doing the same thing behind the scenes.

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.cpp

66

So now m_trace_buffer_size is the size of each trace buffer, regardless of the buffer is for a single thread or a single core?

[wallace]

lldb/docs/lldb-gdb-remote.txt

369

The reason is that if you want to trace a specific thread, it's actually much better to use single-buffer thread tracing than per cpu. You get full fidelity and no data loss in this mode. On the other hand, per cpu tracing is really useful in a thread-unbounded case where you are okay with having no data for unfrequent threads.
Per-core single-tracing is doable, but it's very impractical in the context of LLDB. However, if we ever need to implement it eventually, we for sure can refactor the code for that.

[wallace]

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.cpp

66

yep

lldb/docs/lldb-gdb-remote.txt

 //  This traces specific threads.
 //
 // INPUT SCHEMA
 //  {
 //    "type": <string>,
 //        Tracing technology name, e.g. intel-pt, arm-coresight.
 //
 //    /* thread tracing only */
-//    "tids": [<decimal integer>],
+//    "tids"?: [<decimal integer>],
 //        Individual threads to trace.
 //
 //    ... other parameters specific to the provided tracing type
 //  }
 //
 // NOTES
 //  - If "tids" is not provided, then the operation is "process tracing",
 //    otherwise it's "thread tracing".
 //  - Each tracing technology can have different levels of support for "thread
 //    tracing" and "process tracing".
 //
 // INTEL-PT
 //  intel-pt supports both "thread tracing" and "process tracing".
 //
-//  "Process tracing" is implemented by tracing each thread individually, but
-//  managed by the same "process trace" instance.
-//  Each actual thread trace, either from "process tracing" or "thread tracing",
+//  "Process tracing" is implemented in two different ways. If the
+//  "perCoreTracing" option is false, then each thread is traced individually
+//  but managed by the same "process trace" instance. This means that the
+//  amount of trace buffers used is proportional to the number of running
+//  threads. This is the recommended option unless the number of threads is
+//  huge. If "perCoreTracing" is true, then each cpu core is traced invidually
+//  instead of each thread, which uses a fixed number of trace buffers, but
+//  might result in less data available for less frequent threads. See
+//  "perCoreTracing" below for more information.
+//
+//  Each actual trace buffer, either from "process tracing" or "thread tracing",
 //  is stored in an in-memory circular buffer, which keeps the most recent data.
 //
 //  Additional params in the input schema:
 //   {
-//     "threadBufferSize": <decimal integer>,
-//         Trace buffer size per thread in bytes. It must be a power of 2
-//         greater than or equal to 4096 (2^12) bytes.
+//     "traceBufferSize": <decimal integer>,
+//         Size in bytes used by each individual per-thread or per-core trace
+//         buffer. It must be a power of 2 greater than or equal to 4096 (2^12)
+//         bytes.
 //
 //     "enableTsc": <boolean>,
 //         Whether to enable TSC timestamps or not. This is supported on
 //         all devices that support intel-pt. A TSC timestamp is generated along
 //         with PSB (synchronization) packets, whose frequency can be configured
 //         with the "psbPeriod" parameter.
 //
 //     "psbPeriod"?: <Optional decimal integer>,

 //         raw trace bytes between PSB packets as:
 //
 //             2 ^ (value + 11)
 //
 //          e.g. value 3 means 16KiB between PSB packets. Defaults to
 //          0 if supported.
 //
 //     /* process tracing only */
+//     "perCoreTracing": <boolean>
+//         Instead of having an individual trace buffer per thread, this option
+//         triggers the collection on a per cpu core basis. This effectively
+//         traces the entire activity on all cores. At decoding time, in order
+//         to correctly associate a decoded instruction with a thread, the
+//         context switch trace of each core is needed, as well as a record per
+//         cpu indicating which thread was running on each core when tracing
+//         started. These secondary traces are correlated with the intel-pt
+//         trace by comparing TSC timestamps.
+//
+//         This option forces the capture of TSC timestamps (see "enableTsc").
+//
+//         Note: This option can't be used simulatenously with any other trace
+//         sessions because of its system-wide nature.
+//
+//     /* process tracing only */
 //     "processBufferSizeLimit": <decimal integer>,
 //         Maximum total buffer size per process in bytes.
 //         This limit applies to the sum of the sizes of all trace buffers for
 //         the current process, excluding the ones started with "thread tracing".
 //
-//         Whenever a thread is attempted to be traced due to "process tracing"
-//         and the limit would be reached, the process is stopped with a
-//         "tracing" reason along with a meaningful description, so that the
-//         user can retrace the process if needed.
+//         If "perCoreTracing" is false, whenever a thread is attempted to be
+//         traced due to "process tracing" and the limit would be reached, the
+//         process is stopped with a "tracing" reason along with a meaningful
+//         description, so that the user can retrace the process if needed.
+//
+//         If "perCoreTracing" is true, then starting the system-wide trace
+//         session fails if all the individual per-core trace buffers require
+//         in total more memory that the limit impossed by this parameter.
 //   }
 //
 //  Notes:
 //   - Modifying the parameters of an existing trace is not supported. The user
 //     needs to stop the trace and start a new one.
 //   - If "process tracing" is attempted and there are individual threads
 //     already being traced with "thread tracing", these traces are left
 //     unaffected and the threads not traced twice.

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

struct IntelPTDataKinds {
   static const char *kProcFsCpuInfo;
   static const char *kThreadTraceBuffer;
 };
 
 /// jLLDBTraceStart gdb-remote packet
 /// \{
 struct TraceIntelPTStartRequest : TraceStartRequest {
   /// Size in bytes to use for each thread's trace buffer.
-  int64_t threadBufferSize;
+  int64_t trace_buffer_size;
 
   /// Whether to enable TSC
-  bool enableTsc;
+  bool enable_tsc;
 
   /// PSB packet period
-  llvm::Optional<int64_t> psbPeriod;
+  llvm::Optional<int64_t> psb_period;
 
   /// Required when doing "process tracing".
   ///
   /// Limit in bytes on all the thread traces started by this "process trace"
   /// instance. When a thread is about to be traced and the limit would be hit,
   /// then a "tracing" stop event is triggered.
-  llvm::Optional<int64_t> processBufferSizeLimit;
+  llvm::Optional<int64_t> process_buffer_size_limit;
+
+  /// Whether to have a trace buffer per thread or per cpu core.
+  llvm::Optional<bool> per_core_tracing;
 };
 
 bool fromJSON(const llvm::json::Value &value, TraceIntelPTStartRequest &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceIntelPTStartRequest &packet);
 /// \}
 

lldb/packages/Python/lldbsuite/test/tools/intelpt/intelpt_testcase.py

def getTraceOrCreate(self):
         return self.target().GetTrace()
 
     def assertSBError(self, sberror, error=False):
         if error:
             self.assertTrue(sberror.Fail())
         else:
             self.assertSuccess(sberror)
 
-    def createConfiguration(self, threadBufferSize=None,
+    def createConfiguration(self, traceBufferSize=None,
                             processBufferSizeLimit=None, enableTsc=False,
-                            psbPeriod=None):
+                            psbPeriod=None, perCoreTracing=False):
         obj = {}
         if processBufferSizeLimit is not None:
             obj["processBufferSizeLimit"] = processBufferSizeLimit
-        if threadBufferSize is not None:
-            obj["threadBufferSize"] = threadBufferSize
+        if traceBufferSize is not None:
+            obj["traceBufferSize"] = traceBufferSize
         if psbPeriod is not None:
             obj["psbPeriod"] = psbPeriod
         obj["enableTsc"] = enableTsc
+        obj["perCoreTracing"] = perCoreTracing
 
         configuration = lldb.SBStructuredData()
         configuration.SetFromJSON(json.dumps(obj))
         return configuration
 
     def traceStartThread(self, thread=None, error=False, substrs=None,
-                         threadBufferSize=None, enableTsc=False, psbPeriod=None):
+                         traceBufferSize=None, enableTsc=False, psbPeriod=None):
         if self.USE_SB_API:
             trace = self.getTraceOrCreate()
             thread = thread if thread is not None else self.thread()
             configuration = self.createConfiguration(
-                threadBufferSize=threadBufferSize, enableTsc=enableTsc,
+                traceBufferSize=traceBufferSize, enableTsc=enableTsc,
                 psbPeriod=psbPeriod)
             self.assertSBError(trace.Start(thread, configuration), error)
         else:
             command = "thread trace start"
             if thread is not None:
                 command += " " + str(thread.GetIndexID())
-            if threadBufferSize is not None:
-                command += " -s " + str(threadBufferSize)
+            if traceBufferSize is not None:
+                command += " -s " + str(traceBufferSize)
             if enableTsc:
                 command += " --tsc"
             if psbPeriod is not None:
                 command += " --psb-period " + str(psbPeriod)
             self.expect(command, error=error, substrs=substrs)
 
     def traceStartProcess(self, processBufferSizeLimit=None, error=False,
-                          substrs=None, enableTsc=False, psbPeriod=None):
+                          substrs=None, enableTsc=False, psbPeriod=None,
+                          perCoreTracing=False):
         if self.USE_SB_API:
             trace = self.getTraceOrCreate()
             configuration = self.createConfiguration(
                 processBufferSizeLimit=processBufferSizeLimit, enableTsc=enableTsc,
-                psbPeriod=psbPeriod)
+                psbPeriod=psbPeriod, perCoreTracing=perCoreTracing)
             self.assertSBError(trace.Start(configuration), error=error)
         else:
             command = "process trace start"
             if processBufferSizeLimit != None:
                 command += " -l " + str(processBufferSizeLimit)
             if enableTsc:
                 command += " --tsc"
             if psbPeriod is not None:
                 command += " --psb-period " + str(psbPeriod)
+            if perCoreTracing:
+                command += " --per-core-tracing"
             self.expect(command, error=error, substrs=substrs)
 
     def traceStopProcess(self):
         if self.USE_SB_API:
             self.assertSuccess(self.target().GetTrace().Stop())
         else:
             self.expect("process trace stop")
 

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

 
 Error IntelPTThreadTraceCollection::TraceStart(
     lldb::tid_t tid, const TraceIntelPTStartRequest &request) {
   if (TracesThread(tid))
     return createStringError(inconvertibleErrorCode(),
                              "Thread %" PRIu64 " already traced", tid);
 
   Expected<IntelPTThreadTraceUP> trace_up = IntelPTThreadTrace::Create(
-      m_pid, tid, request.threadBufferSize, request.enableTsc,
-      request.psbPeriod.map([](int64_t period) { return (size_t)period; }));
+      m_pid, tid, request.trace_buffer_size, request.enable_tsc,
+      request.psb_period.map([](int64_t period) { return (size_t)period; }));
   if (!trace_up)
     return trace_up.takeError();
 
   m_total_buffer_size += (*trace_up)->GetTraceBufferSize();
   m_thread_traces.try_emplace(tid, std::move(*trace_up));
   return Error::success();
 }
 
bool IntelPTProcessTrace::TracesThread(lldb::tid_t tid) const {
   return m_thread_traces.TracesThread(tid);
 }
 
 Error IntelPTProcessTrace::TraceStop(lldb::tid_t tid) {
   return m_thread_traces.TraceStop(tid);
 }
 
 Error IntelPTProcessTrace::TraceStart(lldb::tid_t tid) {
-  if (m_thread_traces.GetTotalBufferSize() + m_tracing_params.threadBufferSize >
-      static_cast<size_t>(*m_tracing_params.processBufferSizeLimit))
+  if (m_thread_traces.GetTotalBufferSize() +
+          m_tracing_params.trace_buffer_size >
+      static_cast<size_t>(*m_tracing_params.process_buffer_size_limit))
     return createStringError(
         inconvertibleErrorCode(),
         "Thread %" PRIu64 " can't be traced as the process trace size limit "
         "has been reached. Consider retracing with a higher "
         "limit.",
         tid);
 
   return m_thread_traces.TraceStart(tid, m_tracing_params);
Error IntelPTCollector::TraceStart(
     const TraceIntelPTStartRequest &request,
     const std::vector<lldb::tid_t> &process_threads) {
   if (request.IsProcessTracing()) {
     if (IsProcessTracingEnabled()) {
       return createStringError(
           inconvertibleErrorCode(),
           "Process currently traced. Stop process tracing first");
     }
+    if (request.per_core_tracing.getValueOr(false)) {
+      return createStringError(inconvertibleErrorCode(),
+                               "Per-core tracing is not supported.");
+    }
     m_process_trace = IntelPTProcessTrace(m_pid, request);
 
     Error error = Error::success();
     for (lldb::tid_t tid : process_threads)
       error = joinErrors(std::move(error), m_process_trace->TraceStart(tid));
     return error;
   } else {
     Error error = Error::success();

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.h

public:
 
     Status SetOptionValue(uint32_t option_idx, llvm::StringRef option_arg,
                           ExecutionContext *execution_context) override;
 
     void OptionParsingStarting(ExecutionContext *execution_context) override;
 
     llvm::ArrayRef<OptionDefinition> GetDefinitions() override;
 
-    size_t m_thread_buffer_size;
+    size_t m_trace_buffer_size;
     bool m_enable_tsc;
     llvm::Optional<size_t> m_psb_period;
   };
 
   CommandObjectThreadTraceStartIntelPT(TraceIntelPT &trace,
                                        CommandInterpreter &interpreter)
       : CommandObjectMultipleThreads(
             interpreter, "thread trace start",
public:
 
     Status SetOptionValue(uint32_t option_idx, llvm::StringRef option_arg,
                           ExecutionContext *execution_context) override;
 
     void OptionParsingStarting(ExecutionContext *execution_context) override;
 
     llvm::ArrayRef<OptionDefinition> GetDefinitions() override;
 
-    size_t m_thread_buffer_size;
+    size_t m_trace_buffer_size;
     size_t m_process_buffer_size_limit;
     bool m_enable_tsc;
     llvm::Optional<size_t> m_psb_period;
+    bool m_per_core_tracing;
   };
 
   CommandObjectProcessTraceStartIntelPT(TraceIntelPT &trace,
                                         CommandInterpreter &interpreter)
       : CommandObjectParsed(
             interpreter, "process trace start",
             "Start tracing this process with intel-pt, including future "
-            "threads. "
-            "This is implemented by tracing each thread independently. "
+            "threads. If --per-core-tracing is not provided, this traces each "
+            "thread independently, thus using a trace buffer per thread. "
             "Threads traced with the \"thread trace start\" command are left "
-            "unaffected ant not retraced.",
+            "unaffected ant not retraced. This is the recommended option "
+            "unless the number of threads is huge. If --per-core-tracing is "
+            "passed, each cpu core is traced instead of each thread, which "
+            "uses a fixed number of trace buffers, but might result in less "
+            "data available for less frequent threads.",
             "process trace start [<intel-pt-options>]",
             lldb::eCommandRequiresProcess | lldb::eCommandTryTargetAPILock |
                 lldb::eCommandProcessMustBeLaunched |
                 lldb::eCommandProcessMustBePaused),
         m_trace(trace), m_options() {}
 
   Options *GetOptions() override { return &m_options; }
 

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.cpp

 Status CommandObjectThreadTraceStartIntelPT::CommandOptions::SetOptionValue(
     uint32_t option_idx, llvm::StringRef option_arg,
     ExecutionContext *execution_context) {
   Status error;
   const int short_option = m_getopt_table[option_idx].val;
 
   switch (short_option) {
   case 's': {
-    int64_t thread_buffer_size;
-    if (option_arg.empty() || option_arg.getAsInteger(0, thread_buffer_size) ||
-        thread_buffer_size < 0)
+    int64_t trace_buffer_size;
+    if (option_arg.empty() || option_arg.getAsInteger(0, trace_buffer_size) ||
+        trace_buffer_size < 0)
       error.SetErrorStringWithFormat("invalid integer value for option '%s'",
                                      option_arg.str().c_str());
     else
-      m_thread_buffer_size = thread_buffer_size;
+      m_trace_buffer_size = trace_buffer_size;
     break;
   }
   case 't': {
     m_enable_tsc = true;
     break;
   }
   case 'p': {
     int64_t psb_period;
     if (option_arg.empty() || option_arg.getAsInteger(0, psb_period) ||
         psb_period < 0)
       error.SetErrorStringWithFormat("invalid integer value for option '%s'",
                                      option_arg.str().c_str());
     else
       m_psb_period = psb_period;
     break;
   }
   default:
     llvm_unreachable("Unimplemented option");
   }
   return error;
 }
 
 void CommandObjectThreadTraceStartIntelPT::CommandOptions::
     OptionParsingStarting(ExecutionContext *execution_context) {
-  m_thread_buffer_size = kDefaultThreadBufferSize;
+  m_trace_buffer_size = kDefaultTraceBufferSize;
   m_enable_tsc = kDefaultEnableTscValue;
   m_psb_period = kDefaultPsbPeriod;
 }
 
 llvm::ArrayRef<OptionDefinition>
 CommandObjectThreadTraceStartIntelPT::CommandOptions::GetDefinitions() {
   return llvm::makeArrayRef(g_thread_trace_start_intel_pt_options);
 }
 
 bool CommandObjectThreadTraceStartIntelPT::DoExecuteOnThreads(
     Args &command, CommandReturnObject &result,
     llvm::ArrayRef<lldb::tid_t> tids) {
-  if (Error err = m_trace.Start(tids, m_options.m_thread_buffer_size,
+  if (Error err = m_trace.Start(tids, m_options.m_trace_buffer_size,
                                 m_options.m_enable_tsc, m_options.m_psb_period))
     result.SetError(Status(std::move(err)));
   else
     result.SetStatus(eReturnStatusSuccessFinishResult);
 
   return result.Succeeded();
 }
 
 /// CommandObjectProcessTraceStartIntelPT
 
 #define LLDB_OPTIONS_process_trace_start_intel_pt
 #include "TraceIntelPTCommandOptions.inc"
 
 Status CommandObjectProcessTraceStartIntelPT::CommandOptions::SetOptionValue(
     uint32_t option_idx, llvm::StringRef option_arg,
     ExecutionContext *execution_context) {
   Status error;
   const int short_option = m_getopt_table[option_idx].val;
 
   switch (short_option) {
   case 's': {
-    int64_t thread_buffer_size;
-    if (option_arg.empty() || option_arg.getAsInteger(0, thread_buffer_size) ||
-        thread_buffer_size < 0)
+    int64_t trace_buffer_size;
+    if (option_arg.empty() || option_arg.getAsInteger(0, trace_buffer_size) ||
+        trace_buffer_size < 0)
       error.SetErrorStringWithFormat("invalid integer value for option '%s'",
                                      option_arg.str().c_str());
     else
-      m_thread_buffer_size = thread_buffer_size;
+      m_trace_buffer_size = trace_buffer_size;
     break;
   }
   case 'l': {
     int64_t process_buffer_size_limit;
     if (option_arg.empty() ||
         option_arg.getAsInteger(0, process_buffer_size_limit) ||
         process_buffer_size_limit < 0)
       error.SetErrorStringWithFormat("invalid integer value for option '%s'",
                                      option_arg.str().c_str());
     else
       m_process_buffer_size_limit = process_buffer_size_limit;
     break;
   }
   case 't': {
     m_enable_tsc = true;
     break;
   }
+  case 'c': {
+    m_per_core_tracing = true;
+    break;
+  }
   case 'p': {
     int64_t psb_period;
     if (option_arg.empty() || option_arg.getAsInteger(0, psb_period) ||
         psb_period < 0)
       error.SetErrorStringWithFormat("invalid integer value for option '%s'",
                                      option_arg.str().c_str());
     else
       m_psb_period = psb_period;
     break;
   }
   default:
     llvm_unreachable("Unimplemented option");
   }
   return error;
 }
 
 void CommandObjectProcessTraceStartIntelPT::CommandOptions::
     OptionParsingStarting(ExecutionContext *execution_context) {
-  m_thread_buffer_size = kDefaultThreadBufferSize;
+  m_trace_buffer_size = kDefaultTraceBufferSize;
   m_process_buffer_size_limit = kDefaultProcessBufferSizeLimit;
   m_enable_tsc = kDefaultEnableTscValue;
   m_psb_period = kDefaultPsbPeriod;
+  m_per_core_tracing = kDefaultPerCoreTracing;
 }
 
 llvm::ArrayRef<OptionDefinition>
 CommandObjectProcessTraceStartIntelPT::CommandOptions::GetDefinitions() {
   return llvm::makeArrayRef(g_process_trace_start_intel_pt_options);
 }
 
 bool CommandObjectProcessTraceStartIntelPT::DoExecute(
     Args &command, CommandReturnObject &result) {
-  if (Error err = m_trace.Start(m_options.m_thread_buffer_size,
+  if (Error err = m_trace.Start(m_options.m_trace_buffer_size,
                                 m_options.m_process_buffer_size_limit,
-                                m_options.m_enable_tsc, m_options.m_psb_period))
+                                m_options.m_enable_tsc, m_options.m_psb_period,
+                                m_options.m_per_core_tracing))
     result.SetError(Status(std::move(err)));
   else
     result.SetStatus(eReturnStatusSuccessFinishResult);
 
   return result.Succeeded();
 }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

void DoRefreshLiveProcessState(
       llvm::Expected<TraceGetStateResponse> state) override;
 
   bool IsTraced(lldb::tid_t tid) override;
 
   const char *GetStartConfigurationHelp() override;
 
   /// Start tracing a live process.
   ///
-  /// \param[in] thread_buffer_size
+  /// More information on the parameters below can be found in the
+  /// jLLDBTraceStart section in lldb/docs/lldb-gdb-remote.txt.
+  ///
+  /// \param[in] trace_buffer_size
   ///     Trace size per thread in bytes.
   ///
   /// \param[in] total_buffer_size_limit
   ///     Maximum total trace size per process in bytes.
-  ///     More information in TraceIntelPT::GetStartConfigurationHelp().
   ///
   /// \param[in] enable_tsc
   ///     Whether to use enable TSC timestamps or not.
-  ///     More information in TraceIntelPT::GetStartConfigurationHelp().
   ///
   /// \param[in] psb_period
-  ///
   ///     This value defines the period in which PSB packets will be generated.
-  ///     More information in TraceIntelPT::GetStartConfigurationHelp();
+  ///
+  /// \param[in] per_core_tracing
+  ///     This value defines whether to have a trace buffer per thread or per
+  ///     cpu core.
   ///
   /// \return
   ///     \a llvm::Error::success if the operation was successful, or
   ///     \a llvm::Error otherwise.
-  llvm::Error Start(size_t thread_buffer_size, size_t total_buffer_size_limit,
-                    bool enable_tsc, llvm::Optional<size_t> psb_period);
+  llvm::Error Start(size_t trace_buffer_size, size_t total_buffer_size_limit,
+                    bool enable_tsc, llvm::Optional<size_t> psb_period,
+                    bool m_per_core_tracing);
 
   /// \copydoc Trace::Start
   llvm::Error Start(StructuredData::ObjectSP configuration =
                         StructuredData::ObjectSP()) override;
 
   /// Start tracing live threads.
   ///
+  /// More information on the parameters below can be found in the
+  /// jLLDBTraceStart section in lldb/docs/lldb-gdb-remote.txt.
+  ///
   /// \param[in] tids
   ///     Threads to trace.
   ///
-  /// \param[in] thread_buffer_size
-  ///     Trace size per thread in bytes.
+  /// \param[in] trace_buffer_size
+  ///     Trace size per thread or per core in bytes.
   ///
   /// \param[in] enable_tsc
   ///     Whether to use enable TSC timestamps or not.
-  ///     More information in TraceIntelPT::GetStartConfigurationHelp().
   ///
   /// \param[in] psb_period
-  ///
   ///     This value defines the period in which PSB packets will be generated.
-  ///     More information in TraceIntelPT::GetStartConfigurationHelp().
   ///
   /// \return
   ///     \a llvm::Error::success if the operation was successful, or
   ///     \a llvm::Error otherwise.
-  llvm::Error Start(llvm::ArrayRef<lldb::tid_t> tids, size_t thread_buffer_size,
+  llvm::Error Start(llvm::ArrayRef<lldb::tid_t> tids, size_t trace_buffer_size,
                     bool enable_tsc, llvm::Optional<size_t> psb_period);
 
   /// \copydoc Trace::Start
   llvm::Error Start(llvm::ArrayRef<lldb::tid_t> tids,
                     StructuredData::ObjectSP configuration =
                         StructuredData::ObjectSP()) override;
 
   /// See \a Trace::OnThreadBinaryDataRead().

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

bool TraceIntelPT::IsTraced(lldb::tid_t tid) {
   return m_thread_decoders.count(tid);
 }
 
 // The information here should match the description of the intel-pt section
 // of the jLLDBTraceStart packet in the lldb/docs/lldb-gdb-remote.txt
 // documentation file. Similarly, it should match the CLI help messages of the
 // TraceIntelPTOptions.td file.
 const char *TraceIntelPT::GetStartConfigurationHelp() {
-  return R"(Parameters:
+  static Optional<std::string> message;
+  if (!message) {
+    message.emplace(formatv(R"(Parameters:
 
-  Note: If a parameter is not specified, a default value will be used.
+  See the jLLDBTraceStart section in lldb/docs/lldb-gdb-remote.txt for a
+  description of each parameter below.
 
-  - int threadBufferSize (defaults to 4096 bytes):
+  - int traceBufferSize (defaults to {0} bytes):
     [process and thread tracing]
-    Trace size in bytes per thread. It must be a power of 2 greater
-    than or equal to 4096 (2^12). The trace is circular keeping the
-    the most recent data.
 
-  - boolean enableTsc (default to false):
+  - boolean enableTsc (default to {1}):
     [process and thread tracing]
-    Whether to use enable TSC timestamps or not. This is supported on
-    all devices that support intel-pt.
 
-  - psbPeriod (defaults to null):
+  - int psbPeriod (defaults to {2}):
     [process and thread tracing]
-    This value defines the period in which PSB packets will be generated.
-    A PSB packet is a synchronization packet that contains a TSC
-    timestamp and the current absolute instruction pointer.
 
-    This parameter can only be used if
-
-        /sys/bus/event_source/devices/intel_pt/caps/psb_cyc
-
-    is 1. Otherwise, the PSB period will be defined by the processor.
-
-    If supported, valid values for this period can be found in
-
-        /sys/bus/event_source/devices/intel_pt/caps/psb_periods
-
-    which contains a hexadecimal number, whose bits represent
-    valid values e.g. if bit 2 is set, then value 2 is valid.
-
-    The psb_period value is converted to the approximate number of
-    raw trace bytes between PSB packets as:
-
-        2 ^ (value + 11)
-
-    e.g. value 3 means 16KiB between PSB packets. Defaults to 0 if
-    supported.
-
-  - int processBufferSizeLimit (defaults to 500 MB):
+  - boolean perCoreTracing (default to {3}):
     [process tracing only]
-    Maximum total trace size per process in bytes. This limit applies
-    to the sum of the sizes of all thread traces of this process,
-    excluding the ones created explicitly with "thread tracing".
-    Whenever a thread is attempted to be traced due to this command
-    and the limit would be reached, the process is stopped with a
-    "processor trace" reason, so that the user can retrace the process
-    if needed.)";
+
+  - int processBufferSizeLimit (defaults to {4} MiB):
+    [process tracing only])",
+                            kDefaultTraceBufferSize, kDefaultEnableTscValue,
+                            kDefaultPsbPeriod, kDefaultPerCoreTracing,
+                            kDefaultProcessBufferSizeLimit / 1024 / 1024));
+  }
+  return message->c_str();
 }
 
-Error TraceIntelPT::Start(size_t thread_buffer_size,
+Error TraceIntelPT::Start(size_t trace_buffer_size,
                           size_t total_buffer_size_limit, bool enable_tsc,
-                          Optional<size_t> psb_period) {
+                          Optional<size_t> psb_period, bool per_core_tracing) {
   TraceIntelPTStartRequest request;
-  request.threadBufferSize = thread_buffer_size;
-  request.processBufferSizeLimit = total_buffer_size_limit;
-  request.enableTsc = enable_tsc;
-  request.psbPeriod = psb_period.map([](size_t val) { return (int64_t)val; });
+  request.trace_buffer_size = trace_buffer_size;
+  request.process_buffer_size_limit = total_buffer_size_limit;
+  request.enable_tsc = enable_tsc;
+  request.psb_period = psb_period.map([](size_t val) { return (int64_t)val; });
   request.type = GetPluginName().str();
+  request.per_core_tracing = per_core_tracing;
   return Trace::Start(toJSON(request));
 }
 
 Error TraceIntelPT::Start(StructuredData::ObjectSP configuration) {
-  size_t thread_buffer_size = kDefaultThreadBufferSize;
+  size_t trace_buffer_size = kDefaultTraceBufferSize;
   size_t process_buffer_size_limit = kDefaultProcessBufferSizeLimit;
   bool enable_tsc = kDefaultEnableTscValue;
   Optional<size_t> psb_period = kDefaultPsbPeriod;
+  bool per_core_tracing = kDefaultPerCoreTracing;
 
   if (configuration) {
     if (StructuredData::Dictionary *dict = configuration->GetAsDictionary()) {
-      dict->GetValueForKeyAsInteger("threadBufferSize", thread_buffer_size);
+      dict->GetValueForKeyAsInteger("traceBufferSize", trace_buffer_size);
       dict->GetValueForKeyAsInteger("processBufferSizeLimit",
                                     process_buffer_size_limit);
       dict->GetValueForKeyAsBoolean("enableTsc", enable_tsc);
       dict->GetValueForKeyAsInteger("psbPeriod", psb_period);
+      dict->GetValueForKeyAsBoolean("perCoreTracing", per_core_tracing);
     } else {
       return createStringError(inconvertibleErrorCode(),
                                "configuration object is not a dictionary");
     }
   }
 
-  return Start(thread_buffer_size, process_buffer_size_limit, enable_tsc,
-               psb_period);
+  return Start(trace_buffer_size, process_buffer_size_limit, enable_tsc,
+               psb_period, per_core_tracing);
 }
 
 llvm::Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,
-                                size_t thread_buffer_size, bool enable_tsc,
+                                size_t trace_buffer_size, bool enable_tsc,
                                 Optional<size_t> psb_period) {
   TraceIntelPTStartRequest request;
-  request.threadBufferSize = thread_buffer_size;
-  request.enableTsc = enable_tsc;
-  request.psbPeriod = psb_period.map([](size_t val) { return (int64_t)val; });
+  request.trace_buffer_size = trace_buffer_size;
+  request.enable_tsc = enable_tsc;
+  request.psb_period = psb_period.map([](size_t val) { return (int64_t)val; });
   request.type = GetPluginName().str();
   request.tids.emplace();
   for (lldb::tid_t tid : tids)
     request.tids->push_back(tid);
   return Trace::Start(toJSON(request));
 }
 
 Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,
                           StructuredData::ObjectSP configuration) {
-  size_t thread_buffer_size = kDefaultThreadBufferSize;
+  size_t trace_buffer_size = kDefaultTraceBufferSize;
   bool enable_tsc = kDefaultEnableTscValue;
   Optional<size_t> psb_period = kDefaultPsbPeriod;
 
   if (configuration) {
     if (StructuredData::Dictionary *dict = configuration->GetAsDictionary()) {
-      dict->GetValueForKeyAsInteger("threadBufferSize", thread_buffer_size);
+      dict->GetValueForKeyAsInteger("traceBufferSize", trace_buffer_size);
       dict->GetValueForKeyAsBoolean("enableTsc", enable_tsc);
       dict->GetValueForKeyAsInteger("psbPeriod", psb_period);
     } else {
       return createStringError(inconvertibleErrorCode(),
                                "configuration object is not a dictionary");
     }
   }
 
-  return Start(tids, thread_buffer_size, enable_tsc, psb_period);
+  return Start(tids, trace_buffer_size, enable_tsc, psb_period);
 }
 
 Error TraceIntelPT::OnThreadBufferRead(lldb::tid_t tid,
                                        OnBinaryDataReadCallback callback) {
   return OnThreadBinaryDataRead(tid, IntelPTDataKinds::kThreadTraceBuffer,
                                 callback);
 }
 
 TaskTimer &TraceIntelPT::GetTimer() { return m_task_timer; }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

 
 #include <cstddef>
 
 #include <llvm/ADT/Optional.h>
 
 namespace lldb_private {
 namespace trace_intel_pt {
 
-const size_t kDefaultThreadBufferSize = 4 * 1024;              // 4KB
+const size_t kDefaultTraceBufferSize = 4 * 1024;               // 4KB
 const size_t kDefaultProcessBufferSizeLimit = 5 * 1024 * 1024; // 500MB
 const bool kDefaultEnableTscValue = false;
 const llvm::Optional<size_t> kDefaultPsbPeriod = llvm::None;
+const bool kDefaultPerCoreTracing = false;
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_CONSTANTS_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTOptions.td

Desc<"This value defines the period in which PSB packets will be "
          "contains a hexadecimal number, whose bits represent valid values "
          "e.g. if bit 2 is set, then value 2 is valid. The psb_period value is "
          "converted to the approximate number of raw trace bytes between PSB "
          "packets as: 2 ^ (value + 11), e.g. value 3 means 16KiB between PSB "
          "packets. Defaults to 0 if supported.">;
 }
 
 let Command = "process trace start intel pt" in {
-  def process_trace_start_intel_pt_thread_size: Option<"thread-size", "s">,
+  def process_trace_start_intel_pt_buffer_size: Option<"buffer-size", "s">,
     Group<1>,
     Arg<"Value">,
-    Desc<"Trace size in bytes per thread. It must be a power of 2 greater "
-         "than or equal to 4096 (2^12). The trace is circular keeping "
-         "the most recent data. Defaults to 4096 bytes.">;
+    Desc<"Size in bytes used by each individual per-thread or per-core trace "
+         "buffer. It must be a power of 2 greater than or equal to 4096 (2^12) "
+         "bytes.">;
+  def process_trace_start_intel_pt_per_core_tracing:
+    Option<"per-core-tracing", "c">,
+    Group<1>,
+    Desc<"Instead of having an individual trace buffer per thread, which uses "
+         "a number trace buffers proportional to the number of running "
+         "threads, this option triggers the collection on a per cpu core "
+         "basis. This effectively traces the entire activity on all cores "
+         "using a limited amount of trace buffers regardless of the number of "
+         "threads. This might cause data loss for less frequent threads. This "
+         "option forces the capture of TSC timestamps (see --tsc). Also, this "
+         "option can't be used simulatenously with any other trace sessions "
+         "because of its system-wide nature.">;
   def process_trace_start_intel_pt_process_size_limit: Option<"total-size-limit", "l">,
     Group<1>,
     Arg<"Value">,
     Desc<"Maximum total trace size per process in bytes. This limit applies to "
          "the sum of the sizes of all thread traces of this process, excluding "
          "the ones created with the \"thread trace start\" command. "
          "Whenever a thread is attempted to be traced due to this command and "
          "the limit would be reached, the process is stopped with a "

lldb/source/Utility/TraceIntelPTGDBRemotePackets.cpp

 
 const char *IntelPTDataKinds::kProcFsCpuInfo = "procfsCpuInfo";
 const char *IntelPTDataKinds::kThreadTraceBuffer = "threadTraceBuffer";
 
 bool fromJSON(const json::Value &value, TraceIntelPTStartRequest &packet,
               Path path) {
   ObjectMapper o(value, path);
   if (!o || !fromJSON(value, (TraceStartRequest &)packet, path) ||
-      !o.map("enableTsc", packet.enableTsc) ||
-      !o.map("psbPeriod", packet.psbPeriod) ||
-      !o.map("threadBufferSize", packet.threadBufferSize) ||
-      !o.map("processBufferSizeLimit", packet.processBufferSizeLimit))
+      !o.map("enableTsc", packet.enable_tsc) ||
+      !o.map("psbPeriod", packet.psb_period) ||
+      !o.map("traceBufferSize", packet.trace_buffer_size))
     return false;
-  if (packet.tids && packet.processBufferSizeLimit) {
-    path.report("processBufferSizeLimit must be provided");
-    return false;
-  }
-  if (!packet.tids && !packet.processBufferSizeLimit) {
-    path.report("processBufferSizeLimit must not be provided");
+
+  if (packet.IsProcessTracing()) {
+    if (!o.map("processBufferSizeLimit", packet.process_buffer_size_limit) ||
+        !o.map("perCoreTracing", packet.per_core_tracing))
       return false;
   }
   return true;
 }
 
 json::Value toJSON(const TraceIntelPTStartRequest &packet) {
   json::Value base = toJSON((const TraceStartRequest &)packet);
-  base.getAsObject()->try_emplace("threadBufferSize", packet.threadBufferSize);
-  base.getAsObject()->try_emplace("processBufferSizeLimit",
-                                  packet.processBufferSizeLimit);
-  base.getAsObject()->try_emplace("psbPeriod", packet.psbPeriod);
-  base.getAsObject()->try_emplace("enableTsc", packet.enableTsc);
+  json::Object &obj = *base.getAsObject();
+  obj.try_emplace("traceBufferSize", packet.trace_buffer_size);
+  obj.try_emplace("processBufferSizeLimit", packet.process_buffer_size_limit);
+  obj.try_emplace("psbPeriod", packet.psb_period);
+  obj.try_emplace("enableTsc", packet.enable_tsc);
+  obj.try_emplace("perCoreTracing", packet.per_core_tracing);
   return base;
 }
 
 std::chrono::nanoseconds
 LinuxPerfZeroTscConversion::Convert(uint64_t raw_counter_value) {
   uint64_t quot = raw_counter_value >> m_time_shift;
   uint64_t rem_flag = (((uint64_t)1 << m_time_shift) - 1);
   uint64_t rem = raw_counter_value & rem_flag;

lldb/test/API/commands/trace/TestTraceStartStop.py

class TestTraceStartStop(TraceIntelPTTestCaseBase):
 
     @testSBAPIAndCommands
     def testStartSessionWithWrongSize(self):
         self.expect("file " + os.path.join(self.getSourceDir(), "intelpt-trace", "a.out"))
         self.expect("b main")
         self.expect("r")
 
         self.traceStartThread(
-            error=True, threadBufferSize=2000,
+            error=True, traceBufferSize=2000,
             substrs=["The trace buffer size must be a power of 2", "It was 2000"])
 
         self.traceStartThread(
-            error=True, threadBufferSize=5000,
+            error=True, traceBufferSize=5000,
             substrs=["The trace buffer size must be a power of 2", "It was 5000"])
 
         self.traceStartThread(
-            error=True, threadBufferSize=0,
+            error=True, traceBufferSize=0,
             substrs=["The trace buffer size must be a power of 2", "It was 0"])
 
-        self.traceStartThread(threadBufferSize=1048576)
+        self.traceStartThread(traceBufferSize=1048576)
 
     @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
     def testSBAPIHelp(self):
         self.expect("file " + os.path.join(self.getSourceDir(), "intelpt-trace", "a.out"))
         self.expect("b main")
         self.expect("r")
 
         help = self.getTraceOrCreate().GetStartConfigurationHelp()
-        self.assertIn("threadBufferSize", help)
+        self.assertIn("traceBufferSize", help)
         self.assertIn("processBufferSizeLimit", help)
 
     @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
     def testStoppingAThread(self):
         self.expect("file " + os.path.join(self.getSourceDir(), "intelpt-trace", "a.out"))
         self.expect("b main")
         self.expect("r")
         self.expect("thread trace start")

lldb/test/API/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.py

def testStartMultipleLiveThreadsWithSmallTotalLimit(self):
         self.traceStartProcess(processBufferSizeLimit=5000)
 
         # we get the stop event when trace 2 appears and can't be traced
         self.expect("c", substrs=['Thread', "can't be traced"])
         # we get the stop event when trace 3 appears and can't be traced
         self.expect("c", substrs=['Thread', "can't be traced"])
 
         self.traceStopProcess()
+
+    @skipIf(oslist=no_match(['linux']), archs=no_match(['i386', 'x86_64']))
+    @testSBAPIAndCommands
+    def testStartPerCoreSession(self):
+        self.build()
+        exe = self.getBuildArtifact("a.out")
+        self.dbg.CreateTarget(exe)
+
+        self.expect("b main")
+        self.expect("r")
+
+        self.traceStartProcess(
+            error=True, perCoreTracing=True,
+            substrs=["Per-core tracing is not supported"])