[trace][intel pt] Add a cgroup filter

[wallace]

Summary

Thanks to Gaurav Gaur ([e-mail]), who made most of the investigation to make this happen.

It turns out that cgroup filtering is relatively trivial and works
really nicely. Thid diffs adds automatic cgroup filtering when in
per-cpu mode, unless a new --disable-cgroup-filtering flag is passed in
the start command. At least on Meta machines, all processes are spawned
inside a cgroup by default, which comes super handy, because per cpu
tracing is now much more precise.

A manual test gave me this result

• Without filtering:

Total number of trace items: 36083
Total number of continuous executions found: 229
Number of continuous executions for this thread: 2
Total number of PSB blocks found: 98
Number of PSB blocks for this thread 2
Total number of unattributed PSB blocks found: 38

• With filtering:

Total number of trace items: 87756
Total number of continuous executions found: 123
Number of continuous executions for this thread: 2
Total number of PSB blocks found: 10
Number of PSB blocks for this thread 3
Total number of unattributed PSB blocks found: 2

Filtering gives us great results. The number of instructions collected
more than double (probalby because we have less noise in the trace), and
we have much less unattributed PSBs blocks and unrelated PSBs in
general. The ones that are unrelated probably belong to other processes
in the same cgroup.

[jj10306]

looks good overall, just a couple questions from my end

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

87–90

isn't the cgroup_file path going to have two slashes since slice starts with a slash?

in the case of the image above, wouldn't cgroup_file be "/sys/fs/cgroup//foo.slice/bar.service" instead of "/sys/fs/cgroup/foo.slice/bar.service"

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCpuDecoder.cpp

127

Help me understand this. To me this seems like it is counting all of the PSBs that don't belong to the current thread, whereas I would expect this to only count the PSBs that don't belong to any thread?
So based on my understanding we would be severely overcounting the number of unattributed PSB, but I think I'm just misunderstanding how this code flows.

[wallace]

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

87–90

I think that's fine for system paths. You can have multiple consecutive //// and the system collapses them

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCpuDecoder.cpp

127

the current code is correct, I'll explain:

• on_new_thread_execution is executed on all threads of the same CPU in chronological order
• as on_new_thread_execution is being invoked repeatedly, the it iterator is being traversed and is always moving forwards
• when on_new_thread_execution is invoked, the it iterator will look for psb blocks that happened before the given execution. These blocks do not belong to any thread execution.

Graphically, we have

----exec 1---- ---exec 2---- ---exec 3----
PSB1 PSB2 PSB3 PSB4 PSB5 PSB6

when on_new_thread_execution is invoked for exec2, it will be pointing at PSB3, which is the first PSB after exec 1. PSB3 comes before exec 2, so it'll be unattributed, then it will move to PSB4 and so on

[jj10306]

thanks for answering those questions, lgtm

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

87–90

TIL

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCpuDecoder.cpp

127

makes sense, thanks for the explanation!

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

struct TraceIntelPTStartRequest : TraceStartRequest {
   /// Limit in bytes on all the thread traces started by this "process trace"
   /// instance. When a thread is about to be traced and the limit would be hit,
   /// then a "tracing" stop event is triggered.
   llvm::Optional<uint64_t> process_buffer_size_limit;
 
   /// Whether to have a trace buffer per thread or per cpu cpu.
   llvm::Optional<bool> per_cpu_tracing;
 
+  /// Disable the cgroup filtering that is automatically applied in per cpu
+  /// mode.
+  llvm::Optional<bool> disable_cgroup_filtering;
+
   bool IsPerCpuTracing() const;
 };
 
 bool fromJSON(const llvm::json::Value &value, TraceIntelPTStartRequest &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceIntelPTStartRequest &packet);
 /// \}
struct LinuxPerfZeroTscConversion {
   uint32_t time_mult;
   uint16_t time_shift;
   JSONUINT64 time_zero;
 };
 
 struct TraceIntelPTGetStateResponse : TraceGetStateResponse {
   /// The TSC to wall time conversion if it exists, otherwise \b nullptr.
   llvm::Optional<LinuxPerfZeroTscConversion> tsc_perf_zero_conversion;
+  bool using_cgroup_filtering = false;
 };
 
 bool fromJSON(const llvm::json::Value &value,
               LinuxPerfZeroTscConversion &packet, llvm::json::Path path);
 
 llvm::json::Value toJSON(const LinuxPerfZeroTscConversion &packet);
 
 bool fromJSON(const llvm::json::Value &value,
               TraceIntelPTGetStateResponse &packet, llvm::json::Path path);
 
 llvm::json::Value toJSON(const TraceIntelPTGetStateResponse &packet);
 /// \}
 
 } // namespace lldb_private
 
 #endif // LLDB_UTILITY_TRACEINTELPTGDBREMOTEPACKETS_H

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

 #include "lldb/Utility/StreamString.h"
 
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/MathExtras.h"
 
 #include <algorithm>
 #include <cstddef>
+#include <fcntl.h>
 #include <fstream>
 #include <linux/perf_event.h>
 #include <sstream>
 #include <sys/ioctl.h>
 #include <sys/syscall.h>
 
 using namespace lldb;
 using namespace lldb_private;
if (request.IsProcessTracing()) {
     Error error = Error::success();
     for (int64_t tid : *request.tids)
       error = joinErrors(std::move(error),
                          TraceStop(static_cast<lldb::tid_t>(tid)));
     return error;
   }
 }
 
+/// \return
+///   some file descriptor in /sys/fs/ associated with the cgroup of the given
+///   pid, or \a llvm::None if the pid is not part of a cgroup.
+static Optional<int> GetCGroupFileDescriptor(lldb::pid_t pid) {
+  static Optional<int> fd;
+  if (fd)
+    return fd;
+
+  std::ifstream ifile;
+  ifile.open(formatv("/proc/{0}/cgroup", pid));
+  if (!ifile)
+    return None;
+
+  std::string line;
+  while (std::getline(ifile, line)) {
+    if (line.find("0:") != 0)
+      continue;
+
+    std::string slice = line.substr(line.find_first_of("/"));
+    if (slice.empty())
+      return None;
+    std::string cgroup_file = formatv("/sys/fs/cgroup/{0}", slice);
+    // This cgroup should for the duration of the target, so we don't need to
+    // invoke close ourselves.
+    int maybe_fd = open(cgroup_file.c_str(), O_RDONLY);
+    if (maybe_fd != -1) {
+      fd = maybe_fd;
+      return fd;
+    }
+  }
+  return None;
+}
+
 Error IntelPTCollector::TraceStart(const TraceIntelPTStartRequest &request) {
   if (request.IsProcessTracing()) {
     if (m_process_trace_up) {
       return createStringError(
           inconvertibleErrorCode(),
           "Process currently traced. Stop process tracing first");
     }
     if (request.IsPerCpuTracing()) {
       if (m_thread_traces.GetTracedThreadsCount() > 0)
         return createStringError(
             inconvertibleErrorCode(),
             "Threads currently traced. Stop tracing them first.");
       // CPU tracing is useless if we can't convert tsc to nanos.
       Expected<LinuxPerfZeroTscConversion &> tsc_conversion =
           FetchPerfTscConversionParameters();
       if (!tsc_conversion)
         return tsc_conversion.takeError();
 
-      // We force the enabledment of TSCs, which is needed for correlating the
+      // We force the enablement of TSCs, which is needed for correlating the
       // cpu traces.
       TraceIntelPTStartRequest effective_request = request;
       effective_request.enable_tsc = true;
 
+      // We try to use cgroup filtering whenever possible
+      Optional<int> cgroup_fd;
+      if (!request.disable_cgroup_filtering.getValueOr(false))
+        cgroup_fd = GetCGroupFileDescriptor(m_process.GetID());
+
       if (Expected<IntelPTProcessTraceUP> trace =
               IntelPTMultiCoreTrace::StartOnAllCores(effective_request,
-                                                     m_process)) {
+                                                     m_process, cgroup_fd)) {
         m_process_trace_up = std::move(*trace);
         return Error::success();
       } else {
         return trace.takeError();
       }
     } else {
       std::vector<lldb::tid_t> process_threads;
       for (NativeThreadProtocol &thread : m_process.Threads())

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.h

public:
   /// Start tracing all CPU cores.
   ///
   /// \param[in] request
   ///   Intel PT configuration parameters.
   ///
   /// \param[in] process
   ///   The process being debugged.
   ///
+  ///  \param[in] cgroup_fd
+  ///  A file descriptor in /sys/fs associated with the cgroup of the process to
+  ///  trace. If not \a llvm::None, then the trace sesion will use cgroup
+  ///  filtering.
+  ///
   /// \return
   ///   An \a IntelPTMultiCoreTrace instance if tracing was successful, or
   ///   an \a llvm::Error otherwise.
   static llvm::Expected<std::unique_ptr<IntelPTMultiCoreTrace>>
   StartOnAllCores(const TraceIntelPTStartRequest &request,
-                  NativeProcessProtocol &process);
+                  NativeProcessProtocol &process,
+                  llvm::Optional<int> cgroup_fd = llvm::None);
 
   /// Execute the provided callback on each core that is being traced.
   ///
   /// \param[in] callback.cpu_id
   ///   The core id that is being traced.
   ///
   /// \param[in] callback.core_trace
   ///   The single-buffer trace instance for the given core.

 
 private:
   /// This assumes that all underlying perf_events for each core are part of the
   /// same perf event group.
   IntelPTMultiCoreTrace(
       llvm::DenseMap<lldb::cpu_id_t,
                      std::pair<IntelPTSingleBufferTrace, ContextSwitchTrace>>
           &&traces_per_core,
-      NativeProcessProtocol &process)
-      : m_traces_per_core(std::move(traces_per_core)), m_process(process) {}
+      NativeProcessProtocol &process, bool using_cgroup_filtering)
+      : m_traces_per_core(std::move(traces_per_core)), m_process(process),
+        m_using_cgroup_filtering(using_cgroup_filtering) {}
 
   llvm::DenseMap<lldb::cpu_id_t,
                  std::pair<IntelPTSingleBufferTrace, ContextSwitchTrace>>
       m_traces_per_core;
 
   /// The target process.
   NativeProcessProtocol &m_process;
+  bool m_using_cgroup_filtering;
 };
 
 } // namespace process_linux
 } // namespace lldb_private
 
 #endif // liblldb_IntelPTMultiCoreTrace_H_

lldb/source/Plugins/Process/Linux/IntelPTMultiCoreTrace.cpp

return createStringError(
       inconvertibleErrorCode(),
       "%s\nYou might need to rerun as sudo or to set "
       "/proc/sys/kernel/perf_event_paranoid to a value of 0 or -1.",
       toString(std::move(error)).c_str());
 }
 
 Expected<std::unique_ptr<IntelPTMultiCoreTrace>>
 IntelPTMultiCoreTrace::StartOnAllCores(const TraceIntelPTStartRequest &request,
-                                       NativeProcessProtocol &process) {
+                                       NativeProcessProtocol &process,
+                                       Optional<int> cgroup_fd) {
   Expected<ArrayRef<cpu_id_t>> cpu_ids = GetAvailableLogicalCoreIDs();
   if (!cpu_ids)
     return cpu_ids.takeError();
 
   if (IsTotalBufferLimitReached(*cpu_ids, request))
     return createStringError(
         inconvertibleErrorCode(),
         "The process can't be traced because the process trace size limit "
         "has been reached. Consider retracing with a higher limit.");
 
   DenseMap<cpu_id_t, std::pair<IntelPTSingleBufferTrace, ContextSwitchTrace>>
       traces;
 
   for (cpu_id_t cpu_id : *cpu_ids) {
     Expected<IntelPTSingleBufferTrace> core_trace =
         IntelPTSingleBufferTrace::Start(request, /*tid=*/None, cpu_id,
-                                        /*disabled=*/true);
+                                        /*disabled=*/true, cgroup_fd);
     if (!core_trace)
       return IncludePerfEventParanoidMessageInError(core_trace.takeError());
 
     if (Expected<PerfEvent> context_switch_trace =
             CreateContextSwitchTracePerfEvent(cpu_id,
                                               &core_trace->GetPerfEvent())) {
       traces.try_emplace(cpu_id,
                          std::make_pair(std::move(*core_trace),
                                         std::move(*context_switch_trace)));
     } else {
       return context_switch_trace.takeError();
     }
   }
 
   return std::unique_ptr<IntelPTMultiCoreTrace>(
-      new IntelPTMultiCoreTrace(std::move(traces), process));
+      new IntelPTMultiCoreTrace(std::move(traces), process, (bool)cgroup_fd));
 }
 
 void IntelPTMultiCoreTrace::ForEachCore(
     std::function<void(cpu_id_t cpu_id, IntelPTSingleBufferTrace &core_trace)>
         callback) {
   for (auto &it : m_traces_per_core)
     callback(it.first, it.second.first);
 }
if (Error err = core_trace.Resume()) {
       LLDB_LOG_ERROR(GetLog(POSIXLog::Trace), std::move(err),
                      "Unable to resume the core trace for core {0}", cpu_id);
     }
   });
 }
 
 TraceIntelPTGetStateResponse IntelPTMultiCoreTrace::GetState() {
   TraceIntelPTGetStateResponse state;
+  state.using_cgroup_filtering = m_using_cgroup_filtering;
 
   for (NativeThreadProtocol &thread : m_process.Threads())
     state.traced_threads.push_back(
         TraceThreadState{thread.GetID(), {}});
 
   state.cpus.emplace();
   ForEachCore([&](lldb::cpu_id_t cpu_id,
                   const IntelPTSingleBufferTrace &core_trace,

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.h

public:
   /// \param[in] cpu_id
   ///     The CPU core id where to trace. If \b None, then this traces all CPUs.
   ///
   /// \param[in] disabled
   ///     If \b true, then no data is collected until \a Resume is invoked.
   ///     Similarly, if \b false, data is collected right away until \a Pause is
   ///     invoked.
   ///
+  ///  \param[in] cgroup_fd
+  ///   A file descriptor in /sys/fs associated with the cgroup of the process
+  ///   to trace. If not \a llvm::None, then the trace sesion will use cgroup
+  ///   filtering.
+  ///
   /// \return
   ///   A \a IntelPTSingleBufferTrace instance if tracing was successful, or
   ///   an \a llvm::Error otherwise.
   static llvm::Expected<IntelPTSingleBufferTrace>
   Start(const TraceIntelPTStartRequest &request,
         llvm::Optional<lldb::tid_t> tid,
         llvm::Optional<lldb::cpu_id_t> cpu_id = llvm::None,
-        bool disabled = false);
+        bool disabled = false, llvm::Optional<int> cgroup_fd = llvm::None);
 
   /// \return
   ///    The bytes requested by a jLLDBTraceGetBinaryData packet that was routed
   ///    to this trace instace.
   llvm::Expected<std::vector<uint8_t>>
   GetBinaryData(const TraceGetBinaryDataRequest &request) const;
 
   /// Read the intel pt trace buffer managed by this trace instance. To ensure

lldb/source/Plugins/Process/Linux/IntelPTSingleBufferTrace.cpp

Expected<std::vector<uint8_t>> IntelPTSingleBufferTrace::GetIptTrace() {
   // clearing IA32_RTIT_CTL.TraceEn (see “Disabling Packet Generation” in
   // Section 35.2.7.2).
   //
   // This is achieved by the PERF_EVENT_IOC_DISABLE ioctl request, as
   // mentioned in the man page of perf_event_open.
   return m_perf_event.GetReadOnlyAuxBuffer();
 }
 
-Expected<IntelPTSingleBufferTrace>
-IntelPTSingleBufferTrace::Start(const TraceIntelPTStartRequest &request,
-                                Optional<lldb::tid_t> tid,
-                                Optional<cpu_id_t> cpu_id, bool disabled) {
+Expected<IntelPTSingleBufferTrace> IntelPTSingleBufferTrace::Start(
+    const TraceIntelPTStartRequest &request, Optional<lldb::tid_t> tid,
+    Optional<cpu_id_t> cpu_id, bool disabled, Optional<int> cgroup_fd) {
 #ifndef PERF_ATTR_SIZE_VER5
   return createStringError(inconvertibleErrorCode(),
                            "Intel PT Linux perf event not supported");
 #else
   Log *log = GetLog(POSIXLog::Trace);
 
   LLDB_LOG(log, "Will start tracing thread id {0} and cpu id {1}", tid, cpu_id);
 
Expected<perf_event_attr> attr = CreateIntelPTPerfEventConfiguration(
         return static_cast<uint64_t>(value);
       }));
   if (!attr)
     return attr.takeError();
   attr->disabled = disabled;
 
   LLDB_LOG(log, "Will create intel pt trace buffer of size {0}",
            request.ipt_trace_size);
+  unsigned long flags = 0;
+  if (cgroup_fd) {
+    tid = *cgroup_fd;
+    flags |= PERF_FLAG_PID_CGROUP;
+  }
 
-  if (Expected<PerfEvent> perf_event = PerfEvent::Init(*attr, tid, cpu_id)) {
+  if (Expected<PerfEvent> perf_event =
+          PerfEvent::Init(*attr, tid, cpu_id, -1, flags)) {
     if (Error mmap_err = perf_event->MmapMetadataAndBuffers(
             /*num_data_pages=*/0, aux_buffer_numpages,
             /*data_buffer_write=*/true)) {
       return std::move(mmap_err);
     }
     return IntelPTSingleBufferTrace(std::move(*perf_event));
   } else {
     return perf_event.takeError();
   }
 #endif
 }
 
 const PerfEvent &IntelPTSingleBufferTrace::GetPerfEvent() const {
   return m_perf_event;
 }

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.h

public:
 
     llvm::ArrayRef<OptionDefinition> GetDefinitions() override;
 
     uint64_t m_ipt_trace_size;
     uint64_t m_process_buffer_size_limit;
     bool m_enable_tsc;
     llvm::Optional<uint64_t> m_psb_period;
     bool m_per_cpu_tracing;
+    bool m_disable_cgroup_filtering;
   };
 
   CommandObjectProcessTraceStartIntelPT(TraceIntelPT &trace,
                                         CommandInterpreter &interpreter)
       : CommandObjectParsed(
             interpreter, "process trace start",
             "Start tracing this process with intel-pt, including future "
             "threads. If --per-cpu-tracing is not provided, this traces each "

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.cpp

Status CommandObjectProcessTraceStartIntelPT::CommandOptions::SetOptionValue(
   case 't': {
     m_enable_tsc = true;
     break;
   }
   case 'c': {
     m_per_cpu_tracing = true;
     break;
   }
+  case 'd': {
+    m_disable_cgroup_filtering = true;
+    break;
+  }
   case 'p': {
     int64_t psb_period;
     if (option_arg.empty() || option_arg.getAsInteger(0, psb_period) ||
         psb_period < 0)
       error.SetErrorStringWithFormat("invalid integer value for option '%s'",
                                      option_arg.str().c_str());
     else
       m_psb_period = psb_period;
     break;
   }
   default:
     llvm_unreachable("Unimplemented option");
   }
   return error;
 }
 
 void CommandObjectProcessTraceStartIntelPT::CommandOptions::
     OptionParsingStarting(ExecutionContext *execution_context) {
   m_ipt_trace_size = kDefaultIptTraceSize;
   m_process_buffer_size_limit = kDefaultProcessBufferSizeLimit;
   m_enable_tsc = kDefaultEnableTscValue;
   m_psb_period = kDefaultPsbPeriod;
   m_per_cpu_tracing = kDefaultPerCpuTracing;
+  m_disable_cgroup_filtering = kDefaultDisableCgroupFiltering;
 }
 
 llvm::ArrayRef<OptionDefinition>
 CommandObjectProcessTraceStartIntelPT::CommandOptions::GetDefinitions() {
   return llvm::makeArrayRef(g_process_trace_start_intel_pt_options);
 }
 
 bool CommandObjectProcessTraceStartIntelPT::DoExecute(
     Args &command, CommandReturnObject &result) {
-  if (Error err = m_trace.Start(m_options.m_ipt_trace_size,
-                                m_options.m_process_buffer_size_limit,
-                                m_options.m_enable_tsc, m_options.m_psb_period,
-                                m_options.m_per_cpu_tracing))
+  if (Error err = m_trace.Start(
+          m_options.m_ipt_trace_size, m_options.m_process_buffer_size_limit,
+          m_options.m_per_cpu_tracing, m_options.m_disable_cgroup_filtering))
     result.SetError(Status(std::move(err)));
   else
     result.SetStatus(eReturnStatusSuccessFinishResult);
 
   return result.Succeeded();
 }
 
 Optional<uint64_t>

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

public:
   ///
   /// \param[in] psb_period
   ///     This value defines the period in which PSB packets will be generated.
   ///
   /// \param[in] per_cpu_tracing
   ///     This value defines whether to have an intel pt trace buffer per thread
   ///     or per cpu core.
   ///
+  /// \param[in] disable_cgroup_filtering
+  ///     Disable the cgroup filtering that is automatically applied when doing
+  ///     per cpu tracing.
+  ///
   /// \return
   ///     \a llvm::Error::success if the operation was successful, or
   ///     \a llvm::Error otherwise.
   llvm::Error Start(uint64_t ipt_trace_size, uint64_t total_buffer_size_limit,
                     bool enable_tsc, llvm::Optional<uint64_t> psb_period,
-                    bool m_per_cpu_tracing);
+                    bool m_per_cpu_tracing, bool disable_cgroup_filtering);
 
   /// \copydoc Trace::Start
   llvm::Error Start(StructuredData::ObjectSP configuration =
                         StructuredData::ObjectSP()) override;
 
   /// Start tracing live threads.
   ///
   /// More information on the parameters below can be found in the

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

void TraceIntelPT::DumpTraceInfo(Thread &thread, Stream &s, bool verbose) {
 
   if (storage.multicpu_decoder) {
     s << "\n  Multi-cpu decoding:\n";
     s.Format("    Total number of continuous executions found: {0}\n",
              storage.multicpu_decoder->GetTotalContinuousExecutionsCount());
     s.Format(
         "    Number of continuous executions for this thread: {0}\n",
         storage.multicpu_decoder->GetNumContinuousExecutionsForThread(tid));
+    s.Format("    Total number of PSB blocks found: {0}\n",
+             storage.multicpu_decoder->GetTotalPSBBlocksCount());
+    s.Format("    Number of PSB blocks for this thread {0}\n",
+             storage.multicpu_decoder->GePSBBlocksCountForThread(tid));
+    s.Format("    Total number of unattributed PSB blocks found: {0}\n",
+             storage.multicpu_decoder->GetUnattributedPSBBlocksCount());
   }
 
   // Errors
   {
     s << "\n  Errors:\n";
     const DecodedThread::LibiptErrorsStats &tsc_errors_stats =
         decoded_thread_sp->GetTscErrorsStats();
     s.Format("    Number of TSC decoding errors: {0}\n",
const char *TraceIntelPT::GetStartConfigurationHelp() {
 
   - int psbPeriod (defaults to {2}):
     [process and thread tracing]
 
   - boolean perCpuTracing (default to {3}):
     [process tracing only]
 
   - int processBufferSizeLimit (defaults to {4} MiB):
+    [process tracing only]
+
+  - boolean disableCgroupFiltering (default to {5}):
     [process tracing only])",
                             kDefaultIptTraceSize, kDefaultEnableTscValue,
                             kDefaultPsbPeriod, kDefaultPerCpuTracing,
-                            kDefaultProcessBufferSizeLimit / 1024 / 1024));
+                            kDefaultProcessBufferSizeLimit / 1024 / 1024,
+                            kDefaultDisableCgroupFiltering));
   }
   return message->c_str();
 }
 
 Error TraceIntelPT::Start(uint64_t ipt_trace_size,
                           uint64_t total_buffer_size_limit, bool enable_tsc,
-                          Optional<uint64_t> psb_period, bool per_cpu_tracing) {
+                          Optional<uint64_t> psb_period, bool per_cpu_tracing,
+                          bool disable_cgroup_filtering) {
   TraceIntelPTStartRequest request;
   request.ipt_trace_size = ipt_trace_size;
   request.process_buffer_size_limit = total_buffer_size_limit;
   request.enable_tsc = enable_tsc;
   request.psb_period = psb_period;
   request.type = GetPluginName().str();
   request.per_cpu_tracing = per_cpu_tracing;
+  request.disable_cgroup_filtering = disable_cgroup_filtering;
   return Trace::Start(toJSON(request));
 }
 
 Error TraceIntelPT::Start(StructuredData::ObjectSP configuration) {
   uint64_t ipt_trace_size = kDefaultIptTraceSize;
   uint64_t process_buffer_size_limit = kDefaultProcessBufferSizeLimit;
   bool enable_tsc = kDefaultEnableTscValue;
   Optional<uint64_t> psb_period = kDefaultPsbPeriod;
   bool per_cpu_tracing = kDefaultPerCpuTracing;
+  bool disable_cgroup_filtering = kDefaultDisableCgroupFiltering;
 
   if (configuration) {
     if (StructuredData::Dictionary *dict = configuration->GetAsDictionary()) {
       dict->GetValueForKeyAsInteger("iptTraceSize", ipt_trace_size);
       dict->GetValueForKeyAsInteger("processBufferSizeLimit",
                                     process_buffer_size_limit);
       dict->GetValueForKeyAsBoolean("enableTsc", enable_tsc);
       dict->GetValueForKeyAsInteger("psbPeriod", psb_period);
       dict->GetValueForKeyAsBoolean("perCpuTracing", per_cpu_tracing);
+      dict->GetValueForKeyAsBoolean("disableCgroupFiltering",
+                                    disable_cgroup_filtering);
     } else {
       return createStringError(inconvertibleErrorCode(),
                                "configuration object is not a dictionary");
     }
   }
 
   return Start(ipt_trace_size, process_buffer_size_limit, enable_tsc,
-               psb_period, per_cpu_tracing);
+               psb_period, per_cpu_tracing, disable_cgroup_filtering);
 }
 
 llvm::Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,
                                 uint64_t ipt_trace_size, bool enable_tsc,
                                 Optional<uint64_t> psb_period) {
   TraceIntelPTStartRequest request;
   request.ipt_trace_size = ipt_trace_size;
   request.enable_tsc = enable_tsc;

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

 namespace lldb_private {
 namespace trace_intel_pt {
 
 const size_t kDefaultIptTraceSize = 4 * 1024;                  // 4KB
 const size_t kDefaultProcessBufferSizeLimit = 5 * 1024 * 1024; // 500MB
 const bool kDefaultEnableTscValue = false;
 const llvm::Optional<size_t> kDefaultPsbPeriod = llvm::None;
 const bool kDefaultPerCpuTracing = false;
+const bool kDefaultDisableCgroupFiltering = false;
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_CONSTANTS_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCpuDecoder.h

public:
   ///   whether there's tracing data associated to it or not.
   bool TracesThread(lldb::tid_t tid) const;
 
   /// \return
   ///   The number of continuous executions found for the given \p tid.
   size_t GetNumContinuousExecutionsForThread(lldb::tid_t tid) const;
 
   /// \return
+  ///   The number of PSB blocks for a given thread in all cores.
+  size_t GePSBBlocksCountForThread(lldb::tid_t tid) const;
+
+  /// \return
   ///   The total number of continuous executions found across CPUs.
   size_t GetTotalContinuousExecutionsCount() const;
 
+  /// \return
+  ///   The number of psb blocks in all cores that couldn't be matched with a
+  ///   thread execution coming from context switch traces.
+  size_t GetUnattributedPSBBlocksCount() const;
+
+  /// \return
+  ///   The total number of PSB blocks in all cores.
+  size_t GetTotalPSBBlocksCount() const;
+
 private:
   /// Traverse the context switch traces and the basic intel pt continuous
   /// subtraces and produce a list of continuous executions for each process and
   /// thread.
   ///
   /// See \a DoCorrelateContextSwitchesAndIntelPtTraces.
   ///
   /// Any errors are stored in \a m_setup_error.
private:
   std::set<lldb::tid_t> m_tids;
   llvm::Optional<
       llvm::DenseMap<lldb::tid_t, std::vector<IntelPTThreadContinousExecution>>>
       m_continuous_executions_per_thread;
   llvm::DenseMap<lldb::tid_t, DecodedThreadSP> m_decoded_threads;
   /// This variable will be non-None if a severe error happened during the setup
   /// of the decoder and we don't want decoding to be reattempted.
   llvm::Optional<std::string> m_setup_error;
-  uint64_t m_unattributed_intelpt_subtraces;
+  uint64_t m_unattributed_psb_blocks = 0;
+  uint64_t m_total_psb_blocks = 0;
 };
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPTMULTICPUDECODER_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCpuDecoder.cpp

TraceIntelPTMultiCpuDecoder::DoCorrelateContextSwitchesAndIntelPtTraces() {
   LinuxPerfZeroTscConversion tsc_conversion = *conv_opt;
 
   for (cpu_id_t cpu_id : trace_sp->GetTracedCpus()) {
     Expected<std::vector<IntelPTThreadSubtrace>> intel_pt_subtraces =
         GetIntelPTSubtracesForCpu(*trace_sp, cpu_id);
     if (!intel_pt_subtraces)
       return intel_pt_subtraces.takeError();
 
+    m_total_psb_blocks += intel_pt_subtraces->size();
     // We'll be iterating through the thread continuous executions and the intel
     // pt subtraces sorted by time.
     auto it = intel_pt_subtraces->begin();
     auto on_new_thread_execution =
         [&](const ThreadContinuousExecution &thread_execution) {
           IntelPTThreadContinousExecution execution(thread_execution);
 
           for (; it != intel_pt_subtraces->end() &&
                  it->tsc < thread_execution.GetEndTSC();
                it++) {
             if (it->tsc > thread_execution.GetStartTSC()) {
               execution.intelpt_subtraces.push_back(*it);
             } else {
-              m_unattributed_intelpt_subtraces++;
+              m_unattributed_psb_blocks++;
             }
           }
           continuous_executions_per_thread[thread_execution.tid].push_back(
               execution);
         };
     Error err = trace_sp->OnCpuBinaryDataRead(
         cpu_id, IntelPTDataKinds::kPerfContextSwitchTrace,
         [&](ArrayRef<uint8_t> data) -> Error {
           Expected<std::vector<ThreadContinuousExecution>> executions =
               DecodePerfContextSwitchTrace(data, cpu_id, tsc_conversion);
           if (!executions)
             return executions.takeError();
           for (const ThreadContinuousExecution &exec : *executions)
             on_new_thread_execution(exec);
           return Error::success();
         });
     if (err)
       return std::move(err);
+
+    m_unattributed_psb_blocks += intel_pt_subtraces->end() - it;
   }
   // We now sort the executions of each thread to have them ready for
   // instruction decoding
   for (auto &tid_executions : continuous_executions_per_thread)
     std::sort(tid_executions.second.begin(), tid_executions.second.end());
 
   return continuous_executions_per_thread;
 }

 size_t TraceIntelPTMultiCpuDecoder::GetTotalContinuousExecutionsCount() const {
   if (!m_continuous_executions_per_thread)
     return 0;
   size_t count = 0;
   for (const auto &kv : *m_continuous_executions_per_thread)
     count += kv.second.size();
   return count;
 }
+
+size_t
+TraceIntelPTMultiCpuDecoder::GePSBBlocksCountForThread(lldb::tid_t tid) const {
+  if (!m_continuous_executions_per_thread)
+    return 0;
+  size_t count = 0;
+  auto it = m_continuous_executions_per_thread->find(tid);
+  for (const IntelPTThreadContinousExecution &execution : it->second)
+    count += execution.intelpt_subtraces.size();
+  return count;
+}
+
+size_t TraceIntelPTMultiCpuDecoder::GetUnattributedPSBBlocksCount() const {
+  return m_unattributed_psb_blocks;
+}
+
+size_t TraceIntelPTMultiCpuDecoder::GetTotalPSBBlocksCount() const {
+  return m_total_psb_blocks;
+}

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTOptions.td

Desc<"This value defines the period in which PSB packets will be "
          "the PSB period will be defined by the processor. If supported, valid "
          "values for this period can be found in "
          "/sys/bus/event_source/devices/intel_pt/caps/psb_periods which "
          "contains a hexadecimal number, whose bits represent valid values "
          "e.g. if bit 2 is set, then value 2 is valid. The psb_period value is "
          "converted to the approximate number of raw trace bytes between PSB "
          "packets as: 2 ^ (value + 11), e.g. value 3 means 16KiB between PSB "
          "packets. Defaults to 0 if supported.">;
+  def process_trace_start_intel_pt_disable_cgroup_filtering:
+    Option<"disable-cgroup-filtering", "d">,
+    Desc<"Disable the automatic cgroup filtering that is applied if --per-cpu "
+         "is provided. Cgroup filtering allows collecting intel pt data "
+         "exclusively of processes of the same cgroup as the target.">;
 }

lldb/source/Utility/TraceIntelPTGDBRemotePackets.cpp

bool fromJSON(const json::Value &value, TraceIntelPTStartRequest &packet,
   if (!(o && fromJSON(value, (TraceStartRequest &)packet, path) &&
         o.map("enableTsc", packet.enable_tsc) &&
         o.map("psbPeriod", packet.psb_period) &&
         o.map("iptTraceSize", packet.ipt_trace_size)))
     return false;
 
   if (packet.IsProcessTracing()) {
     if (!o.map("processBufferSizeLimit", packet.process_buffer_size_limit) ||
-        !o.map("perCpuTracing", packet.per_cpu_tracing))
+        !o.map("perCpuTracing", packet.per_cpu_tracing) ||
+        !o.map("disableCgroupTracing", packet.disable_cgroup_filtering))
       return false;
   }
   return true;
 }
 
 json::Value toJSON(const TraceIntelPTStartRequest &packet) {
   json::Value base = toJSON((const TraceStartRequest &)packet);
   json::Object &obj = *base.getAsObject();
   obj.try_emplace("iptTraceSize", packet.ipt_trace_size);
   obj.try_emplace("processBufferSizeLimit", packet.process_buffer_size_limit);
   obj.try_emplace("psbPeriod", packet.psb_period);
   obj.try_emplace("enableTsc", packet.enable_tsc);
   obj.try_emplace("perCpuTracing", packet.per_cpu_tracing);
+  obj.try_emplace("disableCgroupTracing", packet.disable_cgroup_filtering);
   return base;
 }
 
 uint64_t LinuxPerfZeroTscConversion::ToNanos(uint64_t tsc) const {
   uint64_t quot = tsc >> time_shift;
   uint64_t rem_flag = (((uint64_t)1 << time_shift) - 1);
   uint64_t rem = tsc & rem_flag;
   return time_zero.value + quot * time_mult + ((rem * time_mult) >> time_shift);
bool fromJSON(const json::Value &value, LinuxPerfZeroTscConversion &packet,
   packet.time_shift = time_shift;
   return true;
 }
 
 bool fromJSON(const json::Value &value, TraceIntelPTGetStateResponse &packet,
               json::Path path) {
   ObjectMapper o(value, path);
   return o && fromJSON(value, (TraceGetStateResponse &)packet, path) &&
-         o.map("tscPerfZeroConversion", packet.tsc_perf_zero_conversion);
+         o.map("tscPerfZeroConversion", packet.tsc_perf_zero_conversion) &&
+         o.map("usingCgroupFiltering", packet.using_cgroup_filtering);
 }
 
 json::Value toJSON(const TraceIntelPTGetStateResponse &packet) {
   json::Value base = toJSON((const TraceGetStateResponse &)packet);
-  base.getAsObject()->insert(
-      {"tscPerfZeroConversion", packet.tsc_perf_zero_conversion});
+  json::Object &obj = *base.getAsObject();
+  obj.insert({"tscPerfZeroConversion", packet.tsc_perf_zero_conversion});
+  obj.insert({"usingCgroupFiltering", packet.using_cgroup_filtering});
   return base;
 }
 
 } // namespace lldb_private