[trace][intelpt] Support system-wide tracing [1] - Add a method for accessing the list of logical core ids

[wallace]

Summary

In order to open perf events per core, we need to first get the list of
core ids available in the system. So I'm adding a function that does
that by parsing /proc/cpuinfo. That seems to be the simplest and most
portable way to do that.

Besides that, I made a few refactors and renames to reflect better that
the cpu info that we use in lldb-server comes from procfs.

[jj10306]

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

22

why not use an enum here?
having "kinds" in the name already makes it sound like an enum and seeing IntelPTDataKinds:: everywhere immediately made me thing it was an enum.

[wallace]

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

22

this is actually an enum, but it's an enum of strings. So we have a few options:

• have a normal int enum and have another function that converts int -> const char*.
• create a fake enum with class static variables like here
• use header inline variables, but they are not available in the C++ version used by LLDB.

Overall I found this to be the simplest way to implement it.

[wallace]

I made a mistake and I included the second of two commits. This update contains all the code.

[jj10306]

lldb/unittests/Process/Linux/PerfTests.cpp

53

Not directly related to this diff, but I just realized the proc fs parsing logic doesn't belong in the Perf files since it has nothing to do with perf. Feel free to move those functions and these associated tests to a more appropriate location or we can do it in a future diff since this isn't the diff that made that mistake.

[wallace]

lldb/unittests/Process/Linux/PerfTests.cpp

53

that's a valid point. I'll create a new file

[wallace]

move procfs functions to Procfs.h

[wallace]

final nits

[jj10306]

Much cleaner now, thanks for separating out the procfs logic 🙂

[wallace]

thanks for your review :)

lldb/docs/lldb-gdb-remote.txt

 // NOTES
 //   - "traceThreads" includes all thread traced by both "process tracing" and
 //     "thread tracing".
 //
 // INTEL PT
 //
 //  Binary data kinds:
 //    - threadTraceBuffer: trace buffer for a thread.
-//    - cpuInfo: contents of the /proc/cpuinfo file.
+//    - procFsCpuInfo: contents of the /proc/cpuinfo file.
 //
 //  Counter info kinds:
 //    tsc-perf-zero-conversion:
 //
 //    This field allows converting Intel processor's TSC values to a wall time.
 //    It is available through the Linux perf_event API when cap_user_time and cap_user_time_zero
 //    are set.
 //    See the documentation of time_zero in

 //   "size": <decimal>,
 //      Number of bytes in to read starting from the offset.
 //  }
 //
 // INTEL PT
 //
 //  Binary data kinds:
 //    - threadTraceBuffer: trace buffer for a thread.
-//    - cpuInfo: contents of the /proc/cpuinfo file.
+//    - procFsCpuInfo: contents of the /proc/cpuinfo file.
 //----------------------------------------------------------------------
 
 send packet: jLLDBTraceGetBinaryData:{"type":<type>,"kind":<query>,"tid":<tid>,"offset":<offset>,"size":<size>}]
 read packet: <binary data>/E<error code>;AAAAAAAAA
 
 //----------------------------------------------------------------------
 // "qRegisterInfo<hex-reg-id>"
 //

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

 
 #include "llvm/Support/JSON.h"
 
 #include <chrono>
 
 /// See docs/lldb-gdb-remote.txt for more information.
 namespace lldb_private {
 
+// List of data kinds used by jLLDBGetState and jLLDBGetBinaryData.
+struct IntelPTDataKinds {
+  static const char *kProcFsCpuInfo;
+  static const char *kThreadTraceBuffer;
+};
+
 /// jLLDBTraceStart gdb-remote packet
 /// \{
 struct TraceIntelPTStartRequest : TraceStartRequest {
   /// Size in bytes to use for each thread's trace buffer.
   int64_t threadBufferSize;
 
   /// Whether to enable TSC
   bool enableTsc;

lldb/source/Plugins/Process/Linux/IntelPTCollector.cpp

enum IntelPTConfigFileType {
   ZeroOne,
   Decimal,
   // a bit index file always starts with the prefix config: following by an int,
   // which represents the offset of the perf_event_attr.config value where to
   // store a given configuration.
   BitOffset
 };
 
-/// Get the content of /proc/cpuinfo that can be later used to decode traces.
-static Expected<ArrayRef<uint8_t>> GetCPUInfo() {
-  static llvm::Optional<std::vector<uint8_t>> cpu_info;
-  if (!cpu_info) {
-    auto buffer_or_error = errorOrToExpected(getProcFile("cpuinfo"));
-    if (!buffer_or_error)
-      return buffer_or_error.takeError();
-    MemoryBuffer &buffer = **buffer_or_error;
-    cpu_info = std::vector<uint8_t>(
-        reinterpret_cast<const uint8_t *>(buffer.getBufferStart()),
-        reinterpret_cast<const uint8_t *>(buffer.getBufferEnd()));
-  }
-  return *cpu_info;
-}
-
 static Expected<uint32_t> ReadIntelPTConfigFile(const char *file,
                                                 IntelPTConfigFileType type) {
   ErrorOr<std::unique_ptr<MemoryBuffer>> stream =
       MemoryBuffer::getFileAsStream(file);
 
   if (!stream)
     return createStringError(inconvertibleErrorCode(),
                              "Can't open the file '%s'", file);
for (auto part : parts) {
     next = std::copy_n(part.begin(), chunk_size, next);
     bytes_left -= chunk_size;
   }
   dst = dst.drop_back(bytes_left);
 }
 
 TraceThreadState IntelPTThreadTrace::GetState() const {
   return {static_cast<int64_t>(m_tid),
-          {TraceBinaryData{"threadTraceBuffer",
+          {TraceBinaryData{IntelPTDataKinds::kThreadTraceBuffer,
                            static_cast<int64_t>(GetTraceBufferSize())}}};
 }
 
 /// IntelPTThreadTraceCollection
 
 bool IntelPTThreadTraceCollection::TracesThread(lldb::tid_t tid) const {
   return m_thread_traces.count(tid);
 }
Error IntelPTCollector::OnThreadDestroyed(lldb::tid_t tid) {
   if (IsProcessTracingEnabled() && m_process_trace->TracesThread(tid))
     return m_process_trace->TraceStop(tid);
   else if (m_thread_traces.TracesThread(tid))
     return m_thread_traces.TraceStop(tid);
   return Error::success();
 }
 
 Expected<json::Value> IntelPTCollector::GetState() const {
-  Expected<ArrayRef<uint8_t>> cpu_info = GetCPUInfo();
+  Expected<ArrayRef<uint8_t>> cpu_info = GetProcFsCpuInfo();
   if (!cpu_info)
     return cpu_info.takeError();
 
   TraceGetStateResponse state;
-  state.processBinaryData.push_back(
-      {"cpuInfo", static_cast<int64_t>(cpu_info->size())});
+  state.processBinaryData.push_back({IntelPTDataKinds::kProcFsCpuInfo,
+                                     static_cast<int64_t>(cpu_info->size())});
 
   std::vector<TraceThreadState> thread_states =
       m_thread_traces.GetThreadStates();
   state.tracedThreads.insert(state.tracedThreads.end(), thread_states.begin(),
                              thread_states.end());
 
   if (IsProcessTracingEnabled()) {
     thread_states = m_process_trace->GetThreadTraces().GetThreadStates();
     state.tracedThreads.insert(state.tracedThreads.end(), thread_states.begin(),
                                thread_states.end());
   }
   return toJSON(state);
 }
 
 Expected<const IntelPTThreadTrace &>
 IntelPTCollector::GetTracedThread(lldb::tid_t tid) const {
   if (IsProcessTracingEnabled() && m_process_trace->TracesThread(tid))
     return m_process_trace->GetThreadTraces().GetTracedThread(tid);
   return m_thread_traces.GetTracedThread(tid);
 }
 
 Expected<std::vector<uint8_t>>
 IntelPTCollector::GetBinaryData(const TraceGetBinaryDataRequest &request) const {
-  if (request.kind == "threadTraceBuffer") {
+  if (request.kind == IntelPTDataKinds::kThreadTraceBuffer) {
     if (Expected<const IntelPTThreadTrace &> trace =
             GetTracedThread(*request.tid))
       return trace->GetIntelPTBuffer(request.offset, request.size);
     else
       return trace.takeError();
-  } else if (request.kind == "cpuInfo") {
-    return GetCPUInfo();
+  } else if (request.kind == IntelPTDataKinds::kProcFsCpuInfo) {
+    return GetProcFsCpuInfo();
   }
   return createStringError(inconvertibleErrorCode(),
                            "Unsuported trace binary data kind: %s",
                            request.kind.c_str());
 }
 
 void IntelPTCollector::ClearProcessTracing() { m_process_trace = None; }
 

lldb/source/Plugins/Process/Linux/Perf.h

 };
 
 using FileDescriptorUP =
     std::unique_ptr<long, resource_handle::FileDescriptorDeleter>;
 using MmapUP = std::unique_ptr<void, resource_handle::MmapDeleter>;
 
 } // namespace resource_handle
 
+/// \return
+///     The content of /proc/cpuinfo and cache it if errors didn't happen.
+llvm::Expected<llvm::ArrayRef<uint8_t>> GetProcFsCpuInfo();
+
+/// \return
+///     A list of available logical core ids given the contents of
+///     /proc/cpuinfo.
+llvm::Expected<std::vector<int>>
+GetAvailableLogicalCoreIDs(llvm::StringRef cpuinfo);
+
+/// \return
+///     A list with all the logical cores available in the system and cache it
+///     if errors didn't happen.
+llvm::Expected<llvm::ArrayRef<int>> GetAvailableLogicalCoreIDs();
+
 /// Thin wrapper of the perf_event_open API.
 ///
 /// Exposes the metadata page and data and aux buffers of a perf event.
 /// Handles the management of the event's file descriptor and mmap'ed
 /// regions.
 class PerfEvent {
 public:
   /// Create a new performance monitoring event via the perf_event_open syscall.

lldb/source/Plugins/Process/Linux/Perf.cpp

 //===-- Perf.cpp ----------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "Perf.h"
 
+#include "lldb/Host/linux/Support.h"
+
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
+#include "llvm/Support/MemoryBuffer.h"
 
 #include <sys/mman.h>
 #include <sys/syscall.h>
 #include <unistd.h>
 
 using namespace lldb_private;
 using namespace process_linux;
 using namespace llvm;
 
+Expected<ArrayRef<uint8_t>> lldb_private::process_linux::GetProcFsCpuInfo() {
+  static Optional<std::vector<uint8_t>> cpu_info;
+  if (!cpu_info) {
+    auto buffer_or_error = errorOrToExpected(getProcFile("cpuinfo"));
+    if (!buffer_or_error)
+      return buffer_or_error.takeError();
+    MemoryBuffer &buffer = **buffer_or_error;
+    cpu_info = std::vector<uint8_t>(
+        reinterpret_cast<const uint8_t *>(buffer.getBufferStart()),
+        reinterpret_cast<const uint8_t *>(buffer.getBufferEnd()));
+  }
+  return *cpu_info;
+}
+
+Expected<std::vector<int>>
+lldb_private::process_linux::GetAvailableLogicalCoreIDs(StringRef cpuinfo) {
+  SmallVector<StringRef, 8> lines;
+  cpuinfo.split(lines, "\n", /*MaxSplit=*/-1, /*KeepEmpty=*/false);
+  std::vector<int> logical_cores;
+
+  for (StringRef line : lines) {
+    std::pair<StringRef, StringRef> key_value = line.split(':');
+    auto key = key_value.first.trim();
+    auto val = key_value.second.trim();
+    if (key == "processor") {
+      int processor;
+      if (val.getAsInteger(10, processor))
+        return createStringError(
+            inconvertibleErrorCode(),
+            "Failed parsing the /proc/cpuinfo line entry: %s", line.data());
+      logical_cores.push_back(processor);
+    }
+  }
+  return logical_cores;
+}
+
+llvm::Expected<llvm::ArrayRef<int>>
+lldb_private::process_linux::GetAvailableLogicalCoreIDs() {
+  static Optional<std::vector<int>> logical_cores_ids;
+  if (!logical_cores_ids) {
+    // We find the actual list of core ids by parsing /proc/cpuinfo
+    Expected<ArrayRef<uint8_t>> cpuinfo = GetProcFsCpuInfo();
+    if (!cpuinfo)
+      return cpuinfo.takeError();
+
+    Expected<std::vector<int>> core_ids = GetAvailableLogicalCoreIDs(
+        StringRef(reinterpret_cast<const char *>(cpuinfo->data())));
+    if (!core_ids)
+      return core_ids.takeError();
+
+    logical_cores_ids.emplace(std::move(*core_ids));
+  }
+  return *logical_cores_ids;
+}
+
 Expected<LinuxPerfZeroTscConversion>
 lldb_private::process_linux::LoadPerfTscConversionParameters() {
   lldb::pid_t pid = getpid();
   perf_event_attr attr;
   memset(&attr, 0, sizeof(attr));
   attr.size = sizeof(attr);
   attr.type = PERF_TYPE_SOFTWARE;
   attr.config = PERF_COUNT_SW_DUMMY;

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

 
 TraceIntelPT::TraceIntelPT(
     const pt_cpu &cpu_info,
     const std::vector<ThreadPostMortemTraceSP> &traced_threads)
     : m_cpu_info(cpu_info) {
   for (const ThreadPostMortemTraceSP &thread : traced_threads) {
     m_thread_decoders.emplace(thread->GetID(),
                               std::make_unique<ThreadDecoder>(thread, *this));
-    SetPostMortemThreadDataFile(thread->GetID(), "threadTraceBuffer",
+    SetPostMortemThreadDataFile(thread->GetID(),
+                                IntelPTDataKinds::kThreadTraceBuffer,
                                 thread->GetTraceFile());
   }
 }
 
 DecodedThreadSP TraceIntelPT::Decode(Thread &thread) {
   RefreshLiveProcessState();
   if (m_live_refresh_error.hasValue())
     return std::make_shared<DecodedThread>(
llvm::Expected<size_t> TraceIntelPT::GetRawTraceSize(Thread &thread) {
   };
   if (Error err = OnThreadBufferRead(thread.GetID(), callback))
     return std::move(err);
 
   return size;
 }
 
 Expected<pt_cpu> TraceIntelPT::GetCPUInfoForLiveProcess() {
-  Expected<std::vector<uint8_t>> cpu_info = GetLiveProcessBinaryData("cpuInfo");
+  Expected<std::vector<uint8_t>> cpu_info =
+      GetLiveProcessBinaryData(IntelPTDataKinds::kProcFsCpuInfo);
   if (!cpu_info)
     return cpu_info.takeError();
 
   int64_t cpu_family = -1;
   int64_t model = -1;
   int64_t stepping = -1;
   std::string vendor_id;
 
if (configuration) {
     }
   }
 
   return Start(tids, thread_buffer_size, enable_tsc, psb_period);
 }
 
 Error TraceIntelPT::OnThreadBufferRead(lldb::tid_t tid,
                                        OnBinaryDataReadCallback callback) {
-  return OnThreadBinaryDataRead(tid, "threadTraceBuffer", callback);
+  return OnThreadBinaryDataRead(tid, IntelPTDataKinds::kThreadTraceBuffer,
+                                callback);
 }
 
 TaskTimer &TraceIntelPT::GetTimer() { return m_task_timer; }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionSaver.cpp

if (std::error_code ec =
     return llvm::errorCodeToError(ec);
 
   llvm::Expected<JSONTraceIntelPTTrace> json_intel_pt_trace =
       BuildTraceSection(trace_ipt);
   if (!json_intel_pt_trace)
     return json_intel_pt_trace.takeError();
 
   llvm::Expected<JSONTraceSessionBase> json_session_description =
-      TraceSessionSaver::BuildProcessesSection(*live_process,
-                                               "threadTraceBuffer", directory);
+      TraceSessionSaver::BuildProcessesSection(
+          *live_process, IntelPTDataKinds::kThreadTraceBuffer, directory);
 
   if (!json_session_description)
     return json_session_description.takeError();
 
   JSONTraceIntelPTSession json_intel_pt_session{json_intel_pt_trace.get(),
                                                 json_session_description.get()};
 
   return TraceSessionSaver::WriteSessionToFile(

lldb/source/Utility/TraceIntelPTGDBRemotePackets.cpp

 //===-- TraceIntelPTGDBRemotePackets.cpp ------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Utility/TraceIntelPTGDBRemotePackets.h"
 
 using namespace llvm;
 using namespace llvm::json;
 
 namespace lldb_private {
 
+const char *IntelPTDataKinds::kProcFsCpuInfo = "procFsCpuInfo";
+const char *IntelPTDataKinds::kThreadTraceBuffer = "threadTraceBuffer";
+
 bool fromJSON(const json::Value &value, TraceIntelPTStartRequest &packet,
               Path path) {
   ObjectMapper o(value, path);
   if (!o || !fromJSON(value, (TraceStartRequest &)packet, path) ||
       !o.map("enableTsc", packet.enableTsc) ||
       !o.map("psbPeriod", packet.psbPeriod) ||
       !o.map("threadBufferSize", packet.threadBufferSize) ||
       !o.map("processBufferSizeLimit", packet.processBufferSizeLimit))

lldb/unittests/Process/Linux/PerfTests.cpp

 //===-- PerfTests.cpp -----------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifdef __x86_64__
 
 #include "Perf.h"
 
+#include "lldb/Host/linux/Support.h"
+
 #include "llvm/Support/Error.h"
 
+#include "gmock/gmock.h"
 #include "gtest/gtest.h"
+
 #include <chrono>
 #include <cstdint>
 
 using namespace lldb_private;
 using namespace process_linux;
 using namespace llvm;
 
 /// Helper function to read current TSC value.
static Expected<uint64_t> readTsc() {
   unsigned cpu;
   unsigned long rax, rdx;
 
   __asm__ __volatile__("rdtscp\n" : "=a"(rax), "=d"(rdx), "=c"(cpu)::);
 
   return (rdx << 32) + rax;
 }
 
+TEST(Perf, HardcodedLogicalCoreIDs) {
+  Expected<std::vector<int>> core_ids =
+      GetAvailableLogicalCoreIDs(R"(processor       : 13
+vendor_id       : GenuineIntel
+cpu family      : 6
+model           : 85
+model name      : Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
+stepping        : 4
+microcode       : 0x2000065
+cpu MHz         : 2886.370
+cache size      : 28160 KB
+physical id     : 1
+siblings        : 40
+core id         : 19
+cpu cores       : 20
+apicid          : 103
+initial apicid  : 103
+fpu             : yes
+fpu_exception   : yes
+cpuid level     : 22
+power management:
+
+processor       : 24
+vendor_id       : GenuineIntel
+cpu family      : 6
+model           : 85
+model name      : Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
+stepping        : 4
+microcode       : 0x2000065
+cpu MHz         : 2768.494
+cache size      : 28160 KB
+physical id     : 1
+siblings        : 40
+core id         : 20
+cpu cores       : 20
+apicid          : 105
+power management:
+
+processor       : 35
+vendor_id       : GenuineIntel
+cpu family      : 6
+model           : 85
+model name      : Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
+stepping        : 4
+microcode       : 0x2000065
+cpu MHz         : 2884.703
+cache size      : 28160 KB
+physical id     : 1
+siblings        : 40
+core id         : 24
+cpu cores       : 20
+apicid          : 113
+
+processor       : 79
+vendor_id       : GenuineIntel
+cpu family      : 6
+model           : 85
+model name      : Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz
+stepping        : 4
+microcode       : 0x2000065
+cpu MHz         : 3073.955
+cache size      : 28160 KB
+physical id     : 1
+siblings        : 40
+core id         : 28
+cpu cores       : 20
+apicid          : 121
+power management:
+)");
+
+  ASSERT_TRUE((bool)core_ids);
+  ASSERT_THAT(*core_ids, ::testing::ElementsAre(13, 24, 35, 79));
+}
+
+TEST(Perf, RealLogicalCoreIDs) {
+  // We first check we can read /proc/cpuinfo
+  auto buffer_or_error = errorOrToExpected(getProcFile("cpuinfo"));
+  if (!buffer_or_error)
+    GTEST_SKIP() << toString(buffer_or_error.takeError());
+
+  // At this point we shouldn't fail parsing the core ids
+  Expected<ArrayRef<int>> core_ids = GetAvailableLogicalCoreIDs();
+  ASSERT_TRUE((bool)core_ids);
+  ASSERT_GT((int)core_ids->size(), 0) << "We must see at least one core";
+}
+
 // Test TSC to walltime conversion based on perf conversion values.
 TEST(Perf, TscConversion) {
   // This test works by first reading the TSC value directly before
   // and after sleeping, then converting these values to nanoseconds, and
   // finally ensuring the difference is approximately equal to the sleep time.
   //
   // There will be slight overhead associated with the sleep call, so it isn't
   // reasonable to expect the difference to be exactly equal to the sleep time.