[trace][intelpt] Support system-wide tracing [13] - Add context switch decoding

[wallace]

Summary

• Add the logic that parses all cpu context switch traces and produces blocks of continuous executions, which will be later used to assign intel pt subtraces to threads and to identify gaps. This logic can also identify if the context switch trace is malformed.
• The continuous executions blocks are able to indicate when there were some contention issues when producing the context switch trace. See the inline comments for more information.
• Update the 'dump info' command to show information and stats related to the multicore decoding flow, including timing about context switch decoding.
• Add the logic to conver nanoseconds to TSCs.
• Fix a bug when returning the context switches. Now they data returned makes sense and even empty traces can be returned from lldb-server.
• Finish the necessary bits for loading and saving a multi-core trace bundle from disk.
• Change some size_t to uint64_t for compatibility with 32 bit systems.

Tested by saving a trace session of a program that sleeps 100 times, it was able to produce the following 'dump info' text:

(lldb) trace load /tmp/trace3/trace.json                                                                   
(lldb) thread trace dump info                                                                              

Trace technology: intel-pt

thread #1: tid = 4192415
  Total number of instructions: 1

  Memory usage:
    Total approximate memory usage (excluding raw trace): 2.51 KiB
    Average memory usage per instruction (excluding raw trace): 2573.00 bytes

  Timing for this thread:

  Timing for global tasks:
    Context switch trace decoding: 0.00s

  Events:
    Number of instructions with events: 0
    Number of individual events: 0

  Multi-core decoding:
    Total number of continuous executions found: 2499
    Number of continuous executions for this thread: 102

  Errors:
    Number of TSC decoding errors: 0

As you can see, it had 101 context switches, which makes sense given the 100 sleeps, and the total number of continuous executions for all threads is 2499. All the context switches were correctly decoded without a severe failure.

[jj10306]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCoreDecoder.cpp

23–43

should these structures live in Perf.h?

60–103

nit: if you switch the union for two optional values you can remove a lot of the redundancy between these methods.

137–142

can you help me understand how these two cases could happen? they seem fundamentally impossible given the nature of context switching - shouldn't all "continuous" execution be disjoint?

My current understanding is as follows:

Expected:
i1 o1 i4 o4 i9 o9

Impossible:
and this is not possible:
i1 i4 o4 o1 9 o9

Let me know if I'm missing something 🙂

147

why the + 1?

182

Do you think any of the general perf logic related to "decoding" the records should be moved to Perf.h/cpp?

197

nit: it feels weird casting to PerfContextSwitchRecord when we don't yet know if this is actually a context switch event without first looking at the header.
casting to perf_event_header and checking that first before interpreting the record as a context switch record seems like a better approach.
Given that currently only intelpt is using the LLDB's perf "library" this isn't a big deal, but if we wanted to make it more complete/robust, we should revisit this and improve our record handling design so it could be easily extended to support any record types.

199

can you link the documentation that states this?

200–201

using sizeof on uint64_t feels weird since the typename already implies the name. I think moving this value to a constant and explaining its value would make things cleaner.

205–206

same as above, can we link the docs. alternatively, link the docs at the top of the function or in the header and then reference that link at the appropriate spots in the code

264–273

this is lambda inception 😆

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCoreDecoder.h

32–34

Where do these "guesses" come from?

41–62

what about just having two optionals fields to remove the "dangers" that unions introduce? Then the Variant enum can be used to guide whether the start and end values should be non-null just as it's being used to access the union currently

106–108

what is meant by "contention" here? Is this referring to the ipt aux buffer wrapping?

112

is this just the cores that the program ran on or all possible cores the process could have run on?

156

why is this needed?

[wallace]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCoreDecoder.cpp

23–43

I created a new file PerfContextSwitchDecoder. and moved this code there. We can't actually use Perf.h because that's private to lldb-server

60–103

in this case I'm willing to pay the cost of being super explicit in initialization to be able to be super safe when using the context switches. I don't want that for any reason we use context switch that we are not completely sure that is correct, so I'm adding very explicit variable names to prevent myself and others from making mistakes

137–142

this can happen if, for example, there are contention issues. You can have something like this

tid 12 in
tid 12 out
<now you start reading the context switch trace in the collector and update the tail pointer>
    during this time the following context switches happened but you lost them
    tid 13 in
    tid 13 out
    tid 14 in
<here the kernel resumes writing context switches because the tail pointer was updated>
tid 14 out

and now you have <tid 12 out> followed by <tid 14 out>. That means that you can rely on the first execution (tid 12), but you can't trust the second one (tid 14) because you don't really know when it started.

147

let's follow the example above, you have <tid 12 out: tsc A> followed by <tid 14 out: tsc B>

you don't really know when the execution of tid 14 started, but you know it was after tsc A, i.e. tsc A + 1. So that's a hinted start

182

see my first response above

197

makes sense. this is a code smell

199

sure

200–201

+1

205–206

+1

264–273

lol, i'll see how i can simplify it

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCoreDecoder.h

32–34

see above

41–62

see above

106–108

it means when the trace reader thread couldn't keep up with the data and some context switch records were lost

112

all cores on the hardware. We don't know which ones our program ran on

156

i've improved the documentation, but basically this variable holds any fatal decoding error we see to prevent multiple failed decoding attempts. If this is not-null, then this means that we tried to decode but it failed badly, and we don't want to try to redecode again because that's expensive.

[wallace]

This was improved in https://reviews.llvm.org/D127804

lldb/include/lldb/Target/Trace.h

protected:
   ///
   /// \return
   ///     A vector of bytes with the requested data, or an \a llvm::Error in
   ///     case of failures.
   llvm::Expected<std::vector<uint8_t>>
   GetLiveProcessBinaryData(llvm::StringRef kind);
 
   /// Get the size of the data returned by \a GetLiveThreadBinaryData
-  llvm::Optional<size_t> GetLiveThreadBinaryDataSize(lldb::tid_t tid,
+  llvm::Optional<uint64_t> GetLiveThreadBinaryDataSize(lldb::tid_t tid,
                                                        llvm::StringRef kind);
 
   /// Get the size of the data returned by \a GetLiveCoreBinaryData
-  llvm::Optional<size_t> GetLiveCoreBinaryDataSize(lldb::core_id_t core_id,
+  llvm::Optional<uint64_t> GetLiveCoreBinaryDataSize(lldb::core_id_t core_id,
                                                      llvm::StringRef kind);
 
   /// Get the size of the data returned by \a GetLiveProcessBinaryData
-  llvm::Optional<size_t> GetLiveProcessBinaryDataSize(llvm::StringRef kind);
+  llvm::Optional<uint64_t> GetLiveProcessBinaryDataSize(llvm::StringRef kind);
 
   /// Constructor for post mortem processes
   Trace(llvm::ArrayRef<lldb::ProcessSP> postmortem_processes,
         llvm::Optional<std::vector<lldb::core_id_t>> postmortem_cores);
 
   /// Constructor for a live process
   Trace(Process &live_process) : m_live_process(&live_process) {}
 
private:
   std::vector<Process *> m_postmortem_processes;
 
   /// These data kinds are returned by lldb-server when fetching the state of
   /// the tracing session. The size in bytes can be used later for fetching the
   /// data in batches.
   /// \{
 
   /// tid -> data kind -> size
-  llvm::DenseMap<lldb::tid_t, std::unordered_map<std::string, size_t>>
+  llvm::DenseMap<lldb::tid_t, std::unordered_map<std::string, uint64_t>>
       m_live_thread_data;
 
   /// core id -> data kind -> size
-  llvm::DenseMap<lldb::core_id_t, std::unordered_map<std::string, size_t>>
+  llvm::DenseMap<lldb::core_id_t, std::unordered_map<std::string, uint64_t>>
       m_live_core_data;
   /// data kind -> size
-  std::unordered_map<std::string, size_t> m_live_process_data;
+  std::unordered_map<std::string, uint64_t> m_live_process_data;
   /// \}
 
   /// The list of cores being traced. Might be \b None depending on the plug-in.
   llvm::Optional<std::vector<lldb::core_id_t>> m_cores;
 
   /// Postmortem traces can specific additional data files, which are
   /// represented in this variable using a data kind identifier for each file.
   /// \{

lldb/include/lldb/Utility/TraceGDBRemotePackets.h

 struct TraceGetBinaryDataRequest {
   /// Tracing technology name, e.g. intel-pt, arm-coresight.
   std::string type;
   /// Identifier for the data.
   std::string kind;
   /// Optional tid if the data is related to a thread.
   llvm::Optional<lldb::tid_t> tid;
   /// Optional core id if the data is related to a cpu core.
-  llvm::Optional<lldb::tid_t> core_id;
+  llvm::Optional<lldb::core_id_t> core_id;
   /// Offset in bytes from where to start reading the data.
   uint64_t offset;
   /// Number of bytes to read.
   uint64_t size;
 };
 
 bool fromJSON(const llvm::json::Value &value,
               lldb_private::TraceGetBinaryDataRequest &packet,
               llvm::json::Path path);
 
 llvm::json::Value toJSON(const lldb_private::TraceGetBinaryDataRequest &packet);
 /// \}
 
 } // namespace lldb_private
 
 #endif // LLDB_UTILITY_TRACEGDBREMOTEPACKETS_H

lldb/include/lldb/Utility/TraceIntelPTGDBRemotePackets.h

struct LinuxPerfZeroTscConversion {
   //  See 'time_zero' section of
   //  https://man7.org/linux/man-pages/man2/perf_event_open.2.html
   ///
   /// \param[in] tsc
   ///   The TSC value to be converted.
   ///
   /// \return
   ///   Nanosecond wall time.
-  std::chrono::nanoseconds ToNanos(uint64_t tsc);
+  std::chrono::nanoseconds ToNanos(uint64_t tsc) const;
+
+  uint64_t ToTSC(std::chrono::nanoseconds nanos) const;
 
   uint32_t time_mult;
   uint16_t time_shift;
   uint64_t time_zero;
 };
 
 struct TraceIntelPTGetStateResponse : TraceGetStateResponse {
   /// The TSC to wall time conversion if it exists, otherwise \b nullptr.

lldb/source/Plugins/Process/Linux/Perf.cpp

PerfEvent::ReadFlushedOutDataCyclicBuffer(size_t offset, size_t size) {
   }
 
   if (output.size() != size)
     return createStringError(inconvertibleErrorCode(),
                              formatv("Requested {0} bytes of perf_event data "
                                      "buffer but only {1} are available",
                                      size, output.size()));
 
-  return data;
+  return output;
 }
 
 Expected<std::vector<uint8_t>>
 PerfEvent::ReadFlushedOutAuxCyclicBuffer(size_t offset, size_t size) {
   CollectionState previous_state = m_collection_state;
   if (Error err = DisableWithIoctl())
     return std::move(err);
 
PerfEvent::ReadFlushedOutAuxCyclicBuffer(size_t offset, size_t size) {
   }
 
   if (output.size() != size)
     return createStringError(inconvertibleErrorCode(),
                              formatv("Requested {0} bytes of perf_event aux "
                                      "buffer but only {1} are available",
                                      size, output.size()));
 
-  return data;
+  return output;
 }
 
 Error PerfEvent::DisableWithIoctl() {
   if (m_collection_state == CollectionState::Disabled)
     return Error::success();
 
   if (ioctl(*m_fd, PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP) < 0)
     return createStringError(inconvertibleErrorCode(),
return createStringError(inconvertibleErrorCode(),
                              std::strerror(errno));
 
   m_collection_state = CollectionState::Enabled;
   return Error::success();
 }
 
 size_t PerfEvent::GetEffectiveDataBufferSize() const {
   perf_event_mmap_page &mmap_metadata = GetMetadataPage();
-  if (mmap_metadata.data_head <= mmap_metadata.data_size)
+  if (mmap_metadata.data_head < mmap_metadata.data_size)
     return mmap_metadata.data_head;
   else
     return mmap_metadata.data_size; // The buffer has wrapped.
 }

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

if (pid != LLDB_INVALID_PROCESS_ID)
     packet.Printf("p%" PRIx64 ".", pid);
 
   if (tid == UINT64_MAX)
     packet.PutCString("-1");
   else
     packet.Printf("%" PRIx64, tid);
 
   StringExtractorGDBRemote response;
-  if (SendPacketAndWaitForResponse(packet.GetString(), response) 
-      == PacketResult::Success) {
+  if (SendPacketAndWaitForResponse(packet.GetString(), response) ==
+      PacketResult::Success) {
     if (response.IsOKResponse())
       return {{pid, tid}};
 
     /*
      * Connected bare-iron target (like YAMON gdb-stub) may not have support for
      * Hg packet.
      * The reply from '?' packet could be as simple as 'S05'. There is no packet
      * which can
GDBRemoteCommunicationClient::SendTraceGetBinaryData(
   escaped_packet.PutEscapedBytes(json_string.c_str(), json_string.size());
 
   StringExtractorGDBRemote response;
   if (SendPacketAndWaitForResponse(escaped_packet.GetString(), response,
                                    timeout) ==
       GDBRemoteCommunication::PacketResult::Success) {
     if (response.IsErrorResponse())
       return response.GetStatus().ToError();
-    if (response.IsUnsupportedResponse())
-      return llvm::createStringError(llvm::inconvertibleErrorCode(),
-                                     "jLLDBTraceGetBinaryData is unsupported");
     std::string data;
     response.GetEscapedBinaryData(data);
     return std::vector<uint8_t>(data.begin(), data.end());
   }
   LLDB_LOG(log, "failed to send packet: jLLDBTraceGetBinaryData");
   return llvm::createStringError(
       llvm::inconvertibleErrorCode(),
       "failed to send packet: jLLDBTraceGetBinaryData '%s'",

lldb/source/Plugins/Trace/intel-pt/CMakeLists.txt

add_lldb_library(lldbPluginTraceIntelPT PLUGIN
   CommandObjectTraceStartIntelPT.cpp
   DecodedThread.cpp
   TaskTimer.cpp
   LibiptDecoder.cpp
   ThreadDecoder.cpp
   TraceCursorIntelPT.cpp
   TraceIntelPT.cpp
   TraceIntelPTJSONStructs.cpp
+  TraceIntelPTMultiCoreDecoder.cpp
   TraceIntelPTSessionFileParser.cpp
   TraceIntelPTSessionSaver.cpp
 
   LINK_LIBS
     lldbCore
     lldbSymbol
     lldbTarget
     lldbPluginTraceCommon
     ${LIBIPT_LIBRARY}
   LINK_COMPONENTS
     Support
   )
 
 
 add_dependencies(lldbPluginTraceIntelPT TraceIntelPTOptionsGen)

lldb/source/Plugins/Trace/intel-pt/CommandObjectTraceStartIntelPT.h

public:
 
     Status SetOptionValue(uint32_t option_idx, llvm::StringRef option_arg,
                           ExecutionContext *execution_context) override;
 
     void OptionParsingStarting(ExecutionContext *execution_context) override;
 
     llvm::ArrayRef<OptionDefinition> GetDefinitions() override;
 
-    size_t m_trace_buffer_size;
+    uint64_t m_trace_buffer_size;
     bool m_enable_tsc;
-    llvm::Optional<size_t> m_psb_period;
+    llvm::Optional<uint64_t> m_psb_period;
   };
 
   CommandObjectThreadTraceStartIntelPT(TraceIntelPT &trace,
                                        CommandInterpreter &interpreter)
       : CommandObjectMultipleThreads(
             interpreter, "thread trace start",
             "Start tracing one or more threads with intel-pt. "
             "Defaults to the current thread. Thread indices can be "
public:
 
     Status SetOptionValue(uint32_t option_idx, llvm::StringRef option_arg,
                           ExecutionContext *execution_context) override;
 
     void OptionParsingStarting(ExecutionContext *execution_context) override;
 
     llvm::ArrayRef<OptionDefinition> GetDefinitions() override;
 
-    size_t m_trace_buffer_size;
-    size_t m_process_buffer_size_limit;
+    uint64_t m_trace_buffer_size;
+    uint64_t m_process_buffer_size_limit;
     bool m_enable_tsc;
-    llvm::Optional<size_t> m_psb_period;
+    llvm::Optional<uint64_t> m_psb_period;
     bool m_per_core_tracing;
   };
 
   CommandObjectProcessTraceStartIntelPT(TraceIntelPT &trace,
                                         CommandInterpreter &interpreter)
       : CommandObjectParsed(
             interpreter, "process trace start",
             "Start tracing this process with intel-pt, including future "

lldb/source/Plugins/Trace/intel-pt/TaskTimer.h

 #include <chrono>
 #include <functional>
 #include <unordered_map>
 
 namespace lldb_private {
 namespace trace_intel_pt {
 
 /// Class used to track the duration of long running tasks related to a single
-/// thread for reporting.
-class ThreadTaskTimer {
+/// scope for reporting.
+class ScopedTaskTimer {
 public:
   /// Execute the given \p task and record its duration.
   ///
   /// \param[in] name
   ///     The name used to identify this task for reporting.
   ///
   /// \param[in] task
   ///     The task function.
private:
   std::unordered_map<std::string, std::chrono::milliseconds> m_timed_tasks;
 };
 
 /// Class used to track the duration of long running tasks for reporting.
 class TaskTimer {
 public:
   /// \return
   ///     The timer object for the given thread.
-  ThreadTaskTimer &ForThread(lldb::tid_t tid);
+  ScopedTaskTimer &ForThread(lldb::tid_t tid);
+
+  /// \return
+  ///     The timer object for global tasks.
+  ScopedTaskTimer &ForGlobal();
 
 private:
-  llvm::DenseMap<lldb::tid_t, ThreadTaskTimer> m_thread_timers;
+  llvm::DenseMap<lldb::tid_t, ScopedTaskTimer> m_thread_timers;
+  ScopedTaskTimer m_global_timer;
 };
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TASKTIMER_H

lldb/source/Plugins/Trace/intel-pt/TaskTimer.cpp

 #include "TaskTimer.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::trace_intel_pt;
 using namespace llvm;
 
-void ThreadTaskTimer::ForEachTimedTask(
+void ScopedTaskTimer::ForEachTimedTask(
     std::function<void(const std::string &event,
                        std::chrono::milliseconds duration)>
         callback) {
   for (const auto &kv : m_timed_tasks) {
     callback(kv.first, kv.second);
   }
 }
 
-ThreadTaskTimer &TaskTimer::ForThread(lldb::tid_t tid) {
+ScopedTaskTimer &TaskTimer::ForThread(lldb::tid_t tid) {
   auto it = m_thread_timers.find(tid);
   if (it == m_thread_timers.end())
-    it = m_thread_timers.try_emplace(tid, ThreadTaskTimer{}).first;
+    it = m_thread_timers.try_emplace(tid, ScopedTaskTimer{}).first;
   return it->second;
 }
+
+ScopedTaskTimer &TaskTimer::ForGlobal() { return m_global_timer; }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

 //===-- TraceIntelPT.h ------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPT_H
 #define LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPT_H
 
 #include "TaskTimer.h"
 #include "ThreadDecoder.h"
+#include "TraceIntelPTMultiCoreDecoder.h"
 #include "TraceIntelPTSessionFileParser.h"
+
 #include "lldb/Utility/FileSpec.h"
 #include "lldb/lldb-types.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace lldb_private {
 namespace trace_intel_pt {
 
 class TraceIntelPT : public Trace {
public:
   GetThreadTraceStartCommand(CommandInterpreter &interpreter) override;
 
   llvm::StringRef GetSchema() override;
 
   lldb::TraceCursorUP GetCursor(Thread &thread) override;
 
   void DumpTraceInfo(Thread &thread, Stream &s, bool verbose) override;
 
-  llvm::Expected<size_t> GetRawTraceSize(Thread &thread);
+  llvm::Expected<llvm::Optional<uint64_t>> GetRawTraceSize(Thread &thread);
 
   llvm::Error DoRefreshLiveProcessState(TraceGetStateResponse state,
                                         llvm::StringRef json_response) override;
 
   bool IsTraced(lldb::tid_t tid) override;
 
   const char *GetStartConfigurationHelp() override;
 
public:
   ///
   /// \param[in] per_core_tracing
   ///     This value defines whether to have a trace buffer per thread or per
   ///     cpu core.
   ///
   /// \return
   ///     \a llvm::Error::success if the operation was successful, or
   ///     \a llvm::Error otherwise.
-  llvm::Error Start(size_t trace_buffer_size, size_t total_buffer_size_limit,
-                    bool enable_tsc, llvm::Optional<size_t> psb_period,
+  llvm::Error Start(uint64_t trace_buffer_size,
+                    uint64_t total_buffer_size_limit, bool enable_tsc,
+                    llvm::Optional<uint64_t> psb_period,
                     bool m_per_core_tracing);
 
   /// \copydoc Trace::Start
   llvm::Error Start(StructuredData::ObjectSP configuration =
                         StructuredData::ObjectSP()) override;
 
   /// Start tracing live threads.
   ///
public:
   ///     Whether to use enable TSC timestamps or not.
   ///
   /// \param[in] psb_period
   ///     This value defines the period in which PSB packets will be generated.
   ///
   /// \return
   ///     \a llvm::Error::success if the operation was successful, or
   ///     \a llvm::Error otherwise.
-  llvm::Error Start(llvm::ArrayRef<lldb::tid_t> tids, size_t trace_buffer_size,
-                    bool enable_tsc, llvm::Optional<size_t> psb_period);
+  llvm::Error Start(llvm::ArrayRef<lldb::tid_t> tids,
+                    uint64_t trace_buffer_size, bool enable_tsc,
+                    llvm::Optional<uint64_t> psb_period);
 
   /// \copydoc Trace::Start
   llvm::Error Start(llvm::ArrayRef<lldb::tid_t> tids,
                     StructuredData::ObjectSP configuration =
                         StructuredData::ObjectSP()) override;
 
   /// See \a Trace::OnThreadBinaryDataRead().
   llvm::Error OnThreadBufferRead(lldb::tid_t tid,
private:
   /// \return
   ///     A \a DecodedThread shared pointer with the decoded instructions. Any
   ///     errors are embedded in the instruction list.
   DecodedThreadSP Decode(Thread &thread);
 
   /// It is provided by either a session file or a live process' "cpuInfo"
   /// binary data.
   llvm::Optional<pt_cpu> m_cpu_info;
+  llvm::Optional<TraceIntelPTMultiCoreDecoder> m_multicore_decoder;
+  /// These decoders are used for the non-per-core case
   std::map<lldb::tid_t, std::unique_ptr<ThreadDecoder>> m_thread_decoders;
   /// Helper variable used to track long running operations for telemetry.
   TaskTimer m_task_timer;
   /// It is provided by either a session file or a live process to convert TSC
   /// counters to and from nanos. It might not be available on all hosts.
   llvm::Optional<LinuxPerfZeroTscConversion> m_tsc_conversion;
 };
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPT_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

for (const ThreadPostMortemTraceSP &thread : traced_threads) {
     m_thread_decoders.emplace(thread->GetID(),
                               std::make_unique<ThreadDecoder>(thread, *this));
     if (const Optional<FileSpec> &trace_file = thread->GetTraceFile()) {
       SetPostMortemThreadDataFile(thread->GetID(),
                                   IntelPTDataKinds::kTraceBuffer, *trace_file);
     }
   }
   if (session.cores) {
+    std::vector<core_id_t> cores;
+
     for (const JSONCore &core : *session.cores) {
       SetPostMortemCoreDataFile(core.core_id, IntelPTDataKinds::kTraceBuffer,
                                 FileSpec(core.trace_buffer));
       SetPostMortemCoreDataFile(core.core_id,
                                 IntelPTDataKinds::kPerfContextSwitchTrace,
                                 FileSpec(core.context_switch_trace));
+      cores.push_back(core.core_id);
     }
+
+    std::vector<tid_t> tids;
+    for (const JSONProcess &process : session.processes)
+      for (const JSONThread &thread : process.threads)
+        tids.push_back(thread.tid);
+
+    m_multicore_decoder.emplace(*this, cores, tids,
+                                *session.tsc_perf_zero_conversion);
   }
 }
 
 DecodedThreadSP TraceIntelPT::Decode(Thread &thread) {
   if (const char *error = RefreshLiveProcessState())
     return std::make_shared<DecodedThread>(
         thread.shared_from_this(),
         createStringError(inconvertibleErrorCode(), error));
 
+  if (m_multicore_decoder)
+    return m_multicore_decoder->Decode(thread);
+
   auto it = m_thread_decoders.find(thread.GetID());
   if (it == m_thread_decoders.end())
     return std::make_shared<DecodedThread>(
         thread.shared_from_this(),
         createStringError(inconvertibleErrorCode(), "thread not traced"));
   return it->second->Decode();
 }
 
 lldb::TraceCursorUP TraceIntelPT::GetCursor(Thread &thread) {
   return Decode(thread)->GetCursor();
 }
 
 void TraceIntelPT::DumpTraceInfo(Thread &thread, Stream &s, bool verbose) {
   lldb::tid_t tid = thread.GetID();
   s.Format("\nthread #{0}: tid = {1}", thread.GetIndexID(), thread.GetID());
   if (!IsTraced(tid)) {
     s << ", not traced\n";
     return;
   }
   s << "\n";
 
-  Expected<size_t> raw_size = GetRawTraceSize(thread);
-  if (!raw_size) {
-    s.Format("  {0}\n", toString(raw_size.takeError()));
+  Expected<Optional<uint64_t>> raw_size_or_error = GetRawTraceSize(thread);
+  if (!raw_size_or_error) {
+    s.Format("  {0}\n", toString(raw_size_or_error.takeError()));
     return;
   }
+  Optional<uint64_t> raw_size = *raw_size_or_error;
 
   DecodedThreadSP decoded_trace_sp = Decode(thread);
-  size_t insn_len = decoded_trace_sp->GetInstructionsCount();
-  size_t mem_used = decoded_trace_sp->CalculateApproximateMemoryUsage();
+
+  /// Instruction stats
+  {
+    uint64_t insn_len = decoded_trace_sp->GetInstructionsCount();
+    uint64_t mem_used = decoded_trace_sp->CalculateApproximateMemoryUsage();
 
     s.Format("  Total number of instructions: {0}\n", insn_len);
 
     s << "\n  Memory usage:\n";
+    if (raw_size)
       s.Format("    Raw trace size: {0} KiB\n", *raw_size / 1024);
+
     s.Format(
         "    Total approximate memory usage (excluding raw trace): {0:2} KiB\n",
         (double)mem_used / 1024);
     if (insn_len != 0)
-    s.Format("    Average memory usage per instruction (excluding raw trace): "
-             "{0:2} bytes\n",
-             (double)mem_used / insn_len);
+      s.Format(
+          "    Average memory usage per instruction (excluding raw trace): "
+  }
 
-  s << "\n  Timing:\n";
-  GetTimer().ForThread(tid).ForEachTimedTask(
-      [&](const std::string &name, std::chrono::milliseconds duration) {
-        s.Format("    {0}: {1:2}s\n", name, duration.count() / 1000.0);
-      });
+  // Timing
+  {
+    s << "\n  Timing for this thread:\n";
+    auto print_duration = [&](const std::string &name,
+                              std::chrono::milliseconds duration) {
+    };
+    GetTimer().ForThread(tid).ForEachTimedTask(print_duration);
+
+    s << "\n  Timing for global tasks:\n";
+    GetTimer().ForGlobal().ForEachTimedTask(print_duration);
+  }
 
+  // Instruction events stats
+  {
     const DecodedThread::EventsStats &events_stats =
         decoded_trace_sp->GetEventsStats();
     s << "\n  Events:\n";
     s.Format("    Number of instructions with events: {0}\n",
              events_stats.total_instructions_with_events);
-  s.Format("    Number of individual events: {0}\n", events_stats.total_count);
+    s.Format("    Number of individual events: {0}\n",
+             events_stats.total_count);
     for (const auto &event_to_count : events_stats.events_counts) {
       s.Format("      {0}: {1}\n",
                trace_event_utils::EventToDisplayString(event_to_count.first),
                event_to_count.second);
     }
+  }
 
+  // Multicode decoding stats
+  if (m_multicore_decoder) {
+    s << "\n  Multi-core decoding:\n";
+    s.Format("    Total number of continuous executions found: {0}\n",
+             m_multicore_decoder->GetTotalContinuousExecutionsCount());
+    s.Format("    Number of continuous executions for this thread: {0}\n",
+             m_multicore_decoder->GetNumContinuousExecutionsForThread(tid));
+  }
+
+  // Errors
+  {
     s << "\n  Errors:\n";
     const DecodedThread::LibiptErrorsStats &tsc_errors_stats =
         decoded_trace_sp->GetTscErrorsStats();
     s.Format("    Number of TSC decoding errors: {0}\n",
              tsc_errors_stats.total_count);
     for (const auto &error_message_to_count :
          tsc_errors_stats.libipt_errors_counts) {
       s.Format("      {0}: {1}\n", error_message_to_count.first,
                error_message_to_count.second);
     }
   }
+}
 
-llvm::Expected<size_t> TraceIntelPT::GetRawTraceSize(Thread &thread) {
-  size_t size;
+llvm::Expected<Optional<uint64_t>>
+TraceIntelPT::GetRawTraceSize(Thread &thread) {
+  if (m_multicore_decoder)
+    return None; // TODO: calculate the amount of intel pt raw trace associated
+                 // with the given thread.
+  if (GetLiveProcess())
+    return GetLiveThreadBinaryDataSize(thread.GetID(),
+                                       IntelPTDataKinds::kTraceBuffer);
+  uint64_t size;
   auto callback = [&](llvm::ArrayRef<uint8_t> data) {
     size = data.size();
     return Error::success();
   };
   if (Error err = OnThreadBufferRead(thread.GetID(), callback))
     return std::move(err);
 
   return size;

 TraceIntelPT::GetPerfZeroTscConversion() {
   RefreshLiveProcessState();
   return m_tsc_conversion;
 }
 
 Error TraceIntelPT::DoRefreshLiveProcessState(TraceGetStateResponse state,
                                               StringRef json_response) {
   m_thread_decoders.clear();
+  m_tsc_conversion.reset();
+  m_multicore_decoder.reset();
 
+  Expected<TraceIntelPTGetStateResponse> intelpt_state =
+      json::parse<TraceIntelPTGetStateResponse>(json_response,
+                                                "TraceIntelPTGetStateResponse");
+  if (!intelpt_state)
+    return intelpt_state.takeError();
+
+  if (!intelpt_state->cores) {
     for (const TraceThreadState &thread_state : state.traced_threads) {
       ThreadSP thread_sp =
           GetLiveProcess()->GetThreadList().FindThreadByID(thread_state.tid);
       m_thread_decoders.emplace(
           thread_state.tid, std::make_unique<ThreadDecoder>(thread_sp, *this));
     }
+  } else {
+    std::vector<core_id_t> cores;
+    for (const TraceCoreState &core : *intelpt_state->cores)
+      cores.push_back(core.core_id);
+
+    std::vector<tid_t> tids;
+    for (const TraceThreadState &thread : intelpt_state->traced_threads)
+      tids.push_back(thread.tid);
 
-  Expected<TraceIntelPTGetStateResponse> intelpt_state =
-      json::parse<TraceIntelPTGetStateResponse>(json_response,
-                                                "TraceIntelPTGetStateResponse");
-  if (!intelpt_state)
-    return intelpt_state.takeError();
+    if (!intelpt_state->tsc_perf_zero_conversion)
+      return createStringError(inconvertibleErrorCode(),
+                               "Missing perf time_zero conversion values");
+    m_multicore_decoder.emplace(*this, cores, tids,
+                                *intelpt_state->tsc_perf_zero_conversion);
+  }
 
   m_tsc_conversion = intelpt_state->tsc_perf_zero_conversion;
   if (m_tsc_conversion) {
     Log *log = GetLog(LLDBLog::Target);
     LLDB_LOG(log, "TraceIntelPT found TSC conversion information");
   }
   return Error::success();
 }
 
 bool TraceIntelPT::IsTraced(lldb::tid_t tid) {
   RefreshLiveProcessState();
+  if (m_multicore_decoder)
+    return m_multicore_decoder->TracesThread(tid);
   return m_thread_decoders.count(tid);
 }
 
 // The information here should match the description of the intel-pt section
 // of the jLLDBTraceStart packet in the lldb/docs/lldb-gdb-remote.txt
 // documentation file. Similarly, it should match the CLI help messages of the
 // TraceIntelPTOptions.td file.
 const char *TraceIntelPT::GetStartConfigurationHelp() {
- int processBufferSizeLimit (defaults to {4} MiB):
     [process tracing only])",
                             kDefaultTraceBufferSize, kDefaultEnableTscValue,
                             kDefaultPsbPeriod, kDefaultPerCoreTracing,
                             kDefaultProcessBufferSizeLimit / 1024 / 1024));
   }
   return message->c_str();
 }
 
-Error TraceIntelPT::Start(size_t trace_buffer_size,
-                          size_t total_buffer_size_limit, bool enable_tsc,
-                          Optional<size_t> psb_period, bool per_core_tracing) {
+Error TraceIntelPT::Start(uint64_t trace_buffer_size,
+                          uint64_t total_buffer_size_limit, bool enable_tsc,
+                          Optional<uint64_t> psb_period,
+                          bool per_core_tracing) {
   TraceIntelPTStartRequest request;
   request.trace_buffer_size = trace_buffer_size;
   request.process_buffer_size_limit = total_buffer_size_limit;
   request.enable_tsc = enable_tsc;
-  request.psb_period =
-      psb_period.map([](size_t val) { return static_cast<uint64_t>(val); });
+  request.psb_period = psb_period;
   request.type = GetPluginName().str();
   request.per_core_tracing = per_core_tracing;
   return Trace::Start(toJSON(request));
 }
 
 Error TraceIntelPT::Start(StructuredData::ObjectSP configuration) {
-  size_t trace_buffer_size = kDefaultTraceBufferSize;
-  size_t process_buffer_size_limit = kDefaultProcessBufferSizeLimit;
+  uint64_t trace_buffer_size = kDefaultTraceBufferSize;
+  uint64_t process_buffer_size_limit = kDefaultProcessBufferSizeLimit;
   bool enable_tsc = kDefaultEnableTscValue;
-  Optional<size_t> psb_period = kDefaultPsbPeriod;
+  Optional<uint64_t> psb_period = kDefaultPsbPeriod;
   bool per_core_tracing = kDefaultPerCoreTracing;
 
   if (configuration) {
     if (StructuredData::Dictionary *dict = configuration->GetAsDictionary()) {
       dict->GetValueForKeyAsInteger("traceBufferSize", trace_buffer_size);
       dict->GetValueForKeyAsInteger("processBufferSizeLimit",
                                     process_buffer_size_limit);
       dict->GetValueForKeyAsBoolean("enableTsc", enable_tsc);
       dict->GetValueForKeyAsInteger("psbPeriod", psb_period);
       dict->GetValueForKeyAsBoolean("perCoreTracing", per_core_tracing);
     } else {
       return createStringError(inconvertibleErrorCode(),
                                "configuration object is not a dictionary");
     }
   }
 
   return Start(trace_buffer_size, process_buffer_size_limit, enable_tsc,
                psb_period, per_core_tracing);
 }
 
 llvm::Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,
-                                size_t trace_buffer_size, bool enable_tsc,
-                                Optional<size_t> psb_period) {
+                                uint64_t trace_buffer_size, bool enable_tsc,
+                                Optional<uint64_t> psb_period) {
   TraceIntelPTStartRequest request;
   request.trace_buffer_size = trace_buffer_size;
   request.enable_tsc = enable_tsc;
-  request.psb_period =
-      psb_period.map([](size_t val) { return static_cast<uint64_t>(val); });
+  request.psb_period = psb_period;
   request.type = GetPluginName().str();
   request.tids.emplace();
   for (lldb::tid_t tid : tids)
     request.tids->push_back(tid);
   return Trace::Start(toJSON(request));
 }
 
 Error TraceIntelPT::Start(llvm::ArrayRef<lldb::tid_t> tids,

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.cpp

 }
 
 bool fromJSON(const json::Value &value, JSONTraceSession &session, Path path) {
   ObjectMapper o(value, path);
   if (!o || !o.map("processes", session.processes) ||
       !o.map("type", session.type) || !o.map("cores", session.cores) ||
       !o.map("tscPerfZeroConversion", session.tsc_perf_zero_conversion))
     return false;
+  if (session.cores && !session.tsc_perf_zero_conversion) {
+    path.report(
+        "\"tscPerfZeroConversion\" is required when \"cores\" is provided");
+    return false;
+  }
   // We have to do this because the compiler fails at doing it automatically
   // because pt_cpu is not in a namespace
   if (!fromJSON(*value.getAsObject()->get("cpuInfo"), session.cpu_info,
                 path.field("cpuInfo")))
     return false;
   return true;
 }
 
 } // namespace trace_intel_pt
 } // namespace lldb_private

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCoreDecoder.h

+//===-- TraceIntelPTMultiCoreDecoder.h --------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPTMULTICOREDECODER_H
+#define LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPTMULTICOREDECODER_H
+
+#include "ThreadDecoder.h"
+
+namespace lldb_private {
+namespace trace_intel_pt {
+
+/// This class indicates the time interval in which a thread was running
+/// continuously on a cpu core.
+///
+/// In most cases both endpoints of the intervals can be accurately recovered
+/// from a context switch trace, but in some cases one of these endpoints might
+/// be guessed or not known at all, due to contention problems in the trace or
+/// because tracing was interrupted.
+///
+/// Note: we use the terms CPU and cores interchangeably.
+struct ThreadContinuousExecution {
+  enum class Variant {
+    /// Both endpoints are known
+    Complete,
+    /// The end is known and we have a guess for the start
+    HintedStart,
+    /// The start is known and we have a guess for the end
+    HintedEnd,
+    /// We only know the start. This might be the last entry of a core trace.
+    OnlyStart,
+    /// We only know the end. This might be the first entry or a core trace.
+    OnlyEnd,
+  } variant;
+
+  union {
+    struct {
+      uint64_t start;
+      uint64_t end;
+    } complete;
+    struct {
+      uint64_t start;
+    } only_start;
+    struct {
+      uint64_t end;
+    } only_end;
+    /// The following 'hinted' structures are useful when there are contention
+    /// problems in the trace
+    struct {
+      uint64_t hinted_start;
+      uint64_t end;
+    } hinted_start;
+    struct {
+      uint64_t start;
+      uint64_t hinted_end;
+    } hinted_end;
+  } tscs;
+
+  lldb::core_id_t core_id;
+  lldb::tid_t tid;
+
+  /// \return
+  ///   A tsc that we are certain of, either the start or the end.
+  uint64_t GetErrorFreeTSC() const;
+
+  /// Constructors for the different variants of this object
+  ///
+  /// \{
+  static ThreadContinuousExecution
+  CreateCompleteExecution(lldb::core_id_t core_id, lldb::tid_t tid,
+                          uint64_t start, uint64_t end);
+
+  static ThreadContinuousExecution
+  CreateHintedStartExecution(lldb::core_id_t core_id, lldb::tid_t tid,
+                             uint64_t hinted_start, uint64_t end);
+
+  static ThreadContinuousExecution
+  CreateHintedEndExecution(lldb::core_id_t core_id, lldb::tid_t tid,
+                           uint64_t start, uint64_t hinted_end);
+
+  static ThreadContinuousExecution
+  CreateOnlyEndExecution(lldb::core_id_t core_id, lldb::tid_t tid,
+                         uint64_t end);
+
+  static ThreadContinuousExecution
+  CreateOnlyStartExecution(lldb::core_id_t core_id, lldb::tid_t tid,
+                           uint64_t start);
+  /// \}
+
+  /// Comparator by TSCs
+  bool operator<(const ThreadContinuousExecution &o) const;
+
+private:
+  ThreadContinuousExecution(lldb::core_id_t core_id, lldb::tid_t tid)
+      : core_id(core_id), tid(tid) {}
+};
+
+/// Class used to decode a multi-core Intel PT trace. It assumes that each
+/// thread could have potentially been executed on different cores. It uses a
+/// context switch trace per CPU with timestamps to identify which thread owns
+/// each Intel PT decoded instruction and in which order. It also assumes that
+/// the Intel PT data and context switches might have gaps in their traces due
+/// to contention or race conditions.
+class TraceIntelPTMultiCoreDecoder {
+public:
+  /// \param[in] core_ids
+  ///   The list of cores where the traced programs were running on.
+  ///
+  /// \param[in] tid
+  ///   The full list of tids that were traced.
+  ///
+  /// \param[in] tsc_conversion
+  ///   The conversion values for converting between nanoseconds and TSCs.
+  TraceIntelPTMultiCoreDecoder(
+      TraceIntelPT &trace, llvm::ArrayRef<lldb::core_id_t> core_ids,
+      llvm::ArrayRef<lldb::tid_t> tids,
+      const LinuxPerfZeroTscConversion &tsc_conversion);
+
+  /// \return
+  ///   A \a DecodedThread for the \p thread by decoding its instructions on all
+  ///   CPUs, sorted by TSCs.
+  DecodedThreadSP Decode(Thread &thread);
+
+  /// \return
+  ///   \b true if the given \p tid is managed by this decoder, regardless of
+  ///   whether there's tracing data associated to it or not.
+  bool TracesThread(lldb::tid_t tid) const;
+
+  /// \return
+  ///   The number of continuous executions found for the given \p tid.
+  size_t GetNumContinuousExecutionsForThread(lldb::tid_t tid) const;
+
+  /// \return
+  ///   The total number of continuous executions found across CPUs.
+  size_t GetTotalContinuousExecutionsCount() const;
+
+private:
+  /// Traverse the context switch traces and recover the continuous executions
+  /// by thread.
+  llvm::Error DecodeContextSwitchTraces();
+
+  TraceIntelPT &m_trace;
+  std::set<lldb::core_id_t> m_cores;
+  std::set<lldb::tid_t> m_tids;
+  llvm::Optional<
+      llvm::DenseMap<lldb::tid_t, std::vector<ThreadContinuousExecution>>>
+      m_continuous_executions_per_thread;
+  LinuxPerfZeroTscConversion m_tsc_conversion;
+  /// This variable will be non-None if a severe error happened during the setup
+  /// of the decoder.
+  llvm::Optional<std::string> m_setup_error;
+};
+
+} // namespace trace_intel_pt
+} // namespace lldb_private
+
+#endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPTMULTICOREDECODER_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTMultiCoreDecoder.cpp

+//===-- TraceIntelPTMultiCoreDecoder.cpp ----------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "TraceIntelPTMultiCoreDecoder.h"
+
+#include "TraceIntelPT.h"
+
+#include "llvm/Support/Error.h"
+
+#include <linux/perf_event.h>
+
+using namespace lldb;
+using namespace lldb_private;
+using namespace lldb_private::trace_intel_pt;
+using namespace llvm;
+
+struct PerfContextSwitchRecord {
+  struct perf_event_header header;
+  uint32_t next_prev_pid;
+  uint32_t next_prev_tid;
+  uint32_t pid, tid;
+  uint64_t time_in_nanos;
+
+  bool IsOut() const { return header.misc & PERF_RECORD_MISC_SWITCH_OUT; }
+};
+
+struct ContextSwitchRecord {
+  uint64_t tsc;
+  bool is_out;
+  /// A pid of 0 indicates an execution in the kernel
+  lldb::pid_t pid;
+  lldb::tid_t tid;
+
+  bool IsOut() const { return is_out; }
+
+  bool IsIn() const { return !is_out; }
+};
+
+uint64_t ThreadContinuousExecution::GetErrorFreeTSC() const {
+  switch (variant) {
+  case Variant::Complete:
+    return tscs.complete.start; // end would also work
+  case Variant::HintedStart:
+    return tscs.hinted_start.end;
+  case Variant::HintedEnd:
+    return tscs.hinted_end.start;
+  case Variant::OnlyEnd:
+    return tscs.only_end.end;
+  case Variant::OnlyStart:
+    return tscs.only_start.start;
+  }
+}
+
+ThreadContinuousExecution ThreadContinuousExecution::CreateCompleteExecution(
+    lldb::core_id_t core_id, lldb::tid_t tid, uint64_t start, uint64_t end) {
+  ThreadContinuousExecution o(core_id, tid);
+  o.variant = Variant::Complete;
+  o.tscs.complete.start = start;
+  o.tscs.complete.end = end;
+  return o;
+}
+
+ThreadContinuousExecution ThreadContinuousExecution::CreateHintedStartExecution(
+    lldb::core_id_t core_id, lldb::tid_t tid, uint64_t hinted_start,
+    uint64_t end) {
+  ThreadContinuousExecution o(core_id, tid);
+  o.variant = Variant::HintedStart;
+  o.tscs.hinted_start.hinted_start = hinted_start;
+  o.tscs.hinted_start.end = end;
+  return o;
+}
+
+ThreadContinuousExecution ThreadContinuousExecution::CreateHintedEndExecution(
+    lldb::core_id_t core_id, lldb::tid_t tid, uint64_t start,
+    uint64_t hinted_end) {
+  ThreadContinuousExecution o(core_id, tid);
+  o.variant = Variant::HintedEnd;
+  o.tscs.hinted_end.start = start;
+  o.tscs.hinted_end.hinted_end = hinted_end;
+  return o;
+}
+
+ThreadContinuousExecution ThreadContinuousExecution::CreateOnlyEndExecution(
+    lldb::core_id_t core_id, lldb::tid_t tid, uint64_t end) {
+  ThreadContinuousExecution o(core_id, tid);
+  o.variant = Variant::OnlyEnd;
+  o.tscs.only_end.end = end;
+  return o;
+}
+
+ThreadContinuousExecution ThreadContinuousExecution::CreateOnlyStartExecution(
+    lldb::core_id_t core_id, lldb::tid_t tid, uint64_t start) {
+  ThreadContinuousExecution o(core_id, tid);
+  o.variant = Variant::OnlyStart;
+  o.tscs.only_start.start = start;
+  return o;
+}
+
+bool ThreadContinuousExecution::operator<(
+    const ThreadContinuousExecution &o) const {
+  // We can compare by GetErrorFreeTSC because context switches across CPUs can
+  // be sorted by any of its TSC.
+  return GetErrorFreeTSC() < o.GetErrorFreeTSC();
+}
+
+/// Tries to recover a continuous execution by analyzing two consecutive context
+/// switch records.
+static Error
+HandleContextSwitch(core_id_t core_id,
+                    const LinuxPerfZeroTscConversion &tsc_conversion,
+                    const ContextSwitchRecord &record,
+                    const Optional<ContextSwitchRecord> &prev_record,
+                    std::function<void(ThreadContinuousExecution &&execution)>
+                        on_new_thread_execution) {
+  if (!prev_record) {
+    if (record.IsOut())
+      on_new_thread_execution(ThreadContinuousExecution::CreateOnlyEndExecution(
+          core_id, record.tid, record.tsc));
+    // The 'in' case will be handled later when we try to look for its end
+    return Error::success();
+  }
+
+  const ContextSwitchRecord &prev = *prev_record;
+  if (prev.tsc > record.tsc)
+    return createStringError(
+        inconvertibleErrorCode(),
+        formatv("A context switch record out doesn't happen after the previous "
+                "record. Previous TSC= {0}, current TSC = {1}.",
+                prev.tsc, record.tsc));
+
+  if (record.IsIn() && prev.IsIn()) {
+    // We found two consecutive ins, which means that we didn't capture
+    // the end of the previous execution.
+    on_new_thread_execution(ThreadContinuousExecution::CreateHintedEndExecution(
+        core_id, prev.tid, prev.tsc, record.tsc - 1));
+  } else if (record.IsOut() && prev.IsOut()) {
+    // We found two consecutive outs, that means that we didn't capture
+    // the beginning of the current execution.
+    on_new_thread_execution(
+        ThreadContinuousExecution::CreateHintedStartExecution(
+            core_id, record.tid, prev.tsc + 1, record.tsc));
+  } else if (record.IsOut() && prev.IsIn()) {
+    if (record.pid == prev.pid && record.tid == prev.tid) {
+      /// A complete execution
+      on_new_thread_execution(
+          ThreadContinuousExecution::CreateCompleteExecution(
+              core_id, record.tid, prev.tsc, record.tsc));
+    } else {
+      // An out after the in of a different thread. The first one doesn't
+      // have an end, and the second one doesn't have a start.
+      on_new_thread_execution(
+          ThreadContinuousExecution::CreateHintedEndExecution(
+              core_id, prev.tid, prev.tsc, record.tsc - 1));
+      on_new_thread_execution(
+          ThreadContinuousExecution::CreateHintedStartExecution(
+              core_id, record.tid, prev.tsc + 1, record.tsc));
+    }
+  }
+  return Error::success();
+}
+
+/// Decodes a context switch trace gotten with perf_event_open.
+///
+/// \param[in] data
+///   The context switch trace in binary format.
+///
+/// \param[i] core_id
+///   The core_id where the trace were gotten from.
+///
+/// \param[in] tsc_conversion
+///   The conversion values used to confert nanoseconds to TSC.
+///
+/// \param[in] on_new_thread_execution
+///   Callback to be invoked whenever a continuous execution is recovered from
+///   the trace.
+static Error DecodePerfContextSwitchTrace(
+    ArrayRef<uint8_t> data, core_id_t core_id,
+    const LinuxPerfZeroTscConversion &tsc_conversion,
+    std::function<void(ThreadContinuousExecution &&execution)>
+        on_new_thread_execution) {
+  auto CreateError = [&](size_t offset, auto error) -> Error {
+    return createStringError(inconvertibleErrorCode(),
+                             formatv("Malformed perf context switch trace for "
+                                     "cpu {0} at offset {1}. {2}",
+                                     core_id, offset, error));
+  };
+
+  Optional<ContextSwitchRecord> prev_record;
+  for (size_t offset = 0; offset < data.size();) {
+    const PerfContextSwitchRecord &perf_record =
+        *reinterpret_cast<const PerfContextSwitchRecord *>(data.data() +
+                                                           offset);
+    // A record of 1000 uint64_t's or more should mean that the data is wrong
+    if (perf_record.header.size == 0 ||
+        perf_record.header.size > sizeof(uint64_t) * 1000)
+      return CreateError(offset, formatv("A record of {0} bytes was found.",
+                                         perf_record.header.size));
+
+    // We add + 100 to this record because some systems might have custom
+    // records. In any case, we are looking only for abnormal data.
+    if (perf_record.header.type >= PERF_RECORD_MAX + 100)
+      return CreateError(offset, formatv("Invalid record type {0} was found.",
+                                         perf_record.header.type));
+
+    if (perf_record.header.type == PERF_RECORD_SWITCH_CPU_WIDE) {
+      ContextSwitchRecord record{tsc_conversion.ToTSC(std::chrono::nanoseconds(
+                                     perf_record.time_in_nanos)),
+                                 perf_record.IsOut(),
+                                 static_cast<lldb::pid_t>(perf_record.pid),
+                                 static_cast<lldb::tid_t>(perf_record.tid)};
+
+      if (Error err = HandleContextSwitch(core_id, tsc_conversion, record,
+                                          prev_record, on_new_thread_execution))
+        return CreateError(offset, toString(std::move(err)));
+
+      prev_record = record;
+    }
+    offset += perf_record.header.size;
+  }
+
+  // We might have an incomplete last record
+  if (prev_record && prev_record->IsIn())
+    on_new_thread_execution(ThreadContinuousExecution::CreateOnlyStartExecution(
+        core_id, prev_record->tid, prev_record->tsc));
+
+  return Error::success();
+}
+
+TraceIntelPTMultiCoreDecoder::TraceIntelPTMultiCoreDecoder(
+    TraceIntelPT &trace, ArrayRef<core_id_t> core_ids, ArrayRef<tid_t> tids,
+    const LinuxPerfZeroTscConversion &tsc_conversion)
+    : m_trace(trace), m_cores(core_ids.begin(), core_ids.end()),
+      m_tids(tids.begin(), tids.end()), m_tsc_conversion(tsc_conversion) {}
+
+bool TraceIntelPTMultiCoreDecoder::TracesThread(lldb::tid_t tid) const {
+  return m_tids.count(tid);
+}
+
+DecodedThreadSP TraceIntelPTMultiCoreDecoder::Decode(Thread &thread) {
+  if (Error err = DecodeContextSwitchTraces())
+    return std::make_shared<DecodedThread>(thread.shared_from_this(),
+                                           std::move(err));
+
+  return std::make_shared<DecodedThread>(
+      thread.shared_from_this(),
+      createStringError(inconvertibleErrorCode(), "unimplemented"));
+}
+
+Error TraceIntelPTMultiCoreDecoder::DecodeContextSwitchTraces() {
+  if (m_setup_error)
+    return createStringError(inconvertibleErrorCode(), m_setup_error->c_str());
+
+  if (m_continuous_executions_per_thread)
+    return Error::success();
+
+  m_continuous_executions_per_thread.emplace();
+
+  auto do_decode = [&]() -> Error {
+    // We'll decode all context switch traces, identify continuous executions
+    // and group them by thread.
+    for (core_id_t core_id : m_cores) {
+      Error err = m_trace.OnCoreBinaryDataRead(
+          core_id, IntelPTDataKinds::kPerfContextSwitchTrace,
+          [&](ArrayRef<uint8_t> data) -> Error {
+            return DecodePerfContextSwitchTrace(
+                data, core_id, m_tsc_conversion,
+                [&](const ThreadContinuousExecution &execution) {
+                  (*m_continuous_executions_per_thread)[execution.tid]
+                      .push_back(execution);
+                });
+          });
+      if (err) {
+        m_setup_error = toString(std::move(err));
+        return createStringError(inconvertibleErrorCode(),
+                                 m_setup_error->c_str());
+      }
+    }
+    // We now sort the executions of each to have them ready for instruction
+    // decoding
+    for (auto &tid_executions : *m_continuous_executions_per_thread)
+      std::sort(tid_executions.second.begin(), tid_executions.second.end());
+
+    return Error::success();
+  };
+
+  return m_trace.GetTimer().ForGlobal().TimeTask<Error>(
+      "Context switch trace decoding", do_decode);
+}
+
+size_t TraceIntelPTMultiCoreDecoder::GetNumContinuousExecutionsForThread(
+    lldb::tid_t tid) const {
+  if (!m_continuous_executions_per_thread)
+    return 0;
+  auto it = m_continuous_executions_per_thread->find(tid);
+  if (it == m_continuous_executions_per_thread->end())
+    return 0;
+  return it->second.size();
+}
+
+size_t TraceIntelPTMultiCoreDecoder::GetTotalContinuousExecutionsCount() const {
+  if (!m_continuous_executions_per_thread)
+    return 0;
+  size_t count = 0;
+  for (const auto &kv : *m_continuous_executions_per_thread)
+    count += kv.second.size();
+  return count;
+}

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionFileParser.cpp

TraceIntelPTSessionFileParser::ParseSessionFile(
   };
 
   for (const JSONProcess &process : session.processes) {
     if (Expected<ParsedProcess> parsed_process = ParseProcess(process))
       parsed_processes.push_back(std::move(*parsed_process));
     else
       return HandleError(parsed_process.takeError());
   }
+
   return parsed_processes;
 }
 
 StringRef TraceIntelPTSessionFileParser::GetSchema() {
   static std::string schema;
   if (schema.empty()) {
     schema = R"({
   "type": "intel-pt",
StringRef TraceIntelPTSessionFileParser::GetSchema() {
   "cores"?: [
     {
       "coreId": integer,
           // Id of this CPU core.
       "traceBuffer": string,
           // Path to the raw Intel PT buffer for this core.
       "contextSwitchTrace": string,
           // Path to the raw perf_event_open context switch trace file for this core.
+          // The perf_event must have been configured with PERF_SAMPLE_TID and
+          // PERF_SAMPLE_TIME, as well as sample_id_all = 1.
     }
   ],
   "tscPerfZeroConversion"?: {
     // Values used to convert between TSCs and nanoseconds. See the time_zero
     // section in https://man7.org/linux/man-pages/man2/perf_event_open.2.html
     // for for information.
 
     "timeMult": integer,

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionSaver.cpp

 using namespace lldb_private;
 using namespace lldb_private::trace_intel_pt;
 using namespace llvm;
 
 static llvm::Error WriteBytesToDisk(FileSpec &output_file,
                                     ArrayRef<uint8_t> data) {
   std::basic_fstream<char> out_fs = std::fstream(
       output_file.GetPath().c_str(), std::ios::out | std::ios::binary);
-  out_fs.write(reinterpret_cast<const char *>(&data[0]),
-               data.size() * sizeof(uint8_t));
+  if (!data.empty())
+    out_fs.write(reinterpret_cast<const char *>(&data[0]), data.size());
+
   out_fs.close();
   if (!out_fs)
     return createStringError(inconvertibleErrorCode(),
                              formatv("couldn't write to the file {0}",
                                      output_file.GetPath().c_str()));
   return Error::success();
 }
 

lldb/source/Target/Trace.cpp

 
 Expected<std::string> Trace::GetLiveProcessState() {
   if (!m_live_process)
     return createStringError(inconvertibleErrorCode(),
                              "Tracing requires a live process.");
   return m_live_process->TraceGetState(GetPluginName());
 }
 
-Optional<size_t> Trace::GetLiveThreadBinaryDataSize(lldb::tid_t tid,
+Optional<uint64_t> Trace::GetLiveThreadBinaryDataSize(lldb::tid_t tid,
                                                       llvm::StringRef kind) {
   auto it = m_live_thread_data.find(tid);
   if (it == m_live_thread_data.end())
     return None;
-  std::unordered_map<std::string, size_t> &single_thread_data = it->second;
+  std::unordered_map<std::string, uint64_t> &single_thread_data = it->second;
   auto single_thread_data_it = single_thread_data.find(kind.str());
   if (single_thread_data_it == single_thread_data.end())
     return None;
   return single_thread_data_it->second;
 }
 
-Optional<size_t> Trace::GetLiveCoreBinaryDataSize(lldb::core_id_t core_id,
+Optional<uint64_t> Trace::GetLiveCoreBinaryDataSize(lldb::core_id_t core_id,
                                                     llvm::StringRef kind) {
   auto it = m_live_core_data.find(core_id);
   if (it == m_live_core_data.end())
     return None;
-  std::unordered_map<std::string, size_t> &single_core_data = it->second;
+  std::unordered_map<std::string, uint64_t> &single_core_data = it->second;
   auto single_thread_data_it = single_core_data.find(kind.str());
   if (single_thread_data_it == single_core_data.end())
     return None;
   return single_thread_data_it->second;
 }
 
-Optional<size_t> Trace::GetLiveProcessBinaryDataSize(llvm::StringRef kind) {
+Optional<uint64_t> Trace::GetLiveProcessBinaryDataSize(llvm::StringRef kind) {
   auto data_it = m_live_process_data.find(kind.str());
   if (data_it == m_live_process_data.end())
     return None;
   return data_it->second;
 }
 
 Expected<std::vector<uint8_t>>
 Trace::GetLiveThreadBinaryData(lldb::tid_t tid, llvm::StringRef kind) {
   if (!m_live_process)
     return createStringError(inconvertibleErrorCode(),
                              "Tracing requires a live process.");
-  llvm::Optional<size_t> size = GetLiveThreadBinaryDataSize(tid, kind);
+  llvm::Optional<uint64_t> size = GetLiveThreadBinaryDataSize(tid, kind);
   if (!size)
     return createStringError(
         inconvertibleErrorCode(),
         "Tracing data \"%s\" is not available for thread %" PRIu64 ".",
         kind.data(), tid);
 
   TraceGetBinaryDataRequest request{GetPluginName().str(), kind.str(),   tid,
                                     /*core_id=*/None,      /*offset=*/0, *size};
   return m_live_process->TraceGetBinaryData(request);
 }
 
 Expected<std::vector<uint8_t>>
 Trace::GetLiveCoreBinaryData(lldb::core_id_t core_id, llvm::StringRef kind) {
   if (!m_live_process)
     return createStringError(inconvertibleErrorCode(),
                              "Tracing requires a live process.");
-  llvm::Optional<size_t> size = GetLiveCoreBinaryDataSize(core_id, kind);
+  llvm::Optional<uint64_t> size = GetLiveCoreBinaryDataSize(core_id, kind);
   if (!size)
     return createStringError(
         inconvertibleErrorCode(),
         "Tracing data \"%s\" is not available for core_id %" PRIu64 ".",
         kind.data(), core_id);
 
   TraceGetBinaryDataRequest request{GetPluginName().str(), kind.str(),
                                     /*tid=*/None,          core_id,
                                     /*offset=*/0,          *size};
   return m_live_process->TraceGetBinaryData(request);
 }
 
 Expected<std::vector<uint8_t>>
 Trace::GetLiveProcessBinaryData(llvm::StringRef kind) {
   if (!m_live_process)
     return createStringError(inconvertibleErrorCode(),
                              "Tracing requires a live process.");
-  llvm::Optional<size_t> size = GetLiveProcessBinaryDataSize(kind);
+  llvm::Optional<uint64_t> size = GetLiveProcessBinaryDataSize(kind);
   if (!size)
     return createStringError(
         inconvertibleErrorCode(),
         "Tracing data \"%s\" is not available for the process.", kind.data());
 
   TraceGetBinaryDataRequest request{GetPluginName().str(), kind.str(),
                                     /*tid=*/None,          /*core_id*/ None,
                                     /*offset=*/0,          *size};

lldb/source/Utility/TraceGDBRemotePackets.cpp

void TraceGetStateResponse::AddWarning(StringRef warning) {
   if (!warnings)
     warnings.emplace();
   warnings->push_back(warning.data());
 }
 
 bool fromJSON(const json::Value &value, TraceCoreState &packet,
               json::Path path) {
   ObjectMapper o(value, path);
-  int64_t core_id;
+  uint64_t core_id;
   if (!o || !o.map("coreId", core_id) ||
       !o.map("binaryData", packet.binary_data))
     return false;
   packet.core_id = static_cast<lldb::core_id_t>(core_id);
   return true;
 }
 
 json::Value toJSON(const TraceCoreState &packet) {
return json::Value(Object{{"type", packet.type},
                             {"tid", packet.tid},
                             {"coreId", packet.core_id},
                             {"size", packet.size}});
 }
 
 bool fromJSON(const json::Value &value, TraceGetBinaryDataRequest &packet,
               Path path) {
   ObjectMapper o(value, path);
-  return o && o.map("type", packet.type) && o.map("kind", packet.kind) &&
-         o.map("tid", packet.tid) && o.map("offset", packet.offset) &&
-         o.map("size", packet.size) && o.map("coreId", packet.core_id);
+  Optional<uint64_t> core_id;
+  if (!o || !o.map("type", packet.type) || !o.map("kind", packet.kind) ||
+      !o.map("tid", packet.tid) || !o.map("offset", packet.offset) ||
+      !o.map("size", packet.size) || !o.map("coreId", core_id))
+    return false;
+
+  if (core_id)
+    packet.core_id = static_cast<lldb::core_id_t>(*core_id);
+  return true;
 }
 /// \}
 
 } // namespace lldb_private

lldb/source/Utility/TraceIntelPTGDBRemotePackets.cpp

json::Value toJSON(const TraceIntelPTStartRequest &packet) {
   obj.try_emplace("traceBufferSize", packet.trace_buffer_size);
   obj.try_emplace("processBufferSizeLimit", packet.process_buffer_size_limit);
   obj.try_emplace("psbPeriod", packet.psb_period);
   obj.try_emplace("enableTsc", packet.enable_tsc);
   obj.try_emplace("perCoreTracing", packet.per_core_tracing);
   return base;
 }
 
-std::chrono::nanoseconds LinuxPerfZeroTscConversion::ToNanos(uint64_t tsc) {
+std::chrono::nanoseconds
+LinuxPerfZeroTscConversion::ToNanos(uint64_t tsc) const {
   uint64_t quot = tsc >> time_shift;
   uint64_t rem_flag = (((uint64_t)1 << time_shift) - 1);
   uint64_t rem = tsc & rem_flag;
   return std::chrono::nanoseconds{time_zero + quot * time_mult +
                                   ((rem * time_mult) >> time_shift)};
 }
 
+uint64_t
+LinuxPerfZeroTscConversion::ToTSC(std::chrono::nanoseconds nanos) const {
+  uint64_t time = nanos.count() - time_zero;
+  uint64_t quot = time / time_mult;
+  uint64_t rem = time % time_mult;
+  return (quot << time_shift) + (rem << time_shift) / time_mult;
+}
+
 json::Value toJSON(const LinuxPerfZeroTscConversion &packet) {
   return json::Value(json::Object{
       {"timeMult", packet.time_mult},
       {"timeShift", packet.time_shift},
       {"timeZero", packet.time_zero},
   });
 }
 

lldb/test/API/commands/trace/TestTraceDumpInfo.py

 thread #1: tid = 3842849
   Total number of instructions: 21
 
   Memory usage:
     Raw trace size: 4 KiB
     Total approximate memory usage (excluding raw trace): 1.27 KiB
     Average memory usage per instruction (excluding raw trace): 61.76 bytes
 
-  Timing:
-    Decoding instructions: ''', '''s
+  Timing for this thread:
+    Decoding instructions: ''', '''
 
   Events:
     Number of instructions with events: 1
     Number of individual events: 1
       paused: 1
 
   Errors:
     Number of TSC decoding errors: 0'''],
             patterns=["Decoding instructions: \d.\d\ds"])

lldb/test/API/commands/trace/TestTraceLoad.py

 thread #1: tid = 3842849
   Total number of instructions: 21
 
   Memory usage:
     Raw trace size: 4 KiB
     Total approximate memory usage (excluding raw trace): 1.27 KiB
     Average memory usage per instruction (excluding raw trace): 61.76 bytes
 
-  Timing:
-    Decoding instructions: ''', '''s
+  Timing for this thread:
+    Decoding instructions: ''', '''
 
   Events:
     Number of instructions with events: 1
     Number of individual events: 1
       paused: 1
 
   Errors:
     Number of TSC decoding errors: 0'''])

lldb/test/API/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.py

def testStartPerCoreSession(self):
             self.assertTrue(context_switch_size is not None)
             self.assertTrue(trace_buffer_size is not None)
             if context_switch_size > 0:
                 found_non_empty_context_switch = True
 
         # We must have captured the context switch of when the target resumed
         self.assertTrue(found_non_empty_context_switch)
 
+        self.expect("thread trace dump instructions", substrs=['unimplemented'])
 
         self.traceStopProcess()