[trace][intel pt] Support events

[wallace]

Summary

A trace might contain events traced during the target's execution. For
example, a thread might be paused for some period of time due to context
switches or breakpoints, which actually force a context switch. Not only
that, a trace might be paused because the CPU decides to trace only a
specific part of the target, like the address filtering provided by
intel pt, which will cause pause events. Besides this case, other kinds
of events might exist.

This patch adds the method TraceCursor::GetEvents()` that returns the
list of events that happened right before the instruction being pointed
at by the cursor. Some refactors were done to make this change simpler.

Besides this new API, the instruction dumper now supports the -e flag
which shows pause events, like in the following example, where pauses
happened due to breakpoints.

thread #1: tid = 2717361
  a.out`main + 20 at main.cpp:27:20
    0: 0x00000000004023d9    leaq   -0x1200(%rbp), %rax
  [paused]
    1: 0x00000000004023e0    movq   %rax, %rdi
  [paused]
    2: 0x00000000004023e3    callq  0x403a62                  ; std::vector<int, std::allocator<int> >::vector at stl_vector.h:391:7
  a.out`std::vector<int, std::allocator<int> >::vector() at stl_vector.h:391:7
    3: 0x0000000000403a62    pushq  %rbp
    4: 0x0000000000403a63    movq   %rsp, %rbp

The dump info command has also been updated and now it shows the
number of instructions that have associated events.

[wallace]

formatting

[jj10306]

lldb/include/lldb/lldb-enumerations.h

1151

What are some other events you could forsee adding to this enum?

1155

Will this paused event always be due to a context switch? If so, we should rename it as such to reduce ambiguity

lldb/source/Plugins/Trace/intel-pt/DecodedThread.cpp

137

What is being "Set" and why have the extra layer of indirection with the private AppendError(Error &&e)? The name is a little confusing imo because all this does is call Append internally.
Any reason to not have AppendError public and call that directly?

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

75

225

What value does this providing the total number of instructions with events add? I think it could be more useful to provide a specific event as a parameter and return the total number of instructions with that particular event. wdyt?

258

I know this isn't related to these changes, but this should be updated to be consistent with the other instruction_index -> XXX maps in this class.

258

lldb/source/Plugins/Trace/intel-pt/LibiptDecoder.cpp

68

consider adding docs on insn being an "out" parameter so it's a little more clear

186

Why have the internal buffer overflow event as an error instead of a variant of the newly introduced Events enum?

lldb/test/API/commands/trace/TestTraceDumpInfo.py

52

Will this always be 1 or is there a possibility that it could be different depending on the number of context switches?

[wallace]

lldb/include/lldb/lldb-enumerations.h

1151

paging, execution mode change (from 64 to 32 bit mode or viceversa), power state change, core bus ratio change (i.e. frequency change). Some of these might include an additional payload, which should be gotten using other APIs.

1155

not really. Two examples:

• context switch
• ip filtering -> in this case tracing stopped at CPU level because an unwanted memory region was executed. In this case the process is still running

lldb/source/Plugins/Trace/intel-pt/DecodedThread.cpp

137

The reason is that I want all callers to invoke Append() except for exceptional cases. The benefit of Append is that can provide an error with events and timing information. SetAsFailed is receiving only an error without additional data to be used only in extreme situations where there's no event or timing data available, which happens if the decoding setup fails.

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

75

oh, i wanted to mean if and only if. Or is that too mathematical and less colloquial?

225

good idea

258

in this case we can't do that, because when doing random accesses in the trace, we need to quickly find the most recent TSC for the new instruction, which is done with binary search using a map. This is impossible in a hash map like DenseMap

lldb/source/Plugins/Trace/intel-pt/LibiptDecoder.cpp

68

good catch

186

that's because an overflow means that we lost data, and therefore we should show that as an error in the trace. Remember that errors mean that instructions were lost, which is exactly what happens in an overflow case. This is not just an event that could be ignored.

lldb/test/API/commands/trace/TestTraceDumpInfo.py

52

This is using trace load, which uses an offline trace, so it will never change.

[wallace]

address comments. Still pending response about the iff word.

I've just created a stats objects for events with some util functions that make
the code much cleaner.

[jj10306]

lldb/include/lldb/Target/TraceCursor.h

265

nit: maybe this will change in the future to where this will have data and instance members, but if not, consider just using a namespace here to house the utility functions since the class isn't really providing any state?

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

75

Lol my first thought actually was "I wonder if he meant if and only if, so this isn't a typo".
Up to you if you want to keep if or leave it as iff

258

I know this is a pedantic discussion, but isn't lookup in hash tables amortized O(1) (assuming a good hash that doesn't take too long to execute or produce the same hash for different values) whereas in an ordered map it's O(logn)?
IIUC, you should only use an ordered map when you rely on the order property of it.

In any case, we should use uint64_t here instead of size_t to be consistent with the type for instruction index used in the other maps.

[wallace]

lldb/include/lldb/Target/TraceCursor.h

265

great idea

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

75

i'll use if and only if

258

In any case, we should use uint64_t here instead of size_t to be consistent with the type for instruction index used in the other maps.

you are right with this one.

IIUC, you should only use an ordered map when you rely on the order property of it.

yes, and that's what we use for the timestamps. As there are a small number of unique timestamps, we are keeping this map 'insn_index -> timestamp'. We need its order property because when we do GoToId(insn_id) we need to look for most recent entry in the map that has a timestamp, and we use that timestamp for our new instruction. We use map.lower_bound for that. With a hashmap we couldn't do that.

[wallace]

address comments

[jj10306]

Thanks for making those changes - lgtm!

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

258

We use map.lower_bound for that. With a hashmap we couldn't do that.

Ahhh yes, that makes sense - I forgot we were using that

lldb/include/lldb/Target/TraceCursor.h

public:
   /// Get the hardware counter of a given type associated with the current
   /// instruction. Each architecture might support different counters. It might
   /// happen that only some instructions of an entire trace have a given counter
   /// associated with them.
   ///
   /// \param[in] counter_type
   ///    The counter type.
   /// \return
-  ///     The value of the counter or \b llvm::None if not available.
+  ///    The value of the counter or \b llvm::None if not available.
   virtual llvm::Optional<uint64_t> GetCounter(lldb::TraceCounter counter_type) = 0;
 
+  /// Get a bitmask with a list of events that happened chronologically right
+  /// before the current instruction or error, but after the previous
+  /// instruction.
+  ///
+  /// \return
+  ///   The bitmask of events.
+  virtual lldb::TraceEvents GetEvents() = 0;
+
   /// \return
   ///     The \a lldb::TraceInstructionControlFlowType categories the
   ///     instruction the cursor is pointing at falls into. If the cursor points
   ///     to an error in the trace, return \b 0.
   virtual lldb::TraceInstructionControlFlowType
   GetInstructionControlFlowType() = 0;
   /// \}
 
 protected:
   ExecutionContextRef m_exe_ctx_ref;
 
   lldb::TraceInstructionControlFlowType m_granularity =
       lldb::eTraceInstructionControlFlowTypeInstruction;
   bool m_ignore_errors = false;
   bool m_forwards = false;
 };
 
 } // namespace lldb_private
 
 #endif // LLDB_TARGET_TRACE_CURSOR_H

lldb/include/lldb/Target/TraceInstructionDumper.h

struct TraceInstructionDumperOptions {
   /// recent instruction.
   bool forwards = false;
   /// Dump only instruction addresses without disassembly nor symbol
   /// information.
   bool raw = false;
   /// For each instruction, print the corresponding timestamp counter if
   /// available.
   bool show_tsc = false;
+  /// Dump the events that happened between instructions.
+  bool show_events = false;
   /// Optional custom id to start traversing from.
   llvm::Optional<uint64_t> id = llvm::None;
   /// Optional number of instructions to skip from the starting position
   /// of the cursor.
   llvm::Optional<size_t> skip = llvm::None;
 };
 
 /// Class used to dump the instructions of a \a TraceCursor using its current
private:
   void SetNoMoreData();
 
   /// Move the cursor one step.
   ///
   /// \return
   ///     \b true if the cursor moved.
   bool TryMoveOneStep();
 
+  void PrintEvents();
+
   lldb::TraceCursorUP m_cursor_up;
   TraceInstructionDumperOptions m_options;
   Stream &m_s;
   /// If \b true, all the instructions have been traversed.
   bool m_no_more_data = false;
 };
 
 } // namespace lldb_private
 
 #endif // LLDB_TARGET_TRACE_INSTRUCTION_DUMPER_H

lldb/include/lldb/lldb-enumerations.h

 };
 
 // Type of counter values associated with instructions in a trace.
 enum TraceCounter {
   // Timestamp counter, like the one offered by Intel CPUs (TSC).
   eTraceCounterTSC,
 };
 
+// Events that might happen during a trace session.
+FLAGS_ENUM(TraceEvents){
+    // Tracing was paused. If instructions were executed after resuming,
+    // the TraceCursor used to traverse the trace should provide an error
+    // signalinig this data loss.
+    eTraceEventPaused = (1u << 0),
+};
+
 } // namespace lldb
 
 #endif // LLDB_LLDB_ENUMERATIONS_H

lldb/source/Commands/CommandObjectThread.cpp

Status SetOptionValue(uint32_t option_idx, llvm::StringRef option_arg,
       case 'f': {
         m_dumper_options.forwards = true;
         break;
       }
       case 't': {
         m_dumper_options.show_tsc = true;
         break;
       }
+      case 'e': {
+        m_dumper_options.show_events = true;
+        break;
+      }
       case 'C': {
         m_continue = true;
         break;
       }
       default:
         llvm_unreachable("Unimplemented option");
       }
       return error;

lldb/source/Commands/Options.td

def thread_trace_dump_instructions_skip: Option<"skip", "s">, Group<1>,
     "before starting the traversal.">;
   def thread_trace_dump_instructions_raw : Option<"raw", "r">,
     Group<1>,
     Desc<"Dump only instruction address without disassembly nor symbol "
     "information.">;
   def thread_trace_dump_instructions_show_tsc : Option<"tsc", "t">, Group<1>,
     Desc<"For each instruction, print the corresponding timestamp counter if "
     "available.">;
+  def thread_trace_dump_instructions_hide_events : Option<"events", "e">,
+    Group<1>,
+    Desc<"Dump the events that happened during the execution of the target.">;
   def thread_trace_dump_instructions_continue: Option<"continue", "C">,
     Group<1>,
-    Desc<"Continue dumping instructions right where the previous invocation of this "
-    "command was left, or from the beginning if this is the first invocation. The --skip "
-    "argument is discarded and the other arguments are preserved from the previous "
-    "invocation when possible.">;
+    Desc<"Continue dumping instructions right where the previous invocation of "
+    "this command was left, or from the beginning if this is the first "
+    "invocation. The --skip argument is discarded and the other arguments are "
+    "preserved from the previous invocation when possible.">;
 }
 
 let Command = "thread trace dump info" in {
   def thread_trace_dump_info_verbose : Option<"verbose", "v">, Group<1>,
     Desc<"show verbose thread trace dump info">;
 }
 
 let Command = "type summary add" in {

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

 #include "lldb/Target/Trace.h"
 #include "lldb/Utility/TraceIntelPTGDBRemotePackets.h"
 
 #include "intel-pt.h"
 
 namespace lldb_private {
 namespace trace_intel_pt {
 
+/// libipt status utils
+/// \{
+bool IsLibiptError(int libipt_status);
+
+bool IsEndOfStream(int libipt_status);
+
+bool IsTscUnavailable(int libipt_status);
+/// \}
+
 /// Class for representing a libipt decoding error.
 class IntelPTError : public llvm::ErrorInfo<IntelPTError> {
 public:
   static char ID;
 
   /// \param[in] libipt_error_code
   ///     Negative number returned by libipt when decoding the trace and
   ///     signaling errors.
public:
 
   void log(llvm::raw_ostream &OS) const override;
 
 private:
   int m_libipt_error_code;
   lldb::addr_t m_address;
 };
 
+/// Helper struct for building an instruction or error from the decoder.
+/// It holds associated events and timing information.
+struct DecodedInstruction {
+  DecodedInstruction() {
+    pt_insn.ip = LLDB_INVALID_ADDRESS;
+    libipt_error = pte_ok;
+  }
+
+  DecodedInstruction(int libipt_error_code) : DecodedInstruction() {
+    libipt_error = libipt_error_code;
+  }
+
+  /// \return \b true iff this struct holds a libipt error.
+  explicit operator bool() const;
+
+  int libipt_error;
+  lldb::TraceEvents events = (lldb::TraceEvents)0;
+  llvm::Optional<uint64_t> tsc = llvm::None;
+  pt_insn pt_insn;
+};
+
 /// \class DecodedThread
 /// Class holding the instructions and function call hierarchy obtained from
 /// decoding a trace, as well as a position cursor used when reverse debugging
 /// the trace.
 ///
 /// Each decoded thread contains a cursor to the current position the user is
 /// stopped at. See \a Trace::GetCursorPosition for more information.
 class DecodedThread : public std::enable_shared_from_this<DecodedThread> {
struct LibiptErrors {
     void RecordError(int libipt_error_code);
   };
 
   DecodedThread(lldb::ThreadSP thread_sp);
 
   /// Utility constructor that initializes the trace with a provided error.
   DecodedThread(lldb::ThreadSP thread_sp, llvm::Error &&err);
 
-  /// Append a successfully decoded instruction.
-  void AppendInstruction(const pt_insn &instruction);
+  /// Append an instruction or a libipt error.
+  void Append(const DecodedInstruction &insn);
 
-  /// Append a sucessfully decoded instruction with an associated TSC timestamp.
-  void AppendInstruction(const pt_insn &instruction, uint64_t tsc);
-
-  /// Append a decoding error (i.e. an instruction that failed to be decoded).
-  void AppendError(llvm::Error &&error);
+  /// Append an error signaling that decoding completely failed.
+  void SetAsFailed(llvm::Error &&error);
 
-  /// Append a decoding error with a corresponding TSC.
-  void AppendError(llvm::Error &&error, uint64_t tsc);
+  /// Get a bitmask with the events that happened chronologically right before
+  /// the instruction pointed by the given instruction index, but after the
+  /// previous instruction.
+  lldb::TraceEvents GetEvents(int insn_index);
 
   /// Get the total number of instruction pointers from the decoded trace.
   /// This will include instructions that indicate errors (or gaps) in the
   /// trace. For an instruction error, you can access its underlying error
   /// message with the \a GetErrorByInstructionIndex() method.
   size_t GetInstructionsCount() const;
 
   /// \return
public:
   /// Record an error decoding a TSC timestamp.
   ///
   /// See \a GetTscErrors() for more documentation.
   ///
   /// \param[in] libipt_error_code
   ///   An error returned by the libipt library.
   void RecordTscError(int libipt_error_code);
 
+  /// \return
+  ///   The number of instructions with associated events.
+  size_t GetInstructionsWithEventsCount() const;
+
   /// The approximate size in bytes used by this instance,
   /// including all the already decoded instructions.
   size_t CalculateApproximateMemoryUsage() const;
 
   lldb::ThreadSP GetThread();
 
 private:
+  /// Append a decoding error given an llvm::Error.
+  void AppendError(llvm::Error &&error);
+
   /// Notify this class that the last added instruction or error has
   /// an associated TSC.
   void RecordTscForLastInstruction(uint64_t tsc);
 
   /// When adding new members to this class, make sure
   /// to update \a CalculateApproximateMemoryUsage() accordingly.
   lldb::ThreadSP m_thread_sp;
   /// The low level storage of all instruction addresses. Each instruction has
   /// an index in this vector and it will be used in other parts of the code.
   std::vector<lldb::addr_t> m_instruction_ips;
   /// The size in bytes of each instruction.
   std::vector<uint8_t> m_instruction_sizes;
   /// The libipt instruction class for each instruction.
   std::vector<pt_insn_class> m_instruction_classes;
 
   /// This map contains the TSCs of the decoded instructions. It maps
   /// `instruction index -> TSC`, where `instruction index` is the first index
   /// at which the mapped TSC appears. We use this representation because TSCs
   /// are sporadic and we can think of them as ranges. If TSCs are present in
   /// the trace, all instructions will have an associated TSC, including the
   /// first one. Otherwise, this map will be empty.
   std::map<size_t, uint64_t> m_instruction_timestamps;
   /// This is the chronologically last TSC that has been added.
   llvm::Optional<uint64_t> m_last_tsc = llvm::None;
   // This variables stores the messages of all the error instructions in the
   // trace. It maps `instruction index -> error message`.
   llvm::DenseMap<uint64_t, std::string> m_errors;
-  /// The size in bytes of the raw buffer before decoding. It might be None if
-  /// the decoding failed.
-  llvm::Optional<size_t> m_raw_trace_size;
+  /// This variable stores the bitmask of events that happened right before
+  /// the instruction given as a key. It maps `instruction index -> events`.
+  llvm::DenseMap<uint64_t, lldb::TraceEvents> m_events;
   /// All occurrences of libipt errors when decoding TSCs.
   LibiptErrors m_tsc_errors;
   /// Total amount of time spent decoding.
   std::chrono::milliseconds m_total_decoding_time{0};
 };
 
 using DecodedThreadSP = std::shared_ptr<DecodedThread>;
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_DECODEDTHREAD_H

lldb/source/Plugins/Trace/intel-pt/DecodedThread.cpp

 
 #include <memory>
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::trace_intel_pt;
 using namespace llvm;
 
+bool lldb_private::trace_intel_pt::IsLibiptError(int libipt_status) {
+  return libipt_status < 0;
+}
+
+bool lldb_private::trace_intel_pt::IsEndOfStream(int libipt_status) {
+  return libipt_status == -pte_eos;
+}
+
+bool lldb_private::trace_intel_pt::IsTscUnavailable(int libipt_status) {
+  return libipt_status == -pte_no_time;
+}
+
 char IntelPTError::ID;
 
 IntelPTError::IntelPTError(int libipt_error_code, lldb::addr_t address)
-    : m_libipt_error_code(libipt_error_code), m_address(address) {
+    : m_libipt_error_code(-std::abs(libipt_error_code)), m_address(address) {
   assert(libipt_error_code < 0);
 }
 
 void IntelPTError::log(llvm::raw_ostream &OS) const {
   const char *libipt_error_message = pt_errstr(pt_errcode(m_libipt_error_code));
   if (m_address != LLDB_INVALID_ADDRESS && m_address > 0) {
     write_hex(OS, m_address, HexPrintStyle::PrefixLower, 18);
     OS << "    ";
   }
   OS << "error: " << libipt_error_message;
 }
 
+DecodedInstruction::operator bool() const {
+  return !IsLibiptError(libipt_error);
+}
+
 size_t DecodedThread::GetInstructionsCount() const {
   return m_instruction_ips.size();
 }
 
 lldb::addr_t DecodedThread::GetInstructionLoadAddress(size_t insn_index) const {
   return m_instruction_ips[insn_index];
 }
 
if (!m_last_tsc || *m_last_tsc != tsc) {
     // instructions to use the first valid TSC, so that all the trace has TSCs.
     size_t start_index =
         m_instruction_timestamps.empty() ? 0 : m_instruction_ips.size() - 1;
     m_instruction_timestamps.emplace(start_index, tsc);
     m_last_tsc = tsc;
   }
 }
 
-void DecodedThread::AppendInstruction(const pt_insn &insn) {
-  m_instruction_ips.emplace_back(insn.ip);
-  m_instruction_sizes.emplace_back(insn.size);
-  m_instruction_classes.emplace_back(insn.iclass);
-}
-
-void DecodedThread::AppendInstruction(const pt_insn &insn, uint64_t tsc) {
-  AppendInstruction(insn);
-  RecordTscForLastInstruction(tsc);
+void DecodedThread::Append(const DecodedInstruction &insn) {
+  if (insn.libipt_error < 0) {
+    // End of stream shouldn't be a public error
+    if (insn.libipt_error == -pte_eos)
+      return;
+
+    AppendError(make_error<IntelPTError>(insn.libipt_error, insn.pt_insn.ip));
+  } else {
+    m_instruction_ips.emplace_back(insn.pt_insn.ip);
+    m_instruction_sizes.emplace_back(insn.pt_insn.size);
+    m_instruction_classes.emplace_back(insn.pt_insn.iclass);
+  }
+  if (insn.tsc)
+    RecordTscForLastInstruction(*insn.tsc);
+  if (insn.events)
+    m_events.try_emplace(m_instruction_ips.size() - 1, insn.events);
 }
 
 void DecodedThread::AppendError(llvm::Error &&error) {
   m_errors.try_emplace(m_instruction_ips.size(), toString(std::move(error)));
   m_instruction_ips.emplace_back(LLDB_INVALID_ADDRESS);
   m_instruction_sizes.emplace_back(0);
   m_instruction_classes.emplace_back(pt_insn_class::ptic_error);
 }
 
-void DecodedThread::AppendError(llvm::Error &&error, uint64_t tsc) {
+void DecodedThread::SetAsFailed(llvm::Error &&error) {
   AppendError(std::move(error));
-  RecordTscForLastInstruction(tsc);
+}
+
+lldb::TraceEvents DecodedThread::GetEvents(int insn_index) {
+  auto it = m_events.find(insn_index);
+  if (it != m_events.end())
+    return it->second;
+  return (TraceEvents)0;
 }
 
 void DecodedThread::LibiptErrors::RecordError(int libipt_error_code) {
   libipt_errors[pt_errstr(pt_errcode(libipt_error_code))]++;
   total_count++;
 }
 
 void DecodedThread::RecordTscError(int libipt_error_code) {

 lldb::TraceCursorUP DecodedThread::GetCursor() {
   // We insert a fake error signaling an empty trace if needed becasue the
   // TraceCursor requires non-empty traces.
   if (m_instruction_ips.empty())
     AppendError(createStringError(inconvertibleErrorCode(), "empty trace"));
   return std::make_unique<TraceCursorIntelPT>(m_thread_sp, shared_from_this());
 }
 
+size_t DecodedThread::GetInstructionsWithEventsCount() const {
+  return m_events.size();
+}
+
 size_t DecodedThread::CalculateApproximateMemoryUsage() const {
   return sizeof(pt_insn::ip) * m_instruction_ips.size() +
          sizeof(pt_insn::size) * m_instruction_sizes.size() +
          sizeof(pt_insn::iclass) * m_instruction_classes.size() +
          (sizeof(size_t) + sizeof(uint64_t)) * m_instruction_timestamps.size() +
-         m_errors.getMemorySize();
+         m_errors.getMemorySize() + m_events.getMemorySize();
 }
 
 DecodedThread::TscRange::TscRange(std::map<size_t, uint64_t>::const_iterator it,
                                   const DecodedThread &decoded_thread)
     : m_it(it), m_decoded_thread(&decoded_thread) {
   auto next_it = m_it;
   ++next_it;
   m_end_index = (next_it == m_decoded_thread->m_instruction_timestamps.end())

lldb/source/Plugins/Trace/intel-pt/LibiptDecoder.cpp

 // or we just don't yet.
 struct TscInfo {
   uint64_t tsc = 0;
   LazyBool has_tsc = eLazyBoolCalculate;
 
   explicit operator bool() const { return has_tsc == eLazyBoolYes; }
 };
 
-static inline bool IsLibiptError(int libipt_status) {
-  return libipt_status < 0;
-}
-
-static inline bool IsEndOfStream(int libipt_status) {
-  return libipt_status == -pte_eos;
-}
-
-static inline bool IsTscUnavailable(int libipt_status) {
-  return libipt_status == -pte_no_time;
-}
-
 /// Class that decodes a raw buffer for a single thread using the low level
 /// libipt library.
 ///
 /// Throughout this code, the status of the decoder will be used to identify
 /// events needed to be processed or errors in the decoder. The values can be
 /// - negative: actual errors
 /// - positive or zero: not an error, but a list of bits signaling the status
 /// of the decoder, e.g. whether there are events that need to be decoded or
while (!IsLibiptError(status = FindNextSynchronizationPoint())) {
       // We have synchronized, so we can start decoding instructions and
       // events.
       // Multiple loops indicate gaps in the trace.
       DecodeInstructionsAndEvents(status);
     }
   }
 
 private:
+  /// Invoke the low level function \a pt_insn_next and store the decoded
+  /// instruction in the given \a DecodedInstruction.
+  ///
+  /// \return
+  ///   The status returned by pt_insn_next.
+  int DecodeNextInstruction(DecodedInstruction &insn) {
+    return pt_insn_next(&m_decoder, &insn.pt_insn, sizeof(insn.pt_insn));
+  }
+
   /// Decode all the instructions and events until an error is found or the end
   /// of the trace is reached.
   ///
   /// \param[in] status
   ///   The status that was result of synchronizing to the most recent PSB.
   void DecodeInstructionsAndEvents(int status) {
-    pt_insn insn;
-    while (ProcessPTEvents(status)) {
-      status = pt_insn_next(&m_decoder, &insn, sizeof(insn));
-      // The status returned by pt_insn_next will need to be processed by
-      // ProcessPTEvents in the next loop.
-      if (FoundErrors(status, insn.ip))
+    while (DecodedInstruction insn = ProcessPTEvents(status)) {
+      // The status returned by DecodeNextInstruction will need to be processed
+      // by ProcessPTEvents in the next loop if it is not an error.
+      if (IsLibiptError(status = DecodeNextInstruction(insn))) {
+        insn.libipt_error = status;
+        m_decoded_thread.Append(insn);
         break;
-      AppendInstruction(insn);
+      }
+      m_decoded_thread.Append(insn);
     }
   }
 
   /// Move the decoder forward to the next synchronization point (i.e. next PSB
   /// packet).
   ///
   /// Once the decoder is at that synchronization point, it can start decoding
   /// instructions.
   ///
+  /// If errors are found, they will be appended to the trace.
+  ///
   /// \return
   ///   The libipt decoder status after moving to the next PSB. Negative if
   ///   no PSB was found.
   int FindNextSynchronizationPoint() {
     // Try to sync the decoder. If it fails, then get the decoder_offset and
     // try to sync again from the next synchronization point. If the
     // new_decoder_offset is same as decoder_offset then we can't move to the
     // next synchronization point. Otherwise, keep resyncing until either end
if (!IsEndOfStream(status) && IsLibiptError(status)) {
           }
           // We'll try again starting from a new offset.
           decoder_offset = new_decoder_offset;
         }
       }
     }
 
     // We make this call to record any synchronization errors.
-    FoundErrors(status);
+    if (IsLibiptError(status))
+      m_decoded_thread.Append(DecodedInstruction(status));
+
     return status;
   }
 
   /// Before querying instructions, we need to query the events associated that
   /// instruction e.g. timing events like ptev_tick, or paging events like
   /// ptev_paging.
   ///
+  /// If an error is found, it will be appended to the trace.
+  ///
+  /// \param[in] status
+  ///   The status gotten from the previous instruction decoding or PSB
+  ///   synchronization.
+  ///
   /// \return
-  ///   \b true if we could process the events, \b false if errors were found.
-  bool ProcessPTEvents(int status) {
+  ///   A \a DecodedInstruction with event, tsc and error information.
+  DecodedInstruction ProcessPTEvents(int status) {
+    DecodedInstruction insn;
     while (status & pts_event_pending) {
       pt_event event;
       status = pt_insn_event(&m_decoder, &event, sizeof(event));
-      if (IsLibiptError(status))
+      if (IsLibiptError(status)) {
+        insn.libipt_error = status;
+        break;
+      }
+
+      switch (event.type) {
+      case ptev_enabled:
+        // The kernel started or resumed tracing the program.
+        break;
+      case ptev_disabled:
+        // The CPU paused tracing the program, e.g. due to ip filtering.
+      case ptev_async_disabled:
+        // The kernel or user code paused tracing the program, e.g.
+        // a breakpoint or a ioctl invocation pausing the trace, or a
+        // context switch happened.
+
+        if (m_decoded_thread.GetInstructionsCount() > 0) {
+          // A paused event before the first instruction can be safely
+          // discarded.
+          insn.events = (TraceEvents)(insn.events | eTraceEventPaused);
+        }
+        break;
+      case ptev_overflow:
+        // The CPU internal buffer had an overflow error and some instructions
+        // were lost.
+        insn.libipt_error = -pte_overflow;
         break;
+      default:
+        break;
+      }
     }
 
     // We refresh the TSC that might have changed after processing the events.
     // See
     // https://github.com/intel/libipt/blob/master/doc/man/pt_evt_next.3.md
     RefreshTscInfo();
-    return !FoundErrors(status);
+    if (m_tsc_info)
+      insn.tsc = m_tsc_info.tsc;
+    if (!insn)
+      m_decoded_thread.Append(insn);
+    return insn;
   }
 
   /// Query the decoder for the most recent TSC timestamp and update
   /// the inner tsc information accordingly.
   void RefreshTscInfo() {
     if (m_tsc_info.has_tsc == eLazyBoolNo)
       return;
 
if (IsLibiptError(tsc_status = pt_insn_time(&m_decoder, &new_tsc, nullptr,
         m_decoded_thread.RecordTscError(tsc_status);
       }
     } else {
       m_tsc_info.tsc = new_tsc;
       m_tsc_info.has_tsc = eLazyBoolYes;
     }
   }
 
-  /// Check if the given libipt status signals any errors. If errors were found,
-  /// they will be recorded in the decoded trace.
-  ///
-  /// \param[in] ip
-  ///     An optional ip address can be passed if the error is associated with
-  ///     the decoding of a specific instruction.
-  ///
-  /// \return
-  ///     \b true if errors were found, \b false otherwise.
-  bool FoundErrors(int status, lldb::addr_t ip = LLDB_INVALID_ADDRESS) {
-    if (!IsLibiptError(status))
-      return false;
-
-    // We signal a gap only if it's not "end of stream", as that's not a proper
-    // error.
-    if (!IsEndOfStream(status)) {
-      if (m_tsc_info) {
-        m_decoded_thread.AppendError(make_error<IntelPTError>(status, ip),
-                                     m_tsc_info.tsc);
-      } else {
-        m_decoded_thread.AppendError(make_error<IntelPTError>(status, ip));
-      }
-    }
-    return true;
-  }
-
-  void AppendInstruction(const pt_insn &insn) {
-    if (m_tsc_info)
-      m_decoded_thread.AppendInstruction(insn, m_tsc_info.tsc);
-    else
-      m_decoded_thread.AppendInstruction(insn);
-  }
-
 private:
   pt_insn_decoder &m_decoder;
   DecodedThread &m_decoded_thread;
   TscInfo m_tsc_info;
 };
 
 /// Callback used by libipt for reading the process memory.
 ///

 }
 
 void lldb_private::trace_intel_pt::DecodeTrace(DecodedThread &decoded_thread,
                                                TraceIntelPT &trace_intel_pt,
                                                ArrayRef<uint8_t> buffer) {
   Expected<PtInsnDecoderUP> decoder_up =
       CreateInstructionDecoder(decoded_thread, trace_intel_pt, buffer);
   if (!decoder_up)
-    return decoded_thread.AppendError(decoder_up.takeError());
+    return decoded_thread.SetAsFailed(decoder_up.takeError());
 
   LibiptDecoder libipt_decoder(*decoder_up.get(), decoded_thread);
   libipt_decoder.DecodeUntilEndOfTrace();
 }

lldb/source/Plugins/Trace/intel-pt/ThreadDecoder.cpp

return m_trace.GetTimer()
 
         Error err = m_trace.OnThreadBufferRead(
             m_thread_sp->GetID(), [&](llvm::ArrayRef<uint8_t> data) {
               DecodeTrace(*decoded_thread_sp, m_trace, data);
               return Error::success();
             });
 
         if (err)
-          decoded_thread_sp->AppendError(std::move(err));
+          decoded_thread_sp->SetAsFailed(std::move(err));
         return decoded_thread_sp;
       });
 }

lldb/source/Plugins/Trace/intel-pt/TraceCursorIntelPT.h

public:
   virtual bool Next() override;
 
   const char *GetError() override;
 
   lldb::addr_t GetLoadAddress() override;
 
   llvm::Optional<uint64_t> GetCounter(lldb::TraceCounter counter_type) override;
 
+  lldb::TraceEvents GetEvents() override;
+
   lldb::TraceInstructionControlFlowType
   GetInstructionControlFlowType() override;
 
   bool IsError() override;
 
   bool GoToId(lldb::user_id_t id) override;
 
   lldb::user_id_t GetId() const override;

lldb/source/Plugins/Trace/intel-pt/TraceCursorIntelPT.cpp

TraceCursorIntelPT::GetCounter(lldb::TraceCounter counter_type) {
   case lldb::eTraceCounterTSC:
     if (m_tsc_range)
       return m_tsc_range->GetTsc();
     else
       return llvm::None;
   }
 }
 
+lldb::TraceEvents TraceCursorIntelPT::GetEvents() {
+  return m_decoded_thread_sp->GetEvents(m_pos);
+}
+
 TraceInstructionControlFlowType
 TraceCursorIntelPT::GetInstructionControlFlowType() {
   return m_decoded_thread_sp->GetInstructionControlFlowType(m_pos);
 }
 
 bool TraceCursorIntelPT::GoToId(user_id_t id) {
   if (m_decoded_thread_sp->GetInstructionsCount() <= id)
     return false;
   m_pos = id;
   m_tsc_range = m_decoded_thread_sp->CalculateTscRange(m_pos, m_tsc_range);
 
   return true;
 }
 
 user_id_t TraceCursorIntelPT::GetId() const { return m_pos; }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

s.Format("    Average memory usage per instruction (excluding raw trace): "
              (double)mem_used / insn_len);
 
   s << "\n  Timing:\n";
   GetTimer().ForThread(tid).ForEachTimedTask(
       [&](const std::string &name, std::chrono::milliseconds duration) {
         s.Format("    {0}: {1:2}s\n", name, duration.count() / 1000.0);
       });
 
+  s << "\n  Events:\n";
+  s.Format("    Total number of instructions with events: {0}\n", decoded_trace_sp->GetInstructionsWithEventsCount());
+
   s << "\n  Errors:\n";
   const DecodedThread::LibiptErrors &tsc_errors =
       decoded_trace_sp->GetTscErrors();
   s.Format("    Number of TSC decoding errors: {0}\n", tsc_errors.total_count);
   for (const auto &error_message_to_count : tsc_errors.libipt_errors) {
     s.Format("      {0}: {1}\n", error_message_to_count.first,
              error_message_to_count.second);
   }

lldb/source/Target/TraceInstructionDumper.cpp

insn.instruction->Dump(&s, /*max_opcode_byte_size=*/0, /*show_address=*/false,
                          /*disassembly_addr_format=*/nullptr,
                          /*max_address_text_size=*/0);
 }
 
 void TraceInstructionDumper::SetNoMoreData() { m_no_more_data = true; }
 
 bool TraceInstructionDumper::HasMoreData() { return !m_no_more_data; }
 
+void TraceInstructionDumper::PrintEvents() {
+  if (!m_options.show_events)
+    return;
+
+  TraceEvents events = m_cursor_up->GetEvents();
+  if (events & lldb::eTraceEventPaused) {
+    m_s << "  [paused]\n";
+  }
+}
+
 Optional<lldb::tid_t> TraceInstructionDumper::DumpInstructions(size_t count) {
   ThreadSP thread_sp = m_cursor_up->GetExecutionContextRef().GetThreadSP();
   if (!thread_sp) {
     m_s.Printf("invalid thread");
     return None;
   }
 
   bool was_prev_instruction_an_error = false;
Optional<lldb::tid_t> TraceInstructionDumper::DumpInstructions(size_t count) {
 
   Optional<lldb::user_id_t> last_id;
   for (size_t i = 0; i < count; i++) {
     if (!HasMoreData()) {
       m_s.Printf("    no more data\n");
       break;
     }
     last_id = m_cursor_up->GetId();
+    if (m_options.forwards) {
+      // When moving forwards, we first print the event before printing
+      // the actual instruction.
+      PrintEvents();
+    }
 
     if (const char *err = m_cursor_up->GetError()) {
       if (!m_cursor_up->IsForwards() && !was_prev_instruction_an_error)
         printMissingInstructionsMessage();
 
       was_prev_instruction_an_error = true;
 
       printInstructionHeader(m_cursor_up->GetId());
if (const char *err = m_cursor_up->GetError()) {
 
       if (!m_options.raw)
         DumpInstructionDisassembly(m_s, insn_info);
 
       prev_insn_info = insn_info;
     }
 
     m_s.Printf("\n");
+
+    if (!m_options.forwards) {
+      // If we move backwards, we print the events after printing
+      // the actual instruction so that reading chronologically
+      // makes sense.
+      PrintEvents();
+    }
     TryMoveOneStep();
   }
   return last_id;
 }

lldb/test/API/commands/trace/TestTraceDumpInfo.py

def testDumpRawTraceSize(self):
         self.expect("thread trace dump info",
             substrs=['''Trace technology: intel-pt
 
 thread #1: tid = 3842849
   Total number of instructions: 21
 
   Memory usage:
     Raw trace size: 4 KiB
-    Total approximate memory usage (excluding raw trace): 0.27 KiB
-    Average memory usage per instruction (excluding raw trace): 13.00 bytes
+    Total approximate memory usage (excluding raw trace): 1.27 KiB
+    Average memory usage per instruction (excluding raw trace): 61.76 bytes
 
   Timing:
     Decoding instructions: ''', '''s
 
+  Events:
+    Total number of instructions with events: 1
+
   Errors:
     Number of TSC decoding errors: 0'''],
             patterns=["Decoding instructions: \d.\d\ds"])

lldb/test/API/commands/trace/TestTraceEvents.py

+import lldb
+from intelpt_testcase import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+from lldbsuite.test.decorators import *
+
+class TestTraceEvents(TraceIntelPTTestCaseBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    @testSBAPIAndCommands
+    def testPauseEvents(self):
+      '''
+        Everytime the target stops running on the CPU, a 'disabled' event will
+        be emitted, which is represented by the TraceCursor API as a 'paused'
+        event.
+      '''
+      self.expect("target create " +
+            os.path.join(self.getSourceDir(), "intelpt-trace-multi-file", "a.out"))
+      self.expect("b 12")
+      self.expect("r")
+      self.traceStartThread()
+      self.expect("n")
+      self.expect("n")
+      self.expect("si")
+      self.expect("si")
+      self.expect("si")
+      # We ensure that the paused events are printed correctly forward
+      self.expect("thread trace dump instructions -e -f",
+        patterns=[f'''thread #1: tid = .*
+  a.out`main \+ 23 at main.cpp:12
+    0: {ADDRESS_REGEX}    movl .*
+  \[paused\]
+    1: {ADDRESS_REGEX}    addl .*
+    2: {ADDRESS_REGEX}    movl .*
+  \[paused\]
+  a.out`main \+ 34 \[inlined\] inline_function\(\) at main.cpp:4
+    3: {ADDRESS_REGEX}    movl .*
+  a.out`main \+ 41 \[inlined\] inline_function\(\) \+ 7 at main.cpp:5
+    4: {ADDRESS_REGEX}    movl .*
+    5: {ADDRESS_REGEX}    addl .*
+    6: {ADDRESS_REGEX}    movl .*
+  a.out`main \+ 52 \[inlined\] inline_function\(\) \+ 18 at main.cpp:6
+    7: {ADDRESS_REGEX}    movl .*
+  a.out`main \+ 55 at main.cpp:14
+    8: {ADDRESS_REGEX}    movl .*
+    9: {ADDRESS_REGEX}    addl .*
+    10: {ADDRESS_REGEX}    movl .*
+  \[paused\]
+  a.out`main \+ 63 at main.cpp:16
+    11: {ADDRESS_REGEX}    callq  .* ; symbol stub for: foo\(\)
+  \[paused\]
+  a.out`symbol stub for: foo\(\)
+    12: {ADDRESS_REGEX}    jmpq'''])
+
+      # We ensure that the paused events are printed correctly backward
+      self.expect("thread trace dump instructions -e --id 12",
+        patterns=[f'''thread #1: tid = .*
+  a.out`symbol stub for: foo\(\)
+    12: {ADDRESS_REGEX}    jmpq .*
+  \[paused\]
+  a.out`main \+ 63 at main.cpp:16
+    11: {ADDRESS_REGEX}    callq  .* ; symbol stub for: foo\(\)
+  \[paused\]
+  a.out`main \+ 60 at main.cpp:14
+    10: {ADDRESS_REGEX}    movl .*
+    9: {ADDRESS_REGEX}    addl .*
+    8: {ADDRESS_REGEX}    movl .*
+  a.out`main \+ 52 \[inlined\] inline_function\(\) \+ 18 at main.cpp:6
+    7: {ADDRESS_REGEX}    movl .*
+  a.out`main \+ 49 \[inlined\] inline_function\(\) \+ 15 at main.cpp:5
+    6: {ADDRESS_REGEX}    movl .*
+    5: {ADDRESS_REGEX}    addl .*
+    4: {ADDRESS_REGEX}    movl .*
+  a.out`main \+ 34 \[inlined\] inline_function\(\) at main.cpp:4
+    3: {ADDRESS_REGEX}    movl .*
+  \[paused\]
+  a.out`main \+ 31 at main.cpp:12
+    2: {ADDRESS_REGEX}    movl .*
+    1: {ADDRESS_REGEX}    addl .*
+  \[paused\]
+    0: {ADDRESS_REGEX}    movl .*'''])

lldb/test/API/commands/trace/TestTraceLoad.py

def testLoadTrace(self):
         self.expect("thread list", substrs=["Process 1234 stopped", "tid = 3842849"])
         self.expect("thread trace dump info", substrs=['''Trace technology: intel-pt
 
 thread #1: tid = 3842849
   Total number of instructions: 21
 
   Memory usage:
     Raw trace size: 4 KiB
-    Total approximate memory usage (excluding raw trace): 0.27 KiB
-    Average memory usage per instruction (excluding raw trace): 13.00 bytes
+    Total approximate memory usage (excluding raw trace): 1.27 KiB
+    Average memory usage per instruction (excluding raw trace): 61.76 bytes
 
   Timing:
     Decoding instructions: ''', '''s
 
+  Events:
+    Total number of instructions with events: 1
+
   Errors:
     Number of TSC decoding errors: 0'''])
 
     def testLoadInvalidTraces(self):
         src_dir = self.getSourceDir()
         # We test first an invalid type
         self.expect("trace load -v " + os.path.join(src_dir, "intelpt-trace", "trace_bad.json"), error=True,
           substrs=['''error: expected object at traceSession.processes[0]