[trace][intel pt] Simple detection of infinite decoding loops

[wallace]

Summary

The low-level decoder might fall into an infinite decoding loop for
various reasons, the simplest being an infinite direct loop reached due
to wrong handling of self-modified code in the kernel, e.g. it might
reach

0x0A: pause
0x0C: jump to 0x0A

In this case, all the code is sequential and requires no packets to be
decoded. The low-level decoder would produce an output like the
following

0x0A: pause
0x0C: jump to 0x0A
0x0A: pause
0x0C: jump to 0x0A
0x0A: pause
0x0C: jump to 0x0A
... infinite amount of times

These cases require stopping the decoder to avoid infinite work and signal this
at least as a trace error.

• Add a check that breaks decoding of a single PSB once 500k instructions have been decoded since the last packet was processed.
• Add a check that looks for infinite loops after certain amount of instructions have been decoded since the last packet was processed.
• Add some settings properties for tweaking the thresholds of the checks above. This is also nice because it does the basic work needed for future settings.
• Add an AnomalyDetector class that inspects the DecodedThread and the libipt decoder in search for anomalies. These anomalies are then signaled as fatal errors in the trace.
• Add an ErrorStats class that keeps track of all the errors in a DecodedThread, with a special counter for fatal errors.
• Add an entry for decoded thread errors in the dump info command.

Some notes are added in the code and in the documention of the settings,
so please read them.

Besides that, I haven't been unable to create a test case in LLVM style, but
I've found an anomaly in the thread #12 of the trace
72533820-3eb8-4465-b8e4-4e6bf0ccca99 at Meta. We have to figure out how to
artificially create traces with this kind of anomalies in LLVM style.

With this change, that anomalous thread now shows:

(lldb)thread trace dump instructions 12 -e -i 23101

thread #12: tid = 8
    ...missing instructions
    23101: (error) anomalous trace: possible infinite loop detected of size 2
  vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 5 [inlined] rep_nop at processor.h:13:2
    23100: 0xffffffff81342785    pause
  vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 7 at panic.c:87:2
    23099: 0xffffffff81342787    jmp    0xffffffff81342785        ; <+5> [inlined] rep_nop at processor.h:13:2
  vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 5 [inlined] rep_nop at processor.h:13:2
    23098: 0xffffffff81342785    pause
  vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 7 at panic.c:87:2
    23097: 0xffffffff81342787    jmp    0xffffffff81342785        ; <+5> [inlined] rep_nop at processor.h:13:2
  vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 5 [inlined] rep_nop at processor.h:13:2
    23096: 0xffffffff81342785    pause
  vmlinux-5.12.0-0_fbk8_clang_6656_gc85768aa64da`panic_smp_self_stop + 7 at panic.c:87:2
    23095: 0xffffffff81342787    jmp    0xffffffff81342785        ; <+5> [inlined] rep_nop at processor.h:13:2

It used to be in an infinite loop where the decoder never stopped.

Besides that, the dump info command shows

(lldb) thread trace dump info 12

 Errors:
    Number of individual errors: 32
      Number of fatal errors: 1
      Number of other errors: 31

and in json format

(lldb) thread trace dump info 12 -j

 "errors": {
      "totalCount": 32,
      "libiptErrors": {},
      "fatalErrors": 1,
      "otherErrors": 31
    }

[wallace]

.

[jj10306]

looks good overall, mainly some questions and a few nits

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

123–129

135

nice, I was about to add this as part of my diff (:

lldb/source/Plugins/Trace/intel-pt/LibiptDecoder.cpp

206

can you explain why we are increasing the threshold?

235–241

if you move the --item_index in prev_insn_index lambda, would that allow you to remove this duplicated eTraceItemKindInstruction check and instead unconditionally call prev_insn_index?
or would this not work because the intention of the lamda is to skip the current event even if it's already an instruction

281

help me understand this please. I thought pt_insn_get_offset would always return a new, increasing offset every time this function is called.

[wallace]

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

123–129

ahh good one

135

oh nice!

lldb/source/Plugins/Trace/intel-pt/LibiptDecoder.cpp

206

the idea is to check for infinite loops sporadically without making the total checks in O(N^2) and instead do it in O(N)

If we first do a linear check in the trace, which is O(T) after T instructions are appended and there are no loops, we might want to check again in the future. We could wait for the next T instructions and then run another check, and if we fail, wait for the next T and so on. This result in a total time spent of O(T + 2T + 3T + 4T + ... + N) which is O(N^2). Instead, we can run the check after 2T, and then after 4T and then after 8T and so on. This gives us a geometric progression of (N + N/2 + N / 4 + ... + T) which is amortized total O(N). A similar algorithm is vector::push_back (https://cs.stackexchange.com/questions/9380/why-is-push-back-in-c-vectors-constant-amortized) which is total O(N) using a similar approach.

235–241

I like your idea. I think I can simplify the code

281

Not really. pt_insn_get_offset returns the offset of the last packet that was processed, and that single could lead to many individual sequential instructions until the next packet is needed.

Let's imagine that you have this trace

PSB with starting address of 0xAAA
TNT with 4 bits
TIP with address 0xFFF

What the decoder will do is to first read the PSB and start at IP 0xAAA. It'll then decode sequential instructions until it reaches the first branch or jump. It then needs to read the next packet, which is the TNT with 4 bits, so it will help decode the next 4 branches but not the fifth one. So the decoder will change the offset and resume decoding instructions sequentially until that fifth branch (or jump) is reached. Then the decoder will read the next packet, which is a TIP and tells the decoder to jump to address 0xFFF.
So this means that with the PSB, the decoder produced, let's say, 10 instructions, and with the TNT maybe 1000 were produced,, and then the decoder moved to the offset of the TIP for the next instruction.

[wallace]

address issues and comments

lldb/include/lldb/Core/PluginManager.h

public:
   static SymbolVendorCreateInstance
   GetSymbolVendorCreateCallbackAtIndex(uint32_t idx);
 
   // Trace
   static bool RegisterPlugin(
       llvm::StringRef name, llvm::StringRef description,
       TraceCreateInstanceFromBundle create_callback_from_bundle,
       TraceCreateInstanceForLiveProcess create_callback_for_live_process,
-      llvm::StringRef schema);
+      llvm::StringRef schema,
+      DebuggerInitializeCallback debugger_init_callback);
 
   static bool
   UnregisterPlugin(TraceCreateInstanceFromBundle create_callback);
 
   static TraceCreateInstanceFromBundle
   GetTraceCreateCallback(llvm::StringRef plugin_name);
 
   static TraceCreateInstanceForLiveProcess
public:
 
   static lldb::OptionValuePropertiesSP
   GetSettingForProcessPlugin(Debugger &debugger, ConstString setting_name);
 
   static bool CreateSettingForProcessPlugin(
       Debugger &debugger, const lldb::OptionValuePropertiesSP &properties_sp,
       ConstString description, bool is_global_property);
 
+  static bool CreateSettingForTracePlugin(
+      Debugger &debugger, const lldb::OptionValuePropertiesSP &properties_sp,
+      ConstString description, bool is_global_property);
+
   static lldb::OptionValuePropertiesSP
   GetSettingForObjectFilePlugin(Debugger &debugger, ConstString setting_name);
 
   static bool CreateSettingForObjectFilePlugin(
       Debugger &debugger, const lldb::OptionValuePropertiesSP &properties_sp,
       ConstString description, bool is_global_property);
 
   static lldb::OptionValuePropertiesSP

lldb/source/Core/PluginManager.cpp

 #pragma mark Trace
 
 struct TraceInstance
     : public PluginInstance<TraceCreateInstanceFromBundle> {
   TraceInstance(
       llvm::StringRef name, llvm::StringRef description,
       CallbackType create_callback_from_bundle,
       TraceCreateInstanceForLiveProcess create_callback_for_live_process,
-      llvm::StringRef schema)
+      llvm::StringRef schema, DebuggerInitializeCallback debugger_init_callback)
       : PluginInstance<TraceCreateInstanceFromBundle>(
-            name, description, create_callback_from_bundle),
+            name, description, create_callback_from_bundle,
+            debugger_init_callback),
         schema(schema),
         create_callback_for_live_process(create_callback_for_live_process) {}
 
   llvm::StringRef schema;
   TraceCreateInstanceForLiveProcess create_callback_for_live_process;
 };
 
 typedef PluginInstances<TraceInstance> TraceInstances;
 
 static TraceInstances &GetTracePluginInstances() {
   static TraceInstances g_instances;
   return g_instances;
 }
 
 bool PluginManager::RegisterPlugin(
     llvm::StringRef name, llvm::StringRef description,
     TraceCreateInstanceFromBundle create_callback_from_bundle,
     TraceCreateInstanceForLiveProcess create_callback_for_live_process,
-    llvm::StringRef schema) {
+    llvm::StringRef schema, DebuggerInitializeCallback debugger_init_callback) {
   return GetTracePluginInstances().RegisterPlugin(
       name, description, create_callback_from_bundle,
-      create_callback_for_live_process, schema);
+      create_callback_for_live_process, schema, debugger_init_callback);
 }
 
 bool PluginManager::UnregisterPlugin(
     TraceCreateInstanceFromBundle create_callback_from_bundle) {
   return GetTracePluginInstances().UnregisterPlugin(
       create_callback_from_bundle);
 }
 
if (properties_sp) {
     }
   }
   return false;
 }
 
 static const char *kDynamicLoaderPluginName("dynamic-loader");
 static const char *kPlatformPluginName("platform");
 static const char *kProcessPluginName("process");
+static const char *kTracePluginName("trace");
 static const char *kObjectFilePluginName("object-file");
 static const char *kSymbolFilePluginName("symbol-file");
 static const char *kJITLoaderPluginName("jit-loader");
 static const char *kStructuredDataPluginName("structured-data");
 
 lldb::OptionValuePropertiesSP
 PluginManager::GetSettingForDynamicLoaderPlugin(Debugger &debugger,
                                                 ConstString setting_name) {

 bool PluginManager::CreateSettingForProcessPlugin(
     Debugger &debugger, const lldb::OptionValuePropertiesSP &properties_sp,
     ConstString description, bool is_global_property) {
   return CreateSettingForPlugin(debugger, ConstString(kProcessPluginName),
                                 ConstString("Settings for process plug-ins"),
                                 properties_sp, description, is_global_property);
 }
 
+bool PluginManager::CreateSettingForTracePlugin(
+    Debugger &debugger, const lldb::OptionValuePropertiesSP &properties_sp,
+    ConstString description, bool is_global_property) {
+  return CreateSettingForPlugin(debugger, ConstString(kTracePluginName),
+                                ConstString("Settings for trace plug-ins"),
+                                properties_sp, description, is_global_property);
+}
+
 lldb::OptionValuePropertiesSP
 PluginManager::GetSettingForObjectFilePlugin(Debugger &debugger,
                                              ConstString setting_name) {
   return GetSettingForPlugin(debugger, setting_name,
                              ConstString(kObjectFilePluginName));
 }
 
 bool PluginManager::CreateSettingForObjectFilePlugin(

lldb/source/Plugins/Trace/intel-pt/CMakeLists.txt

 if (NOT LIBIPT_INCLUDE_PATH)
   message (FATAL_ERROR "libipt include path not provided")
 endif()
 
 if (NOT EXISTS "${LIBIPT_INCLUDE_PATH}")
   message (FATAL_ERROR "invalid libipt include path provided")
 endif()
 include_directories(${LIBIPT_INCLUDE_PATH})
 
 find_library(LIBIPT_LIBRARY ipt PATHS ${LIBIPT_LIBRARY_PATH} REQUIRED)
 
 lldb_tablegen(TraceIntelPTCommandOptions.inc -gen-lldb-option-defs
   SOURCE TraceIntelPTOptions.td
   TARGET TraceIntelPTOptionsGen)
 
+lldb_tablegen(TraceIntelPTProperties.inc -gen-lldb-property-defs
+  SOURCE TraceIntelPTProperties.td
+  TARGET TraceIntelPTPropertiesGen)
+
+lldb_tablegen(TraceIntelPTPropertiesEnum.inc -gen-lldb-property-enum-defs
+  SOURCE TraceIntelPTProperties.td
+  TARGET TraceIntelPTPropertiesEnumGen)
+
 add_lldb_library(lldbPluginTraceIntelPT PLUGIN
   CommandObjectTraceStartIntelPT.cpp
   DecodedThread.cpp
   TaskTimer.cpp
   LibiptDecoder.cpp
   PerfContextSwitchDecoder.cpp
   ThreadDecoder.cpp
   TraceCursorIntelPT.cpp
LINK_LIBS
     lldbTarget
     lldbPluginTraceCommon
     ${LIBIPT_LIBRARY}
   LINK_COMPONENTS
     Support
   )
 
 
-add_dependencies(lldbPluginTraceIntelPT TraceIntelPTOptionsGen)
+add_dependencies(lldbPluginTraceIntelPT
+  TraceIntelPTOptionsGen
+  TraceIntelPTPropertiesGen
+  TraceIntelPTPropertiesEnumGen)

lldb/source/Plugins/Trace/intel-pt/DecodedThread.h

 /// the trace.
 ///
 /// Each decoded thread contains a cursor to the current position the user is
 /// stopped at. See \a Trace::GetCursorPosition for more information.
 class DecodedThread : public std::enable_shared_from_this<DecodedThread> {
 public:
   using TSC = uint64_t;
 
-  // Struct holding counts for libipts errors;
-  struct LibiptErrorsStats {
-    // libipt error -> count
-    llvm::DenseMap<const char *, int> libipt_errors_counts;
-    size_t total_count = 0;
-
-    void RecordError(int libipt_error_code);
-  };
-
   /// A structure that represents a maximal range of trace items associated to
   /// the same TSC value.
   struct TSCRange {
     TSC tsc;
     /// Number of trace items in this range.
     uint64_t items_count;
     /// Index of the first trace item in this range.
     uint64_t first_item_index;
GetInterpolatedTime(uint64_t item_index, uint64_t beginning_of_time_nanos,
                         const LinuxPerfZeroTscConversion &tsc_conversion) const;
 
     /// \return
     ///   \b true if and only if the given \p item_index is covered by this
     ///   range.
     bool InRange(uint64_t item_index) const;
   };
 
-  // Struct holding counts for events;
+  // Struct holding counts for events
   struct EventsStats {
     /// A count for each individual event kind. We use an unordered map instead
     /// of a DenseMap because DenseMap can't understand enums.
-    std::unordered_map<lldb::TraceEvent, size_t> events_counts;
-    size_t total_count = 0;
+    ///
+    /// Note: We can't use DenseMap because lldb::TraceEvent is not
+    /// automatically handled correctly by DenseMap. We'd need to implement a
+    /// custom DenseMapInfo struct for TraceEvent and that's a bit too much for
+    /// such a simple structure.
+    std::unordered_map<lldb::TraceEvent, uint64_t> events_counts;
+    uint64_t total_count = 0;
 
     void RecordEvent(lldb::TraceEvent event);
   };
 
+  // Struct holding counts for errors
+  struct ErrorStats {
+    /// The following counters are mutually exclusive
+    /// \{
+    uint64_t other_errors = 0;
+    uint64_t fatal_errors = 0;
+    // libipt error -> count
+    llvm::DenseMap<const char *, uint64_t> libipt_errors;
+    /// \}
+
+    uint64_t GetTotalCount() const;
+
+    void RecordError(int libipt_error_code);
+
+    void RecordError(bool fatal);
+  };
+
   DecodedThread(
       lldb::ThreadSP thread_sp,
       const llvm::Optional<LinuxPerfZeroTscConversion> &tsc_conversion);
 
   /// Get the total number of instruction, errors and events from the decoded
   /// trace.
   uint64_t GetItemsCount() const;
 
public:
   ///   The requested nanoseconds range, or \a llvm::None if not available.
   llvm::Optional<DecodedThread::NanosecondsRange>
   GetNanosecondsRangeByIndex(uint64_t item_index);
 
   /// \return
   ///     The load address of the instruction at the given index.
   lldb::addr_t GetInstructionLoadAddress(uint64_t item_index) const;
 
+  /// \return
+  ///     The number of instructions in this trace (not trace items).
+  uint64_t GetTotalInstructionCount() const;
+
   /// Return an object with statistics of the trace events that happened.
   ///
   /// \return
   ///   The stats object of all the events.
   const EventsStats &GetEventsStats() const;
 
+  /// Return an object with statistics of the trace errors that happened.
+  ///
+  /// \return
+  ///   The stats object of all the events.
+  const ErrorStats &GetErrorStats() const;
+
   /// The approximate size in bytes used by this instance,
   /// including all the already decoded instructions.
   size_t CalculateApproximateMemoryUsage() const;
 
   lldb::ThreadSP GetThread();
 
   /// Notify this object that a new tsc has been seen.
   /// If this a new TSC, an event will be created.
   void NotifyTsc(TSC tsc);
 
   /// Notify this object that a CPU has been seen.
   /// If this a new CPU, an event will be created.
   void NotifyCPU(lldb::cpu_id_t cpu_id);
 
   /// Notify this object that a new PSB has been seen.
   void NotifySyncPoint(lldb::addr_t psb_offset);
 
   /// Append a decoding error.
   void AppendError(const IntelPTError &error);
 
   /// Append a custom decoding.
-  void AppendCustomError(llvm::StringRef error);
+  ///
+  /// \param[in] error
+  ///   The error message.
+  ///
+  /// \param[in] fatal
+  ///   If \b true, then the whole decoded thread should be discarded because a
+  ///   fatal anomaly has been found.
+  void AppendCustomError(llvm::StringRef error, bool fatal = false);
 
   /// Append an event.
   void AppendEvent(lldb::TraceEvent);
 
   /// Append an instruction.
   void AppendInstruction(const pt_insn &insn);
 
 private:
private:
   llvm::Optional<uint64_t> m_last_cpu = llvm::None;
 
   // The PSB offsets are stored as a map. It maps `item index -> psb offset`.
   llvm::DenseMap<uint64_t, lldb::addr_t> m_psb_offsets;
 
   /// TSC -> nanos conversion utility.
   llvm::Optional<LinuxPerfZeroTscConversion> m_tsc_conversion;
 
+  /// Statistics of all tracing errors.
+  ErrorStats m_error_stats;
+
   /// Statistics of all tracing events.
   EventsStats m_events_stats;
   /// Total amount of time spent decoding.
   std::chrono::milliseconds m_total_decoding_time{0};
+
+  /// Total number of instructions in the trace.
+  uint64_t m_insn_count = 0;
 };
 
 using DecodedThreadSP = std::shared_ptr<DecodedThread>;
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_DECODEDTHREAD_H

lldb/source/Plugins/Trace/intel-pt/DecodedThread.cpp

 Optional<DecodedThread::NanosecondsRange>
 DecodedThread::GetNanosecondsRangeByIndex(uint64_t item_index) {
   auto next_it = m_nanoseconds.upper_bound(item_index);
   if (next_it == m_nanoseconds.begin())
     return None;
   return prev(next_it)->second;
 }
 
+uint64_t DecodedThread::GetTotalInstructionCount() const {
+  return m_insn_count;
+}
+
 void DecodedThread::AppendEvent(lldb::TraceEvent event) {
   CreateNewTraceItem(lldb::eTraceItemKindEvent).event = event;
   m_events_stats.RecordEvent(event);
 }
 
 void DecodedThread::AppendInstruction(const pt_insn &insn) {
   CreateNewTraceItem(lldb::eTraceItemKindInstruction).load_address = insn.ip;
+  m_insn_count++;
 }
 
 void DecodedThread::AppendError(const IntelPTError &error) {
   CreateNewTraceItem(lldb::eTraceItemKindError).error =
       ConstString(error.message()).AsCString();
+  m_error_stats.RecordError(/*fatal=*/false);
 }
 
-void DecodedThread::AppendCustomError(StringRef err) {
+void DecodedThread::AppendCustomError(StringRef err, bool fatal) {
   CreateNewTraceItem(lldb::eTraceItemKindError).error =
       ConstString(err).AsCString();
+  m_error_stats.RecordError(fatal);
 }
 
 lldb::TraceEvent DecodedThread::GetEventByIndex(int item_index) const {
   return m_item_data[item_index].event;
 }
 
-void DecodedThread::LibiptErrorsStats::RecordError(int libipt_error_code) {
-  libipt_errors_counts[pt_errstr(pt_errcode(libipt_error_code))]++;
-  total_count++;
-}
-
 const DecodedThread::EventsStats &DecodedThread::GetEventsStats() const {
   return m_events_stats;
 }
 
 void DecodedThread::EventsStats::RecordEvent(lldb::TraceEvent event) {
   events_counts[event]++;
   total_count++;
 }
 
+uint64_t DecodedThread::ErrorStats::GetTotalCount() const {
+  uint64_t total = 0;
+  for (const auto &[kind, count] : libipt_errors)
+    total += count;
+
+  return total + other_errors + fatal_errors;
+}
+
+void DecodedThread::ErrorStats::RecordError(bool fatal) {
+  if (fatal)
+    fatal_errors++;
+  else
+    other_errors++;
+}
+
+void DecodedThread::ErrorStats::RecordError(int libipt_error_code) {
+  libipt_errors[pt_errstr(pt_errcode(libipt_error_code))]++;
+}
+
+const DecodedThread::ErrorStats &DecodedThread::GetErrorStats() const {
+  return m_error_stats;
+}
+
 lldb::TraceItemKind
 DecodedThread::GetItemKindByIndex(uint64_t item_index) const {
   return static_cast<lldb::TraceItemKind>(m_item_kinds[item_index]);
 }
 
 const char *DecodedThread::GetErrorByIndex(uint64_t item_index) const {
   return m_item_data[item_index].error;
 }

lldb/source/Plugins/Trace/intel-pt/LibiptDecoder.cpp

CreateQueryDecoder(TraceIntelPT &trace_intel_pt, ArrayRef<uint8_t> buffer) {
 
   pt_query_decoder *decoder_ptr = pt_qry_alloc_decoder(&*config);
   if (!decoder_ptr)
     return make_error<IntelPTError>(-pte_nomem);
 
   return PtQueryDecoderUP(decoder_ptr, QueryDecoderDeleter);
 }
 
+/// Class used to identify anomalies in traces, which should often indicate a
+/// fatal error in the trace.
+class PSBBlockAnomalyDetector {
+public:
+  PSBBlockAnomalyDetector(pt_insn_decoder &decoder,
+                          TraceIntelPT &trace_intel_pt,
+                          DecodedThread &decoded_thread)
+      : m_decoder(decoder), m_decoded_thread(decoded_thread) {
+    m_infinite_decoding_loop_threshold =
+        trace_intel_pt.GetGlobalProperties()
+            .GetInfiniteDecodingLoopVerificationThreshold();
+    m_extremely_large_decoding_threshold =
+        trace_intel_pt.GetGlobalProperties()
+            .GetExtremelyLargeDecodingThreshold();
+    m_next_infinite_decoding_loop_threshold =
+        m_infinite_decoding_loop_threshold;
+  }
+
+  /// \return
+  ///   An \a llvm::Error if an anomaly that includes the last instruction item
+  ///   in the trace, or \a llvm::Error::success otherwise.
+  Error DetectAnomaly() {
+    RefreshPacketOffset();
+    uint64_t insn_added_since_last_packet_offset =
+        m_decoded_thread.GetTotalInstructionCount() -
+        m_insn_count_at_last_packet_offset;
+
+    // We want to check if we might have fallen in an infinite loop. As this
+    // check is not a no-op, we want to do it when we have a strong suggestion
+    // that things went wrong. First, we check how many instructions we have
+    // decoded since we processed an Intel PT packet for the last time. This
+    // number should be low, because at some point we should see branches, jumps
+    // or interrupts that require a new packet to be processed. Once we reach
+    // certain threshold we start analyzing the trace.
+    //
+    // We use the number of decoded instructions since the last Intel PT packet
+    // as a proxy because, in fact, we don't expect a single packet to give,
+    // say, 100k instructions. That would mean that there are 100k sequential
+    // instructions without any single branch, which is highly unlikely, or that
+    // we found an infinite loop using direct jumps, e.g.
+    //
+    //   0x0A: nop or pause
+    //   0x0C: jump to 0x0A
+    //
+    // which is indeed code that is found in the kernel. I presume we reach
+    // this kind of code in the decoder because we don't handle self-modified
+    // code in post-mortem kernel traces.
+    //
+    // We are right now only signaling the anomaly as a trace error, but it
+    // would be more conservative to also discard all the trace items found in
+    // this PSB. I prefer not to do that for the time being to give more
+    // exposure to this kind of anomalies and help debugging. Discarding the
+    // trace items would just make investigation harded.
+    //
+    // Finally, if the user wants to see if a specific thread has an anomaly,
+    // it's enough to run the `thread trace dump info` command and look for the
+    // count of this kind of errors.
+
+    if (insn_added_since_last_packet_offset >=
+        m_extremely_large_decoding_threshold) {
+      // In this case, we have decoded a massive amount of sequential
+      // instructions that don't loop. Honestly I wonder if this will ever
+      // happen, but better safe than sorry.
+      return createStringError(
+          inconvertibleErrorCode(),
+          "anomalous trace: possible infinite trace detected");
+    }
+    if (insn_added_since_last_packet_offset ==
+        m_next_infinite_decoding_loop_threshold) {
+      if (Optional<uint64_t> loop_size = TryIdentifyInfiniteLoop()) {
+        return createStringError(
+            inconvertibleErrorCode(),
+            "anomalous trace: possible infinite loop detected of size %" PRIu64,
+            *loop_size);
+      }
+      m_next_infinite_decoding_loop_threshold *= 2;
+    }
+    return Error::success();
+  }
+
+private:
+  Optional<uint64_t> TryIdentifyInfiniteLoop() {
+    // The infinite decoding loops we'll encounter are due to sequential
+    // instructions that repeat themselves due to direct jumps, therefore in a
+    // cycle each individual address will only appear once. We use this
+    // information to detect cycles by finding the last 2 ocurrences of the last
+    // instruction added to the trace. Then we traverse the trace making sure
+    // that these two instructions where the ends of a repeating loop.
+
+    // This is a utility that returns the most recent instruction index given a
+    // position in the trace. If the given position is an instruction, that
+    // position is returned. It skips non-instruction items.
+    auto most_recent_insn_index =
+        [&](uint64_t item_index) -> Optional<uint64_t> {
+      while (true) {
+        if (m_decoded_thread.GetItemKindByIndex(item_index) ==
+            lldb::eTraceItemKindInstruction) {
+          return item_index;
+        }
+        if (item_index == 0)
+          return None;
+        item_index--;
+      }
+      return None;
+    };
+    // Similar to most_recent_insn_index but skips the starting position.
+    auto prev_insn_index = [&](uint64_t item_index) -> Optional<uint64_t> {
+      if (item_index == 0)
+        return None;
+      return most_recent_insn_index(item_index - 1);
+    };
+
+    // We first find the most recent instruction.
+    Optional<uint64_t> last_insn_index_opt =
+        *prev_insn_index(m_decoded_thread.GetItemsCount());
+    if (!last_insn_index_opt)
+      return None;
+    uint64_t last_insn_index = *last_insn_index_opt;
+
+    // We then find the most recent previous occurrence of that last
+    // instruction.
+    Optional<uint64_t> last_insn_copy_index = prev_insn_index(last_insn_index);
+    uint64_t loop_size = 1;
+    while (last_insn_copy_index &&
+           m_decoded_thread.GetInstructionLoadAddress(*last_insn_copy_index) !=
+               m_decoded_thread.GetInstructionLoadAddress(last_insn_index)) {
+      last_insn_copy_index = prev_insn_index(*last_insn_copy_index);
+      loop_size++;
+    }
+    if (!last_insn_copy_index)
+      return None;
+
+    // Now we check if the segment between these last positions of the last
+    // instruction address is in fact a repeating loop.
+    uint64_t loop_elements_visited = 1;
+    uint64_t insn_index_a = last_insn_index,
+             insn_index_b = *last_insn_copy_index;
+    while (loop_elements_visited < loop_size) {
+      if (Optional<uint64_t> prev = prev_insn_index(insn_index_a))
+        insn_index_a = *prev;
+      else
+        return None;
+      if (Optional<uint64_t> prev = prev_insn_index(insn_index_b))
+        insn_index_b = *prev;
+      else
+        return None;
+      if (m_decoded_thread.GetInstructionLoadAddress(insn_index_a) !=
+          m_decoded_thread.GetInstructionLoadAddress(insn_index_b))
+        return None;
+      loop_elements_visited++;
+    }
+    return loop_size;
+  }
+
+  // Refresh the internal counters if a new packet offset has been visited
+  void RefreshPacketOffset() {
+    lldb::addr_t new_packet_offset;
+    if (!IsLibiptError(pt_insn_get_offset(&m_decoder, &new_packet_offset)) &&
+        new_packet_offset != m_last_packet_offset) {
+      m_last_packet_offset = new_packet_offset;
+      m_next_infinite_decoding_loop_threshold =
+          m_infinite_decoding_loop_threshold;
+      m_insn_count_at_last_packet_offset =
+          m_decoded_thread.GetTotalInstructionCount();
+    }
+  }
+
+  pt_insn_decoder &m_decoder;
+  DecodedThread &m_decoded_thread;
+  lldb::addr_t m_last_packet_offset = LLDB_INVALID_ADDRESS;
+  uint64_t m_insn_count_at_last_packet_offset = 0;
+  uint64_t m_infinite_decoding_loop_threshold;
+  uint64_t m_next_infinite_decoding_loop_threshold;
+  uint64_t m_extremely_large_decoding_threshold;
+};
+
 /// Class that decodes a raw buffer for a single PSB block using the low level
 /// libipt library. It assumes that kernel and user mode instructions are not
 /// mixed in the same PSB block.
 ///
 /// Throughout this code, the status of the decoder will be used to identify
 /// events needed to be processed or errors in the decoder. The values can be
 /// - negative: actual errors
 /// - positive or zero: not an error, but a list of bits signaling the status
public:
   /// \param[in] next_block_ip
   ///     The starting ip at the next PSB block of the same thread if available.
   ///
   /// \param[in] decoded_thread
   ///     A \a DecodedThread object where the decoded instructions will be
   ///     appended to. It might have already some instructions.
   PSBBlockDecoder(PtInsnDecoderUP &&decoder_up, const PSBBlock &psb_block,
                   Optional<lldb::addr_t> next_block_ip,
-                  DecodedThread &decoded_thread)
+                  DecodedThread &decoded_thread, TraceIntelPT &trace_intel_pt)
       : m_decoder_up(std::move(decoder_up)), m_psb_block(psb_block),
-        m_next_block_ip(next_block_ip), m_decoded_thread(decoded_thread) {}
+        m_next_block_ip(next_block_ip), m_decoded_thread(decoded_thread),
+        m_anomaly_detector(*m_decoder_up, trace_intel_pt, decoded_thread) {}
 
   /// \param[in] trace_intel_pt
   ///     The main Trace object that own the PSB block.
   ///
   /// \param[in] decoder
   ///     A decoder configured to start and end within the boundaries of the
   ///     given \p psb_block.
   ///
Create(TraceIntelPT &trace_intel_pt, const PSBBlock &psb_block,
          ArrayRef<uint8_t> buffer, Process &process,
          Optional<lldb::addr_t> next_block_ip, DecodedThread &decoded_thread) {
     Expected<PtInsnDecoderUP> decoder_up =
         CreateInstructionDecoder(trace_intel_pt, buffer, process);
     if (!decoder_up)
       return decoder_up.takeError();
 
     return PSBBlockDecoder(std::move(*decoder_up), psb_block, next_block_ip,
-                           decoded_thread);
+                           decoded_thread, trace_intel_pt);
   }
 
   void DecodePSBBlock() {
     int status = pt_insn_sync_forward(m_decoder_up.get());
     assert(status >= 0 &&
            "Synchronization shouldn't fail because this PSB was previously "
            "decoded correctly.");
 
     // We emit a TSC before a sync event to more easily associate a timestamp to
     // the sync event. If present, the current block's TSC would be the first
     // TSC we'll see when processing events.
     if (m_psb_block.tsc)
       m_decoded_thread.NotifyTsc(*m_psb_block.tsc);
 
     m_decoded_thread.NotifySyncPoint(m_psb_block.psb_offset);
 
     DecodeInstructionsAndEvents(status);
   }
 
 private:
-  /// Decode all the instructions and events of the given PSB block.
-  ///
-  /// \param[in] status
-  ///   The status that was result of synchronizing to the most recent PSB.
+  /// Append an instruction and return \b false if and only if a serious anomaly
+  /// has been detected.
+  bool AppendInstructionAndDetectAnomalies(const pt_insn &insn) {
+    m_decoded_thread.AppendInstruction(insn);
+
+    if (Error err = m_anomaly_detector.DetectAnomaly()) {
+      m_decoded_thread.AppendCustomError(toString(std::move(err)),
+                                         /*fatal=*/true);
+      return false;
+    }
+    return true;
+  }
+  /// Decode all the instructions and events of the given PSB block. The
+  /// decoding loop might stop abruptly if an infinite decoding loop is
+  /// detected.
   void DecodeInstructionsAndEvents(int status) {
     pt_insn insn;
+
     while (true) {
       status = ProcessPTEvents(status);
 
       if (IsLibiptError(status))
         return;
       else if (IsEndOfStream(status))
         break;
 
       // The status returned by pt_insn_next will need to be processed
       // by ProcessPTEvents in the next loop if it is not an error.
       std::memset(&insn, 0, sizeof insn);
       status = pt_insn_next(m_decoder_up.get(), &insn, sizeof(insn));
 
       if (IsLibiptError(status)) {
         m_decoded_thread.AppendError(IntelPTError(status, insn.ip));
         return;
       } else if (IsEndOfStream(status)) {
         break;
       }
-      m_decoded_thread.AppendInstruction(insn);
+
+      if (!AppendInstructionAndDetectAnomalies(insn))
+        return;
     }
 
     // We need to keep querying non-branching instructions until we hit the
     // starting point of the next PSB. We won't see events at this point. This
     // is based on
     // https://github.com/intel/libipt/blob/master/doc/howto_libipt.md#parallel-decode
     if (m_next_block_ip && insn.ip != 0) {
       while (insn.ip != *m_next_block_ip) {
-        m_decoded_thread.AppendInstruction(insn);
+        if (!AppendInstructionAndDetectAnomalies(insn))
+          return;
 
         status = pt_insn_next(m_decoder_up.get(), &insn, sizeof(insn));
 
         if (IsLibiptError(status)) {
           m_decoded_thread.AppendError(IntelPTError(status, insn.ip));
           return;
         }
       }
int ProcessPTEvents(int status) {
     return status;
   }
 
 private:
   PtInsnDecoderUP m_decoder_up;
   PSBBlock m_psb_block;
   Optional<lldb::addr_t> m_next_block_ip;
   DecodedThread &m_decoded_thread;
+  PSBBlockAnomalyDetector m_anomaly_detector;
 };
 
 Error lldb_private::trace_intel_pt::DecodeSingleTraceForThread(
     DecodedThread &decoded_thread, TraceIntelPT &trace_intel_pt,
     ArrayRef<uint8_t> buffer) {
   Expected<std::vector<PSBBlock>> blocks =
       SplitTraceIntoPSBBlock(trace_intel_pt, buffer, /*expect_tscs=*/false);
   if (!blocks)

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

 #include "lldb/lldb-types.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace lldb_private {
 namespace trace_intel_pt {
 
 class TraceIntelPT : public Trace {
 public:
+  /// Properties to be used with the `settings` command.
+  class PluginProperties : public Properties {
+  public:
+    static ConstString GetSettingName();
+
+    PluginProperties();
+
+    ~PluginProperties() override = default;
+
+    uint64_t GetInfiniteDecodingLoopVerificationThreshold();
+
+    uint64_t GetExtremelyLargeDecodingThreshold();
+  };
+
+  /// Return the global properties for this trace plug-in.
+  static PluginProperties &GetGlobalProperties();
+
   void Dump(Stream *s) const override;
 
   llvm::Expected<FileSpec> SaveToDisk(FileSpec directory,
                                       bool compact) override;
 
   ~TraceIntelPT() override = default;
 
   /// PluginInterface protocol
public:
   static llvm::Expected<lldb::TraceSP> CreateInstanceForTraceBundle(
       const llvm::json::Value &trace_bundle_description,
       llvm::StringRef bundle_dir, Debugger &debugger);
 
   static llvm::Expected<lldb::TraceSP>
   CreateInstanceForLiveProcess(Process &process);
 
   static llvm::StringRef GetPluginNameStatic() { return "intel-pt"; }
+
+  static void DebuggerInitialize(Debugger &debugger);
   /// \}
 
   lldb::CommandObjectSP
   GetProcessTraceStartCommand(CommandInterpreter &interpreter) override;
 
   lldb::CommandObjectSP
   GetThreadTraceStartCommand(CommandInterpreter &interpreter) override;
 

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

 #include "../common/ThreadPostMortemTrace.h"
 #include "CommandObjectTraceStartIntelPT.h"
 #include "DecodedThread.h"
 #include "TraceCursorIntelPT.h"
 #include "TraceIntelPTBundleLoader.h"
 #include "TraceIntelPTBundleSaver.h"
 #include "TraceIntelPTConstants.h"
 #include "lldb/Core/PluginManager.h"
+#include "lldb/Interpreter/OptionValueProperties.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Target/Target.h"
 #include "llvm/ADT/None.h"
 
 using namespace lldb;
 using namespace lldb_private;
 using namespace lldb_private::trace_intel_pt;
 using namespace llvm;
 
 LLDB_PLUGIN_DEFINE(TraceIntelPT)
 
 lldb::CommandObjectSP
 TraceIntelPT::GetProcessTraceStartCommand(CommandInterpreter &interpreter) {
   return CommandObjectSP(
       new CommandObjectProcessTraceStartIntelPT(*this, interpreter));
 }
 
 lldb::CommandObjectSP
 TraceIntelPT::GetThreadTraceStartCommand(CommandInterpreter &interpreter) {
   return CommandObjectSP(
       new CommandObjectThreadTraceStartIntelPT(*this, interpreter));
 }
 
+#define LLDB_PROPERTIES_traceintelpt
+#include "TraceIntelPTProperties.inc"
+
+enum {
+#define LLDB_PROPERTIES_traceintelpt
+#include "TraceIntelPTPropertiesEnum.inc"
+};
+
+ConstString TraceIntelPT::PluginProperties::GetSettingName() {
+  return ConstString(TraceIntelPT::GetPluginNameStatic());
+}
+
+TraceIntelPT::PluginProperties::PluginProperties() : Properties() {
+  m_collection_sp = std::make_shared<OptionValueProperties>(GetSettingName());
+  m_collection_sp->Initialize(g_traceintelpt_properties);
+}
+
+uint64_t
+TraceIntelPT::PluginProperties::GetInfiniteDecodingLoopVerificationThreshold() {
+  const uint32_t idx = ePropertyInfiniteDecodingLoopVerificationThreshold;
+  return m_collection_sp->GetPropertyAtIndexAsUInt64(
+      nullptr, idx, g_traceintelpt_properties[idx].default_uint_value);
+}
+
+uint64_t TraceIntelPT::PluginProperties::GetExtremelyLargeDecodingThreshold() {
+  const uint32_t idx = ePropertyExtremelyLargeDecodingThreshold;
+  return m_collection_sp->GetPropertyAtIndexAsUInt64(
+      nullptr, idx, g_traceintelpt_properties[idx].default_uint_value);
+}
+
+TraceIntelPT::PluginProperties &TraceIntelPT::GetGlobalProperties() {
+  static TraceIntelPT::PluginProperties g_settings;
+  return g_settings;
+}
+
 void TraceIntelPT::Initialize() {
-  PluginManager::RegisterPlugin(GetPluginNameStatic(), "Intel Processor Trace",
-                                CreateInstanceForTraceBundle,
-                                CreateInstanceForLiveProcess,
-                                TraceIntelPTBundleLoader::GetSchema());
+  PluginManager::RegisterPlugin(
+      GetPluginNameStatic(), "Intel Processor Trace",
+      CreateInstanceForTraceBundle, CreateInstanceForLiveProcess,
+      TraceIntelPTBundleLoader::GetSchema(), DebuggerInitialize);
+}
+
+void TraceIntelPT::DebuggerInitialize(Debugger &debugger) {
+  if (!PluginManager::GetSettingForProcessPlugin(
+          debugger, PluginProperties::GetSettingName())) {
+    const bool is_global_setting = true;
+    PluginManager::CreateSettingForTracePlugin(
+        debugger, GetGlobalProperties().GetValueProperties(),
+        ConstString("Properties for the intel-pt trace plug-in."),
+        is_global_setting);
+  }
 }
 
 void TraceIntelPT::Terminate() {
   PluginManager::UnregisterPlugin(CreateInstanceForTraceBundle);
 }
 
 StringRef TraceIntelPT::GetSchema() {
   return TraceIntelPTBundleLoader::GetSchema();
// Instruction events stats
     s.Format("    Number of individual events: {0}\n",
              events_stats.total_count);
     for (const auto &event_to_count : events_stats.events_counts) {
       s.Format("      {0}: {1}\n",
                TraceCursor::EventKindToString(event_to_count.first),
                event_to_count.second);
     }
   }
+  // Trace error stats
+  {
+    const DecodedThread::ErrorStats &error_stats =
+        decoded_thread_sp->GetErrorStats();
+    s << "\n  Errors:\n";
+    s.Format("    Number of individual errors: {0}\n",
+             error_stats.GetTotalCount());
+    s.Format("      Number of fatal errors: {0}\n", error_stats.fatal_errors);
+    for (const auto &[kind, count] : error_stats.libipt_errors) {
+      s.Format("     Number of libipt errors of kind [{0}]: {1}\n", kind,
+               count);
+    }
+    s.Format("      Number of other errors: {0}\n", error_stats.other_errors);
+  }
 
   if (storage.multicpu_decoder) {
     s << "\n  Multi-cpu decoding:\n";
     s.Format("    Total number of continuous executions found: {0}\n",
              storage.multicpu_decoder->GetTotalContinuousExecutionsCount());
     s.Format(
         "    Number of continuous executions for this thread: {0}\n",
         storage.multicpu_decoder->GetNumContinuousExecutionsForThread(tid));
json_str.attributeObject("threadStats", [&] {
         json_str.attributeObject("individualCounts", [&] {
           for (const auto &event_to_count : events_stats.events_counts) {
             json_str.attribute(
                 TraceCursor::EventKindToString(event_to_count.first),
                 event_to_count.second);
           }
         });
       });
+      // Trace error stats
+      const DecodedThread::ErrorStats &error_stats =
+          decoded_thread_sp->GetErrorStats();
+      json_str.attributeObject("errors", [&] {
+        json_str.attribute("totalCount", error_stats.GetTotalCount());
+        json_str.attributeObject("libiptErrors", [&] {
+          for (const auto &[kind, count] : error_stats.libipt_errors) {
+            json_str.attribute(kind, count);
+          }
+        });
+        json_str.attribute("fatalErrors", error_stats.fatal_errors);
+        json_str.attribute("otherErrors", error_stats.other_errors);
+      });
 
       if (storage.multicpu_decoder) {
         json_str.attribute(
             "continuousExecutions",
             storage.multicpu_decoder->GetNumContinuousExecutionsForThread(tid));
         json_str.attribute(
             "PSBBlocks",
             storage.multicpu_decoder->GePSBBlocksCountForThread(tid));

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTProperties.td

+include "../../../../include/lldb/Core/PropertiesBase.td"
+
+let Definition = "traceintelpt" in {
+  def InfiniteDecodingLoopVerificationThreshold:
+      Property<"infinite-decoding-loop-verification-threshold", "UInt64">,
+    Global,
+    DefaultUnsignedValue<10000>,
+    Desc<"Specify how many instructions following an individual Intel PT "
+      "packet must have been decoded before triggering the verification of "
+      "infinite decoding loops. If no decoding loop has been found after this "
+      "threshold T, another attempt will be done after 2T instructions, then "
+      "4T, 8T and so on, which guarantees a total linear time spent checking "
+      "this anomaly. If a loop is found, then decoding of the corresponding "
+      "PSB block is stopped. An error is hence emitted in the trace and "
+      "decoding is resumed in the next PSB block.">;
+  def ExtremelyLargeDecodingThreshold:
+      Property<"extremely-large-decoding-threshold", "UInt64">,
+    Global,
+    DefaultUnsignedValue<500000>,
+    Desc<"Specify how many instructions following an individual Intel PT "
+      "packet must have been decoded before stopping the decoding of the "
+      "corresponding PSB block. An error is hence emitted in the trace and "
+      "decoding is resumed in the next PSB block.">;
+}

lldb/test/API/commands/trace/TestTraceDumpInfo.py

"timingInSeconds": {
       "Decoding instructions": 0''', '''
     },
     "events": {
       "totalCount": 3,
       "individualCounts": {
         "software disabled tracing": 2,
         "trace synchronization point": 1
       }
+    },
+    "errors": {
+      "totalCount": 0,
+      "libiptErrors": {},
+      "fatalErrors": 0,
+      "otherErrors": 0
     }
   },
   "globalStats": {
     "timingInSeconds": {}
   }
 }'''])

lldb/test/API/commands/trace/TestTraceLoad.py

"threadStats": {
     },
     "timingInSeconds": {
       "Decoding instructions": ''', '''
     },
     "events": {
       "totalCount": 0,
       "individualCounts": {}
     },
+    "errors": {
+      "totalCount": 0,
+      "libiptErrors": {},
+      "fatalErrors": 0,
+      "otherErrors": 0
+    },
     "continuousExecutions": 0,
     "PSBBlocks": 0
   },
   "globalStats": {
     "timingInSeconds": {
       "Context switch and Intel PT traces correlation": 0
     },
     "totalUnattributedPSBBlocks": 0,
"events": {
       "totalCount": 11,
       "individualCounts": {
         "software disabled tracing": 1,
         "trace synchronization point": 1,
         "CPU core changed": 1,
         "HW clock tick": 8
       }
     },
+    "errors": {
+      "totalCount": 1,
+      "libiptErrors": {},
+      "fatalErrors": 0,
+      "otherErrors": 1
+    },
     "continuousExecutions": 1,
     "PSBBlocks": 1
   },
   "globalStats": {
     "timingInSeconds": {
       "Context switch and Intel PT traces correlation": 0''', '''},
     "totalUnattributedPSBBlocks": 0,
     "totalCountinuosExecutions": 153,