[trace][intel pt] Support a new kernel section in LLDB’s trace bundle schema

[persona0220]

Summary

Add a new "kernel" section with following schema.

"kernel": {
  "loadAddress"?: decimal | hex string | string decimal 
  # This is optional. If it's not specified, use default address 0xffffffff81000000.
  "file": string
  # path to the kernel image
}

Here's more details of the diff:

• If "kernel" section exist, it means current tracing mode is KernelMode.
• If tracing mode is KernelMode, the "processes" section must be empty and the "kernel" and "cpus" section must be provided. This is tested with TestTraceLoad.
• "kernel" section is parsed and turned into a new process with a single module which is the kernel image. The kernel process has N fake threads, one for each cpu.

[wallace]

Are the files in lldb/test/API/commands/trace/intelpt-kernel-trace/cores/ actual kernel traces? If not, just use some trace files that are already present in the repo. You can use relative paths of the form ../../trace_sample/cores/... to refer to them.

Good start for this task, btw!

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

175

enum class

175

@jj10306 , TraceMode or TracingMode?

197

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleLoader.cpp

146–161

this part is the same as the one in ParseProcess. Create a function to dedup this code. Maybe a good name is CreateEmptyProcess

168–176

you can make this simpler

185

try without setting this one. Hopefully you only need to set one file spec. The PlatformFileSpec helps distinguish the binary path on the lldb's machine from the one were collection happened, but I don't think that feature will be useful for this kernel thing

294

307

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleSaver.cpp

376

add this now. You have to update the corresponding tests to make sure that saving a loaded trace bundle produces the same information.

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

25

lldb/test/API/commands/trace/intelpt-kernel-trace/trace.json

21–22

Make "processes" optional, so that it's not even necessary to specify this field if "kernel" is provided.

[jj10306]

Nice work!
qq: Do we plan to add this kernel tracing support for live tracing as well?

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

87

What's the trace_mode being used for? afaict it's unused in this method.

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

175

I don't have a strong preference on the naming, but I was wondering how/if this enum is being used currently?
I understand this would be necessary when configuring a live tracing session bc this enum would control the configuration of the perf_event, but afaict this diff is only adding support for the kernel in the post mortem case so I don't quite understand how it's being used.

189–199

Should these be for postmortem instead of live?

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

25

a couple questsions here:

1. where is this value coming from? Would be useful to provide a link to documentation
2. does this depend on the system?
3. I know you and @wallace were discussing the implications of ASLR offline, but curious what will happen if this load address is not correct due to ASLR or any other reason. Do we have a way to detect this?

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.cpp

165–166

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.h

61

nit: any reason not to use reuse JSONModule here? The only difference I see is that the load address is optional.
Not sure if there's a strong reason against this, but imo it should be the bundle creator's responsibility to provide a load address of the kernel and not LLDB's responsibility to use a default kernel load address.

[wallace]

qq: Do we plan to add this kernel tracing support for live tracing as well?

I think it won't be possible without major changes or at least a big discussion because LLDB is supposed to operated on processes it's debugging, so let's better leave it out of scope for now.

[wallace]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

87

it'll be used later because this diff doesn't make decoding work yet

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

175

for now, live tracing will always be UserMode but postmortem could be any of those.

189–199

I think you are right :)

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

25

Just like the comment I left below, I think it's better not to have this default value here because it's not guaranteed to work on every system.

What we can implement in the collector is the following:

read /proc/kcore and look for the line with the entry SYMBOL(_stext)=ffffffff81000000

then in the documentation we can mention that this a way to get this load address, and we can implement that in the collector

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.h

61

I think JSONKernel here is fine just to allow more easily other fields to be added to this struct, like kernel version or any other information that could be useful in the future.
Regarding the default, after giving a second thought, I think you are right. Let's better not have a default but instead show in the documentation ways to figure out this value.

[persona0220]

• Remove duplicate files under lldb/test/API/commands/trace/intelpt-kernel-trace and use relative path to ../intelpt-multi-core-trace/
• Update TraceIntelBundleSaver to save "kernel" section
• Add test to TestTraceSave
• Add CreateEmptyProcess to dedup between ParseProccess and PareKernel.

[persona0220]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

25

1. It's the address of startup_64, the start address of loaded kernel image. This is a good documentation https://0xax.gitbooks.io/linux-insides/content/Theory/linux-theory-2.html
2. It depends on architecture, and this is for x86. Thus, user can skip "loadAddress" section in "kernel" section in x86, and for other architecture, they have to specify the loadAddress.
3. This is a good question! We don't have a logic to detect it yet. What I can come up roughly is analyzing the given elf file with load address and check whether it's valid?

25

As we discussed offline, kernel load address is always set to ffffffff81000000 for x86, so I'll keep default value and accept an optional load address.

[wallace]

just some minor cosmetic changes :)

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleLoader.cpp

177–184

try to reduce duplication

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleSaver.cpp

345–350

convert these checks into asserts, as they should crash the program if they happen.

403–405

too much duplication. Better first calculate the processes and kernel sections, and then just perform one single invocation to JSONTraceBundleDescription()

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.cpp

150

revert your change that removes parsing processes here, that way you can dedup changes below

170–171

check for its existence because it's now an optional

179–182

remove this

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.h

58

also make this one Optional, following the changes in the schema

lldb/test/API/commands/trace/TestTraceLoad.py

274

also check the load address

[wallace]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

26

mention that github link here

[persona0220]

Resolved comments

[wallace]

almost there!

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

105–106

why did you remove this?

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleLoader.cpp

307

remove this line. We should allow the user to pass an empty array for "processes" or not passing it at all. This will make the json easier to work with

308

please break this into multiple lines of at most 80 chars each

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleSaver.cpp

395–401

that is enough

396

that is enough

lldb/test/API/commands/trace/TestTraceLoad.py

274

275

remove this one

lldb/test/API/commands/trace/intelpt-kernel-trace/trace.json

27

write another test with a custom load address

[wallace]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleLoader.cpp

308

you didn't apply these suggestions:

• add or not passed at all,
• break into multiple lines

[persona0220]

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

105–106

Because it seems that tids vector is not used anywhere

[wallace]

nice job! Now you just need to connect it with the collector :)

[persona0220]

rebasing to the upstream

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h

public:
   ScopedTaskTimer &GetThreadTimer(lldb::tid_t tid);
 
   /// \return
   ///     The global copedTaskTimer object for this trace.
   ScopedTaskTimer &GetGlobalTimer();
 
   TraceIntelPTSP GetSharedPtr();
 
+  enum class TraceMode { UserMode, KernelMode };
+
+  TraceMode GetTraceMode();
+
 private:
   friend class TraceIntelPTBundleLoader;
 
   llvm::Expected<pt_cpu> GetCPUInfoForLiveProcess();
 
   /// Postmortem trace constructor
   ///
   /// \param[in] bundle_description
   ///     The definition file for the postmortem bundle.
   ///
   /// \param[in] traced_processes
-  ///     The processes traced in the live session.
+  ///     The processes traced in the postmortem session.
   ///
   /// \param[in] trace_threads
-  ///     The threads traced in the live session. They must belong to the
+  ///     The threads traced in the postmortem session. They must belong to the
   ///     processes mentioned above.
   ///
+  /// \param[in] trace_mode
+  ///     The tracing mode of the postmortem session.
+  ///
   /// \return
   ///     A TraceIntelPT shared pointer instance.
   /// \{
   static TraceIntelPTSP CreateInstanceForPostmortemTrace(
       JSONTraceBundleDescription &bundle_description,
       llvm::ArrayRef<lldb::ProcessSP> traced_processes,
-      llvm::ArrayRef<lldb::ThreadPostMortemTraceSP> traced_threads);
+      llvm::ArrayRef<lldb::ThreadPostMortemTraceSP> traced_threads,
+      TraceMode trace_mode);
 
   /// This constructor is used by CreateInstanceForPostmortemTrace to get the
   /// instance ready before using shared pointers, which is a limitation of C++.
   TraceIntelPT(JSONTraceBundleDescription &bundle_description,
-               llvm::ArrayRef<lldb::ProcessSP> traced_processes);
+               llvm::ArrayRef<lldb::ProcessSP> traced_processes,
+               TraceMode trace_mode);
   /// \}
 
   /// Constructor for live processes
-  TraceIntelPT(Process &live_process) : Trace(live_process){};
+  TraceIntelPT(Process &live_process)
+      : Trace(live_process), trace_mode(TraceMode::UserMode){};
 
   /// Decode the trace of the given thread that, i.e. recontruct the traced
   /// instructions.
   ///
   /// \param[in] thread
   ///     If \a thread is a \a ThreadTrace, then its internal trace file will be
   ///     decoded. Live threads are not currently supported.
   ///
private:
 
   /// It is provided by either a trace bundle or a live process' "cpuInfo"
   /// binary data. We don't put it in the Storage because this variable doesn't
   /// change.
   llvm::Optional<pt_cpu> m_cpu_info;
 
   /// Get the storage after refreshing the data in the case of a live process.
   Storage &GetUpdatedStorage();
+
+  /// The tracing mode of post mortem trace.
+  TraceMode trace_mode;
 };
 
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPT_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp

Expected<TraceSP> TraceIntelPT::CreateInstanceForLiveProcess(Process &process) {
   process.GetTarget().SetTrace(instance);
   return instance;
 }
 
 TraceIntelPTSP TraceIntelPT::GetSharedPtr() {
   return std::static_pointer_cast<TraceIntelPT>(shared_from_this());
 }
 
+TraceIntelPT::TraceMode TraceIntelPT::GetTraceMode() { return trace_mode; }
+
 TraceIntelPTSP TraceIntelPT::CreateInstanceForPostmortemTrace(
     JSONTraceBundleDescription &bundle_description,
     ArrayRef<ProcessSP> traced_processes,
-    ArrayRef<ThreadPostMortemTraceSP> traced_threads) {
+    ArrayRef<ThreadPostMortemTraceSP> traced_threads, TraceMode trace_mode) {
   TraceIntelPTSP trace_sp(
-      new TraceIntelPT(bundle_description, traced_processes));
+      new TraceIntelPT(bundle_description, traced_processes, trace_mode));
   trace_sp->m_storage.tsc_conversion =
       bundle_description.tsc_perf_zero_conversion;
 
   if (bundle_description.cpus) {
     std::vector<cpu_id_t> cpus;
 
     for (const JSONCpu &cpu : *bundle_description.cpus) {
       trace_sp->SetPostMortemCpuDataFile(cpu.id, IntelPTDataKinds::kIptTrace,
                                          FileSpec(cpu.ipt_trace));
 
       trace_sp->SetPostMortemCpuDataFile(
           cpu.id, IntelPTDataKinds::kPerfContextSwitchTrace,
           FileSpec(cpu.context_switch_trace));
       cpus.push_back(cpu.id);
     }
 
-    std::vector<tid_t> tids;
-    for (const JSONProcess &process : bundle_description.processes)
-      for (const JSONThread &thread : process.threads)
-        tids.push_back(thread.tid);
-
     trace_sp->m_storage.multicpu_decoder.emplace(trace_sp);
   } else {
     for (const ThreadPostMortemTraceSP &thread : traced_threads) {
       trace_sp->m_storage.thread_decoders.try_emplace(
           thread->GetID(), std::make_unique<ThreadDecoder>(thread, *trace_sp));
       if (const Optional<FileSpec> &trace_file = thread->GetTraceFile()) {
         trace_sp->SetPostMortemThreadDataFile(
             thread->GetID(), IntelPTDataKinds::kIptTrace, *trace_file);
       }
     }
   }
 
   for (const ProcessSP &process_sp : traced_processes)
     process_sp->GetTarget().SetTrace(trace_sp);
   return trace_sp;
 }
 
 TraceIntelPT::TraceIntelPT(JSONTraceBundleDescription &bundle_description,
-                           ArrayRef<ProcessSP> traced_processes)
+                           ArrayRef<ProcessSP> traced_processes,
+                           TraceMode trace_mode)
     : Trace(traced_processes, bundle_description.GetCpuIds()),
-      m_cpu_info(bundle_description.cpu_info) {}
+      m_cpu_info(bundle_description.cpu_info), trace_mode(trace_mode) {}
 
 Expected<DecodedThreadSP> TraceIntelPT::Decode(Thread &thread) {
   if (const char *error = RefreshLiveProcessState())
     return createStringError(inconvertibleErrorCode(), error);
 
   Storage &storage = GetUpdatedStorage();
   if (storage.multicpu_decoder)
     return storage.multicpu_decoder->Decode(thread);

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleLoader.h

lldb::ThreadPostMortemTraceSP ParseThread(Process &process,
                                             const JSONThread &thread);
 
   /// Given a bundle description and a list of fully parsed processes,
   /// create an actual Trace instance that "traces" these processes.
   llvm::Expected<lldb::TraceSP>
   CreateTraceIntelPTInstance(JSONTraceBundleDescription &bundle_description,
                              std::vector<ParsedProcess> &parsed_processes);
 
+  /// Create an empty Process object with given pid and target.
+  llvm::Expected<ParsedProcess> CreateEmptyProcess(lldb::pid_t pid,
+                                                   llvm::StringRef triple);
+
   /// Create the corresponding Threads and Process objects given the JSON
   /// process definition.
   ///
   /// \param[in] process
   ///   The JSON process definition
   llvm::Expected<ParsedProcess> ParseProcess(const JSONProcess &process);
 
   /// Create a module associated with the given \p target using the definition
   /// from \p module.
   llvm::Error ParseModule(Target &target, const JSONModule &module);
 
+  /// Create a kernel process and cpu threads given the JSON kernel definition.
+  llvm::Expected<ParsedProcess>
+  ParseKernel(const JSONTraceBundleDescription &bundle_description);
+
   /// Create a user-friendly error message upon a JSON-parsing failure using the
   /// \a json::ObjectMapper functionality.
   ///
   /// \param[in] root
   ///   The \a llvm::json::Path::Root used to parse the JSON \a value.
   ///
   /// \param[in] value
   ///   The json value that failed to parse.

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleLoader.cpp

 //===-- TraceIntelPTBundleLoader.cpp --------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "TraceIntelPTBundleLoader.h"
 
 #include "../common/ThreadPostMortemTrace.h"
 #include "TraceIntelPT.h"
+#include "TraceIntelPTConstants.h"
 #include "TraceIntelPTJSONStructs.h"
 
 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/Module.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Target/Target.h"
 
 using namespace lldb;
TraceIntelPTBundleLoader::ParseThread(Process &process,
 
   ThreadPostMortemTraceSP thread_sp =
       std::make_shared<ThreadPostMortemTrace>(process, tid, trace_file);
   process.GetThreadList().AddThread(thread_sp);
   return thread_sp;
 }
 
 Expected<TraceIntelPTBundleLoader::ParsedProcess>
-TraceIntelPTBundleLoader::ParseProcess(const JSONProcess &process) {
+TraceIntelPTBundleLoader::CreateEmptyProcess(lldb::pid_t pid,
+                                             llvm::StringRef triple) {
   TargetSP target_sp;
   Status error = m_debugger.GetTargetList().CreateTarget(
-      m_debugger, /*user_exe_path*/ StringRef(), process.triple.value_or(""),
-      eLoadDependentsNo,
+      m_debugger, /*user_exe_path*/ StringRef(), triple, eLoadDependentsNo,
       /*platform_options*/ nullptr, target_sp);
 
   if (!target_sp)
     return error.ToError();
 
   ParsedProcess parsed_process;
   parsed_process.target_sp = target_sp;
 
   ProcessSP process_sp = target_sp->CreateProcess(
       /*listener*/ nullptr, "trace",
       /*crash_file*/ nullptr,
       /*can_connect*/ false);
 
-  process_sp->SetID(static_cast<lldb::pid_t>(process.pid));
+  process_sp->SetID(static_cast<lldb::pid_t>(pid));
+
+  return parsed_process;
+}
+
+Expected<TraceIntelPTBundleLoader::ParsedProcess>
+TraceIntelPTBundleLoader::ParseProcess(const JSONProcess &process) {
+  Expected<ParsedProcess> parsed_process =
+      CreateEmptyProcess(process.pid, process.triple.value_or(""));
+
+  if (!parsed_process)
+    return parsed_process.takeError();
+
+  ProcessSP process_sp = parsed_process->target_sp->GetProcessSP();
 
   for (const JSONThread &thread : process.threads)
-    parsed_process.threads.push_back(ParseThread(*process_sp, thread));
+    parsed_process->threads.push_back(ParseThread(*process_sp, thread));
 
   for (const JSONModule &module : process.modules)
-    if (Error err = ParseModule(*target_sp, module))
+    if (Error err = ParseModule(*parsed_process->target_sp, module))
       return std::move(err);
 
   if (!process.threads.empty())
     process_sp->GetThreadList().SetSelectedThreadByIndexID(0);
 
   // We invoke DidAttach to create a correct stopped state for the process and
   // its threads.
   ArchSpec process_arch;
   process_sp->DidAttach(process_arch);
 
   return parsed_process;
 }
 
+Expected<TraceIntelPTBundleLoader::ParsedProcess>
+TraceIntelPTBundleLoader::ParseKernel(
+    const JSONTraceBundleDescription &bundle_description) {
+  Expected<ParsedProcess> parsed_process =
+      CreateEmptyProcess(kDefaultKernelProcessID, "");
+
+  if (!parsed_process)
+    return parsed_process.takeError();
+
+  ProcessSP process_sp = parsed_process->target_sp->GetProcessSP();
+
+  // Add cpus as fake threads
+  for (const JSONCpu &cpu : *bundle_description.cpus) {
+    ThreadPostMortemTraceSP thread_sp = std::make_shared<ThreadPostMortemTrace>(
+        *process_sp, static_cast<lldb::tid_t>(cpu.id), FileSpec(cpu.ipt_trace));
+    thread_sp->SetName(formatv("kernel_cpu_{0}", cpu.id).str().c_str());
+    process_sp->GetThreadList().AddThread(thread_sp);
+    parsed_process->threads.push_back(thread_sp);
+  }
+
+  // Add kernel image
+  FileSpec file_spec(bundle_description.kernel->file);
+  ModuleSpec module_spec;
+  module_spec.GetFileSpec() = file_spec;
+
+  Status error;
+  ModuleSP module_sp =
+      parsed_process->target_sp->GetOrCreateModule(module_spec, false, &error);
+
+  if (error.Fail())
+    return error.ToError();
+
+  lldb::addr_t load_address =
+      bundle_description.kernel->load_address
+          ? bundle_description.kernel->load_address->value
+          : kDefaultKernelLoadAddress;
+
+  bool load_addr_changed = false;
+  module_sp->SetLoadAddress(*parsed_process->target_sp, load_address, false,
+                            load_addr_changed);
+
+  process_sp->GetThreadList().SetSelectedThreadByIndexID(0);
+
+  // We invoke DidAttach to create a correct stopped state for the process and
+  // its threads.
+  ArchSpec process_arch;
+  process_sp->DidAttach(process_arch);
+
+  return parsed_process;
+}
+
 Expected<std::vector<TraceIntelPTBundleLoader::ParsedProcess>>
 TraceIntelPTBundleLoader::LoadBundle(
     const JSONTraceBundleDescription &bundle_description) {
   std::vector<ParsedProcess> parsed_processes;
 
   auto HandleError = [&](Error &&err) {
     // Delete all targets that were created so far in case of failures
     for (ParsedProcess &parsed_process : parsed_processes)
       m_debugger.GetTargetList().DeleteTarget(parsed_process.target_sp);
     return std::move(err);
   };
 
-  for (const JSONProcess &process : bundle_description.processes) {
+  if (bundle_description.processes) {
+    for (const JSONProcess &process : *bundle_description.processes) {
       if (Expected<ParsedProcess> parsed_process = ParseProcess(process))
         parsed_processes.push_back(std::move(*parsed_process));
       else
         return HandleError(parsed_process.takeError());
     }
+  }
+
+  if (bundle_description.kernel) {
+    if (Expected<ParsedProcess> kernel_process =
+            ParseKernel(bundle_description))
+      parsed_processes.push_back(std::move(*kernel_process));
+    else
+      return HandleError(kernel_process.takeError());
+  }
 
   return parsed_processes;
 }
 
 StringRef TraceIntelPTBundleLoader::GetSchema() {
   static std::string schema;
   if (schema.empty()) {
     schema = R"({
   "type": "intel-pt",
   "cpuInfo": {
     // CPU information gotten from, for example, /proc/cpuinfo.
 
     "vendor": "GenuineIntel" | "unknown",
     "family": integer,
     "model": integer,
     "stepping": integer
   },
-  "processes": [
+  "processes?": [
     {
       "pid": integer,
       "triple"?: string,
           // Optional clang/llvm target triple.
       "threads": [
           // A list of known threads for the given process. When context switch
           // data is provided, LLDB will automatically create threads for the
           // this process whenever it finds new threads when traversing the
StringRef TraceIntelPTBundleLoader::GetSchema() {
   "tscPerfZeroConversion"?: {
     // Values used to convert between TSCs and nanoseconds. See the time_zero
     // section in https://man7.org/linux/man-pages/man2/perf_event_open.2.html
     // for for information.
 
     "timeMult": integer,
     "timeShift": integer,
     "timeZero": integer | string decimal | hex string,
+  },
+  "kernel"?: {
+    "loadAddress"?: integer | string decimal | hex string,
+        // Kernel's image load address. Defaults to 0xffffffff81000000, which
+        // is a load address of x86 architecture if KASLR is not enabled.
+    "file": string,
+        // Path to the kernel image.
   }
 }
 
 Notes:
 
-- All paths are either absolute or relative to folder containing the bundle description file.
+- All paths are either absolute or relative to folder containing the bundle
+  description file.
 - "cpus" is provided if and only if processes[].threads[].iptTrace is not provided.
 - "tscPerfZeroConversion" must be provided if "cpus" is provided.
- })";
+- If "kernel" is provided, then the "processes" section must be empty or not
+  passed at all, and the "cpus" section must be provided. This configuration
+  indicates that the kernel was traced and user processes weren't. Besides
+  that, the kernel is treated as a single process with one thread per CPU
+  core. This doesn't handle actual kernel threads, but instead treats
+  all the instructions executed by the kernel on each core as an
+  individual thread.})";
   }
   return schema;
 }
 
 Error TraceIntelPTBundleLoader::AugmentThreadsFromContextSwitches(
     JSONTraceBundleDescription &bundle_description) {
-  if (!bundle_description.cpus)
+  if (!bundle_description.cpus || !bundle_description.processes)
     return Error::success();
 
   if (!bundle_description.tsc_perf_zero_conversion)
     return createStringError(inconvertibleErrorCode(),
                              "TSC to nanos conversion values are needed when "
                              "context switch information is provided.");
 
   DenseMap<lldb::pid_t, JSONProcess *> indexed_processes;
   DenseMap<JSONProcess *, DenseSet<tid_t>> indexed_threads;
 
-  for (JSONProcess &process : bundle_description.processes) {
+  for (JSONProcess &process : *bundle_description.processes) {
     indexed_processes[process.pid] = &process;
     for (JSONThread &thread : process.threads)
       indexed_threads[&process].insert(thread.tid);
   }
 
   auto on_thread_seen = [&](lldb::pid_t pid, tid_t tid) {
     auto proc = indexed_processes.find(pid);
     if (proc == indexed_processes.end())
Expected<TraceSP> TraceIntelPTBundleLoader::CreateTraceIntelPTInstance(
   std::vector<ThreadPostMortemTraceSP> threads;
   std::vector<ProcessSP> processes;
   for (const ParsedProcess &parsed_process : parsed_processes) {
     processes.push_back(parsed_process.target_sp->GetProcessSP());
     threads.insert(threads.end(), parsed_process.threads.begin(),
                    parsed_process.threads.end());
   }
 
+  TraceIntelPT::TraceMode trace_mode = bundle_description.kernel
+                                           ? TraceIntelPT::TraceMode::KernelMode
+                                           : TraceIntelPT::TraceMode::UserMode;
+
   TraceSP trace_instance = TraceIntelPT::CreateInstanceForPostmortemTrace(
-      bundle_description, processes, threads);
+      bundle_description, processes, threads, trace_mode);
   for (const ParsedProcess &parsed_process : parsed_processes)
     parsed_process.target_sp->SetTrace(trace_instance);
 
   return trace_instance;
 }
 
 void TraceIntelPTBundleLoader::NormalizeAllPaths(
     JSONTraceBundleDescription &bundle_description) {
-  for (JSONProcess &process : bundle_description.processes) {
+  if (bundle_description.processes) {
+    for (JSONProcess &process : *bundle_description.processes) {
       for (JSONModule &module : process.modules) {
         module.system_path = NormalizePath(module.system_path).GetPath();
         if (module.file)
           module.file = NormalizePath(*module.file).GetPath();
       }
       for (JSONThread &thread : process.threads) {
         if (thread.ipt_trace)
           thread.ipt_trace = NormalizePath(*thread.ipt_trace).GetPath();
       }
     }
+  }
   if (bundle_description.cpus) {
     for (JSONCpu &cpu : *bundle_description.cpus) {
       cpu.context_switch_trace =
           NormalizePath(cpu.context_switch_trace).GetPath();
       cpu.ipt_trace = NormalizePath(cpu.ipt_trace).GetPath();
     }
   }
+  if (bundle_description.kernel) {
+    bundle_description.kernel->file =
+        NormalizePath(bundle_description.kernel->file).GetPath();
+  }
 }
 
 Expected<TraceSP> TraceIntelPTBundleLoader::Load() {
   json::Path::Root root("traceBundle");
   JSONTraceBundleDescription bundle_description;
   if (!fromJSON(m_bundle_description, bundle_description, root))
     return CreateJSONError(root, m_bundle_description);
 

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTBundleSaver.cpp

 //===-- TraceIntelPTBundleSaver.cpp ---------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "TraceIntelPTBundleSaver.h"
 
 #include "PerfContextSwitchDecoder.h"
 #include "TraceIntelPT.h"
+#include "TraceIntelPTConstants.h"
 #include "TraceIntelPTJSONStructs.h"
 
 #include "lldb/Core/Module.h"
 #include "lldb/Core/ModuleList.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Target/SectionLoadList.h"
 #include "lldb/Target/Target.h"
 #include "lldb/Target/ThreadList.h"
if (llvm::Expected<JSONProcess> json_process =
             BuildProcessSection(*process, directory))
       processes.push_back(std::move(*json_process));
     else
       return json_process.takeError();
   }
   return processes;
 }
 
+static llvm::Expected<JSONKernel>
+BuildKernelSection(TraceIntelPT &trace_ipt, const FileSpec &directory) {
+  JSONKernel json_kernel;
+  std::vector<Process *> processes = trace_ipt.GetAllProcesses();
+  Process *kernel_process = processes[0];
+
+  assert(processes.size() == 1 && "User processeses exist in kernel mode");
+  assert(kernel_process->GetID() == kDefaultKernelProcessID &&
+         "Kernel process not exist");
+
+  Expected<std::vector<JSONModule>> json_modules =
+      BuildModulesSection(*kernel_process, directory);
+  if (!json_modules)
+    return json_modules.takeError();
+
+  JSONModule kernel_image = json_modules.get()[0];
+  return JSONKernel{kernel_image.load_address, kernel_image.system_path};
+}
+
 Expected<FileSpec> TraceIntelPTBundleSaver::SaveToDisk(TraceIntelPT &trace_ipt,
                                                        FileSpec directory,
                                                        bool compact) {
   if (std::error_code ec =
           sys::fs::create_directories(directory.GetPath().c_str()))
     return llvm::errorCodeToError(ec);
 
   Expected<pt_cpu> cpu_info = trace_ipt.GetCPUInfo();
   if (!cpu_info)
     return cpu_info.takeError();
 
   FileSystem::Instance().Resolve(directory);
 
-  Expected<std::vector<JSONProcess>> json_processes =
-      BuildProcessesSection(trace_ipt, directory);
-
-  if (!json_processes)
-    return json_processes.takeError();
-
   Expected<Optional<std::vector<JSONCpu>>> json_cpus =
       BuildCpusSection(trace_ipt, directory, compact);
   if (!json_cpus)
     return json_cpus.takeError();
 
+  Optional<std::vector<JSONProcess>> json_processes;
+  Optional<JSONKernel> json_kernel;
+
+  if (trace_ipt.GetTraceMode() == TraceIntelPT::TraceMode::KernelMode) {
+    Expected<Optional<JSONKernel>> exp_json_kernel =
+        BuildKernelSection(trace_ipt, directory);
+    if (!exp_json_kernel)
+      return exp_json_kernel.takeError();
+    else
+      json_kernel = *exp_json_kernel;
+  } else {
+    Expected<Optional<std::vector<JSONProcess>>> exp_json_processes =
+        BuildProcessesSection(trace_ipt, directory);
+    if (!exp_json_processes)
+      return exp_json_processes.takeError();
+    else
+      json_processes = *exp_json_processes;
+  }
+
   JSONTraceBundleDescription json_intel_pt_bundle_desc{
-      "intel-pt", *cpu_info, *json_processes, *json_cpus,
-      trace_ipt.GetPerfZeroTscConversion()};
+      "intel-pt",
+      *cpu_info,
+      json_processes,
+      *json_cpus,
+      trace_ipt.GetPerfZeroTscConversion(),
+      json_kernel};
 
   return SaveTraceBundleDescription(toJSON(json_intel_pt_bundle_desc),
                                     directory);
 }

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTConstants.h

 namespace trace_intel_pt {
 
 const size_t kDefaultIptTraceSize = 4 * 1024;                  // 4KB
 const size_t kDefaultProcessBufferSizeLimit = 5 * 1024 * 1024; // 500MB
 const bool kDefaultEnableTscValue = false;
 const llvm::Optional<size_t> kDefaultPsbPeriod = llvm::None;
 const bool kDefaultPerCpuTracing = false;
 const bool kDefaultDisableCgroupFiltering = false;
 
+// Physical address where the kernel is loaded in x86 architecture. Refer to
+// https://github.com/torvalds/linux/blob/master/Documentation/x86/x86_64/mm.rst
+// for the start address of kernel text section.
+// The kernel entry point is 0x1000000 by default when KASLR is disabled.
+const lldb::addr_t kDefaultKernelLoadAddress = 0xffffffff81000000;
+const lldb::pid_t kDefaultKernelProcessID = 1;
+
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_CONSTANTS_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.h

 };
 
 struct JSONCpu {
   lldb::cpu_id_t id;
   std::string ipt_trace;
   std::string context_switch_trace;
 };
 
+struct JSONKernel {
+  llvm::Optional<JSONUINT64> load_address;
+  std::string file;
+};
+
 struct JSONTraceBundleDescription {
   std::string type;
   pt_cpu cpu_info;
-  std::vector<JSONProcess> processes;
+  llvm::Optional<std::vector<JSONProcess>> processes;
   llvm::Optional<std::vector<JSONCpu>> cpus;
   llvm::Optional<LinuxPerfZeroTscConversion> tsc_perf_zero_conversion;
+  llvm::Optional<JSONKernel> kernel;
 
   llvm::Optional<std::vector<lldb::cpu_id_t>> GetCpuIds();
 };
 
 llvm::json::Value toJSON(const JSONModule &module);
 
 llvm::json::Value toJSON(const JSONThread &thread);
 
 llvm::json::Value toJSON(const JSONProcess &process);
 
 llvm::json::Value toJSON(const JSONCpu &cpu);
 
 llvm::json::Value toJSON(const pt_cpu &cpu_info);
 
+llvm::json::Value toJSON(const JSONKernel &kernel);
+
 llvm::json::Value toJSON(const JSONTraceBundleDescription &bundle_description);
 
 bool fromJSON(const llvm::json::Value &value, JSONModule &module,
               llvm::json::Path path);
 
 bool fromJSON(const llvm::json::Value &value, JSONThread &thread,
               llvm::json::Path path);
 
 bool fromJSON(const llvm::json::Value &value, JSONProcess &process,
               llvm::json::Path path);
 
 bool fromJSON(const llvm::json::Value &value, JSONCpu &cpu,
               llvm::json::Path path);
 
 bool fromJSON(const llvm::json::Value &value, pt_cpu &cpu_info,
               llvm::json::Path path);
 
+bool fromJSON(const llvm::json::Value &value, JSONModule &kernel,
+              llvm::json::Path path);
+
 bool fromJSON(const llvm::json::Value &value,
               JSONTraceBundleDescription &bundle_description,
               llvm::json::Path path);
 } // namespace trace_intel_pt
 } // namespace lldb_private
 
 #endif // LLDB_SOURCE_PLUGINS_TRACE_INTEL_PT_TRACEINTELPTJSONSTRUCTS_H

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.cpp

if (!(o && o.map("vendor", vendor) && o.map("family", family) &&
     return false;
   cpu_info.vendor = vendor == "GenuineIntel" ? pcv_intel : pcv_unknown;
   cpu_info.family = family;
   cpu_info.model = model;
   cpu_info.stepping = stepping;
   return true;
 }
 
+json::Value toJSON(const JSONKernel &kernel) {
+  json::Object json_module;
+  if (kernel.load_address)
+    json_module["loadAddress"] = toJSON(*kernel.load_address, true);
+  json_module["file"] = kernel.file;
+  return std::move(json_module);
+}
+
+bool fromJSON(const json::Value &value, JSONKernel &kernel, Path path) {
+  ObjectMapper o(value, path);
+  return o && o.map("loadAddress", kernel.load_address) &&
+         o.map("file", kernel.file);
+}
+
 json::Value toJSON(const JSONTraceBundleDescription &bundle_description) {
   return Object{
       {"type", bundle_description.type},
       {"processes", bundle_description.processes},
       // We have to do this because the compiler fails at doing it
       // automatically because pt_cpu is not in a namespace
       {"cpuInfo", toJSON(bundle_description.cpu_info)},
       {"cpus", bundle_description.cpus},
-      {"tscPerfZeroConversion", bundle_description.tsc_perf_zero_conversion}};
+      {"tscPerfZeroConversion", bundle_description.tsc_perf_zero_conversion},
+      {"kernel", bundle_description.kernel}};
 }
 
 bool fromJSON(const json::Value &value,
               JSONTraceBundleDescription &bundle_description, Path path) {
   ObjectMapper o(value, path);
   if (!(o && o.map("processes", bundle_description.processes) &&
         o.map("type", bundle_description.type) &&
         o.map("cpus", bundle_description.cpus) &&
         o.map("tscPerfZeroConversion",
-              bundle_description.tsc_perf_zero_conversion)))
+              bundle_description.tsc_perf_zero_conversion) &&
+        o.map("kernel", bundle_description.kernel)))
     return false;
   if (bundle_description.cpus && !bundle_description.tsc_perf_zero_conversion) {
     path.report(
         "\"tscPerfZeroConversion\" is required when \"cpus\" is provided");
     return false;
   }
   // We have to do this because the compiler fails at doing it automatically
   // because pt_cpu is not in a namespace
   if (!fromJSON(*value.getAsObject()->get("cpuInfo"),
                 bundle_description.cpu_info, path.field("cpuInfo")))
     return false;
+
+  // When kernel section is present, this is kernel-only tracing. Thus, throw an
+  // error if the "processes" section is non-empty or the "cpus" section is not
+  // present.
+  if (bundle_description.kernel) {
+    if (bundle_description.processes &&
+        !bundle_description.processes->empty()) {
+      path.report("\"processes\" must be empty when \"kernel\" is provided");
+      return false;
+    }
+    if (!bundle_description.cpus) {
+      path.report("\"cpus\" is required when \"kernel\" is provided");
+      return false;
+    }
+  } else if (!bundle_description.processes) {
+    // Usermode tracing requires processes section.
+    path.report("\"processes\" is required when \"kernel\" is not provided");
+    return false;
+  }
   return true;
 }
 
 } // namespace trace_intel_pt
 } // namespace lldb_private

lldb/test/API/commands/trace/TestTraceLoad.py

"type": "intel-pt"
         self.assertEqual(self.dbg.GetNumTargets(), 0)
 
     def testLoadTraceCursor(self):
         src_dir = self.getSourceDir()
         trace_description_file_path = os.path.join(src_dir, "intelpt-multi-core-trace", "trace.json")
         traceDescriptionFile = lldb.SBFileSpec(trace_description_file_path, True)
 
         error = lldb.SBError()
         trace = self.dbg.LoadTraceFromFile(error, traceDescriptionFile)
         self.assertSBError(error)
 
         target = self.dbg.GetSelectedTarget()
         process = target.process
 
 
         # 1. Test some expected items of thread 1's trace cursor.
         thread1 = process.threads[1]
         cursor = trace.CreateNewCursor(error, thread1) 
def testLoadTraceCursor(self):
             TEST_SEEK_ID = 3
             randomAccessCursor.GoToId(TEST_SEEK_ID )
             # Reset the sequential cursor 
             sequentialTraversalCursor.Seek(0, lldb.eTraceCursorSeekTypeBeginning)
             sequentialTraversalCursor.SetForwards(True)
             for _ in range(TEST_SEEK_ID): sequentialTraversalCursor.Next()
             assertCurrentTraceCursorItemEqual(sequentialTraversalCursor, randomAccessCursor)
 
+    @testSBAPIAndCommands
+    def testLoadKernelTrace(self):
+        # kernel section without loadAddress (using default loadAddress).
+        src_dir = self.getSourceDir()
+        trace_description_file_path = os.path.join(src_dir, "intelpt-kernel-trace", "trace.json")
+        self.traceLoad(traceDescriptionFilePath=trace_description_file_path, substrs=["intel-pt"])
 
-            
-                
+        self.expect("image list", substrs=["0xffffffff81000000", "modules/m.out"])
+        self.expect("thread list", substrs=[
+            "Process 1 stopped",
+            "* thread #1: tid = 0x002d",
+            "  thread #2: tid = 0x0033"])
+
+        # kernel section with custom loadAddress.
+        trace_description_file_path = os.path.join(src_dir, "intelpt-kernel-trace",
+                "trace_with_loadAddress.json")
+        self.traceLoad(traceDescriptionFilePath=trace_description_file_path, substrs=["intel-pt"])
+        self.expect("image list", substrs=["0x400000", "modules/m.out"])
 
+    @testSBAPIAndCommands
+    def testLoadInvalidKernelTrace(self):
+        src_dir = self.getSourceDir()
 
+        # Test kernel section with non-empty processeses section.
+        trace_description_file_path = os.path.join(src_dir, "intelpt-kernel-trace", "trace_kernel_with_process.json")
+        expected_substrs = ['error: "processes" must be empty when "kernel" is provided when parsing traceBundle']
+        self.traceLoad(traceDescriptionFilePath=trace_description_file_path, error=True, substrs=expected_substrs)
 
-
-        
+        # Test kernel section without cpus section.
+        trace_description_file_path = os.path.join(src_dir, "intelpt-kernel-trace", "trace_kernel_wo_cpus.json")
+        expected_substrs = ['error: "cpus" is required when "kernel" is provided when parsing traceBundle']
+        self.traceLoad(traceDescriptionFilePath=trace_description_file_path, error=True, substrs=expected_substrs)

lldb/test/API/commands/trace/TestTraceSave.py

def testSaveTrace(self):
         # Compare with instructions saved at the first time
         ci.HandleCommand("thread trace dump instructions -c 10 --forwards", res)
         self.assertEqual(res.Succeeded(), True)
         self.assertEqual(res.GetOutput(), first_ten_instructions)
 
         ci.HandleCommand("thread trace dump instructions -c 10", res)
         self.assertEqual(res.Succeeded(), True)
         self.assertEqual(res.GetOutput(), last_ten_instructions)
+
+    def testSaveKernelTrace(self):
+        original_trace_file = os.path.join(self.getSourceDir(), "intelpt-kernel-trace",
+                "trace.json")
+        copied_trace_dir = os.path.join(self.getBuildDir(), "intelpt-kernel-trace")
+        copied_trace_file = os.path.join(copied_trace_dir, "trace.json")
+
+        self.expect("trace load -v " + original_trace_file, substrs=["intel-pt"])
+        self.expect("trace save " + copied_trace_dir)
+
+        # We finally check that the new json has the same information as the original one
+        with open(original_trace_file) as original_file:
+            original_file = json.load(original_file)
+            with open(copied_trace_file) as copy_file:
+                copy_file = json.load(copy_file)
+                self.assertTrue("kernel" in copy_file)
+
+                self.assertEqual(os.path.basename(original_file["kernel"]["file"]),
+                        os.path.basename(copy_file["kernel"]["file"]))

lldb/test/API/commands/trace/intelpt-kernel-trace/trace.json

+{
+  "cpus": [
+    {
+      "contextSwitchTrace": "../intelpt-multi-core-trace/cores/45.perf_context_switch_trace",
+      "id": 45,
+      "iptTrace": "../intelpt-multi-core-trace/cores/45.intelpt_trace"
+    },
+    {
+      "contextSwitchTrace": "../intelpt-multi-core-trace/cores/51.perf_context_switch_trace",
+      "id": 51,
+      "iptTrace": "../intelpt-multi-core-trace/cores/51.intelpt_trace"
+    }
+  ],
+  "cpuInfo": {
+    "family": 6,
+    "model": 85,
+    "stepping": 4,
+    "vendor": "GenuineIntel"
+  },
+  "tscPerfZeroConversion": {
+    "timeMult": 1076264588,
+    "timeShift": 31,
+    "timeZero": 18433473881008870804
+  },
+  "kernel": {
+    "file": "../intelpt-multi-core-trace/modules/m.out"
+  },
+  "type": "intel-pt"
+}

lldb/test/API/commands/trace/intelpt-kernel-trace/trace_kernel_with_process.json

+{
+  "cpus": [
+    {
+      "contextSwitchTrace": "../intelpt-multi-core-trace/cores/45.perf_context_switch_trace",
+      "id": 45,
+      "iptTrace": "../intelpt-multi-core-trace/cores/45.intelpt_trace"
+    },
+    {
+      "contextSwitchTrace": "../intelpt-multi-core-trace/cores/51.perf_context_switch_trace",
+      "id": 51,
+      "iptTrace": "../intelpt-multi-core-trace/cores/51.intelpt_trace"
+    }
+  ],
+  "cpuInfo": {
+    "family": 6,
+    "model": 85,
+    "stepping": 4,
+    "vendor": "GenuineIntel"
+  },
+  "processes": [
+    {
+      "modules": [
+        {
+          "file": "../intelpt-multi-core-trace/modules/m.out",
+          "systemPath": "/tmp/m.out",
+          "loadAddress": 4194304,
+          "uuid": "AEFB0D59-233F-80FF-6D3C-4DED498534CF-11017B3B"
+        }
+      ],
+      "pid": 3497234,
+      "threads": [
+        {
+          "tid": 3497234
+        },
+        {
+          "tid": 3497496
+        },
+        {
+          "tid": 3497497
+        }
+      ]
+    }
+  ],
+  "tscPerfZeroConversion": {
+    "timeMult": 1076264588,
+    "timeShift": 31,
+    "timeZero": 18433473881008870804
+  },
+  "type": "intel-pt",
+  "kernel": {
+    "file": "../intelpt-multi-core-trace/modules/m.out"
+  }
+}

lldb/test/API/commands/trace/intelpt-kernel-trace/trace_kernel_wo_cpus.json

+{
+  "cpuInfo": {
+    "family": 6,
+    "model": 85,
+    "stepping": 4,
+    "vendor": "GenuineIntel"
+  },
+  "processes": [
+  ],
+  "tscPerfZeroConversion": {
+    "timeMult": 1076264588,
+    "timeShift": 31,
+    "timeZero": 18433473881008870804
+  },
+  "kernel": {
+    "file": "../intelpt-multi-core-trace/modules/m.out",
+    "loadAddress": 4194304
+  },
+  "type": "intel-pt"
+}

lldb/test/API/commands/trace/intelpt-kernel-trace/trace_with_loadAddress.json

+{
+  "cpus": [
+    {
+      "contextSwitchTrace": "../intelpt-multi-core-trace/cores/45.perf_context_switch_trace",
+      "id": 45,
+      "iptTrace": "../intelpt-multi-core-trace/cores/45.intelpt_trace"
+    },
+    {
+      "contextSwitchTrace": "../intelpt-multi-core-trace/cores/51.perf_context_switch_trace",
+      "id": 51,
+      "iptTrace": "../intelpt-multi-core-trace/cores/51.intelpt_trace"
+    }
+  ],
+  "cpuInfo": {
+    "family": 6,
+    "model": 85,
+    "stepping": 4,
+    "vendor": "GenuineIntel"
+  },
+  "tscPerfZeroConversion": {
+    "timeMult": 1076264588,
+    "timeShift": 31,
+    "timeZero": 18433473881008870804
+  },
+  "kernel": {
+    "file": "../intelpt-multi-core-trace/modules/m.out",
+    "loadAddress": "0x400000"
+  },
+  "type": "intel-pt"
+}