[lldb] [Process/FreeBSDKernel] Support finding all processes

[mgorny]

Summary

Include the complete list of threads of all running processes
in the FreeBSDKernel plugin. This makes it possible to inspect
the states (including partial register dumps from PCB) of all kernel
and userspace threads at the time of crash, or at the time of reading
/dev/mem first.

[labath]

I think I've managed to figure out what the code is doing, but it wouldn't hurt to include a comment giving a high-level overview of what that code is doing and/or splitting it into smaller functions with helpful names.

Can we expect to have a test for this?

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp

203

sizeof(thread_name)

221

is this going to be unique (for the entire system)?

lldb/source/Plugins/Process/FreeBSDKernel/ThreadFreeBSDKernel.cpp

29

std::move

[mgorny]

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp

221

Yes, I have specifically verified that.

[mgorny]

Update the code per suggestions, update the first test case.

TODO: update the remaining tests, document the code better

[mgorny]

Add code comments

[mgorny]

Update all tests.

Phab doesn't like git binary deltas for some reason, so I've uploaded the patch with --no-binary.

New file sizes:

-rw-r--r-- 1 mgorny mgorny 120K 12-29 15:36 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-amd64-full.bz2
-rw-r--r-- 1 mgorny mgorny 132K 12-29 12:45 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-amd64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny  37K 12-29 15:45 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-arm64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny 136K 12-29 15:38 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-i386-minidump.bz2

[labath]

Can we make the number of threads smaller? Perhaps by patching the linked lists to skip the uninteresting ones? 600 threads is way too much, all we really need is one thread from each of the three categories. Then you can meaningfully assert the names and registers of the existing threads (since their layout is the same, there's probably no point in asserting all registers, but it'd be good to check one or two, just to make sure the starting offset is right).

[mgorny]

That would be a lot of work, and it would need to be repeated whenever the test data needs to be regenerated. How about leaving the threads as-is but just testing the few selected ones from the list?

[labath]

That would deal with the code coverage, but it still leaves us with a fairly large core file, and a lot of uninteresting tests obscuring the output.

How do you generate these core files? Do you actually create a fresh core dump or you just recompute the "interesting" portions from a master file you have around? If you make the changes the the master core file, then they would get automatically picked up during the recomputation.

Alternatively, maybe there is a way to capture a core file without so many processes. Either killing off everything before the core file is written, or by making sure the other processes are never started (something like init=/bin/bash on linux)?

[mgorny]

In D116255#3213612, @labath wrote:

That would deal with the code coverage, but it still leaves us with a fairly large core file, and a lot of uninteresting tests obscuring the output.

How do you generate these core files? Do you actually create a fresh core dump or you just recompute the "interesting" portions from a master file you have around? If you make the changes the the master core file, then they would get automatically picked up during the recomputation.

I do recompute them from master copies (which I finally need to upload somewhere) but the recomputation is really dumb and unaware of the file format. I suppose I could just hack LLDB to stop after grabbing the first N threads but… the first non-dump on-CPU thread is no. 200. Finishing on that will probably make the core smaller but not sure how much smaller. I'll try in a minute.

Alternatively, maybe there is a way to capture a core file without so many processes. Either killing off everything before the core file is written, or by making sure the other processes are never started (something like init=/bin/bash on linux)?

The vast majority of threads are kernel threads. At a quick glance, only about 20 threads look like userspace.

[mgorny]

Ok, so limiting to 200-ish threads halves the data size:

-rw-r--r-- 1 mgorny mgorny 76K 12-30 14:16 /tmp/vmcore.bz2

[labath]

In D116255#3213665, @mgorny wrote:

In D116255#3213612, @labath wrote:

That would deal with the code coverage, but it still leaves us with a fairly large core file, and a lot of uninteresting tests obscuring the output.

How do you generate these core files? Do you actually create a fresh core dump or you just recompute the "interesting" portions from a master file you have around? If you make the changes the the master core file, then they would get automatically picked up during the recomputation.

I do recompute them from master copies (which I finally need to upload somewhere) but the recomputation is really dumb and unaware of the file format.

I am aware how the recomputation works. My point is that if you hack the "master" copy (quotes because you probably still want to keep the original master around) then you only need to redo the hacks after when you're regenerating the master. At that point you'll also need to update all the test expectations, so it doesn't seem like too much extra work. (In fact, I wouldn't be surprised if we just end up adding a new core file instead of updating the old one).

I suppose I could just hack LLDB to stop after grabbing the first N threads but… the first non-dump on-CPU thread is no. 200. Finishing on that will probably make the core smaller but not sure how much smaller. I'll try in a minute.

You may still need to do some hex editing to avoid ending the list with a dangling pointer...

Alternatively, maybe there is a way to capture a core file without so many processes. Either killing off everything before the core file is written, or by making sure the other processes are never started (something like init=/bin/bash on linux)?

The vast majority of threads are kernel threads. At a quick glance, only about 20 threads look like userspace.

yikes. And I thought linux was bad..

[mgorny]

Skip repeating thread name when it's equal to comm. Include thread status in name field (i.e. 'crashed', 'on CPU n').

Update the test tooling to strip "non-interesting" processes from allproc and to grab bt/regs from three first threads. Update amd64 test to include only interesting processes (making the core much smaller), and to test backtrace and regs on one thread of each kind.

[mgorny]

Update all tests and the test documentation.

New sizes:

-rw-r--r-- 1 mgorny mgorny  13K 12-31 15:27 vmcore-amd64-full.bz2
-rw-r--r-- 1 mgorny mgorny  18K 12-31 13:14 vmcore-amd64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny  12K 12-31 16:02 vmcore-arm64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny  14K 12-31 16:31 vmcore-i386-minidump.bz2

[labath]

Thanks for doing this.

Interesting solution to the problem. I'm not sure how long will the diff files remain applicable, but I would've accepted this even without them so I think all is fine.

lldb/test/API/functionalities/postmortem/FreeBSDKernel/TestFreeBSDKernelVMCore.py

60–61

If you want to be even more fancy :)

[mgorny]

lldb/test/API/functionalities/postmortem/FreeBSDKernel/TestFreeBSDKernelVMCore.py

60–61

Neat. I didn't know threads are iterable like this. Unfortunately, swig makes it kinda hard to inspect stuff in a really Pythonic way.

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h

public:
 
   lldb_private::Status DoLoadCore() override;
 
   lldb_private::DynamicLoader *GetDynamicLoader() override;
 
 protected:
   bool DoUpdateThreadList(lldb_private::ThreadList &old_thread_list,
                           lldb_private::ThreadList &new_thread_list) override;
+
+  lldb::addr_t FindSymbol(const char* name);
 };
 
 #endif // LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_PROCESSFREEBSDKERNEL_H

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp

if (old_thread_list.GetSize(false) == 0) {
     case llvm::Triple::aarch64:
     case llvm::Triple::x86:
     case llvm::Triple::x86_64:
       break;
     default:
       return false;
     }
 
-    const Symbol *pcb_sym =
-        GetTarget().GetExecutableModule()->FindFirstSymbolWithNameAndType(
-            ConstString("dumppcb"));
-    ThreadSP thread_sp(new ThreadFreeBSDKernel(
-        *this, 1, pcb_sym ? pcb_sym->GetFileAddress() : LLDB_INVALID_ADDRESS));
+    Status error;
+
+    // struct field offsets are written as symbols so that we don't have
+    // to figure them out ourselves
+    int32_t offset_p_list = ReadSignedIntegerFromMemory(
+        FindSymbol("proc_off_p_list"), 4, -1, error);
+    int32_t offset_p_pid =
+        ReadSignedIntegerFromMemory(FindSymbol("proc_off_p_pid"), 4, -1, error);
+    int32_t offset_p_threads = ReadSignedIntegerFromMemory(
+        FindSymbol("proc_off_p_threads"), 4, -1, error);
+    int32_t offset_p_comm = ReadSignedIntegerFromMemory(
+        FindSymbol("proc_off_p_comm"), 4, -1, error);
+
+    int32_t offset_td_tid = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_tid"), 4, -1, error);
+    int32_t offset_td_plist = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_plist"), 4, -1, error);
+    int32_t offset_td_pcb = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_pcb"), 4, -1, error);
+    int32_t offset_td_oncpu = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_oncpu"), 4, -1, error);
+    int32_t offset_td_name = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_name"), 4, -1, error);
+
+    // fail if we were not able to read any of the offsets
+    if (offset_p_list == -1 || offset_p_pid == -1 || offset_p_threads == -1 ||
+        offset_p_comm == -1 || offset_td_tid == -1 || offset_td_plist == -1 ||
+        offset_td_pcb == -1 || offset_td_oncpu == -1 || offset_td_name == -1)
+      return false;
+
+    // dumptid contains the thread-id of the crashing thread
+    // dumppcb contains its PCB
+    int32_t dumptid =
+        ReadSignedIntegerFromMemory(FindSymbol("dumptid"), 4, -1, error);
+    lldb::addr_t dumppcb = FindSymbol("dumppcb");
+
+    // stoppcbs is an array of PCBs on all CPUs
+    // each element is of size pcb_size
+    int32_t pcbsize =
+        ReadSignedIntegerFromMemory(FindSymbol("pcb_size"), 4, -1, error);
+    lldb::addr_t stoppcbs = FindSymbol("stoppcbs");
+
+    // from FreeBSD sys/param.h
+    constexpr size_t fbsd_maxcomlen = 19;
+
+    // iterate through a linked list of all processes
+    // allproc is a pointer to the first list element, p_list field
+    // (found at offset_p_list) specifies the next element
+    for (lldb::addr_t proc =
+             ReadPointerFromMemory(FindSymbol("allproc"), error);
+         proc != 0 && proc != LLDB_INVALID_ADDRESS;
+         proc = ReadPointerFromMemory(proc + offset_p_list, error)) {
+      int32_t pid =
+          ReadSignedIntegerFromMemory(proc + offset_p_pid, 4, -1, error);
+      // process' command-line string
+      char comm[fbsd_maxcomlen + 1];
+      ReadCStringFromMemory(proc + offset_p_comm, comm, sizeof(comm), error);
+
+      // iterate through a linked list of all process' threads
+      // the initial thread is found in process' p_threads, subsequent
+      // elements are linked via td_plist field
+      for (lldb::addr_t td =
+               ReadPointerFromMemory(proc + offset_p_threads, error);
+           td != 0; td = ReadPointerFromMemory(td + offset_td_plist, error)) {
+        int32_t tid =
+            ReadSignedIntegerFromMemory(td + offset_td_tid, 4, -1, error);
+        lldb::addr_t pcb_addr =
+            ReadPointerFromMemory(td + offset_td_pcb, error);
+        // whether process was on CPU (-1 if not, otherwise CPU number)
+        int32_t oncpu =
+            ReadSignedIntegerFromMemory(td + offset_td_oncpu, 4, -2, error);
+        // thread name
+        char thread_name[fbsd_maxcomlen + 1];
+        ReadCStringFromMemory(td + offset_td_name, thread_name,
+                              sizeof(thread_name), error);
+
+        // if we failed to read TID, ignore this thread
+        if (tid == -1)
+          continue;
+
+        std::string thread_desc = llvm::formatv("(pid {0}) {1}", pid, comm);
+        if (*thread_name && strcmp(thread_name, comm)) {
+          thread_desc += '/';
+          thread_desc += thread_name;
+        }
+
+        // roughly:
+        // 1. if the thread crashed, its PCB is going to be at "dumppcb"
+        // 2. if the thread was on CPU, its PCB is going to be on the CPU
+        // 3. otherwise, its PCB is in the thread struct
+        if (tid == dumptid) {
+          // NB: dumppcb can be LLDB_INVALID_ADDRESS if reading it failed
+          pcb_addr = dumppcb;
+          thread_desc += " (crashed)";
+        } else if (oncpu != -1) {
+          // if we managed to read stoppcbs and pcb_size, use them to find
+          // the correct PCB
+          if (stoppcbs != LLDB_INVALID_ADDRESS && pcbsize > 0)
+            pcb_addr = stoppcbs + oncpu * pcbsize;
+          else
+            pcb_addr = LLDB_INVALID_ADDRESS;
+          thread_desc += llvm::formatv(" (on CPU {0})", oncpu);
+        }
+
+        ThreadSP thread_sp{
+            new ThreadFreeBSDKernel(*this, tid, pcb_addr, thread_desc)};
         new_thread_list.AddThread(thread_sp);
+      }
+    }
   } else {
     const uint32_t num_threads = old_thread_list.GetSize(false);
     for (uint32_t i = 0; i < num_threads; ++i)
       new_thread_list.AddThread(old_thread_list.GetThreadAtIndex(i, false));
   }
   return new_thread_list.GetSize(false) > 0;
 }
 
 Status ProcessFreeBSDKernel::DoLoadCore() {
   // The core is already loaded by CreateInstance().
   return Status();
 }
 
 DynamicLoader *ProcessFreeBSDKernel::GetDynamicLoader() {
   if (m_dyld_up.get() == nullptr)
     m_dyld_up.reset(DynamicLoader::FindPlugin(
         this, DynamicLoaderStatic::GetPluginNameStatic()));
   return m_dyld_up.get();
 }
 
+lldb::addr_t ProcessFreeBSDKernel::FindSymbol(const char *name) {
+  ModuleSP mod_sp = GetTarget().GetExecutableModule();
+  const Symbol *sym = mod_sp->FindFirstSymbolWithNameAndType(ConstString(name));
+  return sym ? sym->GetLoadAddress(&GetTarget()) : LLDB_INVALID_ADDRESS;
+}
+
 #if LLDB_ENABLE_FBSDVMCORE
 
 ProcessFreeBSDKernelFVC::ProcessFreeBSDKernelFVC(lldb::TargetSP target_sp,
                                                  ListenerSP listener_sp,
                                                  fvc_t *fvc)
     : ProcessFreeBSDKernel(target_sp, listener_sp), m_fvc(fvc) {}
 
 ProcessFreeBSDKernelFVC::~ProcessFreeBSDKernelFVC() {

lldb/source/Plugins/Process/FreeBSDKernel/ThreadFreeBSDKernel.h

 //===-- ThreadFreeBSDKernel.h ------------------------------------- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_THREADFREEBSDKERNEL_H
 #define LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_THREADFREEBSDKERNEL_H
 
 #include "lldb/Target/Thread.h"
 
 class ThreadFreeBSDKernel : public lldb_private::Thread {
 public:
   ThreadFreeBSDKernel(lldb_private::Process &process, lldb::tid_t tid,
-                      lldb::addr_t pcb_addr);
+                      lldb::addr_t pcb_addr, std::string thread_name);
 
   ~ThreadFreeBSDKernel() override;
 
   void RefreshStateAfterStop() override;
 
   lldb::RegisterContextSP GetRegisterContext() override;
 
   lldb::RegisterContextSP
   CreateRegisterContextForFrame(lldb_private::StackFrame *frame) override;
 
+  const char *GetName() override {
+    if (m_thread_name.empty())
+      return nullptr;
+    return m_thread_name.c_str();
+  }
+
+  void SetName(const char *name) override {
+    if (name && name[0])
+      m_thread_name.assign(name);
+    else
+      m_thread_name.clear();
+  }
+
 protected:
   bool CalculateStopInfo() override;
 
 private:
+  std::string m_thread_name;
   lldb::RegisterContextSP m_thread_reg_ctx_sp;
   lldb::addr_t m_pcb_addr;
 };
 
 #endif // LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_THREADFREEBSDKERNEL_H

lldb/source/Plugins/Process/FreeBSDKernel/ThreadFreeBSDKernel.cpp

 #include "RegisterContextFreeBSDKernel_i386.h"
 #include "RegisterContextFreeBSDKernel_x86_64.h"
 #include "ThreadFreeBSDKernel.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
 ThreadFreeBSDKernel::ThreadFreeBSDKernel(Process &process, lldb::tid_t tid,
-                                         lldb::addr_t pcb_addr)
-    : Thread(process, tid), m_pcb_addr(pcb_addr) {}
+                                         lldb::addr_t pcb_addr,
+                                         std::string thread_name)
+    : Thread(process, tid), m_thread_name(std::move(thread_name)),
+      m_pcb_addr(pcb_addr) {}
 
 ThreadFreeBSDKernel::~ThreadFreeBSDKernel() {}
 
 void ThreadFreeBSDKernel::RefreshStateAfterStop() {}
 
 lldb::RegisterContextSP ThreadFreeBSDKernel::GetRegisterContext() {
   if (!m_reg_context_sp)
     m_reg_context_sp = CreateRegisterContextForFrame(nullptr);
if (concrete_frame_idx == 0) {
     switch (arch.GetMachine()) {
     case llvm::Triple::aarch64:
       m_thread_reg_ctx_sp =
           std::make_shared<RegisterContextFreeBSDKernel_arm64>(
               *this, std::make_unique<RegisterInfoPOSIX_arm64>(arch, 0),
               m_pcb_addr);
       break;
     case llvm::Triple::x86:
-      m_thread_reg_ctx_sp =
-          std::make_shared<RegisterContextFreeBSDKernel_i386>(
-              *this, new RegisterContextFreeBSD_i386(arch), m_pcb_addr);
+      m_thread_reg_ctx_sp = std::make_shared<RegisterContextFreeBSDKernel_i386>(
       break;
     case llvm::Triple::x86_64:
       m_thread_reg_ctx_sp =
           std::make_shared<RegisterContextFreeBSDKernel_x86_64>(
               *this, new RegisterContextFreeBSD_x86_64(arch), m_pcb_addr);
       break;
     default:
       assert(false && "Unsupported architecture passed to ThreadFreeBSDKernel");

lldb/test/API/functionalities/postmortem/FreeBSDKernel/TestFreeBSDKernelVMCore.py

 import bz2
 import shutil
 import struct
 
 import lldb
 from lldbsuite.test.decorators import *
 from lldbsuite.test.lldbtest import *
 from lldbsuite.test import lldbutil
 
 
 @skipIfFBSDVMCoreSupportMissing
 class FreeBSDKernelVMCoreTestCase(TestBase):
     NO_DEBUG_INFO_TESTCASE = True
+    maxDiff = None
 
     mydir = TestBase.compute_mydir(__file__)
 
     def make_target(self, src_filename):
         src = self.getSourcePath(src_filename)
         dest = self.getBuildArtifact("kernel")
         self.yaml2obj(src, dest, max_size=30*1024*1024)
         return self.dbg.CreateTarget(dest)
 
     def make_vmcore(self, src_filename):
         src = self.getSourcePath(src_filename)
         dest = self.getBuildArtifact("vmcore")
         with bz2.open(src, "rb") as inf:
             with open(dest, "wb") as outf:
                 shutil.copyfileobj(inf, outf)
         return dest
 
-    def do_test(self, kernel_yaml, vmcore_bz2, bt_expected, regs_expected,
-                hz_value=100):
+    def do_test(self, kernel_yaml, vmcore_bz2, numthread, threads={}, hz=100):
         target = self.make_target(kernel_yaml)
         vmcore_file = self.make_vmcore(vmcore_bz2)
         process = target.LoadCore(vmcore_file)
 
         self.assertTrue(process, PROCESS_IS_VALID)
-        self.assertEqual(process.GetNumThreads(), 1)
+        self.assertEqual(process.GetNumThreads(), numthread)
         self.assertEqual(process.GetProcessID(), 0)
 
         # test memory reading
         self.expect("expr -- *(int *) &hz",
-                    substrs=["(int) $0 = %d" % hz_value])
+                    substrs=["(int) $0 = %d" % hz])
 
         main_mod = target.GetModuleAtIndex(0)
         hz_addr = (main_mod.FindSymbols("hz")[0].symbol.addr
                    .GetLoadAddress(target))
         error = lldb.SBError()
         self.assertEqual(process.ReadMemory(hz_addr, 4, error),
-                         struct.pack("<I", hz_value))
+                         struct.pack("<I", hz))
+
+        for thread_index, thread_data in threads.items():
+            bt_expected = thread_data["bt"]
+            regs_expected = thread_data["regs"]
+            thread = process.GetThreadAtIndex(thread_index)
+            self.assertEqual(thread.GetName(), thread_data["name"])
 
             # test backtrace
             self.assertEqual(
-            [process.GetThreadAtIndex(0).GetFrameAtIndex(i).addr
-             .GetLoadAddress(target) for i in range(len(bt_expected))],
+                [frame.addr.GetLoadAddress(target) for frame in thread],
                 bt_expected)
 
             # test registers
-        regs = process.GetThreadAtIndex(0).GetFrameAtIndex(0).GetRegisters()
+            regs = thread.GetFrameAtIndex(0).GetRegisters()
             reg_values = {}
             for regset in regs:
                 for reg in regset:
                     if reg.value is None:
                         continue
                     reg_values[reg.name] = reg.value
             self.assertEqual(reg_values, regs_expected)
 
         self.dbg.DeleteTarget(target)
 
     def test_amd64_full_vmcore(self):
         self.do_test("kernel-amd64.yaml", "vmcore-amd64-full.bz2",
-                     [0xffffffff80c09ade, 0xffffffff80c09916,
+                numthread=13,
+                threads={
+                    0: {"name": "(pid 806) sysctl (crashed)",
+                        "bt": [0xffffffff80c09ade, 0xffffffff80c09916,
                                0xffffffff80c09d90, 0xffffffff80c09b93,
                                0xffffffff80c57d91, 0xffffffff80c19e71,
                                0xffffffff80c192bc, 0xffffffff80c19933,
                                0xffffffff80c1977f, 0xffffffff8108ba8c,
-                      0xffffffff810620ce],
-                     {"rbx": "0x0000000000000000",
+                               0xffffffff810620ce,
+                               ],
+                        "regs": {"rbx": "0x0000000000000000",
                                  "rbp": "0xfffffe0085cb2760",
                                  "rsp": "0xfffffe0085cb2748",
                                  "r12": "0xfffffe0045a6c300",
                                  "r13": "0xfffff800033693a8",
                                  "r14": "0x0000000000000000",
                                  "r15": "0xfffff80003369380",
                                  "rip": "0xffffffff80c09ade",
-                      })
+                                 },
+                        },
+                    1: {"name": "(pid 11) idle/idle: cpu0 (on CPU 0)",
+                        "bt": [0xffffffff81057988, 0xffffffff81057949,
+                               0xffffffff8108a5ff, 0xffffffff81062537,
+                               0xffffffff80c1aa38, 0xffffffff80b9d587,
+                               0xffffffff810575b1, 0xffffffff81063b33,
+                               0xffffffff804e3edb, 0xffffffff8104dc6e,
+                               0xffffffff8104dd1f, 0xffffffff80c3f0b4,
+                               0xffffffff80bc7c5e,
+                               ],
+                        "regs": {"rbx": "0xffffffff81d43950",
+                                 "rbp": "0xffffffff81d43820",
+                                 "rsp": "0xffffffff81d43808",
+                                 "r12": "0xfffff80003374000",
+                                 "r13": "0x00000000027be000",
+                                 "r14": "0x0000000000000000",
+                                 "r15": "0xfffffe00009f7300",
+                                 "rip": "0xffffffff81057988",
+                                 },
+                        },
+                    10: {"name": "(pid 11) idle/idle: cpu9",
+                         "bt": [0xffffffff80c3c8c8, 0xffffffff80c16521,
+                                0xffffffff80c3f0b4, 0xffffffff80bc7c5e,
+                                ],
+                         "regs": {"rbx": "0x000000007fff29f4",
+                                  "rbp": "0xfffffe00007a4ad0",
+                                  "rsp": "0xfffffe00007a4a08",
+                                  "r12": "0xfffffe00009fd300",
+                                  "r13": "0x0000000000000608",
+                                  "r14": "0xfffffe00009250c0",
+                                  "r15": "0xfffffe0045a6c300",
+                                  "rip": "0xffffffff80c3c8c8",
+                                  },
+                         },
 
     def test_amd64_minidump(self):
         self.do_test("kernel-amd64.yaml", "vmcore-amd64-minidump.bz2",
-                     [0xffffffff80c09ade, 0xffffffff80c09916,
+                     numthread=16,
+                     threads={
+                         0: {"name": "(pid 800) sysctl (crashed)",
+                             "bt": [0xffffffff80c09ade, 0xffffffff80c09916,
                                     0xffffffff80c09d90, 0xffffffff80c09b93,
                                     0xffffffff80c57d91, 0xffffffff80c19e71,
                                     0xffffffff80c192bc, 0xffffffff80c19933,
                                     0xffffffff80c1977f, 0xffffffff8108ba8c,
-                      0xffffffff810620ce],
-                     {"rbx": "0x0000000000000000",
+                                    0xffffffff810620ce,
+                                    ],
+                             "regs": {"rbx": "0x0000000000000000",
                                       "rbp": "0xfffffe00798c4760",
                                       "rsp": "0xfffffe00798c4748",
                                       "r12": "0xfffffe0045b11c00",
                                       "r13": "0xfffff800033693a8",
                                       "r14": "0x0000000000000000",
                                       "r15": "0xfffff80003369380",
                                       "rip": "0xffffffff80c09ade",
+                                      },
+                             },
+                         1: {"name": "(pid 28) pagedaemon/dom0 (on CPU 4)",
+                             "bt": [0xffffffff81057988, 0xffffffff81057949,
+                                    0xffffffff8108a5ff, 0xffffffff81062537,
+                                    0xffffffff8107171e, 0xffffffff81075f9c,
+                                    0xffffffff80f4359e, 0xffffffff80f494b4,
+                                    0xffffffff80f47430, 0xffffffff80f46eee,
+                                    0xffffffff80bc7c5e,
+                                    ],
+                             "regs": {"rbx": "0xfffffe00008e2f30",
+                                      "rbp": "0xfffffe00008e2e00",
+                                      "rsp": "0xfffffe00008e2de8",
+                                      "r12": "0xfffff80003845000",
+                                      "r13": "0x00000000027be000",
+                                      "r14": "0x0000000000000004",
+                                      "r15": "0xfffffe00458c2700",
+                                      "rip": "0xffffffff81057988",
+                                      },
+                             },
+                         2: {"name": "(pid 28) pagedaemon/laundry: dom0",
+                             "bt": [0xffffffff80c3c8c8, 0xffffffff80c16521,
+                                    0xffffffff80c15c3b, 0xffffffff80f48dfc,
+                                    0xffffffff80bc7c5e,
+                                    ],
+                             "regs": {"rbx": "0x000000007fff25f1",
+                                      "rbp": "0xfffffe00527dd890",
+                                      "rsp": "0xfffffe00527dd7c8",
+                                      "r12": "0xfffffe0045b13100",
+                                      "r13": "0x0000000000000104",
+                                      "r14": "0xfffffe00008d70c0",
+                                      "r15": "0xfffffe00009f5e00",
+                                      "rip": "0xffffffff80c3c8c8",
+                                      },
+                             },
                          })
 
     def test_arm64_minidump(self):
         self.do_test("kernel-arm64.yaml", "vmcore-arm64-minidump.bz2",
-                     [0xffff0000004b6e78],  # TODO: fix unwinding
-                      {"x0": "0x0000000000000000",
+                     hz=1000,
+                     numthread=10,
+                     threads={
+                         0: {"name": "(pid 939) sysctl (crashed)",
+                             # TODO: fix unwinding
+                             "bt": [0xffff0000004b6e78,
+                                    ],
+                             "regs": {"x0": "0x0000000000000000",
                                       "x1": "0x0000000000000000",
                                       "x2": "0x0000000000000000",
                                       "x3": "0x0000000000000000",
                                       "x4": "0x0000000000000000",
                                       "x5": "0x0000000000000000",
                                       "x6": "0x0000000000000000",
                                       "x7": "0x0000000000000000",
                                       "x8": "0xffffa00001548700",
                                       "x9": "0x0000000000000000",
                                       "x10": "0xffffa00000e04580",
                                       "x11": "0x0000000000000000",
                                       "x12": "0x000000000008950a",
                                       "x13": "0x0000000000089500",
                                       "x14": "0x0000000000000039",
                                       "x15": "0x0000000000000000",
                                       "x16": "0x00000000ffffffd8",
                                       "x17": "0x0000000000000000",
                                       "x18": "0xffff000000e6d380",
                                       "x19": "0xffff000000af9000",
                                       "x20": "0xffff000000b82000",
                                       "x21": "0xffffa00000319da8",
                                       "x22": "0xffff000000b84000",
                                       "x23": "0xffff000000b84000",
                                       "x24": "0xffff000000b55000",
                                       "x25": "0x0000000000000000",
                                       "x26": "0x0000000000040800",
                                       "x27": "0x0000000000000000",
                                       "x28": "0x00000000002019ca",
                                       "fp": "0xffff0000d58f23b0",
                                       "sp": "0xffff0000d58f23b0",
                                       "pc": "0xffff0000004b6e78",
                                       },
-                     hz_value=1000)
+                             },
+                         1: {"name": "(pid 21) syncer (on CPU 6)",
+                             # TODO: fix unwinding
+                             "bt": [0xffff000000811370,
+                                    ],
+                             "regs": {"x0": "0x0000000000000000",
+                                      "x1": "0x0000000000000000",
+                                      "x2": "0x0000000000000000",
+                                      "x3": "0x0000000000000000",
+                                      "x4": "0x0000000000000000",
+                                      "x5": "0x0000000000000000",
+                                      "x6": "0x0000000000000000",
+                                      "x7": "0x0000000000000000",
+                                      "x8": "0x0000000000000006",
+                                      "x9": "0x0000000000000560",
+                                      "x10": "0xffff000000e8f640",
+                                      "x11": "0x0000000000000001",
+                                      "x12": "0x0000000056000000",
+                                      "x13": "0x0000000000002af8",
+                                      "x14": "0x0000000000002710",
+                                      "x15": "0x0000000000000002",
+                                      "x16": "0x00000000ffffffff",
+                                      "x17": "0x0000000000000002",
+                                      "x18": "0xffff000000e6db80",
+                                      "x19": "0x0000000000000006",
+                                      "x20": "0xffff0000853a3670",
+                                      "x21": "0xffff0000009279c1",
+                                      "x22": "0x0000000000000804",
+                                      "x23": "0x0000000000000004",
+                                      "x24": "0xffff000082a93000",
+                                      "x25": "0xffffa0000053f080",
+                                      "x26": "0xffff000000e6391c",
+                                      "x27": "0xffff000000e63000",
+                                      "x28": "0x0000000000000004",
+                                      "fp": "0xffff0000853a35c0",
+                                      "sp": "0xffff0000853a35c0",
+                                      "pc": "0xffff000000811370",
+                                      },
+                             },
+                         4: {"name": "(pid 11) idle/idle: cpu2",
+                             # TODO: fix unwinding
+                             "bt": [0xffff0000004ee99c,
+                                    ],
+                             "regs": {"x0": "0x0000000000000000",
+                                      "x1": "0x0000000000000000",
+                                      "x2": "0x0000000000000000",
+                                      "x3": "0x0000000000000000",
+                                      "x4": "0x0000000000000000",
+                                      "x5": "0x0000000000000000",
+                                      "x6": "0x0000000000000000",
+                                      "x7": "0x0000000000000000",
+                                      "x8": "0x00000000ffffffff",
+                                      "x9": "0x0000000000000001",
+                                      "x10": "0x0000000000002710",
+                                      "x11": "0x000000007ff7e333",
+                                      "x12": "0x000000007ff7ba9c",
+                                      "x13": "0x0000000000002af8",
+                                      "x14": "0x0000000000002897",
+                                      "x15": "0x0000000000002af8",
+                                      "x16": "0x00000000000028e1",
+                                      "x17": "0x00000000ffffffff",
+                                      "x18": "0xffff000000e6d380",
+                                      "x19": "0xffffa0000032e580",
+                                      "x20": "0xffff000000b82000",
+                                      "x21": "0xffff000040517100",
+                                      "x22": "0xffffa00000e04580",
+                                      "x23": "0xffff000000b84000",
+                                      "x24": "0x0000000000000001",
+                                      "x25": "0xffff000000dd1000",
+                                      "x26": "0xffff000082783898",
+                                      "x27": "0xffff000000e26000",
+                                      "x28": "0xffff000000b82000",
+                                      "fp": "0xffff0000827835f0",
+                                      "sp": "0xffff000082783570",
+                                      "pc": "0xffff0000004ee99c",
+                                      },
+                             },
+                         })
 
     def test_i386_minidump(self):
         self.do_test("kernel-i386.yaml", "vmcore-i386-minidump.bz2",
-                     [0x010025c5, 0x01002410, 0x010027d5, 0x01002644,
-                      0x01049a2f, 0x01011077, 0x01010780, 0x01010c7a,
-                      0x01010ab2, 0x013e9e2d, 0xffc033f9],
-                     {"ebp": "0x151968e4",
+                     numthread=13,
+                     threads={
+                         0: {"name": "(pid 806) sysctl (crashed)",
+                             "bt": [0x010025c5, 0x01002410,
+                                    0x010027d5, 0x01002644,
+                                    0x01049a2f, 0x01011077,
+                                    0x01010780, 0x01010c7a,
+                                    0x01010ab2, 0x013e9e2d,
+                                    0xffc033f9,
+                                    ],
+                             "regs": {"ebp": "0x151968e4",
                                       "esp": "0x151968d8",
                                       "esi": "0x0c77aa80",
                                       "edi": "0x03f0dc80",
                                       "eip": "0x010025c5",
+                                      },
+                             },
+                         1: {"name": "(pid 11) idle/idle: cpu0 (on CPU 0)",
+                             "bt": [0x013a91f6, 0x013a91c0,
+                                    0x013e8ce4, 0xffc0319f,
+                                    0x00000028,
+                                    ],
+                             "regs": {"ebp": "0x03d979bc",
+                                      "esp": "0x03d979a0",
+                                      "esi": "0x000007f7",
+                                      "edi": "0x00000000",
+                                      "eip": "0x013a91f6",
+                                      },
+                             },
+                         12: {"name": "(pid 11) idle/idle: cpu11",
+                             "bt": [0x0103012c, 0x0100de0e,
+                                    0x0100b770, 0x010323be,
+                                    0x00fc50b6,
+                                    ],
+                             "regs": {"ebp": "0x03dc4af0",
+                                      "esp": "0x03dc4aa4",
+                                      "esi": "0x03f97e00",
+                                      "edi": "0x000003e8",
+                                      "eip": "0x0103012c",
+                                      },
+                             },
                          })

lldb/test/API/functionalities/postmortem/FreeBSDKernel/kernel-amd64.yaml

 --- !ELF
 FileHeader:
   Class:           ELFCLASS64
   Data:            ELFDATA2LSB
   OSABI:           ELFOSABI_FREEBSD
   Type:            ET_EXEC
   Machine:         EM_X86_64
   Entry:           0xFFFFFFFF8037C000
 Sections:
   - Name:            .bss
     Type:            SHT_NOBITS
     Flags:           [ SHF_WRITE, SHF_ALLOC ]
     Address:         0xFFFFFFFF819BA380
     AddressAlign:    0x80
     Offset:          0x17BA348
     Size:            0x445C80
+  - Name:            .rodata
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_MERGE, SHF_STRINGS ]
+    Address:         0xFFFFFFFF81152D30
+    AddressAlign:    0x10
+    Size:            0x800
 Symbols:
   - Name:            kernbase
     Index:           SHN_ABS
     Binding:         STB_GLOBAL
     Value:           0xFFFFFFFF80000000
   - Name:            KPML4phys
     Type:            STT_OBJECT
     Section:         .bss
     Binding:         STB_GLOBAL
     Value:           0xFFFFFFFF81D47EB8
     Size:            0x8
   - Name:            dumppcb
     Type:            STT_OBJECT
     Section:         .bss
     Value:           0xFFFFFFFF81CA6868
     Size:            0x140
   - Name:            hz
     Type:            STT_OBJECT
     Section:         .bss
     Binding:         STB_GLOBAL
     Value:           0xFFFFFFFF81CD4C0C
     Size:            0x4
+  - Name:            proc_off_p_comm
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA624
+    Size:            0x4
+  - Name:            proc_off_p_hash
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA62C
+    Size:            0x4
+  - Name:            proc_off_p_list
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA628
+    Size:            0x4
+  - Name:            proc_off_p_pid
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA620
+    Size:            0x4
+  - Name:            proc_off_p_threads
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA630
+    Size:            0x4
+  - Name:            thread_off_td_name
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA638
+    Size:            0x4
+  - Name:            thread_off_td_oncpu
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA63C
+    Size:            0x4
+  - Name:            thread_off_td_pcb
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA640
+    Size:            0x4
+  - Name:            thread_off_td_plist
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA644
+    Size:            0x4
+  - Name:            thread_off_td_tid
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA634
+    Size:            0x4
+  - Name:            dumptid
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF81CA69A8
+    Size:            0x4
+  - Name:            pcb_size
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF815CA590
+    Size:            0x4
+  - Name:            stoppcbs
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF81D23E20
+    Size:            0x14000
+  - Name:            allproc
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0xFFFFFFFF81C9A2F0
+    Size:            0x8

lldb/test/API/functionalities/postmortem/FreeBSDKernel/kernel-arm64.yaml

 --- !ELF
 FileHeader:
   Class:           ELFCLASS64
   Data:            ELFDATA2LSB
   Type:            ET_EXEC
   Machine:         EM_AARCH64
   Entry:           0xFFFF000000000800
 Sections:
   - Name:            .bss
     Type:            SHT_NOBITS
     Flags:           [ SHF_WRITE, SHF_ALLOC ]
     Address:         0xFFFF000000C35000
     AddressAlign:    0x1000
     Size:            0x37F000
+  - Name:            .rodata
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_MERGE, SHF_STRINGS ]
+    Address:         0xFFFF0000008A72C0
+    AddressAlign:    0x20
+    Size:            0x1000
 Symbols:
   - Name:            kernbase
     Index:           SHN_ABS
     Binding:         STB_GLOBAL
     Value:           0xFFFF000000000000
   - Name:            dumppcb
     Type:            STT_OBJECT
     Section:         .bss
     Value:           0xFFFF000000DF3790
     Size:            0x560
   - Name:            hz
     Type:            STT_OBJECT
     Section:         .bss
     Binding:         STB_GLOBAL
     Value:           0xFFFF000000E2651C
     Size:            0x4
+  - Name:            proc_off_p_comm
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC30
+    Size:            0x4
+  - Name:            proc_off_p_hash
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC38
+    Size:            0x4
+  - Name:            proc_off_p_list
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC34
+    Size:            0x4
+  - Name:            proc_off_p_pid
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC2C
+    Size:            0x4
+  - Name:            proc_off_p_threads
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC3C
+    Size:            0x4
+  - Name:            thread_off_td_name
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC44
+    Size:            0x4
+  - Name:            thread_off_td_oncpu
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC48
+    Size:            0x4
+  - Name:            thread_off_td_pcb
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC4C
+    Size:            0x4
+  - Name:            thread_off_td_plist
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC50
+    Size:            0x4
+  - Name:            thread_off_td_tid
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC40
+    Size:            0x4
+  - Name:            dumptid
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF000000DF3CF0
+    Size:            0x4
+  - Name:            pcb_size
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF00000096BC0C
+    Size:            0x4
+  - Name:            stoppcbs
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF000000E8F640
+    Size:            0x56000
+  - Name:            allproc
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0xFFFF000000DE7230
+    Size:            0x8

lldb/test/API/functionalities/postmortem/FreeBSDKernel/kernel-i386.yaml

 --- !ELF
 FileHeader:
   Class:           ELFCLASS32
   Data:            ELFDATA2LSB
   OSABI:           ELFOSABI_FREEBSD
   Type:            ET_EXEC
   Machine:         EM_386
   Entry:           0x8F9000
 Sections:
   - Name:            .bss
     Type:            SHT_NOBITS
     Flags:           [ SHF_WRITE, SHF_ALLOC ]
     Address:         0x1AB7B00
     AddressAlign:    0x80
     Offset:          0x12B7AB0
     Size:            0x2D48D8
+  - Name:            .rodata
+    Type:            SHT_PROGBITS
+    Flags:           [ SHF_ALLOC, SHF_MERGE, SHF_STRINGS ]
+    Address:         0x1400290
+    AddressAlign:    0x10
+    Size:            0x800
 Symbols:
   - Name:            kernbase
     Index:           SHN_ABS
     Binding:         STB_GLOBAL
     Value:           0x800000
   - Name:            dumppcb
     Type:            STT_OBJECT
     Section:         .bss
     Value:           0x1D2D9A0
     Size:            0xC0
   - Name:            hz
     Type:            STT_OBJECT
     Section:         .bss
     Binding:         STB_GLOBAL
     Value:           0x1D4053C
     Size:            0x4
   - Name:            IdlePDPT
     Type:            STT_OBJECT
     Section:         .bss
     Binding:         STB_GLOBAL
     Value:           0x1D8B044
     Size:            0x4
+  - Name:            proc_off_p_comm
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809ABC
+    Size:            0x4
+  - Name:            proc_off_p_hash
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AC4
+    Size:            0x4
+  - Name:            proc_off_p_list
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AC0
+    Size:            0x4
+  - Name:            proc_off_p_pid
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AB8
+    Size:            0x4
+  - Name:            proc_off_p_threads
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AC8
+    Size:            0x4
+  - Name:            thread_off_td_name
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AD0
+    Size:            0x4
+  - Name:            thread_off_td_oncpu
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AD4
+    Size:            0x4
+  - Name:            thread_off_td_pcb
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809AD8
+    Size:            0x4
+  - Name:            thread_off_td_plist
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809ADC
+    Size:            0x4
+  - Name:            thread_off_td_tid
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809ACC
+    Size:            0x4
+  - Name:            dumptid
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0x1D2DA60
+    Size:            0x4
+  - Name:            pcb_size
+    Type:            STT_OBJECT
+    Section:         .rodata
+    Binding:         STB_GLOBAL
+    Value:           0x1809A74
+    Size:            0x4
+  - Name:            stoppcbs
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0x1D651A4
+    Size:            0x1800
+  - Name:            allproc
+    Type:            STT_OBJECT
+    Section:         .bss
+    Binding:         STB_GLOBAL
+    Value:           0x1D21540
+    Size:            0x4

lldb/test/API/functionalities/postmortem/FreeBSDKernel/tools/README.rst

 How to create vmcores for tests
 ===============================
 
 1. Boot a FreeBSD VM with as little memory as possible and create a core dump
    per `FreeBSD Handbook Kernel Debugging Chapter`_.  Note that you may need to
    reboot with more memory after the kernel panic as otherwise savecore(8) may
    fail.
 
    For instance, I was able to boot FreeBSD and qemu-system-x86_64 with 128 MiB
    RAM but had to increase it to 256 MiB for the boot after kernel panic.
 
 2. Transfer the kernel image (``/boot/kernel/kernel``) and vmcore
    (``/var/crash/vmcore.latest``) from the VM.
 
-3. Patch libfbsdvmcore using ``libfbsdvmcore-print-offsets.patch`` and build
-   LLDB against the patched library.
+3. Patch libfbsdvmcore using ``libfbsdvmcore-hacks.patch`` and build LLDB
+   against the patched library.
 
-4. Do a test run of ``test.script`` in LLDB against the kernel + vmcore::
+4. Patch LLDB using ``lldb-minimize-processes.patch`` and build it.
+
+   WARNING: LLDB will now modify core files in order to make the resulting
+   test vmcores smaller.  Make a backup if necessary.
+
+5. Do a test run of ``test.script`` in LLDB against the kernel + vmcore::
 
     lldb -b -s test.script --core /path/to/core /path/to/kernel
 
    If everything works fine, the LLDB output should be interspersed with
-   ``%RD`` lines.
+   ``%RD`` lines.  The vmcore will also be modified to shorten the process
+   list in ``allproc``.
+
+6. Open the vmcore in the patched LLDB again and choose interesting threads
+   for testing.  Update thread numbers in ``test.script`` if necessary.
 
-5. Use the ``copy-sparse.py`` tool to create a sparse version of the vmcore::
+7. Use the ``copy-sparse.py`` tool to create a sparse version of the vmcore::
 
        lldb -b -s test.script --core /path/to/core /path/to/kernel |
            grep '^% RD' | python copy-sparse.py /path/to/core vmcore.sparse
 
-6. Compress the sparse vmcore file using ``bzip2``::
+8. Compress the sparse vmcore file using ``bzip2``::
 
        bzip2 -9 vmcore.sparse
 
 
 .. _FreeBSD Handbook Kernel Debugging Chapter:
    https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/

lldb/test/API/functionalities/postmortem/FreeBSDKernel/tools/copy-sparse.py

def main():
     line_re = re.compile(r"^% RD: (\d+) (\d+)")
 
     # copy the first chunk that usually includes ELF headers
     # (not output by patched libfbsdvmcore since libelf reads this)
     outf.write(inf.read(1024))
 
     for l in sys.stdin:
         m = line_re.match(l)
+        if m is None:
+            continue
         offset, size = [int(x) for x in m.groups()]
 
         inf.seek(offset)
         outf.seek(offset)
         outf.write(inf.read(size))
 
 
 if __name__ == "__main__":
     main()

lldb/test/API/functionalities/postmortem/FreeBSDKernel/tools/libfbsdvmcore-hacks.patch

+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 53b401a..ccb289e 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -94,6 +94,7 @@ install(
+     man/fvc_kerndisp.3
+     man/fvc_open.3
+     man/fvc_read.3
++    man/fvc_write.3
+     DESTINATION "${CMAKE_INSTALL_MANDIR}/man3")
+ 
+ install(
 diff --git a/lib/fvc.c b/lib/fvc.c
-index e6b96c1..4033f78 100644
+index e6b96c1..cac7158 100644
 --- a/lib/fvc.c
 +++ b/lib/fvc.c
-@@ -297,6 +297,7 @@ fvc_read(fvc_t *kd, fvc_addr_t kva, void *buf, size_t len)
+@@ -154,7 +154,7 @@ _fvc_open(fvc_t *kd, const char *uf, const char *mf, char *errout)
+    goto failed;
+  }
+ 
+- if ((kd->pmfd = open(mf, O_RDONLY | O_CLOEXEC, 0)) < 0) {
++ if ((kd->pmfd = open(mf, O_RDWR | O_CLOEXEC, 0)) < 0) {
+    _fvc_syserr(kd, kd->program, "%s", mf);
+    goto failed;
+  }
+@@ -297,6 +297,47 @@ fvc_read(fvc_t *kd, fvc_addr_t kva, void *buf, size_t len)
       _fvc_syserr(kd, kd->program, "fvc_read");
       break;
     }
 +   printf("%% RD: %zu %d\n", pa, cc);
++   /*
++    * If ka_kvatop returns a bogus value or our core file is
++    * truncated, we might wind up seeking beyond the end of the
++    * core file in which case the read will return 0 (EOF).
++    */
++   if (cr == 0)
++     break;
++   cp += cr;
++   kva += cr;
++   len -= cr;
++ }
++
++ return (cp - (char *)buf);
++}
++
++ssize_t
++fvc_write(fvc_t *kd, fvc_addr_t kva, const void *buf, size_t len)
++{
++ int cc;
++ ssize_t cr;
++ off_t pa;
++ const char *cp;
++
++ cp = buf;
++ while (len > 0) {
++   cc = kd->arch->ka_kvatop(kd, kva, &pa);
++   if (cc == 0)
++     return (-1);
++   if (cc > (ssize_t)len)
++     cc = len;
++   errno = 0;
++   if (lseek(kd->pmfd, pa, 0) == -1 && errno != 0) {
++     _fvc_syserr(kd, 0, _PATH_MEM);
++     break;
++   }
++   cr = write(kd->pmfd, cp, cc);
++   if (cr < 0) {
++     _fvc_syserr(kd, kd->program, "fvc_write");
++     break;
++   }
     /*
      * If ka_kvatop returns a bogus value or our core file is
      * truncated, we might wind up seeking beyond the end of the
-@@ -331,3 +332,8 @@ fvc_kerndisp(fvc_t *kd)
+@@ -331,3 +372,8 @@ fvc_kerndisp(fvc_t *kd)
  
   return (kd->arch->ka_kerndisp(kd));
  }
 +
 +ssize_t xpread(int fd, void *buf, size_t count, off_t offset) {
 + printf("%% RD: %zu %zu\n", offset, count);
 + return pread(fd, buf, count, offset);
 +}
 diff --git a/lib/fvc.h b/lib/fvc.h
-index 8680079..ff1e0f0 100644
+index 8680079..8cff17c 100644
 --- a/lib/fvc.h
 +++ b/lib/fvc.h
 @@ -54,6 +54,8 @@ typedef unsigned char fvc_vm_prot_t;
  #define  FVC_VM_PROT_WRITE   ((fvc_vm_prot_t) 0x02)
  #define  FVC_VM_PROT_EXECUTE   ((fvc_vm_prot_t) 0x04)
  
 +ssize_t xpread(int fd, void *buf, size_t count, off_t offset);
 +
  struct fvc_page {
   unsigned int  kp_version;
   fvc_addr_t  kp_paddr;
+@@ -76,6 +78,7 @@ fvc_t  *fvc_open
+      (const char *, const char *, char *,
+      int (*)(const char *, fvc_addr_t *, void *), void *);
+ ssize_t    fvc_read(fvc_t *, fvc_addr_t, void *, size_t);
++ssize_t    fvc_write(fvc_t *, fvc_addr_t, const void *, size_t);
+ ssize_t   fvc_kerndisp(fvc_t *);
+ 
+ typedef int fvc_walk_pages_cb_t(struct fvc_page *, void *);
 diff --git a/lib/fvc_amd64.c b/lib/fvc_amd64.c
 index 4d27998..69f1807 100644
 --- a/lib/fvc_amd64.c
 +++ b/lib/fvc_amd64.c
 @@ -205,7 +205,7 @@ _amd64_vatop(fvc_t *kd, fvc_addr_t va, off_t *pa)
     _fvc_err(kd, kd->program, "_amd64_vatop: pdpe_pa not found");
     goto invalid;
   }
@@ -309,7 +316,7 @@ _fvc_pt_init(fvc_t *kd, size_t dump_avail_size, off_t dump_avail_off,
         map_len);
     return (-1);
   }
 - rd = pread(kd->pmfd, kd->pt_map, map_len, map_off);
 + rd = xpread(kd->pmfd, kd->pt_map, map_len, map_off);
   if (rd < 0 || rd != (ssize_t)map_len) {
     _fvc_err(kd, kd->program, "cannot read %zu bytes for bitmap",
         map_len);
+diff --git a/man/fbsdvmcore.3 b/man/fbsdvmcore.3
+index 4285ba2..c0d760c 100644
+--- a/man/fbsdvmcore.3
++++ b/man/fbsdvmcore.3
+@@ -89,4 +89,5 @@ etc.
+ .Xr fvc_geterr 3 ,
+ .Xr fvc_kerndisp 3 ,
+ .Xr fvc_open 3 ,
+-.Xr fvc_read 3
++.Xr fvc_read 3 ,
++.Xr fvc_write 3
+diff --git a/man/fvc_geterr.3 b/man/fvc_geterr.3
+index 964a097..7d74c25 100644
+--- a/man/fvc_geterr.3
++++ b/man/fvc_geterr.3
+@@ -66,7 +66,8 @@ or an error has not been captured for
+ .Sh SEE ALSO
+ .Xr fvc 3 ,
+ .Xr fvc_close 3 ,
+-.Xr fvc_read 3
++.Xr fvc_read 3 ,
++.Xr fvc_write 3
+ .Sh BUGS
+ This routine cannot be used to access error conditions due to a failed
+ .Fn fvc_open
+diff --git a/man/fvc_open.3 b/man/fvc_open.3
+index 1f8e3be..4ea93ed 100644
+--- a/man/fvc_open.3
++++ b/man/fvc_open.3
+@@ -166,5 +166,6 @@ was
+ .Xr fvc_geterr 3 ,
+ .Xr fvc_native 3 ,
+ .Xr fvc_read 3 ,
++.Xr fvc_write 3 ,
+ .Xr kmem 4 ,
+ .Xr mem 4
+diff --git a/man/fvc_read.3 b/man/fvc_read.3
+index 7413d59..c18dadc 100644
+--- a/man/fvc_read.3
++++ b/man/fvc_read.3
+@@ -36,18 +36,24 @@
+ .Dt FVC_READ 3
+ .Os
+ .Sh NAME
+-.Nm fvc_read
+-.Nd read kernel virtual memory
++.Nm fvc_read ,
++.Nm fvc_write
++.Nd read or write kernel virtual memory
+ .Sh LIBRARY
+ .Lb libfbsdvmcore
+ .Sh SYNOPSIS
+ .In fvc.h
+ .Ft ssize_t
+ .Fn fvc_read "fvc_t *kd" "kvaddr_t addr" "void *buf" "size_t nbytes"
++.Ft ssize_t
++.Fn fvc_write "fvc_t *kd" "kvaddr_t addr" "void *buf" "size_t nbytes"
+ .Sh DESCRIPTION
+ The
+ .Fn fvc_read
+ function is used to read a crash dump file.
++.Fn fvc_write
++function is used to overwrite parts of a crash dump file.
++Note that only the fragments already present can be written.
+ See
+ .Fn fvc_open 3
+ for information regarding opening kernel crash dumps.
+@@ -63,6 +69,13 @@ to
+ .Fa buf .
+ .Pp
+ The
++.Fn fvc_write
++function transfers
++.Fa nbytes
++bytes of data from
++.Fa buf
++to the kernel space address
++.Fa addr .
+ .Sh RETURN VALUES
+ Upon success, the number of bytes actually transferred is returned.
+ Otherwise, -1 is returned.
+diff --git a/man/fvc_write.3 b/man/fvc_write.3
+new file mode 100644
+index 0000000..f25fc74
+--- /dev/null
++++ b/man/fvc_write.3
+@@ -0,0 +1 @@
++.so man3/fvc_read.3

lldb/test/API/functionalities/postmortem/FreeBSDKernel/tools/lldb-minimize-processes.patch

+diff --git a/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp b/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp
+index e3707365a9c3..c4a9c82f3c63 100644
+--- a/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp
++++ b/lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp
+@@ -38,6 +38,8 @@ public:
+ 
+   size_t DoReadMemory(lldb::addr_t addr, void *buf, size_t size,
+                       lldb_private::Status &error) override;
++  size_t DoWriteMemory(lldb::addr_t vm_addr, const void *buf,
++                       size_t size, Status &error) override;
+ 
+ private:
+   fvc_t *m_fvc;
+@@ -185,6 +187,7 @@ bool ProcessFreeBSDKernel::DoUpdateThreadList(ThreadList &old_thread_list,
+     // iterate through a linked list of all processes
+     // allproc is a pointer to the first list element, p_list field
+     // (found at offset_p_list) specifies the next element
++    lldb::addr_t prev = 0;
+     for (lldb::addr_t proc =
+              ReadPointerFromMemory(FindSymbol("allproc"), error);
+          proc != 0 && proc != LLDB_INVALID_ADDRESS;
+@@ -195,6 +198,8 @@ bool ProcessFreeBSDKernel::DoUpdateThreadList(ThreadList &old_thread_list,
+       char comm[fbsd_maxcomlen + 1];
+       ReadCStringFromMemory(proc + offset_p_comm, comm, sizeof(comm), error);
+ 
++      bool interesting = false;
++
+       // iterate through a linked list of all process' threads
+       // the initial thread is found in process' p_threads, subsequent
+       // elements are linked via td_plist field
+@@ -231,6 +236,7 @@ bool ProcessFreeBSDKernel::DoUpdateThreadList(ThreadList &old_thread_list,
+           // NB: dumppcb can be LLDB_INVALID_ADDRESS if reading it failed
+           pcb_addr = dumppcb;
+           thread_desc += " (crashed)";
++          interesting = true;
+         } else if (oncpu != -1) {
+           // if we managed to read stoppcbs and pcb_size, use them to find
+           // the correct PCB
+@@ -239,13 +245,27 @@ bool ProcessFreeBSDKernel::DoUpdateThreadList(ThreadList &old_thread_list,
+           else
+             pcb_addr = LLDB_INVALID_ADDRESS;
+           thread_desc += llvm::formatv(" (on CPU {0})", oncpu);
++          interesting = true;
+         }
+ 
+         ThreadSP thread_sp{
+             new ThreadFreeBSDKernel(*this, tid, pcb_addr, thread_desc)};
+         new_thread_list.AddThread(thread_sp);
+       }
++
++      if (interesting) {
++        printf("pid %d is interesting\n", pid);
++        if (prev != 0) {
++          printf("will link %d to %d\n", prev, proc);
++          if (!WritePointerToMemory(prev + offset_p_list, proc, error))
++            assert(0 && "write failed");
++        }
++        prev = proc;
++      }
+     }
++    printf("last: %d\n", prev);
++    if (!WritePointerToMemory(prev + offset_p_list, 0, error))
++      assert(0 && "write failed");
+   } else {
+     const uint32_t num_threads = old_thread_list.GetSize(false);
+     for (uint32_t i = 0; i < num_threads; ++i)
+@@ -295,6 +315,18 @@ size_t ProcessFreeBSDKernelFVC::DoReadMemory(lldb::addr_t addr, void *buf,
+   return rd;
+ }
+ 
++size_t ProcessFreeBSDKernelFVC::DoWriteMemory(lldb::addr_t vm_addr, const void *buf,
++                       size_t size, Status &error) {
++  ssize_t rd = 0;
++  rd = fvc_write(m_fvc, vm_addr, buf, size);
++  printf("fvc_write(%p, %p, %d) -> %d\n", vm_addr, buf, size, rd);
++  if (rd < 0 || static_cast<size_t>(rd) != size) {
++    error.SetErrorStringWithFormat("Writing memory failed: %s", GetError());
++    return rd > 0 ? rd : 0;
++  }
++  return rd;
++}
++
+ const char *ProcessFreeBSDKernelFVC::GetError() { return fvc_geterr(m_fvc); }
+ 
+ #endif // LLDB_ENABLE_FBSDVMCORE

lldb/test/API/functionalities/postmortem/FreeBSDKernel/tools/test.script

 thread list
-register read pc
+register read --all
+bt
+thread select 2
+register read --all
+bt
+thread select 3
+register read --all
 bt
 p *(int*)&hz
 memory read &hz