[lldb] [Process/FreeBSDKernel] Support finding all processes

[mgorny]

Summary

Include the complete list of threads of all running processes
in the FreeBSDKernel plugin. This makes it possible to inspect
the states (including partial register dumps from PCB) of all kernel
and userspace threads at the time of crash, or at the time of reading
/dev/mem first.

[labath]

I think I've managed to figure out what the code is doing, but it wouldn't hurt to include a comment giving a high-level overview of what that code is doing and/or splitting it into smaller functions with helpful names.

Can we expect to have a test for this?

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp

203

sizeof(thread_name)

221

is this going to be unique (for the entire system)?

lldb/source/Plugins/Process/FreeBSDKernel/ThreadFreeBSDKernel.cpp

29

std::move

[mgorny]

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp

221

Yes, I have specifically verified that.

[mgorny]

Update the code per suggestions, update the first test case.

TODO: update the remaining tests, document the code better

[mgorny]

Add code comments

[mgorny]

Update all tests.

Phab doesn't like git binary deltas for some reason, so I've uploaded the patch with --no-binary.

New file sizes:

-rw-r--r-- 1 mgorny mgorny 120K 12-29 15:36 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-amd64-full.bz2
-rw-r--r-- 1 mgorny mgorny 132K 12-29 12:45 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-amd64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny  37K 12-29 15:45 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-arm64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny 136K 12-29 15:38 lldb/test/API/functionalities/postmortem/FreeBSDKernel/vmcore-i386-minidump.bz2

[labath]

Can we make the number of threads smaller? Perhaps by patching the linked lists to skip the uninteresting ones? 600 threads is way too much, all we really need is one thread from each of the three categories. Then you can meaningfully assert the names and registers of the existing threads (since their layout is the same, there's probably no point in asserting all registers, but it'd be good to check one or two, just to make sure the starting offset is right).

[mgorny]

That would be a lot of work, and it would need to be repeated whenever the test data needs to be regenerated. How about leaving the threads as-is but just testing the few selected ones from the list?

[labath]

That would deal with the code coverage, but it still leaves us with a fairly large core file, and a lot of uninteresting tests obscuring the output.

How do you generate these core files? Do you actually create a fresh core dump or you just recompute the "interesting" portions from a master file you have around? If you make the changes the the master core file, then they would get automatically picked up during the recomputation.

Alternatively, maybe there is a way to capture a core file without so many processes. Either killing off everything before the core file is written, or by making sure the other processes are never started (something like init=/bin/bash on linux)?

[mgorny]

In D116255#3213612, @labath wrote:

That would deal with the code coverage, but it still leaves us with a fairly large core file, and a lot of uninteresting tests obscuring the output.

How do you generate these core files? Do you actually create a fresh core dump or you just recompute the "interesting" portions from a master file you have around? If you make the changes the the master core file, then they would get automatically picked up during the recomputation.

I do recompute them from master copies (which I finally need to upload somewhere) but the recomputation is really dumb and unaware of the file format. I suppose I could just hack LLDB to stop after grabbing the first N threads but… the first non-dump on-CPU thread is no. 200. Finishing on that will probably make the core smaller but not sure how much smaller. I'll try in a minute.

Alternatively, maybe there is a way to capture a core file without so many processes. Either killing off everything before the core file is written, or by making sure the other processes are never started (something like init=/bin/bash on linux)?

The vast majority of threads are kernel threads. At a quick glance, only about 20 threads look like userspace.

[mgorny]

Ok, so limiting to 200-ish threads halves the data size:

-rw-r--r-- 1 mgorny mgorny 76K 12-30 14:16 /tmp/vmcore.bz2

[labath]

In D116255#3213665, @mgorny wrote:

In D116255#3213612, @labath wrote:

That would deal with the code coverage, but it still leaves us with a fairly large core file, and a lot of uninteresting tests obscuring the output.

How do you generate these core files? Do you actually create a fresh core dump or you just recompute the "interesting" portions from a master file you have around? If you make the changes the the master core file, then they would get automatically picked up during the recomputation.

I do recompute them from master copies (which I finally need to upload somewhere) but the recomputation is really dumb and unaware of the file format.

I am aware how the recomputation works. My point is that if you hack the "master" copy (quotes because you probably still want to keep the original master around) then you only need to redo the hacks after when you're regenerating the master. At that point you'll also need to update all the test expectations, so it doesn't seem like too much extra work. (In fact, I wouldn't be surprised if we just end up adding a new core file instead of updating the old one).

I suppose I could just hack LLDB to stop after grabbing the first N threads but… the first non-dump on-CPU thread is no. 200. Finishing on that will probably make the core smaller but not sure how much smaller. I'll try in a minute.

You may still need to do some hex editing to avoid ending the list with a dangling pointer...

Alternatively, maybe there is a way to capture a core file without so many processes. Either killing off everything before the core file is written, or by making sure the other processes are never started (something like init=/bin/bash on linux)?

The vast majority of threads are kernel threads. At a quick glance, only about 20 threads look like userspace.

yikes. And I thought linux was bad..

[mgorny]

Skip repeating thread name when it's equal to comm. Include thread status in name field (i.e. 'crashed', 'on CPU n').

Update the test tooling to strip "non-interesting" processes from allproc and to grab bt/regs from three first threads. Update amd64 test to include only interesting processes (making the core much smaller), and to test backtrace and regs on one thread of each kind.

[mgorny]

Update all tests and the test documentation.

New sizes:

-rw-r--r-- 1 mgorny mgorny  13K 12-31 15:27 vmcore-amd64-full.bz2
-rw-r--r-- 1 mgorny mgorny  18K 12-31 13:14 vmcore-amd64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny  12K 12-31 16:02 vmcore-arm64-minidump.bz2
-rw-r--r-- 1 mgorny mgorny  14K 12-31 16:31 vmcore-i386-minidump.bz2

[labath]

Thanks for doing this.

Interesting solution to the problem. I'm not sure how long will the diff files remain applicable, but I would've accepted this even without them so I think all is fine.

lldb/test/API/functionalities/postmortem/FreeBSDKernel/TestFreeBSDKernelVMCore.py

60–61 ↗

(On Diff #396789)

If you want to be even more fancy :)

[mgorny]

lldb/test/API/functionalities/postmortem/FreeBSDKernel/TestFreeBSDKernelVMCore.py

60–61 ↗

(On Diff #396789)

Neat. I didn't know threads are iterable like this. Unfortunately, swig makes it kinda hard to inspect stuff in a really Pythonic way.

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.h

public:
 
   lldb_private::Status DoLoadCore() override;
 
   lldb_private::DynamicLoader *GetDynamicLoader() override;
 
 protected:
   bool DoUpdateThreadList(lldb_private::ThreadList &old_thread_list,
                           lldb_private::ThreadList &new_thread_list) override;
+
+  lldb::addr_t FindSymbol(const char* name);
 };
 
 #endif // LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_PROCESSFREEBSDKERNEL_H

lldb/source/Plugins/Process/FreeBSDKernel/ProcessFreeBSDKernel.cpp

if (old_thread_list.GetSize(false) == 0) {
     case llvm::Triple::aarch64:
     case llvm::Triple::x86:
     case llvm::Triple::x86_64:
       break;
     default:
       return false;
     }
 
-    const Symbol *pcb_sym =
-        GetTarget().GetExecutableModule()->FindFirstSymbolWithNameAndType(
-            ConstString("dumppcb"));
+    Status error;
+
+    // struct field offsets are written as symbols so that we don't have
+    // to figure them out ourselves
+    int32_t offset_p_list = ReadSignedIntegerFromMemory(
+        FindSymbol("proc_off_p_list"), 4, -1, error);
+    int32_t offset_p_pid =
+        ReadSignedIntegerFromMemory(FindSymbol("proc_off_p_pid"), 4, -1, error);
+    int32_t offset_p_threads = ReadSignedIntegerFromMemory(
+        FindSymbol("proc_off_p_threads"), 4, -1, error);
+    int32_t offset_p_comm = ReadSignedIntegerFromMemory(
+        FindSymbol("proc_off_p_comm"), 4, -1, error);
+
+    int32_t offset_td_tid = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_tid"), 4, -1, error);
+    int32_t offset_td_plist = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_plist"), 4, -1, error);
+    int32_t offset_td_pcb = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_pcb"), 4, -1, error);
+    int32_t offset_td_oncpu = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_oncpu"), 4, -1, error);
+    int32_t offset_td_name = ReadSignedIntegerFromMemory(
+        FindSymbol("thread_off_td_name"), 4, -1, error);
+
+    // fail if we were not able to read any of the offsets
+    if (offset_p_list == -1 || offset_p_pid == -1 || offset_p_threads == -1 ||
+        offset_p_comm == -1 || offset_td_tid == -1 || offset_td_plist == -1 ||
+        offset_td_pcb == -1 || offset_td_oncpu == -1 || offset_td_name == -1)
+      return false;
+
+    int32_t dumptid =
+        ReadSignedIntegerFromMemory(FindSymbol("dumptid"), 4, -1, error);
+    lldb::addr_t dumppcb = FindSymbol("dumppcb");
+
+    int32_t pcbsize =
+        ReadSignedIntegerFromMemory(FindSymbol("pcb_size"), 4, -1, error);
+    lldb::addr_t stoppcbs = FindSymbol("stoppcbs");
+
+    // from FreeBSD sys/param.h
+    constexpr size_t fbsd_maxcomlen = 19;
+
+    for (lldb::addr_t proc =
+             ReadPointerFromMemory(FindSymbol("allproc"), error);
+         proc != 0 && proc != LLDB_INVALID_ADDRESS;
+         proc = ReadPointerFromMemory(proc + offset_p_list, error)) {
+      int32_t pid =
+          ReadSignedIntegerFromMemory(proc + offset_p_pid, 4, -1, error);
+      char comm[fbsd_maxcomlen + 1];
+      ReadCStringFromMemory(proc + offset_p_comm, comm, fbsd_maxcomlen + 1,
+                            error);
+
+      for (lldb::addr_t td =
+               ReadPointerFromMemory(proc + offset_p_threads, error);
+           td != 0; td = ReadPointerFromMemory(td + offset_td_plist, error)) {
+        int32_t tid =
+            ReadSignedIntegerFromMemory(td + offset_td_tid, 4, -1, error);
+        lldb::addr_t pcb_addr =
+            ReadPointerFromMemory(td + offset_td_pcb, error);
+        // NB: -1 is a valid value here
+        int32_t oncpu =
+            ReadSignedIntegerFromMemory(td + offset_td_oncpu, 4, -2, error);
+        char thread_name[fbsd_maxcomlen + 1];
+        ReadCStringFromMemory(td + offset_td_name, thread_name,
+                              fbsd_maxcomlen + 1, error);
+
+        if (tid == -1)
+          continue;
+        // roughly:
+        // 1. if the thread crashed, its PCB is going to be at "dumppcb"
+        // 2. if the thread was on CPU, its PCB is going to be on the CPU
+        // 3. otherwise, its PCB is in the thread struct
+        if (tid == dumptid)
+          pcb_addr = dumppcb;
+        else if (oncpu != -1) {
+          if (stoppcbs != LLDB_INVALID_ADDRESS && pcbsize > 0)
+            pcb_addr = stoppcbs + oncpu * pcbsize;
+          else
+            pcb_addr = LLDB_INVALID_ADDRESS;
+        }
+
         ThreadSP thread_sp(new ThreadFreeBSDKernel(
-        *this, 1, pcb_sym ? pcb_sym->GetFileAddress() : LLDB_INVALID_ADDRESS));
+            *this, tid, pcb_addr,
+            llvm::formatv("(pid {0}) {1}/{2}", pid, comm, thread_name)));
         new_thread_list.AddThread(thread_sp);
+      }
+    }
   } else {
     const uint32_t num_threads = old_thread_list.GetSize(false);
     for (uint32_t i = 0; i < num_threads; ++i)
       new_thread_list.AddThread(old_thread_list.GetThreadAtIndex(i, false));
   }
   return new_thread_list.GetSize(false) > 0;
 }
 
 Status ProcessFreeBSDKernel::DoLoadCore() {
   // The core is already loaded by CreateInstance().
   return Status();
 }
 
 DynamicLoader *ProcessFreeBSDKernel::GetDynamicLoader() {
   if (m_dyld_up.get() == nullptr)
     m_dyld_up.reset(DynamicLoader::FindPlugin(
         this, DynamicLoaderStatic::GetPluginNameStatic()));
   return m_dyld_up.get();
 }
 
+lldb::addr_t ProcessFreeBSDKernel::FindSymbol(const char *name) {
+  ModuleSP mod_sp = GetTarget().GetExecutableModule();
+  const Symbol *sym = mod_sp->FindFirstSymbolWithNameAndType(ConstString(name));
+  return sym ? sym->GetLoadAddress(&GetTarget()) : LLDB_INVALID_ADDRESS;
+}
+
 #if LLDB_ENABLE_FBSDVMCORE
 
 ProcessFreeBSDKernelFVC::ProcessFreeBSDKernelFVC(lldb::TargetSP target_sp,
                                                  ListenerSP listener_sp,
                                                  fvc_t *fvc)
     : ProcessFreeBSDKernel(target_sp, listener_sp), m_fvc(fvc) {}
 
 ProcessFreeBSDKernelFVC::~ProcessFreeBSDKernelFVC() {

lldb/source/Plugins/Process/FreeBSDKernel/ThreadFreeBSDKernel.h

 //===-- ThreadFreeBSDKernel.h ------------------------------------- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_THREADFREEBSDKERNEL_H
 #define LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_THREADFREEBSDKERNEL_H
 
 #include "lldb/Target/Thread.h"
 
 class ThreadFreeBSDKernel : public lldb_private::Thread {
 public:
   ThreadFreeBSDKernel(lldb_private::Process &process, lldb::tid_t tid,
-                      lldb::addr_t pcb_addr);
+                      lldb::addr_t pcb_addr, std::string thread_name);
 
   ~ThreadFreeBSDKernel() override;
 
   void RefreshStateAfterStop() override;
 
   lldb::RegisterContextSP GetRegisterContext() override;
 
   lldb::RegisterContextSP
   CreateRegisterContextForFrame(lldb_private::StackFrame *frame) override;
 
+  const char *GetName() override {
+    if (m_thread_name.empty())
+      return nullptr;
+    return m_thread_name.c_str();
+  }
+
+  void SetName(const char *name) override {
+    if (name && name[0])
+      m_thread_name.assign(name);
+    else
+      m_thread_name.clear();
+  }
+
 protected:
   bool CalculateStopInfo() override;
 
 private:
+  std::string m_thread_name;
   lldb::RegisterContextSP m_thread_reg_ctx_sp;
   lldb::addr_t m_pcb_addr;
 };
 
 #endif // LLDB_SOURCE_PLUGINS_PROCESS_FREEBSDKERNEL_THREADFREEBSDKERNEL_H

lldb/source/Plugins/Process/FreeBSDKernel/ThreadFreeBSDKernel.cpp

 #include "RegisterContextFreeBSDKernel_i386.h"
 #include "RegisterContextFreeBSDKernel_x86_64.h"
 #include "ThreadFreeBSDKernel.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
 ThreadFreeBSDKernel::ThreadFreeBSDKernel(Process &process, lldb::tid_t tid,
-                                         lldb::addr_t pcb_addr)
-    : Thread(process, tid), m_pcb_addr(pcb_addr) {}
+                                         lldb::addr_t pcb_addr,
+                                         std::string thread_name)
+    : Thread(process, tid), m_thread_name(thread_name), m_pcb_addr(pcb_addr) {}
 
 ThreadFreeBSDKernel::~ThreadFreeBSDKernel() {}
 
 void ThreadFreeBSDKernel::RefreshStateAfterStop() {}
 
 lldb::RegisterContextSP ThreadFreeBSDKernel::GetRegisterContext() {
   if (!m_reg_context_sp)
     m_reg_context_sp = CreateRegisterContextForFrame(nullptr);
if (concrete_frame_idx == 0) {
     switch (arch.GetMachine()) {
     case llvm::Triple::aarch64:
       m_thread_reg_ctx_sp =
           std::make_shared<RegisterContextFreeBSDKernel_arm64>(
               *this, std::make_unique<RegisterInfoPOSIX_arm64>(arch, 0),
               m_pcb_addr);
       break;
     case llvm::Triple::x86:
-      m_thread_reg_ctx_sp =
-          std::make_shared<RegisterContextFreeBSDKernel_i386>(
-              *this, new RegisterContextFreeBSD_i386(arch), m_pcb_addr);
+      m_thread_reg_ctx_sp = std::make_shared<RegisterContextFreeBSDKernel_i386>(
       break;
     case llvm::Triple::x86_64:
       m_thread_reg_ctx_sp =
           std::make_shared<RegisterContextFreeBSDKernel_x86_64>(
               *this, new RegisterContextFreeBSD_x86_64(arch), m_pcb_addr);
       break;
     default:
       assert(false && "Unsupported architecture passed to ThreadFreeBSDKernel");