Write the # of addressable bits into Mach-O corefiles, read it back out again

[jasonmolenda]

Summary

On systems where we have an address mask, add support to process save-core to note the number of bits used for addressing in the corefile in an LC_NOTE, and look for that value in a corefile and set the mask in the Process.

In a live debug session on Apple Silicon using arm64e, debugserver reports the number of bits used for addressing. Corefiles of those same processes don't currently record this information any place, making them hard to use. This is a simple patch to add the LC_NOTE and read it back in again, nothing fancy.

I added a couple of comments on the masks to note that the mask uses 1 bits to indicate bits not used in addressing, open to disagreements about this but it always seemed backwards to me (I know we're following what the Linux kernel provides) and I will confuse myself if I'm not careful. Basically, leaving notes so Future Jason doesn't have to puzzle over the right answer quite so long, because I'm sure I'll forget this again.

[JDevlieghere]

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

5575–5613

Given that this function and the one below are basically doing the same thing, would it make sense to make this a generic helper that takes a lambda so we can share the logic to get to the LC_NOTE data?

lldb/source/Plugins/Process/mach-core/ProcessMachCore.cpp

544–545

if (addr_t address_mask = core_objfile->GetAddressMask()) {

[shafik]

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

5575–5613

I also noticed in one of the places we are doing this work we do:

memset(data_owner, 0, sizeof(data_owner));

Should we be doing this in all places? We can also do that simpler by:

char data_owner[17]{0}

[jasonmolenda]

Update patch to address Jonas' suggestion of not duplicating the code to write the LC_NOTE load command & payloads (I was annoyed by this too but didn't deal with it in my first pass). Also add a test, although it only is really tested on an arm64e testsuite run where the compiler is doing PAC auth on return addresses.

[jasonmolenda]

One more bit I meant to do -- added output to "process status --verbose" to show the addressing masks in effect. There's no other way to check what lldb is currently using for ptrauth masks currently, to debug/confirm what lldb is using.

There's the target.process.virtual-addressable-bits setting, but this is separate from what lldb might pick up dynamically (from lldb-server, debugserver, a core file) so querying that setting will show 0 if lldb is using the dynamically determined mask.

I couldn't think of any place better to put this, this will do for now.

lldb/include/lldb/Symbol/ObjectFile.h

public:
   ///
   /// \return
   ///     Returns the identifier string if one exists, else an empty
   ///     string.
   virtual std::string GetIdentifierString () {
       return std::string();
   }
 
+  /// Some object files may have the number of bits used for addressing
+  /// embedded in them, e.g. a Mach-O core file using an LC_NOTE.  These
+  /// object files can return the address mask that should be used in
+  /// the Process.
+  /// \return
+  ///     The mask will have bits set which aren't used for addressing --
+  ///     typically, the high bits.
+  ///     Zero is returned when no address bits mask is available.
+  virtual lldb::addr_t GetAddressMask() { return 0; }
+
   /// When the ObjectFile is a core file, lldb needs to locate the "binary" in
   /// the core file.  lldb can iterate over the pages looking for a valid
   /// binary, but some core files may have metadata  describing where the main
   /// binary is exactly which removes ambiguity when there are multiple
   /// binaries present in the captured memory pages.
   ///
   /// \param[out] address
   ///   If the address of the binary is specified, this will be set.

lldb/include/lldb/Target/Process.h

bool m_resume_requested; // If m_currently_handling_event or
                            // Resume will only request a resume, using this
                            // flag to check.
 
   /// This is set at the beginning of Process::Finalize() to stop functions
   /// from looking up or creating things during or after a finalize call.
   std::atomic<bool> m_finalizing;
 
   /// Mask for code an data addresses. The default value (0) means no mask is
-  /// set.
+  /// set.  The bits set to 1 indicate bits that are NOT significant for
+  /// addressing.
   /// @{
   lldb::addr_t m_code_address_mask = 0;
   lldb::addr_t m_data_address_mask = 0;
   /// @}
 
   bool m_clear_thread_plans_on_stop;
   bool m_force_next_event_delivery;
   lldb::StateType m_last_broadcast_state; /// This helps with the Public event

lldb/source/Commands/CommandObjectProcess.cpp

 #include "lldb/Target/Process.h"
 #include "lldb/Target/StopInfo.h"
 #include "lldb/Target/Target.h"
 #include "lldb/Target/Thread.h"
 #include "lldb/Target/UnixSignals.h"
 #include "lldb/Utility/Args.h"
 #include "lldb/Utility/State.h"
 
+#include <bitset>
+
 using namespace lldb;
 using namespace lldb_private;
 
 class CommandObjectProcessLaunchOrAttach : public CommandObjectParsed {
 public:
   CommandObjectProcessLaunchOrAttach(CommandInterpreter &interpreter,
                                      const char *name, const char *help,
                                      const char *syntax, uint32_t flags,
bool DoExecute(Args &command, CommandReturnObject &result) override {
     const uint32_t num_frames = 1;
     const uint32_t num_frames_with_source = 1;
     const bool stop_format = true;
     process->GetStatus(strm);
     process->GetThreadStatus(strm, only_threads_with_stop_reason, start_frame,
                              num_frames, num_frames_with_source, stop_format);
 
     if (m_options.m_verbose) {
+      addr_t code_mask = process->GetCodeAddressMask();
+      addr_t data_mask = process->GetDataAddressMask();
+      if (code_mask != 0) {
+        int bits = std::bitset<64>(~code_mask).count();
+        result.AppendMessageWithFormat(
+            "Addressable code address mask: 0x%" PRIx64 "\n", code_mask);
+        result.AppendMessageWithFormat(
+            "Addressable data address mask: 0x%" PRIx64 "\n", data_mask);
+        result.AppendMessageWithFormat(
+            "Number of bits used in addressing (code): %d\n", bits);
+      }
+
       PlatformSP platform_sp = process->GetTarget().GetPlatform();
       if (!platform_sp) {
         result.AppendError("Couldn'retrieve the target's platform");
         return result.Succeeded();
       }
 
       auto expected_crash_info =
           platform_sp->FetchExtendedCrashInformation(*process);

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.h

 #define LLDB_SOURCE_PLUGINS_OBJECTFILE_MACH_O_OBJECTFILEMACHO_H
 
 #include "lldb/Core/Address.h"
 #include "lldb/Core/FileSpecList.h"
 #include "lldb/Host/SafeMachO.h"
 #include "lldb/Symbol/ObjectFile.h"
 #include "lldb/Utility/FileSpec.h"
 #include "lldb/Utility/RangeMap.h"
+#include "lldb/Utility/StreamString.h"
 #include "lldb/Utility/UUID.h"
 
 // This class needs to be hidden as eventually belongs in a plugin that
 // will export the ObjectFile protocol
 class ObjectFileMachO : public lldb_private::ObjectFile {
 public:
   ObjectFileMachO(const lldb::ModuleSP &module_sp, lldb::DataBufferSP &data_sp,
                   lldb::offset_t data_offset,
public:
   lldb_private::Address GetEntryPointAddress() override;
 
   lldb_private::Address GetBaseAddress() override;
 
   uint32_t GetNumThreadContexts() override;
 
   std::string GetIdentifierString() override;
 
+  lldb::addr_t GetAddressMask() override;
+
   bool GetCorefileMainBinaryInfo(lldb::addr_t &address,
                                  lldb_private::UUID &uuid,
                                  ObjectFile::BinaryType &type) override;
 
   bool LoadCoreFileImages(lldb_private::Process &process) override;
 
   lldb::RegisterContextSP
   GetThreadContextAtIndex(uint32_t idx, lldb_private::Thread &thread) override;
struct MachOCorefileImageEntry {
     std::string filename;
     lldb_private::UUID uuid;
     lldb::addr_t load_address = LLDB_INVALID_ADDRESS;
     bool currently_executing;
     std::vector<std::tuple<lldb_private::ConstString, lldb::addr_t>>
         segment_load_addresses;
   };
 
+  struct LCNoteEntry {
+    LCNoteEntry(uint32_t addr_byte_size, lldb::ByteOrder byte_order)
+        : payload(lldb_private::Stream::eBinary, addr_byte_size, byte_order) {}
+
+    std::string name;
+    lldb::addr_t payload_file_offset = 0;
+    lldb_private::StreamString payload;
+  };
+
   struct MachOCorefileAllImageInfos {
     std::vector<MachOCorefileImageEntry> all_image_infos;
     bool IsValid() { return all_image_infos.size() > 0; }
   };
 
   /// Get the list of binary images that were present in the process
   /// when the corefile was produced.
   /// \return

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

 #endif
 
 #ifndef __APPLE__
 #include "Utility/UuidCompatibility.h"
 #else
 #include <uuid/uuid.h>
 #endif
 
+#include <bitset>
 #include <memory>
 
 #if LLVM_SUPPORT_XCODE_SIGNPOSTS
 // Unfortunately the signpost header pulls in the system MachO header, too.
 #undef CPU_TYPE_ARM
 #undef CPU_TYPE_ARM64
 #undef CPU_TYPE_ARM64_32
 #undef CPU_TYPE_I386
for (uint32_t i = 0; i < m_header.ncmds; ++i) {
           free(buf);
       }
       offset = cmd_offset + ident_command.cmdsize;
     }
   }
   return result;
 }
 
+addr_t ObjectFileMachO::GetAddressMask() {
+  addr_t mask = 0;
+  ModuleSP module_sp(GetModule());
+  if (module_sp) {
+    std::lock_guard<std::recursive_mutex> guard(module_sp->GetMutex());
+    lldb::offset_t offset = MachHeaderSizeFromMagic(m_header.magic);
+    for (uint32_t i = 0; i < m_header.ncmds; ++i) {
+      const uint32_t cmd_offset = offset;
+      llvm::MachO::load_command lc;
+      if (m_data.GetU32(&offset, &lc.cmd, 2) == nullptr)
+        break;
+      if (lc.cmd == LC_NOTE) {
+        char data_owner[17];
+        m_data.CopyData(offset, 16, data_owner);
+        data_owner[16] = '\0';
+        offset += 16;
+        uint64_t fileoff = m_data.GetU64_unchecked(&offset);
+
+        // "addrable bits" has a uint32_t version and a uint32_t
+        // number of bits used in addressing.
+        if (strcmp("addrable bits", data_owner) == 0) {
+          offset = fileoff;
+          uint32_t version;
+          if (m_data.GetU32(&offset, &version, 1) != nullptr) {
+            if (version == 3) {
+              uint32_t num_addr_bits = m_data.GetU32_unchecked(&offset);
+              if (num_addr_bits != 0) {
+                mask = ~((1ULL << num_addr_bits) - 1);
+              }
+              break;
+            }
+          }
+        }
+      }
+      offset = cmd_offset + lc.cmdsize;
+    }
+  }
+  return mask;
+}
+
 bool ObjectFileMachO::GetCorefileMainBinaryInfo(addr_t &address, UUID &uuid,
                                                 ObjectFile::BinaryType &type) {
   address = LLDB_INVALID_ADDRESS;
   uuid.Clear();
   ModuleSP module_sp(GetModule());
   if (module_sp) {
     std::lock_guard<std::recursive_mutex> guard(module_sp->GetMutex());
     lldb::offset_t offset = MachHeaderSizeFromMagic(m_header.magic);
if (make_core) {
         }
 
         // and the size of all LC_THREAD load command
         for (const auto &LC_THREAD_data : LC_THREAD_datas) {
           ++mach_header.ncmds;
           mach_header.sizeofcmds += 8 + LC_THREAD_data.GetSize();
         }
 
+        // Bits will be set to indicate which bits are NOT used in
+        // addressing in this process or 0 for unknown.
+        uint64_t address_mask = process_sp->GetCodeAddressMask();
+        if (address_mask != 0) {
+          // LC_NOTE "addrable bits"
+          mach_header.ncmds++;
+          mach_header.sizeofcmds += sizeof(llvm::MachO::note_command);
+        }
+
         // LC_NOTE "all image infos"
         mach_header.ncmds++;
         mach_header.sizeofcmds += sizeof(llvm::MachO::note_command);
 
         // Write the mach header
         buffer.PutHex32(mach_header.magic);
         buffer.PutHex32(mach_header.cputype);
         buffer.PutHex32(mach_header.cpusubtype);
         buffer.PutHex32(mach_header.filetype);
         buffer.PutHex32(mach_header.ncmds);
         buffer.PutHex32(mach_header.sizeofcmds);
         buffer.PutHex32(mach_header.flags);
         if (addr_byte_size == 8) {
           buffer.PutHex32(mach_header.reserved);
         }
 
         // Skip the mach header and all load commands and align to the next
         // 0x1000 byte boundary
         addr_t file_offset = buffer.GetSize() + mach_header.sizeofcmds;
 
         file_offset = llvm::alignTo(file_offset, 16);
+        std::vector<std::unique_ptr<LCNoteEntry>> lc_notes;
 
-        // Create the "all image infos" LC_NOTE payload
-        StreamString all_image_infos_payload(Stream::eBinary, addr_byte_size,
-                                             byte_order);
-        offset_t all_image_infos_payload_start = file_offset;
-        file_offset = CreateAllImageInfosPayload(process_sp, file_offset,
-                                                 all_image_infos_payload);
-
-        // Add the "all image infos" LC_NOTE load command
-        llvm::MachO::note_command all_image_info_note = {
-            LC_NOTE,                           /* uint32_t cmd */
-            sizeof(llvm::MachO::note_command), /* uint32_t cmdsize */
-            "all image infos",                 /* char data_owner[16] */
-            all_image_infos_payload_start,     /* uint64_t offset */
-            file_offset - all_image_infos_payload_start /* uint64_t size */
-        };
-        buffer.PutHex32(all_image_info_note.cmd);
-        buffer.PutHex32(all_image_info_note.cmdsize);
-        buffer.PutRawBytes(all_image_info_note.data_owner,
-                           sizeof(all_image_info_note.data_owner));
-        buffer.PutHex64(all_image_info_note.offset);
-        buffer.PutHex64(all_image_info_note.size);
+        // Add "addrable bits" LC_NOTE when an address mask is available
+        if (address_mask != 0) {
+          std::unique_ptr<LCNoteEntry> addrable_bits_lcnote_up(
+              new LCNoteEntry(addr_byte_size, byte_order));
+          addrable_bits_lcnote_up->name = "addrable bits";
+          addrable_bits_lcnote_up->payload_file_offset = file_offset;
+          int bits = std::bitset<64>(~address_mask).count();
+          addrable_bits_lcnote_up->payload.PutHex32(3); // version
+          addrable_bits_lcnote_up->payload.PutHex32(
+              bits); // # of bits used for addressing
+          addrable_bits_lcnote_up->payload.PutHex64(0); // unused
+
+          file_offset += addrable_bits_lcnote_up->payload.GetSize();
+
+          lc_notes.push_back(std::move(addrable_bits_lcnote_up));
+        }
+
+        // Add "all image infos" LC_NOTE
+        std::unique_ptr<LCNoteEntry> all_image_infos_lcnote_up(
+            new LCNoteEntry(addr_byte_size, byte_order));
+        all_image_infos_lcnote_up->name = "all image infos";
+        all_image_infos_lcnote_up->payload_file_offset = file_offset;
+        file_offset = CreateAllImageInfosPayload(
+            process_sp, file_offset, all_image_infos_lcnote_up->payload);
+        lc_notes.push_back(std::move(all_image_infos_lcnote_up));
+
+        // Add LC_NOTE load commands
+        for (auto &lcnote : lc_notes) {
+          // Add the LC_NOTE load command to the file.
+          buffer.PutHex32(LC_NOTE);
+          buffer.PutHex32(sizeof(llvm::MachO::note_command));
+          char namebuf[16];
+          memset(namebuf, 0, sizeof(namebuf));
+          // this is the uncommon case where strncpy is exactly
+          // the right one, doesn't need to be nul terminated.
+          strncpy(namebuf, lcnote->name.c_str(), sizeof(namebuf));
+          buffer.PutRawBytes(namebuf, sizeof(namebuf));
+          buffer.PutHex64(lcnote->payload_file_offset);
+          buffer.PutHex64(lcnote->payload.GetSize());
+        }
 
         // Align to 4096-byte page boundary for the LC_SEGMENTs.
         file_offset = llvm::alignTo(file_offset, 4096);
 
         for (auto &segment : segment_load_commands) {
           segment.fileoff = file_offset;
           file_offset += segment.filesize;
         }
if (make_core) {
           // Read 1 page at a time
           uint8_t bytes[0x1000];
           // Write the mach header and load commands out to the core file
           size_t bytes_written = buffer.GetString().size();
           error =
               core_file.get()->Write(buffer.GetString().data(), bytes_written);
           if (error.Success()) {
 
-            if (core_file.get()->SeekFromStart(all_image_info_note.offset) ==
+            for (auto &lcnote : lc_notes) {
+              if (core_file.get()->SeekFromStart(lcnote->payload_file_offset) ==
                   -1) {
-              error.SetErrorStringWithFormat(
-                  "Unable to seek to corefile pos to write all iamge infos");
+                error.SetErrorStringWithFormat("Unable to seek to corefile pos "
+                                               "to write '%s' LC_NOTE payload",
+                                               lcnote->name.c_str());
                 return false;
               }
-
-            bytes_written = all_image_infos_payload.GetString().size();
-            error = core_file.get()->Write(
-                all_image_infos_payload.GetString().data(), bytes_written);
+              bytes_written = lcnote->payload.GetSize();
+              error = core_file.get()->Write(lcnote->payload.GetData(),
+                                             bytes_written);
               if (!error.Success())
                 return false;
+            }
 
             // Now write the file data for all memory segments in the process
             for (const auto &segment : segment_load_commands) {
               if (core_file.get()->SeekFromStart(segment.fileoff) == -1) {
                 error.SetErrorStringWithFormat(
                     "unable to seek to offset 0x%" PRIx64 " in '%s'",
                     segment.fileoff, core_file_path.c_str());
                 break;

lldb/source/Plugins/Process/mach-core/ProcessMachCore.cpp

Status ProcessMachCore::DoLoadCore() {
   // match the core file which is always single arch.
   ArchSpec arch(m_core_module_sp->GetArchitecture());
   if (arch.GetCore() == ArchSpec::eCore_x86_32_i486) {
     arch = Platform::GetAugmentedArchSpec(GetTarget().GetPlatform().get(), "i386");
   }
   if (arch.IsValid())
     GetTarget().SetArchitecture(arch);
 
+  addr_t address_mask = core_objfile->GetAddressMask();
+  if (address_mask != 0) {
+    SetCodeAddressMask(address_mask);
+    SetDataAddressMask(address_mask);
+  }
   return error;
 }
 
 lldb_private::DynamicLoader *ProcessMachCore::GetDynamicLoader() {
   if (m_dyld_up.get() == nullptr)
     m_dyld_up.reset(DynamicLoader::FindPlugin(
         this, m_dyld_plugin_name.IsEmpty() ? nullptr
                                            : m_dyld_plugin_name.GetCString()));

lldb/test/API/macosx/lc-note/addrable-bits/Makefile

+C_SOURCES := main.c
+
+include Makefile.rules

lldb/test/API/macosx/lc-note/addrable-bits/TestAddrableBitsCorefile.py

+"""Test that corefiles with LC_NOTE "addrable bits" load command, creating and reading."""
+
+
+
+import os
+import re
+import subprocess
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class TestAddrableBitsCorefile(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+    NO_DEBUG_INFO_TESTCASE = True
+
+    def initial_setup(self):
+        self.build()
+        self.exe = self.getBuildArtifact("a.out")
+        self.corefile = self.getBuildArtifact("corefile")
+
+    @skipIf(archs=no_match(['arm64e']))
+    @skipUnlessDarwin
+    def test_lc_note_addrable_bits(self):
+        self.initial_setup()
+
+        self.target = self.dbg.CreateTarget(self.exe)
+        err = lldb.SBError()
+        (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, "break here",
+                                          lldb.SBFileSpec('main.c'))
+        self.assertEqual(process.IsValid(), True)
+
+        found_main = False
+        for f in thread.frames:
+          if f.GetFunctionName() == "main":
+            found_main = True
+        self.assertTrue(found_main)
+
+        cmdinterp = self.dbg.GetCommandInterpreter()
+        res = lldb.SBCommandReturnObject()
+        cmdinterp.HandleCommand("process save-core %s" % self.corefile, res)
+        self.assertTrue(res.Succeeded(), True)
+        process.Kill()
+        self.dbg.DeleteTarget(target)
+
+        target = self.dbg.CreateTarget('')
+        process = target.LoadCore(self.corefile)
+        self.assertTrue(process.IsValid(), True)
+        thread = process.GetSelectedThread()
+
+        found_main = False
+        for f in thread.frames:
+          if f.GetFunctionName() == "main":
+            found_main = True
+        self.assertTrue(found_main)
+

lldb/test/API/macosx/lc-note/addrable-bits/main.c

+int pat (int in) { 
+  return in + 5; // break here 
+}
+
+int tat (int in) { return pat(in + 10); }
+
+int mat (int in) { return tat(in + 15); }
+
+int main() {
+ int (*matp)(int) = mat;
+ return matp(10);
+}