Set a default number of addressing bits for Darwin arm64 systems

[jasonmolenda]

Summary

For ARMv8.3 pointer auth signing, lldb needs to know how many bits are valid for addressing, and strip the high bits off of addresses before looking them up. During the initial bringup, we didn't have dynamic ways of getting these values so we hardcoded a default addressing mask in lldb, and there are still some cases where we need to support this, where we still don't have the value dynamically declared.

There are three parts to this patch, all needed to construct a test. The most important part is that when I print ptrauth values, I always print the actual uint64_t with ptrauth bits included, then I strip the bits and see if it resolves to an address that lldb knows about. If so, then I use the normal formatter. For this test case, I needed this behavior in our function pointer formatter. Our normal output might look like

(lldb) p fmain
(int (*)(...)) $0 = 0x0000000100003f90 (a.out`main at main.c:3)

and with an ARMv8.3 ABI with ptrauth bits, you might see

(lldb) p fmain
(int (*)(...)) $0 = 0xe46bff0100003f8c (0xe46bff0100003f8c)

which is not helpful. So this patch does

(lldb) p fmain
(int (*)(...)) $0 = 0xe46bff0100003f8c (actual=0x0000000100003f8c a.out`main at main.c:3)

I never want to hide the ptrauth bit reality from the user - because they may be debugging a problem with ptrauth signing itself. The three parts of the patch:

1. ABIMacOSX_arm64::FixAddress changed so if no mask is specified, uses the default mask.

2. ObjectFileMachO::GetAllArchSpecs when reading a Mach-O binary, we were previously clearing the Vendor from the Triple that we create for the binary (so it would be arm64e--). This loses information - realistically, any binary in a Mach-O container is most likely following the Darwin ABIs and I want to retain the vendor so we can select the correct ABI instead of grabbing any random arm64 ABI.

3. ArchSpec::SetArchitecture - explicitly set the vendor to Apple when we have a Mach-O architecture.

4. formatters::CXXFunctionPointerSummaryProvider the formatter change above.

The test case for this is a main.c with main() and a function pointer global to main, fmain(). Then there is a standalone program that creates this specific corefile without any metadata about the addressing bits -- lldb itself always adds this metadata, so I couldn't use lldb's built-in process save-core functionality. It's a tiny corefile that only has 8 bytes of DATA memory where the function pointer is stored. I have ideas for how I can use this test case for some upcoming changes I want to make, so I expect to get more use out of this test case.

[DavidSpickett]

Glad to see this, it will be useful on AArch64 Linux as well.

General question, is yaml2obj not flexible enough to do this? I guess the issue would be even if it does do MachO the test could only work on a precompiled file, this way allows you to build a.out each time.

I have ideas for how I can use this test case for some upcoming changes I want to make, so I expect to get more use out of this test case.

And having this program to build the corefile allows you the flexibility to do these future things you reference here?

lldb/source/DataFormatters/CXXFunctionPointer.cpp

53

You could do if (Process *process = ....

55

Remove space before the (. Unless this is a code style I'm missing.

58

Stray space ( here too.

61

Where does the closing ) come from, does the so_addr.Dump emit it?

lldb/source/Plugins/ABI/AArch64/ABIMacOSX_arm64.cpp

830

Can you explain this logic? The "else" part I get, it's the sign extension check I'm not sure of.

I have to admit, we have this over in AArch64 Linux and it's been on my list to find out what it actually does and document it there too. My naive assumption is that removing signature bits would just be a mask but clearly I'm missing something.

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

5161

You could pull that out to after the if.

if (header.filetype == MH_KEXT_BUNDLE) {
    base_triple.setVendor(llvm::Triple::Apple);
}
add_triple(base_triple);

lldb/test/API/macosx/corefile-default-ptrauth/Makefile

10

Does a.out get built here because the default testing makefile does that? Please comment that if so, otherwise it looks like you forgot to check in the prebuilt a.out.

lldb/test/API/macosx/corefile-default-ptrauth/TestCorefileDefaultPtrauth.py

51

I'd use the full commands here just for clarity.

Also given the brackets around the "actual" bit seem to be emitted by two different calls, include them in the match as well? Just in case one gets dropped at some point.

lldb/test/API/macosx/corefile-default-ptrauth/create-corefile.c

19

Is the name spelled like that or should it be "addressable"?

31

exit(1) here

56

Should you have some validity check here that the address of main and fmain is not still 0? Just in case you manage to read half way through an nm line and miss one of them.

58

What are the 4s?

60

Would help to have a comment here like:

// Data is written out in the order it will be in the file:
// Mach header
// Segment command
// Thread command
// Segment contents

78

Do I understand correctly, this points to segment_contents?

93

Why /4 not /8 here? Assuming 64 bit registers.

Is count in 32 bit words? (I found some sources on google but they didn't include the arm64 version)

100

And here you've manually put some bits in the position of the signing bits? Please comment with that.

I assume it doesn't matter if they're generated or not, lldb isn't going to try and auth it.

[DavidSpickett]

General question, is yaml2obj not flexible enough to do this? I guess the issue would be even if it does do MachO the test could only work on a precompiled file, this way allows you to build a.out each time.

Plus you get all the Mach specific constant values for free instead of writing umpteen magic numbers in a YAML file. So I think I just answered my own question here.

[jasonmolenda]

Thanks so much for the thorough review Davide, I really appreciate the help and time you spent doing this. I addressed the comments and will update the patch momentarily.

I have another small change I want to make soon where I add a new "load binary" LC_NOTE which will be a way for a corefile to tell lldb to load additional binaries for a corefile, and I'm planning to add this load binary command to the corefile and confirm that lldb loads the a.out correctly via that.

I didn't look at obj2yaml too hard, but the first thing I tried was running it on my created corefile, and it didn't include the memory contents (LC_SEGMENT_64), so I didn't pursue it much further. I knew I was going to add a custom LC_NOTE to this test corefile soon too and I don't think those are handled r.n. It was pretty straightforward to create a simple corefile.

lldb/source/DataFormatters/CXXFunctionPointer.cpp

61

This method (and Address::Dump) are concatenating text into the Stream sstr which is a symbolic description of the address; we want to print

fmain = 0xPTRAUTH (actual=0xACTUAL <name of function, source line, etc>)

so this is prepending the "actual=0xACTUAL " part, replacing the Address we're looking up with the ptrauth-stripped-off Address, and relying on Address::Dump to append its normal symbolication stuff. Then just below in this method we have

if (sstr.GetSize() > 0) {
  stream.Printf("(%s)", sstr.GetData());

where the parens are added.

lldb/source/Plugins/ABI/AArch64/ABIMacOSX_arm64.cpp

830

I intend this to handle the case of code running in high memory, where bit 55 is set, indicating that all of the high bits should be set to 1. Unlike userland memory on Darwin which runs in low memory, where bit 55 is clear and so all the high bits should be cleared to 0 when removing the PAC bits.

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

5161

Good point, altho looking at this bit of code my goal was to force Vendor to apple for any mach-o binary, not just kexts. Whatever, the key part was not clearing the Vendor here.

lldb/test/API/macosx/corefile-default-ptrauth/Makefile

10

OK. I copied a Makefile I used in another test, where I have a helper program and the test binary, and didn't think about how it reads.

lldb/test/API/macosx/corefile-default-ptrauth/create-corefile.c

19

The LC_NOTE names are only 16 characters, so unfortunately, "addrable bits" it is. :)

58

The LC_THREAD load command has four uint32_t's at the start, but you're right and I was even thinking of commenting in this last night & did not. I added a comment with the format of LC_THREAD.

93

It's a Darwin specific feature register contexts; most API accept a register state "flavor" and a size, expressed in number of 32-bit words of content. So if a register context grows, a kernel etc can distinguish between an older smaller regctx and a newer larger one that both use the same flavor enum. (in reality I don't think I've ever seen this done with a register flavor as far as I can remember, but it's a common pattern used elsewhere too). Added a comment.

[jasonmolenda]

Update patchset to address David's feedback.

[DavidSpickett]

lldb/source/Plugins/ABI/AArch64/ABIMacOSX_arm64.cpp

830

Got it! (for Linux I'm used to userspace addresses only)

lldb/source/DataFormatters/CXXFunctionPointer.cpp

 //===-- CXXFunctionPointer.cpp---------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/DataFormatters/CXXFunctionPointer.h"
 
 #include "lldb/Core/ValueObject.h"
+#include "lldb/Target/ABI.h"
 #include "lldb/Target/SectionLoadList.h"
 #include "lldb/Target/Target.h"
 #include "lldb/Utility/Stream.h"
 
 #include <string>
 
 using namespace lldb;
 using namespace lldb_private;
case eAddressTypeHost:
       break;
 
     case eAddressTypeLoad: {
       ExecutionContext exe_ctx(valobj.GetExecutionContextRef());
 
       Address so_addr;
       Target *target = exe_ctx.GetTargetPtr();
       if (target && !target->GetSectionLoadList().IsEmpty()) {
-        if (target->GetSectionLoadList().ResolveLoadAddress(func_ptr_address,
-                                                            so_addr)) {
+        target->GetSectionLoadList().ResolveLoadAddress(func_ptr_address,
+                                                            so_addr);
+        if (so_addr.GetSection() == nullptr) {
+          // If we have an address that doesn't correspond to any symbol,
+          // it might have authentication bits.  Strip them & see if it
+          // now points to a symbol -- if so, do the SymbolContext lookup
+          // based on the stripped address.
+          // If we find a symbol with the ptrauth bits stripped, print the
+          // raw value into the stream, and replace the Address with the
+          // one that points to a symbol for a fuller description.
+          if (Process *process = exe_ctx.GetProcessPtr()) {
+            if (ABISP abi_sp = process->GetABI()) {
+              addr_t fixed_addr = abi_sp->FixCodeAddress(func_ptr_address);
+              if (fixed_addr != func_ptr_address) {
+                Address test_address;
+                test_address.SetLoadAddress(fixed_addr, target);
+                if (test_address.GetSection() != nullptr) {
+                  int addrsize = target->GetArchitecture().GetAddressByteSize();
+                  sstr.Printf("actual=0x%*.*" PRIx64 " ", addrsize * 2, addrsize * 2, fixed_addr);
+                  so_addr = test_address;
+                }
+              }
+            }
+          }
+        }
+
+        if (so_addr.IsValid()) {
           so_addr.Dump(&sstr, exe_ctx.GetBestExecutionContextScope(),
                        Address::DumpStyleResolvedDescription,
                        Address::DumpStyleSectionNameOffset);
         }
       }
     } break;
     }
   }
   if (sstr.GetSize() > 0) {
     stream.Printf("(%s)", sstr.GetData());
     return true;
   } else
     return false;
 }

lldb/source/Plugins/ABI/AArch64/ABIMacOSX_arm64.cpp

           &thread, return_compiler_type, ConstString(""), data);
     }
   }
   return return_valobj_sp;
 }
 
 lldb::addr_t ABIMacOSX_arm64::FixAddress(addr_t pc, addr_t mask) {
   lldb::addr_t pac_sign_extension = 0x0080000000000000ULL;
+  // Darwin systems originally couldn't determine the proper value
+  // dynamically, so the most common value was hardcoded.  This has
+  // largely been cleaned up, but there are still a handful of
+  // environments that assume the default value is set to this value
+  // and there's no dynamic value to correct it.
+  // When no mask is specified, set it to 39 bits of addressing (0..38).
+  if (mask == 0) {
+    // ~((1ULL<<39)-1)
+    mask = 0xffffff8000000000;
+  }
   return (pc & pac_sign_extension) ? pc | mask : pc & (~mask);
 }
 
 void ABIMacOSX_arm64::Initialize() {
   PluginManager::RegisterPlugin(GetPluginNameStatic(), pluginDesc,
                                 CreateInstance);
 }
 
 void ABIMacOSX_arm64::Terminate() {
   PluginManager::UnregisterPlugin(CreateInstance);
 }

lldb/source/Plugins/ObjectFile/Mach-O/ObjectFileMachO.cpp

           triple.setEnvironmentName(os_env.environment);
         add_triple(triple);
       }
     } while (0);
     offset = cmd_offset + load_cmd.cmdsize;
   }
 
   if (!found_any) {
-    if (header.filetype == MH_KEXT_BUNDLE) {
-      base_triple.setVendor(llvm::Triple::Apple);
-      add_triple(base_triple);
-    } else {
-      // We didn't find a LC_VERSION_MIN load command and this isn't a KEXT
-      // so lets not say our Vendor is Apple, leave it as an unspecified
-      // unknown.
-      base_triple.setVendor(llvm::Triple::UnknownVendor);
-      base_triple.setVendorName(llvm::StringRef());
-      add_triple(base_triple);
-    }
+    add_triple(base_triple);
   }
 }
 
 ArchSpec ObjectFileMachO::GetArchitecture(
     ModuleSP module_sp, const llvm::MachO::mach_header &header,
     const lldb_private::DataExtractor &data, lldb::offset_t lc_offset) {
   ModuleSpecList all_specs;
   ModuleSpec base_spec;
   GetAllArchSpecs(header, data, MachHeaderSizeFromMagic(header.magic),
                   base_spec, all_specs);
 
   // If the object file offers multiple alternative load commands,

lldb/source/Utility/ArchSpec.cpp

         m_core = core_def->core;
         update_triple = false;
         // Always use the architecture name because it might be more
         // descriptive than the architecture enum ("armv7" ->
         // llvm::Triple::arm).
         m_triple.setArchName(llvm::StringRef(core_def->name));
         if (arch_type == eArchTypeMachO) {
           m_triple.setVendor(llvm::Triple::Apple);
+          m_triple.setObjectFormat(llvm::Triple::MachO);
 
           // Don't set the OS.  It could be simulator, macosx, ios, watchos,
           // tvos, bridgeos.  We could get close with the cpu type - but we
           // can't get it right all of the time.  Better to leave this unset
           // so other sections of code will set it when they have more
           // information. NB: don't call m_triple.setOS
           // (llvm::Triple::UnknownOS). That sets the OSName to "unknown" and
           // the ArchSpec::TripleVendorWasSpecified() method says that any

lldb/test/API/macosx/corefile-default-ptrauth/Makefile

+C_SOURCES := main.c
+
+# compile a.out and create-corefile
+# create-corefile will create a custom corefile using the symbols
+# addresses from the a.out binary.
+all: a.out create-corefile
+
+create-corefile:
+  $(MAKE) -f $(MAKEFILE_RULES) EXE=create-corefile \
+    C_SOURCES=create-corefile.c
+
+include Makefile.rules

lldb/test/API/macosx/corefile-default-ptrauth/TestCorefileDefaultPtrauth.py

+"""Test that lldb has a default mask for addressable bits on Darwin arm64 ABI"""
+
+
+import os
+import re
+import subprocess
+
+import lldb
+from lldbsuite.test.decorators import *
+from lldbsuite.test.lldbtest import *
+from lldbsuite.test import lldbutil
+
+
+class TestCorefileDefaultPtrauth(TestBase):
+
+    mydir = TestBase.compute_mydir(__file__)
+
+    @skipIf(debug_info=no_match(["dsym"]), bugnumber="This test is looking explicitly for a dSYM")
+    @skipIf(archs=no_match(['arm64','arm64e']))
+    @skipUnlessDarwin
+    def test_lc_note(self):
+        self.build()
+        self.test_exe = self.getBuildArtifact("a.out")
+        self.create_corefile = self.getBuildArtifact("create-corefile")
+        self.corefile = self.getBuildArtifact("core")
+
+        ### Create our corefile
+        retcode = call(self.create_corefile + " " +  self.test_exe + " " + self.corefile, shell=True)
+
+        ## This corefile has no metadata telling us how many bits are
+        ## used for ptrauth signed function pointers.  We will need lldb
+        ## to fall back on its old default value for Darwin arm64 ABIs
+        ## to correctly strip the bits.
+
+        self.target = self.dbg.CreateTarget('')
+        err = lldb.SBError()
+        self.process = self.target.LoadCore(self.corefile)
+        self.assertEqual(self.process.IsValid(), True)
+        modspec = lldb.SBModuleSpec()
+        modspec.SetFileSpec(lldb.SBFileSpec(self.test_exe, True))
+        m = self.target.AddModule(modspec)
+        self.assertTrue(m.IsValid())
+        self.target.SetModuleLoadAddress (m, 0)
+        
+        # target variable should show us both the actual function
+        # pointer with ptrauth bits and the symbol it resolves to,
+        # with the ptrauth bits stripped, e.g.
+        #  (int (*)(...)) fmain = 0xe46bff0100003f90 (actual=0x0000000100003f90 a.out`main at main.c:3)
+
+        self.expect("target variable fmain", substrs=['fmain = 0x', '(actual=0x', 'main at main.c'])

lldb/test/API/macosx/corefile-default-ptrauth/create-corefile.c

+#include <mach-o/loader.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <mach/machine.h>
+#include <string.h>
+#include <mach/machine/thread_state.h>
+#include <inttypes.h>
+#include <sys/syslimits.h>
+
+// Given an executable binary with 
+//   "fmain" (a function pointer to main)
+//   "main"
+// symbols, create a fake arm64e corefile that
+// contains a memory segment for the fmain 
+// function pointer, with the value of the 
+// address of main() with ptrauth bits masked on.
+//
+// The corefile does not include the "addrable bits"
+// LC_NOTE, so lldb will need to fall back on its 
+// default value from the Darwin arm64 ABI.
+
+int main(int argc, char **argv)
+{
+  if (argc != 3) {
+    fprintf (stderr, "usage: %s executable-binary output-file\n", argv[0]);
+    exit(1);
+  }
+  FILE *exe = fopen(argv[1], "r");
+  if (!exe) {
+    fprintf (stderr, "Unable to open executable %s for reading\n", argv[1]);
+    exit(1);
+  }
+  FILE *out = fopen(argv[2], "w");
+  if (!out) {
+    fprintf (stderr, "Unable to open %s for writing\n", argv[2]);
+    exit(1);
+  }
+
+  char buf[PATH_MAX + 6];
+  sprintf (buf, "nm '%s'", argv[1]);
+  FILE *nm = popen(buf, "r");
+  if (!nm) {
+    fprintf (stderr, "Unable to run nm on '%s'", argv[1]);
+    exit (1);
+  }
+  uint64_t main_addr = 0;
+  uint64_t fmain_addr = 0;
+  while (fgets (buf, sizeof(buf), nm)) {
+    if (strstr (buf, "_fmain")) {
+      fmain_addr = strtoul (buf, NULL, 16);
+    }
+    if (strstr (buf, "_main")) {
+      main_addr = strtoul (buf, NULL, 16);
+    }
+  }
+  pclose (nm);
+
+  if (main_addr == 0 || fmain_addr == 0) {
+    fprintf(stderr, "Unable to find address of main or fmain in %s.\n",
+        argv[1]);
+    exit (1);
+  }
+
+  // Write out a corefile with contents in this order:
+  //    1. mach header
+  //    2. LC_THREAD load command
+  //    3. LC_SEGMENT_64 load command
+  //    4. memory segment contents
+
+  // struct thread_command {
+  //       uint32_t        cmd;    
+  //       uint32_t        cmdsize;
+  //       uint32_t flavor      
+  //       uint32_t count       
+  //       struct XXX_thread_state state
+  int size_of_thread_cmd = 4 + 4 + 4 + 4 + sizeof (arm_thread_state64_t);
+
+  struct mach_header_64 mh;
+  mh.magic = 0xfeedfacf;
+  mh.cputype = CPU_TYPE_ARM64;
+  mh.cpusubtype = CPU_SUBTYPE_ARM64E;
+  mh.filetype = MH_CORE;
+  mh.ncmds = 2; // LC_THREAD, LC_SEGMENT_64
+  mh.sizeofcmds = size_of_thread_cmd + sizeof(struct segment_command_64);
+  mh.flags = 0;
+  mh.reserved = 0;
+
+  fwrite(&mh, sizeof (mh), 1, out);
+
+  struct segment_command_64 seg;
+  seg.cmd = LC_SEGMENT_64;
+  seg.cmdsize = sizeof(seg);
+  memset (&seg.segname, 0, 16);
+  seg.vmaddr = fmain_addr;
+  seg.vmsize = 8;
+  // Offset to segment contents
+  seg.fileoff = sizeof (mh) + size_of_thread_cmd + sizeof(seg);
+  seg.filesize = 8;
+  seg.maxprot = 3;
+  seg.initprot = 3;
+  seg.nsects = 0;
+  seg.flags = 0;
+
+  fwrite (&seg, sizeof (seg), 1, out);
+
+  uint32_t cmd = LC_THREAD;
+  fwrite (&cmd, sizeof (cmd), 1, out);
+  uint32_t cmdsize = size_of_thread_cmd;
+  fwrite (&cmdsize, sizeof (cmdsize), 1, out);
+  uint32_t flavor = ARM_THREAD_STATE64;
+  fwrite (&flavor, sizeof (flavor), 1, out);
+  // count is number of uint32_t's of the register context
+  uint32_t count = sizeof (arm_thread_state64_t) / 4;
+  fwrite (&count, sizeof (count), 1, out);
+  arm_thread_state64_t regstate;
+  memset (&regstate, 0, sizeof (regstate));
+  fwrite (&regstate, sizeof (regstate), 1, out);
+
+
+  // Or together a random PAC value from a system using 39 bits 
+  // of addressing with the address of main().  lldb will need
+  // to correctly strip off the high bits to find the address of
+  // main.
+  uint64_t segment_contents = 0xe46bff0000000000 | main_addr;
+
+  fwrite (&segment_contents, sizeof (segment_contents), 1, out);
+
+  fclose (out);
+
+  exit (0);
+}

lldb/test/API/macosx/corefile-default-ptrauth/main.c

+int main();
+int (*fmain)() = main;
+int main () {
+  return fmain();
+}
+