Switch to a different library-loaded notification function breakpoint in Darwin dyld

[jasonmolenda]

Summary

Darwin's dynamic linker, dyld, calls a no-op function whenever libraries are loaded or removed from a process. There are two of these functions - one for older compatibility, which lldb is using, and actual one it calls first.

In a future patch, I'll change how lldb discovers the address of this function. lldb is currently not using the "canonical" one that I'll discover that way, and their arguments are slightly different, so this patch lays the groundwork by switching to this canonical notification function with the same discovery mechanism.

The important difference between these two is that the old function (lldb currently uses) is passed an array of uint64_t addresses of binary mach-o headers. In the new, canonical function, the headers are passed as an array of ptrsize -- so 32-bits, on a target like arm64_32 which is ILP32 (we don't have any genuine 32-bit targets that we still support). Beyond that, it's only a different function name.

[jasonmolenda]

oops, slight misstatement from not reading the patch when I wrote the description. The function we currently use has an array of uint64_t's, the one I am switching to is given an array of dyld_image_info objects, the first element has the ptrsize address of the mach-o header.

[jingham]

This makes sense. I think you should at least log if you can't read the memory dyld told you was where the image info should be. Seems like this is the sort of thing that shouldn't happen, but if it does we probably want to say something. Other than that minor grip, LGTM.

lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderMacOS.cpp

329

Is it expected that this memory read fail? It seems weird to just not handle that case at all? Maybe at least log something here, otherwise it looks like we just get nothing w/o knowing why.

[JDevlieghere]

lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderMacOS.cpp

324–325

Nit: pretty sure GetAddressByteSize() returns an unsigned (uint32_t or uint8_t).

329

+1

[jasonmolenda]

Update patch to address Jim and Jonas' feedback, most importantly report a warning if we can't read one of the mach-o binary addresses that dyld is telling us are being added/removed.

[jasonmolenda]

After reverting to avoid CI bot failures on macos intel, this is an updated patch that (1) corrects the binary image count size, (2) allows for a fallback if it can't find the notification function lldb_image_notifier, by name.

[jasonmolenda]

Update the patch to address the failures we were hitting on the Intel macOS 10.15 CI bot. In addition to lldb_notifier_function, look for "gdb_notifier_function" which is what it was called back in macOS 10.15. Read the notification function address out of the dyld_all_image_infos as a last resort, if dyld_all_image_infos has been initialized.

lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderMacOS.h

protected:
 
   bool IsFullyInitialized() override;
 
   static bool
   NotifyBreakpointHit(void *baton,
                       lldb_private::StoppointCallbackContext *context,
                       lldb::user_id_t break_id, lldb::user_id_t break_loc_id);
 
+  lldb::addr_t GetNotificationFuncAddrFromImageInfos();
+
   bool SetNotificationBreakpoint() override;
 
   void ClearNotificationBreakpoint() override;
 
   void UpdateImageInfosHeaderAndLoadCommands(ImageInfo::collection &image_infos,
                                              uint32_t infos_count,
                                              bool update_executable);
 

lldb/source/Plugins/DynamicLoader/MacOSX-DYLD/DynamicLoaderMacOS.cpp

 // Static callback function that gets called when our DYLD notification
 // breakpoint gets hit. We update all of our image infos and then let our super
 // class DynamicLoader class decide if we should stop or not (based on global
 // preference).
 bool DynamicLoaderMacOS::NotifyBreakpointHit(void *baton,
                                              StoppointCallbackContext *context,
                                              lldb::user_id_t break_id,
                                              lldb::user_id_t break_loc_id) {
-  // Let the event know that the images have changed
-  // DYLD passes three arguments to the notification breakpoint.
-  // Arg1: enum dyld_notify_mode mode - 0 = adding, 1 = removing, 2 = remove
-  // all Arg2: unsigned long icount        - Number of shared libraries
-  // added/removed Arg3: uint64_t mach_headers[]     - Array of load addresses
-  // of binaries added/removed
+  //
+  // Our breakpoint on
+  //
+  // void lldb_image_notifier(enum dyld_image_mode mode, uint32_t infoCount,
+  // const dyld_image_info info[])
+  //
+  // has been hit.  We need to read the arguments.
 
   DynamicLoaderMacOS *dyld_instance = (DynamicLoaderMacOS *)baton;
 
   ExecutionContext exe_ctx(context->exe_ctx_ref);
   Process *process = exe_ctx.GetProcessPtr();
 
   // This is a sanity check just in case this dyld_instance is an old dyld
   // plugin's breakpoint still lying around.
if (abi) {
     TypeSystemClangSP scratch_ts_sp =
         ScratchTypeSystemClang::GetForTarget(process->GetTarget());
     if (!scratch_ts_sp)
       return false;
 
     ValueList argument_values;
 
     Value mode_value;    // enum dyld_notify_mode { dyld_notify_adding=0,
-                         // dyld_notify_removing=1, dyld_notify_remove_all=2 };
-    Value count_value;   // unsigned long count
-    Value headers_value; // uint64_t machHeaders[] (aka void*)
+                         // dyld_notify_removing=1, dyld_notify_remove_all=2,
+                         // dyld_notify_dyld_moved=3 };
+    Value count_value;   // uint32_t
+    Value headers_value; // struct dyld_image_info machHeaders[]
 
     CompilerType clang_void_ptr_type =
         scratch_ts_sp->GetBasicType(eBasicTypeVoid).GetPointerType();
     CompilerType clang_uint32_type =
         scratch_ts_sp->GetBuiltinTypeForEncodingAndBitSize(lldb::eEncodingUint,
                                                            32);
     CompilerType clang_uint64_type =
         scratch_ts_sp->GetBuiltinTypeForEncodingAndBitSize(lldb::eEncodingUint,
                                                            32);
 
     mode_value.SetValueType(Value::ValueType::Scalar);
     mode_value.SetCompilerType(clang_uint32_type);
 
-    if (process->GetTarget().GetArchitecture().GetAddressByteSize() == 4) {
-      count_value.SetValueType(Value::ValueType::Scalar);
-      count_value.SetCompilerType(clang_uint32_type);
-    } else {
-      count_value.SetValueType(Value::ValueType::Scalar);
-      count_value.SetCompilerType(clang_uint64_type);
-    }
 
     headers_value.SetValueType(Value::ValueType::Scalar);
     headers_value.SetCompilerType(clang_void_ptr_type);
 
     argument_values.PushValue(mode_value);
     argument_values.PushValue(count_value);
     argument_values.PushValue(headers_value);
 
     if (abi->GetArgumentValues(exe_ctx.GetThreadRef(), argument_values)) {
       uint32_t dyld_mode =
           argument_values.GetValueAtIndex(0)->GetScalar().UInt(-1);
       if (dyld_mode != static_cast<uint32_t>(-1)) {
         // Okay the mode was right, now get the number of elements, and the
         // array of new elements...
         uint32_t image_infos_count =
             argument_values.GetValueAtIndex(1)->GetScalar().UInt(-1);
         if (image_infos_count != static_cast<uint32_t>(-1)) {
           addr_t header_array =
               argument_values.GetValueAtIndex(2)->GetScalar().ULongLong(-1);
           if (header_array != static_cast<uint64_t>(-1)) {
             std::vector<addr_t> image_load_addresses;
+            // header_array points to an array of image_infos_count elements,
+            // each is
+            // struct dyld_image_info {
+            //   const struct mach_header* imageLoadAddress;
+            //   const char*               imageFilePath;
+            //   uintptr_t                 imageFileModDate;
+            // };
+            //
+            // and we only need the imageLoadAddress fields.
+
+            const int addrsize =
+                process->GetTarget().GetArchitecture().GetAddressByteSize();
             for (uint64_t i = 0; i < image_infos_count; i++) {
               Status error;
-              addr_t addr = process->ReadUnsignedIntegerFromMemory(
-                  header_array + (8 * i), 8, LLDB_INVALID_ADDRESS, error);
-              if (addr != LLDB_INVALID_ADDRESS) {
+              addr_t dyld_image_info = header_array + (addrsize * 3 * i);
+              addr_t addr =
+                  process->ReadPointerFromMemory(dyld_image_info, error);
+              if (error.Success()) {
                 image_load_addresses.push_back(addr);
+              } else {
+                Debugger::ReportWarning(
+                    "DynamicLoaderMacOS::NotifyBreakpointHit unable "
+                    "to read binary mach-o load address at 0x%" PRIx64,
+                    addr);
               }
             }
             if (dyld_mode == 0) {
               // dyld_notify_adding
               if (process->GetTarget().GetImages().GetSize() == 0) {
                 // When all images have been removed, we're doing the
                 // dyld handover from a launch-dyld to a shared-cache-dyld,
                 // and we've just hit our one-shot address breakpoint in
if (abi->GetArgumentValues(exe_ctx.GetThreadRef(), argument_values)) {
               int addr_size =
                   process->GetTarget().GetArchitecture().GetAddressByteSize();
               addr_t notification_location = all_image_infos + 4 + // version
                                              4 +        // infoArrayCount
                                              addr_size; // infoArray
               Status error;
               addr_t notification_addr =
                   process->ReadPointerFromMemory(notification_location, error);
-              if (ABISP abi_sp = process->GetABI())
-                notification_addr = abi_sp->FixCodeAddress(notification_addr);
-
+              if (!error.Success()) {
+                Debugger::ReportWarning(
+                    "DynamicLoaderMacOS::NotifyBreakpointHit unable "
+                    "to read address of dyld-handover notification function at "
+                    "0x%" PRIx64,
+                    notification_location);
+              } else {
+                notification_addr = process->FixCodeAddress(notification_addr);
                 dyld_instance->SetDYLDHandoverBreakpoint(notification_addr);
               }
             }
           }
         }
       }
+    }
   } else {
     Target &target = process->GetTarget();
     Debugger::ReportWarning(
         "no ABI plugin located for triple " +
             target.GetArchitecture().GetTriple().getTriple() +
             ": shared libraries will not be registered",
         target.GetDebugger().GetID());
   }

 
 // Dump the _dyld_all_image_infos members and all current image infos that we
 // have parsed to the file handle provided.
 void DynamicLoaderMacOS::PutToLog(Log *log) const {
   if (log == nullptr)
     return;
 }
 
+// Look in dyld's dyld_all_image_infos structure for the address
+// of the notification function.
+// We can find the address of dyld_all_image_infos by a system
+// call, even if we don't have a dyld binary registered in lldb's
+// image list.
+// At process launch time - before dyld has executed any instructions -
+// the address of the notification function is not a resolved vm address
+// yet.  dyld_all_image_infos also has a field with its own address
+// in it, and this will also be unresolved when we're at this state.
+// So we can compare the address of the object with this field and if
+// they differ, dyld hasn't started executing yet and we can't get the
+// notification address this way.
+addr_t DynamicLoaderMacOS::GetNotificationFuncAddrFromImageInfos() {
+  addr_t notification_addr = LLDB_INVALID_ADDRESS;
+  if (!m_process)
+    return notification_addr;
+
+  addr_t all_image_infos_addr = m_process->GetImageInfoAddress();
+  if (all_image_infos_addr == LLDB_INVALID_ADDRESS)
+    return notification_addr;
+
+  const uint32_t addr_size =
+      m_process->GetTarget().GetArchitecture().GetAddressByteSize();
+  offset_t registered_infos_addr_offset =
+      sizeof(uint32_t) + // version
+      sizeof(uint32_t) + // infoArrayCount
+      addr_size +        // infoArray
+      addr_size +        // notification
+      addr_size +        // processDetachedFromSharedRegion +
+                         // libSystemInitialized + pad
+      addr_size +        // dyldImageLoadAddress
+      addr_size +        // jitInfo
+      addr_size +        // dyldVersion
+      addr_size +        // errorMessage
+      addr_size +        // terminationFlags
+      addr_size +        // coreSymbolicationShmPage
+      addr_size +        // systemOrderFlag
+      addr_size +        // uuidArrayCount
+      addr_size;         // uuidArray
+                         // dyldAllImageInfosAddress
+
+  // If the dyldAllImageInfosAddress does not match
+  // the actual address of this struct, dyld has not started
+  // executing yet.  The 'notification' field can't be used by
+  // lldb until it's resolved to an actual address.
+  Status error;
+  addr_t registered_infos_addr = m_process->ReadPointerFromMemory(
+      all_image_infos_addr + registered_infos_addr_offset, error);
+  if (!error.Success())
+    return notification_addr;
+  if (registered_infos_addr != all_image_infos_addr)
+    return notification_addr;
+
+  offset_t notification_fptr_offset = sizeof(uint32_t) + // version
+                                      sizeof(uint32_t) + // infoArrayCount
+                                      addr_size;         // infoArray
+
+  addr_t notification_fptr = m_process->ReadPointerFromMemory(
+      all_image_infos_addr + notification_fptr_offset, error);
+  if (error.Success())
+    notification_addr = m_process->FixCodeAddress(notification_fptr);
+  return notification_addr;
+}
+
+// We want to put a breakpoint on dyld's lldb_image_notifier()
+// but we may have attached to the process during the
+// transition from on-disk-dyld to shared-cache-dyld, so there's
+// officially no dyld binary loaded in the process (libdyld will
+// report none when asked), but the kernel can find the dyld_all_image_infos
+// struct and the function pointer for lldb_image_notifier is in
+// that struct.
 bool DynamicLoaderMacOS::SetNotificationBreakpoint() {
+
+  // First try to find the notification breakpoint function by name
   if (m_break_id == LLDB_INVALID_BREAK_ID) {
     ModuleSP dyld_sp(GetDYLDModule());
     if (dyld_sp) {
       bool internal = true;
       bool hardware = false;
       LazyBool skip_prologue = eLazyBoolNo;
       FileSpecList *source_files = nullptr;
       FileSpecList dyld_filelist;
       dyld_filelist.Append(dyld_sp->GetFileSpec());
 
       Breakpoint *breakpoint =
           m_process->GetTarget()
               .CreateBreakpoint(&dyld_filelist, source_files,
-                                "_dyld_debugger_notification",
-                                eFunctionNameTypeFull, eLanguageTypeC, 0,
-                                skip_prologue, internal, hardware)
+                                "lldb_image_notifier", eFunctionNameTypeFull,
+                                eLanguageTypeUnknown, 0, skip_prologue,
+                                internal, hardware)
+              .get();
+      breakpoint->SetCallback(DynamicLoaderMacOS::NotifyBreakpointHit, this,
+                              true);
+      breakpoint->SetBreakpointKind("shared-library-event");
+      if (breakpoint->HasResolvedLocations())
+        m_break_id = breakpoint->GetID();
+      else
+        m_process->GetTarget().RemoveBreakpointByID(breakpoint->GetID());
+
+      if (m_break_id == LLDB_INVALID_BREAK_ID) {
+        Breakpoint *breakpoint =
+            m_process->GetTarget()
+                .CreateBreakpoint(&dyld_filelist, source_files,
+                                  "gdb_image_notifier", eFunctionNameTypeFull,
+                                  eLanguageTypeUnknown, 0, skip_prologue,
+                                  internal, hardware)
                 .get();
         breakpoint->SetCallback(DynamicLoaderMacOS::NotifyBreakpointHit, this,
                                 true);
         breakpoint->SetBreakpointKind("shared-library-event");
+        if (breakpoint->HasResolvedLocations())
           m_break_id = breakpoint->GetID();
+        else
+          m_process->GetTarget().RemoveBreakpointByID(breakpoint->GetID());
+      }
+    }
+  }
+
+  // Failing that, find dyld_all_image_infos struct in memory,
+  // read the notification function pointer at the offset.
+  if (m_break_id == LLDB_INVALID_BREAK_ID) {
+    addr_t notification_addr = GetNotificationFuncAddrFromImageInfos();
+    if (notification_addr != LLDB_INVALID_ADDRESS) {
+      Address so_addr;
+      // We may not have a dyld binary mapped to this address yet;
+      // don't try to express the Address object as section+offset,
+      // only as a raw load address.
+      so_addr.SetRawAddress(notification_addr);
+      Breakpoint *dyld_break =
+          m_process->GetTarget().CreateBreakpoint(so_addr, true, false).get();
+      dyld_break->SetCallback(DynamicLoaderMacOS::NotifyBreakpointHit, this,
+                              true);
+      dyld_break->SetBreakpointKind("shared-library-event");
+      if (dyld_break->HasResolvedLocations())
+        m_break_id = dyld_break->GetID();
+      else
+        m_process->GetTarget().RemoveBreakpointByID(dyld_break->GetID());
     }
   }
   return m_break_id != LLDB_INVALID_BREAK_ID;
 }
 
 bool DynamicLoaderMacOS::SetDYLDHandoverBreakpoint(
     addr_t notification_address) {
   if (m_dyld_handover_break_id == LLDB_INVALID_BREAK_ID) {