Advanced guessing of rendezvous breakpoint

[eugene]

Summary

When rendezvous structure is not initialized we need to set up rendezvous breakpoint anyway. In this case the code will locate dynamic loader (interpreter) and look for known function names.
Bug: 25806

[labath]

packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py

371

This test has been disabled for a couple of years. I wouldn't be surprised if the problems you are running into with it are due to bitrot.

I didn't know this test exists, so I've added TestBreakpointInGlobalConstructor.py to test this behavior a week or two ago. That one is known to pass on mac, so it shouldn't suffer from bitrot. If that's the case then maybe we can just remove this test (or the other one -- In any case, I think we don't need both of them).

source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp

271

So, the way I understand this, the main purpose of the entry breakpoint is to set the rendezvous breakpoint (the LoadAllCurrentModules is just a side-effect of us not being able to catch the library load early enough -- if we resolve the rendezvous breakpoint this way, it should not be needed.

I expect we can just omit the entry breakpoint altogether if we've managed to set the rendezvous breakpoint this way (we can probably keep the code as a fallback, but use it only for cases where we don't locate any of the rendezvous symbols).

354

the LLDB_LOG will automatically insert file+function (if you do log enable --file-function) so you can just drop this __FUNCTION__ part

357

s/tring/trying

604

s/moduel/module

[eugene]

• Addressing review comments
• Making sure all tests pass on Linux
• Reformating

[eugene]

packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py

371

I fixed both of them, I think it's fine to keep both.

[eugene]

Don't set entry breakpoint if rendezvous breakpoint was set right after launch.

[eugene]

Fix tests.

[labath]

I'm adding some people with non-linux non-x86/arm targets who use this plugin for more visibility. I have a couple of small comments, but in general, I am happy with this.

packages/Python/lldbsuite/test/functionalities/breakpoint/global_constructor/TestBreakpointInGlobalConstructor.py

24

The right way to handle this is to use the environment which is returned from self.registerSharedLibrariesWithTarget as the launch environment (you'll need to launch using the SB API). The reason that TestLoadUnload does not handle this is probably because it predates this function.

source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp

224

s/enty/entry

379

It should be possible to create the breakpoint via Target::CreateBreakpoint (the overload that lets you specify a list of function names) instead of manually resolving the symbols... Have you tried doing that?

[clayborg]

Looks fine. Set the breakpoint using the list of names and delete the breakpoint if you get no locations and this will be good to go.

source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp

365–368

Will only one of these ever be present?

379

Agreed, I would use the breakpoint setting function in target that allows you to specify one or more names as labath suggested. One question about this code though: how would we ever get a load address from a module if the dynamic loader doesn't load it itself? Are these symbols in the "[vdso]" module and is the [vdso]" module loaded differently?

399

You should delete the breakpoint if the number of locations is zero (for this one by address and also in case you end up setting the breakpoint using a list of names). Otherwise you will have a dangling breakpoint that no one wants.

[eugene]

source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp

365–368

Yes. These are the names of rendezvous functions in different known linkers (Android, Linux, BSD) None of them should have more than one function from this list.

379

OS loads the dynamic loader, if it is requested by the executable, and passes its base address via AT_BASE value in the auxiliary vector (https://refspecs.linuxfoundation.org/LSB_1.3.0/IA64/spec/auxiliaryvector.html). That's how we know where to look for it. (see EvalSpecialModulesStatus)

vdso is yet another special module, but it is not related to the dynamic loader.

[eugene]

Addressing code review comments. Switching from manual symbol resolution to the appropriate overload of CreateBreakpoint.

[labath]

looks good, thanks.

[ted]

Thanks for adding me, Pavel.

Hexagon running Linux uses this plugin. These changes lgtm.

Standalone Hexagon uses its own dyld plugin; I need to look at it and see if I want to pull any of these ideas into it.

[clayborg]

Looks good.

packages/Python/lldbsuite/test/functionalities/breakpoint/global_constructor/TestBreakpointInGlobalConstructor.py

class TestBreakpointInGlobalConstructors(TestBase):
 
     mydir = TestBase.compute_mydir(__file__)
     NO_DEBUG_INFO_TESTCASE = True
 
     def setUp(self):
         TestBase.setUp(self)
         self.line_foo = line_number('foo.cpp', '// !BR_foo')
         self.line_main = line_number('main.cpp', '// !BR_main')
+        if not self.platformIsDarwin():
+          if lldb.remote_platform:
+              wd = lldb.remote_platform.GetWorkingDirectory()
+          else:
+              wd = os.getcwd()
+          self.runCmd(
+              "settings set target.env-vars " +
+              self.dylibPath +
+              "=" +
+              wd)
+
 
-    @expectedFailureAll(bugnumber="llvm.org/pr35480", oslist=["linux"])
     def test(self):
         self.build()
         exe = os.path.join(os.getcwd(), "a.out")
+
+        target = self.dbg.CreateTarget("a.out")
+        self.assertTrue(target, VALID_TARGET)
+        self.registerSharedLibrariesWithTarget(target, ["foo"])
+
         self.runCmd("file %s" % exe)
 
         bp_main = lldbutil.run_break_set_by_file_and_line(
             self, 'main.cpp', self.line_main)
         bp_foo = lldbutil.run_break_set_by_file_and_line(
             self, 'foo.cpp', self.line_foo)
 
         self.runCmd("run")

packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py

def test_step_over_load(self):
 
         # The stop reason should be step end.
         self.expect("thread list", "step over succeeded.",
                     substrs=['stopped',
                              'stop reason = step over'])
 
     @skipIfFreeBSD  # llvm.org/pr14424 - missing FreeBSD Makefiles/testcase support
     @skipIfWindows  # Windows doesn't have dlopen and friends, dynamic libraries work differently
-    @unittest2.expectedFailure("llvm.org/pr25806")
     def test_static_init_during_load(self):
         """Test that we can set breakpoints correctly in static initializers"""
 
         self.build()
         self.copy_shlibs_to_remote()
 
         exe = os.path.join(os.getcwd(), "a.out")
         self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
def test_static_init_during_load(self):
         self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
                     substrs=['stopped',
                              'd_init',
                              'stop reason = breakpoint %d' % d_init_bp_num])
 
         self.runCmd("continue")
         self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
                     substrs=['stopped',
-                             'a_init',
-                             'stop reason = breakpoint %d' % a_init_bp_num])
+                             'b_init',
+                             'stop reason = breakpoint %d' % b_init_bp_num])
         self.expect("thread backtrace",
-                    substrs=['a_init',
+                    substrs=['b_init',
                              'dlopen',
                              'main'])
 
         self.runCmd("continue")
         self.expect("thread list", STOPPED_DUE_TO_BREAKPOINT,
                     substrs=['stopped',
-                             'b_init',
-                             'stop reason = breakpoint %d' % b_init_bp_num])
+                             'a_init',
+                             'stop reason = breakpoint %d' % a_init_bp_num])
         self.expect("thread backtrace",
-                    substrs=['b_init',
+                    substrs=['a_init',
                              'dlopen',
                              'main'])

source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.h

protected:
 
   /// Rendezvous breakpoint.
   lldb::break_id_t m_dyld_bid;
 
   /// Contains AT_SYSINFO_EHDR, which means a vDSO has been
   /// mapped to the address space
   lldb::addr_t m_vdso_base;
 
+  /// Contains AT_BASE, which means a dynamic loader has been
+  /// mapped to the address space
+  lldb::addr_t m_interpreter_base;
+
   /// Loaded module list. (link map for each module)
   std::map<lldb::ModuleWP, lldb::addr_t, std::owner_less<lldb::ModuleWP>>
       m_loaded_modules;
 
-  /// Enables a breakpoint on a function called by the runtime
+  /// If possible sets a breakpoint on a function called by the runtime
   /// linker each time a module is loaded or unloaded.
-  virtual void SetRendezvousBreakpoint();
+  bool SetRendezvousBreakpoint();
 
   /// Callback routine which updates the current list of loaded modules based
   /// on the information supplied by the runtime linker.
   static bool RendezvousBreakpointHit(
       void *baton, lldb_private::StoppointCallbackContext *context,
       lldb::user_id_t break_id, lldb::user_id_t break_loc_id);
 
   /// Helper method for RendezvousBreakpointHit.  Updates LLDB's current set
protected:
   EntryBreakpointHit(void *baton,
                      lldb_private::StoppointCallbackContext *context,
                      lldb::user_id_t break_id, lldb::user_id_t break_loc_id);
 
   /// Helper for the entry breakpoint callback.  Resolves the load addresses
   /// of all dependent modules.
   virtual void LoadAllCurrentModules();
 
-  void LoadVDSO(lldb_private::ModuleList &modules);
+  void LoadVDSO();
+
+  // Loading an interpreter module (if present) assumming m_interpreter_base
+  // already points to its base address.
+  lldb::ModuleSP LoadInterpreterModule();
 
   /// Computes a value for m_load_offset returning the computed address on
   /// success and LLDB_INVALID_ADDRESS on failure.
   lldb::addr_t ComputeLoadOffset();
 
   /// Computes a value for m_entry_point returning the computed address on
   /// success and LLDB_INVALID_ADDRESS on failure.
   lldb::addr_t GetEntryPoint();
 
-  /// Evaluate if Aux vectors contain vDSO information
+  /// Evaluate if Aux vectors contain vDSO and LD information
   /// in case they do, read and assign the address to m_vdso_base
-  void EvalVdsoStatus();
+  /// and m_interpreter_base.
+  void EvalSpecialModulesStatus();
 
   /// Loads Module from inferior process.
   void ResolveExecutableModule(lldb::ModuleSP &module_sp);
 
   bool AlwaysRelyOnEHUnwindInfo(lldb_private::SymbolContext &sym_ctx) override;
 
 private:
   DISALLOW_COPY_AND_ASSIGN(DynamicLoaderPOSIXDYLD);
 };
 
 #endif // liblldb_DynamicLoaderPOSIXDYLD_h_

source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp

if (create)
     return new DynamicLoaderPOSIXDYLD(process);
   return NULL;
 }
 
 DynamicLoaderPOSIXDYLD::DynamicLoaderPOSIXDYLD(Process *process)
     : DynamicLoader(process), m_rendezvous(process),
       m_load_offset(LLDB_INVALID_ADDRESS), m_entry_point(LLDB_INVALID_ADDRESS),
       m_auxv(), m_dyld_bid(LLDB_INVALID_BREAK_ID),
-      m_vdso_base(LLDB_INVALID_ADDRESS) {}
+      m_vdso_base(LLDB_INVALID_ADDRESS),
+      m_interpreter_base(LLDB_INVALID_ADDRESS) {}
 
 DynamicLoaderPOSIXDYLD::~DynamicLoaderPOSIXDYLD() {
   if (m_dyld_bid != LLDB_INVALID_BREAK_ID) {
     m_process->GetTarget().RemoveBreakpointByID(m_dyld_bid);
     m_dyld_bid = LLDB_INVALID_BREAK_ID;
   }
 }
 
if (log)
     log->Printf("DynamicLoaderPOSIXDYLD::%s pid %" PRIu64
                 " executable '%s', load_offset 0x%" PRIx64,
                 __FUNCTION__,
                 m_process ? m_process->GetID() : LLDB_INVALID_PROCESS_ID,
                 executable_sp ? executable_sp->GetFileSpec().GetPath().c_str()
                               : "<null executable>",
                 load_offset);
 
-  EvalVdsoStatus();
+  EvalSpecialModulesStatus();
 
   // if we dont have a load address we cant re-base
   bool rebase_exec = (load_offset == LLDB_INVALID_ADDRESS) ? false : true;
 
   // if we have a valid executable
   if (executable_sp.get()) {
     lldb_private::ObjectFile *obj = executable_sp->GetObjectFile();
     if (obj) {
void DynamicLoaderPOSIXDYLD::DidLaunch() {
 
   ModuleSP executable;
   addr_t load_offset;
 
   m_auxv.reset(new AuxVector(m_process));
 
   executable = GetTargetExecutable();
   load_offset = ComputeLoadOffset();
-  EvalVdsoStatus();
+  EvalSpecialModulesStatus();
 
   if (executable.get() && load_offset != LLDB_INVALID_ADDRESS) {
     ModuleList module_list;
     module_list.Append(executable);
     UpdateLoadedSections(executable, LLDB_INVALID_ADDRESS, load_offset, true);
 
     if (log)
       log->Printf("DynamicLoaderPOSIXDYLD::%s about to call ProbeEntry()",
                   __FUNCTION__);
+
+    if (!SetRendezvousBreakpoint()) {
+      // If we cannot establish rendezvous breakpoint right now
+      // we'll try again at enty point.
       ProbeEntry();
+    }
 
     m_process->GetTarget().ModulesDidLoad(module_list);
   }
 }
 
 Status DynamicLoaderPOSIXDYLD::CanLoadImage() { return Status(); }
 
 void DynamicLoaderPOSIXDYLD::UpdateLoadedSections(ModuleSP module,
if (log)
     log->Printf("DynamicLoaderPOSIXDYLD::%s pid %" PRIu64
                 " GetEntryPoint() returned address 0x%" PRIx64
                 ", setting entry breakpoint",
                 __FUNCTION__,
                 m_process ? m_process->GetID() : LLDB_INVALID_PROCESS_ID,
                 entry);
 
   if (m_process) {
     Breakpoint *const entry_break =
         m_process->GetTarget().CreateBreakpoint(entry, true, false).get();
     entry_break->SetCallback(EntryBreakpointHit, this, true);
     entry_break->SetBreakpointKind("shared-library-event");
 
     // Shoudn't hit this more than once.
     entry_break->SetOneShot(true);
   }
 }
if (log)
                   __FUNCTION__, break_id);
   }
 
   dyld_instance->LoadAllCurrentModules();
   dyld_instance->SetRendezvousBreakpoint();
   return false; // Continue running.
 }
 
-void DynamicLoaderPOSIXDYLD::SetRendezvousBreakpoint() {
+bool DynamicLoaderPOSIXDYLD::SetRendezvousBreakpoint() {
   Log *log(GetLogIfAnyCategoriesSet(LIBLLDB_LOG_DYNAMIC_LOADER));
+  if (m_dyld_bid != LLDB_INVALID_BREAK_ID) {
+    LLDB_LOG(log,
+             "Rendezvous breakpoint breakpoint id {0} for pid {1}"
+             "is already set.",
+             m_dyld_bid,
+             m_process ? m_process->GetID() : LLDB_INVALID_PROCESS_ID);
+    return true;
+  }
 
-  addr_t break_addr = m_rendezvous.GetBreakAddress();
+  addr_t break_addr;
   Target &target = m_process->GetTarget();
 
-  if (m_dyld_bid == LLDB_INVALID_BREAK_ID) {
-    if (log)
-      log->Printf("DynamicLoaderPOSIXDYLD::%s pid %" PRIu64
-                  " setting rendezvous break address at 0x%" PRIx64,
-                  __FUNCTION__,
-                  m_process ? m_process->GetID() : LLDB_INVALID_PROCESS_ID,
-                  break_addr);
-    Breakpoint *dyld_break =
-        target.CreateBreakpoint(break_addr, true, false).get();
-    dyld_break->SetCallback(RendezvousBreakpointHit, this, true);
-    dyld_break->SetBreakpointKind("shared-library-event");
-    m_dyld_bid = dyld_break->GetID();
-  } else {
-    if (log)
-      log->Printf("DynamicLoaderPOSIXDYLD::%s pid %" PRIu64
-                  " reusing break id %" PRIu32 ", address at 0x%" PRIx64,
-                  __FUNCTION__,
-                  m_process ? m_process->GetID() : LLDB_INVALID_PROCESS_ID,
-                  m_dyld_bid, break_addr);
-  }
+  if (m_rendezvous.IsValid()) {
+    break_addr = m_rendezvous.GetBreakAddress();
+  } else {
+    LLDB_LOG(log, "Rendezvous structure is not set up yet. "
+                  "Trying to locate rendezvous breakpoint in the interpreter "
+                  "by symbol name.");
+    ModuleSP interpreter = LoadInterpreterModule();
+    if (!interpreter) {
+      LLDB_LOG(log, "Can't find interpreter, rendezvous breakpoint isn't set.");
+      return false;
+    }
+
+    const Symbol *symbol = nullptr;
+    static const char *DebugStateCandidates[] = {
+        "_dl_debug_state", "rtld_db_dlactivity", "__dl_rtld_db_dlactivity",
+        "r_debug_state",   "_r_debug_state",     "_rtld_debug_state",
+    };
+
+    for (auto name : DebugStateCandidates) {
+      symbol = interpreter->FindFirstSymbolWithNameAndType(ConstString(name));
+      if (symbol)
+        break;
+    }
+    if (symbol == nullptr) {
+      LLDB_LOG(log, "Can't find debug state function in the interpreter.");
+      return false;
+    }
+    break_addr = symbol->GetLoadAddress(&m_process->GetTarget());
+    if (break_addr == LLDB_INVALID_ADDRESS) {
+      LLDB_LOG(log, "Can't resolve address of debug state function {0}.",
+               symbol->GetName());
+      return false;
+    } else {
+      LLDB_LOG(
+          log,
+          "Function {0} from module {1} will be used as rendezvous breakpoint.",
+          symbol->GetName(), interpreter->GetFileSpec().GetFilename());
+    }
+  }
+
+  LLDB_LOG(log, "Setting rendezvous break address for pid {0} at {1:x}",
+  if (dyld_break->GetNumResolvedLocations() == 0)
+    return false;
 
-  // Make sure our breakpoint is at the right address.
-  assert(target.GetBreakpointByID(m_dyld_bid)
-             ->FindLocationByAddress(break_addr)
-             ->GetBreakpoint()
-             .GetID() == m_dyld_bid);
+  m_dyld_bid = dyld_break->GetID();
+  return true;
 }
 
 bool DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit(
     void *baton, StoppointCallbackContext *context, user_id_t break_id,
     user_id_t break_loc_id) {
   assert(baton && "null baton");
   if (!baton)
     return false;
if (addrs.size() > 0) {
     std::sort(start, end);
     addrs.erase(std::unique(start, end), end);
     thread_plan_sp.reset(new ThreadPlanRunToAddress(thread, addrs, stop));
   }
 
   return thread_plan_sp;
 }
 
-void DynamicLoaderPOSIXDYLD::LoadVDSO(ModuleList &modules) {
+void DynamicLoaderPOSIXDYLD::LoadVDSO() {
   if (m_vdso_base == LLDB_INVALID_ADDRESS)
     return;
 
   FileSpec file("[vdso]", false);
 
   MemoryRegionInfo info;
   Status status = m_process->GetMemoryRegionInfo(m_vdso_base, info);
   if (status.Fail()) {
     Log *log(GetLogIfAnyCategoriesSet(LIBLLDB_LOG_DYNAMIC_LOADER));
     LLDB_LOG(log, "Failed to get vdso region info: {0}", status);
     return;
   }
 
   if (ModuleSP module_sp = m_process->ReadModuleFromMemory(
           file, m_vdso_base, info.GetRange().GetByteSize())) {
     UpdateLoadedSections(module_sp, LLDB_INVALID_ADDRESS, m_vdso_base, false);
     m_process->GetTarget().GetImages().AppendIfNeeded(module_sp);
   }
 }
 
+ModuleSP DynamicLoaderPOSIXDYLD::LoadInterpreterModule() {
+  if (m_interpreter_base == LLDB_INVALID_ADDRESS)
+    return nullptr;
+
+  MemoryRegionInfo info;
+  Target &target = m_process->GetTarget();
+  Status status = m_process->GetMemoryRegionInfo(m_interpreter_base, info);
+  if (status.Fail() || info.GetMapped() != MemoryRegionInfo::eYes ||
+      info.GetName().IsEmpty()) {
+    Log *log(GetLogIfAnyCategoriesSet(LIBLLDB_LOG_DYNAMIC_LOADER));
+    LLDB_LOG(log, "Failed to get interpreter region info: {0}", status);
+    return nullptr;
+  }
+
+  FileSpec file(info.GetName().GetCString(), false);
+  ModuleSpec module_spec(file, target.GetArchitecture());
+
+  if (ModuleSP module_sp = target.GetSharedModule(module_spec)) {
+    UpdateLoadedSections(module_sp, LLDB_INVALID_ADDRESS, m_interpreter_base,
+                         false);
+    return module_sp;
+  }
+  return nullptr;
+}
+
 void DynamicLoaderPOSIXDYLD::LoadAllCurrentModules() {
   DYLDRendezvous::iterator I;
   DYLDRendezvous::iterator E;
   ModuleList module_list;
+  Log *log(GetLogIfAnyCategoriesSet(LIBLLDB_LOG_DYNAMIC_LOADER));
 
   if (!m_rendezvous.Resolve()) {
-    Log *log(GetLogIfAnyCategoriesSet(LIBLLDB_LOG_DYNAMIC_LOADER));
     if (log)
       log->Printf("DynamicLoaderPOSIXDYLD::%s unable to resolve POSIX DYLD "
                   "rendezvous address",
                   __FUNCTION__);
     return;
   }
 
   // The rendezvous class doesn't enumerate the main module, so track
   // that ourselves here.
   ModuleSP executable = GetTargetExecutable();
   m_loaded_modules[executable] = m_rendezvous.GetLinkMapAddress();
-  LoadVDSO(module_list);
+  LoadVDSO();
 
   std::vector<FileSpec> module_names;
   for (I = m_rendezvous.begin(), E = m_rendezvous.end(); I != E; ++I)
     module_names.push_back(I->file_spec);
   m_process->PrefetchModuleSpecs(
       module_names, m_process->GetTarget().GetArchitecture().GetTriple());
 
   for (I = m_rendezvous.begin(), E = m_rendezvous.end(); I != E; ++I) {
     ModuleSP module_sp =
         LoadModuleAtAddress(I->file_spec, I->link_addr, I->base_addr, true);
     if (module_sp.get()) {
+      LLDB_LOG(log, "LoadAllCurrentModules loading module: {0}",
+               I->file_spec.GetFilename());
       module_list.Append(module_sp);
     } else {
       Log *log(GetLogIfAnyCategoriesSet(LIBLLDB_LOG_DYNAMIC_LOADER));
       if (log)
         log->Printf(
             "DynamicLoaderPOSIXDYLD::%s failed loading module %s at 0x%" PRIx64,
             __FUNCTION__, I->file_spec.GetCString(), I->base_addr);
     }
addr_t DynamicLoaderPOSIXDYLD::ComputeLoadOffset() {
 
   if (!file_entry.IsValid())
     return LLDB_INVALID_ADDRESS;
 
   m_load_offset = virt_entry - file_entry.GetFileAddress();
   return m_load_offset;
 }
 
-void DynamicLoaderPOSIXDYLD::EvalVdsoStatus() {
-  AuxVector::iterator I = m_auxv->FindEntry(AuxVector::AUXV_AT_SYSINFO_EHDR);
-
-  if (I != m_auxv->end())
+void DynamicLoaderPOSIXDYLD::EvalSpecialModulesStatus() {
+  auto I = m_auxv->FindEntry(AuxVector::AUXV_AT_SYSINFO_EHDR);
+  if (I != m_auxv->end() && I->value != 0)
     m_vdso_base = I->value;
+
+  I = m_auxv->FindEntry(AuxVector::AUXV_AT_BASE);
+  if (I != m_auxv->end() && I->value != 0)
+    m_interpreter_base = I->value;
 }
 
 addr_t DynamicLoaderPOSIXDYLD::GetEntryPoint() {
   if (m_entry_point != LLDB_INVALID_ADDRESS)
     return m_entry_point;
 
   if (m_auxv.get() == NULL)
     return LLDB_INVALID_ADDRESS;