[lldb] Support fetching symbols with dsymForUUID in the background

[JDevlieghere]

Summary

On macOS, LLDB uses the DebugSymbols.framework to locate symbol rich dSYM bundles. [1] The framework uses a variety of methods, one of them calling into a binary or shell script to locate (and download) dSYMs. Internally at Apple, that tool is called dsymForUUID and for simplicity I'm just going to refer to it that way here too, even though it can be be an arbitrary executable.

The most common use case for dsymForUUID is to fetch symbols from the network. This can take a long time, and because the calls to the DebugSymbols.framework are blocking, lldb does not launch the process immediately. This is the expected behavior and many people therefore often don't use this functionality, but instead use add-dsym when they want symbols for a given frame, backtrace or module. This is a little faster because you're only fetching symbols for the module you care about, but it's still a slow, blocking operation.

This patch introduces a hybrid approach between the two. When symbols.enable-background-lookup is enabled, lldb will do the equivalent of add-dsym for every module it doesn't have symbols for in the background. From the user's perspective there is no slowdown, because the process launches immediately, with whatever symbols are available and more symbol information is added over time as the background fetching completes.

[1] https://lldb.llvm.org/use/symbols.html

[jasonmolenda]

Interesting idea, it seems worth trying out and seeing how it works in practice.

My read of the llvm::ThreadPool is that this won't create an unbounded number of threads doing background fetches, right. Some of the big UI apps in macOS these days can have a thousand dylib/frameworks loaded in them.

[jasonmolenda]

I wonder if it might be interesting to target this a bit - only start background downloading the dSYM for a Module once it has appeared in a backtrace, for instance. Any UI application on macOS etc has around 500 libraries loaded, but I believe only a few dozen of those libraries will actually appear in backtraces as they're debugging. We could delay it until the UnwindTable ctor is called for instance -- that's the first thing that will happen when a Module is in a backtrace. Just thinking aloud, not arguing for it especially.

[JDevlieghere]

In D131328#3704791, @jasonmolenda wrote:

Interesting idea, it seems worth trying out and seeing how it works in practice.

My read of the llvm::ThreadPool is that this won't create an unbounded number of threads doing background fetches, right. Some of the big UI apps in macOS these days can have a thousand dylib/frameworks loaded in them.

Correct, the thread pool is limited by the number of cores/physical CPUs (depending on the ThreadPoolStrategy). I contemplated using a dedicated thread pool for this, but didn't like the idea of having to manage its lifetime and pick and arbitrary fixed number. The only risk here is that we starve other work scheduled on this ThreadPool (currently that's building the manual DWARF index, so not really a concern on our platform).

In D131328#3705187, @jasonmolenda wrote:

I wonder if it might be interesting to target this a bit - only start background downloading the dSYM for a Module once it has appeared in a backtrace, for instance. Any UI application on macOS etc has around 500 libraries loaded, but I believe only a few dozen of those libraries will actually appear in backtraces as they're debugging. We could delay it until the UnwindTable ctor is called for instance -- that's the first thing that will happen when a Module is in a backtrace. Just thinking aloud, not arguing for it especially.

I really like this suggestion. It did bother me how naive/wasteful the current approach is. The downside of doing this more lazily is that you potentially had enough time to fetch everything by the time you did the first backtrace. But it's equally likely that you're waiting for those one or two modules you did care about (and are part of the backtrace) while 200 others you don't care about are being pulled in. I think your suggestion hits the sweet spot in terms of latency and doing only meaningful work.

[JDevlieghere]

Implement Jason's suggestion of delaying this until we need the unwind table for a module.

[labath]

I see two problems with this patch.

• it makes shutting down lldb safely impossible (because you never know if there are any unfinished tasks running). We already have problems with shutting down (grep for "intentionally leaked"), but this is guaranteed to make the problem worse. Using the global thread pool would make things better, as then one could flush the pool in SBDebugger::Terminate, or something similar
• there seems to be a race between new debuggers/targets being added, and their existing instances being notified. I'm not 100% sure what will happen in this case, but it seems very easy to miss some debuggers (or targets) that were added after we took the "snapshot" of the current state

(There's also the (obvious) problem of any breakpoints that are set on code within those modules becoming effective at an unpredictable future data, but I am assuming you are aware of that.)

[JDevlieghere]

In D131328#3706202, @labath wrote:

I see two problems with this patch.

• it makes shutting down lldb safely impossible (because you never know if there are any unfinished tasks running). We already have problems with shutting down (grep for "intentionally leaked"), but this is guaranteed to make the problem worse. Using the global thread pool would make things better, as then one could flush the pool in SBDebugger::Terminate, or something similar

It *is* using the global thread pool, but it doesn't look like that's being flushed. I can rework that in a separate patch to make it fit the Initialize/Terminate pattern.

• there seems to be a race between new debuggers/targets being added, and their existing instances being notified. I'm not 100% sure what will happen in this case, but it seems very easy to miss some debuggers (or targets) that were added after we took the "snapshot" of the current state

As I (tried to) explain in the comment, since we set the symbol file in the module before taking the "snapshot", I don't think this is a problem. If a target was created after, it should already know about the symbol file. Did I miss something with that approach?

(There's also the (obvious) problem of any breakpoints that are set on code within those modules becoming effective at an unpredictable future data, but I am assuming you are aware of that.)

Yeah, but I don't see any way around that. The unpredictability is unfortunate, but from UX perspective it's not much worse than modules getting loaded at runtime (although hopefully that's more deterministic).

[labath]

In D131328#3706565, @JDevlieghere wrote:

In D131328#3706202, @labath wrote:

I see two problems with this patch.

• it makes shutting down lldb safely impossible (because you never know if there are any unfinished tasks running). We already have problems with shutting down (grep for "intentionally leaked"), but this is guaranteed to make the problem worse. Using the global thread pool would make things better, as then one could flush the pool in SBDebugger::Terminate, or something similar

It *is* using the global thread pool, but it doesn't look like that's being flushed. I can rework that in a separate patch to make it fit the Initialize/Terminate pattern.

Ok, I'm sorry. I got confused by the static ThreadPoolTaskGroup thingy. In either case, I think we should flush that pool during termination. The reason we didn't need that before is because this is the first time that we have these sorts of fully async tasks (the other task pool tasks are "flushing" themselves).

• there seems to be a race between new debuggers/targets being added, and their existing instances being notified. I'm not 100% sure what will happen in this case, but it seems very easy to miss some debuggers (or targets) that were added after we took the "snapshot" of the current state

As I (tried to) explain in the comment, since we set the symbol file in the module before taking the "snapshot", I don't think this is a problem. If a target was created after, it should already know about the symbol file. Did I miss something with that approach?

Right. I missed that part. I think this should be mostly fine. I mean, I suppose this can still race in the other "direction", where a target picks up the new symbol file, but we still end up calling SymbolsDidLoad manually. But hopefully that is ok (?)

(There's also the (obvious) problem of any breakpoints that are set on code within those modules becoming effective at an unpredictable future data, but I am assuming you are aware of that.)

Yeah, but I don't see any way around that. The unpredictability is unfortunate, but from UX perspective it's not much worse than modules getting loaded at runtime (although hopefully that's more deterministic).

Yeah, if that's the thing you want, then that's the thing you want. One could play some UX games, and add a mechanism to show the downloads that are in progress and/or force the completion of loading of some symbols of whatever, but there's no way to get rid of that completely.

[JDevlieghere]

There was another approach I considered initially which might alleviate some of the issues Pavel raised. Instead of using the global thread pool, that plan consisted of a dedicated thread to fetch symbols in the background. That would allow us to add the symbol info synchronously. For example on every stop we could ask if there are any new symbol files to add instead of trying to add the symbol info as soon as it becomes available.

[labath]

In D131328#3706749, @JDevlieghere wrote:

There was another approach I considered initially which might alleviate some of the issues Pavel raised. Instead of using the global thread pool, that plan consisted of a dedicated thread to fetch symbols in the background. That would allow us to add the symbol info synchronously. For example on every stop we could ask if there are any new symbol files to add instead of trying to add the symbol info as soon as it becomes available.

I kinda like that. Bonus points if you make that single thread download multiple files at the same time (using threads to paralelize downloading is not the best approach, since the task is not cpu-bound). :)

[JDevlieghere]

• Use a debugger event to execute the notification of the targets on the event handler thread

[JDevlieghere]

In D131328#3706814, @labath wrote:

In D131328#3706749, @JDevlieghere wrote:

There was another approach I considered initially which might alleviate some of the issues Pavel raised. Instead of using the global thread pool, that plan consisted of a dedicated thread to fetch symbols in the background. That would allow us to add the symbol info synchronously. For example on every stop we could ask if there are any new symbol files to add instead of trying to add the symbol info as soon as it becomes available.

I kinda like that.

I tried prototyping this. I wasn't super happy with the results:

• This one's minor, but compared to the current patch, there's a bunch of synchronization boilerplate: input/output mutex, condition variable, etc
• Who should own this class? Inherently it's tied to the shared module list, but that never gets destroyed, so that has the flushing issue. The only viable alternative is to have an instance controlled by Debugger::{Initialize/Terminate}, which is exactly what we do for the global thread pool in D131407.
• There's really no good place to poll for this in the target. You'd need to do it from the process.
• Once you poll for new symbol files, you can't flush the newly added modules from the download (unless you store them in the target) because other targets will need to be notified later.

All these things can be overcome, but that was the point where I decided that maybe it wasn't worth it.

Instead I focussed on avoiding doing all this work from an arbitrary thread and using a broadcast event to do it from the event handler thread.

Bonus points if you make that single thread download multiple files at the same time (using threads to paralelize downloading is not the best approach, since the task is not cpu-bound). :)

If LLDB was doing the downloading itself we might be able to do better, but by shelling out to dsymForUUID all we see in an opaque blocking call.

[jasonmolenda]

This is an awesome enhancement for environments where the dSYM can be located programmatically --- downloading only the dSYMs that are actually being used in a debug session makes it low overhead versus grabbing everything, and it's great to have them loaded for you when you actually want to look at them. I think the LocateSymbolFileMacOSX.cpp part of the patch needs to be rebased now that https://reviews.llvm.org/D131303 landed. I hope we can get this integrated, I can't wait to live on it.

[JDevlieghere]

Rebase on top of D131303

[jasonmolenda]

LGTM.

lldb/include/lldb/Core/Debugger.h

class Debugger : public std::enable_shared_from_this<Debugger>,
   friend class SourceManager; // For GetSourceFileCache.
 
 public:
   /// Broadcaster event bits definitions.
   enum {
     eBroadcastBitProgress = (1 << 0),
     eBroadcastBitWarning = (1 << 1),
     eBroadcastBitError = (1 << 2),
+    eBroadcastSymbolChange = (1 << 3),
   };
 
   static ConstString GetStaticBroadcasterClass();
 
   /// Get the public broadcaster for this debugger.
   Broadcaster &GetBroadcaster() { return m_broadcaster; }
   const Broadcaster &GetBroadcaster() const { return m_broadcaster; }
 
public:
   /// \param [in] once
   ///   If a pointer is passed to a std::once_flag, then it will be used to
   ///   ensure the given error is only broadcast once.
   static void
   ReportError(std::string messsage,
               llvm::Optional<lldb::user_id_t> debugger_id = llvm::None,
               std::once_flag *once = nullptr);
 
+  static void ReportSymbolChange(const ModuleSpec &module_spec);
+
 protected:
   friend class CommandInterpreter;
   friend class REPL;
   friend class Progress;
 
   /// Report progress events.
   ///
   /// Progress events will be delivered to any debuggers that have listeners

lldb/include/lldb/Core/DebuggerEvents.h

 //===-- DebuggerEvents.h ----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
+#include "lldb/Core/ModuleSpec.h"
 #include "lldb/Utility/ConstString.h"
 #include "lldb/Utility/Event.h"
 
 #include <string>
 
 #ifndef LLDB_CORE_DEBUGGER_EVENTS_H
 #define LLDB_CORE_DEBUGGER_EVENTS_H
 
protected:
   std::string m_message;
   Type m_type;
   const bool m_debugger_specific;
 
   DiagnosticEventData(const DiagnosticEventData &) = delete;
   const DiagnosticEventData &operator=(const DiagnosticEventData &) = delete;
 };
 
+class SymbolChangeEventData : public EventData {
+public:
+  SymbolChangeEventData(lldb::DebuggerWP debugger_wp, ModuleSpec module_spec)
+      : m_debugger_wp(debugger_wp), m_module_spec(std::move(module_spec)) {}
+
+  static ConstString GetFlavorString();
+  ConstString GetFlavor() const override;
+
+  static const SymbolChangeEventData *
+  GetEventDataFromEvent(const Event *event_ptr);
+
+  void DoOnRemoval(Event *event_ptr) override;
+
+private:
+  lldb::DebuggerWP m_debugger_wp;
+  ModuleSpec m_module_spec;
+
+  SymbolChangeEventData(const SymbolChangeEventData &) = delete;
+  const SymbolChangeEventData &
+  operator=(const SymbolChangeEventData &) = delete;
+};
+
 } // namespace lldb_private
 
 #endif // LLDB_CORE_DEBUGGER_EVENTS_H

lldb/include/lldb/Core/ModuleList.h

 
 public:
   ModuleListProperties();
 
   FileSpec GetClangModulesCachePath() const;
   bool SetClangModulesCachePath(const FileSpec &path);
   bool GetEnableExternalLookup() const;
   bool SetEnableExternalLookup(bool new_value);
+  bool GetEnableBackgroundLookup() const;
   bool GetEnableLLDBIndexCache() const;
   bool SetEnableLLDBIndexCache(bool new_value);
   uint64_t GetLLDBIndexCacheMaxByteSize();
   uint64_t GetLLDBIndexCacheMaxPercent();
   uint64_t GetLLDBIndexCacheExpirationDays();
   FileSpec GetLLDBIndexCachePath() const;
   bool SetLLDBIndexCachePath(const FileSpec &path);
 
GetSharedModule(const ModuleSpec &module_spec, lldb::ModuleSP &module_sp,
                   llvm::SmallVectorImpl<lldb::ModuleSP> *old_modules,
                   bool *did_create_ptr, bool always_create = false);
 
   static bool RemoveSharedModule(lldb::ModuleSP &module_sp);
 
   static void FindSharedModules(const ModuleSpec &module_spec,
                                 ModuleList &matching_module_list);
 
+  static lldb::ModuleSP FindSharedModule(const UUID &uuid);
+
   static size_t RemoveOrphanSharedModules(bool mandatory);
 
   static bool RemoveSharedModuleIfOrphaned(const Module *module_ptr);
 
   void ForEach(std::function<bool(const lldb::ModuleSP &module_sp)> const
                    &callback) const;
 
 protected:

lldb/include/lldb/Symbol/LocateSymbolFile.h

 //===-- LocateSymbolFile.h --------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLDB_SYMBOL_LOCATESYMBOLFILE_H
 #define LLDB_SYMBOL_LOCATESYMBOLFILE_H
 
 #include <cstdint>
 
 #include "lldb/Core/FileSpecList.h"
 #include "lldb/Utility/FileSpec.h"
 #include "lldb/Utility/Status.h"
+#include "lldb/lldb-forward.h"
 
 namespace lldb_private {
 
 class ArchSpec;
 class ModuleSpec;
 class UUID;
 
 class Symbols {
public:
   // repository, or using any other means necessary to locate both the
   // unstripped object file and the debug symbols. The force_lookup argument
   // controls whether the external program is called unconditionally to find
   // the symbol file, or if the user's settings are checked to see if they've
   // enabled the external program before calling.
   //
   static bool DownloadObjectAndSymbolFile(ModuleSpec &module_spec,
                                           Status &error,
-                                          bool force_lookup = true);
+                                          bool force_lookup = true,
+                                          bool copy_executable = true);
+
+  /// Locate the symbol file for the given UUID on a background thread. This
+  /// function returns immediately. Under the hood it uses the debugger's
+  /// thread pool to call DownloadObjectAndSymbolFile. If a symbol file is
+  /// found, this will notify all target which contain the module with the
+  /// given UUID.
+  static void DownloadSymbolFileAsync(const UUID &uuid);
 };
 
 } // namespace lldb_private
 
 #endif // LLDB_SYMBOL_LOCATESYMBOLFILE_H

lldb/include/lldb/Target/Target.h

public:
 
   bool GetEnableAutoApplyFixIts() const;
 
   uint64_t GetNumberOfRetriesWithFixits() const;
 
   bool GetEnableNotifyAboutFixIts() const;
 
   FileSpec GetSaveJITObjectsDir() const;
-  
   bool GetEnableSyntheticValue() const;
 
   uint32_t GetMaxZeroPaddingInFloatFormat() const;
 
   uint32_t GetMaximumNumberOfChildrenToDisplay() const;
 
   /// Get the max depth value, augmented with a bool to indicate whether the
   /// depth is the default.
private:
   void EnvVarsValueChangedCallback();
   void InputPathValueChangedCallback();
   void OutputPathValueChangedCallback();
   void ErrorPathValueChangedCallback();
   void DetachOnErrorValueChangedCallback();
   void DisableASLRValueChangedCallback();
   void InheritTCCValueChangedCallback();
   void DisableSTDIOValueChangedCallback();
-  
   // Settings checker for target.jit-save-objects-dir:
   void CheckJITObjectsDir();
 
   Environment ComputeEnvironment() const;
 
   // Member variables.
   ProcessLaunchInfo m_launch_info;
   std::unique_ptr<TargetExperimentalProperties> m_experimental_properties_up;
public:
   friend class Debugger;
 
   /// Broadcaster event bits definitions.
   enum {
     eBroadcastBitBreakpointChanged = (1 << 0),
     eBroadcastBitModulesLoaded = (1 << 1),
     eBroadcastBitModulesUnloaded = (1 << 2),
     eBroadcastBitWatchpointChanged = (1 << 3),
-    eBroadcastBitSymbolsLoaded = (1 << 4)
+    eBroadcastBitSymbolsLoaded = (1 << 4),
+    eBroadcastBitSymbolsChanged = (1 << 5),
   };
 
   // These two functions fill out the Broadcaster interface:
 
   static ConstString &GetStaticBroadcasterClass();
 
   ConstString &GetBroadcasterClass() const override {
     return GetStaticBroadcasterClass();
public:
   /// \param[in] module_sp
   ///     A shared pointer reference to the module that checked.
   ///
   /// \return \b true if the module should be excluded, \b false otherwise.
   bool
   ModuleIsExcludedForUnconstrainedSearches(const lldb::ModuleSP &module_sp);
 
   const ArchSpec &GetArchitecture() const { return m_arch.GetSpec(); }
-  
   /// Returns the name of the target's ABI plugin.
   llvm::StringRef GetABIName() const;
 
   /// Set the architecture for this target.
   ///
   /// If the current target has no Images read in, then this just sets the
   /// architecture, which will be used to select the architecture of the
   /// ExecutableModule when that is set. If the current target has an
public:
   /// Add a signal for the target.  This will get copied over to the process
   /// if the signal exists on that target.  Only the values with Yes and No are
   /// set, Calculate values will be ignored.
 protected:
   struct DummySignalValues {
     LazyBool pass = eLazyBoolCalculate;
     LazyBool notify = eLazyBoolCalculate;
     LazyBool stop = eLazyBoolCalculate;
-    DummySignalValues(LazyBool pass, LazyBool notify, LazyBool stop) : 
-        pass(pass), notify(notify), stop(stop) {}
+    DummySignalValues(LazyBool pass, LazyBool notify, LazyBool stop)
+        : pass(pass), notify(notify), stop(stop) {}
     DummySignalValues() = default;
   };
   using DummySignalElement = llvm::StringMapEntry<DummySignalValues>;
-  static bool UpdateSignalFromDummy(lldb::UnixSignalsSP signals_sp, 
+  static bool UpdateSignalFromDummy(lldb::UnixSignalsSP signals_sp,
                                     const DummySignalElement &element);
-  static bool ResetSignalFromDummy(lldb::UnixSignalsSP signals_sp, 
+  static bool ResetSignalFromDummy(lldb::UnixSignalsSP signals_sp,
                                    const DummySignalElement &element);
 
 public:
   /// Add a signal to the Target's list of stored signals/actions.  These
   /// values will get copied into any processes launched from
   /// this target.
-  void AddDummySignal(llvm::StringRef name, LazyBool pass, LazyBool print, 
+  void AddDummySignal(llvm::StringRef name, LazyBool pass, LazyBool print,
                       LazyBool stop);
   /// Updates the signals in signals_sp using the stored dummy signals.
   /// If warning_stream_sp is not null, if any stored signals are not found in
   /// the current process, a warning will be emitted here.
-  void UpdateSignalsFromDummy(lldb::UnixSignalsSP signals_sp, 
+  void UpdateSignalsFromDummy(lldb::UnixSignalsSP signals_sp,
                               lldb::StreamSP warning_stream_sp);
   /// Clear the dummy signals in signal_names from the target, or all signals
   /// if signal_names is empty.  Also remove the behaviors they set from the
-  /// process's signals if it exists. 
+  /// process's signals if it exists.
   void ClearDummySignals(Args &signal_names);
   /// Print all the signals set in this target.
   void PrintDummySignals(Stream &strm, Args &signals);
 
 protected:
   /// Implementing of ModuleList::Notifier.
 
   void NotifyModuleAdded(const ModuleList &module_list,
protected:
   bool m_suppress_stop_hooks; /// Used to not run stop hooks for expressions
   bool m_is_dummy_target;
   unsigned m_next_persistent_variable_index = 0;
   /// An optional \a lldb_private::Trace object containing processor trace
   /// information of this target.
   lldb::TraceSP m_trace_sp;
   /// Stores the frame recognizers of this target.
   lldb::StackFrameRecognizerManagerUP m_frame_recognizer_manager_up;
-  /// These are used to set the signal state when you don't have a process and 
+  /// These are used to set the signal state when you don't have a process and
   /// more usefully in the Dummy target where you can't know exactly what
   /// signals you will have.
   llvm::StringMap<DummySignalValues> m_dummy_signals;
 
   static void ImageSearchPathsChanged(const PathMappingList &path_list,
                                       void *baton);
 
   // Utilities for `statistics` command.

lldb/source/Core/CoreProperties.td

 include "../../include/lldb/Core/PropertiesBase.td"
 
 let Definition = "modulelist" in {
   def EnableExternalLookup: Property<"enable-external-lookup", "Boolean">,
     Global,
     DefaultTrue,
     Desc<"Control the use of external tools and repositories to locate symbol files. Directories listed in target.debug-file-search-paths and directory of the executable are always checked first for separate debug info files. Then depending on this setting: On macOS, Spotlight would be also used to locate a matching .dSYM bundle based on the UUID of the executable. On NetBSD, directory /usr/libdata/debug would be also searched. On platforms other than NetBSD directory /usr/lib/debug would be also searched.">;
+  def EnableBackgroundLookup: Property<"enable-background-lookup", "Boolean">,
+    Global,
+    DefaultFalse,
+    Desc<"On macOS, enable calling dsymForUUID (or an equivalent script/binary) in the background to locate symbol files that weren't found.">;
   def ClangModulesCachePath: Property<"clang-modules-cache-path", "FileSpec">,
     Global,
     DefaultStringValue<"">,
     Desc<"The path to the clang modules cache directory (-fmodules-cache-path).">;
   def SymLinkPaths: Property<"debug-info-symlink-paths", "FileSpecList">,
     Global,
     DefaultStringValue<"">,
     Desc<"Debug info path which should be resolved while parsing, relative to the host filesystem.">;

lldb/source/Core/Debugger.cpp

 //===-- Debugger.cpp ------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Core/Debugger.h"
 
 #include "lldb/Breakpoint/Breakpoint.h"
 #include "lldb/Core/DebuggerEvents.h"
 #include "lldb/Core/FormatEntity.h"
 #include "lldb/Core/Mangled.h"
 #include "lldb/Core/ModuleList.h"
+#include "lldb/Core/ModuleSpec.h"
 #include "lldb/Core/PluginManager.h"
 #include "lldb/Core/StreamAsynchronousIO.h"
 #include "lldb/Core/StreamFile.h"
 #include "lldb/DataFormatters/DataVisualization.h"
 #include "lldb/Expression/REPL.h"
 #include "lldb/Host/File.h"
 #include "lldb/Host/FileSystem.h"
 #include "lldb/Host/HostInfo.h"

 void Debugger::ReportError(std::string message,
                            llvm::Optional<lldb::user_id_t> debugger_id,
                            std::once_flag *once) {
 
   ReportDiagnosticImpl(DiagnosticEventData::Type::Error, std::move(message),
                        debugger_id, once);
 }
 
+void Debugger::ReportSymbolChange(const ModuleSpec &module_spec) {
+  if (g_debugger_list_ptr && g_debugger_list_mutex_ptr) {
+    std::lock_guard<std::recursive_mutex> guard(*g_debugger_list_mutex_ptr);
+    for (DebuggerSP debugger_sp : *g_debugger_list_ptr) {
+      EventSP event_sp = std::make_shared<Event>(
+          Debugger::eBroadcastSymbolChange,
+          new SymbolChangeEventData(debugger_sp, module_spec));
+      debugger_sp->GetBroadcaster().BroadcastEvent(event_sp);
+    }
+  }
+}
+
 static std::shared_ptr<LogHandler>
 CreateLogHandler(LogHandlerKind log_handler_kind, int fd, bool should_close,
                  size_t buffer_size) {
   switch (log_handler_kind) {
   case eLogHandlerStream:
     return std::make_shared<StreamLogHandler>(fd, should_close, buffer_size);
   case eLogHandlerCircular:
     return std::make_shared<RotatingLogHandler>(buffer_size);
listener_sp->StartListeningForEventSpec(m_broadcaster_manager_sp,
                                           thread_event_spec);
   listener_sp->StartListeningForEvents(
       m_command_interpreter_up.get(),
       CommandInterpreter::eBroadcastBitQuitCommandReceived |
           CommandInterpreter::eBroadcastBitAsynchronousOutputData |
           CommandInterpreter::eBroadcastBitAsynchronousErrorData);
 
   listener_sp->StartListeningForEvents(
-      &m_broadcaster,
-      eBroadcastBitProgress | eBroadcastBitWarning | eBroadcastBitError);
+      &m_broadcaster, eBroadcastBitProgress | eBroadcastBitWarning |
+                          eBroadcastBitError | eBroadcastSymbolChange);
 
   // Let the thread that spawned us know that we have started up and that we
   // are now listening to all required events so no events get missed
   m_sync_broadcaster.BroadcastEvent(eBroadcastBitEventThreadIsListening);
 
   bool done = false;
   while (!done) {
     EventSP event_sp;

lldb/source/Core/DebuggerEvents.cpp

 //===-- DebuggerEvents.cpp ------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Core/DebuggerEvents.h"
+#include "lldb/Core/Debugger.h"
+#include "lldb/Core/Module.h"
 #include "llvm/Support/WithColor.h"
 
 using namespace lldb_private;
+using namespace lldb;
 
 template <typename T>
 static const T *GetEventDataFromEventImpl(const Event *event_ptr) {
   if (event_ptr)
     if (const EventData *event_data = event_ptr->GetData())
       if (event_data->GetFlavor() == T::GetFlavorString())
         return static_cast<const T *>(event_ptr->GetData());
   return nullptr;

 ConstString DiagnosticEventData::GetFlavor() const {
   return DiagnosticEventData::GetFlavorString();
 }
 
 const DiagnosticEventData *
 DiagnosticEventData::GetEventDataFromEvent(const Event *event_ptr) {
   return GetEventDataFromEventImpl<DiagnosticEventData>(event_ptr);
 }
+
+ConstString SymbolChangeEventData::GetFlavorString() {
+  static ConstString g_flavor("SymbolChangeEventData");
+  return g_flavor;
+}
+
+ConstString SymbolChangeEventData::GetFlavor() const {
+  return SymbolChangeEventData::GetFlavorString();
+}
+
+const SymbolChangeEventData *
+SymbolChangeEventData::GetEventDataFromEvent(const Event *event_ptr) {
+  return GetEventDataFromEventImpl<SymbolChangeEventData>(event_ptr);
+}
+
+void SymbolChangeEventData::DoOnRemoval(Event *event_ptr) {
+  DebuggerSP debugger_sp(m_debugger_wp.lock());
+  if (!debugger_sp)
+    return;
+
+  for (TargetSP target_sp : debugger_sp->GetTargetList().Targets()) {
+    if (ModuleSP module_sp =
+            target_sp->GetImages().FindModule(m_module_spec.GetUUID())) {
+      {
+        std::lock_guard<std::recursive_mutex> guard(module_sp->GetMutex());
+        if (!module_sp->GetSymbolFileFileSpec())
+          module_sp->SetSymbolFileFileSpec(m_module_spec.GetSymbolFileSpec());
+      }
+      ModuleList module_list;
+      module_list.Append(module_sp);
+      target_sp->SymbolsDidLoad(module_list);
+    }
+  }
+}

lldb/source/Core/Module.cpp

 #include "lldb/Core/Section.h"
 #include "lldb/Host/FileSystem.h"
 #include "lldb/Host/Host.h"
 #include "lldb/Host/HostInfo.h"
 #include "lldb/Interpreter/CommandInterpreter.h"
 #include "lldb/Interpreter/ScriptInterpreter.h"
 #include "lldb/Symbol/CompileUnit.h"
 #include "lldb/Symbol/Function.h"
+#include "lldb/Symbol/LocateSymbolFile.h"
 #include "lldb/Symbol/ObjectFile.h"
 #include "lldb/Symbol/Symbol.h"
 #include "lldb/Symbol/SymbolContext.h"
 #include "lldb/Symbol/SymbolFile.h"
 #include "lldb/Symbol/SymbolVendor.h"
 #include "lldb/Symbol/Symtab.h"
 #include "lldb/Symbol/Type.h"
 #include "lldb/Symbol/TypeList.h"

 void Module::LookupInfo::Prune(SymbolContextList &sc_list,
                                size_t start_idx) const {
   if (m_match_name_after_lookup && m_name) {
     SymbolContext sc;
     size_t i = start_idx;
     while (i < sc_list.GetSize()) {
       if (!sc_list.GetContextAtIndex(i, sc))
         break;
-      
       bool keep_it =
           NameMatchesLookupInfo(sc.GetFunctionName(), sc.GetLanguage());
       if (keep_it)
         ++i;
       else
         sc_list.RemoveContextAtIndex(i);
     }
   }
void Module::SectionFileAddressesChanged() {
   ObjectFile *obj_file = GetObjectFile();
   if (obj_file)
     obj_file->SectionFileAddressesChanged();
   if (SymbolFile *symbols = GetSymbolFile())
     symbols->SectionFileAddressesChanged();
 }
 
 UnwindTable &Module::GetUnwindTable() {
-  if (!m_unwind_table)
+  if (!m_unwind_table) {
     m_unwind_table.emplace(*this);
+    if (!m_symfile_spec)
+      Symbols::DownloadSymbolFileAsync(GetUUID());
+  }
   return *m_unwind_table;
 }
 
 SectionList *Module::GetUnifiedSectionList() {
   if (!m_sections_up)
     m_sections_up = std::make_unique<SectionList>();
   return m_sections_up.get();
 }

lldb/source/Core/ModuleList.cpp

return m_collection_sp->GetPropertyAtIndexAsBoolean(
       nullptr, idx, g_modulelist_properties[idx].default_uint_value != 0);
 }
 
 bool ModuleListProperties::SetEnableExternalLookup(bool new_value) {
   return m_collection_sp->SetPropertyAtIndexAsBoolean(
       nullptr, ePropertyEnableExternalLookup, new_value);
 }
 
+bool ModuleListProperties::GetEnableBackgroundLookup() const {
+  const uint32_t idx = ePropertyEnableBackgroundLookup;
+  return m_collection_sp->GetPropertyAtIndexAsBoolean(
+      nullptr, idx, g_modulelist_properties[idx].default_uint_value != 0);
+}
+
 FileSpec ModuleListProperties::GetClangModulesCachePath() const {
   return m_collection_sp
       ->GetPropertyAtIndexAsOptionValueFileSpec(nullptr, false,
                                                 ePropertyClangModulesCachePath)
       ->GetCurrentValue();
 }
 
 bool ModuleListProperties::SetClangModulesCachePath(const FileSpec &path) {
bool ModuleList::ModuleIsInCache(const Module *module_ptr) {
   return false;
 }
 
 void ModuleList::FindSharedModules(const ModuleSpec &module_spec,
                                    ModuleList &matching_module_list) {
   GetSharedModuleList().FindModules(module_spec, matching_module_list);
 }
 
+lldb::ModuleSP ModuleList::FindSharedModule(const UUID &uuid) {
+  return GetSharedModuleList().FindModule(uuid);
+}
+
 size_t ModuleList::RemoveOrphanSharedModules(bool mandatory) {
   return GetSharedModuleList().RemoveOrphans(mandatory);
 }
 
 Status
 ModuleList::GetSharedModule(const ModuleSpec &module_spec, ModuleSP &module_sp,
                             const FileSpecList *module_search_paths_ptr,
                             llvm::SmallVectorImpl<lldb::ModuleSP> *old_modules,

lldb/source/Symbol/LocateSymbolFile.cpp

 //===-- LocateSymbolFile.cpp ----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Symbol/LocateSymbolFile.h"
 
+#include "lldb/Core/Debugger.h"
+#include "lldb/Core/Module.h"
 #include "lldb/Core/ModuleList.h"
 #include "lldb/Core/ModuleSpec.h"
 #include "lldb/Core/Progress.h"
 #include "lldb/Host/FileSystem.h"
 #include "lldb/Symbol/ObjectFile.h"
 #include "lldb/Utility/ArchSpec.h"
 #include "lldb/Utility/DataBuffer.h"
 #include "lldb/Utility/DataExtractor.h"
 #include "lldb/Utility/LLDBLog.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/StreamString.h"
 #include "lldb/Utility/Timer.h"
 #include "lldb/Utility/UUID.h"
 
+#include "llvm/ADT/SmallSet.h"
 #include "llvm/Support/FileSystem.h"
+#include "llvm/Support/ThreadPool.h"
 
 // From MacOSX system header "mach/machine.h"
 typedef int cpu_type_t;
 typedef int cpu_subtype_t;
 
 using namespace lldb;
 using namespace lldb_private;
 
for (size_t idx_file = 0; idx_file < num_files; ++idx_file) {
         }
       }
     }
   }
 
   return LocateExecutableSymbolFileDsym(module_spec);
 }
 
+void Symbols::DownloadSymbolFileAsync(const UUID &uuid) {
+  if (!ModuleList::GetGlobalModuleListProperties().GetEnableBackgroundLookup())
+    return;
+
+  static llvm::SmallSet<UUID, 8> g_seen_uuids;
+  static std::mutex g_mutex;
+  Debugger::GetThreadPool().async([=]() {
+    {
+      std::lock_guard<std::mutex> guard(g_mutex);
+      if (g_seen_uuids.count(uuid))
+        return;
+      g_seen_uuids.insert(uuid);
+    }
+
+    Status error;
+    ModuleSpec module_spec;
+    module_spec.GetUUID() = uuid;
+    if (!Symbols::DownloadObjectAndSymbolFile(module_spec, error,
+                                              /*force_lookup=*/true,
+                                              /*copy_executable=*/false))
+      return;
+
+    if (error.Fail())
+      return;
+
+    Debugger::ReportSymbolChange(module_spec);
+  });
+}
+
 #if !defined(__APPLE__)
 
 FileSpec Symbols::FindSymbolFileInBundle(const FileSpec &symfile_bundle,
                                          const lldb_private::UUID *uuid,
                                          const ArchSpec *arch) {
   // FIXME
   return FileSpec();
 }
 
 bool Symbols::DownloadObjectAndSymbolFile(ModuleSpec &module_spec,
-                                          Status &error, bool force_lookup) {
+                                          Status &error, bool force_lookup,
+                                          bool copy_executable) {
   // Fill in the module_spec.GetFileSpec() for the object file and/or the
   // module_spec.GetSymbolFileSpec() for the debug symbols file.
   return false;
 }
 
 #endif

lldb/source/Symbol/LocateSymbolFileMacOSX.cpp

std::call_once(g_once_flag, [&]() {
 
     // We couldn't find the dsymForUUID binary.
     g_dsymForUUID_executable = {};
   });
   return g_dsymForUUID_executable;
 }
 
 bool Symbols::DownloadObjectAndSymbolFile(ModuleSpec &module_spec,
-                                          Status &error, bool force_lookup) {
+                                          Status &error, bool force_lookup,
+                                          bool copy_executable) {
   const UUID *uuid_ptr = module_spec.GetUUIDPtr();
   const FileSpec *file_spec_ptr = module_spec.GetFileSpecPtr();
 
   llvm::StringRef dbgshell_command = GetDbgShellCommand();
 
   // When dbgshell_command is empty, the user has not enabled the use of an
   // external program to find the symbols, don't run it for them.
   if (!force_lookup && dbgshell_command.empty())
bool Symbols::DownloadObjectAndSymbolFile(ModuleSpec &module_spec,
   const std::string uuid_str = uuid_ptr ? uuid_ptr->GetAsString() : "";
   const std::string file_path_str =
       file_spec_ptr ? file_spec_ptr->GetPath() : "";
 
   Log *log = GetLog(LLDBLog::Host);
 
   // Create the dsymForUUID command.
   StreamString command;
+  const char *copy_executable_arg = copy_executable ? "--copyExecutable " : "";
   if (!uuid_str.empty()) {
-    command.Printf("%s --ignoreNegativeCache --copyExecutable %s",
-                   dsymForUUID_exe_path.c_str(), uuid_str.c_str());
+    command.Printf("%s --ignoreNegativeCache %s%s",
+                   dsymForUUID_exe_path.c_str(), copy_executable_arg,
+                   uuid_str.c_str());
     LLDB_LOGF(log, "Calling %s with UUID %s to find dSYM: %s",
               dsymForUUID_exe_path.c_str(), uuid_str.c_str(),
               command.GetString().data());
   } else if (!file_path_str.empty()) {
-    command.Printf("%s --ignoreNegativeCache --copyExecutable %s",
-                   dsymForUUID_exe_path.c_str(), file_path_str.c_str());
+    command.Printf("%s --ignoreNegativeCache %s%s",
+                   dsymForUUID_exe_path.c_str(), copy_executable_arg,
+                   file_path_str.c_str());
     LLDB_LOGF(log, "Calling %s with file %s to find dSYM: %s",
               dsymForUUID_exe_path.c_str(), file_path_str.c_str(),
               command.GetString().data());
   } else {
     return false;
   }
 
   // Invoke dsymForUUID.