Add functionality to load dynamic libraries temporarily

[Holman]

Summary

Previously, it was possible to load dynamic libraries which would be unloaded on llvm_shutdown(), but recently ManagedStatic removal changed this so that loaded libraries really can't ever be unloaded. This functionality was very useful, and so to add it back in a more explicit way, I've added new getLibrary() and closeLibrary() methods to allow callers to use the very convenient platform independent abstraction that LLVM has for dynamic libraries.

As a specific use case, the onnx-mlir project was using this functionality with an API that allows instancing LLVM so you can compile a shared library, and then load that library, and eventually close the instance (and library) and compile something else. This change to llvm_shutdown causes libraries to leak and also locks the libraries for the entire duration of the program which prevents reusing library names.

[lhames]

I think it makes sense to introduce a closeLibrary operation, and I'm ok with this as an initial implementation.

I'm a little worried about introducing the separate TemporaryHandles set since that will affect lookup order, but we never really promised a sensible lookup order before anyway.

Long term we should refactor sys::DynamicLibrary to (1) be the thinnest possible wrapper around the OS library load / close calls, (2) provide guarantees on lookup order if possible (should we match the order that libraries were opened in? or use system RTLD_GLOBAL or equivalent?). I don't think that kind of refactoring should block this patch though.

@nhaehnle -- There was no library unloading story in https://reviews.llvm.org/D129127 beyond "they'll all get unloaded when the process terminates", right? I think we need a bulk-unloading story. Would it make sense to add a destructor to the Globals struct so that if a library with sys::DynamicLibrary compiled in is closed then all libraries loaded through that interface are closed too?

llvm/include/llvm/Support/DynamicLibrary.h

98

Since you're touching these APIs anyway, could you add a note that they forward to the underlying OS library load/close operations, and that we make no guarantees about when those will actually close them library. (e.g. on Darwin, libraries containing Swift or ObjC will never be closed).

llvm/lib/Support/DynamicLibrary.cpp

49–50

&& binds tighter than || -- I think you'll want parens here.

[Holman]

Fix assert, add comments about unload semantics

[Holman]

In D134763#3821829, @lhames wrote:

I think it makes sense to introduce a closeLibrary operation, and I'm ok with this as an initial implementation.

I'm a little worried about introducing the separate TemporaryHandles set since that will affect lookup order, but we never really promised a sensible lookup order before anyway.

Long term we should refactor sys::DynamicLibrary to (1) be the thinnest possible wrapper around the OS library load / close calls, (2) provide guarantees on lookup order if possible (should we match the order that libraries were opened in? or use system RTLD_GLOBAL or equivalent?). I don't think that kind of refactoring should block this patch though.

I agree, I went this route because it was the least intrusive to the current API and I didn't fully understand the context of why getPermanentLibrary opens a handle and then closes the existing handle if one exists instead of something like the semantics I added, but I think I see now it was about lookup order.

@nhaehnle -- There was no library unloading story in https://reviews.llvm.org/D129127 beyond "they'll all get unloaded when the process terminates", right? I think we need a bulk-unloading story. Would it make sense to add a destructor to the Globals struct so that if a library with sys::DynamicLibrary compiled in is closed then all libraries loaded through that interface are closed too?

HandleSet has a destructor that I believe that should covers this case.

[Holman]

Also, I do not have commit access, so can you please help me get this merged if it looks ok?

llvm/include/llvm/Support/DynamicLibrary.h

 #include <string>
 
 namespace llvm {
 
 class StringRef;
 
 namespace sys {
 
-  /// This class provides a portable interface to dynamic libraries which also
-  /// might be known as shared libraries, shared objects, dynamic shared
-  /// objects, or dynamic link libraries. Regardless of the terminology or the
-  /// operating system interface, this class provides a portable interface that
-  /// allows dynamic libraries to be loaded and searched for externally
-  /// defined symbols. This is typically used to provide "plug-in" support.
-  /// It also allows for symbols to be defined which don't live in any library,
-  /// but rather the main program itself, useful on Windows where the main
-  /// executable cannot be searched.
-  ///
-  /// Note: there is currently no interface for temporarily loading a library,
-  /// or for unloading libraries when the LLVM library is unloaded.
-  class DynamicLibrary {
-    // Placeholder whose address represents an invalid library.
-    // We use this instead of NULL or a pointer-int pair because the OS library
-    // might define 0 or 1 to be "special" handles, such as "search all".
-    static char Invalid;
 
-    // Opaque data used to interface with OS-specific dynamic library handling.
-    void *Data;
 
-  public:
-    explicit DynamicLibrary(void *data = &Invalid) : Data(data) {}
 
-    /// Returns true if the object refers to a valid library.
-    bool isValid() const { return Data != &Invalid; }
 
-    /// Searches through the library for the symbol \p symbolName. If it is
-    /// found, the address of that symbol is returned. If not, NULL is returned.
-    /// Note that NULL will also be returned if the library failed to load.
-    /// Use isValid() to distinguish these cases if it is important.
-    /// Note that this will \e not search symbols explicitly registered by
-    /// AddSymbol().
-    void *getAddressOfSymbol(const char *symbolName);
 
-    /// This function permanently loads the dynamic library at the given path.
-    /// The library will only be unloaded when llvm_shutdown() is called.
-    /// This returns a valid DynamicLibrary instance on success and an invalid
-    /// instance on failure (see isValid()). \p *errMsg will only be modified
-    /// if the library fails to load.
-    ///
-    /// It is safe to call this function multiple times for the same library.
-    /// Open a dynamic library permanently.
-    static DynamicLibrary getPermanentLibrary(const char *filename,
-                                              std::string *errMsg = nullptr);
+  /// This function permanently loads the dynamic library at the given path
+  /// using the library load operation from the host operating system. The
+  /// library instance will only be closed when global destructors run, and
+  /// there is no guarantee when the library will be unloaded.
+  ///
+  /// instance on failure (see isValid()). \p *errMsg will only be modified if
+  /// the library fails to load.
 
-    /// Registers an externally loaded library. The library will be unloaded
-    /// when the program terminates.
-    ///
-    /// It is safe to call this function multiple times for the same library,
-    /// though ownership is only taken if there was no error.
-    ///
-    /// \returns An empty \p DynamicLibrary if the library was already loaded.
-    static DynamicLibrary addPermanentLibrary(void *handle,
-                                              std::string *errMsg = nullptr);
 
-    /// This function permanently loads the dynamic library at the given path.
-    /// Use this instead of getPermanentLibrary() when you won't need to get
-    /// symbols from the library itself.
-    ///
-    /// It is safe to call this function multiple times for the same library.
-    static bool LoadLibraryPermanently(const char *Filename,
-                                       std::string *ErrMsg = nullptr) {
-      return !getPermanentLibrary(Filename, ErrMsg).isValid();
-    }
 
-    enum SearchOrdering {
-      /// SO_Linker - Search as a call to dlsym(dlopen(NULL)) would when
-      /// DynamicLibrary::getPermanentLibrary(NULL) has been called or
-      /// search the list of explcitly loaded symbols if not.
-      SO_Linker,
-      /// SO_LoadedFirst - Search all loaded libraries, then as SO_Linker would.
-      SO_LoadedFirst,
-      /// SO_LoadedLast - Search as SO_Linker would, then loaded libraries.
-      /// Only useful to search if libraries with RTLD_LOCAL have been added.
-      SO_LoadedLast,
-      /// SO_LoadOrder - Or this in to search libraries in the ordered loaded.
-      /// The default bahaviour is to search loaded libraries in reverse.
-      SO_LoadOrder = 4
-    };
-    static SearchOrdering SearchOrder; // = SO_Linker
+  /// This function loads the dynamic library at the given path, using the
+  /// library load operation from the host operating system. The library
+  /// instance will be closed when closeLibrary is called or global destructors
+  /// are run, but there is no guarantee when the library will be unloaded.
+  ///
+  /// This returns a valid DynamicLibrary instance on success and an invalid
+  /// instance on failure (see isValid()). \p *Err will only be modified if the
+  /// library fails to load.
+  ///
+  /// It is safe to call this function multiple times for the same library.
+  static DynamicLibrary getLibrary(const char *FileName,
+                                   std::string *Err = nullptr);
+
+  /// This function closes the dynamic library at the given path, using the
+  /// library close operation of the host operating system, and there is no
+  /// guarantee if or when this will cause the the library to be unloaded.
+  ///
+  /// This function should be called only if the library was loaded using the
+  /// getLibrary() function.
+  static void closeLibrary(DynamicLibrary &Lib);
+
 
-    /// This function will search through all previously loaded dynamic
-    /// libraries for the symbol \p symbolName. If it is found, the address of
-    /// that symbol is returned. If not, null is returned. Note that this will
-    /// search permanently loaded libraries (getPermanentLibrary()) as well
-    /// as explicitly registered symbols (AddSymbol()).
-    /// @throws std::string on error.
-    /// Search through libraries for address of a symbol
-    static void *SearchForAddressOfSymbol(const char *symbolName);
 
-    /// Convenience function for C++ophiles.
-    static void *SearchForAddressOfSymbol(const std::string &symbolName) {
-      return SearchForAddressOfSymbol(symbolName.c_str());
-    }
 
-    /// This functions permanently adds the symbol \p symbolName with the
-    /// value \p symbolValue.  These symbols are searched before any
-    /// libraries.
-    /// Add searchable symbol/value pair.
-    static void AddSymbol(StringRef symbolName, void *symbolValue);
 
-    class HandleSet;
-  };
 
 } // End sys namespace
 } // End llvm namespace
 
 #endif

llvm/lib/Support/DynamicLibrary.cpp

public:
   ~HandleSet();
 
   HandleList::iterator Find(void *Handle) { return find(Handles, Handle); }
 
   bool Contains(void *Handle) {
     return Handle == Process || Find(Handle) != Handles.end();
   }
 
-  bool AddLibrary(void *Handle, bool IsProcess = false, bool CanClose = true) {
+  bool AddLibrary(void *Handle, bool IsProcess = false, bool CanClose = true,
+                  bool AllowDuplicates = false) {
 #ifdef _WIN32
     assert((Handle == this ? IsProcess : !IsProcess) && "Bad Handle.");
 #endif
+    assert((!AllowDuplicates || !CanClose) &&
+           "CanClose must be false if AllowDuplicates is true.");
 
     if (LLVM_LIKELY(!IsProcess)) {
-      if (Find(Handle) != Handles.end()) {
+      if (!AllowDuplicates && Find(Handle) != Handles.end()) {
         if (CanClose)
           DLClose(Handle);
         return false;
       }
       Handles.push_back(Handle);
     } else {
 #ifndef _WIN32
       if (Process) {
         if (CanClose)
           DLClose(Process);
         if (Process == Handle)
           return false;
       }
 #endif
       Process = Handle;
     }
     return true;
   }
 
+  void CloseLibrary(void *Handle) {
+    DLClose(Handle);
+    HandleList::iterator it = Find(Handle);
+    if (it != Handles.end()) {
+      Handles.erase(it);
+    }
+  }
+
   void *LibLookup(const char *Symbol, DynamicLibrary::SearchOrdering Order) {
     if (Order & SO_LoadOrder) {
       for (void *Handle : Handles) {
         if (void *Ptr = DLSym(Handle, Symbol))
           return Ptr;
       }
     } else {
       for (void *Handle : llvm::reverse(Handles)) {

 };
 
 namespace {
 
 struct Globals {
   // Collection of symbol name/value pairs to be searched prior to any
   // libraries.
   llvm::StringMap<void *> ExplicitSymbols;
-  // Collection of known library handles.
+  // Collections of known library handles.
   DynamicLibrary::HandleSet OpenedHandles;
-  // Lock for ExplicitSymbols and OpenedHandles.
+  DynamicLibrary::HandleSet OpenedTemporaryHandles;
+  // Lock for ExplicitSymbols, OpenedHandles, and OpenedTemporaryHandles.
   llvm::sys::SmartMutex<true> SymbolsMutex;
 };
 
 Globals &getGlobals() {
   static Globals G;
   return G;
 }
 
DynamicLibrary DynamicLibrary::addPermanentLibrary(void *Handle,
   // If we've already loaded this library, tell the caller.
   if (!G.OpenedHandles.AddLibrary(Handle, /*IsProcess*/ false,
                                   /*CanClose*/ false))
     *Err = "Library already loaded";
 
   return DynamicLibrary(Handle);
 }
 
+DynamicLibrary DynamicLibrary::getLibrary(const char *FileName,
+                                          std::string *Err) {
+  assert(FileName && "Use getPermanentLibrary() for opening process handle");
+  void *Handle = HandleSet::DLOpen(FileName, Err);
+  if (Handle != &Invalid) {
+    auto &G = getGlobals();
+    SmartScopedLock<true> Lock(G.SymbolsMutex);
+    G.OpenedTemporaryHandles.AddLibrary(Handle, /*IsProcess*/ false,
+                                        /*CanClose*/ false,
+                                        /*AllowDuplicates*/ true);
+  }
+  return DynamicLibrary(Handle);
+}
+
+void DynamicLibrary::closeLibrary(DynamicLibrary &Lib) {
+  auto &G = getGlobals();
+  SmartScopedLock<true> Lock(G.SymbolsMutex);
+  if (Lib.isValid()) {
+    G.OpenedTemporaryHandles.CloseLibrary(Lib.Data);
+    Lib.Data = &Invalid;
+  }
+}
+
 void *DynamicLibrary::getAddressOfSymbol(const char *SymbolName) {
   if (!isValid())
     return nullptr;
   return HandleSet::DLSym(Data, SymbolName);
 }
 
 void *DynamicLibrary::SearchForAddressOfSymbol(const char *SymbolName) {
   {
     auto &G = getGlobals();
     SmartScopedLock<true> Lock(G.SymbolsMutex);
 
     // First check symbols added via AddSymbol().
     StringMap<void *>::iterator i = G.ExplicitSymbols.find(SymbolName);
 
     if (i != G.ExplicitSymbols.end())
       return i->second;
 
     // Now search the libraries.
     if (void *Ptr = G.OpenedHandles.Lookup(SymbolName, SearchOrder))
       return Ptr;
+    if (void *Ptr = G.OpenedTemporaryHandles.Lookup(SymbolName, SearchOrder))
+      return Ptr;
   }
 
   return llvm::SearchForAddressOfSpecialSymbol(SymbolName);
 }
 
 //===----------------------------------------------------------------------===//
 // C API.
 //===----------------------------------------------------------------------===//