[OPENMP] Load plugins from same directory as the libomptarget.so and quick fail mechanism for offloading plugins

[gregrodgers]

Summary

This patch loads plugins from the same directory as libomptarget.so is found. This avoids the need to set LD_LIBRARY_PATH to find plugins whose relative names are hardcoded in this source file.

This patch also provides a quick fail mechanism to avoid doing a dlopen for a platform that is not operational.

This patch also adds the name for the amdgcn plugin.

[ABataev]

openmp/libomptarget/src/rtl.cpp

67

Why do we want to do this? Also, does it mean that cmake configuration must be changed to put plugins in the same directory as libomptarget.so?

72

Why need to use dynamic allocation here?

[JonChesterfield]

@Ravi you're marked as needs changes but I can't see any comments

openmp/libomptarget/src/rtl.cpp

67

Setting LD_LIBRARY_PATH in order to work is nasty. I didn't realise that was the current status.

Loading relative to libomptarget is nicely position independent. Do you have a preferred location?

72

To keep it off the stack, but char[256] seems fine as a local to me too.

Is there a cross platform equivalent to this in the llvm libraries? Seems a bit suspect that dlinfo doesn't take the size of the buffer it writes to.

Plus I thought PATH_MAX was 256 and Google suggests Linux sets it to 260 now, so a wrapper with associated ready made error handling would be nice.

93

I'm not as sure about this. Probably faster, but is dlopen that slow when the file doesn't exist? Seems to run at terminal print speed when I'm debugging

[RaviNarayanaswamy]

I don't recall saying needs changes.

[JonChesterfield]

In D73657#1848238, @RaviNarayanaswamy wrote:

I don't recall saying needs changes.

Apologies, I have misread phabricator. You have an icon that means 'blocking review', not 'request changes'. Thanks!

[Hahnfeld]

As a general comment, I think this patch should be split into at least two separate ones:

1. Load plugins from the same directory as libomptarget.so is found.
2. Quick fail mechanism, although I'm not sure how much this saves? In most cases, we won't have the plugins for a foreign architecture (PPC on x86) anyway, that might only be a concern for accelerators.

I would strongly prefer if adding the amdgcn plugin was part of upstreaming that plugin, otherwise this change is just pointless.

openmp/libomptarget/src/rtl.cpp

69

handle is never closed.

70–71

AFAICT DP doesn't return, so the code will just fall through and case issues down below.

73–74

Again, that's not a thorough error handling.

86

Are you sure that incrementing inside the for loop is legal? I don't know what C++ says about the number of evaluations.

In any case, the code would be more clear if RTLQuickCheckFiles was merged into RTLNames, these two arrays are supposed to belong together.

95–97

You need to handle the case where the code can't find out the location of libomptarget.so. Plus I think the library should allow the user to load plugins from different locations by setting LD_LIBRARY_PATH. At the very least this would preserve compatibility, but it will also be helpful while developing out-of-tree plugins.

[gregrodgers]

I will rework this patch to

• Try dlopen on relative library name first to for LD_LIBRARY_PATH search. If that fails, I will load using full path name.
• Reorganize the names arrays into a single array to avoid a counter.

openmp/libomptarget/src/rtl.cpp

69

can it be closed? This code is in this dl . That is why we are sure it is loaded.

72

Sadly, nothing in the OpenMP runtime uses llvm libraries.
We don't know the size till dlinfo gives us the name. Hmmm, maybe there is a dlinfo on the size of the name.

86

Good idea, that would make for better maintenance and avoid this counter. I will fix this.

93

I am sure this is much faster. Doing a stat on a synthetic file is much faster than opening a real file that may exist but is useless to the running platform. Then the dlopen will attempt to load the useless file. Without this fix, users will be discouraged from building portable applications because of the potential failure on platform A from platform B if application was built for platforms A and B.

95–97

This code src/rtl.cpp is part of libomptarget.so. and it is already loaded. How can it not be found.? But I do like the idea of doing a dlopen on the relative file name, then if that fails use the full_plugin_name. So I will fix this. This extra path search is another good reason to have a quick fail method that skips non-active platforms.

[jdoerfert]

openmp/libomptarget/src/rtl.cpp

72

This all sounds scary. I am by far no expert in this but maybe PATH_MAX is the right size here, see als llvm/lib/Support/Unix/Path.inc

[lebedev.ri]

openmp/libomptarget/src/rtl.cpp

72

It is most certainly not 260, and not 256 on linux, but 4095.
On hurd it is righfully not defined at all - the code should not be making assumptions
about the upper bound on it's length, because clearly it is easy to fall short.

[protze.joachim]

Not sure whether greg still plans to work on this. Anyway a inline comment to the PATH_MAX

openmp/libomptarget/src/rtl.cpp

72

260 seems to be the limit on windows according to this blog article: http://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html

Since dlinfo does not use a size argument, I'd consider this prone to buffer overflow. Or in the best case you get a truncated path.
RTLD_DI_LINKMAP might be the better choice, as it should provide you access to the original name.

[Hahnfeld]

Superseded by D87413, I think.

[gregrodgers]

No longer using this approach

openmp/libomptarget/src/rtl.cpp

 #include "rtl.h"
 
 #include <cassert>
 #include <cstdlib>
 #include <cstring>
 #include <dlfcn.h>
 #include <mutex>
 #include <string>
+#include <sys/stat.h>
 
 // List of all plugins that can support offloading.
 static const char *RTLNames[] = {
     /* PowerPC target */ "libomptarget.rtl.ppc64.so",
     /* x86_64 target  */ "libomptarget.rtl.x86_64.so",
     /* CUDA target    */ "libomptarget.rtl.cuda.so",
+    /* HSA target     */ "libomptarget.rtl.hsa.so",
     /* AArch64 target */ "libomptarget.rtl.aarch64.so"};
 
+// Define the platform quick check files.
+// At least one must be found to attempt to load plugin for that platform.
+#define MAX_PLATFORM_CHECK_FILES 2
+static const char *RTLQuickCheckFiles[][MAX_PLATFORM_CHECK_FILES] = {
+    /* ppc64 has multiple quick check files */
+    {"/sys/firmware/devicetree/base/ibm,firmware-versions/open-power",
+     "/sys/firmware/devicetree/base/cpus/ibm,powerpc-cpu-features"},
+    /* acpi is unique to x86       */ {"/sys/firmware/acpi"},
+    /* nvidia0 is unique with cuda */ {"/dev/nvidia0"},
+    /* kfd is unique to amdgcn     */ {"/dev/kfd"},
+    /* More arm check files needed */ {"/sys/module/mdio_thunder/initstate"}};
+
 RTLsTy RTLs;
 std::mutex RTLsMtx;
 
 HostEntriesBeginToTransTableTy HostEntriesBeginToTransTable;
 std::mutex TrlTblMtx;
 
 HostPtrToTableMapTy HostPtrToTableMap;
 std::mutex TblMapMtx;
 
 void RTLsTy::LoadRTLs() {
 #ifdef OMPTARGET_DEBUG
   if (char *envStr = getenv("LIBOMPTARGET_DEBUG")) {
     DebugLevel = std::stoi(envStr);
   }
 #endif // OMPTARGET_DEBUG
 
   // Parse environment variable OMP_TARGET_OFFLOAD (if set)
   TargetOffloadPolicy = (kmp_target_offload_kind_t) __kmpc_get_target_offload();
   if (TargetOffloadPolicy == tgt_disabled) {
     return;
   }
 
+  // Load plugins from same directory as libomptarget.so
+  std::string full_plugin_name;
+  void *handle = dlopen("libomptarget.so", RTLD_NOW);
+  if (!handle)
+    DP("dlopen() failed: %s\n", dlerror());
+  char *libomptarget_dir_name = new char[256];
+  if (dlinfo(handle, RTLD_DI_ORIGIN, libomptarget_dir_name) == -1)
+    DP("RTLD_DI_ORIGIN failed: %s\n", dlerror());
+  struct stat stat_buffer;
+  int platform_num = 0;
+
   DP("Loading RTLs...\n");
 
   // Attempt to open all the plugins and, if they exist, check if the interface
   // is correct and if they are supporting any devices.
   for (auto *Name : RTLNames) {
-    DP("Loading library '%s'...\n", Name);
-    void *dynlib_handle = dlopen(Name, RTLD_NOW);
+
+    // Only one quick check file required to attempt to load platform plugin
+    bool found = false;
+    for (auto *QuickCheckName : RTLQuickCheckFiles[platform_num++]) {
+      if (QuickCheckName) {
+        if (!strcmp(QuickCheckName, "") ||
+            (stat(QuickCheckName, &stat_buffer) == 0))
+          found = true;
+      }
+    }
+    if (!found) // Not finding quick check files is a faster fail than dlopen
+      continue;
+    full_plugin_name.assign(libomptarget_dir_name).append("/").append(Name);
+    DP("Loading library '%s'...\n", full_plugin_name.c_str());
+    void *dynlib_handle = dlopen(full_plugin_name.c_str(), RTLD_NOW);
 
     if (!dynlib_handle) {
       // Library does not exist or cannot be found.
-      DP("Unable to load library '%s': %s!\n", Name, dlerror());
+      DP("Unable to load '%s': %s!\n", full_plugin_name.c_str(), dlerror());
       continue;
     }
 
-    DP("Successfully loaded library '%s'!\n", Name);
+    DP("Successfully loaded library '%s'!\n", full_plugin_name.c_str());
 
     // Retrieve the RTL information from the runtime library.
     RTLInfoTy R;
 
     R.LibraryHandler = dynlib_handle;
     R.isUsed = false;
 
 #ifdef OMPTARGET_DEBUG
#endif
 
     DP("Registering RTL %s supporting %d devices!\n",
         R.RTLName.c_str(), R.NumberOfDevices);
 
     // The RTL is valid! Will save the information in the RTLs list.
     AllRTLs.push_back(R);
   }
 
+  delete libomptarget_dir_name;
   DP("RTLs loaded!\n");
 
   return;
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 // Functionality for registering libs