[OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in nextgen plugins

[mhalk]

Summary

The purpose of this patch is to Implement registration of callback functions in the generic plugin by looking up corresponding callbacks in libomptarget. The overall design document is https://rice.app.box.com/s/pf3gix2hs4d4o1aatwir1set05xmjljc

Defined an object of type OmptDeviceCallbacksTy in the amdgpu plugin for holding the tool-provided callback functions. Implemented a global constructor in the plugin that creates a connector object to connect with libomptarget. The callbacks that are already registered with libomptarget are looked up and registered with the plugin.

Combined with an internal patch from Dhruva Chakrabarti, which fixes the OMPT initialization ordering.
Achieved through removal of the constructor attribute from ompt_init.

Patch from John Mellor-Crummey <[e-mail]>
With contributions from:
Dhruva Chakrabarti <[e-mail]>
Michael Halkenhaeuser <[e-mail]>

[dhruvachak]

Rebased

[jplehr]

openmp/libomptarget/plugins/amdgpu/src/ompt_callback.cpp

57 ↗

(On Diff #438531)

Minor change to be consistent with other functions printing the function name in these debug messages.

[jplehr]

Updating D124070: [OpenMP] [OMPT] [amdgpu] [4/8] Implemented callback registration in amdgpu plugin

Rebase and follow naming scheme of already landed patches.

[dhruvachak]

openmp/libomptarget/plugins/amdgpu/src/ompt_callback.cpp

22 ↗

(On Diff #496363)

We use OmptDeviceCallbacks variable name in libomptarget. Let's keep it consistent.

35 ↗

(On Diff #496363)

We should be able to move the ifdef to the top. Let's keep it consistent with libomptarget.

[jplehr]

Addressing feedback from Dhruva

[jplehr]

Address compile error in debug build.

[jplehr]

Rebased

[mhalk]

Update

Fix OMPT init ordering and library typo.

[mhalk]

Fixed wrong diff update.

[jhuber6]

These need to be updated for the next-gen plugins.

[mhalk]

Refactored patch for use with nextgen plugins.

[jplehr]

openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp

43

Functions and methods should start with a lower case letter and read as active voice, e.g., initializeOmptDevice.

[mhalk]

As discussed with Dhruva.

[dhruvachak]

openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp

68

The comment says it is a constructor but I don't see the attribute. Are you calling it directly from elsewhere in a subsequent patch?

[mhalk]

openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp

68

Yes, it will be called from PluginInterface (or GenericPluginTy::init() to be more precise) in patch 5/8 (https://reviews.llvm.org/D124652).
I'll remove the constructor comment.

[jdoerfert]

openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp

23

Can we put this stuff into a namespace (ompt::) rather than prefixing it with Ompt?

72

Multiple plugins can be active, are you sure you want static variables here?

openmp/libomptarget/src/rtl.cpp

75

Is this patch run through clang-format (only the patch part)?

[mhalk]

Rebase

[mhalk]

Corrected commit (apologies)

[dhruvachak]

openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp

68

I think LibomptargetConnector and OmptResult need not be statics, locals will suffice. The connector object is used to copy the callback function pointers from libomptarget and once that is done, the object is not required any more. The original intent of the static was to guard against accidental repeat connect attempts. If we add the assert to the finalizer like I suggested, that will still be guarded against. Regarding OmptResult, it is used to communicate some pointers to the other library which caches the pointers, so no need to keep the wrapper object around.

We could make similar changes to the connect logic in libomptarget and add an assert during finalizer registration in libomp but let's keep that for a separate commit, if at all.

openmp/libomptarget/src/OmptCallback.cpp

39

In registerRtl, before assigning to FiniFn, add an assert that it is nullptr.

[mhalk]

Implemented further feedback (removed statics & added corresponding assert)
edit: Please note that the namespace feedback has been addressed in the "next" patch in this stack: D12465

[dhruvachak]

LGTM. I understand that the namespace changes are already made in https://reviews.llvm.org/D124652 I think all other comments have been resolved.

@jdoerfert Okay to land this patch?

[mhalk]

Small fix w.r.t. debug builds.

[tianshilei1992]

I encountered a compile error when building openmp.

[4/143] Building CXX object libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o
FAILED: libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o
/gpfs/jlse-fs0/users/ac.shilei.tian/build/llvm/release/bin/clang++ -DGTEST_HAS_RTTI=0 -DOMPTARGET_DEBUG -I/home/ac.shilei.tian/Documents/vscode/llvm-project/llvm/include -I/gpfs/jlse-fs0/users/ac.shilei.tian/build/llvm/release/include -I/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-enum-constexpr-conversion -Wno-extra -Wno-pedantic -std=c++17 -g -fPIC  -fno-exceptions -funwind-tables -fno-rtti -MD -MT libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o -MF libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o.d -o libomptarget/src/CMakeFiles/omptarget.dir/OmptCallback.cpp.o -c /home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:124:32: error: unknown type name 'ompt_start_tool_result_t'
void ompt_libomptarget_connect(ompt_start_tool_result_t *result) {
                               ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:125:3: error: use of undeclared identifier 'DP'
  DP("OMPT: Enter ompt_libomptarget_connect\n");
  ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:126:7: error: use of undeclared identifier 'OmptEnabled'
  if (OmptEnabled && result) {
      ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:128:5: error: use of undeclared identifier 'LibraryFinalizer'
    LibraryFinalizer.registerRtl(result->finalize);
    ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:132:24: error: use of undeclared identifier 'OmptDeviceCallbacksTy'
    result->initialize(OmptDeviceCallbacksTy::doLookup,
                       ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/OmptCallback.cpp:135:3: error: use of undeclared identifier 'DP'
  DP("OMPT: Leave ompt_libomptarget_connect\n");
  ^
6 errors generated.

Do we want to revert it or a quick fix will be landed soon?

[tianshilei1992]

Okay, I see why it fails here. ompt_start_tool_result_t is in openmp/runtime/src/include/omp-tools.h.var, which will be generated when libomp is built. This makes it dependent on libomp, but the dependences are not set. I'm gonna revert the patch for now. I have another patch (https://reviews.llvm.org/D149617) that will make libomptarget depend on libomp, which can solve the issue automatically. It is reverted because it broke something but it will be landed soon.
Another thing is, I disabled OMPT on my side, and OmptCallback.cpp is added to the source file without any guard. That has to be fixed as well.

[ronlieb]

[AMD Official Use Only - General]

Revert sounds good, thx

Like that you have a related patch queued up that will help fix OMPT-4 patch.

[mhalk]

@tianshilei1992 Thank you very much!
Sorry that I was unavailable for the last hour, just got at the PC.

I'll have a look at the errors and see how I can reproduce them, but I'm honestly surprised (since I built this on multiple systems without problems).

Another thing is, I disabled OMPT on my side, and OmptCallback.cpp is added to the source file without any guard. That has to be fixed as well.

Just to make sure, you're talking about a guard that e.g. prevents OmptCallback.cpp to produce an object file when OMPT is not enabled (e.g. in the CMakeLists), correct?

[dhruvachak]

openmp/libomptarget/src/OmptCallback.cpp

141

This should be at the end of this file since omp-tools.h is included depending on whether OMPT is enabled. Missed this earlier.

[tianshilei1992]

Either only adding the source file when OMPT is enabled, or as suggested by @dhruvachak, guarding the entire file with the OMPT_SUPPORT.

[mhalk]

In D124070#4313476, @tianshilei1992 wrote:

Either only adding the source file when OMPT is enabled, or as suggested by @dhruvachak, guarding the entire file with the OMPT_SUPPORT.

Thanks for getting back so fast -- I'll look into the latter approach of moving #endif // OMPT_SUPPORT to the bottom of OmptCallback.cpp and re-check with different configurations (esp. disabled OMPT).

[mhalk]

Made #ifdef span OmptCallback.cpp to avoid build errors when OMPT is disabled.
Added dummy definition for exported symbol ompt_libomptarget_connect.

[tianshilei1992]

openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt

85

A better practice is to link OMPT if it is enabled.

if (OMPT_ENABLED)
  target_link_libraries(omptarget.rtl.amdgpu.nextgen PRIVATE OMPT)
endif()

[mhalk]

If there are no further problems or requests, I would like to re-land this revision.

openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt

85

Thank you, will be addressed!

[tianshilei1992]

openmp/libomptarget/src/OmptCallback.cpp

137

Will it cause some link error if ompt_libomptarget_connect is not defined but mentioned in exports?

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

137

Yes, also with the guarded target_link_libraries(...) you suggested -- I just tried it.
If there's another approach or better practice, I'd be very interested.

[dhruvachak]

openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt

85

For this to work, you probably want to check
if ((OMPT_TARGET_DEFAULT) AND (LIBOMPTARGET_OMPT_SUPPORT))
the same way libomptarget cmake does.

The former needs to be exported to global scope first in libomptarget. The latter should already be available to the plugin.

[mhalk]

Added conditional linking of OMPT to PluginInterface as well as amdgpu and cuda plugins`.

[mhalk]

openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt

85

Fhe correct values were available (without previous export of OMPT_TARGET_DEFAULT to global scope) when I checked against the different configurations -DLIBOMP[TARGET]_OMPT_SUPPORT=ON|OFF within CMakeLists.txt of PluginInterface and the amdgpu plugin.

[dhruvachak]

LGTM.

[mhalk]

@tianshilei1992 If it is not too much to ask for: Could you please build the current patch with your configuration and let me know if there were issues?

So far, I have not encountered any problems and would like to re-land this revision.

[tianshilei1992]

I think the patch looks good, though I didn't test it yet. I don't want to block here. Go for it. :-)

[mhalk]

In D124070#4320259, @tianshilei1992 wrote:

I think the patch looks good, though I didn't test it yet. I don't want to block here. Go for it. :-)

Much appreciated -- thank you!
However, I wanted to be available if there should be issues.

openmp/libomptarget/include/ompt_device_callbacks.h

 #define OmptBindCallback(Name, Type, Code)                                     \
   Name##_fn = (Name##_t)Lookup(#Name);                                         \
   DP("OMPT: class bound %s=%p\n", #Name, ((void *)(uint64_t)Name##_fn));
 
     FOREACH_OMPT_TARGET_CALLBACK(OmptBindCallback);
 #undef OmptBindCallback
   }
 
+  /// Used to find a callback given its name
+  ompt_interface_fn_t lookupCallback(const char *InterfaceFunctionName) {
+#define OmptLookup(Name, Type, Code)                                           \
+  if (strcmp(InterfaceFunctionName, #Name) == 0)                               \
+    return (ompt_interface_fn_t)Name##_fn;
+
+    FOREACH_OMPT_TARGET_CALLBACK(OmptLookup);
+#undef OmptLookup
+    return (ompt_interface_fn_t) nullptr;
+  }
+
+  /// Wrapper function to find a callback given its name
+  static ompt_interface_fn_t doLookup(const char *InterfaceFunctionName);
+
 private:
   /// Set to true if callbacks for this library have been initialized
   bool Enabled;
 
   /// Callback functions
 #define DeclareName(Name, Type, Code) Name##_t Name##_fn;
   FOREACH_OMPT_TARGET_CALLBACK(DeclareName)
 #undef DeclareName
 };
 
 /// Device callbacks object for the library that performs the instantiation
 extern OmptDeviceCallbacksTy OmptDeviceCallbacks;
 
-#undef DEBUG_PREFIX
-
 #endif // OMPT_SUPPORT
 
 #endif // _OMPT_DEVICE_CALLBACKS_H

openmp/libomptarget/plugins-nextgen/CMakeLists.txt

add_llvm_library("omptarget.rtl.${tmachine_libname}.nextgen"
       ADDITIONAL_HEADER_DIRS
       ${LIBOMPTARGET_INCLUDE_DIR}
       ${LIBOMPTARGET_DEP_LIBFFI_INCLUDE_DIR}
 
         LINK_LIBS
         PRIVATE
         elf_common
         MemoryManager
+        OMPT
         PluginInterface
         ${LIBOMPTARGET_DEP_LIBFFI_LIBRARIES}
         ${OPENMP_PTHREAD_LIB}
 
       NO_INSTALL_RPATH
     )
 
     if (LIBOMP_HAVE_VERSION_SCRIPT_FLAG)

openmp/libomptarget/plugins-nextgen/amdgpu/CMakeLists.txt

add_llvm_library(omptarget.rtl.amdgpu.nextgen SHARED
   LINK_COMPONENTS
   Support
   Object
 
   LINK_LIBS
   PRIVATE
   elf_common
   MemoryManager
   PluginInterface
   ${LIBOMPTARGET_DEP_LIBRARIES}
   ${OPENMP_PTHREAD_LIB}
   ${LDFLAGS_UNDEFINED}
 
   NO_INSTALL_RPATH
 )
 
+if ((OMPT_TARGET_DEFAULT) AND (LIBOMPTARGET_OMPT_SUPPORT))
+  target_link_libraries(omptarget.rtl.amdgpu.nextgen PRIVATE OMPT)
+endif()
+
 if (LIBOMP_HAVE_VERSION_SCRIPT_FLAG)
   target_link_libraries(omptarget.rtl.amdgpu.nextgen PRIVATE
     "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/../exports")
 endif()
 
 target_include_directories(
   omptarget.rtl.amdgpu.nextgen
   PRIVATE

openmp/libomptarget/plugins-nextgen/common/CMakeLists.txt

 ##===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 ##===----------------------------------------------------------------------===##
 #
 # Common parts which can be used by all plugins
 #
 ##===----------------------------------------------------------------------===##
 
+add_subdirectory(OMPT)
 add_subdirectory(PluginInterface)

openmp/libomptarget/plugins-nextgen/common/OMPT/CMakeLists.txt

 ##===----------------------------------------------------------------------===##
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 #
 ##===----------------------------------------------------------------------===##
 #
-# Common parts which can be used by all plugins
+# Aggregation of parts which can be used by OpenMP tools
 #
 ##===----------------------------------------------------------------------===##
 
-# NOTE: Don't try to build `PluginInterface` using `add_llvm_library` because we
-# don't want to export `PluginInterface` while `add_llvm_library` requires that.
-add_library(PluginInterface OBJECT
-  PluginInterface.cpp GlobalHandler.cpp JIT.cpp)
-
-# Only enable JIT for those targets that LLVM can support.
-string(TOUPPER "${LLVM_TARGETS_TO_BUILD}" TargetsSupported)
-foreach(Target ${TargetsSupported})
-  target_compile_definitions(PluginInterface PRIVATE "LIBOMPTARGET_JIT_${Target}")
-endforeach()
+# NOTE: Don't try to build `OMPT` using `add_llvm_library` because we
+# don't want to export `OMPT` while `add_llvm_library` requires that.
+add_library(OMPT OBJECT
+  OmptCallback.cpp)
 
 # This is required when using LLVM libraries.
-llvm_update_compile_flags(PluginInterface)
+llvm_update_compile_flags(OMPT)
 
 if (LLVM_LINK_LLVM_DYLIB)
   set(llvm_libs LLVM)
 else()
   llvm_map_components_to_libnames(llvm_libs
     ${LLVM_TARGETS_TO_BUILD}
     AggressiveInstCombine
     Analysis
llvm_map_components_to_libnames(llvm_libs
     Support
     Target
     TargetParser
     TransformUtils
     Vectorize
   )
 endif()
 
-target_link_libraries(PluginInterface
+target_link_libraries(OMPT
   PUBLIC
     ${llvm_libs}
     elf_common
     MemoryManager
 )
 
 # Define the TARGET_NAME and DEBUG_PREFIX.
-target_compile_definitions(PluginInterface PRIVATE
-  TARGET_NAME="PluginInterface"
-  DEBUG_PREFIX="PluginInterface"
+target_compile_definitions(OMPT PRIVATE
+  TARGET_NAME="OMPT"
+  DEBUG_PREFIX="OMPT"
 )
 
-target_include_directories(PluginInterface
+target_include_directories(OMPT
   INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}
   PRIVATE ${LIBOMPTARGET_INCLUDE_DIR}
 )
 
-set_target_properties(PluginInterface PROPERTIES
+set_target_properties(OMPT PROPERTIES
   POSITION_INDEPENDENT_CODE ON
   CXX_VISIBILITY_PRESET protected)

openmp/libomptarget/plugins-nextgen/common/OMPT/OmptCallback.cpp

+//===---------- OmptCallback.cpp - Generic OMPT callbacks --------- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// OMPT support for PluginInterface
+//
+//===----------------------------------------------------------------------===//
+
+#ifdef OMPT_SUPPORT
+#include <atomic>
+#include <cstdio>
+#include <string.h>
+#include <vector>
+
+#include "Debug.h"
+#include "ompt_connector.h"
+#include "ompt_device_callbacks.h"
+
+/// Object maintaining all the callbacks in the plugin
+OmptDeviceCallbacksTy OmptDeviceCallbacks;
+
+/// Lookup function used for querying callback functions maintained
+/// by the plugin
+ompt_interface_fn_t
+OmptDeviceCallbacksTy::doLookup(const char *InterfaceFunctionName) {
+  // TODO This will be populated with device tracing functions
+  return (ompt_interface_fn_t) nullptr;
+}
+
+/// Used to indicate whether OMPT was enabled for this library
+static bool OmptEnabled = false;
+
+/// This function is passed to libomptarget as part of the OMPT connector
+/// object. It is called by libomptarget during initialization of OMPT in the
+/// plugin. \p lookup to be used to query callbacks registered with libomptarget
+/// \p initial_device_num Initial device num provided by libomptarget
+/// \p tool_data as provided by the tool
+static int OmptDeviceInit(ompt_function_lookup_t lookup, int initial_device_num,
+                          ompt_data_t *tool_data) {
+  DP("OMPT: Enter OmptDeviceInit\n");
+  OmptEnabled = true;
+  // The lookup parameter is provided by libomptarget which already has the tool
+  // callbacks registered at this point. The registration call below causes the
+  // same callback functions to be registered in the plugin as well.
+  OmptDeviceCallbacks.registerCallbacks(lookup);
+  DP("OMPT: Exit OmptDeviceInit\n");
+  return 0;
+}
+
+/// This function is passed to libomptarget as part of the OMPT connector
+/// object. It is called by libomptarget during finalization of OMPT in the
+/// plugin.
+static void OmptDeviceFini(ompt_data_t *tool_data) {
+  DP("OMPT: Executing OmptDeviceFini\n");
+}
+
+/// Used to initialize callbacks implemented by the tool. This interface will
+/// lookup the callbacks table in libomptarget and assign them to the callbacks
+/// table maintained in the calling plugin library.
+void OmptCallbackInit() {
+  DP("OMPT: Entering OmptCallbackInit\n");
+  /// Connect plugin instance with libomptarget
+  OmptLibraryConnectorTy LibomptargetConnector("libomptarget");
+  ompt_start_tool_result_t OmptResult;
+
+  // Initialize OmptResult with the init and fini functions that will be
+  // called by the connector
+  OmptResult.initialize = OmptDeviceInit;
+  OmptResult.finalize = OmptDeviceFini;
+  OmptResult.tool_data.value = 0;
+
+  // Initialize the device callbacks first
+  OmptDeviceCallbacks.init();
+
+  // Now call connect that causes the above init/fini functions to be called
+  LibomptargetConnector.connect(&OmptResult);
+  DP("OMPT: Exiting OmptCallbackInit\n");
+}
+#endif

openmp/libomptarget/plugins-nextgen/common/PluginInterface/CMakeLists.txt

 
 target_link_libraries(PluginInterface
   PUBLIC
     ${llvm_libs}
     elf_common
     MemoryManager
 )
 
+if ((OMPT_TARGET_DEFAULT) AND (LIBOMPTARGET_OMPT_SUPPORT))
+  target_link_libraries(PluginInterface PUBLIC OMPT)
+endif()
+
 # Define the TARGET_NAME and DEBUG_PREFIX.
 target_compile_definitions(PluginInterface PRIVATE
   TARGET_NAME="PluginInterface"
   DEBUG_PREFIX="PluginInterface"
 )
 
 target_include_directories(PluginInterface
   INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}
   PRIVATE ${LIBOMPTARGET_INCLUDE_DIR}
 )
 
 set_target_properties(PluginInterface PROPERTIES
   POSITION_INDEPENDENT_CODE ON
   CXX_VISIBILITY_PRESET protected)

openmp/libomptarget/plugins-nextgen/cuda/CMakeLists.txt

add_llvm_library(omptarget.rtl.cuda.nextgen SHARED
   elf_common
   MemoryManager
   PluginInterface
   ${OPENMP_PTHREAD_LIB}
 
   NO_INSTALL_RPATH
 )
 
+if ((OMPT_TARGET_DEFAULT) AND (LIBOMPTARGET_OMPT_SUPPORT))
+  target_link_libraries(omptarget.rtl.cuda.nextgen PRIVATE OMPT)
+endif()
+
 if (LIBOMP_HAVE_VERSION_SCRIPT_FLAG)
   target_link_libraries(omptarget.rtl.cuda.nextgen PRIVATE
   "-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/../exports,-z,defs")
 endif()
 
 
 if(LIBOMPTARGET_DEP_CUDA_FOUND AND NOT LIBOMPTARGET_FORCE_DLOPEN_LIBCUDA)
   libomptarget_say("Building CUDA plugin linked against libcuda")

openmp/libomptarget/src/CMakeLists.txt

 add_llvm_library(omptarget
   SHARED
 
   api.cpp
   device.cpp
   interface.cpp
   interop.cpp
   omptarget.cpp
-  ompt_callback.cpp
+  OmptCallback.cpp
   rtl.cpp
   LegacyAPI.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LIBOMPTARGET_INCLUDE_DIR}
 
   LINK_COMPONENTS
   Support

openmp/libomptarget/src/OmptCallback.cpp

-//===-- ompt_callback.cpp - Target independent OpenMP target RTL -- C++ -*-===//
+//===-- OmptCallback.cpp - Target independent OpenMP target RTL --- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of OMPT callback interfaces for target independent layer
 //
 //===----------------------------------------------------------------------===//
 
 #ifdef OMPT_SUPPORT
 
 #include <assert.h>
 #include <atomic>
 #include <cstdlib>
 #include <cstring>
 
 #include "omp-tools.h"
 
+#include "Debug.h"
 #include "ompt_connector.h"
 #include "ompt_device_callbacks.h"
 #include "private.h"
 
 #define fnptr_to_ptr(x) ((void *)(uint64_t)x)
 
 /// Used to indicate whether OMPT was enabled for this library
-bool ompt_enabled = false;
+bool OmptEnabled = false;
 /// Object maintaining all the callbacks for this library
 OmptDeviceCallbacksTy OmptDeviceCallbacks;
 
+/// Used to maintain the finalization function that is received
+/// from the plugin during connect
+class LibomptargetRtlFinalizer {
+public:
+  LibomptargetRtlFinalizer() : RtlFinalization(nullptr) {}
+  void registerRtl(ompt_finalize_t FinalizationFunction) {
+    assert((RtlFinalization == nullptr) &&
+           "RTL finalization may only be registered once");
+    RtlFinalization = FinalizationFunction;
+  }
+  void finalize() {
+    if (RtlFinalization)
+      RtlFinalization(nullptr /* tool_data */);
+    RtlFinalization = nullptr;
+  }
+
+private:
+  ompt_finalize_t RtlFinalization;
+};
+
+/// Object that will maintain the RTL finalizer from the plugin
+static LibomptargetRtlFinalizer LibraryFinalizer;
+
+/// Lookup function to be used by libomptarget library
+ompt_interface_fn_t
+OmptDeviceCallbacksTy::doLookup(const char *InterfaceFunctionName) {
+  return OmptDeviceCallbacks.lookupCallback(InterfaceFunctionName);
+}
+
 /// This is the function called by the higher layer (libomp) responsible
 /// for initializing OMPT in this library. This is passed to libomp
 /// as part of the OMPT connector object.
 /// \p lookup to be used to query callbacks registered with libomp
 /// \p initial_device_num Initial device num provided by libomp
 /// \p tool_data as provided by the tool
 static int ompt_libomptarget_initialize(ompt_function_lookup_t lookup,
                                         int initial_device_num,
                                         ompt_data_t *tool_data) {
   DP("enter ompt_libomptarget_initialize!\n");
-  ompt_enabled = true;
+  OmptEnabled = true;
   // The lookup parameter is provided by libomp which already has the
   // tool callbacks registered at this point. The registration call
   // below causes the same callback functions to be registered in
   // libomptarget as well
   OmptDeviceCallbacks.registerCallbacks(lookup);
   DP("exit ompt_libomptarget_initialize!\n");
   return 0;
 }
 
+/// This function is passed to libomp as part of the OMPT connector object.
+/// It is called by libomp during finalization of OMPT in libomptarget.
 static void ompt_libomptarget_finalize(ompt_data_t *data) {
   DP("enter ompt_libomptarget_finalize!\n");
-  ompt_enabled = false;
+  // Before disabling OMPT, call the finalizer (of the plugin) that was
+  // registered with this library
+  LibraryFinalizer.finalize();
+  OmptEnabled = false;
   DP("exit ompt_libomptarget_finalize!\n");
 }
 
 /*****************************************************************************
  * constructor
  *****************************************************************************/
 /// Used to initialize callbacks implemented by the tool. This interface
 /// will lookup the callbacks table in libomp and assign them to the callbacks
-/// maintained in libomptarget. Using priority 102 to have this constructor
-/// run after the init target library constructor with priority 101 (see
-/// rtl.cpp).
-__attribute__((constructor(102))) static void ompt_init(void) {
-  DP("OMPT: Enter ompt_init\n");
+/// maintained in libomptarget.
+void InitOmptLibomp() {
+  DP("OMPT: Enter InitOmptLibomp\n");
   // Connect with libomp
   static OmptLibraryConnectorTy LibompConnector("libomp");
   static ompt_start_tool_result_t OmptResult;
 
   // Initialize OmptResult with the init and fini functions that will be
   // called by the connector
   OmptResult.initialize = ompt_libomptarget_initialize;
   OmptResult.finalize = ompt_libomptarget_finalize;
   OmptResult.tool_data.value = 0;
 
   // Initialize the device callbacks first
   OmptDeviceCallbacks.init();
 
   // Now call connect that causes the above init/fini functions to be called
   LibompConnector.connect(&OmptResult);
-  DP("OMPT: Exit ompt_init\n");
+  DP("OMPT: Exit InitOmptLibomp\n");
 }
 
+extern "C" {
+/// Used for connecting libomptarget with a plugin
+void ompt_libomptarget_connect(ompt_start_tool_result_t *result) {
+  DP("OMPT: Enter ompt_libomptarget_connect\n");
+  if (OmptEnabled && result) {
+    // Cache the fini function so that it can be invoked on exit
+    LibraryFinalizer.registerRtl(result->finalize);
+    // Invoke the provided init function with the lookup function maintained
+    // in this library so that callbacks maintained by this library are
+    // retrieved.
+    result->initialize(OmptDeviceCallbacksTy::doLookup,
+                       0 /* initial_device_num */, nullptr /* tool_data */);
+  }
+  DP("OMPT: Leave ompt_libomptarget_connect\n");
+}
+}
+#else
+extern "C" {
+/// Dummy definition when OMPT is disabled
+void ompt_libomptarget_connect() {}
+}
 #endif // OMPT_SUPPORT

openmp/libomptarget/src/exports

global:
     omp_get_interop_ptr;
     omp_get_interop_str;
     omp_get_interop_int;
     omp_get_interop_name;
     omp_get_interop_type_desc;
     __tgt_interop_init;
     __tgt_interop_use;
     __tgt_interop_destroy;
+    ompt_libomptarget_connect;
   local:
     *;
 };

openmp/libomptarget/src/rtl.cpp

static const char *RTLNames[] = {
     /* AMDGPU target        */ "libomptarget.rtl.amdgpu",
     /* Remote target        */ "libomptarget.rtl.rpc",
 };
 
 PluginManager *PM;
 
 static char *ProfileTraceFile = nullptr;
 
+#ifdef OMPT_SUPPORT
+extern void InitOmptLibomp();
+#endif
+
 __attribute__((constructor(101))) void init() {
   DP("Init target library!\n");
 
   bool UseEventsForAtomicTransfers = true;
   if (const char *ForceAtomicMap = getenv("LIBOMPTARGET_MAP_FORCE_ATOMIC")) {
     std::string ForceAtomicMapStr(ForceAtomicMap);
     if (ForceAtomicMapStr == "false" || ForceAtomicMapStr == "FALSE")
       UseEventsForAtomicTransfers = false;
     else if (ForceAtomicMapStr != "true" && ForceAtomicMapStr != "TRUE")
       fprintf(stderr,
               "Warning: 'LIBOMPTARGET_MAP_FORCE_ATOMIC' accepts only "
               "'true'/'TRUE' or 'false'/'FALSE' as options, '%s' ignored\n",
               ForceAtomicMap);
   }
 
   PM = new PluginManager(UseEventsForAtomicTransfers);
 
   ProfileTraceFile = getenv("LIBOMPTARGET_PROFILE");
   // TODO: add a configuration option for time granularity
   if (ProfileTraceFile)
     timeTraceProfilerInitialize(500 /* us */, "libomptarget");
 
+  #ifdef OMPT_SUPPORT
+    // Initialize OMPT first
+    InitOmptLibomp();
+  #endif
+
   PM->RTLs.loadRTLs();
   PM->registerDelayedLibraries();
 }
 
 __attribute__((destructor(101))) void deinit() {
   DP("Deinit target library!\n");
   delete PM;