[OpenMP] [OMPT] [3/8] Implemented callback registration in libomptarget

[dhruvachak]

Summary

The purpose of this patch is to have tool-provided callbacks registered
in libomptarget. The overall design document is in
https://rice.app.box.com/s/pf3gix2hs4d4o1aatwir1set05xmjljc

Defined a class OmptDeviceCallbacksTy that will be used by libomptarget
and a plugin for callbacks registered by a tool. Once the callbacks are
registered in libomp, a lookup function is passed to libomptarget that is
used to retrieve the callbacks and register them in libomptarget.

Patch from John Mellor-Crummey <[e-mail]>
(With contributions from Dhruva Chakrabarti <[e-mail]>)

[dhruvachak]

Rebased

[dhruvachak]

Rebased

[dhruvachak]

Rebased. Please review.

[jplehr]

My comment about the naming applies to all objects / variables etc.
I'm unsure whether it would help readability, but I have in mind that it makes the external/user-facing API clearly distinguishable from the internal one.

openmp/libomptarget/include/ompt_device_callbacks.h

61

I wonder if we should limit the use of snake_case to only the OMP/OMPT interface functions and types, and stick with the LLVM naming conventions otherwise?

[dhruvachak]

Variable name changed.

[dhruvachak]

openmp/libomptarget/include/ompt_device_callbacks.h

61

Thanks for the comment. Changed.

[jplehr]

LGTM

[dhruvachak]

Anyone else has any comment before I land this patch?

[jmellorcrummey]

Woohoo!

John Mellor-Crummey Professor
Dept of Computer Science Rice University
email: [e-mail] phone: [phone]

[tianshilei1992]

Looks like this patch breaks the build.

FAILED: libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o
/home/ac.shilei.tian/Documents/build/llvm/release/bin/clang++ -DGTEST_HAS_RTTI=0 -DOMPTARGET_DEBUG -DOMPT_SUPPORT=1 -I/home/ac.shilei.tian/Documents/vscode/llvm-project/llvm/include -I/home/ac.shilei.tian/Documents/build/llvm/release/include -I/home/ac.shilei.tian/Documents/build/openmp/release/runtime/src -I/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-enum-constexpr-conversion -Wno-extra -Wno-pedantic -std=c++17 -O3 -DNDEBUG -fPIC  -fno-exceptions -fno-rtti -MD -MT libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o -MF libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o.d -o libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o -c /home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/ompt_callback.cpp
In file included from /home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/src/ompt_callback.cpp:22:
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
    FOREACH_OMPT_TARGET_CALLBACK(OmptBindCallback);
                                 ^
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
/home/ac.shilei.tian/Documents/vscode/llvm-project/openmp/libomptarget/include/ompt_device_callbacks.h:45:34: error: use of undeclared identifier 'DEBUG_PREFIX'
12 errors generated.

[dhruvachak]

In D123974#3965638, @tianshilei1992 wrote:

Looks like this patch breaks the build.

I did not see this problem in my build. I will investigate. For now, I reverted the commit.

[dhruvachak]

I can repro the failure with -DLIBOMPTARGET_ENABLE_DEBUG=ON added. I will update the patch.

[dhruvachak]

Fixed build failure with LIBOMPTARGET_ENABLE_DEBUG.

[dhruvachak]

@tianshilei1992 I tested builds w/ and w/o -DLIBOMPTARGET_ENABLE_DEBUG=ON. Any concern if I land the updated patch?

[tianshilei1992]

Please go ahead land it if it is clear on your side.

openmp/libomptarget/include/ompt_connector.h

 
 #ifdef OMPT_SUPPORT
 
 #include "llvm/Support/DynamicLibrary.h"
 
 #include <memory>
 #include <string>
 
-#include "Debug.h"
 #include "omp-tools.h"
+
+#include "Debug.h"
 #include "omptarget.h"
 
 #define DEBUG_PREFIX "OMPT"
-
 #define LIBOMPTARGET_STRINGIFY(s) #s
 
 /// Type for the function to be invoked for connecting two libraries.
 typedef void (*OmptConnectRtnTy)(ompt_start_tool_result_t *result);
 
 /// Establish connection between openmp runtime libraries
 ///
 /// This class is used to communicate between an OMPT implementation in

openmp/libomptarget/include/ompt_device_callbacks.h

+//===--------- ompt_device_callbacks.h - OMPT callbacks -- C++ ----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Interface used by both target-independent and device-dependent runtimes
+// to coordinate registration and invocation of OMPT callbacks
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _OMPT_DEVICE_CALLBACKS_H
+#define _OMPT_DEVICE_CALLBACKS_H
+
+#ifdef OMPT_SUPPORT
+
+#include "Debug.h"
+#include <omp-tools.h>
+
+#define DEBUG_PREFIX "OMPT"
+
+#define FOREACH_OMPT_TARGET_CALLBACK(macro)                                    \
+  FOREACH_OMPT_DEVICE_EVENT(macro)                                             \
+  FOREACH_OMPT_NOEMI_EVENT(macro)                                              \
+  FOREACH_OMPT_EMI_EVENT(macro)
+
+/// Internal representation for OMPT device callback functions.
+class OmptDeviceCallbacksTy {
+public:
+  /// Initialize the enabled flag and all the callbacks
+  void init() {
+    Enabled = false;
+#define initName(Name, Type, Code) Name##_fn = 0;
+    FOREACH_OMPT_TARGET_CALLBACK(initName)
+#undef initName
+  }
+
+  /// Used to register callbacks. \p Lookup is used to query a given callback
+  /// by name and the result is assigned to the corresponding callback function.
+  void registerCallbacks(ompt_function_lookup_t Lookup) {
+    Enabled = true;
+#define OmptBindCallback(Name, Type, Code)                                     \
+  Name##_fn = (Name##_t)Lookup(#Name);                                         \
+  DP("OMPT: class bound %s=%p\n", #Name, ((void *)(uint64_t)Name##_fn));
+
+    FOREACH_OMPT_TARGET_CALLBACK(OmptBindCallback);
+#undef OmptBindCallback
+  }
+
+private:
+  /// Set to true if callbacks for this library have been initialized
+  bool Enabled;
+
+  /// Callback functions
+#define DeclareName(Name, Type, Code) Name##_t Name##_fn;
+  FOREACH_OMPT_TARGET_CALLBACK(DeclareName)
+#undef DeclareName
+};
+
+/// Device callbacks object for the library that performs the instantiation
+extern OmptDeviceCallbacksTy OmptDeviceCallbacks;
+
+#undef DEBUG_PREFIX
+
+#endif // OMPT_SUPPORT
+
+#endif // _OMPT_DEVICE_CALLBACKS_H

openmp/libomptarget/src/ompt_callback.cpp

 #ifdef OMPT_SUPPORT
 
 #include <assert.h>
 #include <atomic>
 #include <cstdlib>
 #include <cstring>
 
 #include "omp-tools.h"
+
 #include "ompt_connector.h"
+#include "ompt_device_callbacks.h"
 #include "private.h"
 
 #define fnptr_to_ptr(x) ((void *)(uint64_t)x)
 
 /// Used to indicate whether OMPT was enabled for this library
 bool ompt_enabled = false;
+/// Object maintaining all the callbacks for this library
+OmptDeviceCallbacksTy OmptDeviceCallbacks;
 
 /// This is the function called by the higher layer (libomp) responsible
 /// for initializing OMPT in this library. This is passed to libomp
 /// as part of the OMPT connector object.
 /// \p lookup to be used to query callbacks registered with libomp
 /// \p initial_device_num Initial device num provided by libomp
 /// \p tool_data as provided by the tool
 static int ompt_libomptarget_initialize(ompt_function_lookup_t lookup,
                                         int initial_device_num,
                                         ompt_data_t *tool_data) {
   DP("enter ompt_libomptarget_initialize!\n");
   ompt_enabled = true;
-  // TODO use the parameters to populate callbacks in libomptarget
+  // The lookup parameter is provided by libomp which already has the
+  // tool callbacks registered at this point. The registration call
+  // below causes the same callback functions to be registered in
+  // libomptarget as well
+  OmptDeviceCallbacks.registerCallbacks(lookup);
   DP("exit ompt_libomptarget_initialize!\n");
   return 0;
 }
 
 static void ompt_libomptarget_finalize(ompt_data_t *data) {
   DP("enter ompt_libomptarget_finalize!\n");
   ompt_enabled = false;
   DP("exit ompt_libomptarget_finalize!\n");
__attribute__((constructor(102))) static void ompt_init(void) {
   static ompt_start_tool_result_t OmptResult;
 
   // Initialize OmptResult with the init and fini functions that will be
   // called by the connector
   OmptResult.initialize = ompt_libomptarget_initialize;
   OmptResult.finalize = ompt_libomptarget_finalize;
   OmptResult.tool_data.value = 0;
 
+  // Initialize the device callbacks first
+  OmptDeviceCallbacks.init();
+
   // Now call connect that causes the above init/fini functions to be called
   LibompConnector.connect(&OmptResult);
   DP("OMPT: Exit ompt_init\n");
 }
 
 #endif // OMPT_SUPPORT