[OpenMP] [OMPT] [7/8] Invoke tool-supplied callbacks before and after target launch and data transfer operations

[mhalk]

Summary

Implemented RAII objects, initialized at target entry points, that
invoke tool-supplied callbacks. Updated status of target callbacks as
implemented.

Depends on D127365

Patch from John Mellor-Crummey <[e-mail]>
(With contributions from Dhruva Chakrabarti <[e-mail]>)

[dhruvachak]

Rebased

[jplehr]

openmp/libomptarget/src/device.cpp

562

I wonder if these RAII objects are constructed when OMPT support is not built, or if it is optimized away in such cases?

609

In the asynchronous case, the lifetime of the RAII object is probably too short to account for the actual data transfer events.

636

In the asynchronous case, the lifetime of the RAII object is probably too short to account for the actual data transfer events.

[mhalk]

Rebased

[mhalk]

Refactored patch for use with nextgen plugins.

[mhalk]

As discussed with Dhruva.

[jdoerfert]

openmp/libomptarget/src/device.cpp

44

Style, also names of the arguments.

openmp/libomptarget/src/interface.cpp

86

These RAIIs should stay together. If you make them templated you could even get rid of the individual members and potentially have a single RAII to rule them all. (you'd pass the callback rather than the enum, or have a separate switch for the enum to callback translation, either way, you'd then unpack the tuple for the call, see std::apply)

openmp/libomptarget/src/omptarget.cpp

42

if you move the conditional into the con/destructor and you create a common base class for these, you should be able to eliminate all but the logic dealing with the provided values (here NumTeams).

1680

*, not +

[mhalk]

Rebase + implemented feedback so far. Thanks @jdoerfert!
Exception: RAII objects have not been consolidated, yet.

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

54

Some sort of init flag will be needed.
Originally thought the timeline would allow to go with a separate patch.
See D155186

openmp/libomptarget/src/device.cpp

48

Oversight: Six occurrences -- will be removed.

[jdoerfert]

openmp/libomptarget/src/OmptCallback.cpp

39

Why is the comment gone?

openmp/libomptarget/src/omptarget.cpp

45

Commented code?

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

39

Has been moved to its declaration in openmp/libomptarget/src/OmptInterface.h.

openmp/libomptarget/src/omptarget.cpp

45

Yes, my bad -- as stated in my other comment there are six occurrences, all of which will be removed upon the next change.
performIfOmptInitialized already checks the Variable before calling [begin|end]Submit().

[mhalk]

openmp/libomptarget/src/omptarget.cpp

45

FYI: I' will adapt this "part" of the patch (using an Init flag) upon the landing of D155186.

[mhalk]

Implemented templated RAII class.

Please, take a look if this is the desired direction.
If the approach should be changed, let me know.

Other than that, there are a few outstanding issues:
(Currently looking into the first two.)

• Warnings are currently emitted for each RAII instance

warning: 'InterfaceRAII' may not intend to support class template argument
         deduction [-Wctad-maybe-unsupported]
note: add a deduction guide to suppress this warning

• Test coverage should be expanded for target region operations: enter_data, exit_data, update
  • Those cases are handled by the RAII -- but simply not tested.
  • Will add another lit test in another upcoming review.
• Asynchronous timing does not work correctly as stated in a previous comment. RAII objects will be prematurely destroyed in this use case. Please, let me know if this is a requirement for accepting this patch.

[jdoerfert]

Add -Wno-ctad-maybe-unsupported to our cmake config or add sth like:

template <typename FunctionPairTy, typename... ArgsTy>
  InterfaceRAII(FunctionPairTy FunctionPair, ArgsTy... Args) -> InterfaceRAII< FunctionPairTy, ArgsTy...>;

We can fix async in a follow up.

[dhruvachak]

openmp/libomptarget/src/device.cpp

609

I think the asynchronous case is not an issue for the callbacks. The begin and end callbacks are supposed to be invoked before and after the entry-point invocation. The actual data transfer may complete later and is supposed to be captured by device tracing, something that is not part of this patch.

[mhalk]

Fixed warning related to template deduction -- thanks @jdoerfert
Fixed constant numbers being used in target data operations
NFC / improved naming of InterfaceRAII

Known issue:
Missing test coverage for target region operations: enter_data, exit_data, update
Callback my fire before the actual device init and report wrong device number

• Saw this with enter_data when performing a datamapping within a new testcase

Currently, I plan on landing this within the next five hours.
If there are concerns or further feedback, let me know.

*ping* @jdoerfert @dhruvachak

[dhruvachak]

LGTM.

[tianshilei1992]

I don't think the code in common path in this patch has been guarded correctly. It causes build error if OMPT is disabled. Please fix it or get it reverted if it takes long time.

[mhalk]

Thanks @tianshilei1992 for bringing this to my attention (again)!
Sorry for the inconvenience, I will try to fix this ASAP.

[mhalk]

Fixed build errors when OMPT support was disabled.

[jplehr]

LGTM, assuming build works

openmp/libomptarget/src/omptarget.cpp

1682

nit: Why?

[mhalk]

openmp/libomptarget/src/omptarget.cpp

1682

Because we're already checking via #ifdef OMPT_SUPPORT

openmp/libomptarget/include/OmptCallback.h

 #undef DEBUG_PREFIX
 #define DEBUG_PREFIX "OMPT"
 
 #define FOREACH_OMPT_TARGET_CALLBACK(macro)                                    \
   FOREACH_OMPT_DEVICE_EVENT(macro)                                             \
   FOREACH_OMPT_NOEMI_EVENT(macro)                                              \
   FOREACH_OMPT_EMI_EVENT(macro)
 
+#define performIfOmptInitialized(stmt)                                         \
+  do {                                                                         \
+    if (llvm::omp::target::ompt::Initialized) {                                \
+      stmt;                                                                    \
+    }                                                                          \
+  } while (0)
+
 #define performOmptCallback(CallbackName, ...)                                 \
   do {                                                                         \
     if (ompt_callback_##CallbackName##_fn)                                     \
       ompt_callback_##CallbackName##_fn(__VA_ARGS__);                          \
   } while (0)
 
 /// Function type def used for maintaining unique target region, target
 /// operations ids

 /// OMPT initialization status; false if initializeLibrary has not been executed
 extern bool Initialized;
 
 } // namespace ompt
 } // namespace target
 } // namespace omp
 } // namespace llvm
 
+#else
+#define performIfOmptInitialized(stmt)
 #endif // OMPT_SUPPORT
 
 #pragma pop_macro("DEBUG_PREFIX")
 
 #endif // _OMPTCALLBACK_H

openmp/libomptarget/src/OmptCallback.cpp

 
 // Define OMPT callback functions (bound to actual callbacks later on)
 #define defineOmptCallback(Name, Type, Code)                                   \
   Name##_t llvm::omp::target::ompt::Name##_fn = nullptr;
 FOREACH_OMPT_NOEMI_EVENT(defineOmptCallback)
 FOREACH_OMPT_EMI_EVENT(defineOmptCallback)
 #undef defineOmptCallback
 
-/// Thread local state for target region and associated metadata
-thread_local llvm::omp::target::ompt::Interface OmptInterface;
+/// Forward declaration
+class LibomptargetRtlFinalizer;
 
-/// Define function pointers
-ompt_get_task_data_t ompt_get_task_data_fn = nullptr;
+/// Object that will maintain the RTL finalizer from the plugin
+LibomptargetRtlFinalizer *LibraryFinalizer = nullptr;
+
+thread_local Interface llvm::omp::target::ompt::RegionInterface;
+
+bool llvm::omp::target::ompt::Initialized = false;
+
+ompt_get_callback_t llvm::omp::target::ompt::lookupCallbackByCode = nullptr;
+ompt_function_lookup_t llvm::omp::target::ompt::lookupCallbackByName = nullptr;
 ompt_get_target_task_data_t ompt_get_target_task_data_fn = nullptr;
+ompt_get_task_data_t ompt_get_task_data_fn = nullptr;
 
 /// Unique correlation id
 static std::atomic<uint64_t> IdCounter(1);
 
 /// Used to create a new correlation id
 static uint64_t createId() { return IdCounter.fetch_add(1); }
 
 /// Create a new correlation id and update the operations id
 static uint64_t createOpId() {
   uint64_t NewId = createId();
-  OmptInterface.setHostOpId(NewId);
+  RegionInterface.setHostOpId(NewId);
   return NewId;
 }
 
 /// Create a new correlation id and update the target region id
 static uint64_t createRegionId() {
   uint64_t NewId = createId();
-  OmptInterface.setTargetDataValue(NewId);
+  RegionInterface.setTargetDataValue(NewId);
   return NewId;
 }
 
 void Interface::beginTargetDataAlloc(int64_t DeviceId, void *HstPtrBegin,
                                      size_t Size, void *Code) {
   beginTargetDataOperation();
   if (ompt_callback_target_data_op_emi_fn) {
     // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
void Interface::endTargetSubmit(unsigned int numTeams) {
   // Only EMI callback handles end scope
   if (ompt_callback_target_submit_emi_fn) {
     // HostOpId is set by the tool. Invoke the tool supplied target submit EMI
     // callback
     ompt_callback_target_submit_emi_fn(ompt_scope_end, &TargetData, &HostOpId,
                                        numTeams);
   }
 }
+
 void Interface::beginTargetDataEnter(int64_t DeviceId, void *Code) {
   beginTargetRegion();
   if (ompt_callback_target_emi_fn) {
     // Invoke the tool supplied target EMI callback
     ompt_callback_target_emi_fn(ompt_target_enter_data, ompt_scope_begin,
                                 DeviceId, TaskData, TargetTaskData, &TargetData,
                                 Code);
   } else if (ompt_callback_target_fn) {
for (auto FinalizationFunction : RtlFinalizationFunctions)
       FinalizationFunction(/* tool_data */ nullptr);
     RtlFinalizationFunctions.clear();
   }
 
 private:
   llvm::SmallVector<ompt_finalize_t> RtlFinalizationFunctions;
 };
 
-/// Object that will maintain the RTL finalizer from the plugin
-LibomptargetRtlFinalizer *LibraryFinalizer = nullptr;
-
-bool llvm::omp::target::ompt::Initialized = false;
-
-ompt_get_callback_t llvm::omp::target::ompt::lookupCallbackByCode = nullptr;
-ompt_function_lookup_t llvm::omp::target::ompt::lookupCallbackByName = nullptr;
-
 int llvm::omp::target::ompt::initializeLibrary(ompt_function_lookup_t lookup,
                                                int initial_device_num,
                                                ompt_data_t *tool_data) {
   DP("Executing initializeLibrary (libomp)\n");
 #define bindOmptFunctionName(OmptFunction, DestinationFunction)                \
   DestinationFunction = (OmptFunction##_t)lookup(#OmptFunction);               \
   DP("initializeLibrary (libomp) bound %s=%p\n", #DestinationFunction,         \
      ((void *)(uint64_t)DestinationFunction));
 
   bindOmptFunctionName(ompt_get_callback, lookupCallbackByCode);
   bindOmptFunctionName(ompt_get_task_data, ompt_get_task_data_fn);
   bindOmptFunctionName(ompt_get_target_task_data, ompt_get_target_task_data_fn);
 #undef bindOmptFunctionName
 
   // Store pointer of 'ompt_libomp_target_fn_lookup' for use by libomptarget
   lookupCallbackByName = lookup;
 
   assert(lookupCallbackByCode && "lookupCallbackByCode should be non-null");
   assert(lookupCallbackByName && "lookupCallbackByName should be non-null");
+  assert(ompt_get_task_data_fn && "ompt_get_task_data_fn should be non-null");
+  assert(ompt_get_target_task_data_fn &&
+         "ompt_get_target_task_data_fn should be non-null");
   assert(LibraryFinalizer == nullptr &&
          "LibraryFinalizer should not be initialized yet");
 
   LibraryFinalizer = new LibomptargetRtlFinalizer();
 
   Initialized = true;
 
   return 0;
 }
 
 void llvm::omp::target::ompt::finalizeLibrary(ompt_data_t *data) {
   DP("Executing finalizeLibrary (libomp)\n");
   // Before disabling OMPT, call the (plugin) finalizations that were registered
   // with this library
   LibraryFinalizer->finalize();
   delete LibraryFinalizer;
+  Initialized = false;
 }
 
 void llvm::omp::target::ompt::connectLibrary() {
   DP("Entering connectLibrary (libomp)\n");
   // Connect with libomp
   static OmptLibraryConnectorTy LibompConnector("libomp");
   static ompt_start_tool_result_t OmptResult;
 

openmp/libomptarget/src/OmptInterface.h

 //===-------- OmptInterface.h - Target independent OpenMP target RTL ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Declarations for OpenMP Tool callback dispatchers
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _OMPTARGET_OMPTINTERFACE_H
 #define _OMPTARGET_OMPTINTERFACE_H
 
+#include <functional>
+#include <tuple>
+
+#include "OmptCallback.h"
 #include "omp-tools.h"
 
+#include "llvm/Support/ErrorHandling.h"
+
 // If target OMPT support is compiled in
 #ifdef OMPT_SUPPORT
 #define OMPT_IF_BUILT(stmt) stmt
 #else
 #define OMPT_IF_BUILT(stmt)
 #endif
 
+#define OMPT_GET_RETURN_ADDRESS(level) __builtin_return_address(level)
+
 /// Callbacks for target regions require task_data representing the
 /// encountering task.
 /// Callbacks for target regions and target data ops require
 /// target_task_data representing the target task region.
 typedef ompt_data_t *(*ompt_get_task_data_t)();
 typedef ompt_data_t *(*ompt_get_target_task_data_t)();
 
 namespace llvm {
public:
   void endTargetUpdate(int64_t DeviceId, void *Code);
 
   /// Top-level function for invoking callback before target construct
   void beginTarget(int64_t DeviceId, void *Code);
 
   /// Top-level function for invoking callback after target construct
   void endTarget(int64_t DeviceId, void *Code);
 
+  // Functionpair getter: Target data operations
+  template <ompt_target_data_op_t OpType> auto getCallbacks() {
+    if constexpr (OpType == ompt_target_data_alloc ||
+                  OpType == ompt_target_data_alloc_async)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetDataAlloc),
+                            std::mem_fn(&Interface::endTargetDataAlloc));
+
+    if constexpr (OpType == ompt_target_data_delete ||
+                  OpType == ompt_target_data_delete_async)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetDataDelete),
+                            std::mem_fn(&Interface::endTargetDataDelete));
+
+    if constexpr (OpType == ompt_target_data_transfer_to_device ||
+                  OpType == ompt_target_data_transfer_to_device_async)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetDataSubmit),
+                            std::mem_fn(&Interface::endTargetDataSubmit));
+
+    if constexpr (OpType == ompt_target_data_transfer_from_device ||
+                  OpType == ompt_target_data_transfer_from_device_async)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetDataRetrieve),
+                            std::mem_fn(&Interface::endTargetDataRetrieve));
+
+    llvm_unreachable("Unhandled target data operation type!");
+  }
+
+  // Target region operations
+  template <ompt_target_t OpType> auto getCallbacks() {
+    if constexpr (OpType == ompt_target_enter_data ||
+                  OpType == ompt_target_enter_data_nowait)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetDataEnter),
+                            std::mem_fn(&Interface::endTargetDataEnter));
+
+    if constexpr (OpType == ompt_target_exit_data ||
+                  OpType == ompt_target_exit_data_nowait)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetDataExit),
+                            std::mem_fn(&Interface::endTargetDataExit));
+
+    if constexpr (OpType == ompt_target_update ||
+                  OpType == ompt_target_update_nowait)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetUpdate),
+                            std::mem_fn(&Interface::endTargetUpdate));
+
+    if constexpr (OpType == ompt_target || OpType == ompt_target_nowait)
+      return std::make_pair(std::mem_fn(&Interface::beginTarget),
+                            std::mem_fn(&Interface::endTarget));
+
+    llvm_unreachable("Unknown target region operation type!");
+  }
+
+  // Kernel launch operation
+  template <ompt_callbacks_t OpType> auto getCallbacks() {
+    // We use 'ompt_callbacks_t', because no other enum is currently available
+    // to model a kernel launch / target submit operation.
+    if constexpr (OpType == ompt_callback_target_submit)
+      return std::make_pair(std::mem_fn(&Interface::beginTargetSubmit),
+                            std::mem_fn(&Interface::endTargetSubmit));
+
+    llvm_unreachable("Unhandled target operation!");
+  }
+
   /// Setters for target region and target operation correlation ids
   void setTargetDataValue(uint64_t DataValue) { TargetData.value = DataValue; }
   void setTargetDataPtr(void *DataPtr) { TargetData.ptr = DataPtr; }
   void setHostOpId(ompt_id_t OpId) { HostOpId = OpId; }
 
   /// Getters for target region and target operation correlation ids
   uint64_t getTargetDataValue() { return TargetData.value; }
   void *getTargetDataPtr() { return TargetData.ptr; }
private:
 
   /// Used for marking begin of a target region
   void beginTargetRegion();
 
   /// Used for marking end of a target region
   void endTargetRegion();
 };
 
+/// Thread local state for target region and associated metadata
+extern thread_local Interface RegionInterface;
+
+template <typename FuncTy, typename ArgsTy, size_t... IndexSeq>
+void InvokeInterfaceFunction(FuncTy Func, ArgsTy Args,
+                             std::index_sequence<IndexSeq...>) {
+  std::invoke(Func, RegionInterface, std::get<IndexSeq>(Args)...);
+}
+
+template <typename FunctionPairTy, typename... ArgsTy> class InterfaceRAII {
+public:
+  InterfaceRAII(FunctionPairTy FunctionPair, ArgsTy... Args)
+      : Arguments(Args...), beginFunction(std::get<0>(FunctionPair)),
+        endFunction(std::get<1>(FunctionPair)) {
+    performIfOmptInitialized(begin());
+  }
+  ~InterfaceRAII() { performIfOmptInitialized(end()); }
+
+private:
+  void begin() {
+    auto IndexSequence =
+        std::make_index_sequence<std::tuple_size_v<decltype(Arguments)>>{};
+    InvokeInterfaceFunction(beginFunction, Arguments, IndexSequence);
+  }
+
+  void end() {
+    auto IndexSequence =
+        std::make_index_sequence<std::tuple_size_v<decltype(Arguments)>>{};
+    InvokeInterfaceFunction(endFunction, Arguments, IndexSequence);
+  }
+
+  std::tuple<ArgsTy...> Arguments;
+  typename FunctionPairTy::first_type beginFunction;
+  typename FunctionPairTy::second_type endFunction;
+};
+
 } // namespace ompt
 } // namespace target
 } // namespace omp
 } // namespace llvm
 
-extern thread_local llvm::omp::target::ompt::Interface OmptInterface;
-
 #endif // _OMPTARGET_OMPTINTERFACE_H

openmp/libomptarget/src/device.cpp

 //===--------- device.cpp - Target independent OpenMP target RTL ----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Functionality for managing devices that are handled by RTL plugins.
 //
 //===----------------------------------------------------------------------===//
 
 #include "device.h"
+#include "OmptCallback.h"
+#include "OmptInterface.h"
 #include "omptarget.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <cassert>
 #include <climits>
 #include <cstdint>
 #include <cstdio>
 #include <mutex>
 #include <string>
 #include <thread>
 
+using namespace llvm::omp::target::ompt;
+
 int HostDataToTargetTy::addEventIfNecessary(DeviceTy &Device,
                                             AsyncInfoTy &AsyncInfo) const {
   // First, check if the user disabled atomic map transfer/malloc/dealloc.
   if (!PM->UseEventsForAtomicTransfers)
     return OFFLOAD_SUCCESS;
 
   void *Event = getEvent();
   bool NeedNewEvent = Event == nullptr;
   if (NeedNewEvent && Device.createEvent(&Event) != OFFLOAD_SUCCESS) {
     REPORT("Failed to create event\n");
     return OFFLOAD_FAIL;
   }
 
   // We cannot assume the event should not be nullptr because we don't
   // know if the target support event. But if a target doesn't,
   // recordEvent should always return success.
   if (Device.recordEvent(Event, AsyncInfo) != OFFLOAD_SUCCESS) {
     REPORT("Failed to set dependence on event " DPxMOD "\n", DPxPTR(Event));
     return OFFLOAD_FAIL;
   }
 
   if (NeedNewEvent)
     setEvent(Event);
 
   return OFFLOAD_SUCCESS;
 }
 

 
 // Load binary to device.
 __tgt_target_table *DeviceTy::loadBinary(void *Img) {
   std::lock_guard<decltype(RTL->Mtx)> LG(RTL->Mtx);
   return RTL->load_binary(RTLDeviceID, Img);
 }
 
 void *DeviceTy::allocData(int64_t Size, void *HstPtr, int32_t Kind) {
+  /// RAII to establish tool anchors before and after data allocation
+  InterfaceRAII TargetDataAllocRAII(
+      RegionInterface.getCallbacks<ompt_target_data_alloc>(), RTLDeviceID,
+      HstPtr, Size,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   return RTL->data_alloc(RTLDeviceID, Size, HstPtr, Kind);
 }
 
 int32_t DeviceTy::deleteData(void *TgtAllocBegin, int32_t Kind) {
+  /// RAII to establish tool anchors before and after data deletion
+  InterfaceRAII TargetDataDeleteRAII(
+      RegionInterface.getCallbacks<ompt_target_data_delete>(), RTLDeviceID,
+      TgtAllocBegin,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   return RTL->data_delete(RTLDeviceID, TgtAllocBegin, Kind);
 }
 
 static void printCopyInfo(int DeviceId, bool H2D, void *SrcPtrBegin,
                           void *DstPtrBegin, int64_t Size,
                           HostDataToTargetTy *HT) {
 
   INFO(OMP_INFOTYPE_DATA_TRANSFER, DeviceId,
if (getInfoLevel() & OMP_INFOTYPE_DATA_TRANSFER) {
     if (!Entry) {
       LR = lookupMapping(HDTTMap, HstPtrBegin, Size);
       Entry = LR.TPR.getEntry();
     }
     printCopyInfo(DeviceID, /* H2D */ true, HstPtrBegin, TgtPtrBegin, Size,
                   Entry);
   }
 
+  /// RAII to establish tool anchors before and after data submit
+  InterfaceRAII TargetDataSubmitRAII(
+      RegionInterface.getCallbacks<ompt_target_data_transfer_to_device>(),
+      RTLDeviceID, TgtPtrBegin, HstPtrBegin, Size,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   if (!AsyncInfo || !RTL->data_submit_async || !RTL->synchronize)
     return RTL->data_submit(RTLDeviceID, TgtPtrBegin, HstPtrBegin, Size);
   return RTL->data_submit_async(RTLDeviceID, TgtPtrBegin, HstPtrBegin, Size,
                                 AsyncInfo);
 }
 
 // Retrieve data from device
 int32_t DeviceTy::retrieveData(void *HstPtrBegin, void *TgtPtrBegin,
                                int64_t Size, AsyncInfoTy &AsyncInfo,
                                HostDataToTargetTy *Entry) {
   if (getInfoLevel() & OMP_INFOTYPE_DATA_TRANSFER) {
     HDTTMapAccessorTy HDTTMap = HostDataToTargetMap.getExclusiveAccessor(Entry);
     LookupResult LR;
     if (!Entry) {
       LR = lookupMapping(HDTTMap, HstPtrBegin, Size);
       Entry = LR.TPR.getEntry();
     }
     printCopyInfo(DeviceID, /* H2D */ false, TgtPtrBegin, HstPtrBegin, Size,
                   Entry);
   }
 
+  /// RAII to establish tool anchors before and after data retrieval
+  InterfaceRAII TargetDataRetrieveRAII(
+      RegionInterface.getCallbacks<ompt_target_data_transfer_from_device>(),
+      RTLDeviceID, HstPtrBegin, TgtPtrBegin, Size,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   if (!RTL->data_retrieve_async || !RTL->synchronize)
     return RTL->data_retrieve(RTLDeviceID, HstPtrBegin, TgtPtrBegin, Size);
   return RTL->data_retrieve_async(RTLDeviceID, HstPtrBegin, TgtPtrBegin, Size,
                                   AsyncInfo);
 }
 
 // Copy data from current device to destination device directly
 int32_t DeviceTy::dataExchange(void *SrcPtr, DeviceTy &DstDev, void *DstPtr,

openmp/libomptarget/src/interface.cpp

 //===-------- interface.cpp - Target independent OpenMP target RTL --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of the interface to be used by Clang during the codegen of a
 // target region.
 //
 //===----------------------------------------------------------------------===//
 
+#include "OmptCallback.h"
+#include "OmptInterface.h"
 #include "device.h"
 #include "omptarget.h"
 #include "private.h"
 #include "rtl.h"
 
 #include "Utilities.h"
 
 #include <cassert>
 #include <cstdio>
 #include <cstdlib>
 #include <mutex>
 #include <type_traits>
 
+using namespace llvm::omp::target::ompt;
+
 ////////////////////////////////////////////////////////////////////////////////
 /// adds requires flags
 EXTERN void __tgt_register_requires(int64_t Flags) {
   TIMESCOPE();
   PM->RTLs.registerRequires(Flags);
 }
 
 ////////////////////////////////////////////////////////////////////////////////
targetDataMapper(ident_t *Loc, int64_t DeviceId, int32_t ArgNum,
                  TargetDataFuncPtrTy TargetDataFunction,
                  const char *RegionTypeMsg, const char *RegionName) {
   static_assert(std::is_convertible_v<TargetAsyncInfoTy, AsyncInfoTy>,
                 "TargetAsyncInfoTy must be convertible to AsyncInfoTy.");
 
   TIMESCOPE_WITH_IDENT(Loc);
 
   DP("Entering data %s region for device %" PRId64 " with %d mappings\n",
      RegionName, DeviceId, ArgNum);
 
   if (checkDeviceAndCtors(DeviceId, Loc)) {
     DP("Not offloading to device %" PRId64 "\n", DeviceId);
     return;
   }
 
   if (getInfoLevel() & OMP_INFOTYPE_KERNEL_ARGS)
     printKernelArguments(Loc, DeviceId, ArgNum, ArgSizes, ArgTypes, ArgNames,

 /// and passes the data to the device.
 EXTERN void __tgt_target_data_begin_mapper(ident_t *Loc, int64_t DeviceId,
                                            int32_t ArgNum, void **ArgsBase,
                                            void **Args, int64_t *ArgSizes,
                                            int64_t *ArgTypes,
                                            map_var_info_t *ArgNames,
                                            void **ArgMappers) {
   TIMESCOPE_WITH_IDENT(Loc);
+  /// RAII to establish tool anchors before and after data begin
+  InterfaceRAII TargetDataEnterRAII(
+      RegionInterface.getCallbacks<ompt_target_enter_data>(), DeviceId,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   targetDataMapper<AsyncInfoTy>(Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes,
                                 ArgTypes, ArgNames, ArgMappers, targetDataBegin,
                                 "Entering OpenMP data region", "begin");
 }
 
 EXTERN void __tgt_target_data_begin_nowait_mapper(
     ident_t *Loc, int64_t DeviceId, int32_t ArgNum, void **ArgsBase,
     void **Args, int64_t *ArgSizes, int64_t *ArgTypes, map_var_info_t *ArgNames,

 /// created by the last __tgt_target_data_begin.
 EXTERN void __tgt_target_data_end_mapper(ident_t *Loc, int64_t DeviceId,
                                          int32_t ArgNum, void **ArgsBase,
                                          void **Args, int64_t *ArgSizes,
                                          int64_t *ArgTypes,
                                          map_var_info_t *ArgNames,
                                          void **ArgMappers) {
   TIMESCOPE_WITH_IDENT(Loc);
+  /// RAII to establish tool anchors before and after data end
+  InterfaceRAII TargetDataExitRAII(
+      RegionInterface.getCallbacks<ompt_target_exit_data>(), DeviceId,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   targetDataMapper<AsyncInfoTy>(Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes,
                                 ArgTypes, ArgNames, ArgMappers, targetDataEnd,
                                 "Exiting OpenMP data region", "end");
 }
 
 EXTERN void __tgt_target_data_end_nowait_mapper(
     ident_t *Loc, int64_t DeviceId, int32_t ArgNum, void **ArgsBase,
     void **Args, int64_t *ArgSizes, int64_t *ArgTypes, map_var_info_t *ArgNames,
     void **ArgMappers, int32_t DepNum, void *DepList, int32_t NoAliasDepNum,
     void *NoAliasDepList) {
   TIMESCOPE_WITH_IDENT(Loc);
   targetDataMapper<TaskAsyncInfoWrapperTy>(
       Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes, ArgTypes, ArgNames,
       ArgMappers, targetDataEnd, "Exiting OpenMP data region", "end");
 }
 
 EXTERN void __tgt_target_data_update_mapper(ident_t *Loc, int64_t DeviceId,
                                             int32_t ArgNum, void **ArgsBase,
                                             void **Args, int64_t *ArgSizes,
                                             int64_t *ArgTypes,
                                             map_var_info_t *ArgNames,
                                             void **ArgMappers) {
   TIMESCOPE_WITH_IDENT(Loc);
+  /// RAII to establish tool anchors before and after data update
+  InterfaceRAII TargetDataUpdateRAII(
+      RegionInterface.getCallbacks<ompt_target_update>(), DeviceId,
+      /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
+
   targetDataMapper<AsyncInfoTy>(
       Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes, ArgTypes, ArgNames,
       ArgMappers, targetDataUpdate, "Updating OpenMP data", "update");
 }
 
 EXTERN void __tgt_target_data_update_nowait_mapper(
     ident_t *Loc, int64_t DeviceId, int32_t ArgNum, void **ArgsBase,
     void **Args, int64_t *ArgSizes, int64_t *ArgTypes, map_var_info_t *ArgNames,
DP("Entry %2d: Base=" DPxMOD ", Begin=" DPxMOD ", Size=%" PRId64
            ? getNameFromMapping(KernelArgs->ArgNames[I]).c_str()
            : "unknown");
   }
 #endif
 
   DeviceTy &Device = *PM->Devices[DeviceId];
   TargetAsyncInfoTy TargetAsyncInfo(Device);
   AsyncInfoTy &AsyncInfo = TargetAsyncInfo;
+  InterfaceRAII TargetDataAllocRAII(RegionInterface.getCallbacks<ompt_target>(),
+                                    DeviceId,
+                                    /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
 
   int Rc = OFFLOAD_SUCCESS;
   Rc = target(Loc, Device, HostPtr, *KernelArgs, AsyncInfo);
 
   if (Rc == OFFLOAD_SUCCESS)
     Rc = AsyncInfo.synchronize();
 
   handleTargetOutcome(Rc == OFFLOAD_SUCCESS, Loc);
EXTERN int __tgt_target_kernel_replay(ident_t *Loc, int64_t DeviceId,
                                       int32_t NumTeams, int32_t ThreadLimit,
                                       uint64_t LoopTripCount) {
 
   if (checkDeviceAndCtors(DeviceId, Loc)) {
     DP("Not offloading to device %" PRId64 "\n", DeviceId);
     return OMP_TGT_FAIL;
   }
   DeviceTy &Device = *PM->Devices[DeviceId];
+  InterfaceRAII TargetDataAllocRAII(RegionInterface.getCallbacks<ompt_target>(),
+                                    DeviceId,
+                                    /* CodePtr */ OMPT_GET_RETURN_ADDRESS(0));
 
   AsyncInfoTy AsyncInfo(Device);
   int Rc = target_replay(Loc, Device, HostPtr, DeviceMemory, DeviceMemorySize,
                          TgtArgs, TgtOffsets, NumArgs, NumTeams, ThreadLimit,
                          LoopTripCount, AsyncInfo);
   if (Rc == OFFLOAD_SUCCESS)
     Rc = AsyncInfo.synchronize();
   handleTargetOutcome(Rc == OFFLOAD_SUCCESS, Loc);

openmp/libomptarget/src/omptarget.cpp

 //===------ omptarget.cpp - Target independent OpenMP target RTL -- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of the interface to be used by Clang during the codegen of a
 // target region.
 //
 //===----------------------------------------------------------------------===//
 
 #include "omptarget.h"
+#include "OmptCallback.h"
+#include "OmptInterface.h"
 #include "device.h"
 #include "private.h"
 #include "rtl.h"
 
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/bit.h"
 
 #include <cassert>
 #include <cstdint>
 #include <vector>
 
 using llvm::SmallVector;
+using namespace llvm::omp::target::ompt;
 
 int AsyncInfoTy::synchronize() {
   int Result = OFFLOAD_SUCCESS;
   if (!isQueueEmpty()) {
     switch (SyncType) {
     case SyncTy::BLOCKING:
       // If we have a queue we need to synchronize it now.
       Result = Device.synchronize(*this);
       assert(AsyncInfo.Queue == nullptr &&
              "The device plugin should have nulled the queue to indicate there "
              "are no outstanding actions!");
       break;
     case SyncTy::NON_BLOCKING:
       Result = Device.queryAsync(*this);
       break;
     }
   }
 
   // Run any pending post-processing function registered on this async object.
   if (Result == OFFLOAD_SUCCESS && isQueueEmpty())
     Result = runPostProcessing();
 
   return Result;
 }
int target(ident_t *Loc, DeviceTy &Device, void *HostPtr,
   // Launch device execution.
   void *TgtEntryPtr = TargetTable->EntriesBegin[TM->Index].addr;
   DP("Launching target execution %s with pointer " DPxMOD " (index=%d).\n",
      TargetTable->EntriesBegin[TM->Index].name, DPxPTR(TgtEntryPtr), TM->Index);
 
   {
     assert(KernelArgs.NumArgs == TgtArgs.size() && "Argument count mismatch!");
     TIMESCOPE_WITH_NAME_AND_IDENT("Initiate Kernel Launch", Loc);
+
+#ifdef OMPT_SUPPORT
+    assert(KernelArgs.NumTeams[1] == 0 && KernelArgs.NumTeams[2] == 0 &&
+           "Multi dimensional launch not supported yet.");
+    /// RAII to establish tool anchors before and after kernel launch
+    int32_t NumTeams = KernelArgs.NumTeams[0];
+    InterfaceRAII TargetSubmitRAII(
+        RegionInterface.getCallbacks<ompt_callback_target_submit>(), NumTeams);
+#endif
+
     Ret = Device.launchKernel(TgtEntryPtr, TgtArgs.data(), TgtOffsets.data(),
                               KernelArgs, AsyncInfo);
   }
 
   if (Ret != OFFLOAD_SUCCESS) {
     REPORT("Executing target region abort target.\n");
     return OFFLOAD_FAIL;
   }

openmp/libomptarget/test/ompt/veccopy.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op'
-/// CHECK: Could not register callback 'ompt_callback_target'
-/// CHECK: Could not register callback 'ompt_callback_target_submit'
-
 /// CHECK: Callback Init:
 /// CHECK: Callback Load:
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
+
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=0
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
 /// CHECK: Callback Fini:

openmp/libomptarget/test/ompt/veccopy_disallow_both.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op'
-/// CHECK: Could not register callback 'ompt_callback_target'
-/// CHECK: Could not register callback 'ompt_callback_target_submit'
-
 /// CHECK: Callback Init:
 /// CHECK: Callback Load:
+/// CHECK: Callback Target EMI: kind=1 endpoint=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback Target EMI: kind=1 endpoint=2
+/// CHECK: Callback Target EMI: kind=1 endpoint=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=0
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback Target EMI: kind=1 endpoint=2
 /// CHECK: Callback Fini:

openmp/libomptarget/test/ompt/veccopy_emi.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op_emi'
-/// CHECK: Could not register callback 'ompt_callback_target_emi'
-/// CHECK: Could not register callback 'ompt_callback_target_submit_emi'
-
 /// CHECK: Callback Init:
 /// CHECK: Callback Load:
+/// CHECK: Callback Target EMI: kind=1 endpoint=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback Submit EMI: endpoint=1 req_num_teams=1
+/// CHECK: Callback Submit EMI: endpoint=2 req_num_teams=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback Target EMI: kind=1 endpoint=2
+/// CHECK: Callback Target EMI: kind=1 endpoint=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback Submit EMI: endpoint=1 req_num_teams=0
+/// CHECK: Callback Submit EMI: endpoint=2 req_num_teams=0
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback Target EMI: kind=1 endpoint=2
 /// CHECK: Callback Fini:

openmp/libomptarget/test/ompt/veccopy_emi_map.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op_emi'
-/// CHECK: Could not register callback 'ompt_callback_target_emi'
-/// CHECK: Could not register callback 'ompt_callback_target_submit_emi'
-/// CHECK: Could not register callback 'ompt_callback_target_map_emi'
-
+/// CHECK: 0: Could not register callback 'ompt_callback_target_map_emi'
 /// CHECK: Callback Init:
 /// CHECK: Callback Load:
+/// CHECK: Callback Target EMI: kind=1 endpoint=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback Submit EMI: endpoint=1 req_num_teams=1
+/// CHECK: Callback Submit EMI: endpoint=2 req_num_teams=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback Target EMI: kind=1 endpoint=2
+/// CHECK: Callback Target EMI: kind=1 endpoint=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=1
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=2
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=2
+/// CHECK: Callback Submit EMI: endpoint=1 req_num_teams=0
+/// CHECK: Callback Submit EMI: endpoint=2 req_num_teams=0
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=3
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=1 optype=4
+/// CHECK: Callback DataOp EMI: endpoint=2 optype=4
+/// CHECK: Callback Target EMI: kind=1 endpoint=2
 /// CHECK: Callback Fini:

openmp/libomptarget/test/ompt/veccopy_map.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op'
-/// CHECK: Could not register callback 'ompt_callback_target'
-/// CHECK: Could not register callback 'ompt_callback_target_submit'
 
+/// CHECK: 0: Could not register callback 'ompt_callback_target_map'
 /// CHECK: Callback Init:
 /// CHECK: Callback Load:
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
+
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=0
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
 /// CHECK: Callback Fini:

openmp/libomptarget/test/ompt/veccopy_no_device_init.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op'
-/// CHECK: Could not register callback 'ompt_callback_target'
-/// CHECK: Could not register callback 'ompt_callback_target_submit'
-
 /// CHECK-NOT: Callback Init:
 /// CHECK-NOT: Callback Load:
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
+
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=0
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
 /// CHECK-NOT: Callback Fini:

openmp/libomptarget/test/ompt/veccopy_wrong_return.c

for (i = 0; i < N; i++)
     }
 
   if (!rc)
     printf("Success\n");
 
   return rc;
 }
 
-/// CHECK: Could not register callback 'ompt_callback_target_data_op'
-/// CHECK: Could not register callback 'ompt_callback_target'
-/// CHECK: Could not register callback 'ompt_callback_target_submit'
-
 /// CHECK-NOT: Callback Init:
 /// CHECK-NOT: Callback Load:
-/// CHECK-NOT: Callback Fini:
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
+
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=1
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=2
+/// CHECK-NOT: Callback Submit: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] req_num_teams=0
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=3
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback DataOp: target_id=[[TARGET_ID:[0-9]+]] host_op_id=[[HOST_OP_ID:[0-9]+]] optype=4
+/// CHECK-NOT: Callback Target: target_id=[[TARGET_ID:[0-9]+]] kind=1 endpoint=2
+/// CHECK-NOT: Callback Fini

openmp/runtime/src/ompt-event-specific.h

 #define ompt_callback_parallel_begin_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_parallel_end_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_task_create_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_task_schedule_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_implicit_task_implemented ompt_event_MAY_ALWAYS
 
-#define ompt_callback_target_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_emi_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_data_op_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_data_op_emi_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_submit_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_submit_emi_implemented ompt_event_UNIMPLEMENTED
-
+#define ompt_callback_target_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_emi_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_data_op_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_data_op_emi_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_submit_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_submit_emi_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_control_tool_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_device_initialize_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_device_finalize_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_device_load_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_device_unload_implemented ompt_event_UNIMPLEMENTED