[OpenMP] [OMPT] [7/8] Invoke tool-supplied callbacks before and after target launch and data transfer operations

[mhalk]

Summary

Implemented RAII objects, initialized at target entry points, that
invoke tool-supplied callbacks. Updated status of target callbacks as
implemented.

Depends on D127365

Patch from John Mellor-Crummey <[e-mail]>
(With contributions from Dhruva Chakrabarti <[e-mail]>)

[dhruvachak]

Rebased

[jplehr]

openmp/libomptarget/src/device.cpp

619

I wonder if these RAII objects are constructed when OMPT support is not built, or if it is optimized away in such cases?

649

In the asynchronous case, the lifetime of the RAII object is probably too short to account for the actual data transfer events.

675

In the asynchronous case, the lifetime of the RAII object is probably too short to account for the actual data transfer events.

[mhalk]

Rebased

[mhalk]

Refactored patch for use with nextgen plugins.

[mhalk]

As discussed with Dhruva.

[jdoerfert]

openmp/libomptarget/src/device.cpp

42

Style, also names of the arguments.

openmp/libomptarget/src/interface.cpp

85

These RAIIs should stay together. If you make them templated you could even get rid of the individual members and potentially have a single RAII to rule them all. (you'd pass the callback rather than the enum, or have a separate switch for the enum to callback translation, either way, you'd then unpack the tuple for the call, see std::apply)

openmp/libomptarget/src/omptarget.cpp

37

if you move the conditional into the con/destructor and you create a common base class for these, you should be able to eliminate all but the logic dealing with the provided values (here NumTeams).

1723

*, not +

[mhalk]

Rebase + implemented feedback so far. Thanks @jdoerfert!
Exception: RAII objects have not been consolidated, yet.

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

45 ↗

(On Diff #542612)

Some sort of init flag will be needed.
Originally thought the timeline would allow to go with a separate patch.
See D155186

openmp/libomptarget/src/device.cpp

46

Oversight: Six occurrences -- will be removed.

[jdoerfert]

openmp/libomptarget/src/OmptCallback.cpp

39 ↗

(On Diff #542612)

Why is the comment gone?

openmp/libomptarget/src/omptarget.cpp

41

Commented code?

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

39 ↗

(On Diff #542612)

Has been moved to its declaration in openmp/libomptarget/src/OmptInterface.h.

openmp/libomptarget/src/omptarget.cpp

41

Yes, my bad -- as stated in my other comment there are six occurrences, all of which will be removed upon the next change.
performIfOmptInitialized already checks the Variable before calling [begin|end]Submit().

[mhalk]

openmp/libomptarget/src/omptarget.cpp

41

FYI: I' will adapt this "part" of the patch (using an Init flag) upon the landing of D155186.

[mhalk]

Implemented templated RAII class.

Please, take a look if this is the desired direction.
If the approach should be changed, let me know.

Other than that, there are a few outstanding issues:
(Currently looking into the first two.)

• Warnings are currently emitted for each RAII instance

warning: 'InterfaceRAII' may not intend to support class template argument
         deduction [-Wctad-maybe-unsupported]
note: add a deduction guide to suppress this warning

• Test coverage should be expanded for target region operations: enter_data, exit_data, update
  • Those cases are handled by the RAII -- but simply not tested.
  • Will add another lit test in another upcoming review.
• Asynchronous timing does not work correctly as stated in a previous comment. RAII objects will be prematurely destroyed in this use case. Please, let me know if this is a requirement for accepting this patch.

[jdoerfert]

Add -Wno-ctad-maybe-unsupported to our cmake config or add sth like:

template <typename FunctionPairTy, typename... ArgsTy>
  InterfaceRAII(FunctionPairTy FunctionPair, ArgsTy... Args) -> InterfaceRAII< FunctionPairTy, ArgsTy...>;

We can fix async in a follow up.

[dhruvachak]

openmp/libomptarget/src/device.cpp

649

I think the asynchronous case is not an issue for the callbacks. The begin and end callbacks are supposed to be invoked before and after the entry-point invocation. The actual data transfer may complete later and is supposed to be captured by device tracing, something that is not part of this patch.

[mhalk]

Fixed warning related to template deduction -- thanks @jdoerfert
Fixed constant numbers being used in target data operations
NFC / improved naming of InterfaceRAII

Known issue:
Missing test coverage for target region operations: enter_data, exit_data, update
Callback my fire before the actual device init and report wrong device number

• Saw this with enter_data when performing a datamapping within a new testcase

Currently, I plan on landing this within the next five hours.
If there are concerns or further feedback, let me know.

*ping* @jdoerfert @dhruvachak

[dhruvachak]

LGTM.

[tianshilei1992]

I don't think the code in common path in this patch has been guarded correctly. It causes build error if OMPT is disabled. Please fix it or get it reverted if it takes long time.

[mhalk]

Thanks @tianshilei1992 for bringing this to my attention (again)!
Sorry for the inconvenience, I will try to fix this ASAP.

[mhalk]

Fixed build errors when OMPT support was disabled.

[jplehr]

LGTM, assuming build works

openmp/libomptarget/src/omptarget.cpp

1725

nit: Why?

[mhalk]

openmp/libomptarget/src/omptarget.cpp

1725

Because we're already checking via #ifdef OMPT_SUPPORT

openmp/libomptarget/src/device.cpp

 //===--------- device.cpp - Target independent OpenMP target RTL ----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Functionality for managing devices that are handled by RTL plugins.
 //
 //===----------------------------------------------------------------------===//
 
 #include "device.h"
+#include "ompt_callback.h"
 #include "omptarget.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <cassert>
 #include <climits>
 #include <cstdint>
 #include <cstdio>
 #include <string>
 #include <thread>
 
+/// RAII used to invoke callbacks before and after data ops
+struct OmptInterfaceTargetDataOpRAII {
+  OmptInterfaceTargetDataOpRAII(int32_t Id, int64_t Sz, void *HPtr, void *TPtr,
+                                ompt_target_data_op_t Op)
+      : DeviceId{Id}, Size{Sz}, CodePtr{nullptr}, HostPtr{HPtr}, TgtPtr{TPtr},
+        TgtDataOp{Op} {
+    OMPT_IF_BUILT(beginOp());
+  }
+  ~OmptInterfaceTargetDataOpRAII() { OMPT_IF_BUILT(endOp()); }
+
+private:
+  int32_t DeviceId;                // Target device
+  int64_t Size;                    // Size of data transfer
+  void *CodePtr;                   // Return address
+  void *HostPtr;                   // Host data ptr
+  void *TgtPtr;                    // Target data ptr
+  ompt_target_data_op_t TgtDataOp; // Data transfer type
+  void beginOp() {
+    if (!ompt_enabled)
+      return;
+    CodePtr = OMPT_GET_RETURN_ADDRESS(0);
+    switch (TgtDataOp) {
+    case ompt_target_data_alloc:
+    case ompt_target_data_alloc_async:
+      ompt_interface.beginTargetDataAlloc(DeviceId, HostPtr, Size, CodePtr);
+      break;
+    case ompt_target_data_delete:
+    case ompt_target_data_delete_async:
+      ompt_interface.beginTargetDataDelete(DeviceId, TgtPtr, CodePtr);
+      break;
+    case ompt_target_data_transfer_to_device:
+    case ompt_target_data_transfer_to_device_async:
+      ompt_interface.beginTargetDataSubmit(DeviceId, TgtPtr, HostPtr, Size,
+                                           CodePtr);
+      break;
+    case ompt_target_data_transfer_from_device:
+    case ompt_target_data_transfer_from_device_async:
+      ompt_interface.beginTargetDataRetrieve(DeviceId, HostPtr, TgtPtr, Size,
+                                             CodePtr);
+      break;
+    default:
+      break;
+    }
+  }
+  void endOp() {
+    if (!ompt_enabled)
+      return;
+    switch (TgtDataOp) {
+    case ompt_target_data_alloc:
+    case ompt_target_data_alloc_async:
+      ompt_interface.endTargetDataAlloc(DeviceId, HostPtr, Size, CodePtr);
+      break;
+    case ompt_target_data_delete:
+    case ompt_target_data_delete_async:
+      ompt_interface.endTargetDataDelete(DeviceId, TgtPtr, CodePtr);
+      break;
+    case ompt_target_data_transfer_to_device:
+    case ompt_target_data_transfer_to_device_async:
+      ompt_interface.endTargetDataSubmit(DeviceId, TgtPtr, HostPtr, Size,
+                                         CodePtr);
+      break;
+    case ompt_target_data_transfer_from_device:
+    case ompt_target_data_transfer_from_device_async:
+      ompt_interface.endTargetDataRetrieve(DeviceId, HostPtr, TgtPtr, Size,
+                                           CodePtr);
+      break;
+    default:
+      break;
+    }
+  }
+};
+
 int HostDataToTargetTy::addEventIfNecessary(DeviceTy &Device,
                                             AsyncInfoTy &AsyncInfo) const {
   // First, check if the user disabled atomic map transfer/malloc/dealloc.
   if (!PM->UseEventsForAtomicTransfers)
     return OFFLOAD_SUCCESS;
 
   void *Event = getEvent();
   bool NeedNewEvent = Event == nullptr;

 
 // Load binary to device.
 __tgt_target_table *DeviceTy::loadBinary(void *Img) {
   std::lock_guard<decltype(RTL->Mtx)> LG(RTL->Mtx);
   return RTL->load_binary(RTLDeviceID, Img);
 }
 
 void *DeviceTy::allocData(int64_t Size, void *HstPtr, int32_t Kind) {
+  /// RAII to establish tool anchors before and after data allocation
+  OmptInterfaceTargetDataOpRAII TgtDataAlloc(
+      RTLDeviceID, Size, HstPtr, nullptr /* TgtPtr */, ompt_target_data_alloc);
   return RTL->data_alloc(RTLDeviceID, Size, HstPtr, Kind);
 }
 
 int32_t DeviceTy::deleteData(void *TgtPtrBegin, int32_t Kind) {
+  /// RAII to establish tool anchors before and after data deletion
+  OmptInterfaceTargetDataOpRAII TgtDataDelete(
+      RTLDeviceID, 0 /* Size */, nullptr /* HostPtr */, TgtPtrBegin,
+      ompt_target_data_delete);
   return RTL->data_delete(RTLDeviceID, TgtPtrBegin, Kind);
 }
 
 // Submit data to device
 int32_t DeviceTy::submitData(void *TgtPtrBegin, void *HstPtrBegin, int64_t Size,
                              AsyncInfoTy &AsyncInfo) {
   if (getInfoLevel() & OMP_INFOTYPE_DATA_TRANSFER) {
     HDTTMapAccessorTy HDTTMap = HostDataToTargetMap.getExclusiveAccessor();
     LookupResult LR = lookupMapping(HDTTMap, HstPtrBegin, Size);
     auto *HT = &*LR.Entry;
 
     INFO(OMP_INFOTYPE_DATA_TRANSFER, DeviceID,
          "Copying data from host to device, HstPtr=" DPxMOD ", TgtPtr=" DPxMOD
          ", Size=%" PRId64 ", Name=%s\n",
          DPxPTR(HstPtrBegin), DPxPTR(TgtPtrBegin), Size,
          (HT && HT->HstPtrName) ? getNameFromMapping(HT->HstPtrName).c_str()
                                 : "unknown");
   }
 
+  /// RAII to establish tool anchors before and after data submit
+  OmptInterfaceTargetDataOpRAII TargetDataSubmitRAII(
+      RTLDeviceID, Size, HstPtrBegin, TgtPtrBegin,
+      ompt_target_data_transfer_to_device);
+
   if (!AsyncInfo || !RTL->data_submit_async || !RTL->synchronize)
     return RTL->data_submit(RTLDeviceID, TgtPtrBegin, HstPtrBegin, Size);
   return RTL->data_submit_async(RTLDeviceID, TgtPtrBegin, HstPtrBegin, Size,
                                 AsyncInfo);
 }
 
 // Retrieve data from device
 int32_t DeviceTy::retrieveData(void *HstPtrBegin, void *TgtPtrBegin,
                                int64_t Size, AsyncInfoTy &AsyncInfo) {
   if (getInfoLevel() & OMP_INFOTYPE_DATA_TRANSFER) {
     HDTTMapAccessorTy HDTTMap = HostDataToTargetMap.getExclusiveAccessor();
     LookupResult LR = lookupMapping(HDTTMap, HstPtrBegin, Size);
     auto *HT = &*LR.Entry;
     INFO(OMP_INFOTYPE_DATA_TRANSFER, DeviceID,
          "Copying data from device to host, TgtPtr=" DPxMOD ", HstPtr=" DPxMOD
          ", Size=%" PRId64 ", Name=%s\n",
          DPxPTR(TgtPtrBegin), DPxPTR(HstPtrBegin), Size,
          (HT && HT->HstPtrName) ? getNameFromMapping(HT->HstPtrName).c_str()
                                 : "unknown");
   }
 
+  /// RAII to establish tool anchors before and after data retrieval
+  OmptInterfaceTargetDataOpRAII TargetDataRetrieve(
+      RTLDeviceID, Size, HstPtrBegin, TgtPtrBegin,
+      ompt_target_data_transfer_from_device);
+
   if (!RTL->data_retrieve_async || !RTL->synchronize)
     return RTL->data_retrieve(RTLDeviceID, HstPtrBegin, TgtPtrBegin, Size);
   return RTL->data_retrieve_async(RTLDeviceID, HstPtrBegin, TgtPtrBegin, Size,
                                   AsyncInfo);
 }
 
 // Copy data from current device to destination device directly
 int32_t DeviceTy::dataExchange(void *SrcPtr, DeviceTy &DstDev, void *DstPtr,

openmp/libomptarget/src/interface.cpp

 //===-------- interface.cpp - Target independent OpenMP target RTL --------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of the interface to be used by Clang during the codegen of a
 // target region.
 //
 //===----------------------------------------------------------------------===//
 
 #include "device.h"
+#include "ompt_callback.h"
 #include "omptarget.h"
 #include "private.h"
 #include "rtl.h"
 
 #include "Utilities.h"
 
 #include <cassert>
 #include <cstdio>
 #include <cstdlib>
 #include <mutex>
 #include <type_traits>
 
+/// RAII used to invoke callbacks before and after target regions
+struct OmptInterfaceTargetRAII {
+  OmptInterfaceTargetRAII(int64_t Id, ompt_target_t Op)
+      : CodePtr{nullptr}, DeviceId{Id}, TgtOp{Op} {
+    OMPT_IF_BUILT(beginOp());
+  }
+  ~OmptInterfaceTargetRAII() { OMPT_IF_BUILT(endOp()); }
+
+private:
+  void *CodePtr;       // Return address
+  int64_t DeviceId;    // Target device
+  ompt_target_t TgtOp; // Target operation
+  void beginOp() {
+    if (!ompt_enabled)
+      return;
+    CodePtr = OMPT_GET_RETURN_ADDRESS(0);
+    switch (TgtOp) {
+    case ompt_target_enter_data:
+    case ompt_target_enter_data_nowait:
+      ompt_interface.beginTargetDataEnter(DeviceId, CodePtr);
+      break;
+    case ompt_target_exit_data:
+    case ompt_target_exit_data_nowait:
+      ompt_interface.beginTargetDataExit(DeviceId, CodePtr);
+      break;
+    case ompt_target_update:
+    case ompt_target_update_nowait:
+      ompt_interface.beginTargetUpdate(DeviceId, CodePtr);
+      break;
+    case ompt_target:
+    case ompt_target_nowait:
+      ompt_interface.beginTarget(DeviceId, CodePtr);
+      break;
+    }
+  }
+  void endOp() {
+    if (!ompt_enabled)
+      return;
+    switch (TgtOp) {
+    case ompt_target_enter_data:
+    case ompt_target_enter_data_nowait:
+      ompt_interface.endTargetDataEnter(DeviceId, CodePtr);
+      break;
+    case ompt_target_exit_data:
+    case ompt_target_exit_data_nowait:
+      ompt_interface.endTargetDataExit(DeviceId, CodePtr);
+      break;
+    case ompt_target_update:
+    case ompt_target_update_nowait:
+      ompt_interface.endTargetUpdate(DeviceId, CodePtr);
+      break;
+    case ompt_target:
+    case ompt_target_nowait:
+      ompt_interface.endTarget(DeviceId, CodePtr);
+      break;
+    }
+  }
+};
+
 ////////////////////////////////////////////////////////////////////////////////
 /// adds requires flags
 EXTERN void __tgt_register_requires(int64_t Flags) {
   TIMESCOPE();
   PM->RTLs.registerRequires(Flags);
 }
 
 ////////////////////////////////////////////////////////////////////////////////

 /// and passes the data to the device.
 EXTERN void __tgt_target_data_begin_mapper(ident_t *Loc, int64_t DeviceId,
                                            int32_t ArgNum, void **ArgsBase,
                                            void **Args, int64_t *ArgSizes,
                                            int64_t *ArgTypes,
                                            map_var_info_t *ArgNames,
                                            void **ArgMappers) {
   TIMESCOPE_WITH_IDENT(Loc);
+  /// RAII to establish tool anchors before and after data begin
+  OmptInterfaceTargetRAII TargetDataBeginRAII(DeviceId,
+                                              ompt_target_enter_data);
   targetDataMapper<AsyncInfoTy>(Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes,
                                 ArgTypes, ArgNames, ArgMappers, targetDataBegin,
                                 "Entering OpenMP data region", "begin");
 }
 
 EXTERN void __tgt_target_data_begin_nowait_mapper(
     ident_t *Loc, int64_t DeviceId, int32_t ArgNum, void **ArgsBase,
     void **Args, int64_t *ArgSizes, int64_t *ArgTypes, map_var_info_t *ArgNames,

 /// created by the last __tgt_target_data_begin.
 EXTERN void __tgt_target_data_end_mapper(ident_t *Loc, int64_t DeviceId,
                                          int32_t ArgNum, void **ArgsBase,
                                          void **Args, int64_t *ArgSizes,
                                          int64_t *ArgTypes,
                                          map_var_info_t *ArgNames,
                                          void **ArgMappers) {
   TIMESCOPE_WITH_IDENT(Loc);
+  /// RAII to establish tool anchors before and after data end
+  OmptInterfaceTargetRAII TargetDataEndRAII(DeviceId, ompt_target_exit_data);
   targetDataMapper<AsyncInfoTy>(Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes,
                                 ArgTypes, ArgNames, ArgMappers, targetDataEnd,
                                 "Exiting OpenMP data region", "end");
 }
 
 EXTERN void __tgt_target_data_end_nowait_mapper(
     ident_t *Loc, int64_t DeviceId, int32_t ArgNum, void **ArgsBase,
     void **Args, int64_t *ArgSizes, int64_t *ArgTypes, map_var_info_t *ArgNames,
     void **ArgMappers, int32_t DepNum, void *DepList, int32_t NoAliasDepNum,
     void *NoAliasDepList) {
   TIMESCOPE_WITH_IDENT(Loc);
   targetDataMapper<TaskAsyncInfoWrapperTy>(
       Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes, ArgTypes, ArgNames,
       ArgMappers, targetDataEnd, "Exiting OpenMP data region", "end");
 }
 
 EXTERN void __tgt_target_data_update_mapper(ident_t *Loc, int64_t DeviceId,
                                             int32_t ArgNum, void **ArgsBase,
                                             void **Args, int64_t *ArgSizes,
                                             int64_t *ArgTypes,
                                             map_var_info_t *ArgNames,
                                             void **ArgMappers) {
   TIMESCOPE_WITH_IDENT(Loc);
+  /// RAII to establish tool anchors before and after data update
+  OmptInterfaceTargetRAII TargetDataUpdateRAII(DeviceId, ompt_target_update);
   targetDataMapper<AsyncInfoTy>(
       Loc, DeviceId, ArgNum, ArgsBase, Args, ArgSizes, ArgTypes, ArgNames,
       ArgMappers, targetDataUpdate, "Updating OpenMP data", "update");
 }
 
 EXTERN void __tgt_target_data_update_nowait_mapper(
     ident_t *Loc, int64_t DeviceId, int32_t ArgNum, void **ArgsBase,
     void **Args, int64_t *ArgSizes, int64_t *ArgTypes, map_var_info_t *ArgNames,
DP("Entry %2d: Base=" DPxMOD ", Begin=" DPxMOD ", Size=%" PRId64
        (KernelArgs->ArgNames)
            ? getNameFromMapping(KernelArgs->ArgNames[I]).c_str()
            : "unknown");
   }
 #endif
 
   DeviceTy &Device = *PM->Devices[DeviceId];
   TargetAsyncInfoTy TargetAsyncInfo(Device);
+  /// RAII to establish tool anchors before and after target region
+  OmptInterfaceTargetRAII TargetRAII(DeviceId, ompt_target);
   AsyncInfoTy &AsyncInfo = TargetAsyncInfo;
 
   int Rc = OFFLOAD_SUCCESS;
   Rc = target(Loc, Device, HostPtr, *KernelArgs, AsyncInfo);
 
   if (Rc == OFFLOAD_SUCCESS)
     Rc = AsyncInfo.synchronize();
 
EXTERN int __tgt_target_kernel_replay(ident_t *Loc, int64_t DeviceId,
                                       int32_t NumTeams, int32_t ThreadLimit,
                                       uint64_t LoopTripCount) {
 
   if (checkDeviceAndCtors(DeviceId, Loc)) {
     DP("Not offloading to device %" PRId64 "\n", DeviceId);
     return OMP_TGT_FAIL;
   }
   DeviceTy &Device = *PM->Devices[DeviceId];
-
+  /// RAII to establish tool anchors before and after target region
+  OmptInterfaceTargetRAII TargetRAII(DeviceId, ompt_target);
   AsyncInfoTy AsyncInfo(Device);
   int Rc = target_replay(Loc, Device, HostPtr, DeviceMemory, DeviceMemorySize,
                          TgtArgs, TgtOffsets, NumArgs, NumTeams, ThreadLimit,
                          LoopTripCount, AsyncInfo);
   if (Rc == OFFLOAD_SUCCESS)
     Rc = AsyncInfo.synchronize();
   handleTargetOutcome(Rc == OFFLOAD_SUCCESS, Loc);
   assert(Rc == OFFLOAD_SUCCESS &&

openmp/libomptarget/src/ompt_callback.h

 //===------- ompt_callback.h - Target independent OpenMP target RTL ------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Declarations for OpenMP Tool callback dispatchers
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _OMPTARGET_OMPT_CALLBACK_H
 #define _OMPTARGET_OMPT_CALLBACK_H
 
 // If target OMPT support is compiled in
 #ifdef OMPT_SUPPORT
 #define OMPT_IF_BUILT(stmt) stmt
 #else
 #define OMPT_IF_BUILT(stmt)
 #endif
 
+#define OMPT_GET_RETURN_ADDRESS(level) __builtin_return_address(level)
+
 #include "omp-tools.h"
 
 /// Used to maintain execution state for this thread
 struct OmptInterface {
 public:
   /// Top-level function for invoking callback before device data allocation
   void beginTargetDataAlloc(int64_t DeviceId, void *TgtPtrBegin, size_t Size,
                             void *Code);

openmp/libomptarget/src/omptarget.cpp

 //===------ omptarget.cpp - Target independent OpenMP target RTL -- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Implementation of the interface to be used by Clang during the codegen of a
 // target region.
 //
 //===----------------------------------------------------------------------===//
 
 #include "omptarget.h"
 #include "device.h"
+#include "ompt_callback.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <cassert>
 #include <cstdint>
 #include <vector>
 
 using llvm::SmallVector;
 
+/// RAII used to invoke callbacks before and after kernel launch
+struct OmptInterfaceTargetSubmitRAII {
+  OmptInterfaceTargetSubmitRAII(int32_t Teams) : NumTeams{Teams} {
+    OMPT_IF_BUILT(beginSubmit());
+  }
+  ~OmptInterfaceTargetSubmitRAII() { OMPT_IF_BUILT(endSubmit()); }
+
+private:
+  int32_t NumTeams; // Number of teams
+  void beginSubmit() {
+    if (!ompt_enabled)
+      return;
+    ompt_interface.beginTargetSubmit(NumTeams);
+  }
+  void endSubmit() {
+    if (!ompt_enabled)
+      return;
+    ompt_interface.endTargetSubmit(NumTeams);
+  }
+};
+
 int AsyncInfoTy::synchronize() {
   int Result = OFFLOAD_SUCCESS;
   if (!isQueueEmpty()) {
     switch (SyncType) {
     case SyncTy::BLOCKING:
       // If we have a queue we need to synchronize it now.
       Result = Device.synchronize(*this);
       assert(AsyncInfo.Queue == nullptr &&
int target(ident_t *Loc, DeviceTy &Device, void *HostPtr,
   // Launch device execution.
   void *TgtEntryPtr = TargetTable->EntriesBegin[TM->Index].addr;
   DP("Launching target execution %s with pointer " DPxMOD " (index=%d).\n",
      TargetTable->EntriesBegin[TM->Index].name, DPxPTR(TgtEntryPtr), TM->Index);
 
   {
     assert(KernelArgs.NumArgs == TgtArgs.size() && "Argument count mismatch!");
     TIMESCOPE_WITH_NAME_AND_IDENT("Initiate Kernel Launch", Loc);
+
+#ifdef OMPT_SUPPORT
+    /// RAII to establish tool anchors before and after kernel launch
+    int32_t NumTeams = KernelArgs.NumTeams[0] + KernelArgs.NumTeams[1] +
+                       KernelArgs.NumTeams[2];
+    OmptInterfaceTargetSubmitRAII TargetSubmitRAII(NumTeams);
+#endif
+
     Ret = Device.launchKernel(TgtEntryPtr, TgtArgs.data(), TgtOffsets.data(),
                               KernelArgs, AsyncInfo);
   }
 
   if (Ret != OFFLOAD_SUCCESS) {
     REPORT("Executing target region abort target.\n");
     return OFFLOAD_FAIL;
   }

openmp/runtime/src/ompt-event-specific.h

 /******************************************************************************
  * File: ompt-event-specific.h
  *
  * Description:
  *
  *   specify which of the OMPT events are implemented by this runtime system
  *   and the level of their implementation by a runtime system.
  *****************************************************************************/

 #define ompt_callback_parallel_begin_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_parallel_end_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_task_create_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_task_schedule_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_implicit_task_implemented ompt_event_MAY_ALWAYS
 
-#define ompt_callback_target_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_emi_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_data_op_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_data_op_emi_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_submit_implemented ompt_event_UNIMPLEMENTED
-#define ompt_callback_target_submit_emi_implemented ompt_event_UNIMPLEMENTED
-
+#define ompt_callback_target_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_emi_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_data_op_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_data_op_emi_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_submit_implemented ompt_event_MAY_ALWAYS
+#define ompt_callback_target_submit_emi_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_control_tool_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_device_initialize_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_device_finalize_implemented ompt_event_MAY_ALWAYS
 
 #define ompt_callback_device_load_implemented ompt_event_MAY_ALWAYS
 #define ompt_callback_device_unload_implemented ompt_event_UNIMPLEMENTED