[OpenMP] [OMPT] [6/8] Added callback support for target data operations, target submit, and target regions.

[mhalk]

Summary

This patch adds support for invoking target callbacks but does not yet
invoke them. A new structure OmptInterface has been added that tracks
thread local states including correlation ids. This structure defines
methods that will be called from the device independent target library
with information related to a target entry point for which a callback
is invoked. (The callback signature is defined by the OMPT spec and can be
found in omp-tools.h.) These methods in turn use the callback functions maintained
by OmptDeviceCallbacksTy to invoke the tool supplied callbacks.

Depends on D124652

Patch from John Mellor-Crummey <[e-mail]>
(With contributions from Dhruva Chakrabarti <[e-mail]>)

[dhruvachak]

Rebased

[mhalk]

Rebased + spelling fixes

[mhalk]

Refactored patch for use with nextgen plugins.

[mhalk]

As discussed with Dhruva.

[jplehr]

Some initial comments

openmp/libomptarget/include/ompt_device_callbacks.h

101 ↗

(On Diff #506968)

The methods here should probably read in an active voice, e.g., invokeOmptTargetDataOpEmiCallback and alike.

219 ↗

(On Diff #506968)

No return needed.

openmp/libomptarget/src/OmptCallback.cpp

48

These functions should probably be updated to follow the LLVM naming convention, e.g., createId. Idk if this should be done right now, or in an NFC after the OMPT support has landed.

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

48

@NFC I was asking myself the same question.
But of course I could incorporate such NFCs in this set / stack of patches.

[mhalk]

openmp/libomptarget/include/ompt_device_callbacks.h

219 ↗

(On Diff #506968)

Done.

[mhalk]

Update w.r.t. the landing of D124652; implement current state of feedback.

[jplehr]

Added some nits.
Any more comments @jdoerfert @dreachem @dhruvachak before accepting this?

openmp/libomptarget/src/OmptCallback.cpp

66

This naming seems inconsistent, doesn't it?
Why is it called TargetDataValue in the OmptInterface but the function to access it getRegionId.

openmp/libomptarget/src/OmptInterface.h

44

This should probably be a class according to the coding standards (sorry if I'm only pointing to this now).
https://llvm.org/docs/CodingStandards.html#use-of-class-and-struct-keywords

113

Parameter names here and below spelling starting w/ uppercase?

[mhalk]

Thanks for pointing out these nits, I've prepped changes for the latter two (class & uppercase spelling) and will wait for further feedback.

openmp/libomptarget/src/OmptCallback.cpp

66

Yes, that is absolutely right.
Thanks for bringing this up.

Background:
According to the spec the underlying data type ompt_data_t is supposed to represent: "[...] data that is reserved for tool use and that is related to a thread or to a parallel or task region."

My observation so far is, that this is only used for (task / target) "regions".
From the use-case POV, maybe get[Target]RegionDataValue (get[Target]RegionDataPtr)?

But, "correctly", you'd call it getToolDataValue (getToolDataPtr) I guess -- while rather generic, that'd be my pick.

If anyone has a preference / other suggestion, I'm happy to oblige.

[jplehr]

openmp/libomptarget/src/OmptCallback.cpp

66

Ok, I re-read the spec. In the light of the spec the name regionId makes more sense.
Maybe a dumb question, but do we need this function, or is it better to return the ompt_data_t itself, so it is left to the caller what they want to do with it?
I mean, do we need / want / use this?

[mhalk]

openmp/libomptarget/src/OmptCallback.cpp

66

Now, at a closer look, after all the refactoring and moving we can simply remove getRegionId.
AFAIK and as discussed offline, getRegionId would have no other utility because of its visbility.

Its uses will be replaced with direct access to the corresponding TargetData.value within "non-EMI" callback functions.
The EMI callbacks already use the ompt_data_t TargetData field directly.

[mhalk]

Rebased + removed getRegionId as it could be replaced.

openmp/libomptarget/include/OmptCallback.h

#define FOREACH_OMPT_TARGET_CALLBACK(macro)                                    \
   FOREACH_OMPT_EMI_EVENT(macro)
 
 #define performOmptCallback(CallbackName, ...)                                 \
   do {                                                                         \
     if (ompt_callback_##CallbackName##_fn)                                     \
       ompt_callback_##CallbackName##_fn(__VA_ARGS__);                          \
   } while (0)
 
+/// Function type def used for maintaining unique target region, target
+/// operations ids
+typedef uint64_t (*IdInterfaceTy)();
+
 namespace llvm {
 namespace omp {
 namespace target {
 namespace ompt {
 
 #define declareOmptCallback(Name, Type, Code) extern Name##_t Name##_fn;
 FOREACH_OMPT_NOEMI_EVENT(declareOmptCallback)
 FOREACH_OMPT_EMI_EVENT(declareOmptCallback)

openmp/libomptarget/src/OmptCallback.cpp

 
 #include <cstdlib>
 #include <cstring>
 #include <memory>
 
 #include "Debug.h"
 #include "OmptCallback.h"
 #include "OmptConnector.h"
+#include "OmptInterface.h"
 
 #undef DEBUG_PREFIX
 #define DEBUG_PREFIX "OMPT"
 
 using namespace llvm::omp::target::ompt;
 
 // Define OMPT callback functions (bound to actual callbacks later on)
 #define defineOmptCallback(Name, Type, Code)                                   \
   Name##_t llvm::omp::target::ompt::Name##_fn = nullptr;
 FOREACH_OMPT_NOEMI_EVENT(defineOmptCallback)
 FOREACH_OMPT_EMI_EVENT(defineOmptCallback)
 #undef defineOmptCallback
 
+/// Thread local state for target region and associated metadata
+thread_local llvm::omp::target::ompt::Interface OmptInterface;
+
+/// Define function pointers
+ompt_get_task_data_t ompt_get_task_data_fn = nullptr;
+ompt_get_target_task_data_t ompt_get_target_task_data_fn = nullptr;
+
+/// Unique correlation id
+static std::atomic<uint64_t> IdCounter(1);
+
+/// Used to create a new correlation id
+static uint64_t createId() { return IdCounter.fetch_add(1); }
+
+/// Create a new correlation id and update the operations id
+static uint64_t createOpId() {
+  uint64_t NewId = createId();
+  OmptInterface.setHostOpId(NewId);
+  return NewId;
+}
+
+/// Create a new correlation id and update the target region id
+static uint64_t createRegionId() {
+  uint64_t NewId = createId();
+  OmptInterface.setTargetDataValue(NewId);
+  return NewId;
+}
+
+void Interface::beginTargetDataAlloc(int64_t DeviceId, void *HstPtrBegin,
+                                     size_t Size, void *Code) {
+  beginTargetDataOperation();
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(ompt_scope_begin, TargetTaskData,
+                                        &TargetData, &TargetRegionOpId,
+                                        ompt_target_data_alloc, HstPtrBegin,
+                                        DeviceId, /* TgtPtrBegin */ nullptr,
+                                        /* TgtDeviceNum */ 0, Size, Code);
+  } else if (ompt_callback_target_data_op_fn) {
+    // HostOpId is set by the runtime
+    HostOpId = createOpId();
+    // Invoke the tool supplied data op callback
+    ompt_callback_target_data_op_fn(
+        TargetData.value, HostOpId, ompt_target_data_alloc, HstPtrBegin,
+        DeviceId, /* TgtPtrBegin */ nullptr, /* TgtDeviceNum */ 0, Size, Code);
+  }
+}
+
+void Interface::endTargetDataAlloc(int64_t DeviceId, void *HstPtrBegin,
+                                   size_t Size, void *Code) {
+  // Only EMI callback handles end scope
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(ompt_scope_end, TargetTaskData,
+                                        &TargetData, &TargetRegionOpId,
+                                        ompt_target_data_alloc, HstPtrBegin,
+                                        DeviceId, /* TgtPtrBegin */ nullptr,
+                                        /* TgtDeviceNum */ 0, Size, Code);
+  }
+  endTargetDataOperation();
+}
+
+void Interface::beginTargetDataSubmit(int64_t DeviceId, void *TgtPtrBegin,
+                                      void *HstPtrBegin, size_t Size,
+                                      void *Code) {
+  beginTargetDataOperation();
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(
+        ompt_scope_begin, TargetTaskData, &TargetData, &TargetRegionOpId,
+        ompt_target_data_transfer_to_device, HstPtrBegin,
+        /* SrcDeviceNum */ 0, TgtPtrBegin, DeviceId, Size, Code);
+  } else if (ompt_callback_target_data_op_fn) {
+    // HostOpId is set by the runtime
+    HostOpId = createOpId();
+    // Invoke the tool supplied data op callback
+    ompt_callback_target_data_op_fn(
+        TargetData.value, HostOpId, ompt_target_data_transfer_to_device,
+        HstPtrBegin, /* SrcDeviceNum */ 0, TgtPtrBegin, DeviceId, Size, Code);
+  }
+}
+
+void Interface::endTargetDataSubmit(int64_t DeviceId, void *TgtPtrBegin,
+                                    void *HstPtrBegin, size_t Size,
+                                    void *Code) {
+  // Only EMI callback handles end scope
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(
+        ompt_scope_end, TargetTaskData, &TargetData, &TargetRegionOpId,
+        ompt_target_data_transfer_to_device, HstPtrBegin,
+        /* SrcDeviceNum */ 0, TgtPtrBegin, DeviceId, Size, Code);
+  }
+  endTargetDataOperation();
+}
+
+void Interface::beginTargetDataDelete(int64_t DeviceId, void *TgtPtrBegin,
+                                      void *Code) {
+  beginTargetDataOperation();
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(
+        ompt_scope_begin, TargetTaskData, &TargetData, &TargetRegionOpId,
+        ompt_target_data_delete, TgtPtrBegin, DeviceId,
+        /* TgtPtrBegin */ nullptr, /* TgtDeviceNum */ 0, /* Bytes */ 0, Code);
+  } else if (ompt_callback_target_data_op_fn) {
+    // HostOpId is set by the runtime
+    HostOpId = createOpId();
+    // Invoke the tool supplied data op callback
+    ompt_callback_target_data_op_fn(TargetData.value, HostOpId,
+                                    ompt_target_data_delete, TgtPtrBegin,
+                                    DeviceId, /* TgtPtrBegin */ nullptr,
+                                    /* TgtDeviceNum */ 0, /* Bytes */ 0, Code);
+  }
+}
+
+void Interface::endTargetDataDelete(int64_t DeviceId, void *TgtPtrBegin,
+                                    void *Code) {
+  // Only EMI callback handles end scope
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(
+        ompt_scope_end, TargetTaskData, &TargetData, &TargetRegionOpId,
+        ompt_target_data_delete, TgtPtrBegin, DeviceId,
+        /* TgtPtrBegin */ nullptr, /* TgtDeviceNum */ 0, /* Bytes */ 0, Code);
+  }
+  endTargetDataOperation();
+}
+
+void Interface::beginTargetDataRetrieve(int64_t DeviceId, void *HstPtrBegin,
+                                        void *TgtPtrBegin, size_t Size,
+                                        void *Code) {
+  beginTargetDataOperation();
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(ompt_scope_begin, TargetTaskData,
+                                        &TargetData, &TargetRegionOpId,
+                                        ompt_target_data_transfer_from_device,
+                                        TgtPtrBegin, DeviceId, HstPtrBegin,
+                                        /* TgtDeviceNum */ 0, Size, Code);
+  } else if (ompt_callback_target_data_op_fn) {
+    // HostOpId is set by the runtime
+    HostOpId = createOpId();
+    // Invoke the tool supplied data op callback
+    ompt_callback_target_data_op_fn(TargetData.value, HostOpId,
+                                    ompt_target_data_transfer_from_device,
+                                    TgtPtrBegin, DeviceId, HstPtrBegin,
+                                    /* TgtDeviceNum */ 0, Size, Code);
+  }
+}
+
+void Interface::endTargetDataRetrieve(int64_t DeviceId, void *HstPtrBegin,
+                                      void *TgtPtrBegin, size_t Size,
+                                      void *Code) {
+  // Only EMI callback handles end scope
+  if (ompt_callback_target_data_op_emi_fn) {
+    // HostOpId will be set by the tool. Invoke the tool supplied data op EMI
+    // callback
+    ompt_callback_target_data_op_emi_fn(ompt_scope_end, TargetTaskData,
+                                        &TargetData, &TargetRegionOpId,
+                                        ompt_target_data_transfer_from_device,
+                                        TgtPtrBegin, DeviceId, HstPtrBegin,
+                                        /* TgtDeviceNum */ 0, Size, Code);
+  }
+  endTargetDataOperation();
+}
+
+void Interface::beginTargetSubmit(unsigned int numTeams) {
+  if (ompt_callback_target_submit_emi_fn) {
+    // HostOpId is set by the tool. Invoke the tool supplied target submit EMI
+    // callback
+    ompt_callback_target_submit_emi_fn(ompt_scope_begin, &TargetData, &HostOpId,
+                                       numTeams);
+  } else if (ompt_callback_target_submit_fn) {
+    // HostOpId is set by the runtime
+    HostOpId = createOpId();
+    ompt_callback_target_submit_fn(TargetData.value, HostOpId, numTeams);
+  }
+}
+
+void Interface::endTargetSubmit(unsigned int numTeams) {
+  // Only EMI callback handles end scope
+  if (ompt_callback_target_submit_emi_fn) {
+    // HostOpId is set by the tool. Invoke the tool supplied target submit EMI
+    // callback
+    ompt_callback_target_submit_emi_fn(ompt_scope_end, &TargetData, &HostOpId,
+                                       numTeams);
+  }
+}
+void Interface::beginTargetDataEnter(int64_t DeviceId, void *Code) {
+  beginTargetRegion();
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target_enter_data, ompt_scope_begin,
+                                DeviceId, TaskData, TargetTaskData, &TargetData,
+                                Code);
+  } else if (ompt_callback_target_fn) {
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target_enter_data, ompt_scope_begin, DeviceId,
+                            TaskData, TargetData.value, Code);
+  }
+}
+
+void Interface::endTargetDataEnter(int64_t DeviceId, void *Code) {
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target_enter_data, ompt_scope_end,
+                                DeviceId, TaskData, TargetTaskData, &TargetData,
+                                Code);
+  } else if (ompt_callback_target_fn) {
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target_enter_data, ompt_scope_end, DeviceId,
+                            TaskData, TargetData.value, Code);
+  }
+  endTargetRegion();
+}
+
+void Interface::beginTargetDataExit(int64_t DeviceId, void *Code) {
+  beginTargetRegion();
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target_exit_data, ompt_scope_begin,
+                                DeviceId, TaskData, TargetTaskData, &TargetData,
+                                Code);
+  } else if (ompt_callback_target_fn) {
+    TargetData.value = createRegionId();
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target_exit_data, ompt_scope_begin, DeviceId,
+                            TaskData, TargetData.value, Code);
+  }
+}
+
+void Interface::endTargetDataExit(int64_t DeviceId, void *Code) {
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target_exit_data, ompt_scope_end, DeviceId,
+                                TaskData, TargetTaskData, &TargetData, Code);
+  } else if (ompt_callback_target_fn) {
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target_exit_data, ompt_scope_end, DeviceId,
+                            TaskData, TargetData.value, Code);
+  }
+  endTargetRegion();
+}
+
+void Interface::beginTargetUpdate(int64_t DeviceId, void *Code) {
+  beginTargetRegion();
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target_update, ompt_scope_begin, DeviceId,
+                                TaskData, TargetTaskData, &TargetData, Code);
+  } else if (ompt_callback_target_fn) {
+    TargetData.value = createRegionId();
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target_update, ompt_scope_begin, DeviceId,
+                            TaskData, TargetData.value, Code);
+  }
+}
+
+void Interface::endTargetUpdate(int64_t DeviceId, void *Code) {
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target_update, ompt_scope_end, DeviceId,
+                                TaskData, TargetTaskData, &TargetData, Code);
+  } else if (ompt_callback_target_fn) {
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target_update, ompt_scope_end, DeviceId,
+                            TaskData, TargetData.value, Code);
+  }
+  endTargetRegion();
+}
+
+void Interface::beginTarget(int64_t DeviceId, void *Code) {
+  beginTargetRegion();
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target, ompt_scope_begin, DeviceId,
+                                TaskData, TargetTaskData, &TargetData, Code);
+  } else if (ompt_callback_target_fn) {
+    TargetData.value = createRegionId();
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target, ompt_scope_begin, DeviceId, TaskData,
+                            TargetData.value, Code);
+  }
+}
+
+void Interface::endTarget(int64_t DeviceId, void *Code) {
+  if (ompt_callback_target_emi_fn) {
+    // Invoke the tool supplied target EMI callback
+    ompt_callback_target_emi_fn(ompt_target, ompt_scope_end, DeviceId, TaskData,
+                                TargetTaskData, &TargetData, Code);
+  } else if (ompt_callback_target_fn) {
+    // Invoke the tool supplied target callback
+    ompt_callback_target_fn(ompt_target, ompt_scope_end, DeviceId, TaskData,
+                            TargetData.value, Code);
+  }
+  endTargetRegion();
+}
+
+void Interface::beginTargetDataOperation() {
+  DP("in ompt_target_region_begin (TargetRegionOpId = %lu)\n",
+     TargetData.value);
+}
+
+void Interface::endTargetDataOperation() {
+  DP("in ompt_target_region_end (TargetRegionOpId = %lu)\n", TargetData.value);
+}
+
+void Interface::beginTargetRegion() {
+  // Set up task state
+  assert(ompt_get_task_data_fn && "Calling a null task data function");
+  TaskData = ompt_get_task_data_fn();
+  // Set up target task state
+  assert(ompt_get_target_task_data_fn &&
+         "Calling a null target task data function");
+  TargetTaskData = ompt_get_target_task_data_fn();
+  // Target state will be set later
+  TargetData = ompt_data_none;
+}
+
+void Interface::endTargetRegion() {
+  TaskData = 0;
+  TargetTaskData = 0;
+  TargetData = ompt_data_none;
+}
+
 /// Used to maintain the finalization functions that are received
 /// from the plugins during connect.
 /// Note: Currently, there are no plugin-specific finalizations, so each plugin
 /// will call the same (empty) function.
 class LibomptargetRtlFinalizer {
 public:
   LibomptargetRtlFinalizer() {}
 
int llvm::omp::target::ompt::initializeLibrary(ompt_function_lookup_t lookup,
                                                ompt_data_t *tool_data) {
   DP("Executing initializeLibrary (libomp)\n");
 #define bindOmptFunctionName(OmptFunction, DestinationFunction)                \
   DestinationFunction = (OmptFunction##_t)lookup(#OmptFunction);               \
   DP("initializeLibrary (libomp) bound %s=%p\n", #DestinationFunction,         \
      ((void *)(uint64_t)DestinationFunction));
 
   bindOmptFunctionName(ompt_get_callback, lookupCallbackByCode);
+  bindOmptFunctionName(ompt_get_task_data, ompt_get_task_data_fn);
+  bindOmptFunctionName(ompt_get_target_task_data, ompt_get_target_task_data_fn);
 #undef bindOmptFunctionName
 
   // Store pointer of 'ompt_libomp_target_fn_lookup' for use by libomptarget
   lookupCallbackByName = lookup;
 
   assert(lookupCallbackByCode && "lookupCallbackByCode should be non-null");
   assert(lookupCallbackByName && "lookupCallbackByName should be non-null");
   assert(LibraryFinalizer == nullptr &&

openmp/libomptarget/src/OmptInterface.h

+//===-------- OmptInterface.h - Target independent OpenMP target RTL ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Declarations for OpenMP Tool callback dispatchers
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _OMPTARGET_OMPTINTERFACE_H
+#define _OMPTARGET_OMPTINTERFACE_H
+
+#include "omp-tools.h"
+
+// If target OMPT support is compiled in
+#ifdef OMPT_SUPPORT
+#define OMPT_IF_BUILT(stmt) stmt
+#else
+#define OMPT_IF_BUILT(stmt)
+#endif
+
+/// Callbacks for target regions require task_data representing the
+/// encountering task.
+/// Callbacks for target regions and target data ops require
+/// target_task_data representing the target task region.
+typedef ompt_data_t *(*ompt_get_task_data_t)();
+typedef ompt_data_t *(*ompt_get_target_task_data_t)();
+
+namespace llvm {
+namespace omp {
+namespace target {
+namespace ompt {
+
+/// Function pointers that will be used to track task_data and
+/// target_task_data.
+static ompt_get_task_data_t ompt_get_task_data_fn;
+static ompt_get_target_task_data_t ompt_get_target_task_data_fn;
+
+/// Used to maintain execution state for this thread
+class Interface {
+public:
+  /// Top-level function for invoking callback before device data allocation
+  void beginTargetDataAlloc(int64_t DeviceId, void *TgtPtrBegin, size_t Size,
+                            void *Code);
+
+  /// Top-level function for invoking callback after device data allocation
+  void endTargetDataAlloc(int64_t DeviceId, void *TgtPtrBegin, size_t Size,
+                          void *Code);
+
+  /// Top-level function for invoking callback before data submit
+  void beginTargetDataSubmit(int64_t DeviceId, void *HstPtrBegin,
+                             void *TgtPtrBegin, size_t Size, void *Code);
+
+  /// Top-level function for invoking callback after data submit
+  void endTargetDataSubmit(int64_t DeviceId, void *HstPtrBegin,
+                           void *TgtPtrBegin, size_t Size, void *Code);
+
+  /// Top-level function for invoking callback before device data deallocation
+  void beginTargetDataDelete(int64_t DeviceId, void *TgtPtrBegin, void *Code);
+
+  /// Top-level function for invoking callback after device data deallocation
+  void endTargetDataDelete(int64_t DeviceId, void *TgtPtrBegin, void *Code);
+
+  /// Top-level function for invoking callback before data retrieve
+  void beginTargetDataRetrieve(int64_t DeviceId, void *HstPtrBegin,
+                               void *TgtPtrBegin, size_t Size, void *Code);
+
+  /// Top-level function for invoking callback after data retrieve
+  void endTargetDataRetrieve(int64_t DeviceId, void *HstPtrBegin,
+                             void *TgtPtrBegin, size_t Size, void *Code);
+
+  /// Top-level function for invoking callback before kernel dispatch
+  void beginTargetSubmit(unsigned int NumTeams = 1);
+
+  /// Top-level function for invoking callback after kernel dispatch
+  void endTargetSubmit(unsigned int NumTeams = 1);
+
+  // Target region callbacks
+
+  /// Top-level function for invoking callback before target enter data
+  /// construct
+  void beginTargetDataEnter(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback after target enter data
+  /// construct
+  void endTargetDataEnter(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback before target exit data
+  /// construct
+  void beginTargetDataExit(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback after target exit data
+  /// construct
+  void endTargetDataExit(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback before target update construct
+  void beginTargetUpdate(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback after target update construct
+  void endTargetUpdate(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback before target construct
+  void beginTarget(int64_t DeviceId, void *Code);
+
+  /// Top-level function for invoking callback after target construct
+  void endTarget(int64_t DeviceId, void *Code);
+
+  /// Setters for target region and target operation correlation ids
+  void setTargetDataValue(uint64_t DataValue) { TargetData.value = DataValue; }
+  void setTargetDataPtr(void *DataPtr) { TargetData.ptr = DataPtr; }
+  void setHostOpId(ompt_id_t OpId) { HostOpId = OpId; }
+
+  /// Getters for target region and target operation correlation ids
+  uint64_t getTargetDataValue() { return TargetData.value; }
+  void *getTargetDataPtr() { return TargetData.ptr; }
+  ompt_id_t getHostOpId() { return HostOpId; }
+
+private:
+  /// Target operations id
+  ompt_id_t HostOpId = 0;
+
+  /// Target region data
+  ompt_data_t TargetData = ompt_data_none;
+
+  /// Task data representing the encountering task
+  ompt_data_t *TaskData = nullptr;
+
+  /// Target task data representing the target task region
+  ompt_data_t *TargetTaskData = nullptr;
+
+  /// Correlation id that is incremented with target operations
+  uint64_t TargetRegionOpId = 1;
+
+  /// Used for marking begin of a data operation
+  void beginTargetDataOperation();
+
+  /// Used for marking end of a data operation
+  void endTargetDataOperation();
+
+  /// Used for marking begin of a target region
+  void beginTargetRegion();
+
+  /// Used for marking end of a target region
+  void endTargetRegion();
+};
+
+} // namespace ompt
+} // namespace target
+} // namespace omp
+} // namespace llvm
+
+extern thread_local llvm::omp::target::ompt::Interface OmptInterface;
+
+#endif // _OMPTARGET_OMPTINTERFACE_H

openmp/runtime/src/ompt-general.cpp

if (ompt_start_tool_result) {
     kmp_info_t *root_thread = ompt_get_thread();
 
     ompt_set_thread_state(root_thread, ompt_state_overhead);
 
     if (ompt_enabled.ompt_callback_thread_begin) {
       ompt_callbacks.ompt_callback(ompt_callback_thread_begin)(
           ompt_thread_initial, __ompt_get_thread_data_internal());
     }
-    ompt_data_t *task_data;
-    ompt_data_t *parallel_data;
+    ompt_data_t *task_data = nullptr;
+    ompt_data_t *parallel_data = nullptr;
     __ompt_get_task_info_internal(0, NULL, &task_data, NULL, &parallel_data,
                                   NULL);
     if (ompt_enabled.ompt_callback_implicit_task) {
       ompt_callbacks.ompt_callback(ompt_callback_implicit_task)(
           ompt_scope_begin, parallel_data, task_data, 1, 1, ompt_task_initial);
     }
 
     ompt_set_thread_state(root_thread, ompt_state_work_serial);
#define ompt_interface_fn(fn)                                                  \
 
   FOREACH_OMPT_INQUIRY_FN(ompt_interface_fn)
 
 #undef ompt_interface_fn
 
   return NULL;
 }
 
+static ompt_data_t *ompt_get_task_data() { return __ompt_get_task_data(); }
+
+static ompt_data_t *ompt_get_target_task_data() {
+  return __ompt_get_target_task_data();
+}
+
 /// Lookup function to query libomp callbacks registered by the tool
 static ompt_interface_fn_t ompt_libomp_target_fn_lookup(const char *s) {
 #define provide_fn(fn)                                                         \
   if (strcmp(s, #fn) == 0)                                                     \
     return (ompt_interface_fn_t)fn;
 
   provide_fn(ompt_get_callback);
+  provide_fn(ompt_get_task_data);
+  provide_fn(ompt_get_target_task_data);
 #undef provide_fn
 
 #define ompt_interface_fn(fn, type, code)                                      \
   if (strcmp(s, #fn) == 0)                                                     \
     return (ompt_interface_fn_t)ompt_callbacks.ompt_callback(fn);
 
   FOREACH_OMPT_DEVICE_EVENT(ompt_interface_fn)
   FOREACH_OMPT_EMI_EVENT(ompt_interface_fn)

openmp/runtime/src/ompt-internal.h

typedef struct ompt_lw_taskteam_s {
   int heap;
   struct ompt_lw_taskteam_s *parent;
 } ompt_lw_taskteam_t;
 
 typedef struct {
   ompt_data_t thread_data;
   ompt_data_t task_data; /* stored here from implicit barrier-begin until
                             implicit-task-end */
+  ompt_data_t target_task_data; /* required by target support */
   void *return_address; /* stored here on entry of runtime */
   ompt_state_t state;
   ompt_wait_id_t wait_id;
   int ompt_task_yielded;
   int parallel_flags; // information for the last parallel region invoked
   void *idle_frame;
 } ompt_thread_info_t;
 

openmp/runtime/src/ompt-specific.h

 
 void __ompt_lw_taskteam_link(ompt_lw_taskteam_t *lwt, kmp_info_t *thr,
                              int on_heap, bool always = false);
 
 void __ompt_lw_taskteam_unlink(kmp_info_t *thr);
 
 ompt_team_info_t *__ompt_get_teaminfo(int depth, int *size);
 
+ompt_data_t *__ompt_get_task_data();
+
+ompt_data_t *__ompt_get_target_task_data();
+
 ompt_task_info_t *__ompt_get_task_info_object(int depth);
 
 int __ompt_get_parallel_info_internal(int ancestor_level,
                                       ompt_data_t **parallel_data,
                                       int *team_size);
 
 int __ompt_get_task_info_internal(int ancestor_level, int *type,
                                   ompt_data_t **task_data,
                                   ompt_frame_t **task_frame,
                                   ompt_data_t **parallel_data, int *thread_num);
 
 ompt_data_t *__ompt_get_thread_data_internal();
 
 /*
  * Unused currently
 static uint64_t __ompt_get_get_unique_id_internal();
 */
 
 ompt_sync_region_t __ompt_get_barrier_kind(enum barrier_type, kmp_info_t *);
 
 /*****************************************************************************
  * macros
  ****************************************************************************/
 
-#define OMPT_CUR_TASK_INFO(thr) (&(thr->th.th_current_task->ompt_task_info))
+#define OMPT_CUR_TASK_INFO(thr) (&((thr)->th.th_current_task->ompt_task_info))
 #define OMPT_CUR_TASK_DATA(thr)                                                \
-  (&(thr->th.th_current_task->ompt_task_info.task_data))
-#define OMPT_CUR_TEAM_INFO(thr) (&(thr->th.th_team->t.ompt_team_info))
+  (&((thr)->th.th_current_task->ompt_task_info.task_data))
+#define OMPT_CUR_TEAM_INFO(thr) (&((thr)->th.th_team->t.ompt_team_info))
 #define OMPT_CUR_TEAM_DATA(thr)                                                \
-  (&(thr->th.th_team->t.ompt_team_info.parallel_data))
+  (&((thr)->th.th_team->t.ompt_team_info.parallel_data))
 
 #define OMPT_HAVE_WEAK_ATTRIBUTE KMP_HAVE_WEAK_ATTRIBUTE
 #define OMPT_HAVE_PSAPI KMP_HAVE_PSAPI
 #define OMPT_STR_MATCH(haystack, needle) __kmp_str_match(haystack, 0, needle)
 
 inline void *__ompt_load_return_address(int gtid) {
   kmp_info_t *thr = __kmp_threads[gtid];
   void *return_address = thr->th.ompt_thread_info.return_address;

openmp/runtime/src/ompt-specific.cpp

#endif
   }
   //    return lwtask;
 }
 
 //----------------------------------------------------------
 // task support
 //----------------------------------------------------------
 
+ompt_data_t *__ompt_get_task_data() {
+  kmp_info_t *thr = ompt_get_thread();
+  ompt_data_t *task_data = thr ? OMPT_CUR_TASK_DATA(thr) : NULL;
+  return task_data;
+}
+
+ompt_data_t *__ompt_get_target_task_data() {
+  return &__kmp_threads[__kmp_get_gtid()]->th.ompt_thread_info.target_task_data;
+}
+
 int __ompt_get_task_info_internal(int ancestor_level, int *type,
                                   ompt_data_t **task_data,
                                   ompt_frame_t **task_frame,
                                   ompt_data_t **parallel_data,
                                   int *thread_num) {
   if (__kmp_get_gtid() < 0)
     return 0;