[OpenMP][libomptarget] Notify the plugins regarding new mapping/unmappings

[kevinsala]

Summary

The NextGen plugins use the information regarding new mapping/unmappings to lock/unlock the corresponding host buffer and speed up the host-device transfers. Handle also the buffers that were already locked externally, i.e., from the application.

[kevinsala]

Fix format.

[tianshilei1992]

mapped/unmapped is a confusing name, especially in the context of OpenMP, you can always check if a pointer is mapped in libomptarget. Of course I understand the "map" here is not OpenMP map.

[kevinsala]

In D142514#4078741, @tianshilei1992 wrote:

mapped/unmapped is a confusing name, especially in the context of OpenMP, you can always check if a pointer is mapped in libomptarget. Of course I understand the "map" here is not OpenMP map.

True. Any recommendation for the rename?

[tianshilei1992]

In D142514#4078818, @kevinsala wrote:

In D142514#4078741, @tianshilei1992 wrote:

mapped/unmapped is a confusing name, especially in the context of OpenMP, you can always check if a pointer is mapped in libomptarget. Of course I understand the "map" here is not OpenMP map.

True. Any recommendation for the rename?

I'm working on Nvidia GPUs, so IMO pinned memory makes sense. Not sure if AMD people would object, though in the implementation you are using that name. ;-)

[ye-luo]

openmp/libomptarget/include/omptargetplugin.h

219

The same as line 215

[jdoerfert]

@tianshilei1992 This is supposed to mean "mapped" in the OpenMP sense. So the plugins are informed if a mapping occurs.

[jdoerfert]

I think this is fine. Two nits and Ye's comment need to be addressed. @tianshilei1992 please accept or suggest a different name. Since we really mean "OpenMP mapped" here, I find the name fitting.

openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp

1855

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.h

487

Docs

[tianshilei1992]

If it servers as notifying data mapping, __tgt_rtl_notify_data_map or something like that sounds more appropriate to me. __tgt_rtl_data_mapped sounds like checking whether it is mapped, and that's how it confused me initially.

[ye-luo]

In D142514#4078989, @tianshilei1992 wrote:

If it servers as notifying data mapping, __tgt_rtl_notify_data_map or something like that sounds more appropriate to me. __tgt_rtl_data_mapped sounds like checking whether it is mapped, and that's how it confused me initially.

Can be tgt_rtl_data_notify_mapped and tgt_rtl_data_notify_unmapped by keeping __tgt_rtl_data prefix

[ye-luo]

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.cpp

797

I don't understand this part of code.
The implementation actually lock the host buffer instead of just notifying.

[jdoerfert]

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.cpp

797

While it is mapped, we want to keep it (by default) pinned. The escape hatch (env var) is missing though.
This allows us to avoid the pin/unpin on the data transfer back and intermediate updates.

[ye-luo]

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.cpp

797

Unconditionally pinning mapped host memory should be an option to opt-in rather than default.
If I just map 10 scalars, don't pin them, staging can be better.
if users want fine-grain control, use the lock API. For really lazy users, give them the env var to opt-in.

[jdoerfert]

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.cpp

797

Mapping scalars as firstprivate or as tofrom? The former would not go through this mechanism. The latter makes reasonable sense to pin the page, no?

I agree we need an env var, default is the question.

[ye-luo]

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.cpp

797

I was referring to map(tofrom) scalars. firstprivte is not considered a map technically. I don't think you want to pay the cost of pinning 10 pages due to 10 scalars. Pinning is very expensive (interacting with OS).

[jdoerfert]

Let's make it opt-in then. And we can use heuristics as well as we go.

[kevinsala]

Changes:

• Fixing comments.
• Re-writing the PinnedAllocationMapTy functions to be clearer and easier to read.
• Renaming RTL API functions.
• Disable automatic locking of mapped host buffers by default. Can be enabled by setting LIBOMPTARGET_LOCK_MAPPED_HOST_BUFFERS envar.
• Although the previous envar is not set, mapped buffers that were externally locked (e.g, the app allocating HSA memory) are registered into the plugin's map and we use a single-step asynchronous HSA memory transfer. (@ye-luo)

This patch still detects invalid partial overlapping in offloading/complex_reduction.cpp, which I have to fix. The previous patch fails with OpenMC, I've to check if this also fails.

[kevinsala]

openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp

1855

It's an output parameter. I'll add the proper comment.

[kevinsala]

Fix format.

[kevinsala]

Fixing bug. Now complex reduction tests work.

[tianshilei1992]

openmp/libomptarget/include/omptargetplugin.h

215

__tgt_rtl_notify_data_mapped?

[kevinsala]

openmp/libomptarget/include/omptargetplugin.h

215

This name was proposed by @ye-luo to be consistent with the other data-related API functions:

Can be tgt_rtl_data_notify_mapped and tgt_rtl_data_notify_unmapped by keeping __tgt_rtl_data prefix

[tianshilei1992]

LGTM

[ye-luo]

openmp/libomptarget/src/device.cpp

611

"Size=%" missing "%"

[kevinsala]

Fix debug print

[kevinsala]

The locking of mapped host buffers is disabled by default. Can be enabled through the LIBOMPTARGET_LOCK_MAPPED_HOST_BUFFERS envar. The envar accepts boolean values (on/off) and a special option:

• off: Do not lock mapped host buffers (default).
• on: Lock mapped host buffers automatically, but do not report lock failures if the plugin fails to lock them.
• mandatory: Lock mapped host buffers automatically and treat locking failures in the plugins as fatal errors. This option may be useful for debugging purposes.

[DavidSpickett]

This is probably a very late report but I happened to be looking at openmp bots for a different reason, some of the error returns in this change are failing to compile:
https://lab.llvm.org/buildbot/#/builders/4/builds/32052/steps/5/logs/stdio

Despite the "gcc" name, this bot appears to use clang-11 which is still supported for building llvm.

This is a simplified version: https://godbolt.org/z/x91h9KcGq

I'm not sure how we're expected to use Expected (no pun intended) in this scenario. There is probably a helper function somewhere.

[jhuber6]

In D142514#4206591, @DavidSpickett wrote:

This is probably a very late report but I happened to be looking at openmp bots for a different reason, some of the error returns in this change are failing to compile:
https://lab.llvm.org/buildbot/#/builders/4/builds/32052/steps/5/logs/stdio

Despite the "gcc" name, this bot appears to use clang-11 which is still supported for building llvm.

This is a simplified version: https://godbolt.org/z/x91h9KcGq

I'm not sure how we're expected to use Expected (no pun intended) in this scenario. There is probably a helper function somewhere.

Needs an explicit move https://godbolt.org/z/Kj1s7od7e.

[jhuber6]

In D142514#4206591, @DavidSpickett wrote:

This is probably a very late report but I happened to be looking at openmp bots for a different reason, some of the error returns in this change are failing to compile:
https://lab.llvm.org/buildbot/#/builders/4/builds/32052/steps/5/logs/stdio

Despite the "gcc" name, this bot appears to use clang-11 which is still supported for building llvm.

This is a simplified version: https://godbolt.org/z/x91h9KcGq

I'm not sure how we're expected to use Expected (no pun intended) in this scenario. There is probably a helper function somewhere.

This should resolve it, thanks for bringing this to my attention https://reviews.llvm.org/rGedc03550063c.

openmp/libomptarget/include/device.h

int32_t submitData(void *TgtPtrBegin, void *HstPtrBegin, int64_t Size,
                      AsyncInfoTy &AsyncInfo);
   // Copy data from device back to host
   int32_t retrieveData(void *HstPtrBegin, void *TgtPtrBegin, int64_t Size,
                        AsyncInfoTy &AsyncInfo);
   // Copy data from current device to destination device directly
   int32_t dataExchange(void *SrcPtr, DeviceTy &DstDev, void *DstPtr,
                        int64_t Size, AsyncInfoTy &AsyncInfo);
 
+  /// Notify the plugins about a new mapping starting at the host address
+  /// \p HstPtr and \p Size bytes.
+  int32_t dataMapped(void *HstPtr, int64_t Size);
+
+  /// Notify the plugins about an existing mapping being unmapped starting at
+  /// the host address \p HstPtr.
+  int32_t dataUnmapped(void *HstPtr);
+
   // Launch the kernel identified by \p TgtEntryPtr with the given arguments.
   int32_t launchKernel(void *TgtEntryPtr, void **TgtVarsPtr,
                        ptrdiff_t *TgtOffsets, const KernelArgsTy &KernelArgs,
                        AsyncInfoTy &AsyncInfo);
 
   /// Synchronize device/queue/event based on \p AsyncInfo and return
   /// OFFLOAD_SUCCESS/OFFLOAD_FAIL when succeeds/fails.
   int32_t synchronize(AsyncInfoTy &AsyncInfo);

openmp/libomptarget/include/omptargetplugin.h

 
 // lock/pin host memory
 int32_t __tgt_rtl_data_lock(int32_t ID, void *HstPtr, int64_t Size,
                             void **LockedPtr);
 
 // unlock/unpin host memory
 int32_t __tgt_rtl_data_unlock(int32_t ID, void *HstPtr);
 
+// Notify the plugin about a new mapping starting at the host address \p HstPtr
+// and \p Size bytes. The plugin may lock/pin that buffer to achieve optimal
+// memory transfers involving that buffer.
+int32_t __tgt_rtl_data_mapped(int32_t ID, void *HstPtr, int64_t Size);
+
+// Notify the plugin about an existing mapping being unmapped, starting at the
+// host address \p HstPtr and \p Size bytes.
+int32_t __tgt_rtl_data_mapped(int32_t ID, void *HstPtr, int64_t Size);
+
 #ifdef __cplusplus
 }
 #endif
 
 #endif // _OMPTARGETPLUGIN_H_

openmp/libomptarget/include/rtl.h

struct RTLInfoTy {
   typedef int32_t(sync_event_ty)(int32_t, void *);
   typedef int32_t(destroy_event_ty)(int32_t, void *);
   typedef int32_t(release_async_info_ty)(int32_t, __tgt_async_info *);
   typedef int32_t(init_async_info_ty)(int32_t, __tgt_async_info **);
   typedef int64_t(init_device_into_ty)(int64_t, __tgt_device_info *,
                                        const char **);
   typedef int32_t(data_lock_ty)(int32_t, void *, int64_t, void **);
   typedef int32_t(data_unlock_ty)(int32_t, void *);
+  typedef int32_t(data_mapped_ty)(int32_t, void *, int64_t);
+  typedef int32_t(data_unmapped_ty)(int32_t, void *);
 
   int32_t Idx = -1;             // RTL index, index is the number of devices
                                 // of other RTLs that were registered before,
                                 // i.e. the OpenMP index of the first device
                                 // to be registered with this RTL.
   int32_t NumberOfDevices = -1; // Number of devices this RTL deals with.
 
   std::unique_ptr<llvm::sys::DynamicLibrary> LibraryHandler;
#endif
   wait_event_ty *wait_event = nullptr;
   sync_event_ty *sync_event = nullptr;
   destroy_event_ty *destroy_event = nullptr;
   init_async_info_ty *init_async_info = nullptr;
   init_device_into_ty *init_device_info = nullptr;
   release_async_info_ty *release_async_info = nullptr;
   data_lock_ty *data_lock = nullptr;
   data_unlock_ty *data_unlock = nullptr;
+  data_mapped_ty *data_mapped = nullptr;
+  data_unmapped_ty *data_unmapped = nullptr;
 
   // Are there images associated with this RTL.
   bool IsUsed = false;
 
   llvm::DenseSet<const __tgt_device_image *> UsedImages;
 
   // Mutex for thread-safety when calling RTL interface functions.
   // It is easier to enforce thread-safety at the libomptarget level,

openmp/libomptarget/plugins-nextgen/amdgpu/src/rtl.cpp

struct AMDGPUDeviceTy : public GenericDeviceTy, AMDGenericDeviceTy {
   }
 
   /// Unpin the host buffer.
   Error dataUnlockImpl(void *HstPtr) override {
     hsa_status_t Status = hsa_amd_memory_unlock(HstPtr);
     return Plugin::check(Status, "Error in hsa_amd_memory_unlock: %s\n");
   }
 
+  Expected<bool> isPinnedPtrImpl(void *HstPtr, void *&HstAllocPtr,
+                                 void *&DevAccessibleAllocPtr,
+                                 size_t &AllocSize) const override {
+    hsa_amd_pointer_info_t Info;
+    Info.size = sizeof(hsa_amd_pointer_info_t);
+    hsa_status_t Status =
+        hsa_amd_pointer_info(HstPtr, &Info, /* Allocator */ nullptr,
+                             /* Number of accessible agents */ nullptr,
+                             /* Accessible agents */ nullptr);
+
+    if (auto Err = Plugin::check(Status, "Error in hsa_amd_pointer_info: %s"))
+      return Err;
+
+    if (Info.type != HSA_EXT_POINTER_TYPE_LOCKED &&
+        Info.type != HSA_EXT_POINTER_TYPE_HSA)
+      return false;
+
+    assert(Info.hostBaseAddress && "Invalid host pinned address");
+    assert(Info.agentBaseAddress && "Invalid agent pinned address");
+    assert(Info.sizeInBytes > 0 && "Invalid pinned allocation size");
+
+    HstAllocPtr = Info.hostBaseAddress;
+    DevAccessibleAllocPtr = Info.agentBaseAddress;
+    AllocSize = Info.sizeInBytes;
+
+    return true;
+  }
+
   /// Submit data to the device (host to device transfer).
   Error dataSubmitImpl(void *TgtPtr, const void *HstPtr, int64_t Size,
                        AsyncInfoWrapperTy &AsyncInfoWrapper) override {
     // Use one-step asynchronous operation when host memory is already pinned.
     if (void *PinnedPtr =
             PinnedAllocs.getDeviceAccessiblePtrFromPinnedBuffer(HstPtr)) {
       AMDGPUStreamTy &Stream = getStream(AsyncInfoWrapper);
       return Stream.pushPinnedMemoryCopyAsync(TgtPtr, PinnedPtr, Size);

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.h

struct EntryTy {
     /// The pointer that devices' driver should use to transfer data from/to the
     /// pinned allocation. In most plugins, this pointer will be the same as the
     /// host pointer above.
     void *DevAccessiblePtr;
 
     /// The size of the pinned allocation.
     size_t Size;
 
+    /// Indicate whether the allocation was pinned from outside the plugin, for
+    /// instance, from the application. The externally pinned allocations are
+    /// not unlocked when unregistering the last user.
+    bool ExternallyPinned;
+
     /// The number of references to the pinned allocation. The allocation should
     /// remain pinned and registered to the map until the number of references
     /// becomes zero.
     mutable size_t References;
 
-    /// Create an entry with the host and device acessible pointers, and the
-    /// buffer size.
-    EntryTy(void *HstPtr, void *DevAccessiblePtr, size_t Size)
+    /// Create an entry with the host and device acessible pointers, the buffer
+    /// size, and a boolean indicating whether the buffer was locked externally.
+    EntryTy(void *HstPtr, void *DevAccessiblePtr, size_t Size,
+            bool ExternallyPinned = false)
         : HstPtr(HstPtr), DevAccessiblePtr(DevAccessiblePtr), Size(Size),
-          References(1) {}
+          ExternallyPinned(ExternallyPinned), References(1) {}
 
     /// Utility constructor used for std::set searches.
     EntryTy(void *HstPtr)
-        : HstPtr(HstPtr), DevAccessiblePtr(nullptr), Size(0), References(0) {}
+        : HstPtr(HstPtr), DevAccessiblePtr(nullptr), Size(0),
+          ExternallyPinned(false), References(0) {}
   };
 
   /// Comparator of mep entries. Use the host pointer to enforce an order
   /// between entries.
   struct EntryCmpTy {
     bool operator()(const EntryTy &Left, const EntryTy &Right) const {
       return Left.HstPtr < Right.HstPtr;
     }
public:
   /// previously registered using the registerHostBuffer function. When calling
   /// this function, the pinned allocation cannot have any other user.
   Error unregisterHostBuffer(void *HstPtr);
 
   /// Lock the host buffer at \p HstPtr or register a new user if it intersects
   /// with an already existing one. A partial overlapping with extension is not
   /// allowed. The function returns the device accessible pointer of the pinned
   /// buffer. The buffer must be unlocked using the unlockHostBuffer function.
-  Expected<void *> lockHostBuffer(void *HstPtr, size_t Size);
+  /// The last parameter indicates whether the buffer is being locked because of
+  /// a new mapping or due to a explicit lock operation.
+  Expected<void *> lockHostBuffer(void *HstPtr, size_t Size,
+                                  bool FromMapping = false);
 
   /// Unlock the host buffer at \p HstPtr or unregister a user if other users
   /// are still using the pinned allocation. If this was the last user, the
   /// pinned allocation is removed from the map and the memory is unlocked.
   Error unlockHostBuffer(void *HstPtr);
 
   /// Return the device accessible pointer associated to the host pinned
   /// allocation which the \p HstPtr belongs, if any. Return null in case the
struct GenericDeviceTy : public DeviceAllocatorTy {
 
   /// Unpin a host memory buffer that was previously pinned.
   Error dataUnlock(void *HstPtr) {
     return PinnedAllocs.unlockHostBuffer(HstPtr);
   }
 
   virtual Error dataUnlockImpl(void *HstPtr) = 0;
 
+  /// Pin the buffer \p HstPtr with \p Size because a new mapping was created.
+  Error dataMapped(void *HstPtr, int64_t Size);
+
+  /// Unpin the \p HstPtr buffer because the mapping was removed.
+  Error dataUnmapped(void *HstPtr) {
+    return PinnedAllocs.unlockHostBuffer(HstPtr);
+  }
+
+  virtual Expected<bool> isPinnedPtrImpl(void *HstPtr, void *&HstAllocPtr,
+                                         void *&DevAccessibleAllocPtr,
+                                         size_t &AllocSize) const = 0;
+
   /// Submit data to the device (host to device transfer).
   Error dataSubmit(void *TgtPtr, const void *HstPtr, int64_t Size,
                    __tgt_async_info *AsyncInfo);
   virtual Error dataSubmitImpl(void *TgtPtr, const void *HstPtr, int64_t Size,
                                AsyncInfoWrapperTy &AsyncInfoWrapper) = 0;
 
   /// Retrieve data from the device (device to host transfer).
   Error dataRetrieve(void *HstPtr, const void *TgtPtr, int64_t Size,

openmp/libomptarget/plugins-nextgen/common/PluginInterface/PluginInterface.cpp

Error PinnedAllocationMapTy::unregisterHostBuffer(void *HstPtr) {
 
   // Remove the entry from the map.
   Allocs.erase(It);
 
   return Plugin::success();
 }
 
 Expected<void *> PinnedAllocationMapTy::lockHostBuffer(void *HstPtr,
-                                                       size_t Size) {
+                                                       size_t Size,
+                                                       bool FromMapping) {
   assert(HstPtr && "Invalid pointer");
 
   std::lock_guard<std::shared_mutex> Lock(Mutex);
 
   auto It = findIntersecting(HstPtr);
 
   // No intersecting registered allocation found in the map. We must lock and
   // register the memory buffer into the map.
   if (It == Allocs.end()) {
+    // When locking because a new mapping, check whether the buffer is already
+    // locked externally, e.g., the application.
+    if (FromMapping) {
+      size_t AllocSize;
+      void *HstAllocPtr, *DevAccessibleAllocPtr;
+
+      // Check whether the host pointer has been externally allocated.
+      auto IsPinnedOrErr = Device.isPinnedPtrImpl(
+          HstPtr, HstAllocPtr, DevAccessibleAllocPtr, AllocSize);
+      if (!IsPinnedOrErr)
+        return IsPinnedOrErr.takeError();
+
+      // Only insert the original allocation into the map if it is already
+      // pinned.
+      if (*IsPinnedOrErr) {
+        auto Res = Allocs.insert({HstAllocPtr, DevAccessibleAllocPtr, AllocSize,
+                                  /* Externally locked */ true});
+        if (!Res.second)
+          return Plugin::error("Cannot register locked buffer");
+
+        return advanceVoidPtr(DevAccessibleAllocPtr,
+                              getPtrDiff(HstPtr, HstAllocPtr));
+      }
+    }
+
     // First, lock the host buffer and retrieve the device accessible pointer.
     auto PinnedPtrOrErr = Device.dataLockImpl(HstPtr, Size);
     if (!PinnedPtrOrErr)
       return PinnedPtrOrErr.takeError();
 
     // Then, insert the host buffer entry into the map.
     auto Res = Allocs.insert({HstPtr, *PinnedPtrOrErr, Size});
     if (!Res.second)
Error PinnedAllocationMapTy::unlockHostBuffer(void *HstPtr) {
 
   const EntryTy &Entry = *It;
 
   // Decrease the number of references. No need to do anything if there are
   // others using the allocation.
   if (--Entry.References > 0)
     return Plugin::success();
 
-  // This was the last user of the allocation. Unlock the original locked memory
-  // buffer, which is the host pointer stored in the entry.
+  // This was the last user of the allocation. Unlock the original locked buffer
+  // if it was locked by the plugin. Do not unlock it if it was locked by an
+  // external entity. Unlock the buffer using the host pointer of the entry.
+  if (!Entry.ExternallyPinned)
     if (auto Err = Device.dataUnlockImpl(Entry.HstPtr))
       return Err;
 
   // Remove the entry from the map.
   size_t Erased = Allocs.erase(Entry);
   if (!Erased)
     return Plugin::error("Cannot find locked buffer");
 
   return Plugin::success();
 }
Error GenericDeviceTy::dataDelete(void *TgtPtr, TargetAllocTy Kind) {
   // Unregister deallocated pinned memory buffer if the type is host memory.
   if (Kind == TARGET_ALLOC_HOST)
     if (auto Err = PinnedAllocs.unregisterHostBuffer(TgtPtr))
       return Err;
 
   return Plugin::success();
 }
 
+Error GenericDeviceTy::dataMapped(void *HstPtr, int64_t Size) {
+  auto PinnedPtrOrErr = PinnedAllocs.lockHostBuffer(
+      HstPtr, Size, /* check whether already locked */ true);
+  if (!PinnedPtrOrErr)
+    return PinnedPtrOrErr.takeError();
+
+  return Plugin::success();
+}
+
 Error GenericDeviceTy::dataSubmit(void *TgtPtr, const void *HstPtr,
                                   int64_t Size, __tgt_async_info *AsyncInfo) {
   auto Err = Plugin::success();
   AsyncInfoWrapperTy AsyncInfoWrapper(Err, *this, AsyncInfo);
   Err = dataSubmitImpl(TgtPtr, HstPtr, Size, AsyncInfoWrapper);
   return Err;
 }
 
if (Err) {
     REPORT("Failure to unlock memory %p: %s\n", Ptr,
            toString(std::move(Err)).data());
     return OFFLOAD_FAIL;
   }
 
   return OFFLOAD_SUCCESS;
 }
 
+int32_t __tgt_rtl_data_mapped(int32_t DeviceId, void *HstPtr, int64_t Size) {
+  auto Err = Plugin::get().getDevice(DeviceId).dataMapped(HstPtr, Size);
+  if (Err) {
+    REPORT("Failure to register mapped memory %p: %s\n", HstPtr,
+           toString(std::move(Err)).data());
+    return OFFLOAD_FAIL;
+  }
+
+  return OFFLOAD_SUCCESS;
+}
+
+int32_t __tgt_rtl_data_unmapped(int32_t DeviceId, void *HstPtr) {
+  auto Err = Plugin::get().getDevice(DeviceId).dataUnmapped(HstPtr);
+  if (Err) {
+    REPORT("Failure to unregister unmapped memory %p: %s\n", HstPtr,
+           toString(std::move(Err)).data());
+    return OFFLOAD_FAIL;
+  }
+
+  return OFFLOAD_SUCCESS;
+}
+
 int32_t __tgt_rtl_data_submit(int32_t DeviceId, void *TgtPtr, void *HstPtr,
                               int64_t Size) {
   return __tgt_rtl_data_submit_async(DeviceId, TgtPtr, HstPtr, Size,
                                      /* AsyncInfoPtr */ nullptr);
 }
 
 int32_t __tgt_rtl_data_submit_async(int32_t DeviceId, void *TgtPtr,
                                     void *HstPtr, int64_t Size,

openmp/libomptarget/plugins-nextgen/cuda/src/rtl.cpp

struct CUDADeviceTy : public GenericDeviceTy {
 
   Expected<void *> dataLockImpl(void *HstPtr, int64_t Size) override {
     // TODO: Register the buffer as CUDA host memory.
     return HstPtr;
   }
 
   Error dataUnlockImpl(void *HstPtr) override { return Plugin::success(); }
 
+  Expected<bool> isPinnedPtrImpl(void *HstPtr, void *&HstAllocPtr,
+                                 void *&DevAccessibleAllocPtr,
+                                 size_t &AllocSize) const override {
+    return false;
+  }
+
   /// Submit data to the device (host to device transfer).
   Error dataSubmitImpl(void *TgtPtr, const void *HstPtr, int64_t Size,
                        AsyncInfoWrapperTy &AsyncInfoWrapper) override {
     if (auto Err = setContext())
       return Err;
 
     CUstream Stream = getStream(AsyncInfoWrapper);
     if (!Stream)

openmp/libomptarget/plugins-nextgen/generic-elf-64bit/src/rtl.cpp

struct GenELF64DeviceTy : public GenericDeviceTy {
   /// return the same pointer as the device pointer.
   Expected<void *> dataLockImpl(void *HstPtr, int64_t Size) override {
     return HstPtr;
   }
 
   /// Nothing to do when unlocking the buffer.
   Error dataUnlockImpl(void *HstPtr) override { return Plugin::success(); }
 
+  Expected<bool> isPinnedPtrImpl(void *HstPtr, void *&HstAllocPtr,
+                                 void *&DevAccessibleAllocPtr,
+                                 size_t &AllocSize) const override {
+    return false;
+  }
+
   /// Submit data to the device (host to device transfer).
   Error dataSubmitImpl(void *TgtPtr, const void *HstPtr, int64_t Size,
                        AsyncInfoWrapperTy &AsyncInfoWrapper) override {
     std::memcpy(TgtPtr, HstPtr, Size);
     return Plugin::success();
   }
 
   /// Retrieve data from the device (device to host transfer).

openmp/libomptarget/src/device.cpp

int DeviceTy::associatePtr(void *HstPtrBegin, void *TgtPtrBegin, int64_t Size) {
   DP("Creating new map entry: HstBase=" DPxMOD ", HstBegin=" DPxMOD
      ", HstEnd=" DPxMOD ", TgtBegin=" DPxMOD ", DynRefCount=%s, "
      "HoldRefCount=%s\n",
      DPxPTR(NewEntry.HstPtrBase), DPxPTR(NewEntry.HstPtrBegin),
      DPxPTR(NewEntry.HstPtrEnd), DPxPTR(NewEntry.TgtPtrBegin),
      NewEntry.dynRefCountToStr().c_str(), NewEntry.holdRefCountToStr().c_str());
   (void)NewEntry;
 
-  return OFFLOAD_SUCCESS;
+  // Notify about the new mapping.
+  return dataMapped(HstPtrBegin, Size);
 }
 
 int DeviceTy::disassociatePtr(void *HstPtrBegin) {
   HDTTMapAccessorTy HDTTMap = HostDataToTargetMap.getExclusiveAccessor();
 
   auto It = HDTTMap->find(HstPtrBegin);
   if (It != HDTTMap->end()) {
     HostDataToTargetTy &HDTT = *It->HDTT;
     // Mapping exists
     if (HDTT.getHoldRefCount()) {
       // This is based on OpenACC 3.1, sec 3.2.33 "acc_unmap_data", L3656-3657:
       // "It is an error to call acc_unmap_data if the structured reference
       // count for the pointer is not zero."
       REPORT("Trying to disassociate a pointer with a non-zero hold reference "
              "count\n");
     } else if (HDTT.isDynRefCountInf()) {
       DP("Association found, removing it\n");
       void *Event = HDTT.getEvent();
       delete &HDTT;
       if (Event)
         destroyEvent(Event);
       HDTTMap->erase(It);
-      return OFFLOAD_SUCCESS;
+
+      // Notify about the unmapped memory.
+      return dataUnmapped(HstPtrBegin);
     } else {
       REPORT("Trying to disassociate a pointer which was not mapped via "
              "omp_target_associate_ptr\n");
     }
   } else {
     REPORT("Association not found\n");
   }
 
if (LR.Flags.IsContained ||
     INFO(OMP_INFOTYPE_MAPPING_CHANGED, DeviceID,
          "Creating new map entry with HstPtrBase=" DPxMOD
          ", HstPtrBegin=" DPxMOD ", TgtPtrBegin=" DPxMOD ", Size=%ld, "
          "DynRefCount=%s, HoldRefCount=%s, Name=%s\n",
          DPxPTR(HstPtrBase), DPxPTR(HstPtrBegin), DPxPTR(Ptr), Size,
          Entry->dynRefCountToStr().c_str(), Entry->holdRefCountToStr().c_str(),
          (HstPtrName) ? getNameFromMapping(HstPtrName).c_str() : "unknown");
     TargetPointer = (void *)Ptr;
+
+    // Notify about the new mapping.
+    if (dataMapped(HstPtrBegin, Size))
+      return {{false /* IsNewEntry */, false /* IsHostPointer */},
+              nullptr /* Entry */,
+              nullptr /* TargetPointer */};
   } else {
     // This entry is not present and we did not create a new entry for it.
     IsPresent = false;
   }
 
   // If the target pointer is valid, and we need to transfer data, issue the
   // data transfer.
   if (TargetPointer && !IsHostPtr && HasFlagTo && (IsNew || HasFlagAlways)) {
int DeviceTy::deallocTgtPtrAndEntry(HostDataToTargetTy *Entry, int64_t Size) {
 
   void *Event = Entry->getEvent();
   if (Event && destroyEvent(Event) != OFFLOAD_SUCCESS) {
     REPORT("Failed to destroy event " DPxMOD "\n", DPxPTR(Event));
     return OFFLOAD_FAIL;
   }
 
   int Ret = deleteData((void *)Entry->TgtPtrBegin);
+
+  // Notify about the unmapped memory.
+  Ret |= dataUnmapped((void *)Entry->HstPtrBegin);
+
   delete Entry;
 
   return Ret;
 }
 
 /// Init device, should not be called directly.
 void DeviceTy::init() {
   // Make call to init_requires if it exists for this plugin.
if (!AsyncInfo || !RTL->data_exchange_async || !RTL->synchronize) {
     assert(RTL->data_exchange && "RTL->data_exchange is nullptr");
     return RTL->data_exchange(RTLDeviceID, SrcPtr, DstDev.RTLDeviceID, DstPtr,
                               Size);
   }
   return RTL->data_exchange_async(RTLDeviceID, SrcPtr, DstDev.RTLDeviceID,
                                   DstPtr, Size, AsyncInfo);
 }
 
+int32_t DeviceTy::dataMapped(void *HstPtr, int64_t Size) {
+  if (!RTL->data_mapped)
+    return OFFLOAD_SUCCESS;
+
+  DP("Notifying about new mapping: HstPtr=" DPxMOD ", Size=" PRId64 "\n",
+     DPxPTR(HstPtr), Size);
+
+  if (RTL->data_mapped(RTLDeviceID, HstPtr, Size)) {
+    REPORT("Notifiying about data mapping failed.\n");
+    return OFFLOAD_FAIL;
+  }
+  return OFFLOAD_SUCCESS;
+}
+
+int32_t DeviceTy::dataUnmapped(void *HstPtr) {
+  if (!RTL->data_unmapped)
+    return OFFLOAD_SUCCESS;
+
+  DP("Notifying about an unmapping: HstPtr=" DPxMOD "\n", DPxPTR(HstPtr));
+
+  if (RTL->data_unmapped(RTLDeviceID, HstPtr)) {
+    REPORT("Notifiying about data unmapping failed.\n");
+    return OFFLOAD_FAIL;
+  }
+  return OFFLOAD_SUCCESS;
+}
+
 // Run region on device
 int32_t DeviceTy::launchKernel(void *TgtEntryPtr, void **TgtVarsPtr,
                                ptrdiff_t *TgtOffsets,
                                const KernelArgsTy &KernelArgs,
                                AsyncInfoTy &AsyncInfo) {
   return RTL->launch_kernel(RTLDeviceID, TgtEntryPtr, TgtVarsPtr, TgtOffsets,
                             &KernelArgs, AsyncInfo);
 }

openmp/libomptarget/src/omptarget.cpp

for (auto *HostEntriesBegin : PM->HostEntriesBeginRegistrationOrder) {
           HDTTMap->emplace(new HostDataToTargetTy(
               (uintptr_t)CurrHostEntry->addr /*HstPtrBase*/,
               (uintptr_t)CurrHostEntry->addr /*HstPtrBegin*/,
               (uintptr_t)CurrHostEntry->addr +
                   CurrHostEntry->size /*HstPtrEnd*/,
               (uintptr_t)CurrDeviceEntry->addr /*TgtPtrBegin*/,
               false /*UseHoldRefCount*/, CurrHostEntry->name,
               true /*IsRefCountINF*/));
+
+          // Notify about the new mapping.
+          if (Device.dataMapped(CurrHostEntry->addr, CurrHostEntry->size))
+            return OFFLOAD_FAIL;
         }
       }
     }
   }
 
   if (Rc != OFFLOAD_SUCCESS) {
     return Rc;
   }

openmp/libomptarget/src/rtl.cpp

#endif
   *((void **)&RTL.init_async_info) =
       DynLibrary->getAddressOfSymbol("__tgt_rtl_init_async_info");
   *((void **)&RTL.init_device_info) =
       DynLibrary->getAddressOfSymbol("__tgt_rtl_init_device_info");
   *((void **)&RTL.data_lock) =
       DynLibrary->getAddressOfSymbol("__tgt_rtl_data_lock");
   *((void **)&RTL.data_unlock) =
       DynLibrary->getAddressOfSymbol("__tgt_rtl_data_unlock");
+  *((void **)&RTL.data_mapped) =
+      DynLibrary->getAddressOfSymbol("__tgt_rtl_data_mapped");
+  *((void **)&RTL.data_unmapped) =
+      DynLibrary->getAddressOfSymbol("__tgt_rtl_data_unmapped");
 
   RTL.LibraryHandler = std::move(DynLibrary);
 
   // Successfully loaded
   return true;
 }
 
 ////////////////////////////////////////////////////////////////////////////////