[libomptarget][ve] Workaround for the libomptarget memory manager cleanup

[manorom]

Summary

The target memory manager, introduced to libomptarget with rG0289696751e9a959b2413ca26624fc6c91be1eea, will try to free target memory it allocated, using __tgt_rtl_data_delete(), when the host program shuts down.
This is a problem for the VE plugin because it cleans up all its target resources when its destructor is called, which happens before the Memory Manager destructor frees is executed. This means that there is no VE process handle when the Memory Manager tries to free its allocated target memory and the call to veo_free_mem() causes a program crash.

This patch adds a workaround for this problem. It simply makes __tgt_rtl_data_delete() return OFFLOAD_SUCCESS immediately if the plugin has already cleaned up its target memory.

[JonChesterfield]

We really should fix this in libomptarget, but yeah, workaround for now

[JonChesterfield]

A colleague has now written a similar hack for amdgpu. The control flow in libomptarget is difficult to follow, but it is surely possible to move the destructor call.

Until we fix that, I think we should disable the known-broken memory allocator, at least by default.

[tianshilei1992]

I'm trying to fix the issue in D94256. Please take a review. :-)

[tianshilei1992]

The issue has been fixed D94379 by moving the memory manager to plugin. I only enabled CUDA. If VE would like this feature, you can also do it in a similar way.

[manorom]

Obsolete due to D94379

openmp/libomptarget/plugins/ve/src/rtl.cpp

 //===-RTLs/nec-aurora/src/rtl.cpp - Target RTLs Implementation - C++ -*-======//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.txt for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // RTL for NEC Aurora TSUBASA machines
 //
 //===----------------------------------------------------------------------===//
 
 #include <algorithm>
 #include <cassert>
 #include <cerrno>
 #include <cstring>
 #include <list>
 #include <stdlib.h>
 #include <string>
 #include <sys/stat.h>
 #include <ve_offload.h>
 #include <vector>
 #include <veosinfo/veosinfo.h>
 
 #include "Debug.h"
 #include "omptargetplugin.h"
 
 #ifndef TARGET_NAME
 #define TARGET_NAME VE
 #endif
 
 #define DEBUG_PREFIX "Target " GETNAME(TARGET_NAME) " RTL"
 
 #ifndef TARGET_ELF_ID
 #define TARGET_ELF_ID 0
 #endif
 
 #include "../../common/elf_common.c"
 
+static bool isInit = true;
+
 struct DynLibTy {
   char *FileName;
   uint64_t VeoLibHandle;
 };
 
 /// Keep entries table per device.
 struct FuncOrGblEntryTy {
   __tgt_target_table Table;
RTLDeviceInfoTy() {
     DP("Found %i VE devices\n", NumDevices);
     ProcHandles.resize(NumDevices, NULL);
     Contexts.resize(NumDevices, NULL);
     FuncOrGblEntry.resize(NumDevices);
     LibraryHandles.resize(NumDevices);
   }
 
   ~RTLDeviceInfoTy() {
+    isInit = false;
+    DP("Closing VEO contexts\n");
     for (auto &ctx : Contexts) {
       if (ctx != NULL) {
         if (veo_context_close(ctx) != 0) {
           DP("Failed to close VEO context.\n");
         }
       }
     }
 
+    DP("Destroying VEO process handles\n");
     for (auto &hdl : ProcHandles) {
       if (hdl != NULL) {
         veo_proc_destroy(hdl);
       }
     }
 
     for (auto &lib : DynLibs) {
       if (lib.FileName) {
if (ret != 0) {
     return OFFLOAD_FAIL;
   }
   return OFFLOAD_SUCCESS;
 }
 
 // De-allocate the data referenced by target ptr on the device. In case of
 // success, return zero. Otherwise, return an error code.
 int32_t __tgt_rtl_data_delete(int32_t ID, void *TargetPtr) {
+  if (!isInit) {
+    // Workaround for memory manager
+    return OFFLOAD_SUCCESS;
+  }
   int ret =  veo_free_mem(DeviceInfo.ProcHandles[ID], (uint64_t)TargetPtr);
 
   if (ret != 0) {
     DP("veo_free_mem() failed with error code %d\n", ret);
     return OFFLOAD_FAIL;
   }
   return OFFLOAD_SUCCESS;
 }