[Libomptarget] Full implementation of the target-offload-icv

[grokos]

Summary

This patch implements full support for the target-offload-icv and regulates the library's behavior in case of offload failure. There are three offload kinds: DEFAULT, DISABLED and MANDATORY.

Initially, the offload kind is retrieved from libomp. If the offload kind is set to DEFAULT, then libomptarget behaves as if the ICV were set to MANDATORY if at least one device is detected in the system, otherwise it falls back to DISABLED.

The behavior of libomptarget is defined as follows:

1. If the ICV has been set to MANDATORY: If target offload fails, then libomptarget prints a warning and aborts the application. The warning is printed unconditionally, independently from debug/release configuration or the value of LIBOMPTARGET_DEBUG.
2. If the ICV has been set to DISABLED: Offloading is disabled and all calls to __tgt_* functions return immediately.
3. If the ICV is set to DEFAULT and an offload operation fails, then there are two cases: (i) If there have been previous successful offload operations then libomptarget prints out a warning and terminates the application. It is not safe to fall back onto the host as the host may not have the most up-to-date version of the application's data. (ii) If there are no previous successful offload operations, then the ICV is set to DISABLED and execution (including any subsequent offload operations) continues on the host.

[Hahnfeld]

Will this be defined in the next revisions of the standard? I remember OpenMP 4.5 being unclear about these cases.

[RaviNarayanaswamy]

Openmp has defined OMP_TARGET_OFFLOAD to control offloading, We should be using this

OMP_TARGET_OFFLOAD
6 The OMP_TARGET_OFFLOAD environment variable sets the initial value of the target-offload-var
7 ICV. The value of the OMP_TARGET_OFFLOAD environment variable may be set to one of these
8 values:
9 MANDATORY | DISABLED | DEFAULT
10 The MANDATORY value specifies that a device construct or a device memory routine must execute
11 on a target device. If the device construct cannot execute on its target device, or if a device memory
12 routine fails to execute, a warning is issued and the program execution aborts. Device constructs
13 are exempt from this behavior when an if-clause is present and the if-clause expression evaluates to
14 false.
15 The support of DISABLED is implementation defined. If an implementation supports it, the
16 behavior should be that a device construct must execute on the host. The behavior with this
17 environment value is equivalent to an if clause present on all device constructs, where each of these
18 if clause expressions evaluate to false. Device memory routines behave as if all device number
19 parameters are set to the value returned by omp_get_initial_device(). The
20 omp_get_initial_device() routine returns that no target device is available
21 The DEFAULT value specifies that when one or more target devices are available, the runtime
22 behaves as if this environment variable is set to MANDATORY; otherwise, the runtime behaves as if
23 this environment variable is set to DISABLED.

[grokos]

Right, I just read TR6 and everything is defined there. I'll revise the patch and upload a new diff.

[tlwilmar]

We have a patch coming with libomp-side addition of OMP_TARGET_OFFLOAD envirable plus an internal API so that libomptarget can get the value from libomp. Would it help if we get this patch out faster?

[grokos]

Waiting for the libomp-side patch for OMP_TARGET_OFFLOAD, I removed parsing this env var from within libomptarget and now this revision only implements the internal logic to handle whatever libomptarget will get by querying libomp. (there is already some code checking for OMP_TARGET_OFFLOAD==DISABLED, I'm leaving it there in order not to break systems which make use of this option right now)

[grokos]

Updated the patch to query the target offload kind from libomp instead of parsing OMP_TARGET_OFFLOAD.

[grokos]

Rebase after bug fix.

[Hahnfeld]

This has completely gone under my radar. Be sure to ping changes if there is no review and some reviewers are actively working on other parts of LLVM.

In D44522#1039056, @RaviNarayanaswamy wrote:

21 The DEFAULT value specifies that when one or more target devices are available, the runtime
22 behaves as if this environment variable is set to MANDATORY; otherwise, the runtime behaves as if
23 this environment variable is set to DISABLED.

After reviewing the code (see inline comments) I don't see this behavior implemented: It only falls back to DISABLED, but doesn't raise to MANDATORY. I'm not sure though if there have been changes to the spec after Ravi posted the quote.

Another general comment: This patch doesn't handle API routines, I think they should be affected as well?

libomptarget/src/interface.cpp

28–30

If we are exiting, this shouldn't say WARNING. I think the word failure is enough here?

33–35

Likewise

37

Remove dead code

45–59

I'd prefer eliminating these macros and just putting the check in all entries. This also has the advantage of having precise debugging output.

libomptarget/src/omptarget.cpp

31–38

No need to have this here, just put it in interface.cpp and make it static (don't forget to remove the declaration from the header file!)

[Hahnfeld]

Do we want to get this into 7.0?

[AndreyChurbanov]

In D44522#1175241, @Hahnfeld wrote:

Do we want to get this into 7.0?

Note that corresponding libomp change committed long ago (r328046).

[Hahnfeld]

@grokos do you plan to update this patch? If not I'll try to implement this variable according to TR7 which will be very close to what OpenMP 5.0 will have. It's really annoying that target execution falls back to the host if the device is busy or there is an error in the kernel :-(

[Hahnfeld]

Not needed anymore, superseeded by D50522.

libomptarget/src/interface.cpp

 
 #include "device.h"
 #include "private.h"
 #include "rtl.h"
 
 #include <cassert>
 #include <cstdlib>
 
+static inline void checkOffloadResult(int rc) {
+  if (rc == OFFLOAD_FAIL && TargetOffloadKind == OMP_TGT_OFFLOAD_MANDATORY) {
+    fprintf(stderr, "WARNING: Target offload failure, terminating application"
+        "\n");
+    exit(OFFLOAD_FAIL);
+  }
+}
+
+#define CHECK_HOST_FALLBACK_DATA()                                             \
+  {                                                                            \
+    if (TargetOffloadKind == OMP_TGT_OFFLOAD_DISABLED) {                       \
+      DP("Target offload disabled, ignoring \"target data\", "                 \
+          "\"target enter/exit data\" or \"target update\"\n");                \
+      return;                                                                  \
+    }                                                                          \
+  }
+#define CHECK_HOST_FALLBACK_TARGET()                                           \
+  {                                                                            \
+    if (TargetOffloadKind == OMP_TGT_OFFLOAD_DISABLED) {                       \
+      DP("Target offload disabled, executing target region on the host\n");    \
+      return OFFLOAD_FAIL;                                                     \
+    }                                                                          \
+  }
+
 ////////////////////////////////////////////////////////////////////////////////
 /// adds a target shared library to the target execution image
 EXTERN void __tgt_register_lib(__tgt_bin_desc *desc) {
   RTLs.RegisterLib(desc);
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 /// unloads a target shared library
 EXTERN void __tgt_unregister_lib(__tgt_bin_desc *desc) {
   RTLs.UnregisterLib(desc);
 }
 
 // Following datatypes and functions (tgt_oldmap_type, combined_entry_t,
 // translate_map, cleanup_map) will be removed once the compiler starts using
 // the new map types.
 
 // Old map types
 enum tgt_oldmap_type {
   OMP_TGT_OLDMAPTYPE_TO          = 0x001, // copy data from host to device
   OMP_TGT_OLDMAPTYPE_FROM        = 0x002, // copy data from device to host
static void cleanup_map(int32_t new_arg_num, void **new_args_base,
   }
 }
 
 /// creates host-to-target data mapping, stores it in the
 /// libomptarget.so internal structure (an entry in a stack of data maps)
 /// and passes the data to the device.
 EXTERN void __tgt_target_data_begin(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  CHECK_HOST_FALLBACK_DATA();
   DP("Entering data begin region for device %ld with %d mappings\n", device_id,
      arg_num);
 
   // No devices available?
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
     DP("Use default device id %ld\n", device_id);
   }
__tgt_target_data_begin(device_id, arg_num, args_base, args, arg_sizes,
                           arg_types);
 }
 
 /// passes data from the target, releases target memory and destroys
 /// the host-target mapping (top entry from the stack of data maps)
 /// created by the last __tgt_target_data_begin.
 EXTERN void __tgt_target_data_end(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  CHECK_HOST_FALLBACK_DATA();
   DP("Entering data end region with %d mappings\n", arg_num);
 
   // No devices available?
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
   RTLsMtx.lock();
if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, 0);
 
   __tgt_target_data_end(device_id, arg_num, args_base, args, arg_sizes,
                         arg_types);
 }
 
 EXTERN void __tgt_target_data_update(int64_t device_id, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  CHECK_HOST_FALLBACK_DATA();
   DP("Entering data update with %d mappings\n", arg_num);
 
   // No devices available?
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
   if (CheckDeviceAndCtors(device_id) != OFFLOAD_SUCCESS) {
if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, 0);
 
   __tgt_target_data_update(device_id, arg_num, args_base, args, arg_sizes,
                            arg_types);
 }
 
 EXTERN int __tgt_target(int64_t device_id, void *host_ptr, int32_t arg_num,
     void **args_base, void **args, int64_t *arg_sizes, int64_t *arg_types) {
+  CHECK_HOST_FALLBACK_TARGET();
   DP("Entering target region with entry point " DPxMOD " and device Id %ld\n",
      DPxPTR(host_ptr), device_id);
 
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
   if (CheckDeviceAndCtors(device_id) != OFFLOAD_SUCCESS) {
EXTERN int __tgt_target(int64_t device_id, void *host_ptr, int32_t arg_num,
   //    arg_types, 0, 0, false /*team*/, false /*recursive*/);
   int rc = target(device_id, host_ptr, new_arg_num, new_args_base, new_args,
       new_arg_sizes, new_arg_types, 0, 0, false /*team*/);
 
   // Cleanup translation memory
   cleanup_map(new_arg_num, new_args_base, new_args, new_arg_sizes,
       new_arg_types, arg_num, args_base);
 
+  checkOffloadResult(rc);
   return rc;
 }
 
 EXTERN int __tgt_target_nowait(int64_t device_id, void *host_ptr,
     int32_t arg_num, void **args_base, void **args, int64_t *arg_sizes,
     int64_t *arg_types, int32_t depNum, void *depList, int32_t noAliasDepNum,
     void *noAliasDepList) {
   if (depNum + noAliasDepNum > 0)
     __kmpc_omp_taskwait(NULL, 0);
 
   return __tgt_target(device_id, host_ptr, arg_num, args_base, args, arg_sizes,
                       arg_types);
 }
 
 EXTERN int __tgt_target_teams(int64_t device_id, void *host_ptr,
     int32_t arg_num, void **args_base, void **args, int64_t *arg_sizes,
     int64_t *arg_types, int32_t team_num, int32_t thread_limit) {
+  CHECK_HOST_FALLBACK_TARGET();
   DP("Entering target region with entry point " DPxMOD " and device Id %ld\n",
      DPxPTR(host_ptr), device_id);
 
   if (device_id == OFFLOAD_DEVICE_DEFAULT) {
     device_id = omp_get_default_device();
   }
 
   if (CheckDeviceAndCtors(device_id) != OFFLOAD_SUCCESS) {
EXTERN int __tgt_target_teams(int64_t device_id, void *host_ptr,
   //              false /*recursive*/);
   int rc = target(device_id, host_ptr, new_arg_num, new_args_base, new_args,
       new_arg_sizes, new_arg_types, team_num, thread_limit, true /*team*/);
 
   // Cleanup translation memory
   cleanup_map(new_arg_num, new_args_base, new_args, new_arg_sizes,
       new_arg_types, arg_num, args_base);
 
+  checkOffloadResult(rc);
   return rc;
 }
 
 EXTERN int __tgt_target_teams_nowait(int64_t device_id, void *host_ptr,
     int32_t arg_num, void **args_base, void **args, int64_t *arg_sizes,
     int64_t *arg_types, int32_t team_num, int32_t thread_limit, int32_t depNum,
     void *depList, int32_t noAliasDepNum, void *noAliasDepList) {
   if (depNum + noAliasDepNum > 0)

libomptarget/src/omptarget.cpp

 
 #include <cassert>
 #include <vector>
 
 #ifdef OMPTARGET_DEBUG
 int DebugLevel = 0;
 #endif // OMPTARGET_DEBUG
 
+// Target offload ICV - can be MANDATORY, DISABLED or DEFAULT.
+tgt_offload_kind TargetOffloadKind;
+
 /// Map global data and execute pending ctors
 static int InitLibrary(DeviceTy& Device) {
   /*
    * Map global data
    */
   int32_t device_id = Device.DeviceID;
   int rc = OFFLOAD_SUCCESS;
 
   Device.PendingGlobalsMtx.lock();
   TrlTblMtx.lock();
   for (HostEntriesBeginToTransTableTy::iterator
       ii = HostEntriesBeginToTransTable.begin();
       ii != HostEntriesBeginToTransTable.end(); ++ii) {
     TranslationTable *TransTable = &ii->second;
     if (TransTable->TargetsTable[device_id] != 0) {
       // Library entries have already been processed

libomptarget/src/private.h

 extern int CheckDeviceAndCtors(int64_t device_id);
 
 // Implemented in libomp, they are called from within __tgt_* functions.
 #ifdef __cplusplus
 extern "C" {
 #endif
 int omp_get_default_device(void) __attribute__((weak));
 int32_t __kmpc_omp_taskwait(void *loc_ref, int32_t gtid) __attribute__((weak));
+int __kmpc_get_target_offload() __attribute__((weak));
 #ifdef __cplusplus
 }
 #endif
 
 #ifdef OMPTARGET_DEBUG
 extern int DebugLevel;
 
 #define DP(...) \
   do { \
     if (DebugLevel > 0) { \
       DEBUGP("Libomptarget", __VA_ARGS__); \
     } \
   } while (false)
 #else // OMPTARGET_DEBUG
 #define DP(...) {}
 #endif // OMPTARGET_DEBUG
 
+// Target offload ICV - must match definition in libomp
+enum tgt_offload_kind {
+  OMP_TGT_OFFLOAD_DISABLED = 0,
+  OMP_TGT_OFFLOAD_DEFAULT = 1,
+  OMP_TGT_OFFLOAD_MANDATORY = 2
+};
+extern tgt_offload_kind TargetOffloadKind;
+
 #endif

libomptarget/src/rtl.cpp

 
 void RTLsTy::LoadRTLs() {
 #ifdef OMPTARGET_DEBUG
   if (char *envStr = getenv("LIBOMPTARGET_DEBUG")) {
     DebugLevel = std::stoi(envStr);
   }
 #endif // OMPTARGET_DEBUG
 
-  // Parse environment variable OMP_TARGET_OFFLOAD (if set)
-  char *envStr = getenv("OMP_TARGET_OFFLOAD");
-  if (envStr && !strcmp(envStr, "DISABLED")) {
+  // Query libomp about the kind of target offload
+  TargetOffloadKind = (tgt_offload_kind) __kmpc_get_target_offload();
+  if (TargetOffloadKind == OMP_TGT_OFFLOAD_DISABLED) {
     DP("Target offloading disabled by environment\n");
     return;
   }
 
   DP("Loading RTLs...\n");
 
   // Attempt to open all the plugins and, if they exist, check if the interface
   // is correct and if they are supporting any devices.
#endif
     if (!(R.NumberOfDevices = R.number_of_devices())) {
       DP("No devices supported in this RTL\n");
       continue;
     }
 
     DP("Registering RTL %s supporting %d devices!\n",
         R.RTLName.c_str(), R.NumberOfDevices);
 
+    // We have found available devices, so the target offload mode must be set
+    // to MANDATORY.
+    TargetOffloadKind = OMP_TGT_OFFLOAD_MANDATORY;
+    DP("Target-offload-icv set to MANDATORY\n");
+
     // The RTL is valid! Will save the information in the RTLs list.
     AllRTLs.push_back(R);
   }
 
   DP("RTLs loaded!\n");
 
+  // At this point, if the target-offload-icv is still DEFAULT, it means that no
+  // devices were found, so the icv falls back to DISABLED.
+  if (TargetOffloadKind == OMP_TGT_OFFLOAD_DEFAULT) {
+    TargetOffloadKind = OMP_TGT_OFFLOAD_DISABLED;
+    DP("Target-offload-icv set to DISABLED\n");
+  }
+
   return;
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 // Functionality for registering libs
 
 static void RegisterImageIntoTranslationTable(TranslationTable &TT,
     RTLInfoTy &RTL, __tgt_device_image *image) {
for (int32_t i = 0; i < RTL->NumberOfDevices; ++i) {
     Device.PendingGlobalsMtx.unlock();
   }
 }
 
 void RTLsTy::RegisterLib(__tgt_bin_desc *desc) {
   // Attempt to load all plugins available in the system.
   std::call_once(initFlag, &RTLsTy::LoadRTLs, this);
 
+  // Return immediately if offloading has been disabled
+  if (TargetOffloadKind == OMP_TGT_OFFLOAD_DISABLED) {
+    DP("Target offload disabled, skipping registering library\n");
+    return;
+  }
+
   RTLsMtx.lock();
   // Register the images with the RTLs that understand them, if any.
   for (int32_t i = 0; i < desc->NumDeviceImages; ++i) {
     // Obtain the image.
     __tgt_device_image *img = &desc->DeviceImages[i];
 
     RTLInfoTy *FoundRTL = NULL;
 
void RTLsTy::RegisterLib(__tgt_bin_desc *desc) {
   }
   RTLsMtx.unlock();
 
 
   DP("Done registering entries!\n");
 }
 
 void RTLsTy::UnregisterLib(__tgt_bin_desc *desc) {
+  // Return immediately if offloading has been disabled
+  if (TargetOffloadKind == OMP_TGT_OFFLOAD_DISABLED) {
+    DP("Target offload disabled, skipping unregister library\n");
+    return;
+  }
+
   DP("Unloading target library!\n");
 
   RTLsMtx.lock();
   // Find which RTL understands each image, if any.
   for (int32_t i = 0; i < desc->NumDeviceImages; ++i) {
     // Obtain the image.
     __tgt_device_image *img = &desc->DeviceImages[i];