Page MenuHomePhabricator

[Libomptarget] Full implementation of the target-offload-icv
AbandonedPublic

Authored by grokos on Mar 15 2018, 9:16 AM.

Details

Summary

This patch implements full support for the target-offload-icv and regulates the library's behavior in case of offload failure. There are three offload kinds: DEFAULT, DISABLED and MANDATORY.

Initially, the offload kind is retrieved from libomp. If the offload kind is set to DEFAULT, then libomptarget behaves as if the ICV were set to MANDATORY if at least one device is detected in the system, otherwise it falls back to DISABLED.

The behavior of libomptarget is defined as follows:

  1. If the ICV has been set to MANDATORY: If target offload fails, then libomptarget prints a warning and aborts the application. The warning is printed unconditionally, independently from debug/release configuration or the value of LIBOMPTARGET_DEBUG.
  2. If the ICV has been set to DISABLED: Offloading is disabled and all calls to __tgt_* functions return immediately.
  3. If the ICV is set to DEFAULT and an offload operation fails, then there are two cases: (i) If there have been previous successful offload operations then libomptarget prints out a warning and terminates the application. It is not safe to fall back onto the host as the host may not have the most up-to-date version of the application's data. (ii) If there are no previous successful offload operations, then the ICV is set to DISABLED and execution (including any subsequent offload operations) continues on the host.

Diff Detail

Repository
rOMP OpenMP

Event Timeline

grokos created this revision.Mar 15 2018, 9:16 AM

Will this be defined in the next revisions of the standard? I remember OpenMP 4.5 being unclear about these cases.

Openmp has defined OMP_TARGET_OFFLOAD to control offloading, We should be using this

OMP_TARGET_OFFLOAD
6 The OMP_TARGET_OFFLOAD environment variable sets the initial value of the target-offload-var
7 ICV. The value of the OMP_TARGET_OFFLOAD environment variable may be set to one of these
8 values:
9 MANDATORY | DISABLED | DEFAULT
10 The MANDATORY value specifies that a device construct or a device memory routine must execute
11 on a target device. If the device construct cannot execute on its target device, or if a device memory
12 routine fails to execute, a warning is issued and the program execution aborts. Device constructs
13 are exempt from this behavior when an if-clause is present and the if-clause expression evaluates to
14 false.
15 The support of DISABLED is implementation defined. If an implementation supports it, the
16 behavior should be that a device construct must execute on the host. The behavior with this
17 environment value is equivalent to an if clause present on all device constructs, where each of these
18 if clause expressions evaluate to false. Device memory routines behave as if all device number
19 parameters are set to the value returned by omp_get_initial_device(). The
20 omp_get_initial_device() routine returns that no target device is available
21 The DEFAULT value specifies that when one or more target devices are available, the runtime
22 behaves as if this environment variable is set to MANDATORY; otherwise, the runtime behaves as if
23 this environment variable is set to DISABLED.

Right, I just read TR6 and everything is defined there. I'll revise the patch and upload a new diff.

grokos updated this revision to Diff 138615.Mar 15 2018, 1:23 PM
grokos retitled this revision from [Libomptarget] Behavior of library upon target offload failure to [Libomptarget] Full implementation of the target-offload-icv.
grokos edited the summary of this revision. (Show Details)

We have a patch coming with libomp-side addition of OMP_TARGET_OFFLOAD envirable plus an internal API so that libomptarget can get the value from libomp. Would it help if we get this patch out faster?

grokos updated this revision to Diff 138724.Mar 16 2018, 9:27 AM

Waiting for the libomp-side patch for OMP_TARGET_OFFLOAD, I removed parsing this env var from within libomptarget and now this revision only implements the internal logic to handle whatever libomptarget will get by querying libomp. (there is already some code checking for OMP_TARGET_OFFLOAD==DISABLED, I'm leaving it there in order not to break systems which make use of this option right now)

grokos updated this revision to Diff 138760.Mar 16 2018, 1:52 PM
grokos edited the summary of this revision. (Show Details)

Updated the patch to query the target offload kind from libomp instead of parsing OMP_TARGET_OFFLOAD.

grokos updated this revision to Diff 138805.Mar 16 2018, 7:17 PM

Rebase after bug fix.

grokos updated this revision to Diff 139497.Mar 22 2018, 12:57 PM
grokos edited the summary of this revision. (Show Details)Mar 22 2018, 1:02 PM

This has completely gone under my radar. Be sure to ping changes if there is no review and some reviewers are actively working on other parts of LLVM.

21 The DEFAULT value specifies that when one or more target devices are available, the runtime
22 behaves as if this environment variable is set to MANDATORY; otherwise, the runtime behaves as if
23 this environment variable is set to DISABLED.

After reviewing the code (see inline comments) I don't see this behavior implemented: It only falls back to DISABLED, but doesn't raise to MANDATORY. I'm not sure though if there have been changes to the spec after Ravi posted the quote.

Another general comment: This patch doesn't handle API routines, I think they should be affected as well?

libomptarget/src/interface.cpp
28–30

If we are exiting, this shouldn't say WARNING. I think the word failure is enough here?

33–35

Likewise

37

Remove dead code

45–59

I'd prefer eliminating these macros and just putting the check in all entries. This also has the advantage of having precise debugging output.

libomptarget/src/omptarget.cpp
31–38

No need to have this here, just put it in interface.cpp and make it static (don't forget to remove the declaration from the header file!)

Do we want to get this into 7.0?

Do we want to get this into 7.0?

Note that corresponding libomp change committed long ago (r328046).

@grokos do you plan to update this patch? If not I'll try to implement this variable according to TR7 which will be very close to what OpenMP 5.0 will have. It's really annoying that target execution falls back to the host if the device is busy or there is an error in the kernel :-(

Not needed anymore, superseeded by D50522.

grokos abandoned this revision.Aug 16 2018, 9:10 AM