[Libomptarget] Don't report lack of CUDA devices

[jdenny]

Summary

Sometimes libomptarget's CUDA plugin produces unhelpful diagnostics
about a lack of CUDA devices before an application runs:

$ clang -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa hello-world.c
$ ./a.out
CUDA error: Error returned from cuInit
CUDA error: no CUDA-capable device is detected
Hello World: 4

This can happen when the CUDA plugin was built but all CUDA devices
are currently disabled in some manner, perhaps because
CUDA_VISIBLE_DEVICES is set to the empty string. As shown in the
above example, it can even happen when we haven't compiled the
application for offloading to CUDA.

The following code from openmp/libomptarget/plugins/cuda/src/rtl.cpp
appears to be intended to handle this case, and it chooses not to
write a diagnostic to stderr unless debugging is enabled:

if (NumberOfDevices == 0) {
  DP("There are no devices supporting CUDA.\n");
  return;
}

The problem is that the above code is never reached because the
earlier cuInit returns CUDA_ERROR_NO_DEVICE. This patch handles
that cuInit case in the same manner as the above code handles the
NumberOfDevices == 0 case.

[tianshilei1992]

LGTM Thanks for the improvement!

[jdenny]

Thanks for the quick review.

[ye-luo]

I guess you machine has the nvidia driver installed but there is no GPU.
When there is no nvidia driver,

Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so': libcuda.so.1: cannot open shared object file: No such file or directory!

[jdenny]

In D130371#3700740, @ye-luo wrote:

I guess you machine has the nvidia driver installed but there is no GPU.

On my laptop, I saw the problem when I just disabled the (discrete) nvidia gpu in favor of integrated graphics... or when I set CUDA_VISIBLE_DEVICES to the empty string.

The machine that originally motivated this change has cuda installed but not the nvidia driver. This patch helped that case too. However, that machine also experienced other strange behavior I don't have any more time right now to pursue, and I ultimately recommended -DLIBOMPTARGET_BUILD_CUDA_PLUGIN=False to get around it. (I would have reported the behavior upstream, but it might be specific to Clacc.) Anyway, my point is that I'm not sure yet that things always work right with a disabled nvidia driver.

[ye-luo]

In D130371#3706544, @jdenny wrote:

In D130371#3700740, @ye-luo wrote:

I guess you machine has the nvidia driver installed but there is no GPU.

On my laptop, I saw the problem when I just disabled the (discrete) nvidia gpu in favor of integrated graphics... or when I set CUDA_VISIBLE_DEVICES to the empty string.

The machine that originally motivated this change has cuda installed but not the nvidia driver. This patch helped that case too. However, that machine also experienced other strange behavior I don't have any more time right now to pursue, and I ultimately recommended -DLIBOMPTARGET_BUILD_CUDA_PLUGIN=False to get around it. (I would have reported the behavior upstream, but it might be specific to Clacc.) Anyway, my point is that I'm not sure yet that things always work right with a disabled nvidia driver.

Thanks for the info. With your patch
clang++ -fopenmp --offload-arch=sm_80,gfx906 main.cpp
CUDA_VISIBLE_DEVICES="" ./a.out # runs fine on the AMD GPU.
so it is good.

openmp/libomptarget/plugins/cuda/dynamic_cuda/cuda.h

 typedef struct CUctx_st *CUcontext;
 typedef struct CUfunc_st *CUfunction;
 typedef struct CUstream_st *CUstream;
 typedef struct CUevent_st *CUevent;
 
 typedef enum cudaError_enum {
   CUDA_SUCCESS = 0,
   CUDA_ERROR_INVALID_VALUE = 1,
+  CUDA_ERROR_NO_DEVICE = 100,
   CUDA_ERROR_INVALID_HANDLE = 400,
 } CUresult;
 
 typedef enum CUstream_flags_enum {
   CU_STREAM_DEFAULT = 0x0,
   CU_STREAM_NON_BLOCKING = 0x1,
 } CUstream_flags;
 

openmp/libomptarget/plugins/cuda/src/rtl.cpp

DeviceRTLTy()
     DP("Start initializing CUDA\n");
 
     CUresult Err = cuInit(0);
     if (Err == CUDA_ERROR_INVALID_HANDLE) {
       // Can't call cuGetErrorString if dlsym failed
       DP("Failed to load CUDA shared library\n");
       return;
     }
+    if (Err == CUDA_ERROR_NO_DEVICE) {
+      DP("There are no devices supporting CUDA.\n");
+      return;
+    }
     if (!checkResult(Err, "Error returned from cuInit\n")) {
       return;
     }
 
     Err = cuDeviceGetCount(&NumberOfDevices);
     if (!checkResult(Err, "Error returned from cuDeviceGetCount\n"))
       return;
 

openmp/libomptarget/test/offloading/cuda_no_devices.c

+// The CUDA plugin used to complain on stderr when no CUDA devices were enabled,
+// and then it let the application run anyway.  Check that there's no such
+// complaint anymore, especially when the user isn't targeting CUDA.
+
+// RUN: %libomptarget-compile-generic
+// RUN: env CUDA_VISIBLE_DEVICES= \
+// RUN:   %libomptarget-run-generic 2>&1 | %fcheck-generic
+
+#include <stdio.h>
+
+// CHECK-NOT: {{.}}
+//     CHECK: Hello World: 4
+// CHECK-NOT: {{.}}
+int main() {
+  int x = 0;
+  #pragma omp target teams num_teams(2) reduction(+:x)
+  x += 2;
+  printf("Hello World: %d\n", x);
+  return 0;
+}