[driver][CUDA] Use CMake's FindCUDA as default --cuda-path.

[Meinersbur]

Summary

Add the introspection result from find_package(CUDA) as the first candidate when looking for a CUDA installation. This allows specifying the path to the CUDA installation when configuring LLVM instead of in every invocation of clang. Clang and libomptarget both use CMake's introspection to determine the CUDA environment, in particular to determine the default and supported architectures for OpenMP. libomptarget's regression tests assume that clang will find the CUDA installation itself without passing --cuda-path. Ideally, these are consistent. Making Clang also use the CMake introspection was the preferred solution from the OpenMP Multi-Company conference call over adding --cuda-path to every regression test.

To not make the driver regression test dependendent on the build configuration, a new flag --cuda-path-ignore-cmake is added which causes the CUDA_TOOLKIT_ROOT_PATH to not be added to the list of installation candidates.

Some notes:

• Both libomptarget and clang have a workaround for http://bugs.debian.org/882505 (CUDA on Debian and Ubuntu installations with /usr/lib/cuda prefix instead of /usr), but implemented very differently. Also see D40453 and D55588). In my Ubuntu LTS versions 18.04 and 20.04, this seem to be fixed as /usr/lib/cuda still exists but only contain the files version.txt and libdevice.10.bc (though as symbolic link directory), which happen to be the files Clang looks for when searching a CUDA installation. I assume this is a compatibility workaround to not break Clang's compatibility workaround. libomptarget's implementation was to overwrite CUDA_TOOLKIT_ROOT_DIR after invoking FindCUDA, which is an evil thing to do. I replaced it with a PATHS "/usr/lib/cuda" so FindCUDA can check that path itself for older distributions which indeed have CUDA installed there.
• find_package(CUDA) is deprecated by find_package(CUDAToolkit) in CMake 3.17. The required version for LLVM currently is CMake 3.13.4

[tra]

One concern I have is that the path we configure during clang's build is not necessarily the right choice for the user of clang we build. It's likely that the clang in the end will be used on a completely different machine.
E.g. official clang builds can not ever provide the same CUDA path for *all* users who end up using them. Requiring the rest to use a special option to make clang work again looks like an overall usability regression to me.

I think the default should still let clang search for CUDA or require the user to provide correct CUDA path. "Use CUDA path discovered by CMake at build time" should be a non-default configuration option if/when it's needed and appropriate.

Alternatively, I would not object to appending cmake-discovered path to the default search list, or enforcing it for OpenMP builds only if that's what OpenMP needs (though I still don't see how that's going to be a good default for all/most users).

[Hahnfeld]

In D89974#2347938, @tra wrote:

One concern I have is that the path we configure during clang's build is not necessarily the right choice for the user of clang we build. It's likely that the clang in the end will be used on a completely different machine.
E.g. official clang builds can not ever provide the same CUDA path for *all* users who end up using them. Requiring the rest to use a special option to make clang work again looks like an overall usability regression to me.

Not quite, as far as I understand the change it's only one of the searched paths.

I think the default should still let clang search for CUDA or require the user to provide correct CUDA path. "Use CUDA path discovered by CMake at build time" should be a non-default configuration option if/when it's needed and appropriate.

I agree here. It's definitely surprising to make it the *first* path because module loading another CUDA version and putting it into PATH is not recognized anymore.

clang/test/Driver/cuda-detect.cu

9

There's now a space missing before --cuda-path-ignore-env.

[Hahnfeld]

In D89974#2347979, @Hahnfeld wrote:

In D89974#2347938, @tra wrote:

I think the default should still let clang search for CUDA or require the user to provide correct CUDA path. "Use CUDA path discovered by CMake at build time" should be a non-default configuration option if/when it's needed and appropriate.

I agree here. It's definitely surprising to make it the *first* path because module loading another CUDA version and putting it into PATH is not recognized anymore.

Also note that GCC_INSTALL_PREFIX works differently (empty by default) and must be set explicitly during configure to make the build embed a path into the final executable. Maybe that's the better way to go?

[tra]

In D89974#2347979, @Hahnfeld wrote:

In D89974#2347938, @tra wrote:

One concern I have is that the path we configure during clang's build is not necessarily the right choice for the user of clang we build. It's likely that the clang in the end will be used on a completely different machine.
E.g. official clang builds can not ever provide the same CUDA path for *all* users who end up using them. Requiring the rest to use a special option to make clang work again looks like an overall usability regression to me.

Not quite, as far as I understand the change it's only one of the searched paths.

Right. My mistake. I misread the code and the excess of --cuda-path-ignore-cmake in the tests convinced me that it replaced the default search paths.

The point about the path unlikely to be a good default guess still stands. It will also introduce surprising behavior when clang built on a machine with CUDA installed will behave differently compared to clang built on a machine w/o CUDA. IMO clang built from the same sources with the same build configuration options should behave the same way, regardless of what's been installed on the build host during the build.

[Meinersbur]

Re-add accidentally removed space

[tambre]

I echo @tra's concerns.
Having an option for a vendor to append/prepend a toolkit search location seems useful, but currently this seems more like a workaround for an explicit CUDA toolkit path not being passed to the testsuite.
Shortcomings in the testrunner don't seem like a reason to introduce new build-time configured default search paths into the driver.

Aside, here's an overview of CMake+CUDA:

• FindCUDA is deprecated since 3.10 and obsolete since 3.17. Replaced by native language support and FindCUDAToolkit.
• CMake supports Clang as a CUDA compiler since 3.18.
• CMake supports separable compilation with Clang since 3.19.
• CMake doesn't support CUDA with Clang on Windows. I hope to work on it for 3.20.

Ideally LLVM would use the native CMake CUDA support and the testsuite would use the CUDA toolkit found by CMake where necessary.
Users would be able to override the selected toolkit by setting CUDAToolkit_ROOT when configuring Clang.
Unfortunately this would require CMake 3.18, so this'll have to wait a bit.

If there are any questions/issues on the CMake side let me know. I maintain Clang and CUDA support in CMake.

[Meinersbur]

The host's build environment already influences the compiler's defaults, e.g. D88929. Especially libomptarget heavily uses it, and I would argue that these should be consistent, unless explicitly overwritten.

However, I see that OpenMP offloading is not the only use case.

[jdoerfert]

I think the default should still let clang search for CUDA or require the user to provide correct CUDA path. "Use CUDA path discovered by CMake at build time" should be a non-default configuration option if/when it's needed and appropriate.

Having an option for a vendor to append/prepend a toolkit search location seems useful, but currently this seems more like a workaround for an explicit CUDA toolkit path not being passed to the testsuite.
Shortcomings in the testrunner don't seem like a reason to introduce new build-time configured default search paths into the driver.

I thought, right now we would configure clang with a cuda path XYZ and the clang we build still will lookup the default path instead. This requires you to specify XYZ when compiling/linking as well in order to not accidentally pick up a different cuda installation (or none at all). Maybe I misunderstood but if I didn't, the behavior we have right now seems wrong or at least incomplete. I should be able to build a clang on a system with a non-default cuda installation that will work "out-of-the-box" for users as it has the cuda path "backed-in". Again, I might misunderstand all this.

[Hahnfeld]

In D89974#2348124, @jdoerfert wrote:

I thought, right now we would configure clang with a cuda path XYZ

If you speak of the current state in the repository, AFAICT, Clang doesn't know about a CUDA installation that CMake may have found. There's just a list of default paths that are searched during runtime, dynamically enhanced by an entry from searching ptxas in PATH.

[tra]

Shortcomings in the testrunner don't seem like a reason to introduce new build-time configured default search paths into the driver.

+1

The root of the problem is that we rely on CUDA SDK as the external dependency and we have no way to reliably predict where the user will put it.
Generally speaking, the location of that dependency is specific to particular machine. I.e. I can have CUDA installed pretty much anywhere I want. Different distributions may install it anywhere they want. There may be multiple versions installed and clang has no basis to guess which one the user intends to use.

I do not see a way to make it work reliably without explicit input from the end user. Making the heuristic dependent on clang's build environment is not improving the situation, IMO. It may help some users, but it will disrupt others and it will introduce more confusion about where specific clang binary searches for CUDA installations.

CUDA path is sort of a global configuration parameter for all CUDA compilations. Perhaps we should consider allowing the user to specify a CUDA search path candidate via environment variable. This should allow transparently overriding preferred CUDA path without having to adjust all builds. I can't say I like it, but it seems to be the least bad way (that I can think of ATM) to address the dependency on something that only the end user would know for sure.

[Hahnfeld]

In D89974#2348176, @tra wrote:

CUDA path is sort of a global configuration parameter for all CUDA compilations. Perhaps we should consider allowing the user to specify a CUDA search path candidate via environment variable. This should allow transparently overriding preferred CUDA path without having to adjust all builds. I can't say I like it, but it seems to be the least bad way (that I can think of ATM) to address the dependency on something that only the end user would know for sure.

Don't we already have this via PATH? At least that was my motivation back then and it worked without problems.

[tra]

In D89974#2348181, @Hahnfeld wrote:

Don't we already have this via PATH? At least that was my motivation back then and it worked without problems.

We do and it should indeed work for the tests. Setting an environment variable should be reasonably low burden if/when default path is not good enough.

PATH does suffer from the problem that ptxas location is not necessarily the right CUDA location. I.e. sometimes it's found via /usr/bin which I can not drop from the PATH, so one should be careful to *prepend* desired ptxas location.

[rnk]

My opinion carries less weight since I don't use CUDA, but I agree with everything Art said. Here's some input, if it helps.

I like the PATH search for ptxas as a way to make things work out of the box as often as possible.

I don't like the idea of CMake auto-detecting CUDA for the reasons Art listed. It's a great way to implicitly leak more details about the build environment to the user.

An explicit CMake option to override the CUDA installation path seems fine. A distribution could use this if they vend the CUDA SDK somewhere standard, for example.

Lastly, if those things don't work, there is the CCC_OVERRIDE_OPTIONS environment variable, so the user always has that escape hatch. A separate environment variable would also be fine. It reminds me of PYTHON_HOME, with similar drawbacks.

[tra]

In D89974#2348310, @rnk wrote:

Lastly, if those things don't work, there is the CCC_OVERRIDE_OPTIONS environment variable, so the user always has that escape hatch. A separate environment variable would also be fine. It reminds me of PYTHON_HOME, with similar drawbacks.

Oh. I didn't know about CCC_OVERRIDE_OPTIONS. It appears that does a bit more that just adding extra options. Is it documented somewhere?

Also, clang appears to use SDKROOT environment for specifying XCode path on MacOs, presumably for the reasons similar to what we're dealing with here with CUDA.

[Meinersbur]

Abandoned in favor of D101266.

clang/CMakeLists.txt

 
 # OpenMP offloading requires at least sm_35 because we use shuffle instructions
 # to generate efficient code for reductions and the atomicMax instruction on
 # 64-bit integers in the implementation of conditional lastprivate.
 set(CUDA_ARCH_FLAGS "sm_35")
 
 # Try to find the highest Nvidia GPU architecture the system supports
 if (NOT DEFINED CLANG_OPENMP_NVPTX_DEFAULT_ARCH)
-  find_package(CUDA QUIET)
+  # How CUDA is detected should be consistent with libomptarget.
+  #
+  # /usr/lib/cuda is a special case for Debian/Ubuntu to have nvidia-cuda-toolkit
+  # work out of the box. More info on http://bugs.debian.org/882505
+  find_package(CUDA QUIET PATHS "/usr/lib/cuda")
   if (CUDA_FOUND)
     cuda_select_nvcc_arch_flags(CUDA_ARCH_FLAGS)
   endif()
 else()
   set(CUDA_ARCH_FLAGS ${CLANG_OPENMP_NVPTX_DEFAULT_ARCH})
 endif()
 
 string(REGEX MATCH "sm_([0-9]+)" CUDA_ARCH_MATCH ${CUDA_ARCH_FLAGS})

clang/include/clang/Config/config.h.cmake

 #cmakedefine CLANG_CONFIG_FILE_USER_DIR "${CLANG_CONFIG_FILE_USER_DIR}"
 
 /* Default <path> to all compiler invocations for --sysroot=<path>. */
 #define DEFAULT_SYSROOT "${DEFAULT_SYSROOT}"
 
 /* Directory where gcc is installed. */
 #define GCC_INSTALL_PREFIX "${GCC_INSTALL_PREFIX}"
 
+/* Directory where CUDA is installed. */
+#define CLANG_CUDA_TOOLKIT_ROOT_DIR "${CUDA_TOOLKIT_ROOT_DIR}"
+
 /* Define if we have libxml2 */
 #cmakedefine CLANG_HAVE_LIBXML ${CLANG_HAVE_LIBXML}
 
 /* Define if we have sys/resource.h (rlimits) */
 #cmakedefine CLANG_HAVE_RLIMITS ${CLANG_HAVE_RLIMITS}
 
 /* The LLVM product name and version */
 #define BACKEND_PACKAGE_STRING "${BACKEND_PACKAGE_STRING}"

clang/include/clang/Driver/Options.td

 def cuda_noopt_device_debug : Flag<["--"], "cuda-noopt-device-debug">,
   HelpText<"Enable device-side debug info generation. Disables ptxas optimizations.">;
 def no_cuda_version_check : Flag<["--"], "no-cuda-version-check">,
   HelpText<"Don't error out if the detected version of the CUDA install is "
            "too low for the requested CUDA gpu architecture.">;
 def no_cuda_noopt_device_debug : Flag<["--"], "no-cuda-noopt-device-debug">;
 def cuda_path_EQ : Joined<["--"], "cuda-path=">, Group<i_Group>,
   HelpText<"CUDA installation path">;
+def cuda_path_ignore_cmake : Flag<["--"], "cuda-path-ignore-cmake">, Group<i_Group>,
+  HelpText<"Ignore clang's build-time introspection to detect CUDA installation">;
 def cuda_path_ignore_env : Flag<["--"], "cuda-path-ignore-env">, Group<i_Group>,
   HelpText<"Ignore environment variables to detect CUDA installation">;
 def ptxas_path_EQ : Joined<["--"], "ptxas-path=">, Group<i_Group>,
   HelpText<"Path to ptxas (used for compiling CUDA code)">;
 def fcuda_flush_denormals_to_zero : Flag<["-"], "fcuda-flush-denormals-to-zero">,
   HelpText<"Flush denormal floating point values to zero in CUDA device mode.">;
 def fno_cuda_flush_denormals_to_zero : Flag<["-"], "fno-cuda-flush-denormals-to-zero">;
 defm cuda_approx_transcendentals : OptInFFlag<"cuda-approx-transcendentals", "Use", "Don't use",

clang/lib/Driver/ToolChains/Cuda.cpp

CudaInstallationDetector::CudaInstallationDetector(
 
   // In decreasing order so we prefer newer versions to older versions.
   std::initializer_list<const char *> Versions = {"8.0", "7.5", "7.0"};
   auto &FS = D.getVFS();
 
   if (Args.hasArg(clang::driver::options::OPT_cuda_path_EQ)) {
     Candidates.emplace_back(
         Args.getLastArgValue(clang::driver::options::OPT_cuda_path_EQ).str());
-  } else if (HostTriple.isOSWindows()) {
+  } else {
+    // Use the CUDA installation detected by CMake as the first candidate. This
+    // also the one used to compile libomptarget's CUDA plugin and its
+    // regression tests.
+    if (!Args.hasArg(clang::driver::options::OPT_cuda_path_ignore_cmake)) {
+      std::string CMakeCudaPath = CLANG_CUDA_TOOLKIT_ROOT_DIR;
+      if (!CMakeCudaPath.empty())
+        Candidates.emplace_back(CMakeCudaPath);
+    }
+
+    if (HostTriple.isOSWindows()) {
       for (const char *Ver : Versions)
         Candidates.emplace_back(
             D.SysRoot + "/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v" +
             Ver);
     } else {
       if (!Args.hasArg(clang::driver::options::OPT_cuda_path_ignore_env)) {
         // Try to find ptxas binary. If the executable is located in a directory
         // called 'bin/', its parent directory might be a good guess for a valid
         // CUDA installation.
-      // However, some distributions might installs 'ptxas' to /usr/bin. In that
-      // case the candidate would be '/usr' which passes the following checks
-      // because '/usr/include' exists as well. To avoid this case, we always
-      // check for the directory potentially containing files for libdevice,
-      // even if the user passes -nocudalib.
+        // However, some distributions might installs 'ptxas' to /usr/bin. In
+        // that case the candidate would be '/usr' which passes the following
+        // checks because '/usr/include' exists as well. To avoid this case, we
+        // always check for the directory potentially containing files for
+        // libdevice, even if the user passes -nocudalib.
         if (llvm::ErrorOr<std::string> ptxas =
                 llvm::sys::findProgramByName("ptxas")) {
           SmallString<256> ptxasAbsolutePath;
           llvm::sys::fs::real_path(*ptxas, ptxasAbsolutePath);
 
           StringRef ptxasDir = llvm::sys::path::parent_path(ptxasAbsolutePath);
           if (llvm::sys::path::filename(ptxasDir) == "bin")
             Candidates.emplace_back(
                 std::string(llvm::sys::path::parent_path(ptxasDir)),
                 /*StrictChecking=*/true);
         }
       }
 
       Candidates.emplace_back(D.SysRoot + "/usr/local/cuda");
       for (const char *Ver : Versions)
         Candidates.emplace_back(D.SysRoot + "/usr/local/cuda-" + Ver);
 
       Distro Dist(FS, llvm::Triple(llvm::sys::getProcessTriple()));
       if (Dist.IsDebian() || Dist.IsUbuntu())
         // Special case for Debian to have nvidia-cuda-toolkit work
         // out of the box. More info on http://bugs.debian.org/882505
         Candidates.emplace_back(D.SysRoot + "/usr/lib/cuda");
     }
+  }
 
   bool NoCudaLib = Args.hasArg(options::OPT_nogpulib);
 
   for (const auto &Candidate : Candidates) {
     InstallPath = Candidate.Path;
     if (InstallPath.empty() || !FS.exists(InstallPath))
       continue;
 

clang/test/Driver/cuda-detect-path.cu

 // REQUIRES: clang-driver
 // REQUIRES: x86-registered-target
 // REQUIRES: nvptx-registered-target
 // This tests uses the PATH environment variable.
 // REQUIRES: !system-windows
 
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s
 
 
 // Check that we follow ptxas binaries that are symlinks.
 // RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
-// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
 // RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
-// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
 // RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
-// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
 // RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
-// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
 
 
 // We only take a CUDA installation from PATH if it contains libdevice.
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 
 // We even require libdevice if -nocudalib is passed to avoid false positives
 // if the distribution merges CUDA into /usr and ptxas ends up /usr/bin.
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake -nocudalib \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake -nocudalib \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake -nocudalib \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake -nocudalib \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 
 
 // Check that the CUDA installation in PATH is not taken when passing
 // the option --cuda-path-ignore-env.
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake --cuda-path-ignore-env \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake --cuda-path-ignore-env \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake --cuda-path-ignore-env \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 // RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
-// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-cmake --cuda-path-ignore-env \
 // RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
 
 // CHECK: Found CUDA installation: {{.*}}/Inputs/CUDA/usr/local/cuda
 // SYMLINKS: Found CUDA installation: {{.*}}/Inputs/CUDA-symlinks/opt/cuda
 // NOCUDA-NOT: Found CUDA installation:

clang/test/Driver/cuda-detect.cu

 // REQUIRES: clang-driver
 // REQUIRES: x86-registered-target
 // REQUIRES: nvptx-registered-target
 //
 // Check that we properly detect CUDA installation.
-// RUN: %clang -v --target=i386-unknown-linux \
+// RUN: %clang -v --target=i386-unknown-linux --cuda-path-ignore-cmake  \
 // RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
-// RUN: %clang -v --target=i386-apple-macosx \
+// RUN: %clang -v --target=i386-apple-macosx --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
-// RUN: %clang -v --target=x86_64-unknown-linux \
+// RUN: %clang -v --target=x86_64-unknown-linux --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
-// RUN: %clang -v --target=x86_64-apple-macosx \
+// RUN: %clang -v --target=x86_64-apple-macosx --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
 
-
-// RUN: %clang -v --target=i386-unknown-linux \
+// RUN: %clang -v --target=i386-unknown-linux --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA --cuda-path-ignore-env 2>&1 | FileCheck %s
-// RUN: %clang -v --target=i386-apple-macosx \
+// RUN: %clang -v --target=i386-apple-macosx --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA --cuda-path-ignore-env 2>&1 | FileCheck %s
 
 // RUN: %clang -v --target=i386-unknown-linux \
 // RUN:   --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 | FileCheck %s
 // RUN: %clang -v --target=i386-apple-macosx \
 // RUN:   --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 | FileCheck %s
 
 // Check that we don't find a CUDA installation without libdevice ...
-// RUN: %clang -v --target=i386-unknown-linux \
+// RUN: %clang -v --target=i386-unknown-linux --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
-// RUN: %clang -v --target=i386-apple-macosx \
+// RUN: %clang -v --target=i386-apple-macosx --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
-// RUN: %clang -v --target=x86_64-unknown-linux \
+// RUN: %clang -v --target=x86_64-unknown-linux --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
-// RUN: %clang -v --target=x84_64-apple-macosx \
+// RUN: %clang -v --target=x84_64-apple-macosx --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
 
 // ... unless the user doesn't need libdevice
-// RUN: %clang -v --target=i386-unknown-linux -nocudalib \
+// RUN: %clang -v --target=i386-unknown-linux -nocudalib --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
-// RUN: %clang -v --target=i386-apple-macosx -nocudalib \
+// RUN: %clang -v --target=i386-apple-macosx -nocudalib --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
-// RUN: %clang -v --target=x86_64-unknown-linux -nocudalib \
+// RUN: %clang -v --target=x86_64-unknown-linux -nocudalib --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
-// RUN: %clang -v --target=x86_64-apple-macosx -nocudalib \
+// RUN: %clang -v --target=x86_64-apple-macosx -nocudalib --cuda-path-ignore-cmake \
 // RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
 
 
 // Make sure we map libdevice bitcode files to proper GPUs. These
 // tests use Inputs/CUDA_80 which has full set of libdevice files.
 // However, libdevice mapping only matches CUDA-7.x at the moment.
 // sm_2x, sm_32 -> compute_20
 // RUN: %clang -### -v --target=i386-unknown-linux --cuda-gpu-arch=sm_21 \

openmp/libomptarget/cmake/Modules/LibomptargetGetDependencies.cmake

 
 mark_as_advanced(
   LIBOMPTARGET_DEP_LIBFFI_INCLUDE_DIRS 
   LIBOMPTARGET_DEP_LIBFFI_LIBRARIES)
   
 ################################################################################
 # Looking for CUDA...
 ################################################################################
-if (CUDA_TOOLKIT_ROOT_DIR)
-  set(LIBOMPTARGET_CUDA_TOOLKIT_ROOT_DIR_PRESET TRUE)
-endif()
-find_package(CUDA QUIET)
+
+# How CUDA is detected should be consistent with Clang.
+#
+# /usr/lib/cuda is a special case for Debian/Ubuntu to have nvidia-cuda-toolkit
+# work out of the box. More info on http://bugs.debian.org/882505
+find_package(CUDA QUIET PATHS "/usr/lib/cuda")
 
 # Try to get the highest Nvidia GPU architecture the system supports
 if (CUDA_FOUND)
   cuda_select_nvcc_arch_flags(CUDA_ARCH_FLAGS)
   string(REGEX MATCH "sm_([0-9]+)" CUDA_ARCH_MATCH_OUTPUT ${CUDA_ARCH_FLAGS})
   if (NOT DEFINED CUDA_ARCH_MATCH_OUTPUT OR "${CMAKE_MATCH_1}" LESS 35)
     libomptarget_warning_say("Setting Nvidia GPU architecture support for OpenMP target runtime library to sm_35 by default")
     set(LIBOMPTARGET_DEP_CUDA_ARCH "35")
find_package_handle_standard_args(
   LIBOMPTARGET_DEP_VEO_LIBRARIES
   LIBOMPTARGET_DEP_VEOSINFO_LIBRARIES
   LIBOMPTARGET_DEP_VEO_INCLUDE_DIRS)
 
 mark_as_advanced(
   LIBOMPTARGET_DEP_VEO_FOUND
   LIBOMPTARGET_DEP_VEO_INCLUDE_DIRS)
 
-# Looking for CUDA libdevice subdirectory
-#
-# Special case for Debian/Ubuntu to have nvidia-cuda-toolkit work
-# out of the box. More info on http://bugs.debian.org/882505
-################################################################################
-
-set(LIBOMPTARGET_CUDA_LIBDEVICE_SUBDIR nvvm/libdevice)
-
-# Don't alter CUDA_TOOLKIT_ROOT_DIR if the user specified it, if a value was
-# already cached for it, or if it already has libdevice.  Otherwise, on
-# Debian/Ubuntu, look where the nvidia-cuda-toolkit package normally installs
-# libdevice.
-if (NOT LIBOMPTARGET_CUDA_TOOLKIT_ROOT_DIR_PRESET AND
-    NOT EXISTS
-      "${CUDA_TOOLKIT_ROOT_DIR}/${LIBOMPTARGET_CUDA_LIBDEVICE_SUBDIR}")
-  find_program(LSB_RELEASE lsb_release)
-  if (LSB_RELEASE)
-    execute_process(COMMAND ${LSB_RELEASE} -is
-      OUTPUT_VARIABLE LSB_RELEASE_ID
-      OUTPUT_STRIP_TRAILING_WHITESPACE)
-    set(candidate_dir /usr/lib/cuda)
-    if ((LSB_RELEASE_ID STREQUAL "Debian" OR LSB_RELEASE_ID STREQUAL "Ubuntu")
-        AND EXISTS "${candidate_dir}/${LIBOMPTARGET_CUDA_LIBDEVICE_SUBDIR}")
-      set(CUDA_TOOLKIT_ROOT_DIR "${candidate_dir}" CACHE PATH
-          "Toolkit location." FORCE)
-    endif()
-  endif()
-endif()