[CUDA] Detect installation in PATH

[Hahnfeld]

Summary

If the CUDA toolkit is not installed to its default locations
in /usr/local/cuda, the user is forced to specify --cuda-path.
This is tedious and the driver can be smarter if well-known tools
(like ptxas) can already be found in the PATH environment variable.

Add option --cuda-path-ignore-env if the user wants to ignore
set environment variables. Also use it in the tests to make sure
the driver always finds the same CUDA installation, regardless
of the user's environment.

[tra]

Some linux distributions integrate CUDA into the standard directory structure. I.e. binaries go into /usr/bin, headers into /usr/include, bitcode goes somewhere else, etc. ptxas will be found, but we would still fail to detect CUDA. I'd add one more test case to make sure that's still the case.

Otherwise - LGTM.

[jlebar]

Can we document this behavior in https://llvm.org/docs/CompileCudaWithLLVM.html (in the LLVM repo)? Totally fine if you want to do this in a separate patch.

[Hahnfeld]

In D42642#990976, @tra wrote:

Some linux distributions integrate CUDA into the standard directory structure. I.e. binaries go into /usr/bin, headers into /usr/include, bitcode goes somewhere else, etc. ptxas will be found, but we would still fail to detect CUDA. I'd add one more test case to make sure that's still the case.

I'm not sure how this can work, we only require bin/ and include/ to exist, and nvvm/libdevice/ if -nocudalib isn't specified. I agree this can be a problem because the defaults might detect an invalid installation...

In D42642#991013, @jlebar wrote:

Can we document this behavior in https://llvm.org/docs/CompileCudaWithLLVM.html (in the LLVM repo)? Totally fine if you want to do this in a separate patch.

Sure, will do!

[tra]

In D42642#991127, @Hahnfeld wrote:

In D42642#990976, @tra wrote:

Some linux distributions integrate CUDA into the standard directory structure. I.e. binaries go into /usr/bin, headers into /usr/include, bitcode goes somewhere else, etc. ptxas will be found, but we would still fail to detect CUDA. I'd add one more test case to make sure that's still the case.

I'm not sure how this can work, we only require bin/ and include/ to exist, and nvvm/libdevice/ if -nocudalib isn't specified. I agree this can be a problem because the defaults might detect an invalid installation...

Good point. Perhaps we want to be more strict about CUDA installation checking if we've found it indirectly via PATH (as opposed to explicit --cuda-path or a known common install path).
Should we always check for nvvm/libdevice directory? It's unlikely to be under /usr or /usr/local and it will be always present in a CUDA installation of all currently supported CUDA versions.

[Hahnfeld]

Check for libdevice in candidates from PATH.

[Hahnfeld]

In D42642#991154, @tra wrote:

In D42642#991127, @Hahnfeld wrote:

In D42642#990976, @tra wrote:

Some linux distributions integrate CUDA into the standard directory structure. I.e. binaries go into /usr/bin, headers into /usr/include, bitcode goes somewhere else, etc. ptxas will be found, but we would still fail to detect CUDA. I'd add one more test case to make sure that's still the case.

I'm not sure how this can work, we only require bin/ and include/ to exist, and nvvm/libdevice/ if -nocudalib isn't specified. I agree this can be a problem because the defaults might detect an invalid installation...

Good point. Perhaps we want to be more strict about CUDA installation checking if we've found it indirectly via PATH (as opposed to explicit --cuda-path or a known common install path).
Should we always check for nvvm/libdevice directory? It's unlikely to be under /usr or /usr/local and it will be always present in a CUDA installation of all currently supported CUDA versions.

That sounds like a sane check. I updated the implementation and added corresponding checks.

[tra]

lib/Driver/ToolChains/Cuda.cpp

218

I think this may be too strict.

Checking directory structure for the purposes of detecting CUDA SDK should work well enough to weed out false detection for 'split' CUDA installation and we've verified libdevice directory presence above.

Checking for libdevice bitcode is somewhat orthogonal to this. IMO, regardless of how we've found the installation directory, whether we have suitable libdevice version there should not matter if user explicitly passed -nocudalib. Insisting on libdevice presence in this situation would be somewhat surprising.

[Hahnfeld]

lib/Driver/ToolChains/Cuda.cpp

218

So you are suggesting to revert the change to this line, right?

[tra]

I've thought a bit more about this and there's another quirk -- symlinks.

What if we've found /usr/bin/ptxas and is a symlink pointing to the real ptxas in the CUDA installation? If we add /usr to the list of candidates it will not help us at all. We should probably find the real path and add another candidate path derived from it.

lib/Driver/ToolChains/Cuda.cpp

218

Yes, if you agree with my reasoning.

[Hahnfeld]

In D42642#992137, @tra wrote:

I've thought a bit more about this and there's another quirk -- symlinks.

What if we've found /usr/bin/ptxas and is a symlink pointing to the real ptxas in the CUDA installation? If we add /usr to the list of candidates it will not help us at all. We should probably find the real path and add another candidate path derived from it.

Yes, this might be a valid use case, for example linking /usr/bin/ptxas to /opt/cuda/bin/ptxas or something. I think there should be an LLVM function to get the realpath, let me check that.
(In fact I have symlinks in my environment but it doesn't matter in this case because we are just linking the complete CUDA installation to another path...)

lib/Driver/ToolChains/Cuda.cpp

218

I generally agree that the working bitcode files are orthogonal to the directory structure. However I would argue that there won't be a real world scenario of an empty directory called nvvm/libdevice without any libdevice files in there. That said it probably doesn't make a difference, so I'll follow the separation of concerns.

[Hahnfeld]

Follow symlinked ptxas executables.

[tra]

lib/Driver/ToolChains/Cuda.cpp

96–105

Another corner case:
Debian scatters CUDA install all over the filesystem. To make it work with clang it has a 'shim' package which re-creates complete CUDA install using symlinks to its scattered bits. https://bugs.llvm.org/show_bug.cgi?id=35249. If PATH includes such a shim with a symlink pointing to location somewhere else in the filesystem, this variant of the patch will not work.

I'd add another candidate derived from the path returned by find. This should cover all reasonable scenarios I can think of.

Caveat: clang on Debian already has a special case to add this shim to the list of candidates ( D40453 ), so this patch should not affect it. Still, it's possible for the similar case to happen somewhere else where we do not have any explicit workarounds in clang.

BTW, should this heuristic apply on Windows, too? IIRC cuda installer does add CUDA's bin dir to PATH.

[Hahnfeld]

lib/Driver/ToolChains/Cuda.cpp

96–105

I'd rather not complicate this further. I asked on D40453 and the reply was

In D40453#936114, @sylvestre.ledru wrote:

Debian packages don't update the PATH and we are aiming at providing packages working out of the box.

So IMO if distributions opt to fully diverge from CUDA's standard directory layout they should add special cases here to make it work.

I'm not really sure regarding Windows:

1. From what I recall the PATH variable was somewhat dubious under Windows...
2. Does the installer actually allow to customize the installation path? If not, how likely is it that users move the directory manually?

I can't test on Windows, so I'd generally prefer if you adjust the code and add tests in a separate patch...

[tra]

LGTM.

lib/Driver/ToolChains/Cuda.cpp

96–105

I don't have windows either.
OK, we can handle oddball CUDA installations and exotic platforms until someone wants it.

[Hahnfeld]

Add some sysroot arguments to new test to make sure it doesn't accidentally find CUDA installations in the system.

[aganea]

Just a quick heads-up - the cuda-detect-path.cu test fails on WSL/Ubuntu 18.04:

aganea@MTL-BJ842:/mnt/f/svn/buildWSL$ /usr/bin/python3.6 bin/llvm-lit -vv ../clang/test/Driver/cuda-detect-path.cu
llvm-lit: /mnt/f/svn/llvm/utils/lit/lit/llvm/config.py:341: note: using clang: /mnt/f/svn/buildWSL/bin/clang
-- Testing: 1 tests, single process --
FAIL: Clang :: Driver/cuda-detect-path.cu (1 of 1)
******************** TEST 'Clang :: Driver/cuda-detect-path.cu' FAILED ********************

(output chopped)

+ env PATH=/mnt/f/svn/clang/test/Driver/Inputs/CUDA-symlinks/usr/bin /mnt/f/svn/buildWSL/bin/clang -v --target=i386-unknown-linux --sysroot=/mnt/f/svn/clang/test/Driver/no-cuda-there
+ /mnt/f/svn/buildWSL/bin/FileCheck /mnt/f/svn/clang/test/Driver/cuda-detect-path.cu --check-prefix SYMLINKS
/mnt/f/svn/clang/test/Driver/cuda-detect-path.cu:82:14: error: SYMLINKS: expected string not found in input
// SYMLINKS: Found CUDA installation: {{.*}}/Inputs/CUDA-symlinks/opt/cuda
             ^
<stdin>:1:1: note: scanning from here
clang version 9.0.0 (trunk)
^
<stdin>:4:1: note: possible intended match here
InstalledDir: /mnt/f/svn/buildWSL/bin
^

aganea@MTL-BJ842:/mnt/f/svn/buildWSL$ env PATH=/mnt/f/svn/clang/test/Driver/Inputs/CUDA-symlinks/usr/bin /mnt/f/svn/buildWSL/bin/clang -v --target=i386-unknown-linux --sysroot=/mnt/f/svn/clang/test/Driver/no-cuda-there
clang version 9.0.0 (trunk)
Target: i386-unknown-linux
Thread model: posix
InstalledDir: /mnt/f/svn/buildWSL/bin
aganea@MTL-BJ842:/mnt/f/svn/buildWSL$ ls -l /mnt/f/svn/clang/test/Driver/Inputs/CUDA-symlinks/usr/bin
total 0
-rwxrwxrwx 1 aganea aganea 29 Feb  1  2018 ptxas

[Hahnfeld]

In D42642#1479013, @aganea wrote:

Just a quick heads-up - the cuda-detect-path.cu test fails on WSL/Ubuntu 18.04:

Hmm, maybe that's because of symlinks. By chance, can you debug print the values of real_path and parent_path? They are supposed to resolve the links which is required for the candidate search to work and the test to pass.

cfe/trunk/lib/Driver/ToolChains/Cuda.cpp

98–101 ↗

(On Diff #132104)

These calls.

[aganea]

So it turns out this is a symlink issue. The file clang/trunk/test/Driver/Inputs/CUDA-symlinks/usr/bin/ptxas has been synchronized on my Windows 10 PC as a regular text file, not a symlink. It looks like TortoiseSVN doesn't implement symlinks. As WSL inherits of my file system, it will not find the symbolic link. I suppose REQUIRES: !system-windows isn't enough for cuda-detect-path.cu, and it would need something like REQUIRES: symlinks along with support in lit. @rnk

include/clang/Driver/Options.td

 def cuda_noopt_device_debug : Flag<["--"], "cuda-noopt-device-debug">,
   HelpText<"Enable device-side debug info generation. Disables ptxas optimizations.">;
 def no_cuda_version_check : Flag<["--"], "no-cuda-version-check">,
   HelpText<"Don't error out if the detected version of the CUDA install is "
            "too low for the requested CUDA gpu architecture.">;
 def no_cuda_noopt_device_debug : Flag<["--"], "no-cuda-noopt-device-debug">;
 def cuda_path_EQ : Joined<["--"], "cuda-path=">, Group<i_Group>,
   HelpText<"CUDA installation path">;
+def cuda_path_ignore_env : Joined<["--"], "cuda-path-ignore-env">, Group<i_Group>,
+  HelpText<"Ignore environment variables to detect CUDA installation">;
 def ptxas_path_EQ : Joined<["--"], "ptxas-path=">, Group<i_Group>,
   HelpText<"Path to ptxas (used for compiling CUDA code)">;
 def fcuda_flush_denormals_to_zero : Flag<["-"], "fcuda-flush-denormals-to-zero">,
   Flags<[CC1Option]>, HelpText<"Flush denormal floating point values to zero in CUDA device mode.">;
 def fno_cuda_flush_denormals_to_zero : Flag<["-"], "fno-cuda-flush-denormals-to-zero">;
 def fcuda_approx_transcendentals : Flag<["-"], "fcuda-approx-transcendentals">,
   Flags<[CC1Option]>, HelpText<"Use approximate transcendental functions">;
 def fno_cuda_approx_transcendentals : Flag<["-"], "fno-cuda-approx-transcendentals">;

lib/Driver/ToolChains/Cuda.cpp

 //===--- Cuda.cpp - Cuda Tool and ToolChain Implementations -----*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "Cuda.h"
-#include "InputInfo.h"
 #include "CommonArgs.h"
+#include "InputInfo.h"
 #include "clang/Basic/Cuda.h"
-#include "clang/Config/config.h"
 #include "clang/Basic/VirtualFileSystem.h"
-#include "clang/Driver/Distro.h"
+#include "clang/Config/config.h"
 #include "clang/Driver/Compilation.h"
+#include "clang/Driver/Distro.h"
 #include "clang/Driver/Driver.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/Options.h"
 #include "llvm/Option/ArgList.h"
+#include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
 #include <system_error>
 
 using namespace clang::driver;
 using namespace clang::driver::toolchains;
 using namespace clang::driver::tools;
 using namespace clang;
 using namespace llvm::opt;
 
if (Major == 9 && Minor == 1)
     return CudaVersion::CUDA_91;
   return CudaVersion::UNKNOWN;
 }
 
 CudaInstallationDetector::CudaInstallationDetector(
     const Driver &D, const llvm::Triple &HostTriple,
     const llvm::opt::ArgList &Args)
     : D(D) {
-  SmallVector<std::string, 4> CudaPathCandidates;
+  struct Candidate {
+    std::string Path;
+    bool StrictChecking;
+
+    Candidate(std::string Path, bool StrictChecking = false)
+        : Path(Path), StrictChecking(StrictChecking) {}
+  };
+  SmallVector<Candidate, 4> Candidates;
 
   // In decreasing order so we prefer newer versions to older versions.
   std::initializer_list<const char *> Versions = {"8.0", "7.5", "7.0"};
 
   if (Args.hasArg(clang::driver::options::OPT_cuda_path_EQ)) {
-    CudaPathCandidates.push_back(
-        Args.getLastArgValue(clang::driver::options::OPT_cuda_path_EQ));
+    Candidates.emplace_back(
+        Args.getLastArgValue(clang::driver::options::OPT_cuda_path_EQ).str());
   } else if (HostTriple.isOSWindows()) {
     for (const char *Ver : Versions)
-      CudaPathCandidates.push_back(
+      Candidates.emplace_back(
           D.SysRoot + "/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v" +
           Ver);
   } else {
-    CudaPathCandidates.push_back(D.SysRoot + "/usr/local/cuda");
+    if (!Args.hasArg(clang::driver::options::OPT_cuda_path_ignore_env)) {
+      // Try to find ptxas binary. If the executable is located in a directory
+      // called 'bin/', its parent directory might be a good guess for a valid
+      // CUDA installation.
+      // However, some distributions might installs 'ptxas' to /usr/bin. In that
+      // case the candidate would be '/usr' which passes the following checks
+      // because '/usr/include' exists as well. To avoid this case, we always
+      // check for the directory potentially containing files for libdevice,
+      // even if the user passes -nocudalib.
+      if (llvm::ErrorOr<std::string> ptxas =
+              llvm::sys::findProgramByName("ptxas")) {
+        SmallString<256> ptxasAbsolutePath;
+        llvm::sys::fs::real_path(*ptxas, ptxasAbsolutePath);
+
+        StringRef ptxasDir = llvm::sys::path::parent_path(ptxasAbsolutePath);
+        if (llvm::sys::path::filename(ptxasDir) == "bin")
+          Candidates.emplace_back(llvm::sys::path::parent_path(ptxasDir),
+                                  /*StrictChecking=*/true);
+      }
+    }
+
+    Candidates.emplace_back(D.SysRoot + "/usr/local/cuda");
     for (const char *Ver : Versions)
-      CudaPathCandidates.push_back(D.SysRoot + "/usr/local/cuda-" + Ver);
+      Candidates.emplace_back(D.SysRoot + "/usr/local/cuda-" + Ver);
 
     if (Distro(D.getVFS()).IsDebian())
       // Special case for Debian to have nvidia-cuda-toolkit work
       // out of the box. More info on http://bugs.debian.org/882505
-      CudaPathCandidates.push_back(D.SysRoot + "/usr/lib/cuda");
+      Candidates.emplace_back(D.SysRoot + "/usr/lib/cuda");
   }
 
-  for (const auto &CudaPath : CudaPathCandidates) {
-    if (CudaPath.empty() || !D.getVFS().exists(CudaPath))
+  bool NoCudaLib = Args.hasArg(options::OPT_nocudalib);
+
+  for (const auto &Candidate : Candidates) {
+    InstallPath = Candidate.Path;
+    if (InstallPath.empty() || !D.getVFS().exists(InstallPath))
       continue;
 
-    InstallPath = CudaPath;
-    BinPath = CudaPath + "/bin";
+    BinPath = InstallPath + "/bin";
     IncludePath = InstallPath + "/include";
     LibDevicePath = InstallPath + "/nvvm/libdevice";
 
     auto &FS = D.getVFS();
     if (!(FS.exists(IncludePath) && FS.exists(BinPath)))
       continue;
+    bool CheckLibDevice = (!NoCudaLib || Candidate.StrictChecking);
+    if (CheckLibDevice && !FS.exists(LibDevicePath))
+      continue;
 
     // On Linux, we have both lib and lib64 directories, and we need to choose
     // based on our triple.  On MacOS, we have only a lib directory.
     //
     // It's sufficient for our purposes to be flexible: If both lib and lib64
     // exist, we choose whichever one matches our triple.  Otherwise, if only
     // lib exists, we use it.
     if (HostTriple.isArch64Bit() && FS.exists(InstallPath + "/lib64"))
if (Version >= CudaVersion::CUDA_90) {
             LibDeviceMap["sm_53"] = FilePath;
           }
         }
       }
     }
 
     // Check that we have found at least one libdevice that we can link in if
     // -nocudalib hasn't been specified.
-    if (LibDeviceMap.empty() && !Args.hasArg(options::OPT_nocudalib))
+    if (LibDeviceMap.empty() && !NoCudaLib)
       continue;
 
     IsValid = true;
     break;
   }
 }
 
 void CudaInstallationDetector::AddCudaIncludeArgs(

test/Driver/Inputs/CUDA-symlinks/usr/bin/ptxas

+../../opt/cuda/bin/ptxas
 No newline at end of file

test/Driver/cuda-detect-path.cu

+// REQUIRES: clang-driver
+// REQUIRES: x86-registered-target
+// REQUIRES: nvptx-registered-target
+// This tests uses the PATH environment variable.
+// REQUIRES: !system-windows
+
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s
+
+
+// Check that we follow ptxas binaries that are symlinks.
+// RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
+// RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
+// RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
+// RUN: env PATH=%S/Inputs/CUDA-symlinks/usr/bin \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix SYMLINKS
+
+
+// We only take a CUDA installation from PATH if it contains libdevice.
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+
+// We even require libdevice if -nocudalib is passed to avoid false positives
+// if the distribution merges CUDA into /usr and ptxas ends up /usr/bin.
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA-nolibdevice/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there -nocudalib \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+
+
+// Check that the CUDA installation in PATH is not taken when passing
+// the option --cuda-path-ignore-env.
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=i386-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-unknown-linux --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+// RUN: env PATH=%S/Inputs/CUDA/usr/local/cuda/bin \
+// RUN:    %clang -v --target=x86_64-apple-macosx --sysroot=%S/no-cuda-there --cuda-path-ignore-env \
+// RUN:    2>&1 | FileCheck %s --check-prefix NOCUDA
+
+// CHECK: Found CUDA installation: {{.*}}/Inputs/CUDA/usr/local/cuda
+// SYMLINKS: Found CUDA installation: {{.*}}/Inputs/CUDA-symlinks/opt/cuda
+// NOCUDA-NOT: Found CUDA installation:

test/Driver/cuda-detect.cu

 // REQUIRES: clang-driver
 // REQUIRES: x86-registered-target
 // REQUIRES: nvptx-registered-target
 //
 // Check that we properly detect CUDA installation.
 // RUN: %clang -v --target=i386-unknown-linux \
-// RUN:   --sysroot=%S/no-cuda-there 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
 // RUN: %clang -v --target=i386-apple-macosx \
-// RUN:   --sysroot=%S/no-cuda-there 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN: %clang -v --target=x86_64-unknown-linux \
+// RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN: %clang -v --target=x86_64-apple-macosx \
+// RUN:   --sysroot=%S/no-cuda-there --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
+
 
 // RUN: %clang -v --target=i386-unknown-linux \
 // RUN:   --sysroot=%S/Inputs/CUDA 2>&1 | FileCheck %s
 // RUN: %clang -v --target=i386-apple-macosx \
 // RUN:   --sysroot=%S/Inputs/CUDA 2>&1 | FileCheck %s
 
 // RUN: %clang -v --target=i386-unknown-linux \
 // RUN:   --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 | FileCheck %s
 // RUN: %clang -v --target=i386-apple-macosx \
 // RUN:   --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 | FileCheck %s
 
 // Check that we don't find a CUDA installation without libdevice ...
 // RUN: %clang -v --target=i386-unknown-linux \
-// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
 // RUN: %clang -v --target=i386-apple-macosx \
-// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN: %clang -v --target=x86_64-unknown-linux \
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
+// RUN: %clang -v --target=x84_64-apple-macosx \
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NOCUDA
 
 // ... unless the user doesn't need libdevice
 // RUN: %clang -v --target=i386-unknown-linux -nocudalib \
-// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
 // RUN: %clang -v --target=i386-apple-macosx -nocudalib \
-// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
+// RUN: %clang -v --target=x86_64-unknown-linux -nocudalib \
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
+// RUN: %clang -v --target=x86_64-apple-macosx -nocudalib \
+// RUN:   --sysroot=%S/Inputs/CUDA-nolibdevice --cuda-path-ignore-env 2>&1 | FileCheck %s -check-prefix NO-LIBDEVICE
 
 
 // Make sure we map libdevice bitcode files to proper GPUs. These
 // tests use Inputs/CUDA_80 which has full set of libdevice files.
 // However, libdevice mapping only matches CUDA-7.x at the moment.
 // sm_2x, sm_32 -> compute_20
 // RUN: %clang -### -v --target=i386-unknown-linux --cuda-gpu-arch=sm_21 \
 // RUN:   --cuda-path=%S/Inputs/CUDA_80/usr/local/cuda %s 2>&1 \

test/Driver/cuda-not-found.cu

 // REQUIRES: clang-driver
 
 // Check that we raise an error if we're trying to compile CUDA code but can't
 // find a CUDA install, unless -nocudainc was passed.
 
-// RUN: %clang -### --sysroot=%s/no-cuda-there %s 2>&1 | FileCheck %s --check-prefix ERR
+// RUN: %clang -### --sysroot=%s/no-cuda-there --cuda-path-ignore-env %s 2>&1 | FileCheck %s --check-prefix ERR
 // RUN: %clang -### --cuda-path=%s/no-cuda-there %s 2>&1 | FileCheck %s --check-prefix ERR
 // ERR: cannot find CUDA installation
 
-// RUN: %clang -### -nocudainc --sysroot=%s/no-cuda-there %s 2>&1 | FileCheck %s --check-prefix OK
+// RUN: %clang -### -nocudainc --sysroot=%s/no-cuda-there --cuda-path-ignore-env %s 2>&1 | FileCheck %s --check-prefix OK
 // RUN: %clang -### -nocudainc --cuda-path=%s/no-cuda-there %s 2>&1 | FileCheck %s --check-prefix OK
 // OK-NOT: cannot find CUDA installation

test/Driver/cuda-version-check.cu

 // REQUIRES: clang-driver
 // REQUIRES: x86-registered-target
 // REQUIRES: nvptx-registered-target
 
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_20 --sysroot=%S/Inputs/CUDA 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_20 --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_20 --sysroot=%S/Inputs/CUDA_80 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_20 --cuda-path=%S/Inputs/CUDA_80/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --sysroot=%S/Inputs/CUDA_80 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA_80/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
 
 // The installation at Inputs/CUDA is CUDA 7.0, which doesn't support sm_60.
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --sysroot=%S/Inputs/CUDA 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60
 
 // This should only complain about sm_60, not sm_35.
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-gpu-arch=sm_35 \
-// RUN:    --sysroot=%S/Inputs/CUDA 2>&1 %s | \
+// RUN:    --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60 --check-prefix=OK_SM35
 
 // We should get two errors here, one for sm_60 and one for sm_61.
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-gpu-arch=sm_61 \
-// RUN:    --sysroot=%S/Inputs/CUDA 2>&1 %s | \
+// RUN:    --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60 --check-prefix=ERR_SM61
 
 // We should still get an error if we pass -nocudainc, because this compilation
 // would invoke ptxas, and we do a version check on that, too.
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 -nocudainc --sysroot=%S/Inputs/CUDA 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 -nocudainc --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60
 
 // If with -nocudainc and -E, we don't touch the CUDA install, so we
 // shouldn't get an error.
 // RUN: %clang --target=x86_64-linux -v -### -E --cuda-device-only --cuda-gpu-arch=sm_60 -nocudainc \
-// RUN:    --sysroot=%S/Inputs/CUDA 2>&1 %s | \
+// RUN:    --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
 
 // --no-cuda-version-check should suppress all of these errors.
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --sysroot=%S/Inputs/CUDA 2>&1 \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 \
 // RUN:    --no-cuda-version-check %s | \
 // RUN:    FileCheck %s --check-prefix=OK
 
 // We need to make sure the version check is done only for the device toolchain,
 // therefore we should not get an error in host-only mode. We use the -S here
 // to avoid the error being produced in case by the assembler tool, which does
 // the same check.
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-host-only --sysroot=%S/Inputs/CUDA -S 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-host-only --cuda-path=%S/Inputs/CUDA/usr/local/cuda -S 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-device-only --sysroot=%S/Inputs/CUDA -S 2>&1 %s | \
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-device-only --cuda-path=%S/Inputs/CUDA/usr/local/cuda -S 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60
 
 // OK-NOT: error: GPU arch
 
 // OK_SM35-NOT: error: GPU arch sm_35
 
 // We should only get one error per architecture.
 // ERR_SM60: error: GPU arch sm_60 {{.*}}
 // ERR_SM60-NOT: error: GPU arch sm_60
 
 // ERR_SM61: error: GPU arch sm_61 {{.*}}
 // ERR_SM61-NOT: error: GPU arch sm_61