[CUDA] Improve CUDA version detection and diagnostics.

[tra]

Summary

Always use cuda.h to detect CUDA version. It's a more universal approach
compared to version.txt which is no longer present in recent CUDA versions.

Split the 'unknown CUDA version' warning in two:

• when detected CUDA version is partially supported by clang. It's expected to

work in general, at the feature parity with the latest supported CUDA
version. and may be missing support for the new features/instructions/GPU
variants. Clang will issue a warning.

• when detected version is new. Recent CUDA versions have been working with

clang reasonably well, and will likely to work similarly to the partially
supported ones above. Or it may not work at all. Clang will issue a warning and
proceed as if the latest known CUDA version was detected.

[Hahnfeld]

clang/test/Driver/cuda-version-check.cu

75

Can we keep the [unknown] out of the warning message? I understand that we cannot nicely print a version identifier that we don't know about, so maybe only print if the version is known and newer than PARTIALLY_SUPPORTED?

[yaxunl]

The change about amdgpu LGTM

[tra]

Do not report the version if we don't know it.

[tra]

clang/test/Driver/cuda-version-check.cu

75

Done.

[Hahnfeld]

Otherwise LGTM

clang/lib/Driver/ToolChains/Cuda.cpp

110–111

I think you have to prepend the space in order to match CUDA version%0 is newer than, or move the %0. But I didn't try locally, might be missing something.

[tra]

Prepend space to the version string.

[tra]

clang/lib/Driver/ToolChains/Cuda.cpp

110–111

Good catch. Fixed.

[Hahnfeld]

clang/lib/Driver/ToolChains/Cuda.cpp

218–220

The compiler is now warning here because of the assignment to VERSION in the ternary operator

[tra]

clang/lib/Driver/ToolChains/Cuda.cpp

218–220

I'll fix it shortly. I also need to figure out why my build does not produce the warning.

[tra]

Fixed an error spotted by reviewer.

[Hahnfeld]

clang/lib/Driver/ToolChains/Cuda.cpp

218–220

The exact warning is

LLVM/src/clang/lib/Driver/ToolChains/Cuda.cpp: In constructor ‘clang::driver::CudaInstallationDetector::CudaInstallationDetector(const clang::driver::Driver&, const llvm::Triple&, const llvm::opt::ArgList&)’:              
LLVM/src/clang/lib/Driver/ToolChains/Cuda.cpp:207:15: warning: operation on ‘((clang::driver::CudaInstallationDetector*)this)->clang::driver::CudaInstallationDetector::Version’ may be undefined [-Wsequence-point]          
       Version = FS.exists(LibDevicePath + "/libdevice.10.bc")
       ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                     ? Version = CudaVersion::NEW
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                     : Version = CudaVersion::CUDA_70;
                     ~~~~~~~~~~~~~~~~~~~~~~~

Re-reading the code, I actually think this is a false positive of GCC 8.3.1 (on CentOS 8) and Version is never undefined. But it was redundant, so it's good to remove it anyway.

[tambre]

This unfortunately breaks using Debian distribution CUDA.
The Debian/Ubuntu special case is now useless.

On Debian:

• cuda.h is in /usr/include, as it must be per policy.
• libdevice.10.bc is in /usr/lib/cuda/nvidia-cuda-toolkit/libdevice and /usr/lib/cuda/nvvm/libdevice is symlinked to that.

Passing --cuda-path=/usr/lib/cuda previously worked, but now results in trying to parse from /usr/lib/cuda/include/cuda.h (doesn't exist) and Clang defaulting to the newest available version, which isn't supported by the ptxas currently in Debian.
NVCC has no issues with this layout.

Symlinking /usr/lib/cuda/include to /usr/include on Debian's side seems like the obvious short-term solution for trunk.
Could probably be backported to ensure LTSes will work with Clang 14.

If any thoughts for an alternative solution let me know.
I have filed Debian bug #995122.

[tra]

So, what's the current state of affairs regarding CUDA SDK layout in debian?
Clang does rely on very particular ordering of includes, so having CUDA headers in a location clang does not expect will lead to issues sooner or later.
If the headers are not available where --cuda-path points clang to, I'm not sure what clang is supposed to do. Second-guessing explicit user input is not a good idea in general.

CUDA SDK no longer ships version.txt, so cuda.h is the only reliable mechanism for detecting CUDA version and clang does need to know it.

In case the version has not been found we also can not choose an arbitrary old version as the default.

So, one workaround would be to install CUDA-11.4. AFAICT, we do find the headers and ptxas, but misdetect CUDA version.
Another workaround would be to place a fake /usr/lib/cuda/include/cuda.h with something like this:

#pragma push_macro("CUDA_VERSION")
#define CUDA_VERSION 11030
#pragma pop_macro("CUDA_VERSION")
#include_next <cuda.h>

[tambre]

In D108247#3025238, @tra wrote:

So, what's the current state of affairs regarding CUDA SDK layout in debian?

/usr/bin: most executables, nvcc and nvprof shims
/usr/lib/nvidia-cuda-toolkit/bin: cicc, crt, nvcc, nvprof, g++ and gcc wrappers for supporting different compilers
/usr/lib/nvidia-cuda-toolkit/libdevice: libdevice.10.bc
/usr/lib/cuda/{bin,include,lib64}: empty directories
/usr/lib/cuda/nvvm/libdevice: symlinked to /usr/lib/nvidia-cuda-toolkit/libdevice
/usr/include: headers

CUDA SDK no longer ships version.txt, so cuda.h is the only reliable mechanism for detecting CUDA version and clang does need to know it.
In case the version has not been found we also can not choose an arbitrary old version as the default.

Agreed, sensible.

So, one workaround would be to install CUDA-11.4. AFAICT, we do find the headers and ptxas, but misdetect CUDA version.

I don't think that's going to be acceptable for most distributions, which might ship a Clang supporting a newer CUDA than the one they ship.

Another workaround would be to place a fake /usr/lib/cuda/include/cuda.h with something like this:

My CMake CI bot encountered cmath template errors after I manually symlinked /usr/lib/cuda/include to /usr/include, though that solved the version detection.
Adding a shim like the one you show fixes everything and compiling works.

Clang does rely on very particular ordering of includes, so having CUDA headers in a location clang does not expect will lead to issues sooner or later.

Having the headers in the system-wide /usr/include shouldn't be a problem though, right?
I'm not quite sure why simply having /usr/lib/cuda/include → /usr/include symlink breaks as seen on my bot.
Prior to this an empty /usr/lib/cuda/include and Clang finding them through the regular include search path worked.

If the headers are not available where --cuda-path points clang to, I'm not sure what clang is supposed to do. Second-guessing explicit user input is not a good idea in general.

I agree that second-guessing is bad, but expecting it to be available in the include search path otherwise seems reasonable.

[tra]

In D108247#3025415, @tambre wrote:

Another workaround would be to place a fake /usr/lib/cuda/include/cuda.h with something like this:

My CMake CI bot [[ https://open.cdash.org/test/498767666 | encountered cmath template errors ]] after I manually symlinked /usr/lib/cuda/include to /usr/include, though that solved the version detection.
Adding a shim like the one you show fixes everything and compiling works.

Clang does rely on very particular ordering of includes, so having CUDA headers in a location clang does not expect will lead to issues sooner or later.

Having the headers in the system-wide /usr/include shouldn't be a problem though, right?
I'm not quite sure why simply having /usr/lib/cuda/include → /usr/include symlink breaks as seen on my bot.

If you look at the logs of the broken build you will see that /usr/lib/cuda/include/ appears fairly early in the search path, before the standard library C++ headers.
Symlinking it to /usr/include places a lot of other includes and that breaks the include order for the C++ library which has its own assumptions and requirements.

Prior to this an empty /usr/lib/cuda/include and Clang finding them through the regular include search path worked.

It's possible that the CUDA headers themselves do not need to be placed in the include path that early. We do need cuda_wrappers to be there, but at the moment I do not recall why we ended up placing cuda/include there as well. The fact that it worked for you with the empty directory suggests that it may not be necessary.

If we move cuda includes down to where /usr/include is in the search list, it should allow symlinking it to /usr/include.
I'll check whether that's feasible.

If the headers are not available where --cuda-path points clang to, I'm not sure what clang is supposed to do. Second-guessing explicit user input is not a good idea in general.

I agree that second-guessing is bad, but expecting it to be available in the include search path otherwise seems reasonable.

The issue with the include search path is that it's something that's not readily available at the clang driver level at the point in time where we need to detect cuda version.
The driver does construct the include arguments for cc1 invocations, but all that happens quite a bit later after CUDA detection is done.

clang/include/clang/Basic/Cuda.h

enum class CudaVersion {
   CUDA_100,
   CUDA_101,
   CUDA_102,
   CUDA_110,
   CUDA_111,
   CUDA_112,
   CUDA_113,
   CUDA_114,
-  LATEST = CUDA_114,
-  LATEST_SUPPORTED = CUDA_101,
+  FULLY_SUPPORTED = CUDA_101,
+  PARTIALLY_SUPPORTED =
+      CUDA_114, // Partially supported. Proceed with a warning.
+  NEW = 10000,  // Too new. Issue a warning, but allow using it.
 };
 const char *CudaVersionToString(CudaVersion V);
 // Input is "Major.Minor"
 CudaVersion CudaStringToVersion(const llvm::Twine &S);
 
 enum class CudaArch {
   UNUSED,
   UNKNOWN,

clang/include/clang/Basic/DiagnosticDriverKinds.td

 def err_drv_undetermined_amdgpu_arch : Error<
   "cannot determine AMDGPU architecture: %0; consider passing it via "
   "'--march'">;
 def err_drv_cuda_version_unsupported : Error<
   "GPU arch %0 is supported by CUDA versions between %1 and %2 (inclusive), "
   "but installation at %3 is %4; use '--cuda-path' to specify a different CUDA "
   "install, pass a different GPU arch with '--cuda-gpu-arch', or pass "
   "'--no-cuda-version-check'">;
-def warn_drv_unknown_cuda_version: Warning<
-  "unknown CUDA version: %0; assuming the latest supported version %1">,
+def warn_drv_new_cuda_version: Warning<
+  "CUDA version %0 is newer than the latest%select{| partially}1 supported version %2">,
+  InGroup<CudaUnknownVersion>;
+def warn_drv_partially_supported_cuda_version: Warning<
+  "CUDA version %0 is only partially supported">,
   InGroup<CudaUnknownVersion>;
 def err_drv_cuda_host_arch : Error<
   "unsupported architecture '%0' for host compilation">;
 def err_drv_mix_cuda_hip : Error<
   "mixed CUDA and HIP compilation is not supported">;
 def err_drv_bad_target_id : Error<
   "invalid target ID '%0'; format is a processor name followed by an optional "
   "colon-delimited list of features followed by an enable/disable sign (e.g., "

clang/lib/Basic/Cuda.cpp

const char *CudaVersionToString(CudaVersion V) {
   case CudaVersion::CUDA_111:
     return "11.1";
   case CudaVersion::CUDA_112:
     return "11.2";
   case CudaVersion::CUDA_113:
     return "11.3";
   case CudaVersion::CUDA_114:
     return "11.4";
+  case CudaVersion::NEW:
+    return "[new]";
   }
   llvm_unreachable("invalid enum");
 }
 
 CudaVersion CudaStringToVersion(const llvm::Twine &S) {
   return llvm::StringSwitch<CudaVersion>(S.str())
       .Case("7.0", CudaVersion::CUDA_70)
       .Case("7.5", CudaVersion::CUDA_75)
CudaVersion MinVersionForCudaArch(CudaArch A) {
   default:
     llvm_unreachable("invalid enum");
   }
 }
 
 CudaVersion MaxVersionForCudaArch(CudaArch A) {
   // AMD GPUs do not depend on CUDA versions.
   if (IsAMDGpuArch(A))
-    return CudaVersion::LATEST;
+    return CudaVersion::NEW;
 
   switch (A) {
   case CudaArch::UNKNOWN:
     return CudaVersion::UNKNOWN;
   case CudaArch::SM_20:
   case CudaArch::SM_21:
     return CudaVersion::CUDA_80;
   case CudaArch::SM_30:
     return CudaVersion::CUDA_110;
   default:
-    return CudaVersion::LATEST;
+    return CudaVersion::NEW;
   }
 }
 
 CudaVersion ToCudaVersion(llvm::VersionTuple Version) {
   int IVer =
       Version.getMajor() * 10 + Version.getMinor().getValueOr(0);
   switch(IVer) {
   case 70:

clang/lib/Driver/ToolChains/Cuda.h

 
 /// A class to find a viable CUDA installation
 class CudaInstallationDetector {
 private:
   const Driver &D;
   bool IsValid = false;
   CudaVersion Version = CudaVersion::UNKNOWN;
   std::string DetectedVersion;
-  bool DetectedVersionIsNotSupported = false;
+  bool DetectedVersionIsPartiallySupported = false;
+  bool DetectedVersionIsTooNew = false;
   std::string InstallPath;
   std::string BinPath;
   std::string LibPath;
   std::string LibDevicePath;
   std::string IncludePath;
   llvm::StringMap<std::string> LibDeviceMap;
 
   // CUDA architectures for which we have raised an error in
public:
   void CheckCudaVersionSupportsArch(CudaArch Arch) const;
 
   /// Check whether we detected a valid Cuda install.
   bool isValid() const { return IsValid; }
   /// Print information about the detected CUDA installation.
   void print(raw_ostream &OS) const;
 
   /// Get the detected Cuda install's version.
-  CudaVersion version() const { return Version; }
+  CudaVersion version() const {
+    return Version == CudaVersion::NEW ? CudaVersion::PARTIALLY_SUPPORTED
+                                       : Version;
+  }
   /// Get the detected Cuda installation path.
   StringRef getInstallPath() const { return InstallPath; }
   /// Get the detected path to Cuda's bin directory.
   StringRef getBinPath() const { return BinPath; }
   /// Get the detected Cuda Include path.
   StringRef getIncludePath() const { return IncludePath; }
   /// Get the detected Cuda library path.
   StringRef getLibPath() const { return LibPath; }

clang/lib/Driver/ToolChains/Cuda.cpp

 #include "clang/Config/config.h"
 #include "clang/Driver/Compilation.h"
 #include "clang/Driver/Distro.h"
 #include "clang/Driver/Driver.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/InputInfo.h"
 #include "clang/Driver/Options.h"
 #include "llvm/ADT/Optional.h"
+#include "llvm/ADT/StringExtras.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Host.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/TargetParser.h"
 #include "llvm/Support/VirtualFileSystem.h"
 #include <system_error>
 
 using namespace clang::driver;
 using namespace clang::driver::toolchains;
 using namespace clang::driver::tools;
 using namespace clang;
 using namespace llvm::opt;
 
 namespace {
 struct CudaVersionInfo {
-  std::string DetectedVersion;
+  std::string DetectedVersionString;
   CudaVersion Version;
 };
-// Parses the contents of version.txt in an CUDA installation.  It should
-// contain one line of the from e.g. "CUDA Version 7.5.2".
-CudaVersionInfo parseCudaVersionFile(llvm::StringRef V) {
-  V = V.trim();
-  if (!V.startswith("CUDA Version "))
-    return {V.str(), CudaVersion::UNKNOWN};
-  V = V.substr(strlen("CUDA Version "));
-  SmallVector<StringRef,4> VersionParts;
-  V.split(VersionParts, '.');
-  return {"version.txt: " + V.str() + ".",
-          VersionParts.size() < 2
-              ? CudaVersion::UNKNOWN
-              : CudaStringToVersion(
-                    join_items(".", VersionParts[0], VersionParts[1]))};
-}
 
 CudaVersion getCudaVersion(uint32_t raw_version) {
   if (raw_version < 7050)
     return CudaVersion::CUDA_70;
   if (raw_version < 8000)
     return CudaVersion::CUDA_75;
   if (raw_version < 9000)
     return CudaVersion::CUDA_80;
CudaVersion getCudaVersion(uint32_t raw_version) {
   if (raw_version < 11020)
     return CudaVersion::CUDA_111;
   if (raw_version < 11030)
     return CudaVersion::CUDA_112;
   if (raw_version < 11040)
     return CudaVersion::CUDA_113;
   if (raw_version < 11050)
     return CudaVersion::CUDA_114;
-  return CudaVersion::LATEST;
+  return CudaVersion::NEW;
 }
 
 CudaVersionInfo parseCudaHFile(llvm::StringRef Input) {
   // Helper lambda which skips the words if the line starts with them or returns
   // None otherwise.
   auto StartsWithWords =
       [](llvm::StringRef Line,
          const SmallVector<StringRef, 3> words) -> llvm::Optional<StringRef> {
     for (StringRef word : words) {
       if (!Line.consume_front(word))
         return {};
       Line = Line.ltrim();
     }
     return Line;
   };
 
   Input = Input.ltrim();
   while (!Input.empty()) {
     if (auto Line =
             StartsWithWords(Input.ltrim(), {"#", "define", "CUDA_VERSION"})) {
       uint32_t RawVersion;
       Line->consumeInteger(10, RawVersion);
-      return {"cuda.h: CUDA_VERSION=" + Twine(RawVersion).str() + ".",
-              getCudaVersion(RawVersion)};
+      return {llvm::utostr(RawVersion), getCudaVersion(RawVersion)};
     }
     // Find next non-empty line.
     Input = Input.drop_front(Input.find_first_of("\n\r")).ltrim();
   }
-  return {"cuda.h: CUDA_VERSION not found.", CudaVersion::UNKNOWN};
+  return {"CUDA_VERSION not found.", CudaVersion::UNKNOWN};
 }
 } // namespace
 
 void CudaInstallationDetector::WarnIfUnsupportedVersion() {
-  if (DetectedVersionIsNotSupported)
-    D.Diag(diag::warn_drv_unknown_cuda_version)
+  if (DetectedVersionIsTooNew)
+    D.Diag(diag::warn_drv_new_cuda_version)
         << DetectedVersion
-        << CudaVersionToString(CudaVersion::LATEST_SUPPORTED);
+        << (CudaVersion::PARTIALLY_SUPPORTED != CudaVersion::FULLY_SUPPORTED)
+        << CudaVersionToString(CudaVersion::PARTIALLY_SUPPORTED);
+  else if (DetectedVersionIsPartiallySupported)
+    D.Diag(diag::warn_drv_partially_supported_cuda_version) << DetectedVersion;
 }
 
 CudaInstallationDetector::CudaInstallationDetector(
     const Driver &D, const llvm::Triple &HostTriple,
     const llvm::opt::ArgList &Args)
     : D(D) {
   struct Candidate {
     std::string Path;
for (const auto &Candidate : Candidates) {
     if (HostTriple.isArch64Bit() && FS.exists(InstallPath + "/lib64"))
       LibPath = InstallPath + "/lib64";
     else if (FS.exists(InstallPath + "/lib"))
       LibPath = InstallPath + "/lib";
     else
       continue;
 
     CudaVersionInfo VersionInfo = {"", CudaVersion::UNKNOWN};
-    if (auto VersionFile = FS.getBufferForFile(InstallPath + "/version.txt"))
-      VersionInfo = parseCudaVersionFile((*VersionFile)->getBuffer());
-    // If version file didn't give us the version, try to find it in cuda.h
-    if (VersionInfo.Version == CudaVersion::UNKNOWN)
-      if (auto CudaHFile = FS.getBufferForFile(InstallPath + "/include/cuda.h"))
-        VersionInfo = parseCudaHFile((*CudaHFile)->getBuffer());
-    // As the last resort, make an educated guess between CUDA-7.0, (which had
-    // no version.txt file and had old-style libdevice bitcode ) and an unknown
-    // recent CUDA version (no version.txt, new style bitcode).
+    // As the last resort, make an educated guess between CUDA-7.0, which had
+    // old-style libdevice bitcode, and an unknown recent CUDA version.
     if (VersionInfo.Version == CudaVersion::UNKNOWN) {
-      VersionInfo.Version = (FS.exists(LibDevicePath + "/libdevice.10.bc"))
-                                ? Version = CudaVersion::LATEST
+      VersionInfo.Version = FS.exists(LibDevicePath + "/libdevice.10.bc")
+                                ? Version = CudaVersion::NEW
                                 : Version = CudaVersion::CUDA_70;
-      VersionInfo.DetectedVersion = "no version found in version.txt or cuda.h";
+      VersionInfo.DetectedVersionString = "[unknown]";
     }
 
     Version = VersionInfo.Version;
-    DetectedVersion = VersionInfo.DetectedVersion;
+    DetectedVersion = VersionInfo.DetectedVersionString;
 
-    // TODO(tra): remove the warning once we have all features of 10.2
-    // and 11.0 implemented.
-    DetectedVersionIsNotSupported = Version > CudaVersion::LATEST_SUPPORTED;
+    DetectedVersionIsPartiallySupported =
+        Version > CudaVersion::FULLY_SUPPORTED;
+    DetectedVersionIsTooNew = Version > CudaVersion::PARTIALLY_SUPPORTED;
 
     if (Version >= CudaVersion::CUDA_90) {
       // CUDA-9+ uses single libdevice file for all GPU variants.
       std::string FilePath = LibDevicePath + "/libdevice.10.bc";
       if (FS.exists(FilePath)) {
         for (int Arch = (int)CudaArch::SM_30, E = (int)CudaArch::LAST; Arch < E;
              ++Arch) {
           CudaArch GpuArch = static_cast<CudaArch>(Arch);

clang/test/Driver/Inputs/CUDA-new/usr/local/cuda/include/cuda.h

+//
+// Placeholder file for testing CUDA version detection
+//
+
+#define CUDA_VERSION 99010
+
+//

clang/test/Driver/Inputs/CUDA-unknown/usr/local/cuda/version.txt

-CUDA Version 999.999.999

clang/test/Driver/Inputs/CUDA_80/usr/local/cuda/include/cuda.h

+//
+// Placeholder file for testing CUDA version detection
+//
+
+#define CUDA_VERSION 8000
+
+//

clang/test/Driver/Inputs/CUDA_80/usr/local/cuda/version.txt

-CUDA Version 8.0.42

clang/test/Driver/Inputs/CUDA_90/usr/local/cuda/include/cuda.h

+//
+// Placeholder file for testing CUDA version detection
+//
+
+#define CUDA_VERSION 9000
+
+//

clang/test/Driver/cuda-version-check.cu

 // REQUIRES: clang-driver
 // REQUIRES: x86-registered-target
 // REQUIRES: nvptx-registered-target
 
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_20 --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_20 --cuda-path=%S/Inputs/CUDA_80/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA_80/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=OK
-// Test version guess when no version.txt or cuda.h are found
+// Test version guess when cuda.h has not been found
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA-unknown/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=UNKNOWN_VERSION
-// Unknown version with version.txt present
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA_102/usr/local/cuda 2>&1 %s | \
-// RUN:    FileCheck %s --check-prefix=UNKNOWN_VERSION_V
-// Unknown version with no version.txt but with version info present in cuda.h
-// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA_111/usr/local/cuda 2>&1 %s | \
-// RUN:    FileCheck %s --check-prefix=UNKNOWN_VERSION_H
+// Unknown version info present in cuda.h
+// RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA-new/usr/local/cuda 2>&1 %s | \
+// RUN:    FileCheck %s --check-prefix=NEW_VERSION
 // Make sure that we don't warn about CUDA version during C++ compilation.
 // RUN: %clang --target=x86_64-linux -v -### -x c++ --cuda-gpu-arch=sm_60 \
 // RUN:    --cuda-path=%S/Inputs/CUDA-unknown/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=UNKNOWN_VERSION_CXX
 
 // The installation at Inputs/CUDA is CUDA 7.0, which doesn't support sm_60.
 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-path=%S/Inputs/CUDA/usr/local/cuda 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60

 // RUN: %clang --target=x86_64-linux -v -### --cuda-gpu-arch=sm_60 --cuda-device-only --cuda-path=%S/Inputs/CUDA/usr/local/cuda -S 2>&1 %s | \
 // RUN:    FileCheck %s --check-prefix=ERR_SM60
 
 // OK-NOT: error: GPU arch
 
 // OK_SM35-NOT: error: GPU arch sm_35
 
 // We should only get one error per architecture.
+// ERR_SM20: error: GPU arch sm_20 {{.*}}
+// ERR_SM20-NOT: error: GPU arch sm_20
+
 // ERR_SM60: error: GPU arch sm_60 {{.*}}
 // ERR_SM60-NOT: error: GPU arch sm_60
 
 // ERR_SM61: error: GPU arch sm_61 {{.*}}
 // ERR_SM61-NOT: error: GPU arch sm_61
 
-// UNKNOWN_VERSION_V: unknown CUDA version: version.txt:{{.*}}; assuming the latest supported version
-// UNKNOWN_VERSION_H: unknown CUDA version: cuda.h: CUDA_VERSION={{.*}}; assuming the latest supported version
-// UNKNOWN_VERSION: unknown CUDA version: no version found in version.txt or cuda.h; assuming the latest supported version
+// UNKNOWN_VERSION: CUDA version [unknown] is newer than the latest{{.*}} supported version
+// NEW_VERSION: CUDA version 99010 is newer than the latest{{.*}} supported version
 // UNKNOWN_VERSION_CXX-NOT: unknown CUDA version