[AMDGCN] Update search path for device libraries

[scchan]

Summary

• Add support for finding device libraries in new ROCm directory

structure

• Simplify and remove the handling of legacy ROCm directory structure

Change-Id: I04da3bc9da85ced4b56b0225efb6b94448b8c5a1

[yaxunl]

files under clang/test/Driver/Inputs/rocm_resource_dir/lib/amdgcn/bitcode-no-abi-ver seem not used.

[scchan]

Remove unused bitcode files

[scchan]

In D140315#4011440, @yaxunl wrote:

files under clang/test/Driver/Inputs/rocm_resource_dir/lib/amdgcn/bitcode-no-abi-ver seem not used.

removed

[yaxunl]

LGTM. Thanks.

[mgorny]

I'm sorry for reporting it this late (clang's been broken for over two weeks, so we couldn't periodically test it) but this change is causing test regressions on Gentoo/amd64:

FAIL: Clang :: Driver/hip-device-libs.hip (9304 of 17486)
******************** TEST 'Clang :: Driver/hip-device-libs.hip' FAILED ********************
[...]
+ : 'RUN: at line 92'
+ /var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/bin/clang -### --target=x86_64-linux-gnu --offload-arch=gfx803 -nogpuinc -resource-dir=/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir /var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/hip_multiple_inputs/b.hip
+ /usr/lib/llvm/16/bin/FileCheck /var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/hip-device-libs.hip --check-prefixes=ALL,FLUSHD,RESDIR
/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/hip-device-libs.hip:203:13: error: ALL-NOT: excluded string found in input
// ALL-NOT: error:
            ^
<stdin>:5:11: note: found here
clang-16: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library
          ^~~~~~

Input file: <stdin>
Check file: /var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/hip-device-libs.hip

-dump-input=help explains the following input dump.

Input was:
<<<<<<
         1: clang version 16.0.0 
         2: Target: x86_64-unknown-linux-gnu 
         3: Thread model: posix 
         4: InstalledDir: /var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/bin 
         5: clang-16: error: cannot find ROCm device library; provide its path via '--rocm-path' or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library 
not:203               !~~~~~                                                                                                                                                            error: no match expected
         6:  "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/bin/clang-16" "-cc1" "-triple" "amdgcn-amd-amdhsa" "-aux-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "-mrelax-all" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "b.hip" "-mrelocation-model" "pic" "-pic-level" "1" "-fhalf-no-semantic-interposition" "-mframe-pointer=all" "-fdenormal-fp-math-f32=preserve-sign,preserve-sign" "-fno-rounding-math" "-mconstructor-aliases" "-aux-target-cpu" "x86-64" "-fcuda-is-device" "-mllvm" "-amdgpu-internalize-symbols" "-fcuda-allow-variadic-functions" "-fvisibility=hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx803" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-resource-dir" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/backward" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/backward" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdeprecated-macro" "-fno-autolink" "-fdebug-compilation-dir=/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/test/Driver" "-ferror-limit" "19" "-fhip-new-launch-api" "-fgnuc-version=4.2.1" "-fcxx-exceptions" "-fexceptions" "-cuid=baca1261ad77b3c1" "-fcuda-allow-variadic-functions" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-aed965/b-gfx803.o" "-x" "hip" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/hip_multiple_inputs/b.hip" 
         7:  "/usr/lib/llvm/16/bin/lld" "-flavor" "gnu" "-m" "elf64_amdgpu" "--no-undefined" "-shared" "-plugin-opt=-amdgpu-internalize-symbols" "-plugin-opt=mcpu=gfx803" "-o" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-8854b0/b-gfx803.out" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-aed965/b-gfx803.o" 
         8:  "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/bin/clang-offload-bundler" "-type=o" "-bundle-align=4096" "-targets=host-x86_64-unknown-linux,hipv4-amdgcn-amd-amdhsa--gfx803" "-input=/dev/null" "-input=/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-8854b0/b-gfx803.out" "-output=/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-54bd59.hipfb" 
         9:  "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/bin/clang-16" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-aux-triple" "amdgcn-amd-amdhsa" "-emit-obj" "-mrelax-all" "-disable-free" "-clear-ast-before-backend" "-disable-llvm-verifier" "-discard-value-names" "-main-file-name" "b.hip" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-fcoverage-compilation-dir=/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/test/Driver" "-resource-dir" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/backward" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/include/g++-v12/backward" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-internal-isystem" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/rocm_resource_dir/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-fdeprecated-macro" "-fdebug-compilation-dir=/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/x/y/clang-abi_x86_64.amd64/test/Driver" "-ferror-limit" "19" "-fhip-new-launch-api" "-fgnuc-version=4.2.1" "-fcxx-exceptions" "-fexceptions" "-fcuda-include-gpubinary" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-54bd59.hipfb" "-cuid=baca1261ad77b3c1" "-fcuda-allow-variadic-functions" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-af8098.o" "-x" "hip" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/work/clang/test/Driver/Inputs/hip_multiple_inputs/b.hip" 
        10:  "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/bin/ld" "-pie" "--hash-style=gnu" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "a.out" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../lib64/Scrt1.o" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../lib64/crti.o" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/crtbeginS.o" "-L/usr/lib/gcc/x86_64-pc-linux-gnu/12" "-L/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../lib64" "-L/lib/../lib64" "-L/usr/lib/../lib64" "-L/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../x86_64-pc-linux-gnu/lib" "-L/lib" "-L/usr/lib" "/var/tmp/portage/sys-devel/clang-16.0.0_pre20230124/temp/b-af8098.o" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "-lc" "-lgcc" "--as-needed" "-lgcc_s" "--no-as-needed" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/crtendS.o" "/usr/lib/gcc/x86_64-pc-linux-gnu/12/../../../../lib64/crtn.o" 
>>>>>>

--

********************

After reverting this commit locally, all tests pass.

Full build log (15M uncompressed):

sys-devel:clang-16.0.0_pre20230124:20230124-104711.log.xz367 KBDownload

[yaxunl]

@scchan the test expects CLANG_INSTALL_LIBDIR_BASENAME to be lib. Maybe in the failed test it was not true.

[mgorny]

In D140315#4077193, @yaxunl wrote:

@scchan the test expects CLANG_INSTALL_LIBDIR_BASENAME to be lib. Maybe in the failed test it was not true.

CMAKE_INSTALL_LIBDIR=lib64, so I guess _BASENAME is that as well.

[yaxunl]

clang/test/Driver/hip-device-libs.hip

206

this might be fixed by (lib|lib64)

[scchan]

In D140315#4076614, @mgorny wrote:

I'm sorry for reporting it this late (clang's been broken for over two weeks, so we couldn't periodically test it) but this change is causing test regressions on Gentoo/amd64:

Thanks for reporting the issue. I'm working on a patch to fix the test.

[hvdijk]

In D140315#4077611, @scchan wrote:

In D140315#4076614, @mgorny wrote:

I'm sorry for reporting it this late (clang's been broken for over two weeks, so we couldn't periodically test it) but this change is causing test regressions on Gentoo/amd64:

Thanks for reporting the issue. I'm working on a patch to fix the test.

I don't think it's actually the test that was at fault, it doesn't look right that this diff uses CLANG_INSTALL_LIBDIR_BASENAME for an unrelated purpose to determining the basename of the libdir, especially considering the comments continue to say clang looks in lib/amdgcn/bitcode rather than <LIBDIR>/amdgcn/bitcode, and your original test specifically assumed that it would always be lib. Should the code perhaps just be updated to hardcode "lib" as it did before?

Either way, the test can still fail after D142506, CLANG_INSTALL_LIBDIR_BASENAME may be any arbitrary string and cannot be assumed to be either lib or lib64. With Debian-style multiarch I have CLANG_INSTALL_LIBDIR_BASENAME=lib/x86_64-linux-gnu, so after updating from Clang 15 to Clang 16 I am now also seeing this test fail.

clang/lib/Driver/ToolChains/AMDGPU.cpp

 //===--- AMDGPU.cpp - AMDGPU ToolChain Implementations ----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "AMDGPU.h"
 #include "CommonArgs.h"
 #include "clang/Basic/TargetID.h"
+#include "clang/Config/config.h"
 #include "clang/Driver/Compilation.h"
 #include "clang/Driver/Distro.h"
 #include "clang/Driver/DriverDiagnostic.h"
 #include "clang/Driver/InputInfo.h"
 #include "clang/Driver/Options.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/Host.h"
if (!LibDevicePath.empty()) {
     if (!FS.exists(LibDevicePath))
       return;
 
     scanLibDevicePath(LibDevicePath);
     HasDeviceLibrary = allGenericLibsValid() && !LibDeviceMap.empty();
     return;
   }
 
-  // The install path situation in old versions of ROCm is a real mess, and
-  // use a different install layout. Multiple copies of the device libraries
-  // exist for each frontend project, and differ depending on which build
-  // system produced the packages. Standalone OpenCL builds also have a
-  // different directory structure from the ROCm OpenCL package.
-  auto &ROCmDirs = getInstallationPathCandidates();
-  for (const auto &Candidate : ROCmDirs) {
-    auto CandidatePath = Candidate.Path;
-
-    // Check device library exists at the given path.
-    auto CheckDeviceLib = [&](StringRef Path) {
-      bool CheckLibDevice = (!NoBuiltinLibs || Candidate.StrictChecking);
-      if (CheckLibDevice && !FS.exists(Path))
-        return false;
+  auto CheckDeviceLib = [&](StringRef Path, bool StrictChecking) {
+    bool CheckLibDevice = (!NoBuiltinLibs || StrictChecking);
 
-      scanLibDevicePath(Path);
 
-      if (!NoBuiltinLibs) {
-        // Check that the required non-target libraries are all available.
-        if (!allGenericLibsValid())
-          return false;
 
-        // Check that we have found at least one libdevice that we can link in
-        // if -nobuiltinlib hasn't been specified.
-        if (LibDeviceMap.empty())
-          return false;
-      }
-      return true;
-    };
 
-    // The possible structures are:
-    // - ${ROCM_ROOT}/amdgcn/bitcode/*
-    // - ${ROCM_ROOT}/lib/*
-    // - ${ROCM_ROOT}/lib/bitcode/*
-    // so try to detect these layouts.
-    static constexpr std::array<const char *, 2> SubDirsList[] = {
-        {"amdgcn", "bitcode"},
-        {"lib", ""},
-        {"lib", "bitcode"},
-    };
-
-    // Make a path by appending sub-directories to InstallPath.
-    auto MakePath = [&](const llvm::ArrayRef<const char *> &SubDirs) {
-      auto Path = CandidatePath;
-      for (auto SubDir : SubDirs)
-        llvm::sys::path::append(Path, SubDir);
-      return Path;
-    };
+  // Find device libraries in <LLVM_DIR>/lib/clang/<ver>/lib/amdgcn/bitcode
+  LibDevicePath = D.ResourceDir;
+  llvm::sys::path::append(LibDevicePath, CLANG_INSTALL_LIBDIR_BASENAME,
+                          "amdgcn", "bitcode");
+  HasDeviceLibrary = CheckDeviceLib(LibDevicePath, true);
+  if (HasDeviceLibrary)
+    return;
 
-    for (auto SubDirs : SubDirsList) {
-      LibDevicePath = MakePath(SubDirs);
-      HasDeviceLibrary = CheckDeviceLib(LibDevicePath);
-      if (HasDeviceLibrary)
-        return;
-    }
-  }
-}
+  // Find device libraries in a legacy ROCm directory structure
+  // ${ROCM_ROOT}/amdgcn/bitcode/*
+  auto &ROCmDirs = getInstallationPathCandidates();
+  for (const auto &Candidate : ROCmDirs) {
+    LibDevicePath = Candidate.Path;
+    llvm::sys::path::append(LibDevicePath, "amdgcn", "bitcode");
+    HasDeviceLibrary = CheckDeviceLib(LibDevicePath, Candidate.StrictChecking);
 
 void RocmInstallationDetector::detectHIPRuntime() {
   SmallVector<Candidate, 4> HIPSearchDirs;
   if (!HIPPathArg.empty())
     HIPSearchDirs.emplace_back(HIPPathArg.str(), /*StrictChecking=*/true);
   else
     HIPSearchDirs.append(getInstallationPathCandidates());
   auto &FS = D.getVFS();

clang/test/Driver/hip-device-libs.hip

 // REQUIRES: x86-registered-target
 // REQUIRES: amdgpu-registered-target
 
 // Test if oclc_daz_opt_on or if oclc_daz_opt_off is linked depending on
 // expected denormal mode.
 
 // Test subtarget with flushing on by default.
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:  --cuda-gpu-arch=gfx803 \
 // RUN:  --rocm-path=%S/Inputs/rocm   \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
 
 // Test subtarget with flushing off by ddefault.
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:  --cuda-gpu-arch=gfx900 \
 // RUN:  --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD,ROCMDIR
 
 
 // Test explicit flag, opposite of target default.
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx900 \
 // RUN:   -fgpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
 
 // Test explicit flag, opposite of target default.
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx803 \
 // RUN:   -fno-gpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD,ROCMDIR
 
 
 // Test explicit flag, same as target default.
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx900 \
 // RUN:   -fno-gpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD,ROCMDIR
 
 
 // Test explicit flag, same as target default.
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx803 \
 // RUN:   -fgpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
 
 // Test last flag wins, not flushing
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx803 \
 // RUN:   -fgpu-flush-denormals-to-zero -fno-gpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD,ROCMDIR
 
 
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx900 \
 // RUN:   -fgpu-flush-denormals-to-zero -fno-gpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm   \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,NOFLUSHD,ROCMDIR
 
 
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx900 \
 // RUN:   -fno-gpu-flush-denormals-to-zero -fgpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm   \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
 
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx803 \
 // RUN:   -fno-gpu-flush-denormals-to-zero -fgpu-flush-denormals-to-zero \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
+// Test finding device lib in resource dir
+// RUN: %clang -### --target=x86_64-linux-gnu \
+// RUN:   --offload-arch=gfx803 -nogpuinc \
+// RUN:   -resource-dir=%S/Inputs/rocm_resource_dir \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,RESDIR
 
 // Test --hip-device-lib-path flag
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx803 -nogpuinc \
 // RUN:   --hip-device-lib-path=%S/Inputs/rocm/amdgcn/bitcode   \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
+// Test --hip-device-lib-path wins over -resource-dir
+// RUN: %clang -### --target=x86_64-linux-gnu \
+// RUN:   --cuda-gpu-arch=gfx803 -nogpuinc \
+// RUN:   --hip-device-lib-path=%S/Inputs/rocm/amdgcn/bitcode   \
+// RUN:   -resource-dir=%S/Inputs/rocm_resource_dir \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,FLUSHD,ROCMDIR
 
 // Test environment variable HIP_DEVICE_LIB_PATH
 // RUN: env HIP_DEVICE_LIB_PATH=%S/Inputs/rocm/amdgcn/bitcode \
 // RUN:   %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx900 -nogpuinc \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,ROCMDIR
+
+// Test environment variable HIP_DEVICE_LIB_PATH wins over -resource-dir
+// RUN: env HIP_DEVICE_LIB_PATH=%S/Inputs/rocm/amdgcn/bitcode \
+// RUN:   %clang -### --target=x86_64-linux-gnu \
+// RUN:   --cuda-gpu-arch=gfx900 -nogpuinc \
+// RUN:   -resource-dir=%S/Inputs/rocm_resource_dir \
+// RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,ROCMDIR
 
 // Test --gpu-instrument-lib
 // RUN: %clang -### --target=x86_64-linux-gnu \
 // RUN:   --cuda-gpu-arch=gfx900 \
 // RUN:   --rocm-path=%S/Inputs/rocm \
 // RUN:   --gpu-instrument-lib=%S/Inputs/hip_multiple_inputs/instrument.bc \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
-// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,INST
+// RUN: 2>&1 | FileCheck %s --check-prefixes=ALL,INST,ROCMDIR
 
 // Test -fast-math
 // RUN: %clang -### --target=x86_64-linux-gnu --offload-arch=gfx900 \
 // RUN:   -ffast-math --rocm-path=%S/Inputs/rocm \
 // RUN:   %S/Inputs/hip_multiple_inputs/b.hip \
 // RUN: 2>&1 | FileCheck %s --check-prefixes=FAST
 
 // Test -ffinite-math-only

 // RUN: %clang -### --target=x86_64-linux-gnu --offload-arch=gfx900 \
 // RUN:   -mcode-object-version=5 \
 // RUN:   --hip-device-lib-path=%S/Inputs/rocm/amdgcn/bitcode-no-abi-ver   \
 // RUN:   --rocm-path=%S/Inputs/rocm %S/Inputs/hip_multiple_inputs/b.hip \
 // RUN: 2>&1 | FileCheck %s --check-prefixes=NOABI5
 
 // ALL-NOT: error:
 // ALL: {{"[^"]*clang[^"]*"}}
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}hip.bc"
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}ocml.bc"
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}ockl.bc"
-
-// FLUSHD-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_daz_opt_on.bc"
-// NOFLUSHD-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_daz_opt_off.bc"
-
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_unsafe_math_off.bc"
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_finite_only_off.bc"
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_correctly_rounded_sqrt_on.bc"
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_wavefrontsize64_on.bc"
-// ALL-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_isa_version_{{[0-9]+}}.bc"
+
+// RESDIR-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR:[^"]+(/|\\\\)rocm_resource_dir(/|\\\\)lib(/|\\\\)amdgcn(/|\\\\).*]]hip.bc"
+// ROCMDIR-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR:[^"]+(/|\\\\)rocm(/|\\\\)amdgcn(/|\\\\).*]]hip.bc"
+
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]ocml.bc"
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]ockl.bc"
+
+// FLUSHD-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_daz_opt_on.bc"
+// NOFLUSHD-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_daz_opt_off.bc"
+
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_unsafe_math_off.bc"
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_finite_only_off.bc"
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_correctly_rounded_sqrt_on.bc"
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_wavefrontsize64_on.bc"
+// ALL-SAME: "-mlink-builtin-bitcode" "[[DEVICELIB_DIR]]oclc_isa_version_{{[0-9]+}}.bc"
 // INST-SAME: "-mlink-builtin-bitcode" "{{.*}}instrument.bc"
 
 // FAST: "-mlink-builtin-bitcode" "{{.*}}oclc_daz_opt_off.bc"
 // FAST-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_unsafe_math_on.bc"
 // FAST-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_finite_only_on.bc"
 // FAST-SAME: "-mlink-builtin-bitcode" "{{.*}}oclc_correctly_rounded_sqrt_on.bc"
 
 // FINITE: "-mlink-builtin-bitcode" "{{.*}}oclc_daz_opt_off.bc"