[HIP] Fix spack HIP device lib detection

[yaxunl]

Summary

spack HIP device library is installed at amdgcn directory under llvm/clang
directory.

This patch fixes detection of HIP device library for spack.

[tra]

Context for the changes in this patch: https://reviews.llvm.org/D97340#2775477

LGTM syntax-wise. I can't say whether it's the right change for spack. I'll let @haampie make the call on the functionality.

[haampie]

Hi @yaxunl, generally looks good to me. I've just tried this on top of https://github.com/spack/spack/pull/23859 and I'm getting:

$ /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/bin/clang++ -x hip -print-rocm-search-dirs hi.cc 
ROCm installation search path (Spack 4.2.0): /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/lib/clang/12.0.0
ROCm installation search path: /opt/rocm
ROCm installation search path (Spack 4.2.0): /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/lib/clang/12.0.0
ROCm installation search path: /opt/rocm
clang-12: error: cannot find HIP runtime. Provide its path via --rocm-path, or pass -nogpuinc to build without HIP runtime.
clang-12: error: cannot find HIP runtime. Provide its path via --rocm-path, or pass -nogpuinc to build without HIP runtime.

$ echo /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0* | xargs -n1
/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0-siong3jqyqtwb62zvauv43eo2zc4eup7
/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0-wqgj4zleuboc3jpqwvvvwz3p4sdzxbpt
/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0-xg4uyi67oq6tdtegf6h3ppezztr44sh3

So, bitcode is found, hip is not found because of multiple matches.

My suggestion remains to drop the magic that looks for the hip install prefix based on the spack directory layout, since in some cases it may be incorrect (upsteam hip install and an unrelated local hip install would load the wrong hip, or user has a different config for the directory structure).

If you use llvm with amd support, you have to install hip anyways, and when hip is installed, you can just load it with spack, and it will set environment variables that will be sufficient for clang to know where hip is:

$ spack env activate --sh hip
export HSA_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hsa-rocr-dev-4.2.0-fukddlz3oiwrvylxato7kff2pg6dd5oj;
export ROCM_PATH=/path/to/environment/.spack-env/view;
export LLVM_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo;
export ROCMINFO_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/rocminfo-4.2.0-briz73u35rpzdj3roxlaqlzjc5sw5b67;
export HIP_COMPILER=clang;
export HIP_DEVICE_LIB_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/amdgcn/bitcode;
export HIP_CLANG_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/bin;
export DEVICE_LIB_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/amdgcn/bitcode;
export PATH='/path/to/environment/.spack-env/view/bin:...';
export HIP_PATH=/path/to/environment/.spack-env/view;
export HIP_PLATFORM=amd;

this is how it's used:

$ spack env activate hip
$ clang++ -x hip -print-rocm-search-dirs hi.cc 
ROCm installation search path: /path/to/environment/.spack-env/view
ROCm installation search path: /path/to/environment/.spack-env/view
$ ./a.out 
hello world

[haampie]

Accepting this, with the comment that I think it's better to follow up with a patch that reverts autodetecting hip from clang based on the spack directory structure.

[yaxunl]

In D103281#2796206, @haampie wrote:

Accepting this, with the comment that I think it's better to follow up with a patch that reverts autodetecting hip from clang based on the spack directory structure.

Thanks. will remove auto-detect of Spack HIP package in a separate patch.

[haampie]

Is @tra able to land this? I'm not

clang/lib/Driver/ToolChains/AMDGPU.cpp

RocmInstallationDetector::getInstallationPathCandidates() {
   // Deduce ROCm path by the real path of the invoked clang, resolving symbolic
   // link of clang itself.
   llvm::SmallString<256> RealClangPath;
   llvm::sys::fs::real_path(D.getClangProgramPath(), RealClangPath);
   auto ParentPath = llvm::sys::path::parent_path(RealClangPath);
   if (ParentPath != InstallDir)
     ROCmSearchDirs.emplace_back(DeduceROCmPath(ParentPath));
 
-  // Device library may be installed in clang resource directory.
+  // Device library may be installed in clang or resource directory.
+  auto ClangRoot = llvm::sys::path::parent_path(InstallDir);
+  auto RealClangRoot = llvm::sys::path::parent_path(ParentPath);
+  ROCmSearchDirs.emplace_back(ClangRoot.str(), /*StrictChecking=*/true);
+  if (RealClangRoot != ClangRoot)
+    ROCmSearchDirs.emplace_back(RealClangRoot.str(), /*StrictChecking=*/true);
   ROCmSearchDirs.emplace_back(D.ResourceDir,
                               /*StrictChecking=*/true);
 
   ROCmSearchDirs.emplace_back(D.SysRoot + "/opt/rocm",
                               /*StrictChecking=*/true);
 
   // Find the latest /opt/rocm-{release} directory.
   std::error_code EC;
for (const auto &Candidate : ROCmDirs) {
     static constexpr std::array<const char *, 2> SubDirsList[] = {
         {"amdgcn", "bitcode"},
         {"lib", ""},
         {"lib", "bitcode"},
     };
 
     // Make a path by appending sub-directories to InstallPath.
     auto MakePath = [&](const llvm::ArrayRef<const char *> &SubDirs) {
-      // Device library built by SPACK is installed to
-      // <rocm_root>/rocm-device-libs-<rocm_release_string>-<hash> directory.
-      auto SPACKPath = findSPACKPackage(Candidate, "rocm-device-libs");
-      auto Path = SPACKPath.empty() ? CandidatePath : SPACKPath;
+      auto Path = CandidatePath;
       for (auto SubDir : SubDirs)
         llvm::sys::path::append(Path, SubDir);
       return Path;
     };
 
     for (auto SubDirs : SubDirsList) {
       LibDevicePath = MakePath(SubDirs);
       HasDeviceLibrary = CheckDeviceLib(LibDevicePath);

clang/test/Driver/rocm-detect.hip

 // RUN:   -target x86_64-linux-gnu --cuda-gpu-arch=gfx900 --print-rocm-search-dirs %s 2>&1 \
 // RUN:   | FileCheck -check-prefixes=SPACK %s
 
 // Test SPACK installation with multiple hip and rocm-device-libs packages of the same
 // ROCm release. Clang cannot determine which one to use and emit diagnostics. --hip-path
 // and --rocm-device-lib-path can be used to specify them.
 
 // RUN: cp -r %T/rocm-spack/hip-* %T/rocm-spack/hip-4.0.0-abcd
-// RUN: cp -r %T/rocm-spack/rocm-device-libs-* %T/rocm-spack/rocm-device-libs-4.0.0-efgh
 // RUN: %T/rocm-spack/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin/clang -### -v \
 // RUN:   -target x86_64-linux-gnu --cuda-gpu-arch=gfx900 %s 2>&1 \
 // RUN:   | FileCheck -check-prefixes=SPACK-MULT %s
 // RUN: %T/rocm-spack/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin/clang -### -v \
 // RUN:   -target x86_64-linux-gnu --cuda-gpu-arch=gfx900 \
 // RUN:   --hip-path=%T/rocm-spack/hip-4.0.0-abcd \
-// RUN:   --rocm-device-lib-path=%T/rocm-spack/rocm-device-libs-4.0.0-efgh/amdgcn/bitcode \
 // RUN:    %s 2>&1 | FileCheck -check-prefixes=SPACK-SET %s
 
 // Test invalid SPACK ROCm installation missing hip and rocm-device-libs packages.
 // The message about SPACK is emitted only if -v is specified.
 
 // RUN: rm -rf %T/rocm-spack/hip-*
-// RUN: rm -rf %T/rocm-spack/rocm-device-libs-*
+// RUN: rm -rf %T/rocm-spack/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/amdgcn
 // RUN: %T/rocm-spack/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin/clang -### -v \
 // RUN:   -target x86_64-linux-gnu --cuda-gpu-arch=gfx900 %s 2>&1 \
 // RUN:   | FileCheck -check-prefixes=SPACK-MISS %s
 // RUN: %T/rocm-spack/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin/clang --version 2>&1 \
 // RUN:   | FileCheck -check-prefixes=SPACK-MISS-SILENT %s
 
 // GFX902-DEFAULTLIBS: error: cannot find ROCm device library for gfx902. Provide its path via --rocm-path or --rocm-device-lib-path, or pass -nogpulib to build without ROCm device library
 
 // NODEFAULTLIBS-NOT: error: cannot find
 
 // COMMON: "-triple" "amdgcn-amd-amdhsa"
 
 // ROCM-ENV: ROCm installation search path: {{.*}}/Inputs/rocm
 
 // ROCM-REL: ROCm installation search path: {{.*}}/opt/rocm
 // ROCM-REL: ROCm installation search path: {{.*}}/opt/rocm-3.10.0
 
 // SPACK: ROCm installation search path (Spack 4.0.0): [[DIR:.*]]
 // SPACK: ROCm installation search path: [[CLANG:.*]]
+// SPACK: ROCm installation search path: [[DIR]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z
 // SPACK: ROCm installation search path: [[CLANG]]/{{(llvm/)?}}lib{{[0-9]*}}/clang/{{[0-9.]+}}
 // SPACK: ROCm installation search path: /opt/rocm
 // SPACK: InstalledDir: [[DIR]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin
 // SPACK: Found HIP installation: [[DIR]]/hip-4.0.0-5f63slrursbrvfe2txrrjkynbsywsob5, version 4.0.20214-a2917cd
 // SPACK: "-triple" "amdgcn-amd-amdhsa"
-// SPACK-SAME: "-mlink-builtin-bitcode" "[[DIR]]/rocm-device-libs-4.0.0-6wnyzz4hgl3hr7uswasnagt7j2adctbs/amdgcn/bitcode/hip.bc"
+// SPACK-SAME: "-mlink-builtin-bitcode" "[[DIR]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/amdgcn/bitcode/hip.bc"
 // SPACK-SAME: "-internal-isystem" "[[DIR]]/hip-4.0.0-5f63slrursbrvfe2txrrjkynbsywsob5/include"
 
 // SPACK-MULT: InstalledDir: [[DIR:.*]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin
 // SPACK-MULT-DAG: Cannot use SPACK package hip-4.0.0 at [[DIR]] due to multiple installations for the same version
-// SPACK-MULT-DAG: Cannot use SPACK package rocm-device-libs-4.0.0 at [[DIR]] due to multiple installations for the same version
 // SPACK-MULT-NOT: Found HIP installation: [[DIR]]/hip-4.0.0-5f63slrursbrvfe2txrrjkynbsywsob5, version 4.0.20214-a2917cd
-// SPACK-MULT-NOT: "-mlink-builtin-bitcode" "[[DIR]]/rocm-device-libs-4.0.0-6wnyzz4hgl3hr7uswasnagt7j2adctbs/amdgcn/bitcode/hip.bc"
 // SPACK-MULT-NOT: "-internal-isystem" "[[DIR]]/hip-4.0.0-5f63slrursbrvfe2txrrjkynbsywsob5/include"
 
 // SPACK-SET: InstalledDir: [[DIR:.*]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin
 // SPACK-SET: Found HIP installation: [[DIR]]/hip-4.0.0-abcd, version 4.0.20214-a2917cd
 // SPACK-SET: "-triple" "amdgcn-amd-amdhsa"
-// SPACK-SET-SAME: "-mlink-builtin-bitcode" "[[DIR]]/rocm-device-libs-4.0.0-efgh/amdgcn/bitcode/hip.bc"
+// SPACK-SET-SAME: "-mlink-builtin-bitcode" "[[DIR]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/amdgcn/bitcode/hip.bc"
 // SPACK-SET-SAME: "-internal-isystem" "[[DIR]]/hip-4.0.0-abcd/include"
 
 // SPACK-MISS: InstalledDir: [[DIR:.*]]/llvm-amdgpu-4.0.0-ieagcs7inf7runpyfvepqkurasoglq4z/bin
 // SPACK-MISS-DAG: SPACK package hip-4.0.0 not found at [[DIR]]
-// SPACK-MISS-DAG: SPACK package rocm-device-libs-4.0.0 not found at [[DIR]]
 // SPACK-MISS-NOT: Found HIP installation: [[DIR]]/hip-4.0.0-5f63slrursbrvfe2txrrjkynbsywsob5, version 4.0.20214-a2917cd
-// SPACK-MISS-NOT: "-mlink-builtin-bitcode" "[[DIR]]/rocm-device-libs-4.0.0-6wnyzz4hgl3hr7uswasnagt7j2adctbs/amdgcn/bitcode/hip.bc"
 // SPACK-MISS-NOT: "-internal-isystem" "[[DIR]]/hip-4.0.0-5f63slrursbrvfe2txrrjkynbsywsob5/include"
 
 // SPACK-MISS-SILENT-NOT: SPACK package hip-{{.*}} not found at
-// SPACK-MISS-SILENT-NOT: SPACK package rocm-device-libs-{{.*}} not found at
 // SPACK-MISS-SILENT-NOT: Found HIP installation