Page MenuHomePhabricator

[HIP] Fix spack HIP device lib detection
ClosedPublic

Authored by yaxunl on May 27 2021, 12:59 PM.

Details

Summary

spack HIP device library is installed at amdgcn directory under llvm/clang
directory.

This patch fixes detection of HIP device library for spack.

Diff Detail

Event Timeline

yaxunl created this revision.May 27 2021, 12:59 PM
yaxunl requested review of this revision.May 27 2021, 12:59 PM
tra accepted this revision.Jun 1 2021, 11:36 AM

Context for the changes in this patch: https://reviews.llvm.org/D97340#2775477

LGTM syntax-wise. I can't say whether it's the right change for spack. I'll let @haampie make the call on the functionality.

This revision is now accepted and ready to land.Jun 1 2021, 11:36 AM
haampie added a comment.EditedJun 3 2021, 6:57 AM

Hi @yaxunl, generally looks good to me. I've just tried this on top of https://github.com/spack/spack/pull/23859 and I'm getting:

$ /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/bin/clang++ -x hip -print-rocm-search-dirs hi.cc 
ROCm installation search path (Spack 4.2.0): /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/lib/clang/12.0.0
ROCm installation search path: /opt/rocm
ROCm installation search path (Spack 4.2.0): /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo
ROCm installation search path: /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/lib/clang/12.0.0
ROCm installation search path: /opt/rocm
clang-12: error: cannot find HIP runtime. Provide its path via --rocm-path, or pass -nogpuinc to build without HIP runtime.
clang-12: error: cannot find HIP runtime. Provide its path via --rocm-path, or pass -nogpuinc to build without HIP runtime.
$ echo /spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0* | xargs -n1
/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0-siong3jqyqtwb62zvauv43eo2zc4eup7
/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0-wqgj4zleuboc3jpqwvvvwz3p4sdzxbpt
/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hip-4.2.0-xg4uyi67oq6tdtegf6h3ppezztr44sh3

So, bitcode is found, hip is not found because of multiple matches.

My suggestion remains to drop the magic that looks for the hip install prefix based on the spack directory layout, since in some cases it may be incorrect (upsteam hip install and an unrelated local hip install would load the wrong hip, or user has a different config for the directory structure).

If you use llvm with amd support, you have to install hip anyways, and when hip is installed, you can just load it with spack, and it will set environment variables that will be sufficient for clang to know where hip is:

$ spack env activate --sh hip
export HSA_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/hsa-rocr-dev-4.2.0-fukddlz3oiwrvylxato7kff2pg6dd5oj;
export ROCM_PATH=/path/to/environment/.spack-env/view;
export LLVM_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo;
export ROCMINFO_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/rocminfo-4.2.0-briz73u35rpzdj3roxlaqlzjc5sw5b67;
export HIP_COMPILER=clang;
export HIP_DEVICE_LIB_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/amdgcn/bitcode;
export HIP_CLANG_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/bin;
export DEVICE_LIB_PATH=/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-amdgpu-4.2.0-7jyccemdxiz7rc7uqcqpxaybxbjm4ndo/amdgcn/bitcode;
export PATH='/path/to/environment/.spack-env/view/bin:...';
export HIP_PATH=/path/to/environment/.spack-env/view;
export HIP_PLATFORM=amd;

this is how it's used:

$ spack env activate hip
$ clang++ -x hip -print-rocm-search-dirs hi.cc 
ROCm installation search path: /path/to/environment/.spack-env/view
ROCm installation search path: /path/to/environment/.spack-env/view
$ ./a.out 
hello world
haampie accepted this revision.Jun 3 2021, 7:00 AM

Accepting this, with the comment that I think it's better to follow up with a patch that reverts autodetecting hip from clang based on the spack directory structure.

yaxunl added a comment.Jun 3 2021, 7:29 AM

Accepting this, with the comment that I think it's better to follow up with a patch that reverts autodetecting hip from clang based on the spack directory structure.

Thanks. will remove auto-detect of Spack HIP package in a separate patch.

Is @tra able to land this? I'm not

This revision was landed with ongoing or failed builds.Jun 4 2021, 6:13 AM
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptJun 4 2021, 6:13 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript