AMDGPU: Search for new ROCm bitcode library structure

[arsenm]

Summary

The current install situation is a mess, but I'm working on fixing
it. Search for the target layout instead of one of the N options that
exist today.

[arsenm]

Forgot to commit part

[arsenm]

ping

[yaxunl]

If we really want to do this, device lib change and hipcc change need to be ready. Since once this is committed without corresponding device lib and hipcc change, HIP will break.

On ROCM, device lib is currently installed at /opt/rocm/lib with extension .amdgcn.bc. This change requires device libs to be installed to /opt/rocm/amdgcn/bitcode and with extension .bc (drop amdgcn in extension). Also in hipcc -device-lib-path option needs to point to /opt/rocm/amdgcn/bitcode or dropped.

On Windows, device lib Makefiles need to be changed to drop amdgcn in extension.

Another issue: is /opt/rocm/amdgcn/bitcode the right installation path for device lib? Shouldn't it be /opt/rocm/lib/amdgcn/bitcode or /opt/rocm/lib/amdgcn ?

[arsenm]

In D77885#1983708, @yaxunl wrote:

If we really want to do this, device lib change and hipcc change need to be ready. Since once this is committed without corresponding device lib and hipcc change, HIP will break.

The library change is also waiting for review

On ROCM, device lib is currently installed at /opt/rocm/lib with extension .amdgcn.bc. This change requires device libs to be installed to /opt/rocm/amdgcn/bitcode and with extension .bc (drop amdgcn in extension). Also in hipcc -device-lib-path option needs to point to /opt/rocm/amdgcn/bitcode or dropped.

HIP is fixed/merged with the CL handling in D76957

On Windows, device lib Makefiles need to be changed to drop amdgcn in extension.

Another issue: is /opt/rocm/amdgcn/bitcode the right installation path for device lib? Shouldn't it be /opt/rocm/lib/amdgcn/bitcode or /opt/rocm/lib/amdgcn ?

These aren't host libraries, so I don't think these belong under /lib. CUDA installs the functionally equivalent in /usr/local/cuda/nvvm/libdevice/*.bc

[yaxunl]

clang/lib/Driver/ToolChains/AMDGPU.cpp

75

with your change, the structure becomes

${ROCM_ROOT}/amdgcn/bitcode/*.bc

Does the toolchain support ${OPENCL_ROOT} ?

[arsenm]

clang/lib/Driver/ToolChains/AMDGPU.cpp

75

This is just pseudocode and no environment variable is read. This corresponds to --rocm-path

[arsenm]

Device library change submitted. 123bee602a260150ff55c74287f583a67ee78f36

clang/lib/Driver/ToolChains/AMDGPU.cpp

RocmInstallationDetector::RocmInstallationDetector(
 
   bool NoBuiltinLibs = Args.hasArg(options::OPT_nogpulib);
 
   for (const auto &Candidate : Candidates) {
     InstallPath = Candidate.Path;
     if (InstallPath.empty() || !D.getVFS().exists(InstallPath))
       continue;
 
-    // FIXME: The install path situation is a real mess.
-
-    // For a cmake install, these are placed directly in
-    // ${INSTALL_PREFIX}/lib
-
-    // In the separate OpenCL builds, the bitcode libraries are placed in
-    // ${OPENCL_ROOT}/lib/x86_64/bitcode/*
-
-    // For the rocm installed packages, these are placed at
-    // /opt/rocm/opencl/lib/x86_64/bitcode
-
-    // An additional copy is installed, in scattered locations between
-    // /opt/rocm/hcc/rocdl/oclc
-    // /opt/rocm/hcc/rocdl/ockl
-    // /opt/rocm/hcc/rocdl/lib
+    // The install path situation in old versions of ROCm is a real mess, and
+    // use a different install layout. Multiple copies of the device libraries
+    // exist for each frontend project, and differ depending on which build
+    // system produced the packages. Standalone OpenCL builds also have a
+    // different directory structure from the ROCm OpenCL package.
     //
-    // Yet another complete set is installed to
-    // /opt/rocm/hcc/rocdl/lib
-
-    // For now just recognize the opencl package layout.
+    // The desired structure is (${ROCM_ROOT} or
+    // ${OPENCL_ROOT})/amdgcn/bitcode/*, so try to detect this layout.
 
     // BinPath = InstallPath + "/bin";
     llvm::sys::path::append(IncludePath, InstallPath, "include");
-    llvm::sys::path::append(LibDevicePath, InstallPath, "lib");
+    llvm::sys::path::append(LibDevicePath, InstallPath, "amdgcn", "bitcode");
 
     auto &FS = D.getVFS();
 
     // We don't need the include path for OpenCL, since clang already ships with
     // the default header.
 
     bool CheckLibDevice = (!NoBuiltinLibs || Candidate.StrictChecking);
     if (CheckLibDevice && !FS.exists(LibDevicePath))
       continue;
 
-    const StringRef Suffix(".amdgcn.bc");
+    const StringRef Suffix(".bc");
 
     std::error_code EC;
     for (llvm::sys::fs::directory_iterator LI(LibDevicePath, EC), LE;
          !EC && LI != LE; LI = LI.increment(EC)) {
       StringRef FilePath = LI->path();
       StringRef FileName = llvm::sys::path::filename(FilePath);
       if (!FileName.endswith(Suffix))
         continue;

clang/test/Driver/rocm-device-libs.cl

 // RUN:   -x cl -mcpu=gfx803  -mno-wavefrontsize64  \
 // RUN:   --rocm-path=%S/Inputs/rocm-device-libs    \
 // RUN:   %S/opencl.cl \
 // RUN: 2>&1 | FileCheck -dump-input-on-failure --check-prefixes=COMMMON,GFX803,WAVE64 %s
 
 
 
 // COMMON: "-triple" "amdgcn-amd-amdhsa"
-// COMMON-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/opencl.amdgcn.bc"
-// COMMON-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/ocml.amdgcn.bc"
-// COMMON-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/ockl.amdgcn.bc"
+// COMMON-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/opencl.bc"
+// COMMON-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/ocml.bc"
+// COMMON-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/ockl.bc"
 
-// GFX900-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_daz_opt_off.amdgcn.bc"
-// GFX803-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_daz_opt_on.amdgcn.bc"
-// GFX700-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_daz_opt_on.amdgcn.bc"
-// COMMON-DAZ-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_daz_opt_on.amdgcn.bc"
+// GFX900-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_daz_opt_off.bc"
+// GFX803-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_daz_opt_on.bc"
+// GFX700-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_daz_opt_on.bc"
+// COMMON-DAZ-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_daz_opt_on.bc"
 
 
-// COMMON-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_unsafe_math_off.amdgcn.bc"
-// COMMON-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_finite_only_off.amdgcn.bc"
-// COMMON-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_correctly_rounded_sqrt_off.amdgcn.bc"
+// COMMON-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_unsafe_math_off.bc"
+// COMMON-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_finite_only_off.bc"
+// COMMON-DEFAULT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_correctly_rounded_sqrt_off.bc"
 
 
-// COMMON-FINITE-ONLY-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_unsafe_math_off.amdgcn.bc"
-// COMMON-FINITE-ONLY-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_finite_only_on.amdgcn.bc"
-// COMMON-FINITE-ONLY-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_correctly_rounded_sqrt_off.amdgcn.bc"
+// COMMON-FINITE-ONLY-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_unsafe_math_off.bc"
+// COMMON-FINITE-ONLY-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_finite_only_on.bc"
+// COMMON-FINITE-ONLY-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_correctly_rounded_sqrt_off.bc"
 
 
-// COMMON-CORRECT-SQRT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_unsafe_math_off.amdgcn.bc"
-// COMMON-CORRECT-SQRT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_finite_only_off.amdgcn.bc"
-// COMMON-CORRECT-SQRT-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_correctly_rounded_sqrt_on.amdgcn.bc"
+// COMMON-CORRECT-SQRT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_unsafe_math_off.bc"
+// COMMON-CORRECT-SQRT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_finite_only_off.bc"
+// COMMON-CORRECT-SQRT-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc"
 
 
-// COMMON-FAST-RELAXED-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_unsafe_math_on.amdgcn.bc"
-// COMMON-FAST-RELAXED-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_finite_only_on.amdgcn.bc"
-// COMMON-FAST-RELAXED-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_correctly_rounded_sqrt_off.amdgcn.bc"
+// COMMON-FAST-RELAXED-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_unsafe_math_on.bc"
+// COMMON-FAST-RELAXED-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_finite_only_on.bc"
+// COMMON-FAST-RELAXED-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_correctly_rounded_sqrt_off.bc"
 
 
-// COMMON-UNSAFE-MATH-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_unsafe_math_on.amdgcn.bc"
-// COMMON-UNSAFE-MATH-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_finite_only_off.amdgcn.bc"
-// COMMON-UNSAFE-MATH-SAME: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_correctly_rounded_sqrt_off.amdgcn.bc"
+// COMMON-UNSAFE-MATH-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_unsafe_math_on.bc"
+// COMMON-UNSAFE-MATH-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_finite_only_off.bc"
+// COMMON-UNSAFE-MATH-SAME: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_correctly_rounded_sqrt_off.bc"
 
-// WAVE64: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_wavefrontsize64_on.amdgcn.bc"
-// WAVE32: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_wavefrontsize64_off.amdgcn.bc"
+// WAVE64: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_wavefrontsize64_on.bc"
+// WAVE32: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_wavefrontsize64_off.bc"
 
 
-// GFX900: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_isa_version_900.amdgcn.bc"
-// GFX803: "-mlink-builtin-bitcode" "{{.*}}/lib/oclc_isa_version_803.amdgcn.bc"
+// GFX900: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_isa_version_900.bc"
+// GFX803: "-mlink-builtin-bitcode" "{{.*}}/amdgcn/bitcode/oclc_isa_version_803.bc"