Page MenuHomePhabricator

[HIPSPV] Convert HIP kernels to SPIR-V kernels

Authored by linjamaki on Sep 15 2021, 3:06 AM.



This patch translates HIP kernels to SPIR-V kernels when the HIP
compilation mode is targeting SPIR-S. This involves:

  • Setting Cuda calling convention to CC_OpenCLKernel (which maps to SPIR_KERNEL in LLVM IR later on).
  • Coercing pointer arguments with default address space (AS) qualifier to CrossWorkGroup AS (__global in OpenCL). HIPSPV's device code is ultimately SPIR-V for OpenCL execution environment (as starter/default) where Generic or Function (OpenCL's private) is not supported as storage class for kernel pointer types. This leaves the CrossWorkGroup to be the only reasonable choice for HIP buffers.

Diff Detail

Unit TestsFailed

50 msx64 debian > ORC-x86_64-linux.TestCases/Linux/x86-64::trivial-cxa-atexit.S
Script: -- : 'RUN: at line 3'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang -m64 -c -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/orc/X86_64LinuxConfig/TestCases/Linux/x86-64/Output/trivial-cxa-atexit.S.tmp /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/orc/TestCases/Linux/x86-64/trivial-cxa-atexit.S
70 msx64 debian > ORC-x86_64-linux.TestCases/Linux/x86-64::trivial-static-initializer.S
Script: -- : 'RUN: at line 7'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang -m64 -c -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/orc/X86_64LinuxConfig/TestCases/Linux/x86-64/Output/trivial-static-initializer.S.tmp /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/orc/TestCases/Linux/x86-64/trivial-static-initializer.S
70 msx64 debian > ORC-x86_64-linux.TestCases/Linux/x86-64::trivial-tls.S
Script: -- : 'RUN: at line 1'; /var/lib/buildkite-agent/builds/llvm-project/build/./bin/clang -m64 -c -o /var/lib/buildkite-agent/builds/llvm-project/build/projects/compiler-rt/test/orc/X86_64LinuxConfig/TestCases/Linux/x86-64/Output/trivial-tls.S.tmp /var/lib/buildkite-agent/builds/llvm-project/compiler-rt/test/orc/TestCases/Linux/x86-64/trivial-tls.S

Event Timeline

linjamaki created this revision.Sep 15 2021, 3:06 AM
linjamaki edited the summary of this revision. (Show Details)Mon, Sep 20, 2:16 AM
linjamaki published this revision for review.Mon, Sep 20, 2:20 AM
linjamaki added reviewers: Anastasia, bader.
Herald added a project: Restricted Project. · View Herald TranscriptMon, Sep 20, 2:20 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
yaxunl accepted this revision.Mon, Sep 20, 9:17 AM

LGTM. Thanks!

This revision is now accepted and ready to land.Mon, Sep 20, 9:17 AM
Anastasia added inline comments.Tue, Sep 21, 2:50 AM

It feels like this needs to be in SPIRVABIInfo or something? Or can this be generalized to both - SPIR and SPIR-V?


Can you explain why this mapping is needed? We already have an address space map to perform the mapping of address spaces b/w language and target. It would be good if we don't replicate similar logic in too many places.


Again this feels like we need SPIRVTargetCodeGenInfo unless the logic is generalizable to both.

linjamaki added inline comments.Tue, Sep 21, 10:44 PM

A comment was added in D109144 to state that the SPIRABIInfo is an ABI implementation for both the SPIR and SPIR-V. For now, there is not much difference between SPIR and SPIR-V for this class. Would it be satisfactory if the class is renamed to something more general (like CommonSPIRABIInfo)?


HIP does not require address space qualifiers on kernel pointer arguments (e.g. see hipspv-kernel.cpp) nor there are AS qualifiers that can be placed on them. With the default logic, provided by SPIRVTargetInfo’s address space map, the kernel pointer arguments get converted to generic pointers which are not allowed by the OpenCL SPIR-V Environment Specification.

Anastasia added inline comments.Wed, Sep 22, 2:34 AM

It might be reasonable to rename indeed or we can just amend its documentation to clarify what it is used for both...

However if you need to specialize different logic I think the conventional way in clang would be to create a subclass...

This is related to a wider discussion that is ongoing about the best way to reuse SPIR for SPIR-V and their evolution... CCing @bader here too.


I feel that it is the same for SYCL... It might be good to check with @bader whether there is already a way to handle this that can be reused for HIP...

bader added inline comments.Wed, Sep 22, 4:08 AM

We need to do similar transformation for SYCL, but it's not exactly the same.
For SYCL kernels, which represented as function objects, compiler generates SPIR kernel function and fixes up the address space for pointer arguments in compiler generated declaration. For more details, see the description of and handlePointerType function code in clang/lib/Sema/SemaSYCL.cpp of this review request (lines 848-876). As address space is fixed in Sema, it works for all targets SYCL currently supports SPIR, NVPTX and AMDGPU.

If I understand it correctly, we are trying to do minimal amount of work for convert HIP kernel function to SPIR kernel function, i.e. fix calling convention and address spaces.
Are these two fixes enough or we need more fixes to enable more sophisticated kernels?

linjamaki added inline comments.Thu, Sep 23, 2:43 AM

This patch and D108621 covers only the calling convention and the address space aspects in HIP->SPIR-V conversion. There are still various HIP features [1] which need to be expanded or emulated afterwards. A fully linked device program is needed before the fixes can be applied, so these fixes won’t be implemented at Sema nor CodeGen which operate per translation unit. The full-program fixes are applied by the HIPSPV tool chain by applying LLVM passes provided by a HIPSPV runtime [2]. For the time being, we are not seeing a need to specialize SPIR target infos further.

[1]: “HIP code expansion” section of the “[RFC][HIPSPV] Emitting HIP device code as SPIR-V”
[2]: clang/lib/Driver/ToolChains/HIPSPV.cpp:constructEmitSpirvCommand() @

linjamaki updated this revision to Diff 375233.Mon, Sep 27, 6:04 AM

Put HIPSPV logic into new SPIRV{ABI,TargetCodeGen}Info subclassed from

linjamaki marked 2 inline comments as done.Mon, Sep 27, 6:06 AM