[VE] fastcc and vreg-to-vreg copy

[simoll]

Summary

This defines a 'fastcc' for the VE target and implements vreg-to-vreg copy for parameter passing. The 'fastcc' extends the standard CC for SX-Aurora with register passing of vector-typed parameters and return values.

[kaz7]

It is better to make a different NFC patch for modifications on the mechanism to retrieve the calling convention. And need a regression test from caller side.

llvm/lib/Target/VE/VEISelLowering.cpp

43

These modifications are independent from passing vector registers on registers patch.

[simoll]

• Rebased onto commited CC switching patch D90911
• added call tests

[simoll]

llvm/lib/Target/VE/VECallingConv.td

165–167

I'll commit the CSR_preserve_all change standalone with D90912

[simoll]

• Rebased.
• NFC.

[simoll]

• Rebasing onto landed D90833 and adapted tests.

[kaz7]

Thank you for adding caller side test. However, it is not possible to examine these tests whether fastcc is implemented correctly or not, unfortunately. How about upstream vector load and vector store first, then upstream this with caller side tests something like load and call, and return and store? After that, upstream vector copy with callee side tests.

[simoll]

Here is why the patches are structured this way relying on fastcc:
My actual goal is to put a patch on phabricator that demonstrates the hpce/develop isel mechanism for vector instructions. This should come with single vector instruction tests (one vector inst emitted per test function).

Using fastcc we can write single vector instruction tests that do not require any vld nor vst nor vor, if we order the vector operands in IR to match the vreg order of the CC.
However, to emit any valid vector instruction code in the first place, i need LVLGen upstream (D90850) and to upstream LVLGen we need a test for it. The VOR required for fastcc is such a test and it does not require any isel patterns.

This current patch (D90842) is just a stub for fastcc to make D90850 smaller.

[simoll]

Merged into one fastcc vreg-to-vreg patch

[kaz7]

Why is there no caller_varg_pass_v0 or caller_varg_pass_v0_and_v1 tests?

[simoll]

In D90842#2394965, @kaz7 wrote:

Why is there no caller_varg_pass_v0 or caller_varg_pass_v0_and_v1 tests?

fastcc does not support vargs. Citing the LLVM LangRef :

“fastcc” - The fast calling convention
[..]
This calling convention does not support varargs and requires the prototype of all callees to exactly match the prototype of the function definition.

[kaz7]

I don't say varargs. I meant vector arguments. This patch define fastcc and declare how to pass vector arguments and vector return values using vector registers. Why don't you make tests to check function calls using vector registers correctly? Your tests don't show %v0 or %v1 in checking assembly code, so it is not possible to check the correctness, IMO.

[simoll]

• added vector param tests that trigger explicit vreg copies.

[kaz7]

Thank you for adding acceptable tests. LGTM.

llvm/lib/Target/VE/VECallingConv.td

CCIfType<[i64, f64],
            CCAssignToReg<[SX0, SX1, SX2, SX3, SX4, SX5, SX6, SX7]>>,
 
   // long double --> pair of generic 64 bit registers
   CCIfType<[f128],
            CCAssignToRegWithShadow<[Q0, Q1, Q2, Q3],
                                    [SX0, SX1, SX3, SX5]>>,
 ]>;
 
+///// Custom fastcc /////
+//
+// This passes vector params and return values in registers.  Scalar values are
+// handled conforming to the standard cc.
+def CC_VE_Fast : CallingConv<[
+  // vector --> generic vector registers
+  CCIfType<[v2i32,   v2i64,   v2f32,   v2f64,
+            v4i32,   v4i64,   v4f32,   v4f64,
+            v8i32,   v8i64,   v8f32,   v8f64,
+            v16i32,  v16i64,  v16f32,  v16f64,
+            v32i32,  v32i64,  v32f32,  v32f64,
+            v64i32,  v64i64,  v64f32,  v64f64,
+            v128i32, v128i64, v128f32, v128f64,
+            v256i32, v256f32, v256i64, v256f64],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+  // TODO: make this conditional on packed mode
+  CCIfType<[v512i32, v512f32],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+
+  // vector mask --> generic vector mask registers
+  CCIfType<[v256i1],
+           CCAssignToReg<[VM1, VM2, VM3, VM4, VM5, VM6, VM7]>>,
+
+  // pair of vector mask --> generic vector mask registers
+  CCIfType<[v512i1],
+           CCAssignToRegWithShadow<[VMP1, VMP2, VMP3],
+                                   [VM1, VM1, VM3]>>,
+
+  // Follow the standard C CC for scalars.
+  CCDelegateTo<CC_VE_C>
+]>;
+
+def RetCC_VE_Fast : CallingConv<[
+  // vector --> generic vector registers
+  CCIfType<[v2i32,   v2i64,   v2f32,   v2f64,
+            v4i32,   v4i64,   v4f32,   v4f64,
+            v8i32,   v8i64,   v8f32,   v8f64,
+            v16i32,  v16i64,  v16f32,  v16f64,
+            v32i32,  v32i64,  v32f32,  v32f64,
+            v64i32,  v64i64,  v64f32,  v64f64,
+            v128i32, v128i64, v128f32, v128f64,
+            v256i32, v256f32, v256i64, v256f64],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+  // TODO: make this conditional on packed mode
+  CCIfType<[v512i32, v512f32],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+
+  // vector mask --> generic vector mask registers
+  CCIfType<[v256i1],
+           CCAssignToReg<[VM1, VM2, VM3, VM4, VM5, VM6, VM7]>>,
+
+  // pair of vector mask --> generic vector mask registers
+  CCIfType<[v512i1],
+           CCAssignToRegWithShadow<[VMP1, VMP2, VMP3],
+                                   [VM1, VM1, VM3]>>,
+
+  // Follow the standard C CC for scalars.
+  CCDelegateTo<RetCC_VE_C>
+]>;
+
 // Callee-saved registers
 def CSR : CalleeSavedRegs<(add (sequence "SX%u", 18, 33))>;
 def CSR_NoRegs : CalleeSavedRegs<(add)>;
 
 // PreserveAll (clobbers s62,s63) - used for ve_grow_stack
 def CSR_preserve_all : CalleeSavedRegs<(add (sequence "SX%u", 0, 61),
                                             (sequence "V%u", 0, 63),
                                             (sequence "VM%u", 1, 15))>;

llvm/lib/Target/VE/VEISelLowering.cpp

 #define DEBUG_TYPE "ve-lower"
 
 //===----------------------------------------------------------------------===//
 // Calling Convention Implementation
 //===----------------------------------------------------------------------===//
 
 #include "VEGenCallingConv.inc"
 
 CCAssignFn *getReturnCC(CallingConv::ID CallConv) {
   switch (CallConv) {
   default:
     return RetCC_VE_C;
+  case CallingConv::Fast:
+    return RetCC_VE_Fast;
   }
 }
 
 CCAssignFn *getParamCC(CallingConv::ID CallConv, bool IsVarArg) {
   if (IsVarArg)
     return CC_VE2;
   switch (CallConv) {
   default:
     return CC_VE_C;
+  case CallingConv::Fast:
+    return CC_VE_Fast;
   }
 }
 
 bool VETargetLowering::CanLowerReturn(
     CallingConv::ID CallConv, MachineFunction &MF, bool IsVarArg,
     const SmallVectorImpl<ISD::OutputArg> &Outs, LLVMContext &Context) const {
   CCAssignFn *RetCC = getReturnCC(CallConv);
   SmallVector<CCValAssign, 16> RVLocs;

llvm/lib/Target/VE/VERegisterInfo.cpp

 #include "VEGenRegisterInfo.inc"
 
 // VE uses %s10 == %lp to keep return address
 VERegisterInfo::VERegisterInfo() : VEGenRegisterInfo(VE::SX10) {}
 
 const MCPhysReg *
 VERegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   switch (MF->getFunction().getCallingConv()) {
+  case CallingConv::Fast:
+    // Being explicit (same as standard CC).
   default:
     return CSR_SaveList;
   case CallingConv::PreserveAll:
     return CSR_preserve_all_SaveList;
   }
 }
 
 const uint32_t *VERegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                                      CallingConv::ID CC) const {
   switch (CC) {
+  case CallingConv::Fast:
+    // Being explicit (same as standard CC).
   default:
     return CSR_RegMask;
   case CallingConv::PreserveAll:
     return CSR_preserve_all_RegMask;
   }
 }
 
 const uint32_t *VERegisterInfo::getNoPreservedMask() const {

llvm/test/CodeGen/VE/Vector/call_fastcc.ll

+; RUN: llc < %s -mtriple=ve-unknown-unknown -mattr=+vpu | FileCheck %s
+
+
+declare i32 @sample_add(i32, i32)
+declare i32 @stack_callee_int(i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)
+declare i32 @stack_callee_int_szext(i1 signext, i8 zeroext, i32, i32, i32, i32, i32, i32, i16 zeroext, i8 signext)
+declare float @stack_callee_float(float, float, float, float, float, float, float, float, float, float)
+declare void @test(i64)
+
+; Scalar argument passing must not change (same tests as in VE/Scalar/call.ll below - this time with +vpu)
+
+define fastcc i32 @sample_call() {
+; CHECK-LABEL: sample_call:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    lea %s0, sample_add@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, sample_add@hi(, %s0)
+; CHECK-NEXT:    or %s0, 1, (0)1
+; CHECK-NEXT:    or %s1, 2, (0)1
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %r = tail call i32 @sample_add(i32 1, i32 2)
+  ret i32 %r
+}
+
+define fastcc i32 @stack_call_int() {
+; CHECK-LABEL: stack_call_int:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    or %s0, 10, (0)1
+; CHECK-NEXT:    st %s0, 248(, %s11)
+; CHECK-NEXT:    or %s34, 9, (0)1
+; CHECK-NEXT:    lea %s0, stack_callee_int@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, stack_callee_int@hi(, %s0)
+; CHECK-NEXT:    or %s0, 1, (0)1
+; CHECK-NEXT:    or %s1, 2, (0)1
+; CHECK-NEXT:    or %s2, 3, (0)1
+; CHECK-NEXT:    or %s3, 4, (0)1
+; CHECK-NEXT:    or %s4, 5, (0)1
+; CHECK-NEXT:    or %s5, 6, (0)1
+; CHECK-NEXT:    or %s6, 7, (0)1
+; CHECK-NEXT:    or %s7, 8, (0)1
+; CHECK-NEXT:    st %s34, 240(, %s11)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %r = tail call i32 @stack_callee_int(i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10)
+  ret i32 %r
+}
+
+define fastcc i32 @stack_call_int_szext() {
+; CHECK-LABEL: stack_call_int_szext:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    or %s0, -1, (0)1
+; CHECK-NEXT:    st %s0, 248(, %s11)
+; CHECK-NEXT:    lea %s34, 65535
+; CHECK-NEXT:    lea %s1, stack_callee_int_szext@lo
+; CHECK-NEXT:    and %s1, %s1, (32)0
+; CHECK-NEXT:    lea.sl %s12, stack_callee_int_szext@hi(, %s1)
+; CHECK-NEXT:    lea %s1, 255
+; CHECK-NEXT:    or %s2, 3, (0)1
+; CHECK-NEXT:    or %s3, 4, (0)1
+; CHECK-NEXT:    or %s4, 5, (0)1
+; CHECK-NEXT:    or %s5, 6, (0)1
+; CHECK-NEXT:    or %s6, 7, (0)1
+; CHECK-NEXT:    or %s7, 8, (0)1
+; CHECK-NEXT:    st %s34, 240(, %s11)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %r = tail call i32 @stack_callee_int_szext(i1 -1, i8 -1, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i16 -1, i8 -1)
+  ret i32 %r
+}
+
+define fastcc float @stack_call_float() {
+; CHECK-LABEL: stack_call_float:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    lea.sl %s0, 1092616192
+; CHECK-NEXT:    st %s0, 248(, %s11)
+; CHECK-NEXT:    lea.sl %s34, 1091567616
+; CHECK-NEXT:    lea.sl %s0, 1065353216
+; CHECK-NEXT:    lea.sl %s1, 1073741824
+; CHECK-NEXT:    lea.sl %s2, 1077936128
+; CHECK-NEXT:    lea.sl %s3, 1082130432
+; CHECK-NEXT:    lea.sl %s4, 1084227584
+; CHECK-NEXT:    lea.sl %s5, 1086324736
+; CHECK-NEXT:    lea.sl %s6, 1088421888
+; CHECK-NEXT:    lea.sl %s7, 1090519040
+; CHECK-NEXT:    lea %s35, stack_callee_float@lo
+; CHECK-NEXT:    and %s35, %s35, (32)0
+; CHECK-NEXT:    lea.sl %s12, stack_callee_float@hi(, %s35)
+; CHECK-NEXT:    st %s34, 240(, %s11)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %r = tail call float @stack_callee_float(float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0, float 9.0, float 10.0)
+  ret float %r
+}
+
+define fastcc float @stack_call_float2(float %p0) {
+; CHECK-LABEL: stack_call_float2:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    st %s0, 248(, %s11)
+; CHECK-NEXT:    lea %s1, stack_callee_float@lo
+; CHECK-NEXT:    and %s1, %s1, (32)0
+; CHECK-NEXT:    lea.sl %s12, stack_callee_float@hi(, %s1)
+; CHECK-NEXT:    st %s0, 240(, %s11)
+; CHECK-NEXT:    or %s1, 0, %s0
+; CHECK-NEXT:    or %s2, 0, %s0
+; CHECK-NEXT:    or %s3, 0, %s0
+; CHECK-NEXT:    or %s4, 0, %s0
+; CHECK-NEXT:    or %s5, 0, %s0
+; CHECK-NEXT:    or %s6, 0, %s0
+; CHECK-NEXT:    or %s7, 0, %s0
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %r = tail call float @stack_callee_float(float %p0, float %p0, float %p0, float %p0, float %p0, float %p0, float %p0, float %p0, float %p0, float %p0)
+  ret float %r
+}
+
+; Vector argument passing (fastcc feature)
+; 
+declare fastcc <256 x i32> @get_v256i32()
+declare fastcc void @vsample_v(<256 x i32>)
+declare fastcc void @vsample_iv(i32, <256 x i32>)
+
+define void @caller_vret() {
+; CHECK:       caller_vret:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    lea %s0, get_v256i32@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, get_v256i32@hi(, %s0)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %r = tail call fastcc <256 x i32> @get_v256i32()
+  ret void
+}
+
+define void @caller_vret_pass_p0() {
+; CHECK-LABEL: caller_vret_pass_p0:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK:         lea %s0, get_v256i32@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, get_v256i32@hi(, %s0)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    lea %s0, vsample_v@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, vsample_v@hi(, %s0)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    or %s11, 0, %s9
+  %p = tail call fastcc <256 x i32> @get_v256i32()
+  call fastcc void @vsample_v(<256 x i32> %p)
+  ret void
+}
+
+define void @caller_vret_pass_p1(i32 %s) {
+; CHECK-LABEL: caller_vret_pass_p1:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK:         or %s18, 0, %s0
+; CHECK-NEXT:    lea %s0, get_v256i32@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, get_v256i32@hi(, %s0)
+; CHECK-NEXT:    bsic %s10, (, %s12)
+; CHECK-NEXT:    lea %s0, vsample_iv@lo
+; CHECK-NEXT:    and %s0, %s0, (32)0
+; CHECK-NEXT:    lea.sl %s12, vsample_iv@hi(, %s0)
+; CHECK-NEXT:    or %s0, 0, %s18
+; CHECK-NEXT:    bsic %s10, (, %s12)
+  %p = tail call fastcc <256 x i32> @get_v256i32()
+  call fastcc void @vsample_iv(i32 %s, <256 x i32> %p)
+  ret void
+}

llvm/test/CodeGen/VE/Vector/callee_fastcc.ll

+; RUN: llc < %s -mtriple=ve-unknown-unknown -mattr=+vpu | FileCheck %s
+
+; Scalar argument passing must not change (same tests as in VE/Scalar/callee.ll below - this time with +vpu)
+
+define fastcc i32 @stack_stack_arg_i32_r9(i1 %0, i8 %1, i16 %2, i32 %3, i64 %4, i32 %5, i32 %6, i32 %7, i32 %8, i32 %9) {
+; CHECK-LABEL: stack_stack_arg_i32_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldl.sx %s0, 424(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret i32 %9
+}
+
+define fastcc i64 @stack_stack_arg_i64_r9(i1 %0, i8 %1, i16 %2, i32 %3, i64 %4, i64 %5, i64 %6, i64 %7, i64 %8, i64 %9) {
+; CHECK-LABEL: stack_stack_arg_i64_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ld %s0, 424(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret i64 %9
+}
+
+define fastcc float @stack_stack_arg_f32_r9(float %p0, float %p1, float %p2, float %p3, float %p4, float %p5, float %p6, float %p7, float %s0, float %s1) {
+; CHECK-LABEL: stack_stack_arg_f32_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldu %s0, 428(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret float %s1
+}
+
+define fastcc i32 @stack_stack_arg_i32f32_r8(i32 %p0, float %p1, i32 %p2, float %p3, i32 %p4, float %p5, i32 %p6, float %p7, i32 %s0, float %s1) {
+; CHECK-LABEL: stack_stack_arg_i32f32_r8:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldl.sx %s0, 416(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret i32 %s0
+}
+
+define fastcc float @stack_stack_arg_i32f32_r9(i32 %p0, float %p1, i32 %p2, float %p3, i32 %p4, float %p5, i32 %p6, float %p7, i32 %s0, float %s1) {
+; CHECK-LABEL: stack_stack_arg_i32f32_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldu %s0, 428(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret float %s1
+}
+
+; Vector argument passing (fastcc feature)
+
+; v0-to-v0 passthrough case without vreg copy.
+define fastcc <256 x i32> @vreg_arg_v256i32_r0(<256 x i32> %p0) {
+; CHECK-LABEL: vreg_arg_v256i32_r0:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret <256 x i32> %p0
+}
+
+; TODO: Uncomment tests when vreg-to-vreg copy is upstream.
+; define fastcc <256 x i32> @vreg_arg_v256i32_r1(<256 x i32> %p0, <256 x i32> %p1) {
+;   ret <256 x i32> %p1
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r2(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2) {
+;   ret <256 x i32> %p2
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r3(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3) {
+;   ret <256 x i32> %p3
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r4(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4) {
+;   ret <256 x i32> %p4
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r5(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5) {
+;   ret <256 x i32> %p5
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r6(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5, <256 x i32> %p6) {
+;   ret <256 x i32> %p6
+; }
+; 
+; TODO: Uncomment test when vector loads are upstream (vreg stack passing).
+; define <256 x i32> @vreg_arg_v256i32_r7(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5, <256 x i32> %p6, <256 x i32> %p7) {
+;   ret <256 x i32> %p7
+; }
+; define <256 x i32> @vreg_arg_v256i32_r8(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5, <256 x i32> %p6, <256 x i32> %p7, <256 x i32> %p8) {
+;   ret <256 x i32> %p8
+; }