[VE] fastcc and vreg-to-vreg copy

[simoll]

Summary

This defines a 'fastcc' for the VE target and implements vreg-to-vreg copy for parameter passing. The 'fastcc' extends the standard CC for SX-Aurora with register passing of vector-typed parameters and return values.

[kaz7]

It is better to make a different NFC patch for modifications on the mechanism to retrieve the calling convention. And need a regression test from caller side.

llvm/lib/Target/VE/VEISelLowering.cpp

43

These modifications are independent from passing vector registers on registers patch.

[simoll]

• Rebased onto commited CC switching patch D90911
• added call tests

[simoll]

llvm/lib/Target/VE/VECallingConv.td

165–167

I'll commit the CSR_preserve_all change standalone with D90912

[simoll]

• Rebased.
• NFC.

[simoll]

• Rebasing onto landed D90833 and adapted tests.

[kaz7]

Thank you for adding caller side test. However, it is not possible to examine these tests whether fastcc is implemented correctly or not, unfortunately. How about upstream vector load and vector store first, then upstream this with caller side tests something like load and call, and return and store? After that, upstream vector copy with callee side tests.

[simoll]

Here is why the patches are structured this way relying on fastcc:
My actual goal is to put a patch on phabricator that demonstrates the hpce/develop isel mechanism for vector instructions. This should come with single vector instruction tests (one vector inst emitted per test function).

Using fastcc we can write single vector instruction tests that do not require any vld nor vst nor vor, if we order the vector operands in IR to match the vreg order of the CC.
However, to emit any valid vector instruction code in the first place, i need LVLGen upstream (D90850) and to upstream LVLGen we need a test for it. The VOR required for fastcc is such a test and it does not require any isel patterns.

This current patch (D90842) is just a stub for fastcc to make D90850 smaller.

[simoll]

Merged into one fastcc vreg-to-vreg patch

[kaz7]

Why is there no caller_varg_pass_v0 or caller_varg_pass_v0_and_v1 tests?

[simoll]

In D90842#2394965, @kaz7 wrote:

Why is there no caller_varg_pass_v0 or caller_varg_pass_v0_and_v1 tests?

fastcc does not support vargs. Citing the LLVM LangRef :

“fastcc” - The fast calling convention
[..]
This calling convention does not support varargs and requires the prototype of all callees to exactly match the prototype of the function definition.

[kaz7]

I don't say varargs. I meant vector arguments. This patch define fastcc and declare how to pass vector arguments and vector return values using vector registers. Why don't you make tests to check function calls using vector registers correctly? Your tests don't show %v0 or %v1 in checking assembly code, so it is not possible to check the correctness, IMO.

[simoll]

• added vector param tests that trigger explicit vreg copies.

[kaz7]

Thank you for adding acceptable tests. LGTM.

llvm/lib/Target/VE/VECallingConv.td

 def CC_VE_C_Stack: CallingConv<[
   // F128 are assigned to the stack in 16-byte aligned units
   CCIfType<[f128], CCAssignToStackWithShadow<16, 16, [SX7]>>,
 
   // All of the rest are assigned to the stack in 8-byte aligned units.
   CCAssignToStack<0, 8>
 ]>;
 
-def CC_VE : CallingConv<[
+///// C Calling Convention (VE ABI v2.1) /////
+//
+// Reference: https://www.nec.com/en/global/prod/hpc/aurora/document/VE-ABI_v2.1.pdf
+//
+def CC_VE_C : CallingConv<[
   // All arguments get passed in generic registers if there is space.
 
   // Promote i1/i8/i16/i32 arguments to i64.
   CCIfType<[i1, i8, i16, i32], CCPromoteToType<i64>>,
 
   // Convert float arguments to i64 with padding.
   //     63     31   0
   //    +------+------+
def CC_VE_C : CallingConv<[
   CCIfType<[f128],
            CCAssignToRegWithShadow<[Q0, Q1, Q2, Q3],
                                    [SX0, SX1, SX3, SX5]>>,
 
   // Alternatively, they are assigned to the stack in 8-byte aligned units.
   CCDelegateTo<CC_VE_C_Stack>
 ]>;
 
+///// Standard vararg C Calling Convention (VE ABI v2.1) /////
 // All arguments get passed in stack for varargs function or non-prototyped
 // function.
 def CC_VE2 : CallingConv<[
   // Promote i1/i8/i16/i32 arguments to i64.
   CCIfType<[i1, i8, i16, i32], CCPromoteToType<i64>>,
 
   // Convert float arguments to i64 with padding.
   //     63     31   0
   //    +------+------+
   //    | float|   0  |
   //    +------+------+
   CCIfType<[f32], CCBitConvertToType<i64>>,
 
   // F128 are assigned to the stack in 16-byte aligned units
   CCIfType<[f128], CCAssignToStack<16, 16>>,
 
   CCAssignToStack<0, 8>
 ]>;
 
-def RetCC_VE : CallingConv<[
+def RetCC_VE_C : CallingConv<[
   // Promote i1/i8/i16/i32 return values to i64.
   CCIfType<[i1, i8, i16, i32], CCPromoteToType<i64>>,
 
   // Convert float return values to i64 with padding.
   //     63     31   0
   //    +------+------+
   //    | float|   0  |
   //    +------+------+
   CCIfType<[f32], CCBitConvertToType<i64>>,
 
   // bool, char, int, enum, long, long long, float, double
   //     --> generic 64 bit registers
   CCIfType<[i64, f64],
            CCAssignToReg<[SX0, SX1, SX2, SX3, SX4, SX5, SX6, SX7]>>,
 
   // long double --> pair of generic 64 bit registers
   CCIfType<[f128],
            CCAssignToRegWithShadow<[Q0, Q1, Q2, Q3],
                                    [SX0, SX1, SX3, SX5]>>,
 ]>;
 
+///// Custom fastcc /////
+//
+// This passes vector params and return values in registers.  Scalar values are
+// handled conforming to the standard cc.
+def CC_VE_Fast : CallingConv<[
+  // vector --> generic vector registers
+  CCIfType<[v2i32,   v2i64,   v2f32,   v2f64,
+            v4i32,   v4i64,   v4f32,   v4f64,
+            v8i32,   v8i64,   v8f32,   v8f64,
+            v16i32,  v16i64,  v16f32,  v16f64,
+            v32i32,  v32i64,  v32f32,  v32f64,
+            v64i32,  v64i64,  v64f32,  v64f64,
+            v128i32, v128i64, v128f32, v128f64,
+            v256i32, v256f32, v256i64, v256f64],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+  // TODO: make this conditional on packed mode
+  CCIfType<[v512i32, v512f32],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+
+  // vector mask --> generic vector mask registers
+  CCIfType<[v256i1],
+           CCAssignToReg<[VM1, VM2, VM3, VM4, VM5, VM6, VM7]>>,
+
+  // pair of vector mask --> generic vector mask registers
+  CCIfType<[v512i1],
+           CCAssignToRegWithShadow<[VMP1, VMP2, VMP3],
+                                   [VM1, VM1, VM3]>>,
+
+  // Follow the standard C CC for scalars.
+  CCDelegateTo<CC_VE_C>
+]>;
+
+def RetCC_VE_Fast : CallingConv<[
+  // vector --> generic vector registers
+  CCIfType<[v2i32,   v2i64,   v2f32,   v2f64,
+            v4i32,   v4i64,   v4f32,   v4f64,
+            v8i32,   v8i64,   v8f32,   v8f64,
+            v16i32,  v16i64,  v16f32,  v16f64,
+            v32i32,  v32i64,  v32f32,  v32f64,
+            v64i32,  v64i64,  v64f32,  v64f64,
+            v128i32, v128i64, v128f32, v128f64,
+            v256i32, v256f32, v256i64, v256f64],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+  // TODO: make this conditional on packed mode
+  CCIfType<[v512i32, v512f32],
+           CCAssignToReg<[V0, V1, V2, V3, V4, V5, V6, V7]>>,
+
+  // vector mask --> generic vector mask registers
+  CCIfType<[v256i1],
+           CCAssignToReg<[VM1, VM2, VM3, VM4, VM5, VM6, VM7]>>,
+
+  // pair of vector mask --> generic vector mask registers
+  CCIfType<[v512i1],
+           CCAssignToRegWithShadow<[VMP1, VMP2, VMP3],
+                                   [VM1, VM1, VM3]>>,
+
+  // Follow the standard C CC for scalars.
+  CCDelegateTo<RetCC_VE_C>
+]>;
+
 // Callee-saved registers
 def CSR : CalleeSavedRegs<(add (sequence "SX%u", 18, 33))>;
 def CSR_NoRegs : CalleeSavedRegs<(add)>;
 
 // PreserveAll (clobbers s62,s63) - used for ve_grow_stack
-def CSR_preserve_all : CalleeSavedRegs<(add (sequence "SX%u", 0, 61))>;
+def CSR_preserve_all : CalleeSavedRegs<(add (sequence "SX%u", 0, 61),
+                                            (sequence "V%u", 0, 63),
+                                            (sequence "VM%u", 1, 15))>;

llvm/lib/Target/VE/VEISelLowering.cpp

 #define DEBUG_TYPE "ve-lower"
 
 //===----------------------------------------------------------------------===//
 // Calling Convention Implementation
 //===----------------------------------------------------------------------===//
 
 #include "VEGenCallingConv.inc"
 
+CCAssignFn *getReturnCC(CallingConv::ID CallConv) {
+  switch (CallConv) {
+  default:
+    return RetCC_VE_C;
+  case CallingConv::Fast:
+    return RetCC_VE_Fast;
+  }
+}
+
+CCAssignFn *getParamCC(CallingConv::ID CallConv, bool IsVarArg) {
+  if (IsVarArg)
+    return CC_VE2;
+  switch (CallConv) {
+  default:
+    return CC_VE_C;
+  case CallingConv::Fast:
+    return CC_VE_Fast;
+  }
+}
+
 bool VETargetLowering::CanLowerReturn(
     CallingConv::ID CallConv, MachineFunction &MF, bool IsVarArg,
     const SmallVectorImpl<ISD::OutputArg> &Outs, LLVMContext &Context) const {
-  CCAssignFn *RetCC = RetCC_VE;
+  CCAssignFn *RetCC = getReturnCC(CallConv);
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCInfo(CallConv, IsVarArg, MF, RVLocs, Context);
   return CCInfo.CheckReturn(Outs, RetCC);
 }
 
 void VETargetLowering::initRegisterClasses() {
   // Set up the register classes.
   addRegisterClass(MVT::i32, &VE::I32RegClass);
VETargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
   // CCValAssign - represent the assignment of the return value to locations.
   SmallVector<CCValAssign, 16> RVLocs;
 
   // CCState - Info about the registers and stack slot.
   CCState CCInfo(CallConv, IsVarArg, DAG.getMachineFunction(), RVLocs,
                  *DAG.getContext());
 
   // Analyze return values.
-  CCInfo.AnalyzeReturn(Outs, RetCC_VE);
+  CCInfo.AnalyzeReturn(Outs, getReturnCC(CallConv));
 
   SDValue Flag;
   SmallVector<SDValue, 4> RetOps(1, Chain);
 
   // Copy the result values into the output registers.
   for (unsigned i = 0; i != RVLocs.size(); ++i) {
     CCValAssign &VA = RVLocs[i];
     assert(VA.isRegLoc() && "Can only return in registers!");
SDValue VETargetLowering::LowerFormalArguments(
   // Analyze arguments according to CC_VE.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CallConv, IsVarArg, DAG.getMachineFunction(), ArgLocs,
                  *DAG.getContext());
   // Allocate the preserved area first.
   CCInfo.AllocateStack(ArgsPreserved, Align(8));
   // We already allocated the preserved area, so the stack offset computed
   // by CC_VE would be correct now.
-  CCInfo.AnalyzeFormalArguments(Ins, CC_VE);
+  CCInfo.AnalyzeFormalArguments(Ins, getParamCC(CallConv, false));
 
   for (unsigned i = 0, e = ArgLocs.size(); i != e; ++i) {
     CCValAssign &VA = ArgLocs[i];
     if (VA.isRegLoc()) {
       // This argument is passed in a register.
       // All integer register arguments are promoted by the caller to i64.
 
       // Create a virtual register for the promoted live-in value.
SDValue VETargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   // Analyze operands of the call, assigning locations to each operand.
   SmallVector<CCValAssign, 16> ArgLocs;
   CCState CCInfo(CLI.CallConv, CLI.IsVarArg, DAG.getMachineFunction(), ArgLocs,
                  *DAG.getContext());
   // Allocate the preserved area first.
   CCInfo.AllocateStack(ArgsPreserved, Align(8));
   // We already allocated the preserved area, so the stack offset computed
   // by CC_VE would be correct now.
-  CCInfo.AnalyzeCallOperands(CLI.Outs, CC_VE);
+  CCInfo.AnalyzeCallOperands(CLI.Outs, getParamCC(CLI.CallConv, false));
 
   // VE requires to use both register and stack for varargs or no-prototyped
   // functions.
   bool UseBoth = CLI.IsVarArg;
 
   // Analyze operands again if it is required to store BOTH.
   SmallVector<CCValAssign, 16> ArgLocs2;
   CCState CCInfo2(CLI.CallConv, CLI.IsVarArg, DAG.getMachineFunction(),
                   ArgLocs2, *DAG.getContext());
   if (UseBoth)
-    CCInfo2.AnalyzeCallOperands(CLI.Outs, CC_VE2);
+    CCInfo2.AnalyzeCallOperands(CLI.Outs,
+                                getParamCC(CLI.CallConv, CLI.IsVarArg));
 
   // Get the size of the outgoing arguments stack space requirement.
   unsigned ArgsSize = CCInfo.getNextStackOffset();
 
   // Keep stack frames 16-byte aligned.
   ArgsSize = alignTo(ArgsSize, 16);
 
   // Adjust the stack pointer to make room for the arguments.
SDValue VETargetLowering::LowerCall(TargetLowering::CallLoweringInfo &CLI,
   CCState RVInfo(CLI.CallConv, CLI.IsVarArg, DAG.getMachineFunction(), RVLocs,
                  *DAG.getContext());
 
   // Set inreg flag manually for codegen generated library calls that
   // return float.
   if (CLI.Ins.size() == 1 && CLI.Ins[0].VT == MVT::f32 && !CLI.CB)
     CLI.Ins[0].Flags.setInReg();
 
-  RVInfo.AnalyzeCallResult(CLI.Ins, RetCC_VE);
+  RVInfo.AnalyzeCallResult(CLI.Ins, getReturnCC(CLI.CallConv));
 
   // Copy all of the result registers out of their specified physreg.
   for (unsigned i = 0; i != RVLocs.size(); ++i) {
     CCValAssign &VA = RVLocs[i];
     unsigned Reg = VA.getLocReg();
 
     // When returning 'inreg {i32, i32 }', two consecutive i32 arguments can
     // reside in the same register in the high and low bits. Reuse the

llvm/lib/Target/VE/VERegisterInfo.cpp

 #include "VEGenRegisterInfo.inc"
 
 // VE uses %s10 == %lp to keep return address
 VERegisterInfo::VERegisterInfo() : VEGenRegisterInfo(VE::SX10) {}
 
 const MCPhysReg *
 VERegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   switch (MF->getFunction().getCallingConv()) {
+  case CallingConv::Fast:
+    // Being explicit (same as standard CC).
   default:
     return CSR_SaveList;
   case CallingConv::PreserveAll:
     return CSR_preserve_all_SaveList;
   }
 }
 
 const uint32_t *VERegisterInfo::getCallPreservedMask(const MachineFunction &MF,
                                                      CallingConv::ID CC) const {
   switch (CC) {
+  case CallingConv::Fast:
+    // Being explicit (same as standard CC).
   default:
     return CSR_RegMask;
   case CallingConv::PreserveAll:
     return CSR_preserve_all_RegMask;
   }
 }
 
 const uint32_t *VERegisterInfo::getNoPreservedMask() const {

llvm/test/CodeGen/VE/Vector/fastcc.ll

+; RUN: llc < %s -mtriple=ve-unknown-unknown -mattr=+vpu | FileCheck %s
+
+; Scalar argument passing must not change (same tests as in VE/Scalar/callee.ll below - this time with +vpu)
+
+define fastcc i32 @stack_stack_arg_i32_r9(i1 %0, i8 %1, i16 %2, i32 %3, i64 %4, i32 %5, i32 %6, i32 %7, i32 %8, i32 %9) {
+; CHECK-LABEL: stack_stack_arg_i32_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldl.sx %s0, 424(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret i32 %9
+}
+
+define fastcc i64 @stack_stack_arg_i64_r9(i1 %0, i8 %1, i16 %2, i32 %3, i64 %4, i64 %5, i64 %6, i64 %7, i64 %8, i64 %9) {
+; CHECK-LABEL: stack_stack_arg_i64_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ld %s0, 424(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret i64 %9
+}
+
+define fastcc float @stack_stack_arg_f32_r9(float %p0, float %p1, float %p2, float %p3, float %p4, float %p5, float %p6, float %p7, float %s0, float %s1) {
+; CHECK-LABEL: stack_stack_arg_f32_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldu %s0, 428(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret float %s1
+}
+
+define fastcc i32 @stack_stack_arg_i32f32_r8(i32 %p0, float %p1, i32 %p2, float %p3, i32 %p4, float %p5, i32 %p6, float %p7, i32 %s0, float %s1) {
+; CHECK-LABEL: stack_stack_arg_i32f32_r8:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldl.sx %s0, 416(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret i32 %s0
+}
+
+define fastcc float @stack_stack_arg_i32f32_r9(i32 %p0, float %p1, i32 %p2, float %p3, i32 %p4, float %p5, i32 %p6, float %p7, i32 %s0, float %s1) {
+; CHECK-LABEL: stack_stack_arg_i32f32_r9:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    ldu %s0, 428(, %s11)
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret float %s1
+}
+
+; Vector argument passing (fastcc feature)
+
+; v0-to-v0 passthrough case without vreg copy.
+define fastcc <256 x i32> @vreg_arg_v256i32_r0(<256 x i32> %p0) {
+; CHECK-LABEL: vreg_arg_v256i32_r0:
+; CHECK:       .LBB{{[0-9]+}}_2:
+; CHECK-NEXT:    or %s11, 0, %s9
+  ret <256 x i32> %p0
+}
+
+; TODO: Uncomment tests when vreg-to-vreg copy is upstream.
+; define fastcc <256 x i32> @vreg_arg_v256i32_r1(<256 x i32> %p0, <256 x i32> %p1) {
+;   ret <256 x i32> %p1
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r2(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2) {
+;   ret <256 x i32> %p2
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r3(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3) {
+;   ret <256 x i32> %p3
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r4(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4) {
+;   ret <256 x i32> %p4
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r5(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5) {
+;   ret <256 x i32> %p5
+; }
+; 
+; define fastcc <256 x i32> @vreg_arg_v256i32_r6(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5, <256 x i32> %p6) {
+;   ret <256 x i32> %p6
+; }
+; 
+; define <256 x i32> @vreg_arg_v256i32_r7(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5, <256 x i32> %p6, <256 x i32> %p7) {
+;   ret <256 x i32> %p7
+; }
+
+; TODO: Uncomment test when vector loads are upstream (vreg stack passing).
+; define <256 x i32> @vreg_arg_v256i32_r8(<256 x i32> %p0, <256 x i32> %p1, <256 x i32> %p2, <256 x i32> %p3, <256 x i32> %p4, <256 x i32> %p5, <256 x i32> %p6, <256 x i32> %p7, <256 x i32> %p8) {
+;   ret <256 x i32> %p8
+; }