[X86] ABI breaking change for x86-32: pass 3 vector arguments in-register instead of 4, except on Darwin

[mkuper]

Summary

This changes the ABI used on 32-bit x86 for passing vector arguments.
Currently, clang passes 4 arguments in-register, and everything else on the stack, on all platforms.

This is different from the behavior of gcc, icc, and msvc, all of which pass 3 arguments in-register. The 3-register convention is documented, unofficially, in Agner's calling convention guide, and, officially, in the recently released version 1.0 of the i386 psABI (see https://01.org/sites/default/files/file_attach/intel386-psabi-1.0.pdf ).

This patch changes the behavior on all platforms except Darwin. The Darwin ABI is kept as is, because:
a) Clang is the mainstream compiler for the platform, which means that existing code is much more likely to rely on the current ABI.
b) The current behavior is documented (in the OS X ABI Function Call Guide).

See also PR21510.

Please feel free to add more reviewers that have a stake in this.

[rnk]

I think this is definitely the right change for Linux, Darwin, and Windows.

What about the BSDs, though? They switched to clang as the system compiler some time ago. They might want either the Darwin behavior for compatibility or the standard Sys V behavior for better conformance. You should bug Ed Maste and Joerg is already CCd.

[mkuper]

Thanks, Reid!

And yes, that's why I added Joerg.
(I recall talking to him about this on IRC when this originally came up in December, and he did not object to making the change on BSD, but I may be mis-remembering...)

[joerg]

I don't think it matters either way for NetBSD, at most a few things in pkgsrc are affected.

[emaste]

I didn't spot this when first posted, but I'll belatedly add that I think this is the right change for FreeBSD as well.

Clang has been our system compiler for quite some time but a reasonable portion of the ports tree is built with GCC, and interop and psABI conformance concerns probably override a potential slightly bumpy transition.

llvm/trunk/lib/Target/X86/X86CallingConv.td

 def CC_X86_64_AnyReg : CallingConv<[
   CCCustom<"CC_X86_AnyReg_Error">
 ]>;
 
 //===----------------------------------------------------------------------===//
 // X86 C Calling Convention
 //===----------------------------------------------------------------------===//
 
+/// CC_X86_32_Vector_Common - In all X86-32 calling conventions, extra vector
+/// values are spilled on the stack.
+def CC_X86_32_Vector_Common : CallingConv<[
+  // Other SSE vectors get 16-byte stack slots that are 16-byte aligned.
+  CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64], CCAssignToStack<16, 16>>,
+
+  // 256-bit AVX vectors get 32-byte stack slots that are 32-byte aligned.
+  CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+           CCAssignToStack<32, 32>>,
+
+  // 512-bit AVX 512-bit vectors get 64-byte stack slots that are 64-byte aligned.
+  CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+           CCAssignToStack<64, 64>>
+]>;
+
+// CC_X86_32_Vector_Standard - The first 3 vector arguments are passed in
+// vector registers
+def CC_X86_32_Vector_Standard : CallingConv<[
+  // SSE vector arguments are passed in XMM registers.
+  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+                CCAssignToReg<[XMM0, XMM1, XMM2]>>>,
+
+  // AVX 256-bit vector arguments are passed in YMM registers.
+  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+                CCIfSubtarget<"hasFp256()",
+                CCAssignToReg<[YMM0, YMM1, YMM2]>>>>,
+
+  // AVX 512-bit vector arguments are passed in ZMM registers.
+  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+                CCAssignToReg<[ZMM0, ZMM1, ZMM2]>>>,
+
+  CCDelegateTo<CC_X86_32_Vector_Common>
+]>;
+
+// CC_X86_32_Vector_Darwin - The first 4 vector arguments are passed in
+// vector registers.
+def CC_X86_32_Vector_Darwin : CallingConv<[
+  // SSE vector arguments are passed in XMM registers.
+  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+                CCAssignToReg<[XMM0, XMM1, XMM2, XMM3]>>>,
+
+  // AVX 256-bit vector arguments are passed in YMM registers.
+  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
+                CCIfSubtarget<"hasFp256()",
+                CCAssignToReg<[YMM0, YMM1, YMM2, YMM3]>>>>,
+
+  // AVX 512-bit vector arguments are passed in ZMM registers.
+  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
+                CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3]>>>,
+
+  CCDelegateTo<CC_X86_32_Vector_Common>
+]>;
+
 /// CC_X86_32_Common - In all X86-32 calling conventions, extra integers and FP
-/// values are spilled on the stack, and the first 4 vector values go in XMM
-/// regs.
+/// values are spilled on the stack.
 def CC_X86_32_Common : CallingConv<[
   // Handles byval parameters.
   CCIfByVal<CCPassByVal<4, 4>>,
 
   // The first 3 float or double arguments, if marked 'inreg' and if the call
   // is not a vararg call and if SSE2 is available, are passed in SSE registers.
   CCIfNotVarArg<CCIfInReg<CCIfType<[f32,f64],
                 CCIfSubtarget<"hasSSE2()",
def CC_X86_32_Common : CallingConv<[
   // since the boolean types in AVX/AVX2 are promoted by default.
   CCIfType<[v2i1],  CCPromoteToType<v2i64>>,
   CCIfType<[v4i1],  CCPromoteToType<v4i32>>,
   CCIfType<[v8i1],  CCPromoteToType<v8i16>>,
   CCIfType<[v16i1], CCPromoteToType<v16i8>>,
   CCIfType<[v32i1], CCPromoteToType<v32i8>>,
   CCIfType<[v64i1], CCPromoteToType<v64i8>>,
 
-  // The first 4 SSE vector arguments are passed in XMM registers.
-  CCIfNotVarArg<CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
-                CCAssignToReg<[XMM0, XMM1, XMM2, XMM3]>>>,
-
-  // The first 4 AVX 256-bit vector arguments are passed in YMM registers.
-  CCIfNotVarArg<CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
-                CCIfSubtarget<"hasFp256()",
-                CCAssignToReg<[YMM0, YMM1, YMM2, YMM3]>>>>,
-
-  // The first 4 AVX 512-bit vector arguments are passed in ZMM registers.
-  CCIfNotVarArg<CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
-                CCAssignToReg<[ZMM0, ZMM1, ZMM2, ZMM3]>>>,
-
-  // Other SSE vectors get 16-byte stack slots that are 16-byte aligned.
-  CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64], CCAssignToStack<16, 16>>,
-
-  // 256-bit AVX vectors get 32-byte stack slots that are 32-byte aligned.
-  CCIfType<[v32i8, v16i16, v8i32, v4i64, v8f32, v4f64],
-           CCAssignToStack<32, 32>>,
-
-  // 512-bit AVX 512-bit vectors get 64-byte stack slots that are 64-byte aligned.
-  CCIfType<[v64i8, v32i16, v16i32, v8i64, v16f32, v8f64],
-           CCAssignToStack<64, 64>>,
-
   // __m64 vectors get 8-byte stack slots that are 4-byte aligned. They are
   // passed in the parameter area.
-  CCIfType<[x86mmx], CCAssignToStack<8, 4>>]>;
+  CCIfType<[x86mmx], CCAssignToStack<8, 4>>,
+
+  // Darwin passes vectors in a form that differs from the i386 psABI
+  CCIfSubtarget<"isTargetDarwin()", CCDelegateTo<CC_X86_32_Vector_Darwin>>,
+
+  // Otherwise, drop to 'normal' X86-32 CC
+  CCDelegateTo<CC_X86_32_Vector_Standard>
+]>;
 
 def CC_X86_32_C : CallingConv<[
   // Promote i1/i8/i16 arguments to i32.
   CCIfType<[i1, i8, i16], CCPromoteToType<i32>>,
 
   // The 'nest' parameter, if any, is passed in ECX.
   CCIfNest<CCAssignToReg<[ECX]>>,
 

llvm/trunk/test/CodeGen/X86/fp-trunc.ll

 ; RUN: llc < %s -march=x86 -mcpu=corei7 | FileCheck %s
 ; RUN: llc < %s -march=x86 -mcpu=core-avx-i | FileCheck %s --check-prefix=AVX
 
+target triple = "i686-pc-linux-gnu"
+
 define <1 x float> @test1(<1 x double> %x) nounwind {
 ; CHECK-LABEL: test1:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    pushl %eax
 ; CHECK-NEXT:    movsd {{[0-9]+}}(%esp), %xmm0
 ; CHECK-NEXT:    cvtsd2ss %xmm0, %xmm0
 ; CHECK-NEXT:    movss %xmm0, (%esp)
 ; CHECK-NEXT:    flds (%esp)

 ; AVX-NEXT:    retl
   %y = fptrunc <4 x double> %x to <4 x float>
   ret <4 x float> %y
 }
 
 define <8 x float> @test4(<8 x double> %x) nounwind {
 ; CHECK-LABEL: test4:
 ; CHECK:       # BB#0:
+; CHECK-NEXT:    subl $12, %esp
 ; CHECK-NEXT:    cvtpd2ps %xmm1, %xmm1
 ; CHECK-NEXT:    cvtpd2ps %xmm0, %xmm0
 ; CHECK-NEXT:    unpcklpd {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; CHECK-NEXT:    cvtpd2ps %xmm3, %xmm3
 ; CHECK-NEXT:    cvtpd2ps %xmm2, %xmm1
-; CHECK-NEXT:    unpcklpd {{.*#+}} xmm1 = xmm1[0],xmm3[0]
+; CHECK-NEXT:    cvtpd2ps 16(%esp), %xmm2
+; CHECK-NEXT:    unpcklpd {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; CHECK-NEXT:    addl $12, %esp
 ; CHECK-NEXT:    retl
 ;
 ; AVX-LABEL: test4:
 ; AVX:       # BB#0:
 ; AVX-NEXT:    vcvtpd2psy %ymm0, %xmm0
 ; AVX-NEXT:    vcvtpd2psy %ymm1, %xmm1
 ; AVX-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX-NEXT:    retl
   %y = fptrunc <8 x double> %x to <8 x float>
   ret <8 x float> %y
 }
 
 

llvm/trunk/test/CodeGen/X86/illegal-vector-args-return.ll

 ; RUN: llc < %s -march=x86 -mattr=+sse2 -mcpu=nehalem | grep "mulpd %xmm3, %xmm1"
 ; RUN: llc < %s -march=x86 -mattr=+sse2 -mcpu=nehalem | grep "mulpd %xmm2, %xmm0"
 ; RUN: llc < %s -march=x86 -mattr=+sse2 -mcpu=nehalem | grep "addps %xmm3, %xmm1"
 ; RUN: llc < %s -march=x86 -mattr=+sse2 -mcpu=nehalem | grep "addps %xmm2, %xmm0"
 
+target triple = "i686-apple-darwin8"
+
 define <4 x double> @foo(<4 x double> %x, <4 x double> %z) {
   %y = fmul <4 x double> %x, %z
   ret <4 x double> %y
 }
 
 define <8 x float> @bar(<8 x float> %x, <8 x float> %z) {
   %y = fadd <8 x float> %x, %z
   ret <8 x float> %y
 }

llvm/trunk/test/CodeGen/X86/x86-32-vector-calling-conv.ll

+; RUN: llc < %s -mtriple=i686-apple-darwin -mattr=+avx512f | FileCheck %s --check-prefix=DARWIN
+; RUN: llc < %s -mtriple=i686-pc-linux -mattr=+avx512f | FileCheck %s --check-prefix=LINUX
+
+; CHECK-LABEL: test_sse:
+; DARWIN-DAG: vpaddd  %xmm1, %xmm0, %xmm0
+; DARWIN-DAG: vpaddd  %xmm3, %xmm2, %xmm1
+; DARWIN: vpaddd  %xmm1, %xmm0, %xmm0
+; LINUX-DAG:  vpaddd  %xmm1, %xmm0, %xmm0
+; LINUX-DAG:  vpaddd  {{[0-9]+}}(%e{{s|b}}p), %xmm2, %xmm1
+; LINUX:  vpaddd  %xmm1, %xmm0, %xmm0
+define <4 x i32> @test_sse(<4 x i32> %a, <4 x i32> %b, <4 x i32> %c, <4 x i32> %d) nounwind {
+  %r0 = add <4 x i32> %a, %b
+  %r1 = add <4 x i32> %c, %d
+  %ret = add <4 x i32> %r0, %r1
+  ret <4 x i32> %ret
+}
+
+; CHECK-LABEL: test_avx:
+; DARWIN-DAG: vpaddd  %ymm1, %ymm0, %ymm0
+; DARWIN-DAG: vpaddd  %ymm3, %ymm2, %ymm1
+; DARWIN: vpaddd  %ymm1, %ymm0, %ymm0
+; LINUX-DAG:  vpaddd  %ymm1, %ymm0, %ymm0
+; LINUX-DAG:  vpaddd  {{[0-9]+}}(%e{{s|b}}p), %ymm2, %ymm1
+; LINUX:  vpaddd  %ymm1, %ymm0, %ymm0
+define <8 x i32> @test_avx(<8 x i32> %a, <8 x i32> %b, <8 x i32> %c, <8 x i32> %d) nounwind {
+  %r0 = add <8 x i32> %a, %b
+  %r1 = add <8 x i32> %c, %d
+  %ret = add <8 x i32> %r0, %r1
+  ret <8 x i32> %ret
+}
+
+; CHECK-LABEL: test_avx512:
+; DARWIN-DAG: vpaddd  %zmm1, %zmm0, %zmm0
+; DARWIN-DAG: vpaddd  %zmm3, %zmm2, %zmm1
+; DARWIN: vpaddd  %zmm1, %zmm0, %zmm0
+; LINUX-DAG:  vpaddd  %zmm1, %zmm0, %zmm0
+; LINUX-DAG:  vpaddd  {{[0-9]+}}(%e{{s|b}}p), %zmm2, %zmm1
+; LINUX:  vpaddd  %zmm1, %zmm0, %zmm0
+define <16 x i32> @test_avx512(<16 x i32> %a, <16 x i32> %b, <16 x i32> %c, <16 x i32> %d) nounwind {
+  %r0 = add <16 x i32> %a, %b
+  %r1 = add <16 x i32> %c, %d
+  %ret = add <16 x i32> %r0, %r1
+  ret <16 x i32> %ret
+}