[X86] Don't allow illegal vector types to return by direct value on x86-64.

[craig.topper]

Summary

The backend can't lower this correctly and will try to split the return value into multiple registers.

This patches forces it to return via memory similar to what was already done for arguments.

Fixes PR39501.

[hjl.tools]

lib/CodeGen/TargetInfo.cpp

2862

Doesn't -m32 have the same issue?

[craig.topper]

Retitling to just the x86-64 case. 32-bit mode has issues on arguments too I think and will need more work. The IsIllegalVectorType function is a member of the X86_64ABIInfo so we need to refactor or add a new one for 32-bit.

[efriedma]

As far as I know, according to the ABI docs, it's impossible to return a 256-bit vector from a function if AVX is disabled.

Given that we aren't rejecting the testcase, we're effectively implementing a new "no-AVX" ABI variant. That variant is not defined anywhere, so we might as well pick the fastest convention, returning the value in registers. I'm not sure why you would want to return the vector in memory instead.

[hjl.tools]

In D53919#1282692, @efriedma wrote:

As far as I know, according to the ABI docs, it's impossible to return a 256-bit vector from a function if AVX is disabled.

Given that we aren't rejecting the testcase, we're effectively implementing a new "no-AVX" ABI variant. That variant is not defined anywhere, so we might as well pick the fastest convention, returning the value in registers. I'm not sure why you would want to return the vector in memory instead.

Not a new "no-AVX" ABI variant. It should be pre-AVX ABI.

[efriedma]

Whatever you call it, the question remains; what specification and/or compiler are we trying to be compatible with?

[hjl.tools]

In D53919#1282734, @efriedma wrote:

Whatever you call it, the question remains; what specification and/or compiler are we trying to be compatible with?

The ABI before AVX was introduced.

[efriedma]

The ABI document before AVX was introduced didn't say anything at all about 256-bit vectors. So we're talking about compatibility with some other compiler. Which compiler?

[hjl.tools]

In D53919#1282748, @efriedma wrote:

The ABI document before AVX was introduced didn't say anything at all about 256-bit vectors. So we're talking about compatibility with some other compiler. Which compiler?

X86-64 compilers created before AVX was introduced. Say clang 1.0 doesn't know AVX and returns it in memory. So should clang 7.0 with -mno-avx.

[efriedma]

I just tried clang 3.3, and as far as I can tell it behaves the same way as trunk.

If the argument is that we should be compatible with gcc on Linux, that's fine, but we should explicitly state that in a comment.

[hjl.tools]

In D53919#1282797, @efriedma wrote:

I just tried clang 3.3, and as far as I can tell it behaves the same way as trunk.

If the argument is that we should be compatible with gcc on Linux, that's fine, but we should explicitly state that in a comment.

Does clang 3.3 support AVX512?

[efriedma]

No, AVX512 wasn't even announced when clang 3.3 came out.

[hjl.tools]

In D53919#1282811, @efriedma wrote:

No, AVX512 wasn't even announced when clang 3.3 came out.

Try this with clang 3.3 and clang 7.0.

[hjl@gnu-cfl-1 tmp]$ cat z.c
typedef int attribute((mode(SI))) si;
typedef si attribute((vector_size (64))) v;

v
foo (void)
{
v y = {'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f' };
return y;
}
[hjl@gnu-cfl-1 tmp]$ gcc -S -O2 z.c
z.c: In function ‘foo’:
z.c:6:1: warning: AVX512F vector return without AVX512F enabled changes the ABI [-Wpsabi]
{
^
[hjl@gnu-cfl-1 tmp]$

[efriedma]

With both 3.3 and trunk (I don't have a 7.0 handy; I can build it if it would be helpful):

movaps  .LCPI0_0(%rip), %xmm0   # xmm0 = [48,49,50,51]
movaps  .LCPI0_1(%rip), %xmm1   # xmm1 = [52,53,54,55]
movaps  .LCPI0_2(%rip), %xmm2   # xmm2 = [56,57,97,98]
movaps  .LCPI0_3(%rip), %xmm3   # xmm3 = [99,100,101,102]
retq

[hjl.tools]

In D53919#1282952, @efriedma wrote:

With both 3.3 and trunk (I don't have a 7.0 handy; I can build it if it would be helpful):

Please try clang 2.6 on both testcases.

[echristo]

In D53919#1282994, @hjl.tools wrote:

In D53919#1282952, @efriedma wrote:

With both 3.3 and trunk (I don't have a 7.0 handy; I can build it if it would be helpful):

Please try clang 2.6 on both testcases.

From the releases:

23 Oct 2009 2.6

... why would you care about a 9 year old version of clang?

[hjl.tools]

In D53919#1287510, @echristo wrote:

In D53919#1282994, @hjl.tools wrote:

In D53919#1282952, @efriedma wrote:

With both 3.3 and trunk (I don't have a 7.0 handy; I can build it if it would be helpful):

Please try clang 2.6 on both testcases.

From the releases:

23 Oct 2009 2.6

... why would you care about a 9 year old version of clang?

It is about ABI consistency. When AVX isn't enabled, the old and compilers should
use the same calling convention.

lib/CodeGen/TargetInfo.cpp

if (const RecordType *RT = Ty->getAs<RecordType>()) {
 
     postMerge(Size, Lo, Hi);
   }
 }
 
 ABIArgInfo X86_64ABIInfo::getIndirectReturnResult(QualType Ty) const {
   // If this is a scalar LLVM value then assume LLVM will pass it in the right
   // place naturally.
-  if (!isAggregateTypeForABI(Ty)) {
+  if (!isAggregateTypeForABI(Ty) && !IsIllegalVectorType(Ty)) {
     // Treat an enum type as its underlying type.
     if (const EnumType *EnumTy = Ty->getAs<EnumType>())
       Ty = EnumTy->getDecl()->getIntegerType();
 
     return (Ty->isPromotableIntegerType() ? ABIArgInfo::getExtend(Ty)
                                           : ABIArgInfo::getDirect());
   }
 

test/CodeGen/x86_64-arguments.c

 struct t65 {
   __m256 m;
   int : 0;
 };
 // SSE-LABEL: @f65(%struct.t65* byval align 32 %{{[^,)]+}})
 // AVX: @f65(<8 x float> %{{[^,)]+}})
 void f65(struct t65 a0) {
 }
+
+// SSE-LABEL: @f66(<32 x i8>* noalias sret %{{[^,)]+}})
+// AVX: <32 x i8> @f66()
+typedef char v66 __attribute__((vector_size (32))) ;
+v66 f66()
+{
+ v66 y = {'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f',
+          '0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'};
+ return y;
+}