[X86] Fix bug in vectorcall calling convention

[wxiao3]

Summary

Original implementation can't correctly handle m256 and m512 types passed
by reference through stack. This patch fixes it.

[rnk]

lib/Target/X86/X86CallingConv.cpp

165

One way to deal with this would be to find MCRegisterInfo and ask if this is a sub/super register of XMM0 so we don't have to enumerate all the X,Y,Z cases and inevitable W someday in the future. But I'd accept this fix as is because I'm not sure how to get the register info data here.

[wxiao3]

I have updated the patch so that we don't have to enumerate all the X,Y,Z cases and inevitable W someday in the future. Is it ok for merge?

[rnk]

lgtm

lib/Target/X86/X86CallingConv.cpp

165

Eh, now I feel this made the code less readable since we still have to say XMM4/5 twice. Anyway, I'm not that concerned.

[wxiao3]

lib/Target/X86/X86CallingConv.cpp

165

The XMM4/5 only appear once. Can we merge it now?

[rnk]

lib/Target/X86/X86CallingConv.cpp

165

Sure, I already approved the patch, but it needs a committer to submit it. I see this is your first patch in phab, so I'll go ahead and do that. Thanks for finding regsOverlap!

lib/Target/X86/X86CallingConv.cpp

if (!ArgFlags.isHva() || ArgFlags.isHvaStart()) {
     (void)State.AllocateReg(CC_X86_64_VectorCallGetGPRs());
 
     // Assign XMM register - (shadow for HVA and non-shadow for non HVA).
     if (unsigned Reg = State.AllocateReg(CC_X86_VectorCallGetSSEs(ValVT))) {
       // In Vectorcall Calling convention, additional shadow stack can be
       // created on top of the basic 32 bytes of win64.
       // It can happen if the fifth or sixth argument is vector type or HVA.
       // At that case for each argument a shadow stack of 8 bytes is allocated.
-      if (Reg == X86::XMM4 || Reg == X86::XMM5)
+      if (Reg == X86::XMM4 || Reg == X86::XMM5 ||
+          Reg == X86::YMM4 || Reg == X86::YMM5 ||
+          Reg == X86::ZMM4 || Reg == X86::ZMM5)
         State.AllocateStack(8, 8);
 
       if (!ArgFlags.isHva()) {
         State.addLoc(CCValAssign::getReg(ValNo, ValVT, Reg, LocVT, LocInfo));
         return true; // Allocated a register - Stop the search.
       }
     }
   }

test/CodeGen/X86/x86-64-veccallcc.ll

+; RUN: llc -mtriple=x86_64-pc-windows-msvc < %s | FileCheck %s
+
+; Test 1st and 2nd arguments passed in YMM0 and YMM1.
+; Test 7nd argument passed by reference in stack: 56(%rsp).
+define x86_vectorcallcc <8 x float> @test_m256_7(<8 x float> %a, <8 x float> %b, <8 x float> %c, <8 x float> %d, <8 x float> %e, <8 x float> %f, <8 x float> %g) #0 {
+  ; CHECK-LABEL: test_m256_7@@224:
+  ; CHECK: movq 56(%rsp), %rax
+  ; CHECK: vaddps %ymm1, %ymm0, %ymm0
+  ; CHECK: vsubps (%rax), %ymm0, %ymm0
+  %add.i = fadd <8 x float> %a, %b
+  %sub.i = fsub <8 x float> %add.i, %g
+  ret <8 x float> %sub.i
+}
+
+attributes #0 = { nounwind "target-cpu"="core-avx2" }