[X86ISel] Implement byval lowering for Win64 calling convention

[loladiro]

Summary

The IR reference for the byval attribute states:

This indicates that the pointer parameter should really be passed by value
to the function. The attribute implies that a hidden copy of the pointee is
made between the caller and the callee, so the callee is unable to modify
the value in the caller. This attribute is only valid on LLVM pointer arguments.

However, on Win64, this attribute is unimplemented and the raw pointer is
passed to the callee instead. This is problematic, because frontend authors
relying on the implicit hidden copy (as happens for every other calling
convention) will see the passed value silently (if mutable memory) or
loudly (by means of a crash) modified because the callee treats the
location as scratch memory space it is allowed to mutate.

At this point, it's worth taking a step back to understand the context.
In most calling conventions, aggregates that are too large to be passed
in registers, instead get *copied* to the stack at a fixed (computable
from the signature) offset of the stack pointer. At the LLVM, we hide
this hidden copy behind the byval attribute. The caller passes a pointer
to the desired data and the callee receives a pointer, but these pointers
are not the same. In particular, the pointer that the callee receives
points to temporary stack memory allocated as part of the call lowering.
In most calling conventions, this pointer is never realized in registers
or memory. The temporary memory is simply defined by an implicit
offset from the stack pointer at function entry.

Win64, uniquely, works differently. The structure is still passed in
memory, but instead of being stored at an implicit memory offset, the
caller computes a pointer to the temporary memory and passes it to
the callee as a regular pointer (taking up a register, or if all
registers are taken up, an additional stack slot). Presumably, this
was done to allow eliding the copy when passing aggregates through
several functions on the stack.

This explains why ignoring the byval attribute mostly works on Win64.
The argument simply gets passed as a pointer and as long as we're ok
with the callee trampling all over that memory, there are no ill effects.
However, it does contradict the documentation of the byval attribute
which specifies that there is to be an implicit copy.

Frontends can of course work around this by never emitting the byval
attribute for Win64 and creating allocas for the requisite temporary
stack slots (and that does appear to be what frontends are doing).
However, the presence of the byval attribute is not a trap for
frontend authors, since it seems to work, but silently modifies the
passed memory contrary to documentation.

I see two solutions:

• Disallow the byval attribute in the verifier if using the Win64 calling convention.
• Make it work by simply emitting a temporary stack copy as we would with any other calling convention (frontends can of course always not use the attribute if they want to elide the copy).

This patch implements the second option (make it work), though I would
be fine with the first also.

Ref: https://github.com/JuliaLang/julia/issues/28338

[rnk]

lgtm with a tweaked comment. Thanks!

lib/Target/X86/X86ISelLowering.cpp

3625

I wouldn't call this a spill slot, I would say, copy the argument so that the caller does not see modifications.

[efriedma]

If you're a frontend author, I would suggest not using byval, ever; it mostly exists for compatibility with existing C calling conventions. (Of course, it should still work if the frontend requests it.)

[loladiro]

Fix comment as requested by review, fix a small bug found in more extensive testing,
add a test case for the latter.

lib/Target/X86/X86CallingConv.td

def CC_X86_64_HHVM_C : CallingConv<[
   CCIfType<[i64], CCAssignToReg<[RBP]>>,
 
   // Otherwise it's the same as the regular C calling convention.
   CCDelegateTo<CC_X86_64_C>
 ]>;
 
 // Calling convention used on Win64
 def CC_X86_Win64_C : CallingConv<[
-  // FIXME: Handle byval stuff.
   // FIXME: Handle varargs.
 
+  // Byval aggregates are passed by pointer
+  CCIfByVal<CCPassIndirect<i64>>,
+
   // Promote i1/v1i1 arguments to i8.
   CCIfType<[i1, v1i1], CCPromoteToType<i8>>,
 
   // The 'nest' parameter, if any, is passed in R10.
   CCIfNest<CCAssignToReg<[R10]>>,
 
   // A SwiftError is passed in R12.
   CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[R12]>>>,

lib/Target/X86/X86ISelLowering.cpp

if (VA.isRegLoc()) {
       }
     } else {
       assert(VA.isMemLoc());
       ArgValue =
           LowerMemArgument(Chain, CallConv, Ins, dl, DAG, VA, MFI, InsIndex);
     }
 
     // If value is passed via pointer - do a load.
-    if (VA.getLocInfo() == CCValAssign::Indirect)
+    if (VA.getLocInfo() == CCValAssign::Indirect && !Ins[I].Flags.isByVal())
       ArgValue =
           DAG.getLoad(VA.getValVT(), dl, Chain, ArgValue, MachinePointerInfo());
 
     InVals.push_back(ArgValue);
   }
 
   for (unsigned I = 0, E = Ins.size(); I != E; ++I) {
     // Swift calling convention does not require we copy the sret argument
case CCValAssign::AExt:
         Arg = getMOVL(DAG, dl, MVT::v2i64, DAG.getUNDEF(MVT::v2i64), Arg);
       } else
         Arg = DAG.getNode(ISD::ANY_EXTEND, dl, RegVT, Arg);
       break;
     case CCValAssign::BCvt:
       Arg = DAG.getBitcast(RegVT, Arg);
       break;
     case CCValAssign::Indirect: {
+      if (isByVal) {
+        // Memcpy the argument to a temporary stack slot to prevent
+        // the caller from seeing any modifications the callee may make
+        // as guaranteed by the `byval` attribute.
+        int FrameIdx = MF.getFrameInfo().CreateStackObject(
+            Flags.getByValSize(), std::max(16, (int)Flags.getByValAlign()),
+            false);
+        SDValue StackSlot =
+            DAG.getFrameIndex(FrameIdx, getPointerTy(DAG.getDataLayout()));
+        Chain =
+            CreateCopyOfByValArgument(Arg, StackSlot, Chain, Flags, DAG, dl);
+        // From now on treat this as a regular pointer
+        Arg = StackSlot;
+        isByVal = false;
+      } else {
         // Store the argument.
         SDValue SpillSlot = DAG.CreateStackTemporary(VA.getValVT());
         int FI = cast<FrameIndexSDNode>(SpillSlot)->getIndex();
         Chain = DAG.getStore(
             Chain, dl, Arg, SpillSlot,
             MachinePointerInfo::getFixedStack(DAG.getMachineFunction(), FI));
         Arg = SpillSlot;
+      }
       break;
     }
     }
 
     if (VA.needsCustom()) {
       assert(VA.getValVT() == MVT::v64i1 &&
              "Currently the only custom case is when we split v64i1 to 2 regs");
       // Split v64i1 value into two registers

test/CodeGen/X86/win64-byval.ll

+; RUN: llc -mtriple x86_64-w64-mingw32 %s -o - | FileCheck %s
+
+declare void @foo({ float, double }* byval)
+@G = external constant { float, double }
+
+define void @bar()
+{
+; Make sure we're creating a temporary stack slot, rather than just passing
+; the pointer through unmodified.
+; CHECK-LABEL: @bar
+; CHECK: movq    G+8(%rip), %rax
+; CHECK: movq    %rax, 40(%rsp)
+; CHECK: movq    G(%rip), %rax
+; CHECK: movq    %rax, 32(%rsp)
+; CHECK: leaq    32(%rsp), %rcx
+    call void @foo({ float, double }* byval @G)
+    ret void
+}
+
+define void @baz({ float, double }* byval %arg)
+{
+; On Win64 the byval is effectively ignored on declarations, since we do
+; pass a real pointer in registers. However, by our semantics if we pass
+; the pointer on to another byval function, we do need to make a copy.
+; CHECK-LABEL: @baz
+; CHECK: movq (%rcx), %rax
+; CHECK: movq 8(%rcx), %rcx
+; CHECK: movq %rcx, 40(%rsp)
+; CHECK: movq %rax, 32(%rsp)
+; CHECK: leaq 32(%rsp), %rcx
+    call void @foo({ float, double }* byval %arg)
+    ret void
+}