[AArch64][GlobalISel] Lower formal arguments of AAPCS & ms_abi variadic functions.

[dzhidzhoev]

Summary

Reimplemented SelectionDAG code for GlobalISel.

Fixes https://github.com/llvm/llvm-project/issues/54079

[dzhidzhoev]

Fix pre-merge checks fail.

[paquette]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

561

It looks like these arrays already exist in AArch64CallingConvention.cpp; is it possible to use them here? (Or should we?)

572

nit: can you use static_cast for greppability?

573

Can you explain why this is the case in a comment? Even referencing some win64 doc would be useful?

576

nit: static_cast?

580

can you pull LLT::pointer(0, 64) out into like

const LLT p0 = LLT::pointer(0, 64);

and use that everywhere?

582

similar to above, can you create a

LLT s64 = LLT::scalar(64);

and use that everywhere?

[dzhidzhoev]

Addressing @paquette comments.

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

I'm not sure I have clean understanding of this line. It was introduced here https://reviews.llvm.org/D35720

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

clear*

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

The reason why it's always 8, is that if we're reserving space for a number of 8 byte registers and align it to 16, we either will have a 8 byte gap, or no gap at all.

As for the exact reasons why we manually added a gap in the SelectionDAG implementation, I don't really remember anything more than what you can read in the old phabricator review unfortunately.

[dzhidzhoev]

Rebase & remove placeholder stack object.

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

It seems like these lines don't affect test output. Let's try to remove them.

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

I can't guarantee that there aren't gaps in the testcases for this combination though, so it'd be good if you'd manually try to see if there are other cases that can be triggered (with either more or less formal parameters to the function, and/or more or less regular stack allocation.

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

I've tested it on modified sample from Github issue https://godbolt.org/z/ndbx96M9r .
It has shown the same output for GlobalISel and SelectionDAG.
Could you possibly suggest something else to check?

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

I don't know about that particular testcase or the GlobalISel implementation, but if referring to the existing code,

if (GPRSaveSize & 15)
  // The extra size here, if triggered, will always be 8.
  MFI.CreateFixedObject(16 - (GPRSaveSize & 15), -(int)alignTo(GPRSaveSize, 16), false);

then if this is removed, three existing testcases under test/CodeGen/AArch64 fail/change output, aarch64_win64cc_vararg.ll, sponentry.ll and win64_vararg.ll. Have a look at what changes in their output. E.g. for the win64_vararg.ll testcase; there, the function f7 ends up broken, looking like this:

f7:
        sub     sp, sp, #16
        add     x8, sp, #8
        add     x0, sp, #8
        stp     x8, x7, [sp], #16
        ret

(Here, both x8 and x7 are stored on the stack outside of the stack allocation.)

Please check at least all those cases that differ with the SelectionDAG implementation when this logic is removed, that they work correctly with GlobalISel too.

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

Unfortunately, va_start translation for Windows has not been fully implemented in GlobalISel yet, that is the reason why I hadn't changed test files you mentioned.
As far as I can test, I see no problems in these tests output with va_start call removed.

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

Ok, so this only handles some part of the variadic functions, but not everything of it? In that case I guess you won't hit this case quite yet, as that relates specifically to dumping the variadic arguments from registers, i.e. part of va_start.

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

aarch64_win64cc_vararg.ll covers variable dumping, doesn't it? It triggers lines being discussed and seems to be ok without them.
What else you think should be checked to hit the case? Sorry for misunderstanding.

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

I don't think there's anything else involved in that; the quirky code lines from the SelectionDAG implementation were specifically about variable dumping - so if that's not relevant here (yet), then you probably can leave out that bit entirely, for now.

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

Thank you so much for the help!

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

562

You should consistently get the calling convention from CCState, not the parent function. It's wrong in outgoing contexts

[rnk]

+@efriedma @hans for Win/ARM64

[dzhidzhoev]

Updated & rebased.

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

1945

Did you really mean the caller CC here, not the callee?

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

1945

In these lines, we're lowering G_VASTART instruction. The branch is supposed to execute if the calling convention of MachineFunction containing G_VASTART is Win64.

Do you think that this code does something else?

[dzhidzhoev]

Rebased.

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

1945

I know nothing about varargs, I just thought it was suspicious to see selection dependent on the parent calling convention

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

1945

Got it. Thank you for paying attention.

[mstorsjo]

I had somewhat missed the scope of this patch - I had only expected it to affect how functions marked ms_abi on non-Windows platforms are lowered, while it affects all regular Windows targets too.

This caused a pretty severe regression for Windows targets, reproducible with this snippet:

$ cat test.c
double func(double f) {
    return f;
}
$ clang -target aarch64-windows -S -o - test.c -fno-asynchronous-unwind-tables
func:
	sub	sp, sp, #16
	str	x0, [sp, #8]
	ldr	d0, [sp, #8]
	add	sp, sp, #16
	ret

(The -fno-asynchronous-unwind-tables option is only relevant for clarity of the output.)

I'll go ahead and push a revert in a moment.

[dzhidzhoev]

Don't treat all Win64CC functions as vararg.

[dzhidzhoev]

In D130903#3991285, @mstorsjo wrote:

I had somewhat missed the scope of this patch - I had only expected it to affect how functions marked ms_abi on non-Windows platforms are lowered, while it affects all regular Windows targets too.

This caused a pretty severe regression for Windows targets, reproducible with this snippet:

$ cat test.c
double func(double f) {
    return f;
}
$ clang -target aarch64-windows -S -o - test.c -fno-asynchronous-unwind-tables
func:
	sub	sp, sp, #16
	str	x0, [sp, #8]
	ldr	d0, [sp, #8]
	add	sp, sp, #16
	ret

(The -fno-asynchronous-unwind-tables option is only relevant for clarity of the output.)

I'll go ahead and push a revert in a moment.

Thank you for catching it! I wonder how win64 tests didn't fail on that. The problem was that during rebase I accidentally changed CCAssignFn arguments, so that all Win64 function was treated as vararg. Fixed it now. Should I recommit?

[mstorsjo]

In D130903#3992834, @dzhidzhoev wrote:

In D130903#3991285, @mstorsjo wrote:

I had somewhat missed the scope of this patch - I had only expected it to affect how functions marked ms_abi on non-Windows platforms are lowered, while it affects all regular Windows targets too.

This caused a pretty severe regression for Windows targets, reproducible with this snippet:

$ cat test.c
double func(double f) {
    return f;
}
$ clang -target aarch64-windows -S -o - test.c -fno-asynchronous-unwind-tables
func:
	sub	sp, sp, #16
	str	x0, [sp, #8]
	ldr	d0, [sp, #8]
	add	sp, sp, #16
	ret

(The -fno-asynchronous-unwind-tables option is only relevant for clarity of the output.)

I'll go ahead and push a revert in a moment.

Thank you for catching it! I wonder how win64 tests didn't fail on that. The problem was that during rebase I accidentally changed CCAssignFn arguments, so that all Win64 function was treated as vararg. Fixed it now. Should I recommit?

Thanks for the update! I can run a bunch of tests on it that specifically exercise calling conventions (that I don't otherwise run regularly) - I'll let you know how it fares.

[mstorsjo]

In D130903#3992928, @mstorsjo wrote:

In D130903#3992834, @dzhidzhoev wrote:

Thank you for catching it! I wonder how win64 tests didn't fail on that. The problem was that during rebase I accidentally changed CCAssignFn arguments, so that all Win64 function was treated as vararg. Fixed it now. Should I recommit?

Thanks for the update! I can run a bunch of tests on it that specifically exercise calling conventions (that I don't otherwise run regularly) - I'll let you know how it fares.

The tests I had ran just fine now, thanks!

However I don't see any added test for this case - could you add that too? (I guess it's probably covered by some existing test - just extending that to run with globalisel probably is enough?)

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

687

Parameter name suggest this should just be isVarArg

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

687

https://github.com/llvm/llvm-project/blob/9408164254d26d5305fe4e0267b668c41c1266ed/llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp#L602
On the trunk we pass false even if the function is vararg. Are you sure I should change this behavior for non win64 calling conventions?

[efriedma]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

687

The way AArch64TargetLowering::CCAssignFnForCall works is a little weird. For Darwin calling conventions, "fixed" arguments to varargs functions are passed differently from "variadic" arguments. Instead of recording that information in the arguments themselves, we call a different AssignFn for fixed vs. variadic arguments.

There are no such thing as variadic parameters, though; you can only access variadic arguments through va_arg. So we don't need to handle variadic arguments here.

Maybe there's a better way to implement the whole thing (if we record whether an argument is fixed in ArgFlagsTy, we could unify the Darwin calling conventions), but this is consistent with the way it currently works for SelectionDAG ISel.

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

687

I vaguely remember encountering this mess before

[mstorsjo]

In D130903#3993407, @mstorsjo wrote:

However I don't see any added test for this case - could you add that too? (I guess it's probably covered by some existing test - just extending that to run with globalisel probably is enough?)

Before reapplying, I'd still want to have a testcase added for the case that broke.

[dzhidzhoev]

Addressed @mstorsjo comment. Added "foo" test function, causing
regression for Windows target on previous (buggy) patch version.

[mstorsjo]

In D130903#4101303, @dzhidzhoev wrote:

Addressed @mstorsjo comment. Added "foo" test function, causing
regression for Windows target on previous (buggy) patch version.

Thanks - I believe this should be ok to try relanding it then! I'll follow up again if it triggers something else somewhere, but it at least seems fine in initial testing now.

[dzhidzhoev]

In D130903#4112826, @mstorsjo wrote:

In D130903#4101303, @dzhidzhoev wrote:

Addressed @mstorsjo comment. Added "foo" test function, causing
regression for Windows target on previous (buggy) patch version.

Thanks - I believe this should be ok to try relanding it then! I'll follow up again if it triggers something else somewhere, but it at least seems fine in initial testing now.

Thank you!

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

It turns out that the missing case of (GPRSaveSize & 15) did indeed break something.

aarch64_win64cc_vararg.ll tests much fewer cases than win64_vararg.ll, and during this review, I was led to believe that this change wouldn't hit those cases.

I see that win64_vararg.ll fails to translate with GlobalISel (don't quite know why) so I see why you weren't able to test that easily.

However I believe the issue should have been possible to test by just testing various functions with both even and odd numbers of parameters. Unfortunately, the issue is quite nonobvious; without the fix, things do seem to work right, but the location of stack objects mismatch between AArch64FrameLowering and the stack objects that are listed in MIR - readding this removed line fixes that.

The issue caused by this is https://github.com/llvm/llvm-project/issues/64740. I've got a patch coming up, readding the lines that were left out here.

[dzhidzhoev]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

573

Thank you for clarifying and fixing it!

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum Rounding {
   RP = 1,    // Round towards Plus infinity
   RM = 2,    // Round towards Minus infinity
   RZ = 3,    // Round towards Zero
   rmMask = 3 // Bit mask selecting rounding mode
 };
 
 // Bit position of rounding mode bits in FPCR.
 const unsigned RoundingBitsPos = 22;
+
+// Registers used to pass function arguments.
+const ArrayRef<MCPhysReg> getGPRArgRegs();
+const ArrayRef<MCPhysReg> getFPRArgRegs();
+
 } // namespace AArch64
 
 class AArch64Subtarget;
 
 class AArch64TargetLowering : public TargetLowering {
 public:
   explicit AArch64TargetLowering(const TargetMachine &TM,
                                  const AArch64Subtarget &STI);

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 // bottleneck after this transform on high end CPU. So this max leaf node
 // limitation is guard cmp+ccmp will be profitable.
 static cl::opt<unsigned> MaxXors("aarch64-max-xors", cl::init(16), cl::Hidden,
                                  cl::desc("Maximum of xors"));
 
 /// Value type used for condition codes.
 static const MVT MVT_CC = MVT::i32;
 
+static const MCPhysReg GPRArgRegs[] = {AArch64::X0, AArch64::X1, AArch64::X2,
+                                       AArch64::X3, AArch64::X4, AArch64::X5,
+                                       AArch64::X6, AArch64::X7};
+static const MCPhysReg FPRArgRegs[] = {AArch64::Q0, AArch64::Q1, AArch64::Q2,
+                                       AArch64::Q3, AArch64::Q4, AArch64::Q5,
+                                       AArch64::Q6, AArch64::Q7};
+
+const ArrayRef<MCPhysReg> llvm::AArch64::getGPRArgRegs() { return GPRArgRegs; }
+
+const ArrayRef<MCPhysReg> llvm::AArch64::getFPRArgRegs() { return FPRArgRegs; }
+
 static inline EVT getPackedSVEVectorVT(EVT VT) {
   switch (VT.getSimpleVT().SimpleTy) {
   default:
     llvm_unreachable("unexpected element type for vector");
   case MVT::i8:
     return MVT::nxv16i8;
   case MVT::i16:
     return MVT::nxv8i16;
void AArch64TargetLowering::saveVarArgRegisters(CCState &CCInfo,
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
   auto PtrVT = getPointerTy(DAG.getDataLayout());
   bool IsWin64 = Subtarget->isCallingConvWin64(MF.getFunction().getCallingConv());
 
   SmallVector<SDValue, 8> MemOps;
 
-  static const MCPhysReg GPRArgRegs[] = { AArch64::X0, AArch64::X1, AArch64::X2,
-                                          AArch64::X3, AArch64::X4, AArch64::X5,
-                                          AArch64::X6, AArch64::X7 };
-  unsigned NumGPRArgRegs = std::size(GPRArgRegs);
+  auto GPRArgRegs = AArch64::getGPRArgRegs();
+  unsigned NumGPRArgRegs = GPRArgRegs.size();
   if (Subtarget->isWindowsArm64EC()) {
     // In the ARM64EC ABI, only x0-x3 are used to pass arguments to varargs
     // functions.
     NumGPRArgRegs = 4;
   }
   unsigned FirstVariadicGPR = CCInfo.getFirstUnallocated(GPRArgRegs);
 
   unsigned GPRSaveSize = 8 * (NumGPRArgRegs - FirstVariadicGPR);
for (unsigned i = FirstVariadicGPR; i < NumGPRArgRegs; ++i) {
       FIN =
           DAG.getNode(ISD::ADD, DL, PtrVT, FIN, DAG.getConstant(8, DL, PtrVT));
     }
   }
   FuncInfo->setVarArgsGPRIndex(GPRIdx);
   FuncInfo->setVarArgsGPRSize(GPRSaveSize);
 
   if (Subtarget->hasFPARMv8() && !IsWin64) {
-    static const MCPhysReg FPRArgRegs[] = {
-        AArch64::Q0, AArch64::Q1, AArch64::Q2, AArch64::Q3,
-        AArch64::Q4, AArch64::Q5, AArch64::Q6, AArch64::Q7};
-    static const unsigned NumFPRArgRegs = std::size(FPRArgRegs);
+    auto FPRArgRegs = AArch64::getFPRArgRegs();
+    const unsigned NumFPRArgRegs = FPRArgRegs.size();
     unsigned FirstVariadicFPR = CCInfo.getFirstUnallocated(FPRArgRegs);
 
     unsigned FPRSaveSize = 16 * (NumFPRArgRegs - FirstVariadicFPR);
     int FPRIdx = 0;
     if (FPRSaveSize != 0) {
       FPRIdx = MFI.CreateStackObject(FPRSaveSize, Align(16), false);
 
       SDValue FIN = DAG.getFrameIndex(FPRIdx, PtrVT);

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.h

 
 private:
   using RegHandler = std::function<void(MachineIRBuilder &, Type *, unsigned,
                                         CCValAssign &)>;
 
   using MemHandler =
       std::function<void(MachineIRBuilder &, int, CCValAssign &)>;
 
+  void saveVarArgRegisters(MachineIRBuilder &MIRBuilder,
+                           CallLowering::IncomingValueHandler &Handler,
+                           CCState &CCInfo) const;
+
   bool lowerTailCall(MachineIRBuilder &MIRBuilder, CallLoweringInfo &Info,
                      SmallVectorImpl<ArgInfo> &OutArgs) const;
 
   bool
   doCallerAndCalleePassArgsTheSameWay(CallLoweringInfo &Info,
                                       MachineFunction &MF,
                                       SmallVectorImpl<ArgInfo> &InArgs) const;
 
   bool
   areCalleeOutgoingArgsTailCallable(CallLoweringInfo &Info, MachineFunction &MF,
                                     SmallVectorImpl<ArgInfo> &OutArgs) const;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_LIB_TARGET_AARCH64_AARCH64CALLLOWERING_H

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

 /// This file implements the lowering of LLVM calls to machine code calls for
 /// GlobalISel.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "AArch64CallLowering.h"
 #include "AArch64ISelLowering.h"
 #include "AArch64MachineFunctionInfo.h"
+#include "AArch64RegisterInfo.h"
 #include "AArch64Subtarget.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Analysis/ObjCARCUtil.h"
 #include "llvm/CodeGen/Analysis.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/CodeGen/FunctionLoweringInfo.h"
 #include "llvm/CodeGen/GlobalISel/MachineIRBuilder.h"
bool AArch64CallLowering::fallBackToDAGISel(const MachineFunction &MF) const {
   SMEAttrs Attrs(F);
   if (Attrs.hasNewZAInterface() ||
       (!Attrs.hasStreamingInterface() && Attrs.hasStreamingBody()))
     return true;
 
   return false;
 }
 
+void AArch64CallLowering::saveVarArgRegisters(
+    MachineIRBuilder &MIRBuilder, CallLowering::IncomingValueHandler &Handler,
+    CCState &CCInfo) const {
+  auto GPRArgRegs = AArch64::getGPRArgRegs();
+  auto FPRArgRegs = AArch64::getFPRArgRegs();
+
+  MachineFunction &MF = MIRBuilder.getMF();
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
+  auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+  bool IsWin64CC =
+      Subtarget.isCallingConvWin64(CCInfo.getCallingConv());
+  const LLT p0 = LLT::pointer(0, 64);
+  const LLT s64 = LLT::scalar(64);
+
+  unsigned FirstVariadicGPR = CCInfo.getFirstUnallocated(GPRArgRegs);
+  unsigned NumVariadicGPRArgRegs = GPRArgRegs.size() - FirstVariadicGPR + 1;
+
+  unsigned GPRSaveSize = 8 * (GPRArgRegs.size() - FirstVariadicGPR);
+  int GPRIdx = 0;
+  if (GPRSaveSize != 0) {
+    if (IsWin64CC) {
+      GPRIdx = MFI.CreateFixedObject(GPRSaveSize,
+                                     -static_cast<int>(GPRSaveSize), false);
+    } else
+      GPRIdx = MFI.CreateStackObject(GPRSaveSize, Align(8), false);
+
+    auto FIN = MIRBuilder.buildFrameIndex(p0, GPRIdx);
+    auto Offset =
+        MIRBuilder.buildConstant(MRI.createGenericVirtualRegister(s64), 8);
+
+    for (unsigned i = FirstVariadicGPR; i < GPRArgRegs.size(); ++i) {
+      Register Val = MRI.createGenericVirtualRegister(s64);
+      Handler.assignValueToReg(
+          Val, GPRArgRegs[i],
+          CCValAssign::getReg(i + MF.getFunction().getNumOperands(), MVT::i64,
+                              GPRArgRegs[i], MVT::i64, CCValAssign::Full));
+      auto MPO = IsWin64CC ? MachinePointerInfo::getFixedStack(
+                               MF, GPRIdx, (i - FirstVariadicGPR) * 8)
+                         : MachinePointerInfo::getStack(MF, i * 8);
+      MIRBuilder.buildStore(Val, FIN, MPO, inferAlignFromPtrInfo(MF, MPO));
+
+      FIN = MIRBuilder.buildPtrAdd(MRI.createGenericVirtualRegister(p0),
+                                   FIN.getReg(0), Offset);
+    }
+  }
+  FuncInfo->setVarArgsGPRIndex(GPRIdx);
+  FuncInfo->setVarArgsGPRSize(GPRSaveSize);
+
+  if (Subtarget.hasFPARMv8() && !IsWin64CC) {
+    unsigned FirstVariadicFPR = CCInfo.getFirstUnallocated(FPRArgRegs);
+
+    unsigned FPRSaveSize = 16 * (FPRArgRegs.size() - FirstVariadicFPR);
+    int FPRIdx = 0;
+    if (FPRSaveSize != 0) {
+      FPRIdx = MFI.CreateStackObject(FPRSaveSize, Align(16), false);
+
+      auto FIN = MIRBuilder.buildFrameIndex(p0, FPRIdx);
+      auto Offset =
+          MIRBuilder.buildConstant(MRI.createGenericVirtualRegister(s64), 16);
+
+      for (unsigned i = FirstVariadicFPR; i < FPRArgRegs.size(); ++i) {
+        Register Val = MRI.createGenericVirtualRegister(LLT::scalar(128));
+        Handler.assignValueToReg(
+            Val, FPRArgRegs[i],
+            CCValAssign::getReg(
+                i + MF.getFunction().getNumOperands() + NumVariadicGPRArgRegs,
+                MVT::f128, FPRArgRegs[i], MVT::f128, CCValAssign::Full));
+
+        auto MPO = MachinePointerInfo::getStack(MF, i * 16);
+        MIRBuilder.buildStore(Val, FIN, MPO, inferAlignFromPtrInfo(MF, MPO));
+
+        FIN = MIRBuilder.buildPtrAdd(MRI.createGenericVirtualRegister(p0),
+                                     FIN.getReg(0), Offset);
+      }
+    }
+    FuncInfo->setVarArgsFPRIndex(FPRIdx);
+    FuncInfo->setVarArgsFPRSize(FPRSaveSize);
+  }
+}
+
 bool AArch64CallLowering::lowerFormalArguments(
     MachineIRBuilder &MIRBuilder, const Function &F,
     ArrayRef<ArrayRef<Register>> VRegs, FunctionLoweringInfo &FLI) const {
   MachineFunction &MF = MIRBuilder.getMF();
   MachineBasicBlock &MBB = MIRBuilder.getMBB();
   MachineRegisterInfo &MRI = MF.getRegInfo();
   auto &DL = F.getParent()->getDataLayout();
+  auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+  // TODO: Support Arm64EC
+  bool IsWin64 = Subtarget.isCallingConvWin64(F.getCallingConv()) && !Subtarget.isWindowsArm64EC();
 
   SmallVector<ArgInfo, 8> SplitArgs;
   SmallVector<std::pair<Register, Register>> BoolArgs;
 
   // Insert the hidden sret parameter if the return value won't fit in the
   // return registers.
   if (!FLI.CanLowerReturn)
     insertSRetIncomingArgument(F, SplitArgs, FLI.DemoteRegister, MRI, DL);
for (auto &Arg : F.args()) {
     splitToValueTypes(OrigArg, SplitArgs, DL, F.getCallingConv());
     ++i;
   }
 
   if (!MBB.empty())
     MIRBuilder.setInstr(*MBB.begin());
 
   const AArch64TargetLowering &TLI = *getTLI<AArch64TargetLowering>();
-  CCAssignFn *AssignFn =
-      TLI.CCAssignFnForCall(F.getCallingConv(), /*IsVarArg=*/false);
+  CCAssignFn *AssignFn = TLI.CCAssignFnForCall(F.getCallingConv(), IsWin64 && F.isVarArg());
 
   AArch64IncomingValueAssigner Assigner(AssignFn, AssignFn);
   FormalArgHandler Handler(MIRBuilder, MRI);
-  if (!determineAndHandleAssignments(Handler, Assigner, SplitArgs, MIRBuilder,
-                                     F.getCallingConv(), F.isVarArg()))
+  SmallVector<CCValAssign, 16> ArgLocs;
+  CCState CCInfo(F.getCallingConv(), F.isVarArg(), MF, ArgLocs, F.getContext());
+  if (!determineAssignments(Assigner, SplitArgs, CCInfo) ||
+      !handleAssignments(Handler, SplitArgs, CCInfo, ArgLocs, MIRBuilder))
     return false;
 
   if (!BoolArgs.empty()) {
     for (auto &KV : BoolArgs) {
       Register OrigReg = KV.first;
       Register WideReg = KV.second;
       LLT WideTy = MRI.getType(WideReg);
       assert(MRI.getType(OrigReg).getScalarSizeInBits() == 1 &&
              "Unexpected bit size of a bool arg");
       MIRBuilder.buildTrunc(
           OrigReg, MIRBuilder.buildAssertZExt(WideTy, WideReg, 1).getReg(0));
     }
   }
 
   AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
   uint64_t StackOffset = Assigner.StackOffset;
   if (F.isVarArg()) {
-    auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
-    if (!Subtarget.isTargetDarwin()) {
-        // FIXME: we need to reimplement saveVarArgsRegisters from
-      // AArch64ISelLowering.
+    if ((!Subtarget.isTargetDarwin() && !Subtarget.isWindowsArm64EC()) || IsWin64) {
+      // The AAPCS variadic function ABI is identical to the non-variadic
+      // one. As a result there may be more arguments in registers and we should
+      // save them for future reference.
+      // Win64 variadic functions also pass arguments in registers, but all
+      // float arguments are passed in integer registers.
+      saveVarArgRegisters(MIRBuilder, Handler, CCInfo);
+    } else if (Subtarget.isWindowsArm64EC()) {
       return false;
     }
 
     // We currently pass all varargs at 8-byte alignment, or 4 in ILP32.
     StackOffset =
         alignTo(Assigner.StackOffset, Subtarget.isTargetILP32() ? 4 : 8);
 
     auto &MFI = MIRBuilder.getMF().getFrameInfo();
bool AArch64CallLowering::lowerFormalArguments(
   }
 
   // When we tail call, we need to check if the callee's arguments
   // will fit on the caller's stack. So, whenever we lower formal arguments,
   // we should keep track of this information, since we might lower a tail call
   // in this function later.
   FuncInfo->setBytesInStackArgArea(StackOffset);
 
-  auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   if (Subtarget.hasCustomCallingConv())
     Subtarget.getRegisterInfo()->UpdateCustomCalleeSavedRegs(MF);
 
   handleMustTailForwardedRegisters(MIRBuilder, AssignFn);
 
   // Move back to the end of the basic block.
   MIRBuilder.setMBB(MBB);
 

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

 
 bool AArch64InstructionSelector::selectVaStartDarwin(
     MachineInstr &I, MachineFunction &MF, MachineRegisterInfo &MRI) const {
   AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
   Register ListReg = I.getOperand(0).getReg();
 
   Register ArgsAddrReg = MRI.createVirtualRegister(&AArch64::GPR64RegClass);
 
+  int FrameIdx = FuncInfo->getVarArgsStackIndex();
+  if (MF.getSubtarget<AArch64Subtarget>().isCallingConvWin64(
+          MF.getFunction().getCallingConv())) {
+    FrameIdx = FuncInfo->getVarArgsGPRSize() > 0
+                   ? FuncInfo->getVarArgsGPRIndex()
+                   : FuncInfo->getVarArgsStackIndex();
+  }
+
   auto MIB =
       BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(AArch64::ADDXri))
           .addDef(ArgsAddrReg)
-          .addFrameIndex(FuncInfo->getVarArgsStackIndex())
+          .addFrameIndex(FrameIdx)
           .addImm(0)
           .addImm(0);
 
   constrainSelectedInstRegOperands(*MIB, TII, TRI, RBI);
 
   MIB = BuildMI(*I.getParent(), I, I.getDebugLoc(), TII.get(AArch64::STRXui))
             .addUse(ArgsAddrReg)
             .addUse(ListReg)

llvm/test/CodeGen/AArch64/GlobalISel/aapcs_vararg_frame.ll

+; RUN: llc < %s --global-isel=0 -mtriple=aarch64-linux-gnu -mattr=+fp-armv8 | FileCheck %s
+; RUN: llc < %s --global-isel=1 -mtriple=aarch64-linux-gnu -mattr=+fp-armv8 | FileCheck %s --check-prefix=GISEL
+
+define void @va(i32 %count, half %f, ...) nounwind {
+; CHECK-LABEL: va:
+; CHECK:       // %bb.0: // %entry
+; CHECK-NEXT:    sub sp, sp, #176
+; CHECK-NEXT:    stp x4, x5, [sp, #144]
+; CHECK-NEXT:    stp x2, x3, [sp, #128]
+; CHECK-NEXT:    str x1, [sp, #120]
+; CHECK-NEXT:    stp x6, x7, [sp, #160]
+; CHECK-NEXT:    stp q1, q2, [sp]
+; CHECK-NEXT:    stp q3, q4, [sp, #32]
+; CHECK-NEXT:    stp q5, q6, [sp, #64]
+; CHECK-NEXT:    str q7, [sp, #96]
+; CHECK-NEXT:    add sp, sp, #176
+; CHECK-NEXT:    ret
+;
+; GISEL-LABEL: va:
+; GISEL:       // %bb.0: // %entry
+; GISEL-NEXT:    sub sp, sp, #176
+; GISEL-NEXT:    stp x1, x2, [sp, #120]
+; GISEL-NEXT:    stp x3, x4, [sp, #136]
+; GISEL-NEXT:    stp x5, x6, [sp, #152]
+; GISEL-NEXT:    str x7, [sp, #168]
+; GISEL-NEXT:    stp q1, q2, [sp]
+; GISEL-NEXT:    stp q3, q4, [sp, #32]
+; GISEL-NEXT:    stp q5, q6, [sp, #64]
+; GISEL-NEXT:    str q7, [sp, #96]
+; GISEL-NEXT:    add sp, sp, #176
+; GISEL-NEXT:    ret
+entry:
+  ret void
+}

llvm/test/CodeGen/AArch64/aarch64_win64cc_vararg.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s
+; RUN: llc < %s --global-isel=1 -mtriple=aarch64-apple-darwin | FileCheck %s --check-prefix=DARWIN
 
 define win64cc void @pass_va(i32 %count, ...) nounwind {
 ; CHECK-LABEL: pass_va:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    sub sp, sp, #96
 ; CHECK-NEXT:    add x8, sp, #40
 ; CHECK-NEXT:    add x0, sp, #40
 ; CHECK-NEXT:    stp x30, x18, [sp, #16] // 16-byte Folded Spill
 ; CHECK-NEXT:    stp x3, x4, [sp, #56]
 ; CHECK-NEXT:    stp x1, x2, [sp, #40]
 ; CHECK-NEXT:    stp x5, x6, [sp, #72]
 ; CHECK-NEXT:    str x7, [sp, #88]
 ; CHECK-NEXT:    str x8, [sp, #8]
 ; CHECK-NEXT:    bl other_func
 ; CHECK-NEXT:    ldp x30, x18, [sp, #16] // 16-byte Folded Reload
 ; CHECK-NEXT:    add sp, sp, #96
 ; CHECK-NEXT:    ret
+;
+; DARWIN:       ; %bb.0: ; %entry
+; DARWIN-DAG:     stp x3, x4, [sp, #56]
+; DARWIN-DAG:     stp x1, x2, [sp, #40]
+; DARWIN-DAG:     stp x5, x6, [sp, #72]
+; DARWIN-DAG:     str x7, [sp, #88]
 entry:
   %ap = alloca ptr, align 8
   call void @llvm.va_start(ptr %ap)
   %ap2 = load ptr, ptr %ap, align 8
   call void @other_func(ptr %ap2)
   ret void
 }
 
 declare void @other_func(ptr) local_unnamed_addr
 
 declare void @llvm.va_start(ptr) nounwind
 declare void @llvm.va_copy(ptr, ptr) nounwind
 
 define win64cc ptr @f9(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, i64 %a7, i64 %a8, ...) nounwind {
 ; CHECK-LABEL: f9:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    str x18, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    add x8, sp, #24
 ; CHECK-NEXT:    add x0, sp, #24
 ; CHECK-NEXT:    str x8, [sp, #8]
 ; CHECK-NEXT:    ldr x18, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
+;
+; DARWIN-LABEL: _f9:
+; DARWIN:      ; %bb.0:                                ; %entry
+; DARWIN-NEXT:   str x18, [sp, #-16]!                ; 8-byte Folded Spill
+; DARWIN-NEXT:   add x8, sp, #8
+; DARWIN-NEXT:   add x9, sp, #24
+; DARWIN-NEXT:   str x9, [x8]
+; DARWIN-NEXT:   ldr x0, [sp, #8]
+; DARWIN-NEXT:   ldr x18, [sp], #16                  ; 8-byte Folded Reload
+; DARWIN-NEXT:   ret
 entry:
   %ap = alloca ptr, align 8
   call void @llvm.va_start(ptr %ap)
   %ap2 = load ptr, ptr %ap, align 8
   ret ptr %ap2
 }
 
 define win64cc ptr @f8(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, i64 %a7, ...) nounwind {
 ; CHECK-LABEL: f8:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    str x18, [sp, #-16]! // 8-byte Folded Spill
 ; CHECK-NEXT:    add x8, sp, #16
 ; CHECK-NEXT:    add x0, sp, #16
 ; CHECK-NEXT:    str x8, [sp, #8]
 ; CHECK-NEXT:    ldr x18, [sp], #16 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
+;
+; DARWIN-LABEL: _f8:
+; DARWIN:      ; %bb.0:                                ; %entry
+; DARWIN-NEXT:   str x18, [sp, #-16]!                ; 8-byte Folded Spill
+; DARWIN-NEXT:   add x8, sp, #8
+; DARWIN-NEXT:   add x9, sp, #16
+; DARWIN-NEXT:   str x9, [x8]
+; DARWIN-NEXT:   ldr x0, [sp, #8]
+; DARWIN-NEXT:   ldr x18, [sp], #16                  ; 8-byte Folded Reload
+; DARWIN-NEXT:   ret
 entry:
   %ap = alloca ptr, align 8
   call void @llvm.va_start(ptr %ap)
   %ap2 = load ptr, ptr %ap, align 8
   ret ptr %ap2
 }
 
 define win64cc ptr @f7(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, ...) nounwind {
 ; CHECK-LABEL: f7:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    str x18, [sp, #-32]! // 8-byte Folded Spill
 ; CHECK-NEXT:    add x8, sp, #24
 ; CHECK-NEXT:    add x0, sp, #24
 ; CHECK-NEXT:    str x7, [sp, #24]
 ; CHECK-NEXT:    str x8, [sp, #8]
 ; CHECK-NEXT:    ldr x18, [sp], #32 // 8-byte Folded Reload
 ; CHECK-NEXT:    ret
+;
+; DARWIN-LABEL: _f7:
+; DARWIN:      ; %bb.0:                                ; %entry
+; DARWIN-NEXT:   str x18, [sp, #-32]!                ; 8-byte Folded Spill
+; DARWIN-NEXT:   add x8, sp, #8
+; DARWIN-NEXT:   add x9, sp, #24
+; DARWIN-NEXT:   str x7, [sp, #24]
+; DARWIN-NEXT:   str x9, [x8]
+; DARWIN-NEXT:   ldr x0, [sp, #8]
+; DARWIN-NEXT:   ldr x18, [sp], #32                  ; 8-byte Folded Reload
+; DARWIN-NEXT:   ret
 entry:
   %ap = alloca ptr, align 8
   call void @llvm.va_start(ptr %ap)
   %ap2 = load ptr, ptr %ap, align 8
   ret ptr %ap2
 }

llvm/test/CodeGen/AArch64/win64_vararg_float_cc.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s --check-prefixes=DAGISEL
 ; RUN: llc < %s -mtriple=aarch64-windows -verify-machineinstrs -O0 -fast-isel | FileCheck %s --check-prefixes=O0,FASTISEL
 ; RUN: llc < %s -mtriple=aarch64-windows -verify-machineinstrs -O0 -global-isel | FileCheck %s --check-prefixes=O0,GISEL
 
+; Check that non-vararg functions compilation is not broken
+define win64cc float @foo(float %arg) nounwind {
+; GISEL-LABEL: foo:
+; GISEL-NEXT:  // %bb.0: // %entry
+; GISEL-NEXT:  ret
+entry:
+  ret float %arg
+}
+
 define win64cc void @float_va_fn(float %a, i32 %b, ...) nounwind {
 ; DAGISEL-LABEL: float_va_fn:
 ; DAGISEL:       // %bb.0: // %entry
 ; DAGISEL-NEXT:    str x30, [sp, #-64]! // 8-byte Folded Spill
 ; DAGISEL-NEXT:    add x8, sp, #16
 ; DAGISEL-NEXT:    fmov s0, w0
 ; DAGISEL-NEXT:    add x0, sp, #16
 ; DAGISEL-NEXT:    stp x3, x4, [sp, #24]