[AArch64] Fix windows vararg functions with floats in the fixed args

[mstorsjo]

Summary

On Windows, float arguments are normally passed in float registers
in the calling convention for regular functions. For variable
argument functions, floats are passed in integer registers. This
already was done correctly since many years.

However, the surprising bit was that floats among the fixed arguments
also are supposed to be passed in integer registers, contrary to regular
functions. (This also seems to be the behaviour on ARM though, both
on Windows, but also on e.g. hardfloat linux.)

In the calling convention, don't promote shorter floats to f64, but
convert them to integers of the same length. (Floats passed as part of
the actual variable arguments are promoted to double already on the
C/Clang level; the LLVM vararg calling convention doesn't do any
extra promotion of f32 to f64 - this matches how it works on X86 too.)

Technically, this is an ABI break compared to older LLVM versions,
but it fixes compatibility with the official platform ABI. (In practice,
floats among the fixed arguments in variable argument functions is
a pretty rare construct.)

[rnk]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

561–562

Can we put the logic here?

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

561–562

Hmm, that might be possible, I'll have a go at it.

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

561–562

Actually, it looks to me like it'd be more of a mess than what this patch is; getAssignFnsForCC is called in 5 places, and we'd need to have access to the MachineFunction (for querying the subtarget, whether the calling convention is a windows one, and for querying whether the function is a variadic one). E.g. in the case of doCallerAndCalleePassArgsTheSameWay below, we have the MachineFunction for the caller, but for the callee, we only have a CallLoweringInfo struct which only contains a MachineOperand.

[rnk]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

175

I'm guessing this would become UseVarArgsCCForFixed = IsWin && IsVarArg;

249

Really, I wanted to find a way to make this if more self-documenting.

261–262

Let's pre-calculate a boolean, something like UseVarArgsCCForFixed, and put that in the if above.

561–562

I think AArch64TargetLowering has the subtarget, but I see that it is private. You can use TLI to get the TargetMachine, which will tell you if we are on Windows, but it doesn't have access to the isCallingConvWin64 logic in AArch64Subtarget. That logic could be reimplemented inline here, but I think I agree, let's find another way.

[mstorsjo]

Adjusted AArch64CallLowering.cpp to use a UseVarArgsCCForFixed variable instead.

[rnk]

lgtm

[maxim-kuvyrkov]

@mstorsjo
Hi Martin,

Please consider backporting this and your other WoA fixes to release/12.x branch. They may or may not hit the upcoming LLVM 12.0.0 release, but, at least, they'll get into the subsequent update release.

[mstorsjo]

In D100365#2690807, @maxim-kuvyrkov wrote:

@mstorsjo
Hi Martin,

Please consider backporting this and your other WoA fixes to release/12.x branch. They may or may not hit the upcoming LLVM 12.0.0 release, but, at least, they'll get into the subsequent update release.

@rnk - What do you think of backporting this kind of fix to 12.0.1 - is it too "big" a change (as it changes the ABI we produce, even if it was wrong before) for that part of the release process?

[rnk]

In D100365#2692146, @mstorsjo wrote:

In D100365#2690807, @maxim-kuvyrkov wrote:

Please consider backporting this and your other WoA fixes to release/12.x branch. They may or may not hit the upcoming LLVM 12.0.0 release, but, at least, they'll get into the subsequent update release.

@rnk - What do you think of backporting this kind of fix to 12.0.1 - is it too "big" a change (as it changes the ABI we produce, even if it was wrong before) for that part of the release process?

I'd say go ahead and backport it. I think there are probably more MSVC/Clang WoA ABI boundaries than Clang <12.0.0/Clang 12.0.1 WoA ABI boundaries.

[arsenm]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

167

You don't need to add this parameter, you can check the CCState::isVarArg in the assignment function

[mstorsjo]

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

167

Ah, thanks! Patch upcoming.

llvm/lib/Target/AArch64/AArch64CallingConvention.td

def RetCC_AArch64_AAPCS : CallingConv<[
 
   CCIfType<[nxv2i1, nxv4i1, nxv8i1, nxv16i1],
            CCAssignToReg<[P0, P1, P2, P3]>>
 ]>;
 
 // Vararg functions on windows pass floats in integer registers
 let Entry = 1 in
 def CC_AArch64_Win64_VarArg : CallingConv<[
-  CCIfType<[f16, bf16, f32], CCPromoteToType<f64>>,
+  CCIfType<[f16, bf16], CCBitConvertToType<i16>>,
+  CCIfType<[f32], CCBitConvertToType<i32>>,
   CCIfType<[f64], CCBitConvertToType<i64>>,
   CCDelegateTo<CC_AArch64_AAPCS>
 ]>;
 
 // Windows Control Flow Guard checks take a single argument (the target function
 // address) and have no return value.
 let Entry = 1 in
 def CC_AArch64_Win64_CFGuard_Check : CallingConv<[

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (Ins[i].isOrigArg()) {
                                   /*AllowUnknown*/ true);
       MVT ActualMVT = ActualVT.isSimple() ? ActualVT.getSimpleVT() : MVT::Other;
       // If ActualMVT is i1/i8/i16, we should set LocVT to i8/i8/i16.
       if (ActualMVT == MVT::i1 || ActualMVT == MVT::i8)
         ValVT = MVT::i8;
       else if (ActualMVT == MVT::i16)
         ValVT = MVT::i16;
     }
-    CCAssignFn *AssignFn = CCAssignFnForCall(CallConv, /*IsVarArg=*/false);
+    bool UseVarArgCC = false;
+    if (IsWin64)
+      UseVarArgCC = isVarArg;
+    CCAssignFn *AssignFn = CCAssignFnForCall(CallConv, UseVarArgCC);
     bool Res =
         AssignFn(i, ValVT, ValVT, CCValAssign::Full, Ins[i].Flags, CCInfo);
     assert(!Res && "Call operand has unhandled type");
     (void)Res;
   }
   SmallVector<SDValue, 16> ArgValues;
   unsigned ExtraArgLocs = 0;
   for (unsigned i = 0, e = Ins.size(); i != e; ++i) {
AArch64TargetLowering::LowerCall(CallLoweringInfo &CLI,
 
   MachineFunction &MF = DAG.getMachineFunction();
   MachineFunction::CallSiteInfo CSInfo;
   bool IsThisReturn = false;
 
   AArch64FunctionInfo *FuncInfo = MF.getInfo<AArch64FunctionInfo>();
   bool TailCallOpt = MF.getTarget().Options.GuaranteedTailCallOpt;
   bool IsSibCall = false;
+  bool IsWin64 =
+      Subtarget->isCallingConvWin64(MF.getFunction().getCallingConv());
 
   // Check callee args/returns for SVE registers and set calling convention
   // accordingly.
   if (CallConv == CallingConv::C || CallConv == CallingConv::Fast) {
     bool CalleeOutSVE = any_of(Outs, [](ISD::OutputArg &Out){
       return Out.VT.isScalableVector();
     });
     bool CalleeInSVE = any_of(Ins, [](ISD::InputArg &In){
if (IsVarArg) {
 
     for (unsigned i = 0; i != NumArgs; ++i) {
       MVT ArgVT = Outs[i].VT;
       if (!Outs[i].IsFixed && ArgVT.isScalableVector())
         report_fatal_error("Passing SVE types to variadic functions is "
                            "currently not supported");
 
       ISD::ArgFlagsTy ArgFlags = Outs[i].Flags;
-      CCAssignFn *AssignFn = CCAssignFnForCall(CallConv,
-                                               /*IsVarArg=*/ !Outs[i].IsFixed);
+      bool UseVarArgCC = !Outs[i].IsFixed;
+      // On Windows, the fixed arguments in a vararg call are passed in GPRs
+      // too, so use the vararg CC to force them to integer registers.
+      if (IsWin64)
+        UseVarArgCC = true;
+      CCAssignFn *AssignFn = CCAssignFnForCall(CallConv, UseVarArgCC);
       bool Res = AssignFn(i, ArgVT, ArgVT, CCValAssign::Full, ArgFlags, CCInfo);
       assert(!Res && "Call operand has unhandled type");
       (void)Res;
     }
   } else {
     // At this point, Outs[].VT may already be promoted to i32. To correctly
     // handle passing i8 as i8 instead of i32 on stack, we pass in both i32 and
     // i8 to CC_AArch64_AAPCS with i32 being ValVT and i8 being LocVT.

llvm/lib/Target/AArch64/GISel/AArch64CallLowering.cpp

ReturnedArgCallReturnHandler(MachineIRBuilder &MIRBuilder,
       : CallReturnHandler(MIRBuilder, MRI, MIB, AssignFn) {}
 
   void markPhysRegUsed(MCRegister PhysReg) override {}
 };
 
 struct OutgoingArgHandler : public CallLowering::OutgoingValueHandler {
   OutgoingArgHandler(MachineIRBuilder &MIRBuilder, MachineRegisterInfo &MRI,
                      MachineInstrBuilder MIB, CCAssignFn *AssignFn,
-                     CCAssignFn *AssignFnVarArg, bool IsTailCall = false,
-                     int FPDiff = 0)
+                     CCAssignFn *AssignFnVarArg, bool IsVarArg,
+                     bool IsTailCall = false, int FPDiff = 0)
       : OutgoingValueHandler(MIRBuilder, MRI, AssignFn), MIB(MIB),
         AssignFnVarArg(AssignFnVarArg), IsTailCall(IsTailCall), FPDiff(FPDiff),
-        StackSize(0), SPReg(0) {}
+        StackSize(0), SPReg(0) {
+    MachineFunction &MF = MIRBuilder.getMF();
+    const auto &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+    bool IsWin =
+        Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
+    UseVarArgsCCForFixed = IsVarArg && IsWin;
+  }
 
   Register getStackAddress(uint64_t Size, int64_t Offset,
                            MachinePointerInfo &MPO,
                            ISD::ArgFlagsTy Flags) override {
     MachineFunction &MF = MIRBuilder.getMF();
     LLT p0 = LLT::pointer(0, 64);
     LLT s64 = LLT::scalar(64);
 
struct OutgoingArgHandler : public CallLowering::OutgoingValueHandler {
   }
 
   bool assignArg(unsigned ValNo, MVT ValVT, MVT LocVT,
                  CCValAssign::LocInfo LocInfo,
                  const CallLowering::ArgInfo &Info,
                  ISD::ArgFlagsTy Flags,
                  CCState &State) override {
     bool Res;
-    if (Info.IsFixed)
+    if (Info.IsFixed && !UseVarArgsCCForFixed)
       Res = AssignFn(ValNo, ValVT, LocVT, LocInfo, Flags, State);
     else
       Res = AssignFnVarArg(ValNo, ValVT, LocVT, LocInfo, Flags, State);
 
     StackSize = State.getNextStackOffset();
     return Res;
   }
 
   MachineInstrBuilder MIB;
   CCAssignFn *AssignFnVarArg;
   bool IsTailCall;
+  bool UseVarArgsCCForFixed;
 
   /// For tail calls, the byte offset of the call's argument area from the
   /// callee's. Unused elsewhere.
   int FPDiff;
   uint64_t StackSize;
 
   // Cache the SP register vreg if we need it more than once in this call site.
   Register SPReg;
 };
for (unsigned i = 0; i < SplitEVTs.size(); ++i) {
       if (CurVReg != CurArgInfo.Regs[0]) {
         CurArgInfo.Regs[0] = CurVReg;
         // Reset the arg flags after modifying CurVReg.
         setArgFlags(CurArgInfo, AttributeList::ReturnIndex, DL, F);
       }
       splitToValueTypes(CurArgInfo, SplitArgs, DL, CC);
     }
 
-    OutgoingArgHandler Handler(MIRBuilder, MRI, MIB, AssignFn, AssignFn);
+    OutgoingArgHandler Handler(MIRBuilder, MRI, MIB, AssignFn, AssignFn,
+                               F.isVarArg());
     Success =
         handleAssignments(MIRBuilder, SplitArgs, Handler, CC, F.isVarArg());
   }
 
   if (SwiftErrorVReg) {
     MIB.addUse(AArch64::X21, RegState::Implicit);
     MIRBuilder.buildCopy(AArch64::X21, SwiftErrorVReg);
   }
static bool mayTailCallThisCC(CallingConv::ID CC) {
   default:
     return canGuaranteeTCO(CC);
   }
 }
 
 /// Returns a pair containing the fixed CCAssignFn and the vararg CCAssignFn for
 /// CC.
 static std::pair<CCAssignFn *, CCAssignFn *>
 getAssignFnsForCC(CallingConv::ID CC, const AArch64TargetLowering &TLI) {
   return {TLI.CCAssignFnForCall(CC, false), TLI.CCAssignFnForCall(CC, true)};
 }
 
 bool AArch64CallLowering::doCallerAndCalleePassArgsTheSameWay(
     CallLoweringInfo &Info, MachineFunction &MF,
     SmallVectorImpl<ArgInfo> &InArgs) const {
   const Function &CallerF = MF.getFunction();
   CallingConv::ID CalleeCC = Info.CallConv;
   CallingConv::ID CallerCC = CallerF.getCallingConv();
if (!IsSibCall) {
     // satisfy the same constraint.
     assert(FPDiff % 16 == 0 && "unaligned stack on tail call");
   }
 
   const auto &Forwards = FuncInfo->getForwardedMustTailRegParms();
 
   // Do the actual argument marshalling.
   OutgoingArgHandler Handler(MIRBuilder, MRI, MIB, AssignFnFixed,
-                             AssignFnVarArg, true, FPDiff);
+                             AssignFnVarArg, Info.IsVarArg, true, FPDiff);
   if (!handleAssignments(MIRBuilder, OutArgs, Handler, CalleeCC, Info.IsVarArg))
     return false;
 
   Mask = getMaskForArgs(OutArgs, Info, *TRI, MF);
 
   if (Info.IsVarArg && Info.IsMustTailCall) {
     // Now we know what's being passed to the function. Add uses to the call for
     // the forwarded registers that we *aren't* passing as parameters. This will
bool AArch64CallLowering::lowerCall(MachineIRBuilder &MIRBuilder,
   MIB.add(Info.Callee);
 
   // Tell the call which registers are clobbered.
   const uint32_t *Mask;
   const auto *TRI = MF.getSubtarget<AArch64Subtarget>().getRegisterInfo();
 
   // Do the actual argument marshalling.
   OutgoingArgHandler Handler(MIRBuilder, MRI, MIB, AssignFnFixed,
-                             AssignFnVarArg, false);
+                             AssignFnVarArg, Info.IsVarArg, false);
   if (!handleAssignments(MIRBuilder, OutArgs, Handler, Info.CallConv,
                          Info.IsVarArg))
     return false;
 
   Mask = getMaskForArgs(OutArgs, Info, *TRI, MF);
 
   if (MF.getSubtarget<AArch64Subtarget>().hasCustomCallingConv())
     TRI->UpdateCustomCallPreservedMask(MF, &Mask);

llvm/test/CodeGen/AArch64/GlobalISel/call-translator-tail-call.ll

define void @test_varargs() {
   ; DARWIN:   $x1 = COPY [[C2]](s64)
   ; DARWIN:   TCRETURNdi @varargs, 0, csr_darwin_aarch64_aapcs, implicit $sp, implicit $w0, implicit $d0, implicit $x1
   ; WINDOWS-LABEL: name: test_varargs
   ; WINDOWS: bb.1 (%ir-block.0):
   ; WINDOWS:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; WINDOWS:   [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 1.000000e+00
   ; WINDOWS:   [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 12
   ; WINDOWS:   $w0 = COPY [[C]](s32)
-  ; WINDOWS:   $d0 = COPY [[C1]](s64)
-  ; WINDOWS:   $x1 = COPY [[C2]](s64)
-  ; WINDOWS:   TCRETURNdi @varargs, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0, implicit $d0, implicit $x1
+  ; WINDOWS:   $x1 = COPY [[C1]](s64)
+  ; WINDOWS:   $x2 = COPY [[C2]](s64)
+  ; WINDOWS:   TCRETURNdi @varargs, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0, implicit $x1, implicit $x2
   tail call void(i32, double, i64, ...) @varargs(i32 42, double 1.0, i64 12)
   ret void
 }
 
 ; Darwin should not tail call here, because the last parameter to @varargs is
 ; not fixed. So, it's passed on the stack, which will make us not fit. On
 ; Windows, it's passed in a register, so it's safe to tail call.
 define void @test_varargs_2() {
define void @test_varargs_2() {
   ; DARWIN:   RET_ReallyLR
   ; WINDOWS-LABEL: name: test_varargs_2
   ; WINDOWS: bb.1 (%ir-block.0):
   ; WINDOWS:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; WINDOWS:   [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 1.000000e+00
   ; WINDOWS:   [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 12
   ; WINDOWS:   [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 314
   ; WINDOWS:   $w0 = COPY [[C]](s32)
-  ; WINDOWS:   $d0 = COPY [[C1]](s64)
-  ; WINDOWS:   $x1 = COPY [[C2]](s64)
-  ; WINDOWS:   $x2 = COPY [[C3]](s64)
-  ; WINDOWS:   TCRETURNdi @varargs, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0, implicit $d0, implicit $x1, implicit $x2
+  ; WINDOWS:   $x1 = COPY [[C1]](s64)
+  ; WINDOWS:   $x2 = COPY [[C2]](s64)
+  ; WINDOWS:   $x3 = COPY [[C3]](s64)
+  ; WINDOWS:   TCRETURNdi @varargs, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0, implicit $x1, implicit $x2, implicit $x3
   tail call void(i32, double, i64, ...) @varargs(i32 42, double 1.0, i64 12, i64 314)
   ret void
 }
 
 ; Same deal here, even though we have enough room to fit. On Darwin, we'll pass
 ; the last argument to @varargs on the stack. We don't allow tail calling
 ; varargs arguments that are on the stack.
 define void @test_varargs_3([8 x <2 x double>], <4 x half> %arg) {
define void @test_varargs_3([8 x <2 x double>], <4 x half> %arg) {
   ; WINDOWS:   [[COPY7:%[0-9]+]]:_(<2 x s64>) = COPY $q7
   ; WINDOWS:   [[FRAME_INDEX:%[0-9]+]]:_(p0) = G_FRAME_INDEX %fixed-stack.0
   ; WINDOWS:   [[LOAD:%[0-9]+]]:_(<4 x s16>) = G_LOAD [[FRAME_INDEX]](p0) :: (invariant load 8 from %fixed-stack.0, align 16)
   ; WINDOWS:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 42
   ; WINDOWS:   [[C1:%[0-9]+]]:_(s64) = G_FCONSTANT double 1.000000e+00
   ; WINDOWS:   [[C2:%[0-9]+]]:_(s64) = G_CONSTANT i64 12
   ; WINDOWS:   [[C3:%[0-9]+]]:_(s64) = G_CONSTANT i64 314
   ; WINDOWS:   $w0 = COPY [[C]](s32)
-  ; WINDOWS:   $d0 = COPY [[C1]](s64)
-  ; WINDOWS:   $x1 = COPY [[C2]](s64)
-  ; WINDOWS:   $x2 = COPY [[C3]](s64)
-  ; WINDOWS:   TCRETURNdi @varargs, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0, implicit $d0, implicit $x1, implicit $x2
+  ; WINDOWS:   $x1 = COPY [[C1]](s64)
+  ; WINDOWS:   $x2 = COPY [[C2]](s64)
+  ; WINDOWS:   $x3 = COPY [[C3]](s64)
+  ; WINDOWS:   TCRETURNdi @varargs, 0, csr_aarch64_aapcs, implicit $sp, implicit $w0, implicit $x1, implicit $x2, implicit $x3
   tail call void(i32, double, i64, ...) @varargs(i32 42, double 1.0, i64 12, i64 314)
   ret void
 }
 
 ; Unsupported calling convention for tail calls. Make sure we never tail call
 ; it.
 declare ghccc void @bad_call_conv_fn()
 define void @test_bad_call_conv() {

llvm/test/CodeGen/AArch64/win64_vararg_float.ll

+; RUN: llc < %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s --check-prefixes=CHECK,DAGISEL
+; RUN: llc < %s -mtriple=aarch64-windows -verify-machineinstrs -O0 -fast-isel | FileCheck %s --check-prefixes=CHECK,O0
+; RUN: llc < %s -mtriple=aarch64-windows -verify-machineinstrs -O0 -global-isel | FileCheck %s --check-prefixes=CHECK,O0
+
+define void @float_va_fn(float %a, i32 %b, ...) nounwind {
+entry:
+; CHECK-LABEL: float_va_fn:
+; O0: str x7, [sp, #72]
+; O0: str x6, [sp, #64]
+; O0: str x5, [sp, #56]
+; O0: str x4, [sp, #48]
+; O0: str x3, [sp, #40]
+; O0: str x2, [sp, #32]
+; CHECK: fmov s0, w0
+; O0: add x8, sp, #32
+; O0: str x8, [sp, #8]
+; O0: ldr x0, [sp, #8]
+; DAGISEL: add x0, sp, #16
+; DAGISEL: stp x2, x3, [sp, #16]
+; DAGISEL: stp x4, x5, [sp, #32]
+; DAGISEL: stp x6, x7, [sp, #48]
+; CHECK: bl f_va_list
+  %ap = alloca i8*, align 8
+  %0 = bitcast i8** %ap to i8*
+  call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %0)
+  call void @llvm.va_start(i8* nonnull %0)
+  %1 = load i8*, i8** %ap, align 8
+  call void @f_va_list(float %a, i8* %1)
+  call void @llvm.va_end(i8* nonnull %0)
+  call void @llvm.lifetime.end.p0i8(i64 8, i8* nonnull %0)
+  ret void
+}
+
+declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture)
+declare void @llvm.va_start(i8*)
+declare void @f_va_list(float, i8*)
+declare void @llvm.va_end(i8*)
+declare void @llvm.lifetime.end.p0i8(i64 immarg, i8* nocapture)
+
+define void @double_va_fn(double %a, i32 %b, ...) nounwind {
+entry:
+; CHECK-LABEL: double_va_fn:
+; O0: str x7, [sp, #72]
+; O0: str x6, [sp, #64]
+; O0: str x5, [sp, #56]
+; O0: str x4, [sp, #48]
+; O0: str x3, [sp, #40]
+; O0: str x2, [sp, #32]
+; CHECK: fmov d0, x0
+; O0: add x8, sp, #32
+; O0: str x8, [sp, #8]
+; O0: ldr x0, [sp, #8]
+; DAGISEL: add x0, sp, #16
+; DAGISEL: stp x2, x3, [sp, #16]
+; DAGISEL: stp x4, x5, [sp, #32]
+; DAGISEL: stp x6, x7, [sp, #48]
+; CHECK: bl d_va_list
+  %ap = alloca i8*, align 8
+  %0 = bitcast i8** %ap to i8*
+  call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %0)
+  call void @llvm.va_start(i8* nonnull %0)
+  %1 = load i8*, i8** %ap, align 8
+  call void @d_va_list(double %a, i8* %1)
+  call void @llvm.va_end(i8* nonnull %0)
+  call void @llvm.lifetime.end.p0i8(i64 8, i8* nonnull %0)
+  ret void
+}
+
+declare void @d_va_list(double, i8*)
+
+define void @call_f_va() nounwind {
+entry:
+; CHECK-LABEL: call_f_va:
+; DAGISEL: mov w0, #1065353216
+; FASTISEL: mov w0, #1065353216
+; GISEL: fmov s0, #1.00000000
+; GISEL: fmov w0, s0
+; CHECK: mov w1, #2
+; DAGISEL: mov x2, #4613937818241073152
+; FASTISEL: mov x2, #4613937818241073152
+; GISEL: fmov d0, #3.00000000
+; GISEL: fmov x2, d0
+; CHECK: mov w3, #4
+; CHECK: b other_f_va_fn
+  tail call void (float, i32, ...) @other_f_va_fn(float 1.000000e+00, i32 2, double 3.000000e+00, i32 4)
+  ret void
+}
+
+declare void @other_f_va_fn(float, i32, ...)
+
+define void @call_d_va() nounwind {
+entry:
+; CHECK-LABEL: call_d_va:
+; DAGISEL: mov x0, #4607182418800017408
+; FASTISEL: mov x0, #4607182418800017408
+; GISEL: fmov d0, #1.00000000
+; GISEL: fmov x0, d0
+; CHECK: mov w1, #2
+; DAGISEL: mov x2, #4613937818241073152
+; FASTISEL: mov x2, #4613937818241073152
+; GISEL: fmov d0, #3.00000000
+; CHECK: mov w3, #4
+; CHECK: b other_d_va_fn
+  tail call void (double, i32, ...) @other_d_va_fn(double 1.000000e+00, i32 2, double 3.000000e+00, i32 4) #4
+  ret void
+}
+
+declare dso_local void @other_d_va_fn(double, i32, ...)