[SystemZ] [z/OS] Add XPLINK64 Calling Convention to SystemZ

[Everybody0523]

Summary

This patch adds the XPLINK64 calling convention to the SystemZ backend. It specifies and implements the argument passing and return value conventions.

[Everybody0523]

Formatting/style changes.

[Everybody0523]

More style/format issues.

[uweigand]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

27

Please remove this unnecessary whitespace change here.

166

Why are you changing the sequence here? I didn't think the order of CSRs in that list should make any particular difference ...

177

Please revisit the comments here and below and look for wording and formatting issues. This comment would fit on one line and needs a '.' and the end.

188

The last sentence doesn't seem to be correct -- according to the CSR definition above there are call-saved FPRs.

194

Space after ','.

199

Should we also provide more vector return registers for ABI extensions? Usually it should be fine to use all argument registers for returns too.

212

If you want to have a comment here describing how the ABI works in general terms, that's of course good, but I'm not sure if the long table here is particularly useful. Maybe describing it in terms of rules would be better? Or, even better, providing a link to an offical ABI doc?

249

So a difference to Linux ABI code here is that in Linux we use CCIfExtend, i.e. the extension only happens if the argument is marked using "sext" or "zext" in the IR (which gets set from clang depending on the source type, or omitted e.g. if we're passing a struct in register that cannot be "extended" as such). This code here calls CCPromoteToType unconditionally, so you'll get AnyExtend nodes in cases where there is no "sext" or "zext" marker. Is this intended?

306

That last comment describes an action that doesn't appear to be done here, right?

[Everybody0523]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

166

The CSR mask is generated in order based on this list. However, FPRs are preserved in the save area in reverse order, which is contrary to the current CSR mask. The previous commit not reversing the order of the callee-saved FPRs was an oversight.

With all that said and done, if you prefer I can keep the sequence unchanged for now and create a separate patch that fixes this bug.

[uweigand]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

166

If you need to save FPRs in some specific place in the save area for ABI reasons, that should be handled by code in SystemZFrameLowering.cpp (like is done for GPRs on Linux -- for FPRs there is no defined place where they need to be saved so it doesn't matter).

But in any case this is an independent question, so I agree that this should best be handled in a separate patch.

[Kai]

llvm/lib/Target/SystemZ/SystemZCallingConv.h

187

Add . (dot) at end of sentence.

[kpn]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

194

What is this ABI non-compliant code that's getting support?

How does a long double complex get returned?

212

The comment seems misleading. Integer types get expanded to fill a whole GPR. So passing three chars uses 3 registers -> 24 bytes. Talking about bytes therefore doesn't seem useful. Worse, it's misleading to refer to "bytes" for GPR and not for the FPR since it implies that parms get packed or something. I see that you addressed this below, but it'd be better to not even provoke the thought in someone reading the code. I'd stick with "registers" instead of "bytes" in this sentence.

One has to look closely at the table to see that FP parms can prevent a GPR from being used that would be used if there were no FP parms. So that bit could use a description in english.

A link to the ABI doc would be useful, or at least the name of it. Since links go stale, the name is probably a minimum.

Actually, how about copying the tables that are (were?) in the ABI documentation? Last time I looked IBM had a list of examples showing how FP values can push around GPR values, and how large numbers of integer parms can prevent FP values from being passed in FP registers despite FP registers being available. (I admit it's been years since I looked at the documentation.)

279

Don't you need to hedge a bit in this comment? Some FP arguments aren't passed in registers if there are too many integer arguments. (Unless I'm wrong and the restriction only applies to 32-bit XPLINK?)

[Everybody0523]

Formatting issues in tablegen file + remove fix for unrelated issue.

[Everybody0523]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

279

If I am not mistaken you are referring to the requirement that a FP argument is within 15 words of the previous FP argument in the argument list?

If so, then yes - XPLINK64 has no such clause.

[kpn]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

279

That's the one, yes. It must be a 31-bit only restriction then. Noted. Thanks for the clarification!

[kpn]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

212

Now that I've looked at the current version of Vendor Interfaces, the table I was thinking of is in Appendix B, from pages 924 to 931. That's ... a lot.

Oh, and appendix B doesn't include the documentation that is elsewhere in the book. XPLINK calling conventions require that much documentation. So a reference to the documentation may be better than trying to duplicate it.

Ulrich, how about this: Instead of having this table below, and instead of trying to put 6-7 pages of documentation in comments here, how about a comment about how there are a number of common tricky cases and they're documented in <documentation bib entry here>?

[Everybody0523]

Added comment on how complex f128 types are passed. Added explanation on how floating point parameters could affect availability of GPRs for parameter passing. Added name of document that contains full xplink spec. Fixed some grammatical issues. Added CCIfExtend clause to XPLINK argument promotion.

[Everybody0523]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

249

The clang front-end should generate sext/zext for those types anyways so we should be using CCIfExtend as well. Thanks!

[Everybody0523]

Add all param passing vector registers as return value regs too. Fix up some punctuation.

[Everybody0523]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

212

I've updated the comments to include a note on when the full specification of the ABI can be found.

I am hesitant to include a link for reason you mentioned above - links can go stale. Given the name of the document, it shouldn't be hard to find the latest version of the LE vendor interfaces.

[uweigand]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

212

I agree the current wording of the comment seems fine to me.

224

One more thing I just noticed here (b.t.w. applies also to the return value convention): I don't think we can ever end up with i1, i8, or i16 here, because those types are not "legal" in the SystemZ back-end, so they will already have been converted to i32 by common code (type legalization). That is why the Linux ABI code only checks for i32 here.

I think this would still hold true for the XPLINK convention -- if so, it would be preferable to remove the small types here as well. (This should have no effect on codegen, just remove an unnecessary difference between the code for the two calling conventions.)

235

Just to re-confirm: this handler will pass an i128 via implicit pointer (i.e. push the i128 to the stack somewhere outside the argument list, and pass a pointer to that location as argument). Is that actually correct for XPLINK? Reading the specs in the documentation referred to above, I get the impression that XPLINK never passes arguments via implicit pointers ...

260

One more comment wording nit-pick: I'd say "The first 2 named long double arguments" just like the in previous two clauses, and then remove the "Unless ..." sentence. We should just the same wording for the same concept across those different clauses.

[uweigand]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

194

Non ABI-compliant code here refers to LLVM front-ends other than clang, e.g. for languages other than C/C++ or for JIT use cases. Some of those use cases require multiple return values to be passed efficiently. While you could require any such users to define their own calling convention for that, this is quite tedious (and requires back-end changes). Therefore, it most LLVM back-ends provide an extension in their "default" calling convention for multiple return values, which is never used by LLVM IR generated by clang, but is available for these other use case.

However, if it is indeed true that in XPLINK, the C/C++ complex long double is already returned in four FPRs, then that would not could as "non ABI-compliant" use. But that's just a comment wording issue.

286

Note that the ELF calling convention makes special provisions for "short" vector types, i.e. those defined using attribute((vector_size(8))) or shorter. They may be passed in VRs like the "normal" 16 byte vectors, but if none are available they'll not get passed in a normal 16-byte stack slot, but rather in an 8-byte stack slot.

I notice that this provision is not in place with the XPLINK convention. Now I guess this is a decision for you to make, given that such types would be an extension to the ABI anyway. I just wanted to make sure that this is considered choice and not just happening by accident.

[kpn]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

212

Fair enough!

[Everybody0523]

Remove special handling for i1, i8, i16 in calling convention. SDAG legalize types phase should already extend these, there is no need to add handling to extend these again.

[Everybody0523]

llvm/lib/Target/SystemZ/SystemZCallingConv.td

235

Yes, that's the intention.

286

Hi Uli, we actually hadn't considered that - thank you for pointing this out.

After thinking about it for a while I think we should keep the current behavior, ie. pass in a 16-byte stack slot. The reason for this is if were to copy the ELF behavior, and pass short vector types in a 8-byte stack slot if passed on the stack, we would have to do one of two things.

1. Have short vector types passed-by-register "shadow" 8 bytes in the argument area. However this would mean that vector registers can shadow a variable amount of space on the argument list depending of the type of the parameter contained within, which is very strange.
2. Have short vector types passed-by-register "shadow" 16 bytes in the argument area, but those passed-by-stack occupy 8 bytes. This means that two params of the same type would occupy a different amount of memory in the argument list depending on whether it is passed-by-register or passed-by-stack - also very strange.

[uweigand]

So it looks like this is good to go now. LGTM, thanks!

llvm/lib/Target/SystemZ/SystemZCallingConv.td

235

OK, thanks for confirming.

286

I see, that does make sense to me as well.

llvm/lib/Target/SystemZ/SystemZCallingConv.h

 //===-- SystemZCallingConv.h - Calling conventions for SystemZ --*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_LIB_TARGET_SYSTEMZ_SYSTEMZCALLINGCONV_H
 #define LLVM_LIB_TARGET_SYSTEMZ_SYSTEMZCALLINGCONV_H
 
+#include "SystemZSubtarget.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/MC/MCRegisterInfo.h"
 
 namespace llvm {
 namespace SystemZ {
   const unsigned ELFNumArgGPRs = 5;
   extern const MCPhysReg ELFArgGPRs[ELFNumArgGPRs];
 
   const unsigned ELFNumArgFPRs = 4;
   extern const MCPhysReg ELFArgFPRs[ELFNumArgFPRs];
+
+  const unsigned XPLINK64NumArgGPRs = 3;
+  extern const MCPhysReg XPLINK64ArgGPRs[XPLINK64NumArgGPRs];
+
+  const unsigned XPLINK64NumArgFPRs = 4;
+  extern const MCPhysReg XPLINK64ArgFPRs[XPLINK64NumArgFPRs];
 } // end namespace SystemZ
 
 class SystemZCCState : public CCState {
 private:
   /// Records whether the value was a fixed argument.
   /// See ISD::OutputArg::IsFixed.
   SmallVector<bool, 4> ArgIsFixed;
 
inline bool CC_SystemZ_I128Indirect(unsigned &ValNo, MVT &ValVT,
   PendingMembers.push_back(CCValAssign::getPending(ValNo, ValVT,
                                                    LocVT, LocInfo));
   if (!ArgFlags.isSplitEnd())
     return true;
 
   // OK, we've collected all parts in the pending list.  Allocate
   // the location (register or stack slot) for the indirect pointer.
   // (This duplicates the usual i64 calling convention rules.)
-  unsigned Reg = State.AllocateReg(SystemZ::ELFArgGPRs);
+  unsigned Reg;
+  const SystemZSubtarget &Subtarget =
+      State.getMachineFunction().getSubtarget<SystemZSubtarget>();
+  if (Subtarget.isTargetELF())
+    Reg = State.AllocateReg(SystemZ::ELFArgGPRs);
+  else if (Subtarget.isTargetXPLINK64())
+    Reg = State.AllocateReg(SystemZ::XPLINK64ArgGPRs);
+  else
+    llvm_unreachable("Unknown Calling Convention!");
+
   unsigned Offset = Reg ? 0 : State.AllocateStack(8, Align(8));
 
   // Use that same location for all the pending parts.
   for (auto &It : PendingMembers) {
     if (Reg)
       It.convertToReg(Reg);
     else
       It.convertToMem(Offset);
     State.addLoc(It);
   }
 
   PendingMembers.clear();
 
   return true;
 }
 
+inline bool CC_XPLINK64_Shadow_Reg(unsigned &ValNo, MVT &ValVT, MVT &LocVT,
+                                   CCValAssign::LocInfo &LocInfo,
+                                   ISD::ArgFlagsTy &ArgFlags, CCState &State) {
+  if (LocVT == MVT::f32 || LocVT == MVT::f64) {
+    State.AllocateReg(SystemZ::XPLINK64ArgGPRs);
+  }
+  if (LocVT == MVT::f128 || LocVT.is128BitVector()) {
+    // Shadow next two GPRs, if available.
+    State.AllocateReg(SystemZ::XPLINK64ArgGPRs);
+    State.AllocateReg(SystemZ::XPLINK64ArgGPRs);
+
+    // Quad precision floating point needs to
+    // go inside pre-defined FPR pair.
+    if (LocVT == MVT::f128) {
+      for (unsigned I = 0; I < SystemZ::XPLINK64NumArgFPRs; I += 2)
+        if (State.isAllocated(SystemZ::XPLINK64ArgFPRs[I]))
+          State.AllocateReg(SystemZ::XPLINK64ArgFPRs[I + 1]);
+    }
+  }
+  return false;
+}
+
+inline bool CC_XPLINK64_Allocate128BitVararg(unsigned &ValNo, MVT &ValVT,
+                                             MVT &LocVT,
+                                             CCValAssign::LocInfo &LocInfo,
+                                             ISD::ArgFlagsTy &ArgFlags,
+                                             CCState &State) {
+  if (LocVT.getSizeInBits() < 128)
+    return false;
+
+  if (static_cast<SystemZCCState *>(&State)->IsFixed(ValNo))
+    return false;
+
+  // For any C or C++ program, this should always be
+  // false, since it is illegal to have a function
+  // where the first argument is variadic. Therefore
+  // the first fixed argument should already have
+  // allocated GPR1 either through shadowing it or
+  // using it for parameter passing.
+  State.AllocateReg(SystemZ::R1D);
+
+  bool AllocGPR2 = State.AllocateReg(SystemZ::R2D);
+  bool AllocGPR3 = State.AllocateReg(SystemZ::R3D);
+
+  // If GPR2 and GPR3 are available, then we may pass vararg in R2Q.
+  if (AllocGPR2 && AllocGPR3) {
+    State.addLoc(
+        CCValAssign::getReg(ValNo, ValVT, SystemZ::R2Q, LocVT, LocInfo));
+    return true;
+  }
+
+  // If only GPR3 is available, we allocate on stack but need to
+  // set custom handling to copy hi bits into GPR3.
+  if (!AllocGPR2 && AllocGPR3) {
+    auto Offset = State.AllocateStack(16, Align(8));
+    State.addLoc(
+        CCValAssign::getCustomMem(ValNo, ValVT, Offset, LocVT, LocInfo));
+    return true;
+  }
+
+  return false;
+}
+
+inline bool RetCC_SystemZ_Error(unsigned &, MVT &, MVT &,
+                                CCValAssign::LocInfo &, ISD::ArgFlagsTy &,
+                                CCState &) {
+  llvm_unreachable("Return value calling convention currently unsupported.");
+}
+
+inline bool CC_SystemZ_Error(unsigned &, MVT &, MVT &, CCValAssign::LocInfo &,
+                             ISD::ArgFlagsTy &, CCState &) {
+  llvm_unreachable("Argument calling convention currently unsupported.");
+}
+
 inline bool CC_SystemZ_GHC_Error(unsigned &, MVT &, MVT &,
                                  CCValAssign::LocInfo &, ISD::ArgFlagsTy &,
                                  CCState &) {
   report_fatal_error("No registers left in GHC calling convention");
   return false;
 }
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/SystemZ/SystemZCallingConv.cpp

 
 const MCPhysReg SystemZ::ELFArgGPRs[SystemZ::ELFNumArgGPRs] = {
   SystemZ::R2D, SystemZ::R3D, SystemZ::R4D, SystemZ::R5D, SystemZ::R6D
 };
 
 const MCPhysReg SystemZ::ELFArgFPRs[SystemZ::ELFNumArgFPRs] = {
   SystemZ::F0D, SystemZ::F2D, SystemZ::F4D, SystemZ::F6D
 };
+
+// The XPLINK64 ABI-defined param passing general purpose registers
+const MCPhysReg SystemZ::XPLINK64ArgGPRs[SystemZ::XPLINK64NumArgGPRs] = {
+    SystemZ::R1D, SystemZ::R2D, SystemZ::R3D
+};
+
+// The XPLINK64 ABI-defined param passing floating point registers
+const MCPhysReg SystemZ::XPLINK64ArgFPRs[SystemZ::XPLINK64NumArgFPRs] = {
+    SystemZ::F0D, SystemZ::F2D, SystemZ::F4D, SystemZ::F6D
+};

llvm/lib/Target/SystemZ/SystemZCallingConv.td

class CCIfSubtarget<string F, CCAction A>
   : CCIf<!strconcat("static_cast<const SystemZSubtarget&>"
                     "(State.getMachineFunction().getSubtarget()).", F),
          A>;
 
 // Match if this specific argument is a fixed (i.e. named) argument.
 class CCIfFixed<CCAction A>
     : CCIf<"static_cast<SystemZCCState *>(&State)->IsFixed(ValNo)", A>;
 
+// Match if this specific argument is not a fixed (i.e. vararg) argument.
+class CCIfNotFixed<CCAction A>
+    : CCIf<"!(static_cast<SystemZCCState *>(&State)->IsFixed(ValNo))", A>;
+
 // Match if this specific argument was widened from a short vector type.
 class CCIfShortVector<CCAction A>
     : CCIf<"static_cast<SystemZCCState *>(&State)->IsShortVector(ValNo)", A>;
 
 
 //===----------------------------------------------------------------------===//
 // z/Linux return value calling convention
 //===----------------------------------------------------------------------===//
 def RetCC_SystemZ_ELF : CallingConv<[
   // Promote i32 to i64 if it has an explicit extension type.
   CCIfType<[i32], CCIfExtend<CCPromoteToType<i64>>>,
 
   // A SwiftError is returned in R9.
def CSR_SystemZ_AllRegs_Vector : CalleeSavedRegs<(add (sequence "R%dD", 2, 15),
                                                       (sequence "V%d", 0, 31))>;
 
 def CSR_SystemZ_NoRegs : CalleeSavedRegs<(add)>;
 
 //===----------------------------------------------------------------------===//
 // z/OS XPLINK64 callee-saved registers
 //===----------------------------------------------------------------------===//
 def CSR_SystemZ_XPLINK64 : CalleeSavedRegs<(add (sequence "R%dD", 8, 15),
                                                 (sequence "F%dD", 8, 15))>;
 
+def CSR_SystemZ_XPLINK64_Vector : CalleeSavedRegs<(add (sequence "R%dD", 8, 15),
+                                                       (sequence "F%dD", 15, 8),
+                                                       (sequence "V%d", 23, 16))>;
+
+//===----------------------------------------------------------------------===//
+// z/OS XPLINK64 return value calling convention
+//===----------------------------------------------------------------------===//
+def RetCC_SystemZ_XPLINK64 : CallingConv<[
+  // XPLINK64 ABI compliant code widens i1, i8, i16, i32 to i64.
+  CCIfType<[i1, i8, i16, i32], CCPromoteToType<i64>>,
+
+  // Structs of size 1-24 bytes are returned in R1D, R2D, and R3D.
+  CCIfType<[i64], CCIfInReg<CCAssignToReg<[R1D, R2D, R3D]>>>,
+  // An i64 is returned in R3D. R2D and R1D provided for ABI non-compliant
+  // code.
+  CCIfType<[i64], CCAssignToReg<[R3D, R2D, R1D]>>,
+
+  // ABI compliant code returns floating point values in FPR0, FPR2, FPR4
+  // and FPR6, using as many registers as required.
+  // All floating point return-value registers are call-clobbered.
+  CCIfType<[f32], CCAssignToReg<[F0S, F2S, F4S, F6S]>>,
+  CCIfType<[f64], CCAssignToReg<[F0D, F2D, F4D, F6D]>>,
+
+  // ABI compliant code returns f128 in F0D and F2D, hence F0Q.
+  // F4D and F6D, hence F4Q are used for complex long double types.
+  CCIfType<[f128], CCAssignToReg<[F0Q,F4Q]>>,
+
+  // ABI compliant code returns vectors in VR24
+  CCIfSubtarget<"hasVector()",
+    CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+             CCAssignToReg<[V24]>>>
+]>;
+
+//===----------------------------------------------------------------------===//
+// z/OS XPLINK64 argument calling conventions
+//===----------------------------------------------------------------------===//
+// XPLink uses a logical argument list consisting of contiguous register-size
+// words (8 bytes in 64-Bit mode) where some arguments are passed in registers
+// and some in storage.
+// Even though 3 GPRs, 4 FPRs, and 8 VRs may be used,
+// space must be reserved for all the args on stack.
+// The first three register-sized words of the parameter area are passed in
+// GPRs 1-3. FP values and vector-type arguments are instead passed in FPRs
+// and VRs respectively, but if a FP value or vector argument occupies one of
+// the first three register-sized words of the parameter area, the corresponding
+// GPR's value is not used to pass arguments. 
+//
+// The XPLINK64 Calling Convention is fully specified in Chapter 22 of the z/OS
+// Language Environment Vendor Interfaces. Appendix B of the same document contains
+// examples.
+
+def CC_SystemZ_XPLINK64 : CallingConv<[
+  // XPLINK64 ABI compliant code
+  // widens i1, i8, i16, i32 to i64
+  // before placing the parameters either on the stack or in registers
+  CCIfType<[i1, i8, i16, i32], CCIfExtend<CCPromoteToType<i64>>>,
+
+  // A SwiftSelf is passed in callee-saved R10.
+  CCIfSwiftSelf<CCIfType<[i64], CCAssignToReg<[R10D]>>>,
+
+  // A SwiftError is passed in R0.
+  CCIfSwiftError<CCIfType<[i64], CCAssignToReg<[R0D]>>>,
+
+  // First i128 values. These are already split into two i64 here,
+  // so we have to use a custom handler and assign into registers, if possible
+  // We need to deal with this first
+  CCIfType<[i64], CCCustom<"CC_SystemZ_I128Indirect">>,
+  // The first 3 integer arguments are passed in registers R1D-R3D.
+  // The rest will be passed in the user area. The address offset of the user
+  // area can be found in register R4D
+  CCIfType<[i32], CCAssignToReg<[R1L, R2L, R3L]>>,
+  CCIfType<[i64], CCAssignToReg<[R1D, R2D, R3D]>>,
+
+  // The first 8 named vector arguments are passed in V24-V31.  Sub-128 vectors
+  // are passed in the same way, but they're widened to one of these types
+  // during type legalization.
+  CCIfSubtarget<"hasVector()",
+    CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+             CCIfFixed<CCCustom<"CC_XPLINK64_Shadow_Reg">>>>,
+  CCIfSubtarget<"hasVector()",
+    CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+             CCIfFixed<CCAssignToReg<[V24, V25, V26, V27,
+                                      V28, V29, V30, V31]>>>>,
+
+  // The first 4 named  float and double arguments are passed in registers FPR0-FPR6.
+  // The rest will be passed in the user area.
+  CCIfType<[f32, f64], CCIfFixed<CCCustom<"CC_XPLINK64_Shadow_Reg">>>,
+  CCIfType<[f32], CCIfFixed<CCAssignToReg<[F0S, F2S, F4S, F6S]>>>,
+  CCIfType<[f64], CCIfFixed<CCAssignToReg<[F0D, F2D, F4D, F6D]>>>,
+  // The first 2 long double arguments are passed in register FPR0/FPR2
+  // and FPR4/FPR6. The rest are passed on the stack
+  // Unless the arguments are passed as VARARGS
+  CCIfType<[f128], CCIfFixed<CCCustom<"CC_XPLINK64_Shadow_Reg">>>,
+  CCIfType<[f128], CCIfFixed<CCAssignToReg<[F0Q, F4Q]>>>,
+
+  // Non fixed floats are passed in GPRs
+  // Promote f32 to f64, if it needs to be passed in GPRs
+  CCIfType<[f32], CCIfNotFixed<CCPromoteToType<f64>>>,
+  // Assign f64 varargs to their proper GPRs
+  CCIfType<[f64], CCIfNotFixed<CCAssignToReg<[R1D, R2D, R3D]>>>,
+  // long double, can only be passed in GPR2 and GPR3, if available,
+  // hence R2Q
+  CCIfType<[f128], CCIfNotFixed<CCCustom<"CC_XPLINK64_Allocate128BitVararg">>>,
+
+  // Non fixed vector arguments are treated in the same way as long
+  // doubles.
+  CCIfSubtarget<"hasVector()",
+    CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+      CCIfNotFixed<CCCustom<"CC_XPLINK64_Allocate128BitVararg">>>>,
+
+  // Other arguments are passed in 8-byte-aligned 8-byte stack slots.
+  CCIfType<[i32, i64, f32, f64], CCAssignToStack<8, 8>>,
+  // Other f128 arguments are passed in 8-byte-aligned 16-byte stack slots.
+  CCIfType<[f128], CCAssignToStack<16, 8>>,
+  // Vector arguments are passed in 8-byte-alinged 16-byte stack slots too.
+  CCIfSubtarget<"hasVector()",
+    CCIfType<[v16i8, v8i16, v4i32, v2i64, v4f32, v2f64],
+             CCAssignToStack<16, 8>>>
+]>;
+
 //===----------------------------------------------------------------------===//
 // s390x return value calling convention
 //===----------------------------------------------------------------------===//
 
 def RetCC_SystemZ : CallingConv<[
+  // zOS XPLINK64
+  CCIfSubtarget<"isTargetXPLINK64()", CCDelegateTo<RetCC_SystemZ_XPLINK64>>,
 
   // ELF Linux SystemZ
   CCIfSubtarget<"isTargetELF()", CCDelegateTo<RetCC_SystemZ_ELF>>
 ]>;
 
 
 //===----------------------------------------------------------------------===//
 // s390x argument calling conventions
 //===----------------------------------------------------------------------===//
 def CC_SystemZ : CallingConv<[
+  // zOS XPLINK64
+  CCIfSubtarget<"isTargetXPLINK64()", CCDelegateTo<CC_SystemZ_XPLINK64>>,
 
   // ELF Linux SystemZ
   CCIfSubtarget<"isTargetELF()", CCDelegateTo<CC_SystemZ_ELF>>
 ]>;