[X86][ABI] Don't preserve return regs for preserve_all/preserve_most CCs

[AntonBikineev]

Summary

Currently both calling conventions preserve registers that are used to store
a return value. This causes the returned value to be lost:

define i32 @bar() {
  %1 = call preserve_mostcc i32 @foo()
   ret i32 %1
}

define preserve_mostcc i32 @foo() {
  ret i32 2
  ; preserve_mostcc will restore %rax, whatever it was before the call.
}

This contradicts the current documentation (preserve_allcc "behaves
identical to the C calling conventions on how arguments and return
values are passed") and also breaks [[clang::preserve_most]].

This change makes CSRs be preserved iff they are not used to store a
return value (e.g. %rax for scalars, {%rax:%rdx} for __int128, %xmm0
for double). For void functions no additional registers are
preserved, i.e. the behaviour is backward compatible with existing
code.

Juergen, can you please take a look, since you added the CCs originally?
Phoebe, I added you as a reviewer as well, since you touched the X86 CC code recently :)

Thanks!

[AntonBikineev]

Roman, could you please take a look at it (or suggest a reviewer)?

(The context: we want to use the attribute for cold calls in Chromium/V8. Early experiments are quite promising (e.g. showed to improve the hot path by up to 2x)).

[pengfei]

I took a look but fogot to comment. TBH, I know little about preserve_mostcc. I checked the existing test cases all of which declare void return with it. So is it not legal to return value with the CC? If we do not preserve rax, we probably have compatibility issue with pre-built code since it assumes rax is saved by callee.

[AntonBikineev]

So is it not legal to return value with the CC?

According to the documentation, it is legal. In the current implementation it isn't though.

If we do not preserve rax, we probably have compatibility issue with pre-built code since it assumes rax is saved by callee.

That's true - the change may break the existing precompiled code.

As I see it, we should either

• have consistency between the implementation and the docs and early diagnose attribute application on non-void functions, or
• allow %rax to be clobbered by the callee.

[lebedev.ri]

This seems to be missing the code change itself.

[AntonBikineev]

In D141020#4037505, @lebedev.ri wrote:

This seems to be missing the code change itself.

(the code change is essentially in X86CallingConv.td)

[efriedma]

Having a register be both preserved and used to store a return value is obviously a contradiction. However, it's consistent to say that a register is preserved if and only if it is not used to return a value. (Not sure off the top of my head what changes are required to make that work correctly.)

The C calling convention uses RAX/RDX/XMM0/XMM1 to return values, and LLVM additionally allows using RCX/XMM2/XMM3 for certain IR signatures. So I don't think just removing RAX from the list does the right thing.

[pengfei]

In D141020#4038220, @efriedma wrote:

Having a register be both preserved and used to store a return value is obviously a contradiction. However, it's consistent to say that a register is preserved if and only if it is not used to return a value. (Not sure off the top of my head what changes are required to make that work correctly.)

The C calling convention uses RAX/RDX/XMM0/XMM1 to return values, and LLVM additionally allows using RCX/XMM2/XMM3 for certain IR signatures. So I don't think just removing RAX from the list does the right thing.

Agreed! FPRs is not a problem as the doc says they are not preserved. But we do use RDX sometime, e.g.: https://godbolt.org/z/sWKKbjMvz
So yes, we may need a method to override the preserve list when RAX/RDX are used as return registers. Not sure if the current framework support or not.

[efriedma]

FPRs is not a problem as the doc says they are not preserved

MostRegs doesn't, but AllRegs does?

Not sure if the current framework support or not.

X86RegisterInfo::getCalleeSavedRegs has enough information to figure out the correct set of registers; not sure if there's any convenient way to compute it, though. Not sure X86RegisterInfo::getCallPreservedMask has enough information to tell; it looks like the actual signature isn't passed. The callers should have that information available, though.

[ributzka]

If this can be fixed without affecting any potential clients, then I am all for it. In hindsight it might have been better to make a use case specific calling convention instead and not a generic one that everyone could depend on. Oen possible solution would be to make a Chromium specific calling convention and also name it as such. This would provide clear expectations what it is intended for and how is supposed to use it. Furthermore this would give you the option of changing it in the future if the need should arise.

[AntonBikineev]

Thanks everybody for the feedback! Sorry for the delayed response - I was occupied with the regular work.

However, it's consistent to say that a register is preserved if and only if it is not used to return a value.

I thought about it first, but just found cloberring %rax as a quick fix. But I 100% agree - having a flexible CC would be the best solution:

• would allow callers to use more registers when callees are void,
• would stay backward compatible, assuming the clients are not using the CCs with the return value (as otherwise their code would be broken).

MostRegs doesn't, but AllRegs does?

Yeah, all return registers must be addressed..

Oen possible solution would be to make a Chromium specific calling convention and also name it as such.

I thought about this as well, but preserve_most is conceptually already what we need. It would be better to save a chromium-specific CC for a future use case :)

There is already logic that nicely allows to disable CSRs dynamically, which was introduced for the Intel's regcall CC (here https://reviews.llvm.org/D28566). So I'll just piggyback on it.

[AntonBikineev]

Ptal. Now the change preserves all the declared callee-saved-regs as long as they are not used to return a value. The change should be backward compatible, assuming the user code was correct.

[pengfei]

Looks great!

llvm/test/CodeGen/X86/preserve_allcc64.ll

9

I think it's better to put these 2 tests into a seperate file. It just waste time to run multi times on the other 2 tests.

11–112

This will brings difficulties in future updates, though I think it's rare this file will be affected.

[AntonBikineev]

llvm/test/CodeGen/X86/preserve_allcc64.ll

9

Sure (moved tests returning double into separate files)

11–112

What exactly could break? The order of CSRs?

[AntonBikineev]

Moved unparametrized tests (i.e. ones returning doubles) into separate files.

[AntonBikineev]

(friendly ping...)

We've already started using the CC on void-functions in Chromium/V8, which helped us to improve the runtime (as well as the binary size). Looking forward to apply it more broadly.

[AntonBikineev]

(friendly ping)

[pengfei]

LGTM.

[AntonBikineev]

Thanks, everybody!

[hans]

We bisected Chromium crashes to this revision, see https://crbug.com/1410225#c4 for how to reproduce.

I'll revert this for now.

[hans]

@tstellar We may want to consider the revert for the 16 branch.

llvm/docs/LangRef.rst

"``preserve_mostcc``" - The `PreserveMost` calling convention
     calling convention on how arguments and return values are passed, but it
     uses a different set of caller/callee-saved registers. This alleviates the
     burden of saving and recovering a large register set before and after the
     call in the caller. If the arguments are passed in callee-saved registers,
     then they will be preserved by the callee across the call. This doesn't
     apply for values returned in callee-saved registers.
 
     - On X86-64 the callee preserves all general purpose registers, except for
-      R11. R11 can be used as a scratch register. Floating-point registers
-      (XMMs/YMMs) are not preserved and need to be saved by the caller.
+      R11 and return registers, if any. R11 can be used as a scratch register.
+      Floating-point registers (XMMs/YMMs) are not preserved and need to be
+      saved by the caller.
 
     The idea behind this convention is to support calls to runtime functions
     that have a hot path and a cold path. The hot path is usually a small piece
     of code that doesn't use many registers. The cold path might need to call out to
     another function and therefore only needs to preserve the caller-saved
     registers, which haven't already been saved by the caller. The
     `PreserveMost` calling convention is very similar to the `cold` calling
     convention in terms of caller/callee-saved registers, but they are used for

llvm/lib/Target/X86/X86CallingConv.td

                                              R8, R9, R10, R11)>;
 
 // CSRs that are handled by prologue, epilogue.
 def CSR_64_CXX_TLS_Darwin_PE : CalleeSavedRegs<(add RBP)>;
 
 // CSRs that are handled explicitly via copies.
 def CSR_64_CXX_TLS_Darwin_ViaCopy : CalleeSavedRegs<(sub CSR_64_TLS_Darwin, RBP)>;
 
-// All GPRs - except r11
+// All GPRs - except r11 and return registers.
 def CSR_64_RT_MostRegs : CalleeSavedRegs<(add CSR_64, RAX, RCX, RDX, RSI, RDI,
                                               R8, R9, R10)>;
 
-// All registers - except r11
+// All registers - except r11 and return registers.
 def CSR_64_RT_AllRegs     : CalleeSavedRegs<(add CSR_64_RT_MostRegs,
                                                  (sequence "XMM%u", 0, 15))>;
 def CSR_64_RT_AllRegs_AVX : CalleeSavedRegs<(add CSR_64_RT_MostRegs,
                                                  (sequence "YMM%u", 0, 15))>;
 
 def CSR_64_MostRegs : CalleeSavedRegs<(add RBX, RCX, RDX, RSI, RDI, R8, R9, R10,
                                            R11, R12, R13, R14, R15, RBP,
                                            (sequence "XMM%u", 0, 15))>;

llvm/lib/Target/X86/X86ISelLowering.cpp

 /// crashing.
 static void errorUnsupported(SelectionDAG &DAG, const SDLoc &dl,
                              const char *Msg) {
   MachineFunction &MF = DAG.getMachineFunction();
   DAG.getContext()->diagnose(
       DiagnosticInfoUnsupported(MF.getFunction(), Msg, dl.getDebugLoc()));
 }
 
+/// Returns true if a CC can dynamically exclude a register from the list of
+/// callee-saved-registers (TargetRegistryInfo::getCalleeSavedRegs()) based on
+/// params/returns.
+static bool shouldDisableCalleeSavedRegisterCC(CallingConv::ID CC) {
+  switch (CC) {
+  default:
+    return false;
+  case CallingConv::X86_RegCall:
+  case CallingConv::PreserveMost:
+  case CallingConv::PreserveAll:
+    return true;
+  }
+}
+
 X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
                                      const X86Subtarget &STI)
     : TargetLowering(TM), Subtarget(STI) {
   bool UseX87 = !Subtarget.useSoftFloat() && Subtarget.hasX87();
   MVT PtrVT = MVT::getIntegerVT(TM.getPointerSizeInBits(0));
 
   // Set up the TargetLowering object.
 

                                const SmallVectorImpl<SDValue> &OutVals,
                                const SDLoc &dl, SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
   X86MachineFunctionInfo *FuncInfo = MF.getInfo<X86MachineFunctionInfo>();
 
   // In some cases we need to disable registers from the default CSR list.
   // For example, when they are used for argument passing.
   bool ShouldDisableCalleeSavedRegister =
-      CallConv == CallingConv::X86_RegCall ||
+      shouldDisableCalleeSavedRegisterCC(CallConv) ||
       MF.getFunction().hasFnAttribute("no_caller_saved_registers");
 
   if (CallConv == CallingConv::X86_INTR && !Outs.empty())
     report_fatal_error("X86 interrupts may not return any value");
 
   SmallVector<CCValAssign, 16> RVLocs;
   CCState CCInfo(CallConv, isVarArg, MF, RVLocs, *DAG.getContext());
   CCInfo.AnalyzeReturn(Outs, RetCC_X86);

       // same, so the size of funclets' (mostly empty) frames is dictated by
       // how far this slot is from the bottom (since they allocate just enough
       // space to accommodate holding this slot at the correct offset).
       int PSPSymFI = MFI.CreateStackObject(8, Align(8), /*isSpillSlot=*/false);
       EHInfo->PSPSymFrameIdx = PSPSymFI;
     }
   }
 
-  if (CallConv == CallingConv::X86_RegCall ||
+  if (shouldDisableCalleeSavedRegisterCC(CallConv) ||
       F.hasFnAttribute("no_caller_saved_registers")) {
     MachineRegisterInfo &MRI = MF.getRegInfo();
     for (std::pair<Register, Register> Pair : MRI.liveins())
       MRI.disableCalleeSavedRegister(Pair.first);
   }
 
   return Chain;
 }

       Mask = RegInfo->getNoPreservedMask();
   }
 
   // Define a new register mask from the existing mask.
   uint32_t *RegMask = nullptr;
 
   // In some calling conventions we need to remove the used physical registers
   // from the reg mask.
-  if (CallConv == CallingConv::X86_RegCall || HasNCSR) {
+  if (shouldDisableCalleeSavedRegisterCC(CallConv) || HasNCSR) {
     const TargetRegisterInfo *TRI = Subtarget.getRegisterInfo();
 
     // Allocate a new Reg Mask and copy Mask.
     RegMask = MF.allocateRegMask();
     unsigned RegMaskSize = MachineOperand::getRegMaskSize(TRI->getNumRegs());
     memcpy(RegMask, Mask, sizeof(RegMask[0]) * RegMaskSize);
 
     // Make sure all sub registers of the argument registers are reset

llvm/test/CodeGen/X86/preserve_allcc64-ret-double.ll

+; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7     | FileCheck --check-prefixes=ALL,SSE %s
+; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7-avx | FileCheck --check-prefixes=ALL,AVX %s
+
+define preserve_allcc double @preserve_allcc1() nounwind {
+entry:
+;ALL-LABEL:   preserve_allcc1
+;SSE:         movaps %xmm1
+;SSE-NOT:     movaps %xmm0
+;AVX:         vmovups %ymm1
+;AVX-NOT:     vmovups %ymm0
+;SSE-NOT:     movaps {{.*}} %xmm0
+;SSE:         movaps {{.*}} %xmm1
+;AVX-NOT:     vmovups {{.*}} %ymm0
+;AVX:         vmovups {{.*}} %ymm1
+  call void asm sideeffect "", "~{rax},~{rbx},~{rcx},~{rdx},~{rsi},~{rdi},~{r8},~{r9},~{r10},~{r11},~{r12},~{r13},~{r14},~{r15},~{rbp},~{xmm0},~{xmm1},~{xmm2},~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15}"()
+  ret double 0.
+}
+
+; Make sure XMM0 (return register) and R11 are saved before the call
+declare preserve_allcc double @bar_double(i64, i64)
+define void @preserve_allcc2() nounwind {
+entry:
+;SSE-LABEL: preserve_allcc2
+;SSE:       movq %r11, [[REG1:%[a-z0-9]+]]
+;SSE:       movaps %xmm0, [[REG2:[-0-9]*\(%r[sb]p\)]]
+;SSE:       movq [[REG1]], %r11
+;SSE:       movaps [[REG2]], %xmm0
+  %a0 = call i64 asm sideeffect "", "={rax}"() nounwind
+  %a1 = call i64 asm sideeffect "", "={rcx}"() nounwind
+  %a2 = call i64 asm sideeffect "", "={rdx}"() nounwind
+  %a3 = call i64 asm sideeffect "", "={r8}"() nounwind
+  %a4 = call i64 asm sideeffect "", "={r9}"() nounwind
+  %a5 = call i64 asm sideeffect "", "={r10}"() nounwind
+  %a6 = call i64 asm sideeffect "", "={r11}"() nounwind
+  %a10 = call <2 x double> asm sideeffect "", "={xmm0}"() nounwind
+  %a11 = call <2 x double> asm sideeffect "", "={xmm1}"() nounwind
+  %a12 = call <2 x double> asm sideeffect "", "={xmm2}"() nounwind
+  %a13 = call <2 x double> asm sideeffect "", "={xmm3}"() nounwind
+  %a14 = call <2 x double> asm sideeffect "", "={xmm4}"() nounwind
+  %a15 = call <2 x double> asm sideeffect "", "={xmm5}"() nounwind
+  %a16 = call <2 x double> asm sideeffect "", "={xmm6}"() nounwind
+  %a17 = call <2 x double> asm sideeffect "", "={xmm7}"() nounwind
+  %a18 = call <2 x double> asm sideeffect "", "={xmm8}"() nounwind
+  %a19 = call <2 x double> asm sideeffect "", "={xmm9}"() nounwind
+  %a20 = call <2 x double> asm sideeffect "", "={xmm10}"() nounwind
+  %a21 = call <2 x double> asm sideeffect "", "={xmm11}"() nounwind
+  %a22 = call <2 x double> asm sideeffect "", "={xmm12}"() nounwind
+  %a23 = call <2 x double> asm sideeffect "", "={xmm13}"() nounwind
+  %a24 = call <2 x double> asm sideeffect "", "={xmm14}"() nounwind
+  %a25 = call <2 x double> asm sideeffect "", "={xmm15}"() nounwind
+  call preserve_allcc double @bar_double(i64 1, i64 2)
+  call void asm sideeffect "", "{rax},{rcx},{rdx},{r8},{r9},{r10},{r11},{xmm0},{xmm1},{xmm2},{xmm3},{xmm4},{xmm5},{xmm6},{xmm7},{xmm8},{xmm9},{xmm10},{xmm11},{xmm12},{xmm13},{xmm14},{xmm15}"(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, <2 x double> %a10, <2 x double> %a11, <2 x double> %a12, <2 x double> %a13, <2 x double> %a14, <2 x double> %a15, <2 x double> %a16, <2 x double> %a17, <2 x double> %a18, <2 x double> %a19, <2 x double> %a20, <2 x double> %a21, <2 x double> %a22, <2 x double> %a23, <2 x double> %a24, <2 x double> %a25)
+  ret void
+}

llvm/test/CodeGen/X86/preserve_allcc64.ll

-; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7     | FileCheck --check-prefix=SSE %s
-; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7-avx | FileCheck --check-prefix=AVX %s
+; RUN: sed -e "s/RETTYPE/void/;s/RETVAL//" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck --check-prefixes=ALL,SSE,VOID %s
+; RUN: sed -e "s/RETTYPE/i32/;s/RETVAL/undef/" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck --check-prefixes=ALL,SSE,INT %s
+; RUN: sed -e "s/RETTYPE/\{i64\,i64\}/;s/RETVAL/undef/" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck --check-prefixes=ALL,SSE,INT128 %s
+;
+; RUN: sed -e "s/RETTYPE/void/;s/RETVAL//" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7-avx | FileCheck --check-prefixes=ALL,AVX,VOID %s
+; RUN: sed -e "s/RETTYPE/i32/;s/RETVAL/undef/" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7-avx | FileCheck --check-prefixes=ALL,AVX,INT %s
+; RUN: sed -e "s/RETTYPE/\{i64\,i64\}/;s/RETVAL/undef/" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7-avx | FileCheck --check-prefixes=ALL,AVX,INT128 %s
 
-define preserve_allcc void @preserve_allcc1() nounwind {
+define preserve_allcc RETTYPE @preserve_allcc1() nounwind {
 entry:
-;SSE-LABEL: preserve_allcc1
-;SSE:       pushq %r10
-;SSE-NEXT:  pushq %r9
-;SSE-NEXT:  pushq %r8
-;SSE-NEXT:  pushq %rdi
-;SSE-NEXT:  pushq %rsi
-;SSE-NEXT:  pushq %rdx
-;SSE-NEXT:  pushq %rcx
-;SSE-NEXT:  pushq %rax
-;SSE-NEXT:  pushq %rbp
-;SSE-NEXT:  pushq %r15
-;SSE-NEXT:  pushq %r14
-;SSE-NEXT:  pushq %r13
-;SSE-NEXT:  pushq %r12
-;SSE-NEXT:  pushq %rbx
-;SSE:       movaps %xmm15
-;SSE-NEXT:  movaps %xmm14
-;SSE-NEXT:  movaps %xmm13
-;SSE-NEXT:  movaps %xmm12
-;SSE-NEXT:  movaps %xmm11
-;SSE-NEXT:  movaps %xmm10
-;SSE-NEXT:  movaps %xmm9
-;SSE-NEXT:  movaps %xmm8
-;SSE-NEXT:  movaps %xmm7
-;SSE-NEXT:  movaps %xmm6
-;SSE-NEXT:  movaps %xmm5
-;SSE-NEXT:  movaps %xmm4
-;SSE-NEXT:  movaps %xmm3
-;SSE-NEXT:  movaps %xmm2
-;SSE-NEXT:  movaps %xmm1
-;SSE-NEXT:  movaps %xmm0
-;AVX-LABEL: preserve_allcc1
-;AVX:       pushq %r10
-;AVX-NEXT:  pushq %r9
-;AVX-NEXT:  pushq %r8
-;AVX-NEXT:  pushq %rdi
-;AVX-NEXT:  pushq %rsi
-;AVX-NEXT:  pushq %rdx
-;AVX-NEXT:  pushq %rcx
-;AVX-NEXT:  pushq %rax
-;AVX-NEXT:  pushq %rbp
-;AVX-NEXT:  pushq %r15
-;AVX-NEXT:  pushq %r14
-;AVX-NEXT:  pushq %r13
-;AVX-NEXT:  pushq %r12
-;AVX-NEXT:  pushq %rbx
-;AVX:       vmovups %ymm15
-;AVX-NEXT:  vmovups %ymm14
-;AVX-NEXT:  vmovups %ymm13
-;AVX-NEXT:  vmovups %ymm12
-;AVX-NEXT:  vmovups %ymm11
-;AVX-NEXT:  vmovups %ymm10
-;AVX-NEXT:  vmovups %ymm9
-;AVX-NEXT:  vmovups %ymm8
-;AVX-NEXT:  vmovups %ymm7
-;AVX-NEXT:  vmovups %ymm6
-;AVX-NEXT:  vmovups %ymm5
-;AVX-NEXT:  vmovups %ymm4
-;AVX-NEXT:  vmovups %ymm3
-;AVX-NEXT:  vmovups %ymm2
-;AVX-NEXT:  vmovups %ymm1
-;AVX-NEXT:  vmovups %ymm0
+;ALL-LABEL:   preserve_allcc1
+;ALL:         pushq %r10
+;ALL-NEXT:    pushq %r9
+;ALL-NEXT:    pushq %r8
+;ALL-NEXT:    pushq %rdi
+;ALL-NEXT:    pushq %rsi
+;VOID-NEXT:   pushq %rdx
+;INT-NEXT:    pushq %rdx
+;INT128-NOT:  pushq %rdx
+;ALL-NEXT:    pushq %rcx
+;VOID-NEXT:   pushq %rax
+;INT-NOT:     pushq %rax
+;INT128-NOT:  pushq %rax
+;ALL-NEXT:    pushq %rbp
+;ALL-NEXT:    pushq %r15
+;ALL-NEXT:    pushq %r14
+;ALL-NEXT:    pushq %r13
+;ALL-NEXT:    pushq %r12
+;ALL-NEXT:    pushq %rbx
+;SSE:         movaps %xmm15
+;SSE-NEXT:    movaps %xmm14
+;SSE-NEXT:    movaps %xmm13
+;SSE-NEXT:    movaps %xmm12
+;SSE-NEXT:    movaps %xmm11
+;SSE-NEXT:    movaps %xmm10
+;SSE-NEXT:    movaps %xmm9
+;SSE-NEXT:    movaps %xmm8
+;SSE-NEXT:    movaps %xmm7
+;SSE-NEXT:    movaps %xmm6
+;SSE-NEXT:    movaps %xmm5
+;SSE-NEXT:    movaps %xmm4
+;SSE-NEXT:    movaps %xmm3
+;SSE-NEXT:    movaps %xmm2
+;SSE-NEXT:    movaps %xmm1
+;SSE-NEXT:    movaps %xmm0
+;AVX:         vmovups %ymm15
+;AVX-NEXT:    vmovups %ymm14
+;AVX-NEXT:    vmovups %ymm13
+;AVX-NEXT:    vmovups %ymm12
+;AVX-NEXT:    vmovups %ymm11
+;AVX-NEXT:    vmovups %ymm10
+;AVX-NEXT:    vmovups %ymm9
+;AVX-NEXT:    vmovups %ymm8
+;AVX-NEXT:    vmovups %ymm7
+;AVX-NEXT:    vmovups %ymm6
+;AVX-NEXT:    vmovups %ymm5
+;AVX-NEXT:    vmovups %ymm4
+;AVX-NEXT:    vmovups %ymm3
+;AVX-NEXT:    vmovups %ymm2
+;AVX-NEXT:    vmovups %ymm1
+;AVX-NEXT:    vmovups %ymm0
+;SSE:         movaps {{.*}} %xmm0
+;SSE-NEXT:    movaps {{.*}} %xmm1
+;SSE-NEXT:    movaps {{.*}} %xmm2
+;SSE-NEXT:    movaps {{.*}} %xmm3
+;SSE-NEXT:    movaps {{.*}} %xmm4
+;SSE-NEXT:    movaps {{.*}} %xmm5
+;SSE-NEXT:    movaps {{.*}} %xmm6
+;SSE-NEXT:    movaps {{.*}} %xmm7
+;SSE-NEXT:    movaps {{.*}} %xmm8
+;SSE-NEXT:    movaps {{.*}} %xmm9
+;SSE-NEXT:    movaps {{.*}} %xmm10
+;SSE-NEXT:    movaps {{.*}} %xmm11
+;SSE-NEXT:    movaps {{.*}} %xmm12
+;SSE-NEXT:    movaps {{.*}} %xmm13
+;SSE-NEXT:    movaps {{.*}} %xmm14
+;SSE-NEXT:    movaps {{.*}} %xmm15
+;AVX:         vmovups {{.*}} %ymm0
+;AVX-NEXT:    vmovups {{.*}} %ymm1
+;AVX-NEXT:    vmovups {{.*}} %ymm2
+;AVX-NEXT:    vmovups {{.*}} %ymm3
+;AVX-NEXT:    vmovups {{.*}} %ymm4
+;AVX-NEXT:    vmovups {{.*}} %ymm5
+;AVX-NEXT:    vmovups {{.*}} %ymm6
+;AVX-NEXT:    vmovups {{.*}} %ymm7
+;AVX-NEXT:    vmovups {{.*}} %ymm8
+;AVX-NEXT:    vmovups {{.*}} %ymm9
+;AVX-NEXT:    vmovups {{.*}} %ymm10
+;AVX-NEXT:    vmovups {{.*}} %ymm11
+;AVX-NEXT:    vmovups {{.*}} %ymm12
+;AVX-NEXT:    vmovups {{.*}} %ymm13
+;AVX-NEXT:    vmovups {{.*}} %ymm14
+;AVX-NEXT:    vmovups {{.*}} %ymm15
+;ALL:         popq    %rbx
+;ALL-NEXT:    popq    %r12
+;ALL-NEXT:    popq    %r13
+;ALL-NEXT:    popq    %r14
+;ALL-NEXT:    popq    %r15
+;ALL-NEXT:    popq    %rbp
+;VOID-NEXT:   popq    %rax
+;INT-NOT:     popq    %rax
+;INT128-NOT:  popq    %rax
+;ALL-NEXT:    popq    %rcx
+;VOID-NEXT:   popq    %rdx
+;INT-NEXT:    popq    %rdx
+;INT128-NOT:  popq    %rdx
+;ALL-NEXT:    popq    %rsi
+;ALL-NEXT:    popq    %rdi
+;ALL-NEXT:    popq    %r8
+;ALL-NEXT:    popq    %r9
+;ALL-NEXT:    popq    %r10
   call void asm sideeffect "", "~{rax},~{rbx},~{rcx},~{rdx},~{rsi},~{rdi},~{r8},~{r9},~{r10},~{r11},~{r12},~{r13},~{r14},~{r15},~{rbp},~{xmm0},~{xmm1},~{xmm2},~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15}"()
-  ret void
+  ret RETTYPE RETVAL
 }
 
-; Make sure only R11 is saved before the call
-declare preserve_allcc void @bar(i64, i64, double, double)
+; Make sure R11 and return registers are saved before the call
+declare preserve_allcc RETTYPE @bar(i64, i64, double, double)
 define void @preserve_allcc2() nounwind {
 entry:
-;SSE-LABEL: preserve_allcc2
-;SSE:       movq %r11, [[REG:%[a-z0-9]+]]
-;SSE-NOT:   movaps %xmm
-;SSE:       movq [[REG]], %r11
+;ALL-LABEL: preserve_allcc2
+;VOID-NOT:  movq %rax, [[REG1:%[a-z0-9]+]]
+;INT:       movq %rax, [[REG1:%[a-z0-9]+]]
+;INT128:    movq %rax, [[REG1:%[a-z0-9]+]]
+;VOID-NOT:  movq %rdx, [[REG2:%[a-z0-9]+]]
+;INT-NOT:   movq %rdx, [[REG2:%[a-z0-9]+]]
+;INT128:    movq %rdx, [[REG2:%[a-z0-9]+]]
+;ALL:       movq %r11, [[REG3:%[a-z0-9]+]]
+;ALL-NOT:   movaps %xmm
+;VOID-NOT:  movq {{.*}}, %rax
+;INT:       movq [[REG1]], %rax
+;INT128:    movq [[REG1]], %rax
+;VOID-NOT:  movq {{.*}}, %rdx
+;INT-NOT:   movq {{.*}}, %rdx
+;INT128:    movq [[REG2]], %rdx
+;ALL:       movq [[REG3]], %r11
   %a0 = call i64 asm sideeffect "", "={rax}"() nounwind
   %a1 = call i64 asm sideeffect "", "={rcx}"() nounwind
   %a2 = call i64 asm sideeffect "", "={rdx}"() nounwind
   %a3 = call i64 asm sideeffect "", "={r8}"() nounwind
   %a4 = call i64 asm sideeffect "", "={r9}"() nounwind
   %a5 = call i64 asm sideeffect "", "={r10}"() nounwind
   %a6 = call i64 asm sideeffect "", "={r11}"() nounwind
   %a10 = call <2 x double> asm sideeffect "", "={xmm2}"() nounwind
   %a11 = call <2 x double> asm sideeffect "", "={xmm3}"() nounwind
   %a12 = call <2 x double> asm sideeffect "", "={xmm4}"() nounwind
   %a13 = call <2 x double> asm sideeffect "", "={xmm5}"() nounwind
   %a14 = call <2 x double> asm sideeffect "", "={xmm6}"() nounwind
   %a15 = call <2 x double> asm sideeffect "", "={xmm7}"() nounwind
   %a16 = call <2 x double> asm sideeffect "", "={xmm8}"() nounwind
   %a17 = call <2 x double> asm sideeffect "", "={xmm9}"() nounwind
   %a18 = call <2 x double> asm sideeffect "", "={xmm10}"() nounwind
   %a19 = call <2 x double> asm sideeffect "", "={xmm11}"() nounwind
   %a20 = call <2 x double> asm sideeffect "", "={xmm12}"() nounwind
   %a21 = call <2 x double> asm sideeffect "", "={xmm13}"() nounwind
   %a22 = call <2 x double> asm sideeffect "", "={xmm14}"() nounwind
   %a23 = call <2 x double> asm sideeffect "", "={xmm15}"() nounwind
-  call preserve_allcc void @bar(i64 1, i64 2, double 3.0, double 4.0)
+  call preserve_allcc RETTYPE @bar(i64 1, i64 2, double 3.0, double 4.0)
   call void asm sideeffect "", "{rax},{rcx},{rdx},{r8},{r9},{r10},{r11},{xmm2},{xmm3},{xmm4},{xmm5},{xmm6},{xmm7},{xmm8},{xmm9},{xmm10},{xmm11},{xmm12},{xmm13},{xmm14},{xmm15}"(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, <2 x double> %a10, <2 x double> %a11, <2 x double> %a12, <2 x double> %a13, <2 x double> %a14, <2 x double> %a15, <2 x double> %a16, <2 x double> %a17, <2 x double> %a18, <2 x double> %a19, <2 x double> %a20, <2 x double> %a21, <2 x double> %a22, <2 x double> %a23)
   ret void
 }

llvm/test/CodeGen/X86/preserve_mostcc64-ret-double.ll

+; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck %s
+
+; Make sure XMM0 (return register) and R11 are saved before the call
+declare preserve_mostcc double @foo_double(i64, i64)
+define void @preserve_mostcc1() nounwind {
+entry:
+;CHECK-LABEL: preserve_mostcc1
+;CHECK:       movq %r11, [[REG1:%[a-z0-9]+]]
+;CHECK:       movaps %xmm0, [[REG2:[-0-9]*\(%r[sb]p\)]]
+;CHECK:       call
+;CHECK:       movq [[REG1]], %r11
+;CHECK:       movaps [[REG2]], %xmm0
+  %a0 = call i64 asm sideeffect "", "={rax}"() nounwind
+  %a1 = call i64 asm sideeffect "", "={rcx}"() nounwind
+  %a2 = call i64 asm sideeffect "", "={rdx}"() nounwind
+  %a3 = call i64 asm sideeffect "", "={r8}"() nounwind
+  %a4 = call i64 asm sideeffect "", "={r9}"() nounwind
+  %a5 = call i64 asm sideeffect "", "={r10}"() nounwind
+  %a6 = call i64 asm sideeffect "", "={r11}"() nounwind
+  %a10 = call <2 x double> asm sideeffect "", "={xmm0}"() nounwind
+  %a11 = call <2 x double> asm sideeffect "", "={xmm1}"() nounwind
+  %a12 = call <2 x double> asm sideeffect "", "={xmm2}"() nounwind
+  %a13 = call <2 x double> asm sideeffect "", "={xmm3}"() nounwind
+  %a14 = call <2 x double> asm sideeffect "", "={xmm4}"() nounwind
+  %a15 = call <2 x double> asm sideeffect "", "={xmm5}"() nounwind
+  %a16 = call <2 x double> asm sideeffect "", "={xmm6}"() nounwind
+  %a17 = call <2 x double> asm sideeffect "", "={xmm7}"() nounwind
+  %a18 = call <2 x double> asm sideeffect "", "={xmm8}"() nounwind
+  %a19 = call <2 x double> asm sideeffect "", "={xmm9}"() nounwind
+  %a20 = call <2 x double> asm sideeffect "", "={xmm10}"() nounwind
+  %a21 = call <2 x double> asm sideeffect "", "={xmm11}"() nounwind
+  %a22 = call <2 x double> asm sideeffect "", "={xmm12}"() nounwind
+  %a23 = call <2 x double> asm sideeffect "", "={xmm13}"() nounwind
+  %a24 = call <2 x double> asm sideeffect "", "={xmm14}"() nounwind
+  %a25 = call <2 x double> asm sideeffect "", "={xmm15}"() nounwind
+  call preserve_mostcc double @foo_double(i64 1, i64 2)
+  call void asm sideeffect "", "{rax},{rcx},{rdx},{r8},{r9},{r10},{r11},{xmm0},{xmm1},{xmm2},{xmm3},{xmm4},{xmm5},{xmm6},{xmm7},{xmm8},{xmm9},{xmm10},{xmm11},{xmm12},{xmm13},{xmm14},{xmm15}"(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, <2 x double> %a10, <2 x double> %a11, <2 x double> %a12, <2 x double> %a13, <2 x double> %a14, <2 x double> %a15, <2 x double> %a16, <2 x double> %a17, <2 x double> %a18, <2 x double> %a19, <2 x double> %a20, <2 x double> %a21, <2 x double> %a22, <2 x double> %a23, <2 x double> %a24, <2 x double> %a25)
+  ret void
+}

llvm/test/CodeGen/X86/preserve_mostcc64.ll

-; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7     | FileCheck --check-prefix=SSE %s
-; RUN: llc < %s -mtriple=x86_64-apple-darwin -mcpu=corei7-avx | FileCheck --check-prefix=AVX %s
+; RUN: sed -e "s/RETTYPE/void/;s/RETVAL//" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck --check-prefixes=ALL,VOID %s
+; RUN: sed -e "s/RETTYPE/i32/;s/RETVAL/undef/" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck --check-prefixes=ALL,INT %s
+; RUN: sed -e "s/RETTYPE/\{i64\,i64\}/;s/RETVAL/undef/" %s | llc -mtriple=x86_64-apple-darwin -mcpu=corei7 | FileCheck --check-prefixes=ALL,INT128 %s
 
-; Every GPR should be saved - except r11
-define preserve_mostcc void @preserve_mostcc1() nounwind {
+; Every GPR should be saved - except r11 and return registers
+define preserve_mostcc RETTYPE @preserve_mostcc1() nounwind {
 entry:
-;SSE-LABEL: preserve_mostcc1
-;SSE:       pushq %r10
-;SSE-NEXT:  pushq %r9
-;SSE-NEXT:  pushq %r8
-;SSE-NEXT:  pushq %rdi
-;SSE-NEXT:  pushq %rsi
-;SSE-NEXT:  pushq %rdx
-;SSE-NEXT:  pushq %rcx
-;SSE-NEXT:  pushq %rax
-;SSE-NEXT:  pushq %rbp
-;SSE-NEXT:  pushq %r15
-;SSE-NEXT:  pushq %r14
-;SSE-NEXT:  pushq %r13
-;SSE-NEXT:  pushq %r12
-;SSE-NEXT:  pushq %rbx
-;AVX-LABEL: preserve_mostcc1
-;AVX:       pushq %r10
-;AVX-NEXT:  pushq %r9
-;AVX-NEXT:  pushq %r8
-;AVX-NEXT:  pushq %rdi
-;AVX-NEXT:  pushq %rsi
-;AVX-NEXT:  pushq %rdx
-;AVX-NEXT:  pushq %rcx
-;AVX-NEXT:  pushq %rax
-;AVX-NEXT:  pushq %rbp
-;AVX-NEXT:  pushq %r15
-;AVX-NEXT:  pushq %r14
-;AVX-NEXT:  pushq %r13
-;AVX-NEXT:  pushq %r12
-;AVX-NEXT:  pushq %rbx
+;ALL-LABEL:   preserve_mostcc1
+;ALL:         pushq %r10
+;ALL-NEXT:    pushq %r9
+;ALL-NEXT:    pushq %r8
+;ALL-NEXT:    pushq %rdi
+;ALL-NEXT:    pushq %rsi
+;VOID-NEXT:   pushq %rdx
+;INT-NEXT:    pushq %rdx
+;INT128-NOT:  pushq %rdx
+;ALL-NEXT:    pushq %rcx
+;VOID-NEXT:   pushq %rax
+;INT-NOT:     pushq %rax
+;INT128-NOT:  pushq %rax
+;ALL-NEXT:    pushq %rbp
+;ALL-NEXT:    pushq %r15
+;ALL-NEXT:    pushq %r14
+;ALL-NEXT:    pushq %r13
+;ALL-NEXT:    pushq %r12
+;ALL-NEXT:    pushq %rbx
+;ALL:         popq    %rbx
+;ALL-NEXT:    popq    %r12
+;ALL-NEXT:    popq    %r13
+;ALL-NEXT:    popq    %r14
+;ALL-NEXT:    popq    %r15
+;ALL-NEXT:    popq    %rbp
+;VOID-NEXT:   popq    %rax
+;INT-NOT:     popq    %rax
+;INT128-NOT:  popq    %rax
+;ALL-NEXT:    popq    %rcx
+;VOID-NEXT:   popq    %rdx
+;INT-NEXT:    popq    %rdx
+;INT128-NOT:  popq    %rdx
+;ALL-NEXT:    popq    %rsi
+;ALL-NEXT:    popq    %rdi
+;ALL-NEXT:    popq    %r8
+;ALL-NEXT:    popq    %r9
+;ALL-NEXT:    popq    %r10
   call void asm sideeffect "", "~{rax},~{rbx},~{rcx},~{rdx},~{rsi},~{rdi},~{r8},~{r9},~{r10},~{r11},~{r12},~{r13},~{r14},~{r15},~{rbp},~{xmm0},~{xmm1},~{xmm2},~{xmm3},~{xmm4},~{xmm5},~{xmm6},~{xmm7},~{xmm8},~{xmm9},~{xmm10},~{xmm11},~{xmm12},~{xmm13},~{xmm14},~{xmm15}"()
-  ret void
+  ret RETTYPE RETVAL
 }
 
-; Make sure R11 and XMMs are saved before the call
-declare preserve_mostcc void @foo(i64, i64, double, double)
+; Make sure R11, return registers and XMMs are saved before the call
+declare preserve_mostcc RETTYPE @foo(i64, i64, double, double)
 define void @preserve_mostcc2() nounwind {
 entry:
-;SSE-LABEL: preserve_mostcc2
-;SSE:       movq %r11, [[REG:%[a-z0-9]+]]
-;SSE:       movaps %xmm2
-;SSE:       movaps %xmm3
-;SSE:       movaps %xmm4
-;SSE:       movaps %xmm5
-;SSE:       movaps %xmm6
-;SSE:       movaps %xmm7
-;SSE:       movaps %xmm8
-;SSE:       movaps %xmm9
-;SSE:       movaps %xmm10
-;SSE:       movaps %xmm11
-;SSE:       movaps %xmm12
-;SSE:       movaps %xmm13
-;SSE:       movaps %xmm14
-;SSE:       movaps %xmm15
-;SSE:       movq [[REG]], %r11
+;ALL-LABEL: preserve_mostcc2
+;VOID-NOT:  movq %rax, [[REG1:%[a-z0-9]+]]
+;INT:       movq %rax, [[REG1:%[a-z0-9]+]]
+;INT128:    movq %rax, [[REG1:%[a-z0-9]+]]
+;VOID-NOT:  movq %rdx, [[REG2:%[a-z0-9]+]]
+;INT-NOT:   movq %rdx, [[REG2:%[a-z0-9]+]]
+;INT128:    movq %rdx, [[REG2:%[a-z0-9]+]]
+;ALL:       movq %r11, [[REG3:%[a-z0-9]+]]
+;ALL:       movaps %xmm2
+;ALL:       movaps %xmm3
+;ALL:       movaps %xmm4
+;ALL:       movaps %xmm5
+;ALL:       movaps %xmm6
+;ALL:       movaps %xmm7
+;ALL:       movaps %xmm8
+;ALL:       movaps %xmm9
+;ALL:       movaps %xmm10
+;ALL:       movaps %xmm11
+;ALL:       movaps %xmm12
+;ALL:       movaps %xmm13
+;ALL:       movaps %xmm14
+;ALL:       movaps %xmm15
+;ALL:       call
+;VOID-NOT:  movq {{.*}}, %rax
+;INT:       movq [[REG1]], %rax
+;INT128:    movq [[REG1]], %rax
+;VOID-NOT:  movq {{.*}}, %rdx
+;INT-NOT:   movq {{.*}}, %rdx
+;INT128:    movq [[REG2]], %rdx
+;ALL:       movq [[REG3]], %r11
+;ALL:       movaps {{.*}} %xmm2
+;ALL:       movaps {{.*}} %xmm3
+;ALL:       movaps {{.*}} %xmm4
+;ALL:       movaps {{.*}} %xmm5
+;ALL:       movaps {{.*}} %xmm6
+;ALL:       movaps {{.*}} %xmm7
+;ALL:       movaps {{.*}} %xmm8
+;ALL:       movaps {{.*}} %xmm9
+;ALL:       movaps {{.*}} %xmm10
+;ALL:       movaps {{.*}} %xmm11
+;ALL:       movaps {{.*}} %xmm12
+;ALL:       movaps {{.*}} %xmm13
+;ALL:       movaps {{.*}} %xmm14
+;ALL:       movaps {{.*}} %xmm15
   %a0 = call i64 asm sideeffect "", "={rax}"() nounwind
   %a1 = call i64 asm sideeffect "", "={rcx}"() nounwind
   %a2 = call i64 asm sideeffect "", "={rdx}"() nounwind
   %a3 = call i64 asm sideeffect "", "={r8}"() nounwind
   %a4 = call i64 asm sideeffect "", "={r9}"() nounwind
   %a5 = call i64 asm sideeffect "", "={r10}"() nounwind
   %a6 = call i64 asm sideeffect "", "={r11}"() nounwind
   %a10 = call <2 x double> asm sideeffect "", "={xmm2}"() nounwind
   %a11 = call <2 x double> asm sideeffect "", "={xmm3}"() nounwind
   %a12 = call <2 x double> asm sideeffect "", "={xmm4}"() nounwind
   %a13 = call <2 x double> asm sideeffect "", "={xmm5}"() nounwind
   %a14 = call <2 x double> asm sideeffect "", "={xmm6}"() nounwind
   %a15 = call <2 x double> asm sideeffect "", "={xmm7}"() nounwind
   %a16 = call <2 x double> asm sideeffect "", "={xmm8}"() nounwind
   %a17 = call <2 x double> asm sideeffect "", "={xmm9}"() nounwind
   %a18 = call <2 x double> asm sideeffect "", "={xmm10}"() nounwind
   %a19 = call <2 x double> asm sideeffect "", "={xmm11}"() nounwind
   %a20 = call <2 x double> asm sideeffect "", "={xmm12}"() nounwind
   %a21 = call <2 x double> asm sideeffect "", "={xmm13}"() nounwind
   %a22 = call <2 x double> asm sideeffect "", "={xmm14}"() nounwind
   %a23 = call <2 x double> asm sideeffect "", "={xmm15}"() nounwind
-  call preserve_mostcc void @foo(i64 1, i64 2, double 3.0, double 4.0)
+  call preserve_mostcc RETTYPE @foo(i64 1, i64 2, double 3.0, double 4.0)
   call void asm sideeffect "", "{rax},{rcx},{rdx},{r8},{r9},{r10},{r11},{xmm2},{xmm3},{xmm4},{xmm5},{xmm6},{xmm7},{xmm8},{xmm9},{xmm10},{xmm11},{xmm12},{xmm13},{xmm14},{xmm15}"(i64 %a0, i64 %a1, i64 %a2, i64 %a3, i64 %a4, i64 %a5, i64 %a6, <2 x double> %a10, <2 x double> %a11, <2 x double> %a12, <2 x double> %a13, <2 x double> %a14, <2 x double> %a15, <2 x double> %a16, <2 x double> %a17, <2 x double> %a18, <2 x double> %a19, <2 x double> %a20, <2 x double> %a21, <2 x double> %a22, <2 x double> %a23)
   ret void
 }