[X86][VARARG] Avoid spilling xmm vararg arguments.

[avl]

Summary

That patch fixes https://bugs.llvm.org/show_bug.cgi?id=42219 bug.
Related review D62639.

For the noimplicitfloat mode, the compiler mustn't generate
floating-point code if it was not asked directly to do so.
This rule does not work with variable function arguments currently.
Though compiler correctly guards block of code, which copies xmm vararg parameters with a check for %al,
it does not protect spills for xmm registers. Thus, such spills generated in non-protected areas and could break code, which does not expect floating-point data.
The problem happens in -O0 optimization mode. With this optimization level
there is used FastRegisterAllocator, which spills virtual registers at basic block boundaries.
Register Allocator does not protect spills with additional control-flow modifications.
Thus to resolve that problem, it is suggested to not copy incoming physical
registers into virtual registers. Instead, store incoming physical xmm registers into the memory from scratch.

Another variant of this problem happens with high optimization modes with thunk functions.
At a high optimization level, the Greedy Register Allocator is used.
For the following test case(CodeGen/X86/musttail-varargs.ll) :

define void @f_thunk(i8* %this, ...) {
  ; Use va_start so that we exercise the combination.  %ap = alloca [4 x i8*], align 16
  %ap_i8 = bitcast [4 x i8*]* %ap to i8*
  call void @llvm.va_start(i8* %ap_i8)

  %fptr = call void(i8*, ...)*(i8*) @get_f(i8* %this)  <<<<<<<<<<<<<<<<<<<
  musttail call void (i8*, ...) %fptr(i8* %this, ...)
  ret void
}

f_thunk function needs to propagate all their parameters into %fptr. But, it needs to store/restore virtual registers
corresponding to incoming xmm registers around the call to get_f(). So the final code contains unprotected stores/restores for xmm registers. The solution for that case is also to avoid using virtual registers. Copy incoming physical xmm registers into the memory at the function entry. Restore them from memory right before tail-call jump instruction. New asm code for this case would look like this:

f_thunk: 
        testb   %al, %al
        je      .LBB0_2
# %bb.1:
        movaps  %xmm0, 96(%rsp)  <<< store incoming xmm registers on to stack

.LBB0_2:
        callq   get_f
        testb   %al, %al   <<< check for existence of xmm varargs parameters
        je      .LBB0_4
# %bb.3:
        movaps  96(%rsp), %xmm0        <<< restore xmm varargs parameters before tailcall jump. 
        jmpq    *%r11                   # TAILCALL
.LBB0_4:
        jmpq    *%r11                   # TAILCALL

[asl]

llvm/lib/Target/X86/X86ISelLowering.cpp

3699

What is the status of all these TODO here and there?

llvm/lib/Target/X86/X86ISelLowering.h

264

Do you really need these changes?

[avl]

llvm/lib/Target/X86/X86ISelLowering.cpp

3699

I assume they would be done later. They are not required for correctness.

1. not storing xmm registers if "no fp, only musttail calls, noimplcitfloat". I am going to do after this patch and D62639 would be integrated.
2. support of YMM and ZMM would be done by someone who would implement support of YMM and ZMM registers for varargs. this is separate task.
3. comment for AArch64 - for someone who would implement that on AArch64.

llvm/lib/Target/X86/X86ISelLowering.h

264

no, I don`t . it was done by clang-format.

[asl]

llvm/lib/Target/X86/X86ISelLowering.h

264

Please do not add unrelated changes

[avl]

llvm/lib/Target/X86/X86ISelLowering.h

264

Ok, I will delete them. clang-format added above changes since VARARG_THUNK_SAVE_XMM_REGS was added. In that sense the changes are related. But OK, I will delete them.

[avl]

removed formatting done by clang-format for X86ISelLowering.h. ping...

[avl]

fix small typo introduced by previous update.

[efriedma]

Is it possible to avoid expanding VARARG_THUNK_SAVE_XMM_REGS until after register allocation? I would rather not make the liveness rules more complicated just for the sake of working around a limitation of fast regalloc.

[avl]

Is it possible to avoid expanding VARARG_THUNK_SAVE_XMM_REGS until after register allocation? I would rather not make the liveness rules more complicated just for the sake of working around a limitation of fast regalloc.

yes, it is. Would do that way. Thanks.

[avl]

addressed comments(moved expansion of xmm registers code until after register allocator).

[avl]

ping.

[avl]

ping..

[avl]

ping.

[avl]

rebased. ping.

[avl]

ping

[avl]

ping.

Colleagues, any comments on this review?

[avl]

ping.

[efriedma]

Target-independent changes look fine.

I'm not planning to review the x86 backend changes.

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

3543

First letter of variables should be capitalized.

3692

!guardedXmmRegs.empty()

[rnk]

For the noimplicitfloat mode, the compiler mustn't generate
floating-point code if it was not asked directly to do so.

I don't think checking AL on function entry will work reliably. In general, musttail is meant to forward all possible register parameters. Normal arguments may be passed in XMM registers without setting AL. Consider this code:

typedef float v4f32 __attribute__((__vector_size__(16), __aligned__(16)));
int gv_i32;
v4f32 gv_v4f32;
void bar(int x, v4f32 v) {
  gv_i32 = x;
  gv_v4f32 = v;
}
void foo() {
  v4f32 v = {};
  bar(42, v);
}

It should be possible to use musttail thunk with a prototype of void (int, ...) between the call to foo and bar. And, if that thunk contains a function call, it will need to spill&fill all XMM argument registers. If you look at the generated assembly for calling bar, AL is not set to 1:

# %bb.0:                                # %entry
        pushq   %rax
        .cfi_def_cfa_offset 16
        xorps   %xmm0, %xmm0
        movl    $42, %edi
        callq   _Z3bariDv4_f

Is there actually a use case for musttail thunks and noimplicitfloat? Could we instead just declare that such a thunk represents explicit FP usage?

Furthermore, this will probably do the wrong thing on Windows, where AL is never set, to my knowledge. Consider this C++ code that uses musttail in the MS ABI:

int gv_i32;
double gv_f64;
class Foo {
  void virtual bar(int x, double v);
  void foo();
};
void Foo::bar(int x, double v) {
  gv_i32 = x;
  gv_f64 = v;
}
void Foo::foo() {
  auto mp = &Foo::bar;
  (this->*mp)(42, 0.0);
}

The virtual member pointer thunk needs to pass along XMM2, although it doesn't contain a function call.

llvm/lib/Target/X86/X86ExpandPseudo.cpp

250

This looks like a lot of very fragile pattern matching. I would greatly prefer it if we didn't have to do this.

llvm/lib/Target/X86/X86ISelLowering.cpp

3574

We don't in general reference the issue tracker from code comments, only from test cases.

[avl]

In D69372#1818166, @rnk wrote:

For the noimplicitfloat mode, the compiler mustn't generate
floating-point code if it was not asked directly to do so.

I don't think checking AL on function entry will work reliably. In general, musttail is meant to forward all possible register parameters. Normal arguments may be passed in XMM registers without setting AL. Consider this code:

typedef float v4f32 __attribute__((__vector_size__(16), __aligned__(16)));
int gv_i32;
v4f32 gv_v4f32;
void bar(int x, v4f32 v) {
  gv_i32 = x;
  gv_v4f32 = v;
}
void foo() {
  v4f32 v = {};
  bar(42, v);
}

It should be possible to use musttail thunk with a prototype of void (int, ...) between the call to foo and bar.

if I correctly understood the example - such a situation should not occur.
My understanding is that musttail thunk and it`s target _must_ have identical signatures.

In the above example, thunk assumed this definition: void (int, ...) while the target has this definition: void (int x, v4f32 v), that is wrong.

There should be either:

thunk: void (int, ...)
target: void (int, ...)

either:

thunk: void (int x, v4f32 v)
target: void (int x, v4f32 v)

In both these cases, this patch would work correctly.
Thus, assuming identical declarations, checking AL should work correctly.

And, if that thunk contains a function call, it will need to spill&fill all XMM argument registers. If you look at the generated assembly for calling bar, AL is not set to 1:

# %bb.0:                                # %entry
        pushq   %rax
        .cfi_def_cfa_offset 16
        xorps   %xmm0, %xmm0
        movl    $42, %edi
        callq   _Z3bariDv4_f

Right. AL is not set to 1. That is so because bar() does not use variable arguments according to its declaration.
Thunk function should assume the same signature and do not use AL trick then.

Is there actually a use case for musttail thunks and noimplicitfloat? Could we instead just declare that such a thunk represents explicit FP usage?

I do not know the real examples. But in general, it looks like a valid use case(musttail thunks and noimplicitfloat).

Furthermore, this will probably do the wrong thing on Windows, where AL is never set, to my knowledge. Consider this C++ code that uses musttail in the MS ABI:

int gv_i32;
double gv_f64;
class Foo {
  void virtual bar(int x, double v);
  void foo();
};
void Foo::bar(int x, double v) {
  gv_i32 = x;
  gv_f64 = v;
}
void Foo::foo() {
  auto mp = &Foo::bar;
  (this->*mp)(42, 0.0);
}

The virtual member pointer thunk needs to pass along XMM2, although it doesn't contain a function call.

This patch does not work on windows . It is only for amd64 ABI:

bool X86CallLowering::lowerCall()
...

if (STI.is64Bit() && !IsFixed && !STI.isCallingConvWin64(Info.CallConv)) {
  // From AMD64 ABI document:
  // For calls that may call functions that use varargs or stdargs
  // (prototype-less calls or calls to functions containing ellipsis (...) in
  // the declaration) %al is used as hidden argument to specify the number
  // of SSE registers used. The contents of %al do not need to match exactly
  // the number of registers, but must be an ubound on the number of SSE
  // registers used and is in the range 0 - 8 inclusive.

  MIRBuilder.buildInstr(X86::MOV8ri)
      .addDef(X86::AL)
      .addImm(Handler.getNumXmmRegs());
  MIB.addUse(X86::AL, RegState::Implicit);
}

X86ISelLowering.cpp:
// FIXME: Get this from tablegen.
static ArrayRef<MCPhysReg> get64BitArgumentXMMs(MachineFunction &MF,

                                              CallingConv::ID CallConv,
                                              const X86Subtarget &Subtarget) {
assert(Subtarget.is64Bit());
if (Subtarget.isCallingConvWin64(CallConv)) {
  // The XMM registers which might contain var arg parameters are shadowed
  // in their paired GPR.  So we only need to save the GPR to their home
  // slots.
  // TODO: __vectorcall will change this.
  return None;
}

[avl]

llvm/lib/Target/X86/X86ExpandPseudo.cpp

250

would rewrite it.

[rnk]

In D69372#1820099, @avl wrote:

In D69372#1818166, @rnk wrote:

It should be possible to use musttail thunk with a prototype of void (int, ...) between the call to foo and bar.

if I correctly understood the example - such a situation should not occur.
My understanding is that musttail thunk and it`s target _must_ have identical signatures.

The third point under musttail is meant to create an exception for for perfectly forwarding thunks: https://llvm.org/docs/LangRef.html#id325

If the musttail call appears in a function with the "thunk" attribute and the caller and callee both have varargs, than any unprototyped arguments in register or memory are forwarded to the callee. Similarly, the return value of the callee is returned to the caller’s caller, even if a void return type is in use.

In retrospect, I think it may have been a mistake to repurpose varargs to indicate that all remaining argument registers should be preserved. Maybe we don't need to use a varargs function prototype to implement perfectly forwarding thunks. We ended up needing the "thunk" function attribute, so we could base it on that instead.

This patch does not work on windows . It is only for amd64 ABI:

OK, but the C++ code I provided illustrates the use case of perfect forwarding, which might come up on Linux. It sets up a chain of calls that looks like:

• indirect call to thunk, pass FP in XMM ->
• in thunk, save XMM, use XMM, restore XMM, tail call to callee ->
• receive XMM value in callee

The use case for perfectly forwarding thunks is pretty rare. I don't think there is a way to convince clang to generate one for Linux, just Windows.

---

Anyway, it seems like you could do away with a fair amount of the complexity in this patch for handling guarding musttail forwarding by declaring them to be an "explicit FP use", since it is not, in general, possible for a perfectly forwarding thunk to know if XMM arguments have been used.

[avl]

In D69372#1820845, @rnk wrote:

In D69372#1820099, @avl wrote:

In D69372#1818166, @rnk wrote:

It should be possible to use musttail thunk with a prototype of void (int, ...) between the call to foo and bar.

if I correctly understood the example - such a situation should not occur.
My understanding is that musttail thunk and it`s target _must_ have identical signatures.

The third point under musttail is meant to create an exception for for perfectly forwarding thunks: https://llvm.org/docs/LangRef.html#id325

If the musttail call appears in a function with the "thunk" attribute and the caller and callee both have varargs, than any unprototyped arguments in register or memory are forwarded to the callee. Similarly, the return value of the callee is returned to the caller’s caller, even if a void return type is in use.

In retrospect, I think it may have been a mistake to repurpose varargs to indicate that all remaining argument registers should be preserved. Maybe we don't need to use a varargs function prototype to implement perfectly forwarding thunks. We ended up needing the "thunk" function attribute, so we could base it on that instead.

It looks like point 3 does not allow to have thunk signature to be "void (int, ...)" and callee signature "void (int x, v4f32 v)".
That point means that if caller and callee both have varargs, all varargs arguments should be properly forwarded from caller to callee.

There is another rule which looks relevant:

"The callee must be varargs iff the caller is varargs. Bitcasting a non-varargs function to the appropriate varargs type is legal so long as the non-varargs prefixes obey the other rules."

i.e. if caller/thunk signature "void (int, ...)", then the callee signature could be "void (int x, v4f32 v)".
Caller while calls thunk should use "void (int, ...)" signature.
Caller must set AL because of thunk signature "void (int, ...)".

that scenario should work correctly with/without this patch.

if currently there is a use case when caller/thunk have this signature "void (int, ...)" and AL is not set by caller(for AMD64 ABI) - then it looks like a bug in implementation.

This patch does not work on windows . It is only for amd64 ABI:

OK, but the C++ code I provided illustrates the use case of perfect forwarding, which might come up on Linux. It sets up a chain of calls that looks like:

• indirect call to thunk, pass FP in XMM ->
• in thunk, save XMM, use XMM, restore XMM, tail call to callee ->
• receive XMM value in callee

The use case for perfectly forwarding thunks is pretty rare. I don't think there is a way to convince clang to generate one for Linux, just Windows.

It seems to me that this case is already properly handled and my patch does not break it.
This example does not have varargs in function declarations.
Thus AL trick would not be used.

Both thunk and target have this signature : void (int x, double v).
It explicitly uses XMM(as per ABI for func params).
All following manipulations would be explicitly done:

• indirect call to thunk, pass FP in XMM -> OK
• in thunk, save XMM, use XMM, restore XMM, tail call to callee -> OK
• receive XMM value in callee -> OK

What would start to be wrong with this patch?

[avl]

@rnk

Anyway, it seems like you could do away with a fair amount of the complexity in this patch for handling guarding musttail forwarding by declaring them to be an "explicit FP use", since it is not, in general, possible for a perfectly forwarding thunk to know if XMM arguments have been used.

This effectively means that any program containing thunks could not be compiled for non-floating point environment even if it does not use fp at all.
Taking into account that thunks are usually inserted by the compiler - that could be unexpected and hard to solve for the user.
i.e. The program could not have floating point code at all but it would not be possible to compile it because of thunks.
I think it is not a good alternative.

In retrospect, I think it may have been a mistake to repurpose varargs to indicate that all remaining argument registers should be preserved. Maybe we don't need to use a varargs function prototype to implement perfectly forwarding thunks. We ended up needing the "thunk" function attribute, so we could base it on that instead.

I agree that it is a mistake to use varargs for perfectly forwarding thunks purposes. Since it does not conform to current behavior and documented rules:

1. testcase from llvm/test/CodeGen/X86/musttail-varargs.ll:

declare void @llvm.va_start(i8*) nounwind

declare void(i8*, ...)* @get_f(i8* %this)

define void @f_thunk(i8* %this, ...) {
  %ap = alloca [4 x i8*], align 16
  %ap_i8 = bitcast [4 x i8*]* %ap to i8*
  call void @llvm.va_start(i8* %ap_i8)

  %fptr = call void(i8*, ...)*(i8*) @get_f(i8* %this)
  musttail call void (i8*, ...) %fptr(i8* %this, ...)
  ret void
}

It has "f_thunk" which has llvm.va_start. Thus it saves incoming xmm registers into va_start area :

; LINUX-NEXT:    testb %al, %al
; LINUX-NEXT:    je .LBB0_2
; LINUX-NEXT:  # %bb.1:
; LINUX-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm1, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm2, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm3, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm4, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm5, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
; LINUX-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
; LINUX-NEXT:  .LBB0_2:

if f_thunk would be called without setting AL it would work incorrectly.

2. calling "void (int, ...)" function without setting AL is AMD64 ABI violation.

3. https://llvm.org/docs/LangRef.html#id325
   
   The caller and callee prototypes must match. Pointer types of parameters or return types may differ in pointee type, but not in address space. The calling conventions of the caller and callee must match. All ABI-impacting function attributes, such as sret, byval, inreg, returned, and inalloca, must match.
   
   These rules clearly states that thunk signature and callee signature should agree on calling conventions. if thunk assumes that AL should be set then it is an error to not setting it.

---

Probably we could fix implementation of perfectly forwarding thunks in an ABI compatible way ?

[avl]

@rnk Reid, I am trying to research the question of using varargs thunk for non-varargs methods.
(like using "void (int, ...)" thunk with "void (int x, v4f32 v)" signature as from the previous discussion).
I found following place which looks like precisely the case which uses musttail for not only varargs case:

CGVTables.cpp:EmitCallAndReturnForThunk():

// If perfect forwarding is required a variadic method, a method using
// inalloca, or an unprototyped thunk, use musttail. Emit an error if this
// thunk requires a return adjustment, since that is impossible with musttail.
if (CurFnInfo->usesInAlloca() || CurFnInfo->isVariadic() || IsUnprototyped) {

In cited fragments "CurFnInfo->isVariadic()" is a common usage for musttail.
"CurFnInfo->usesInAlloca()" and "IsUnprototyped" looks like extended usages.
Both of these features are specific for MS ABI:

1. InAlloca

https://llvm.org/docs/InAlloca.html

Primarily, this feature is required for compatibility with the Microsoft C++ ABI.

2. Unprototyped thunk

CGVTables.cpp:EmitCallAndReturnForThunk()
// Arrange a function prototype appropriate for a function definition. In some
// cases in the MS ABI, we may need to build an unprototyped musttail thunk.
const CGFunctionInfo &FnInfo =
    IsUnprototyped ? CGM.getTypes().arrangeUnprototypedMustTailThunk(MD)
                   : CGM.getTypes().arrangeGlobalDeclaration(GD);
llvm::FunctionType *ThunkFnTy = CGM.getTypes().GetFunctionType(FnInfo);

Thus it looks like the case using musttail for non-varargs functions never happens with AMD64 ABI.
So It is safe for this patch(Avoid spilling xmm vararg arguments) to rely on ABI requirement, and AL register should always be properly set for vararg functions.

The ability to avoid floating point usages is essential for environments not using floating point. If thunks would be considered as "explicit FP use" then many simple C++ programs could not be compiled for non-floating point environment because of unintended FP usage.

[avl]

addressed comments:

Do not create guarded registers for functions with "thunk" attribute.
Resolve style issues.
Check for MachineInstr::FrameDestroy instead of checking instruction patterns.

[merge_guards_bot]

Unit tests: fail. 62440 tests passed, 1 failed and 845 were skipped.

failed: libc++.std/thread/thread_mutex/thread_mutex_requirements/thread_mutex_requirements_mutex/thread_mutex_class/try_lock.pass.cpp

clang-tidy: fail. clang-tidy found 0 errors and 2 warnings. 0 of them are added as review comments below (why?).

clang-format: fail. Please format your changes with clang-format by running git-clang-format HEAD^ or applying this patch.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[avl]

clang-format recommendations: I was asked to not apply them in this review.
clang-tidy - would apply.
unit-test - looks like not reproduced, though I would check additionally.

[avl]

applied clang-tidy reccomendations, removed icall_branch_funnel workaround.

[avl]

Unit tests pass. 62573 tests passed, 0 failed and 842 were skipped.

clang-tidy pass.

clang-format fail.

[rnk]

llvm/lib/Target/X86/X86ISelLowering.cpp

3711

I haven't reviewed all of this code, but we have to find some way to refactor LowerCall. It was already poorly factored and long, but this is just too much complexity, too many lines of code. We need to find some way to separate concerns.

llvm/test/CodeGen/X86/musttail-varargs.ll

299–300

This seems unfortunate. :( Does all this go away if you put the "thunk" attribute on it? Everything in this file is meant to be a test for universal thunks, so adding the attribute is reasonable.

[avl]

llvm/lib/Target/X86/X86ISelLowering.cpp

3711

Ok, I would refactor it. Would it be OK if that refactoring would be a part of this patch ? Or Do I need to make it previously in separate patch ?

llvm/test/CodeGen/X86/musttail-varargs.ll

299–300

yes. all this xmm save/restore code would go away if "thunk" is specified. I will add it to the test case.

additionally, I would like to make separate patch which would NOT do this xmm store/restore if noimplicitfloat=false. So that store/restore code is generated for only noimplicitfloat=true case.

thus, I assume following patches would be done:

1. this patch. xmm stores/restores through phys regs would be generated for all _usual|_ thunks(not including universal thunks)
2. patch which will fix ABI breakage for noimplicitfloat case - D62639
3. do not generate xmm store/restore through phys regs for noimplicitfloat=false case.

[avl]

already fixed by D80163.

llvm/include/llvm/CodeGen/CallingConvLower.h

 //===- llvm/CallingConvLower.h - Calling Conventions ------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file declares the CCState and CCValAssign classes, used for lowering
 // and implementing calling conventions.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CODEGEN_CALLINGCONVLOWER_H
 #define LLVM_CODEGEN_CALLINGCONVLOWER_H
 
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/TargetCallingConv.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/Support/Alignment.h"
 
public:
   }
 
   bool isUpperBitsInLoc() const {
     return HTP == AExtUpper || HTP == SExtUpper || HTP == ZExtUpper;
   }
 };
 
 /// Describes a register that needs to be forwarded from the prologue to a
-/// musttail call.
+/// musttail call. Specifying VReg == 0 means that the register should be
+/// put into guarded area and no virtual register was created for it.
 struct ForwardedRegister {
   ForwardedRegister(unsigned VReg, MCPhysReg PReg, MVT VT)
       : VReg(VReg), PReg(PReg), VT(VT) {}
+  bool isGuarded() const { return VReg == 0; }
   unsigned VReg;
   MCPhysReg PReg;
   MVT VT;
 };
 
 /// CCAssignFn - This function assigns a location for Val, updating State to
 /// reflect the change.  It returns 'true' if it failed to handle Val.
 typedef bool CCAssignFn(unsigned ValNo, MVT ValVT,
public:
   /// Compute the remaining unused register parameters that would be used for
   /// the given value type. This is useful when varargs are passed in the
   /// registers that normal prototyped parameters would be passed in, or for
   /// implementing perfect forwarding.
   void getRemainingRegParmsForType(SmallVectorImpl<MCPhysReg> &Regs, MVT VT,
                                    CCAssignFn Fn);
 
   /// Compute the set of registers that need to be preserved and forwarded to
-  /// any musttail calls.
+  /// any musttail calls. Some platforms(AMD64) allow to guard access to
+  /// certain kind of input registers. f.e. Accesses to xmm registers should
+  /// be done according to the state of %al register. This function set
+  /// IsGuarded bit according to the specified set of GuardedForwardedRegs.
   void analyzeMustTailForwardedRegisters(
-      SmallVectorImpl<ForwardedRegister> &Forwards, ArrayRef<MVT> RegParmTypes,
-      CCAssignFn Fn);
+      SmallVectorImpl<ForwardedRegister> &Forwards,
+      const SmallDenseSet<MCPhysReg, 8> &GuardedForwardedRegs,
+      ArrayRef<MVT> RegParmTypes, CCAssignFn Fn);
 
   /// Returns true if the results of the two calling conventions are compatible.
   /// This is usually part of the check for tailcall eligibility.
   static bool resultsCompatible(CallingConv::ID CalleeCC,
                                 CallingConv::ID CallerCC, MachineFunction &MF,
                                 LLVMContext &C,
                                 const SmallVectorImpl<ISD::InputArg> &Ins,
                                 CCAssignFn CalleeFn, CCAssignFn CallerFn);

llvm/lib/CodeGen/CallingConvLower.cpp

#endif
   // as allocated so that future queries don't return the same registers, i.e.
   // when i64 and f64 are both passed in GPRs.
   StackOffset = SavedStackOffset;
   MaxStackArgAlign = SavedMaxStackArgAlign;
   Locs.resize(NumLocs);
 }
 
 void CCState::analyzeMustTailForwardedRegisters(
-    SmallVectorImpl<ForwardedRegister> &Forwards, ArrayRef<MVT> RegParmTypes,
-    CCAssignFn Fn) {
+    SmallVectorImpl<ForwardedRegister> &Forwards,
+    const SmallDenseSet<MCPhysReg, 8> &GuardedForwardedRegs,
+    ArrayRef<MVT> RegParmTypes, CCAssignFn Fn) {
   // Oftentimes calling conventions will not user register parameters for
   // variadic functions, so we need to assume we're not variadic so that we get
   // all the registers that might be used in a non-variadic call.
   SaveAndRestore<bool> SavedVarArg(IsVarArg, false);
   SaveAndRestore<bool> SavedMustTail(AnalyzingMustTailForwardedRegs, true);
 
   for (MVT RegVT : RegParmTypes) {
     SmallVector<MCPhysReg, 8> RemainingRegs;
     getRemainingRegParmsForType(RemainingRegs, RegVT, Fn);
     const TargetLowering *TL = MF.getSubtarget().getTargetLowering();
     const TargetRegisterClass *RC = TL->getRegClassFor(RegVT);
     for (MCPhysReg PReg : RemainingRegs) {
+      if (GuardedForwardedRegs.count(PReg) == 0) {
         unsigned VReg = MF.addLiveIn(PReg, RC);
         Forwards.push_back(ForwardedRegister(VReg, PReg, RegVT));
+      } else
+        Forwards.push_back(ForwardedRegister(0, PReg, RegVT));
     }
   }
 }
 
 bool CCState::resultsCompatible(CallingConv::ID CalleeCC,
                                 CallingConv::ID CallerCC, MachineFunction &MF,
                                 LLVMContext &C,
                                 const SmallVectorImpl<ISD::InputArg> &Ins,

llvm/lib/Target/AArch64/AArch64CallLowering.cpp

CCState CCInfo(F.getCallingConv(), /*IsVarArg=*/true, MF, ArgLocs,
                  F.getContext());
   SmallVector<MVT, 2> RegParmTypes;
   RegParmTypes.push_back(MVT::i64);
   RegParmTypes.push_back(MVT::f128);
 
   // Later on, we can use this vector to restore the registers if necessary.
   SmallVectorImpl<ForwardedRegister> &Forwards =
       FuncInfo->getForwardedMustTailRegParms();
-  CCInfo.analyzeMustTailForwardedRegisters(Forwards, RegParmTypes, AssignFn);
+
+  SmallDenseSet<MCPhysReg, 8> GuardedRegs;
+  CCInfo.analyzeMustTailForwardedRegisters(Forwards, GuardedRegs, RegParmTypes,
+                                           AssignFn);
 
   // Conservatively forward X8, since it might be used for an aggregate
   // return.
   if (!CCInfo.isAllocated(AArch64::X8)) {
     unsigned X8VReg = MF.addLiveIn(AArch64::X8, &AArch64::GPR64RegClass);
     Forwards.push_back(ForwardedRegister(X8VReg, AArch64::X8, MVT::i64));
   }
 

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (isVarArg) {
 
     if (MFI.hasMustTailInVarArgFunc()) {
       SmallVector<MVT, 2> RegParmTypes;
       RegParmTypes.push_back(MVT::i64);
       RegParmTypes.push_back(MVT::f128);
       // Compute the set of forwarded registers. The rest are scratch.
       SmallVectorImpl<ForwardedRegister> &Forwards =
                                        FuncInfo->getForwardedMustTailRegParms();
-      CCInfo.analyzeMustTailForwardedRegisters(Forwards, RegParmTypes,
-                                               CC_AArch64_AAPCS);
+
+      SmallDenseSet<MCPhysReg, 8> GuardedRegs;
+      CCInfo.analyzeMustTailForwardedRegisters(Forwards, GuardedRegs,
+                                               RegParmTypes, CC_AArch64_AAPCS);
 
       // Conservatively forward X8, since it might be used for aggregate return.
       if (!CCInfo.isAllocated(AArch64::X8)) {
         unsigned X8VReg = MF.addLiveIn(AArch64::X8, &AArch64::GPR64RegClass);
         Forwards.push_back(ForwardedRegister(X8VReg, AArch64::X8, MVT::i64));
       }
     }
   }

llvm/lib/Target/X86/X86ExpandPseudo.cpp

public:
   }
 
 private:
   void ExpandICallBranchFunnel(MachineBasicBlock *MBB,
                                MachineBasicBlock::iterator MBBI);
 
   bool ExpandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
   bool ExpandMBB(MachineBasicBlock &MBB);
+
+  void expandSaveVarargXmmRegs(MachineBasicBlock *MBB,
+                               MachineBasicBlock::iterator MBBI) const;
+
+  void createTailCallBlocksPair(MachineBasicBlock &OriginalTailCallBlk,
+                                MachineBasicBlock::iterator &TCPseudoInstr);
 };
 char X86ExpandPseudo::ID = 0;
 
 } // End anonymous namespace.
 
 INITIALIZE_PASS(X86ExpandPseudo, DEBUG_TYPE, X86_EXPAND_PSEUDO_NAME, false,
                 false)
 
void X86ExpandPseudo::ExpandICallBranchFunnel(
   for (auto P : TargetMBBs) {
     MF->insert(InsPt, P.first);
     BuildMI(P.first, DL, TII->get(X86::TAILJMPd64))
         .add(JTInst->getOperand(3 + 2 * P.second));
   }
   JTMBB->erase(JTInst);
 }
 
+// this function replaces original tail call instruction with two versions
+// of tailcall instruction. One is fully similar to original, another has xmm
+// registers restoring code inserted previously. Additionally there is created a
+// branch which checks %al and selects proper version of tailcall. This %al
+// trick is AMD64 specific thing.
+//
+//   f_thunk:                     f_thunk:
+//   # %bb.1:             =>      # %bb.1:
+//   addq    32, %rsp             testb   %al, %al
+//   jmpq    tc_func              je      .LBB0_2
+//                                # %bb.2:
+//                                movaps  96(%rsp), %xmm0
+//                                addq    32, %rsp
+//                                jmpq    tc_func
+//                                .LBB0_2:
+//                                # %bb.3:
+//                                addq    32, %rsp
+//                                jmpq    tc_func
+//
+void X86ExpandPseudo::createTailCallBlocksPair(
+    MachineBasicBlock &OriginalTailCallBlk,
+    MachineBasicBlock::iterator &TCPseudoInstr) {
+
+  MachineFunction *Func = OriginalTailCallBlk.getParent();
+  X86MachineFunctionInfo *X86Info = Func->getInfo<X86MachineFunctionInfo>();
+  const auto &Forwards = X86Info->getForwardedMustTailRegParms();
+  const BasicBlock *BB = OriginalTailCallBlk.getBasicBlock();
+
+  MachineBasicBlock::iterator TailCallMInstr = std::prev(TCPseudoInstr);
+  DebugLoc DL = TCPseudoInstr->getDebugLoc();
+
+  // create two blocks for tailcalls.
+  MachineFunction::iterator MBBIter = ++OriginalTailCallBlk.getIterator();
+  MachineBasicBlock *TailCallBlkWithGuardedRegs =
+      Func->CreateMachineBasicBlock(BB);
+  MachineBasicBlock *TailCallBlk = Func->CreateMachineBasicBlock(BB);
+  Func->insert(MBBIter, TailCallBlkWithGuardedRegs);
+  Func->insert(MBBIter, TailCallBlk);
+
+  TailCallBlk->transferSuccessors(&OriginalTailCallBlk);
+  OriginalTailCallBlk.addSuccessor(TailCallBlkWithGuardedRegs);
+  OriginalTailCallBlk.addSuccessor(TailCallBlk);
+
+  // search for the first stack frame destroying instruction
+  MachineBasicBlock::iterator FirstStackFrameDestroyingInstr = &*TailCallMInstr;
+  for (MachineBasicBlock::iterator I = TCPseudoInstr;
+       I != OriginalTailCallBlk.begin(); --I) {
+    MachineBasicBlock::iterator PI = std::prev(I);
+    if (PI->getFlag(MachineInstr::FrameDestroy))
+      FirstStackFrameDestroyingInstr = PI;
+  }
+
+  // copy stack restoring code and tailcall instruction into
+  // two created blocks. Delete copied instructions from the
+  // OriginalTailCallBlk.
+  MachineBasicBlock::iterator CurInstr = FirstStackFrameDestroyingInstr;
+  do {
+    // duplicate instructions and put them into new blocks.
+    // handle CFI instructions separately.
+    if (CurInstr->isCFIInstruction()) {
+      BuildMI(*TailCallBlkWithGuardedRegs, TailCallBlkWithGuardedRegs->end(),
+              TailCallBlkWithGuardedRegs->findDebugLoc(
+                  TailCallBlkWithGuardedRegs->begin()),
+              TII->get(TargetOpcode::CFI_INSTRUCTION))
+          .addCFIIndex(CurInstr->getOperand(0).getCFIIndex());
+      BuildMI(*TailCallBlk, TailCallBlk->end(),
+              TailCallBlk->findDebugLoc(TailCallBlk->begin()),
+              TII->get(TargetOpcode::CFI_INSTRUCTION))
+          .addCFIIndex(CurInstr->getOperand(0).getCFIIndex());
+    } else {
+      TII->duplicate(*TailCallBlkWithGuardedRegs,
+                     TailCallBlkWithGuardedRegs->end(), *CurInstr);
+
+      TII->duplicate(*TailCallBlk, TailCallBlk->end(), *CurInstr);
+    }
+
+    // stop copying if we achieved tail call instruction
+    if (CurInstr->getOpcode() == TailCallMInstr->getOpcode()) {
+      OriginalTailCallBlk.erase(CurInstr);
+      break;
+    }
+
+    CurInstr = &*OriginalTailCallBlk.erase(CurInstr);
+  } while (CurInstr != OriginalTailCallBlk.end());
+
+  // copy call site information into new tail call instructions
+  OriginalTailCallBlk.getParent()->copyCallSiteInfo(
+      &*TCPseudoInstr, &*TailCallBlkWithGuardedRegs->getLastNonDebugInstr());
+
+  OriginalTailCallBlk.getParent()->copyCallSiteInfo(
+      &*TCPseudoInstr, &*TailCallBlk->getLastNonDebugInstr());
+
+  // If %al is 0, branch around the XMM save block.
+  BuildMI(&OriginalTailCallBlk, DL, TII->get(X86::TEST8rr))
+      .addReg(X86::AL)
+      .addReg(X86::AL);
+  BuildMI(&OriginalTailCallBlk, DL, TII->get(X86::JCC_1))
+      .addMBB(TailCallBlk)
+      .addImm(X86::COND_E);
+
+  // add code restoring xmm regsiters into start of TailCallInstrFromGuardedBlk
+  MachineInstr &TailCallInstrFromGuardedBlk =
+      *TailCallBlkWithGuardedRegs->getLastNonDebugInstr();
+
+  // TODO: take into account YMM, ZMM here
+  unsigned MOVOpc = STI->hasAVX() ? X86::VMOVAPSrm : X86::MOVAPSrm;
+
+  unsigned BaseReg;
+  int64_t FrameOffset = X86FL->getFrameIndexReference(
+      *Func, X86Info->getThunkRegSaveFrameIndex(), BaseReg);
+  int64_t SaveAreaOffset =
+      (Func->getFrameInfo().hasVAStart() ? X86Info->getVarArgsFPOffset() : 0);
+
+  int RegIdx = 0;
+  for (const auto &Fwd : Forwards) {
+    if (Fwd.isGuarded()) {
+      int64_t Offset = FrameOffset + SaveAreaOffset + RegIdx * 16;
+
+      MachineMemOperand *MMO = Func->getMachineMemOperand(
+          MachinePointerInfo::getFixedStack(
+              *Func, X86Info->getThunkRegSaveFrameIndex(), Offset),
+          MachineMemOperand::MOLoad,
+          /*Size=*/16, /*Align=*/16);
+
+      BuildMI(*TailCallBlkWithGuardedRegs, TailCallBlkWithGuardedRegs->begin(),
+              DL, TII->get(MOVOpc), Fwd.PReg)
+          .addReg(BaseReg)
+          .addImm(/*Scale=*/1)
+          .addReg(/*IndexReg=*/0)
+          .addImm(/*Disp=*/Offset)
+          .addReg(/*Segment=*/0)
+          .addMemOperand(MMO);
+
+      TailCallInstrFromGuardedBlk.addOperand(
+          MachineOperand::CreateReg(Fwd.PReg, false /*IsDef*/, true /*IsImp*/));
+      RegIdx++;
+    }
+  }
+
+  // add liveins into newly created blocks
+  for (auto &MO : TCPseudoInstr->operands()) {
+    if (MO.isReg() && Register::isPhysicalRegister(MO.getReg())) {
+      TailCallBlk->addLiveIn(MO.getReg());
+      TailCallBlkWithGuardedRegs->addLiveIn(MO.getReg());
+    }
+  }
+}
+
 /// If \p MBBI is a pseudo instruction, this method expands
 /// it to the corresponding (sequence of) actual instruction(s).
 /// \returns true if \p MBBI has been expanded.
 bool X86ExpandPseudo::ExpandMI(MachineBasicBlock &MBB,
                                MachineBasicBlock::iterator MBBI) {
   MachineInstr &MI = *MBBI;
   unsigned Opcode = MI.getOpcode();
   DebugLoc DL = MBBI->getDebugLoc();
case X86::TCRETURNmi64: {
     } else {
       JumpTarget.setIsKill();
       BuildMI(MBB, MBBI, DL, TII->get(X86::TAILJMPr))
           .add(JumpTarget);
     }
 
     MachineInstr &NewMI = *std::prev(MBBI);
     NewMI.copyImplicitOps(*MBBI->getParent()->getParent(), *MBBI);
-    MBB.getParent()->moveCallSiteInfo(&*MBBI, &NewMI);
+    MBB.getParent()->copyCallSiteInfo(&*MBBI, &NewMI);
+
+    MachineFunction *Func = MBB.getParent();
+    X86MachineFunctionInfo *X86Info = Func->getInfo<X86MachineFunctionInfo>();
+    const auto &Forwards = X86Info->getForwardedMustTailRegParms();
+
+    // if tailcall return sequence is a "musttail"
+    // and some of forwarded registers should be guarded
+    // then replace current tailcall return sequence
+    // with two return sequences: one which restores
+    // guarded registers and another one which does not.
+    // Otherwise, leave current tailcall return sequence as is.
+    if (Func->getFrameInfo().hasMustTailInVarArgFunc()) {
+      for (const auto &F : Forwards)
+        if (F.isGuarded()) {
+          createTailCallBlocksPair(MBB, MBBI);
+          break;
+        }
+    }
 
     // Delete the pseudo instruction TCRETURN.
+    MBB.getParent()->eraseCallSiteInfo(&*MBBI);
     MBB.erase(MBBI);
 
     return true;
   }
   case X86::EH_RETURN:
   case X86::EH_RETURN64: {
     MachineOperand &DestAddr = MBBI->getOperand(0);
     assert(DestAddr.isReg() && "Offset should be in register!");
case X86::LCMPXCHG16B_SAVE_RBX: {
 
     // Delete the pseudo.
     MBBI->eraseFromParent();
     return true;
   }
   case TargetOpcode::ICALL_BRANCH_FUNNEL:
     ExpandICallBranchFunnel(&MBB, MBBI);
     return true;
+
+  case X86::SAVE_VARARG_XMM_REGS:
+    expandSaveVarargXmmRegs(&MBB, MBBI);
+    return true;
   }
   llvm_unreachable("Previous switch has a fallthrough?");
 }
 
 /// Expand all pseudo instructions contained in \p MBB.
 /// \returns true if any expansion occurred for \p MBB.
 bool X86ExpandPseudo::ExpandMBB(MachineBasicBlock &MBB) {
   bool Modified = false;
 
   // MBBI may be invalidated by the expansion.
   MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
   while (MBBI != E) {
     MachineBasicBlock::iterator NMBBI = std::next(MBBI);
     Modified |= ExpandMI(MBB, MBBI);
     MBBI = NMBBI;
   }
 
   return Modified;
 }
 
+/// This function replaces X86::SAVE_VARARG_XMM_REGS pseudo instruction
+/// with set of copying instructions for specified xmm vararg registers.
+///
+/// [0] parameter of X86::SAVE_VARARG_XMM_REGS is frame index of stack area,
+///     where registers should be stored
+/// [1] parameter of X86::SAVE_VARARG_XMM_REGS is offset inside stack frame
+///     to the area where registers should be stored
+/// [2] - [till end] parameters of X86::SAVE_VARARG_XMM_REGS are set of
+///     xmm registers which should be stored.
+void X86ExpandPseudo::expandSaveVarargXmmRegs(
+    MachineBasicBlock *GuardedBlock,
+    MachineBasicBlock::iterator SaveVarargXmmRegsInstr) const {
+  assert(SaveVarargXmmRegsInstr->getOpcode() == X86::SAVE_VARARG_XMM_REGS);
+
+  MachineFunction *Func = GuardedBlock->getParent();
+  DebugLoc DL = SaveVarargXmmRegsInstr->getDebugLoc();
+
+  int64_t FrameIndex = SaveVarargXmmRegsInstr->getOperand(0).getImm();
+  unsigned BaseReg;
+  int64_t FrameOffset =
+      X86FL->getFrameIndexReference(*Func, FrameIndex, BaseReg);
+  int64_t VarArgsRegsOffset = SaveVarargXmmRegsInstr->getOperand(1).getImm();
+
+  // TODO: add support for YMM and ZMM here.
+  unsigned MOVOpc = STI->hasAVX() ? X86::VMOVAPSmr : X86::MOVAPSmr;
+
+  // In the XMM save block, save all the XMM argument registers.
+  for (int64_t OpndIdx = 2, RegIdx = 0;
+       OpndIdx < SaveVarargXmmRegsInstr->getNumOperands();
+       OpndIdx++, RegIdx++) {
+
+    int64_t Offset = FrameOffset + VarArgsRegsOffset + RegIdx * 16;
+
+    MachineMemOperand *MMO = Func->getMachineMemOperand(
+        MachinePointerInfo::getFixedStack(*Func, FrameIndex, Offset),
+        MachineMemOperand::MOStore,
+        /*Size=*/16, /*Align=*/16);
+
+    BuildMI(GuardedBlock, DL, TII->get(MOVOpc))
+        .addReg(BaseReg)
+        .addImm(/*Scale=*/1)
+        .addReg(/*IndexReg=*/0)
+        .addImm(/*Disp=*/Offset)
+        .addReg(/*Segment=*/0)
+        .addReg(SaveVarargXmmRegsInstr->getOperand(OpndIdx).getReg())
+        .addMemOperand(MMO);
+    assert(Register::isPhysicalRegister(
+        SaveVarargXmmRegsInstr->getOperand(OpndIdx).getReg()));
+
+    GuardedBlock->addLiveIn(
+        SaveVarargXmmRegsInstr->getOperand(OpndIdx).getReg());
+  }
+
+  // Delete the pseudo.
+  SaveVarargXmmRegsInstr->eraseFromParent();
+}
+
 bool X86ExpandPseudo::runOnMachineFunction(MachineFunction &MF) {
   STI = &static_cast<const X86Subtarget &>(MF.getSubtarget());
   TII = STI->getInstrInfo();
   TRI = STI->getRegisterInfo();
   X86FI = MF.getInfo<X86MachineFunctionInfo>();
   X86FL = STI->getFrameLowering();
 
   bool Modified = false;

llvm/lib/Target/X86/X86ISelLowering.h

 #include "llvm/CodeGen/TargetLowering.h"
 
 namespace llvm {
   class X86Subtarget;
   class X86TargetMachine;
 
   namespace X86ISD {
     // X86 Specific DAG Nodes
     enum NodeType : unsigned {
       // Start the numbering where the builtin ops leave off.
       FIRST_NUMBER = ISD::BUILTIN_OP_END,
 
       /// Bit scan forward.
       BSF,
       /// Bit scan reverse.
       BSR,
 
enum NodeType : unsigned {
 
       // Thread Local Storage. A call to get the start address
       // of the TLS block for the current module.
       TLSBASEADDR,
 
       // Thread Local Storage.  When calling to an OS provided
       // thunk at the address from an earlier relocation.
       TLSCALL,
 
       // Exception Handling helpers.
       EH_RETURN,
 
       // SjLj exception handling setjmp.
       EH_SJLJ_SETJMP,
 
       // SjLj exception handling longjmp.
       EH_SJLJ_LONGJMP,
enum NodeType : unsigned {
       // Dot product of BF16 pairs to accumulated into
       // packed single precision.
       DPBF16PS,
 
       // Save xmm argument registers to the stack, according to %al. An operator
       // is needed so that this can be expanded with control flow.
       VASTART_SAVE_XMM_REGS,
 
+      // Save xmm argument registers of the vararg thunk function to the stack,
+      // according to %al. An operator is needed so that this can be expanded with
+      // control flow.
+      MUSTTAIL_SAVE_GUARDED_REGS,
+
       // Windows's _chkstk call to do stack probing.
       WIN_ALLOCA,
 
       // For allocating variable amounts of stack space when using
       // segmented stacks. Check if the current stacklet has enough space, and
       // falls back to heap allocation if not.
       SEG_ALLOCA,
 
private:
 
     bool needsCmpXchgNb(Type *MemType) const;
 
     void SetupEntryBlockForSjLj(MachineInstr &MI, MachineBasicBlock *MBB,
                                 MachineBasicBlock *DispatchBB, int FI) const;
 
     // Utility function to emit the low-level va_arg code for X86-64.
     MachineBasicBlock *
-    EmitVAARG64WithCustomInserter(MachineInstr &MI,
+    emitVAARG64WithCustomInserter(MachineInstr &MI,
                                   MachineBasicBlock *MBB) const;
 
     /// Utility function to emit the xmm reg save portion of va_start.
     MachineBasicBlock *
-    EmitVAStartSaveXMMRegsWithCustomInserter(MachineInstr &BInstr,
+    emitVAStartSaveXMMRegsWithCustomInserter(MachineInstr &BInstr,
                                              MachineBasicBlock *BB) const;
 
+    /// Utility function to emit the guarded xmm regs saving block.
+    MachineBasicBlock *
+    emitVarargThunkSaveXMMRegsWithCustomInserter(MachineInstr &BInstr,
+                                                 MachineBasicBlock *BB) const;
+
+    void addSaveVarargXmmRegsPseudo(MachineBasicBlock *GuardedRegsBlk,
+                                    MachineBasicBlock *TailBlk,
+                                    MachineInstr &SrcPseudoInstr) const;
+
     MachineBasicBlock *EmitLoweredCascadedSelect(MachineInstr &MI1,
                                                  MachineInstr &MI2,
                                                  MachineBasicBlock *BB) const;
 
     MachineBasicBlock *EmitLoweredSelect(MachineInstr &I,
                                          MachineBasicBlock *BB) const;
 
     MachineBasicBlock *EmitLoweredAtomicFP(MachineInstr &I,

llvm/lib/Target/X86/X86ISelLowering.cpp

if (MFI.hasVAStart() &&
     FuncInfo->setVarArgsFrameIndex(MFI.CreateFixedObject(1, StackSize, true));
   }
 
   // Figure out if XMM registers are in use.
   assert(!(Subtarget.useSoftFloat() &&
            F.hasFnAttribute(Attribute::NoImplicitFloat)) &&
          "SSE register cannot be used when SSE is disabled!");
 
+  SmallDenseSet<MCPhysReg, 8> GuardedRegs;
+  SmallVector<SDValue, 6> LiveGPRs;
+  SmallVector<SDValue, 8> LiveXMMRegs;
+  SDValue ALVal;
+
   // 64-bit calling conventions support varargs and register parameters, so we
   // have to do extra work to spill them in the prologue.
-  if (Is64Bit && isVarArg && MFI.hasVAStart()) {
+  if (Is64Bit && isVarArg) {
     // Find the first unallocated argument registers.
     ArrayRef<MCPhysReg> ArgGPRs = get64BitArgumentGPRs(CallConv, Subtarget);
     ArrayRef<MCPhysReg> ArgXMMs = get64BitArgumentXMMs(MF, CallConv, Subtarget);
     unsigned NumIntRegs = CCInfo.getFirstUnallocated(ArgGPRs);
     unsigned NumXMMRegs = CCInfo.getFirstUnallocated(ArgXMMs);
     assert(!(NumXMMRegs && !Subtarget.hasSSE1()) &&
            "SSE register cannot be used when SSE is disabled!");
 
     // Gather all the live in physical registers.
-    SmallVector<SDValue, 6> LiveGPRs;
-    SmallVector<SDValue, 8> LiveXMMRegs;
-    SDValue ALVal;
     for (MCPhysReg Reg : ArgGPRs.slice(NumIntRegs)) {
       unsigned GPR = MF.addLiveIn(Reg, &X86::GR64RegClass);
       LiveGPRs.push_back(
           DAG.getCopyFromReg(Chain, dl, GPR, MVT::i64));
     }
+
     if (!ArgXMMs.empty()) {
       unsigned AL = MF.addLiveIn(X86::AL, &X86::GR8RegClass);
       ALVal = DAG.getCopyFromReg(Chain, dl, AL, MVT::i8);
       for (MCPhysReg Reg : ArgXMMs.slice(NumXMMRegs)) {
-        unsigned XMMReg = MF.addLiveIn(Reg, &X86::VR128RegClass);
-        LiveXMMRegs.push_back(
-            DAG.getCopyFromReg(Chain, dl, XMMReg, MVT::v4f32));
+        // FastRegisterAllocator spills virtual registers at basic
+        // block boundary. That leads to usages of xmm registers
+        // outside of check for %al. Pass physical registers to
+        // VASTART_SAVE_XMM_REGS to avoid unneccessary spilling.
+        MF.getRegInfo().addLiveIn(Reg);
+        LiveXMMRegs.push_back(DAG.getRegister(Reg, MVT::v4f32));
+
+        // 'musttail' calls forward input registers from thunk function to
+        // callee. AMD64 ABI allows to avoid access to xmm registers by guarding
+        // them with %al register. That behavior is important for
+        // NoImplicitFloat case. To implement that behavior we create set of
+        // registers which should be guarded while forwarding.
+        //
+        // We do not guard registers for functions with "thunk" attribute
+        // currently. Attribute "thunk" points to special kind of thunk
+        // function: "perfectly forwarding thunk". It is assumed that functions
+        // with "thunk" attribute should not be used in NoImplicitFloat case.
+        if (!F.hasFnAttribute("thunk"))
+          GuardedRegs.insert(Reg);
       }
     }
 
+    if (MFI.hasVAStart()) {
       if (IsWin64) {
         // Get to the caller-allocated home save location.  Add 8 to account
         // for the return address.
         int HomeOffset = TFI.getOffsetOfLocalArea() + 8;
         FuncInfo->setRegSaveFrameIndex(
             MFI.CreateFixedObject(1, NumIntRegs * 8 + HomeOffset, false));
         // Fixup to set vararg frame on shadow area (4 x i64).
         if (NumIntRegs < 4)
           FuncInfo->setVarArgsFrameIndex(FuncInfo->getRegSaveFrameIndex());
       } else {
         // For X86-64, if there are vararg parameters that are passed via
         // registers, then we must store them to their spots on the stack so
         // they may be loaded by dereferencing the result of va_next.
         FuncInfo->setVarArgsGPOffset(NumIntRegs * 8);
         FuncInfo->setVarArgsFPOffset(ArgGPRs.size() * 8 + NumXMMRegs * 16);
         FuncInfo->setRegSaveFrameIndex(MFI.CreateStackObject(
             ArgGPRs.size() * 8 + ArgXMMs.size() * 16, 16, false));
       }
 
       // Store the integer parameter registers.
       SmallVector<SDValue, 8> MemOps;
       SDValue RSFIN = DAG.getFrameIndex(FuncInfo->getRegSaveFrameIndex(),
                                         getPointerTy(DAG.getDataLayout()));
       unsigned Offset = FuncInfo->getVarArgsGPOffset();
       for (SDValue Val : LiveGPRs) {
-      SDValue FIN = DAG.getNode(ISD::ADD, dl, getPointerTy(DAG.getDataLayout()),
-                                RSFIN, DAG.getIntPtrConstant(Offset, dl));
+        SDValue FIN =
+            DAG.getNode(ISD::ADD, dl, getPointerTy(DAG.getDataLayout()), RSFIN,
+                        DAG.getIntPtrConstant(Offset, dl));
         SDValue Store =
             DAG.getStore(Val.getValue(1), dl, Val, FIN,
                          MachinePointerInfo::getFixedStack(
                              DAG.getMachineFunction(),
                              FuncInfo->getRegSaveFrameIndex(), Offset));
         MemOps.push_back(Store);
         Offset += 8;
       }
 
       if (!ArgXMMs.empty() && NumXMMRegs != ArgXMMs.size()) {
         // Now store the XMM (fp + vector) parameter registers.
         SmallVector<SDValue, 12> SaveXMMOps;
         SaveXMMOps.push_back(Chain);
         SaveXMMOps.push_back(ALVal);
-      SaveXMMOps.push_back(DAG.getIntPtrConstant(
-                             FuncInfo->getRegSaveFrameIndex(), dl));
-      SaveXMMOps.push_back(DAG.getIntPtrConstant(
-                             FuncInfo->getVarArgsFPOffset(), dl));
+        SaveXMMOps.push_back(
+            DAG.getIntPtrConstant(FuncInfo->getRegSaveFrameIndex(), dl));
+        SaveXMMOps.push_back(
+            DAG.getIntPtrConstant(FuncInfo->getVarArgsFPOffset(), dl));
         SaveXMMOps.insert(SaveXMMOps.end(), LiveXMMRegs.begin(),
                           LiveXMMRegs.end());
         MemOps.push_back(DAG.getNode(X86ISD::VASTART_SAVE_XMM_REGS, dl,
                                      MVT::Other, SaveXMMOps));
       }
 
       if (!MemOps.empty())
         Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, MemOps);
     }
+  }
 
   if (isVarArg && MFI.hasMustTailInVarArgFunc()) {
     // Find the largest legal vector type.
     MVT VecVT = MVT::Other;
     // FIXME: Only some x86_32 calling conventions support AVX512.
     if (Subtarget.useAVX512Regs() &&
         (Is64Bit || (CallConv == CallingConv::X86_VectorCall ||
                      CallConv == CallingConv::Intel_OCL_BI)))
       VecVT = MVT::v16f32;
     else if (Subtarget.hasAVX())
       VecVT = MVT::v8f32;
     else if (Subtarget.hasSSE2())
       VecVT = MVT::v4f32;
 
     // We forward some GPRs and some vector types.
     SmallVector<MVT, 2> RegParmTypes;
     MVT IntVT = Is64Bit ? MVT::i64 : MVT::i32;
     RegParmTypes.push_back(IntVT);
     if (VecVT != MVT::Other)
       RegParmTypes.push_back(VecVT);
 
     // Compute the set of forwarded registers. The rest are scratch.
     SmallVectorImpl<ForwardedRegister> &Forwards =
         FuncInfo->getForwardedMustTailRegParms();
-    CCInfo.analyzeMustTailForwardedRegisters(Forwards, RegParmTypes, CC_X86);
+    CCInfo.analyzeMustTailForwardedRegisters(Forwards, GuardedRegs,
+                                             RegParmTypes, CC_X86);
 
     // Forward AL for SysV x86_64 targets, since it is used for varargs.
     if (Is64Bit && !IsWin64 && !CCInfo.isAllocated(X86::AL)) {
       unsigned ALVReg = MF.addLiveIn(X86::AL, &X86::GR8RegClass);
       Forwards.push_back(ForwardedRegister(ALVReg, X86::AL, MVT::i8));
     }
 
     // Copy all forwards from physical to virtual registers.
     for (ForwardedRegister &FR : Forwards) {
       // FIXME: Can we use a less constrained schedule?
+      if (!FR.isGuarded()) {
         SDValue RegVal = DAG.getCopyFromReg(Chain, dl, FR.VReg, FR.VT);
         FR.VReg = MF.getRegInfo().createVirtualRegister(getRegClassFor(FR.VT));
         Chain = DAG.getCopyToReg(Chain, dl, FR.VReg, RegVal);
       }
     }
 
+    if (!GuardedRegs.empty()) {
+      if (MFI.hasVAStart()) {
+        // all incoming xmm registers are already stored by VAStart
+        // handling. Reuse these stored values for thunk forwarded
+        // parameters here.
+        FuncInfo->setThunkRegSaveFrameIndex(FuncInfo->getRegSaveFrameIndex());
+      } else {
+        // TODO: implement support for YMM, ZMM vararg registers
+
+        // allocate stack space to save registers which should be guarded by
+        // ABI, 16 is size of XMM
+        FuncInfo->setThunkRegSaveFrameIndex(
+            MFI.CreateStackObject(GuardedRegs.size() * 16, 16, false));
+
+        // Save guarded forwards into guarded area
+        SmallVector<SDValue, 8> VarargMemOps;
+        SmallVector<SDValue, 12> VarargXMMOps;
+        VarargXMMOps.push_back(Chain);
+        VarargXMMOps.push_back(ALVal);
+        VarargXMMOps.push_back(
+            DAG.getIntPtrConstant(FuncInfo->getThunkRegSaveFrameIndex(), dl));
+        VarargXMMOps.push_back(DAG.getIntPtrConstant(0, dl));
+        VarargXMMOps.insert(VarargXMMOps.end(), LiveXMMRegs.begin(),
+                            LiveXMMRegs.end());
+        VarargMemOps.push_back(DAG.getNode(X86ISD::MUSTTAIL_SAVE_GUARDED_REGS,
+                                           dl, MVT::Other, VarargXMMOps));
+        if (!VarargMemOps.empty())
+          Chain = DAG.getNode(ISD::TokenFactor, dl, MVT::Other, VarargMemOps);
+      }
+    }
+  }
+
   // Some CCs need callee pop.
   if (X86::isCalleePop(CallConv, Is64Bit, isVarArg,
                        MF.getTarget().Options.GuaranteedTailCallOpt)) {
     FuncInfo->setBytesToPopOnReturn(StackSize); // Callee pops everything.
   } else if (CallConv == CallingConv::X86_INTR && Ins.size() == 2) {
     // X86 interrupts must pop the error code (and the alignment padding) if
     // present.
     FuncInfo->setBytesToPopOnReturn(Is64Bit ? 16 : 4);
   } else {
     FuncInfo->setBytesToPopOnReturn(0); // Callee pops nothing.
     // If this is an sret function, the return should pop the hidden pointer.
     if (!Is64Bit && !canGuaranteeTCO(CallConv) &&
         !Subtarget.getTargetTriple().isOSMSVCRT() &&
         argsAreStructReturn(Ins, Subtarget.isTargetMCU()) == StackStructReturn)
       FuncInfo->setBytesToPopOnReturn(4);
   }
 
   if (!Is64Bit) {
-    // RegSaveFrameIndex is X86-64 only.
+    // RegSaveFrameIndex and ThunkRegSaveFrameIndex is X86-64 only.
     FuncInfo->setRegSaveFrameIndex(0xAAAAAAA);
+    FuncInfo->setThunkRegSaveFrameIndex(0xAAAAAAA);
     if (CallConv == CallingConv::X86_FastCall ||
         CallConv == CallingConv::X86_ThisCall)
       // fastcc functions can't have varargs.
       FuncInfo->setVarArgsFrameIndex(0xAAAAAAA);
   }
 
   FuncInfo->setArgumentStackSize(StackSize);
 
if (Is64Bit && isVarArg && !IsWin64 && !IsMustTail) {
     RegsToPass.push_back(std::make_pair(unsigned(X86::AL),
                                         DAG.getConstant(NumXMMRegs, dl,
                                                         MVT::i8)));
   }
 
   if (isVarArg && IsMustTail) {
     const auto &Forwards = X86Info->getForwardedMustTailRegParms();
     for (const auto &F : Forwards) {
+      if (!F.isGuarded()) {
         SDValue Val = DAG.getCopyFromReg(Chain, dl, F.VReg, F.VT);
         RegsToPass.push_back(std::make_pair(unsigned(F.PReg), Val));
       }
     }
+  }
 
   // For tail calls lower the arguments to the 'real' stack slots.  Sibcalls
   // don't need this because the eligibility check rejects calls that require
   // shuffling arguments passed in memory.
   if (!IsSibcall && isTailCall) {
     // Force all the incoming stack arguments to be loaded from the stack
     // before any new outgoing arguments are stored to the stack, because the
     // outgoing stack slots may alias the incoming argument stack slots, and
#define NODE_NAME_CASE(NODE) case X86ISD::NODE: return "X86ISD::" #NODE;
   NODE_NAME_CASE(VRANGE_SAE)
   NODE_NAME_CASE(VRANGES)
   NODE_NAME_CASE(VRANGES_SAE)
   NODE_NAME_CASE(PMULUDQ)
   NODE_NAME_CASE(PMULDQ)
   NODE_NAME_CASE(PSADBW)
   NODE_NAME_CASE(DBPSADBW)
   NODE_NAME_CASE(VASTART_SAVE_XMM_REGS)
+  NODE_NAME_CASE(MUSTTAIL_SAVE_GUARDED_REGS)
   NODE_NAME_CASE(VAARG_64)
   NODE_NAME_CASE(WIN_ALLOCA)
   NODE_NAME_CASE(MEMBARRIER)
   NODE_NAME_CASE(MFENCE)
   NODE_NAME_CASE(SEG_ALLOCA)
   NODE_NAME_CASE(PROBED_ALLOCA)
   NODE_NAME_CASE(RDRAND)
   NODE_NAME_CASE(RDSEED)
static MachineBasicBlock *emitXBegin(MachineInstr &MI, MachineBasicBlock *MBB,
   BuildMI(*sinkMBB, sinkMBB->begin(), DL, TII->get(X86::PHI), DstReg)
       .addReg(mainDstReg).addMBB(mainMBB)
       .addReg(fallDstReg).addMBB(fallMBB);
 
   MI.eraseFromParent();
   return sinkMBB;
 }
 
 
 
 MachineBasicBlock *
-X86TargetLowering::EmitVAARG64WithCustomInserter(MachineInstr &MI,
+X86TargetLowering::emitVAARG64WithCustomInserter(MachineInstr &MI,
                                                  MachineBasicBlock *MBB) const {
   // Emit va_arg instruction on X86-64.
 
   // Operands to this pseudo-instruction:
   // 0  ) Output        : destination address (reg)
   // 1-5) Input         : va_list address (addr, i64mem)
   // 6  ) ArgSize       : Size (in bytes) of vararg type
   // 7  ) ArgMode       : 0=overflow only, 1=use gp_offset, 2=use fp_offset
X86TargetLowering::emitVAARG64WithCustomInserter(MachineInstr &MI,
   }
 
   // Erase the pseudo instruction
   MI.eraseFromParent();
 
   return endMBB;
 }
 
-MachineBasicBlock *X86TargetLowering::EmitVAStartSaveXMMRegsWithCustomInserter(
-    MachineInstr &MI, MachineBasicBlock *MBB) const {
-  // Emit code to save XMM registers to the stack. The ABI says that the
-  // number of registers to save is given in %al, so it's theoretically
-  // possible to do an indirect jump trick to avoid saving all of them,
-  // however this code takes a simpler approach and just executes all
-  // of the stores if %al is non-zero. It's less code, and it's probably
-  // easier on the hardware branch predictor, and stores aren't all that
-  // expensive anyway.
+// This function creates additional block for storing varargs guarded
+// registers. It adds check for %al into entry block, to skip
+// GuardedRegsBlk if xmm registers should not be stored.
+//
+//     EntryBlk[VAPseudoInstr]          EntryBlk
+//        |                              |     .
+//        |                              |        .
+//        |                              |   GuardedRegsBlk
+//        |                      =>      |        .
+//        |                              |     .
+//        |                             TailBlk[VAPseudoInstr]
+//        |                              |
+//        |                              |
+//
+static std::pair<MachineBasicBlock *, MachineBasicBlock *>
+createGuardedRegsBlock(MachineBasicBlock *EntryBlk, MachineInstr &VAPseudoInstr,
+                       const X86Subtarget &Subtarget) {
+
+  MachineFunction *Func = EntryBlk->getParent();
+  const TargetInstrInfo *TII = Subtarget.getInstrInfo();
+  DebugLoc DL = VAPseudoInstr.getDebugLoc();
+  Register CountReg = VAPseudoInstr.getOperand(0).getReg();
 
   // Create the new basic blocks. One block contains all the XMM stores,
   // and one block is the final destination regardless of whether any
   // stores were performed.
-  const BasicBlock *LLVM_BB = MBB->getBasicBlock();
-  MachineFunction *F = MBB->getParent();
-  MachineFunction::iterator MBBIter = ++MBB->getIterator();
-  MachineBasicBlock *XMMSaveMBB = F->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *EndMBB = F->CreateMachineBasicBlock(LLVM_BB);
-  F->insert(MBBIter, XMMSaveMBB);
-  F->insert(MBBIter, EndMBB);
+  const BasicBlock *LLVMBlk = EntryBlk->getBasicBlock();
+  MachineFunction::iterator EntryBlkIter = ++EntryBlk->getIterator();
+  MachineBasicBlock *GuardedRegsBlk = Func->CreateMachineBasicBlock(LLVMBlk);
+  MachineBasicBlock *TailBlk = Func->CreateMachineBasicBlock(LLVMBlk);
+  Func->insert(EntryBlkIter, GuardedRegsBlk);
+  Func->insert(EntryBlkIter, TailBlk);
 
   // Transfer the remainder of MBB and its successor edges to EndMBB.
-  EndMBB->splice(EndMBB->begin(), MBB,
-                 std::next(MachineBasicBlock::iterator(MI)), MBB->end());
-  EndMBB->transferSuccessorsAndUpdatePHIs(MBB);
+  TailBlk->splice(TailBlk->begin(), EntryBlk,
+                  std::next(MachineBasicBlock::iterator(VAPseudoInstr)),
+                  EntryBlk->end());
+  TailBlk->transferSuccessorsAndUpdatePHIs(EntryBlk);
 
   // The original block will now fall through to the XMM save block.
-  MBB->addSuccessor(XMMSaveMBB);
+  EntryBlk->addSuccessor(GuardedRegsBlk);
   // The XMMSaveMBB will fall through to the end block.
-  XMMSaveMBB->addSuccessor(EndMBB);
-
-  // Now add the instructions.
-  const TargetInstrInfo *TII = Subtarget.getInstrInfo();
-  DebugLoc DL = MI.getDebugLoc();
-
-  Register CountReg = MI.getOperand(0).getReg();
-  int64_t RegSaveFrameIndex = MI.getOperand(1).getImm();
-  int64_t VarArgsFPOffset = MI.getOperand(2).getImm();
+  GuardedRegsBlk->addSuccessor(TailBlk);
 
-  if (!Subtarget.isCallingConvWin64(F->getFunction().getCallingConv())) {
+  if (!Subtarget.isCallingConvWin64(Func->getFunction().getCallingConv())) {
     // If %al is 0, branch around the XMM save block.
-    BuildMI(MBB, DL, TII->get(X86::TEST8rr)).addReg(CountReg).addReg(CountReg);
-    BuildMI(MBB, DL, TII->get(X86::JCC_1)).addMBB(EndMBB).addImm(X86::COND_E);
-    MBB->addSuccessor(EndMBB);
+    BuildMI(EntryBlk, DL, TII->get(X86::TEST8rr))
+        .addReg(CountReg)
+        .addReg(CountReg);
+    BuildMI(EntryBlk, DL, TII->get(X86::JCC_1))
+        .addMBB(TailBlk)
+        .addImm(X86::COND_E);
+    EntryBlk->addSuccessor(TailBlk);
   }
 
+  return std::make_pair(GuardedRegsBlk, TailBlk);
+}
+
+void X86TargetLowering::addSaveVarargXmmRegsPseudo(
+    MachineBasicBlock *GuardedRegsBlk, MachineBasicBlock *TailBlk,
+    MachineInstr &SrcPseudoInstr) const {
   // Make sure the last operand is EFLAGS, which gets clobbered by the branch
   // that was just emitted, but clearly shouldn't be "saved".
-  assert((MI.getNumOperands() <= 3 ||
-          !MI.getOperand(MI.getNumOperands() - 1).isReg() ||
-          MI.getOperand(MI.getNumOperands() - 1).getReg() == X86::EFLAGS) &&
+  assert((SrcPseudoInstr.getNumOperands() <= 3 ||
+          !SrcPseudoInstr.getOperand(SrcPseudoInstr.getNumOperands() - 1)
+               .isReg() ||
+          SrcPseudoInstr.getOperand(SrcPseudoInstr.getNumOperands() - 1)
+                  .getReg() == X86::EFLAGS) &&
          "Expected last argument to be EFLAGS");
-  unsigned MOVOpc = Subtarget.hasAVX() ? X86::VMOVAPSmr : X86::MOVAPSmr;
-  // In the XMM save block, save all the XMM argument registers.
-  for (int i = 3, e = MI.getNumOperands() - 1; i != e; ++i) {
-    int64_t Offset = (i - 3) * 16 + VarArgsFPOffset;
-    MachineMemOperand *MMO = F->getMachineMemOperand(
-        MachinePointerInfo::getFixedStack(*F, RegSaveFrameIndex, Offset),
-        MachineMemOperand::MOStore,
-        /*Size=*/16, /*Align=*/16);
-    BuildMI(XMMSaveMBB, DL, TII->get(MOVOpc))
-        .addFrameIndex(RegSaveFrameIndex)
-        .addImm(/*Scale=*/1)
-        .addReg(/*IndexReg=*/0)
-        .addImm(/*Disp=*/Offset)
-        .addReg(/*Segment=*/0)
-        .addReg(MI.getOperand(i).getReg())
-        .addMemOperand(MMO);
-  }
+
+  // create SAVE_VARARG_XMM_REGS pseudo
+  MachineInstrBuilder MIB =
+      BuildMI(GuardedRegsBlk, SrcPseudoInstr.getDebugLoc(),
+              Subtarget.getInstrInfo()->get(X86::SAVE_VARARG_XMM_REGS));
+
+  // set Frame Index
+  MIB.addImm(SrcPseudoInstr.getOperand(1).getImm());
+
+  // set ArgsOffset
+  MIB.addImm(SrcPseudoInstr.getOperand(2).getImm());
+
+  for (unsigned OpndIdx = 3, RegIdx = 0;
+       OpndIdx + 1 < SrcPseudoInstr.getNumOperands(); OpndIdx++, RegIdx++)
+    MIB.addReg(SrcPseudoInstr.getOperand(OpndIdx).getReg(),
+               RegState::InternalRead);
+
+  SrcPseudoInstr.eraseFromParent(); // The pseudo instruction is gone now.
 
-  MI.eraseFromParent(); // The pseudo instruction is gone now.
+MachineBasicBlock *X86TargetLowering::emitVAStartSaveXMMRegsWithCustomInserter(
+    MachineInstr &PseudoVaStartInstr, MachineBasicBlock *EntryBlk) const {
+  // Emit code to save XMM registers to the stack. The ABI says that the
+  // number of registers to save is given in %al, so it's theoretically
+  // possible to do an indirect jump trick to avoid saving all of them,
+  // however this code takes a simpler approach and just executes all
+  // of the stores if %al is non-zero. It's less code, and it's probably
+  // easier on the hardware branch predictor, and stores aren't all that
+  // expensive anyway.
+
+  MachineBasicBlock *GuardedRegsBlk = nullptr;
+  MachineBasicBlock *TailBlk = nullptr;
 
-  return EndMBB;
+  std::tie(GuardedRegsBlk, TailBlk) =
+      createGuardedRegsBlock(EntryBlk, PseudoVaStartInstr, Subtarget);
+
+  addSaveVarargXmmRegsPseudo(GuardedRegsBlk, TailBlk, PseudoVaStartInstr);
+
+  return TailBlk;
+}
+
+MachineBasicBlock *
+X86TargetLowering::emitVarargThunkSaveXMMRegsWithCustomInserter(
+    MachineInstr &PseudoVarargThunkInstr, MachineBasicBlock *EntryBlk) const {
+  MachineBasicBlock *GuardedRegsBlk = nullptr;
+  MachineBasicBlock *TailBlk = nullptr;
+  MachineFunction *Func = EntryBlk->getParent();
+
+  // check whether GuardedRegsBlk is already created by VASTART handling code
+  assert(Func->begin() != Func->end());
+  for (auto &Succ : (*Func->begin()).successors()) {
+
+    for (auto &Instr : Succ->instrs()) {
+      if (Instr.getOpcode() == X86::SAVE_VARARG_XMM_REGS) {
+        // GuardedRegsBlk is already created by VASTART handling code
+        assert(Func->getFrameInfo().hasVAStart());
+        GuardedRegsBlk = Succ;
+        TailBlk = *GuardedRegsBlk->succ_begin();
+        break;
+      }
+    }
+
+    if (GuardedRegsBlk)
+      break;
+  }
+
+  if (GuardedRegsBlk == nullptr)
+    std::tie(GuardedRegsBlk, TailBlk) =
+        createGuardedRegsBlock(EntryBlk, PseudoVarargThunkInstr, Subtarget);
+
+  addSaveVarargXmmRegsPseudo(GuardedRegsBlk, TailBlk, PseudoVarargThunkInstr);
+
+  return TailBlk;
 }
 
 // The EFLAGS operand of SelectItr might be missing a kill marker
 // because there were multiple uses of EFLAGS, and ISel didn't know
 // which to mark. Figure out whether SelectItr should have had a
 // kill marker, and set it if it should. Returns the correct kill
 // marker value.
 static bool checkAndUpdateEFLAGSKill(MachineBasicBlock::iterator SelectItr,
case X86::FP80_TO_INT64_IN_MEM: {
     return BB;
   }
 
   // xbegin
   case X86::XBEGIN:
     return emitXBegin(MI, BB, Subtarget.getInstrInfo());
 
   case X86::VASTART_SAVE_XMM_REGS:
-    return EmitVAStartSaveXMMRegsWithCustomInserter(MI, BB);
+    return emitVAStartSaveXMMRegsWithCustomInserter(MI, BB);
+
+  case X86::MUSTTAIL_SAVE_GUARDED_REGS:
+    return emitVarargThunkSaveXMMRegsWithCustomInserter(MI, BB);
 
   case X86::VAARG_64:
-    return EmitVAARG64WithCustomInserter(MI, BB);
+    return emitVAARG64WithCustomInserter(MI, BB);
 
   case X86::EH_SjLj_SetJmp32:
   case X86::EH_SjLj_SetJmp64:
     return emitEHSjLjSetJmp(MI, BB);
 
   case X86::EH_SjLj_LongJmp32:
   case X86::EH_SjLj_LongJmp64:
     return emitEHSjLjLongJmp(MI, BB);

llvm/lib/Target/X86/X86InstrCompiler.td

def ADJCALLSTACKUP64   : I<0, Pseudo, (outs), (ins i32imm:$amt1, i32imm:$amt2),
                            [(X86callseq_end timm:$amt1, timm:$amt2)]>,
                            Requires<[IsLP64]>;
 }
 def : Pat<(X86callseq_start timm:$amt1, timm:$amt2),
         (ADJCALLSTACKDOWN64 i32imm:$amt1, i32imm:$amt2, 0)>, Requires<[IsLP64]>;
 
 let SchedRW = [WriteSystem] in {
 
+let hasSideEffects = 1 in {
+def SAVE_VARARG_XMM_REGS : I<0, Pseudo,
+                           (outs),
+                           (ins i64imm:$regsavefi, i64imm:$offset,
+                             variable_ops),
+                          "#SAVE_VARARG_XMM_REGS $regsavefi, $offset",
+                          []>;
+}
+
 // x86-64 va_start lowering magic.
 let usesCustomInserter = 1, Defs = [EFLAGS] in {
 def VASTART_SAVE_XMM_REGS : I<0, Pseudo,
                               (outs),
                               (ins GR8:$al,
                                    i64imm:$regsavefi, i64imm:$offset,
                                    variable_ops),
                               "#VASTART_SAVE_XMM_REGS $al, $regsavefi, $offset",
                               [(X86vastart_save_xmm_regs GR8:$al,
                                                          imm:$regsavefi,
                                                          imm:$offset),
                                (implicit EFLAGS)]>;
 
+// x86-64 %al guarded thunk arguments lowering magic.
+def MUSTTAIL_SAVE_GUARDED_REGS : I<0, Pseudo,
+                              (outs),
+                              (ins GR8:$al,
+                                   i64imm:$regsavefi, i64imm:$offset,
+                                   variable_ops),
+                              "#MUSTTAIL_SAVE_GUARDED_REGS $al, $regsavefi, $offset",
+                              [(X86musttail_save_guarded_regs GR8:$al,
+                                                         imm:$regsavefi,
+                                                         imm:$offset),
+                               (implicit EFLAGS)]>;
+
+
 // The VAARG_64 pseudo-instruction takes the address of the va_list,
 // and places the address of the next argument into a register.
 let Defs = [EFLAGS] in
 def VAARG_64 : I<0, Pseudo,
                  (outs GR64:$dst),
                  (ins i8mem:$ap, i32imm:$size, i8imm:$mode, i32imm:$align),
                  "#VAARG_64 $dst, $ap, $size, $mode, $align",
                  [(set GR64:$dst,

llvm/lib/Target/X86/X86InstrInfo.td

 def SDT_X86Call   : SDTypeProfile<0, -1, [SDTCisVT<0, iPTR>]>;
 
 def SDT_X86NtBrind : SDTypeProfile<0, -1, [SDTCisVT<0, iPTR>]>;
 
 def SDT_X86VASTART_SAVE_XMM_REGS : SDTypeProfile<0, -1, [SDTCisVT<0, i8>,
                                                          SDTCisVT<1, iPTR>,
                                                          SDTCisVT<2, iPTR>]>;
 
+def SDT_X86MUSTTAIL_SAVE_GUARDED_REGS : SDTypeProfile<0, -1, [SDTCisVT<0, i8>,
+                                                         SDTCisVT<1, iPTR>,
+                                                         SDTCisVT<2, iPTR>]>;
+
+
 def SDT_X86VAARG_64 : SDTypeProfile<1, -1, [SDTCisPtrTy<0>,
                                             SDTCisPtrTy<1>,
                                             SDTCisVT<2, i32>,
                                             SDTCisVT<3, i8>,
                                             SDTCisVT<4, i32>]>;
 
 def SDTX86RepStr  : SDTypeProfile<0, 1, [SDTCisVT<0, OtherVT>]>;
 
def X86retflag : SDNode<"X86ISD::RET_FLAG", SDTX86Ret,
                         [SDNPHasChain, SDNPOptInGlue, SDNPVariadic]>;
 def X86iret : SDNode<"X86ISD::IRET", SDTX86Ret,
                         [SDNPHasChain, SDNPOptInGlue]>;
 
 def X86vastart_save_xmm_regs :
                  SDNode<"X86ISD::VASTART_SAVE_XMM_REGS",
                         SDT_X86VASTART_SAVE_XMM_REGS,
                         [SDNPHasChain, SDNPVariadic]>;
+
+def X86musttail_save_guarded_regs :
+                 SDNode<"X86ISD::MUSTTAIL_SAVE_GUARDED_REGS",
+                        SDT_X86MUSTTAIL_SAVE_GUARDED_REGS,
+                        [SDNPHasChain, SDNPVariadic]>;
+
 def X86vaarg64 :
                  SDNode<"X86ISD::VAARG_64", SDT_X86VAARG_64,
                         [SDNPHasChain, SDNPMayLoad, SDNPMayStore,
                          SDNPMemOperand]>;
 def X86callseq_start :
                  SDNode<"ISD::CALLSEQ_START", SDT_X86CallSeqStart,
                         [SDNPHasChain, SDNPOutGlue]>;
 def X86callseq_end :

llvm/lib/Target/X86/X86MachineFunctionInfo.h

class X86MachineFunctionInfo : public MachineFunctionInfo {
   /// use as the global base register. This is used for PIC in some PIC
   /// relocation models.
   unsigned GlobalBaseReg = 0;
 
   /// VarArgsFrameIndex - FrameIndex for start of varargs area.
   int VarArgsFrameIndex = 0;
   /// RegSaveFrameIndex - X86-64 vararg func register save area.
   int RegSaveFrameIndex = 0;
+  /// ThunkRegSaveFrameIndex - X86-64 vararg func register save area for thunk
+  /// functions.
+  int ThunkRegSaveFrameIndex = 0;
   /// VarArgsGPOffset - X86-64 vararg func int reg offset.
   unsigned VarArgsGPOffset = 0;
   /// VarArgsFPOffset - X86-64 vararg func fp reg offset.
   unsigned VarArgsFPOffset = 0;
   /// ArgumentStackSize - The number of bytes on stack consumed by the arguments
   /// being passed on the stack.
   unsigned ArgumentStackSize = 0;
   /// NumLocalDynamics - Number of local-dynamic TLS accesses.
public:
   void setGlobalBaseReg(unsigned Reg) { GlobalBaseReg = Reg; }
 
   int getVarArgsFrameIndex() const { return VarArgsFrameIndex; }
   void setVarArgsFrameIndex(int Idx) { VarArgsFrameIndex = Idx; }
 
   int getRegSaveFrameIndex() const { return RegSaveFrameIndex; }
   void setRegSaveFrameIndex(int Idx) { RegSaveFrameIndex = Idx; }
 
+  int getThunkRegSaveFrameIndex() const { return ThunkRegSaveFrameIndex; }
+  void setThunkRegSaveFrameIndex(int Idx) { ThunkRegSaveFrameIndex = Idx; }
+
   unsigned getVarArgsGPOffset() const { return VarArgsGPOffset; }
   void setVarArgsGPOffset(unsigned Offset) { VarArgsGPOffset = Offset; }
 
   unsigned getVarArgsFPOffset() const { return VarArgsFPOffset; }
   void setVarArgsFPOffset(unsigned Offset) { VarArgsFPOffset = Offset; }
 
   unsigned getArgumentStackSize() const { return ArgumentStackSize; }
   void setArgumentStackSize(unsigned size) { ArgumentStackSize = size; }

llvm/test/CodeGen/X86/icall-branch-funnel.ll

 declare void @f9()
 
 declare void @llvm.icall.branch.funnel(...)
 
 define void @jt2(i8* nest, ...) {
   ; CHECK: jt2:
   ; CHECK:      leaq g+1(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB0_1
+  ; CHECK-NEXT: jae .LBB0_3
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f0
-  ; CHECK-NEXT: .LBB0_1:
+  ; CHECK-NEXT: .LBB0_3:
   ; CHECK-NEXT: jmp f1
   musttail call void (...) @llvm.icall.branch.funnel(
       i8* %0,
       i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,
       i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,
       ...
   )
   ret void
 }
 
 define void @jt3(i8* nest, ...) {
   ; CHECK: jt3:
   ; CHECK:      leaq g+1(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB1_1
+  ; CHECK-NEXT: jae .LBB1_3
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f0
-  ; CHECK-NEXT: .LBB1_1:
-  ; CHECK-NEXT: jne .LBB1_2
+  ; CHECK-NEXT: .LBB1_3:
+  ; CHECK-NEXT: jne .LBB1_4
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f1
-  ; CHECK-NEXT: .LBB1_2:
+  ; CHECK-NEXT: .LBB1_4:
   ; CHECK-NEXT: jmp f2
   musttail call void (...) @llvm.icall.branch.funnel(
       i8* %0,
       i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,
       i8* getelementptr (i8, i8* @g, i64 2), void ()* @f2,
       i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,
       ...
   )
   ret void
 }
 
 define void @jt7(i8* nest, ...) {
   ; CHECK: jt7:
   ; CHECK:      leaq g+3(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB2_1
+  ; CHECK-NEXT: jae .LBB2_3
   ; CHECK-NEXT: #
   ; CHECK-NEXT: leaq g+1(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB2_6
+  ; CHECK-NEXT: jae .LBB2_8
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f0
-  ; CHECK-NEXT: .LBB2_1:
-  ; CHECK-NEXT: jne .LBB2_2
+  ; CHECK-NEXT: .LBB2_3:
+  ; CHECK-NEXT: jne .LBB2_4
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f3
-  ; CHECK-NEXT: .LBB2_6:
-  ; CHECK-NEXT: jne .LBB2_7
+  ; CHECK-NEXT: .LBB2_8:
+  ; CHECK-NEXT: jne .LBB2_9
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f1
-  ; CHECK-NEXT: .LBB2_2:
+  ; CHECK-NEXT: .LBB2_4:
   ; CHECK-NEXT: leaq g+5(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB2_3
+  ; CHECK-NEXT: jae .LBB2_5
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f4
-  ; CHECK-NEXT: .LBB2_7:
+  ; CHECK-NEXT: .LBB2_9:
   ; CHECK-NEXT: jmp f2
-  ; CHECK-NEXT: .LBB2_3:
-  ; CHECK-NEXT: jne .LBB2_4
+  ; CHECK-NEXT: .LBB2_5:
+  ; CHECK-NEXT: jne .LBB2_6
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f5
-  ; CHECK-NEXT: .LBB2_4:
+  ; CHECK-NEXT: .LBB2_6:
   ; CHECK-NEXT: jmp f6
   musttail call void (...) @llvm.icall.branch.funnel(
       i8* %0,
       i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,
       i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,
       i8* getelementptr (i8, i8* @g, i64 2), void ()* @f2,
       i8* getelementptr (i8, i8* @g, i64 3), void ()* @f3,
       i8* getelementptr (i8, i8* @g, i64 4), void ()* @f4,
       i8* getelementptr (i8, i8* @g, i64 5), void ()* @f5,
       i8* getelementptr (i8, i8* @g, i64 6), void ()* @f6,
       ...
   )
   ret void
 }
 
 define void @jt10(i8* nest, ...) {
   ; CHECK: jt10:
   ; CHECK:      leaq g+5(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB3_1
+  ; CHECK-NEXT: jae .LBB3_3
   ; CHECK-NEXT: #
   ; CHECK-NEXT: leaq g+1(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB3_7
+  ; CHECK-NEXT: jae .LBB3_9
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f0
-  ; CHECK-NEXT: .LBB3_1:
-  ; CHECK-NEXT: jne .LBB3_2
+  ; CHECK-NEXT: .LBB3_3:
+  ; CHECK-NEXT: jne .LBB3_4
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f5
-  ; CHECK-NEXT: .LBB3_7:
-  ; CHECK-NEXT: jne .LBB3_8
+  ; CHECK-NEXT: .LBB3_9:
+  ; CHECK-NEXT: jne .LBB3_10
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f1
-  ; CHECK-NEXT: .LBB3_2:
+  ; CHECK-NEXT: .LBB3_4:
   ; CHECK-NEXT: leaq g+7(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB3_3
+  ; CHECK-NEXT: jae .LBB3_5
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f6
-  ; CHECK-NEXT: .LBB3_8:
+  ; CHECK-NEXT: .LBB3_10:
   ; CHECK-NEXT: leaq g+3(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB3_9
+  ; CHECK-NEXT: jae .LBB3_11
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f2
-  ; CHECK-NEXT: .LBB3_3:
-  ; CHECK-NEXT: jne .LBB3_4
+  ; CHECK-NEXT: .LBB3_5:
+  ; CHECK-NEXT: jne .LBB3_6
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f7
-  ; CHECK-NEXT: .LBB3_9:
-  ; CHECK-NEXT: jne .LBB3_10
+  ; CHECK-NEXT: .LBB3_11:
+  ; CHECK-NEXT: jne .LBB3_12
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f3
-  ; CHECK-NEXT: .LBB3_4:
+  ; CHECK-NEXT: .LBB3_6:
   ; CHECK-NEXT: leaq g+9(%rip), %r11
   ; CHECK-NEXT: cmpq %r11, %r10
-  ; CHECK-NEXT: jae .LBB3_5
+  ; CHECK-NEXT: jae .LBB3_7
   ; CHECK-NEXT: #
   ; CHECK-NEXT: jmp f8
-  ; CHECK-NEXT: .LBB3_10:
+  ; CHECK-NEXT: .LBB3_12:
   ; CHECK-NEXT: jmp f4
-  ; CHECK-NEXT: .LBB3_5:
+  ; CHECK-NEXT: .LBB3_7:
   ; CHECK-NEXT: jmp f9
   musttail call void (...) @llvm.icall.branch.funnel(
       i8* %0,
       i8* getelementptr (i8, i8* @g, i64 0), void ()* @f0,
       i8* getelementptr (i8, i8* @g, i64 1), void ()* @f1,
       i8* getelementptr (i8, i8* @g, i64 2), void ()* @f2,
       i8* getelementptr (i8, i8* @g, i64 3), void ()* @f3,
       i8* getelementptr (i8, i8* @g, i64 4), void ()* @f4,

llvm/test/CodeGen/X86/musttail-varargs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs < %s -enable-tail-merge=0 -mtriple=x86_64-linux | FileCheck %s --check-prefix=LINUX
+; RUN: llc -verify-machineinstrs -O0 < %s -enable-tail-merge=0 -mtriple=x86_64-linux | FileCheck %s --check-prefix=LINUX-OPT0
 ; RUN: llc -verify-machineinstrs < %s -enable-tail-merge=0 -mtriple=x86_64-linux-gnux32 | FileCheck %s --check-prefix=LINUX-X32
+; RUN: llc -verify-machineinstrs -O0 < %s -enable-tail-merge=0 -mtriple=x86_64-linux-gnux32 | FileCheck %s --check-prefix=LINUX-X32-OPT0
 ; RUN: llc -verify-machineinstrs < %s -enable-tail-merge=0 -mtriple=x86_64-windows | FileCheck %s --check-prefix=WINDOWS
+; RUN: llc -verify-machineinstrs -O0 < %s -enable-tail-merge=0 -mtriple=x86_64-windows | FileCheck %s --check-prefix=WINDOWS-OPT0
 ; RUN: llc -verify-machineinstrs < %s -enable-tail-merge=0 -mtriple=i686-windows | FileCheck %s --check-prefix=X86 --check-prefix=X86-NOSSE
+; RUN: llc -verify-machineinstrs -O0 < %s -enable-tail-merge=0 -mtriple=i686-windows | FileCheck %s --check-prefix=X86 --check-prefix=X86-NOSSE-OPT0
 ; RUN: llc -verify-machineinstrs < %s -enable-tail-merge=0 -mtriple=i686-windows -mattr=+sse2 | FileCheck %s --check-prefix=X86 --check-prefix=X86-SSE
+; RUN: llc -verify-machineinstrs -O0 < %s -enable-tail-merge=0 -mtriple=i686-windows -mattr=+sse2 | FileCheck %s --check-prefix=X86 --check-prefix=X86-SSE-OPT0
 
 ; Test that we actually spill and reload all arguments in the variadic argument
 ; pack. Doing a normal call will clobber all argument registers, and we will
 ; spill around it. A simple adjustment should not require any XMM spills.
 
 declare void @llvm.va_start(i8*) nounwind
 
 declare void(i8*, ...)* @get_f(i8* %this)

 ; LINUX-NEXT:    pushq %r14
 ; LINUX-NEXT:    .cfi_def_cfa_offset 32
 ; LINUX-NEXT:    pushq %r13
 ; LINUX-NEXT:    .cfi_def_cfa_offset 40
 ; LINUX-NEXT:    pushq %r12
 ; LINUX-NEXT:    .cfi_def_cfa_offset 48
 ; LINUX-NEXT:    pushq %rbx
 ; LINUX-NEXT:    .cfi_def_cfa_offset 56
-; LINUX-NEXT:    subq $360, %rsp # imm = 0x168
-; LINUX-NEXT:    .cfi_def_cfa_offset 416
+; LINUX-NEXT:    subq $232, %rsp
+; LINUX-NEXT:    .cfi_def_cfa_offset 288
 ; LINUX-NEXT:    .cfi_offset %rbx, -56
 ; LINUX-NEXT:    .cfi_offset %r12, -48
 ; LINUX-NEXT:    .cfi_offset %r13, -40
 ; LINUX-NEXT:    .cfi_offset %r14, -32
 ; LINUX-NEXT:    .cfi_offset %r15, -24
 ; LINUX-NEXT:    .cfi_offset %rbp, -16
 ; LINUX-NEXT:    movq %r9, %r15
 ; LINUX-NEXT:    movq %r8, %r12
 ; LINUX-NEXT:    movq %rcx, %r13
 ; LINUX-NEXT:    movq %rdx, %rbp
 ; LINUX-NEXT:    movq %rsi, %rbx
 ; LINUX-NEXT:    movq %rdi, %r14
+; LINUX-NEXT:    movq %rsi, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %r8, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %r9, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movb %al, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
 ; LINUX-NEXT:    testb %al, %al
 ; LINUX-NEXT:    je .LBB0_2
 ; LINUX-NEXT:  # %bb.1:
 ; LINUX-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm1, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm2, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm3, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm4, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm5, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:  .LBB0_2:
-; LINUX-NEXT:    movq %rbx, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %rbp, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %r13, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %r12, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %r15, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
 ; LINUX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
 ; LINUX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movabsq $206158430216, %rax # imm = 0x3000000008
 ; LINUX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movq %r14, %rdi
-; LINUX-NEXT:    movaps %xmm7, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm6, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm5, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm4, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; LINUX-NEXT:    callq get_f
 ; LINUX-NEXT:    movq %rax, %r11
 ; LINUX-NEXT:    movq %r14, %rdi
 ; LINUX-NEXT:    movq %rbx, %rsi
 ; LINUX-NEXT:    movq %rbp, %rdx
 ; LINUX-NEXT:    movq %r13, %rcx
 ; LINUX-NEXT:    movq %r12, %r8
 ; LINUX-NEXT:    movq %r15, %r9
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm0 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm1 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm2 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm3 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm4 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm5 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm6 # 16-byte Reload
-; LINUX-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm7 # 16-byte Reload
 ; LINUX-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
-; LINUX-NEXT:    addq $360, %rsp # imm = 0x168
+; LINUX-NEXT:    testb %al, %al
+; LINUX-NEXT:    je .LBB0_4
+; LINUX-NEXT:  # %bb.3:
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm7
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm6
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm5
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm4
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm3
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm2
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm1
+; LINUX-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm0
+; LINUX-NEXT:    addq $232, %rsp
+; LINUX-NEXT:    .cfi_def_cfa_offset 56
+; LINUX-NEXT:    popq %rbx
+; LINUX-NEXT:    .cfi_def_cfa_offset 48
+; LINUX-NEXT:    popq %r12
+; LINUX-NEXT:    .cfi_def_cfa_offset 40
+; LINUX-NEXT:    popq %r13
+; LINUX-NEXT:    .cfi_def_cfa_offset 32
+; LINUX-NEXT:    popq %r14
+; LINUX-NEXT:    .cfi_def_cfa_offset 24
+; LINUX-NEXT:    popq %r15
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-NEXT:    popq %rbp
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-NEXT:  .LBB0_4:
+; LINUX-NEXT:    .cfi_def_cfa_offset 288
+; LINUX-NEXT:    addq $232, %rsp
 ; LINUX-NEXT:    .cfi_def_cfa_offset 56
 ; LINUX-NEXT:    popq %rbx
 ; LINUX-NEXT:    .cfi_def_cfa_offset 48
 ; LINUX-NEXT:    popq %r12
 ; LINUX-NEXT:    .cfi_def_cfa_offset 40
 ; LINUX-NEXT:    popq %r13
 ; LINUX-NEXT:    .cfi_def_cfa_offset 32
 ; LINUX-NEXT:    popq %r14
 ; LINUX-NEXT:    .cfi_def_cfa_offset 24
 ; LINUX-NEXT:    popq %r15
 ; LINUX-NEXT:    .cfi_def_cfa_offset 16
 ; LINUX-NEXT:    popq %rbp
 ; LINUX-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-NEXT:    jmpq *%r11 # TAILCALL
 ;
+; LINUX-OPT0-LABEL: f_thunk:
+; LINUX-OPT0:       # %bb.0:
+; LINUX-OPT0-NEXT:    subq $328, %rsp # imm = 0x148
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 336
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    movb %al, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; LINUX-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    je .LBB0_2
+; LINUX-OPT0-NEXT:  # %bb.1:
+; LINUX-OPT0-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm1, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm2, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm3, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm4, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm5, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:  .LBB0_2:
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rsi, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rdi, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %r8b # 1-byte Reload
+; LINUX-OPT0-NEXT:    leaq {{[0-9]+}}(%rsp), %r9
+; LINUX-OPT0-NEXT:    movq %r9, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    leaq {{[0-9]+}}(%rsp), %r9
+; LINUX-OPT0-NEXT:    movq %r9, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movl $48, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movl $8, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r9, %rdi
+; LINUX-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rax, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movb %r8b, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; LINUX-OPT0-NEXT:    callq get_f
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %r10b # 1-byte Reload
+; LINUX-OPT0-NEXT:    movq %rax, (%rsp) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movb %r10b, %al
+; LINUX-OPT0-NEXT:    movq (%rsp), %r11 # 8-byte Reload
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    je .LBB0_4
+; LINUX-OPT0-NEXT:  # %bb.3:
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm7
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm6
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm5
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm4
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm3
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm2
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm1
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm0
+; LINUX-OPT0-NEXT:    addq $328, %rsp # imm = 0x148
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-OPT0-NEXT:  .LBB0_4:
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 336
+; LINUX-OPT0-NEXT:    addq $328, %rsp # imm = 0x148
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+;
 ; LINUX-X32-LABEL: f_thunk:
 ; LINUX-X32:       # %bb.0:
 ; LINUX-X32-NEXT:    pushq %rbp
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
 ; LINUX-X32-NEXT:    pushq %r15
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 24
 ; LINUX-X32-NEXT:    pushq %r14
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 32
 ; LINUX-X32-NEXT:    pushq %r13
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 40
 ; LINUX-X32-NEXT:    pushq %r12
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 48
 ; LINUX-X32-NEXT:    pushq %rbx
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 56
-; LINUX-X32-NEXT:    subl $344, %esp # imm = 0x158
-; LINUX-X32-NEXT:    .cfi_def_cfa_offset 400
+; LINUX-X32-NEXT:    subl $216, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 272
 ; LINUX-X32-NEXT:    .cfi_offset %rbx, -56
 ; LINUX-X32-NEXT:    .cfi_offset %r12, -48
 ; LINUX-X32-NEXT:    .cfi_offset %r13, -40
 ; LINUX-X32-NEXT:    .cfi_offset %r14, -32
 ; LINUX-X32-NEXT:    .cfi_offset %r15, -24
 ; LINUX-X32-NEXT:    .cfi_offset %rbp, -16
 ; LINUX-X32-NEXT:    movq %r9, %r15
 ; LINUX-X32-NEXT:    movq %r8, %r12
 ; LINUX-X32-NEXT:    movq %rcx, %r13
 ; LINUX-X32-NEXT:    movq %rdx, %rbp
 ; LINUX-X32-NEXT:    movq %rsi, %rbx
 ; LINUX-X32-NEXT:    movl %edi, %r14d
+; LINUX-X32-NEXT:    movq %rsi, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %rdx, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %rcx, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %r8, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %r9, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
 ; LINUX-X32-NEXT:    testb %al, %al
 ; LINUX-X32-NEXT:    je .LBB0_2
 ; LINUX-X32-NEXT:  # %bb.1:
 ; LINUX-X32-NEXT:    movaps %xmm0, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm1, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm2, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm3, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm4, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm5, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm6, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm7, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:  .LBB0_2:
-; LINUX-X32-NEXT:    movq %rbx, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %rbp, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %r13, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %r12, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %r15, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    leal {{[0-9]+}}(%rsp), %eax
 ; LINUX-X32-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    leal {{[0-9]+}}(%rsp), %eax
 ; LINUX-X32-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movabsq $206158430216, %rax # imm = 0x3000000008
 ; LINUX-X32-NEXT:    movq %rax, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movl %r14d, %edi
-; LINUX-X32-NEXT:    movaps %xmm7, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm6, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm5, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm4, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm3, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm2, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; LINUX-X32-NEXT:    callq get_f
 ; LINUX-X32-NEXT:    movl %eax, %r11d
 ; LINUX-X32-NEXT:    movl %r14d, %edi
 ; LINUX-X32-NEXT:    movq %rbx, %rsi
 ; LINUX-X32-NEXT:    movq %rbp, %rdx
 ; LINUX-X32-NEXT:    movq %r13, %rcx
 ; LINUX-X32-NEXT:    movq %r12, %r8
 ; LINUX-X32-NEXT:    movq %r15, %r9
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm2 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm3 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm4 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm5 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm6 # 16-byte Reload
-; LINUX-X32-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm7 # 16-byte Reload
 ; LINUX-X32-NEXT:    movb {{[-0-9]+}}(%e{{[sb]}}p), %al # 1-byte Reload
-; LINUX-X32-NEXT:    addl $344, %esp # imm = 0x158
+; LINUX-X32-NEXT:    testb %al, %al
+; LINUX-X32-NEXT:    je .LBB0_4
+; LINUX-X32-NEXT:  # %bb.3:
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm3
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-NEXT:    movaps {{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-NEXT:    addl $216, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 56
+; LINUX-X32-NEXT:    popq %rbx
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 48
+; LINUX-X32-NEXT:    popq %r12
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 40
+; LINUX-X32-NEXT:    popq %r13
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 32
+; LINUX-X32-NEXT:    popq %r14
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 24
+; LINUX-X32-NEXT:    popq %r15
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-X32-NEXT:    popq %rbp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-NEXT:  .LBB0_4:
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 272
+; LINUX-X32-NEXT:    addl $216, %esp
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 56
 ; LINUX-X32-NEXT:    popq %rbx
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 48
 ; LINUX-X32-NEXT:    popq %r12
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 40
 ; LINUX-X32-NEXT:    popq %r13
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 32
 ; LINUX-X32-NEXT:    popq %r14
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 24
 ; LINUX-X32-NEXT:    popq %r15
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
 ; LINUX-X32-NEXT:    popq %rbp
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
 ;
+; LINUX-X32-OPT0-LABEL: f_thunk:
+; LINUX-X32-OPT0:       # %bb.0:
+; LINUX-X32-OPT0-NEXT:    subl $312, %esp # imm = 0x138
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 320
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; LINUX-X32-OPT0-NEXT:    je .LBB0_2
+; LINUX-X32-OPT0-NEXT:  # %bb.1:
+; LINUX-X32-OPT0-NEXT:    movaps %xmm0, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm1, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm2, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm3, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm4, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm5, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm6, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm7, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:  .LBB0_2:
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rax # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rax, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rcx, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rsi, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rdi # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rdi, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movb {{[-0-9]+}}(%e{{[sb]}}p), %r8b # 1-byte Reload
+; LINUX-X32-OPT0-NEXT:    leal {{[0-9]+}}(%rsp), %r9d
+; LINUX-X32-OPT0-NEXT:    movl %r9d, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    leal {{[0-9]+}}(%rsp), %r9d
+; LINUX-X32-OPT0-NEXT:    movl %r9d, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movl $48, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movl $8, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %r9d # 4-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rdi, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movl %r9d, %edi
+; LINUX-X32-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rax, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movb %r8b, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; LINUX-X32-OPT0-NEXT:    callq get_f
+; LINUX-X32-OPT0-NEXT:    movl %eax, %eax
+; LINUX-X32-OPT0-NEXT:    movl %eax, %ecx
+; LINUX-X32-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edi # 4-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r10 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rcx, (%esp) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r10, %rcx
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movb {{[-0-9]+}}(%e{{[sb]}}p), %al # 1-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq (%esp), %r11 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    je .LBB0_4
+; LINUX-X32-OPT0-NEXT:  # %bb.3:
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm3
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-OPT0-NEXT:    addl $312, %esp # imm = 0x138
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-OPT0-NEXT:  .LBB0_4:
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 320
+; LINUX-X32-OPT0-NEXT:    addl $312, %esp # imm = 0x138
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+;
 ; WINDOWS-LABEL: f_thunk:
 ; WINDOWS:       # %bb.0:
 ; WINDOWS-NEXT:    pushq %r14
 ; WINDOWS-NEXT:    .seh_pushreg %r14
 ; WINDOWS-NEXT:    pushq %rsi
 ; WINDOWS-NEXT:    .seh_pushreg %rsi
 ; WINDOWS-NEXT:    pushq %rdi
 ; WINDOWS-NEXT:    .seh_pushreg %rdi

 ; WINDOWS-NEXT:    popq %rdi
 ; WINDOWS-NEXT:    popq %rsi
 ; WINDOWS-NEXT:    popq %r14
 ; WINDOWS-NEXT:    rex64 jmpq *%rax # TAILCALL
 ; WINDOWS-NEXT:    .seh_handlerdata
 ; WINDOWS-NEXT:    .text
 ; WINDOWS-NEXT:    .seh_endproc
 ;
+; WINDOWS-OPT0-LABEL: f_thunk:
+; WINDOWS-OPT0:       # %bb.0:
+; WINDOWS-OPT0-NEXT:    subq $104, %rsp
+; WINDOWS-OPT0-NEXT:    .seh_stackalloc 104
+; WINDOWS-OPT0-NEXT:    .seh_endprologue
+; WINDOWS-OPT0-NEXT:    movq %r9, {{[0-9]+}}(%rsp)
+; WINDOWS-OPT0-NEXT:    movq %r8, {{[0-9]+}}(%rsp)
+; WINDOWS-OPT0-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
+; WINDOWS-OPT0-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
+; WINDOWS-OPT0-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
+; WINDOWS-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    callq get_f
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    addq $104, %rsp
+; WINDOWS-OPT0-NEXT:    rex64 jmpq *%rax # TAILCALL
+; WINDOWS-OPT0-NEXT:    .seh_handlerdata
+; WINDOWS-OPT0-NEXT:    .text
+; WINDOWS-OPT0-NEXT:    .seh_endproc
+;
 ; X86-NOSSE-LABEL: f_thunk:
 ; X86-NOSSE:       # %bb.0:
 ; X86-NOSSE-NEXT:    pushl %ebp
 ; X86-NOSSE-NEXT:    movl %esp, %ebp
 ; X86-NOSSE-NEXT:    pushl %esi
 ; X86-NOSSE-NEXT:    andl $-16, %esp
 ; X86-NOSSE-NEXT:    subl $32, %esp
 ; X86-NOSSE-NEXT:    movl 8(%ebp), %esi
 ; X86-NOSSE-NEXT:    leal 12(%ebp), %eax
 ; X86-NOSSE-NEXT:    movl %eax, (%esp)
 ; X86-NOSSE-NEXT:    pushl %esi
 ; X86-NOSSE-NEXT:    calll _get_f
 ; X86-NOSSE-NEXT:    addl $4, %esp
 ; X86-NOSSE-NEXT:    movl %esi, 8(%ebp)
 ; X86-NOSSE-NEXT:    leal -4(%ebp), %esp
 ; X86-NOSSE-NEXT:    popl %esi
 ; X86-NOSSE-NEXT:    popl %ebp
 ; X86-NOSSE-NEXT:    jmpl *%eax # TAILCALL
 ;
+; X86-NOSSE-OPT0-LABEL: f_thunk:
+; X86-NOSSE-OPT0:       # %bb.0:
+; X86-NOSSE-OPT0-NEXT:    pushl %ebp
+; X86-NOSSE-OPT0-NEXT:    movl %esp, %ebp
+; X86-NOSSE-OPT0-NEXT:    andl $-16, %esp
+; X86-NOSSE-OPT0-NEXT:    subl $48, %esp
+; X86-NOSSE-OPT0-NEXT:    movl 8(%ebp), %eax
+; X86-NOSSE-OPT0-NEXT:    leal 12(%ebp), %ecx
+; X86-NOSSE-OPT0-NEXT:    movl %ecx, {{[0-9]+}}(%esp)
+; X86-NOSSE-OPT0-NEXT:    movl %esp, %ecx
+; X86-NOSSE-OPT0-NEXT:    movl %eax, (%ecx)
+; X86-NOSSE-OPT0-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-NOSSE-OPT0-NEXT:    calll _get_f
+; X86-NOSSE-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-NOSSE-OPT0-NEXT:    movl %ecx, 8(%ebp)
+; X86-NOSSE-OPT0-NEXT:    movl %ebp, %esp
+; X86-NOSSE-OPT0-NEXT:    popl %ebp
+; X86-NOSSE-OPT0-NEXT:    jmpl *%eax # TAILCALL
+;
 ; X86-SSE-LABEL: f_thunk:
 ; X86-SSE:       # %bb.0:
 ; X86-SSE-NEXT:    pushl %ebp
 ; X86-SSE-NEXT:    movl %esp, %ebp
 ; X86-SSE-NEXT:    pushl %esi
 ; X86-SSE-NEXT:    andl $-16, %esp
 ; X86-SSE-NEXT:    subl $80, %esp
 ; X86-SSE-NEXT:    movaps %xmm2, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X86-SSE-NEXT:    movaps %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X86-SSE-NEXT:    movaps %xmm0, (%esp) # 16-byte Spill
 ; X86-SSE-NEXT:    movl 8(%ebp), %esi
 ; X86-SSE-NEXT:    leal 12(%ebp), %eax
 ; X86-SSE-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; X86-SSE-NEXT:    pushl %esi
 ; X86-SSE-NEXT:    calll _get_f
 ; X86-SSE-NEXT:    addl $4, %esp
 ; X86-SSE-NEXT:    movl %esi, 8(%ebp)
 ; X86-SSE-NEXT:    movaps (%esp), %xmm0 # 16-byte Reload
 ; X86-SSE-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
 ; X86-SSE-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm2 # 16-byte Reload
 ; X86-SSE-NEXT:    leal -4(%ebp), %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %ebp
 ; X86-SSE-NEXT:    jmpl *%eax # TAILCALL
+;
+; X86-SSE-OPT0-LABEL: f_thunk:
+; X86-SSE-OPT0:       # %bb.0:
+; X86-SSE-OPT0-NEXT:    pushl %ebp
+; X86-SSE-OPT0-NEXT:    movl %esp, %ebp
+; X86-SSE-OPT0-NEXT:    andl $-16, %esp
+; X86-SSE-OPT0-NEXT:    subl $112, %esp
+; X86-SSE-OPT0-NEXT:    movl 8(%ebp), %eax
+; X86-SSE-OPT0-NEXT:    leal 12(%ebp), %ecx
+; X86-SSE-OPT0-NEXT:    movl %ecx, {{[0-9]+}}(%esp)
+; X86-SSE-OPT0-NEXT:    movl %esp, %ecx
+; X86-SSE-OPT0-NEXT:    movl %eax, (%ecx)
+; X86-SSE-OPT0-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X86-SSE-OPT0-NEXT:    movaps %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X86-SSE-OPT0-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-SSE-OPT0-NEXT:    movaps %xmm2, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X86-SSE-OPT0-NEXT:    calll _get_f
+; X86-SSE-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ecx # 4-byte Reload
+; X86-SSE-OPT0-NEXT:    movl %ecx, 8(%ebp)
+; X86-SSE-OPT0-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm2 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movl %ebp, %esp
+; X86-SSE-OPT0-NEXT:    popl %ebp
+; X86-SSE-OPT0-NEXT:    jmpl *%eax # TAILCALL
   %ap = alloca [4 x i8*], align 16
   %ap_i8 = bitcast [4 x i8*]* %ap to i8*
   call void @llvm.va_start(i8* %ap_i8)
 
   %fptr = call void(i8*, ...)*(i8*) @get_f(i8* %this)
   musttail call void (i8*, ...) %fptr(i8* %this, ...)
   ret void
 }
 
 ; Save and restore 6 GPRs, 8 XMMs, and AL around the call.
 
 ; No regparms on normal x86 conventions.
 
-; This thunk shouldn't require any spills and reloads, assuming the register
-; allocator knows what it's doing.
+; This thunk stores xmms on entry and restores them before jumping.
+; Storing and restoring xmms could be optimized out for this concrete case.
 
 define void @g_thunk(i8* %fptr_i8, ...) {
 ; LINUX-LABEL: g_thunk:
 ; LINUX:       # %bb.0:
+; LINUX-NEXT:    pushq %rax
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-NEXT:    testb %al, %al
+; LINUX-NEXT:    je .LBB1_2
+; LINUX-NEXT:  # %bb.1:
+; LINUX-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm3, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm4, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm5, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm6, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm7, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:  .LBB1_2:
+; LINUX-NEXT:    testb %al, %al
+; LINUX-NEXT:    je .LBB1_4
+; LINUX-NEXT:  # %bb.3:
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm7
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm6
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm5
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm4
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm3
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm2
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
+; LINUX-NEXT:    popq %r11
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-NEXT:    jmpq *%rdi # TAILCALL
+; LINUX-NEXT:  .LBB1_4:
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-NEXT:    popq %r11
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-NEXT:    jmpq *%rdi # TAILCALL
 ;
+; LINUX-OPT0-LABEL: g_thunk:
+; LINUX-OPT0:       # %bb.0:
+; LINUX-OPT0-NEXT:    subq $72, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 80
+; LINUX-OPT0-NEXT:    movb %al, %r10b
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movb %r10b, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; LINUX-OPT0-NEXT:    je .LBB1_2
+; LINUX-OPT0-NEXT:  # %bb.1:
+; LINUX-OPT0-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm3, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm4, (%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm5, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:  .LBB1_2:
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r11 # 8-byte Reload
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    je .LBB1_4
+; LINUX-OPT0-NEXT:  # %bb.3:
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm7
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm6
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm5
+; LINUX-OPT0-NEXT:    movaps (%rsp), %xmm4
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm3
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm2
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
+; LINUX-OPT0-NEXT:    addq $72, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-OPT0-NEXT:  .LBB1_4:
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 80
+; LINUX-OPT0-NEXT:    addq $72, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+;
 ; LINUX-X32-LABEL: g_thunk:
 ; LINUX-X32:       # %bb.0:
+; LINUX-X32-NEXT:    pushq %rax
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-X32-NEXT:    testb %al, %al
+; LINUX-X32-NEXT:    je .LBB1_2
+; LINUX-X32-NEXT:  # %bb.1:
+; LINUX-X32-NEXT:    movaps %xmm0, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm1, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm2, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm3, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm4, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm5, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm6, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm7, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:  .LBB1_2:
 ; LINUX-X32-NEXT:    movl %edi, %r11d
+; LINUX-X32-NEXT:    testb %al, %al
+; LINUX-X32-NEXT:    je .LBB1_4
+; LINUX-X32-NEXT:  # %bb.3:
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm3
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-NEXT:    addl $8, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-NEXT:  .LBB1_4:
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-X32-NEXT:    addl $8, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
 ;
+; LINUX-X32-OPT0-LABEL: g_thunk:
+; LINUX-X32-OPT0:       # %bb.0:
+; LINUX-X32-OPT0-NEXT:    subl $72, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 80
+; LINUX-X32-OPT0-NEXT:    movb %al, %r10b
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movb %r10b, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; LINUX-X32-OPT0-NEXT:    je .LBB1_2
+; LINUX-X32-OPT0-NEXT:  # %bb.1:
+; LINUX-X32-OPT0-NEXT:    movaps %xmm0, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm1, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm2, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm3, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm4, (%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm5, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm6, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm7, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:  .LBB1_2:
+; LINUX-X32-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; LINUX-X32-OPT0-NEXT:    movl %eax, %ecx
+; LINUX-X32-OPT0-NEXT:    movl %ecx, %edx
+; LINUX-X32-OPT0-NEXT:    movl %eax, %edi
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r8, %rdx
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movb {{[-0-9]+}}(%e{{[sb]}}p), %al # 1-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r11 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    je .LBB1_4
+; LINUX-X32-OPT0-NEXT:  # %bb.3:
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-OPT0-NEXT:    movaps (%esp), %xmm4
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm3
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-OPT0-NEXT:    addl $72, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-OPT0-NEXT:  .LBB1_4:
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 80
+; LINUX-X32-OPT0-NEXT:    addl $72, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+;
 ; WINDOWS-LABEL: g_thunk:
 ; WINDOWS:       # %bb.0:
 ; WINDOWS-NEXT:    rex64 jmpq *%rcx # TAILCALL
 ;
+; WINDOWS-OPT0-LABEL: g_thunk:
+; WINDOWS-OPT0:       # %bb.0:
+; WINDOWS-OPT0-NEXT:    pushq %rax
+; WINDOWS-OPT0-NEXT:    .seh_stackalloc 8
+; WINDOWS-OPT0-NEXT:    .seh_endprologue
+; WINDOWS-OPT0-NEXT:    movq %rcx, (%rsp) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq (%rsp), %rax # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    popq %r10
+; WINDOWS-OPT0-NEXT:    rex64 jmpq *%rax # TAILCALL
+; WINDOWS-OPT0-NEXT:    .seh_handlerdata
+; WINDOWS-OPT0-NEXT:    .text
+; WINDOWS-OPT0-NEXT:    .seh_endproc
+;
 ; X86-LABEL: g_thunk:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; X86-NEXT:    jmpl *%eax # TAILCALL
   %fptr = bitcast i8* %fptr_i8 to void (i8*, ...)*
   musttail call void (i8*, ...) %fptr(i8* %fptr_i8, ...)
   ret void
 }
 
 ; Do a simple multi-exit multi-bb test.
 
 %struct.Foo = type { i1, i8*, i8* }
 
 @g = external global i32
 
 define void @h_thunk(%struct.Foo* %this, ...) {
 ; LINUX-LABEL: h_thunk:
 ; LINUX:       # %bb.0:
+; LINUX-NEXT:    pushq %rax
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-NEXT:    testb %al, %al
+; LINUX-NEXT:    je .LBB2_2
+; LINUX-NEXT:  # %bb.1:
+; LINUX-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm3, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm4, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm5, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm6, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movaps %xmm7, -{{[0-9]+}}(%rsp)
+; LINUX-NEXT:  .LBB2_2:
 ; LINUX-NEXT:    cmpb $1, (%rdi)
-; LINUX-NEXT:    jne .LBB2_2
-; LINUX-NEXT:  # %bb.1: # %then
+; LINUX-NEXT:    jne .LBB2_4
+; LINUX-NEXT:  # %bb.3: # %then
 ; LINUX-NEXT:    movq 8(%rdi), %r11
+; LINUX-NEXT:    testb %al, %al
+; LINUX-NEXT:    je .LBB2_6
+; LINUX-NEXT:  # %bb.5: # %then
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm7
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm6
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm5
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm4
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm3
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm2
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
+; LINUX-NEXT:    addq $8, %rsp
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-NEXT:    jmpq *%r11 # TAILCALL
-; LINUX-NEXT:  .LBB2_2: # %else
+; LINUX-NEXT:  .LBB2_4: # %else
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
 ; LINUX-NEXT:    movq 16(%rdi), %r11
 ; LINUX-NEXT:    movl $42, {{.*}}(%rip)
+; LINUX-NEXT:    testb %al, %al
+; LINUX-NEXT:    je .LBB2_8
+; LINUX-NEXT:  # %bb.7: # %else
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm7
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm6
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm5
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm4
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm3
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm2
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1
+; LINUX-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
+; LINUX-NEXT:    addq $8, %rsp
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-NEXT:  .LBB2_6: # %then
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-NEXT:    addq $8, %rsp
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-NEXT:  .LBB2_8: # %else
+; LINUX-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-NEXT:    addq $8, %rsp
+; LINUX-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-NEXT:    jmpq *%r11 # TAILCALL
 ;
+; LINUX-OPT0-LABEL: h_thunk:
+; LINUX-OPT0:       # %bb.0:
+; LINUX-OPT0-NEXT:    subq $88, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-OPT0-NEXT:    movb %al, %r10b
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    movq %rdi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movb %r10b, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
+; LINUX-OPT0-NEXT:    je .LBB2_4
+; LINUX-OPT0-NEXT:  # %bb.3:
+; LINUX-OPT0-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm3, (%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm4, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm5, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
+; LINUX-OPT0-NEXT:  .LBB2_4:
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; LINUX-OPT0-NEXT:    testb $1, (%rax)
+; LINUX-OPT0-NEXT:    jne .LBB2_1
+; LINUX-OPT0-NEXT:    jmp .LBB2_2
+; LINUX-OPT0-NEXT:  .LBB2_1: # %then
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq 8(%rax), %rcx
+; LINUX-OPT0-NEXT:    movq %rax, %rdi
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r8, %rcx
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r11 # 8-byte Reload
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    je .LBB2_6
+; LINUX-OPT0-NEXT:  # %bb.5: # %then
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm7
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm6
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm5
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm4
+; LINUX-OPT0-NEXT:    movaps (%rsp), %xmm3
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm2
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
+; LINUX-OPT0-NEXT:    addq $88, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-OPT0-NEXT:  .LBB2_6: # %then
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-OPT0-NEXT:    addq $88, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-OPT0-NEXT:  .LBB2_2: # %else
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq 16(%rax), %rcx
+; LINUX-OPT0-NEXT:    movl $42, {{.*}}(%rip)
+; LINUX-OPT0-NEXT:    movq %rax, %rdi
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; LINUX-OPT0-NEXT:    movq %r8, %rcx
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-OPT0-NEXT:    movb {{[-0-9]+}}(%r{{[sb]}}p), %al # 1-byte Reload
+; LINUX-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r11 # 8-byte Reload
+; LINUX-OPT0-NEXT:    testb %al, %al
+; LINUX-OPT0-NEXT:    je .LBB2_8
+; LINUX-OPT0-NEXT:  # %bb.7: # %else
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm7
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm6
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm5
+; LINUX-OPT0-NEXT:    movaps {{[0-9]+}}(%rsp), %xmm4
+; LINUX-OPT0-NEXT:    movaps (%rsp), %xmm3
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm2
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm1
+; LINUX-OPT0-NEXT:    movaps -{{[0-9]+}}(%rsp), %xmm0
+; LINUX-OPT0-NEXT:    addq $88, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-OPT0-NEXT:  .LBB2_8: # %else
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-OPT0-NEXT:    addq $88, %rsp
+; LINUX-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+;
 ; LINUX-X32-LABEL: h_thunk:
 ; LINUX-X32:       # %bb.0:
+; LINUX-X32-NEXT:    pushq %rax
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-X32-NEXT:    testb %al, %al
+; LINUX-X32-NEXT:    je .LBB2_2
+; LINUX-X32-NEXT:  # %bb.1:
+; LINUX-X32-NEXT:    movaps %xmm0, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm1, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm2, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm3, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm4, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm5, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm6, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movaps %xmm7, -{{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:  .LBB2_2:
 ; LINUX-X32-NEXT:    cmpb $1, (%edi)
-; LINUX-X32-NEXT:    jne .LBB2_2
-; LINUX-X32-NEXT:  # %bb.1: # %then
+; LINUX-X32-NEXT:    jne .LBB2_4
+; LINUX-X32-NEXT:  # %bb.3: # %then
 ; LINUX-X32-NEXT:    movl 4(%edi), %r11d
+; LINUX-X32-NEXT:    testb %al, %al
+; LINUX-X32-NEXT:    je .LBB2_6
+; LINUX-X32-NEXT:  # %bb.5: # %then
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm3
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-NEXT:    addl $8, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
-; LINUX-X32-NEXT:  .LBB2_2: # %else
+; LINUX-X32-NEXT:  .LBB2_4: # %else
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
 ; LINUX-X32-NEXT:    movl 8(%edi), %r11d
 ; LINUX-X32-NEXT:    movl $42, {{.*}}(%rip)
+; LINUX-X32-NEXT:    testb %al, %al
+; LINUX-X32-NEXT:    je .LBB2_8
+; LINUX-X32-NEXT:  # %bb.7: # %else
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm3
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-NEXT:    addl $8, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-NEXT:  .LBB2_6: # %then
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-X32-NEXT:    addl $8, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-NEXT:  .LBB2_8: # %else
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 16
+; LINUX-X32-NEXT:    addl $8, %esp
+; LINUX-X32-NEXT:    .cfi_def_cfa_offset 8
 ; LINUX-X32-NEXT:    jmpq *%r11 # TAILCALL
 ;
+; LINUX-X32-OPT0-LABEL: h_thunk:
+; LINUX-X32-OPT0:       # %bb.0:
+; LINUX-X32-OPT0-NEXT:    subl $88, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-X32-OPT0-NEXT:    movb %al, %r10b
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    movl %edi, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rsi, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movb %r10b, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
+; LINUX-X32-OPT0-NEXT:    je .LBB2_4
+; LINUX-X32-OPT0-NEXT:  # %bb.3:
+; LINUX-X32-OPT0-NEXT:    movaps %xmm0, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm1, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm2, -{{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm3, (%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm4, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm5, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm6, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:    movaps %xmm7, {{[0-9]+}}(%esp)
+; LINUX-X32-OPT0-NEXT:  .LBB2_4:
+; LINUX-X32-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; LINUX-X32-OPT0-NEXT:    testb $1, (%eax)
+; LINUX-X32-OPT0-NEXT:    jne .LBB2_1
+; LINUX-X32-OPT0-NEXT:    jmp .LBB2_2
+; LINUX-X32-OPT0-NEXT:  .LBB2_1: # %then
+; LINUX-X32-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; LINUX-X32-OPT0-NEXT:    movl 4(%eax), %ecx
+; LINUX-X32-OPT0-NEXT:    movl %ecx, %edx
+; LINUX-X32-OPT0-NEXT:    movl %eax, %edi
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r8, %rdx
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movb {{[-0-9]+}}(%e{{[sb]}}p), %al # 1-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r11 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    je .LBB2_6
+; LINUX-X32-OPT0-NEXT:  # %bb.5: # %then
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-OPT0-NEXT:    movaps (%esp), %xmm3
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-OPT0-NEXT:    addl $88, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-OPT0-NEXT:  .LBB2_6: # %then
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-X32-OPT0-NEXT:    addl $88, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-OPT0-NEXT:  .LBB2_2: # %else
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-X32-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; LINUX-X32-OPT0-NEXT:    movl 8(%eax), %ecx
+; LINUX-X32-OPT0-NEXT:    movl %ecx, %edx
+; LINUX-X32-OPT0-NEXT:    movl $42, {{.*}}(%rip)
+; LINUX-X32-OPT0-NEXT:    movl %eax, %edi
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rsi # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%e{{[sb]}}p) # 8-byte Spill
+; LINUX-X32-OPT0-NEXT:    movq %r8, %rdx
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %rcx # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r8 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r9 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    movb {{[-0-9]+}}(%e{{[sb]}}p), %al # 1-byte Reload
+; LINUX-X32-OPT0-NEXT:    movq {{[-0-9]+}}(%e{{[sb]}}p), %r11 # 8-byte Reload
+; LINUX-X32-OPT0-NEXT:    testb %al, %al
+; LINUX-X32-OPT0-NEXT:    je .LBB2_8
+; LINUX-X32-OPT0-NEXT:  # %bb.7: # %else
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm7
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm6
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm5
+; LINUX-X32-OPT0-NEXT:    movaps {{[0-9]+}}(%esp), %xmm4
+; LINUX-X32-OPT0-NEXT:    movaps (%esp), %xmm3
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm2
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm1
+; LINUX-X32-OPT0-NEXT:    movaps -{{[0-9]+}}(%esp), %xmm0
+; LINUX-X32-OPT0-NEXT:    addl $88, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+; LINUX-X32-OPT0-NEXT:  .LBB2_8: # %else
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 96
+; LINUX-X32-OPT0-NEXT:    addl $88, %esp
+; LINUX-X32-OPT0-NEXT:    .cfi_def_cfa_offset 8
+; LINUX-X32-OPT0-NEXT:    jmpq *%r11 # TAILCALL
+;
 ; WINDOWS-LABEL: h_thunk:
 ; WINDOWS:       # %bb.0:
 ; WINDOWS-NEXT:    cmpb $1, (%rcx)
 ; WINDOWS-NEXT:    jne .LBB2_2
 ; WINDOWS-NEXT:  # %bb.1: # %then
 ; WINDOWS-NEXT:    movq 8(%rcx), %rax
 ; WINDOWS-NEXT:    rex64 jmpq *%rax # TAILCALL
 ; WINDOWS-NEXT:  .LBB2_2: # %else
 ; WINDOWS-NEXT:    movq 16(%rcx), %rax
 ; WINDOWS-NEXT:    movl $42, {{.*}}(%rip)
 ; WINDOWS-NEXT:    rex64 jmpq *%rax # TAILCALL
 ;
-; X86-LABEL: h_thunk:
-; X86:       # %bb.0:
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    cmpb $1, (%eax)
-; X86-NEXT:    jne LBB2_2
-; X86-NEXT:  # %bb.1: # %then
-; X86-NEXT:    movl 4(%eax), %ecx
-; X86-NEXT:    movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT:    jmpl *%ecx # TAILCALL
-; X86-NEXT:  LBB2_2: # %else
-; X86-NEXT:    movl 8(%eax), %ecx
-; X86-NEXT:    movl $42, _g
-; X86-NEXT:    movl %eax, {{[0-9]+}}(%esp)
-; X86-NEXT:    jmpl *%ecx # TAILCALL
+; WINDOWS-OPT0-LABEL: h_thunk:
+; WINDOWS-OPT0:       # %bb.0:
+; WINDOWS-OPT0-NEXT:    subq $48, %rsp
+; WINDOWS-OPT0-NEXT:    .seh_stackalloc 48
+; WINDOWS-OPT0-NEXT:    .seh_endprologue
+; WINDOWS-OPT0-NEXT:    testb $1, (%rcx)
+; WINDOWS-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %rdx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %r8, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %r9, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    jne .LBB2_1
+; WINDOWS-OPT0-NEXT:    jmp .LBB2_2
+; WINDOWS-OPT0-NEXT:  .LBB2_1: # %then
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq 8(%rax), %rcx
+; WINDOWS-OPT0-NEXT:    movq %rcx, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %rax, %rcx
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r10 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    addq $48, %rsp
+; WINDOWS-OPT0-NEXT:    rex64 jmpq *%r10 # TAILCALL
+; WINDOWS-OPT0-NEXT:  .LBB2_2: # %else
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rax # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq 16(%rax), %rcx
+; WINDOWS-OPT0-NEXT:    movl $42, {{.*}}(%rip)
+; WINDOWS-OPT0-NEXT:    movq %rcx, (%rsp) # 8-byte Spill
+; WINDOWS-OPT0-NEXT:    movq %rax, %rcx
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %rdx # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r8 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq {{[-0-9]+}}(%r{{[sb]}}p), %r9 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    movq (%rsp), %r10 # 8-byte Reload
+; WINDOWS-OPT0-NEXT:    addq $48, %rsp
+; WINDOWS-OPT0-NEXT:    rex64 jmpq *%r10 # TAILCALL
+; WINDOWS-OPT0-NEXT:    .seh_handlerdata
+; WINDOWS-OPT0-NEXT:    .text
+; WINDOWS-OPT0-NEXT:    .seh_endproc
+;
+; X86-NOSSE-LABEL: h_thunk:
+; X86-NOSSE:       # %bb.0:
+; X86-NOSSE-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOSSE-NEXT:    cmpb $1, (%eax)
+; X86-NOSSE-NEXT:    jne LBB2_2
+; X86-NOSSE-NEXT:  # %bb.1: # %then
+; X86-NOSSE-NEXT:    movl 4(%eax), %ecx
+; X86-NOSSE-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-NOSSE-NEXT:    jmpl *%ecx # TAILCALL
+; X86-NOSSE-NEXT:  LBB2_2: # %else
+; X86-NOSSE-NEXT:    movl 8(%eax), %ecx
+; X86-NOSSE-NEXT:    movl $42, _g
+; X86-NOSSE-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-NOSSE-NEXT:    jmpl *%ecx # TAILCALL
+;
+; X86-NOSSE-OPT0-LABEL: h_thunk:
+; X86-NOSSE-OPT0:       # %bb.0:
+; X86-NOSSE-OPT0-NEXT:    pushl %eax
+; X86-NOSSE-OPT0-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-NOSSE-OPT0-NEXT:    testb $1, (%eax)
+; X86-NOSSE-OPT0-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X86-NOSSE-OPT0-NEXT:    jne LBB2_1
+; X86-NOSSE-OPT0-NEXT:    jmp LBB2_2
+; X86-NOSSE-OPT0-NEXT:  LBB2_1: # %then
+; X86-NOSSE-OPT0-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X86-NOSSE-OPT0-NEXT:    movl 4(%eax), %ecx
+; X86-NOSSE-OPT0-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-NOSSE-OPT0-NEXT:    popl %eax
+; X86-NOSSE-OPT0-NEXT:    jmpl *%ecx # TAILCALL
+; X86-NOSSE-OPT0-NEXT:  LBB2_2: # %else
+; X86-NOSSE-OPT0-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X86-NOSSE-OPT0-NEXT:    movl 8(%eax), %ecx
+; X86-NOSSE-OPT0-NEXT:    movl $42, _g
+; X86-NOSSE-OPT0-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-NOSSE-OPT0-NEXT:    popl %eax
+; X86-NOSSE-OPT0-NEXT:    jmpl *%ecx # TAILCALL
+;
+; X86-SSE-LABEL: h_thunk:
+; X86-SSE:       # %bb.0:
+; X86-SSE-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-SSE-NEXT:    cmpb $1, (%eax)
+; X86-SSE-NEXT:    jne LBB2_2
+; X86-SSE-NEXT:  # %bb.1: # %then
+; X86-SSE-NEXT:    movl 4(%eax), %ecx
+; X86-SSE-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-SSE-NEXT:    jmpl *%ecx # TAILCALL
+; X86-SSE-NEXT:  LBB2_2: # %else
+; X86-SSE-NEXT:    movl 8(%eax), %ecx
+; X86-SSE-NEXT:    movl $42, _g
+; X86-SSE-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-SSE-NEXT:    jmpl *%ecx # TAILCALL
+;
+; X86-SSE-OPT0-LABEL: h_thunk:
+; X86-SSE-OPT0:       # %bb.0:
+; X86-SSE-OPT0-NEXT:    subl $76, %esp
+; X86-SSE-OPT0-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; X86-SSE-OPT0-NEXT:    testb $1, (%eax)
+; X86-SSE-OPT0-NEXT:    movups %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X86-SSE-OPT0-NEXT:    movups %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; X86-SSE-OPT0-NEXT:    movl %eax, {{[-0-9]+}}(%e{{[sb]}}p) # 4-byte Spill
+; X86-SSE-OPT0-NEXT:    movups %xmm2, (%esp) # 16-byte Spill
+; X86-SSE-OPT0-NEXT:    jne LBB2_1
+; X86-SSE-OPT0-NEXT:    jmp LBB2_2
+; X86-SSE-OPT0-NEXT:  LBB2_1: # %then
+; X86-SSE-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-SSE-OPT0-NEXT:    movl 4(%eax), %ecx
+; X86-SSE-OPT0-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-SSE-OPT0-NEXT:    movups {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movups {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movups (%esp), %xmm2 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    addl $76, %esp
+; X86-SSE-OPT0-NEXT:    jmpl *%ecx # TAILCALL
+; X86-SSE-OPT0-NEXT:  LBB2_2: # %else
+; X86-SSE-OPT0-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %eax # 4-byte Reload
+; X86-SSE-OPT0-NEXT:    movl 8(%eax), %ecx
+; X86-SSE-OPT0-NEXT:    movl $42, _g
+; X86-SSE-OPT0-NEXT:    movl %eax, {{[0-9]+}}(%esp)
+; X86-SSE-OPT0-NEXT:    movups {{[-0-9]+}}(%e{{[sb]}}p), %xmm0 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movups {{[-0-9]+}}(%e{{[sb]}}p), %xmm1 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    movups (%esp), %xmm2 # 16-byte Reload
+; X86-SSE-OPT0-NEXT:    addl $76, %esp
+; X86-SSE-OPT0-NEXT:    jmpl *%ecx # TAILCALL
   %cond_p = getelementptr %struct.Foo, %struct.Foo* %this, i32 0, i32 0
   %cond = load i1, i1* %cond_p
   br i1 %cond, label %then, label %else
 
 then:
   %a_p = getelementptr %struct.Foo, %struct.Foo* %this, i32 0, i32 1
   %a_i8 = load i8*, i8** %a_p
   %a = bitcast i8* %a_i8 to void (%struct.Foo*, ...)*

llvm/test/CodeGen/X86/vastart-defs-eflags.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc %s -o - | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.10.0"
 
 ; Check that vastart handling doesn't get between testb and je for the branch.
 define i32 @check_flag(i32 %flags, ...) nounwind {
 ; CHECK-LABEL: check_flag:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    subq $56, %rsp
+; CHECK-NEXT:    movq %rsi, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %r8, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %r9, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je LBB0_2
 ; CHECK-NEXT:  ## %bb.1: ## %entry
 ; CHECK-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm3, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm4, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm5, (%rsp)
 ; CHECK-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:  LBB0_2: ## %entry
-; CHECK-NEXT:    movq %rsi, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %r8, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %r9, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testl $512, %edi ## imm = 0x200
 ; CHECK-NEXT:    je LBB0_4
 ; CHECK-NEXT:  ## %bb.3: ## %if.then
 ; CHECK-NEXT:    leaq -{{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movq %rax, 16
 ; CHECK-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movq %rax, 8

llvm/test/CodeGen/X86/x32-va_start.ll

 
 define i32 @foo(float %a, i8* nocapture readnone %fmt, ...) nounwind {
 entry:
   %ap = alloca [1 x %struct.__va_list_tag], align 16
   %0 = bitcast [1 x %struct.__va_list_tag]* %ap to i8*
   call void @llvm.lifetime.start.p0i8(i64 16, i8* %0) #2
   call void @llvm.va_start(i8* %0)
 ; SSE: subl $72, %esp
+; CHECK-DAG: movq %r9
+; CHECK-DAG: movq %r8
+; CHECK-DAG: movq %rcx
+; CHECK-DAG: movq %rdx
+; CHECK-DAG: movq %rsi
 ; SSE: testb %al, %al
 ; SSE: je .[[NOFP:.*]]
 ; SSE-DAG: movaps %xmm1
 ; SSE-DAG: movaps %xmm2
 ; SSE-DAG: movaps %xmm3
 ; SSE-DAG: movaps %xmm4
 ; SSE-DAG: movaps %xmm5
 ; SSE-DAG: movaps %xmm6
 ; SSE-DAG: movaps %xmm7
 ; NOSSE-NOT: xmm
 ; SSE: .[[NOFP]]:
-; CHECK-DAG: movq %r9
-; CHECK-DAG: movq %r8
-; CHECK-DAG: movq %rcx
-; CHECK-DAG: movq %rdx
-; CHECK-DAG: movq %rsi
   %gp_offset_p = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %ap, i32 0, i32 0, i32 0
   %gp_offset = load i32, i32* %gp_offset_p, align 16
   %fits_in_gp = icmp ult i32 %gp_offset, 41
   br i1 %fits_in_gp, label %vaarg.in_reg, label %vaarg.in_mem
 ; CHECK: cmpl $40, [[COUNT:.*]]
 ; CHECK: ja .[[IN_MEM:.*]]
 
 vaarg.in_reg:                                     ; preds = %entry

llvm/test/CodeGen/X86/xmm-vararg-noopt.ll

+; RUN: llc -O0 -mtriple=x86_64-unknown-unknown < %s | FileCheck %s
+
+; CHECK-LABEL: testvarargs
+; Ensure that xmm registers are not used before testing %al
+; CHECK-NOT: xmm
+; CHECK: testb %al, %al
+; CHECK-NOT: xmm
+; CHECK: # %bb.1
+; CHECK-NEXT: %xmm0, {{.*}}%rsp
+; CHECK-NEXT: %xmm1, {{.*}}%rsp
+; CHECK-NEXT: %xmm2, {{.*}}%rsp
+; CHECK-NEXT: %xmm3, {{.*}}%rsp
+; CHECK-NEXT: %xmm4, {{.*}}%rsp
+; CHECK-NEXT: %xmm5, {{.*}}%rsp
+; CHECK-NEXT: %xmm6, {{.*}}%rsp
+; CHECK-NEXT: %xmm7, {{.*}}%rsp
+
+; ModuleID = 'variadic.c'
+source_filename = "variadic.c"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux"
+
+%struct.__va_list_tag = type { i32, i32, i8*, i8* }
+
+@.str = private unnamed_addr constant [9 x i8] c"\0A hello \00", align 1
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @testvarargs(i8* %fmt, ...) {
+entry:
+  %fmt.addr = alloca i8*, align 8
+  %va = alloca [1 x %struct.__va_list_tag], align 16
+  store i8* %fmt, i8** %fmt.addr, align 8
+  %arraydecay = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %va, i64 0, i64 0
+  %arraydecay1 = bitcast %struct.__va_list_tag* %arraydecay to i8*
+  call void @llvm.va_start(i8* %arraydecay1)
+  %arraydecay2 = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %va, i64 0, i64 0
+  %arraydecay23 = bitcast %struct.__va_list_tag* %arraydecay2 to i8*
+  call void @llvm.va_end(i8* %arraydecay23)
+  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([9 x i8], [9 x i8]* @.str, i64 0, i64 0))
+  ret void
+}
+
+; Function Attrs: nounwind
+declare void @llvm.va_start(i8*)
+
+; Function Attrs: nounwind
+declare void @llvm.va_end(i8*)
+
+declare dso_local i32 @printf(i8*, ...)