[X86][VARARG] Avoid spilling xmm registers for va_start.

[avl]

Summary

That review is extracted from D69372.
It fixes https://bugs.llvm.org/show_bug.cgi?id=42219 bug.

For the noimplicitfloat mode, the compiler mustn't generate
floating-point code if it was not asked directly to do so.
This rule does not work with variable function arguments currently.
Though compiler correctly guards block of code, which copies xmm vararg
parameters with a check for %al, it does not protect spills for xmm registers.
Thus, such spills are generated in non-protected areas and could break code,
which does not expect floating-point data. The problem happens in -O0
optimization mode. With this optimization level there is used
FastRegisterAllocator, which spills virtual registers at basic block boundaries.
Register Allocator does not protect spills with additional control-flow modifications.
Thus to resolve that problem, it is suggested to not copy incoming physical
registers into virtual registers. Instead, store incoming physical xmm registers
into the memory from scratch.

[aeubanks]

not knowledgeable enough on this

[avl]

ping

[avl]

ping.

[avl]

ping.

[RKSimon]

Do we need better i686- triple testing?

I don't know much about this area - @craig.topper @efriedma any thoughts?

llvm/lib/Target/X86/X86ExpandPseudo.cpp

506

Any reason not to deal with YMM/ZMM as well?

llvm/test/CodeGen/X86/x32-va_start.ll

3–5

Regenerate with update_llc ?

[avl]

Do we need better i686- triple testing?

Will rebase and retest, including clang bootstrap and clang bootstrap with address sanitizer.

llvm/lib/Target/X86/X86ExpandPseudo.cpp

506

The reason is to not complicate the patch. YMM/ZMM is not supported for varargs currently. So YMM/ZMM support needs its own patch independent from noimplicitfloat problem.

llvm/test/CodeGen/X86/x32-va_start.ll

3–5

will do that way. thanks.

[avl]

Do we need better i686- triple testing?

It looks like I misunderstood the question. This patch should affect X86-64(AMD64 abi) only and should not affect 32-bit abi. Thus I would add checking for i386,i686 triples into the llvm/test/CodeGen/X86/x32-va_start.ll. That test would show that 32-bit abi is not affected(as it currently is checked for -mtriple=x86_64-linux-gnux32 -mattr=-sse).

[efriedma]

@RKSimon The general approach of using a pseudo to hide the control flow until after register allocation seems fine. I'm not going to try to review the exact instruction sequences.

[avl]

rebased,
retested(bootstrap with check-all),
added i386/i686 triples into X86/x32-va_start.ll,
regenerated tests with update_llc_test_checks.py.

[avl]

@RKSimon @craig.topper Would you mind to take a look at this review, please?

[craig.topper]

llvm/lib/Target/X86/X86ExpandPseudo.cpp

531

Why do we only add them ass liveins this late? Shouldn't they have been livein when the block was created in the custom inserter?

llvm/lib/Target/X86/X86ISelLowering.cpp

31654

Is RegIdx used here?

31657

I'm not familiar with InternalRead. The documentation says it means it was defined by the same instruction. Is that case here?

[avl]

llvm/lib/Target/X86/X86ExpandPseudo.cpp

531

Things were done that way(fill liveins when the block was created
in the custom inserter) in the first version of the fix.
But it creates new liveness rule for physical registers
(newly added block should have physical registers in its liveins
but it is not allowed before regalloc):

https://reviews.llvm.org/D69372?id=227838#change-TJhqJRPUjpDM

So there was a request to do things in other way, not requiring
to change liveness rule:

https://reviews.llvm.org/D69372#1734756

The solution done in that patch creates a block with SAVE_VARARG_XMM_REGS
pseudo instruction. That pseudo instruction defines xmm registers
(used by varargs). After register allocator, SAVE_VARARG_XMM_REGS pseudo
instruction is expanded to the real copy instructions and put xmm registers
into the block liveins.

llvm/lib/Target/X86/X86ISelLowering.cpp

31654

no. will delete it.

31657

that is how the solution works: SAVE_VARARG_XMM_REGS pseudo declares
itself as an instruction which reads the registers defined by itself. And later
(when SAVE_VARARG_XMM_REGS is expanded) the correct definitions
would be put into the liveins.

[avl]

llvm/lib/Target/X86/X86ExpandPseudo.cpp

531

But... Reading this recent comment https://reviews.llvm.org/D80163#2224926 I think I implemented not exactly what was requested. Implementation from this patch does not hide control flow. Would do implementation which will hide control flow.

[avl]

moved control-flow expansion for VASTART_SAVE_XMM_REGS until after regalloc.

[avl]

rebased.

[avl]

@craig.topper Would you mind to take a look at this updated patch once more, please?

[craig.topper]

@pengfei or @RKSimon do you mind taking a look?

[pengfei]

I had a check on the tests, the changes LGTM.

llvm/lib/Target/X86/X86ISelLowering.cpp

3445

Does the change affect GreedyRA?

llvm/test/CodeGen/X86/xmm-vararg-noopt.ll

2

Maybe we can also generate the test by update_llc_test_checks?

[avl]

llvm/lib/Target/X86/X86ISelLowering.cpp

3445

Yes, it does. GreedyRA might have the same problem if it would generate a spill in entry block for the xmm varargs. Though, it would probably be some specific artificial case(in contrast to FastRA which always spills live ranges on the block boundary).

llvm/test/CodeGen/X86/xmm-vararg-noopt.ll

2

Ok, will do that way.

[avl]

updated the test.

[pengfei]

LGTM. Thanks.

[avl]

@pengfei @craig.topper Thank you for the review.

llvm/lib/Target/X86/X86ExpandPseudo.cpp

public:
   }
 
 private:
   void ExpandICallBranchFunnel(MachineBasicBlock *MBB,
                                MachineBasicBlock::iterator MBBI);
 
   bool ExpandMI(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI);
   bool ExpandMBB(MachineBasicBlock &MBB);
+
+  void ExpandSaveVarargXmmRegs(MachineBasicBlock *MBB,
+                               MachineBasicBlock::iterator MBBI) const;
 };
 char X86ExpandPseudo::ID = 0;
 
 } // End anonymous namespace.
 
 INITIALIZE_PASS(X86ExpandPseudo, DEBUG_TYPE, X86_EXPAND_PSEUDO_NAME, false,
                 false)
 
TII->copyPhysReg(MBB, MBBI, DL, BasePointer, SaveRbx,
                      /*SrcIsKill*/ true);
     // Delete the pseudo.
     MBBI->eraseFromParent();
     return true;
   }
   case TargetOpcode::ICALL_BRANCH_FUNNEL:
     ExpandICallBranchFunnel(&MBB, MBBI);
     return true;
+
+  case X86::SAVE_VARARG_XMM_REGS:
+    ExpandSaveVarargXmmRegs(&MBB, MBBI);
+    return true;
   }
   llvm_unreachable("Previous switch has a fallthrough?");
 }
 
+/// This function replaces X86::SAVE_VARARG_XMM_REGS pseudo instruction
+/// with set of copying instructions for specified xmm vararg registers.
+///
+/// [0] parameter of X86::SAVE_VARARG_XMM_REGS is frame index of stack area,
+///     where registers should be stored
+/// [1] parameter of X86::SAVE_VARARG_XMM_REGS is offset inside stack frame
+///     to the area where registers should be stored
+/// [2] - [till end] parameters of X86::SAVE_VARARG_XMM_REGS are set of
+///     xmm registers which should be stored.
+void X86ExpandPseudo::ExpandSaveVarargXmmRegs(
+    MachineBasicBlock *GuardedBlock,
+    MachineBasicBlock::iterator SaveVarargXmmRegsInstr) const {
+  assert(SaveVarargXmmRegsInstr->getOpcode() == X86::SAVE_VARARG_XMM_REGS);
+
+  MachineFunction *Func = GuardedBlock->getParent();
+  DebugLoc DL = SaveVarargXmmRegsInstr->getDebugLoc();
+
+  int64_t FrameIndex = SaveVarargXmmRegsInstr->getOperand(0).getImm();
+  Register BaseReg;
+  int64_t FrameOffset =
+      X86FL->getFrameIndexReference(*Func, FrameIndex, BaseReg);
+  int64_t VarArgsRegsOffset = SaveVarargXmmRegsInstr->getOperand(1).getImm();
+
+  // TODO: add support for YMM and ZMM here.
+  unsigned MOVOpc = STI->hasAVX() ? X86::VMOVAPSmr : X86::MOVAPSmr;
+
+  // In the XMM save block, save all the XMM argument registers.
+  for (int64_t OpndIdx = 2, RegIdx = 0;
+       OpndIdx < SaveVarargXmmRegsInstr->getNumOperands();
+       OpndIdx++, RegIdx++) {
+
+    int64_t Offset = FrameOffset + VarArgsRegsOffset + RegIdx * 16;
+
+    MachineMemOperand *MMO = Func->getMachineMemOperand(
+        MachinePointerInfo::getFixedStack(*Func, FrameIndex, Offset),
+        MachineMemOperand::MOStore,
+        /*Size=*/16, Align(16));
+
+    BuildMI(GuardedBlock, DL, TII->get(MOVOpc))
+        .addReg(BaseReg)
+        .addImm(/*Scale=*/1)
+        .addReg(/*IndexReg=*/0)
+        .addImm(/*Disp=*/Offset)
+        .addReg(/*Segment=*/0)
+        .addReg(SaveVarargXmmRegsInstr->getOperand(OpndIdx).getReg())
+        .addMemOperand(MMO);
+    assert(Register::isPhysicalRegister(
+        SaveVarargXmmRegsInstr->getOperand(OpndIdx).getReg()));
+
+    GuardedBlock->addLiveIn(
+        SaveVarargXmmRegsInstr->getOperand(OpndIdx).getReg());
+  }
+
+  // Delete the pseudo.
+  SaveVarargXmmRegsInstr->eraseFromParent();
+}
+
 /// Expand all pseudo instructions contained in \p MBB.
 /// \returns true if any expansion occurred for \p MBB.
 bool X86ExpandPseudo::ExpandMBB(MachineBasicBlock &MBB) {
   bool Modified = false;
 
   // MBBI may be invalidated by the expansion.
   MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
   while (MBBI != E) {

llvm/lib/Target/X86/X86ISelLowering.h

private:
     EmitVAARG64WithCustomInserter(MachineInstr &MI,
                                   MachineBasicBlock *MBB) const;
 
     /// Utility function to emit the xmm reg save portion of va_start.
     MachineBasicBlock *
     EmitVAStartSaveXMMRegsWithCustomInserter(MachineInstr &BInstr,
                                              MachineBasicBlock *BB) const;
 
+    void AddSaveVarargXmmRegsPseudo(MachineBasicBlock *GuardedRegsBlk,
+                                    MachineBasicBlock *TailBlk,
+                                    MachineInstr &SrcPseudoInstr) const;
+
     MachineBasicBlock *EmitLoweredCascadedSelect(MachineInstr &MI1,
                                                  MachineInstr &MI2,
                                                  MachineBasicBlock *BB) const;
 
     MachineBasicBlock *EmitLoweredSelect(MachineInstr &I,
                                          MachineBasicBlock *BB) const;
 
     MachineBasicBlock *EmitLoweredCatchRet(MachineInstr &MI,

llvm/lib/Target/X86/X86ISelLowering.cpp

for (MCPhysReg Reg : ArgGPRs.slice(NumIntRegs)) {
       Register GPR = TheMachineFunction.addLiveIn(Reg, &X86::GR64RegClass);
       LiveGPRs.push_back(DAG.getCopyFromReg(Chain, DL, GPR, MVT::i64));
     }
     const auto &AvailableXmms = ArgXMMs.slice(NumXMMRegs);
     if (!AvailableXmms.empty()) {
       Register AL = TheMachineFunction.addLiveIn(X86::AL, &X86::GR8RegClass);
       ALVal = DAG.getCopyFromReg(Chain, DL, AL, MVT::i8);
       for (MCPhysReg Reg : AvailableXmms) {
-        Register XMMReg = TheMachineFunction.addLiveIn(Reg, &X86::VR128RegClass);
-        LiveXMMRegs.push_back(
-            DAG.getCopyFromReg(Chain, DL, XMMReg, MVT::v4f32));
+        // FastRegisterAllocator spills virtual registers at basic
+        // block boundary. That leads to usages of xmm registers
+        // outside of check for %al. Pass physical registers to
+        // VASTART_SAVE_XMM_REGS to avoid unneccessary spilling.
+        TheMachineFunction.getRegInfo().addLiveIn(Reg);
+        LiveXMMRegs.push_back(DAG.getRegister(Reg, MVT::v4f32));
       }
     }
 
     // Store the integer parameter registers.
     SmallVector<SDValue, 8> MemOps;
     SDValue RSFIN =
         DAG.getFrameIndex(FuncInfo->getRegSaveFrameIndex(),
                           TargLowering.getPointerTy(DAG.getDataLayout()));
X86TargetLowering::EmitVAARG64WithCustomInserter(MachineInstr &MI,
   }
 
   // Erase the pseudo instruction
   MI.eraseFromParent();
 
   return endMBB;
 }
 
-MachineBasicBlock *X86TargetLowering::EmitVAStartSaveXMMRegsWithCustomInserter(
-    MachineInstr &MI, MachineBasicBlock *MBB) const {
-  // Emit code to save XMM registers to the stack. The ABI says that the
-  // number of registers to save is given in %al, so it's theoretically
-  // possible to do an indirect jump trick to avoid saving all of them,
-  // however this code takes a simpler approach and just executes all
-  // of the stores if %al is non-zero. It's less code, and it's probably
-  // easier on the hardware branch predictor, and stores aren't all that
-  // expensive anyway.
+// This function creates additional block for storing varargs guarded
+// registers. It adds check for %al into entry block, to skip
+// GuardedRegsBlk if xmm registers should not be stored.
+//
+//     EntryBlk[VAPseudoInstr]          EntryBlk
+//        |                              |     .
+//        |                              |        .
+//        |                              |   GuardedRegsBlk
+//        |                      =>      |        .
+//        |                              |     .
+//        |                             TailBlk[VAPseudoInstr]
+//        |                              |
+//        |                              |
+//
+static std::pair<MachineBasicBlock *, MachineBasicBlock *>
+createGuardedRegsBlock(MachineBasicBlock *EntryBlk, MachineInstr &VAPseudoInstr,
+                       const X86Subtarget &Subtarget) {
+  MachineFunction *Func = EntryBlk->getParent();
+  const TargetInstrInfo *TII = Subtarget.getInstrInfo();
+  DebugLoc DL = VAPseudoInstr.getDebugLoc();
+  Register CountReg = VAPseudoInstr.getOperand(0).getReg();
 
   // Create the new basic blocks. One block contains all the XMM stores,
   // and one block is the final destination regardless of whether any
   // stores were performed.
-  const BasicBlock *LLVM_BB = MBB->getBasicBlock();
-  MachineFunction *F = MBB->getParent();
-  MachineFunction::iterator MBBIter = ++MBB->getIterator();
-  MachineBasicBlock *XMMSaveMBB = F->CreateMachineBasicBlock(LLVM_BB);
-  MachineBasicBlock *EndMBB = F->CreateMachineBasicBlock(LLVM_BB);
-  F->insert(MBBIter, XMMSaveMBB);
-  F->insert(MBBIter, EndMBB);
+  const BasicBlock *LLVMBlk = EntryBlk->getBasicBlock();
+  MachineFunction::iterator EntryBlkIter = ++EntryBlk->getIterator();
+  MachineBasicBlock *GuardedRegsBlk = Func->CreateMachineBasicBlock(LLVMBlk);
+  MachineBasicBlock *TailBlk = Func->CreateMachineBasicBlock(LLVMBlk);
+  Func->insert(EntryBlkIter, GuardedRegsBlk);
+  Func->insert(EntryBlkIter, TailBlk);
 
   // Transfer the remainder of MBB and its successor edges to EndMBB.
-  EndMBB->splice(EndMBB->begin(), MBB,
-                 std::next(MachineBasicBlock::iterator(MI)), MBB->end());
-  EndMBB->transferSuccessorsAndUpdatePHIs(MBB);
+  TailBlk->splice(TailBlk->begin(), EntryBlk,
+                  std::next(MachineBasicBlock::iterator(VAPseudoInstr)),
+                  EntryBlk->end());
+  TailBlk->transferSuccessorsAndUpdatePHIs(EntryBlk);
 
   // The original block will now fall through to the XMM save block.
-  MBB->addSuccessor(XMMSaveMBB);
+  EntryBlk->addSuccessor(GuardedRegsBlk);
   // The XMMSaveMBB will fall through to the end block.
-  XMMSaveMBB->addSuccessor(EndMBB);
-
-  // Now add the instructions.
-  const TargetInstrInfo *TII = Subtarget.getInstrInfo();
-  DebugLoc DL = MI.getDebugLoc();
-
-  Register CountReg = MI.getOperand(0).getReg();
-  int64_t RegSaveFrameIndex = MI.getOperand(1).getImm();
-  int64_t VarArgsFPOffset = MI.getOperand(2).getImm();
+  GuardedRegsBlk->addSuccessor(TailBlk);
 
-  if (!Subtarget.isCallingConvWin64(F->getFunction().getCallingConv())) {
+  if (!Subtarget.isCallingConvWin64(Func->getFunction().getCallingConv())) {
     // If %al is 0, branch around the XMM save block.
-    BuildMI(MBB, DL, TII->get(X86::TEST8rr)).addReg(CountReg).addReg(CountReg);
-    BuildMI(MBB, DL, TII->get(X86::JCC_1)).addMBB(EndMBB).addImm(X86::COND_E);
-    MBB->addSuccessor(EndMBB);
+    BuildMI(EntryBlk, DL, TII->get(X86::TEST8rr))
+        .addReg(CountReg)
+        .addReg(CountReg);
+    BuildMI(EntryBlk, DL, TII->get(X86::JCC_1))
+        .addMBB(TailBlk)
+        .addImm(X86::COND_E);
+    EntryBlk->addSuccessor(TailBlk);
   }
 
+  return std::make_pair(GuardedRegsBlk, TailBlk);
+}
+
+void X86TargetLowering::AddSaveVarargXmmRegsPseudo(
+    MachineBasicBlock *GuardedRegsBlk, MachineBasicBlock *TailBlk,
+    MachineInstr &SrcPseudoInstr) const {
   // Make sure the last operand is EFLAGS, which gets clobbered by the branch
   // that was just emitted, but clearly shouldn't be "saved".
-  assert((MI.getNumOperands() <= 3 ||
-          !MI.getOperand(MI.getNumOperands() - 1).isReg() ||
-          MI.getOperand(MI.getNumOperands() - 1).getReg() == X86::EFLAGS) &&
+  assert((SrcPseudoInstr.getNumOperands() <= 3 ||
+          !SrcPseudoInstr.getOperand(SrcPseudoInstr.getNumOperands() - 1)
+               .isReg() ||
+          SrcPseudoInstr.getOperand(SrcPseudoInstr.getNumOperands() - 1)
+                  .getReg() == X86::EFLAGS) &&
          "Expected last argument to be EFLAGS");
-  unsigned MOVOpc = Subtarget.hasAVX() ? X86::VMOVAPSmr : X86::MOVAPSmr;
-  // In the XMM save block, save all the XMM argument registers.
-  for (int i = 3, e = MI.getNumOperands() - 1; i != e; ++i) {
-    int64_t Offset = (i - 3) * 16 + VarArgsFPOffset;
-    MachineMemOperand *MMO = F->getMachineMemOperand(
-        MachinePointerInfo::getFixedStack(*F, RegSaveFrameIndex, Offset),
-        MachineMemOperand::MOStore,
-        /*Size=*/16, Align(16));
-    BuildMI(XMMSaveMBB, DL, TII->get(MOVOpc))
-        .addFrameIndex(RegSaveFrameIndex)
-        .addImm(/*Scale=*/1)
-        .addReg(/*IndexReg=*/0)
-        .addImm(/*Disp=*/Offset)
-        .addReg(/*Segment=*/0)
-        .addReg(MI.getOperand(i).getReg())
-        .addMemOperand(MMO);
-  }
+
+  // create SAVE_VARARG_XMM_REGS pseudo
+  MachineInstrBuilder MIB =
+      BuildMI(GuardedRegsBlk, SrcPseudoInstr.getDebugLoc(),
+              Subtarget.getInstrInfo()->get(X86::SAVE_VARARG_XMM_REGS));
+
+  // set Frame Index
+  MIB.addImm(SrcPseudoInstr.getOperand(1).getImm());
+
+  // set ArgsOffset
+  MIB.addImm(SrcPseudoInstr.getOperand(2).getImm());
+
+  for (unsigned OpndIdx = 3, RegIdx = 0;
+       OpndIdx + 1 < SrcPseudoInstr.getNumOperands(); OpndIdx++, RegIdx++)
+    MIB.addReg(SrcPseudoInstr.getOperand(OpndIdx).getReg(),
+               RegState::InternalRead);
+
+  SrcPseudoInstr.eraseFromParent(); // The pseudo instruction is gone now.
 
-  MI.eraseFromParent(); // The pseudo instruction is gone now.
+MachineBasicBlock *X86TargetLowering::EmitVAStartSaveXMMRegsWithCustomInserter(
+    MachineInstr &PseudoVaStartInstr, MachineBasicBlock *EntryBlk) const {
+  // Emit code to save XMM registers to the stack. The ABI says that the
+  // number of registers to save is given in %al, so it's theoretically
+  // possible to do an indirect jump trick to avoid saving all of them,
+  // however this code takes a simpler approach and just executes all
+  // of the stores if %al is non-zero. It's less code, and it's probably
+  // easier on the hardware branch predictor, and stores aren't all that
+  // expensive anyway.
+
+  MachineBasicBlock *GuardedRegsBlk = nullptr;
+  MachineBasicBlock *TailBlk = nullptr;
+
+  std::tie(GuardedRegsBlk, TailBlk) =
+      createGuardedRegsBlock(EntryBlk, PseudoVaStartInstr, Subtarget);
+
+  AddSaveVarargXmmRegsPseudo(GuardedRegsBlk, TailBlk, PseudoVaStartInstr);
 
-  return EndMBB;
+  return TailBlk;
 }
 
 // The EFLAGS operand of SelectItr might be missing a kill marker
 // because there were multiple uses of EFLAGS, and ISel didn't know
 // which to mark. Figure out whether SelectItr should have had a
 // kill marker, and set it if it should. Returns the correct kill
 // marker value.
 static bool checkAndUpdateEFLAGSKill(MachineBasicBlock::iterator SelectItr,

llvm/lib/Target/X86/X86InstrCompiler.td

def ADJCALLSTACKUP64   : I<0, Pseudo, (outs), (ins i32imm:$amt1, i32imm:$amt2),
                            [(X86callseq_end timm:$amt1, timm:$amt2)]>,
                            Requires<[IsLP64]>;
 }
 def : Pat<(X86callseq_start timm:$amt1, timm:$amt2),
         (ADJCALLSTACKDOWN64 i32imm:$amt1, i32imm:$amt2, 0)>, Requires<[IsLP64]>;
 
 let SchedRW = [WriteSystem] in {
 
+let hasSideEffects = 1, mayStore = 1 in {
+def SAVE_VARARG_XMM_REGS : I<0, Pseudo,
+                           (outs),
+                           (ins i64imm:$regsavefi, i64imm:$offset,
+                             variable_ops),
+                          "#SAVE_VARARG_XMM_REGS $regsavefi, $offset",
+                          []>;
+}
+
 // x86-64 va_start lowering magic.
 let usesCustomInserter = 1, Defs = [EFLAGS] in {
 def VASTART_SAVE_XMM_REGS : I<0, Pseudo,
                               (outs),
                               (ins GR8:$al,
                                    i64imm:$regsavefi, i64imm:$offset,
                                    variable_ops),
                               "#VASTART_SAVE_XMM_REGS $al, $regsavefi, $offset",

llvm/test/CodeGen/X86/musttail-varargs.ll

 ; LINUX-NEXT:    subq $360, %rsp # imm = 0x168
 ; LINUX-NEXT:    .cfi_def_cfa_offset 416
 ; LINUX-NEXT:    .cfi_offset %rbx, -56
 ; LINUX-NEXT:    .cfi_offset %r12, -48
 ; LINUX-NEXT:    .cfi_offset %r13, -40
 ; LINUX-NEXT:    .cfi_offset %r14, -32
 ; LINUX-NEXT:    .cfi_offset %r15, -24
 ; LINUX-NEXT:    .cfi_offset %rbp, -16
+; LINUX-NEXT:    movaps %xmm7, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm6, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm5, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm4, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
+; LINUX-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; LINUX-NEXT:    movq %r9, %r15
 ; LINUX-NEXT:    movq %r8, %r12
 ; LINUX-NEXT:    movq %rcx, %r13
 ; LINUX-NEXT:    movq %rdx, %rbp
 ; LINUX-NEXT:    movq %rsi, %rbx
 ; LINUX-NEXT:    movq %rdi, %r14
+; LINUX-NEXT:    movq %rsi, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %rdx, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %rcx, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %r8, {{[0-9]+}}(%rsp)
+; LINUX-NEXT:    movq %r9, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movb %al, {{[-0-9]+}}(%r{{[sb]}}p) # 1-byte Spill
 ; LINUX-NEXT:    testb %al, %al
 ; LINUX-NEXT:    je .LBB0_2
 ; LINUX-NEXT:  # %bb.1:
 ; LINUX-NEXT:    movaps %xmm0, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm1, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm2, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm3, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm4, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm5, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:  .LBB0_2:
-; LINUX-NEXT:    movq %rbx, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %rbp, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %r13, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %r12, {{[0-9]+}}(%rsp)
-; LINUX-NEXT:    movq %r15, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
 ; LINUX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
 ; LINUX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movabsq $206158430216, %rax # imm = 0x3000000008
 ; LINUX-NEXT:    movq %rax, {{[0-9]+}}(%rsp)
 ; LINUX-NEXT:    movq %r14, %rdi
-; LINUX-NEXT:    movaps %xmm7, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm6, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm5, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm4, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm3, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm2, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm1, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; LINUX-NEXT:    movaps %xmm0, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; LINUX-NEXT:    callq get_f
 ; LINUX-NEXT:    movq %rax, %r11
 ; LINUX-NEXT:    movq %r14, %rdi
 ; LINUX-NEXT:    movq %rbx, %rsi
 ; LINUX-NEXT:    movq %rbp, %rdx
 ; LINUX-NEXT:    movq %r13, %rcx
 ; LINUX-NEXT:    movq %r12, %r8
 ; LINUX-NEXT:    movq %r15, %r9

 ; LINUX-X32-NEXT:    subl $344, %esp # imm = 0x158
 ; LINUX-X32-NEXT:    .cfi_def_cfa_offset 400
 ; LINUX-X32-NEXT:    .cfi_offset %rbx, -56
 ; LINUX-X32-NEXT:    .cfi_offset %r12, -48
 ; LINUX-X32-NEXT:    .cfi_offset %r13, -40
 ; LINUX-X32-NEXT:    .cfi_offset %r14, -32
 ; LINUX-X32-NEXT:    .cfi_offset %r15, -24
 ; LINUX-X32-NEXT:    .cfi_offset %rbp, -16
+; LINUX-X32-NEXT:    movaps %xmm7, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm6, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm5, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm4, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm3, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm2, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
+; LINUX-X32-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; LINUX-X32-NEXT:    movq %r9, %r15
 ; LINUX-X32-NEXT:    movq %r8, %r12
 ; LINUX-X32-NEXT:    movq %rcx, %r13
 ; LINUX-X32-NEXT:    movq %rdx, %rbp
 ; LINUX-X32-NEXT:    movq %rsi, %rbx
 ; LINUX-X32-NEXT:    movl %edi, %r14d
+; LINUX-X32-NEXT:    movq %rsi, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %rdx, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %rcx, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %r8, {{[0-9]+}}(%esp)
+; LINUX-X32-NEXT:    movq %r9, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movb %al, {{[-0-9]+}}(%e{{[sb]}}p) # 1-byte Spill
 ; LINUX-X32-NEXT:    testb %al, %al
 ; LINUX-X32-NEXT:    je .LBB0_2
 ; LINUX-X32-NEXT:  # %bb.1:
 ; LINUX-X32-NEXT:    movaps %xmm0, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm1, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm2, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm3, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm4, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm5, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm6, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movaps %xmm7, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:  .LBB0_2:
-; LINUX-X32-NEXT:    movq %rbx, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %rbp, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %r13, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %r12, {{[0-9]+}}(%esp)
-; LINUX-X32-NEXT:    movq %r15, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    leal {{[0-9]+}}(%rsp), %eax
 ; LINUX-X32-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    leal {{[0-9]+}}(%rsp), %eax
 ; LINUX-X32-NEXT:    movl %eax, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movabsq $206158430216, %rax # imm = 0x3000000008
 ; LINUX-X32-NEXT:    movq %rax, {{[0-9]+}}(%esp)
 ; LINUX-X32-NEXT:    movl %r14d, %edi
-; LINUX-X32-NEXT:    movaps %xmm7, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm6, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm5, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm4, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm3, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm2, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm1, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; LINUX-X32-NEXT:    movaps %xmm0, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; LINUX-X32-NEXT:    callq get_f
 ; LINUX-X32-NEXT:    movl %eax, %r11d
 ; LINUX-X32-NEXT:    movl %r14d, %edi
 ; LINUX-X32-NEXT:    movq %rbx, %rsi
 ; LINUX-X32-NEXT:    movq %rbp, %rdx
 ; LINUX-X32-NEXT:    movq %r13, %rcx
 ; LINUX-X32-NEXT:    movq %r12, %r8
 ; LINUX-X32-NEXT:    movq %r15, %r9

llvm/test/CodeGen/X86/vastart-defs-eflags.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc %s -o - | FileCheck %s
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.10.0"
 
 ; Check that vastart handling doesn't get between testb and je for the branch.
 define i32 @check_flag(i32 %flags, ...) nounwind {
 ; CHECK-LABEL: check_flag:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    subq $56, %rsp
+; CHECK-NEXT:    movq %rsi, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %r8, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movq %r9, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    testb %al, %al
 ; CHECK-NEXT:    je LBB0_2
 ; CHECK-NEXT:  ## %bb.1: ## %entry
 ; CHECK-NEXT:    movaps %xmm0, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm1, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm2, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm3, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm4, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm5, (%rsp)
 ; CHECK-NEXT:    movaps %xmm6, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movaps %xmm7, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:  LBB0_2: ## %entry
-; CHECK-NEXT:    movq %rsi, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %rdx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %rcx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %r8, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movq %r9, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    testl $512, %edi ## imm = 0x200
 ; CHECK-NEXT:    je LBB0_4
 ; CHECK-NEXT:  ## %bb.3: ## %if.then
 ; CHECK-NEXT:    leaq -{{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movq %rax, 16
 ; CHECK-NEXT:    leaq {{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movq %rax, 8

llvm/test/CodeGen/X86/x32-va_start.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-linux-gnux32 | FileCheck %s -check-prefix=CHECK -check-prefix=SSE
 ; RUN: llc < %s -mtriple=x86_64-linux-gnux32 -mattr=-sse | FileCheck %s -check-prefix=CHECK -check-prefix=NOSSE
+; RUN: llc < %s -mtriple=i386-linux-gnux32 | FileCheck %s -check-prefix=CHECK -check-prefix=32BITABI
+; RUN: llc < %s -mtriple=i686-linux-gnux32 | FileCheck %s -check-prefix=CHECK -check-prefix=32BITABI
 ;
 ; Verifies that x32 va_start lowering is sane. To regenerate this test, use
 ; cat <<EOF |
 ; #include <stdarg.h>
 ;
 ; int foo(float a, const char* fmt, ...) {
 ;   va_list ap;
 ;   va_start(ap, fmt);
 ;   int value = va_arg(ap, int);
 ;   va_end(ap);
 ;   return value;
 ; }
 ; EOF
 ; build/bin/clang -mx32 -O3 -o- -S -emit-llvm -xc -
 ;
 target datalayout = "e-m:e-p:32:32-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnux32"
 
 %struct.__va_list_tag = type { i32, i32, i8*, i8* }
 
 define i32 @foo(float %a, i8* nocapture readnone %fmt, ...) nounwind {
+; SSE-LABEL: foo:
+; SSE:       # %bb.0: # %entry
+; SSE-NEXT:    subl $72, %esp
+; SSE-NEXT:    movq %rsi, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movq %rdx, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movq %rcx, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movq %r8, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movq %r9, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    testb %al, %al
+; SSE-NEXT:    je .LBB0_2
+; SSE-NEXT:  # %bb.1: # %entry
+; SSE-NEXT:    movaps %xmm1, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movaps %xmm2, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movaps %xmm3, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movaps %xmm4, (%esp)
+; SSE-NEXT:    movaps %xmm5, {{[0-9]+}}(%esp)
+; SSE-NEXT:    movaps %xmm6, {{[0-9]+}}(%esp)
+; SSE-NEXT:    movaps %xmm7, {{[0-9]+}}(%esp)
+; SSE-NEXT:  .LBB0_2: # %entry
+; SSE-NEXT:    leal -{{[0-9]+}}(%rsp), %eax
+; SSE-NEXT:    movl %eax, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    leal {{[0-9]+}}(%rsp), %eax
+; SSE-NEXT:    movl %eax, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movabsq $274877906952, %rax # imm = 0x4000000008
+; SSE-NEXT:    movq %rax, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    movl $8, %ecx
+; SSE-NEXT:    cmpl $40, %ecx
+; SSE-NEXT:    ja .LBB0_4
+; SSE-NEXT:  # %bb.3: # %vaarg.in_reg
+; SSE-NEXT:    movl -{{[0-9]+}}(%esp), %eax
+; SSE-NEXT:    addl %ecx, %eax
+; SSE-NEXT:    addl $8, %ecx
+; SSE-NEXT:    movl %ecx, -{{[0-9]+}}(%esp)
+; SSE-NEXT:    jmp .LBB0_5
+; SSE-NEXT:  .LBB0_4: # %vaarg.in_mem
+; SSE-NEXT:    movl -{{[0-9]+}}(%esp), %eax
+; SSE-NEXT:    leal 8(%rax), %ecx
+; SSE-NEXT:    movl %ecx, -{{[0-9]+}}(%esp)
+; SSE-NEXT:  .LBB0_5: # %vaarg.end
+; SSE-NEXT:    movl (%eax), %eax
+; SSE-NEXT:    addl $72, %esp
+; SSE-NEXT:    retq
+;
+; NOSSE-LABEL: foo:
+; NOSSE:       # %bb.0: # %entry
+; NOSSE-NEXT:    movq %rsi, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movq %rdx, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movq %rcx, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movq %r8, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movq %r9, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    leal -{{[0-9]+}}(%rsp), %eax
+; NOSSE-NEXT:    movl %eax, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    leal {{[0-9]+}}(%rsp), %eax
+; NOSSE-NEXT:    movl %eax, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movabsq $206158430216, %rax # imm = 0x3000000008
+; NOSSE-NEXT:    movq %rax, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movl $8, %ecx
+; NOSSE-NEXT:    cmpl $40, %ecx
+; NOSSE-NEXT:    ja .LBB0_2
+; NOSSE-NEXT:  # %bb.1: # %vaarg.in_reg
+; NOSSE-NEXT:    movl -{{[0-9]+}}(%esp), %eax
+; NOSSE-NEXT:    addl %ecx, %eax
+; NOSSE-NEXT:    addl $8, %ecx
+; NOSSE-NEXT:    movl %ecx, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movl (%eax), %eax
+; NOSSE-NEXT:    retq
+; NOSSE-NEXT:  .LBB0_2: # %vaarg.in_mem
+; NOSSE-NEXT:    movl -{{[0-9]+}}(%esp), %eax
+; NOSSE-NEXT:    leal 8(%rax), %ecx
+; NOSSE-NEXT:    movl %ecx, -{{[0-9]+}}(%esp)
+; NOSSE-NEXT:    movl (%eax), %eax
+; NOSSE-NEXT:    retq
+;
+; 32BITABI-LABEL: foo:
+; 32BITABI:       # %bb.0: # %entry
+; 32BITABI-NEXT:    subl $28, %esp
+; 32BITABI-NEXT:    leal {{[0-9]+}}(%esp), %ecx
+; 32BITABI-NEXT:    movl %ecx, (%esp)
+; 32BITABI-NEXT:    cmpl $40, %ecx
+; 32BITABI-NEXT:    ja .LBB0_2
+; 32BITABI-NEXT:  # %bb.1: # %vaarg.in_reg
+; 32BITABI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; 32BITABI-NEXT:    addl %ecx, %eax
+; 32BITABI-NEXT:    addl $8, %ecx
+; 32BITABI-NEXT:    movl %ecx, (%esp)
+; 32BITABI-NEXT:    jmp .LBB0_3
+; 32BITABI-NEXT:  .LBB0_2: # %vaarg.in_mem
+; 32BITABI-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; 32BITABI-NEXT:    leal 8(%eax), %ecx
+; 32BITABI-NEXT:    movl %ecx, {{[0-9]+}}(%esp)
+; 32BITABI-NEXT:  .LBB0_3: # %vaarg.end
+; 32BITABI-NEXT:    movl (%eax), %eax
+; 32BITABI-NEXT:    addl $28, %esp
+; 32BITABI-NEXT:    retl
 entry:
   %ap = alloca [1 x %struct.__va_list_tag], align 16
   %0 = bitcast [1 x %struct.__va_list_tag]* %ap to i8*
   call void @llvm.lifetime.start.p0i8(i64 16, i8* %0) #2
   call void @llvm.va_start(i8* %0)
-; SSE: subl $72, %esp
-; SSE: testb %al, %al
-; SSE: je .[[NOFP:.*]]
-; SSE-DAG: movaps %xmm1
-; SSE-DAG: movaps %xmm2
-; SSE-DAG: movaps %xmm3
-; SSE-DAG: movaps %xmm4
-; SSE-DAG: movaps %xmm5
-; SSE-DAG: movaps %xmm6
-; SSE-DAG: movaps %xmm7
-; NOSSE-NOT: xmm
-; SSE: .[[NOFP]]:
-; CHECK-DAG: movq %r9
-; CHECK-DAG: movq %r8
-; CHECK-DAG: movq %rcx
-; CHECK-DAG: movq %rdx
-; CHECK-DAG: movq %rsi
   %gp_offset_p = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %ap, i32 0, i32 0, i32 0
   %gp_offset = load i32, i32* %gp_offset_p, align 16
   %fits_in_gp = icmp ult i32 %gp_offset, 41
   br i1 %fits_in_gp, label %vaarg.in_reg, label %vaarg.in_mem
-; CHECK: cmpl $40, [[COUNT:.*]]
-; CHECK: ja .[[IN_MEM:.*]]
 
 vaarg.in_reg:                                     ; preds = %entry
   %1 = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %ap, i32 0, i32 0, i32 3
   %reg_save_area = load i8*, i8** %1, align 4
   %2 = getelementptr i8, i8* %reg_save_area, i32 %gp_offset
   %3 = add i32 %gp_offset, 8
   store i32 %3, i32* %gp_offset_p, align 16
   br label %vaarg.end
-; CHECK: movl {{[^,]*}}, [[ADDR:.*]]
-; CHECK: addl [[COUNT]], [[ADDR]]
-; SSE: jmp .[[END:.*]]
-; NOSSE: movl ([[ADDR]]), %eax
-; NOSSE: retq
-; CHECK: .[[IN_MEM]]:
 vaarg.in_mem:                                     ; preds = %entry
   %overflow_arg_area_p = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %ap, i32 0, i32 0, i32 2
   %overflow_arg_area = load i8*, i8** %overflow_arg_area_p, align 8
   %overflow_arg_area.next = getelementptr i8, i8* %overflow_arg_area, i32 8
   store i8* %overflow_arg_area.next, i8** %overflow_arg_area_p, align 8
   br label %vaarg.end
-; CHECK: movl {{[^,]*}}, [[ADDR]]
-; NOSSE: movl ([[ADDR]]), %eax
-; NOSSE: retq
-; SSE: .[[END]]:
 
 vaarg.end:                                        ; preds = %vaarg.in_mem, %vaarg.in_reg
   %vaarg.addr.in = phi i8* [ %2, %vaarg.in_reg ], [ %overflow_arg_area, %vaarg.in_mem ]
   %vaarg.addr = bitcast i8* %vaarg.addr.in to i32*
   %4 = load i32, i32* %vaarg.addr, align 4
   call void @llvm.va_end(i8* %0)
   call void @llvm.lifetime.end.p0i8(i64 16, i8* %0) #2
   ret i32 %4
-; SSE: movl ([[ADDR]]), %eax
-; SSE: retq
 }
 
 ; Function Attrs: nounwind argmemonly
 declare void @llvm.lifetime.start.p0i8(i64, i8* nocapture) nounwind
 
 ; Function Attrs: nounwind
 declare void @llvm.va_start(i8*) nounwind
 
 ; Function Attrs: nounwind
 declare void @llvm.va_end(i8*) nounwind
 
 ; Function Attrs: nounwind argmemonly
 declare void @llvm.lifetime.end.p0i8(i64, i8* nocapture) nounwind
 

llvm/test/CodeGen/X86/xmm-vararg-noopt.ll

+; RUN: llc -O0 -mtriple=x86_64-unknown-unknown < %s | FileCheck %s
+
+; CHECK-LABEL: testvarargs
+; Ensure that xmm registers are not used before testing %al
+; CHECK-NOT: xmm
+; CHECK: testb %al, %al
+; CHECK-NOT: xmm
+; CHECK: # %bb.1
+; CHECK-NEXT: %xmm0, {{.*}}%rsp
+; CHECK-NEXT: %xmm1, {{.*}}%rsp
+; CHECK-NEXT: %xmm2, {{.*}}%rsp
+; CHECK-NEXT: %xmm3, {{.*}}%rsp
+; CHECK-NEXT: %xmm4, {{.*}}%rsp
+; CHECK-NEXT: %xmm5, {{.*}}%rsp
+; CHECK-NEXT: %xmm6, {{.*}}%rsp
+; CHECK-NEXT: %xmm7, {{.*}}%rsp
+
+; ModuleID = 'variadic.c'
+source_filename = "variadic.c"
+target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux"
+
+%struct.__va_list_tag = type { i32, i32, i8*, i8* }
+
+@.str = private unnamed_addr constant [9 x i8] c"\0A hello \00", align 1
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @testvarargs(i8* %fmt, ...) {
+entry:
+  %fmt.addr = alloca i8*, align 8
+  %va = alloca [1 x %struct.__va_list_tag], align 16
+  store i8* %fmt, i8** %fmt.addr, align 8
+  %arraydecay = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %va, i64 0, i64 0
+  %arraydecay1 = bitcast %struct.__va_list_tag* %arraydecay to i8*
+  call void @llvm.va_start(i8* %arraydecay1)
+  %arraydecay2 = getelementptr inbounds [1 x %struct.__va_list_tag], [1 x %struct.__va_list_tag]* %va, i64 0, i64 0
+  %arraydecay23 = bitcast %struct.__va_list_tag* %arraydecay2 to i8*
+  call void @llvm.va_end(i8* %arraydecay23)
+  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([9 x i8], [9 x i8]* @.str, i64 0, i64 0))
+  ret void
+}
+
+; Function Attrs: nounwind
+declare void @llvm.va_start(i8*)
+
+; Function Attrs: nounwind
+declare void @llvm.va_end(i8*)
+
+declare dso_local i32 @printf(i8*, ...)