[x86/SLH] Significantly optimize the interprocedural hardening strategy.

[chandlerc]

Summary

Previously, SLH would pedantically thread the predicate state in and out
of the stack pointer around every single call. This is unnecessary, and
there are vague indications it is fairly expensive.

Instead, observe a few things:

1. We can be lazy about restoring the predicate state from the stack pointer until we need it to do some hardening.
2. Once we have put the predicate state for a basic block into the stack pointer, that predicate state never changes due to code *in* that basic block. All subsequent updates within the basic block's execution occur inside of called functions and are communicated back to the block *in the stack pointer*. So once we first merge our predicate state into the stack pointer in a basic block, the most current state remains in the stack pointer for the rest of the block. We can completely skip subsequent merges into the stack pointer.
3. If we already have the predicate state in the stack pointer when we see a return, that may be sufficient to harden the return. This is still being checked, so is currently disabled by a flag.

I'm hopeful this will somewhat help with the instruction density
overhead of SLH, and significantly help benchmarks and applications
where there are tight long chains of calls. I haven't yet gotten
performance data with this optimization in place, but once I do I will
update all of the documentation.

[chandlerc]

Before anyone gets too excited about the performance improvements here, I should mention that I am working on implementing another component of the high level SLH design that I'm worried will completely defeat all of these improvements. Maybe even hold off reviewing, as this whole thing may not be relevant. =/ Sorry for the noise.

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

 
 static cl::opt<bool> HardenIndirectCallsAndJumps(
     PASS_KEY "-indirect",
     cl::desc("Harden indirect calls and jumps against using speculatively "
              "stored attacker controlled addresses. This is designed to "
              "mitigate Spectre v1.2 style attacks."),
     cl::init(true), cl::Hidden);
 
+static cl::opt<bool> HardenReturnsNearby(
+    PASS_KEY "-returns-nearby",
+    cl::desc("Force hardening RSP nearby to the RET instruction. Relying an "
+             "earlier hardening of RSP is appealing but hasn't had its "
+             "effectiveness vetted."),
+    cl::init(true), cl::Hidden);
+
 namespace llvm {
 
 void initializeX86SpeculativeLoadHardeningPassPass(PassRegistry &);
 
 } // end namespace llvm
 
 namespace {
 
void restoreEFLAGS(MachineBasicBlock &MBB,
                      unsigned OFReg);
 
   void mergePredStateIntoSP(MachineBasicBlock &MBB,
                             MachineBasicBlock::iterator InsertPt, DebugLoc Loc,
                             unsigned PredStateReg);
   unsigned extractPredStateFromSP(MachineBasicBlock &MBB,
                                   MachineBasicBlock::iterator InsertPt,
                                   DebugLoc Loc);
+  unsigned getCurrentPredState(MachineBasicBlock &MBB,
+                               MachineBasicBlock::iterator InsertPt,
+                               DebugLoc Loc, bool &IsPredStateSSACurrent);
 
   void
   hardenLoadAddr(MachineInstr &MI, MachineOperand &BaseMO,
-                 MachineOperand &IndexMO,
+                 MachineOperand &IndexMO, bool &IsPredStateSSACurrent,
                  SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg);
   MachineInstr *
   sinkPostLoadHardenedInst(MachineInstr &MI,
                            SmallPtrSetImpl<MachineInstr *> &HardenedInstrs);
   bool canHardenRegister(unsigned Reg);
   unsigned hardenValueInRegister(unsigned Reg, MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator InsertPt,
-                                 DebugLoc Loc);
-  unsigned hardenPostLoad(MachineInstr &MI);
-  void hardenReturnInstr(MachineInstr &MI);
+                                 DebugLoc Loc, bool &IsPredStateSSACurrent);
+  unsigned hardenPostLoad(MachineInstr &MI, bool &IsPredStateSSACurrent);
+  void hardenReturnInstr(MachineInstr &MI, bool IsPredStateInSP);
   void hardenIndirectCallOrJumpInstr(
-      MachineInstr &MI,
+      MachineInstr &MI, bool &IsPredStateSSACurrent,
       SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg);
 };
 
 } // end anonymous namespace
 
 char X86SpeculativeLoadHardeningPass::ID = 0;
 
 void X86SpeculativeLoadHardeningPass::getAnalysisUsage(
if (HardenLoads)
         if (IndexReg)
           HardenedAddrRegs.insert(IndexReg);
 
         for (MachineOperand &Def : MI.defs())
           if (Def.isReg())
             LoadDepRegs.set(Def.getReg());
       }
 
+    // While actually hardening in this block we also trace the predicate state
+    // in and out of functions being called. However, within a basic block there
+    // will never be *local* updates to the predicate state. So once we merge
+    // the predicate state into the stack pointer, the stack pointer will
+    // always have the latest predicate state for this function. We can also
+    // delay extracting the predicate state from the stack pointer until we need
+    // it for some hardening operation. We use two flags to coordinate this. The
+    // first tracks whether we have the current predicate state in the stack
+    // pointer, and therefore no longer need to merge it. The second tracks
+    // whether the predicate state in the SSA updater is current, or if it needs
+    // to be freshly extracted from the stack pointer.
+    bool IsPredStateInSP = false;
+    bool IsPredStateSSACurrent = true;
+
+    // Lastly, we remember the last call instruction (if any) where the
+    // predicate state ceased to be current in the SSA updater. We use this to
+    // figure out where to extract the current predicate state if needed for
+    // successors of the block.
+    MachineInstr *LastCall;
+
     // Now re-walk the instructions in the basic block, and apply whichever
     // hardening strategy we have elected. Note that we do this in a second
     // pass specifically so that we have the complete set of instructions for
     // which we will do post-load hardening and can defer it in certain
     // circumstances.
     //
     // FIXME: This could probably be made even more effective by doing it
     // across the entire function. Rather than just walking the flat list
for (MachineInstr &MI : MBB) {
           assert(MemRefBeginIdx >= 0 && "Cannot have an invalid index here!");
 
           MemRefBeginIdx += X86II::getOperandBias(Desc);
 
           MachineOperand &BaseMO =
               MI.getOperand(MemRefBeginIdx + X86::AddrBaseReg);
           MachineOperand &IndexMO =
               MI.getOperand(MemRefBeginIdx + X86::AddrIndexReg);
-          hardenLoadAddr(MI, BaseMO, IndexMO, AddrRegToHardenedReg);
+          hardenLoadAddr(MI, BaseMO, IndexMO, IsPredStateSSACurrent, AddrRegToHardenedReg);
           continue;
         }
 
         // Test if this instruction is one of our post load instructions (and
         // remove it from the set if so).
         if (HardenPostLoad.erase(&MI)) {
           assert(!MI.isCall() && "Must not try to post-load harden a call!");
 
for (MachineInstr &MI : MBB) {
                 continue;
 
               // Otherwise, add this to the set of defs we harden.
               HardenPostLoad.insert(SunkMI);
               continue;
             }
           }
 
-          unsigned HardenedReg = hardenPostLoad(MI);
+          unsigned HardenedReg = hardenPostLoad(MI, IsPredStateSSACurrent);
 
           // Mark the resulting hardened register as such so we don't re-harden.
           AddrRegToHardenedReg[HardenedReg] = HardenedReg;
 
           continue;
         }
 
         // Check for an indirect call or branch that may need its input hardened
         // even if we couldn't find the specific load used, or were able to
         // avoid hardening it for some reason. Note that here we cannot break
         // out afterward as we may still need to handle any call aspect of this
         // instruction.
         if ((MI.isCall() || MI.isBranch()) && HardenIndirectCallsAndJumps)
-          hardenIndirectCallOrJumpInstr(MI, AddrRegToHardenedReg);
+          hardenIndirectCallOrJumpInstr(MI, IsPredStateSSACurrent, AddrRegToHardenedReg);
       }
 
       // After we finish hardening loads we handle interprocedural hardening if
       // enabled and relevant for this instruction.
       if (!HardenInterprocedurally)
         continue;
       if (!MI.isCall() && !MI.isReturn())
         continue;
 
       // If this is a direct return (IE, not a tail call) just directly harden
-      // it.
+      // it and then we're done with the block.
       if (MI.isReturn() && !MI.isCall()) {
-        hardenReturnInstr(MI);
-        continue;
+        hardenReturnInstr(MI, IsPredStateInSP);
+        break;
       }
 
       // Otherwise we have a call. We need to handle transferring the predicate
       // state into a call and recovering it after the call returns unless this
       // is a tail call.
       assert(MI.isCall() && "Should only reach here for calls!");
       auto InsertPt = MI.getIterator();
       DebugLoc Loc = MI.getDebugLoc();
 
       // First, we transfer the predicate state into the called function by
-      // merging it into the stack pointer. This will kill the current def of
-      // the state.
-      unsigned StateReg = PS->SSA.GetValueAtEndOfBlock(&MBB);
-      mergePredStateIntoSP(MBB, InsertPt, Loc, StateReg);
+      // merging it into the stack pointer if it is not already there.
+      if (!IsPredStateInSP) {
+        assert(IsPredStateSSACurrent &&
+               "PredState must be in SSA until it enters the SP!");
+        mergePredStateIntoSP(MBB, InsertPt, Loc,
+                             PS->SSA.GetValueAtEndOfBlock(&MBB));
+        IsPredStateInSP = true;
+      }
 
       // If this call is also a return, it is a tail call and we don't need
-      // anything else to handle it so just continue.
+      // anything else to handle it, and we're done with the block.
       if (MI.isReturn())
-        continue;
+        break;
 
-      // We need to step past the call and recover the predicate
-      // state from SP after the return, and make this new state available.
-      ++InsertPt;
-      unsigned NewStateReg = extractPredStateFromSP(MBB, InsertPt, Loc);
-      PS->SSA.AddAvailableValue(&MBB, NewStateReg);
+      // Mark that the predicate state in the SSA updater is no longer current
+      // so that we will extract it from the stack pointer if it is needed, and
+      // this instruction as the last safe place to extract fresh predicate
+      // state if needed.
+      IsPredStateSSACurrent = false;
+      LastCall = &MI;
+    }
+
+    // If we end the block without the current predicate state in the SSA
+    // updater and there are any successors to the block, we need to forcibly
+    // extract it and add it to the SSA updater. Otherwise we won't correctly
+    // PHI together the virtual registers for it along interesting CFGs.
+    if (!IsPredStateSSACurrent && !MBB.succ_empty()) {
+      unsigned StateReg = extractPredStateFromSP(
+          MBB, std::next(LastCall->getIterator()), LastCall->getDebugLoc());
+      PS->SSA.AddAvailableValue(&MBB, StateReg);
     }
 
     HardenPostLoad.clear();
     HardenLoadAddr.clear();
     HardenedAddrRegs.clear();
     AddrRegToHardenedReg.clear();
 
     // Currently, we only track data-dependent loads within a basic block.
auto ShiftI =
           .addReg(TmpReg, RegState::Kill)
           .addImm(TRI->getRegSizeInBits(*PS->RC) - 1);
   ShiftI->addRegisterDead(X86::EFLAGS, TRI);
   ++NumInstsInserted;
 
   return PredStateReg;
 }
 
+/// Gets the current predicate state.
+///
+/// This will either return the predicate state in the SSA updater if current,
+/// or it will extract fresh predicate state from the stack pointer, make the
+/// SSA updater current with that and return it for immediate use.
+unsigned X86SpeculativeLoadHardeningPass::getCurrentPredState(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator InsertPt, DebugLoc Loc,
+    bool &IsPredStateSSACurrent) {
+  // If the SSA updater already has the current value, just return it.
+  if (IsPredStateSSACurrent)
+    return PS->SSA.GetValueAtEndOfBlock(&MBB);
+
+  // Otherwise we need to compute a fresh value from the stack pointer.
+  unsigned PredStateReg = extractPredStateFromSP(MBB, InsertPt, Loc);
+
+  // Tell the SSA updater about this now current value and mark that it is in
+  // fact now current.
+  PS->SSA.AddAvailableValue(&MBB, PredStateReg);
+  IsPredStateSSACurrent = true;
+
+  return PredStateReg;
+}
+
 void X86SpeculativeLoadHardeningPass::hardenLoadAddr(
     MachineInstr &MI, MachineOperand &BaseMO, MachineOperand &IndexMO,
+    bool &IsPredStateSSACurrent,
     SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg) {
   MachineBasicBlock &MBB = *MI.getParent();
   DebugLoc Loc = MI.getDebugLoc();
 
   // Check if EFLAGS are alive by seeing if there is a def of them or they
   // live-in, and then seeing if that def is in turn used.
   bool EFLAGSLive = isEFLAGSLive(MBB, MI.getIterator(), *TRI);
 
llvm::erase_if(HardenOpRegs, [&](MachineOperand *Op) {
     // Otherwise, we can directly update this operand and remove it.
     Op->setReg(It->second);
     return true;
   });
   // If there are none left, we're done.
   if (HardenOpRegs.empty())
     return;
 
-  // Compute the current predicate state.
-  unsigned StateReg = PS->SSA.GetValueAtEndOfBlock(&MBB);
-
   auto InsertPt = MI.getIterator();
 
   // If EFLAGS are live and we don't have access to instructions that avoid
   // clobbering EFLAGS we need to save and restore them. This in turn makes
   // the EFLAGS no longer live.
   unsigned FlagsReg = 0;
   if (EFLAGSLive && !Subtarget->hasBMI2()) {
     EFLAGSLive = false;
     FlagsReg = saveEFLAGS(MBB, InsertPt, Loc);
   }
 
+  // Compute the current predicate state. This may emit instructions that
+  // clobber EFLAGS!
+  unsigned StateReg =
+      getCurrentPredState(MBB, InsertPt, Loc, IsPredStateSSACurrent);
+
   for (MachineOperand *Op : HardenOpRegs) {
     unsigned OpReg = Op->getReg();
     auto *OpRC = MRI->getRegClass(OpReg);
     unsigned TmpReg = MRI->createVirtualRegister(OpRC);
 
     // If this is a vector register, we'll need somewhat custom logic to handle
     // hardening it.
     if (!Subtarget->hasVLX() && (OpRC->hasSuperClassEq(&X86::VR128RegClass) ||

 /// `Reg` must be a virtual register. Currently, it is required to be a GPR no
 /// larger than the predicate state register. FIXME: We should support vector
 /// registers here by broadcasting the predicate state.
 ///
 /// The new, hardened virtual register is returned. It will have the same
 /// register class as `Reg`.
 unsigned X86SpeculativeLoadHardeningPass::hardenValueInRegister(
     unsigned Reg, MachineBasicBlock &MBB, MachineBasicBlock::iterator InsertPt,
-    DebugLoc Loc) {
+    DebugLoc Loc, bool &IsPredStateSSACurrent) {
   assert(canHardenRegister(Reg) && "Cannot harden this register!");
   assert(TRI->isVirtualRegister(Reg) && "Cannot harden a physical register!");
 
+  unsigned FlagsReg = 0;
+  if (isEFLAGSLive(MBB, InsertPt, *TRI))
+    FlagsReg = saveEFLAGS(MBB, InsertPt, Loc);
+
+  // Compute the current predicate state. This may emit instructions that
+  // clobber EFLAGS!
+  unsigned StateReg =
+      getCurrentPredState(MBB, InsertPt, Loc, IsPredStateSSACurrent);
+
   auto *RC = MRI->getRegClass(Reg);
   int Bytes = TRI->getRegSizeInBits(*RC) / 8;
 
-  unsigned StateReg = PS->SSA.GetValueAtEndOfBlock(&MBB);
-
   // FIXME: Need to teach this about 32-bit mode.
   if (Bytes != 8) {
     unsigned SubRegImms[] = {X86::sub_8bit, X86::sub_16bit, X86::sub_32bit};
     unsigned SubRegImm = SubRegImms[Log2_32(Bytes)];
     unsigned NarrowStateReg = MRI->createVirtualRegister(RC);
     BuildMI(MBB, InsertPt, Loc, TII->get(TargetOpcode::COPY), NarrowStateReg)
         .addReg(StateReg, 0, SubRegImm);
     StateReg = NarrowStateReg;
   }
 
-  unsigned FlagsReg = 0;
-  if (isEFLAGSLive(MBB, InsertPt, *TRI))
-    FlagsReg = saveEFLAGS(MBB, InsertPt, Loc);
-
   unsigned NewReg = MRI->createVirtualRegister(RC);
   unsigned OrOpCodes[] = {X86::OR8rr, X86::OR16rr, X86::OR32rr, X86::OR64rr};
   unsigned OrOpCode = OrOpCodes[Log2_32(Bytes)];
   auto OrI = BuildMI(MBB, InsertPt, Loc, TII->get(OrOpCode), NewReg)
                  .addReg(StateReg)
                  .addReg(Reg);
   OrI->addRegisterDead(X86::EFLAGS, TRI);
   ++NumInstsInserted;

 ///
 /// We can harden a non-leaking load into a register without touching the
 /// address by just hiding all of the loaded bits during misspeculation. We use
 /// an `or` instruction to do this because we set up our poison value as all
 /// ones. And the goal is just for the loaded bits to not be exposed to
 /// execution and coercing them to one is sufficient.
 ///
 /// Returns the newly hardened register.
-unsigned X86SpeculativeLoadHardeningPass::hardenPostLoad(MachineInstr &MI) {
+unsigned
+X86SpeculativeLoadHardeningPass::hardenPostLoad(MachineInstr &MI,
+                                                bool &IsPredStateSSACurrent) {
   MachineBasicBlock &MBB = *MI.getParent();
   DebugLoc Loc = MI.getDebugLoc();
 
   auto &DefOp = MI.getOperand(0);
   unsigned OldDefReg = DefOp.getReg();
   auto *DefRC = MRI->getRegClass(OldDefReg);
 
   // Because we want to completely replace the uses of this def'ed value with
   // the hardened value, create a dedicated new register that will only be used
   // to communicate the unhardened value to the hardening.
   unsigned UnhardenedReg = MRI->createVirtualRegister(DefRC);
   DefOp.setReg(UnhardenedReg);
 
   // Now harden this register's value, getting a hardened reg that is safe to
   // use. Note that we insert the instructions to compute this *after* the
   // defining instruction, not before it.
-  unsigned HardenedReg = hardenValueInRegister(
-      UnhardenedReg, MBB, std::next(MI.getIterator()), Loc);
+  unsigned HardenedReg =
+      hardenValueInRegister(UnhardenedReg, MBB, std::next(MI.getIterator()),
+                            Loc, IsPredStateSSACurrent);
 
   // Finally, replace the old register (which now only has the uses of the
   // original def) with the hardened register.
   MRI->replaceRegWith(/*FromReg*/ OldDefReg, /*ToReg*/ HardenedReg);
 
   ++NumPostLoadRegsHardened;
   return HardenedReg;
 }

 /// benefit: it allows us to pass the predicate state accumulated in this
 /// function back to the caller. In the absence of a BCBS attack on the return,
 /// the caller will typically be resumed and speculatively executed due to the
 /// Return Stack Buffer (RSB) prediction which is very accurate and has a high
 /// priority. It is possible that some code from the caller will be executed
 /// speculatively even during a BCBS-attacked return until the steering takes
 /// effect. Whenever this happens, the caller can recover the (poisoned)
 /// predicate state from the stack pointer and continue to harden loads.
-void X86SpeculativeLoadHardeningPass::hardenReturnInstr(MachineInstr &MI) {
+void X86SpeculativeLoadHardeningPass::hardenReturnInstr(MachineInstr &MI,
+                                                        bool IsPredStateInSP) {
   MachineBasicBlock &MBB = *MI.getParent();
   DebugLoc Loc = MI.getDebugLoc();
   auto InsertPt = MI.getIterator();
 
   if (FenceCallAndRet) {
     // Simply forcibly block speculation of loads out of the function by using
     // an LFENCE. This is potentially a heavy-weight mitigation strategy, but
     // should be secure, is simple from an ABI perspective, and the cost can be
     // minimized through inlining.
     //
     // FIXME: We should investigate ways to establish a strong data-dependency
     // on the return. However, poisoning the stack pointer is unlikely to work
     // because the return is *predicted* rather than relying on the load of the
     // return address to actually resolve.
     BuildMI(MBB, InsertPt, Loc, TII->get(X86::LFENCE));
     ++NumInstsInserted;
     ++NumLFENCEsInserted;
     return;
   }
 
+  // If we already have the predicate state in RSP, nothing to do here.
+  // FIXME: This is currently disabled by a flag until confirmed that using an
+  // earlier tagged stack pointer remains sufficient to mitigate BCBS attacks on
+  // the return address.
+  if (!HardenReturnsNearby && IsPredStateInSP)
+    return;
+
   // Take our predicate state, shift it to the high 17 bits (so that we keep
   // pointers canonical) and merge it into RSP. This will allow the caller to
   // extract it when we return (speculatively).
   mergePredStateIntoSP(MBB, InsertPt, Loc, PS->SSA.GetValueAtEndOfBlock(&MBB));
 }
 
 /// An attacker may speculatively store over a value that is then speculatively
 /// loaded and used as the target of an indirect call or jump instruction. This
 /// is called Spectre v1.2 or Bounds Check Bypass Store (BCBS) and is described
 /// in this paper:
 /// https://people.csail.mit.edu/vlk/spectre11.pdf
 ///
 /// When this happens, the speculative execution of the call or jump will end up
 /// being steered to this attacker controlled address. While most such loads
 /// will be adequately hardened already, we want to ensure that they are
 /// definitively treated as needing post-load hardening. While address hardening
 /// is sufficient to prevent secret data from leaking to the attacker, it may
 /// not be sufficient to prevent an attacker from steering speculative
 /// execution. We forcibly unfolded all relevant loads above and so will always
 /// have an opportunity to post-load harden here, we just need to scan for cases
 /// not already flagged and add them.
 void X86SpeculativeLoadHardeningPass::hardenIndirectCallOrJumpInstr(
-    MachineInstr &MI,
+    MachineInstr &MI, bool &IsPredStateSSACurrent,
     SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg) {
   switch (MI.getOpcode()) {
   case X86::FARCALL16m:
   case X86::FARCALL32m:
   case X86::FARCALL64:
   case X86::FARJMP16m:
   case X86::FARJMP32m:
   case X86::FARJMP64:
void X86SpeculativeLoadHardeningPass::hardenIndirectCallOrJumpInstr(
   // If we don't have a hardened register yet, compute one. Otherwise, just use
   // the already hardened register.
   //
   // FIXME: It is a little suspect that we use partially hardened registers that
   // only feed addresses. The complexity of partial hardening with SHRX
   // continues to pile up. Should definitively measure its value and consider
   // eliminating it.
   if (!HardenedTargetReg)
-    HardenedTargetReg = hardenValueInRegister(
-        OldTargetReg, *MI.getParent(), MI.getIterator(), MI.getDebugLoc());
+    HardenedTargetReg =
+        hardenValueInRegister(OldTargetReg, *MI.getParent(), MI.getIterator(),
+                              MI.getDebugLoc(), IsPredStateSSACurrent);
 
   // Set the target operand to the hardened register.
   TargetOp.setReg(HardenedTargetReg);
 
   ++NumCallsOrJumpsHardened;
 }
 
 INITIALIZE_PASS_BEGIN(X86SpeculativeLoadHardeningPass, DEBUG_TYPE,
                       "X86 speculative load hardener", false, false)
 INITIALIZE_PASS_END(X86SpeculativeLoadHardeningPass, DEBUG_TYPE,
                     "X86 speculative load hardener", false, false)
 
 FunctionPass *llvm::createX86SpeculativeLoadHardeningPass() {
   return new X86SpeculativeLoadHardeningPass();
 }

llvm/test/CodeGen/X86/speculative-load-hardening-indirect.ll

@global_blockaddrs = constant [4 x i8*] [
   i8* blockaddress(@test_indirectbr_global, %bb1),
   i8* blockaddress(@test_indirectbr_global, %bb2),
   i8* blockaddress(@test_indirectbr_global, %bb3)
 ]
 
 define i32 @test_indirect_call(i32 ()** %ptr) nounwind {
 ; X64-LABEL: test_indirect_call:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    movq $-1, %rcx
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movq (%rdi), %rcx
-; X64-NEXT:    orq %rax, %rcx
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    callq *%rcx
-; X64-NEXT:    movq %rsp, %rcx
-; X64-NEXT:    sarq $63, %rcx
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    movq %rsp, %rbx
+; X64-NEXT:    movq $-1, %rax
+; X64-NEXT:    sarq $63, %rbx
+; X64-NEXT:    movq (%rdi), %rax
+; X64-NEXT:    orq %rbx, %rax
+; X64-NEXT:    movq %rbx, %rcx
 ; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    orq %rcx, %rsp
-; X64-NEXT:    popq %rcx
+; X64-NEXT:    callq *%rax
+; X64-NEXT:    shlq $47, %rbx
+; X64-NEXT:    orq %rbx, %rsp
+; X64-NEXT:    popq %rbx
 ; X64-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_indirect_call:
 ; X64-RETPOLINE:       # %bb.0: # %entry
-; X64-RETPOLINE-NEXT:    pushq %rax
-; X64-RETPOLINE-NEXT:    movq %rsp, %rax
-; X64-RETPOLINE-NEXT:    movq $-1, %rcx
-; X64-RETPOLINE-NEXT:    sarq $63, %rax
+; X64-RETPOLINE-NEXT:    pushq %rbx
+; X64-RETPOLINE-NEXT:    movq %rsp, %rbx
+; X64-RETPOLINE-NEXT:    movq $-1, %rax
+; X64-RETPOLINE-NEXT:    sarq $63, %rbx
 ; X64-RETPOLINE-NEXT:    movq (%rdi), %r11
-; X64-RETPOLINE-NEXT:    orq %rax, %r11
+; X64-RETPOLINE-NEXT:    orq %rbx, %r11
+; X64-RETPOLINE-NEXT:    movq %rbx, %rax
 ; X64-RETPOLINE-NEXT:    shlq $47, %rax
 ; X64-RETPOLINE-NEXT:    orq %rax, %rsp
 ; X64-RETPOLINE-NEXT:    callq __llvm_retpoline_r11
-; X64-RETPOLINE-NEXT:    movq %rsp, %rcx
-; X64-RETPOLINE-NEXT:    sarq $63, %rcx
-; X64-RETPOLINE-NEXT:    shlq $47, %rcx
-; X64-RETPOLINE-NEXT:    orq %rcx, %rsp
-; X64-RETPOLINE-NEXT:    popq %rcx
+; X64-RETPOLINE-NEXT:    shlq $47, %rbx
+; X64-RETPOLINE-NEXT:    orq %rbx, %rsp
+; X64-RETPOLINE-NEXT:    popq %rbx
 ; X64-RETPOLINE-NEXT:    retq
 entry:
   %fp = load i32 ()*, i32 ()** %ptr
   %v = call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_tail_call(i32 ()** %ptr) nounwind {
entry:
   %fp = load i32 ()*, i32 ()** %ptr
   %v = tail call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_call_global() nounwind {
 ; X64-LABEL: test_indirect_call_global:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rax
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    movq $-1, %rcx
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movq {{.*}}(%rip), %rcx
-; X64-NEXT:    orq %rax, %rcx
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    callq *%rcx
-; X64-NEXT:    movq %rsp, %rcx
-; X64-NEXT:    sarq $63, %rcx
+; X64-NEXT:    pushq %rbx
+; X64-NEXT:    movq %rsp, %rbx
+; X64-NEXT:    movq $-1, %rax
+; X64-NEXT:    sarq $63, %rbx
+; X64-NEXT:    movq {{.*}}(%rip), %rax
+; X64-NEXT:    orq %rbx, %rax
+; X64-NEXT:    movq %rbx, %rcx
 ; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    orq %rcx, %rsp
-; X64-NEXT:    popq %rcx
+; X64-NEXT:    callq *%rax
+; X64-NEXT:    shlq $47, %rbx
+; X64-NEXT:    orq %rbx, %rsp
+; X64-NEXT:    popq %rbx
 ; X64-NEXT:    retq
 ;
 ; X64-RETPOLINE-LABEL: test_indirect_call_global:
 ; X64-RETPOLINE:       # %bb.0: # %entry
-; X64-RETPOLINE-NEXT:    pushq %rax
-; X64-RETPOLINE-NEXT:    movq %rsp, %rax
-; X64-RETPOLINE-NEXT:    movq $-1, %rcx
-; X64-RETPOLINE-NEXT:    sarq $63, %rax
+; X64-RETPOLINE-NEXT:    pushq %rbx
+; X64-RETPOLINE-NEXT:    movq %rsp, %rbx
+; X64-RETPOLINE-NEXT:    movq $-1, %rax
+; X64-RETPOLINE-NEXT:    sarq $63, %rbx
 ; X64-RETPOLINE-NEXT:    movq {{.*}}(%rip), %r11
+; X64-RETPOLINE-NEXT:    movq %rbx, %rax
 ; X64-RETPOLINE-NEXT:    shlq $47, %rax
 ; X64-RETPOLINE-NEXT:    orq %rax, %rsp
 ; X64-RETPOLINE-NEXT:    callq __llvm_retpoline_r11
-; X64-RETPOLINE-NEXT:    movq %rsp, %rcx
-; X64-RETPOLINE-NEXT:    sarq $63, %rcx
-; X64-RETPOLINE-NEXT:    shlq $47, %rcx
-; X64-RETPOLINE-NEXT:    orq %rcx, %rsp
-; X64-RETPOLINE-NEXT:    popq %rcx
+; X64-RETPOLINE-NEXT:    shlq $47, %rbx
+; X64-RETPOLINE-NEXT:    orq %rbx, %rsp
+; X64-RETPOLINE-NEXT:    popq %rbx
 ; X64-RETPOLINE-NEXT:    retq
 entry:
   %fp = load i32 ()*, i32 ()** @global_fnptr
   %v = call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_tail_call_global() nounwind {

llvm/test/CodeGen/X86/speculative-load-hardening.ll

 ; X64-NEXT:    cmovgq %rcx, %rax
 ; X64-NEXT:    movslq %edi, %rcx
 ; X64-NEXT:    movl (%rsi,%rcx,4), %ebp
 ; X64-NEXT:    orl %eax, %ebp
 ; X64-NEXT:    movl $4, %edi
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq __cxa_allocate_exception
-; X64-NEXT:    movq %rsp, %rcx
-; X64-NEXT:    sarq $63, %rcx
 ; X64-NEXT:    movl %ebp, (%rax)
 ; X64-NEXT:  .Ltmp0:
 ; X64-NEXT:    xorl %esi, %esi
 ; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    movq %rax, %rdi
-; X64-NEXT:    orq %rcx, %rsp
 ; X64-NEXT:    callq __cxa_throw
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:  .Ltmp1:
 ; X64-NEXT:    jmp .LBB4_3
 ; X64-NEXT:  .LBB4_1:
 ; X64-NEXT:    cmovleq %rcx, %rax
 ; X64-NEXT:  .LBB4_3: # %exit

 ; X64-NEXT:    addl (%rbx), %eax
 ; X64-NEXT:    cltq
 ; X64-NEXT:    orq %rcx, %rax
 ; X64-NEXT:    movl (%r14,%rax,4), %edi
 ; X64-NEXT:    orl %ecx, %edi
 ; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    orq %rcx, %rsp
 ; X64-NEXT:    callq sink
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
 ;
 ; X64-LFENCE-LABEL: test_basic_eh:
 ; X64-LFENCE:       # %bb.0: # %entry
 ; X64-LFENCE-NEXT:    pushq %rbp
 ; X64-LFENCE-NEXT:    pushq %r14
 ; X64-LFENCE-NEXT:    pushq %rbx
 ; X64-LFENCE-NEXT:    cmpl $41, %edi
 ; X64-LFENCE-NEXT:    jg .LBB4_2

 declare void @sink_double(double)
 
 ; Test direct and converting loads of floating point values.
 define void @test_fp_loads(float* %fptr, double* %dptr, i32* %i32ptr, i64* %i64ptr) nounwind {
 ; X64-LABEL: test_fp_loads:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    pushq %r15
 ; X64-NEXT:    pushq %r14
+; X64-NEXT:    pushq %r13
 ; X64-NEXT:    pushq %r12
 ; X64-NEXT:    pushq %rbx
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    movq %rcx, %r15
+; X64-NEXT:    movq %rcx, %r12
 ; X64-NEXT:    movq %rdx, %r14
-; X64-NEXT:    movq %rsi, %rbx
-; X64-NEXT:    movq %rdi, %r12
+; X64-NEXT:    movq %rsi, %r15
+; X64-NEXT:    movq %rdi, %r13
 ; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r12
+; X64-NEXT:    orq %rax, %r13
 ; X64-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink_float
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %rbx
+; X64-NEXT:    movq %rsp, %rbx
+; X64-NEXT:    sarq $63, %rbx
+; X64-NEXT:    orq %r15, %rbx
 ; X64-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink_double
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; X64-NEXT:    cvtsd2ss %xmm0, %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink_float
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; X64-NEXT:    cvtss2sd %xmm0, %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink_double
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r14
+; X64-NEXT:    movq %rsp, %r15
+; X64-NEXT:    sarq $63, %r15
+; X64-NEXT:    orq %r14, %r15
 ; X64-NEXT:    xorps %xmm0, %xmm0
-; X64-NEXT:    cvtsi2ssl (%r14), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    cvtsi2ssl (%r15), %xmm0
 ; X64-NEXT:    callq sink_float
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r15
+; X64-NEXT:    movq %rsp, %rbx
+; X64-NEXT:    sarq $63, %rbx
+; X64-NEXT:    orq %rbx, %r12
 ; X64-NEXT:    xorps %xmm0, %xmm0
-; X64-NEXT:    cvtsi2sdq (%r15), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    cvtsi2sdq (%r12), %xmm0
 ; X64-NEXT:    callq sink_double
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    xorps %xmm0, %xmm0
-; X64-NEXT:    cvtsi2ssq (%r15), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    cvtsi2ssq (%r12), %xmm0
 ; X64-NEXT:    callq sink_float
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    xorps %xmm0, %xmm0
-; X64-NEXT:    cvtsi2sdl (%r14), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    cvtsi2sdl (%r15), %xmm0
 ; X64-NEXT:    callq sink_double
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    addq $8, %rsp
+; X64-NEXT:    shlq $47, %rbx
+; X64-NEXT:    orq %rbx, %rsp
 ; X64-NEXT:    popq %rbx
 ; X64-NEXT:    popq %r12
+; X64-NEXT:    popq %r13
 ; X64-NEXT:    popq %r14
 ; X64-NEXT:    popq %r15
 ; X64-NEXT:    retq
 ;
 ; X64-LFENCE-LABEL: test_fp_loads:
 ; X64-LFENCE:       # %bb.0: # %entry
 ; X64-LFENCE-NEXT:    pushq %r15
 ; X64-LFENCE-NEXT:    pushq %r14

 ; X64-LABEL: test_vec_loads:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    pushq %r15
 ; X64-NEXT:    pushq %r14
 ; X64-NEXT:    pushq %r13
 ; X64-NEXT:    pushq %r12
 ; X64-NEXT:    pushq %rbx
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    movq %r9, %r14
-; X64-NEXT:    movq %r8, %r15
-; X64-NEXT:    movq %rcx, %r12
-; X64-NEXT:    movq %rdx, %r13
+; X64-NEXT:    movq %r9, %r13
+; X64-NEXT:    movq %r8, %r14
+; X64-NEXT:    movq %rcx, %r15
+; X64-NEXT:    movq %rdx, %r12
 ; X64-NEXT:    movq %rsi, %rbx
 ; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    orq %rax, %rdi
 ; X64-NEXT:    movaps (%rdi), %xmm0
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink_v4f32
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %rbx
-; X64-NEXT:    movaps (%rbx), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    orq %rbx, %rax
+; X64-NEXT:    movaps (%rax), %xmm0
 ; X64-NEXT:    callq sink_v2f64
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r13
-; X64-NEXT:    movaps (%r13), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    orq %r12, %rax
+; X64-NEXT:    movaps (%rax), %xmm0
 ; X64-NEXT:    callq sink_v16i8
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r12
-; X64-NEXT:    movaps (%r12), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    orq %r15, %rax
+; X64-NEXT:    movaps (%rax), %xmm0
 ; X64-NEXT:    callq sink_v8i16
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r15
-; X64-NEXT:    movaps (%r15), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    orq %r14, %rax
+; X64-NEXT:    movaps (%rax), %xmm0
 ; X64-NEXT:    callq sink_v4i32
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    orq %rax, %r14
-; X64-NEXT:    movaps (%r14), %xmm0
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    movq %rsp, %rbx
+; X64-NEXT:    sarq $63, %rbx
+; X64-NEXT:    orq %rbx, %r13
+; X64-NEXT:    movaps (%r13), %xmm0
 ; X64-NEXT:    callq sink_v2i64
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    shlq $47, %rbx
+; X64-NEXT:    orq %rbx, %rsp
 ; X64-NEXT:    popq %rbx
 ; X64-NEXT:    popq %r12
 ; X64-NEXT:    popq %r13
 ; X64-NEXT:    popq %r14
 ; X64-NEXT:    popq %r15
 ; X64-NEXT:    retq
 ;
 ; X64-LFENCE-LABEL: test_vec_loads:
entry:
   %x6 = load <2 x i64>, <2 x i64>* %v2i64ptr
   call void @sink_v2i64(<2 x i64> %x6)
   ret void
 }
 
 define void @test_deferred_hardening(i32* %ptr1, i32* %ptr2, i32 %x) nounwind {
 ; X64-LABEL: test_deferred_hardening:
 ; X64:       # %bb.0: # %entry
+; X64-NEXT:    pushq %r15
 ; X64-NEXT:    pushq %r14
 ; X64-NEXT:    pushq %rbx
-; X64-NEXT:    pushq %rax
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    movq %rsi, %r14
-; X64-NEXT:    movq %rdi, %rbx
+; X64-NEXT:    movq %rdi, %r15
 ; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    movl (%rdi), %edi
 ; X64-NEXT:    incl %edi
 ; X64-NEXT:    imull %edx, %edi
 ; X64-NEXT:    orl %eax, %edi
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movl (%rbx), %ecx
+; X64-NEXT:    movl (%r15), %ecx
 ; X64-NEXT:    movl (%r14), %edx
 ; X64-NEXT:    leal 1(%rcx,%rdx), %edi
+; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    orl %eax, %edi
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movl (%rbx), %edi
+; X64-NEXT:    movl (%r15), %edi
 ; X64-NEXT:    shll $7, %edi
+; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    orl %eax, %edi
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink
 ; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movzwl (%rbx), %ecx
+; X64-NEXT:    movzwl (%r15), %ecx
 ; X64-NEXT:    sarw $7, %cx
 ; X64-NEXT:    movzwl %cx, %edi
 ; X64-NEXT:    notl %edi
+; X64-NEXT:    sarq $63, %rax
 ; X64-NEXT:    orl %eax, %edi
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
 ; X64-NEXT:    callq sink
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    movzwl (%rbx), %ecx
-; X64-NEXT:    rolw $9, %cx
-; X64-NEXT:    movswl %cx, %edi
+; X64-NEXT:    movq %rsp, %rbx
+; X64-NEXT:    movzwl (%r15), %eax
+; X64-NEXT:    rolw $9, %ax
+; X64-NEXT:    movswl %ax, %edi
 ; X64-NEXT:    negl %edi
-; X64-NEXT:    orl %eax, %edi
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    sarq $63, %rbx
+; X64-NEXT:    orl %ebx, %edi
 ; X64-NEXT:    callq sink
-; X64-NEXT:    movq %rsp, %rax
-; X64-NEXT:    sarq $63, %rax
-; X64-NEXT:    shlq $47, %rax
-; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    addq $8, %rsp
+; X64-NEXT:    shlq $47, %rbx
+; X64-NEXT:    orq %rbx, %rsp
 ; X64-NEXT:    popq %rbx
 ; X64-NEXT:    popq %r14
+; X64-NEXT:    popq %r15
 ; X64-NEXT:    retq
 ;
 ; X64-LFENCE-LABEL: test_deferred_hardening:
 ; X64-LFENCE:       # %bb.0: # %entry
 ; X64-LFENCE-NEXT:    pushq %r14
 ; X64-LFENCE-NEXT:    pushq %rbx
 ; X64-LFENCE-NEXT:    pushq %rax
 ; X64-LFENCE-NEXT:    movq %rsi, %r14