[x86/SLH] Teach the x86 speculative load hardening pass to harden against v1.2 BCBS attacks directly.

[chandlerc]

Summary

Attacks using spectre v1.2 (a subset of BCBS) are described in the paper
here:
https://people.csail.mit.edu/vlk/spectre11.pdf

The core idea is to speculatively store over the address in a vtable,
jumptable, or other target of indirect control flow that will be
subsequently loaded. Speculative execution after such a store can
forward the stored value to subsequent loads, and if called or jumped
to, the speculative execution will be steered to this potentially
attacker controlled address.

Up until now, this could be mitigated by enableing retpolines. However,
that is a relatively expensive technique to mitigate this particular
flavor. Especially because in most cases SLH will have already mitigated
this. To fully mitigate this with SLH, we need to do two core things:

1. Unfold loads from calls and jumps, allowing the loads to be post-load hardened.
2. Force hardening of incoming registers even if we didn't end up needing to harden the load itself.

The reason we need to do these two things is because hardening calls and
jumps from this particular variant is importantly different from
hardening against leak of secret data. Because the "bad" data here isn't
a secret, but in fact speculatively stored by the attacker, it may be
loaded from any address, regardless of whether it is read-only memory,
mapped memory, or a "hardened" address. The only 100% effective way to
harden these instructions is to harden the their operand itself. But to
the extent possible, we'd like to take advantage of all the other
hardening going on, we just need a fallback in case none of that
happened to cover the particular input to the control transfer
instruction.

This patch implements all of this, but it isn't quite ready to go yet.
First, there is some duplicated code between this patch and the
post-load hardening. I'll work on refactoring that code separately and
then this patch will simplify when rebased. I also need to test to see
what (if anything) we need to do so that these hardening steps are
naturally skipped when retpolines are in fact enabled, but that should
be trivial.

However, this patch shows all of the important mechanics here, including
the hoops we have to go through to unfold the loads from all of the
different instructions and then harden their incoming registers. It also
shows the expected result on the indirect test case.

For users of SLH, currently they are paing 2% to 6% performance overhead
for retpolines, but this mechanism is expected to be substantially
cheaper.

[craig.topper]

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

843

Why not use getOpcodeAfterMemoryUnfold?

[echristo]

Couple of small nits, ctopper's suggestion seems reasonable too. Only "better" way I had of dealing with this would be not folding the load at all in the first place, but it seems harder to justify given the existing unfolding machinery.

-eric

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

2233–2236

Mind sinking this closer to the if statement?

2244

I suppose technically you want < 8?

[chandlerc]

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

843

Because that doesn't (trivially) give me the register class...

But I can use that here to nuke a few of these lines.

If I'm missing a way to write this more simply let me know. The tricky part is passing in the register to the unfold API.

[chandlerc]

Split the TCRETURN folding into https://reviews.llvm.org/D49717 to make the
messy situation there more clear.

Updated this patch based on review feedback.

[chandlerc]

Rebase on top of the cleaned up folding patch, and now without any copy/paste
code smells. We just call out to the generic low-level register hardening
routine.

Other than checking for interactions w/ retpolines, this should be ready to go.

[chandlerc]

Confirmed that this does-the-right-thing with retpolines (as they get
a different instruction).

Notably, we still harden a loaded value that is potentially a secret, because
speculating even a retpoline has a risk of disclosing the loaded value.
However, when the target of the retpoline is not considered a secret (it is
loaded from RO memory for example), we don't force hardening the target as the
retpoline will block any BCBS-style attack.

I've added a retpoline mode to the indirect test which shows both of these
behaviors.

I think this patch is now good-to-go for a last round of review.

[echristo]

Couple of small nits, but that's it. Still LGTM.

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

2182–2183

Nit: Flow.

2219

Nit: I know you don't mean it, but it sounds like you're talking about r0 on some architecture :)

[chandlerc]

All done and submitting! Thanks!

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

2219

Doh, yeah. Simpler to just say 'the first operand of the instruction'. Thanks.

llvm/lib/Target/X86/X86InstrFoldTables.cpp

static const X86MemoryFoldTableEntry MemoryFoldTable0[] = {
   { X86::SETNPr,              X86::SETNPm,              TB_FOLDED_STORE },
   { X86::SETNSr,              X86::SETNSm,              TB_FOLDED_STORE },
   { X86::SETOr,               X86::SETOm,               TB_FOLDED_STORE },
   { X86::SETPr,               X86::SETPm,               TB_FOLDED_STORE },
   { X86::SETSr,               X86::SETSm,               TB_FOLDED_STORE },
   { X86::TAILJMPr,            X86::TAILJMPm,            TB_FOLDED_LOAD },
   { X86::TAILJMPr64,          X86::TAILJMPm64,          TB_FOLDED_LOAD },
   { X86::TAILJMPr64_REX,      X86::TAILJMPm64_REX,      TB_FOLDED_LOAD },
+  { X86::TCRETURNri,          X86::TCRETURNmi,          TB_FOLDED_LOAD },
+  { X86::TCRETURNri64,        X86::TCRETURNmi64,        TB_FOLDED_LOAD },
   { X86::TEST16ri,            X86::TEST16mi,            TB_FOLDED_LOAD },
   { X86::TEST16rr,            X86::TEST16mr,            TB_FOLDED_LOAD },
   { X86::TEST32ri,            X86::TEST32mi,            TB_FOLDED_LOAD },
   { X86::TEST32rr,            X86::TEST32mr,            TB_FOLDED_LOAD },
   { X86::TEST64ri32,          X86::TEST64mi32,          TB_FOLDED_LOAD },
   { X86::TEST64rr,            X86::TEST64mr,            TB_FOLDED_LOAD },
   { X86::TEST8ri,             X86::TEST8mi,             TB_FOLDED_LOAD },
   { X86::TEST8rr,             X86::TEST8mr,             TB_FOLDED_LOAD },

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

 /// https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html
 /// https://spectreattack.com/spectre.pdf
 ///
 //===----------------------------------------------------------------------===//
 
 #include "X86.h"
 #include "X86InstrBuilder.h"
 #include "X86InstrInfo.h"
+#include "X86InstrFoldTables.h"
 #include "X86Subtarget.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"

 #define DEBUG_TYPE PASS_KEY
 
 STATISTIC(NumCondBranchesTraced, "Number of conditional branches traced");
 STATISTIC(NumBranchesUntraced, "Number of branches unable to trace");
 STATISTIC(NumAddrRegsHardened,
           "Number of address mode used registers hardaned");
 STATISTIC(NumPostLoadRegsHardened,
           "Number of post-load register values hardened");
+STATISTIC(NumCallsOrJumpsHardened,
+          "Number of calls or jumps requiring extra hardening");
 STATISTIC(NumInstsInserted, "Number of instructions inserted");
 STATISTIC(NumLFENCEsInserted, "Number of lfence instructions inserted");
 
 static cl::opt<bool> HardenEdgesWithLFENCE(
     PASS_KEY "-lfence",
     cl::desc(
         "Use LFENCE along each conditional edge to harden against speculative "
         "loads rather than conditional movs and poisoned pointers."),
static cl::opt<bool> HardenInterprocedurally(
     cl::init(true), cl::Hidden);
 
 static cl::opt<bool>
     HardenLoads(PASS_KEY "-loads",
                 cl::desc("Sanitize loads from memory. When disable, no "
                          "significant security is provided."),
                 cl::init(true), cl::Hidden);
 
+static cl::opt<bool> HardenIndirectCallsAndJumps(
+    PASS_KEY "-indirect",
+    cl::desc("Harden indirect calls and jumps against using speculatively "
+             "stored attacker controlled addresses. This is designed to "
+             "mitigate Spectre v1.2 style attacks."),
+    cl::init(true), cl::Hidden);
+
 namespace llvm {
 
 void initializeX86SpeculativeLoadHardeningPassPass(PassRegistry &);
 
 } // end namespace llvm
 
 namespace {
 
private:
 
   void hardenEdgesWithLFENCE(MachineFunction &MF);
 
   SmallVector<BlockCondInfo, 16> collectBlockCondInfo(MachineFunction &MF);
 
   SmallVector<MachineInstr *, 16>
   tracePredStateThroughCFG(MachineFunction &MF, ArrayRef<BlockCondInfo> Infos);
 
+  void unfoldCallAndJumpLoads(MachineFunction &MF);
+
   void hardenAllLoads(MachineFunction &MF);
 
   unsigned saveEFLAGS(MachineBasicBlock &MBB,
                       MachineBasicBlock::iterator InsertPt, DebugLoc Loc);
   void restoreEFLAGS(MachineBasicBlock &MBB,
                      MachineBasicBlock::iterator InsertPt, DebugLoc Loc,
                      unsigned OFReg);
 
hardenLoadAddr(MachineInstr &MI, MachineOperand &BaseMO,
                  MachineOperand &IndexMO,
                  SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg);
   MachineInstr *
   sinkPostLoadHardenedInst(MachineInstr &MI,
                            SmallPtrSetImpl<MachineInstr *> &HardenedInstrs);
   bool canHardenRegister(unsigned Reg);
   void hardenPostLoad(MachineInstr &MI);
   void hardenReturnInstr(MachineInstr &MI);
+  void hardenIndirectCallOrJumpInstr(
+      MachineInstr &MI,
+      SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg);
 };
 
 } // end anonymous namespace
 
 char X86SpeculativeLoadHardeningPass::ID = 0;
 
 void X86SpeculativeLoadHardeningPass::getAnalysisUsage(
     AnalysisUsage &AU) const {
for (MachineBasicBlock &MBB : MF) {
       if (!MBB.isEHPad())
         continue;
       PS->SSA.AddAvailableValue(
           &MBB,
           extractPredStateFromSP(MBB, MBB.SkipPHIsAndLabels(MBB.begin()), Loc));
     }
   }
 
+  // If we are going to harden calls and jumps we need to unfold their memory
+  // operands.
+  if (HardenIndirectCallsAndJumps)
+    unfoldCallAndJumpLoads(MF);
+
   // Now harden all of the loads in the function using the predicate state.
   hardenAllLoads(MF);
 
   // Now rewrite all the uses of the pred state using the SSA updater so that
   // we track updates through the CFG.
   for (MachineInstr *CMovI : CMovs)
     for (MachineOperand &Op : CMovI->operands()) {
       if (!Op.isReg() || Op.getReg() != PS->InitialReg)
for (const BlockCondInfo &Info : Infos) {
     // Build a checking version of the successor.
     BuildCheckingBlockForSuccAndConds(MBB, *UncondSucc, /*SuccCount*/ 1,
                                       UncondBr, UncondBr, UncondCodeSeq);
   }
 
   return CMovs;
 }
 
+/// Compute the register class for the unfolded load.
+///
+/// FIXME: This should probably live in X86InstrInfo, potentially by adding
+/// a way to unfold into a newly created vreg rather than requiring a register
+/// input.
+static const TargetRegisterClass *
+getRegClassForUnfoldedLoad(MachineFunction &MF, const X86InstrInfo &TII,
+                           unsigned Opcode) {
+  const X86MemoryFoldTableEntry *I = lookupUnfoldTable(Opcode);
+  if (I == nullptr)
+    return nullptr;
+  unsigned UnfoldedOpc = I->DstOp;
+  unsigned Index = I->Flags & TB_INDEX_MASK;
+  const MCInstrDesc &MCID = TII.get(UnfoldedOpc);
+  return TII.getRegClass(MCID, Index, &TII.getRegisterInfo(), MF);
+}
+
+void X86SpeculativeLoadHardeningPass::unfoldCallAndJumpLoads(
+    MachineFunction &MF) {
+  for (MachineBasicBlock &MBB : MF)
+    for (auto MII = MBB.instr_begin(), MIE = MBB.instr_end(); MII != MIE;) {
+      // Grab a reference and increment the iterator so we can remove this
+      // instruction if needed without disturbing the iteration.
+      MachineInstr &MI = *MII++;
+
+      // Must either be a call or a branch.
+      if (!MI.isCall() && !MI.isBranch())
+        continue;
+      // We only care about loading variants of these instructions.
+      if (!MI.mayLoad())
+        continue;
+
+      switch (MI.getOpcode()) {
+      default: {
+        LLVM_DEBUG(
+            dbgs() << "ERROR: Found an unexpected loading branch or call "
+                      "instruction:\n";
+            MI.dump(); dbgs() << "\n");
+        report_fatal_error("Unexpected loading branch or call!");
+      }
+
+      case X86::FARCALL16m:
+      case X86::FARCALL32m:
+      case X86::FARCALL64:
+      case X86::FARJMP16m:
+      case X86::FARJMP32m:
+      case X86::FARJMP64:
+        // We cannot mitigate far jumps or calls, but we also don't expect them
+        // to be vulnerable to Spectre v1.2 style attacks.
+        continue;
+
+      case X86::CALL16m:
+      case X86::CALL16m_NT:
+      case X86::CALL32m:
+      case X86::CALL32m_NT:
+      case X86::CALL64m:
+      case X86::CALL64m_NT:
+      case X86::JMP16m:
+      case X86::JMP16m_NT:
+      case X86::JMP32m:
+      case X86::JMP32m_NT:
+      case X86::JMP64m:
+      case X86::JMP64m_NT:
+      case X86::TAILJMPm64:
+      case X86::TAILJMPm64_REX:
+      case X86::TAILJMPm:
+      case X86::TCRETURNmi64:
+      case X86::TCRETURNmi: {
+        // Use the generic unfold logic now that we know we're dealing with
+        // expected instructions.
+        // FIXME: We don't have test coverage for all of these!
+        auto *UnfoldedRC = getRegClassForUnfoldedLoad(MF, *TII, MI.getOpcode());
+        if (!UnfoldedRC) {
+          LLVM_DEBUG(dbgs()
+                         << "ERROR: Unable to unfold load from instruction:\n";
+                     MI.dump(); dbgs() << "\n");
+          report_fatal_error("Unable to unfold load!");
+        }
+        unsigned Reg = MRI->createVirtualRegister(UnfoldedRC);
+        SmallVector<MachineInstr *, 2> NewMIs;
+        // If we were able to compute an unfolded reg class, any failure here
+        // is just a programming error so just assert.
+        bool Unfolded =
+            TII->unfoldMemoryOperand(MF, MI, Reg, /*UnfoldLoad*/ true,
+                                     /*UnfoldStore*/ false, NewMIs);
+        (void)Unfolded;
+        assert(Unfolded &&
+               "Computed unfolded register class but failed to unfold");
+        // Now stitch the new instructions into place and erase the old one.
+        for (auto *NewMI : NewMIs)
+          MBB.insert(MI.getIterator(), NewMI);
+        MI.eraseFromParent();
+        LLVM_DEBUG({
+          dbgs() << "Unfolded load successfully into:\n";
+          for (auto *NewMI : NewMIs) {
+            NewMI->dump();
+            dbgs() << "\n";
+          }
+        });
+        continue;
+      }
+      }
+      llvm_unreachable("Escaped switch with default!");
+    }
+}
+
 /// Returns true if the instruction has no behavior (specified or otherwise)
 /// that is based on the value of any of its register operands
 ///
 /// A classical example of something that is inherently not data invariant is an
 /// indirect jump -- the destination is loaded into icache based on the bits set
 /// in the jump destination register.
 ///
 /// FIXME: This should become part of our instruction tables.
for (MachineInstr &MI : MBB) {
         // it to itself.
         AddrRegToHardenedReg[MI.getOperand(0).getReg()] =
             MI.getOperand(0).getReg();
 
         hardenPostLoad(MI);
         continue;
       }
 
+      // Check for an indirect call or branch that may need its input hardened
+      // even if we couldn't find the specific load used, or were able to avoid
+      // hardening it for some reason. Note that here we cannot break out
+      // afterward as we may still need to handle any call aspect of this
+      // instruction.
+      if ((MI.isCall() || MI.isBranch()) && HardenIndirectCallsAndJumps)
+        hardenIndirectCallOrJumpInstr(MI, AddrRegToHardenedReg);
+
       // After we finish processing the instruction and doing any hardening
       // necessary for it, we need to handle transferring the predicate state
       // into a call and recovering it after the call returns (if it returns).
       if (!MI.isCall())
         continue;
 
       // If we're not hardening interprocedurally, we can just skip calls.
       if (!HardenInterprocedurally)
void X86SpeculativeLoadHardeningPass::hardenReturnInstr(MachineInstr &MI) {
   }
 
   // Take our predicate state, shift it to the high 17 bits (so that we keep
   // pointers canonical) and merge it into RSP. This will allow the caller to
   // extract it when we return (speculatively).
   mergePredStateIntoSP(MBB, InsertPt, Loc, PS->SSA.GetValueAtEndOfBlock(&MBB));
 }
 
+/// An attacker may speculatively store over a value that is then speculatively
+/// loaded and used as the target of an indirect call or jump instruction. This
+/// is called Spectre v1.2 or Bounds Check Bypass Store (BCBS) and is described
+/// in this paper:
+/// https://people.csail.mit.edu/vlk/spectre11.pdf
+///
+/// When this happens, the speculative
+/// execution of the call or jump will end up being steered to this attacker
+/// controlled address. While most such loads will be adequately hardened
+/// already, we want to ensure that they are definitively treated as needing
+/// post-load hardening. While address hardening is sufficient to prevent secret
+/// data from leaking to the attacker, it may not be sufficient to prevent an
+/// attacker from steering speculative execution. We forcibly unfolded all
+/// relevant loads above and so will always have an opportunity to post-load
+/// harden here, we just need to scan for cases not already flagged and add
+/// them.
+void X86SpeculativeLoadHardeningPass::hardenIndirectCallOrJumpInstr(
+    MachineInstr &MI,
+    SmallDenseMap<unsigned, unsigned, 32> &AddrRegToHardenedReg) {
+  switch (MI.getOpcode()) {
+  case X86::FARCALL16m:
+  case X86::FARCALL32m:
+  case X86::FARCALL64:
+  case X86::FARJMP16m:
+  case X86::FARJMP32m:
+  case X86::FARJMP64:
+    // We don't need to harden either far calls or far jumps as they are
+    // safe from Spectre.
+    return;
+
+  default:
+    break;
+  }
+
+  // We should never see a loading instruction at this point, as those should
+  // have been unfolded.
+  assert(!MI.mayLoad() && "Found a lingering loading instruction!");
+
+  // If the first operand isn't a register, this is a branch or call
+  // instruction with an immediate operand which doesn't need to be hardened.
+  if (!MI.getOperand(0).isReg())
+    return;
+
+  // For all of these, the target register is zero.
+  auto &TargetOp = MI.getOperand(0);
+  unsigned OldTargetReg = TargetOp.getReg();
+  assert(canHardenRegister(OldTargetReg) &&
+         "Cannot harden this instruction's target register!");
+
+  // If we have already hardened the target register, there is nothing to do.
+  if (AddrRegToHardenedReg.count(OldTargetReg))
+    return;
+
+  MachineBasicBlock &MBB = *MI.getParent();
+  DebugLoc Loc = MI.getDebugLoc();
+  auto InsertPt = MI.getIterator();
+
+  // FIXME: All the hardening code is essentially duplicated with the post-load
+  // hardening code. Should refactor that to be sharable here.
+  auto *TargetRC = MRI->getRegClass(OldTargetReg);
+  int TargetRegBytes = TRI->getRegSizeInBits(*TargetRC) / 8;
+
+  unsigned FlagsReg = 0;
+  if (isEFLAGSLive(MBB, InsertPt, *TRI))
+    FlagsReg = saveEFLAGS(MBB, InsertPt, Loc);
+
+  unsigned StateReg = PS->SSA.GetValueAtEndOfBlock(&MBB);
+  // FIXME: Need to teach this about 32-bit mode.
+  if (TargetRegBytes != 8) {
+    unsigned SubRegImms[] = {X86::sub_8bit, X86::sub_16bit, X86::sub_32bit};
+    unsigned SubRegImm = SubRegImms[Log2_32(TargetRegBytes)];
+    unsigned NarrowStateReg = MRI->createVirtualRegister(TargetRC);
+    BuildMI(MBB, InsertPt, Loc, TII->get(TargetOpcode::COPY), NarrowStateReg)
+        .addReg(StateReg, 0, SubRegImm);
+    StateReg = NarrowStateReg;
+  }
+
+  unsigned NewTargetReg = MRI->createVirtualRegister(TargetRC);
+  unsigned OrOpCodes[] = {X86::OR8rr, X86::OR16rr, X86::OR32rr, X86::OR64rr};
+  unsigned OrOpCode = OrOpCodes[Log2_32(TargetRegBytes)];
+  auto OrI = BuildMI(MBB, InsertPt, Loc, TII->get(OrOpCode), NewTargetReg)
+                 .addReg(StateReg)
+                 .addReg(OldTargetReg);
+  OrI->addRegisterDead(X86::EFLAGS, TRI);
+  TargetOp.setReg(NewTargetReg);
+  ++NumInstsInserted;
+  LLVM_DEBUG(dbgs() << "  Inserting or: "; OrI->dump(); dbgs() << "\n");
+
+  if (FlagsReg)
+    restoreEFLAGS(MBB, InsertPt, Loc, FlagsReg);
+
+  ++NumCallsOrJumpsHardened;
+}
+
 INITIALIZE_PASS_BEGIN(X86SpeculativeLoadHardeningPass, DEBUG_TYPE,
                       "X86 speculative load hardener", false, false)
 INITIALIZE_PASS_END(X86SpeculativeLoadHardeningPass, DEBUG_TYPE,
                     "X86 speculative load hardener", false, false)
 
 FunctionPass *llvm::createX86SpeculativeLoadHardeningPass() {
   return new X86SpeculativeLoadHardeningPass();
 }

llvm/test/CodeGen/X86/speculative-load-hardening-indirect.ll

@global_blockaddrs = constant [4 x i8*] [
   i8* blockaddress(@test_indirectbr_global, %bb1),
   i8* blockaddress(@test_indirectbr_global, %bb2),
   i8* blockaddress(@test_indirectbr_global, %bb3)
 ]
 
 define i32 @test_indirect_call(i32 ()** %ptr) nounwind {
 ; X64-LABEL: test_indirect_call:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    pushq %rbx
-; X64-NEXT:    movq %rsp, %rbx
-; X64-NEXT:    movq $-1, %rax
-; X64-NEXT:    sarq $63, %rbx
-; X64-NEXT:    orq %rbx, %rdi
-; X64-NEXT:    callq *(%rdi)
-; X64-NEXT:    shlq $47, %rbx
-; X64-NEXT:    orq %rbx, %rsp
-; X64-NEXT:    popq %rbx
+; X64-NEXT:    pushq %rax
+; X64-NEXT:    movq %rsp, %rax
+; X64-NEXT:    movq $-1, %rcx
+; X64-NEXT:    sarq $63, %rax
+; X64-NEXT:    movq (%rdi), %rcx
+; X64-NEXT:    orq %rax, %rcx
+; X64-NEXT:    shlq $47, %rax
+; X64-NEXT:    orq %rax, %rsp
+; X64-NEXT:    callq *%rcx
+; X64-NEXT:    movq %rsp, %rcx
+; X64-NEXT:    sarq $63, %rcx
+; X64-NEXT:    shlq $47, %rcx
+; X64-NEXT:    orq %rcx, %rsp
+; X64-NEXT:    popq %rcx
 ; X64-NEXT:    retq
 entry:
   %fp = load i32 ()*, i32 ()** %ptr
   %v = call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_tail_call(i32 ()** %ptr) nounwind {
 ; X64-LABEL: test_indirect_tail_call:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
+; X64-NEXT:    movq (%rdi), %rcx
+; X64-NEXT:    orq %rax, %rcx
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    jmpq *(%rdi) # TAILCALL
+; X64-NEXT:    jmpq *%rcx # TAILCALL
 entry:
   %fp = load i32 ()*, i32 ()** %ptr
   %v = tail call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_call_global() nounwind {
 ; X64-LABEL: test_indirect_call_global:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    pushq %rax
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
+; X64-NEXT:    movq {{.*}}(%rip), %rcx
+; X64-NEXT:    orq %rax, %rcx
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    callq *{{.*}}(%rip)
+; X64-NEXT:    callq *%rcx
 ; X64-NEXT:    movq %rsp, %rcx
 ; X64-NEXT:    sarq $63, %rcx
 ; X64-NEXT:    shlq $47, %rcx
 ; X64-NEXT:    orq %rcx, %rsp
 ; X64-NEXT:    popq %rcx
 ; X64-NEXT:    retq
 entry:
   %fp = load i32 ()*, i32 ()** @global_fnptr
   %v = call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirect_tail_call_global() nounwind {
 ; X64-LABEL: test_indirect_tail_call_global:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rax
 ; X64-NEXT:    movq $-1, %rcx
 ; X64-NEXT:    sarq $63, %rax
+; X64-NEXT:    movq {{.*}}(%rip), %rcx
+; X64-NEXT:    orq %rax, %rcx
 ; X64-NEXT:    shlq $47, %rax
 ; X64-NEXT:    orq %rax, %rsp
-; X64-NEXT:    jmpq *{{.*}}(%rip) # TAILCALL
+; X64-NEXT:    jmpq *%rcx # TAILCALL
 entry:
   %fp = load i32 ()*, i32 ()** @global_fnptr
   %v = tail call i32 %fp()
   ret i32 %v
 }
 
 define i32 @test_indirectbr(i8** %ptr) nounwind {
 ; X64-LABEL: test_indirectbr:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rcx
 ; X64-NEXT:    movq $-1, %rax
 ; X64-NEXT:    sarq $63, %rcx
-; X64-NEXT:    orq %rcx, %rdi
-; X64-NEXT:    jmpq *(%rdi)
+; X64-NEXT:    movq (%rdi), %rax
+; X64-NEXT:    orq %rcx, %rax
+; X64-NEXT:    jmpq *%rax
 ; X64-NEXT:  .LBB4_1: # %bb0
 ; X64-NEXT:    movl $2, %eax
 ; X64-NEXT:    jmp .LBB4_2
 ; X64-NEXT:  .LBB4_4: # %bb2
 ; X64-NEXT:    movl $13, %eax
 ; X64-NEXT:    jmp .LBB4_2
 ; X64-NEXT:  .LBB4_5: # %bb3
 ; X64-NEXT:    movl $42, %eax

 
 define i32 @test_indirectbr_global(i32 %idx) nounwind {
 ; X64-LABEL: test_indirectbr_global:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movq %rsp, %rcx
 ; X64-NEXT:    movq $-1, %rax
 ; X64-NEXT:    sarq $63, %rcx
 ; X64-NEXT:    movslq %edi, %rax
+; X64-NEXT:    movq global_blockaddrs(,%rax,8), %rax
 ; X64-NEXT:    orq %rcx, %rax
-; X64-NEXT:    jmpq *global_blockaddrs(,%rax,8)
+; X64-NEXT:    jmpq *%rax
 ; X64-NEXT:  .Ltmp0: # Block address taken
 ; X64-NEXT:  .LBB5_1: # %bb0
 ; X64-NEXT:    movl $2, %eax
 ; X64-NEXT:    jmp .LBB5_2
 ; X64-NEXT:  .Ltmp1: # Block address taken
 ; X64-NEXT:  .LBB5_4: # %bb2
 ; X64-NEXT:    movl $13, %eax
 ; X64-NEXT:    jmp .LBB5_2

 ; X64-NEXT:    movq %rsp, %rcx
 ; X64-NEXT:    movq $-1, %rax
 ; X64-NEXT:    sarq $63, %rcx
 ; X64-NEXT:    cmpl $3, %edi
 ; X64-NEXT:    ja .LBB6_2
 ; X64-NEXT:  # %bb.1: # %entry
 ; X64-NEXT:    cmovaq %rax, %rcx
 ; X64-NEXT:    movl %edi, %eax
+; X64-NEXT:    movq .LJTI6_0(,%rax,8), %rax
 ; X64-NEXT:    orq %rcx, %rax
-; X64-NEXT:    jmpq *.LJTI6_0(,%rax,8)
+; X64-NEXT:    jmpq *%rax
 ; X64-NEXT:  .LBB6_3: # %bb1
 ; X64-NEXT:    movl $7, %eax
 ; X64-NEXT:    jmp .LBB6_4
 ; X64-NEXT:  .LBB6_2: # %bb0
 ; X64-NEXT:    cmovbeq %rax, %rcx
 ; X64-NEXT:    movl $2, %eax
 ; X64-NEXT:    jmp .LBB6_4
 ; X64-NEXT:  .LBB6_5: # %bb2