This is an archive of the discontinued LLVM Phabricator instance.

Differential D70273

[z86][SLH] Move isDataInvariant* functions
AbandonedPublic

Authored by zbrid on Nov 14 2019, 2:16 PM.

Details

Reviewers
craig.topper
rnk
Summary

I moved these functions to X86InstrInfo.cpp so I can use them in another
pass I'm working on. Thanks for the suggestion, Craig!

Tested via llvm-lit llvm/test/CodeGen/X86/speculative-load-hardening*

As noted in the FIXMEs that I moved along with the fuctions, these
should eventually be moved to the tablegen instruction tables. There
didn't seem a straightforward way to do that due to the difficulties
trying to handle the check for when EFLAGS is dead for target
independent opcodes.

Diff Detail

Event Timeline

zbrid created this revision.Nov 14 2019, 2:16 PM
Herald added a project: Restricted Project. · View Herald TranscriptNov 14 2019, 2:16 PM
Herald added subscribers: llvm-commits, hiraditya. · View Herald Transcript
Harbormaster completed remote builds in B40984: Diff 229405.Nov 14 2019, 2:16 PM
zbrid updated this revision to Diff 229408.Nov 14 2019, 2:26 PM

Update commit message to explain SLH specific behavior req'd

zbrid planned changes to this revision.Nov 14 2019, 2:35 PM

I'm going to update this to separate the check for dead EFLAGS from the isDataInvariant function.

Harbormaster completed remote builds in B40985: Diff 229408.Nov 14 2019, 2:36 PM
zbrid abandoned this revision.Mar 4 2020, 1:05 PM

Creating a new revision for easier workflow.

Revision Contents

PathSize
llvm/
lib/
Target/
X86/
26 lines
369 lines
402 lines

Diff 229408

llvm/lib/Target/X86/X86InstrInfo.h

Show First 20 Lines • Show All 177 Lines • ▼ Show 20 Linespublic:
/// extension instruction. That is, it's like a copy where it's legal for the /// extension instruction. That is, it's like a copy where it's legal for the
/// source to overlap the destination. e.g. X86::MOVSX64rr32. If this returns /// source to overlap the destination. e.g. X86::MOVSX64rr32. If this returns
/// true, then it's expected the pre-extension value is available as a subreg /// true, then it's expected the pre-extension value is available as a subreg
/// of the result register. This also returns the sub-register index in /// of the result register. This also returns the sub-register index in
/// SubIdx. /// SubIdx.
bool isCoalescableExtInstr(const MachineInstr &MI, unsigned &SrcReg, bool isCoalescableExtInstr(const MachineInstr &MI, unsigned &SrcReg,
unsigned &DstReg, unsigned &SubIdx) const override; unsigned &DstReg, unsigned &SubIdx) const override;
/// Returns true if the instruction has no behavior (specified or otherwise)
/// that is based on the value of any of its register operands
///
/// A classical example of something that is inherently not data invariant is an
/// indirect jump -- the destination is loaded into icache based on the bits set
/// in the jump destination register.
///
/// FIXME: This should become part of our instruction tables.
bool isDataInvariant(MachineInstr &MI) const;
/// Returns true if the instruction has no behavior (specified or otherwise)
/// that is based on the value loaded from memory or the value of any
/// non-address register operands.
///
/// For example, if the latency of the instruction is dependent on the
/// particular bits set in any of the registers *or* any of the bits loaded from
/// memory.
///
/// A classical example of something that is inherently not data invariant is an
/// indirect jump -- the destination is loaded into icache based on the bits set
/// in the jump destination register.
///
/// FIXME: This should become part of our instruction tables.
bool isDataInvariantLoad(MachineInstr &MI) const;
unsigned isLoadFromStackSlot(const MachineInstr &MI, unsigned isLoadFromStackSlot(const MachineInstr &MI,
int &FrameIndex) const override; int &FrameIndex) const override;
unsigned isLoadFromStackSlot(const MachineInstr &MI, unsigned isLoadFromStackSlot(const MachineInstr &MI,
int &FrameIndex, int &FrameIndex,
unsigned &MemBytes) const override; unsigned &MemBytes) const override;
/// isLoadFromStackSlotPostFE - Check for post-frame ptr elimination /// isLoadFromStackSlotPostFE - Check for post-frame ptr elimination
/// stack locations as well. This uses a heuristic so it isn't /// stack locations as well. This uses a heuristic so it isn't
/// reliable for correctness. /// reliable for correctness.
▲ Show 20 Lines • Show All 405 Lines • Show Last 20 Lines

llvm/lib/Target/X86/X86InstrInfo.cpp

  • This file is larger than 256 KB, so syntax highlighting is disabled by default.
Show First 20 Lines • Show All 129 Lines • ▼ Show 20 Lines case X86::MOVSX64rr32:
break; break;
} }
return true; return true;
} }
} }
return false; return false;
} }
bool X86InstrInfo::isDataInvariant(MachineInstr &MI) const {
switch (MI.getOpcode()) {
default:
// By default, assume that the instruction is not data invariant.
return false;
// Some target-independent operations that trivially lower to data-invariant
// instructions.
case TargetOpcode::COPY:
case TargetOpcode::INSERT_SUBREG:
case TargetOpcode::SUBREG_TO_REG:
return true;
// On x86 it is believed that imul is constant time w.r.t. the loaded data.
// However, they set flags and are perhaps the most surprisingly constant
// time operations so we call them out here separately.
case X86::IMUL16rr:
case X86::IMUL16rri8:
case X86::IMUL16rri:
case X86::IMUL32rr:
case X86::IMUL32rri8:
case X86::IMUL32rri:
case X86::IMUL64rr:
case X86::IMUL64rri32:
case X86::IMUL64rri8:
// Bit scanning and counting instructions that are somewhat surprisingly
// constant time as they scan across bits and do other fairly complex
// operations like popcnt, but are believed to be constant time on x86.
// However, these set flags.
case X86::BSF16rr:
case X86::BSF32rr:
case X86::BSF64rr:
case X86::BSR16rr:
case X86::BSR32rr:
case X86::BSR64rr:
case X86::LZCNT16rr:
case X86::LZCNT32rr:
case X86::LZCNT64rr:
case X86::POPCNT16rr:
case X86::POPCNT32rr:
case X86::POPCNT64rr:
case X86::TZCNT16rr:
case X86::TZCNT32rr:
case X86::TZCNT64rr:
// Bit manipulation instructions are effectively combinations of basic
// arithmetic ops, and should still execute in constant time. These also
// set flags.
case X86::BLCFILL32rr:
case X86::BLCFILL64rr:
case X86::BLCI32rr:
case X86::BLCI64rr:
case X86::BLCIC32rr:
case X86::BLCIC64rr:
case X86::BLCMSK32rr:
case X86::BLCMSK64rr:
case X86::BLCS32rr:
case X86::BLCS64rr:
case X86::BLSFILL32rr:
case X86::BLSFILL64rr:
case X86::BLSI32rr:
case X86::BLSI64rr:
case X86::BLSIC32rr:
case X86::BLSIC64rr:
case X86::BLSMSK32rr:
case X86::BLSMSK64rr:
case X86::BLSR32rr:
case X86::BLSR64rr:
case X86::TZMSK32rr:
case X86::TZMSK64rr:
// Bit extracting and clearing instructions should execute in constant time,
// and set flags.
case X86::BEXTR32rr:
case X86::BEXTR64rr:
case X86::BEXTRI32ri:
case X86::BEXTRI64ri:
case X86::BZHI32rr:
case X86::BZHI64rr:
// Shift and rotate.
case X86::ROL8r1: case X86::ROL16r1: case X86::ROL32r1: case X86::ROL64r1:
case X86::ROL8rCL: case X86::ROL16rCL: case X86::ROL32rCL: case X86::ROL64rCL:
case X86::ROL8ri: case X86::ROL16ri: case X86::ROL32ri: case X86::ROL64ri:
case X86::ROR8r1: case X86::ROR16r1: case X86::ROR32r1: case X86::ROR64r1:
case X86::ROR8rCL: case X86::ROR16rCL: case X86::ROR32rCL: case X86::ROR64rCL:
case X86::ROR8ri: case X86::ROR16ri: case X86::ROR32ri: case X86::ROR64ri:
case X86::SAR8r1: case X86::SAR16r1: case X86::SAR32r1: case X86::SAR64r1:
case X86::SAR8rCL: case X86::SAR16rCL: case X86::SAR32rCL: case X86::SAR64rCL:
case X86::SAR8ri: case X86::SAR16ri: case X86::SAR32ri: case X86::SAR64ri:
case X86::SHL8r1: case X86::SHL16r1: case X86::SHL32r1: case X86::SHL64r1:
case X86::SHL8rCL: case X86::SHL16rCL: case X86::SHL32rCL: case X86::SHL64rCL:
case X86::SHL8ri: case X86::SHL16ri: case X86::SHL32ri: case X86::SHL64ri:
case X86::SHR8r1: case X86::SHR16r1: case X86::SHR32r1: case X86::SHR64r1:
case X86::SHR8rCL: case X86::SHR16rCL: case X86::SHR32rCL: case X86::SHR64rCL:
case X86::SHR8ri: case X86::SHR16ri: case X86::SHR32ri: case X86::SHR64ri:
case X86::SHLD16rrCL: case X86::SHLD32rrCL: case X86::SHLD64rrCL:
case X86::SHLD16rri8: case X86::SHLD32rri8: case X86::SHLD64rri8:
case X86::SHRD16rrCL: case X86::SHRD32rrCL: case X86::SHRD64rrCL:
case X86::SHRD16rri8: case X86::SHRD32rri8: case X86::SHRD64rri8:
// Basic arithmetic is constant time on the input but does set flags.
case X86::ADC8rr: case X86::ADC8ri:
case X86::ADC16rr: case X86::ADC16ri: case X86::ADC16ri8:
case X86::ADC32rr: case X86::ADC32ri: case X86::ADC32ri8:
case X86::ADC64rr: case X86::ADC64ri8: case X86::ADC64ri32:
case X86::ADD8rr: case X86::ADD8ri:
case X86::ADD16rr: case X86::ADD16ri: case X86::ADD16ri8:
case X86::ADD32rr: case X86::ADD32ri: case X86::ADD32ri8:
case X86::ADD64rr: case X86::ADD64ri8: case X86::ADD64ri32:
case X86::AND8rr: case X86::AND8ri:
case X86::AND16rr: case X86::AND16ri: case X86::AND16ri8:
case X86::AND32rr: case X86::AND32ri: case X86::AND32ri8:
case X86::AND64rr: case X86::AND64ri8: case X86::AND64ri32:
case X86::OR8rr: case X86::OR8ri:
case X86::OR16rr: case X86::OR16ri: case X86::OR16ri8:
case X86::OR32rr: case X86::OR32ri: case X86::OR32ri8:
case X86::OR64rr: case X86::OR64ri8: case X86::OR64ri32:
case X86::SBB8rr: case X86::SBB8ri:
case X86::SBB16rr: case X86::SBB16ri: case X86::SBB16ri8:
case X86::SBB32rr: case X86::SBB32ri: case X86::SBB32ri8:
case X86::SBB64rr: case X86::SBB64ri8: case X86::SBB64ri32:
case X86::SUB8rr: case X86::SUB8ri:
case X86::SUB16rr: case X86::SUB16ri: case X86::SUB16ri8:
case X86::SUB32rr: case X86::SUB32ri: case X86::SUB32ri8:
case X86::SUB64rr: case X86::SUB64ri8: case X86::SUB64ri32:
case X86::XOR8rr: case X86::XOR8ri:
case X86::XOR16rr: case X86::XOR16ri: case X86::XOR16ri8:
case X86::XOR32rr: case X86::XOR32ri: case X86::XOR32ri8:
case X86::XOR64rr: case X86::XOR64ri8: case X86::XOR64ri32:
// Arithmetic with just 32-bit and 64-bit variants and no immediates.
case X86::ADCX32rr: case X86::ADCX64rr:
case X86::ADOX32rr: case X86::ADOX64rr:
case X86::ANDN32rr: case X86::ANDN64rr:
// Unary arithmetic operations.
case X86::DEC8r: case X86::DEC16r: case X86::DEC32r: case X86::DEC64r:
case X86::INC8r: case X86::INC16r: case X86::INC32r: case X86::INC64r:
case X86::NEG8r: case X86::NEG16r: case X86::NEG32r: case X86::NEG64r:
// Check whether the EFLAGS implicit-def is dead. We assume that this will
// always find the implicit-def because this code should only be reached
// for instructions that do in fact implicitly def this.
if (!MI.findRegisterDefOperand(X86::EFLAGS)->isDead()) {
// If we would clobber EFLAGS that are used, just bail for now.
LLVM_DEBUG(dbgs() << " Unable to harden post-load due to EFLAGS: ";
MI.dump(); dbgs() << "\n");
return false;
}
// Otherwise, fallthrough to handle these the same as instructions that
// don't set EFLAGS.
LLVM_FALLTHROUGH;
// Unlike other arithmetic, NOT doesn't set EFLAGS.
case X86::NOT8r: case X86::NOT16r: case X86::NOT32r: case X86::NOT64r:
// Various move instructions used to zero or sign extend things. Note that we
// intentionally don't support the _NOREX variants as we can't handle that
// register constraint anyways.
case X86::MOVSX16rr8:
case X86::MOVSX32rr8: case X86::MOVSX32rr16:
case X86::MOVSX64rr8: case X86::MOVSX64rr16: case X86::MOVSX64rr32:
case X86::MOVZX16rr8:
case X86::MOVZX32rr8: case X86::MOVZX32rr16:
case X86::MOVZX64rr8: case X86::MOVZX64rr16:
case X86::MOV32rr:
// Arithmetic instructions that are both constant time and don't set flags.
case X86::RORX32ri:
case X86::RORX64ri:
case X86::SARX32rr:
case X86::SARX64rr:
case X86::SHLX32rr:
case X86::SHLX64rr:
case X86::SHRX32rr:
case X86::SHRX64rr:
// LEA doesn't actually access memory, and its arithmetic is constant time.
case X86::LEA16r:
case X86::LEA32r:
case X86::LEA64_32r:
case X86::LEA64r:
return true;
}
}
bool X86InstrInfo::isDataInvariantLoad(MachineInstr &MI) const {
switch (MI.getOpcode()) {
default:
// By default, assume that the load will immediately leak.
return false;
// On x86 it is believed that imul is constant time w.r.t. the loaded data.
// However, they set flags and are perhaps the most surprisingly constant
// time operations so we call them out here separately.
case X86::IMUL16rm:
case X86::IMUL16rmi8:
case X86::IMUL16rmi:
case X86::IMUL32rm:
case X86::IMUL32rmi8:
case X86::IMUL32rmi:
case X86::IMUL64rm:
case X86::IMUL64rmi32:
case X86::IMUL64rmi8:
// Bit scanning and counting instructions that are somewhat surprisingly
// constant time as they scan across bits and do other fairly complex
// operations like popcnt, but are believed to be constant time on x86.
// However, these set flags.
case X86::BSF16rm:
case X86::BSF32rm:
case X86::BSF64rm:
case X86::BSR16rm:
case X86::BSR32rm:
case X86::BSR64rm:
case X86::LZCNT16rm:
case X86::LZCNT32rm:
case X86::LZCNT64rm:
case X86::POPCNT16rm:
case X86::POPCNT32rm:
case X86::POPCNT64rm:
case X86::TZCNT16rm:
case X86::TZCNT32rm:
case X86::TZCNT64rm:
// Bit manipulation instructions are effectively combinations of basic
// arithmetic ops, and should still execute in constant time. These also
// set flags.
case X86::BLCFILL32rm:
case X86::BLCFILL64rm:
case X86::BLCI32rm:
case X86::BLCI64rm:
case X86::BLCIC32rm:
case X86::BLCIC64rm:
case X86::BLCMSK32rm:
case X86::BLCMSK64rm:
case X86::BLCS32rm:
case X86::BLCS64rm:
case X86::BLSFILL32rm:
case X86::BLSFILL64rm:
case X86::BLSI32rm:
case X86::BLSI64rm:
case X86::BLSIC32rm:
case X86::BLSIC64rm:
case X86::BLSMSK32rm:
case X86::BLSMSK64rm:
case X86::BLSR32rm:
case X86::BLSR64rm:
case X86::TZMSK32rm:
case X86::TZMSK64rm:
// Bit extracting and clearing instructions should execute in constant time,
// and set flags.
case X86::BEXTR32rm:
case X86::BEXTR64rm:
case X86::BEXTRI32mi:
case X86::BEXTRI64mi:
case X86::BZHI32rm:
case X86::BZHI64rm:
// Basic arithmetic is constant time on the input but does set flags.
case X86::ADC8rm:
case X86::ADC16rm:
case X86::ADC32rm:
case X86::ADC64rm:
case X86::ADCX32rm:
case X86::ADCX64rm:
case X86::ADD8rm:
case X86::ADD16rm:
case X86::ADD32rm:
case X86::ADD64rm:
case X86::ADOX32rm:
case X86::ADOX64rm:
case X86::AND8rm:
case X86::AND16rm:
case X86::AND32rm:
case X86::AND64rm:
case X86::ANDN32rm:
case X86::ANDN64rm:
case X86::OR8rm:
case X86::OR16rm:
case X86::OR32rm:
case X86::OR64rm:
case X86::SBB8rm:
case X86::SBB16rm:
case X86::SBB32rm:
case X86::SBB64rm:
case X86::SUB8rm:
case X86::SUB16rm:
case X86::SUB32rm:
case X86::SUB64rm:
case X86::XOR8rm:
case X86::XOR16rm:
case X86::XOR32rm:
case X86::XOR64rm:
// Check whether the EFLAGS implicit-def is dead. We assume that this will
// always find the implicit-def because this code should only be reached
// for instructions that do in fact implicitly def this.
if (!MI.findRegisterDefOperand(X86::EFLAGS)->isDead()) {
// If we would clobber EFLAGS that are used, just bail for now.
LLVM_DEBUG(dbgs() << " Unable to harden post-load due to EFLAGS: ";
MI.dump(); dbgs() << "\n");
return false;
}
// Otherwise, fallthrough to handle these the same as instructions that
// don't set EFLAGS.
LLVM_FALLTHROUGH;
// Integer multiply w/o affecting flags is still believed to be constant
// time on x86. Called out separately as this is among the most surprising
// instructions to exhibit that behavior.
case X86::MULX32rm:
case X86::MULX64rm:
// Arithmetic instructions that are both constant time and don't set flags.
case X86::RORX32mi:
case X86::RORX64mi:
case X86::SARX32rm:
case X86::SARX64rm:
case X86::SHLX32rm:
case X86::SHLX64rm:
case X86::SHRX32rm:
case X86::SHRX64rm:
// Conversions are believed to be constant time and don't set flags.
case X86::CVTTSD2SI64rm: case X86::VCVTTSD2SI64rm: case X86::VCVTTSD2SI64Zrm:
case X86::CVTTSD2SIrm: case X86::VCVTTSD2SIrm: case X86::VCVTTSD2SIZrm:
case X86::CVTTSS2SI64rm: case X86::VCVTTSS2SI64rm: case X86::VCVTTSS2SI64Zrm:
case X86::CVTTSS2SIrm: case X86::VCVTTSS2SIrm: case X86::VCVTTSS2SIZrm:
case X86::CVTSI2SDrm: case X86::VCVTSI2SDrm: case X86::VCVTSI2SDZrm:
case X86::CVTSI2SSrm: case X86::VCVTSI2SSrm: case X86::VCVTSI2SSZrm:
case X86::CVTSI642SDrm: case X86::VCVTSI642SDrm: case X86::VCVTSI642SDZrm:
case X86::CVTSI642SSrm: case X86::VCVTSI642SSrm: case X86::VCVTSI642SSZrm:
case X86::CVTSS2SDrm: case X86::VCVTSS2SDrm: case X86::VCVTSS2SDZrm:
case X86::CVTSD2SSrm: case X86::VCVTSD2SSrm: case X86::VCVTSD2SSZrm:
// AVX512 added unsigned integer conversions.
case X86::VCVTTSD2USI64Zrm:
case X86::VCVTTSD2USIZrm:
case X86::VCVTTSS2USI64Zrm:
case X86::VCVTTSS2USIZrm:
case X86::VCVTUSI2SDZrm:
case X86::VCVTUSI642SDZrm:
case X86::VCVTUSI2SSZrm:
case X86::VCVTUSI642SSZrm:
// Loads to register don't set flags.
case X86::MOV8rm:
case X86::MOV8rm_NOREX:
case X86::MOV16rm:
case X86::MOV32rm:
case X86::MOV64rm:
case X86::MOVSX16rm8:
case X86::MOVSX32rm16:
case X86::MOVSX32rm8:
case X86::MOVSX32rm8_NOREX:
case X86::MOVSX64rm16:
case X86::MOVSX64rm32:
case X86::MOVSX64rm8:
case X86::MOVZX16rm8:
case X86::MOVZX32rm16:
case X86::MOVZX32rm8:
case X86::MOVZX32rm8_NOREX:
case X86::MOVZX64rm16:
case X86::MOVZX64rm8:
return true;
}
}
int X86InstrInfo::getSPAdjust(const MachineInstr &MI) const { int X86InstrInfo::getSPAdjust(const MachineInstr &MI) const {
const MachineFunction *MF = MI.getParent()->getParent(); const MachineFunction *MF = MI.getParent()->getParent();
const TargetFrameLowering *TFI = MF->getSubtarget().getFrameLowering(); const TargetFrameLowering *TFI = MF->getSubtarget().getFrameLowering();
if (isFrameInstr(MI)) { if (isFrameInstr(MI)) {
unsigned StackAlign = TFI->getStackAlignment(); unsigned StackAlign = TFI->getStackAlignment();
int SPAdj = alignTo(getFrameSize(MI), StackAlign); int SPAdj = alignTo(getFrameSize(MI), StackAlign);
SPAdj -= getFrameAdjustment(MI); SPAdj -= getFrameAdjustment(MI);
▲ Show 20 Lines • Show All 7,980 Lines • Show Last 20 Lines

llvm/lib/Target/X86/X86SpeculativeLoadHardening.cpp

Show First 20 Lines • Show All 1,189 Lines • ▼ Show 20 Lines for (MachineBasicBlock &MBB : MF) {
// predicate state. // predicate state.
PS->SSA.AddAvailableValue(&MBB, UpdatedStateReg); PS->SSA.AddAvailableValue(&MBB, UpdatedStateReg);
} }
// Return all the newly inserted cmov instructions of the predicate state. // Return all the newly inserted cmov instructions of the predicate state.
return CMovs; return CMovs;
} }
/// Returns true if the instruction has no behavior (specified or otherwise)
/// that is based on the value of any of its register operands
///
/// A classical example of something that is inherently not data invariant is an
/// indirect jump -- the destination is loaded into icache based on the bits set
/// in the jump destination register.
///
/// FIXME: This should become part of our instruction tables.
static bool isDataInvariant(MachineInstr &MI) {
switch (MI.getOpcode()) {
default:
// By default, assume that the instruction is not data invariant.
return false;
// Some target-independent operations that trivially lower to data-invariant
// instructions.
case TargetOpcode::COPY:
case TargetOpcode::INSERT_SUBREG:
case TargetOpcode::SUBREG_TO_REG:
return true;
// On x86 it is believed that imul is constant time w.r.t. the loaded data.
// However, they set flags and are perhaps the most surprisingly constant
// time operations so we call them out here separately.
case X86::IMUL16rr:
case X86::IMUL16rri8:
case X86::IMUL16rri:
case X86::IMUL32rr:
case X86::IMUL32rri8:
case X86::IMUL32rri:
case X86::IMUL64rr:
case X86::IMUL64rri32:
case X86::IMUL64rri8:
// Bit scanning and counting instructions that are somewhat surprisingly
// constant time as they scan across bits and do other fairly complex
// operations like popcnt, but are believed to be constant time on x86.
// However, these set flags.
case X86::BSF16rr:
case X86::BSF32rr:
case X86::BSF64rr:
case X86::BSR16rr:
case X86::BSR32rr:
case X86::BSR64rr:
case X86::LZCNT16rr:
case X86::LZCNT32rr:
case X86::LZCNT64rr:
case X86::POPCNT16rr:
case X86::POPCNT32rr:
case X86::POPCNT64rr:
case X86::TZCNT16rr:
case X86::TZCNT32rr:
case X86::TZCNT64rr:
// Bit manipulation instructions are effectively combinations of basic
// arithmetic ops, and should still execute in constant time. These also
// set flags.
case X86::BLCFILL32rr:
case X86::BLCFILL64rr:
case X86::BLCI32rr:
case X86::BLCI64rr:
case X86::BLCIC32rr:
case X86::BLCIC64rr:
case X86::BLCMSK32rr:
case X86::BLCMSK64rr:
case X86::BLCS32rr:
case X86::BLCS64rr:
case X86::BLSFILL32rr:
case X86::BLSFILL64rr:
case X86::BLSI32rr:
case X86::BLSI64rr:
case X86::BLSIC32rr:
case X86::BLSIC64rr:
case X86::BLSMSK32rr:
case X86::BLSMSK64rr:
case X86::BLSR32rr:
case X86::BLSR64rr:
case X86::TZMSK32rr:
case X86::TZMSK64rr:
// Bit extracting and clearing instructions should execute in constant time,
// and set flags.
case X86::BEXTR32rr:
case X86::BEXTR64rr:
case X86::BEXTRI32ri:
case X86::BEXTRI64ri:
case X86::BZHI32rr:
case X86::BZHI64rr:
// Shift and rotate.
case X86::ROL8r1: case X86::ROL16r1: case X86::ROL32r1: case X86::ROL64r1:
case X86::ROL8rCL: case X86::ROL16rCL: case X86::ROL32rCL: case X86::ROL64rCL:
case X86::ROL8ri: case X86::ROL16ri: case X86::ROL32ri: case X86::ROL64ri:
case X86::ROR8r1: case X86::ROR16r1: case X86::ROR32r1: case X86::ROR64r1:
case X86::ROR8rCL: case X86::ROR16rCL: case X86::ROR32rCL: case X86::ROR64rCL:
case X86::ROR8ri: case X86::ROR16ri: case X86::ROR32ri: case X86::ROR64ri:
case X86::SAR8r1: case X86::SAR16r1: case X86::SAR32r1: case X86::SAR64r1:
case X86::SAR8rCL: case X86::SAR16rCL: case X86::SAR32rCL: case X86::SAR64rCL:
case X86::SAR8ri: case X86::SAR16ri: case X86::SAR32ri: case X86::SAR64ri:
case X86::SHL8r1: case X86::SHL16r1: case X86::SHL32r1: case X86::SHL64r1:
case X86::SHL8rCL: case X86::SHL16rCL: case X86::SHL32rCL: case X86::SHL64rCL:
case X86::SHL8ri: case X86::SHL16ri: case X86::SHL32ri: case X86::SHL64ri:
case X86::SHR8r1: case X86::SHR16r1: case X86::SHR32r1: case X86::SHR64r1:
case X86::SHR8rCL: case X86::SHR16rCL: case X86::SHR32rCL: case X86::SHR64rCL:
case X86::SHR8ri: case X86::SHR16ri: case X86::SHR32ri: case X86::SHR64ri:
case X86::SHLD16rrCL: case X86::SHLD32rrCL: case X86::SHLD64rrCL:
case X86::SHLD16rri8: case X86::SHLD32rri8: case X86::SHLD64rri8:
case X86::SHRD16rrCL: case X86::SHRD32rrCL: case X86::SHRD64rrCL:
case X86::SHRD16rri8: case X86::SHRD32rri8: case X86::SHRD64rri8:
// Basic arithmetic is constant time on the input but does set flags.
case X86::ADC8rr: case X86::ADC8ri:
case X86::ADC16rr: case X86::ADC16ri: case X86::ADC16ri8:
case X86::ADC32rr: case X86::ADC32ri: case X86::ADC32ri8:
case X86::ADC64rr: case X86::ADC64ri8: case X86::ADC64ri32:
case X86::ADD8rr: case X86::ADD8ri:
case X86::ADD16rr: case X86::ADD16ri: case X86::ADD16ri8:
case X86::ADD32rr: case X86::ADD32ri: case X86::ADD32ri8:
case X86::ADD64rr: case X86::ADD64ri8: case X86::ADD64ri32:
case X86::AND8rr: case X86::AND8ri:
case X86::AND16rr: case X86::AND16ri: case X86::AND16ri8:
case X86::AND32rr: case X86::AND32ri: case X86::AND32ri8:
case X86::AND64rr: case X86::AND64ri8: case X86::AND64ri32:
case X86::OR8rr: case X86::OR8ri:
case X86::OR16rr: case X86::OR16ri: case X86::OR16ri8:
case X86::OR32rr: case X86::OR32ri: case X86::OR32ri8:
case X86::OR64rr: case X86::OR64ri8: case X86::OR64ri32:
case X86::SBB8rr: case X86::SBB8ri:
case X86::SBB16rr: case X86::SBB16ri: case X86::SBB16ri8:
case X86::SBB32rr: case X86::SBB32ri: case X86::SBB32ri8:
case X86::SBB64rr: case X86::SBB64ri8: case X86::SBB64ri32:
case X86::SUB8rr: case X86::SUB8ri:
case X86::SUB16rr: case X86::SUB16ri: case X86::SUB16ri8:
case X86::SUB32rr: case X86::SUB32ri: case X86::SUB32ri8:
case X86::SUB64rr: case X86::SUB64ri8: case X86::SUB64ri32:
case X86::XOR8rr: case X86::XOR8ri:
case X86::XOR16rr: case X86::XOR16ri: case X86::XOR16ri8:
case X86::XOR32rr: case X86::XOR32ri: case X86::XOR32ri8:
case X86::XOR64rr: case X86::XOR64ri8: case X86::XOR64ri32:
// Arithmetic with just 32-bit and 64-bit variants and no immediates.
case X86::ADCX32rr: case X86::ADCX64rr:
case X86::ADOX32rr: case X86::ADOX64rr:
case X86::ANDN32rr: case X86::ANDN64rr:
// Unary arithmetic operations.
case X86::DEC8r: case X86::DEC16r: case X86::DEC32r: case X86::DEC64r:
case X86::INC8r: case X86::INC16r: case X86::INC32r: case X86::INC64r:
case X86::NEG8r: case X86::NEG16r: case X86::NEG32r: case X86::NEG64r:
// Check whether the EFLAGS implicit-def is dead. We assume that this will
// always find the implicit-def because this code should only be reached
// for instructions that do in fact implicitly def this.
if (!MI.findRegisterDefOperand(X86::EFLAGS)->isDead()) {
// If we would clobber EFLAGS that are used, just bail for now.
LLVM_DEBUG(dbgs() << " Unable to harden post-load due to EFLAGS: ";
MI.dump(); dbgs() << "\n");
return false;
}
// Otherwise, fallthrough to handle these the same as instructions that
// don't set EFLAGS.
LLVM_FALLTHROUGH;
// Unlike other arithmetic, NOT doesn't set EFLAGS.
case X86::NOT8r: case X86::NOT16r: case X86::NOT32r: case X86::NOT64r:
// Various move instructions used to zero or sign extend things. Note that we
// intentionally don't support the _NOREX variants as we can't handle that
// register constraint anyways.
case X86::MOVSX16rr8:
case X86::MOVSX32rr8: case X86::MOVSX32rr16:
case X86::MOVSX64rr8: case X86::MOVSX64rr16: case X86::MOVSX64rr32:
case X86::MOVZX16rr8:
case X86::MOVZX32rr8: case X86::MOVZX32rr16:
case X86::MOVZX64rr8: case X86::MOVZX64rr16:
case X86::MOV32rr:
// Arithmetic instructions that are both constant time and don't set flags.
case X86::RORX32ri:
case X86::RORX64ri:
case X86::SARX32rr:
case X86::SARX64rr:
case X86::SHLX32rr:
case X86::SHLX64rr:
case X86::SHRX32rr:
case X86::SHRX64rr:
// LEA doesn't actually access memory, and its arithmetic is constant time.
case X86::LEA16r:
case X86::LEA32r:
case X86::LEA64_32r:
case X86::LEA64r:
return true;
}
}
/// Returns true if the instruction has no behavior (specified or otherwise)
/// that is based on the value loaded from memory or the value of any
/// non-address register operands.
///
/// For example, if the latency of the instruction is dependent on the
/// particular bits set in any of the registers *or* any of the bits loaded from
/// memory.
///
/// A classical example of something that is inherently not data invariant is an
/// indirect jump -- the destination is loaded into icache based on the bits set
/// in the jump destination register.
///
/// FIXME: This should become part of our instruction tables.
static bool isDataInvariantLoad(MachineInstr &MI) {
switch (MI.getOpcode()) {
default:
// By default, assume that the load will immediately leak.
return false;
// On x86 it is believed that imul is constant time w.r.t. the loaded data.
// However, they set flags and are perhaps the most surprisingly constant
// time operations so we call them out here separately.
case X86::IMUL16rm:
case X86::IMUL16rmi8:
case X86::IMUL16rmi:
case X86::IMUL32rm:
case X86::IMUL32rmi8:
case X86::IMUL32rmi:
case X86::IMUL64rm:
case X86::IMUL64rmi32:
case X86::IMUL64rmi8:
// Bit scanning and counting instructions that are somewhat surprisingly
// constant time as they scan across bits and do other fairly complex
// operations like popcnt, but are believed to be constant time on x86.
// However, these set flags.
case X86::BSF16rm:
case X86::BSF32rm:
case X86::BSF64rm:
case X86::BSR16rm:
case X86::BSR32rm:
case X86::BSR64rm:
case X86::LZCNT16rm:
case X86::LZCNT32rm:
case X86::LZCNT64rm:
case X86::POPCNT16rm:
case X86::POPCNT32rm:
case X86::POPCNT64rm:
case X86::TZCNT16rm:
case X86::TZCNT32rm:
case X86::TZCNT64rm:
// Bit manipulation instructions are effectively combinations of basic
// arithmetic ops, and should still execute in constant time. These also
// set flags.
case X86::BLCFILL32rm:
case X86::BLCFILL64rm:
case X86::BLCI32rm:
case X86::BLCI64rm:
case X86::BLCIC32rm:
case X86::BLCIC64rm:
case X86::BLCMSK32rm:
case X86::BLCMSK64rm:
case X86::BLCS32rm:
case X86::BLCS64rm:
case X86::BLSFILL32rm:
case X86::BLSFILL64rm:
case X86::BLSI32rm:
case X86::BLSI64rm:
case X86::BLSIC32rm:
case X86::BLSIC64rm:
case X86::BLSMSK32rm:
case X86::BLSMSK64rm:
case X86::BLSR32rm:
case X86::BLSR64rm:
case X86::TZMSK32rm:
case X86::TZMSK64rm:
// Bit extracting and clearing instructions should execute in constant time,
// and set flags.
case X86::BEXTR32rm:
case X86::BEXTR64rm:
case X86::BEXTRI32mi:
case X86::BEXTRI64mi:
case X86::BZHI32rm:
case X86::BZHI64rm:
// Basic arithmetic is constant time on the input but does set flags.
case X86::ADC8rm:
case X86::ADC16rm:
case X86::ADC32rm:
case X86::ADC64rm:
case X86::ADCX32rm:
case X86::ADCX64rm:
case X86::ADD8rm:
case X86::ADD16rm:
case X86::ADD32rm:
case X86::ADD64rm:
case X86::ADOX32rm:
case X86::ADOX64rm:
case X86::AND8rm:
case X86::AND16rm:
case X86::AND32rm:
case X86::AND64rm:
case X86::ANDN32rm:
case X86::ANDN64rm:
case X86::OR8rm:
case X86::OR16rm:
case X86::OR32rm:
case X86::OR64rm:
case X86::SBB8rm:
case X86::SBB16rm:
case X86::SBB32rm:
case X86::SBB64rm:
case X86::SUB8rm:
case X86::SUB16rm:
case X86::SUB32rm:
case X86::SUB64rm:
case X86::XOR8rm:
case X86::XOR16rm:
case X86::XOR32rm:
case X86::XOR64rm:
// Check whether the EFLAGS implicit-def is dead. We assume that this will
// always find the implicit-def because this code should only be reached
// for instructions that do in fact implicitly def this.
if (!MI.findRegisterDefOperand(X86::EFLAGS)->isDead()) {
// If we would clobber EFLAGS that are used, just bail for now.
LLVM_DEBUG(dbgs() << " Unable to harden post-load due to EFLAGS: ";
MI.dump(); dbgs() << "\n");
return false;
}
// Otherwise, fallthrough to handle these the same as instructions that
// don't set EFLAGS.
LLVM_FALLTHROUGH;
// Integer multiply w/o affecting flags is still believed to be constant
// time on x86. Called out separately as this is among the most surprising
// instructions to exhibit that behavior.
case X86::MULX32rm:
case X86::MULX64rm:
// Arithmetic instructions that are both constant time and don't set flags.
case X86::RORX32mi:
case X86::RORX64mi:
case X86::SARX32rm:
case X86::SARX64rm:
case X86::SHLX32rm:
case X86::SHLX64rm:
case X86::SHRX32rm:
case X86::SHRX64rm:
// Conversions are believed to be constant time and don't set flags.
case X86::CVTTSD2SI64rm: case X86::VCVTTSD2SI64rm: case X86::VCVTTSD2SI64Zrm:
case X86::CVTTSD2SIrm: case X86::VCVTTSD2SIrm: case X86::VCVTTSD2SIZrm:
case X86::CVTTSS2SI64rm: case X86::VCVTTSS2SI64rm: case X86::VCVTTSS2SI64Zrm:
case X86::CVTTSS2SIrm: case X86::VCVTTSS2SIrm: case X86::VCVTTSS2SIZrm:
case X86::CVTSI2SDrm: case X86::VCVTSI2SDrm: case X86::VCVTSI2SDZrm:
case X86::CVTSI2SSrm: case X86::VCVTSI2SSrm: case X86::VCVTSI2SSZrm:
case X86::CVTSI642SDrm: case X86::VCVTSI642SDrm: case X86::VCVTSI642SDZrm:
case X86::CVTSI642SSrm: case X86::VCVTSI642SSrm: case X86::VCVTSI642SSZrm:
case X86::CVTSS2SDrm: case X86::VCVTSS2SDrm: case X86::VCVTSS2SDZrm:
case X86::CVTSD2SSrm: case X86::VCVTSD2SSrm: case X86::VCVTSD2SSZrm:
// AVX512 added unsigned integer conversions.
case X86::VCVTTSD2USI64Zrm:
case X86::VCVTTSD2USIZrm:
case X86::VCVTTSS2USI64Zrm:
case X86::VCVTTSS2USIZrm:
case X86::VCVTUSI2SDZrm:
case X86::VCVTUSI642SDZrm:
case X86::VCVTUSI2SSZrm:
case X86::VCVTUSI642SSZrm:
// Loads to register don't set flags.
case X86::MOV8rm:
case X86::MOV8rm_NOREX:
case X86::MOV16rm:
case X86::MOV32rm:
case X86::MOV64rm:
case X86::MOVSX16rm8:
case X86::MOVSX32rm16:
case X86::MOVSX32rm8:
case X86::MOVSX32rm8_NOREX:
case X86::MOVSX64rm16:
case X86::MOVSX64rm32:
case X86::MOVSX64rm8:
case X86::MOVZX16rm8:
case X86::MOVZX32rm16:
case X86::MOVZX32rm8:
case X86::MOVZX32rm8_NOREX:
case X86::MOVZX64rm16:
case X86::MOVZX64rm8:
return true;
}
}
static bool isEFLAGSLive(MachineBasicBlock &MBB, MachineBasicBlock::iterator I, static bool isEFLAGSLive(MachineBasicBlock &MBB, MachineBasicBlock::iterator I,
const TargetRegisterInfo &TRI) { const TargetRegisterInfo &TRI) {
// Check if EFLAGS are alive by seeing if there is a def of them or they // Check if EFLAGS are alive by seeing if there is a def of them or they
// live-in, and then seeing if that def is in turn used. // live-in, and then seeing if that def is in turn used.
for (MachineInstr &MI : llvm::reverse(llvm::make_range(MBB.begin(), I))) { for (MachineInstr &MI : llvm::reverse(llvm::make_range(MBB.begin(), I))) {
if (MachineOperand *DefOp = MI.findRegisterDefOperand(X86::EFLAGS)) { if (MachineOperand *DefOp = MI.findRegisterDefOperand(X86::EFLAGS)) {
// If the def is dead, then EFLAGS is not live. // If the def is dead, then EFLAGS is not live.
if (DefOp->isDead()) if (DefOp->isDead())
▲ Show 20 Lines • Show All 139 Lines • ▼ Show 20 Lines if (HardenLoads)
(IndexReg && LoadDepRegs.test(IndexReg))) (IndexReg && LoadDepRegs.test(IndexReg)))
continue; continue;
// If post-load hardening is enabled, this load is compatible with // If post-load hardening is enabled, this load is compatible with
// post-load hardening, and we aren't already going to harden one of the // post-load hardening, and we aren't already going to harden one of the
// address registers, queue it up to be hardened post-load. Notably, // address registers, queue it up to be hardened post-load. Notably,
// even once hardened this won't introduce a useful dependency that // even once hardened this won't introduce a useful dependency that
// could prune out subsequent loads. // could prune out subsequent loads.
if (EnablePostLoadHardening && isDataInvariantLoad(MI) && if (EnablePostLoadHardening && TII->isDataInvariantLoad(MI) &&
MI.getDesc().getNumDefs() == 1 && MI.getOperand(0).isReg() && MI.getDesc().getNumDefs() == 1 && MI.getOperand(0).isReg() &&
canHardenRegister(MI.getOperand(0).getReg()) && canHardenRegister(MI.getOperand(0).getReg()) &&
!HardenedAddrRegs.count(BaseReg) && !HardenedAddrRegs.count(BaseReg) &&
!HardenedAddrRegs.count(IndexReg)) { !HardenedAddrRegs.count(IndexReg)) {
HardenPostLoad.insert(&MI); HardenPostLoad.insert(&MI);
HardenedAddrRegs.insert(MI.getOperand(0).getReg()); HardenedAddrRegs.insert(MI.getOperand(0).getReg());
continue; continue;
} }
Show All 40 Lines for (MachineInstr &MI : MBB) {
// Test if this instruction is one of our post load instructions (and // Test if this instruction is one of our post load instructions (and
// remove it from the set if so). // remove it from the set if so).
if (HardenPostLoad.erase(&MI)) { if (HardenPostLoad.erase(&MI)) {
assert(!MI.isCall() && "Must not try to post-load harden a call!"); assert(!MI.isCall() && "Must not try to post-load harden a call!");
// If this is a data-invariant load, we want to try and sink any // If this is a data-invariant load, we want to try and sink any
// hardening as far as possible. // hardening as far as possible.
if (isDataInvariantLoad(MI)) { if (TII->isDataInvariantLoad(MI)) {
// Sink the instruction we'll need to harden as far as we can down // Sink the instruction we'll need to harden as far as we can down
// the graph. // the graph.
MachineInstr *SunkMI = sinkPostLoadHardenedInst(MI, HardenPostLoad); MachineInstr *SunkMI = sinkPostLoadHardenedInst(MI, HardenPostLoad);
// If we managed to sink this instruction, update everything so we // If we managed to sink this instruction, update everything so we
// harden that instruction when we reach it in the instruction // harden that instruction when we reach it in the instruction
// sequence. // sequence.
if (SunkMI != &MI) { if (SunkMI != &MI) {
▲ Show 20 Lines • Show All 333 Lines • ▼ Show 20 Linesvoid X86SpeculativeLoadHardeningPass::hardenLoadAddr(
// And restore the flags if needed. // And restore the flags if needed.
if (FlagsReg) if (FlagsReg)
restoreEFLAGS(MBB, InsertPt, Loc, FlagsReg); restoreEFLAGS(MBB, InsertPt, Loc, FlagsReg);
} }
MachineInstr *X86SpeculativeLoadHardeningPass::sinkPostLoadHardenedInst( MachineInstr *X86SpeculativeLoadHardeningPass::sinkPostLoadHardenedInst(
MachineInstr &InitialMI, SmallPtrSetImpl<MachineInstr *> &HardenedInstrs) { MachineInstr &InitialMI, SmallPtrSetImpl<MachineInstr *> &HardenedInstrs) {
assert(isDataInvariantLoad(InitialMI) && assert(TII->isDataInvariantLoad(InitialMI) &&
"Cannot get here with a non-invariant load!"); "Cannot get here with a non-invariant load!");
// See if we can sink hardening the loaded value. // See if we can sink hardening the loaded value.
auto SinkCheckToSingleUse = auto SinkCheckToSingleUse =
[&](MachineInstr &MI) -> Optional<MachineInstr *> { [&](MachineInstr &MI) -> Optional<MachineInstr *> {
Register DefReg = MI.getOperand(0).getReg(); Register DefReg = MI.getOperand(0).getReg();
// We need to find a single use which we can sink the check. We can // We need to find a single use which we can sink the check. We can
// primarily do this because many uses may already end up checked on their // primarily do this because many uses may already end up checked on their
// own. // own.
MachineInstr *SingleUseMI = nullptr; MachineInstr *SingleUseMI = nullptr;
for (MachineInstr &UseMI : MRI->use_instructions(DefReg)) { for (MachineInstr &UseMI : MRI->use_instructions(DefReg)) {
// If we're already going to harden this use, it is data invariant and // If we're already going to harden this use, it is data invariant and
// within our block. // within our block.
if (HardenedInstrs.count(&UseMI)) { if (HardenedInstrs.count(&UseMI)) {
if (!isDataInvariantLoad(UseMI)) { if (!TII->isDataInvariantLoad(UseMI)) {
// If we've already decided to harden a non-load, we must have sunk // If we've already decided to harden a non-load, we must have sunk
// some other post-load hardened instruction to it and it must itself // some other post-load hardened instruction to it and it must itself
// be data-invariant. // be data-invariant.
assert(isDataInvariant(UseMI) && assert(TII->isDataInvariant(UseMI) &&
"Data variant instruction being hardened!"); "Data variant instruction being hardened!");
continue; continue;
} }
// Otherwise, this is a load and the load component can't be data // Otherwise, this is a load and the load component can't be data
// invariant so check how this register is being used. // invariant so check how this register is being used.
const MCInstrDesc &Desc = UseMI.getDesc(); const MCInstrDesc &Desc = UseMI.getDesc();
int MemRefBeginIdx = X86II::getMemoryOperandNo(Desc.TSFlags); int MemRefBeginIdx = X86II::getMemoryOperandNo(Desc.TSFlags);
Show All 15 Lines for (MachineInstr &UseMI : MRI->use_instructions(DefReg)) {
} }
if (SingleUseMI) if (SingleUseMI)
// We already have a single use, this would make two. Bail. // We already have a single use, this would make two. Bail.
return {}; return {};
// If this single use isn't data invariant, isn't in this block, or has // If this single use isn't data invariant, isn't in this block, or has
// interfering EFLAGS, we can't sink the hardening to it. // interfering EFLAGS, we can't sink the hardening to it.
if (!isDataInvariant(UseMI) || UseMI.getParent() != MI.getParent()) if (!TII->isDataInvariant(UseMI) || UseMI.getParent() != MI.getParent())
return {}; return {};
// If this instruction defines multiple registers bail as we won't harden // If this instruction defines multiple registers bail as we won't harden
// all of them. // all of them.
if (UseMI.getDesc().getNumDefs() > 1) if (UseMI.getDesc().getNumDefs() > 1)
return {}; return {};
// If this register isn't a virtual register we can't walk uses of sanely, // If this register isn't a virtual register we can't walk uses of sanely,
▲ Show 20 Lines • Show All 438 Lines • Show Last 20 Lines