This is an archive of the discontinued LLVM Phabricator instance.

Add inline assembly load hardening mitigation for Load Value Injection (LVI) on X86 [6/6]
ClosedPublic

Authored by sconstab on Mar 13 2020, 1:39 PM.

Details

Summary

Added code to X86AsmParser::emitInstruction() to add an LFENCE after each instruction that may load, and emit a warning if it encounters an instruction that may be vulnerable, but cannot be automatically mitigated.

Diff Detail

Event Timeline

sconstab created this revision.Mar 13 2020, 1:39 PM
sconstab updated this revision to Diff 250583.Mar 16 2020, 9:22 AM

Added warnings for instructions that cannot be automatically mitigating by inserting an LFENCE.

sconstab edited the summary of this revision. (Show Details)Mar 16 2020, 9:23 AM
sconstab retitled this revision from Add inline assembly load hardening mitigation for Load Value Injection (LVI) on X86 to Add inline assembly load hardening mitigation for Load Value Injection (LVI) on X86 [6/6].Mar 16 2020, 9:31 AM
craig.topper added inline comments.Mar 16 2020, 10:10 AM
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3254

Can we move this code out to a separate method called from here? Something like performLVIMitigation?

zbrid accepted this revision.Mar 16 2020, 11:36 AM

LGTM. This is awesome, thanks! Someone else should give another LGTM before submitting.

This revision is now accepted and ready to land.Mar 16 2020, 11:36 AM
sconstab updated this revision to Diff 250619.Mar 16 2020, 12:58 PM

Separated out the load/cfi hardening functionality into two separate member functions.

sconstab marked an inline comment as done.Mar 16 2020, 12:59 PM
sconstab added inline comments.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3254

I created two new methods. Let me know what you think.

mattdr added a subscriber: mattdr.Mar 16 2020, 2:11 PM

This seems like a much more robust approach for mitigating inline asm. Thanks for that!

For now, my biggest question is around layering and separation of concerns. Do we really want to do transformations in the parser? It seems like that opens the door to making the parser API surprising and less helpful. For example, if the LVI mitigation is enabled the Parse and Emit operations of the parser no longer round-trip, which is very new and very nonobvious.

I definitely understand that you put the code here because it was the one path you could find that all attempts to emit ASM would go through. And I know it's frustrating I don't have a concrete suggestion for a different approach. Still working on that.

llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3165

Let's add a comment about how this list was created, or maybe a reference to public documentation if this list is pulled from there. Otherwise folks coming afterward have no way to tell if this list is correct.

3193

Once the details of LVI have faded, future maintainers will really appreciate a comment here about what this mitigation *is*, what it's doing, etc.

3199

Same comment as for the list above.

sconstab updated this revision to Diff 250804.Mar 17 2020, 9:40 AM

Added comments with pointers to LVI public documentation.

First, want to apologize for the high latency -- thanks to (gestures vaguely in the air) I've been mostly offline during the day and trying to catch up at night.

I think I've probably said my piece here. Summarizing themes from my comments:

  1. the parser seems like a weird place to put this mitigation
  2. the lists of instructions feel very magic because there's no explanation of how they were arrived at. They can also be incomplete or become out of date. Consider fixing both by matching instructions by interesting properties instead of opcodes. Include an explanation of why the properties make the instructions interesting in the context of LVI.
  3. the warnings we fire aren't actionable

Ultimately, though, these are code-quality and maintainability issues and I'm not a maintainer. Especially since I'm only occasionally-online right now, please don't consider me a blocker for the review.

llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3174

Following up on my previous comment: it seems like this is supposed to be the full set of instructions that combine a load from memory with a control flow change dependent on that load.

How are we sure the list is complete? Just from a quick look at the return instructions LLVM has for X86, it looks like LRETL and LRETQ exist and would not be mitigated.

Can we just look at the MCInstDesc and warn for instructions that mayLoad and mayAffectControlFlow? Then this list will stay up to date even if new instructions are added.

3193

This warning should at least tell users *why* this is vulnerable, and ideally offer the next step for fixing it.

3213

I didn't find any mention of these instructions, why they would "require additional considerations", or what folks should actually *do* if they hit this warning by reading https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection.

Here's my best guess:
This is a list of [[ https://cpu.fyi/d/484#G7.340223 | string instructions that can be used with REP ]] and which update ZF in EFLAGS instead of relying on an a count in rcx to terminate". When used with REP or REPNE, these instructions become interesting because an attacker could use LVI to change the target's speculative control flow by injecting a value for a load. Since the instruction is indivisible, there's no place to put an LFENCE, so like RET these instructions need to be split into load and compare/branch components.

Is that accurate?

If so, we could explain it that way and replace the hardcoded list with hasImplicitDefOfPhysReg(X86::EFLAGS)

zbrid accepted this revision.Apr 2 2020, 5:11 PM
zbrid added a subscriber: jyknight.

LGTM, but I'm not sure whether or not this should be upstreamed due to Matt's concerns and due to my own lack of what's okay within LLVM for maintainers. Could @jyknight or @george.burgess.iv chime in? @craig.topper, is there an owner to this component who could chime in?

sconstab marked 2 inline comments as done.Apr 2 2020, 10:30 PM
sconstab added inline comments.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3174

I looked into this and unfortunately the mayLoad attribute is not set for any of the RET family (see https://github.com/llvm/llvm-project/blob/b1d581019f5d4d176ad70ddffee13db247a13ef1/llvm/lib/Target/X86/X86InstrControl.td#L21). Either this is a bug or I don't actually understand what the mayLoad attribute actually means. It seems pretty strange since the POPr family has mayLoad. @craig.topper : opinion?

3193

The manual mitigation is a little bit more involved than what would typically be conveyed in a compiler warning. Would it be acceptable for the warning to point to the LVI deep dive document?

sconstab marked 2 inline comments as not done.Apr 3 2020, 12:17 PM
craig.topper added inline comments.Apr 3 2020, 3:04 PM
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3174

I suspect it doesn't have the mayLoad flag because from the perspective of the codegen passes that normally use that flag, the load of the return address isn't really something we would care about. It could just as well be reading the return address from some stack dedicated return address stack or a stack implemented in hardware or something.

Not sure what the implications of just adding the flag would be to codegen.

3211

I don't think this works if the rep prefix is on the line before the cmps/scas instruction. That will assemble to the same thing but the parser sees it as two instructions.

mattdr added inline comments.Apr 7 2020, 2:56 AM
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3174

Yikes, okay. It's incredibly surprising that mayLoad isn't set for the RET family, and if anything stops working when we set it that's almost certainly a bug we want to know about.

But I can accept that fixing that is out of scope for this change -- at least, for the initial version. If so, though, that makes it all the more important that we describe how we arrived at this particular list of instructions, in enough detail that someone else can retrace our steps.

3193

Sure, a link to an external site seems fine. Can we please link to the specific part of the page that explains the mitigations for this specific set of instructions?

3211

@craig.topper any notion what the best workaround would be? One option is keeping track of whether we've seen a REP or REPNE before this.

Although perhaps a simpler approach is just not worrying about whether we saw REP. We're here to try to warn users about instructions that we can't mitigate. What's the likelihood that we see a CMPSB without REP and it turns out it _doesn't_ need mitigation?

3213

And if we hit the same problem as above with RET, where somehow the instruction tables are inaccurate and we can't rely on them yet, then it's similarly okay with me if we leave the hardcoded list of instructions as long as we provide really specific comments describing why the list is all and only these instructions.

sconstab marked 3 inline comments as done.Apr 7 2020, 1:25 PM
sconstab added inline comments.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3193

@mattdr Working on it!

3211

@mattdr The difference is that CMPSB without REP can be mitigated simply by following it with an LFENCE. REP CMPSB must be manually decomposed into a loop, into which an LFENCE can be inserted.

I wonder if it would suffice to simply emit a warning/error if a lone REP is encountered to indicate that the user may need to manually mitigate?

3213

Similar to providing a link to a public document in a warning message, I think it should suffice to have a link to said document (and specifically the relevant subsection of that document) in the comment.

sconstab updated this revision to Diff 256471.Apr 9 2020, 6:39 PM

There is now a more complete set of documentation for instructions that must be manually mitigated:

https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection#specialinstructions

  • The asm parser now emits a warning containing the above link whenever it encounters an instruction that cannot be automatically mitigated.
  • Now emitting a warning if a REP/REPNE prefix is encountered without an accompanying instruction on the same line.
sconstab updated this revision to Diff 259745.Apr 23 2020, 3:49 PM
sconstab edited the summary of this revision. (Show Details)

Added a CLI option to enable the inline asm hardening feature, which is now disabled by default. There is also a disclaimer in the CLI description that this feature is experimental.

craig.topper added inline comments.Apr 23 2020, 4:39 PM
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3163

Move the command line option to the top of the file below the includes.

The two other examples I know of start with "-x86-experimental-" instead of having experimental at the end.

sconstab updated this revision to Diff 259795.Apr 23 2020, 9:03 PM

Addressed comment by @craig.topper about the position of the CLI argument.

sconstab updated this revision to Diff 259797.Apr 23 2020, 9:08 PM

Previous diff had an error.

mattdr accepted this revision.Apr 23 2020, 11:16 PM

Whew. I still think putting this in the X86 assembly parser (so it doesn't round-trip) is crazypants, and I wish we were relying more on instruction metadata rather than hardcoded lists, but if I take those decisions as given then this implementation seems fine.

Thanks for all the new citations and your work to make the intent more obvious.

Take my "Accept" to mean "I can read this code and understand what is going on here". As for the rest, ¯\_(ツ)_/¯

This revision is now accepted and ready to land.Apr 23 2020, 11:16 PM
craig.topper added inline comments.Apr 25 2020, 6:00 PM
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp
3191

RET is a pseudo. It will never be parsed.
RETW is missing.
LRETL, LRETW, LRETQ, LRETIL, LRETIW, LRETIQ are missing based on what binutils does.

3200

JMP64m_REX is a special codegen pseudo instruction. It won't be parsed.

3201

binutils does not handle the FARJMP/FARCALL. Is that a miss or them or something else?

sconstab updated this revision to Diff 260173.Apr 26 2020, 10:12 AM

Addressed comments from @craig.topper.

The coverage of the LVI mitigations for LLVM should match the coverage of the mitigations implemented for binutils. Both should adhere to the guidance outlined in this document: https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection.

Mitigations have only been recommended for SGX enclave code. Since SGX enclaves do not support far procedure calls and branches (and therefore far returns), these features do not need to be covered by either mitigation tool.

Herald added a project: Restricted Project. · View Herald TranscriptApr 26 2020, 10:12 AM
sconstab updated this revision to Diff 261970.May 4 2020, 5:12 PM

Rebase onto master.

mattdr accepted this revision.EditedMay 7 2020, 2:59 PM

The coverage of the LVI mitigations for LLVM should match the coverage of the mitigations implemented for binutils. Both should adhere to the guidance outlined in this document: https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection.

I mean, yes, sure, I want that too, but that doc just says the Intel patches to GNU binutils work by "by inserting an LFENCE instruction after each instruction that performs a load". And the binutils implementation does seem to be trying to make that true?

Mitigations have only been recommended for SGX enclave code. Since SGX enclaves do not support far procedure calls and branches (and therefore far returns), these features do not need to be covered by either mitigation tool.

That reasoning also tracks, but then here is the change where binutils explicitly adds support for LRET: https://github.com/bminor/binutils-gdb/commit/a09f656b267b9a684f038fba7cadfe98e2f18892#diff-c3c1bdcf15ebcd70b899275b3486272bR4594

Anyway, if the controlling factor is "we are only mitigating code that works in SGX and that excludes instructions like <blah>" that at least merits a comment for posterity. Otherwise -- because these are just magic lists of instructions without an explanation of how they were arrived at -- there is no way to tell the difference between an oversight and a purposeful omission, and others will probably independently come to the same conclusions @craig.topper did.

My "accept" remains with previously-described provisos.

The coverage of the LVI mitigations for LLVM should match the coverage of the mitigations implemented for binutils. Both should adhere to the guidance outlined in this document: https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection.

I mean, yes, sure, I want that too, but that doc just says the Intel patches to GNU binutils work by "by inserting an LFENCE instruction after each instruction that performs a load". And the binutils implementation does seem to be trying to make that true?

Mitigations have only been recommended for SGX enclave code. Since SGX enclaves do not support far procedure calls and branches (and therefore far returns), these features do not need to be covered by either mitigation tool.

That reasoning also tracks, but then here is the change where binutils explicitly adds support for LRET: https://github.com/bminor/binutils-gdb/commit/a09f656b267b9a684f038fba7cadfe98e2f18892#diff-c3c1bdcf15ebcd70b899275b3486272bR4594

Isn't that binutils patch adding IRET (opcode 0xcf)? LRET(opcodes 0xca and 0xcb) were already there.

Anyway, if the controlling factor is "we are only mitigating code that works in SGX and that excludes instructions like <blah>" that at least merits a comment for posterity. Otherwise -- because these are just magic lists of instructions without an explanation of how they were arrived at -- there is no way to tell the difference between an oversight and a purposeful omission, and others will probably independently come to the same conclusions @craig.topper did.

My "accept" remains with previously-described provisos.

mattdr added a comment.May 7 2020, 4:35 PM

Isn't that binutils patch adding IRET (opcode 0xcf)? LRET(opcodes 0xca and 0xcb) were already there.

I admit I don't know the opcodes offhand, but I agree with your reading of the code. From what I can see that commit does seem to be the first time lret appears in comments, and it appears to add a number of lret test cases.

Isn't that binutils patch adding IRET (opcode 0xcf)? LRET(opcodes 0xca and 0xcb) were already there.

I admit I don't know the opcodes offhand, but I agree with your reading of the code. From what I can see that commit does seem to be the first time lret appears in comments, and it appears to add a number of lret test cases.

I just had a discussion with the author of that commit to binutils. He acknowledged that the inclusion of IRET and LRET was a mistake, and plans to fix it soon.

This revision was automatically updated to reflect the committed changes.