Added code to X86AsmParser::emitInstruction() to add an LFENCE after each instruction that may load, and emit a warning if it encounters an instruction that may be vulnerable, but cannot be automatically mitigated.
Details
Diff Detail
- Repository
- rG LLVM Github Monorepo
Event Timeline
Added warnings for instructions that cannot be automatically mitigating by inserting an LFENCE.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3254 | Can we move this code out to a separate method called from here? Something like performLVIMitigation? |
LGTM. This is awesome, thanks! Someone else should give another LGTM before submitting.
Separated out the load/cfi hardening functionality into two separate member functions.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3254 | I created two new methods. Let me know what you think. |
This seems like a much more robust approach for mitigating inline asm. Thanks for that!
For now, my biggest question is around layering and separation of concerns. Do we really want to do transformations in the parser? It seems like that opens the door to making the parser API surprising and less helpful. For example, if the LVI mitigation is enabled the Parse and Emit operations of the parser no longer round-trip, which is very new and very nonobvious.
I definitely understand that you put the code here because it was the one path you could find that all attempts to emit ASM would go through. And I know it's frustrating I don't have a concrete suggestion for a different approach. Still working on that.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3165 | Let's add a comment about how this list was created, or maybe a reference to public documentation if this list is pulled from there. Otherwise folks coming afterward have no way to tell if this list is correct. | |
3193 | Once the details of LVI have faded, future maintainers will really appreciate a comment here about what this mitigation *is*, what it's doing, etc. | |
3199 | Same comment as for the list above. |
First, want to apologize for the high latency -- thanks to (gestures vaguely in the air) I've been mostly offline during the day and trying to catch up at night.
I think I've probably said my piece here. Summarizing themes from my comments:
- the parser seems like a weird place to put this mitigation
- the lists of instructions feel very magic because there's no explanation of how they were arrived at. They can also be incomplete or become out of date. Consider fixing both by matching instructions by interesting properties instead of opcodes. Include an explanation of why the properties make the instructions interesting in the context of LVI.
- the warnings we fire aren't actionable
Ultimately, though, these are code-quality and maintainability issues and I'm not a maintainer. Especially since I'm only occasionally-online right now, please don't consider me a blocker for the review.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3174 | Following up on my previous comment: it seems like this is supposed to be the full set of instructions that combine a load from memory with a control flow change dependent on that load. How are we sure the list is complete? Just from a quick look at the return instructions LLVM has for X86, it looks like LRETL and LRETQ exist and would not be mitigated. Can we just look at the MCInstDesc and warn for instructions that mayLoad and mayAffectControlFlow? Then this list will stay up to date even if new instructions are added. | |
3193 | This warning should at least tell users *why* this is vulnerable, and ideally offer the next step for fixing it. | |
3213 | I didn't find any mention of these instructions, why they would "require additional considerations", or what folks should actually *do* if they hit this warning by reading https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection. Here's my best guess: Is that accurate? If so, we could explain it that way and replace the hardcoded list with hasImplicitDefOfPhysReg(X86::EFLAGS) |
LGTM, but I'm not sure whether or not this should be upstreamed due to Matt's concerns and due to my own lack of what's okay within LLVM for maintainers. Could @jyknight or @george.burgess.iv chime in? @craig.topper, is there an owner to this component who could chime in?
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3174 | I looked into this and unfortunately the mayLoad attribute is not set for any of the RET family (see https://github.com/llvm/llvm-project/blob/b1d581019f5d4d176ad70ddffee13db247a13ef1/llvm/lib/Target/X86/X86InstrControl.td#L21). Either this is a bug or I don't actually understand what the mayLoad attribute actually means. It seems pretty strange since the POPr family has mayLoad. @craig.topper : opinion? | |
3193 | The manual mitigation is a little bit more involved than what would typically be conveyed in a compiler warning. Would it be acceptable for the warning to point to the LVI deep dive document? |
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3174 | I suspect it doesn't have the mayLoad flag because from the perspective of the codegen passes that normally use that flag, the load of the return address isn't really something we would care about. It could just as well be reading the return address from some stack dedicated return address stack or a stack implemented in hardware or something. Not sure what the implications of just adding the flag would be to codegen. | |
3211 | I don't think this works if the rep prefix is on the line before the cmps/scas instruction. That will assemble to the same thing but the parser sees it as two instructions. |
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3174 | Yikes, okay. It's incredibly surprising that mayLoad isn't set for the RET family, and if anything stops working when we set it that's almost certainly a bug we want to know about. But I can accept that fixing that is out of scope for this change -- at least, for the initial version. If so, though, that makes it all the more important that we describe how we arrived at this particular list of instructions, in enough detail that someone else can retrace our steps. | |
3193 | Sure, a link to an external site seems fine. Can we please link to the specific part of the page that explains the mitigations for this specific set of instructions? | |
3211 | @craig.topper any notion what the best workaround would be? One option is keeping track of whether we've seen a REP or REPNE before this. Although perhaps a simpler approach is just not worrying about whether we saw REP. We're here to try to warn users about instructions that we can't mitigate. What's the likelihood that we see a CMPSB without REP and it turns out it _doesn't_ need mitigation? | |
3213 | And if we hit the same problem as above with RET, where somehow the instruction tables are inaccurate and we can't rely on them yet, then it's similarly okay with me if we leave the hardcoded list of instructions as long as we provide really specific comments describing why the list is all and only these instructions. |
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3193 | @mattdr Working on it! | |
3211 | @mattdr The difference is that CMPSB without REP can be mitigated simply by following it with an LFENCE. REP CMPSB must be manually decomposed into a loop, into which an LFENCE can be inserted. I wonder if it would suffice to simply emit a warning/error if a lone REP is encountered to indicate that the user may need to manually mitigate? | |
3213 | Similar to providing a link to a public document in a warning message, I think it should suffice to have a link to said document (and specifically the relevant subsection of that document) in the comment. |
There is now a more complete set of documentation for instructions that must be manually mitigated:
- The asm parser now emits a warning containing the above link whenever it encounters an instruction that cannot be automatically mitigated.
- Now emitting a warning if a REP/REPNE prefix is encountered without an accompanying instruction on the same line.
Added a CLI option to enable the inline asm hardening feature, which is now disabled by default. There is also a disclaimer in the CLI description that this feature is experimental.
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3163 | Move the command line option to the top of the file below the includes. The two other examples I know of start with "-x86-experimental-" instead of having experimental at the end. |
Whew. I still think putting this in the X86 assembly parser (so it doesn't round-trip) is crazypants, and I wish we were relying more on instruction metadata rather than hardcoded lists, but if I take those decisions as given then this implementation seems fine.
Thanks for all the new citations and your work to make the intent more obvious.
Take my "Accept" to mean "I can read this code and understand what is going on here". As for the rest, ¯\_(ツ)_/¯
llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
3191 | RET is a pseudo. It will never be parsed. | |
3200 | JMP64m_REX is a special codegen pseudo instruction. It won't be parsed. | |
3201 | binutils does not handle the FARJMP/FARCALL. Is that a miss or them or something else? |
Addressed comments from @craig.topper.
The coverage of the LVI mitigations for LLVM should match the coverage of the mitigations implemented for binutils. Both should adhere to the guidance outlined in this document: https://software.intel.com/security-software-guidance/insights/deep-dive-load-value-injection.
Mitigations have only been recommended for SGX enclave code. Since SGX enclaves do not support far procedure calls and branches (and therefore far returns), these features do not need to be covered by either mitigation tool.
I mean, yes, sure, I want that too, but that doc just says the Intel patches to GNU binutils work by "by inserting an LFENCE instruction after each instruction that performs a load". And the binutils implementation does seem to be trying to make that true?
Mitigations have only been recommended for SGX enclave code. Since SGX enclaves do not support far procedure calls and branches (and therefore far returns), these features do not need to be covered by either mitigation tool.
That reasoning also tracks, but then here is the change where binutils explicitly adds support for LRET: https://github.com/bminor/binutils-gdb/commit/a09f656b267b9a684f038fba7cadfe98e2f18892#diff-c3c1bdcf15ebcd70b899275b3486272bR4594
Anyway, if the controlling factor is "we are only mitigating code that works in SGX and that excludes instructions like <blah>" that at least merits a comment for posterity. Otherwise -- because these are just magic lists of instructions without an explanation of how they were arrived at -- there is no way to tell the difference between an oversight and a purposeful omission, and others will probably independently come to the same conclusions @craig.topper did.
My "accept" remains with previously-described provisos.
Isn't that binutils patch adding IRET (opcode 0xcf)? LRET(opcodes 0xca and 0xcb) were already there.
Anyway, if the controlling factor is "we are only mitigating code that works in SGX and that excludes instructions like <blah>" that at least merits a comment for posterity. Otherwise -- because these are just magic lists of instructions without an explanation of how they were arrived at -- there is no way to tell the difference between an oversight and a purposeful omission, and others will probably independently come to the same conclusions @craig.topper did.
My "accept" remains with previously-described provisos.
I admit I don't know the opcodes offhand, but I agree with your reading of the code. From what I can see that commit does seem to be the first time lret appears in comments, and it appears to add a number of lret test cases.
I just had a discussion with the author of that commit to binutils. He acknowledged that the inclusion of IRET and LRET was a mistake, and plans to fix it soon.
Move the command line option to the top of the file below the includes.
The two other examples I know of start with "-x86-experimental-" instead of having experimental at the end.