This patch includes everything we have in D36788 plus it covers PR19251. I was forced to refactor and to re-design the current implementation because it was not adopted for various use cases of prefixes. Now it should fix all known issues.
Details
Diff Detail
Event Timeline
include/llvm/MC/MCInst.h | ||
---|---|---|
165 | sence->sense |
lib/Target/X86/Disassembler/X86Disassembler.cpp | ||
---|---|---|
243 | Flags |= 1 ? | |
251 | These hard coded values for the flag bits are going to be tricky to maintain. Perhaps an enum ? enum OpPrefixFlag { OPF_OpSize = 1, OPF_AdSize = 2, OPF_REP = 4, OPF_REP = 8, }; void clearFlags() { Flags = 0; } void setFlag(OpPrefixFlag F) { Flags |= (unsigned)F; } bool isFlagSet(OpPrefixFlag F) const { return !!(Flags & (unsigned)F); } |
Craig,
You reviewed https://reviews.llvm.org/D36788 which was the first step of fixing issues with prefixes: could you review this new patch as well? In fact it's simply an extended version of D36788 and covers issues raised by echristo.
I'm not sure I can approve growing the size of MCInst. Though I can't see anyway around it. @rafael what do you think?
The new test cases don't test anything because they aren't running FileCheck.
test/MC/Disassembler/X86/prefixes-i386.s | ||
---|---|---|
1 | This needs to be named .txt not .s You also need to pass the output to FileCheck | |
test/MC/Disassembler/X86/prefixes-x86_64.s | ||
1 | This file needs be named .txt not .s You also need to pass the output to FileCheck |
I fixed tests mentioned by Craig. And about extension of MCInst: this opaque data (not simple flag) could be really useful for other components (not only for disassembler).
There was a email thread about the issues in this patch. To keep track of those emails I'm putting them here:
It was added when I’ve start poking around with prefixes, to implement the proper recognition of xaquire/xrelease (rL311309).
I can suggest some additional views to the matter at hand:
Enhancing prefix digestion by the parser is highly recommended – aforementioned FIXME note describes those issues I’ve found on a brief exploring, surly there’s more.
Currently it is emitted as a standalone instruction, which I don’t see much sense in.
IMHO, we should aggregate prefixes till we have consumed an ‘actual’ instruction, and then make queries about whether they form a legal combination, and I deem it to be the right course for the disassembler as well, unless we don’t care about tolerating nonsense in disassembly (how does others disassemblers handle such phenomena?)
Personally, I see prefixes as a part of a particular instruction, so at least on the concept level I’m in favor of ‘Flags’.
More generally, whether producing multiple MCInsts, using flag or whatever other approach – it’s technicalities.
Agreement should be first reached on what would be considered as a proper handling for both ends (parser, disassembler).
p.s. can you guyz please use Phab for further discussion? Hard (for me) to keep track on mail correspondence.
From: Andrew Tischenko [mailto:tishenandr@xenzu.com]
BTW, JFYI, I found the following comment in the source:
// FIXME: // Enhace prefixes integrity robustness. for example, following forms // are currently tolerated: // repz repnz <insn> ; GAS errors for the use of two similar prefixes // lock addq %rax, %rbx ; Destination operand must be of memory type // xacquire <insn> ; xacquire must be accompanied by 'lock'
The approach with Flag will allow to implement it.
Andrew
On 27.09.2017 12:18, Andrew Tischenko wrote:
OK, I'll try to change the assembler properly but there are some questions: Should I do it in the same patch? Currently if we have: repz repnz repe cmpsb then we produce with 'llvm-mc -triple x86_64-unknown-unknown -x86-asm-syntax=intel -show-encoding intel-syntax.s': rep # encoding: [0xf3] repne # encoding: [0xf2] rep # encoding: [0xf3] cmpsb %es:(%rdi), (%rsi) # encoding: [0xa6] but after the change we'll get only the following: rep # encoding: [0xf3] cmpsb %es:(%rdi), (%rsi) # encoding: [0xa6] Is it OK? (IMHO, Yes.) If YES, do we need any warnings here? (IMHO, No.) Andrew On 26.09.2017 22:31, Rafael Avila de Espindola wrote: The assembler and disassembler should use the same path. I would be OK with always producing 1 or N instructions, as long as both the assembler and disassembler do the same. That is, it is OK to have Flags, as long as the assembler uses that instead of creating a separate instruction for prefixes. It seems that allowing the disassembler to create multiple instructions would have the advantage of not needing Flags, but that is secondary IMHO. Cheers, Rafael Craig Topper <craig.topper@gmail.com> writes: Here's my understanding of what I think happens today. -For a very select few instructions if the AsmParser sees a repne/repe prefix it creates a special version of the instruction that has the REP bits set in TSFlags. For any other instruction it emits the repne/rep/repe as a separate MCInst. -For the disassembler if it sees a repne/repe byte at the start that it doesn't think goes with an instruction it will emit a MCInst containing just the REP. -If the disassembler encounters a repne/repe byte not at the start of the instruction that doesn't go with the instruction we drop it and don't print anything. The disassembler interface only allows us to return one instruction. So we can't return a separate repne/repe instruction and a real instruction from the same byte sequence. I don't believe the assembler can ever produce a byte sequence that hits this case, but that doesn't mean some binary couldn't contain that string of bytes created by hand. So this patch is trying to preserve the extra prefix information in the one MCInst we're allowed to emit. Maybe another option would be to allow creating multiple MCInsts from the disassembler? ~Craig On Tue, Sep 26, 2017 at 10:37 AM, Rafael Avila de Espindola < rafael.espindola@gmail.com> wrote: The question is why it is different for disassembler than for the assembler? How does the assembler handle trepne? Cheers, Rafael Andrew Tischenko <tishenandr@xenzu.com> writes: It is not a simple flag, it's some data. And this data could be useful for any other component because it's some opaque info which could be send via MCInst from one low level target component to another one. Without this (additional) data MCInst loosing (potentially very useful) info about the given instruction. Andrew On 25.09.2017 22:05, Rafael Avila de Espindola wrote: Having a flag field that is used only on disassembly seems wrong. Don't we support parsing our own output? I don't see trepne in any .s test for example. Cheers, Rafael
Rafael, Craig,
After of some investigation of AsmParser I realised that assembler modification to support Flags for proper prefixes elaboration will require signature change of at least 2 virtual functions: ParseInstruction and MatchAndEmitInstruction. The reason of such change is very simple: ParseInstruction does real parsing (and as result could track prefixes) but does not deal with MCInst while MatchAndEmitInstruction create MCInst. This signature modifications will force massive changes in the sources that's why it will be difficult to review.
The question is: could we accept the current patch as it is (only disassembler is being supported) and follow up with a new patch which will include the corresponding assembler changes?
Or the big patch is OK?
Alternatively, I could add a new X86Operand type (e.g. Prefix): as result I will not change any signature but it will be X86-specific only. What's better from your point of view?
I re-implemented assembler in case of working with X86-prefixes. Now both X86-assembler and X86-disassembler use the new Flags field from MCInst. As result now it's possible to track several prefixes for one instr and now one prefix is not a separate instruction but only is the parameter of the one. I tried to keep the current tests unchanged where it's possible. And I did not extend/change any diagnostic related to prefixes: it should be done in the follow up patches.
I added a test to cover PR32809.
Guys, could you speed up the review? In fact you already reviewed everything except X86-ASM changes: I did it to have one path for both assembler and disassembler as we agreed in our previuos discussions.
lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
2865 ↗ | (On Diff #118602) | Use isPrefix()? |
2866 ↗ | (On Diff #118602) | I'm not sure this isn't a use after free. pop_back_val is going to return a std::unique_ptr which you dereferenced, but I'm not convinced anything is keeping the std::unique_ptr alive. |
lib/Target/X86/AsmParser/X86Operand.h | ||
395 ↗ | (On Diff #118602) | isPrefix* |
lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
2866 ↗ | (On Diff #118602) | It seems that everything is OK here because C++ Utilities library Dynamic memory management std::unique_ptr typename std::add_lvalue_reference<T>::type operator*() const; (1) (since C++11) operator* and operator-> provide access to the object owned by *this. Parameters Return value
Am I right? |
I added tests covering PR21640. Now this patch covers PR7709, PR17697, PR19251, PR32809 and PR21640.
lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
2866 ↗ | (On Diff #118602) | The problem isn't with the operator*. It's about the ordering of when the destructor for the std::unqiue_ptr returned by pop_back_val is executed. I believe it will happen as soon as that line ends since it's an unnamed temporary. So the destructor will run and delete the object the unique_ptr is pointing at. But you're still holding a reference to that object. So it's a use after free. I think you should assign to Prefixes while the object is still in the vector using .back(), and then once you're done with that just call pop_back() on the vector. |
lib/Target/X86/AsmParser/X86AsmParser.cpp | ||
---|---|---|
2866 ↗ | (On Diff #118602) | I'm not sure you're right because they say about "end of scope" to invoke destructor and here the "scope" should be "if" statement. But to be absolutely safe I'll do it. Tnx. |
Thanks for fixing that. Scope only applies to named objects, the unique_ptr here has no name. https://stackoverflow.com/questions/2298781/why-do-un-named-c-objects-destruct-before-the-scope-block-ends.
LGTM other than the one comment.
sence->sense