Fix x86 return pattern detection

[mortimer]

Summary

Replace 0xc9 (LEAVE) with 0xcb (RETF) in ret_pattern_p(). Also put 0xc3 first, since it is the most common form and will match first.

[labath]

Could you please write a test for this? You can take a look at unittests/UnwindAssembly/x86/Testx86AssemblyInspectionEngine.cpp for other x86AssemblyInspectionEngine tests.

Also, I'm curious how you found this bug. (i.e. which functionality was broken with the old implementation).

[mortimer]

For sure I can add a test.

I found this only because I was looking to fix something else (prologue detection on OpenBSD with -fret-protector) and noticed that the return pattern detect function was wrong. So nothing was broken that I found, though the inclusion of 0xc9 as a return pattern could conceivably cause problems under some circumstances. In any event, I figured it would be easy enough to submit a fix, since it’s a tiny change.

[mortimer]

Added a test to verify that the unwind state is reset after each kind of return.

[jasonmolenda]

Thanks for fixing this and adding a testcase. I don't know why 0xc9 leave was here; it's handled over in x86AssemblyInspectionEngine::leave_pattern_p. Do you have commit access?

[mortimer]

Happy to help out. :-) I do not have commit access.

[labath]

Thanks for the fix and the test.

lldb/source/Plugins/UnwindAssembly/x86/x86AssemblyInspectionEngine.cpp

if (*p == 0x89) {
 
     regno = ((*(p + 1) >> 3) & 0x7) | src_reg_prefix_bit;
     rbp_offset = offset > 0 ? offset : -offset;
     return true;
   }
   return false;
 }
 
-// ret [0xc9] or [0xc2 imm8] or [0xca imm8]
+// ret [0xc3] or [0xcb] or [0xc2 imm16] or [0xca imm16]
 bool x86AssemblyInspectionEngine::ret_pattern_p() {
   uint8_t *p = m_cur_insn;
-  return *p == 0xc9 || *p == 0xc2 || *p == 0xca || *p == 0xc3;
+  return *p == 0xc3 || *p == 0xc2 || *p == 0xca || *p == 0xcb;
 }
 
 uint32_t x86AssemblyInspectionEngine::extract_4(uint8_t *b) {
   uint32_t v = 0;
   for (int i = 3; i >= 0; i--)
     v = (v << 8) | b[i];
   return v;
 }