This is an archive of the discontinued LLVM Phabricator instance.

[HWASan] Save + print registers when tag mismatch occurs in AArch64.
ClosedPublic

Authored by hctim on Mar 1 2019, 4:50 PM.

Details

Summary

This change change the instrumentation to allow users to view the registers at the point at which tag mismatch occured. Most of the heavy lifting is done in the runtime library, where we save the registers to the stack and emit unwind information. This allows us to reduce the overhead, as very little additional work needs to be done in each __hwasan_check instance.

In this implementation, the fast path of hwasan_check is unmodified. There are an additional 4 instructions (16B) emitted in the slow path in every hwasan_check instance. This may increase binary size somewhat, but as most of the work is done in the runtime library, it's manageable.

The failure trace now contains a list of registers at the point of which the failure occured, in a format similar to that of Android's tombstones. It currently has the following format:

Registers where the failure occurred (pc 0x0055555561b4):

x0  0000000000000014  x1  0000007ffffff6c0  x2  1100007ffffff6d0  x3  12000056ffffe025
x4  0000007fff800000  x5  0000000000000014  x6  0000007fff800000  x7  0000000000000001
x8  12000056ffffe020  x9  0200007700000000  x10 0200007700000000  x11 0000000000000000
x12 0000007fffffdde0  x13 0000000000000000  x14 02b65b01f7a97490  x15 0000000000000000
x16 0000007fb77376b8  x17 0000000000000012  x18 0000007fb7ed6000  x19 0000005555556078
x20 0000007ffffff768  x21 0000007ffffff778  x22 0000000000000001  x23 0000000000000000
x24 0000000000000000  x25 0000000000000000  x26 0000000000000000  x27 0000000000000000
x28 0000000000000000  x29 0000007ffffff6f0  x30 00000055555561b4

... and prints after the dump of memory tags around the buggy address.

Every register is saved exactly as it was at the point where the tag mismatch occurs, with the exception of x16/x17. These registers are used in the tag mismatch calculation as scratch registers during __hwasan_check, and cannot be saved without affecting the fast path. As these registers are designated as scratch registers for linking, there should be no important information in them that could aid in debugging.

Diff Detail

Event Timeline

hctim created this revision.Mar 1 2019, 4:50 PM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptMar 1 2019, 4:50 PM
hctim edited the summary of this revision. (Show Details)Mar 1 2019, 4:51 PM

This needs a compiler-rt test.

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S
47 ↗(On Diff #189011)

This does not match the actual file name. Intentional?

62 ↗(On Diff #189011)

This does not seem to match stack frame description in the comment above, or the IR test. It looks like x29 offset should be 16, and x30 offset should be 24.

87 ↗(On Diff #189011)

Could this use "b" instruction, so that the reporting code does not need to remove a stack frame?

95 ↗(On Diff #189011)

use NO_EXEC_STACK_DIRECTIVE

ormris removed a subscriber: ormris.Mar 4 2019, 10:33 AM
hctim updated this revision to Diff 189173.Mar 4 2019, 11:28 AM
hctim marked 3 inline comments as done.

Addressed part of eugenis@ comments. Changed to macro no-exec stack and changed DWARF file name in HWASan '.S' file.

hctim added inline comments.Mar 4 2019, 11:29 AM
compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S
47 ↗(On Diff #189011)

Nope, fixed.

62 ↗(On Diff #189011)

My interpretation is the following layout is the current one:

// |              ...                |
// |              ...                |
// | Previous stack frames...        |
// +=================================+ <-- [CFA (x29 + 16)]
// | Return address (x30) for caller |
// | of __hwasan_check_*.            |
// +---------------------------------+ <-- [&x30 == CFA - 8]
// | Frame address (x29) for caller  |
// | of __hwasan_check_*             |
// +---------------------------------+ <-- [x29 / FP]
// | Saved x1, as __hwasan_check_*   |     [&x29 == CFA - 16]
// | clobbers it.                    |
// +---------------------------------+
// | Saved x0, likewise above.       |
// +---------------------------------+ <-- [x30 / SP]

Just stepped through GDB and it looks like the frame locates the saved registers (x29/x30) in the address they're correctly saved in. Do you see a different layout than what I'm seeing?

87 ↗(On Diff #189011)

We could, but unfortunately it still sets up a frame for hwasan_tag_mismatch, but misattributes it to the hwasan_check (i.e. we have two entries for __hwasan_check at the bottom of the stack trace).

We could change it to a direct branch and avoid the extra frame by never adjusting the frame pointer, but this means we also lose any debug-ability in __hwasan_tag_mismatch. This would also nullify the comment above regarding CFA correctness.

If you'd like me to make this tradeoff, LMK.

eugenis added inline comments.Mar 4 2019, 11:44 AM
compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S
62 ↗(On Diff #189011)

You are right, I misread the code.

87 ↗(On Diff #189011)

Yeah, that would be wrong.

But it looks like now that we have a stack frame, we can easily implement -fsanitize-recover with check outlining. Does not have to be part of the same CL.

hctim marked 6 inline comments as done.Mar 4 2019, 12:54 PM
hctim updated this revision to Diff 189373.Mar 5 2019, 11:18 AM

Added a register-dump test.

eugenis added inline comments.Mar 5 2019, 11:28 AM
compiler-rt/lib/hwasan/hwasan_linux.cc
390 ↗(On Diff #189373)

check that trace && size > 0 just in case

compiler-rt/test/hwasan/TestCases/register-dump-read.c
5 ↗(On Diff #189373)

REQUIRES: aarch64-target-arch

21 ↗(On Diff #189373)

Check that the top frame in the report is as expected, with both settings of fast_unwind_on_fatal.

hctim updated this revision to Diff 189394.Mar 5 2019, 1:16 PM
  • Register stack frame fast-forward has additional validity checks.
  • Changed register dump test requirements from 'android' to 'aarch64-target-arch'.
  • Changed test to check top frame of stack trace for both options of 'fast_unwind_on_fatal'.
hctim marked 3 inline comments as done.Mar 5 2019, 1:17 PM
eugenis accepted this revision.Mar 5 2019, 1:43 PM

LGTM

This revision is now accepted and ready to land.Mar 5 2019, 1:43 PM
pcc added inline comments.Mar 5 2019, 5:08 PM
compiler-rt/lib/hwasan/CMakeLists.txt
24 ↗(On Diff #189394)

Sort alphabetically.

compiler-rt/lib/hwasan/hwasan_linux.cc
417 ↗(On Diff #189394)

The function name should be moved into the implementation namespace, i.e. prefix with __.

compiler-rt/lib/hwasan/hwasan_report.cc
440 ↗(On Diff #189394)

This should match the name of the function.

452 ↗(On Diff #189394)

The math here seems a little hard to reason about. Would it be better to store the registers in numerical order so that e.g. x0 is in frame[0], x1 is in frame[1] and so on?

compiler-rt/lib/hwasan/hwasan_tag_mismatch_aarch64.S
94 ↗(On Diff #189394)

I think this needs to be outside the #if..#endif to prevent the stack from becoming executable on non-aarch64 machines.

compiler-rt/test/hwasan/TestCases/register-dump-read.c
28 ↗(On Diff #189394)

Is there some way to test that the correct register values are being printed? Maybe one way would be compile this translation unit with a few -ffixed-x* arguments (to prevent them from being clobbered by codegen; this is now possible with r353957), insert inline asm in this function that sets the fixed registers to specific values and test for those values here.

llvm/utils/gn/secondary/compiler-rt/lib/hwasan/BUILD.gn
45 ↗(On Diff #189394)

Sort alphabetically.

hctim updated this revision to Diff 189783.Mar 7 2019, 2:11 PM
hctim marked 8 inline comments as done.

Addressed pcc's comments.

compiler-rt/lib/hwasan/hwasan_report.cc
452 ↗(On Diff #189394)

Have changed to do ordering for x2 - x28, but as x0, x1, x29, and x30 are saved in __hwasan_check they'll have to stay a bit out-of-order.

pcc added inline comments.Mar 7 2019, 4:10 PM
compiler-rt/lib/hwasan/hwasan_report.cc
452 ↗(On Diff #189394)

Not necessarily. I think you could do this in __hwasan_check_*:

stp x0, x1, [sp, #-248]!
stp x29, x30, [sp, #232]

Now all of your stores in __hwasan_tag_mismatch can look like this:

stp xN, xN+1, [sp, #(N * 8)]
hctim marked 2 inline comments as done.Mar 8 2019, 11:02 AM
hctim added inline comments.
compiler-rt/lib/hwasan/hwasan_report.cc
452 ↗(On Diff #189394)

Done, except for a minor change to ensure that the stack pointer is 16-byte aligned (see ARM procedure call standard Sec 5.2.2.1.)

hctim updated this revision to Diff 189887.Mar 8 2019, 11:02 AM
hctim marked an inline comment as done.
pcc added inline comments.Mar 8 2019, 11:06 AM
compiler-rt/lib/hwasan/hwasan_report.cc
452 ↗(On Diff #189394)

Thanks, makes sense.

llvm/test/CodeGen/AArch64/hwasan-check-memaccess.ll
46 ↗(On Diff #189887)

I think this test case needs to be updated. Same below.

hctim updated this revision to Diff 189891.Mar 8 2019, 11:15 AM
hctim marked 2 inline comments as done.
hctim added inline comments.
llvm/test/CodeGen/AArch64/hwasan-check-memaccess.ll
46 ↗(On Diff #189887)

Thanks, done.

pcc accepted this revision.Mar 8 2019, 12:49 PM

LGTM

compiler-rt/lib/hwasan/hwasan_linux.cpp
418 ↗(On Diff #189891)

nit: reformat

hctim updated this revision to Diff 189909.Mar 8 2019, 12:53 PM
hctim marked an inline comment as done.
This revision was automatically updated to reflect the committed changes.