This is an archive of the discontinued LLVM Phabricator instance.

[ELF][MTE] Add --android-memtag-* options to synthesize ELF notes
ClosedPublic

Authored by hctim on Feb 9 2022, 3:51 PM.

Details

Summary

This ELF note is aarch64 and Android-specific. It specifies to the
dynamic loader that specific work should be scheduled to enable MTE
protection of stack and heap regions.

Current synthesis of the ".note.android.memtag" ELF note is done in the
Android build system. We'd like to move that to the compiler. This patch
adds the --memtag-stack, --memtag-heap, and --memtag-mode={async, sync,
none} flags to the linker, which synthesises the note for us.

Future changes will add -fsanitize=memtag* flags to clang which will
pass these through to lld.

Depends on D119381.

Diff Detail

Event Timeline

hctim created this revision.Feb 9 2022, 3:51 PM
hctim requested review of this revision.Feb 9 2022, 3:51 PM
Herald added a project: Restricted Project. · View Herald TranscriptFeb 9 2022, 3:51 PM
hctim updated this revision to Diff 413581.Mar 7 2022, 11:58 AM

Rebase onto landed D119381, PTAL.

Herald added a project: Restricted Project. · View Herald TranscriptMar 7 2022, 11:58 AM

(friendly ping)

I know little about MTE. Do you have a companion Bionic change? If yes, seems useful to share the link here so that people will see the big picture.

Currently --memtag-* seems Android specific while Android specific options should have android in the names.
If --memtag-* may be used by regular AArch64 programs, then using --memtag-* seems fine.

Do you expect further iterations on the option semantics? Do we reserve rights to rename and adjust semantics as appropriate, and try having little backward compatibility burden?

lld/ELF/SyntheticSections.cpp
3857

This section is modeled on Elf64_Nhdr, so it's clear these fields are related to nhdr. No need to mention this again.

3859
lld/test/ELF/aarch64-memtag-android-abi.s
4

This block of comments may be confusing to readers. Are the tags added to serve backward compatibility purposes for the old versions of Android, or are they needed by new Android?

10

(technically, any system that can parse ELF, but I'm not rewriting it in python to run on Windows...).

This sentence can be removed.

llvm-readelf -n is a robust way testing ELF note sections. There seems no point to use Python to dump the information.

The concept of memtag is not android specific, but the current implementation is. It might grow support for other OS targets in the future.

I think it would be convenient to reserve the right to change the option semantics at least until we hear about another target implementation?

MaskRay added a comment.EditedMar 14 2022, 1:48 PM

The concept of memtag is not android specific, but the current implementation is. It might grow support for other OS targets in the future.

(I will be out of town from Wednesday til next Monday and will have little time replying.)

If you are not in hurry, the ideal way forward as I see is to create a proposal on https://github.com/ARM-software/abi-aa and get it approved.
(Tag @peter.smith as an ABI approver. Tag @zatrazz who has some experience with MTE on glibc side.)
Then Android probably doesn't need to invent its own mechanism.

I think existing ld.so implementations only inspect GNU properties.
So we probably need something similar to GNU_PROPERTY_AARCH64_FEATURE_1_BTI.


If you need something soon for experiments, the Android ABI may need to diverge from the generic AArch64 ABI.
(Something similar to --pack-dyn-relocs=android+relr.)
I think Android needs to use --android-memtag-* instead of the generic option names --memtag-*.
We can consider the options experimental. When AArch64 ABI gets a proposal and Android is happy to switch to the standard, we should remove the experimental options.

I think it would be convenient to reserve the right to change the option semantics at least until we hear about another target implementation?

Agreed.

There is also a way not adding linker options.
You may just synthesize an object file according to sync/async and stack/heap:

.section .note.gnu.property, "a", %note
.p2align 3
...

and let clang Driver link it into the output.
This may be a bit inconvenient as it needs object files for all combinations, but don't need to let linker commit to an unstable interface.

Some initial thoughts.

  • If this is likely to be rolled out to other platforms such as Linux/BSD etc. It would be useful to have something common written up in the ABI. As MaskRay suggests the best way forward would be to raise an issue or a pull request https://github.com/ARM-software/abi-aa . You are of course welcome to implement an Android specific platform ABI, but it would be good to a similar one across multiple platforms if possible.
  • For BTI, for better or worse, we chose to https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#62program-property as this mechanism was going to be needed for Intel platforms. Ideally this would be documented in the SysVABI doc, but that didn't exist at the time the property was created.
  • Do you need per-object marking that a linker will aggregate? For example do you enable when at least one object needs MTE, or do you need all objects to be compatible with MTE? A per object marking that can be overridden at link time will give you a chance to diagnose objects that may not have the MTE support. It can help to have an assembler option to fabricate the note if objects are used.
  • A colleague mentioned that the choice of command line options were quite generic. If they are aarch64 and Android specific they probably ought to be behind -z and perhaps with AArch64 and Android in the name.

I know little about MTE. Do you have a companion Bionic change? If yes, seems useful to share the link here so that people will see the big picture.

The memtag code has existed on Android since Android 11 (2021): https://cs.android.com/android/platform/superproject/+/master:bionic/libc/bionic/libc_init_static.cpp;drc=ba842429a777575fb683e21c31c5d962305b93fd;l=207

Unfortunately, this makes the implementation, for Android, inflexible, as we'd like binaries built with the new toolchain to work on older versions..

Currently --memtag-* seems Android specific while Android specific options should have android in the names.
If --memtag-* may be used by regular AArch64 programs, then using --memtag-* seems fine.

If you need something soon for experiments, the Android ABI may need to diverge from the generic AArch64 ABI.
(Something similar to --pack-dyn-relocs=android+relr.)
I think Android needs to use --android-memtag-* instead of the generic option names --memtag-*.
We can consider the options experimental. When AArch64 ABI gets a proposal and Android is happy to switch to the standard, we should remove the experimental options.

I don't see renaming the current linker options to --android-memtag* as a problem, and seems to remove the concerns of "this ABI is nonstandard and potentially android-specific." Especially given that the name of the note is "Android", which can't be changed due to the aforementioned need for backwards-compatibility.

There is also a way not adding linker options.
You may just synthesize an object file according to sync/async and stack/heap:

.section .note.gnu.property, "a", %note
.p2align 3
...

and let clang Driver link it into the output.
This may be a bit inconvenient as it needs object files for all combinations, but don't need to let linker commit to an unstable interface.

This is what is currently done on Android, but we'd like to move the behaviour to the toolchain as it's necessary to implement MTE stack, as it need collaboration from the loader.

Some initial thoughts.

  • If this is likely to be rolled out to other platforms such as Linux/BSD etc. It would be useful to have something common written up in the ABI. As MaskRay suggests the best way forward would be to raise an issue or a pull request https://github.com/ARM-software/abi-aa . You are of course welcome to implement an Android specific platform ABI, but it would be good to a similar one across multiple platforms if possible.
  • For BTI, for better or worse, we chose to https://github.com/ARM-software/abi-aa/blob/main/aaelf64/aaelf64.rst#62program-property as this mechanism was going to be needed for Intel platforms. Ideally this would be documented in the SysVABI doc, but that didn't exist at the time the property was created.

GNU_PROPERTY_AARCH64_FEATURE_1_AND seems like a reasonable place to implement these features in a generic standpoint, but we still need the android-specific behaviour for backwards compatibility.

  • Do you need per-object marking that a linker will aggregate? For example do you enable when at least one object needs MTE, or do you need all objects to be compatible with MTE? A per object marking that can be overridden at link time will give you a chance to diagnose objects that may not have the MTE support. It can help to have an assembler option to fabricate the note if objects are used.

The ELF note is coupled with the executable, not the DSOs, and thus the executable decides what MTE options are enabled. In future, there will be a per-DSO progbits section for MTE global descriptors, but these progbits sections will be ignored if the main executable doesn't include the MTE ELF note. And, of course, MTE stack is enabled on a per-CU level.

  • A colleague mentioned that the choice of command line options were quite generic. If they are aarch64 and Android specific they probably ought to be behind -z and perhaps with AArch64 and Android in the name.

As above, agreed regarding the *android* naming of the notes. I don't think they need to be platform-specific, as I'd expect a generic --android-memtag* to be able to correctly synthesize notes for AArch64 and any future platforms that support memory tagging.

  • Do you need per-object marking that a linker will aggregate? For example do you enable when at least one object needs MTE, or do you need all objects to be compatible with MTE? A per object marking that can be overridden at link time will give you a chance to diagnose objects that may not have the MTE support. It can help to have an assembler option to fabricate the note if objects are used.

The ELF note is coupled with the executable, not the DSOs, and thus the executable decides what MTE options are enabled. In future, there will be a per-DSO progbits section for MTE global descriptors, but these progbits sections will be ignored if the main executable doesn't include the MTE ELF note. And, of course, MTE stack is enabled on a per-CU level.

The reason we do not use per-object feature bits is that, even though stack tagging requires a compiler flag for code instrumentation, the note is global to a process (not even a DSO). Setting PROT_MTE on stack pages can increase memory traffic (by shipping tags back and forth). I can see some common object file or a static library (crtbegin, libatomic, libunwind, etc) in the NDK built with stack tagging instrumentation, but we would not want that to impose PROT_MTE on any binary that links it.

hctim updated this revision to Diff 418720.Mar 28 2022, 3:55 PM
hctim marked 4 inline comments as done.

Rebase, rename to --android-memtag*, and address Ray's comments.

Generally looks good, with some nits

lld/ELF/Driver.cpp
725

unknown --android-memtag-mode value: memtagModeArg, should be one of ...

lld/test/ELF/aarch64-memtag-android-abi.s
9

"none-gnu" in -triple=aarch64-linux-none-gnu seems invalid.

Did you mean aarch64-none-linux-android?

15

Nit: These llvm-readelf -n %t RUN lines do not need a wrap. They aren't so long.

56

As of Android 12 stack MTE is unimplemented. ...

59

Align

68

Just place the message on one single line. This is more friendly to diagnostic search.
We don't enforce 80-line rule to test files anyway

MaskRay retitled this revision from [MTE] [lld] Add --memtag-* options to synthetise ELF notes. to [ELF][MTE] Add --android-memtag-* options to synthesize ELF notes.Mar 29 2022, 11:29 AM
hctim updated this revision to Diff 419859.Apr 1 2022, 2:15 PM
hctim marked 6 inline comments as done.

Address comments.

MaskRay accepted this revision.Apr 1 2022, 7:26 PM

LG. It will be nice to have a link to the Android bionic implementation, so that when iterating on the semantics we can get a reference.

This revision is now accepted and ready to land.Apr 1 2022, 7:26 PM
hctim added a comment.Apr 4 2022, 11:07 AM

LG. It will be nice to have a link to the Android bionic implementation, so that when iterating on the semantics we can get a reference.

Done (in the commit).