[MTE] Add -fsanitize=memtag* and friends.

[hctim]

Summary

Currently, enablement of heap MTE on Android is specified by an ELF note, which
signals to the linker to enable heap MTE. This change allows
-fsanitize=memtag-heap to synthesize these notes, rather than adding them
through the build system. We need to extend this feature to also signal the
linker to do special work for MTE globals (in future) and MTE stack (currently
implemented in the toolchain, but not implemented in the loader).

Current Android uses a non-backwards-compatible ELF note, called
".note.android.memtag". Stack MTE is an ABI break anyway, so we don't mind that
we won't be able to run executables with stack MTE on Android 11/12 devices.

The current expectation is to support the verbiage used by Android, in
that "SYNC" means MTE Synchronous mode, and "ASYNC" effectively means
"fast", using the Kernel auto-upgrade feature that allows
hardware-specific and core-specific configuration as to whether "ASYNC"
would end up being Asynchronous, Asymmetric, or Synchronous on that
particular core, whichever has a reasonable performance delta. Of
course, this is platform and loader-specific.

[MaskRay]

I haven't investigated the use case yet, just commented a few things. Please split the patch into 3:

• BinaryFormat/ELF.h (see an inline comment)
• lld/ELF
• clang

lld/ELF/SyntheticSections.h

1235 ↗

(On Diff #405798)

keep variable names alphabetically

[MaskRay]

lld/ELF/SyntheticSections.cpp

3842 ↗

(On Diff #405798)

3846 ↗

(On Diff #405798)

See GnuPropertySection::writeTo

lld/test/ELF/memtag-android-abi.s

1 ↗

(On Diff #405798)

REQUIRES: aarch64

Use # for comments. Rename this to aarch64-*

19 ↗

(On Diff #405798)

use llvm-readelf -x .note.android.memtag to dump the bytes of a section.

If this is a note section, consider llvm-readelf -n

llvm/include/llvm/BinaryFormat/ELF.h

1536 ↗

(On Diff #405798)

If Android wants to define notes, can it use a namespace NT_ANDROID_* like GNU?

Please see https://reviews.llvm.org/D107949 how I add ELF attributes and related tests when new additions are added to BinaryFormat/ELF.h

[MaskRay]

lld/ELF/SyntheticSections.cpp

3871 ↗

(On Diff #405798)

/*namesz=*/

lld/ELF/SyntheticSections.h

1195 ↗

(On Diff #405798)

/*alignment=*/4

lld/ELF/Writer.cpp

367 ↗

(On Diff #405798)

[hctim]

llvm/include/llvm/BinaryFormat/ELF.h

1536 ↗

(On Diff #405798)

Android currently defines their internal note types in this naming scheme (https://cs.android.com/android/platform/superproject/+/master:bionic/libc/private/bionic_asm_note.h).

However, it seems like these are not used outside of Android internally, so let me slap some prefixes on them. Maybe I'll even get around to updating them in the Android tree as well, who knows.

[hctim]

Address Ray's comments, and made it so that stack MTE doesn't imply heap.

[hctim]

In D118948#3295321, @MaskRay wrote:

I haven't investigated the use case yet, just commented a few things. Please split the patch into 3:

• BinaryFormat/ELF.h (see an inline comment)
• lld/ELF
• clang

Can you clarify what you'd like (from D107949) in patch #1?

[hctim]

(forgot to add REQUIRES: aarch64 to the lld test)

[eugenis]

clang/include/clang/Basic/DiagnosticDriverKinds.td

484–485

Split out renaming of memtag to memtag-stack first? This will remove a lot of diff from this patch.

clang/include/clang/Driver/Options.td

1646

Let's make "sync" the default mode.

lld/ELF/Config.h

346 ↗

(On Diff #405842)

NT_MEMTAG_DEFAULT -> NT_MEMTAG_LEVEL_NONE

lld/ELF/Driver.cpp

691 ↗

(On Diff #405842)

Should it be required? We could assume default Sync here the same as in clang frontend, that's one less flag to pass around or list in manual linker invocations.

[MaskRay]

In D118948#3295589, @hctim wrote:

In D118948#3295321, @MaskRay wrote:

I haven't investigated the use case yet, just commented a few things. Please split the patch into 3:

• BinaryFormat/ELF.h (see an inline comment)
• lld/ELF
• clang

Can you clarify what you'd like (from D107949) in patch #1?

For patch #1, make yaml2obj support the NT_* tags (grep NT_ llvm-readobj/ELF/note-*.test) and obj2yaml able to dump the tag. Add llvm-readobj support.

llvm-objdump support may be unneeded.

[MaskRay]

lld/ELF/Driver.cpp

690 ↗

(On Diff #405842)

lld uses the diagnostic format specified by https://llvm.org/docs/CodingStandards.html#error-and-warning-messages

Please see other error and warn messages. No capitalization. No trailing perioid.

lld/ELF/Options.td

728 ↗

(On Diff #405842)

B/Eq is for legacy options which accept single-dash forms. Newer options should use BB and EEq.

lld/ELF/SyntheticSections.h

1195 ↗

(On Diff #405798)

no space after /*

lld/test/ELF/aarch64-memtag-android-abi.s

3 ↗

(On Diff #405842)

In lld and llvm binary utilities, we use ## for non-RUN-non-CHECK comments.

[hctim]

Pushed the [1/3] ELF note parsing over to D119381. Will leave this review for the clang-side changes, and send a new patch for the LLD changes. All of your comments are completed in my internal branch, and I'll upload that after the lld changes are landed.

clang/include/clang/Basic/DiagnosticDriverKinds.td

484–485

splitting into elf -> lld -> clang as per Ray's suggestion, should reduce the diff enough.

[hctim]

Rebase onto D119384, removing all the llvm and lld-specific parts. This patch now only contains the clang-specific bits.

[MaskRay]

clang/include/clang/Driver/SanitizerArgs.h

67

clang/lib/Driver/SanitizerArgs.cpp

657

Prefer the more common == to equals

clang/lib/Driver/ToolChains/CommonArgs.cpp

1015

1017

clang/test/Driver/memtag-ld.c

2

Pure driver tests (-###) generally don't need REQUIRES: aarch64-registered-target. You can test that the test should still pass by removing AArch64 from LLVM_TARGETS_TO_BUILD

clang/test/Lexer/has_feature_memtag_heap.cpp

1 ↗

(On Diff #413625)

The file can be merged into has_feature_memtag_stack.cpp. Just rename one MemTagSanitizerEnabled to avoid duplicate.

clang/test/Lexer/has_feature_memtag_stack.cpp

2 ↗

(On Diff #413625)

%clang is generally only used in test/Driver tests. (There are violations but the patch should not add more).

[hctim]

clang/test/Lexer/has_feature_memtag_stack.cpp

2 ↗

(On Diff #413625)

It looks like the SanitizerGroups aren't parsed in in the cc1 invocation, given that using -cc1 provides an error of:

error: invalid value 'memtag' in '-fsanitize='

non-cc1 invocations are thus standard (see clang/test/Lexer/has_feature_undefined_behavior_sanitizer.cpp for an example) for testing sanitizers.

(fwiw - I couldn't find out *why* there's a discrepancy between sanitizers and sanitizer groups, otherwise we could potentially fix it, but it would only be for the sake of "making tests all use _cc1")

[hctim]

(comments)

[MaskRay]

clang/test/CodeGen/memtag-attr.cpp

13

By placing a space before HasSanitizeMemTag and placing a (, we test that the function name is exact.

clang/test/Driver/memtag-ld.c

37

This will be easier to read if you make the strings after : aligned.

Is -memtag-mode=async --android-memtag-mode=async now?

39

Prefer CHECK-HEAP-SAME to CHECK-HEAP whenever feasible.

[hctim]

(comments)

[hctim]

clang/test/Driver/memtag-ld.c

37

thanks for the catch

39

There's no other CHECK-HEAP before this, so -SAME not possible

[tschuett]

Is -fsanitize=memtag-heap Android specific or target independent? It passes Android flags to the linker?!?

I missed the IsAndroid checks. Sorry.

[hctim]

In D118948#3427344, @tschuett wrote:

Is -fsanitize=memtag-heap Android specific or target independent? It passes Android flags to the linker?!?

The frontend flag should be target-independent. The clang driver knows to only pass Android flags to the linker where TC.getTriple().isAndroid() is true. This should allow a glibc-based implementation to live under the same -fsanitize=memtag-heap flag, just chainging the implementation based on the target.

[MaskRay]

In D118948#3427401, @hctim wrote:

In D118948#3427344, @tschuett wrote:

Is -fsanitize=memtag-heap Android specific or target independent? It passes Android flags to the linker?!?

The frontend flag should be target-independent. The clang driver knows to only pass Android flags to the linker where TC.getTriple().isAndroid() is true. This should allow a glibc-based implementation to live under the same -fsanitize=memtag-heap flag, just chainging the implementation based on the target.

Since this driver option has no-op for glibc/musl/other ELF OS, by convention it should report an err_drv_unsupported_opt_for_target error.
This allows configure-time detection.

[hctim]

.

[hctim]

In D118948#3428157, @MaskRay wrote:

Since this driver option has no-op for glibc/musl/other ELF OS, by convention it should report an err_drv_unsupported_opt_for_target error.
This allows configure-time detection.

Done.

[MaskRay]

clang/include/clang/Basic/DiagnosticDriverKinds.td

484–485

The convention is to omit period for the last sentence.

clang/include/clang/Driver/Options.td

1646

Omit period for the last sentence

clang/lib/Driver/ToolChains/CommonArgs.cpp

1013

clang/test/Lexer/has_feature_memtag.cpp

5

[hctim]

Comments

[hctim]

Over to @eugenis for any final touches

[eugenis]

LGTM

clang/include/clang/Basic/DiagnosticDriverKinds.td

484–485

Mention armv9a, too?
Something like "Try compiling with -march=armv9a, or -march=armv8a+memtag"

[hctim]

.

clang/include/clang/Basic/DiagnosticDriverKinds.td

484–485

mentioned it similar to armv8, as mentioned we're not 100% clear whether MTE is mandatory on armv9.

[aaron.ballman]

I'm sorry, but I had to revert this (in 4aaf25b4f7d7695ada313a3159d234b84d2896b2) as it broke the bots.

https://lab.llvm.org/buildbot/#/builders/109/builds/36233

[hctim]

Thanks, I'll take a look.

clang/include/clang/Basic/DiagnosticDriverKinds.td

def note_use_dashdash : Note<
   "use '--' to treat subsequent arguments as filenames">;
 
 def err_drv_ropi_rwpi_incompatible_with_pic : Error<
   "embedded and GOT-based position independence are incompatible">;
 def err_drv_ropi_incompatible_with_cxx : Error<
   "ROPI is not compatible with c++">;
 
 def err_stack_tagging_requires_hardware_feature : Error<
-  "'-fsanitize=memtag' requires hardware support (+memtag)">;
+  "'-fsanitize=memtag-stack' requires hardware support (+memtag). For Armv8, "
+  "try compiling with -march=armv8a+memtag.">;
 
 def err_cmse_pi_are_incompatible : Error<
   "cmse is not compatible with %select{RWPI|ROPI}0">;
 
 def warn_target_unsupported_nan2008 : Warning<
   "ignoring '-mnan=2008' option because the '%0' architecture does not support it">,
   InGroup<UnsupportedNan>;
 def warn_target_unsupported_nanlegacy : Warning<

clang/include/clang/Basic/Features.def

 FEATURE(address_sanitizer,
         LangOpts.Sanitize.hasOneOf(SanitizerKind::Address |
                                    SanitizerKind::KernelAddress))
 FEATURE(leak_sanitizer,
         LangOpts.Sanitize.has(SanitizerKind::Leak))
 FEATURE(hwaddress_sanitizer,
         LangOpts.Sanitize.hasOneOf(SanitizerKind::HWAddress |
                                    SanitizerKind::KernelHWAddress))
-FEATURE(memtag_sanitizer, LangOpts.Sanitize.has(SanitizerKind::MemTag))
+FEATURE(memtag_stack,
+        LangOpts.Sanitize.has(SanitizerKind::MemtagStack))
+FEATURE(memtag_heap,
+        LangOpts.Sanitize.has(SanitizerKind::MemtagHeap))
 FEATURE(xray_instrument, LangOpts.XRayInstrument)
 FEATURE(undefined_behavior_sanitizer,
         LangOpts.Sanitize.hasOneOf(SanitizerKind::Undefined))
 FEATURE(coverage_sanitizer, LangOpts.SanitizeCoverage)
 FEATURE(assume_nonnull, true)
 FEATURE(attribute_analyzer_noreturn, true)
 FEATURE(attribute_availability, true)
 FEATURE(attribute_availability_with_message, true)

clang/include/clang/Basic/Sanitizers.def

 
 // Hardware-assisted AddressSanitizer
 SANITIZER("hwaddress", HWAddress)
 
 // Kernel Hardware-assisted AddressSanitizer (KHWASan)
 SANITIZER("kernel-hwaddress", KernelHWAddress)
 
 // A variant of AddressSanitizer using AArch64 MTE extension.
-SANITIZER("memtag", MemTag)
+SANITIZER("memtag-stack", MemtagStack)
+SANITIZER("memtag-heap", MemtagHeap)
+SANITIZER_GROUP("memtag", MemTag, MemtagStack | MemtagHeap)
 
 // MemorySanitizer
 SANITIZER("memory", Memory)
 
 // Kernel MemorySanitizer (KMSAN)
 SANITIZER("kernel-memory", KernelMemory)
 
 // libFuzzer

clang/include/clang/Driver/Options.td

def fno_sanitize_memory_track_origins : Flag<["-"], "fno-sanitize-memory-track-origins">,
                                         Flags<[CoreOption, NoXarchOption]>,
                                         HelpText<"Disable origins tracking in MemorySanitizer">;
 def fsanitize_address_outline_instrumentation : Flag<["-"], "fsanitize-address-outline-instrumentation">,
                                                 Group<f_clang_Group>,
                                                 HelpText<"Always generate function calls for address sanitizer instrumentation">;
 def fno_sanitize_address_outline_instrumentation : Flag<["-"], "fno-sanitize-address-outline-instrumentation">,
                                                    Group<f_clang_Group>,
                                                    HelpText<"Use default code inlining logic for the address sanitizer">;
+def fsanitize_memtag_mode_EQ : Joined<["-"], "fsanitize-memtag-mode=">,
+                                        Group<f_clang_Group>,
+                                        HelpText<"Set default MTE mode to 'sync' (default) or 'async'.">;
 def fsanitize_hwaddress_experimental_aliasing
   : Flag<["-"], "fsanitize-hwaddress-experimental-aliasing">,
     Group<f_clang_Group>,
     HelpText<"Enable aliasing mode in HWAddressSanitizer">;
 def fno_sanitize_hwaddress_experimental_aliasing
   : Flag<["-"], "fno-sanitize-hwaddress-experimental-aliasing">,
     Group<f_clang_Group>, Flags<[CoreOption, NoXarchOption]>,
     HelpText<"Disable aliasing mode in HWAddressSanitizer">;

clang/include/clang/Driver/SanitizerArgs.h

class SanitizerArgs {
   bool MinimalRuntime = false;
   // True if cross-dso CFI support if provided by the system (i.e. Android).
   bool ImplicitCfiRuntime = false;
   bool NeedsMemProfRt = false;
   bool HwasanUseAliases = false;
   llvm::AsanDetectStackUseAfterReturnMode AsanUseAfterReturn =
       llvm::AsanDetectStackUseAfterReturnMode::Invalid;
 
+  std::string MemtagMode;
+
 public:
   /// Parses the sanitizer arguments from an argument list.
   SanitizerArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
                 bool DiagnoseErrors = true);
 
   bool needsSharedRt() const { return SharedRuntime; }
 
   bool needsMemProfRt() const { return NeedsMemProfRt; }
public:
   bool requiresMinimalRuntime() const { return MinimalRuntime; }
   bool needsDfsanRt() const { return Sanitizers.has(SanitizerKind::DataFlow); }
   bool needsSafeStackRt() const { return SafeStackRuntime; }
   bool needsCfiRt() const;
   bool needsCfiDiagRt() const;
   bool needsStatsRt() const { return Stats; }
   bool needsScudoRt() const { return Sanitizers.has(SanitizerKind::Scudo); }
 
+  bool hasMemTag() const { return hasMemtagHeap() || hasMemtagStack(); }
+  bool hasMemtagHeap() const {
+    return Sanitizers.has(SanitizerKind::MemtagHeap);
+  }
+  bool hasMemtagStack() const {
+    return Sanitizers.has(SanitizerKind::MemtagStack);
+  }
+  const std::string &getMemtagMode() const {
+    assert(!MemtagMode.empty());
+    return MemtagMode;
+  }
+
   bool requiresPIE() const;
   bool needsUnwindTables() const;
   bool needsLTO() const;
   bool linkRuntimes() const { return LinkRuntimes; }
   bool linkCXXRuntimes() const { return LinkCXXRuntimes; }
   bool hasCrossDsoCfi() const { return CfiCrossDso; }
   bool hasAnySanitizer() const { return !Sanitizers.empty(); }
   void addArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
                llvm::opt::ArgStringList &CmdArgs, types::ID InputType) const;
 };
 
 }  // namespace driver
 }  // namespace clang
 
 #endif

clang/lib/CodeGen/CGDeclCXX.cpp

llvm::Function *CodeGenModule::CreateGlobalInitOrCleanUpFunction(
   if (getLangOpts().Sanitize.has(SanitizerKind::HWAddress) &&
       !isInNoSanitizeList(SanitizerKind::HWAddress, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeHWAddress);
 
   if (getLangOpts().Sanitize.has(SanitizerKind::KernelHWAddress) &&
       !isInNoSanitizeList(SanitizerKind::KernelHWAddress, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeHWAddress);
 
-  if (getLangOpts().Sanitize.has(SanitizerKind::MemTag) &&
-      !isInNoSanitizeList(SanitizerKind::MemTag, Fn, Loc))
+  if (getLangOpts().Sanitize.has(SanitizerKind::MemtagStack) &&
+      !isInNoSanitizeList(SanitizerKind::MemtagStack, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeMemTag);
 
   if (getLangOpts().Sanitize.has(SanitizerKind::Thread) &&
       !isInNoSanitizeList(SanitizerKind::Thread, Fn, Loc))
     Fn->addFnAttr(llvm::Attribute::SanitizeThread);
 
   if (getLangOpts().Sanitize.has(SanitizerKind::Memory) &&
       !isInNoSanitizeList(SanitizerKind::Memory, Fn, Loc))

clang/lib/CodeGen/CodeGenFunction.cpp

if (ShouldSkipSanitizerInstrumentation()) {
     CurFn->addFnAttr(llvm::Attribute::DisableSanitizerInstrumentation);
   } else {
     // Apply sanitizer attributes to the function.
     if (SanOpts.hasOneOf(SanitizerKind::Address | SanitizerKind::KernelAddress))
       Fn->addFnAttr(llvm::Attribute::SanitizeAddress);
     if (SanOpts.hasOneOf(SanitizerKind::HWAddress |
                          SanitizerKind::KernelHWAddress))
       Fn->addFnAttr(llvm::Attribute::SanitizeHWAddress);
-    if (SanOpts.has(SanitizerKind::MemTag))
+    if (SanOpts.has(SanitizerKind::MemtagStack))
       Fn->addFnAttr(llvm::Attribute::SanitizeMemTag);
     if (SanOpts.has(SanitizerKind::Thread))
       Fn->addFnAttr(llvm::Attribute::SanitizeThread);
     if (SanOpts.hasOneOf(SanitizerKind::Memory | SanitizerKind::KernelMemory))
       Fn->addFnAttr(llvm::Attribute::SanitizeMemory);
   }
   if (SanOpts.has(SanitizerKind::SafeStack))
     Fn->addFnAttr(llvm::Attribute::SafeStack);

clang/lib/CodeGen/CodeGenModule.cpp

void CodeGenModule::Release() {
   // needed for LTO, where the function attributes are inside bitcode
   // serialised into a global variable by the time build attributes are
   // emitted, so we can't access them.
   if (Context.getTargetInfo().hasFeature("ptrauth") &&
       LangOpts.getSignReturnAddressScope() !=
           LangOptions::SignReturnAddressScopeKind::None)
     getModule().addModuleFlag(llvm::Module::Override,
                               "sign-return-address-buildattr", 1);
-  if (LangOpts.Sanitize.has(SanitizerKind::MemTag))
+  if (LangOpts.Sanitize.has(SanitizerKind::MemtagStack))
     getModule().addModuleFlag(llvm::Module::Override,
                               "tag-stack-memory-buildattr", 1);
 
   if (Arch == llvm::Triple::thumb || Arch == llvm::Triple::thumbeb ||
       Arch == llvm::Triple::arm || Arch == llvm::Triple::armeb ||
       Arch == llvm::Triple::aarch64 || Arch == llvm::Triple::aarch64_32 ||
       Arch == llvm::Triple::aarch64_be) {
     getModule().addModuleFlag(llvm::Module::Error, "branch-target-enforcement",

clang/lib/Driver/SanitizerArgs.cpp

 static const SanitizerMask RequiresPIE =
     SanitizerKind::DataFlow | SanitizerKind::HWAddress | SanitizerKind::Scudo;
 static const SanitizerMask NeedsUnwindTables =
     SanitizerKind::Address | SanitizerKind::HWAddress | SanitizerKind::Thread |
     SanitizerKind::Memory | SanitizerKind::DataFlow;
 static const SanitizerMask SupportsCoverage =
     SanitizerKind::Address | SanitizerKind::HWAddress |
     SanitizerKind::KernelAddress | SanitizerKind::KernelHWAddress |
-    SanitizerKind::MemTag | SanitizerKind::Memory |
-    SanitizerKind::KernelMemory | SanitizerKind::Leak |
+    SanitizerKind::MemtagStack | SanitizerKind::MemtagHeap |
+    SanitizerKind::Memory | SanitizerKind::KernelMemory | SanitizerKind::Leak |
     SanitizerKind::Undefined | SanitizerKind::Integer | SanitizerKind::Bounds |
     SanitizerKind::ImplicitConversion | SanitizerKind::Nullability |
     SanitizerKind::DataFlow | SanitizerKind::Fuzzer |
     SanitizerKind::FuzzerNoLink | SanitizerKind::FloatDivideByZero |
     SanitizerKind::SafeStack | SanitizerKind::ShadowCallStack |
     SanitizerKind::Thread | SanitizerKind::ObjCCast;
 static const SanitizerMask RecoverableByDefault =
     SanitizerKind::Undefined | SanitizerKind::Integer |
static const SanitizerMask TrappingSupported =
     SanitizerKind::ObjCCast;
 static const SanitizerMask TrappingDefault = SanitizerKind::CFI;
 static const SanitizerMask CFIClasses =
     SanitizerKind::CFIVCall | SanitizerKind::CFINVCall |
     SanitizerKind::CFIMFCall | SanitizerKind::CFIDerivedCast |
     SanitizerKind::CFIUnrelatedCast;
 static const SanitizerMask CompatibleWithMinimalRuntime =
     TrappingSupported | SanitizerKind::Scudo | SanitizerKind::ShadowCallStack |
-    SanitizerKind::MemTag;
+    SanitizerKind::MemtagStack | SanitizerKind::MemtagHeap;
 
 enum CoverageFeature {
   CoverageFunc = 1 << 0,
   CoverageBB = 1 << 1,
   CoverageEdge = 1 << 2,
   CoverageIndirCall = 1 << 3,
   CoverageTraceBB = 1 << 4, // Deprecated.
   CoverageTraceCmp = 1 << 5,
MsanParamRetval = Args.hasFlag(
         options::OPT_fno_sanitize_memory_param_retval, MsanParamRetval);
     NeedPIE |= !(TC.getTriple().isOSLinux() &&
                  TC.getTriple().getArch() == llvm::Triple::x86_64);
   } else {
     MsanUseAfterDtor = false;
     MsanParamRetval = false;
   }
 
+  if (AllAddedKinds & SanitizerKind::MemTag) {
+    StringRef S =
+        Args.getLastArgValue(options::OPT_fsanitize_memtag_mode_EQ, "sync");
+    if (S == "async" || S == "sync") {
+      MemtagMode = S.str();
+    } else {
+      D.Diag(clang::diag::err_drv_invalid_value_with_suggestion)
+          << "-fsanitize-memtag-mode=" << S << "{async, sync}";
+    }
+  }
+
   if (AllAddedKinds & SanitizerKind::Thread) {
     TsanMemoryAccess = Args.hasFlag(
         options::OPT_fsanitize_thread_memory_access,
         options::OPT_fno_sanitize_thread_memory_access, TsanMemoryAccess);
     TsanFuncEntryExit = Args.hasFlag(
         options::OPT_fsanitize_thread_func_entry_exit,
         options::OPT_fno_sanitize_thread_func_entry_exit, TsanFuncEntryExit);
     TsanAtomics =
void SanitizerArgs::addArgs(const ToolChain &TC, const llvm::opt::ArgList &Args,
   if (Sanitizers.hasOneOf(CFIClasses) && !TC.getTriple().isOSWindows() &&
       !Args.hasArg(options::OPT_fvisibility_EQ)) {
     TC.getDriver().Diag(clang::diag::err_drv_argument_only_allowed_with)
         << lastArgumentForMask(TC.getDriver(), Args,
                                Sanitizers.Mask & CFIClasses)
         << "-fvisibility=";
   }
 
-  if (Sanitizers.has(SanitizerKind::MemTag) && !hasTargetFeatureMTE(CmdArgs))
+  if (Sanitizers.has(SanitizerKind::MemtagStack) &&
+      !hasTargetFeatureMTE(CmdArgs))
     TC.getDriver().Diag(diag::err_stack_tagging_requires_hardware_feature);
 }
 
 SanitizerMask parseArgValues(const Driver &D, const llvm::opt::Arg *A,
                              bool DiagnoseErrors) {
   assert((A->getOption().matches(options::OPT_fsanitize_EQ) ||
           A->getOption().matches(options::OPT_fno_sanitize_EQ) ||
           A->getOption().matches(options::OPT_fsanitize_recover_EQ) ||

clang/lib/Driver/ToolChains/CommonArgs.cpp

bool tools::addSanitizerRuntimes(const ToolChain &TC, const ArgList &Args,
   // If there is a static runtime with no dynamic list, force all the symbols
   // to be dynamic to be sure we export sanitizer interface functions.
   if (AddExportDynamic)
     CmdArgs.push_back("--export-dynamic");
 
   if (SanArgs.hasCrossDsoCfi() && !AddExportDynamic)
     CmdArgs.push_back("--export-dynamic-symbol=__cfi_check");
 
+  if (SanArgs.hasMemTag()) {
+    if (!TC.getTriple().isAndroid()) {
+      TC.getDriver().Diag(diag::err_drv_unsupported_opt_for_target)
+          << ("-fsanitize=memtag*") << TC.getTriple().str();
+    }
+    CmdArgs.push_back(
+        Args.MakeArgString("--android-memtag-mode=" + SanArgs.getMemtagMode()));
+    if (SanArgs.hasMemtagHeap())
+      CmdArgs.push_back("--android-memtag-heap");
+    if (SanArgs.hasMemtagStack())
+      CmdArgs.push_back("--android-memtag-stack");
+  }
+
   return !StaticRuntimes.empty() || !NonWholeStaticRuntimes.empty();
 }
 
 bool tools::addXRayRuntime(const ToolChain&TC, const ArgList &Args, ArgStringList &CmdArgs) {
   if (Args.hasArg(options::OPT_shared))
     return false;
 
   if (TC.getXRayArgs().needsXRayRt()) {

clang/test/CodeGen/memtag-attr.cpp

 // Make sure the sanitize_memtag attribute is emitted when using MemTag sanitizer.
 // Make sure __attribute__((no_sanitize("memtag")) disables instrumentation.
 
 // RUN: %clang_cc1 -triple aarch64-unknown-linux -disable-O0-optnone \
 // RUN:   -emit-llvm -o - %s | FileCheck -check-prefix=CHECK-NO %s
 
-// RUN: %clang_cc1 -triple aarch64-unknown-linux -fsanitize=memtag \
+// RUN: %clang_cc1 -triple aarch64-unknown-linux -fsanitize=memtag-stack \
 // RUN:   -disable-O0-optnone -emit-llvm -o - %s | \
 // RUN:   FileCheck -check-prefix=CHECK-MEMTAG %s
 
-int HasSanitizeMemTag() { return 1; }
-// CHECK-NO: {{Function Attrs: mustprogress noinline nounwind$}}
-// CHECK-MEMTAG: Function Attrs: mustprogress noinline nounwind sanitize_memtag
+// RUN: %clang --target=aarch64-unknown-linux -march=armv8a+memtag \
+// RUN:   -fsanitize=memtag -disable-O0-optnone -S -emit-llvm -o - %s | \
+// RUN:   FileCheck -check-prefix=CHECK-MEMTAG %s
+
+extern "C" int HasSanitizeMemTag() { return 1; }
+// CHECK-NO: Function Attrs
+// CHECK-NO-NOT: sanitize_memtag
+// CHECK-NO-NEXT: define {{.*}} @HasSanitizeMemTag(
+// CHECK-MEMTAG: Function Attrs: {{.*}} sanitize_memtag
+// CHECK-MEMTAG-NEXT: define {{.*}} @HasSanitizeMemTag(
 
-__attribute__((no_sanitize("memtag"))) int NoSanitizeQuoteAddress() {
+extern "C" __attribute__((no_sanitize("memtag"))) int NoSanitizeQuoteAddress() {
   return 0;
 }
-// CHECK-NO: {{Function Attrs: mustprogress noinline nounwind$}}
-// CHECK-MEMTAG: {{Function Attrs: mustprogress noinline nounwind$}}
+// CHECK-NO: Function Attrs
+// CHECK-NO-NOT: sanitize_memtag
+// CHECK-NO-NEXT: define {{.*}} @NoSanitizeQuoteAddress(
+// CHECK-MEMTAG: Function Attrs
+// CHECK-MEMTAG-NOT: sanitize_memtag
+// CHECK-MEMTAG-NEXT: define {{.*}} @NoSanitizeQuoteAddress(

clang/test/Driver/fsanitize.c

 // CHECK-SANMT-SANHA: '-fsanitize=memtag' not allowed with '-fsanitize=hwaddress'
 
 // RUN: %clang -target i386-linux-android -fsanitize=memtag -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANMT-BAD-ARCH
 // RUN: %clang -target x86_64-linux-android -fsanitize=memtag -fno-rtti %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANMT-BAD-ARCH
 // CHECK-SANMT-BAD-ARCH: unsupported option '-fsanitize=memtag' for target
 
 // RUN: %clang -target aarch64-linux -fsanitize=memtag -march=armv8-a+memtag %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANMT-MT
 // CHECK-SANMT-MT: "-target-feature" "+mte"
-// CHECK-SANMT-MT-SAME: "-fsanitize=memtag"
+// CHECK-SANMT-MT-SAME: "-fsanitize=memtag-stack,memtag-heap"
 
 // RUN: %clang -target aarch64-linux -fsanitize=memtag %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANMT-NOMT-0
-// CHECK-SANMT-NOMT-0: '-fsanitize=memtag' requires hardware support (+memtag)
+// CHECK-SANMT-NOMT-0: '-fsanitize=memtag-stack' requires hardware support (+memtag)
 
 // RUN: %clang -target aarch64-linux -fsanitize=memtag -I +mte %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-SANMT-NOMT-1
-// CHECK-SANMT-NOMT-1: '-fsanitize=memtag' requires hardware support (+memtag)
+// CHECK-SANMT-NOMT-1: '-fsanitize=memtag-stack' requires hardware support (+memtag)
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address -fsanitize-address-use-after-scope %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-USE-AFTER-SCOPE
 // RUN: %clang_cl --target=x86_64-windows -fsanitize=address -fsanitize-address-use-after-scope -### -- %s 2>&1 | FileCheck %s --check-prefix=CHECK-USE-AFTER-SCOPE
 // CHECK-USE-AFTER-SCOPE: -cc1{{.*}}-fsanitize-address-use-after-scope
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=address -fno-sanitize-address-use-after-scope %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-USE-AFTER-SCOPE-OFF
 // RUN: %clang_cl --target=x86_64-windows -fsanitize=address -fno-sanitize-address-use-after-scope -### -- %s 2>&1 | FileCheck %s --check-prefix=CHECK-USE-AFTER-SCOPE-OFF
 // CHECK-USE-AFTER-SCOPE-OFF-NOT: -cc1{{.*}}address-use-after-scope

 // RUN: %clang -target x86_64-linux-gnu -fsanitize=integer -fsanitize-trap=integer %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-INTSAN-TRAP
 // CHECK-INTSAN-TRAP: "-fsanitize-trap=integer-divide-by-zero,shift-base,shift-exponent,signed-integer-overflow,unsigned-integer-overflow,unsigned-shift-base,implicit-unsigned-integer-truncation,implicit-signed-integer-truncation,implicit-integer-sign-change"
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=integer -fsanitize-minimal-runtime %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-INTSAN-MINIMAL
 // CHECK-INTSAN-MINIMAL: "-fsanitize=integer-divide-by-zero,shift-base,shift-exponent,signed-integer-overflow,unsigned-integer-overflow,unsigned-shift-base,implicit-unsigned-integer-truncation,implicit-signed-integer-truncation,implicit-integer-sign-change"
 // CHECK-INTSAN-MINIMAL: "-fsanitize-minimal-runtime"
 
 // RUN: %clang -target aarch64-linux-android -march=armv8-a+memtag -fsanitize=memtag -fsanitize-minimal-runtime %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-MEMTAG-MINIMAL
-// CHECK-MEMTAG-MINIMAL: "-fsanitize=memtag"
+// CHECK-MEMTAG-MINIMAL: "-fsanitize=memtag-stack,memtag-heap"
 // CHECK-MEMTAG-MINIMAL: "-fsanitize-minimal-runtime"
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=undefined -fsanitize=function -fsanitize-minimal-runtime %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-UBSAN-FUNCTION-MINIMAL
 // CHECK-UBSAN-FUNCTION-MINIMAL: error: invalid argument '-fsanitize=function' not allowed with '-fsanitize-minimal-runtime'
 
 // RUN: %clang -target x86_64-linux-gnu -fsanitize=undefined -fsanitize=vptr -fsanitize-minimal-runtime %s -### 2>&1 | FileCheck %s --check-prefix=CHECK-UBSAN-VPTR-MINIMAL
 // CHECK-UBSAN-VPTR-MINIMAL: error: invalid argument '-fsanitize=vptr' not allowed with '-fsanitize-minimal-runtime'
 

clang/test/Driver/memtag-ld.c

+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag %s 2>&1 | FileCheck %s \
+// RUN:   --check-prefixes=CHECK-SYNC,CHECK-HEAP,CHECK-STACK
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag-stack %s 2>&1 | FileCheck %s \
+// RUN:   --check-prefixes=CHECK-SYNC,CHECK-NO-HEAP,CHECK-STACK
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag-heap %s 2>&1 | FileCheck %s \
+// RUN:   --check-prefixes=CHECK-SYNC,CHECK-HEAP,CHECK-NO-STACK
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag -fsanitize-memtag-mode=async %s 2>&1 | FileCheck %s \
+// RUN:   --check-prefixes=CHECK-ASYNC,CHECK-HEAP,CHECK-STACK
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag-stack -fsanitize-memtag-mode=async %s 2>&1 \
+// RUN:   | FileCheck %s \
+// RUN:   --check-prefixes=CHECK-ASYNC,CHECK-NO-HEAP,CHECK-STACK
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag-heap -fsanitize-memtag-mode=async %s 2>&1 \
+// RUN:   | FileCheck %s \
+// RUN:   --check-prefixes=CHECK-ASYNC,CHECK-HEAP,CHECK-NO-STACK
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag-heap -fsanitize-memtag-mode=asymm %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefixes=CHECK-INVALID-MODE
+
+// RUN: %clang -### --target=aarch64-linux-android -march=armv8+memtag \
+// RUN:   -fsanitize=memtag-stack -fsanitize=memtag-heap \
+// RUN:   -fsanitize-memtag-mode=asymm -fno-sanitize=memtag %s 2>&1 \
+// RUN:   | FileCheck %s --check-prefixes=CHECK-NONE
+
+// CHECK-ASYNC:             ld{{.*}} "--android-memtag-mode=async"
+// CHECK-SYNC:              ld{{.*}} "--android-memtag-mode=sync"
+// CHECK-HEAP:              "--android-memtag-heap"
+// CHECK-NO-HEAP-NOT:       "--android-memtag-heap"
+// CHECK-STACK:             "--android-memtag-stack"
+// CHECK-NO-STACK-NOT:      "--android-memtag-stack"
+// CHECK-INVALID-MODE:      invalid value 'asymm' in '-fsanitize-memtag-mode=',
+// CHECK-INVALID-MODE-SAME: expected one of: {async, sync}
+// CHECK-NONE-NOT:          ld{{.*}} "--android-memtag
+
+void f() {}

clang/test/Driver/memtag-stack.c

 // RUN: %clang     -target aarch64-unknown-linux -march=armv8+memtag -fsanitize=memtag -mllvm -stack-safety-print=1 %s -S -o - 2>&1 | FileCheck %s --check-prefix=CHECK-NO-SAFETY
 // RUN: %clang -O1 -target aarch64-unknown-linux -march=armv8+memtag -fsanitize=memtag -mllvm -stack-safety-print=1 %s -S -o - 2>&1 | FileCheck %s --check-prefix=CHECK-SAFETY
 // RUN: %clang -O2 -target aarch64-unknown-linux -march=armv8+memtag -fsanitize=memtag -mllvm -stack-safety-print=1 %s -S -o - 2>&1 | FileCheck %s --check-prefix=CHECK-SAFETY
 // RUN: %clang -O3 -target aarch64-unknown-linux -march=armv8+memtag -fsanitize=memtag -mllvm -stack-safety-print=1 %s -S -o - 2>&1 | FileCheck %s --check-prefix=CHECK-SAFETY
 
 // REQUIRES: aarch64-registered-target
 
 int z;
 __attribute__((noinline)) void use(int *p) { *p = z; }
 int foo() {
   int x;
   use(&x);
   return x;
 }
 
 // CHECK-NO-SAFETY-NOT: allocas uses
 
 // CHECK-SAFETY-LABEL: @foo
 // CHECK-SAFETY-LABEL: allocas uses:
 // CHECK-SAFETY-NEXT: [4]: [0,4)

clang/test/Lexer/has_feature_memtag.cpp

+// RUN: %clang_cc1 -E -fsanitize=memtag-stack %s -o - | FileCheck --check-prefix=CHECK-MEMTAG-STACK %s
+// RUN: %clang_cc1 -E -fsanitize=memtag-heap %s -o -  | FileCheck --check-prefix=CHECK-MEMTAG-HEAP %s
+// RUN: %clang -E -fsanitize=memtag --target=aarch64-unknown-linux -march=armv8a+memtag %s -o - \
+// RUN:     | FileCheck --check-prefixes=CHECK-MEMTAG-STACK,CHECK-MEMTAG-HEAP %s
+// RUN: %clang_cc1 -E  %s -o - | FileCheck --check-prefix=CHECK-NO-MEMTAG %s
+
+#if __has_feature(memtag_stack)
+int MemTagSanitizerStack();
+#else
+int MemTagSanitizerNoStack();
+#endif
+
+#if __has_feature(memtag_heap)
+int MemTagSanitizerHeap();
+#else
+int MemTagSanitizerNoHeap();
+#endif
+
+// CHECK-MEMTAG-STACK: MemTagSanitizerStack
+// CHECK-MEMTAG-HEAP: MemTagSanitizerHeap
+
+// CHECK-NO-MEMTAG: MemTagSanitizerNoStack
+// CHECK-NO-MEMTAG: MemTagSanitizerNoHeap

clang/test/Lexer/has_feature_memtag_sanitizer.cpp

-// RUN: %clang_cc1 -E -fsanitize=memtag %s -o - | FileCheck --check-prefix=CHECK-MEMTAG %s
-// RUN: %clang_cc1 -E  %s -o - | FileCheck --check-prefix=CHECK-NO-MEMTAG %s
-
-#if __has_feature(memtag_sanitizer)
-int MemTagSanitizerEnabled();
-#else
-int MemTagSanitizerDisabled();
-#endif
-
-// CHECK-MEMTAG: MemTagSanitizerEnabled
-// CHECK-NO-MEMTAG: MemTagSanitizerDisabled