[AArch64] Add support for MTE intrinsics

[javed.absar]

Summary

This patch provides intrinsics support for Memory Tagging Extension (MTE),
which was introduced with the Armv8.5-a architecture.
This is clang patch. Corresponding llvm-patch is at: https://reviews.llvm.org/D60486.

These intrinsics are available when __ARM_FEATURE_MEMORY_TAGGING is defined.
Each intrinsic is described in detail in the latest ACLE Q1 2019 documentation:

https://developer.arm.com/docs/101028/latest

However, below we also list the intrinsics:

1. T* __arm_mte_create_random_tag(T* src, uint64_t mask); This intrinsic returns a pointer containing a randomly created logical address tag.
2. T* __arm_mte_increment_tag(T* src, unsigned offset); This intrinsic returns a pointer which is a copy of the input pointer src but with the logical address tag part offset by a specified offset value.
3. uint64_t __arm_mte_exclude_tag(T* src, uint64_t excluded); This intrinsic adds a logical tag to the set of excluded logical tags.
4. void __arm_mte_set_tag(T* tag_address); This intrinsic stores an allocation tag, computed from the logical tag, to the tag memory thereby setting the allocation tag for the 16-byte granule of memory.
5. T* __arm_mte_get_tag(T* address); This intrinsic loads the allocation tag from tag memory and returns the corresponding logical tag as part of the returned pointer value.
6. ptrdiff_t __arm_mte_ptrdiff(T* a, T* b); The intrinsic calculates the difference between the address parts of the two pointers, ignoring the tags.

[DavidSpickett]

Do we need a separate file for that one "arm_mte_ptrdiff" test? Wouldn't it be easier to wrap the whole function in "__cplusplus" and put it in the same file.

[javed.absar]

Tests merged as suggested.

[t.p.northover]

include/clang/Basic/DiagnosticSemaKinds.td

9626

I think these diagnostics could do with a bit more context for consistency. They seem to take "MTE builtin" for granted, whereas most Clang messages mention what they're talking about.

I'm not saying "MTE builtin" is the best we can come up with, BTW, just that something more would be nice.

lib/Headers/arm_acle.h

610–615

Why are the builtin names so different from the ones exposed. GCC compatibility? LLVM?

lib/Sema/SemaChecking.cpp

6117

I think this will generate an unused variable warning in a non-asserts build.

[javed.absar]

include/clang/Basic/DiagnosticSemaKinds.td

9626

I thought of doing that too , so can prefix these warnings with 'mte builtin' if that's what you meant. But the intrinsic called kind of names same thing (__arm_mte_..).

lib/Headers/arm_acle.h

610–615

The builtin name (e.g. _mte_irg) is reflecting the instruction that implements the otherwise meaningful ACLE names (mte_create_tag). Its just that the instruction names are sometimes cryptic (e.g. stg, ldg). I could change the names to __builtin_arm_create_tag etc and push the meaningful name -> insn level name to intrinsic level or further down but that would mean lots of name changes to current patch and tests.

[t.p.northover]

include/clang/Basic/DiagnosticSemaKinds.td

9626

"Builtin" or even "function" (fixed up to be grammatical) would suffice if you don't like the repetition.

lib/Headers/arm_acle.h

610–615

OK, sounds reasonable to leave it as is then.

lib/Sema/SemaChecking.cpp

6116

Stray space here between '(' and 'unsigned', BTW.

[javed.absar]

Hi Tim:
Have made the changes as suggested. Please have a look.

[t.p.northover]

include/clang/Sema/Sema.h

10762

Slightly misaligned.

lib/CodeGen/CGBuiltin.cpp

7129–7131

I think this should be fixed now. It looks like technical debt from the fact that the instructions only fairly recently gained that feature after the intrinsics were implemented internally. There's no good way to justify the current semantics to someone unaware of that history.

[javed.absar]

lib/CodeGen/CGBuiltin.cpp

7129–7131

Not quite that really. So the instruction did gain the feature recently like you mentioned. But the ACLE/intrinsics were designed and agreed upon after it and it was decided in ACLE discussions that the exta feature added complexity that need not be exposed at ACLE level yet. No big use case to justify complicating the ACLE MTE spec yet. Directly assembly can use that instruction though.

[t.p.northover]

lib/CodeGen/CGBuiltin.cpp

7129–7131

I think the ACLE decision was a mistake, but since it happened I withdraw this request. I expect (and hope) far more people will use these through ACLE than as compiler-specific builtins.

[vitalybuka]

http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap-msan/builds/12035

-- Testing: 14692 tests, 96 threads --
Testing: 
FAIL: Clang :: AST/float16.cpp (132 of 14692)
******************** TEST 'Clang :: AST/float16.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';   /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/clang -cc1 -internal-isystem /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/lib/clang/9.0.0/include -nostdsysteminc -std=c++11 -ast-dump -triple aarch64-linux-gnu /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/test/AST/float16.cpp | /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/test/AST/float16.cpp --strict-whitespace
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/clang -cc1 -internal-isystem /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/lib/clang/9.0.0/include -nostdsysteminc -std=c++11 -ast-dump -triple aarch64-linux-gnu -fnative-half-type /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/test/AST/float16.cpp | /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/test/AST/float16.cpp --check-prefix=CHECK-NATIVE --strict-whitespace
--
Exit Code: 2

Command Output (stderr):
--
==12261==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0xd4212a8 in clang::targets::AArch64TargetInfo::getTargetDefines(clang::LangOptions const&, clang::MacroBuilder&) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Basic/Targets/AArch64.cpp:197:7
    #1 0xd422b5b in clang::targets::AArch64leTargetInfo::getTargetDefines(clang::LangOptions const&, clang::MacroBuilder&) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Basic/Targets/AArch64.cpp:463:22
    #2 0xd3ee87e in clang::targets::OSTargetInfo<clang::targets::AArch64leTargetInfo>::getTargetDefines(clang::LangOptions const&, clang::MacroBuilder&) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Basic/Targets/OSTargets.h:33:14
    #3 0x784d4c7 in InitializePredefinedMacros(clang::TargetInfo const&, clang::LangOptions const&, clang::FrontendOptions const&, clang::MacroBuilder&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Frontend/InitPreprocessor.cpp:1087:6
    #4 0x7830b1f in clang::InitializePreprocessor(clang::Preprocessor&, clang::PreprocessorOptions const&, clang::PCHContainerReader const&, clang::FrontendOptions const&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Frontend/InitPreprocessor.cpp:1117:5
    #5 0x76a550e in clang::CompilerInstance::createPreprocessor(clang::TranslationUnitKind) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Frontend/CompilerInstance.cpp:392:3
    #6 0x779314b in clang::FrontendAction::BeginSourceFile(clang::CompilerInstance&, clang::FrontendInputFile const&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Frontend/FrontendAction.cpp:742:8
    #7 0x76b49f8 in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Frontend/CompilerInstance.cpp:943:13
    #8 0x79d07f6 in clang::ExecuteCompilerInvocation(clang::CompilerInstance*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:271:25
    #9 0xb19a3a in cc1_main(llvm::ArrayRef<char const*>, char const*, void*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/tools/driver/cc1_main.cpp:225:15
    #10 0xb12983 in ExecuteCC1Tool /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/tools/driver/driver.cpp:309:12
    #11 0xb12983 in main /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/tools/driver/driver.cpp:381
    #12 0x7f69ddc1c2e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)
    #13 0xa91139 in _start (/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/clang-9+0xa91139)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/lib/Basic/Targets/AArch64.cpp:197:7 in clang::targets::AArch64TargetInfo::getTargetDefines(clang::LangOptions const&, clang::MacroBuilder&) const
Exiting
FileCheck error: '-' is empty.
FileCheck command line:  /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm/tools/clang/test/AST/float16.cpp --strict-whitespace

FYI @hctim

[vitalybuka]

fixed with r359366

include/clang/Basic/BuiltinsAArch64.def

 BUILTIN(__builtin_arm_crc32cb, "UiUiUc", "nc")
 BUILTIN(__builtin_arm_crc32h, "UiUiUs", "nc")
 BUILTIN(__builtin_arm_crc32ch, "UiUiUs", "nc")
 BUILTIN(__builtin_arm_crc32w, "UiUiUi", "nc")
 BUILTIN(__builtin_arm_crc32cw, "UiUiUi", "nc")
 BUILTIN(__builtin_arm_crc32d, "UiUiWUi", "nc")
 BUILTIN(__builtin_arm_crc32cd, "UiUiWUi", "nc")
 
+// Memory Tagging Extensions (MTE)
+BUILTIN(__builtin_arm_irg, "v*v*Ui", "t")
+BUILTIN(__builtin_arm_addg, "v*v*Ui", "t")
+BUILTIN(__builtin_arm_gmi, "Uiv*Ui", "t")
+BUILTIN(__builtin_arm_ldg, "v*v*", "t")
+BUILTIN(__builtin_arm_stg, "vv*", "t")
+BUILTIN(__builtin_arm_subp, "Uiv*v*", "t")
+
 // Memory barrier
 BUILTIN(__builtin_arm_dmb, "vUi", "nc")
 BUILTIN(__builtin_arm_dsb, "vUi", "nc")
 BUILTIN(__builtin_arm_isb, "vUi", "nc")
 
 // Prefetch
 BUILTIN(__builtin_arm_prefetch, "vvC*UiUiUiUi", "nc")
 

include/clang/Basic/DiagnosticSemaKinds.td

def err_spaceship_argument_narrowing : Error<
   "evaluates to %1, which cannot be narrowed to type %2}0">;
 def err_std_compare_type_not_supported : Error<
   "standard library implementation of %0 is not supported; "
    "%select{member '%2' does not have expected form|"
    "member '%2' is missing|"
    "the type is not trivially copyable|"
    "the type does not have the expected form}1">;
 
+// Memory Tagging Extensions (MTE) diagnostics
+def err_memtag_arg_null_or_pointer : Error<
+  "%0 argument of MTE builtin function must be a null or a pointer (%1 invalid)">;
+def err_memtag_any2arg_pointer : Error<
+  "at least one argument of MTE builtin function must be a pointer (%0, %1 invalid)">;
+def err_memtag_arg_must_be_pointer : Error<
+  "%0 argument of MTE builtin function must be a pointer (%1 invalid)">;
+def err_memtag_arg_must_be_integer : Error<
+  "%0 argument of MTE builtin function must be an integer type (%1 invalid)">;
+def err_memtag_arg_must_be_unsigned : Error<
+  "%0 argument  of MTE builtin function must be an unsigned integer type (%1 invalid)">;
+
 def warn_dereference_of_noderef_type : Warning<
   "dereferencing %0; was declared with a 'noderef' type">, InGroup<NoDeref>;
 def warn_dereference_of_noderef_type_no_decl : Warning<
   "dereferencing expression marked as 'noderef'">, InGroup<NoDeref>;
 def warn_noderef_on_non_pointer_or_array : Warning<
   "'noderef' can only be used on an array or pointer type">, InGroup<IgnoredAttributes>;
 def warn_noderef_to_dereferenceable_pointer : Warning<
   "casting to dereferenceable pointer removes 'noderef' attribute">, InGroup<NoDeref>;
 
 def err_builtin_launder_invalid_arg : Error<
   "%select{non-pointer|function pointer|void pointer}0 argument to "
   "'__builtin_launder' is not allowed">;
 
 } // end of sema component.

include/clang/Sema/Sema.h

bool SemaBuiltinConstantArg(CallExpr *TheCall, int ArgNum,
                               llvm::APSInt &Result);
   bool SemaBuiltinConstantArgRange(CallExpr *TheCall, int ArgNum, int Low,
                                    int High, bool RangeIsError = true);
   bool SemaBuiltinConstantArgMultiple(CallExpr *TheCall, int ArgNum,
                                       unsigned Multiple);
   bool SemaBuiltinARMSpecialReg(unsigned BuiltinID, CallExpr *TheCall,
                                 int ArgNum, unsigned ExpectedFieldNum,
                                 bool AllowName);
+  bool SemaBuiltinARMMemoryTaggingCall(unsigned BuiltinID, CallExpr *TheCall);
 public:
   enum FormatStringType {
     FST_Scanf,
     FST_Printf,
     FST_NSString,
     FST_Strftime,
     FST_Strfmon,
     FST_Kprintf,

lib/Basic/Targets/AArch64.h

class LLVM_LIBRARY_VISIBILITY AArch64TargetInfo : public TargetInfo {
 
   unsigned FPU;
   unsigned CRC;
   unsigned Crypto;
   unsigned Unaligned;
   unsigned HasFullFP16;
   unsigned HasDotProd;
   unsigned HasFP16FML;
+  unsigned HasMTE;
   llvm::AArch64::ArchKind ArchKind;
 
   static const Builtin::Info BuiltinInfo[];
 
   std::string ABI;
 
 public:
   AArch64TargetInfo(const llvm::Triple &Triple, const TargetOptions &Opts);

lib/Basic/Targets/AArch64.cpp

void AArch64TargetInfo::getTargetDefines(const LangOptions &Opts,
   if ((FPU & NeonMode) && HasFullFP16)
     Builder.defineMacro("__ARM_FEATURE_FP16_VECTOR_ARITHMETIC", "1");
   if (HasFullFP16)
    Builder.defineMacro("__ARM_FEATURE_FP16_SCALAR_ARITHMETIC", "1");
 
   if (HasDotProd)
     Builder.defineMacro("__ARM_FEATURE_DOTPROD", "1");
 
+  if (HasMTE)
+    Builder.defineMacro("__ARM_FEATURE_MEMORY_TAGGING", "1");
+
   if ((FPU & NeonMode) && HasFP16FML)
     Builder.defineMacro("__ARM_FEATURE_FP16FML", "1");
 
   switch (ArchKind) {
   default:
     break;
   case llvm::AArch64::ArchKind::ARMV8_1A:
     getTargetDefinesARMV81A(Opts, Builder);
for (const auto &Feature : Features) {
     if (Feature == "+v8.2a")
       ArchKind = llvm::AArch64::ArchKind::ARMV8_2A;
     if (Feature == "+fullfp16")
       HasFullFP16 = 1;
     if (Feature == "+dotprod")
       HasDotProd = 1;
     if (Feature == "+fp16fml")
       HasFP16FML = 1;
+    if (Feature == "+mte")
+      HasMTE = 1;
   }
 
   setDataLayout();
 
   return true;
 }
 
 TargetInfo::CallingConvCheckResult

lib/CodeGen/CGBuiltin.cpp

if (CRCIntrinsicID != Intrinsic::not_intrinsic) {
     Function *F = CGM.getIntrinsic(CRCIntrinsicID);
 
     llvm::Type *DataTy = F->getFunctionType()->getParamType(1);
     Arg1 = Builder.CreateZExtOrBitCast(Arg1, DataTy);
 
     return Builder.CreateCall(F, {Arg0, Arg1});
   }
 
+  // Memory Tagging Extensions (MTE) Intrinsics
+  Intrinsic::ID MTEIntrinsicID = Intrinsic::not_intrinsic;
+  switch (BuiltinID) {
+  case AArch64::BI__builtin_arm_irg:
+    MTEIntrinsicID = Intrinsic::aarch64_irg; break;
+  case  AArch64::BI__builtin_arm_addg:
+    MTEIntrinsicID = Intrinsic::aarch64_addg; break;
+  case  AArch64::BI__builtin_arm_gmi:
+    MTEIntrinsicID = Intrinsic::aarch64_gmi; break;
+  case  AArch64::BI__builtin_arm_ldg:
+    MTEIntrinsicID = Intrinsic::aarch64_ldg; break;
+  case AArch64::BI__builtin_arm_stg:
+    MTEIntrinsicID = Intrinsic::aarch64_stg; break;
+  case AArch64::BI__builtin_arm_subp:
+    MTEIntrinsicID = Intrinsic::aarch64_subp; break;
+  }
+
+  if (MTEIntrinsicID != Intrinsic::not_intrinsic) {
+    llvm::Type *T = ConvertType(E->getType());
+
+    if (MTEIntrinsicID == Intrinsic::aarch64_irg) {
+      Value *Pointer = EmitScalarExpr(E->getArg(0));
+      Value *Mask = EmitScalarExpr(E->getArg(1));
+
+      Pointer = Builder.CreatePointerCast(Pointer, Int8PtrTy);
+      Mask = Builder.CreateZExt(Mask, Int64Ty);
+      Value *RV = Builder.CreateCall(
+                       CGM.getIntrinsic(MTEIntrinsicID), {Pointer, Mask});
+       return Builder.CreatePointerCast(RV, T);
+    }
+    if (MTEIntrinsicID == Intrinsic::aarch64_addg) {
+      Value *Pointer = EmitScalarExpr(E->getArg(0));
+      Value *TagOffset = EmitScalarExpr(E->getArg(1));
+
+      Pointer = Builder.CreatePointerCast(Pointer, Int8PtrTy);
+      TagOffset = Builder.CreateZExt(TagOffset, Int64Ty);
+      Value *RV = Builder.CreateCall(
+                       CGM.getIntrinsic(MTEIntrinsicID), {Pointer, TagOffset});
+      return Builder.CreatePointerCast(RV, T);
+    }
+    if (MTEIntrinsicID == Intrinsic::aarch64_gmi) {
+      Value *Pointer = EmitScalarExpr(E->getArg(0));
+      Value *ExcludedMask = EmitScalarExpr(E->getArg(1));
+
+      ExcludedMask = Builder.CreateZExt(ExcludedMask, Int64Ty);
+      Pointer = Builder.CreatePointerCast(Pointer, Int8PtrTy);
+      return Builder.CreateCall(
+                       CGM.getIntrinsic(MTEIntrinsicID), {Pointer, ExcludedMask});
+    }
+    // Although it is possible to supply a different return
+    // address (first arg) to this intrinsic, for now we set
+    // return address same as input address.
+    if (MTEIntrinsicID == Intrinsic::aarch64_ldg) {
+      Value *TagAddress = EmitScalarExpr(E->getArg(0));
+      TagAddress = Builder.CreatePointerCast(TagAddress, Int8PtrTy);
+      Value *RV = Builder.CreateCall(
+                    CGM.getIntrinsic(MTEIntrinsicID), {TagAddress, TagAddress});
+      return Builder.CreatePointerCast(RV, T);
+    }
+    // Although it is possible to supply a different tag (to set)
+    // to this intrinsic (as first arg), for now we supply
+    // the tag that is in input address arg (common use case).
+    if (MTEIntrinsicID == Intrinsic::aarch64_stg) {
+        Value *TagAddress = EmitScalarExpr(E->getArg(0));
+        TagAddress = Builder.CreatePointerCast(TagAddress, Int8PtrTy);
+        return Builder.CreateCall(
+                 CGM.getIntrinsic(MTEIntrinsicID), {TagAddress, TagAddress});
+    }
+    if (MTEIntrinsicID == Intrinsic::aarch64_subp) {
+      Value *PointerA = EmitScalarExpr(E->getArg(0));
+      Value *PointerB = EmitScalarExpr(E->getArg(1));
+      PointerA = Builder.CreatePointerCast(PointerA, Int8PtrTy);
+      PointerB = Builder.CreatePointerCast(PointerB, Int8PtrTy);
+      return Builder.CreateCall(
+                       CGM.getIntrinsic(MTEIntrinsicID), {PointerA, PointerB});
+    }
+  }
+
   if (BuiltinID == AArch64::BI__builtin_arm_rsr ||
       BuiltinID == AArch64::BI__builtin_arm_rsr64 ||
       BuiltinID == AArch64::BI__builtin_arm_rsrp ||
       BuiltinID == AArch64::BI__builtin_arm_wsr ||
       BuiltinID == AArch64::BI__builtin_arm_wsr64 ||
       BuiltinID == AArch64::BI__builtin_arm_wsrp) {
 
     bool IsRead = BuiltinID == AArch64::BI__builtin_arm_rsr ||

lib/Headers/arm_acle.h

 /* 10.1 Special register intrinsics */
 #define __arm_rsr(sysreg) __builtin_arm_rsr(sysreg)
 #define __arm_rsr64(sysreg) __builtin_arm_rsr64(sysreg)
 #define __arm_rsrp(sysreg) __builtin_arm_rsrp(sysreg)
 #define __arm_wsr(sysreg, v) __builtin_arm_wsr(sysreg, v)
 #define __arm_wsr64(sysreg, v) __builtin_arm_wsr64(sysreg, v)
 #define __arm_wsrp(sysreg, v) __builtin_arm_wsrp(sysreg, v)
 
+// Memory Tagging Extensions (MTE) Intrinsics
+#if __ARM_FEATURE_MEMORY_TAGGING
+#define __arm_mte_create_random_tag(__ptr, __mask)  __builtin_arm_irg(__ptr, __mask)
+#define __arm_mte_increment_tag(__ptr, __tag_offset)  __builtin_arm_addg(__ptr, __tag_offset)
+#define __arm_mte_exclude_tag(__ptr, __excluded)  __builtin_arm_gmi(__ptr, __excluded)
+#define __arm_mte_get_tag(__ptr) __builtin_arm_ldg(__ptr)
+#define __arm_mte_set_tag(__ptr) __builtin_arm_stg(__ptr)
+#define __arm_mte_ptrdiff(__ptra, __ptrb) __builtin_arm_subp(__ptra, __ptrb)
+#endif
+
 #if defined(__cplusplus)
 }
 #endif
 
 #endif /* __ARM_ACLE_H */

lib/Sema/SemaChecking.cpp

return SemaBuiltinConstantArgRange(TheCall, 1, 0, 1) ||
       SemaBuiltinConstantArgRange(TheCall, 3, 0, 1) ||
       SemaBuiltinConstantArgRange(TheCall, 4, 0, 1);
   }
 
   if (BuiltinID == AArch64::BI__builtin_arm_rsr64 ||
       BuiltinID == AArch64::BI__builtin_arm_wsr64)
     return SemaBuiltinARMSpecialReg(BuiltinID, TheCall, 0, 5, true);
 
+  // Memory Tagging Extensions (MTE) Intrinsics
+  if (BuiltinID == AArch64::BI__builtin_arm_irg ||
+      BuiltinID == AArch64::BI__builtin_arm_addg ||
+      BuiltinID == AArch64::BI__builtin_arm_gmi ||
+      BuiltinID == AArch64::BI__builtin_arm_ldg ||
+      BuiltinID == AArch64::BI__builtin_arm_stg ||
+      BuiltinID == AArch64::BI__builtin_arm_subp) {
+    return SemaBuiltinARMMemoryTaggingCall(BuiltinID, TheCall);
+  }
+
   if (BuiltinID == AArch64::BI__builtin_arm_rsr ||
       BuiltinID == AArch64::BI__builtin_arm_rsrp ||
       BuiltinID == AArch64::BI__builtin_arm_wsr ||
       BuiltinID == AArch64::BI__builtin_arm_wsrp)
     return SemaBuiltinARMSpecialReg(BuiltinID, TheCall, 0, 5, true);
 
   // Only check the valid encoding range. Any constant in this range would be
   // converted to a register of the form S1_2_C3_C4_5. Let the hardware throw
bool Sema::SemaBuiltinConstantArgMultiple(CallExpr *TheCall, int ArgNum,
 
   if (Result.getSExtValue() % Num != 0)
     return Diag(TheCall->getBeginLoc(), diag::err_argument_not_multiple)
            << Num << Arg->getSourceRange();
 
   return false;
 }
 
+/// SemaBuiltinARMMemoryTaggingCall - Handle calls of memory tagging extensions
+bool Sema::SemaBuiltinARMMemoryTaggingCall(unsigned BuiltinID, CallExpr *TheCall) {
+  if (BuiltinID == AArch64::BI__builtin_arm_irg) {
+    if (checkArgCount(*this, TheCall, 2))
+      return true;
+    Expr *Arg0 = TheCall->getArg(0);
+    Expr *Arg1 = TheCall->getArg(1);
+
+    ExprResult FirstArg = DefaultFunctionArrayLvalueConversion(Arg0);
+    if (FirstArg.isInvalid())
+      return true;
+    QualType FirstArgType = FirstArg.get()->getType();
+    if (!FirstArgType->isAnyPointerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_must_be_pointer)
+               << "first" << FirstArgType << Arg0->getSourceRange();
+    TheCall->setArg(0, FirstArg.get());
+
+    ExprResult SecArg = DefaultLvalueConversion(Arg1);
+    if (SecArg.isInvalid())
+      return true;
+    QualType SecArgType = SecArg.get()->getType();
+    if (!SecArgType->isIntegerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_must_be_integer)
+               << "second" << SecArgType << Arg1->getSourceRange();
+
+    // Derive the return type from the pointer argument.
+    TheCall->setType(FirstArgType);
+    return false;
+  }
+
+  if (BuiltinID == AArch64::BI__builtin_arm_addg) {
+    if (checkArgCount(*this, TheCall, 2))
+      return true;
+
+    Expr *Arg0 = TheCall->getArg(0);
+    ExprResult FirstArg = DefaultFunctionArrayLvalueConversion(Arg0);
+    if (FirstArg.isInvalid())
+      return true;
+    QualType FirstArgType = FirstArg.get()->getType();
+    if (!FirstArgType->isAnyPointerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_must_be_pointer)
+               << "first" << FirstArgType << Arg0->getSourceRange();
+    TheCall->setArg(0, FirstArg.get());
+
+    // Derive the return type from the pointer argument.
+    TheCall->setType(FirstArgType);
+
+    // Second arg must be an constant in range [0,15]
+    return SemaBuiltinConstantArgRange(TheCall, 1, 0, 15);
+  }
+
+  if (BuiltinID == AArch64::BI__builtin_arm_gmi) {
+    if (checkArgCount(*this, TheCall, 2))
+      return true;
+    Expr *Arg0 = TheCall->getArg(0);
+    Expr *Arg1 = TheCall->getArg(1);
+
+    ExprResult FirstArg = DefaultFunctionArrayLvalueConversion(Arg0);
+    if (FirstArg.isInvalid())
+      return true;
+    QualType FirstArgType = FirstArg.get()->getType();
+    if (!FirstArgType->isAnyPointerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_must_be_pointer)
+               << "first" << FirstArgType << Arg0->getSourceRange();
+
+    QualType SecArgType = Arg1->getType();
+    if (!SecArgType->isIntegerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_must_be_integer)
+               << "second" << SecArgType << Arg1->getSourceRange();
+    TheCall->setType(Context.IntTy);
+    return false;
+  }
+
+  if (BuiltinID == AArch64::BI__builtin_arm_ldg ||
+      BuiltinID == AArch64::BI__builtin_arm_stg) {
+    if (checkArgCount(*this, TheCall, 1))
+      return true;
+    Expr *Arg0 = TheCall->getArg(0);
+    ExprResult FirstArg = DefaultFunctionArrayLvalueConversion(Arg0);
+    if (FirstArg.isInvalid())
+      return true;
+
+    QualType FirstArgType = FirstArg.get()->getType();
+    if (!FirstArgType->isAnyPointerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_must_be_pointer)
+               << "first" << FirstArgType << Arg0->getSourceRange();
+    TheCall->setArg(0, FirstArg.get());
+
+    // Derive the return type from the pointer argument.
+    if (BuiltinID == AArch64::BI__builtin_arm_ldg)
+      TheCall->setType(FirstArgType);
+    return false;
+  }
+
+  if (BuiltinID == AArch64::BI__builtin_arm_subp) {
+    Expr *ArgA = TheCall->getArg(0);
+    Expr *ArgB = TheCall->getArg(1);
+
+    ExprResult ArgExprA = DefaultFunctionArrayLvalueConversion(ArgA);
+    ExprResult ArgExprB = DefaultFunctionArrayLvalueConversion(ArgB);
+
+    if (ArgExprA.isInvalid() || ArgExprB.isInvalid())
+      return true;
+
+    QualType ArgTypeA = ArgExprA.get()->getType();
+    QualType ArgTypeB = ArgExprB.get()->getType();
+
+    auto isNull = [&] (Expr *E) -> bool {
+      return E->isNullPointerConstant(
+                        Context, Expr::NPC_ValueDependentIsNotNull); };
+
+    // argument should be either a pointer or null
+    if (!ArgTypeA->isAnyPointerType() && !isNull(ArgA))
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_null_or_pointer)
+        << "first" << ArgTypeA << ArgA->getSourceRange();
+
+    if (!ArgTypeB->isAnyPointerType() && !isNull(ArgB))
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_arg_null_or_pointer)
+        << "second" << ArgTypeB << ArgB->getSourceRange();
+
+    // Ensure Pointee types are compatible
+    if (ArgTypeA->isAnyPointerType() && !isNull(ArgA) &&
+        ArgTypeB->isAnyPointerType() && !isNull(ArgB)) {
+      QualType pointeeA = ArgTypeA->getPointeeType();
+      QualType pointeeB = ArgTypeB->getPointeeType();
+      if (!Context.typesAreCompatible(
+             Context.getCanonicalType(pointeeA).getUnqualifiedType(),
+             Context.getCanonicalType(pointeeB).getUnqualifiedType())) {
+        return Diag(TheCall->getBeginLoc(), diag::err_typecheck_sub_ptr_compatible)
+          << ArgTypeA <<  ArgTypeB << ArgA->getSourceRange()
+          << ArgB->getSourceRange();
+      }
+    }
+
+    // at least one argument should be pointer type
+    if (!ArgTypeA->isAnyPointerType() && !ArgTypeB->isAnyPointerType())
+      return Diag(TheCall->getBeginLoc(), diag::err_memtag_any2arg_pointer)
+        <<  ArgTypeA << ArgTypeB << ArgA->getSourceRange();
+
+    if (isNull(ArgA)) // adopt type of the other pointer
+      ArgExprA = ImpCastExprToType(ArgExprA.get(), ArgTypeB, CK_NullToPointer);
+
+    if (isNull(ArgB))
+      ArgExprB = ImpCastExprToType(ArgExprB.get(), ArgTypeA, CK_NullToPointer);
+
+    TheCall->setArg(0, ArgExprA.get());
+    TheCall->setArg(1, ArgExprB.get());
+    TheCall->setType(Context.LongLongTy);
+    return false;
+  }
+  assert(false && "Unhandled ARM MTE intrinsic");
+  return true;
+}
+
 /// SemaBuiltinARMSpecialReg - Handle a check if argument ArgNum of CallExpr
 /// TheCall is an ARM/AArch64 special register string literal.
 bool Sema::SemaBuiltinARMSpecialReg(unsigned BuiltinID, CallExpr *TheCall,
                                     int ArgNum, unsigned ExpectedFieldNum,
                                     bool AllowName) {
   bool IsARMBuiltin = BuiltinID == ARM::BI__builtin_arm_rsr64 ||
                       BuiltinID == ARM::BI__builtin_arm_wsr64 ||
                       BuiltinID == ARM::BI__builtin_arm_rsr ||

test/CodeGen/arm64-mte.c

+// Test memory tagging extension intrinsics
+// RUN: %clang_cc1 -triple aarch64-none-linux-eabi -target-feature +mte -O3 -S -emit-llvm -o - %s  | FileCheck %s
+#include <stddef.h>
+#include <arm_acle.h>
+
+// CHECK-LABEL: define i32* @create_tag1
+int *create_tag1(int *a, unsigned b) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = zext i32 %b to i64
+// CHECK: [[T2:%[0-9]+]] = tail call i8* @llvm.aarch64.irg(i8* [[T0]], i64 [[T1]])
+// CHECK: bitcast i8* [[T2]] to i32*
+        return __arm_mte_create_random_tag(a,b);
+}
+
+// CHECK-LABEL: define i16* @create_tag2
+short *create_tag2(short *a, unsigned b) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i16* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = zext i32 %b to i64
+// CHECK: [[T2:%[0-9]+]] = tail call i8* @llvm.aarch64.irg(i8* [[T0]], i64 [[T1]])
+// CHECK: bitcast i8* [[T2]] to i16*
+        return __arm_mte_create_random_tag(a,b);
+}
+
+// CHECK-LABEL: define i8* @create_tag3
+char *create_tag3(char *a, unsigned b) {
+// CHECK: [[T1:%[0-9]+]] = zext i32 %b to i64
+// CHECK: [[T2:%[0-9]+]] = tail call i8* @llvm.aarch64.irg(i8* %a, i64 [[T1]])
+// CHECK: ret i8* [[T2:%[0-9]+]]
+        return __arm_mte_create_random_tag(a,b);
+}
+
+// CHECK-LABEL: define i8* @increment_tag1
+char *increment_tag1(char *a) {
+// CHECK: call i8* @llvm.aarch64.addg(i8* %a, i64 3)
+        return __arm_mte_increment_tag(a,3);
+}
+
+// CHECK-LABEL: define i16* @increment_tag2
+short *increment_tag2(short *a) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i16* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = tail call i8* @llvm.aarch64.addg(i8* [[T0]], i64 3)
+// CHECK: [[T2:%[0-9]+]]  = bitcast i8* [[T1]] to i16*
+        return __arm_mte_increment_tag(a,3);
+}
+
+// CHECK-LABEL: define i32 @exclude_tag
+unsigned exclude_tag(int *a, unsigned m) {
+// CHECK: [[T0:%[0-9]+]] = zext i32 %m to i64
+// CHECK: [[T1:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: [[T2:%[0-9]+]] = tail call i64 @llvm.aarch64.gmi(i8* [[T1]], i64 [[T0]])
+// CHECK: trunc i64 [[T2]] to i32
+  return __arm_mte_exclude_tag(a, m);
+}
+
+// CHECK-LABEL: define i32* @get_tag1
+int *get_tag1(int *a) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = tail call i8* @llvm.aarch64.ldg(i8* [[T0]], i8* [[T0]])
+// CHECK: [[T2:%[0-9]+]]  = bitcast i8* [[T1]] to i32*
+   return __arm_mte_get_tag(a);
+}
+
+// CHECK-LABEL: define i16* @get_tag2
+short *get_tag2(short *a) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i16* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = tail call i8* @llvm.aarch64.ldg(i8* [[T0]], i8* [[T0]])
+// CHECK: [[T2:%[0-9]+]]  = bitcast i8* [[T1]] to i16*
+   return __arm_mte_get_tag(a);
+}
+
+// CHECK-LABEL: define void @set_tag1
+void set_tag1(int *a) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: tail call void @llvm.aarch64.stg(i8* [[T0]], i8* [[T0]])
+   __arm_mte_set_tag(a);
+}
+
+// CHECK-LABEL: define i64 @subtract_pointers
+ptrdiff_t subtract_pointers(int *a, int *b) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = bitcast i32* %b to i8*
+// CHECK: [[T2:%[0-9]+]] = tail call i64 @llvm.aarch64.subp(i8* [[T0]], i8* [[T1]])
+// CHECK: ret i64 [[T2]]
+   return __arm_mte_ptrdiff(a, b);
+}
+
+// CHECK-LABEL: define i64 @subtract_pointers_null_1
+ptrdiff_t subtract_pointers_null_1(int *a) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = tail call i64 @llvm.aarch64.subp(i8* [[T0]], i8* null)
+// CHECK: ret i64 [[T1]]
+   return __arm_mte_ptrdiff(a, NULL);
+}
+
+// CHECK-LABEL: define i64 @subtract_pointers_null_2
+ptrdiff_t subtract_pointers_null_2(int *a) {
+// CHECK: [[T0:%[0-9]+]] = bitcast i32* %a to i8*
+// CHECK: [[T1:%[0-9]+]] = tail call i64 @llvm.aarch64.subp(i8* null, i8* [[T0]])
+// CHECK: ret i64 [[T1]]
+   return __arm_mte_ptrdiff(NULL, a);
+}
+
+// Check arithmetic promotion on return type
+// CHECK-LABEL: define i32 @subtract_pointers4
+int subtract_pointers4(void* a, void *b) {
+// CHECK: [[T0:%[0-9]+]] = tail call i64 @llvm.aarch64.subp(i8* %a, i8* %b)
+// CHECK-NEXT: %cmp = icmp slt i64 [[T0]], 1
+// CHECK-NEXT:  = zext i1 %cmp to i32
+  return __arm_mte_ptrdiff(a,b) <= 0;
+}

test/Preprocessor/aarch64-target-features.c

 // ================== Check whether -march accepts mixed-case features.
 // RUN: %clang -target aarch64 -march=ARMV8.1A+CRYPTO -### -c %s 2>&1 | FileCheck -check-prefix=CHECK-V81A-FEATURE-1 %s
 // RUN: %clang -target aarch64 -march=Armv8.1a+NOcrypto -### -c %s 2>&1 | FileCheck -check-prefix=CHECK-V81A-FEATURE-2 %s
 // RUN: %clang -target aarch64 -march=armv8.1a+noSIMD -### -c %s 2>&1 | FileCheck -check-prefix=CHECK-V81A-FEATURE-3 %s
 // CHECK-V81A-FEATURE-1: "-cc1"{{.*}} "-triple" "aarch64{{.*}}" "-target-feature" "+neon" "-target-feature" "+v8.1a" "-target-feature" "+crypto"
 // CHECK-V81A-FEATURE-2: "-cc1"{{.*}} "-triple" "aarch64{{.*}}" "-target-feature" "+neon" "-target-feature" "+v8.1a" "-target-feature" "-crypto"
 // CHECK-V81A-FEATURE-3: "-cc1"{{.*}} "-triple" "aarch64{{.*}}" "-target-feature" "+v8.1a" "-target-feature" "-neon"
 
+// ================== Check Memory Tagging Extensions (MTE).
+// RUN: %clang -target arm64-none-linux-gnu -march=armv8.5-a+memtag -x c -E -dM %s -o - 2>&1 | FileCheck -check-prefix=CHECK-MEMTAG %s
+// CHECK-MEMTAG: __ARM_FEATURE_MEMORY_TAGGING 1

test/Sema/builtins-arm64-mte.c

+// RUN: %clang_cc1 -triple arm64-arm-eabi %s -target-feature +mte -fsyntax-only -verify
+// RUN: %clang_cc1 -triple arm64-arm-eabi %s -target-feature +mte -x c++ -fsyntax-only -verify
+#include <stddef.h>
+#include <arm_acle.h>
+
+int  *create_tag1(int a, unsigned b) {
+  // expected-error@+1 {{first argument of MTE builtin function must be a pointer ('int' invalid)}}
+  return __arm_mte_create_random_tag(a,b);
+}
+
+int  *create_tag2(int *a, unsigned *b) {
+  // expected-error@+1 {{second argument of MTE builtin function must be an integer type ('unsigned int *' invalid)}}
+  return __arm_mte_create_random_tag(a,b);
+}
+
+int  *create_tag3(const int *a, unsigned b) {
+#ifdef __cplusplus
+  // expected-error@+1 {{cannot initialize return object of type 'int *' with an rvalue of type 'const int *'}}
+  return __arm_mte_create_random_tag(a,b);
+#else
+  // expected-warning@+1 {{returning 'const int *' from a function with result type 'int *' discards qualifiers}}
+  return __arm_mte_create_random_tag(a,b);
+#endif
+}
+
+int  *create_tag4(volatile int *a, unsigned b) {
+#ifdef __cplusplus
+  // expected-error@+1 {{cannot initialize return object of type 'int *' with an rvalue of type 'volatile int *'}}
+  return __arm_mte_create_random_tag(a,b);
+#else
+  // expected-warning@+1 {{returning 'volatile int *' from a function with result type 'int *' discards qualifiers}}
+  return __arm_mte_create_random_tag(a,b);
+#endif
+}
+
+int  *increment_tag1(int *a, unsigned b) {
+  // expected-error@+1 {{argument to '__builtin_arm_addg' must be a constant integer}}
+  return __arm_mte_increment_tag(a,b);
+}
+
+int  *increment_tag2(int *a) {
+  // expected-error@+1 {{argument value 16 is outside the valid range [0, 15]}}
+  return __arm_mte_increment_tag(a,16);
+}
+
+int  *increment_tag3(int *a) {
+  // expected-error@+1 {{argument value -1 is outside the valid range [0, 15]}}
+  return __arm_mte_increment_tag(a,-1);
+}
+
+int  *increment_tag4(const int *a) {
+#ifdef __cplusplus
+  // expected-error@+1 {{cannot initialize return object of type 'int *' with an rvalue of type 'const int *'}}
+  return __arm_mte_increment_tag(a,5);
+#else
+  // expected-warning@+1 {{returning 'const int *' from a function with result type 'int *' discards qualifiers}}
+  return __arm_mte_increment_tag(a,5);
+#endif
+}
+
+int *increment_tag5(const volatile int *a) {
+#ifdef __cplusplus
+  // expected-error@+1 {{cannot initialize return object of type 'int *' with an rvalue of type 'const volatile int *'}}
+  return __arm_mte_increment_tag(a,5);
+#else
+  // expected-warning@+1 {{returning 'const volatile int *' from a function with result type 'int *' discards qualifiers}}
+  return __arm_mte_increment_tag(a,5);
+#endif
+}
+
+unsigned exclude_tag1(int *ptr, unsigned m) {
+   // expected-error@+1 {{first argument of MTE builtin function must be a pointer ('int' invalid)}}
+   return  __arm_mte_exclude_tag(*ptr, m);
+}
+
+unsigned exclude_tag2(int *ptr, int *m) {
+   // expected-error@+1 {{second argument of MTE builtin function must be an integer type ('int *' invalid)}}
+   return  __arm_mte_exclude_tag(ptr, m);
+}
+
+void get_tag1() {
+   // expected-error@+1 {{too few arguments to function call, expected 1, have 0}}
+   __arm_mte_get_tag();
+}
+
+int *get_tag2(int ptr) {
+   // expected-error@+1 {{first argument of MTE builtin function must be a pointer ('int' invalid)}}
+   return __arm_mte_get_tag(ptr);
+}
+
+int *get_tag3(const volatile int *ptr) {
+#ifdef __cplusplus
+  // expected-error@+1 {{cannot initialize return object of type 'int *' with an rvalue of type 'const volatile int *'}}
+  return __arm_mte_get_tag(ptr);
+#else
+  // expected-warning@+1 {{returning 'const volatile int *' from a function with result type 'int *' discards qualifiers}}
+  return __arm_mte_get_tag(ptr);
+#endif
+}
+
+void set_tag1() {
+   // expected-error@+1 {{too few arguments to function call, expected 1, have 0}}
+   __arm_mte_set_tag();
+}
+
+void set_tag2(int ptr) {
+   // expected-error@+1 {{first argument of MTE builtin function must be a pointer ('int' invalid)}}
+   __arm_mte_set_tag(ptr);
+}
+
+ptrdiff_t subtract_pointers1(int a, int *b) {
+  // expected-error@+1 {{first argument of MTE builtin function must be a null or a pointer ('int' invalid)}}
+  return __arm_mte_ptrdiff(a, b);
+}
+
+ptrdiff_t subtract_pointers2(int *a, int b) {
+  // expected-error@+1 {{second argument of MTE builtin function must be a null or a pointer ('int' invalid)}}
+  return __arm_mte_ptrdiff(a, b);
+}
+
+ptrdiff_t subtract_pointers3(char *a, int *b) {
+  // expected-error@+1 {{'char *' and 'int *' are not pointers to compatible types}}
+  return __arm_mte_ptrdiff(a, b);
+}
+
+ptrdiff_t subtract_pointers4(int *a, char *b) {
+  // expected-error@+1 {{'int *' and 'char *' are not pointers to compatible types}}
+  return __arm_mte_ptrdiff(a, b);
+}
+
+#ifdef __cplusplus
+ptrdiff_t subtract_pointers5() {
+  // expected-error@+1 {{at least one argument of MTE builtin function must be a pointer ('nullptr_t', 'nullptr_t' invalid)}}
+  return __arm_mte_ptrdiff(nullptr, nullptr);
+}
+#endif