[AArch64] Add support for Transactional Memory Extension (TME)

[chill]

Summary

  TME is a future architecture technology, documented in
  
  https://developer.arm.com/architectures/cpu-architecture/a-profile/exploration-tools
  https://developer.arm.com/docs/ddi0601/a
  
  More about the future architectures:
  
  https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/new-technologies-for-the-arm-a-profile-architecture
  
  This patch adds support for the TME instructions TSTART, TTEST, TCOMMIT, and
  TCANCEL and the target feature/arch extension "tme".
  
  It also implements TME builtin functions, defined in ACLE Q2 2019
  (https://developer.arm.com/docs/101028/latest)
  
Patch by Javed Absar and Momchil Velikov

[ostannard]

clang/lib/CodeGen/CGBuiltin.cpp

7223 ↗

(On Diff #208668)

This error checking should be done in Sema::CheckBuiltinFunctionCall, not CodeGen.

clang/test/CodeGen/aarch64-tme-errors.c

1 ↗

(On Diff #208668)

Tests for error messages should use clang's -verify option, and be in clang/test/Sema.

llvm/include/llvm/IR/IntrinsicsAArch64.td

713 ↗

(On Diff #208668)

Why is this IR intrinsic marked as Throws, but the C intrinsic is nothrows (n)?

715 ↗

(On Diff #208668)

Are you sure it's safe for this to be IntrNoMem and not IntrHasSideEffects? I think this means that all calls to this intrinsic can be folded together, even across transaction boundaries.

More generally, I wonder if it would be safer to model all of these intrinsics as reading and writing memory, because we don't want them to be re-ordered with respect to any other memory accesses (though maybe IntrHasSideEffects is enough for that). Do we have any documentation on how these intrinsics change (if at all) the C or LLVM rules regarding data races?

llvm/lib/Target/AArch64/AArch64InstrFormats.td

1064 ↗

(On Diff #208668)

This isn't how we use instruction format classes. There should be one class here for each group with the same operands and encoding. Some of these might be able to re-use existing format classes if they fit into existing parts of the encoding space.

llvm/test/CodeGen/AArch64/tme.ll

29 ↗

(On Diff #208668)

It would be better to test each intrinsic in a separate function, so that this won't be affected by irrelevant codegen changes (e.g. basic block layout).

79 ↗

(On Diff #208668)

Most of the attributes and metadata can be removed to simplify the test.

[chill]

Thanks for the review. I've addressed the comments.

[chill]

llvm/include/llvm/IR/IntrinsicsAArch64.td

713 ↗

(On Diff #208668)

I've now marked both as nothrow.

715 ↗

(On Diff #208668)

It's indeed probably unsafe. I've changed ttest now to have side effects, so it's on the chain with tstart/tcommit/tcancel.

I would like to impose the minimun set of restrictions regarding this set of intrinsics and associated instructions. There isn't yet a specification how this extension fits in the AArch64 or C/C++ memory model. Intuitively, regarding reordering of memory accesses, I would expect tstart to have Acquire semantics and tcommit - Release; but then that's speculation at this point, moreover I couldn't find a way to express that.

So, now

• tstart and tcommit both read/write memory and have side efects;
• ttest has side-effects (so it's not reordered with tstart and tcommit, but otherwise everythign is free to move around it.
• tcancel is the only one left with IntrNoMem and IntrHasSideEffects - it's IntrNoReturn so every operation following it is dead; if such an operation was to be reordered with tcancel it does not become less dead as tcancel discards its effects. Same for reordering in the opposite direction.

[ostannard]

LGTM, thanks.

[RKSimon]

@chill This is failing on buildbots with EXPENSIVE_CHECKS enabled: http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/18684/steps/test-check-all/logs/stdio

[chill]

I reverted the patch, have to rework tcancel.

[chill]

Changed tcancel implementation.

[ostannard]

LGTM

[t.p.northover]

clang/test/Sema/aarch64-tme-errors.c

1 ↗

(On Diff #210768)

I don't think the Sema checks need to be split over so many files. One for the whole of transactional seems enough. Similarly for CodeGen really.

llvm/test/CodeGen/AArch64/tme-tcancel.ll

6 ↗

(On Diff #210768)

Testing more values than 0 would be a good idea.

[RKSimon]

llvm/test/CodeGen/AArch64/tme-tcancel.ll

1 ↗

(On Diff #210768)

Would it make sense to add -verify-machineinstrs to all these Codegen/AArch64/tme-*.ll tests?

[chill]

If there are no objections, I'll go ahead with committing that Soon(tm) on the basis of previous acceptance.

[chill]

Rebase on top of master.

cfe/trunk/include/clang/Basic/BuiltinsAArch64.def

 LANGBUILTIN(__wfe,   "v", "",   ALL_MS_LANGUAGES)
 LANGBUILTIN(__wfi,   "v", "",   ALL_MS_LANGUAGES)
 LANGBUILTIN(__sev,   "v", "",   ALL_MS_LANGUAGES)
 LANGBUILTIN(__sevl,  "v", "",   ALL_MS_LANGUAGES)
 
 // Misc
 BUILTIN(__builtin_sponentry, "v*", "c")
 
+// Transactional Memory Extension
+BUILTIN(__builtin_arm_tstart, "WUi", "nj")
+BUILTIN(__builtin_arm_tcommit, "v", "n")
+BUILTIN(__builtin_arm_tcancel, "vWUIi", "nr")
+BUILTIN(__builtin_arm_ttest, "WUi", "nc")
+
 TARGET_HEADER_BUILTIN(_BitScanForward, "UcUNi*UNi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_BitScanReverse, "UcUNi*UNi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_BitScanForward64, "UcUNi*ULLi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_BitScanReverse64, "UcUNi*ULLi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 
 TARGET_HEADER_BUILTIN(_InterlockedAdd,           "LiLiD*Li",    "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedAnd64,         "LLiLLiD*LLi", "nh", "intrin.h", ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedDecrement64,   "LLiLLiD*",    "nh", "intrin.h", ALL_MS_LANGUAGES, "")

cfe/trunk/lib/Basic/Targets/AArch64.h

class LLVM_LIBRARY_VISIBILITY AArch64TargetInfo : public TargetInfo {
   unsigned FPU;
   bool HasCRC;
   bool HasCrypto;
   bool HasUnaligned;
   bool HasFullFP16;
   bool HasDotProd;
   bool HasFP16FML;
   bool HasMTE;
+  bool HasTME;
 
   llvm::AArch64::ArchKind ArchKind;
 
   static const Builtin::Info BuiltinInfo[];
 
   std::string ABI;
 
 public:

cfe/trunk/lib/Basic/Targets/AArch64.cpp

void AArch64TargetInfo::getTargetDefines(const LangOptions &Opts,
    Builder.defineMacro("__ARM_FEATURE_FP16_SCALAR_ARITHMETIC", "1");
 
   if (HasDotProd)
     Builder.defineMacro("__ARM_FEATURE_DOTPROD", "1");
 
   if (HasMTE)
     Builder.defineMacro("__ARM_FEATURE_MEMORY_TAGGING", "1");
 
+  if (HasTME)
+    Builder.defineMacro("__ARM_FEATURE_TME", "1");
+
   if ((FPU & NeonMode) && HasFP16FML)
     Builder.defineMacro("__ARM_FEATURE_FP16FML", "1");
 
   switch (ArchKind) {
   default:
     break;
   case llvm::AArch64::ArchKind::ARMV8_1A:
     getTargetDefinesARMV81A(Opts, Builder);
bool AArch64TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
   FPU = FPUMode;
   HasCRC = false;
   HasCrypto = false;
   HasUnaligned = true;
   HasFullFP16 = false;
   HasDotProd = false;
   HasFP16FML = false;
   HasMTE = false;
+  HasTME = false;
   ArchKind = llvm::AArch64::ArchKind::ARMV8A;
 
   for (const auto &Feature : Features) {
     if (Feature == "+neon")
       FPU |= NeonMode;
     if (Feature == "+sve")
       FPU |= SveMode;
     if (Feature == "+crc")
for (const auto &Feature : Features) {
     if (Feature == "+fullfp16")
       HasFullFP16 = true;
     if (Feature == "+dotprod")
       HasDotProd = true;
     if (Feature == "+fp16fml")
       HasFP16FML = true;
     if (Feature == "+mte")
       HasMTE = true;
+    if (Feature == "+tme")
+      HasTME = true;
   }
 
   setDataLayout();
 
   return true;
 }
 
 TargetInfo::CallingConvCheckResult

cfe/trunk/lib/Headers/arm_acle.h

 /* 10.1 Special register intrinsics */
 #define __arm_rsr(sysreg) __builtin_arm_rsr(sysreg)
 #define __arm_rsr64(sysreg) __builtin_arm_rsr64(sysreg)
 #define __arm_rsrp(sysreg) __builtin_arm_rsrp(sysreg)
 #define __arm_wsr(sysreg, v) __builtin_arm_wsr(sysreg, v)
 #define __arm_wsr64(sysreg, v) __builtin_arm_wsr64(sysreg, v)
 #define __arm_wsrp(sysreg, v) __builtin_arm_wsrp(sysreg, v)
 
-// Memory Tagging Extensions (MTE) Intrinsics
+/* Memory Tagging Extensions (MTE) Intrinsics */
 #if __ARM_FEATURE_MEMORY_TAGGING
 #define __arm_mte_create_random_tag(__ptr, __mask)  __builtin_arm_irg(__ptr, __mask)
 #define __arm_mte_increment_tag(__ptr, __tag_offset)  __builtin_arm_addg(__ptr, __tag_offset)
 #define __arm_mte_exclude_tag(__ptr, __excluded)  __builtin_arm_gmi(__ptr, __excluded)
 #define __arm_mte_get_tag(__ptr) __builtin_arm_ldg(__ptr)
 #define __arm_mte_set_tag(__ptr) __builtin_arm_stg(__ptr)
 #define __arm_mte_ptrdiff(__ptra, __ptrb) __builtin_arm_subp(__ptra, __ptrb)
 #endif
 
+/* Transactional Memory Extension (TME) Intrinsics */
+#if __ARM_FEATURE_TME
+
+#define _TMFAILURE_REASON  0x00007fffu
+#define _TMFAILURE_RTRY    0x00008000u
+#define _TMFAILURE_CNCL    0x00010000u
+#define _TMFAILURE_MEM     0x00020000u
+#define _TMFAILURE_IMP     0x00040000u
+#define _TMFAILURE_ERR     0x00080000u
+#define _TMFAILURE_SIZE    0x00100000u
+#define _TMFAILURE_NEST    0x00200000u
+#define _TMFAILURE_DBG     0x00400000u
+#define _TMFAILURE_INT     0x00800000u
+#define _TMFAILURE_TRIVIAL 0x01000000u
+
+#define __tstart()        __builtin_arm_tstart()
+#define __tcommit()       __builtin_arm_tcommit()
+#define __tcancel(__arg)  __builtin_arm_tcancel(__arg)
+#define __ttest()         __builtin_arm_ttest()
+
+#endif /* __ARM_FEATURE_TME */
+
 #if defined(__cplusplus)
 }
 #endif
 
 #endif /* __ARM_ACLE_H */

cfe/trunk/lib/Sema/SemaChecking.cpp

bool Sema::CheckAArch64BuiltinFunctionCall(unsigned BuiltinID,
   // For intrinsics which take an immediate value as part of the instruction,
   // range check them here.
   unsigned i = 0, l = 0, u = 0;
   switch (BuiltinID) {
   default: return false;
   case AArch64::BI__builtin_arm_dmb:
   case AArch64::BI__builtin_arm_dsb:
   case AArch64::BI__builtin_arm_isb: l = 0; u = 15; break;
+  case AArch64::BI__builtin_arm_tcancel: l = 0; u = 65535; break;
   }
 
   return SemaBuiltinConstantArgRange(TheCall, i, l, u + l);
 }
 
 bool Sema::CheckHexagonBuiltinCpu(unsigned BuiltinID, CallExpr *TheCall) {
   struct BuiltinAndString {
     unsigned BuiltinID;

cfe/trunk/test/CodeGen/aarch64-tme-tcancel-arg.cpp

+// RUN: %clang_cc1 -triple aarch64-eabi -target-feature +tme -S -emit-llvm %s -o - | FileCheck %s
+
+#define A -1
+constexpr int f() { return 65536; }
+
+void t_cancel() {
+  __builtin_arm_tcancel(f() + A);
+}
+
+// CHECK: call void @llvm.aarch64.tcancel(i64 65535)

cfe/trunk/test/CodeGen/aarch64-tme.c

+// RUN: %clang_cc1 -triple aarch64-eabi -target-feature +tme -S -emit-llvm %s -o - | FileCheck %s
+// RUN: %clang_cc1 -DUSE_ACLE  -triple aarch64-eabi -target-feature +tme -S -emit-llvm %s -o - | FileCheck %s
+
+#ifdef USE_ACLE
+#include "arm_acle.h"
+void test_tme_funcs() {
+  __tstart();
+  (void)__ttest();
+  __tcommit();
+  __tcancel(0x789a);
+}
+#else
+void test_tme_funcs() {
+  __builtin_arm_tstart();
+  (void)__builtin_arm_ttest();
+  __builtin_arm_tcommit();
+  __builtin_arm_tcancel(0x789a);
+}
+#endif
+// CHECK: call i64 @llvm.aarch64.tstart()
+// CHECK: call i64 @llvm.aarch64.ttest()
+// CHECK: call void @llvm.aarch64.tcommit()
+// CHECK: call void @llvm.aarch64.tcancel(i64 30874)
+
+// CHECK: declare i64 @llvm.aarch64.tstart() #1
+// CHECK: declare i64 @llvm.aarch64.ttest() #1
+// CHECK: declare void @llvm.aarch64.tcommit() #1
+// CHECK: declare void @llvm.aarch64.tcancel(i64 immarg) #2
+
+#ifdef __ARM_FEATURE_TME
+void arm_feature_tme_defined() {}
+#endif
+// CHECK: define void @arm_feature_tme_defined()
+
+// CHECK: attributes #1 = { nounwind }
+// CHECK: attributes #2 = { noreturn nounwind }

cfe/trunk/test/Sema/aarch64-tme-errors.c

+// RUN: %clang_cc1 -triple aarch64-eabi -verify %s
+
+#include "arm_acle.h"
+
+void test_no_tme_funcs() {
+  __tstart();         // expected-warning{{implicit declaration of function '__tstart'}}
+  __builtin_tstart(); // expected-error{{use of unknown builtin '__builtin_tstart'}}
+}

cfe/trunk/test/Sema/aarch64-tme-tcancel-const-error.c

+// RUN: %clang_cc1 -triple aarch64-eabi -target-feature +tme -verify %s
+void t_cancel(unsigned short u) {
+  __builtin_arm_tcancel(u); // expected-error{{argument to '__builtin_arm_tcancel' must be a constant integer}}
+}

cfe/trunk/test/Sema/aarch64-tme-tcancel-range-error.c

+// RUN: %clang_cc1 -triple aarch64-eabi -target-feature +tme -verify %s
+void t_cancel() {
+  __builtin_arm_tcancel(0x12345u); // expected-error{{argument value 74565 is outside the valid range [0, 65535]}}
+}

llvm/trunk/include/llvm/IR/IntrinsicsAArch64.td

def int_aarch64_gmi   : Intrinsic<[llvm_i64_ty], [llvm_ptr_ty, llvm_i64_ty],
     [IntrNoMem]>;
 def int_aarch64_ldg   : Intrinsic<[llvm_ptr_ty], [llvm_ptr_ty, llvm_ptr_ty],
     [IntrReadMem]>;
 def int_aarch64_stg   : Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty],
     [IntrWriteMem]>;
 def int_aarch64_subp :  Intrinsic<[llvm_i64_ty], [llvm_ptr_ty, llvm_ptr_ty],
     [IntrNoMem]>;
 }
+
+// Transactional Memory Extension (TME) Intrinsics
+let TargetPrefix = "aarch64" in {
+def int_aarch64_tstart  : GCCBuiltin<"__builtin_arm_tstart">,
+                         Intrinsic<[llvm_i64_ty]>;
+
+def int_aarch64_tcommit : GCCBuiltin<"__builtin_arm_tcommit">, Intrinsic<[]>;
+
+def int_aarch64_tcancel : GCCBuiltin<"__builtin_arm_tcancel">,
+                          Intrinsic<[], [llvm_i64_ty],
+                                    [ImmArg<0>, IntrNoMem, IntrHasSideEffects,
+                                     IntrNoReturn]>;
+
+def int_aarch64_ttest   : GCCBuiltin<"__builtin_arm_ttest">,
+                          Intrinsic<[llvm_i64_ty], [],
+                                    [IntrNoMem, IntrHasSideEffects]>;
+}

llvm/trunk/include/llvm/Support/AArch64TargetParser.h

enum ArchExtKind : unsigned {
   AEK_SSBS =        1 << 20,
   AEK_SB =          1 << 21,
   AEK_PREDRES =     1 << 22,
   AEK_SVE2 =        1 << 23,
   AEK_SVE2AES =     1 << 24,
   AEK_SVE2SM4 =     1 << 25,
   AEK_SVE2SHA3 =    1 << 26,
   AEK_BITPERM =     1 << 27,
+  AEK_TME =         1 << 28,
 };
 
 enum class ArchKind {
 #define AARCH64_ARCH(NAME, ID, CPU_ATTR, SUB_ARCH, ARCH_ATTR, ARCH_FPU, ARCH_BASE_EXT) ID,
 #include "AArch64TargetParser.def"
 };
 
 const ARM::ArchNames<ArchKind> AArch64ARCHNames[] = {

llvm/trunk/include/llvm/Support/AArch64TargetParser.def

 AARCH64_ARCH_EXT_NAME("sve2-sha3", AArch64::AEK_SVE2SHA3, "+sve2-sha3", "-sve2-sha3")
 AARCH64_ARCH_EXT_NAME("bitperm",   AArch64::AEK_BITPERM,  "+bitperm", "-bitperm")
 AARCH64_ARCH_EXT_NAME("rcpc",      AArch64::AEK_RCPC,     "+rcpc", "-rcpc")
 AARCH64_ARCH_EXT_NAME("rng",       AArch64::AEK_RAND,     "+rand",  "-rand")
 AARCH64_ARCH_EXT_NAME("memtag",    AArch64::AEK_MTE,      "+mte",   "-mte")
 AARCH64_ARCH_EXT_NAME("ssbs",      AArch64::AEK_SSBS,     "+ssbs",  "-ssbs")
 AARCH64_ARCH_EXT_NAME("sb",        AArch64::AEK_SB,       "+sb",    "-sb")
 AARCH64_ARCH_EXT_NAME("predres",   AArch64::AEK_PREDRES,  "+predres", "-predres")
+AARCH64_ARCH_EXT_NAME("tme",       AArch64::AEK_TME,      "+tme",   "-tme")
 #undef AARCH64_ARCH_EXT_NAME
 
 #ifndef AARCH64_CPU_NAME
 #define AARCH64_CPU_NAME(NAME, ID, DEFAULT_FPU, IS_DEFAULT, DEFAULT_EXT)
 #endif
 AARCH64_CPU_NAME("cortex-a35", ARMV8A, FK_CRYPTO_NEON_FP_ARMV8, false,
                  (AArch64::AEK_CRC))
 AARCH64_CPU_NAME("cortex-a53", ARMV8A, FK_CRYPTO_NEON_FP_ARMV8, true,

llvm/trunk/lib/Target/AArch64/AArch64.td

def FeatureBranchTargetId : SubtargetFeature<"bti", "HasBTI",
     "true", "Enable Branch Target Identification" >;
 
 def FeatureRandGen : SubtargetFeature<"rand", "HasRandGen",
     "true", "Enable Random Number generation instructions" >;
 
 def FeatureMTE : SubtargetFeature<"mte", "HasMTE",
     "true", "Enable Memory Tagging Extension" >;
 
+def FeatureTME : SubtargetFeature<"tme", "HasTME",
+    "true", "Enable Transactional Memory Extension" >;
+
 //===----------------------------------------------------------------------===//
 // Architectures.
 //
 
 def HasV8_1aOps : SubtargetFeature<"v8.1a", "HasV8_1aOps", "true",
   "Support ARM v8.1a instructions", [FeatureCRC, FeatureLSE, FeatureRDM,
   FeaturePAN, FeatureLOR, FeatureVH]>;
 

llvm/trunk/lib/Target/AArch64/AArch64InstrFormats.td

 }
 def logical_imm32_not : Operand<i32> {
   let ParserMatchClass = LogicalImm32NotOperand;
 }
 def logical_imm64_not : Operand<i64> {
   let ParserMatchClass = LogicalImm64NotOperand;
 }
 
-// imm0_65535 predicate - True if the immediate is in the range [0,65535].
-def imm0_65535 : Operand<i32>, ImmLeaf<i32, [{
+// iXX_imm0_65535 predicates - True if the immediate is in the range [0,65535].
+let ParserMatchClass = AsmImmRange<0, 65535>, PrintMethod = "printImmHex" in {
+def i32_imm0_65535 : Operand<i32>, ImmLeaf<i32, [{
   return ((uint32_t)Imm) < 65536;
-}]> {
-  let ParserMatchClass = AsmImmRange<0, 65535>;
-  let PrintMethod = "printImmHex";
+}]>;
+
+def i64_imm0_65535 : Operand<i64>, ImmLeaf<i64, [{
+  return ((uint64_t)Imm) < 65536;
+}]>;
 }
 
 // imm0_255 predicate - True if the immediate is in the range [0,255].
 def Imm0_255Operand : AsmImmRange<0,255>;
 
 def imm0_255 : Operand<i32>, ImmLeaf<i32, [{
   return ((uint32_t)Imm) < 256;
 }]> {

 // System instructions which have an Rt register.
 class RtSystemI<bit L, dag oops, dag iops, string asm, string operands>
     : BaseSystemI<L, oops, iops, asm, operands>,
       Sched<[WriteSys]> {
   bits<5> Rt;
   let Inst{4-0} = Rt;
 }
 
+// System instructions for transactional memory extension
+class TMBaseSystemI<bit L, bits<4> CRm, bits<3> op2, dag oops, dag iops,
+                    string asm, string operands, list<dag> pattern>
+    : BaseSystemI<L, oops, iops, asm, operands, pattern>,
+      Sched<[WriteSys]> {
+  let Inst{20-12} = 0b000110011;
+  let Inst{11-8} = CRm;
+  let Inst{7-5} = op2;
+  let DecoderMethod = "";
+
+  let mayLoad = 1;
+  let mayStore = 1;
+}
+
+// System instructions for transactional memory - single input operand
+class TMSystemI<bits<4> CRm, string asm, list<dag> pattern>
+    : TMBaseSystemI<0b1, CRm, 0b011,
+                    (outs GPR64:$Rt), (ins), asm, "\t$Rt", pattern> {
+  bits<5> Rt;
+  let Inst{4-0} = Rt;
+}
+
+// System instructions for transactional memory - no operand
+class TMSystemINoOperand<bits<4> CRm, string asm, list<dag> pattern>
+    : TMBaseSystemI<0b0, CRm, 0b011, (outs), (ins), asm, "", pattern> {
+  let Inst{4-0} = 0b11111;
+}
+
+// System instructions for exit from transactions
+let mayLoad = 0, mayStore = 0, hasSideEffects = 1 in
+class TMSystemException<bits<3> op1, string asm, list<dag> pattern>
+    : I<(outs), (ins i64_imm0_65535:$imm), asm, "\t$imm", "", pattern>,
+      Sched<[WriteSys]> {
+  bits<16> imm;
+  let Inst{31-24} = 0b11010100;
+  let Inst{23-21} = op1;
+  let Inst{20-5}  = imm;
+  let Inst{4-0}   = 0b00000;
+}
+
 // Hint instructions that take both a CRm and a 3-bit immediate.
 // NOTE: ideally, this would have mayStore = 0, mayLoad = 0, but we cannot
 // model patterns with sufficiently fine granularity
 let mayStore = 1, mayLoad = 1, hasSideEffects = 1 in
   class HintI<string mnemonic>
       : SimpleSystemI<0, (ins imm0_127:$imm), mnemonic#"\t$imm", "",
                       [(int_aarch64_hint imm0_127:$imm)]>,
         Sched<[WriteHint]> {

 }
 
 //---
 // Exception generation
 //---
 
 let mayLoad = 0, mayStore = 0, hasSideEffects = 1 in
 class ExceptionGeneration<bits<3> op1, bits<2> ll, string asm>
-    : I<(outs), (ins imm0_65535:$imm), asm, "\t$imm", "", []>,
+    : I<(outs), (ins i32_imm0_65535:$imm), asm, "\t$imm", "", []>,
       Sched<[WriteSys]> {
   bits<16> imm;
   let Inst{31-24} = 0b11010100;
   let Inst{23-21} = op1;
   let Inst{20-5}  = imm;
   let Inst{4-2}   = 0b000;
   let Inst{1-0}   = ll;
 }

llvm/trunk/lib/Target/AArch64/AArch64InstrInfo.td

 def HasPredRes      : Predicate<"Subtarget->hasPredRes()">,
                        AssemblerPredicate<"FeaturePredRes", "predres">;
 def HasCCDP          : Predicate<"Subtarget->hasCCDP()">,
                        AssemblerPredicate<"FeatureCacheDeepPersist", "ccdp">;
 def HasBTI           : Predicate<"Subtarget->hasBTI()">,
                        AssemblerPredicate<"FeatureBranchTargetId", "bti">;
 def HasMTE           : Predicate<"Subtarget->hasMTE()">,
                        AssemblerPredicate<"FeatureMTE", "mte">;
+def HasTME           : Predicate<"Subtarget->hasTME()">,
+                       AssemblerPredicate<"FeatureTME", "tme">;
 def IsLE             : Predicate<"Subtarget->isLittleEndian()">;
 def IsBE             : Predicate<"!Subtarget->isLittleEndian()">;
 def IsWindows        : Predicate<"Subtarget->isTargetWindows()">;
 def UseAlternateSExtLoadCVTF32
     : Predicate<"Subtarget->useAlternateSExtLoadCVTF32Pattern()">;
 
 def UseNegativeImmediates
     : Predicate<"false">, AssemblerPredicate<"!FeatureNoNegativeImmediates",

 // Generic system instructions
 def SYSxt  : SystemXtI<0, "sys">;
 def SYSLxt : SystemLXtI<1, "sysl">;
 
 def : InstAlias<"sys $op1, $Cn, $Cm, $op2",
                 (SYSxt imm0_7:$op1, sys_cr_op:$Cn,
                  sys_cr_op:$Cm, imm0_7:$op2, XZR)>;
 
+
+let Predicates = [HasTME] in {
+
+def TSTART : TMSystemI<0b0000, "tstart", [(set GPR64:$Rt, (int_aarch64_tstart))]>;
+
+def TCOMMIT : TMSystemINoOperand<0b0000, "tcommit", [(int_aarch64_tcommit)]>;
+
+let mayLoad = 0, mayStore = 0 in {
+def TTEST : TMSystemI<0b0001, "ttest", [(set GPR64:$Rt, (int_aarch64_ttest))]>;
+def TCANCEL : TMSystemException<0b011, "tcancel", [(int_aarch64_tcancel i64_imm0_65535:$imm)]> {
+  let isBarrier = 1;
+}
+}
+} // HasTME
+
 //===----------------------------------------------------------------------===//
 // Move immediate instructions.
 //===----------------------------------------------------------------------===//
 
 defm MOVK : InsertImmediate<0b11, "movk">;
 defm MOVN : MoveImmediate<0b00, "movn">;
 
 let PostEncoderMethod = "fixMOVZ" in
 defm MOVZ : MoveImmediate<0b10, "movz">;
 
 // First group of aliases covers an implicit "lsl #0".
-def : InstAlias<"movk $dst, $imm", (MOVKWi GPR32:$dst, imm0_65535:$imm, 0), 0>;
-def : InstAlias<"movk $dst, $imm", (MOVKXi GPR64:$dst, imm0_65535:$imm, 0), 0>;
-def : InstAlias<"movn $dst, $imm", (MOVNWi GPR32:$dst, imm0_65535:$imm, 0)>;
-def : InstAlias<"movn $dst, $imm", (MOVNXi GPR64:$dst, imm0_65535:$imm, 0)>;
-def : InstAlias<"movz $dst, $imm", (MOVZWi GPR32:$dst, imm0_65535:$imm, 0)>;
-def : InstAlias<"movz $dst, $imm", (MOVZXi GPR64:$dst, imm0_65535:$imm, 0)>;
+def : InstAlias<"movk $dst, $imm", (MOVKWi GPR32:$dst, i32_imm0_65535:$imm, 0), 0>;
+def : InstAlias<"movk $dst, $imm", (MOVKXi GPR64:$dst, i32_imm0_65535:$imm, 0), 0>;
+def : InstAlias<"movn $dst, $imm", (MOVNWi GPR32:$dst, i32_imm0_65535:$imm, 0)>;
+def : InstAlias<"movn $dst, $imm", (MOVNXi GPR64:$dst, i32_imm0_65535:$imm, 0)>;
+def : InstAlias<"movz $dst, $imm", (MOVZWi GPR32:$dst, i32_imm0_65535:$imm, 0)>;
+def : InstAlias<"movz $dst, $imm", (MOVZXi GPR64:$dst, i32_imm0_65535:$imm, 0)>;
 
 // Next, we have various ELF relocations with the ":XYZ_g0:sym" syntax.
 def : InstAlias<"movz $Rd, $sym", (MOVZXi GPR64:$Rd, movz_symbol_g3:$sym, 48)>;
 def : InstAlias<"movz $Rd, $sym", (MOVZXi GPR64:$Rd, movz_symbol_g2:$sym, 32)>;
 def : InstAlias<"movz $Rd, $sym", (MOVZXi GPR64:$Rd, movz_symbol_g1:$sym, 16)>;
 def : InstAlias<"movz $Rd, $sym", (MOVZXi GPR64:$Rd, movz_symbol_g0:$sym, 0)>;
 
 def : InstAlias<"movn $Rd, $sym", (MOVNXi GPR64:$Rd, movz_symbol_g3:$sym, 48)>;

llvm/trunk/lib/Target/AArch64/AArch64Subtarget.h

protected:
   bool HasSpecRestrict = false;
   bool HasSSBS = false;
   bool HasSB = false;
   bool HasPredRes = false;
   bool HasCCDP = false;
   bool HasBTI = false;
   bool HasRandGen = false;
   bool HasMTE = false;
+  bool HasTME = false;
 
   // Arm SVE2 extensions
   bool HasSVE2AES = false;
   bool HasSVE2SM4 = false;
   bool HasSVE2SHA3 = false;
   bool HasSVE2BitPerm = false;
 
   // HasZeroCycleRegMove - Has zero-cycle register mov instructions.
public:
   bool hasSpecRestrict() const { return HasSpecRestrict; }
   bool hasSSBS() const { return HasSSBS; }
   bool hasSB() const { return HasSB; }
   bool hasPredRes() const { return HasPredRes; }
   bool hasCCDP() const { return HasCCDP; }
   bool hasBTI() const { return HasBTI; }
   bool hasRandGen() const { return HasRandGen; }
   bool hasMTE() const { return HasMTE; }
+  bool hasTME() const { return HasTME; }
   // Arm SVE2 extensions
   bool hasSVE2AES() const { return HasSVE2AES; }
   bool hasSVE2SM4() const { return HasSVE2SM4; }
   bool hasSVE2SHA3() const { return HasSVE2SHA3; }
   bool hasSVE2BitPerm() const { return HasSVE2BitPerm; }
 
   bool isLittleEndian() const { return IsLittle; }
 

llvm/trunk/test/CodeGen/AArch64/tme-tcancel.ll

+; RUN: llc %s -o - | FileCheck %s
+
+target triple = "aarch64-unknown-unknown-eabi"
+
+define void @test_tcancel() #0 {
+  tail call void @llvm.aarch64.tcancel(i64 0) #1
+  unreachable
+}
+
+declare void @llvm.aarch64.tcancel(i64 immarg) #1
+
+attributes #0 = { "target-features"="+tme" }
+attributes #1 = { nounwind noreturn }
+
+; CHECK-LABEL: test_tcancel
+; CHECK: tcancel

llvm/trunk/test/CodeGen/AArch64/tme-tcommit.ll

+; RUN: llc %s -o - | FileCheck %s
+
+target triple = "aarch64-unknown-unknown-eabi"
+
+define void @test_tcommit() #0 {
+  tail call void @llvm.aarch64.tcommit()
+  ret void
+}
+
+declare void @llvm.aarch64.tcommit() #1
+
+attributes #0 = { "target-features"="+tme" }
+attributes #1 = { nounwind }
+
+; CHECK-LABEL: test_tcommit
+; CHECK: tcommit

llvm/trunk/test/CodeGen/AArch64/tme-tstart.ll

+; RUN: llc %s -o - | FileCheck %s
+
+target triple = "aarch64-unknown-unknown-eabi"
+
+define i64 @test_tstart() #0 {
+  %r = tail call i64 @llvm.aarch64.tstart()
+  ret i64 %r
+}
+
+declare i64 @llvm.aarch64.tstart() #1
+
+attributes #0 = { "target-features"="+tme" }
+attributes #1 = { nounwind }
+
+; CHECK-LABEL: test_tstart
+; CHECK: tstart x

llvm/trunk/test/CodeGen/AArch64/tme-ttest.ll

+; RUN: llc %s -o - | FileCheck %s
+
+target triple = "aarch64-unknown-unknown-eabi"
+
+define i64 @test_ttest() #0 {
+  %r = tail call i64 @llvm.aarch64.ttest()
+  ret i64 %r
+}
+
+declare i64 @llvm.aarch64.ttest() #1
+
+attributes #0 = { "target-features"="+tme" }
+attributes #1 = { nounwind }
+
+; CHECK-LABEL: test_ttest
+; CHECK: ttest x

llvm/trunk/test/MC/AArch64/tme-error.s

+// Tests for transactional memory extension instructions
+// RUN: not llvm-mc -triple aarch64 -show-encoding -mattr=+tme < %s 2>&1   | FileCheck %s
+
+tstart
+// CHECK: error: too few operands for instruction
+// CHECK-NEXT: tstart
+tstart  x4, x5
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: tstart x4, x5
+tstart  x4, #1
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: tstart x4, #1
+tstart  sp
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: tstart sp
+
+ttest
+// CHECK: error: too few operands for instruction
+// CHECK-NEXT: ttest
+ttest  x4, x5
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: ttest x4, x5
+ttest  x4, #1
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: ttest x4, #1
+ttest  sp
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: ttest sp
+
+tcommit  x4
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: tcommit x4
+tcommit  sp
+// CHECK: error: invalid operand for instruction
+// CHECK-NEXT: tcommit sp
+
+
+tcancel
+// CHECK: error: too few operands for instruction
+// CHECK-NEXT tcancel
+tcancel x0
+// CHECK: error: immediate must be an integer in range [0, 65535]
+// CHECK-NEXT tcancel
+tcancel #65536
+// CHECK: error: immediate must be an integer in range [0, 65535]
+// CHECK-NEXT: tcancel #65536
+

llvm/trunk/test/MC/AArch64/tme.s

+// Tests for transaction memory extension instructions
+//
+// RUN:     llvm-mc -triple aarch64 -show-encoding -mattr=+tme   < %s      | FileCheck %s
+// RUN: not llvm-mc -triple aarch64 -show-encoding -mattr=-tme   < %s 2>&1 | FileCheck %s --check-prefix=NOTME
+
+tstart x3
+ttest  x4
+tcommit
+tcancel #0x1234
+
+// CHECK: tstart x3         // encoding: [0x63,0x30,0x23,0xd5]
+// CHECK: ttest x4          // encoding: [0x64,0x31,0x23,0xd5]
+// CHECK: tcommit           // encoding: [0x7f,0x30,0x03,0xd5]
+// CHECK: tcancel #0x1234   // encoding: [0x80,0x46,0x62,0xd4]
+
+
+// NOTME: instruction requires: tme
+// NOTME-NEXT: tstart x3
+// NOTME: instruction requires: tme
+// NOTME-NEXT: ttest  x4
+// NOTME: instruction requires: tme
+// NOTME-NEXT: tcommit
+// NOTME: instruction requires: tme
+// NOTME-NEXT: tcancel #0x1234

llvm/trunk/test/MC/Disassembler/AArch64/tme.txt

+# Tests for transaction memory extension instructions
+# RUN:     llvm-mc -triple=aarch64 -mattr=+tme   -disassemble < %s      | FileCheck %s
+# RUN: not llvm-mc -triple=aarch64 -mattr=-tme   -disassemble < %s 2>&1 | FileCheck %s --check-prefix=NOTME
+
+[0x63,0x30,0x23,0xd5]
+[0x64,0x31,0x23,0xd5]
+[0x7f,0x30,0x03,0xd5]
+[0x80,0x46,0x62,0xd4]
+
+# CHECK: tstart x3
+# CHECK: ttest  x4
+# CHECK: tcommit
+# CHECK: tcancel #0x1234
+
+# NOTEME: mrs
+# NOTEME-NEXT: mrs
+# NOTEME-NEXT: msr
+# NOTME:      warning: invalid instruction encoding
+# NOTME-NEXT: [0x80,0x46,0x62,0xd4]

llvm/trunk/unittests/Support/TargetParserTest.cpp

const char *ArchExt[][4] = {{"crc", "nocrc", "+crc", "-crc"},
                                "-sve2-sm4"},
                               {"sve2-sha3", "nosve2-sha3", "+sve2-sha3",
                                "-sve2-sha3"},
                               {"bitperm", "nobitperm", "+bitperm", "-bitperm"},
                               {"dotprod", "nodotprod", "+dotprod", "-dotprod"},
                               {"rcpc", "norcpc", "+rcpc", "-rcpc" },
                               {"rng", "norng", "+rand", "-rand"},
                               {"memtag", "nomemtag", "+mte", "-mte"},
+                              {"tme", "notme", "+tme", "-tme"},
                               {"ssbs", "nossbs", "+ssbs", "-ssbs"},
                               {"sb", "nosb", "+sb", "-sb"},
                               {"predres", "nopredres", "+predres", "-predres"}
 };
 
   for (unsigned i = 0; i < array_lengthof(ArchExt); i++) {
     EXPECT_EQ(StringRef(ArchExt[i][2]),
               AArch64::getArchExtFeature(ArchExt[i][0]));
     EXPECT_EQ(StringRef(ArchExt[i][3]),
               AArch64::getArchExtFeature(ArchExt[i][1]));
   }
 }
 }