[AArch64] Add SVE2.1 target feature for Armv9-A 2022 Architecture Extension

[david-arm]

Summary

First patch in a series adding MC layer support for SVE2.1.

This patch adds the following feature:

sve2p1

Some of the existing SVE instructions added for SME are now
also available under the sve2p1 feature, which are now guarded
by the HasSVE2p1_or_HasSME predicate.

The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09

[sdesmalen]

I think there is a case for this missing in AArch64AsmParser.cpp for the march assembler directive. Can you add that with a corresponding test? (see e.g. llvm/test/MC/AArch64/SME2/directive-arch.s in D135455)

[david-arm]

• Moved existing SVE instructions added for SME into AArch64SVEInstrInfo.td and guarded them a new HasSVE2p1OrSME predicate, since these can now be enabled by SVE2p1.
• Added directive-arch tests for the new SVE2 feature.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64.td

166

Extra whitespace

llvm/lib/Target/AArch64/AArch64InstrInfo.td

159

I know the precedent has been set but I find this awkward to parse. Is HasSVE2p1_or_HasSME a terrible idea? If agreeable we can always convert the other instance later on.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3570

Existing precedent is just End HasSVE2p1orSME.

llvm/test/MC/AArch64/SME2/feature-sve2p1-implies-sve2.s

7 ↗

(On Diff #469272)

Does the CHECK-NOT line provide any value? By which I mean such a message will be followed by an exit(1) and so the lit test will fail anyway?

[david-arm]

llvm/lib/Target/AArch64/AArch64InstrInfo.td

159

I don't mind changing it, but I do think it makes sense to also fix up the other names too for consistency (perhaps in a later patch?). I guess your reason for wanting to put 'Has' twice in the name is to make it easily searchable?

@sdesmalen any thoughts?

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64InstrInfo.td

159

Sure, definitely a later patch for the existing cases. The double Has is to reflect this is boolean logic of two existing feature flags, which as you say means they'll show up in the typical search query,

[david-arm]

• Changed predicate from HasSVE2p1orSME2 to HasSVE2p1_or_HasSME2.
• Removed CHECK-NOT from directive.s test.

[david-arm]

llvm/lib/Target/AArch64/AArch64.td

166

The whitespace is deliberate because all the other features have a newline separating them too. I did this for consistency that's all.

[sdesmalen]

llvm/lib/Target/AArch64/AArch64InstrInfo.td

159

Yes it would make searching easier, so I'm happy with the suggestion

[sdesmalen]

llvm/lib/Target/AArch64/AArch64InstrInfo.td

162–164

nit: Should there be a new-line between def HasSVE2p1_or_HasSME and def HasSVE2p1_or_HasSME2 ?

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64.td

166

Perhaps you've misunderstood my comment? I just meant there is a double whitespace within Extension 2.1 instructions.

[david-arm]

• As a result of @c-rhodes's request on https://reviews.llvm.org/D136427 I'm updating the chain of patches to move all tests into a new llvm/test/MC/AArch64/SVE2p1 directory.
• Removed extra space feature.

[david-arm]

llvm/lib/Target/AArch64/AArch64.td

166

Ah yes I see what you mean now. The colour scheme on my browser basically just shows two shades of yellow so it's not very obvious. :)

llvm/include/llvm/Support/AArch64TargetParser.h

enum ArchExtKind : uint64_t {
   AEK_FLAGM =       1ULL << 36, // FEAT_FlagM
   AEK_SME =         1ULL << 37, // FEAT_SME
   AEK_SMEF64F64 =   1ULL << 38, // FEAT_SME_F64F64
   AEK_SMEI16I64 =   1ULL << 39, // FEAT_SME_I16I64
   AEK_HBC =         1ULL << 40, // FEAT_HBC
   AEK_MOPS =        1ULL << 41, // FEAT_MOPS
   AEK_PERFMON =     1ULL << 42, // FEAT_PMUv3
   AEK_SME2 =        1ULL << 43, // FEAT_SME2
+  AEK_SVE2p1 =      1ULL << 44, // FEAT_SVE2p1
 };
 
 enum class ArchKind {
 #define AARCH64_ARCH(NAME, ID, CPU_ATTR, SUB_ARCH, ARCH_ATTR, ARCH_FPU, ARCH_BASE_EXT) ID,
 #include "AArch64TargetParser.def"
 };
 
 const ARM::ArchNames<ArchKind> AArch64ARCHNames[] = {

llvm/include/llvm/Support/AArch64TargetParser.def

 AARCH64_ARCH_EXT_NAME("profile",      AArch64::AEK_PROFILE,     "+spe",          "-spe")
 AARCH64_ARCH_EXT_NAME("ras",          AArch64::AEK_RAS,         "+ras",          "-ras")
 AARCH64_ARCH_EXT_NAME("sve",          AArch64::AEK_SVE,         "+sve",          "-sve")
 AARCH64_ARCH_EXT_NAME("sve2",         AArch64::AEK_SVE2,        "+sve2",         "-sve2")
 AARCH64_ARCH_EXT_NAME("sve2-aes",     AArch64::AEK_SVE2AES,     "+sve2-aes",     "-sve2-aes")
 AARCH64_ARCH_EXT_NAME("sve2-sm4",     AArch64::AEK_SVE2SM4,     "+sve2-sm4",     "-sve2-sm4")
 AARCH64_ARCH_EXT_NAME("sve2-sha3",    AArch64::AEK_SVE2SHA3,    "+sve2-sha3",    "-sve2-sha3")
 AARCH64_ARCH_EXT_NAME("sve2-bitperm", AArch64::AEK_SVE2BITPERM, "+sve2-bitperm", "-sve2-bitperm")
+AARCH64_ARCH_EXT_NAME("sve2p1",       AArch64::AEK_SVE2p1,      "+sve2p1",       "-sve2p1")
 AARCH64_ARCH_EXT_NAME("rcpc",         AArch64::AEK_RCPC,        "+rcpc",         "-rcpc")
 AARCH64_ARCH_EXT_NAME("rng",          AArch64::AEK_RAND,        "+rand",         "-rand")
 AARCH64_ARCH_EXT_NAME("memtag",       AArch64::AEK_MTE,         "+mte",          "-mte")
 AARCH64_ARCH_EXT_NAME("ssbs",         AArch64::AEK_SSBS,        "+ssbs",         "-ssbs")
 AARCH64_ARCH_EXT_NAME("sb",           AArch64::AEK_SB,          "+sb",           "-sb")
 AARCH64_ARCH_EXT_NAME("predres",      AArch64::AEK_PREDRES,     "+predres",      "-predres")
 AARCH64_ARCH_EXT_NAME("bf16",         AArch64::AEK_BF16,        "+bf16",         "-bf16")
 AARCH64_ARCH_EXT_NAME("i8mm",         AArch64::AEK_I8MM,        "+i8mm",         "-i8mm")

llvm/lib/Target/AArch64/AArch64.td

def FeatureSVE2SM4 : SubtargetFeature<"sve2-sm4", "HasSVE2SM4", "true",
   "Enable SM4 SVE2 instructions (FEAT_SVE_SM4)", [FeatureSVE2, FeatureSM4]>;
 
 def FeatureSVE2SHA3 : SubtargetFeature<"sve2-sha3", "HasSVE2SHA3", "true",
   "Enable SHA3 SVE2 instructions (FEAT_SVE_SHA3)", [FeatureSVE2, FeatureSHA3]>;
 
 def FeatureSVE2BitPerm : SubtargetFeature<"sve2-bitperm", "HasSVE2BitPerm", "true",
   "Enable bit permutation SVE2 instructions (FEAT_SVE_BitPerm)", [FeatureSVE2]>;
 
+def FeatureSVE2p1: SubtargetFeature<"sve2p1", "HasSVE2p1", "true",
+  "Enable Scalable Vector Extension 2.1 instructions", [FeatureSVE2]>;
+
 def FeatureZCRegMove : SubtargetFeature<"zcm", "HasZeroCycleRegMove", "true",
                                         "Has zero-cycle register moves">;
 
 def FeatureZCZeroingGP : SubtargetFeature<"zcz-gp", "HasZeroCycleZeroingGP", "true",
                                         "Has zero-cycle zeroing instructions for generic registers">;
 
 // It is generally beneficial to rewrite "fmov s0, wzr" to "movi d0, #0".
 // as movi is more efficient across all cores. Newer cores can eliminate
let F = [HasSVE, HasSVE2, HasSVE2AES, HasSVE2SM4, HasSVE2SHA3,
            HasSVE2BitPerm, HasSVEorSME, HasSVE2orSME];
 }
 
 def PAUnsupported : AArch64Unsupported {
   let F = [HasPAuth];
 }
 
 def SMEUnsupported : AArch64Unsupported {
-  let F = [HasSME, HasSMEF64F64, HasSMEI16I64, HasSME2];
+  let F = [HasSME, HasSMEF64F64, HasSMEI16I64, HasSME2, HasSVE2p1_or_HasSME2];
 }
 
 include "AArch64SchedA53.td"
 include "AArch64SchedA55.td"
 include "AArch64SchedA57.td"
 include "AArch64SchedCyclone.td"
 include "AArch64SchedFalkor.td"
 include "AArch64SchedKryo.td"

llvm/lib/Target/AArch64/AArch64InstrInfo.td

def HasSPE           : Predicate<"Subtarget->hasSPE()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSPE), "spe">;
 def HasFuseAES       : Predicate<"Subtarget->hasFuseAES()">,
                                  AssemblerPredicateWithAll<(all_of FeatureFuseAES),
                                  "fuse-aes">;
 def HasSVE           : Predicate<"Subtarget->hasSVE()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE), "sve">;
 def HasSVE2          : Predicate<"Subtarget->hasSVE2()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2), "sve2">;
+def HasSVE2p1        : Predicate<"Subtarget->hasSVE2p1()">,
+                                 AssemblerPredicate<(any_of FeatureSVE2p1), "sve2p1">;
 def HasSVE2AES       : Predicate<"Subtarget->hasSVE2AES()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2AES), "sve2-aes">;
 def HasSVE2SM4       : Predicate<"Subtarget->hasSVE2SM4()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2SM4), "sve2-sm4">;
 def HasSVE2SHA3      : Predicate<"Subtarget->hasSVE2SHA3()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2SHA3), "sve2-sha3">;
 def HasSVE2BitPerm   : Predicate<"Subtarget->hasSVE2BitPerm()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2BitPerm), "sve2-bitperm">;

 def HasSVEorSME
     : Predicate<"Subtarget->hasSVE() || Subtarget->hasSME()">,
                 AssemblerPredicateWithAll<(any_of FeatureSVE, FeatureSME),
                 "sve or sme">;
 def HasSVE2orSME
     : Predicate<"Subtarget->hasSVE2() || Subtarget->hasSME()">,
                 AssemblerPredicateWithAll<(any_of FeatureSVE2, FeatureSME),
                 "sve2 or sme">;
+def HasSVE2p1_or_HasSME
+    : Predicate<"Subtarget->hasSVE2p1() || Subtarget->hasSME()">,
+                 AssemblerPredicate<(any_of FeatureSME, FeatureSVE2p1), "sme or sve2p1">;
+def HasSVE2p1_or_HasSME2
+    : Predicate<"Subtarget->hasSVE2p1() || Subtarget->hasSME2()">,
+                 AssemblerPredicate<(any_of FeatureSME2, FeatureSVE2p1), "sme2 or sve2p1">;
 // A subset of NEON instructions are legal in Streaming SVE execution mode,
 // they should be enabled if either has been specified.
 def HasNEONorSME
     : Predicate<"Subtarget->hasNEON() || Subtarget->hasSME()">,
                 AssemblerPredicateWithAll<(any_of FeatureNEON, FeatureSME),
                 "neon or sme">;
 def HasRCPC          : Predicate<"Subtarget->hasRCPC()">,
                                  AssemblerPredicateWithAll<(all_of FeatureRCPC), "rcpc">;

llvm/lib/Target/AArch64/AArch64SMEInstrInfo.td

 def : Pat<(int_aarch64_sme_set_tpidr2 i64:$val),
           (MSR 0xde85, GPR64:$val)>;
 def : Pat<(i64 (int_aarch64_sme_get_tpidr2)),
           (MRS 0xde85)>;
 
 def OBSCURE_COPY : Pseudo<(outs GPR64:$dst), (ins GPR64:$idx), []>, Sched<[]> { }
 def : Pat<(i64 (AArch64ObscureCopy (i64 GPR64:$idx))),
           (OBSCURE_COPY GPR64:$idx)>;
-
-//===----------------------------------------------------------------------===//
-// SVE2 instructions
-//===----------------------------------------------------------------------===//
-
-defm REVD_ZPmZ : sve2_int_perm_revd<"revd", AArch64revd_mt>;
-
-defm SCLAMP_ZZZ : sve2_clamp<"sclamp", 0b0, int_aarch64_sve_sclamp>;
-defm UCLAMP_ZZZ : sve2_clamp<"uclamp", 0b1, int_aarch64_sve_uclamp>;
-
-defm PSEL_PPPRI : sve2_int_perm_sel_p<"psel", int_aarch64_sve_psel>;
-
 } // End let Predicates = [HasSME]
 
 //===----------------------------------------------------------------------===//
 // SME2 Instructions
 //===----------------------------------------------------------------------===//
 let Predicates = [HasSME2] in {
 defm ADD_VG2_M2ZZ_S  : sme2_mla_add_sub_array_vg2_single_S<"add", 0b10>;
 defm ADD_VG4_M4ZZ_S  : sme2_mla_add_sub_array_vg4_single_S<"add", 0b10>;

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

 } // End HasSVE2SHA3
 
 let Predicates = [HasSVE2BitPerm] in {
   // SVE2 bitwise permute
   defm BEXT_ZZZ : sve2_misc_bitwise<0b1100, "bext", int_aarch64_sve_bext_x>;
   defm BDEP_ZZZ : sve2_misc_bitwise<0b1101, "bdep", int_aarch64_sve_bdep_x>;
   defm BGRP_ZZZ : sve2_misc_bitwise<0b1110, "bgrp", int_aarch64_sve_bgrp_x>;
 } // End HasSVE2BitPerm
+
+//===----------------------------------------------------------------------===//
+// SME or SVE2.1 instructions
+//===----------------------------------------------------------------------===//
+
+let Predicates = [HasSVE2p1_or_HasSME] in {
+defm REVD_ZPmZ : sve2_int_perm_revd<"revd", AArch64revd_mt>;
+
+defm SCLAMP_ZZZ : sve2_clamp<"sclamp", 0b0, int_aarch64_sve_sclamp>;
+defm UCLAMP_ZZZ : sve2_clamp<"uclamp", 0b1, int_aarch64_sve_uclamp>;
+
+defm PSEL_PPPRI : sve2_int_perm_sel_p<"psel", int_aarch64_sve_psel>;
+} // End HasSVE2p1_or_HasSME

llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp

const FeatureBitset Features;
     {"rcpc", {AArch64::FeatureRCPC}},
     {"rng", {AArch64::FeatureRandGen}},
     {"sve", {AArch64::FeatureSVE}},
     {"sve2", {AArch64::FeatureSVE2}},
     {"sve2-aes", {AArch64::FeatureSVE2AES}},
     {"sve2-sm4", {AArch64::FeatureSVE2SM4}},
     {"sve2-sha3", {AArch64::FeatureSVE2SHA3}},
     {"sve2-bitperm", {AArch64::FeatureSVE2BitPerm}},
+    {"sve2p1", {AArch64::FeatureSVE2p1}},
     {"ls64", {AArch64::FeatureLS64}},
     {"xs", {AArch64::FeatureXS}},
     {"pauth", {AArch64::FeaturePAuth}},
     {"flagm", {AArch64::FeatureFlagM}},
     {"rme", {AArch64::FeatureRME}},
     {"sme", {AArch64::FeatureSME}},
     {"sme-f64f64", {AArch64::FeatureSMEF64F64}},
     {"sme-i16i64", {AArch64::FeatureSMEI16I64}},

llvm/test/MC/AArch64/SVE2p1/directive-arch-negative.s

+// RUN: not llvm-mc -triple aarch64 -filetype asm -o - %s 2>&1 | FileCheck %s
+
+.arch armv9-a+sve2p1
+.arch armv9-a+nosve2p1
+sclamp z0.s, z1.s, z2.s
+// CHECK: error: instruction requires: sme or sve2p1
+// CHECK: sclamp z0.s, z1.s, z2.s

llvm/test/MC/AArch64/SVE2p1/directive-arch.s

+// RUN: llvm-mc -triple aarch64 -filetype asm -o - %s 2>&1 | FileCheck %s
+
+.arch armv9-a+sve2p1
+sclamp z0.s, z1.s, z2.s
+// CHECK: sclamp z0.s, z1.s, z2.s

llvm/test/MC/AArch64/SVE2p1/directive-arch_extension-negative.s

+// RUN: not llvm-mc -triple aarch64 -filetype asm -o - %s 2>&1 | FileCheck %s
+
+.arch_extension sve2p1
+.arch_extension nosve2p1
+sclamp z0.s, z1.s, z2.s
+// CHECK: error: instruction requires: sme or sve2p1
+// CHECK: sclamp z0.s, z1.s, z2.s

llvm/test/MC/AArch64/SVE2p1/directive-arch_extension.s

+// RUN: llvm-mc -triple aarch64 -filetype asm -o - %s 2>&1 | FileCheck %s
+
+.arch_extension sve2p1
+sclamp z0.s, z1.s, z2.s
+// CHECK: sclamp z0.s, z1.s, z2.s

llvm/test/MC/AArch64/SVE2p1/feature-sve2p1-implies-sve2.s

+// This test verifies SVE2p1 implies SVE2.
+
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1  2>&1 < %s \
+// RUN:        | FileCheck %s
+
+cmla   z0.b, z1.b, z2.b, #0
+// CHECK: cmla   z0.b, z1.b, z2.b, #0

llvm/unittests/Support/TargetParserTest.cpp

std::vector<uint64_t> Extensions = {
       AArch64::AEK_SVE2SM4, AArch64::AEK_SVE2SHA3, AArch64::AEK_SVE2BITPERM,
       AArch64::AEK_RCPC,    AArch64::AEK_RAND,     AArch64::AEK_MTE,
       AArch64::AEK_SSBS,    AArch64::AEK_SB,       AArch64::AEK_PREDRES,
       AArch64::AEK_BF16,    AArch64::AEK_I8MM,     AArch64::AEK_F32MM,
       AArch64::AEK_F64MM,   AArch64::AEK_TME,      AArch64::AEK_LS64,
       AArch64::AEK_BRBE,    AArch64::AEK_PAUTH,    AArch64::AEK_FLAGM,
       AArch64::AEK_SME,     AArch64::AEK_SMEF64F64, AArch64::AEK_SMEI16I64,
       AArch64::AEK_SME2,    AArch64::AEK_HBC,      AArch64::AEK_MOPS,
-      AArch64::AEK_PERFMON};
+      AArch64::AEK_PERFMON, AArch64::AEK_SVE2p1};
 
   std::vector<StringRef> Features;
 
   uint64_t ExtVal = 0;
   for (auto Ext : Extensions)
     ExtVal |= Ext;
 
   // INVALID and NONE have no feature names.
TEST(TargetParserTest, AArch64ExtensionFeatures) {
   EXPECT_TRUE(llvm::is_contained(Features, "+spe"));
   EXPECT_TRUE(llvm::is_contained(Features, "+ras"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-aes"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-sm4"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-sha3"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-bitperm"));
+  EXPECT_TRUE(llvm::is_contained(Features, "+sve2p1"));
   EXPECT_TRUE(llvm::is_contained(Features, "+rcpc"));
   EXPECT_TRUE(llvm::is_contained(Features, "+rand"));
   EXPECT_TRUE(llvm::is_contained(Features, "+mte"));
   EXPECT_TRUE(llvm::is_contained(Features, "+ssbs"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sb"));
   EXPECT_TRUE(llvm::is_contained(Features, "+predres"));
   EXPECT_TRUE(llvm::is_contained(Features, "+bf16"));
   EXPECT_TRUE(llvm::is_contained(Features, "+i8mm"));
const char *ArchExt[][4] = {
       {"ras", "noras", "+ras", "-ras"},
       {"lse", "nolse", "+lse", "-lse"},
       {"rdm", "nordm", "+rdm", "-rdm"},
       {"sve", "nosve", "+sve", "-sve"},
       {"sve2", "nosve2", "+sve2", "-sve2"},
       {"sve2-aes", "nosve2-aes", "+sve2-aes", "-sve2-aes"},
       {"sve2-sm4", "nosve2-sm4", "+sve2-sm4", "-sve2-sm4"},
       {"sve2-sha3", "nosve2-sha3", "+sve2-sha3", "-sve2-sha3"},
+      {"sve2p1", "nosve2p1", "+sve2p1", "-sve2p1"},
       {"sve2-bitperm", "nosve2-bitperm", "+sve2-bitperm", "-sve2-bitperm"},
       {"dotprod", "nodotprod", "+dotprod", "-dotprod"},
       {"rcpc", "norcpc", "+rcpc", "-rcpc"},
       {"rng", "norng", "+rand", "-rand"},
       {"memtag", "nomemtag", "+mte", "-mte"},
       {"tme", "notme", "+tme", "-tme"},
       {"pauth", "nopauth", "+pauth", "-pauth"},
       {"ssbs", "nossbs", "+ssbs", "-ssbs"},