[AArch64][SVE2] Add the SVE2.1 BF16 instructions

[david-arm]

Summary

This patch adds the new FEAT_B16B16 feature as well as the
assembly/disassembly for all of the B16B16 instructions:

bfadd: BFloat16 floating-point add vectors
bfsub: BFloat16 floating-point subtract vectors
bfmul: BFloat16 floating-point multiply vectors
bfclamp: BFloat16 floating-point clamp to minimum/maximum number
bfmax: BFloat16 floating-point maximum
bfmaxnm: BFloat16 floating-point maximum number
bfmin: BFloat16 floating-point minimum
bfminnm: BFloat16 floating-point minimum number
bfmla: BFloat16 floating-point fused multiply-add vectors
bfmls: BFloat16 floating-point fused multiply-subtract vectors

The reference can be found here:
https://developer.arm.com/documentation/ddi0602/2022-09

[c-rhodes]

llvm/lib/Target/AArch64/AArch64.td

166

unrelated to this patch but I noticed this is missing (FEAT_SVE2p1) like the other feature descriptions

169

BFloat16 to BFloat16

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3701

SVE2.1 or SME2.1 non-widening BFloat16 to BFloat16 instructions?

llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td

22

https://reviews.llvm.org/D137245#inline-1324912 applies here as well

llvm/lib/Target/AArch64/SVEInstrFormats.td

2195

single def so this should be a class

2195

sve2p1 like the class below?

2257

single def so this should be a class

llvm/test/MC/AArch64/SVE2p1/bfadd-diagnostics.s

7

movprfx is only necessary for the last test

14

only merging predication can be specified? This test is no different from the above, I'd either remove it or change it so the predicate register is valid but with zeroing prediction.

23

m

28–35

not sure what's going on here, it's the unpredicated form that can't be preceded by movprfx?

llvm/test/MC/AArch64/SVE2p1/bfmax-diagnostics.s

7

same comments as bfadd-diagnostics.s apply here

llvm/test/MC/AArch64/SVE2p1/bfmaxnm-diagnostics.s

7

same comments as bfadd-diagnostics.s apply here

llvm/test/MC/AArch64/SVE2p1/bfmin-diagnostics.s

7

same comments as bfadd-diagnostics.s apply here

llvm/test/MC/AArch64/SVE2p1/bfminnm-diagnostics.s

7

same comments as bfadd-diagnostics.s apply here

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3707–3708

Do you mind grouping these by class and adding a space between those groups?

llvm/lib/Target/AArch64/SVEInstrFormats.td

2175

Can this just be bits<2> opc? given it's really a 2-bit opcode and then push this on to the uses of sve_fp_fma_by_indexed_elem rather than hiding the fact within sve_fp_fma_by_indexed_elem.

2195

Whilst true, ultimately we're going to require code gen patterns so we'll need this to be a multiclass later on anyway.

[david-arm]

• Rebased on top of D137410 to pick up the SME2p1 feature.
• Corrected some tests that were a bit confused with regards to movprfx.
• Tidied up some of the classes and instruction layouts.

llvm/lib/Target/AArch64/SVEInstrFormats.td

2257

Given Paul's comments above I've just left this as a multiclass if that's ok?

[c-rhodes]

llvm/lib/Target/AArch64/SVEInstrFormats.td

2257

Given Paul's comments above I've just left this as a multiclass if that's ok?

sure that's fine

llvm/include/llvm/Support/AArch64TargetParser.h

enum ArchExtKind : uint64_t {
   AEK_SMEF64F64 =   1ULL << 38, // FEAT_SME_F64F64
   AEK_SMEI16I64 =   1ULL << 39, // FEAT_SME_I16I64
   AEK_HBC =         1ULL << 40, // FEAT_HBC
   AEK_MOPS =        1ULL << 41, // FEAT_MOPS
   AEK_PERFMON =     1ULL << 42, // FEAT_PMUv3
   AEK_SME2 =        1ULL << 43, // FEAT_SME2
   AEK_SVE2p1 =      1ULL << 44, // FEAT_SVE2p1
   AEK_SME2p1 =      1ULL << 45, // FEAT_SME2p1
+  AEK_B16B16 =      1ULL << 46  // FEAT_B16B16
 };
 
 enum class ArchKind {
 #define AARCH64_ARCH(NAME, ID, CPU_ATTR, SUB_ARCH, ARCH_ATTR, ARCH_FPU, ARCH_BASE_EXT) ID,
 #include "AArch64TargetParser.def"
 };
 
 const ARM::ArchNames<ArchKind> AArch64ARCHNames[] = {

llvm/include/llvm/Support/AArch64TargetParser.def

 AARCH64_ARCH_EXT_NAME("ras",          AArch64::AEK_RAS,         "+ras",          "-ras")
 AARCH64_ARCH_EXT_NAME("sve",          AArch64::AEK_SVE,         "+sve",          "-sve")
 AARCH64_ARCH_EXT_NAME("sve2",         AArch64::AEK_SVE2,        "+sve2",         "-sve2")
 AARCH64_ARCH_EXT_NAME("sve2-aes",     AArch64::AEK_SVE2AES,     "+sve2-aes",     "-sve2-aes")
 AARCH64_ARCH_EXT_NAME("sve2-sm4",     AArch64::AEK_SVE2SM4,     "+sve2-sm4",     "-sve2-sm4")
 AARCH64_ARCH_EXT_NAME("sve2-sha3",    AArch64::AEK_SVE2SHA3,    "+sve2-sha3",    "-sve2-sha3")
 AARCH64_ARCH_EXT_NAME("sve2-bitperm", AArch64::AEK_SVE2BITPERM, "+sve2-bitperm", "-sve2-bitperm")
 AARCH64_ARCH_EXT_NAME("sve2p1",       AArch64::AEK_SVE2p1,      "+sve2p1",       "-sve2p1")
+AARCH64_ARCH_EXT_NAME("b16b16",       AArch64::AEK_B16B16,      "+b16b16",       "-b16b16")
 AARCH64_ARCH_EXT_NAME("rcpc",         AArch64::AEK_RCPC,        "+rcpc",         "-rcpc")
 AARCH64_ARCH_EXT_NAME("rng",          AArch64::AEK_RAND,        "+rand",         "-rand")
 AARCH64_ARCH_EXT_NAME("memtag",       AArch64::AEK_MTE,         "+mte",          "-mte")
 AARCH64_ARCH_EXT_NAME("ssbs",         AArch64::AEK_SSBS,        "+ssbs",         "-ssbs")
 AARCH64_ARCH_EXT_NAME("sb",           AArch64::AEK_SB,          "+sb",           "-sb")
 AARCH64_ARCH_EXT_NAME("predres",      AArch64::AEK_PREDRES,     "+predres",      "-predres")
 AARCH64_ARCH_EXT_NAME("bf16",         AArch64::AEK_BF16,        "+bf16",         "-bf16")
 AARCH64_ARCH_EXT_NAME("i8mm",         AArch64::AEK_I8MM,        "+i8mm",         "-i8mm")

llvm/lib/Target/AArch64/AArch64.td

 
 def FeatureSVE2SHA3 : SubtargetFeature<"sve2-sha3", "HasSVE2SHA3", "true",
   "Enable SHA3 SVE2 instructions (FEAT_SVE_SHA3)", [FeatureSVE2, FeatureSHA3]>;
 
 def FeatureSVE2BitPerm : SubtargetFeature<"sve2-bitperm", "HasSVE2BitPerm", "true",
   "Enable bit permutation SVE2 instructions (FEAT_SVE_BitPerm)", [FeatureSVE2]>;
 
 def FeatureSVE2p1: SubtargetFeature<"sve2p1", "HasSVE2p1", "true",
   "Enable Scalable Vector Extension 2.1 instructions", [FeatureSVE2]>;
 
+def FeatureB16B16 : SubtargetFeature<"b16b16", "HasB16B16", "true",
+  "Enable SVE2.1 or SME2.1 non-widening BFloat16 to BFloat16 instructions (FEAT_B16B16)", []>;
+
 def FeatureZCRegMove : SubtargetFeature<"zcm", "HasZeroCycleRegMove", "true",
                                         "Has zero-cycle register moves">;
 
 def FeatureZCZeroingGP : SubtargetFeature<"zcz-gp", "HasZeroCycleZeroingGP", "true",
                                         "Has zero-cycle zeroing instructions for generic registers">;
 
 // It is generally beneficial to rewrite "fmov s0, wzr" to "movi d0, #0".
 // as movi is more efficient across all cores. Newer cores can eliminate

 //===----------------------------------------------------------------------===//
 // Unsupported features to disable for scheduling models
 //===----------------------------------------------------------------------===//
 
 class AArch64Unsupported { list<Predicate> F; }
 
 def SVEUnsupported : AArch64Unsupported {
   let F = [HasSVE, HasSVE2, HasSVE2AES, HasSVE2SM4, HasSVE2SHA3,
-           HasSVE2BitPerm, HasSVEorSME, HasSVE2orSME];
+           HasSVE2BitPerm, HasSVEorSME, HasSVE2p1, HasSVE2orSME, HasSVE2p1_or_HasSME2p1];
 }
 
 def PAUnsupported : AArch64Unsupported {
   let F = [HasPAuth];
 }
 
 def SMEUnsupported : AArch64Unsupported {
-  let F = [HasSME, HasSMEF64F64, HasSMEI16I64, HasSME2, HasSVE2p1_or_HasSME2];
+  let F = [HasSME, HasSMEF64F64, HasSMEI16I64, HasSME2, HasSVE2p1_or_HasSME2, HasSVE2p1_or_HasSME2p1];
 }
 
 include "AArch64SchedA53.td"
 include "AArch64SchedA55.td"
 include "AArch64SchedA57.td"
 include "AArch64SchedCyclone.td"
 include "AArch64SchedFalkor.td"
 include "AArch64SchedKryo.td"

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 def HasSVE2AES       : Predicate<"Subtarget->hasSVE2AES()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2AES), "sve2-aes">;
 def HasSVE2SM4       : Predicate<"Subtarget->hasSVE2SM4()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2SM4), "sve2-sm4">;
 def HasSVE2SHA3      : Predicate<"Subtarget->hasSVE2SHA3()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2SHA3), "sve2-sha3">;
 def HasSVE2BitPerm   : Predicate<"Subtarget->hasSVE2BitPerm()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSVE2BitPerm), "sve2-bitperm">;
+def HasB16B16        : Predicate<"Subtarget->hasB16B16()">,
+                                 AssemblerPredicateWithAll<(all_of FeatureB16B16), "b16b16">;
 def HasSME           : Predicate<"Subtarget->hasSME()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSME), "sme">;
 def HasSMEF64F64     : Predicate<"Subtarget->hasSMEF64F64()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSMEF64F64), "sme-f64f64">;
 def HasSMEI16I64     : Predicate<"Subtarget->hasSMEI16I64()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSMEI16I64), "sme-i16i64">;
 def HasSME2          : Predicate<"Subtarget->hasSME2()">,
                                  AssemblerPredicateWithAll<(all_of FeatureSME2), "sme2">;

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVEorSME] in {
   defm : fma<nxv2f64, nxv2i1, "D">;
 } // End HasSVEorSME
 
 let Predicates = [HasSVE] in {
   defm FTMAD_ZZI : sve_fp_ftmad<"ftmad", int_aarch64_sve_ftmad_x>;
 } // End HasSVE
 
 let Predicates = [HasSVEorSME] in {
-  defm FMLA_ZZZI : sve_fp_fma_by_indexed_elem<0b0, "fmla", int_aarch64_sve_fmla_lane>;
-  defm FMLS_ZZZI : sve_fp_fma_by_indexed_elem<0b1, "fmls", int_aarch64_sve_fmls_lane>;
+  defm FMLA_ZZZI : sve_fp_fma_by_indexed_elem<0b00, "fmla", int_aarch64_sve_fmla_lane>;
+  defm FMLS_ZZZI : sve_fp_fma_by_indexed_elem<0b01, "fmls", int_aarch64_sve_fmls_lane>;
 
   defm FCMLA_ZZZI : sve_fp_fcmla_by_indexed_elem<"fcmla", int_aarch64_sve_fcmla_lane>;
   defm FMUL_ZZZI   : sve_fp_fmul_by_indexed_elem<"fmul", int_aarch64_sve_fmul_lane>;
 } // End HasSVEorSME
 
 let Predicates = [HasSVE] in {
   // SVE floating point reductions.
   defm FADDA_VPZ   : sve_fp_2op_p_vd<0b000, "fadda",   AArch64fadda_p>;

 defm WHILEGT_CXX  : sve2p1_int_while_rr_pn<"whilegt", 0b001>;
 defm WHILELT_CXX  : sve2p1_int_while_rr_pn<"whilelt", 0b010>;
 defm WHILELE_CXX  : sve2p1_int_while_rr_pn<"whilele", 0b011>;
 defm WHILEHS_CXX  : sve2p1_int_while_rr_pn<"whilehs", 0b100>;
 defm WHILEHI_CXX  : sve2p1_int_while_rr_pn<"whilehi", 0b101>;
 defm WHILELO_CXX  : sve2p1_int_while_rr_pn<"whilelo", 0b110>;
 defm WHILELS_CXX  : sve2p1_int_while_rr_pn<"whilels", 0b111>;
 } // End HasSVE2p1_or_HasSME2
+
+//===----------------------------------------------------------------------===//
+// SVE2.1 non-widening BFloat16 to BFloat16 instructions
+//===----------------------------------------------------------------------===//
+
+let Predicates = [HasSVE2p1_or_HasSME2p1, HasB16B16] in {
+def BFADD_ZZZ : sve_fp_3op_u_zd<0b00, 0b000, "bfadd", ZPR16>;
+def BFSUB_ZZZ : sve_fp_3op_u_zd<0b00, 0b001, "bfsub", ZPR16>;
+def BFMUL_ZZZ : sve_fp_3op_u_zd<0b00, 0b010, "bfmul", ZPR16>;
+
+def BFMLA_ZPmZZ : sve_fp_3op_p_zds_a<0b00, 0b00, "bfmla", ZPR16>;
+def BFMLS_ZPmZZ : sve_fp_3op_p_zds_a<0b00, 0b01, "bfmls", ZPR16>;
+
+def BFADD_ZPZmZ   : sve_fp_2op_p_zds<0b00, 0b0000, "bfadd", ZPR16>;
+def BFSUB_ZPZmZ   : sve_fp_2op_p_zds<0b00, 0b0001, "bfsub", ZPR16>;
+def BFMUL_ZPZmZ   : sve_fp_2op_p_zds<0b00, 0b0010, "bfmul", ZPR16>;
+def BFMAXNM_ZPZmZ : sve_fp_2op_p_zds<0b00, 0b0100, "bfmaxnm", ZPR16>;
+def BFMINNM_ZPZmZ : sve_fp_2op_p_zds<0b00, 0b0101, "bfminnm", ZPR16>;
+def BFMAX_ZPZmZ   : sve_fp_2op_p_zds<0b00, 0b0110, "bfmax", ZPR16>;
+def BFMIN_ZPZmZ   : sve_fp_2op_p_zds<0b00, 0b0111, "bfmin", ZPR16>;
+
+defm BFMLA_ZZZI : sve2p1_fp_bfma_by_indexed_elem<"bfmla", 0b10>;
+defm BFMLS_ZZZI : sve2p1_fp_bfma_by_indexed_elem<"bfmls", 0b11>;
+
+defm BFMUL_ZZZI : sve2p1_fp_bfmul_by_indexed_elem<"bfmul">;
+
+def BFCLAMP_ZZZ : sve2p1_fclamp<"bfclamp", 0b00, ZPR16>;
+} // End HasSVE2p1_or_HasSME2p1, HasB16B16

llvm/lib/Target/AArch64/AArch64SchedNeoverseN2.td

 def NeoverseN2Model : SchedMachineModel {
   let IssueWidth            =  10; // Micro-ops dispatched at a time.
   let MicroOpBufferSize     = 160; // Entries in micro-op re-order buffer.
   let LoadLatency           =   4; // Optimistic load latency.
   let MispredictPenalty     =  10; // Extra cycles for mispredicted branch.
   let LoopMicroOpBufferSize =  16; // NOTE: Copied from Cortex-A57.
   let CompleteModel         =   1;
 
-  list<Predicate> UnsupportedFeatures = SMEUnsupported.F;
+  list<Predicate> UnsupportedFeatures = !listconcat(SMEUnsupported.F,
+                                                    [HasSVE2p1]);
 }
 
 //===----------------------------------------------------------------------===//
 // Define each kind of processor resource and number available on Neoverse N2.
 // Instructions are first fetched and then decoded into internal macro-ops
 // (MOPs). From there, the MOPs proceed through register renaming and dispatch
 // stages. A MOP can be split into two micro-ops further down the pipeline
 // after the decode stage. Once dispatched, micro-ops wait for their operands

llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp

const FeatureBitset Features;
     {"rng", {AArch64::FeatureRandGen}},
     {"sve", {AArch64::FeatureSVE}},
     {"sve2", {AArch64::FeatureSVE2}},
     {"sve2-aes", {AArch64::FeatureSVE2AES}},
     {"sve2-sm4", {AArch64::FeatureSVE2SM4}},
     {"sve2-sha3", {AArch64::FeatureSVE2SHA3}},
     {"sve2-bitperm", {AArch64::FeatureSVE2BitPerm}},
     {"sve2p1", {AArch64::FeatureSVE2p1}},
+    {"b16b16", {AArch64::FeatureB16B16}},
     {"ls64", {AArch64::FeatureLS64}},
     {"xs", {AArch64::FeatureXS}},
     {"pauth", {AArch64::FeaturePAuth}},
     {"flagm", {AArch64::FeatureFlagM}},
     {"rme", {AArch64::FeatureRME}},
     {"sme", {AArch64::FeatureSME}},
     {"sme-f64f64", {AArch64::FeatureSMEF64F64}},
     {"sme-i16i64", {AArch64::FeatureSMEI16I64}},

llvm/lib/Target/AArch64/SVEInstrFormats.td

: I<(outs zprty:$Zda), (ins PPR3bAny:$Pg, zprty:$_Zda, zprty:$Zn, zprty:$Zm),
   let Inst{15}    = 0b0;
   let Inst{14-13} = opc;
   let Inst{12-10} = Pg;
   let Inst{9-5}   = Zn;
   let Inst{4-0}   = Zda;
 
   let Constraints = "$Zda = $_Zda";
   let ElementSize = zprty.ElementSize;
+  let DestructiveInstType = DestructiveTernaryCommWithRev;
 }
 
 multiclass sve_fp_3op_p_zds_a<bits<2> opc, string asm, string Ps,
                               SDPatternOperator op, string revname,
                               bit isReverseInstr=0> {
-  let DestructiveInstType = DestructiveTernaryCommWithRev in {
   def _H : sve_fp_3op_p_zds_a<0b01, opc, asm, ZPR16>,
            SVEPseudo2Instr<Ps # _H, 1>, SVEInstr2Rev<NAME # _H, revname # _H, isReverseInstr>;
   def _S : sve_fp_3op_p_zds_a<0b10, opc, asm, ZPR32>,
            SVEPseudo2Instr<Ps # _S, 1>, SVEInstr2Rev<NAME # _S, revname # _S, isReverseInstr>;
   def _D : sve_fp_3op_p_zds_a<0b11, opc, asm, ZPR64>,
            SVEPseudo2Instr<Ps # _D, 1>, SVEInstr2Rev<NAME # _D, revname # _D, isReverseInstr>;
-  }
 
   def : SVE_4_Op_Pat<nxv8f16, op, nxv8i1, nxv8f16, nxv8f16, nxv8f16, !cast<Instruction>(NAME # _H)>;
   def : SVE_4_Op_Pat<nxv4f32, op, nxv4i1, nxv4f32, nxv4f32, nxv4f32, !cast<Instruction>(NAME # _S)>;
   def : SVE_4_Op_Pat<nxv2f64, op, nxv2i1, nxv2f64, nxv2f64, nxv2f64, !cast<Instruction>(NAME # _D)>;
 }
 
 class sve_fp_3op_p_zds_b<bits<2> sz, bits<2> opc, string asm,
                          ZPRRegOp zprty>
multiclass sve_fp_3op_p_zds_zx {
   def _UNDEF_S : PredThreeOpPseudo<NAME # _S, ZPR32, FalseLanesUndef>;
   def _UNDEF_D : PredThreeOpPseudo<NAME # _D, ZPR64, FalseLanesUndef>;
 }
 
 //===----------------------------------------------------------------------===//
 // SVE Floating Point Multiply-Add - Indexed Group
 //===----------------------------------------------------------------------===//
 
-class sve_fp_fma_by_indexed_elem<bits<2> sz, bit opc, string asm,
+class sve_fp_fma_by_indexed_elem<bits<2> sz, bits<2> opc, string asm,
                                  ZPRRegOp zprty1,
                                  ZPRRegOp zprty2, Operand itype>
 : I<(outs zprty1:$Zda), (ins zprty1:$_Zda, zprty1:$Zn, zprty2:$Zm, itype:$iop),
   asm, "\t$Zda, $Zn, $Zm$iop", "", []>, Sched<[]> {
   bits<5> Zda;
   bits<5> Zn;
   let Inst{31-24} = 0b01100100;
   let Inst{23-22} = sz;
   let Inst{21}    = 0b1;
-  let Inst{15-11} = 0;
-  let Inst{10}    = opc;
+  let Inst{15-12} = 0b0000;
+  let Inst{11-10} = opc;
   let Inst{9-5}   = Zn;
   let Inst{4-0}   = Zda;
 
   let Constraints = "$Zda = $_Zda";
   let DestructiveInstType = DestructiveOther;
   let ElementSize = ElementSizeNone;
 }
 
-multiclass sve_fp_fma_by_indexed_elem<bit opc, string asm,
+multiclass sve2p1_fp_bfma_by_indexed_elem<string asm, bits<2> opc> {
+  def NAME : sve_fp_fma_by_indexed_elem<{0, ?}, opc, asm, ZPR16, ZPR3b16,
+                                         VectorIndexH32b> {
+    bits<3> Zm;
+    bits<3> iop;
+    let Inst{22} = iop{2};
+    let Inst{20-19} = iop{1-0};
+    let Inst{18-16} = Zm;
+  }
+}
+
+multiclass sve_fp_fma_by_indexed_elem<bits<2> opc, string asm,
                                       SDPatternOperator op> {
   def _H : sve_fp_fma_by_indexed_elem<{0, ?}, opc, asm, ZPR16, ZPR3b16, VectorIndexH32b> {
     bits<3> Zm;
     bits<3> iop;
     let Inst{22} = iop{2};
     let Inst{20-19} = iop{1-0};
     let Inst{18-16} = Zm;
   }
def : Pat<(nxv2f64 (op nxv2f64:$Op1, nxv2f64:$Op2, nxv2f64:$Op3, (i32 VectorIndexD32b_timm:$idx))),
             (!cast<Instruction>(NAME # _D) $Op1, $Op2, $Op3, VectorIndexD32b_timm:$idx)>;
 }
 
 
 //===----------------------------------------------------------------------===//
 // SVE Floating Point Multiply - Indexed Group
 //===----------------------------------------------------------------------===//
 
-class sve_fp_fmul_by_indexed_elem<bits<2> sz, string asm, ZPRRegOp zprty,
-                                      ZPRRegOp zprty2, Operand itype>
+class sve_fp_fmul_by_indexed_elem<bits<2> sz, bit o2, string asm, ZPRRegOp zprty,
 : I<(outs zprty:$Zd), (ins zprty:$Zn, zprty2:$Zm, itype:$iop),
   asm, "\t$Zd, $Zn, $Zm$iop", "", []>, Sched<[]> {
   bits<5> Zd;
   bits<5> Zn;
   let Inst{31-24} = 0b01100100;
   let Inst{23-22} = sz;
   let Inst{21}    = 0b1;
-  let Inst{15-10} = 0b001000;
+  let Inst{15-12} = 0b0010;
+  let Inst{11}    = o2;
+  let Inst{10}    = 0b0;
   let Inst{9-5}   = Zn;
   let Inst{4-0}   = Zd;
 }
 
+multiclass sve2p1_fp_bfmul_by_indexed_elem<string asm> {
+  def NAME : sve_fp_fmul_by_indexed_elem<{0, ?}, 0b1, asm, ZPR16, ZPR3b16, VectorIndexH32b> {
+    bits<3> Zm;
+    bits<3> iop;
+    let Inst{22} = iop{2};
+    let Inst{20-19} = iop{1-0};
+    let Inst{18-16} = Zm;
+  }
+}
+
 multiclass sve_fp_fmul_by_indexed_elem<string asm, SDPatternOperator op> {
-  def _H : sve_fp_fmul_by_indexed_elem<{0, ?}, asm, ZPR16, ZPR3b16, VectorIndexH32b> {
+  def _H : sve_fp_fmul_by_indexed_elem<{0, ?}, 0b0, asm, ZPR16, ZPR3b16, VectorIndexH32b> {
     bits<3> Zm;
     bits<3> iop;
     let Inst{22} = iop{2};
     let Inst{20-19} = iop{1-0};
     let Inst{18-16} = Zm;
   }
-  def _S : sve_fp_fmul_by_indexed_elem<0b10, asm, ZPR32, ZPR3b32, VectorIndexS32b> {
+  def _S : sve_fp_fmul_by_indexed_elem<0b10, 0b0, asm, ZPR32, ZPR3b32, VectorIndexS32b> {
     bits<3> Zm;
     bits<2> iop;
     let Inst{20-19} = iop;
     let Inst{18-16} = Zm;
   }
-  def _D : sve_fp_fmul_by_indexed_elem<0b11, asm, ZPR64, ZPR4b64, VectorIndexD32b> {
+  def _D : sve_fp_fmul_by_indexed_elem<0b11, 0b0, asm, ZPR64, ZPR4b64, VectorIndexD32b> {
     bits<4> Zm;
     bit iop;
     let Inst{20} = iop;
     let Inst{19-16} = Zm;
   }
 
   def : Pat<(nxv8f16 (op nxv8f16:$Op1, nxv8f16:$Op2, (i32 VectorIndexH32b_timm:$idx))),
             (!cast<Instruction>(NAME # _H) $Op1, $Op2, VectorIndexH32b_timm:$idx)>;

llvm/test/MC/AArch64/SVE2p1/bfadd-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfadd z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfadd z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfadd z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfadd z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfadd z23.h, p1/m, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfadd z23.h, p1/m, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfadd z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfadd z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid use of movprfx
+
+movprfx z23.h, p1/m, z31.h
+bfadd z23.h, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: instruction is unpredictable when following a movprfx, suggest replacing movprfx with mov
+// CHECK-NEXT: bfadd z23.h, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfadd.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+movprfx  z23.h, p3/m, z31.h
+bfadd   z23.h, p3/m, z23.h, z13.h  // 01100101-00000000-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfadd   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65008db7 <unknown>
+
+movprfx z23, z31
+bfadd   z23.h, p3/m, z23.h, z13.h  // 01100101-00000000-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfadd   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65008db7 <unknown>
+
+bfadd   z0.h, p0/m, z0.h, z0.h  // 01100101-00000000-10000000-00000000
+// CHECK-INST: bfadd   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65008000 <unknown>
+
+bfadd   z21.h, p5/m, z21.h, z10.h  // 01100101-00000000-10010101-01010101
+// CHECK-INST: bfadd   z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65009555 <unknown>
+
+bfadd   z23.h, p3/m, z23.h, z13.h  // 01100101-00000000-10001101-10110111
+// CHECK-INST: bfadd   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65008db7 <unknown>
+
+bfadd   z31.h, p7/m, z31.h, z31.h  // 01100101-00000000-10011111-11111111
+// CHECK-INST: bfadd   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65009fff <unknown>
+
+bfadd   z0.h, z0.h, z0.h  // 01100101-00000000-00000000-00000000
+// CHECK-INST: bfadd   z0.h, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x00,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65000000 <unknown>
+
+bfadd   z21.h, z10.h, z21.h  // 01100101-00010101-00000001-01010101
+// CHECK-INST: bfadd   z21.h, z10.h, z21.h
+// CHECK-ENCODING: [0x55,0x01,0x15,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65150155 <unknown>
+
+bfadd   z23.h, z13.h, z8.h  // 01100101-00001000-00000001-10110111
+// CHECK-INST: bfadd   z23.h, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x01,0x08,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 650801b7 <unknown>
+
+bfadd   z31.h, z31.h, z31.h  // 01100101-00011111-00000011-11111111
+// CHECK-INST: bfadd   z31.h, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x03,0x1f,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 651f03ff <unknown>

llvm/test/MC/AArch64/SVE2p1/bfclamp-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfclamp z23.h, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfclamp z23.h, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfclamp z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfclamp z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfclamp.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx z23, z31
+bfclamp z23.h, z13.h, z8.h  // 01100100-00101000-00100101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfclamp z23.h, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x25,0x28,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 642825b7 <unknown>
+
+bfclamp z0.h, z0.h, z0.h  // 01100100-00100000-00100100-00000000
+// CHECK-INST: bfclamp z0.h, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x24,0x20,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64202400 <unknown>
+
+bfclamp z21.h, z10.h, z21.h  // 01100100-00110101-00100101-01010101
+// CHECK-INST: bfclamp z21.h, z10.h, z21.h
+// CHECK-ENCODING: [0x55,0x25,0x35,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64352555 <unknown>
+
+bfclamp z23.h, z13.h, z8.h  // 01100100-00101000-00100101-10110111
+// CHECK-INST: bfclamp z23.h, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x25,0x28,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 642825b7 <unknown>
+
+bfclamp z31.h, z31.h, z31.h  // 01100100-00111111-00100111-11111111
+// CHECK-INST: bfclamp z31.h, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x27,0x3f,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 643f27ff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfmax-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfmax z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfmax z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmax z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmax z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfmax z23.h, p1/z, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmax z23.h, p1/z, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmax z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmax z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfmax.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx  z23.h, p3/m, z31.h
+bfmax   z23.h, p3/m, z23.h, z13.h  // 01100101-00000110-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfmax   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x06,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65068db7 <unknown>
+
+movprfx z23, z31
+bfmax   z23.h, p3/m, z23.h, z13.h  // 01100101-00000110-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmax   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x06,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65068db7 <unknown>
+
+bfmax   z0.h, p0/m, z0.h, z0.h  // 01100101-00000110-10000000-00000000
+// CHECK-INST: bfmax   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x06,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65068000 <unknown>
+
+bfmax   z21.h, p5/m, z21.h, z10.h  // 01100101-00000110-10010101-01010101
+// CHECK-INST: bfmax   z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x06,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65069555 <unknown>
+
+bfmax   z23.h, p3/m, z23.h, z13.h  // 01100101-00000110-10001101-10110111
+// CHECK-INST: bfmax   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x06,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65068db7 <unknown>
+
+bfmax   z31.h, p7/m, z31.h, z31.h  // 01100101-00000110-10011111-11111111
+// CHECK-INST: bfmax   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x06,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65069fff <unknown>

llvm/test/MC/AArch64/SVE2p1/bfmaxnm-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfmaxnm z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfmaxnm z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmaxnm z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmaxnm z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfmaxnm z23.h, p1/z, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmaxnm z23.h, p1/z, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmaxnm z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmaxnm z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfmaxnm.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx  z23.h, p3/m, z31.h
+bfmaxnm z23.h, p3/m, z23.h, z13.h  // 01100101-00000100-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfmaxnm z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x04,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65048db7 <unknown>
+
+movprfx z23, z31
+bfmaxnm z23.h, p3/m, z23.h, z13.h  // 01100101-00000100-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmaxnm z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x04,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65048db7 <unknown>
+
+bfmaxnm z0.h, p0/m, z0.h, z0.h  // 01100101-00000100-10000000-00000000
+// CHECK-INST: bfmaxnm z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x04,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65048000 <unknown>
+
+bfmaxnm z21.h, p5/m, z21.h, z10.h  // 01100101-00000100-10010101-01010101
+// CHECK-INST: bfmaxnm z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x04,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65049555 <unknown>
+
+bfmaxnm z23.h, p3/m, z23.h, z13.h  // 01100101-00000100-10001101-10110111
+// CHECK-INST: bfmaxnm z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x04,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65048db7 <unknown>
+
+bfmaxnm z31.h, p7/m, z31.h, z31.h  // 01100101-00000100-10011111-11111111
+// CHECK-INST: bfmaxnm z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x04,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65049fff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfmin-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfmin z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfmin z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmin z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmin z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfmin z23.h, p1/z, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmin z23.h, p1/z, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmin z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmin z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfmin.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx  z23.h, p3/m, z31.h
+bfmin   z23.h, p3/m, z23.h, z13.h  // 01100101-00000111-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfmin   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x07,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65078db7 <unknown>
+
+movprfx z23, z31
+bfmin   z23.h, p3/m, z23.h, z13.h  // 01100101-00000111-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmin   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x07,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65078db7 <unknown>
+
+bfmin   z0.h, p0/m, z0.h, z0.h  // 01100101-00000111-10000000-00000000
+// CHECK-INST: bfmin   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x07,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65078000 <unknown>
+
+bfmin   z21.h, p5/m, z21.h, z10.h  // 01100101-00000111-10010101-01010101
+// CHECK-INST: bfmin   z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x07,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65079555 <unknown>
+
+bfmin   z23.h, p3/m, z23.h, z13.h  // 01100101-00000111-10001101-10110111
+// CHECK-INST: bfmin   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x07,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65078db7 <unknown>
+
+bfmin   z31.h, p7/m, z31.h, z31.h  // 01100101-00000111-10011111-11111111
+// CHECK-INST: bfmin   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x07,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65079fff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfminnm-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfminnm z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfminnm z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfminnm z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfminnm z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfminnm z23.h, p1/z, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfminnm z23.h, p1/z, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfminnm z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfminnm z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfminnm.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx  z23.h, p3/m, z31.h
+bfminnm z23.h, p3/m, z23.h, z13.h  // 01100101-00000101-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfminnm z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x05,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65058db7 <unknown>
+
+movprfx z23, z31
+bfminnm z23.h, p3/m, z23.h, z13.h  // 01100101-00000101-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfminnm z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x05,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65058db7 <unknown>
+
+bfminnm z0.h, p0/m, z0.h, z0.h  // 01100101-00000101-10000000-00000000
+// CHECK-INST: bfminnm z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x05,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65058000 <unknown>
+
+bfminnm z21.h, p5/m, z21.h, z10.h  // 01100101-00000101-10010101-01010101
+// CHECK-INST: bfminnm z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x05,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65059555 <unknown>
+
+bfminnm z23.h, p3/m, z23.h, z13.h  // 01100101-00000101-10001101-10110111
+// CHECK-INST: bfminnm z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x05,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65058db7 <unknown>
+
+bfminnm z31.h, p7/m, z31.h, z31.h  // 01100101-00000101-10011111-11111111
+// CHECK-INST: bfminnm z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x05,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65059fff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfmla-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid vector lane index
+
+bfmla z0.h, z0.h, z0.h[8]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: vector lane must be an integer in range [0, 7].
+// CHECK-NEXT: bfmla z0.h, z0.h, z0.h[8]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmla z0.h, z0.h, z0.h[-1]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: vector lane must be an integer in range [0, 7].
+// CHECK-NEXT: bfmla z0.h, z0.h, z0.h[-1]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmla z0.h, z0.h, z8.h[2]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: Invalid restricted vector register, expected z0.h..z7.h
+// CHECK-NEXT: bfmla z0.h, z0.h, z8.h[2]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfmla z0.h, z0.s, z0.s[0]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmla z0.h, z0.s, z0.s[0]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmla z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmla z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid use of movprfx
+
+movprfx z23.h, p1/m, z31.h
+bfmla z23.h, z12.h, z0.h[0]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: instruction is unpredictable when following a predicated movprfx, suggest using unpredicated movprfx
+// CHECK-NEXT: bfmla z23.h, z12.h, z0.h[0]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfmla.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx z23, z31
+bfmla   z23.h, z13.h, z0.h[5]  // 01100100-01101000-00001001-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmla   z23.h, z13.h, z0.h[5]
+// CHECK-ENCODING: [0xb7,0x09,0x68,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 646809b7 <unknown>
+
+bfmla   z0.h, z0.h, z0.h[0]  // 01100100-00100000-00001000-00000000
+// CHECK-INST: bfmla   z0.h, z0.h, z0.h[0]
+// CHECK-ENCODING: [0x00,0x08,0x20,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64200800 <unknown>
+
+bfmla   z21.h, z10.h, z5.h[6]  // 01100100-01110101-00001001-01010101
+// CHECK-INST: bfmla   z21.h, z10.h, z5.h[6]
+// CHECK-ENCODING: [0x55,0x09,0x75,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64750955 <unknown>
+
+bfmla   z23.h, z13.h, z0.h[5]  // 01100100-01101000-00001001-10110111
+// CHECK-INST: bfmla   z23.h, z13.h, z0.h[5]
+// CHECK-ENCODING: [0xb7,0x09,0x68,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 646809b7 <unknown>
+
+bfmla   z31.h, z31.h, z7.h[7]  // 01100100-01111111-00001011-11111111
+// CHECK-INST: bfmla   z31.h, z31.h, z7.h[7]
+// CHECK-ENCODING: [0xff,0x0b,0x7f,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 647f0bff <unknown>
+
+
+movprfx  z23.h, p3/m, z31.h
+bfmla   z23.h, p3/m, z13.h, z8.h  // 01100101-00101000-00001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfmla   z23.h, p3/m, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x0d,0x28,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65280db7 <unknown>
+
+movprfx z23, z31
+bfmla   z23.h, p3/m, z13.h, z8.h  // 01100101-00101000-00001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmla   z23.h, p3/m, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x0d,0x28,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65280db7 <unknown>
+
+bfmla   z0.h, p0/m, z0.h, z0.h  // 01100101-00100000-00000000-00000000
+// CHECK-INST: bfmla   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x00,0x20,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65200000 <unknown>
+
+bfmla   z21.h, p5/m, z10.h, z21.h  // 01100101-00110101-00010101-01010101
+// CHECK-INST: bfmla   z21.h, p5/m, z10.h, z21.h
+// CHECK-ENCODING: [0x55,0x15,0x35,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65351555 <unknown>
+
+bfmla   z23.h, p3/m, z13.h, z8.h  // 01100101-00101000-00001101-10110111
+// CHECK-INST: bfmla   z23.h, p3/m, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x0d,0x28,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65280db7 <unknown>
+
+bfmla   z31.h, p7/m, z31.h, z31.h  // 01100101-00111111-00011111-11111111
+// CHECK-INST: bfmla   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x1f,0x3f,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 653f1fff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfmls-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid vector lane index
+
+bfmls z0.h, z0.h, z0.h[8]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: vector lane must be an integer in range [0, 7].
+// CHECK-NEXT: bfmls z0.h, z0.h, z0.h[8]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmls z0.h, z0.h, z0.h[-1]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: vector lane must be an integer in range [0, 7].
+// CHECK-NEXT: bfmls z0.h, z0.h, z0.h[-1]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmls z0.h, z0.h, z8.h[2]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: Invalid restricted vector register, expected z0.h..z7.h
+// CHECK-NEXT: bfmls z0.h, z0.h, z8.h[2]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfmls z0.h, z0.s, z0.s[0]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmls z0.h, z0.s, z0.s[0]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmls z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmls z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid use of movprfx
+
+movprfx z23.h, p1/m, z31.h
+bfmls z23.h, z12.h, z0.h[0]
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: instruction is unpredictable when following a predicated movprfx, suggest using unpredicated movprfx
+// CHECK-NEXT: bfmls z23.h, z12.h, z0.h[0]
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfmls.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+
+movprfx z23, z31
+bfmls   z23.h, z13.h, z0.h[5]  // 01100100-01101000-00001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmls   z23.h, z13.h, z0.h[5]
+// CHECK-ENCODING: [0xb7,0x0d,0x68,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64680db7 <unknown>
+
+bfmls   z0.h, z0.h, z0.h[0]  // 01100100-00100000-00001100-00000000
+// CHECK-INST: bfmls   z0.h, z0.h, z0.h[0]
+// CHECK-ENCODING: [0x00,0x0c,0x20,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64200c00 <unknown>
+
+bfmls   z21.h, z10.h, z5.h[6]  // 01100100-01110101-00001101-01010101
+// CHECK-INST: bfmls   z21.h, z10.h, z5.h[6]
+// CHECK-ENCODING: [0x55,0x0d,0x75,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64750d55 <unknown>
+
+bfmls   z23.h, z13.h, z0.h[5]  // 01100100-01101000-00001101-10110111
+// CHECK-INST: bfmls   z23.h, z13.h, z0.h[5]
+// CHECK-ENCODING: [0xb7,0x0d,0x68,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64680db7 <unknown>
+
+bfmls   z31.h, z31.h, z7.h[7]  // 01100100-01111111-00001111-11111111
+// CHECK-INST: bfmls   z31.h, z31.h, z7.h[7]
+// CHECK-ENCODING: [0xff,0x0f,0x7f,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 647f0fff <unknown>
+
+
+movprfx  z23.h, p3/m, z31.h
+bfmls   z23.h, p3/m, z13.h, z8.h  // 01100101-00101000-00101101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfmls   z23.h, p3/m, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x2d,0x28,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65282db7 <unknown>
+
+movprfx z23, z31
+bfmls   z23.h, p3/m, z13.h, z8.h  // 01100101-00101000-00101101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmls   z23.h, p3/m, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x2d,0x28,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65282db7 <unknown>
+
+bfmls   z0.h, p0/m, z0.h, z0.h  // 01100101-00100000-00100000-00000000
+// CHECK-INST: bfmls   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x20,0x20,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65202000 <unknown>
+
+bfmls   z21.h, p5/m, z10.h, z21.h  // 01100101-00110101-00110101-01010101
+// CHECK-INST: bfmls   z21.h, p5/m, z10.h, z21.h
+// CHECK-ENCODING: [0x55,0x35,0x35,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65353555 <unknown>
+
+bfmls   z23.h, p3/m, z13.h, z8.h  // 01100101-00101000-00101101-10110111
+// CHECK-INST: bfmls   z23.h, p3/m, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x2d,0x28,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65282db7 <unknown>
+
+bfmls   z31.h, p7/m, z31.h, z31.h  // 01100101-00111111-00111111-11111111
+// CHECK-INST: bfmls   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x3f,0x3f,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 653f3fff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfmul-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfmul z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfmul z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmul z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfmul z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfmul z23.h, p1/m, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmul z23.h, p1/m, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfmul z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfmul z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid use of movprfx
+
+movprfx z23.h, p1/m, z31.h
+bfmul z23.h, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: instruction is unpredictable when following a movprfx, suggest replacing movprfx with mov
+// CHECK-NEXT: bfmul z23.h, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfmul.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+bfmul   z0.h, z0.h, z0.h[0]  // 01100100-00100000-00101000-00000000
+// CHECK-INST: bfmul   z0.h, z0.h, z0.h[0]
+// CHECK-ENCODING: [0x00,0x28,0x20,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64202800 <unknown>
+
+bfmul   z21.h, z10.h, z5.h[6]  // 01100100-01110101-00101001-01010101
+// CHECK-INST: bfmul   z21.h, z10.h, z5.h[6]
+// CHECK-ENCODING: [0x55,0x29,0x75,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 64752955 <unknown>
+
+bfmul   z23.h, z13.h, z0.h[5]  // 01100100-01101000-00101001-10110111
+// CHECK-INST: bfmul   z23.h, z13.h, z0.h[5]
+// CHECK-ENCODING: [0xb7,0x29,0x68,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 646829b7 <unknown>
+
+bfmul   z31.h, z31.h, z7.h[7]  // 01100100-01111111-00101011-11111111
+// CHECK-INST: bfmul   z31.h, z31.h, z7.h[7]
+// CHECK-ENCODING: [0xff,0x2b,0x7f,0x64]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 647f2bff <unknown>
+
+movprfx  z23.h, p3/m, z31.h
+bfmul   z23.h, p3/m, z23.h, z13.h  // 01100101-00000010-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfmul   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x02,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65028db7 <unknown>
+
+movprfx z23, z31
+bfmul   z23.h, p3/m, z23.h, z13.h  // 01100101-00000010-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfmul   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x02,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65028db7 <unknown>
+
+bfmul   z0.h, p0/m, z0.h, z0.h  // 01100101-00000010-10000000-00000000
+// CHECK-INST: bfmul   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x02,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65028000 <unknown>
+
+bfmul   z21.h, p5/m, z21.h, z10.h  // 01100101-00000010-10010101-01010101
+// CHECK-INST: bfmul   z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x02,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65029555 <unknown>
+
+bfmul   z23.h, p3/m, z23.h, z13.h  // 01100101-00000010-10001101-10110111
+// CHECK-INST: bfmul   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x02,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65028db7 <unknown>
+
+bfmul   z31.h, p7/m, z31.h, z31.h  // 01100101-00000010-10011111-11111111
+// CHECK-INST: bfmul   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x02,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65029fff <unknown>
+
+bfmul   z0.h, z0.h, z0.h  // 01100101-00000000-00001000-00000000
+// CHECK-INST: bfmul   z0.h, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x08,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65000800 <unknown>
+
+bfmul   z21.h, z10.h, z21.h  // 01100101-00010101-00001001-01010101
+// CHECK-INST: bfmul   z21.h, z10.h, z21.h
+// CHECK-ENCODING: [0x55,0x09,0x15,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65150955 <unknown>
+
+bfmul   z23.h, z13.h, z8.h  // 01100101-00001000-00001001-10110111
+// CHECK-INST: bfmul   z23.h, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x09,0x08,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 650809b7 <unknown>
+
+bfmul   z31.h, z31.h, z31.h  // 01100101-00011111-00001011-11111111
+// CHECK-INST: bfmul   z31.h, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x0b,0x1f,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 651f0bff <unknown>
+

llvm/test/MC/AArch64/SVE2p1/bfsub-diagnostics.s

+// RUN: not llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 2>&1 < %s | FileCheck %s
+
+// --------------------------------------------------------------------------//
+// Invalid predicate register
+
+bfsub z23.h, p8/m, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid restricted predicate register, expected p0..p7 (without element suffix)
+// CHECK-NEXT: bfsub z23.h, p8/m, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfsub z23.h, p1/z, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid operand for instruction
+// CHECK-NEXT: bfsub z23.h, p1/z, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid vector suffix
+
+bfsub z23.h, p1/m, z23.s, z13.s
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfsub z23.h, p1/m, z23.s, z13.s
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+bfsub z23.s, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: invalid element width
+// CHECK-NEXT: bfsub z23.s, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:
+
+// --------------------------------------------------------------------------//
+// Invalid use of movprfx
+
+movprfx z23.h, p1/m, z31.h
+bfsub z23.h, z23.h, z13.h
+// CHECK: [[@LINE-1]]:{{[0-9]+}}: error: instruction is unpredictable when following a movprfx, suggest replacing movprfx with mov
+// CHECK-NEXT: bfsub z23.h, z23.h, z13.h
+// CHECK-NOT: [[@LINE-1]]:{{[0-9]+}}:

llvm/test/MC/AArch64/SVE2p1/bfsub.s

+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+// RUN: not llvm-mc -triple=aarch64 -show-encoding < %s 2>&1 \
+// RUN:        | FileCheck %s --check-prefix=CHECK-ERROR
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --no-print-imm-hex --mattr=+sve2p1,+b16b16 - | FileCheck %s --check-prefix=CHECK-INST
+// RUN: llvm-mc -triple=aarch64 -filetype=obj -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | llvm-objdump -d --mattr=-b16b16 - | FileCheck %s --check-prefix=CHECK-UNKNOWN
+// RUN: llvm-mc -triple=aarch64 -show-encoding -mattr=+sve2p1,+b16b16 < %s \
+// RUN:        | sed '/.text/d' | sed 's/.*encoding: //g' \
+// RUN:        | llvm-mc -triple=aarch64 -mattr=+sve2p1,+b16b16 -disassemble -show-encoding \
+// RUN:        | FileCheck %s --check-prefixes=CHECK-ENCODING,CHECK-INST
+
+movprfx  z23.h, p3/m, z31.h
+bfsub   z23.h, p3/m, z23.h, z13.h  // 01100101-00000001-10001101-10110111
+// CHECK-INST:  movprfx  z23.h, p3/m, z31.h
+// CHECK-INST: bfsub   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x01,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65018db7 <unknown>
+
+movprfx z23, z31
+bfsub   z23.h, p3/m, z23.h, z13.h  // 01100101-00000001-10001101-10110111
+// CHECK-INST:  movprfx z23, z31
+// CHECK-INST: bfsub   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x01,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65018db7 <unknown>
+
+bfsub   z0.h, p0/m, z0.h, z0.h  // 01100101-00000001-10000000-00000000
+// CHECK-INST: bfsub   z0.h, p0/m, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x80,0x01,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65018000 <unknown>
+
+bfsub   z21.h, p5/m, z21.h, z10.h  // 01100101-00000001-10010101-01010101
+// CHECK-INST: bfsub   z21.h, p5/m, z21.h, z10.h
+// CHECK-ENCODING: [0x55,0x95,0x01,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65019555 <unknown>
+
+bfsub   z23.h, p3/m, z23.h, z13.h  // 01100101-00000001-10001101-10110111
+// CHECK-INST: bfsub   z23.h, p3/m, z23.h, z13.h
+// CHECK-ENCODING: [0xb7,0x8d,0x01,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65018db7 <unknown>
+
+bfsub   z31.h, p7/m, z31.h, z31.h  // 01100101-00000001-10011111-11111111
+// CHECK-INST: bfsub   z31.h, p7/m, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x9f,0x01,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65019fff <unknown>
+
+bfsub   z0.h, z0.h, z0.h  // 01100101-00000000-00000100-00000000
+// CHECK-INST: bfsub   z0.h, z0.h, z0.h
+// CHECK-ENCODING: [0x00,0x04,0x00,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65000400 <unknown>
+
+bfsub   z21.h, z10.h, z21.h  // 01100101-00010101-00000101-01010101
+// CHECK-INST: bfsub   z21.h, z10.h, z21.h
+// CHECK-ENCODING: [0x55,0x05,0x15,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 65150555 <unknown>
+
+bfsub   z23.h, z13.h, z8.h  // 01100101-00001000-00000101-10110111
+// CHECK-INST: bfsub   z23.h, z13.h, z8.h
+// CHECK-ENCODING: [0xb7,0x05,0x08,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 650805b7 <unknown>
+
+bfsub   z31.h, z31.h, z31.h  // 01100101-00011111-00000111-11111111
+// CHECK-INST: bfsub   z31.h, z31.h, z31.h
+// CHECK-ENCODING: [0xff,0x07,0x1f,0x65]
+// CHECK-ERROR: instruction requires: b16b16 sme2p1 or sve2p1
+// CHECK-UNKNOWN: 651f07ff <unknown>

llvm/unittests/Support/TargetParserTest.cpp

std::vector<uint64_t> Extensions = {
       AArch64::AEK_SVE2SM4, AArch64::AEK_SVE2SHA3, AArch64::AEK_SVE2BITPERM,
       AArch64::AEK_RCPC,    AArch64::AEK_RAND,     AArch64::AEK_MTE,
       AArch64::AEK_SSBS,    AArch64::AEK_SB,       AArch64::AEK_PREDRES,
       AArch64::AEK_BF16,    AArch64::AEK_I8MM,     AArch64::AEK_F32MM,
       AArch64::AEK_F64MM,   AArch64::AEK_TME,      AArch64::AEK_LS64,
       AArch64::AEK_BRBE,    AArch64::AEK_PAUTH,    AArch64::AEK_FLAGM,
       AArch64::AEK_SME,     AArch64::AEK_SMEF64F64, AArch64::AEK_SMEI16I64,
       AArch64::AEK_SME2,    AArch64::AEK_HBC,      AArch64::AEK_MOPS,
-      AArch64::AEK_PERFMON, AArch64::AEK_SVE2p1,   AArch64::AEK_SME2p1};
+      AArch64::AEK_PERFMON, AArch64::AEK_SVE2p1,   AArch64::AEK_SME2p1,
+      AArch64::AEK_B16B16};
 
   std::vector<StringRef> Features;
 
   uint64_t ExtVal = 0;
   for (auto Ext : Extensions)
     ExtVal |= Ext;
 
   // INVALID and NONE have no feature names.
TEST(TargetParserTest, AArch64ExtensionFeatures) {
   EXPECT_TRUE(llvm::is_contained(Features, "+ras"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-aes"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-sm4"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-sha3"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2-bitperm"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sve2p1"));
+  EXPECT_TRUE(llvm::is_contained(Features, "+b16b16"));
   EXPECT_TRUE(llvm::is_contained(Features, "+rcpc"));
   EXPECT_TRUE(llvm::is_contained(Features, "+rand"));
   EXPECT_TRUE(llvm::is_contained(Features, "+mte"));
   EXPECT_TRUE(llvm::is_contained(Features, "+ssbs"));
   EXPECT_TRUE(llvm::is_contained(Features, "+sb"));
   EXPECT_TRUE(llvm::is_contained(Features, "+predres"));
   EXPECT_TRUE(llvm::is_contained(Features, "+bf16"));
   EXPECT_TRUE(llvm::is_contained(Features, "+i8mm"));
const char *ArchExt[][4] = {
       {"lse", "nolse", "+lse", "-lse"},
       {"rdm", "nordm", "+rdm", "-rdm"},
       {"sve", "nosve", "+sve", "-sve"},
       {"sve2", "nosve2", "+sve2", "-sve2"},
       {"sve2-aes", "nosve2-aes", "+sve2-aes", "-sve2-aes"},
       {"sve2-sm4", "nosve2-sm4", "+sve2-sm4", "-sve2-sm4"},
       {"sve2-sha3", "nosve2-sha3", "+sve2-sha3", "-sve2-sha3"},
       {"sve2p1", "nosve2p1", "+sve2p1", "-sve2p1"},
+      {"b16b16", "nob16b16", "+b16b16", "-b16b16"},
       {"sve2-bitperm", "nosve2-bitperm", "+sve2-bitperm", "-sve2-bitperm"},
       {"dotprod", "nodotprod", "+dotprod", "-dotprod"},
       {"rcpc", "norcpc", "+rcpc", "-rcpc"},
       {"rng", "norng", "+rand", "-rand"},
       {"memtag", "nomemtag", "+mte", "-mte"},
       {"tme", "notme", "+tme", "-tme"},
       {"pauth", "nopauth", "+pauth", "-pauth"},
       {"ssbs", "nossbs", "+ssbs", "-ssbs"},