[AArch64] Adding "armv8.8-a" memcpy/memset support.

[tmatheson]

Summary

This change introduces the Arm 8.8-A instructions for FEAT_MOPS. Documentation for these instructions can be found in the A64 ISA documentation.

There are a lot of variants so I won't link to all of the documentation individually here. There are four basic operations with the following prefixes:

• CPYP*: Memory copy
• CPYF*: Memory copy, forward-only
• SET*: Memory Set
• SETG*: Memory Set with tag setting, which requires FEAT_MTE in addition to FEAT_MOPS

For each of these operations there are three instructions differentiated by P/M/E for preconditioning/main/end. For example, SETGP, SETGM, SETGE.

For each set of three instructions there are also variants for privileged/unprivileged and temporal/non-temporal e.g. CPYPTN, CPYMTN, CPYETN

For the copy instructions, you can separately specify the read and write translations (so that kernels can safely use these instructions in syscall handlers, to memcpy between the calling process's user-space memory map and the kernel's own privileged one).

Some of these instructions write back to multiple registers to indicate progress in the event of interrupts. Contrary to the previous description of these instructions, they are not required to be placed in a loop.

Patch by Simon Tatham, Victor Campos, Tomas Matheson and Sam Elliott

[nickdesaulniers]

The latest publicly accessible Arm ARM doesn't yet mention the v8.8 extensions, IIUC.
https://developer.arm.com/documentation/ddi0487/gb/
Is there perhaps a draft of the next revision I can refer to to review this patch?

[tmatheson]

I've updated the description with documentation.

[dmgreen]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

11455

Should this be hasNoSchedulingInfo? Can we remove this or include something very basic?

Should something here be marking them MayLoad and MayStore? They may already be hasSideEffects, as nothing marks them otherwise and they do not have codegen patterns, which is more strict. But it's probably good to be explicit.

[tmatheson]

Add mayLoad/mayStore/hasSideEffects

[tmatheson]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

11455

Should this be hasNoSchedulingInfo? Can we remove this or include something very basic?

I'm not sure why this is set, @simon_tatham might remember. My guess would be that the alternative is adding Sched<[]> but that doesn't seem to work on the multiclass - CodeGenSchedModels::checkCompleteness() does not recognise the final defs as having the Sched subclass.

[simon_tatham]

llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp

4975

Surely this ought to say "invalid CPY instruction", and similarly the two below it?

[simon_tatham]

llvm/lib/Target/AArch64/AArch64InstrFormats.td

11455

No, sorry – I think when I wrote the internal version of this patch I just did whatever would most quickly get Tablegen to stop complaining about incomplete scheduling models. If @dmgreen thinks it's wrong, I'd trust him over me :-)

[tmatheson]

SET->CPY

[tmatheson]

Use Sched<[]> rather than hasNoSchedulingInfo. This requried updating the regexes used by various scheduling models to apply InstRW to CPY* instructions so that they do not capture the new MOPS instructions.

[dmgreen]

Use Sched<[]> rather than hasNoSchedulingInfo. This requried updating the regexes used by various scheduling models to apply InstRW to CPY* instructions so that they do not capture the new MOPS instructions.

Yeah, I got confused as to why this worked differently to the other new instructions like ST64B. Enough to go and have a look at what was going on, and came to the same conclusion.

I think it's worth renaming the old "CPY" instruction just for clarity. It is not anything to do with any CPY instructions after all. I put up D116655 for that.

Otherwise that and the other point here, this patch LGTM.

llvm/lib/Target/AArch64/AArch64InstrFormats.td

11456

Sorry, I should have been more clear. I'm not sure this actually needs hasSideEffects. I was just trying to explain that it would already have it set (implicitly), so missing mayLoad/mayStore would not technically be a problem, as it already had sideeffects. With mayLoad and mayStore, I don't believe there are any other side effects of these instructions. They just update registers and load/store memory.

[dmgreen]

Oh. I just thought that these instructions will unlikely be scheduled if we are generating then from pseudos. Perhaps the scheduling info doesn't matter either way.

Should they be setting or using NZCV though?

[tmatheson]

Should they be setting or using NZCV though?

Not as far as I am aware.

I'll remove the hasSideEffects before landing.

Thanks everyone for the comments.

llvm/lib/Target/AArch64/AArch64.td

def FeatureHCX : SubtargetFeature<
     "hcx", "HasHCX", "true", "Enable Armv8.7-A HCRX_EL2 system register">;
 
 def FeatureLS64 : SubtargetFeature<"ls64", "HasLS64",
     "true", "Enable Armv8.7-A LD64B/ST64B Accelerator Extension">;
 
 def FeatureHBC : SubtargetFeature<"hbc", "HasHBC",
     "true", "Enable Armv8.8-A Hinted Conditional Branches Extension">;
 
+def FeatureMOPS : SubtargetFeature<"mops", "HasMOPS",
+    "true", "Enable Armv8.8-A memcpy and memset acceleration instructions">;
+
 def FeatureBRBE : SubtargetFeature<"brbe", "HasBRBE",
     "true", "Enable Branch Record Buffer Extension">;
 
 def FeatureSPE_EEF : SubtargetFeature<"spe-eef", "HasSPE_EEF",
     "true", "Enable extra register in the Statistical Profiling Extension">;
 
 def FeatureFineGrainedTraps : SubtargetFeature<"fgt", "HasFineGrainedTraps",
     "true", "Enable fine grained virtualization traps extension">;
def HasV8_6aOps : SubtargetFeature<
    FeatureEnhancedCounterVirtualization, FeatureMatMulInt8]>;
 
 def HasV8_7aOps : SubtargetFeature<
   "v8.7a", "HasV8_7aOps", "true", "Support ARM v8.7a instructions",
   [HasV8_6aOps, FeatureXS, FeatureWFxT, FeatureHCX]>;
 
 def HasV8_8aOps : SubtargetFeature<
   "v8.8a", "HasV8_8aOps", "true", "Support ARM v8.8a instructions",
-  [HasV8_7aOps, FeatureHBC]>;
+  [HasV8_7aOps, FeatureHBC, FeatureMOPS]>;
 
 def HasV9_0aOps : SubtargetFeature<
   "v9a", "HasV9_0aOps", "true", "Support ARM v9a instructions",
   [HasV8_5aOps, FeatureSVE2]>;
 
 def HasV9_1aOps : SubtargetFeature<
   "v9.1a", "HasV9_1aOps", "true", "Support ARM v9.1a instructions",
   [HasV8_6aOps, HasV9_0aOps]>;

llvm/lib/Target/AArch64/AArch64InstrFormats.td

 
 class Store64BV<bits<3> opc, string asm_inst, list<dag> pat = []>
     : LoadStore64B_base<opc, asm_inst, "\t$Rs, $Rt, [$Rn]",
                        (ins GPR64x8:$Rt, GPR64sp:$Rn), (outs GPR64:$Rs), pat> {
   bits<5> Rs;
   let Inst{20-16} = Rs;
 }
 
+class MOPSMemoryCopyMoveBase<bit isMove, bits<2> opcode, bits<2> op1,
+                             bits<2> op2, string asm>
+  : I<(outs GPR64common:$Rd_wb, GPR64common:$Rs_wb, GPR64:$Rn_wb),
+      (ins GPR64common:$Rd, GPR64common:$Rs, GPR64:$Rn),
+      asm, "\t[$Rd]!, [$Rs]!, $Rn!",
+      "$Rd = $Rd_wb,$Rs = $Rs_wb,$Rn = $Rn_wb", []>,
+    Sched<[]> {
+  bits<5> Rd;
+  bits<5> Rs;
+  bits<5> Rn;
+  let Inst{31-27} = 0b00011;
+  let Inst{26} = isMove;
+  let Inst{25-24} = 0b01;
+  let Inst{23-22} = opcode;
+  let Inst{21} = 0b0;
+  let Inst{20-16} = Rs;
+  let Inst{15-14} = op2;
+  let Inst{13-12} = op1;
+  let Inst{11-10} = 0b01;
+  let Inst{9-5} = Rn;
+  let Inst{4-0} = Rd;
+
+  let DecoderMethod = "DecodeCPYMemOpInstruction";
+  let mayLoad = 1;
+  let mayStore = 1;
+}
+
+class MOPSMemoryCopy<bits<2> opcode, bits<2> op1, bits<2> op2, string asm>
+  : MOPSMemoryCopyMoveBase<0, opcode, op1, op2, asm>;
+
+class MOPSMemoryMove<bits<2> opcode, bits<2> op1, bits<2> op2, string asm>
+  : MOPSMemoryCopyMoveBase<1, opcode, op1, op2, asm>;
+
+class MOPSMemorySetBase<bit isTagging, bits<2> opcode, bit op1, bit op2,
+                        string asm>
+  : I<(outs GPR64common:$Rd_wb, GPR64:$Rn_wb),
+      (ins GPR64common:$Rd, GPR64:$Rn, GPR64:$Rm),
+      asm, "\t[$Rd]!, $Rn!, $Rm",
+      "$Rd = $Rd_wb,$Rn = $Rn_wb", []>,
+    Sched<[]> {
+  bits<5> Rd;
+  bits<5> Rn;
+  bits<5> Rm;
+  let Inst{31-27} = 0b00011;
+  let Inst{26} = isTagging;
+  let Inst{25-21} = 0b01110;
+  let Inst{20-16} = Rm;
+  let Inst{15-14} = opcode;
+  let Inst{13} = op2;
+  let Inst{12} = op1;
+  let Inst{11-10} = 0b01;
+  let Inst{9-5} = Rn;
+  let Inst{4-0} = Rd;
+
+  let DecoderMethod = "DecodeSETMemOpInstruction";
+  let mayLoad = 0;
+  let mayStore = 1;
+}
+
+class MOPSMemorySet<bits<2> opcode, bit op1, bit op2, string asm>
+  : MOPSMemorySetBase<0, opcode, op1, op2, asm>;
+
+class MOPSMemorySetTagging<bits<2> opcode, bit op1, bit op2, string asm>
+  : MOPSMemorySetBase<1, opcode, op1, op2, asm>;
+
+multiclass MOPSMemoryCopyInsns<bits<2> opcode, string asm> {
+  def ""   : MOPSMemoryCopy<opcode, 0b00, 0b00, asm>;
+  def WN   : MOPSMemoryCopy<opcode, 0b00, 0b01, asm # "wn">;
+  def RN   : MOPSMemoryCopy<opcode, 0b00, 0b10, asm # "rn">;
+  def N    : MOPSMemoryCopy<opcode, 0b00, 0b11, asm # "n">;
+  def WT   : MOPSMemoryCopy<opcode, 0b01, 0b00, asm # "wt">;
+  def WTWN : MOPSMemoryCopy<opcode, 0b01, 0b01, asm # "wtwn">;
+  def WTRN : MOPSMemoryCopy<opcode, 0b01, 0b10, asm # "wtrn">;
+  def WTN  : MOPSMemoryCopy<opcode, 0b01, 0b11, asm # "wtn">;
+  def RT   : MOPSMemoryCopy<opcode, 0b10, 0b00, asm # "rt">;
+  def RTWN : MOPSMemoryCopy<opcode, 0b10, 0b01, asm # "rtwn">;
+  def RTRN : MOPSMemoryCopy<opcode, 0b10, 0b10, asm # "rtrn">;
+  def RTN  : MOPSMemoryCopy<opcode, 0b10, 0b11, asm # "rtn">;
+  def T    : MOPSMemoryCopy<opcode, 0b11, 0b00, asm # "t">;
+  def TWN  : MOPSMemoryCopy<opcode, 0b11, 0b01, asm # "twn">;
+  def TRN  : MOPSMemoryCopy<opcode, 0b11, 0b10, asm # "trn">;
+  def TN   : MOPSMemoryCopy<opcode, 0b11, 0b11, asm # "tn">;
+}
+
+multiclass MOPSMemoryMoveInsns<bits<2> opcode, string asm> {
+  def ""   : MOPSMemoryMove<opcode, 0b00, 0b00, asm>;
+  def WN   : MOPSMemoryMove<opcode, 0b00, 0b01, asm # "wn">;
+  def RN   : MOPSMemoryMove<opcode, 0b00, 0b10, asm # "rn">;
+  def N    : MOPSMemoryMove<opcode, 0b00, 0b11, asm # "n">;
+  def WT   : MOPSMemoryMove<opcode, 0b01, 0b00, asm # "wt">;
+  def WTWN : MOPSMemoryMove<opcode, 0b01, 0b01, asm # "wtwn">;
+  def WTRN : MOPSMemoryMove<opcode, 0b01, 0b10, asm # "wtrn">;
+  def WTN  : MOPSMemoryMove<opcode, 0b01, 0b11, asm # "wtn">;
+  def RT   : MOPSMemoryMove<opcode, 0b10, 0b00, asm # "rt">;
+  def RTWN : MOPSMemoryMove<opcode, 0b10, 0b01, asm # "rtwn">;
+  def RTRN : MOPSMemoryMove<opcode, 0b10, 0b10, asm # "rtrn">;
+  def RTN  : MOPSMemoryMove<opcode, 0b10, 0b11, asm # "rtn">;
+  def T    : MOPSMemoryMove<opcode, 0b11, 0b00, asm # "t">;
+  def TWN  : MOPSMemoryMove<opcode, 0b11, 0b01, asm # "twn">;
+  def TRN  : MOPSMemoryMove<opcode, 0b11, 0b10, asm # "trn">;
+  def TN   : MOPSMemoryMove<opcode, 0b11, 0b11, asm # "tn">;
+}
+
+multiclass MOPSMemorySetInsns<bits<2> opcode, string asm> {
+  def "" : MOPSMemorySet<opcode, 0, 0, asm>;
+  def T  : MOPSMemorySet<opcode, 1, 0, asm # "t">;
+  def N  : MOPSMemorySet<opcode, 0, 1, asm # "n">;
+  def TN : MOPSMemorySet<opcode, 1, 1, asm # "tn">;
+}
+
+multiclass MOPSMemorySetTaggingInsns<bits<2> opcode, string asm> {
+  def "" : MOPSMemorySetTagging<opcode, 0, 0, asm>;
+  def T  : MOPSMemorySetTagging<opcode, 1, 0, asm # "t">;
+  def N  : MOPSMemorySetTagging<opcode, 0, 1, asm # "n">;
+  def TN : MOPSMemorySetTagging<opcode, 1, 1, asm # "tn">;
+}
+
 //----------------------------------------------------------------------------
 // Allow the size specifier tokens to be upper case, not just lower.
 def : TokenAlias<".4B", ".4b">;  // Add dot product
 def : TokenAlias<".8B", ".8b">;
 def : TokenAlias<".4H", ".4h">;
 def : TokenAlias<".2S", ".2s">;
 def : TokenAlias<".1D", ".1d">;
 def : TokenAlias<".16B", ".16b">;

llvm/lib/Target/AArch64/AArch64InstrInfo.td

 def HasWFxT          : Predicate<"Subtarget->hasWFxT()">,
                        AssemblerPredicate<(all_of FeatureWFxT), "wfxt">;
 def HasLS64          : Predicate<"Subtarget->hasLS64()">,
                        AssemblerPredicate<(all_of FeatureLS64), "ls64">;
 def HasBRBE          : Predicate<"Subtarget->hasBRBE()">,
                        AssemblerPredicate<(all_of FeatureBRBE), "brbe">;
 def HasSPE_EEF       : Predicate<"Subtarget->hasSPE_EEF()">,
                        AssemblerPredicate<(all_of FeatureSPE_EEF), "spe-eef">;
-def HasHBC            : Predicate<"Subtarget->hasHBC()">,
+def HasHBC           : Predicate<"Subtarget->hasHBC()">,
                        AssemblerPredicate<(all_of FeatureHBC), "hbc">;
+def HasMOPS          : Predicate<"Subtarget->hasMOPS()">,
+                       AssemblerPredicate<(all_of FeatureMOPS), "mops">;
 def IsLE             : Predicate<"Subtarget->isLittleEndian()">;
 def IsBE             : Predicate<"!Subtarget->isLittleEndian()">;
 def IsWindows        : Predicate<"Subtarget->isTargetWindows()">;
 def UseExperimentalZeroingPseudos
     : Predicate<"Subtarget->useExperimentalZeroingPseudos()">;
 def UseAlternateSExtLoadCVTF32
     : Predicate<"Subtarget->useAlternateSExtLoadCVTF32Pattern()">;
 
class ST64BPattern<Intrinsic intrinsic, Instruction instruction>
     : Pat<(intrinsic GPR64sp:$addr, GPR64:$x0, GPR64:$x1, GPR64:$x2, GPR64:$x3, GPR64:$x4, GPR64:$x5, GPR64:$x6, GPR64:$x7),
           (instruction (REG_SEQUENCE GPR64x8Class, $x0, x8sub_0, $x1, x8sub_1, $x2, x8sub_2, $x3, x8sub_3, $x4, x8sub_4, $x5, x8sub_5, $x6, x8sub_6, $x7, x8sub_7), $addr)>;
 
   def : ST64BPattern<int_aarch64_st64b, ST64B>;
   def : ST64BPattern<int_aarch64_st64bv, ST64BV>;
   def : ST64BPattern<int_aarch64_st64bv0, ST64BV0>;
 }
 
+let Predicates = [HasMOPS] in {
+  defm CPYFP : MOPSMemoryCopyInsns<0b00, "cpyfp">;
+  defm CPYFM : MOPSMemoryCopyInsns<0b01, "cpyfm">;
+  defm CPYFE : MOPSMemoryCopyInsns<0b10, "cpyfe">;
+
+  defm CPYP : MOPSMemoryMoveInsns<0b00, "cpyp">;
+  defm CPYM : MOPSMemoryMoveInsns<0b01, "cpym">;
+  defm CPYE : MOPSMemoryMoveInsns<0b10, "cpye">;
+
+  defm SETP : MOPSMemorySetInsns<0b00, "setp">;
+  defm SETM : MOPSMemorySetInsns<0b01, "setm">;
+  defm SETE : MOPSMemorySetInsns<0b10, "sete">;
+}
+let Predicates = [HasMOPS, HasMTE] in {
+  defm SETGP     : MOPSMemorySetTaggingInsns<0b00, "setgp">;
+  defm SETGM     : MOPSMemorySetTaggingInsns<0b01, "setgm">;
+  // Can't use SETGE because it's a reserved name in TargetSelectionDAG.td
+  defm MOPSSETGE : MOPSMemorySetTaggingInsns<0b10, "setge">;
+}
+
 let Defs = [X16, X17], mayStore = 1, isCodeGenOnly = 1 in
 def StoreSwiftAsyncContext
       : Pseudo<(outs), (ins GPR64:$ctx, GPR64sp:$base, simm9:$offset),
                []>, Sched<[]>;
 
 def AArch64AssertZExtBool : SDNode<"AArch64ISD::ASSERT_ZEXT_BOOL", SDT_assert>;
 def : Pat<(AArch64AssertZExtBool GPR32:$op),
           (i32 GPR32:$op)>;
 
 include "AArch64InstrAtomics.td"
 include "AArch64SVEInstrInfo.td"
 include "AArch64SMEInstrInfo.td"
 include "AArch64InstrGISel.td"

llvm/lib/Target/AArch64/AArch64SchedA57.td

 //   Q form - v16i8, v8i16, v4i32
 //   D form - v1i8, v1i16, v1i32, v1i64
 //   Q form - v16i8, v8i16, v4i32, v2i64
 
 // ASIMD bitwise insert, Q-form
 def : InstRW<[A57Write_3cyc_2V], (instregex "^(BIF|BIT|BSL|BSP)v16i8")>;
 
 // ASIMD duplicate, gen reg, D-form and Q-form
-def : InstRW<[A57Write_8cyc_1L_1V], (instregex "^CPY")>;
+def : InstRW<[A57Write_8cyc_1L_1V], (instregex "^CPY[^PMEF]")>;
 def : InstRW<[A57Write_8cyc_1L_1V], (instregex "^DUPv.+gpr")>;
 
 // ASIMD move, saturating
 def : InstRW<[A57Write_4cyc_1X], (instregex "^[SU]QXTU?N")>;
 
 // ASIMD reciprocal estimate, D-form
 def : InstRW<[A57Write_5cyc_1V], (instregex "^[FU](RECP|RSQRT)(E|X)(v2f32|v1i32|v2i32|v1i64)")>;
 // ASIMD reciprocal estimate, Q-form

llvm/lib/Target/AArch64/AArch64SchedA64FX.td

 // ASIMD count, D-form
 // ASIMD count, Q-form
 def : InstRW<[A64FXWrite_4Cyc_GI0],
             (instregex "^CLSv", "^CLZv", "^CNTv")>;
 
 // ASIMD duplicate, gen reg
 // ASIMD duplicate, element
 def : InstRW<[A64FXWrite_DUPGENERAL], (instregex "^DUPv")>;
-def : InstRW<[A64FXWrite_6Cyc_GI0], (instregex "^CPY")>;
+def : InstRW<[A64FXWrite_6Cyc_GI0], (instregex "^CPY[^PMEF]")>;
 def : InstRW<[A64FXWrite_6Cyc_GI0], (instregex "^DUPv.+gpr")>;
 
 // ASIMD extract
 def : InstRW<[A64FXWrite_6Cyc_GI0], (instregex "^EXTv")>;
 
 // ASIMD extract narrow
 def : InstRW<[A64FXWrite_6Cyc_GI3], (instregex "^XTNv")>;
 

llvm/lib/Target/AArch64/AArch64SchedExynosM3.td

 
 // ASIMD miscellaneous instructions.
 def : InstRW<[M3WriteNALU1], (instregex "^RBITv")>;
 def : InstRW<[M3WriteNALU1], (instregex "^(BIF|BIT|BSL|BSP)v")>;
 def : InstRW<[M3WriteNEONB], (instregex "^DUPv.+gpr")>;
 def : InstRW<[M3WriteNSHF1], (instregex "^DUPv.+lane")>;
 def : InstRW<[M3WriteNSHF1], (instregex "^EXTv")>;
 def : InstRW<[M3WriteNSHF1], (instregex "^[SU]?Q?XTU?Nv")>;
-def : InstRW<[M3WriteNSHF1], (instregex "^CPY")>;
+def : InstRW<[M3WriteNSHF1], (instregex "^CPY[^PMEF]")>;
 def : InstRW<[M3WriteNSHF1], (instregex "^INSv.+lane")>;
 def : InstRW<[M3WriteMOVI],  (instregex "^MOVI")>;
 def : InstRW<[M3WriteNALU1], (instregex "^FMOVv")>;
 def : InstRW<[M3WriteFCVT4], (instregex "^[FU](RECP|RSQRT)Ev[248]")>;
 def : InstRW<[M3WriteFMAC4,
               M3ReadFMAC],   (instregex "^F(RECP|RSQRT)Sv")>;
 def : InstRW<[M3WriteNSHF1], (instregex "^REV(16|32|64)v")>;
 def : InstRW<[M3WriteNSHF1], (instregex "^TB[LX]v")>;

llvm/lib/Target/AArch64/AArch64SchedExynosM4.td

 def : InstRW<[M4WriteNEONX],  (instrs FSQRTv4f32)>;
 def : InstRW<[M4WriteNEONY],  (instrs FSQRTv2f64)>;
 
 // ASIMD miscellaneous instructions.
 def : InstRW<[M4WriteNALU1],  (instregex "^RBITv")>;
 def : InstRW<[M4WriteNALU1],  (instregex "^(BIF|BIT|BSL|BSP)v")>;
 def : InstRW<[M4WriteNALU1],  (instregex "^CL[STZ]v")>;
 def : InstRW<[M4WriteNEONB],  (instregex "^DUPv.+gpr")>;
-def : InstRW<[M4WriteNSHF1],  (instregex "^CPY")>;
+def : InstRW<[M4WriteNSHF1],  (instregex "^CPY[^PMEF]")>;
 def : InstRW<[M4WriteNSHF1],  (instregex "^DUPv.+lane")>;
 def : InstRW<[M4WriteNSHF1],  (instregex "^EXTv")>;
 def : InstRW<[M4WriteNSHT4A], (instregex "^XTNv")>;
 def : InstRW<[M4WriteNSHT4A], (instregex "^[SU]?QXTU?Nv")>;
 def : InstRW<[M4WriteNEONB],  (instregex "^INSv.+gpr")>;
 def : InstRW<[M4WriteNSHF1],  (instregex "^INSv.+lane")>;
 def : InstRW<[M4WriteMOVI],   (instregex "^(MOV|MVN)I")>;
 def : InstRW<[M4WriteNALU1H], (instregex "^FMOVv.f16")>;

llvm/lib/Target/AArch64/AArch64SchedExynosM5.td

 def : InstRW<[M5WriteNEONX],  (instrs FSQRTv4f32)>;
 def : InstRW<[M5WriteNEONY],  (instrs FSQRTv2f64)>;
 
 // ASIMD miscellaneous instructions.
 def : InstRW<[M5WriteNALU2],  (instregex "^RBITv")>;
 def : InstRW<[M5WriteNALU2],  (instregex "^(BIF|BIT|BSL|BSP)v")>;
 def : InstRW<[M5WriteNALU2],  (instregex "^CL[STZ]v")>;
 def : InstRW<[M5WriteNEONB],  (instregex "^DUPv.+gpr")>;
-def : InstRW<[M5WriteNSHF2],  (instregex "^CPY")>;
+def : InstRW<[M5WriteNSHF2],  (instregex "^CPY[^PMEF]")>;
 def : InstRW<[M5WriteNSHF2],  (instregex "^DUPv.+lane")>;
 def : InstRW<[M5WriteNSHF2],  (instregex "^EXTv")>;
 def : InstRW<[M5WriteNSHT4A], (instregex "^XTNv")>;
 def : InstRW<[M5WriteNSHT4A], (instregex "^[SU]?QXTU?Nv")>;
 def : InstRW<[M5WriteNEONB],  (instregex "^INSv.+gpr")>;
 def : InstRW<[M5WriteNSHF2],  (instregex "^INSv.+lane")>;
 def : InstRW<[M5WriteMOVI],   (instregex "^(MOV|MVN)I")>;
 def : InstRW<[M5WriteNALU1],  (instregex "^FMOVv.f(16|32|64)")>;

llvm/lib/Target/AArch64/AArch64SchedThunderX2T99.td

 // ASIMD count, D-form
 // ASIMD count, Q-form
 def : InstRW<[THX2T99Write_5Cyc_F01],
             (instregex "^CLSv", "^CLZv", "^CNTv")>;
 
 // ASIMD duplicate, gen reg
 // ASIMD duplicate, element
 def : InstRW<[THX2T99Write_5Cyc_F01], (instregex "^DUPv")>;
-def : InstRW<[THX2T99Write_5Cyc_F01], (instregex "^CPY")>;
+def : InstRW<[THX2T99Write_5Cyc_F01], (instregex "^CPY[^PMEF]")>;
 def : InstRW<[THX2T99Write_5Cyc_F01], (instregex "^DUPv.+gpr")>;
 
 // ASIMD extract
 def : InstRW<[THX2T99Write_5Cyc_F01], (instregex "^EXTv")>;
 
 // ASIMD extract narrow
 def : InstRW<[THX2T99Write_7Cyc_F01], (instregex "^XTNv")>;
 

llvm/lib/Target/AArch64/AArch64SchedThunderX3T110.td

 // ASIMD count, D-form
 // ASIMD count, Q-form
 def : InstRW<[THX3T110Write_3_4Cyc_F23_F0123],
             (instregex "^CLSv", "^CLZv", "^CNTv")>;
 
 // ASIMD duplicate, gen reg
 // ASIMD duplicate, element
 def : InstRW<[THX3T110Write_5Cyc_F0123], (instregex "^DUPv")>;
-def : InstRW<[THX3T110Write_5Cyc_F0123], (instregex "^CPY")>;
+def : InstRW<[THX3T110Write_5Cyc_F0123], (instregex "^CPY[^PMEF]")>;
 def : InstRW<[THX3T110Write_5Cyc_F0123], (instregex "^DUPv.+gpr")>;
 
 // ASIMD extract
 def : InstRW<[THX3T110Write_5Cyc_F0123], (instregex "^EXTv")>;
 
 // ASIMD extract narrow
 def : InstRW<[THX3T110Write_5Cyc_F0123], (instregex "^XTNv")>;
 

llvm/lib/Target/AArch64/AArch64Subtarget.h

protected:
   // Armv8.7-A Extensions
   bool HasXS = false;
   bool HasWFxT = false;
   bool HasHCX = false;
   bool HasLS64 = false;
 
   // Armv8.8-A Extensions
   bool HasHBC = false;
+  bool HasMOPS = false;
 
   // Arm SVE2 extensions
   bool HasSVE2 = false;
   bool HasSVE2AES = false;
   bool HasSVE2SM4 = false;
   bool HasSVE2SHA3 = false;
   bool HasSVE2BitPerm = false;
 
public:
   bool hasLS64() const { return HasLS64; }
   bool hasSEL2() const { return HasSEL2; }
   bool hasTLB_RMI() const { return HasTLB_RMI; }
   bool hasFlagM() const { return HasFlagM; }
   bool hasRCPC_IMMO() const { return HasRCPC_IMMO; }
   bool hasEL2VMSA() const { return HasEL2VMSA; }
   bool hasEL3() const { return HasEL3; }
   bool hasHBC() const { return HasHBC; }
+  bool hasMOPS() const { return HasMOPS; }
 
   bool fixCortexA53_835769() const { return FixCortexA53_835769; }
 
   bool addrSinkUsingGEPs() const override {
     // Keeping GEPs inbounds is important for exploiting AArch64
     // addressing-modes in ILP32 mode.
     return useAA() || isTargetILP32();
   }

llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp

case AArch64::LDRAAwriteback: {
     if (Xt == Xn)
       return Error(Loc[0],
           "unpredictable LDRA instruction, writeback base"
           " is also a destination");
     break;
   }
   }
 
+  // Check v8.8-A memops instructions.
+  switch (Inst.getOpcode()) {
+  case AArch64::CPYFP:
+  case AArch64::CPYFPWN:
+  case AArch64::CPYFPRN:
+  case AArch64::CPYFPN:
+  case AArch64::CPYFPWT:
+  case AArch64::CPYFPWTWN:
+  case AArch64::CPYFPWTRN:
+  case AArch64::CPYFPWTN:
+  case AArch64::CPYFPRT:
+  case AArch64::CPYFPRTWN:
+  case AArch64::CPYFPRTRN:
+  case AArch64::CPYFPRTN:
+  case AArch64::CPYFPT:
+  case AArch64::CPYFPTWN:
+  case AArch64::CPYFPTRN:
+  case AArch64::CPYFPTN:
+  case AArch64::CPYFM:
+  case AArch64::CPYFMWN:
+  case AArch64::CPYFMRN:
+  case AArch64::CPYFMN:
+  case AArch64::CPYFMWT:
+  case AArch64::CPYFMWTWN:
+  case AArch64::CPYFMWTRN:
+  case AArch64::CPYFMWTN:
+  case AArch64::CPYFMRT:
+  case AArch64::CPYFMRTWN:
+  case AArch64::CPYFMRTRN:
+  case AArch64::CPYFMRTN:
+  case AArch64::CPYFMT:
+  case AArch64::CPYFMTWN:
+  case AArch64::CPYFMTRN:
+  case AArch64::CPYFMTN:
+  case AArch64::CPYFE:
+  case AArch64::CPYFEWN:
+  case AArch64::CPYFERN:
+  case AArch64::CPYFEN:
+  case AArch64::CPYFEWT:
+  case AArch64::CPYFEWTWN:
+  case AArch64::CPYFEWTRN:
+  case AArch64::CPYFEWTN:
+  case AArch64::CPYFERT:
+  case AArch64::CPYFERTWN:
+  case AArch64::CPYFERTRN:
+  case AArch64::CPYFERTN:
+  case AArch64::CPYFET:
+  case AArch64::CPYFETWN:
+  case AArch64::CPYFETRN:
+  case AArch64::CPYFETN:
+  case AArch64::CPYP:
+  case AArch64::CPYPWN:
+  case AArch64::CPYPRN:
+  case AArch64::CPYPN:
+  case AArch64::CPYPWT:
+  case AArch64::CPYPWTWN:
+  case AArch64::CPYPWTRN:
+  case AArch64::CPYPWTN:
+  case AArch64::CPYPRT:
+  case AArch64::CPYPRTWN:
+  case AArch64::CPYPRTRN:
+  case AArch64::CPYPRTN:
+  case AArch64::CPYPT:
+  case AArch64::CPYPTWN:
+  case AArch64::CPYPTRN:
+  case AArch64::CPYPTN:
+  case AArch64::CPYM:
+  case AArch64::CPYMWN:
+  case AArch64::CPYMRN:
+  case AArch64::CPYMN:
+  case AArch64::CPYMWT:
+  case AArch64::CPYMWTWN:
+  case AArch64::CPYMWTRN:
+  case AArch64::CPYMWTN:
+  case AArch64::CPYMRT:
+  case AArch64::CPYMRTWN:
+  case AArch64::CPYMRTRN:
+  case AArch64::CPYMRTN:
+  case AArch64::CPYMT:
+  case AArch64::CPYMTWN:
+  case AArch64::CPYMTRN:
+  case AArch64::CPYMTN:
+  case AArch64::CPYE:
+  case AArch64::CPYEWN:
+  case AArch64::CPYERN:
+  case AArch64::CPYEN:
+  case AArch64::CPYEWT:
+  case AArch64::CPYEWTWN:
+  case AArch64::CPYEWTRN:
+  case AArch64::CPYEWTN:
+  case AArch64::CPYERT:
+  case AArch64::CPYERTWN:
+  case AArch64::CPYERTRN:
+  case AArch64::CPYERTN:
+  case AArch64::CPYET:
+  case AArch64::CPYETWN:
+  case AArch64::CPYETRN:
+  case AArch64::CPYETN: {
+    unsigned Xd_wb = Inst.getOperand(0).getReg();
+    unsigned Xs_wb = Inst.getOperand(1).getReg();
+    unsigned Xn_wb = Inst.getOperand(2).getReg();
+    unsigned Xd = Inst.getOperand(3).getReg();
+    unsigned Xs = Inst.getOperand(4).getReg();
+    unsigned Xn = Inst.getOperand(5).getReg();
+    if (Xd_wb != Xd)
+      return Error(Loc[0],
+                   "invalid CPY instruction, Xd_wb and Xd do not match");
+    if (Xs_wb != Xs)
+      return Error(Loc[0],
+                   "invalid CPY instruction, Xs_wb and Xs do not match");
+    if (Xn_wb != Xn)
+      return Error(Loc[0],
+                   "invalid CPY instruction, Xn_wb and Xn do not match");
+    if (Xd == Xs)
+      return Error(Loc[0], "invalid CPY instruction, destination and source"
+                           " registers are the same");
+    if (Xd == Xn)
+      return Error(Loc[0], "invalid CPY instruction, destination and size"
+                           " registers are the same");
+    if (Xs == Xn)
+      return Error(Loc[0], "invalid CPY instruction, source and size"
+                           " registers are the same");
+    break;
+  }
+  case AArch64::SETP:
+  case AArch64::SETPT:
+  case AArch64::SETPN:
+  case AArch64::SETPTN:
+  case AArch64::SETM:
+  case AArch64::SETMT:
+  case AArch64::SETMN:
+  case AArch64::SETMTN:
+  case AArch64::SETE:
+  case AArch64::SETET:
+  case AArch64::SETEN:
+  case AArch64::SETETN:
+  case AArch64::SETGP:
+  case AArch64::SETGPT:
+  case AArch64::SETGPN:
+  case AArch64::SETGPTN:
+  case AArch64::SETGM:
+  case AArch64::SETGMT:
+  case AArch64::SETGMN:
+  case AArch64::SETGMTN:
+  case AArch64::MOPSSETGE:
+  case AArch64::MOPSSETGET:
+  case AArch64::MOPSSETGEN:
+  case AArch64::MOPSSETGETN: {
+    unsigned Xd_wb = Inst.getOperand(0).getReg();
+    unsigned Xn_wb = Inst.getOperand(1).getReg();
+    unsigned Xd = Inst.getOperand(2).getReg();
+    unsigned Xn = Inst.getOperand(3).getReg();
+    unsigned Xm = Inst.getOperand(4).getReg();
+    if (Xd_wb != Xd)
+      return Error(Loc[0],
+                   "invalid SET instruction, Xd_wb and Xd do not match");
+    if (Xn_wb != Xn)
+      return Error(Loc[0],
+                   "invalid SET instruction, Xn_wb and Xn do not match");
+    if (Xd == Xn)
+      return Error(Loc[0], "invalid SET instruction, destination and size"
+                           " registers are the same");
+    if (Xd == Xm)
+      return Error(Loc[0], "invalid SET instruction, destination and source"
+                           " registers are the same");
+    if (Xn == Xm)
+      return Error(Loc[0], "invalid SET instruction, source and size"
+                           " registers are the same");
+    break;
+  }
+  }
 
   // Now check immediate ranges. Separate from the above as there is overlap
   // in the instructions being checked and this keeps the nested conditionals
   // to a minimum.
   switch (Inst.getOpcode()) {
   case AArch64::ADDSWri:
   case AArch64::ADDSXri:
   case AArch64::ADDWri:

llvm/lib/Target/AArch64/Disassembler/AArch64Disassembler.cpp

static DecodeStatus DecodeSImm(MCInst &Inst, uint64_t Imm, uint64_t Address,
                                const void *Decoder);
 template <int ElementWidth>
 static DecodeStatus DecodeImm8OptLsl(MCInst &Inst, unsigned Imm,
                                      uint64_t Addr, const void *Decoder);
 static DecodeStatus DecodeSVEIncDecImm(MCInst &Inst, unsigned Imm,
                                        uint64_t Addr, const void *Decoder);
 static DecodeStatus DecodeSVCROp(MCInst &Inst, unsigned Imm, uint64_t Address,
                                  const void *Decoder);
+static DecodeStatus DecodeCPYMemOpInstruction(MCInst &Inst, uint32_t insn,
+                                              uint64_t Addr,
+                                              const void *Decoder);
+static DecodeStatus DecodeSETMemOpInstruction(MCInst &Inst, uint32_t insn,
+                                              uint64_t Addr,
+                                              const void *Decoder);
 
 static bool Check(DecodeStatus &Out, DecodeStatus In) {
   switch (In) {
     case MCDisassembler::Success:
       // Out stays the same.
       return true;
     case MCDisassembler::SoftFail:
       Out = In;

 static DecodeStatus DecodeSVCROp(MCInst &Inst, unsigned Imm, uint64_t Address,
                                  const void *Decoder) {
   if (AArch64SVCR::lookupSVCRByEncoding(Imm)) {
     Inst.addOperand(MCOperand::createImm(Imm));
     return Success;
   }
   return Fail;
 }
+
+static DecodeStatus DecodeCPYMemOpInstruction(MCInst &Inst, uint32_t insn,
+                                              uint64_t Addr,
+                                              const void *Decoder) {
+  unsigned Rd = fieldFromInstruction(insn, 0, 5);
+  unsigned Rs = fieldFromInstruction(insn, 16, 5);
+  unsigned Rn = fieldFromInstruction(insn, 5, 5);
+
+  // None of the registers may alias: if they do, then the instruction is not
+  // merely unpredictable but actually entirely unallocated.
+  if (Rd == Rs || Rs == Rn || Rd == Rn)
+    return MCDisassembler::Fail;
+
+  // All three register operands are written back, so they all appear
+  // twice in the operand list, once as outputs and once as inputs.
+  if (!DecodeGPR64commonRegisterClass(Inst, Rd, Addr, Decoder) ||
+      !DecodeGPR64commonRegisterClass(Inst, Rs, Addr, Decoder) ||
+      !DecodeGPR64RegisterClass(Inst, Rn, Addr, Decoder) ||
+      !DecodeGPR64commonRegisterClass(Inst, Rd, Addr, Decoder) ||
+      !DecodeGPR64commonRegisterClass(Inst, Rs, Addr, Decoder) ||
+      !DecodeGPR64RegisterClass(Inst, Rn, Addr, Decoder))
+    return MCDisassembler::Fail;
+
+  return MCDisassembler::Success;
+}
+
+static DecodeStatus DecodeSETMemOpInstruction(MCInst &Inst, uint32_t insn,
+                                              uint64_t Addr,
+                                              const void *Decoder) {
+  unsigned Rd = fieldFromInstruction(insn, 0, 5);
+  unsigned Rm = fieldFromInstruction(insn, 16, 5);
+  unsigned Rn = fieldFromInstruction(insn, 5, 5);
+
+  // None of the registers may alias: if they do, then the instruction is not
+  // merely unpredictable but actually entirely unallocated.
+  if (Rd == Rm || Rm == Rn || Rd == Rn)
+    return MCDisassembler::Fail;
+
+  // Rd and Rn (not Rm) register operands are written back, so they appear
+  // twice in the operand list, once as outputs and once as inputs.
+  if (!DecodeGPR64commonRegisterClass(Inst, Rd, Addr, Decoder) ||
+      !DecodeGPR64RegisterClass(Inst, Rn, Addr, Decoder) ||
+      !DecodeGPR64commonRegisterClass(Inst, Rd, Addr, Decoder) ||
+      !DecodeGPR64RegisterClass(Inst, Rn, Addr, Decoder) ||
+      !DecodeGPR64RegisterClass(Inst, Rm, Addr, Decoder))
+    return MCDisassembler::Fail;
+
+  return MCDisassembler::Success;
+}
 No newline at end of file

llvm/test/MC/AArch64/armv8.8a-mops.s

+// RUN: not llvm-mc -triple aarch64-none-linux-gnu -show-encoding -mattr=+mops,+mte < %s 2> %t | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MTE
+// RUN: FileCheck --check-prefix=CHECK-ERROR %s < %t
+// RUN: not llvm-mc -triple aarch64-none-linux-gnu -show-encoding -mattr=+v8.8a,+mte < %s 2> %t | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-MTE
+// RUN: FileCheck --check-prefix=CHECK-ERROR %s < %t
+// RUN: not llvm-mc -triple aarch64-none-linux-gnu -show-encoding -mattr=+mops < %s 2> %t | FileCheck %s --check-prefix=CHECK
+// RUN: FileCheck --check-prefix=CHECK-NO-MTE-ERR %s < %t
+// RUN: not llvm-mc -triple aarch64-none-linux-gnu -show-encoding -mattr=+v8.8a < %s 2> %t | FileCheck %s --check-prefix=CHECK
+// RUN: FileCheck --check-prefix=CHECK-NO-MTE-ERR %s < %t
+// RUN: not llvm-mc -triple aarch64-none-linux-gnu < %s 2> %t
+// RUN: FileCheck --check-prefix=CHECK-NO-MOPS-ERR --check-prefix=CHECK-NO-MOPSMTE-ERR %s < %t
+
+// CHECK:      [0x40,0x04,0x01,0x19]
+// CHECK-NEXT: [0x40,0x44,0x01,0x19]
+// CHECK-NEXT: [0x40,0x84,0x01,0x19]
+// CHECK-NEXT: [0x40,0xc4,0x01,0x19]
+// CHECK-NEXT: [0x40,0x14,0x01,0x19]
+// CHECK-NEXT: [0x40,0x54,0x01,0x19]
+// CHECK-NEXT: [0x40,0x94,0x01,0x19]
+// CHECK-NEXT: [0x40,0xd4,0x01,0x19]
+// CHECK-NEXT: [0x40,0x24,0x01,0x19]
+// CHECK-NEXT: [0x40,0x64,0x01,0x19]
+// CHECK-NEXT: [0x40,0xa4,0x01,0x19]
+// CHECK-NEXT: [0x40,0xe4,0x01,0x19]
+// CHECK-NEXT: [0x40,0x34,0x01,0x19]
+// CHECK-NEXT: [0x40,0x74,0x01,0x19]
+// CHECK-NEXT: [0x40,0xb4,0x01,0x19]
+// CHECK-NEXT: [0x40,0xf4,0x01,0x19]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+cpyfp [x0]!, [x1]!, x2!
+cpyfpwn [x0]!, [x1]!, x2!
+cpyfprn [x0]!, [x1]!, x2!
+cpyfpn [x0]!, [x1]!, x2!
+cpyfpwt [x0]!, [x1]!, x2!
+cpyfpwtwn [x0]!, [x1]!, x2!
+cpyfpwtrn [x0]!, [x1]!, x2!
+cpyfpwtn [x0]!, [x1]!, x2!
+cpyfprt [x0]!, [x1]!, x2!
+cpyfprtwn [x0]!, [x1]!, x2!
+cpyfprtrn [x0]!, [x1]!, x2!
+cpyfprtn [x0]!, [x1]!, x2!
+cpyfpt [x0]!, [x1]!, x2!
+cpyfptwn [x0]!, [x1]!, x2!
+cpyfptrn [x0]!, [x1]!, x2!
+cpyfptn [x0]!, [x1]!, x2!
+
+// CHECK:      [0x40,0x04,0x41,0x19]
+// CHECK-NEXT: [0x40,0x44,0x41,0x19]
+// CHECK-NEXT: [0x40,0x84,0x41,0x19]
+// CHECK-NEXT: [0x40,0xc4,0x41,0x19]
+// CHECK-NEXT: [0x40,0x14,0x41,0x19]
+// CHECK-NEXT: [0x40,0x54,0x41,0x19]
+// CHECK-NEXT: [0x40,0x94,0x41,0x19]
+// CHECK-NEXT: [0x40,0xd4,0x41,0x19]
+// CHECK-NEXT: [0x40,0x24,0x41,0x19]
+// CHECK-NEXT: [0x40,0x64,0x41,0x19]
+// CHECK-NEXT: [0x40,0xa4,0x41,0x19]
+// CHECK-NEXT: [0x40,0xe4,0x41,0x19]
+// CHECK-NEXT: [0x40,0x34,0x41,0x19]
+// CHECK-NEXT: [0x40,0x74,0x41,0x19]
+// CHECK-NEXT: [0x40,0xb4,0x41,0x19]
+// CHECK-NEXT: [0x40,0xf4,0x41,0x19]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+cpyfm [x0]!, [x1]!, x2!
+cpyfmwn [x0]!, [x1]!, x2!
+cpyfmrn [x0]!, [x1]!, x2!
+cpyfmn [x0]!, [x1]!, x2!
+cpyfmwt [x0]!, [x1]!, x2!
+cpyfmwtwn [x0]!, [x1]!, x2!
+cpyfmwtrn [x0]!, [x1]!, x2!
+cpyfmwtn [x0]!, [x1]!, x2!
+cpyfmrt [x0]!, [x1]!, x2!
+cpyfmrtwn [x0]!, [x1]!, x2!
+cpyfmrtrn [x0]!, [x1]!, x2!
+cpyfmrtn [x0]!, [x1]!, x2!
+cpyfmt [x0]!, [x1]!, x2!
+cpyfmtwn [x0]!, [x1]!, x2!
+cpyfmtrn [x0]!, [x1]!, x2!
+cpyfmtn [x0]!, [x1]!, x2!
+
+// CHECK:      [0x40,0x04,0x81,0x19]
+// CHECK-NEXT: [0x40,0x44,0x81,0x19]
+// CHECK-NEXT: [0x40,0x84,0x81,0x19]
+// CHECK-NEXT: [0x40,0xc4,0x81,0x19]
+// CHECK-NEXT: [0x40,0x14,0x81,0x19]
+// CHECK-NEXT: [0x40,0x54,0x81,0x19]
+// CHECK-NEXT: [0x40,0x94,0x81,0x19]
+// CHECK-NEXT: [0x40,0xd4,0x81,0x19]
+// CHECK-NEXT: [0x40,0x24,0x81,0x19]
+// CHECK-NEXT: [0x40,0x64,0x81,0x19]
+// CHECK-NEXT: [0x40,0xa4,0x81,0x19]
+// CHECK-NEXT: [0x40,0xe4,0x81,0x19]
+// CHECK-NEXT: [0x40,0x34,0x81,0x19]
+// CHECK-NEXT: [0x40,0x74,0x81,0x19]
+// CHECK-NEXT: [0x40,0xb4,0x81,0x19]
+// CHECK-NEXT: [0x40,0xf4,0x81,0x19]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+cpyfe [x0]!, [x1]!, x2!
+cpyfewn [x0]!, [x1]!, x2!
+cpyfern [x0]!, [x1]!, x2!
+cpyfen [x0]!, [x1]!, x2!
+cpyfewt [x0]!, [x1]!, x2!
+cpyfewtwn [x0]!, [x1]!, x2!
+cpyfewtrn [x0]!, [x1]!, x2!
+cpyfewtn [x0]!, [x1]!, x2!
+cpyfert [x0]!, [x1]!, x2!
+cpyfertwn [x0]!, [x1]!, x2!
+cpyfertrn [x0]!, [x1]!, x2!
+cpyfertn [x0]!, [x1]!, x2!
+cpyfet [x0]!, [x1]!, x2!
+cpyfetwn [x0]!, [x1]!, x2!
+cpyfetrn [x0]!, [x1]!, x2!
+cpyfetn [x0]!, [x1]!, x2!
+
+// CHECK:      [0x40,0x04,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x44,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x84,0x01,0x1d]
+// CHECK-NEXT: [0x40,0xc4,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x14,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x54,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x94,0x01,0x1d]
+// CHECK-NEXT: [0x40,0xd4,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x24,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x64,0x01,0x1d]
+// CHECK-NEXT: [0x40,0xa4,0x01,0x1d]
+// CHECK-NEXT: [0x40,0xe4,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x34,0x01,0x1d]
+// CHECK-NEXT: [0x40,0x74,0x01,0x1d]
+// CHECK-NEXT: [0x40,0xb4,0x01,0x1d]
+// CHECK-NEXT: [0x40,0xf4,0x01,0x1d]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+cpyp [x0]!, [x1]!, x2!
+cpypwn [x0]!, [x1]!, x2!
+cpyprn [x0]!, [x1]!, x2!
+cpypn [x0]!, [x1]!, x2!
+cpypwt [x0]!, [x1]!, x2!
+cpypwtwn [x0]!, [x1]!, x2!
+cpypwtrn [x0]!, [x1]!, x2!
+cpypwtn [x0]!, [x1]!, x2!
+cpyprt [x0]!, [x1]!, x2!
+cpyprtwn [x0]!, [x1]!, x2!
+cpyprtrn [x0]!, [x1]!, x2!
+cpyprtn [x0]!, [x1]!, x2!
+cpypt [x0]!, [x1]!, x2!
+cpyptwn [x0]!, [x1]!, x2!
+cpyptrn [x0]!, [x1]!, x2!
+cpyptn [x0]!, [x1]!, x2!
+
+// CHECK:      [0x40,0x04,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x44,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x84,0x41,0x1d]
+// CHECK-NEXT: [0x40,0xc4,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x14,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x54,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x94,0x41,0x1d]
+// CHECK-NEXT: [0x40,0xd4,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x24,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x64,0x41,0x1d]
+// CHECK-NEXT: [0x40,0xa4,0x41,0x1d]
+// CHECK-NEXT: [0x40,0xe4,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x34,0x41,0x1d]
+// CHECK-NEXT: [0x40,0x74,0x41,0x1d]
+// CHECK-NEXT: [0x40,0xb4,0x41,0x1d]
+// CHECK-NEXT: [0x40,0xf4,0x41,0x1d]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+cpym [x0]!, [x1]!, x2!
+cpymwn [x0]!, [x1]!, x2!
+cpymrn [x0]!, [x1]!, x2!
+cpymn [x0]!, [x1]!, x2!
+cpymwt [x0]!, [x1]!, x2!
+cpymwtwn [x0]!, [x1]!, x2!
+cpymwtrn [x0]!, [x1]!, x2!
+cpymwtn [x0]!, [x1]!, x2!
+cpymrt [x0]!, [x1]!, x2!
+cpymrtwn [x0]!, [x1]!, x2!
+cpymrtrn [x0]!, [x1]!, x2!
+cpymrtn [x0]!, [x1]!, x2!
+cpymt [x0]!, [x1]!, x2!
+cpymtwn [x0]!, [x1]!, x2!
+cpymtrn [x0]!, [x1]!, x2!
+cpymtn [x0]!, [x1]!, x2!
+
+// CHECK:      [0x40,0x04,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x44,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x84,0x81,0x1d]
+// CHECK-NEXT: [0x40,0xc4,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x14,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x54,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x94,0x81,0x1d]
+// CHECK-NEXT: [0x40,0xd4,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x24,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x64,0x81,0x1d]
+// CHECK-NEXT: [0x40,0xa4,0x81,0x1d]
+// CHECK-NEXT: [0x40,0xe4,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x34,0x81,0x1d]
+// CHECK-NEXT: [0x40,0x74,0x81,0x1d]
+// CHECK-NEXT: [0x40,0xb4,0x81,0x1d]
+// CHECK-NEXT: [0x40,0xf4,0x81,0x1d]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+cpye [x0]!, [x1]!, x2!
+cpyewn [x0]!, [x1]!, x2!
+cpyern [x0]!, [x1]!, x2!
+cpyen [x0]!, [x1]!, x2!
+cpyewt [x0]!, [x1]!, x2!
+cpyewtwn [x0]!, [x1]!, x2!
+cpyewtrn [x0]!, [x1]!, x2!
+cpyewtn [x0]!, [x1]!, x2!
+cpyert [x0]!, [x1]!, x2!
+cpyertwn [x0]!, [x1]!, x2!
+cpyertrn [x0]!, [x1]!, x2!
+cpyertn [x0]!, [x1]!, x2!
+cpyet [x0]!, [x1]!, x2!
+cpyetwn [x0]!, [x1]!, x2!
+cpyetrn [x0]!, [x1]!, x2!
+cpyetn [x0]!, [x1]!, x2!
+
+// CHECK:      [0x20,0x04,0xc2,0x19]
+// CHECK-NEXT: [0x20,0x14,0xc2,0x19]
+// CHECK-NEXT: [0x20,0x24,0xc2,0x19]
+// CHECK-NEXT: [0x20,0x34,0xc2,0x19]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+setp [x0]!, x1!, x2
+setpt [x0]!, x1!, x2
+setpn [x0]!, x1!, x2
+setptn [x0]!, x1!, x2
+
+// CHECK: [0x20,0x44,0xc2,0x19]
+// CHECK: [0x20,0x54,0xc2,0x19]
+// CHECK: [0x20,0x64,0xc2,0x19]
+// CHECK: [0x20,0x74,0xc2,0x19]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+setm [x0]!, x1!, x2
+setmt [x0]!, x1!, x2
+setmn [x0]!, x1!, x2
+setmtn [x0]!, x1!, x2
+
+// CHECK: [0x20,0x84,0xc2,0x19]
+// CHECK: [0x20,0x94,0xc2,0x19]
+// CHECK: [0x20,0xa4,0xc2,0x19]
+// CHECK: [0x20,0xb4,0xc2,0x19]
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+// CHECK-NO-MOPS-ERR: error: instruction requires: mops
+sete [x0]!, x1!, x2
+setet [x0]!, x1!, x2
+seten [x0]!, x1!, x2
+setetn [x0]!, x1!, x2
+
+// CHECK-MTE: [0x20,0x04,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x14,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x24,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x34,0xc2,0x1d]
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+setgp [x0]!, x1!, x2
+setgpt [x0]!, x1!, x2
+setgpn [x0]!, x1!, x2
+setgptn [x0]!, x1!, x2
+
+// CHECK-MTE: [0x20,0x44,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x54,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x64,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x74,0xc2,0x1d]
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+setgm [x0]!, x1!, x2
+setgmt [x0]!, x1!, x2
+setgmn [x0]!, x1!, x2
+setgmtn [x0]!, x1!, x2
+
+// CHECK-MTE: [0x20,0x84,0xc2,0x1d]
+// CHECK-MTE: [0x20,0x94,0xc2,0x1d]
+// CHECK-MTE: [0x20,0xa4,0xc2,0x1d]
+// CHECK-MTE: [0x20,0xb4,0xc2,0x1d]
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MTE-ERR: error: instruction requires: mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+// CHECK-NO-MOPSMTE-ERR: error: instruction requires: mops mte
+setge [x0]!, x1!, x2
+setget [x0]!, x1!, x2
+setgen [x0]!, x1!, x2
+setgetn [x0]!, x1!, x2
+
+// All operand must be different from each other
+
+// CHECK-ERROR: error: invalid CPY instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, source and size registers are the same
+cpyfp [x0]!, [x0]!, x1!
+cpyfp [x0]!, [x1]!, x0!
+cpyfp [x1]!, [x0]!, x0!
+
+// CHECK-ERROR: error: invalid CPY instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, source and size registers are the same
+cpyfm [x0]!, [x0]!, x1!
+cpyfm [x0]!, [x1]!, x0!
+cpyfm [x1]!, [x0]!, x0!
+
+// CHECK-ERROR: error: invalid CPY instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, source and size registers are the same
+cpyfe [x0]!, [x0]!, x1!
+cpyfe [x0]!, [x1]!, x0!
+cpyfe [x1]!, [x0]!, x0!
+
+// CHECK-ERROR: error: invalid CPY instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, source and size registers are the same
+cpyp [x0]!, [x0]!, x1!
+cpyp [x0]!, [x1]!, x0!
+cpyp [x1]!, [x0]!, x0!
+
+// CHECK-ERROR: error: invalid CPY instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, source and size registers are the same
+cpym [x0]!, [x0]!, x1!
+cpym [x0]!, [x1]!, x0!
+cpym [x1]!, [x0]!, x0!
+
+// CHECK-ERROR: error: invalid CPY instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid CPY instruction, source and size registers are the same
+cpye [x0]!, [x0]!, x1!
+cpye [x0]!, [x1]!, x0!
+cpye [x1]!, [x0]!, x0!
+
+// CHECK-ERROR: error: invalid SET instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid SET instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid SET instruction, source and size registers are the same
+setp [x0]!, x0!, x1
+setp [x0]!, x1!, x0
+setp [x1]!, x0!, x0
+
+// CHECK-ERROR: error: invalid SET instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid SET instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid SET instruction, source and size registers are the same
+setm [x0]!, x0!, x1
+setm [x0]!, x1!, x0
+setm [x1]!, x0!, x0
+
+// CHECK-ERROR: error: invalid SET instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid SET instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid SET instruction, source and size registers are the same
+sete [x0]!, x0!, x1
+sete [x0]!, x1!, x0
+sete [x1]!, x0!, x0
+
+// CHECK-ERROR: error: invalid SET instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid SET instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid SET instruction, source and size registers are the same
+setgp [x0]!, x0!, x1
+setgp [x0]!, x1!, x0
+setgp [x1]!, x0!, x0
+
+// CHECK-ERROR: error: invalid SET instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid SET instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid SET instruction, source and size registers are the same
+setgm [x0]!, x0!, x1
+setgm [x0]!, x1!, x0
+setgm [x1]!, x0!, x0
+
+// CHECK-ERROR: error: invalid SET instruction, destination and size registers are the same
+// CHECK-ERROR: error: invalid SET instruction, destination and source registers are the same
+// CHECK-ERROR: error: invalid SET instruction, source and size registers are the same
+setge [x0]!, x0!, x1
+setge [x0]!, x1!, x0
+setge [x1]!, x0!, x0
+
+// SP cannot be used as argument at any position
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+cpyfp [sp]!, [x1]!, x2!
+cpyfp [x0]!, [sp]!, x2!
+cpyfp [x0]!, [x1]!, sp!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+cpyfm [sp]!, [x1]!, x2!
+cpyfm [x0]!, [sp]!, x2!
+cpyfm [x0]!, [x1]!, sp!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+cpyfe [sp]!, [x1]!, x2!
+cpyfe [x0]!, [sp]!, x2!
+cpyfe [x0]!, [x1]!, sp!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+cpyp [sp]!, [x2]!, x2!
+cpyp [x0]!, [sp]!, x2!
+cpyp [x0]!, [x1]!, sp!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+cpym [sp]!, [x2]!, x2!
+cpym [x0]!, [sp]!, x2!
+cpym [x0]!, [x1]!, sp!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+cpye [sp]!, [x2]!, x2!
+cpye [x0]!, [sp]!, x2!
+cpye [x0]!, [x1]!, sp!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+setp [sp]!, x1!, x2
+setp [x0]!, sp!, x2
+setp [x0]!, x1!, sp
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+setm [sp]!, x1!, x2
+setm [x0]!, sp!, x2
+setm [x0]!, x1!, sp
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+sete [sp]!, x1!, x2
+sete [x0]!, sp!, x2
+sete [x0]!, x1!, sp
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+setgp [sp]!, x1!, x2
+setgp [x0]!, sp!, x2
+setgp [x0]!, x1!, sp
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+setgm [sp]!, x1!, x2
+setgm [x0]!, sp!, x2
+setgm [x0]!, x1!, sp
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+setge [sp]!, x1!, x2
+setge [x0]!, sp!, x2
+setge [x0]!, x1!, sp
+
+// XZR can only be used at:
+//  - the size operand in CPY.
+//  - the size or source operands in SET.
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       cpyfp [x0]!, [x1]!, xzr!
+cpyfp [xzr]!, [x1]!, x2!
+cpyfp [x0]!, [xzr]!, x2!
+cpyfp [x0]!, [x1]!, xzr!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       cpyfm [x0]!, [x1]!, xzr!
+cpyfm [xzr]!, [x1]!, x2!
+cpyfm [x0]!, [xzr]!, x2!
+cpyfm [x0]!, [x1]!, xzr!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       cpyfe [x0]!, [x1]!, xzr!
+cpyfe [xzr]!, [x1]!, x2!
+cpyfe [x0]!, [xzr]!, x2!
+cpyfe [x0]!, [x1]!, xzr!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       cpyp [x0]!, [x1]!, xzr!
+cpyp [xzr]!, [x2]!, x2!
+cpyp [x0]!, [xzr]!, x2!
+cpyp [x0]!, [x1]!, xzr!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       cpym [x0]!, [x1]!, xzr!
+cpym [xzr]!, [x2]!, x2!
+cpym [x0]!, [xzr]!, x2!
+cpym [x0]!, [x1]!, xzr!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       cpye [x0]!, [x1]!, xzr!
+cpye [xzr]!, [x2]!, x2!
+cpye [x0]!, [xzr]!, x2!
+cpye [x0]!, [x1]!, xzr!
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       setp [x0]!, xzr!, x2
+// CHECK:       setp [x0]!, x1!, xzr
+setp [xzr]!, x1!, x2
+setp [x0]!, xzr!, x2
+setp [x0]!, x1!, xzr
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       setm [x0]!, xzr!, x2
+// CHECK:       setm [x0]!, x1!, xzr
+setm [xzr]!, x1!, x2
+setm [x0]!, xzr!, x2
+setm [x0]!, x1!, xzr
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK:       sete [x0]!, xzr!, x2
+// CHECK:       sete [x0]!, x1!, xzr
+sete [xzr]!, x1!, x2
+sete [x0]!, xzr!, x2
+sete [x0]!, x1!, xzr
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-MTE:   setgp [x0]!, xzr!, x2
+// CHECK-MTE:   setgp [x0]!, x1!, xzr
+setgp [xzr]!, x1!, x2
+setgp [x0]!, xzr!, x2
+setgp [x0]!, x1!, xzr
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-MTE:   setgm [x0]!, xzr!, x2
+// CHECK-MTE:   setgm [x0]!, x1!, xzr
+setgm [xzr]!, x1!, x2
+setgm [x0]!, xzr!, x2
+setgm [x0]!, x1!, xzr
+
+// CHECK-ERROR: error: invalid operand for instruction
+// CHECK-MTE:   setge [x0]!, xzr!, x2
+// CHECK-MTE:   setge [x0]!, x1!, xzr
+setge [xzr]!, x1!, x2
+setge [x0]!, xzr!, x2
+setge [x0]!, x1!, xzr

llvm/test/MC/Disassembler/AArch64/armv8.8a-mops.txt

+# RUN: not llvm-mc -triple aarch64-arm-none-eabi -mattr=+mops,+mte -disassemble < %s 2> %t | FileCheck %s --check-prefixes=CHECK-MOPS,CHECK-MTE
+# RUN: FileCheck %s --check-prefix=CHECK-INVALID < %t
+# RUN: not llvm-mc -triple aarch64-arm-none-eabi -mattr=+v8.8a,+mte -disassemble < %s 2> %t | FileCheck %s --check-prefixes=CHECK-MOPS,CHECK-MTE
+# RUN: FileCheck %s --check-prefix=CHECK-INVALID < %t
+# RUN: not llvm-mc -triple aarch64-arm-none-eabi -mattr=+mops -disassemble < %s 2> %t | FileCheck %s --check-prefix=CHECK-MOPS
+# RUN: FileCheck %s --check-prefixes=CHECK-INVALID,CHECK-NO-MTE < %t
+# RUN: not llvm-mc -triple aarch64-arm-none-eabi -mattr=+v8.8a -disassemble < %s 2> %t | FileCheck %s --check-prefix=CHECK-MOPS
+# RUN: FileCheck %s --check-prefixes=CHECK-INVALID,CHECK-NO-MTE < %t
+# RUN: not llvm-mc -triple aarch64-arm-none-eabi -disassemble < %s 2> %t
+# RUN: FileCheck %s --check-prefixes=CHECK-INVALID,CHECK-NO-MOPS,CHECK-NO-MTE < %t
+
+
+[0x40,0x04,0x01,0x19]
+[0x40,0x44,0x01,0x19]
+[0x40,0x84,0x01,0x19]
+[0x40,0xc4,0x01,0x19]
+[0x40,0x14,0x01,0x19]
+[0x40,0x54,0x01,0x19]
+[0x40,0x94,0x01,0x19]
+[0x40,0xd4,0x01,0x19]
+[0x40,0x24,0x01,0x19]
+[0x40,0x64,0x01,0x19]
+[0x40,0xa4,0x01,0x19]
+[0x40,0xe4,0x01,0x19]
+[0x40,0x34,0x01,0x19]
+[0x40,0x74,0x01,0x19]
+[0x40,0xb4,0x01,0x19]
+[0x40,0xf4,0x01,0x19]
+# CHECK-MOPS: cpyfp [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfprn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpwt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpwtwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpwtrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpwtn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfprt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfprtwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfprtrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfprtn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfpt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfptwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfptrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfptn [x0]!, [x1]!, x2!
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x40,0x04,0x41,0x19]
+[0x40,0x44,0x41,0x19]
+[0x40,0x84,0x41,0x19]
+[0x40,0xc4,0x41,0x19]
+[0x40,0x14,0x41,0x19]
+[0x40,0x54,0x41,0x19]
+[0x40,0x94,0x41,0x19]
+[0x40,0xd4,0x41,0x19]
+[0x40,0x24,0x41,0x19]
+[0x40,0x64,0x41,0x19]
+[0x40,0xa4,0x41,0x19]
+[0x40,0xe4,0x41,0x19]
+[0x40,0x34,0x41,0x19]
+[0x40,0x74,0x41,0x19]
+[0x40,0xb4,0x41,0x19]
+[0x40,0xf4,0x41,0x19]
+# CHECK-MOPS: cpyfm [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmwt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmwtwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmwtrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmwtn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmrt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmrtwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmrtrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmrtn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmtwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmtrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfmtn [x0]!, [x1]!, x2!
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x40,0x04,0x81,0x19]
+[0x40,0x44,0x81,0x19]
+[0x40,0x84,0x81,0x19]
+[0x40,0xc4,0x81,0x19]
+[0x40,0x14,0x81,0x19]
+[0x40,0x54,0x81,0x19]
+[0x40,0x94,0x81,0x19]
+[0x40,0xd4,0x81,0x19]
+[0x40,0x24,0x81,0x19]
+[0x40,0x64,0x81,0x19]
+[0x40,0xa4,0x81,0x19]
+[0x40,0xe4,0x81,0x19]
+[0x40,0x34,0x81,0x19]
+[0x40,0x74,0x81,0x19]
+[0x40,0xb4,0x81,0x19]
+[0x40,0xf4,0x81,0x19]
+# CHECK-MOPS: cpyfe [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfewn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfern [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfen  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfewt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfewtwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfewtrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfewtn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfert [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfertwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfertrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfertn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfet  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfetwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfetrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyfetn [x0]!, [x1]!, x2!
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x40,0x04,0x01,0x1d]
+[0x40,0x44,0x01,0x1d]
+[0x40,0x84,0x01,0x1d]
+[0x40,0xc4,0x01,0x1d]
+[0x40,0x14,0x01,0x1d]
+[0x40,0x54,0x01,0x1d]
+[0x40,0x94,0x01,0x1d]
+[0x40,0xd4,0x01,0x1d]
+[0x40,0x24,0x01,0x1d]
+[0x40,0x64,0x01,0x1d]
+[0x40,0xa4,0x01,0x1d]
+[0x40,0xe4,0x01,0x1d]
+[0x40,0x34,0x01,0x1d]
+[0x40,0x74,0x01,0x1d]
+[0x40,0xb4,0x01,0x1d]
+[0x40,0xf4,0x01,0x1d]
+# CHECK-MOPS: cpyp  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyprn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypwt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypwtwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypwtrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypwtn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyprt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyprtwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyprtrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyprtn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpypt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyptwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyptrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyptn  [x0]!, [x1]!, x2!
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x40,0x04,0x41,0x1d]
+[0x40,0x44,0x41,0x1d]
+[0x40,0x84,0x41,0x1d]
+[0x40,0xc4,0x41,0x1d]
+[0x40,0x14,0x41,0x1d]
+[0x40,0x54,0x41,0x1d]
+[0x40,0x94,0x41,0x1d]
+[0x40,0xd4,0x41,0x1d]
+[0x40,0x24,0x41,0x1d]
+[0x40,0x64,0x41,0x1d]
+[0x40,0xa4,0x41,0x1d]
+[0x40,0xe4,0x41,0x1d]
+[0x40,0x34,0x41,0x1d]
+[0x40,0x74,0x41,0x1d]
+[0x40,0xb4,0x41,0x1d]
+[0x40,0xf4,0x41,0x1d]
+# CHECK-MOPS: cpym  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymwt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymwtwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymwtrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymwtn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymrt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymrtwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymrtrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymrtn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymt [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymtwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymtrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpymtn  [x0]!, [x1]!, x2!
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x40,0x04,0x81,0x1d]
+[0x40,0x44,0x81,0x1d]
+[0x40,0x84,0x81,0x1d]
+[0x40,0xc4,0x81,0x1d]
+[0x40,0x14,0x81,0x1d]
+[0x40,0x54,0x81,0x1d]
+[0x40,0x94,0x81,0x1d]
+[0x40,0xd4,0x81,0x1d]
+[0x40,0x24,0x81,0x1d]
+[0x40,0x64,0x81,0x1d]
+[0x40,0xa4,0x81,0x1d]
+[0x40,0xe4,0x81,0x1d]
+[0x40,0x34,0x81,0x1d]
+[0x40,0x74,0x81,0x1d]
+[0x40,0xb4,0x81,0x1d]
+[0x40,0xf4,0x81,0x1d]
+# CHECK-MOPS: cpye  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyewn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyern  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyen [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyewt  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyewtwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyewtrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyewtn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyert  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyertwn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyertrn  [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyertn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyet [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyetwn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyetrn [x0]!, [x1]!, x2!
+# CHECK-MOPS: cpyetn  [x0]!, [x1]!, x2!
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x20,0x04,0xc2,0x19]
+[0x20,0x14,0xc2,0x19]
+[0x20,0x24,0xc2,0x19]
+[0x20,0x34,0xc2,0x19]
+# CHECK-MOPS: setp  [x0]!, x1!, x2
+# CHECK-MOPS: setpt [x0]!, x1!, x2
+# CHECK-MOPS: setpn [x0]!, x1!, x2
+# CHECK-MOPS: setptn  [x0]!, x1!, x2
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x20,0x44,0xc2,0x19]
+[0x20,0x54,0xc2,0x19]
+[0x20,0x64,0xc2,0x19]
+[0x20,0x74,0xc2,0x19]
+# CHECK-MOPS: setm  [x0]!, x1!, x2
+# CHECK-MOPS: setmt [x0]!, x1!, x2
+# CHECK-MOPS: setmn [x0]!, x1!, x2
+# CHECK-MOPS: setmtn  [x0]!, x1!, x2
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x20,0x84,0xc2,0x19]
+[0x20,0x94,0xc2,0x19]
+[0x20,0xa4,0xc2,0x19]
+[0x20,0xb4,0xc2,0x19]
+# CHECK-MOPS: sete  [x0]!, x1!, x2
+# CHECK-MOPS: setet [x0]!, x1!, x2
+# CHECK-MOPS: seten [x0]!, x1!, x2
+# CHECK-MOPS: setetn  [x0]!, x1!, x2
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+# CHECK-NO-MOPS: warning: invalid instruction encoding
+
+[0x20,0x04,0xc2,0x1d]
+[0x20,0x14,0xc2,0x1d]
+[0x20,0x24,0xc2,0x1d]
+[0x20,0x34,0xc2,0x1d]
+# CHECK-MTE: setgp  [x0]!, x1!, x2
+# CHECK-MTE: setgpt [x0]!, x1!, x2
+# CHECK-MTE: setgpn [x0]!, x1!, x2
+# CHECK-MTE: setgptn  [x0]!, x1!, x2
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+
+[0x20,0x44,0xc2,0x1d]
+[0x20,0x54,0xc2,0x1d]
+[0x20,0x64,0xc2,0x1d]
+[0x20,0x74,0xc2,0x1d]
+# CHECK-MTE: setgm  [x0]!, x1!, x2
+# CHECK-MTE: setgmt [x0]!, x1!, x2
+# CHECK-MTE: setgmn [x0]!, x1!, x2
+# CHECK-MTE: setgmtn  [x0]!, x1!, x2
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+
+[0x20,0x84,0xc2,0x1d]
+[0x20,0x94,0xc2,0x1d]
+[0x20,0xa4,0xc2,0x1d]
+[0x20,0xb4,0xc2,0x1d]
+# CHECK-MTE: setge  [x0]!, x1!, x2
+# CHECK-MTE: setget [x0]!, x1!, x2
+# CHECK-MTE: setgen [x0]!, x1!, x2
+# CHECK-MTE: setgetn  [x0]!, x1!, x2
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+# CHECK-NO-MTE: warning: invalid instruction encoding
+
+
+# Register number 31 (SP or XZR) is not allowed in address positions.
+# cpyfp
+[0x5f,0x04,0x01,0x19]
+[0x40,0x04,0x1f,0x19]
+# cpyfm
+[0x5f,0x04,0x41,0x19]
+[0x40,0x04,0x5f,0x19]
+# cpyfe
+[0x5f,0x04,0x81,0x19]
+[0x40,0x04,0x9f,0x19]
+# cpyp
+[0x5f,0x04,0x01,0x1d]
+[0x40,0x04,0x1f,0x1d]
+# cpym
+[0x5f,0x04,0x41,0x1d]
+[0x40,0x04,0x5f,0x1d]
+# cpye
+[0x5f,0x04,0x81,0x1d]
+[0x40,0x04,0x9f,0x1d]
+# setp
+[0x5f,0x04,0xc2,0x19]
+# setm
+[0x5f,0x44,0xc2,0x19]
+# sete
+[0x5f,0x84,0xc2,0x19]
+# setgp
+[0x5f,0x04,0xc2,0x1d]
+# setgm
+[0x5f,0x44,0xc2,0x1d]
+# setge
+[0x5f,0x84,0xc2,0x1d]
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding
+# CHECK-INVALID: warning: invalid instruction encoding