[RISCV] Add vendor-defined XTheadMAC (multiply-accumulate) extension

[philipp.tomsich]

Summary

The vendor-defined XTHeadMAC (no comparable standard extension exists
at the time of writing) extension adds multiply accumulate instructions.

It is supported by the C9xx cores (e.g., found in the wild in the
Allwinner D1) by Alibaba T-Head.

The current (as of this commit) public documentation for this
extension is available at:

https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf

Support for these instructions has already landed in GNU Binutils:

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=4041e11db3ec3611921d10150572a92689aa3154

Depends on D143439

[craig.topper]

llvm/lib/Target/RISCV/RISCVFeatures.td

491

Identation is off here

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

96

I think we usually use $rd_wb for the destination on tied instructions

360

You can probably use binop_allwusers<add> instead of (sext_inreg (add ...)). Similar for sub. Then add these instructions to the switch in RISCVDAGToDAGISel::doPeepholeSExtW with ADDIW/ADDW/etc.

363

Also better to use binop_allwusers here.

365

"Zbb" -> "Zbb or THeadBb"?

379

Same here

llvm/test/CodeGen/RISCV/attributes.ll

95

xtheadMAC -> xtheadmac

llvm/test/MC/RISCV/rv32xtheadmac-invalid.s

3

Add {{$}} to the end of the CHECK to make sure there is nothing else listed.

[philipp.tomsich]

• rework for review comments

[philipp.tomsich]

• update comment to not refer to sext_inreg

[philipp.tomsich]

We plan to have a major rework tomorrow:

• introduce a sexti16 complex pattern
• reduce the number of individual patterns (handling the sext_inreg and shl+sra case together)

[philipp.tomsich]

• introduce sexti16 as a complex pattern (matches values that are sign-extended to an i16 and returns the underlying node --- i.e, what is being fed into the sign-extension)
• refactors sexti32 to share implementation with sexti16 (similarily to what is implemented for zexti32)
• simplifies XTHeadMac implementation by matching sexti16 instead of having multiple cases spelled out for shl+sra and sext_inreg

[craig.topper]

LGTM fix those formatting changes.

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

91

I think I missed it in previous reviews, but hasSideEffects should line up with Predicates on the previous line. That's the prevailing style in RISCVInstrInfo.td

94

(ins is over indented

llvm/docs/RISCVUsage.rst

``XTHeadBa``
   LLVM implements `the THeadBa (address-generation) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
 
 ``XTHeadBb``
   LLVM implements `the THeadBb (basic bit-manipulation) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
 
 ``XTHeadBs``
   LLVM implements `the THeadBs (single-bit operations) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
 
+``XTheadMac``
+  LLVM implements `the XTheadMac (multiply-accumulate instructions) vendor-defined instructions specified in <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.2/xthead-2023-01-30-2.2.2.pdf>`_  by T-HEAD of Alibaba.  Instructions are prefixed with `th.` as described in the specification.
+
 ``XTHeadVdot``
   LLVM implements `version 1.0.0 of the THeadV-family custom instructions specification <https://github.com/T-head-Semi/thead-extension-spec/releases/download/2.2.0/xthead-2022-12-04-2.2.0.pdf>`_ by T-HEAD of Alibaba.  All instructions are prefixed with `th.` as described in the specification, and the riscv-toolchain-convention document linked above.
 
 ``XVentanaCondOps``
   LLVM implements `version 1.0.0 of the VTx-family custom instructions specification <https://github.com/ventanamicro/ventana-custom-extensions/releases/download/v1.0.0/ventana-custom-extensions-v1.0.0.pdf>`_ by Ventana Micro Systems.  All instructions are prefixed with `vt.` as described in the specification, and the riscv-toolchain-convention document linked above.  These instructions are only available for riscv64 at this time.
 
 Specification Documents
 =======================
 For ratified specifications, please refer to the `official RISC-V International
 page <https://riscv.org/technical/specifications/>`_.  Make sure to check the
 `wiki for not yet integrated extensions
 <https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions>`_.
 

llvm/docs/ReleaseNotes.rst

 
 * Assembler support for version 1.0.1 of the Zcb extension was added.
 * Zca, Zcf, and Zcd extensions were upgraded to version 1.0.1.
 * vsetvli intrinsics no longer have side effects. They may now be combined,
   moved, deleted, etc. by optimizations.
 * Adds support for the vendor-defined XTHeadBa (address-generation) extension.
 * Adds support for the vendor-defined XTHeadBb (basic bit-manipulation) extension.
 * Adds support for the vendor-defined XTHeadBs (single-bit) extension.
+* Adds support for the vendor-defined XTHeadMac (multiply-accumulate instructions) extension.
 
 Changes to the WebAssembly Backend
 ----------------------------------
 
 * ...
 
 Changes to the Windows Target
 -----------------------------

llvm/lib/Support/RISCVISAInfo.cpp

static const RISCVSupportedExtension SupportedExtensions[] = {
     {"svnapot", RISCVExtensionVersion{1, 0}},
     {"svpbmt", RISCVExtensionVersion{1, 0}},
     {"svinval", RISCVExtensionVersion{1, 0}},
 
     // vendor-defined ('X') extensions
     {"xtheadba", RISCVExtensionVersion{1, 0}},
     {"xtheadbb", RISCVExtensionVersion{1, 0}},
     {"xtheadbs", RISCVExtensionVersion{1, 0}},
+    {"xtheadmac", RISCVExtensionVersion{1, 0}},
     {"xtheadvdot", RISCVExtensionVersion{1, 0}},
     {"xventanacondops", RISCVExtensionVersion{1, 0}},
 };
 
 static const RISCVSupportedExtension SupportedExperimentalExtensions[] = {
     {"zihintntl", RISCVExtensionVersion{0, 2}},
 
     {"zca", RISCVExtensionVersion{1, 0}},

llvm/lib/Target/RISCV/Disassembler/RISCVDisassembler.cpp

if ((Bytes[0] & 0x3) == 0x3) {
     }
     if (STI.getFeatureBits()[RISCV::FeatureVendorXTHeadBs]) {
       LLVM_DEBUG(dbgs() << "Trying XTHeadBs custom opcode table:\n");
       Result = decodeInstruction(DecoderTableTHeadBs32, MI, Insn, Address, this,
                                  STI);
       if (Result != MCDisassembler::Fail)
         return Result;
     }
+    if (STI.getFeatureBits()[RISCV::FeatureVendorXTHeadMac]) {
+      LLVM_DEBUG(dbgs() << "Trying XTHeadMac custom opcode table:\n");
+      Result = decodeInstruction(DecoderTableTHeadMac32, MI, Insn, Address,
+                                 this, STI);
+      if (Result != MCDisassembler::Fail)
+        return Result;
+    }
     if (STI.getFeatureBits()[RISCV::FeatureVendorXTHeadVdot]) {
       LLVM_DEBUG(dbgs() << "Trying XTHeadVdot custom opcode table:\n");
       Result =
           decodeInstruction(DecoderTableTHeadV32, MI, Insn, Address, this, STI);
       if (Result != MCDisassembler::Fail)
         return Result;
     }
 

llvm/lib/Target/RISCV/RISCVFeatures.td

 
 def FeatureVendorXTHeadBs
     : SubtargetFeature<"xtheadbs", "HasVendorXTHeadBs", "true",
                        "'xtheadbs' (T-Head single-bit instructions)">;
 def HasVendorXTHeadBs : Predicate<"Subtarget->hasVendorXTHeadBs()">,
                                   AssemblerPredicate<(all_of FeatureVendorXTHeadBs),
                                   "'xtheadbs' (T-Head single-bit instructions)">;
 
+def FeatureVendorXTHeadMac
+    : SubtargetFeature<"xtheadmac", "HasVendorXTHeadMac", "true",
+                       "'xtheadmac' (T-Head Multiply-Accumulate Instructions)">;
+def HasVendorXTHeadMac : Predicate<"Subtarget->hasVendorXTHeadMac()">,
+                                  AssemblerPredicate<(all_of FeatureVendorXTHeadMac),
+                                  "'xtheadmac' (T-Head Multiply-Accumulate Instructions)">;
+
 def FeatureVendorXTHeadVdot
     : SubtargetFeature<"xtheadvdot", "HasVendorXTHeadVdot", "true",
                        "'xtheadvdot' (T-Head Vector Extensions for Dot)",
                        [FeatureStdExtV]>;
 def HasVendorXTHeadVdot : Predicate<"Subtarget->hasVendorXTHeadVdot()">,
                                     AssemblerPredicate<(all_of FeatureVendorXTHeadVdot),
                                     "'xtheadvdot' (T-Head Vector Extensions for Dot)">;
 

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

 
 let Predicates = [HasVendorXTHeadBb, IsRV64], DecoderNamespace = "THeadBb",
   hasSideEffects = 0, mayLoad = 0, mayStore = 0 in
 class THShiftW_ri<bits<7> funct7, bits<3> funct3, string opcodestr>
     : RVInstIShiftW<funct7, funct3, OPC_CUSTOM_0, (outs GPR:$rd),
                     (ins GPR:$rs1, uimm5:$shamt),
         opcodestr, "$rd, $rs1, $shamt">;
 
+
+let Predicates = [HasVendorXTHeadMac], DecoderNamespace = "THeadMac",
+  hasSideEffects = 0, mayLoad = 0, mayStore = 0, isCommutable = 1 in
+class THMulAccumulate_rr<bits<7> funct7, string opcodestr>
+    : RVInstR<funct7, 0b001, OPC_CUSTOM_0, (outs GPR:$rd_up),
+                    (ins GPR:$rd, GPR:$rs1, GPR:$rs2),
+        opcodestr, "$rd, $rs1, $rs2"> {
+  let Constraints = "$rd_up = $rd";
+}
+
+
 //===----------------------------------------------------------------------===//
 // Combination of instruction classes.
 // Use these multiclasses to define instructions more easily.
 //===----------------------------------------------------------------------===//
 multiclass THVdotVMAQA_VX<string opcodestr, bits<6> funct6> {
   def _VX : THVdotALUrVX<funct6, OPMVX, opcodestr # ".vx">;
 }
 

 } // Predicates = [HasVendorXTHeadBb, IsRV64]
 
 let Predicates = [HasVendorXTHeadBs], DecoderNamespace = "THeadBs" in {
 let IsSignExtendingOpW = 1 in
 def TH_TST : RVBShift_ri<0b10001, 0b001, OPC_CUSTOM_0, "th.tst">,
              Sched<[WriteSingleBitImm, ReadSingleBitImm]>;
 } // Predicates = [HasVendorXTHeadBs]
 
+let Predicates = [HasVendorXTHeadMac] in {
+def TH_MULA  : THMulAccumulate_rr<0b0010000, "th.mula">;
+def TH_MULAH : THMulAccumulate_rr<0b0010100, "th.mulah">;
+def TH_MULS  : THMulAccumulate_rr<0b0010001, "th.muls">;
+def TH_MULSH : THMulAccumulate_rr<0b0010101, "th.mulsh">;
+} // Predicates = [HasVendorXTHeadMac]
+
+let Predicates = [HasVendorXTHeadMac, IsRV64], IsSignExtendingOpW = 1 in {
+def TH_MULAW : THMulAccumulate_rr<0b0010010, "th.mulaw">;
+def TH_MULSW : THMulAccumulate_rr<0b0010011, "th.mulsw">;
+} // Predicates = [HasVendorXTHeadMac, IsRV64]
+
 let Predicates = [HasVendorXTHeadVdot],
     Constraints = "@earlyclobber $vd",
     RVVConstraint = WidenV in {
 defm THVdotVMAQA      : THVdotVMAQA<"th.vmaqa",     0b100000>;
 defm THVdotVMAQAU     : THVdotVMAQA<"th.vmaqau",    0b100010>;
 defm THVdotVMAQASU    : THVdotVMAQA<"th.vmaqasu",   0b100100>;
 defm THVdotVMAQAUS    : THVdotVMAQA_VX<"th.vmaqaus",0b100110>;
 }

 
 let Predicates = [HasVendorXTHeadBs] in {
 def : Pat<(and (srl GPR:$rs1, uimmlog2xlen:$shamt), 1),
           (TH_TST GPR:$rs1, uimmlog2xlen:$shamt)>;
 def : Pat<(seteq (and GPR:$rs1, SingleBitSetMask:$mask), 0),
           (TH_TST (XORI GPR:$rs1, -1), SingleBitSetMask:$mask)>;
 } // Predicates = [HasVendorXTHeadBs]
 
+let Predicates = [HasVendorXTHeadMac] in {
+def : Pat<(add GPR:$rd, (mul GPR:$rs1, GPR:$rs2)), (TH_MULA GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+def : Pat<(sub GPR:$rd, (mul GPR:$rs1, GPR:$rs2)), (TH_MULS GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+} // Predicates = [HasVendorXTHeadMac]
+
+let Predicates = [HasVendorXTHeadMac, IsRV64] in {
+// mulaw, mulsw are available only in RV64.
+def : Pat<(sext_inreg (add GPR:$rd, (mul GPR:$rs1, GPR:$rs2)), i32), (TH_MULAW GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+def : Pat<(sext_inreg (sub GPR:$rd, (mul GPR:$rs1, GPR:$rs2)), i32), (TH_MULSW GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+// mulah, mulsh need an outer (sext_inreg ..., i32) in RV64.
+def : Pat<(sext_inreg (add GPR:$rd, (mul (sext_inreg GPR:$rs1, i16), (sext_inreg GPR:$rs2, i16))), i32), (TH_MULAH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+def : Pat<(sext_inreg (sub GPR:$rd, (mul (sext_inreg GPR:$rs1, i16), (sext_inreg GPR:$rs2, i16))), i32), (TH_MULSH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+// Also try match for mulah, mulsh in the absense of Zbb.
+def : Pat<(sext_inreg (add GPR:$rd, (mul
+            (sra (shl GPR:$rs1, (i64 48)), (i64 48)),
+      (sra (shl GPR:$rs2, (i64 48)), (i64 48)))),
+     i32), (TH_MULAH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+def : Pat<(sext_inreg (sub GPR:$rd, (mul
+            (sra (shl GPR:$rs1, (i64 48)), (i64 48)),
+      (sra (shl GPR:$rs2, (i64 48)), (i64 48)))),
+     i32), (TH_MULSH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+} // Predicates = [HasVendorXTHeadMac, IsRV64]
+
+let Predicates = [HasVendorXTHeadMac, IsRV32] in {
+def : Pat<(add GPR:$rd, (mul (sext_inreg GPR:$rs1, i16), (sext_inreg GPR:$rs2, i16))), (TH_MULAH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+def : Pat<(sub GPR:$rd, (mul (sext_inreg GPR:$rs1, i16), (sext_inreg GPR:$rs2, i16))), (TH_MULSH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+// Also try match for mulah, mulsh in the absense of Zbb.
+def : Pat<(add GPR:$rd, (mul
+            (sra (shl GPR:$rs1, (i32 16)), (i32 16)),
+      (sra (shl GPR:$rs2, (i32 16)), (i32 16)))),
+     (TH_MULAH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+def : Pat<(sub GPR:$rd, (mul
+            (sra (shl GPR:$rs1, (i32 16)), (i32 16)),
+      (sra (shl GPR:$rs2, (i32 16)), (i32 16)))),
+     (TH_MULSH GPR:$rd, GPR:$rs1, GPR:$rs2)>;
+} // Predicates = [HasVendorXTHeadMac, IsRV32]
 
 defm PseudoTHVdotVMAQA      : VPseudoVMAQA_VV_VX;
 defm PseudoTHVdotVMAQAU     : VPseudoVMAQA_VV_VX;
 defm PseudoTHVdotVMAQASU    : VPseudoVMAQA_VV_VX;
 defm PseudoTHVdotVMAQAUS    : VPseudoVMAQA_VX;
 
 let Predicates = [HasVendorXTHeadVdot] in {
 defm : VPatTernaryVMAQA_VV_VX<"int_riscv_th_vmaqa",  "PseudoTHVdotVMAQA",  AllQuadWidenableInt8NoVLMulVectors>;
 defm : VPatTernaryVMAQA_VV_VX<"int_riscv_th_vmaqau", "PseudoTHVdotVMAQAU", AllQuadWidenableInt8NoVLMulVectors>;
 defm : VPatTernaryVMAQA_VV_VX<"int_riscv_th_vmaqasu","PseudoTHVdotVMAQASU",AllQuadWidenableInt8NoVLMulVectors>;
 defm : VPatTernaryVMAQA_VX<"int_riscv_th_vmaqaus",   "PseudoTHVdotVMAQAUS",AllQuadWidenableInt8NoVLMulVectors>;
 }

llvm/test/CodeGen/RISCV/attributes.ll

 ; RUN: llc -mtriple=riscv32 -mattr=+zbkb,+zbkc,+zbkx,+zkne,+zknd,+zknh %s -o - | FileCheck --check-prefixes=CHECK,RV32COMBINEINTOZKN %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zbkb,+zbkc,+zbkx,+zksed,+zksh %s -o - | FileCheck --check-prefixes=CHECK,RV32COMBINEINTOZKS %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicbom %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICBOM %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicboz %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICBOZ %s
 ; RUN: llc -mtriple=riscv32 -mattr=+zicbop %s -o - | FileCheck --check-prefixes=CHECK,RV32ZICBOP %s
 ; RUN: llc -mtriple=riscv32 -mattr=+svnapot %s -o - | FileCheck --check-prefixes=CHECK,RV32SVNAPOT %s
 ; RUN: llc -mtriple=riscv32 -mattr=+svpbmt %s -o - | FileCheck --check-prefixes=CHECK,RV32SVPBMT %s
 ; RUN: llc -mtriple=riscv32 -mattr=+svinval %s -o - | FileCheck --check-prefixes=CHECK,RV32SVINVAL %s
+; RUN: llc -mtriple=riscv32 -mattr=+xtheadmac %s -o - | FileCheck --check-prefixes=CHECK,RV32XTHEADMAC %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zca %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCA %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcb %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCB %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcd %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCD %s
 ; RUN: llc -mtriple=riscv32 -mattr=+experimental-zcf %s -o - | FileCheck --check-prefixes=CHECK,RV32ZCF %s
 
 ; RUN: llc -mtriple=riscv64 %s -o - | FileCheck %s
 ; RUN: llc -mtriple=riscv64 -mattr=+m %s -o - | FileCheck --check-prefixes=CHECK,RV64M %s
 ; RUN: llc -mtriple=riscv64 -mattr=+zmmul %s -o - | FileCheck --check-prefixes=CHECK,RV64ZMMUL %s

 ; RUN: llc -mtriple=riscv64 -mattr=+zicbop %s -o - | FileCheck --check-prefixes=CHECK,RV64ZICBOP %s
 ; RUN: llc -mtriple=riscv64 -mattr=+svnapot %s -o - | FileCheck --check-prefixes=CHECK,RV64SVNAPOT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+svpbmt %s -o - | FileCheck --check-prefixes=CHECK,RV64SVPBMT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+svinval %s -o - | FileCheck --check-prefixes=CHECK,RV64SVINVAL %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xventanacondops %s -o - | FileCheck --check-prefixes=CHECK,RV64XVENTANACONDOPS %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadba %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADBA %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadbb %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADBB %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadbs %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADBS %s
+; RUN: llc -mtriple=riscv64 -mattr=+xtheadMAC %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADMAC %s
 ; RUN: llc -mtriple=riscv64 -mattr=+xtheadvdot %s -o - | FileCheck --check-prefixes=CHECK,RV64XTHEADVDOT %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zawrs %s -o - | FileCheck --check-prefixes=CHECK,RV64ZAWRS %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-ztso %s -o - | FileCheck --check-prefixes=CHECK,RV64ZTSO %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zca %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCA %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcb %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCB %s
 ; RUN: llc -mtriple=riscv64 -mattr=+experimental-zcd %s -o - | FileCheck --check-prefixes=CHECK,RV64ZCD %s
 
 ; CHECK: .attribute 4, 16

 ; RV32COMBINEINTOZKN: .attribute 5, "rv32i2p0_zbkb1p0_zbkc1p0_zbkx1p0_zkn1p0_zknd1p0_zkne1p0_zknh1p0"
 ; RV32COMBINEINTOZKS: .attribute 5, "rv32i2p0_zbkb1p0_zbkc1p0_zbkx1p0_zks1p0_zksed1p0_zksh1p0"
 ; RV32ZICBOM: .attribute 5, "rv32i2p0_zicbom1p0"
 ; RV32ZICBOZ: .attribute 5, "rv32i2p0_zicboz1p0"
 ; RV32ZICBOP: .attribute 5, "rv32i2p0_zicbop1p0"
 ; RV32SVNAPOT: .attribute 5, "rv32i2p0_svnapot1p0"
 ; RV32SVPBMT: .attribute 5, "rv32i2p0_svpbmt1p0"
 ; RV32SVINVAL: .attribute 5, "rv32i2p0_svinval1p0"
+; RV32XTHEADMAC: .attribute 5, "rv32i2p0_xtheadmac1p0"
 ; RV32ZCA: .attribute 5, "rv32i2p0_zca1p0"
 ; RV32ZCB: .attribute 5, "rv32i2p0_zca1p0_zcb1p0"
 ; RV32ZCD: .attribute 5, "rv32i2p0_zcd1p0"
 ; RV32ZCF: .attribute 5, "rv32i2p0_zcf1p0"
 
 ; RV64M: .attribute 5, "rv64i2p0_m2p0"
 ; RV64ZMMUL: .attribute 5, "rv64i2p0_zmmul1p0"
 ; RV64MZMMUL: .attribute 5, "rv64i2p0_m2p0_zmmul1p0"

 ; RV64ZICBOP: .attribute 5, "rv64i2p0_zicbop1p0"
 ; RV64SVNAPOT: .attribute 5, "rv64i2p0_svnapot1p0"
 ; RV64SVPBMT: .attribute 5, "rv64i2p0_svpbmt1p0"
 ; RV64SVINVAL: .attribute 5, "rv64i2p0_svinval1p0"
 ; RV64XVENTANACONDOPS: .attribute 5, "rv64i2p0_xventanacondops1p0"
 ; RV64XTHEADBA: .attribute 5, "rv64i2p0_xtheadba1p0"
 ; RV64XTHEADBB: .attribute 5, "rv64i2p0_xtheadbb1p0"
 ; RV64XTHEADBS: .attribute 5, "rv64i2p0_xtheadbs1p0"
+; RV64XTHEADMAC: .attribute 5, "rv64i2p0_xtheadmac1p0"
 ; RV64XTHEADVDOT: .attribute 5, "rv64i2p0_f2p0_d2p0_v1p0_zve32f1p0_zve32x1p0_zve64d1p0_zve64f1p0_zve64x1p0_zvl128b1p0_zvl32b1p0_zvl64b1p0_xtheadvdot1p0"
 ; RV64ZTSO: .attribute 5, "rv64i2p0_ztso0p1"
 ; RV64ZCA: .attribute 5, "rv64i2p0_zca1p0"
 ; RV64ZCB: .attribute 5, "rv64i2p0_zca1p0_zcb1p0"
 ; RV64ZCD: .attribute 5, "rv64i2p0_zcd1p0"
 
 define i32 @addi(i32 %a) {
   %1 = add i32 %a, 1
   ret i32 %1
 }

llvm/test/CodeGen/RISCV/xtheadmac.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv32 -mattr=+xtheadmac -mattr=+m -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s -check-prefix=RV32XTHEADMAC
+; RUN: llc -mtriple=riscv64 -mattr=+xtheadmac -mattr=+m -verify-machineinstrs < %s \
+; RUN:   | FileCheck %s -check-prefix=RV64XTHEADMAC
+
+define i32 @mula_i32(i32 %a, i32 %b, i32 %c) {
+; RV32XTHEADMAC-LABEL: mula_i32:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    th.mula a0, a1, a2
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: mula_i32:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mulaw a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = mul i32 %b, %c
+  %e = add i32 %a, %d
+  ret i32 %e
+}
+
+define i32 @muls_i32(i32 %a, i32 %b, i32 %c) {
+; RV32XTHEADMAC-LABEL: muls_i32:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    th.muls a0, a1, a2
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: muls_i32:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mulsw a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = mul i32 %b, %c
+  %e = sub i32 %a, %d
+  ret i32 %e
+}
+
+define i64 @mula_i64(i64 %a, i64 %b, i64 %c) {
+; RV32XTHEADMAC-LABEL: mula_i64:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    mulhu a6, a2, a4
+; RV32XTHEADMAC-NEXT:    th.mula a6, a2, a5
+; RV32XTHEADMAC-NEXT:    th.mula a6, a3, a4
+; RV32XTHEADMAC-NEXT:    th.mula a2, a0, a4
+; RV32XTHEADMAC-NEXT:    sltu a0, a2, a0
+; RV32XTHEADMAC-NEXT:    add a0, a1, a0
+; RV32XTHEADMAC-NEXT:    add a1, a0, a6
+; RV32XTHEADMAC-NEXT:    mv a0, a2
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: mula_i64:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mula a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = mul i64 %b, %c
+  %f = add i64 %a, %d
+  ret i64 %f
+}
+
+define i64 @mulaw_i64(i32 %a, i32 %b, i32 %c) {
+; RV32XTHEADMAC-LABEL: mulaw_i64:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    th.mula a0, a1, a2
+; RV32XTHEADMAC-NEXT:    srai a1, a0, 31
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: mulaw_i64:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mulaw a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = mul i32 %b, %c
+  %e = add i32 %a, %d
+  %f = sext i32 %e to i64
+  ret i64 %f
+}
+
+define i64 @mulah_i64(i32 %a, i16 %b, i16 %c) {
+; RV32XTHEADMAC-LABEL: mulah_i64:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    th.mulah a0, a1, a2
+; RV32XTHEADMAC-NEXT:    srai a1, a0, 31
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: mulah_i64:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mulah a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = sext i16 %b to i32
+  %e = sext i16 %c to i32
+  %f = mul i32 %d, %e
+  %g = add i32 %a, %f
+  %h = sext i32 %g to i64
+  ret i64 %h
+}
+
+define i64 @muls_i64(i64 %a, i64 %b, i64 %c) {
+; RV32XTHEADMAC-LABEL: muls_i64:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    mulhu a6, a2, a4
+; RV32XTHEADMAC-NEXT:    th.mula a6, a2, a5
+; RV32XTHEADMAC-NEXT:    th.mula a6, a3, a4
+; RV32XTHEADMAC-NEXT:    mul a3, a2, a4
+; RV32XTHEADMAC-NEXT:    th.muls a2, a0, a4
+; RV32XTHEADMAC-NEXT:    sltu a0, a0, a3
+; RV32XTHEADMAC-NEXT:    sub a1, a1, a0
+; RV32XTHEADMAC-NEXT:    sub a1, a1, a6
+; RV32XTHEADMAC-NEXT:    mv a0, a2
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: muls_i64:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.muls a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = mul i64 %b, %c
+  %f = sub i64 %a, %d
+  ret i64 %f
+}
+
+define i64 @mulsw_i64(i32 %a, i32 %b, i32 %c) {
+; RV32XTHEADMAC-LABEL: mulsw_i64:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    th.muls a0, a1, a2
+; RV32XTHEADMAC-NEXT:    srai a1, a0, 31
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: mulsw_i64:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mulsw a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = mul i32 %b, %c
+  %e = sub i32 %a, %d
+  %f = sext i32 %e to i64
+  ret i64 %f
+}
+
+define i64 @mulsh_i64(i32 %a, i16 %b, i16 %c) {
+; RV32XTHEADMAC-LABEL: mulsh_i64:
+; RV32XTHEADMAC:       # %bb.0:
+; RV32XTHEADMAC-NEXT:    th.mulsh a0, a1, a2
+; RV32XTHEADMAC-NEXT:    srai a1, a0, 31
+; RV32XTHEADMAC-NEXT:    ret
+;
+; RV64XTHEADMAC-LABEL: mulsh_i64:
+; RV64XTHEADMAC:       # %bb.0:
+; RV64XTHEADMAC-NEXT:    th.mulsh a0, a1, a2
+; RV64XTHEADMAC-NEXT:    ret
+  %d = sext i16 %b to i32
+  %e = sext i16 %c to i32
+  %f = mul i32 %d, %e
+  %g = sub i32 %a, %f
+  %h = sext i32 %g to i64
+  ret i64 %h
+}

llvm/test/MC/RISCV/rv32xtheadmac-invalid.s

+# RUN: not llvm-mc -triple riscv32 -mattr=+xtheadmac < %s 2>&1 | FileCheck %s
+
+th.mulaw  t0, t1, t2     # CHECK: :[[@LINE]]:1: error: instruction requires the following: RV64I Base
+th.mulsw  t0, t1, t2     # CHECK: :[[@LINE]]:1: error: instruction requires the following: RV64I Base

llvm/test/MC/RISCV/rv32xtheadmac-valid.s

+# RUN: llvm-mc %s -triple=riscv32 -mattr=+xtheadmac -show-encoding \
+# RUN:     | FileCheck -check-prefixes=CHECK-ASM,CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc -filetype=obj -triple=riscv32 -mattr=+xtheadmac < %s \
+# RUN:     | llvm-objdump --mattr=+xtheadmac -d -r - \
+# RUN:     | FileCheck --check-prefix=CHECK-ASM-AND-OBJ %s
+
+# CHECK-ASM-AND-OBJ: th.mula a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x20]
+th.mula  a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.mulah a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x28]
+th.mulah a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.muls a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x22]
+th.muls  a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.mulsh a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x2a]
+th.mulsh a0, a1, a2

llvm/test/MC/RISCV/rv64xtheadmac-valid.s

+# RUN: llvm-mc %s -triple=riscv64 -mattr=+xtheadmac -show-encoding \
+# RUN:     | FileCheck -check-prefixes=CHECK-ASM,CHECK-ASM-AND-OBJ %s
+# RUN: llvm-mc -filetype=obj -triple=riscv64 -mattr=+xtheadmac < %s \
+# RUN:     | llvm-objdump --mattr=+xtheadmac -d -r - \
+# RUN:     | FileCheck --check-prefix=CHECK-ASM-AND-OBJ %s
+
+# CHECK-ASM-AND-OBJ: th.mula a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x20]
+th.mula  a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.mulah a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x28]
+th.mulah a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.mulaw a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x24]
+th.mulaw a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.muls a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x22]
+th.muls  a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.mulsh a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x2a]
+th.mulsh a0, a1, a2
+
+# CHECK-ASM-AND-OBJ: th.mulsw a0, a1, a2
+# CHECK-ASM: encoding: [0x0b,0x95,0xc5,0x26]
+th.mulsw a0, a1, a2

llvm/test/MC/RISCV/xtheadmac-invalid.s

+# RUN: not llvm-mc -triple riscv32 -mattr=+xtheadmac < %s 2>&1 | FileCheck %s
+# RUN: not llvm-mc -triple riscv64 -mattr=+xtheadmac < %s 2>&1 | FileCheck %s
+
+th.mula  t0, t1, 0(t2)  # CHECK: :[[@LINE]]:18: error: invalid operand for instruction
+th.muls  t0, 12, t2     # CHECK: :[[@LINE]]:14: error: invalid operand for instruction
+th.mulsh t0, t1, 34     # CHECK: :[[@LINE]]:18: error: invalid operand for instruction
+th.mulsw t0, -56, t2    # CHECK: :[[@LINE]]:14: error: invalid operand for instruction