[RISCV] Model all 3 arithmetic sources of vector FMA at MC layer.

[craig.topper]

Summary

For the most part, MC version of vector instructions don't model when
the destination is also a source. This primarily occurs for mask/tail
undisturbed. The MC layer can't see the policy bits so this kind of
makes sense.

We also lumped FMA instructions into this, but the destination of
FMA is an arithmetic source not just an undisturbed value. This needs
to be correct for llvm-mca to understand the dependency for the FMA
instructions. Though every other instruction is still wrong for
tail/mask undisturbed.

This patch models the FMA instructions correctly at the MCA layer.
This necessitates changes to the assembler to offset operand numbers.

We also have to remove hasMergeOp from the pseudo versions. This
property was originally for RISCVAsmPrinter to know to drop the
tail/mask undisturbed value when converting to the MC version. It's
been hijacked by the peepholes in RISCVISelDAGToDAG.cpp too so I had
to make some changes there. I'm going to look at ways to remove this
usage as a follow up.

I've added the extra sched class operand and fixed the operand order
for the scalar read class.

[frasercrmck]

I think cleaning up HasMergeOp makes sense. If we're now adding let HasMergeOp = 0 in various places it's telling me the design of our tablegen classes isn't working well.

llvm/lib/Target/RISCV/RISCVISelDAGToDAG.cpp

3190 ↗

(On Diff #527223)

Do we need the extra parens on this first term (which was there before)?

llvm/lib/Target/RISCV/RISCVInstrInfoZvfbf.td

29

I wonder if it's worth having ExtraConstraints which are automatically appended to Constraints, to avoid us having to duplicate the "inherent" tied operand constraints that the base class should be setting.

I don't know how practical that is given our class hierarchies, just a thought. It might make things harder to understand, not easier.

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoZvfbf.td

29

automatically appended in tablegen itself or somewhere in our class hierarchy?

[craig.topper]

Try to address review comments

[craig.topper]

More updates

[craig.topper]

Remove parentheses from RISCVISelDAGToDAG.cpp

[frasercrmck]

llvm/lib/Target/RISCV/RISCVInstrInfoV.td

1318–1319

Aren't these properties (including WidenV) inherent to VWMAC_MV_V_X and VWMAC_MV_X instructions? I wonder why we're not setting them in the classes themselves.

1421–1422

Same question about properties inherent to VWMAC_FV_V_F

llvm/lib/Target/RISCV/RISCVInstrInfoZvfbf.td

29

It wasn't very well fleshed out, sorry. I had something in mind using in our class hierarchy. I think what you've done in your latest update has assuaged most of my concerns. I'll say what I had in mind anyway though.

I was considering there being inside the base class let Constraints = !strconcat("@earlyclobber $vd_wb", ExtraConstraints) (which defaults to being empty), and then any defs like this one can do let ExtraConstraints = ", $vd = $vd_wb".

The notion of additive constraints makes sense to me, as I don't think changing or removing such things from the base classes is easy to follow, and repetition is more error-prone. But I think it's very specific to the class hierarchies whether this idea actually makes anything simpler or not.

[craig.topper]

Push WidenV constraint into class.
Finish using the EarlyClobber parameter I added to remove constraints.

[craig.topper]

Rebase

[craig.topper]

Rebase on D152039

[craig.topper]

Rebase

[craig.topper]

@frasercrmck does this look ok now?

[craig.topper]

Ping

[craig.topper]

Ping

[rogfer01]

I could not test it fully as now some of the hunks can't be applied but the change LGTM.

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

3081

I'm not sure I get this: were they never commutable in the first place due to their widening behaviour?

[craig.topper]

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

3081

I think this might be left over from when I added an extra parameter to the class so I had to list this explicitly. Now that there is no parameter, we don't need this change.

But I think we should make vector instructions commutable so MachineCSE can swap operands to find CSE opportunities.

[wangpc]

llvm/lib/Target/RISCV/RISCVInstrInfoV.td

625

We may not need comments for arguments since we have supported named arguments.

llvm/lib/Target/RISCV/AsmParser/RISCVAsmParser.cpp

if (MCID.TSFlags & RISCVII::VS2Constraint) {
       if (VCIXDst == VCIXRs2)
         return Error(VCIXDstLoc, "The destination vector register group cannot"
                                  " overlap the source vector register group.");
     }
     return false;
   }
 
   unsigned DestReg = Inst.getOperand(0).getReg();
+  unsigned Offset = 0;
+  int TiedOp = MCID.getOperandConstraint(1, MCOI::TIED_TO);
+  if (TiedOp == 0)
+    Offset = 1;
+
   // Operands[1] will be the first operand, DestReg.
   SMLoc Loc = Operands[1]->getStartLoc();
   if (MCID.TSFlags & RISCVII::VS2Constraint) {
-    unsigned CheckReg = Inst.getOperand(1).getReg();
+    unsigned CheckReg = Inst.getOperand(Offset + 1).getReg();
     if (DestReg == CheckReg)
       return Error(Loc, "The destination vector register group cannot overlap"
                         " the source vector register group.");
   }
-  if ((MCID.TSFlags & RISCVII::VS1Constraint) && (Inst.getOperand(2).isReg())) {
-    unsigned CheckReg = Inst.getOperand(2).getReg();
+  if ((MCID.TSFlags & RISCVII::VS1Constraint) && Inst.getOperand(Offset + 2).isReg()) {
+    unsigned CheckReg = Inst.getOperand(Offset + 2).getReg();
     if (DestReg == CheckReg)
       return Error(Loc, "The destination vector register group cannot overlap"
                         " the source vector register group.");
   }
   if ((MCID.TSFlags & RISCVII::VMConstraint) && (DestReg == RISCV::V0)) {
     // vadc, vsbc are special cases. These instructions have no mask register.
     // The destination register could not be V0.
     if (Opcode == RISCV::VADC_VVM || Opcode == RISCV::VADC_VXM ||

llvm/lib/Target/RISCV/RISCVInstrInfoV.td

 class SchedBinaryMC<string write, string read0, string read1,
                     bit forceMasked = 1>:
   SchedBinary<write, read0, read1, "WorstCase", forceMasked=forceMasked>;
 
 // For instructions with three operands.
 class SchedTernary<string write, string read0, string read1, string read2,
                    string mx, int sew = 0, bit forceMasked = 0>:
   SchedNary<write, [read0, read1, read2], mx, sew, forceMasked>;
+class SchedTernaryMC<string write, string read0, string read1, string read2,
+                     int sew = 0, bit forceMasked = 1>:
+  SchedNary<write, [read0, read1, read2], "WorstCase", sew, forceMasked>;
 
 // For reduction instructions.
 class SchedReduction<string write, string read, string mx, int sew>:
   SchedCommon<[!cast<SchedWrite>(write # "_" # mx # "_E" # sew)],
               !listsplat(!cast<SchedRead>(read), 3), mx, sew>;
 class SchedReductionMC<string write, string readV, string readV0>:
   SchedCommon<[!cast<SchedWrite>(write # "_WorstCase")],
               [!cast<SchedRead>(readV), !cast<SchedRead>(readV0)],

 class VALUmVV<bits<6> funct6, RISCVVFormat opv, string opcodestr>
     : RVInstVV<funct6, opv, (outs VR:$vd),
                 (ins VR:$vs2, VR:$vs1, VMV0:$v0),
                 opcodestr, "$vd, $vs2, $vs1, v0"> {
   let vm = 0;
 }
 
 // op vd, vs1, vs2, vm (reverse the order of vs1 and vs2)
-class VALUrVV<bits<6> funct6, RISCVVFormat opv, string opcodestr>
-    : RVInstVV<funct6, opv, (outs VR:$vd),
-                (ins VR:$vs1, VR:$vs2, VMaskOp:$vm),
-                opcodestr, "$vd, $vs1, $vs2$vm">;
+class VALUrVV<bits<6> funct6, RISCVVFormat opv, string opcodestr,
+              bit EarlyClobber = 0>
+    : RVInstVV<funct6, opv, (outs VR:$vd_wb),
+                (ins VR:$vd, VR:$vs1, VR:$vs2, VMaskOp:$vm),
+                opcodestr, "$vd, $vs1, $vs2$vm"> {
+  let Constraints = !if(EarlyClobber, "@earlyclobber $vd_wb, $vd = $vd_wb",
+                                      "$vd = $vd_wb");
+}
 
 // op vd, vs2, vs1
 class VALUVVNoVm<bits<6> funct6, RISCVVFormat opv, string opcodestr>
     : RVInstVV<funct6, opv, (outs VR:$vd),
                (ins VR:$vs2, VR:$vs1),
                opcodestr, "$vd, $vs2, $vs1"> {
   let vm = 1;
 }
 
 // op vd, vs2, rs1, vm
 class VALUVX<bits<6> funct6, RISCVVFormat opv, string opcodestr>
     : RVInstVX<funct6, opv, (outs VR:$vd),
                 (ins VR:$vs2, GPR:$rs1, VMaskOp:$vm),
                 opcodestr, "$vd, $vs2, $rs1$vm">;
 
 // op vd, vs2, rs1, v0 (without mask, use v0 as carry input)
 class VALUmVX<bits<6> funct6, RISCVVFormat opv, string opcodestr>
     : RVInstVX<funct6, opv, (outs VR:$vd),
                 (ins VR:$vs2, GPR:$rs1, VMV0:$v0),
                 opcodestr, "$vd, $vs2, $rs1, v0"> {
   let vm = 0;
 }
 
 // op vd, rs1, vs2, vm (reverse the order of rs1 and vs2)
-class VALUrVX<bits<6> funct6, RISCVVFormat opv, string opcodestr>
-    : RVInstVX<funct6, opv, (outs VR:$vd),
-                (ins GPR:$rs1, VR:$vs2, VMaskOp:$vm),
-                opcodestr, "$vd, $rs1, $vs2$vm">;
+class VALUrVX<bits<6> funct6, RISCVVFormat opv, string opcodestr,
+              bit EarlyClobber = 0>
+    : RVInstVX<funct6, opv, (outs VR:$vd_wb),
+                (ins VR:$vd, GPR:$rs1, VR:$vs2, VMaskOp:$vm),
+                opcodestr, "$vd, $rs1, $vs2$vm"> {
+  let Constraints = !if(EarlyClobber, "@earlyclobber $vd_wb, $vd = $vd_wb",
+                                      "$vd = $vd_wb");
+}
 
 // op vd, vs1, vs2
 class VALUVXNoVm<bits<6> funct6, RISCVVFormat opv, string opcodestr>
     : RVInstVX<funct6, opv, (outs VR:$vd),
                (ins VR:$vs2, GPR:$rs1),
                opcodestr, "$vd, $vs2, $rs1"> {
   let vm = 1;
 }

 
 // op vd, vs2, rs1, vm (Float)
 class VALUVF<bits<6> funct6, RISCVVFormat opv, string opcodestr>
     : RVInstVX<funct6, opv, (outs VR:$vd),
                 (ins VR:$vs2, FPR32:$rs1, VMaskOp:$vm),
                 opcodestr, "$vd, $vs2, $rs1$vm">;
 
 // op vd, rs1, vs2, vm (Float) (with mask, reverse the order of rs1 and vs2)
-class VALUrVF<bits<6> funct6, RISCVVFormat opv, string opcodestr>
-    : RVInstVX<funct6, opv, (outs VR:$vd),
-                (ins FPR32:$rs1, VR:$vs2, VMaskOp:$vm),
-                opcodestr, "$vd, $rs1, $vs2$vm">;
+class VALUrVF<bits<6> funct6, RISCVVFormat opv, string opcodestr,
+              bit EarlyClobber = 0>
+    : RVInstVX<funct6, opv, (outs VR:$vd_wb),
+                (ins VR:$vd, FPR32:$rs1, VR:$vs2, VMaskOp:$vm),
+                opcodestr, "$vd, $rs1, $vs2$vm"> {
+  let Constraints = !if(EarlyClobber, "@earlyclobber $vd_wb, $vd = $vd_wb",
+                                      "$vd = $vd_wb");
+}
 
 // op vd, vs2, vm (use vs1 as instruction encoding)
 class VALUVs2<bits<6> funct6, bits<5> vs1, RISCVVFormat opv, string opcodestr>
     : RVInstV<funct6, vs1, opv, (outs VR:$vd),
                (ins VR:$vs2, VMaskOp:$vm),
                opcodestr, "$vd, $vs2$vm">;
 
 // op vd, vs2 (use vs1 as instruction encoding)
multiclass VALU_MV_V_X<string opcodestr, bits<6> funct6, string vw> {
   def V  : VALUVV<funct6, OPMVV, opcodestr # "." # vw # "v">,
            SchedBinaryMC<"WriteVIWALUV", "ReadVIWALUV", "ReadVIWALUV">;
   def X  : VALUVX<funct6, OPMVX, opcodestr # "." # vw # "x">,
            SchedBinaryMC<"WriteVIWALUX", "ReadVIWALUV", "ReadVIWALUX">;
 }
 
 multiclass VMAC_MV_V_X<string opcodestr, bits<6> funct6> {
   def V : VALUrVV<funct6, OPMVV, opcodestr # ".vv">,
-          SchedBinaryMC<"WriteVIMulAddV", "ReadVIMulAddV", "ReadVIMulAddV">;
+          SchedTernaryMC<"WriteVIMulAddV", "ReadVIMulAddV", "ReadVIMulAddV",
+                         "ReadVIMulAddV">;
   def X : VALUrVX<funct6, OPMVX, opcodestr # ".vx">,
-          SchedBinaryMC<"WriteVIMulAddX", "ReadVIMulAddV", "ReadVIMulAddX">;
+          SchedTernaryMC<"WriteVIMulAddX", "ReadVIMulAddV", "ReadVIMulAddX",
+                         "ReadVIMulAddV">;
 }
 
 multiclass VWMAC_MV_X<string opcodestr, bits<6> funct6> {
+  let RVVConstraint = WidenV in
   def X : VALUrVX<funct6, OPMVX, opcodestr # ".vx">,
-          SchedBinaryMC<"WriteVIWMulAddX", "ReadVIWMulAddV", "ReadVIWMulAddX">;
+          SchedTernaryMC<"WriteVIWMulAddX", "ReadVIWMulAddV", "ReadVIWMulAddX",
+                         "ReadVIWMulAddV">;
 }
 
 multiclass VWMAC_MV_V_X<string opcodestr, bits<6> funct6>
    : VWMAC_MV_X<opcodestr, funct6> {
-  def V : VALUrVV<funct6, OPMVV, opcodestr # ".vv">,
-          SchedBinaryMC<"WriteVIWMulAddV", "ReadVIWMulAddV", "ReadVIWMulAddV">;
+  let RVVConstraint = WidenV in
+  def V : VALUrVV<funct6, OPMVV, opcodestr # ".vv", /*EarlyClobber*/1>,
+          SchedTernaryMC<"WriteVIWMulAddV", "ReadVIWMulAddV", "ReadVIWMulAddV",
+                         "ReadVIWMulAddV">;
 }
 
 multiclass VALU_MV_VS2<string opcodestr, bits<6> funct6, bits<5> vs1> {
   def "" : VALUVs2<funct6, vs1, OPMVV, opcodestr>,
            SchedUnaryMC<"WriteVExtV", "ReadVExtV">;
 }
 
 multiclass VMRG_IV_V_X_I<string opcodestr, bits<6> funct6> {
multiclass VWMUL_FV_V_F<string opcodestr, bits<6> funct6> {
   def V : VALUVV<funct6, OPFVV, opcodestr # ".vv">,
           SchedBinaryMC<"WriteVFWMulV", "ReadVFWMulV", "ReadVFWMulV">;
   def F : VALUVF<funct6, OPFVF, opcodestr # ".vf">,
           SchedBinaryMC<"WriteVFWMulF", "ReadVFWMulV", "ReadVFWMulF">;
 }
 
 multiclass VMAC_FV_V_F<string opcodestr, bits<6> funct6> {
   def V : VALUrVV<funct6, OPFVV, opcodestr # ".vv">,
-          SchedBinaryMC<"WriteVFMulAddV", "ReadVFMulAddV", "ReadVFMulAddV">;
+          SchedTernaryMC<"WriteVFMulAddV", "ReadVFMulAddV", "ReadVFMulAddV",
+                         "ReadVFMulAddV">;
   def F : VALUrVF<funct6, OPFVF, opcodestr # ".vf">,
-          SchedBinaryMC<"WriteVFMulAddF", "ReadVFMulAddV", "ReadVFMulAddF">;
+          SchedTernaryMC<"WriteVFMulAddF", "ReadVFMulAddV", "ReadVFMulAddF",
+                         "ReadVFMulAddV">;
 }
 
 multiclass VWMAC_FV_V_F<string opcodestr, bits<6> funct6> {
-  def V : VALUrVV<funct6, OPFVV, opcodestr # ".vv">,
-          SchedBinaryMC<"WriteVFWMulAddV", "ReadVFWMulAddV", "ReadVFWMulAddV">;
-  def F : VALUrVF<funct6, OPFVF, opcodestr # ".vf">,
-          SchedBinaryMC<"WriteVFWMulAddF", "ReadVFWMulAddV", "ReadVFWMulAddF">;
+  let RVVConstraint = WidenV in {
+  def V : VALUrVV<funct6, OPFVV, opcodestr # ".vv", /*EarlyClobber*/1>,
+          SchedTernaryMC<"WriteVFWMulAddV", "ReadVFWMulAddV", "ReadVFWMulAddV",
+                         "ReadVFWMulAddV">;
+  def F : VALUrVF<funct6, OPFVF, opcodestr # ".vf", /*EarlyClobber*/1>,
+          SchedTernaryMC<"WriteVFWMulAddF", "ReadVFWMulAddV", "ReadVFWMulAddF",
+                         "ReadVFWMulAddV">;
+  }
 }
 
 multiclass VSQR_FV_VS2<string opcodestr, bits<6> funct6, bits<5> vs1> {
   def "" : VALUVs2<funct6, vs1, OPFVV, opcodestr>,
            SchedUnaryMC<"WriteVFSqrtV", "ReadVFSqrtV">;
 }
 
 multiclass VRCP_FV_VS2<string opcodestr, bits<6> funct6, bits<5> vs1> {

 } // Constraints = "@earlyclobber $vd", RVVConstraint = WidenV
 
 // Vector Single-Width Integer Multiply-Add Instructions
 defm VMACC_V : VMAC_MV_V_X<"vmacc", 0b101101>;
 defm VNMSAC_V : VMAC_MV_V_X<"vnmsac", 0b101111>;
 defm VMADD_V : VMAC_MV_V_X<"vmadd", 0b101001>;
 defm VNMSUB_V : VMAC_MV_V_X<"vnmsub", 0b101011>;
 
 // Vector Widening Integer Multiply-Add Instructions
-let Constraints = "@earlyclobber $vd", RVVConstraint = WidenV in {
 defm VWMACCU_V : VWMAC_MV_V_X<"vwmaccu", 0b111100>;
 defm VWMACC_V : VWMAC_MV_V_X<"vwmacc", 0b111101>;
 defm VWMACCSU_V : VWMAC_MV_V_X<"vwmaccsu", 0b111111>;
 defm VWMACCUS_V : VWMAC_MV_X<"vwmaccus", 0b111110>;
-} // Constraints = "@earlyclobber $vd", RVVConstraint = WidenV
 
 // Vector Integer Merge Instructions
 defm VMERGE_V : VMRG_IV_V_X_I<"vmerge", 0b010111>;
 
 // Vector Integer Move Instructions
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0, vs2 = 0, vm = 1,
     RVVConstraint = NoConstraint  in {
 // op vd, vs1

 defm VFMSAC_V : VMAC_FV_V_F<"vfmsac", 0b101110>;
 defm VFNMSAC_V : VMAC_FV_V_F<"vfnmsac", 0b101111>;
 defm VFMADD_V : VMAC_FV_V_F<"vfmadd", 0b101000>;
 defm VFNMADD_V : VMAC_FV_V_F<"vfnmadd", 0b101001>;
 defm VFMSUB_V : VMAC_FV_V_F<"vfmsub", 0b101010>;
 defm VFNMSUB_V : VMAC_FV_V_F<"vfnmsub", 0b101011>;
 }
 
 // Vector Widening Floating-Point Fused Multiply-Add Instructions
-let Constraints = "@earlyclobber $vd", RVVConstraint = WidenV,
-    Uses = [FRM], mayRaiseFPException = true in {
+let Uses = [FRM], mayRaiseFPException = true in {
 defm VFWMACC_V : VWMAC_FV_V_F<"vfwmacc", 0b111100>;
 defm VFWNMACC_V : VWMAC_FV_V_F<"vfwnmacc", 0b111101>;
 defm VFWMSAC_V : VWMAC_FV_V_F<"vfwmsac", 0b111110>;
 defm VFWNMSAC_V : VWMAC_FV_V_F<"vfwnmsac", 0b111111>;
 } // Constraints = "@earlyclobber $vd", RVVConstraint = WidenV, Uses = [FRM], mayRaiseFPException = true
 
 // Vector Floating-Point Square-Root Instruction
 let Uses = [FRM], mayRaiseFPException = true in {

llvm/lib/Target/RISCV/RISCVInstrInfoVPseudos.td

multiclass VPseudoTernaryV_VF_AAXA_RM<LMULInfo m, FPR_Info f, string Constraint = ""> {
   defm "_V" # f.FX : VPseudoTernaryWithPolicyRoundingMode<m.vrclass, f.fprclass,
                                                           m.vrclass, m, Constraint,
                                                           Commutable=1>;
 }
 
 multiclass VPseudoTernaryW_VV<LMULInfo m> {
   defvar constraint = "@earlyclobber $rd";
   defm _VV : VPseudoTernaryWithPolicy<m.wvrclass, m.vrclass, m.vrclass, m,
                                       constraint>;
 }
 
 multiclass VPseudoTernaryW_VV_RM<LMULInfo m> {
   defvar constraint = "@earlyclobber $rd";
   defm _VV : VPseudoTernaryWithPolicyRoundingMode<m.wvrclass, m.vrclass, m.vrclass, m,
                                                   constraint>;
 }
 

 
 multiclass VPseudoVMAC_VV_VX_AAXA<string Constraint = ""> {
   foreach m = MxList in {
     defvar mx = m.MX;
     defm "" : VPseudoTernaryV_VV_AAXA<m, Constraint>,
               SchedTernary<"WriteVIMulAddV", "ReadVIMulAddV", "ReadVIMulAddV",
                            "ReadVIMulAddV", mx>;
     defm "" : VPseudoTernaryV_VX_AAXA<m, Constraint>,
-              SchedTernary<"WriteVIMulAddX", "ReadVIMulAddV", "ReadVIMulAddV",
-                           "ReadVIMulAddX", mx>;
+              SchedTernary<"WriteVIMulAddX", "ReadVIMulAddV", "ReadVIMulAddX",
+                           "ReadVIMulAddV", mx>;
   }
 }
 
 multiclass VPseudoVMAC_VV_VF_AAXA<string Constraint = ""> {
   foreach m = MxListF in {
     defm "" : VPseudoTernaryV_VV_AAXA<m, Constraint>,
               SchedTernary<"WriteVFMulAddV", "ReadVFMulAddV", "ReadVFMulAddV",
                            "ReadVFMulAddV", m.MX>;
   }
 
   foreach f = FPList in {
     foreach m = f.MxList in {
       defm "" : VPseudoTernaryV_VF_AAXA<m, f, Constraint>,
-                SchedTernary<"WriteVFMulAddF", "ReadVFMulAddV", "ReadVFMulAddV",
-                             "ReadVFMulAddF", m.MX>;
+                SchedTernary<"WriteVFMulAddF", "ReadVFMulAddV", "ReadVFMulAddF",
+                             "ReadVFMulAddV", m.MX>;
     }
   }
 }
 
 multiclass VPseudoVMAC_VV_VF_AAXA_RM<string Constraint = ""> {
   foreach m = MxListF in {
     defm "" : VPseudoTernaryV_VV_AAXA_RM<m, Constraint>,
               SchedTernary<"WriteVFMulAddV", "ReadVFMulAddV", "ReadVFMulAddV",
                            "ReadVFMulAddV", m.MX>;
   }
 
   foreach f = FPList in {
     foreach m = f.MxList in {
       defm "" : VPseudoTernaryV_VF_AAXA_RM<m, f, Constraint>,
-                SchedTernary<"WriteVFMulAddF", "ReadVFMulAddV", "ReadVFMulAddV",
-                             "ReadVFMulAddF", m.MX>;
+                SchedTernary<"WriteVFMulAddF", "ReadVFMulAddV", "ReadVFMulAddF",
+                             "ReadVFMulAddV", m.MX>;
     }
   }
 }
 
 multiclass VPseudoVSLD_VX_VI<Operand ImmType = simm5, string Constraint = ""> {
   foreach m = MxList in {
     defvar mx = m.MX;
     defm "" : VPseudoVSLDV_VX<m, Constraint>,
               SchedTernary<"WriteVISlideX", "ReadVISlideV", "ReadVISlideV",
                            "ReadVISlideX", mx>;
     defm "" : VPseudoVSLDV_VI<ImmType, m, Constraint>,
               SchedBinary<"WriteVISlideI", "ReadVISlideV", "ReadVISlideV", mx>;
   }
 }
 
 multiclass VPseudoVWMAC_VV_VX {
   foreach m = MxListW in {
     defvar mx = m.MX;
     defm "" : VPseudoTernaryW_VV<m>,
               SchedTernary<"WriteVIWMulAddV", "ReadVIWMulAddV", "ReadVIWMulAddV",
                            "ReadVIWMulAddV", mx>;
     defm "" : VPseudoTernaryW_VX<m>,
-              SchedTernary<"WriteVIWMulAddX", "ReadVIWMulAddV", "ReadVIWMulAddV",
-                           "ReadVIWMulAddX", mx>;
+              SchedTernary<"WriteVIWMulAddX", "ReadVIWMulAddV", "ReadVIWMulAddX",
+                           "ReadVIWMulAddV", mx>;
   }
 }
 
 multiclass VPseudoVWMAC_VX {
   foreach m = MxListW in {
     defm "" : VPseudoTernaryW_VX<m>,
-              SchedTernary<"WriteVIWMulAddX", "ReadVIWMulAddV", "ReadVIWMulAddV",
-                           "ReadVIWMulAddX", m.MX>;
+              SchedTernary<"WriteVIWMulAddX", "ReadVIWMulAddV", "ReadVIWMulAddX",
+                           "ReadVIWMulAddV", m.MX>;
   }
 }
 
 multiclass VPseudoVWMAC_VV_VF_RM {
   foreach m = MxListFW in {
     defm "" : VPseudoTernaryW_VV_RM<m>,
               SchedTernary<"WriteVFWMulAddV", "ReadVFWMulAddV",
                            "ReadVFWMulAddV", "ReadVFWMulAddV", m.MX>;
   }
 
   foreach f = FPListW in {
     foreach m = f.MxListFW in {
       defm "" : VPseudoTernaryW_VF_RM<m, f>,
                 SchedTernary<"WriteVFWMulAddF", "ReadVFWMulAddV",
-                             "ReadVFWMulAddV", "ReadVFWMulAddF", m.MX>;
+                             "ReadVFWMulAddF", "ReadVFWMulAddV", m.MX>;
     }
   }
 }
 
 multiclass VPseudoVWMAC_VV_VF_BF_RM {
   foreach m = MxListFW in {
     defvar mx = m.MX;
     defvar WriteVFWMulAddV_MX = !cast<SchedWrite>("WriteVFWMulAddV_" # mx);
     defvar ReadVFWMulAddV_MX = !cast<SchedRead>("ReadVFWMulAddV_" # mx);
  
     defm "" : VPseudoTernaryW_VV_RM<m>,
               Sched<[WriteVFWMulAddV_MX, ReadVFWMulAddV_MX,
                      ReadVFWMulAddV_MX, ReadVFWMulAddV_MX, ReadVMask]>;
   }
  
   foreach f = BFPListW in {
     foreach m = f.MxListFW in {
       defvar mx = m.MX;
       defvar WriteVFWMulAddF_MX = !cast<SchedWrite>("WriteVFWMulAddF_" # mx);
       defvar ReadVFWMulAddV_MX = !cast<SchedRead>("ReadVFWMulAddV_" # mx);
       defvar ReadVFWMulAddF_MX = !cast<SchedRead>("ReadVFWMulAddF_" # mx);
  
       defm "" : VPseudoTernaryW_VF_BF_RM<m, f>,
                 Sched<[WriteVFWMulAddF_MX, ReadVFWMulAddV_MX,
-                       ReadVFWMulAddV_MX, ReadVFWMulAddF_MX, ReadVMask]>;
+                       ReadVFWMulAddF_MX, ReadVFWMulAddV_MX, ReadVMask]>;
     }
   }
 }
 
 multiclass VPseudoVCMPM_VV_VX_VI {
   foreach m = MxList in {
     defvar mx = m.MX;
     defm "" : VPseudoBinaryM_VV<m>,

llvm/lib/Target/RISCV/RISCVInstrInfoXTHead.td

class THInstVdotVX<bits<6> funct6, RISCVVFormat opv, dag outs, dag ins,
     : RVInstVX<funct6, opv, outs, ins, opcodestr, argstr> {
   let Inst{26} = 1;
   let Inst{6-0} = OPC_CUSTOM_0.Value;
   let DecoderNamespace = "XTHeadVdot";
 }
 
 let hasSideEffects = 0, mayLoad = 0, mayStore = 0 in {
 // op vd, vs1, vs2, vm (reverse the order of vs1 and vs2)
-class THVdotALUrVV<bits<6> funct6, RISCVVFormat opv, string opcodestr>
-    : THInstVdotVV<funct6, opv, (outs VR:$vd),
-                   (ins VR:$vs1, VR:$vs2, VMaskOp:$vm),
-                   opcodestr, "$vd, $vs1, $vs2$vm">;
+class THVdotALUrVV<bits<6> funct6, RISCVVFormat opv, string opcodestr,
+                   bit EarlyClobber>
+    : THInstVdotVV<funct6, opv, (outs VR:$vd_wb),
+                   (ins VR:$vd, VR:$vs1, VR:$vs2, VMaskOp:$vm),
+                   opcodestr, "$vd, $vs1, $vs2$vm"> {
+  let Constraints = !if(EarlyClobber, "@earlyclobber $vd_wb, $vd = $vd_wb",
+                                      "$vd = $vd_wb");
+}
 
 // op vd, rs1, vs2, vm (reverse the order of rs1 and vs2)
-class THVdotALUrVX<bits<6> funct6, RISCVVFormat opv, string opcodestr>
-    : THInstVdotVX<funct6, opv, (outs VR:$vd),
-                   (ins GPR:$rs1, VR:$vs2, VMaskOp:$vm),
-                   opcodestr, "$vd, $rs1, $vs2$vm">;
+class THVdotALUrVX<bits<6> funct6, RISCVVFormat opv, string opcodestr,
+                   bit EarlyClobber>
+    : THInstVdotVX<funct6, opv, (outs VR:$vd_wb),
+                   (ins VR:$vd, GPR:$rs1, VR:$vs2, VMaskOp:$vm),
+                   opcodestr, "$vd, $rs1, $vs2$vm"> {
+  let Constraints = !if(EarlyClobber, "@earlyclobber $vd_wb, $vd = $vd_wb",
+                                      "$vd = $vd_wb");
+}
 } // hasSideEffects = 0, mayLoad = 0, mayStore = 0
 
 let Predicates = [HasVendorXTHeadBa], DecoderNamespace = "XTHeadBa",
     hasSideEffects = 0, mayLoad = 0, mayStore = 0 in
 class THShiftALU_rri<bits<3> funct3, string opcodestr>
     : RVInstRBase<funct3, OPC_CUSTOM_0, (outs GPR:$rd),
                   (ins GPR:$rs1, GPR:$rs2, uimm2:$uimm2),
                   opcodestr, "$rd, $rs1, $rs2, $uimm2"> {

 }
 
 //===----------------------------------------------------------------------===//
 // Combination of instruction classes.
 // Use these multiclasses to define instructions more easily.
 //===----------------------------------------------------------------------===//
 
 multiclass THVdotVMAQA_VX<string opcodestr, bits<6> funct6> {
-  def _VX : THVdotALUrVX<funct6, OPMVX, opcodestr # ".vx">;
+  let RVVConstraint = WidenV in
+  def _VX : THVdotALUrVX<funct6, OPMVX, opcodestr # ".vx", /*EarlyClobber*/1>;
 }
 
-multiclass THVdotVMAQA<string opcodestr, bits<6> funct6> {
-  def _VV   : THVdotALUrVV<funct6, OPMVX, opcodestr # ".vv">;
-  defm ""   : THVdotVMAQA_VX<opcodestr, funct6>;
+multiclass THVdotVMAQA<string opcodestr, bits<6> funct6>
+    : THVdotVMAQA_VX<opcodestr, funct6> {
+  let RVVConstraint = WidenV in
+  def _VV   : THVdotALUrVV<funct6, OPMVX, opcodestr # ".vv", /*EarlyClobber*/1>;
 }
 
 //===----------------------------------------------------------------------===//
 // Instructions
 //===----------------------------------------------------------------------===//
 
 let Predicates = [HasVendorXTHeadBa] in
 def TH_ADDSL : THShiftALU_rri<0b001, "th.addsl">,

 let Predicates = [HasVendorXTHeadFMemIdx, HasStdExtD, IsRV64],
     DecoderNamespace = "XTHeadFMemIdx" in {
 def TH_FLURD : THLoadIndexed<FPR64, 0b01110, "th.flurd">,
                Sched<[WriteFLD64, ReadFMemBase]>;
 def TH_FSURD : THStoreIndexed<FPR64, 0b01110, "th.fsurd">,
                Sched<[WriteFST64, ReadFStoreData, ReadFMemBase]>;
 }
 
-let Predicates = [HasVendorXTHeadVdot],
-    Constraints = "@earlyclobber $vd",
-    RVVConstraint = WidenV in {
+let Predicates = [HasVendorXTHeadVdot] in {
 defm THVdotVMAQA      : THVdotVMAQA<"th.vmaqa",     0b100000>;
 defm THVdotVMAQAU     : THVdotVMAQA<"th.vmaqau",    0b100010>;
 defm THVdotVMAQASU    : THVdotVMAQA<"th.vmaqasu",   0b100100>;
 defm THVdotVMAQAUS    : THVdotVMAQA_VX<"th.vmaqaus",0b100110>;
 }
 
 // Associate LMUL with tablegen records of register classes.
 def THVdotV_M1  : LMULInfo<0b000,  8,   VR, VR,   VR,   VR,   VR, "M1">;

llvm/lib/Target/RISCV/RISCVInstrInfoZvfbf.td

 let Predicates = [HasStdExtZvfbfmin], Constraints = "@earlyclobber $vd",
     mayRaiseFPException = true in {
 let RVVConstraint = WidenCvt in
 defm VFWCVTBF16_F_F_V : VWCVTF_FV_VS2<"vfwcvtbf16.f.f.v", 0b010010, 0b01101>;
 let Uses = [FRM] in
 defm VFNCVTBF16_F_F_W : VNCVTF_FV_VS2<"vfncvtbf16.f.f.w", 0b010010, 0b11101>;
 }
 
-let Predicates = [HasStdExtZvfbfwma], Constraints = "@earlyclobber $vd",
+let Predicates = [HasStdExtZvfbfwma],
+    Constraints = "@earlyclobber $vd_wb, $vd = $vd_wb",
     RVVConstraint = WidenV, Uses = [FRM], mayRaiseFPException = true in {
 defm VFWMACCBF16_V : VWMAC_FV_V_F<"vfwmaccbf16", 0b111011>;
 }