[AMDGPU] Fix delay alu for VOPD with src2acc

[Joe_Nash]

Summary

V_FMAC_F32 and V_DOT2C_F32_F16 have a dummy src2 operand tied to vdst to
inform passes that the instructions read the dst operand. The VOPD
versions of these instructions lacked the dummy operand, which was a
problem for inserting s_delay_alu.
Introduce the dummy src2 operand on the VOPD versions, and fix the VOPD operand
tracking logic to account for it.

[dp]

Overall looks fine, but I want somebody to look at codegen changes. You could leave MC code refactoring to me (but I'm not sure if I'll be able to improve code that much).

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

8567

This may be moved out of the loop (together with CInfo.hasSrc2Acc()).

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp

540

hasRegularSrcOperand looks counterintuitive now because tied src2 is actually a quite special operand. Counting this operand in ComponentProps.SrcOperandsNum is also debatable. Maybe handling this operand explicitly would make code more clear. It will also make error position correct. But it will require changes in parser and codegen which I believe you tried to avoid.

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

580

I believe this change may be avoided by making ComponentLayout a subclass of ComponentProps.

llvm/test/MC/AMDGPU/gfx11_asm_vopd_errs.s

269

It's a pity.

[foad]

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

8567

Maybe const auto & here and on the declaration of InstInfo?

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

563–565

Don't need any of these trailing underscores.

[foad]

Overall looks fine, but I want somebody to look at codegen changes.

The changes in test/CodeGen/AMDGPU/vopd-src2acc-delay.mir are good.

[dp]

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

563–565

You are right, but I'm not sure if ... : Kind(Kind), ... is a good and clear style.

[dp]

LGTM (with or without MC code corrections).

llvm/lib/Target/AMDGPU/VOP2Instructions.td

439

I think these two lines are too long even for td files.

[foad]

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

563–565

It is ubiquitous, and I think it's much better than inventing lots of different conventions (like trailing underscore) to make the argument name different frmo the field name.

[Joe_Nash]

use const auto &, hoist loop invariant code, shorten td line length, delete parameter name underscores

Thanks for the comments. I will plan to land this after rebasing and testing, and leave aside the remaining refactoring opportunities for now.

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp

540

Yes, I have tried to avoid special case code in the parser and codegen. Counting the operand in SrcOperandsNum is indeed debatable but it is a standard tied operand from a CodeGen perspective and the VOPD rules are the same for dst and src2, so it works out cleanly. I'm open to an NFC (except for error caret position) refactor in the future.

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

580

It seems natural to delete ComponentLayout and incorporate its functionaliy into ComponentInfo. I'd prefer that was in a separate patch.

llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp

auto addOp = [&](uint16_t i) { // NOLINT:function pointer
     }
     if (Op.isImm()) {
       Op.addImmOperands(Inst, 1);
       return;
     }
     llvm_unreachable("Unhandled operand type in cvtVOPD");
   };
 
-  auto InstInfo = getVOPDInstInfo(Inst.getOpcode(), &MII);
+  const auto &InstInfo = getVOPDInstInfo(Inst.getOpcode(), &MII);
 
   // MCInst operands are ordered as follows:
   //   dstX, dstY, src0X [, other OpX operands], src0Y [, other OpY operands]
 
   for (auto CompIdx : VOPD::COMPONENTS) {
     addOp(InstInfo[CompIdx].getParsedDstIndex());
   }
 
   for (auto CompIdx : VOPD::COMPONENTS) {
+    const auto &CInfo = InstInfo[CompIdx];
+    bool CompHasSrc2Acc = CInfo.hasSrc2Acc();
     auto SrcOperandsNum = InstInfo[CompIdx].getSrcOperandsNum();
     for (unsigned SrcIdx = 0; SrcIdx < SrcOperandsNum; ++SrcIdx) {
-      addOp(InstInfo[CompIdx].getParsedSrcIndex(SrcIdx));
+      addOp(CInfo.getParsedSrcIndex(SrcIdx, CompHasSrc2Acc));
     }
   }
 }
 
 //===----------------------------------------------------------------------===//
 // dpp
 //===----------------------------------------------------------------------===//
 

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h

private:
   static constexpr unsigned FIRST_MC_SRC_IDX[] = {1, 2, 2 /* + OpXSrcNum */};
 
   // Parsed operands of regular instructions are ordered as follows:
   //   Mnemo dst src0 [vsrc1 ...]
   // Parsed VOPD operands are ordered as follows:
   //   OpXMnemo dstX src0X [vsrc1X|imm vsrc1X|vsrc1X imm] '::'
   //   OpYMnemo dstY src0Y [vsrc1Y|imm vsrc1Y|vsrc1Y imm]
   // Each ComponentKind has operand indices defined below.
-  static constexpr unsigned PARSED_DST_IDX[] = {1, 1, 4 /* + OpXSrcNum */};
-  static constexpr unsigned FIRST_PARSED_SRC_IDX[] = {2, 2,
-                                                      5 /* + OpXSrcNum */};
+  static constexpr unsigned PARSED_DST_IDX[] = {1, 1,
+                                                4 /* + ParsedOpXSrcNum */};
+  static constexpr unsigned FIRST_PARSED_SRC_IDX[] = {
+      2, 2, 5 /* + ParsedOpXSrcNum */};
 
 private:
   ComponentKind Kind;
   unsigned OpXSrcNum;
+  unsigned ParsedOpXSrcNum;
 
 public:
-  ComponentLayout(ComponentKind Kind_ = ComponentKind::SINGLE,
-                  unsigned OpXSrcNum_ = 0)
-      : Kind(Kind_), OpXSrcNum(OpXSrcNum_) {
+  ComponentLayout(ComponentKind Kind = ComponentKind::SINGLE,
+                  unsigned OpXSrcNum = 0, unsigned ParsedOpXSrcNum = 0)
+      : Kind(Kind), OpXSrcNum(OpXSrcNum), ParsedOpXSrcNum(ParsedOpXSrcNum) {
     assert(Kind <= ComponentKind::MAX);
     assert((Kind == ComponentKind::COMPONENT_Y) == (OpXSrcNum > 0));
   }
 
 public:
   unsigned getDstIndex() const { return MC_DST_IDX[Kind]; }
   unsigned getSrcIndex(unsigned SrcIdx) const {
     assert(SrcIdx < Component::MAX_SRC_NUM);
     return FIRST_MC_SRC_IDX[Kind] + OpXSrcNum + SrcIdx;
   }
 
   unsigned getParsedDstIndex() const {
-    return PARSED_DST_IDX[Kind] + OpXSrcNum;
+    return PARSED_DST_IDX[Kind] + ParsedOpXSrcNum;
   }
-  unsigned getParsedSrcIndex(unsigned SrcIdx) const {
+  unsigned getParsedSrcIndex(unsigned SrcIdx, bool ComponentHasSrc2Acc) const {
     assert(SrcIdx < Component::MAX_SRC_NUM);
-    return FIRST_PARSED_SRC_IDX[Kind] + OpXSrcNum + SrcIdx;
+    // FMAC and DOT2C have a src2 operand on the MCInst but
+    // not on the asm representation. src2 is tied to dst.
+    if (ComponentHasSrc2Acc && SrcIdx == (MAX_SRC_NUM - 1))
+      return getParsedDstIndex();
+    return FIRST_PARSED_SRC_IDX[Kind] + ParsedOpXSrcNum + SrcIdx;
   }
 };
 
 // Properties of VOPD components.
 class ComponentProps {
 private:
   unsigned SrcOperandsNum;
   Optional<unsigned> MandatoryLiteralIdx;
private:
   }
 };
 
 // Layout and properties of VOPD components.
 class ComponentInfo : public ComponentLayout, public ComponentProps {
 public:
   ComponentInfo(const MCInstrDesc &OpDesc,
                 ComponentKind Kind = ComponentKind::SINGLE,
-                unsigned OpXSrcNum = 0)
-      : ComponentLayout(Kind, OpXSrcNum), ComponentProps(OpDesc) {}
+                unsigned OpXSrcNum = 0, unsigned ParsedOpXSrcNum = 0)
+      : ComponentLayout(Kind, OpXSrcNum, ParsedOpXSrcNum),
+        ComponentProps(OpDesc) {}
 
   // Map MC operand index to parsed operand index.
   // Return 0 if the specified operand does not exist.
   unsigned getParsedOperandIndex(unsigned OprIdx) const;
 };
 
 // Properties of VOPD instructions.
 class InstInfo {

llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp

ComponentProps::ComponentProps(const MCInstrDesc &OpDesc) {
   assert(OpDesc.getNumDefs() == Component::DST_NUM);
 
   assert(OpDesc.getOperandConstraint(Component::SRC0, MCOI::TIED_TO) == -1);
   assert(OpDesc.getOperandConstraint(Component::SRC1, MCOI::TIED_TO) == -1);
   auto TiedIdx = OpDesc.getOperandConstraint(Component::SRC2, MCOI::TIED_TO);
   assert(TiedIdx == -1 || TiedIdx == Component::DST);
   HasSrc2Acc = TiedIdx != -1;
 
-  SrcOperandsNum = OpDesc.getNumOperands() - OpDesc.getNumDefs() - HasSrc2Acc;
+  SrcOperandsNum = OpDesc.getNumOperands() - OpDesc.getNumDefs();
   assert(SrcOperandsNum <= Component::MAX_SRC_NUM);
 
-  auto OperandsNum = OpDesc.getNumOperands() - HasSrc2Acc;
+  auto OperandsNum = OpDesc.getNumOperands();
   for (unsigned OprIdx = Component::SRC1; OprIdx < OperandsNum; ++OprIdx) {
     if (OpDesc.OpInfo[OprIdx].OperandType == AMDGPU::OPERAND_KIMM32) {
       MandatoryLiteralIdx = OprIdx;
       break;
     }
   }
 }
 
 unsigned ComponentInfo::getParsedOperandIndex(unsigned OprIdx) const {
   assert(OprIdx < Component::MAX_OPR_NUM);
 
   if (OprIdx == Component::DST)
     return getParsedDstIndex();
 
   auto SrcIdx = OprIdx - Component::DST_NUM;
   if (SrcIdx < getSrcOperandsNum())
-    return getParsedSrcIndex(SrcIdx);
+    return getParsedSrcIndex(SrcIdx, hasSrc2Acc());
 
   // The specified operand does not exist.
   return 0;
 }
 
 Optional<unsigned> InstInfo::getInvalidOperandIndex(
     std::function<unsigned(unsigned, unsigned)> GetRegIdx) const {
 
InstInfo::RegIndices InstInfo::getRegIndices(
   unsigned DstReg = GetRegIdx(ComponentIdx, Comp.getDstIndex());
   unsigned Src0Reg = GetRegIdx(ComponentIdx, Comp.getSrcIndex(0));
 
   unsigned Src1Reg = 0;
   if (Comp.hasRegularSrcOperand(1))
     Src1Reg = GetRegIdx(ComponentIdx, Comp.getSrcIndex(1));
 
   unsigned Src2Reg = 0;
   if (Comp.hasRegularSrcOperand(2))
     Src2Reg = GetRegIdx(ComponentIdx, Comp.getSrcIndex(2));
-  else if (Comp.hasSrc2Acc())
-    Src2Reg = DstReg;
 
   return {DstReg, Src0Reg, Src1Reg, Src2Reg};
 }
 
 } // namespace VOPD
 
 VOPD::InstInfo getVOPDInstInfo(const MCInstrDesc &OpX, const MCInstrDesc &OpY) {
   return VOPD::InstInfo(OpX, OpY);
 }
 
 VOPD::InstInfo getVOPDInstInfo(unsigned VOPDOpcode,
                                const MCInstrInfo *InstrInfo) {
   auto [OpX, OpY] = getVOPDComponents(VOPDOpcode);
   const auto &OpXDesc = InstrInfo->get(OpX);
   const auto &OpYDesc = InstrInfo->get(OpY);
   VOPD::ComponentInfo OpXInfo(OpXDesc, VOPD::ComponentKind::COMPONENT_X);
-  VOPD::ComponentInfo OpYInfo(OpYDesc, VOPD::ComponentKind::COMPONENT_Y,
-                              OpXInfo.getSrcOperandsNum());
+  VOPD::ComponentInfo OpYInfo(
+      OpYDesc, VOPD::ComponentKind::COMPONENT_Y, OpXInfo.getSrcOperandsNum(),
+      OpXInfo.getSrcOperandsNum() - OpXInfo.hasSrc2Acc());
   return VOPD::InstInfo(OpXInfo, OpYInfo);
 }
 
 namespace IsaInfo {
 
 AMDGPUTargetID::AMDGPUTargetID(const MCSubtargetInfo &STI)
     : STI(STI), XnackSetting(TargetIDSetting::Any),
       SramEccSetting(TargetIDSetting::Any) {

llvm/lib/Target/AMDGPU/VOP2Instructions.td

let InsDPP = (ins Src0ModDPP:$src0_modifiers, Src0DPP:$src0,
                     Src1ModDPP:$src1_modifiers, Src1DPP:$src1,
                     getVregSrcForVT<Src2VT>.ret:$src2, // stub argument
                     dpp_ctrl:$dpp_ctrl, row_mask:$row_mask,
                     bank_mask:$bank_mask, bound_ctrl:$bound_ctrl);
   let InsDPP16 = !con(InsDPP, (ins FI:$fi));
   let InsVOP3Base  = getIns64<Src0VOP3DPP, Src1RC64, RegisterOperand<VGPR_32>, 3,
                        0, HasModifiers, HasModifiers, HasOMod,
                        Src0Mod, Src1Mod, Src2Mod>.ret;
+  // We need a dummy src2 tied to dst to track the use of that register for s_delay_alu
+  let InsVOPDX = (ins Src0RC32:$src0X, Src1RC32:$vsrc1X, VGPRSrc_32:$src2X);
+  let InsVOPDXDeferred =
+    (ins !if(!eq(Src0VT.Size, 32), VSrc_f32_Deferred, VSrc_f16_Deferred):$src0X,
+         VGPR_32:$vsrc1X, VGPRSrc_32:$src2X);
+  let InsVOPDY = (ins Src0RC32:$src0Y, Src1RC32:$vsrc1Y, VGPRSrc_32:$src2Y);
+  let InsVOPDYDeferred =
+    (ins !if(!eq(Src1VT.Size, 32), VSrc_f32_Deferred, VSrc_f16_Deferred):$src0Y,
+         VGPR_32:$vsrc1Y, VGPRSrc_32:$src2Y);
 
   let InsDPP8 = (ins Src0ModDPP:$src0_modifiers, Src0DPP:$src0,
                      Src1ModDPP:$src1_modifiers, Src1DPP:$src1,
                      getVregSrcForVT<Src2VT>.ret:$src2, // stub argument
                      dpp8:$dpp8, FI:$fi);
   let InsSDWA = (ins Src0ModSDWA:$src0_modifiers, Src0SDWA:$src0,
                      Src1ModSDWA:$src1_modifiers, Src1SDWA:$src1,
                      getVregSrcForVT<Src2VT>.ret:$src2, // stub argument
                      clampmod:$clamp, omod:$omod,

llvm/lib/Target/AMDGPU/VOPDInstructions.td

class VOPD_Base<dag outs, dag ins, string asm, VOP_Pseudo VDX, VOP_Pseudo VDY,
   let WaveSizePredicate = isWave32;
   let isCodeGenOnly = 0;
   let SubtargetPredicate = isGFX11Plus;
   let AsmMatchConverter  = "cvtVOPD";
   let Size = 8;
   let ReadsModeReg = !or(VDX.ReadsModeReg, VDY.ReadsModeReg);
   let mayRaiseFPException = ReadsModeReg;
 
+  // V_DUAL_FMAC and V_DUAL_DOT2ACC_F32_F16 need a dummy src2 tied to dst for
+  // passes to track its uses. Its presence does not affect VOPD formation rules
+  // because the rules for src2 and dst are the same. src2X and src2Y should not
+  // be encoded.
+  bit hasSrc2AccX = !or(!eq(VDX.Mnemonic, "v_fmac_f32"), !eq(VDX.Mnemonic, "v_dot2c_f32_f16"));
+  bit hasSrc2AccY = !or(!eq(VDY.Mnemonic, "v_fmac_f32"), !eq(VDY.Mnemonic, "v_dot2c_f32_f16"));
+  string ConstraintsX = !if(hasSrc2AccX, "$src2X = $vdstX", "");
+  string ConstraintsY = !if(hasSrc2AccY, "$src2Y = $vdstY", "");
+  let Constraints =
+      ConstraintsX # !if(!and(hasSrc2AccX, hasSrc2AccY), ", ", "") # ConstraintsY;
+  string DisableEncodingX = !if(hasSrc2AccX, "$src2X", "");
+  string DisableEncodingY = !if(hasSrc2AccY, "$src2Y", "");
+  let DisableEncoding =
+      DisableEncodingX # !if(!and(hasSrc2AccX, hasSrc2AccY), ", ", "") # DisableEncodingY;
+
   let Uses = RegListUnion<VDX.Uses, VDY.Uses>.ret;
   let Defs = RegListUnion<VDX.Defs, VDY.Defs>.ret;
   let SchedRW = !listconcat(VDX.SchedRW, VDY.SchedRW);
 }
 
 class VOPD<dag outs, dag ins, string asm, VOP_Pseudo VDX, VOP_Pseudo VDY,
            VOPD_Component XasVC, VOPD_Component YasVC>
     : VOPD_Base<outs, ins, asm, VDX, VDY, XasVC, YasVC>,

llvm/test/CodeGen/AMDGPU/vopd-combine.mir

bb.0:
     ; SCHED-NEXT: $vgpr3 = IMPLICIT_DEF
     ; SCHED-NEXT: $vgpr5 = V_FMAMK_F32 killed $vgpr0, 10, killed $vgpr3, implicit $mode, implicit $exec
     ; SCHED-NEXT: $vgpr2 = V_FMAC_F32_e32 killed $vgpr1, $vgpr1, killed $vgpr2, implicit $mode, implicit $exec
     ; PAIR-LABEL: name: vopd_fmamk
     ; PAIR: $vgpr2 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr0 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr1 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr3 = IMPLICIT_DEF
-    ; PAIR-NEXT: $vgpr5, $vgpr2 = V_DUAL_FMAMK_F32_X_FMAC_F32_e32 killed $vgpr0, 10, killed $vgpr3, killed $vgpr1, $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    ; PAIR-NEXT: $vgpr5, $vgpr2 = V_DUAL_FMAMK_F32_X_FMAC_F32_e32 killed $vgpr0, 10, killed $vgpr3, killed $vgpr1, $vgpr1, killed $vgpr2, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
     $vgpr0 = IMPLICIT_DEF
     $vgpr1 = IMPLICIT_DEF
     $vgpr2 = IMPLICIT_DEF
     $vgpr3 = IMPLICIT_DEF
     ; should pair
     $vgpr2 = V_FMAC_F32_e32 $vgpr1, $vgpr1, $vgpr2, implicit $mode, implicit $exec
     $vgpr5 = V_FMAMK_F32 $vgpr0, 10, $vgpr3, implicit $mode, implicit $exec
 
bb.0:
     ; SCHED-NEXT: $vgpr9 = V_CNDMASK_B32_e32 killed $sgpr20, killed $vgpr3, implicit $mode, implicit $exec, implicit $vcc
     ; PAIR-LABEL: name: vopd_cndmask
     ; PAIR: $vgpr2 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr0 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr1 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr3 = IMPLICIT_DEF
     ; PAIR-NEXT: $sgpr20 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr4 = V_FMAMK_F32 $sgpr20, 12345, $vgpr3, implicit $mode, implicit $exec
-    ; PAIR-NEXT: $vgpr2, $vgpr5 = V_DUAL_FMAC_F32_e32_X_CNDMASK_B32_e32 $sgpr20, killed $vgpr1, $vgpr0, $vgpr3, implicit $exec, implicit $vcc, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec, implicit $vcc
+    ; PAIR-NEXT: $vgpr2, $vgpr5 = V_DUAL_FMAC_F32_e32_X_CNDMASK_B32_e32 $sgpr20, killed $vgpr1, killed $vgpr2, $vgpr0, $vgpr3, implicit $exec, implicit $vcc, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec, implicit $vcc
     ; PAIR-NEXT: $vgpr7 = V_CNDMASK_B32_e32 killed $vgpr0, $vgpr3, implicit $mode, implicit $exec, implicit $vcc
     ; PAIR-NEXT: $vgpr6 = V_ADD_F32_e32 $sgpr20, $vgpr3, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr9 = V_CNDMASK_B32_e32 killed $sgpr20, killed $vgpr3, implicit $mode, implicit $exec, implicit $vcc
     $vgpr0 = IMPLICIT_DEF
     $vgpr1 = IMPLICIT_DEF
     $vgpr2 = IMPLICIT_DEF
     $vgpr3 = IMPLICIT_DEF
     $sgpr20 = IMPLICIT_DEF
bb.0:
     ; SCHED-NEXT: $vgpr3 = IMPLICIT_DEF
     ; SCHED-NEXT: $vgpr5 = V_FMAMK_F32 killed $vgpr0, 100, killed $vgpr3, implicit $mode, implicit $exec
     ; SCHED-NEXT: $vgpr2 = V_FMAC_F32_e32 4, killed $vgpr1, killed $vgpr2, implicit $mode, implicit $exec
     ; PAIR-LABEL: name: vopd_constants_inlinable
     ; PAIR: $vgpr2 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr0 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr1 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr3 = IMPLICIT_DEF
-    ; PAIR-NEXT: $vgpr5, $vgpr2 = V_DUAL_FMAMK_F32_X_FMAC_F32_e32 killed $vgpr0, 100, killed $vgpr3, 4, killed $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    ; PAIR-NEXT: $vgpr5, $vgpr2 = V_DUAL_FMAMK_F32_X_FMAC_F32_e32 killed $vgpr0, 100, killed $vgpr3, 4, killed $vgpr1, killed $vgpr2, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
     $vgpr0 = IMPLICIT_DEF
     $vgpr1 = IMPLICIT_DEF
     $vgpr2 = IMPLICIT_DEF
     $vgpr3 = IMPLICIT_DEF
     ; can pair since 4 is inlinable
     $vgpr2 = V_FMAC_F32_e32 4, $vgpr1, $vgpr2, implicit $mode, implicit $exec
     $vgpr5 = V_FMAMK_F32 $vgpr0, 100, $vgpr3, implicit $mode, implicit $exec
 
bb.0:
     ; SCHED-NEXT: $vgpr3 = IMPLICIT_DEF
     ; SCHED-NEXT: $vgpr5 = V_FMAMK_F32 killed $vgpr0, 100, killed $vgpr3, implicit $mode, implicit $exec
     ; SCHED-NEXT: $vgpr2 = V_FMAC_F32_e32 100, killed $vgpr1, killed $vgpr2, implicit $mode, implicit $exec
     ; PAIR-LABEL: name: vopd_constants_same
     ; PAIR: $vgpr2 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr0 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr1 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr3 = IMPLICIT_DEF
-    ; PAIR-NEXT: $vgpr5, $vgpr2 = V_DUAL_FMAMK_F32_X_FMAC_F32_e32 killed $vgpr0, 100, killed $vgpr3, 100, killed $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    ; PAIR-NEXT: $vgpr5, $vgpr2 = V_DUAL_FMAMK_F32_X_FMAC_F32_e32 killed $vgpr0, 100, killed $vgpr3, 100, killed $vgpr1, killed $vgpr2, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
     $vgpr0 = IMPLICIT_DEF
     $vgpr1 = IMPLICIT_DEF
     $vgpr2 = IMPLICIT_DEF
     $vgpr3 = IMPLICIT_DEF
     ; should be able to pair using 1 deduplicated literal
     $vgpr2 = V_FMAC_F32_e32 100, $vgpr1, $vgpr2, implicit $mode, implicit $exec
     $vgpr5 = V_FMAMK_F32 $vgpr0, 100, $vgpr3, implicit $mode, implicit $exec
 
bb.0:
     ; PAIR-NEXT: $vgpr0 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr1 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr20 = IMPLICIT_DEF
     ; PAIR-NEXT: $vgpr16, $vgpr35 = V_DUAL_SUB_F32_e32_X_MUL_F32_e32 $vgpr1, $vgpr1, $vgpr0, $vgpr0, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr3, $vgpr6 = V_DUAL_SUB_F32_e32_X_MUL_F32_e32 $vgpr1, $vgpr1, $vgpr0, $vgpr0, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr2 = V_FMAC_F32_e32 10, $vgpr1, killed $vgpr2, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr2 = V_ADD_F32_e32 $vgpr1, $vgpr1, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr4, $vgpr29 = V_DUAL_SUB_F32_e32_X_CNDMASK_B32_e32 $vgpr1, $vgpr1, $vgpr0, $vgpr3, implicit $exec, implicit $vcc, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec, implicit $vcc
-    ; PAIR-NEXT: $vgpr19, $vgpr20 = V_DUAL_CNDMASK_B32_e32_X_FMAC_F32_e32 $vgpr0, $vgpr3, 10, $vgpr1, implicit $exec, implicit $mode, implicit $vcc, implicit $mode, implicit $exec, implicit $vcc, implicit $mode, implicit $exec
+    ; PAIR-NEXT: $vgpr19, $vgpr20 = V_DUAL_CNDMASK_B32_e32_X_FMAC_F32_e32 $vgpr0, $vgpr3, 10, $vgpr1, killed $vgpr20, implicit $exec, implicit $mode, implicit $vcc, implicit $mode, implicit $exec, implicit $vcc, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr15 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $mode, implicit $exec, implicit $vcc
     ; PAIR-NEXT: $vgpr10, $vgpr17 = V_DUAL_CNDMASK_B32_e32_X_MUL_F32_e32 $vgpr1, $vgpr2, $vgpr0, $vgpr0, implicit $exec, implicit $mode, implicit $vcc, implicit $mode, implicit $exec, implicit $vcc, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr11, $vgpr12 = V_DUAL_CNDMASK_B32_e32_X_ADD_F32_e32 $vgpr0, $vgpr3, $vgpr1, $vgpr1, implicit $exec, implicit $mode, implicit $vcc, implicit $mode, implicit $exec, implicit $vcc, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr37, $vgpr14 = V_DUAL_CNDMASK_B32_e32_X_SUB_F32_e32 $vgpr0, killed $vgpr3, $vgpr1, $vgpr1, implicit $exec, implicit $mode, implicit $vcc, implicit $mode, implicit $exec, implicit $vcc, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr20 = V_ADD_F32_e32 $vgpr1, $vgpr1, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr21, $vgpr24 = V_DUAL_SUB_F32_e32_X_MUL_F32_e32 $vgpr1, $vgpr1, killed $vgpr0, $vgpr0, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
     ; PAIR-NEXT: $vgpr28 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $mode, implicit $exec, implicit $vcc
     ; PAIR-NEXT: $vgpr22 = V_SUB_F32_e32 $vgpr1, $vgpr1, implicit $mode, implicit $exec

llvm/test/CodeGen/AMDGPU/vopd-fmac-delay.mir

-# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
-# RUN: llc -march=amdgcn -mcpu=gfx1100 -verify-machineinstrs -run-pass=gcn-create-vopd,amdgpu-insert-delay-alu %s -o - | FileCheck %s
-
-# FIXME: Second VOPD pair reads vgpr0 and vgpr1 written by first pair, so there
-# should be a delay.
----
-name: vopd_fmac_fmac
-tracksRegLiveness: true
-body: |
-  bb.0:
-    ; CHECK-LABEL: name: vopd_fmac_fmac
-    ; CHECK: $vgpr0 = IMPLICIT_DEF
-    ; CHECK-NEXT: $vgpr1 = IMPLICIT_DEF
-    ; CHECK-NEXT: $vgpr2 = IMPLICIT_DEF
-    ; CHECK-NEXT: $vgpr3 = IMPLICIT_DEF
-    ; CHECK-NEXT: $vgpr4 = IMPLICIT_DEF
-    ; CHECK-NEXT: $vgpr0, $vgpr1 = V_DUAL_FMAC_F32_e32_X_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr3, $vgpr4, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
-    ; CHECK-NEXT: $vgpr0, $vgpr1 = V_DUAL_FMAC_F32_e32_X_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr3, $vgpr4, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
-    $vgpr0 = IMPLICIT_DEF
-    $vgpr1 = IMPLICIT_DEF
-    $vgpr2 = IMPLICIT_DEF
-    $vgpr3 = IMPLICIT_DEF
-    $vgpr4 = IMPLICIT_DEF
-    $vgpr0 = V_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr0, implicit $mode, implicit $exec
-    $vgpr1 = V_FMAC_F32_e32 $vgpr3, $vgpr4, $vgpr1, implicit $mode, implicit $exec
-    $vgpr0 = V_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr0, implicit $mode, implicit $exec
-    $vgpr1 = V_FMAC_F32_e32 $vgpr3, $vgpr4, $vgpr1, implicit $mode, implicit $exec
-...

llvm/test/CodeGen/AMDGPU/vopd-src2acc-delay.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=amdgcn -mcpu=gfx1100 -verify-machineinstrs -run-pass=gcn-create-vopd,amdgpu-insert-delay-alu %s -o - | FileCheck %s
+
+---
+name: vopd_fmac_fmac
+tracksRegLiveness: true
+body: |
+  bb.0:
+    ; CHECK-LABEL: name: vopd_fmac_fmac
+    ; CHECK: $vgpr0 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr1 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr2 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr3 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr4 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr0, $vgpr1 = V_DUAL_FMAC_F32_e32_X_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr0, $vgpr3, $vgpr4, $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    ; CHECK-NEXT: S_DELAY_ALU 1
+    ; CHECK-NEXT: $vgpr0, $vgpr1 = V_DUAL_FMAC_F32_e32_X_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr0, $vgpr3, $vgpr4, $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    $vgpr0 = IMPLICIT_DEF
+    $vgpr1 = IMPLICIT_DEF
+    $vgpr2 = IMPLICIT_DEF
+    $vgpr3 = IMPLICIT_DEF
+    $vgpr4 = IMPLICIT_DEF
+    $vgpr0 = V_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr0, implicit $mode, implicit $exec
+    $vgpr1 = V_FMAC_F32_e32 $vgpr3, $vgpr4, $vgpr1, implicit $mode, implicit $exec
+    $vgpr0 = V_FMAC_F32_e32 $vgpr2, $vgpr3, $vgpr0, implicit $mode, implicit $exec
+    $vgpr1 = V_FMAC_F32_e32 $vgpr3, $vgpr4, $vgpr1, implicit $mode, implicit $exec
+...
+---
+name: vopd_dot2c_dot2c
+tracksRegLiveness: true
+body: |
+  bb.0:
+    ; CHECK-LABEL: name: vopd_dot2c_dot2c
+    ; CHECK: $vgpr0 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr1 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr2 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr3 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr4 = IMPLICIT_DEF
+    ; CHECK-NEXT: $vgpr0, $vgpr1 = V_DUAL_DOT2C_F32_F16_e32_X_DOT2C_F32_F16_e32 $vgpr2, $vgpr3, $vgpr0, $vgpr3, $vgpr4, $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    ; CHECK-NEXT: S_DELAY_ALU 1
+    ; CHECK-NEXT: $vgpr0, $vgpr1 = V_DUAL_DOT2C_F32_F16_e32_X_DOT2C_F32_F16_e32 $vgpr2, $vgpr3, $vgpr0, $vgpr3, $vgpr4, $vgpr1, implicit $exec, implicit $mode, implicit $mode, implicit $exec, implicit $mode, implicit $exec
+    $vgpr0 = IMPLICIT_DEF
+    $vgpr1 = IMPLICIT_DEF
+    $vgpr2 = IMPLICIT_DEF
+    $vgpr3 = IMPLICIT_DEF
+    $vgpr4 = IMPLICIT_DEF
+    $vgpr0 = V_DOT2C_F32_F16_e32 $vgpr2, $vgpr3, $vgpr0, implicit $mode, implicit $exec
+    $vgpr1 = V_DOT2C_F32_F16_e32 $vgpr3, $vgpr4, $vgpr1, implicit $mode, implicit $exec
+    $vgpr0 = V_DOT2C_F32_F16_e32 $vgpr2, $vgpr3, $vgpr0, implicit $mode, implicit $exec
+    $vgpr1 = V_DOT2C_F32_F16_e32 $vgpr3, $vgpr4, $vgpr1, implicit $mode, implicit $exec
+...

llvm/test/MC/AMDGPU/gfx11_asm_vopd_errs.s

 v_dual_fmac_f32     v7, v1, v2                   :: v_dual_fmamk_f32      v6, v2, 0xaf123456, v3
 // GFX11: error: src2 operands must use different VGPR banks
 // GFX11-NEXT:{{^}}v_dual_fmac_f32     v7, v1, v2                   :: v_dual_fmamk_f32      v6, v2, 0xaf123456, v3
 // GFX11-NEXT:{{^}}                                                                                              ^
 
 v_dual_fmamk_f32    v6, v1, 0xaf123456, v3       :: v_dual_fmac_f32       v5, v2, v3
 // GFX11: error: src2 operands must use different VGPR banks
 // GFX11-NEXT:{{^}}v_dual_fmamk_f32    v6, v1, 0xaf123456, v3       :: v_dual_fmac_f32       v5, v2, v3
-// GFX11-NEXT:{{^}}                                        ^
+// GFX11-NEXT:{{^}}                                                                          ^