[AMDGPU][GFX11] Mitigate VALU mask write hazard

[critson]

Summary

VALU use of an SGPR (pair) as mask followed by SALU write to the
same SGPR can cause incorrect execution of subsequent SALU reads
of the SGPR.

[rampitec]

llvm/lib/Target/AMDGPU/AMDGPU.td

707

You do not really need a feature bit for this, just a subtarget check.

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

2719

You seem to start from the instruction in the middle and have no check for the 3rd instruction. This makes the w/a overrestrictive as far as I understand. This maybe fine, but probably deserves a comment.

2725

A mask can only be a 64 bit register in wave64. You could check the size and bail rather then listing 32 bit registers.

2732

Isn't that less error prone to just check for an implicit vcc use? I assume it shall not be true unless it reads a mask.

2741

As far as I understand this can only be a VCC too.

2804

Call runOnInstruction() on this new instruction.

[critson]

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

2719

Yes, this intentionally does not do a forward check to look for hazard expiry.
From my analysis we rarely reach expiry, <10% of the time in graphics.
So I considered the forward check a waste of compile time.
I will add a comment to explain this.

2725

Mask can only be 64 bit, but SALU write to either 32 bit half is a hazard.

2732

I am not sure I understand?
These instructions always implicitly read VCC.

2741

As above, my understanding is partial overlap is also a hazard.

[critson]

• Address reviewer comments

[rampitec]

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

2719

Thanks for adding the comment. In fact forward search would not work, we do not have such mechanism. There are cases of 3 instruction hazards where we always start from the last instruction, but that would be an overkill here.

2732

I mean instead of listing instruction opcodes it can use a general check is there is an implicit $vcc operand. But OK, if you think this list is comprehensive.

[foad]

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

2732

The _e32 versions have an implicit $vcc, but the _e64 versions do not.

There is still a small problem that the list of instructions here is missing exotic forms like V_CNDMASK_B32_dpp.

[critson]

• Add DPP forms and associated tests

I was hanging on to this while I waited for some performance data.
After reviewing this closely, I think I need to switch the workaround mechanism.

[foad]

Sure, although...

There is still a small problem that the list of instructions here is missing exotic forms like V_CNDMASK_B32_dpp.

... this still bothers me.

[critson]

In D134151#3823059, @foad wrote:

Sure, although...

There is still a small problem that the list of instructions here is missing exotic forms like V_CNDMASK_B32_dpp.

... this still bothers me.

Beyond DPP which other exotic forms are you looking for?

[critson]

• Change mitigation strategy to s_waitcnt_depctr after SALU

[foad]

In D134151#3823083, @critson wrote:

In D134151#3823059, @foad wrote:

Sure, although...

There is still a small problem that the list of instructions here is missing exotic forms like V_CNDMASK_B32_dpp.

... this still bothers me.

Beyond DPP which other exotic forms are you looking for?

The ones I can find at the moment are:

V_ADDC_U32_sdwa
V_CNDMASK_B16_sdwa
V_CNDMASK_B32_sdwa
V_SUBBREV_U32_sdwa
V_SUBB_U32_sdwa

[critson]

In D134151#3823142, @foad wrote:

In D134151#3823083, @critson wrote:

Beyond DPP which other exotic forms are you looking for?

The ones I can find at the moment are:

V_ADDC_U32_sdwa
V_CNDMASK_B16_sdwa
V_CNDMASK_B32_sdwa
V_SUBBREV_U32_sdwa
V_SUBB_U32_sdwa

GFX11 doesn't have SDWA which is why I did not add them.

[foad]

In D134151#3825327, @critson wrote:

In D134151#3823142, @foad wrote:

In D134151#3823083, @critson wrote:

Beyond DPP which other exotic forms are you looking for?

The ones I can find at the moment are:

V_ADDC_U32_sdwa
V_CNDMASK_B16_sdwa
V_CNDMASK_B32_sdwa
V_SUBBREV_U32_sdwa
V_SUBB_U32_sdwa

GFX11 doesn't have SDWA which is why I did not add them.

OK, good point!

llvm/lib/Target/AMDGPU/AMDGPU.td

 def FeatureNSAMaxSize13 : SubtargetFeatureNSAMaxSize<13>;
 
 def FeatureVOPD : SubtargetFeature<"vopd",
   "HasVOPDInsts",
   "true",
   "Has VOPD dual issue wave32 instructions"
 >;
 
+def FeatureVALUMaskWriteHazard : SubtargetFeature<"valu-mask-write-hazard",
+  "HasVALUMaskWriteHazard",
+  "true",
+  "VALU mask read followed by SALU write to same SGPR can cause incorrect SALU execution"
+>;
+
 //===------------------------------------------------------------===//
 // Subtarget Features (options and debugging)
 //===------------------------------------------------------------===//
 
 class FeatureMaxPrivateElementSize<int size> : SubtargetFeature<
   "max-private-element-size-"#size,
   "MaxPrivateElementSize",
   !cast<string>(size),
def FeatureGFX11 : GCNSubtargetFeatureGeneration<"GFX11",
    FeatureGFX11Insts, FeatureVOP3P, FeatureVOPD, FeatureTrue16BitInsts,
    FeatureMovrel, FeatureFastFMAF32, FeatureDPP, FeatureIntClamp,
    FeatureFlatInstOffsets, FeatureFlatGlobalInsts, FeatureFlatScratchInsts,
    FeatureAddNoCarryInsts, FeatureFmaMixInsts,
    FeatureNoSdstCMPX, FeatureVscnt,
    FeatureVOP3Literal, FeatureDPP8, FeatureExtendedImageInsts,
    FeatureNoDataDepHazard, FeaturePkFmacF16Inst,
    FeatureGFX10A16, FeatureFastDenormalF32, FeatureG16,
-   FeatureUnalignedBufferAccess, FeatureUnalignedDSAccess
+   FeatureUnalignedBufferAccess, FeatureUnalignedDSAccess,
+   FeatureVALUMaskWriteHazard
   ]
 >;
 
 class FeatureSet<list<SubtargetFeature> Features_> {
   list<SubtargetFeature> Features = Features_;
 }
 
 def FeatureISAVersion6_0_0 : FeatureSet<[FeatureSouthernIslands,

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.h

private:
   bool fixVcmpxExecWARHazard(MachineInstr *MI);
   bool fixLdsBranchVmemWARHazard(MachineInstr *MI);
   bool fixLdsDirectVALUHazard(MachineInstr *MI);
   bool fixLdsDirectVMEMHazard(MachineInstr *MI);
   bool fixVALUPartialForwardingHazard(MachineInstr *MI);
   bool fixVALUTransUseHazard(MachineInstr *MI);
   bool fixWMMAHazards(MachineInstr *MI);
   bool fixShift64HighRegBug(MachineInstr *MI);
+  bool fixVALUMaskWriteHazard(MachineInstr *MI);
 
   int checkMAIHazards(MachineInstr *MI);
   int checkMAIHazards908(MachineInstr *MI);
   int checkMAIHazards90A(MachineInstr *MI);
   /// Pad the latency between neighboring MFMA instructions with s_nops. The
   /// percentage of wait states to fill with s_nops is specified by the command
   /// line option '-amdgpu-mfma-padding-ratio'.
   ///

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

void GCNHazardRecognizer::fixHazards(MachineInstr *MI) {
   if (ST.hasLdsDirect()) {
     fixLdsDirectVALUHazard(MI);
     fixLdsDirectVMEMHazard(MI);
   }
   fixVALUPartialForwardingHazard(MI);
   fixVALUTransUseHazard(MI);
   fixWMMAHazards(MI);
   fixShift64HighRegBug(MI);
+  fixVALUMaskWriteHazard(MI);
 }
 
 bool GCNHazardRecognizer::fixVcmpxPermlaneHazards(MachineInstr *MI) {
   if (!ST.hasVcmpxPermlaneHazard() || !isPermlane(*MI))
     return false;
 
   const SIInstrInfo *TII = ST.getInstrInfo();
   const SIRegisterInfo *TRI = ST.getRegisterInfo();
bool GCNHazardRecognizer::ShouldPreferAnother(SUnit *SU) {
   if (IsMFMAFn(*MI)) {
     int W = getWaitStatesSince(IsMFMAFn, 16);
     if (MAI)
       return W < (int)TSchedModel.computeInstrLatency(MAI);
   }
 
   return false;
 }
+
+bool GCNHazardRecognizer::fixVALUMaskWriteHazard(MachineInstr *MI) {
+  if (!ST.isWave64())
+    return false;
+  if (!ST.hasVALUMaskWriteHazard())
+    return false;
+  if (!SIInstrInfo::isSALU(*MI))
+    return false;
+
+  const MachineOperand *SDSTOp = TII.getNamedOperand(*MI, AMDGPU::OpName::sdst);
+  if (!SDSTOp || !SDSTOp->isReg())
+    return false;
+
+  const Register HazardReg = SDSTOp->getReg();
+  if (HazardReg == AMDGPU::EXEC ||
+      HazardReg == AMDGPU::EXEC_LO ||
+      HazardReg == AMDGPU::EXEC_HI ||
+      HazardReg == AMDGPU::M0)
+    return false;
+
+  auto IsHazardFn = [HazardReg, this](const MachineInstr &I) {
+    switch (I.getOpcode()) {
+    case AMDGPU::V_ADDC_U32_e32:
+    case AMDGPU::V_CNDMASK_B16_e32:
+    case AMDGPU::V_CNDMASK_B32_e32:
+    case AMDGPU::V_DIV_FMAS_F32_e64:
+    case AMDGPU::V_DIV_FMAS_F64_e64:
+    case AMDGPU::V_SUBB_U32_e32:
+    case AMDGPU::V_SUBBREV_U32_e32:
+      // These implicitly read VCC as mask source.
+      return HazardReg == AMDGPU::VCC ||
+             HazardReg == AMDGPU::VCC_LO ||
+             HazardReg == AMDGPU::VCC_HI;
+    case AMDGPU::V_ADDC_U32_e64:
+    case AMDGPU::V_CNDMASK_B16_e64:
+    case AMDGPU::V_CNDMASK_B32_e64:
+    case AMDGPU::V_SUBB_U32_e64:
+    case AMDGPU::V_SUBBREV_U32_e64: {
+      // Only check mask register overlaps.
+      const MachineOperand *SSRCOp = TII.getNamedOperand(I, AMDGPU::OpName::src2);
+      assert(SSRCOp);
+      return TRI.regsOverlap(SSRCOp->getReg(), HazardReg);
+    }
+    default:
+      return false;
+    }
+  };
+
+  const MachineRegisterInfo &MRI = MF.getRegInfo();
+  auto IsExpiredFn = [&MRI, this](const MachineInstr &I, int) {
+    // VALU access to any SGPR or literal constant other than HazardReg
+    // mitigates hazard. No need to check HazardReg here as this will
+    // only be called when !IsHazardFn.
+    if (!SIInstrInfo::isVALU(I))
+      return false;
+    for (int OpNo = 0, End = I.getNumOperands(); OpNo < End; ++OpNo) {
+      const MachineOperand &Op = I.getOperand(OpNo);
+      if (Op.isReg()) {
+        Register OpReg = Op.getReg();
+        // Only consider uses
+        if (!Op.isUse())
+          continue;
+        // Ignore EXEC
+        if (OpReg == AMDGPU::EXEC ||
+            OpReg == AMDGPU::EXEC_LO ||
+            OpReg == AMDGPU::EXEC_HI)
+          continue;
+        // Ignore all implicit uses except VCC
+        if (Op.isImplicit()) {
+          if (OpReg == AMDGPU::VCC ||
+              OpReg == AMDGPU::VCC_LO ||
+              OpReg == AMDGPU::VCC_HI)
+            return true;
+          continue;
+        }
+        if (TRI.isSGPRReg(MRI, OpReg))
+          return true;
+      } else {
+        const MCInstrDesc &InstDesc = I.getDesc();
+        const MCOperandInfo &OpInfo = InstDesc.OpInfo[OpNo];
+        if (TII.isLiteralConstant(Op, OpInfo))
+          return true;
+      }
+    }
+    return false;
+  };
+
+  // Check for hazard
+  if (::getWaitStatesSince(IsHazardFn, MI, IsExpiredFn) ==
+      std::numeric_limits<int>::max())
+    return false;
+
+  // Add VALU which reads an SGPR before SGPR write to mitigate hazard.
+  // FIXME: we could select VGPR/SGPR used to avoid unintended dependency delays
+  BuildMI(*MI->getParent(), MI, MI->getDebugLoc(),
+          TII.get(AMDGPU::V_XOR3_B32_e64), AMDGPU::VGPR0)
+    .addReg(AMDGPU::VGPR0)
+    .addReg(AMDGPU::SGPR0)
+    .addReg(AMDGPU::SGPR0);
+
+  return true;
+}

llvm/lib/Target/AMDGPU/GCNSubtarget.h

protected:
   bool HasNSAtoVMEMBug = false;
   bool HasNSAClauseBug = false;
   bool HasOffset3fBug = false;
   bool HasFlatSegmentOffsetBug = false;
   bool HasImageStoreD16Bug = false;
   bool HasImageGather4D16Bug = false;
   bool HasVOPDInsts = false;
 
+  bool HasVALUMaskWriteHazard = false;
+
   // Dummy feature to use for assembler in tablegen.
   bool FeatureDisable = false;
 
   SelectionDAGTargetInfo TSInfo;
 private:
   SIInstrInfo InstrInfo;
   SITargetLowering TLInfo;
   SIFrameLowering FrameLowering;
public:
   bool hasLdsDirect() const { return getGeneration() >= GFX11; }
 
   bool hasVALUPartialForwardingHazard() const {
     return getGeneration() >= GFX11;
   }
 
   bool hasVALUTransUseHazard() const { return getGeneration() >= GFX11; }
 
+  bool hasVALUMaskWriteHazard() const { return HasVALUMaskWriteHazard; }
+
   /// Return if operations acting on VGPR tuples require even alignment.
   bool needsAlignedVGPRs() const { return GFX90AInsts; }
 
   /// Return true if the target has the S_PACK_HL_B32_B16 instruction.
   bool hasSPackHL() const { return GFX11Insts; }
 
   /// Return true if the target's EXP instruction has the COMPR flag, which
   /// affects the meaning of the EN (enable) bits.

llvm/test/CodeGen/AMDGPU/valu-mask-write-hazard.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=amdgcn -mcpu=gfx1100 -mattr=+wavefrontsize64 -verify-machineinstrs -run-pass post-RA-hazard-rec -o - %s | FileCheck -check-prefix=GCN %s
+
+---
+name:            mask_hazard_getpc1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_getpc1
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr0_sgpr1 = S_GETPC_B64
+    ; GCN-NEXT: $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    $sgpr0_sgpr1 = S_GETPC_B64
+    $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_getpc2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_getpc2
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    ; GCN-NEXT: BUNDLE implicit-def $sgpr0_sgpr1 {
+    ; GCN-NEXT:   $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT:   $sgpr0_sgpr1 = S_GETPC_B64
+    ; GCN-NEXT:   $sgpr0 = S_ADD_U32 $sgpr0, 4, implicit-def $scc
+    ; GCN-NEXT:   $sgpr1 = S_ADDC_U32 $sgpr1, 12, implicit-def $scc, implicit $scc
+    ; GCN-NEXT: }
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    BUNDLE implicit-def $sgpr0_sgpr1 {
+      $sgpr0_sgpr1 = S_GETPC_B64
+      $sgpr0 = S_ADD_U32 $sgpr0, 4, implicit-def $scc
+      $sgpr1 = S_ADDC_U32 $sgpr1, 12, implicit-def $scc, implicit $scc
+    }
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_vcc1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_vcc1
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_vcc2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_vcc2
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_addc1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_addc1
+    ; GCN: $vgpr1, $vcc = V_ADDC_U32_e64 0, $vgpr1, $sgpr2_sgpr3, 0, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1, $vcc = V_ADDC_U32_e64 0, $vgpr1, $sgpr2_sgpr3, 0, implicit $exec
+    $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_addc2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_addc2
+    ; GCN: $vgpr1 = V_ADDC_U32_e32 0, $vgpr1, implicit-def $vcc, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_ADDC_U32_e32 0, $vgpr1, implicit-def $vcc, implicit $vcc, implicit $exec
+    $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_subb1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subb1
+    ; GCN: $vgpr1, $vcc = V_SUBB_U32_e64 0, $vgpr1, $sgpr2_sgpr3, 0, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1, $vcc = V_SUBB_U32_e64 0, $vgpr1, $sgpr2_sgpr3, 0, implicit $exec
+    $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_subb2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subb2
+    ; GCN: $vgpr1 = V_SUBB_U32_e32 0, $vgpr1, implicit-def $vcc, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_SUBB_U32_e32 0, $vgpr1, implicit-def $vcc, implicit $vcc, implicit $exec
+    $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_subbrev1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subbrev1
+    ; GCN: $vgpr1, $vcc = V_SUBBREV_U32_e64 0, $vgpr1, $sgpr2_sgpr3, 0, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1, $vcc = V_SUBBREV_U32_e64 0, $vgpr1, $sgpr2_sgpr3, 0, implicit $exec
+    $sgpr2_sgpr3 = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_subbrev2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subbrev2
+    ; GCN: $vgpr1 = V_SUBBREV_U32_e32 0, $vgpr1, implicit-def $vcc, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_SUBBREV_U32_e32 0, $vgpr1, implicit-def $vcc, implicit $vcc, implicit $exec
+    $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_div_fmas_f32
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_div_fmas_f32
+    ; GCN: $vgpr0 = V_DIV_FMAS_F32_e64 0, $vgpr1, 0, $vgpr2, 0, $vgpr3, 0, 0, implicit $mode, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr0 = V_DIV_FMAS_F32_e64 0, $vgpr1, 0, $vgpr2, 0, $vgpr3, 0, 0, implicit $mode, implicit $vcc, implicit $exec
+    $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+---
+name:            mask_hazard_div_fmas_f64
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_div_fmas_f64
+    ; GCN: $vgpr0_vgpr1 = V_DIV_FMAS_F64_e64 0, $vgpr0_vgpr1, 0, $vgpr2_vgpr3, 0, $vgpr4_vgpr5, 0, 0, implicit $mode, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr0_vgpr1 = V_DIV_FMAS_F64_e64 0, $vgpr0_vgpr1, 0, $vgpr2_vgpr3, 0, $vgpr4_vgpr5, 0, 0, implicit $mode, implicit $vcc, implicit $exec
+    $vcc = S_CSELECT_B64 -1, 0, implicit $scc
+    S_ENDPGM 0
+...
+
+# Check low word overlap
+---
+name:            mask_hazard_subreg1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subreg1
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr2 = S_MOV_B32 0
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    $sgpr2 = S_MOV_B32 0
+    S_ENDPGM 0
+...
+
+# Check high word overlap
+---
+name:            mask_hazard_subreg2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subreg2
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr3 = S_MOV_B32 0
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    $sgpr3 = S_MOV_B32 0
+    S_ENDPGM 0
+...
+
+# Check multiple subreg overlap
+---
+name:            mask_hazard_subreg3
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subreg3
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr2 = S_MOV_B32 0
+    ; GCN-NEXT: $sgpr3 = S_MOV_B32 0
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    $sgpr2 = S_MOV_B32 0
+    $sgpr3 = S_MOV_B32 0
+    S_ENDPGM 0
+...
+
+# Check vcc_lo overlap
+---
+name:            mask_hazard_subreg4
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subreg4
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc_lo = S_MOV_B32 0
+    ; GCN-NEXT: $sgpr2 = S_MOV_B32 $vcc_lo
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    $vcc_lo = S_MOV_B32 0
+    $sgpr2 = S_MOV_B32 $vcc_lo
+    S_ENDPGM 0
+...
+
+# Check vcc_hi overlap
+---
+name:            mask_hazard_subreg5
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_subreg5
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $vcc_hi = S_MOV_B32 0
+    ; GCN-NEXT: $sgpr2 = S_MOV_B32 $vcc_hi
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    $vcc_hi = S_MOV_B32 0
+    $sgpr2 = S_MOV_B32 $vcc_hi
+    S_ENDPGM 0
+...
+
+# S_WAITCNT does not mitigate hazard
+---
+name:            mask_hazard_waitcnt
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_waitcnt
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr0_sgpr1 = S_GETPC_B64
+    ; GCN-NEXT: $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    S_WAITCNT 0
+    $sgpr0_sgpr1 = S_GETPC_B64
+    $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    S_ENDPGM 0
+...
+
+# Check implicit $exec
+---
+name:            mask_hazard_gap1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_gap1
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr2 = V_MOV_B32_e32 0, implicit $exec
+    ; GCN-NEXT: $vgpr3 = V_MOV_B32_e32 0, implicit $exec
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr0_sgpr1 = S_GETPC_B64
+    ; GCN-NEXT: $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    $vgpr2 = V_MOV_B32_e32 0, implicit $exec
+    $vgpr3 = V_MOV_B32_e32 0, implicit $exec
+    $sgpr0_sgpr1 = S_GETPC_B64
+    $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    S_ENDPGM 0
+...
+
+# Check implicit $mode
+---
+name:            mask_hazard_gap2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_gap2
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr2 = V_MOV_B32_e32 0, implicit $exec, implicit $mode
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr0_sgpr1 = S_GETPC_B64
+    ; GCN-NEXT: $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    $vgpr2 = V_MOV_B32_e32 0, implicit $exec, implicit $mode
+    $sgpr0_sgpr1 = S_GETPC_B64
+    $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    S_ENDPGM 0
+...
+
+# Check explicit $exec
+---
+name:            mask_hazard_gap3
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_gap3
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr2 = V_WRITELANE_B32 $exec_lo, 0, $vgpr2
+    ; GCN-NEXT: $vgpr0 = V_XOR3_B32_e64 $vgpr0, $sgpr0, $sgpr0, implicit $exec
+    ; GCN-NEXT: $sgpr0_sgpr1 = S_GETPC_B64
+    ; GCN-NEXT: $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr0_sgpr1, implicit $exec
+    $vgpr2 = V_WRITELANE_B32 $exec_lo, 0, $vgpr2
+    $sgpr0_sgpr1 = S_GETPC_B64
+    $sgpr0 = S_ADD_U32 $sgpr0, 0, implicit-def $scc
+    S_ENDPGM 0
+...
+
+# Different SGPR write
+---
+name:            mask_hazard_no_hazard1
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_no_hazard1
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    ; GCN-NEXT: $sgpr0 = S_MOV_B32 0
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e64 0, $vgpr1, 0, $vgpr2, $sgpr2_sgpr3, implicit $exec
+    $sgpr0 = S_MOV_B32 0
+    S_ENDPGM 0
+...
+
+# Different SGPR write with mask read overlap
+---
+name:            mask_hazard_no_hazard2
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_no_hazard2
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $sgpr0_sgpr1 = S_MOV_B64 $vcc
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    $sgpr0_sgpr1 = S_MOV_B64 $vcc
+    S_ENDPGM 0
+...
+
+# Overlapping VGPR write
+---
+name:            mask_hazard_no_hazard3
+body:            |
+  bb.0:
+    ; GCN-LABEL: name: mask_hazard_no_hazard3
+    ; GCN: $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    ; GCN-NEXT: $vgpr2 = V_MOV_B32_e32 0, implicit $exec
+    ; GCN-NEXT: S_ENDPGM 0
+    $vgpr1 = V_CNDMASK_B32_e32 $vgpr1, $vgpr2, implicit $vcc, implicit $exec
+    $vgpr2 = V_MOV_B32_e32 0, implicit $exec
+    S_ENDPGM 0
+...