[AMDGPU] W/a hazard if 64 bit shift amount is a highest allocated VGPR

[rampitec]

Summary

In this case gfx90a uses v0 instead of the correct register. Swap
the value temporarily with a lower register and then swap it back.

Unfortunately hazard recognizer works after wait count insertion,
so we cannot simply reuse an arbitrary register, hence w/a also
includes a full waitcount. This can be avoided if we run it from
expandPostRAPseudo, but that is a complete misplacement.

[rampitec]

Also we could still constrain src0 of these to a decimated RC in the finalizeLowering and keep this w/a just in case further optimizations would replace the operand. In that case this will be an extremely rare situation we hit the w/a itself. Although constraining the RC will have its own negative impact on the register allocation.

[rampitec]

The other problem each of these new instructions can create a new hazard just by itself. So we probably need to reset the iterator to a first new instruction or run recognizer on each of them.

Alternatively it can be possible to really run this earlier, from post-ra pseudo expansion.

[foad]

Also we could still constrain src0 of these to a decimated RC in the finalizeLowering and keep this w/a just in case further optimizations would replace the operand.

If you use a different RC for src0 then surely you don't need the workaround? The rest of the compiler should respect the RC, and should not "replace the operand" with a physreg that is not in the right RC.

[rampitec]

In D133067#3763494, @foad wrote:

Also we could still constrain src0 of these to a decimated RC in the finalizeLowering and keep this w/a just in case further optimizations would replace the operand.

If you use a different RC for src0 then surely you don't need the workaround? The rest of the compiler should respect the RC, and should not "replace the operand" with a physreg that is not in the right RC.

That is if I define a whole new set of shifts for this target. This would create a whole new set of problems. If I just constrain existing vgpr32 operand to a decimated RC it will eventually be replaced by some subsequent passes.

[foad]

That is if I define a whole new set of shifts for this target. This would create a whole new set of problems.

OK, understood. We are doing something similar for 16-bit operands for GFX11, but it is a lot of work.

constrain existing vgpr32 operand to a decimated RC

"octimated", surely? :-) :-)

[rampitec]

In D133067#3763528, @foad wrote:

constrain existing vgpr32 operand to a decimated RC

"octimated", surely? :-) :-)

For sure yes, although list operator is DECIMATE, even if with an 8 as argument.

[rampitec]

One more thing to note: in most of the cases shift amount is an immediate, and even an inline literal. So we need to be really lucky to get here.

[rampitec]

Added run of the recognizer on the newly inserted swaps and tests for it.

[rampitec]

ping

[kerbowa]

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

311

Do you need to reset CurrCycleInstr after calling this function? Should this work with process bundles?

[rampitec]

Fixed work inside a bundle.

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

311

AdvanceCycle() sets CurrCycleInstr to nullptr itself.

Good point about bundle, fixed.

[kerbowa]

LGTM

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.h

private:
   /// \returns the number of wait states before another MFMA instruction can be
   /// issued after \p MI.
   unsigned getMFMAPipelineWaitStates(const MachineInstr &MI) const;
 
   // Advance over a MachineInstr bundle. Look for hazards in the bundled
   // instructions.
   void processBundle();
 
+  // Run on an individual instruction in hazard recognizer mode. This can be
+  // used on a newly inserted instruction before returning from PreEmitNoops.
+  void runOnInstruction(MachineInstr *MI);
+
   int getWaitStatesSince(IsHazardFn IsHazard, int Limit);
   int getWaitStatesSinceDef(unsigned Reg, IsHazardFn IsHazardDef, int Limit);
   int getWaitStatesSinceSetReg(IsHazardFn IsHazard, int Limit);
 
   int checkSoftClauseHazards(MachineInstr *SMEM);
   int checkSMRDHazards(MachineInstr *SMRD);
   int checkVMEMHazards(MachineInstr* VMEM);
   int checkDPPHazards(MachineInstr *DPP);
private:
   bool fixSMEMtoVectorWriteHazards(MachineInstr *MI);
   bool fixVcmpxExecWARHazard(MachineInstr *MI);
   bool fixLdsBranchVmemWARHazard(MachineInstr *MI);
   bool fixLdsDirectVALUHazard(MachineInstr *MI);
   bool fixLdsDirectVMEMHazard(MachineInstr *MI);
   bool fixVALUPartialForwardingHazard(MachineInstr *MI);
   bool fixVALUTransUseHazard(MachineInstr *MI);
   bool fixWMMAHazards(MachineInstr *MI);
+  bool fixShift64HighRegBug(MachineInstr *MI);
 
   int checkMAIHazards(MachineInstr *MI);
   int checkMAIHazards908(MachineInstr *MI);
   int checkMAIHazards90A(MachineInstr *MI);
   /// Pad the latency between neighboring MFMA instructions with s_nops. The
   /// percentage of wait states to fill with s_nops is specified by the command
   /// line option '-amdgpu-mfma-padding-ratio'.
   ///

llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp

for (unsigned i = 0, e = std::min(WaitStates, MaxLookAhead - 1); i < e; ++i)
       EmittedInstrs.push_front(nullptr);
 
     EmittedInstrs.push_front(CurrCycleInstr);
     EmittedInstrs.resize(MaxLookAhead);
   }
   CurrCycleInstr = nullptr;
 }
 
+void GCNHazardRecognizer::runOnInstruction(MachineInstr *MI) {
+  assert(IsHazardRecognizerMode);
+
+  unsigned NumPreNoops = PreEmitNoops(MI);
+  EmitNoops(NumPreNoops);
+  if (MI->isInsideBundle())
+    insertNoopsInBundle(MI, TII, NumPreNoops);
+  else
+    TII.insertNoops(*MI->getParent(), MachineBasicBlock::iterator(MI),
+                    NumPreNoops);
+  EmitInstruction(MI);
+  AdvanceCycle();
+}
+
 unsigned GCNHazardRecognizer::PreEmitNoops(MachineInstr *MI) {
   IsHazardRecognizerMode = true;
   CurrCycleInstr = MI;
   unsigned W = PreEmitNoopsCommon(MI);
   fixHazards(MI);
   CurrCycleInstr = nullptr;
   return W;
 }
void GCNHazardRecognizer::fixHazards(MachineInstr *MI) {
   fixLdsBranchVmemWARHazard(MI);
   if (ST.hasLdsDirect()) {
     fixLdsDirectVALUHazard(MI);
     fixLdsDirectVMEMHazard(MI);
   }
   fixVALUPartialForwardingHazard(MI);
   fixVALUTransUseHazard(MI);
   fixWMMAHazards(MI);
+  fixShift64HighRegBug(MI);
 }
 
 bool GCNHazardRecognizer::fixVcmpxPermlaneHazards(MachineInstr *MI) {
   if (!ST.hasVcmpxPermlaneHazard() || !isPermlane(*MI))
     return false;
 
   const SIInstrInfo *TII = ST.getInstrInfo();
   const SIRegisterInfo *TRI = ST.getRegisterInfo();
if (::getWaitStatesSince(IsHazardFn, MI, IsExpiredFn) ==
       std::numeric_limits<int>::max())
     return false;
 
   BuildMI(*MI->getParent(), MI, MI->getDebugLoc(), TII->get(AMDGPU::V_NOP_e32));
 
   return true;
 }
 
+bool GCNHazardRecognizer::fixShift64HighRegBug(MachineInstr *MI) {
+  if (!ST.hasShift64HighRegBug())
+    return false;
+
+  switch (MI->getOpcode()) {
+  default:
+    return false;
+  case AMDGPU::V_LSHLREV_B64_e64:
+  case AMDGPU::V_LSHRREV_B64_e64:
+  case AMDGPU::V_ASHRREV_I64_e64:
+    break;
+  }
+
+  MachineOperand *Amt = TII.getNamedOperand(*MI, AMDGPU::OpName::src0);
+  if (!Amt->isReg())
+    return false;
+
+  Register AmtReg = Amt->getReg();
+  const MachineRegisterInfo &MRI = MF.getRegInfo();
+  // Check if this is a last VGPR in the allocation block.
+  if (!TRI.isVGPR(MRI, AmtReg) || ((AmtReg - AMDGPU::VGPR0) & 7) != 7)
+    return false;
+
+  if (AmtReg != AMDGPU::VGPR255 && MRI.isPhysRegUsed(AmtReg + 1))
+    return false;
+
+  MachineOperand *Src1 = TII.getNamedOperand(*MI, AMDGPU::OpName::src1);
+  bool OverlappedSrc = Src1->isReg() && TRI.regsOverlap(Src1->getReg(), AmtReg);
+  bool OverlappedDst = MI->modifiesRegister(AmtReg, &TRI);
+  bool Overlapped = OverlappedSrc || OverlappedDst;
+
+  assert(!OverlappedDst || !OverlappedSrc ||
+         Src1->getReg() == MI->getOperand(0).getReg());
+  assert(ST.needsAlignedVGPRs());
+  static_assert(AMDGPU::VGPR0 + 1 == AMDGPU::VGPR1);
+
+  Register NewReg;
+  for (MCRegister Reg : Overlapped ? AMDGPU::VReg_64_Align2RegClass
+                                   : AMDGPU::VGPR_32RegClass) {
+    if (!MI->modifiesRegister(Reg, &TRI) && !MI->readsRegister(Reg, &TRI)) {
+      NewReg = Reg;
+      break;
+    }
+  }
+
+  Register NewAmt = Overlapped ? (Register)TRI.getSubReg(NewReg, AMDGPU::sub1)
+                               : NewReg;
+  Register NewAmtLo;
+
+  if (Overlapped)
+    NewAmtLo = TRI.getSubReg(NewReg, AMDGPU::sub0);
+
+  DebugLoc DL = MI->getDebugLoc();
+  MachineBasicBlock *MBB = MI->getParent();
+  // Insert a full wait count because found register might be pending a wait.
+  BuildMI(*MBB, MI, DL, TII.get(AMDGPU::S_WAITCNT))
+      .addImm(0);
+
+  // Insert V_SWAP_B32 instruction(s) and run hazard recognizer on them.
+  if (Overlapped)
+    runOnInstruction(
+        BuildMI(*MBB, MI, DL, TII.get(AMDGPU::V_SWAP_B32), NewAmtLo)
+            .addDef(AmtReg - 1)
+            .addReg(AmtReg - 1)
+            .addReg(NewAmtLo));
+  runOnInstruction(BuildMI(*MBB, MI, DL, TII.get(AMDGPU::V_SWAP_B32), NewAmt)
+                       .addDef(AmtReg)
+                       .addReg(AmtReg)
+                       .addReg(NewAmt));
+
+  // Instructions emitted after the current instruction will be processed by the
+  // parent loop of the hazard recognizer in a natural way.
+  BuildMI(*MBB, std::next(MI->getIterator()), DL, TII.get(AMDGPU::V_SWAP_B32),
+          AmtReg)
+      .addDef(NewAmt)
+      .addReg(NewAmt)
+      .addReg(AmtReg);
+  if (Overlapped)
+    BuildMI(*MBB, std::next(MI->getIterator()), DL, TII.get(AMDGPU::V_SWAP_B32),
+            AmtReg - 1)
+        .addDef(NewAmtLo)
+        .addReg(NewAmtLo)
+        .addReg(AmtReg - 1);
+
+  // Re-running hazard recognizer on the modified instruction is not necessary,
+  // inserted V_SWAP_B32 has already both read and write new registers so
+  // hazards related to these register has already been handled.
+  Amt->setReg(NewAmt);
+  Amt->setIsKill(false);
+  if (OverlappedDst)
+    MI->getOperand(0).setReg(NewReg);
+  if (OverlappedSrc) {
+    Src1->setReg(NewReg);
+    Src1->setIsKill(false);
+  }
+
+  return true;
+}
+
 int GCNHazardRecognizer::checkNSAtoVMEMHazard(MachineInstr *MI) {
   int NSAtoVMEMWaitStates = 1;
 
   if (!ST.hasNSAtoVMEMBug())
     return 0;
 
   if (!SIInstrInfo::isMUBUF(*MI) && !SIInstrInfo::isMTBUF(*MI))
     return 0;

llvm/lib/Target/AMDGPU/GCNSubtarget.h

public:
   bool hasVcmpxExecWARHazard() const {
     return HasVcmpxExecWARHazard;
   }
 
   bool hasLdsBranchVmemWARHazard() const {
     return HasLdsBranchVmemWARHazard;
   }
 
+  // Shift amount of a 64 bit shift cannot be a highest allocated register
+  // if also at the end of the allocation block.
+  bool hasShift64HighRegBug() const {
+    return GFX90AInsts && !GFX940Insts;
+  }
+
   // Has one cycle hazard on transcendental instruction feeding a
   // non transcendental VALU.
   bool hasTransForwardingHazard() const { return GFX940Insts; }
 
   // Has one cycle hazard on a VALU instruction partially writing dst with
   // a shift of result bits feeding another VALU instruction.
   bool hasDstSelForwardingHazard() const { return GFX940Insts; }
 

llvm/test/CodeGen/AMDGPU/hazard-shift64.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -march=amdgcn -mcpu=gfx90a -verify-machineinstrs -run-pass post-RA-hazard-rec %s -o - | FileCheck -check-prefix=GCN %s
+
+---
+name:            highest_reg_shift_amt_v7
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_v7
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr2_vgpr3 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr0, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr0, implicit $exec
+    ; GCN-NEXT: renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr0, killed $vgpr2_vgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr2_vgpr3 = IMPLICIT_DEF
+    renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr2_vgpr3, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_v15
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_v15
+    ; GCN: $vgpr15 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr2_vgpr3 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr0, $vgpr15 = V_SWAP_B32 $vgpr15, $vgpr0, implicit $exec
+    ; GCN-NEXT: renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr0, killed $vgpr2_vgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr15, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr15, implicit $exec
+    $vgpr15 = IMPLICIT_DEF
+    $vgpr2_vgpr3 = IMPLICIT_DEF
+    renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 killed $vgpr15, killed $vgpr2_vgpr3, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_v255
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_v255
+    ; GCN: $vgpr255 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr2_vgpr3 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr0, $vgpr255 = V_SWAP_B32 $vgpr255, $vgpr0, implicit $exec
+    ; GCN-NEXT: renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr0, killed $vgpr2_vgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr255, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr255, implicit $exec
+    $vgpr255 = IMPLICIT_DEF
+    $vgpr2_vgpr3 = IMPLICIT_DEF
+    renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 killed $vgpr255, killed $vgpr2_vgpr3, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_used_v0_dst
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_used_v0_dst
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr2_vgpr3 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr4, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr4, implicit $exec
+    ; GCN-NEXT: renamable $vgpr0_vgpr1 = V_LSHRREV_B64_e64 $vgpr4, killed $vgpr2_vgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr4 = V_SWAP_B32 $vgpr4, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr2_vgpr3 = IMPLICIT_DEF
+    renamable $vgpr0_vgpr1 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr2_vgpr3, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_used_v0_src
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_used_v0_src
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr0_vgpr1 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr4, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr4, implicit $exec
+    ; GCN-NEXT: renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr4, killed $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr4 = V_SWAP_B32 $vgpr4, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr0_vgpr1 = IMPLICIT_DEF
+    renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr0_vgpr1, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_used_v0_both
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_used_v0_both
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr0_vgpr1 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr2, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr2, implicit $exec
+    ; GCN-NEXT: renamable $vgpr0_vgpr1 = V_LSHRREV_B64_e64 $vgpr2, killed $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr2 = V_SWAP_B32 $vgpr2, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr0_vgpr1 = IMPLICIT_DEF
+    renamable $vgpr0_vgpr1 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr0_vgpr1, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_overlapped_src
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_overlapped_src
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr6_vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr2, $vgpr6 = V_SWAP_B32 $vgpr6, $vgpr2, implicit $exec
+    ; GCN-NEXT: $vgpr3, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr3, implicit $exec
+    ; GCN-NEXT: renamable $vgpr0_vgpr1 = V_LSHRREV_B64_e64 $vgpr3, $vgpr2_vgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr6, $vgpr2 = V_SWAP_B32 $vgpr2, $vgpr6, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr3 = V_SWAP_B32 $vgpr3, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr6_vgpr7 = IMPLICIT_DEF
+    renamable $vgpr0_vgpr1 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr6_vgpr7, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_overlapped_dst
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_overlapped_dst
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr0_vgpr1 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr2, $vgpr6 = V_SWAP_B32 $vgpr6, $vgpr2, implicit $exec
+    ; GCN-NEXT: $vgpr3, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr3, implicit $exec
+    ; GCN-NEXT: $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr3, killed $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr6, $vgpr2 = V_SWAP_B32 $vgpr2, $vgpr6, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr3 = V_SWAP_B32 $vgpr3, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr0_vgpr1 = IMPLICIT_DEF
+    renamable $vgpr6_vgpr7 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr0_vgpr1, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_overlapped_both
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_overlapped_both
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr6_vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr0, $vgpr6 = V_SWAP_B32 $vgpr6, $vgpr0, implicit $exec
+    ; GCN-NEXT: $vgpr1, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr0_vgpr1 = V_LSHRREV_B64_e64 $vgpr1, $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr6, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr6, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr1 = V_SWAP_B32 $vgpr1, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr6_vgpr7 = IMPLICIT_DEF
+    renamable $vgpr6_vgpr7 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr6_vgpr7, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_hazard_in_swap
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_hazard_in_swap
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr0_vgpr1 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr4_vgpr5 = V_MFMA_F64_4X4X4F64_vgprcd_e64 $vgpr0_vgpr1, $vgpr0_vgpr1, $vgpr0_vgpr1, 0, 0, 0, implicit $mode, implicit $exec
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: S_NOP 4
+    ; GCN-NEXT: $vgpr4, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr4, implicit $exec
+    ; GCN-NEXT: renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr4, killed $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr4 = V_SWAP_B32 $vgpr4, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr0_vgpr1 = IMPLICIT_DEF
+    $vgpr4_vgpr5 = V_MFMA_F64_4X4X4F64_vgprcd_e64 $vgpr0_vgpr1, $vgpr0_vgpr1, $vgpr0_vgpr1, 0, 0, 0, implicit $mode, implicit $exec
+    renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr0_vgpr1, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_hazard_in_swap2
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_hazard_in_swap2
+    ; GCN: $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr6_vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT: $vgpr1 = V_DOT4C_I32_I8_e32 $vgpr7, $vgpr7, $vgpr1, implicit $exec
+    ; GCN-NEXT: S_WAITCNT 0
+    ; GCN-NEXT: $vgpr0, $vgpr6 = V_SWAP_B32 $vgpr6, $vgpr0, implicit $exec
+    ; GCN-NEXT: S_NOP 0
+    ; GCN-NEXT: $vgpr1, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr0_vgpr1 = V_LSHRREV_B64_e64 $vgpr1, $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: $vgpr6, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr6, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr1 = V_SWAP_B32 $vgpr1, $vgpr7, implicit $exec
+    $vgpr7 = IMPLICIT_DEF
+    $vgpr6_vgpr7 = IMPLICIT_DEF
+    $vgpr1 = V_DOT4C_I32_I8_e32 $vgpr7, $vgpr7, $vgpr1, implicit $exec
+    renamable $vgpr6_vgpr7 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr6_vgpr7, implicit $exec
+...
+
+---
+name:            highest_reg_shift_amt_v7_bundle
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_v7_bundle
+    ; GCN: BUNDLE implicit-def $vgpr2_vgpr3, implicit-def $vgpr7 {
+    ; GCN-NEXT:   $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT:   $vgpr2_vgpr3 = IMPLICIT_DEF
+    ; GCN-NEXT:   S_WAITCNT 0
+    ; GCN-NEXT:   $vgpr0, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr0, implicit $exec
+    ; GCN-NEXT:   renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 $vgpr0, killed $vgpr2_vgpr3, implicit $exec
+    ; GCN-NEXT: }
+    ; GCN-NEXT: $vgpr7, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr7, implicit $exec
+    BUNDLE implicit-def $vgpr2_vgpr3, implicit-def $vgpr7 {
+      $vgpr7 = IMPLICIT_DEF
+      $vgpr2_vgpr3 = IMPLICIT_DEF
+      renamable $vgpr2_vgpr3 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr2_vgpr3, implicit $exec
+    }
+...
+
+---
+name:            highest_reg_shift_amt_hazard_in_swap2_bundle
+body:             |
+  bb.0:
+
+    ; GCN-LABEL: name: highest_reg_shift_amt_hazard_in_swap2_bundle
+    ; GCN: BUNDLE implicit-def $vgpr1, implicit-def $vgpr6_vgpr7 {
+    ; GCN-NEXT:   $vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT:   $vgpr6_vgpr7 = IMPLICIT_DEF
+    ; GCN-NEXT:   $vgpr1 = V_DOT4C_I32_I8_e32 $vgpr7, $vgpr7, $vgpr1, implicit $exec
+    ; GCN-NEXT:   S_WAITCNT 0
+    ; GCN-NEXT:   $vgpr0, $vgpr6 = V_SWAP_B32 $vgpr6, $vgpr0, implicit $exec
+    ; GCN-NEXT:   S_NOP 0
+    ; GCN-NEXT:   $vgpr1, $vgpr7 = V_SWAP_B32 $vgpr7, $vgpr1, implicit $exec
+    ; GCN-NEXT:   $vgpr0_vgpr1 = V_LSHRREV_B64_e64 $vgpr1, $vgpr0_vgpr1, implicit $exec
+    ; GCN-NEXT: }
+    ; GCN-NEXT: $vgpr6, $vgpr0 = V_SWAP_B32 $vgpr0, $vgpr6, implicit $exec
+    ; GCN-NEXT: $vgpr7, $vgpr1 = V_SWAP_B32 $vgpr1, $vgpr7, implicit $exec
+    BUNDLE implicit-def $vgpr1, implicit-def $vgpr6_vgpr7 {
+      $vgpr7 = IMPLICIT_DEF
+      $vgpr6_vgpr7 = IMPLICIT_DEF
+      $vgpr1 = V_DOT4C_I32_I8_e32 $vgpr7, $vgpr7, $vgpr1, implicit $exec
+      renamable $vgpr6_vgpr7 = V_LSHRREV_B64_e64 killed $vgpr7, killed $vgpr6_vgpr7, implicit $exec
+    }
+...