AMDGPU: Fix dbg_value handling when forming soft clause bundles

[arsenm]

Summary

DBG_VALUES placed between memory instructions would change
codegen. Skip over these and re-insert them after the bundle instead
of giving up on bundling.

[arsenm]

Forgot to git add test

[dfukalov]

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

389

Why don't just create DbgInstrs here?

llvm/test/CodeGen/AMDGPU/soft-clause-dbg-value.mir

2 ↗

(On Diff #320357)

Am I right that we need phi-node-elimination, before to remove IsSSA property to avoid verifier' fail?
Or, dummy copy of first instruction %0:sreg_64 = COPY $sgpr4_sgpr5 ; defeat IsSSA detection can be added.

[scott.linder]

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

325

Since https://reviews.llvm.org/D92522, if you don't have a strong reason for choosing 8

391–393

When does Next != BundleNext after this for terminates? It seems like you should be able to just use Next below when re-inserting the DBG instructions?

[rampitec]

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

389

Right, it can be local to this block and cleared/destroyed after re-insertion. It will take less memory.

[arsenm]

Address comments

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

389

This avoids reallocating on subsequent bundles if it's necessary

391–393

It doesn't make a difference

llvm/test/CodeGen/AMDGPU/soft-clause-dbg-value.mir

2 ↗

(On Diff #320357)

No. The pass cleared properties now clears SSA

[arsenm]

41877b82f07224041a2a994f9032332fe01e4d1b

[scott.linder]

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

391–393

I disagree that it doesn't matter, after reading to this point I have to assume either (a) I don't understand the code, or (b) there is a missing assert(BundleNext == Next);

Can you either delete BundleNext or add an assert?

[arsenm]

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

391–393

BundleNext is needed for incrementing the inner loop up to Next

[scott.linder]

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

389

I didn't notice this when I first read, but it doesn't really seem necessary to buffer up the debug instrs at all. Can't we just MBB.insert(Next, BI->removeFromParent()) below? Did you do the buffering and extra loop because it makes tracking when to end iteration simpler, and because the largest the vector can get is amdgpu-max-memory-clause anyway?

391–393

Thank you for changing this in the committed version!

llvm/lib/Target/AMDGPU/GCNRegPressure.cpp

bool GCNDownwardRPTracker::advanceBeforeNext() {
 
   MaxPressure = max(MaxPressure, CurPressure);
 
   return true;
 }
 
 void GCNDownwardRPTracker::advanceToNext() {
   LastTrackedMI = &*NextMI++;
+  NextMI = skipDebugInstructionsForward(NextMI, MBBEnd);
 
   // Add new registers or mask bits.
   for (const auto &MO : LastTrackedMI->operands()) {
     if (!MO.isReg() || !MO.isDef())
       continue;
     Register Reg = MO.getReg();
     if (!Reg.isVirtual())
       continue;

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

 
 static bool isSMEMClauseInst(const MachineInstr &MI) {
   return SIInstrInfo::isSMRD(MI);
 }
 
 // There no sense to create store clauses, they do not define anything,
 // thus there is nothing to set early-clobber.
 static bool isValidClauseInst(const MachineInstr &MI, bool IsVMEMClause) {
-  if (MI.isDebugValue() || MI.isBundled())
+  assert(!MI.isDebugInstr() && "debug instructions should not reach here");
+  if (MI.isBundled())
     return false;
   if (!MI.mayLoad() || MI.mayStore())
     return false;
   if (AMDGPU::getAtomicNoRetOp(MI.getOpcode()) != -1 ||
       AMDGPU::getAtomicRetOp(MI.getOpcode()) != -1)
     return false;
   if (IsVMEMClause && !isVMEMClauseInst(MI))
     return false;
bool SIFormMemoryClauses::runOnMachineFunction(MachineFunction &MF) {
   SlotIndexes *Ind = LIS->getSlotIndexes();
   bool Changed = false;
 
   MaxVGPRs = TRI->getAllocatableSet(MF, &AMDGPU::VGPR_32RegClass).count();
   MaxSGPRs = TRI->getAllocatableSet(MF, &AMDGPU::SGPR_32RegClass).count();
   unsigned FuncMaxClause = AMDGPU::getIntegerAttribute(
       MF.getFunction(), "amdgpu-max-memory-clause", MaxClause);
 
+  SmallVector<MachineInstr *, 8> DbgInstrs;
+
   for (MachineBasicBlock &MBB : MF) {
     GCNDownwardRPTracker RPT(*LIS);
     MachineBasicBlock::instr_iterator Next;
     for (auto I = MBB.instr_begin(), E = MBB.instr_end(); I != E; I = Next) {
       MachineInstr &MI = *I;
       Next = std::next(I);
 
+      if (MI.isDebugInstr())
+        continue;
+
       bool IsVMEM = isVMEMClauseInst(MI);
 
       if (!isValidClauseInst(MI, IsVMEM))
         continue;
 
       if (!RPT.getNext().isValid())
         RPT.reset(MI);
       else { // Advance the state to the current MI.
         RPT.advance(MachineBasicBlock::const_iterator(MI));
         RPT.advanceBeforeNext();
       }
 
       const GCNRPTracker::LiveRegSet LiveRegsCopy(RPT.getLiveRegs());
       RegUse Defs, Uses;
       if (!processRegUses(MI, Defs, Uses, RPT)) {
         RPT.reset(MI, &LiveRegsCopy);
         continue;
       }
 
       unsigned Length = 1;
       for ( ; Next != E && Length < FuncMaxClause; ++Next) {
+        // Debug instructions should not change the bundling. We need to move
+        // these after the bundle
+        if (Next->isDebugInstr()) {
+          //DbgInstrs.push_back(&*Next);
+          continue;
+        }
+
         if (!isValidClauseInst(*Next, IsVMEM))
           break;
 
         // A load from pointer which was loaded inside the same bundle is an
         // impossible clause because we will need to write and read the same
         // register inside. In this case processRegUses will return false.
         if (!processRegUses(*Next, Defs, Uses, RPT))
           break;
 
         ++Length;
       }
       if (Length < 2) {
         RPT.reset(MI, &LiveRegsCopy);
         continue;
       }
 
       Changed = true;
       MFI->limitOccupancy(LastRecordedOccupancy);
 
       auto B = BuildMI(MBB, I, DebugLoc(), TII->get(TargetOpcode::BUNDLE));
       Ind->insertMachineInstrInMaps(*B);
 
       // Restore the state after processing the bundle.
       RPT.reset(*B, &LiveRegsCopy);
+      DbgInstrs.clear();
+
+      auto BundleNext = I;
+      for (auto BI = I; BI != Next; BI = BundleNext) {
+        BundleNext = std::next(BI);
+
+        if (BI->isDebugValue()) {
+          DbgInstrs.push_back(BI->removeFromParent());
+          continue;
+        }
 
-      for (auto BI = I; BI != Next; ++BI) {
         BI->bundleWithPred();
         Ind->removeSingleMachineInstrFromMaps(*BI);
 
         for (MachineOperand &MO : BI->defs())
           if (MO.readsReg())
             MO.setIsInternalRead(true);
       }
 
+      // Replace any debug instructions after the new bundle.
+      for (MachineInstr *DbgInst : DbgInstrs)
+        MBB.insert(BundleNext, DbgInst);
+
       for (auto &&R : Defs) {
         forAllLanes(R.first, R.second.second, [&R, &B](unsigned SubReg) {
           unsigned S = R.second.first | RegState::EarlyClobber;
           if (!SubReg)
             S &= ~(RegState::Undef | RegState::Dead);
           B.addDef(R.first, S, SubReg);
         });
       }