[AMDGPU] Skip unclusterd rescheduling w/o ld/st

[rampitec]

Summary

We are attempting rescheduling without load store clustering
if occupancy limits were not met with clustering. Skip this
for regions which do not have any loads or stores at all.

In a set of kernels I am experimenting with this improves
scheduling time by ~30%.

[rampitec]

Check for actual clusters in the region. This is a more precise method adding a bit more of speed.

[arsenm]

llvm/lib/Target/AMDGPU/GCNSchedStrategy.h

53

Needs comment

[rampitec]

Added comment.

[vpykhtin]

Looks good, but should we use just a single dedicated pass over SUs to check if there're clustered ops after first scheduling to make the logic slightly easier?

[rampitec]

In D97342#2590002, @vpykhtin wrote:

Looks good, but should we use just a single dedicated pass over SUs to check if there're clustered ops after first scheduling to make the logic slightly easier?

My problem with that it has to be done in the schedule() method or somewhere else within GCNScheduleDAGMILive. The only way to get an SUnit there is to call getSUnit() passing a MachineInstr and that is a map lookup. I.e. it is simply slower and I am trying to squeeze as much speed as I could.

[rampitec]

In D97342#2590737, @rampitec wrote:

In D97342#2590002, @vpykhtin wrote:

Looks good, but should we use just a single dedicated pass over SUs to check if there're clustered ops after first scheduling to make the logic slightly easier?

My problem with that it has to be done in the schedule() method or somewhere else within GCNScheduleDAGMILive. The only way to get an SUnit there is to call getSUnit() passing a MachineInstr and that is a map lookup. I.e. it is simply slower and I am trying to squeeze as much speed as I could.

Actually it is not even possible. The place where I can do it does not have mutations applied yet.

[rampitec]

In D97342#2590876, @rampitec wrote:

In D97342#2590737, @rampitec wrote:

In D97342#2590002, @vpykhtin wrote:

Looks good, but should we use just a single dedicated pass over SUs to check if there're clustered ops after first scheduling to make the logic slightly easier?

My problem with that it has to be done in the schedule() method or somewhere else within GCNScheduleDAGMILive. The only way to get an SUnit there is to call getSUnit() passing a MachineInstr and that is a map lookup. I.e. it is simply slower and I am trying to squeeze as much speed as I could.

Actually it is not even possible. The place where I can do it does not have mutations applied yet.

In fact I did the experiment, it may look better but I've got 3.6% slower scheduling with that separate loop.

[vpykhtin]

LGTM

llvm/lib/Target/AMDGPU/GCNSchedStrategy.h

class GCNMaxOccupancySchedStrategy final : public GenericScheduler {
 
   unsigned SGPRExcessLimit;
   unsigned VGPRExcessLimit;
   unsigned SGPRCriticalLimit;
   unsigned VGPRCriticalLimit;
 
   unsigned TargetOccupancy;
 
+  bool HasClusteredNodes;
+
   MachineFunction *MF;
 
 public:
   GCNMaxOccupancySchedStrategy(const MachineSchedContext *C);
 
   SUnit *pickNode(bool &IsTopNode) override;
 
   void initialize(ScheduleDAGMI *DAG) override;

llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp

 #include "GCNSchedStrategy.h"
 #include "SIMachineFunctionInfo.h"
 
 #define DEBUG_TYPE "machine-scheduler"
 
 using namespace llvm;
 
 GCNMaxOccupancySchedStrategy::GCNMaxOccupancySchedStrategy(
     const MachineSchedContext *C) :
-    GenericScheduler(C), TargetOccupancy(0), MF(nullptr) { }
+    GenericScheduler(C), TargetOccupancy(0), HasClusteredNodes(false),
+    MF(nullptr) { }
 
 void GCNMaxOccupancySchedStrategy::initialize(ScheduleDAGMI *DAG) {
   GenericScheduler::initialize(DAG);
 
   const SIRegisterInfo *SRI = static_cast<const SIRegisterInfo*>(TRI);
 
   MF = &DAG->MF;
 
do {
     }
   } while (SU->isScheduled);
 
   if (SU->isTopReady())
     Top.removeReady(SU);
   if (SU->isBottomReady())
     Bot.removeReady(SU);
 
+  if (!HasClusteredNodes && SU->getInstr()->mayLoadOrStore()) {
+    for (SDep &Dep : SU->Preds) {
+      if (Dep.isCluster()) {
+        HasClusteredNodes = true;
+        break;
+      }
+    }
+  }
+
   LLVM_DEBUG(dbgs() << "Scheduling SU(" << SU->NodeNum << ") "
                     << *SU->getInstr());
   return SU;
 }
 
 GCNScheduleDAGMILive::GCNScheduleDAGMILive(MachineSchedContext *C,
                         std::unique_ptr<MachineSchedStrategy> S) :
   ScheduleDAGMILive(C, std::move(S)),
if (LIS) {
     LLVM_DEBUG(dbgs() << "Pressure before scheduling:\nRegion live-ins:";
                GCNRPTracker::printLiveRegs(dbgs(), LiveIns[RegionIdx], MRI);
                dbgs() << "Region live-in pressure:  ";
                llvm::getRegPressure(MRI, LiveIns[RegionIdx]).print(dbgs());
                dbgs() << "Region register pressure: ";
                PressureBefore.print(dbgs()));
   }
 
+  GCNMaxOccupancySchedStrategy &S = (GCNMaxOccupancySchedStrategy&)*SchedImpl;
+  // Set HasClusteredNodes to true for late stages where we are not interested
+  // in it anymore. That way pickNode() will not scan SDep's when not needed.
+  S.HasClusteredNodes = Stage >= UnclusteredReschedule;
   ScheduleDAGMILive::schedule();
   Regions[RegionIdx] = std::make_pair(RegionBegin, RegionEnd);
   RescheduleRegions[RegionIdx] = false;
 
   if (!LIS)
     return;
 
   // Check the results of scheduling.
-  GCNMaxOccupancySchedStrategy &S = (GCNMaxOccupancySchedStrategy&)*SchedImpl;
   auto PressureAfter = getRealRegPressure();
 
   LLVM_DEBUG(dbgs() << "Pressure after scheduling: ";
              PressureAfter.print(dbgs()));
 
   if (PressureAfter.getSGPRNum() <= S.SGPRCriticalLimit &&
       PressureAfter.getVGPRNum(ST.hasGFX90AInsts()) <= S.VGPRCriticalLimit) {
     Pressure[RegionIdx] = PressureAfter;
void GCNScheduleDAGMILive::schedule() {
   if (WavesAfter >= MinOccupancy) {
     if (Stage == UnclusteredReschedule &&
         !PressureAfter.less(ST, PressureBefore)) {
       LLVM_DEBUG(dbgs() << "Unclustered reschedule did not help.\n");
     } else if (WavesAfter > MFI.getMinWavesPerEU() ||
         PressureAfter.less(ST, PressureBefore) ||
         !RescheduleRegions[RegionIdx]) {
       Pressure[RegionIdx] = PressureAfter;
+      if (!S.HasClusteredNodes && (Stage + 1) == UnclusteredReschedule)
+        RescheduleRegions[RegionIdx] = false;
       return;
     } else {
       LLVM_DEBUG(dbgs() << "New pressure will result in more spilling.\n");
     }
   }
 
   LLVM_DEBUG(dbgs() << "Attempting to revert scheduling.\n");
-  RescheduleRegions[RegionIdx] = true;
+  RescheduleRegions[RegionIdx] = S.HasClusteredNodes ||
+                                 (Stage + 1) != UnclusteredReschedule;
   RegionEnd = RegionBegin;
   for (MachineInstr *MI : Unsched) {
     if (MI->isDebugInstr())
       continue;
 
     if (MI->getIterator() != RegionEnd) {
       BB->remove(MI);
       BB->insert(RegionEnd, MI);