[AMDGPU] Calculate minimum allowed occupancy based on threads per lane

[foad]

Summary

Previously the minimum allowed occupancy (for memory-bound functions)
was a constant 4 waves per SIMD. Now it is a constant 16 threads per
SIMD lane, which better accounts for the architectural changes in GFX10.

[rampitec]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp

200

This is not expressable in a number of VGPRs. The limiter is about memory and not about register budget at all. Meanwhile it looks like 8 waves might be more reasonable for wave32, but wave64 shall still use the same value across the targets, because at the end of the day it boils down to a number of concurrent memory operations.

[nhaehnle]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp

200

I agree that using number of VGPRs here is an awkward roundabout way of doing it. The point isn't really about VGPRs, it's about how many threads are alive per lane in order to allow a good mix of VALU and VMEM instruction issue in the face of certain (IMHO underspecified) memory pressure situations.

I also disagree that 8 is the right answer for Wave32, for precisely the same reason: it is about the number of concurrent memory operations. 16x Wave32 gives you the same amount of memory operations per CU (2 SIMD * 16 waves * 32 = 1024 workitems) as 4 waves on <=gfx9 (4 SIMD * 4 waves * 64 = 1024 workitems).

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h

919

The ST argument requirement seems annoying. Wouldn't it make sense for the MachineFunctionInfo to know the underlying MachineFunction, which would allow access to the GCNSubtarget (i.e. most-derived TargetSubtargetInfo)?

[arsenm]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h

919

No, I'm actively trying to remove references to the machine function state from MachineFunctionInfo

[rampitec]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp

200

I can understand the logic behind it, but using of TotalNumVGPRs() is not really what we want here. It probably needs to use getMaxWavesPerEU() and wavefront size to estimate a number of threads in flight. Besides the same logic will give answer 16 even on gfx1030 where 16 is the absolute maximum. The latter does not sound right.

[nhaehnle]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h

919

Why is that? MachineFunctionInfo is explicitly created with the MachineFunction as a constructor argument?

[arsenm]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h

919

That's what I'm trying to fix in D80249

[nhaehnle]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp

200

I agree that gfx10 can probably drop the requirement from the 16 that would be equivalent to gfx9. I've heard numbers in the 10-12 range being thrown around as where dropping lower than that hurts performance. On gfx9, the consensus seemed to be 4 waves for the same kind of folklore. I don't think we really have enough experience and thorough performance studies with gfx10 yet to decide this finally. Maybe it should be a cl::opt... (not a function attribute, since this is specifically about the default heuristic).

[rampitec]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp

200

Probably, and I do not think we can use simple divide x by y logic here.

[nhaehnle]

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h

919

I see. I commented on that change because I don't think it's quite right, but regardless of the outcome of that discussion: D80249 still gives the MFI accesses to the TargetSubtargetInfo, which is what we're talking about here. Even with that change, passing the ST yet again into this method should be unnecessary.

llvm/lib/Target/AMDGPU/GCNIterativeScheduler.cpp

unsigned GCNIterativeScheduler::tryMaximizeOccupancy(unsigned TargetOcc) {
 
   return std::max(NewOcc, Occ);
 }
 
 void GCNIterativeScheduler::scheduleLegacyMaxOccupancy(
   bool TryMaximizeOccupancy) {
   const auto &ST = MF.getSubtarget<GCNSubtarget>();
   SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();
-  auto TgtOcc = MFI->getMinAllowedOccupancy();
+  auto TgtOcc = MFI->getMinAllowedOccupancy(ST);
 
   sortRegionsByPressure(TgtOcc);
   auto Occ = Regions.front()->MaxPressure.getOccupancy(ST);
 
   if (TryMaximizeOccupancy && Occ < TgtOcc)
     Occ = tryMaximizeOccupancy(TgtOcc);
 
   // This is really weird but for some magic scheduling regions twice

 
 ///////////////////////////////////////////////////////////////////////////////
 // ILP scheduler port
 
 void GCNIterativeScheduler::scheduleILP(
   bool TryMaximizeOccupancy) {
   const auto &ST = MF.getSubtarget<GCNSubtarget>();
   SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();
-  auto TgtOcc = MFI->getMinAllowedOccupancy();
+  auto TgtOcc = MFI->getMinAllowedOccupancy(ST);
 
   sortRegionsByPressure(TgtOcc);
   auto Occ = Regions.front()->MaxPressure.getOccupancy(ST);
 
   if (TryMaximizeOccupancy && Occ < TgtOcc)
     Occ = tryMaximizeOccupancy(TgtOcc);
 
   TgtOcc = std::min(Occ, TgtOcc);

llvm/lib/Target/AMDGPU/GCNSchedStrategy.cpp

LLVM_DEBUG(dbgs() << "Occupancy before scheduling: " << WavesBefore
                     << ", after " << WavesAfter << ".\n");
 
   // We could not keep current target occupancy because of the just scheduled
   // region. Record new occupancy for next scheduling cycle.
   unsigned NewOccupancy = std::max(WavesAfter, WavesBefore);
   // Allow memory bound functions to drop to 4 waves if not limited by an
   // attribute.
   if (WavesAfter < WavesBefore && WavesAfter < MinOccupancy &&
-      WavesAfter >= MFI.getMinAllowedOccupancy()) {
+      WavesAfter >= MFI.getMinAllowedOccupancy(ST)) {
     LLVM_DEBUG(dbgs() << "Function is memory bound, allow occupancy drop up to "
-                      << MFI.getMinAllowedOccupancy() << " waves\n");
+                      << MFI.getMinAllowedOccupancy(ST) << " waves\n");
     NewOccupancy = WavesAfter;
   }
   if (NewOccupancy < MinOccupancy) {
     MinOccupancy = NewOccupancy;
     MFI.limitOccupancy(MinOccupancy);
     LLVM_DEBUG(dbgs() << "Occupancy lowered for the function to "
                       << MinOccupancy << ".\n");
   }

llvm/lib/Target/AMDGPU/SIFormMemoryClauses.cpp

bool SIFormMemoryClauses::checkPressure(const MachineInstr &MI,
                                         GCNDownwardRPTracker &RPT) {
   // NB: skip advanceBeforeNext() call. Since all defs will be marked
   // early-clobber they will all stay alive at least to the end of the
   // clause. Therefor we should not decrease pressure even if load
   // pointer becomes dead and could otherwise be reused for destination.
   RPT.advanceToNext();
   GCNRegPressure MaxPressure = RPT.moveMaxPressure();
   unsigned Occupancy = MaxPressure.getOccupancy(*ST);
-  if (Occupancy >= MFI->getMinAllowedOccupancy() &&
+  if (Occupancy >= MFI->getMinAllowedOccupancy(*ST) &&
       MaxPressure.getVGPRNum() <= MaxVGPRs &&
       MaxPressure.getSGPRNum() <= MaxSGPRs) {
     LastRecordedOccupancy = Occupancy;
     return true;
   }
   return false;
 }
 

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h

 #include "llvm/Support/ErrorHandling.h"
 #include <array>
 #include <cassert>
 #include <utility>
 #include <vector>
 
 namespace llvm {
 
+class GCNSubtarget;
 class MachineFrameInfo;
 class MachineFunction;
 class TargetRegisterClass;
 
 class AMDGPUPseudoSourceValue : public PseudoSourceValue {
 public:
   enum AMDGPUPSVKind : unsigned {
     PSVBuffer = PseudoSourceValue::TargetCustom,
const AMDGPUGWSResourcePseudoSourceValue *getGWSPSV(const SIInstrInfo &TII) {
 
     return GWSResourcePSV.get();
   }
 
   unsigned getOccupancy() const {
     return Occupancy;
   }
 
-  unsigned getMinAllowedOccupancy() const {
-    if (!isMemoryBound() && !needsWaveLimiter())
-      return Occupancy;
-    return (Occupancy < 4) ? Occupancy : 4;
-  }
+  unsigned getMinAllowedOccupancy(const GCNSubtarget &ST) const;
 
   void limitOccupancy(const MachineFunction &MF);
 
   void limitOccupancy(unsigned Limit) {
     if (Occupancy > Limit)
       Occupancy = Limit;
   }
 

llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp

 //===- SIMachineFunctionInfo.cpp - SI Machine Function Info ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "SIMachineFunctionInfo.h"
SIMachineFunctionInfo::SIMachineFunctionInfo(const MachineFunction &MF)
   if (!S.empty())
     S.consumeInteger(0, HighBitsOf32BitAddress);
 
   S = F.getFnAttribute("amdgpu-gds-size").getValueAsString();
   if (!S.empty())
     S.consumeInteger(0, GDSSize);
 }
 
+unsigned SIMachineFunctionInfo::getMinAllowedOccupancy(const GCNSubtarget &ST) const {
+  if (!isMemoryBound() && !needsWaveLimiter())
+    return Occupancy;
+  // Allow a minimum of 16 threads per SIMD lane, which works out as:
+  // - 4 waves per SIMD for GFX9 and below
+  // - 8 waves per SIMD for GFX10 wave64
+  // - 16 waves per SIMD for GFX10 wave32
+  unsigned MinOccupancy = ST.getTotalNumVGPRs() / 64;
+  return std::min(Occupancy, MinOccupancy);
+}
+
 void SIMachineFunctionInfo::limitOccupancy(const MachineFunction &MF) {
   limitOccupancy(getMaxWavesPerEU());
   const GCNSubtarget& ST = MF.getSubtarget<GCNSubtarget>();
   limitOccupancy(ST.getOccupancyWithLocalMemSize(getLDSSize(),
                  MF.getFunction()));
 }
 
 Register SIMachineFunctionInfo::addPrivateSegmentBuffer(