AMDGPU/SI: Pass whether to use the SI scheduler via Target Attribute

[axeldavy]

Summary

Currently the SI scheduler can be selected via command line option,
but it turned out it would be better if it was selectable via a Target Attribute.

This patch adds "si-scheduler" attribute to the backend.

[• tstellarAMD]

LGTM.

[arsenm]

I thought @echristo wanted to move away from doing things like this

[axeldavy]

The way with the command line argument had some issues with Mesa.
There can be several llvm users at the same time.

For example, d3d apps can use both sw rendering and hw rendering. We'd want the command line
option to apply only to the radeonsi backend, and not to llvmpipe.

Even with that put aside, setting the command line option in a call_once function didn't work reliably
(crash complaining the command line option is set twice):
Several d3d apps were failing. I suspect these apps do create gl contexts in addition to d3d context,
and Mesa does duplicate all the drivers for the d3d lib binary, thus resulting in two call_once functions ...

I guess if app is using clover + gl there would be similar issue.

[echristo]

Getting away from this would be a good thing, i.e. you just added more cleanup now. What are you trying to do here?

-eric

[• tstellarAMD]

In D16192#327175, @echristo wrote:

Getting away from this would be a good thing, i.e. you just added more cleanup now. What are you trying to do here?

-eric

This attribute is for enabling an alternative machine scheduler in the AMDGPU backend. I had originally asked Axel not to add the subtarget feature and instead have Mesa enable the scheduler by passing a command line argument to the C API function LLVMParseCommandLineOptions(), but he ran into some problems as described in his comment.

[echristo]

For your target in particular you only have a single subtarget and so the command line option should be valid at any time you're instantiating a JIT for the gpu yes? The comment wasn't enough for me to piece together what's happening here.

-eric

[• tstellarAMD]

In D16192#327225, @echristo wrote:

For your target in particular you only have a single subtarget and so the command line option should be valid at any time you're instantiating a JIT for the gpu yes? The comment wasn't enough for me to piece together what's happening here.

-eric

The problem is that the command line parser is global, so you can only call it once with a particular option. An application can possibly load multiple drivers which both need to set the command line option: e.g. OpenGL and OpenCL. These drivers have no knowledge of each other and are completely separate. When this happens the second driver to call LLVMParseCommandLine() assertion fails, because this fucntion does not allow a command line option to be set twice.

[echristo]

The way this is set up isn't going to work in general the same as other subtarget features, but as we discussed on IRC if you're willing to keep a separate TargetMachine for each set of subtarget feature changes that you might want to do that affect the pass and code generation pipeline it should be fine.

As a note, this is the sort of thing that the other targets handle in their TargetMachine where it does the caching etc and the subtarget features are set on the function.

-eric

llvm/trunk/lib/Target/AMDGPU/AMDGPU.td

def FeatureSGPRInitBug : SubtargetFeature<"sgpr-init-bug",
         "true",
         "VI SGPR initilization bug requiring a fixed SGPR allocation size">;
 
 def FeatureEnableHugeScratchBuffer : SubtargetFeature<"huge-scratch-buffer",
         "EnableHugeScratchBuffer",
         "true",
         "Enable scratch buffer sizes greater than 128 GB">;
 
+def FeatureEnableSIScheduler : SubtargetFeature<"si-scheduler",
+        "EnableSIScheduler",
+        "true",
+        "Enable SI Machine Scheduler">;
+
 class SubtargetFeatureFetchLimit <string Value> :
                           SubtargetFeature <"fetch"#Value,
         "TexVTXClauseSize",
         Value,
         "Limit the maximum number of fetches in a clause to "#Value>;
 
 def FeatureFetchLimit8 : SubtargetFeatureFetchLimit <"8">;
 def FeatureFetchLimit16 : SubtargetFeatureFetchLimit <"16">;

llvm/trunk/lib/Target/AMDGPU/AMDGPUSubtarget.h

private:
   bool IsGCN;
   bool GCN1Encoding;
   bool GCN3Encoding;
   bool CIInsts;
   bool FeatureDisable;
   int LDSBankCount;
   unsigned IsaVersion;
   bool EnableHugeScratchBuffer;
+  bool EnableSIScheduler;
 
   std::unique_ptr<AMDGPUFrameLowering> FrameLowering;
   std::unique_ptr<AMDGPUTargetLowering> TLInfo;
   std::unique_ptr<AMDGPUInstrInfo> InstrInfo;
   InstrItineraryData InstrItins;
   Triple TargetTriple;
 
 public:
public:
   StringRef getDeviceName() const {
     return DevName;
   }
 
   bool enableHugeScratchBuffer() const {
     return EnableHugeScratchBuffer;
   }
 
+  bool enableSIScheduler() const {
+    return EnableSIScheduler;
+  }
+
   bool dumpCode() const {
     return DumpCode;
   }
   bool r600ALUEncoding() const {
     return R600ALUInst;
   }
   bool isAmdHsaOS() const {
     return TargetTriple.getOS() == Triple::AMDHSA;

llvm/trunk/lib/Target/AMDGPU/AMDGPUSubtarget.cpp

: AMDGPUGenSubtargetInfo(TT, GPU, FS), DevName(GPU), Is64bit(false),
       EnableIfCvt(true), EnableLoadStoreOpt(false),
       EnableUnsafeDSOffsetFolding(false),
       EnableXNACK(false),
       WavefrontSize(0), CFALUBug(false),
       LocalMemorySize(0),
       EnableVGPRSpilling(false), SGPRInitBug(false), IsGCN(false),
       GCN1Encoding(false), GCN3Encoding(false), CIInsts(false), LDSBankCount(0),
       IsaVersion(ISAVersion0_0_0), EnableHugeScratchBuffer(false),
-      FrameLowering(nullptr),
+      EnableSIScheduler(false), FrameLowering(nullptr),
       InstrItins(getInstrItineraryForCPU(GPU)), TargetTriple(TT) {
 
   initializeSubtargetDependencies(TT, GPU, FS);
 
   const unsigned MaxStackAlign = 64 * 16; // Maximum stack alignment (long16)
 
   if (getGeneration() <= AMDGPUSubtarget::NORTHERN_ISLANDS) {
     InstrInfo.reset(new R600InstrInfo(*this));

llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

AMDGPUTargetMachine &getAMDGPUTargetMachine() const {
     return getTM<AMDGPUTargetMachine>();
   }
 
   ScheduleDAGInstrs *
   createMachineScheduler(MachineSchedContext *C) const override {
     const AMDGPUSubtarget &ST = *getAMDGPUTargetMachine().getSubtargetImpl();
     if (ST.getGeneration() <= AMDGPUSubtarget::NORTHERN_ISLANDS)
       return createR600MachineScheduler(C);
+    else if (ST.enableSIScheduler())
+      return createSIMachineScheduler(C);
     return nullptr;
   }
 
   void addIRPasses() override;
   void addCodeGenPrepare() override;
   bool addPreISel() override;
   bool addInstSelector() override;
   bool addGCPasses() override;