Use Flat For 64-bit Global Buffer

[cfang]

Summary

Generate flat instructions for 64-bit global buffer instructions. In this patch, we defined
a subtarget feature: FlatForGlobal, and add this feature for CI. This means for CI, we will
generate a flat instruction for a 64-bit global buffer reference.

[• tstellarAMD]

The commit message should be prefixed with: AMDGPU/SI:

lib/Target/AMDGPU/AMDGPU.td

240

We don't want to enable FeatureFlatForGlobal globally. It should only be used for HSA.

lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

417–418

I think this fix should be in a separate patch.

lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

995–998

You should move the check for useFlatForGlobal() into SelectMUBUF() that way you only need to add it in one place.

1064–1068

You won't need this once you have the check in SelectMUBUF().

lib/Target/AMDGPU/AMDGPUSubtarget.h

73

You need to initialize this value to something in the AMDGPUSubtarget constructor. 'true' when targeting HSA, 'false' otherwise.

lib/Target/AMDGPU/VIInstructions.td

107–112

SInce the patterns below this will be used for both CI and VI now, they should be moved into CIInstructions.td

test/CodeGen/AMDGPU/ci-use-flat-for-global.ll

1–2

Should add a RUN line for non-hsa: -march=amdgcn instead of -mtriple=amdgcn--amdhsa to make sure we are still using the buffer instructions.

[cfang]

Update based on Tom's comments.

[• tstellarAMD]

lib/Target/AMDGPU/AMDGPUSubtarget.cpp

102–105 ↗

(On Diff #43042)

You need to set this value before you call initializeSubtargetDependencies() otherwise users won't be able to toggle this using -mattr=flat-for-global.

test/CodeGen/AMDGPU/ci-use-flat-for-global.ll

2–3

I would add two more run line here to test setting the non-default value for each target using -mattr=

7

You can drop the spir_kernel calling convention here.

30–37

Since we are disabling buffer instructions globally, I think you can use a much more simple test case:

void @test(i32 addrspace(1)* %out) {
entry:
  store i32 0, i32 addrspace(1)* %out
  ret void
}

[cfang]

update based on Tom's reviews:

1. set FlatForGlobal before "initializeSubtargetDependencies()"
2. update LIT test for a simple one and also check around -mattr=(+/-)flat-for-global

[• tstellarAMD]

LGTM.

[• tstellarAMD]

lib/Target/AMDGPU/AMDGPUSubtarget.cpp

81–82 ↗

(On Diff #43053)

The test case you added fails now, because the generation has not been initialized yet. I think we can just check for AmdHsaOS() here.

[cfang]

Hello, tom:

I am taking this afternoon again. Call you make the change or I can do it tomorrow!

Thanks

Changpeng

[cfang]

Don't check the subtarget before the subtarget is set up.

[cfang]

Change the way of initialization for FlatForGlobal to make -mattr=-flat-for-global work;
Fix LIT test failures due to generating flat instructions instead of buffer ones.

[• tstellarAMD]

LGTM.

I think it would be good to explain why we are making this change in the commit message. Here is the commit message I wrote when I was getting ready to commit an earlier version of the change. Feel free to use it:

AMDGPU/SI: Use flat for global load/store when targeting HSA

For some reason doing executing an MUBUF instruction with the addr64
bit set and a zero base pointer in the resource descriptor causes
the memory operation to be dropped when the shader is executed using
the HSA runtime.

This kind of MUBUF instruction is commonly used when the pointer is
stored in VGPRs.  The base pointer field in the resource descriptor
is set to zero and and the pointer is stored in the vaddr field.

This patch resolves the issue by only using flat instructions for
global memory operations when targeting HSA. This is an overly
conservative fix as all other configurations of MUBUF instructions
appear to work.

lib/Target/AMDGPU/CIInstructions.td

36–37 ↗

(On Diff #43415)

Unnecessary whitepsace change, please drop this before you commit.

[cfang]

Commited but was reverted because of a lIT failure on llvm.dbg.ll.

But this is the test I modified and my local test of this passed.

What could happen?

Thanks;

Changpeng

lib/Target/AMDGPU/AMDGPU.td

         "true",
         "Force using DS instruction immediate offsets on SI">;
 
+def FeatureFlatForGlobal : SubtargetFeature<"flat-for-global",
+        "FlatForGlobal",
+        "true",
+        "Force to generate flat instruction for global">;
+
 def FeatureFlatAddressSpace : SubtargetFeature<"flat-address-space",
         "FlatAddressSpace",
         "true",

 def FeatureSeaIslands : SubtargetFeatureGeneration<"SEA_ISLANDS",
         [Feature64BitPtr, FeatureFP64, FeatureLocalMemorySize65536,
          FeatureWavefrontSize64, FeatureGCN, FeatureFlatAddressSpace,
-         FeatureGCN1Encoding, FeatureCIInsts]>;
+         FeatureFlatForGlobal, FeatureGCN1Encoding, FeatureCIInsts]>;
 
 def FeatureVolcanicIslands : SubtargetFeatureGeneration<"VOLCANIC_ISLANDS",
         [Feature64BitPtr, FeatureFP64, FeatureLocalMemorySize65536,

lib/Target/AMDGPU/AMDGPUAsmPrinter.cpp

     }
   }
 
-  if (VCCUsed)
+  if (VCCUsed || FlatUsed)
     MaxSGPR += 2;
 
   if (FlatUsed)

lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp

   SDValue Ptr, Offen, Idxen, Addr64;
 
   // addr64 bit was removed for volcanic islands.
-  if (Subtarget->getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS)
+  // Or Subtarget prefers to use flat instruction.
+  if (Subtarget->getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS ||
+      Subtarget->useFlatForGlobal())
     return false;
 
   SelectMUBUF(Addr, Ptr, VAddr, SOffset, Offset, Offen, Idxen, Addr64,

                                            SDValue &SOffset, SDValue &Offset,
                                            SDValue &GLC, SDValue &SLC,
                                            SDValue &TFE) const {
+
+  // Subtarget prefers to use flat instruction.
+  if (Subtarget->useFlatForGlobal())
+    return false;
+
   SDValue Ptr, VAddr, Offen, Idxen, Addr64;
   const SIInstrInfo *TII =
     static_cast<const SIInstrInfo *>(Subtarget->getInstrInfo());

lib/Target/AMDGPU/AMDGPUSubtarget.h

   bool FastFMAF32;
   bool CaymanISA;
   bool FlatAddressSpace;
+  bool FlatForGlobal;
   bool EnableIRStructurizer;
   bool EnablePromoteAlloca;
   bool EnableIfCvt;

     return FlatAddressSpace;
   }
 
+  bool useFlatForGlobal() const {
+    return FlatForGlobal;
+  }
+
   bool hasBFE() const {
     return (getGeneration() >= EVERGREEN);
   }

lib/Target/AMDGPU/VIInstructions.td

 // SMEM Patterns
 //===----------------------------------------------------------------------===//
 
-let Predicates = [isVI] in {
+def useFlatForGlobal : Predicate <
+  "Subtarget->useFlatForGlobal() || "
+  "Subtarget->getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS">;
+
+let Predicates = [useFlatForGlobal] in {
 
 // 1. Offset as 20bit DWORD immediate
 def : Pat <

test/CodeGen/AMDGPU/ci-use-flat-for-global.ll

+; RUN: llc < %s -mtriple=amdgcn--amdhsa -mcpu=kaveri | FileCheck %s
+
+
+; CHECK-NOT: buffer_store_dword
+; CHECK: flat_store_dword
+define spir_kernel void @test_store(i32 %arg, i32 addrspace(1)* nocapture %arg1, i32 %arg2) #0 {
+bb:
+  %tmp = tail call noalias i8 addrspace(2)* @llvm.amdgcn.dispatch.ptr() #2
+  %tmp7 = getelementptr i8, i8 addrspace(2)* %tmp, i64 4
+  %tmp8 = bitcast i8 addrspace(2)* %tmp7 to i32 addrspace(2)*
+  %tmp9 = load i32, i32 addrspace(2)* %tmp8, align 4
+  %tmp10 = and i32 %tmp9, 65535
+  %tmp11 = tail call i32 @llvm.r600.read.tgid.x() #2
+  %tmp12 = tail call i32 @llvm.r600.read.tidig.x() #2
+  %tmp13 = mul i32 %tmp11, %tmp10
+  %tmp14 = add i32 %tmp12, %tmp13
+  %tmp15 = lshr i32 %tmp9, 16
+  %tmp16 = tail call i32 @llvm.r600.read.tgid.y() #2
+  %tmp17 = tail call i32 @llvm.r600.read.tidig.y() #2
+  %tmp18 = mul i32 %tmp16, %tmp15
+  %tmp19 = add i32 %tmp17, %tmp18
+  %tmp20 = icmp eq i32 %arg, 0
+  %tmp21 = mul nsw i32 %tmp19, %arg2
+  %tmp22 = add nsw i32 %tmp14, %tmp21
+  %tmp23 = sext i32 %tmp22 to i64
+  %tmp24 = getelementptr inbounds i32, i32 addrspace(1)* %arg1, i64 %tmp23
+  br i1 %tmp20, label %bb26, label %bb25
+
+bb25:                                             ; preds = %bb
+  store i32 3, i32 addrspace(1)* %tmp24, align 4
+  br label %bb26
+
+bb26:                                             ; preds = %bb25, %bb
+  store i32 1, i32 addrspace(1)* %tmp24, align 4 
+  ret void
+}
+
+; Function Attrs: nounwind readnone
+declare i8 addrspace(2)* @llvm.amdgcn.dispatch.ptr() #1
+
+; Function Attrs: nounwind readnone
+declare i32 @llvm.r600.read.tgid.x() #1
+
+; Function Attrs: nounwind readnone
+declare i32 @llvm.r600.read.tidig.x() #1
+
+; Function Attrs: nounwind readnone
+declare i32 @llvm.r600.read.tgid.y() #1
+
+; Function Attrs: nounwind readnone
+declare i32 @llvm.r600.read.tidig.y() #1
+