[AMDGPU] Always use IDX for load/store format intrinsics.

[bnieuwenhuizen]

Summary

On Vega, whether the buffer size in the descriptor is interpreted
in bytes or in units of stride depends on whether IDXEN is enabled.

This modifies the format intrinsics to always use IDXEN for GFX9,
so that we can always rely on it being units of stride. Before this
the intrinsics were pretty much unusable as we cannot fill in the
size field without knowing if LLVM will apply the optimization.

I heard people on IRC that they preferred modifying the existing
intrinsic. If you want me to create new ones for the pattern, that
is OK with me.

[nhaehnle]

We need tests for the store case as well.

I think it's fine to keep the existing intrinsics, but this causes a subtle change (or clarification) of semantics. Could you please add a comment in IntrinsicsAMDGPU.td explaining that buffer.{load,store}.format.* expects a buffer with stride != 0, while the non-format intrinsics expect a buffer with stride == 0?

[bnieuwenhuizen]

Added a run fox GFX9 in the store tests and added the comments to the
intrinsics.

[arsenm]

I think the unit change is a hint we should use different intrinsics for this

[mareko]

In D44401#1040175, @arsenm wrote:

I think the unit change is a hint we should use different intrinsics for this

The current intrinsic is unusable on GFX9 though. Why should we keep it as-is if it's unusable? This patch is reasonable because it picks the most expected behavior for GFX9 while it disallows the least expected and problematic behavior.

[nhaehnle]

In D44401#1040539, @mareko wrote:

In D44401#1040175, @arsenm wrote:

I think the unit change is a hint we should use different intrinsics for this

The current intrinsic is unusable on GFX9 though. Why should we keep it as-is if it's unusable? This patch is reasonable because it picks the most expected behavior for GFX9 while it disallows the least expected and problematic behavior.

I agree. We don't have a use for "format" instructions without stride, and we don't have a use for "non-format" instructions with stride, so this change is pragmatic. Arguably, we should remove the "index" parameter from the "non-format" intrinsics, but that's a change requiring a flag-day.

[tpr]

I don't have any objection to this change going in, but I think we need more:

1. LLPC generates indexed atomic ops. This change does not fix that. We have a case where LLVM optimizes an index that is constant 0 to idxen=0, causing a breakage. But probably not all atomic ops are intended to be indexed.

2. This change doesn't do anything to tbuffer ops.

3. Longer term, surely it makes sense to represent the full semantic capability of the ISA by having non-indexed and indexed (raw and structured) variants of all of the intrinsics. That could be done compatibly after this change by assuming the formatted buffer ops are structured and introducing raw variants, and assuming the other buffer ops are raw and introducing structured variants.

[tpr]

Also (in terms of extra changes needed in buffer intrinsics over and above this change):

David Stuttard pointed me at
https://reviews.llvm.org/D30687
in which Tom Stellard pointed out that the way the buffer intrinsics conflate the three offsets (inst_offset, sgpr_offset and vgpr_offset) breaks swizzling, and possibly range checking in swizzling too.

So the suggestion is to have new intrinsics with the three offsets in separate args, in the same way as David's tbuffer intrinsic. Then the new intrinsics would also need to have raw and structured variants.

[nhaehnle]

Indexed atomic ops are a good point. radeonsi and radv should be generating those as well for image atomics on texture buffers.

I doubt we'll ever use *format intrinsics with non-strided buffers, so we will be able to leave those as they are with this change, but for the other buffer intrinsics it looks like we indeed have to go to two sets of intrinsics, with and without stride. But that could be done in a separate change.

include/llvm/IR/IntrinsicsAMDGPU.td

 class AMDGPUBufferLoad : Intrinsic <
   [llvm_anyfloat_ty],
   [llvm_v4i32_ty,     // rsrc(SGPR)
    llvm_i32_ty,       // vindex(VGPR)
    llvm_i32_ty,       // offset(SGPR/VGPR/imm)
    llvm_i1_ty,        // glc(imm)
    llvm_i1_ty],       // slc(imm)
   [IntrReadMem], "", [SDNPMemOperand]>;
+
+// Assumes the buffer has stride != 0.
 def int_amdgcn_buffer_load_format : AMDGPUBufferLoad;
+
+// Assumes the buffer has stride == 0
 def int_amdgcn_buffer_load : AMDGPUBufferLoad;
 
 class AMDGPUBufferStore : Intrinsic <
   [],
   [llvm_anyfloat_ty,  // vdata(VGPR) -- can currently only select f32, v2f32, v4f32
    llvm_v4i32_ty,     // rsrc(SGPR)
    llvm_i32_ty,       // vindex(VGPR)
    llvm_i32_ty,       // offset(SGPR/VGPR/imm)
    llvm_i1_ty,        // glc(imm)
    llvm_i1_ty],       // slc(imm)
   [IntrWriteMem], "", [SDNPMemOperand]>;
+
+// Assumes the buffer has stride != 0
 def int_amdgcn_buffer_store_format : AMDGPUBufferStore;
+
+// Assumes the buffer has stride == 0
 def int_amdgcn_buffer_store : AMDGPUBufferStore;
 
 def int_amdgcn_tbuffer_load : Intrinsic <
     [llvm_any_ty],    // overloaded for types f32/i32, v2f32/v2i32, v4f32/v4i32
     [llvm_v4i32_ty,   // rsrc(SGPR)
      llvm_i32_ty,     // vindex(VGPR)
      llvm_i32_ty,     // voffset(VGPR)
      llvm_i32_ty,     // soffset(SGPR)

lib/Target/AMDGPU/AMDGPU.td

def HasIntClamp : Predicate<"Subtarget->hasIntClamp()">,
   AssemblerPredicate<"FeatureIntClamp">;
 
 def HasMadMixInsts : Predicate<"Subtarget->hasMadMixInsts()">,
   AssemblerPredicate<"FeatureMadMixInsts">;
 
 def EnableLateCFGStructurize : Predicate<
   "EnableLateStructurizeCFG">;
 
+// On GFX9 IDXEN matters for interpretation of buffer sizes.
+def IDXENEliminationAlwaysAllowed : Predicate <
+  "Subtarget->getGeneration() < AMDGPUSubtarget::GFX9">;
+
 // Exists to help track down where SubtargetPredicate isn't set rather
 // than letting tablegen crash with an unhelpful error.
 def InvalidPred : Predicate<"predicate not set on instruction or pattern">;
 
 class PredicateControl {
   Predicate SubtargetPredicate = InvalidPred;
   Predicate SIAssemblerPredicate = isSICI;
   Predicate VIAssemblerPredicate = isVI;

lib/Target/AMDGPU/BUFInstructions.td

(vt (name v4i32:$rsrc, i32:$vindex,
               imm:$glc, imm:$slc)),
     (!cast<MUBUF_Pseudo>(opcode # _BOTHEN)
       (REG_SEQUENCE VReg_64, $vindex, sub0, $voffset, sub1),
       $rsrc, $soffset, (as_i16imm $offset),
       (as_i1imm $glc), (as_i1imm $slc), 0)
   >;
 }
 
-defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format, f32, "BUFFER_LOAD_FORMAT_X">;
-defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format, v2f32, "BUFFER_LOAD_FORMAT_XY">;
-defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format, v4f32, "BUFFER_LOAD_FORMAT_XYZW">;
+multiclass MUBUF_IndexedLoadIntrinsicPat<SDPatternOperator name, ValueType vt,
+                                         string opcode> {
+  let SubtargetPredicate = IDXENEliminationAlwaysAllowed in {
+    def : GCNPat<
+      (vt (name v4i32:$rsrc, 0,
+                (MUBUFIntrinsicOffset i32:$soffset, i16:$offset),
+                imm:$glc, imm:$slc)),
+      (!cast<MUBUF_Pseudo>(opcode # _OFFSET) $rsrc, $soffset, (as_i16imm $offset),
+        (as_i1imm $glc), (as_i1imm $slc), 0)
+    >;
+
+    def : GCNPat<
+      (vt (name v4i32:$rsrc, 0,
+                (MUBUFIntrinsicVOffset i32:$soffset, i16:$offset, i32:$voffset),
+                imm:$glc, imm:$slc)),
+      (!cast<MUBUF_Pseudo>(opcode # _OFFEN) $voffset, $rsrc, $soffset, (as_i16imm $offset),
+        (as_i1imm $glc), (as_i1imm $slc), 0)
+    >;
+  }
+
+  def : GCNPat<
+    (vt (name v4i32:$rsrc, i32:$vindex,
+              (MUBUFIntrinsicOffset i32:$soffset, i16:$offset),
+              imm:$glc, imm:$slc)),
+    (!cast<MUBUF_Pseudo>(opcode # _IDXEN) $vindex, $rsrc, $soffset, (as_i16imm $offset),
+      (as_i1imm $glc), (as_i1imm $slc), 0)
+  >;
+
+  def : GCNPat<
+    (vt (name v4i32:$rsrc, i32:$vindex,
+              (MUBUFIntrinsicVOffset i32:$soffset, i16:$offset, i32:$voffset),
+              imm:$glc, imm:$slc)),
+    (!cast<MUBUF_Pseudo>(opcode # _BOTHEN)
+      (REG_SEQUENCE VReg_64, $vindex, sub0, $voffset, sub1),
+      $rsrc, $soffset, (as_i16imm $offset),
+      (as_i1imm $glc), (as_i1imm $slc), 0)
+  >;
+}
+
+defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format, f32, "BUFFER_LOAD_FORMAT_X">;
+defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format, v2f32, "BUFFER_LOAD_FORMAT_XY">;
+defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format, v4f32, "BUFFER_LOAD_FORMAT_XYZW">;
 
 let SubtargetPredicate = HasUnpackedD16VMem in {
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format, f16, "BUFFER_LOAD_FORMAT_D16_X_gfx80">;
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format_d16, v2i32, "BUFFER_LOAD_FORMAT_D16_XY_gfx80">;
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format_d16, v4i32, "BUFFER_LOAD_FORMAT_D16_XYZW_gfx80">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format, f16, "BUFFER_LOAD_FORMAT_D16_X_gfx80">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format_d16, v2i32, "BUFFER_LOAD_FORMAT_D16_XY_gfx80">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format_d16, v4i32, "BUFFER_LOAD_FORMAT_D16_XYZW_gfx80">;
 } // End HasUnpackedD16VMem.
 
 let SubtargetPredicate = HasPackedD16VMem in {
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format, f16, "BUFFER_LOAD_FORMAT_D16_X">;
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format, v2f16, "BUFFER_LOAD_FORMAT_D16_XY">;
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format_d16, i32, "BUFFER_LOAD_FORMAT_D16_XY">;
-  defm : MUBUF_LoadIntrinsicPat<SIbuffer_load_format_d16, v2i32, "BUFFER_LOAD_FORMAT_D16_XYZW">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format, f16, "BUFFER_LOAD_FORMAT_D16_X">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format, v2f16, "BUFFER_LOAD_FORMAT_D16_XY">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format_d16, i32, "BUFFER_LOAD_FORMAT_D16_XY">;
+  defm : MUBUF_IndexedLoadIntrinsicPat<SIbuffer_load_format_d16, v2i32, "BUFFER_LOAD_FORMAT_D16_XYZW">;
 } // End HasPackedD16VMem.
 
 defm : MUBUF_LoadIntrinsicPat<SIbuffer_load, f32, "BUFFER_LOAD_DWORD">;
 defm : MUBUF_LoadIntrinsicPat<SIbuffer_load, v2f32, "BUFFER_LOAD_DWORDX2">;
 defm : MUBUF_LoadIntrinsicPat<SIbuffer_load, v4f32, "BUFFER_LOAD_DWORDX4">;
 
 multiclass MUBUF_StoreIntrinsicPat<SDPatternOperator name, ValueType vt,
                                    string opcode> {
def : GCNPat<
     (!cast<MUBUF_Pseudo>(opcode # _BOTHEN_exact)
       $vdata,
       (REG_SEQUENCE VReg_64, $vindex, sub0, $voffset, sub1),
       $rsrc, $soffset, (as_i16imm $offset),
       (as_i1imm $glc), (as_i1imm $slc), 0)
   >;
 }
 
-defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format, f32, "BUFFER_STORE_FORMAT_X">;
-defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format, v2f32, "BUFFER_STORE_FORMAT_XY">;
-defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format, v4f32, "BUFFER_STORE_FORMAT_XYZW">;
+multiclass MUBUF_IndexedStoreIntrinsicPat<SDPatternOperator name, ValueType vt,
+                                          string opcode> {
+  let SubtargetPredicate = IDXENEliminationAlwaysAllowed in {
+    def : GCNPat<
+      (name vt:$vdata, v4i32:$rsrc, 0,
+            (MUBUFIntrinsicOffset i32:$soffset, i16:$offset),
+            imm:$glc, imm:$slc),
+      (!cast<MUBUF_Pseudo>(opcode # _OFFSET_exact) $vdata, $rsrc, $soffset, (as_i16imm $offset),
+                                      (as_i1imm $glc), (as_i1imm $slc), 0)
+    >;
+
+    def : GCNPat<
+      (name vt:$vdata, v4i32:$rsrc, 0,
+            (MUBUFIntrinsicVOffset i32:$soffset, i16:$offset, i32:$voffset),
+            imm:$glc, imm:$slc),
+      (!cast<MUBUF_Pseudo>(opcode # _OFFEN_exact) $vdata, $voffset, $rsrc, $soffset,
+                                     (as_i16imm $offset), (as_i1imm $glc),
+                                     (as_i1imm $slc), 0)
+  >;
+  }
+
+  def : GCNPat<
+    (name vt:$vdata, v4i32:$rsrc, i32:$vindex,
+          (MUBUFIntrinsicOffset i32:$soffset, i16:$offset),
+          imm:$glc, imm:$slc),
+    (!cast<MUBUF_Pseudo>(opcode # _IDXEN_exact) $vdata, $vindex, $rsrc, $soffset,
+                                   (as_i16imm $offset), (as_i1imm $glc),
+                                   (as_i1imm $slc), 0)
+  >;
+
+  def : GCNPat<
+    (name vt:$vdata, v4i32:$rsrc, i32:$vindex,
+          (MUBUFIntrinsicVOffset i32:$soffset, i16:$offset, i32:$voffset),
+          imm:$glc, imm:$slc),
+    (!cast<MUBUF_Pseudo>(opcode # _BOTHEN_exact)
+      $vdata,
+      (REG_SEQUENCE VReg_64, $vindex, sub0, $voffset, sub1),
+      $rsrc, $soffset, (as_i16imm $offset),
+      (as_i1imm $glc), (as_i1imm $slc), 0)
+  >;
+}
+
+defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format, f32, "BUFFER_STORE_FORMAT_X">;
+defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format, v2f32, "BUFFER_STORE_FORMAT_XY">;
+defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format, v4f32, "BUFFER_STORE_FORMAT_XYZW">;
 
 let SubtargetPredicate = HasUnpackedD16VMem in {
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, f16, "BUFFER_STORE_FORMAT_D16_X_gfx80">;
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, v2i32, "BUFFER_STORE_FORMAT_D16_XY_gfx80">;
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, v4i32, "BUFFER_STORE_FORMAT_D16_XYZW_gfx80">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, f16, "BUFFER_STORE_FORMAT_D16_X_gfx80">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, v2i32, "BUFFER_STORE_FORMAT_D16_XY_gfx80">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, v4i32, "BUFFER_STORE_FORMAT_D16_XYZW_gfx80">;
 } // End HasUnpackedD16VMem.
 
 let SubtargetPredicate = HasPackedD16VMem in {
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, f16, "BUFFER_STORE_FORMAT_D16_X">;
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, v2f16, "BUFFER_STORE_FORMAT_D16_XY">;
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, i32, "BUFFER_STORE_FORMAT_D16_XY">;
-  defm : MUBUF_StoreIntrinsicPat<SIbuffer_store_format_d16, v2i32, "BUFFER_STORE_FORMAT_D16_XYZW">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, f16, "BUFFER_STORE_FORMAT_D16_X">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, v2f16, "BUFFER_STORE_FORMAT_D16_XY">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, i32, "BUFFER_STORE_FORMAT_D16_XY">;
+  defm : MUBUF_IndexedStoreIntrinsicPat<SIbuffer_store_format_d16, v2i32, "BUFFER_STORE_FORMAT_D16_XYZW">;
 } // End HasPackedD16VMem.
 
 defm : MUBUF_StoreIntrinsicPat<SIbuffer_store, f32, "BUFFER_STORE_DWORD">;
 defm : MUBUF_StoreIntrinsicPat<SIbuffer_store, v2f32, "BUFFER_STORE_DWORDX2">;
 defm : MUBUF_StoreIntrinsicPat<SIbuffer_store, v4f32, "BUFFER_STORE_DWORDX4">;
 
 //===----------------------------------------------------------------------===//
 // buffer_atomic patterns

test/CodeGen/AMDGPU/llvm.amdgcn.buffer.load.format.ll

 ;RUN: llc < %s -march=amdgcn -mcpu=verde -verify-machineinstrs | FileCheck %s -check-prefix=CHECK -check-prefix=SICI
 ;RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s -check-prefix=CHECK -check-prefix=VI
+;RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck %s -check-prefix=CHECK -check-prefix=GFX9
 
 ;CHECK-LABEL: {{^}}buffer_load:
-;CHECK: buffer_load_format_xyzw v[0:3], off, s[0:3], 0
-;CHECK: buffer_load_format_xyzw v[4:7], off, s[0:3], 0 glc
-;CHECK: buffer_load_format_xyzw v[8:11], off, s[0:3], 0 slc
+;SICI: buffer_load_format_xyzw v[0:3], off, s[0:3], 0
+;SICI: buffer_load_format_xyzw v[4:7], off, s[0:3], 0 glc
+;SICI: buffer_load_format_xyzw v[8:11], off, s[0:3], 0 slc
+;VI: buffer_load_format_xyzw v[0:3], off, s[0:3], 0
+;VI: buffer_load_format_xyzw v[4:7], off, s[0:3], 0 glc
+;VI: buffer_load_format_xyzw v[8:11], off, s[0:3], 0 slc
+;GFX9: buffer_load_format_xyzw v[0:3], {{v[0-9]+}}, s[0:3], 0 idxen
+;GFX9: buffer_load_format_xyzw v[4:7], {{v[0-9]+}}, s[0:3], 0 idxen glc
+;GFX9: buffer_load_format_xyzw v[8:11], {{v[0-9]+}}, s[0:3], 0 idxen slc
 ;CHECK: s_waitcnt
 define amdgpu_ps {<4 x float>, <4 x float>, <4 x float>} @buffer_load(<4 x i32> inreg) {
 main_body:
   %data = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 0, i32 0, i1 0, i1 0)
   %data_glc = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 0, i32 0, i1 1, i1 0)
   %data_slc = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 0, i32 0, i1 0, i1 1)
   %r0 = insertvalue {<4 x float>, <4 x float>, <4 x float>} undef, <4 x float> %data, 0
   %r1 = insertvalue {<4 x float>, <4 x float>, <4 x float>} %r0, <4 x float> %data_glc, 1
   %r2 = insertvalue {<4 x float>, <4 x float>, <4 x float>} %r1, <4 x float> %data_slc, 2
   ret {<4 x float>, <4 x float>, <4 x float>} %r2
 }
 
 ;CHECK-LABEL: {{^}}buffer_load_immoffs:
-;CHECK: buffer_load_format_xyzw v[0:3], off, s[0:3], 0 offset:42
+;SICI: buffer_load_format_xyzw v[0:3], off, s[0:3], 0 offset:42
+;VI: buffer_load_format_xyzw v[0:3], off, s[0:3], 0 offset:42
+;GFX9: buffer_load_format_xyzw v[0:3], {{v[0-9]+}}, s[0:3], 0 idxen offset:42
 ;CHECK: s_waitcnt
 define amdgpu_ps <4 x float> @buffer_load_immoffs(<4 x i32> inreg) {
 main_body:
   %data = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 0, i32 42, i1 0, i1 0)
   ret <4 x float> %data
 }
 
 ;CHECK-LABEL: {{^}}buffer_load_immoffs_large:

 ;CHECK: s_waitcnt
 define amdgpu_ps <4 x float> @buffer_load_idx(<4 x i32> inreg, i32) {
 main_body:
   %data = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 %1, i32 0, i1 0, i1 0)
   ret <4 x float> %data
 }
 
 ;CHECK-LABEL: {{^}}buffer_load_ofs:
-;CHECK: buffer_load_format_xyzw v[0:3], v0, s[0:3], 0 offen
+;SICI: buffer_load_format_xyzw v[0:3], v0, s[0:3], 0 offen
+;VI: buffer_load_format_xyzw v[0:3], v0, s[0:3], 0 offen
+;GFX9: buffer_load_format_xyzw v[0:3], v[0:1], s[0:3], 0 idxen offen
 ;CHECK: s_waitcnt
 define amdgpu_ps <4 x float> @buffer_load_ofs(<4 x i32> inreg, i32) {
 main_body:
   %data = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 0, i32 %1, i1 0, i1 0)
   ret <4 x float> %data
 }
 
 ;CHECK-LABEL: {{^}}buffer_load_ofs_imm:
-;CHECK: buffer_load_format_xyzw v[0:3], v0, s[0:3], 0 offen offset:60
+;SICI: buffer_load_format_xyzw v[0:3], v0, s[0:3], 0 offen offset:60
+;VI: buffer_load_format_xyzw v[0:3], v0, s[0:3], 0 offen offset:60
+;GFX9: buffer_load_format_xyzw v[0:3], v[0:1], s[0:3], 0 idxen offen offset:60
 ;CHECK: s_waitcnt
 define amdgpu_ps <4 x float> @buffer_load_ofs_imm(<4 x i32> inreg, i32) {
 main_body:
   %ofs = add i32 %1, 60
   %data = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 0, i32 %ofs, i1 0, i1 0)
   ret <4 x float> %data
 }
 

 ;CHECK: s_waitcnt
 define amdgpu_ps <4 x float> @buffer_load_both_reversed(<4 x i32> inreg, i32, i32) {
 main_body:
   %data = call <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32> %0, i32 %2, i32 %1, i1 0, i1 0)
   ret <4 x float> %data
 }
 
 ;CHECK-LABEL: {{^}}buffer_load_x:
-;CHECK: buffer_load_format_x v0, off, s[0:3], 0
+;SICI: buffer_load_format_x v0, off, s[0:3], 0
+;VI: buffer_load_format_x v0, off, s[0:3], 0
+;GFX9: buffer_load_format_x v0, v0, s[0:3], 0 idxen
 ;CHECK: s_waitcnt
 define amdgpu_ps float @buffer_load_x(<4 x i32> inreg %rsrc) {
 main_body:
   %data = call float @llvm.amdgcn.buffer.load.format.f32(<4 x i32> %rsrc, i32 0, i32 0, i1 0, i1 0)
   ret float %data
 }
 
 ;CHECK-LABEL: {{^}}buffer_load_xy:
-;CHECK: buffer_load_format_xy v[0:1], off, s[0:3], 0
+;SICI: buffer_load_format_xy v[0:1], off, s[0:3], 0
+;VI: buffer_load_format_xy v[0:1], off, s[0:3], 0
+;GFX9: buffer_load_format_xy v[0:1], v0, s[0:3], 0 idxen
 ;CHECK: s_waitcnt
 define amdgpu_ps <2 x float> @buffer_load_xy(<4 x i32> inreg %rsrc) {
 main_body:
   %data = call <2 x float> @llvm.amdgcn.buffer.load.format.v2f32(<4 x i32> %rsrc, i32 0, i32 0, i1 0, i1 0)
   ret <2 x float> %data
 }
 
 declare float @llvm.amdgcn.buffer.load.format.f32(<4 x i32>, i32, i32, i1, i1) #0
 declare <2 x float> @llvm.amdgcn.buffer.load.format.v2f32(<4 x i32>, i32, i32, i1, i1) #0
 declare <4 x float> @llvm.amdgcn.buffer.load.format.v4f32(<4 x i32>, i32, i32, i1, i1) #0
 
 attributes #0 = { nounwind readonly }

test/CodeGen/AMDGPU/llvm.amdgcn.buffer.store.format.ll

-;RUN: llc < %s -march=amdgcn -mcpu=verde -verify-machineinstrs | FileCheck %s
-;RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s
+;RUN: llc < %s -march=amdgcn -mcpu=verde -verify-machineinstrs | FileCheck %s -check-prefix=CHECK -check-prefix=SICIVI
+;RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s -check-prefix=CHECK -check-prefix=SICIVI
+;RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck %s -check-prefix=CHECK -check-prefix=GFX9
 
 ;CHECK-LABEL: {{^}}buffer_store:
 ;CHECK-NOT: s_waitcnt
-;CHECK: buffer_store_format_xyzw v[0:3], off, s[0:3], 0
-;CHECK: buffer_store_format_xyzw v[4:7], off, s[0:3], 0 glc
-;CHECK: buffer_store_format_xyzw v[8:11], off, s[0:3], 0 slc
+;SICIVI: buffer_store_format_xyzw v[0:3], off, s[0:3], 0
+;SICIVI: buffer_store_format_xyzw v[4:7], off, s[0:3], 0 glc
+;SICIVI: buffer_store_format_xyzw v[8:11], off, s[0:3], 0 slc
+;GFX9: buffer_store_format_xyzw v[0:3], {{v[0-9]+}}, s[0:3], 0 idxen
+;GFX9: buffer_store_format_xyzw v[4:7], {{v[0-9]+}}, s[0:3], 0 idxen glc
+;GFX9: buffer_store_format_xyzw v[8:11], {{v[0-9]+}}, s[0:3], 0 idxen slc
 define amdgpu_ps void @buffer_store(<4 x i32> inreg, <4 x float>, <4 x float>, <4 x float>) {
 main_body:
   call void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float> %1, <4 x i32> %0, i32 0, i32 0, i1 0, i1 0)
   call void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float> %2, <4 x i32> %0, i32 0, i32 0, i1 1, i1 0)
   call void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float> %3, <4 x i32> %0, i32 0, i32 0, i1 0, i1 1)
   ret void
 }
 
 ;CHECK-LABEL: {{^}}buffer_store_immoffs:
 ;CHECK-NOT: s_waitcnt
-;CHECK: buffer_store_format_xyzw v[0:3], off, s[0:3], 0 offset:42
+;SICIVI: buffer_store_format_xyzw v[0:3], off, s[0:3], 0 offset:42
+;GFX9: buffer_store_format_xyzw v[0:3], {{v[0-9]+}}, s[0:3], 0 idxen offset:42
 define amdgpu_ps void @buffer_store_immoffs(<4 x i32> inreg, <4 x float>) {
 main_body:
   call void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float> %1, <4 x i32> %0, i32 0, i32 42, i1 0, i1 0)
   ret void
 }
 
 ;CHECK-LABEL: {{^}}buffer_store_idx:
 ;CHECK-NOT: s_waitcnt
 ;CHECK: buffer_store_format_xyzw v[0:3], v4, s[0:3], 0 idxen
 define amdgpu_ps void @buffer_store_idx(<4 x i32> inreg, <4 x float>, i32) {
 main_body:
   call void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float> %1, <4 x i32> %0, i32 %2, i32 0, i1 0, i1 0)
   ret void
 }
 
 ;CHECK-LABEL: {{^}}buffer_store_ofs:
 ;CHECK-NOT: s_waitcnt
-;CHECK: buffer_store_format_xyzw v[0:3], v4, s[0:3], 0 offen
+;SICIVI: buffer_store_format_xyzw v[0:3], v4, s[0:3], 0 offen
+;GFX9: buffer_store_format_xyzw v[0:3], v[4:5], s[0:3], 0 idxen offen
 define amdgpu_ps void @buffer_store_ofs(<4 x i32> inreg, <4 x float>, i32) {
 main_body:
   call void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float> %1, <4 x i32> %0, i32 0, i32 %2, i1 0, i1 0)
   ret void
 }
 
 ;CHECK-LABEL: {{^}}buffer_store_both:
 ;CHECK-NOT: s_waitcnt