[AMDGPU] Add intrinsics for tbuffer load and store

[dstuttard]

Summary

Intrinsic already existed for llvm.SI.tbuffer.store

Needed tbuffer.load and also re-implementing the intrinsic as llvm.amdgcn.tbuffer.*

Added CodeGen tests for the 2 new variants added.
Left the original llvm.SI.tbuffer.store implementation to avoid issues with existing code

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

443

This should be part of the mangled return type, I don't think there's any reason to have to add this here

449–453

I don't think we can directly expose offen/idxen.

tfe definitely can't be exposed this way because it changes the register class of the output

[tstellar]

include/llvm/IR/IntrinsicsAMDGPU.td

449–453

We need some solution to deal with swizzled addressing, because these have really complicated clamping rules. The easiest thing to do is to expose all fields so the user is responsible for making sure the operands are correct.

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

449–453

I was wondering if we could use fat pointers, and then have a constant argument hint for what the swizzle factor in the resource is so we can match the addressing mode. Would that work?

[tstellar]

include/llvm/IR/IntrinsicsAMDGPU.td

449–453

I think so, you could also use different address spaces to indicate swizzling vs non-swizzling.

[dstuttard]

See inline comments.

Is this worth looking at for now if there is a different re-factor already in progress?

include/llvm/IR/IntrinsicsAMDGPU.td

443

Isn't the issue that the 3 dword variant can't be generated as we can only have 1,2 and 4 dword return types?

449–453

I think I need to look at this again, in particular the fat pointers. This initial implementation was based on extending the pre-existing llvm.SI.tbuffer.store that already existed.
Matt has also pointed out that there is a re-factor of this in progress already (for store). I guess it makes sense to wait for that as well?

[arsenm]

include/llvm/IR/IntrinsicsAMDGPU.td

443

We can have a 3x dword return type in the intrinsic. The only issue is the DAG doesn't treat this as a legal machine type. I have patches to fix this, and we can work around this in codegen. Long term this won't be an issue with global isel.

[dstuttard]

I've rewritten the implementation of tbuffer load and store and based it around buffer.load.format
and buffer.store.format

This has resulted in a cleaner intrinsic that doesn't include all the instruction fields (no enable
bits, no tfe, single index and offset fields that determine what to do based on what is passed in)

No support yet for the 3 dword variants - but this is a step along the way

This doesn't include fat pointer support - something that can be added later when this is rolled out
more widely

I've also removed the old llvm.SI.tbuffer.store implementation and replaced the uses of it in the lit
testing (this part can be reverted if required).

[tstellar]

include/llvm/IR/IntrinsicsAMDGPU.td

456

Will these be able to support swizzled addressing with only one offset field?

[dstuttard]

include/llvm/IR/IntrinsicsAMDGPU.td

456

This implementation mirrors the buffer.load.format and buffer.store.format intrinsics, so presumably they would suffer from the same problem?

You can generate an instruction with both the index and offset enable bits set (and vgpr offset and index), as well as an offset like this:

%offs.2  = add i32 %offs, 52
%vdata   = call <4 x i32> @llvm.amdgcn.tbuffer.load.v4i32(<4 x i32> %0, i32 %vindex, i32 %offs.2, i32 14, i32 4, i1 0, i1 0)

which results in an instruction like this:

	tbuffer_load_format_xyzw v[0:3], v[0:1], s[0:3],  dfmt:14,  nfmt:4, 0 idxen offen offset:52

does that give you everything required for a swizzle?

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

3050–3051

Replacing the uses in the test is fine, but we need to keep this around until mesa is updated to use the new intrinsic

[dstuttard]

Reintroduce the legacy llvm.SI.tbuffer.store intrinsic

I've left the tests using the new implementation, but have included the old test for this
intrinsic.

In the end the easiest way to do this was to re-insert the old code and rename with "legacy" tags as
appropriate.
Hopefully at some point this can be removed.

[tstellar]

include/llvm/IR/IntrinsicsAMDGPU.td

456

You should review the "Range Checking" section in the ISA docs for all generations. If you arbitrarily move offset values between soffset, inst_offset and vgpr_offset, you risk failing the range checks, which would turn the instruction into a nop.

I think the existing intrinsics are probably broken in the same way, but I think we need to come up with some solution here to
avoid generating broken code.

I think the easiest thing to do is to expose all offset fields, inst_offset, vgpr_offset, and soffset through the intrinsic, and then have the backend not try to optimize the offsets.

We can always go in later and add compiler hints to let the compiler know when it's safe to move the offsets.

[dstuttard]

As suggested by Tom Stellard (due to potential issues with range checking) , I've changed the
intrinsics to have explicit operands for vindex, voffset, soffset and offset. The backend no longer
attempts to optimise by folding in any offsets it can spot in preceding instructions.

I've used separate vindex and voffset operands which means that the idxen and offen flags aren't
required in the intrinsic itself (and the compiler does the appropriate thing to fold the values
into a reg_sequence when both are used). To me this seems a little cleaner.

A later change (again suggested by Tom) might be to re-enable some of these optimisations with
appropriate compiler hints.

[dstuttard]

ping

[dstuttard]

Any chance of another review of this change?
Does it look good to go now?
Thanks

[arsenm]

Needs some assembler tests for the new nfmt/dfmt parsing

include/llvm/IR/IntrinsicsAMDGPU.td

441

any_ty, so f32 is a valid type too

lib/Target/AMDGPU/SIISelLowering.cpp

2998–3001

This isn't the right place for this, although it is what other intrinsics are doing right now. As a follow up patch it would be good to move the MMO creation into getTgtMemIntrinsic

test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.load.ll

5

Should use GCN check prefix (and space after ;)

test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.store.ll

3

Ditto

[dstuttard]

Data can now be float or int
I decided that since the load variant would support this, so should the store. This meant re-jigging
the implementation slightly to use an approach more similar to load to enable the use of any for the
store as well (unless you can see a more efficient way to do it).

Noted about the MMO creation - I'll look into a later patch for this.

Made suggested changes to the tbuffer.load and tbuffer.store tests.
Added some more tests for the assembler etc. (See extra tests in MC/AMDGPU and
MC/Disassembler/AMDGPU

[dstuttard]

Forgot to update comment about float as an option for overloading

[arsenm]

lib/Target/AMDGPU/BUFInstructions.td

85

You shouldn't need to change anything about the encoding for the legacy intrinsics. They should purely be an input issue. I don'ts any repeated MTBUF encoding class, so this seems to be an unnecessary rename?

1246–1247

Again we don't need separate machine instruction or encoding definitions to support the old intrinsics

lib/Target/AMDGPU/SIISelLowering.cpp

3141

You shouldn't need a separate node here. You can insert whatever code is necessary to convert to the new intrinsic/node operands are here, such as bitcasting the types or inserting missing constants etc.

[dstuttard]

See responses to your individual comments - it could be that I've missed something in the way this could be done.

I was reluctant to make big changes to the original implementation (other than renaming) as:

• I wanted to get the change in relatively quickly
• I didn't want to risk breaking something in the legacy support that would mean errors creeping in to Mesa implementation for example

I think that the implementation could be simplified if the SIISelLowering code for the legacy intrinsic did the transformation to the new style at that point (as you suggest), but the issue with that is that the new implementation doesn't support the same fields as the old one (TFE, IDXEN, OFFEN). It could be that ignoring TFE is a valid thing to do and won't break anything, and it should be ok to ignore IDXEN and OFFEN, but is that definite?

lib/Target/AMDGPU/BUFInstructions.td

85

Unless I'm missing something - isn't the problem here that the pseudo instructions are using a different approach to representing the offset and idx enable (amongst others).
I was wary of changing the old implementation too much in order to preserve the old mechanism without any risk of breaking it.

lib/Target/AMDGPU/SIISelLowering.cpp

3141

Part of the issue is that the legacy intrinsic has TFE which isn't present in the new one. If TFE is never used it could be done - but it would effectively ignore this field (unless the intrinsic is changed to remove it too).

We could get this lowering code to check that the offen, idxen are correct given the parameters that have been passed in, and perhaps to make sure that tfe is never used (and assert if it is).

Not sure this is necessarily the approach we should take given that the only reason the legacy support is still there is to make sure that code using this approach doesn't break.

[arsenm]

lib/Target/AMDGPU/BUFInstructions.td

85

The pseudo instructions here are not related to the operands at all. We define pseudo and real encoding classes for each instruction encoding type because in VI, the instruction encoding changed. The pseudo instructions allow codegen to ignore this detail, and then the MC emission can select the right real encoding opcode at the last possible moment. It is entirely disconnected from the intrinsics.

MUBUF instructions have all the combinations of idxen and offen enabled, this again isn't related to the input intrinsics. There are actually this many different addressing mode options that need to be represented with different instructions, this isn't changing.

[dstuttard]

lib/Target/AMDGPU/BUFInstructions.td

85

I misunderstood your original comment. I took it to mean that the legacy and new implementation could be unified at this point.

This is the original implementation of the legacy intrinsic (with the name changed to indicate it is the old implementation), I didn't think there was much point in tidying up the implementation. However, if you think it is worthwhile I'll make the necessary changes (I'm inclined to agree that the legacy code should be shortened if there is an easy option to do so).

[dstuttard]

Removed the bulk of the legacy implementation and now lower to the new form earlier

Some issues to note:

1. tfe can't be used in the legacy intrinsic (asserts if this is detected). The old implementation

wouldn't have worked if tfe was enabled anyway so this isn't a loss of functionality

2. idxen and offen can't both be used - this is supported by the new intrinsic, but the legacy

implementation wouldn't have worked in this mode anyway. Rather than re-write the legacy intrinsic
to support 1 or 2 dword VAddr operands (which is what is required) I've just added an assert to
catch this case. Again - this isn't a loss of functionality over the old intrinsic anyway.

3. The legacy intrinsic supports 3-vec form (TBUFFER_STORE_FORMAT_XYZ) which I've also added in this

updated implementation for the legacy intrinsic. The new intrinsic doesn't support it at the moment
and will have to be added at some point.

[arsenm]

LGTM besides formatting issues

lib/Target/AMDGPU/SIISelLowering.cpp

3130

80 column limit (a few other places too)

3136

Variable name styles

[dstuttard]

Corrected 80 column formatting and variable names

[dstuttard]

Inadvertently included extra changes in last diff

[arsenm]

LGTM

include/llvm/IR/IntrinsicsAMDGPU.td

class AMDGPUBufferStore : Intrinsic <
    llvm_i32_ty,       // vindex(VGPR)
    llvm_i32_ty,       // offset(SGPR/VGPR/imm)
    llvm_i1_ty,        // glc(imm)
    llvm_i1_ty],       // slc(imm)
   [IntrWriteMem]>;
 def int_amdgcn_buffer_store_format : AMDGPUBufferStore;
 def int_amdgcn_buffer_store : AMDGPUBufferStore;
 
+def int_amdgcn_tbuffer_load : Intrinsic <
+    [llvm_anyint_ty], // overloaded for types i32, v2i32, v4i32
+    [llvm_v4i32_ty,   // rsrc(SGPR)
+     llvm_i32_ty,     // num_channels(imm), selects opcode suffix: 1=X, 2=XY, 3=XYZ, 4=XYZW
+     llvm_i32_ty,     // vaddr(VGPR)
+     llvm_i32_ty,     // soffset(SGPR)
+     llvm_i32_ty,     // inst_offset(imm)
+     llvm_i32_ty,     // dfmt(imm)
+     llvm_i32_ty,     // nfmt(imm)
+     llvm_i32_ty,     // offen(imm)
+     llvm_i32_ty,     // idxen(imm)
+     llvm_i32_ty,     // glc(imm)
+     llvm_i32_ty,     // slc(imm)
+     llvm_i32_ty],    // tfe(imm)
+    []>;
+
+def int_amdgcn_tbuffer_store : Intrinsic <
+    [],
+    [llvm_v4i32_ty,  // rsrc(SGPR)
+     llvm_anyint_ty, // vdata(VGPR), overloaded for types i32, v2i32, v4i32
+     llvm_i32_ty,    // num_channels(imm), selects opcode suffix: 1=X, 2=XY, 3=XYZ, 4=XYZW
+     llvm_i32_ty,    // vaddr(VGPR)
+     llvm_i32_ty,    // soffset(SGPR)
+     llvm_i32_ty,    // inst_offset(imm)
+     llvm_i32_ty,    // dfmt(imm)
+     llvm_i32_ty,    // nfmt(imm)
+     llvm_i32_ty,    // offen(imm)
+     llvm_i32_ty,    // idxen(imm)
+     llvm_i32_ty,    // glc(imm)
+     llvm_i32_ty,    // slc(imm)
+     llvm_i32_ty],   // tfe(imm)
+    []>;
+
 class AMDGPUBufferAtomic : Intrinsic <
   [llvm_i32_ty],
   [llvm_i32_ty,       // vdata(VGPR)
    llvm_v4i32_ty,     // rsrc(SGPR)
    llvm_i32_ty,       // vindex(VGPR)
    llvm_i32_ty,       // offset(SGPR/VGPR/imm)
    llvm_i1_ty],       // slc(imm)
   []>;

lib/Target/AMDGPU/AMDGPUISelLowering.h

enum NodeType : unsigned {
   INTERP_P2,
   PC_ADD_REL_OFFSET,
   KILL,
   DUMMY_CHAIN,
   FIRST_MEM_OPCODE_NUMBER = ISD::FIRST_TARGET_MEMORY_OPCODE,
   STORE_MSKOR,
   LOAD_CONSTANT,
   TBUFFER_STORE_FORMAT,
+  TBUFFER_LOAD_FORMAT,
   ATOMIC_CMP_SWAP,
   ATOMIC_INC,
   ATOMIC_DEC,
   BUFFER_LOAD,
   BUFFER_LOAD_FORMAT,
   LAST_AMDGPU_ISD_NUMBER
 };
 
 
 } // End namespace AMDGPUISD
 
 } // End namespace llvm
 
 #endif

lib/Target/AMDGPU/AMDGPUISelLowering.cpp

const char* AMDGPUTargetLowering::getTargetNodeName(unsigned Opcode) const {
   NODE_NAME_CASE(SENDMSG)
   NODE_NAME_CASE(SENDMSGHALT)
   NODE_NAME_CASE(INTERP_MOV)
   NODE_NAME_CASE(INTERP_P1)
   NODE_NAME_CASE(INTERP_P2)
   NODE_NAME_CASE(STORE_MSKOR)
   NODE_NAME_CASE(LOAD_CONSTANT)
   NODE_NAME_CASE(TBUFFER_STORE_FORMAT)
+  NODE_NAME_CASE(TBUFFER_LOAD_FORMAT)
   NODE_NAME_CASE(ATOMIC_CMP_SWAP)
   NODE_NAME_CASE(ATOMIC_INC)
   NODE_NAME_CASE(ATOMIC_DEC)
   NODE_NAME_CASE(BUFFER_LOAD)
   NODE_NAME_CASE(BUFFER_LOAD_FORMAT)
   case AMDGPUISD::LAST_AMDGPU_ISD_NUMBER: break;
   }
   return nullptr;

lib/Target/AMDGPU/BUFInstructions.td

class MTBUF_Pseudo <string opName, dag outs, dag ins,
   let MTBUF = 1;
   let Uses = [EXEC];
 
   let hasSideEffects = 0;
   let SchedRW = [WriteVMEM];
 }
 
 class MTBUF_Real <MTBUF_Pseudo ps> :
   InstSI <ps.OutOperandList, ps.InOperandList, ps.Mnemonic # ps.AsmOperands, []>,
   Enc64 {
 
   let isPseudo = 0;
   let isCodeGenOnly = 0;
 
   // copy relevant pseudo op flags
   let SubtargetPredicate = ps.SubtargetPredicate;
   let AsmMatchConverter  = ps.AsmMatchConverter;

 
 def BUFFER_WBINVL1 : MUBUF_Invalidate <"buffer_wbinvl1",
                                        int_amdgcn_buffer_wbinvl1>;
 
 //===----------------------------------------------------------------------===//
 // MTBUF Instructions
 //===----------------------------------------------------------------------===//
 
-//def TBUFFER_LOAD_FORMAT_X    : MTBUF_ <0, "tbuffer_load_format_x", []>;
-//def TBUFFER_LOAD_FORMAT_XY   : MTBUF_ <1, "tbuffer_load_format_xy", []>;
-//def TBUFFER_LOAD_FORMAT_XYZ  : MTBUF_ <2, "tbuffer_load_format_xyz", []>;
+def TBUFFER_LOAD_FORMAT_X     : MTBUF_Load_Pseudo  <"tbuffer_load_format_x", VGPR_32>;
+def TBUFFER_LOAD_FORMAT_XY    : MTBUF_Load_Pseudo  <"tbuffer_load_format_xy", VReg_64>;
+def TBUFFER_LOAD_FORMAT_XYZ   : MTBUF_Load_Pseudo  <"tbuffer_load_format_xyz", VReg_128>;
 def TBUFFER_LOAD_FORMAT_XYZW  : MTBUF_Load_Pseudo  <"tbuffer_load_format_xyzw", VReg_128>;
 def TBUFFER_STORE_FORMAT_X    : MTBUF_Store_Pseudo <"tbuffer_store_format_x", VGPR_32>;
 def TBUFFER_STORE_FORMAT_XY   : MTBUF_Store_Pseudo <"tbuffer_store_format_xy", VReg_64>;
 def TBUFFER_STORE_FORMAT_XYZ  : MTBUF_Store_Pseudo <"tbuffer_store_format_xyz", VReg_128>;
 def TBUFFER_STORE_FORMAT_XYZW : MTBUF_Store_Pseudo <"tbuffer_store_format_xyzw", VReg_128>;
 
 } // End let SubtargetPredicate = isGCN
 

 def : MUBUFScratchStorePat <BUFFER_STORE_DWORD_OFFEN, i32, store_private>;
 def : MUBUFScratchStorePat <BUFFER_STORE_DWORDX2_OFFEN, v2i32, store_private>;
 def : MUBUFScratchStorePat <BUFFER_STORE_DWORDX4_OFFEN, v4i32, store_private>;
 
 //===----------------------------------------------------------------------===//
 // MTBUF Patterns
 //===----------------------------------------------------------------------===//
 
+// TBUFFER_LOAD_FORMAT_*, addr64=0
+class MTBUF_LoadResource <ValueType vt, int num_channels, MTBUF_Pseudo opcode> : Pat<
+  (vt (SItbuffer_load v4i32:$rsrc, num_channels, i32:$vaddr,
+                      i32:$soffset, imm:$inst_offset, imm:$dfmt,
+                      imm:$nfmt, imm:$offen, imm:$idxen,
+                      imm:$glc, imm:$slc, imm:$tfe)),
+  (opcode
+    (as_i16imm $inst_offset), (as_i1imm $offen), (as_i1imm $idxen),
+    (as_i1imm $glc), 0, (as_i8imm $dfmt), (as_i8imm $nfmt), $vaddr, $rsrc,
+    (as_i1imm $slc), (as_i1imm $tfe), $soffset)
+>;
+
+def : MTBUF_LoadResource <i32, 1, TBUFFER_LOAD_FORMAT_X>;
+def : MTBUF_LoadResource <v2i32, 2, TBUFFER_LOAD_FORMAT_XY>;
+def : MTBUF_LoadResource <v4i32, 3, TBUFFER_LOAD_FORMAT_XYZ>;
+def : MTBUF_LoadResource <v4i32, 4, TBUFFER_LOAD_FORMAT_XYZW>;
+
 // TBUFFER_STORE_FORMAT_*, addr64=0
 class MTBUF_StoreResource <ValueType vt, int num_channels, MTBUF_Pseudo opcode> : Pat<
   (SItbuffer_store v4i32:$rsrc, vt:$vdata, num_channels, i32:$vaddr,
                    i32:$soffset, imm:$inst_offset, imm:$dfmt,
                    imm:$nfmt, imm:$offen, imm:$idxen,
                    imm:$glc, imm:$slc, imm:$tfe),
   (opcode
     $vdata, (as_i16imm $inst_offset), (as_i1imm $offen), (as_i1imm $idxen),
class MTBUF_Real_si <bits<3> op, MTBUF_Pseudo ps> :
   let AssemblerPredicate=isSICI;
   let DecoderNamespace="SICI";
 
   bits<1> addr64;
   let Inst{15}    = addr64;
   let Inst{18-16} = op;
 }
 
+def TBUFFER_LOAD_FORMAT_X_si     : MTBUF_Real_si <0, TBUFFER_LOAD_FORMAT_X>;
+def TBUFFER_LOAD_FORMAT_XY_si    : MTBUF_Real_si <1, TBUFFER_LOAD_FORMAT_XY>;
+def TBUFFER_LOAD_FORMAT_XYZ_si   : MTBUF_Real_si <2, TBUFFER_LOAD_FORMAT_XYZ>;
 def TBUFFER_LOAD_FORMAT_XYZW_si  : MTBUF_Real_si <3, TBUFFER_LOAD_FORMAT_XYZW>;
 def TBUFFER_STORE_FORMAT_X_si    : MTBUF_Real_si <4, TBUFFER_STORE_FORMAT_X>;
 def TBUFFER_STORE_FORMAT_XY_si   : MTBUF_Real_si <5, TBUFFER_STORE_FORMAT_XY>;
 def TBUFFER_STORE_FORMAT_XYZ_si  : MTBUF_Real_si <6, TBUFFER_STORE_FORMAT_XYZ>;
 def TBUFFER_STORE_FORMAT_XYZW_si : MTBUF_Real_si <7, TBUFFER_STORE_FORMAT_XYZW>;
 
 
 //===----------------------------------------------------------------------===//
 // CI
 //===----------------------------------------------------------------------===//
 
 class MUBUF_Real_ci <bits<7> op, MUBUF_Pseudo ps> :
   MUBUF_Real_si<op, ps> {
   let AssemblerPredicate=isCIOnly;
   let DecoderNamespace="CI";
class MTBUF_Real_vi <bits<4> op, MTBUF_Pseudo ps> :
   MTBUF_Real<ps>,
   SIMCInstr<ps.PseudoInstr, SIEncodingFamily.VI> {
   let AssemblerPredicate=isVI;
   let DecoderNamespace="VI";
 
   let Inst{18-15} = op;
 }
 
+def TBUFFER_LOAD_FORMAT_X_vi     : MTBUF_Real_vi <0, TBUFFER_LOAD_FORMAT_X>;
+def TBUFFER_LOAD_FORMAT_XY_vi    : MTBUF_Real_vi <1, TBUFFER_LOAD_FORMAT_XY>;
+def TBUFFER_LOAD_FORMAT_XYZ_vi   : MTBUF_Real_vi <2, TBUFFER_LOAD_FORMAT_XYZ>;
 def TBUFFER_LOAD_FORMAT_XYZW_vi  : MTBUF_Real_vi <3, TBUFFER_LOAD_FORMAT_XYZW>;
 def TBUFFER_STORE_FORMAT_X_vi    : MTBUF_Real_vi <4, TBUFFER_STORE_FORMAT_X>;
 def TBUFFER_STORE_FORMAT_XY_vi   : MTBUF_Real_vi <5, TBUFFER_STORE_FORMAT_XY>;
 def TBUFFER_STORE_FORMAT_XYZ_vi  : MTBUF_Real_vi <6, TBUFFER_STORE_FORMAT_XYZ>;
 def TBUFFER_STORE_FORMAT_XYZW_vi : MTBUF_Real_vi <7, TBUFFER_STORE_FORMAT_XYZW>;
 

lib/Target/AMDGPU/SIISelLowering.cpp

default:
     return AMDGPUTargetLowering::LowerOperation(Op, DAG);
   }
 }
 
 SDValue SITargetLowering::LowerINTRINSIC_W_CHAIN(SDValue Op,
                                                  SelectionDAG &DAG) const {
   unsigned IntrID = cast<ConstantSDNode>(Op.getOperand(1))->getZExtValue();
   SDLoc DL(Op);
+  MachineFunction &MF = DAG.getMachineFunction();
+
   switch (IntrID) {
   case Intrinsic::amdgcn_atomic_inc:
   case Intrinsic::amdgcn_atomic_dec: {
     MemSDNode *M = cast<MemSDNode>(Op);
     unsigned Opc = (IntrID == Intrinsic::amdgcn_atomic_inc) ?
       AMDGPUISD::ATOMIC_INC : AMDGPUISD::ATOMIC_DEC;
     SDValue Ops[] = {
       M->getOperand(0), // Chain
case Intrinsic::amdgcn_buffer_load_format: {
     SDValue Ops[] = {
       Op.getOperand(0), // Chain
       Op.getOperand(2), // rsrc
       Op.getOperand(3), // vindex
       Op.getOperand(4), // offset
       Op.getOperand(5), // glc
       Op.getOperand(6)  // slc
     };
-    MachineFunction &MF = DAG.getMachineFunction();
     SIMachineFunctionInfo *MFI = MF.getInfo<SIMachineFunctionInfo>();
 
     unsigned Opc = (IntrID == Intrinsic::amdgcn_buffer_load) ?
         AMDGPUISD::BUFFER_LOAD : AMDGPUISD::BUFFER_LOAD_FORMAT;
     EVT VT = Op.getValueType();
     EVT IntVT = VT.changeTypeToInteger();
 
     MachineMemOperand *MMO = MF.getMachineMemOperand(
       MachinePointerInfo(MFI->getBufferPSV()),
       MachineMemOperand::MOLoad,
       VT.getStoreSize(), VT.getStoreSize());
 
     return DAG.getMemIntrinsicNode(Opc, DL, Op->getVTList(), Ops, IntVT, MMO);
   }
+  case Intrinsic::amdgcn_tbuffer_load: {
+    SDValue Ops[] = {
+      Op.getOperand(0),  // Chain
+      Op.getOperand(2),  // rsrc
+      Op.getOperand(3),  // num channels
+      Op.getOperand(4),  // vaddr
+      Op.getOperand(5),  // soffset
+      Op.getOperand(6),  // instr offset
+      Op.getOperand(7),  // dfmt
+      Op.getOperand(8),  // nfmt
+      Op.getOperand(9),  // offen
+      Op.getOperand(10),  // idxen
+      Op.getOperand(11), // glc
+      Op.getOperand(12), // slc
+      Op.getOperand(13)  // tfe
+    };
+
+    EVT VT = Op.getOperand(2).getValueType();
+
+    MachineMemOperand *MMO = MF.getMachineMemOperand(
+      MachinePointerInfo(),
+      MachineMemOperand::MOLoad,
+      VT.getStoreSize(), 4);
+    return DAG.getMemIntrinsicNode(AMDGPUISD::TBUFFER_LOAD_FORMAT, DL,
+                                   Op->getVTList(), Ops, VT, MMO);    
+  }
   default:
     return SDValue();
   }
 }
 
 SDValue SITargetLowering::LowerINTRINSIC_VOID(SDValue Op,
                                               SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
SDValue SITargetLowering::LowerINTRINSIC_VOID(SDValue Op,
   case Intrinsic::amdgcn_s_sendmsghalt: {
     unsigned NodeOp = (IntrinsicID == Intrinsic::amdgcn_s_sendmsg) ?
       AMDGPUISD::SENDMSG : AMDGPUISD::SENDMSGHALT;
     Chain = copyToM0(DAG, Chain, DL, Op.getOperand(3));
     SDValue Glue = Chain.getValue(1);
     return DAG.getNode(NodeOp, DL, MVT::Other, Chain,
                        Op.getOperand(2), Glue);
   }
-  case AMDGPUIntrinsic::SI_tbuffer_store: {
+  case AMDGPUIntrinsic::SI_tbuffer_store: // Retained for backward compatibility
+  case Intrinsic::amdgcn_tbuffer_store: {
     SDValue Ops[] = {
       Chain,
       Op.getOperand(2),
       Op.getOperand(3),
       Op.getOperand(4),
       Op.getOperand(5),
       Op.getOperand(6),
       Op.getOperand(7),
       Op.getOperand(8),
case AMDGPUIntrinsic::SI_export: { // Legacy intrinsic.
     const ConstantSDNode *Tgt = cast<ConstantSDNode>(Op.getOperand(5));
     const ConstantSDNode *Compr = cast<ConstantSDNode>(Op.getOperand(6));
 
     const SDValue Ops[] = {
       Chain,
       DAG.getTargetConstant(Tgt->getZExtValue(), DL, MVT::i8),
       DAG.getTargetConstant(En->getZExtValue(), DL, MVT::i8),
       Op.getOperand(7),  // src0
       Op.getOperand(8),  // src1
       Op.getOperand(9),  // src2
       Op.getOperand(10), // src3
       DAG.getTargetConstant(Compr->getZExtValue(), DL, MVT::i1),
       DAG.getTargetConstant(VM->getZExtValue(), DL, MVT::i1)
     };
 
     unsigned Opc = Done->isNullValue() ?
       AMDGPUISD::EXPORT : AMDGPUISD::EXPORT_DONE;
     return DAG.getNode(Opc, DL, Op->getVTList(), Ops);
   }
   default:
     return SDValue();
   }
 }
 
 SDValue SITargetLowering::LowerLOAD(SDValue Op, SelectionDAG &DAG) const {
   SDLoc DL(Op);
   LoadSDNode *Load = cast<LoadSDNode>(Op);
   ISD::LoadExtType ExtType = Load->getExtensionType();

lib/Target/AMDGPU/SIInstrInfo.td

 def SIatomic_inc : SDNode<"AMDGPUISD::ATOMIC_INC", SDTAtomic2,
   [SDNPMayLoad, SDNPMayStore, SDNPMemOperand, SDNPHasChain]
 >;
 
 def SIatomic_dec : SDNode<"AMDGPUISD::ATOMIC_DEC", SDTAtomic2,
   [SDNPMayLoad, SDNPMayStore, SDNPMemOperand, SDNPHasChain]
 >;
 
+def SItbuffer_load : SDNode<"AMDGPUISD::TBUFFER_LOAD_FORMAT",
+  SDTypeProfile<1, 12,
+    [SDTCisVT<0, iAny>,   // vdata(VGPR)
+     SDTCisVT<1, v4i32>,  // rsrc(SGPR)
+     SDTCisVT<2, i32>,    // num_channels(imm)
+     SDTCisVT<3, i32>,    // vaddr(VGPR)
+     SDTCisVT<4, i32>,    // soffset(SGPR)
+     SDTCisVT<5, i32>,    // inst_offset(imm)
+     SDTCisVT<6, i32>,    // dfmt(imm)
+     SDTCisVT<7, i32>,    // nfmt(imm)
+     SDTCisVT<8, i32>,    // offen(imm)
+     SDTCisVT<9, i32>,    // idxen(imm)
+     SDTCisVT<10, i32>,   // glc(imm)
+     SDTCisVT<11, i32>,   // slc(imm)
+     SDTCisVT<12, i32>    // tfe(imm)
+    ]>,
+  [SDNPMayLoad, SDNPMemOperand, SDNPHasChain]
+>;
+
 def SItbuffer_store : SDNode<"AMDGPUISD::TBUFFER_STORE_FORMAT",
   SDTypeProfile<0, 13,
     [SDTCisVT<0, v4i32>,   // rsrc(SGPR)
      SDTCisVT<1, iAny>,   // vdata(VGPR)
      SDTCisVT<2, i32>,    // num_channels(imm)
      SDTCisVT<3, i32>,    // vaddr(VGPR)
      SDTCisVT<4, i32>,    // soffset(SGPR)
      SDTCisVT<5, i32>,    // inst_offset(imm)

lib/Target/AMDGPU/SIIntrinsics.td

def int_SI_export : Intrinsic <[],
     llvm_float_ty,  // src2
     llvm_float_ty], // src3
     []
   >;
 
   def int_SI_load_const : Intrinsic <[llvm_float_ty], [llvm_anyint_ty, llvm_i32_ty], [IntrNoMem]>;
   def int_SI_vs_load_input : Intrinsic <[llvm_v4f32_ty], [llvm_anyint_ty, llvm_i16_ty, llvm_i32_ty], [IntrNoMem]> ;
 
+  // Fully-flexible TBUFFER_LOAD_FORMAT_* except for the ADDR64 bit, which is not exposed
+  def int_SI_tbuffer_load : Intrinsic <
+    [llvm_anyint_ty], // vdata(VGPR), overloaded for types i32, v2i32, v4i32
+    [llvm_anyint_ty,  // rsrc(SGPR)
+     llvm_i32_ty,     // num_channels(imm), selects opcode suffix: 1=X, 2=XY, 3=XYZ, 4=XYZW
+     llvm_i32_ty,     // vaddr(VGPR)
+     llvm_i32_ty,     // soffset(SGPR)
+     llvm_i32_ty,     // inst_offset(imm)
+     llvm_i32_ty,     // dfmt(imm)
+     llvm_i32_ty,     // nfmt(imm)
+     llvm_i32_ty,     // offen(imm)
+     llvm_i32_ty,     // idxen(imm)
+     llvm_i32_ty,     // glc(imm)
+     llvm_i32_ty,     // slc(imm)
+     llvm_i32_ty],    // tfe(imm)
+    []>;
+
   // Fully-flexible TBUFFER_STORE_FORMAT_* except for the ADDR64 bit, which is not exposed
   def int_SI_tbuffer_store : Intrinsic <
     [],
     [llvm_anyint_ty, // rsrc(SGPR)
      llvm_anyint_ty, // vdata(VGPR), overloaded for types i32, v2i32, v4i32
      llvm_i32_ty,    // num_channels(imm), selects opcode suffix: 1=X, 2=XY, 3=XYZ, 4=XYZW
      llvm_i32_ty,    // vaddr(VGPR)
      llvm_i32_ty,    // soffset(SGPR)

test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.load.ll

+;RUN: llc < %s -march=amdgcn -mcpu=verde -verify-machineinstrs | FileCheck %s
+;RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s
+
+;CHECK-LABEL: {{^}}test1:
+;CHECK: tbuffer_load_format_xyzw {{v\[[0-9]+:[0-9]+\]}}, 0x20, -1, 0, -1, 0, 14, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs <4 x i32> @test1(i32 %vaddr) {
+    %vdata = call <4 x i32> @llvm.amdgcn.tbuffer.load.v4i32(<4 x i32> undef,
+        i32 4, i32 %vaddr, i32 0, i32 32, i32 14, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret <4 x i32> %vdata
+}
+
+;CHECK-LABEL: {{^}}test2:
+;CHECK: tbuffer_load_format_xyz {{v\[[0-9]+:[0-9]+\]}}, 0x18, -1, 0, -1, 0, 13, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs <4 x i32> @test2(i32 %vaddr) {
+    %vdata = call <4 x i32> @llvm.amdgcn.tbuffer.load.v4i32(<4 x i32> undef,
+        i32 3, i32 %vaddr, i32 0, i32 24, i32 13, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret <4 x i32> %vdata
+}
+
+;CHECK-LABEL: {{^}}test3:
+;CHECK: tbuffer_load_format_xy {{v\[[0-9]+:[0-9]+\]}}, 0x10, -1, 0, -1, 0, 11, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs <2 x i32> @test3(i32 %vaddr) {
+    %vdata = call <2 x i32> @llvm.amdgcn.tbuffer.load.v2i32(<4 x i32> undef,
+        i32 2, i32 %vaddr, i32 0, i32 16, i32 11, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret <2 x i32> %vdata
+}
+
+;CHECK-LABEL: {{^}}test4:
+;CHECK: tbuffer_load_format_x {{v[0-9]+}}, 0x8, -1, 0, -1, 0, 4, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs float @test4(i32 %vaddr) {
+    %vdata = call i32 @llvm.amdgcn.tbuffer.load.i32(<4 x i32> undef,
+        i32 1, i32 %vaddr, i32 0, i32 8, i32 4, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    %vdata.f = bitcast i32 %vdata to float
+    ret float %vdata.f
+}
+
+declare i32 @llvm.amdgcn.tbuffer.load.i32(<4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)
+declare <2 x i32> @llvm.amdgcn.tbuffer.load.v2i32(<4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)
+declare <4 x i32> @llvm.amdgcn.tbuffer.load.v4i32(<4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)

test/CodeGen/AMDGPU/llvm.amdgcn.tbuffer.store.ll

+;RUN: llc < %s -march=amdgcn -mcpu=verde -verify-machineinstrs | FileCheck %s
+;RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck %s
+
+;CHECK-LABEL: {{^}}test1:
+;CHECK: tbuffer_store_format_xyzw {{v\[[0-9]+:[0-9]+\]}}, 0x20, -1, 0, -1, 0, 14, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs void @test1(i32 %a1, i32 %vaddr) {
+    %vdata = insertelement <4 x i32> undef, i32 %a1, i32 0
+    call void @llvm.amdgcn.tbuffer.store.v4i32(<4 x i32> undef, <4 x i32> %vdata,
+        i32 4, i32 %vaddr, i32 0, i32 32, i32 14, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret void
+}
+
+;CHECK-LABEL: {{^}}test2:
+;CHECK: tbuffer_store_format_xyz {{v\[[0-9]+:[0-9]+\]}}, 0x18, -1, 0, -1, 0, 13, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs void @test2(i32 %a1, i32 %vaddr) {
+    %vdata = insertelement <4 x i32> undef, i32 %a1, i32 0
+    call void @llvm.amdgcn.tbuffer.store.v4i32(<4 x i32> undef, <4 x i32> %vdata,
+        i32 3, i32 %vaddr, i32 0, i32 24, i32 13, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret void
+}
+
+;CHECK-LABEL: {{^}}test3:
+;CHECK: tbuffer_store_format_xy {{v\[[0-9]+:[0-9]+\]}}, 0x10, -1, 0, -1, 0, 11, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs void @test3(i32 %a1, i32 %vaddr) {
+    %vdata = insertelement <2 x i32> undef, i32 %a1, i32 0
+    call void @llvm.amdgcn.tbuffer.store.v2i32(<4 x i32> undef, <2 x i32> %vdata,
+        i32 2, i32 %vaddr, i32 0, i32 16, i32 11, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret void
+}
+
+;CHECK-LABEL: {{^}}test4:
+;CHECK: tbuffer_store_format_x {{v[0-9]+}}, 0x8, -1, 0, -1, 0, 4, 4, {{v[0-9]+}}, {{s\[[0-9]+:[0-9]+\]}}, -1, 0, 0
+define amdgpu_vs void @test4(i32 %vdata, i32 %vaddr) {
+    call void @llvm.amdgcn.tbuffer.store.i32(<4 x i32> undef, i32 %vdata,
+        i32 1, i32 %vaddr, i32 0, i32 8, i32 4, i32 4, i32 1, i32 0, i32 1,
+        i32 1, i32 0)
+    ret void
+}
+
+declare void @llvm.amdgcn.tbuffer.store.i32(<4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)
+declare void @llvm.amdgcn.tbuffer.store.v2i32(<4 x i32>, <2 x i32>, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)
+declare void @llvm.amdgcn.tbuffer.store.v4i32(<4 x i32>, <4 x i32>, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32)