[AMDGPU] Add amdgcn_s_buffer_load_imm intrinsic

[piotr]

Summary

Add int_amdgcn_s_buffer_load_imm instinsic for gfx9+, similar
to the existing int_amdgcn_s_buffer_load, but with immediate
instruction offset.

This exposes an immediate field of the instruction to the front-ends,
and can potentially help generate better code, especially in cases
of complex address expressions where the offset is located in
a different block than the load.

Basic handling also added in the global-isel path with the fall-back
to the old intrinsic. It is not clear at this point, whether
the new intrinsic will help global-isel.

[piotr]

Updated instcombine code as well.

[arsenm]

What's the problem with using a constant offset with the existing intrinsic?

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

4915

Extra newline

[piotr]

Removed extra newline.

[piotr]

In D137066#3898225, @arsenm wrote:

What's the problem with using a constant offset with the existing intrinsic?

That would work fine, but the problem is that the existing intrinsic has just one offset field and relies on the isel to extract the const part to utilize the instruction offset immediate field. The new intrinsic in addition to the existing scalar offset exposes the constant part that lets us use the immediate field of the instruction directly. So the new intrinsic would let us include "1024" as an immediate in the example below:

%load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 1024, i32 0)

(I know there are new tests with a constant for soffset and 0 for the immediate offset field, but these are to ensure the generated code is the same as with the old intrinsic. The use case with two constant offsets is really not a real-world case - the main use case is a scalar offset with separate constant offset).

[nhaehnle]

I think the question is really: what IR examples are there that could use scalar loads with immediate offsets but don't because instruction selection fails to extract the constant; and why does extracting the constant fail?

[piotr]

In D137066#3902068, @nhaehnle wrote:

I think the question is really: what IR examples are there that could use scalar loads with immediate offsets but don't because instruction selection fails to extract the constant; and why does extracting the constant fail?

All cases of isel not being able to extract the constant I looked at were due to the nodes being scattered over different basic blocks.

[nhaehnle]

Ah, I see. Maybe add a comment in IntrinsicsAMDGPU.td that we should aim to get rid of the intrinsic again after we've transitioned to GlobalISel?

[nhaehnle]

llvm/include/llvm/IR/IntrinsicsAMDGPU.td

1017–1019

Document whether there is an upper bound to the offset argument, or whether codegen will handle it.

[piotr]

Extended comments.

[arsenm]

In D137066#3902128, @piotr wrote:

In D137066#3902068, @nhaehnle wrote:

I think the question is really: what IR examples are there that could use scalar loads with immediate offsets but don't because instruction selection fails to extract the constant; and why does extracting the constant fail?

All cases of isel not being able to extract the constant I looked at were due to the nodes being scattered over different basic blocks.

This is the kind of case that CodeGenPrepare works around for addressing mode matching. I'd rather add that sort of optimization rather than changing the IR by adding new intrinsics to workaround this

[piotr]

In D137066#3903754, @arsenm wrote:

In D137066#3902128, @piotr wrote:

In D137066#3902068, @nhaehnle wrote:

I think the question is really: what IR examples are there that could use scalar loads with immediate offsets but don't because instruction selection fails to extract the constant; and why does extracting the constant fail?

All cases of isel not being able to extract the constant I looked at were due to the nodes being scattered over different basic blocks.

This is the kind of case that CodeGenPrepare works around for addressing mode matching. I'd rather add that sort of optimization rather than changing the IR by adding new intrinsics to workaround this

I did consider extending CodeGenPrepare instead. The advantage of using the intrinsic is that we can run the code through the optimizer (CodeGenPrepare is run very late), which yields even better code.

While the GlobalISel may not need the new intrinsic, one could argue that adding the intrinsic is not a workaround and it should have been implemented for gfx9 already alongside other changes. The new SMEM format features (including the separate immediate offset field) were originally omitted - see https://github.com/llvm/llvm-project/issues/38652. It was only about a couple of months ago, when @kosarev fixed this by adding support for the new format in a series of commits.

[piotr]

Another problem with codegenprepare is that it does not understand our intrinsic, so it does not treat the i32 offset field as a part of address expression, or in any special way.

We do operate on loads with fat pointers in the (part of) front-end, but we convert them to intrinsics before entering codegen. The aspiration is to do the conversion later (or skip the intrinsics altogether) effectively adding support for fat pointers in the backend. But we are not there yet, and that would surely have some other issues, which we are not aware of yet.

[piotr]

Ping.

[tsymalla]

What is the current state of this?

[piotr]

In D137066#4198531, @tsymalla wrote:

What is the current state of this?

I am now convinced that this is not the right fix.

There is no easy solution available - need to experiment more. Matt suggested taking advantage of addressing mode matching in CodeGenPrepare, but this pass doesn't work for our intrinsics.

llvm/include/llvm/IR/IntrinsicsAMDGPU.td

 def int_amdgcn_s_buffer_load : Intrinsic <
   [llvm_any_ty],
   [llvm_v4i32_ty,     // rsrc(SGPR)
    llvm_i32_ty,       // byte offset
    llvm_i32_ty],      // cachepolicy(imm; bit 0 = glc, bit 2 = dlc)
   [IntrNoMem, IntrWillReturn, ImmArg<ArgIndex<2>>]>,
   AMDGPURsrcIntrinsic<0>;
 
+// Generate intrinsic similar to int_amdgcn_s_buffer_load, but with immediate
+// instruction offset.
+// Codegen will generate an extra s_mov instruction for any overflow of 20 bit.
+// The intrinsic helps generating better code for offset expressions spanning
+// separate basic blocks, it may be not needed for GlobalISel.
+def int_amdgcn_s_buffer_load_imm : Intrinsic <
+  [llvm_any_ty],
+  [llvm_v4i32_ty,     // rsrc(SGPR)
+   llvm_i32_ty,       // byte offset
+   llvm_i32_ty,       // unsigned imm offset (imm), 20-bit
+   llvm_i32_ty],      // cachepolicy(imm; bit 0 = glc, bit 2 = dlc)
+  [IntrNoMem, IntrWillReturn, ImmArg<ArgIndex<2>>, ImmArg<ArgIndex<3>>]>,
+  AMDGPURsrcIntrinsic<0>;
+
 class AMDGPUBufferStore<LLVMType data_ty = llvm_any_ty> : Intrinsic <
   [],
   [data_ty,          // vdata(VGPR)
    llvm_v4i32_ty,     // rsrc(SGPR)
    llvm_i32_ty,       // vindex(VGPR)
    llvm_i32_ty,       // offset(SGPR/VGPR/imm)
    llvm_i1_ty,        // glc(imm)
    llvm_i1_ty],       // slc(imm)

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

if (UnusedComponentsAtFront > 0) {
       static const unsigned InvalidOffsetIdx = 0xf;
 
       unsigned OffsetIdx;
       switch (II.getIntrinsicID()) {
       case Intrinsic::amdgcn_raw_buffer_load:
         OffsetIdx = 1;
         break;
       case Intrinsic::amdgcn_s_buffer_load:
+      case Intrinsic::amdgcn_s_buffer_load_imm:
         // If resulting type is vec3, there is no point in trimming the
         // load with updated offset, as the vec3 would most likely be widened to
         // vec4 anyway during lowering.
         if (ActiveBits == 4 && UnusedComponentsAtFront == 1)
           OffsetIdx = InvalidOffsetIdx;
         else
           OffsetIdx = 1;
         break;
std::function<void(Instruction *, unsigned, APInt, APInt &)>
         SimplifyAndSetOp) const {
   switch (II.getIntrinsicID()) {
   case Intrinsic::amdgcn_buffer_load:
   case Intrinsic::amdgcn_buffer_load_format:
   case Intrinsic::amdgcn_raw_buffer_load:
   case Intrinsic::amdgcn_raw_buffer_load_format:
   case Intrinsic::amdgcn_raw_tbuffer_load:
   case Intrinsic::amdgcn_s_buffer_load:
+  case Intrinsic::amdgcn_s_buffer_load_imm:
   case Intrinsic::amdgcn_struct_buffer_load:
   case Intrinsic::amdgcn_struct_buffer_load_format:
   case Intrinsic::amdgcn_struct_tbuffer_load:
   case Intrinsic::amdgcn_tbuffer_load:
     return simplifyAMDGCNMemoryIntrinsicDemanded(IC, II, DemandedElts);
   default: {
     if (getAMDGPUImageDMaskIntrinsic(II.getIntrinsicID())) {
       return simplifyAMDGCNMemoryIntrinsicDemanded(IC, II, DemandedElts, 0);
     }
     break;
   }
   }
   return None;
 }

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

if (BaseOpcode->Atomic) {
     Register VData0 = MI.getOperand(2).getReg();
     LLT Ty = MRI->getType(VData0);
 
     // TODO: Allow atomic swap and bit ops for v2s16/v4s16
     if (Ty.isVector())
       return false;
 
     if (BaseOpcode->AtomicX2) {
       Register VData1 = MI.getOperand(3).getReg();
       // The two values are packed in one register.
       LLT PackedTy = LLT::fixed_vector(2, Ty);
       auto Concat = B.buildBuildVector(PackedTy, {VData0, VData1});
       MI.getOperand(2).setReg(Concat.getReg(0));
       MI.getOperand(3).setReg(AMDGPU::NoRegister);
     }
   }
 
bool AMDGPULegalizerInfo::legalizeSBufferLoad(
 
   if (shouldBitcastLoadStoreType(ST, Ty, LLT::scalar(Size))) {
     Ty = getBitcastRegisterType(Ty);
     Helper.bitcastDst(MI, Ty, 0);
     Dst = MI.getOperand(0).getReg();
     B.setInsertPt(B.getMBB(), MI);
   }
 
+  auto Intr = MI.getIntrinsicID();
+
   // FIXME: We don't really need this intermediate instruction. The intrinsic
   // should be fixed to have a memory operand. Since it's readnone, we're not
   // allowed to add one.
   MI.setDesc(B.getTII().get(AMDGPU::G_AMDGPU_S_BUFFER_LOAD));
   MI.removeOperand(1); // Remove intrinsic ID
 
+  if (Intr == Intrinsic::amdgcn_s_buffer_load) {
+    // Add instr offset 0 before cachepolicy.
+    unsigned CachePolicy = MI.getOperand(3).getImm();
+    MI.addOperand(MachineOperand::CreateImm(CachePolicy));
+    MI.getOperand(3).setImm(0);
+  } else {
+    // Fall-back to old scheme with instr offset 0.
+    // TODO: Proper support for instr offset.
+    assert(Intr == Intrinsic::amdgcn_s_buffer_load_imm);
+
+    const LLT S32 = LLT::scalar(32);
+    unsigned InstrOffsetUint = MI.getOperand(3).getImm();
+    if (InstrOffsetUint > 0) {
+      auto InstrOffset = B.buildConstant(S32, InstrOffsetUint).getReg(0);
+      auto NewSOffset = B.buildAdd(S32, MI.getOperand(2), InstrOffset);
+      MI.getOperand(2).setReg(NewSOffset.getReg(0));
+      MI.getOperand(3).setImm(0);
+    }
+  }
+
   // FIXME: When intrinsic definition is fixed, this should have an MMO already.
   // TODO: Should this use datalayout alignment?
   const unsigned MemSize = (Size + 7) / 8;
   const Align MemAlign(4);
   MachineMemOperand *MMO = MF.getMachineMemOperand(
       MachinePointerInfo(),
       MachineMemOperand::MOLoad | MachineMemOperand::MODereferenceable |
           MachineMemOperand::MOInvariant,
bool AMDGPULegalizerInfo::legalizeIntrinsic(LegalizerHelper &Helper,
   case Intrinsic::amdgcn_is_private:
     return legalizeIsAddrSpace(MI, MRI, B, AMDGPUAS::PRIVATE_ADDRESS);
   case Intrinsic::amdgcn_wavefrontsize: {
     B.buildConstant(MI.getOperand(0), ST.getWavefrontSize());
     MI.eraseFromParent();
     return true;
   }
   case Intrinsic::amdgcn_s_buffer_load:
+  case Intrinsic::amdgcn_s_buffer_load_imm:
     return legalizeSBufferLoad(Helper, MI);
   case Intrinsic::amdgcn_raw_buffer_store:
   case Intrinsic::amdgcn_struct_buffer_store:
     return legalizeBufferStore(MI, MRI, B, false, false);
   case Intrinsic::amdgcn_raw_buffer_store_format:
   case Intrinsic::amdgcn_struct_buffer_store_format:
     return legalizeBufferStore(MI, MRI, B, false, true);
   case Intrinsic::amdgcn_raw_tbuffer_store:

llvm/lib/Target/AMDGPU/AMDGPURegisterBankInfo.cpp

AMDGPURegisterBankInfo::getInstrAlternativeMappingsIntrinsic(
   }
 }
 
 RegisterBankInfo::InstructionMappings
 AMDGPURegisterBankInfo::getInstrAlternativeMappingsIntrinsicWSideEffects(
     const MachineInstr &MI, const MachineRegisterInfo &MRI) const {
 
   switch (MI.getIntrinsicID()) {
-  case Intrinsic::amdgcn_s_buffer_load: {
+  case Intrinsic::amdgcn_s_buffer_load:
+  case Intrinsic::amdgcn_s_buffer_load_imm: {
     static const OpRegBankEntry<2> Table[4] = {
       // Perfectly legal.
       { { AMDGPU::SGPRRegBankID, AMDGPU::SGPRRegBankID }, 1 },
 
       // Only need 1 register in loop
       { { AMDGPU::SGPRRegBankID, AMDGPU::VGPRRegBankID }, 300 },
 
       // Have to waterfall the resource.

llvm/lib/Target/AMDGPU/SIISelLowering.h

private:
 
   SDValue LowerGlobalAddress(AMDGPUMachineFunction *MFI, SDValue Op,
                              SelectionDAG &DAG) const override;
   SDValue lowerImplicitZextParam(SelectionDAG &DAG, SDValue Op,
                                  MVT VT, unsigned Offset) const;
   SDValue lowerImage(SDValue Op, const AMDGPU::ImageDimIntrinsicInfo *Intr,
                      SelectionDAG &DAG, bool WithChain) const;
   SDValue lowerSBuffer(EVT VT, SDLoc DL, SDValue Rsrc, SDValue Offset,
-                       SDValue CachePolicy, SelectionDAG &DAG) const;
+                       SDValue ImmOffset, SDValue CachePolicy,
+                       SelectionDAG &DAG) const;
 
   SDValue lowerRawBufferAtomicIntrin(SDValue Op, SelectionDAG &DAG,
                                      unsigned NewOpcode) const;
   SDValue lowerStructBufferAtomicIntrin(SDValue Op, SelectionDAG &DAG,
                                         unsigned NewOpcode) const;
 
   SDValue lowerWorkitemID(SelectionDAG &DAG, SDValue Op, unsigned Dim,
                           const ArgDescriptor &ArgDesc) const;
public:
   /// expanded into a set of cmp/select instructions.
   static bool shouldExpandVectorDynExt(unsigned EltSize, unsigned NumElem,
                                        bool IsDivergentIdx,
                                        const GCNSubtarget *Subtarget);
 
   bool shouldExpandVectorDynExt(SDNode *N) const;
 
 private:
-  // Analyze a combined offset from an amdgcn_buffer_ intrinsic and store the
-  // three offsets (voffset, soffset and instoffset) into the SDValue[3] array
-  // pointed to by Offsets.
-  void setBufferOffsets(SDValue CombinedOffset, SelectionDAG &DAG,
-                        SDValue *Offsets, Align Alignment = Align(4)) const;
+  // Analyze a combined offset from an amdgcn_buffer_ intrinsic, together with
+  // immediate instruction offset, and store the three offsets (voffset, soffset
+  // and instoffset) into the SDValue[3] array pointed to by Offsets.
+  void setBufferOffsets(SDValue CombinedOffset, unsigned InstImmOffset,
+                        SelectionDAG &DAG, SDValue *Offsets,
+                        Align Alignment = Align(4)) const;
 
   // Handle 8 bit and 16 bit buffer loads
   SDValue handleByteShortBufferLoads(SelectionDAG &DAG, EVT LoadVT, SDLoc DL,
                                      ArrayRef<SDValue> Ops, MemSDNode *M) const;
 
   // Handle 8 bit and 16 bit buffer stores
   SDValue handleByteShortBufferStores(SelectionDAG &DAG, EVT VDataType,
                                       SDLoc DL, SDValue Ops[],

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

if (BaseOpcode->Store)
     return SDValue(NewNode, 0);
   return constructRetValue(DAG, NewNode,
                            OrigResultTypes, IsTexFail,
                            Subtarget->hasUnpackedD16VMem(), IsD16,
                            DMaskLanes, NumVDataDwords, DL);
 }
 
 SDValue SITargetLowering::lowerSBuffer(EVT VT, SDLoc DL, SDValue Rsrc,
-                                       SDValue Offset, SDValue CachePolicy,
+                                       SDValue Offset, SDValue ImmOffset,
+                                       SDValue CachePolicy,
                                        SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
 
   const DataLayout &DataLayout = DAG.getDataLayout();
   Align Alignment =
       DataLayout.getABITypeAlign(VT.getTypeForEVT(*DAG.getContext()));
 
   MachineMemOperand *MMO = MF.getMachineMemOperand(
       MachinePointerInfo(),
       MachineMemOperand::MOLoad | MachineMemOperand::MODereferenceable |
           MachineMemOperand::MOInvariant,
       VT.getStoreSize(), Alignment);
 
   if (!Offset->isDivergent()) {
-    SDValue Ops[] = {
-        Rsrc,
-        Offset, // Offset
-        CachePolicy
-    };
+
+    SDValue Ops[4];
+    Ops[0] = Rsrc;
+
+    // In the unlikely event the immediate offset overflows 20 bit we add the
+    // overflow to soffset.
+    // The encoding has 21 bits, but using a negative offset is unsafe, so
+    // we only make use of 20 bits and treat the offset as unsigned.
+    unsigned ImmOffsetUint = cast<ConstantSDNode>(ImmOffset)->getZExtValue();
+    unsigned MaxOffset = (1 << 20) - 1;
+    if (ImmOffsetUint > MaxOffset) {
+      unsigned High = ImmOffsetUint & ~MaxOffset;
+      unsigned Low = ImmOffsetUint & MaxOffset;
+
+      SDValue Overflow = DAG.getConstant(High, DL, MVT::i32);
+      SDValue CombinedOffset =
+          DAG.getNode(ISD::ADD, DL, MVT::i32, Offset, Overflow);
+      SDValue ValidImmOffset = DAG.getTargetConstant(Low, DL, MVT::i32);
+
+      Ops[1] = CombinedOffset;
+      Ops[2] = ValidImmOffset;
+    } else {
+      Ops[1] = Offset;
+      Ops[2] = ImmOffset;
+    }
+    Ops[3] = CachePolicy;
 
     // Widen vec3 load to vec4.
     if (VT.isVector() && VT.getVectorNumElements() == 3) {
       EVT WidenedVT =
           EVT::getVectorVT(*DAG.getContext(), VT.getVectorElementType(), 4);
       auto WidenedOp = DAG.getMemIntrinsicNode(
           AMDGPUISD::SBUFFER_LOAD, DL, DAG.getVTList(WidenedVT), Ops, WidenedVT,
           MF.getMachineMemOperand(MMO, 0, WidenedVT.getStoreSize()));
SDValue Ops[] = {
       {},                                               // soffset
       {},                                               // offset
       CachePolicy,                                      // cachepolicy
       DAG.getTargetConstant(0, DL, MVT::i1),            // idxen
   };
 
   // Use the alignment to ensure that the required offsets will fit into the
   // immediate offsets.
-  setBufferOffsets(Offset, DAG, &Ops[3],
+  setBufferOffsets(Offset, (cast<ConstantSDNode>(ImmOffset))->getZExtValue(),
+                   DAG, &Ops[3],
                    NumLoads > 1 ? Align(16 * NumLoads) : Align(4));
 
   uint64_t InstOffset = cast<ConstantSDNode>(Ops[5])->getZExtValue();
   for (unsigned i = 0; i < NumLoads; ++i) {
     Ops[5] = DAG.getTargetConstant(InstOffset + 16 * i, DL, MVT::i32);
     Loads.push_back(getMemIntrinsicNode(AMDGPUISD::BUFFER_LOAD, DL, VTList, Ops,
                                         LoadVT, MMO, DAG));
   }
case Intrinsic::amdgcn_workitem_id_x:
     return lowerWorkitemID(DAG, Op, 0, MFI->getArgInfo().WorkItemIDX);
   case Intrinsic::amdgcn_workitem_id_y:
     return lowerWorkitemID(DAG, Op, 1, MFI->getArgInfo().WorkItemIDY);
   case Intrinsic::amdgcn_workitem_id_z:
     return lowerWorkitemID(DAG, Op, 2, MFI->getArgInfo().WorkItemIDZ);
   case Intrinsic::amdgcn_wavefrontsize:
     return DAG.getConstant(MF.getSubtarget<GCNSubtarget>().getWavefrontSize(),
                            SDLoc(Op), MVT::i32);
-  case Intrinsic::amdgcn_s_buffer_load: {
-    unsigned CPol = cast<ConstantSDNode>(Op.getOperand(3))->getZExtValue();
-    if (CPol & ~AMDGPU::CPol::ALL)
+  case Intrinsic::amdgcn_s_buffer_load:
+  case Intrinsic::amdgcn_s_buffer_load_imm: {
+    SDValue ImmOffset, CPol;
+    if (IntrinsicID == Intrinsic::amdgcn_s_buffer_load_imm) {
+      ImmOffset = Op.getOperand(3);
+      CPol = Op.getOperand(4);
+    } else {
+      ImmOffset = DAG.getTargetConstant(0, DL, MVT::i32);
+      CPol = Op.getOperand(3);
+    }
+
+    if (cast<ConstantSDNode>(CPol)->getZExtValue() & ~AMDGPU::CPol::ALL)
       return Op;
-    return lowerSBuffer(VT, DL, Op.getOperand(1), Op.getOperand(2), Op.getOperand(3),
-                        DAG);
+
+    return lowerSBuffer(VT, DL, Op.getOperand(1), Op.getOperand(2), ImmOffset,
+                        CPol, DAG);
   }
   case Intrinsic::amdgcn_fdiv_fast:
     return lowerFDIV_FAST(Op, DAG);
   case Intrinsic::amdgcn_sin:
     return DAG.getNode(AMDGPUISD::SIN_HW, DL, VT, Op.getOperand(1));
 
   case Intrinsic::amdgcn_cos:
     return DAG.getNode(AMDGPUISD::COS_HW, DL, VT, Op.getOperand(1));
SDValue Ops[] = {
       Op.getOperand(2), // rsrc
       Op.getOperand(3), // vindex
       SDValue(),        // voffset -- will be set by setBufferOffsets
       SDValue(),        // soffset -- will be set by setBufferOffsets
       SDValue(),        // offset -- will be set by setBufferOffsets
       DAG.getTargetConstant(Glc | (Slc << 1), DL, MVT::i32), // cachepolicy
       DAG.getTargetConstant(IdxEn, DL, MVT::i1), // idxen
     };
-    setBufferOffsets(Op.getOperand(4), DAG, &Ops[3]);
+    setBufferOffsets(Op.getOperand(4), 0, DAG, &Ops[3]);
 
     unsigned Opc = (IntrID == Intrinsic::amdgcn_buffer_load) ?
         AMDGPUISD::BUFFER_LOAD : AMDGPUISD::BUFFER_LOAD_FORMAT;
 
     EVT VT = Op.getValueType();
     EVT IntVT = VT.changeTypeToInteger();
     auto *M = cast<MemSDNode>(Op);
     updateBufferMMO(M->getMemOperand(), Ops[3], Ops[4], Ops[5], Ops[2]);
SDValue Ops[] = {
       Op.getOperand(3), // rsrc
       Op.getOperand(4), // vindex
       SDValue(),        // voffset -- will be set by setBufferOffsets
       SDValue(),        // soffset -- will be set by setBufferOffsets
       SDValue(),        // offset -- will be set by setBufferOffsets
       DAG.getTargetConstant(Slc << 1, DL, MVT::i32), // cachepolicy
       DAG.getTargetConstant(IdxEn, DL, MVT::i1), // idxen
     };
-    setBufferOffsets(Op.getOperand(5), DAG, &Ops[4]);
+    setBufferOffsets(Op.getOperand(5), 0, DAG, &Ops[4]);
 
     EVT VT = Op.getValueType();
 
     auto *M = cast<MemSDNode>(Op);
     updateBufferMMO(M->getMemOperand(), Ops[4], Ops[5], Ops[6], Ops[3]);
     unsigned Opcode = 0;
 
     switch (IntrID) {
SDValue Ops[] = {
       Op.getOperand(4), // rsrc
       Op.getOperand(5), // vindex
       SDValue(),        // voffset -- will be set by setBufferOffsets
       SDValue(),        // soffset -- will be set by setBufferOffsets
       SDValue(),        // offset -- will be set by setBufferOffsets
       DAG.getTargetConstant(Slc << 1, DL, MVT::i32), // cachepolicy
       DAG.getTargetConstant(IdxEn, DL, MVT::i1), // idxen
     };
-    setBufferOffsets(Op.getOperand(6), DAG, &Ops[5]);
+    setBufferOffsets(Op.getOperand(6), 0, DAG, &Ops[5]);
 
     EVT VT = Op.getValueType();
     auto *M = cast<MemSDNode>(Op);
     updateBufferMMO(M->getMemOperand(), Ops[5], Ops[6], Ops[7], Ops[4]);
 
     return DAG.getMemIntrinsicNode(AMDGPUISD::BUFFER_ATOMIC_CMPSWAP, DL,
                                    Op->getVTList(), Ops, VT, M->getMemOperand());
   }
SDValue Ops[] = {
       Op.getOperand(3), // rsrc
       Op.getOperand(4), // vindex
       SDValue(), // voffset -- will be set by setBufferOffsets
       SDValue(), // soffset -- will be set by setBufferOffsets
       SDValue(), // offset -- will be set by setBufferOffsets
       DAG.getTargetConstant(Glc | (Slc << 1), DL, MVT::i32), // cachepolicy
       DAG.getTargetConstant(IdxEn, DL, MVT::i1), // idxen
     };
-    setBufferOffsets(Op.getOperand(5), DAG, &Ops[4]);
+    setBufferOffsets(Op.getOperand(5), 0, DAG, &Ops[4]);
 
     unsigned Opc = IntrinsicID == Intrinsic::amdgcn_buffer_store ?
                    AMDGPUISD::BUFFER_STORE : AMDGPUISD::BUFFER_STORE_FORMAT;
     Opc = IsD16 ? AMDGPUISD::BUFFER_STORE_FORMAT_D16 : Opc;
     MemSDNode *M = cast<MemSDNode>(Op);
     updateBufferMMO(M->getMemOperand(), Ops[4], Ops[5], Ops[6], Ops[3]);
 
     // Handle BUFFER_STORE_BYTE/SHORT overloaded intrinsics
if (!C1)
     C1 = cast<ConstantSDNode>(DAG.getTargetConstant(0, DL, MVT::i32));
   return {N0, SDValue(C1, 0)};
 }
 
 // Analyze a combined offset from an amdgcn_buffer_ intrinsic and store the
 // three offsets (voffset, soffset and instoffset) into the SDValue[3] array
 // pointed to by Offsets.
 void SITargetLowering::setBufferOffsets(SDValue CombinedOffset,
+                                        uint32_t InstImmOffset,
                                         SelectionDAG &DAG, SDValue *Offsets,
                                         Align Alignment) const {
   SDLoc DL(CombinedOffset);
   if (auto C = dyn_cast<ConstantSDNode>(CombinedOffset)) {
-    uint32_t Imm = C->getZExtValue();
+    uint64_t Imm = C->getZExtValue() + InstImmOffset;
     uint32_t SOffset, ImmOffset;
-    if (AMDGPU::splitMUBUFOffset(Imm, SOffset, ImmOffset, Subtarget,
-                                 Alignment)) {
+    if (Imm <= UINT_MAX && AMDGPU::splitMUBUFOffset(Imm, SOffset, ImmOffset,
+                                                    Subtarget, Alignment)) {
       Offsets[0] = DAG.getConstant(0, DL, MVT::i32);
       Offsets[1] = DAG.getConstant(SOffset, DL, MVT::i32);
       Offsets[2] = DAG.getTargetConstant(ImmOffset, DL, MVT::i32);
       return;
     }
   }
   if (DAG.isBaseWithConstantOffset(CombinedOffset)) {
     SDValue N0 = CombinedOffset.getOperand(0);
     SDValue N1 = CombinedOffset.getOperand(1);
     uint32_t SOffset, ImmOffset;
-    int Offset = cast<ConstantSDNode>(N1)->getSExtValue();
-    if (Offset >= 0 && AMDGPU::splitMUBUFOffset(Offset, SOffset, ImmOffset,
-                                                Subtarget, Alignment)) {
+    int64_t Imm =
+        cast<ConstantSDNode>(N1)->getSExtValue() + (int64_t)InstImmOffset;
+    if (Imm >= 0 && Imm <= UINT_MAX &&
+        AMDGPU::splitMUBUFOffset(Imm, SOffset, ImmOffset, Subtarget,
+                                 Alignment)) {
       Offsets[0] = N0;
       Offsets[1] = DAG.getConstant(SOffset, DL, MVT::i32);
       Offsets[2] = DAG.getTargetConstant(ImmOffset, DL, MVT::i32);
       return;
     }
   }
+  if (InstImmOffset) {
+    // The extra imm offset in buffer offsets is only supported for an intrinsic
+    // available in gfx9+. This also means that the later call to
+    // splitMUBUFOffset will always succeed.
+    assert(Subtarget->getGeneration() >= AMDGPUSubtarget::GFX9);
+    uint32_t SOffset, ImmOffset;
+    uint32_t Imm = InstImmOffset;
+    bool Res =
+        AMDGPU::splitMUBUFOffset(Imm, SOffset, ImmOffset, Subtarget, Alignment);
+    (void)Res;
+    assert(Res && "Unexpected overflow from InstImmOffset.");
+    Offsets[0] = CombinedOffset;
+    Offsets[1] = DAG.getConstant(SOffset, DL, MVT::i32);
+    Offsets[2] = DAG.getTargetConstant(ImmOffset, DL, MVT::i32);
+    return;
+  }
   Offsets[0] = CombinedOffset;
   Offsets[1] = DAG.getConstant(0, DL, MVT::i32);
   Offsets[2] = DAG.getTargetConstant(0, DL, MVT::i32);
 }
 
 // Handle 8 bit and 16 bit buffer loads
 SDValue SITargetLowering::handleByteShortBufferLoads(SelectionDAG &DAG,
                                                      EVT LoadVT, SDLoc DL,

llvm/lib/Target/AMDGPU/SIInstrInfo.td

 
 //===----------------------------------------------------------------------===//
 // SI DAG Nodes
 //===----------------------------------------------------------------------===//
 
 def AMDGPUclamp : SDNode<"AMDGPUISD::CLAMP", SDTFPUnaryOp>;
 
 def SIsbuffer_load : SDNode<"AMDGPUISD::SBUFFER_LOAD",
-  SDTypeProfile<1, 3, [SDTCisVT<1, v4i32>, SDTCisVT<2, i32>, SDTCisVT<3, i32>]>,
+  SDTypeProfile<1, 4, [SDTCisVT<1, v4i32>, SDTCisVT<2, i32>, SDTCisVT<3, i32>, SDTCisVT<4, i32>]>,
   [SDNPMayLoad, SDNPMemOperand]
 >;
 
 def SIds_ordered_count : SDNode<"AMDGPUISD::DS_ORDERED_COUNT",
   SDTypeProfile<1, 2, [SDTCisVT<0, i32>, SDTCisVT<1, i32>, SDTCisVT<2, i16>]>,
   [SDNPMayLoad, SDNPMayStore, SDNPMemOperand, SDNPHasChain, SDNPInGlue]
 >;
 

llvm/lib/Target/AMDGPU/SIInstructions.td

def G_AMDGPU_BUFFER_ATOMIC_CMPSWAP : AMDGPUGenericInstruction {
   let mayStore = 1;
 }
 
 // Wrapper around llvm.amdgcn.s.buffer.load. This is mostly needed as
 // a workaround for the intrinsic being defined as readnone, but
 // really needs a memory operand.
 def G_AMDGPU_S_BUFFER_LOAD : AMDGPUGenericInstruction {
   let OutOperandList = (outs type0:$dst);
-  let InOperandList = (ins type1:$rsrc, type2:$offset, untyped_imm_0:$cachepolicy);
+  let InOperandList = (ins type1:$rsrc, type2:$offset, untyped_imm_0:$imm_offset, untyped_imm_0:$cachepolicy);
   let hasSideEffects = 0;
   let mayLoad = 1;
   let mayStore = 0;
 }
 
 // This is equivalent to the G_INTRINSIC*, but the operands may have
 // been legalized depending on the subtarget requirements.
 def G_AMDGPU_INTRIN_IMAGE_LOAD : AMDGPUGenericInstruction {

llvm/lib/Target/AMDGPU/SMInstructions.td

def : GCNPat <
     (vt (smrd_load (i64 SReg_64:$sbase))),
     (vt (!cast<SM_Pseudo>(Instr#"_IMM") i64:$sbase, 0, 0))
   >;
 }
 
 multiclass SMLoad_Pattern <string Instr, ValueType vt> {
   // 1. Offset as an immediate
   def : GCNPat <
-    (SIsbuffer_load v4i32:$sbase, (SMRDBufferImm i32:$offset), timm:$cachepolicy),
+    (SIsbuffer_load v4i32:$sbase, (SMRDBufferImm i32:$offset), 0, timm:$cachepolicy),
     (vt (!cast<SM_Pseudo>(Instr#"_IMM") SReg_128:$sbase, i32imm:$offset, (extract_cpol $cachepolicy)))> {
     let AddedComplexity = 2;
   }
 
   // 2. 32-bit IMM offset on CI
   def : GCNPat <
-    (vt (SIsbuffer_load v4i32:$sbase, (SMRDBufferImm32 i32:$offset), timm:$cachepolicy)),
+    (vt (SIsbuffer_load v4i32:$sbase, (SMRDBufferImm32 i32:$offset), 0, timm:$cachepolicy)),
     (!cast<InstSI>(Instr#"_IMM_ci") SReg_128:$sbase, smrd_literal_offset:$offset,
                                     (extract_cpol $cachepolicy))> {
     let OtherPredicates = [isGFX7Only];
     let AddedComplexity = 1;
   }
 
   // 3. Offset loaded in an 32bit SGPR
   def : GCNPat <
-    (SIsbuffer_load v4i32:$sbase, i32:$soffset, timm:$cachepolicy),
+    (SIsbuffer_load v4i32:$sbase, i32:$soffset, 0, timm:$cachepolicy),
     (vt (!cast<SM_Pseudo>(Instr#"_SGPR") SReg_128:$sbase, SReg_32:$soffset, (extract_cpol $cachepolicy)))
   >;
 
   // 4. Offset as an 32-bit SGPR + immediate
   def : GCNPat <
-    (SIsbuffer_load v4i32:$sbase, (SMRDBufferSgprImm i32:$soffset, i32:$offset),
+    (SIsbuffer_load v4i32:$sbase, (SMRDBufferSgprImm i32:$soffset, i32:$offset), 0,
                     timm:$cachepolicy),
     (vt (!cast<SM_Pseudo>(Instr#"_SGPR_IMM") SReg_128:$sbase, SReg_32:$soffset, i32imm:$offset,
                                              (extract_cpol $cachepolicy)))> {
     let OtherPredicates = [isGFX9Plus];
   }
+
+  // 5. Offset as a 32-bit SGPR and separate immediate instruction offset.
+  def : GCNPat <
+    (SIsbuffer_load v4i32:$sbase, i32:$soffset, i32:$offset,
+                    timm:$cachepolicy),
+    (vt (!cast<SM_Pseudo>(Instr#"_SGPR_IMM") SReg_128:$sbase, SReg_32:$soffset, smrd_literal_offset:$offset,
+                                             (extract_cpol $cachepolicy)))> {
+    let OtherPredicates = [isGFX9Plus];
+  }
 }
 
 // Global and constant loads can be selected to either MUBUF or SMRD
 // instructions, but SMRD instructions are faster so we want the instruction
 // selector to prefer those.
 let AddedComplexity = 100 in {
 
 foreach vt = Reg32Types.types in {

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-llvm.amdgcn.s.buffer.load.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple=amdgcn-mesa-mesa3d -mcpu=tahiti -run-pass=legalizer %s -o - | FileCheck -check-prefix=GCN %s
 # RUN: llc -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -run-pass=legalizer %s -o - | FileCheck -check-prefix=GCN %s
 
 ---
 name: s_buffer_load_s32
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_s32
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(s32) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s32))
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(s32) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s32))
     ; GCN-NEXT: S_ENDPGM 0, implicit [[AMDGPU_S_BUFFER_LOAD]](s32)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...
 
 ---
 name: s_buffer_load_v3s32
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_v3s32
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32), [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
     ; GCN-NEXT: [[BUILD_VECTOR:%[0-9]+]]:_(<3 x s32>) = G_BUILD_VECTOR [[UV]](s32), [[UV1]](s32), [[UV2]](s32)
     ; GCN-NEXT: S_ENDPGM 0, implicit [[BUILD_VECTOR]](<3 x s32>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<3 x s32>) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...
 
 ---
 name: s_buffer_load_v3p3
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_v3p3
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32), [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
     ; GCN-NEXT: [[BUILD_VECTOR:%[0-9]+]]:_(<3 x s32>) = G_BUILD_VECTOR [[UV]](s32), [[UV1]](s32), [[UV2]](s32)
     ; GCN-NEXT: [[BITCAST:%[0-9]+]]:_(<3 x p3>) = G_BITCAST [[BUILD_VECTOR]](<3 x s32>)
     ; GCN-NEXT: S_ENDPGM 0, implicit [[BITCAST]](<3 x p3>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<3 x p3>) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...
 
 ---
 name: s_buffer_load_v6s16
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_v6s16
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32), [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
     ; GCN-NEXT: [[BUILD_VECTOR:%[0-9]+]]:_(<3 x s32>) = G_BUILD_VECTOR [[UV]](s32), [[UV1]](s32), [[UV2]](s32)
     ; GCN-NEXT: [[BITCAST:%[0-9]+]]:_(<6 x s16>) = G_BITCAST [[BUILD_VECTOR]](<3 x s32>)
     ; GCN-NEXT: S_ENDPGM 0, implicit [[BITCAST]](<6 x s16>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<6 x s16>) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...
 
 ---
 name: s_buffer_load_v6s32
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_v6s32
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<8 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s192), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<8 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s192), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32), [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32), [[UV4:%[0-9]+]]:_(s32), [[UV5:%[0-9]+]]:_(s32), [[UV6:%[0-9]+]]:_(s32), [[UV7:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<8 x s32>)
     ; GCN-NEXT: [[BUILD_VECTOR:%[0-9]+]]:_(<6 x s32>) = G_BUILD_VECTOR [[UV]](s32), [[UV1]](s32), [[UV2]](s32), [[UV3]](s32), [[UV4]](s32), [[UV5]](s32)
     ; GCN-NEXT: S_ENDPGM 0, implicit [[BUILD_VECTOR]](<6 x s32>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<6 x s32>) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...
 
 ---
 name: s_buffer_load_v3s64
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_v3s64
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s64>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s192), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s64>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s192), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s64), [[UV1:%[0-9]+]]:_(s64), [[UV2:%[0-9]+]]:_(s64), [[UV3:%[0-9]+]]:_(s64) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s64>)
     ; GCN-NEXT: [[BUILD_VECTOR:%[0-9]+]]:_(<3 x s64>) = G_BUILD_VECTOR [[UV]](s64), [[UV1]](s64), [[UV2]](s64)
     ; GCN-NEXT: S_ENDPGM 0, implicit [[BUILD_VECTOR]](<3 x s64>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<3 x s64>) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...
 
 ---
 name: s_buffer_load_v12s8
 body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_v12s8
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32), [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
     ; GCN-NEXT: [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 8
     ; GCN-NEXT: [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[UV]], [[C1]](s32)
     ; GCN-NEXT: [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 16
     ; GCN-NEXT: [[LSHR1:%[0-9]+]]:_(s32) = G_LSHR [[UV]], [[C2]](s32)
     ; GCN-NEXT: [[C3:%[0-9]+]]:_(s32) = G_CONSTANT i32 24
     ; GCN-NEXT: [[LSHR2:%[0-9]+]]:_(s32) = G_LSHR [[UV]], [[C3]](s32)
     ; GCN-NEXT: [[LSHR3:%[0-9]+]]:_(s32) = G_LSHR [[UV1]], [[C1]](s32)
body:             |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3
 
     ; GCN-LABEL: name: s_buffer_load_s96
     ; GCN: liveins: $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: {{  $}}
     ; GCN-NEXT: [[COPY:%[0-9]+]]:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; GCN-NEXT: [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
-    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+    ; GCN-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
     ; GCN-NEXT: [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32), [[UV2:%[0-9]+]]:_(s32), [[UV3:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
     ; GCN-NEXT: [[BUILD_VECTOR:%[0-9]+]]:_(<3 x s32>) = G_BUILD_VECTOR [[UV]](s32), [[UV1]](s32), [[UV2]](s32)
     ; GCN-NEXT: S_ENDPGM 0, implicit [[BUILD_VECTOR]](<3 x s32>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = G_CONSTANT i32 0
     %2:_(<3 x s32>) = G_INTRINSIC intrinsic(@llvm.amdgcn.s.buffer.load), %0, %1, 0
     S_ENDPGM 0, implicit %2
 
 ...

llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn-s-buffer-load.mir

body: |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4
 
     ; CHECK-LABEL: name: buffer_load_ss
     ; CHECK: liveins: $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:sgpr(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
-    ; CHECK-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[COPY1]](s32), 0
+    ; CHECK-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[COPY1]](s32), 0, 0
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = COPY $sgpr4
-    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0
+    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0, 0
 
 ...
 
 ---
 name: buffer_load_sv
 legalized: true
 tracksRegLiveness: true
 body: |
   bb.0:
     liveins: $sgpr0_sgpr1_sgpr2_sgpr3, $vgpr0
 
     ; CHECK-LABEL: name: buffer_load_sv
     ; CHECK: liveins: $sgpr0_sgpr1_sgpr2_sgpr3, $vgpr0
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:sgpr(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:vgpr(s32) = COPY $vgpr0
     ; CHECK-NEXT: [[C:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 0
     ; CHECK-NEXT: [[C1:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
     ; CHECK-NEXT: [[AMDGPU_BUFFER_LOAD:%[0-9]+]]:vgpr(<4 x s32>) = G_AMDGPU_BUFFER_LOAD [[COPY]](<4 x s32>), [[C1]](s32), [[COPY1]], [[C]], 0, 0, 0 :: (dereferenceable invariant load (s128), align 4)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = COPY $vgpr0
-    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0
+    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0, 0
 
 ...
 
 ---
 name: buffer_load_vs
 legalized: true
 tracksRegLiveness: true
 body: |
bb.0:
     ; CHECK-NEXT: .3:
     ; CHECK-NEXT: successors: %bb.4(0x80000000)
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: $exec = S_MOV_B64_term [[S_MOV_B64_]]
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: .4:
     %0:_(<4 x s32>) = COPY $vgpr0_vgpr1_vgpr2_vgpr3
     %1:_(s32) = COPY $sgpr0
-    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0
+    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0, 0
 
 ...
 
 ---
 name: buffer_load_vv
 legalized: true
 tracksRegLiveness: true
 body: |
bb.0:
     ; CHECK-NEXT: .3:
     ; CHECK-NEXT: successors: %bb.4(0x80000000)
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: $exec = S_MOV_B64_term [[S_MOV_B64_]]
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: .4:
     %0:_(<4 x s32>) = COPY $vgpr0_vgpr1_vgpr2_vgpr3
     %1:_(s32) = COPY $vgpr4
-    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0
+    %2:_(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD %0, %1, 0, 0
 
 ...

llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.s.buffer.load.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 ; RUN: llc -amdgpu-global-isel-new-legality -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -simplify-mir -stop-after=regbankselect -regbankselect-fast -o - %s | FileCheck %s
 ; RUN: llc -amdgpu-global-isel-new-legality -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=hawaii -simplify-mir -stop-after=regbankselect -regbankselect-greedy -o - %s | FileCheck %s --check-prefix=GREEDY
 
 ; Natural mapping
 define amdgpu_ps i32 @s_buffer_load_i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; CHECK-LABEL: name: s_buffer_load_i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; CHECK-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(s32) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s32))
+  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(s32) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s32))
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[AMDGPU_S_BUFFER_LOAD]](s32)
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; CHECK-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; CHECK-NEXT:   SI_RETURN_TO_EPILOG implicit $sgpr0
   ; GREEDY-LABEL: name: s_buffer_load_i32
   ; GREEDY: bb.1 (%ir-block.0):
   ; GREEDY-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; GREEDY-NEXT: {{  $}}
   ; GREEDY-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; GREEDY-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; GREEDY-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; GREEDY-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; GREEDY-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; GREEDY-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(s32) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s32))
+  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(s32) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s32))
   ; GREEDY-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[AMDGPU_S_BUFFER_LOAD]](s32)
   ; GREEDY-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; GREEDY-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; GREEDY-NEXT:   SI_RETURN_TO_EPILOG implicit $sgpr0
   %val = call i32 @llvm.amdgcn.s.buffer.load.i32(<4 x i32> %rsrc, i32 %soffset, i32 0)
   ret i32 %val
 }
 
 define amdgpu_ps <2 x i32> @s_buffer_load_v2i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; CHECK-LABEL: name: s_buffer_load_v2i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; CHECK-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<2 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s64), align 4)
+  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<2 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s64), align 4)
   ; CHECK-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<2 x s32>)
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; CHECK-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; CHECK-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; CHECK-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; CHECK-NEXT:   SI_RETURN_TO_EPILOG implicit $sgpr0, implicit $sgpr1
   ; GREEDY-LABEL: name: s_buffer_load_v2i32
   ; GREEDY: bb.1 (%ir-block.0):
   ; GREEDY-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; GREEDY-NEXT: {{  $}}
   ; GREEDY-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; GREEDY-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; GREEDY-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; GREEDY-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; GREEDY-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; GREEDY-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<2 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s64), align 4)
+  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<2 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s64), align 4)
   ; GREEDY-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<2 x s32>)
   ; GREEDY-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; GREEDY-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; GREEDY-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; GREEDY-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; GREEDY-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; GREEDY-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; GREEDY-NEXT:   SI_RETURN_TO_EPILOG implicit $sgpr0, implicit $sgpr1
   %val = call <2 x i32> @llvm.amdgcn.s.buffer.load.v2i32(<4 x i32> %rsrc, i32 %soffset, i32 0)
   ret <2 x i32> %val
 }
 
 define amdgpu_ps <3 x i32> @s_buffer_load_v3i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; CHECK-LABEL: name: s_buffer_load_v3i32
   ; CHECK: bb.1 (%ir-block.0):
   ; CHECK-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; CHECK-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
   ; CHECK-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32), [[UV2:%[0-9]+]]:sgpr(s32), [[UV3:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; CHECK-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; CHECK-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; CHECK-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[UV2]](s32)
   ; CHECK-NEXT:   [[INT2:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY7]](s32)
   ; CHECK-NEXT:   $sgpr2 = COPY [[INT2]](s32)
   ; CHECK-NEXT:   SI_RETURN_TO_EPILOG implicit $sgpr0, implicit $sgpr1, implicit $sgpr2
   ; GREEDY-LABEL: name: s_buffer_load_v3i32
   ; GREEDY: bb.1 (%ir-block.0):
   ; GREEDY-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; GREEDY-NEXT: {{  $}}
   ; GREEDY-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; GREEDY-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; GREEDY-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; GREEDY-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; GREEDY-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; GREEDY-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s96), align 4)
+  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<4 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s96), align 4)
   ; GREEDY-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32), [[UV2:%[0-9]+]]:sgpr(s32), [[UV3:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<4 x s32>)
   ; GREEDY-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; GREEDY-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; GREEDY-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; GREEDY-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; GREEDY-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; GREEDY-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; GREEDY-NEXT:   [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[UV2]](s32)
define amdgpu_ps <8 x i32> @s_buffer_load_v8i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; CHECK-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; CHECK-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<8 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s256), align 4)
+  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<8 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s256), align 4)
   ; CHECK-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32), [[UV2:%[0-9]+]]:sgpr(s32), [[UV3:%[0-9]+]]:sgpr(s32), [[UV4:%[0-9]+]]:sgpr(s32), [[UV5:%[0-9]+]]:sgpr(s32), [[UV6:%[0-9]+]]:sgpr(s32), [[UV7:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<8 x s32>)
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; CHECK-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; CHECK-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; CHECK-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[UV2]](s32)
define amdgpu_ps <8 x i32> @s_buffer_load_v8i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; GREEDY-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; GREEDY-NEXT: {{  $}}
   ; GREEDY-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; GREEDY-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; GREEDY-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; GREEDY-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; GREEDY-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; GREEDY-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<8 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s256), align 4)
+  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<8 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s256), align 4)
   ; GREEDY-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32), [[UV2:%[0-9]+]]:sgpr(s32), [[UV3:%[0-9]+]]:sgpr(s32), [[UV4:%[0-9]+]]:sgpr(s32), [[UV5:%[0-9]+]]:sgpr(s32), [[UV6:%[0-9]+]]:sgpr(s32), [[UV7:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<8 x s32>)
   ; GREEDY-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; GREEDY-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; GREEDY-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; GREEDY-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; GREEDY-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; GREEDY-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; GREEDY-NEXT:   [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[UV2]](s32)
define amdgpu_ps <16 x i32> @s_buffer_load_v16i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; CHECK-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; CHECK-NEXT: {{  $}}
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; CHECK-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; CHECK-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; CHECK-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; CHECK-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<16 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s512), align 4)
+  ; CHECK-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<16 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s512), align 4)
   ; CHECK-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32), [[UV2:%[0-9]+]]:sgpr(s32), [[UV3:%[0-9]+]]:sgpr(s32), [[UV4:%[0-9]+]]:sgpr(s32), [[UV5:%[0-9]+]]:sgpr(s32), [[UV6:%[0-9]+]]:sgpr(s32), [[UV7:%[0-9]+]]:sgpr(s32), [[UV8:%[0-9]+]]:sgpr(s32), [[UV9:%[0-9]+]]:sgpr(s32), [[UV10:%[0-9]+]]:sgpr(s32), [[UV11:%[0-9]+]]:sgpr(s32), [[UV12:%[0-9]+]]:sgpr(s32), [[UV13:%[0-9]+]]:sgpr(s32), [[UV14:%[0-9]+]]:sgpr(s32), [[UV15:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<16 x s32>)
   ; CHECK-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; CHECK-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; CHECK-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; CHECK-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; CHECK-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; CHECK-NEXT:   [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[UV2]](s32)
define amdgpu_ps <16 x i32> @s_buffer_load_v16i32(<4 x i32> inreg %rsrc, i32 inreg %soffset) {
   ; GREEDY-NEXT:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6
   ; GREEDY-NEXT: {{  $}}
   ; GREEDY-NEXT:   [[COPY:%[0-9]+]]:sgpr(s32) = COPY $sgpr2
   ; GREEDY-NEXT:   [[COPY1:%[0-9]+]]:sgpr(s32) = COPY $sgpr3
   ; GREEDY-NEXT:   [[COPY2:%[0-9]+]]:sgpr(s32) = COPY $sgpr4
   ; GREEDY-NEXT:   [[COPY3:%[0-9]+]]:sgpr(s32) = COPY $sgpr5
   ; GREEDY-NEXT:   [[BUILD_VECTOR:%[0-9]+]]:sgpr(<4 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32)
   ; GREEDY-NEXT:   [[COPY4:%[0-9]+]]:sgpr(s32) = COPY $sgpr6
-  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<16 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0 :: (dereferenceable invariant load (s512), align 4)
+  ; GREEDY-NEXT:   [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(<16 x s32>) = G_AMDGPU_S_BUFFER_LOAD [[BUILD_VECTOR]](<4 x s32>), [[COPY4]](s32), 0, 0 :: (dereferenceable invariant load (s512), align 4)
   ; GREEDY-NEXT:   [[UV:%[0-9]+]]:sgpr(s32), [[UV1:%[0-9]+]]:sgpr(s32), [[UV2:%[0-9]+]]:sgpr(s32), [[UV3:%[0-9]+]]:sgpr(s32), [[UV4:%[0-9]+]]:sgpr(s32), [[UV5:%[0-9]+]]:sgpr(s32), [[UV6:%[0-9]+]]:sgpr(s32), [[UV7:%[0-9]+]]:sgpr(s32), [[UV8:%[0-9]+]]:sgpr(s32), [[UV9:%[0-9]+]]:sgpr(s32), [[UV10:%[0-9]+]]:sgpr(s32), [[UV11:%[0-9]+]]:sgpr(s32), [[UV12:%[0-9]+]]:sgpr(s32), [[UV13:%[0-9]+]]:sgpr(s32), [[UV14:%[0-9]+]]:sgpr(s32), [[UV15:%[0-9]+]]:sgpr(s32) = G_UNMERGE_VALUES [[AMDGPU_S_BUFFER_LOAD]](<16 x s32>)
   ; GREEDY-NEXT:   [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[UV]](s32)
   ; GREEDY-NEXT:   [[INT:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY5]](s32)
   ; GREEDY-NEXT:   $sgpr0 = COPY [[INT]](s32)
   ; GREEDY-NEXT:   [[COPY6:%[0-9]+]]:vgpr(s32) = COPY [[UV1]](s32)
   ; GREEDY-NEXT:   [[INT1:%[0-9]+]]:sgpr(s32) = G_INTRINSIC intrinsic(@llvm.amdgcn.readfirstlane), [[COPY6]](s32)
   ; GREEDY-NEXT:   $sgpr1 = COPY [[INT1]](s32)
   ; GREEDY-NEXT:   [[COPY7:%[0-9]+]]:vgpr(s32) = COPY [[UV2]](s32)

llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-amdgcn.s.buffer.load.mir

bb.0:
     ; GREEDY-NEXT: [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 0
     ; GREEDY-NEXT: [[C2:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
     ; GREEDY-NEXT: [[AMDGPU_BUFFER_LOAD:%[0-9]+]]:vgpr(s32) = G_AMDGPU_BUFFER_LOAD [[COPY]](<4 x s32>), [[C2]](s32), [[COPY2]], [[C1]], 256, 0, 0 :: (dereferenceable invariant load (s32))
     ; GREEDY-NEXT: S_ENDPGM 0, implicit [[AMDGPU_BUFFER_LOAD]](s32)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = COPY $sgpr0
     %2:vgpr(s32) = G_CONSTANT i32 256
     %3:_(s32) = G_ADD %1, %2
-    %4:_(s32) = G_AMDGPU_S_BUFFER_LOAD %0, %3, 0
+    %4:_(s32) = G_AMDGPU_S_BUFFER_LOAD %0, %3, 0, 0
     S_ENDPGM 0, implicit %4
 
 ...
 
 ---
 name:            s_buffer_load_negative_offset
 legalized:       true
 tracksRegLiveness: true
bb.0:
     ; GREEDY-NEXT: [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 0
     ; GREEDY-NEXT: [[C2:%[0-9]+]]:vgpr(s32) = G_CONSTANT i32 0
     ; GREEDY-NEXT: [[AMDGPU_BUFFER_LOAD:%[0-9]+]]:vgpr(s32) = G_AMDGPU_BUFFER_LOAD [[COPY]](<4 x s32>), [[C2]](s32), [[ADD]], [[C1]], 0, 0, 0 :: (dereferenceable invariant load (s32))
     ; GREEDY-NEXT: S_ENDPGM 0, implicit [[AMDGPU_BUFFER_LOAD]](s32)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(s32) = COPY $vgpr0
     %2:_(s32) = G_CONSTANT i32 -60
     %3:_(s32) = G_ADD %1, %2
-    %4:_(s32) = G_AMDGPU_S_BUFFER_LOAD %0, %3, 0
+    %4:_(s32) = G_AMDGPU_S_BUFFER_LOAD %0, %3, 0, 0
     S_ENDPGM 0, implicit %4
 
 ...

llvm/test/CodeGen/AMDGPU/GlobalISel/regbankselect-insert-vector-elt.mir

bb.0:
 
     ; CHECK-LABEL: name: insert_vector_elt_with_s_buffer_load
     ; CHECK: liveins: $vgpr0, $sgpr0_sgpr1_sgpr2_sgpr3, $sgpr4_sgpr5
     ; CHECK-NEXT: {{  $}}
     ; CHECK-NEXT: [[COPY:%[0-9]+]]:sgpr(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     ; CHECK-NEXT: [[COPY1:%[0-9]+]]:sgpr(<2 x s32>) = COPY $sgpr4_sgpr5
     ; CHECK-NEXT: [[COPY2:%[0-9]+]]:vgpr(s32) = COPY $vgpr0
     ; CHECK-NEXT: [[C:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 0
-    ; CHECK-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(s32) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0 :: (dereferenceable invariant load (s32))
+    ; CHECK-NEXT: [[AMDGPU_S_BUFFER_LOAD:%[0-9]+]]:sgpr(s32) = G_AMDGPU_S_BUFFER_LOAD [[COPY]](<4 x s32>), [[C]](s32), 0, 0 :: (dereferenceable invariant load (s32))
     ; CHECK-NEXT: [[COPY3:%[0-9]+]]:vgpr(<2 x s32>) = COPY [[COPY1]](<2 x s32>)
     ; CHECK-NEXT: [[UV:%[0-9]+]]:vgpr(s32), [[UV1:%[0-9]+]]:vgpr(s32) = G_UNMERGE_VALUES [[COPY3]](<2 x s32>)
     ; CHECK-NEXT: [[C1:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 0
     ; CHECK-NEXT: [[ICMP:%[0-9]+]]:vcc(s1) = G_ICMP intpred(eq), [[COPY2]](s32), [[C1]]
     ; CHECK-NEXT: [[COPY4:%[0-9]+]]:vgpr(s32) = COPY [[AMDGPU_S_BUFFER_LOAD]](s32)
     ; CHECK-NEXT: [[SELECT:%[0-9]+]]:vgpr(s32) = G_SELECT [[ICMP]](s1), [[COPY4]], [[UV]]
     ; CHECK-NEXT: [[C2:%[0-9]+]]:sgpr(s32) = G_CONSTANT i32 1
     ; CHECK-NEXT: [[ICMP1:%[0-9]+]]:vcc(s1) = G_ICMP intpred(eq), [[COPY2]](s32), [[C2]]
     ; CHECK-NEXT: [[COPY5:%[0-9]+]]:vgpr(s32) = COPY [[AMDGPU_S_BUFFER_LOAD]](s32)
     ; CHECK-NEXT: [[SELECT1:%[0-9]+]]:vgpr(s32) = G_SELECT [[ICMP1]](s1), [[COPY5]], [[UV1]]
     ; CHECK-NEXT: [[BUILD_VECTOR:%[0-9]+]]:vgpr(<2 x s32>) = G_BUILD_VECTOR [[SELECT]](s32), [[SELECT1]](s32)
     ; CHECK-NEXT: S_ENDPGM 0, implicit [[BUILD_VECTOR]](<2 x s32>)
     %0:_(<4 x s32>) = COPY $sgpr0_sgpr1_sgpr2_sgpr3
     %1:_(<2 x s32>) = COPY $sgpr4_sgpr5
     %2:_(s32) = COPY $vgpr0
     %3:_(s32) = G_CONSTANT i32 0
-    %4:_(s32) = G_AMDGPU_S_BUFFER_LOAD %0(<4 x s32>), %3(s32), 0 :: (dereferenceable invariant load (s32))
+    %4:_(s32) = G_AMDGPU_S_BUFFER_LOAD %0(<4 x s32>), %3(s32), 0, 0 :: (dereferenceable invariant load (s32))
     %5:_(<2 x s32>) = G_INSERT_VECTOR_ELT %1, %4(s32), %2(s32)
 
     S_ENDPGM 0, implicit %5
 
 ...

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.s.buffer.load.imm.ll

+; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck %s -check-prefix=GCN
+; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck %s -check-prefix=GCN
+
+; The tests check code generated for @llvm.amdgcn.s.buffer.load.imm.
+; The first group of tests is similar to tests @llvm.amdgcn.s.buffer.load.i32 with instruction immediate offset being 0.
+; The second group of tests checks cases where instruction immediate offset is not equal to 0.
+
+;GCN-LABEL: {{^}}s_buffer_load_imm:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dword s{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0x4
+define amdgpu_ps void @s_buffer_load_imm(<4 x i32> inreg %desc) {
+main_body:
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 4, i32 0, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_load_index:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dword s{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
+define amdgpu_ps void @s_buffer_load_index(<4 x i32> inreg %desc, i32 inreg %index) {
+main_body:
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_load_index_divergent:
+;GCN-NOT: s_waitcnt;
+;GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+define amdgpu_ps void @s_buffer_load_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+main_body:
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx2_imm:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x40
+define amdgpu_ps void @s_buffer_loadx2_imm(<4 x i32> inreg %desc) {
+main_body:
+  %load = call <2 x i32> @llvm.amdgcn.s.buffer.load.imm.v2i32(<4 x i32> %desc, i32 64, i32 0, i32 0)
+  %bitcast = bitcast <2 x i32> %load to <2 x float>
+  %x = extractelement <2 x float> %bitcast, i32 0
+  %y = extractelement <2 x float> %bitcast, i32 1
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx2_index:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
+define amdgpu_ps void @s_buffer_loadx2_index(<4 x i32> inreg %desc, i32 inreg %index) {
+main_body:
+  %load = call <2 x i32> @llvm.amdgcn.s.buffer.load.imm.v2i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast <2 x i32> %load to <2 x float>
+  %x = extractelement <2 x float> %bitcast, i32 0
+  %y = extractelement <2 x float> %bitcast, i32 1
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx2_index_divergent:
+;GCN-NOT: s_waitcnt;
+;GCN: buffer_load_dwordx2 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+define amdgpu_ps void @s_buffer_loadx2_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+main_body:
+  %load = call <2 x i32> @llvm.amdgcn.s.buffer.load.imm.v2i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast <2 x i32> %load to <2 x float>
+  %x = extractelement <2 x float> %bitcast, i32 0
+  %y = extractelement <2 x float> %bitcast, i32 1
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx3_imm:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x40
+define amdgpu_ps void @s_buffer_loadx3_imm(<4 x i32> inreg %desc) {
+main_body:
+  %load = call <3 x i32> @llvm.amdgcn.s.buffer.load.imm.v3i32(<4 x i32> %desc, i32 64, i32 0, i32 0)
+  %bitcast = bitcast <3 x i32> %load to <3 x float>
+  %x = extractelement <3 x float> %bitcast, i32 0
+  %y = extractelement <3 x float> %bitcast, i32 1
+  %z = extractelement <3 x float> %bitcast, i32 2
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx3_index:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
+define amdgpu_ps void @s_buffer_loadx3_index(<4 x i32> inreg %desc, i32 inreg %index) {
+main_body:
+  %load = call <3 x i32> @llvm.amdgcn.s.buffer.load.imm.v3i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast <3 x i32> %load to <3 x float>
+  %x = extractelement <3 x float> %bitcast, i32 0
+  %y = extractelement <3 x float> %bitcast, i32 1
+  %z = extractelement <3 x float> %bitcast, i32 2
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx3_index_divergent:
+;GCN-NOT: s_waitcnt;
+;GCN: buffer_load_dwordx3 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+define amdgpu_ps void @s_buffer_loadx3_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+main_body:
+  %load = call <3 x i32> @llvm.amdgcn.s.buffer.load.imm.v3i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast <3 x i32> %load to <3 x float>
+  %x = extractelement <3 x float> %bitcast, i32 0
+  %y = extractelement <3 x float> %bitcast, i32 1
+  %z = extractelement <3 x float> %bitcast, i32 2
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx4_imm:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0xc8
+define amdgpu_ps void @s_buffer_loadx4_imm(<4 x i32> inreg %desc) {
+main_body:
+  %load = call <4 x i32> @llvm.amdgcn.s.buffer.load.imm.v4i32(<4 x i32> %desc, i32 200, i32 0, i32 0)
+  %bitcast = bitcast <4 x i32> %load to <4 x float>
+  %x = extractelement <4 x float> %bitcast, i32 0
+  %y = extractelement <4 x float> %bitcast, i32 1
+  %z = extractelement <4 x float> %bitcast, i32 2
+  %w = extractelement <4 x float> %bitcast, i32 3
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float %w, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx4_index:
+;GCN-NOT: s_waitcnt;
+;GCN: buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], s{{[0-9]+}}
+define amdgpu_ps void @s_buffer_loadx4_index(<4 x i32> inreg %desc, i32 inreg %index) {
+main_body:
+  %load = call <4 x i32> @llvm.amdgcn.s.buffer.load.imm.v4i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast <4 x i32> %load to <4 x float>
+  %x = extractelement <4 x float> %bitcast, i32 0
+  %y = extractelement <4 x float> %bitcast, i32 1
+  %z = extractelement <4 x float> %bitcast, i32 2
+  %w = extractelement <4 x float> %bitcast, i32 3
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float %w, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_loadx4_index_divergent:
+;GCN-NOT: s_waitcnt;
+;GCN: buffer_load_dwordx4 v[{{[0-9]+:[0-9]+}}], v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+define amdgpu_ps void @s_buffer_loadx4_index_divergent(<4 x i32> inreg %desc, i32 %index) {
+main_body:
+  %load = call <4 x i32> @llvm.amdgcn.s.buffer.load.imm.v4i32(<4 x i32> %desc, i32 %index, i32 0, i32 0)
+  %bitcast = bitcast <4 x i32> %load to <4 x float>
+  %x = extractelement <4 x float> %bitcast, i32 0
+  %y = extractelement <4 x float> %bitcast, i32 1
+  %z = extractelement <4 x float> %bitcast, i32 2
+  %w = extractelement <4 x float> %bitcast, i32 3
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float %w, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_load_imm_mergex2:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx2 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x4
+define amdgpu_ps void @s_buffer_load_imm_mergex2(<4 x i32> inreg %desc) {
+main_body:
+  %load0 = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 4, i32 0, i32 0)
+  %load1 = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 8, i32 0, i32 0)
+  %x = bitcast i32 %load0 to float
+  %y = bitcast i32 %load1 to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_load_imm_mergex4:
+;GCN-NOT: s_waitcnt;
+;GCN: s_buffer_load_dwordx4 s[{{[0-9]+:[0-9]+}}], s[{{[0-9]+:[0-9]+}}], 0x8
+define amdgpu_ps void @s_buffer_load_imm_mergex4(<4 x i32> inreg %desc) {
+main_body:
+  %load0 = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 8, i32 0, i32 0)
+  %load1 = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 12, i32 0, i32 0)
+  %load2 = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 16, i32 0, i32 0)
+  %load3 = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 20, i32 0, i32 0)
+  %x = bitcast i32 %load0 to float
+  %y = bitcast i32 %load1 to float
+  %z = bitcast i32 %load2 to float
+  %w = bitcast i32 %load3 to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %x, float %y, float %z, float %w, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_load_index_across_bb:
+;GCN-NOT: s_waitcnt;
+;GCN: v_or_b32
+;GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+define amdgpu_ps void @s_buffer_load_index_across_bb(<4 x i32> inreg %desc, i32 %index) {
+main_body:
+  %tmp = shl i32 %index, 4
+  br label %bb1
+
+bb1:                                              ; preds = %main_body
+  %tmp1 = or i32 %tmp, 8
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %tmp1, i32 0, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+;GCN-LABEL: {{^}}s_buffer_load_index_across_bb_merged:
+;GCN-NOT: s_waitcnt;
+;GCN: v_or_b32
+;GCN: v_or_b32
+;GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+;GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen
+define amdgpu_ps void @s_buffer_load_index_across_bb_merged(<4 x i32> inreg %desc, i32 %index) {
+main_body:
+  %tmp = shl i32 %index, 4
+  br label %bb1
+
+bb1:                                              ; preds = %main_body
+  %tmp1 = or i32 %tmp, 8
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %tmp1, i32 0, i32 0)
+  %tmp2 = or i32 %tmp1, 4
+  %load2 = tail call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %tmp2, i32 0, i32 0)
+  %bitcast = bitcast i32 %load to float
+  %bitcast2 = bitcast i32 %load2 to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float %bitcast2, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_neg1:
+; GCN: s_mov_b32 [[K:s[0-9]+]], -1{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_neg1(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 -1, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_neg4:
+; GCN: s_mov_b32 [[K:s[0-9]+]], -4{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_neg4(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 -4, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_neg8:
+; GCN: s_mov_b32 [[K:s[0-9]+]], -8{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_neg8(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 -8, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_bit31:
+; GCN: s_brev_b32 [[K:s[0-9]+]], 1{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_bit31(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 -2147483648, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_bit30:
+; GCN: s_mov_b32 [[K:s[0-9]+]], 2.0{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_bit30(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1073741824, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_bit29:
+; GCN: s_brev_b32 [[K:s[0-9]+]], 4{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_bit29(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 536870912, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_bit21:
+; GCN: s_mov_b32 [[K:s[0-9]+]], 0x200000{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_bit21(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 2097152, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_bit20:
+; GCN: s_mov_b32 [[K:s[0-9]+]], 0x100000{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_bit20(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1048576, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_neg_bit20:
+; GCN: s_mov_b32 [[K:s[0-9]+]], 0xfff00000{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_neg_bit20(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32  -1048576, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_bit19:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x80000{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_bit19(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 524288, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_neg_bit19:
+; GCN: s_mov_b32 [[K:s[0-9]+]], 0xfff80000{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], [[K]]{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_neg_bit19(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 -524288, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_255:
+; GCN: s_buffer_load_dword s0, s[0:3], 0xff{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_255(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 255, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_256:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x100{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_256(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 256, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_1016:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x3f8{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_1016(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1016, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_1020:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x3fc{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_1020(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1020, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_1021:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x3fd{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_1021(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1021, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_1024:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x400{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_1024(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1024, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_imm_1025:
+; GCN: s_buffer_load_dword s0, s[0:3], 0x401{{$}}
+define amdgpu_ps i32 @s_buffer_load_imm_1025(<4 x i32> inreg %desc) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 1025, i32 0, i32 0)
+  ret i32 %load
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_index_imm_1024:
+; GCN: s_buffer_load_dword s0, s[0:3], s4 offset:0x400{{$}}
+define amdgpu_ps void @s_buffer_load_index_imm_1024(<4 x i32> inreg %desc, i32 inreg %index) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 1024, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_index_imm_0xffffc:
+; GCN: s_buffer_load_dword s0, s[0:3], s4 offset:0xffffc{{$}}
+define amdgpu_ps void @s_buffer_load_index_imm_0xffffc(<4 x i32> inreg %desc, i32 inreg %index) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 1048572, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_index_imm_overflow_0x100000:
+; GCN: s_add_i32 s4, s4, 0x100000{{$}}
+; GCN: s_buffer_load_dword s0, s[0:3], s4{{$}}
+define amdgpu_ps void @s_buffer_load_index_imm_overflow_0x100000(<4 x i32> inreg %desc, i32 inreg %index) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 1048576, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_index_imm_overflow_0x100004:
+; GCN: s_buffer_load_dword s0, s[0:3], s4 offset:0x4{{$}}
+define amdgpu_ps void @s_buffer_load_index_imm_overflow_0x100004(<4 x i32> inreg %desc, i32 inreg %index) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 1048580, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_divergent_index_imm_1024:
+; GCN: buffer_load_dword v{{[0-9]+}}, v{{[0-9]+}}, s[{{[0-9]+:[0-9]+}}], 0 offen offset:1024{{$}}
+define amdgpu_ps void @s_buffer_load_divergent_index_imm_1024(<4 x i32> inreg %desc, i32 %index) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 1024, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_divergent_index_imm_5000:
+; GCN: s_movk_i32 s4, 0xffc
+; GCN: buffer_load_dword v0, v0, s[0:3], s4 offen offset:908{{$}}
+define amdgpu_ps void @s_buffer_load_divergent_index_imm_5000(<4 x i32> inreg %desc, i32 %index) {
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %index, i32 5000, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_divergent_index_add_imm_1024:
+; GCN: buffer_load_dword v0, v0, s[0:3], 0 offen offset:1028{{$}}
+define amdgpu_ps void @s_buffer_load_divergent_index_add_imm_1024(<4 x i32> inreg %desc, i32 %index) {
+  %add = add i32 %index, 4
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %add, i32 1024, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_buffer_load_divergent_index_add_imm_5000:
+; GCN: s_movk_i32 s4, 0xffc
+; GCN: buffer_load_dword v0, v0, s[0:3], s4 offen offset:912{{$}}
+define amdgpu_ps void @s_buffer_load_divergent_index_add_imm_5000(<4 x i32> inreg %desc, i32 %index) {
+  %add = add i32 %index, 4
+  %load = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> %desc, i32 %add, i32 5000, i32 0)
+  %bitcast = bitcast i32 %load to float
+  call void @llvm.amdgcn.exp.f32(i32 0, i32 15, float %bitcast, float undef, float undef, float undef, i1 true, i1 true)
+  ret void
+}
+
+declare void @llvm.amdgcn.exp.f32(i32, i32, float, float, float, float, i1, i1)
+declare i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32>, i32, i32, i32)
+declare <2 x i32> @llvm.amdgcn.s.buffer.load.imm.v2i32(<4 x i32>, i32, i32, i32)
+declare <3 x i32> @llvm.amdgcn.s.buffer.load.imm.v3i32(<4 x i32>, i32, i32, i32)
+declare <4 x i32> @llvm.amdgcn.s.buffer.load.imm.v4i32(<4 x i32>, i32, i32, i32)
+

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-demanded-vector-elts.ll

 }
 
 declare i8 @llvm.amdgcn.s.buffer.load.i8(<4 x i32>, i32, i32) #1
 declare <2 x i8> @llvm.amdgcn.s.buffer.load.v2i8(<4 x i32>, i32, i32) #1
 declare <3 x i8> @llvm.amdgcn.s.buffer.load.v3i8(<4 x i32>, i32, i32) #1
 declare <4 x i8> @llvm.amdgcn.s.buffer.load.v4i8(<4 x i32>, i32, i32) #1
 
 ; --------------------------------------------------------------------
+; llvm.amdgcn.s.buffer.load.imm
+; --------------------------------------------------------------------
+
+define amdgpu_ps float @s_buffer_load_imm_f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @s_buffer_load_imm_f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  ret float %data
+}
+
+define amdgpu_ps <2 x float> @s_buffer_load_imm_v2f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @s_buffer_load_imm_v2f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x float> [[DATA]]
+;
+  %data = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  ret <2 x float> %data
+}
+
+define amdgpu_ps <4 x float> @s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <4 x float> [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  ret <4 x float> %data
+}
+
+define amdgpu_ps float @extract_elt0_s_buffer_load_imm_v2f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_s_buffer_load_imm_v2f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt0 = extractelement <2 x float> %data, i32 0
+  ret float %elt0
+}
+
+define amdgpu_ps float @extract_elt1_s_buffer_load_imm_v2f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v2f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 4
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <2 x float> %data, i32 1
+  ret float %elt1
+}
+
+define amdgpu_ps float @extract_elt0_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt0 = extractelement <4 x float> %data, i32 0
+  ret float %elt0
+}
+
+define amdgpu_ps float @extract_elt1_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 4
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x float> %data, i32 1
+  ret float %elt1
+}
+
+define amdgpu_ps float @extract_elt2_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt2_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 8
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x float> %data, i32 2
+  ret float %elt1
+}
+
+define amdgpu_ps float @extract_elt3_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt3_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 12
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x float> %data, i32 3
+  ret float %elt1
+}
+
+define amdgpu_ps <2 x float> @extract_elt0_elt1_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_elt1_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x float> [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x float> %data, <4 x float> poison, <2 x i32> <i32 0, i32 1>
+  ret <2 x float> %shuf
+}
+
+define amdgpu_ps <2 x float> @extract_elt1_elt2_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_elt2_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 4
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x float> [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x float> %data, <4 x float> poison, <2 x i32> <i32 1, i32 2>
+  ret <2 x float> %shuf
+}
+
+define amdgpu_ps <2 x float> @extract_elt2_elt3_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt2_elt3_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 8
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x float> [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x float> %data, <4 x float> poison, <2 x i32> <i32 2, i32 3>
+  ret <2 x float> %shuf
+}
+
+define amdgpu_ps <3 x float> @extract_elt0_elt1_elt2_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_elt1_elt2_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <3 x float> [[DATA]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x float> %data, <4 x float> poison, <3 x i32> <i32 0, i32 1, i32 2>
+  ret <3 x float> %shuf
+}
+
+define amdgpu_ps <3 x float> @extract_elt0_elt2_elt3_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_elt2_elt3_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    [[SHUF:%.*]] = shufflevector <4 x float> [[DATA]], <4 x float> poison, <3 x i32> <i32 0, i32 2, i32 3>
+; CHECK-NEXT:    ret <3 x float> [[SHUF]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x float> %data, <4 x float> poison, <3 x i32> <i32 0, i32 2, i32 3>
+  ret <3 x float> %shuf
+}
+
+define amdgpu_ps float @extract_elt0_s_buffer_load_imm_v3f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_s_buffer_load_imm_v3f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt0 = extractelement <3 x float> %data, i32 0
+  ret float %elt0
+}
+
+define amdgpu_ps float @extract_elt1_s_buffer_load_imm_v3f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v3f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 4
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <3 x float> %data, i32 1
+  ret float %elt1
+}
+
+define amdgpu_ps float @extract_elt2_s_buffer_load_imm_v3f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt2_s_buffer_load_imm_v3f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 8
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <3 x float> %data, i32 2
+  ret float %elt1
+}
+
+define amdgpu_ps <2 x float> @extract_elt0_elt1_s_buffer_load_imm_v3f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_elt1_s_buffer_load_imm_v3f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x float> [[DATA]]
+;
+  %data = call <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <3 x float> %data, <3 x float> poison, <2 x i32> <i32 0, i32 1>
+  ret <2 x float> %shuf
+}
+
+define amdgpu_ps <2 x float> @extract_elt1_elt2_s_buffer_load_imm_v3f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_elt2_s_buffer_load_imm_v3f32(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 4
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x float> [[DATA]]
+;
+  %data = call <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <3 x float> %data, <3 x float> poison, <2 x i32> <i32 1, i32 2>
+  ret <2 x float> %shuf
+}
+
+; Do not trim to vec3 s_buffer_load_imm in instcombine, as the load will most likely be widened
+; to vec4 anyway during lowering.
+define amdgpu_ps <3 x float> @extract_elt1_elt2_elt3_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_elt2_elt3_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    [[SHUF:%.*]] = shufflevector <4 x float> [[DATA]], <4 x float> poison, <3 x i32> <i32 1, i32 2, i32 3>
+; CHECK-NEXT:    ret <3 x float> [[SHUF]]
+;
+  %data = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x float> %data, <4 x float> poison, <3 x i32> <i32 1, i32 2, i32 3>
+  ret <3 x float> %shuf
+}
+
+define i32 @extract0_bitcast_s_buffer_load_imm_v4f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract0_bitcast_s_buffer_load_imm_v4f32(
+; CHECK-NEXT:    [[VAR:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    [[VAR2:%.*]] = bitcast float [[VAR]] to i32
+; CHECK-NEXT:    ret i32 [[VAR2]]
+;
+  %var = call <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %var1 = bitcast <4 x float> %var to <4 x i32>
+  %var2 = extractelement <4 x i32> %var1, i32 0
+  ret i32 %var2
+}
+
+define float @extract0_bitcast_s_buffer_load_imm_v4i32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract0_bitcast_s_buffer_load_imm_v4i32(
+; CHECK-NEXT:    [[VAR:%.*]] = call i32 @llvm.amdgcn.s.buffer.load.imm.i32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    [[VAR2:%.*]] = bitcast i32 [[VAR]] to float
+; CHECK-NEXT:    ret float [[VAR2]]
+;
+  %var = call <4 x i32> @llvm.amdgcn.s.buffer.load.imm.v4i32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %var1 = bitcast <4 x i32> %var to <4 x float>
+  %var2 = extractelement <4 x float> %var1, i32 0
+  ret float %var2
+}
+
+define amdgpu_ps float @preserve_metadata_extract_elt0_s_buffer_load_imm_v2f32(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @preserve_metadata_extract_elt0_s_buffer_load_imm_v2f32(
+; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0), !fpmath !0
+; CHECK-NEXT:    ret float [[DATA]]
+;
+  %data = call <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0), !fpmath !0
+  %elt0 = extractelement <2 x float> %data, i32 0
+  ret float %elt0
+}
+
+declare float @llvm.amdgcn.s.buffer.load.imm.f32(<4 x i32>, i32, i32, i32) #1
+declare <2 x float> @llvm.amdgcn.s.buffer.load.imm.v2f32(<4 x i32>, i32, i32, i32) #1
+declare <3 x float> @llvm.amdgcn.s.buffer.load.imm.v3f32(<4 x i32>, i32, i32, i32) #1
+declare <4 x float> @llvm.amdgcn.s.buffer.load.imm.v4f32(<4 x i32>, i32, i32, i32) #1
+declare <4 x i32> @llvm.amdgcn.s.buffer.load.imm.v4i32(<4 x i32>, i32, i32, i32) #1
+
+define amdgpu_ps half @extract_elt0_s_buffer_load_imm_v2f16(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_s_buffer_load_imm_v2f16(
+; CHECK-NEXT:    [[DATA:%.*]] = call half @llvm.amdgcn.s.buffer.load.imm.f16(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret half [[DATA]]
+;
+  %data = call <2 x half> @llvm.amdgcn.s.buffer.load.imm.v2f16(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt0 = extractelement <2 x half> %data, i32 0
+  ret half %elt0
+}
+
+define amdgpu_ps half @extract_elt1_s_buffer_load_imm_v2f16(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v2f16(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 2
+; CHECK-NEXT:    [[DATA:%.*]] = call half @llvm.amdgcn.s.buffer.load.imm.f16(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret half [[DATA]]
+;
+  %data = call <2 x half> @llvm.amdgcn.s.buffer.load.imm.v2f16(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <2 x half> %data, i32 1
+  ret half %elt1
+}
+
+define amdgpu_ps half @extract_elt1_s_buffer_load_imm_v3f16(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v3f16(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 2
+; CHECK-NEXT:    [[DATA:%.*]] = call half @llvm.amdgcn.s.buffer.load.imm.f16(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret half [[DATA]]
+;
+  %data = call <3 x half> @llvm.amdgcn.s.buffer.load.imm.v3f16(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <3 x half> %data, i32 1
+  ret half %elt1
+}
+
+define amdgpu_ps half @extract_elt1_s_buffer_load_imm_v4f16(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v4f16(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 2
+; CHECK-NEXT:    [[DATA:%.*]] = call half @llvm.amdgcn.s.buffer.load.imm.f16(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret half [[DATA]]
+;
+  %data = call <4 x half> @llvm.amdgcn.s.buffer.load.imm.v4f16(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x half> %data, i32 1
+  ret half %elt1
+}
+
+
+define amdgpu_ps half @extract_elt3_s_buffer_load_imm_v4f16(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt3_s_buffer_load_imm_v4f16(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 6
+; CHECK-NEXT:    [[DATA:%.*]] = call half @llvm.amdgcn.s.buffer.load.imm.f16(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret half [[DATA]]
+;
+  %data = call <4 x half> @llvm.amdgcn.s.buffer.load.imm.v4f16(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x half> %data, i32 3
+  ret half %elt1
+}
+
+define amdgpu_ps <2 x half> @extract_elt0_elt1_s_buffer_load_imm_v4f16(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_elt1_s_buffer_load_imm_v4f16(
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x half> @llvm.amdgcn.s.buffer.load.imm.v2f16(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x half> [[DATA]]
+;
+  %data = call <4 x half> @llvm.amdgcn.s.buffer.load.imm.v4f16(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x half> %data, <4 x half> poison, <2 x i32> <i32 0, i32 1>
+  ret <2 x half> %shuf
+}
+
+declare half @llvm.amdgcn.s.buffer.load.imm.f16(<4 x i32>, i32, i32, i32) #1
+declare <2 x half> @llvm.amdgcn.s.buffer.load.imm.v2f16(<4 x i32>, i32, i32, i32) #1
+declare <3 x half> @llvm.amdgcn.s.buffer.load.imm.v3f16(<4 x i32>, i32, i32, i32) #1
+declare <4 x half> @llvm.amdgcn.s.buffer.load.imm.v4f16(<4 x i32>, i32, i32, i32) #1
+
+define amdgpu_ps i8 @extract_elt0_s_buffer_load_imm_v2i8(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_s_buffer_load_imm_v2i8(
+; CHECK-NEXT:    [[DATA:%.*]] = call i8 @llvm.amdgcn.s.buffer.load.imm.i8(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret i8 [[DATA]]
+;
+  %data = call <2 x i8> @llvm.amdgcn.s.buffer.load.imm.v2i8(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt0 = extractelement <2 x i8> %data, i32 0
+  ret i8 %elt0
+}
+
+define amdgpu_ps i8 @extract_elt1_s_buffer_load_imm_v2i8(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v2i8(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 1
+; CHECK-NEXT:    [[DATA:%.*]] = call i8 @llvm.amdgcn.s.buffer.load.imm.i8(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret i8 [[DATA]]
+;
+  %data = call <2 x i8> @llvm.amdgcn.s.buffer.load.imm.v2i8(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <2 x i8> %data, i32 1
+  ret i8 %elt1
+}
+
+define amdgpu_ps i8 @extract_elt1_s_buffer_load_imm_v3i8(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v3i8(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 1
+; CHECK-NEXT:    [[DATA:%.*]] = call i8 @llvm.amdgcn.s.buffer.load.imm.i8(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret i8 [[DATA]]
+;
+  %data = call <3 x i8> @llvm.amdgcn.s.buffer.load.imm.v3i8(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <3 x i8> %data, i32 1
+  ret i8 %elt1
+}
+
+define amdgpu_ps i8 @extract_elt1_s_buffer_load_imm_v4i8(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt1_s_buffer_load_imm_v4i8(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 1
+; CHECK-NEXT:    [[DATA:%.*]] = call i8 @llvm.amdgcn.s.buffer.load.imm.i8(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret i8 [[DATA]]
+;
+  %data = call <4 x i8> @llvm.amdgcn.s.buffer.load.imm.v4i8(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x i8> %data, i32 1
+  ret i8 %elt1
+}
+
+define amdgpu_ps i8 @extract_elt3_s_buffer_load_imm_v4i8(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt3_s_buffer_load_imm_v4i8(
+; CHECK-NEXT:    [[TMP1:%.*]] = add i32 [[OFS:%.*]], 3
+; CHECK-NEXT:    [[DATA:%.*]] = call i8 @llvm.amdgcn.s.buffer.load.imm.i8(<4 x i32> [[RSRC:%.*]], i32 [[TMP1]], i32 4, i32 0)
+; CHECK-NEXT:    ret i8 [[DATA]]
+;
+  %data = call <4 x i8> @llvm.amdgcn.s.buffer.load.imm.v4i8(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %elt1 = extractelement <4 x i8> %data, i32 3
+  ret i8 %elt1
+}
+
+define amdgpu_ps <2 x i8> @extract_elt0_elt1_s_buffer_load_imm_v4i8(<4 x i32> inreg %rsrc, i32 %ofs) #0 {
+; CHECK-LABEL: @extract_elt0_elt1_s_buffer_load_imm_v4i8(
+; CHECK-NEXT:    [[DATA:%.*]] = call <2 x i8> @llvm.amdgcn.s.buffer.load.imm.v2i8(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 4, i32 0)
+; CHECK-NEXT:    ret <2 x i8> [[DATA]]
+;
+  %data = call <4 x i8> @llvm.amdgcn.s.buffer.load.imm.v4i8(<4 x i32> %rsrc, i32 %ofs, i32 4, i32 0)
+  %shuf = shufflevector <4 x i8> %data, <4 x i8> poison, <2 x i32> <i32 0, i32 1>
+  ret <2 x i8> %shuf
+}
+
+declare i8 @llvm.amdgcn.s.buffer.load.imm.i8(<4 x i32>, i32, i32, i32) #1
+declare <2 x i8> @llvm.amdgcn.s.buffer.load.imm.v2i8(<4 x i32>, i32, i32, i32) #1
+declare <3 x i8> @llvm.amdgcn.s.buffer.load.imm.v3i8(<4 x i32>, i32, i32, i32) #1
+declare <4 x i8> @llvm.amdgcn.s.buffer.load.imm.v4i8(<4 x i32>, i32, i32, i32) #1
+
+
+; --------------------------------------------------------------------
 ; llvm.amdgcn.raw.buffer.load.format
 ; --------------------------------------------------------------------
 
 define amdgpu_ps float @raw_buffer_load_format_f32(<4 x i32> inreg %rsrc, i32 %ofs, i32 %sofs) #0 {
 ; CHECK-LABEL: @raw_buffer_load_format_f32(
 ; CHECK-NEXT:    [[DATA:%.*]] = call float @llvm.amdgcn.raw.buffer.load.format.f32(<4 x i32> [[RSRC:%.*]], i32 [[OFS:%.*]], i32 [[SOFS:%.*]], i32 0)
 ; CHECK-NEXT:    ret float [[DATA]]
 ;