[AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Summary

[AMDGPU] Implement hardware bug workaround for image instructions

This implements a workaround for a hardware bug in gfx8 and gfx9,
where register usage is not estimated correctly for image_store and
image_gather4 instructions when D16 is used.

Change-Id: I4e30744da6796acac53a9b5ad37ac1c2035c8899

[arsenm]

Also needs to update the GlobalISel path

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

5370–5371

IsD16 && (!hasUnpackedD16() || (Gather4 && hasGatherbug))?

6700

You can initialize this to the target size and then avoid push_back

6701

This is the same as just MVT::v2i16

6703

Formatting

[rdomingu]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

5370–5371

We want NumVDataDwords = DMaskLanes for Gather4 && IsD16 && hasGatherBug. That wouldn't be the case with your suggestion.

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3221–3222 ↗

(On Diff #276533)

Brace formatting

3226–3228 ↗

(On Diff #276533)

There's no obstacle to handling v3 here, it should work in the other cases

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3226–3228 ↗

(On Diff #276533)

Note that v3f16 is not handled in SIISelLowering.cpp either (line 7255). Are you suggesting we implement v3f16 there too?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3226–3228 ↗

(On Diff #276533)

There is a v3f16 definition now, so you could. I think it's more important for globalisel to be more complete

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3286 ↗

(On Diff #282042)

No else after return

3301 ↗

(On Diff #282042)

No else after return

3303 ↗

(On Diff #282042)

llvm_unreachable instead of assert(alse)

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp

317

I know clang-format really wants to pack these onto a single line, but it's a terrible idea and you shouldn't listen to it

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Ping.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3410 ↗

(On Diff #284443)

If you just resize to 8 below, why not use an array?

3411–3415 ↗

(On Diff #284443)

It would be preferable to emit a concat_vectors of <2 x s16> pieces here

3414 ↗

(On Diff #284443)

Resize here is weird. You can push back, or constructed PackedRegs to the desired size?

3420 ↗

(On Diff #284443)

Same as above

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

6701

++I

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3411–3415 ↗

(On Diff #284443)

Sorry, I'm new to this. Why would concat_vectors be preferable than build_vector? Could you please elaborate?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3411–3415 ↗

(On Diff #284443)

Because a G_BUILD_VECTOR with 16-bit sources isn't naturally legal. This works, it just adds more work for the legalizer to reprocess these when you could produce something that's legal to begin with to save compile time

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3411–3415 ↗

(On Diff #284443)

I see. But how would you go from v3f16 to concat_vectors of <2 x 16> to v4f32 (which is what we want at the end)?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3411–3415 ↗

(On Diff #284443)

I think I'm missing something. Why is this going from <3 x s16> to <4 x s32>? Isn't this the unpacked layout case? Why isn't this just an G_ANYEXT from <3 x s16> to <3 x s32>?

3414 ↗

(On Diff #284443)

You can do this in the initial construction, also the small size should just be 8? Or use std::array?

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3411–3415 ↗

(On Diff #284443)

This is the image workaround for the packed layout case. We don't want to change the data layout which is why we shouldn't use G_ANYEXT. We just want to make the compiler think the data is twice as big.

llvm/lib/Target/AMDGPU/AMDGPU.td

 >;
 
 def FeatureOffset3fBug : SubtargetFeature<"offset-3f-bug",
   "HasOffset3fBug",
   "true",
   "Branch offset of 3f hardware bug"
 >;
 
+def FeatureImageStoreD16Bug : SubtargetFeature<"image-store-d16-bug",
+  "HasImageStoreD16Bug",
+  "true",
+  "Image Store D16 hardware bug"
+>;
+
+def FeatureImageGather4D16Bug : SubtargetFeature<"image-gather4-d16-bug",
+  "HasImageGather4D16Bug",
+  "true",
+  "Image Gather4 D16 hardware bug"
+>;
+
 class SubtargetFeatureLDSBankCount <int Value> : SubtargetFeature <
   "ldsbankcount"#Value,
   "LDSBankCount",
   !cast<string>(Value),
   "The number of LDS banks per compute unit."
 >;
 
 def FeatureLDSBankCount16 : SubtargetFeatureLDSBankCount<16>;
def FeatureISAVersion8_0_3 : FeatureSet<
    FeatureUnpackedD16VMem,
    FeatureDoesNotSupportXNACK,
    FeatureCodeObjectV3]>;
 
 def FeatureISAVersion8_1_0 : FeatureSet<
   [FeatureVolcanicIslands,
    FeatureLDSBankCount16,
    FeatureXNACK,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageStoreD16Bug,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_0 : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureCodeObjectV3,
    FeatureDoesNotSupportXNACK,
-   FeatureDoesNotSupportSRAMECC]>;
+   FeatureDoesNotSupportSRAMECC,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_2 : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureXNACK,
    FeatureDoesNotSupportSRAMECC,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_4 : FeatureSet<
   [FeatureGFX9,
    FeatureLDSBankCount32,
    FeatureFmaMixInsts,
    FeatureDoesNotSupportXNACK,
    FeatureDoesNotSupportSRAMECC,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_6 : FeatureSet<
   [FeatureGFX9,
    HalfRate64Ops,
    FeatureFmaMixInsts,
    FeatureLDSBankCount32,
    FeatureDLInsts,
    FeatureDot1Insts,
    FeatureDot2Insts,
    FeatureDoesNotSupportXNACK,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_8 : FeatureSet<
   [FeatureGFX9,
    HalfRate64Ops,
    FeatureFmaMixInsts,
    FeatureLDSBankCount32,
    FeatureDLInsts,
    FeatureDot1Insts,
    FeatureDot2Insts,
    FeatureDot3Insts,
    FeatureDot4Insts,
    FeatureDot5Insts,
    FeatureDot6Insts,
    FeatureMAIInsts,
    FeaturePkFmacF16Inst,
    FeatureAtomicFaddInsts,
    FeatureSRAMECC,
    FeatureMFMAInlineLiteralBug,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_9 : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureXNACK,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 // TODO: Organize more features into groups.
 def FeatureGroup {
   // Bugs present on gfx10.1.
   list<SubtargetFeature> GFX10_1_Bugs = [
     FeatureVcmpxPermlaneHazard,
     FeatureVMEMtoScalarWriteHazard,
     FeatureSMEMtoVectorWriteHazard,

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h

protected:
   bool HasVMEMtoScalarWriteHazard;
   bool HasSMEMtoVectorWriteHazard;
   bool HasInstFwdPrefetchBug;
   bool HasVcmpxExecWARHazard;
   bool HasLdsBranchVmemWARHazard;
   bool HasNSAtoVMEMBug;
   bool HasOffset3fBug;
   bool HasFlatSegmentOffsetBug;
+  bool HasImageStoreD16Bug;
+  bool HasImageGather4D16Bug;
 
   // Dummy feature to use for assembler in tablegen.
   bool FeatureDisable;
 
   SelectionDAGTargetInfo TSInfo;
 private:
   SIInstrInfo InstrInfo;
   SITargetLowering TLInfo;
public:
   bool hasGFX10A16() const {
     return HasGFX10A16;
   }
 
   bool hasOffset3fBug() const {
     return HasOffset3fBug;
   }
 
+  bool hasImageStoreD16Bug() const {
+    return HasImageStoreD16Bug;
+  }
+
+  bool hasImageGather4D16Bug() const {
+    return HasImageGather4D16Bug;
+  }
+
   bool hasNSAEncoding() const {
     return HasNSAEncoding;
   }
 
   bool hasMadF16() const;
 
   bool enableSIScheduler() const {
     return EnableSIScheduler;

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp

GCNSubtarget::GCNSubtarget(const Triple &TT, StringRef GPU, StringRef FS,
     HasVMEMtoScalarWriteHazard(false),
     HasSMEMtoVectorWriteHazard(false),
     HasInstFwdPrefetchBug(false),
     HasVcmpxExecWARHazard(false),
     HasLdsBranchVmemWARHazard(false),
     HasNSAtoVMEMBug(false),
     HasOffset3fBug(false),
     HasFlatSegmentOffsetBug(false),
+    HasImageStoreD16Bug(false),
+    HasImageGather4D16Bug(false),
 
     FeatureDisable(false),
     InstrInfo(initializeSubtargetDependencies(TT, GPU, FS)),
     TLInfo(TM, *this),
     FrameLowering(TargetFrameLowering::StackGrowsUp, getStackAlignment(), 0) {
   MaxWavesPerEU = AMDGPU::IsaInfo::getMaxWavesPerEU(this);
   CallLoweringInfo.reset(new AMDGPUCallLowering(*getTargetLowering()));
   Legalizer.reset(new AMDGPULegalizerInfo(*this, TM));
unsigned GCNSubtarget::getConstantBusLimit(unsigned Opcode) const {
   case AMDGPU::V_LSHRREV_B64:
   case AMDGPU::V_LSHRREV_B64_gfx10:
   case AMDGPU::V_LSHR_B64:
   case AMDGPU::V_ASHRREV_I64:
   case AMDGPU::V_ASHRREV_I64_gfx10:
   case AMDGPU::V_ASHR_I64:
     return 1;
   }
 
   return 2;
 }
 
 unsigned AMDGPUSubtarget::getMaxLocalMemSizeWithWaveCount(unsigned NWaves,
   const Function &F) const {
   if (NWaves == 1)
     return getLocalMemorySize();
   unsigned WorkGroupSize = getFlatWorkGroupSizes(F).second;

llvm/lib/Target/AMDGPU/SIISelLowering.h

SDValue lowerIntrinsicLoad(MemSDNode *M, bool IsFormat, SelectionDAG &DAG,
                              ArrayRef<SDValue> Ops) const;
 
   // Call DAG.getMemIntrinsicNode for a load, but first widen a dwordx3 type to
   // dwordx4 if on SI.
   SDValue getMemIntrinsicNode(unsigned Opcode, const SDLoc &DL, SDVTList VTList,
                               ArrayRef<SDValue> Ops, EVT MemVT,
                               MachineMemOperand *MMO, SelectionDAG &DAG) const;
 
-  SDValue handleD16VData(SDValue VData, SelectionDAG &DAG) const;
+  SDValue handleD16VData(SDValue VData, SelectionDAG &DAG,
+                         bool Image = false) const;
 
   /// Converts \p Op, which must be of floating point type, to the
   /// floating point type \p VT, by either extending or truncating it.
   SDValue getFPExtOrFPTrunc(SelectionDAG &DAG,
                             SDValue Op,
                             const SDLoc &DL,
                             EVT VT) const;
 

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

MVT DataDwordVT = NumDataDwords == 1 ?
     MVT::i32 : MVT::getVectorVT(MVT::i32, NumDataDwords);
 
   MVT MaskPopVT = MaskPopDwords == 1 ?
     MVT::i32 : MVT::getVectorVT(MVT::i32, MaskPopDwords);
 
   SDValue Data(Result, 0);
   SDValue TexFail;
 
-  if (IsTexFail) {
+  if (DMaskPop > 0 && Data.getValueType() != MaskPopVT) {
     SDValue ZeroIdx = DAG.getConstant(0, DL, MVT::i32);
     if (MaskPopVT.isVector()) {
       Data = DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, MaskPopVT,
                          SDValue(Result, 0), ZeroIdx);
     } else {
       Data = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MaskPopVT,
                          SDValue(Result, 0), ZeroIdx);
     }
-
-    TexFail = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i32,
-                          SDValue(Result, 0),
-                          DAG.getConstant(MaskPopDwords, DL, MVT::i32));
   }
 
   if (DataDwordVT.isVector())
     Data = padEltsToUndef(DAG, DL, DataDwordVT, Data,
                           NumDataDwords - MaskPopDwords);
 
   if (IsD16)
     Data = adjustLoadValueTypeImpl(Data, ReqRetVT, DL, DAG, Unpacked);
 
   if (!ReqRetVT.isVector())
     Data = DAG.getNode(ISD::TRUNCATE, DL, ReqRetVT.changeTypeToInteger(), Data);
 
   Data = DAG.getNode(ISD::BITCAST, DL, ReqRetVT, Data);
 
-  if (TexFail)
+  if (IsTexFail) {
+    TexFail = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i32,
+                          SDValue(Result, 0),
+                          DAG.getConstant(MaskPopDwords, DL, MVT::i32));
+
     return DAG.getMergeValues({Data, TexFail, SDValue(Result, 1)}, DL);
+  }
 
   if (Result->getNumValues() == 1)
     return Data;
 
   return DAG.getMergeValues({Data, SDValue(Result, 1)}, DL);
 }
 
 static bool parseTexFail(SDValue TexFailCtrl, SelectionDAG &DAG, SDValue *TFE,
if (BaseOpcode->Store) {
       VData = Op.getOperand(2);
 
       MVT StoreVT = VData.getSimpleValueType();
       if (StoreVT.getScalarType() == MVT::f16) {
         if (!Subtarget->hasD16Images() || !BaseOpcode->HasD16)
           return Op; // D16 is unsupported for this instruction
 
         IsD16 = true;
-        VData = handleD16VData(VData, DAG);
+        VData = handleD16VData(VData, DAG, true);
       }
 
       NumVDataDwords = (VData.getValueType().getSizeInBits() + 31) / 32;
     } else {
       // Work out the num dwords based on the dmask popcount and underlying type
       // and whether packing is supported.
       MVT LoadVT = ResultTypes[0].getSimpleVT();
       if (LoadVT.getScalarType() == MVT::f16) {
         if (!Subtarget->hasD16Images() || !BaseOpcode->HasD16)
           return Op; // D16 is unsupported for this instruction
 
         IsD16 = true;
       }
 
       // Confirm that the return type is large enough for the dmask specified
       if ((LoadVT.isVector() && LoadVT.getVectorNumElements() < DMaskLanes) ||
           (!LoadVT.isVector() && DMaskLanes > 1))
           return Op;
 
-      if (IsD16 && !Subtarget->hasUnpackedD16VMem())
+      // The sq block of gfx8 and gfx9 do not estimate register use correctly
+      // for d16 image_gather4, image_gather4_l, and image_gather4_lz
+      // instructions.
+      if (IsD16 && !Subtarget->hasUnpackedD16VMem() &&
+          !(BaseOpcode->Gather4 && Subtarget->hasImageGather4D16Bug()))
         NumVDataDwords = (DMaskLanes + 1) / 2;
       else
         NumVDataDwords = DMaskLanes;
 
       AdjustRetType = true;
     }
 
     AddrIdx = DMaskIdx + 1;
if (WidenedVT != VT) {
     auto Extract = DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, VT, NewOp,
                                DAG.getVectorIdxConstant(0, DL));
     NewOp = DAG.getMergeValues({ Extract, SDValue(NewOp.getNode(), 1) }, DL);
   }
   return NewOp;
 }
 
 SDValue SITargetLowering::handleD16VData(SDValue VData,
-                                         SelectionDAG &DAG) const {
+                                         SelectionDAG &DAG,
+                                         bool Image) const {
   EVT StoreVT = VData.getValueType();
 
   // No change for f16 and legal vector D16 types.
   if (!StoreVT.isVector())
     return VData;
 
   SDLoc DL(VData);
   assert((StoreVT.getVectorNumElements() != 3) && "Handle v3f16");
 
   if (Subtarget->hasUnpackedD16VMem()) {
     // We need to unpack the packed data to store.
     EVT IntStoreVT = StoreVT.changeTypeToInteger();
     SDValue IntVData = DAG.getNode(ISD::BITCAST, DL, IntStoreVT, VData);
 
     EVT EquivStoreVT = EVT::getVectorVT(*DAG.getContext(), MVT::i32,
                                         StoreVT.getVectorNumElements());
     SDValue ZExt = DAG.getNode(ISD::ZERO_EXTEND, DL, EquivStoreVT, IntVData);
     return DAG.UnrollVectorOp(ZExt.getNode());
   }
 
+  // The sq block of gfx8.1 does not estimate register use correctly for d16
+  // image store instructions. The data operand is computed as if it were not a
+  // d16 image instruction.
+  if (Image && Subtarget->hasImageStoreD16Bug()) {
+    // Bitcast to i16
+    EVT IntStoreVT = StoreVT.changeTypeToInteger();
+    SDValue IntVData = DAG.getNode(ISD::BITCAST, DL, IntStoreVT, VData);
+
+    // Decompose into scalars
+    SmallVector<SDValue, 4> Elts;
+    DAG.ExtractVectorElements(IntVData, Elts);
+
+    // Group pairs of i16 into v2i16 and bitcast to i32
+    SmallVector<SDValue, 4> PackedElts;
+    EVT Vec2VT = EVT::getVectorVT(*DAG.getContext(), MVT::i16, 2);
+    for (unsigned i = 0 ; i < Elts.size() / 2 ; i += 1)
+    {
+      SDValue Pair = DAG.getBuildVector(Vec2VT, DL, {Elts[i * 2],
+                                        Elts[i * 2 +  1]});
+      SDValue IntPair = DAG.getNode(ISD::BITCAST, DL, MVT::i32, Pair);
+      PackedElts.push_back(IntPair);
+    }
+
+    // Pad using UNDEF
+    PackedElts.resize(PackedElts.size() * 2, DAG.getUNDEF(MVT::i32));
+
+    // Build final vector
+    EVT VecVT = EVT::getVectorVT(*DAG.getContext(), MVT::i32,
+                                 PackedElts.size());
+    return DAG.getBuildVector(VecVT, DL, PackedElts);
+  }
+
   assert(isTypeLegal(StoreVT));
   return VData;
 }
 
 SDValue SITargetLowering::LowerINTRINSIC_VOID(SDValue Op,
                                               SelectionDAG &DAG) const {
   SDLoc DL(Op);
   SDValue Chain = Op.getOperand(0);

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.d16.dim.ll

 ; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck -check-prefixes=GCN,UNPACKED,GFX89 %s
-; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX81,GFX89 %s
+; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX81,GFX89 %s
 ; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX9,GFX89 %s
 ; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX10 %s
 
 ; GCN-LABEL: {{^}}image_load_f16:
 ; GFX89: image_load v0, v[0:1], s[0:7] dmask:0x1 unorm d16{{$}}
 ; GFX10: image_load v0, v[0:1], s[0:7] dmask:0x1 dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps half @image_load_f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t) {
 main_body:
main_body:
   ret void
 }
 
 ; GCN-LABEL: {{^}}image_store_v2f16
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: image_store v[{{[0-9:]+}}], v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
 ; PACKED: image_store v2, v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
+; GFX81: image_store v[2:3], v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
 ; GFX10: image_store v2, v[0:1], s[0:7] dmask:0x3 dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps void @image_store_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, float %in) {
 main_body:
   %data = bitcast float %in to <2 x half>
   call void @llvm.amdgcn.image.store.2d.v2f16.i32(<2 x half> %data, i32 3, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 ; GCN-LABEL: {{^}}image_store_v4f16
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: image_store v[{{[0-9:]+}}], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; PACKED: image_store v[2:3], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
+; GFX81: image_store v[2:5], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; GFX10: image_store v[2:3], v[0:1], s[0:7] dmask:0xf dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps void @image_store_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <2 x float> %in) {
 main_body:
   %data = bitcast <2 x float> %in to <4 x half>
   call void @llvm.amdgcn.image.store.2d.v4f16.i32(<4 x half> %data, i32 15, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.d16.dim.ll

 ; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck -check-prefixes=GCN,UNPACKED,GFX89 %s
-; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX81,GFX89 %s
-; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX9,GFX89 %s
+; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX81,GFX89 %s
+; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX9,GFX89 %s
 ; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX10 %s
 
 ; GCN-LABEL: {{^}}image_gather4_b_2d_v4f16:
 ; UNPACKED: image_gather4_b v[0:3], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
 ; PACKED: image_gather4_b v[0:1], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
+; GFX810: image_gather4_b v[0:3], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
+; GFX9: image_gather4_b v[0:3], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
 ; GFX10: image_gather4_b v[0:1], v[0:2], s[0:7], s[8:11] dmask:0x4 dim:SQ_RSRC_IMG_2D d16{{$}}
 define amdgpu_ps <2 x float> @image_gather4_b_2d_v4f16(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %s, float %t) {
 main_body:
   %tex = call <4 x half> @llvm.amdgcn.image.gather4.b.2d.v4f16.f32.f32(i32 4, float %bias, float %s, float %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 false, i32 0, i32 0)
   %r = bitcast <4 x half> %tex to <2 x float>
   ret <2 x float> %r
 }
 
 declare <4 x half> @llvm.amdgcn.image.gather4.b.2d.v4f16.f32.f32(i32, float, float, float, <8 x i32>, <4 x i32>, i1, i32, i32) #1
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readonly }
 attributes #2 = { nounwind readnone }