[AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Summary

[AMDGPU] Implement hardware bug workaround for image instructions

This implements a workaround for a hardware bug in gfx8 and gfx9,
where register usage is not estimated correctly for image_store and
image_gather4 instructions when D16 is used.

Change-Id: I4e30744da6796acac53a9b5ad37ac1c2035c8899

[arsenm]

Also needs to update the GlobalISel path

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

5890–5891

IsD16 && (!hasUnpackedD16() || (Gather4 && hasGatherbug))?

7280

You can initialize this to the target size and then avoid push_back

7281

This is the same as just MVT::v2i16

7283

Formatting

[rdomingu]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

5890–5891

We want NumVDataDwords = DMaskLanes for Gather4 && IsD16 && hasGatherBug. That wouldn't be the case with your suggestion.

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3221–3222

Brace formatting

3226–3228

There's no obstacle to handling v3 here, it should work in the other cases

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3226–3228

Note that v3f16 is not handled in SIISelLowering.cpp either (line 7255). Are you suggesting we implement v3f16 there too?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3226–3228

There is a v3f16 definition now, so you could. I think it's more important for globalisel to be more complete

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3228

No else after return

3243

No else after return

3245

llvm_unreachable instead of assert(alse)

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp

197

I know clang-format really wants to pack these onto a single line, but it's a terrible idea and you shouldn't listen to it

[rdomingu]

Updating D81172: [AMDGPU] Implement hardware bug workaround for image instructions

[rdomingu]

Ping.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3217

Same as above

3231

If you just resize to 8 below, why not use an array?

3232–3236

It would be preferable to emit a concat_vectors of <2 x s16> pieces here

3235

Resize here is weird. You can push back, or constructed PackedRegs to the desired size?

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

7281

++I

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3232–3236

Sorry, I'm new to this. Why would concat_vectors be preferable than build_vector? Could you please elaborate?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3232–3236

Because a G_BUILD_VECTOR with 16-bit sources isn't naturally legal. This works, it just adds more work for the legalizer to reprocess these when you could produce something that's legal to begin with to save compile time

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3232–3236

I see. But how would you go from v3f16 to concat_vectors of <2 x 16> to v4f32 (which is what we want at the end)?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3232–3236

I think I'm missing something. Why is this going from <3 x s16> to <4 x s32>? Isn't this the unpacked layout case? Why isn't this just an G_ANYEXT from <3 x s16> to <3 x s32>?

3235

You can do this in the initial construction, also the small size should just be 8? Or use std::array?

[rdomingu]

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

3232–3236

This is the image workaround for the packed layout case. We don't want to change the data layout which is why we shouldn't use G_ANYEXT. We just want to make the compiler think the data is twice as big.

llvm/lib/Target/AMDGPU/AMDGPU.td

 >;
 
 def FeatureOffset3fBug : SubtargetFeature<"offset-3f-bug",
   "HasOffset3fBug",
   "true",
   "Branch offset of 3f hardware bug"
 >;
 
+def FeatureImageStoreD16Bug : SubtargetFeature<"image-store-d16-bug",
+  "HasImageStoreD16Bug",
+  "true",
+  "Image Store D16 hardware bug"
+>;
+
+def FeatureImageGather4D16Bug : SubtargetFeature<"image-gather4-d16-bug",
+  "HasImageGather4D16Bug",
+  "true",
+  "Image Gather4 D16 hardware bug"
+>;
+
 class SubtargetFeatureLDSBankCount <int Value> : SubtargetFeature <
   "ldsbankcount"#Value,
   "LDSBankCount",
   !cast<string>(Value),
   "The number of LDS banks per compute unit."
 >;
 
 def FeatureLDSBankCount16 : SubtargetFeatureLDSBankCount<16>;
def FeatureISAVersion8_0_3 : FeatureSet<
    FeatureUnpackedD16VMem,
    FeatureDoesNotSupportXNACK,
    FeatureCodeObjectV3]>;
 
 def FeatureISAVersion8_1_0 : FeatureSet<
   [FeatureVolcanicIslands,
    FeatureLDSBankCount16,
    FeatureXNACK,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageStoreD16Bug,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_0 : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureCodeObjectV3,
    FeatureDoesNotSupportXNACK,
-   FeatureDoesNotSupportSRAMECC]>;
+   FeatureDoesNotSupportSRAMECC,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_2 : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureXNACK,
    FeatureDoesNotSupportSRAMECC,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_4 : FeatureSet<
   [FeatureGFX9,
    FeatureLDSBankCount32,
    FeatureFmaMixInsts,
    FeatureDoesNotSupportXNACK,
    FeatureDoesNotSupportSRAMECC,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_6 : FeatureSet<
   [FeatureGFX9,
    HalfRate64Ops,
    FeatureFmaMixInsts,
    FeatureLDSBankCount32,
    FeatureDLInsts,
    FeatureDot1Insts,
    FeatureDot2Insts,
    FeatureDoesNotSupportXNACK,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_8 : FeatureSet<
   [FeatureGFX9,
    HalfRate64Ops,
    FeatureFmaMixInsts,
    FeatureLDSBankCount32,
    FeatureDLInsts,
    FeatureDot1Insts,
    FeatureDot2Insts,
    FeatureDot3Insts,
    FeatureDot4Insts,
    FeatureDot5Insts,
    FeatureDot6Insts,
    FeatureMAIInsts,
    FeaturePkFmacF16Inst,
    FeatureAtomicFaddInsts,
    FeatureSRAMECC,
    FeatureMFMAInlineLiteralBug,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 def FeatureISAVersion9_0_9 : FeatureSet<
   [FeatureGFX9,
    FeatureMadMixInsts,
    FeatureLDSBankCount32,
    FeatureXNACK,
-   FeatureCodeObjectV3]>;
+   FeatureCodeObjectV3,
+   FeatureImageGather4D16Bug]>;
 
 // TODO: Organize more features into groups.
 def FeatureGroup {
   // Bugs present on gfx10.1.
   list<SubtargetFeature> GFX10_1_Bugs = [
     FeatureVcmpxPermlaneHazard,
     FeatureVMEMtoScalarWriteHazard,
     FeatureSMEMtoVectorWriteHazard,

llvm/lib/Target/AMDGPU/AMDGPUInstructionSelector.cpp

if (BaseOpcode->AtomicX2) {
       NumVDataDwords = Is64Bit ? 2 : 1;
     }
   } else {
     const int DMaskIdx = 2; // Input/output + intrinsic ID.
 
     DMask = MI.getOperand(DMaskIdx).getImm();
     DMaskLanes = BaseOpcode->Gather4 ? 4 : countPopulation(DMask);
 
-    if (BaseOpcode->Store) {
-      VDataIn = MI.getOperand(1).getReg();
-      VDataTy = MRI->getType(VDataIn);
-      NumVDataDwords = (VDataTy.getSizeInBits() + 31) / 32;
-    } else {
-      VDataOut = MI.getOperand(0).getReg();
-      VDataTy = MRI->getType(VDataOut);
-      NumVDataDwords = DMaskLanes;
-
-      // One memoperand is mandatory, except for getresinfo.
-      // FIXME: Check this in verifier.
-      if (!MI.memoperands_empty()) {
-        const MachineMemOperand *MMO = *MI.memoperands_begin();
 
-        // Infer d16 from the memory size, as the register type will be mangled by
-        // unpacked subtargets, or by TFE.
-        IsD16 = ((8 * MMO->getSize()) / DMaskLanes) < 32;
+    }
+
+    if (BaseOpcode->Store) {
+      VDataIn = MI.getOperand(1).getReg();
+      VDataTy = MRI->getType(VDataIn);
+      NumVDataDwords = (VDataTy.getSizeInBits() + 31) / 32;
+    } else {
+      VDataOut = MI.getOperand(0).getReg();
+      VDataTy = MRI->getType(VDataOut);
+      NumVDataDwords = DMaskLanes;
 
-        if (IsD16 && !STI.hasUnpackedD16VMem())
-          NumVDataDwords = (DMaskLanes + 1) / 2;
-      }
-    }
-  }
 
   // Optimize _L to _LZ when _L is zero
   if (LZMappingInfo) {
     // The legalizer replaced the register with an immediate 0 if we need to
     // change the opcode.
     const MachineOperand &Lod = MI.getOperand(VAddrIdx + NumVAddr - 1);
     if (Lod.isImm()) {
       assert(Lod.getImm() == 0);

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.h

bool legalizeImplicitArgPtr(MachineInstr &MI, MachineRegisterInfo &MRI,
                               MachineIRBuilder &B) const;
   bool legalizeIsAddrSpace(MachineInstr &MI, MachineRegisterInfo &MRI,
                            MachineIRBuilder &B, unsigned AddrSpace) const;
 
   std::tuple<Register, unsigned, unsigned>
   splitBufferOffsets(MachineIRBuilder &B, Register OrigOffset) const;
 
   Register handleD16VData(MachineIRBuilder &B, MachineRegisterInfo &MRI,
-                          Register Reg) const;
+                          Register Reg, bool ImageStore = false) const;
   bool legalizeRawBufferStore(MachineInstr &MI, MachineRegisterInfo &MRI,
                               MachineIRBuilder &B, bool IsFormat) const;
   bool legalizeRawBufferLoad(MachineInstr &MI, MachineRegisterInfo &MRI,
                              MachineIRBuilder &B, bool IsFormat) const;
   Register fixStoreSourceType(MachineIRBuilder &B, Register VData,
                               bool IsFormat) const;
 
   bool legalizeBufferStore(MachineInstr &MI, MachineRegisterInfo &MRI,

llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp

if (!BaseReg)
     BaseReg = B.buildConstant(S32, 0).getReg(0);
 
   return std::make_tuple(BaseReg, ImmOffset, TotalConstOffset);
 }
 
 /// Handle register layout difference for f16 images for some subtargets.
 Register AMDGPULegalizerInfo::handleD16VData(MachineIRBuilder &B,
                                              MachineRegisterInfo &MRI,
-                                             Register Reg) const {
-  if (!ST.hasUnpackedD16VMem())
-    return Reg;
-
+                                             Register Reg,
+                                             bool ImageStore) const {
   const LLT S16 = LLT::scalar(16);
   const LLT S32 = LLT::scalar(32);
   LLT StoreVT = MRI.getType(Reg);
   assert(StoreVT.isVector() && StoreVT.getElementType() == S16);
 
+  if (ST.hasUnpackedD16VMem()) {
     auto Unmerge = B.buildUnmerge(S16, Reg);
 
     SmallVector<Register, 4> WideRegs;
     for (int I = 0, E = Unmerge->getNumOperands() - 1; I != E; ++I)
       WideRegs.push_back(B.buildAnyExt(S32, Unmerge.getReg(I)).getReg(0));
 
     int NumElts = StoreVT.getNumElements();
 
     return B.buildBuildVector(LLT::vector(NumElts, S32), WideRegs).getReg(0);
   }
 
+  if (ImageStore && ST.hasImageStoreD16Bug()) {
+    if (StoreVT.getNumElements() == 2) {
+      SmallVector<Register, 4> PackedRegs;
+      Reg = B.buildBitcast(S32, Reg).getReg(0);
+      PackedRegs.push_back(Reg);
+      PackedRegs.resize(2, B.buildUndef(S32).getReg(0));
+      return B.buildBuildVector(LLT::vector(2, S32), PackedRegs).getReg(0);
+    } else if (StoreVT.getNumElements() == 3) {
+      SmallVector<Register, 4> PackedRegs;
+      auto Unmerge = B.buildUnmerge(S16, Reg);
+      for (int I = 0, E = Unmerge->getNumOperands() - 1; I != E; ++I)
+        PackedRegs.push_back(Unmerge.getReg(I));
+      PackedRegs.resize(8, B.buildUndef(S16).getReg(0));
+      Reg = B.buildBuildVector(LLT::vector(8, S16), PackedRegs).getReg(0);
+      return B.buildBitcast(LLT::vector(4, S32), Reg).getReg(0);
+    } else if (StoreVT.getNumElements() == 4) {
+      SmallVector<Register, 4> PackedRegs;
+      Reg = B.buildBitcast(LLT::vector(2, S32), Reg).getReg(0);
+      auto Unmerge = B.buildUnmerge(S32, Reg);
+      for (int I = 0, E = Unmerge->getNumOperands() - 1; I != E; ++I)
+        PackedRegs.push_back(Unmerge.getReg(I));
+      PackedRegs.resize(4, B.buildUndef(S32).getReg(0));
+      return B.buildBuildVector(LLT::vector(4, S32), PackedRegs).getReg(0);
+    } else {
+      assert(false && "Data type not supported");
+    }
+  }
+
+  return Reg;
+}
+
 Register AMDGPULegalizerInfo::fixStoreSourceType(
   MachineIRBuilder &B, Register VData, bool IsFormat) const {
   MachineRegisterInfo *MRI = B.getMRI();
   LLT Ty = MRI->getType(VData);
 
   const LLT S16 = LLT::scalar(16);
 
   // Fixup illegal register types for i8 stores.
bool AMDGPULegalizerInfo::legalizeImageIntrinsic(
 
   if (BaseOpcode->Store) { // No TFE for stores?
     // TODO: Handle dmask trim
     Register VData = MI.getOperand(1).getReg();
     LLT Ty = MRI->getType(VData);
     if (!Ty.isVector() || Ty.getElementType() != S16)
       return true;
 
-    Register RepackedReg = handleD16VData(B, *MRI, VData);
+    Register RepackedReg = handleD16VData(B, *MRI, VData, true);
     if (RepackedReg != VData) {
       MI.getOperand(1).setReg(RepackedReg);
     }
 
     return true;
   }
 
   Register DstReg = MI.getOperand(0).getReg();

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.h

protected:
   bool HasVMEMtoScalarWriteHazard;
   bool HasSMEMtoVectorWriteHazard;
   bool HasInstFwdPrefetchBug;
   bool HasVcmpxExecWARHazard;
   bool HasLdsBranchVmemWARHazard;
   bool HasNSAtoVMEMBug;
   bool HasOffset3fBug;
   bool HasFlatSegmentOffsetBug;
+  bool HasImageStoreD16Bug;
+  bool HasImageGather4D16Bug;
 
   // Dummy feature to use for assembler in tablegen.
   bool FeatureDisable;
 
   SelectionDAGTargetInfo TSInfo;
 private:
   SIInstrInfo InstrInfo;
   SITargetLowering TLInfo;
public:
   bool hasA16() const { return hasR128A16() || hasGFX10A16(); }
 
   bool hasG16() const { return HasG16; }
 
   bool hasOffset3fBug() const {
     return HasOffset3fBug;
   }
 
-  bool hasNSAEncoding() const {
-    return HasNSAEncoding;
-  }
+  bool hasImageStoreD16Bug() const { return HasImageStoreD16Bug; }
+
+  bool hasImageGather4D16Bug() const { return HasImageGather4D16Bug; }
+
+  bool hasNSAEncoding() const { return HasNSAEncoding; }
 
   bool hasGFX10_BEncoding() const {
     return GFX10_BEncoding;
   }
 
   bool hasGFX10_3Insts() const {
     return GFX10_3Insts;
   }

llvm/lib/Target/AMDGPU/AMDGPUSubtarget.cpp

AMDGPUSubtarget::AMDGPUSubtarget(const Triple &TT) :
   EnablePromoteAlloca(false),
   HasTrigReducedRange(false),
   MaxWavesPerEU(10),
   LocalMemorySize(0),
   WavefrontSizeLog2(0)
   { }
 
 GCNSubtarget::GCNSubtarget(const Triple &TT, StringRef GPU, StringRef FS,
-                           const GCNTargetMachine &TM) :
-    AMDGPUGenSubtargetInfo(TT, GPU, FS),
-    AMDGPUSubtarget(TT),
+                           const GCNTargetMachine &TM)
+    : AMDGPUGenSubtargetInfo(TT, GPU, FS), AMDGPUSubtarget(TT),
       TargetTriple(TT),
       Gen(TT.getOS() == Triple::AMDHSA ? SEA_ISLANDS : SOUTHERN_ISLANDS),
-    InstrItins(getInstrItineraryForCPU(GPU)),
-    LDSBankCount(0),
+      InstrItins(getInstrItineraryForCPU(GPU)), LDSBankCount(0),
       MaxPrivateElementSize(0),
 
-    FastFMAF32(false),
-    FastDenormalF32(false),
-    HalfRate64Ops(false),
-
-    FlatForGlobal(false),
-    AutoWaitcntBeforeBarrier(false),
-    CodeObjectV3(false),
-    UnalignedScratchAccess(false),
+      FastFMAF32(false), FastDenormalF32(false), HalfRate64Ops(false),
+
+      FlatForGlobal(false), AutoWaitcntBeforeBarrier(false),
+      CodeObjectV3(false), UnalignedScratchAccess(false),
       UnalignedBufferAccess(false),
 
-    HasApertureRegs(false),
-    EnableXNACK(false),
-    DoesNotSupportXNACK(false),
-    EnableCuMode(false),
-    TrapHandler(false),
-
-    EnableLoadStoreOpt(false),
-    EnableUnsafeDSOffsetFolding(false),
-    EnableSIScheduler(false),
-    EnableDS128(false),
-    EnablePRTStrictNull(false),
+      HasApertureRegs(false), EnableXNACK(false), DoesNotSupportXNACK(false),
+      EnableCuMode(false), TrapHandler(false),
+
+      EnableLoadStoreOpt(false), EnableUnsafeDSOffsetFolding(false),
+      EnableSIScheduler(false), EnableDS128(false), EnablePRTStrictNull(false),
       DumpCode(false),
 
-    FP64(false),
-    GCN3Encoding(false),
-    CIInsts(false),
-    GFX8Insts(false),
-    GFX9Insts(false),
-    GFX10Insts(false),
-    GFX10_3Insts(false),
-    GFX7GFX8GFX9Insts(false),
-    SGPRInitBug(false),
-    HasSMemRealTime(false),
-    HasIntClamp(false),
-    HasFmaMixInsts(false),
-    HasMovrel(false),
-    HasVGPRIndexMode(false),
-    HasScalarStores(false),
-    HasScalarAtomics(false),
-    HasSDWAOmod(false),
-    HasSDWAScalar(false),
-    HasSDWASdst(false),
-    HasSDWAMac(false),
-    HasSDWAOutModsVOPC(false),
-    HasDPP(false),
-    HasDPP8(false),
-    HasR128A16(false),
-    HasGFX10A16(false),
-    HasG16(false),
-    HasNSAEncoding(false),
-    GFX10_BEncoding(false),
-    HasDLInsts(false),
-    HasDot1Insts(false),
-    HasDot2Insts(false),
-    HasDot3Insts(false),
-    HasDot4Insts(false),
-    HasDot5Insts(false),
-    HasDot6Insts(false),
-    HasMAIInsts(false),
-    HasPkFmacF16Inst(false),
-    HasAtomicFaddInsts(false),
-    EnableSRAMECC(false),
-    DoesNotSupportSRAMECC(false),
-    HasNoSdstCMPX(false),
-    HasVscnt(false),
-    HasGetWaveIdInst(false),
-    HasSMemTimeInst(false),
-    HasRegisterBanking(false),
-    HasVOP3Literal(false),
-    HasNoDataDepHazard(false),
-    FlatAddressSpace(false),
-    FlatInstOffsets(false),
-    FlatGlobalInsts(false),
-    FlatScratchInsts(false),
-    ScalarFlatScratchInsts(false),
-    AddNoCarryInsts(false),
-    HasUnpackedD16VMem(false),
-    LDSMisalignedBug(false),
+      FP64(false), GCN3Encoding(false), CIInsts(false), GFX8Insts(false),
+      GFX9Insts(false), GFX10Insts(false), GFX10_3Insts(false),
+      GFX7GFX8GFX9Insts(false), SGPRInitBug(false), HasSMemRealTime(false),
+      HasIntClamp(false), HasFmaMixInsts(false), HasMovrel(false),
+      HasVGPRIndexMode(false), HasScalarStores(false), HasScalarAtomics(false),
+      HasSDWAOmod(false), HasSDWAScalar(false), HasSDWASdst(false),
+      HasSDWAMac(false), HasSDWAOutModsVOPC(false), HasDPP(false),
+      HasDPP8(false), HasR128A16(false), HasGFX10A16(false), HasG16(false),
+      HasNSAEncoding(false), GFX10_BEncoding(false), HasDLInsts(false),
+      HasDot1Insts(false), HasDot2Insts(false), HasDot3Insts(false),
+      HasDot4Insts(false), HasDot5Insts(false), HasDot6Insts(false),
+      HasMAIInsts(false), HasPkFmacF16Inst(false), HasAtomicFaddInsts(false),
+      EnableSRAMECC(false), DoesNotSupportSRAMECC(false), HasNoSdstCMPX(false),
+      HasVscnt(false), HasGetWaveIdInst(false), HasSMemTimeInst(false),
+      HasRegisterBanking(false), HasVOP3Literal(false),
+      HasNoDataDepHazard(false), FlatAddressSpace(false),
+      FlatInstOffsets(false), FlatGlobalInsts(false), FlatScratchInsts(false),
+      ScalarFlatScratchInsts(false), AddNoCarryInsts(false),
+      HasUnpackedD16VMem(false), LDSMisalignedBug(false),
       HasMFMAInlineLiteralBug(false),
 
       ScalarizeGlobal(false),
 
-    HasVcmpxPermlaneHazard(false),
-    HasVMEMtoScalarWriteHazard(false),
-    HasSMEMtoVectorWriteHazard(false),
-    HasInstFwdPrefetchBug(false),
-    HasVcmpxExecWARHazard(false),
-    HasLdsBranchVmemWARHazard(false),
-    HasNSAtoVMEMBug(false),
-    HasOffset3fBug(false),
-    HasFlatSegmentOffsetBug(false),
+      HasVcmpxPermlaneHazard(false), HasVMEMtoScalarWriteHazard(false),
+      HasSMEMtoVectorWriteHazard(false), HasInstFwdPrefetchBug(false),
+      HasVcmpxExecWARHazard(false), HasLdsBranchVmemWARHazard(false),
+      HasNSAtoVMEMBug(false), HasOffset3fBug(false),
+      HasFlatSegmentOffsetBug(false), HasImageStoreD16Bug(false),
+      HasImageGather4D16Bug(false),
 
       FeatureDisable(false),
       InstrInfo(initializeSubtargetDependencies(TT, GPU, FS)),
       TLInfo(TM, *this),
       FrameLowering(TargetFrameLowering::StackGrowsUp, getStackAlignment(), 0) {
   MaxWavesPerEU = AMDGPU::IsaInfo::getMaxWavesPerEU(this);
   CallLoweringInfo.reset(new AMDGPUCallLowering(*getTargetLowering()));
   InlineAsmLoweringInfo.reset(new InlineAsmLowering(getTargetLowering()));
   Legalizer.reset(new AMDGPULegalizerInfo(*this, TM));
   RegBankInfo.reset(new AMDGPURegisterBankInfo(*this));
   InstSelector.reset(new AMDGPUInstructionSelector(
   *this, *static_cast<AMDGPURegisterBankInfo *>(RegBankInfo.get()), TM));
 }

llvm/lib/Target/AMDGPU/SIISelLowering.h

SDValue lowerIntrinsicLoad(MemSDNode *M, bool IsFormat, SelectionDAG &DAG,
                              ArrayRef<SDValue> Ops) const;
 
   // Call DAG.getMemIntrinsicNode for a load, but first widen a dwordx3 type to
   // dwordx4 if on SI.
   SDValue getMemIntrinsicNode(unsigned Opcode, const SDLoc &DL, SDVTList VTList,
                               ArrayRef<SDValue> Ops, EVT MemVT,
                               MachineMemOperand *MMO, SelectionDAG &DAG) const;
 
-  SDValue handleD16VData(SDValue VData, SelectionDAG &DAG) const;
+  SDValue handleD16VData(SDValue VData, SelectionDAG &DAG,
+                         bool ImageStore = false) const;
 
   /// Converts \p Op, which must be of floating point type, to the
   /// floating point type \p VT, by either extending or truncating it.
   SDValue getFPExtOrFPRound(SelectionDAG &DAG,
                             SDValue Op,
                             const SDLoc &DL,
                             EVT VT) const;
 

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

MVT DataDwordVT = NumDataDwords == 1 ?
     MVT::i32 : MVT::getVectorVT(MVT::i32, NumDataDwords);
 
   MVT MaskPopVT = MaskPopDwords == 1 ?
     MVT::i32 : MVT::getVectorVT(MVT::i32, MaskPopDwords);
 
   SDValue Data(Result, 0);
   SDValue TexFail;
 
-  if (IsTexFail) {
+  if (DMaskPop > 0 && Data.getValueType() != MaskPopVT) {
     SDValue ZeroIdx = DAG.getConstant(0, DL, MVT::i32);
     if (MaskPopVT.isVector()) {
       Data = DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, MaskPopVT,
                          SDValue(Result, 0), ZeroIdx);
     } else {
       Data = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MaskPopVT,
                          SDValue(Result, 0), ZeroIdx);
     }
-
-    TexFail = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i32,
-                          SDValue(Result, 0),
-                          DAG.getConstant(MaskPopDwords, DL, MVT::i32));
   }
 
   if (DataDwordVT.isVector())
     Data = padEltsToUndef(DAG, DL, DataDwordVT, Data,
                           NumDataDwords - MaskPopDwords);
 
   if (IsD16)
     Data = adjustLoadValueTypeImpl(Data, ReqRetVT, DL, DAG, Unpacked);
 
   if (!ReqRetVT.isVector())
     Data = DAG.getNode(ISD::TRUNCATE, DL, ReqRetVT.changeTypeToInteger(), Data);
 
   Data = DAG.getNode(ISD::BITCAST, DL, ReqRetVT, Data);
 
-  if (TexFail)
+  if (IsTexFail) {
+    TexFail =
+        DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i32, SDValue(Result, 0),
+                    DAG.getConstant(MaskPopDwords, DL, MVT::i32));
+
     return DAG.getMergeValues({Data, TexFail, SDValue(Result, 1)}, DL);
+  }
 
   if (Result->getNumValues() == 1)
     return Data;
 
   return DAG.getMergeValues({Data, SDValue(Result, 1)}, DL);
 }
 
 static bool parseTexFail(SDValue TexFailCtrl, SelectionDAG &DAG, SDValue *TFE,
if (BaseOpcode->Store) {
       VData = Op.getOperand(2);
 
       MVT StoreVT = VData.getSimpleValueType();
       if (StoreVT.getScalarType() == MVT::f16) {
         if (!Subtarget->hasD16Images() || !BaseOpcode->HasD16)
           return Op; // D16 is unsupported for this instruction
 
         IsD16 = true;
-        VData = handleD16VData(VData, DAG);
+        VData = handleD16VData(VData, DAG, true);
       }
 
       NumVDataDwords = (VData.getValueType().getSizeInBits() + 31) / 32;
     } else {
       // Work out the num dwords based on the dmask popcount and underlying type
       // and whether packing is supported.
       MVT LoadVT = ResultTypes[0].getSimpleVT();
       if (LoadVT.getScalarType() == MVT::f16) {
         if (!Subtarget->hasD16Images() || !BaseOpcode->HasD16)
           return Op; // D16 is unsupported for this instruction
 
         IsD16 = true;
       }
 
       // Confirm that the return type is large enough for the dmask specified
       if ((LoadVT.isVector() && LoadVT.getVectorNumElements() < DMaskLanes) ||
           (!LoadVT.isVector() && DMaskLanes > 1))
           return Op;
 
-      if (IsD16 && !Subtarget->hasUnpackedD16VMem())
+      // The sq block of gfx8 and gfx9 do not estimate register use correctly
+      // for d16 image_gather4, image_gather4_l, and image_gather4_lz
+      // instructions.
+      if (IsD16 && !Subtarget->hasUnpackedD16VMem() &&
+          !(BaseOpcode->Gather4 && Subtarget->hasImageGather4D16Bug()))
         NumVDataDwords = (DMaskLanes + 1) / 2;
       else
         NumVDataDwords = DMaskLanes;
 
       AdjustRetType = true;
     }
 
     AddrIdx = DMaskIdx + 1;
SDValue SITargetLowering::getMemIntrinsicNode(unsigned Opcode, const SDLoc &DL,
   if (WidenedVT != VT) {
     auto Extract = DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, VT, NewOp,
                                DAG.getVectorIdxConstant(0, DL));
     NewOp = DAG.getMergeValues({ Extract, SDValue(NewOp.getNode(), 1) }, DL);
   }
   return NewOp;
 }
 
-SDValue SITargetLowering::handleD16VData(SDValue VData,
-                                         SelectionDAG &DAG) const {
+SDValue SITargetLowering::handleD16VData(SDValue VData, SelectionDAG &DAG,
+                                         bool ImageStore) const {
   EVT StoreVT = VData.getValueType();
 
   // No change for f16 and legal vector D16 types.
   if (!StoreVT.isVector())
     return VData;
 
   SDLoc DL(VData);
   assert((StoreVT.getVectorNumElements() != 3) && "Handle v3f16");
 
   if (Subtarget->hasUnpackedD16VMem()) {
     // We need to unpack the packed data to store.
     EVT IntStoreVT = StoreVT.changeTypeToInteger();
     SDValue IntVData = DAG.getNode(ISD::BITCAST, DL, IntStoreVT, VData);
 
     EVT EquivStoreVT = EVT::getVectorVT(*DAG.getContext(), MVT::i32,
                                         StoreVT.getVectorNumElements());
     SDValue ZExt = DAG.getNode(ISD::ZERO_EXTEND, DL, EquivStoreVT, IntVData);
     return DAG.UnrollVectorOp(ZExt.getNode());
   }
 
+  // The sq block of gfx8.1 does not estimate register use correctly for d16
+  // image store instructions. The data operand is computed as if it were not a
+  // d16 image instruction.
+  if (ImageStore && Subtarget->hasImageStoreD16Bug()) {
+    // Bitcast to i16
+    EVT IntStoreVT = StoreVT.changeTypeToInteger();
+    SDValue IntVData = DAG.getNode(ISD::BITCAST, DL, IntStoreVT, VData);
+
+    // Decompose into scalars
+    SmallVector<SDValue, 4> Elts;
+    DAG.ExtractVectorElements(IntVData, Elts);
+
+    // Group pairs of i16 into v2i16 and bitcast to i32
+    SmallVector<SDValue, 4> PackedElts;
+    for (unsigned I = 0; I < Elts.size() / 2; I += 1) {
+      SDValue Pair =
+          DAG.getBuildVector(MVT::v2i16, DL, {Elts[I * 2], Elts[I * 2 + 1]});
+      SDValue IntPair = DAG.getNode(ISD::BITCAST, DL, MVT::i32, Pair);
+      PackedElts.push_back(IntPair);
+    }
+
+    // Pad using UNDEF
+    PackedElts.resize(PackedElts.size() * 2, DAG.getUNDEF(MVT::i32));
+
+    // Build final vector
+    EVT VecVT =
+        EVT::getVectorVT(*DAG.getContext(), MVT::i32, PackedElts.size());
+    return DAG.getBuildVector(VecVT, DL, PackedElts);
+  }
+
   assert(isTypeLegal(StoreVT));
   return VData;
 }
 
 SDValue SITargetLowering::LowerINTRINSIC_VOID(SDValue Op,
                                               SelectionDAG &DAG) const {
   SDLoc DL(Op);
   SDValue Chain = Op.getOperand(0);

llvm/test/CodeGen/AMDGPU/GlobalISel/legalize-llvm.amdgcn.image.store.2d.d16.ll

 ; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tonga -stop-after=legalizer -o - %s | FileCheck -check-prefix=UNPACKED %s
-; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx810 -stop-after=legalizer -o - %s | FileCheck -check-prefix=PACKED %s
+; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx810 -stop-after=legalizer -o - %s | FileCheck -check-prefix=GFX81 %s
 
 define amdgpu_ps void @image_store_f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, half %data) {
   ; UNPACKED-LABEL: name: image_store_f16
   ; UNPACKED: bb.1 (%ir-block.0):
   ; UNPACKED:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2
   ; UNPACKED:   [[COPY:%[0-9]+]]:_(s32) = COPY $sgpr2
   ; UNPACKED:   [[COPY1:%[0-9]+]]:_(s32) = COPY $sgpr3
   ; UNPACKED:   [[COPY2:%[0-9]+]]:_(s32) = COPY $sgpr4
define amdgpu_ps void @image_store_f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, half %data) {
   ; PACKED:   [[COPY8:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; PACKED:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; PACKED:   [[COPY10:%[0-9]+]]:_(s32) = COPY $vgpr2
   ; PACKED:   [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[COPY10]](s32)
   ; PACKED:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
   ; PACKED:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
   ; PACKED:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[TRUNC]](s16), 1, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 2 into custom "TargetCustom8")
   ; PACKED:   S_ENDPGM 0
+  ; GFX81-LABEL: name: image_store_f16
+  ; GFX81: bb.1 (%ir-block.0):
+  ; GFX81:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2
+  ; GFX81:   [[COPY:%[0-9]+]]:_(s32) = COPY $sgpr2
+  ; GFX81:   [[COPY1:%[0-9]+]]:_(s32) = COPY $sgpr3
+  ; GFX81:   [[COPY2:%[0-9]+]]:_(s32) = COPY $sgpr4
+  ; GFX81:   [[COPY3:%[0-9]+]]:_(s32) = COPY $sgpr5
+  ; GFX81:   [[COPY4:%[0-9]+]]:_(s32) = COPY $sgpr6
+  ; GFX81:   [[COPY5:%[0-9]+]]:_(s32) = COPY $sgpr7
+  ; GFX81:   [[COPY6:%[0-9]+]]:_(s32) = COPY $sgpr8
+  ; GFX81:   [[COPY7:%[0-9]+]]:_(s32) = COPY $sgpr9
+  ; GFX81:   [[COPY8:%[0-9]+]]:_(s32) = COPY $vgpr0
+  ; GFX81:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
+  ; GFX81:   [[COPY10:%[0-9]+]]:_(s32) = COPY $vgpr2
+  ; GFX81:   [[TRUNC:%[0-9]+]]:_(s16) = G_TRUNC [[COPY10]](s32)
+  ; GFX81:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
+  ; GFX81:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
+  ; GFX81:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[TRUNC]](s16), 1, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 2 into custom "TargetCustom8")
+  ; GFX81:   S_ENDPGM 0
   call void @llvm.amdgcn.image.store.2d.f16.i32(half %data, i32 1, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_ps void @image_store_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <2 x half> %in) {
   ; UNPACKED-LABEL: name: image_store_v2f16
   ; UNPACKED: bb.1 (%ir-block.0):
   ; UNPACKED:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2
define amdgpu_ps void @image_store_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <2 x half> %in) {
   ; PACKED:   [[COPY7:%[0-9]+]]:_(s32) = COPY $sgpr9
   ; PACKED:   [[COPY8:%[0-9]+]]:_(s32) = COPY $vgpr0
   ; PACKED:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; PACKED:   [[COPY10:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr2
   ; PACKED:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
   ; PACKED:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
   ; PACKED:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[COPY10]](<2 x s16>), 3, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 4 into custom "TargetCustom8")
   ; PACKED:   S_ENDPGM 0
+  ; GFX81-LABEL: name: image_store_v2f16
+  ; GFX81: bb.1 (%ir-block.0):
+  ; GFX81:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2
+  ; GFX81:   [[COPY:%[0-9]+]]:_(s32) = COPY $sgpr2
+  ; GFX81:   [[COPY1:%[0-9]+]]:_(s32) = COPY $sgpr3
+  ; GFX81:   [[COPY2:%[0-9]+]]:_(s32) = COPY $sgpr4
+  ; GFX81:   [[COPY3:%[0-9]+]]:_(s32) = COPY $sgpr5
+  ; GFX81:   [[COPY4:%[0-9]+]]:_(s32) = COPY $sgpr6
+  ; GFX81:   [[COPY5:%[0-9]+]]:_(s32) = COPY $sgpr7
+  ; GFX81:   [[COPY6:%[0-9]+]]:_(s32) = COPY $sgpr8
+  ; GFX81:   [[COPY7:%[0-9]+]]:_(s32) = COPY $sgpr9
+  ; GFX81:   [[COPY8:%[0-9]+]]:_(s32) = COPY $vgpr0
+  ; GFX81:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
+  ; GFX81:   [[COPY10:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr2
+  ; GFX81:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
+  ; GFX81:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
+  ; GFX81:   [[BITCAST:%[0-9]+]]:_(s32) = G_BITCAST [[COPY10]](<2 x s16>)
+  ; GFX81:   [[DEF:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
+  ; GFX81:   [[BUILD_VECTOR2:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[BITCAST]](s32), [[DEF]](s32)
+  ; GFX81:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[BUILD_VECTOR2]](<2 x s32>), 3, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 4 into custom "TargetCustom8")
+  ; GFX81:   S_ENDPGM 0
   call void @llvm.amdgcn.image.store.2d.v2f16.i32(<2 x half> %in, i32 3, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_ps void @image_store_v3f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <3 x half> %in) {
   ; UNPACKED-LABEL: name: image_store_v3f16
   ; UNPACKED: bb.1 (%ir-block.0):
   ; UNPACKED:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2, $vgpr3
define amdgpu_ps void @image_store_v3f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <3 x half> %in) {
   ; PACKED:   [[COPY11:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr3
   ; PACKED:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
   ; PACKED:   [[DEF:%[0-9]+]]:_(<2 x s16>) = G_IMPLICIT_DEF
   ; PACKED:   [[CONCAT_VECTORS:%[0-9]+]]:_(<6 x s16>) = G_CONCAT_VECTORS [[COPY10]](<2 x s16>), [[COPY11]](<2 x s16>), [[DEF]](<2 x s16>)
   ; PACKED:   [[UV:%[0-9]+]]:_(<3 x s16>), [[UV1:%[0-9]+]]:_(<3 x s16>) = G_UNMERGE_VALUES [[CONCAT_VECTORS]](<6 x s16>)
   ; PACKED:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
   ; PACKED:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[UV]](<3 x s16>), 7, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 6 into custom "TargetCustom8", align 8)
   ; PACKED:   S_ENDPGM 0
+  ; GFX81-LABEL: name: image_store_v3f16
+  ; GFX81: bb.1 (%ir-block.0):
+  ; GFX81:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2, $vgpr3
+  ; GFX81:   [[COPY:%[0-9]+]]:_(s32) = COPY $sgpr2
+  ; GFX81:   [[COPY1:%[0-9]+]]:_(s32) = COPY $sgpr3
+  ; GFX81:   [[COPY2:%[0-9]+]]:_(s32) = COPY $sgpr4
+  ; GFX81:   [[COPY3:%[0-9]+]]:_(s32) = COPY $sgpr5
+  ; GFX81:   [[COPY4:%[0-9]+]]:_(s32) = COPY $sgpr6
+  ; GFX81:   [[COPY5:%[0-9]+]]:_(s32) = COPY $sgpr7
+  ; GFX81:   [[COPY6:%[0-9]+]]:_(s32) = COPY $sgpr8
+  ; GFX81:   [[COPY7:%[0-9]+]]:_(s32) = COPY $sgpr9
+  ; GFX81:   [[COPY8:%[0-9]+]]:_(s32) = COPY $vgpr0
+  ; GFX81:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
+  ; GFX81:   [[COPY10:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr2
+  ; GFX81:   [[COPY11:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr3
+  ; GFX81:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
+  ; GFX81:   [[DEF:%[0-9]+]]:_(<2 x s16>) = G_IMPLICIT_DEF
+  ; GFX81:   [[CONCAT_VECTORS:%[0-9]+]]:_(<6 x s16>) = G_CONCAT_VECTORS [[COPY10]](<2 x s16>), [[COPY11]](<2 x s16>), [[DEF]](<2 x s16>)
+  ; GFX81:   [[UV:%[0-9]+]]:_(<3 x s16>), [[UV1:%[0-9]+]]:_(<3 x s16>) = G_UNMERGE_VALUES [[CONCAT_VECTORS]](<6 x s16>)
+  ; GFX81:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
+  ; GFX81:   [[DEF1:%[0-9]+]]:_(<4 x s16>) = G_IMPLICIT_DEF
+  ; GFX81:   [[INSERT:%[0-9]+]]:_(<4 x s16>) = G_INSERT [[DEF1]], [[UV]](<3 x s16>), 0
+  ; GFX81:   [[UV2:%[0-9]+]]:_(<2 x s16>), [[UV3:%[0-9]+]]:_(<2 x s16>) = G_UNMERGE_VALUES [[INSERT]](<4 x s16>)
+  ; GFX81:   [[BITCAST:%[0-9]+]]:_(s32) = G_BITCAST [[UV2]](<2 x s16>)
+  ; GFX81:   [[C:%[0-9]+]]:_(s32) = G_CONSTANT i32 16
+  ; GFX81:   [[LSHR:%[0-9]+]]:_(s32) = G_LSHR [[BITCAST]], [[C]](s32)
+  ; GFX81:   [[BITCAST1:%[0-9]+]]:_(s32) = G_BITCAST [[UV3]](<2 x s16>)
+  ; GFX81:   [[LSHR1:%[0-9]+]]:_(s32) = G_LSHR [[BITCAST1]], [[C]](s32)
+  ; GFX81:   [[C1:%[0-9]+]]:_(s32) = G_CONSTANT i32 65535
+  ; GFX81:   [[COPY12:%[0-9]+]]:_(s32) = COPY [[BITCAST]](s32)
+  ; GFX81:   [[AND:%[0-9]+]]:_(s32) = G_AND [[COPY12]], [[C1]]
+  ; GFX81:   [[COPY13:%[0-9]+]]:_(s32) = COPY [[LSHR]](s32)
+  ; GFX81:   [[AND1:%[0-9]+]]:_(s32) = G_AND [[COPY13]], [[C1]]
+  ; GFX81:   [[SHL:%[0-9]+]]:_(s32) = G_SHL [[AND1]], [[C]](s32)
+  ; GFX81:   [[OR:%[0-9]+]]:_(s32) = G_OR [[AND]], [[SHL]]
+  ; GFX81:   [[BITCAST2:%[0-9]+]]:_(<2 x s16>) = G_BITCAST [[OR]](s32)
+  ; GFX81:   [[COPY14:%[0-9]+]]:_(s32) = COPY [[BITCAST1]](s32)
+  ; GFX81:   [[AND2:%[0-9]+]]:_(s32) = G_AND [[COPY14]], [[C1]]
+  ; GFX81:   [[C2:%[0-9]+]]:_(s32) = G_CONSTANT i32 0
+  ; GFX81:   [[SHL1:%[0-9]+]]:_(s32) = G_SHL [[C2]], [[C]](s32)
+  ; GFX81:   [[OR1:%[0-9]+]]:_(s32) = G_OR [[AND2]], [[SHL1]]
+  ; GFX81:   [[BITCAST3:%[0-9]+]]:_(<2 x s16>) = G_BITCAST [[OR1]](s32)
+  ; GFX81:   [[OR2:%[0-9]+]]:_(s32) = G_OR [[C2]], [[SHL1]]
+  ; GFX81:   [[BITCAST4:%[0-9]+]]:_(<2 x s16>) = G_BITCAST [[OR2]](s32)
+  ; GFX81:   [[COPY15:%[0-9]+]]:_(s32) = COPY [[OR2]](s32)
+  ; GFX81:   [[BITCAST5:%[0-9]+]]:_(<2 x s16>) = G_BITCAST [[COPY15]](s32)
+  ; GFX81:   [[CONCAT_VECTORS1:%[0-9]+]]:_(<8 x s16>) = G_CONCAT_VECTORS [[BITCAST2]](<2 x s16>), [[BITCAST3]](<2 x s16>), [[BITCAST4]](<2 x s16>), [[BITCAST5]](<2 x s16>)
+  ; GFX81:   [[BITCAST6:%[0-9]+]]:_(<4 x s32>) = G_BITCAST [[CONCAT_VECTORS1]](<8 x s16>)
+  ; GFX81:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[BITCAST6]](<4 x s32>), 7, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 6 into custom "TargetCustom8", align 8)
+  ; GFX81:   S_ENDPGM 0
   call void @llvm.amdgcn.image.store.2d.v3f16.i32(<3 x half> %in, i32 7, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_ps void @image_store_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <4 x half> %in) {
   ; UNPACKED-LABEL: name: image_store_v4f16
   ; UNPACKED: bb.1 (%ir-block.0):
   ; UNPACKED:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2, $vgpr3
define amdgpu_ps void @image_store_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <4 x half> %in) {
   ; PACKED:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
   ; PACKED:   [[COPY10:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr2
   ; PACKED:   [[COPY11:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr3
   ; PACKED:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
   ; PACKED:   [[CONCAT_VECTORS:%[0-9]+]]:_(<4 x s16>) = G_CONCAT_VECTORS [[COPY10]](<2 x s16>), [[COPY11]](<2 x s16>)
   ; PACKED:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
   ; PACKED:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[CONCAT_VECTORS]](<4 x s16>), 15, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 8 into custom "TargetCustom8")
   ; PACKED:   S_ENDPGM 0
+  ; GFX81-LABEL: name: image_store_v4f16
+  ; GFX81: bb.1 (%ir-block.0):
+  ; GFX81:   liveins: $sgpr2, $sgpr3, $sgpr4, $sgpr5, $sgpr6, $sgpr7, $sgpr8, $sgpr9, $vgpr0, $vgpr1, $vgpr2, $vgpr3
+  ; GFX81:   [[COPY:%[0-9]+]]:_(s32) = COPY $sgpr2
+  ; GFX81:   [[COPY1:%[0-9]+]]:_(s32) = COPY $sgpr3
+  ; GFX81:   [[COPY2:%[0-9]+]]:_(s32) = COPY $sgpr4
+  ; GFX81:   [[COPY3:%[0-9]+]]:_(s32) = COPY $sgpr5
+  ; GFX81:   [[COPY4:%[0-9]+]]:_(s32) = COPY $sgpr6
+  ; GFX81:   [[COPY5:%[0-9]+]]:_(s32) = COPY $sgpr7
+  ; GFX81:   [[COPY6:%[0-9]+]]:_(s32) = COPY $sgpr8
+  ; GFX81:   [[COPY7:%[0-9]+]]:_(s32) = COPY $sgpr9
+  ; GFX81:   [[COPY8:%[0-9]+]]:_(s32) = COPY $vgpr0
+  ; GFX81:   [[COPY9:%[0-9]+]]:_(s32) = COPY $vgpr1
+  ; GFX81:   [[COPY10:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr2
+  ; GFX81:   [[COPY11:%[0-9]+]]:_(<2 x s16>) = COPY $vgpr3
+  ; GFX81:   [[BUILD_VECTOR:%[0-9]+]]:_(<8 x s32>) = G_BUILD_VECTOR [[COPY]](s32), [[COPY1]](s32), [[COPY2]](s32), [[COPY3]](s32), [[COPY4]](s32), [[COPY5]](s32), [[COPY6]](s32), [[COPY7]](s32)
+  ; GFX81:   [[CONCAT_VECTORS:%[0-9]+]]:_(<4 x s16>) = G_CONCAT_VECTORS [[COPY10]](<2 x s16>), [[COPY11]](<2 x s16>)
+  ; GFX81:   [[BUILD_VECTOR1:%[0-9]+]]:_(<2 x s32>) = G_BUILD_VECTOR [[COPY8]](s32), [[COPY9]](s32)
+  ; GFX81:   [[BITCAST:%[0-9]+]]:_(<2 x s32>) = G_BITCAST [[CONCAT_VECTORS]](<4 x s16>)
+  ; GFX81:   [[UV:%[0-9]+]]:_(s32), [[UV1:%[0-9]+]]:_(s32) = G_UNMERGE_VALUES [[BITCAST]](<2 x s32>)
+  ; GFX81:   [[DEF:%[0-9]+]]:_(s32) = G_IMPLICIT_DEF
+  ; GFX81:   [[BUILD_VECTOR2:%[0-9]+]]:_(<4 x s32>) = G_BUILD_VECTOR [[UV]](s32), [[UV1]](s32), [[DEF]](s32), [[DEF]](s32)
+  ; GFX81:   G_AMDGPU_INTRIN_IMAGE_STORE intrinsic(@llvm.amdgcn.image.store.2d), [[BUILD_VECTOR2]](<4 x s32>), 15, [[BUILD_VECTOR1]](<2 x s32>), $noreg, [[BUILD_VECTOR]](<8 x s32>), 0, 0, 0 :: (dereferenceable store 8 into custom "TargetCustom8")
+  ; GFX81:   S_ENDPGM 0
   call void @llvm.amdgcn.image.store.2d.v4f16.i32(<4 x half> %in, i32 15, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 declare void @llvm.amdgcn.image.store.2d.f16.i32(half, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 declare void @llvm.amdgcn.image.store.2d.v2f16.i32(<2 x half>, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 declare void @llvm.amdgcn.image.store.2d.v3f16.i32(<3 x half>, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 declare void @llvm.amdgcn.image.store.2d.v4f16.i32(<4 x half>, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 
 attributes #0 = { nounwind writeonly }

llvm/test/CodeGen/AMDGPU/GlobalISel/llvm.amdgcn.image.store.2d.d16.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=tonga -o - %s | FileCheck -check-prefix=UNPACKED %s
-; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx810 -o - %s | FileCheck -check-prefix=PACKED %s
+; RUN: llc -global-isel -mtriple=amdgcn-mesa-mesa3d -mcpu=gfx810 -o - %s | FileCheck -check-prefix=GFX81 %s
 
 define amdgpu_ps void @image_store_f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, half %data) {
 ; UNPACKED-LABEL: image_store_f16:
 ; UNPACKED:       ; %bb.0:
 ; UNPACKED-NEXT:    s_mov_b32 s0, s2
 ; UNPACKED-NEXT:    s_mov_b32 s1, s3
 ; UNPACKED-NEXT:    s_mov_b32 s2, s4
 ; UNPACKED-NEXT:    s_mov_b32 s3, s5
 ; UNPACKED-NEXT:    s_mov_b32 s4, s6
 ; UNPACKED-NEXT:    s_mov_b32 s5, s7
 ; UNPACKED-NEXT:    s_mov_b32 s6, s8
 ; UNPACKED-NEXT:    s_mov_b32 s7, s9
-; UNPACKED-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x1 unorm
+; UNPACKED-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x1 unorm d16
 ; UNPACKED-NEXT:    s_endpgm
 ;
 ; PACKED-LABEL: image_store_f16:
 ; PACKED:       ; %bb.0:
 ; PACKED-NEXT:    s_mov_b32 s0, s2
 ; PACKED-NEXT:    s_mov_b32 s1, s3
 ; PACKED-NEXT:    s_mov_b32 s2, s4
 ; PACKED-NEXT:    s_mov_b32 s3, s5
 ; PACKED-NEXT:    s_mov_b32 s4, s6
 ; PACKED-NEXT:    s_mov_b32 s5, s7
 ; PACKED-NEXT:    s_mov_b32 s6, s8
 ; PACKED-NEXT:    s_mov_b32 s7, s9
-; PACKED-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x1 unorm
+; PACKED-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x1 unorm d16
 ; PACKED-NEXT:    s_endpgm
+;
+; GFX81-LABEL: image_store_f16:
+; GFX81:       ; %bb.0:
+; GFX81-NEXT:    s_mov_b32 s0, s2
+; GFX81-NEXT:    s_mov_b32 s1, s3
+; GFX81-NEXT:    s_mov_b32 s2, s4
+; GFX81-NEXT:    s_mov_b32 s3, s5
+; GFX81-NEXT:    s_mov_b32 s4, s6
+; GFX81-NEXT:    s_mov_b32 s5, s7
+; GFX81-NEXT:    s_mov_b32 s6, s8
+; GFX81-NEXT:    s_mov_b32 s7, s9
+; GFX81-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x1 unorm d16
+; GFX81-NEXT:    s_endpgm
   call void @llvm.amdgcn.image.store.2d.f16.i32(half %data, i32 1, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_ps void @image_store_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <2 x half> %in) {
 ; UNPACKED-LABEL: image_store_v2f16:
 ; UNPACKED:       ; %bb.0:
 ; UNPACKED-NEXT:    s_mov_b32 s0, s2
 ; UNPACKED-NEXT:    s_mov_b32 s1, s3
 ; UNPACKED-NEXT:    s_mov_b32 s2, s4
 ; UNPACKED-NEXT:    s_mov_b32 s3, s5
 ; UNPACKED-NEXT:    s_mov_b32 s4, s6
 ; UNPACKED-NEXT:    s_mov_b32 s5, s7
 ; UNPACKED-NEXT:    s_mov_b32 s6, s8
 ; UNPACKED-NEXT:    s_mov_b32 s7, s9
 ; UNPACKED-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; UNPACKED-NEXT:    image_store v[2:3], v[0:1], s[0:7] dmask:0x3 unorm
+; UNPACKED-NEXT:    image_store v[2:3], v[0:1], s[0:7] dmask:0x3 unorm d16
 ; UNPACKED-NEXT:    s_endpgm
 ;
 ; PACKED-LABEL: image_store_v2f16:
 ; PACKED:       ; %bb.0:
 ; PACKED-NEXT:    s_mov_b32 s0, s2
 ; PACKED-NEXT:    s_mov_b32 s1, s3
 ; PACKED-NEXT:    s_mov_b32 s2, s4
 ; PACKED-NEXT:    s_mov_b32 s3, s5
 ; PACKED-NEXT:    s_mov_b32 s4, s6
 ; PACKED-NEXT:    s_mov_b32 s5, s7
 ; PACKED-NEXT:    s_mov_b32 s6, s8
 ; PACKED-NEXT:    s_mov_b32 s7, s9
-; PACKED-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x3 unorm
+; PACKED-NEXT:    image_store v2, v[0:1], s[0:7] dmask:0x3 unorm d16
 ; PACKED-NEXT:    s_endpgm
+;
+; GFX81-LABEL: image_store_v2f16:
+; GFX81:       ; %bb.0:
+; GFX81-NEXT:    s_mov_b32 s0, s2
+; GFX81-NEXT:    s_mov_b32 s1, s3
+; GFX81-NEXT:    s_mov_b32 s2, s4
+; GFX81-NEXT:    s_mov_b32 s3, s5
+; GFX81-NEXT:    s_mov_b32 s4, s6
+; GFX81-NEXT:    s_mov_b32 s5, s7
+; GFX81-NEXT:    s_mov_b32 s6, s8
+; GFX81-NEXT:    s_mov_b32 s7, s9
+; GFX81-NEXT:    image_store v[2:3], v[0:1], s[0:7] dmask:0x3 unorm d16
+; GFX81-NEXT:    s_endpgm
   call void @llvm.amdgcn.image.store.2d.v2f16.i32(<2 x half> %in, i32 3, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 ; FIXME: Broken
 ; define amdgpu_ps void @image_store_v3f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <3 x half> %in) {
 ;   call void @llvm.amdgcn.image.store.2d.v3f16.i32(<3 x half> %in, i32 7, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
 ;   ret void

 ; UNPACKED-NEXT:    s_mov_b32 s3, s5
 ; UNPACKED-NEXT:    s_mov_b32 s4, s6
 ; UNPACKED-NEXT:    s_mov_b32 s5, s7
 ; UNPACKED-NEXT:    s_mov_b32 s6, s8
 ; UNPACKED-NEXT:    s_mov_b32 s7, s9
 ; UNPACKED-NEXT:    v_mov_b32_e32 v5, v0
 ; UNPACKED-NEXT:    v_lshrrev_b32_e32 v2, 16, v1
 ; UNPACKED-NEXT:    v_lshrrev_b32_e32 v4, 16, v3
-; UNPACKED-NEXT:    image_store v[1:4], v[5:6], s[0:7] dmask:0xf unorm
+; UNPACKED-NEXT:    image_store v[1:4], v[5:6], s[0:7] dmask:0xf unorm d16
 ; UNPACKED-NEXT:    s_endpgm
 ;
 ; PACKED-LABEL: image_store_v4f16:
 ; PACKED:       ; %bb.0:
 ; PACKED-NEXT:    s_mov_b32 s0, s2
 ; PACKED-NEXT:    s_mov_b32 s1, s3
 ; PACKED-NEXT:    s_mov_b32 s2, s4
 ; PACKED-NEXT:    s_mov_b32 s3, s5
 ; PACKED-NEXT:    s_mov_b32 s4, s6
 ; PACKED-NEXT:    s_mov_b32 s5, s7
 ; PACKED-NEXT:    s_mov_b32 s6, s8
 ; PACKED-NEXT:    s_mov_b32 s7, s9
-; PACKED-NEXT:    image_store v[2:3], v[0:1], s[0:7] dmask:0xf unorm
+; PACKED-NEXT:    image_store v[2:3], v[0:1], s[0:7] dmask:0xf unorm d16
 ; PACKED-NEXT:    s_endpgm
+;
+; GFX81-LABEL: image_store_v4f16:
+; GFX81:       ; %bb.0:
+; GFX81-NEXT:    s_mov_b32 s0, s2
+; GFX81-NEXT:    s_mov_b32 s1, s3
+; GFX81-NEXT:    s_mov_b32 s2, s4
+; GFX81-NEXT:    s_mov_b32 s3, s5
+; GFX81-NEXT:    s_mov_b32 s4, s6
+; GFX81-NEXT:    s_mov_b32 s5, s7
+; GFX81-NEXT:    s_mov_b32 s6, s8
+; GFX81-NEXT:    s_mov_b32 s7, s9
+; GFX81-NEXT:    image_store v[2:5], v[0:1], s[0:7] dmask:0xf unorm d16
+; GFX81-NEXT:    s_endpgm
   call void @llvm.amdgcn.image.store.2d.v4f16.i32(<4 x half> %in, i32 15, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 declare void @llvm.amdgcn.image.store.2d.f16.i32(half, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 declare void @llvm.amdgcn.image.store.2d.v2f16.i32(<2 x half>, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 declare void @llvm.amdgcn.image.store.2d.v3f16.i32(<3 x half>, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 declare void @llvm.amdgcn.image.store.2d.v4f16.i32(<4 x half>, i32 immarg, i32, i32, <8 x i32>, i32 immarg, i32 immarg) #0
 
 attributes #0 = { nounwind writeonly }

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.d16.dim.ll

 ; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck -check-prefixes=GCN,UNPACKED,GFX89 %s
-; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX81,GFX89 %s
+; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX81,GFX89 %s
 ; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX9,GFX89 %s
 ; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX10 %s
 
 ; GCN-LABEL: {{^}}image_load_f16:
 ; GFX89: image_load v0, v[0:1], s[0:7] dmask:0x1 unorm d16{{$}}
 ; GFX10: image_load v0, v[0:1], s[0:7] dmask:0x1 dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps half @image_load_f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t) {
 main_body:
   %tex = call half @llvm.amdgcn.image.load.2d.f16.i32(i32 1, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret half %tex
 }
 
 ; GCN-LABEL: {{^}}image_load_v2f16:
 ; UNPACKED: image_load v[0:1], v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
 ; PACKED: image_load v0, v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
+; GFX81: image_load v0, v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
 ; GFX10: image_load v0, v[0:1], s[0:7] dmask:0x3 dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps float @image_load_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t) {
 main_body:
   %tex = call <2 x half> @llvm.amdgcn.image.load.2d.v2f16.i32(i32 3, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   %r = bitcast <2 x half> %tex to float
   ret float %r
 }
 
 ; GCN-LABEL: {{^}}image_load_v4f16:
 ; UNPACKED: image_load v[0:3], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; PACKED: image_load v[0:1], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
+; GFX81: image_load v[0:1], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; GFX10: image_load v[0:1], v[0:1], s[0:7] dmask:0xf dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps <2 x float> @image_load_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t) {
 main_body:
   %tex = call <4 x half> @llvm.amdgcn.image.load.2d.v4f16.i32(i32 15, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   %r = bitcast <4 x half> %tex to <2 x float>
   ret <2 x float> %r
 }
 
 ; GCN-LABEL: {{^}}image_load_mip_v4f16:
 ; UNPACKED: image_load_mip v[0:3], v[0:2], s[0:7] dmask:0xf unorm d16{{$}}
 ; PACKED: image_load_mip v[0:1], v[0:2], s[0:7] dmask:0xf unorm d16{{$}}
+; GFX81: image_load_mip v[0:1], v[0:2], s[0:7] dmask:0xf unorm d16{{$}}
 ; GFX10: image_load_mip v[0:1], v[0:2], s[0:7] dmask:0xf dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps <2 x float> @image_load_mip_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, i32 %mip) {
 main_body:
   %tex = call <4 x half> @llvm.amdgcn.image.load.mip.2d.v4f16.i32(i32 15, i32 %s, i32 %t, i32 %mip, <8 x i32> %rsrc, i32 0, i32 0)
   %r = bitcast <4 x half> %tex to <2 x float>
   ret <2 x float> %r
 }
 
 ; GCN-LABEL: {{^}}image_load_3d_v2f16:
 ; UNPACKED: image_load v[0:1], v[0:2], s[0:7] dmask:0x3 unorm d16{{$}}
 ; PACKED: image_load v0, v[0:2], s[0:7] dmask:0x3 unorm d16{{$}}
+; GFX81: image_load v0, v[0:2], s[0:7] dmask:0x3 unorm d16{{$}}
 ; GFX10: image_load v0, v[0:2], s[0:7] dmask:0x3 dim:SQ_RSRC_IMG_3D unorm d16{{$}}
 define amdgpu_ps float @image_load_3d_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, i32 %r) {
 main_body:
   %tex = call <2 x half> @llvm.amdgcn.image.load.3d.v2f16.i32(i32 3, i32 %s, i32 %t, i32 %r, <8 x i32> %rsrc, i32 0, i32 0)
   %x = bitcast <2 x half> %tex to float
   ret float %x
 }
 
 ; GCN-LABEL: {{^}}image_store_f16
 ; GFX89: image_store v2, v[0:1], s[0:7] dmask:0x1 unorm d16{{$}}
 ; GFX10: image_store v2, v[0:1], s[0:7] dmask:0x1 dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps void @image_store_f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, half %data) {
 main_body:
   call void @llvm.amdgcn.image.store.2d.f16.i32(half %data, i32 1, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 ; GCN-LABEL: {{^}}image_store_v2f16
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: image_store v[{{[0-9:]+}}], v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
 ; PACKED: image_store v2, v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
+; GFX81: image_store v[2:3], v[0:1], s[0:7] dmask:0x3 unorm d16{{$}}
 ; GFX10: image_store v2, v[0:1], s[0:7] dmask:0x3 dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps void @image_store_v2f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, float %in) {
 main_body:
   %data = bitcast float %in to <2 x half>
   call void @llvm.amdgcn.image.store.2d.v2f16.i32(<2 x half> %data, i32 3, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 ; GCN-LABEL: {{^}}image_store_v4f16
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: image_store v[{{[0-9:]+}}], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; PACKED: image_store v[2:3], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
+; GFX81: image_store v[2:5], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; GFX10: image_store v[2:3], v[0:1], s[0:7] dmask:0xf dim:SQ_RSRC_IMG_2D unorm d16{{$}}
 define amdgpu_ps void @image_store_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %t, <2 x float> %in) {
 main_body:
   %data = bitcast <2 x float> %in to <4 x half>
   call void @llvm.amdgcn.image.store.2d.v4f16.i32(<4 x half> %data, i32 15, i32 %s, i32 %t, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 ; GCN-LABEL: {{^}}image_store_mip_1d_v4f16
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: v_lshrrev_b32_e32
 ; UNPACKED: v_and_b32_e32
 ; UNPACKED: image_store_mip v[{{[0-9:]+}}], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; PACKED: image_store_mip v[2:3], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
+; GFX81: image_store_mip v[2:5], v[0:1], s[0:7] dmask:0xf unorm d16{{$}}
 ; GFX10: image_store_mip v[2:3], v[0:1], s[0:7] dmask:0xf dim:SQ_RSRC_IMG_1D unorm d16{{$}}
 define amdgpu_ps void @image_store_mip_1d_v4f16(<8 x i32> inreg %rsrc, i32 %s, i32 %mip, <2 x float> %in) {
 main_body:
   %data = bitcast <2 x float> %in to <4 x half>
   call void @llvm.amdgcn.image.store.mip.1d.v4f16.i32(<4 x half> %data, i32 15, i32 %s, i32 %mip, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.d16.dim.ll

 ; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck -check-prefixes=GCN,UNPACKED,GFX89 %s
-; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX81,GFX89 %s
-; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck -check-prefixes=GCN,PACKED,GFX9,GFX89 %s
+; RUN: llc < %s -march=amdgcn -mcpu=gfx810 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX81,GFX89 %s
+; RUN: llc < %s -march=amdgcn -mcpu=gfx900 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX9,GFX89 %s
 ; RUN: llc < %s -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs | FileCheck -check-prefixes=GCN,GFX10 %s
 
 ; GCN-LABEL: {{^}}image_gather4_b_2d_v4f16:
 ; UNPACKED: image_gather4_b v[0:3], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
 ; PACKED: image_gather4_b v[0:1], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
+; GFX810: image_gather4_b v[0:3], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
+; GFX9: image_gather4_b v[0:3], v[0:2], s[0:7], s[8:11] dmask:0x4 d16{{$}}
 ; GFX10: image_gather4_b v[0:1], v[0:2], s[0:7], s[8:11] dmask:0x4 dim:SQ_RSRC_IMG_2D d16{{$}}
 define amdgpu_ps <2 x float> @image_gather4_b_2d_v4f16(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %s, float %t) {
 main_body:
   %tex = call <4 x half> @llvm.amdgcn.image.gather4.b.2d.v4f16.f32.f32(i32 4, float %bias, float %s, float %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 false, i32 0, i32 0)
   %r = bitcast <4 x half> %tex to <2 x float>
   ret <2 x float> %r
 }
 
 declare <4 x half> @llvm.amdgcn.image.gather4.b.2d.v4f16.f32.f32(i32, float, float, float, <8 x i32>, <4 x i32>, i1, i32, i32) #1
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readonly }
 attributes #2 = { nounwind readnone }