[AMDGPU] Trim zero components from buffer and image stores

[matejam]

Summary

For image and buffer stores the default behaviour on GFX11 and
older is to set all unset components to zero. So if we pass
only X component it will be the same as X000, or XY same as XY00.

This patch simplifies the passed vector of components in InstCombine
by removing zero components from the end.

For image stores it also trims DMask if necessary.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

354

Can you use possiblyDemandedEltsInMask or something else from vector utils? This doesn't feel like something you would need to invent yourself

356

This is an unchecked dyn_cast. You already did a dyn_cast at the call site, so just pass in Instruction to begin with?

359

SmallVector

360

Don't use std::map. Also, can this just be a SmallVector in the reverse direction per component?

380

Replace the opcode check with the dyn_cast

395–396

Replace the opcode check with the dyn_cast

1177

Don't bother using APInt here?

[matejam]

Thanks for the review!

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

354

I thought about using that, but this is a bit more than that.
It doesn't just find which elements were set, but also takes into consideration the DMask for image intrinsics and also ignores zeros that were inserted at the end of the vector, which is the primary goal of this patch.
There is also findDemandedEltsByAllUsers, but that works in a different direction, meaning that it will find the uses of the given Value * and update DemandedElts that way.

In my case I look at the definitions (insertelement, shufflevector or ConstantVector) of the given Value * and "recursively" go up and update DemandedElts accordingly.

359

Will be done.

360

I will look into that.

380

Will be done.

395–396

Will be done.

1177

I see I can do the same thing a lot more easier.
Like "if (DMaskVal > (1 << VWidth))"

[matejam]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

360

What about MapVector?
I'm not sure how would I use SmallVector to implement a map?

[matejam]

Using SmallVector instead of std::vector, MapVector instead of std::map, eliminated some unnecessary dyn_casts and a few more small changes.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

360

I meant it's the value to vector indexes? You could just go the other direction and find the value from the index

381

Unchecked dyn_cast, need a test with a variable vector index

[matejam]

Use of SmallVector instead of VectorMap for tracking which components were already added.
Remove some unnecessary dyn_casts.

[matejam]

Rebase and minor changes.

[matejam]

Rebase.

@foad @arsenm would you please review this?
Thank you.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

359–362

Maybe call them ComponentIndices and ComponentValues?

365

I wonder if the whole of this loop could be replaced by calling llvm::computeKnownBits with an appropriate DemandedElts mask to test each element from the last to the first.

1141

Use getArgOperand throughout for consistency?

1144

Why does this need to be an Instruction? Couldn't it be a Constant? E.g. if it was an all zeroes constant, that should be simplified.

1176

Should be >=

1348–1349

Keep the braces around this, because it is more than one physical line.

[foad]

[AMDGPU] Default component broadcast store

There is no "broadcasting" here. Maybe call it something like "trim zero components from buffer and image stores"?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

1138

Don't see why the element type would matter? They're a bit artificial

[matejam]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

365

I looked that up.
computeKnownBits works only for integers, pointers and vector of integers.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

365

You might be able to use the brand new computeKnownFPClass instead, and check if the result is exactly fcPosZero?

[matejam]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

365

Thanks.

[matejam]

Changes in findDemandedElts, use computeKnownFPClass.

[foad]

When I suggested using computeKnownFPClass I meant something like this: https://reviews.llvm.org/differential/diff/517952/
Further cleanups are possible.

[matejam]

Thank you @foad.
findDemandedElts with correct usage of computeKnownFPClass.

[matejam]

Rebase and change in comments.

[matejam]

Please review this. @foad @arsenm

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

352

This function needs a better name now to explain what it does, and a comment, and should return APInt instead of taking an APInt& argument.

1152–1155

It would be simpler to use dyn_cast and break if it fails.

1161

dyn_cast should be isa since you don't use the result. But what is this actually testing for? Would it be better to test hasNamedOperand?

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

1161

I think these intrinsics all have immarg on the dmask argument, so it should always be a constant?

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

1161

Yes, should be able to just cast<ConstantInt>

[matejam]

Thanks for the review.

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

352

Is the name trimTrailingZerosInVector more explanatory?

1161

Both image and buffer intrinsics have the same body in this switch.
For buffer instructions the first operand is not a constant, that is why I used dyn_cast<ConstantInt>.
I could use isa<ConstantInt>.
I'm not sure how would I check for hasNamedOperand, because II is a call instruction.

[matejam]

Change the name from findDemandedElts to trimTrailingZerosInVector.
Remove some unnecessary dyn_casts.
Refactor and rebase.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

352

Sure.

1153

You don't use IIVTy for anything else so just test isa<FixedVectorType>(...) instead.

1161

You definitely should not rely on the second argument of the buffer intrinsics not being a constant! Instead you should test the opcode.

[matejam]

Use Intrinsic opcode to know if the instructions has DMask instead of testing if the instruction has a ConstantInt as the second operand.
Add more run-lines to the test.

[foad]

Code looks good now, thanks.

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-intrinsics.ll

4554

This looks like we have lost an optimization that converts the "mip" form to the non-"mip" form.

[matejam]

Do the optimizations for image instructions that were done prior to this patch.

[matejam]

Refactor.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

1191–1194

Instead of duplicating this code, I think it would be neater to move it after the whole switch (...) {...} statement. Then everything should Just Work.

[matejam]

Move the default case out of the switch.

[foad]

LGTM, thanks!

[foad]

matejam added a reverting change: rG9c8c31eea439: Revert "[AMDGPU] Trim zero components from buffer and image stores".

Have you got any more details about what was wrong with it?

[matejam]

In D146737#4353046, @foad wrote:

matejam added a reverting change: rG9c8c31eea439: Revert "[AMDGPU] Trim zero components from buffer and image stores".

Have you got any more details about what was wrong with it?

No, we're having issues accessing amd network.
I don't know what tests exactly are failing and why.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

360

I don't think you want to loop to go down to i = 0. If the 0'th element is zero then you will remove the whole store instruction, which would not be right. Can you add a test for that case please?

[matejam]

Change condition in for loop, instead of i >= 0, put i > 0. We don't want to optimize out the 0th element.

[matejam]

This patch was reverted on upstream, because of failed cts tests.

[matejam]

Add test case with all zero components.

[matejam]

Remove *_buffer_store instructions from being optimized.

[foad]

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-simplify-image-buffer-stores.ll

12–16

Nit: could just use zeroinitializer.

[foad]

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

369

Could also handle undef here and treat it as zero.

llvm/lib/Target/AMDGPU/AMDGPUInstCombineIntrinsic.cpp

bool GCNTTIImpl::canSimplifyLegacyMulToMul(const Value *Op0, const Value *Op1,
   if (isKnownNeverInfinity(Op0, TLI) && isKnownNeverNaN(Op0, TLI) &&
       isKnownNeverInfinity(Op1, TLI) && isKnownNeverNaN(Op1, TLI)) {
     // Neither operand is infinity or NaN.
     return true;
   }
   return false;
 }
 
+// Demanded elements are those that are covered by DMask, without the
+// zeros at the end of the passed vector.
+static APInt findDemandedElts(Value *V, unsigned DMaskVal = 15) {
+  unsigned VWidth = cast<FixedVectorType>(V->getType())->getNumElements();
+  Instruction *CurInst = dyn_cast<Instruction>(V);
+  Value *FirstArg = CurInst;
+
+  std::vector<int> Components(VWidth, -1);
+  std::map<Value *, int> MapValueInt;
+  int LastElem = 0;
+
+  while (!isa<UndefValue>(FirstArg)) {
+    CurInst = dyn_cast<Instruction>(FirstArg);
+    if (!CurInst) {
+      if (auto *ConstVec = dyn_cast<ConstantVector>(FirstArg)) {
+        // Components[i] == -1 iff the value was set to 0 or wasn't set.
+        for (unsigned i = 0; i < ConstVec->getNumOperands(); ++i) {
+          if (Components[i] != -1)
+            continue;
+
+          auto *ConstFP = dyn_cast<ConstantFP>(ConstVec->getOperand(i));
+          // Map all non-zero constants to -2 in Components vector.
+          if (!ConstFP->isExactlyValue(0))
+            Components[i] = -2;
+        }
+        break;
+      }
+    }
+    if (CurInst->getOpcode() == Instruction::InsertElement) {
+      InsertElementInst *IEI = dyn_cast<InsertElementInst>(CurInst);
+      auto *IEIIndex = dyn_cast<ConstantInt>(IEI->getOperand(2));
+      auto *Comp = IEI->getOperand(1);
+      unsigned CompIdx = IEIIndex->getZExtValue();
+
+      auto Val = MapValueInt.find(Comp);
+      if (Val != MapValueInt.end())
+        Components[CompIdx] = Val->second;
+      else {
+        MapValueInt.insert({Comp, LastElem});
+        Components[CompIdx] = LastElem++;
+      }
+
+      FirstArg = CurInst->getOperand(0);
+    } else if (CurInst->getOpcode() == Instruction::ShuffleVector) {
+      ShuffleVectorInst *Shuffle = dyn_cast<ShuffleVectorInst>(CurInst);
+
+      for (unsigned i = 0; i < VWidth; i++) {
+        unsigned MaskVal = Shuffle->getMaskValue(i);
+        if (MaskVal == -1u || MaskVal >= 2 * VWidth)
+          continue;
+        if (MaskVal < VWidth)
+          Components[i] = MaskVal;
+        if (MaskVal >= VWidth && MaskVal < 2 * VWidth)
+          Components[i] = MaskVal - VWidth;
+      }
+      break;
+    }
+  }
+
+  APInt DemandedElts = APInt::getAllOnes(VWidth);
+  for (int i = VWidth - 1; i >= 0; --i) {
+    // If the component isn't set, is equal to zero or is set but DMask doesn't
+    // cover it, then don't include that component in the DemandedElts.
+    if (Components[i] == -1 || (Components[i] != -1 && !(DMaskVal & (1 << i))))
+      DemandedElts.clearBit(i);
+    else
+      break;
+  }
+
+  return DemandedElts;
+}
+
+static Value *simplifyAMDGCNMemoryIntrinsicDemanded(InstCombiner &IC,
+                                                    IntrinsicInst &II,
+                                                    APInt DemandedElts,
+                                                    int DMaskIdx = -1,
+                                                    bool IsLoad = true);
+
 std::optional<Instruction *>
 GCNTTIImpl::instCombineIntrinsic(InstCombiner &IC, IntrinsicInst &II) const {
   Intrinsic::ID IID = II.getIntrinsicID();
   switch (IID) {
   case Intrinsic::amdgcn_rcp: {
     Value *Src = II.getArgOperand(0);
 
     // TODO: Move to ConstantFolding/InstSimplify?
GCNTTIImpl::instCombineIntrinsic(InstCombiner &IC, IntrinsicInst &II) const {
   case Intrinsic::amdgcn_is_private: {
     if (isa<UndefValue>(II.getArgOperand(0)))
       return IC.replaceInstUsesWith(II, UndefValue::get(II.getType()));
 
     if (isa<ConstantPointerNull>(II.getArgOperand(0)))
       return IC.replaceInstUsesWith(II, ConstantInt::getFalse(II.getType()));
     break;
   }
+  case Intrinsic::amdgcn_buffer_store:
+  case Intrinsic::amdgcn_buffer_store_format:
+  case Intrinsic::amdgcn_raw_buffer_store:
+  case Intrinsic::amdgcn_raw_buffer_store_format:
+  case Intrinsic::amdgcn_raw_tbuffer_store:
+  case Intrinsic::amdgcn_struct_buffer_store:
+  case Intrinsic::amdgcn_struct_buffer_store_format:
+  case Intrinsic::amdgcn_struct_tbuffer_store:
+  case Intrinsic::amdgcn_tbuffer_store: {
+    if (II.getOperand(0)->getType()->isFloatingPointTy())
+      break;
+
+    auto *IIVTy = cast<FixedVectorType>(II.getOperand(0)->getType());
+    unsigned VWidth = IIVTy->getNumElements();
+    APInt DemandedElts = APInt::getAllOnes(VWidth);
+    if (auto *SrcVec = dyn_cast<Instruction>(II.getArgOperand(0)))
+      DemandedElts = findDemandedElts(SrcVec);
+
+    if (Value *V = simplifyAMDGCNMemoryIntrinsicDemanded(
+            IC, II, DemandedElts, -1, false))
+      return IC.eraseInstFromFunction(II);
+
+    break;
+  }
+  case Intrinsic::amdgcn_image_store_1d:
+  case Intrinsic::amdgcn_image_store_1darray:
+  case Intrinsic::amdgcn_image_store_2d:
+  case Intrinsic::amdgcn_image_store_2darray:
+  case Intrinsic::amdgcn_image_store_2darraymsaa:
+  case Intrinsic::amdgcn_image_store_2dmsaa:
+  case Intrinsic::amdgcn_image_store_3d:
+  case Intrinsic::amdgcn_image_store_cube:
+  case Intrinsic::amdgcn_image_store_mip_1d:
+  case Intrinsic::amdgcn_image_store_mip_1darray:
+  case Intrinsic::amdgcn_image_store_mip_2d:
+  case Intrinsic::amdgcn_image_store_mip_2darray:
+  case Intrinsic::amdgcn_image_store_mip_3d:
+  case Intrinsic::amdgcn_image_store_mip_cube: {
+    if (II.getOperand(0)->getType()->isFloatingPointTy())
+      break;
+
+    auto *IIVTy = cast<FixedVectorType>(II.getOperand(0)->getType());
+    unsigned VWidth = IIVTy->getNumElements();
+    ConstantInt *DMask = cast<ConstantInt>(II.getArgOperand(1));
+    unsigned DMaskVal = DMask->getZExtValue() & 0xf;
+
+    // If DMask has a value that needs more than VWidth bits for its representation,
+    // that means that the intrinsic has already been optimized.
+    if (4 - APInt(4, DMaskVal).countLeadingZeros() > VWidth)
+      break;
+
+    APInt DemandedElts = APInt::getAllOnes(VWidth);
+    if (auto *SrcVec = dyn_cast<Instruction>(II.getArgOperand(0)))
+      DemandedElts = findDemandedElts(SrcVec, DMaskVal);
+
+    if (Value *V = simplifyAMDGCNMemoryIntrinsicDemanded(
+            IC, II, DemandedElts, 1, false))
+      return IC.eraseInstFromFunction(II);
+
+    break;
+  }
   default: {
     if (const AMDGPU::ImageDimIntrinsicInfo *ImageDimIntr =
             AMDGPU::getImageDimIntrinsicInfo(II.getIntrinsicID())) {
       return simplifyAMDGCNImageIntrinsic(ST, ImageDimIntr, II, IC);
     }
   }
   }
   return std::nullopt;
 }
 
 /// Implement SimplifyDemandedVectorElts for amdgcn buffer and image intrinsics.
 ///
+/// The result of simplifying amdgcn image and buffer store intrinsics is updating
+/// definitions of the first vector argument of the intrinsic, not uses
+/// of the result like image and buffer loads.
 /// Note: This only supports non-TFE/LWE image intrinsic calls; those have
 ///       struct returns.
 static Value *simplifyAMDGCNMemoryIntrinsicDemanded(InstCombiner &IC,
                                                     IntrinsicInst &II,
                                                     APInt DemandedElts,
-                                                    int DMaskIdx = -1) {
+                                                    int DMaskIdx, bool IsLoad) {
 
-  auto *IIVTy = cast<FixedVectorType>(II.getType());
+  auto *IIVTy = cast<FixedVectorType>(IsLoad ? II.getType()
+                                             : II.getOperand(0)->getType());
   unsigned VWidth = IIVTy->getNumElements();
   if (VWidth == 1)
     return nullptr;
   Type *EltTy = IIVTy->getElementType();
 
   IRBuilderBase::InsertPointGuard Guard(IC.Builder);
   IC.Builder.SetInsertPoint(&II);
 
static Value *simplifyAMDGCNMemoryIntrinsicDemanded(InstCombiner &IC,
   SmallVector<Type *, 6> OverloadTys;
   if (!Intrinsic::getIntrinsicSignature(II.getCalledFunction(), OverloadTys))
     return nullptr;
 
   Type *NewTy =
       (NewNumElts == 1) ? EltTy : FixedVectorType::get(EltTy, NewNumElts);
   OverloadTys[0] = NewTy;
 
+  if (!IsLoad) {
+    SmallVector<int, 8> EltMask;
+    for (unsigned OrigStoreIdx = 0; OrigStoreIdx < VWidth; ++OrigStoreIdx) {
+      if (DMaskIdx < 0) {
+        // Buffer case.
+        if (DemandedElts[OrigStoreIdx])
+          EltMask.push_back(OrigStoreIdx);
+      } else {
+        // Image case.
+        ConstantInt *DMask = cast<ConstantInt>(Args[DMaskIdx]);
+        unsigned DMaskVal = DMask->getZExtValue() & 0xf;
+        if (DMaskVal & (1 << OrigStoreIdx))
+          EltMask.push_back(OrigStoreIdx);
+      }
+    }
+
+    if (NewNumElts == 1)
+      Args[0] = IC.Builder.CreateExtractElement(II.getOperand(0), EltMask[0]);
+    else
+      Args[0] = IC.Builder.CreateShuffleVector(II.getOperand(0), EltMask);
+  }
+
   Function *NewIntrin = Intrinsic::getDeclaration(
       II.getModule(), II.getIntrinsicID(), OverloadTys);
   CallInst *NewCall = IC.Builder.CreateCall(NewIntrin, Args);
   NewCall->takeName(&II);
   NewCall->copyMetadata(II);
 
-  if (NewNumElts == 1) {
+  if (IsLoad) {
+    if (NewNumElts == 1)
       return IC.Builder.CreateInsertElement(UndefValue::get(IIVTy), NewCall,
                                             DemandedElts.countr_zero());
-  }
 
     SmallVector<int, 8> EltMask;
     unsigned NewLoadIdx = 0;
     for (unsigned OrigLoadIdx = 0; OrigLoadIdx < VWidth; ++OrigLoadIdx) {
       if (!!DemandedElts[OrigLoadIdx])
         EltMask.push_back(NewLoadIdx++);
       else
         EltMask.push_back(NewNumElts);
     }
 
-  Value *Shuffle = IC.Builder.CreateShuffleVector(NewCall, EltMask);
+    auto *Shuffle = IC.Builder.CreateShuffleVector(NewCall, EltMask);
 
     return Shuffle;
   }
 
+  return NewCall;
+}
+
 std::optional<Value *> GCNTTIImpl::simplifyDemandedVectorEltsIntrinsic(
     InstCombiner &IC, IntrinsicInst &II, APInt DemandedElts, APInt &UndefElts,
     APInt &UndefElts2, APInt &UndefElts3,
     std::function<void(Instruction *, unsigned, APInt, APInt &)>
         SimplifyAndSetOp) const {
   switch (II.getIntrinsicID()) {
   case Intrinsic::amdgcn_buffer_load:
   case Intrinsic::amdgcn_buffer_load_format:

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-intrinsics.ll

 ; CHECK-NEXT:    ret double 0x3F97D05F417D05F4
 ;
   %val = call double @llvm.amdgcn.rcp.f64(double 4.300000e+01) nounwind readnone
   ret double %val
 }
 
 define float @test_constant_fold_rcp_f32_43_strictfp() nounwind strictfp {
 ; CHECK-LABEL: @test_constant_fold_rcp_f32_43_strictfp(
-; CHECK-NEXT:    [[VAL:%.*]] = call float @llvm.amdgcn.rcp.f32(float 4.300000e+01) #[[ATTR14:[0-9]+]]
+; CHECK-NEXT:    [[VAL:%.*]] = call float @llvm.amdgcn.rcp.f32(float 4.300000e+01) #[[ATTR12:[0-9]+]]
 ; CHECK-NEXT:    ret float [[VAL]]
 ;
   %val = call float @llvm.amdgcn.rcp.f32(float 4.300000e+01) strictfp nounwind readnone
   ret float %val
 }
 
 ; --------------------------------------------------------------------
 ; llvm.amdgcn.sqrt

 ; CHECK-NEXT:    ret double 0x7FF8000000000000
 ;
   %val = call double @llvm.amdgcn.sqrt.f64(double undef) nounwind readnone
   ret double %val
 }
 
 define half @test_constant_fold_sqrt_f16_0() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_f16_0(
-; CHECK-NEXT:    [[VAL:%.*]] = call half @llvm.amdgcn.sqrt.f16(half 0xH0000) #[[ATTR15:[0-9]+]]
+; CHECK-NEXT:    [[VAL:%.*]] = call half @llvm.amdgcn.sqrt.f16(half 0xH0000) #[[ATTR13:[0-9]+]]
 ; CHECK-NEXT:    ret half [[VAL]]
 ;
   %val = call half @llvm.amdgcn.sqrt.f16(half 0.0) nounwind readnone
   ret half %val
 }
 
 define float @test_constant_fold_sqrt_f32_0() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_f32_0(
-; CHECK-NEXT:    [[VAL:%.*]] = call float @llvm.amdgcn.sqrt.f32(float 0.000000e+00) #[[ATTR15]]
+; CHECK-NEXT:    [[VAL:%.*]] = call float @llvm.amdgcn.sqrt.f32(float 0.000000e+00) #[[ATTR13]]
 ; CHECK-NEXT:    ret float [[VAL]]
 ;
   %val = call float @llvm.amdgcn.sqrt.f32(float 0.0) nounwind readnone
   ret float %val
 }
 
 define double @test_constant_fold_sqrt_f64_0() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_f64_0(
-; CHECK-NEXT:    [[VAL:%.*]] = call double @llvm.amdgcn.sqrt.f64(double 0.000000e+00) #[[ATTR15]]
+; CHECK-NEXT:    [[VAL:%.*]] = call double @llvm.amdgcn.sqrt.f64(double 0.000000e+00) #[[ATTR13]]
 ; CHECK-NEXT:    ret double [[VAL]]
 ;
   %val = call double @llvm.amdgcn.sqrt.f64(double 0.0) nounwind readnone
   ret double %val
 }
 
 define half @test_constant_fold_sqrt_f16_neg0() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_f16_neg0(
-; CHECK-NEXT:    [[VAL:%.*]] = call half @llvm.amdgcn.sqrt.f16(half 0xH8000) #[[ATTR15]]
+; CHECK-NEXT:    [[VAL:%.*]] = call half @llvm.amdgcn.sqrt.f16(half 0xH8000) #[[ATTR13]]
 ; CHECK-NEXT:    ret half [[VAL]]
 ;
   %val = call half @llvm.amdgcn.sqrt.f16(half -0.0) nounwind readnone
   ret half %val
 }
 
 define float @test_constant_fold_sqrt_f32_neg0() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_f32_neg0(
-; CHECK-NEXT:    [[VAL:%.*]] = call float @llvm.amdgcn.sqrt.f32(float -0.000000e+00) #[[ATTR15]]
+; CHECK-NEXT:    [[VAL:%.*]] = call float @llvm.amdgcn.sqrt.f32(float -0.000000e+00) #[[ATTR13]]
 ; CHECK-NEXT:    ret float [[VAL]]
 ;
   %val = call float @llvm.amdgcn.sqrt.f32(float -0.0) nounwind readnone
   ret float %val
 }
 
 define double @test_constant_fold_sqrt_f64_neg0() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_f64_neg0(
-; CHECK-NEXT:    [[VAL:%.*]] = call double @llvm.amdgcn.sqrt.f64(double -0.000000e+00) #[[ATTR15]]
+; CHECK-NEXT:    [[VAL:%.*]] = call double @llvm.amdgcn.sqrt.f64(double -0.000000e+00) #[[ATTR13]]
 ; CHECK-NEXT:    ret double [[VAL]]
 ;
   %val = call double @llvm.amdgcn.sqrt.f64(double -0.0) nounwind readnone
   ret double %val
 }
 
 define double @test_constant_fold_sqrt_snan_f64() nounwind {
 ; CHECK-LABEL: @test_constant_fold_sqrt_snan_f64(

 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.amdgcn.class.f32(float %x, i32 3)
   ret i1 %val
 }
 
 define i1 @test_class_isnan_f32_strict(float %x) nounwind {
 ; CHECK-LABEL: @test_class_isnan_f32_strict(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.amdgcn.class.f32(float [[X:%.*]], i32 3) #[[ATTR16:[0-9]+]]
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.amdgcn.class.f32(float [[X:%.*]], i32 3) #[[ATTR14:[0-9]+]]
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.amdgcn.class.f32(float %x, i32 3) strictfp
   ret i1 %val
 }
 
 define i1 @test_class_is_p0_n0_f32(float %x) nounwind {
 ; CHECK-LABEL: @test_class_is_p0_n0_f32(
 ; CHECK-NEXT:    [[VAL:%.*]] = fcmp oeq float [[X:%.*]], 0.000000e+00
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.amdgcn.class.f32(float %x, i32 96)
   ret i1 %val
 }
 
 define i1 @test_class_is_p0_n0_f32_strict(float %x) nounwind {
 ; CHECK-LABEL: @test_class_is_p0_n0_f32_strict(
-; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.amdgcn.class.f32(float [[X:%.*]], i32 96) #[[ATTR16]]
+; CHECK-NEXT:    [[VAL:%.*]] = call i1 @llvm.amdgcn.class.f32(float [[X:%.*]], i32 96) #[[ATTR14]]
 ; CHECK-NEXT:    ret i1 [[VAL]]
 ;
   %val = call i1 @llvm.amdgcn.class.f32(float %x, i32 96) strictfp
   ret i1 %val
 }
 
 define i1 @test_constant_class_snan_test_snan_f64() nounwind {
 ; CHECK-LABEL: @test_constant_class_snan_test_snan_f64(

 ; CHECK-NEXT:    ret i32 0
 ;
   %bfe = call i32 @llvm.amdgcn.ubfe.i32(i32 %src, i32 0, i32 0)
   ret i32 %bfe
 }
 
 define i32 @ubfe_offset_0_width_3(i32 %src) {
 ; CHECK-LABEL: @ubfe_offset_0_width_3(
-; CHECK-NEXT:    [[TMP1:%.*]] = and i32 [[SRC:%.*]], 7
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[BFE:%.*]] = and i32 [[SRC:%.*]], 7
+; CHECK-NEXT:    ret i32 [[BFE]]
 ;
   %bfe = call i32 @llvm.amdgcn.ubfe.i32(i32 %src, i32 0, i32 3)
   ret i32 %bfe
 }
 
 define i32 @ubfe_offset_3_width_1(i32 %src) {
 ; CHECK-LABEL: @ubfe_offset_3_width_1(
 ; CHECK-NEXT:    [[TMP1:%.*]] = lshr i32 [[SRC:%.*]], 3

 ; CHECK-NEXT:    ret i64 0
 ;
   %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 9, i32 8, i32 32)
   ret i64 %result
 }
 
 define i64 @icmp_constant_inputs_true() {
 ; CHECK-LABEL: @icmp_constant_inputs_true(
-; CHECK-NEXT:    [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0:![0-9]+]]) #[[ATTR17:[0-9]+]]
+; CHECK-NEXT:    [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0:![0-9]+]]) #[[ATTR15:[0-9]+]]
 ; CHECK-NEXT:    ret i64 [[RESULT]]
 ;
   %result = call i64 @llvm.amdgcn.icmp.i64.i32(i32 9, i32 8, i32 34)
   ret i64 %result
 }
 
 define i64 @icmp_constant_to_rhs_slt(i32 %x) {
 ; CHECK-LABEL: @icmp_constant_to_rhs_slt(

 ; CHECK-NEXT:    ret i64 0
 ;
   %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float 2.0, float 4.0, i32 1)
   ret i64 %result
 }
 
 define i64 @fcmp_constant_inputs_true() {
 ; CHECK-LABEL: @fcmp_constant_inputs_true(
-; CHECK-NEXT:    [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]]) #[[ATTR17]]
+; CHECK-NEXT:    [[RESULT:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]]) #[[ATTR15]]
 ; CHECK-NEXT:    ret i64 [[RESULT]]
 ;
   %result = call i64 @llvm.amdgcn.fcmp.i64.f32(float 2.0, float 4.0, i32 4)
   ret i64 %result
 }
 
 define i64 @fcmp_constant_to_rhs_olt(float %x) {
 ; CHECK-LABEL: @fcmp_constant_to_rhs_olt(

 ; CHECK-NEXT:    ret i64 0
 ;
   %b = call i64 @llvm.amdgcn.ballot.i64(i1 0)
   ret i64 %b
 }
 
 define i64 @ballot_one_64() {
 ; CHECK-LABEL: @ballot_one_64(
-; CHECK-NEXT:    [[B:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]]) #[[ATTR17]]
+; CHECK-NEXT:    [[B:%.*]] = call i64 @llvm.read_register.i64(metadata [[META0]]) #[[ATTR15]]
 ; CHECK-NEXT:    ret i64 [[B]]
 ;
   %b = call i64 @llvm.amdgcn.ballot.i64(i1 1)
   ret i64 %b
 }
 
 define i32 @ballot_nocombine_32(i1 %i) {
 ; CHECK-LABEL: @ballot_nocombine_32(

 ; CHECK-NEXT:    ret i32 0
 ;
   %b = call i32 @llvm.amdgcn.ballot.i32(i1 0)
   ret i32 %b
 }
 
 define i32 @ballot_one_32() {
 ; CHECK-LABEL: @ballot_one_32(
-; CHECK-NEXT:    [[B:%.*]] = call i32 @llvm.read_register.i32(metadata [[META1:![0-9]+]]) #[[ATTR17]]
+; CHECK-NEXT:    [[B:%.*]] = call i32 @llvm.read_register.i32(metadata [[META1:![0-9]+]]) #[[ATTR15]]
 ; CHECK-NEXT:    ret i32 [[B]]
 ;
   %b = call i32 @llvm.amdgcn.ballot.i32(i1 1)
   ret i32 %b
 }
 
 ; --------------------------------------------------------------------
 ; llvm.amdgcn.wqm.vote
main_body:
   store <4 x float> %v, ptr addrspace(1) %out
   ret void
 }
 
 
 define amdgpu_kernel void @store_mip_1d(ptr addrspace(1) %out, <8 x i32> inreg %rsrc, <4 x float> %vdata, i32 %s) {
 ; CHECK-LABEL: @store_mip_1d(
 ; CHECK-NEXT:  main_body:
-; CHECK-NEXT:    call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; CHECK-NEXT:    call void @llvm.amdgcn.image.store.mip.1d.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 0, <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
 ; CHECK-NEXT:    ret void
 ;
 main_body:
   call void @llvm.amdgcn.image.store.mip.1d.v4f32.i32(<4 x float> %vdata, i32 15, i32 %s, i32 0, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_kernel void @store_mip_2d(ptr addrspace(1) %out, <8 x i32> inreg %rsrc, <4 x float> %vdata, i32 %s, i32 %t) {
 ; CHECK-LABEL: @store_mip_2d(
 ; CHECK-NEXT:  main_body:
-; CHECK-NEXT:    call void @llvm.amdgcn.image.store.2d.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; CHECK-NEXT:    call void @llvm.amdgcn.image.store.mip.2d.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 0, <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
 ; CHECK-NEXT:    ret void
 ;
 main_body:
   call void @llvm.amdgcn.image.store.mip.2d.v4f32.i32(<4 x float> %vdata, i32 15, i32 %s, i32 %t, i32 0, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_kernel void @store_mip_3d(ptr addrspace(1) %out, <8 x i32> inreg %rsrc, <4 x float> %vdata, i32 %s, i32 %t, i32 %u) {
 ; CHECK-LABEL: @store_mip_3d(
 ; CHECK-NEXT:  main_body:
-; CHECK-NEXT:    call void @llvm.amdgcn.image.store.3d.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 [[U:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; CHECK-NEXT:    call void @llvm.amdgcn.image.store.mip.3d.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 [[U:%.*]], i32 0, <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
 ; CHECK-NEXT:    ret void
 ;
 main_body:
   call void @llvm.amdgcn.image.store.mip.3d.v4f32.i32(<4 x float> %vdata, i32 15, i32 %s, i32 %t, i32 %u, i32 0, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_kernel void @store_mip_1darray(ptr addrspace(1) %out, <8 x i32> inreg %rsrc, <4 x float> %vdata, i32 %s, i32 %t) {
 ; CHECK-LABEL: @store_mip_1darray(
 ; CHECK-NEXT:  main_body:
-; CHECK-NEXT:    call void @llvm.amdgcn.image.store.1darray.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; CHECK-NEXT:    call void @llvm.amdgcn.image.store.mip.1darray.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 0, <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
 ; CHECK-NEXT:    ret void
 ;
 main_body:
   call void @llvm.amdgcn.image.store.mip.1darray.v4f32.i32(<4 x float> %vdata, i32 15, i32 %s, i32 %t, i32 0, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_kernel void @store_mip_2darray(ptr addrspace(1) %out, <8 x i32> inreg %rsrc, <4 x float> %vdata, i32 %s, i32 %t, i32 %u) {
 ; CHECK-LABEL: @store_mip_2darray(
 ; CHECK-NEXT:  main_body:
-; CHECK-NEXT:    call void @llvm.amdgcn.image.store.2darray.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 [[U:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; CHECK-NEXT:    call void @llvm.amdgcn.image.store.mip.2darray.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 [[U:%.*]], i32 0, <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
 ; CHECK-NEXT:    ret void
 ;
 main_body:
   call void @llvm.amdgcn.image.store.mip.2darray.v4f32.i32(<4 x float> %vdata, i32 15, i32 %s, i32 %t, i32 %u, i32 0, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 define amdgpu_kernel void @store_mip_cube(ptr addrspace(1) %out, <8 x i32> inreg %rsrc, <4 x float> %vdata, i32 %s, i32 %t, i32 %u) {
 ; CHECK-LABEL: @store_mip_cube(
 ; CHECK-NEXT:  main_body:
-; CHECK-NEXT:    call void @llvm.amdgcn.image.store.cube.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 [[U:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; CHECK-NEXT:    call void @llvm.amdgcn.image.store.mip.cube.v4f32.i32(<4 x float> [[VDATA:%.*]], i32 15, i32 [[S:%.*]], i32 [[T:%.*]], i32 [[U:%.*]], i32 0, <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
 ; CHECK-NEXT:    ret void
 ;
 main_body:
   call void @llvm.amdgcn.image.store.mip.cube.v4f32.i32(<4 x float> %vdata, i32 15, i32 %s, i32 %t, i32 %u, i32 0, <8 x i32> %rsrc, i32 0, i32 0)
   ret void
 }
 
 declare <4 x float> @llvm.amdgcn.image.load.mip.1d.v4f32.i32(i32, i32, i32, <8 x i32>, i32, i32) #1

 ; CHECK-NEXT:    ret double [[VAL]]
 ;
   %val = call double @llvm.amdgcn.trig.preop.f64(double 3.454350e+02, i32 5)
   ret double %val
 }
 
 define double @trig_preop_constfold_strictfp() {
 ; CHECK-LABEL: @trig_preop_constfold_strictfp(
-; CHECK-NEXT:    [[VAL:%.*]] = call double @llvm.amdgcn.trig.preop.f64(double 3.454350e+02, i32 5) #[[ATTR16]]
+; CHECK-NEXT:    [[VAL:%.*]] = call double @llvm.amdgcn.trig.preop.f64(double 3.454350e+02, i32 5) #[[ATTR14]]
 ; CHECK-NEXT:    ret double [[VAL]]
 ;
   %val = call double @llvm.amdgcn.trig.preop.f64(double 3.454350e+02, i32 5) strictfp
   ret double %val
 }
 
 define double @trig_preop_constfold_0.0__0() {
 ; CHECK-LABEL: @trig_preop_constfold_0.0__0(

llvm/test/Transforms/InstCombine/AMDGPU/amdgcn-simplify-image-buffer-stores.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt -mcpu=gfx1100 -S -passes=instcombine -mtriple=amdgcn-amd-amdhsa %s | FileCheck -check-prefixes=GFX11 %s
+  
+define amdgpu_ps void @image_store_1d_store_insert_zeros_at_end(<8 x i32> inreg %rsrc, float %vdata1, i32 %s) #0 {
+; GFX11-LABEL: @image_store_1d_store_insert_zeros_at_end(
+; GFX11-NEXT:    call void @llvm.amdgcn.image.store.1d.f32.i32(float [[VDATA1:%.*]], i32 1, i32 [[S:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; GFX11-NEXT:    ret void
+;
+  %newvdata1 = insertelement <4 x float> undef, float %vdata1, i32 0
+  %newvdata2 = insertelement <4 x float> %newvdata1, float 0.0, i32 1
+  %newvdata3 = insertelement <4 x float> %newvdata2, float 0.0, i32 2
+  %newvdata4 = insertelement <4 x float> %newvdata3, float 0.0, i32 3
+  call void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float> %newvdata4, i32 15, i32 %s, <8 x i32> %rsrc, i32 0, i32 0)
+  ret void
+}
+
+define amdgpu_ps void @image_store_mip_1d_store_insert_zeros_at_end(<8 x i32> inreg %rsrc, float %vdata1, float %vdata2, i32 %s, i32 %mip) #0 {
+; GFX11-LABEL: @image_store_mip_1d_store_insert_zeros_at_end(
+; GFX11-NEXT:    [[TMP1:%.*]] = insertelement <3 x float> <float 0.000000e+00, float poison, float poison>, float [[VDATA1:%.*]], i64 1
+; GFX11-NEXT:    [[TMP2:%.*]] = insertelement <3 x float> [[TMP1]], float [[VDATA2:%.*]], i64 2
+; GFX11-NEXT:    call void @llvm.amdgcn.image.store.mip.1d.v3f32.i32(<3 x float> [[TMP2]], i32 7, i32 [[S:%.*]], i32 [[MIP:%.*]], <8 x i32> [[RSRC:%.*]], i32 0, i32 0)
+; GFX11-NEXT:    ret void
+;
+  %newvdata1 = insertelement <4 x float> undef, float 0.0, i32 0
+  %newvdata2 = insertelement <4 x float> %newvdata1, float %vdata1, i32 1
+  %newvdata3 = insertelement <4 x float> %newvdata2, float %vdata2, i32 2
+  %newvdata4 = insertelement <4 x float> %newvdata3, float 0.0, i32 3
+  call void @llvm.amdgcn.image.store.mip.1d.v4f32.i32(<4 x float> %newvdata4, i32 7, i32 %s, i32 %mip, <8 x i32> %rsrc, i32 0, i32 0)
+  ret void
+}
+
+define amdgpu_ps void @buffer_store_insert_zeros_at_end(<4 x i32> inreg %a, float %vdata1, i32 %b) {
+; GFX11-LABEL: @buffer_store_insert_zeros_at_end(
+; GFX11-NEXT:    [[TMP1:%.*]] = insertelement <2 x float> undef, float [[VDATA1:%.*]], i64 0
+; GFX11-NEXT:    [[TMP2:%.*]] = shufflevector <2 x float> [[TMP1]], <2 x float> poison, <2 x i32> zeroinitializer
+; GFX11-NEXT:    call void @llvm.amdgcn.buffer.store.v2f32(<2 x float> [[TMP2]], <4 x i32> [[A:%.*]], i32 [[B:%.*]], i32 0, i1 false, i1 false)
+; GFX11-NEXT:    ret void
+;
+  %newvdata1 = insertelement <4 x float> undef, float %vdata1, i32 0
+  %newvdata2 = insertelement <4 x float> %newvdata1, float %vdata1, i32 1
+  %newvdata3 = insertelement <4 x float> %newvdata2, float 0.0, i32 2
+  %newvdata4 = insertelement <4 x float> %newvdata3, float 0.0, i32 3
+  call void @llvm.amdgcn.buffer.store.v4f32(<4 x float> %newvdata4, <4 x i32> %a, i32 %b, i32 0, i1 0, i1 0)
+  ret void
+}
+
+define amdgpu_ps void @struct_buffer_store_insert_zeros(<4 x i32> inreg %a, float %vdata1, i32 %b) {
+; GFX11-LABEL: @struct_buffer_store_insert_zeros(
+; GFX11-NEXT:    [[TMP1:%.*]] = insertelement <3 x float> <float poison, float 0.000000e+00, float poison>, float [[VDATA1:%.*]], i64 0
+; GFX11-NEXT:    [[TMP2:%.*]] = insertelement <3 x float> [[TMP1]], float [[VDATA1]], i64 2
+; GFX11-NEXT:    call void @llvm.amdgcn.struct.buffer.store.v3f32(<3 x float> [[TMP2]], <4 x i32> [[A:%.*]], i32 [[B:%.*]], i32 0, i32 42, i32 0)
+; GFX11-NEXT:    ret void
+;
+  %newvdata1 = insertelement <4 x float> undef, float %vdata1, i32 0
+  %newvdata2 = insertelement <4 x float> %newvdata1, float 0.0, i32 1
+  %newvdata3 = insertelement <4 x float> %newvdata2, float %vdata1, i32 2
+  %newvdata4 = insertelement <4 x float> %newvdata3, float 0.0, i32 3
+  call void @llvm.amdgcn.struct.buffer.store.v4f32(<4 x float> %newvdata4, <4 x i32> %a, i32 %b, i32 0, i32 42, i32 0)
+  ret void
+}
+
+define amdgpu_ps void @struct_tbuffer_store_insert_zeros_at_beginning(<4 x i32> inreg %a, float %vdata1, i32 %b) {
+; GFX11-LABEL: @struct_tbuffer_store_insert_zeros_at_beginning(
+; GFX11-NEXT:    [[NEWVDATA4:%.*]] = insertelement <4 x float> <float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float poison>, float [[VDATA1:%.*]], i64 3
+; GFX11-NEXT:    call void @llvm.amdgcn.struct.tbuffer.store.v4f32(<4 x float> [[NEWVDATA4]], <4 x i32> [[A:%.*]], i32 [[B:%.*]], i32 0, i32 42, i32 0, i32 15)
+; GFX11-NEXT:    ret void
+;
+  %newvdata1 = insertelement <4 x float> undef, float 0.0, i32 0
+  %newvdata2 = insertelement <4 x float> %newvdata1, float 0.0, i32 1
+  %newvdata3 = insertelement <4 x float> %newvdata2, float 0.0, i32 2
+  %newvdata4 = insertelement <4 x float> %newvdata3, float %vdata1, i32 3
+  call void @llvm.amdgcn.struct.tbuffer.store.v4f32(<4 x float> %newvdata4, <4 x i32> %a, i32 %b, i32 0, i32 42, i32 0, i32 15)
+  ret void
+}
+
+declare void @llvm.amdgcn.raw.buffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i32) #2
+declare void @llvm.amdgcn.raw.buffer.store.format.v4f32(<4 x float>, <4 x i32>, i32, i32, i32) #2
+declare void @llvm.amdgcn.buffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i1, i1) #0
+declare void @llvm.amdgcn.buffer.store.format.v4f32(<4 x float>, <4 x i32>, i32, i32, i1, i1) #1
+declare void @llvm.amdgcn.struct.buffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i32, i32) #2
+declare void @llvm.amdgcn.struct.buffer.store.format.v4f32(<4 x float>, <4 x i32>, i32, i32, i32, i32) #2
+declare void @llvm.amdgcn.struct.tbuffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i32, i32, i32) #0
+declare void @llvm.amdgcn.raw.tbuffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i32, i32) #0
+declare void @llvm.amdgcn.tbuffer.store.v4f32(<4 x float>, <4 x i32>, i32, i32, i32, i32, i32, i32, i1, i1) #0
+declare void @llvm.amdgcn.image.store.1d.v4f32.i32(<4 x float>, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.2d.v4f32.i32(<4 x float>, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.3d.v4f32.i32(<4 x float>, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.cube.v4f32.i32(<4 x float>, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.1darray.v4f32.i32(<4 x float>, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.2darray.v4f32.i32(<4 x float>, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.2dmsaa.v4f32.i32(<4 x float>, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.2darraymsaa.v4f32.i32(<4 x float>, i32, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.mip.1d.v4f32.i32(<4 x float>, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.mip.2d.v4f32.i32(<4 x float>, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.mip.3d.v4f32.i32(<4 x float>, i32, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.mip.cube.v4f32.i32(<4 x float>, i32, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.mip.1darray.v4f32.i32(<4 x float>, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+declare void @llvm.amdgcn.image.store.mip.2darray.v4f32.i32(<4 x float>, i32, i32, i32, i32, i32, <8 x i32>, i32, i32) #0
+
+attributes #0 = { nounwind }
+attributes #1 = { nounwind writeonly }
+attributes #2 = { nounwind }