[AMDGPU] Fix lowering a16 image intrinsics

[sebastian-ne]

Summary

scalar_to_vector takes only one argument, not two.
The a16 tests now also check the packing of coordinates into registers.

[arsenm]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.a16.encode.ll

3 ↗

(On Diff #240589)

if this is an encoding test, it should use -show-mc-encoding?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

21

These tests changes seem independent, but I think having the separate scalars has more value for testing the packing code actually works

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.a16.encode.ll

3 ↗

(On Diff #240589)

Maybe the name was chosen uncarefully, this test should ensure that all components are actually converted and packed into one register.

Before this patch, a16 is not working because the y component just gets ignored, the compiler only ensures that the x component is present.
Still, all other a16 tests are passing. So I tried to introduce this test as a regression test.
It should make sure that actually all components are "encoded" into the packed coordinates.

I guess renaming it to llvm.amdgcn.image.a16.pack.ll would make more sense?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

21

Well, before this patch the packing did not work but the tests passed ;)
To test the packing, I added the new test.

nhaehnle said the amdgpu_ps calling convention is not build to handle f16 arguments, so it is not clear if they are packed or not?
The llvm.amdgcn.image.a16.dim.ll test, which takes i16 instead of halfs, also uses packed arguments. It groups all arguments into <2 x i16>s.

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

21

I forgot to add, some testcases later down used e.g. v[2:9] as arguments, using the packed arguments ensures that always v[0:…] is used so the test should not be influenced by other optimizations or changes of the compiler.

[rtaylor]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.a16.encode.ll

3 ↗

(On Diff #240589)

Should we be checking the encoding for all existing tests instead?

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

4–14

What is the point of combining the halves into vectors? I think this makes the test code unnecessarily complex and less readable.

[rtaylor]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

5446–5447

Couldn't this just be a SCALAR_TO_VECTOR to get the 0th element and create the vector? I think Nicolai mentioned this before?

[arsenm]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

21

f16 arguments work. They are not packed

[nhaehnle]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

5446–5447

Yes, please change this.

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.a16.encode.ll

3 ↗

(On Diff #240589)

Yes, rename the file to avoid confusion.

Yes, we should add encoding tests, but that should happen in a separate patch that actually fixes the encoding for gfx1010 :)

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

4–14

Ideally, this test should be changed into one that uses the update_llc_test_checks script; with that, having the inputs packed like this will highlight some inefficiencies in the codegen that we should fix.

[sebastian-ne]

Address comments

• Use scalar_to_vector instead of two vector inserts
• Rename packing test
• Convert tests to use update script

[rtaylor]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.a16.encode.ll

3 ↗

(On Diff #240589)

I'm not sure what this file is doing that the current tests don't do?

; GCN-LABEL: {{^}}sample_2d:
; GCN: image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
define amdgpu_ps <4 x float> @sample_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t) {
main_body:

%v = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f16(i32 15, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
ret <4 x float> %v

}

5–16 ↗

(On Diff #240589)

Do you need all this checking or can you just check the image_sample instruction? If so, it would be cleaner and easier to read and easier to tell what you are checking with just the GFX9: image_sample...

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

4–14

Nicolai, can you be more specific, what inefficiencies?

[sebastian-ne]

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.a16.encode.ll

5–16 ↗

(On Diff #240589)

Testing the image_sample instruction is not enough.
In the previous tests, only the image_sample was tested and the tests passed. However, a16 was not working because packing the f16 arguments into the registers did work only partially.
So the previous tests were incomplete.
This is exactly the part that this test fills. The significant part of this test is not the image_sample, but the conversion and packing of all arguments into the registers.
It tests the part that went wrong before this patch.

Is the intent of this test clear? I can try to explain it more detailed.

With my updated patch, the other tests check more details, so we could also remove this small test. Any opinions on this?

[sebastian-ne]

Remove superfluous test

[rtaylor]

LGTM.

[nhaehnle]

LGTM

[sebastian-ne]

I changed the way the two f16 values get packed.
This way seems to be a lot better for optimization and results in less instructions sometimes.

[nhaehnle]

Yes, that does look better, thanks!

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

SDValue SITargetLowering::lowerImage(SDValue Op,
   if (((VAddrScalarVT == MVT::f16) || (VAddrScalarVT == MVT::i16))) {
     // Illegal to use a16 images
     if (!ST->hasFeature(AMDGPU::FeatureR128A16))
       return Op;
 
     IsA16 = true;
     const MVT VectorVT = VAddrScalarVT == MVT::f16 ? MVT::v2f16 : MVT::v2i16;
     for (unsigned i = AddrIdx; i < (AddrIdx + NumMIVAddrs); ++i) {
-      SDValue AddrLo, AddrHi;
+      SDValue AddrLo;
       // Push back extra arguments.
       if (i < DimIdx) {
         AddrLo = Op.getOperand(i);
       } else {
-        AddrLo = Op.getOperand(i);
+        AddrLo = DAG.getNode(ISD::SCALAR_TO_VECTOR, DL, VectorVT, Op.getOperand(i));
         // Dz/dh, dz/dv and the last odd coord are packed with undef. Also,
         // in 1D, derivatives dx/dh and dx/dv are packed with undef.
         if (((i + 1) >= (AddrIdx + NumMIVAddrs)) ||
             ((NumGradients / 2) % 2 == 1 &&
             (i == DimIdx + (NumGradients / 2) - 1 ||
              i == DimIdx + NumGradients - 1))) {
-          AddrHi = DAG.getUNDEF(MVT::f16);
         } else {
-          AddrHi = Op.getOperand(i + 1);
           i++;
+          AddrLo = DAG.getNode(ISD::INSERT_VECTOR_ELT, DL, VectorVT, AddrLo,
+                            Op.getOperand(i), DAG.getConstant(1, DL, MVT::i32));
         }
-        AddrLo = DAG.getNode(ISD::SCALAR_TO_VECTOR, DL, VectorVT,
-                             {AddrLo, AddrHi});
         AddrLo = DAG.getBitcast(MVT::i32, AddrLo);
       }
       VAddrs.push_back(AddrLo);
     }
   } else {
     for (unsigned i = 0; i < NumMIVAddrs; ++i)
       VAddrs.push_back(Op.getOperand(AddrIdx + i));
   }
 
   // If the register allocator cannot place the address registers contiguously

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.gather4.a16.dim.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN %s
 
-; GCN-LABEL: {{^}}gather4_2d:
-; GCN: image_gather4 v[0:3], v0, s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t) {
+; GCN-LABEL: gather4_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4 v[0:3], v0, s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.2d.v4f32.f16(i32 1, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_cube:
-; GCN: image_gather4 v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16 da{{$}}
 define amdgpu_ps <4 x float> @gather4_cube(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %face) {
+; GCN-LABEL: gather4_cube:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4 v[0:3], v[1:2], s[0:7], s[8:11] dmask:0x1 a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.cube.v4f32.f16(i32 1, half %s, half %t, half %face, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_2darray:
-; GCN: image_gather4 v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16 da{{$}}
 define amdgpu_ps <4 x float> @gather4_2darray(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %slice) {
+; GCN-LABEL: gather4_2darray:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4 v[0:3], v[1:2], s[0:7], s[8:11] dmask:0x1 a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.2darray.v4f32.f16(i32 1, half %s, half %t, half %slice, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_c_2d:
-; GCN: image_gather4_c v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_c_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t) {
+; GCN-LABEL: gather4_c_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_c v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.c.2d.v4f32.f32(i32 1, float %zcompare, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_cl_2d:
-; GCN: image_gather4_cl v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %clamp) {
+; GCN-LABEL: gather4_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_cl v[0:3], v[1:2], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.cl.2d.v4f32.f16(i32 1, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_c_cl_2d:
-; GCN: image_gather4_c_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_c_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t, half %clamp) {
+; GCN-LABEL: gather4_c_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_mov_b32_e32 v5, v3
+; GCN-NEXT:    v_mov_b32_e32 v3, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v4, v2, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_c_cl v[0:3], v[3:6], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.c.cl.2d.v4f32.f32(i32 1, float %zcompare, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_b_2d:
-; GCN: image_gather4_b v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_b_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, half %s, half %t) {
+; GCN-LABEL: gather4_b_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_b v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.b.2d.v4f32.f32.f16(i32 1, float %bias, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_c_b_2d:
-; GCN: image_gather4_c_b v[0:3], v[0:3], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_c_b_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %zcompare, half %s, half %t) {
+; GCN-LABEL: gather4_c_b_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GCN-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_c_b v[0:3], v[0:3], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.c.b.2d.v4f32.f32.f16(i32 1, float %bias, float %zcompare, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_b_cl_2d:
-; GCN: image_gather4_b_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_b_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, half %s, half %t, half %clamp) {
+; GCN-LABEL: gather4_b_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_mov_b32_e32 v5, v3
+; GCN-NEXT:    v_mov_b32_e32 v3, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v4, v2, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_b_cl v[0:3], v[3:6], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.b.cl.2d.v4f32.f32.f16(i32 1, float %bias, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_c_b_cl_2d:
-; GCN: image_gather4_c_b_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_c_b_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %zcompare, half %s, half %t, half %clamp) {
+; GCN-LABEL: gather4_c_b_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_mov_b32_e32 v7, v4
+; GCN-NEXT:    v_mov_b32_e32 v4, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v2
+; GCN-NEXT:    v_mov_b32_e32 v5, v1
+; GCN-NEXT:    v_lshl_or_b32 v6, v3, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_gather4_c_b_cl v[0:3], v[4:7], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.c.b.cl.2d.v4f32.f32.f16(i32 1, float %bias, float %zcompare, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_l_2d:
-; GCN: image_gather4_l v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_l_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %lod) {
+; GCN-LABEL: gather4_l_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    image_gather4_l v[0:3], v[1:2], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.l.2d.v4f32.f16(i32 1, half %s, half %t, half %lod, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_c_l_2d:
-; GCN: image_gather4_c_l v[0:3], v[0:3], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_c_l_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t, half %lod) {
+; GCN-LABEL: gather4_c_l_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v5, v3
+; GCN-NEXT:    v_mov_b32_e32 v3, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v4, v2, 16, v0
+; GCN-NEXT:    image_gather4_c_l v[0:3], v[3:6], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.c.l.2d.v4f32.f32(i32 1, float %zcompare, half %s, half %t, half %lod, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_lz_2d:
-; GCN: image_gather4_lz v[0:3], v0, s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_lz_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t) {
+; GCN-LABEL: gather4_lz_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    image_gather4_lz v[0:3], v0, s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.lz.2d.v4f32.f16(i32 1, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}gather4_c_lz_2d:
-; GCN: image_gather4_c_lz v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16{{$}}
 define amdgpu_ps <4 x float> @gather4_c_lz_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t) {
+; GCN-LABEL: gather4_c_lz_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    image_gather4_c_lz v[0:3], v[0:1], s[0:7], s[8:11] dmask:0x1 a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.gather4.c.lz.2d.v4f32.f32(i32 1, float %zcompare, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
 declare <4 x float> @llvm.amdgcn.image.gather4.2d.v4f32.f16(i32, half, half, <8 x i32>, <4 x i32>, i1, i32, i32) #1
 declare <4 x float> @llvm.amdgcn.image.gather4.cube.v4f32.f16(i32, half, half, half, <8 x i32>, <4 x i32>, i1, i32, i32) #1
 declare <4 x float> @llvm.amdgcn.image.gather4.2darray.v4f32.f16(i32, half, half, half, <8 x i32>, <4 x i32>, i1, i32, i32) #1

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.sample.a16.dim.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefixes=GCN %s
-; GCN: image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s) {
+; GCN-LABEL: sample_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f16(i32 15, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_2d:
-; GCN: image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t) {
+; GCN-LABEL: sample_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f16(i32 15, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_3d:
-; GCN: image_sample v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_3d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %r) {
+; GCN-LABEL: sample_3d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample v[0:3], v[1:2], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.3d.v4f32.f16(i32 15, half %s, half %t, half %r, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cube:
-; GCN: image_sample v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16 da{{$}}
 define amdgpu_ps <4 x float> @sample_cube(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %face) {
+; GCN-LABEL: sample_cube:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample v[0:3], v[1:2], s[0:7], s[8:11] dmask:0xf a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cube.v4f32.f16(i32 15, half %s, half %t, half %face, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_1darray:
-; GCN: image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16 da{{$}}
 define amdgpu_ps <4 x float> @sample_1darray(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %slice) {
+; GCN-LABEL: sample_1darray:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.1darray.v4f32.f16(i32 15, half %s, half %slice, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_2darray:
-; GCN: image_sample v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16 da{{$}}
 define amdgpu_ps <4 x float> @sample_2darray(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %slice) {
+; GCN-LABEL: sample_2darray:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample v[0:3], v[1:2], s[0:7], s[8:11] dmask:0xf a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.2darray.v4f32.f16(i32 15, half %s, half %t, half %slice, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_1d:
-; GCN: image_sample_c v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s) {
+; GCN-LABEL: sample_c_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.1d.v4f32.f16(i32 15, float %zcompare, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_2d:
-; GCN: image_sample_c v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t) {
+; GCN-LABEL: sample_c_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.2d.v4f32.f16(i32 15, float %zcompare, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cl_1d:
-; GCN: image_sample_cl v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %clamp) {
+; GCN-LABEL: sample_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_cl v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cl.1d.v4f32.f16(i32 15, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cl_2d:
-; GCN: image_sample_cl v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_cl v[0:3], v[1:2], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cl.2d.v4f32.f16(i32 15, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_cl_1d:
-; GCN: image_sample_c_cl v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %clamp) {
+; GCN-LABEL: sample_c_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c_cl v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.cl.1d.v4f32.f16(i32 15, float %zcompare, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_cl_2d:
-; GCN: image_sample_c_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_c_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_mov_b32_e32 v5, v3
+; GCN-NEXT:    v_mov_b32_e32 v3, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v4, v2, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c_cl v[0:3], v[3:6], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.cl.2d.v4f32.f16(i32 15, float %zcompare, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_b_1d:
-; GCN: image_sample_b v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_b_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, half %s) {
+; GCN-LABEL: sample_b_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_b v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.b.1d.v4f32.f32.f16(i32 15, float %bias, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_b_2d:
-; GCN: image_sample_b v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_b_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, half %s, half %t) {
+; GCN-LABEL: sample_b_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_b v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.b.2d.v4f32.f32.f16(i32 15, float %bias, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_b_1d:
-; GCN: image_sample_c_b v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_b_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %zcompare, half %s) {
+; GCN-LABEL: sample_c_b_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c_b v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.b.1d.v4f32.f32.f16(i32 15, float %bias, float %zcompare, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_b_2d:
-; GCN: image_sample_c_b v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_b_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %zcompare, half %s, half %t) {
+; GCN-LABEL: sample_c_b_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GCN-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c_b v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.b.2d.v4f32.f32.f16(i32 15, float %bias, float %zcompare, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_b_cl_1d:
-; GCN: image_sample_b_cl v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_b_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, half %s, half %clamp) {
+; GCN-LABEL: sample_b_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_b_cl v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.b.cl.1d.v4f32.f32.f16(i32 15, float %bias, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_b_cl_2d:
-; GCN: image_sample_b_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_b_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_b_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_mov_b32_e32 v5, v3
+; GCN-NEXT:    v_mov_b32_e32 v3, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v4, v2, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_b_cl v[0:3], v[3:6], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.b.cl.2d.v4f32.f32.f16(i32 15, float %bias, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_b_cl_1d:
-; GCN: image_sample_c_b_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_b_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %zcompare, half %s, half %clamp) {
+; GCN-LABEL: sample_c_b_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GCN-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c_b_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.b.cl.1d.v4f32.f32.f16(i32 15, float %bias, float %zcompare, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_b_cl_2d:
-; GCN: image_sample_c_b_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_b_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %bias, float %zcompare, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_c_b_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    s_mov_b64 s[12:13], exec
+; GCN-NEXT:    s_wqm_b64 exec, exec
+; GCN-NEXT:    v_mov_b32_e32 v7, v4
+; GCN-NEXT:    v_mov_b32_e32 v4, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v2
+; GCN-NEXT:    v_mov_b32_e32 v5, v1
+; GCN-NEXT:    v_lshl_or_b32 v6, v3, 16, v0
+; GCN-NEXT:    s_and_b64 exec, exec, s[12:13]
+; GCN-NEXT:    image_sample_c_b_cl v[0:3], v[4:7], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.b.cl.2d.v4f32.f32.f16(i32 15, float %bias, float %zcompare, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_d_1d:
-; GCN: image_sample_d v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_d_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dsdv, half %s) {
+; GCN-LABEL: sample_d_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    image_sample_d v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.d.1d.v4f32.f16.f16(i32 15, half %dsdh, half %dsdv, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_d_2d:
-; GCN: image_sample_d v[0:3], v[1:4], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_d_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t) {
+; GCN-LABEL: sample_d_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v6, 0xffff
+; GCN-NEXT:    v_and_b32_e32 v4, v6, v4
+; GCN-NEXT:    v_and_b32_e32 v2, v6, v2
+; GCN-NEXT:    v_and_b32_e32 v0, v6, v0
+; GCN-NEXT:    v_lshl_or_b32 v3, v3, 16, v2
+; GCN-NEXT:    v_lshl_or_b32 v4, v5, 16, v4
+; GCN-NEXT:    v_lshl_or_b32 v2, v1, 16, v0
+; GCN-NEXT:    image_sample_d v[0:3], v[2:5], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.d.2d.v4f32.f16.f16(i32 15, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABAL: {{^}}sample_d_3d:
-; GCN: image_sample_d v[0:3], v[2:9], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_d_3d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dtdh, half %drdh, half %dsdv, half %dtdv, half %drdv, half %s, half %t, half %r) {
+; GCN-LABEL: sample_d_3d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v12, v8
+; GCN-NEXT:    v_mov_b32_e32 v8, v2
+; GCN-NEXT:    v_mov_b32_e32 v2, 0xffff
+; GCN-NEXT:    v_mov_b32_e32 v10, v5
+; GCN-NEXT:    v_and_b32_e32 v5, v2, v6
+; GCN-NEXT:    v_and_b32_e32 v3, v2, v3
+; GCN-NEXT:    v_and_b32_e32 v0, v2, v0
+; GCN-NEXT:    v_lshl_or_b32 v11, v7, 16, v5
+; GCN-NEXT:    v_lshl_or_b32 v9, v4, 16, v3
+; GCN-NEXT:    v_lshl_or_b32 v7, v1, 16, v0
+; GCN-NEXT:    image_sample_d v[0:3], v[7:14], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.d.3d.v4f32.f16.f16(i32 15, half %dsdh, half %dtdh, half %drdh, half %dsdv, half %dtdv, half %drdv, half %s, half %t, half %r, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_d_1d:
-; GCN: image_sample_c_d v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_d_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dsdv, half %s) {
+; GCN-LABEL: sample_c_d_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    image_sample_c_d v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.d.1d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dsdv, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_d_2d:
-; GCN: image_sample_c_d v[0:3], v[1:4], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_d_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t) {
+; GCN-LABEL: sample_c_d_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v9, 0xffff
+; GCN-NEXT:    v_mov_b32_e32 v8, v2
+; GCN-NEXT:    v_mov_b32_e32 v7, v3
+; GCN-NEXT:    v_and_b32_e32 v2, v9, v5
+; GCN-NEXT:    v_and_b32_e32 v1, v9, v1
+; GCN-NEXT:    v_lshl_or_b32 v3, v6, 16, v2
+; GCN-NEXT:    v_and_b32_e32 v2, v9, v7
+; GCN-NEXT:    v_lshl_or_b32 v2, v4, 16, v2
+; GCN-NEXT:    v_lshl_or_b32 v1, v8, 16, v1
+; GCN-NEXT:    image_sample_c_d v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.d.2d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_d_cl_1d:
-; GCN: image_sample_d_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_d_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dsdv, half %s, half %clamp) {
+; GCN-LABEL: sample_d_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GCN-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
+; GCN-NEXT:    image_sample_d_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.d.cl.1d.v4f32.f16.f16(i32 15, half %dsdh, half %dsdv, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_d_cl_2d:
-; GCN: image_sample_d_cl v[0:3], v[2:5], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_d_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_d_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v7, 0xffff
+; GCN-NEXT:    v_and_b32_e32 v4, v7, v4
+; GCN-NEXT:    v_and_b32_e32 v2, v7, v2
+; GCN-NEXT:    v_and_b32_e32 v0, v7, v0
+; GCN-NEXT:    v_lshl_or_b32 v5, v5, 16, v4
+; GCN-NEXT:    v_lshl_or_b32 v4, v3, 16, v2
+; GCN-NEXT:    v_lshl_or_b32 v3, v1, 16, v0
+; GCN-NEXT:    image_sample_d_cl v[0:3], v[3:6], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.d.cl.2d.v4f32.f16.f16(i32 15, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_d_cl_1d:
-; GCN: image_sample_c_d_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_d_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dsdv, half %s, half %clamp) {
+; GCN-LABEL: sample_c_d_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v3, 0xffff, v3
+; GCN-NEXT:    v_lshl_or_b32 v3, v4, 16, v3
+; GCN-NEXT:    image_sample_c_d_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.d.cl.1d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dsdv, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_d_cl_2d:
-; GCN: image_sample_c_d_cl v[0:3], v[2:9], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_d_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_c_d_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v11, v7
+; GCN-NEXT:    v_mov_b32_e32 v7, v0
+; GCN-NEXT:    v_mov_b32_e32 v0, 0xffff
+; GCN-NEXT:    v_and_b32_e32 v5, v0, v5
+; GCN-NEXT:    v_and_b32_e32 v3, v0, v3
+; GCN-NEXT:    v_and_b32_e32 v0, v0, v1
+; GCN-NEXT:    v_lshl_or_b32 v10, v6, 16, v5
+; GCN-NEXT:    v_lshl_or_b32 v9, v4, 16, v3
+; GCN-NEXT:    v_lshl_or_b32 v8, v2, 16, v0
+; GCN-NEXT:    image_sample_c_d_cl v[0:3], v[7:14], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.d.cl.2d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cd_1d:
-; GCN: image_sample_cd v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_cd_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dsdv, half %s) {
+; GCN-LABEL: sample_cd_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    image_sample_cd v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cd.1d.v4f32.f16.f16(i32 15, half %dsdh, half %dsdv, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cd_2d:
-; GCN: image_sample_cd v[0:3], v[1:4], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_cd_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t) {
+; GCN-LABEL: sample_cd_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v6, 0xffff
+; GCN-NEXT:    v_and_b32_e32 v4, v6, v4
+; GCN-NEXT:    v_and_b32_e32 v2, v6, v2
+; GCN-NEXT:    v_and_b32_e32 v0, v6, v0
+; GCN-NEXT:    v_lshl_or_b32 v3, v3, 16, v2
+; GCN-NEXT:    v_lshl_or_b32 v4, v5, 16, v4
+; GCN-NEXT:    v_lshl_or_b32 v2, v1, 16, v0
+; GCN-NEXT:    image_sample_cd v[0:3], v[2:5], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cd.2d.v4f32.f16.f16(i32 15, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_cd_1d:
-; GCN: image_sample_c_cd v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_cd_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dsdv, half %s) {
+; GCN-LABEL: sample_c_cd_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    image_sample_c_cd v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.cd.1d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dsdv, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_cd_2d:
-; GCN: image_sample_c_cd v[0:3], v[1:4], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_cd_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t) {
+; GCN-LABEL: sample_c_cd_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v9, 0xffff
+; GCN-NEXT:    v_mov_b32_e32 v8, v2
+; GCN-NEXT:    v_mov_b32_e32 v7, v3
+; GCN-NEXT:    v_and_b32_e32 v2, v9, v5
+; GCN-NEXT:    v_and_b32_e32 v1, v9, v1
+; GCN-NEXT:    v_lshl_or_b32 v3, v6, 16, v2
+; GCN-NEXT:    v_and_b32_e32 v2, v9, v7
+; GCN-NEXT:    v_lshl_or_b32 v2, v4, 16, v2
+; GCN-NEXT:    v_lshl_or_b32 v1, v8, 16, v1
+; GCN-NEXT:    image_sample_c_cd v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.cd.2d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cd_cl_1d:
-; GCN: image_sample_cd_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_cd_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dsdv, half %s, half %clamp) {
+; GCN-LABEL: sample_cd_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GCN-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
+; GCN-NEXT:    image_sample_cd_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cd.cl.1d.v4f32.f16.f16(i32 15, half %dsdh, half %dsdv, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_cd_cl_2d:
-; GCN: image_sample_cd_cl v[0:3], v[2:5], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_cd_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_cd_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v7, 0xffff
+; GCN-NEXT:    v_and_b32_e32 v4, v7, v4
+; GCN-NEXT:    v_and_b32_e32 v2, v7, v2
+; GCN-NEXT:    v_and_b32_e32 v0, v7, v0
+; GCN-NEXT:    v_lshl_or_b32 v5, v5, 16, v4
+; GCN-NEXT:    v_lshl_or_b32 v4, v3, 16, v2
+; GCN-NEXT:    v_lshl_or_b32 v3, v1, 16, v0
+; GCN-NEXT:    image_sample_cd_cl v[0:3], v[3:6], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.cd.cl.2d.v4f32.f16.f16(i32 15, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_cd_cl_1d:
-; GCN: image_sample_c_cd_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_cd_cl_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dsdv, half %s, half %clamp) {
+; GCN-LABEL: sample_c_cd_cl_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v3, 0xffff, v3
+; GCN-NEXT:    v_lshl_or_b32 v3, v4, 16, v3
+; GCN-NEXT:    image_sample_c_cd_cl v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.cd.cl.1d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dsdv, half %s, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_cd_cl_2d:
-; GCN: image_sample_c_cd_cl v[0:3], v[2:9], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_cd_cl_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp) {
+; GCN-LABEL: sample_c_cd_cl_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v11, v7
+; GCN-NEXT:    v_mov_b32_e32 v7, v0
+; GCN-NEXT:    v_mov_b32_e32 v0, 0xffff
+; GCN-NEXT:    v_and_b32_e32 v5, v0, v5
+; GCN-NEXT:    v_and_b32_e32 v3, v0, v3
+; GCN-NEXT:    v_and_b32_e32 v0, v0, v1
+; GCN-NEXT:    v_lshl_or_b32 v10, v6, 16, v5
+; GCN-NEXT:    v_lshl_or_b32 v9, v4, 16, v3
+; GCN-NEXT:    v_lshl_or_b32 v8, v2, 16, v0
+; GCN-NEXT:    image_sample_c_cd_cl v[0:3], v[7:14], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.cd.cl.2d.v4f32.f32.f16(i32 15, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %clamp, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_l_1d:
-; GCN: image_sample_l v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_l_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %lod) {
+; GCN-LABEL: sample_l_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    image_sample_l v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.l.1d.v4f32.f16(i32 15, half %s, half %lod, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_l_2d:
-; GCN: image_sample_l v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_l_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t, half %lod) {
+; GCN-LABEL: sample_l_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v1, v1, 16, v0
+; GCN-NEXT:    image_sample_l v[0:3], v[1:2], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.l.2d.v4f32.f16(i32 15, half %s, half %t, half %lod, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_l_1d:
-; GCN: image_sample_c_l v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_l_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %lod) {
+; GCN-LABEL: sample_c_l_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    image_sample_c_l v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.l.1d.v4f32.f16(i32 15, float %zcompare, half %s, half %lod, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_l_2d:
-; GCN: image_sample_c_l v[0:3], v[0:3], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_l_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t, half %lod) {
+; GCN-LABEL: sample_c_l_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v5, v3
+; GCN-NEXT:    v_mov_b32_e32 v3, v0
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v4, v2, 16, v0
+; GCN-NEXT:    image_sample_c_l v[0:3], v[3:6], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.l.2d.v4f32.f16(i32 15, float %zcompare, half %s, half %t, half %lod, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_lz_1d:
-; GCN: image_sample_lz v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_lz_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s) {
+; GCN-LABEL: sample_lz_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    image_sample_lz v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.lz.1d.v4f32.f16(i32 15, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_lz_2d:
-; GCN: image_sample_lz v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_lz_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, half %s, half %t) {
+; GCN-LABEL: sample_lz_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GCN-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
+; GCN-NEXT:    image_sample_lz v[0:3], v0, s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.lz.2d.v4f32.f16(i32 15, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_lz_1d:
-; GCN: image_sample_c_lz v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_lz_1d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s) {
+; GCN-LABEL: sample_c_lz_1d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    image_sample_c_lz v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.lz.1d.v4f32.f16(i32 15, float %zcompare, half %s, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_lz_2d:
-; GCN: image_sample_c_lz v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16{{$}}
 define amdgpu_ps <4 x float> @sample_c_lz_2d(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %zcompare, half %s, half %t) {
+; GCN-LABEL: sample_c_lz_2d:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GCN-NEXT:    v_lshl_or_b32 v1, v2, 16, v1
+; GCN-NEXT:    image_sample_c_lz v[0:3], v[0:1], s[0:7], s[8:11] dmask:0xf a16
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <4 x float> @llvm.amdgcn.image.sample.c.lz.2d.v4f32.f16(i32 15, float %zcompare, half %s, half %t, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <4 x float> %v
 }
 
-; GCN-LABEL: {{^}}sample_c_d_o_2darray_V1:
-; GCN: image_sample_c_d_o v0, v[2:9], s[0:7], s[8:11] dmask:0x4 a16 da{{$}}
 define amdgpu_ps float @sample_c_d_o_2darray_V1(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, i32 %offset, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %slice) {
+; GCN-LABEL: sample_c_d_o_2darray_V1:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v13, v8
+; GCN-NEXT:    v_mov_b32_e32 v8, v0
+; GCN-NEXT:    v_mov_b32_e32 v0, 0xffff
+; GCN-NEXT:    v_mov_b32_e32 v9, v1
+; GCN-NEXT:    v_and_b32_e32 v1, v0, v6
+; GCN-NEXT:    v_lshl_or_b32 v12, v7, 16, v1
+; GCN-NEXT:    v_and_b32_e32 v1, v0, v4
+; GCN-NEXT:    v_and_b32_e32 v0, v0, v2
+; GCN-NEXT:    v_lshl_or_b32 v11, v5, 16, v1
+; GCN-NEXT:    v_lshl_or_b32 v10, v3, 16, v0
+; GCN-NEXT:    image_sample_c_d_o v0, v[8:15], s[0:7], s[8:11] dmask:0x4 a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call float @llvm.amdgcn.image.sample.c.d.o.2darray.f32.f16.f16(i32 4, i32 %offset, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %slice, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret float %v
 }
 
-; GCN-LABEL: {{^}}sample_c_d_o_2darray_V2:
-; GCN: image_sample_c_d_o v[0:1], v[2:9], s[0:7], s[8:11] dmask:0x6 a16 da{{$}}
 define amdgpu_ps <2 x float> @sample_c_d_o_2darray_V2(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, i32 %offset, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %slice) {
+; GCN-LABEL: sample_c_d_o_2darray_V2:
+; GCN:       ; %bb.0: ; %main_body
+; GCN-NEXT:    v_mov_b32_e32 v13, v8
+; GCN-NEXT:    v_mov_b32_e32 v8, v0
+; GCN-NEXT:    v_mov_b32_e32 v0, 0xffff
+; GCN-NEXT:    v_mov_b32_e32 v9, v1
+; GCN-NEXT:    v_and_b32_e32 v1, v0, v6
+; GCN-NEXT:    v_lshl_or_b32 v12, v7, 16, v1
+; GCN-NEXT:    v_and_b32_e32 v1, v0, v4
+; GCN-NEXT:    v_and_b32_e32 v0, v0, v2
+; GCN-NEXT:    v_lshl_or_b32 v11, v5, 16, v1
+; GCN-NEXT:    v_lshl_or_b32 v10, v3, 16, v0
+; GCN-NEXT:    image_sample_c_d_o v[0:1], v[8:15], s[0:7], s[8:11] dmask:0x6 a16 da
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    ; return to shader part epilog
 main_body:
   %v = call <2 x float> @llvm.amdgcn.image.sample.c.d.o.2darray.v2f32.f32.f16(i32 6, i32 %offset, float %zcompare, half %dsdh, half %dtdh, half %dsdv, half %dtdv, half %s, half %t, half %slice, <8 x i32> %rsrc, <4 x i32> %samp, i1 0, i32 0, i32 0)
   ret <2 x float> %v
 }
 
 declare <4 x float> @llvm.amdgcn.image.sample.1d.v4f32.f16(i32, half, <8 x i32>, <4 x i32>, i1, i32, i32) #1
 declare <8 x float> @llvm.amdgcn.image.sample.1d.v8f32.f16(i32, half, <8 x i32>, <4 x i32>, i1, i32, i32) #1
 declare <4 x float> @llvm.amdgcn.image.sample.2d.v4f32.f16(i32, half, half, <8 x i32>, <4 x i32>, i1, i32, i32) #1