AMDGPU/SI: Make i16 a legal type for VI subtargets

[• tstellarAMD]

Summary

Patch by: Wei Ding

[arsenm]

All of these tests should go in the existing test files and be run on the other subtargets as well.

I also expected way more load and store test additions. We need to have extload tests for i8->i16, i16->i32, i16->i64, some of which probably exist already.

This patch should limit itself to the basic set of required operations on i16: load and store, add, sub, bitshifts, and conversions. Optimizations like min/max should come later.

lib/Target/AMDGPU/VIInstructions.td

170

This pattern is necessary. I believe I had a test for this in my original patch

178–195

Dead code should be removed

199

Should follow camel case naming convention

201

No spaces around the :s

204–205

Should be indented like other Pats in the file

210

This is incorrect if this is a scalar zext, which currently doesn't happen because there are no scalar i16 instructions (although we may want pseudos for these). To be consistent, this should use S_MOV_B32 to materialize the 0

240–241

These should be using the signed min/max. I also think the min/max matching should be a separate patch

246

These should be done in a separate patch

249–267

Dead code should be removed. These instruction's dont exist. However, tests should be added to the rotl/rotr/bswap/ctlz/cttz to make sure these are properly expanded when i16 is added as legal since most operations by default are assumed to be legal if the type is

[arsenm]

This is also missing the required register info changes and SIISelLowering changes

[arsenm]

This is also missing the required register info changes and SIISelLowering changes

[wdng]

1. global-extload-i16.ll contains extload tests for i16->i32, i16->i64. Test for load/store i8->i16 is added into file global-extload-i8.ll
2. Dead codes have been removed.
3. Fixed indentation issues.
4. Fixed scalar zext, used S_MOV_B32 to materialize the 0
5. Added tests for shl, sra, sub
6. Removed min/max, which will be committed into another separate patch.

[• tstellarAMD]

lib/Target/AMDGPU/SIISelLowering.cpp

264–268 ↗

(On Diff #51139)

Are these really necessary? I thought making a type legal marked these operations legal by default.

270–276 ↗

(On Diff #51139)

Same with these too. Are they really necessary?

1743–1756 ↗

(On Diff #51139)

Why does i16 needs special handling here. These seems to be nearly identical to the block of code directly below.

2022–2031 ↗

(On Diff #51139)

We do we need to custom lower i16 stores? Can't we just mark then as promote?

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

270–276 ↗

(On Diff #51139)

min/max need to be explicitly made legal, but that should be a separate patch. setcc should be promote for now until those are added later

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

65 ↗

(On Diff #51139)

I did this already, so this should check Subtarget->has16BitInsts()

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

2022–2031 ↗

(On Diff #51139)

Load/store promote expects an equal size type for a bitcast promote. This is the same problem that i1 has, so it should follow that example

[wdng]

Please temporarily ignore code changes in function LowerFDV32(). I will finish the code development for fp32 div in next patch.

[• tstellarAMD]

Can you send a new patch without the FP32 Div change, so I can commit it.

[wdng]

Revert DFIV32 code changes. This patch contains i16 implementation only based on Tom / Matt comments.

[• tstellarAMD]

I tried to apply this patch, but I get a lot of failing lit tests in test/CodeGen/AMDGPU. Do all the tests in this directory pass for you?

lib/Target/AMDGPU/SIISelLowering.cpp

65 ↗

(On Diff #52514)

Do we still need this comment?

1740 ↗

(On Diff #52514)

Does this need to be removed?

1748 ↗

(On Diff #52514)

Coding style. Variable names should start with a captial.

[• tstellarAMD]

Rebase on top of master.

[kzhuravl]

LGTM overall with minor fixes:

lib/Target/AMDGPU/AMDGPUISelLowering.cpp

600 ↗

(On Diff #74904)

Subtarget->has16BitInsts()

611 ↗

(On Diff #74904)

Subtarget->has16BitInsts()

2369 ↗

(On Diff #74904)

Space after //. Capitalize.

2370 ↗

(On Diff #74904)

Subtarget->has16BitInsts()

lib/Target/AMDGPU/SIISelLowering.cpp

37 ↗

(On Diff #74904)

Alphabetize

82 ↗

(On Diff #74904)

I do not think we need this comment

83 ↗

(On Diff #74904)

Subtarget->has16BitInsts()

229 ↗

(On Diff #74904)

Subtarget->has16BitInsts()

273 ↗

(On Diff #74904)

Detabify

275 ↗

(On Diff #74904)

Detabify

277 ↗

(On Diff #74904)

Detabify

279 ↗

(On Diff #74904)

Detabify

280 ↗

(On Diff #74904)

Remove extra new line

[arsenm]

lib/Target/AMDGPU/BUFInstructions.td

962–963 ↗

(On Diff #74904)

Originally I was avoiding these by promoting the load return type and truncating. Is this the same or is there an advantage from known bits types of things knowing it is really a 32-bit extload?

lib/Target/AMDGPU/SIISelLowering.cpp

1916 ↗

(On Diff #74904)

Why is this part of the patch? This looks unrelated

3428–3429 ↗

(On Diff #74904)

Define on same line

[• tstellarAMD]

Address review comments.

[• tstellarAMD]

lib/Target/AMDGPU/BUFInstructions.td

962–963 ↗

(On Diff #74904)

The main problem is that extload must be legal, so it's hard to custom lower all extloads. I attempted to do this, and I think it would require a lot of work in the backend and also the legalizer to make this work. I'm not sure it's worth the effort at this point.

[arsenm]

lib/Target/AMDGPU/BUFInstructions.td

960 ↗

(On Diff #76087)

This can be refined to a has 16-bit predicate

lib/Target/AMDGPU/VOP1Instructions.td

614–618 ↗

(On Diff #76087)

Commented out code

lib/Target/AMDGPU/VOP2Instructions.td

348 ↗

(On Diff #76087)

This should only need to be around the actual defs, not the multiclasses

355 ↗

(On Diff #76087)

I don't think you need to repeat the type in the output here

lib/Target/AMDGPU/VOP3Instructions.td

232 ↗

(On Diff #76087)

Line wrapping

[• tstellarAMD]

Address review comments.

[arsenm]

LGTM

lib/Target/AMDGPU/VIInstructions.td

 >;
 
 class ZExt_i16_i1_Pat <SDNode ext> : Pat <
   (i16 (ext i1:$src)),
   (V_CNDMASK_B32_e64 (i32 0), (i32 1), $src)
 >;
 
 def : ZExt_i16_i1_Pat<zext>;
-
-
+//def : ZExt_i16_i1_Pat<anyext>;
 
 def : Pat <
   (i16 (select i1:$src0, i16:$src1, i16:$src2)),
   (V_CNDMASK_B32_e64 $src2, $src1, $src0)
 >;
 
 // 16-bit instructions produce a 0 result in the high 16-bits.
+/*
 def : Pat <
   (add i16:$src0, i16:$src1),
   (V_ADD_U16_e32 $src0, $src1)
 >;
 
 def : Pat <
   (i32 (zext (add i16:$src0, i16:$src1))),
   (V_ADD_U16_e32 i16:$src0, i16:$src1)
 >;
 
 def : Pat <
   (i64 (zext (add i16:$src0, i16:$src1))),
    (REG_SEQUENCE VReg_64,
      (V_ADD_U16_e32 i16:$src0, i16:$src1), sub0,
      (V_MOV_B32_e32 (i32 0)), sub1)
 >;
+*/
+//----------------------------------------------------------------------
+// Note: 16-bit instructions produce a 0 result in the high 16-bits.
+
+multiclass i16_arithmetic_pats <SDPatternOperator op, Instruction inst> {
+
+def : Pat<(op i16 : $src0, i16 : $src1),
+            (inst i16:$src0, i16:$src1)>;
+
+def : Pat<(i32 (zext (op i16:$src0, i16:$src1))),
+          (inst i16:$src0, i16:$src1)>;
+
+def : Pat<(i64 (zext (op i16:$src0, i16:$src1))),
+           (REG_SEQUENCE VReg_64,
+           (inst i16:$src0, i16:$src1), sub0,
+           (V_MOV_B32_e32 (i32 0)), sub1)>;
+}
+
+//////////////////////////////////////////////////////////////////////////////////
+multiclass i16_bits_op_pats <SDPatternOperator op, Instruction inst> {
+
+def : Pat<(op i16:$src0, i32:$src1),
+       (inst i16:$src0, i32:$src1)
+>;
+
+def : Pat <(i32 (zext (op i16:$src0, i32:$src1))),
+          (inst i16:$src0, i32:$src1)
+>;
+
+def : Pat <(i64 (zext (op i16:$src0, i32:$src1))),
+           (REG_SEQUENCE VReg_64,
+           (inst i16:$src0, i32:$src1), sub0,
+           (V_MOV_B32_e32 (i32 0)), sub1)
+>;
+}
+//////////////////////////////////////////////////////////////////////////////////
+
+
+defm : i16_arithmetic_pats<add, V_ADD_U16_e32>;
+defm : i16_arithmetic_pats<sub, V_SUB_U16_e32>;
+
+defm : i16_bits_op_pats<shl, V_LSHLREV_B16_e32>;
+defm : i16_bits_op_pats<srl, V_LSHRREV_B16_e32>;
+defm : i16_bits_op_pats<sra, V_ASHRREV_B16_e32>;
+
+defm : i16_arithmetic_pats<smin, V_MIN_U16_e32>;
+defm : i16_arithmetic_pats<smax, V_MAX_U16_e32>;
+defm : i16_arithmetic_pats<umin, V_MIN_U16_e32>;
+defm : i16_arithmetic_pats<umax, V_MAX_U16_e32>;
+
+
+//Fixme: do we need setcc instruction?
+//defm : i16_arithmetic_pats<setcc, V_ASHRREV_B16_e32>;
+
+//Looks like we don't need the and, or, xor instructions for i16
+//defm : i16_arithmetic_pats<and, V_AND_B16_e32>;
+//defm : i16_arithmetic_pats<or, V_OR_B16_e32>;
+//defm : i16_arithmetic_pats<xor, V_XOR_B16_e32>;
+
+//Fixme: do we need rotr, rotl instructions?
+//defm : i16_arithmetic_pats<rotr, V_ROTR_B32_e32>;  //???
+//defm : i16_arithmetic_pats<rotl, V_ROTL_B32_e32>;  //???
+
+//defm : i16_arithmetic_pats<sdiv, V_SDIV_B32_e32>;  // ???
+//defm : i16_arithmetic_pats<udiv, V_UDIV_B32_e32>; // ???
+
+//defm : i16_arithmetic_pats<srem, V_SREM_B32_e32>; // ???
+//defm : i16_arithmetic_pats<urem, V_UREM_B32_e32>; // ???
+defm : i16_arithmetic_pats<mul, V_MUL_F16_e32>; //???
+
+//defm : i16_arithmetic_pats<bswap, V_BSWAP_U16_e32>;
+//defm : i16_arithmetic_pats<cttz, V_CTTZ_U16_e32>;
+//defm : i16_arithmetic_pats<crrz_zero_undef, V_MUL_U16_e32>;
+
 
 } // End Predicates = [isVI]

test/CodeGen/AMDGPU/max.i16.ll

+; RUN: llc -march=amdgcn < %s | FileCheck -check-prefix=VI -check-prefix=FUNC %s
+
+declare i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+
+; FUNC-LABEL: {{^}}v_test_imax_sge_i16:
+; VI: v_max_i16_e32
+define void @v_test_imax_sge_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp sge i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}v_test_imax_sge_v4i16:
+; VI: v_max_i16_e32
+; VI: v_max_i16_e32
+; VI: v_max_i16_e32
+; VI: v_max_i16_e32
+define void @v_test_imax_sge_v4i16(<4 x i16> addrspace(1)* %out, <4 x i16> addrspace(1)* %aptr, <4 x i16> addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr <4 x i16>, <4 x i16> addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr <4 x i16>, <4 x i16> addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr <4 x i16>, <4 x i16> addrspace(1)* %out, i32 %tid
+  %a = load <4 x i16>, <4 x i16> addrspace(1)* %gep0, align 4
+  %b = load <4 x i16>, <4 x i16> addrspace(1)* %gep1, align 4
+  %cmp = icmp sge <4 x i16> %a, %b
+  %val = select <4 x i1> %cmp, <4 x i16> %a, <4 x i16> %b
+  store <4 x i16> %val, <4 x i16> addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_imax_sge_i16
+; VI: s_max_i16
+define void @s_test_imax_sge_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp sge i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imax_sge_imm_i16:
+; VI: s_max_i16 {{s[0-9]+}}, {{s[0-9]+}}, 9
+define void @s_test_imax_sge_imm_i16(i16 addrspace(1)* %out, i16 %a) nounwind {
+  %cmp = icmp sge i16 %a, 9
+  %val = select i1 %cmp, i16 %a, i16 9
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imax_sgt_imm_i16:
+; VI: s_max_i16 {{s[0-9]+}}, {{s[0-9]+}}, 9
+define void @s_test_imax_sgt_imm_i16(i16 addrspace(1)* %out, i16 %a) nounwind {
+  %cmp = icmp sgt i16 %a, 9
+  %val = select i1 %cmp, i16 %a, i16 9
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imax_sgt_imm_v2i16:
+; VI: s_max_i16 {{s[0-9]+}}, {{s[0-9]+}}, 9
+; VI: s_max_i16 {{s[0-9]+}}, {{s[0-9]+}}, 9
+define void @s_test_imax_sgt_imm_v2i16(<2 x i16> addrspace(1)* %out, <2 x i16> %a) nounwind {
+  %cmp = icmp sgt <2 x i16> %a, <i16 9, i16 9>
+  %val = select <2 x i1> %cmp, <2 x i16> %a, <2 x i16> <i16 9, i16 9>
+  store <2 x i16> %val, <2 x i16> addrspace(1)* %out, align 4
+  ret void
+}
+; FUNC-LABEL: @v_test_imax_sgt_i16
+; VI: v_max_i16_e32
+define void @v_test_imax_sgt_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp sgt i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_imax_sgt_i16
+; VI: s_max_i16
+define void @s_test_imax_sgt_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp sgt i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: @v_test_umax_uge_i16
+; VI: v_max_u32_e32
+define void @v_test_umax_uge_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp uge i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_umax_uge_i16
+; VI: s_max_u32
+define void @s_test_umax_uge_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp uge i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_umax_uge_v3i16:
+; VI: s_max_u32
+; VI: s_max_u32
+; VI: s_max_u32
+; VI-NOT: s_max_u32
+; VI: s_endpgm
+define void @s_test_umax_uge_v3i16(<3 x i16> addrspace(1)* %out, <3 x i16> %a, <3 x i16> %b) nounwind {
+  %cmp = icmp uge <3 x i16> %a, %b
+  %val = select <3 x i1> %cmp, <3 x i16> %a, <3 x i16> %b
+  store <3 x i16> %val, <3 x i16> addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: @v_test_umax_ugt_i16
+; VI: v_max_u32_e32
+define void @v_test_umax_ugt_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp ugt i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_umax_ugt_i16:
+; VI: s_max_u32
+define void @s_test_umax_ugt_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp ugt i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_umax_ugt_imm_v2i16:
+; VI: s_max_u32 {{s[0-9]+}}, {{s[0-9]+}}, 15
+; VI: s_max_u32 {{s[0-9]+}}, {{s[0-9]+}}, 23
+define void @s_test_umax_ugt_imm_v2i16(<2 x i16> addrspace(1)* %out, <2 x i16> %a) nounwind {
+  %cmp = icmp ugt <2 x i16> %a, <i16 15, i16 23>
+  %val = select <2 x i1> %cmp, <2 x i16> %a, <2 x i16> <i16 15, i16 23>
+  store <2 x i16> %val, <2 x i16> addrspace(1)* %out, align 4
+  ret void
+}
+

test/CodeGen/AMDGPU/min_test.ll

+; RUN: llc -march=amdgcn < %s | FileCheck -check-prefix=VI -check-prefix=FUNC %s
+
+declare i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+
+; FUNC-LABEL: {{^}}v_test_imin_sle_i16:
+; VI: v_min_i16_e32
+define void @v_test_imin_sle_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp sle i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imin_sle_i16:
+; VI: s_min_i16
+define void @s_test_imin_sle_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp sle i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imin_sle_v1i16:
+; VI: s_min_i16
+define void @s_test_imin_sle_v1i16(<1 x i16> addrspace(1)* %out, <1 x i16> %a, <1 x i16> %b) nounwind {
+  %cmp = icmp sle <1 x i16> %a, %b
+  %val = select <1 x i1> %cmp, <1 x i16> %a, <1 x i16> %b
+  store <1 x i16> %val, <1 x i16> addrspace(1)* %out
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imin_sle_v4i16:
+; VI: v_min_i16
+; VI: v_min_i16
+; VI: v_min_i16
+; VI: v_min_i16
+define void @s_test_imin_sle_v4i16(<4 x i16> addrspace(1)* %out, <4 x i16> %a, <4 x i16> %b) nounwind {
+  %cmp = icmp sle <4 x i16> %a, %b
+  %val = select <4 x i1> %cmp, <4 x i16> %a, <4 x i16> %b
+  store <4 x i16> %val, <4 x i16> addrspace(1)* %out
+  ret void
+}
+
+; FUNC-LABEL: @v_test_imin_slt_i16
+; VI: v_min_i16_e32
+define void @v_test_imin_slt_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp slt i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_imin_slt_i16
+; VI: s_min_i16
+define void @s_test_imin_slt_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp slt i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imin_slt_v2i16:
+; VI: s_min_i16
+; VI: s_min_i16
+define void @s_test_imin_slt_v2i16(<2 x i16> addrspace(1)* %out, <2 x i16> %a, <2 x i16> %b) nounwind {
+  %cmp = icmp slt <2 x i16> %a, %b
+  %val = select <2 x i1> %cmp, <2 x i16> %a, <2 x i16> %b
+  store <2 x i16> %val, <2 x i16> addrspace(1)* %out
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imin_slt_imm_i16:
+; VI: s_min_i16 {{s[0-9]+}}, {{s[0-9]+}}, 8
+define void @s_test_imin_slt_imm_i16(i16 addrspace(1)* %out, i16 %a) nounwind {
+  %cmp = icmp slt i16 %a, 8
+  %val = select i1 %cmp, i16 %a, i16 8
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_imin_sle_imm_i16:
+; VI: s_min_i16 {{s[0-9]+}}, {{s[0-9]+}}, 8
+define void @s_test_imin_sle_imm_i16(i16 addrspace(1)* %out, i16 %a) nounwind {
+  %cmp = icmp sle i16 %a, 8
+  %val = select i1 %cmp, i16 %a, i16 8
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: @v_test_umin_ule_i16
+; VI: v_min_u16_e32
+define void @v_test_umin_ule_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp ule i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: @v_test_umin_ule_v3i16
+; VI: v_min_u16_e32
+; VI: v_min_u16_e32
+; VI: v_min_u16_e32
+; VI-NOT: v_min_u16_e32
+; VI: s_endpgm
+define void @v_test_umin_ule_v3i16(<3 x i16> addrspace(1)* %out, <3 x i16> %a, <3 x i16> %b) nounwind {
+  %cmp = icmp ule <3 x i16> %a, %b
+  %val = select <3 x i1> %cmp, <3 x i16> %a, <3 x i16> %b
+  store <3 x i16> %val, <3 x i16> addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_umin_ule_i16
+; VI: s_min_u16
+define void @s_test_umin_ule_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp ule i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: @v_test_umin_ult_i16
+; VI: v_min_u16_e32
+define void @v_test_umin_ult_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 addrspace(1)* %bptr) nounwind {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone
+  %gep0 = getelementptr i16, i16 addrspace(1)* %aptr, i32 %tid
+  %gep1 = getelementptr i16, i16 addrspace(1)* %bptr, i32 %tid
+  %outgep = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep0, align 4
+  %b = load i16, i16 addrspace(1)* %gep1, align 4
+  %cmp = icmp ult i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %outgep, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_umin_ult_i16
+; VI: s_min_u16
+define void @s_test_umin_ult_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) nounwind {
+  %cmp = icmp ult i16 %a, %b
+  %val = select i1 %cmp, i16 %a, i16 %b
+  store i16 %val, i16 addrspace(1)* %out, align 4
+  ret void
+}
+
+; FUNC-LABEL: @s_test_umin_ult_v1i16
+; VI: s_min_u16
+define void @s_test_umin_ult_v1i16(<1 x i16> addrspace(1)* %out, <1 x i16> %a, <1 x i16> %b) nounwind {
+  %cmp = icmp ult <1 x i16> %a, %b
+  %val = select <1 x i1> %cmp, <1 x i16> %a, <1 x i16> %b
+  store <1 x i16> %val, <1 x i16> addrspace(1)* %out
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_test_umin_ult_v8i16:
+; VI: s_min_u16
+; VI: s_min_u16
+; VI: s_min_u16
+; VI: s_min_u16
+; VI: s_min_u16
+; VI: s_min_u16
+; VI: s_min_u16
+; VI: s_min_u16
+define void @s_test_umin_ult_v8i16(<8 x i16> addrspace(1)* %out, <8 x i16> %a, <8 x i16> %b) nounwind {
+  %cmp = icmp ult <8 x i16> %a, %b
+  %val = select <8 x i1> %cmp, <8 x i16> %a, <8 x i16> %b
+  store <8 x i16> %val, <8 x i16> addrspace(1)* %out
+  ret void
+}
+
+

test/CodeGen/AMDGPU/shl.i16.ll

+; RUN: llc < %s -march=amdgcn -mcpu=tonga -verify-machineinstrs | FileCheck -check-prefix=GCN -check-prefix=VI %s
+
+declare i32 @llvm.r600.read.tidig.x() #0
+
+;VI: {{^}}shl_v2i16:
+;VI: v_lshlrev_b32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+;VI: v_lshlrev_b32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+
+define void @shl_v2i16(<2 x i16> addrspace(1)* %out, <2 x i16> addrspace(1)* %in) {
+  %b_ptr = getelementptr <2 x i16>, <2 x i16> addrspace(1)* %in, i32 1
+  %a = load <2 x i16>, <2 x i16> addrspace(1) * %in
+  %b = load <2 x i16>, <2 x i16> addrspace(1) * %b_ptr
+  %result = shl <2 x i16> %a, %b
+  store <2 x i16> %result, <2 x i16> addrspace(1)* %out
+  ret void
+}
+
+;VI: {{^}}shl_v4i16:
+;VI: v_lshlrev_b32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+;VI: v_lshlrev_b32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+;VI: v_lshlrev_b32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+;VI: v_lshlrev_b32_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+
+define void @shl_v4i16(<4 x i16> addrspace(1)* %out, <4 x i16> addrspace(1)* %in) {
+  %b_ptr = getelementptr <4 x i16>, <4 x i16> addrspace(1)* %in, i32 1
+  %a = load <4 x i16>, <4 x i16> addrspace(1) * %in
+  %b = load <4 x i16>, <4 x i16> addrspace(1) * %b_ptr
+  %result = shl <4 x i16> %a, %b
+  store <4 x i16> %result, <4 x i16> addrspace(1)* %out
+  ret void
+}
+
+
+;VI: {{^}}shl_i16:
+;VI: v_lshlrev_b64 {{v\[[0-9]+:[0-9]+\], v[0-9]+, v\[[0-9]+:[0-9]+\]}}
+
+define void @shl_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %in) {
+  %b_ptr = getelementptr i16, i16 addrspace(1)* %in, i16 1
+  %a = load i16, i16 addrspace(1) * %in
+  %b = load i16, i16 addrspace(1) * %b_ptr
+  %result = shl i16 %a, %b
+  store i16 %result, i16 addrspace(1)* %out
+  ret void
+}
+
+; FUNC-LABEL: {{^}}v_shl_i16_32_bit_constant:
+; SI-DAG: buffer_load_dword [[VAL:v[0-9]+]]
+; SI-DAG: s_mov_b32 s[[KLO:[0-9]+]], 0x12d687{{$}}
+; SI-DAG: s_mov_b32 s[[KHI:[0-9]+]], 0{{$}}
+; SI: v_lshl_b64 {{v\[[0-9]+:[0-9]+\]}}, s{{\[}}[[KLO]]:[[KHI]]{{\]}}, [[VAL]]
+define void @v_shl_i16_32_bit_constant(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr) {
+  %a = load i16, i16 addrspace(1)* %aptr, align 8
+  %shl = shl i16 1234567, %a
+  store i16 %shl, i16 addrspace(1)* %out, align 8
+  ret void
+}
+
+; FUNC-LABEL: {{^}}v_shl_inline_imm_8_i16:
+; SI: v_lshl_b64 {{v\[[0-9]+:[0-9]+\]}}, 64, {{v[0-9]+}}
+define void @v_shl_inline_imm_64_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr) {
+  %a = load i16, i16 addrspace(1)* %aptr, align 8
+  %shl = shl i16 8, %a
+  store i16 %shl, i16 addrspace(1)* %out, align 8
+  ret void
+}
+
+; FUNC-LABEL: {{^}}s_shl_inline_imm_1_i16:
+; SI: s_lshl_b64 s{{\[[0-9]+:[0-9]+\]}}, 1, s{{[0-9]+}}
+define void @s_shl_inline_imm_1_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %aptr, i16 %a) {
+  %shl = shl i16 1, %a
+  store i16 %shl, i16 addrspace(1)* %out, align 8
+  ret void
+}
+
+attributes #0 = { nounwind readnone }

test/CodeGen/AMDGPU/sra.i16.ll

+; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck -check-prefix=GCN -check-prefix=VI -check-prefix=FUNC %s
+
+declare i32 @llvm.r600.read.tidig.x() #0
+
+; FUNC-LABEL: {{^}}ashr_v2i16:
+
+; VI: v_ashrrev_i16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+; VI: v_ashrrev_i16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+
+define void @ashr_v2i16(<2 x i16> addrspace(1)* %out, <2 x i16> addrspace(1)* %in) {
+  %b_ptr = getelementptr <2 x i16>, <2 x i16> addrspace(1)* %in, i16 1
+  %a = load <2 x i16>, <2 x i16> addrspace(1)* %in
+  %b = load <2 x i16>, <2 x i16> addrspace(1)* %b_ptr
+  %result = ashr <2 x i16> %a, %b
+  store <2 x i16> %result, <2 x i16> addrspace(1)* %out
+  ret void
+}
+
+; FUNC-LABEL: {{^}}ashr_v4i16:
+
+; VI: v_ashrrev_i16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+; VI: v_ashrrev_i16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+; VI: v_ashrrev_i16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+; VI: v_ashrrev_i16_e32 v{{[0-9]+, v[0-9]+, v[0-9]+}}
+
+define void @ashr_v4i16(<4 x i16> addrspace(1)* %out, <4 x i16> addrspace(1)* %in) {
+  %b_ptr = getelementptr <4 x i16>, <4 x i16> addrspace(1)* %in, i16 1
+  %a = load <4 x i16>, <4 x i16> addrspace(1)* %in
+  %b = load <4 x i16>, <4 x i16> addrspace(1)* %b_ptr
+  %result = ashr <4 x i16> %a, %b
+  store <4 x i16> %result, <4 x i16> addrspace(1)* %out
+  ret void
+}
+
+
+; FUNC-LABEL: {{^}}ashr_i16_2:
+
+; VI: v_ashrrev_i16 {{v\[[0-9]+:[0-9]+\], v[0-9]+, v\[[0-9]+:[0-9]+\]}}
+
+define void @ashr_i16_2(i16 addrspace(1)* %out, i16 addrspace(1)* %in) {
+entry:
+  %b_ptr = getelementptr i16, i16 addrspace(1)* %in, i16 1
+  %a = load i16, i16 addrspace(1)* %in
+  %b = load i16, i16 addrspace(1)* %b_ptr
+  %result = ashr i16 %a, %b
+  store i16 %result, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_ashr_2_i16:
+; SI: buffer_load_dword v[[HI:[0-9]+]], {{v\[[0-9]+:[0-9]+\]}}, {{s\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:4
+; VI: flat_load_dword v[[HI:[0-9]+]]
+; GCN: v_ashrrev_i16_e32 v[[SHIFT:[0-9]+]], 31, v[[HI]]
+; GCN: {{buffer|flat}}_store_dwordx2 {{.*}}v{{\[}}[[HI]]:[[SHIFT]]{{\]}}
+define void @v_ashr_32_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %in) {
+  %tid = call i32 @llvm.r600.read.tidig.x() #0
+  %gep.in = getelementptr i16, i16 addrspace(1)* %in, i32 %tid
+  %gep.out = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep.in
+  %result = ashr i16 %a, 2
+  store i16 %result, i16 addrspace(1)* %gep.out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_ashr_4_i16:
+; GCN-DAG: s_load_dword s[[HI:[0-9]+]], {{s\[[0-9]+:[0-9]+\]}}, {{0xc|0x30}}
+; GCN: s_ashr_i16 s[[SHIFT:[0-9]+]], s[[HI]], 31
+; GCN: s_mov_b32 s[[COPYSHIFT:[0-9]+]], s[[SHIFT]]
+; GCN: s_add_u32 {{s[0-9]+}}, s[[HI]], {{s[0-9]+}}
+; GCN: s_addc_u32 {{s[0-9]+}}, s[[COPYSHIFT]], {{s[0-9]+}}
+define void @s_ashr_63_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) {
+  %result = ashr i16 %a, 4
+  %add = add i16 %result, %b
+  store i16 %add, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_ashr_8_i16:
+; VI: flat_load_dword v[[HI:[0-9]+]]
+define void @v_ashr_63_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %in) {
+  %tid = call i32 @llvm.r600.read.tidig.x() #0
+  %gep.in = getelementptr i16, i16 addrspace(1)* %in, i32 %tid
+  %gep.out = getelementptr i16, i16 addrspace(1)* %out, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep.in
+  %result = ashr i16 %a, 8
+  store i16 %result, i16 addrspace(1)* %gep.out
+  ret void
+}
+
+attributes #0 = { nounwind readnone }

test/CodeGen/AMDGPU/sub.i16.ll

+; RUN: llc -march=amdgcn -mcpu=tonga -verify-machineinstrs < %s | FileCheck -check-prefix=VI -check-prefix=GCN %s
+
+; GCN-LABEL: {{^}}v_test_sub_i16:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: flat_load_ushort [[B:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], [[A]], [[B]]
+; VI-NEXT: buffer_store_short [[ADD]]
+define void @v_test_sub_i16(i16 addrspace(1)* %out, i16 addrspace(1)* %in0, i16 addrspace(1)* %in1) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i16, i16 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %gep.in1 = getelementptr inbounds i16, i16 addrspace(1)* %in1, i32 %tid
+  %a = load volatile i16, i16 addrspace(1)* %gep.in0
+  %b = load volatile i16, i16 addrspace(1)* %gep.in1
+  %sub = sub i16 %a, %b
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_constant:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], 0x7b, [[A]]
+; VI-NEXT: buffer_store_short [[ADD]]
+define void @v_test_sub_i16_constant(i16 addrspace(1)* %out, i16 addrspace(1)* %in0) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i16, i16 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %a = load volatile i16, i16 addrspace(1)* %gep.in0
+  %sub = sub i16 %a, 123
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_neg_constant:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], 0xfffffcb3, [[A]]
+; VI-NEXT: buffer_store_short [[ADD]]
+define void @v_test_sub_i16_neg_constant(i16 addrspace(1)* %out, i16 addrspace(1)* %in0) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i16, i16 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %a = load volatile i16, i16 addrspace(1)* %gep.in0
+  %sub = sub i16 %a, -845
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_inline_neg1:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], -1, [[A]]
+; VI-NEXT: buffer_store_short [[ADD]]
+define void @v_test_sub_i16_inline_neg1(i16 addrspace(1)* %out, i16 addrspace(1)* %in0) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i16, i16 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %a = load volatile i16, i16 addrspace(1)* %gep.in0
+  %sub = sub i16 %a, -1
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_zext_to_i32:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: flat_load_ushort [[B:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], [[A]], [[B]]
+; VI-NEXT: buffer_store_dword [[ADD]]
+define void @v_test_sub_i16_zext_to_i32(i32 addrspace(1)* %out, i16 addrspace(1)* %in0, i16 addrspace(1)* %in1) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i32, i32 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %gep.in1 = getelementptr inbounds i16, i16 addrspace(1)* %in1, i32 %tid
+  %a = load volatile i16, i16 addrspace(1)* %gep.in0
+  %b = load volatile i16, i16 addrspace(1)* %gep.in1
+  %sub = sub i16 %a, %b
+  %ext = zext i16 %sub to i32
+  store i32 %ext, i32 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_zext_to_i64:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: flat_load_ushort [[B:v[0-9]+]]
+; VI-DAG: v_sub_u16_e32 v[[ADD:[0-9]+]], [[A]], [[B]]
+; VI-DAG: v_mov_b32_e32 v[[VZERO:[0-9]+]], 0{{$}}
+; VI: buffer_store_dwordx2 v{{\[}}[[ADD]]:[[VZERO]]{{\]}}
+define void @v_test_sub_i16_zext_to_i64(i64 addrspace(1)* %out, i16 addrspace(1)* %in0, i16 addrspace(1)* %in1) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i64, i64 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %gep.in1 = getelementptr inbounds i16, i16 addrspace(1)* %in1, i32 %tid
+  %a = load volatile i16, i16 addrspace(1)* %gep.in0
+  %b = load volatile i16, i16 addrspace(1)* %gep.in1
+  %sub = sub i16 %a, %b
+  %ext = zext i16 %sub to i64
+  store i64 %ext, i64 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_sext_to_i32:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: flat_load_ushort [[B:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], [[A]], [[B]]
+; VI-NEXT: v_bfe_i32 [[SEXT:v[0-9]+]], [[ADD]], 0, 16
+; VI-NEXT: buffer_store_dword [[SEXT]]
+define void @v_test_sub_i16_sext_to_i32(i32 addrspace(1)* %out, i16 addrspace(1)* %in0, i16 addrspace(1)* %in1) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i32, i32 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %gep.in1 = getelementptr inbounds i16, i16 addrspace(1)* %in1, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep.in0
+  %b = load i16, i16 addrspace(1)* %gep.in1
+  %sub = sub i16 %a, %b
+  %ext = sext i16 %sub to i32
+  store i32 %ext, i32 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}v_test_sub_i16_sext_to_i64:
+; VI: flat_load_ushort [[A:v[0-9]+]]
+; VI: flat_load_ushort [[B:v[0-9]+]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], [[A]], [[B]]
+; VI-NEXT: v_bfe_i32 v[[LO:[0-9]+]], [[ADD]], 0, 16
+; VI-NEXT: v_ashrrev_i32_e32 v[[HI:[0-9]+]], 31, v[[LO]]
+; VI-NEXT: buffer_store_dwordx2 v{{\[}}[[LO]]:[[HI]]{{\]}}
+define void @v_test_sub_i16_sext_to_i64(i64 addrspace(1)* %out, i16 addrspace(1)* %in0, i16 addrspace(1)* %in1) #1 {
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %gep.out = getelementptr inbounds i64, i64 addrspace(1)* %out, i32 %tid
+  %gep.in0 = getelementptr inbounds i16, i16 addrspace(1)* %in0, i32 %tid
+  %gep.in1 = getelementptr inbounds i16, i16 addrspace(1)* %in1, i32 %tid
+  %a = load i16, i16 addrspace(1)* %gep.in0
+  %b = load i16, i16 addrspace(1)* %gep.in1
+  %sub = sub i16 %a, %b
+  %ext = sext i16 %sub to i64
+  store i64 %ext, i64 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI: v_sub_u16_e32 [[RESULT:v[0-9]+]], [[B]], [[VA]]
+; VI-NEXT: buffer_store_short [[RESULT]]
+define void @s_test_sub_i16(i16 addrspace(1)* %out, i16 %a, i16 %b) #1 {
+  %sub = sub i16 %a, %b
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16_zeroext_args:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI: v_sub_u16_e32 [[RESULT:v[0-9]+]], [[B]], [[VA]]
+; VI-NEXT: buffer_store_short [[RESULT]]
+define void @s_test_sub_i16_zeroext_args(i16 addrspace(1)* %out, i16 zeroext %a, i16 zeroext %b) #1 {
+  %sub = sub i16 %a, %b
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16_signext_args:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI: v_sub_u16_e32 [[RESULT:v[0-9]+]], [[B]], [[VA]]
+; VI-NEXT: buffer_store_short [[RESULT]]
+define void @s_test_sub_i16_signext_args(i16 addrspace(1)* %out, i16 signext %a, i16 signext %b) #1 {
+  %sub = sub i16 %a, %b
+  store i16 %sub, i16 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16_zext_to_i32:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], [[B]], [[VA]]
+; VI-NEXT: buffer_store_dword [[RESULT]]
+define void @s_test_sub_i16_zext_to_i32(i32 addrspace(1)* %out, i16 zeroext %a, i16 zeroext %b) #1 {
+  %sub = sub i16 %a, %b
+  %ext = zext i16 %sub to i32
+  store i32 %ext, i32 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16_zext_to_i64:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI-DAG: v_sub_u16_e32 v[[LO:[0-9]+]], [[B]], [[VA]]
+; VI-DAG: v_mov_b32_e32 v[[HI:[0-9]+]], 0{{$}}
+; VI-NEXT: buffer_store_dwordx2 v{{\[}}[[LO]]:[[HI]]{{\]}}
+define void @s_test_sub_i16_zext_to_i64(i64 addrspace(1)* %out, i16 zeroext %a, i16 zeroext %b) #1 {
+  %sub = sub i16 %a, %b
+  %ext = zext i16 %sub to i64
+  store i64 %ext, i64 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16_sext_to_i32:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI: v_sub_u16_e32 [[ADD:v[0-9]+]], [[B]], [[VA]]
+; VI-NEXT: v_bfe_i32 [[RESULT:v[0-9]+]], [[ADD]], 0, 16
+; VI-NEXT: buffer_store_dword [[RESULT]]
+define void @s_test_sub_i16_sext_to_i32(i32 addrspace(1)* %out, i16 signext %a, i16 signext %b) #1 {
+  %sub = sub i16 %a, %b
+  %ext = sext i16 %sub to i32
+  store i32 %ext, i32 addrspace(1)* %out
+  ret void
+}
+
+; GCN-LABEL: {{^}}s_test_sub_i16_sext_to_i64:
+; VI-DAG: s_load_dword [[A:s[0-9]+]], s[0:1], 0x2c
+; VI-DAG: s_load_dword [[B:s[0-9]+]], s[0:1], 0x30
+; VI-DAG: v_mov_b32_e32 [[VA:v[0-9]+]], [[A]]
+; VI: v_sub_u16_e32 v[[LO:[0-9]+]], [[B]], [[VA]]
+; VI-NEXT: v_bfe_i32 v[[LO:[0-9]+]], [[ADD]], 0, 16
+; VI-NEXT: v_ashrrev_i32_e32 v[[HI:[0-9]+]], 31, v[[LO]]
+; VI-NEXT: buffer_store_dwordx2 v{{\[}}[[LO]]:[[HI]]{{\]}}
+define void @s_test_sub_i16_sext_to_i64(i64 addrspace(1)* %out, i16 signext %a, i16 signext %b) #1 {
+  %sub = sub i16 %a, %b
+  %ext = sext i16 %sub to i64
+  store i64 %ext, i64 addrspace(1)* %out
+  ret void
+}
+
+declare i32 @llvm.amdgcn.workitem.id.x() #0
+
+attributes #0 = { nounwind readnone }
+attributes #1 = { nounwind }