[AMDGPU] Support idot2 pattern.

[FarhanaAleen]

Summary

Transform add (mul ((i32)S0.x, (i32)S1.x),

add( mul ((i32)S0.y, (i32)S1.y), (i32)S3) => i/udot2((v2i16)S0, (v2i16)S1, (i32)S3)

[arsenm]

This seems to be a lot of code to try different commutations. Why can't you just use table gen patterns which would generate these for you?

Also seem to be missing a lot of hasOneUse checks

lib/Target/AMDGPU/SIISelLowering.cpp

4949–4954 ↗

(On Diff #158091)

I would split the intrinsics into a separate patch

test/CodeGen/AMDGPU/idot2.ll

272–273

Avoid using function calls in tests that aren't specifically testing calls

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

7354–7355 ↗

(On Diff #158091)

return the SDValue results rather than bool + out argument?

7358 ↗

(On Diff #158091)

Don't need explicit initializers

[arsenm]

lib/Target/AMDGPU/SIISelLowering.cpp

4949–4954 ↗

(On Diff #158091)

The intrinsic definition also seems to be missing from the patch

[kzhuravl]

lib/Target/AMDGPU/SIISelLowering.cpp

4949–4954 ↗

(On Diff #158091)

Intrinsic definitions were already committed.

[FarhanaAleen]

Thanks Matt.

In D50024#1182247, @arsenm wrote:

This seems to be a lot of code to try different commutations. Why can't you just use table gen patterns which would generate these for you?

We can't use table gen patterns because all the types are promoted by then. Also, recognition of vector elements would be very complex.

Also seem to be missing a lot of hasOneUse checks

They are not missing, I did not add them intentionally. In general, we care about throughput on GPU, right? With multiple uses, generating dot instruction saves 2 instructions in the best case(1 op having multiple uses), 0 instructions in the worse case(all 3 ops having multiple uses).

As per latency, I was thinking dot instruction should be as fast as one MAD since it will be performing the multiplication on packed data which can be as fast as performing one 16bit multiplication. With multiple uses, if dot is not generated, we will be generating 2 MADs. So, it should be better to generate 1 dot instead of 2 MADs, right?

[FarhanaAleen]

• Removed SDValue initialization
• Removed function calls from the testcases.
• Returned SDValue instead of bool+SDValue

[arsenm]

In D50024#1183200, @FarhanaAleen wrote:

Thanks Matt.

In D50024#1182247, @arsenm wrote:

This seems to be a lot of code to try different commutations. Why can't you just use table gen patterns which would generate these for you?

We can't use table gen patterns because all the types are promoted by then. Also, recognition of vector elements would be very complex.

But all the types are legal already?

We should think about treating the vector extracts as legal, and then cleaning up the mess from selecting those later. I think at this point doing what we do is causing more problems than it's worth

Also seem to be missing a lot of hasOneUse checks

They are not missing, I did not add them intentionally. In general, we care about throughput on GPU, right? With multiple uses, generating dot instruction saves 2 instructions in the best case(1 op having multiple uses), 0 instructions in the worse case(all 3 ops having multiple uses).

Register pressure also matters, probably more. In the every instruction has another use case, you're just increasing the register count without saving anything. I would need to think more carefully about the other cases.

Either way, there should be tests with the different operands with multiple uses.

As per latency, I was thinking dot instruction should be as fast as one MAD since it will be performing the multiplication on packed data which can be as fast as performing one 16bit multiplication. With multiple uses, if dot is not generated, we will be generating 2 MADs. So, it should be better to generate 1 dot instead of 2 MADs, right?

Are these really full rate instructions? They are marked as such now, but I thought these were quarter rate.

test/CodeGen/AMDGPU/idot2.ll

25

What happens if everything is done in i16? Can this still be matched?

[FarhanaAleen]

Supported the transformation using table gen patterns.

[FarhanaAleen]

But all the types are legal already?

Yes, they are legal types but we have packed instructions that can operation on a pair of 16bits, therefore packed types can be treated as 32bit scalar type.

We should think about treating the vector extracts as legal, and then cleaning up the mess from selecting those later. I think at this point doing what we do is causing more problems than it's worth

I agree. I also realized that we don't need to worry about the exact types. We should treat them as 32 bit since we have packed instructions.

test/CodeGen/AMDGPU/idot2.ll

25

The pattern will not be matched if everything is done in 16. I will support it in a separate patch.

[FarhanaAleen]

If all the operations happen in 16bit, the pattern is detected. Added a testcase of that pattern.

[arsenm]

lib/Target/AMDGPU/VOP3PInstructions.td

180–181

Patterns matching sext_inreg directly is kind of unusual, especially for a specific size. I would expect this to be a number of known sign bits check?

test/CodeGen/AMDGPU/idot2.ll

10–13

These tests are a bit thin. Perhaps use update_llc_test_checks?

87–91

Should include a test with the explicit sext_inreg patterns done on i32, and with different bit widths than i16

[arsenm]

test/CodeGen/AMDGPU/idot2.ll

3

Should also include run lines with a CI and VI target to make sure those don't break

[FarhanaAleen]

Added - a testcase with sign_extend_inreg happening on 32bit from 8bit.

• checks for SI and VI
• update_llc_test checks

lib/Target/AMDGPU/VOP3PInstructions.td

180–181

It's not unusual, I see all the other targets doing this sext_inreg matching with a specific size, specially for vectors. For vectors lying in a 32bit register, we need to make sure that each element is lying on a specific location inside the register.

I feel like doing the known bits check would be redundant since DAG combiner already performed this check before generating sign_extend_inreg. Also, performing this check does not provide any benefit in our vector case since we cannot allow other sizes being sign extended to 16 or higher. It has to be exactly coming from the lower/upper 16bit of a 32bit register unless we rearrange the data orientation inside the 32bit register.

[FarhanaAleen]

It looks like there are no further comments. In that case, I will go ahead and check it in.

[arsenm]

LGTM

lib/Target/AMDGPU/AMDGPUInstructions.td

 def add_oneuse : HasOneUseBinOp<add>;
 def sub_oneuse : HasOneUseBinOp<sub>;
 
 def srl_oneuse : HasOneUseBinOp<srl>;
 def shl_oneuse : HasOneUseBinOp<shl>;
 
 def select_oneuse : HasOneUseTernaryOp<select>;
 
+def AMDGPUmul_u24_oneuse : HasOneUseBinOp<AMDGPUmul_u24>;
+def AMDGPUmul_i24_oneuse : HasOneUseBinOp<AMDGPUmul_i24>;
+
 def srl_16 : PatFrag<
   (ops node:$src0), (srl_oneuse node:$src0, (i32 16))
 >;
 
 
 def hi_i16_elt : PatFrag<
   (ops node:$src0), (i16 (trunc (i32 (srl_16 node:$src0))))
 >;

lib/Target/AMDGPU/VOP3PInstructions.td

 let ClampLo = 0, ClampHi = 1 in {
 def V_FMA_MIXHI_F16 : VOP3_VOP3PInst<"v_fma_mixhi_f16", VOP3_Profile<VOP_F16_F16_F16_F16, VOP3_OPSEL>, 1>;
 }
 }
 
 defm : MadFmaMixPats<fma, V_FMA_MIX_F32, V_FMA_MIXLO_F16, V_FMA_MIXHI_F16>;
 }
 
+class UDot2Pat<Instruction Inst> : GCNPat <
+  (add (add_oneuse (AMDGPUmul_u24_oneuse (srl i32:$src0, (i32 16)),
+                                         (srl i32:$src1, (i32 16))), i32:$src2),
+       (AMDGPUmul_u24_oneuse (and i32:$src0, (i32 65535)),
+                             (and i32:$src1, (i32 65535)))
+   ),
+  (Inst (i32 8), $src0, (i32 8), $src1, (i32 8), $src2, (i1 0))
+>;
+
+class SDot2Pat<Instruction Inst> : GCNPat <
+  (add (add_oneuse (AMDGPUmul_i24_oneuse (sra i32:$src0, (i32 16)),
+                                         (sra i32:$src1, (i32 16))), i32:$src2),
+       (AMDGPUmul_i24_oneuse (sext_inreg i32:$src0, i16),
+                             (sext_inreg i32:$src1, i16))
+   ),
+  (Inst (i32 8), $src0, (i32 8), $src1, (i32 8), $src2, (i1 0))
+>;
+
 let SubtargetPredicate = HasDLInsts in {
 
 def V_DOT2_F32_F16 : VOP3PInst<"v_dot2_f32_f16", VOP3_Profile<VOP_F32_V2F16_V2F16_F32>, AMDGPUfdot2>;
 def V_DOT2_I32_I16 : VOP3PInst<"v_dot2_i32_i16", VOP3_Profile<VOP_I32_V2I16_V2I16_I32>, int_amdgcn_sdot2>;
 def V_DOT2_U32_U16 : VOP3PInst<"v_dot2_u32_u16", VOP3_Profile<VOP_I32_V2I16_V2I16_I32>, int_amdgcn_udot2>;
 def V_DOT4_I32_I8  : VOP3PInst<"v_dot4_i32_i8", VOP3_Profile<VOP_I32_I32_I32_I32, VOP3_PACKED>, int_amdgcn_sdot4>;
 def V_DOT4_U32_U8  : VOP3PInst<"v_dot4_u32_u8", VOP3_Profile<VOP_I32_I32_I32_I32, VOP3_PACKED>, int_amdgcn_udot4>;
 def V_DOT8_I32_I4  : VOP3PInst<"v_dot8_i32_i4", VOP3_Profile<VOP_I32_I32_I32_I32, VOP3_PACKED>, int_amdgcn_sdot8>;
 def V_DOT8_U32_U4  : VOP3PInst<"v_dot8_u32_u4", VOP3_Profile<VOP_I32_I32_I32_I32, VOP3_PACKED>, int_amdgcn_udot8>;
 
+def : UDot2Pat<V_DOT2_U32_U16>;
+def : SDot2Pat<V_DOT2_I32_I16>;
+
 } // End SubtargetPredicate = HasDLInsts
 
 multiclass VOP3P_Real_vi<bits<10> op> {
   def _vi : VOP3P_Real<!cast<VOP3_Pseudo>(NAME), SIEncodingFamily.VI>,
             VOP3Pe <op, !cast<VOP3_Pseudo>(NAME).Pfl> {
     let AssemblerPredicates = [HasVOP3PInsts];
     let DecoderNamespace = "VI";
   }

test/CodeGen/AMDGPU/idot2.ll

+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck %s  -check-prefixes=GCN,GFX900
+; RUN: llc -march=amdgcn -mcpu=gfx906 -verify-machineinstrs < %s | FileCheck %s  -check-prefixes=GCN,GFX906
+
+; add(mul(S0.x, S1.y),
+;     add (mul (S0.y, S1.y), S3)) -> v_dot2_{I|U}32_{I|U}16(S1, S2, S3)
+
+; GCN-LABEL: {{^}}udot2
+
+; GFX900: v_mad_u32_u24
+; GFX900: v_mad_u32_u24
+
+; GFX906: v_dot2_u32_u16
+define amdgpu_kernel void @udot2(<2 x i16> addrspace(1)* %src1,
+                                 <2 x i16> addrspace(1)* %src2,
+                                 i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul nuw i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul nuw i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; TODO: Support this pattern
+;      add(S3,
+;          add (mul (S0.y, S1.y), mul (S0.y, S1.y))) -> v_dot2_{I|U}32_{I|U}16(S1, S2, S3)
+; GCN-LABEL: {{^}}udot2_MulMul
+
+; GFX900: v_add_u32_e32
+
+; GFX906: v_add_u32_e32
+define amdgpu_kernel void @udot2_MulMul(<2 x i16> addrspace(1)* %src1,
+                                        <2 x i16> addrspace(1)* %src2,
+                                        i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul nuw i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul nuw i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %mul1
+  %add6 = add i32 %add, %s3
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}idot2
+
+; GFX900: v_mad_i32_i24
+; GFX900: v_mad_i32_i24
+
+; GFX906: v_dot2_i32_i16
+define amdgpu_kernel void @idot2(<2 x i16> addrspace(1)* %src1,
+                                 <2 x i16> addrspace(1)* %src2,
+                                 i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = sext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = sext i16 %s2.elt1 to i32
+  %mul1 = mul nuw i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = sext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = sext i16 %s2.elt2 to i32
+  %mul2 = mul nuw i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}idot2_MixedTypedMul
+
+; GFX900: v_mad_i32_i24
+
+; GFX906: v_mad_i32_i24
+define amdgpu_kernel void @idot2_MixedTypedMul(<2 x i16> addrspace(1)* %src1,
+                                               <2 x i16> addrspace(1)* %src2,
+                                               i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = sext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = sext i16 %s2.elt1 to i32
+  %mul1 = mul nuw i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul nuw i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}udot2_alt_AddOperands
+
+; GFX900: v_mad_u32_u24
+; GFX900: v_mad_u32_u24
+
+; GFX906: v_dot2_u32_u16
+define amdgpu_kernel void @udot2_alt_AddOperands(<2 x i16> addrspace(1)* %src1,
+                                                 <2 x i16> addrspace(1)* %src2,
+                                                 i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul nuw i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul nuw i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %s3, %mul2
+  %add6 = add i32 %mul1, %add
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}idot2_MixedExt
+
+; GFX900: v_mad_i32_i24
+; GFX900: v_mad_i32_i24
+
+; GFX906: v_mad_i32_i24
+define amdgpu_kernel void @idot2_MixedExt(<2 x i16> addrspace(1)* %src1,
+                                          <2 x i16> addrspace(1)* %src2,
+                                          i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = sext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul nuw i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = sext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = sext i16 %s2.elt2 to i32
+  %mul2 = mul nuw i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}notudot2_SameVec
+
+; GFX900: v_mad_u32_u24
+
+; GFX906: v_mad_u32_u24
+define amdgpu_kernel void @notudot2_SameVec(<2 x i16> addrspace(1)* %src1,
+                                            <2 x i16> addrspace(1)* %src2,
+                                            i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}udot2_v4i16
+
+; GFX900: v_mad_u32_u24
+
+; GFX906: v_dot2_u32_u16 v2
+define amdgpu_kernel void @udot2_v4i16(<4 x i16> addrspace(1)* %src1,
+                                       <4 x i16> addrspace(1)* %src2,
+                                       i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <4 x i16>, <4 x i16> addrspace(1)* %src1
+  %vec2 = load <4 x i16>, <4 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <4 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <4 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <4 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <4 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}udot2_v4i16_Hi
+
+; GFX900: v_mad_u32_u24
+
+; GFX906: v_dot2_u32_u16 v2
+define amdgpu_kernel void @udot2_v4i16_Hi(<4 x i16> addrspace(1)* %src1,
+                                          <4 x i16> addrspace(1)* %src2,
+                                          i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <4 x i16>, <4 x i16> addrspace(1)* %src1
+  %vec2 = load <4 x i16>, <4 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <4 x i16> %vec1, i64 2
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <4 x i16> %vec2, i64 2
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <4 x i16> %vec1, i64 3
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <4 x i16> %vec2, i64 3
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+
+; GCN-LABEL: {{^}}notudot2_v4i16_Even
+
+; GFX900: v_mad_u32_u24
+
+; GFX906-NOT: v_dot2_u32_u16
+; GFX906-NOT: v_dot2_i32_i16
+define amdgpu_kernel void @notudot2_v4i16_Even(<4 x i16> addrspace(1)* %src1,
+                                               <4 x i16> addrspace(1)* %src2,
+                                               i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <4 x i16>, <4 x i16> addrspace(1)* %src1
+  %vec2 = load <4 x i16>, <4 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <4 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <4 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <4 x i16> %vec1, i64 2
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <4 x i16> %vec2, i64 2
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}notudot2_v4i16_Middle
+
+; GFX900: v_mad_u32_u24
+
+; GFX906-NOT: v_dot2_u32_u16
+; GFX906-NOT: v_dot2_i32_i16
+define amdgpu_kernel void @notudot2_v4i16_Middle(<4 x i16> addrspace(1)* %src1,
+                                                 <4 x i16> addrspace(1)* %src2,
+                                                 i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <4 x i16>, <4 x i16> addrspace(1)* %src1
+  %vec2 = load <4 x i16>, <4 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <4 x i16> %vec1, i64 1
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <4 x i16> %vec2, i64 1
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <4 x i16> %vec1, i64 2
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <4 x i16> %vec2, i64 2
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}notudot2_DiffIndex
+
+; GFX900: v_mad_u32_u24
+
+; GFX906-NOT: v_dot2_u32_u16
+; GFX906-NOT: v_dot2_i32_i16
+define amdgpu_kernel void @notudot2_DiffIndex(<2 x i16> addrspace(1)* %src1,
+                                              <2 x i16> addrspace(1)* %src2,
+                                              i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 1
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 0
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add = add i32 %mul2, %s3
+  %add6 = add i32 %add, %mul1
+  store i32 %add6, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}udot2_MultipleUses_add1
+
+; GFX900: v_mad_u32_u24
+
+; GFX906-NOT: v_dot2_u32_u16
+; GFX906-NOT: v_dot2_i32_i16
+define amdgpu_kernel void @udot2_MultipleUses_add1(<2 x i16> addrspace(1)* %src1,
+                                                  <2 x i16> addrspace(1)* %src2,
+                                                  i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add1 = add i32 %mul2, %s3
+  %add2 = add i32 %add1, %mul1
+
+  %res = add i32 %add2, %add1
+  store i32 %res, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}idot2_MultipleUses_add1
+
+; GFX900: v_mad_i32_i24
+
+; GFX906-NOT: v_dot2_i32_i16
+; GFX906-NOT: v_dot2_u32_u16
+define amdgpu_kernel void @idot2_MultipleUses_add1(<2 x i16> addrspace(1)* %src1,
+                                                   <2 x i16> addrspace(1)* %src2,
+                                                   i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = sext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = sext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = sext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = sext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add1 = add i32 %mul2, %s3
+  %add2 = add i32 %add1, %mul1
+
+  %res = add i32 %add2, %add1
+  store i32 %res, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}udot2_MultipleUses_mul1
+
+; GFX900: v_mad_u32_u24
+
+; GFX906-NOT: v_dot2_u32_u16
+; GFX906-NOT: v_dot2_i32_i16
+define amdgpu_kernel void @udot2_MultipleUses_mul1(<2 x i16> addrspace(1)* %src1,
+                                                   <2 x i16> addrspace(1)* %src2,
+                                                   i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add0 = add i32 %mul1, %s3
+
+  %add1 = add i32 %mul2, %add0
+  %add2 = add i32 %add1, %mul1
+
+  store i32 %add2, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}idot2_MultipleUses_mul1
+
+; GFX900: v_mad_i32_i24
+
+; GFX906-NOT: v_dot2_i32_i16
+; GFX906-NOT: v_dot2_u32_u16
+define amdgpu_kernel void @idot2_MultipleUses_mul1(<2 x i16> addrspace(1)* %src1,
+                                                   <2 x i16> addrspace(1)* %src2,
+                                                   i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = sext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = sext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = sext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = sext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add0 = add i32 %mul1, %s3
+
+  %add1 = add i32 %mul2, %add0
+  %add2 = add i32 %add1, %mul1
+
+  store i32 %add2, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}udot2_MultipleUses_mul2
+
+; GFX900: v_mad_u32_u24
+; GFX900: v_mad_u32_u24
+; GFX900: v_mad_u32_u24
+
+; GFX906-NOT: v_dot2_i32_i16
+; GFX906-NOT: v_dot2_u32_u16
+define amdgpu_kernel void @udot2_MultipleUses_mul2(<2 x i16> addrspace(1)* %src1,
+                                                   <2 x i16> addrspace(1)* %src2,
+                                                   i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = zext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = zext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = zext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = zext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add0 = add i32 %mul2, %s3
+
+  %add1 = add i32 %mul2, %add0
+  %add2 = add i32 %add1, %mul1
+
+  store i32 %add2, i32 addrspace(1)* %dst, align 4
+  ret void
+}
+
+; GCN-LABEL: {{^}}idot2_MultipleUses_mul2
+
+; GFX900: v_mad_i32_i24
+; GFX900: v_mad_i32_i24
+; GFX900: v_mad_i32_i24
+
+; GFX906-NOT: v_dot2_i32_i16
+; GFX906-NOT: v_dot2_u32_u16
+define amdgpu_kernel void @idot2_MultipleUses_mul2(<2 x i16> addrspace(1)* %src1,
+                                                   <2 x i16> addrspace(1)* %src2,
+                                                   i32 addrspace(1)* nocapture %dst) {
+entry:
+  %vec1 = load <2 x i16>, <2 x i16> addrspace(1)* %src1
+  %vec2 = load <2 x i16>, <2 x i16> addrspace(1)* %src2
+
+  %s1.elt1 = extractelement <2 x i16> %vec1, i64 0
+  %conv = sext i16 %s1.elt1 to i32
+  %s2.elt1 = extractelement <2 x i16> %vec2, i64 0
+  %conv2 = sext i16 %s2.elt1 to i32
+  %mul1 = mul i32 %conv2, %conv
+
+  %s1.elt2 = extractelement <2 x i16> %vec1, i64 1
+  %conv3 = sext i16 %s1.elt2 to i32
+  %s2.elt2 = extractelement <2 x i16> %vec2, i64 1
+  %conv4 = sext i16 %s2.elt2 to i32
+  %mul2 = mul i32 %conv4, %conv3
+
+  %s3 = load i32, i32 addrspace(1)* %dst, align 4
+  %add0 = add i32 %mul2, %s3
+
+  %add1 = add i32 %mul2, %add0
+  %add2 = add i32 %add1, %mul1
+
+  store i32 %add2, i32 addrspace(1)* %dst, align 4
+  ret void
+}