[AMDGPU/wmma] - Disable 3-address syntax for f16

[OutOfCache]

Summary

Always keep wmma instructions with a 16-bit floating-point accumulator
as two-address instruction.

This is a prerequesite for an upcoming optimization for wmma with 16-bit
accumulator matrices.
We want to pack the results of two separate
wmmas into the same register, so one matrix
is in the lower half while the other matrix is
in the upper half of the registers.

We pack the values into the registers before using them
in the first wmma as input:

v_wmma_f16_16x16x16_f16 v[0:7], v[8:15], v[16:23], v[0:7]
v_wmma_f16_16x16x16_f16 v[0:7], v[24:31], v[32:49], v[0:7] op_sel:[0,0,1]

Therefore, both instructions need to write to the same registers
and overwrite the values of the input matrices.

We have verified the correct behavior by
running nod.ai's Stable Diffusion with these
changes in data layout.
On average, this change reduced the vgpr count by 17.17% (in 88 shaders
that the change applied to).

[arsenm]

llvm/lib/Target/AMDGPU/VOP3PInstructions.td

863

Shouldn’t lie about properties, disable at a different point?

[OutOfCache]

llvm/lib/Target/AMDGPU/VOP3PInstructions.td

863

Where would you suggest?

[arsenm]

llvm/lib/Target/AMDGPU/VOP3PInstructions.td

863

Does this happen in two address instructions? I assume the current heuristic assumes a simple register and isn’t accounting for large tuple increasing pressure

[OutOfCache]

llvm/lib/Target/AMDGPU/VOP3PInstructions.td

863

Sorry, what do you mean by 'does this happen'? If you are talking about the conversion from two address to three address instruction, then yes, it can happen. I encountered issues when compiling Stable Diffusion shaders that initialize the matrices via the zeroinitializer constant outside a loop.

In the loop case, each matrix receives a dedicated zero matrix. Without the loop, the same zero matrix is reused for multiple wmmas. So we have the following scenario:

v_wmma_f16_16x16x16_f16 v[0:7],   ..., v[24:31]
v_wmma_f16_16x16x16_f16 v[32:30], ..., v[24:31]

So we use v[24:31 as zero matrix as input, but a different destination matrix (e.g., v[0:7]). The problem arises once we try to pack:

v_wmma_f16_16x16x16_f16 v[0:7],     ..., v[0:7] op_sel [0,0,1]

We have no guarantee that the upper halves of v[0:7] are zero initialized. In practice, this indeed caused issues (completely black images as SD output). This change fixed it.

You are right that disabling the ability to convert is not the best solution, and I would be happy to move this change elsewhere. Currently I don't know a better place, though. Can we maybe adapt the heuristics to always keep the two address mode for constant input matrices like the zeroinitializer? Or does that not make sense either?

[OutOfCache]

Updated test

[piotr]

I am trying to understand the failing case better. Can the issue only happen with the extra patch with packing? Is the issue only with zeroinitializers (constant matrices), or it is just where the problem was found?

It would be preferable to keep the 3-address syntax, as it can produce better code (see the extra moves in the updated ir tests).

[OutOfCache]

Sorry for responding so late.

The problem was only found with zeroinitializer matrices, which were reused as input for multiple wmma_f16 instructions. However, it could happen for other, non-constant matrices as well, as long as the input and output accumulator registers are different.

After further discussion with other compiler engineers, we want to add a new pseudo instruction for the tied instruction. Then we can update the intrinsics in the packing patch to lower to these specific pseudos. That way, the original wmma_f16 instruction can still be a three-address instruction in cases outside the patch.

[OutOfCache]

• [AMDGPU/VOP3P] - Simplify wmma instruction defs
• [AMDGPU/VOP3P] - Add tied wmma_f16 pseudos

Use new pseudo instructions instead of modifying existing ones.

[piotr]

Thanks, that would avoid the regression. However, I still do not fully understand the failing mode - can you show the test case + extra code that triggers the issue?

[foad]

The patch as-is needs some codegen tests for the new intrinsics. (I have no opinion on whether adding new intrinsics is actually the right approach here.)

[OutOfCache]

Added codegen tests for new intrinsics

[OutOfCache]

In D158059#4640554, @piotr wrote:

Thanks, that would avoid the regression. However, I still do not fully understand the failing mode - can you show the test case + extra code that triggers the issue?

I admit, it is probably hard to follow without the full context. I'll try again.

During cooperative matrix calculation, we typically have a loop. This loop calculates one or more accumulator matrices.
Every iteration calculates the partial result of the accumulator, while iterating over different factor matrices.

However, some shaders do not use a loop.

Suppose we have multiple wmma instructions. They all use the same C matrix. In our case, they all use a zero matrix as input at first, because there is no previous result.

v_mov v24, 0
; ...
; v[24:31] has all zeros
v_wmma_f16_16x16x16_f16 v[0:7],   ..., v[24:31]
v_wmma_f16_16x16x16_f16 v[32:39], ..., v[24:31]

Since these are the wmma_f16 instructions, they write the result into the lower 16 bit of the result registers.
Therefore, after the instructions, the content of v0 is 0xIJKL????, where IJKL is the result of the wmma, while the ???? is the previous content of v0, from before the wmma.
In other words, even though the input accumulator has all zeros in its registers, these zeros are not copied into the result registers.

Then, we have other wmmas, which update the result matrices. We take the previous result as input accumulator, and swap out the factor matrices for different ones.

v_wmma_f16_16x16x16_f16 v[0:7],   ..., v[0:7]
v_wmma_f16_16x16x16_f16 v[32:39], ..., v[32:39]

After that, the content of v0 is 0xPQRS????.

Usually, this is not an issue. But the future patch tries to fill the ???? with the values of another matrix, so we can save VGPRs.
We can do that thanks to the op_sel argument. We want the matrix values inside v[32:39] to be inside v[0:7].
Essentally, we want this:

v_wmma_f16_16x16x16_f16 v[0:7],   ..., v[0:7]
v_wmma_f16_16x16x16_f16 v[0:7],   ..., v[0:7] op_sel [0, 0, 1] ; instead of v[32:39]

The second instructions reads from the upper 16 bit of v0, and also writes the result into the upper 16 bit of v0.
That normally does not cause an issue. However, when we intially calculated v0 with the first wmma, we had the content 0xIJKL????.
Now, any wmma with the op_sel tries to read the upper sixteen bit, the ???? as input accumulator. These upper bits were not initialized to zeros.

That is the root of the issue, since now the previous content of the registers is added to the wmma result, and not 0 as expected.

Tying the input accumulator to the result accumulator solves this issue. We no longer have a separate zero matrix, which we reuse over multiple wmmas.
Instead, we correctly initialize the output matrix to all zeros and then calculate the results.

v_mov v0, 0
; ...
v_wmma_f16_16x16x16_f16 v[0:7],   ..., v[0:7]

After that, the content of v0 is now 0xIJKL0000, as expected.

Does that clear things up? Let me know if I skipped some detail.

[arsenm]

This shouldn't call for introducing new intrinsics

[arsenm]

Is the problem the rematerialization doesn't understand the liveness with op_sel?

[OutOfCache]

I'd be happy to change the approach, but I can't think of a better way to preserve the old behavior while also guaranteeing the correct initialization of register values.
I assume the current behavior of wmma (only writing to one half of the register while leaving the other half untouched) is correct, or should it copy the content of the input accumulator into the other half?

[sebastian-ne]

I’ll take a try at explaining what the new tied-intrinsic does and where it is useful:

Our frontend sees matrix multiplications of multiple, different 16-bit matrices. Each of these matrices takes 8 VGPRs. However, a 16-bit matrix only uses either the high or the low half of each of the 8 VGPRs.
So, what our compiler tries to do, is merging a pair of two independent 16-bit matrices into 8 VGPRs. One matrix taking the low half of each register, one matrix taking the high half.

In IR, we have intrinsics that work on such a combined matrix-pair:

%combined = call <8 x float> @combine_halfs(<8 x float> %lo_16_bit_matrix, <8 x float> %hi_16_bit_matrix)  ; type is like <16 x half> with every second half in use
%low_half_multiplied = call <8 x float> @wmma(%a, %b, <8 x float> %combined, i1 false /* low half */)
%high_half_multiplied = call <8 x float> @wmma(%a, %b, <8 x float> %low_half_multiplied, i1 true /* high half */)

So far so good, we divide VGPR usage by two, now we need to lower our @wmma intrinsic to llvm.amdgcn intrinsics.
For this, we need a wmma intrinsic that uses the low or high half of our value as accumulator (c matrix) and preserves the other half of the value.
If we do not preserve the other half, we would loose the second part of the packed matrix.

This is where the new tied intrinsic comes into play, it guarantees to preserve the “untouched” part of the value by tying input and output to the same physical VGPRs.

[piotr]

Thanks for the extra info; I understand the problem now: currently there seems to be no way to take advantage of the opsel bit to reuse the same destination matrix registers for two wmma instructions.

One way to fix that would be as proposed here, at the intrinsic level. In this approach the intrinsic would always take the full register (C, D matrices), operate on the specified half, and preserve the other half.

But maybe this can be fixed in the codegen without adding the new intrinsic? I think the underlying problem is that we are not modelling correctly the fact that these opsel instructions do not touch the other half. If we could do that I would expect the two address pass to re-use the dest regs of the first instruction, because the low halves would still be live at the end of the second instruction.

%combined = call <8 x float> @combine_halfs(<8 x float> %lo_16_bit_matrix, <8 x float> %hi_16_bit_matrix)
%low_half_multiplied = call <8 x float> @wmma(%a, %b, <8 x float> %combined, i1 false /* low half */)
%high_half_multiplied = call <8 x float> @wmma(%a, %b, <8 x float> %combined, i1 true /* high half */)

[OutOfCache]

That's true @piotr, that is the underlying issue. The twoaddress pass needs to somehow notice this behavior. It makes sense to tackle this problem at the root, so I will look into the current decision process in the pass and think about solutions.

I am open to hear any suggestions. Has there been a case similar to this before?

[sebastian-ne]

In D158059#4643932, @piotr wrote:

Thanks for the extra info; I understand the problem now: currently there seems to be no way to take advantage of the opsel bit to reuse the same destination matrix registers for two wmma instructions.

One way to fix that would be as proposed here, at the intrinsic level. In this approach the intrinsic would always take the full register (C, D matrices), operate on the specified half, and preserve the other half.

But maybe this can be fixed in the codegen without adding the new intrinsic? I think the underlying problem is that we are not modelling correctly the fact that these opsel instructions do not touch the other half. If we could do that I would expect the two address pass to re-use the dest regs of the first instruction, because the low halves would still be live at the end of the second instruction.

So, if you have multiple packed operations in a row, you would need to “re-combine” the matrices?
I.e.

%combined = call <8 x float> @combine_halfs(<8 x float> %lo_16_bit_matrix, <8 x float> %hi_16_bit_matrix)
%low_half_multiplied = call <8 x float> @wmma(%a, %b, <8 x float> %combined, i1 false /* low half */)
%high_half_multiplied = call <8 x float> @wmma(%a, %b, <8 x float> %combined, i1 true /* high half */)
%combined2 = call <8 x float> @combine_halfs(<8 x float> %low_half_multiplied, <8 x float> %high_half_multiplied)
%low_half_multiplied2 = call <8 x float> @wmma(%a, %b, <8 x float> %combined2, i1 false /* low half */)
%high_half_multiplied2 = call <8 x float> @wmma(%a, %b, <8 x float> %combined2, i1 true /* high half */)

Then, as you say, our register allocation needs to be intelligent enough to keep the matrices packed.
How would you define the instructions for this to work?

[piotr]

Then, as you say, our register allocation needs to be intelligent enough to keep the matrices packed.
How would you define the instructions for this to work?

Unfortunately, looking at that a bit more I don't think the scheme I proposed is feasible. Even if we add some extra copies to preserve the other half, the twoaddressinstruction pass will not be able to understand that.

The only alternative I could suggest instead of adding new intrinsics seems to be to implement the packing entirely in the codegen (e.g. after twoaddressinstruction pass).

[OutOfCache]

In D158059#4647410, @piotr wrote:

Then, as you say, our register allocation needs to be intelligent enough to keep the matrices packed.
How would you define the instructions for this to work?

Unfortunately, looking at that a bit more I don't think the scheme I proposed is feasible. Even if we add some extra copies to preserve the other half, the twoaddressinstruction pass will not be able to understand that.

The only alternative I could suggest instead of adding new intrinsics seems to be to implement the packing entirely in the codegen (e.g. after twoaddressinstruction pass).

Thank you for looking into it! I don't know if the packing is feasible at that stage, though. The packing affects the users of the matrices as well, and we base our solution heavily on finding the users of the lgc intrinisics. Plus, the changes can remove some intrinsic calls entirely, so the code size is reduced in any further pass. Delaying the changes to such a late stage makes the packing process more complicated and we would keep more code in each pass, which might be removed later on.

llvm/lib/Target/AMDGPU/VOP3PInstructions.td

 // 2) The pass twoaddressinstruction checks if src2 is live and if that is the case
 //    it converts the default pseudo to the pseudo where src2 is not the same as vdst.
 // 3) @earlyclobber on the destination satisfies the constraint during RA.
 
 multiclass WMMAInst<string Suffix, string Instr, VOPProfile P, SDPatternOperator node = null_frag, RegisterOperand _Src01RC64 = VRegSrc_256, WMMAType Type> {
 
   defvar WMMAConstraints2Addr = "@earlyclobber $vdst,$vdst = $src2";
   defvar WMMAConstraints3Addr = "@earlyclobber $vdst";
+  defvar isConvertableTo3Addr = !cond(!eq(Instr, "v_wmma_f16_16x16x16_f16"): 0, true: 1);
 
   defvar WMMAProfile = VOPProfileWMMA<P, Suffix, _Src01RC64, Type.hasClamp, Type.hasOpsel>;
   if !eq(Suffix, "_w32") then {
     let Mnemonic = Instr, mayRaiseFPException = 0, ReadsModeReg = 0 in {
-      let Constraints = WMMAConstraints2Addr, isConvertibleToThreeAddress = 1 in {
+      let Constraints = WMMAConstraints2Addr, isConvertibleToThreeAddress = isConvertableTo3Addr in {
         def _twoaddr_w32 : VOP3P_Pseudo<Instr # Suffix, WMMAProfile>;
       }
       let Constraints = WMMAConstraints3Addr, SchedRW = [Write32Bit, Write32Bit] in {
         def _threeaddr_w32 : VOP3P_Pseudo<Instr # Suffix, WMMAProfile>;
       }
     }
     def : WMMAOpcodeMapping<!cast<Instruction>(NAME # _twoaddr_w32),
                             !cast<Instruction>(NAME # _threeaddr_w32)>;
   } else if !eq(Suffix, "_w64") then {
     let Mnemonic = Instr, mayRaiseFPException = 0, ReadsModeReg = 0 in {
-      let Constraints = WMMAConstraints2Addr, isConvertibleToThreeAddress = 1 in {
+      let Constraints = WMMAConstraints2Addr, isConvertibleToThreeAddress = isConvertableTo3Addr in {
         def _twoaddr_w64 : VOP3P_Pseudo<Instr # Suffix, WMMAProfile>;
       }
       let Constraints = WMMAConstraints3Addr, SchedRW = [Write32Bit, Write32Bit] in {
         def _threeaddr_w64 : VOP3P_Pseudo<Instr # Suffix, WMMAProfile>;
       }
     }
     def : WMMAOpcodeMapping<!cast<Instruction>(NAME # _twoaddr_w64),
                             !cast<Instruction>(NAME # _threeaddr_w64)>;

llvm/test/CodeGen/AMDGPU/twoaddr-wmma.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 2
 # RUN: llc -march=amdgcn -mcpu=gfx1100 %s -run-pass twoaddressinstruction -verify-machineinstrs -o - | FileCheck -check-prefix=GCN %s
 
-# GCN-LABEL: name: test_v_wmma_f32_16x16x16_f16_twoaddr_w32
-# GCN: early-clobber %2:vreg_256 = V_WMMA_F32_16X16X16_F16_threeaddr_w32 8, killed %1, 8, killed %1, 8, %0, 0, 0, implicit $exec
-
 ---
 name:            test_v_wmma_f32_16x16x16_f16_twoaddr_w32
 registers:
   - { id: 0, class: vreg_256 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_256 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_f32_16x16x16_f16_twoaddr_w32
+    ; GCN: [[DEF:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_256 = V_WMMA_F32_16X16X16_F16_threeaddr_w32 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_256 = V_WMMA_F32_16X16X16_F16_twoaddr_w32 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_256, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_f32_16x16x16_bf16_twoaddr_w32
-# GCN: early-clobber %2:vreg_256 = V_WMMA_F32_16X16X16_BF16_threeaddr_w32 8, killed %1, 8, killed %1, 8, %0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_f32_16x16x16_bf16_twoaddr_w32
 registers:
   - { id: 0, class: vreg_256 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_256 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_f32_16x16x16_bf16_twoaddr_w32
+    ; GCN: [[DEF:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_256 = V_WMMA_F32_16X16X16_BF16_threeaddr_w32 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_256 = V_WMMA_F32_16X16X16_BF16_twoaddr_w32 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_256, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_f16_16x16x16_f16_twoaddr_w32
-# GCN: early-clobber %2:vreg_256 = V_WMMA_F16_16X16X16_F16_threeaddr_w32 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_f16_16x16x16_f16_twoaddr_w32
 registers:
   - { id: 0, class: vreg_256 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_256 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_f16_16x16x16_f16_twoaddr_w32
+    ; GCN: [[DEF:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[COPY:%[0-9]+]]:vreg_256 = COPY [[DEF]]
+    ; GCN-NEXT: early-clobber [[COPY]]:vreg_256 = V_WMMA_F16_16X16X16_F16_twoaddr_w32 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[COPY]], 0, 0, 0, 0, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_256 = V_WMMA_F16_16X16X16_F16_twoaddr_w32 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_256, 0, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_bf16_16x16x16_bf16_twoaddr_w32
-# GCN: early-clobber %2:vreg_256 = V_WMMA_BF16_16X16X16_BF16_threeaddr_w32 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_bf16_16x16x16_bf16_twoaddr_w32
 registers:
   - { id: 0, class: vreg_256 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_256 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_bf16_16x16x16_bf16_twoaddr_w32
+    ; GCN: [[DEF:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_256 = V_WMMA_BF16_16X16X16_BF16_threeaddr_w32 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_256 = V_WMMA_BF16_16X16X16_BF16_twoaddr_w32 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_256, 0, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu8_twoaddr_w32
-# GCN: early-clobber %2:vreg_256 = V_WMMA_I32_16X16X16_IU8_threeaddr_w32 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_i32_16x16x16_iu8_twoaddr_w32
 registers:
   - { id: 0, class: vreg_256 }
   - { id: 1, class: vreg_128 }
   - { id: 2, class: vreg_256 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu8_twoaddr_w32
+    ; GCN: [[DEF:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_256 = V_WMMA_I32_16X16X16_IU8_threeaddr_w32 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_256 = V_WMMA_I32_16X16X16_IU8_twoaddr_w32 8, killed %1:vreg_128, 8, killed %1:vreg_128, 8, %0:vreg_256, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu4_twoaddr_w32
-# GCN: early-clobber %2:vreg_256 = V_WMMA_I32_16X16X16_IU4_threeaddr_w32 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_i32_16x16x16_iu4_twoaddr_w32
 registers:
   - { id: 0, class: vreg_256 }
   - { id: 1, class: vreg_64 }
   - { id: 2, class: vreg_256 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu4_twoaddr_w32
+    ; GCN: [[DEF:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_64 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_256 = V_WMMA_I32_16X16X16_IU4_threeaddr_w32 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_256 = V_WMMA_I32_16X16X16_IU4_twoaddr_w32 8, killed %1:vreg_64, 8, killed %1:vreg_64, 8, %0:vreg_256, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_f32_16x16x16_f16_twoaddr_w64
-# GCN: early-clobber %2:vreg_128 = V_WMMA_F32_16X16X16_F16_threeaddr_w64 8, killed %1, 8, killed %1, 8, %0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_f32_16x16x16_f16_twoaddr_w64
 registers:
   - { id: 0, class: vreg_128 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_128 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_f32_16x16x16_f16_twoaddr_w64
+    ; GCN: [[DEF:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_128 = V_WMMA_F32_16X16X16_F16_threeaddr_w64 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_128 = V_WMMA_F32_16X16X16_F16_twoaddr_w64 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_128, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_f32_16x16x16_bf16_twoaddr_w64
-# GCN: early-clobber %2:vreg_128 = V_WMMA_F32_16X16X16_BF16_threeaddr_w64 8, killed %1, 8, killed %1, 8, %0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_f32_16x16x16_bf16_twoaddr_w64
 registers:
   - { id: 0, class: vreg_128 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_128 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_f32_16x16x16_bf16_twoaddr_w64
+    ; GCN: [[DEF:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_128 = V_WMMA_F32_16X16X16_BF16_threeaddr_w64 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_128 = V_WMMA_F32_16X16X16_BF16_twoaddr_w64 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_128, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_f16_16x16x16_f16_twoaddr_w64
-# GCN: early-clobber %2:vreg_128 = V_WMMA_F16_16X16X16_F16_threeaddr_w64 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_f16_16x16x16_f16_twoaddr_w64
 registers:
   - { id: 0, class: vreg_128 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_128 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_f16_16x16x16_f16_twoaddr_w64
+    ; GCN: [[DEF:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: [[COPY:%[0-9]+]]:vreg_128 = COPY [[DEF]]
+    ; GCN-NEXT: early-clobber [[COPY]]:vreg_128 = V_WMMA_F16_16X16X16_F16_twoaddr_w64 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[COPY]], 0, 0, 0, 0, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_128 = V_WMMA_F16_16X16X16_F16_twoaddr_w64 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_128, 0, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_bf16_16x16x16_bf16_twoaddr_w64
-# GCN: early-clobber %2:vreg_128 = V_WMMA_BF16_16X16X16_BF16_threeaddr_w64 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_bf16_16x16x16_bf16_twoaddr_w64
 registers:
   - { id: 0, class: vreg_128 }
   - { id: 1, class: vreg_256 }
   - { id: 2, class: vreg_128 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_bf16_16x16x16_bf16_twoaddr_w64
+    ; GCN: [[DEF:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_256 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_128 = V_WMMA_BF16_16X16X16_BF16_threeaddr_w64 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_128 = V_WMMA_BF16_16X16X16_BF16_twoaddr_w64 8, killed %1:vreg_256, 8, killed %1:vreg_256, 8, %0:vreg_128, 0, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu8_twoaddr_w64
-# GCN: early-clobber %2:vreg_128 = V_WMMA_I32_16X16X16_IU8_threeaddr_w64 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_i32_16x16x16_iu8_twoaddr_w64
 registers:
   - { id: 0, class: vreg_128 }
   - { id: 1, class: vreg_128 }
   - { id: 2, class: vreg_128 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu8_twoaddr_w64
+    ; GCN: [[DEF:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_128 = V_WMMA_I32_16X16X16_IU8_threeaddr_w64 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_128 = V_WMMA_I32_16X16X16_IU8_twoaddr_w64 8, killed %1:vreg_128, 8, killed %1:vreg_128, 8, %0:vreg_128, 0, 0, 0, implicit $exec
 
 ...
 
-# GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu4_twoaddr_w64
-# GCN: early-clobber %2:vreg_128 = V_WMMA_I32_16X16X16_IU4_threeaddr_w64 8, killed %1, 8, killed %1, 8, %0, 0, 0, 0, implicit $exec
 
 ---
 name:            test_v_wmma_i32_16x16x16_iu4_twoaddr_w64
 registers:
   - { id: 0, class: vreg_128 }
   - { id: 1, class: vreg_64 }
   - { id: 2, class: vreg_128 }
 body:             |
   bb.0:
 
+    ; GCN-LABEL: name: test_v_wmma_i32_16x16x16_iu4_twoaddr_w64
+    ; GCN: [[DEF:%[0-9]+]]:vreg_128 = IMPLICIT_DEF
+    ; GCN-NEXT: [[DEF1:%[0-9]+]]:vreg_64 = IMPLICIT_DEF
+    ; GCN-NEXT: early-clobber %2:vreg_128 = V_WMMA_I32_16X16X16_IU4_threeaddr_w64 8, killed [[DEF1]], 8, killed [[DEF1]], 8, [[DEF]], 0, 0, 0, implicit $exec, implicit $exec
     %0 = IMPLICIT_DEF
     %1 = IMPLICIT_DEF
     early-clobber %2:vreg_128 = V_WMMA_I32_16X16X16_IU4_twoaddr_w64 8, killed %1:vreg_64, 8, killed %1:vreg_64, 8, %0:vreg_128, 0, 0, 0, implicit $exec
 
 ...

llvm/test/CodeGen/AMDGPU/wmma_multiple_32.ll

bb:
   ret void
 }
 
 ; @llvm.amdgcn.wmma.f16.16x16x16.f16
 
 define amdgpu_ps void @test_wmma_f16_16x16x16_f16_lo(<16 x half> %A, <16 x half> %B, <16 x half> %C, ptr addrspace(1) %out, ptr addrspace(1) %out2) {
 ; W32-LABEL: test_wmma_f16_16x16x16_f16_lo:
 ; W32:       ; %bb.0: ; %bb
-; W32-NEXT:    v_wmma_f16_16x16x16_f16 v[28:35], v[0:7], v[8:15], v[16:23]
+; W32-NEXT:    v_dual_mov_b32 v35, v23 :: v_dual_mov_b32 v34, v22
+; W32-NEXT:    v_dual_mov_b32 v33, v21 :: v_dual_mov_b32 v32, v20
+; W32-NEXT:    v_dual_mov_b32 v31, v19 :: v_dual_mov_b32 v30, v18
+; W32-NEXT:    v_dual_mov_b32 v29, v17 :: v_dual_mov_b32 v28, v16
 ; W32-NEXT:    v_wmma_f16_16x16x16_f16 v[16:23], v[8:15], v[8:15], v[16:23]
+; W32-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; W32-NEXT:    v_wmma_f16_16x16x16_f16 v[28:35], v[0:7], v[8:15], v[28:35]
 ; W32-NEXT:    s_clause 0x1
 ; W32-NEXT:    global_store_b128 v[24:25], v[32:35], off offset:16
 ; W32-NEXT:    global_store_b128 v[24:25], v[28:31], off
 ; W32-NEXT:    s_clause 0x1
 ; W32-NEXT:    global_store_b128 v[26:27], v[20:23], off offset:16
 ; W32-NEXT:    global_store_b128 v[26:27], v[16:19], off
 ; W32-NEXT:    s_nop 0
 ; W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; W32-NEXT:    s_endpgm
 bb:
   %res = call <16 x half> @llvm.amdgcn.wmma.f16.16x16x16.f16(<16 x half> %A, <16 x half> %B, <16 x half> %C, i1 0)
   %res2 = call <16 x half> @llvm.amdgcn.wmma.f16.16x16x16.f16(<16 x half> %B, <16 x half> %B, <16 x half> %C, i1 0)
   store <16 x half> %res, ptr addrspace(1) %out, align 32
   store <16 x half> %res2, ptr addrspace(1) %out2, align 32
   ret void
 }
 
 define amdgpu_ps void @test_wmma_f16_16x16x16_f16_hi(<16 x half> %A, <16 x half> %B, <16 x half> %C, ptr addrspace(1) %out, ptr addrspace(1) %out2) {
 ; W32-LABEL: test_wmma_f16_16x16x16_f16_hi:
 ; W32:       ; %bb.0: ; %bb
-; W32-NEXT:    v_wmma_f16_16x16x16_f16 v[28:35], v[0:7], v[8:15], v[16:23] op_sel:[0,0,1]
+; W32-NEXT:    v_dual_mov_b32 v35, v23 :: v_dual_mov_b32 v34, v22
+; W32-NEXT:    v_dual_mov_b32 v33, v21 :: v_dual_mov_b32 v32, v20
+; W32-NEXT:    v_dual_mov_b32 v31, v19 :: v_dual_mov_b32 v30, v18
+; W32-NEXT:    v_dual_mov_b32 v29, v17 :: v_dual_mov_b32 v28, v16
 ; W32-NEXT:    v_wmma_f16_16x16x16_f16 v[16:23], v[8:15], v[8:15], v[16:23] op_sel:[0,0,1]
+; W32-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; W32-NEXT:    v_wmma_f16_16x16x16_f16 v[28:35], v[0:7], v[8:15], v[28:35] op_sel:[0,0,1]
 ; W32-NEXT:    s_clause 0x1
 ; W32-NEXT:    global_store_b128 v[24:25], v[32:35], off offset:16
 ; W32-NEXT:    global_store_b128 v[24:25], v[28:31], off
 ; W32-NEXT:    s_clause 0x1
 ; W32-NEXT:    global_store_b128 v[26:27], v[20:23], off offset:16
 ; W32-NEXT:    global_store_b128 v[26:27], v[16:19], off
 ; W32-NEXT:    s_nop 0
 ; W32-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)

llvm/test/CodeGen/AMDGPU/wmma_multiple_64.ll

bb:
   ret void
 }
 
 ; @llvm.amdgcn.wmma.f16.16x16x16.f16
 
 define amdgpu_ps void @test_wmma_f16_16x16x16_f16_lo(<16 x half> %A, <16 x half> %B, <8 x half> %C, ptr addrspace(1) %out, ptr addrspace(1) %out2) {
 ; W64-LABEL: test_wmma_f16_16x16x16_f16_lo:
 ; W64:       ; %bb.0: ; %bb
-; W64-NEXT:    v_wmma_f16_16x16x16_f16 v[24:27], v[0:7], v[8:15], v[16:19]
+; W64-NEXT:    v_mov_b32_e32 v27, v19
+; W64-NEXT:    v_mov_b32_e32 v26, v18
+; W64-NEXT:    v_mov_b32_e32 v25, v17
+; W64-NEXT:    v_mov_b32_e32 v24, v16
 ; W64-NEXT:    v_wmma_f16_16x16x16_f16 v[16:19], v[8:15], v[8:15], v[16:19]
+; W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; W64-NEXT:    v_wmma_f16_16x16x16_f16 v[24:27], v[0:7], v[8:15], v[24:27]
 ; W64-NEXT:    global_store_b128 v[20:21], v[24:27], off
 ; W64-NEXT:    global_store_b128 v[22:23], v[16:19], off
 ; W64-NEXT:    s_nop 0
 ; W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; W64-NEXT:    s_endpgm
 bb:
   %res = call <8 x half> @llvm.amdgcn.wmma.f16.16x16x16.f16(<16 x half> %A, <16 x half> %B, <8 x half> %C, i1 0)
   %res2 = call <8 x half> @llvm.amdgcn.wmma.f16.16x16x16.f16(<16 x half> %B, <16 x half> %B, <8 x half> %C, i1 0)
   store <8 x half> %res, ptr addrspace(1) %out, align 16
   store <8 x half> %res2, ptr addrspace(1) %out2, align 16
   ret void
 }
 
 define amdgpu_ps void @test_wmma_f16_16x16x16_f16_hi(<16 x half> %A, <16 x half> %B, <8 x half> %C, ptr addrspace(1) %out, ptr addrspace(1) %out2) {
 ; W64-LABEL: test_wmma_f16_16x16x16_f16_hi:
 ; W64:       ; %bb.0: ; %bb
-; W64-NEXT:    v_wmma_f16_16x16x16_f16 v[24:27], v[0:7], v[8:15], v[16:19] op_sel:[0,0,1]
+; W64-NEXT:    v_mov_b32_e32 v27, v19
+; W64-NEXT:    v_mov_b32_e32 v26, v18
+; W64-NEXT:    v_mov_b32_e32 v25, v17
+; W64-NEXT:    v_mov_b32_e32 v24, v16
 ; W64-NEXT:    v_wmma_f16_16x16x16_f16 v[16:19], v[8:15], v[8:15], v[16:19] op_sel:[0,0,1]
+; W64-NEXT:    s_delay_alu instid0(VALU_DEP_2)
+; W64-NEXT:    v_wmma_f16_16x16x16_f16 v[24:27], v[0:7], v[8:15], v[24:27] op_sel:[0,0,1]
 ; W64-NEXT:    global_store_b128 v[20:21], v[24:27], off
 ; W64-NEXT:    global_store_b128 v[22:23], v[16:19], off
 ; W64-NEXT:    s_nop 0
 ; W64-NEXT:    s_sendmsg sendmsg(MSG_DEALLOC_VGPRS)
 ; W64-NEXT:    s_endpgm
 bb:
   %res = call <8 x half> @llvm.amdgcn.wmma.f16.16x16x16.f16(<16 x half> %A, <16 x half> %B, <8 x half> %C, i1 1)
   %res2 = call <8 x half> @llvm.amdgcn.wmma.f16.16x16x16.f16(<16 x half> %B, <16 x half> %B, <8 x half> %C, i1 1)