AMDGPU: Fix illegally constant folding from V_MOV_B32_sdwa

[arsenm]

Summary

This was assumed to be a simple move, and interpreting the immediate
modifier operand as a materialized immediate. Apparently the SDWA pass
never produces these, but GlobalISel does emit these for some vector
shuffles.

[rampitec]

Technically even SDWA or DPP for can move immediate, aren't they? I.e.:

v_mov_b32_sdwa v0, 1 dst_sel:DWORD dst_unused:UNUSED_PRESERVE src0_sel:DWORD ; encoding: [0xf9,0x02,0x00,0x7e,0x81,0x16,0x86,0x00]

Should not SDWA peephole be fixed instead? Interpretation of modifiers as materialized immediate seems to be just a bug.

Moreover, we seem to have another bug in v_mov_b32:

<stdin>:1:15: error: not a valid operand.
v_mov_b32 v0, -v1

This is a legal form but we do not support it. If we fix that then we probably will have issue with src0_modifiers again, right?

[arsenm]

In D80065#2042073, @rampitec wrote:

Technically even SDWA or DPP for can move immediate, aren't they? I.e.:

v_mov_b32_sdwa v0, 1 dst_sel:DWORD dst_unused:UNUSED_PRESERVE src0_sel:DWORD ; encoding: [0xf9,0x02,0x00,0x7e,0x81,0x16,0x86,0x00]

Should not SDWA peephole be fixed instead? Interpretation of modifiers as materialized immediate seems to be just a bug.

We shouldn't emit a simple immediate materialize using SDWA in the first place, so it's not worth the complexity of trying to handle it

Moreover, we seem to have another bug in v_mov_b32:

<stdin>:1:15: error: not a valid operand.
v_mov_b32 v0, -v1

This is a legal form but we do not support it. If we fix that then we probably will have issue with src0_modifiers again, right?

I think I saw a note before that v_mov_b32 doesn't actually support a VOP3 encoding, so I'm not sure it's valid. We never emit V_MOV_B32_e64 anyway

[rampitec]

In D80065#2042113, @arsenm wrote:

In D80065#2042073, @rampitec wrote:

Technically even SDWA or DPP for can move immediate, aren't they? I.e.:

v_mov_b32_sdwa v0, 1 dst_sel:DWORD dst_unused:UNUSED_PRESERVE src0_sel:DWORD ; encoding: [0xf9,0x02,0x00,0x7e,0x81,0x16,0x86,0x00]

Should not SDWA peephole be fixed instead? Interpretation of modifiers as materialized immediate seems to be just a bug.

We shouldn't emit a simple immediate materialize using SDWA in the first place, so it's not worth the complexity of trying to handle it

Moreover, we seem to have another bug in v_mov_b32:

<stdin>:1:15: error: not a valid operand.
v_mov_b32 v0, -v1

This is a legal form but we do not support it. If we fix that then we probably will have issue with src0_modifiers again, right?

I think I saw a note before that v_mov_b32 doesn't actually support a VOP3 encoding, so I'm not sure it's valid. We never emit V_MOV_B32_e64 anyway

It looks like it does support VOP3 form. I can encode 'v_mov_b32 v0, 100'.

[arsenm]

In D80065#2042115, @rampitec wrote:

In D80065#2042113, @arsenm wrote:

In D80065#2042073, @rampitec wrote:

Technically even SDWA or DPP for can move immediate, aren't they? I.e.:

v_mov_b32_sdwa v0, 1 dst_sel:DWORD dst_unused:UNUSED_PRESERVE src0_sel:DWORD ; encoding: [0xf9,0x02,0x00,0x7e,0x81,0x16,0x86,0x00]

Should not SDWA peephole be fixed instead? Interpretation of modifiers as materialized immediate seems to be just a bug.

We shouldn't emit a simple immediate materialize using SDWA in the first place, so it's not worth the complexity of trying to handle it

Moreover, we seem to have another bug in v_mov_b32:

<stdin>:1:15: error: not a valid operand.
v_mov_b32 v0, -v1

This is a legal form but we do not support it. If we fix that then we probably will have issue with src0_modifiers again, right?

I think I saw a note before that v_mov_b32 doesn't actually support a VOP3 encoding, so I'm not sure it's valid. We never emit V_MOV_B32_e64 anyway

It looks like it does support VOP3 form. I can encode 'v_mov_b32 v0, 100'.

That's still VOP1 with literal

[rampitec]

In D80065#2042189, @arsenm wrote:

It looks like it does support VOP3 form. I can encode 'v_mov_b32 v0, 100'.

That's still VOP1 with literal

I have to disagree. It has VOP1 and VOP3 encodings.

[rampitec]

Ugh. I have found that modifiers can be encoded, but do not work :( LGTM then.

[arsenm]

b27a538dda4caf2752ed8c6c731361ddf9458b87

llvm/lib/Target/AMDGPU/VOP1Instructions.td

!if(P.HasModifiers,
                                                   i1:$clamp, i32:$omod))))],
             [(set P.DstVT:$vdst, (node P.Src0VT:$src0))]
         )
     );
 }
 
 multiclass VOP1Inst <string opName, VOPProfile P,
                      SDPatternOperator node = null_frag> {
+  // We only want to set this on the basic, non-SDWA or DPP forms.
+  defvar should_mov_imm = !eq(opName, "v_mov_b32");
+
+  let isMoveImm = should_mov_imm in {
     def _e32 : VOP1_Pseudo <opName, P>;
     def _e64 : VOP3_Pseudo <opName, P, getVOP1Pat64<node, P>.ret>;
+  }
 
   foreach _ = BoolToList<P.HasExtSDWA>.ret in
     def _sdwa : VOP1_SDWA_Pseudo <opName, P>;
 
   foreach _ = BoolToList<P.HasExtDPP>.ret in
     def _dpp : VOP1_DPP_Pseudo <opName, P>;
 
   def : MnemonicAlias<opName#"_e32", opName>, LetDummies;

 //===----------------------------------------------------------------------===//
 // VOP1 Instructions
 //===----------------------------------------------------------------------===//
 
 let VOPAsmPrefer32Bit = 1 in {
 defm V_NOP : VOP1Inst <"v_nop", VOP_NONE>;
 }
 
-let isMoveImm = 1, isReMaterializable = 1, isAsCheapAsAMove = 1 in {
+let isReMaterializable = 1, isAsCheapAsAMove = 1 in {
 defm V_MOV_B32 : VOP1Inst <"v_mov_b32", VOP_I32_I32>;
 } // End isMoveImm = 1
 
 // FIXME: Specify SchedRW for READFIRSTLANE_B32
 // TODO: Make profile for this, there is VOP3 encoding also
 def V_READFIRSTLANE_B32 :
   InstSI <(outs SReg_32:$vdst),
     (ins VRegOrLds_32:$src0),

llvm/test/CodeGen/AMDGPU/atomic_optimizations_local_pointer.ll

 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_and_b32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_and_b32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_and_b32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_or_b32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_or_b32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_or_b32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_or_b32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_xor_b32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_xor_b32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_xor_b32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_xor_b32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_max_i32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_max_i32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_max_i32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_min_i32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_min_i32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_min_i32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_max_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_max_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_max_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_max_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

 ; GFX1064-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_min_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_min_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s2, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s2
 ; GFX1064-NEXT:    v_min_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s2, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 31
 ; GFX1064-NEXT:    v_readlane_b32 s6, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s2, 16
 ; GFX1064-NEXT:    s_mov_b32 s2, -1
 ; GFX1064-NEXT:    v_writelane_b32 v1, s3, 32
 ; GFX1064-NEXT:    v_readlane_b32 s3, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s6, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[4:5]

llvm/test/CodeGen/AMDGPU/atomic_optimizations_pixelshader.ll

 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1064-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1064-NEXT:    v_readlane_b32 s12, v2, 31
 ; GFX1064-NEXT:    v_mov_b32_e32 v3, s12
 ; GFX1064-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xc bank_mask:0xf
-; GFX1064-NEXT:    v_readlane_b32 s12, v2, 15
 ; GFX1064-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1064-NEXT:    v_readlane_b32 s12, v2, 15
 ; GFX1064-NEXT:    v_readlane_b32 s13, v2, 31
 ; GFX1064-NEXT:    v_writelane_b32 v1, s12, 16
 ; GFX1064-NEXT:    v_readlane_b32 s12, v2, 63
 ; GFX1064-NEXT:    v_writelane_b32 v1, s13, 32
 ; GFX1064-NEXT:    v_readlane_b32 s13, v2, 47
 ; GFX1064-NEXT:    v_writelane_b32 v1, s13, 48
 ; GFX1064-NEXT:    s_mov_b64 exec, s[10:11]
 ; GFX1064-NEXT:    v_cmp_eq_u32_e32 vcc, 0, v0

 ; GFX1032-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:1 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1032-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:2 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1032-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:4 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1032-NEXT:    v_add_nc_u32_dpp v2, v2, v2 row_shr:8 row_mask:0xf bank_mask:0xf bound_ctrl:0
 ; GFX1032-NEXT:    v_mov_b32_e32 v3, v2
 ; GFX1032-NEXT:    v_permlanex16_b32 v3, v3, -1, -1
 ; GFX1032-NEXT:    v_add_nc_u32_dpp v2, v3, v2 quad_perm:[0,1,2,3] row_mask:0xa bank_mask:0xf
 ; GFX1032-NEXT:    v_readlane_b32 s10, v2, 31
-; GFX1032-NEXT:    v_readlane_b32 s11, v2, 15
 ; GFX1032-NEXT:    v_mov_b32_dpp v1, v2 row_shr:1 row_mask:0xf bank_mask:0xf
+; GFX1032-NEXT:    v_readlane_b32 s11, v2, 15
 ; GFX1032-NEXT:    v_writelane_b32 v1, s11, 16
 ; GFX1032-NEXT:    s_mov_b32 exec_lo, s9
 ; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
 ; GFX1032-NEXT:    ; implicit-def: $vgpr0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s9, vcc_lo
 ; GFX1032-NEXT:    s_cbranch_execz BB1_3
 ; GFX1032-NEXT:  ; %bb.2:
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, s10

llvm/test/CodeGen/AMDGPU/constant-fold-imm-immreg.mir

bb.0:
 
     %0:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
     %1:vgpr_32 = V_MOV_B32_e32 16, implicit $exec
     %2:vgpr_32 = V_MOV_B32_e32 12345, implicit $exec
     %3:vgpr_32 = V_AND_OR_B32 %0, %1, %2, implicit $exec
     S_ENDPGM 0, implicit %3
 
 ...
+
+# This used to incorrectly interpret V_MOV_B32_sdwa as being a move
+# immediate, and interpreting the src0_modifiers field as a
+# materialized immediate.
+
+---
+# GCN-LABEL: name: no_fold_sdwa_mov_imm
+# GCN: %2:vgpr_32 = V_MOV_B32_sdwa 0, %0, 0, 5, 2, 4, implicit $exec, implicit %0(tied-def 0)
+# GCN-NEXT: [[SHIFT:%[0-9]+]]:vgpr_32 = V_LSHRREV_B32_e64 16, %2, implicit $exec
+# GCN-NEXT: S_ENDPGM 0, implicit [[SHIFT]]
+
+name:            no_fold_sdwa_mov_imm
+alignment:       0
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+tracksRegLiveness: true
+
+body:             |
+  bb.0:
+    liveins: $vgpr0
+    %0:vgpr_32 = COPY $vgpr0
+    %1:vgpr_32 = V_MOV_B32_e32 16, implicit $exec
+    %2:vgpr_32 = V_MOV_B32_sdwa 0, %0:vgpr_32, 0, 5, 2, 4, implicit $exec, implicit %0:vgpr_32(tied-def 0)
+    %3:vgpr_32 = V_MOV_B32_e32 16, implicit $exec
+    %4:vgpr_32 = V_LSHRREV_B32_e64 %3:vgpr_32, %2:vgpr_32, implicit $exec
+    S_ENDPGM 0, implicit %4
+
+...