[AMDGPU] Make SREG_LO16 legal

[rampitec]

Summary

This includes a hack to fix illegal 32 to 16 bit copies.
The problem is when we make 16 bit subregs legal it creates
a huge amount of failures which can only be resolved at once
without a temporary hack like this.

The next step is to change operands, instruction definitions
and patterns until this hack is not needed.

There are also quite obvious regressions as seen in the tests,
but surprisingly not as huge as I expected.

I am not really sure it needs to be pushed right away, but
I definitely want to review it now and agree on the general
direction.

[arsenm]

Could we leave all instructions as 32-bit defs, but then have a 16-bit subreg copy as the only use?

%0:vgpr_32 = V_FOO_U16
%1:vgpr_16lo = COPY %0.sub16_lo
....

The register allocator would understand this and fold out the copy

llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp

315 ↗

(On Diff #259979)

This is a red flag, this should never happen

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

544–547

We should make it so that MachineCopyPropagation can deal with this case

545

Why can't you just erase the instruction?

[rampitec]

In D78829#2002912, @arsenm wrote:

Could we leave all instructions as 32-bit defs, but then have a 16-bit subreg copy as the only use?

%0:vgpr_32 = V_FOO_U16
%1:vgpr_16lo = COPY %0.sub16_lo
....

The register allocator would understand this and fold out the copy

So you want to keep SReg_32 as an RC for i16 and f16? And add a copy to each instruction producing a 16 bit value?

I am afraid it will not work. Instruction needs to produce i16 for which legal class is SReg_32, we will return SReg_lo16. Selector will either complain, not match or emit yet another copy. The problem is a 16 bit value needs to leave either in a 32 bit or a 16 bit RC, but not both.

llvm/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp

315 ↗

(On Diff #259979)

This is in place of redundant mov v0, v0 when one side is 16 bit and another 32. When all is done it should go away.

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

544–547

Hm... So you want it always? I thought this is a red flag.

545

Because caller decrements the iterator expecting new instruction to be there. It then transfers implicit operands there. This is also a hack and frankly breaks if we emit more than one instruction. Maybe we need to emit a bundle instead of a sequence.

This is also why DELETED is created.

[arsenm]

In D78829#2002946, @rampitec wrote:

In D78829#2002912, @arsenm wrote:

Could we leave all instructions as 32-bit defs, but then have a 16-bit subreg copy as the only use?

%0:vgpr_32 = V_FOO_U16
%1:vgpr_16lo = COPY %0.sub16_lo
....

The register allocator would understand this and fold out the copy

So you want to keep SReg_32 as an RC for i16 and f16? And add a copy to each instruction producing a 16 bit value?

I am afraid it will not work. Instruction needs to produce i16 for which legal class is SReg_32, we will return SReg_lo16. Selector will either complain, not match or emit yet another copy. The problem is a 16 bit value needs to leave either in a 32 bit or a 16 bit RC, but not both.

This would probably require new support in the InstrEmitter or patterns but I don't think is impossible

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

544–547

I mean MCP should eliminate these identity copies. It shouldn't see the src == dest due to 16-bit obscuring this (at least for > -O0)

[arsenm]

In D78829#2003007, @arsenm wrote:

In D78829#2002946, @rampitec wrote:

In D78829#2002912, @arsenm wrote:

Could we leave all instructions as 32-bit defs, but then have a 16-bit subreg copy as the only use?

%0:vgpr_32 = V_FOO_U16
%1:vgpr_16lo = COPY %0.sub16_lo
....

The register allocator would understand this and fold out the copy

So you want to keep SReg_32 as an RC for i16 and f16? And add a copy to each instruction producing a 16 bit value?

I am afraid it will not work. Instruction needs to produce i16 for which legal class is SReg_32, we will return SReg_lo16. Selector will either complain, not match or emit yet another copy. The problem is a 16 bit value needs to leave either in a 32 bit or a 16 bit RC, but not both.

This would probably require new support in the InstrEmitter or patterns but I don't think is impossible

This is what it really needs to look like, otherwise it's lying about producing a 16-bit result. If we pretend it really only writes 16-bits, we can't make use of the different high bit handling behaviors

[rampitec]

In D78829#2003535, @arsenm wrote:

In D78829#2003007, @arsenm wrote:

In D78829#2002946, @rampitec wrote:

In D78829#2002912, @arsenm wrote:

Could we leave all instructions as 32-bit defs, but then have a 16-bit subreg copy as the only use?

%0:vgpr_32 = V_FOO_U16
%1:vgpr_16lo = COPY %0.sub16_lo
....

The register allocator would understand this and fold out the copy

So you want to keep SReg_32 as an RC for i16 and f16? And add a copy to each instruction producing a 16 bit value?

I am afraid it will not work. Instruction needs to produce i16 for which legal class is SReg_32, we will return SReg_lo16. Selector will either complain, not match or emit yet another copy. The problem is a 16 bit value needs to leave either in a 32 bit or a 16 bit RC, but not both.

This would probably require new support in the InstrEmitter or patterns but I don't think is impossible

This is what it really needs to look like, otherwise it's lying about producing a 16-bit result. If we pretend it really only writes 16-bits, we can't make use of the different high bit handling behaviors

Hm... That's true. Source operands must be 16 bit and destination 32.

[rampitec]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

544–547

Could it? Assume:

$vgpr1 = $vgpr2_lo16
%vgpr1 = %vgpr2.lo16

It does not tell anything about the contents of a high half after the copy. Basically semantics of these is unclear. We can argue a reverse situation is a truncation, but really undefined as well because it would be unrepresentable.

[rampitec]

• Rebased.
• Dropped DELETED node, just return an empty BUNDLE.

I still do not believe we need to make SReg_LO16 legal just yet, but maybe we need the rest.

[arsenm]

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

544–547

I'm not sure I exactly mean MCP. I'm sort of thinking the allocator would have turned the 16-bit subreg into the same 32-bit reg

[arsenm]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

151–152

These are still here, although contrary to your last comment that we don't need to make these legal yet? (which I interpreted as using as a legal type reg class)

[rampitec]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

151–152

Yes, this is still here just for discussion purposes. I think I will drop this part and revert tests as well, but add a mir test for copyPhysReg() instead.

[rampitec]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

151–152

CopyPhysReg is extracted into D79119.

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

SITargetLowering::SITargetLowering(const TargetMachine &TM,
 
   addRegisterClass(MVT::v8i32, &AMDGPU::SGPR_256RegClass);
   addRegisterClass(MVT::v8f32, &AMDGPU::VReg_256RegClass);
 
   addRegisterClass(MVT::v16i32, &AMDGPU::SGPR_512RegClass);
   addRegisterClass(MVT::v16f32, &AMDGPU::VReg_512RegClass);
 
   if (Subtarget->has16BitInsts()) {
-    addRegisterClass(MVT::i16, &AMDGPU::SReg_32RegClass);
-    addRegisterClass(MVT::f16, &AMDGPU::SReg_32RegClass);
+    addRegisterClass(MVT::i16, &AMDGPU::SGPR_LO16RegClass);
+    addRegisterClass(MVT::f16, &AMDGPU::SGPR_LO16RegClass);
 
     // Unless there are also VOP3P operations, not operations are really legal.
     addRegisterClass(MVT::v2i16, &AMDGPU::SReg_32RegClass);
     addRegisterClass(MVT::v2f16, &AMDGPU::SReg_32RegClass);
     addRegisterClass(MVT::v4i16, &AMDGPU::SReg_64RegClass);
     addRegisterClass(MVT::v4f16, &AMDGPU::SReg_64RegClass);
   }
 

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

 
 // Must be at least 4 to be able to branch over minimum unconditional branch
 // code. This is only for making it possible to write reasonably small tests for
 // long branches.
 static cl::opt<unsigned>
 BranchOffsetBits("amdgpu-s-branch-bits", cl::ReallyHidden, cl::init(16),
                  cl::desc("Restrict range of branch instructions (DEBUG)"));
 
+static cl::opt<bool> Fix16BitCopies(
+  "amdgpu-fix-16-bit-physreg-copies",
+  cl::desc("Fix copies between 32 and 16 bit registers by extending to 32 bit"),
+  cl::init(true),
+  cl::ReallyHidden);
+
 SIInstrInfo::SIInstrInfo(const GCNSubtarget &ST)
   : AMDGPUGenInstrInfo(AMDGPU::ADJCALLSTACKUP, AMDGPU::ADJCALLSTACKDOWN),
     RI(ST), ST(ST) {
   SchedModel.init(&ST);
 }
 
 //===----------------------------------------------------------------------===//
 // TargetInstrInfo callbacks

 }
 
 void SIInstrInfo::copyPhysReg(MachineBasicBlock &MBB,
                               MachineBasicBlock::iterator MI,
                               const DebugLoc &DL, MCRegister DestReg,
                               MCRegister SrcReg, bool KillSrc) const {
   const TargetRegisterClass *RC = RI.getPhysRegClass(DestReg);
 
+  // FIXME: This is hack to resolve copies between 16 bit and 32 bit
+  // registers until all patterns are fixed.
+  if (Fix16BitCopies &&
+      ((RI.getRegSizeInBits(*RC) == 16) ^
+       (RI.getRegSizeInBits(*RI.getPhysRegClass(SrcReg)) == 16))) {
+    MCRegister &RegToFix = (RI.getRegSizeInBits(*RC) == 16) ? DestReg : SrcReg;
+    RegToFix = RI.get32BitRegister(RegToFix);
+
+    if (DestReg == SrcReg) {
+      // Insert empty bundle since ExpandPostRA expects an instruction here.
+      BuildMI(MBB, MI, DL, get(AMDGPU::BUNDLE));
+      return;
+    }
+
+    RC = RI.getPhysRegClass(DestReg);
+  }
+
   if (RC == &AMDGPU::VGPR_32RegClass) {
     assert(AMDGPU::VGPR_32RegClass.contains(SrcReg) ||
            AMDGPU::SReg_32RegClass.contains(SrcReg) ||
            AMDGPU::AGPR_32RegClass.contains(SrcReg));
     unsigned Opc = AMDGPU::AGPR_32RegClass.contains(SrcReg) ?
                      AMDGPU::V_ACCVGPR_READ_B32 : AMDGPU::V_MOV_B32_e32;
     BuildMI(MBB, MI, DL, get(Opc), DestReg)
       .addReg(SrcReg, getKillRegState(KillSrc));
bool SIInstrInfo::FoldImmediate(MachineInstr &UseMI, MachineInstr &DefMI,
 
   const MachineOperand *ImmOp = getNamedOperand(DefMI, AMDGPU::OpName::src0);
   assert(ImmOp);
   // FIXME: We could handle FrameIndex values here.
   if (!ImmOp->isImm())
     return false;
 
   unsigned Opc = UseMI.getOpcode();
-  if (Opc == AMDGPU::COPY) {
+  // FIXME: it is still possible to fold immediate even with a subreg.
+  if (Opc == AMDGPU::COPY &&
+      (!UseMI.getOperand(1).getSubReg() &&
+       RI.getRegSizeInBits(
+         *RI.getRegClassForReg(*MRI, UseMI.getOperand(0).getReg())) == 32)) {
     bool isVGPRCopy = RI.isVGPR(*MRI, UseMI.getOperand(0).getReg());
     unsigned NewOpc = isVGPRCopy ? AMDGPU::V_MOV_B32_e32 : AMDGPU::S_MOV_B32;
     if (RI.isAGPR(*MRI, UseMI.getOperand(0).getReg())) {
       if (!isInlineConstant(*ImmOp, AMDGPU::OPERAND_REG_INLINE_AC_INT32))
         return false;
       NewOpc = AMDGPU::V_ACCVGPR_WRITE_B32;
     }
     UseMI.setDesc(get(NewOpc));

llvm/test/CodeGen/AMDGPU/bitreverse.ll

 ; SI-NEXT:    v_or_b32_e32 v0, v2, v0
 ; SI-NEXT:    v_or_b32_e32 v1, v3, v1
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; FLAT-LABEL: s_brev_i64:
 ; FLAT:       ; %bb.0:
 ; FLAT-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x2c
-; FLAT-NEXT:    v_mov_b32_e32 v0, 0x10203
-; FLAT-NEXT:    s_mov_b32 s4, 0xf0f0f0f
+; FLAT-NEXT:    s_mov_b32 s4, 0x10203
+; FLAT-NEXT:    s_mov_b32 s5, 0xf0f0f0f
 ; FLAT-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; FLAT-NEXT:    s_waitcnt lgkmcnt(0)
-; FLAT-NEXT:    v_perm_b32 v2, 0, s2, v0
-; FLAT-NEXT:    v_perm_b32 v4, 0, s3, v0
+; FLAT-NEXT:    v_mov_b32_e32 v0, s2
+; FLAT-NEXT:    v_mov_b32_e32 v1, s3
+; FLAT-NEXT:    v_perm_b32 v2, 0, v0, s4
+; FLAT-NEXT:    v_perm_b32 v4, 0, v1, s4
 ; FLAT-NEXT:    s_mov_b32 s2, 0xf0f0f0f0
-; FLAT-NEXT:    v_and_b32_e32 v1, s4, v2
-; FLAT-NEXT:    v_and_b32_e32 v0, s4, v4
+; FLAT-NEXT:    v_and_b32_e32 v1, s5, v2
+; FLAT-NEXT:    v_and_b32_e32 v0, s5, v4
 ; FLAT-NEXT:    v_and_b32_e32 v3, s2, v2
 ; FLAT-NEXT:    v_and_b32_e32 v2, s2, v4
 ; FLAT-NEXT:    v_lshlrev_b64 v[0:1], 4, v[0:1]
 ; FLAT-NEXT:    v_lshrrev_b64 v[2:3], 4, v[2:3]
 ; FLAT-NEXT:    s_mov_b32 s2, 0x33333333
 ; FLAT-NEXT:    v_or_b32_e32 v2, v2, v0
 ; FLAT-NEXT:    v_or_b32_e32 v3, v3, v1
 ; FLAT-NEXT:    v_and_b32_e32 v1, s2, v3

 ; SI-NEXT:    v_or_b32_e32 v1, v7, v5
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; FLAT-LABEL: s_brev_v2i64:
 ; FLAT:       ; %bb.0:
 ; FLAT-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
 ; FLAT-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x34
-; FLAT-NEXT:    v_mov_b32_e32 v4, 0x10203
-; FLAT-NEXT:    s_mov_b32 s8, 0xf0f0f0f
-; FLAT-NEXT:    s_mov_b32 s9, 0xcccccccc
-; FLAT-NEXT:    s_mov_b32 s10, 0x55555555
+; FLAT-NEXT:    s_mov_b32 s8, 0x10203
+; FLAT-NEXT:    s_mov_b32 s9, 0x33333333
+; FLAT-NEXT:    s_mov_b32 s10, 0xcccccccc
+; FLAT-NEXT:    s_mov_b32 s11, 0x55555555
 ; FLAT-NEXT:    s_waitcnt lgkmcnt(0)
-; FLAT-NEXT:    v_perm_b32 v3, 0, s2, v4
-; FLAT-NEXT:    v_perm_b32 v2, 0, s3, v4
-; FLAT-NEXT:    s_mov_b32 s2, 0xf0f0f0f0
-; FLAT-NEXT:    v_and_b32_e32 v0, s8, v2
-; FLAT-NEXT:    v_and_b32_e32 v1, s8, v3
-; FLAT-NEXT:    v_and_b32_e32 v2, s2, v2
-; FLAT-NEXT:    v_and_b32_e32 v3, s2, v3
+; FLAT-NEXT:    v_mov_b32_e32 v0, s2
+; FLAT-NEXT:    v_mov_b32_e32 v1, s3
+; FLAT-NEXT:    v_perm_b32 v3, 0, v0, s8
+; FLAT-NEXT:    v_perm_b32 v2, 0, v1, s8
+; FLAT-NEXT:    s_mov_b32 s2, 0xf0f0f0f
+; FLAT-NEXT:    s_mov_b32 s3, 0xf0f0f0f0
+; FLAT-NEXT:    v_and_b32_e32 v0, s2, v2
+; FLAT-NEXT:    v_and_b32_e32 v1, s2, v3
+; FLAT-NEXT:    v_and_b32_e32 v2, s3, v2
+; FLAT-NEXT:    v_and_b32_e32 v3, s3, v3
+; FLAT-NEXT:    v_mov_b32_e32 v4, s0
 ; FLAT-NEXT:    v_lshlrev_b64 v[0:1], 4, v[0:1]
 ; FLAT-NEXT:    v_lshrrev_b64 v[2:3], 4, v[2:3]
-; FLAT-NEXT:    v_perm_b32 v7, 0, s0, v4
-; FLAT-NEXT:    v_perm_b32 v6, 0, s1, v4
+; FLAT-NEXT:    v_perm_b32 v7, 0, v4, s8
+; FLAT-NEXT:    v_mov_b32_e32 v4, s1
 ; FLAT-NEXT:    v_or_b32_e32 v2, v2, v0
-; FLAT-NEXT:    s_mov_b32 s3, 0x33333333
 ; FLAT-NEXT:    v_or_b32_e32 v3, v3, v1
-; FLAT-NEXT:    v_and_b32_e32 v0, s3, v2
-; FLAT-NEXT:    v_and_b32_e32 v1, s3, v3
-; FLAT-NEXT:    v_and_b32_e32 v4, s8, v6
-; FLAT-NEXT:    v_and_b32_e32 v5, s8, v7
-; FLAT-NEXT:    v_and_b32_e32 v2, s9, v2
-; FLAT-NEXT:    v_and_b32_e32 v3, s9, v3
-; FLAT-NEXT:    v_and_b32_e32 v6, s2, v6
-; FLAT-NEXT:    v_and_b32_e32 v7, s2, v7
+; FLAT-NEXT:    v_perm_b32 v6, 0, v4, s8
+; FLAT-NEXT:    v_and_b32_e32 v0, s9, v2
+; FLAT-NEXT:    v_and_b32_e32 v1, s9, v3
+; FLAT-NEXT:    v_and_b32_e32 v4, s2, v6
+; FLAT-NEXT:    v_and_b32_e32 v5, s2, v7
+; FLAT-NEXT:    v_and_b32_e32 v2, s10, v2
+; FLAT-NEXT:    v_and_b32_e32 v3, s10, v3
+; FLAT-NEXT:    v_and_b32_e32 v6, s3, v6
+; FLAT-NEXT:    v_and_b32_e32 v7, s3, v7
 ; FLAT-NEXT:    v_lshlrev_b64 v[0:1], 2, v[0:1]
 ; FLAT-NEXT:    v_lshrrev_b64 v[2:3], 2, v[2:3]
 ; FLAT-NEXT:    v_lshlrev_b64 v[4:5], 4, v[4:5]
 ; FLAT-NEXT:    v_lshrrev_b64 v[6:7], 4, v[6:7]
 ; FLAT-NEXT:    v_or_b32_e32 v2, v2, v0
 ; FLAT-NEXT:    v_or_b32_e32 v3, v3, v1
 ; FLAT-NEXT:    v_or_b32_e32 v6, v6, v4
 ; FLAT-NEXT:    v_or_b32_e32 v7, v7, v5
-; FLAT-NEXT:    s_mov_b32 s11, 0xaaaaaaaa
-; FLAT-NEXT:    v_and_b32_e32 v0, s10, v2
-; FLAT-NEXT:    v_and_b32_e32 v1, s10, v3
-; FLAT-NEXT:    v_and_b32_e32 v4, s3, v6
-; FLAT-NEXT:    v_and_b32_e32 v5, s3, v7
-; FLAT-NEXT:    v_and_b32_e32 v2, s11, v2
-; FLAT-NEXT:    v_and_b32_e32 v3, s11, v3
-; FLAT-NEXT:    v_and_b32_e32 v6, s9, v6
-; FLAT-NEXT:    v_and_b32_e32 v7, s9, v7
+; FLAT-NEXT:    s_mov_b32 s12, 0xaaaaaaaa
+; FLAT-NEXT:    v_and_b32_e32 v0, s11, v2
+; FLAT-NEXT:    v_and_b32_e32 v1, s11, v3
+; FLAT-NEXT:    v_and_b32_e32 v4, s9, v6
+; FLAT-NEXT:    v_and_b32_e32 v5, s9, v7
+; FLAT-NEXT:    v_and_b32_e32 v2, s12, v2
+; FLAT-NEXT:    v_and_b32_e32 v3, s12, v3
+; FLAT-NEXT:    v_and_b32_e32 v6, s10, v6
+; FLAT-NEXT:    v_and_b32_e32 v7, s10, v7
 ; FLAT-NEXT:    v_lshlrev_b64 v[0:1], 1, v[0:1]
 ; FLAT-NEXT:    v_lshrrev_b64 v[2:3], 1, v[2:3]
 ; FLAT-NEXT:    v_lshlrev_b64 v[4:5], 2, v[4:5]
 ; FLAT-NEXT:    v_lshrrev_b64 v[6:7], 2, v[6:7]
 ; FLAT-NEXT:    v_or_b32_e32 v2, v2, v0
 ; FLAT-NEXT:    v_or_b32_e32 v0, v6, v4
 ; FLAT-NEXT:    v_or_b32_e32 v7, v7, v5
-; FLAT-NEXT:    v_and_b32_e32 v5, s10, v7
-; FLAT-NEXT:    v_and_b32_e32 v4, s10, v0
-; FLAT-NEXT:    v_and_b32_e32 v6, s11, v0
-; FLAT-NEXT:    v_and_b32_e32 v7, s11, v7
+; FLAT-NEXT:    v_and_b32_e32 v5, s11, v7
+; FLAT-NEXT:    v_and_b32_e32 v4, s11, v0
+; FLAT-NEXT:    v_and_b32_e32 v6, s12, v0
+; FLAT-NEXT:    v_and_b32_e32 v7, s12, v7
 ; FLAT-NEXT:    v_lshlrev_b64 v[4:5], 1, v[4:5]
 ; FLAT-NEXT:    v_lshrrev_b64 v[6:7], 1, v[6:7]
 ; FLAT-NEXT:    v_or_b32_e32 v3, v3, v1
 ; FLAT-NEXT:    s_mov_b32 s7, 0xf000
 ; FLAT-NEXT:    s_mov_b32 s6, -1
 ; FLAT-NEXT:    v_or_b32_e32 v0, v6, v4
 ; FLAT-NEXT:    v_or_b32_e32 v1, v7, v5
 ; FLAT-NEXT:    buffer_store_dwordx4 v[0:3], off, s[4:7], 0

llvm/test/CodeGen/AMDGPU/bswap.ll

 ; SI-NEXT:    v_bfi_b32 v1, s6, v1, v0
 ; SI-NEXT:    v_bfi_b32 v0, s6, v3, v2
 ; SI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: test_bswap_i64:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x10203
+; VI-NEXT:    s_mov_b32 s8, 0x10203
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_load_dwordx2 s[4:5], s[6:7], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_perm_b32 v1, 0, s4, v0
-; VI-NEXT:    v_perm_b32 v0, 0, s5, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s4
+; VI-NEXT:    v_mov_b32_e32 v2, s5
+; VI-NEXT:    v_perm_b32 v1, 0, v0, s8
+; VI-NEXT:    v_perm_b32 v0, 0, v2, s8
 ; VI-NEXT:    buffer_store_dwordx2 v[0:1], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %val = load i64, i64 addrspace(1)* %in, align 8
   %bswap = call i64 @llvm.bswap.i64(i64 %val) nounwind readnone
   store i64 %bswap, i64 addrspace(1)* %out, align 8
   ret void
 }
 

 ; SI-NEXT:    v_bfi_b32 v1, s8, v6, v5
 ; SI-NEXT:    v_bfi_b32 v0, s8, v8, v7
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: test_bswap_v2i64:
 ; VI:       ; %bb.0:
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v0, 0x10203
+; VI-NEXT:    s_mov_b32 s8, 0x10203
 ; VI-NEXT:    s_mov_b32 s3, 0xf000
 ; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_mov_b32 s0, s4
 ; VI-NEXT:    s_mov_b32 s1, s5
 ; VI-NEXT:    s_load_dwordx4 s[4:7], s[6:7], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_perm_b32 v3, 0, s6, v0
-; VI-NEXT:    v_perm_b32 v2, 0, s7, v0
-; VI-NEXT:    v_perm_b32 v1, 0, s4, v0
-; VI-NEXT:    v_perm_b32 v0, 0, s5, v0
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    v_mov_b32_e32 v1, s7
+; VI-NEXT:    v_perm_b32 v3, 0, v0, s8
+; VI-NEXT:    v_mov_b32_e32 v0, s5
+; VI-NEXT:    v_mov_b32_e32 v4, s4
+; VI-NEXT:    v_perm_b32 v2, 0, v1, s8
+; VI-NEXT:    v_perm_b32 v1, 0, v4, s8
+; VI-NEXT:    v_perm_b32 v0, 0, v0, s8
 ; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %val = load <2 x i64>, <2 x i64> addrspace(1)* %in, align 16
   %bswap = call <2 x i64> @llvm.bswap.v2i64(<2 x i64> %val) nounwind readnone
   store <2 x i64> %bswap, <2 x i64> addrspace(1)* %out, align 16
   ret void
 }
 

 ; SI-NEXT:    v_bfi_b32 v5, s12, v14, v13
 ; SI-NEXT:    v_bfi_b32 v4, s12, v16, v15
 ; SI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[8:11], 0 offset:16
 ; SI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
 ; SI-NEXT:    s_endpgm
 ;
 ; VI-LABEL: test_bswap_v4i64:
 ; VI:       ; %bb.0:
-; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; VI-NEXT:    v_mov_b32_e32 v4, 0x10203
-; VI-NEXT:    s_mov_b32 s11, 0xf000
-; VI-NEXT:    s_mov_b32 s10, -1
+; VI-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; VI-NEXT:    s_mov_b32 s12, 0x10203
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    s_mov_b32 s8, s0
-; VI-NEXT:    s_mov_b32 s9, s1
-; VI-NEXT:    s_load_dwordx8 s[0:7], s[2:3], 0x0
+; VI-NEXT:    s_mov_b32 s0, s4
+; VI-NEXT:    s_mov_b32 s1, s5
+; VI-NEXT:    s_load_dwordx8 s[4:11], s[6:7], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
-; VI-NEXT:    v_perm_b32 v3, 0, s2, v4
-; VI-NEXT:    v_perm_b32 v2, 0, s3, v4
-; VI-NEXT:    v_perm_b32 v1, 0, s0, v4
-; VI-NEXT:    v_perm_b32 v0, 0, s1, v4
-; VI-NEXT:    v_perm_b32 v7, 0, s6, v4
-; VI-NEXT:    v_perm_b32 v6, 0, s7, v4
-; VI-NEXT:    v_perm_b32 v5, 0, s4, v4
-; VI-NEXT:    v_perm_b32 v4, 0, s5, v4
-; VI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[8:11], 0 offset:16
-; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[8:11], 0
+; VI-NEXT:    v_mov_b32_e32 v1, s7
+; VI-NEXT:    v_mov_b32_e32 v4, s4
+; VI-NEXT:    v_perm_b32 v2, 0, v1, s12
+; VI-NEXT:    v_perm_b32 v1, 0, v4, s12
+; VI-NEXT:    v_mov_b32_e32 v4, s10
+; VI-NEXT:    v_perm_b32 v7, 0, v4, s12
+; VI-NEXT:    v_mov_b32_e32 v4, s11
+; VI-NEXT:    v_perm_b32 v6, 0, v4, s12
+; VI-NEXT:    v_mov_b32_e32 v4, s8
+; VI-NEXT:    v_mov_b32_e32 v0, s6
+; VI-NEXT:    v_perm_b32 v5, 0, v4, s12
+; VI-NEXT:    v_mov_b32_e32 v4, s9
+; VI-NEXT:    v_perm_b32 v3, 0, v0, s12
+; VI-NEXT:    v_mov_b32_e32 v0, s5
+; VI-NEXT:    v_perm_b32 v4, 0, v4, s12
+; VI-NEXT:    v_perm_b32 v0, 0, v0, s12
+; VI-NEXT:    buffer_store_dwordx4 v[4:7], off, s[0:3], 0 offset:16
+; VI-NEXT:    buffer_store_dwordx4 v[0:3], off, s[0:3], 0
 ; VI-NEXT:    s_endpgm
   %val = load <4 x i64>, <4 x i64> addrspace(1)* %in, align 32
   %bswap = call <4 x i64> @llvm.bswap.v4i64(<4 x i64> %val) nounwind readnone
   store <4 x i64> %bswap, <4 x i64> addrspace(1)* %out, align 32
   ret void
 }
 
 define float @missing_truncate_promote_bswap(i32 %arg) {

llvm/test/CodeGen/AMDGPU/cgp-bitfield-extract.ll

 
 ; GCN-LABEL: {{^}}sink_ubfe_i16:
 ; GCN-NOT: lshr
 ; VI: s_load_dword [[ARG:s[0-9]+]], s[0:1], 0x2c
 ; VI: s_bfe_u32 [[BFE:s[0-9]+]], [[ARG]], 0xc0004
 ; GCN: s_cbranch_scc{{[0-1]}}
 
 ; SI: s_bfe_u32 s{{[0-9]+}}, s{{[0-9]+}}, 0x70004
-; VI: v_mov_b32_e32 v{{[0-9]+}}, 0x7f
+; VI: v_and_b32_e32 v{{[0-9]+}}, 0x7f
 
 ; GCN: BB2_3:
 ; SI: s_bfe_u32 s{{[0-9]+}}, s{{[0-9]+}}, 0x80004
-; VI: v_mov_b32_e32 v{{[0-9]+}}, 0xff
+; VI: v_and_b32_e32 v{{[0-9]+}}, 0xff
 
 ; GCN: buffer_store_short
 ; GCN: s_endpgm
 define amdgpu_kernel void @sink_ubfe_i16(i16 addrspace(1)* %out, i16 %arg1) #0 {
 entry:
   %shr = lshr i16 %arg1, 4
   br i1 undef, label %bb0, label %bb1
 

llvm/test/CodeGen/AMDGPU/fmul-2-combine-multi-use.ll

define amdgpu_kernel void @multiple_fadd_use_test_f16(half addrspace(1)* %out, i16 zeroext %x.arg, i16 zeroext %y.arg, i16 zeroext %z.arg) #0 {
   %a18 = fmul half %a17, %a17
   %a19 = fmul half %a18, %a17
   %a20 = fsub half 1.0, %a19
   store half %a20, half addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}multiple_use_fadd_fmac_f16:
-; GCN-DAG: v_add_f16_e64 [[MUL2:v[0-9]+]], [[X:s[0-9]+]], s{{[0-9]+}}
+; GCN: s_load_dword [[X:s[0-9]+]],
+; GCN-DAG: v_add_f16_e64 [[MUL2:v[0-9]+]], [[X]], s{{[0-9]+}}
 
-; VI-FLUSH-DAG:     v_mac_f16_e64 [[MAD:v[0-9]+]], [[X]], 2.0
-; VI-DENORM-DAG:    v_fma_f16 [[MAD:v[0-9]+]], [[X]], 2.0, v{{[0-9]+}}
+; VI-DAG: v_mov_b32_e32 [[VX:v[0-9]+]], [[X]]
+
+; VI-FLUSH-DAG:     v_mad_f16 [[MAD:v[0-9]+]], [[VX]], 2.0, {{[vs][0-9]+}}
+; VI-DENORM-DAG:    v_fma_f16 [[MAD:v[0-9]+]], [[VX]], 2.0, {{[vs][0-9]+}}
 ; GFX10-FLUSH-DAG:  v_add_f16_e32 [[MAD:v[0-9]+]], s{{[0-9]+}}, [[MUL2]]
 ; GFX10-DENORM-DAG: v_fma_f16 [[MAD:v[0-9]+]], [[X]], 2.0, s{{[0-9]+}}
 
 ; GCN-DAG: buffer_store_short [[MUL2]]
 ; GCN-DAG: buffer_store_short [[MAD]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @multiple_use_fadd_fmac_f16(half addrspace(1)* %out, i16 zeroext %x.arg, i16 zeroext %y.arg) #0 {
   %x = bitcast i16 %x.arg to half
   %y = bitcast i16 %y.arg to half
   %out.gep.1 = getelementptr half, half addrspace(1)* %out, i32 1
   %mul2 = fmul fast half %x, 2.0
   %mad = fadd fast half %mul2, %y
   store volatile half %mul2, half addrspace(1)* %out
   store volatile half %mad, half addrspace(1)* %out.gep.1
   ret void
 }
 
 ; GCN-LABEL: {{^}}multiple_use_fadd_fmad_f16:
-; GCN-DAG: v_add_f16_e64 [[MUL2:v[0-9]+]], |[[X:s[0-9]+]]|, |s{{[0-9]+}}|
+; GCN: s_load_dword [[X:s[0-9]+]],
+; GCN-DAG: v_add_f16_e64 [[MUL2:v[0-9]+]], |[[X]]|, |s{{[0-9]+}}|
+
+; VI-DAG: v_mov_b32_e32 [[VX:v[0-9]+]], [[X]]
 
-; VI-FLUSH-DAG:     v_mad_f16 [[MAD:v[0-9]+]], |[[X]]|, 2.0, v{{[0-9]+}}
-; VI-DENORM-DAG:    v_fma_f16 [[MAD:v[0-9]+]], |[[X]]|, 2.0, v{{[0-9]+}}
+; VI-FLUSH-DAG:     v_mad_f16 [[MAD:v[0-9]+]], |[[VX]]|, 2.0, {{[vs][0-9]+}}
+; VI-DENORM-DAG:    v_fma_f16 [[MAD:v[0-9]+]], |[[VX]]|, 2.0, {{[vs][0-9]+}}
 ; GFX10-FLUSH-DAG:  v_add_f16_e32 [[MAD:v[0-9]+]], s{{[0-9]+}}, [[MUL2]]
 ; GFX10-DENORM-DAG: v_fma_f16 [[MAD:v[0-9]+]], |[[X]]|, 2.0, s{{[0-9]+}}
 
 ; GCN-DAG: buffer_store_short [[MUL2]]
 ; GCN-DAG: buffer_store_short [[MAD]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @multiple_use_fadd_fmad_f16(half addrspace(1)* %out, i16 zeroext %x.arg, i16 zeroext %y.arg) #0 {
   %x = bitcast i16 %x.arg to half
define amdgpu_kernel void @fmul_x2_xn2_f16(half addrspace(1)* %out, i16 zeroext %x.arg, i16 zeroext %y.arg) #0 {
   %mul2 = fmul fast half %x, 2.0
   %muln2 = fmul fast half %x, -2.0
   %mul = fmul fast half %mul2, %muln2
   store volatile half %mul, half addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}fmul_x2_xn3_f16:
-; SIVI:  v_mov_b32_e32 [[K:v[0-9]+]], 0xc600
-; SIVI:  v_mul_f16_e32 [[TMP0:v[0-9]+]], [[X:s[0-9]+]], [[K]]
+; GCN:   s_load_dword [[X:s[0-9]+]],
+; SI:    v_mov_b32_e32 [[K:v[0-9]+]], 0xc600
+; SI:    v_mul_f16_e32 [[TMP0:v[0-9]+]], [[X]], [[K]]
+; VI:    v_mov_b32_e32 [[VX:v[0-9]+]], [[X]]
+; VI:    v_mul_f16_e32 [[TMP0:v[0-9]+]], 0xc600, [[VX]]
 ; GFX10: v_mul_f16_e64 [[TMP0:v[0-9]+]], 0xc600, [[X:s[0-9]+]]
 ; GCN:   v_mul_f16_e32 [[RESULT:v[0-9]+]], [[X]], [[TMP0]]
 ; GCN:   buffer_store_short [[RESULT]]
 define amdgpu_kernel void @fmul_x2_xn3_f16(half addrspace(1)* %out, i16 zeroext %x.arg, i16 zeroext %y.arg) #0 {
   %x = bitcast i16 %x.arg to half
   %y = bitcast i16 %y.arg to half
   %out.gep.1 = getelementptr half, half addrspace(1)* %out, i32 1
   %mul2 = fmul fast half %x, 2.0
   %muln2 = fmul fast half %x, -3.0
   %mul = fmul fast half %mul2, %muln2
   store volatile half %mul, half addrspace(1)* %out
   ret void
 }
 
 attributes #0 = { nounwind "unsafe-fp-math"="true" }
 attributes #1 = { nounwind readnone }

llvm/test/CodeGen/AMDGPU/fshr.ll

 ; GFX9-NEXT:    v_mov_b32_e32 v7, 15
 ; GFX9-NEXT:    v_and_b32_e32 v6, 15, v4
 ; GFX9-NEXT:    v_mov_b32_e32 v8, 0xffff
 ; GFX9-NEXT:    v_and_b32_sdwa v7, v4, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_and_b32_e32 v6, v8, v6
 ; GFX9-NEXT:    v_lshl_or_b32 v6, v7, 16, v6
 ; GFX9-NEXT:    v_pk_lshrrev_b16 v7, v6, v2
 ; GFX9-NEXT:    v_pk_sub_i16 v6, 16, v6 op_sel_hi:[0,1]
-; GFX9-NEXT:    s_mov_b32 s6, 0xf000f
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v0, v6, v0
-; GFX9-NEXT:    v_and_b32_e32 v4, s6, v4
+; GFX9-NEXT:    v_and_b32_e32 v4, 0xf000f, v4
 ; GFX9-NEXT:    v_or_b32_e32 v0, v0, v7
 ; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, 0, v4
 ; GFX9-NEXT:    v_mov_b32_e32 v7, 0
 ; GFX9-NEXT:    v_cndmask_b32_e32 v6, v0, v2, vcc
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
 ; GFX9-NEXT:    v_cmp_eq_u16_sdwa s[4:5], v4, v7 src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_cndmask_b32_e64 v0, v0, v2, s[4:5]
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v5
 ; GFX9-NEXT:    v_and_b32_e32 v2, v8, v2
 ; GFX9-NEXT:    v_pk_lshrrev_b16 v4, v2, v3
 ; GFX9-NEXT:    v_pk_sub_i16 v2, 16, v2
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v1, v2, v1
-; GFX9-NEXT:    v_and_b32_e32 v2, s6, v5
+; GFX9-NEXT:    v_and_b32_e32 v2, 0xf000f, v5
 ; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, 0, v2
 ; GFX9-NEXT:    v_or_b32_e32 v1, v1, v4
 ; GFX9-NEXT:    v_and_b32_e32 v2, v8, v6
 ; GFX9-NEXT:    v_cndmask_b32_e32 v1, v1, v3, vcc
 ; GFX9-NEXT:    v_lshl_or_b32 v0, v0, 16, v2
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; R600-LABEL: v_fshr_v3i16:

 ; GFX9-NEXT:    v_mov_b32_e32 v7, 15
 ; GFX9-NEXT:    v_and_b32_e32 v6, 15, v5
 ; GFX9-NEXT:    v_mov_b32_e32 v9, 0xffff
 ; GFX9-NEXT:    v_and_b32_sdwa v8, v5, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_and_b32_e32 v6, v9, v6
 ; GFX9-NEXT:    v_lshl_or_b32 v6, v8, 16, v6
 ; GFX9-NEXT:    v_pk_lshrrev_b16 v8, v6, v3
 ; GFX9-NEXT:    v_pk_sub_i16 v6, 16, v6 op_sel_hi:[0,1]
-; GFX9-NEXT:    s_mov_b32 s6, 0xf000f
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v1, v6, v1
-; GFX9-NEXT:    v_and_b32_e32 v5, s6, v5
+; GFX9-NEXT:    v_and_b32_e32 v5, 0xf000f, v5
 ; GFX9-NEXT:    v_or_b32_e32 v1, v1, v8
 ; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, 0, v5
 ; GFX9-NEXT:    v_mov_b32_e32 v8, 0
 ; GFX9-NEXT:    v_cndmask_b32_e32 v6, v1, v3, vcc
 ; GFX9-NEXT:    v_cmp_eq_u16_sdwa s[4:5], v5, v8 src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v3, 16, v3
 ; GFX9-NEXT:    v_cndmask_b32_e64 v1, v1, v3, s[4:5]
 ; GFX9-NEXT:    v_and_b32_e32 v3, 15, v4
 ; GFX9-NEXT:    v_and_b32_sdwa v5, v4, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_and_b32_e32 v3, v9, v3
 ; GFX9-NEXT:    v_lshl_or_b32 v3, v5, 16, v3
 ; GFX9-NEXT:    v_pk_lshrrev_b16 v5, v3, v2
 ; GFX9-NEXT:    v_pk_sub_i16 v3, 16, v3 op_sel_hi:[0,1]
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v0, v3, v0
-; GFX9-NEXT:    v_and_b32_e32 v3, s6, v4
+; GFX9-NEXT:    v_and_b32_e32 v3, 0xf000f, v4
 ; GFX9-NEXT:    v_or_b32_e32 v0, v0, v5
 ; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, 0, v3
 ; GFX9-NEXT:    v_cndmask_b32_e32 v4, v0, v2, vcc
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
 ; GFX9-NEXT:    v_cmp_eq_u16_sdwa s[4:5], v3, v8 src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_cndmask_b32_e64 v0, v0, v2, s[4:5]
 ; GFX9-NEXT:    v_and_b32_e32 v2, v9, v4

llvm/test/CodeGen/AMDGPU/function-returns.ll

define <16 x i8> @v16i8_func_void() #0 {
   %ptr = load volatile <16 x i8> addrspace(1)*, <16 x i8> addrspace(1)* addrspace(4)* undef
   %val = load <16 x i8>, <16 x i8> addrspace(1)* %ptr
   ret <16 x i8> %val
 }
 
 ; FIXME: Should pack
 ; GCN-LABEL: {{^}}v4i8_func_void:
 ; GCN: buffer_load_dword v0
-; GCN-DAG: v_lshrrev_b32_e32 v1, 8, v0
-; GCN-DAG: v_lshrrev_b32_e32 v2, 16, v0
-; GCN-DAG: v_lshrrev_b32_e32 v3, 24, v0
+; CI-DAG: v_lshrrev_b32_e32 v1, 8, v0
+; CI-DAG: v_lshrrev_b32_e32 v2, 16, v0
+; CI-DAG: v_lshrrev_b32_e32 v3, 24, v0
+
+; FIXME: should shift right into resulting registers
+
+; GFX89-DAG: v_lshrrev_b32_e32 v2, 16, v0
+; GFX89-DAG: v_lshrrev_b32_e32 v3, 8, v0
+; GFX89-DAG: v_lshrrev_b32_e32 v0, 24, v0
 ; GCN: s_setpc_b64
 define <4  x i8> @v4i8_func_void() #0 {
   %ptr = load volatile <4  x i8> addrspace(1)*, <4  x i8> addrspace(1)* addrspace(4)* undef
   %val = load <4  x i8>, <4  x i8> addrspace(1)* %ptr
   ret <4  x i8> %val
 }
 
 ; GCN-LABEL: {{^}}struct_i8_i32_func_void:

llvm/test/CodeGen/AMDGPU/idiv-licm.ll

 }
 
 define amdgpu_kernel void @udiv16_invariant_denom(i16 addrspace(1)* nocapture %arg, i16 %arg1) {
 ; GFX9-LABEL: udiv16_invariant_denom:
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_load_dword s3, s[0:1], 0x2c
 ; GFX9-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    s_mov_b32 s0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s3, s2, s3
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s3
+; GFX9-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-NEXT:    s_movk_i32 s3, 0x400
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v0
 ; GFX9-NEXT:  BB4_1: ; %bb3
 ; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    v_mov_b32_e32 v4, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, s2, v4
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v8, v2
-; GFX9-NEXT:    v_lshlrev_b64 v[5:6], 1, v[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v7, s5
-; GFX9-NEXT:    v_add_co_u32_e64 v5, s[0:1], s4, v5
-; GFX9-NEXT:    v_mul_f32_e32 v2, v8, v1
-; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
-; GFX9-NEXT:    v_addc_co_u32_e64 v6, s[0:1], v7, v6, s[0:1]
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v7, v2
-; GFX9-NEXT:    v_add_u16_e32 v4, 1, v4
-; GFX9-NEXT:    v_mad_f32 v2, -v2, v0, v8
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, v0
-; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, s3, v4
-; GFX9-NEXT:    v_addc_co_u32_e64 v2, s[0:1], 0, v7, s[0:1]
+; GFX9-NEXT:    v_cvt_f32_u32_sdwa v7, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0
+; GFX9-NEXT:    v_add_u16_e32 v8, 1, v4
+; GFX9-NEXT:    v_lshlrev_b64 v[4:5], 1, v[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v6, s5
+; GFX9-NEXT:    v_add_co_u32_e64 v4, s[0:1], s4, v4
+; GFX9-NEXT:    v_addc_co_u32_e64 v5, s[0:1], v6, v5, s[0:1]
+; GFX9-NEXT:    v_mul_f32_e32 v6, v7, v1
+; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
+; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, s3, v8
+; GFX9-NEXT:    v_mov_b32_e32 v2, v8
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v8, v6
+; GFX9-NEXT:    v_mad_f32 v6, -v6, v0, v7
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v6|, v0
 ; GFX9-NEXT:    s_and_b64 vcc, exec, vcc
-; GFX9-NEXT:    global_store_short v[5:6], v2, off
+; GFX9-NEXT:    v_addc_co_u32_e64 v6, s[0:1], 0, v8, s[0:1]
+; GFX9-NEXT:    global_store_short v[4:5], v6, off
 ; GFX9-NEXT:    s_cbranch_vccz BB4_1
 ; GFX9-NEXT:  ; %bb.2: ; %bb2
 ; GFX9-NEXT:    s_endpgm
 bb:
   br label %bb3
 
 bb2:                                              ; preds = %bb3
   ret void

 }
 
 define amdgpu_kernel void @urem16_invariant_denom(i16 addrspace(1)* nocapture %arg, i16 %arg1) {
 ; GFX9-LABEL: urem16_invariant_denom:
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_load_dword s3, s[0:1], 0x2c
 ; GFX9-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
-; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    s_movk_i32 s6, 0x400
+; GFX9-NEXT:    s_mov_b32 s0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v4, s0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s3, s2, s3
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s3
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    v_mov_b32_e32 v3, 0
+; GFX9-NEXT:    s_movk_i32 s6, 0x400
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v0
 ; GFX9-NEXT:  BB5_1: ; %bb3
 ; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; GFX9-NEXT:    v_and_b32_e32 v2, s2, v4
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v8, v2
 ; GFX9-NEXT:    v_lshlrev_b64 v[5:6], 1, v[2:3]
 ; GFX9-NEXT:    v_mov_b32_e32 v7, s5
 ; GFX9-NEXT:    v_add_co_u32_e64 v5, s[0:1], s4, v5
bb3:                                              ; preds = %bb3, %bb
   br i1 %tmp8, label %bb2, label %bb3
 }
 
 define amdgpu_kernel void @sdiv16_invariant_denom(i16 addrspace(1)* nocapture %arg, i16 %arg1) {
 ; GFX9-LABEL: sdiv16_invariant_denom:
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    s_movk_i32 s3, 0x400
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_sext_i32_i16 s2, s2
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s2
+; GFX9-NEXT:    s_movk_i32 s3, 0x400
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v0
 ; GFX9-NEXT:  BB6_1: ; %bb3
 ; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    v_mov_b32_e32 v4, v2
 ; GFX9-NEXT:    v_bfe_i32 v5, v4, 0, 16
 ; GFX9-NEXT:    v_and_b32_e32 v2, 0xffff, v4
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v9, v5
+; GFX9-NEXT:    v_add_u16_e32 v7, 1, v4
 ; GFX9-NEXT:    v_xor_b32_e32 v8, s2, v5
-; GFX9-NEXT:    v_lshlrev_b64 v[5:6], 1, v[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v7, s5
-; GFX9-NEXT:    v_add_co_u32_e64 v5, s[0:1], s4, v5
-; GFX9-NEXT:    v_addc_co_u32_e64 v6, s[0:1], v7, v6, s[0:1]
-; GFX9-NEXT:    v_mul_f32_e32 v7, v9, v1
-; GFX9-NEXT:    v_trunc_f32_e32 v7, v7
-; GFX9-NEXT:    v_ashrrev_i32_e32 v2, 30, v8
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v8, v7
-; GFX9-NEXT:    v_mad_f32 v7, -v7, v0, v9
-; GFX9-NEXT:    v_add_u16_e32 v4, 1, v4
-; GFX9-NEXT:    v_or_b32_e32 v2, 1, v2
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v7|, |v0|
-; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, s3, v4
-; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, v2, s[0:1]
-; GFX9-NEXT:    v_add_u32_e32 v2, v8, v2
+; GFX9-NEXT:    v_lshlrev_b64 v[4:5], 1, v[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v6, s5
+; GFX9-NEXT:    v_add_co_u32_e64 v4, s[0:1], s4, v4
+; GFX9-NEXT:    v_addc_co_u32_e64 v5, s[0:1], v6, v5, s[0:1]
+; GFX9-NEXT:    v_mul_f32_e32 v6, v9, v1
+; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
+; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, s3, v7
+; GFX9-NEXT:    v_mov_b32_e32 v2, v7
+; GFX9-NEXT:    v_ashrrev_i32_e32 v7, 30, v8
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v8, v6
+; GFX9-NEXT:    v_mad_f32 v6, -v6, v0, v9
+; GFX9-NEXT:    v_or_b32_e32 v7, 1, v7
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v6|, |v0|
+; GFX9-NEXT:    v_cndmask_b32_e64 v6, 0, v7, s[0:1]
 ; GFX9-NEXT:    s_and_b64 vcc, exec, vcc
-; GFX9-NEXT:    global_store_short v[5:6], v2, off
+; GFX9-NEXT:    v_add_u32_e32 v6, v8, v6
+; GFX9-NEXT:    global_store_short v[4:5], v6, off
 ; GFX9-NEXT:    s_cbranch_vccz BB6_1
 ; GFX9-NEXT:  ; %bb.2: ; %bb2
 ; GFX9-NEXT:    s_endpgm
 bb:
   br label %bb3
 
 bb2:                                              ; preds = %bb3
   ret void
bb3:                                              ; preds = %bb3, %bb
   br i1 %tmp8, label %bb2, label %bb3
 }
 
 define amdgpu_kernel void @srem16_invariant_denom(i16 addrspace(1)* nocapture %arg, i16 %arg1) {
 ; GFX9-LABEL: srem16_invariant_denom:
 ; GFX9:       ; %bb.0: ; %bb
 ; GFX9-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; GFX9-NEXT:    s_mov_b32 s0, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v3, 0
-; GFX9-NEXT:    s_movk_i32 s3, 0x400
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_sext_i32_i16 s2, s2
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s2
+; GFX9-NEXT:    s_movk_i32 s3, 0x400
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v1, v0
 ; GFX9-NEXT:  BB7_1: ; %bb3
 ; GFX9-NEXT:    ; =>This Inner Loop Header: Depth=1
+; GFX9-NEXT:    v_mov_b32_e32 v4, v2
 ; GFX9-NEXT:    v_bfe_i32 v7, v4, 0, 16
 ; GFX9-NEXT:    v_and_b32_e32 v2, 0xffff, v4
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v10, v7
-; GFX9-NEXT:    v_lshlrev_b64 v[5:6], 1, v[2:3]
-; GFX9-NEXT:    v_mov_b32_e32 v8, s5
-; GFX9-NEXT:    v_add_co_u32_e64 v5, s[0:1], s4, v5
-; GFX9-NEXT:    v_addc_co_u32_e64 v6, s[0:1], v8, v6, s[0:1]
-; GFX9-NEXT:    v_mul_f32_e32 v8, v10, v1
+; GFX9-NEXT:    v_add_u16_e32 v8, 1, v4
+; GFX9-NEXT:    v_lshlrev_b64 v[4:5], 1, v[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v6, s5
+; GFX9-NEXT:    v_add_co_u32_e64 v4, s[0:1], s4, v4
+; GFX9-NEXT:    v_addc_co_u32_e64 v5, s[0:1], v6, v5, s[0:1]
+; GFX9-NEXT:    v_mul_f32_e32 v6, v10, v1
 ; GFX9-NEXT:    v_xor_b32_e32 v9, s2, v7
-; GFX9-NEXT:    v_trunc_f32_e32 v8, v8
-; GFX9-NEXT:    v_ashrrev_i32_e32 v2, 30, v9
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v9, v8
-; GFX9-NEXT:    v_mad_f32 v8, -v8, v0, v10
-; GFX9-NEXT:    v_or_b32_e32 v2, 1, v2
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v8|, |v0|
-; GFX9-NEXT:    v_cndmask_b32_e64 v2, 0, v2, s[0:1]
-; GFX9-NEXT:    v_add_u32_e32 v2, v9, v2
-; GFX9-NEXT:    v_mul_lo_u32 v2, v2, s2
-; GFX9-NEXT:    v_add_u16_e32 v4, 1, v4
-; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, s3, v4
+; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
+; GFX9-NEXT:    v_cmp_eq_u16_e32 vcc, s3, v8
+; GFX9-NEXT:    v_mov_b32_e32 v2, v8
+; GFX9-NEXT:    v_ashrrev_i32_e32 v8, 30, v9
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v9, v6
+; GFX9-NEXT:    v_mad_f32 v6, -v6, v0, v10
+; GFX9-NEXT:    v_or_b32_e32 v8, 1, v8
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v6|, |v0|
+; GFX9-NEXT:    v_cndmask_b32_e64 v6, 0, v8, s[0:1]
+; GFX9-NEXT:    v_add_u32_e32 v6, v9, v6
+; GFX9-NEXT:    v_mul_lo_u32 v6, v6, s2
 ; GFX9-NEXT:    s_and_b64 vcc, exec, vcc
-; GFX9-NEXT:    v_sub_u32_e32 v2, v7, v2
-; GFX9-NEXT:    global_store_short v[5:6], v2, off
+; GFX9-NEXT:    v_sub_u32_e32 v6, v7, v6
+; GFX9-NEXT:    global_store_short v[4:5], v6, off
 ; GFX9-NEXT:    s_cbranch_vccz BB7_1
 ; GFX9-NEXT:  ; %bb.2: ; %bb2
 ; GFX9-NEXT:    s_endpgm
 bb:
   br label %bb3
 
 bb2:                                              ; preds = %bb3
   ret void

llvm/test/CodeGen/AMDGPU/idot4u.ll

 ; GFX8-NEXT:    v_mad_u32_u24 v2, s3, v3, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot4_acc16_vecMul:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, 0xffff
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v6, 0xffff
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NODL-NEXT:    s_lshr_b32 s5, s2, 16
-; GFX9-NODL-NEXT:    s_lshr_b32 s7, s3, 16
-; GFX9-NODL-NEXT:    s_lshr_b32 s4, s2, 24
-; GFX9-NODL-NEXT:    v_and_b32_sdwa v4, v0, s5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-NODL-NEXT:    s_lshr_b32 s6, s3, 24
-; GFX9-NODL-NEXT:    v_and_b32_sdwa v3, v0, s7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-NODL-NEXT:    v_lshl_or_b32 v3, s6, 16, v3
-; GFX9-NODL-NEXT:    v_lshl_or_b32 v4, s4, 16, v4
-; GFX9-NODL-NEXT:    v_pk_mul_lo_u16 v3, v4, v3
-; GFX9-NODL-NEXT:    v_and_b32_sdwa v4, v0, s3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-NODL-NEXT:    v_lshrrev_b16_e64 v2, 8, s3
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s2
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v2, s3
 ; GFX9-NODL-NEXT:    v_lshrrev_b16_e64 v1, 8, s2
-; GFX9-NODL-NEXT:    v_and_b32_sdwa v0, v0, s2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-NODL-NEXT:    v_lshl_or_b32 v2, v2, 16, v4
+; GFX9-NODL-NEXT:    v_and_b32_sdwa v0, v6, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NODL-NEXT:    v_lshrrev_b16_e64 v3, 8, s3
+; GFX9-NODL-NEXT:    v_and_b32_sdwa v2, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NODL-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
 ; GFX9-NODL-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
 ; GFX9-NODL-NEXT:    v_pk_mul_lo_u16 v2, v0, v2
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
-; GFX9-NODL-NEXT:    global_load_ushort v4, v[0:1], off
+; GFX9-NODL-NEXT:    global_load_ushort v3, v[0:1], off
+; GFX9-NODL-NEXT:    s_lshr_b32 s4, s2, 24
+; GFX9-NODL-NEXT:    s_lshr_b32 s2, s2, 16
+; GFX9-NODL-NEXT:    s_lshr_b32 s5, s3, 24
+; GFX9-NODL-NEXT:    s_lshr_b32 s3, s3, 16
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s2
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s3
+; GFX9-NODL-NEXT:    v_and_b32_sdwa v5, v6, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NODL-NEXT:    v_and_b32_sdwa v4, v6, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-NODL-NEXT:    v_lshl_or_b32 v5, s5, 16, v5
+; GFX9-NODL-NEXT:    v_lshl_or_b32 v4, s4, 16, v4
+; GFX9-NODL-NEXT:    v_pk_mul_lo_u16 v4, v4, v5
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NODL-NEXT:    v_add_u32_e32 v4, v2, v4
-; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NODL-NEXT:    v_add_u32_e32 v2, v2, v3
-; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NODL-NEXT:    v_add_u32_e32 v3, v2, v3
+; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NODL-NEXT:    v_add_u32_e32 v2, v2, v4
+; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NODL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_acc16_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0xffff
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_lshr_b32 s5, s2, 16
-; GFX9-DL-NEXT:    s_lshr_b32 s7, s3, 16
-; GFX9-DL-NEXT:    s_lshr_b32 s4, s2, 24
-; GFX9-DL-NEXT:    v_and_b32_sdwa v4, v0, s5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-DL-NEXT:    s_lshr_b32 s6, s3, 24
-; GFX9-DL-NEXT:    v_and_b32_sdwa v3, v0, s7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-DL-NEXT:    v_lshl_or_b32 v3, s6, 16, v3
-; GFX9-DL-NEXT:    v_lshl_or_b32 v4, s4, 16, v4
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v3, v4, v3
-; GFX9-DL-NEXT:    v_and_b32_sdwa v4, v0, s3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-DL-NEXT:    v_lshrrev_b16_e64 v2, 8, s3
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s2
+; GFX9-DL-NEXT:    v_mov_b32_e32 v2, s3
 ; GFX9-DL-NEXT:    v_lshrrev_b16_e64 v1, 8, s2
-; GFX9-DL-NEXT:    v_and_b32_sdwa v0, v0, s2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
-; GFX9-DL-NEXT:    v_lshl_or_b32 v2, v2, 16, v4
+; GFX9-DL-NEXT:    v_and_b32_sdwa v0, v6, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-DL-NEXT:    v_lshrrev_b16_e64 v3, 8, s3
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v6, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-DL-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
 ; GFX9-DL-NEXT:    v_lshl_or_b32 v0, v1, 16, v0
 ; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v2, v0, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
-; GFX9-DL-NEXT:    global_load_ushort v4, v[0:1], off
+; GFX9-DL-NEXT:    global_load_ushort v3, v[0:1], off
+; GFX9-DL-NEXT:    s_lshr_b32 s4, s2, 24
+; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 16
+; GFX9-DL-NEXT:    s_lshr_b32 s5, s3, 24
+; GFX9-DL-NEXT:    s_lshr_b32 s3, s3, 16
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s2
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s3
+; GFX9-DL-NEXT:    v_and_b32_sdwa v5, v6, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-DL-NEXT:    v_and_b32_sdwa v4, v6, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
+; GFX9-DL-NEXT:    v_lshl_or_b32 v5, s5, 16, v5
+; GFX9-DL-NEXT:    v_lshl_or_b32 v4, s4, 16, v4
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v4, v4, v5
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u32_e32 v4, v2, v4
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v3
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u32_e32 v3, v2, v3
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v4
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot4_acc16_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v3, 0xffff
 ; GFX10-DL-NEXT:    ; implicit-def: $vcc_hi

 ; GFX8-LABEL: udot4_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
-; GFX8-NEXT:    s_movk_i32 s0, 0xff
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 24
 ; GFX8-NEXT:    s_lshr_b32 s4, s1, 24
 ; GFX8-NEXT:    s_lshr_b32 s3, s0, 16
-; GFX8-NEXT:    v_mov_b32_e32 v4, s0
-; GFX8-NEXT:    v_mov_b32_e32 v5, s1
+; GFX8-NEXT:    v_mov_b32_e32 v3, s0
+; GFX8-NEXT:    v_mov_b32_e32 v4, s1
 ; GFX8-NEXT:    s_mul_i32 s0, s0, s1
 ; GFX8-NEXT:    s_lshr_b32 s5, s1, 16
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v4, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
-; GFX8-NEXT:    v_and_b32_e32 v3, s0, v3
-; GFX8-NEXT:    v_mov_b32_e32 v6, s4
-; GFX8-NEXT:    v_mov_b32_e32 v7, s2
-; GFX8-NEXT:    v_or_b32_e32 v3, v3, v4
-; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s3, v5
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v7, v6 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v3, v3, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
+; GFX8-NEXT:    v_mov_b32_e32 v4, s5
+; GFX8-NEXT:    v_mov_b32_e32 v7, s0
+; GFX8-NEXT:    v_mov_b32_e32 v5, s4
+; GFX8-NEXT:    v_mov_b32_e32 v6, s2
+; GFX8-NEXT:    v_or_b32_sdwa v3, v7, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX8-NEXT:    v_mul_u32_u24_e32 v4, s3, v4
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v5, v6, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX8-NEXT:    v_and_b32_e32 v3, 0xffff, v3
-; GFX8-NEXT:    v_or_b32_sdwa v4, v5, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_or_b32_e32 v4, v3, v4
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
 ; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.class.f16.ll

entry:
   %r.val.sext = sext i1 %r.val to i32
   store i32 %r.val.sext, i32 addrspace(1)* %r
   ret void
 }
 
 ; GCN-LABEL: {{^}}class_f16_fabs:
 ; GCN: s_load_dword s[[SA_F16:[0-9]+]]
 ; GCN: s_load_dword s[[SB_I32:[0-9]+]]
-; GCN: v_mov_b32_e32 [[V_B_I32:v[0-9]+]], s[[SB_I32]]
-; VI:  v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], |s[[SA_F16]]|, [[V_B_I32]]
+; GCN: v_mov_b32_e32 [[V_A_F16:v[0-9]+]], s[[SA_F16]]
+; VI:  v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], |[[V_A_F16]]|, s[[SB_I32]]
 ; VI:  v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, [[CMP]]
 ; GCN: buffer_store_dword v[[VR_I32]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @class_f16_fabs(
   i32 addrspace(1)* %r,
   [8 x i32],
   half %a.val,
   [8 x i32],
   i32 %b.val) {
 entry:
   %a.val.fabs = call half @llvm.fabs.f16(half %a.val)
   %r.val = call i1 @llvm.amdgcn.class.f16(half %a.val.fabs, i32 %b.val)
   %r.val.sext = sext i1 %r.val to i32
   store i32 %r.val.sext, i32 addrspace(1)* %r
   ret void
 }
 
 ; GCN-LABEL: {{^}}class_f16_fneg:
 ; GCN: s_load_dword s[[SA_F16:[0-9]+]]
 ; GCN: s_load_dword s[[SB_I32:[0-9]+]]
-; GCN: v_mov_b32_e32 [[V_B_I32:v[0-9]+]], s[[SB_I32]]
-; VI:  v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], -s[[SA_F16]], [[V_B_I32]]
+; GCN: v_mov_b32_e32 [[V_A_F16:v[0-9]+]], s[[SA_F16]]
+; VI:  v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], -[[V_A_F16]], s[[SB_I32]]
 ; VI:  v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, [[CMP]]
 ; GCN: buffer_store_dword v[[VR_I32]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @class_f16_fneg(
   i32 addrspace(1)* %r,
   [8 x i32],
   half %a.val,
   [8 x i32],
   i32 %b.val) {
 entry:
   %a.val.fneg = fsub half -0.0, %a.val
   %r.val = call i1 @llvm.amdgcn.class.f16(half %a.val.fneg, i32 %b.val)
   %r.val.sext = sext i1 %r.val to i32
   store i32 %r.val.sext, i32 addrspace(1)* %r
   ret void
 }
 
 ; GCN-LABEL: {{^}}class_f16_fabs_fneg:
 ; GCN: s_load_dword s[[SA_F16:[0-9]+]]
 ; GCN: s_load_dword s[[SB_I32:[0-9]+]]
-; GCN: v_mov_b32_e32 [[V_B_I32:v[0-9]+]], s[[SB_I32]]
-; VI: v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], -|s[[SA_F16]]|, [[V_B_I32]]
+; GCN: v_mov_b32_e32 [[V_A_F16:v[0-9]+]], s[[SA_F16]]
+; VI: v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], -|[[V_A_F16]]|, s[[SB_I32]]
 ; VI: v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, [[CMP]]
 ; GCN: buffer_store_dword v[[VR_I32]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @class_f16_fabs_fneg(
   i32 addrspace(1)* %r,
   [8 x i32],
   half %a.val,
   [8 x i32],
entry:
   %r.val = call i1 @llvm.amdgcn.class.f16(half %a.val, i32 64)
   %r.val.sext = sext i1 %r.val to i32
   store i32 %r.val.sext, i32 addrspace(1)* %r
   ret void
 }
 
 ; GCN-LABEL: {{^}}class_f16_full_mask:
 ; GCN: s_load_dword s[[SA_F16:[0-9]+]]
-; VI:  v_mov_b32_e32 v[[MASK:[0-9]+]], 0x3ff{{$}}
-; VI:  v_cmp_class_f16_e32 vcc, s[[SA_F16]], v[[MASK]]
-; VI:  v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, vcc
+; VI:  s_movk_i32 s[[MASK:[0-9]+]], 0x3ff{{$}}
+; VI:  v_mov_b32_e32 v[[V_A_F16:[0-9]+]], s[[SA_F16]]
+; VI:  v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], v[[V_A_F16]], s[[MASK]]
+; VI:  v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, [[CMP]]
 ; GCN: buffer_store_dword v[[VR_I32]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @class_f16_full_mask(
   i32 addrspace(1)* %r,
   half %a.val) {
 entry:
   %r.val = call i1 @llvm.amdgcn.class.f16(half %a.val, i32 1023)
   %r.val.sext = sext i1 %r.val to i32
   store i32 %r.val.sext, i32 addrspace(1)* %r
   ret void
 }
 
 ; GCN-LABEL: {{^}}class_f16_nine_bit_mask:
 ; GCN: s_load_dword s[[SA_F16:[0-9]+]]
-; VI:  v_mov_b32_e32 v[[MASK:[0-9]+]], 0x1ff{{$}}
-; VI:  v_cmp_class_f16_e32 vcc, s[[SA_F16]], v[[MASK]]
-; VI:  v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, vcc
+; VI:  s_movk_i32 s[[MASK:[0-9]+]], 0x1ff{{$}}
+; VI:  v_mov_b32_e32 v[[V_A_F16:[0-9]+]], s[[SA_F16]]
+; VI:  v_cmp_class_f16_e64 [[CMP:s\[[0-9]+:[0-9]+\]]], v[[V_A_F16]], s[[MASK]]
+; VI:  v_cndmask_b32_e64 v[[VR_I32:[0-9]+]], 0, -1, [[CMP]]
 ; GCN: buffer_store_dword v[[VR_I32]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @class_f16_nine_bit_mask(
   i32 addrspace(1)* %r,
   half %a.val) {
 entry:
   %r.val = call i1 @llvm.amdgcn.class.f16(half %a.val, i32 511)
   %r.val.sext = sext i1 %r.val to i32
   store i32 %r.val.sext, i32 addrspace(1)* %r
   ret void
 }

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fcmp.ll

 ; SI: v_cmp_neq_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_one(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 6)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_ogt:
-; VI: v_cmp_gt_f16_e64
+; VI: v_cmp_lt_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_lt_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_ogt(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 2)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_oge:
-; VI: v_cmp_ge_f16_e64
+; VI: v_cmp_le_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_le_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_oge(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 3)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_olt:
-; VI: v_cmp_lt_f16_e64
+; VI: v_cmp_gt_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_gt_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_olt(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 4)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_ole:
-; VI: v_cmp_le_f16_e64
+; VI: v_cmp_ge_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_ge_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_ole(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 5)
   store i64 %result, i64 addrspace(1)* %out
   ret void

 ; SI: v_cmp_neq_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_une(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 14)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_ugt:
-; VI: v_cmp_nle_f16_e64
+; VI: v_cmp_nge_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_nge_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_ugt(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 10)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_uge:
-; VI: v_cmp_nlt_f16_e64
+; VI: v_cmp_ngt_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_ngt_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_uge(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 11)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_ult:
-; VI: v_cmp_nge_f16_e64
+; VI: v_cmp_nle_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_nle_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_ult(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 12)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fcmp_f16_ule:
-; VI: v_cmp_ngt_f16_e64
+; VI: v_cmp_nlt_f16_e64
 
 ; SI-DAG: s_mov_b32 [[K:s[0-9]+]], 0x42c80000
 ; SI-DAG: v_cvt_f32_f16_e32 [[CVT:v[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_nlt_f32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[K]], [[CVT]]
 define amdgpu_kernel void @v_fcmp_f16_ule(i64 addrspace(1)* %out, half %src) {
   %result = call i64 @llvm.amdgcn.fcmp.f16(half %src, half 100.00, i32 13)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 attributes #0 = { nounwind readnone convergent }

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.icmp.ll

 ; SI: v_cmp_ne_u32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_ne(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 33)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_ugt:
-; VI: v_cmp_gt_u16_e64
+; VI: v_cmp_lt_u16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_and_b32 [[CVT:s[0-9]+]], s{{[0-9]+}}, 0xffff{{$}}
 ; SI: v_cmp_gt_u32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_ugt(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 34)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_uge:
-; VI: v_cmp_ge_u16_e64
+; VI: v_cmp_le_u16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_and_b32 [[CVT:s[0-9]+]], s{{[0-9]+}}, 0xffff{{$}}
 ; SI: v_cmp_ge_u32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_uge(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 35)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_ult:
-; VI: v_cmp_lt_u16_e64
+; VI: v_cmp_gt_u16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_and_b32 [[CVT:s[0-9]+]], s{{[0-9]+}}, 0xffff{{$}}
 ; SI: v_cmp_lt_u32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_ult(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 36)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_ule:
-; VI: v_cmp_le_u16_e64
+; VI: v_cmp_ge_u16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_and_b32 [[CVT:s[0-9]+]], s{{[0-9]+}}, 0xffff{{$}}
 ; SI: v_cmp_le_u32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_ule(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 37)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_sgt:
-; VI: v_cmp_gt_i16_e64
+; VI: v_cmp_lt_i16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_sext_i32_i16 [[CVT:s[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_gt_i32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_sgt(i64 addrspace(1)* %out, i16 %src) #1 {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 38)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_sge:
-; VI: v_cmp_ge_i16_e64
+; VI: v_cmp_le_i16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_sext_i32_i16 [[CVT:s[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_ge_i32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_sge(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 39)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_icmp_i16_slt:
-; VI: v_cmp_lt_i16_e64
+; VI: v_cmp_gt_i16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_sext_i32_i16 [[CVT:s[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_lt_i32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_slt(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 40)
   store i64 %result, i64 addrspace(1)* %out
   ret void
 }
 ; GCN-LABEL: {{^}}v_icmp_i16_sle:
-; VI: v_cmp_le_i16_e64
+; VI: v_cmp_ge_i16_e64
 
 ; SI-DAG: v_mov_b32_e32 [[K:v[0-9]+]], 0x64
 ; SI-DAG: s_sext_i32_i16 [[CVT:s[0-9]+]], s{{[0-9]+}}
 ; SI: v_cmp_le_i32_e64 s{{\[[0-9]+:[0-9]+\]}}, [[CVT]], [[K]]
 define amdgpu_kernel void @v_icmp_i16_sle(i64 addrspace(1)* %out, i16 %src) {
   %result = call i64 @llvm.amdgcn.icmp.i16(i16 %src, i16 100, i32 41)
   store i64 %result, i64 addrspace(1)* %out
   ret void

llvm/test/CodeGen/AMDGPU/llvm.amdgcn.image.sample.d16.dim.ll

main_body:
   ret half %tex
 }
 
 define amdgpu_ps half @image_sample_2d_f16_tfe(<8 x i32> inreg %rsrc, <4 x i32> inreg %samp, float %s, float %t, i32 addrspace(1)* inreg %out) {
 ; TONGA-LABEL: image_sample_2d_f16_tfe:
 ; TONGA:       ; %bb.0: ; %main_body
 ; TONGA-NEXT:    s_mov_b64 s[14:15], exec
 ; TONGA-NEXT:    s_wqm_b64 exec, exec
-; TONGA-NEXT:    v_mov_b32_e32 v2, 0
-; TONGA-NEXT:    v_mov_b32_e32 v4, s12
-; TONGA-NEXT:    v_mov_b32_e32 v5, s13
-; TONGA-NEXT:    v_mov_b32_e32 v3, v2
+; TONGA-NEXT:    v_mov_b32_e32 v4, 0
+; TONGA-NEXT:    v_mov_b32_e32 v2, s12
+; TONGA-NEXT:    v_mov_b32_e32 v3, s13
+; TONGA-NEXT:    v_mov_b32_e32 v5, v4
 ; TONGA-NEXT:    s_and_b64 exec, exec, s[14:15]
-; TONGA-NEXT:    image_sample v[2:3], v[0:1], s[0:7], s[8:11] dmask:0x1 tfe d16
+; TONGA-NEXT:    image_sample v[4:5], v[0:1], s[0:7], s[8:11] dmask:0x1 tfe d16
 ; TONGA-NEXT:    s_waitcnt vmcnt(0)
-; TONGA-NEXT:    v_mov_b32_e32 v0, v2
-; TONGA-NEXT:    flat_store_dword v[4:5], v3
+; TONGA-NEXT:    v_mov_b32_e32 v0, v4
+; TONGA-NEXT:    flat_store_dword v[2:3], v5
 ; TONGA-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; TONGA-NEXT:    ; return to shader part epilog
 ;
 ; GFX81-LABEL: image_sample_2d_f16_tfe:
 ; GFX81:       ; %bb.0: ; %main_body
 ; GFX81-NEXT:    s_mov_b64 s[14:15], exec
 ; GFX81-NEXT:    s_wqm_b64 exec, exec
-; GFX81-NEXT:    v_mov_b32_e32 v2, 0
-; GFX81-NEXT:    v_mov_b32_e32 v4, s12
-; GFX81-NEXT:    v_mov_b32_e32 v5, s13
-; GFX81-NEXT:    v_mov_b32_e32 v3, v2
+; GFX81-NEXT:    v_mov_b32_e32 v4, 0
+; GFX81-NEXT:    v_mov_b32_e32 v2, s12
+; GFX81-NEXT:    v_mov_b32_e32 v3, s13
+; GFX81-NEXT:    v_mov_b32_e32 v5, v4
 ; GFX81-NEXT:    s_and_b64 exec, exec, s[14:15]
 ; GFX81-NEXT:    s_nop 0
-; GFX81-NEXT:    image_sample v[2:3], v[0:1], s[0:7], s[8:11] dmask:0x1 tfe d16
+; GFX81-NEXT:    image_sample v[4:5], v[0:1], s[0:7], s[8:11] dmask:0x1 tfe d16
 ; GFX81-NEXT:    s_waitcnt vmcnt(0)
-; GFX81-NEXT:    v_mov_b32_e32 v0, v2
+; GFX81-NEXT:    v_mov_b32_e32 v0, v4
 ; GFX81-NEXT:    s_nop 0
 ; GFX81-NEXT:    s_nop 0
-; GFX81-NEXT:    flat_store_dword v[4:5], v3
+; GFX81-NEXT:    flat_store_dword v[2:3], v5
 ; GFX81-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GFX81-NEXT:    ; return to shader part epilog
 ;
 ; GFX9-LABEL: image_sample_2d_f16_tfe:
 ; GFX9:       ; %bb.0: ; %main_body
 ; GFX9-NEXT:    s_mov_b64 s[14:15], exec
 ; GFX9-NEXT:    s_wqm_b64 exec, exec
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0
-; GFX9-NEXT:    v_mov_b32_e32 v4, s12
-; GFX9-NEXT:    v_mov_b32_e32 v5, s13
-; GFX9-NEXT:    v_mov_b32_e32 v3, v2
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    v_mov_b32_e32 v2, s12
+; GFX9-NEXT:    v_mov_b32_e32 v3, s13
+; GFX9-NEXT:    v_mov_b32_e32 v5, v4
 ; GFX9-NEXT:    s_and_b64 exec, exec, s[14:15]
-; GFX9-NEXT:    image_sample v[2:3], v[0:1], s[0:7], s[8:11] dmask:0x1 tfe d16
+; GFX9-NEXT:    image_sample v[4:5], v[0:1], s[0:7], s[8:11] dmask:0x1 tfe d16
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mov_b32_e32 v0, v2
-; GFX9-NEXT:    global_store_dword v[4:5], v3, off
+; GFX9-NEXT:    v_mov_b32_e32 v0, v4
+; GFX9-NEXT:    global_store_dword v[2:3], v5, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    ; return to shader part epilog
 ;
 ; GFX10-LABEL: image_sample_2d_f16_tfe:
 ; GFX10:       ; %bb.0: ; %main_body
 ; GFX10-NEXT:    s_mov_b32 s14, exec_lo
 ; GFX10-NEXT:    ; implicit-def: $vcc_hi
 ; GFX10-NEXT:    s_wqm_b32 exec_lo, exec_lo

llvm/test/CodeGen/AMDGPU/partial-shift-shrink.ll

-; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX9 %s
+; RUN: llc -march=amdgcn -mcpu=gfx900 -asm-verbose=0 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefixes=GCN,GFX9 %s
 
 ; Test combine to reduce the width of a 64-bit shift to 32-bit if
 ; truncated to 16-bit.
 
 ; GCN-LABEL: {{^}}trunc_srl_i64_16_to_i16:
 ; GCN: s_waitcnt
 ; GCN-NEXT: v_lshrrev_b32_e32 v0, 16, v0
 ; GCN-NEXT: s_setpc_b64

llvm/test/CodeGen/AMDGPU/setcc-opt.ll

define amdgpu_kernel void @zext_bool_icmp_ne_neg1(i1 addrspace(1)* %out, i32 %a, i32 %b) nounwind {
   store i1 %icmp1, i1 addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}cmp_zext_k_i8max:
 ; SI: s_load_dword [[VALUE:s[0-9]+]], s{{\[[0-9]+:[0-9]+\]}}, 0xb
 ; VI: s_load_dword [[VALUE:s[0-9]+]], s{{\[[0-9]+:[0-9]+\]}}, 0x2c
 ; GCN: s_movk_i32 [[K255:s[0-9]+]], 0xff
-; GCN-DAG: v_mov_b32_e32 [[VK255:v[0-9]+]], [[K255]]
+; SI-DAG: v_mov_b32_e32 [[VK255:v[0-9]+]], [[K255]]
 ; SI-DAG: s_and_b32 [[B:s[0-9]+]], [[VALUE]], [[K255]]
 ; SI: v_cmp_ne_u32_e32 vcc, [[B]], [[VK255]]
 
-; VI-DAG: v_and_b32_e32 [[B:v[0-9]+]], [[VALUE]], [[VK255]]
+; VI-DAG: v_mov_b32_e32 [[VVALUE:v[0-9]+]], [[VALUE]]
+; VI-DAG: v_and_b32_e32 [[B:v[0-9]+]], [[K255]], [[VVALUE]]
 ; VI: v_cmp_ne_u16_e32 vcc, [[K255]], [[B]]
 
 ; GCN: v_cndmask_b32_e64 [[RESULT:v[0-9]+]], 0, 1, vcc
 ; GCN: buffer_store_byte [[RESULT]]
 ; GCN: s_endpgm
 define amdgpu_kernel void @cmp_zext_k_i8max(i1 addrspace(1)* %out, i8 %b) nounwind {
   %b.ext = zext i8 %b to i32
   %icmp0 = icmp ne i32 %b.ext, 255