[AMDGPU] Filtering out the inactive lanes bits when lowering copy to SCC

[alex-t]

Summary

Normally, given that the DA results are kept consistent over the selection DAG, uniform comparisons get selected to S_CMP_* but divergent to V_CMP_*. Sometimes, for the sake of efficiency, SSA subgraphs may be converted to VALU to avoid repeatedly copying data back and forth. Hence we have to be able to sustain the correctness passing the i1 from VALU to SALU context and vice versa.

VALU operations only process the active lanes of the VGPR and ignore inactive ones.
Active lanes correspond to 1 bit in the EXEC mask register.
SALU represents i1 as just one bit but VALU as 64bits: 0/1 and 0/(0xffffffffffffffff & EXEC) respectively.
SALU uses one-bit conditional flag SCC but VALU - VCC that is a pair of 32-bit SGPRs

To expose SCC to the VALU context we need to convert the one-bit boolean value to the appropriate 64bit.
To return back to the SALU context we need to do the opposite.

To correctly convert 64bit VALU boolean to either 0 or 1 we need to filter out the bits corresponding to the inactive lanes.

[ruiling]

When we pass the scalar i1 values to uniform VALU instructions
we use the fllowing representation: false - 0, true - 0xffffffffffffffff.
VALU instructions only process the olanes that are active, that is controlled by the EXEC mask.
We need to filter out odd bits when copy the computation result back to SCC.

Could you add some more description what's going on in failure cases? So that is easy for others to understand the problem. And we also need a test to show the problem.

[arsenm]

Needs a more targeted test

[piotr]

The patch looks good to me (modulo the lint warnings).

For the record, the patch fixes a bug uncovered by D108925 (Vulkan CTS failure) - thanks.

[alex-t]

• detailed summary
• MIR test added
• formatting corrected

[ruiling]

IMO, it is more useful to use an end-to-end test ( from LLVM IR to assembly). We do lots of work scattered in different places to deal with boolean values. Things may change in the future, and we may move this logic to other passes.
Btw, I don't know why you were creating a file with the mode 755. I don't know whether it matters in a patch.

llvm/lib/Target/AMDGPU/SIFixSGPRCopies.cpp

604–619

I guess this is just used to skip the newly inserted instruction, no functional change?

612

Please use Exec as variable name.

[alex-t]

In D109900#3007652, @ruiling wrote:

IMO, it is more useful to use an end-to-end test ( from LLVM IR to assembly). We do lots of work scattered in different places to deal with boolean values. Things may change in the future, and we may move this logic to other passes.

In fact, SIFixSGPRCopies is the only pass that is affected by the change and it runs right after the instruction selection.
Anyway, if you like the *.ll more, so be it.

Btw, I don't know why you were creating a file with the mode 755. I don't know whether it matters in a patch.

It does not matter - just a glitch.
The patch was created on windows and then fed to the 'git apply' on Ubuntu.
Some win2linux magic apparently.

llvm/lib/Target/AMDGPU/SIFixSGPRCopies.cpp

604–619

Yes.

[alex-t]

Test changed to end-to-end variant.
Minor change regarding the variable name.

[ruiling]

The patch LGTM. Any different idea from others?

llvm/lib/Target/AMDGPU/SIFixSGPRCopies.cpp

for (MachineBasicBlock::iterator I = MBB->begin(), E = MBB->end(); I != E;
             I = BuildMI(*MI.getParent(),
                         std::next(MachineBasicBlock::iterator(MI)),
                         MI.getDebugLoc(),
                         TII->get(ST.isWave32() ? AMDGPU::S_CSELECT_B32
                                                : AMDGPU::S_CSELECT_B64),
                         SCCCopy)
                     .addImm(-1)
                     .addImm(0);
-            BuildMI(*MI.getParent(), std::next(I), I->getDebugLoc(),
+            I = BuildMI(*MI.getParent(), std::next(I), I->getDebugLoc(),
                         TII->get(AMDGPU::COPY), DstReg)
                     .addReg(SCCCopy);
             MI.eraseFromParent();
             continue;
+          } else if (DstReg == AMDGPU::SCC) {
+            unsigned Opcode =
+                ST.isWave64() ? AMDGPU::S_AND_B64 : AMDGPU::S_AND_B32;
+            Register Exec = ST.isWave64() ? AMDGPU::EXEC : AMDGPU::EXEC_LO;
+            Register Tmp = MRI->createVirtualRegister(TRI->getBoolRC());
+            I = BuildMI(*MI.getParent(),
+                        std::next(MachineBasicBlock::iterator(MI)),
+                        MI.getDebugLoc(), TII->get(Opcode))
+                    .addReg(Tmp, getDefRegState(true))
+                    .addReg(SrcReg)
+                    .addReg(Exec);
+            MI.eraseFromParent();
+            continue;
           }
         }
 
         if (!DstReg.isVirtual()) {
           // If the destination register is a physical register there isn't
           // really much we can do to fix this.
           // Some special instructions use M0 as an input. Some even only use
           // the first lane. Insert a readfirstlane and hope for the best.

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX9-NEXT:    global_store_short v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: sdiv_i16:
 ; GFX90A:       ; %bb.0:
 ; GFX90A-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX90A-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX90A-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX90A-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX90A-NEXT:    s_ashr_i32 s0, s4, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s0
 ; GFX90A-NEXT:    s_sext_i32_i16 s1, s4
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v2, s1
 ; GFX90A-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX90A-NEXT:    global_store_short v1, v0, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = sdiv i16 %x, %y
   store i16 %r, i16 addrspace(1)* %out
   ret void
 }

 ; GFX9-NEXT:    s_ashr_i32 s2, s2, 30
 ; GFX9-NEXT:    s_or_b32 s6, s2, 1
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mul_f32_e32 v2, v1, v2
 ; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX9-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[2:3], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_and_b64 s[2:3], s[2:3], exec
 ; GFX9-NEXT:    s_cselect_b32 s2, s6, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s2, v2
 ; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s5
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_store_short v1, v0, s[0:1]
 ; GFX9-NEXT:    s_endpgm

 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s6, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX90A-NEXT:    v_mul_lo_u32 v0, v0, s5
 ; GFX90A-NEXT:    v_sub_u32_e32 v0, s4, v0
 ; GFX90A-NEXT:    global_store_short v1, v0, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = srem i16 %x, %y
   store i16 %r, i16 addrspace(1)* %out

 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX9-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: sdiv_i8:
 ; GFX90A:       ; %bb.0:
 ; GFX90A-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX90A-NEXT:    s_load_dword s4, s[0:1], 0x2c
 ; GFX90A-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX90A-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX90A-NEXT:    s_bfe_i32 s0, s4, 0x80008
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s0
 ; GFX90A-NEXT:    s_sext_i32_i8 s1, s4
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v2, s1
 ; GFX90A-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX90A-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = sdiv i8 %x, %y
   store i8 %r, i8 addrspace(1)* %out
   ret void
 }

 ; GFX9-NEXT:    s_ashr_i32 s2, s2, 30
 ; GFX9-NEXT:    s_lshr_b32 s5, s4, 8
 ; GFX9-NEXT:    s_or_b32 s6, s2, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v2, v1, v2
 ; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX9-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[2:3], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_and_b64 s[2:3], s[2:3], exec
 ; GFX9-NEXT:    s_cselect_b32 s2, s6, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s2, v2
 ; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s5
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_store_byte v1, v0, s[0:1]

 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_lshr_b32 s5, s4, 8
 ; GFX90A-NEXT:    s_or_b32 s6, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v1, v2
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v1|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v1, s0, v3
 ; GFX90A-NEXT:    v_mul_lo_u32 v1, v1, s5
 ; GFX90A-NEXT:    v_sub_u32_e32 v1, s4, v1
 ; GFX90A-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = srem i8 %x, %y
   store i8 %r, i8 addrspace(1)* %out

 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s8, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v1, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v1, -v3, v0, v1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX9-NEXT:    s_ashr_i32 s1, s6, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX9-NEXT:    s_ashr_i32 s4, s4, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, s4
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v0
 ; GFX9-NEXT:    v_add_u32_e32 v3, s0, v3
 ; GFX9-NEXT:    v_mul_f32_e32 v4, v1, v4
 ; GFX9-NEXT:    s_xor_b32 s0, s4, s1
 ; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    v_mad_f32 v1, -v4, v0, v1
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
 ; GFX9-NEXT:    s_sext_i32_i16 s1, s7
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_add_u32_e32 v4, s0, v4
 ; GFX9-NEXT:    s_sext_i32_i16 s0, s5
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v0
 ; GFX9-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v5, v1, v5
 ; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX9-NEXT:    v_mad_f32 v1, -v5, v0, v1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    s_ashr_i32 s1, s7, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX9-NEXT:    v_add_u32_e32 v1, s0, v5
 ; GFX9-NEXT:    s_ashr_i32 s0, s5, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v6, v0
 ; GFX9-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX9-NEXT:    v_mad_f32 v5, -v6, v0, v5
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_mov_b32_e32 v5, 0xffff
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v6
 ; GFX9-NEXT:    v_and_b32_e32 v1, v5, v1
 ; GFX9-NEXT:    v_lshl_or_b32 v1, v0, 16, v1
 ; GFX9-NEXT:    v_and_b32_e32 v0, v5, v3
 ; GFX9-NEXT:    v_lshl_or_b32 v0, v4, 16, v0
 ; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]

 ; GFX90A-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s8, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v1, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v1, -v3, v0, v1
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX90A-NEXT:    s_ashr_i32 s1, s6, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX90A-NEXT:    s_ashr_i32 s4, s4, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v1, s4
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v4, v0
 ; GFX90A-NEXT:    v_add_u32_e32 v3, s0, v3
 ; GFX90A-NEXT:    v_mul_f32_e32 v4, v1, v4
 ; GFX90A-NEXT:    s_xor_b32 s0, s4, s1
 ; GFX90A-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_mad_f32 v1, -v4, v0, v1
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
 ; GFX90A-NEXT:    s_sext_i32_i16 s1, s7
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v4, s0, v4
 ; GFX90A-NEXT:    s_sext_i32_i16 s0, s5
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v1, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v5, v0
 ; GFX90A-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v5, v1, v5
 ; GFX90A-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_mad_f32 v1, -v5, v0, v1
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    s_ashr_i32 s1, s7, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX90A-NEXT:    v_add_u32_e32 v1, s0, v5
 ; GFX90A-NEXT:    s_ashr_i32 s0, s5, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v5, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v6, v0
 ; GFX90A-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX90A-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX90A-NEXT:    v_mad_f32 v5, -v6, v0, v5
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_mov_b32_e32 v5, 0xffff
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v6
 ; GFX90A-NEXT:    v_and_b32_e32 v1, v5, v1
 ; GFX90A-NEXT:    v_lshl_or_b32 v1, v0, 16, v1
 ; GFX90A-NEXT:    v_and_b32_e32 v0, v5, v3
 ; GFX90A-NEXT:    v_lshl_or_b32 v0, v4, 16, v0
 ; GFX90A-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]

 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s8, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v1, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v1, -v3, v0, v1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX9-NEXT:    s_ashr_i32 s9, s6, 16
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s9
 ; GFX9-NEXT:    s_ashr_i32 s8, s4, 16
 ; GFX9-NEXT:    v_add_u32_e32 v1, s0, v3
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s8
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v0
 ; GFX9-NEXT:    s_xor_b32 s0, s8, s9
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    v_mul_lo_u32 v1, v1, s6
 ; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX9-NEXT:    v_mad_f32 v3, -v4, v0, v3
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; GFX9-NEXT:    s_or_b32 s6, s0, 1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v4
 ; GFX9-NEXT:    s_sext_i32_i16 s0, s7
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s0
 ; GFX9-NEXT:    s_sext_i32_i16 s1, s5
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s1
 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s6, s0, 1
 ; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s9
 ; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX9-NEXT:    v_mad_f32 v4, -v5, v3, v4
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX9-NEXT:    s_ashr_i32 s6, s7, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s6
 ; GFX9-NEXT:    v_add_u32_e32 v3, s0, v5
 ; GFX9-NEXT:    v_mul_lo_u32 v3, v3, s7
 ; GFX9-NEXT:    s_ashr_i32 s7, s5, 16
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, s7
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v6, v4
 ; GFX9-NEXT:    s_xor_b32 s0, s7, s6
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s9, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX9-NEXT:    v_mad_f32 v5, -v6, v4, v5
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v4|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s9, 0
 ; GFX9-NEXT:    v_add_u32_e32 v4, s0, v6
 ; GFX9-NEXT:    v_mul_lo_u32 v4, v4, s6
 ; GFX9-NEXT:    v_sub_u32_e32 v5, s4, v1
 ; GFX9-NEXT:    v_sub_u32_e32 v1, s5, v3
 ; GFX9-NEXT:    v_sub_u32_e32 v0, s8, v0
 ; GFX9-NEXT:    v_sub_u32_e32 v3, s7, v4
 ; GFX9-NEXT:    v_mov_b32_e32 v4, 0xffff

 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s8, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v1, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v1, -v3, v0, v1
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX90A-NEXT:    s_ashr_i32 s8, s6, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v1, s8
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX90A-NEXT:    v_mul_lo_u32 v0, v0, s6
 ; GFX90A-NEXT:    v_sub_u32_e32 v0, s4, v0
 ; GFX90A-NEXT:    s_ashr_i32 s4, s4, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s4
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v4, v1
 ; GFX90A-NEXT:    s_xor_b32 s0, s4, s8
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s6, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GFX90A-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX90A-NEXT:    v_mad_f32 v3, -v4, v1, v3
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v1|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v1, s0, v4
 ; GFX90A-NEXT:    s_sext_i32_i16 s0, s7
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s0
 ; GFX90A-NEXT:    v_mul_lo_u32 v1, v1, s8
 ; GFX90A-NEXT:    s_sext_i32_i16 s1, s5
 ; GFX90A-NEXT:    v_sub_u32_e32 v4, s4, v1
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v1, s1
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX90A-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v5, v1, v5
 ; GFX90A-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_mad_f32 v1, -v5, v3, v1
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v3|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    s_ashr_i32 s4, s7, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s4
 ; GFX90A-NEXT:    v_add_u32_e32 v1, s0, v5
 ; GFX90A-NEXT:    v_mul_lo_u32 v1, v1, s7
 ; GFX90A-NEXT:    v_sub_u32_e32 v1, s5, v1
 ; GFX90A-NEXT:    s_ashr_i32 s5, s5, 16
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v5, s5
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v6, v3
 ; GFX90A-NEXT:    s_xor_b32 s0, s5, s4
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s6, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX90A-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX90A-NEXT:    v_mad_f32 v5, -v6, v3, v5
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v3|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v3, s0, v6
 ; GFX90A-NEXT:    v_mul_lo_u32 v3, v3, s4
 ; GFX90A-NEXT:    v_mov_b32_e32 v5, 0xffff
 ; GFX90A-NEXT:    v_sub_u32_e32 v3, s5, v3
 ; GFX90A-NEXT:    v_and_b32_e32 v1, v5, v1
 ; GFX90A-NEXT:    v_and_b32_e32 v0, v5, v0
 ; GFX90A-NEXT:    v_lshl_or_b32 v1, v3, 16, v1

 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX9-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX9-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: sdiv_i3:
 ; GFX90A:       ; %bb.0:

 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX90A-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX90A-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = sdiv i3 %x, %y
   store i3 %r, i3 addrspace(1)* %out
   ret void

 ; GFX9-NEXT:    s_ashr_i32 s2, s2, 30
 ; GFX9-NEXT:    s_lshr_b32 s5, s4, 8
 ; GFX9-NEXT:    s_or_b32 s6, s2, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v2, v1, v2
 ; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX9-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[2:3], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[2:3], 0
+; GFX9-NEXT:    s_and_b64 s[2:3], s[2:3], exec
 ; GFX9-NEXT:    s_cselect_b32 s2, s6, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s2, v2
 ; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s5
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    v_sub_u32_e32 v0, s4, v0
 ; GFX9-NEXT:    v_and_b32_e32 v0, 7, v0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_lshr_b32 s5, s4, 8
 ; GFX90A-NEXT:    s_or_b32 s6, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v1, v2
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v1|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s6, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v1, s0, v3
 ; GFX90A-NEXT:    v_mul_lo_u32 v1, v1, s5
 ; GFX90A-NEXT:    v_sub_u32_e32 v1, s4, v1
 ; GFX90A-NEXT:    v_and_b32_e32 v1, 7, v1
 ; GFX90A-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = srem i3 %x, %y

 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_or_b32 s8, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX9-NEXT:    s_ashr_i32 s1, s6, 16
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX9-NEXT:    s_ashr_i32 s4, s4, 16
 ; GFX9-NEXT:    v_add_u32_e32 v2, s0, v3
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s4
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v0
 ; GFX9-NEXT:    s_xor_b32 s0, s4, s1
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX9-NEXT:    v_mad_f32 v3, -v4, v0, v3
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v0|
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
 ; GFX9-NEXT:    s_sext_i32_i16 s1, s7
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_add_u32_e32 v3, s0, v4
 ; GFX9-NEXT:    s_sext_i32_i16 s0, s5
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v0
 ; GFX9-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX9-NEXT:    v_mad_f32 v4, -v5, v0, v4
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v5
 ; GFX9-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX9-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
 ; GFX9-NEXT:    global_store_short v1, v0, s[2:3] offset:4
 ; GFX9-NEXT:    global_store_dword v1, v2, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;

 ; GFX90A-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_or_b32 s8, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX90A-NEXT:    s_ashr_i32 s1, s6, 16
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX90A-NEXT:    s_ashr_i32 s4, s4, 16
 ; GFX90A-NEXT:    v_add_u32_e32 v2, s0, v3
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s4
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v4, v0
 ; GFX90A-NEXT:    s_xor_b32 s0, s4, s1
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GFX90A-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX90A-NEXT:    v_mad_f32 v3, -v4, v0, v3
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v0|
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
 ; GFX90A-NEXT:    s_sext_i32_i16 s1, s7
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v0, s1
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v3, s0, v4
 ; GFX90A-NEXT:    s_sext_i32_i16 s0, s5
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v4, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v5, v0
 ; GFX90A-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX90A-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_mad_f32 v4, -v5, v0, v4
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v5
 ; GFX90A-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX90A-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
 ; GFX90A-NEXT:    global_store_short v1, v0, s[2:3] offset:4
 ; GFX90A-NEXT:    global_store_dword v1, v2, s[2:3]
 ; GFX90A-NEXT:    s_endpgm
   %r = sdiv <3 x i16> %x, %y

 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v2, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s10, s0, 1
 ; GFX9-NEXT:    s_sext_i32_i16 s3, s3
 ; GFX9-NEXT:    v_mul_f32_e32 v2, v1, v2
 ; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
 ; GFX9-NEXT:    v_mad_f32 v1, -v2, v0, v1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s10, 0
 ; GFX9-NEXT:    s_ashr_i32 s2, s2, 16
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v2, v2
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s2
 ; GFX9-NEXT:    s_ashr_i32 s6, s6, 16
 ; GFX9-NEXT:    v_add_u32_e32 v1, s0, v2
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v2, s6
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX9-NEXT:    s_xor_b32 s0, s6, s2
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    v_mul_lo_u32 v1, v1, s8
 ; GFX9-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX9-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX9-NEXT:    s_or_b32 s8, s0, 1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v2, s3
 ; GFX9-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX9-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s2
 ; GFX9-NEXT:    s_sext_i32_i16 s2, s7
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s2
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v2
 ; GFX9-NEXT:    s_xor_b32 s0, s2, s3
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s7, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX9-NEXT:    v_mad_f32 v3, -v4, v2, v3
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v2|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s7, 0
 ; GFX9-NEXT:    v_add_u32_e32 v2, s0, v4
 ; GFX9-NEXT:    v_mul_lo_u32 v2, v2, s3
 ; GFX9-NEXT:    v_sub_u32_e32 v1, s9, v1
 ; GFX9-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-NEXT:    v_sub_u32_e32 v0, s6, v0
 ; GFX9-NEXT:    v_sub_u32_e32 v2, s2, v2
 ; GFX9-NEXT:    v_and_b32_e32 v1, 0xffff, v1

 ; GFX90A-NEXT:    s_xor_b32 s0, s9, s8
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v3, v0
 ; GFX90A-NEXT:    s_or_b32 s10, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v3, v2, v3
 ; GFX90A-NEXT:    v_trunc_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_mad_f32 v2, -v3, v0, v2
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v0|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s10, 0
 ; GFX90A-NEXT:    s_ashr_i32 s6, s6, 16
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v3, v3
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v2, s6
 ; GFX90A-NEXT:    s_ashr_i32 s4, s4, 16
 ; GFX90A-NEXT:    v_add_u32_e32 v0, s0, v3
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s4
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v4, v2
 ; GFX90A-NEXT:    s_xor_b32 s0, s4, s6
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_mul_lo_u32 v0, v0, s8
 ; GFX90A-NEXT:    v_mul_f32_e32 v4, v3, v4
 ; GFX90A-NEXT:    v_trunc_f32_e32 v4, v4
 ; GFX90A-NEXT:    v_mad_f32 v3, -v4, v2, v3
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v4, v4
 ; GFX90A-NEXT:    s_or_b32 s8, s0, 1
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v2|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v2, s0, v4
 ; GFX90A-NEXT:    v_mul_lo_u32 v2, v2, s6
 ; GFX90A-NEXT:    s_sext_i32_i16 s6, s7
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s6
 ; GFX90A-NEXT:    v_sub_u32_e32 v2, s4, v2
 ; GFX90A-NEXT:    s_sext_i32_i16 s4, s5
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v4, s4
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX90A-NEXT:    s_xor_b32 s0, s4, s6
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    s_or_b32 s5, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX90A-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_mad_f32 v4, -v5, v3, v4
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s5, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v3, s0, v5
 ; GFX90A-NEXT:    v_sub_u32_e32 v0, s9, v0
 ; GFX90A-NEXT:    v_mul_lo_u32 v3, v3, s6
 ; GFX90A-NEXT:    v_sub_u32_e32 v3, s4, v3
 ; GFX90A-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX90A-NEXT:    v_lshl_or_b32 v0, v2, 16, v0
 ; GFX90A-NEXT:    global_store_short v1, v3, s[2:3] offset:4

 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
 ; GFX9-NEXT:    s_or_b32 s5, s0, 1
 ; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX9-NEXT:    v_mad_f32 v4, -v5, v3, v4
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX9-NEXT:    s_cselect_b32 s0, s5, 0
 ; GFX9-NEXT:    s_bfe_i32 s1, s6, 0xf000f
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s1
 ; GFX9-NEXT:    v_add_u32_e32 v4, s0, v5
 ; GFX9-NEXT:    s_bfe_i32 s0, s4, 0xf000f
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v6, v3
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s6
 ; GFX9-NEXT:    v_alignbit_b32 v1, s7, v1, 30
 ; GFX9-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX9-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    v_mad_f32 v5, -v6, v3, v5
 ; GFX9-NEXT:    v_bfe_i32 v1, v1, 0, 15
 ; GFX9-NEXT:    s_or_b32 s4, s0, 1
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v3|
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, v1
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX9-NEXT:    v_bfe_i32 v0, v0, 0, 15
 ; GFX9-NEXT:    v_add_u32_e32 v5, s0, v6
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v6, v0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v7, v3
 ; GFX9-NEXT:    v_xor_b32_e32 v0, v0, v1
 ; GFX9-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
 ; GFX9-NEXT:    v_or_b32_e32 v0, 1, v0

 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_alignbit_b32 v0, s5, v0, 30
 ; GFX90A-NEXT:    s_or_b32 s5, s0, 1
 ; GFX90A-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX90A-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX90A-NEXT:    v_mad_f32 v4, -v5, v3, v4
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX90A-NEXT:    s_cselect_b32 s0, s5, 0
 ; GFX90A-NEXT:    s_bfe_i32 s1, s6, 0xf000f
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, s1
 ; GFX90A-NEXT:    v_add_u32_e32 v4, s0, v5
 ; GFX90A-NEXT:    s_bfe_i32 s0, s4, 0xf000f
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v5, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v6, v3
 ; GFX90A-NEXT:    v_mov_b32_e32 v1, s6
 ; GFX90A-NEXT:    v_alignbit_b32 v1, s7, v1, 30
 ; GFX90A-NEXT:    s_xor_b32 s0, s0, s1
 ; GFX90A-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX90A-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_mad_f32 v5, -v6, v3, v5
 ; GFX90A-NEXT:    v_bfe_i32 v1, v1, 0, 15
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v3|
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v3, v1
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_bfe_i32 v0, v0, 0, 15
 ; GFX90A-NEXT:    v_add_u32_e32 v5, s0, v6
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v6, v0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v7, v3
 ; GFX90A-NEXT:    v_xor_b32_e32 v0, v0, v1
 ; GFX90A-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
 ; GFX90A-NEXT:    v_or_b32_e32 v0, 1, v0

 ; GFX9-NEXT:    s_lshr_b32 s9, s4, 15
 ; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0xf000f
 ; GFX9-NEXT:    v_alignbit_b32 v1, s7, v1, 30
 ; GFX9-NEXT:    s_lshr_b32 s7, s6, 15
 ; GFX9-NEXT:    s_bfe_u32 s10, s6, 0xf000f
 ; GFX9-NEXT:    s_or_b32 s11, s0, 1
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v2|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s11, 0
 ; GFX9-NEXT:    v_add_u32_e32 v2, s0, v4
 ; GFX9-NEXT:    s_bfe_i32 s0, s10, 0xf0000
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s0
 ; GFX9-NEXT:    s_bfe_i32 s1, s5, 0xf0000
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s1
 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX9-NEXT:    s_or_b32 s5, s0, 1
 ; GFX9-NEXT:    v_and_b32_e32 v1, s8, v1
 ; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
 ; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
 ; GFX9-NEXT:    v_mad_f32 v4, -v5, v3, v4
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX9-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX9-NEXT:    s_cselect_b32 s0, s5, 0
 ; GFX9-NEXT:    v_bfe_i32 v4, v1, 0, 15
 ; GFX9-NEXT:    v_add_u32_e32 v3, s0, v5
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, v4
 ; GFX9-NEXT:    v_and_b32_e32 v0, s8, v0
 ; GFX9-NEXT:    v_bfe_i32 v6, v0, 0, 15
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v7, v6
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v8, v5

 ; GFX90A-NEXT:    v_alignbit_b32 v0, s5, v0, 30
 ; GFX90A-NEXT:    s_lshr_b32 s5, s4, 15
 ; GFX90A-NEXT:    s_bfe_u32 s9, s4, 0xf000f
 ; GFX90A-NEXT:    v_alignbit_b32 v1, s7, v1, 30
 ; GFX90A-NEXT:    s_lshr_b32 s7, s6, 15
 ; GFX90A-NEXT:    s_bfe_u32 s10, s6, 0xf000f
 ; GFX90A-NEXT:    s_or_b32 s11, s0, 1
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s11, 0
 ; GFX90A-NEXT:    v_add_u32_e32 v3, s0, v5
 ; GFX90A-NEXT:    s_bfe_i32 s0, s10, 0xf0000
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v4, s0
 ; GFX90A-NEXT:    s_bfe_i32 s1, s9, 0xf0000
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v5, s1
 ; GFX90A-NEXT:    s_xor_b32 s0, s1, s0
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v6, v4
 ; GFX90A-NEXT:    v_mul_lo_u32 v3, v3, s6
 ; GFX90A-NEXT:    s_ashr_i32 s0, s0, 30
 ; GFX90A-NEXT:    v_sub_u32_e32 v3, s4, v3
 ; GFX90A-NEXT:    v_mul_f32_e32 v6, v5, v6
 ; GFX90A-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX90A-NEXT:    v_mad_f32 v5, -v6, v4, v5
 ; GFX90A-NEXT:    v_cvt_i32_f32_e32 v6, v6
 ; GFX90A-NEXT:    s_or_b32 s4, s0, 1
 ; GFX90A-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v4|
 ; GFX90A-NEXT:    v_and_b32_e32 v1, s8, v1
-; GFX90A-NEXT:    s_cmp_lg_u64 s[0:1], 0
+; GFX90A-NEXT:    s_and_b64 s[0:1], s[0:1], exec
 ; GFX90A-NEXT:    s_cselect_b32 s0, s4, 0
 ; GFX90A-NEXT:    v_bfe_i32 v5, v1, 0, 15
 ; GFX90A-NEXT:    v_add_u32_e32 v4, s0, v6
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v6, v5
 ; GFX90A-NEXT:    v_and_b32_e32 v0, s8, v0
 ; GFX90A-NEXT:    v_bfe_i32 v7, v0, 0, 15
 ; GFX90A-NEXT:    v_cvt_f32_i32_e32 v8, v7
 ; GFX90A-NEXT:    v_rcp_iflag_f32_e32 v9, v6

llvm/test/CodeGen/AMDGPU/copy_to_scc.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -march=amdgcn -mcpu=gfx900 -verify-machineinstrs < %s | FileCheck -check-prefix=GCN %s
+
+define amdgpu_kernel void @copy_to_scc(i32 addrspace(1)* %out, i32 addrspace(1)* %in, <4 x i32> addrspace(4)* %addrSrc) {
+; GCN-LABEL: copy_to_scc:
+; GCN:       ; %bb.0: ; %entry
+; GCN-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
+; GCN-NEXT:    s_load_dwordx2 s[8:9], s[0:1], 0x34
+; GCN-NEXT:    v_mov_b32_e32 v0, 0
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    s_load_dwordx4 s[0:3], s[8:9], 0x0
+; GCN-NEXT:    s_load_dword s10, s[6:7], 0x0
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    buffer_load_dword v1, off, s[0:3], 0 offset:252
+; GCN-NEXT:    s_cmp_lg_u32 s10, 0
+; GCN-NEXT:    s_cselect_b64 s[6:7], -1, 0
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v1
+; GCN-NEXT:    s_xor_b64 s[0:1], s[6:7], vcc
+; GCN-NEXT:    s_and_b64 s[0:1], s[0:1], exec
+; GCN-NEXT:    s_cselect_b32 s0, 2, 3
+; GCN-NEXT:    v_mov_b32_e32 v1, s0
+; GCN-NEXT:    global_store_dword v0, v1, s[4:5]
+; GCN-NEXT:    s_endpgm
+entry:                                             ; preds = %1009
+  %0 = load i32, i32 addrspace(1)* %in, align 4
+  %1 = load <4 x i32>, <4 x i32> addrspace(4)* %addrSrc, align 16
+  %2 = icmp ne i32 %0, 0
+  %3 = call i32 @llvm.amdgcn.raw.buffer.load.i32(<4 x i32> %1, i32 252, i32 0, i32 0)
+  %4 = icmp ne i32 %3, 0
+  %5 = xor i1 %2, %4
+  %result = select i1 %5, i32 2, i32 3
+  store i32 %result, i32 addrspace(1)* %out
+  ret void
+}
+
+declare i32 @llvm.amdgcn.raw.buffer.load.i32(<4 x i32>, i32, i32, i32 immarg)