[DAG] getNode() - fold (zext (trunc x)) -> x iff the upper bits are known zero - add SRL support

[RKSimon]

Summary

This is part of the work to address the D155472 regressions, there's a number of issues with generalizing this fold which is why I'm just adding SRL support atm.

The fold encourages the creation of ISD::FSHR nodes which was breaking a lot of the AMDGPUISD::PERM tests, so I've added ISD::FSHR handling to the matchPERM helper

[goldstein.w.n]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

5698

Why does the ISD::AssertZext need one use check?

But while you're at it think ISD::AND also work here right?

[jrbyrnes]

what is the best way to generalize the performOrCombine handling to support ISD::FSHR as well?

We should ideally produce AMDGPUISD::PERM whenever we have an i32 that is byte permutation of two sources (except some special cases) and it's best to do this via DAGCombining due to target-specific reasons. In practice, typically such byte permutations have ISD::OR as the root of the tree (which is why I have begun with that combine), however, other cases exist where the tree does not have ISD::OR as the root (e.g. AMDGPUISD::PERM, ISD::FSHR, etc). It seems to me, the best way to generalize is to extract the common code from ISD::OR, and use it to enable new roots / entry points -- this is an extension of the work we should do anyway. I hacked together an implementation of what I have just described and this patch no longer causes the v_perm miss regressions. @foad @arsenm objections?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

5698

Everything should work here - eventually the getOpcode() filters will go entirely - the problem we're having is a number of AArch64 mul folds create various zext(trunc()) patterns and then expect them to still exist later in the combine.

As usual with LLVM it comes down to yak shaving :)

[RKSimon]

@jrbyrnes I've pulled the AMDGPUISD::PERM matching out of performOrCombine, made it into a 'matchPERM' helper and setup ISD::FSHR to use it - is this what you had in mind?

[jrbyrnes]

Disregard previous version of comment -- I did not see the most recent changes.

[jrbyrnes]

Can you please bring in tests from https://github.com/llvm/llvm-project/pull/66965

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

10775 ↗

(On Diff #557146)

Can we also fail on vector types

10791 ↗

(On Diff #557146)

This seems to be broken if Index = 1; BytesProvided; = 2 and ByteShift = 3 -- we will attempt to get the 2nd byte of a 16 bit operand which is out of range (only 0,1 are valid bytes).

Perhaps it was meant to be: Index + (BytesProvided - ByteShift) ? But this still doesnt account for the case where ByteShift > n * BytesProvided.

How do you feel about

case ISD::FSHR: {
  auto ShiftOp = dyn_cast<ConstantSDNode>(Op->getOperand(2));
  if (!ShiftOp || Op.getValueType().isVector())
    return std::nullopt;

  uint64_t BitsProvided = Op.getValueSizeInBits();
  if (BitsProvided % 8 != 0)
    return std::nullopt;

  uint64_t BitShift = ShiftOp->getAPIntValue().urem(BitsProvided);
  if (BitShift % 8)
    return std::nullopt;

  uint64_t ConcatSizeInBytes = BitsProvided / 4;
  uint64_t ByteShift = BitShift / 8;

  auto NewIndex = (Index + ByteShift) % ConcatSizeInBytes;
  auto BytesProvided = BitsProvided / 8;
  auto NextOp = Op.getOperand(NewIndex >= BytesProvided ? 0 : 1);
  NewIndex %= BytesProvided;
  return calculateSrcByte(NextOp, StartingIndex, NewIndex);
}

llvm/test/CodeGen/AMDGPU/permute.ll

127 ↗

(On Diff #557146)

I can look into this after this patch lands.

[jrbyrnes]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

10791 ↗

(On Diff #557146)

Oops -- should have been

return calculateByteProvider(NextOp, NewIndex, Depth + 1, StartingIndex

[RKSimon]

Update with @jrbyrnes suggestions and add early-out so we prefer an existing ISD::FSHR node to a AMDGPUISD::PERM equivalent.

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

5698

@goldstein.w.n Hopefully D159537 will address the last regressions which will allow me to remove the OpOp.getOpcode() filter entirely.

[jrbyrnes]

Thanks -- the AMDGPUISD::PERM side LGTM

[RKSimon]

Thanks - any further comments?

[foad]

@jrbyrnes shouldn't we land the AMDGPU changes separately first? Or would it all be dead code without the rest of this patch?

[RKSimon]

In D159533#4649980, @foad wrote:

@jrbyrnes shouldn't we land the AMDGPU changes separately first? Or would it all be dead code without the rest of this patch?

I can pre-commit the matchPERM helper as a NFC-ish to reduce the diff if that helps?

[RKSimon]

rebase

[jrbyrnes]

In D159533#4649980, @foad wrote:

@jrbyrnes shouldn't we land the AMDGPU changes separately first? Or would it all be dead code without the rest of this patch?

The changes to calculateByteProvider may actually be NFC because sub 32bit ISD::FSHRs get legalized into ISD::OR which are already handled, but enabling ISD::FSHR combine and installing matchPERM there is not.

The changes look okay, but it seems it would be best to land them separately first.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

case ISD::ZERO_EXTEND:
     assert(N1.getValueType().bitsLT(VT) && "Invalid zext node, dst < src!");
     if (OpOpcode == ISD::ZERO_EXTEND) // (zext (zext x)) -> (zext x)
       return getNode(ISD::ZERO_EXTEND, DL, VT, N1.getOperand(0));
     if (OpOpcode == ISD::UNDEF)
       // zext(undef) = 0, because the top bits will be zero.
       return getConstant(0, DL, VT);
 
     // Skip unnecessary zext_inreg pattern:
-    // (zext (trunc (assertzext x))) -> (assertzext x)
+    // (zext (trunc x)) -> x iff the upper bits are known zero.
     // TODO: Generalize to MaskedValueIsZero check?
     if (OpOpcode == ISD::TRUNCATE) {
       SDValue OpOp = N1.getOperand(0);
       if (OpOp.getValueType() == VT) {
-        if (OpOp.getOpcode() == ISD::AssertZext && N1->hasOneUse()) {
+        if ((OpOp.getOpcode() == ISD::AssertZext && N1->hasOneUse()) ||
+            OpOp.getOpcode() == ISD::SRL) {
           APInt HiBits = APInt::getBitsSetFrom(VT.getScalarSizeInBits(),
                                                N1.getScalarValueSizeInBits());
           if (MaskedValueIsZero(OpOp, HiBits)) {
             transferDbgValues(N1, OpOp);
             return OpOp;
           }
         }
       }

llvm/test/CodeGen/AMDGPU/ctpop16.ll

 ; SI-NEXT:    s_branch .LBB14_2
 ;
 ; VI-LABEL: ctpop_i16_in_br:
 ; VI:       ; %bb.0: ; %entry
 ; VI-NEXT:    s_load_dword s4, s[0:1], 0x34
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshr_b32 s5, s4, 16
-; VI-NEXT:    v_cmp_ne_u16_e64 s[6:7], s5, 0
-; VI-NEXT:    s_and_b64 vcc, exec, s[6:7]
-; VI-NEXT:    s_cbranch_vccz .LBB14_4
+; VI-NEXT:    s_cmp_lg_u32 s5, 0
+; VI-NEXT:    s_cbranch_scc0 .LBB14_4
 ; VI-NEXT:  ; %bb.1: ; %else
 ; VI-NEXT:    s_mov_b32 s11, 0xf000
 ; VI-NEXT:    s_mov_b32 s10, -1
 ; VI-NEXT:    s_mov_b32 s8, s2
 ; VI-NEXT:    s_mov_b32 s9, s3
 ; VI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0 offset:2
 ; VI-NEXT:    s_mov_b64 s[2:3], 0
 ; VI-NEXT:    s_cbranch_execnz .LBB14_3

llvm/test/CodeGen/AMDGPU/permute_i8.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck %s -check-prefixes=GFX10
 ; RUN: llc -mtriple=amdgcn -mcpu=gfx908  -verify-machineinstrs < %s | FileCheck %s -check-prefixes=GFX9
 
 define hidden void @shuffle6766(ptr addrspace(1) %in0, ptr addrspace(1) %in1, ptr addrspace(1) %out0) {
 ; GFX10-LABEL: shuffle6766:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v0, v[2:3], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_perm_b32 v0, v0, v0, 0x6060706
+; GFX10-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX10-NEXT:    v_lshlrev_b16 v2, 8, v1
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX10-NEXT:    global_store_dword v[4:5], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: shuffle6766:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v0, v[2:3], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x6060706
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v0, v0, s4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX9-NEXT:    v_lshlrev_b16_e32 v2, 8, v1
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX9-NEXT:    global_store_dword v[4:5], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %vec0 = load <4 x i8>, ptr addrspace(1) %in0, align 4
   %vec1 = load <4 x i8>, ptr addrspace(1) %in1, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> %vec1, <4 x i32> <i32 6, i32 7, i32 6, i32 6>
   store <4 x i8> %shuffle0_0, ptr addrspace(1) %out0, align 4
   ret void

 
 
 define hidden void @shuffle7330ud2(ptr addrspace(1) %in0, ptr addrspace(1) %out0) {
 ; GFX10-LABEL: shuffle7330ud2:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v0, v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_perm_b32 v0, v0, v0, 0x4070706
+; GFX10-NEXT:    v_lshlrev_b16 v1, 8, v0
+; GFX10-NEXT:    v_or_b32_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX10-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: shuffle7330ud2:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v0, v[0:1], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x4070706
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v0, v0, s4
+; GFX9-NEXT:    v_lshlrev_b16_e32 v1, 8, v0
+; GFX9-NEXT:    v_or_b32_sdwa v1, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX9-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %vec0 = load <4 x i8>, ptr addrspace(1) %in0, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> undef, <4 x i32> <i32 7, i32 3, i32 3, i32 0>
   store <4 x i8> %shuffle0_0, ptr addrspace(1) %out0, align 4
   ret void
 }
 
 define hidden void @shuffle5341ud2(ptr addrspace(1) %in0, ptr addrspace(1) %out0) {
 ; GFX10-LABEL: shuffle5341ud2:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v0, v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_perm_b32 v0, v0, v0, 0x5040706
+; GFX10-NEXT:    v_alignbit_b32 v0, v0, v0, 16
 ; GFX10-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: shuffle5341ud2:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v0, v[0:1], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x5040706
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v0, v0, s4
+; GFX9-NEXT:    v_alignbit_b32 v0, v0, v0, 16
 ; GFX9-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %vec0 = load <4 x i8>, ptr addrspace(1) %in0, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> undef, <4 x i32> <i32 5, i32 3, i32 4, i32 1>
   store <4 x i8> %shuffle0_0, ptr addrspace(1) %out0, align 4
   ret void
 }

 
 
 define hidden void @shuffle4327ud2(ptr addrspace(1) %in0, ptr addrspace(1) %out0) {
 ; GFX10-LABEL: shuffle4327ud2:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v0, v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_perm_b32 v0, v0, v0, 0x7060706
+; GFX10-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX10-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX10-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: shuffle4327ud2:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v0, v[0:1], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x7060706
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v0, v0, s4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX9-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX9-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %vec0 = load <4 x i8>, ptr addrspace(1) %in0, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> undef, <4 x i32> <i32 4, i32 3, i32 2, i32 7>
   store <4 x i8> %shuffle0_0, ptr addrspace(1) %out0, align 4
   ret void
 }

 }
 
 define hidden void @shuffle2763ud2(ptr addrspace(1) %in0, ptr addrspace(1) %out0) {
 ; GFX10-LABEL: shuffle2763ud2:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v0, v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_perm_b32 v0, v0, v0, 0x7060706
+; GFX10-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX10-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX10-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: shuffle2763ud2:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v0, v[0:1], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x7060706
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v0, v0, s4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 16, v0
+; GFX9-NEXT:    v_alignbit_b32 v0, v1, v0, 16
 ; GFX9-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %vec0 = load <4 x i8>, ptr addrspace(1) %in0, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> undef, <4 x i32> <i32 2, i32 7, i32 6, i32 3>
   store <4 x i8> %shuffle0_0, ptr addrspace(1) %out0, align 4
   ret void
 }

 ; GFX10-NEXT:    v_and_b32_e32 v4, 0x3ff, v31
 ; GFX10-NEXT:    v_lshlrev_b32_e32 v4, 2, v4
 ; GFX10-NEXT:    v_add_co_u32 v2, vcc_lo, v2, v4
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v3, vcc_lo, 0, v3, vcc_lo
 ; GFX10-NEXT:    v_add_co_u32 v0, vcc_lo, v0, v4
 ; GFX10-NEXT:    v_add_co_ci_u32_e32 v1, vcc_lo, 0, v1, vcc_lo
 ; GFX10-NEXT:    global_load_dword v4, v[2:3], off
 ; GFX10-NEXT:    global_load_dword v9, v[0:1], off
+; GFX10-NEXT:    v_mov_b32_e32 v0, 0xff
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-NEXT:    v_bfe_i32 v0, v4, 0, 8
+; GFX10-NEXT:    v_bfe_i32 v1, v4, 0, 8
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_ashrrev_i32_e32 v2, 24, v9
-; GFX10-NEXT:    v_bfe_i32 v3, v4, 8, 8
-; GFX10-NEXT:    v_bfe_i32 v1, v9, 16, 8
-; GFX10-NEXT:    v_bfe_i32 v10, v4, 16, 8
-; GFX10-NEXT:    v_cvt_f32_i32_e32 v13, v0
-; GFX10-NEXT:    v_ashrrev_i32_e32 v11, 24, v4
-; GFX10-NEXT:    v_xor_b32_e32 v15, v2, v3
-; GFX10-NEXT:    v_cvt_f32_i32_e32 v3, v3
-; GFX10-NEXT:    v_xor_b32_e32 v12, v1, v0
-; GFX10-NEXT:    v_rcp_iflag_f32_e32 v16, v13
+; GFX10-NEXT:    v_ashrrev_i32_e32 v3, 24, v9
+; GFX10-NEXT:    v_bfe_i32 v10, v4, 8, 8
+; GFX10-NEXT:    v_bfe_i32 v2, v9, 16, 8
+; GFX10-NEXT:    v_bfe_i32 v11, v4, 16, 8
 ; GFX10-NEXT:    v_cvt_f32_i32_e32 v14, v1
-; GFX10-NEXT:    v_xor_b32_e32 v1, v1, v10
+; GFX10-NEXT:    v_ashrrev_i32_e32 v12, 24, v4
+; GFX10-NEXT:    v_xor_b32_e32 v16, v3, v10
 ; GFX10-NEXT:    v_cvt_f32_i32_e32 v10, v10
-; GFX10-NEXT:    v_rcp_iflag_f32_e32 v17, v3
-; GFX10-NEXT:    v_xor_b32_e32 v0, v0, v11
+; GFX10-NEXT:    v_xor_b32_e32 v13, v2, v1
+; GFX10-NEXT:    v_rcp_iflag_f32_e32 v17, v14
+; GFX10-NEXT:    v_cvt_f32_i32_e32 v15, v2
+; GFX10-NEXT:    v_xor_b32_e32 v2, v2, v11
 ; GFX10-NEXT:    v_cvt_f32_i32_e32 v11, v11
-; GFX10-NEXT:    v_cvt_f32_i32_e32 v2, v2
 ; GFX10-NEXT:    v_rcp_iflag_f32_e32 v18, v10
-; GFX10-NEXT:    v_ashrrev_i32_e32 v12, 30, v12
-; GFX10-NEXT:    v_mul_f32_e32 v16, v14, v16
+; GFX10-NEXT:    v_xor_b32_e32 v1, v1, v12
+; GFX10-NEXT:    v_cvt_f32_i32_e32 v12, v12
+; GFX10-NEXT:    v_cvt_f32_i32_e32 v3, v3
 ; GFX10-NEXT:    v_rcp_iflag_f32_e32 v19, v11
-; GFX10-NEXT:    v_ashrrev_i32_e32 v15, 30, v15
-; GFX10-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
-; GFX10-NEXT:    v_mul_f32_e32 v17, v2, v17
-; GFX10-NEXT:    v_trunc_f32_e32 v16, v16
-; GFX10-NEXT:    v_or_b32_e32 v12, 1, v12
-; GFX10-NEXT:    v_or_b32_e32 v15, 1, v15
-; GFX10-NEXT:    v_mul_f32_e32 v18, v14, v18
+; GFX10-NEXT:    v_ashrrev_i32_e32 v13, 30, v13
+; GFX10-NEXT:    v_mul_f32_e32 v17, v15, v17
+; GFX10-NEXT:    v_rcp_iflag_f32_e32 v20, v12
+; GFX10-NEXT:    v_ashrrev_i32_e32 v16, 30, v16
+; GFX10-NEXT:    v_ashrrev_i32_e32 v2, 30, v2
+; GFX10-NEXT:    v_mul_f32_e32 v18, v3, v18
 ; GFX10-NEXT:    v_trunc_f32_e32 v17, v17
-; GFX10-NEXT:    v_mad_f32 v20, -v16, v13, v14
-; GFX10-NEXT:    v_mul_f32_e32 v19, v13, v19
-; GFX10-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
+; GFX10-NEXT:    v_or_b32_e32 v13, 1, v13
+; GFX10-NEXT:    v_or_b32_e32 v16, 1, v16
+; GFX10-NEXT:    v_mul_f32_e32 v19, v15, v19
 ; GFX10-NEXT:    v_trunc_f32_e32 v18, v18
-; GFX10-NEXT:    v_mad_f32 v2, -v17, v3, v2
-; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v20|, |v13|
+; GFX10-NEXT:    v_mad_f32 v21, -v17, v14, v15
+; GFX10-NEXT:    v_mul_f32_e32 v20, v14, v20
+; GFX10-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
 ; GFX10-NEXT:    v_trunc_f32_e32 v19, v19
+; GFX10-NEXT:    v_mad_f32 v3, -v18, v10, v3
+; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v21|, |v14|
+; GFX10-NEXT:    v_trunc_f32_e32 v20, v20
+; GFX10-NEXT:    v_or_b32_e32 v2, 1, v2
+; GFX10-NEXT:    v_mad_f32 v15, -v19, v11, v15
 ; GFX10-NEXT:    v_or_b32_e32 v1, 1, v1
-; GFX10-NEXT:    v_mad_f32 v14, -v18, v10, v14
-; GFX10-NEXT:    v_or_b32_e32 v0, 1, v0
-; GFX10-NEXT:    v_cndmask_b32_e32 v12, 0, v12, vcc_lo
-; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v2|, |v3|
-; GFX10-NEXT:    v_mad_f32 v21, -v19, v11, v13
-; GFX10-NEXT:    v_cvt_i32_f32_e32 v16, v16
+; GFX10-NEXT:    v_cndmask_b32_e32 v13, 0, v13, vcc_lo
+; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v3|, |v10|
+; GFX10-NEXT:    v_mad_f32 v22, -v20, v12, v14
 ; GFX10-NEXT:    v_cvt_i32_f32_e32 v17, v17
 ; GFX10-NEXT:    v_cvt_i32_f32_e32 v18, v18
-; GFX10-NEXT:    v_cndmask_b32_e32 v2, 0, v15, vcc_lo
-; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v14|, |v10|
 ; GFX10-NEXT:    v_cvt_i32_f32_e32 v19, v19
-; GFX10-NEXT:    v_add_nc_u32_e32 v3, v16, v12
-; GFX10-NEXT:    v_add_nc_u32_sdwa v2, v17, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_cndmask_b32_e32 v3, 0, v16, vcc_lo
+; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v15|, |v11|
+; GFX10-NEXT:    v_cvt_i32_f32_e32 v20, v20
+; GFX10-NEXT:    v_add_nc_u32_e32 v10, v17, v13
+; GFX10-NEXT:    v_and_b32_sdwa v0, v9, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-NEXT:    v_add_nc_u32_sdwa v3, v18, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_cndmask_b32_e32 v2, 0, v2, vcc_lo
+; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v22|, |v12|
+; GFX10-NEXT:    v_lshlrev_b16 v4, 8, v4
+; GFX10-NEXT:    v_or_b32_sdwa v3, v10, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_add_nc_u32_e32 v2, v19, v2
 ; GFX10-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc_lo
-; GFX10-NEXT:    v_cmp_ge_f32_e64 vcc_lo, |v21|, |v11|
-; GFX10-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-NEXT:    v_add_nc_u32_e32 v1, v18, v1
-; GFX10-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc_lo
-; GFX10-NEXT:    v_add_nc_u32_sdwa v0, v19, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-NEXT:    v_or_b32_sdwa v0, v1, v0 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-NEXT:    v_perm_b32 v1, v9, v4, 0x60706
-; GFX10-NEXT:    v_or_b32_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-NEXT:    global_store_dword v[5:6], v0, off
-; GFX10-NEXT:    global_store_dword v[7:8], v1, off
+; GFX10-NEXT:    v_or_b32_e32 v0, v0, v4
+; GFX10-NEXT:    v_add_nc_u32_sdwa v1, v20, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_alignbit_b32 v0, v0, v9, 16
+; GFX10-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    global_store_dword v[5:6], v1, off
+; GFX10-NEXT:    global_store_dword v[7:8], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: sdiv_store_div:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_and_b32_e32 v4, 0x3ff, v31
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 2, v4
 ; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, v0, v4
 ; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v2, v4
 ; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
-; GFX9-NEXT:    global_load_dword v4, v[2:3], off
-; GFX9-NEXT:    global_load_dword v9, v[0:1], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x60706
+; GFX9-NEXT:    global_load_dword v4, v[0:1], off
+; GFX9-NEXT:    global_load_dword v9, v[2:3], off
+; GFX9-NEXT:    s_movk_i32 s4, 0xff
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_bfe_i32 v1, v4, 0, 8
+; GFX9-NEXT:    v_ashrrev_i32_e32 v10, 24, v4
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v9, v4, s4
-; GFX9-NEXT:    v_bfe_i32 v2, v9, 16, 8
-; GFX9-NEXT:    v_ashrrev_i32_e32 v3, 24, v9
-; GFX9-NEXT:    v_bfe_i32 v9, v4, 8, 8
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v12, v1
-; GFX9-NEXT:    v_bfe_i32 v10, v4, 16, 8
-; GFX9-NEXT:    v_ashrrev_i32_e32 v4, 24, v4
-; GFX9-NEXT:    v_xor_b32_e32 v14, v3, v9
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v9, v9
-; GFX9-NEXT:    v_xor_b32_e32 v11, v2, v1
+; GFX9-NEXT:    v_bfe_i32 v2, v9, 0, 8
+; GFX9-NEXT:    v_bfe_i32 v11, v9, 8, 8
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v13, v2
-; GFX9-NEXT:    v_xor_b32_e32 v2, v2, v10
+; GFX9-NEXT:    v_and_b32_sdwa v0, v4, s4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e32 v1, 8, v9
+; GFX9-NEXT:    v_bfe_i32 v3, v4, 16, 8
+; GFX9-NEXT:    v_bfe_i32 v12, v9, 16, 8
+; GFX9-NEXT:    v_ashrrev_i32_e32 v9, 24, v9
+; GFX9-NEXT:    v_xor_b32_e32 v15, v10, v11
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v11, v11
+; GFX9-NEXT:    v_or_b32_e32 v0, v0, v1
+; GFX9-NEXT:    v_xor_b32_e32 v1, v3, v2
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v14, v3
+; GFX9-NEXT:    v_xor_b32_e32 v3, v3, v12
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v12, v12
+; GFX9-NEXT:    v_xor_b32_e32 v2, v2, v9
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v9, v9
+; GFX9-NEXT:    v_alignbit_b32 v0, v0, v4, 16
+; GFX9-NEXT:    v_ashrrev_i32_e32 v4, 30, v15
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v15, v13
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v10, v10
-; GFX9-NEXT:    v_xor_b32_e32 v1, v1, v4
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, v4
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v15, v12
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, v3
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v16, v9
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v17, v10
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v18, v4
-; GFX9-NEXT:    v_mul_f32_e32 v15, v13, v15
-; GFX9-NEXT:    v_mul_f32_e32 v16, v3, v16
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v16, v11
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v17, v12
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v18, v9
+; GFX9-NEXT:    v_mul_f32_e32 v15, v14, v15
+; GFX9-NEXT:    v_mul_f32_e32 v16, v10, v16
 ; GFX9-NEXT:    v_trunc_f32_e32 v15, v15
-; GFX9-NEXT:    v_ashrrev_i32_e32 v11, 30, v11
-; GFX9-NEXT:    v_mul_f32_e32 v17, v13, v17
-; GFX9-NEXT:    v_mul_f32_e32 v18, v12, v18
+; GFX9-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
+; GFX9-NEXT:    v_mul_f32_e32 v17, v14, v17
+; GFX9-NEXT:    v_mul_f32_e32 v18, v13, v18
 ; GFX9-NEXT:    v_trunc_f32_e32 v16, v16
-; GFX9-NEXT:    v_mad_f32 v19, -v15, v12, v13
-; GFX9-NEXT:    v_ashrrev_i32_e32 v14, 30, v14
-; GFX9-NEXT:    v_or_b32_e32 v11, 1, v11
+; GFX9-NEXT:    v_mad_f32 v19, -v15, v13, v14
+; GFX9-NEXT:    v_or_b32_e32 v1, 1, v1
 ; GFX9-NEXT:    v_trunc_f32_e32 v17, v17
 ; GFX9-NEXT:    v_trunc_f32_e32 v18, v18
-; GFX9-NEXT:    v_mad_f32 v3, -v16, v9, v3
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v19|, |v12|
-; GFX9-NEXT:    v_ashrrev_i32_e32 v2, 30, v2
-; GFX9-NEXT:    v_or_b32_e32 v14, 1, v14
+; GFX9-NEXT:    v_mad_f32 v10, -v16, v11, v10
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v19|, |v13|
+; GFX9-NEXT:    v_ashrrev_i32_e32 v3, 30, v3
+; GFX9-NEXT:    v_or_b32_e32 v4, 1, v4
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v15, v15
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v16, v16
-; GFX9-NEXT:    v_mad_f32 v13, -v17, v10, v13
+; GFX9-NEXT:    v_mad_f32 v14, -v17, v12, v14
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v17, v17
-; GFX9-NEXT:    v_mad_f32 v20, -v18, v4, v12
+; GFX9-NEXT:    v_mad_f32 v20, -v18, v9, v13
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v18, v18
-; GFX9-NEXT:    v_cndmask_b32_e32 v11, 0, v11, vcc
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v9|
-; GFX9-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
+; GFX9-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v10|, |v11|
+; GFX9-NEXT:    v_ashrrev_i32_e32 v2, 30, v2
+; GFX9-NEXT:    v_or_b32_e32 v3, 1, v3
+; GFX9-NEXT:    v_cndmask_b32_e32 v4, 0, v4, vcc
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v14|, |v12|
 ; GFX9-NEXT:    v_or_b32_e32 v2, 1, v2
-; GFX9-NEXT:    v_cndmask_b32_e32 v3, 0, v14, vcc
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v13|, |v10|
-; GFX9-NEXT:    v_or_b32_e32 v1, 1, v1
+; GFX9-NEXT:    v_cndmask_b32_e32 v3, 0, v3, vcc
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v20|, |v9|
 ; GFX9-NEXT:    v_cndmask_b32_e32 v2, 0, v2, vcc
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v20|, |v4|
-; GFX9-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
-; GFX9-NEXT:    v_add_u32_e32 v4, v15, v11
-; GFX9-NEXT:    v_add_u32_sdwa v3, v16, v3 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_add_u32_e32 v2, v17, v2
-; GFX9-NEXT:    v_add_u32_sdwa v1, v18, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v3, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v1, v2, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v1, v3, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_add_u32_e32 v1, v15, v1
+; GFX9-NEXT:    v_add_u32_sdwa v4, v16, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_add_u32_e32 v3, v17, v3
+; GFX9-NEXT:    v_add_u32_sdwa v2, v18, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX9-NEXT:    global_store_dword v[5:6], v1, off
 ; GFX9-NEXT:    global_store_dword v[7:8], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep0 = getelementptr <4 x i8>, ptr addrspace(1) %in0, i32 %tid
   %gep1 = getelementptr <4 x i8>, ptr addrspace(1) %in1, i32 %tid
   %vec0 = load <4 x i8>, ptr addrspace(1) %gep0, align 4