[DAG] getNode() - fold (zext (trunc x)) -> x iff the upper bits are known zero - add SRL support

[RKSimon]

Summary

This is part of the work to address the D155472 regressions, there's a number of issues with generalizing this fold which is why I'm just adding SRL support atm.

The fold encourages the creation of ISD::FSHR nodes which was breaking a lot of the AMDGPUISD::PERM tests, so I've added ISD::FSHR handling to the matchPERM helper

[goldstein.w.n]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

5698

Why does the ISD::AssertZext need one use check?

But while you're at it think ISD::AND also work here right?

[jrbyrnes]

what is the best way to generalize the performOrCombine handling to support ISD::FSHR as well?

We should ideally produce AMDGPUISD::PERM whenever we have an i32 that is byte permutation of two sources (except some special cases) and it's best to do this via DAGCombining due to target-specific reasons. In practice, typically such byte permutations have ISD::OR as the root of the tree (which is why I have begun with that combine), however, other cases exist where the tree does not have ISD::OR as the root (e.g. AMDGPUISD::PERM, ISD::FSHR, etc). It seems to me, the best way to generalize is to extract the common code from ISD::OR, and use it to enable new roots / entry points -- this is an extension of the work we should do anyway. I hacked together an implementation of what I have just described and this patch no longer causes the v_perm miss regressions. @foad @arsenm objections?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

5698

Everything should work here - eventually the getOpcode() filters will go entirely - the problem we're having is a number of AArch64 mul folds create various zext(trunc()) patterns and then expect them to still exist later in the combine.

As usual with LLVM it comes down to yak shaving :)

[RKSimon]

@jrbyrnes I've pulled the AMDGPUISD::PERM matching out of performOrCombine, made it into a 'matchPERM' helper and setup ISD::FSHR to use it - is this what you had in mind?

[jrbyrnes]

Disregard previous version of comment -- I did not see the most recent changes.

[jrbyrnes]

Can you please bring in tests from https://github.com/llvm/llvm-project/pull/66965

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

10775

Can we also fail on vector types

10791

This seems to be broken if Index = 1; BytesProvided; = 2 and ByteShift = 3 -- we will attempt to get the 2nd byte of a 16 bit operand which is out of range (only 0,1 are valid bytes).

Perhaps it was meant to be: Index + (BytesProvided - ByteShift) ? But this still doesnt account for the case where ByteShift > n * BytesProvided.

How do you feel about

case ISD::FSHR: {
  auto ShiftOp = dyn_cast<ConstantSDNode>(Op->getOperand(2));
  if (!ShiftOp || Op.getValueType().isVector())
    return std::nullopt;

  uint64_t BitsProvided = Op.getValueSizeInBits();
  if (BitsProvided % 8 != 0)
    return std::nullopt;

  uint64_t BitShift = ShiftOp->getAPIntValue().urem(BitsProvided);
  if (BitShift % 8)
    return std::nullopt;

  uint64_t ConcatSizeInBytes = BitsProvided / 4;
  uint64_t ByteShift = BitShift / 8;

  auto NewIndex = (Index + ByteShift) % ConcatSizeInBytes;
  auto BytesProvided = BitsProvided / 8;
  auto NextOp = Op.getOperand(NewIndex >= BytesProvided ? 0 : 1);
  NewIndex %= BytesProvided;
  return calculateSrcByte(NextOp, StartingIndex, NewIndex);
}

llvm/test/CodeGen/AMDGPU/permute.ll

127

I can look into this after this patch lands.

[jrbyrnes]

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

10791

Oops -- should have been

return calculateByteProvider(NextOp, NewIndex, Depth + 1, StartingIndex

[RKSimon]

Update with @jrbyrnes suggestions and add early-out so we prefer an existing ISD::FSHR node to a AMDGPUISD::PERM equivalent.

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

5698

@goldstein.w.n Hopefully D159537 will address the last regressions which will allow me to remove the OpOp.getOpcode() filter entirely.

[jrbyrnes]

Thanks -- the AMDGPUISD::PERM side LGTM

[RKSimon]

Thanks - any further comments?

[foad]

@jrbyrnes shouldn't we land the AMDGPU changes separately first? Or would it all be dead code without the rest of this patch?

[RKSimon]

In D159533#4649980, @foad wrote:

@jrbyrnes shouldn't we land the AMDGPU changes separately first? Or would it all be dead code without the rest of this patch?

I can pre-commit the matchPERM helper as a NFC-ish to reduce the diff if that helps?

[RKSimon]

rebase

[jrbyrnes]

In D159533#4649980, @foad wrote:

@jrbyrnes shouldn't we land the AMDGPU changes separately first? Or would it all be dead code without the rest of this patch?

The changes to calculateByteProvider may actually be NFC because sub 32bit ISD::FSHRs get legalized into ISD::OR which are already handled, but enabling ISD::FSHR combine and installing matchPERM there is not.

The changes look okay, but it seems it would be best to land them separately first.

llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

case ISD::ZERO_EXTEND:
     assert(N1.getValueType().bitsLT(VT) && "Invalid zext node, dst < src!");
     if (OpOpcode == ISD::ZERO_EXTEND) // (zext (zext x)) -> (zext x)
       return getNode(ISD::ZERO_EXTEND, DL, VT, N1.getOperand(0));
     if (OpOpcode == ISD::UNDEF)
       // zext(undef) = 0, because the top bits will be zero.
       return getConstant(0, DL, VT);
 
     // Skip unnecessary zext_inreg pattern:
-    // (zext (trunc (assertzext x))) -> (assertzext x)
+    // (zext (trunc x)) -> x iff the upper bits are known zero.
     // TODO: Generalize to MaskedValueIsZero check?
     if (OpOpcode == ISD::TRUNCATE) {
       SDValue OpOp = N1.getOperand(0);
       if (OpOp.getValueType() == VT) {
-        if (OpOp.getOpcode() == ISD::AssertZext && N1->hasOneUse()) {
+        if ((OpOp.getOpcode() == ISD::AssertZext && N1->hasOneUse()) ||
+            OpOp.getOpcode() == ISD::SRL) {
           APInt HiBits = APInt::getBitsSetFrom(VT.getScalarSizeInBits(),
                                                N1.getScalarValueSizeInBits());
           if (MaskedValueIsZero(OpOp, HiBits)) {
             transferDbgValues(N1, OpOp);
             return OpOp;
           }
         }
       }

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

setTargetDAGCombine({ISD::ADD,
                        ISD::SMIN,
                        ISD::SMAX,
                        ISD::UMIN,
                        ISD::UMAX,
                        ISD::SETCC,
                        ISD::AND,
                        ISD::OR,
                        ISD::XOR,
+                       ISD::FSHR,
                        ISD::SINT_TO_FP,
                        ISD::UINT_TO_FP,
                        ISD::FCANONICALIZE,
                        ISD::SCALAR_TO_VECTOR,
                        ISD::ZERO_EXTEND,
                        ISD::SIGN_EXTEND_INREG,
                        ISD::EXTRACT_VECTOR_ELT,
                        ISD::INSERT_VECTOR_ELT,
if ((IndexMask & BitMask) != IndexMask) {
       if (IndexMask & BitMask)
         return std::nullopt;
       return ByteProvider<SDValue>::getConstantZero();
     }
 
     return calculateSrcByte(Op->getOperand(0), StartingIndex, Index);
   }
 
+  case ISD::FSHR: {
+    // fshr(X,Y,Z): (X << (BW - (Z % BW))) | (Y >> (Z % BW))
+    auto ShiftOp = dyn_cast<ConstantSDNode>(Op->getOperand(2));
+    if (!ShiftOp)
+      return std::nullopt;
+
+    uint64_t BitsProvided = Op.getScalarValueSizeInBits();
+    if (BitsProvided % 8 != 0)
+      return std::nullopt;
+
+    uint64_t BitShift = ShiftOp->getAPIntValue().urem(BitsProvided);
+    if (BitShift % 8)
+      return std::nullopt;
+
+    uint64_t BytesProvided = BitsProvided / 8;
+    uint64_t ByteShift = BitShift / 8;
+    return BytesProvided - ByteShift > Index
+               ? calculateByteProvider(Op.getOperand(1), Index + ByteShift,
+                                       Depth + 1, StartingIndex)
+               : calculateByteProvider(Op.getOperand(0),
+                                       Index - (BytesProvided - ByteShift),
+                                       Depth + 1, StartingIndex);
+  }
+
   case ISD::SRA:
   case ISD::SRL: {
     auto ShiftOp = dyn_cast<ConstantSDNode>(Op->getOperand(1));
     if (!ShiftOp)
       return std::nullopt;
 
     uint64_t BitShift = ShiftOp->getZExtValue();
     if (BitShift % 8)
auto OtherOpIs16Bit = TempOtherOp.getValueSizeInBits() == 16 ||
                         isExtendedFrom16Bits(TempOtherOp);
   if (!OtherOpIs16Bit)
     return true;
 
   // Do we cleanly address both
   return !addresses16Bits(Low16) || !addresses16Bits(Hi16);
 }
 
+static SDValue matchPERM(SDNode *N, TargetLowering::DAGCombinerInfo &DCI) {
+  SelectionDAG &DAG = DCI.DAG;
+  SDValue LHS = N->getOperand(0);
+  SDValue RHS = N->getOperand(1);
+  EVT VT = N->getValueType(0);
+
+  if (VT != MVT::i32)
+    return SDValue();
+
+  // VT is known to be MVT::i32, so we need to provide 4 bytes.
+  SmallVector<ByteProvider<SDValue>, 8> PermNodes;
+  for (int i = 0; i < 4; i++) {
+    // Find the ByteProvider that provides the ith byte of the result of OR
+    std::optional<ByteProvider<SDValue>> P =
+        calculateByteProvider(SDValue(N, 0), i, 0, /*StartingIndex = */ i);
+    // TODO support constantZero
+    if (!P || P->isConstantZero())
+      return SDValue();
+
+    PermNodes.push_back(*P);
+  }
+  if (PermNodes.size() != 4)
+    return SDValue();
+
+  int FirstSrc = 0;
+  std::optional<int> SecondSrc;
+  uint64_t PermMask = 0x00000000;
+  for (size_t i = 0; i < PermNodes.size(); i++) {
+    auto PermOp = PermNodes[i];
+    // Since the mask is applied to Src1:Src2, Src1 bytes must be offset
+    // by sizeof(Src2) = 4
+    int SrcByteAdjust = 4;
+
+    if (!PermOp.hasSameSrc(PermNodes[FirstSrc])) {
+      if (SecondSrc.has_value())
+        if (!PermOp.hasSameSrc(PermNodes[*SecondSrc]))
+          return SDValue();
+
+      // Set the index of the second distinct Src node
+      SecondSrc = i;
+      assert(!(PermNodes[*SecondSrc].Src->getValueSizeInBits() % 8));
+      SrcByteAdjust = 0;
+    }
+    assert(PermOp.SrcOffset + SrcByteAdjust < 8);
+    assert(!DAG.getDataLayout().isBigEndian());
+    PermMask |= (PermOp.SrcOffset + SrcByteAdjust) << (i * 8);
+  }
+
+  SDValue Op = *PermNodes[FirstSrc].Src;
+  SDValue OtherOp = SecondSrc.has_value() ? *PermNodes[*SecondSrc].Src
+                                          : *PermNodes[FirstSrc].Src;
+
+  // Check that we are not just extracting the bytes in order from an op
+  if (Op == OtherOp && Op.getValueSizeInBits() == 32) {
+    int Low16 = PermMask & 0xffff;
+    int Hi16 = (PermMask & 0xffff0000) >> 16;
+
+    bool WellFormedLow = (Low16 == 0x0504) || (Low16 == 0x0100);
+    bool WellFormedHi = (Hi16 == 0x0706) || (Hi16 == 0x0302);
+
+    // The perm op would really just produce Op. So combine into Op
+    if (WellFormedLow && WellFormedHi)
+      return DAG.getBitcast(MVT::getIntegerVT(32), Op);
+  }
+
+  if (hasNon16BitAccesses(PermMask, Op, OtherOp)) {
+    SDLoc DL(N);
+    assert(Op.getValueType().isByteSized() &&
+           OtherOp.getValueType().isByteSized());
+
+    // If the ultimate src is less than 32 bits, then we will only be
+    // using bytes 0: Op.getValueSizeInBytes() - 1 in the or.
+    // CalculateByteProvider would not have returned Op as source if we
+    // used a byte that is outside its ValueType. Thus, we are free to
+    // ANY_EXTEND as the extended bits are dont-cares.
+    Op = DAG.getBitcastedAnyExtOrTrunc(Op, DL, MVT::i32);
+    OtherOp = DAG.getBitcastedAnyExtOrTrunc(OtherOp, DL, MVT::i32);
+
+    return DAG.getNode(AMDGPUISD::PERM, DL, MVT::i32, Op, OtherOp,
+                       DAG.getConstant(PermMask, DL, MVT::i32));
+  }
+
+  return SDValue();
+}
+
 SDValue SITargetLowering::performOrCombine(SDNode *N,
                                            DAGCombinerInfo &DCI) const {
   SelectionDAG &DAG = DCI.DAG;
   SDValue LHS = N->getOperand(0);
   SDValue RHS = N->getOperand(1);
 
   EVT VT = N->getValueType(0);
   if (VT == MVT::i1) {
if (LHSMask != ~0u && RHSMask != ~0u) {
         SDLoc DL(N);
 
         return DAG.getNode(AMDGPUISD::PERM, DL, MVT::i32,
                            LHS.getOperand(0), RHS.getOperand(0),
                            DAG.getConstant(Sel, DL, MVT::i32));
       }
     }
     if (LHSMask == ~0u || RHSMask == ~0u) {
-      SmallVector<ByteProvider<SDValue>, 8> PermNodes;
-
-      // VT is known to be MVT::i32, so we need to provide 4 bytes.
-      assert(VT == MVT::i32);
-      for (int i = 0; i < 4; i++) {
-        // Find the ByteProvider that provides the ith byte of the result of OR
-        std::optional<ByteProvider<SDValue>> P =
-            calculateByteProvider(SDValue(N, 0), i, 0, /*StartingIndex = */ i);
-        // TODO support constantZero
-        if (!P || P->isConstantZero())
-          return SDValue();
-
-        PermNodes.push_back(*P);
-      }
-      if (PermNodes.size() != 4)
-        return SDValue();
-
-      int FirstSrc = 0;
-      std::optional<int> SecondSrc;
-      uint64_t PermMask = 0x00000000;
-      for (size_t i = 0; i < PermNodes.size(); i++) {
-        auto PermOp = PermNodes[i];
-        // Since the mask is applied to Src1:Src2, Src1 bytes must be offset
-        // by sizeof(Src2) = 4
-        int SrcByteAdjust = 4;
-
-        if (!PermOp.hasSameSrc(PermNodes[FirstSrc])) {
-          if (SecondSrc.has_value())
-            if (!PermOp.hasSameSrc(PermNodes[*SecondSrc]))
-              return SDValue();
-
-          // Set the index of the second distinct Src node
-          SecondSrc = i;
-          assert(!(PermNodes[*SecondSrc].Src->getValueSizeInBits() % 8));
-          SrcByteAdjust = 0;
-        }
-        assert(PermOp.SrcOffset + SrcByteAdjust < 8);
-        assert(!DAG.getDataLayout().isBigEndian());
-        PermMask |= (PermOp.SrcOffset + SrcByteAdjust) << (i * 8);
-      }
-
-      SDValue Op = *PermNodes[FirstSrc].Src;
-      SDValue OtherOp = SecondSrc.has_value() ? *PermNodes[*SecondSrc].Src
-                                              : *PermNodes[FirstSrc].Src;
-
-      // Check that we are not just extracting the bytes in order from an op
-      if (Op == OtherOp && Op.getValueSizeInBits() == 32) {
-        int Low16 = PermMask & 0xffff;
-        int Hi16 = (PermMask & 0xffff0000) >> 16;
-
-        bool WellFormedLow = (Low16 == 0x0504) || (Low16 == 0x0100);
-        bool WellFormedHi = (Hi16 == 0x0706) || (Hi16 == 0x0302);
-
-        // The perm op would really just produce Op. So combine into Op
-        if (WellFormedLow && WellFormedHi)
-          return DAG.getBitcast(MVT::getIntegerVT(32), Op);
-      }
-
-      if (hasNon16BitAccesses(PermMask, Op, OtherOp)) {
-        SDLoc DL(N);
-        assert(Op.getValueType().isByteSized() &&
-               OtherOp.getValueType().isByteSized());
-
-        // If the ultimate src is less than 32 bits, then we will only be
-        // using bytes 0: Op.getValueSizeInBytes() - 1 in the or.
-        // CalculateByteProvider would not have returned Op as source if we
-        // used a byte that is outside its ValueType. Thus, we are free to
-        // ANY_EXTEND as the extended bits are dont-cares.
-        Op = DAG.getBitcastedAnyExtOrTrunc(Op, DL, MVT::i32);
-        OtherOp = DAG.getBitcastedAnyExtOrTrunc(OtherOp, DL, MVT::i32);
-
-        return DAG.getNode(AMDGPUISD::PERM, DL, MVT::i32, Op, OtherOp,
-                           DAG.getConstant(PermMask, DL, MVT::i32));
-      }
+      if (SDValue Perm = matchPERM(N, DCI))
+        return Perm;
     }
   }
 
   if (VT != MVT::i64 || DCI.isBeforeLegalizeOps())
     return SDValue();
 
   // TODO: This could be a generic combine with a predicate for extracting the
   // high half of an integer being free.
SDValue SITargetLowering::PerformDAGCombine(SDNode *N,
   case AMDGPUISD::FMAX_LEGACY:
     return performMinMaxCombine(N, DCI);
   case ISD::FMA:
     return performFMACombine(N, DCI);
   case ISD::AND:
     return performAndCombine(N, DCI);
   case ISD::OR:
     return performOrCombine(N, DCI);
+  case ISD::FSHR: {
+    const SIInstrInfo *TII = getSubtarget()->getInstrInfo();
+    if (N->getValueType(0) == MVT::i32 && N->isDivergent() &&
+        TII->pseudoToMCOpcode(AMDGPU::V_PERM_B32_e64) != -1) {
+      return matchPERM(N, DCI);
+    }
+    break;
+  }
   case ISD::XOR:
     return performXorCombine(N, DCI);
   case ISD::ZERO_EXTEND:
     return performZeroExtendCombine(N, DCI);
   case ISD::SIGN_EXTEND_INREG:
     return performSignExtendInRegCombine(N , DCI);
   case AMDGPUISD::FP_CLASS:
     return performClassCombine(N, DCI);

llvm/test/CodeGen/AMDGPU/ctpop16.ll

 ; SI-NEXT:    s_branch .LBB14_2
 ;
 ; VI-LABEL: ctpop_i16_in_br:
 ; VI:       ; %bb.0: ; %entry
 ; VI-NEXT:    s_load_dword s4, s[0:1], 0x34
 ; VI-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_lshr_b32 s5, s4, 16
-; VI-NEXT:    v_cmp_ne_u16_e64 s[6:7], s5, 0
-; VI-NEXT:    s_and_b64 vcc, exec, s[6:7]
-; VI-NEXT:    s_cbranch_vccz .LBB14_4
+; VI-NEXT:    s_cmp_lg_u32 s5, 0
+; VI-NEXT:    s_cbranch_scc0 .LBB14_4
 ; VI-NEXT:  ; %bb.1: ; %else
 ; VI-NEXT:    s_mov_b32 s11, 0xf000
 ; VI-NEXT:    s_mov_b32 s10, -1
 ; VI-NEXT:    s_mov_b32 s8, s2
 ; VI-NEXT:    s_mov_b32 s9, s3
 ; VI-NEXT:    buffer_load_ushort v0, off, s[8:11], 0 offset:2
 ; VI-NEXT:    s_mov_b64 s[2:3], 0
 ; VI-NEXT:    s_cbranch_execnz .LBB14_3

llvm/test/CodeGen/AMDGPU/permute.ll

 }
 
 define amdgpu_kernel void @lsh8_or_lsr24(ptr addrspace(1) nocapture %arg, i32 %arg1) {
 ; GCN-LABEL: lsh8_or_lsr24:
 ; GCN:       ; %bb.0: ; %bb
 ; GCN-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GCN-NEXT:    s_load_dword s0, s[0:1], 0x2c
 ; GCN-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
+; GCN-NEXT:    v_mov_b32_e32 v3, 0x2010007
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    v_mov_b32_e32 v1, s3
 ; GCN-NEXT:    v_add_u32_e32 v0, vcc, s2, v0
 ; GCN-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GCN-NEXT:    flat_load_dword v2, v[0:1]
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
-; GCN-NEXT:    v_alignbit_b32 v2, v2, s0, 24
+; GCN-NEXT:    v_perm_b32 v2, s0, v2, v3
 ; GCN-NEXT:    flat_store_dword v[0:1], v2
 ; GCN-NEXT:    s_endpgm
 bb:
   %id = tail call i32 @llvm.amdgcn.workitem.id.x()
   %gep = getelementptr i32, ptr addrspace(1) %arg, i32 %id
   %tmp = load i32, ptr addrspace(1) %gep, align 4
   %tmp2 = shl i32 %tmp, 8
   %tmp3 = lshr i32 %arg1, 24

llvm/test/CodeGen/AMDGPU/permute_i8.ll

 }
 
 define hidden void @shuffle5341ud2(ptr addrspace(1) %in0, ptr addrspace(1) %out0) {
 ; GFX10-LABEL: shuffle5341ud2:
 ; GFX10:       ; %bb.0:
 ; GFX10-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX10-NEXT:    global_load_dword v0, v[0:1], off
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_perm_b32 v0, v0, v0, 0x5040706
+; GFX10-NEXT:    v_alignbit_b32 v0, v0, v0, 16
 ; GFX10-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: shuffle5341ud2:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v0, v[0:1], off
-; GFX9-NEXT:    s_mov_b32 s4, 0x5040706
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_perm_b32 v0, v0, v0, s4
+; GFX9-NEXT:    v_alignbit_b32 v0, v0, v0, 16
 ; GFX9-NEXT:    global_store_dword v[2:3], v0, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %vec0 = load <4 x i8>, ptr addrspace(1) %in0, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> undef, <4 x i32> <i32 5, i32 3, i32 4, i32 1>
   store <4 x i8> %shuffle0_0, ptr addrspace(1) %out0, align 4
   ret void
 }

 ; GFX10-NEXT:    global_load_dword v10, v[2:3], off
 ; GFX10-NEXT:    v_mov_b32_e32 v0, 16
 ; GFX10-NEXT:    v_mov_b32_e32 v1, 0xff
 ; GFX10-NEXT:    v_lshlrev_b16 v2, 8, v4
 ; GFX10-NEXT:    s_waitcnt vmcnt(1)
 ; GFX10-NEXT:    v_lshrrev_b32_sdwa v0, v0, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX10-NEXT:    s_waitcnt vmcnt(0)
 ; GFX10-NEXT:    v_and_b32_sdwa v1, v10, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10-NEXT:    v_or_b32_sdwa v0, v9, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v9, v0 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX10-NEXT:    v_or_b32_e32 v1, v1, v2
-; GFX10-NEXT:    v_lshlrev_b32_e32 v2, 16, v0
-; GFX10-NEXT:    v_alignbit_b32 v0, v0, v10, 16
-; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-NEXT:    global_store_dword v[5:6], v1, off
-; GFX10-NEXT:    global_store_dword v[7:8], v0, off
+; GFX10-NEXT:    v_or_b32_sdwa v0, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-NEXT:    v_perm_b32 v1, v10, v9, 0x2000706
+; GFX10-NEXT:    global_store_dword v[5:6], v0, off
+; GFX10-NEXT:    global_store_dword v[7:8], v1, off
 ; GFX10-NEXT:    s_setpc_b64 s[30:31]
 ;
 ; GFX9-LABEL: ive_store_div:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX9-NEXT:    v_and_b32_e32 v9, 0x3ff, v31
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v9, 2, v9
 ; GFX9-NEXT:    v_add_co_u32_e32 v0, vcc, v0, v9
 ; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX9-NEXT:    v_add_co_u32_e32 v2, vcc, v2, v9
 ; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
 ; GFX9-NEXT:    global_load_dword v9, v[0:1], off
 ; GFX9-NEXT:    global_load_dword v10, v[2:3], off
 ; GFX9-NEXT:    s_movk_i32 s4, 0xff
 ; GFX9-NEXT:    v_lshlrev_b16_e32 v0, 8, v4
+; GFX9-NEXT:    s_mov_b32 s5, 0x2000706
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 16, v9
-; GFX9-NEXT:    v_lshlrev_b16_e32 v1, 8, v1
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_and_b32_sdwa v2, v10, s4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v1, v9, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e32 v1, 8, v1
 ; GFX9-NEXT:    v_or_b32_e32 v0, v2, v0
-; GFX9-NEXT:    v_alignbit_b32 v2, v1, v10, 16
-; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; GFX9-NEXT:    v_or_b32_sdwa v1, v9, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_perm_b32 v3, v10, v9, s5
 ; GFX9-NEXT:    global_store_dword v[5:6], v0, off
-; GFX9-NEXT:    global_store_dword v[7:8], v2, off
+; GFX9-NEXT:    global_store_dword v[7:8], v3, off
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    s_setpc_b64 s[30:31]
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %gep0 = getelementptr <4 x i8>, ptr addrspace(1) %in0, i32 %tid
   %gep1 = getelementptr <4 x i8>, ptr addrspace(1) %in1, i32 %tid
   %vec0 = load <4 x i8>, ptr addrspace(1) %gep0, align 4
   %vec1 = load <4 x i8>, ptr addrspace(1) %gep1, align 4
   %shuffle0_0 = shufflevector <4 x i8> %vec0, <4 x i8> %vec1, <4 x i32> <i32 6, i32 7, i32 0, i32 2>