[TargetLowering] Add SimplifyMultipleUseDemandedBits

[RKSimon]

Summary

This patch introduces the DAG version of SimplifyMultipleUseDemandedBits, which attempts to peek through ops (mainly and/or/xor so far) that don't contribute to the demandedbits/elts of a node - which means we can do this even in cases where we have multiple uses of an op, which normally requires us to demanded all bits/elts. The intention is to remove a similar instruction - SelectionDAG::GetDemandedBits - once SimplifyMultipleUseDemandedBits has matured.

The InstCombine version of SimplifyMultipleUseDemandedBits can constant fold which I haven't added here yet, and so far I've only wired this up to some basic binops (and/or/xor/add/sub/mul) to demonstrate its use.

We do see a couple of regressions that need to be addressed:

• AMDGPU unsigned dot product codegen retains an AND mask that it previously removed (but otherwise the dotproduct codegen is a lot better)
• X86/AVX2 has poor handling of vector ANY_EXTEND/ANY_EXTEND_VECTOR_INREG - it prematurely gets converted to ZERO_EXTEND_VECTOR_INREG

Both these cases look like they can be fixed up afterwards but I'd like to get code owner's feedback first.

[nikic]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

664

Why are there explicit SimplifyMultipleUseDemandedBits() calls for individual opcodes, rather than a general call in the !hasOneUse branch here? This is what we do in InstCombine.

[RKSimon]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

664

Because we need to regenerate a single op - afaict we'd have to refactor the entire SimplifyDemandedBits / TargetLoweringOpt system to enable us to replace individual args of a single node.

[craig.topper]

I'm ok with the X86 changes if we can fix them after.

[RKSimon]

@arsenm Any comments?

[RKSimon]

Rebase

[RKSimon]

ping?

[RKSimon]

ping?

[RKSimon]

rebase

[RKSimon]

@arsenm @nhaehnle Please can you take a look at the AMDGPU changes? @craig.topper is happy with the X86 changes (and temp regression) and I'm keen to get this + a couple of followups in before the release branch.

[RKSimon]

rebase

[RKSimon]

ping

[nhaehnle]

The AMDGPU changes seem fine to me overall.

[RKSimon]

@craig.topper Are OK with me committing this? I can then work on the remaining regressions

[craig.topper]

In D63281#1596051, @RKSimon wrote:

@craig.topper Are OK with me committing this? I can then work on the remaining regressions

Yes

include/llvm/CodeGen/TargetLowering.h

bool SimplifyDemandedBits(SDValue Op, const APInt &DemandedBits,
                             unsigned Depth = 0,
                             bool AssumeSingleUse = false) const;
 
   /// Helper wrapper around SimplifyDemandedBits.
   /// Adds Op back to the worklist upon success.
   bool SimplifyDemandedBits(SDValue Op, const APInt &DemandedMask,
                             DAGCombinerInfo &DCI) const;
 
+  /// More limited version of SimplifyDemandedBits that can be used to "look
+  /// through" ops that don't contribute to the DemandedBits/DemandedElts -
+  /// bitwise ops etc.
+  SDValue SimplifyMultipleUseDemandedBits(SDValue Op, const APInt &DemandedBits,
+                                          const APInt &DemandedElts,
+                                          SelectionDAG &DAG,
+                                          unsigned Depth) const;
+
   /// Look at Vector Op. At this point, we know that only the DemandedElts
   /// elements of the result of Op are ever used downstream.  If we can use
   /// this information to simplify Op, create a new simplified DAG node and
   /// return true, storing the original and new nodes in TLO.
   /// Otherwise, analyze the expression and return a mask of KnownUndef and
   /// KnownZero elements for the expression (used to simplify the caller).
   /// The KnownUndef/Zero elements may only be accurate for those bits
   /// in the DemandedMask.
public:
   /// be accurate for those bits in the Demanded masks.
   virtual bool SimplifyDemandedBitsForTargetNode(SDValue Op,
                                                  const APInt &DemandedBits,
                                                  const APInt &DemandedElts,
                                                  KnownBits &Known,
                                                  TargetLoweringOpt &TLO,
                                                  unsigned Depth = 0) const;
 
+  /// More limited version of SimplifyDemandedBits that can be used to "look
+  /// through" ops that don't contribute to the DemandedBits/DemandedElts -
+  /// bitwise ops etc.
+  SDValue SimplifyMultipleUseDemandedBitsForTargetNode(
+      SDValue Op, const APInt &DemandedBits, const APInt &DemandedElts,
+      SelectionDAG &DAG, unsigned Depth) const;
+
   /// This method returns the constant pool value that will be loaded by LD.
   /// NOTE: You must check for implicit extensions of the constant by LD.
   virtual const Constant *getTargetConstantFromLoad(LoadSDNode *LD) const;
 
   /// If \p SNaN is false, \returns true if \p Op is known to never be any
   /// NaN. If \p sNaN is true, returns if \p Op is known to never be a signaling
   /// NaN.
   virtual bool isKnownNeverNaNForTargetNode(SDValue Op,

lib/CodeGen/SelectionDAG/TargetLowering.cpp

bool TargetLowering::SimplifyDemandedBits(SDValue Op, const APInt &DemandedBits,
   EVT VT = Op.getValueType();
   APInt DemandedElts = VT.isVector()
                            ? APInt::getAllOnesValue(VT.getVectorNumElements())
                            : APInt(1, 1);
   return SimplifyDemandedBits(Op, DemandedBits, DemandedElts, Known, TLO, Depth,
                               AssumeSingleUse);
 }
 
+// TODO: Can we merge SelectionDAG::GetDemandedBits into this?
+// TODO: Under what circumstances can we create nodes? BITCAST? Constant?
+SDValue TargetLowering::SimplifyMultipleUseDemandedBits(
+    SDValue Op, const APInt &DemandedBits, const APInt &DemandedElts,
+    SelectionDAG &DAG, unsigned Depth) const {
+  KnownBits LHSKnown, RHSKnown;
+  switch (Op.getOpcode()) {
+  case ISD::AND: {
+    LHSKnown = DAG.computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
+    RHSKnown = DAG.computeKnownBits(Op.getOperand(1), DemandedElts, Depth + 1);
+
+    // If all of the demanded bits are known 1 on one side, return the other.
+    // These bits cannot contribute to the result of the 'and' in this
+    // context.
+    if (DemandedBits.isSubsetOf(LHSKnown.Zero | RHSKnown.One))
+      return Op.getOperand(0);
+    if (DemandedBits.isSubsetOf(RHSKnown.Zero | LHSKnown.One))
+      return Op.getOperand(1);
+    break;
+  }
+  case ISD::OR: {
+    LHSKnown = DAG.computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
+    RHSKnown = DAG.computeKnownBits(Op.getOperand(1), DemandedElts, Depth + 1);
+
+    // If all of the demanded bits are known zero on one side, return the
+    // other.  These bits cannot contribute to the result of the 'or' in this
+    // context.
+    if (DemandedBits.isSubsetOf(LHSKnown.One | RHSKnown.Zero))
+      return Op.getOperand(0);
+    if (DemandedBits.isSubsetOf(RHSKnown.One | LHSKnown.Zero))
+      return Op.getOperand(1);
+    break;
+  }
+  case ISD::XOR: {
+    LHSKnown = DAG.computeKnownBits(Op.getOperand(0), DemandedElts, Depth + 1);
+    RHSKnown = DAG.computeKnownBits(Op.getOperand(1), DemandedElts, Depth + 1);
+
+    // If all of the demanded bits are known zero on one side, return the
+    // other.
+    if (DemandedBits.isSubsetOf(RHSKnown.Zero))
+      return Op.getOperand(0);
+    if (DemandedBits.isSubsetOf(LHSKnown.Zero))
+      return Op.getOperand(1);
+    break;
+  }
+  default:
+    if (Op.getOpcode() >= ISD::BUILTIN_OP_END)
+      if (SDValue V = SimplifyMultipleUseDemandedBitsForTargetNode(
+              Op, DemandedBits, DemandedElts, DAG, Depth))
+        return V;
+    break;
+  }
+  return SDValue();
+}
+
 /// Look at Op. At this point, we know that only the OriginalDemandedBits of the
 /// result of Op are ever used downstream. If we can use this information to
 /// simplify Op, create a new simplified DAG node and return true, returning the
 /// original and new nodes in Old and New. Otherwise, analyze the expression and
 /// return a mask of Known bits for the expression (used to simplify the
 /// caller).  The Known bits may only be accurate for those bits in the
 /// OriginalDemandedBits and OriginalDemandedElts.
 bool TargetLowering::SimplifyDemandedBits(
if (Op.getOpcode() == ISD::Constant) {
     Known.One = cast<ConstantSDNode>(Op)->getAPIntValue();
     Known.Zero = ~Known.One;
     return false;
   }
 
   // Other users may use these bits.
   EVT VT = Op.getValueType();
   if (!Op.getNode()->hasOneUse() && !AssumeSingleUse) {
     if (Depth != 0) {
       // If not at the root, Just compute the Known bits to
       // simplify things downstream.
       Known = TLO.DAG.computeKnownBits(Op, DemandedElts, Depth);
       return false;
     }
     // If this is the root being simplified, allow it to have multiple uses,
     // just set the DemandedBits/Elts to all bits.
     DemandedBits = APInt::getAllOnesValue(BitWidth);
if (SimplifyDemandedBits(Op1, DemandedBits, DemandedElts, Known, TLO,
                              Depth + 1))
       return true;
     assert(!Known.hasConflict() && "Bits known to be one AND zero?");
     if (SimplifyDemandedBits(Op0, ~Known.Zero & DemandedBits, DemandedElts,
                              Known2, TLO, Depth + 1))
       return true;
     assert(!Known2.hasConflict() && "Bits known to be one AND zero?");
 
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    if (!DemandedBits.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+      SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+          Op0, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+      SDValue DemandedOp1 = SimplifyMultipleUseDemandedBits(
+          Op1, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+      if (DemandedOp0 || DemandedOp1) {
+        Op0 = DemandedOp0 ? DemandedOp0 : Op0;
+        Op1 = DemandedOp1 ? DemandedOp1 : Op1;
+        SDValue NewOp = TLO.DAG.getNode(Op.getOpcode(), dl, VT, Op0, Op1);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
+
     // If all of the demanded bits are known one on one side, return the other.
     // These bits cannot contribute to the result of the 'and'.
     if (DemandedBits.isSubsetOf(Known2.Zero | Known.One))
       return TLO.CombineTo(Op, Op0);
     if (DemandedBits.isSubsetOf(Known.Zero | Known2.One))
       return TLO.CombineTo(Op, Op1);
     // If all of the demanded bits in the inputs are known zeros, return zero.
     if (DemandedBits.isSubsetOf(Known.Zero | Known2.Zero))
if (SimplifyDemandedBits(Op1, DemandedBits, DemandedElts, Known, TLO,
                              Depth + 1))
       return true;
     assert(!Known.hasConflict() && "Bits known to be one AND zero?");
     if (SimplifyDemandedBits(Op0, ~Known.One & DemandedBits, DemandedElts,
                              Known2, TLO, Depth + 1))
       return true;
     assert(!Known2.hasConflict() && "Bits known to be one AND zero?");
 
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    if (!DemandedBits.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+      SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+          Op0, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+      SDValue DemandedOp1 = SimplifyMultipleUseDemandedBits(
+          Op1, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+      if (DemandedOp0 || DemandedOp1) {
+        Op0 = DemandedOp0 ? DemandedOp0 : Op0;
+        Op1 = DemandedOp1 ? DemandedOp1 : Op1;
+        SDValue NewOp = TLO.DAG.getNode(Op.getOpcode(), dl, VT, Op0, Op1);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
+
     // If all of the demanded bits are known zero on one side, return the other.
     // These bits cannot contribute to the result of the 'or'.
     if (DemandedBits.isSubsetOf(Known2.One | Known.Zero))
       return TLO.CombineTo(Op, Op0);
     if (DemandedBits.isSubsetOf(Known.One | Known2.Zero))
       return TLO.CombineTo(Op, Op1);
     // If the RHS is a constant, see if we can simplify it.
     if (ShrinkDemandedConstant(Op, DemandedBits, TLO))
if (SimplifyDemandedBits(Op1, DemandedBits, DemandedElts, Known, TLO,
                              Depth + 1))
       return true;
     assert(!Known.hasConflict() && "Bits known to be one AND zero?");
     if (SimplifyDemandedBits(Op0, DemandedBits, DemandedElts, Known2, TLO,
                              Depth + 1))
       return true;
     assert(!Known2.hasConflict() && "Bits known to be one AND zero?");
 
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    if (!DemandedBits.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
+      SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+          Op0, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+      SDValue DemandedOp1 = SimplifyMultipleUseDemandedBits(
+          Op1, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+      if (DemandedOp0 || DemandedOp1) {
+        Op0 = DemandedOp0 ? DemandedOp0 : Op0;
+        Op1 = DemandedOp1 ? DemandedOp1 : Op1;
+        SDValue NewOp = TLO.DAG.getNode(Op.getOpcode(), dl, VT, Op0, Op1);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
+
     // If all of the demanded bits are known zero on one side, return the other.
     // These bits cannot contribute to the result of the 'xor'.
     if (DemandedBits.isSubsetOf(Known.Zero))
       return TLO.CombineTo(Op, Op0);
     if (DemandedBits.isSubsetOf(Known2.Zero))
       return TLO.CombineTo(Op, Op1);
     // If the operation can be done in a smaller type, do so.
     if (ShrinkDemandedOp(Op, BitWidth, DemandedBits, TLO))
case ISD::BITCAST: {
     break;
   }
   case ISD::ADD:
   case ISD::MUL:
   case ISD::SUB: {
     // Add, Sub, and Mul don't demand any bits in positions beyond that
     // of the highest bit demanded of them.
     SDValue Op0 = Op.getOperand(0), Op1 = Op.getOperand(1);
+    SDNodeFlags Flags = Op.getNode()->getFlags();
     unsigned DemandedBitsLZ = DemandedBits.countLeadingZeros();
     APInt LoMask = APInt::getLowBitsSet(BitWidth, BitWidth - DemandedBitsLZ);
     if (SimplifyDemandedBits(Op0, LoMask, DemandedElts, Known2, TLO,
                              Depth + 1) ||
         SimplifyDemandedBits(Op1, LoMask, DemandedElts, Known2, TLO,
                              Depth + 1) ||
         // See if the operation should be performed at a smaller bit width.
         ShrinkDemandedOp(Op, BitWidth, DemandedBits, TLO)) {
-      SDNodeFlags Flags = Op.getNode()->getFlags();
       if (Flags.hasNoSignedWrap() || Flags.hasNoUnsignedWrap()) {
         // Disable the nsw and nuw flags. We can no longer guarantee that we
         // won't wrap after simplification.
         Flags.setNoSignedWrap(false);
         Flags.setNoUnsignedWrap(false);
         SDValue NewOp =
             TLO.DAG.getNode(Op.getOpcode(), dl, VT, Op0, Op1, Flags);
         return TLO.CombineTo(Op, NewOp);
       }
       return true;
     }
 
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    // TODO: Add SDNodeFlags handling.
+    if (!Flags.hasNoSignedWrap() && !Flags.hasNoUnsignedWrap() &&
+        !LoMask.isAllOnesValue()) {
+      SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
+          Op0, LoMask, DemandedElts, TLO.DAG, Depth + 1);
+      SDValue DemandedOp1 = SimplifyMultipleUseDemandedBits(
+          Op1, LoMask, DemandedElts, TLO.DAG, Depth + 1);
+      if (DemandedOp0 || DemandedOp1) {
+        Op0 = DemandedOp0 ? DemandedOp0 : Op0;
+        Op1 = DemandedOp1 ? DemandedOp1 : Op1;
+        SDValue NewOp =
+            TLO.DAG.getNode(Op.getOpcode(), dl, VT, Op0, Op1, Flags);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
+
     // If we have a constant operand, we may be able to turn it into -1 if we
     // do not demand the high bits. This can make the constant smaller to
     // encode, allow more general folding, or match specialized instruction
     // patterns (eg, 'blsr' on x86). Don't bother changing 1 to -1 because that
     // is probably not useful (and could be detrimental).
     ConstantSDNode *C = isConstOrConstSplat(Op1);
     APInt HighMask = APInt::getHighBitsSet(BitWidth, DemandedBitsLZ);
     if (C && !C->isAllOnesValue() && !C->isOne() &&
assert((Op.getOpcode() >= ISD::BUILTIN_OP_END ||
           Op.getOpcode() == ISD::INTRINSIC_W_CHAIN ||
           Op.getOpcode() == ISD::INTRINSIC_VOID) &&
          "Should use SimplifyDemandedBits if you don't know whether Op"
          " is a target node!");
   computeKnownBitsForTargetNode(Op, Known, DemandedElts, TLO.DAG, Depth);
   return false;
 }
 
+SDValue TargetLowering::SimplifyMultipleUseDemandedBitsForTargetNode(
+    SDValue Op, const APInt &DemandedBits, const APInt &DemandedElts,
+    SelectionDAG &DAG, unsigned Depth) const {
+  assert(
+      (Op.getOpcode() >= ISD::BUILTIN_OP_END ||
+       Op.getOpcode() == ISD::INTRINSIC_WO_CHAIN ||
+       Op.getOpcode() == ISD::INTRINSIC_W_CHAIN ||
+       Op.getOpcode() == ISD::INTRINSIC_VOID) &&
+      "Should use SimplifyMultipleUseDemandedBits if you don't know whether Op"
+      " is a target node!");
+  return SDValue();
+}
+
 const Constant *TargetLowering::getTargetConstantFromLoad(LoadSDNode*) const {
   return nullptr;
 }
 
 bool TargetLowering::isKnownNeverNaNForTargetNode(SDValue Op,
                                                   const SelectionDAG &DAG,
                                                   bool SNaN,
                                                   unsigned Depth) const {

test/CodeGen/AArch64/bitfield-insert.ll

; CHECK-NEXT:    ret
 
   ret void
 }
 
 ; Tests when all the bits from one operand are not useful
 define i32 @test_nouseful_bits(i8 %a, i32 %b) {
 ; CHECK-LABEL: test_nouseful_bits:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, w0
-; CHECK-NEXT:    bfi w8, w8, #8, #24
+; CHECK-NEXT:    orr w8, w0, w8, lsl #8
 ; CHECK-NEXT:    mov w9, w0
 ; CHECK-NEXT:    bfi w9, w8, #8, #24
 ; CHECK-NEXT:    bfi w0, w9, #8, #24
 ; CHECK-NEXT:    lsl w0, w0, #8
 ; CHECK-NEXT:    ret
   %conv = zext i8 %a to i32     ;   0  0  0  A
   %shl = shl i32 %b, 8          ;   B2 B1 B0 0
   %or = or i32 %conv, %shl      ;   B2 B1 B0 A

test/CodeGen/AMDGPU/idot4s.ll

 
 define amdgpu_kernel void @idot4_acc16_vecMul(<4 x i8> addrspace(1)* %src1,
 ; GFX7-LABEL: idot4_acc16_vecMul:
 ; GFX7:       ; %bb.0: ; %entry
 ; GFX7-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
 ; GFX7-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7-NEXT:    s_mov_b32 s2, -1
-; GFX7-NEXT:    s_mov_b32 s8, 0xffff
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; GFX7-NEXT:    buffer_load_ushort v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_load_dword s5, s[6:7], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_sext_i32_i8 s6, s4
-; GFX7-NEXT:    s_bfe_i32 s7, s4, 0x80008
-; GFX7-NEXT:    s_sext_i32_i8 s10, s5
+; GFX7-NEXT:    s_ashr_i32 s6, s4, 24
+; GFX7-NEXT:    s_bfe_i32 s7, s4, 0x80010
+; GFX7-NEXT:    s_bfe_i32 s10, s5, 0x80010
 ; GFX7-NEXT:    s_bfe_i32 s11, s5, 0x80008
-; GFX7-NEXT:    s_bfe_i32 s12, s5, 0x80010
-; GFX7-NEXT:    s_ashr_i32 s5, s5, 24
-; GFX7-NEXT:    v_mov_b32_e32 v3, s11
-; GFX7-NEXT:    v_mov_b32_e32 v4, s10
-; GFX7-NEXT:    s_bfe_i32 s9, s4, 0x80010
-; GFX7-NEXT:    v_mov_b32_e32 v2, s12
-; GFX7-NEXT:    s_ashr_i32 s4, s4, 24
+; GFX7-NEXT:    s_ashr_i32 s9, s5, 24
+; GFX7-NEXT:    s_sext_i32_i8 s5, s5
+; GFX7-NEXT:    s_bfe_i32 s8, s4, 0x80008
+; GFX7-NEXT:    s_sext_i32_i8 s4, s4
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s5
-; GFX7-NEXT:    v_mul_i32_i24_e32 v1, s4, v1
-; GFX7-NEXT:    v_mul_i32_i24_e32 v2, s9, v2
-; GFX7-NEXT:    v_mul_i32_i24_e32 v3, s7, v3
-; GFX7-NEXT:    v_mul_i32_i24_e32 v4, s6, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, s8, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s8, v4
-; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, v4, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 16, v1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s11
+; GFX7-NEXT:    v_mov_b32_e32 v3, s10
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s4, v1, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s8, v2, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s7, v3, v0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s9
+; GFX7-NEXT:    v_mad_i32_i24 v0, s6, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot4_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)

test/CodeGen/AMDGPU/idot4u.ll

 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    s_movk_i32 s8, 0xff
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_load_dword s4, s[4:5], 0x0
 ; GFX7-NEXT:    buffer_load_ushort v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_load_dword s5, s[6:7], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_and_b32 s11, s4, s8
-; GFX7-NEXT:    s_bfe_u32 s6, s4, 0x80008
-; GFX7-NEXT:    s_bfe_u32 s9, s5, 0x80008
-; GFX7-NEXT:    s_lshr_b32 s10, s5, 24
-; GFX7-NEXT:    s_and_b32 s8, s5, s8
-; GFX7-NEXT:    v_mov_b32_e32 v4, s9
-; GFX7-NEXT:    s_lshr_b32 s7, s4, 24
-; GFX7-NEXT:    v_mov_b32_e32 v2, s10
-; GFX7-NEXT:    s_bfe_u32 s5, s5, 0x80010
-; GFX7-NEXT:    v_mov_b32_e32 v3, s8
-; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s7, v2
-; GFX7-NEXT:    v_mul_u32_u24_e32 v4, s6, v4
-; GFX7-NEXT:    s_bfe_u32 s4, s4, 0x80010
+; GFX7-NEXT:    s_lshr_b32 s6, s4, 24
+; GFX7-NEXT:    s_bfe_u32 s7, s4, 0x80008
+; GFX7-NEXT:    s_bfe_u32 s10, s5, 0x80008
+; GFX7-NEXT:    s_bfe_u32 s12, s5, 0x80010
+; GFX7-NEXT:    s_lshr_b32 s9, s5, 24
+; GFX7-NEXT:    s_and_b32 s5, s5, s8
+; GFX7-NEXT:    s_bfe_u32 s11, s4, 0x80010
+; GFX7-NEXT:    s_and_b32 s4, s4, s8
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s5
-; GFX7-NEXT:    v_mul_u32_u24_e32 v1, s4, v1
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s11, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
-; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
-; GFX7-NEXT:    v_or_b32_e32 v2, v3, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 16, v1
+; GFX7-NEXT:    v_mov_b32_e32 v2, s10
+; GFX7-NEXT:    v_mov_b32_e32 v3, s12
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s4, v1, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s7, v2, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s11, v3, v0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s9
+; GFX7-NEXT:    v_mad_u32_u24 v0, s6, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot4_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX7-NEXT:    v_mov_b32_e32 v2, s11
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s10
 ; GFX7-NEXT:    s_lshr_b32 s9, s4, 24
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s12
 ; GFX7-NEXT:    s_mul_i32 s4, s4, s5
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v1, s9, v1
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s7, v2
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s6, v3
-; GFX7-NEXT:    s_and_b32 s4, s4, s8
+; GFX7-NEXT:    s_and_b32 s5, s4, s8
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, s8, v2
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v3
 ; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, s4, v3
+; GFX7-NEXT:    v_or_b32_e32 v2, s5, v3
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 8, v1
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 24, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 24, v1
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v2, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot4_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_movk_i32 s8, 0xff
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX8-NEXT:    v_mov_b32_e32 v3, s0
+; GFX8-NEXT:    v_mov_b32_e32 v4, s1
+; GFX8-NEXT:    s_and_b32 s7, s1, s8
 ; GFX8-NEXT:    s_lshr_b32 s2, s0, 24
 ; GFX8-NEXT:    s_lshr_b32 s3, s1, 24
 ; GFX8-NEXT:    s_bfe_u32 s6, s1, 0x80010
-; GFX8-NEXT:    s_and_b32 s7, s1, s8
-; GFX8-NEXT:    v_mov_b32_e32 v3, s0
-; GFX8-NEXT:    v_mov_b32_e32 v4, s1
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v3, v3, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX8-NEXT:    s_bfe_u32 s4, s0, 0x80010
-; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_and_b32 s5, s0, s8
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s7
+; GFX8-NEXT:    v_mul_u32_u24_e32 v4, s5, v4
+; GFX8-NEXT:    s_bfe_u32 s4, s0, 0x80010
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    v_mov_b32_e32 v6, s3
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s2
-; GFX8-NEXT:    v_mul_u32_u24_e32 v4, s5, v4
+; GFX8-NEXT:    v_or_b32_sdwa v3, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s4, v5
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v7, v6 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX8-NEXT:    v_or_b32_sdwa v5, v5, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_sdwa v3, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX8-NEXT:    v_lshrrev_b32_e32 v4, 8, v3
+; GFX8-NEXT:    v_or_b32_e32 v4, v3, v5
+; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v4, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-NODL-LABEL: udot4_acc8_vecMul:
 ; GFX9-NODL:       ; %bb.0: ; %entry
 ; GFX9-NODL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NODL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NODL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NODL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NODL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NODL-NEXT:    s_lshr_b32 s0, s2, 16
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX9-NODL-NEXT:    s_lshr_b32 s1, s3, 16
 ; GFX9-NODL-NEXT:    s_lshr_b32 s4, s3, 24
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v3, s2, v3
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v4, s2, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s1
 ; GFX9-NODL-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NODL-NEXT:    v_mov_b32_e32 v5, s1
 ; GFX9-NODL-NEXT:    s_lshr_b32 s5, s2, 24
 ; GFX9-NODL-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_sdwa v4, s5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NODL-NEXT:    v_mul_lo_u16_e32 v5, s0, v5
+; GFX9-NODL-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX9-NODL-NEXT:    v_or_b32_sdwa v4, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NODL-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v4, 8, v3
+; GFX9-NODL-NEXT:    v_or_b32_e32 v4, v3, v4
+; GFX9-NODL-NEXT:    v_lshrrev_b32_e32 v5, 8, v4
 ; GFX9-NODL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NODL-NEXT:    v_add_u32_e32 v2, v3, v2
-; GFX9-NODL-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-NODL-NEXT:    v_add_u32_e32 v2, v2, v5
+; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NODL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NODL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NODL-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot4_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s3, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_lshr_b32 s0, s2, 16
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s3
 ; GFX9-DL-NEXT:    s_lshr_b32 s1, s3, 16
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s3, 24
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v3, s2, v3
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v4, s2, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:BYTE_1
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s1
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s1
 ; GFX9-DL-NEXT:    s_lshr_b32 s5, s2, 24
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v4, s5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v5, s0, v5
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v4, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v4, 8, v3
+; GFX9-DL-NEXT:    v_or_b32_e32 v4, v3, v4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v4
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v3, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                              <4 x i8> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <4 x i8>, <4 x i8> addrspace(1)* %src1
   %vec2 = load <4 x i8>, <4 x i8> addrspace(1)* %src2
 

test/CodeGen/AMDGPU/idot8s.ll

 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_bfe_i32 s0, s2, 0x40000
 ; GFX8-NEXT:    s_bfe_i32 s1, s4, 0x40000
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_i32 s5, s4, 0x40004
-; GFX8-NEXT:    s_bfe_i32 s6, s4, 0x40008
 ; GFX8-NEXT:    s_lshr_b32 s1, s2, 12
-; GFX8-NEXT:    s_lshr_b32 s7, s4, 12
-; GFX8-NEXT:    s_bfe_i32 s8, s2, 0x40004
-; GFX8-NEXT:    s_bfe_i32 s9, s2, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v4, s6
-; GFX8-NEXT:    v_mov_b32_e32 v7, s5
+; GFX8-NEXT:    s_lshr_b32 s6, s4, 12
+; GFX8-NEXT:    s_bfe_i32 s8, s4, 0x40008
+; GFX8-NEXT:    v_mov_b32_e32 v4, s5
+; GFX8-NEXT:    s_bfe_i32 s7, s2, 0x40004
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
-; GFX8-NEXT:    v_lshlrev_b16_e64 v6, 12, s7
-; GFX8-NEXT:    v_mul_i32_i24_e32 v4, s9, v4
-; GFX8-NEXT:    s_bfe_i32 s10, s4, 0x40010
+; GFX8-NEXT:    v_lshlrev_b16_e64 v6, 12, s6
+; GFX8-NEXT:    v_mov_b32_e32 v7, s8
+; GFX8-NEXT:    s_bfe_i32 s5, s2, 0x40008
+; GFX8-NEXT:    s_bfe_i32 s1, s4, 0x40010
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX8-NEXT:    s_bfe_i32 s12, s4, 0x40014
-; GFX8-NEXT:    s_bfe_i32 s11, s2, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v8, s10
-; GFX8-NEXT:    s_bfe_i32 s14, s4, 0x40018
-; GFX8-NEXT:    s_bfe_i32 s13, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v9, s12
-; GFX8-NEXT:    s_bfe_i32 s15, s2, 0x40018
+; GFX8-NEXT:    s_bfe_i32 s8, s4, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v8, s1
+; GFX8-NEXT:    s_bfe_i32 s6, s2, 0x40010
+; GFX8-NEXT:    s_bfe_i32 s9, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v9, s8
+; GFX8-NEXT:    s_bfe_i32 s1, s2, 0x40014
+; GFX8-NEXT:    s_bfe_i32 s8, s2, 0x40018
 ; GFX8-NEXT:    s_ashr_i32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v10, s14
+; GFX8-NEXT:    v_mov_b32_e32 v10, s9
 ; GFX8-NEXT:    s_ashr_i32 s2, s2, 28
+; GFX8-NEXT:    v_mov_b32_e32 v11, s4
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX8-NEXT:    v_mad_i32_i24 v2, s7, v4, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s5, v7, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s11, v8, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s13, v9, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s15, v10, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s4
-; GFX8-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s6, v8, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s1, v9, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v10, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s2, v11, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc16:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ushort v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s0, s2, 0x40000
 ; GFX9-NEXT:    s_bfe_i32 s1, s4, 0x40000
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_i32 s5, s4, 0x40004
-; GFX9-NEXT:    s_bfe_i32 s6, s4, 0x40008
 ; GFX9-NEXT:    s_lshr_b32 s1, s2, 12
-; GFX9-NEXT:    s_lshr_b32 s7, s4, 12
-; GFX9-NEXT:    s_bfe_i32 s8, s2, 0x40004
-; GFX9-NEXT:    s_bfe_i32 s9, s2, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v4, s6
-; GFX9-NEXT:    v_mov_b32_e32 v7, s5
+; GFX9-NEXT:    s_lshr_b32 s6, s4, 12
+; GFX9-NEXT:    s_bfe_i32 s8, s4, 0x40008
+; GFX9-NEXT:    v_mov_b32_e32 v4, s5
+; GFX9-NEXT:    s_bfe_i32 s7, s2, 0x40004
 ; GFX9-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
-; GFX9-NEXT:    v_lshlrev_b16_e64 v6, 12, s7
-; GFX9-NEXT:    v_mul_i32_i24_e32 v4, s9, v4
-; GFX9-NEXT:    s_bfe_i32 s10, s4, 0x40010
+; GFX9-NEXT:    v_lshlrev_b16_e64 v6, 12, s6
+; GFX9-NEXT:    v_mov_b32_e32 v7, s8
+; GFX9-NEXT:    s_bfe_i32 s5, s2, 0x40008
+; GFX9-NEXT:    s_bfe_i32 s1, s4, 0x40010
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-NEXT:    s_bfe_i32 s12, s4, 0x40014
-; GFX9-NEXT:    s_bfe_i32 s11, s2, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v8, s10
-; GFX9-NEXT:    s_bfe_i32 s14, s4, 0x40018
-; GFX9-NEXT:    s_bfe_i32 s13, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v9, s12
-; GFX9-NEXT:    s_bfe_i32 s15, s2, 0x40018
+; GFX9-NEXT:    s_bfe_i32 s8, s4, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v8, s1
+; GFX9-NEXT:    s_bfe_i32 s6, s2, 0x40010
+; GFX9-NEXT:    s_bfe_i32 s9, s4, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v9, s8
+; GFX9-NEXT:    s_bfe_i32 s1, s2, 0x40014
+; GFX9-NEXT:    s_bfe_i32 s8, s2, 0x40018
 ; GFX9-NEXT:    s_ashr_i32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v10, s14
+; GFX9-NEXT:    v_mov_b32_e32 v10, s9
 ; GFX9-NEXT:    s_ashr_i32 s2, s2, 28
+; GFX9-NEXT:    v_mov_b32_e32 v11, s4
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX9-NEXT:    v_mad_i32_i24 v2, s7, v4, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s5, v7, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s11, v8, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s13, v9, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s15, v10, v2
-; GFX9-NEXT:    v_mov_b32_e32 v3, s4
-; GFX9-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s6, v8, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s1, v9, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v10, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s2, v11, v2
 ; GFX9-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc16:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ushort v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_bfe_i32 s0, s2, 0x40000
 ; GFX9-DL-NEXT:    s_bfe_i32 s1, s4, 0x40000
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    s_bfe_i32 s5, s4, 0x40004
-; GFX9-DL-NEXT:    s_bfe_i32 s6, s4, 0x40008
 ; GFX9-DL-NEXT:    s_lshr_b32 s1, s2, 12
-; GFX9-DL-NEXT:    s_lshr_b32 s7, s4, 12
-; GFX9-DL-NEXT:    s_bfe_i32 s8, s2, 0x40004
-; GFX9-DL-NEXT:    s_bfe_i32 s9, s2, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s6
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s5
+; GFX9-DL-NEXT:    s_lshr_b32 s6, s4, 12
+; GFX9-DL-NEXT:    s_bfe_i32 s8, s4, 0x40008
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
+; GFX9-DL-NEXT:    s_bfe_i32 s7, s2, 0x40004
 ; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s7
-; GFX9-DL-NEXT:    v_mul_i32_i24_e32 v4, s9, v4
-; GFX9-DL-NEXT:    s_bfe_i32 s10, s4, 0x40010
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v6, 12, s6
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s8
+; GFX9-DL-NEXT:    s_bfe_i32 s5, s2, 0x40008
+; GFX9-DL-NEXT:    s_bfe_i32 s1, s4, 0x40010
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
-; GFX9-DL-NEXT:    s_bfe_i32 s12, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_i32 s11, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s10
-; GFX9-DL-NEXT:    s_bfe_i32 s14, s4, 0x40018
-; GFX9-DL-NEXT:    s_bfe_i32 s13, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s12
-; GFX9-DL-NEXT:    s_bfe_i32 s15, s2, 0x40018
+; GFX9-DL-NEXT:    s_bfe_i32 s8, s4, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s1
+; GFX9-DL-NEXT:    s_bfe_i32 s6, s2, 0x40010
+; GFX9-DL-NEXT:    s_bfe_i32 s9, s4, 0x40018
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s8
+; GFX9-DL-NEXT:    s_bfe_i32 s1, s2, 0x40014
+; GFX9-DL-NEXT:    s_bfe_i32 s8, s2, 0x40018
 ; GFX9-DL-NEXT:    s_ashr_i32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v10, s14
+; GFX9-DL-NEXT:    v_mov_b32_e32 v10, s9
 ; GFX9-DL-NEXT:    s_ashr_i32 s2, s2, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v11, s4
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s7, v4, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s5, v7, v2
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s11, v8, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s13, v9, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s15, v10, v2
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s2, v3, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s6, v8, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s1, v9, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v10, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s2, v11, v2
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                        <8 x i4> addrspace(1)* %src2,
                                        i16 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_lshr_b32 s4, s0, 12
 ; GFX8-NEXT:    s_bfe_i32 s7, s1, 0x40000
 ; GFX8-NEXT:    s_lshr_b32 s5, s1, 12
 ; GFX8-NEXT:    s_bfe_i32 s9, s1, 0x40004
-; GFX8-NEXT:    s_bfe_i32 s11, s1, 0x40008
 ; GFX8-NEXT:    s_bfe_i32 s6, s0, 0x40000
-; GFX8-NEXT:    v_mov_b32_e32 v6, s7
-; GFX8-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
-; GFX8-NEXT:    v_lshlrev_b16_e64 v5, 12, s5
+; GFX8-NEXT:    v_mov_b32_e32 v5, s7
+; GFX8-NEXT:    v_lshlrev_b16_e64 v3, 12, s4
+; GFX8-NEXT:    v_lshlrev_b16_e64 v4, 12, s5
+; GFX8-NEXT:    s_bfe_i32 s11, s1, 0x40008
 ; GFX8-NEXT:    s_bfe_i32 s8, s0, 0x40004
-; GFX8-NEXT:    s_bfe_i32 s10, s0, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v3, s11
-; GFX8-NEXT:    v_mov_b32_e32 v7, s9
+; GFX8-NEXT:    v_mov_b32_e32 v6, s9
+; GFX8-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX8-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
-; GFX8-NEXT:    v_mul_i32_i24_e32 v3, s10, v3
+; GFX8-NEXT:    s_bfe_i32 s10, s0, 0x40008
+; GFX8-NEXT:    v_mov_b32_e32 v7, s11
 ; GFX8-NEXT:    s_bfe_i32 s13, s1, 0x40010
+; GFX8-NEXT:    v_and_b32_e32 v3, s2, v3
 ; GFX8-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX8-NEXT:    v_and_b32_e32 v5, s2, v5
 ; GFX8-NEXT:    s_bfe_i32 s15, s1, 0x40014
 ; GFX8-NEXT:    s_bfe_i32 s12, s0, 0x40010
 ; GFX8-NEXT:    v_mov_b32_e32 v8, s13
 ; GFX8-NEXT:    s_bfe_i32 s17, s1, 0x40018
 ; GFX8-NEXT:    s_bfe_i32 s14, s0, 0x40014
 ; GFX8-NEXT:    v_mov_b32_e32 v9, s15
 ; GFX8-NEXT:    s_bfe_i32 s16, s0, 0x40018
 ; GFX8-NEXT:    s_ashr_i32 s1, s1, 28
 ; GFX8-NEXT:    v_mov_b32_e32 v10, s17
 ; GFX8-NEXT:    s_ashr_i32 s0, s0, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_i32_i24 v2, s6, v6, v2
-; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX8-NEXT:    v_mad_u32_u24 v2, v4, v5, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s6, v5, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s8, v6, v2
+; GFX8-NEXT:    v_mad_i32_i24 v2, s10, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s12, v8, v2
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s14, v9, v2
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s16, v10, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc8:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_lshr_b32 s4, s0, 12
 ; GFX9-NEXT:    s_bfe_i32 s7, s1, 0x40000
 ; GFX9-NEXT:    s_lshr_b32 s5, s1, 12
 ; GFX9-NEXT:    s_bfe_i32 s9, s1, 0x40004
-; GFX9-NEXT:    s_bfe_i32 s11, s1, 0x40008
 ; GFX9-NEXT:    s_bfe_i32 s6, s0, 0x40000
-; GFX9-NEXT:    v_mov_b32_e32 v6, s7
-; GFX9-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
-; GFX9-NEXT:    v_lshlrev_b16_e64 v5, 12, s5
+; GFX9-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-NEXT:    v_lshlrev_b16_e64 v3, 12, s4
+; GFX9-NEXT:    v_lshlrev_b16_e64 v4, 12, s5
+; GFX9-NEXT:    s_bfe_i32 s11, s1, 0x40008
 ; GFX9-NEXT:    s_bfe_i32 s8, s0, 0x40004
-; GFX9-NEXT:    s_bfe_i32 s10, s0, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v3, s11
-; GFX9-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-NEXT:    v_mov_b32_e32 v6, s9
+; GFX9-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
-; GFX9-NEXT:    v_mul_i32_i24_e32 v3, s10, v3
+; GFX9-NEXT:    s_bfe_i32 s10, s0, 0x40008
+; GFX9-NEXT:    v_mov_b32_e32 v7, s11
 ; GFX9-NEXT:    s_bfe_i32 s13, s1, 0x40010
+; GFX9-NEXT:    v_and_b32_e32 v3, s2, v3
 ; GFX9-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX9-NEXT:    v_and_b32_e32 v5, s2, v5
 ; GFX9-NEXT:    s_bfe_i32 s15, s1, 0x40014
 ; GFX9-NEXT:    s_bfe_i32 s12, s0, 0x40010
 ; GFX9-NEXT:    v_mov_b32_e32 v8, s13
 ; GFX9-NEXT:    s_bfe_i32 s17, s1, 0x40018
 ; GFX9-NEXT:    s_bfe_i32 s14, s0, 0x40014
 ; GFX9-NEXT:    v_mov_b32_e32 v9, s15
 ; GFX9-NEXT:    s_bfe_i32 s16, s0, 0x40018
 ; GFX9-NEXT:    s_ashr_i32 s1, s1, 28
 ; GFX9-NEXT:    v_mov_b32_e32 v10, s17
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_i32_i24 v2, s6, v6, v2
-; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX9-NEXT:    v_mad_u32_u24 v2, v4, v5, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s6, v5, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s8, v6, v2
+; GFX9-NEXT:    v_mad_i32_i24 v2, s10, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s12, v8, v2
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s14, v9, v2
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s16, v10, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc8:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_movk_i32 s2, 0xff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_load_dword s0, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s1, s[6:7], 0x0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s0, 12
 ; GFX9-DL-NEXT:    s_bfe_i32 s7, s1, 0x40000
 ; GFX9-DL-NEXT:    s_lshr_b32 s5, s1, 12
 ; GFX9-DL-NEXT:    s_bfe_i32 s9, s1, 0x40004
-; GFX9-DL-NEXT:    s_bfe_i32 s11, s1, 0x40008
 ; GFX9-DL-NEXT:    s_bfe_i32 s6, s0, 0x40000
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s7
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v5, 12, s5
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v3, 12, s4
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v4, 12, s5
+; GFX9-DL-NEXT:    s_bfe_i32 s11, s1, 0x40008
 ; GFX9-DL-NEXT:    s_bfe_i32 s8, s0, 0x40004
-; GFX9-DL-NEXT:    s_bfe_i32 s10, s0, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s11
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s9
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
-; GFX9-DL-NEXT:    v_mul_i32_i24_e32 v3, s10, v3
+; GFX9-DL-NEXT:    s_bfe_i32 s10, s0, 0x40008
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s11
 ; GFX9-DL-NEXT:    s_bfe_i32 s13, s1, 0x40010
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, s2, v3
 ; GFX9-DL-NEXT:    v_and_b32_e32 v4, s2, v4
-; GFX9-DL-NEXT:    v_and_b32_e32 v5, s2, v5
 ; GFX9-DL-NEXT:    s_bfe_i32 s15, s1, 0x40014
 ; GFX9-DL-NEXT:    s_bfe_i32 s12, s0, 0x40010
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s13
 ; GFX9-DL-NEXT:    s_bfe_i32 s17, s1, 0x40018
 ; GFX9-DL-NEXT:    s_bfe_i32 s14, s0, 0x40014
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s15
 ; GFX9-DL-NEXT:    s_bfe_i32 s16, s0, 0x40018
 ; GFX9-DL-NEXT:    s_ashr_i32 s1, s1, 28
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v10, s17
 ; GFX9-DL-NEXT:    s_ashr_i32 s0, s0, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s6, v6, v2
-; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v7, v2
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v4, v5, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s6, v5, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s8, v6, v2
+; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s10, v7, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s12, v8, v2
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s14, v9, v2
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s16, v10, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    v_mad_i32_i24 v2, s0, v3, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                        <8 x i4> addrspace(1)* %src2,

 ; TODO: Support this pattern.
 define amdgpu_kernel void @idot8_acc16_vecMul(<8 x i4> addrspace(1)* %src1,
 ; GFX7-LABEL: idot8_acc16_vecMul:
 ; GFX7:       ; %bb.0: ; %entry
 ; GFX7-NEXT:    s_load_dwordx4 s[8:11], s[0:1], 0x9
 ; GFX7-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0xd
 ; GFX7-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX7-NEXT:    s_mov_b32 s6, -1
-; GFX7-NEXT:    s_mov_b32 s0, 0xffff
+; GFX7-NEXT:    s_mov_b32 s2, 0xffff
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_load_dword s1, s[8:9], 0x0
+; GFX7-NEXT:    s_load_dword s0, s[8:9], 0x0
 ; GFX7-NEXT:    buffer_load_ushort v0, off, s[4:7], 0
-; GFX7-NEXT:    s_load_dword s2, s[10:11], 0x0
+; GFX7-NEXT:    s_load_dword s1, s[10:11], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_bfe_i32 s8, s1, 0x40010
-; GFX7-NEXT:    s_bfe_i32 s9, s1, 0x40014
-; GFX7-NEXT:    s_bfe_i32 s15, s2, 0x40010
-; GFX7-NEXT:    s_bfe_i32 s16, s2, 0x40014
-; GFX7-NEXT:    s_bfe_i32 s17, s2, 0x40018
-; GFX7-NEXT:    s_ashr_i32 s18, s2, 28
-; GFX7-NEXT:    s_bfe_i32 s19, s2, 0x40000
-; GFX7-NEXT:    s_bfe_i32 s20, s2, 0x40004
-; GFX7-NEXT:    s_bfe_i32 s21, s2, 0x40008
-; GFX7-NEXT:    s_bfe_i32 s2, s2, 0x4000c
-; GFX7-NEXT:    s_bfe_i32 s10, s1, 0x40018
-; GFX7-NEXT:    s_ashr_i32 s11, s1, 28
-; GFX7-NEXT:    s_bfe_i32 s12, s1, 0x40000
+; GFX7-NEXT:    s_ashr_i32 s8, s0, 28
+; GFX7-NEXT:    s_bfe_i32 s9, s0, 0x40018
+; GFX7-NEXT:    s_bfe_i32 s16, s1, 0x40018
+; GFX7-NEXT:    s_bfe_i32 s17, s1, 0x40014
+; GFX7-NEXT:    s_bfe_i32 s18, s1, 0x40010
+; GFX7-NEXT:    s_bfe_i32 s19, s1, 0x40000
+; GFX7-NEXT:    s_bfe_i32 s20, s1, 0x40004
+; GFX7-NEXT:    s_bfe_i32 s21, s1, 0x40008
+; GFX7-NEXT:    s_ashr_i32 s15, s1, 28
+; GFX7-NEXT:    s_bfe_i32 s1, s1, 0x4000c
+; GFX7-NEXT:    s_bfe_i32 s10, s0, 0x40014
+; GFX7-NEXT:    s_bfe_i32 s11, s0, 0x40010
+; GFX7-NEXT:    s_bfe_i32 s12, s0, 0x40000
 ; GFX7-NEXT:    v_mov_b32_e32 v4, s19
-; GFX7-NEXT:    s_bfe_i32 s13, s1, 0x40004
+; GFX7-NEXT:    s_bfe_i32 s13, s0, 0x40004
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s20
-; GFX7-NEXT:    s_bfe_i32 s14, s1, 0x40008
+; GFX7-NEXT:    s_bfe_i32 s14, s0, 0x40008
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s21
-; GFX7-NEXT:    s_bfe_i32 s1, s1, 0x4000c
-; GFX7-NEXT:    v_mov_b32_e32 v1, s2
-; GFX7-NEXT:    v_mov_b32_e32 v5, s18
-; GFX7-NEXT:    v_mov_b32_e32 v6, s17
-; GFX7-NEXT:    v_mul_i32_i24_e32 v1, s1, v1
-; GFX7-NEXT:    v_mul_i32_i24_e32 v2, s14, v2
+; GFX7-NEXT:    s_bfe_i32 s0, s0, 0x4000c
+; GFX7-NEXT:    v_mov_b32_e32 v1, s1
+; GFX7-NEXT:    v_mul_i32_i24_e32 v1, s0, v1
+; GFX7-NEXT:    v_mul_i32_i24_e32 v8, s14, v2
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v3, s13, v3
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v4, s12, v4
-; GFX7-NEXT:    v_mul_i32_i24_e32 v5, s11, v5
-; GFX7-NEXT:    v_mul_i32_i24_e32 v6, s10, v6
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, s0, v2
+; GFX7-NEXT:    v_and_b32_e32 v8, s2, v8
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s0, v4
-; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, v4, v3
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v6, s0, v6
+; GFX7-NEXT:    v_and_b32_e32 v4, s2, v4
+; GFX7-NEXT:    v_or_b32_e32 v3, v4, v3
+; GFX7-NEXT:    v_or_b32_e32 v1, v8, v1
+; GFX7-NEXT:    v_alignbit_b32 v4, v1, v3, 16
+; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
+; GFX7-NEXT:    v_mov_b32_e32 v5, s18
+; GFX7-NEXT:    v_mov_b32_e32 v6, s17
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s16
-; GFX7-NEXT:    v_mov_b32_e32 v8, s15
-; GFX7-NEXT:    v_or_b32_e32 v3, v6, v5
-; GFX7-NEXT:    v_alignbit_b32 v5, v1, v2, 16
-; GFX7-NEXT:    v_mul_i32_i24_e32 v7, s9, v7
-; GFX7-NEXT:    v_mul_i32_i24_e32 v8, s8, v8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 16, v7
-; GFX7-NEXT:    v_and_b32_e32 v8, s0, v8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v1
-; GFX7-NEXT:    v_or_b32_e32 v4, v8, v7
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v3
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s14, v2, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
+; GFX7-NEXT:    v_mad_i32_i24 v0, s11, v5, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s10, v6, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s9, v7, v0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s15
+; GFX7-NEXT:    v_mad_i32_i24 v0, s8, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v3, 12, s2
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v4, 12, s4
 ; GFX8-NEXT:    s_lshr_b32 s0, s2, 4
-; GFX8-NEXT:    s_lshr_b32 s1, s2, 8
-; GFX8-NEXT:    s_lshr_b32 s5, s4, 4
+; GFX8-NEXT:    s_lshr_b32 s1, s4, 4
+; GFX8-NEXT:    v_lshlrev_b16_e64 v5, 12, s0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v6, 12, s1
+; GFX8-NEXT:    s_lshr_b32 s5, s2, 8
 ; GFX8-NEXT:    s_lshr_b32 s6, s4, 8
-; GFX8-NEXT:    v_lshlrev_b16_e64 v5, 12, s1
-; GFX8-NEXT:    v_lshlrev_b16_e64 v6, 12, s0
-; GFX8-NEXT:    v_lshlrev_b16_e64 v7, 12, s6
-; GFX8-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
+; GFX8-NEXT:    v_lshlrev_b16_e64 v7, 12, s5
+; GFX8-NEXT:    v_lshlrev_b16_e64 v8, 12, s6
 ; GFX8-NEXT:    s_lshr_b32 s0, s2, 12
 ; GFX8-NEXT:    s_lshr_b32 s1, s4, 12
-; GFX8-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
-; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
+; GFX8-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v9, 12, s0
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v10, 12, s1
 ; GFX8-NEXT:    s_lshr_b32 s5, s2, 16
 ; GFX8-NEXT:    s_lshr_b32 s6, s4, 16
-; GFX8-NEXT:    v_mul_u32_u24_e32 v5, v5, v7
+; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
+; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v11, 12, s5
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v12, 12, s6
 ; GFX8-NEXT:    s_lshr_b32 s0, s2, 20
 ; GFX8-NEXT:    s_lshr_b32 s1, s4, 20
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v13, 12, s0
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v14, 12, s1
 ; GFX8-NEXT:    s_lshr_b32 s5, s2, 24
 ; GFX8-NEXT:    s_lshr_b32 s6, s4, 24
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v15, 12, s5
-; GFX8-NEXT:    v_lshlrev_b16_e64 v17, 12, s6
+; GFX8-NEXT:    v_lshlrev_b16_e64 v16, 12, s6
 ; GFX8-NEXT:    s_lshr_b32 s0, s2, 28
 ; GFX8-NEXT:    s_lshr_b32 s1, s4, 28
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
-; GFX8-NEXT:    v_lshlrev_b16_e64 v16, 12, s0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v17, 12, s0
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v18, 12, s1
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
-; GFX8-NEXT:    v_ashrrev_i16_e32 v17, 12, v17
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v16, 12, v16
+; GFX8-NEXT:    v_ashrrev_i16_e32 v17, 12, v17
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v18, 12, v18
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, v3, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v6, v8, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v5, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX8-NEXT:    v_mad_u32_u24 v2, v5, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, v7, v8, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, v9, v10, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, v11, v12, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, v13, v14, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v15, v17, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, v16, v18, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, v15, v16, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, v17, v18, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc16_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX9-NEXT:    v_pk_lshlrev_b16 v10, 12, s0 op_sel_hi:[0,1]
 ; GFX9-NEXT:    v_pk_mul_lo_u16 v5, v5, v9
 ; GFX9-NEXT:    v_pk_ashrrev_i16 v6, 12, v6 op_sel_hi:[0,1]
 ; GFX9-NEXT:    v_pk_ashrrev_i16 v10, 12, v10 op_sel_hi:[0,1]
 ; GFX9-NEXT:    v_pk_mul_lo_u16 v6, v6, v10
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_add_u32_e32 v2, v3, v2
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v6
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;

 ; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v10, 12, s0 op_sel_hi:[0,1]
 ; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v5, v5, v9
 ; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v6, 12, v6 op_sel_hi:[0,1]
 ; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v10, 12, v10 op_sel_hi:[0,1]
 ; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v6, v6, v10
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v3, v2
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:WORD_0
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v6
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                               <8 x i4> addrspace(1)* %src2,

 ; GFX7-NEXT:    v_mov_b32_e32 v3, s21
 ; GFX7-NEXT:    s_bfe_i32 s15, s2, 0x40018
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s22
 ; GFX7-NEXT:    s_ashr_i32 s2, s2, 28
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s8
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v1, s2, v1
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v2, s15, v2
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v3, s14, v3
-; GFX7-NEXT:    v_mul_i32_i24_e32 v4, s13, v4
+; GFX7-NEXT:    v_mul_i32_i24_e32 v9, s13, v4
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v5, s12, v5
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v6, s11, v6
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v7, s10, v7
 ; GFX7-NEXT:    v_mul_i32_i24_e32 v8, s9, v8
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, s0, v2
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s0, v4
+; GFX7-NEXT:    v_and_b32_e32 v9, s0, v9
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 8, v5
 ; GFX7-NEXT:    v_and_b32_e32 v6, s0, v6
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 8, v7
 ; GFX7-NEXT:    v_and_b32_e32 v8, s0, v8
 ; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, v4, v3
+; GFX7-NEXT:    v_or_b32_e32 v2, v9, v3
 ; GFX7-NEXT:    v_or_b32_e32 v3, v6, v5
-; GFX7-NEXT:    v_or_b32_e32 v4, v8, v7
+; GFX7-NEXT:    v_or_b32_e32 v5, v8, v7
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, s1, v2
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, s1, v4
+; GFX7-NEXT:    v_and_b32_e32 v5, s1, v5
 ; GFX7-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_or_b32_e32 v2, v4, v3
+; GFX7-NEXT:    v_or_b32_e32 v2, v5, v3
 ; GFX7-NEXT:    v_alignbit_b32 v3, v1, v2, 8
-; GFX7-NEXT:    v_alignbit_b32 v4, v1, v2, 16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 8, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 24, v1
+; GFX7-NEXT:    v_alignbit_b32 v5, v1, v2, 16
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 8, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 24, v1
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
+; GFX7-NEXT:    v_mad_i32_i24 v0, s13, v4, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX8-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v3, v3, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v4, v8 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v5, v5, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v6, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    s_lshr_b32 s5, s4, 20
+; GFX8-NEXT:    s_lshr_b32 s6, s4, 16
 ; GFX8-NEXT:    s_lshr_b32 s0, s2, 20
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v6, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v5, v5, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX8-NEXT:    s_lshr_b32 s1, s2, 16
-; GFX8-NEXT:    s_lshr_b32 s5, s2, 28
-; GFX8-NEXT:    s_lshr_b32 s2, s2, 24
-; GFX8-NEXT:    s_lshr_b32 s6, s4, 20
-; GFX8-NEXT:    s_lshr_b32 s7, s4, 16
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v3, v3, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v4, v8 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    s_lshr_b32 s7, s2, 28
 ; GFX8-NEXT:    s_lshr_b32 s8, s4, 28
+; GFX8-NEXT:    v_lshlrev_b16_e64 v9, 12, s1
+; GFX8-NEXT:    v_lshlrev_b16_e64 v10, 12, s0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v13, 12, s6
+; GFX8-NEXT:    v_lshlrev_b16_e64 v14, 12, s5
+; GFX8-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 24
 ; GFX8-NEXT:    v_or_b32_sdwa v5, v6, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_sdwa v3, v5, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v7, 12, s2
-; GFX8-NEXT:    v_lshlrev_b16_e64 v8, 12, s5
-; GFX8-NEXT:    v_lshlrev_b16_e64 v9, 12, s1
-; GFX8-NEXT:    v_lshlrev_b16_e64 v10, 12, s0
+; GFX8-NEXT:    v_lshlrev_b16_e64 v8, 12, s7
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v11, 12, s4
 ; GFX8-NEXT:    v_lshlrev_b16_e64 v12, 12, s8
-; GFX8-NEXT:    v_lshlrev_b16_e64 v13, 12, s7
-; GFX8-NEXT:    v_lshlrev_b16_e64 v14, 12, s6
-; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
-; GFX8-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
-; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
-; GFX8-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
+; GFX8-NEXT:    v_or_b32_sdwa v3, v5, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
 ; GFX8-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
-; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v7, v7, v11 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v8, v8, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
+; GFX8-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
+; GFX8-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
+; GFX8-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v9, v9, v13 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v10, v10, v14 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v3
 ; GFX8-NEXT:    v_or_b32_sdwa v9, v9, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v7, v7, v11 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v8, v8, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_0
 ; GFX8-NEXT:    v_or_b32_sdwa v7, v7, v8 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_sdwa v4, v9, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX8-NEXT:    v_and_b32_e32 v4, 0xffff, v9
+; GFX8-NEXT:    v_or_b32_e32 v5, v4, v7
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v5
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v6, v2
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v6
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v7, v2
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v5, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v5, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: idot8_acc8_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX9-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
+; GFX9-NEXT:    s_lshr_b32 s0, s2, 20
+; GFX9-NEXT:    s_lshr_b32 s5, s4, 20
+; GFX9-NEXT:    s_lshr_b32 s6, s4, 16
+; GFX9-NEXT:    s_lshr_b32 s1, s2, 16
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v6, v6, v10
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v5, v5, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v4, v4, v8 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v3, v3, v7
-; GFX9-NEXT:    s_lshr_b32 s0, s2, 20
-; GFX9-NEXT:    s_lshr_b32 s1, s2, 16
-; GFX9-NEXT:    s_lshr_b32 s5, s2, 28
-; GFX9-NEXT:    s_lshr_b32 s2, s2, 24
-; GFX9-NEXT:    s_lshr_b32 s6, s4, 20
-; GFX9-NEXT:    s_lshr_b32 s7, s4, 16
+; GFX9-NEXT:    s_lshr_b32 s7, s2, 28
 ; GFX9-NEXT:    s_lshr_b32 s8, s4, 28
+; GFX9-NEXT:    v_lshlrev_b16_e64 v10, 12, s1
+; GFX9-NEXT:    v_lshlrev_b16_e64 v11, 12, s0
+; GFX9-NEXT:    v_lshlrev_b16_e64 v14, 12, s6
+; GFX9-NEXT:    v_lshlrev_b16_e64 v15, 12, s5
+; GFX9-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 24
 ; GFX9-NEXT:    v_or_b32_sdwa v5, v6, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e64 v8, 12, s2
+; GFX9-NEXT:    v_lshlrev_b16_e64 v9, 12, s7
+; GFX9-NEXT:    v_lshlrev_b16_e64 v12, 12, s4
+; GFX9-NEXT:    v_lshlrev_b16_e64 v13, 12, s8
 ; GFX9-NEXT:    v_or_b32_sdwa v3, v5, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    v_lshlrev_b16_e64 v9, 12, s2
-; GFX9-NEXT:    v_lshlrev_b16_e64 v10, 12, s5
-; GFX9-NEXT:    v_lshlrev_b16_e64 v11, 12, s1
-; GFX9-NEXT:    v_lshlrev_b16_e64 v12, 12, s0
-; GFX9-NEXT:    v_lshlrev_b16_e64 v13, 12, s4
-; GFX9-NEXT:    v_lshlrev_b16_e64 v14, 12, s8
-; GFX9-NEXT:    v_lshlrev_b16_e64 v15, 12, s7
-; GFX9-NEXT:    v_lshlrev_b16_e64 v16, 12, s6
-; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX9-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
+; GFX9-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX9-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
-; GFX9-NEXT:    v_ashrrev_i16_e32 v16, 12, v16
-; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v12, v12, v16 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v11, v11, v15
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v10, v10, v14 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v9, v9, v13
-; GFX9-NEXT:    v_or_b32_sdwa v7, v11, v12 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v8, v9, v10 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v4, v7, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
+; GFX9-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v11, v11, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_e32 v10, v10, v14
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v3
+; GFX9-NEXT:    v_or_b32_sdwa v7, v10, v11 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v9, v9, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_e32 v8, v8, v12
+; GFX9-NEXT:    v_or_b32_sdwa v8, v8, v9 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v4, 0xffff, v7
+; GFX9-NEXT:    v_or_b32_e32 v5, v4, v8
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_add_u32_e32 v2, v3, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
+; GFX9-NEXT:    v_add_u32_e32 v2, v2, v6
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-NEXT:    v_lshrrev_b32_e32 v3, 8, v4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v3, 8, v5
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v3
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v3, 12, v3
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v7, 12, v7
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v4, 12, v4
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v5, 12, v5
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v6, 12, v6
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
+; GFX9-DL-NEXT:    s_lshr_b32 s0, s2, 20
+; GFX9-DL-NEXT:    s_lshr_b32 s5, s4, 20
+; GFX9-DL-NEXT:    s_lshr_b32 s6, s4, 16
+; GFX9-DL-NEXT:    s_lshr_b32 s1, s2, 16
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v6, v6, v10
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v5, v5, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v4, v4, v8 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v3, v3, v7
-; GFX9-DL-NEXT:    s_lshr_b32 s0, s2, 20
-; GFX9-DL-NEXT:    s_lshr_b32 s1, s2, 16
-; GFX9-DL-NEXT:    s_lshr_b32 s5, s2, 28
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 24
-; GFX9-DL-NEXT:    s_lshr_b32 s6, s4, 20
-; GFX9-DL-NEXT:    s_lshr_b32 s7, s4, 16
+; GFX9-DL-NEXT:    s_lshr_b32 s7, s2, 28
 ; GFX9-DL-NEXT:    s_lshr_b32 s8, s4, 28
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s1
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s0
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v14, 12, s6
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v15, 12, s5
+; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 24
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 24
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v5, v6, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v8, 12, s2
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s7
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s4
+; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v13, 12, s8
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v5, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v9, 12, s2
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v10, 12, s5
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v11, 12, s1
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v12, 12, s0
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v13, 12, s4
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v14, 12, s8
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v15, 12, s7
-; GFX9-DL-NEXT:    v_lshlrev_b16_e64 v16, 12, s6
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v10, 12, v10
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v14, 12, v14
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v11, 12, v11
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v15, 12, v15
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v8, 12, v8
 ; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v12, 12, v12
-; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v16, 12, v16
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v12, v12, v16 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v11, v11, v15
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v10, v10, v14 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v9, v9, v13
-; GFX9-DL-NEXT:    v_or_b32_sdwa v7, v11, v12 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_sdwa v8, v9, v10 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_sdwa v4, v7, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v9, 12, v9
+; GFX9-DL-NEXT:    v_ashrrev_i16_e32 v13, 12, v13
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v11, v11, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v10, v10, v14
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v3
+; GFX9-DL-NEXT:    v_or_b32_sdwa v7, v10, v11 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v9, v9, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v8, v8, v12
+; GFX9-DL-NEXT:    v_or_b32_sdwa v8, v8, v9 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v4, 0xffff, v7
+; GFX9-DL-NEXT:    v_or_b32_e32 v5, v4, v8
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v3, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v6
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v4
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v5
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v3
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

test/CodeGen/AMDGPU/idot8u.ll

 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
+; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
+; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
 ; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX8-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX8-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v6, s4
-; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s8
-; GFX8-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v8, s10
-; GFX8-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v9, s12
+; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v8, s9
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX8-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s14
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc16:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ushort v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
+; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
+; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
 ; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX9-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX9-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX9-NEXT:    v_mov_b32_e32 v6, s4
-; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40014
 ; GFX9-NEXT:    v_mov_b32_e32 v7, s8
-; GFX9-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v8, s10
-; GFX9-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX9-NEXT:    v_mov_b32_e32 v9, s12
+; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v8, s9
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX9-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX9-NEXT:    v_mov_b32_e32 v3, s14
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc16:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ushort v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
-; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX9-DL-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX9-DL-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s4
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s8
-; GFX9-DL-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s10
-; GFX9-DL-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s12
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s14
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
+; GFX9-DL-NEXT:    v_dot8_u32_u4 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                        <8 x i4> addrspace(1)* %src2,
                                        i16 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
+; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
+; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
 ; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX8-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX8-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v6, s4
-; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s8
-; GFX8-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v8, s10
-; GFX8-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v9, s12
+; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v8, s9
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX8-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 0xff, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s14
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc8:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
+; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40008
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
+; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
 ; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX9-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX9-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX9-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX9-NEXT:    v_mov_b32_e32 v6, s4
-; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40014
 ; GFX9-NEXT:    v_mov_b32_e32 v7, s8
-; GFX9-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v8, s10
-; GFX9-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX9-NEXT:    v_mov_b32_e32 v9, s12
+; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v8, s9
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX9-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-NEXT:    v_and_b32_e32 v2, 0xff, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX9-NEXT:    v_mov_b32_e32 v3, s14
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc8:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
-; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX9-DL-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX9-DL-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s4
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s8
-; GFX9-DL-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s10
-; GFX9-DL-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s12
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 0xff, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s14
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
+; GFX9-DL-NEXT:    v_dot8_u32_u4 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                       <8 x i4> addrspace(1)* %src2,
                                       i8 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2
 

 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v5, s7
-; GFX8-NEXT:    v_mov_b32_e32 v6, s6
-; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX8-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v7, s9
-; GFX8-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v8, s11
-; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v7, s8
+; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v8, s9
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v9, s13
+; GFX8-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc4:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX9-NEXT:    v_mov_b32_e32 v5, s7
-; GFX9-NEXT:    v_mov_b32_e32 v6, s6
-; GFX9-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX9-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v7, s9
-; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX9-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v8, s11
-; GFX9-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v7, s8
+; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v8, s9
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v9, s13
+; GFX9-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc4:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
-; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s6
-; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX9-DL-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
-; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX9-DL-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s11
-; GFX9-DL-NEXT:    s_bfe_u32 s14, s2, 0x40018
-; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s13
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-DL-NEXT:    v_dot8_u32_u4 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                       <8 x i4> addrspace(1)* %src2,
                                       i4 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2

 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v5, s7
-; GFX8-NEXT:    v_mov_b32_e32 v6, s6
-; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX8-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v7, s9
-; GFX8-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v8, s11
-; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v7, s8
+; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v8, s9
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v9, s13
+; GFX8-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v5
-; GFX8-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_CommutationInsideMAD:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX9-NEXT:    v_mov_b32_e32 v5, s7
-; GFX9-NEXT:    v_mov_b32_e32 v6, s6
-; GFX9-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX9-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v7, s9
-; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX9-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v8, s11
-; GFX9-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v7, s8
+; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v8, s9
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v9, s13
+; GFX9-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v5, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_CommutationInsideMAD:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-DL-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s6
-; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX9-DL-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
-; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX9-DL-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s11
-; GFX9-DL-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX9-DL-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s8
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX9-DL-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s9
+; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX9-DL-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s13
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v5, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                                       <8 x i4> addrspace(1)* %src2,
                                                       i4 addrspace(1)* nocapture %dst) {
 entry:

 ; GFX7-NEXT:    s_load_dword s1, s[10:11], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_bfe_u32 s11, s0, 0x40004
 ; GFX7-NEXT:    s_bfe_u32 s13, s0, 0x4000c
 ; GFX7-NEXT:    s_bfe_u32 s18, s1, 0x40004
 ; GFX7-NEXT:    s_bfe_u32 s20, s1, 0x4000c
 ; GFX7-NEXT:    v_mov_b32_e32 v2, s20
 ; GFX7-NEXT:    v_mov_b32_e32 v4, s18
-; GFX7-NEXT:    s_bfe_u32 s14, s1, 0x40014
-; GFX7-NEXT:    s_bfe_u32 s15, s1, 0x40010
-; GFX7-NEXT:    s_lshr_b32 s16, s1, 28
-; GFX7-NEXT:    s_bfe_u32 s17, s1, 0x40018
+; GFX7-NEXT:    s_bfe_u32 s15, s1, 0x40018
+; GFX7-NEXT:    s_bfe_u32 s16, s1, 0x40014
+; GFX7-NEXT:    s_bfe_u32 s17, s1, 0x40010
 ; GFX7-NEXT:    s_and_b32 s19, s1, 15
+; GFX7-NEXT:    s_lshr_b32 s14, s1, 28
 ; GFX7-NEXT:    s_bfe_u32 s1, s1, 0x40008
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s13, v2
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v4, s11, v4
-; GFX7-NEXT:    s_bfe_u32 s2, s0, 0x40014
-; GFX7-NEXT:    s_bfe_u32 s8, s0, 0x40010
-; GFX7-NEXT:    s_lshr_b32 s9, s0, 28
-; GFX7-NEXT:    v_mov_b32_e32 v6, s16
-; GFX7-NEXT:    s_bfe_u32 s10, s0, 0x40018
+; GFX7-NEXT:    s_lshr_b32 s2, s0, 28
+; GFX7-NEXT:    s_bfe_u32 s8, s0, 0x40018
+; GFX7-NEXT:    s_bfe_u32 s9, s0, 0x40014
+; GFX7-NEXT:    s_bfe_u32 s10, s0, 0x40010
 ; GFX7-NEXT:    s_and_b32 s12, s0, 15
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s19
 ; GFX7-NEXT:    s_bfe_u32 s0, s0, 0x40008
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s1
-; GFX7-NEXT:    v_mov_b32_e32 v5, s17
-; GFX7-NEXT:    v_mul_u32_u24_e32 v6, s9, v6
-; GFX7-NEXT:    v_mul_u32_u24_e32 v1, s0, v1
+; GFX7-NEXT:    v_mul_u32_u24_e32 v8, s0, v1
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s12, v3
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
-; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
-; GFX7-NEXT:    v_or_b32_e32 v2, v3, v4
-; GFX7-NEXT:    v_mul_u32_u24_e32 v5, s10, v5
-; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
-; GFX7-NEXT:    v_mov_b32_e32 v8, s14
-; GFX7-NEXT:    v_or_b32_e32 v3, v5, v6
-; GFX7-NEXT:    v_alignbit_b32 v5, v1, v2, 16
+; GFX7-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX7-NEXT:    v_or_b32_e32 v2, v8, v2
+; GFX7-NEXT:    v_alignbit_b32 v4, v2, v3, 16
+; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 16, v2
+; GFX7-NEXT:    v_mov_b32_e32 v5, s17
+; GFX7-NEXT:    v_mov_b32_e32 v6, s16
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s15
-; GFX7-NEXT:    v_mul_u32_u24_e32 v8, s2, v8
-; GFX7-NEXT:    v_mul_u32_u24_e32 v7, s8, v7
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 16, v8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v1
-; GFX7-NEXT:    v_or_b32_e32 v4, v7, v8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v3
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s0, v1, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
+; GFX7-NEXT:    v_mad_u32_u24 v0, s10, v5, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s9, v6, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s8, v7, v0
+; GFX7-NEXT:    v_mov_b32_e32 v1, s14
+; GFX7-NEXT:    v_mad_u32_u24 v0, s2, v1, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[4:7], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX8-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ushort v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
+; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40008
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
+; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
 ; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s12, s4, 0x40018
-; GFX8-NEXT:    s_lshr_b32 s14, s4, 28
-; GFX8-NEXT:    s_bfe_u32 s4, s4, 0x4000c
-; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x40008
-; GFX8-NEXT:    v_mov_b32_e32 v5, s5
-; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v6, s4
-; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s8
-; GFX8-NEXT:    s_bfe_u32 s11, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v8, s10
-; GFX8-NEXT:    s_bfe_u32 s13, s2, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v9, s12
+; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v8, s9
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
+; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX8-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v6, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s11, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s13, v9, v2
-; GFX8-NEXT:    v_mov_b32_e32 v3, s14
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
+; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    flat_store_short v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc16_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34

 ; GFX9-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX9-NEXT:    v_pk_mul_lo_u16 v5, s5, v5
 ; GFX9-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX9-NEXT:    v_mov_b32_e32 v6, s1
 ; GFX9-NEXT:    v_pk_mul_lo_u16 v6, s0, v6
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_add_u32_e32 v2, v3, v2
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:BYTE_0
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v6
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;

 ; GFX9-DL-NEXT:    s_pack_ll_b32_b16 s1, s1, s4
 ; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v5, s5, v5
 ; GFX9-DL-NEXT:    s_pack_ll_b32_b16 s0, s0, s2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s1
 ; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v6, s0, v6
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v3, v2
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:BYTE_0
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_0
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v6
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    global_store_short v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                               <8 x i4> addrspace(1)* %src2,

 ; GFX7-NEXT:    s_mov_b32 s7, 0xf000
 ; GFX7-NEXT:    s_mov_b32 s6, -1
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_load_dword s0, s[8:9], 0x0
 ; GFX7-NEXT:    buffer_load_ubyte v0, off, s[4:7], 0
 ; GFX7-NEXT:    s_load_dword s1, s[10:11], 0x0
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_bfe_u32 s2, s0, 0x4000c
-; GFX7-NEXT:    s_lshr_b32 s11, s0, 28
+; GFX7-NEXT:    s_bfe_u32 s9, s0, 0x40004
 ; GFX7-NEXT:    s_bfe_u32 s14, s1, 0x4000c
+; GFX7-NEXT:    s_bfe_u32 s16, s1, 0x40004
 ; GFX7-NEXT:    s_lshr_b32 s18, s1, 28
-; GFX7-NEXT:    s_bfe_u32 s20, s1, 0x40014
-; GFX7-NEXT:    v_mov_b32_e32 v4, s18
+; GFX7-NEXT:    v_mov_b32_e32 v6, s16
 ; GFX7-NEXT:    v_mov_b32_e32 v8, s14
 ; GFX7-NEXT:    s_bfe_u32 s15, s1, 0x40008
-; GFX7-NEXT:    s_bfe_u32 s16, s1, 0x40004
 ; GFX7-NEXT:    s_and_b32 s17, s1, 15
 ; GFX7-NEXT:    s_bfe_u32 s19, s1, 0x40018
-; GFX7-NEXT:    s_bfe_u32 s1, s1, 0x40010
-; GFX7-NEXT:    s_bfe_u32 s13, s0, 0x40014
-; GFX7-NEXT:    v_mov_b32_e32 v2, s20
-; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s13, v2
+; GFX7-NEXT:    s_bfe_u32 s20, s1, 0x40014
+; GFX7-NEXT:    s_lshr_b32 s11, s0, 28
+; GFX7-NEXT:    v_mov_b32_e32 v4, s18
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v4, s11, v4
+; GFX7-NEXT:    v_mul_u32_u24_e32 v6, s9, v6
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v8, s2, v8
+; GFX7-NEXT:    s_bfe_u32 s1, s1, 0x40010
 ; GFX7-NEXT:    s_bfe_u32 s8, s0, 0x40008
 ; GFX7-NEXT:    v_mov_b32_e32 v7, s15
-; GFX7-NEXT:    s_bfe_u32 s9, s0, 0x40004
-; GFX7-NEXT:    v_mov_b32_e32 v6, s16
 ; GFX7-NEXT:    s_and_b32 s10, s0, 15
+; GFX7-NEXT:    v_mov_b32_e32 v5, s17
 ; GFX7-NEXT:    s_bfe_u32 s12, s0, 0x40018
 ; GFX7-NEXT:    v_mov_b32_e32 v3, s19
+; GFX7-NEXT:    s_bfe_u32 s13, s0, 0x40014
+; GFX7-NEXT:    v_mov_b32_e32 v2, s20
+; GFX7-NEXT:    v_mul_u32_u24_e32 v2, s13, v2
 ; GFX7-NEXT:    s_bfe_u32 s0, s0, 0x40010
 ; GFX7-NEXT:    v_mov_b32_e32 v1, s1
-; GFX7-NEXT:    v_mov_b32_e32 v5, s17
-; GFX7-NEXT:    v_mul_u32_u24_e32 v6, s9, v6
-; GFX7-NEXT:    v_mul_u32_u24_e32 v1, s0, v1
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v3, s12, v3
-; GFX7-NEXT:    v_mul_u32_u24_e32 v7, s8, v7
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 8, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 8, v8
-; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
-; GFX7-NEXT:    v_or_b32_e32 v2, v3, v4
-; GFX7-NEXT:    v_or_b32_e32 v4, v7, v8
 ; GFX7-NEXT:    v_mul_u32_u24_e32 v5, s10, v5
+; GFX7-NEXT:    v_mul_u32_u24_e32 v7, s8, v7
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; GFX7-NEXT:    v_or_b32_e32 v3, v5, v6
-; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
-; GFX7-NEXT:    v_or_b32_e32 v1, v1, v2
-; GFX7-NEXT:    v_or_b32_e32 v2, v3, v4
-; GFX7-NEXT:    v_alignbit_b32 v3, v1, v2, 8
-; GFX7-NEXT:    v_alignbit_b32 v4, v1, v2, 16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 8, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 24, v1
+; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 8, v8
+; GFX7-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX7-NEXT:    v_or_b32_e32 v4, v5, v6
+; GFX7-NEXT:    v_or_b32_e32 v5, v7, v8
+; GFX7-NEXT:    v_mul_u32_u24_e32 v9, s0, v1
+; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 8, v2
+; GFX7-NEXT:    v_or_b32_e32 v2, v9, v2
+; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
+; GFX7-NEXT:    v_or_b32_e32 v2, v2, v3
+; GFX7-NEXT:    v_or_b32_e32 v3, v4, v5
+; GFX7-NEXT:    v_alignbit_b32 v4, v2, v3, 8
+; GFX7-NEXT:    v_alignbit_b32 v5, v2, v3, 16
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 8, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 24, v2
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v3, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v4, v0
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v5, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v1
 ; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v6, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v7, v0
-; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v8, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, s0, v1, v0
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v7
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v8
+; GFX7-NEXT:    v_add_i32_e32 v0, vcc, v0, v2
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[4:7], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX8-NEXT:    v_mov_b32_e32 v6, s6
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s5
 ; GFX8-NEXT:    v_mov_b32_e32 v8, s0
 ; GFX8-NEXT:    s_and_b32 s9, s2, 15
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v4, v5, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX8-NEXT:    v_mul_u32_u24_e32 v3, s10, v3
 ; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s9, v6
 ; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v8, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    s_bfe_u32 s0, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s1, s4, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40010
+; GFX8-NEXT:    s_lshr_b32 s7, s2, 28
+; GFX8-NEXT:    s_lshr_b32 s8, s4, 28
 ; GFX8-NEXT:    v_or_b32_e32 v5, v5, v6
 ; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX8-NEXT:    v_or_b32_sdwa v3, v5, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX8-NEXT:    s_bfe_u32 s0, s2, 0x40014
-; GFX8-NEXT:    s_lshr_b32 s1, s2, 28
-; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40010
-; GFX8-NEXT:    s_lshr_b32 s7, s4, 28
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x40010
 ; GFX8-NEXT:    s_bfe_u32 s4, s4, 0x40018
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40010
-; GFX8-NEXT:    s_bfe_u32 s2, s2, 0x40018
-; GFX8-NEXT:    v_mov_b32_e32 v6, s4
+; GFX8-NEXT:    v_mov_b32_e32 v6, s8
 ; GFX8-NEXT:    v_mov_b32_e32 v7, s7
-; GFX8-NEXT:    v_mov_b32_e32 v8, s1
-; GFX8-NEXT:    v_mov_b32_e32 v9, s6
-; GFX8-NEXT:    v_mov_b32_e32 v10, s5
-; GFX8-NEXT:    v_mov_b32_e32 v11, s0
+; GFX8-NEXT:    v_mov_b32_e32 v8, s5
+; GFX8-NEXT:    v_mov_b32_e32 v9, s1
+; GFX8-NEXT:    v_mov_b32_e32 v10, s0
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v6, v7, v6 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_mul_u32_u24_e32 v7, s6, v8
+; GFX8-NEXT:    v_mul_u32_u24_sdwa v8, v10, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    s_bfe_u32 s2, s2, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v4, s4
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v7, v8, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_mul_u32_u24_e32 v6, s2, v6
-; GFX8-NEXT:    v_mul_u32_u24_e32 v8, s8, v9
-; GFX8-NEXT:    v_mul_u32_u24_sdwa v9, v11, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_e32 v8, v8, v9
-; GFX8-NEXT:    v_or_b32_sdwa v6, v6, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_sdwa v4, v8, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v4
+; GFX8-NEXT:    v_mul_u32_u24_e32 v4, s2, v4
+; GFX8-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX8-NEXT:    v_or_b32_sdwa v4, v4, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v6, 0xffff, v7
+; GFX8-NEXT:    v_or_b32_e32 v4, v6, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v3
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v4
 ; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v2, v6
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
+; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v7, v2
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_add_u32_sdwa v2, vcc, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc8_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)

 ; GFX9-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40008
 ; GFX9-NEXT:    v_mov_b32_e32 v6, s7
 ; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x4000c
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v3, s0, v3
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v4, s8, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_mul_lo_u16_e32 v5, s9, v5
 ; GFX9-NEXT:    v_mul_lo_u16_sdwa v6, s10, v6 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    s_bfe_u32 s0, s4, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s1, s4, 0x40014
 ; GFX9-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX9-NEXT:    v_or_b32_sdwa v4, v5, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    s_bfe_u32 s1, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40018
-; GFX9-NEXT:    s_bfe_u32 s0, s4, 0x40010
-; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x40018
 ; GFX9-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v4, s0
-; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v5, s1
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40018
-; GFX9-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
-; GFX9-NEXT:    v_mov_b32_e32 v7, s4
-; GFX9-NEXT:    v_mul_lo_u16_e32 v4, s6, v4
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v5, s7, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v6, s8, v6
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v7, s2, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_e32 v4, v4, v5
-; GFX9-NEXT:    v_or_b32_sdwa v5, v6, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
+; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX9-NEXT:    v_mov_b32_e32 v7, s0
+; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v8, s1
+; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s0, s2, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v9, s7
+; GFX9-NEXT:    s_lshr_b32 s1, s2, 28
+; GFX9-NEXT:    v_mov_b32_e32 v10, s4
+; GFX9-NEXT:    v_mul_lo_u16_e32 v7, s5, v7
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v8, s6, v8 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v3
+; GFX9-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX9-NEXT:    v_mul_lo_u16_e32 v9, s0, v9
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v10, s1, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v8, v9, v10 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v5, 0xffff, v7
+; GFX9-NEXT:    v_or_b32_e32 v4, v5, v8
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_add_u32_e32 v2, v3, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
+; GFX9-NEXT:    v_add_u32_e32 v2, v2, v6
+; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v4
+; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v3, 8, v4
 ; GFX9-NEXT:    v_add_u32_e32 v2, v2, v3
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc8_vecMul:

 ; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-DL-NEXT:    s_bfe_u32 s9, s2, 0x40008
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s7
 ; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x4000c
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v3, s0, v3
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v4, s8, v4 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v5, s9, v5
 ; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, s10, v6 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    s_bfe_u32 s0, s4, 0x40010
+; GFX9-DL-NEXT:    s_bfe_u32 s1, s4, 0x40014
 ; GFX9-DL-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v4, v5, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    s_bfe_u32 s1, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40018
-; GFX9-DL-NEXT:    s_bfe_u32 s0, s4, 0x40010
-; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x40018
 ; GFX9-DL-NEXT:    v_or_b32_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s0
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s1
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x40018
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s5
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s4
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v4, s6, v4
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v5, s7, v5 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v6, s8, v6
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v7, s2, v7 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_e32 v4, v4, v5
-; GFX9-DL-NEXT:    v_or_b32_sdwa v5, v6, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_or_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v3
+; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s0
+; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40010
+; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s1
+; GFX9-DL-NEXT:    s_bfe_u32 s6, s2, 0x40014
+; GFX9-DL-NEXT:    s_bfe_u32 s0, s2, 0x40018
+; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s7
+; GFX9-DL-NEXT:    s_lshr_b32 s1, s2, 28
+; GFX9-DL-NEXT:    v_mov_b32_e32 v10, s4
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, s5, v7
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v8, s6, v8 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v3
+; GFX9-DL-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v9, s0, v9
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v10, s1, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_or_b32_sdwa v8, v9, v10 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, 0xffff, v7
+; GFX9-DL-NEXT:    v_or_b32_e32 v4, v5, v8
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v3, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:BYTE_2
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v6
+; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v4
+; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v3, 8, v4
 ; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v3
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_3
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {

 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX8-NEXT:    flat_load_ubyte v2, v[0:1]
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    s_and_b32 s0, s2, 15
 ; GFX8-NEXT:    s_and_b32 s1, s4, 15
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX8-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX8-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX8-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX8-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX8-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX8-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX8-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX8-NEXT:    v_mov_b32_e32 v5, s7
-; GFX8-NEXT:    v_mov_b32_e32 v6, s6
-; GFX8-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX8-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX8-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX8-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX8-NEXT:    v_mov_b32_e32 v7, s9
-; GFX8-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX8-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX8-NEXT:    v_mov_b32_e32 v8, s11
-; GFX8-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX8-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX8-NEXT:    v_mov_b32_e32 v6, s7
+; GFX8-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX8-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX8-NEXT:    v_mov_b32_e32 v7, s8
+; GFX8-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX8-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX8-NEXT:    v_mov_b32_e32 v8, s9
+; GFX8-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX8-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX8-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX8-NEXT:    v_mov_b32_e32 v9, s13
+; GFX8-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX8-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, v5, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX8-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX8-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX8-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX8-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc4_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_and_b32 s0, s2, 15
 ; GFX9-NEXT:    s_and_b32 s1, s4, 15
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s1
 ; GFX9-NEXT:    s_bfe_u32 s5, s4, 0x40004
 ; GFX9-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
 ; GFX9-NEXT:    v_mov_b32_e32 v4, s5
 ; GFX9-NEXT:    s_bfe_u32 s1, s2, 0x40004
+; GFX9-NEXT:    s_bfe_u32 s7, s4, 0x4000c
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
 ; GFX9-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX9-NEXT:    v_mov_b32_e32 v5, s7
-; GFX9-NEXT:    v_mov_b32_e32 v6, s6
-; GFX9-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX9-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX9-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX9-NEXT:    v_mov_b32_e32 v7, s9
-; GFX9-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX9-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX9-NEXT:    v_mov_b32_e32 v8, s11
-; GFX9-NEXT:    s_bfe_u32 s14, s2, 0x40018
+; GFX9-NEXT:    s_bfe_u32 s8, s4, 0x40010
+; GFX9-NEXT:    v_mov_b32_e32 v6, s7
+; GFX9-NEXT:    s_bfe_u32 s6, s2, 0x4000c
+; GFX9-NEXT:    s_bfe_u32 s9, s4, 0x40014
+; GFX9-NEXT:    v_mov_b32_e32 v7, s8
+; GFX9-NEXT:    s_bfe_u32 s7, s2, 0x40010
+; GFX9-NEXT:    s_bfe_u32 s10, s4, 0x40018
+; GFX9-NEXT:    v_mov_b32_e32 v8, s9
+; GFX9-NEXT:    s_bfe_u32 s8, s2, 0x40014
+; GFX9-NEXT:    s_bfe_u32 s9, s2, 0x40018
 ; GFX9-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-NEXT:    v_mov_b32_e32 v9, s13
+; GFX9-NEXT:    v_mov_b32_e32 v9, s10
 ; GFX9-NEXT:    s_lshr_b32 s2, s2, 28
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX9-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s5, v5, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s6, v6, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s7, v7, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s8, v8, v2
+; GFX9-NEXT:    v_mad_u32_u24 v2, s9, v9, v2
 ; GFX9-NEXT:    v_mov_b32_e32 v3, s4
 ; GFX9-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
 ; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc4_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    s_load_dword s2, s[4:5], 0x0
 ; GFX9-DL-NEXT:    s_load_dword s4, s[6:7], 0x0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX9-DL-NEXT:    global_load_ubyte v2, v[0:1], off
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    s_and_b32 s0, s2, 15
-; GFX9-DL-NEXT:    s_and_b32 s1, s4, 15
-; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s1
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s4, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s6, s4, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s7, s4, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v4, s5
-; GFX9-DL-NEXT:    s_bfe_u32 s1, s2, 0x40004
-; GFX9-DL-NEXT:    s_bfe_u32 s5, s2, 0x40008
-; GFX9-DL-NEXT:    s_bfe_u32 s8, s2, 0x4000c
-; GFX9-DL-NEXT:    v_mov_b32_e32 v5, s7
-; GFX9-DL-NEXT:    v_mov_b32_e32 v6, s6
-; GFX9-DL-NEXT:    v_mul_u32_u24_e32 v5, s8, v5
-; GFX9-DL-NEXT:    s_bfe_u32 s9, s4, 0x40010
-; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
-; GFX9-DL-NEXT:    s_bfe_u32 s11, s4, 0x40014
-; GFX9-DL-NEXT:    s_bfe_u32 s10, s2, 0x40010
-; GFX9-DL-NEXT:    v_mov_b32_e32 v7, s9
-; GFX9-DL-NEXT:    s_bfe_u32 s13, s4, 0x40018
-; GFX9-DL-NEXT:    s_bfe_u32 s12, s2, 0x40014
-; GFX9-DL-NEXT:    v_mov_b32_e32 v8, s11
-; GFX9-DL-NEXT:    s_bfe_u32 s14, s2, 0x40018
-; GFX9-DL-NEXT:    s_lshr_b32 s4, s4, 28
-; GFX9-DL-NEXT:    v_mov_b32_e32 v9, s13
-; GFX9-DL-NEXT:    s_lshr_b32 s2, s2, 28
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s0, v3, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s1, v4, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s5, v6, v2
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX9-DL-NEXT:    v_add_u32_e32 v2, v2, v5
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s10, v7, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s12, v8, v2
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s14, v9, v2
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, s4
-; GFX9-DL-NEXT:    v_mad_u32_u24 v2, s2, v3, v2
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-DL-NEXT:    v_dot8_u32_u4 v2, s2, v3, v2
 ; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX9-DL-NEXT:    global_store_byte v[0:1], v2, off
 ; GFX9-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i4 addrspace(1)* nocapture %dst) {
 entry:
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %src1
   %vec2 = load <8 x i4>, <8 x i4> addrspace(1)* %src2

test/CodeGen/AMDGPU/sdiv.ll

 ; GCN-NEXT:    buffer_load_ushort v2, off, s[8:11], 0 offset:4
 ; GCN-NEXT:    s_waitcnt vmcnt(3)
 ; GCN-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GCN-NEXT:    s_waitcnt vmcnt(2)
 ; GCN-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; GCN-NEXT:    s_waitcnt vmcnt(1)
 ; GCN-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
-; GCN-NEXT:    v_or_b32_e32 v1, v2, v3
-; GCN-NEXT:    v_xor_b32_e32 v2, v0, v1
+; GCN-NEXT:    v_or_b32_e32 v2, v2, v3
+; GCN-NEXT:    v_xor_b32_e32 v1, v1, v3
+; GCN-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
 ; GCN-NEXT:    v_cvt_f32_i32_e32 v0, v0
-; GCN-NEXT:    v_cvt_f32_i32_e32 v1, v1
-; GCN-NEXT:    v_ashrrev_i32_e32 v2, 30, v2
-; GCN-NEXT:    v_rcp_iflag_f32_e32 v3, v1
-; GCN-NEXT:    v_or_b32_e32 v2, 1, v2
+; GCN-NEXT:    v_cvt_f32_i32_e32 v2, v2
+; GCN-NEXT:    v_or_b32_e32 v1, 1, v1
+; GCN-NEXT:    v_rcp_iflag_f32_e32 v3, v2
 ; GCN-NEXT:    v_mul_f32_e32 v3, v0, v3
 ; GCN-NEXT:    v_trunc_f32_e32 v3, v3
-; GCN-NEXT:    v_mad_f32 v0, -v3, v1, v0
+; GCN-NEXT:    v_mad_f32 v0, -v3, v2, v0
 ; GCN-NEXT:    v_cvt_i32_f32_e32 v3, v3
-; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v0|, |v1|
-; GCN-NEXT:    v_cndmask_b32_e32 v0, 0, v2, vcc
+; GCN-NEXT:    v_cmp_ge_f32_e64 vcc, |v0|, |v2|
+; GCN-NEXT:    v_cndmask_b32_e32 v0, 0, v1, vcc
 ; GCN-NEXT:    v_add_i32_e32 v0, vcc, v0, v3
 ; GCN-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; GCN-NEXT:    buffer_store_dword v0, off, s[4:7], 0
 ; GCN-NEXT:    s_endpgm
 ;
 ; TONGA-LABEL: v_sdiv_i24:
 ; TONGA:       ; %bb.0:
 ; TONGA-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24

 ; TONGA-NEXT:    buffer_load_ushort v1, off, s[4:7], 0 offset:4
 ; TONGA-NEXT:    buffer_load_sbyte v2, off, s[4:7], 0 offset:6
 ; TONGA-NEXT:    buffer_load_ushort v3, off, s[4:7], 0
 ; TONGA-NEXT:    s_waitcnt vmcnt(3)
 ; TONGA-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
 ; TONGA-NEXT:    s_waitcnt vmcnt(1)
 ; TONGA-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
 ; TONGA-NEXT:    v_or_b32_e32 v1, v1, v2
-; TONGA-NEXT:    v_cvt_f32_i32_e32 v2, v1
+; TONGA-NEXT:    v_cvt_f32_i32_e32 v1, v1
 ; TONGA-NEXT:    s_waitcnt vmcnt(0)
-; TONGA-NEXT:    v_or_b32_e32 v0, v3, v0
-; TONGA-NEXT:    v_cvt_f32_i32_e32 v3, v0
-; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v1
-; TONGA-NEXT:    v_rcp_iflag_f32_e32 v4, v2
+; TONGA-NEXT:    v_or_b32_e32 v3, v3, v0
+; TONGA-NEXT:    v_cvt_f32_i32_e32 v3, v3
+; TONGA-NEXT:    v_xor_b32_e32 v0, v0, v2
+; TONGA-NEXT:    v_rcp_iflag_f32_e32 v4, v1
 ; TONGA-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
 ; TONGA-NEXT:    v_or_b32_e32 v0, 1, v0
-; TONGA-NEXT:    v_mul_f32_e32 v1, v3, v4
-; TONGA-NEXT:    v_trunc_f32_e32 v1, v1
-; TONGA-NEXT:    v_mad_f32 v3, -v1, v2, v3
-; TONGA-NEXT:    v_cvt_i32_f32_e32 v1, v1
-; TONGA-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v2|
+; TONGA-NEXT:    v_mul_f32_e32 v2, v3, v4
+; TONGA-NEXT:    v_trunc_f32_e32 v2, v2
+; TONGA-NEXT:    v_mad_f32 v3, -v2, v1, v3
+; TONGA-NEXT:    v_cvt_i32_f32_e32 v2, v2
+; TONGA-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, |v1|
 ; TONGA-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
-; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v1
+; TONGA-NEXT:    v_add_u32_e32 v0, vcc, v0, v2
 ; TONGA-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; TONGA-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; TONGA-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: v_sdiv_i24:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_mov_b32 s3, 0xf000

 ; GFX9-NEXT:    buffer_load_ushort v2, off, s[4:7], 0 offset:4
 ; GFX9-NEXT:    buffer_load_sbyte v3, off, s[4:7], 0 offset:6
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
 ; GFX9-NEXT:    v_or_b32_e32 v0, v0, v1
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
 ; GFX9-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, v2
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, v0
-; GFX9-NEXT:    v_xor_b32_e32 v0, v0, v2
-; GFX9-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v3
-; GFX9-NEXT:    v_or_b32_e32 v0, 1, v0
-; GFX9-NEXT:    v_mul_f32_e32 v2, v1, v4
-; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v2
-; GFX9-NEXT:    v_mad_f32 v1, -v2, v3, v1
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v3|
-; GFX9-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v2, v2
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, v0
+; GFX9-NEXT:    v_xor_b32_e32 v1, v1, v3
+; GFX9-NEXT:    v_ashrrev_i32_e32 v1, 30, v1
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v2
+; GFX9-NEXT:    v_or_b32_e32 v1, 1, v1
+; GFX9-NEXT:    v_mul_f32_e32 v3, v0, v4
+; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v3
+; GFX9-NEXT:    v_mad_f32 v0, -v3, v2, v0
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v0|, |v2|
+; GFX9-NEXT:    v_cndmask_b32_e32 v0, 0, v1, vcc
 ; GFX9-NEXT:    v_add_u32_e32 v0, v4, v0
 ; GFX9-NEXT:    v_bfe_i32 v0, v0, 0, 24
 ; GFX9-NEXT:    buffer_store_dword v0, off, s[0:3], 0
 ; GFX9-NEXT:    s_endpgm
 ;
 ; EG-LABEL: v_sdiv_i24:
 ; EG:       ; %bb.0:
 ; EG-NEXT:    ALU 0, @14, KC0[CB0:0-32], KC1[]

test/CodeGen/SystemZ/store_nonbytesized_vecs.ll

; CHECK-NEXT:    br %r14
   store <8 x i31> %tmp, <8 x i31>* %p
   ret void
 }
 
 ; Load and store a <3 x i31> vector (test widening).
 define void @fun3(<3 x i31>* %src, <3 x i31>* %p)
 ; CHECK-LABEL: fun3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    llgf %r1, 0(%r2)
 ; CHECK-NEXT:    llgf %r0, 3(%r2)
-; CHECK-NEXT:    sllg %r4, %r1, 62
+; CHECK-NEXT:    llgf %r1, 6(%r2)
+; CHECK-NEXT:    llgf %r2, 0(%r2)
+; CHECK-NEXT:    rosbg %r1, %r0, 0, 32, 31
+; CHECK-NEXT:    sllg %r4, %r2, 62
 ; CHECK-NEXT:    rosbg %r4, %r0, 0, 32, 31
-; CHECK-NEXT:    llgf %r0, 6(%r2)
-; CHECK-NEXT:    ogr %r0, %r4
-; CHECK-NEXT:    st %r0, 8(%r3)
 ; CHECK-NEXT:    srlg %r0, %r4, 32
-; CHECK-NEXT:    sllg %r1, %r1, 30
+; CHECK-NEXT:    st %r1, 8(%r3)
+; CHECK-NEXT:    sllg %r1, %r2, 30
 ; CHECK-NEXT:    lr %r1, %r0
 ; CHECK-NEXT:    nihh %r1, 8191
 ; CHECK-NEXT:    stg %r1, 0(%r3)
 ; CHECK-NEXT:    br %r14
 {
   %tmp = load <3 x i31>, <3 x i31>* %src
   store <3 x i31> %tmp, <3 x i31>* %p
   ret void
 }

test/CodeGen/X86/2012-08-07-CmpISelBug.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-apple-macosx | FileCheck %s
 
 ; Cmp lowering should not look past the truncate unless the high bits are known
 ; zero.
 ; rdar://12027825
 
 define void @foo(i8 %arg4, i32 %arg5, i32* %arg14) nounwind {
 ; CHECK-LABEL: foo:
 ; CHECK:       ## %bb.0: ## %bb
+; CHECK-NEXT:    ## kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    andl $32, %edi
-; CHECK-NEXT:    orl $1601159181, %edi ## imm = 0x5F6FC00D
-; CHECK-NEXT:    andl %edi, %esi
-; CHECK-NEXT:    xorb $-14, %dil
-; CHECK-NEXT:    addb $82, %dil
-; CHECK-NEXT:    shrl $5, %esi
-; CHECK-NEXT:    movzbl %dil, %eax
-; CHECK-NEXT:    testb %sil, %sil
+; CHECK-NEXT:    leal 13(%rdi), %eax
+; CHECK-NEXT:    xorb $-14, %al
+; CHECK-NEXT:    addb $82, %al
+; CHECK-NEXT:    movzbl %al, %eax
+; CHECK-NEXT:    testl %esi, %edi
 ; CHECK-NEXT:    movl $1, %ecx
 ; CHECK-NEXT:    cmovnel %eax, %ecx
 ; CHECK-NEXT:    xorb $81, %cl
 ; CHECK-NEXT:    movzbl %cl, %eax
 ; CHECK-NEXT:    addl $2062143348, %eax ## imm = 0x7AE9CF74
 ; CHECK-NEXT:    movl %eax, (%rdx)
 ; CHECK-NEXT:    retq
 bb:

test/CodeGen/X86/vector-fshl-128.ll

 ; X32-SSE-NEXT:    retl
   %res = call <4 x i32> @llvm.fshl.v4i32(<4 x i32> %x, <4 x i32> %y, <4 x i32> %amt)
   ret <4 x i32> %res
 }
 
 define <8 x i16> @var_funnnel_v8i16(<8 x i16> %x, <8 x i16> %y, <8 x i16> %amt) nounwind {
 ; SSE2-LABEL: var_funnnel_v8i16:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    pand {{.*}}(%rip), %xmm2
 ; SSE2-NEXT:    movdqa {{.*#+}} xmm3 = [16,16,16,16,16,16,16,16]
 ; SSE2-NEXT:    psubw %xmm2, %xmm3
 ; SSE2-NEXT:    psllw $12, %xmm3
 ; SSE2-NEXT:    movdqa %xmm3, %xmm4
 ; SSE2-NEXT:    psraw $15, %xmm4
 ; SSE2-NEXT:    movdqa %xmm4, %xmm5
 ; SSE2-NEXT:    pandn %xmm1, %xmm5
 ; SSE2-NEXT:    psrlw $8, %xmm1

 ; SSE2-NEXT:    pand %xmm4, %xmm1
 ; SSE2-NEXT:    por %xmm5, %xmm1
 ; SSE2-NEXT:    paddw %xmm3, %xmm3
 ; SSE2-NEXT:    psraw $15, %xmm3
 ; SSE2-NEXT:    movdqa %xmm3, %xmm4
 ; SSE2-NEXT:    pandn %xmm1, %xmm4
 ; SSE2-NEXT:    psrlw $1, %xmm1
 ; SSE2-NEXT:    pand %xmm3, %xmm1
+; SSE2-NEXT:    pand {{.*}}(%rip), %xmm2
 ; SSE2-NEXT:    pxor %xmm3, %xmm3
 ; SSE2-NEXT:    movdqa %xmm2, %xmm5
 ; SSE2-NEXT:    punpckhwd {{.*#+}} xmm5 = xmm5[4],xmm3[4],xmm5[5],xmm3[5],xmm5[6],xmm3[6],xmm5[7],xmm3[7]
 ; SSE2-NEXT:    pslld $23, %xmm5
 ; SSE2-NEXT:    movdqa {{.*#+}} xmm6 = [1065353216,1065353216,1065353216,1065353216]
 ; SSE2-NEXT:    paddd %xmm6, %xmm5
 ; SSE2-NEXT:    cvttps2dq %xmm5, %xmm5
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm5 = xmm5[0,2,2,3,4,5,6,7]

 ; XOP-NEXT:    vpor %xmm1, %xmm3, %xmm1
 ; XOP-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; XOP-NEXT:    vpcomeqw %xmm3, %xmm2, %xmm2
 ; XOP-NEXT:    vpblendvb %xmm2, %xmm0, %xmm1, %xmm0
 ; XOP-NEXT:    retq
 ;
 ; X32-SSE-LABEL: var_funnnel_v8i16:
 ; X32-SSE:       # %bb.0:
-; X32-SSE-NEXT:    pand {{\.LCPI.*}}, %xmm2
 ; X32-SSE-NEXT:    movdqa {{.*#+}} xmm3 = [16,16,16,16,16,16,16,16]
 ; X32-SSE-NEXT:    psubw %xmm2, %xmm3
 ; X32-SSE-NEXT:    psllw $12, %xmm3
 ; X32-SSE-NEXT:    movdqa %xmm3, %xmm4
 ; X32-SSE-NEXT:    psraw $15, %xmm4
 ; X32-SSE-NEXT:    movdqa %xmm4, %xmm5
 ; X32-SSE-NEXT:    pandn %xmm1, %xmm5
 ; X32-SSE-NEXT:    psrlw $8, %xmm1

 ; X32-SSE-NEXT:    pand %xmm4, %xmm1
 ; X32-SSE-NEXT:    por %xmm5, %xmm1
 ; X32-SSE-NEXT:    paddw %xmm3, %xmm3
 ; X32-SSE-NEXT:    psraw $15, %xmm3
 ; X32-SSE-NEXT:    movdqa %xmm3, %xmm4
 ; X32-SSE-NEXT:    pandn %xmm1, %xmm4
 ; X32-SSE-NEXT:    psrlw $1, %xmm1
 ; X32-SSE-NEXT:    pand %xmm3, %xmm1
+; X32-SSE-NEXT:    pand {{\.LCPI.*}}, %xmm2
 ; X32-SSE-NEXT:    pxor %xmm3, %xmm3
 ; X32-SSE-NEXT:    movdqa %xmm2, %xmm5
 ; X32-SSE-NEXT:    punpckhwd {{.*#+}} xmm5 = xmm5[4],xmm3[4],xmm5[5],xmm3[5],xmm5[6],xmm3[6],xmm5[7],xmm3[7]
 ; X32-SSE-NEXT:    pslld $23, %xmm5
 ; X32-SSE-NEXT:    movdqa {{.*#+}} xmm6 = [1065353216,1065353216,1065353216,1065353216]
 ; X32-SSE-NEXT:    paddd %xmm6, %xmm5
 ; X32-SSE-NEXT:    cvttps2dq %xmm5, %xmm5
 ; X32-SSE-NEXT:    pshuflw {{.*#+}} xmm5 = xmm5[0,2,2,3,4,5,6,7]

test/CodeGen/X86/vector-reduce-mul-widen.ll

 ; AVX1-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vpextrb $0, %xmm0, %eax
 ; AVX1-NEXT:    # kill: def $al killed $al killed $eax
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v16i8:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
-; AVX2-NEXT:    vpmovzxbw {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX2-NEXT:    vpmovzxbw {{.*#+}} xmm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
-; AVX2-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpmovzxbw {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero,xmm0[8],zero,xmm0[9],zero,xmm0[10],zero,xmm0[11],zero,xmm0[12],zero,xmm0[13],zero,xmm0[14],zero,xmm0[15],zero
+; AVX2-NEXT:    vpmovzxbw {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero,xmm1[8],zero,xmm1[9],zero,xmm1[10],zero,xmm1[11],zero,xmm1[12],zero,xmm1[13],zero,xmm1[14],zero,xmm1[15],zero
+; AVX2-NEXT:    vpmullw %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    vpand {{.*}}(%rip), %ymm0, %ymm0
+; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm1
+; AVX2-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; AVX2-NEXT:    vpmovzxbw {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero,xmm1[8],zero,xmm1[9],zero,xmm1[10],zero,xmm1[11],zero,xmm1[12],zero,xmm1[13],zero,xmm1[14],zero,xmm1[15],zero
+; AVX2-NEXT:    vpmullw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm1 = [255,255,255,255,255,255,255,255]
-; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm2
-; AVX2-NEXT:    vpshufd {{.*#+}} xmm2 = xmm2[1,1,2,3]
-; AVX2-NEXT:    vpmovzxbw {{.*#+}} xmm2 = xmm2[0],zero,xmm2[1],zero,xmm2[2],zero,xmm2[3],zero,xmm2[4],zero,xmm2[5],zero,xmm2[6],zero,xmm2[7],zero
-; AVX2-NEXT:    vpmullw %xmm2, %xmm0, %xmm0
-; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm2
+; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm2
+; AVX2-NEXT:    vpackuswb %xmm0, %xmm2, %xmm2
 ; AVX2-NEXT:    vpsrld $16, %xmm2, %xmm2
 ; AVX2-NEXT:    vpmovzxbw {{.*#+}} xmm2 = xmm2[0],zero,xmm2[1],zero,xmm2[2],zero,xmm2[3],zero,xmm2[4],zero,xmm2[5],zero,xmm2[6],zero,xmm2[7],zero
 ; AVX2-NEXT:    vpmullw %xmm2, %xmm0, %xmm0
 ; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm1
 ; AVX2-NEXT:    vpsrlw $8, %xmm1, %xmm1
 ; AVX2-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpextrb $0, %xmm0, %eax
 ; AVX2-NEXT:    # kill: def $al killed $al killed $eax
+; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512BW-LABEL: test_v16i8:
 ; AVX512BW:       # %bb.0:
 ; AVX512BW-NEXT:    vpshufd {{.*#+}} xmm1 = xmm0[2,3,0,1]
 ; AVX512BW-NEXT:    vpmovzxbw {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero,xmm0[8],zero,xmm0[9],zero,xmm0[10],zero,xmm0[11],zero,xmm0[12],zero,xmm0[13],zero,xmm0[14],zero,xmm0[15],zero
 ; AVX512BW-NEXT:    vpmovzxbw {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero,xmm1[8],zero,xmm1[9],zero,xmm1[10],zero,xmm1[11],zero,xmm1[12],zero,xmm1[13],zero,xmm1[14],zero,xmm1[15],zero
 ; AVX512BW-NEXT:    vpmullw %ymm1, %ymm0, %ymm0

test/CodeGen/X86/vector-reduce-mul.ll

 ; AVX1-NEXT:    # kill: def $al killed $al killed $eax
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test_v16i8:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpunpckhbw {{.*#+}} xmm1 = xmm0[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
 ; AVX2-NEXT:    vpmovzxbw {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
 ; AVX2-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vmovdqa {{.*#+}} xmm1 = [255,255,255,255,255,255,255,255]
-; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpshufd {{.*#+}} xmm2 = xmm0[2,3,2,3]
-; AVX2-NEXT:    vpmullw %xmm2, %xmm0, %xmm0
-; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpshufd {{.*#+}} xmm2 = xmm0[1,1,2,3]
-; AVX2-NEXT:    vpmullw %xmm2, %xmm0, %xmm0
-; AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX2-NEXT:    vpshufb {{.*#+}} xmm1 = xmm0[8],zero,xmm0[10],zero,xmm0[12],zero,xmm0[14],zero,xmm0[u],zero,xmm0[u],zero,xmm0[u],zero,xmm0[u],zero
+; AVX2-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpshufb {{.*#+}} xmm1 = xmm0[4],zero,xmm0[6],zero,xmm0[u],zero,xmm0[u],zero,xmm0[u],zero,xmm0[u],zero,xmm0[u],zero,xmm0[u],zero
+; AVX2-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshuflw {{.*#+}} xmm1 = xmm0[1,1,2,3,4,5,6,7]
 ; AVX2-NEXT:    vpmullw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpextrb $0, %xmm0, %eax
 ; AVX2-NEXT:    # kill: def $al killed $al killed $eax
 ; AVX2-NEXT:    retq
 ;
 ; AVX512BW-LABEL: test_v16i8:
 ; AVX512BW:       # %bb.0: