[DAGCombiner] allow truncation of binops after legalization if desirable

[spatel]

Summary

This is an alternative to D58210 that achieves similar results for SystemZ. I'm proposing to extend the general (trunc (binop X, Y)) transform by using the existing TLI hook isTypeDesirableForOp() if we are post-legalization. This allows eliminating the similar fold in distributeTruncateThroughAnd() that is oddly constrained by starting the pattern match only from shift/rotate.

About the test diffs:

1. AMDGPU: 'widen-smrd-loads' is an improvement and 'idot' diffs are regressions?
2. PowerPC: neutral
3. SystemZ: improvements or neutral
4. x86: mostly neutral, improvements with 'shld/shrd', and regressions for 'vector-sext'.

I did look at the x86 'vector-sext' regressions, and the seemingly unnecessary 'movzbl' are being inserted by an isel pattern because:

// anyext. Define these to do an explicit zero-extend to
// avoid partial-register updates.

So that conflicts with the x86 setting that says 8-bit ops are desirable. Ideally, we would defer partial-reg optimizations to a later pass (and I know we already do this for some cases, so maybe that just needs to be adjusted a bit).

[arsenm]

IIRC the idot patterns are looking for very specific patterns and avoiding commuting to save compile time, so maybe those just need to be updated?

[jonpa]

I tried this patch on SystemZ / SPEC, and as before this seems to have a relatively very minor impact on the number of files changed (7), and on the performance (seemingly unaffected).

I think the SystemZ test looks good, but I leave the final approval to Uli as usual.

[uweigand]

In D58521#1409255, @jonpa wrote:

I tried this patch on SystemZ / SPEC, and as before this seems to have a relatively very minor impact on the number of files changed (7), and on the performance (seemingly unaffected).

I think the SystemZ test looks good, but I leave the final approval to Uli as usual.

Ah, the SystemZ test changes certainly LGTM, and would be a welcome improvement ...

[spatel]

Ping.

AMDGPU comment/update on the idot tests?

Not sure what we want to do about x86 (D58703), but it's probably not important enough to hold up the general improvement?

[rampitec]

In D58521#1406128, @arsenm wrote:

IIRC the idot patterns are looking for very specific patterns and avoiding commuting to save compile time, so maybe those just need to be updated?

They can be updated, but with the patch these tests produced considerably more code.

llvm/test/CodeGen/AMDGPU/widen-smrd-loads.ll

243

In fact this is regression as well. A scalar operation (s_) is preferable over vector (v_).

[RKSimon]

please can you rebase this?

[spatel]

Patch updated:
x86 vector sext is no longer a problem; AMDGPU appears to still show regressions.

[arsenm]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

12202

Due to the lack of register bank awareness, the AMDGPU answer depends on whether the source node is divergent. This needs to pass the node itself, not just the opcode and type to answer properly

[RKSimon]

@spatel Is this necessary any more?

[spatel]

In D58521#2655302, @RKSimon wrote:

@spatel Is this necessary any more?

The SystemZ code hasn't changed, so the code is still not ideal for that target. I'm not sure what we can do about the remaining regressions. I'll rebase, so we at least have an updated view of test diffs

[spatel]

Rebased (regenerated test diffs).

[lebedev.ri]

Re x86 - do we really want to narrow all the way to bytes?
I was under impression that wasn't a good idea.

[spatel]

In D58521#2656581, @lebedev.ri wrote:

Re x86 - do we really want to narrow all the way to bytes?
I was under impression that wasn't a good idea.

It's probably not a good idea in general, but the diffs I scanned here are all feeding into 8-bit op/store, so should be neutral.
This patch is trying to remove code and help SystemZ without hurting anything else, but maybe it's not worth the churn. We could probably add a TLI hook instead.

[RKSimon]

llvm/test/CodeGen/AMDGPU/cgp-bitfield-extract.ll

191

This file needs (manually) regenerating so the codegen diffs are more obvious

[RKSimon]

Needs rebasing again and cgp-bitfield-extract.ll cleaning up to show the diffs

[arsenm]

Is this still relevant?

[spatel]

In D58521#3822024, @arsenm wrote:

Is this still relevant?

Sorry - I lost track of this patch. It was trying to help SystemZ, but it got stuck on the regressions for AMDGPU, and I don't think there was a quick fix. I don't have immediate plans to work on this, so I'll abandon. Could try a SystemZ-specific transform instead?

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

bool tryStoreMergeOfLoads(SmallVectorImpl<MemOpLink> &StoreNodes,
                               SDNode *Root, bool AllowVectors,
                               bool IsNonTemporalStore, bool IsNonTemporalLoad);
 
     /// Merge consecutive store operations into a wide store.
     /// This optimization uses wide integers or vectors when possible.
     /// \return true if stores were merged.
     bool mergeConsecutiveStores(StoreSDNode *St);
 
-    /// Try to transform a truncation where C is a constant:
-    ///     (trunc (and X, C)) -> (and (trunc X), (trunc C))
-    ///
-    /// \p N needs to be a truncation and its first operand an AND. Other
-    /// requirements are checked by the function (e.g. that trunc is
-    /// single-use) and if missed an empty SDValue is returned.
-    SDValue distributeTruncateThroughAnd(SDNode *N);
-
     /// Helper function to determine whether the target supports operation
     /// given by \p Opcode for type \p VT, that is, whether the operation
     /// is legal or custom before legalizing operations, and whether is
     /// legal (but not custom) after legalization.
     bool hasOperation(unsigned Opcode, EVT VT) {
       return TLI.isOperationLegalOrCustom(Opcode, VT, LegalOperations);
     }
 
SDValue NewRHS = DAG.getNode(N->getOpcode(), DL, VT, LHS.getOperand(1),
                                N->getOperand(1));
   assert(isa<ConstantSDNode>(NewRHS) && "Folding was not successful!");
 
   SDValue NewShift = DAG.getNode(N->getOpcode(), DL, VT, LHS.getOperand(0),
                                  N->getOperand(1));
   return DAG.getNode(LHS.getOpcode(), DL, VT, NewShift, NewRHS);
 }
 
-SDValue DAGCombiner::distributeTruncateThroughAnd(SDNode *N) {
-  assert(N->getOpcode() == ISD::TRUNCATE);
-  assert(N->getOperand(0).getOpcode() == ISD::AND);
-
-  // (truncate:TruncVT (and N00, N01C)) -> (and (truncate:TruncVT N00), TruncC)
-  EVT TruncVT = N->getValueType(0);
-  if (N->hasOneUse() && N->getOperand(0).hasOneUse() &&
-      TLI.isTypeDesirableForOp(ISD::AND, TruncVT)) {
-    SDValue N01 = N->getOperand(0).getOperand(1);
-    if (isConstantOrConstantVector(N01, /* NoOpaques */ true)) {
-      SDLoc DL(N);
-      SDValue N00 = N->getOperand(0).getOperand(0);
-      SDValue Trunc00 = DAG.getNode(ISD::TRUNCATE, DL, TruncVT, N00);
-      SDValue Trunc01 = DAG.getNode(ISD::TRUNCATE, DL, TruncVT, N01);
-      AddToWorklist(Trunc00.getNode());
-      AddToWorklist(Trunc01.getNode());
-      return DAG.getNode(ISD::AND, DL, TruncVT, Trunc00, Trunc01);
-    }
-  }
-
-  return SDValue();
-}
-
 SDValue DAGCombiner::visitRotate(SDNode *N) {
   SDLoc dl(N);
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N->getValueType(0);
   unsigned Bitsize = VT.getScalarSizeInBits();
 
   // fold (rot x, 0) -> x
SDValue DAGCombiner::visitRotate(SDNode *N) {
   if (RotAmtC && RotAmtC->getAPIntValue() == 8 &&
       VT.getScalarSizeInBits() == 16 && hasOperation(ISD::BSWAP, VT))
     return DAG.getNode(ISD::BSWAP, dl, VT, N0);
 
   // Simplify the operands using demanded-bits information.
   if (SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
-  // fold (rot* x, (trunc (and y, c))) -> (rot* x, (and (trunc y), (trunc c))).
-  if (N1.getOpcode() == ISD::TRUNCATE &&
-      N1.getOperand(0).getOpcode() == ISD::AND) {
-    if (SDValue NewOp1 = distributeTruncateThroughAnd(N1.getNode()))
-      return DAG.getNode(N->getOpcode(), dl, VT, N0, NewOp1);
-  }
-
   unsigned NextOp = N0.getOpcode();
   // fold (rot* (rot* x, c2), c1) -> (rot* x, c1 +- c2 % bitsize)
   if (NextOp == ISD::ROTL || NextOp == ISD::ROTR) {
     SDNode *C1 = DAG.isConstantIntBuildVectorOrConstantInt(N1);
     SDNode *C2 = DAG.isConstantIntBuildVectorOrConstantInt(N0.getOperand(1));
     if (C1 && C2 && C1->getValueType(0) == C2->getValueType(0)) {
       EVT ShiftVT = C1->getValueType(0);
       bool SameSide = (N->getOpcode() == NextOp);
SDValue DAGCombiner::visitSHL(SDNode *N) {
   if (SDValue NewSel = foldBinOpIntoSelect(N))
     return NewSel;
 
   // if (shl x, c) is known to be zero, return 0
   if (DAG.MaskedValueIsZero(SDValue(N, 0),
                             APInt::getAllOnesValue(OpSizeInBits)))
     return DAG.getConstant(0, SDLoc(N), VT);
 
-  // fold (shl x, (trunc (and y, c))) -> (shl x, (and (trunc y), (trunc c))).
-  if (N1.getOpcode() == ISD::TRUNCATE &&
-      N1.getOperand(0).getOpcode() == ISD::AND) {
-    if (SDValue NewOp1 = distributeTruncateThroughAnd(N1.getNode()))
-      return DAG.getNode(ISD::SHL, SDLoc(N), VT, N0, NewOp1);
-  }
-
   if (SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
   // fold (shl (shl x, c1), c2) -> 0 or (shl x, (add c1, c2))
   if (N0.getOpcode() == ISD::SHL) {
     auto MatchOutOfRange = [OpSizeInBits](ConstantSDNode *LHS,
                                           ConstantSDNode *RHS) {
       APInt c1 = LHS->getAPIntValue();
if (ConstantSDNode *AddC = isConstOrConstSplat(N0.getOperand(1))) {
         SDValue ShiftC = DAG.getConstant(AddC->getAPIntValue().lshr(ShiftAmt).
                              trunc(TruncVT.getScalarSizeInBits()), DL, TruncVT);
         SDValue Add = DAG.getNode(ISD::ADD, DL, TruncVT, Trunc, ShiftC);
         return DAG.getSExtOrTrunc(Add, DL, VT);
       }
     }
   }
 
-  // fold (sra x, (trunc (and y, c))) -> (sra x, (and (trunc y), (trunc c))).
-  if (N1.getOpcode() == ISD::TRUNCATE &&
-      N1.getOperand(0).getOpcode() == ISD::AND) {
-    if (SDValue NewOp1 = distributeTruncateThroughAnd(N1.getNode()))
-      return DAG.getNode(ISD::SRA, SDLoc(N), VT, N0, NewOp1);
-  }
-
   // fold (sra (trunc (sra x, c1)), c2) -> (trunc (sra x, c1 + c2))
   // fold (sra (trunc (srl x, c1)), c2) -> (trunc (sra x, c1 + c2))
   //      if c1 is equal to the number of bits the trunc removes
   // TODO - support non-uniform vector shift amounts.
   if (N0.getOpcode() == ISD::TRUNCATE &&
       (N0.getOperand(0).getOpcode() == ISD::SRL ||
        N0.getOperand(0).getOpcode() == ISD::SRA) &&
       N0.getOperand(0).hasOneUse() &&
if (UnknownBits.isPowerOf2()) {
       }
 
       SDLoc DL(N);
       return DAG.getNode(ISD::XOR, DL, VT,
                          Op, DAG.getConstant(1, DL, VT));
     }
   }
 
-  // fold (srl x, (trunc (and y, c))) -> (srl x, (and (trunc y), (trunc c))).
-  if (N1.getOpcode() == ISD::TRUNCATE &&
-      N1.getOperand(0).getOpcode() == ISD::AND) {
-    if (SDValue NewOp1 = distributeTruncateThroughAnd(N1.getNode()))
-      return DAG.getNode(ISD::SRL, SDLoc(N), VT, N0, NewOp1);
-  }
-
   // fold operands of srl based on knowledge that the low bits are not
   // demanded.
   if (SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
   if (N1C && !N1C->isOpaque())
     if (SDValue NewSRL = visitShiftByConstant(N))
       return NewSRL;
SDValue DAGCombiner::visitTRUNCATE(SDNode *N) {
   // this transform.
   switch (N0.getOpcode()) {
   case ISD::ADD:
   case ISD::SUB:
   case ISD::MUL:
   case ISD::AND:
   case ISD::OR:
   case ISD::XOR:
-    if (!LegalOperations && N0.hasOneUse() &&
+    if ((!LegalOperations || TLI.isTypeDesirableForOp(N0.getOpcode(), VT)) && N0.hasOneUse() &&
         (isConstantOrConstantVector(N0.getOperand(0), true) ||
          isConstantOrConstantVector(N0.getOperand(1), true))) {
       // TODO: We already restricted this to pre-legalization, but for vectors
       // we are extra cautious to not create an unsupported operation.
       // Target-specific changes are likely needed to avoid regressions here.
       if (VT.isScalarInteger() || TLI.isOperationLegal(N0.getOpcode(), VT)) {
         SDLoc DL(N);
         SDValue NarrowL = DAG.getNode(ISD::TRUNCATE, DL, VT, N0.getOperand(0));

llvm/test/CodeGen/AArch64/fp16-v4-instructions.ll

; CHECK-COMMON-NEXT:  ret
   ret <4 x i16> %1
 }
 
 define <4 x i1> @test_fcmp_une(<4 x half> %a, <4 x half> %b) #0 {
 ; CHECK-CVT-LABEL: test_fcmp_une:
 ; CHECK-CVT: fcvtl
 ; CHECK-CVT: fcvtl
 ; CHECK-CVT: fcmeq
-; CHECK-CVT: mvn
 ; CHECK-CVT: xtn
+; CHECK-CVT: mvn
 ; CHECK-CVT: ret
 
 ; CHECK-FP16-LABEL: test_fcmp_une:
 ; CHECK-FP16-NOT:   fcvt
 ; CHECK-FP16:       fcmeq v{{[0-9]}}.4h, v{{[0-9]}}.4h
   %1 = fcmp une <4 x half> %a, %b
   ret <4 x i1> %1
 }

llvm/test/CodeGen/AArch64/srem-seteq-illegal-types.ll

 ; CHECK-NEXT:    fmov d3, x11
 ; CHECK-NEXT:    add x9, x9, x10
 ; CHECK-NEXT:    mov v3.d[1], x8
 ; CHECK-NEXT:    fmov d4, x9
 ; CHECK-NEXT:    and v4.16b, v4.16b, v0.16b
 ; CHECK-NEXT:    and v0.16b, v3.16b, v0.16b
 ; CHECK-NEXT:    cmeq v0.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmeq v1.2d, v4.2d, v2.2d
-; CHECK-NEXT:    mvn v0.16b, v0.16b
-; CHECK-NEXT:    mvn v1.16b, v1.16b
 ; CHECK-NEXT:    xtn v0.2s, v0.2d
 ; CHECK-NEXT:    xtn v1.2s, v1.2d
+; CHECK-NEXT:    mvn v0.8b, v0.8b
+; CHECK-NEXT:    mvn v1.8b, v1.8b
 ; CHECK-NEXT:    mov w1, v0.s[1]
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    fmov w2, s1
 ; CHECK-NEXT:    ret
   %srem = srem <3 x i33> %X, <i33 9, i33 9, i33 -9>
   %cmp = icmp ne <3 x i33> %srem, <i33 3, i33 -3, i33 3>
   ret <3 x i1> %cmp
 }

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-idiv.ll

-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: opt -S -mtriple=amdgcn-- -mcpu=tahiti -amdgpu-codegenprepare -amdgpu-bypass-slow-div=0 %s | FileCheck %s
 ; RUN: llc -mtriple=amdgcn-- -mcpu=tahiti -amdgpu-bypass-slow-div=0 < %s | FileCheck -check-prefix=GFX6 %s
 ; RUN: llc -mtriple=amdgcn-- -mcpu=gfx900 -amdgpu-bypass-slow-div=0 < %s | FileCheck -check-prefix=GFX9 %s
 
 define amdgpu_kernel void @udiv_i32(i32 addrspace(1)* %out, i32 %x, i32 %y) {
 ; CHECK-LABEL: @udiv_i32(
 ; CHECK-NEXT:    [[TMP1:%.*]] = uitofp i32 [[Y:%.*]] to float

 ; GFX9-LABEL: udiv_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX9-NEXT:    v_mov_b32_e32 v1, s4
+; GFX9-NEXT:    s_and_b32 s0, s6, s8
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, s0
 ; GFX9-NEXT:    s_and_b32 s0, s4, s8
-; GFX9-NEXT:    s_and_b32 s1, s6, s8
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, s1
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v4, s0
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v3, s0
 ; GFX9-NEXT:    s_bfe_u32 s0, s6, 0xf000f
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v0
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v6, s0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v1
-; GFX9-NEXT:    v_mov_b32_e32 v3, s6
-; GFX9-NEXT:    s_bfe_u32 s1, s4, 0xf000f
-; GFX9-NEXT:    v_alignbit_b32 v3, s7, v3, 30
-; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v7, s1
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
+; GFX9-NEXT:    s_bfe_u32 s0, s4, 0xf000f
+; GFX9-NEXT:    v_alignbit_b32 v5, s7, v5, 30
+; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v7, s0
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v8, v6
+; GFX9-NEXT:    v_and_b32_e32 v5, s8, v5
+; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
+; GFX9-NEXT:    v_mad_f32 v3, -v4, v0, v3
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v4, v4
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v5, v5
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, v0
+; GFX9-NEXT:    v_alignbit_b32 v1, s5, v1, 30
+; GFX9-NEXT:    v_mul_f32_e32 v0, v7, v8
+; GFX9-NEXT:    v_and_b32_e32 v1, s8, v1
+; GFX9-NEXT:    v_trunc_f32_e32 v0, v0
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v4, vcc
+; GFX9-NEXT:    v_mad_f32 v4, -v0, v6, v7
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v0, v0
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, v1
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v7, v5
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, v6
+; GFX9-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v0, vcc
+; GFX9-NEXT:    v_and_b32_e32 v4, s8, v4
+; GFX9-NEXT:    v_mul_f32_e32 v0, v1, v7
+; GFX9-NEXT:    v_trunc_f32_e32 v0, v0
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v6, v0
+; GFX9-NEXT:    v_mad_f32 v0, -v0, v5, v1
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v0|, v5
 ; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
-; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
-; GFX9-NEXT:    v_mad_f32 v4, -v5, v1, v4
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v3, v3
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, v1
-; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
-; GFX9-NEXT:    v_mul_f32_e32 v1, v7, v8
-; GFX9-NEXT:    v_and_b32_e32 v0, s8, v0
-; GFX9-NEXT:    v_trunc_f32_e32 v1, v1
-; GFX9-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v5, vcc
-; GFX9-NEXT:    v_mad_f32 v5, -v1, v6, v7
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v1, v1
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v0, v0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v7, v3
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v5|, v6
-; GFX9-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v1, vcc
-; GFX9-NEXT:    v_mul_f32_e32 v1, v0, v7
-; GFX9-NEXT:    v_trunc_f32_e32 v1, v1
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v6, v1
-; GFX9-NEXT:    v_mad_f32 v0, -v1, v3, v0
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v0|, v3
-; GFX9-NEXT:    v_and_b32_e32 v3, s8, v4
 ; GFX9-NEXT:    v_addc_co_u32_e32 v0, vcc, 0, v6, vcc
-; GFX9-NEXT:    v_and_b32_e32 v4, s8, v5
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 15, v4
 ; GFX9-NEXT:    v_lshlrev_b64 v[0:1], 30, v[0:1]
 ; GFX9-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX9-NEXT:    v_or_b32_e32 v0, v3, v0
 ; GFX9-NEXT:    global_store_dword v2, v0, s[2:3]
 ; GFX9-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX9-NEXT:    global_store_short v2, v0, s[2:3] offset:4
 ; GFX9-NEXT:    s_endpgm

 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    v_mov_b32_e32 v0, s4
-; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
-; GFX9-NEXT:    s_and_b32 s5, s6, s8
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, s5
+; GFX9-NEXT:    s_and_b32 s0, s6, s8
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v1, s0
 ; GFX9-NEXT:    s_and_b32 s0, s4, s8
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v4, s0
-; GFX9-NEXT:    s_bfe_u32 s5, s6, 0xf000f
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v1
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v6, s5
-; GFX9-NEXT:    v_mov_b32_e32 v3, s6
-; GFX9-NEXT:    v_alignbit_b32 v3, s7, v3, 30
-; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
-; GFX9-NEXT:    v_mad_f32 v4, -v5, v1, v4
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v5, v5
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, v1
+; GFX9-NEXT:    s_bfe_u32 s1, s6, 0xf000f
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v6, s1
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v3, s0
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v1
 ; GFX9-NEXT:    s_bfe_u32 s1, s4, 0xf000f
-; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
-; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v5, vcc
-; GFX9-NEXT:    v_cvt_f32_u32_e32 v5, v3
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v7, s1
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v8, v6
+; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
+; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
+; GFX9-NEXT:    v_mad_f32 v3, -v4, v1, v3
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v4, v4
+; GFX9-NEXT:    v_mov_b32_e32 v5, s6
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v3|, v1
+; GFX9-NEXT:    v_alignbit_b32 v5, s7, v5, 30
+; GFX9-NEXT:    v_mul_f32_e32 v3, v7, v8
+; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
+; GFX9-NEXT:    v_and_b32_e32 v5, s8, v5
+; GFX9-NEXT:    v_addc_co_u32_e32 v1, vcc, 0, v4, vcc
+; GFX9-NEXT:    v_mad_f32 v4, -v3, v6, v7
+; GFX9-NEXT:    v_cvt_f32_u32_e32 v7, v5
+; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
 ; GFX9-NEXT:    v_and_b32_e32 v0, s8, v0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v9, v5
-; GFX9-NEXT:    s_lshr_b32 s0, s6, 15
-; GFX9-NEXT:    v_mul_f32_e32 v4, v7, v8
 ; GFX9-NEXT:    v_cvt_f32_u32_e32 v8, v0
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v9, v7
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, v6
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v3, v3
+; GFX9-NEXT:    s_lshr_b32 s0, s6, 15
+; GFX9-NEXT:    v_mul_f32_e32 v4, v8, v9
 ; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
-; GFX9-NEXT:    v_mad_f32 v7, -v4, v6, v7
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v7|, v6
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v4, v4
-; GFX9-NEXT:    v_mul_f32_e32 v6, v8, v9
-; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
-; GFX9-NEXT:    v_cvt_u32_f32_e32 v7, v6
-; GFX9-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v4, vcc
-; GFX9-NEXT:    v_mad_f32 v6, -v6, v5, v8
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v6|, v5
-; GFX9-NEXT:    v_mul_lo_u32 v4, v4, s0
-; GFX9-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v7, vcc
-; GFX9-NEXT:    v_mul_lo_u32 v3, v5, v3
+; GFX9-NEXT:    v_cvt_u32_f32_e32 v6, v4
+; GFX9-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
+; GFX9-NEXT:    v_mad_f32 v4, -v4, v7, v8
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v4|, v7
+; GFX9-NEXT:    v_addc_co_u32_e32 v4, vcc, 0, v6, vcc
+; GFX9-NEXT:    v_mul_lo_u32 v3, v3, s0
+; GFX9-NEXT:    v_mul_lo_u32 v4, v4, v5
 ; GFX9-NEXT:    v_mul_lo_u32 v1, v1, s6
 ; GFX9-NEXT:    s_lshr_b32 s0, s4, 15
-; GFX9-NEXT:    v_sub_u32_e32 v4, s0, v4
-; GFX9-NEXT:    v_and_b32_e32 v4, s8, v4
+; GFX9-NEXT:    v_sub_u32_e32 v3, s0, v3
+; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
 ; GFX9-NEXT:    v_sub_u32_e32 v5, s4, v1
-; GFX9-NEXT:    v_sub_u32_e32 v0, v0, v3
+; GFX9-NEXT:    v_sub_u32_e32 v0, v0, v4
 ; GFX9-NEXT:    v_lshlrev_b64 v[0:1], 30, v[0:1]
-; GFX9-NEXT:    v_and_b32_e32 v3, s8, v5
-; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 15, v4
-; GFX9-NEXT:    v_or_b32_e32 v3, v3, v4
+; GFX9-NEXT:    v_and_b32_e32 v4, s8, v5
+; GFX9-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
+; GFX9-NEXT:    v_or_b32_e32 v3, v4, v3
 ; GFX9-NEXT:    v_or_b32_e32 v0, v3, v0
 ; GFX9-NEXT:    global_store_dword v2, v0, s[2:3]
 ; GFX9-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX9-NEXT:    global_store_short v2, v0, s[2:3] offset:4
 ; GFX9-NEXT:    s_endpgm
   %r = urem <3 x i15> %x, %y
   store <3 x i15> %r, <3 x i15> addrspace(1)* %out
   ret void

 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    v_mov_b32_e32 v2, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    s_bfe_i32 s1, s4, 0xf0000
 ; GFX9-NEXT:    s_bfe_i32 s0, s6, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s0
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s1
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, s1
 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
-; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
-; GFX9-NEXT:    s_or_b32 s5, s0, 1
-; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
-; GFX9-NEXT:    v_mad_f32 v4, -v5, v3, v4
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v0
+; GFX9-NEXT:    s_or_b32 s8, s0, 1
+; GFX9-NEXT:    v_mul_f32_e32 v3, v1, v3
+; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
+; GFX9-NEXT:    v_mad_f32 v1, -v3, v0, v1
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
 ; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
-; GFX9-NEXT:    s_cselect_b32 s0, s5, 0
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
+; GFX9-NEXT:    s_cselect_b32 s0, s8, 0
 ; GFX9-NEXT:    s_bfe_i32 s1, s6, 0xf000f
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s1
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s1
+; GFX9-NEXT:    s_bfe_i32 s8, s4, 0xf000f
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, s8
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v0
+; GFX9-NEXT:    v_add_u32_e32 v3, s0, v3
+; GFX9-NEXT:    v_mul_f32_e32 v4, v1, v4
+; GFX9-NEXT:    s_xor_b32 s0, s8, s1
+; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
+; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
+; GFX9-NEXT:    v_mad_f32 v1, -v4, v0, v1
+; GFX9-NEXT:    s_or_b32 s8, s0, 1
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
 ; GFX9-NEXT:    v_mov_b32_e32 v1, s6
-; GFX9-NEXT:    v_add_u32_e32 v4, s0, v5
-; GFX9-NEXT:    s_bfe_i32 s0, s4, 0xf000f
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, s0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v6, v3
 ; GFX9-NEXT:    v_alignbit_b32 v1, s7, v1, 30
-; GFX9-NEXT:    s_xor_b32 s0, s0, s1
-; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
-; GFX9-NEXT:    v_mul_f32_e32 v6, v5, v6
-; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
-; GFX9-NEXT:    v_mad_f32 v5, -v6, v3, v5
 ; GFX9-NEXT:    v_bfe_i32 v1, v1, 0, 15
-; GFX9-NEXT:    s_or_b32 s4, s0, 1
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v5|, |v3|
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, v1
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v6, v6
-; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
-; GFX9-NEXT:    s_cselect_b32 s0, s4, 0
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, v1
+; GFX9-NEXT:    v_mov_b32_e32 v0, s4
+; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
 ; GFX9-NEXT:    v_bfe_i32 v0, v0, 0, 15
-; GFX9-NEXT:    v_add_u32_e32 v5, s0, v6
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v6, v0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v7, v3
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v7, v5
 ; GFX9-NEXT:    v_xor_b32_e32 v0, v0, v1
-; GFX9-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
-; GFX9-NEXT:    v_or_b32_e32 v0, 1, v0
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
+; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
 ; GFX9-NEXT:    v_mul_f32_e32 v1, v6, v7
 ; GFX9-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v7, v1
-; GFX9-NEXT:    v_mad_f32 v1, -v1, v3, v6
-; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v3|
+; GFX9-NEXT:    v_ashrrev_i32_e32 v0, 30, v0
+; GFX9-NEXT:    v_mad_f32 v1, -v1, v5, v6
+; GFX9-NEXT:    s_cselect_b32 s0, s8, 0
+; GFX9-NEXT:    v_or_b32_e32 v0, 1, v0
+; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v1|, |v5|
 ; GFX9-NEXT:    v_cndmask_b32_e32 v0, 0, v0, vcc
+; GFX9-NEXT:    v_add_u32_e32 v4, s0, v4
 ; GFX9-NEXT:    s_movk_i32 s0, 0x7fff
 ; GFX9-NEXT:    v_add_u32_e32 v0, v7, v0
-; GFX9-NEXT:    v_and_b32_e32 v3, s0, v4
-; GFX9-NEXT:    v_and_b32_e32 v4, s0, v5
-; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 15, v4
+; GFX9-NEXT:    v_and_b32_e32 v4, s0, v4
 ; GFX9-NEXT:    v_lshlrev_b64 v[0:1], 30, v[0:1]
+; GFX9-NEXT:    v_and_b32_e32 v3, s0, v3
+; GFX9-NEXT:    v_lshlrev_b32_e32 v4, 15, v4
 ; GFX9-NEXT:    v_or_b32_e32 v3, v3, v4
 ; GFX9-NEXT:    v_or_b32_e32 v0, v3, v0
 ; GFX9-NEXT:    global_store_dword v2, v0, s[2:3]
 ; GFX9-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX9-NEXT:    global_store_short v2, v0, s[2:3] offset:4
 ; GFX9-NEXT:    s_endpgm
   %r = sdiv <3 x i15> %x, %y
   store <3 x i15> %r, <3 x i15> addrspace(1)* %out

 ;
 ; GFX9-LABEL: srem_v3i15:
 ; GFX9:       ; %bb.0:
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x2c
 ; GFX9-NEXT:    s_load_dwordx2 s[6:7], s[0:1], 0x34
 ; GFX9-NEXT:    s_movk_i32 s8, 0x7fff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    s_and_b32 s0, s4, s8
-; GFX9-NEXT:    s_and_b32 s1, s6, s8
-; GFX9-NEXT:    s_bfe_i32 s1, s1, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v2, s1
+; GFX9-NEXT:    s_and_b32 s1, s4, s8
+; GFX9-NEXT:    s_and_b32 s0, s6, s8
 ; GFX9-NEXT:    s_bfe_i32 s0, s0, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s0
-; GFX9-NEXT:    s_xor_b32 s0, s0, s1
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v4, v2
-; GFX9-NEXT:    v_mov_b32_e32 v0, s4
-; GFX9-NEXT:    v_mov_b32_e32 v1, s6
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v0, s0
+; GFX9-NEXT:    s_bfe_i32 s1, s1, 0xf0000
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, s1
+; GFX9-NEXT:    s_xor_b32 s0, s1, s0
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v2, v0
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
-; GFX9-NEXT:    v_mul_f32_e32 v4, v3, v4
-; GFX9-NEXT:    v_trunc_f32_e32 v4, v4
-; GFX9-NEXT:    v_mad_f32 v3, -v4, v2, v3
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v4, v4
-; GFX9-NEXT:    v_alignbit_b32 v0, s5, v0, 30
-; GFX9-NEXT:    v_alignbit_b32 v1, s7, v1, 30
-; GFX9-NEXT:    s_or_b32 s11, s0, 1
 ; GFX9-NEXT:    s_lshr_b32 s9, s4, 15
-; GFX9-NEXT:    s_bfe_u32 s5, s4, 0xf000f
-; GFX9-NEXT:    s_lshr_b32 s7, s6, 15
-; GFX9-NEXT:    s_bfe_u32 s10, s6, 0xf000f
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v3|, |v2|
+; GFX9-NEXT:    s_lshr_b32 s10, s6, 15
+; GFX9-NEXT:    v_mul_f32_e32 v2, v1, v2
+; GFX9-NEXT:    v_trunc_f32_e32 v2, v2
+; GFX9-NEXT:    v_mad_f32 v1, -v2, v0, v1
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v2, v2
+; GFX9-NEXT:    s_or_b32 s11, s0, 1
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v1|, |v0|
 ; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
 ; GFX9-NEXT:    s_cselect_b32 s0, s11, 0
-; GFX9-NEXT:    v_add_u32_e32 v2, s0, v4
-; GFX9-NEXT:    s_bfe_i32 s0, s10, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v3, s0
-; GFX9-NEXT:    s_bfe_i32 s1, s5, 0xf0000
-; GFX9-NEXT:    v_cvt_f32_i32_e32 v4, s1
+; GFX9-NEXT:    v_add_u32_e32 v0, s0, v2
+; GFX9-NEXT:    s_bfe_u32 s0, s6, 0xf000f
+; GFX9-NEXT:    s_bfe_i32 s0, s0, 0xf0000
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v1, s0
+; GFX9-NEXT:    s_bfe_u32 s1, s4, 0xf000f
+; GFX9-NEXT:    s_bfe_i32 s1, s1, 0xf0000
+; GFX9-NEXT:    v_cvt_f32_i32_e32 v2, s1
+; GFX9-NEXT:    v_rcp_iflag_f32_e32 v3, v1
 ; GFX9-NEXT:    s_xor_b32 s0, s1, s0
-; GFX9-NEXT:    v_rcp_iflag_f32_e32 v5, v3
 ; GFX9-NEXT:    s_ashr_i32 s0, s0, 30
-; GFX9-NEXT:    s_or_b32 s5, s0, 1
-; GFX9-NEXT:    v_and_b32_e32 v1, s8, v1
-; GFX9-NEXT:    v_mul_f32_e32 v5, v4, v5
-; GFX9-NEXT:    v_trunc_f32_e32 v5, v5
-; GFX9-NEXT:    v_mad_f32 v4, -v5, v3, v4
-; GFX9-NEXT:    v_cvt_i32_f32_e32 v5, v5
-; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v4|, |v3|
+; GFX9-NEXT:    s_or_b32 s11, s0, 1
+; GFX9-NEXT:    v_mul_f32_e32 v3, v2, v3
+; GFX9-NEXT:    v_trunc_f32_e32 v3, v3
+; GFX9-NEXT:    v_mad_f32 v2, -v3, v1, v2
+; GFX9-NEXT:    v_cvt_i32_f32_e32 v3, v3
+; GFX9-NEXT:    v_cmp_ge_f32_e64 s[0:1], |v2|, |v1|
 ; GFX9-NEXT:    s_cmp_lg_u64 s[0:1], 0
-; GFX9-NEXT:    s_cselect_b32 s0, s5, 0
-; GFX9-NEXT:    v_bfe_i32 v4, v1, 0, 15
-; GFX9-NEXT:    v_add_u32_e32 v3, s0, v5
+; GFX9-NEXT:    s_cselect_b32 s0, s11, 0
+; GFX9-NEXT:    v_add_u32_e32 v1, s0, v3
+; GFX9-NEXT:    v_mov_b32_e32 v3, s6
+; GFX9-NEXT:    v_alignbit_b32 v3, s7, v3, 30
+; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
+; GFX9-NEXT:    v_bfe_i32 v4, v3, 0, 15
+; GFX9-NEXT:    v_mov_b32_e32 v2, s4
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v5, v4
-; GFX9-NEXT:    v_and_b32_e32 v0, s8, v0
-; GFX9-NEXT:    v_bfe_i32 v6, v0, 0, 15
+; GFX9-NEXT:    v_alignbit_b32 v2, s5, v2, 30
+; GFX9-NEXT:    v_and_b32_e32 v2, s8, v2
+; GFX9-NEXT:    v_bfe_i32 v6, v2, 0, 15
 ; GFX9-NEXT:    v_cvt_f32_i32_e32 v7, v6
 ; GFX9-NEXT:    v_rcp_iflag_f32_e32 v8, v5
 ; GFX9-NEXT:    v_xor_b32_e32 v4, v6, v4
 ; GFX9-NEXT:    v_ashrrev_i32_e32 v4, 30, v4
 ; GFX9-NEXT:    v_or_b32_e32 v4, 1, v4
 ; GFX9-NEXT:    v_mul_f32_e32 v6, v7, v8
 ; GFX9-NEXT:    v_trunc_f32_e32 v6, v6
 ; GFX9-NEXT:    v_cvt_i32_f32_e32 v8, v6
 ; GFX9-NEXT:    v_mad_f32 v6, -v6, v5, v7
 ; GFX9-NEXT:    v_cmp_ge_f32_e64 vcc, |v6|, |v5|
 ; GFX9-NEXT:    v_cndmask_b32_e32 v4, 0, v4, vcc
-; GFX9-NEXT:    v_mul_lo_u32 v3, v3, s7
+; GFX9-NEXT:    v_mul_lo_u32 v1, v1, s10
 ; GFX9-NEXT:    v_add_u32_e32 v4, v8, v4
-; GFX9-NEXT:    v_mul_lo_u32 v2, v2, s6
-; GFX9-NEXT:    v_mul_lo_u32 v1, v4, v1
-; GFX9-NEXT:    v_sub_u32_e32 v3, s9, v3
-; GFX9-NEXT:    v_and_b32_e32 v3, s8, v3
-; GFX9-NEXT:    v_sub_u32_e32 v2, s4, v2
-; GFX9-NEXT:    v_sub_u32_e32 v0, v0, v1
+; GFX9-NEXT:    v_mul_lo_u32 v0, v0, s6
+; GFX9-NEXT:    v_mul_lo_u32 v3, v4, v3
+; GFX9-NEXT:    v_sub_u32_e32 v6, s9, v1
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0
+; GFX9-NEXT:    v_sub_u32_e32 v5, s4, v0
+; GFX9-NEXT:    v_sub_u32_e32 v0, v2, v3
+; GFX9-NEXT:    v_and_b32_e32 v3, s8, v6
 ; GFX9-NEXT:    v_lshlrev_b64 v[0:1], 30, v[0:1]
-; GFX9-NEXT:    v_and_b32_e32 v2, s8, v2
+; GFX9-NEXT:    v_and_b32_e32 v2, s8, v5
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v3, 15, v3
 ; GFX9-NEXT:    v_or_b32_e32 v2, v2, v3
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
 ; GFX9-NEXT:    v_or_b32_e32 v0, v2, v0
 ; GFX9-NEXT:    global_store_dword v4, v0, s[2:3]
 ; GFX9-NEXT:    v_and_b32_e32 v0, 0x1fff, v1
 ; GFX9-NEXT:    global_store_short v4, v0, s[2:3] offset:4
 ; GFX9-NEXT:    s_endpgm
   %r = srem <3 x i15> %x, %y
   store <3 x i15> %r, <3 x i15> addrspace(1)* %out
   ret void

llvm/test/CodeGen/AMDGPU/cgp-bitfield-extract.ll

 ; OPT: br label
 
 ; OPT: ret:
 ; OPT: store
 ; OPT: ret
 
 ; For GFX8: since i16 is legal type, we cannot sink lshr into BBs.
 
-; GCN-LABEL: {{^}}sink_ubfe_i16:
-; GCN-NOT: lshr
-; VI: s_load_dword [[ARG:s[0-9]+]], s[0:1], 0x2c
-; VI: s_bfe_u32 [[BFE:s[0-9]+]], [[ARG]], 0xc0004
-; GCN: s_cbranch_scc{{[0-1]}}
-
-; SI: s_bfe_u32 s{{[0-9]+}}, s{{[0-9]+}}, 0x70004
-; VI: v_mov_b32_e32 v{{[0-9]+}}, 0x7f
-
-; GCN: BB2_3:
-; SI: s_bfe_u32 s{{[0-9]+}}, s{{[0-9]+}}, 0x80004
-; VI: v_mov_b32_e32 v{{[0-9]+}}, 0xff
-
-; GCN: buffer_store_short
-; GCN: s_endpgm
 define amdgpu_kernel void @sink_ubfe_i16(i16 addrspace(1)* %out, i16 %arg1) #0 {
+; SI-LABEL: sink_ubfe_i16:
+; SI:       ; %bb.0: ; %entry
+; SI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x9
+; SI-NEXT:    s_load_dword s1, s[0:1], 0xb
+; SI-NEXT:    s_cbranch_scc0 BB2_2
+; SI-NEXT:  ; %bb.1: ; %bb1
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_bfe_u32 s0, s1, 0x70004
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    buffer_store_short v0, off, s[0:3], 0
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:    s_mov_b64 vcc, exec
+; SI-NEXT:    s_cbranch_execz BB2_3
+; SI-NEXT:    s_branch BB2_4
+; SI-NEXT:  BB2_2:
+; SI-NEXT:    s_mov_b64 s[2:3], -1
+; SI-NEXT:    s_mov_b32 s0, 0
+; SI-NEXT:    s_andn2_b64 vcc, exec, s[2:3]
+; SI-NEXT:    s_waitcnt lgkmcnt(0)
+; SI-NEXT:    s_mov_b64 vcc, vcc
+; SI-NEXT:    s_cbranch_vccnz BB2_4
+; SI-NEXT:  BB2_3: ; %bb0
+; SI-NEXT:    s_bfe_u32 s0, s1, 0x80004
+; SI-NEXT:    s_mov_b32 s3, 0xf000
+; SI-NEXT:    s_mov_b32 s2, -1
+; SI-NEXT:    s_waitcnt expcnt(0)
+; SI-NEXT:    v_mov_b32_e32 v0, 0
+; SI-NEXT:    buffer_store_short v0, off, s[0:3], 0
+; SI-NEXT:    s_waitcnt vmcnt(0)
+; SI-NEXT:  BB2_4: ; %ret
+; SI-NEXT:    s_mov_b32 s7, 0xf000
+; SI-NEXT:    s_mov_b32 s6, -1
+; SI-NEXT:    s_waitcnt expcnt(0)
+; SI-NEXT:    v_mov_b32_e32 v0, s0
+; SI-NEXT:    buffer_store_short v0, off, s[4:7], 0
+; SI-NEXT:    s_endpgm
+;
+; VI-LABEL: sink_ubfe_i16:
+; VI:       ; %bb.0: ; %entry
+; VI-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x24
+; VI-NEXT:    s_load_dword s0, s[0:1], 0x2c
+; VI-NEXT:    v_mov_b32_e32 v0, 0xfff
+; VI-NEXT:    s_waitcnt lgkmcnt(0)
+; VI-NEXT:    s_lshr_b32 s0, s0, 4
+; VI-NEXT:    v_and_b32_e32 v1, s0, v0
+; VI-NEXT:    s_cbranch_scc0 BB2_2
+; VI-NEXT:  ; %bb.1: ; %bb1
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    v_mov_b32_e32 v2, 0
+; VI-NEXT:    buffer_store_short v2, off, s[0:3], 0
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:    v_and_b32_e32 v0, 0x7f, v1
+; VI-NEXT:    s_cbranch_execz BB2_3
+; VI-NEXT:    s_branch BB2_4
+; VI-NEXT:  BB2_2:
+; VI-NEXT:    ; implicit-def: $vgpr0
+; VI-NEXT:  BB2_3: ; %bb0
+; VI-NEXT:    v_and_b32_e32 v0, 0xff, v1
+; VI-NEXT:    s_mov_b32 s3, 0xf000
+; VI-NEXT:    s_mov_b32 s2, -1
+; VI-NEXT:    v_mov_b32_e32 v1, 0
+; VI-NEXT:    buffer_store_short v1, off, s[0:3], 0
+; VI-NEXT:    s_waitcnt vmcnt(0)
+; VI-NEXT:  BB2_4: ; %ret
+; VI-NEXT:    s_mov_b32 s7, 0xf000
+; VI-NEXT:    s_mov_b32 s6, -1
+; VI-NEXT:    buffer_store_short v0, off, s[4:7], 0
+; VI-NEXT:    s_endpgm
 entry:
   %shr = lshr i16 %arg1, 4
   br i1 undef, label %bb0, label %bb1
 
 bb0:
   %val0 = and i16 %shr, 255
   store volatile i16 0, i16 addrspace(1)* undef
   br label %ret

llvm/test/CodeGen/AMDGPU/idot8s.ll

 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0xffff
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; GFX9-NEXT:    v_mov_b32_e32 v5, 15
+; GFX9-NEXT:    v_mov_b32_e32 v4, 0xffff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_load_dword v3, v0, s[4:5]
-; GFX9-NEXT:    global_load_dword v4, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v1, v0, s[4:5]
+; GFX9-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    global_load_ushort v3, v0, s[2:3]
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(2)
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 20, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 12, v1
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_and_b32_e32 v10, 15, v3
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_and_b32_e32 v17, 15, v4
-; GFX9-NEXT:    v_bfe_u32 v0, v3, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v3, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v3, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v13, v4, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v4, 8, 4
-; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 28, v3
-; GFX9-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v9, v3, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v3, v3, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v10, v2, v10
-; GFX9-NEXT:    v_bfe_u32 v11, v4, 24, 4
-; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 28, v4
-; GFX9-NEXT:    v_bfe_u32 v14, v4, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v16, v4, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v4, v4, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v17, v2, v17
-; GFX9-NEXT:    v_lshl_or_b32 v3, v3, 16, v10
-; GFX9-NEXT:    v_lshl_or_b32 v4, v4, 16, v17
-; GFX9-NEXT:    v_pk_lshlrev_b16 v3, 12, v3 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_lshlrev_b16 v4, 12, v4 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_ashrrev_i16 v3, 12, v3 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_ashrrev_i16 v4, 12, v4 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_mul_lo_u16 v3, v3, v4
-; GFX9-NEXT:    global_load_ushort v4, v1, s[2:3]
-; GFX9-NEXT:    v_and_b32_e32 v8, v2, v8
-; GFX9-NEXT:    v_and_b32_e32 v0, v2, v0
-; GFX9-NEXT:    v_and_b32_e32 v15, v2, v15
-; GFX9-NEXT:    v_lshl_or_b32 v0, v5, 16, v0
+; GFX9-NEXT:    v_lshrrev_b32_e32 v15, 4, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
+; GFX9-NEXT:    v_and_b32_e32 v16, 15, v1
+; GFX9-NEXT:    v_and_b32_sdwa v17, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v18, 15, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 12, v2
+; GFX9-NEXT:    v_and_b32_e32 v1, v4, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-NEXT:    v_and_b32_sdwa v19, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v5, 15, v10
+; GFX9-NEXT:    v_and_b32_e32 v10, v4, v16
+; GFX9-NEXT:    v_and_b32_e32 v16, v4, v17
+; GFX9-NEXT:    v_and_b32_e32 v17, v4, v18
+; GFX9-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX9-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_lshl_or_b32 v5, v5, 16, v10
+; GFX9-NEXT:    v_lshl_or_b32 v6, v15, 16, v17
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_and_b32_e32 v2, v4, v2
+; GFX9-NEXT:    v_pk_lshlrev_b16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_lshlrev_b16 v6, 12, v6 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-NEXT:    v_and_b32_e32 v8, v4, v8
+; GFX9-NEXT:    v_and_b32_e32 v18, v4, v19
+; GFX9-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-NEXT:    v_and_b32_e32 v4, v4, v13
 ; GFX9-NEXT:    v_lshl_or_b32 v8, v9, 16, v8
-; GFX9-NEXT:    v_lshl_or_b32 v5, v16, 16, v15
-; GFX9-NEXT:    v_and_b32_e32 v6, v2, v6
+; GFX9-NEXT:    v_lshl_or_b32 v4, v14, 16, v4
+; GFX9-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_ashrrev_i16 v6, 12, v6 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v8, 12, v8 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_lshlrev_b16 v5, 12, v5 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_and_b32_e32 v13, v2, v13
-; GFX9-NEXT:    v_and_b32_e32 v2, v2, v11
-; GFX9-NEXT:    v_lshl_or_b32 v6, v7, 16, v6
-; GFX9-NEXT:    v_lshl_or_b32 v7, v14, 16, v13
-; GFX9-NEXT:    v_lshl_or_b32 v2, v12, 16, v2
+; GFX9-NEXT:    v_pk_lshlrev_b16 v4, 12, v4 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-NEXT:    v_pk_mul_lo_u16 v5, v5, v6
+; GFX9-NEXT:    v_lshl_or_b32 v7, v7, 16, v16
+; GFX9-NEXT:    v_lshl_or_b32 v2, v11, 16, v2
+; GFX9-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-NEXT:    v_add_u16_e32 v3, v5, v3
+; GFX9-NEXT:    v_lshl_or_b32 v10, v12, 16, v18
 ; GFX9-NEXT:    v_pk_ashrrev_i16 v8, 12, v8 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_lshlrev_b16 v6, 12, v6 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_lshlrev_b16 v0, 12, v0 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_ashrrev_i16 v4, 12, v4 op_sel_hi:[0,1]
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v7, 12, v7 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_lshlrev_b16 v1, 12, v1 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_lshlrev_b16 v9, 12, v10 op_sel_hi:[0,1]
 ; GFX9-NEXT:    v_pk_lshlrev_b16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_mul_lo_u16 v5, v8, v5
-; GFX9-NEXT:    v_pk_ashrrev_i16 v6, 12, v6 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_ashrrev_i16 v0, 12, v0 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_ashrrev_i16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
-; GFX9-NEXT:    v_pk_mul_lo_u16 v0, v0, v2
-; GFX9-NEXT:    v_pk_mul_lo_u16 v2, v6, v7
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u16_e32 v4, v3, v4
-; GFX9-NEXT:    v_add_u16_sdwa v3, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u16_e32 v3, v3, v5
+; GFX9-NEXT:    v_pk_mul_lo_u16 v4, v8, v4
 ; GFX9-NEXT:    v_add_u16_sdwa v3, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    v_add_u16_e32 v3, v3, v4
+; GFX9-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_ashrrev_i16 v1, 12, v1 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_ashrrev_i16 v2, 12, v2 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_ashrrev_i16 v9, 12, v9 op_sel_hi:[0,1]
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-NEXT:    v_pk_mul_lo_u16 v2, v7, v9
+; GFX9-NEXT:    v_add_u16_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-NEXT:    v_add_u16_e32 v3, v3, v2
 ; GFX9-NEXT:    v_add_u16_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u16_e32 v2, v2, v0
-; GFX9-NEXT:    v_add_u16_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX9-NEXT:    v_add_u16_e32 v2, v2, v1
+; GFX9-NEXT:    v_add_u16_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    global_store_short v0, v1, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: idot8_acc16_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-DL-NEXT:    v_mov_b32_e32 v2, 0xffff
-; GFX9-DL-NEXT:    v_mov_b32_e32 v1, 0
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, 15
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    global_load_dword v3, v0, s[4:5]
-; GFX9-DL-NEXT:    global_load_dword v4, v0, s[6:7]
+; GFX9-DL-NEXT:    global_load_dword v1, v0, s[4:5]
+; GFX9-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-DL-NEXT:    global_load_ushort v3, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 12, v1
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v3
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_and_b32_e32 v17, 15, v4
-; GFX9-DL-NEXT:    v_bfe_u32 v0, v3, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v3, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v3, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v4, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v4, 8, 4
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 28, v3
-; GFX9-DL-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v3, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v3, v3, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v10, v2, v10
-; GFX9-DL-NEXT:    v_bfe_u32 v11, v4, 24, 4
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 28, v4
-; GFX9-DL-NEXT:    v_bfe_u32 v14, v4, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v4, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v4, v4, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v17, v2, v17
-; GFX9-DL-NEXT:    v_lshl_or_b32 v3, v3, 16, v10
-; GFX9-DL-NEXT:    v_lshl_or_b32 v4, v4, 16, v17
-; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v3, 12, v3 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v4, 12, v4 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v3, 12, v3 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v4, 12, v4 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v3, v3, v4
-; GFX9-DL-NEXT:    global_load_ushort v4, v1, s[2:3]
-; GFX9-DL-NEXT:    v_and_b32_e32 v8, v2, v8
-; GFX9-DL-NEXT:    v_and_b32_e32 v0, v2, v0
-; GFX9-DL-NEXT:    v_and_b32_e32 v15, v2, v15
-; GFX9-DL-NEXT:    v_lshl_or_b32 v0, v5, 16, v0
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v15, 4, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
+; GFX9-DL-NEXT:    v_and_b32_e32 v16, 15, v1
+; GFX9-DL-NEXT:    v_and_b32_sdwa v17, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v18, 15, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 12, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v1, v4, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v19, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v10
+; GFX9-DL-NEXT:    v_and_b32_e32 v10, v4, v16
+; GFX9-DL-NEXT:    v_and_b32_e32 v16, v4, v17
+; GFX9-DL-NEXT:    v_and_b32_e32 v17, v4, v18
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX9-DL-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_lshl_or_b32 v5, v5, 16, v10
+; GFX9-DL-NEXT:    v_lshl_or_b32 v6, v15, 16, v17
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_and_b32_e32 v2, v4, v2
+; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v6, 12, v6 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, v4, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v18, v4, v19
+; GFX9-DL-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-DL-NEXT:    v_and_b32_e32 v4, v4, v13
 ; GFX9-DL-NEXT:    v_lshl_or_b32 v8, v9, 16, v8
-; GFX9-DL-NEXT:    v_lshl_or_b32 v5, v16, 16, v15
-; GFX9-DL-NEXT:    v_and_b32_e32 v6, v2, v6
+; GFX9-DL-NEXT:    v_lshl_or_b32 v4, v14, 16, v4
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v6, 12, v6 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
 ; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v8, 12, v8 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v5, 12, v5 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_and_b32_e32 v13, v2, v13
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, v2, v11
-; GFX9-DL-NEXT:    v_lshl_or_b32 v6, v7, 16, v6
-; GFX9-DL-NEXT:    v_lshl_or_b32 v7, v14, 16, v13
-; GFX9-DL-NEXT:    v_lshl_or_b32 v2, v12, 16, v2
+; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v4, 12, v4 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v5, v5, v6
+; GFX9-DL-NEXT:    v_lshl_or_b32 v7, v7, 16, v16
+; GFX9-DL-NEXT:    v_lshl_or_b32 v2, v11, 16, v2
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
+; GFX9-DL-NEXT:    v_add_u16_e32 v3, v5, v3
+; GFX9-DL-NEXT:    v_lshl_or_b32 v10, v12, 16, v18
 ; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v8, 12, v8 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v6, 12, v6 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v0, 12, v0 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v4, 12, v4 op_sel_hi:[0,1]
 ; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v7, 12, v7 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v1, 12, v1 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v9, 12, v10 op_sel_hi:[0,1]
 ; GFX9-DL-NEXT:    v_pk_lshlrev_b16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v5, v8, v5
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v6, 12, v6 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v0, 12, v0 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v0, v0, v2
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v2, v6, v7
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u16_e32 v4, v3, v4
-; GFX9-DL-NEXT:    v_add_u16_sdwa v3, v4, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u16_e32 v3, v3, v5
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v4, v8, v4
 ; GFX9-DL-NEXT:    v_add_u16_sdwa v3, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u16_e32 v3, v3, v4
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v1, 12, v1 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v2, 12, v2 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_ashrrev_i16 v9, 12, v9 op_sel_hi:[0,1]
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v2, v7, v9
+; GFX9-DL-NEXT:    v_add_u16_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
 ; GFX9-DL-NEXT:    v_add_u16_e32 v3, v3, v2
 ; GFX9-DL-NEXT:    v_add_u16_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u16_e32 v2, v2, v0
-; GFX9-DL-NEXT:    v_add_u16_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX9-DL-NEXT:    v_add_u16_e32 v2, v2, v1
+; GFX9-DL-NEXT:    v_add_u16_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    global_store_short v0, v1, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-XNACK-LABEL: idot8_acc16_vecMul:
 ; GFX10-DL-XNACK:       ; %bb.0: ; %entry
 ; GFX10-DL-XNACK-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-XNACK-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-XNACK-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX10-DL-XNACK-NEXT:    v_mov_b32_e32 v4, 0xffff
+; GFX10-DL-XNACK-NEXT:    v_mov_b32_e32 v23, 0xffff
+; GFX10-DL-XNACK-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX10-DL-XNACK-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-XNACK-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-XNACK-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-XNACK-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-XNACK-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-XNACK-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-XNACK-NEXT:    s_clause 0x1
-; GFX10-DL-XNACK-NEXT:    global_load_dword v1, v0, s[4:5]
+; GFX10-DL-XNACK-NEXT:    global_load_dword v19, v0, s[4:5]
 ; GFX10-DL-XNACK-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX10-DL-XNACK-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-XNACK-NEXT:    global_load_ushort v3, v0, s[0:1]
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v11, 15, v1
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v10, 4, v19
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v13, 15, v2
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v7, v1, 16, 4
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v9, v1, 8, 4
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v5, v1, 24, 4
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v8, v1, 20, 4
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v10, v1, 12, 4
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v1, v1, 4, 4
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v11, v4, v11
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v16, v2, 4, 4
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v13, v4, v13
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v18, v2, 8, 4
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v9, v4, v9
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v1, v1, 16, v11
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v7, v4, v7
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v11, v16, 16, v13
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v19, v2, 24, 4
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v14, 28, v2
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v1, 12, v1 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v15, v2, 16, 4
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v17, v2, 20, 4
-; GFX10-DL-XNACK-NEXT:    v_bfe_u32 v2, v2, 12, 4
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v13, v4, v18
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v11, 12, v11 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v9, v10, 16, v9
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v7, v8, 16, v7
-; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v1, 12, v1 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v2, v2, 16, v13
-; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v8, 12, v11 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v15, 4, v2
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v16, 15, v19
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v18, 15, v2
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v8, 8, v19
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v12, 20, v2
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v14, 12, v2
+; GFX10-DL-XNACK-NEXT:    v_and_b32_sdwa v17, v19, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_and_b32_sdwa v1, v19, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX10-DL-XNACK-NEXT:    v_and_b32_sdwa v22, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_and_b32_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v5, 15, v10
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v10, v23, v16
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v16, v23, v18
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v9, 12, v19
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v5, v5, 16, v10
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v10, v15, 16, v16
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v8, v23, v8
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v10, 12, v10 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v13, v23, v13
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v8, v9, 16, v8
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v7, 20, v19
+; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v10, 12, v10 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v9, v14, 16, v13
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v8, 12, v8 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v17, v23, v17
+; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v5, v5, v10
 ; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v9, 12, v9 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v10, v4, v15
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v7, 12, v7 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v1, v1, v8
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v13, v23, v22
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v7, v7, 16, v17
+; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v8, 12, v8 op_sel_hi:[0,1]
 ; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v9, 12, v9 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v8, v17, 16, v10
-; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v10, 16, v1
 ; GFX10-DL-XNACK-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v3
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v3, v4, v5
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v5, 12, v8 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v4, v4, v19
-; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v2, v9, v2
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v10
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v3, v6, 16, v3
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v3, v5, v3
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v10, v12, 16, v13
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v12, 16, v5
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v7, 12, v7 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v2, v23, v2
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v19
+; GFX10-DL-XNACK-NEXT:    v_and_b32_e32 v1, v23, v1
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v5, 12, v10 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v4, v8, v9
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v12
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v2, v11, 16, v2
+; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
 ; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_lshl_or_b32 v4, v14, 16, v4
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v2
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v2, 12, v3 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v3, 12, v4 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v6, 16, v4
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v4
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v1, 12, v1 op_sel_hi:[0,1]
+; GFX10-DL-XNACK-NEXT:    v_pk_lshlrev_b16 v2, 12, v2 op_sel_hi:[0,1]
 ; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v4, v7, v5
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v6
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v6
+; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v1, 12, v1 op_sel_hi:[0,1]
 ; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v2, 12, v2 op_sel_hi:[0,1]
-; GFX10-DL-XNACK-NEXT:    v_pk_ashrrev_i16 v3, 12, v3 op_sel_hi:[0,1]
 ; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v5, 16, v4
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v7, v1, v4
-; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v2, v2, v3
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v7, v5
-; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v2
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v4
+; GFX10-DL-XNACK-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v2, v3, v5
+; GFX10-DL-XNACK-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
+; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v2, v1
 ; GFX10-DL-XNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v3
 ; GFX10-DL-XNACK-NEXT:    global_store_short v0, v1, s[0:1]
 ; GFX10-DL-XNACK-NEXT:    s_endpgm
 ;
 ; GFX10-DL-NOXNACK-LABEL: idot8_acc16_vecMul:
 ; GFX10-DL-NOXNACK:       ; %bb.0: ; %entry
 ; GFX10-DL-NOXNACK-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NOXNACK-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-NOXNACK-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX10-DL-NOXNACK-NEXT:    v_mov_b32_e32 v2, 0
-; GFX10-DL-NOXNACK-NEXT:    v_mov_b32_e32 v4, 0xffff
+; GFX10-DL-NOXNACK-NEXT:    v_mov_b32_e32 v23, 0xffff
+; GFX10-DL-NOXNACK-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX10-DL-NOXNACK-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NOXNACK-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NOXNACK-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NOXNACK-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NOXNACK-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NOXNACK-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NOXNACK-NEXT:    s_clause 0x1
-; GFX10-DL-NOXNACK-NEXT:    global_load_dword v1, v0, s[4:5]
+; GFX10-DL-NOXNACK-NEXT:    global_load_dword v19, v0, s[4:5]
 ; GFX10-DL-NOXNACK-NEXT:    global_load_dword v0, v0, s[6:7]
 ; GFX10-DL-NOXNACK-NEXT:    global_load_ushort v3, v2, s[0:1]
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v11, 15, v1
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v10, 4, v19
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v13, 15, v0
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v7, v1, 16, 4
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v9, v1, 8, 4
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v5, v1, 24, 4
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v8, v1, 20, 4
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v10, v1, 12, 4
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v1, v1, 4, 4
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v11, v4, v11
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v16, v0, 4, 4
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v13, v4, v13
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v18, v0, 8, 4
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v9, v4, v9
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v1, v1, 16, v11
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v7, v4, v7
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v11, v16, 16, v13
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v19, v0, 24, 4
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v14, 28, v0
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v1, 12, v1 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v15, v0, 16, 4
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v17, v0, 20, 4
-; GFX10-DL-NOXNACK-NEXT:    v_bfe_u32 v0, v0, 12, 4
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v13, v4, v18
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v11, 12, v11 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v9, v10, 16, v9
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v7, v8, 16, v7
-; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v1, 12, v1 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v0, v0, 16, v13
-; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v8, 12, v11 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v15, 4, v0
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v16, 15, v19
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v18, 15, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v8, 8, v19
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v12, 20, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v13, 8, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v14, 12, v0
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_sdwa v17, v19, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_sdwa v1, v19, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v11, 28, v0
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_sdwa v22, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_sdwa v0, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v5, 15, v10
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v10, v23, v16
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v16, v23, v18
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v9, 12, v19
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v5, v5, 16, v10
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v10, v15, 16, v16
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v8, v23, v8
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v10, 12, v10 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v13, v23, v13
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v8, v9, 16, v8
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v7, 20, v19
+; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v10, 12, v10 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v9, v14, 16, v13
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v8, 12, v8 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v17, v23, v17
+; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v5, v5, v10
 ; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v9, 12, v9 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v10, v4, v15
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v7, 12, v7 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v0, 12, v0 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v1, v1, v8
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v13, v23, v22
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v7, v7, 16, v17
+; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v8, 12, v8 op_sel_hi:[0,1]
 ; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v9, 12, v9 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v8, v17, 16, v10
-; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v0, 12, v0 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v10, 16, v1
 ; GFX10-DL-NOXNACK-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v3
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v3, v4, v5
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v5, 12, v8 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v4, v4, v19
-; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v0, v9, v0
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v1, v1, v10
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v3, v6, 16, v3
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v3, v5, v3
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v10, v12, 16, v13
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v12, 16, v5
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v7, 12, v7 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v0, v23, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v6, 28, v19
+; GFX10-DL-NOXNACK-NEXT:    v_and_b32_e32 v1, v23, v1
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v5, 12, v10 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v4, v8, v9
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v12
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v0, v11, 16, v0
+; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v7, 12, v7 op_sel_hi:[0,1]
 ; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v5, 12, v5 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_lshl_or_b32 v4, v14, 16, v4
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v6, 16, v0
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v0, v1, v0
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v1, 12, v3 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v3, 12, v4 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v6, 16, v4
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v4
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v1, 12, v1 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_pk_lshlrev_b16 v0, 12, v0 op_sel_hi:[0,1]
 ; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v4, v7, v5
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v7, v0, v6
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v6
 ; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v1, 12, v1 op_sel_hi:[0,1]
-; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v3, 12, v3 op_sel_hi:[0,1]
+; GFX10-DL-NOXNACK-NEXT:    v_pk_ashrrev_i16 v0, 12, v0 op_sel_hi:[0,1]
 ; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v5, 16, v4
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v0, v7, v4
-; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v1, v1, v3
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v0, v0, v5
-; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
-; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v0, v0, v1
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v3, v3, v4
+; GFX10-DL-NOXNACK-NEXT:    v_pk_mul_lo_u16 v0, v1, v0
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v1, v3, v5
+; GFX10-DL-NOXNACK-NEXT:    v_lshrrev_b32_e32 v3, 16, v0
+; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v0, v1, v0
 ; GFX10-DL-NOXNACK-NEXT:    v_add_nc_u16_e64 v0, v0, v3
 ; GFX10-DL-NOXNACK-NEXT:    global_store_short v2, v0, s[0:1]
 ; GFX10-DL-NOXNACK-NEXT:    s_endpgm
 ; GFX10-DL-LABEL: idot8_acc16_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[4:5], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-NEXT:    s_mov_b32 s12, SCRATCH_RSRC_DWORD0

llvm/test/CodeGen/AMDGPU/idot8u.ll

 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc16:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
-; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dword v4, v[0:1]
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s7
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
-; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s0
-; GFX8-NEXT:    flat_load_ushort v18, v[2:3]
+; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    flat_load_ushort v1, v[2:3]
+; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
 ; GFX8-NEXT:    s_add_u32 s8, s8, s3
 ; GFX8-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(2)
-; GFX8-NEXT:    v_lshrrev_b32_e32 v1, 28, v4
-; GFX8-NEXT:    v_bfe_u32 v5, v4, 24, 4
-; GFX8-NEXT:    v_bfe_u32 v6, v4, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v7, v4, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v8, v4, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v9, v4, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v10, v4, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 4, v4
+; GFX8-NEXT:    v_and_b32_e32 v16, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 20, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 12, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v9, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(1)
+; GFX8-NEXT:    v_and_b32_e32 v17, 15, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 4, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 20, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v13, 12, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v14, 8, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 28, v4
+; GFX8-NEXT:    v_and_b32_sdwa v18, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 28, v0
-; GFX8-NEXT:    v_bfe_u32 v12, v0, 24, 4
-; GFX8-NEXT:    v_bfe_u32 v13, v0, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v14, v0, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v15, v0, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v16, v0, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v17, v0, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX8-NEXT:    v_and_b32_sdwa v19, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v0, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v10
+; GFX8-NEXT:    v_and_b32_e32 v10, 15, v15
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v18
-; GFX8-NEXT:    v_mad_u16 v0, v10, v17, v0
-; GFX8-NEXT:    v_mad_u16 v0, v9, v16, v0
-; GFX8-NEXT:    v_mad_u16 v0, v8, v15, v0
-; GFX8-NEXT:    v_mad_u16 v0, v7, v14, v0
-; GFX8-NEXT:    v_mad_u16 v0, v6, v13, v0
-; GFX8-NEXT:    v_mad_u16 v0, v5, v12, v0
-; GFX8-NEXT:    v_mad_u16 v0, v1, v11, v0
+; GFX8-NEXT:    v_mad_u16 v1, v16, v17, v1
+; GFX8-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX8-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX8-NEXT:    v_mad_u16 v1, v5, v10, v1
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX8-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX8-NEXT:    v_mad_u16 v1, v9, v14, v1
+; GFX8-NEXT:    v_mad_u16 v1, v8, v13, v1
+; GFX8-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX8-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX8-NEXT:    v_mad_u16 v1, v18, v19, v1
+; GFX8-NEXT:    v_mad_u16 v1, v7, v12, v1
+; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v1
+; GFX8-NEXT:    v_mad_u16 v0, v6, v11, v0
 ; GFX8-NEXT:    flat_store_short v[2:3], v0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc16:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-NEXT:    global_load_ushort v17, v1, s[2:3]
+; GFX9-NEXT:    global_load_ushort v4, v1, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-NEXT:    v_bfe_u32 v4, v2, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 4, v2
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 4, v3
+; GFX9-NEXT:    v_and_b32_e32 v15, 15, v2
+; GFX9-NEXT:    v_and_b32_e32 v16, 15, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 20, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 12, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 8, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-NEXT:    v_and_b32_sdwa v17, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-NEXT:    v_bfe_u32 v11, v3, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-NEXT:    v_and_b32_sdwa v18, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v3, 15, v9
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v14
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-NEXT:    v_mad_legacy_u16 v4, v15, v16, v4
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v3, v9, v4
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v8, v13, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v7, v12, v3
+; GFX9-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX9-NEXT:    v_and_b32_e32 v11, 15, v11
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v17, v18, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v6, v11, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v5, v10, v0
 ; GFX9-NEXT:    global_store_short v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc16:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-DL-NEXT:    global_load_ushort v17, v1, s[2:3]
+; GFX9-DL-NEXT:    global_load_ushort v4, v1, s[2:3]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v4, v2, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 4, v2
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 4, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, 15, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v16, 15, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 20, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 12, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 8, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v17, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-DL-NEXT:    v_bfe_u32 v11, v3, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-DL-NEXT:    v_and_b32_sdwa v18, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v14
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v4, v15, v16, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v3, v9, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v8, v13, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v7, v12, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX9-DL-NEXT:    v_and_b32_e32 v11, 15, v11
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v17, v18, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v6, v11, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v5, v10, v0
 ; GFX9-DL-NEXT:    global_store_short v1, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc16:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
-; GFX10-DL-NEXT:    global_load_dword v2, v0, s[4:5]
-; GFX10-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX10-DL-NEXT:    global_load_ushort v4, v1, s[2:3]
+; GFX10-DL-NEXT:    global_load_dword v11, v0, s[4:5]
+; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-DL-NEXT:    global_load_ushort v3, v0, s[2:3]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NEXT:    v_and_b32_e32 v11, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 4, v11
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v3
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 4, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 4, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v11
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v5
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v4
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 8, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 12, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 12, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 16, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 16, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 24, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 24, 4
+; GFX10-DL-NEXT:    v_mad_u16 v3, v6, v13, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 12, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v5, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v6
+; GFX10-DL-NEXT:    v_mov_b32_e32 v5, 15
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v7
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v8, v9, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 20, v2
+; GFX10-DL-NEXT:    v_and_b32_sdwa v9, v11, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v10, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 28, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v7
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v8
+; GFX10-DL-NEXT:    v_mad_u16 v3, v9, v10, v3
+; GFX10-DL-NEXT:    v_and_b32_sdwa v10, v11, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v5, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 28, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 28, v3
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v2, v3, v0
-; GFX10-DL-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_mad_u16 v3, v10, v5, v3
+; GFX10-DL-NEXT:    v_mad_u16 v1, v1, v2, v3
+; GFX10-DL-NEXT:    global_store_short v0, v1, s[2:3]
 ; GFX10-DL-NEXT:    s_endpgm
                                        <8 x i4> addrspace(1)* %src2,
                                        i16 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %gep1
   %gep2 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src2, i32 %idx

 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
-; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dword v4, v[0:1]
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s7
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
-; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s0
-; GFX8-NEXT:    flat_load_ubyte v18, v[2:3]
+; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    flat_load_ubyte v1, v[2:3]
+; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
 ; GFX8-NEXT:    s_add_u32 s8, s8, s3
 ; GFX8-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(2)
-; GFX8-NEXT:    v_lshrrev_b32_e32 v1, 28, v4
-; GFX8-NEXT:    v_bfe_u32 v5, v4, 24, 4
-; GFX8-NEXT:    v_bfe_u32 v6, v4, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v7, v4, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v8, v4, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v9, v4, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v10, v4, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 4, v4
+; GFX8-NEXT:    v_and_b32_e32 v16, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 20, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 12, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v9, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(1)
+; GFX8-NEXT:    v_and_b32_e32 v17, 15, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 4, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 20, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v13, 12, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v14, 8, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 28, v4
+; GFX8-NEXT:    v_and_b32_sdwa v18, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 28, v0
-; GFX8-NEXT:    v_bfe_u32 v12, v0, 24, 4
-; GFX8-NEXT:    v_bfe_u32 v13, v0, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v14, v0, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v15, v0, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v16, v0, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v17, v0, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX8-NEXT:    v_and_b32_sdwa v19, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v0, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v10
+; GFX8-NEXT:    v_and_b32_e32 v10, 15, v15
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v18
-; GFX8-NEXT:    v_mad_u16 v0, v10, v17, v0
-; GFX8-NEXT:    v_mad_u16 v0, v9, v16, v0
-; GFX8-NEXT:    v_mad_u16 v0, v8, v15, v0
-; GFX8-NEXT:    v_mad_u16 v0, v7, v14, v0
-; GFX8-NEXT:    v_mad_u16 v0, v6, v13, v0
-; GFX8-NEXT:    v_mad_u16 v0, v5, v12, v0
-; GFX8-NEXT:    v_mad_u16 v0, v1, v11, v0
+; GFX8-NEXT:    v_mad_u16 v1, v16, v17, v1
+; GFX8-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX8-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX8-NEXT:    v_mad_u16 v1, v5, v10, v1
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX8-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX8-NEXT:    v_mad_u16 v1, v9, v14, v1
+; GFX8-NEXT:    v_mad_u16 v1, v8, v13, v1
+; GFX8-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX8-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX8-NEXT:    v_mad_u16 v1, v18, v19, v1
+; GFX8-NEXT:    v_mad_u16 v1, v7, v12, v1
+; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v1
+; GFX8-NEXT:    v_mad_u16 v0, v6, v11, v0
 ; GFX8-NEXT:    flat_store_byte v[2:3], v0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc8:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-NEXT:    global_load_ubyte v17, v1, s[2:3]
+; GFX9-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-NEXT:    v_bfe_u32 v4, v2, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 4, v2
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 4, v3
+; GFX9-NEXT:    v_and_b32_e32 v15, 15, v2
+; GFX9-NEXT:    v_and_b32_e32 v16, 15, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 20, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 12, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 8, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-NEXT:    v_and_b32_sdwa v17, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-NEXT:    v_bfe_u32 v11, v3, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-NEXT:    v_and_b32_sdwa v18, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v3, 15, v9
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v14
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-NEXT:    v_mad_legacy_u16 v4, v15, v16, v4
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v3, v9, v4
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v8, v13, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v7, v12, v3
+; GFX9-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX9-NEXT:    v_and_b32_e32 v11, 15, v11
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v17, v18, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v6, v11, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v5, v10, v0
 ; GFX9-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc8:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-DL-NEXT:    global_load_ubyte v17, v1, s[2:3]
+; GFX9-DL-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v4, v2, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 4, v2
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 4, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, 15, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v16, 15, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 20, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 12, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 8, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v17, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-DL-NEXT:    v_bfe_u32 v11, v3, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-DL-NEXT:    v_and_b32_sdwa v18, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v14
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v4, v15, v16, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v3, v9, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v8, v13, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v7, v12, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX9-DL-NEXT:    v_and_b32_e32 v11, 15, v11
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v17, v18, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v6, v11, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v5, v10, v0
 ; GFX9-DL-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc8:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
-; GFX10-DL-NEXT:    global_load_dword v2, v0, s[4:5]
-; GFX10-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX10-DL-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX10-DL-NEXT:    global_load_dword v11, v0, s[4:5]
+; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-DL-NEXT:    global_load_ubyte v3, v0, s[2:3]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NEXT:    v_and_b32_e32 v11, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 4, v11
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v3
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 4, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 4, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v11
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v5
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v4
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 8, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 12, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 12, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 16, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 16, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 24, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 24, 4
+; GFX10-DL-NEXT:    v_mad_u16 v3, v6, v13, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 12, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v5, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v6
+; GFX10-DL-NEXT:    v_mov_b32_e32 v5, 15
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v7
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v8, v9, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 20, v2
+; GFX10-DL-NEXT:    v_and_b32_sdwa v9, v11, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v10, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 28, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v7
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v8
+; GFX10-DL-NEXT:    v_mad_u16 v3, v9, v10, v3
+; GFX10-DL-NEXT:    v_and_b32_sdwa v10, v11, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v5, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 28, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 28, v3
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v2, v3, v0
-; GFX10-DL-NEXT:    global_store_byte v1, v0, s[2:3]
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_mad_u16 v3, v10, v5, v3
+; GFX10-DL-NEXT:    v_mad_u16 v1, v1, v2, v3
+; GFX10-DL-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX10-DL-NEXT:    s_endpgm
                                       <8 x i4> addrspace(1)* %src2,
                                       i8 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %gep1
   %gep2 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src2, i32 %idx

 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc4:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
-; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dword v4, v[0:1]
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s7
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
-; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s0
-; GFX8-NEXT:    flat_load_ubyte v18, v[2:3]
+; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    flat_load_ubyte v1, v[2:3]
+; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
 ; GFX8-NEXT:    s_add_u32 s8, s8, s3
 ; GFX8-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(2)
-; GFX8-NEXT:    v_lshrrev_b32_e32 v1, 28, v4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 24, v4
-; GFX8-NEXT:    v_bfe_u32 v6, v4, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v7, v4, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v8, v4, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v9, v4, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v10, v4, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 4, v4
+; GFX8-NEXT:    v_and_b32_e32 v18, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 20, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v9, 12, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(1)
-; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 28, v0
-; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 24, v0
-; GFX8-NEXT:    v_bfe_u32 v13, v0, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v14, v0, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v15, v0, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v16, v0, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v17, v0, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX8-NEXT:    v_and_b32_e32 v19, 15, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v17, 4, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v14, 20, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 12, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v16, 8, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 28, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 24, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 28, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v13, 24, v0
+; GFX8-NEXT:    v_and_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v0, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v11
+; GFX8-NEXT:    v_and_b32_e32 v11, 15, v17
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v18
-; GFX8-NEXT:    v_mad_u16 v0, v10, v17, v0
-; GFX8-NEXT:    v_mad_u16 v0, v9, v16, v0
-; GFX8-NEXT:    v_mad_u16 v0, v8, v15, v0
-; GFX8-NEXT:    v_mad_u16 v0, v7, v14, v0
-; GFX8-NEXT:    v_mad_u16 v0, v6, v13, v0
-; GFX8-NEXT:    v_mad_u16 v0, v5, v12, v0
-; GFX8-NEXT:    v_mad_u16 v0, v1, v11, v0
+; GFX8-NEXT:    v_mad_u16 v1, v18, v19, v1
+; GFX8-NEXT:    v_and_b32_e32 v10, 15, v10
+; GFX8-NEXT:    v_and_b32_e32 v16, 15, v16
+; GFX8-NEXT:    v_mad_u16 v1, v5, v11, v1
+; GFX8-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX8-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX8-NEXT:    v_mad_u16 v1, v10, v16, v1
+; GFX8-NEXT:    v_mad_u16 v1, v9, v15, v1
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX8-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v1
+; GFX8-NEXT:    v_mad_u16 v0, v8, v14, v0
+; GFX8-NEXT:    v_mad_u16 v0, v7, v13, v0
+; GFX8-NEXT:    v_mad_u16 v0, v6, v12, v0
 ; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX8-NEXT:    flat_store_byte v[2:3], v0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc4:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-NEXT:    global_load_ubyte v17, v1, s[2:3]
+; GFX9-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-NEXT:    v_lshrrev_b32_e32 v4, 24, v2
-; GFX9-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 4, v2
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 24, v3
-; GFX9-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v16, 4, v3
+; GFX9-NEXT:    v_and_b32_e32 v17, 15, v2
+; GFX9-NEXT:    v_and_b32_e32 v18, 15, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 12, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 20, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 12, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v15, 8, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 28, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 24, v3
+; GFX9-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v3, 15, v10
+; GFX9-NEXT:    v_and_b32_e32 v10, 15, v16
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-NEXT:    v_mad_legacy_u16 v4, v17, v18, v4
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v3, v10, v4
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v9, v15, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v8, v14, v3
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v7, v13, v0
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v6, v12, v0
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v5, v11, v0
 ; GFX9-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX9-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc4:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-DL-NEXT:    global_load_ubyte v17, v1, s[2:3]
+; GFX9-DL-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v4, 24, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 4, v2
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 24, v3
-; GFX9-DL-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v16, 4, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v17, 15, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v18, 15, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 12, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 20, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 12, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v15, 8, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 24, v3
+; GFX9-DL-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v10
+; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v16
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v4, v17, v18, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v3, v10, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v9, v15, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v8, v14, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v7, v13, v0
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v6, v12, v0
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v5, v11, v0
 ; GFX9-DL-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX9-DL-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc4:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
-; GFX10-DL-NEXT:    global_load_dword v2, v0, s[4:5]
-; GFX10-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX10-DL-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX10-DL-NEXT:    global_load_dword v11, v0, s[4:5]
+; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-DL-NEXT:    global_load_ubyte v3, v0, s[2:3]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NEXT:    v_and_b32_e32 v11, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 4, v11
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v3
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 4, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 4, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v11
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v5
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v4
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 8, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 12, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 12, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 16, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 16, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v11, 24, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 24, v3
+; GFX10-DL-NEXT:    v_mad_u16 v3, v6, v13, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 12, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v5, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v6
+; GFX10-DL-NEXT:    v_mov_b32_e32 v5, 15
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v7
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v8, v9, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 20, v2
+; GFX10-DL-NEXT:    v_and_b32_sdwa v10, v11, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v5, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 28, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v7
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v8
+; GFX10-DL-NEXT:    v_mad_u16 v3, v10, v5, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 24, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 24, v11
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 28, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 28, v3
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v2, v3, v0
-; GFX10-DL-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX10-DL-NEXT:    global_store_byte v1, v0, s[2:3]
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_mad_u16 v3, v5, v10, v3
+; GFX10-DL-NEXT:    v_mad_u16 v1, v1, v2, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v1, 15, v1
+; GFX10-DL-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX10-DL-NEXT:    s_endpgm
                                       <8 x i4> addrspace(1)* %src2,
                                       i4 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %gep1
   %gep2 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src2, i32 %idx

 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_CommutationInsideMAD:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
-; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dword v4, v[0:1]
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s7
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
-; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s0
-; GFX8-NEXT:    flat_load_ubyte v18, v[2:3]
+; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    flat_load_ubyte v1, v[2:3]
+; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
 ; GFX8-NEXT:    s_add_u32 s8, s8, s3
 ; GFX8-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(2)
-; GFX8-NEXT:    v_lshrrev_b32_e32 v1, 28, v4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v5, 24, v4
-; GFX8-NEXT:    v_bfe_u32 v6, v4, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v7, v4, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v8, v4, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v9, v4, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v10, v4, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 4, v4
+; GFX8-NEXT:    v_and_b32_e32 v18, 15, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 20, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v9, 12, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 8, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(1)
-; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 28, v0
-; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 24, v0
-; GFX8-NEXT:    v_bfe_u32 v13, v0, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v14, v0, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v15, v0, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v16, v0, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v17, v0, 4, 4
-; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX8-NEXT:    v_and_b32_e32 v19, 15, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v17, 4, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v14, 20, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 12, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v16, 8, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 28, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 24, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 28, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v13, 24, v0
+; GFX8-NEXT:    v_and_b32_sdwa v4, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v0, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v5, 15, v11
+; GFX8-NEXT:    v_and_b32_e32 v11, 15, v17
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v18
-; GFX8-NEXT:    v_mad_u16 v0, v10, v17, v0
-; GFX8-NEXT:    v_mad_u16 v0, v9, v16, v0
-; GFX8-NEXT:    v_mad_u16 v0, v8, v15, v0
-; GFX8-NEXT:    v_mad_u16 v0, v7, v14, v0
-; GFX8-NEXT:    v_mad_u16 v0, v6, v13, v0
-; GFX8-NEXT:    v_mad_u16 v0, v5, v12, v0
-; GFX8-NEXT:    v_mad_u16 v0, v1, v11, v0
+; GFX8-NEXT:    v_mad_u16 v1, v18, v19, v1
+; GFX8-NEXT:    v_and_b32_e32 v10, 15, v10
+; GFX8-NEXT:    v_and_b32_e32 v16, 15, v16
+; GFX8-NEXT:    v_mad_u16 v1, v5, v11, v1
+; GFX8-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX8-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX8-NEXT:    v_mad_u16 v1, v10, v16, v1
+; GFX8-NEXT:    v_mad_u16 v1, v9, v15, v1
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX8-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v1
+; GFX8-NEXT:    v_mad_u16 v0, v8, v14, v0
+; GFX8-NEXT:    v_mad_u16 v0, v7, v13, v0
+; GFX8-NEXT:    v_mad_u16 v0, v6, v12, v0
 ; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX8-NEXT:    flat_store_byte v[2:3], v0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_CommutationInsideMAD:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-NEXT:    global_load_ubyte v17, v1, s[2:3]
+; GFX9-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-NEXT:    v_lshrrev_b32_e32 v4, 24, v2
-; GFX9-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 4, v2
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 24, v3
-; GFX9-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v16, 4, v3
+; GFX9-NEXT:    v_and_b32_e32 v17, 15, v2
+; GFX9-NEXT:    v_and_b32_e32 v18, 15, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 12, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 20, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 12, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v15, 8, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 28, v3
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 24, v3
+; GFX9-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v3, 15, v10
+; GFX9-NEXT:    v_and_b32_e32 v10, 15, v16
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-NEXT:    v_mad_legacy_u16 v4, v17, v18, v4
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v3, v10, v4
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v9, v15, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v3, v8, v14, v3
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v7, v13, v0
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v6, v12, v0
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v5, v11, v0
 ; GFX9-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX9-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_CommutationInsideMAD:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX9-DL-NEXT:    global_load_ubyte v17, v1, s[2:3]
+; GFX9-DL-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v0, 28, v2
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v4, 24, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v5, v2, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v7, v2, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v2, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v2, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 4, v2
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 28, v3
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 24, v3
-; GFX9-DL-NEXT:    v_bfe_u32 v12, v3, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v3, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v14, v3, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v3, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v3, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v16, 4, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v17, 15, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v18, 15, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 12, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 20, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 12, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v15, 8, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 28, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 24, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v3
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 24, v3
+; GFX9-DL-NEXT:    v_and_b32_sdwa v0, v3, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v3, 15, v10
+; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v16
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v2, v3, v17
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v9, v16, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v8, v15, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v7, v14, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v6, v13, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v5, v12, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v2, v4, v11, v2
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v0, v10, v2
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v4, v17, v18, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, 15, v15
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v3, v10, v4
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v9, v15, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v3, v8, v14, v3
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v2, v0, v3
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v7, v13, v0
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v6, v12, v0
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v5, v11, v0
 ; GFX9-DL-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX9-DL-NEXT:    global_store_byte v1, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_CommutationInsideMAD:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX10-DL-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
-; GFX10-DL-NEXT:    global_load_dword v2, v0, s[4:5]
-; GFX10-DL-NEXT:    global_load_dword v3, v0, s[6:7]
-; GFX10-DL-NEXT:    global_load_ubyte v4, v1, s[2:3]
+; GFX10-DL-NEXT:    global_load_dword v11, v0, s[4:5]
+; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
+; GFX10-DL-NEXT:    global_load_ubyte v3, v0, s[2:3]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NEXT:    v_and_b32_e32 v11, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 4, v11
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v3
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 4, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 4, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v11
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 8, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX10-DL-NEXT:    v_and_b32_e32 v5, 15, v5
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v4
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 8, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 12, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 12, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v2, 16, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v3, 16, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v11, 24, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 24, v3
+; GFX10-DL-NEXT:    v_mad_u16 v3, v6, v13, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 12, v11
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 12, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v5, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v6
+; GFX10-DL-NEXT:    v_mov_b32_e32 v5, 15
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v7
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v8, v9, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 20, v2
+; GFX10-DL-NEXT:    v_and_b32_sdwa v10, v11, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v5, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 28, v11
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, 15, v7
+; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v8
+; GFX10-DL-NEXT:    v_mad_u16 v3, v10, v5, v3
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 24, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 24, v11
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 28, v2
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 28, v3
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v7, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v11, v5, v0
-; GFX10-DL-NEXT:    v_mad_u16 v0, v2, v3, v0
-; GFX10-DL-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX10-DL-NEXT:    global_store_byte v1, v0, s[2:3]
+; GFX10-DL-NEXT:    v_mad_u16 v3, v4, v6, v3
+; GFX10-DL-NEXT:    v_mad_u16 v3, v5, v10, v3
+; GFX10-DL-NEXT:    v_mad_u16 v1, v1, v2, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v1, 15, v1
+; GFX10-DL-NEXT:    global_store_byte v0, v1, s[2:3]
 ; GFX10-DL-NEXT:    s_endpgm
                                                       <8 x i4> addrspace(1)* %src2,
                                                       i4 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx
   %vec1 = load <8 x i4>, <8 x i4> addrspace(1)* %gep1
   %gep2 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src2, i32 %idx

 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
-; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    flat_load_dword v4, v[0:1]
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s7
-; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v3, s1
-; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
 ; GFX8-NEXT:    v_mov_b32_e32 v2, s0
-; GFX8-NEXT:    flat_load_ushort v18, v[2:3]
+; GFX8-NEXT:    flat_load_dword v0, v[0:1]
+; GFX8-NEXT:    flat_load_ushort v1, v[2:3]
+; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
 ; GFX8-NEXT:    s_add_u32 s8, s8, s3
 ; GFX8-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(2)
-; GFX8-NEXT:    v_and_b32_e32 v1, 15, v4
-; GFX8-NEXT:    v_bfe_u32 v5, v4, 4, 4
-; GFX8-NEXT:    v_bfe_u32 v6, v4, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v7, v4, 12, 4
-; GFX8-NEXT:    v_bfe_u32 v8, v4, 16, 4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 4, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 12, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v9, 20, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 28, v4
 ; GFX8-NEXT:    s_waitcnt vmcnt(1)
-; GFX8-NEXT:    v_and_b32_e32 v11, 15, v0
-; GFX8-NEXT:    v_bfe_u32 v12, v0, 4, 4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 4, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 8, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v13, 12, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v14, 20, v0
+; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 28, v0
+; GFX8-NEXT:    v_and_b32_sdwa v16, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v17, v4, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v18, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v5, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_e32 v4, 15, v4
+; GFX8-NEXT:    v_and_b32_e32 v0, 15, v0
+; GFX8-NEXT:    v_and_b32_e32 v6, 15, v6
+; GFX8-NEXT:    v_and_b32_e32 v11, 15, v11
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_mad_u16 v1, v1, v11, v18
-; GFX8-NEXT:    v_bfe_u32 v13, v0, 8, 4
-; GFX8-NEXT:    v_mad_u16 v1, v5, v12, v1
-; GFX8-NEXT:    v_bfe_u32 v14, v0, 12, 4
-; GFX8-NEXT:    v_mad_u16 v1, v6, v13, v1
-; GFX8-NEXT:    v_bfe_u32 v15, v0, 16, 4
-; GFX8-NEXT:    v_mad_u16 v1, v7, v14, v1
-; GFX8-NEXT:    v_bfe_u32 v9, v4, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v16, v0, 20, 4
-; GFX8-NEXT:    v_mad_u16 v1, v8, v15, v1
-; GFX8-NEXT:    v_bfe_u32 v10, v4, 24, 4
-; GFX8-NEXT:    v_bfe_u32 v17, v0, 24, 4
-; GFX8-NEXT:    v_mad_u16 v1, v9, v16, v1
-; GFX8-NEXT:    v_lshrrev_b32_e32 v4, 28, v4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v0, 28, v0
-; GFX8-NEXT:    v_mad_u16 v1, v10, v17, v1
 ; GFX8-NEXT:    v_mad_u16 v0, v4, v0, v1
+; GFX8-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX8-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX8-NEXT:    v_mad_u16 v0, v6, v11, v0
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX8-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX8-NEXT:    v_mad_u16 v0, v7, v12, v0
+; GFX8-NEXT:    v_mad_u16 v0, v8, v13, v0
+; GFX8-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX8-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX8-NEXT:    v_mad_u16 v0, v17, v5, v0
+; GFX8-NEXT:    v_mad_u16 v0, v9, v14, v0
+; GFX8-NEXT:    v_mad_u16 v0, v16, v18, v0
+; GFX8-NEXT:    v_mad_u16 v0, v10, v15, v0
 ; GFX8-NEXT:    flat_store_short v[2:3], v0
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc16_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-NEXT:    v_mov_b32_e32 v2, 0xffff
-; GFX9-NEXT:    v_mov_b32_e32 v1, 0
+; GFX9-NEXT:    v_mov_b32_e32 v4, 15
+; GFX9-NEXT:    v_mov_b32_e32 v5, 0xffff
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_load_dword v3, v0, s[4:5]
-; GFX9-NEXT:    global_load_dword v4, v0, s[6:7]
+; GFX9-NEXT:    global_load_dword v1, v0, s[4:5]
+; GFX9-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-NEXT:    global_load_ushort v3, v0, s[2:3]
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
+; GFX9-NEXT:    s_waitcnt vmcnt(2)
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_bfe_u32 v0, v3, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v3, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v3, 8, 4
-; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_bfe_u32 v11, v4, 24, 4
-; GFX9-NEXT:    v_bfe_u32 v13, v4, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v4, 8, 4
-; GFX9-NEXT:    v_and_b32_e32 v17, 15, v4
-; GFX9-NEXT:    v_and_b32_e32 v10, 15, v3
-; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 28, v3
-; GFX9-NEXT:    v_and_b32_e32 v0, v2, v0
-; GFX9-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX9-NEXT:    v_and_b32_e32 v6, v2, v6
-; GFX9-NEXT:    v_bfe_u32 v9, v3, 12, 4
-; GFX9-NEXT:    v_and_b32_e32 v8, v2, v8
-; GFX9-NEXT:    v_bfe_u32 v3, v3, 4, 4
-; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 28, v4
-; GFX9-NEXT:    v_bfe_u32 v14, v4, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v16, v4, 12, 4
-; GFX9-NEXT:    v_bfe_u32 v4, v4, 4, 4
-; GFX9-NEXT:    v_and_b32_e32 v17, v2, v17
-; GFX9-NEXT:    v_and_b32_e32 v11, v2, v11
-; GFX9-NEXT:    v_and_b32_e32 v13, v2, v13
-; GFX9-NEXT:    v_and_b32_e32 v15, v2, v15
-; GFX9-NEXT:    v_and_b32_e32 v2, v2, v10
-; GFX9-NEXT:    v_lshl_or_b32 v4, v4, 16, v17
-; GFX9-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
-; GFX9-NEXT:    v_pk_mul_lo_u16 v2, v2, v4
-; GFX9-NEXT:    global_load_ushort v4, v1, s[2:3]
-; GFX9-NEXT:    v_lshl_or_b32 v0, v5, 16, v0
-; GFX9-NEXT:    v_lshl_or_b32 v6, v7, 16, v6
-; GFX9-NEXT:    v_lshl_or_b32 v5, v14, 16, v13
-; GFX9-NEXT:    v_lshl_or_b32 v7, v16, 16, v15
-; GFX9-NEXT:    v_lshl_or_b32 v8, v9, 16, v8
-; GFX9-NEXT:    v_pk_mul_lo_u16 v3, v6, v5
-; GFX9-NEXT:    v_pk_mul_lo_u16 v5, v8, v7
-; GFX9-NEXT:    v_lshl_or_b32 v10, v12, 16, v11
-; GFX9-NEXT:    v_pk_mul_lo_u16 v0, v0, v10
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 12, v2
+; GFX9-NEXT:    v_and_b32_e32 v16, 15, v1
+; GFX9-NEXT:    v_and_b32_e32 v18, 15, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v15, 4, v2
+; GFX9-NEXT:    v_and_b32_sdwa v19, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 20, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 12, v1
+; GFX9-NEXT:    v_and_b32_sdwa v17, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v2, v5, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX9-NEXT:    v_and_b32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v4, 15, v10
+; GFX9-NEXT:    v_and_b32_e32 v10, 15, v15
+; GFX9-NEXT:    v_and_b32_e32 v16, v5, v16
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_and_b32_e32 v18, v5, v18
+; GFX9-NEXT:    v_and_b32_e32 v1, v5, v1
+; GFX9-NEXT:    v_lshl_or_b32 v2, v11, 16, v2
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-NEXT:    v_and_b32_e32 v17, v5, v17
+; GFX9-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-NEXT:    v_and_b32_e32 v15, v5, v19
+; GFX9-NEXT:    v_and_b32_e32 v11, v5, v13
+; GFX9-NEXT:    v_and_b32_e32 v5, v5, v8
+; GFX9-NEXT:    v_lshl_or_b32 v8, v10, 16, v18
+; GFX9-NEXT:    v_lshl_or_b32 v4, v4, 16, v16
+; GFX9-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-NEXT:    v_pk_mul_lo_u16 v4, v4, v8
+; GFX9-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-NEXT:    v_lshl_or_b32 v6, v12, 16, v15
+; GFX9-NEXT:    v_lshl_or_b32 v7, v7, 16, v17
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u16_e32 v4, v2, v4
-; GFX9-NEXT:    v_add_u16_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u16_e32 v2, v2, v5
-; GFX9-NEXT:    v_add_u16_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u16_e32 v2, v2, v3
-; GFX9-NEXT:    v_add_u16_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    v_add_u16_e32 v2, v2, v0
-; GFX9-NEXT:    v_add_u16_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX9-NEXT:    v_add_u16_e32 v3, v4, v3
+; GFX9-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-NEXT:    v_pk_mul_lo_u16 v2, v7, v6
+; GFX9-NEXT:    v_lshl_or_b32 v6, v14, 16, v11
+; GFX9-NEXT:    v_lshl_or_b32 v5, v9, 16, v5
+; GFX9-NEXT:    v_pk_mul_lo_u16 v5, v5, v6
+; GFX9-NEXT:    v_add_u16_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    v_add_u16_e32 v3, v3, v5
+; GFX9-NEXT:    v_add_u16_sdwa v3, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    v_add_u16_e32 v3, v3, v2
+; GFX9-NEXT:    v_add_u16_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    v_add_u16_e32 v2, v2, v1
+; GFX9-NEXT:    v_add_u16_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-NEXT:    global_store_short v0, v1, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc16_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
-; GFX9-DL-NEXT:    v_mov_b32_e32 v2, 0xffff
-; GFX9-DL-NEXT:    v_mov_b32_e32 v1, 0
+; GFX9-DL-NEXT:    v_mov_b32_e32 v4, 15
+; GFX9-DL-NEXT:    v_mov_b32_e32 v5, 0xffff
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-DL-NEXT:    global_load_dword v3, v0, s[4:5]
-; GFX9-DL-NEXT:    global_load_dword v4, v0, s[6:7]
+; GFX9-DL-NEXT:    global_load_dword v1, v0, s[4:5]
+; GFX9-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 0
+; GFX9-DL-NEXT:    global_load_ushort v3, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
+; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT:    v_bfe_u32 v0, v3, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v3, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v3, 8, 4
-; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_bfe_u32 v11, v4, 24, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v4, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v4, 8, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v17, 15, v4
-; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v3
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 28, v3
-; GFX9-DL-NEXT:    v_and_b32_e32 v0, v2, v0
-; GFX9-DL-NEXT:    v_bfe_u32 v7, v3, 20, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v6, v2, v6
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v3, 12, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v8, v2, v8
-; GFX9-DL-NEXT:    v_bfe_u32 v3, v3, 4, 4
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 28, v4
-; GFX9-DL-NEXT:    v_bfe_u32 v14, v4, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v4, 12, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v4, v4, 4, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v17, v2, v17
-; GFX9-DL-NEXT:    v_and_b32_e32 v11, v2, v11
-; GFX9-DL-NEXT:    v_and_b32_e32 v13, v2, v13
-; GFX9-DL-NEXT:    v_and_b32_e32 v15, v2, v15
-; GFX9-DL-NEXT:    v_and_b32_e32 v2, v2, v10
-; GFX9-DL-NEXT:    v_lshl_or_b32 v4, v4, 16, v17
-; GFX9-DL-NEXT:    v_lshl_or_b32 v2, v3, 16, v2
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v2, v2, v4
-; GFX9-DL-NEXT:    global_load_ushort v4, v1, s[2:3]
-; GFX9-DL-NEXT:    v_lshl_or_b32 v0, v5, 16, v0
-; GFX9-DL-NEXT:    v_lshl_or_b32 v6, v7, 16, v6
-; GFX9-DL-NEXT:    v_lshl_or_b32 v5, v14, 16, v13
-; GFX9-DL-NEXT:    v_lshl_or_b32 v7, v16, 16, v15
-; GFX9-DL-NEXT:    v_lshl_or_b32 v8, v9, 16, v8
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v3, v6, v5
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v5, v8, v7
-; GFX9-DL-NEXT:    v_lshl_or_b32 v10, v12, 16, v11
-; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v0, v0, v10
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 12, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v16, 15, v1
+; GFX9-DL-NEXT:    v_and_b32_e32 v18, 15, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v15, 4, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v19, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 12, v1
+; GFX9-DL-NEXT:    v_and_b32_sdwa v17, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v2, v5, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX9-DL-NEXT:    v_and_b32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v4, 15, v10
+; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v15
+; GFX9-DL-NEXT:    v_and_b32_e32 v16, v5, v16
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_and_b32_e32 v18, v5, v18
+; GFX9-DL-NEXT:    v_and_b32_e32 v1, v5, v1
+; GFX9-DL-NEXT:    v_lshl_or_b32 v2, v11, 16, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-DL-NEXT:    v_and_b32_e32 v17, v5, v17
+; GFX9-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, v5, v19
+; GFX9-DL-NEXT:    v_and_b32_e32 v11, v5, v13
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, v5, v8
+; GFX9-DL-NEXT:    v_lshl_or_b32 v8, v10, 16, v18
+; GFX9-DL-NEXT:    v_lshl_or_b32 v4, v4, 16, v16
+; GFX9-DL-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v4, v4, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX9-DL-NEXT:    v_lshl_or_b32 v6, v12, 16, v15
+; GFX9-DL-NEXT:    v_lshl_or_b32 v7, v7, 16, v17
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u16_e32 v4, v2, v4
-; GFX9-DL-NEXT:    v_add_u16_sdwa v2, v4, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u16_e32 v2, v2, v5
-; GFX9-DL-NEXT:    v_add_u16_sdwa v2, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u16_e32 v2, v2, v3
-; GFX9-DL-NEXT:    v_add_u16_sdwa v2, v2, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    v_add_u16_e32 v2, v2, v0
-; GFX9-DL-NEXT:    v_add_u16_sdwa v0, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
-; GFX9-DL-NEXT:    global_store_short v1, v0, s[2:3]
+; GFX9-DL-NEXT:    v_add_u16_e32 v3, v4, v3
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v2, v7, v6
+; GFX9-DL-NEXT:    v_lshl_or_b32 v6, v14, 16, v11
+; GFX9-DL-NEXT:    v_lshl_or_b32 v5, v9, 16, v5
+; GFX9-DL-NEXT:    v_pk_mul_lo_u16 v5, v5, v6
+; GFX9-DL-NEXT:    v_add_u16_sdwa v3, v3, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u16_e32 v3, v3, v5
+; GFX9-DL-NEXT:    v_add_u16_sdwa v3, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u16_e32 v3, v3, v2
+; GFX9-DL-NEXT:    v_add_u16_sdwa v2, v3, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    v_add_u16_e32 v2, v2, v1
+; GFX9-DL-NEXT:    v_add_u16_sdwa v1, v2, v1 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_1
+; GFX9-DL-NEXT:    global_store_short v0, v1, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc16_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
-; GFX10-DL-NEXT:    v_mov_b32_e32 v4, 0xffff
+; GFX10-DL-NEXT:    v_mov_b32_e32 v19, 0xffff
+; GFX10-DL-NEXT:    v_mov_b32_e32 v4, 15
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX10-DL-NEXT:    global_load_ushort v3, v0, s[0:1]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NEXT:    v_and_b32_e32 v7, 15, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 4, v1
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_and_b32_e32 v6, 15, v2
-; GFX10-DL-NEXT:    v_bfe_u32 v9, v1, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v15, v2, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v13, v2, 8, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v7, v4, v7
-; GFX10-DL-NEXT:    v_and_b32_e32 v6, v4, v6
-; GFX10-DL-NEXT:    v_bfe_u32 v19, v1, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v10, v2, 12, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v13, v4, v13
-; GFX10-DL-NEXT:    v_lshl_or_b32 v7, v9, 16, v7
-; GFX10-DL-NEXT:    v_lshl_or_b32 v6, v15, 16, v6
-; GFX10-DL-NEXT:    v_bfe_u32 v9, v1, 12, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v12, v4, v19
-; GFX10-DL-NEXT:    v_bfe_u32 v11, v1, 16, 4
-; GFX10-DL-NEXT:    v_lshl_or_b32 v10, v10, 16, v13
-; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v6, v7, v6
-; GFX10-DL-NEXT:    v_bfe_u32 v7, v2, 16, 4
-; GFX10-DL-NEXT:    v_lshl_or_b32 v9, v9, 16, v12
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v1, 24, 4
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v23, 28, v1
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v12, 16, v6
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v14, 4, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v16, 15, v1
+; GFX10-DL-NEXT:    v_and_b32_e32 v18, 15, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v12, 8, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v10, 15, v10
+; GFX10-DL-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX10-DL-NEXT:    v_and_b32_e32 v15, v19, v18
+; GFX10-DL-NEXT:    v_and_b32_e32 v16, v19, v16
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 12, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v13, 12, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX10-DL-NEXT:    v_lshl_or_b32 v14, v14, 16, v15
+; GFX10-DL-NEXT:    v_lshl_or_b32 v15, v10, 16, v16
+; GFX10-DL-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX10-DL-NEXT:    v_and_b32_e32 v12, v19, v12
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, v19, v8
+; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v10, v15, v14
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 20, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v11, 20, v2
+; GFX10-DL-NEXT:    v_and_b32_sdwa v17, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshl_or_b32 v12, v13, 16, v12
+; GFX10-DL-NEXT:    v_lshl_or_b32 v15, v9, 16, v8
+; GFX10-DL-NEXT:    v_and_b32_sdwa v14, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 16, v10
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v6, v3
-; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v9, v9, v10
-; GFX10-DL-NEXT:    v_bfe_u32 v1, v1, 20, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v11, v4, v11
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v2, 20, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v7, v4, v7
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v3, v12
-; GFX10-DL-NEXT:    v_bfe_u32 v10, v2, 24, 4
-; GFX10-DL-NEXT:    v_lshl_or_b32 v1, v1, 16, v11
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v10, v3
+; GFX10-DL-NEXT:    v_and_b32_e32 v10, 15, v11
+; GFX10-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, v19, v17
+; GFX10-DL-NEXT:    v_and_b32_e32 v11, v19, v14
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v3, v9
+; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v8, v15, v12
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX10-DL-NEXT:    v_and_b32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v4, v2, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_lshl_or_b32 v9, v10, 16, v11
+; GFX10-DL-NEXT:    v_lshl_or_b32 v7, v7, 16, v13
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 16, v8
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v3, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v1, v19, v1
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v2, 28, v2
-; GFX10-DL-NEXT:    v_lshl_or_b32 v6, v6, 16, v7
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 16, v9
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v14, v3, v9
-; GFX10-DL-NEXT:    v_and_b32_e32 v9, v4, v10
-; GFX10-DL-NEXT:    v_and_b32_e32 v4, v4, v5
-; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v1, v1, v6
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v14, v7
-; GFX10-DL-NEXT:    v_lshl_or_b32 v2, v2, 16, v9
-; GFX10-DL-NEXT:    v_lshl_or_b32 v4, v23, 16, v4
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v3, v1
-; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v2, v4, v2
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v1, v3, v5
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 16, v2
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v1, v1, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v4, v19, v4
+; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v5, v7, v9
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v3, v10
+; GFX10-DL-NEXT:    v_lshl_or_b32 v1, v6, 16, v1
+; GFX10-DL-NEXT:    v_lshl_or_b32 v2, v2, 16, v4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v4, 16, v5
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v3, v5
+; GFX10-DL-NEXT:    v_pk_mul_lo_u16 v1, v1, v2
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v2, v3, v4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v3, 16, v1
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v1, v2, v1
 ; GFX10-DL-NEXT:    v_add_nc_u16_e64 v1, v1, v3
 ; GFX10-DL-NEXT:    global_store_short v0, v1, s[0:1]
 ; GFX10-DL-NEXT:    s_endpgm
                                               <8 x i4> addrspace(1)* %src2,
                                               i16 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx

 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0
+; GFX8-NEXT:    v_mov_b32_e32 v5, 15
 ; GFX8-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
-; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s5
 ; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s4, v2
 ; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
-; GFX8-NEXT:    v_mov_b32_e32 v3, s7
-; GFX8-NEXT:    v_add_u32_e32 v2, vcc, s6, v2
-; GFX8-NEXT:    v_addc_u32_e32 v3, vcc, 0, v3, vcc
-; GFX8-NEXT:    flat_load_dword v4, v[0:1]
-; GFX8-NEXT:    flat_load_dword v2, v[2:3]
+; GFX8-NEXT:    flat_load_dword v3, v[0:1]
+; GFX8-NEXT:    v_mov_b32_e32 v1, s7
+; GFX8-NEXT:    v_add_u32_e32 v0, vcc, s6, v2
+; GFX8-NEXT:    v_addc_u32_e32 v1, vcc, 0, v1, vcc
+; GFX8-NEXT:    flat_load_dword v2, v[0:1]
 ; GFX8-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX8-NEXT:    v_mov_b32_e32 v1, s1
-; GFX8-NEXT:    flat_load_ubyte v5, v[0:1]
+; GFX8-NEXT:    flat_load_ubyte v4, v[0:1]
+; GFX8-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX8-NEXT:    s_mov_b32 s10, -1
 ; GFX8-NEXT:    s_mov_b32 s11, 0xe80000
 ; GFX8-NEXT:    s_add_u32 s8, s8, s3
 ; GFX8-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX8-NEXT:    s_waitcnt vmcnt(2)
-; GFX8-NEXT:    v_bfe_u32 v3, v4, 20, 4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 20, v3
+; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 8, v3
+; GFX8-NEXT:    v_lshrrev_b32_e32 v9, 12, v3
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 28, v3
 ; GFX8-NEXT:    s_waitcnt vmcnt(1)
-; GFX8-NEXT:    v_bfe_u32 v13, v2, 20, 4
-; GFX8-NEXT:    v_bfe_u32 v7, v4, 24, 4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 28, v4
-; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 28, v2
-; GFX8-NEXT:    v_bfe_u32 v14, v2, 24, 4
-; GFX8-NEXT:    v_mul_lo_u16_sdwa v3, v3, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_bfe_u32 v6, v4, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v12, v2, 16, 4
-; GFX8-NEXT:    v_bfe_u32 v9, v4, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v16, v2, 8, 4
-; GFX8-NEXT:    v_bfe_u32 v10, v4, 12, 4
-; GFX8-NEXT:    v_and_b32_e32 v11, 15, v4
-; GFX8-NEXT:    v_bfe_u32 v17, v2, 12, 4
-; GFX8-NEXT:    v_and_b32_e32 v18, 15, v2
-; GFX8-NEXT:    v_bfe_u32 v4, v4, 4, 4
-; GFX8-NEXT:    v_bfe_u32 v2, v2, 4, 4
-; GFX8-NEXT:    v_mul_lo_u16_e32 v13, v7, v14
-; GFX8-NEXT:    v_mul_lo_u16_sdwa v8, v8, v15 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_mul_lo_u16_e32 v19, v6, v12
-; GFX8-NEXT:    v_mul_lo_u16_e32 v9, v9, v16
-; GFX8-NEXT:    v_mul_lo_u16_sdwa v10, v10, v17 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_mul_lo_u16_e32 v11, v11, v18
-; GFX8-NEXT:    v_mul_lo_u16_sdwa v4, v4, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX8-NEXT:    v_or_b32_e32 v8, v13, v8
-; GFX8-NEXT:    v_or_b32_e32 v9, v9, v10
-; GFX8-NEXT:    v_or_b32_e32 v10, v11, v4
-; GFX8-NEXT:    v_lshlrev_b32_e32 v11, 16, v8
-; GFX8-NEXT:    v_or_b32_e32 v3, v19, v3
-; GFX8-NEXT:    v_or_b32_sdwa v3, v3, v11 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 16, v9
-; GFX8-NEXT:    v_or_b32_e32 v4, v4, v2
-; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 8, v3
+; GFX8-NEXT:    v_lshrrev_b32_e32 v12, 20, v2
+; GFX8-NEXT:    v_lshrrev_b32_e32 v13, 8, v2
+; GFX8-NEXT:    v_lshrrev_b32_e32 v14, 12, v2
+; GFX8-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX8-NEXT:    v_and_b32_sdwa v17, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v18, v3, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v19, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX8-NEXT:    v_and_b32_sdwa v5, v2, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX8-NEXT:    v_mul_lo_u16_sdwa v6, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_mul_lo_u16_e32 v11, v18, v5
+; GFX8-NEXT:    v_lshrrev_b32_e32 v10, 4, v3
+; GFX8-NEXT:    v_lshrrev_b32_e32 v15, 4, v2
+; GFX8-NEXT:    v_and_b32_e32 v9, 15, v9
+; GFX8-NEXT:    v_and_b32_e32 v14, 15, v14
+; GFX8-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX8-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX8-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX8-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX8-NEXT:    v_and_b32_e32 v16, 15, v3
+; GFX8-NEXT:    v_and_b32_e32 v3, 15, v2
+; GFX8-NEXT:    v_and_b32_e32 v2, 15, v10
+; GFX8-NEXT:    v_and_b32_e32 v10, 15, v15
+; GFX8-NEXT:    v_mul_lo_u16_e32 v15, v17, v19
+; GFX8-NEXT:    v_mul_lo_u16_sdwa v7, v7, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_or_b32_e32 v6, v11, v6
+; GFX8-NEXT:    v_mul_lo_u16_e32 v8, v8, v13
+; GFX8-NEXT:    v_mul_lo_u16_sdwa v9, v9, v14 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_mul_lo_u16_sdwa v10, v2, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 16, v6
+; GFX8-NEXT:    v_mul_lo_u16_e32 v3, v16, v3
+; GFX8-NEXT:    v_or_b32_e32 v8, v8, v9
+; GFX8-NEXT:    v_or_b32_e32 v7, v15, v7
+; GFX8-NEXT:    v_or_b32_e32 v9, v3, v10
+; GFX8-NEXT:    v_or_b32_sdwa v3, v7, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 16, v8
+; GFX8-NEXT:    v_lshrrev_b32_e32 v7, 8, v3
+; GFX8-NEXT:    v_or_b32_e32 v10, v10, v2
 ; GFX8-NEXT:    v_lshrrev_b64 v[2:3], 24, v[2:3]
-; GFX8-NEXT:    v_lshrrev_b32_e32 v4, 8, v4
+; GFX8-NEXT:    v_lshrrev_b32_e32 v3, 8, v10
 ; GFX8-NEXT:    s_waitcnt vmcnt(0)
-; GFX8-NEXT:    v_add_u16_e32 v3, v10, v5
-; GFX8-NEXT:    v_add_u16_e32 v3, v3, v4
-; GFX8-NEXT:    v_add_u16_e32 v3, v3, v9
+; GFX8-NEXT:    v_add_u16_e32 v4, v9, v4
+; GFX8-NEXT:    v_add_u16_e32 v3, v4, v3
+; GFX8-NEXT:    v_add_u16_e32 v3, v3, v8
 ; GFX8-NEXT:    v_add_u16_e32 v2, v3, v2
-; GFX8-NEXT:    v_mad_u16 v2, v6, v12, v2
-; GFX8-NEXT:    v_add_u16_e32 v2, v2, v11
-; GFX8-NEXT:    v_lshrrev_b32_e32 v8, 8, v8
-; GFX8-NEXT:    v_mad_u16 v2, v7, v14, v2
-; GFX8-NEXT:    v_add_u16_e32 v2, v2, v8
+; GFX8-NEXT:    v_mad_u16 v2, v17, v19, v2
+; GFX8-NEXT:    v_add_u16_e32 v2, v2, v7
+; GFX8-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
+; GFX8-NEXT:    v_mad_u16 v2, v18, v5, v2
+; GFX8-NEXT:    v_add_u16_e32 v2, v2, v6
 ; GFX8-NEXT:    flat_store_byte v[0:1], v2
 ; GFX8-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: udot8_acc8_vecMul:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-NEXT:    s_mov_b32 s10, -1
 ; GFX9-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX9-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX9-NEXT:    global_load_ubyte v4, v3, s[2:3]
+; GFX9-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-NEXT:    v_bfe_u32 v0, v1, 20, 4
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 20, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 8, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v8, 12, v1
 ; GFX9-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-NEXT:    v_bfe_u32 v12, v2, 20, 4
-; GFX9-NEXT:    v_bfe_u32 v6, v1, 24, 4
-; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 28, v1
-; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 28, v2
-; GFX9-NEXT:    v_bfe_u32 v13, v2, 24, 4
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v0, v0, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_bfe_u32 v5, v1, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v11, v2, 16, 4
-; GFX9-NEXT:    v_bfe_u32 v8, v1, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v15, v2, 8, 4
-; GFX9-NEXT:    v_bfe_u32 v9, v1, 12, 4
-; GFX9-NEXT:    v_and_b32_e32 v10, 15, v1
-; GFX9-NEXT:    v_bfe_u32 v16, v2, 12, 4
-; GFX9-NEXT:    v_and_b32_e32 v17, 15, v2
-; GFX9-NEXT:    v_bfe_u32 v1, v1, 4, 4
-; GFX9-NEXT:    v_bfe_u32 v2, v2, 4, 4
-; GFX9-NEXT:    v_mul_lo_u16_e32 v12, v6, v13
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v7, v7, v14 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v18, v5, v11
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v2, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v8, v8, v15
-; GFX9-NEXT:    v_mul_lo_u16_sdwa v9, v9, v16 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_mul_lo_u16_e32 v10, v10, v17
-; GFX9-NEXT:    v_or_b32_e32 v7, v12, v7
-; GFX9-NEXT:    v_or_b32_e32 v8, v8, v9
-; GFX9-NEXT:    v_or_b32_e32 v1, v18, v0
-; GFX9-NEXT:    v_or_b32_e32 v9, v10, v2
-; GFX9-NEXT:    v_lshlrev_b32_e32 v10, 16, v7
-; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 16, v8
-; GFX9-NEXT:    v_or_b32_e32 v2, v2, v0
-; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 8, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v10, 20, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v12, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v13, 12, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-NEXT:    v_and_b32_sdwa v16, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v18, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v6, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_e32 v11, v16, v18
+; GFX9-NEXT:    v_lshrrev_b32_e32 v9, 4, v1
+; GFX9-NEXT:    v_lshrrev_b32_e32 v14, 4, v2
+; GFX9-NEXT:    v_and_b32_e32 v15, 15, v1
+; GFX9-NEXT:    v_and_b32_sdwa v17, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v1, 15, v2
+; GFX9-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-NEXT:    v_and_b32_e32 v10, 15, v10
+; GFX9-NEXT:    v_and_b32_e32 v0, 15, v9
+; GFX9-NEXT:    v_and_b32_e32 v9, 15, v14
+; GFX9-NEXT:    v_mul_lo_u16_e32 v14, v17, v2
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v5, v5, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_e32 v6, v11, v6
+; GFX9-NEXT:    v_mul_lo_u16_e32 v7, v7, v12
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v8, v8, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_mul_lo_u16_sdwa v9, v0, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_e32 v0, v14, v5
+; GFX9-NEXT:    v_lshlrev_b32_e32 v5, 16, v6
+; GFX9-NEXT:    v_mul_lo_u16_e32 v1, v15, v1
+; GFX9-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX9-NEXT:    v_or_b32_e32 v8, v1, v9
+; GFX9-NEXT:    v_or_b32_sdwa v1, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b32_e32 v0, 16, v7
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 8, v1
+; GFX9-NEXT:    v_or_b32_e32 v9, v9, v0
 ; GFX9-NEXT:    v_lshrrev_b64 v[0:1], 24, v[0:1]
-; GFX9-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
+; GFX9-NEXT:    v_lshrrev_b32_e32 v1, 8, v9
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_add_u16_e32 v1, v9, v4
-; GFX9-NEXT:    v_add_u16_e32 v1, v1, v2
-; GFX9-NEXT:    v_add_u16_e32 v1, v1, v8
+; GFX9-NEXT:    v_add_u16_e32 v4, v8, v4
+; GFX9-NEXT:    v_add_u16_e32 v1, v4, v1
+; GFX9-NEXT:    v_add_u16_e32 v1, v1, v7
 ; GFX9-NEXT:    v_add_u16_e32 v0, v1, v0
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v5, v11, v0
-; GFX9-NEXT:    v_add_u16_e32 v0, v0, v10
-; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 8, v7
-; GFX9-NEXT:    v_mad_legacy_u16 v0, v6, v13, v0
-; GFX9-NEXT:    v_add_u16_e32 v0, v0, v7
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v17, v2, v0
+; GFX9-NEXT:    v_add_u16_e32 v0, v0, v5
+; GFX9-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
+; GFX9-NEXT:    v_mad_legacy_u16 v0, v16, v18, v0
+; GFX9-NEXT:    v_add_u16_e32 v0, v0, v6
 ; GFX9-NEXT:    global_store_byte v3, v0, s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX9-DL-LABEL: udot8_acc8_vecMul:
 ; GFX9-DL:       ; %bb.0: ; %entry
 ; GFX9-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX9-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX9-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX9-DL-NEXT:    s_mov_b32 s11, 0xe00000
 ; GFX9-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX9-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX9-DL-NEXT:    s_load_dwordx2 s[2:3], s[0:1], 0x34
 ; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX9-DL-NEXT:    v_mov_b32_e32 v3, 0
 ; GFX9-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX9-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX9-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX9-DL-NEXT:    global_load_dword v2, v0, s[6:7]
 ; GFX9-DL-NEXT:    global_load_ubyte v4, v3, s[2:3]
+; GFX9-DL-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX9-DL-NEXT:    v_bfe_u32 v0, v1, 20, 4
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 20, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v8, 12, v1
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX9-DL-NEXT:    v_bfe_u32 v12, v2, 20, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v6, v1, 24, 4
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 28, v1
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 28, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v13, v2, 24, 4
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v0, v0, v12 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_bfe_u32 v5, v1, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v11, v2, 16, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v8, v1, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v15, v2, 8, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v9, v1, 12, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v1
-; GFX9-DL-NEXT:    v_bfe_u32 v16, v2, 12, 4
-; GFX9-DL-NEXT:    v_and_b32_e32 v17, 15, v2
-; GFX9-DL-NEXT:    v_bfe_u32 v1, v1, 4, 4
-; GFX9-DL-NEXT:    v_bfe_u32 v2, v2, 4, 4
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v12, v6, v13
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v7, v7, v14 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v18, v5, v11
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v2, v1, v2 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v8, v8, v15
-; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v9, v9, v16 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v10, v10, v17
-; GFX9-DL-NEXT:    v_or_b32_e32 v7, v12, v7
-; GFX9-DL-NEXT:    v_or_b32_e32 v8, v8, v9
-; GFX9-DL-NEXT:    v_or_b32_e32 v1, v18, v0
-; GFX9-DL-NEXT:    v_or_b32_e32 v9, v10, v2
-; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v10, 16, v7
-; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v1, v10 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v8
-; GFX9-DL-NEXT:    v_or_b32_e32 v2, v2, v0
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 8, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v10, 20, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v12, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v13, 12, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v16, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_sdwa v18, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v6, v6, v11 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v11, v16, v18
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v9, 4, v1
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v14, 4, v2
+; GFX9-DL-NEXT:    v_and_b32_e32 v15, 15, v1
+; GFX9-DL-NEXT:    v_and_b32_sdwa v17, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v1, 15, v2
+; GFX9-DL-NEXT:    v_and_b32_sdwa v2, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_and_b32_e32 v8, 15, v8
+; GFX9-DL-NEXT:    v_and_b32_e32 v13, 15, v13
+; GFX9-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX9-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX9-DL-NEXT:    v_and_b32_e32 v5, 15, v5
+; GFX9-DL-NEXT:    v_and_b32_e32 v10, 15, v10
+; GFX9-DL-NEXT:    v_and_b32_e32 v0, 15, v9
+; GFX9-DL-NEXT:    v_and_b32_e32 v9, 15, v14
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v14, v17, v2
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v5, v5, v10 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_or_b32_e32 v6, v11, v6
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v7, v7, v12
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v8, v8, v13 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_mul_lo_u16_sdwa v9, v0, v9 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-DL-NEXT:    v_or_b32_e32 v0, v14, v5
+; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v5, 16, v6
+; GFX9-DL-NEXT:    v_mul_lo_u16_e32 v1, v15, v1
+; GFX9-DL-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX9-DL-NEXT:    v_or_b32_e32 v8, v1, v9
+; GFX9-DL-NEXT:    v_or_b32_sdwa v1, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v7
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v5, 8, v1
+; GFX9-DL-NEXT:    v_or_b32_e32 v9, v9, v0
 ; GFX9-DL-NEXT:    v_lshrrev_b64 v[0:1], 24, v[0:1]
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v2, 8, v2
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v9
 ; GFX9-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v9, v4
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v2
-; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v8
+; GFX9-DL-NEXT:    v_add_u16_e32 v4, v8, v4
+; GFX9-DL-NEXT:    v_add_u16_e32 v1, v4, v1
+; GFX9-DL-NEXT:    v_add_u16_e32 v1, v1, v7
 ; GFX9-DL-NEXT:    v_add_u16_e32 v0, v1, v0
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v5, v11, v0
-; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v10
-; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v7
-; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v6, v13, v0
-; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v7
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v17, v2, v0
+; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v5
+; GFX9-DL-NEXT:    v_lshrrev_b32_e32 v6, 8, v6
+; GFX9-DL-NEXT:    v_mad_legacy_u16 v0, v16, v18, v0
+; GFX9-DL-NEXT:    v_add_u16_e32 v0, v0, v6
 ; GFX9-DL-NEXT:    global_store_byte v3, v0, s[2:3]
 ; GFX9-DL-NEXT:    s_endpgm
 ;
 ; GFX10-DL-LABEL: udot8_acc8_vecMul:
 ; GFX10-DL:       ; %bb.0: ; %entry
 ; GFX10-DL-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX10-DL-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX10-DL-NEXT:    v_mov_b32_e32 v19, 0
 ; GFX10-DL-NEXT:    s_mov_b32 s8, SCRATCH_RSRC_DWORD0
 ; GFX10-DL-NEXT:    s_mov_b32 s9, SCRATCH_RSRC_DWORD1
 ; GFX10-DL-NEXT:    s_mov_b32 s10, -1
 ; GFX10-DL-NEXT:    s_mov_b32 s11, 0x31c16000
 ; GFX10-DL-NEXT:    s_add_u32 s8, s8, s3
 ; GFX10-DL-NEXT:    s_addc_u32 s9, s9, 0
 ; GFX10-DL-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX10-DL-NEXT:    s_clause 0x1
 ; GFX10-DL-NEXT:    global_load_dword v1, v0, s[4:5]
 ; GFX10-DL-NEXT:    global_load_dword v2, v0, s[6:7]
+; GFX10-DL-NEXT:    v_mov_b32_e32 v0, 15
 ; GFX10-DL-NEXT:    global_load_ubyte v3, v19, s[0:1]
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(2)
-; GFX10-DL-NEXT:    v_bfe_u32 v9, v1, 12, 4
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v8, 12, v1
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(1)
-; GFX10-DL-NEXT:    v_bfe_u32 v10, v2, 12, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v8, v1, 8, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v13, v2, 8, 4
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 28, v1
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v14, 28, v2
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v9, v9, v10
-; GFX10-DL-NEXT:    v_bfe_u32 v5, v1, 16, 4
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v8, v8, v13
-; GFX10-DL-NEXT:    v_bfe_u32 v0, v1, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v6, v1, 24, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v11, 15, v1
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v9, 8, v9
-; GFX10-DL-NEXT:    v_bfe_u32 v1, v1, 4, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v15, v2, 4, 4
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v7, v7, v14
-; GFX10-DL-NEXT:    v_bfe_u32 v10, v2, 20, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v13, v2, 24, 4
-; GFX10-DL-NEXT:    v_bfe_u32 v23, v2, 16, 4
-; GFX10-DL-NEXT:    v_and_b32_e32 v2, 15, v2
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v1, v1, v15
-; GFX10-DL-NEXT:    v_or_b32_e32 v8, v8, v9
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v9, v0, v10
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v10, v6, v13
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v7, 8, v7
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, v11, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v13, 12, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v7, 8, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v12, 8, v2
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v5, 20, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v6, 28, v1
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v9, 4, v1
+; GFX10-DL-NEXT:    v_and_b32_e32 v15, 15, v1
+; GFX10-DL-NEXT:    v_and_b32_sdwa v31, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v17, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_e32 v1, 15, v8
+; GFX10-DL-NEXT:    v_and_b32_e32 v8, 15, v13
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v14, 4, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v7, 15, v7
+; GFX10-DL-NEXT:    v_and_b32_e32 v12, 15, v12
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v11, 28, v2
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v1, v1, v8
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 20, v2
+; GFX10-DL-NEXT:    v_and_b32_e32 v13, 15, v2
+; GFX10-DL-NEXT:    v_and_b32_sdwa v18, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_3 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_sdwa v27, v2, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_and_b32_e32 v0, 15, v9
+; GFX10-DL-NEXT:    v_and_b32_e32 v2, 15, v14
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v7, v7, v12
 ; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v1, 8, v1
-; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v8
-; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v11, v5, v23
-; GFX10-DL-NEXT:    v_or_b32_e32 v7, v10, v7
-; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v9, 8, v9
-; GFX10-DL-NEXT:    v_or_b32_sdwa v10, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_or_b32_e32 v1, v2, v1
-; GFX10-DL-NEXT:    v_or_b32_e32 v2, v11, v9
-; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v9, 16, v7
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 8, v10
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v6, v6, v11
+; GFX10-DL-NEXT:    v_and_b32_e32 v23, 15, v5
+; GFX10-DL-NEXT:    v_and_b32_e32 v9, 15, v10
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v0, v0, v2
+; GFX10-DL-NEXT:    v_or_b32_e32 v7, v7, v1
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v5, 8, v6
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v11, v31, v18
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v2, v23, v9
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v6, 8, v0
+; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v0, 16, v7
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v9, v15, v13
+; GFX10-DL-NEXT:    v_mul_lo_u16_e64 v15, v17, v27
+; GFX10-DL-NEXT:    v_lshlrev_b16_e64 v2, 8, v2
+; GFX10-DL-NEXT:    v_or_b32_e32 v5, v11, v5
+; GFX10-DL-NEXT:    v_or_b32_sdwa v1, v6, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX10-DL-NEXT:    v_or_b32_e32 v6, v9, v6
+; GFX10-DL-NEXT:    v_or_b32_e32 v2, v15, v2
+; GFX10-DL-NEXT:    v_lshlrev_b32_e32 v9, 16, v5
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v10, 8, v1
 ; GFX10-DL-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v1, v3
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v3, v6, v3
 ; GFX10-DL-NEXT:    v_or_b32_sdwa v1, v2, v9 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v9, v3, v10
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v6, v3, v10
 ; GFX10-DL-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
 ; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v1
-; GFX10-DL-NEXT:    v_add_nc_u16_e64 v0, v9, v8
+; GFX10-DL-NEXT:    v_add_nc_u16_e64 v0, v6, v7
 ; GFX10-DL-NEXT:    v_add_nc_u16_e64 v0, v0, v2
-; GFX10-DL-NEXT:    v_mad_u16 v0, v5, v23, v0
+; GFX10-DL-NEXT:    v_mad_u16 v0, v17, v27, v0
 ; GFX10-DL-NEXT:    v_add_nc_u16_e64 v0, v0, v1
-; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v7
-; GFX10-DL-NEXT:    v_mad_u16 v0, v6, v13, v0
+; GFX10-DL-NEXT:    v_lshrrev_b32_e32 v1, 8, v5
+; GFX10-DL-NEXT:    v_mad_u16 v0, v31, v18, v0
 ; GFX10-DL-NEXT:    v_add_nc_u16_e64 v0, v0, v1
 ; GFX10-DL-NEXT:    global_store_byte v19, v0, s[0:1]
 ; GFX10-DL-NEXT:    s_endpgm
                                              <8 x i4> addrspace(1)* %src2,
                                              i8 addrspace(1)* nocapture %dst) {
 entry:
   %idx = call i32 @llvm.amdgcn.workitem.id.x()
   %gep1 = getelementptr <8 x i4>, <8 x i4> addrspace(1)* %src1, i32 %idx

llvm/test/CodeGen/AMDGPU/widen-smrd-loads.ll

 ; VI-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x24
 ; VI-NEXT:    v_mov_b32_e32 v0, 44
 ; VI-NEXT:    v_mov_b32_e32 v1, 3
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_load_dword s0, s[0:1], 0x0
 ; VI-NEXT:    s_waitcnt lgkmcnt(0)
 ; VI-NEXT:    s_and_b32 s1, s0, 0xffff
 ; VI-NEXT:    v_mov_b32_e32 v2, s0
-; VI-NEXT:    s_add_i32 s1, s1, 12
 ; VI-NEXT:    v_add_u32_sdwa v0, vcc, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:BYTE_1
-; VI-NEXT:    s_or_b32 s0, s1, 4
+; VI-NEXT:    s_add_i32 s1, s1, 12
 ; VI-NEXT:    v_or_b32_sdwa v0, v0, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; VI-NEXT:    v_mov_b32_e32 v1, s0
+; VI-NEXT:    v_or_b32_e64 v1, s1, 4
 ; VI-NEXT:    v_or_b32_sdwa v2, v1, v0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
 ; VI-NEXT:    v_mov_b32_e32 v0, 0
 ; VI-NEXT:    v_mov_b32_e32 v1, 0
 ; VI-NEXT:    flat_store_short v[0:1], v2
 ; VI-NEXT:    s_endpgm
   %load = load <2 x i8>, <2 x i8> addrspace(4)* %arg, align 4
   %add = add <2 x i8> %load, <i8 12, i8 44>
   %or = or <2 x i8> %add, <i8 4, i8 3>
   store <2 x i8> %or, <2 x i8> addrspace(1)* null

llvm/test/CodeGen/SystemZ/scalar-ctlz.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=s390x-linux-gnu -mcpu=z13 | FileCheck %s
 ;
 ; FIXME: two consecutive immediate adds not fused in i16/i8 functions.
 
 declare i64 @llvm.ctlz.i64(i64, i1)
 declare i32 @llvm.ctlz.i32(i32, i1)
 declare i16 @llvm.ctlz.i16(i16, i1)
 declare i8 @llvm.ctlz.i8(i8, i1)
 
 define i64 @f0(i64 %arg) {
 ; CHECK-LABEL: f0:
-; CHECK-LABEL: %bb.0:
-; CHECK-NOT:   %bb.1:
-; CHECK: flogr
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    flogr %r2, %r2
+; CHECK-NEXT:    # kill: def $r2d killed $r2d killed $r2q
+; CHECK-NEXT:    br %r14
   %1 = tail call i64 @llvm.ctlz.i64(i64 %arg, i1 false)
   ret i64 %1
 }
 
 define i64 @f1(i64 %arg) {
 ; CHECK-LABEL: f1:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: flogr
-; CHECK-NEXT: # kill
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    flogr %r2, %r2
+; CHECK-NEXT:    # kill: def $r2d killed $r2d killed $r2q
+; CHECK-NEXT:    br %r14
   %1 = tail call i64 @llvm.ctlz.i64(i64 %arg, i1 true)
   ret i64 %1
 }
 
 define i32 @f2(i32 %arg) {
 ; CHECK-LABEL: f2:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: llgfr %r0, %r2
-; CHECK-NEXT: flogr %r2, %r0
-; CHECK-NEXT: aghi  %r2, -32
-; CHECK-NEXT: # kill
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    llgfr %r0, %r2
+; CHECK-NEXT:    flogr %r2, %r0
+; CHECK-NEXT:    ahi %r2, -32
+; CHECK-NEXT:    br %r14
   %1 = tail call i32 @llvm.ctlz.i32(i32 %arg, i1 false)
   ret i32 %1
 }
 
 define i32 @f3(i32 %arg) {
 ; CHECK-LABEL: f3:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: llgfr %r0, %r2
-; CHECK-NEXT: flogr %r2, %r0
-; CHECK-NEXT: aghi  %r2, -32
-; CHECK-NEXT: # kill
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    llgfr %r0, %r2
+; CHECK-NEXT:    flogr %r2, %r0
+; CHECK-NEXT:    ahi %r2, -32
+; CHECK-NEXT:    br %r14
   %1 = tail call i32 @llvm.ctlz.i32(i32 %arg, i1 true)
   ret i32 %1
 }
 
 define i16 @f4(i16 %arg) {
 ; CHECK-LABEL: f4:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: # kill
-; CHECK-NEXT: llghr %r0, %r2
-; CHECK-NEXT: flogr %r0, %r0
-; CHECK-NEXT: aghi  %r0, -32
-; CHECK-NEXT: ahik  %r2, %r0, -16
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $r2l killed $r2l def $r2d
+; CHECK-NEXT:    llghr %r0, %r2
+; CHECK-NEXT:    flogr %r2, %r0
+; CHECK-NEXT:    ahi %r2, -48
+; CHECK-NEXT:    br %r14
   %1 = tail call i16 @llvm.ctlz.i16(i16 %arg, i1 false)
   ret i16 %1
 }
 
 define i16 @f5(i16 %arg) {
 ; CHECK-LABEL: f5:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: # kill
-; CHECK-NEXT: llghr %r0, %r2
-; CHECK-NEXT: flogr %r0, %r0
-; CHECK-NEXT: aghi  %r0, -32
-; CHECK-NEXT: ahik  %r2, %r0, -16
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $r2l killed $r2l def $r2d
+; CHECK-NEXT:    llghr %r0, %r2
+; CHECK-NEXT:    flogr %r2, %r0
+; CHECK-NEXT:    ahi %r2, -48
+; CHECK-NEXT:    br %r14
   %1 = tail call i16 @llvm.ctlz.i16(i16 %arg, i1 true)
   ret i16 %1
 }
 
 define i8 @f6(i8 %arg) {
 ; CHECK-LABEL: f6:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: # kill
-; CHECK-NEXT: llgcr %r0, %r2
-; CHECK-NEXT: flogr %r0, %r0
-; CHECK-NEXT: aghi  %r0, -32
-; CHECK-NEXT: ahik  %r2, %r0, -24
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $r2l killed $r2l def $r2d
+; CHECK-NEXT:    llgcr %r0, %r2
+; CHECK-NEXT:    flogr %r2, %r0
+; CHECK-NEXT:    ahi %r2, -56
+; CHECK-NEXT:    br %r14
   %1 = tail call i8 @llvm.ctlz.i8(i8 %arg, i1 false)
   ret i8 %1
 }
 
 define i8 @f7(i8 %arg) {
 ; CHECK-LABEL: f7:
-; CHECK-LABEL: %bb.0:
-; CHECK-NEXT: # kill
-; CHECK-NEXT: llgcr %r0, %r2
-; CHECK-NEXT: flogr %r0, %r0
-; CHECK-NEXT: aghi  %r0, -32
-; CHECK-NEXT: ahik  %r2, %r0, -24
-; CHECK-NEXT: br %r14
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    # kill: def $r2l killed $r2l def $r2d
+; CHECK-NEXT:    llgcr %r0, %r2
+; CHECK-NEXT:    flogr %r2, %r0
+; CHECK-NEXT:    ahi %r2, -56
+; CHECK-NEXT:    br %r14
   %1 = tail call i8 @llvm.ctlz.i8(i8 %arg, i1 true)
   ret i8 %1
 }

llvm/test/CodeGen/X86/and-encoding.ll

 ; CHECK-NEXT:    retq # encoding: [0xc3]
   %foo = alloca i8, align 32
   ret void
 }
 
 define void @f2(i16 %x, i1 *%y) nounwind  {
 ; CHECK-LABEL: f2:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    andl $1, %edi # encoding: [0x83,0xe7,0x01]
+; CHECK-NEXT:    andb $1, %dil # encoding: [0x40,0x80,0xe7,0x01]
 ; CHECK-NEXT:    movb %dil, (%rsi) # encoding: [0x40,0x88,0x3e]
 ; CHECK-NEXT:    retq # encoding: [0xc3]
   %c = trunc i16 %x to i1
   store i1 %c, i1* %y
   ret void
 }
 
 define void @f3(i32 %x, i1 *%y) nounwind {
 ; CHECK-LABEL: f3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    andl $1, %edi # encoding: [0x83,0xe7,0x01]
+; CHECK-NEXT:    andb $1, %dil # encoding: [0x40,0x80,0xe7,0x01]
 ; CHECK-NEXT:    movb %dil, (%rsi) # encoding: [0x40,0x88,0x3e]
 ; CHECK-NEXT:    retq # encoding: [0xc3]
   %c = trunc i32 %x to i1
   store i1 %c, i1* %y
   ret void
 }
 
 ; The immediate (0x0ffffff0) can be made into an i8 by making it negative.

llvm/test/CodeGen/X86/avx512-calling-conv.ll

 ; KNL-NEXT:    kmovw %k1, %ebp
 ; KNL-NEXT:    kshiftrw $11, %k0, %k1
 ; KNL-NEXT:    kmovw %k1, %ecx
 ; KNL-NEXT:    kshiftrw $12, %k0, %k1
 ; KNL-NEXT:    kmovw %k1, %edx
 ; KNL-NEXT:    kshiftrw $13, %k0, %k1
 ; KNL-NEXT:    kmovw %k1, %edi
 ; KNL-NEXT:    kshiftrw $14, %k0, %k1
-; KNL-NEXT:    andl $1, %r8d
+; KNL-NEXT:    andb $1, %r8b
 ; KNL-NEXT:    movb %r8b, 2(%rax)
 ; KNL-NEXT:    kmovw %k0, %r8d
 ; KNL-NEXT:    andl $1, %r8d
 ; KNL-NEXT:    andl $1, %r9d
 ; KNL-NEXT:    leal (%r8,%r9,2), %r8d
 ; KNL-NEXT:    kmovw %k1, %r9d
 ; KNL-NEXT:    kshiftrw $15, %k0, %k0
 ; KNL-NEXT:    andl $1, %r10d

 ; SKX-NEXT:    kmovd %k1, %ebp
 ; SKX-NEXT:    kshiftrd $11, %k0, %k1
 ; SKX-NEXT:    kmovd %k1, %ecx
 ; SKX-NEXT:    kshiftrd $12, %k0, %k1
 ; SKX-NEXT:    kmovd %k1, %edx
 ; SKX-NEXT:    kshiftrd $13, %k0, %k1
 ; SKX-NEXT:    kmovd %k1, %edi
 ; SKX-NEXT:    kshiftrd $14, %k0, %k1
-; SKX-NEXT:    andl $1, %r8d
+; SKX-NEXT:    andb $1, %r8b
 ; SKX-NEXT:    movb %r8b, 2(%rax)
 ; SKX-NEXT:    kmovd %k0, %r8d
 ; SKX-NEXT:    andl $1, %r8d
 ; SKX-NEXT:    andl $1, %r9d
 ; SKX-NEXT:    leal (%r8,%r9,2), %r8d
 ; SKX-NEXT:    kmovd %k1, %r9d
 ; SKX-NEXT:    kshiftrd $15, %k0, %k0
 ; SKX-NEXT:    andl $1, %r10d

 ; KNL_X32-NEXT:    kmovw %k1, %edi
 ; KNL_X32-NEXT:    kshiftrw $3, %k0, %k1
 ; KNL_X32-NEXT:    kmovw %k1, %ebp
 ; KNL_X32-NEXT:    kshiftrw $4, %k0, %k1
 ; KNL_X32-NEXT:    kmovw %k1, %edx
 ; KNL_X32-NEXT:    kshiftrw $5, %k0, %k1
 ; KNL_X32-NEXT:    kmovw %k1, %ecx
 ; KNL_X32-NEXT:    kshiftrw $6, %k0, %k1
-; KNL_X32-NEXT:    andl $1, %ebx
+; KNL_X32-NEXT:    andb $1, %bl
 ; KNL_X32-NEXT:    movb %bl, 2(%eax)
 ; KNL_X32-NEXT:    kmovw %k0, %ebx
 ; KNL_X32-NEXT:    andl $1, %ebx
 ; KNL_X32-NEXT:    andl $1, %esi
 ; KNL_X32-NEXT:    leal (%ebx,%esi,2), %esi
 ; KNL_X32-NEXT:    kmovw %k1, %ebx
 ; KNL_X32-NEXT:    kshiftrw $7, %k0, %k1
 ; KNL_X32-NEXT:    andl $1, %edi

 ; FASTISEL-NEXT:    kmovd %k1, %ebp
 ; FASTISEL-NEXT:    kshiftrd $11, %k0, %k1
 ; FASTISEL-NEXT:    kmovd %k1, %ecx
 ; FASTISEL-NEXT:    kshiftrd $12, %k0, %k1
 ; FASTISEL-NEXT:    kmovd %k1, %edx
 ; FASTISEL-NEXT:    kshiftrd $13, %k0, %k1
 ; FASTISEL-NEXT:    kmovd %k1, %edi
 ; FASTISEL-NEXT:    kshiftrd $14, %k0, %k1
-; FASTISEL-NEXT:    andl $1, %r8d
+; FASTISEL-NEXT:    andb $1, %r8b
 ; FASTISEL-NEXT:    movb %r8b, 2(%rax)
 ; FASTISEL-NEXT:    kmovd %k0, %r8d
 ; FASTISEL-NEXT:    andl $1, %r8d
 ; FASTISEL-NEXT:    andl $1, %r9d
 ; FASTISEL-NEXT:    leal (%r8,%r9,2), %r8d
 ; FASTISEL-NEXT:    kmovd %k1, %r9d
 ; FASTISEL-NEXT:    kshiftrd $15, %k0, %k0
 ; FASTISEL-NEXT:    andl $1, %r10d

llvm/test/CodeGen/X86/avx512-mask-op.ll

entry:
   ret void
 }
 
 declare void @f2(i32) #1
 
 define void @store_i16_i1(i16 %x, i1 *%y) {
 ; CHECK-LABEL: store_i16_i1:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    andl $1, %edi
+; CHECK-NEXT:    andb $1, %dil
 ; CHECK-NEXT:    movb %dil, (%rsi)
 ; CHECK-NEXT:    retq
 ;
 ; X86-LABEL: store_i16_i1:
 ; X86:       ## %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
-; X86-NEXT:    andl $1, %ecx
+; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X86-NEXT:    andb $1, %cl
 ; X86-NEXT:    movb %cl, (%eax)
 ; X86-NEXT:    retl
   %c = trunc i16 %x to i1
   store i1 %c, i1* %y
   ret void
 }
 
 define void @store_i8_i1(i8 %x, i1 *%y) {
 ; CHECK-LABEL: store_i8_i1:
 ; CHECK:       ## %bb.0:
-; CHECK-NEXT:    andl $1, %edi
+; CHECK-NEXT:    andb $1, %dil
 ; CHECK-NEXT:    movb %dil, (%rsi)
 ; CHECK-NEXT:    retq
 ;
 ; X86-LABEL: store_i8_i1:
 ; X86:       ## %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %cl
 ; X86-NEXT:    andb $1, %cl

llvm/test/CodeGen/X86/bool-math.ll

 
 define i1 @opaque_constant(i48 %x, i48 %y) {
 ; X64-LABEL: opaque_constant:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rsi, %rax
 ; X64-NEXT:    shrq $32, %rdi
 ; X64-NEXT:    shrq $32, %rax
 ; X64-NEXT:    xorl %edi, %eax
-; X64-NEXT:    andl $1, %eax
+; X64-NEXT:    andb $1, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $rax
 ; X64-NEXT:    retq
 ;
 ; X32-LABEL: opaque_constant:
 ; X32:       # %bb.0:
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    xorl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    andl $1, %eax

llvm/test/CodeGen/X86/clz.ll

; X64-CLZ-NEXT:    retq
   ret i64 %tmp
 }
 
 define i8 @ctlz_i8(i8 %x) {
 ; X86-LABEL: ctlz_i8:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    bsrl %eax, %eax
-; X86-NEXT:    xorl $7, %eax
+; X86-NEXT:    xorb $7, %al
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: ctlz_i8:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    bsrl %eax, %eax
-; X64-NEXT:    xorl $7, %eax
+; X64-NEXT:    xorb $7, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ;
 ; X86-CLZ-LABEL: ctlz_i8:
 ; X86-CLZ:       # %bb.0:
 ; X86-CLZ-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-CLZ-NEXT:    lzcntl %eax, %eax
-; X86-CLZ-NEXT:    addl $-24, %eax
+; X86-CLZ-NEXT:    addb $-24, %al
 ; X86-CLZ-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-CLZ-NEXT:    retl
 ;
 ; X64-CLZ-LABEL: ctlz_i8:
 ; X64-CLZ:       # %bb.0:
 ; X64-CLZ-NEXT:    movzbl %dil, %eax
 ; X64-CLZ-NEXT:    lzcntl %eax, %eax
-; X64-CLZ-NEXT:    addl $-24, %eax
+; X64-CLZ-NEXT:    addb $-24, %al
 ; X64-CLZ-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-CLZ-NEXT:    retq
   %tmp2 = call i8 @llvm.ctlz.i8( i8 %x, i1 true )
   ret i8 %tmp2
 }
 
 define i16 @ctlz_i16(i16 %x) {
 ; X86-LABEL: ctlz_i16:

 ; X86-LABEL: ctlz_i8_zero_test:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X86-NEXT:    testb %al, %al
 ; X86-NEXT:    je .LBB8_1
 ; X86-NEXT:  # %bb.2: # %cond.false
 ; X86-NEXT:    movzbl %al, %eax
 ; X86-NEXT:    bsrl %eax, %eax
-; X86-NEXT:    xorl $7, %eax
+; X86-NEXT:    xorb $7, %al
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
 ; X86-NEXT:  .LBB8_1:
 ; X86-NEXT:    movb $8, %al
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: ctlz_i8_zero_test:
 ; X64:       # %bb.0:
 ; X64-NEXT:    testb %dil, %dil
 ; X64-NEXT:    je .LBB8_1
 ; X64-NEXT:  # %bb.2: # %cond.false
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    bsrl %eax, %eax
-; X64-NEXT:    xorl $7, %eax
+; X64-NEXT:    xorb $7, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB8_1:
 ; X64-NEXT:    movb $8, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ;
 ; X86-CLZ-LABEL: ctlz_i8_zero_test:
 ; X86-CLZ:       # %bb.0:
 ; X86-CLZ-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-CLZ-NEXT:    lzcntl %eax, %eax
-; X86-CLZ-NEXT:    addl $-24, %eax
+; X86-CLZ-NEXT:    addb $-24, %al
 ; X86-CLZ-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-CLZ-NEXT:    retl
 ;
 ; X64-CLZ-LABEL: ctlz_i8_zero_test:
 ; X64-CLZ:       # %bb.0:
 ; X64-CLZ-NEXT:    movzbl %dil, %eax
 ; X64-CLZ-NEXT:    lzcntl %eax, %eax
-; X64-CLZ-NEXT:    addl $-24, %eax
+; X64-CLZ-NEXT:    addb $-24, %al
 ; X64-CLZ-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-CLZ-NEXT:    retq
   %tmp1 = call i8 @llvm.ctlz.i8(i8 %n, i1 false)
   ret i8 %tmp1
 }
 
 ; Generate a test and branch to handle zero inputs because bsr/bsf are very slow.
 define i16 @ctlz_i16_zero_test(i16 %n) {

 
 define i8 @ctlz_i8_knownbits(i8 %x)  {
 ; X86-LABEL: ctlz_i8_knownbits:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X86-NEXT:    orb $64, %al
 ; X86-NEXT:    movzbl %al, %eax
 ; X86-NEXT:    bsrl %eax, %eax
-; X86-NEXT:    xorl $7, %eax
+; X86-NEXT:    xorb $7, %al
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: ctlz_i8_knownbits:
 ; X64:       # %bb.0:
 ; X64-NEXT:    orb $64, %dil
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    bsrl %eax, %eax
-; X64-NEXT:    xorl $7, %eax
+; X64-NEXT:    xorb $7, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ;
 ; X86-CLZ-LABEL: ctlz_i8_knownbits:
 ; X86-CLZ:       # %bb.0:
 ; X86-CLZ-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; X86-CLZ-NEXT:    orb $64, %al
 ; X86-CLZ-NEXT:    movzbl %al, %eax
 ; X86-CLZ-NEXT:    lzcntl %eax, %eax
-; X86-CLZ-NEXT:    addl $-24, %eax
+; X86-CLZ-NEXT:    addb $-24, %al
 ; X86-CLZ-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-CLZ-NEXT:    retl
 ;
 ; X64-CLZ-LABEL: ctlz_i8_knownbits:
 ; X64-CLZ:       # %bb.0:
 ; X64-CLZ-NEXT:    orb $64, %dil
 ; X64-CLZ-NEXT:    movzbl %dil, %eax
 ; X64-CLZ-NEXT:    lzcntl %eax, %eax
-; X64-CLZ-NEXT:    addl $-24, %eax
+; X64-CLZ-NEXT:    addb $-24, %al
 ; X64-CLZ-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-CLZ-NEXT:    retq
 
   %x2 = or i8 %x, 64
   %tmp = call i8 @llvm.ctlz.i8(i8 %x2, i1 true )
   %tmp2 = and i8 %tmp, 1
   ret i8 %tmp2
 }

llvm/test/CodeGen/X86/fast-isel-cmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s                               -mtriple=x86_64-apple-darwin10 | FileCheck %s --check-prefix=SDAG
 ; RUN: llc < %s -fast-isel -fast-isel-abort=1 -mtriple=x86_64-apple-darwin10 | FileCheck %s --check-prefixes=FAST,FAST_NOAVX
 ; RUN: llc < %s -fast-isel -fast-isel-abort=1 -mtriple=x86_64-apple-darwin10 -mattr=avx | FileCheck %s --check-prefixes=FAST,FAST_AVX
 ; RUN: llc < %s -fast-isel -fast-isel-abort=1 -mtriple=x86_64-apple-darwin10 -mattr=avx512f | FileCheck %s --check-prefixes=FAST,FAST_AVX
 
 define zeroext i1 @fcmp_oeq(float %x, float %y) {
 ; SDAG-LABEL: fcmp_oeq:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    cmpeqss %xmm1, %xmm0
 ; SDAG-NEXT:    movd %xmm0, %eax
-; SDAG-NEXT:    andl $1, %eax
+; SDAG-NEXT:    andb $1, %al
 ; SDAG-NEXT:    ## kill: def $al killed $al killed $eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST_NOAVX-LABEL: fcmp_oeq:
 ; FAST_NOAVX:       ## %bb.0:
 ; FAST_NOAVX-NEXT:    ucomiss %xmm1, %xmm0
 ; FAST_NOAVX-NEXT:    sete %al
 ; FAST_NOAVX-NEXT:    setnp %cl
; FAST_AVX-NEXT:    retq
   ret i1 %1
 }
 
 define zeroext i1 @fcmp_une(float %x, float %y) {
 ; SDAG-LABEL: fcmp_une:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    cmpneqss %xmm1, %xmm0
 ; SDAG-NEXT:    movd %xmm0, %eax
-; SDAG-NEXT:    andl $1, %eax
+; SDAG-NEXT:    andb $1, %al
 ; SDAG-NEXT:    ## kill: def $al killed $al killed $eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST_NOAVX-LABEL: fcmp_une:
 ; FAST_NOAVX:       ## %bb.0:
 ; FAST_NOAVX-NEXT:    ucomiss %xmm1, %xmm0
 ; FAST_NOAVX-NEXT:    setne %al
 ; FAST_NOAVX-NEXT:    setp %cl

 }
 
 define zeroext i1 @fcmp_oeq3(float %x) {
 ; SDAG-LABEL: fcmp_oeq3:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    xorps %xmm1, %xmm1
 ; SDAG-NEXT:    cmpeqss %xmm0, %xmm1
 ; SDAG-NEXT:    movd %xmm1, %eax
-; SDAG-NEXT:    andl $1, %eax
+; SDAG-NEXT:    andb $1, %al
 ; SDAG-NEXT:    ## kill: def $al killed $al killed $eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST_NOAVX-LABEL: fcmp_oeq3:
 ; FAST_NOAVX:       ## %bb.0:
 ; FAST_NOAVX-NEXT:    xorps %xmm1, %xmm1
 ; FAST_NOAVX-NEXT:    ucomiss %xmm1, %xmm0
 ; FAST_NOAVX-NEXT:    sete %al

 }
 
 define zeroext i1 @fcmp_une3(float %x) {
 ; SDAG-LABEL: fcmp_une3:
 ; SDAG:       ## %bb.0:
 ; SDAG-NEXT:    xorps %xmm1, %xmm1
 ; SDAG-NEXT:    cmpneqss %xmm0, %xmm1
 ; SDAG-NEXT:    movd %xmm1, %eax
-; SDAG-NEXT:    andl $1, %eax
+; SDAG-NEXT:    andb $1, %al
 ; SDAG-NEXT:    ## kill: def $al killed $al killed $eax
 ; SDAG-NEXT:    retq
 ;
 ; FAST_NOAVX-LABEL: fcmp_une3:
 ; FAST_NOAVX:       ## %bb.0:
 ; FAST_NOAVX-NEXT:    xorps %xmm1, %xmm1
 ; FAST_NOAVX-NEXT:    ucomiss %xmm1, %xmm0
 ; FAST_NOAVX-NEXT:    setne %al

llvm/test/CodeGen/X86/fptosi-sat-scalar.ll

 ; X86-X87-NEXT:    movl %edi, %edx
 ; X86-X87-NEXT:    movl %esi, %eax
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebp # 4-byte Reload
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB8_12:
 ; X86-X87-NEXT:    movl %ebx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %eax, (%ecx)
-; X86-X87-NEXT:    andl $15, %edx
+; X86-X87-NEXT:    andb $15, %dl
 ; X86-X87-NEXT:    movb %dl, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $44, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    ucomiss %xmm0, %xmm0
 ; X86-SSE-NEXT:    cmovpl %ebp, %eax
 ; X86-SSE-NEXT:    cmovpl %ebp, %ecx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edi
 ; X86-SSE-NEXT:    movl %edi, 8(%esi)
 ; X86-SSE-NEXT:    movl %edx, 4(%esi)
 ; X86-SSE-NEXT:    movl %ecx, (%esi)
-; X86-SSE-NEXT:    andl $15, %eax
+; X86-SSE-NEXT:    andb $15, %al
 ; X86-SSE-NEXT:    movb %al, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $28, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    popl %ebp
 ; X86-SSE-NEXT:    retl $4

 ; X86-X87-NEXT:    movl %edi, %edx
 ; X86-X87-NEXT:    movl %esi, %eax
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebp # 4-byte Reload
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB18_12:
 ; X86-X87-NEXT:    movl %ebx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %eax, (%ecx)
-; X86-X87-NEXT:    andl $15, %edx
+; X86-X87-NEXT:    andb $15, %dl
 ; X86-X87-NEXT:    movb %dl, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $60, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    ucomisd %xmm0, %xmm0
 ; X86-SSE-NEXT:    cmovpl %ebp, %eax
 ; X86-SSE-NEXT:    cmovpl %ebp, %ecx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edi
 ; X86-SSE-NEXT:    movl %edi, 8(%esi)
 ; X86-SSE-NEXT:    movl %edx, 4(%esi)
 ; X86-SSE-NEXT:    movl %ecx, (%esi)
-; X86-SSE-NEXT:    andl $15, %eax
+; X86-SSE-NEXT:    andb $15, %al
 ; X86-SSE-NEXT:    movb %al, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $44, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    popl %ebp
 ; X86-SSE-NEXT:    retl $4

 ; X86-X87-NEXT:    movl %edi, %edx
 ; X86-X87-NEXT:    movl %esi, %eax
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebp # 4-byte Reload
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB28_12:
 ; X86-X87-NEXT:    movl %ebx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %eax, (%ecx)
-; X86-X87-NEXT:    andl $15, %edx
+; X86-X87-NEXT:    andb $15, %dl
 ; X86-X87-NEXT:    movb %dl, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $44, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    ucomiss %xmm0, %xmm0
 ; X86-SSE-NEXT:    cmovpl %ebp, %eax
 ; X86-SSE-NEXT:    cmovpl %ebp, %ecx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edi
 ; X86-SSE-NEXT:    movl %edi, 8(%esi)
 ; X86-SSE-NEXT:    movl %edx, 4(%esi)
 ; X86-SSE-NEXT:    movl %ecx, (%esi)
-; X86-SSE-NEXT:    andl $15, %eax
+; X86-SSE-NEXT:    andb $15, %al
 ; X86-SSE-NEXT:    movb %al, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $44, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    popl %ebp
 ; X86-SSE-NEXT:    retl $4

 ; X86-X87-NEXT:    movl %edi, %edx
 ; X86-X87-NEXT:    movl %esi, %eax
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebp # 4-byte Reload
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %ebx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB38_12:
 ; X86-X87-NEXT:    movl %ebx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %eax, (%ecx)
-; X86-X87-NEXT:    andl $15, %edx
+; X86-X87-NEXT:    andb $15, %dl
 ; X86-X87-NEXT:    movb %dl, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $60, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    fucompi %st(0), %st
 ; X86-SSE-NEXT:    cmovpl %ebp, %eax
 ; X86-SSE-NEXT:    cmovpl %ebp, %ecx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edx
 ; X86-SSE-NEXT:    cmovpl %ebp, %edi
 ; X86-SSE-NEXT:    movl %edi, 8(%esi)
 ; X86-SSE-NEXT:    movl %edx, 4(%esi)
 ; X86-SSE-NEXT:    movl %ecx, (%esi)
-; X86-SSE-NEXT:    andl $15, %eax
+; X86-SSE-NEXT:    andb $15, %al
 ; X86-SSE-NEXT:    movb %al, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $44, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    popl %ebp
 ; X86-SSE-NEXT:    retl $4

llvm/test/CodeGen/X86/fptoui-sat-scalar.ll

 ; X86-X87-NEXT:  # %bb.9:
 ; X86-X87-NEXT:    movl %ebx, %edi
 ; X86-X87-NEXT:    movl %esi, %ebp
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB8_10:
 ; X86-X87-NEXT:    movl %edx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %edi, (%ecx)
-; X86-X87-NEXT:    andl $15, %eax
+; X86-X87-NEXT:    andb $15, %al
 ; X86-X87-NEXT:    movb %al, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $44, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    cmovbel %edi, %ebx
 ; X86-SSE-NEXT:    movl $-1, %edi
 ; X86-SSE-NEXT:    cmoval %edi, %edx
 ; X86-SSE-NEXT:    cmoval %edi, %ecx
 ; X86-SSE-NEXT:    cmoval %edi, %eax
 ; X86-SSE-NEXT:    movl %eax, 8(%esi)
 ; X86-SSE-NEXT:    movl %ecx, 4(%esi)
 ; X86-SSE-NEXT:    movl %edx, (%esi)
-; X86-SSE-NEXT:    andl $15, %ebx
+; X86-SSE-NEXT:    andb $15, %bl
 ; X86-SSE-NEXT:    movb %bl, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $32, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    retl $4
 ;

 ; X86-X87-NEXT:  # %bb.9:
 ; X86-X87-NEXT:    movl %ebx, %edi
 ; X86-X87-NEXT:    movl %esi, %ebp
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB18_10:
 ; X86-X87-NEXT:    movl %edx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %edi, (%ecx)
-; X86-X87-NEXT:    andl $15, %eax
+; X86-X87-NEXT:    andb $15, %al
 ; X86-X87-NEXT:    movb %al, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $44, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    cmovbel %edi, %ebx
 ; X86-SSE-NEXT:    movl $-1, %edi
 ; X86-SSE-NEXT:    cmoval %edi, %edx
 ; X86-SSE-NEXT:    cmoval %edi, %ecx
 ; X86-SSE-NEXT:    cmoval %edi, %eax
 ; X86-SSE-NEXT:    movl %eax, 8(%esi)
 ; X86-SSE-NEXT:    movl %ecx, 4(%esi)
 ; X86-SSE-NEXT:    movl %edx, (%esi)
-; X86-SSE-NEXT:    andl $15, %ebx
+; X86-SSE-NEXT:    andb $15, %bl
 ; X86-SSE-NEXT:    movb %bl, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $32, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    retl $4
 ;

 ; X86-X87-NEXT:  # %bb.9:
 ; X86-X87-NEXT:    movl %ebx, %edi
 ; X86-X87-NEXT:    movl %esi, %ebp
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB28_10:
 ; X86-X87-NEXT:    movl %edx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %edi, (%ecx)
-; X86-X87-NEXT:    andl $15, %eax
+; X86-X87-NEXT:    andb $15, %al
 ; X86-X87-NEXT:    movb %al, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $44, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    cmovbel %edi, %ebx
 ; X86-SSE-NEXT:    movl $-1, %edi
 ; X86-SSE-NEXT:    cmoval %edi, %edx
 ; X86-SSE-NEXT:    cmoval %edi, %ecx
 ; X86-SSE-NEXT:    cmoval %edi, %eax
 ; X86-SSE-NEXT:    movl %eax, 8(%esi)
 ; X86-SSE-NEXT:    movl %ecx, 4(%esi)
 ; X86-SSE-NEXT:    movl %edx, (%esi)
-; X86-SSE-NEXT:    andl $15, %ebx
+; X86-SSE-NEXT:    andb $15, %bl
 ; X86-SSE-NEXT:    movb %bl, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $32, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    retl $4
 ;

 ; X86-X87-NEXT:  # %bb.9:
 ; X86-X87-NEXT:    movl %ebx, %edi
 ; X86-X87-NEXT:    movl %esi, %ebp
 ; X86-X87-NEXT:    movl {{[-0-9]+}}(%e{{[sb]}}p), %edx # 4-byte Reload
 ; X86-X87-NEXT:  .LBB38_10:
 ; X86-X87-NEXT:    movl %edx, 8(%ecx)
 ; X86-X87-NEXT:    movl %ebp, 4(%ecx)
 ; X86-X87-NEXT:    movl %edi, (%ecx)
-; X86-X87-NEXT:    andl $15, %eax
+; X86-X87-NEXT:    andb $15, %al
 ; X86-X87-NEXT:    movb %al, 12(%ecx)
 ; X86-X87-NEXT:    movl %ecx, %eax
 ; X86-X87-NEXT:    addl $60, %esp
 ; X86-X87-NEXT:    popl %esi
 ; X86-X87-NEXT:    popl %edi
 ; X86-X87-NEXT:    popl %ebx
 ; X86-X87-NEXT:    popl %ebp
 ; X86-X87-NEXT:    retl $4

 ; X86-SSE-NEXT:    cmovbel %edi, %ebx
 ; X86-SSE-NEXT:    movl $-1, %edi
 ; X86-SSE-NEXT:    cmoval %edi, %edx
 ; X86-SSE-NEXT:    cmoval %edi, %ecx
 ; X86-SSE-NEXT:    cmoval %edi, %eax
 ; X86-SSE-NEXT:    movl %eax, 8(%esi)
 ; X86-SSE-NEXT:    movl %ecx, 4(%esi)
 ; X86-SSE-NEXT:    movl %edx, (%esi)
-; X86-SSE-NEXT:    andl $15, %ebx
+; X86-SSE-NEXT:    andb $15, %bl
 ; X86-SSE-NEXT:    movb %bl, 12(%esi)
 ; X86-SSE-NEXT:    movl %esi, %eax
 ; X86-SSE-NEXT:    addl $48, %esp
 ; X86-SSE-NEXT:    popl %esi
 ; X86-SSE-NEXT:    popl %edi
 ; X86-SSE-NEXT:    popl %ebx
 ; X86-SSE-NEXT:    retl $4
 ;

llvm/test/CodeGen/X86/funnel-shift.ll

 ; X64-AVX2-NEXT:    movq %rdx, %rcx
 ; X64-AVX2-NEXT:    movabsq $-2492803253203993461, %rdx # imm = 0xDD67C8A60DD67C8B
 ; X64-AVX2-NEXT:    movq %rcx, %rax
 ; X64-AVX2-NEXT:    mulq %rdx
 ; X64-AVX2-NEXT:    shrq $5, %rdx
 ; X64-AVX2-NEXT:    leal (%rdx,%rdx,8), %eax
 ; X64-AVX2-NEXT:    leal (%rdx,%rax,4), %eax
 ; X64-AVX2-NEXT:    subl %eax, %ecx
-; X64-AVX2-NEXT:    addl $27, %ecx
+; X64-AVX2-NEXT:    addb $27, %cl
 ; X64-AVX2-NEXT:    shlq $27, %rsi
 ; X64-AVX2-NEXT:    # kill: def $cl killed $cl killed $rcx
 ; X64-AVX2-NEXT:    shrdq %cl, %rdi, %rsi
 ; X64-AVX2-NEXT:    movq %rsi, %rax
 ; X64-AVX2-NEXT:    retq
   %f = call i37 @llvm.fshr.i37(i37 %x, i37 %y, i37 %z)
   ret i37 %f
 }
; X64-AVX2-NEXT:    retq
   %res = call i32 @llvm.fshl.i32(i32 undef, i32 %a0, i32 %a1)
   ret i32 %res
 }
 
 define i32 @fshl_i32_undef0_msk(i32 %a0, i32 %a1) nounwind {
 ; X32-SSE2-LABEL: fshl_i32_undef0_msk:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE2-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-SSE2-NEXT:    andl $7, %ecx
-; X32-SSE2-NEXT:    # kill: def $cl killed $cl killed $ecx
+; X32-SSE2-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X32-SSE2-NEXT:    andb $7, %cl
 ; X32-SSE2-NEXT:    shldl %cl, %eax, %eax
 ; X32-SSE2-NEXT:    retl
 ;
 ; X64-AVX2-LABEL: fshl_i32_undef0_msk:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    movl %esi, %ecx
-; X64-AVX2-NEXT:    andl $7, %ecx
+; X64-AVX2-NEXT:    andb $7, %cl
 ; X64-AVX2-NEXT:    # kill: def $cl killed $cl killed $ecx
 ; X64-AVX2-NEXT:    shldl %cl, %edi, %eax
 ; X64-AVX2-NEXT:    retq
   %m = and i32 %a1, 7
   %res = call i32 @llvm.fshl.i32(i32 undef, i32 %a0, i32 %m)
   ret i32 %res
 }
 
; X64-AVX2-NEXT:    retq
   %res = call i32 @llvm.fshr.i32(i32 %a0, i32 undef, i32 %a1)
   ret i32 %res
 }
 
 define i32 @fshr_i32_undef1_msk(i32 %a0, i32 %a1) nounwind {
 ; X32-SSE2-LABEL: fshr_i32_undef1_msk:
 ; X32-SSE2:       # %bb.0:
 ; X32-SSE2-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE2-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-SSE2-NEXT:    andl $7, %ecx
-; X32-SSE2-NEXT:    # kill: def $cl killed $cl killed $ecx
+; X32-SSE2-NEXT:    movb {{[0-9]+}}(%esp), %cl
+; X32-SSE2-NEXT:    andb $7, %cl
 ; X32-SSE2-NEXT:    shrdl %cl, %eax, %eax
 ; X32-SSE2-NEXT:    retl
 ;
 ; X64-AVX2-LABEL: fshr_i32_undef1_msk:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    movl %esi, %ecx
-; X64-AVX2-NEXT:    andl $7, %ecx
+; X64-AVX2-NEXT:    andb $7, %cl
 ; X64-AVX2-NEXT:    # kill: def $cl killed $cl killed $ecx
 ; X64-AVX2-NEXT:    shrdl %cl, %edi, %eax
 ; X64-AVX2-NEXT:    retq
   %m = and i32 %a1, 7
   %res = call i32 @llvm.fshr.i32(i32 %a0, i32 undef, i32 %m)
   ret i32 %res
 }
 

llvm/test/CodeGen/X86/load-local-v4i5.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
 @0 = internal unnamed_addr constant [4 x i5] [i5 2, i5 0, i5 2, i5 -1], align 1
 
 ; Function Attrs: nobuiltin nounwind
 define void @_start() {
 ; CHECK-LABEL: _start:
 ; CHECK:       # %bb.0: # %Entry
 ; CHECK-NEXT:    movl {{.*}}(%rip), %eax
 ; CHECK-NEXT:    movl %eax, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movb -{{[0-9]+}}(%rsp), %cl
 ; CHECK-NEXT:    movzbl -{{[0-9]+}}(%rsp), %edx
 ; CHECK-NEXT:    movzbl -{{[0-9]+}}(%rsp), %esi
+; CHECK-NEXT:    movzbl %cl, %edi
+; CHECK-NEXT:    shrb %cl
+; CHECK-NEXT:    movb %cl, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    andl $31, %eax
 ; CHECK-NEXT:    andl $31, %esi
 ; CHECK-NEXT:    shll $5, %esi
 ; CHECK-NEXT:    orl %eax, %esi
 ; CHECK-NEXT:    andl $31, %edx
 ; CHECK-NEXT:    shll $10, %edx
 ; CHECK-NEXT:    orl %esi, %edx
-; CHECK-NEXT:    movzbl %cl, %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    shll $15, %ecx
-; CHECK-NEXT:    orl %edx, %ecx
-; CHECK-NEXT:    movw %cx, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrl $16, %ecx
-; CHECK-NEXT:    andl $15, %ecx
-; CHECK-NEXT:    movb %cl, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movb %al, -{{[0-9]+}}(%rsp)
-; CHECK-NEXT:    cmpb $31, %al
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shll $15, %eax
+; CHECK-NEXT:    orl %edx, %eax
+; CHECK-NEXT:    movw %ax, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movb %dil, -{{[0-9]+}}(%rsp)
+; CHECK-NEXT:    cmpb $31, %dil
 ; CHECK-NEXT:    je .LBB0_2
 ; CHECK-NEXT:  # %bb.1: # %Then
 ; CHECK-NEXT:    int3
 ; CHECK-NEXT:  .LBB0_2: # %EndIf
 ; CHECK-NEXT:    retq
 Entry:
   %x = alloca [4 x i5], align 1
   %y = alloca <4 x i5>, align 4

llvm/test/CodeGen/X86/lzcnt.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- -mattr=+lzcnt | FileCheck %s --check-prefix=X86
 ; RUN: llc < %s -mtriple=x86_64-linux-gnux32  -mattr=+lzcnt | FileCheck %s --check-prefix=X32
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+lzcnt | FileCheck %s --check-prefix=X64
 
 declare i8 @llvm.ctlz.i8(i8, i1) nounwind readnone
 declare i16 @llvm.ctlz.i16(i16, i1) nounwind readnone
 declare i32 @llvm.ctlz.i32(i32, i1) nounwind readnone
 declare i64 @llvm.ctlz.i64(i64, i1) nounwind readnone
 
 define i8 @t1(i8 %x) nounwind  {
 ; X86-LABEL: t1:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    lzcntl %eax, %eax
-; X86-NEXT:    addl $-24, %eax
+; X86-NEXT:    addb $-24, %al
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
 ;
 ; X32-LABEL: t1:
 ; X32:       # %bb.0:
 ; X32-NEXT:    movzbl %dil, %eax
 ; X32-NEXT:    lzcntl %eax, %eax
-; X32-NEXT:    addl $-24, %eax
+; X32-NEXT:    addb $-24, %al
 ; X32-NEXT:    # kill: def $al killed $al killed $eax
 ; X32-NEXT:    retq
 ;
 ; X64-LABEL: t1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    lzcntl %eax, %eax
-; X64-NEXT:    addl $-24, %eax
+; X64-NEXT:    addb $-24, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %tmp = tail call i8 @llvm.ctlz.i8( i8 %x, i1 false )
   ret i8 %tmp
 }
 
 define i16 @t2(i16 %x) nounwind  {
 ; X86-LABEL: t2:
; X64-NEXT:    retq
   ret i64 %tmp
 }
 
 define i8 @t5(i8 %x) nounwind  {
 ; X86-LABEL: t5:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    lzcntl %eax, %eax
-; X86-NEXT:    addl $-24, %eax
+; X86-NEXT:    addb $-24, %al
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    retl
 ;
 ; X32-LABEL: t5:
 ; X32:       # %bb.0:
 ; X32-NEXT:    movzbl %dil, %eax
 ; X32-NEXT:    lzcntl %eax, %eax
-; X32-NEXT:    addl $-24, %eax
+; X32-NEXT:    addb $-24, %al
 ; X32-NEXT:    # kill: def $al killed $al killed $eax
 ; X32-NEXT:    retq
 ;
 ; X64-LABEL: t5:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl %dil, %eax
 ; X64-NEXT:    lzcntl %eax, %eax
-; X64-NEXT:    addl $-24, %eax
+; X64-NEXT:    addb $-24, %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %tmp = tail call i8 @llvm.ctlz.i8( i8 %x, i1 true )
   ret i8 %tmp
 }
 
 define i16 @t6(i16 %x) nounwind  {
 ; X86-LABEL: t6:

llvm/test/CodeGen/X86/masked_store_trunc.ll

 ; AVX1-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm1
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB1_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB1_3
 ; AVX1-NEXT:  .LBB1_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB1_5

 ; AVX2-NEXT:    vpblendw {{.*#+}} ymm1 = ymm1[0],ymm3[1,2,3],ymm1[4],ymm3[5,6,7],ymm1[8],ymm3[9,10,11],ymm1[12],ymm3[13,14,15]
 ; AVX2-NEXT:    vpblendw {{.*#+}} ymm0 = ymm0[0],ymm3[1,2,3],ymm0[4],ymm3[5,6,7],ymm0[8],ymm3[9,10,11],ymm0[12],ymm3[13,14,15]
 ; AVX2-NEXT:    vpackusdw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
 ; AVX2-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,1,3]
 ; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB1_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB1_3
 ; AVX2-NEXT:  .LBB1_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB1_5

 ; AVX1-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm1
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB2_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB2_3
 ; AVX1-NEXT:  .LBB2_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB2_5

 ; AVX2-NEXT:    vpand %ymm4, %ymm0, %ymm0
 ; AVX2-NEXT:    vpackusdw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
 ; AVX2-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,1,3]
 ; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB2_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB2_3
 ; AVX2-NEXT:  .LBB2_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB2_5

 
 define void @truncstore_v4i64_v4i32(<4 x i64> %x, <4 x i32>* %p, <4 x i32> %mask) {
 ; SSE2-LABEL: truncstore_v4i64_v4i32:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm3, %xmm3
 ; SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,2],xmm1[0,2]
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE2-NEXT:    movmskps %xmm3, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB3_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB3_3
 ; SSE2-NEXT:  .LBB3_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB3_5

 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v4i64_v4i32:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm3, %xmm3
 ; SSE4-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,2],xmm1[0,2]
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE4-NEXT:    movmskps %xmm3, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB3_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB3_3
 ; SSE4-NEXT:  .LBB3_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB3_5

 ; SSE2-NEXT:    pxor %xmm3, %xmm3
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE2-NEXT:    movmskps %xmm3, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB4_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB4_3
 ; SSE2-NEXT:  .LBB4_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB4_5

 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm3, %xmm3
 ; SSE4-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0],xmm3[1,2,3],xmm1[4],xmm3[5,6,7]
 ; SSE4-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0],xmm3[1,2,3],xmm0[4],xmm3[5,6,7]
 ; SSE4-NEXT:    packusdw %xmm1, %xmm0
 ; SSE4-NEXT:    packusdw %xmm0, %xmm0
 ; SSE4-NEXT:    pcmpeqd %xmm3, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB4_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB4_3
 ; SSE4-NEXT:  .LBB4_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB4_5

 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm3 = xmm3[0],xmm2[1,2,3],xmm3[4],xmm2[5,6,7]
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0],xmm2[1,2,3],xmm0[4],xmm2[5,6,7]
 ; AVX1-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB4_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB4_3
 ; AVX1-NEXT:  .LBB4_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB4_5

 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpblendw {{.*#+}} xmm3 = xmm3[0],xmm2[1,2,3],xmm3[4],xmm2[5,6,7]
 ; AVX2-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0],xmm2[1,2,3],xmm0[4],xmm2[5,6,7]
 ; AVX2-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB4_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB4_3
 ; AVX2-NEXT:  .LBB4_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB4_5

 ; SSE2-NEXT:    movdqa {{.*#+}} xmm4 = [255,0,0,0,255,0,0,0]
 ; SSE2-NEXT:    pand %xmm4, %xmm1
 ; SSE2-NEXT:    pand %xmm4, %xmm0
 ; SSE2-NEXT:    packuswb %xmm1, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE2-NEXT:    movmskps %xmm3, %ecx
-; SSE2-NEXT:    xorl $15, %ecx
+; SSE2-NEXT:    xorb $15, %cl
 ; SSE2-NEXT:    testb $1, %cl
 ; SSE2-NEXT:    movd %xmm0, %eax
 ; SSE2-NEXT:    jne .LBB5_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %cl
 ; SSE2-NEXT:    jne .LBB5_3
 ; SSE2-NEXT:  .LBB5_4: # %else2
 ; SSE2-NEXT:    testb $4, %cl

 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm3, %xmm3
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm4 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; SSE4-NEXT:    pshufb %xmm4, %xmm1
 ; SSE4-NEXT:    pshufb %xmm4, %xmm0
 ; SSE4-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE4-NEXT:    movmskps %xmm3, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB5_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB5_3
 ; SSE4-NEXT:  .LBB5_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB5_5

 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm4 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX1-NEXT:    vpshufb %xmm4, %xmm3, %xmm3
 ; AVX1-NEXT:    vpshufb %xmm4, %xmm0, %xmm0
 ; AVX1-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB5_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB5_3
 ; AVX1-NEXT:  .LBB5_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB5_5

 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm4 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm3, %xmm3
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm0, %xmm0
 ; AVX2-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB5_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB5_3
 ; AVX2-NEXT:  .LBB5_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB5_5

 ; SSE2-LABEL: truncstore_v2i64_v2i32:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm2, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB6_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB6_3
 ; SSE2-NEXT:  .LBB6_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB6_1: # %cond.store
 ; SSE2-NEXT:    movd %xmm0, (%rdi)
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    je .LBB6_4
 ; SSE2-NEXT:  .LBB6_3: # %cond.store1
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,1,1]
 ; SSE2-NEXT:    movd %xmm0, 4(%rdi)
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v2i64_v2i32:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm2
 ; SSE4-NEXT:    movmskpd %xmm2, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB6_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB6_3
 ; SSE4-NEXT:  .LBB6_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB6_1: # %cond.store

 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm2, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB7_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB7_3
 ; SSE2-NEXT:  .LBB7_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB7_1: # %cond.store
 ; SSE2-NEXT:    movd %xmm0, %ecx
 ; SSE2-NEXT:    movw %cx, (%rdi)
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    je .LBB7_4
 ; SSE2-NEXT:  .LBB7_3: # %cond.store1
 ; SSE2-NEXT:    pextrw $1, %xmm0, %eax
 ; SSE2-NEXT:    movw %ax, 2(%rdi)
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v2i64_v2i16:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE4-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm2
 ; SSE4-NEXT:    movmskpd %xmm2, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB7_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB7_3
 ; SSE4-NEXT:  .LBB7_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB7_1: # %cond.store
 ; SSE4-NEXT:    pextrw $0, %xmm0, (%rdi)
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    je .LBB7_4
 ; SSE4-NEXT:  .LBB7_3: # %cond.store1
 ; SSE4-NEXT:    pextrw $1, %xmm0, 2(%rdi)
 ; SSE4-NEXT:    retq
 ;
 ; AVX-LABEL: truncstore_v2i64_v2i16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; AVX-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; AVX-NEXT:    vpcmpeqq %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskpd %xmm1, %eax
-; AVX-NEXT:    xorl $3, %eax
+; AVX-NEXT:    xorb $3, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB7_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB7_3
 ; AVX-NEXT:  .LBB7_4: # %else2
 ; AVX-NEXT:    retq
 ; AVX-NEXT:  .LBB7_1: # %cond.store

 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    movd %xmm0, %ecx
 ; SSE2-NEXT:    jne .LBB8_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB8_3
 ; SSE2-NEXT:  .LBB8_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB8_1: # %cond.store
 ; SSE2-NEXT:    movb %cl, (%rdi)
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    je .LBB8_4
 ; SSE2-NEXT:  .LBB8_3: # %cond.store1
 ; SSE2-NEXT:    movb %ch, 1(%rdi)
 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v2i64_v2i8:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pshufb {{.*#+}} xmm0 = xmm0[0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm2
 ; SSE4-NEXT:    movmskpd %xmm2, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB8_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB8_3
 ; SSE4-NEXT:  .LBB8_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB8_1: # %cond.store
 ; SSE4-NEXT:    pextrb $0, %xmm0, (%rdi)
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    je .LBB8_4
 ; SSE4-NEXT:  .LBB8_3: # %cond.store1
 ; SSE4-NEXT:    pextrb $1, %xmm0, 1(%rdi)
 ; SSE4-NEXT:    retq
 ;
 ; AVX-LABEL: truncstore_v2i64_v2i8:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u]
 ; AVX-NEXT:    vpcmpeqq %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskpd %xmm1, %eax
-; AVX-NEXT:    xorl $3, %eax
+; AVX-NEXT:    xorb $3, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB8_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB8_3
 ; AVX-NEXT:  .LBB8_4: # %else2
 ; AVX-NEXT:    retq
 ; AVX-NEXT:  .LBB8_1: # %cond.store

 ; AVX1-NEXT:    vpshufb %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB11_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB11_3
 ; AVX1-NEXT:  .LBB11_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB11_5

 ;
 ; AVX2-LABEL: truncstore_v8i32_v8i16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vpshufb {{.*#+}} ymm0 = ymm0[0,1,4,5,8,9,12,13,u,u,u,u,u,u,u,u,16,17,20,21,24,25,28,29,u,u,u,u,u,u,u,u]
 ; AVX2-NEXT:    vpermq {{.*#+}} ymm0 = ymm0[0,2,2,3]
 ; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB11_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB11_3
 ; AVX2-NEXT:  .LBB11_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB11_5

 ; AVX1-NEXT:    vpshufb %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB12_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB12_3
 ; AVX1-NEXT:  .LBB12_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB12_5

 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm4 = <0,4,8,12,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm3, %xmm3
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm0, %xmm0
 ; AVX2-NEXT:    vpunpckldq {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1]
 ; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB12_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB12_3
 ; AVX2-NEXT:  .LBB12_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB12_5

 ; SSE2-LABEL: truncstore_v4i32_v4i16:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm2, %xmm2
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,4,6,6,7]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    movmskps %xmm2, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB13_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB13_3
 ; SSE2-NEXT:  .LBB13_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB13_5

 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v4i32_v4i16:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pshufb {{.*#+}} xmm0 = xmm0[0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
 ; SSE4-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB13_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB13_3
 ; SSE4-NEXT:  .LBB13_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB13_5

 ; SSE4-NEXT:    retq
 ;
 ; AVX-LABEL: truncstore_v4i32_v4i16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
 ; AVX-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskps %xmm1, %eax
-; AVX-NEXT:    xorl $15, %eax
+; AVX-NEXT:    xorb $15, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB13_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB13_3
 ; AVX-NEXT:  .LBB13_4: # %else2
 ; AVX-NEXT:    testb $4, %al
 ; AVX-NEXT:    jne .LBB13_5

 ; SSE2-LABEL: truncstore_v4i32_v4i8:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm2, %xmm2
 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    movmskps %xmm2, %ecx
-; SSE2-NEXT:    xorl $15, %ecx
+; SSE2-NEXT:    xorb $15, %cl
 ; SSE2-NEXT:    testb $1, %cl
 ; SSE2-NEXT:    movd %xmm0, %eax
 ; SSE2-NEXT:    jne .LBB14_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %cl
 ; SSE2-NEXT:    jne .LBB14_3
 ; SSE2-NEXT:  .LBB14_4: # %else2
 ; SSE2-NEXT:    testb $4, %cl

 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v4i32_v4i8:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pshufb {{.*#+}} xmm0 = xmm0[0,4,8,12,u,u,u,u,u,u,u,u,u,u,u,u]
 ; SSE4-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB14_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB14_3
 ; SSE4-NEXT:  .LBB14_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB14_5

 ; SSE4-NEXT:    retq
 ;
 ; AVX-LABEL: truncstore_v4i32_v4i8:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[0,4,8,12,u,u,u,u,u,u,u,u,u,u,u,u]
 ; AVX-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskps %xmm1, %eax
-; AVX-NEXT:    xorl $15, %eax
+; AVX-NEXT:    xorb $15, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB14_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB14_3
 ; AVX-NEXT:  .LBB14_4: # %else2
 ; AVX-NEXT:    testb $4, %al
 ; AVX-NEXT:    jne .LBB14_5

llvm/test/CodeGen/X86/masked_store_trunc_ssat.ll

 ; AVX1-NEXT:    vpackssdw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackssdw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm1
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB1_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB1_3
 ; AVX1-NEXT:  .LBB1_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB1_5

 ; AVX2-NEXT:    vpcmpgtq %ymm4, %ymm0, %ymm5
 ; AVX2-NEXT:    vblendvpd %ymm5, %ymm0, %ymm4, %ymm0
 ; AVX2-NEXT:    vpackssdw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
 ; AVX2-NEXT:    vpackssdw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,1,3]
 ; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB1_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB1_3
 ; AVX2-NEXT:  .LBB1_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB1_5

 ; AVX1-NEXT:    vpackssdw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm1
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB2_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB2_3
 ; AVX1-NEXT:  .LBB2_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB2_5

 ; AVX2-NEXT:    vblendvpd %ymm5, %ymm0, %ymm4, %ymm0
 ; AVX2-NEXT:    vpackssdw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
 ; AVX2-NEXT:    vpackssdw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,1,3]
 ; AVX2-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB2_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB2_3
 ; AVX2-NEXT:  .LBB2_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB2_5

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; SSE2-NEXT:    por %xmm4, %xmm0
 ; SSE2-NEXT:    pand %xmm0, %xmm5
 ; SSE2-NEXT:    pandn %xmm8, %xmm0
 ; SSE2-NEXT:    por %xmm5, %xmm0
 ; SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,2],xmm1[0,2]
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm9
 ; SSE2-NEXT:    movmskps %xmm9, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB3_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB3_3
 ; SSE2-NEXT:  .LBB3_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB3_5

 ; SSE4-NEXT:    movdqa %xmm1, %xmm3
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm5, %xmm3
 ; SSE4-NEXT:    movapd %xmm6, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm6, %xmm1
 ; SSE4-NEXT:    shufps {{.*#+}} xmm1 = xmm1[0,2],xmm3[0,2]
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm4
 ; SSE4-NEXT:    movmskps %xmm4, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB3_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB3_3
 ; SSE4-NEXT:  .LBB3_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB3_5

 ; SSE2-NEXT:    por %xmm4, %xmm0
 ; SSE2-NEXT:    pand %xmm0, %xmm5
 ; SSE2-NEXT:    pandn %xmm8, %xmm0
 ; SSE2-NEXT:    por %xmm5, %xmm0
 ; SSE2-NEXT:    packssdw %xmm1, %xmm0
 ; SSE2-NEXT:    packssdw %xmm0, %xmm0
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm9
 ; SSE2-NEXT:    movmskps %xmm9, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB4_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB4_3
 ; SSE2-NEXT:  .LBB4_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB4_5

 ; SSE4-NEXT:    blendvpd %xmm0, %xmm5, %xmm3
 ; SSE4-NEXT:    movapd %xmm6, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm1, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm6, %xmm1
 ; SSE4-NEXT:    packssdw %xmm3, %xmm1
 ; SSE4-NEXT:    packssdw %xmm1, %xmm1
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm4
 ; SSE4-NEXT:    movmskps %xmm4, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB4_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB4_3
 ; SSE4-NEXT:  .LBB4_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB4_5

 ; AVX1-NEXT:    vblendvpd %xmm5, %xmm3, %xmm4, %xmm3
 ; AVX1-NEXT:    vpcmpgtq %xmm6, %xmm3, %xmm4
 ; AVX1-NEXT:    vblendvpd %xmm4, %xmm3, %xmm6, %xmm3
 ; AVX1-NEXT:    vblendvpd %xmm7, %xmm0, %xmm6, %xmm0
 ; AVX1-NEXT:    vpackssdw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackssdw %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB4_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB4_3
 ; AVX1-NEXT:  .LBB4_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB4_5

 ; AVX2-NEXT:    vpbroadcastq {{.*#+}} ymm3 = [18446744073709518848,18446744073709518848,18446744073709518848,18446744073709518848]
 ; AVX2-NEXT:    vpcmpgtq %ymm3, %ymm0, %ymm4
 ; AVX2-NEXT:    vblendvpd %ymm4, %ymm0, %ymm3, %ymm0
 ; AVX2-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpackssdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackssdw %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB4_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB4_3
 ; AVX2-NEXT:  .LBB4_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB4_5

 ; SSE2-NEXT:    movdqa {{.*#+}} xmm3 = [255,0,0,0,255,0,0,0]
 ; SSE2-NEXT:    pand %xmm3, %xmm1
 ; SSE2-NEXT:    pand %xmm3, %xmm0
 ; SSE2-NEXT:    packuswb %xmm1, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    packuswb %xmm0, %xmm0
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm9
 ; SSE2-NEXT:    movmskps %xmm9, %ecx
-; SSE2-NEXT:    xorl $15, %ecx
+; SSE2-NEXT:    xorb $15, %cl
 ; SSE2-NEXT:    testb $1, %cl
 ; SSE2-NEXT:    movd %xmm0, %eax
 ; SSE2-NEXT:    jne .LBB5_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %cl
 ; SSE2-NEXT:    jne .LBB5_3
 ; SSE2-NEXT:  .LBB5_4: # %else2
 ; SSE2-NEXT:    testb $4, %cl

 ; SSE4-NEXT:    pcmpgtq %xmm3, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm6, %xmm3
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm0 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; SSE4-NEXT:    pshufb %xmm0, %xmm3
 ; SSE4-NEXT:    pshufb %xmm0, %xmm1
 ; SSE4-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm3[0],xmm1[1],xmm3[1],xmm1[2],xmm3[2],xmm1[3],xmm3[3]
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm4
 ; SSE4-NEXT:    movmskps %xmm4, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB5_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB5_3
 ; SSE4-NEXT:  .LBB5_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB5_5

 ; AVX1-NEXT:    vblendvpd %xmm4, %xmm3, %xmm6, %xmm3
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm4 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX1-NEXT:    vpshufb %xmm4, %xmm3, %xmm3
 ; AVX1-NEXT:    vblendvpd %xmm7, %xmm0, %xmm6, %xmm0
 ; AVX1-NEXT:    vpshufb %xmm4, %xmm0, %xmm0
 ; AVX1-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB5_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB5_3
 ; AVX1-NEXT:  .LBB5_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB5_5

 ; AVX2-NEXT:    vblendvpd %ymm4, %ymm0, %ymm3, %ymm0
 ; AVX2-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm4 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm3, %xmm3
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm0, %xmm0
 ; AVX2-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB5_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB5_3
 ; AVX2-NEXT:  .LBB5_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB5_5

 ; SSE2-NEXT:    pand %xmm3, %xmm5
 ; SSE2-NEXT:    pandn {{.*}}(%rip), %xmm3
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm3[0,2,2,3]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB6_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB6_3
 ; SSE2-NEXT:  .LBB6_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB6_1: # %cond.store

 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm4
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [18446744071562067968,18446744071562067968]
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm2, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm4, %xmm2
 ; SSE4-NEXT:    pshufd {{.*#+}} xmm0 = xmm2[0,2,2,3]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm3
 ; SSE4-NEXT:    movmskpd %xmm3, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB6_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB6_3
 ; SSE4-NEXT:  .LBB6_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB6_1: # %cond.store

 ; SSE2-NEXT:    pandn {{.*}}(%rip), %xmm3
 ; SSE2-NEXT:    por %xmm5, %xmm3
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm3[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB7_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB7_3
 ; SSE2-NEXT:  .LBB7_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB7_1: # %cond.store

 ; SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [18446744073709518848,18446744073709518848]
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm2, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm4, %xmm2
 ; SSE4-NEXT:    pshufd {{.*#+}} xmm0 = xmm2[0,2,2,3]
 ; SSE4-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm3
 ; SSE4-NEXT:    movmskpd %xmm3, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB7_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB7_3
 ; SSE4-NEXT:  .LBB7_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB7_1: # %cond.store

 ; AVX-NEXT:    vblendvpd %xmm4, %xmm0, %xmm3, %xmm0
 ; AVX-NEXT:    vmovdqa {{.*#+}} xmm3 = [18446744073709518848,18446744073709518848]
 ; AVX-NEXT:    vpcmpgtq %xmm3, %xmm0, %xmm4
 ; AVX-NEXT:    vblendvpd %xmm4, %xmm0, %xmm3, %xmm0
 ; AVX-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; AVX-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; AVX-NEXT:    vpcmpeqq %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskpd %xmm1, %eax
-; AVX-NEXT:    xorl $3, %eax
+; AVX-NEXT:    xorb $3, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB7_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB7_3
 ; AVX-NEXT:  .LBB7_4: # %else2
 ; AVX-NEXT:    retq
 ; AVX-NEXT:  .LBB7_1: # %cond.store

 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm3
 ; SSE2-NEXT:    packuswb %xmm3, %xmm3
 ; SSE2-NEXT:    packuswb %xmm3, %xmm3
 ; SSE2-NEXT:    packuswb %xmm3, %xmm3
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm0
 ; SSE2-NEXT:    movmskpd %xmm0, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    movd %xmm3, %ecx
 ; SSE2-NEXT:    jne .LBB8_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB8_3
 ; SSE2-NEXT:  .LBB8_4: # %else2
 ; SSE2-NEXT:    retq

 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm4
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [18446744073709551488,18446744073709551488]
 ; SSE4-NEXT:    movapd %xmm4, %xmm0
 ; SSE4-NEXT:    pcmpgtq %xmm2, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm4, %xmm2
 ; SSE4-NEXT:    pshufb {{.*#+}} xmm2 = xmm2[0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm3
 ; SSE4-NEXT:    movmskpd %xmm3, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB8_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB8_3
 ; SSE4-NEXT:  .LBB8_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB8_1: # %cond.store

 ; AVX-NEXT:    vpcmpgtq %xmm0, %xmm3, %xmm4
 ; AVX-NEXT:    vblendvpd %xmm4, %xmm0, %xmm3, %xmm0
 ; AVX-NEXT:    vmovdqa {{.*#+}} xmm3 = [18446744073709551488,18446744073709551488]
 ; AVX-NEXT:    vpcmpgtq %xmm3, %xmm0, %xmm4
 ; AVX-NEXT:    vblendvpd %xmm4, %xmm0, %xmm3, %xmm0
 ; AVX-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u]
 ; AVX-NEXT:    vpcmpeqq %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskpd %xmm1, %eax
-; AVX-NEXT:    xorl $3, %eax
+; AVX-NEXT:    xorb $3, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB8_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB8_3
 ; AVX-NEXT:  .LBB8_4: # %else2
 ; AVX-NEXT:    retq
 ; AVX-NEXT:  .LBB8_1: # %cond.store

 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm2
 ; AVX1-NEXT:    vpackssdw %xmm2, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB11_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB11_3
 ; AVX1-NEXT:  .LBB11_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB11_5

 ;
 ; AVX2-LABEL: truncstore_v8i32_v8i16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpackssdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB11_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB11_3
 ; AVX2-NEXT:  .LBB11_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB11_5

 ; AVX1-NEXT:    vpackssdw %xmm2, %xmm0, %xmm0
 ; AVX1-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB12_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB12_3
 ; AVX1-NEXT:  .LBB12_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB12_5

 ; AVX2-LABEL: truncstore_v8i32_v8i8:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpackssdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB12_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB12_3
 ; AVX2-NEXT:  .LBB12_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB12_5

 
 define void @truncstore_v4i32_v4i16(<4 x i32> %x, <4 x i16>* %p, <4 x i32> %mask) {
 ; SSE2-LABEL: truncstore_v4i32_v4i16:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pxor %xmm2, %xmm2
 ; SSE2-NEXT:    packssdw %xmm0, %xmm0
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    movmskps %xmm2, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB13_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB13_3
 ; SSE2-NEXT:  .LBB13_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB13_5

 ; SSE2-NEXT:    retq
 ;
 ; SSE4-LABEL: truncstore_v4i32_v4i16:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    packssdw %xmm0, %xmm0
 ; SSE4-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB13_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB13_3
 ; SSE4-NEXT:  .LBB13_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB13_5

 ; SSE4-NEXT:    retq
 ;
 ; AVX-LABEL: truncstore_v4i32_v4i16:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX-NEXT:    vpackssdw %xmm0, %xmm0, %xmm0
 ; AVX-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskps %xmm1, %eax
-; AVX-NEXT:    xorl $15, %eax
+; AVX-NEXT:    xorb $15, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB13_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB13_3
 ; AVX-NEXT:  .LBB13_4: # %else2
 ; AVX-NEXT:    testb $4, %al
 ; AVX-NEXT:    jne .LBB13_5

 ; SSE2-NEXT:    pand %xmm3, %xmm4
 ; SSE2-NEXT:    pandn %xmm0, %xmm3
 ; SSE2-NEXT:    por %xmm4, %xmm3
 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm3
 ; SSE2-NEXT:    packuswb %xmm3, %xmm3
 ; SSE2-NEXT:    packuswb %xmm3, %xmm3
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    movmskps %xmm2, %ecx
-; SSE2-NEXT:    xorl $15, %ecx
+; SSE2-NEXT:    xorb $15, %cl
 ; SSE2-NEXT:    testb $1, %cl
 ; SSE2-NEXT:    movd %xmm3, %eax
 ; SSE2-NEXT:    jne .LBB14_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %cl
 ; SSE2-NEXT:    jne .LBB14_3
 ; SSE2-NEXT:  .LBB14_4: # %else2
 ; SSE2-NEXT:    testb $4, %cl

 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pminsd {{.*}}(%rip), %xmm0
 ; SSE4-NEXT:    pmaxsd {{.*}}(%rip), %xmm0
 ; SSE4-NEXT:    packssdw %xmm0, %xmm0
 ; SSE4-NEXT:    packsswb %xmm0, %xmm0
 ; SSE4-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB14_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB14_3
 ; SSE4-NEXT:  .LBB14_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB14_5

 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vpminsd {{.*}}(%rip), %xmm0, %xmm0
 ; AVX1-NEXT:    vpmaxsd {{.*}}(%rip), %xmm0, %xmm0
 ; AVX1-NEXT:    vpackssdw %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB14_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB14_3
 ; AVX1-NEXT:  .LBB14_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB14_5

 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm3 = [127,127,127,127]
 ; AVX2-NEXT:    vpminsd %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm3 = [4294967168,4294967168,4294967168,4294967168]
 ; AVX2-NEXT:    vpmaxsd %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackssdw %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpacksswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB14_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB14_3
 ; AVX2-NEXT:  .LBB14_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB14_5

llvm/test/CodeGen/X86/masked_store_trunc_usat.ll

 ; AVX1-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm1
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB1_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB1_3
 ; AVX1-NEXT:  .LBB1_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB1_5

 ; AVX2-NEXT:    vpcmpgtq %ymm5, %ymm7, %ymm5
 ; AVX2-NEXT:    vblendvpd %ymm5, %ymm0, %ymm4, %ymm0
 ; AVX2-NEXT:    vpackusdw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
 ; AVX2-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,1,3]
 ; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB1_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB1_3
 ; AVX2-NEXT:  .LBB1_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB1_5

 ; AVX1-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm1
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm2, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB2_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB2_3
 ; AVX1-NEXT:  .LBB2_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB2_5

 ; AVX2-NEXT:    vblendvpd %ymm5, %ymm0, %ymm4, %ymm0
 ; AVX2-NEXT:    vpackusdw %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
 ; AVX2-NEXT:    vpackusdw %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,1,3]
 ; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm3, %ymm2, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB2_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB2_3
 ; AVX2-NEXT:  .LBB2_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB2_5

 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
 ; SSE2-NEXT:    por %xmm5, %xmm1
 ; SSE2-NEXT:    pand %xmm1, %xmm0
 ; SSE2-NEXT:    pandn %xmm8, %xmm1
 ; SSE2-NEXT:    por %xmm0, %xmm1
 ; SSE2-NEXT:    shufps {{.*#+}} xmm1 = xmm1[0,2],xmm4[0,2]
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE2-NEXT:    movmskps %xmm3, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB3_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB3_3
 ; SSE2-NEXT:  .LBB3_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB3_5

 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm3
 ; SSE4-NEXT:    pxor %xmm8, %xmm7
 ; SSE4-NEXT:    pcmpgtq %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm5
 ; SSE4-NEXT:    shufps {{.*#+}} xmm5 = xmm5[0,2],xmm3[0,2]
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm6
 ; SSE4-NEXT:    movmskps %xmm6, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB3_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB3_3
 ; SSE4-NEXT:  .LBB3_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB3_5

 ; SSE2-NEXT:    por %xmm1, %xmm0
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm1 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm4[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm3
 ; SSE2-NEXT:    movmskps %xmm3, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB4_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB4_3
 ; SSE2-NEXT:  .LBB4_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB4_5

 ; SSE4-NEXT:    pxor %xmm8, %xmm7
 ; SSE4-NEXT:    pcmpgtq %xmm7, %xmm4
 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm8, %xmm5
 ; SSE4-NEXT:    packusdw %xmm3, %xmm5
 ; SSE4-NEXT:    packusdw %xmm5, %xmm5
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm6
 ; SSE4-NEXT:    movmskps %xmm6, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB4_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB4_3
 ; SSE4-NEXT:  .LBB4_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB4_5

 ; AVX1-NEXT:    vpcmpgtq %xmm3, %xmm5, %xmm3
 ; AVX1-NEXT:    vmovapd {{.*#+}} xmm5 = [65535,65535]
 ; AVX1-NEXT:    vblendvpd %xmm3, %xmm6, %xmm5, %xmm3
 ; AVX1-NEXT:    vblendvpd %xmm4, %xmm0, %xmm5, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB4_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB4_3
 ; AVX1-NEXT:  .LBB4_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB4_5

 ; AVX2-NEXT:    vpbroadcastq {{.*#+}} ymm5 = [9223372036854841343,9223372036854841343,9223372036854841343,9223372036854841343]
 ; AVX2-NEXT:    vpcmpgtq %ymm4, %ymm5, %ymm4
 ; AVX2-NEXT:    vblendvpd %ymm4, %ymm0, %ymm3, %ymm0
 ; AVX2-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB4_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB4_3
 ; AVX2-NEXT:  .LBB4_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB4_5

 ; SSE2-NEXT:    por %xmm1, %xmm0
 ; SSE2-NEXT:    pand %xmm8, %xmm0
 ; SSE2-NEXT:    pand %xmm8, %xmm4
 ; SSE2-NEXT:    packuswb %xmm0, %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    pcmpeqd %xmm2, %xmm9
 ; SSE2-NEXT:    movmskps %xmm9, %ecx
-; SSE2-NEXT:    xorl $15, %ecx
+; SSE2-NEXT:    xorb $15, %cl
 ; SSE2-NEXT:    testb $1, %cl
 ; SSE2-NEXT:    movd %xmm4, %eax
 ; SSE2-NEXT:    jne .LBB5_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %cl
 ; SSE2-NEXT:    jne .LBB5_3
 ; SSE2-NEXT:  .LBB5_4: # %else2
 ; SSE2-NEXT:    testb $4, %cl

 ; SSE4-NEXT:    movdqa %xmm4, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm1, %xmm7
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm0 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; SSE4-NEXT:    pshufb %xmm0, %xmm7
 ; SSE4-NEXT:    pshufb %xmm0, %xmm5
 ; SSE4-NEXT:    punpcklwd {{.*#+}} xmm5 = xmm5[0],xmm7[0],xmm5[1],xmm7[1],xmm5[2],xmm7[2],xmm5[3],xmm7[3]
 ; SSE4-NEXT:    pcmpeqd %xmm2, %xmm8
 ; SSE4-NEXT:    movmskps %xmm8, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB5_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB5_3
 ; SSE4-NEXT:  .LBB5_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB5_5

 ; AVX1-NEXT:    vblendvpd %xmm3, %xmm6, %xmm5, %xmm3
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm6 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX1-NEXT:    vpshufb %xmm6, %xmm3, %xmm3
 ; AVX1-NEXT:    vblendvpd %xmm4, %xmm0, %xmm5, %xmm0
 ; AVX1-NEXT:    vpshufb %xmm6, %xmm0, %xmm0
 ; AVX1-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB5_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB5_3
 ; AVX1-NEXT:  .LBB5_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB5_5

 ; AVX2-NEXT:    vblendvpd %ymm4, %ymm0, %ymm3, %ymm0
 ; AVX2-NEXT:    vextractf128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm4 = <0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u>
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm3, %xmm3
 ; AVX2-NEXT:    vpshufb %xmm4, %xmm0, %xmm0
 ; AVX2-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB5_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB5_3
 ; AVX2-NEXT:  .LBB5_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB5_5

 ; SSE2-NEXT:    pand %xmm4, %xmm0
 ; SSE2-NEXT:    pandn {{.*}}(%rip), %xmm4
 ; SSE2-NEXT:    por %xmm0, %xmm4
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm4[0,2,2,3]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB6_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB6_3
 ; SSE2-NEXT:  .LBB6_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB6_1: # %cond.store

 ; SSE4-NEXT:    movdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; SSE4-NEXT:    pxor %xmm0, %xmm5
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm0 = [9223372041149743103,9223372041149743103]
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm4
 ; SSE4-NEXT:    pshufd {{.*#+}} xmm0 = xmm4[0,2,2,3]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm3
 ; SSE4-NEXT:    movmskpd %xmm3, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB6_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB6_3
 ; SSE4-NEXT:  .LBB6_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB6_1: # %cond.store

 ; SSE2-NEXT:    pandn {{.*}}(%rip), %xmm4
 ; SSE2-NEXT:    por %xmm0, %xmm4
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm4[0,2,2,3]
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB7_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB7_3
 ; SSE2-NEXT:  .LBB7_4: # %else2
 ; SSE2-NEXT:    retq
 ; SSE2-NEXT:  .LBB7_1: # %cond.store

 ; SSE4-NEXT:    pxor %xmm0, %xmm5
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm0 = [9223372036854841343,9223372036854841343]
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm4
 ; SSE4-NEXT:    pshufd {{.*#+}} xmm0 = xmm4[0,2,2,3]
 ; SSE4-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm3
 ; SSE4-NEXT:    movmskpd %xmm3, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB7_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB7_3
 ; SSE4-NEXT:  .LBB7_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB7_1: # %cond.store

 ; AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm4
 ; AVX-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854841343,9223372036854841343]
 ; AVX-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX-NEXT:    vblendvpd %xmm4, %xmm0, %xmm3, %xmm0
 ; AVX-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; AVX-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
 ; AVX-NEXT:    vpcmpeqq %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskpd %xmm1, %eax
-; AVX-NEXT:    xorl $3, %eax
+; AVX-NEXT:    xorb $3, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB7_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB7_3
 ; AVX-NEXT:  .LBB7_4: # %else2
 ; AVX-NEXT:    retq
 ; AVX-NEXT:  .LBB7_1: # %cond.store

 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm2[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm2, %xmm0
 ; SSE2-NEXT:    movmskpd %xmm0, %eax
-; SSE2-NEXT:    xorl $3, %eax
+; SSE2-NEXT:    xorb $3, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    movd %xmm4, %ecx
 ; SSE2-NEXT:    jne .LBB8_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB8_3
 ; SSE2-NEXT:  .LBB8_4: # %else2
 ; SSE2-NEXT:    retq

 ; SSE4-NEXT:    movdqa {{.*#+}} xmm5 = [9223372036854775808,9223372036854775808]
 ; SSE4-NEXT:    pxor %xmm0, %xmm5
 ; SSE4-NEXT:    movdqa {{.*#+}} xmm0 = [9223372036854776063,9223372036854776063]
 ; SSE4-NEXT:    pcmpgtq %xmm5, %xmm0
 ; SSE4-NEXT:    blendvpd %xmm0, %xmm2, %xmm3
 ; SSE4-NEXT:    pshufb {{.*#+}} xmm3 = xmm3[0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u]
 ; SSE4-NEXT:    pcmpeqq %xmm1, %xmm4
 ; SSE4-NEXT:    movmskpd %xmm4, %eax
-; SSE4-NEXT:    xorl $3, %eax
+; SSE4-NEXT:    xorb $3, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB8_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB8_3
 ; SSE4-NEXT:  .LBB8_4: # %else2
 ; SSE4-NEXT:    retq
 ; SSE4-NEXT:  .LBB8_1: # %cond.store

 ; AVX-NEXT:    vmovapd {{.*#+}} xmm3 = [255,255]
 ; AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm4
 ; AVX-NEXT:    vmovdqa {{.*#+}} xmm5 = [9223372036854776063,9223372036854776063]
 ; AVX-NEXT:    vpcmpgtq %xmm4, %xmm5, %xmm4
 ; AVX-NEXT:    vblendvpd %xmm4, %xmm0, %xmm3, %xmm0
 ; AVX-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[0,8,u,u,u,u,u,u,u,u,u,u,u,u,u,u]
 ; AVX-NEXT:    vpcmpeqq %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vmovmskpd %xmm1, %eax
-; AVX-NEXT:    xorl $3, %eax
+; AVX-NEXT:    xorb $3, %al
 ; AVX-NEXT:    testb $1, %al
 ; AVX-NEXT:    jne .LBB8_1
 ; AVX-NEXT:  # %bb.2: # %else
 ; AVX-NEXT:    testb $2, %al
 ; AVX-NEXT:    jne .LBB8_3
 ; AVX-NEXT:  .LBB8_4: # %else2
 ; AVX-NEXT:    retq
 ; AVX-NEXT:  .LBB8_1: # %cond.store

 ; AVX1-NEXT:    vpminud %xmm3, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm2, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB11_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB11_3
 ; AVX1-NEXT:  .LBB11_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB11_5

 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} ymm3 = [65535,65535,65535,65535,65535,65535,65535,65535]
 ; AVX2-NEXT:    vpminud %ymm3, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB11_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB11_3
 ; AVX2-NEXT:  .LBB11_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB11_5

 ; AVX1-NEXT:    vpackusdw %xmm2, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
 ; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
 ; AVX1-NEXT:    vmovmskps %ymm1, %eax
-; AVX1-NEXT:    notl %eax
+; AVX1-NEXT:    notb %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB12_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB12_3
 ; AVX1-NEXT:  .LBB12_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB12_5

 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} ymm3 = [255,255,255,255,255,255,255,255]
 ; AVX2-NEXT:    vpminud %ymm3, %ymm0, %ymm0
 ; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm3
 ; AVX2-NEXT:    vpackusdw %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm1
 ; AVX2-NEXT:    vmovmskps %ymm1, %eax
-; AVX2-NEXT:    notl %eax
+; AVX2-NEXT:    notb %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB12_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB12_3
 ; AVX2-NEXT:  .LBB12_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB12_5

 ; SSE2-NEXT:    pand %xmm4, %xmm0
 ; SSE2-NEXT:    pandn {{.*}}(%rip), %xmm4
 ; SSE2-NEXT:    por %xmm0, %xmm4
 ; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm4[0,2,2,3,4,5,6,7]
 ; SSE2-NEXT:    pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,4,6,6,7]
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    movmskps %xmm2, %eax
-; SSE2-NEXT:    xorl $15, %eax
+; SSE2-NEXT:    xorb $15, %al
 ; SSE2-NEXT:    testb $1, %al
 ; SSE2-NEXT:    jne .LBB13_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %al
 ; SSE2-NEXT:    jne .LBB13_3
 ; SSE2-NEXT:  .LBB13_4: # %else2
 ; SSE2-NEXT:    testb $4, %al
 ; SSE2-NEXT:    jne .LBB13_5

 ;
 ; SSE4-LABEL: truncstore_v4i32_v4i16:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pminud {{.*}}(%rip), %xmm0
 ; SSE4-NEXT:    packusdw %xmm0, %xmm0
 ; SSE4-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB13_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB13_3
 ; SSE4-NEXT:  .LBB13_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB13_5

 ;
 ; AVX1-LABEL: truncstore_v4i32_v4i16:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vpminud {{.*}}(%rip), %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB13_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB13_3
 ; AVX1-NEXT:  .LBB13_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB13_5

 ; AVX2-LABEL: truncstore_v4i32_v4i16:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm3 = [65535,65535,65535,65535]
 ; AVX2-NEXT:    vpminud %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB13_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB13_3
 ; AVX2-NEXT:  .LBB13_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB13_5

 ; SSE2-NEXT:    pand %xmm4, %xmm0
 ; SSE2-NEXT:    pandn {{.*}}(%rip), %xmm4
 ; SSE2-NEXT:    por %xmm0, %xmm4
 ; SSE2-NEXT:    pand {{.*}}(%rip), %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    packuswb %xmm4, %xmm4
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE2-NEXT:    movmskps %xmm2, %ecx
-; SSE2-NEXT:    xorl $15, %ecx
+; SSE2-NEXT:    xorb $15, %cl
 ; SSE2-NEXT:    testb $1, %cl
 ; SSE2-NEXT:    movd %xmm4, %eax
 ; SSE2-NEXT:    jne .LBB14_1
 ; SSE2-NEXT:  # %bb.2: # %else
 ; SSE2-NEXT:    testb $2, %cl
 ; SSE2-NEXT:    jne .LBB14_3
 ; SSE2-NEXT:  .LBB14_4: # %else2
 ; SSE2-NEXT:    testb $4, %cl

 ; SSE4-LABEL: truncstore_v4i32_v4i8:
 ; SSE4:       # %bb.0:
 ; SSE4-NEXT:    pxor %xmm2, %xmm2
 ; SSE4-NEXT:    pminud {{.*}}(%rip), %xmm0
 ; SSE4-NEXT:    packusdw %xmm0, %xmm0
 ; SSE4-NEXT:    packuswb %xmm0, %xmm0
 ; SSE4-NEXT:    pcmpeqd %xmm1, %xmm2
 ; SSE4-NEXT:    movmskps %xmm2, %eax
-; SSE4-NEXT:    xorl $15, %eax
+; SSE4-NEXT:    xorb $15, %al
 ; SSE4-NEXT:    testb $1, %al
 ; SSE4-NEXT:    jne .LBB14_1
 ; SSE4-NEXT:  # %bb.2: # %else
 ; SSE4-NEXT:    testb $2, %al
 ; SSE4-NEXT:    jne .LBB14_3
 ; SSE4-NEXT:  .LBB14_4: # %else2
 ; SSE4-NEXT:    testb $4, %al
 ; SSE4-NEXT:    jne .LBB14_5

 ; AVX1-LABEL: truncstore_v4i32_v4i8:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vpminud {{.*}}(%rip), %xmm0, %xmm0
 ; AVX1-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vmovmskps %xmm1, %eax
-; AVX1-NEXT:    xorl $15, %eax
+; AVX1-NEXT:    xorb $15, %al
 ; AVX1-NEXT:    testb $1, %al
 ; AVX1-NEXT:    jne .LBB14_1
 ; AVX1-NEXT:  # %bb.2: # %else
 ; AVX1-NEXT:    testb $2, %al
 ; AVX1-NEXT:    jne .LBB14_3
 ; AVX1-NEXT:  .LBB14_4: # %else2
 ; AVX1-NEXT:    testb $4, %al
 ; AVX1-NEXT:    jne .LBB14_5

 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm3 = [255,255,255,255]
 ; AVX2-NEXT:    vpminud %xmm3, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackusdw %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpackuswb %xmm0, %xmm0, %xmm0
 ; AVX2-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vmovmskps %xmm1, %eax
-; AVX2-NEXT:    xorl $15, %eax
+; AVX2-NEXT:    xorb $15, %al
 ; AVX2-NEXT:    testb $1, %al
 ; AVX2-NEXT:    jne .LBB14_1
 ; AVX2-NEXT:  # %bb.2: # %else
 ; AVX2-NEXT:    testb $2, %al
 ; AVX2-NEXT:    jne .LBB14_3
 ; AVX2-NEXT:  .LBB14_4: # %else2
 ; AVX2-NEXT:    testb $4, %al
 ; AVX2-NEXT:    jne .LBB14_5

llvm/test/CodeGen/X86/movmsk-cmp.ll

 
 define i1 @movmsk_and_v2i64(<2 x i64> %x, <2 x i64> %y) {
 ; SSE2-LABEL: movmsk_and_v2i64:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm0
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm0[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm0, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorl $3, %eax
-; SSE2-NEXT:    cmpb $3, %al
+; SSE2-NEXT:    testb %al, %al
 ; SSE2-NEXT:    sete %al
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: movmsk_and_v2i64:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    pcmpeqq %xmm1, %xmm0
 ; SSE41-NEXT:    movmskpd %xmm0, %eax
-; SSE41-NEXT:    xorl $3, %eax
-; SSE41-NEXT:    cmpb $3, %al
+; SSE41-NEXT:    testb %al, %al
 ; SSE41-NEXT:    sete %al
 ; SSE41-NEXT:    retq
 ;
 ; AVX1OR2-LABEL: movmsk_and_v2i64:
 ; AVX1OR2:       # %bb.0:
 ; AVX1OR2-NEXT:    vpcmpeqq %xmm1, %xmm0, %xmm0
 ; AVX1OR2-NEXT:    vmovmskpd %xmm0, %eax
-; AVX1OR2-NEXT:    xorl $3, %eax
-; AVX1OR2-NEXT:    cmpb $3, %al
+; AVX1OR2-NEXT:    testb %al, %al
 ; AVX1OR2-NEXT:    sete %al
 ; AVX1OR2-NEXT:    retq
 ;
 ; KNL-LABEL: movmsk_and_v2i64:
 ; KNL:       # %bb.0:
 ; KNL-NEXT:    # kill: def $xmm1 killed $xmm1 def $zmm1
 ; KNL-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; KNL-NEXT:    vpcmpneqq %zmm1, %zmm0, %k0

 
 define i1 @movmsk_or_v2i64(<2 x i64> %x, <2 x i64> %y) {
 ; SSE2-LABEL: movmsk_or_v2i64:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pcmpeqd %xmm1, %xmm0
 ; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm0[1,0,3,2]
 ; SSE2-NEXT:    pand %xmm0, %xmm1
 ; SSE2-NEXT:    movmskpd %xmm1, %eax
-; SSE2-NEXT:    xorb $3, %al
+; SSE2-NEXT:    cmpb $3, %al
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: movmsk_or_v2i64:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    pcmpeqq %xmm1, %xmm0
 ; SSE41-NEXT:    movmskpd %xmm0, %eax
-; SSE41-NEXT:    xorb $3, %al
+; SSE41-NEXT:    cmpb $3, %al
 ; SSE41-NEXT:    setne %al
 ; SSE41-NEXT:    retq
 ;
 ; AVX1OR2-LABEL: movmsk_or_v2i64:
 ; AVX1OR2:       # %bb.0:
 ; AVX1OR2-NEXT:    vpcmpeqq %xmm1, %xmm0, %xmm0
 ; AVX1OR2-NEXT:    vmovmskpd %xmm0, %eax
-; AVX1OR2-NEXT:    xorb $3, %al
+; AVX1OR2-NEXT:    cmpb $3, %al
 ; AVX1OR2-NEXT:    setne %al
 ; AVX1OR2-NEXT:    retq
 ;
 ; KNL-LABEL: movmsk_or_v2i64:
 ; KNL:       # %bb.0:
 ; KNL-NEXT:    # kill: def $xmm1 killed $xmm1 def $zmm1
 ; KNL-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; KNL-NEXT:    vpcmpneqq %zmm1, %zmm0, %k0

llvm/test/CodeGen/X86/mul-constant-i8.ll

 }
 
 define i8 @test_mul_by_neg10(i8 %x) {
 ; X64-LABEL: test_mul_by_neg10:
 ; X64:       # %bb.0:
 ; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    addl %edi, %edi
 ; X64-NEXT:    leal (%rdi,%rdi,4), %eax
-; X64-NEXT:    negl %eax
+; X64-NEXT:    negb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = mul i8 %x, -10
   ret i8 %m
 }
 
 define i8 @test_mul_by_neg36(i8 %x) {
 ; X64-LABEL: test_mul_by_neg36:
 ; X64:       # %bb.0:
 ; X64-NEXT:    # kill: def $edi killed $edi def $rdi
 ; X64-NEXT:    shll $2, %edi
 ; X64-NEXT:    leal (%rdi,%rdi,8), %eax
-; X64-NEXT:    negl %eax
+; X64-NEXT:    negb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = mul i8 %x, -36
   ret i8 %m
 }
 

llvm/test/CodeGen/X86/parity.ll

 ; X64-NOPOPCNT-NEXT:    xorl %edi, %eax
 ; X64-NOPOPCNT-NEXT:    xorb %ah, %al
 ; X64-NOPOPCNT-NEXT:    setnp %al
 ; X64-NOPOPCNT-NEXT:    retq
 ;
 ; X86-POPCNT-LABEL: parity_32_trunc:
 ; X86-POPCNT:       # %bb.0:
 ; X86-POPCNT-NEXT:    popcntl {{[0-9]+}}(%esp), %eax
-; X86-POPCNT-NEXT:    andl $1, %eax
+; X86-POPCNT-NEXT:    andb $1, %al
 ; X86-POPCNT-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-POPCNT-NEXT:    retl
 ;
 ; X64-POPCNT-LABEL: parity_32_trunc:
 ; X64-POPCNT:       # %bb.0:
 ; X64-POPCNT-NEXT:    popcntl %edi, %eax
-; X64-POPCNT-NEXT:    andl $1, %eax
+; X64-POPCNT-NEXT:    andb $1, %al
 ; X64-POPCNT-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-POPCNT-NEXT:    retq
   %1 = tail call i32 @llvm.ctpop.i32(i32 %x)
   %2 = trunc i32 %1 to i8
   %3 = and i8 %2, 1
   ret i8 %3
 }
 

llvm/test/CodeGen/X86/pr15267.ll

; CHECK-NEXT:    retq
   ret <4 x i64> %sext
 }
 
 define <16 x i4> @test4(<16 x i4>* %in) nounwind {
 ; CHECK-LABEL: test4:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movq (%rdi), %rax
 ; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    shrl $4, %ecx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    shrb $4, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    movl %eax, %edx
-; CHECK-NEXT:    andl $15, %edx
+; CHECK-NEXT:    andb $15, %dl
+; CHECK-NEXT:    movzbl %dl, %edx
 ; CHECK-NEXT:    vmovd %edx, %xmm0
 ; CHECK-NEXT:    vpinsrb $1, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movl %eax, %ecx
 ; CHECK-NEXT:    shrl $8, %ecx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $2, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movl %eax, %ecx
 ; CHECK-NEXT:    shrl $12, %ecx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $3, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movl %eax, %ecx
 ; CHECK-NEXT:    shrl $16, %ecx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $4, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movl %eax, %ecx
 ; CHECK-NEXT:    shrl $20, %ecx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $5, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movl %eax, %ecx
 ; CHECK-NEXT:    shrl $24, %ecx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $6, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movl %eax, %ecx
 ; CHECK-NEXT:    shrl $28, %ecx
 ; CHECK-NEXT:    vpinsrb $7, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $32, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $8, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $36, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $9, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $40, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $10, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $44, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $11, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $48, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $12, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $52, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $13, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    movq %rax, %rcx
 ; CHECK-NEXT:    shrq $56, %rcx
-; CHECK-NEXT:    andl $15, %ecx
+; CHECK-NEXT:    andb $15, %cl
+; CHECK-NEXT:    movzbl %cl, %ecx
 ; CHECK-NEXT:    vpinsrb $14, %ecx, %xmm0, %xmm0
 ; CHECK-NEXT:    shrq $60, %rax
 ; CHECK-NEXT:    vpinsrb $15, %eax, %xmm0, %xmm0
 ; CHECK-NEXT:    retq
   %ret = load <16 x i4>, <16 x i4>* %in, align 1
   ret <16 x i4> %ret
 }

llvm/test/CodeGen/X86/pr40539.ll

 ; CHECK-NEXT:    fld1
 ; CHECK-NEXT:    fxch %st(1)
 ; CHECK-NEXT:    fyl2x
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    fstps (%esp)
 ; CHECK-NEXT:    xorps %xmm0, %xmm0
 ; CHECK-NEXT:    cmpeqss (%esp), %xmm0
 ; CHECK-NEXT:    movd %xmm0, %eax
-; CHECK-NEXT:    andl $1, %eax
+; CHECK-NEXT:    andb $1, %al
 ; CHECK-NEXT:    # kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    popl %ecx
 ; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 entry:
   %0 = load float, float* @f1, align 4
   %1 = fpext float %0 to x86_fp80
   %2 = tail call x86_fp80 asm "fld1; fxch; fyl2x", "={st},0,~{st(1)},~{dirflag},~{fpsr},~{flags}"(x86_fp80 %1)

llvm/test/CodeGen/X86/replace-load-and-with-bzhi.ll

 define i32 @f32_bzhi(i32 %x, i32 %y) local_unnamed_addr {
 ; CHECK-LABEL: f32_bzhi:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    bzhil %esi, %edi, %eax
 ; CHECK-NEXT:    retq
 ;
 ; CHECK32-LABEL: f32_bzhi:
 ; CHECK32:       # %bb.0: # %entry
-; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; CHECK32-NEXT:    bzhil %eax, {{[0-9]+}}(%esp), %eax
 ; CHECK32-NEXT:    retl
 entry:
   %idxprom = sext i32 %y to i64
   %arrayidx = getelementptr inbounds [32 x i32], [32 x i32]* @fill_table32, i64 0, i64 %idxprom
   %0 = load i32, i32* %arrayidx, align 4
   %and = and i32 %0, %x
   ret i32 %and
 }
 
 define i32 @f32_bzhi_partial(i32 %x, i32 %y) local_unnamed_addr {
 ; CHECK-LABEL: f32_bzhi_partial:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    bzhil %esi, %edi, %eax
 ; CHECK-NEXT:    retq
 ;
 ; CHECK32-LABEL: f32_bzhi_partial:
 ; CHECK32:       # %bb.0: # %entry
-; CHECK32-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK32-NEXT:    movb {{[0-9]+}}(%esp), %al
 ; CHECK32-NEXT:    bzhil %eax, {{[0-9]+}}(%esp), %eax
 ; CHECK32-NEXT:    retl
 entry:
   %idxprom = sext i32 %y to i64
   %arrayidx = getelementptr inbounds [17 x i32], [17 x i32]* @fill_table32_partial, i64 0, i64 %idxprom
   %0 = load i32, i32* %arrayidx, align 4
   %and = and i32 %0, %x
   ret i32 %and

llvm/test/CodeGen/X86/setoeq.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- -mattr=+sse2 | FileCheck %s
 
 define zeroext i8 @t(double %x) nounwind readnone {
 ; CHECK-LABEL: t:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; CHECK-NEXT:    cvttpd2dq %xmm0, %xmm1
 ; CHECK-NEXT:    cvtdq2pd %xmm1, %xmm1
 ; CHECK-NEXT:    cmpeqsd %xmm0, %xmm1
 ; CHECK-NEXT:    movd %xmm1, %eax
-; CHECK-NEXT:    andl $1, %eax
+; CHECK-NEXT:    andb $1, %al
 ; CHECK-NEXT:    # kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retl
 entry:
   %0 = fptosi double %x to i32    ; <i32> [#uses=1]
   %1 = sitofp i32 %0 to double    ; <double> [#uses=1]
   %2 = fcmp oeq double %1, %x    ; <i1> [#uses=1]
   %retval12 = zext i1 %2 to i8    ; <i8> [#uses=1]
   ret i8 %retval12
 }
 
 define zeroext i8 @u(double %x) nounwind readnone {
 ; CHECK-LABEL: u:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movsd {{.*#+}} xmm0 = mem[0],zero
 ; CHECK-NEXT:    cvttpd2dq %xmm0, %xmm1
 ; CHECK-NEXT:    cvtdq2pd %xmm1, %xmm1
 ; CHECK-NEXT:    cmpneqsd %xmm0, %xmm1
 ; CHECK-NEXT:    movd %xmm1, %eax
-; CHECK-NEXT:    andl $1, %eax
+; CHECK-NEXT:    andb $1, %al
 ; CHECK-NEXT:    # kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retl
 entry:
   %0 = fptosi double %x to i32    ; <i32> [#uses=1]
   %1 = sitofp i32 %0 to double    ; <double> [#uses=1]
   %2 = fcmp une double %1, %x    ; <i1> [#uses=1]
   %retval12 = zext i1 %2 to i8    ; <i8> [#uses=1]
   ret i8 %retval12
 }

llvm/test/CodeGen/X86/vector-compare-all_of.ll

; AVX512-NEXT:    retq
   ret i1 %d
 }
 
 define i1 @bool_reduction_v4i32(<4 x i32> %x, <4 x i32> %y) {
 ; SSE-LABEL: bool_reduction_v4i32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    pcmpeqd %xmm1, %xmm0
 ; SSE-NEXT:    movmskps %xmm0, %eax
-; SSE-NEXT:    xorl $15, %eax
-; SSE-NEXT:    cmpb $15, %al
+; SSE-NEXT:    testb %al, %al
 ; SSE-NEXT:    sete %al
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: bool_reduction_v4i32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpcmpeqd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vmovmskps %xmm0, %eax
-; AVX-NEXT:    xorl $15, %eax
-; AVX-NEXT:    cmpb $15, %al
+; AVX-NEXT:    testb %al, %al
 ; AVX-NEXT:    sete %al
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: bool_reduction_v4i32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vpcmpneqd %xmm1, %xmm0, %k0
 ; AVX512-NEXT:    kmovd %k0, %eax
 ; AVX512-NEXT:    cmpb $15, %al

llvm/test/CodeGen/X86/vector-compare-any_of.ll

; AVX512-NEXT:    retq
   ret i1 %d
 }
 
 define i1 @bool_reduction_v4i32(<4 x i32> %x, <4 x i32> %y) {
 ; SSE-LABEL: bool_reduction_v4i32:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    pcmpeqd %xmm1, %xmm0
 ; SSE-NEXT:    movmskps %xmm0, %eax
-; SSE-NEXT:    xorb $15, %al
+; SSE-NEXT:    cmpb $15, %al
 ; SSE-NEXT:    setne %al
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: bool_reduction_v4i32:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpcmpeqd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    vmovmskps %xmm0, %eax
-; AVX-NEXT:    xorb $15, %al
+; AVX-NEXT:    cmpb $15, %al
 ; AVX-NEXT:    setne %al
 ; AVX-NEXT:    retq
 ;
 ; AVX512-LABEL: bool_reduction_v4i32:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vpcmpneqd %xmm1, %xmm0, %k0
 ; AVX512-NEXT:    kmovd %k0, %eax
 ; AVX512-NEXT:    testb %al, %al