[DAG] Move lshr narrowing from visitANDLike to SimplifyDemandedBits

[RKSimon]

Summary

Inspired by some of the cases from D145468

Let SimplifyDemandedBits handle the narrowing of lshr to half-width if we don't require the upper bits, the narrowed shift is profitable and the zext/trunc are free.

[RKSimon]

llvm/test/CodeGen/X86/bswap.ll

171

Looks like an equivalent patch for shl narrowing would be useful

232

Not sure if shl narrowing would solve this or we need better zext/trunc handling in the bswap matcher

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505421)

Regression (WIP)

81 ↗

(On Diff #505421)

Regression (WIP)

95 ↗

(On Diff #505421)

Regression (WIP)

llvm/test/CodeGen/X86/illegal-bitfield-loadstore.ll

119 ↗

(On Diff #505421)

Regression (WIP) - dead store folds might fail on load-ext / store-trunc mixtures?

[RKSimon]

rebase

[RKSimon]

drop useless vector handling

[RKSimon]

rebase

[RKSimon]

rebase again (and actually update the changed test this time....)

[goldstein.w.n]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505570)

I think this is okay here. We only get the right codegen by chance here and I don't think its something we can reasonably control during DAG isel.

I tried to improve this with D141653. Looked good for the tests but caused infinite loop in bootstrap build.

I think this (along with other imm level optimizations), need to be moved to a new pass (or function in isel) that runs at the very end.

[kazu]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505570)

think this (along with other imm level optimizations), need to be moved to a new pass (or function in isel) that runs at the very end.

I was just thinking about something similar. Specifically, optimizations to achieve smaller encoding with the same opcode should move to a new pass. Otherwise, we would have to see through ISD::ZERO_EXTEND and ISD::TRUNCATE everywhere, and that would be prone to missed optimizations.

Do we have known bits and demanded bits infrastructure at the x86 MIR level? (I'm guessing not.) Also, I am wondering whether a new pass would be more effective if we use information across basic blocks.

[RKSimon]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505570)

Adding value tracking at that level would be a huge amount of work - both X86 and AMDGPU currently using the DAG narrowing code, but ideally we'd be working to enable it on other targets as well.

[goldstein.w.n]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505570)

I was thinking it would be best saved for a pass between DAG narrowing and MIR. I.e DAG narrowing -> Imm Fixup -> MIR. Imm Fixup could work on SDValue types.

[pengfei]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505570)

Any possibility it can be solved in a new ISel mechanism like GlobalISel?

[RKSimon]

Update - I'm going to investigate splitting this between the regular SimplifyDemandedBits DAG combines and some specific narrowing in X86ISelDAGToDAG.cpp + ISel - that way we have the best chance of making use of the extensive value tracking code we already have in SelectionDAG.

However, a lot of the DAG combines are already in place, we're just missing them due to poor combine ordering - so getting D127115 completed once and for all will likely help us the most, so I'm looking at the remaining regressions there first, and then will revisit this for cleanup.

[RKSimon]

rebasing this now that D127115 has landed, solving most of the issues

[RKSimon]

llvm/test/CodeGen/X86/bswap.ll

232

This looks to be the last regression - MatchBSwapHWordLow is very pattern specific, and can't peek through zext (or ignore AND masks for known zero bits)

[RKSimon]

llvm/test/CodeGen/X86/bswap.ll

232

I've confirmed this is fixed by adding an equivalent SHL narrowing fold in SimplifyDemandedBits, which I intend to do as a follow up.

[pengfei]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505421)

I saw you removed WIP in the title. Are you not considering this as regression or you will do it as a follow up?

[RKSimon]

llvm/test/CodeGen/X86/h-register-addressing-64.ll

67 ↗

(On Diff #505421)

Let me take another look - it seems to be due to matchAddressRecursively not being very good at peeking through ZERO_EXTEND nodes.

[RKSimon]

rebase

[goldstein.w.n]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1871

Doesn't the InDemandedMask.countLeadingZeros() >= (BitWidth / 2) check the same thing as TLO.DAG.MaskedValueIsZero(Op0, APInt::getHighBitsSet(BitWidth, BitWidth / 2)) What is the rationale for having both?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1871

No, InDemandedMask checks if we don't care about the upper bits at all (i.e. the SRL is used by a truncate or a AND mask like in the original implementation), and the MaskedValueIsZero alternatively checks if the upper bits are already known to be zero, in which case the (free) zext/trunc can preserve those bits correctly, using a more profitable narrower op, and possibly allowing further folds to occur. Removing either will result in test changes.

[RKSimon]

any more thoughts/comments?

[goldstein.w.n]

In D146121#4491315, @RKSimon wrote:

any more thoughts/comments?

One more comment.

I noticed in: D154805 that for the scalars the (shl (add x, c1), c2) is not properly folding to (add (shl x, c2), c1 << c2). This is because before reaching visitSHL (and performing the fold), the shl gets shrunk and its wrapped in an ANY_EXTEND.

This patch seems like to cause similiar issues. I'm still generally infavor as we can always update the folds, but is a slight concern that we may be adding a layer of indirection that not all folds work with.

[RKSimon]

I agree - a great deal of folds are poorly designed - we hit a lot of this with D127115 and are now seeing very similar things with D152928 - if you find examples, PLEASE write up an issue ticket as it speeds up triage a great deal.

[RKSimon]

ping?

[foad]

AMDGPU changes seem good.

[pengfei]

X86 changes look good to me expect one nit.

llvm/test/CodeGen/X86/2009-05-30-ISelBug.ll

12

This results in scale to be 4, which may do bad for performance?

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (ConstantSDNode *ADDI = dyn_cast<ConstantSDNode>(N0.getOperand(1))) {
               return SDValue(N, 0);
             }
           }
         }
       }
     }
   }
 
-  // Reduce bit extract of low half of an integer to the narrower type.
-  // (and (srl i64:x, K), KMask) ->
-  //   (i64 zero_extend (and (srl (i32 (trunc i64:x)), K)), KMask)
-  if (N0.getOpcode() == ISD::SRL && N0.hasOneUse()) {
-    if (ConstantSDNode *CAnd = dyn_cast<ConstantSDNode>(N1)) {
-      if (ConstantSDNode *CShift = dyn_cast<ConstantSDNode>(N0.getOperand(1))) {
-        unsigned Size = VT.getSizeInBits();
-        const APInt &AndMask = CAnd->getAPIntValue();
-        unsigned ShiftBits = CShift->getZExtValue();
-
-        // Bail out, this node will probably disappear anyway.
-        if (ShiftBits == 0)
-          return SDValue();
-
-        unsigned MaskBits = AndMask.countr_one();
-        EVT HalfVT = EVT::getIntegerVT(*DAG.getContext(), Size / 2);
-
-        if (AndMask.isMask() &&
-            // Required bits must not span the two halves of the integer and
-            // must fit in the half size type.
-            (ShiftBits + MaskBits <= Size / 2) &&
-            TLI.isNarrowingProfitable(VT, HalfVT) &&
-            TLI.isTypeDesirableForOp(ISD::AND, HalfVT) &&
-            TLI.isTypeDesirableForOp(ISD::SRL, HalfVT) &&
-            TLI.isTruncateFree(VT, HalfVT) &&
-            TLI.isZExtFree(HalfVT, VT)) {
-          // The isNarrowingProfitable is to avoid regressions on PPC and
-          // AArch64 which match a few 64-bit bit insert / bit extract patterns
-          // on downstream users of this. Those patterns could probably be
-          // extended to handle extensions mixed in.
-
-          SDValue SL(N0);
-          assert(MaskBits <= Size);
-
-          // Extracting the highest bit of the low half.
-          EVT ShiftVT = TLI.getShiftAmountTy(HalfVT, DAG.getDataLayout());
-          SDValue Trunc = DAG.getNode(ISD::TRUNCATE, SL, HalfVT,
-                                      N0.getOperand(0));
-
-          SDValue NewMask = DAG.getConstant(AndMask.trunc(Size / 2), SL, HalfVT);
-          SDValue ShiftK = DAG.getConstant(ShiftBits, SL, ShiftVT);
-          SDValue Shift = DAG.getNode(ISD::SRL, SL, HalfVT, Trunc, ShiftK);
-          SDValue And = DAG.getNode(ISD::AND, SL, HalfVT, Shift, NewMask);
-          return DAG.getNode(ISD::ZERO_EXTEND, SL, VT, And);
-        }
-      }
-    }
-  }
-
   return SDValue();
 }
 
 bool DAGCombiner::isAndLoadExtLoad(ConstantSDNode *AndC, LoadSDNode *LoadN,
                                    EVT LoadResultTy, EVT &ExtVT) {
   if (!AndC->getAPIntValue().isMask())
     return false;
 

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (const APInt *SA =
 
       APInt InDemandedMask = (DemandedBits << ShAmt);
 
       // If the shift is exact, then it does demand the low bits (and knows that
       // they are zero).
       if (Op->getFlags().hasExact())
         InDemandedMask.setLowBits(ShAmt);
 
+      // Narrow shift to lower half - similar to ShrinkDemandedOp.
+      // (srl i64:x, K) -> (i64 zero_extend (srl (i32 (trunc i64:x)), K))
+      if ((BitWidth % 2) == 0 && !VT.isVector() &&
+          ((InDemandedMask.countLeadingZeros() >= (BitWidth / 2)) ||
+           TLO.DAG.MaskedValueIsZero(
+               Op0, APInt::getHighBitsSet(BitWidth, BitWidth / 2)))) {
+        EVT HalfVT = EVT::getIntegerVT(*TLO.DAG.getContext(), BitWidth / 2);
+        if (isNarrowingProfitable(VT, HalfVT) &&
+            isTypeDesirableForOp(ISD::SRL, HalfVT) &&
+            isTruncateFree(VT, HalfVT) && isZExtFree(HalfVT, VT) &&
+            (!TLO.LegalOperations() || isOperationLegal(ISD::SRL, VT))) {
+          SDValue NewOp = TLO.DAG.getNode(ISD::TRUNCATE, dl, HalfVT, Op0);
+          SDValue NewShiftAmt = TLO.DAG.getShiftAmountConstant(
+              ShAmt, HalfVT, dl, TLO.LegalTypes());
+          SDValue NewShift =
+              TLO.DAG.getNode(ISD::SRL, dl, HalfVT, NewOp, NewShiftAmt);
+          return TLO.CombineTo(
+              Op, TLO.DAG.getNode(ISD::ZERO_EXTEND, dl, VT, NewShift));
+        }
+      }
+
       // Compute the new bits that are at the top now.
       if (SimplifyDemandedBits(Op0, InDemandedMask, DemandedElts, Known, TLO,
                                Depth + 1))
         return true;
       assert(!Known.hasConflict() && "Bits known to be one AND zero?");
       Known.Zero.lshrInPlace(ShAmt);
       Known.One.lshrInPlace(ShAmt);
       // High bits known zero.

llvm/test/CodeGen/AMDGPU/idot4s.ll

 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_i32 v3, v2, 16, 8
 ; GFX7-NEXT:    v_bfe_i32 v4, v2, 0, 8
+; GFX7-NEXT:    v_bfe_i32 v3, v2, 16, 8
+; GFX7-NEXT:    s_waitcnt vmcnt(1)
+; GFX7-NEXT:    v_bfe_i32 v7, v0, 0, 8
 ; GFX7-NEXT:    v_ashrrev_i32_e32 v5, 24, v2
 ; GFX7-NEXT:    v_bfe_i32 v2, v2, 8, 8
-; GFX7-NEXT:    s_waitcnt vmcnt(1)
+; GFX7-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX7-NEXT:    v_bfe_i32 v6, v0, 16, 8
-; GFX7-NEXT:    v_bfe_i32 v7, v0, 0, 8
 ; GFX7-NEXT:    v_ashrrev_i32_e32 v8, 24, v0
 ; GFX7-NEXT:    v_bfe_i32 v0, v0, 8, 8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; GFX7-NEXT:    v_and_b32_e32 v4, 0xffff, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
 ; GFX7-NEXT:    v_and_b32_e32 v7, 0xffff, v7
-; GFX7-NEXT:    v_alignbit_b32 v2, 0, v2, 16
-; GFX7-NEXT:    v_alignbit_b32 v0, 0, v0, 16
+; GFX7-NEXT:    v_and_b32_e32 v2, 0xffff, v2
+; GFX7-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v7, v1
 ; GFX7-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX7-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
 ; GFX7-NEXT:    v_and_b32_e32 v5, 0xffff, v5
 ; GFX7-NEXT:    v_and_b32_e32 v8, 0xffff, v8
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v6, v0

llvm/test/CodeGen/AMDGPU/idot4u.ll

 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_and_b32_e32 v3, 0xff00, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 24, v2
-; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_and_b32_e32 v6, 0xff00, v0
+; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 24, v2
 ; GFX7-NEXT:    v_and_b32_e32 v5, 0xff, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 24, v0
-; GFX7-NEXT:    v_alignbit_b32 v2, v4, v2, 16
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v3
-; GFX7-NEXT:    v_and_b32_e32 v4, 0xff, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
-; GFX7-NEXT:    v_alignbit_b32 v0, v7, v0, 16
-; GFX7-NEXT:    v_alignbit_b32 v3, 0, v3, 16
-; GFX7-NEXT:    v_alignbit_b32 v6, 0, v6, 16
+; GFX7-NEXT:    s_waitcnt vmcnt(1)
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v0
+; GFX7-NEXT:    v_and_b32_e32 v7, 0xff, v0
+; GFX7-NEXT:    v_bfe_u32 v4, v2, 8, 8
+; GFX7-NEXT:    v_alignbit_b32 v2, v3, v2, 16
+; GFX7-NEXT:    v_bfe_u32 v3, v0, 8, 8
+; GFX7-NEXT:    v_alignbit_b32 v0, v6, v0, 16
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v4, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v2
+; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v7, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
 ; GFX7-NEXT:    v_and_b32_e32 v2, 0xff, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v0
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 16, v0
 ; GFX7-NEXT:    v_and_b32_e32 v0, 0xff, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v3, v6, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v3, v1
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v8, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v5, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot4_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

llvm/test/CodeGen/AMDGPU/idot8s.ll

 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_i32 v8, v2, 0, 4
-; GFX7-NEXT:    v_bfe_i32 v7, v2, 4, 4
-; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_bfe_i32 v15, v0, 0, 4
-; GFX7-NEXT:    v_and_b32_e32 v8, 0xffff, v8
-; GFX7-NEXT:    v_bfe_i32 v14, v0, 4, 4
-; GFX7-NEXT:    v_and_b32_e32 v15, 0xffff, v15
-; GFX7-NEXT:    v_bfe_i32 v6, v2, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v7, 0xffff, v7
-; GFX7-NEXT:    v_bfe_i32 v13, v0, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v14, 0xffff, v14
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v15, v1
+; GFX7-NEXT:    v_bfe_i32 v6, v2, 0, 4
 ; GFX7-NEXT:    v_bfe_i32 v3, v2, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v4, v2, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v5, v2, 16, 4
-; GFX7-NEXT:    v_ashrrev_i32_e32 v9, 28, v2
-; GFX7-NEXT:    v_bfe_i32 v2, v2, 12, 4
+; GFX7-NEXT:    s_waitcnt vmcnt(1)
+; GFX7-NEXT:    v_bfe_i32 v13, v0, 0, 4
+; GFX7-NEXT:    v_bfe_i32 v4, v2, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v5, v2, 8, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v7, 28, v2
+; GFX7-NEXT:    v_bfe_i32 v8, v2, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v9, v2, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v2, v2, 4, 4
 ; GFX7-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX7-NEXT:    v_bfe_i32 v10, v0, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v11, v0, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v12, v0, 16, 4
-; GFX7-NEXT:    v_ashrrev_i32_e32 v16, 28, v0
-; GFX7-NEXT:    v_bfe_i32 v0, v0, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v11, v0, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v12, v0, 8, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v14, 28, v0
+; GFX7-NEXT:    v_bfe_i32 v15, v0, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v16, v0, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v0, v0, 4, 4
 ; GFX7-NEXT:    v_and_b32_e32 v13, 0xffff, v13
-; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v14, v1
 ; GFX7-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX7-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v13, v1
 ; GFX7-NEXT:    v_and_b32_e32 v5, 0xffff, v5
 ; GFX7-NEXT:    v_and_b32_e32 v12, 0xffff, v12
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    v_and_b32_e32 v9, 0xffff, v9
+; GFX7-NEXT:    v_and_b32_e32 v16, 0xffff, v16
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
 ; GFX7-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX7-NEXT:    v_and_b32_e32 v11, 0xffff, v11
-; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
+; GFX7-NEXT:    v_and_b32_e32 v8, 0xffff, v8
+; GFX7-NEXT:    v_and_b32_e32 v15, 0xffff, v15
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
 ; GFX7-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX7-NEXT:    v_and_b32_e32 v10, 0xffff, v10
-; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
-; GFX7-NEXT:    v_and_b32_e32 v9, 0xffff, v9
-; GFX7-NEXT:    v_and_b32_e32 v16, 0xffff, v16
+; GFX7-NEXT:    v_mad_u32_u24 v0, v8, v15, v0
+; GFX7-NEXT:    v_and_b32_e32 v7, 0xffff, v7
+; GFX7-NEXT:    v_and_b32_e32 v14, 0xffff, v14
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v10, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v14, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

 ; GFX7-NEXT:    v_bfe_i32 v16, v0, 12, 4
 ; GFX7-NEXT:    v_bfe_i32 v0, v0, 4, 4
 ; GFX7-NEXT:    v_and_b32_e32 v14, 0xff, v14
 ; GFX7-NEXT:    v_and_b32_e32 v2, 0xff, v2
 ; GFX7-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
 ; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v14, v1
 ; GFX7-NEXT:    v_and_b32_e32 v6, 0xff, v6
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 24, v9
 ; GFX7-NEXT:    v_and_b32_e32 v13, 0xff, v13
-; GFX7-NEXT:    v_lshlrev_b32_e32 v16, 24, v16
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_alignbit_b32 v9, 0, v9, 24
-; GFX7-NEXT:    v_alignbit_b32 v16, 0, v16, 24
+; GFX7-NEXT:    v_and_b32_e32 v9, 0xff, v9
+; GFX7-NEXT:    v_and_b32_e32 v16, 0xff, v16
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v13, v0
 ; GFX7-NEXT:    v_and_b32_e32 v5, 0xff, v5
 ; GFX7-NEXT:    v_and_b32_e32 v12, 0xff, v12
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
 ; GFX7-NEXT:    v_and_b32_e32 v4, 0xff, v4
 ; GFX7-NEXT:    v_and_b32_e32 v11, 0xff, v11
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
 ; GFX7-NEXT:    v_and_b32_e32 v3, 0xff, v3

llvm/test/CodeGen/AMDGPU/idot8u.ll

 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ubyte v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_and_b32_e32 v9, 15, v2
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v3, 28, v2
-; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_and_b32_e32 v16, 15, v0
 ; GFX7-NEXT:    v_bfe_u32 v4, v2, 24, 4
 ; GFX7-NEXT:    v_bfe_u32 v5, v2, 20, 4
 ; GFX7-NEXT:    v_bfe_u32 v6, v2, 16, 4
-; GFX7-NEXT:    v_bfe_u32 v7, v2, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v8, v2, 4, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 12, v2
+; GFX7-NEXT:    v_bfe_u32 v7, v2, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v8, v2, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v9, v2, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX7-NEXT:    s_waitcnt vmcnt(1)
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 28, v0
 ; GFX7-NEXT:    v_bfe_u32 v11, v0, 24, 4
 ; GFX7-NEXT:    v_bfe_u32 v12, v0, 20, 4
 ; GFX7-NEXT:    v_bfe_u32 v13, v0, 16, 4
-; GFX7-NEXT:    v_bfe_u32 v14, v0, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v15, v0, 4, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 12, v0
+; GFX7-NEXT:    v_bfe_u32 v14, v0, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v15, v0, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v16, v0, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v0, 15, v0
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v9, v16, v1
-; GFX7-NEXT:    v_and_b32_e32 v2, 0xf000000, v2
-; GFX7-NEXT:    v_and_b32_e32 v0, 0xf000000, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v15, v1
-; GFX7-NEXT:    v_alignbit_b32 v2, s10, v2, 24
-; GFX7-NEXT:    v_alignbit_b32 v0, 0, v0, 24
-; GFX7-NEXT:    v_mad_u32_u24 v1, v7, v14, v1
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v16, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v8, v15, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v14, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v13, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v10, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8_vecMul:

llvm/test/CodeGen/AMDGPU/integer-mad-patterns.ll

entry:
   %add6 = mul <2 x i16> %mul521, %add422
   ret <2 x i16> %add6
 }
 
 define <3 x i16> @clpeak_imad_pat_v3i16(<3 x i16> %x, <3 x i16> %y) {
 ; GFX67-SDAG-LABEL: clpeak_imad_pat_v3i16:
 ; GFX67-SDAG:       ; %bb.0: ; %entry
 ; GFX67-SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX67-SDAG-NEXT:    v_add_i32_e32 v1, vcc, 1, v1
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v7, 16, v1
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, 1, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v7, 0, v7, 16
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_add_i32_e32 v1, vcc, 1, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v7, v4, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v2, vcc, 1, v2
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v9, v7, v4
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v8, v3, v0
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v8, v8, v3, 1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v2
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v9, v8, v4
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v6, v3, v0
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v8, v4, v1
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v6, v6, v3, 1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v5
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v1, 0, v1, 16
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v8
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v9
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v6, v5, v2
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v7, v0, v3
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v7, v5, v2
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v8, v0, v3
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v4
-; GFX67-SDAG-NEXT:    v_or_b32_e32 v8, v9, v8
+; GFX67-SDAG-NEXT:    v_or_b32_e32 v6, v9, v6
 ; GFX67-SDAG-NEXT:    s_mov_b32 s4, 0x10000
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v0, v3, 1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_add_i32_e32 v8, vcc, s4, v8
+; GFX67-SDAG-NEXT:    v_add_i32_e32 v6, vcc, s4, v6
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v4, v2, v5
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v6, v6, v5, 1
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v7, v7, v5, 1
 ; GFX67-SDAG-NEXT:    v_or_b32_e32 v0, v3, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v3, 0, v8, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v3, 16, v6
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v7
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v2, v5, 1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v7
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v8
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v8
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v3
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v5, v5, v7
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v6
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v5, v5, v6
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v4, 0, v0, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v4, 16, v0
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v5
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v0, v5, v0
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v4
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v2, v3, v2
 ; GFX67-SDAG-NEXT:    s_setpc_b64 s[30:31]
entry:
   ret <3 x i16> %add33
 }
 
 define <4 x i16> @clpeak_imad_pat_v4i16(<4 x i16> %x, <4 x i16> %y) {
 ; GFX67-SDAG-LABEL: clpeak_imad_pat_v4i16:
 ; GFX67-SDAG:       ; %bb.0: ; %entry
 ; GFX67-SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v3, vcc, 1, v3
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v10, 0xffff, v3
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v11, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v7
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v2, vcc, 1, v2
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v3, v11, v7, v3
+; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, 1, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v9, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v1, vcc, 1, v1
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v3, v10, v7, v3
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
-; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, 1, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v9, 0, v9, 16
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v10, 0xffff, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v5
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v13, v10, v7
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v13, v11, v7
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v7
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v7, v8, v6, 1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v11, 0xffff, v0
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v8, v6, v2
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v9, v5, v1
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v7, v9, v6, 1
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v12, v10, v5
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v9, v6, v2
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v8, v4, v0
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v10, v5, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v7
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v8, 16, v13
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v12, v9, v5
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v11, v4, v0
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX67-SDAG-NEXT:    v_or_b32_e32 v7, v8, v7
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v8, v11, v4, 1
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v1, 0, v1, 16
+; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v13
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v8, v8, v4, 1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX67-SDAG-NEXT:    v_or_b32_e32 v7, v9, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v8
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v10, 16, v12
+; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v12
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v9, v0, v4
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v10, v0, v4
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v5
 ; GFX67-SDAG-NEXT:    s_mov_b32 s4, 0x10000
-; GFX67-SDAG-NEXT:    v_or_b32_e32 v8, v10, v8
+; GFX67-SDAG-NEXT:    v_or_b32_e32 v8, v9, v8
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v0, v4, 1
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v5, v2, v6
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v8, vcc, s4, v8
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v2, v6, 1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v4, 16, v1
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v7, vcc, s4, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v6, 16, v3
 ; GFX67-SDAG-NEXT:    v_or_b32_e32 v0, v4, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v4, 0, v8, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v4, 16, v8
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_or_b32_e32 v2, v6, v2
 ; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v6, 16, v7
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v9, 0xffff, v9
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v9, 0xffff, v10
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v8
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v4
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v5
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v2, vcc, s4, v2
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v8, v9, v8
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v4, v4, v5
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v6
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v5, 0, v0, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v5, 16, v0
 ; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v8
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v0, v7, v0
entry:
   %add6 = mul <2 x i16> %mul521, %add422
   ret <2 x i16> %add6
 }
 
 define <3 x i16> @clpeak_umad_pat_v3i16(<3 x i16> %x, <3 x i16> %y) {
 ; GFX67-SDAG-LABEL: clpeak_umad_pat_v3i16:
 ; GFX67-SDAG:       ; %bb.0: ; %entry
 ; GFX67-SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; GFX67-SDAG-NEXT:    v_add_i32_e32 v1, vcc, 1, v1
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v7, 16, v1
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, 1, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v7, 0, v7, 16
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_add_i32_e32 v1, vcc, 1, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v7, v4, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v2, vcc, 1, v2
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v9, v7, v4
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v8, v3, v0
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v8, v8, v3, 1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v2
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v9, v8, v4
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v6, v3, v0
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v8, v4, v1
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v6, v6, v3, 1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v5
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v1, 0, v1, 16
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v8
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v9
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v6, v5, v2
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v7, v0, v3
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v7, v5, v2
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v8, v0, v3
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v4
-; GFX67-SDAG-NEXT:    v_or_b32_e32 v8, v9, v8
+; GFX67-SDAG-NEXT:    v_or_b32_e32 v6, v9, v6
 ; GFX67-SDAG-NEXT:    s_mov_b32 s4, 0x10000
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v0, v3, 1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_add_i32_e32 v8, vcc, s4, v8
+; GFX67-SDAG-NEXT:    v_add_i32_e32 v6, vcc, s4, v6
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v3, 16, v1
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v4, v2, v5
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v6, v6, v5, 1
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v7, v7, v5, 1
 ; GFX67-SDAG-NEXT:    v_or_b32_e32 v0, v3, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v3, 0, v8, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v3, 16, v6
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v7
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v2, v5, 1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v7
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v8
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v8
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v3
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v5, v5, v7
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v6
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v5, v5, v6
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v4, 0, v0, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v4, 16, v0
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v5
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v0, v5, v0
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v4
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v2, v3, v2
 ; GFX67-SDAG-NEXT:    s_setpc_b64 s[30:31]
entry:
   ret <3 x i16> %add33
 }
 
 define <4 x i16> @clpeak_umad_pat_v4i16(<4 x i16> %x, <4 x i16> %y) {
 ; GFX67-SDAG-LABEL: clpeak_umad_pat_v4i16:
 ; GFX67-SDAG:       ; %bb.0: ; %entry
 ; GFX67-SDAG-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v3, vcc, 1, v3
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v10, 0xffff, v3
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v11, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v7
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v2, vcc, 1, v2
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v3, v11, v7, v3
+; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, 1, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v9, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v1, vcc, 1, v1
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v3, v10, v7, v3
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v6, 0xffff, v6
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
-; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, 1, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v9, 0, v9, 16
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v10, 0xffff, v1
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v5
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v13, v10, v7
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v13, v11, v7
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v7
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v7, v8, v6, 1
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v11, 0xffff, v0
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v8, v6, v2
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v9, v5, v1
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v7, v9, v6, 1
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v12, v10, v5
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v9, v6, v2
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v8, v4, v0
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v1, v10, v5, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v7
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v8, 16, v13
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v12, v9, v5
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v11, v4, v0
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX67-SDAG-NEXT:    v_or_b32_e32 v7, v8, v7
-; GFX67-SDAG-NEXT:    v_mad_u32_u24 v8, v11, v4, 1
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v1, 0, v1, 16
+; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v13
+; GFX67-SDAG-NEXT:    v_mad_u32_u24 v8, v8, v4, 1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
+; GFX67-SDAG-NEXT:    v_or_b32_e32 v7, v9, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v8
-; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v10, 16, v12
+; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v9, 16, v12
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
-; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v9, v0, v4
+; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v10, v0, v4
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v5
 ; GFX67-SDAG-NEXT:    s_mov_b32 s4, 0x10000
-; GFX67-SDAG-NEXT:    v_or_b32_e32 v8, v10, v8
+; GFX67-SDAG-NEXT:    v_or_b32_e32 v8, v9, v8
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v0, v0, v4, 1
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v5, v2, v6
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v8, vcc, s4, v8
 ; GFX67-SDAG-NEXT:    v_mad_u32_u24 v2, v2, v6, 1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v4, 16, v1
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v7, vcc, s4, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_lshlrev_b32_e32 v6, 16, v3
 ; GFX67-SDAG-NEXT:    v_or_b32_e32 v0, v4, v0
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v4, 0, v8, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v4, 16, v8
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_or_b32_e32 v2, v6, v2
 ; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v6, 16, v7
-; GFX67-SDAG-NEXT:    v_and_b32_e32 v9, 0xffff, v9
+; GFX67-SDAG-NEXT:    v_and_b32_e32 v9, 0xffff, v10
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v8, 0xffff, v8
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v1, v1, v4
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v5
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v5, 0xffff, v7
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v2, vcc, s4, v2
 ; GFX67-SDAG-NEXT:    v_add_i32_e32 v0, vcc, s4, v0
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v8, v9, v8
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v4, v4, v5
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v3, v3, v6
-; GFX67-SDAG-NEXT:    v_alignbit_b32 v5, 0, v0, 16
+; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v5, 16, v0
 ; GFX67-SDAG-NEXT:    v_lshrrev_b32_e32 v6, 16, v2
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v7, 0xffff, v8
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v0, 0xffff, v0
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v1, 0xffff, v1
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v2, 0xffff, v2
 ; GFX67-SDAG-NEXT:    v_and_b32_e32 v3, 0xffff, v3
 ; GFX67-SDAG-NEXT:    v_mul_u32_u24_e32 v0, v7, v0

llvm/test/CodeGen/AMDGPU/partial-shift-shrink.ll

; GCN-NEXT:    s_setpc_b64 s[30:31]
   ret i16 %trunc
 }
 
 define i32 @trunc_srl_i64_25_to_i26(i64 %x) {
 ; GCN-LABEL: trunc_srl_i64_25_to_i26:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    v_and_b32_e32 v0, 0xa000000, v0
-; GCN-NEXT:    v_alignbit_b32 v0, 0, v0, 25
+; GCN-NEXT:    v_lshrrev_b32_e32 v0, 25, v0
 ; GCN-NEXT:    v_add_u32_e32 v0, 55, v0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %value.knownbits2 = and i64 %x, 167772160 ; 0xA000000
   %shift = lshr i64 %value.knownbits2, 25
   %trunc = trunc i64 %shift to i26
   %add = add i26 %trunc, 55
   %ext = zext i26 %add to i32
   ret i32 %ext
 }

llvm/test/CodeGen/AMDGPU/sdwa-peephole.ll

 ; NOSDWA-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
 ; NOSDWA-NEXT:    s_waitcnt lgkmcnt(0)
 ; NOSDWA-NEXT:    v_mov_b32_e32 v0, s0
 ; NOSDWA-NEXT:    v_mov_b32_e32 v1, s1
 ; NOSDWA-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
 ; NOSDWA-NEXT:    v_mov_b32_e32 v2, s2
 ; NOSDWA-NEXT:    v_mov_b32_e32 v3, s3
 ; NOSDWA-NEXT:    s_waitcnt vmcnt(0)
-; NOSDWA-NEXT:    v_lshrrev_b64 v[4:5], 24, v[0:1]
-; NOSDWA-NEXT:    v_and_b32_e32 v6, 0xff, v0
-; NOSDWA-NEXT:    v_lshrrev_b32_e32 v7, 8, v0
+; NOSDWA-NEXT:    v_and_b32_e32 v4, 0xff, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v5, 8, v0
+; NOSDWA-NEXT:    v_lshrrev_b32_e32 v6, 24, v0
 ; NOSDWA-NEXT:    v_lshrrev_b32_e32 v0, 16, v0
-; NOSDWA-NEXT:    v_and_b32_e32 v5, 0xff, v1
+; NOSDWA-NEXT:    v_and_b32_e32 v7, 0xff, v1
 ; NOSDWA-NEXT:    v_lshrrev_b32_e32 v8, 8, v1
 ; NOSDWA-NEXT:    v_lshrrev_b32_e32 v9, 24, v1
 ; NOSDWA-NEXT:    v_lshrrev_b32_e32 v1, 16, v1
-; NOSDWA-NEXT:    v_lshlrev_b16_e32 v7, 8, v7
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v5, 8, v5
+; NOSDWA-NEXT:    v_lshlrev_b16_e32 v6, 8, v6
 ; NOSDWA-NEXT:    v_and_b32_e32 v0, 0xff, v0
 ; NOSDWA-NEXT:    v_lshlrev_b16_e32 v8, 8, v8
 ; NOSDWA-NEXT:    v_lshlrev_b16_e32 v9, 8, v9
 ; NOSDWA-NEXT:    v_and_b32_e32 v1, 0xff, v1
-; NOSDWA-NEXT:    v_lshlrev_b16_e32 v4, 8, v4
-; NOSDWA-NEXT:    v_or_b32_e32 v6, v6, v7
-; NOSDWA-NEXT:    v_or_b32_e32 v5, v5, v8
+; NOSDWA-NEXT:    v_or_b32_e32 v4, v4, v5
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v6
+; NOSDWA-NEXT:    v_or_b32_e32 v5, v7, v8
 ; NOSDWA-NEXT:    v_or_b32_e32 v1, v1, v9
-; NOSDWA-NEXT:    v_or_b32_e32 v0, v0, v4
-; NOSDWA-NEXT:    v_and_b32_e32 v6, 0xffff, v6
-; NOSDWA-NEXT:    v_and_b32_e32 v4, 0xffff, v5
-; NOSDWA-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_and_b32_e32 v4, 0xffff, v4
 ; NOSDWA-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; NOSDWA-NEXT:    v_or_b32_e32 v0, v6, v0
-; NOSDWA-NEXT:    v_or_b32_e32 v1, v4, v1
+; NOSDWA-NEXT:    v_and_b32_e32 v5, 0xffff, v5
+; NOSDWA-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
+; NOSDWA-NEXT:    v_or_b32_e32 v0, v4, v0
+; NOSDWA-NEXT:    v_or_b32_e32 v1, v5, v1
 ; NOSDWA-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; NOSDWA-NEXT:    s_endpgm
 ;
 ; GFX89-LABEL: pulled_out_test:
 ; GFX89:       ; %bb.0: ; %entry
 ; GFX89-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX89-NEXT:    v_mov_b32_e32 v6, 8
-; GFX89-NEXT:    v_mov_b32_e32 v7, 0xff
+; GFX89-NEXT:    v_mov_b32_e32 v4, 8
+; GFX89-NEXT:    v_mov_b32_e32 v5, 0xff
 ; GFX89-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX89-NEXT:    v_mov_b32_e32 v0, s0
 ; GFX89-NEXT:    v_mov_b32_e32 v1, s1
 ; GFX89-NEXT:    flat_load_dwordx2 v[0:1], v[0:1]
 ; GFX89-NEXT:    v_mov_b32_e32 v2, s2
 ; GFX89-NEXT:    v_mov_b32_e32 v3, s3
 ; GFX89-NEXT:    s_waitcnt vmcnt(0)
-; GFX89-NEXT:    v_lshrrev_b64 v[4:5], 24, v[0:1]
-; GFX89-NEXT:    v_lshrrev_b32_sdwa v8, v6, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX89-NEXT:    v_lshrrev_b32_sdwa v6, v6, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_lshrrev_b32_sdwa v6, v4, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_lshrrev_b32_e32 v7, 24, v0
+; GFX89-NEXT:    v_lshrrev_b32_sdwa v4, v4, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX89-NEXT:    v_lshrrev_b32_e32 v9, 24, v1
-; GFX89-NEXT:    v_and_b32_sdwa v5, v0, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX89-NEXT:    v_and_b32_sdwa v7, v1, v7 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX89-NEXT:    v_lshlrev_b16_e32 v6, 8, v9
-; GFX89-NEXT:    v_lshlrev_b16_e32 v4, 8, v4
-; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v8 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX89-NEXT:    v_or_b32_sdwa v6, v7, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX89-NEXT:    v_and_b32_sdwa v8, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX89-NEXT:    v_and_b32_sdwa v5, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_lshlrev_b16_e32 v6, 8, v7
+; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX89-NEXT:    v_lshlrev_b16_e32 v4, 8, v9
+; GFX89-NEXT:    v_or_b32_sdwa v6, v8, v6 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX89-NEXT:    v_or_b32_sdwa v4, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX89-NEXT:    v_or_b32_sdwa v1, v1, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX89-NEXT:    flat_store_dwordx2 v[2:3], v[0:1]
 ; GFX89-NEXT:    s_endpgm
 ;
 ; GFX9-LABEL: pulled_out_test:
 ; GFX9:       ; %bb.0: ; %entry
 ; GFX9-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX9-NEXT:    v_mov_b32_e32 v4, 0
-; GFX9-NEXT:    v_mov_b32_e32 v5, 8
+; GFX9-NEXT:    v_mov_b32_e32 v2, 0
+; GFX9-NEXT:    v_mov_b32_e32 v3, 8
 ; GFX9-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX9-NEXT:    global_load_dwordx2 v[0:1], v4, s[0:1]
+; GFX9-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
 ; GFX9-NEXT:    s_movk_i32 s0, 0xff
 ; GFX9-NEXT:    s_waitcnt vmcnt(0)
-; GFX9-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
-; GFX9-NEXT:    v_lshrrev_b32_sdwa v6, v5, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_lshrrev_b32_sdwa v5, v5, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_sdwa v4, v3, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_lshrrev_b32_e32 v5, 24, v0
+; GFX9-NEXT:    v_lshrrev_b32_sdwa v3, v3, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
 ; GFX9-NEXT:    v_lshrrev_b32_e32 v7, 24, v1
-; GFX9-NEXT:    v_and_b32_sdwa v3, v0, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX9-NEXT:    v_and_b32_sdwa v6, v0, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
 ; GFX9-NEXT:    v_and_b32_sdwa v8, v1, s0 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_lshlrev_b16_e32 v5, 8, v7
-; GFX9-NEXT:    v_lshlrev_b16_e32 v2, 8, v2
-; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v5, v8, v5 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v2, v3, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX9-NEXT:    global_store_dwordx2 v4, v[0:1], s[2:3]
+; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e32 v4, 8, v5
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX9-NEXT:    v_lshlrev_b16_e32 v3, 8, v7
+; GFX9-NEXT:    v_or_b32_sdwa v4, v6, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v3, v8, v3 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v0, v0, v4 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
+; GFX9-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
 ; GFX9-NEXT:    s_endpgm
 ;
 ; GFX10-LABEL: pulled_out_test:
 ; GFX10:       ; %bb.0: ; %entry
 ; GFX10-NEXT:    s_load_dwordx4 s[0:3], s[0:1], 0x24
-; GFX10-NEXT:    v_mov_b32_e32 v4, 0
-; GFX10-NEXT:    v_mov_b32_e32 v5, 8
-; GFX10-NEXT:    v_mov_b32_e32 v6, 0xff
-; GFX10-NEXT:    v_mov_b32_e32 v7, 24
-; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX10-NEXT:    global_load_dwordx2 v[0:1], v4, s[0:1]
-; GFX10-NEXT:    s_waitcnt vmcnt(0)
-; GFX10-NEXT:    v_lshrrev_b64 v[2:3], 24, v[0:1]
-; GFX10-NEXT:    v_lshrrev_b32_sdwa v3, v5, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-NEXT:    v_and_b32_sdwa v8, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10-NEXT:    v_lshrrev_b32_sdwa v5, v5, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-NEXT:    v_lshrrev_b32_sdwa v7, v7, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-NEXT:    v_and_b32_sdwa v6, v1, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
-; GFX10-NEXT:    v_lshlrev_b16 v2, 8, v2
-; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
-; GFX10-NEXT:    v_or_b32_sdwa v3, v6, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
-; GFX10-NEXT:    v_or_b32_sdwa v2, v8, v2 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_mov_b32_e32 v2, 0
+; GFX10-NEXT:    v_mov_b32_e32 v3, 8
+; GFX10-NEXT:    v_mov_b32_e32 v4, 24
+; GFX10-NEXT:    v_mov_b32_e32 v5, 0xff
+; GFX10-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX10-NEXT:    global_load_dwordx2 v[0:1], v2, s[0:1]
+; GFX10-NEXT:    s_waitcnt vmcnt(0)
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v6, v3, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v7, v4, v0 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_and_b32_sdwa v8, v0, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v3, v3, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_lshrrev_b32_sdwa v4, v4, v1 dst_sel:BYTE_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_and_b32_sdwa v5, v1, v5 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v6, v8, v7 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:BYTE_0 src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v3, v5, v4 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:DWORD
+; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v6 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
 ; GFX10-NEXT:    v_or_b32_sdwa v1, v1, v3 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-NEXT:    v_or_b32_sdwa v0, v0, v2 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_0 src1_sel:DWORD
-; GFX10-NEXT:    global_store_dwordx2 v4, v[0:1], s[2:3]
+; GFX10-NEXT:    global_store_dwordx2 v2, v[0:1], s[2:3]
 ; GFX10-NEXT:    s_endpgm
 entry:
   %idxprom = ashr exact i64 15, 32
   %arrayidx = getelementptr inbounds <8 x i8>, ptr addrspace(1) %sourceA, i64 %idxprom
   %tmp = load <8 x i8>, ptr addrspace(1) %arrayidx, align 8
 
   %tmp1 = extractelement <8 x i8> %tmp, i32 0
   %tmp2 = extractelement <8 x i8> %tmp, i32 1

llvm/test/CodeGen/AMDGPU/shift-i128.ll

 
 define i128 @v_lshr_i128_kv(i128 %rhs) {
 ; GCN-LABEL: v_lshr_i128_kv:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    s_mov_b64 s[4:5], 0x41
 ; GCN-NEXT:    v_lshr_b64 v[1:2], s[4:5], v0
 ; GCN-NEXT:    v_cmp_gt_u32_e32 vcc, 64, v0
-; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v0
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
-; GCN-NEXT:    v_mov_b32_e32 v3, 0x41
-; GCN-NEXT:    s_and_b64 vcc, s[4:5], vcc
-; GCN-NEXT:    v_cndmask_b32_e64 v0, v3, v1, s[4:5]
-; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
+; GCN-NEXT:    v_mov_b32_e32 v2, 0x41
+; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; GCN-NEXT:    v_cndmask_b32_e32 v0, v2, v1, vcc
+; GCN-NEXT:    v_mov_b32_e32 v1, 0
 ; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    v_mov_b32_e32 v3, 0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = lshr i128 65, %rhs
   ret i128 %shl
 }
 
 define i128 @v_ashr_i128_kv(i128 %rhs) {
 ; GCN-LABEL: v_ashr_i128_kv:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    v_lshr_b64 v[1:2], 33, v0
 ; GCN-NEXT:    v_cmp_gt_u32_e32 vcc, 64, v0
-; GCN-NEXT:    v_cmp_ne_u32_e64 s[4:5], 0, v0
 ; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v1, vcc
-; GCN-NEXT:    s_and_b64 vcc, s[4:5], vcc
-; GCN-NEXT:    v_cndmask_b32_e64 v0, 33, v1, s[4:5]
-; GCN-NEXT:    v_cndmask_b32_e32 v1, 0, v2, vcc
+; GCN-NEXT:    v_cmp_ne_u32_e32 vcc, 0, v0
+; GCN-NEXT:    v_cndmask_b32_e32 v0, 33, v1, vcc
+; GCN-NEXT:    v_mov_b32_e32 v1, 0
 ; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    v_mov_b32_e32 v3, 0
 ; GCN-NEXT:    s_setpc_b64 s[30:31]
   %shl = ashr i128 33, %rhs
   ret i128 %shl
 }
 
 define amdgpu_kernel void @s_shl_i128_ss(i128 %lhs, i128 %rhs) {

llvm/test/CodeGen/AMDGPU/wave32.ll

 ; GFX1032-NEXT:    v_sub_nc_u32_e32 v0, v0, v1
 ; GFX1032-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
 ; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s0, v0
 ; GFX1032-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
 ; GFX1032-NEXT:    v_subrev_nc_u32_e32 v1, s0, v0
 ; GFX1032-NEXT:    v_cmp_le_u32_e32 vcc_lo, s0, v0
 ; GFX1032-NEXT:    v_cndmask_b32_e32 v0, v0, v1, vcc_lo
 ; GFX1032-NEXT:    v_cmp_eq_u32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    v_alignbit_b32 v0, 0, vcc_lo, 1
-; GFX1032-NEXT:    v_readfirstlane_b32 s0, v0
+; GFX1032-NEXT:    s_lshr_b32 s0, vcc_lo, 1
 ; GFX1032-NEXT:    s_ff1_i32_b32 s0, s0
 ; GFX1032-NEXT:    s_min_u32 s0, s0, s1
 ; GFX1032-NEXT:    s_cmp_gt_u32 s0, 9
 ; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
 ; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
 ; GFX1032-NEXT:  ; %bb.1: ; %if.then
 ; GFX1032-NEXT:    ; divergent unreachable

 ; GFX1032-NEXT:    v_fma_f32 v5, -v1, v3, v4
 ; GFX1032-NEXT:    v_fmac_f32_e32 v3, v5, v2
 ; GFX1032-NEXT:    v_fma_f32 v1, -v1, v3, v4
 ; GFX1032-NEXT:    v_div_fmas_f32 v1, v1, v2, v3
 ; GFX1032-NEXT:    v_div_fixup_f32 v1, v1, s0, v0
 ; GFX1032-NEXT:    v_trunc_f32_e32 v1, v1
 ; GFX1032-NEXT:    v_fma_f32 v0, -v1, s0, v0
 ; GFX1032-NEXT:    v_cmp_eq_f32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    v_alignbit_b32 v1, 0, vcc_lo, 1
+; GFX1032-NEXT:    s_lshr_b32 s0, vcc_lo, 1
 ; GFX1032-NEXT:    v_cmp_nlg_f32_e32 vcc_lo, 0, v0
-; GFX1032-NEXT:    v_readfirstlane_b32 s0, v1
 ; GFX1032-NEXT:    s_ff1_i32_b32 s0, s0
 ; GFX1032-NEXT:    s_min_u32 s0, s0, s1
 ; GFX1032-NEXT:    s_cmp_gt_u32 s0, 9
 ; GFX1032-NEXT:    s_cselect_b32 s0, -1, 0
 ; GFX1032-NEXT:    s_and_b32 s0, vcc_lo, s0
 ; GFX1032-NEXT:    s_and_saveexec_b32 s1, s0
 ; GFX1032-NEXT:  ; %bb.1: ; %if.then
 ; GFX1032-NEXT:    ; divergent unreachable

llvm/test/CodeGen/X86/2008-05-12-tailmerge-5.ll

 define void @passing2(i64 %str.0, i64 %str.1, i16 signext  %s, i32 %j, i8 signext  %c, i16 signext  %t, i16 signext  %u, i8 signext  %d) nounwind optsize {
 ; CHECK-LABEL: passing2:
 ; CHECK:       ## %bb.0: ## %entry
 ; CHECK-NEXT:    subq $40, %rsp
 ; CHECK-NEXT:    movq %rsi, %rax
 ; CHECK-NEXT:    movq %rdi, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movb %al, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movb %ah, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrq $16, %rsi
-; CHECK-NEXT:    movb %sil, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    shrq $24, %rax
+; CHECK-NEXT:    shrq $16, %rax
 ; CHECK-NEXT:    movb %al, {{[0-9]+}}(%rsp)
+; CHECK-NEXT:    movb %ah, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movw %dx, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movl %ecx, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movb %r8b, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movw %r9w, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    shll $14, %edi
 ; CHECK-NEXT:    sarl $23, %edi
 ; CHECK-NEXT:    cmpl %ecx, %edi
 ; CHECK-NEXT:    jne LBB0_6

llvm/test/CodeGen/X86/2009-05-30-ISelBug.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
 
 define void @BZ2_bzDecompress_bb5_2E_outer_bb35_2E_i_bb54_2E_i(ptr, i32 %c_nblock_used.2.i, i32 %.reload51, ptr %.out, ptr %.out1, ptr %.out2, ptr %.out3) nounwind {
 ; CHECK-LABEL: BZ2_bzDecompress_bb5_2E_outer_bb35_2E_i_bb54_2E_i:
 ; CHECK:       # %bb.0: # %newFuncRoot
 ; CHECK-NEXT:    movq {{[0-9]+}}(%rsp), %rax
 ; CHECK-NEXT:    movl %edx, %edx
 ; CHECK-NEXT:    movl (%rdi,%rdx,4), %edx
 ; CHECK-NEXT:    movzbl %dl, %r10d
+; CHECK-NEXT:    # kill: def $edx killed $edx def $rdx
+; CHECK-NEXT:    shrl $8, %edx
 ; CHECK-NEXT:    addl $4, %r10d
-; CHECK-NEXT:    shrq $6, %rdx
-; CHECK-NEXT:    andl $67108860, %edx # imm = 0x3FFFFFC
-; CHECK-NEXT:    movl (%rdi,%rdx), %edx
+; CHECK-NEXT:    movl (%rdi,%rdx,4), %edx
 ; CHECK-NEXT:    movzbl %dl, %edi
 ; CHECK-NEXT:    shrl $8, %edx
 ; CHECK-NEXT:    addl $5, %esi
 ; CHECK-NEXT:    movl %r10d, (%rcx)
 ; CHECK-NEXT:    movl %edi, (%r8)
 ; CHECK-NEXT:    movl %edx, (%r9)
 ; CHECK-NEXT:    movl %esi, (%rax)
 ; CHECK-NEXT:    retq

llvm/test/CodeGen/X86/3addr-or.ll

 }
 
 ; This test no longer requires or to be converted to 3 addr form because we are
 ; are able to use a zero extend instead of an 'and' which gives the register
 ; allocator freedom.
 define i64 @test2(i8 %A, i8 %B) nounwind {
 ; CHECK-LABEL: test2:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    shll $4, %edi
 ; CHECK-NEXT:    andl $48, %edi
 ; CHECK-NEXT:    movzbl %sil, %eax
-; CHECK-NEXT:    shrq $4, %rax
-; CHECK-NEXT:    orq %rdi, %rax
+; CHECK-NEXT:    shrl $4, %eax
+; CHECK-NEXT:    orl %edi, %eax
 ; CHECK-NEXT:    retq
   %C = zext i8 %A to i64
   %D = shl i64 %C, 4
   %E = and i64 %D, 48
   %F = zext i8 %B to i64
   %G = lshr i64 %F, 4
   %H = or i64 %G, %E
   ret i64 %H

llvm/test/CodeGen/X86/and-shift.ll

 ; X86-NEXT:    shrl $30, %eax
 ; X86-NEXT:    andl $-2, %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: shift30_and2_i64:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
-; X64-NEXT:    shrq $30, %rax
-; X64-NEXT:    andl $2, %eax
+; X64-NEXT:    shrl $30, %eax
+; X64-NEXT:    andl $-2, %eax
 ; X64-NEXT:    retq
   %shr = lshr i64 %x, 30
   %and = and i64 %shr, 2
   ret i64 %and
 }

llvm/test/CodeGen/X86/bswap.ll

 ; CHECK-NEXT:    shll $8, %eax
 ; CHECK-NEXT:    orl %ecx, %eax
 ; CHECK-NEXT:    xorl %edx, %edx
 ; CHECK-NEXT:    retl
 ;
 ; CHECK64-LABEL: not_bswap:
 ; CHECK64:       # %bb.0:
 ; CHECK64-NEXT:    movzwl var16(%rip), %eax
-; CHECK64-NEXT:    movq %rax, %rcx
-; CHECK64-NEXT:    shrq $8, %rcx
+; CHECK64-NEXT:    movl %eax, %ecx
+; CHECK64-NEXT:    shrl $8, %ecx
 ; CHECK64-NEXT:    shlq $8, %rax
 ; CHECK64-NEXT:    orq %rcx, %rax
 ; CHECK64-NEXT:    retq
   %init = load i16, ptr @var16
   %big = zext i16 %init to i64
 
   %hishifted = lshr i64 %big, 8
   %loshifted = shl i64 %big, 8
 

 ; CHECK-NEXT:    movzwl var16, %eax
 ; CHECK-NEXT:    bswapl %eax
 ; CHECK-NEXT:    shrl $16, %eax
 ; CHECK-NEXT:    xorl %edx, %edx
 ; CHECK-NEXT:    retl
 ;
 ; CHECK64-LABEL: finally_useful_bswap:
 ; CHECK64:       # %bb.0:
-; CHECK64-NEXT:    movzwl var16(%rip), %eax
-; CHECK64-NEXT:    bswapq %rax
-; CHECK64-NEXT:    shrq $48, %rax
+; CHECK64-NEXT:    movzwl var16(%rip), %ecx
+; CHECK64-NEXT:    movzbl %cl, %eax
+; CHECK64-NEXT:    # kill: def $ecx killed $ecx killed $rcx def $rcx
+; CHECK64-NEXT:    shrl $8, %ecx
+; CHECK64-NEXT:    shlq $8, %rax
+; CHECK64-NEXT:    orq %rcx, %rax
 ; CHECK64-NEXT:    retq
   %init = load i16, ptr @var16
   %big = zext i16 %init to i64
 
   %hishifted = lshr i64 %big, 8
   %lomasked = and i64 %big, 255
   %loshifted = shl i64 %lomasked, 8
 

llvm/test/CodeGen/X86/combine-bitreverse.ll

 ; X64-NEXT:    shrl %eax
 ; X64-NEXT:    andl $1431655765, %eax # imm = 0x55555555
 ; X64-NEXT:    leal (%rax,%rcx,2), %eax
 ; X64-NEXT:    shlq $33, %rax
 ; X64-NEXT:    bswapq %rax
 ; X64-NEXT:    movl %eax, %ecx
 ; X64-NEXT:    andl $235867919, %ecx # imm = 0xE0F0F0F
 ; X64-NEXT:    shlq $4, %rcx
-; X64-NEXT:    shrq $4, %rax
+; X64-NEXT:    shrl $4, %eax
 ; X64-NEXT:    andl $252645135, %eax # imm = 0xF0F0F0F
 ; X64-NEXT:    orq %rcx, %rax
 ; X64-NEXT:    movl %eax, %ecx
 ; X64-NEXT:    andl $590558003, %ecx # imm = 0x23333333
-; X64-NEXT:    shrq $2, %rax
+; X64-NEXT:    shrl $2, %eax
 ; X64-NEXT:    andl $858993459, %eax # imm = 0x33333333
 ; X64-NEXT:    leaq (%rax,%rcx,4), %rax
-; X64-NEXT:    movabsq $6148914691236517205, %rcx # imm = 0x5555555555555555
-; X64-NEXT:    movq %rax, %rdx
-; X64-NEXT:    andq %rcx, %rdx
-; X64-NEXT:    shrq %rax
-; X64-NEXT:    andq %rcx, %rax
-; X64-NEXT:    leaq (%rax,%rdx,2), %rax
+; X64-NEXT:    movl %eax, %ecx
+; X64-NEXT:    andl $357913941, %ecx # imm = 0x15555555
+; X64-NEXT:    shrl %eax
+; X64-NEXT:    andl $1431655765, %eax # imm = 0x55555555
+; X64-NEXT:    leaq (%rax,%rcx,2), %rax
 ; X64-NEXT:    retq
     %1 = call i64 @llvm.bitreverse.i64(i64 %a)
     %2 = shl i64 %1, 33
     %3 = call i64 @llvm.bitreverse.i64(i64 %2)
     ret i64 %3
 }
 
 define <4 x i32> @test_demandedbits_bitreverse(<4 x i32> %a0) nounwind {

llvm/test/CodeGen/X86/const-shift-of-constmasked.ll

 ; X86-NEXT:    shrl %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test_i64_2147483647_mask_lshr_1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    andl $2147483646, %eax # imm = 0x7FFFFFFE
-; X64-NEXT:    shrq %rax
+; X64-NEXT:    shrl %eax
 ; X64-NEXT:    retq
   %t0 = and i64 %a0, 2147483647
   %t1 = lshr i64 %t0, 1
   ret i64 %t1
 }
 
 define i64 @test_i64_140737488289792_mask_lshr_15(i64 %a0) {
 ; X86-LABEL: test_i64_140737488289792_mask_lshr_15:

 ; X86-NEXT:    shrl %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test_i64_2147483647_mask_ashr_1:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movq %rdi, %rax
 ; X64-NEXT:    andl $2147483646, %eax # imm = 0x7FFFFFFE
-; X64-NEXT:    shrq %rax
+; X64-NEXT:    shrl %eax
 ; X64-NEXT:    retq
   %t0 = and i64 %a0, 2147483647
   %t1 = ashr i64 %t0, 1
   ret i64 %t1
 }
 
 define i64 @test_i64_140737488289792_mask_ashr_15(i64 %a0) {
 ; X86-LABEL: test_i64_140737488289792_mask_ashr_15:

llvm/test/CodeGen/X86/extract-bits.ll

 ; X86-BMITBM-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-BMITBM-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-BMITBM-NEXT:    bextrl $2581, (%ecx), %ecx # imm = 0xA15
 ; X86-BMITBM-NEXT:    incl (%eax,%ecx,4)
 ; X86-BMITBM-NEXT:    retl
 ;
 ; X64-NOBMI-LABEL: pr38938:
 ; X64-NOBMI:       # %bb.0:
-; X64-NOBMI-NEXT:    movq (%rsi), %rax
-; X64-NOBMI-NEXT:    shrq $19, %rax
-; X64-NOBMI-NEXT:    andl $4092, %eax # imm = 0xFFC
-; X64-NOBMI-NEXT:    incl (%rdi,%rax)
+; X64-NOBMI-NEXT:    movl (%rsi), %eax
+; X64-NOBMI-NEXT:    shrl $21, %eax
+; X64-NOBMI-NEXT:    andl $1023, %eax # imm = 0x3FF
+; X64-NOBMI-NEXT:    incl (%rdi,%rax,4)
 ; X64-NOBMI-NEXT:    retq
 ;
 ; X64-BMINOTBM-LABEL: pr38938:
 ; X64-BMINOTBM:       # %bb.0:
 ; X64-BMINOTBM-NEXT:    movl $2581, %eax # imm = 0xA15
-; X64-BMINOTBM-NEXT:    bextrq %rax, (%rsi), %rax
+; X64-BMINOTBM-NEXT:    bextrl %eax, (%rsi), %eax
 ; X64-BMINOTBM-NEXT:    incl (%rdi,%rax,4)
 ; X64-BMINOTBM-NEXT:    retq
 ;
 ; X64-BMITBM-LABEL: pr38938:
 ; X64-BMITBM:       # %bb.0:
-; X64-BMITBM-NEXT:    bextrq $2581, (%rsi), %rax # imm = 0xA15
+; X64-BMITBM-NEXT:    bextrl $2581, (%rsi), %eax # imm = 0xA15
 ; X64-BMITBM-NEXT:    incl (%rdi,%rax,4)
 ; X64-BMITBM-NEXT:    retq
   %tmp = load i64, ptr %a1, align 8
   %tmp1 = lshr i64 %tmp, 21
   %tmp2 = and i64 %tmp1, 1023
   %tmp3 = getelementptr inbounds i32, ptr %a0, i64 %tmp2
   %tmp4 = load i32, ptr %tmp3, align 4
   %tmp5 = add nsw i32 %tmp4, 1

llvm/test/CodeGen/X86/h-registers-0.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mattr=-bmi -mtriple=x86_64-linux | FileCheck %s -check-prefix=X86-64 -check-prefix=X64
 ; RUN: llc < %s -mattr=-bmi -mtriple=x86_64-linux-gnux32 | FileCheck %s -check-prefix=X86-64 -check-prefix=X32
 ; RUN: llc < %s -mattr=-bmi -mtriple=x86_64-win32 | FileCheck %s -check-prefix=WIN64
 ; RUN: llc < %s -mattr=-bmi -mtriple=i686--    | FileCheck %s -check-prefix=X86-32
 
 ; Use h registers. On x86-64, codegen doesn't support general allocation
 ; of h registers yet, due to x86 encoding complications.
 
 define void @bar64(i64 inreg %x, ptr inreg %p) nounwind {
 ; X64-LABEL: bar64:
 ; X64:       # %bb.0:
-; X64-NEXT:    shrq $8, %rdi
+; X64-NEXT:    shrl $8, %edi
 ; X64-NEXT:    incb %dil
 ; X64-NEXT:    movb %dil, (%rsi)
 ; X64-NEXT:    retq
 ;
 ; X32-LABEL: bar64:
 ; X32:       # %bb.0:
-; X32-NEXT:    shrq $8, %rdi
+; X32-NEXT:    shrl $8, %edi
 ; X32-NEXT:    incb %dil
 ; X32-NEXT:    movb %dil, (%esi)
 ; X32-NEXT:    retq
 ;
 ; WIN64-LABEL: bar64:
 ; WIN64:       # %bb.0:
-; WIN64-NEXT:    shrq $8, %rcx
+; WIN64-NEXT:    shrl $8, %ecx
 ; WIN64-NEXT:    incb %cl
 ; WIN64-NEXT:    movb %cl, (%rdx)
 ; WIN64-NEXT:    retq
 ;
 ; X86-32-LABEL: bar64:
 ; X86-32:       # %bb.0:
 ; X86-32-NEXT:    incb %ah
 ; X86-32-NEXT:    movb %ah, (%ecx)

llvm/test/CodeGen/X86/lzcnt-cmp.ll

 ; X86-LZCNT-NEXT:    cmovel %eax, %ecx
 ; X86-LZCNT-NEXT:    testb $64, %cl
 ; X86-LZCNT-NEXT:    setne %al
 ; X86-LZCNT-NEXT:    retl
 ;
 ; X64-BSR-LABEL: lshr_ctlz_undef_cmpeq_one_i64:
 ; X64-BSR:       # %bb.0:
 ; X64-BSR-NEXT:    bsrq %rdi, %rax
-; X64-BSR-NEXT:    shrq $6, %rax
+; X64-BSR-NEXT:    shrl $6, %eax
 ; X64-BSR-NEXT:    cmpl $1, %eax
 ; X64-BSR-NEXT:    sete %al
 ; X64-BSR-NEXT:    retq
 ;
 ; X64-LZCNT-LABEL: lshr_ctlz_undef_cmpeq_one_i64:
 ; X64-LZCNT:       # %bb.0:
 ; X64-LZCNT-NEXT:    lzcntq %rdi, %rax
-; X64-LZCNT-NEXT:    shrq $6, %rax
+; X64-LZCNT-NEXT:    shrl $6, %eax
 ; X64-LZCNT-NEXT:    cmpl $1, %eax
 ; X64-LZCNT-NEXT:    sete %al
 ; X64-LZCNT-NEXT:    retq
   %ctlz = call i64 @llvm.ctlz.i64(i64 %in, i1 -1)
   %lshr = lshr i64 %ctlz, 6
   %icmp = icmp eq i64 %lshr, 1
   ret i1 %icmp
 }

 ; X86-LZCNT-NEXT:    cmovel %eax, %ecx
 ; X86-LZCNT-NEXT:    testb $64, %cl
 ; X86-LZCNT-NEXT:    setne %al
 ; X86-LZCNT-NEXT:    retl
 ;
 ; X64-BSR-LABEL: lshr_ctlz_undef_cmpne_zero_i64:
 ; X64-BSR:       # %bb.0:
 ; X64-BSR-NEXT:    bsrq %rdi, %rax
-; X64-BSR-NEXT:    testq $-64, %rax
+; X64-BSR-NEXT:    testl $-64, %eax
 ; X64-BSR-NEXT:    setne %al
 ; X64-BSR-NEXT:    retq
 ;
 ; X64-LZCNT-LABEL: lshr_ctlz_undef_cmpne_zero_i64:
 ; X64-LZCNT:       # %bb.0:
 ; X64-LZCNT-NEXT:    lzcntq %rdi, %rax
 ; X64-LZCNT-NEXT:    testb $64, %al
 ; X64-LZCNT-NEXT:    setne %al

llvm/test/CodeGen/X86/zext-logicop-shift-load.ll

 ; X86-NEXT:    movzbl (%eax), %eax
 ; X86-NEXT:    shrl $2, %eax
 ; X86-NEXT:    andl $-4, %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test4:
 ; X64:       # %bb.0: # %entry
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    shrq $2, %rax
-; X64-NEXT:    andl $60, %eax
+; X64-NEXT:    movzbl (%rdi), %eax
+; X64-NEXT:    shrl $2, %eax
+; X64-NEXT:    andl $-4, %eax
 ; X64-NEXT:    retq
 entry:
   %bf.load = load i8, ptr %data, align 4
   %bf.clear = lshr i8 %bf.load, 2
   %0 = and i8 %bf.clear, 60
   %1 = zext i8 %0 to i64
   ret i64 %1
 }
 
 define i64 @test5(ptr %data) {
 ; X86-LABEL: test5:
 ; X86:       # %bb.0: # %entry
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzbl (%eax), %eax
 ; X86-NEXT:    shrl $2, %eax
 ; X86-NEXT:    xorl $60, %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test5:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movzbl (%rdi), %eax
-; X64-NEXT:    shrq $2, %rax
+; X64-NEXT:    shrl $2, %eax
 ; X64-NEXT:    xorq $60, %rax
 ; X64-NEXT:    retq
 entry:
   %bf.load = load i8, ptr %data, align 4
   %bf.clear = lshr i8 %bf.load, 2
   %0 = xor i8 %bf.clear, 60
   %1 = zext i8 %0 to i64
   ret i64 %1
 }
 
 define i64 @test6(ptr %data) {
 ; X86-LABEL: test6:
 ; X86:       # %bb.0: # %entry
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzbl (%eax), %eax
 ; X86-NEXT:    shrl $2, %eax
 ; X86-NEXT:    orl $60, %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: test6:
 ; X64:       # %bb.0: # %entry
 ; X64-NEXT:    movzbl (%rdi), %eax
-; X64-NEXT:    shrq $2, %rax
+; X64-NEXT:    shrl $2, %eax
 ; X64-NEXT:    orq $60, %rax
 ; X64-NEXT:    retq
 entry:
   %bf.load = load i8, ptr %data, align 4
   %bf.clear = lshr i8 %bf.load, 2
   %0 = or i8 %bf.clear, 60
   %1 = zext i8 %0 to i64
   ret i64 %1

llvm/test/CodeGen/X86/zext-lshr.ll

 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    shrl $5, %eax
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: i64_zext_shift_i16_zext_i8:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl %dil, %eax
-; X64-NEXT:    shrq $5, %rax
+; X64-NEXT:    shrl $5, %eax
 ; X64-NEXT:    retq
   %t0 = zext i8 %a0 to i16
   %t1 = lshr i16 %t0, 5
   %t2 = zext i16 %t1 to i64
   ret i64 %t2
 }
 
 define i64 @i64_zext_shift_i32_zext_i8(i8 %a0) nounwind {

 ; X86-NEXT:    movl $0, 12(%eax)
 ; X86-NEXT:    movl $0, 8(%eax)
 ; X86-NEXT:    movl $0, 4(%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: i128_zext_shift_i64_zext_i8:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl %dil, %eax
-; X64-NEXT:    shrq $4, %rax
+; X64-NEXT:    shrl $4, %eax
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    retq
   %t0 = zext i8 %a0 to i64
   %t1 = lshr i64 %t0, 4
   %t2 = zext i64 %t1 to i128
   ret i128 %t2
 }
 
 define i128 @i128_zext_shift_i64_zext_i16(i16 %a0) nounwind {
 ; X86-LABEL: i128_zext_shift_i64_zext_i16:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzwl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    shrl $7, %ecx
 ; X86-NEXT:    movl %ecx, (%eax)
 ; X86-NEXT:    movl $0, 12(%eax)
 ; X86-NEXT:    movl $0, 8(%eax)
 ; X86-NEXT:    movl $0, 4(%eax)
 ; X86-NEXT:    retl $4
 ;
 ; X64-LABEL: i128_zext_shift_i64_zext_i16:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzwl %di, %eax
-; X64-NEXT:    shrq $7, %rax
+; X64-NEXT:    shrl $7, %eax
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    retq
   %t0 = zext i16 %a0 to i64
   %t1 = lshr i64 %t0,7
   %t2 = zext i64 %t1 to i128
   ret i128 %t2
 }