[TargetLowering] Improve SimplifyDemandedBits for AND and OR

[foad]

Summary

Simplify each operand of AND and OR using demanded bits based on the
known bits of the other operand.

[foad]

@nikic this could perhaps increase compile time. Could you test it please?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1216

If we're worried about the cost of the extra SimplifyDemandedBits, we might be able to just create Op0DemandedBits/Op1DemandedBits from the knowbits and only use these on the SimplifyMultipleUseDemandedBits cases below? The calls currently only use the original DemandedBits.

[nikic]

In D87464#2265954, @foad wrote:

@nikic this could perhaps increase compile time. Could you test it please?

Not seeing any impact from this change (https://llvm-compile-time-tracker.com/compare.php?from=a5168bdb4a25485ac62e18bdc538b4842bc9fbd9&to=c23aa3c1c163fc5343b1df5254b8c0782d4324cb&stat=instructions).

[foad]

In D87464#2267330, @nikic wrote:

In D87464#2265954, @foad wrote:

@nikic this could perhaps increase compile time. Could you test it please?

Not seeing any impact from this change (https://llvm-compile-time-tracker.com/compare.php?from=a5168bdb4a25485ac62e18bdc538b4842bc9fbd9&to=c23aa3c1c163fc5343b1df5254b8c0782d4324cb&stat=instructions).

Thanks!

[foad]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1216

I did try using Op0DemandedBits/Op1DemandedBits for the calls to SimplifyMultipleUseDemandedBits, in addition to the current patch. It only affected one test case and the diff isn't very compelling:

diff --git a/llvm/test/CodeGen/X86/shift-parts.ll b/llvm/test/CodeGen/X86/shift-parts.ll
index da00f377020d..ef82065ad12a 100644
--- a/llvm/test/CodeGen/X86/shift-parts.ll
+++ b/llvm/test/CodeGen/X86/shift-parts.ll
@@ -12,11 +12,12 @@ define i32 @int87(i32 %uint64p_8, i1 %cond) nounwind {
 ; CHECK-NEXT:    movq g_144+{{.*}}(%rip), %rax
 ; CHECK-NEXT:    movq g_144+{{.*}}(%rip), %rcx
 ; CHECK-NEXT:    movzbl %sil, %edx
+; CHECK-NEXT:    andl $1, %edx
 ; CHECK-NEXT:    shll $6, %edx
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_1: # %for.cond
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    testb $64, %dl
+; CHECK-NEXT:    testb %dl, %dl
 ; CHECK-NEXT:    movq %rcx, %rsi
 ; CHECK-NEXT:    cmovneq %rax, %rsi
 ; CHECK-NEXT:    orl $0, %esi

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1267

Can we trust Known2.One's value is correct for the bits that weren't checked due to Known.One being used to filter the DemandedBits for the previous call?

[foad]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1267

No we can't according to the comment on SimplifyDemandedBits:

/// expression (used to simplify the caller).  The KnownZero/One bits may only
/// be accurate for those bits in the Demanded masks.

This was a pre-existing problem with the call on line 1264.

[foad]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1267

It seems to me that there are various pre-existing problems in this area.

This should be a no-op but causes test case diffs: https://github.com/jayfoad/llvm-project/commit/dfdc519d44ac4f5ad00d9d5322d641e2fcc5881a

This fixes the pre-existing problem I pointed out on line 1264, but causes various code quality regressions: https://github.com/jayfoad/llvm-project/commit/90a5c28ba29edf54fb22ea36bf1ba2625ce557a4

I'm not sure how to proceed now. Should we try to get to a state where known bits information is trustworthy for undemanded bits?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1267

If you're willing to try to do the audit then by all means - we should at least be able to get to a position where all the KnownBits One/Zero bits will not be set unless they are definitely known - but some cases (e.g. bitcasts through vectors, partial demanded elts etc.) we might need to play safe.

[xbolva00]

Reping

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1202–1203

Pull out

APInt Op0DemandedBits = ~Known.Zero & DemandedBits;

1221

Op0DemandedBits ?

1223

Op1DemandedBits ?

[foad]

Rebase. Pass Op0/1DemandedBits into SimplifyMultipleUseDemandedBits.

[foad]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

1221

Done. The only effect on the tests is to add an instruction outside the loop in X86/shift-parts.ll which does not seem like a significant regression to me.

[foad]

In D87464#2476205, @xbolva00 wrote:

Reping

The status is: I'm still nervous about this patch because of the issue @craig.topper pointed out with SimplifyDemandedBits not necessarily returning accurate "known" information for bits that are not demanded.

[foad]

Rebase.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (ConstantSDNode *RHSC = isConstOrConstSplat(Op1)) {
         SDValue Xor = TLO.DAG.getNode(ISD::XOR, dl, VT, Op0.getOperand(0), Op1);
         return TLO.CombineTo(Op, Xor);
       }
     }
 
     if (SimplifyDemandedBits(Op1, DemandedBits, DemandedElts, Known, TLO,
                              Depth + 1))
       return true;
     assert(!Known.hasConflict() && "Bits known to be one AND zero?");
-    if (SimplifyDemandedBits(Op0, ~Known.Zero & DemandedBits, DemandedElts,
-                             Known2, TLO, Depth + 1))
+    APInt Op0DemandedBits = ~Known.Zero & DemandedBits;
+    if (SimplifyDemandedBits(Op0, Op0DemandedBits, DemandedElts, Known2, TLO,
+                             Depth + 1))
       return true;
     assert(!Known2.hasConflict() && "Bits known to be one AND zero?");
 
+    // If we have learned that some more bits of Op1 are not demanded due to
+    // known bits in Op0, try simplifying Op1 again.
+    APInt Op1DemandedBits = ~Known2.Zero & DemandedBits;
+    if (Op1DemandedBits != DemandedBits &&
+        SimplifyDemandedBits(Op1, Op1DemandedBits, DemandedElts, Known, TLO,
+                             Depth + 1))
+      return true;
+    assert(!Known.hasConflict() && "Bits known to be one AND zero?");
+
     // Attempt to avoid multi-use ops if we don't need anything from them.
     if (!DemandedBits.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
       SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
-          Op0, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+          Op0, Op0DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
       SDValue DemandedOp1 = SimplifyMultipleUseDemandedBits(
-          Op1, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
+          Op1, Op1DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
       if (DemandedOp0 || DemandedOp1) {
         Op0 = DemandedOp0 ? DemandedOp0 : Op0;
         Op1 = DemandedOp1 ? DemandedOp1 : Op1;
         SDValue NewOp = TLO.DAG.getNode(Op.getOpcode(), dl, VT, Op0, Op1);
         return TLO.CombineTo(Op, NewOp);
       }
     }
 
if (SimplifyDemandedBits(Op1, DemandedBits, DemandedElts, Known, TLO,
                              Depth + 1))
       return true;
     assert(!Known.hasConflict() && "Bits known to be one AND zero?");
     if (SimplifyDemandedBits(Op0, ~Known.One & DemandedBits, DemandedElts,
                              Known2, TLO, Depth + 1))
       return true;
     assert(!Known2.hasConflict() && "Bits known to be one AND zero?");
 
+    // If we have learned that some more bits of Op1 are not demanded due to
+    // known bits in Op0, try simplifying Op1 again.
+    APInt Op1DemandedBits = ~Known2.One & DemandedBits;
+    if (Op1DemandedBits != DemandedBits &&
+        SimplifyDemandedBits(Op1, Op1DemandedBits, DemandedElts, Known, TLO,
+                             Depth + 1))
+      return true;
+    assert(!Known.hasConflict() && "Bits known to be one AND zero?");
+
     // Attempt to avoid multi-use ops if we don't need anything from them.
     if (!DemandedBits.isAllOnesValue() || !DemandedElts.isAllOnesValue()) {
       SDValue DemandedOp0 = SimplifyMultipleUseDemandedBits(
           Op0, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
       SDValue DemandedOp1 = SimplifyMultipleUseDemandedBits(
           Op1, DemandedBits, DemandedElts, TLO.DAG, Depth + 1);
       if (DemandedOp0 || DemandedOp1) {
         Op0 = DemandedOp0 ? DemandedOp0 : Op0;

llvm/test/CodeGen/AMDGPU/idot4s.ll

 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v8, off, s[0:3], 0
 ; GFX7-NEXT:    s_mov_b32 s4, 0xffff
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_i32 v1, v2, 8, 8
 ; GFX7-NEXT:    v_bfe_i32 v3, v2, 0, 8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
+; GFX7-NEXT:    v_bfe_i32 v1, v2, 16, 8
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_bfe_i32 v5, v0, 8, 8
 ; GFX7-NEXT:    v_bfe_i32 v6, v0, 0, 8
-; GFX7-NEXT:    v_or_b32_e32 v1, v3, v1
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v5, s4, v6
-; GFX7-NEXT:    v_bfe_i32 v7, v0, 16, 8
-; GFX7-NEXT:    v_or_b32_e32 v3, v5, v3
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v7
-; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_ashrrev_i32_e32 v4, 24, v2
+; GFX7-NEXT:    v_bfe_i32 v2, v2, 8, 8
+; GFX7-NEXT:    v_bfe_i32 v5, v0, 16, 8
+; GFX7-NEXT:    v_ashrrev_i32_e32 v7, 24, v0
+; GFX7-NEXT:    v_bfe_i32 v0, v0, 8, 8
 ; GFX7-NEXT:    v_and_b32_e32 v3, s4, v3
-; GFX7-NEXT:    v_bfe_i32 v4, v2, 16, 8
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v1, v3, v8
-; GFX7-NEXT:    v_ashrrev_i32_e32 v2, 24, v2
-; GFX7-NEXT:    v_ashrrev_i32_e32 v0, 24, v0
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v7, v1
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
 ; GFX7-NEXT:    v_and_b32_e32 v2, s4, v2
 ; GFX7-NEXT:    v_and_b32_e32 v0, s4, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v6, v1
-; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_mad_u32_u24 v3, v3, v6, v8
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
+; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v3
+; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
+; GFX7-NEXT:    v_and_b32_e32 v7, s4, v7
+; GFX7-NEXT:    v_mad_u32_u24 v0, v1, v5, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v7, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot4_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

llvm/test/CodeGen/AMDGPU/idot4u.ll

 ; GFX7-NEXT:    s_mov_b32 s10, 0
 ; GFX7-NEXT:    s_mov_b32 s11, s3
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
-; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
-; GFX7-NEXT:    buffer_load_ushort v8, off, s[0:3], 0
+; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
+; GFX7-NEXT:    buffer_load_ushort v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_mov_b32 s4, 0xff00
 ; GFX7-NEXT:    s_movk_i32 s5, 0xff
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_and_b32_e32 v1, s4, v2
-; GFX7-NEXT:    v_and_b32_e32 v3, s5, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 8, v1
+; GFX7-NEXT:    v_and_b32_e32 v3, s4, v2
+; GFX7-NEXT:    v_and_b32_e32 v5, s5, v2
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_and_b32_e32 v5, s4, v0
-; GFX7-NEXT:    v_or_b32_e32 v1, v3, v1
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 8, v5
-; GFX7-NEXT:    v_and_b32_e32 v6, s5, v0
-; GFX7-NEXT:    v_or_b32_e32 v3, v6, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 16, v3
-; GFX7-NEXT:    v_and_b32_e32 v1, s5, v1
-; GFX7-NEXT:    v_and_b32_e32 v3, s5, v3
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v0
+; GFX7-NEXT:    v_and_b32_e32 v8, s5, v0
+; GFX7-NEXT:    v_bfe_u32 v3, v3, 8, 16
+; GFX7-NEXT:    v_bfe_u32 v6, v6, 8, 16
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v1, v1, v3, v8
+; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v8, v1
 ; GFX7-NEXT:    v_bfe_u32 v4, v2, 16, 8
 ; GFX7-NEXT:    v_bfe_u32 v7, v0, 16, 8
-; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v6, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v3, v6, v1
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v2, 24, v2
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v0, 24, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v7, v1
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot4_acc16_vecMul:

llvm/test/CodeGen/AMDGPU/idot8s.ll

 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    s_mov_b32 s10, 0
 ; GFX7-NEXT:    s_mov_b32 s11, s3
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v3, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
-; GFX7-NEXT:    s_mov_b32 s4, 0xffff
-; GFX7-NEXT:    v_mov_b32_e32 v2, 0xffff
 ; GFX7-NEXT:    s_mov_b32 s2, -1
+; GFX7-NEXT:    v_mov_b32_e32 v2, 0xffff
+; GFX7-NEXT:    s_mov_b32 s4, 0xffff
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_bfe_i32 v1, v3, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v4, v3, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v5, v3, 4, 4
-; GFX7-NEXT:    v_bfe_i32 v6, v3, 0, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_bfe_i32 v10, v0, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v11, v0, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v12, v0, 4, 4
-; GFX7-NEXT:    v_bfe_i32 v13, v0, 0, 4
-; GFX7-NEXT:    v_or_b32_e32 v1, v4, v1
-; GFX7-NEXT:    v_or_b32_e32 v4, v6, v5
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v10
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v11
-; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 16, v12
-; GFX7-NEXT:    v_and_b32_e32 v11, v2, v13
-; GFX7-NEXT:    v_bfe_i32 v7, v3, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v8, v3, 8, 4
+; GFX7-NEXT:    v_bfe_i32 v1, v3, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v4, v3, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v5, v3, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v6, v3, 8, 4
+; GFX7-NEXT:    v_bfe_i32 v7, v3, 4, 4
+; GFX7-NEXT:    v_bfe_i32 v8, v3, 0, 4
 ; GFX7-NEXT:    v_ashrrev_i32_e32 v9, 28, v3
 ; GFX7-NEXT:    v_bfe_i32 v3, v3, 12, 4
-; GFX7-NEXT:    v_bfe_i32 v14, v0, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v15, v0, 8, 4
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_bfe_i32 v10, v0, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v11, v0, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v12, v0, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v13, v0, 8, 4
+; GFX7-NEXT:    v_bfe_i32 v14, v0, 4, 4
+; GFX7-NEXT:    v_bfe_i32 v15, v0, 0, 4
 ; GFX7-NEXT:    v_ashrrev_i32_e32 v16, 28, v0
 ; GFX7-NEXT:    v_bfe_i32 v0, v0, 12, 4
-; GFX7-NEXT:    v_or_b32_e32 v5, v6, v5
-; GFX7-NEXT:    v_or_b32_e32 v6, v11, v10
-; GFX7-NEXT:    v_and_b32_e32 v12, v2, v14
-; GFX7-NEXT:    v_and_b32_e32 v13, v2, v15
-; GFX7-NEXT:    v_and_b32_e32 v14, v2, v16
-; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v16, 16, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 16, v6
 ; GFX7-NEXT:    v_and_b32_e32 v3, v2, v3
 ; GFX7-NEXT:    v_and_b32_e32 v9, v2, v9
+; GFX7-NEXT:    v_and_b32_e32 v13, v2, v13
+; GFX7-NEXT:    v_and_b32_e32 v14, v2, v14
+; GFX7-NEXT:    v_and_b32_e32 v15, v2, v15
 ; GFX7-NEXT:    v_and_b32_e32 v0, v2, v0
-; GFX7-NEXT:    v_and_b32_e32 v4, v2, v4
-; GFX7-NEXT:    v_and_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_and_b32_e32 v6, v2, v6
-; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v2, v2, v5
-; GFX7-NEXT:    buffer_load_ushort v5, off, s[0:3], 0
+; GFX7-NEXT:    v_and_b32_e32 v2, v2, v16
+; GFX7-NEXT:    buffer_load_ushort v16, off, s[0:3], 0
 ; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
 ; GFX7-NEXT:    v_and_b32_e32 v7, s4, v7
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
+; GFX7-NEXT:    v_and_b32_e32 v12, s4, v12
+; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
+; GFX7-NEXT:    v_and_b32_e32 v11, s4, v11
+; GFX7-NEXT:    v_and_b32_e32 v1, s4, v1
+; GFX7-NEXT:    v_and_b32_e32 v10, s4, v10
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v4, v4, v6, v5
-; GFX7-NEXT:    v_mad_u32_u24 v4, v16, v11, v4
-; GFX7-NEXT:    v_mad_u32_u24 v4, v8, v13, v4
-; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v0, v4
-; GFX7-NEXT:    v_mad_u32_u24 v0, v1, v2, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v15, v10, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v12, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v14, v0
+; GFX7-NEXT:    v_mad_u32_u24 v8, v8, v15, v16
+; GFX7-NEXT:    v_mad_u32_u24 v7, v7, v14, v8
+; GFX7-NEXT:    v_mad_u32_u24 v6, v6, v13, v7
+; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v0, v6
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v1, v10, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v2, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

 ; GFX7-NEXT:    s_add_u32 s12, s12, s3
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
 ; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    s_mov_b32 s10, 0
 ; GFX7-NEXT:    s_mov_b32 s11, s3
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
-; GFX7-NEXT:    buffer_load_dword v4, v[0:1], s[8:11], 0 addr64
+; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
-; GFX7-NEXT:    s_movk_i32 s4, 0xff
 ; GFX7-NEXT:    s_mov_b32 s2, -1
-; GFX7-NEXT:    v_mov_b32_e32 v2, 0xff
+; GFX7-NEXT:    buffer_load_ubyte v1, off, s[0:3], 0
+; GFX7-NEXT:    s_movk_i32 s4, 0xff
+; GFX7-NEXT:    v_mov_b32_e32 v3, 0xff
 ; GFX7-NEXT:    s_mov_b32 s5, 0xffff
-; GFX7-NEXT:    v_mov_b32_e32 v3, 0xffff
+; GFX7-NEXT:    v_mov_b32_e32 v4, 0xffff
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
+; GFX7-NEXT:    s_waitcnt vmcnt(2)
+; GFX7-NEXT:    v_bfe_i32 v6, v2, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v7, v2, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v8, v2, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v9, v2, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v10, v2, 8, 4
+; GFX7-NEXT:    v_bfe_i32 v11, v2, 4, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v5, 28, v2
+; GFX7-NEXT:    v_bfe_i32 v2, v2, 0, 4
+; GFX7-NEXT:    v_lshlrev_b32_e32 v12, 8, v5
+; GFX7-NEXT:    v_and_b32_e32 v6, s4, v6
+; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 8, v7
+; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
+; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 8, v9
+; GFX7-NEXT:    v_and_b32_e32 v10, s4, v10
+; GFX7-NEXT:    v_lshlrev_b32_e32 v11, 8, v11
+; GFX7-NEXT:    v_and_b32_e32 v2, v3, v2
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_bfe_i32 v6, v4, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v7, v4, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v8, v4, 12, 4
-; GFX7-NEXT:    v_bfe_i32 v9, v4, 8, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 8, v6
-; GFX7-NEXT:    v_and_b32_e32 v7, s4, v7
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 8, v8
-; GFX7-NEXT:    v_and_b32_e32 v9, s4, v9
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_bfe_i32 v13, v0, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v16, v0, 12, 4
-; GFX7-NEXT:    v_or_b32_e32 v6, v7, v6
-; GFX7-NEXT:    v_or_b32_e32 v7, v9, v8
-; GFX7-NEXT:    v_and_b32_e32 v9, v2, v13
-; GFX7-NEXT:    v_lshlrev_b32_e32 v13, 8, v16
-; GFX7-NEXT:    buffer_load_ubyte v16, off, s[0:3], 0
-; GFX7-NEXT:    v_bfe_i32 v5, v4, 24, 4
-; GFX7-NEXT:    v_bfe_i32 v10, v4, 4, 4
-; GFX7-NEXT:    v_ashrrev_i32_e32 v1, 28, v4
-; GFX7-NEXT:    v_bfe_i32 v4, v4, 0, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v11, 8, v1
-; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
-; GFX7-NEXT:    v_ashrrev_i32_e32 v12, 28, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 8, v10
-; GFX7-NEXT:    v_and_b32_e32 v4, v2, v4
-; GFX7-NEXT:    v_bfe_i32 v14, v0, 20, 4
-; GFX7-NEXT:    v_bfe_i32 v15, v0, 16, 4
-; GFX7-NEXT:    v_bfe_i32 v17, v0, 8, 4
-; GFX7-NEXT:    v_bfe_i32 v18, v0, 4, 4
+; GFX7-NEXT:    v_ashrrev_i32_e32 v13, 28, v0
+; GFX7-NEXT:    v_bfe_i32 v14, v0, 24, 4
+; GFX7-NEXT:    v_bfe_i32 v15, v0, 20, 4
+; GFX7-NEXT:    v_bfe_i32 v16, v0, 16, 4
+; GFX7-NEXT:    v_bfe_i32 v17, v0, 12, 4
+; GFX7-NEXT:    v_bfe_i32 v18, v0, 8, 4
+; GFX7-NEXT:    v_bfe_i32 v19, v0, 4, 4
 ; GFX7-NEXT:    v_bfe_i32 v0, v0, 0, 4
-; GFX7-NEXT:    v_or_b32_e32 v5, v5, v11
-; GFX7-NEXT:    v_or_b32_e32 v4, v4, v10
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 8, v12
-; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 8, v14
-; GFX7-NEXT:    v_and_b32_e32 v11, v2, v15
-; GFX7-NEXT:    v_and_b32_e32 v14, v2, v17
-; GFX7-NEXT:    v_lshlrev_b32_e32 v15, 8, v18
-; GFX7-NEXT:    v_and_b32_e32 v0, v2, v0
-; GFX7-NEXT:    v_or_b32_e32 v0, v0, v15
-; GFX7-NEXT:    v_or_b32_e32 v8, v9, v8
-; GFX7-NEXT:    v_or_b32_e32 v9, v11, v10
-; GFX7-NEXT:    v_lshlrev_b32_e32 v5, 16, v5
-; GFX7-NEXT:    v_and_b32_e32 v6, s5, v6
-; GFX7-NEXT:    v_or_b32_e32 v10, v14, v13
-; GFX7-NEXT:    v_or_b32_e32 v5, v6, v5
-; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v8
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 16, v10
+; GFX7-NEXT:    v_or_b32_e32 v6, v6, v12
+; GFX7-NEXT:    v_or_b32_e32 v9, v10, v9
+; GFX7-NEXT:    v_or_b32_e32 v10, v2, v11
+; GFX7-NEXT:    v_or_b32_e32 v7, v8, v7
+; GFX7-NEXT:    v_and_b32_e32 v12, v3, v14
+; GFX7-NEXT:    v_lshlrev_b32_e32 v11, 8, v13
+; GFX7-NEXT:    v_lshlrev_b32_e32 v14, 8, v15
+; GFX7-NEXT:    v_and_b32_e32 v15, v3, v16
+; GFX7-NEXT:    v_lshlrev_b32_e32 v16, 8, v17
+; GFX7-NEXT:    v_and_b32_e32 v17, v3, v18
+; GFX7-NEXT:    v_lshlrev_b32_e32 v18, 8, v19
 ; GFX7-NEXT:    v_and_b32_e32 v0, v3, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v7, 16, v7
-; GFX7-NEXT:    v_and_b32_e32 v4, s5, v4
-; GFX7-NEXT:    v_or_b32_e32 v0, v0, v8
-; GFX7-NEXT:    v_or_b32_e32 v4, v4, v7
-; GFX7-NEXT:    v_and_b32_e32 v7, v3, v9
-; GFX7-NEXT:    v_or_b32_e32 v3, v7, v6
-; GFX7-NEXT:    v_and_b32_e32 v7, v2, v4
-; GFX7-NEXT:    v_and_b32_e32 v13, v2, v0
-; GFX7-NEXT:    v_bfe_u32 v8, v4, 8, 8
-; GFX7-NEXT:    v_bfe_u32 v14, v0, 8, 8
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 24, v4
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 24, v0
-; GFX7-NEXT:    v_bfe_u32 v4, v4, 16, 8
-; GFX7-NEXT:    v_bfe_u32 v0, v0, 16, 8
-; GFX7-NEXT:    v_and_b32_e32 v1, v2, v1
-; GFX7-NEXT:    v_and_b32_e32 v12, v2, v12
-; GFX7-NEXT:    v_and_b32_e32 v9, v2, v5
-; GFX7-NEXT:    v_and_b32_e32 v2, v2, v3
-; GFX7-NEXT:    v_bfe_u32 v10, v5, 8, 8
-; GFX7-NEXT:    v_bfe_u32 v15, v3, 8, 8
-; GFX7-NEXT:    v_bfe_u32 v5, v5, 16, 8
-; GFX7-NEXT:    v_bfe_u32 v3, v3, 16, 8
+; GFX7-NEXT:    v_and_b32_e32 v5, v3, v5
+; GFX7-NEXT:    v_and_b32_e32 v3, v3, v13
+; GFX7-NEXT:    v_or_b32_e32 v11, v12, v11
+; GFX7-NEXT:    v_or_b32_e32 v12, v15, v14
+; GFX7-NEXT:    v_or_b32_e32 v14, v0, v18
+; GFX7-NEXT:    v_lshlrev_b32_e32 v6, 16, v6
+; GFX7-NEXT:    v_and_b32_e32 v7, s5, v7
+; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 16, v9
+; GFX7-NEXT:    v_and_b32_e32 v10, s5, v10
+; GFX7-NEXT:    v_or_b32_e32 v13, v17, v16
+; GFX7-NEXT:    v_or_b32_e32 v6, v7, v6
+; GFX7-NEXT:    v_or_b32_e32 v7, v10, v9
+; GFX7-NEXT:    v_and_b32_e32 v10, v4, v12
+; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 16, v11
+; GFX7-NEXT:    v_lshlrev_b32_e32 v11, 16, v13
+; GFX7-NEXT:    v_and_b32_e32 v4, v4, v14
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v11
+; GFX7-NEXT:    v_bfe_u32 v11, v7, 8, 8
+; GFX7-NEXT:    v_bfe_u32 v14, v4, 8, 8
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v7, v7, v13, v16
-; GFX7-NEXT:    v_mad_u32_u24 v7, v8, v14, v7
-; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v0, v7
-; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v11, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v9, v2, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v10, v15, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
+; GFX7-NEXT:    v_or_b32_e32 v9, v10, v9
+; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 24, v7
+; GFX7-NEXT:    v_lshrrev_b32_e32 v13, 24, v4
+; GFX7-NEXT:    v_bfe_u32 v7, v7, 16, 8
+; GFX7-NEXT:    v_bfe_u32 v4, v4, 16, 8
+; GFX7-NEXT:    v_mad_u32_u24 v0, v11, v14, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v4, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v10, v13, v0
+; GFX7-NEXT:    v_bfe_u32 v12, v6, 8, 8
+; GFX7-NEXT:    v_bfe_u32 v16, v9, 8, 8
+; GFX7-NEXT:    v_mad_u32_u24 v0, v8, v15, v0
+; GFX7-NEXT:    v_bfe_u32 v6, v6, 16, 8
+; GFX7-NEXT:    v_bfe_u32 v9, v9, 16, 8
+; GFX7-NEXT:    v_mad_u32_u24 v0, v12, v16, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v9, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v3, v0
-; GFX7-NEXT:    v_mad_u32_u24 v0, v1, v12, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: idot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

llvm/test/CodeGen/AMDGPU/idot8u.ll

 ; GFX7-NEXT:    s_mov_b32 s12, SCRATCH_RSRC_DWORD0
 ; GFX7-NEXT:    s_mov_b32 s13, SCRATCH_RSRC_DWORD1
 ; GFX7-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x9
 ; GFX7-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0xd
 ; GFX7-NEXT:    s_mov_b32 s14, -1
 ; GFX7-NEXT:    s_mov_b32 s15, 0xe8f000
 ; GFX7-NEXT:    s_add_u32 s12, s12, s3
 ; GFX7-NEXT:    s_mov_b32 s3, 0xf000
-; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    s_mov_b32 s10, 0
 ; GFX7-NEXT:    s_mov_b32 s11, s3
+; GFX7-NEXT:    s_waitcnt lgkmcnt(0)
+; GFX7-NEXT:    s_mov_b64 s[8:9], s[4:5]
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
 ; GFX7-NEXT:    buffer_load_ushort v16, off, s[0:3], 0
-; GFX7-NEXT:    s_mov_b32 s4, 0xf0000
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_bfe_u32 v7, v2, 20, 4
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 12, v2
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v1, 28, v2
 ; GFX7-NEXT:    v_bfe_u32 v3, v2, 24, 4
-; GFX7-NEXT:    v_bfe_u32 v4, v2, 12, 4
-; GFX7-NEXT:    v_bfe_u32 v5, v2, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v6, 15, v2
-; GFX7-NEXT:    v_alignbit_b32 v2, v7, v2, 16
-; GFX7-NEXT:    v_and_b32_e32 v7, s4, v8
+; GFX7-NEXT:    v_bfe_u32 v4, v2, 20, 4
+; GFX7-NEXT:    v_bfe_u32 v5, v2, 16, 4
+; GFX7-NEXT:    v_bfe_u32 v6, v2, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v7, v2, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v8, v2, 4, 4
+; GFX7-NEXT:    v_and_b32_e32 v2, 15, v2
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_lshlrev_b32_e32 v8, 12, v0
-; GFX7-NEXT:    v_or_b32_e32 v6, v6, v7
-; GFX7-NEXT:    v_and_b32_e32 v7, s4, v8
-; GFX7-NEXT:    v_and_b32_e32 v13, 15, v0
-; GFX7-NEXT:    v_or_b32_e32 v7, v13, v7
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 16, v6
-; GFX7-NEXT:    v_lshrrev_b32_e32 v13, 16, v7
-; GFX7-NEXT:    v_and_b32_e32 v6, 15, v6
-; GFX7-NEXT:    v_and_b32_e32 v7, 15, v7
-; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v6, v6, v7, v16
-; GFX7-NEXT:    v_bfe_u32 v12, v0, 8, 4
-; GFX7-NEXT:    v_mad_u32_u24 v6, v8, v13, v6
-; GFX7-NEXT:    v_bfe_u32 v14, v0, 20, 4
 ; GFX7-NEXT:    v_lshrrev_b32_e32 v9, 28, v0
 ; GFX7-NEXT:    v_bfe_u32 v10, v0, 24, 4
-; GFX7-NEXT:    v_bfe_u32 v11, v0, 12, 4
-; GFX7-NEXT:    v_alignbit_b32 v0, v14, v0, 16
-; GFX7-NEXT:    v_mad_u32_u24 v5, v5, v12, v6
-; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 16, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v14, 16, v0
-; GFX7-NEXT:    v_and_b32_e32 v2, 15, v2
+; GFX7-NEXT:    v_bfe_u32 v11, v0, 20, 4
+; GFX7-NEXT:    v_bfe_u32 v12, v0, 16, 4
+; GFX7-NEXT:    v_bfe_u32 v13, v0, 12, 4
+; GFX7-NEXT:    v_bfe_u32 v14, v0, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v15, v0, 4, 4
 ; GFX7-NEXT:    v_and_b32_e32 v0, 15, v0
-; GFX7-NEXT:    v_mad_u32_u24 v4, v4, v11, v5
-; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v4
-; GFX7-NEXT:    v_mad_u32_u24 v0, v15, v14, v0
+; GFX7-NEXT:    s_waitcnt vmcnt(0)
+; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v16
+; GFX7-NEXT:    v_mad_u32_u24 v0, v8, v15, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v14, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v6, v13, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v12, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v4, v11, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v3, v10, v0
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v1, v9, v0
 ; GFX7-NEXT:    buffer_store_short v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc16_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24

 ; GFX7-NEXT:    s_mov_b32 s10, 0
 ; GFX7-NEXT:    s_mov_b32 s11, s3
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 2, v0
 ; GFX7-NEXT:    v_mov_b32_e32 v1, 0
 ; GFX7-NEXT:    buffer_load_dword v2, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b64 s[8:9], s[6:7]
 ; GFX7-NEXT:    buffer_load_dword v0, v[0:1], s[8:11], 0 addr64
 ; GFX7-NEXT:    s_mov_b32 s2, -1
-; GFX7-NEXT:    buffer_load_ubyte v16, off, s[0:3], 0
+; GFX7-NEXT:    buffer_load_ubyte v1, off, s[0:3], 0
 ; GFX7-NEXT:    s_movk_i32 s4, 0xf00
 ; GFX7-NEXT:    v_mov_b32_e32 v3, 0xf00
 ; GFX7-NEXT:    s_movk_i32 s5, 0xf0f
 ; GFX7-NEXT:    s_addc_u32 s13, s13, 0
 ; GFX7-NEXT:    s_waitcnt vmcnt(2)
-; GFX7-NEXT:    v_lshrrev_b32_e32 v6, 28, v2
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 4, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 4, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v8, 12, v2
-; GFX7-NEXT:    v_bfe_u32 v1, v2, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v5, 15, v2
-; GFX7-NEXT:    v_bfe_u32 v7, v2, 16, 4
-; GFX7-NEXT:    v_alignbit_b32 v2, v6, v2, 24
-; GFX7-NEXT:    v_and_b32_e32 v6, s4, v9
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 28, v2
+; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 4, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 4, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v9, 12, v2
+; GFX7-NEXT:    v_bfe_u32 v4, v2, 8, 4
+; GFX7-NEXT:    v_and_b32_e32 v6, 15, v2
+; GFX7-NEXT:    v_bfe_u32 v8, v2, 16, 4
+; GFX7-NEXT:    v_alignbit_b32 v2, v7, v2, 24
+; GFX7-NEXT:    v_and_b32_e32 v7, s4, v10
 ; GFX7-NEXT:    s_waitcnt vmcnt(1)
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 4, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v9, 4, v0
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v4
-; GFX7-NEXT:    v_or_b32_e32 v5, v5, v6
-; GFX7-NEXT:    v_and_b32_e32 v6, v3, v9
-; GFX7-NEXT:    v_bfe_u32 v10, v0, 8, 4
-; GFX7-NEXT:    v_and_b32_e32 v3, v3, v11
-; GFX7-NEXT:    v_or_b32_e32 v1, v1, v4
-; GFX7-NEXT:    v_or_b32_e32 v3, v10, v3
-; GFX7-NEXT:    v_and_b32_e32 v12, 15, v0
-; GFX7-NEXT:    v_lshlrev_b32_e32 v1, 16, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v13, 28, v0
-; GFX7-NEXT:    v_or_b32_e32 v6, v12, v6
-; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GFX7-NEXT:    v_lshlrev_b32_e32 v10, 4, v0
+; GFX7-NEXT:    v_lshrrev_b32_e32 v12, 4, v0
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v5
+; GFX7-NEXT:    v_and_b32_e32 v10, v3, v10
+; GFX7-NEXT:    v_bfe_u32 v11, v0, 8, 4
+; GFX7-NEXT:    v_and_b32_e32 v3, v3, v12
+; GFX7-NEXT:    v_and_b32_e32 v13, 15, v0
+; GFX7-NEXT:    v_or_b32_e32 v4, v4, v5
+; GFX7-NEXT:    v_or_b32_e32 v3, v11, v3
+; GFX7-NEXT:    v_lshrrev_b32_e32 v14, 28, v0
+; GFX7-NEXT:    v_and_b32_e32 v9, s4, v9
 ; GFX7-NEXT:    v_and_b32_e32 v2, s5, v2
-; GFX7-NEXT:    v_or_b32_e32 v3, v6, v3
-; GFX7-NEXT:    v_lshrrev_b32_e32 v15, 12, v0
-; GFX7-NEXT:    v_bfe_u32 v14, v0, 16, 4
-; GFX7-NEXT:    v_alignbit_b32 v0, v13, v0, 24
-; GFX7-NEXT:    v_and_b32_e32 v8, s4, v8
-; GFX7-NEXT:    v_or_b32_e32 v1, v5, v1
-; GFX7-NEXT:    v_and_b32_e32 v4, s4, v15
+; GFX7-NEXT:    v_lshrrev_b32_e32 v16, 12, v0
+; GFX7-NEXT:    v_bfe_u32 v15, v0, 16, 4
+; GFX7-NEXT:    v_alignbit_b32 v0, v14, v0, 24
+; GFX7-NEXT:    v_or_b32_e32 v7, v6, v7
+; GFX7-NEXT:    v_lshlrev_b32_e32 v4, 16, v4
+; GFX7-NEXT:    v_or_b32_e32 v10, v13, v10
+; GFX7-NEXT:    v_lshlrev_b32_e32 v3, 16, v3
+; GFX7-NEXT:    v_and_b32_e32 v5, s4, v16
 ; GFX7-NEXT:    v_and_b32_e32 v0, s5, v0
-; GFX7-NEXT:    v_or_b32_e32 v7, v7, v8
+; GFX7-NEXT:    v_or_b32_e32 v3, v10, v3
+; GFX7-NEXT:    v_or_b32_e32 v9, v8, v9
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v2, 16, v2
-; GFX7-NEXT:    v_and_b32_e32 v6, 15, v1
-; GFX7-NEXT:    v_and_b32_e32 v12, 15, v3
-; GFX7-NEXT:    v_or_b32_e32 v2, v7, v2
-; GFX7-NEXT:    v_or_b32_e32 v4, v14, v4
+; GFX7-NEXT:    v_or_b32_e32 v4, v7, v4
+; GFX7-NEXT:    v_or_b32_e32 v2, v9, v2
+; GFX7-NEXT:    v_or_b32_e32 v5, v15, v5
 ; GFX7-NEXT:    v_lshlrev_b32_e32 v0, 16, v0
-; GFX7-NEXT:    v_bfe_u32 v7, v1, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v13, v3, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v9, v4, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v14, v3, 8, 4
 ; GFX7-NEXT:    s_waitcnt vmcnt(0)
-; GFX7-NEXT:    v_mad_u32_u24 v6, v6, v12, v16
-; GFX7-NEXT:    v_or_b32_e32 v0, v4, v0
-; GFX7-NEXT:    v_lshrrev_b32_e32 v4, 24, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v10, 24, v3
-; GFX7-NEXT:    v_bfe_u32 v1, v1, 16, 4
+; GFX7-NEXT:    v_mad_u32_u24 v1, v6, v13, v1
+; GFX7-NEXT:    v_or_b32_e32 v0, v5, v0
+; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v4
+; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 24, v3
+; GFX7-NEXT:    v_bfe_u32 v4, v4, 16, 4
 ; GFX7-NEXT:    v_bfe_u32 v3, v3, 16, 4
-; GFX7-NEXT:    v_mad_u32_u24 v6, v7, v13, v6
-; GFX7-NEXT:    v_mad_u32_u24 v1, v1, v3, v6
-; GFX7-NEXT:    v_and_b32_e32 v8, 15, v2
-; GFX7-NEXT:    v_and_b32_e32 v14, 15, v0
-; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v10, v1
-; GFX7-NEXT:    v_bfe_u32 v9, v2, 8, 4
-; GFX7-NEXT:    v_bfe_u32 v15, v0, 8, 4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v14, v1
-; GFX7-NEXT:    v_lshrrev_b32_e32 v5, 24, v2
-; GFX7-NEXT:    v_lshrrev_b32_e32 v11, 24, v0
+; GFX7-NEXT:    v_mad_u32_u24 v1, v9, v14, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v4, v3, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v5, v11, v1
+; GFX7-NEXT:    v_bfe_u32 v10, v2, 8, 4
+; GFX7-NEXT:    v_bfe_u32 v16, v0, 8, 4
+; GFX7-NEXT:    v_mad_u32_u24 v1, v8, v15, v1
+; GFX7-NEXT:    v_lshrrev_b32_e32 v7, 24, v2
+; GFX7-NEXT:    v_lshrrev_b32_e32 v12, 24, v0
 ; GFX7-NEXT:    v_bfe_u32 v2, v2, 16, 4
 ; GFX7-NEXT:    v_bfe_u32 v0, v0, 16, 4
-; GFX7-NEXT:    v_mad_u32_u24 v1, v9, v15, v1
+; GFX7-NEXT:    v_mad_u32_u24 v1, v10, v16, v1
 ; GFX7-NEXT:    v_mad_u32_u24 v0, v2, v0, v1
-; GFX7-NEXT:    v_mad_u32_u24 v0, v5, v11, v0
+; GFX7-NEXT:    v_mad_u32_u24 v0, v7, v12, v0
 ; GFX7-NEXT:    buffer_store_byte v0, off, s[0:3], 0
 ; GFX7-NEXT:    s_endpgm
 ;
 ; GFX8-LABEL: udot8_acc8_vecMul:
 ; GFX8:       ; %bb.0: ; %entry
 ; GFX8-NEXT:    s_load_dwordx4 s[4:7], s[0:1], 0x24
 ; GFX8-NEXT:    s_load_dwordx2 s[0:1], s[0:1], 0x34
 ; GFX8-NEXT:    v_lshlrev_b32_e32 v2, 2, v0

llvm/test/CodeGen/RISCV/srem-lkk.ll

 ;
 ; RV64IM-LABEL: fold_srem_positive_odd:
 ; RV64IM:       # %bb.0:
 ; RV64IM-NEXT:    sext.w a1, a0
 ; RV64IM-NEXT:    lui a2, 706409
 ; RV64IM-NEXT:    addiw a2, a2, 389
 ; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    srli a1, a1, 32
+; RV64IM-NEXT:    add a2, a1, a0
+; RV64IM-NEXT:    srliw a2, a2, 31
 ; RV64IM-NEXT:    addw a1, a1, a0
-; RV64IM-NEXT:    srliw a2, a1, 31
 ; RV64IM-NEXT:    srli a1, a1, 6
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    addi a2, zero, 95
 ; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    subw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = srem i32 %x, 95
   ret i32 %1

 ;
 ; RV64IM-LABEL: combine_srem_sdiv:
 ; RV64IM:       # %bb.0:
 ; RV64IM-NEXT:    sext.w a1, a0
 ; RV64IM-NEXT:    lui a2, 706409
 ; RV64IM-NEXT:    addiw a2, a2, 389
 ; RV64IM-NEXT:    mul a1, a1, a2
 ; RV64IM-NEXT:    srli a1, a1, 32
-; RV64IM-NEXT:    addw a2, a1, a0
-; RV64IM-NEXT:    srliw a2, a2, 31
 ; RV64IM-NEXT:    add a1, a1, a0
+; RV64IM-NEXT:    srliw a2, a1, 31
 ; RV64IM-NEXT:    sraiw a1, a1, 6
 ; RV64IM-NEXT:    add a1, a1, a2
 ; RV64IM-NEXT:    addi a2, zero, 95
 ; RV64IM-NEXT:    mul a2, a1, a2
 ; RV64IM-NEXT:    sub a0, a0, a2
 ; RV64IM-NEXT:    addw a0, a0, a1
 ; RV64IM-NEXT:    ret
   %1 = srem i32 %x, 95

llvm/test/CodeGen/RISCV/srem-seteq-illegal-types.ll

 ; RV32-NEXT:    sw s1, 20(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s2, 16(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s3, 12(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s4, 8(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s5, 4(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    sw s6, 0(sp) # 4-byte Folded Spill
 ; RV32-NEXT:    mv s0, a0
 ; RV32-NEXT:    lw a0, 4(a0)
-; RV32-NEXT:    lbu a1, 12(s0)
-; RV32-NEXT:    lw a2, 8(s0)
-; RV32-NEXT:    andi a3, a0, 1
-; RV32-NEXT:    neg s2, a3
-; RV32-NEXT:    slli a3, a1, 30
-; RV32-NEXT:    srli a4, a2, 2
-; RV32-NEXT:    or s3, a4, a3
-; RV32-NEXT:    srli a1, a1, 2
-; RV32-NEXT:    andi a1, a1, 1
-; RV32-NEXT:    neg s1, a1
-; RV32-NEXT:    slli a1, a2, 31
+; RV32-NEXT:    lw a1, 8(s0)
+; RV32-NEXT:    andi a2, a0, 1
+; RV32-NEXT:    neg s2, a2
+; RV32-NEXT:    slli a2, a1, 31
 ; RV32-NEXT:    srli a0, a0, 1
-; RV32-NEXT:    or a0, a0, a1
+; RV32-NEXT:    or s3, a0, a2
+; RV32-NEXT:    lbu a2, 12(s0)
+; RV32-NEXT:    srli a0, a1, 1
+; RV32-NEXT:    andi a0, a0, 1
+; RV32-NEXT:    neg s1, a0
+; RV32-NEXT:    slli a0, a2, 30
+; RV32-NEXT:    srli a1, a1, 2
+; RV32-NEXT:    or a0, a1, a0
 ; RV32-NEXT:    lw s4, 0(s0)
-; RV32-NEXT:    srli a1, a2, 1
+; RV32-NEXT:    srli a1, a2, 2
 ; RV32-NEXT:    andi a1, a1, 1
 ; RV32-NEXT:    neg a1, a1
-; RV32-NEXT:    addi a2, zero, 7
-; RV32-NEXT:    mv a3, zero
+; RV32-NEXT:    addi a2, zero, -5
+; RV32-NEXT:    addi a3, zero, -1
 ; RV32-NEXT:    call __moddi3@plt
 ; RV32-NEXT:    mv s5, a0
 ; RV32-NEXT:    mv s6, a1
-; RV32-NEXT:    addi a2, zero, -5
-; RV32-NEXT:    addi a3, zero, -1
+; RV32-NEXT:    addi a2, zero, 7
 ; RV32-NEXT:    mv a0, s3
 ; RV32-NEXT:    mv a1, s1
+; RV32-NEXT:    mv a3, zero
 ; RV32-NEXT:    call __moddi3@plt
 ; RV32-NEXT:    mv s1, a0
 ; RV32-NEXT:    mv s3, a1
 ; RV32-NEXT:    addi a2, zero, 6
 ; RV32-NEXT:    mv a0, s4
 ; RV32-NEXT:    mv a1, s2
 ; RV32-NEXT:    mv a3, zero
 ; RV32-NEXT:    call __moddi3@plt
-; RV32-NEXT:    xori a2, s1, 2
+; RV32-NEXT:    xori a2, s1, 1
 ; RV32-NEXT:    or a2, a2, s3
 ; RV32-NEXT:    snez a2, a2
-; RV32-NEXT:    xori a3, s5, 1
+; RV32-NEXT:    xori a3, s5, 2
 ; RV32-NEXT:    or a3, a3, s6
 ; RV32-NEXT:    snez a3, a3
 ; RV32-NEXT:    or a0, a0, a1
 ; RV32-NEXT:    snez a0, a0
 ; RV32-NEXT:    neg a1, a3
 ; RV32-NEXT:    neg a4, a2
 ; RV32-NEXT:    neg a5, a0
 ; RV32-NEXT:    sw a5, 0(s0)
-; RV32-NEXT:    slli a3, a3, 1
-; RV32-NEXT:    sub a0, a0, a3
+; RV32-NEXT:    slli a2, a2, 1
+; RV32-NEXT:    sub a0, a0, a2
 ; RV32-NEXT:    sw a0, 4(s0)
-; RV32-NEXT:    slli a0, a2, 2
-; RV32-NEXT:    srli a2, a4, 30
-; RV32-NEXT:    sub a2, a2, a0
-; RV32-NEXT:    andi a2, a2, 7
-; RV32-NEXT:    sb a2, 12(s0)
-; RV32-NEXT:    srli a2, a1, 31
-; RV32-NEXT:    andi a1, a1, 1
-; RV32-NEXT:    slli a1, a1, 1
-; RV32-NEXT:    or a1, a2, a1
-; RV32-NEXT:    sub a0, a1, a0
+; RV32-NEXT:    srli a0, a4, 31
+; RV32-NEXT:    or a0, a0, a2
+; RV32-NEXT:    slli a2, a3, 2
+; RV32-NEXT:    sub a0, a0, a2
 ; RV32-NEXT:    sw a0, 8(s0)
+; RV32-NEXT:    srli a0, a1, 30
+; RV32-NEXT:    sub a0, a0, a2
+; RV32-NEXT:    andi a0, a0, 7
+; RV32-NEXT:    sb a0, 12(s0)
 ; RV32-NEXT:    lw s6, 0(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s5, 4(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s4, 8(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s3, 12(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s2, 16(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s1, 20(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw s0, 24(sp) # 4-byte Folded Reload
 ; RV32-NEXT:    lw ra, 28(sp) # 4-byte Folded Reload

 ; RV32M-NEXT:    sw s1, 20(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s2, 16(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s3, 12(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s4, 8(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s5, 4(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    sw s6, 0(sp) # 4-byte Folded Spill
 ; RV32M-NEXT:    mv s0, a0
 ; RV32M-NEXT:    lw a0, 4(a0)
-; RV32M-NEXT:    lbu a1, 12(s0)
-; RV32M-NEXT:    lw a2, 8(s0)
-; RV32M-NEXT:    andi a3, a0, 1
-; RV32M-NEXT:    neg s2, a3
-; RV32M-NEXT:    slli a3, a1, 30
-; RV32M-NEXT:    srli a4, a2, 2
-; RV32M-NEXT:    or s3, a4, a3
-; RV32M-NEXT:    srli a1, a1, 2
-; RV32M-NEXT:    andi a1, a1, 1
-; RV32M-NEXT:    neg s1, a1
-; RV32M-NEXT:    slli a1, a2, 31
+; RV32M-NEXT:    lw a1, 8(s0)
+; RV32M-NEXT:    andi a2, a0, 1
+; RV32M-NEXT:    neg s2, a2
+; RV32M-NEXT:    slli a2, a1, 31
 ; RV32M-NEXT:    srli a0, a0, 1
-; RV32M-NEXT:    or a0, a0, a1
+; RV32M-NEXT:    or s3, a0, a2
+; RV32M-NEXT:    lbu a2, 12(s0)
+; RV32M-NEXT:    srli a0, a1, 1
+; RV32M-NEXT:    andi a0, a0, 1
+; RV32M-NEXT:    neg s1, a0
+; RV32M-NEXT:    slli a0, a2, 30
+; RV32M-NEXT:    srli a1, a1, 2
+; RV32M-NEXT:    or a0, a1, a0
 ; RV32M-NEXT:    lw s4, 0(s0)
-; RV32M-NEXT:    srli a1, a2, 1
+; RV32M-NEXT:    srli a1, a2, 2
 ; RV32M-NEXT:    andi a1, a1, 1
 ; RV32M-NEXT:    neg a1, a1
-; RV32M-NEXT:    addi a2, zero, 7
-; RV32M-NEXT:    mv a3, zero
+; RV32M-NEXT:    addi a2, zero, -5
+; RV32M-NEXT:    addi a3, zero, -1
 ; RV32M-NEXT:    call __moddi3@plt
 ; RV32M-NEXT:    mv s5, a0
 ; RV32M-NEXT:    mv s6, a1
-; RV32M-NEXT:    addi a2, zero, -5
-; RV32M-NEXT:    addi a3, zero, -1
+; RV32M-NEXT:    addi a2, zero, 7
 ; RV32M-NEXT:    mv a0, s3
 ; RV32M-NEXT:    mv a1, s1
+; RV32M-NEXT:    mv a3, zero
 ; RV32M-NEXT:    call __moddi3@plt
 ; RV32M-NEXT:    mv s1, a0
 ; RV32M-NEXT:    mv s3, a1
 ; RV32M-NEXT:    addi a2, zero, 6
 ; RV32M-NEXT:    mv a0, s4
 ; RV32M-NEXT:    mv a1, s2
 ; RV32M-NEXT:    mv a3, zero
 ; RV32M-NEXT:    call __moddi3@plt
-; RV32M-NEXT:    xori a2, s1, 2
+; RV32M-NEXT:    xori a2, s1, 1
 ; RV32M-NEXT:    or a2, a2, s3
 ; RV32M-NEXT:    snez a2, a2
-; RV32M-NEXT:    xori a3, s5, 1
+; RV32M-NEXT:    xori a3, s5, 2
 ; RV32M-NEXT:    or a3, a3, s6
 ; RV32M-NEXT:    snez a3, a3
 ; RV32M-NEXT:    or a0, a0, a1
 ; RV32M-NEXT:    snez a0, a0
 ; RV32M-NEXT:    neg a1, a3
 ; RV32M-NEXT:    neg a4, a2
 ; RV32M-NEXT:    neg a5, a0
 ; RV32M-NEXT:    sw a5, 0(s0)
-; RV32M-NEXT:    slli a3, a3, 1
-; RV32M-NEXT:    sub a0, a0, a3
+; RV32M-NEXT:    slli a2, a2, 1
+; RV32M-NEXT:    sub a0, a0, a2
 ; RV32M-NEXT:    sw a0, 4(s0)
-; RV32M-NEXT:    slli a0, a2, 2
-; RV32M-NEXT:    srli a2, a4, 30
-; RV32M-NEXT:    sub a2, a2, a0
-; RV32M-NEXT:    andi a2, a2, 7
-; RV32M-NEXT:    sb a2, 12(s0)
-; RV32M-NEXT:    srli a2, a1, 31
-; RV32M-NEXT:    andi a1, a1, 1
-; RV32M-NEXT:    slli a1, a1, 1
-; RV32M-NEXT:    or a1, a2, a1
-; RV32M-NEXT:    sub a0, a1, a0
+; RV32M-NEXT:    srli a0, a4, 31
+; RV32M-NEXT:    or a0, a0, a2
+; RV32M-NEXT:    slli a2, a3, 2
+; RV32M-NEXT:    sub a0, a0, a2
 ; RV32M-NEXT:    sw a0, 8(s0)
+; RV32M-NEXT:    srli a0, a1, 30
+; RV32M-NEXT:    sub a0, a0, a2
+; RV32M-NEXT:    andi a0, a0, 7
+; RV32M-NEXT:    sb a0, 12(s0)
 ; RV32M-NEXT:    lw s6, 0(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw s5, 4(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw s4, 8(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw s3, 12(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw s2, 16(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw s1, 20(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw s0, 24(sp) # 4-byte Folded Reload
 ; RV32M-NEXT:    lw ra, 28(sp) # 4-byte Folded Reload

llvm/test/CodeGen/Thumb2/mve-vecreduce-addpred.ll

 ; CHECK-NEXT:    cset r0, eq
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    csetm r0, ne
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    cset r1, eq
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    csetm r1, ne
 ; CHECK-NEXT:    vmov q1[2], q1[0], r1, r0
-; CHECK-NEXT:    vmov q1[3], q1[1], r1, r0
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vmov r0, s2
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r1, s3
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    adds r0, r0, r3
 ; CHECK-NEXT:    adcs r1, r2
 ; CHECK-NEXT:    bx lr

 ; CHECK-NEXT:    vcmp.i32 ne, q1, zr
 ; CHECK-NEXT:    vmov.i64 q1, #0xffff
 ; CHECK-NEXT:    vmrs r0, p0
 ; CHECK-NEXT:    and r2, r0, #1
 ; CHECK-NEXT:    ubfx r1, r0, #4, #1
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r2, r1
-; CHECK-NEXT:    vmov q3[3], q3[1], r2, r1
 ; CHECK-NEXT:    vmov.u16 r1, q0[1]
 ; CHECK-NEXT:    vmov.u16 r2, q0[0]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r2, r1
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r1, s15
 ; CHECK-NEXT:    vmov r2, s13
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    orrs r1, r2
 ; CHECK-NEXT:    vmov r2, s14
 ; CHECK-NEXT:    add r2, r3
 ; CHECK-NEXT:    ubfx r3, r0, #12, #1
 ; CHECK-NEXT:    ubfx r0, r0, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r0, r3
-; CHECK-NEXT:    vmov q3[3], q3[1], r0, r3
 ; CHECK-NEXT:    vmov.u16 r0, q0[3]
 ; CHECK-NEXT:    vmov.u16 r3, q0[2]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r3, r0
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    vmov r0, s13
 ; CHECK-NEXT:    adds r2, r2, r3

 ; CHECK-NEXT:    vmov q3[3], q3[1], r3, r1
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r1, p0
 ; CHECK-NEXT:    and r0, r1, #1
 ; CHECK-NEXT:    ubfx r3, r1, #4, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r0, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r0, r3
 ; CHECK-NEXT:    vmov.u16 r0, q0[5]
 ; CHECK-NEXT:    vmov.u16 r3, q0[4]
 ; CHECK-NEXT:    vmov q3[2], q3[0], r3, r0
 ; CHECK-NEXT:    vand q3, q3, q1
 ; CHECK-NEXT:    vand q2, q3, q2
 ; CHECK-NEXT:    vmov r3, s8
 ; CHECK-NEXT:    vmov r0, s9
 ; CHECK-NEXT:    adds r2, r2, r3
 ; CHECK-NEXT:    vmov r3, s11
 ; CHECK-NEXT:    adc.w r12, r12, r0
 ; CHECK-NEXT:    vmov r0, s10
 ; CHECK-NEXT:    adds r0, r0, r2
 ; CHECK-NEXT:    adc.w r2, r12, r3
 ; CHECK-NEXT:    ubfx r3, r1, #12, #1
 ; CHECK-NEXT:    ubfx r1, r1, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r1, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r1, r3
 ; CHECK-NEXT:    vmov.u16 r1, q0[7]
 ; CHECK-NEXT:    vmov.u16 r3, q0[6]
 ; CHECK-NEXT:    vmov q0[2], q0[0], r3, r1
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r1, s1
 ; CHECK-NEXT:    adds r0, r0, r3

 ; CHECK-NEXT:    cset r0, eq
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    csetm r0, ne
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    cset r1, eq
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    csetm r1, ne
 ; CHECK-NEXT:    vmov q1[2], q1[0], r1, r0
-; CHECK-NEXT:    vmov q1[3], q1[1], r1, r0
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vmov r0, s2
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    add r0, r1
 ; CHECK-NEXT:    vmov r1, s3
 ; CHECK-NEXT:    orrs r1, r2
 ; CHECK-NEXT:    bx lr

 ; CHECK-NEXT:    vcmp.i32 ne, q1, zr
 ; CHECK-NEXT:    vmov.i64 q1, #0xff
 ; CHECK-NEXT:    vmrs r0, p0
 ; CHECK-NEXT:    and r2, r0, #1
 ; CHECK-NEXT:    ubfx r1, r0, #4, #1
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov q6[2], q6[0], r2, r1
-; CHECK-NEXT:    vmov q6[3], q6[1], r2, r1
 ; CHECK-NEXT:    vmov.u8 r1, q0[1]
 ; CHECK-NEXT:    vmov.u8 r2, q0[0]
 ; CHECK-NEXT:    vmov q7[2], q7[0], r2, r1
 ; CHECK-NEXT:    vand q7, q7, q1
 ; CHECK-NEXT:    vand q6, q7, q6
 ; CHECK-NEXT:    vmov r1, s27
 ; CHECK-NEXT:    vmov r2, s25
 ; CHECK-NEXT:    vmov r3, s24
 ; CHECK-NEXT:    orrs r1, r2
 ; CHECK-NEXT:    vmov r2, s26
 ; CHECK-NEXT:    add r2, r3
 ; CHECK-NEXT:    ubfx r3, r0, #12, #1
 ; CHECK-NEXT:    ubfx r0, r0, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    vmov q6[2], q6[0], r0, r3
-; CHECK-NEXT:    vmov q6[3], q6[1], r0, r3
 ; CHECK-NEXT:    vmov.u8 r0, q0[3]
 ; CHECK-NEXT:    vmov.u8 r3, q0[2]
 ; CHECK-NEXT:    vmov q7[2], q7[0], r3, r0
 ; CHECK-NEXT:    vand q7, q7, q1
 ; CHECK-NEXT:    vand q6, q7, q6
 ; CHECK-NEXT:    vmov r3, s24
 ; CHECK-NEXT:    vmov r0, s25
 ; CHECK-NEXT:    adds r2, r2, r3

 ; CHECK-NEXT:    vmov q6[3], q6[1], r3, r1
 ; CHECK-NEXT:    vcmp.i32 ne, q6, zr
 ; CHECK-NEXT:    vmrs r1, p0
 ; CHECK-NEXT:    and r0, r1, #1
 ; CHECK-NEXT:    ubfx r3, r1, #4, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q5[2], q5[0], r0, r3
-; CHECK-NEXT:    vmov q5[3], q5[1], r0, r3
 ; CHECK-NEXT:    vmov.u8 r0, q0[5]
 ; CHECK-NEXT:    vmov.u8 r3, q0[4]
 ; CHECK-NEXT:    vmov q6[2], q6[0], r3, r0
 ; CHECK-NEXT:    vand q6, q6, q1
 ; CHECK-NEXT:    vand q5, q6, q5
 ; CHECK-NEXT:    vmov r3, s20
 ; CHECK-NEXT:    vmov r0, s21
 ; CHECK-NEXT:    adds r2, r2, r3
 ; CHECK-NEXT:    vmov r3, s23
 ; CHECK-NEXT:    adc.w r12, r12, r0
 ; CHECK-NEXT:    vmov r0, s22
 ; CHECK-NEXT:    adds r0, r0, r2
 ; CHECK-NEXT:    adc.w r2, r12, r3
 ; CHECK-NEXT:    ubfx r3, r1, #12, #1
 ; CHECK-NEXT:    ubfx r1, r1, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov q5[2], q5[0], r1, r3
-; CHECK-NEXT:    vmov q5[3], q5[1], r1, r3
 ; CHECK-NEXT:    vmov.u8 r1, q0[7]
 ; CHECK-NEXT:    vmov.u8 r3, q0[6]
 ; CHECK-NEXT:    vmov q6[2], q6[0], r3, r1
 ; CHECK-NEXT:    vand q6, q6, q1
 ; CHECK-NEXT:    vand q5, q6, q5
 ; CHECK-NEXT:    vmov r3, s20
 ; CHECK-NEXT:    vmov r1, s21
 ; CHECK-NEXT:    adds r0, r0, r3

 ; CHECK-NEXT:    vmov q3[3], q3[1], r3, r2
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    and r0, r2, #1
 ; CHECK-NEXT:    ubfx r3, r2, #4, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r0, r3
-; CHECK-NEXT:    vmov q3[3], q3[1], r0, r3
 ; CHECK-NEXT:    vmov.u8 r0, q0[9]
 ; CHECK-NEXT:    vmov.u8 r3, q0[8]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r3, r0
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    vmov r0, s13
 ; CHECK-NEXT:    adds.w r3, r3, r12
 ; CHECK-NEXT:    adc.w r12, r1, r0
 ; CHECK-NEXT:    vmov r0, s14
 ; CHECK-NEXT:    vmov r1, s15
 ; CHECK-NEXT:    adds r0, r0, r3
 ; CHECK-NEXT:    ubfx r3, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsb.w r3, r3, #0
 ; CHECK-NEXT:    rsb.w r2, r2, #0
 ; CHECK-NEXT:    adc.w r1, r1, r12
 ; CHECK-NEXT:    vmov q3[2], q3[0], r2, r3
-; CHECK-NEXT:    vmov q3[3], q3[1], r2, r3
 ; CHECK-NEXT:    vmov.u8 r2, q0[11]
 ; CHECK-NEXT:    vmov.u8 r3, q0[10]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r3, r2
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    vmov r2, s13
 ; CHECK-NEXT:    adds r0, r0, r3

 ; CHECK-NEXT:    vmov q3[3], q3[1], r3, r2
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    and r0, r2, #1
 ; CHECK-NEXT:    ubfx r3, r2, #4, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r0, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r0, r3
 ; CHECK-NEXT:    vmov.u8 r0, q0[13]
 ; CHECK-NEXT:    vmov.u8 r3, q0[12]
 ; CHECK-NEXT:    vmov q3[2], q3[0], r3, r0
 ; CHECK-NEXT:    vand q3, q3, q1
 ; CHECK-NEXT:    vand q2, q3, q2
 ; CHECK-NEXT:    vmov r3, s8
 ; CHECK-NEXT:    vmov r0, s9
 ; CHECK-NEXT:    adds.w r3, r3, r12
 ; CHECK-NEXT:    adc.w r12, r1, r0
 ; CHECK-NEXT:    vmov r0, s10
 ; CHECK-NEXT:    vmov r1, s11
 ; CHECK-NEXT:    adds r0, r0, r3
 ; CHECK-NEXT:    ubfx r3, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsb.w r3, r3, #0
 ; CHECK-NEXT:    rsb.w r2, r2, #0
 ; CHECK-NEXT:    adc.w r1, r1, r12
 ; CHECK-NEXT:    vmov q2[2], q2[0], r2, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r2, r3
 ; CHECK-NEXT:    vmov.u8 r2, q0[15]
 ; CHECK-NEXT:    vmov.u8 r3, q0[14]
 ; CHECK-NEXT:    vmov q0[2], q0[0], r3, r2
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    adds r0, r0, r3

 ; CHECK-NEXT:    vcmp.i32 ne, q1, zr
 ; CHECK-NEXT:    vmov.i64 q1, #0xffff
 ; CHECK-NEXT:    vmrs r0, p0
 ; CHECK-NEXT:    and r2, r0, #1
 ; CHECK-NEXT:    ubfx r1, r0, #4, #1
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r2, r1
-; CHECK-NEXT:    vmov q3[3], q3[1], r2, r1
 ; CHECK-NEXT:    vmov.u16 r1, q0[1]
 ; CHECK-NEXT:    vmov.u16 r2, q0[0]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r2, r1
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r1, s15
 ; CHECK-NEXT:    vmov r2, s13
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    orrs r1, r2
 ; CHECK-NEXT:    vmov r2, s14
 ; CHECK-NEXT:    add r2, r3
 ; CHECK-NEXT:    ubfx r3, r0, #12, #1
 ; CHECK-NEXT:    ubfx r0, r0, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r0, r3
-; CHECK-NEXT:    vmov q3[3], q3[1], r0, r3
 ; CHECK-NEXT:    vmov.u16 r0, q0[3]
 ; CHECK-NEXT:    vmov.u16 r3, q0[2]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r3, r0
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    vmov r0, s13
 ; CHECK-NEXT:    adds r2, r2, r3

 ; CHECK-NEXT:    vmov q3[3], q3[1], r3, r1
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r1, p0
 ; CHECK-NEXT:    and r0, r1, #1
 ; CHECK-NEXT:    ubfx r3, r1, #4, #1
 ; CHECK-NEXT:    rsbs r0, r0, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r0, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r0, r3
 ; CHECK-NEXT:    vmov.u16 r0, q0[5]
 ; CHECK-NEXT:    vmov.u16 r3, q0[4]
 ; CHECK-NEXT:    vmov q3[2], q3[0], r3, r0
 ; CHECK-NEXT:    vand q3, q3, q1
 ; CHECK-NEXT:    vand q2, q3, q2
 ; CHECK-NEXT:    vmov r3, s8
 ; CHECK-NEXT:    vmov r0, s9
 ; CHECK-NEXT:    adds r2, r2, r3
 ; CHECK-NEXT:    vmov r3, s11
 ; CHECK-NEXT:    adc.w r12, r12, r0
 ; CHECK-NEXT:    vmov r0, s10
 ; CHECK-NEXT:    adds r0, r0, r2
 ; CHECK-NEXT:    adc.w r2, r12, r3
 ; CHECK-NEXT:    ubfx r3, r1, #12, #1
 ; CHECK-NEXT:    ubfx r1, r1, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r1, r1, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r1, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r1, r3
 ; CHECK-NEXT:    vmov.u16 r1, q0[7]
 ; CHECK-NEXT:    vmov.u16 r3, q0[6]
 ; CHECK-NEXT:    vmov q0[2], q0[0], r3, r1
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r1, s1
 ; CHECK-NEXT:    adds r0, r0, r3

 ; CHECK-NEXT:    cset r0, eq
 ; CHECK-NEXT:    cmp r0, #0
 ; CHECK-NEXT:    csetm r0, ne
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    cset r1, eq
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    csetm r1, ne
 ; CHECK-NEXT:    vmov q1[2], q1[0], r1, r0
-; CHECK-NEXT:    vmov q1[3], q1[1], r1, r0
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vmov r0, s2
 ; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    add r0, r1
 ; CHECK-NEXT:    vmov r1, s3
 ; CHECK-NEXT:    orrs r1, r2
 ; CHECK-NEXT:    bx lr

 ; CHECK-NEXT:    cset r2, eq
 ; CHECK-NEXT:    cmp r2, #0
 ; CHECK-NEXT:    csetm r2, ne
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    cset r3, eq
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    csetm r3, ne
 ; CHECK-NEXT:    vmov q1[2], q1[0], r3, r2
-; CHECK-NEXT:    vmov q1[3], q1[1], r3, r2
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vmov r2, s2
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r12, s3
 ; CHECK-NEXT:    vmov lr, s1
 ; CHECK-NEXT:    adds r2, r2, r3
 ; CHECK-NEXT:    adc.w r3, lr, r12
 ; CHECK-NEXT:    adds r0, r0, r2

 ; CHECK-NEXT:    vcmp.i32 ne, q1, zr
 ; CHECK-NEXT:    vmov.i64 q1, #0xffff
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    ubfx r3, r2, #4, #1
 ; CHECK-NEXT:    rsb.w r12, r3, #0
 ; CHECK-NEXT:    and r3, r2, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r3, r12
-; CHECK-NEXT:    vmov q3[3], q3[1], r3, r12
 ; CHECK-NEXT:    vmov.u16 r12, q0[1]
 ; CHECK-NEXT:    vmov.u16 r3, q0[0]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r3, r12
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r12, s15
 ; CHECK-NEXT:    vmov r3, s13
 ; CHECK-NEXT:    vmov lr, s14
 ; CHECK-NEXT:    orr.w r12, r12, r3
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    add lr, r3
 ; CHECK-NEXT:    ubfx r3, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r2, r3
-; CHECK-NEXT:    vmov q3[3], q3[1], r2, r3
 ; CHECK-NEXT:    vmov.u16 r2, q0[3]
 ; CHECK-NEXT:    vmov.u16 r3, q0[2]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r3, r2
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r3, s12
 ; CHECK-NEXT:    vmov r2, s13
 ; CHECK-NEXT:    adds.w lr, lr, r3

 ; CHECK-NEXT:    vmov q3[3], q3[1], r3, r2
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    and r4, r2, #1
 ; CHECK-NEXT:    ubfx r3, r2, #4, #1
 ; CHECK-NEXT:    rsbs r4, r4, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r4, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r4, r3
 ; CHECK-NEXT:    vmov.u16 r3, q0[5]
 ; CHECK-NEXT:    vmov.u16 r4, q0[4]
 ; CHECK-NEXT:    vmov q3[2], q3[0], r4, r3
 ; CHECK-NEXT:    vand q3, q3, q1
 ; CHECK-NEXT:    vand q2, q3, q2
 ; CHECK-NEXT:    vmov r4, s8
 ; CHECK-NEXT:    vmov r3, s9
 ; CHECK-NEXT:    adds.w lr, lr, r4
 ; CHECK-NEXT:    vmov r4, s10
 ; CHECK-NEXT:    adc.w r12, r12, r3
 ; CHECK-NEXT:    vmov r3, s11
 ; CHECK-NEXT:    adds.w r4, r4, lr
 ; CHECK-NEXT:    adc.w r12, r12, r3
 ; CHECK-NEXT:    ubfx r3, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r2, r3
-; CHECK-NEXT:    vmov q2[3], q2[1], r2, r3
 ; CHECK-NEXT:    vmov.u16 r2, q0[7]
 ; CHECK-NEXT:    vmov.u16 r3, q0[6]
 ; CHECK-NEXT:    vmov q0[2], q0[0], r3, r2
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:    vmov r2, s1
 ; CHECK-NEXT:    adds r3, r3, r4

 ; CHECK-NEXT:    cset r2, eq
 ; CHECK-NEXT:    cmp r2, #0
 ; CHECK-NEXT:    csetm r2, ne
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    cset r3, eq
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    csetm r3, ne
 ; CHECK-NEXT:    vmov q1[2], q1[0], r3, r2
-; CHECK-NEXT:    vmov q1[3], q1[1], r3, r2
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vmov r2, s3
 ; CHECK-NEXT:    vmov r3, s1
 ; CHECK-NEXT:    orr.w r12, r3, r2
 ; CHECK-NEXT:    vmov r3, s2
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    add r2, r3
 ; CHECK-NEXT:    adds r0, r0, r2

 ; CHECK-NEXT:    vcmp.i32 ne, q1, zr
 ; CHECK-NEXT:    vmov.i64 q1, #0xff
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    ubfx r3, r2, #4, #1
 ; CHECK-NEXT:    rsb.w r12, r3, #0
 ; CHECK-NEXT:    and r3, r2, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q6[2], q6[0], r3, r12
-; CHECK-NEXT:    vmov q6[3], q6[1], r3, r12
 ; CHECK-NEXT:    vmov.u8 r12, q0[1]
 ; CHECK-NEXT:    vmov.u8 r3, q0[0]
 ; CHECK-NEXT:    vmov q7[2], q7[0], r3, r12
 ; CHECK-NEXT:    vand q7, q7, q1
 ; CHECK-NEXT:    vand q6, q7, q6
 ; CHECK-NEXT:    vmov r12, s27
 ; CHECK-NEXT:    vmov r3, s25
 ; CHECK-NEXT:    vmov lr, s26
 ; CHECK-NEXT:    orr.w r12, r12, r3
 ; CHECK-NEXT:    vmov r3, s24
 ; CHECK-NEXT:    add lr, r3
 ; CHECK-NEXT:    ubfx r3, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    vmov q6[2], q6[0], r2, r3
-; CHECK-NEXT:    vmov q6[3], q6[1], r2, r3
 ; CHECK-NEXT:    vmov.u8 r2, q0[3]
 ; CHECK-NEXT:    vmov.u8 r3, q0[2]
 ; CHECK-NEXT:    vmov q7[2], q7[0], r3, r2
 ; CHECK-NEXT:    vand q7, q7, q1
 ; CHECK-NEXT:    vand q6, q7, q6
 ; CHECK-NEXT:    vmov r3, s24
 ; CHECK-NEXT:    vmov r2, s25
 ; CHECK-NEXT:    adds.w lr, lr, r3

 ; CHECK-NEXT:    vmov q6[3], q6[1], r3, r2
 ; CHECK-NEXT:    vcmp.i32 ne, q6, zr
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    and r4, r2, #1
 ; CHECK-NEXT:    ubfx r3, r2, #4, #1
 ; CHECK-NEXT:    rsbs r4, r4, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q5[2], q5[0], r4, r3
-; CHECK-NEXT:    vmov q5[3], q5[1], r4, r3
 ; CHECK-NEXT:    vmov.u8 r3, q0[5]
 ; CHECK-NEXT:    vmov.u8 r4, q0[4]
 ; CHECK-NEXT:    vmov q6[2], q6[0], r4, r3
 ; CHECK-NEXT:    vand q6, q6, q1
 ; CHECK-NEXT:    vand q5, q6, q5
 ; CHECK-NEXT:    vmov r4, s20
 ; CHECK-NEXT:    vmov r3, s21
 ; CHECK-NEXT:    adds.w lr, lr, r4
 ; CHECK-NEXT:    vmov r4, s22
 ; CHECK-NEXT:    adc.w r12, r12, r3
 ; CHECK-NEXT:    vmov r3, s23
 ; CHECK-NEXT:    adds.w r4, r4, lr
 ; CHECK-NEXT:    adc.w r12, r12, r3
 ; CHECK-NEXT:    ubfx r3, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    vmov q5[2], q5[0], r2, r3
-; CHECK-NEXT:    vmov q5[3], q5[1], r2, r3
 ; CHECK-NEXT:    vmov.u8 r2, q0[7]
 ; CHECK-NEXT:    vmov.u8 r3, q0[6]
 ; CHECK-NEXT:    vmov q6[2], q6[0], r3, r2
 ; CHECK-NEXT:    vand q6, q6, q1
 ; CHECK-NEXT:    vand q5, q6, q5
 ; CHECK-NEXT:    vmov r3, s20
 ; CHECK-NEXT:    vmov r2, s21
 ; CHECK-NEXT:    adds.w lr, r4, r3

 ; CHECK-NEXT:    vmov q3[3], q3[1], r4, r2
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r2, p0
 ; CHECK-NEXT:    and r3, r2, #1
 ; CHECK-NEXT:    ubfx r4, r2, #4, #1
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    rsbs r4, r4, #0
 ; CHECK-NEXT:    vmov q3[2], q3[0], r3, r4
-; CHECK-NEXT:    vmov q3[3], q3[1], r3, r4
 ; CHECK-NEXT:    vmov.u8 r3, q0[9]
 ; CHECK-NEXT:    vmov.u8 r4, q0[8]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r4, r3
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r4, s12
 ; CHECK-NEXT:    vmov r3, s13
 ; CHECK-NEXT:    adds.w r5, r12, r4
 ; CHECK-NEXT:    vmov r4, s14
 ; CHECK-NEXT:    adc.w r12, lr, r3
 ; CHECK-NEXT:    vmov r3, s15
 ; CHECK-NEXT:    adds r5, r5, r4
 ; CHECK-NEXT:    ubfx r4, r2, #12, #1
 ; CHECK-NEXT:    ubfx r2, r2, #8, #1
 ; CHECK-NEXT:    rsb.w r4, r4, #0
 ; CHECK-NEXT:    rsb.w r2, r2, #0
 ; CHECK-NEXT:    adc.w r3, r3, r12
 ; CHECK-NEXT:    vmov q3[2], q3[0], r2, r4
-; CHECK-NEXT:    vmov q3[3], q3[1], r2, r4
 ; CHECK-NEXT:    vmov.u8 r2, q0[11]
 ; CHECK-NEXT:    vmov.u8 r4, q0[10]
 ; CHECK-NEXT:    vmov q4[2], q4[0], r4, r2
 ; CHECK-NEXT:    vand q4, q4, q1
 ; CHECK-NEXT:    vand q3, q4, q3
 ; CHECK-NEXT:    vmov r4, s12
 ; CHECK-NEXT:    vmov r2, s13
 ; CHECK-NEXT:    adds r5, r5, r4

 ; CHECK-NEXT:    vmov q3[3], q3[1], r4, r3
 ; CHECK-NEXT:    vcmp.i32 ne, q3, zr
 ; CHECK-NEXT:    vmrs r3, p0
 ; CHECK-NEXT:    and r2, r3, #1
 ; CHECK-NEXT:    ubfx r4, r3, #4, #1
 ; CHECK-NEXT:    rsbs r2, r2, #0
 ; CHECK-NEXT:    rsbs r4, r4, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r2, r4
-; CHECK-NEXT:    vmov q2[3], q2[1], r2, r4
 ; CHECK-NEXT:    vmov.u8 r2, q0[13]
 ; CHECK-NEXT:    vmov.u8 r4, q0[12]
 ; CHECK-NEXT:    vmov q3[2], q3[0], r4, r2
 ; CHECK-NEXT:    vand q3, q3, q1
 ; CHECK-NEXT:    vand q2, q3, q2
 ; CHECK-NEXT:    vmov r4, s8
 ; CHECK-NEXT:    vmov r2, s9
 ; CHECK-NEXT:    adds r5, r5, r4
 ; CHECK-NEXT:    vmov r4, s11
 ; CHECK-NEXT:    adc.w r12, r12, r2
 ; CHECK-NEXT:    vmov r2, s10
 ; CHECK-NEXT:    adds r2, r2, r5
 ; CHECK-NEXT:    adc.w r5, r12, r4
 ; CHECK-NEXT:    ubfx r4, r3, #12, #1
 ; CHECK-NEXT:    ubfx r3, r3, #8, #1
 ; CHECK-NEXT:    rsbs r4, r4, #0
 ; CHECK-NEXT:    rsbs r3, r3, #0
 ; CHECK-NEXT:    vmov q2[2], q2[0], r3, r4
-; CHECK-NEXT:    vmov q2[3], q2[1], r3, r4
 ; CHECK-NEXT:    vmov.u8 r3, q0[15]
 ; CHECK-NEXT:    vmov.u8 r4, q0[14]
 ; CHECK-NEXT:    vmov q0[2], q0[0], r4, r3
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vand q0, q0, q2
 ; CHECK-NEXT:    vmov r4, s0
 ; CHECK-NEXT:    vmov r3, s1
 ; CHECK-NEXT:    adds r2, r2, r4

 ; CHECK-NEXT:    cset r2, eq
 ; CHECK-NEXT:    cmp r2, #0
 ; CHECK-NEXT:    csetm r2, ne
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    cset r3, eq
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    csetm r3, ne
 ; CHECK-NEXT:    vmov q1[2], q1[0], r3, r2
-; CHECK-NEXT:    vmov q1[3], q1[1], r3, r2
 ; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    vmov r2, s3
 ; CHECK-NEXT:    vmov r3, s1
 ; CHECK-NEXT:    orr.w r12, r3, r2
 ; CHECK-NEXT:    vmov r3, s2
 ; CHECK-NEXT:    vmov r2, s0
 ; CHECK-NEXT:    add r2, r3
 ; CHECK-NEXT:    adds r0, r0, r2

llvm/test/CodeGen/X86/movmsk-cmp.ll

 ; SSE-LABEL: movmsk_v16i8:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    pcmpeqb %xmm1, %xmm0
 ; SSE-NEXT:    pmovmskb %xmm0, %eax
 ; SSE-NEXT:    movl %eax, %ecx
 ; SSE-NEXT:    shrl $15, %ecx
 ; SSE-NEXT:    movl %eax, %edx
 ; SSE-NEXT:    shrl $8, %edx
-; SSE-NEXT:    andl $1, %edx
-; SSE-NEXT:    andl $8, %eax
 ; SSE-NEXT:    shrl $3, %eax
 ; SSE-NEXT:    xorl %edx, %eax
 ; SSE-NEXT:    andl %ecx, %eax
 ; SSE-NEXT:    # kill: def $al killed $al killed $eax
 ; SSE-NEXT:    retq
 ;
 ; AVX1OR2-LABEL: movmsk_v16i8:
 ; AVX1OR2:       # %bb.0:
 ; AVX1OR2-NEXT:    vpcmpeqb %xmm1, %xmm0, %xmm0
 ; AVX1OR2-NEXT:    vpmovmskb %xmm0, %eax
 ; AVX1OR2-NEXT:    movl %eax, %ecx
 ; AVX1OR2-NEXT:    shrl $15, %ecx
 ; AVX1OR2-NEXT:    movl %eax, %edx
 ; AVX1OR2-NEXT:    shrl $8, %edx
-; AVX1OR2-NEXT:    andl $1, %edx
-; AVX1OR2-NEXT:    andl $8, %eax
 ; AVX1OR2-NEXT:    shrl $3, %eax
 ; AVX1OR2-NEXT:    xorl %edx, %eax
 ; AVX1OR2-NEXT:    andl %ecx, %eax
 ; AVX1OR2-NEXT:    # kill: def $al killed $al killed $eax
 ; AVX1OR2-NEXT:    retq
 ;
 ; KNL-LABEL: movmsk_v16i8:
 ; KNL:       # %bb.0:

llvm/test/CodeGen/X86/pr34137.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 
 @var_3 = external dso_local global i16, align 2
 @var_13 = external dso_local global i16, align 2
 @var_212 = external dso_local global i64, align 8
 
 define void @pr34127() {
 ; CHECK-LABEL: pr34127:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movzwl {{.*}}(%rip), %eax
 ; CHECK-NEXT:    movzwl {{.*}}(%rip), %ecx
 ; CHECK-NEXT:    andl %eax, %ecx
 ; CHECK-NEXT:    movl %eax, %edx
 ; CHECK-NEXT:    andl %ecx, %edx
-; CHECK-NEXT:    movzwl %dx, %edx
 ; CHECK-NEXT:    movl %edx, -{{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    xorl %edx, %edx
 ; CHECK-NEXT:    testw %cx, %ax
 ; CHECK-NEXT:    sete %dl
 ; CHECK-NEXT:    andl %eax, %edx
 ; CHECK-NEXT:    movq %rdx, {{.*}}(%rip)
 ; CHECK-NEXT:    movw $0, (%rax)
 ; CHECK-NEXT:    retq

llvm/test/CodeGen/X86/shift-parts.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=x86_64-- < %s | FileCheck %s
 ; PR4736
 
 %0 = type { i32, i8, [35 x i8] }
 
 @g_144 = external dso_local global %0, align 8              ; <%0*> [#uses=1]
 
 define i32 @int87(i32 %uint64p_8, i1 %cond) nounwind {
 ; CHECK-LABEL: int87:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movq g_144+{{.*}}(%rip), %rax
 ; CHECK-NEXT:    movq g_144+{{.*}}(%rip), %rcx
 ; CHECK-NEXT:    movzbl %sil, %edx
+; CHECK-NEXT:    andl $1, %edx
 ; CHECK-NEXT:    shll $6, %edx
 ; CHECK-NEXT:    .p2align 4, 0x90
 ; CHECK-NEXT:  .LBB0_1: # %for.cond
 ; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    testb $64, %dl
+; CHECK-NEXT:    testb %dl, %dl
 ; CHECK-NEXT:    movq %rcx, %rsi
 ; CHECK-NEXT:    cmovneq %rax, %rsi
 ; CHECK-NEXT:    orl $0, %esi
 ; CHECK-NEXT:    je .LBB0_1
 ; CHECK-NEXT:  # %bb.2: # %if.then
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    retq
 entry: