[DAGCombiner] improve throughput of shift+logic+shift

[spatel]

Summary

The motivating case for this is a long way from here:
https://bugs.llvm.org/show_bug.cgi?id=43146
...but I think this is where we have to start.

We need to canonicalize/optimize sequences of shift and logic to ease pattern matching for things like bswap and improve perf in general. But without the artificial limit of '!LegalTypes' (early combining), there are a lot of test diffs, and not all are good.

In the minimal tests added for this proposal, x86 should have better throughput in all cases. AArch64 is neutral because it can fold shifts into bitwise logic ops.

There are 3 shift opcodes and 3 logic opcodes for a total of 9 possible patterns:
https://rise4fun.com/Alive/VlI
https://rise4fun.com/Alive/n1m
https://rise4fun.com/Alive/1Vn

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7322–7323 ↗

(On Diff #218173)

I forgot to delete the existing check of the TLI hook when I copied it. That should be an NFC preliminary commit, but I need to confirm.

[efriedma]

llvm/test/CodeGen/AArch64/bitfield-insert.ll

269 ↗

(On Diff #218173)

Not really related to your patch, but I'm surprised we aren't matching the and+lsl to ubfiz. Something odd going on with the zext, I guess: we lower the zext to a mask, but then don't switch the "and" with the shift like instcombine would. (Simplified example: "int a(unsigned char x) { return x*8; }", vs. "int a(int x) { return ((unsigned char)x)*8; }".)

If that gets fixed, it looks like this saves one instruction, maybe? Not sure what effect that fix would have on the old code.

[lebedev.ri]

llvm/test/CodeGen/AArch64/bitfield-insert.ll

269 ↗

(On Diff #218173)

Random mention that i'm seeing similar issue in https://reviews.llvm.org/D62100#change-c0aDpsCUsZ0a
That patch is stuck due to that i think :/

[spatel]

Patch updated:
Rebased after cleanup - rL370587.

[spatel]

Patch updated:
Rebased after improving test coverage - cycle through various integer and vector types.
It shows we miss an 'LEA' opportunity on x86, but I don't think that's a big deal.

[lebedev.ri]

I think this looks ok, some comments.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7251–7252 ↗

(On Diff #218223)

I think you want to make matchFirstShift() return that X.

7266 ↗

(On Diff #218223)

It would be less confusing to C1 = Shift->getOperand(1); somewhere up there, and use C1 here.

[spatel]

Patch updated:
Try to improve readability.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7251–7252 ↗

(On Diff #218223)

That looked more awkward to me than capturing the shifted value by reference along with the shift amount.
See if the updated code is easier to read.

7266 ↗

(On Diff #218223)

I've never found a good way to juggle the naming of the various SDValue, ConstantSDNode, and underlying APInt variables. See if the update is easier to read.

[lebedev.ri]

Thanks, that is better.
LG with some thoughts.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7207–7209 ↗

(On Diff #218257)

+ the shifts must be identical

7217–7219 ↗

(On Diff #218257)

Is there some function already that has whitelist of such bitwise ops?
I suspect there are more candidates.

7240–7243 ↗

(On Diff #218257)

This looks suspicious to be honest.
Is there a case where that is so?
If not, can this be an assert until then?

7247 ↗

(On Diff #218257)

I'd think bool APInt::ult(uint64_t RHS) would be fine to use too

[spatel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7217–7219 ↗

(On Diff #218257)

Target-specific opcodes? And/or/xor are the only generic opcodes that I am aware of.

7240–7243 ↗

(On Diff #218257)

It's real - SDAG is full of surprising situations like this. I hit an assert in a regression test without this.

That was before I added the !LegalTypes constraint, so it might be hidden now, but there are few bounds down here.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7217–7219 ↗

(On Diff #218257)

Right yes, i was talking about target-specific ones.

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7240–7243 ↗

(On Diff #218257)

getShiftAmountTy gets used in most cases when a combine creates a new shift, so it is likely to have occurred before legalization.

[spatel]

llvm/test/CodeGen/AArch64/bitfield-insert.ll

269 ↗

(On Diff #218173)

Filed as:
https://bugs.llvm.org/show_bug.cgi?id=43191

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitXOR(SDNode *N) {
 
   // Simplify the expression using non-local knowledge.
   if (SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
   return SDValue();
 }
 
+/// If we have a shift-by-constant of a bitwise logic op that itself has a
+/// shift-by-constant operand with identical opcode, we may be able to convert
+/// that into 2 independent shifts followed by the logic op. This is a
+/// throughput improvement.
+static SDValue combineShiftOfShiftedLogic(SDNode *Shift, SelectionDAG &DAG) {
+  // Match a one-use bitwise logic op.
+  SDValue LogicOp = Shift->getOperand(0);
+  if (!LogicOp.hasOneUse())
+    return SDValue();
+
+  unsigned LogicOpcode = LogicOp.getOpcode();
+  if (LogicOpcode != ISD::AND && LogicOpcode != ISD::OR &&
+      LogicOpcode != ISD::XOR)
+    return SDValue();
+
+  // Find a matching one-use shift by constant.
+  unsigned ShiftOpcode = Shift->getOpcode();
+  SDValue C1 = Shift->getOperand(1);
+  ConstantSDNode *C1Node = isConstOrConstSplat(C1);
+  assert(C1Node && "Expected a shift with constant operand");
+  const APInt &C1Val = C1Node->getAPIntValue();
+  auto matchFirstShift = [&](SDValue V, SDValue &ShiftOp,
+                             const APInt *&ShiftAmtVal) {
+    if (V.getOpcode() != ShiftOpcode || !V.hasOneUse())
+      return false;
+
+    ConstantSDNode *ShiftCNode = isConstOrConstSplat(V.getOperand(1));
+    if (!ShiftCNode)
+      return false;
+
+    // Capture the shifted operand and shift amount value.
+    ShiftOp = V.getOperand(0);
+    ShiftAmtVal = &ShiftCNode->getAPIntValue();
+
+    // Shift amount types do not have to match their operand type, so check that
+    // the constants are the same width.
+    if (ShiftAmtVal->getBitWidth() != C1Val.getBitWidth())
+      return false;
+
+    // The fold is not valid if the sum of the shift values exceeds bitwidth.
+    if ((*ShiftAmtVal + C1Val).uge(V.getScalarValueSizeInBits()))
+      return false;
+
+    return true;
+  };
+
+  // Logic ops are commutative, so check each operand for a match.
+  SDValue X, Y;
+  const APInt *C0Val;
+  if (matchFirstShift(LogicOp.getOperand(0), X, C0Val))
+    Y = LogicOp.getOperand(1);
+  else if (matchFirstShift(LogicOp.getOperand(1), X, C0Val))
+    Y = LogicOp.getOperand(0);
+  else
+    return SDValue();
+
+  // shift (logic (shift X, C0), Y), C1 -> logic (shift X, C0+C1), (shift Y, C1)
+  SDLoc DL(Shift);
+  EVT VT = Shift->getValueType(0);
+  EVT ShiftAmtVT = Shift->getOperand(1).getValueType();
+  SDValue ShiftSumC = DAG.getConstant(*C0Val + C1Val, DL, ShiftAmtVT);
+  SDValue NewShift1 = DAG.getNode(ShiftOpcode, DL, VT, X, ShiftSumC);
+  SDValue NewShift2 = DAG.getNode(ShiftOpcode, DL, VT, Y, C1);
+  return DAG.getNode(LogicOpcode, DL, VT, NewShift1, NewShift2);
+}
+
 /// Handle transforms common to the three shifts, when the shift amount is a
 /// constant.
 /// We are looking for: (shift being one of shl/sra/srl)
 ///   shift (binop X, C0), C1
 /// And want to transform into:
 ///   binop (shift X, C1), (shift C0, C1)
 SDValue DAGCombiner::visitShiftByConstant(SDNode *N) {
   assert(isConstOrConstSplat(N->getOperand(1)) && "Expected constant operand");
 
   // Do not turn a 'not' into a regular xor.
   if (isBitwiseNot(N->getOperand(0)))
     return SDValue();
 
   // The inner binop must be one-use, since we want to replace it.
   SDValue LHS = N->getOperand(0);
   if (!LHS.hasOneUse() || !TLI.isDesirableToCommuteWithShift(N, Level))
     return SDValue();
 
+  // TODO: This is limited to early combining because it may reveal regressions
+  //       otherwise. But since we just checked a target hook to see if this is
+  //       desirable, that should have filtered out cases where this interferes
+  //       with some other pattern matching.
+  if (!LegalTypes)
+    if (SDValue R = combineShiftOfShiftedLogic(N, DAG))
+      return R;
+
   // We want to pull some binops through shifts, so that we have (and (shift))
   // instead of (shift (and)), likewise for add, or, xor, etc.  This sort of
   // thing happens with address calculations, so it's important to canonicalize
   // it.
   switch (LHS.getOpcode()) {
   default:
     return SDValue();
   case ISD::OR:

llvm/trunk/test/CodeGen/AArch64/bitfield-insert.ll

; CHECK-NEXT:    ret
 
   ret void
 }
 
 ; Tests when all the bits from one operand are not useful
 define i32 @test_nouseful_bits(i8 %a, i32 %b) {
 ; CHECK-LABEL: test_nouseful_bits:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov w8, w0
-; CHECK-NEXT:    bfi w8, w8, #8, #24
-; CHECK-NEXT:    mov w9, w0
-; CHECK-NEXT:    bfi w9, w8, #8, #24
-; CHECK-NEXT:    bfi w0, w9, #8, #24
-; CHECK-NEXT:    lsl w0, w0, #8
+; CHECK-NEXT:    and w8, w0, #0xff
+; CHECK-NEXT:    lsl w8, w8, #8
+; CHECK-NEXT:    mov w9, w8
+; CHECK-NEXT:    bfxil w9, w0, #0, #8
+; CHECK-NEXT:    bfi w8, w9, #16, #16
+; CHECK-NEXT:    mov w0, w8
 ; CHECK-NEXT:    ret
   %conv = zext i8 %a to i32     ;   0  0  0  A
   %shl = shl i32 %b, 8          ;   B2 B1 B0 0
   %or = or i32 %conv, %shl      ;   B2 B1 B0 A
   %shl.1 = shl i32 %or, 8       ;   B1 B0 A 0
   %or.1 = or i32 %conv, %shl.1  ;   B1 B0 A A
   %shl.2 = shl i32 %or.1, 8     ;   B0 A A 0
   %or.2 = or i32 %conv, %shl.2  ;   B0 A A A

llvm/trunk/test/CodeGen/AArch64/shift-logic.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=aarch64-- | FileCheck %s
 
 define i8 @shl_and(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: shl_and:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    and w8, w1, w0, lsl #3
-; CHECK-NEXT:    lsl w0, w8, #2
+; CHECK-NEXT:    lsl w8, w0, #5
+; CHECK-NEXT:    and w0, w8, w1, lsl #2
 ; CHECK-NEXT:    ret
   %sh0 = shl i8 %x, 3
   %r = and i8 %sh0, %y
   %sh1 = shl i8 %r, 2
   ret i8 %sh1
 }
 
 define i16 @shl_or(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: shl_or:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    orr w8, w1, w0, lsl #5
-; CHECK-NEXT:    lsl w0, w8, #7
+; CHECK-NEXT:    lsl w8, w0, #12
+; CHECK-NEXT:    orr w0, w8, w1, lsl #7
 ; CHECK-NEXT:    ret
   %sh0 = shl i16 %x, 5
   %r = or i16 %y, %sh0
   %sh1 = shl i16 %r, 7
   ret i16 %sh1
 }
 
 define i32 @shl_xor(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: shl_xor:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor w8, w1, w0, lsl #5
-; CHECK-NEXT:    lsl w0, w8, #7
+; CHECK-NEXT:    lsl w8, w0, #12
+; CHECK-NEXT:    eor w0, w8, w1, lsl #7
 ; CHECK-NEXT:    ret
   %sh0 = shl i32 %x, 5
   %r = xor i32 %sh0, %y
   %sh1 = shl i32 %r, 7
   ret i32 %sh1
 }
 
 define i64 @lshr_and(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: lshr_and:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    and x8, x1, x0, lsr #5
-; CHECK-NEXT:    lsr x0, x8, #7
+; CHECK-NEXT:    lsr x8, x0, #12
+; CHECK-NEXT:    and x0, x8, x1, lsr #7
 ; CHECK-NEXT:    ret
   %sh0 = lshr i64 %x, 5
   %r = and i64 %y, %sh0
   %sh1 = lshr i64 %r, 7
   ret i64 %sh1
 }
 
 define <4 x i32> @lshr_or(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: lshr_or:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ushr v0.4s, v0.4s, #5
+; CHECK-NEXT:    ushr v1.4s, v1.4s, #7
+; CHECK-NEXT:    ushr v0.4s, v0.4s, #12
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    ushr v0.4s, v0.4s, #7
 ; CHECK-NEXT:    ret
   %sh0 = lshr <4 x i32> %x, <i32 5, i32 5, i32 5, i32 5>
   %r = or <4 x i32> %sh0, %y
   %sh1 = lshr <4 x i32> %r, <i32 7, i32 7, i32 7, i32 7>
   ret <4 x i32> %sh1
 }
 
 define <8 x i16> @lshr_xor(<8 x i16> %x, <8 x i16> %y) nounwind {
 ; CHECK-LABEL: lshr_xor:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ushr v0.8h, v0.8h, #5
-; CHECK-NEXT:    eor v0.16b, v1.16b, v0.16b
-; CHECK-NEXT:    ushr v0.8h, v0.8h, #7
+; CHECK-NEXT:    ushr v1.8h, v1.8h, #7
+; CHECK-NEXT:    ushr v0.8h, v0.8h, #12
+; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %sh0 = lshr <8 x i16> %x, <i16 5, i16 5, i16 5, i16 5, i16 5, i16 5, i16 5, i16 5>
   %r = xor <8 x i16> %y, %sh0
   %sh1 = lshr <8 x i16> %r, <i16 7, i16 7, i16 7, i16 7, i16 7, i16 7, i16 7, i16 7>
   ret <8 x i16> %sh1
 }
 
 
 define <16 x i8> @ashr_and(<16 x i8> %x, <16 x i8> %y) nounwind {
 ; CHECK-LABEL: ashr_and:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    sshr v0.16b, v0.16b, #3
-; CHECK-NEXT:    and v0.16b, v1.16b, v0.16b
-; CHECK-NEXT:    sshr v0.16b, v0.16b, #2
+; CHECK-NEXT:    sshr v1.16b, v1.16b, #2
+; CHECK-NEXT:    sshr v0.16b, v0.16b, #5
+; CHECK-NEXT:    and v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %sh0 = ashr <16 x i8> %x, <i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3>
   %r = and <16 x i8> %y, %sh0
   %sh1 = ashr <16 x i8> %r, <i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2>
   ret <16 x i8> %sh1
 }
 
 define <2 x i64> @ashr_or(<2 x i64> %x, <2 x i64> %y) nounwind {
 ; CHECK-LABEL: ashr_or:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    sshr v0.2d, v0.2d, #5
+; CHECK-NEXT:    sshr v1.2d, v1.2d, #7
+; CHECK-NEXT:    sshr v0.2d, v0.2d, #12
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    sshr v0.2d, v0.2d, #7
 ; CHECK-NEXT:    ret
   %sh0 = ashr <2 x i64> %x, <i64 5, i64 5>
   %r = or <2 x i64> %sh0, %y
   %sh1 = ashr <2 x i64> %r, <i64 7, i64 7>
   ret <2 x i64> %sh1
 }
 
 define i32 @ashr_xor(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: ashr_xor:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor w8, w1, w0, asr #5
-; CHECK-NEXT:    asr w0, w8, #7
+; CHECK-NEXT:    asr w8, w0, #12
+; CHECK-NEXT:    eor w0, w8, w1, asr #7
 ; CHECK-NEXT:    ret
   %sh0 = ashr i32 %x, 5
   %r = xor i32 %y, %sh0
   %sh1 = ashr i32 %r, 7
   ret i32 %sh1
 }
 
 define i32 @shr_mismatch_xor(i32 %x, i32 %y) nounwind {

llvm/trunk/test/CodeGen/Thumb2/2010-03-15-AsmCCClobber.ll

 %s1 = type { %s3, i32, %s4, i8*, void (i8*, i8*)*, i8*, i32*, i32*, i32*, i32, i64, [1 x i32] }
 %s2 = type { i32 (...)**, %s4 }
 %s3 = type { %s2, i32, i32, i32*, [4 x i8], float, %s4, i8*, i8* }
 %s4 = type { %s5 }
 %s5 = type { i32 }
 
 ; Make sure the cmp is not scheduled before the InlineAsm that clobbers cc.
 ; CHECK: bl _f2
-; CHECK: cmp {{r[0-9]+}}, #0
-; CHECK-NEXT: it       eq
-; CHECK-NEXT: addeq    {{r[0-9]+}}, #1
-; CHECK-NEXT: lsls
+; CHECK: clz {{r[0-9]+}}
+; CHECK-DAG: lsrs    {{r[0-9]+}}
+; CHECK-DAG: lsls    {{r[0-9]+}}
+; CHECK-NEXT: orr.w   {{r[0-9]+}}
 ; CHECK-NEXT: InlineAsm Start
 define void @test(%s1* %this, i32 %format, i32 %w, i32 %h, i32 %levels, i32* %s, i8* %data, i32* nocapture %rowbytes, void (i8*, i8*)* %release, i8* %info) nounwind {
 entry:
   %tmp1 = getelementptr inbounds %s1, %s1* %this, i32 0, i32 0, i32 0, i32 1, i32 0, i32 0
   store volatile i32 1, i32* %tmp1, align 4
   %tmp12 = getelementptr inbounds %s1, %s1* %this, i32 0, i32 1
   store i32 %levels, i32* %tmp12, align 4
   %tmp13 = getelementptr inbounds %s1, %s1* %this, i32 0, i32 3

llvm/trunk/test/CodeGen/X86/shift-logic.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-- | FileCheck %s
 
 define i8 @shl_and(i8 %x, i8 %y) nounwind {
 ; CHECK-LABEL: shl_and:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
-; CHECK-NEXT:    leal (,%rdi,8), %eax
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    shlb $2, %sil
+; CHECK-NEXT:    shlb $5, %al
 ; CHECK-NEXT:    andb %sil, %al
-; CHECK-NEXT:    shlb $2, %al
 ; CHECK-NEXT:    # kill: def $al killed $al killed $eax
 ; CHECK-NEXT:    retq
   %sh0 = shl i8 %x, 3
   %r = and i8 %sh0, %y
   %sh1 = shl i8 %r, 2
   ret i8 %sh1
 }
 
 define i16 @shl_or(i16 %x, i16 %y) nounwind {
 ; CHECK-LABEL: shl_or:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    shll $5, %eax
+; CHECK-NEXT:    shll $7, %esi
+; CHECK-NEXT:    shll $12, %eax
 ; CHECK-NEXT:    orl %esi, %eax
-; CHECK-NEXT:    shll $7, %eax
 ; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %sh0 = shl i16 %x, 5
   %r = or i16 %y, %sh0
   %sh1 = shl i16 %r, 7
   ret i16 %sh1
 }
 
 define i32 @shl_xor(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: shl_xor:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    shll $5, %eax
+; CHECK-NEXT:    shll $7, %esi
+; CHECK-NEXT:    shll $12, %eax
 ; CHECK-NEXT:    xorl %esi, %eax
-; CHECK-NEXT:    shll $7, %eax
 ; CHECK-NEXT:    retq
   %sh0 = shl i32 %x, 5
   %r = xor i32 %sh0, %y
   %sh1 = shl i32 %r, 7
   ret i32 %sh1
 }
 
 define i64 @lshr_and(i64 %x, i64 %y) nounwind {
 ; CHECK-LABEL: lshr_and:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movq %rdi, %rax
-; CHECK-NEXT:    shrq $5, %rax
+; CHECK-NEXT:    shrq $7, %rsi
+; CHECK-NEXT:    shrq $12, %rax
 ; CHECK-NEXT:    andq %rsi, %rax
-; CHECK-NEXT:    shrq $7, %rax
 ; CHECK-NEXT:    retq
   %sh0 = lshr i64 %x, 5
   %r = and i64 %y, %sh0
   %sh1 = lshr i64 %r, 7
   ret i64 %sh1
 }
 
 define <4 x i32> @lshr_or(<4 x i32> %x, <4 x i32> %y) nounwind {
 ; CHECK-LABEL: lshr_or:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    psrld $5, %xmm0
+; CHECK-NEXT:    psrld $7, %xmm1
+; CHECK-NEXT:    psrld $12, %xmm0
 ; CHECK-NEXT:    por %xmm1, %xmm0
-; CHECK-NEXT:    psrld $7, %xmm0
 ; CHECK-NEXT:    retq
   %sh0 = lshr <4 x i32> %x, <i32 5, i32 5, i32 5, i32 5>
   %r = or <4 x i32> %sh0, %y
   %sh1 = lshr <4 x i32> %r, <i32 7, i32 7, i32 7, i32 7>
   ret <4 x i32> %sh1
 }
 
 define <8 x i16> @lshr_xor(<8 x i16> %x, <8 x i16> %y) nounwind {
 ; CHECK-LABEL: lshr_xor:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    psrlw $5, %xmm0
+; CHECK-NEXT:    psrlw $7, %xmm1
+; CHECK-NEXT:    psrlw $12, %xmm0
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
-; CHECK-NEXT:    psrlw $7, %xmm0
 ; CHECK-NEXT:    retq
   %sh0 = lshr <8 x i16> %x, <i16 5, i16 5, i16 5, i16 5, i16 5, i16 5, i16 5, i16 5>
   %r = xor <8 x i16> %y, %sh0
   %sh1 = lshr <8 x i16> %r, <i16 7, i16 7, i16 7, i16 7, i16 7, i16 7, i16 7, i16 7>
   ret <8 x i16> %sh1
 }
 
 
 define <16 x i8> @ashr_and(<16 x i8> %x, <16 x i8> %y) nounwind {
 ; CHECK-LABEL: ashr_and:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    psrlw $3, %xmm0
+; CHECK-NEXT:    psrlw $2, %xmm1
+; CHECK-NEXT:    pand {{.*}}(%rip), %xmm1
+; CHECK-NEXT:    movdqa {{.*#+}} xmm2 = [32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32]
+; CHECK-NEXT:    pxor %xmm2, %xmm1
+; CHECK-NEXT:    psubb %xmm2, %xmm1
+; CHECK-NEXT:    psrlw $5, %xmm0
 ; CHECK-NEXT:    pand {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    movdqa {{.*#+}} xmm2 = [16,16,16,16,16,16,16,16,16,16,16,16,16,16,16,16]
+; CHECK-NEXT:    movdqa {{.*#+}} xmm2 = [4,4,4,4,4,4,4,4,4,4,4,4,4,4,4,4]
 ; CHECK-NEXT:    pxor %xmm2, %xmm0
 ; CHECK-NEXT:    psubb %xmm2, %xmm0
 ; CHECK-NEXT:    pand %xmm1, %xmm0
-; CHECK-NEXT:    psrlw $2, %xmm0
-; CHECK-NEXT:    pand {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32]
-; CHECK-NEXT:    pxor %xmm1, %xmm0
-; CHECK-NEXT:    psubb %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %sh0 = ashr <16 x i8> %x, <i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3, i8 3>
   %r = and <16 x i8> %y, %sh0
   %sh1 = ashr <16 x i8> %r, <i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2, i8 2>
   ret <16 x i8> %sh1
 }
 
 define <2 x i64> @ashr_or(<2 x i64> %x, <2 x i64> %y) nounwind {
 ; CHECK-LABEL: ashr_or:
 ; CHECK:       # %bb.0:
+; CHECK-NEXT:    movdqa %xmm1, %xmm2
+; CHECK-NEXT:    psrad $7, %xmm2
+; CHECK-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[1,3,2,3]
+; CHECK-NEXT:    psrlq $7, %xmm1
+; CHECK-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
+; CHECK-NEXT:    punpckldq {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1]
 ; CHECK-NEXT:    movdqa %xmm0, %xmm2
-; CHECK-NEXT:    psrad $5, %xmm2
+; CHECK-NEXT:    psrad $12, %xmm2
 ; CHECK-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[1,3,2,3]
-; CHECK-NEXT:    psrlq $5, %xmm0
+; CHECK-NEXT:    psrlq $12, %xmm0
 ; CHECK-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; CHECK-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
 ; CHECK-NEXT:    por %xmm1, %xmm0
-; CHECK-NEXT:    movdqa %xmm0, %xmm1
-; CHECK-NEXT:    psrad $7, %xmm1
-; CHECK-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,3,2,3]
-; CHECK-NEXT:    psrlq $7, %xmm0
-; CHECK-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
-; CHECK-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
 ; CHECK-NEXT:    retq
   %sh0 = ashr <2 x i64> %x, <i64 5, i64 5>
   %r = or <2 x i64> %sh0, %y
   %sh1 = ashr <2 x i64> %r, <i64 7, i64 7>
   ret <2 x i64> %sh1
 }
 
 define i32 @ashr_xor(i32 %x, i32 %y) nounwind {
 ; CHECK-LABEL: ashr_xor:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    sarl $5, %eax
+; CHECK-NEXT:    sarl $7, %esi
+; CHECK-NEXT:    sarl $12, %eax
 ; CHECK-NEXT:    xorl %esi, %eax
-; CHECK-NEXT:    sarl $7, %eax
 ; CHECK-NEXT:    retq
   %sh0 = ashr i32 %x, 5
   %r = xor i32 %y, %sh0
   %sh1 = ashr i32 %r, 7
   ret i32 %sh1
 }
 
 define i32 @shr_mismatch_xor(i32 %x, i32 %y) nounwind {