[DAGCombine] Fold (x & ~y) | y patterns

[nikic]

Summary

This is an alternative to D59066. We introduce DAGCombine patterns for (x & ~y) | y to x | y.

This allows us to lower the expansion of vselect c, x, -1, which is (x & ~c) | (-1 & c) to just x | y, recovering the uaddsat results from D59066.

[nikic]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6927 ↗

(On Diff #189984)

An alternative and probably better solution would be to delay BIC formation until after legalization (at least vec op legalization), so there is more opportunity for generic combines to apply first. Is there a way to know in here which legalization stage we are currently in and would it be okay to check it?

llvm/test/CodeGen/AArch64/sat-add.ll

607 ↗

(On Diff #189984)

There's still some redundant bic's left here. The reason is that isBitwiseNot (and the whole isConstOrConstSplat functionality) does not support build vector constants. Handling this in isBitwiseNot is possible (and apparently only affects this test across all targets), but I think the more robust solution would be to migrate the whole isConstOrConstSplat functionality to return APInt and support implicit build vector truncation.

[nikic]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6927 ↗

(On Diff #189984)

I've opened D59187 for this variant.

llvm/test/CodeGen/AArch64/sat-add.ll

607 ↗

(On Diff #189984)

I've opened https://bugs.llvm.org/show_bug.cgi?id=41020 to keep track of this.

[RKSimon]

These seem to be managing special cases of the bit select pattern OR(AND(X,Y),AND(Z,~Y)) - would we benefit from adding a helper function to detect these, similar to what we have for isBitwiseNot ? Although in this case we'd have to generate the -1 constant.

[nikic]

In D59174#1423934, @RKSimon wrote:

These seem to be managing special cases of the bit select pattern OR(AND(X,Y),AND(Z,~Y)) - would we benefit from adding a helper function to detect these, similar to what we have for isBitwiseNot ? Although in this case we'd have to generate the -1 constant.

Don't think a helper would be useful here, as this is specifically looking for degenerate forms of the bit select pattern where one of the ands has been folded away.

[nikic]

Remove AArch64 specific portion, which has been handled by rL356299.

[lebedev.ri]

In D59174#1423934, @RKSimon wrote:

These seem to be managing special cases of the bit select pattern OR(AND(X,Y),AND(Z,~Y)) - would we benefit from adding a helper function to detect these, similar to what we have for isBitwiseNot ? Although in this case we'd have to generate the -1 constant.

I'm not sure why/how/for what it would be useful?

This looks good, but is partial, nit added.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429 ↗

(On Diff #190898)

There are 4 variations of this pattern, both or and and are commutative.

[nikic]

Handle more commuted variants.

[nikic]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429 ↗

(On Diff #190898)

Right you are. I've added these now, which resulted in a few more changes to the variablemask tests. I was under the mistaken impression that (and X, (xor Y, -1)) is a canonicalized and-not pattern.

llvm/test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

344 ↗

(On Diff #191012)

Looks like this doesn't fold for the SSE1 case, because FXOR and friends are created via custom DAG combine very early on.

[lebedev.ri]

Thanks, looks ok, other than repetition..

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429 ↗

(On Diff #190898)

That is indeed a very subtle (and costly!) thing.
(and X, (xor Y, -1)) will be in that order (well, some deterministic order).
But what if X and Y aren't arguments, but some values (instructions)?
Then they will also affect the weights and thus ordering.

5429 ↗

(On Diff #190898)

Hmm, nobody likes that there is no pattern matching in back end :/
But this duplication isn't too nice too.
Can you try this instead:

// Fold all 4 commutative variations of    or (and (xor Y, -1), X), Y
// to                                      or X, Y
static SDValue foldDegradedBitselect(SDNode *N, SelectionDAG &DAG) {
  assert(N->getOpcode() == ISD::OR && "outermost node should be 'or'.");

  SDValue X, Y;
  auto outerMatcher = [N, &X, &Y](bool swap) {
    SDValue And = N->getOperand(0);
    Y = N->getOperand(1);

    if (And.getOpcode() != ISD::AND || swap)
      std::swap(And, Y);
    if (And.getOpcode() != ISD::AND)
      return false;

    auto innerMatcher = [And, Y, &X](bool swap) {
      assert(And.getOpcode() == ISD::AND);

      SDValue NotY = And.getOperand(0);
      X = And.getOperand(1);

      if (!isBitwiseNot(NotY) || swap)
        std::swap(NotY, X);
      if (!isBitwiseNot(NotY))
        return false;

      SDValue Y2 = NotY.getOperand(0);

      return Y == Y2;
    };

    return innerMatcher(/*swap=*/false) || innerMatcher(/*swap=*/true);
  };

  if (!(outerMatcher(/*swap=*/false) || outerMatcher(/*swap=*/true)))
    return SDValue();

  return DAG.getNode(ISD::OR, SDLoc(Y), Y.getValueType(), X, Y);
}

?

If that doesn't work, at least factor out these 4 duplicate if's into a function.

llvm/test/CodeGen/AArch64/unfold-masked-merge-scalar-variablemask.ll

358 ↗

(On Diff #191012)

(good thing i wrote all these then-pointless redundant tests i guess)

[nikic]

Add visitORCommutative() function to reduce code duplication when matching commuting patterns.

[nikic]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429 ↗

(On Diff #190898)

I've extracted a function that will be called with N0, N1 and N1, N0 to avoid repeating the outer pattern, and because this seems useful for potential future transforms as well.

This still repeats the and commutation, but I think the explicit repetition here is more obvious than the operand swapping variant.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429 ↗

(On Diff #190898)

Ok, in this case i guess that works too.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5418–5419 ↗

(On Diff #191017)

Actually, there already is this function, but it is only called in one order.
I suppose a follow-up to move contents of visitORCommutative() into visitORLike(),
and calling visitORLike() second time may be a good idea.

[nikic]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5418–5419 ↗

(On Diff #191017)

The existing transforms inside visitORLike wouldn't benefit from being called with commuted arguments (apart from the very first one maybe), as they check for symmetric patterns. Looking at other folds in visitOR most everything either has one constant operand (which is always on the RHS) or deals with symmetric patterns, or is a very simple fold that probably should stay at the start. I think that MatchRotate could in principle be refactored to be called twice with commuted arguments, but it would be larger change to the logic.

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (N0.getOpcode() == ISD::AND &&
     SDValue X = DAG.getNode(ISD::OR, SDLoc(N0), VT,
                             N0.getOperand(1), N1.getOperand(1));
     return DAG.getNode(ISD::AND, DL, VT, N0.getOperand(0), X);
   }
 
   return SDValue();
 }
 
+/// OR combines for which the commuted variant will be tried as well.
+static SDValue visitORCommutative(
+    SelectionDAG &DAG, SDValue N0, SDValue N1, SDNode *N) {
+  EVT VT = N0.getValueType();
+  if (N0.getOpcode() == ISD::AND) {
+    // fold (or (and X, (xor Y, -1)), Y) -> (or X, Y)
+    if (isBitwiseNot(N0.getOperand(1)) && N0.getOperand(1).getOperand(0) == N1)
+      return DAG.getNode(ISD::OR, SDLoc(N), VT, N0.getOperand(0), N1);
+
+    // fold (or (and (xor Y, -1), X), Y) -> (or X, Y)
+    if (isBitwiseNot(N0.getOperand(0)) && N0.getOperand(0).getOperand(0) == N1)
+      return DAG.getNode(ISD::OR, SDLoc(N), VT, N0.getOperand(1), N1);
+  }
+
+  return SDValue();
+}
+
 SDValue DAGCombiner::visitOR(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   EVT VT = N1.getValueType();
 
   // x | x --> x
   if (N0 == N1)
     return N0;
if (N0.getOpcode() == ISD::AND && N0.getNode()->hasOneUse() &&
     if (SDValue COR = DAG.FoldConstantArithmetic(
             ISD::OR, SDLoc(N1), VT, N1.getNode(), N0.getOperand(1).getNode())) {
       SDValue IOR = DAG.getNode(ISD::OR, SDLoc(N0), VT, N0.getOperand(0), N1);
       AddToWorklist(IOR.getNode());
       return DAG.getNode(ISD::AND, SDLoc(N), VT, COR, IOR);
     }
   }
 
+  if (SDValue Combined = visitORCommutative(DAG, N0, N1, N))
+    return Combined;
+  if (SDValue Combined = visitORCommutative(DAG, N1, N0, N))
+    return Combined;
+
   // Simplify: (or (op x...), (op y...))  -> (op (or x, y))
   if (N0.getOpcode() == N1.getOpcode())
     if (SDValue V = hoistLogicOpWithSameOpcodeHands(N))
       return V;
 
   // See if this is some rotate idiom.
   if (SDNode *Rot = MatchRotate(N0, N1, SDLoc(N)))
     return SDValue(Rot, 0);

llvm/trunk/test/CodeGen/AArch64/sat-add.ll

 }
 
 define <4 x i32> @unsigned_sat_constant_v4i32_using_cmp_sum(<4 x i32> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v4i32_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %a = add <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42>
   %c = icmp ugt <4 x i32> %x, %a
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
 define <4 x i32> @unsigned_sat_constant_v4i32_using_cmp_notval(<4 x i32> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v4i32_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
 ; CHECK-NEXT:    mvni v2.4s, #42
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v2.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %a = add <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42>
   %c = icmp ugt <4 x i32> %x, <i32 -43, i32 -43, i32 -43, i32 -43>
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_min(<2 x i64> %x) {

 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_cmp_sum(<2 x i64> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v2i64_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #42
 ; CHECK-NEXT:    dup v1.2d, x8
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %a = add <2 x i64> %x, <i64 42, i64 42>
   %c = icmp ugt <2 x i64> %x, %a
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_cmp_notval(<2 x i64> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v2i64_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #42
 ; CHECK-NEXT:    mov x9, #-43
 ; CHECK-NEXT:    dup v1.2d, x8
 ; CHECK-NEXT:    dup v2.2d, x9
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v2.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %a = add <2 x i64> %x, <i64 42, i64 42>
   %c = icmp ugt <2 x i64> %x, <i64 -43, i64 -43>
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 
 define <16 x i8> @unsigned_sat_variable_v16i8_using_min(<16 x i8> %x, <16 x i8> %y) {
; CHECK-NEXT:    ret
   ret <4 x i32> %r
 }
 
 define <4 x i32> @unsigned_sat_variable_v4i32_using_cmp_sum(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v4i32_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %a = add <4 x i32> %x, %y
   %c = icmp ugt <4 x i32> %x, %a
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
 define <4 x i32> @unsigned_sat_variable_v4i32_using_cmp_notval(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v4i32_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mvn v2.16b, v1.16b
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v2.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %noty = xor <4 x i32> %y, <i32 -1, i32 -1, i32 -1, i32 -1>
   %a = add <4 x i32> %x, %y
   %c = icmp ugt <4 x i32> %x, %noty
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
; CHECK-NEXT:    ret
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_variable_v2i64_using_cmp_sum(<2 x i64> %x, <2 x i64> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v2i64_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %a = add <2 x i64> %x, %y
   %c = icmp ugt <2 x i64> %x, %a
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_variable_v2i64_using_cmp_notval(<2 x i64> %x, <2 x i64> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v2i64_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mvn v2.16b, v1.16b
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v2.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
-; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %noty = xor <2 x i64> %y, <i64 -1, i64 -1>
   %a = add <2 x i64> %x, %y
   %c = icmp ugt <2 x i64> %x, %noty
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-scalar-variablemask.ll

; CHECK-NEXT:    ret
   %n1 = and i32 %n0, %mask
   %r = xor i32 %n1, -1
   ret i32 %r
 }
 ; This is not a canonical form. Testing for completeness only.
 define i32 @out_constant_varx_mone_invmask(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-LABEL: out_constant_varx_mone_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic w8, w0, w2
-; CHECK-NEXT:    orr w0, w8, w2
+; CHECK-NEXT:    orr w0, w0, w2
 ; CHECK-NEXT:    ret
   %notmask = xor i32 %mask, -1
   %mx = and i32 %notmask, %x
   %my = and i32 %mask, -1
   %r = or i32 %mx, %my
   ret i32 %r
 }
 ; This is not a canonical form. Testing for completeness only.
; CHECK-NEXT:    ret
   %n0 = xor i32 %x, 42 ; %x
   %n1 = and i32 %n0, %notmask
   %r = xor i32 %n1, 42
   ret i32 %r
 }
 define i32 @out_constant_mone_vary(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-LABEL: out_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic w8, w1, w2
-; CHECK-NEXT:    orr w0, w2, w8
+; CHECK-NEXT:    orr w0, w1, w2
 ; CHECK-NEXT:    ret
   %notmask = xor i32 %mask, -1
   %mx = and i32 %mask, -1
   %my = and i32 %notmask, %y
   %r = or i32 %mx, %my
   ret i32 %r
 }
 define i32 @in_constant_mone_vary(i32 %x, i32 %y, i32 %mask) {

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask-const.ll

; CHECK-NEXT:    ret
   %r = xor <4 x i32> %n1, <i32 -1, i32 -1, i32 -1, i32 -1>
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_varx_mone_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_varx_mone_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %notmask, %x
   %my = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
; CHECK-NEXT:    ret
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v1.16b, v2.16b
-; CHECK-NEXT:    orr v0.16b, v2.16b, v0.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v1.16b, v2.16b
-; CHECK-NEXT:    orr v0.16b, v2.16b, v0.16b
+; CHECK-NEXT:    orr v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.

llvm/trunk/test/CodeGen/X86/unfold-masked-merge-scalar-variablemask.ll

; CHECK-BMI-NEXT:    retq
   %n1 = and i32 %n0, %mask
   %r = xor i32 %n1, -1
   ret i32 %r
 }
 ; This is not a canonical form. Testing for completeness only.
 define i32 @out_constant_varx_mone_invmask(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-NOBMI-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-NOBMI:       # %bb.0:
-; CHECK-NOBMI-NEXT:    movl %edx, %eax
-; CHECK-NOBMI-NEXT:    notl %eax
-; CHECK-NOBMI-NEXT:    andl %edi, %eax
+; CHECK-NOBMI-NEXT:    movl %edi, %eax
 ; CHECK-NOBMI-NEXT:    orl %edx, %eax
 ; CHECK-NOBMI-NEXT:    retq
 ;
 ; CHECK-BMI-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-BMI:       # %bb.0:
-; CHECK-BMI-NEXT:    andnl %edi, %edx, %eax
+; CHECK-BMI-NEXT:    movl %edi, %eax
 ; CHECK-BMI-NEXT:    orl %edx, %eax
 ; CHECK-BMI-NEXT:    retq
   %notmask = xor i32 %mask, -1
   %mx = and i32 %notmask, %x
   %my = and i32 %mask, -1
   %r = or i32 %mx, %my
   ret i32 %r
 }
; CHECK-BMI-NEXT:    retq
   %n0 = xor i32 %x, 42 ; %x
   %n1 = and i32 %n0, %notmask
   %r = xor i32 %n1, 42
   ret i32 %r
 }
 define i32 @out_constant_mone_vary(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-NOBMI-LABEL: out_constant_mone_vary:
 ; CHECK-NOBMI:       # %bb.0:
-; CHECK-NOBMI-NEXT:    movl %edx, %eax
-; CHECK-NOBMI-NEXT:    notl %eax
-; CHECK-NOBMI-NEXT:    andl %esi, %eax
+; CHECK-NOBMI-NEXT:    movl %esi, %eax
 ; CHECK-NOBMI-NEXT:    orl %edx, %eax
 ; CHECK-NOBMI-NEXT:    retq
 ;
 ; CHECK-BMI-LABEL: out_constant_mone_vary:
 ; CHECK-BMI:       # %bb.0:
-; CHECK-BMI-NEXT:    andnl %esi, %edx, %eax
+; CHECK-BMI-NEXT:    movl %esi, %eax
 ; CHECK-BMI-NEXT:    orl %edx, %eax
 ; CHECK-BMI-NEXT:    retq
   %notmask = xor i32 %mask, -1
   %mx = and i32 %mask, -1
   %my = and i32 %notmask, %y
   %r = or i32 %mx, %my
   ret i32 %r
 }

llvm/trunk/test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

; CHECK-XOP-NEXT:    retq
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_varx_mone_invmask(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
-; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andnps (%rsi), %xmm1
-; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm0
+; CHECK-SSE1-NEXT:    orps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
-; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rdi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovaps (%rdi), %xmm0
+; CHECK-XOP-NEXT:    vorps (%rdx), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %notmask, %x
   %my = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = or <4 x i32> %mx, %my
; CHECK-XOP-NEXT:    retq
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_mone_vary(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: out_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
-; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
-; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE1-NEXT:    orps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
-; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0
-; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm0
+; CHECK-SSE2-NEXT:    orps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: out_constant_mone_vary:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rsi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vmovaps (%rsi), %xmm0
+; CHECK-XOP-NEXT:    vorps (%rdx), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_mone_vary(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
 ; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
-; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0
-; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm0
+; CHECK-SSE2-NEXT:    orps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_mone_vary:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rsi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vmovaps (%rsi), %xmm0
+; CHECK-XOP-NEXT:    vorps (%rdx), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r