[DAGCombine] Fold (x & ~y) | y patterns

[nikic]

Summary

This is an alternative to D59066. We introduce DAGCombine patterns for (x & ~y) | y to x | y.

This allows us to lower the expansion of vselect c, x, -1, which is (x & ~c) | (-1 & c) to just x | y, recovering the uaddsat results from D59066.

[nikic]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6927 ↗

(On Diff #189984)

An alternative and probably better solution would be to delay BIC formation until after legalization (at least vec op legalization), so there is more opportunity for generic combines to apply first. Is there a way to know in here which legalization stage we are currently in and would it be okay to check it?

llvm/test/CodeGen/AArch64/sat-add.ll

607–608

There's still some redundant bic's left here. The reason is that isBitwiseNot (and the whole isConstOrConstSplat functionality) does not support build vector constants. Handling this in isBitwiseNot is possible (and apparently only affects this test across all targets), but I think the more robust solution would be to migrate the whole isConstOrConstSplat functionality to return APInt and support implicit build vector truncation.

[nikic]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

6927 ↗

(On Diff #189984)

I've opened D59187 for this variant.

llvm/test/CodeGen/AArch64/sat-add.ll

607–608

I've opened https://bugs.llvm.org/show_bug.cgi?id=41020 to keep track of this.

[RKSimon]

These seem to be managing special cases of the bit select pattern OR(AND(X,Y),AND(Z,~Y)) - would we benefit from adding a helper function to detect these, similar to what we have for isBitwiseNot ? Although in this case we'd have to generate the -1 constant.

[nikic]

In D59174#1423934, @RKSimon wrote:

These seem to be managing special cases of the bit select pattern OR(AND(X,Y),AND(Z,~Y)) - would we benefit from adding a helper function to detect these, similar to what we have for isBitwiseNot ? Although in this case we'd have to generate the -1 constant.

Don't think a helper would be useful here, as this is specifically looking for degenerate forms of the bit select pattern where one of the ands has been folded away.

[nikic]

Remove AArch64 specific portion, which has been handled by rL356299.

[lebedev.ri]

In D59174#1423934, @RKSimon wrote:

These seem to be managing special cases of the bit select pattern OR(AND(X,Y),AND(Z,~Y)) - would we benefit from adding a helper function to detect these, similar to what we have for isBitwiseNot ? Although in this case we'd have to generate the -1 constant.

I'm not sure why/how/for what it would be useful?

This looks good, but is partial, nit added.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429

There are 4 variations of this pattern, both or and and are commutative.

[nikic]

Handle more commuted variants.

[nikic]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429

Right you are. I've added these now, which resulted in a few more changes to the variablemask tests. I was under the mistaken impression that (and X, (xor Y, -1)) is a canonicalized and-not pattern.

llvm/test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

344

Looks like this doesn't fold for the SSE1 case, because FXOR and friends are created via custom DAG combine very early on.

[lebedev.ri]

Thanks, looks ok, other than repetition..

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429

That is indeed a very subtle (and costly!) thing.
(and X, (xor Y, -1)) will be in that order (well, some deterministic order).
But what if X and Y aren't arguments, but some values (instructions)?
Then they will also affect the weights and thus ordering.

5429

Hmm, nobody likes that there is no pattern matching in back end :/
But this duplication isn't too nice too.
Can you try this instead:

// Fold all 4 commutative variations of    or (and (xor Y, -1), X), Y
// to                                      or X, Y
static SDValue foldDegradedBitselect(SDNode *N, SelectionDAG &DAG) {
  assert(N->getOpcode() == ISD::OR && "outermost node should be 'or'.");

  SDValue X, Y;
  auto outerMatcher = [N, &X, &Y](bool swap) {
    SDValue And = N->getOperand(0);
    Y = N->getOperand(1);

    if (And.getOpcode() != ISD::AND || swap)
      std::swap(And, Y);
    if (And.getOpcode() != ISD::AND)
      return false;

    auto innerMatcher = [And, Y, &X](bool swap) {
      assert(And.getOpcode() == ISD::AND);

      SDValue NotY = And.getOperand(0);
      X = And.getOperand(1);

      if (!isBitwiseNot(NotY) || swap)
        std::swap(NotY, X);
      if (!isBitwiseNot(NotY))
        return false;

      SDValue Y2 = NotY.getOperand(0);

      return Y == Y2;
    };

    return innerMatcher(/*swap=*/false) || innerMatcher(/*swap=*/true);
  };

  if (!(outerMatcher(/*swap=*/false) || outerMatcher(/*swap=*/true)))
    return SDValue();

  return DAG.getNode(ISD::OR, SDLoc(Y), Y.getValueType(), X, Y);
}

?

If that doesn't work, at least factor out these 4 duplicate if's into a function.

llvm/test/CodeGen/AArch64/unfold-masked-merge-scalar-variablemask.ll

358

(good thing i wrote all these then-pointless redundant tests i guess)

[nikic]

Add visitORCommutative() function to reduce code duplication when matching commuting patterns.

[nikic]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429

I've extracted a function that will be called with N0, N1 and N1, N0 to avoid repeating the outer pattern, and because this seems useful for potential future transforms as well.

This still repeats the and commutation, but I think the explicit repetition here is more obvious than the operand swapping variant.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5429

Ok, in this case i guess that works too.

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5401–5402

Actually, there already is this function, but it is only called in one order.
I suppose a follow-up to move contents of visitORCommutative() into visitORLike(),
and calling visitORLike() second time may be a good idea.

[nikic]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5401–5402

The existing transforms inside visitORLike wouldn't benefit from being called with commuted arguments (apart from the very first one maybe), as they check for symmetric patterns. Looking at other folds in visitOR most everything either has one constant operand (which is always on the RHS) or deals with symmetric patterns, or is a very simple fold that probably should stay at the start. I think that MatchRotate could in principle be refactored to be called twice with commuted arguments, but it would be larger change to the logic.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitOR(SDNode *N) {
 
   if (SDValue NewSel = foldBinOpIntoSelect(N))
     return NewSel;
 
   // fold (or x, c) -> c iff (x & ~c) == 0
   if (N1C && DAG.MaskedValueIsZero(N0, ~N1C->getAPIntValue()))
     return N1;
 
   if (SDValue Combined = visitORLike(N0, N1, N))
     return Combined;
 
   // Recognize halfword bswaps as (bswap + rotl 16) or (bswap + shl 16)
   if (SDValue BSwap = MatchBSwapHWord(N, N0, N1))
     return BSwap;
   if (SDValue BSwap = MatchBSwapHWordLow(N, N0, N1))
     return BSwap;
 
   // reassociate or
if (N0.getOpcode() == ISD::AND && N0.getNode()->hasOneUse() &&
     if (SDValue COR = DAG.FoldConstantArithmetic(
             ISD::OR, SDLoc(N1), VT, N1.getNode(), N0.getOperand(1).getNode())) {
       SDValue IOR = DAG.getNode(ISD::OR, SDLoc(N0), VT, N0.getOperand(0), N1);
       AddToWorklist(IOR.getNode());
       return DAG.getNode(ISD::AND, SDLoc(N), VT, COR, IOR);
     }
   }
 
+  if (N0.getOpcode() == ISD::AND) {
+    // fold (or (and X, (xor Y, -1)), Y) -> (or X, Y)
+    if (isBitwiseNot(N0.getOperand(1)) && N0.getOperand(1).getOperand(0) == N1)
+      return DAG.getNode(ISD::OR, SDLoc(N), VT, N0.getOperand(0), N1);
+
+    // fold (or (and (xor Y, -1), X), Y) -> (or X, Y)
+    if (isBitwiseNot(N0.getOperand(0)) && N0.getOperand(0).getOperand(0) == N1)
+      return DAG.getNode(ISD::OR, SDLoc(N), VT, N0.getOperand(1), N1);
+  }
+
+  if (N1.getOpcode() == ISD::AND) {
+    // fold (or X, (and Y, (xor X, -1))) -> (or X, Y)
+    if (isBitwiseNot(N1.getOperand(1)) && N1.getOperand(1).getOperand(0) == N0)
+      return DAG.getNode(ISD::OR, SDLoc(N), VT, N0, N1.getOperand(0));
+
+    // fold (or X, (and (xor X, -1), Y)) -> (or X, Y)
+    if (isBitwiseNot(N1.getOperand(0)) && N1.getOperand(0).getOperand(0) == N0)
+      return DAG.getNode(ISD::OR, SDLoc(N), VT, N0, N1.getOperand(1));
+  }
+
   // Simplify: (or (op x...), (op y...))  -> (op (or x, y))
   if (N0.getOpcode() == N1.getOpcode())
     if (SDValue V = hoistLogicOpWithSameOpcodeHands(N))
       return V;
 
   // See if this is some rotate idiom.
   if (SDNode *Rot = MatchRotate(N0, N1, SDLoc(N)))
     return SDValue(Rot, 0);

llvm/test/CodeGen/AArch64/sat-add.ll

 }
 
 define <4 x i32> @unsigned_sat_constant_v4i32_using_cmp_sum(<4 x i32> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v4i32_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %a = add <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42>
   %c = icmp ugt <4 x i32> %x, %a
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
 define <4 x i32> @unsigned_sat_constant_v4i32_using_cmp_notval(<4 x i32> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v4i32_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
 ; CHECK-NEXT:    mvni v2.4s, #42
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v2.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %a = add <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42>
   %c = icmp ugt <4 x i32> %x, <i32 -43, i32 -43, i32 -43, i32 -43>
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 

 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_cmp_sum(<2 x i64> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v2i64_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #42
 ; CHECK-NEXT:    dup v1.2d, x8
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %a = add <2 x i64> %x, <i64 42, i64 42>
   %c = icmp ugt <2 x i64> %x, %a
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_constant_v2i64_using_cmp_notval(<2 x i64> %x) {
 ; CHECK-LABEL: unsigned_sat_constant_v2i64_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mov w8, #42
 ; CHECK-NEXT:    mov x9, #-43
 ; CHECK-NEXT:    dup v1.2d, x8
 ; CHECK-NEXT:    dup v2.2d, x9
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v2.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %a = add <2 x i64> %x, <i64 42, i64 42>
   %c = icmp ugt <2 x i64> %x, <i64 -43, i64 -43>
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 

 }
 
 define <8 x i16> @unsigned_sat_variable_v8i16_using_cmp_notval(<8 x i16> %x, <8 x i16> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v8i16_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mvn v2.16b, v1.16b
 ; CHECK-NEXT:    add v1.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    cmhi v0.8h, v0.8h, v2.8h
 ; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %noty = xor <8 x i16> %y, <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>
   %a = add <8 x i16> %x, %y
   %c = icmp ugt <8 x i16> %x, %noty
   %r = select <8 x i1> %c, <8 x i16> <i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1, i16 -1>, <8 x i16> %a
   ret <8 x i16> %r
 }
 
; CHECK-NEXT:    ret
   ret <4 x i32> %r
 }
 
 define <4 x i32> @unsigned_sat_variable_v4i32_using_cmp_sum(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v4i32_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v1.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %a = add <4 x i32> %x, %y
   %c = icmp ugt <4 x i32> %x, %a
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
 
 define <4 x i32> @unsigned_sat_variable_v4i32_using_cmp_notval(<4 x i32> %x, <4 x i32> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v4i32_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mvn v2.16b, v1.16b
 ; CHECK-NEXT:    add v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmhi v0.4s, v0.4s, v2.4s
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %noty = xor <4 x i32> %y, <i32 -1, i32 -1, i32 -1, i32 -1>
   %a = add <4 x i32> %x, %y
   %c = icmp ugt <4 x i32> %x, %noty
   %r = select <4 x i1> %c, <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, <4 x i32> %a
   ret <4 x i32> %r
 }
; CHECK-NEXT:    ret
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_variable_v2i64_using_cmp_sum(<2 x i64> %x, <2 x i64> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v2i64_using_cmp_sum:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %a = add <2 x i64> %x, %y
   %c = icmp ugt <2 x i64> %x, %a
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 
 define <2 x i64> @unsigned_sat_variable_v2i64_using_cmp_notval(<2 x i64> %x, <2 x i64> %y) {
 ; CHECK-LABEL: unsigned_sat_variable_v2i64_using_cmp_notval:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    mvn v2.16b, v1.16b
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v2.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %noty = xor <2 x i64> %y, <i64 -1, i64 -1>
   %a = add <2 x i64> %x, %y
   %c = icmp ugt <2 x i64> %x, %noty
   %r = select <2 x i1> %c, <2 x i64> <i64 -1, i64 -1>, <2 x i64> %a
   ret <2 x i64> %r
 }
 

llvm/test/CodeGen/AArch64/uadd_sat_vec.ll

; CHECK-NEXT:    ret
   ret <16 x i32> %z
 }
 
 define <2 x i64> @v2i64(<2 x i64> %x, <2 x i64> %y) nounwind {
 ; CHECK-LABEL: v2i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v1.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v1.2d
-; CHECK-NEXT:    bic v1.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v1.16b
 ; CHECK-NEXT:    ret
   %z = call <2 x i64> @llvm.uadd.sat.v2i64(<2 x i64> %x, <2 x i64> %y)
   ret <2 x i64> %z
 }
 
 define <4 x i64> @v4i64(<4 x i64> %x, <4 x i64> %y) nounwind {
 ; CHECK-LABEL: v4i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v2.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    add v3.2d, v1.2d, v3.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    cmhi v1.2d, v1.2d, v3.2d
-; CHECK-NEXT:    bic v2.16b, v2.16b, v0.16b
-; CHECK-NEXT:    bic v3.16b, v3.16b, v1.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    orr v1.16b, v1.16b, v3.16b
 ; CHECK-NEXT:    ret
   %z = call <4 x i64> @llvm.uadd.sat.v4i64(<4 x i64> %x, <4 x i64> %y)
   ret <4 x i64> %z
 }
 
 define <8 x i64> @v8i64(<8 x i64> %x, <8 x i64> %y) nounwind {
 ; CHECK-LABEL: v8i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    add v4.2d, v0.2d, v4.2d
 ; CHECK-NEXT:    add v5.2d, v1.2d, v5.2d
 ; CHECK-NEXT:    add v6.2d, v2.2d, v6.2d
 ; CHECK-NEXT:    add v7.2d, v3.2d, v7.2d
 ; CHECK-NEXT:    cmhi v0.2d, v0.2d, v4.2d
 ; CHECK-NEXT:    cmhi v1.2d, v1.2d, v5.2d
 ; CHECK-NEXT:    cmhi v2.2d, v2.2d, v6.2d
 ; CHECK-NEXT:    cmhi v3.2d, v3.2d, v7.2d
-; CHECK-NEXT:    bic v4.16b, v4.16b, v0.16b
-; CHECK-NEXT:    bic v5.16b, v5.16b, v1.16b
-; CHECK-NEXT:    bic v6.16b, v6.16b, v2.16b
-; CHECK-NEXT:    bic v7.16b, v7.16b, v3.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v4.16b
 ; CHECK-NEXT:    orr v1.16b, v1.16b, v5.16b
 ; CHECK-NEXT:    orr v2.16b, v2.16b, v6.16b
 ; CHECK-NEXT:    orr v3.16b, v3.16b, v7.16b
 ; CHECK-NEXT:    ret
   %z = call <8 x i64> @llvm.uadd.sat.v8i64(<8 x i64> %x, <8 x i64> %y)
   ret <8 x i64> %z
 }

llvm/test/CodeGen/AArch64/unfold-masked-merge-scalar-variablemask.ll

 ; CHECK-NEXT:    bic w8, w2, w0
 ; CHECK-NEXT:    mvn w0, w8
 ; CHECK-NEXT:    ret
   %n0 = xor i32 %x, -1 ; %x
   %n1 = and i32 %n0, %mask
   %r = xor i32 %n1, -1
   ret i32 %r
 }
 ; This is not a canonical form. Testing for completeness only.
 define i32 @out_constant_varx_mone_invmask(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-LABEL: out_constant_varx_mone_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic w8, w0, w2
-; CHECK-NEXT:    orr w0, w8, w2
+; CHECK-NEXT:    orr w0, w0, w2
 ; CHECK-NEXT:    ret
   %notmask = xor i32 %mask, -1
   %mx = and i32 %notmask, %x
   %my = and i32 %mask, -1
   %r = or i32 %mx, %my
   ret i32 %r
 }
 ; This is not a canonical form. Testing for completeness only.
; CHECK-NEXT:    ret
   %n0 = xor i32 %x, 42 ; %x
   %n1 = and i32 %n0, %notmask
   %r = xor i32 %n1, 42
   ret i32 %r
 }
 define i32 @out_constant_mone_vary(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-LABEL: out_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic w8, w1, w2
-; CHECK-NEXT:    orr w0, w2, w8
+; CHECK-NEXT:    orr w0, w2, w1
 ; CHECK-NEXT:    ret
   %notmask = xor i32 %mask, -1
   %mx = and i32 %mask, -1
   %my = and i32 %notmask, %y
   %r = or i32 %mx, %my
   ret i32 %r
 }
 define i32 @in_constant_mone_vary(i32 %x, i32 %y, i32 %mask) {

llvm/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask-const.ll

; CHECK-NEXT:    ret
   %r = xor <4 x i32> %n1, <i32 -1, i32 -1, i32 -1, i32 -1>
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_varx_mone_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_varx_mone_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    orr v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %notmask, %x
   %my = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
; CHECK-NEXT:    ret
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v1.16b, v2.16b
-; CHECK-NEXT:    orr v0.16b, v2.16b, v0.16b
+; CHECK-NEXT:    orr v0.16b, v2.16b, v1.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v1.16b, v2.16b
-; CHECK-NEXT:    orr v0.16b, v2.16b, v0.16b
+; CHECK-NEXT:    orr v0.16b, v2.16b, v1.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.

llvm/test/CodeGen/X86/unfold-masked-merge-scalar-variablemask.ll

; CHECK-BMI-NEXT:    retq
   %n1 = and i32 %n0, %mask
   %r = xor i32 %n1, -1
   ret i32 %r
 }
 ; This is not a canonical form. Testing for completeness only.
 define i32 @out_constant_varx_mone_invmask(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-NOBMI-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-NOBMI:       # %bb.0:
-; CHECK-NOBMI-NEXT:    movl %edx, %eax
-; CHECK-NOBMI-NEXT:    notl %eax
-; CHECK-NOBMI-NEXT:    andl %edi, %eax
+; CHECK-NOBMI-NEXT:    movl %edi, %eax
 ; CHECK-NOBMI-NEXT:    orl %edx, %eax
 ; CHECK-NOBMI-NEXT:    retq
 ;
 ; CHECK-BMI-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-BMI:       # %bb.0:
-; CHECK-BMI-NEXT:    andnl %edi, %edx, %eax
+; CHECK-BMI-NEXT:    movl %edi, %eax
 ; CHECK-BMI-NEXT:    orl %edx, %eax
 ; CHECK-BMI-NEXT:    retq
   %notmask = xor i32 %mask, -1
   %mx = and i32 %notmask, %x
   %my = and i32 %mask, -1
   %r = or i32 %mx, %my
   ret i32 %r
 }
; CHECK-BMI-NEXT:    retq
   %n0 = xor i32 %x, 42 ; %x
   %n1 = and i32 %n0, %notmask
   %r = xor i32 %n1, 42
   ret i32 %r
 }
 define i32 @out_constant_mone_vary(i32 %x, i32 %y, i32 %mask) {
 ; CHECK-NOBMI-LABEL: out_constant_mone_vary:
 ; CHECK-NOBMI:       # %bb.0:
-; CHECK-NOBMI-NEXT:    movl %edx, %eax
-; CHECK-NOBMI-NEXT:    notl %eax
-; CHECK-NOBMI-NEXT:    andl %esi, %eax
+; CHECK-NOBMI-NEXT:    movl %esi, %eax
 ; CHECK-NOBMI-NEXT:    orl %edx, %eax
 ; CHECK-NOBMI-NEXT:    retq
 ;
 ; CHECK-BMI-LABEL: out_constant_mone_vary:
 ; CHECK-BMI:       # %bb.0:
-; CHECK-BMI-NEXT:    andnl %esi, %edx, %eax
+; CHECK-BMI-NEXT:    movl %esi, %eax
 ; CHECK-BMI-NEXT:    orl %edx, %eax
 ; CHECK-BMI-NEXT:    retq
   %notmask = xor i32 %mask, -1
   %mx = and i32 %mask, -1
   %my = and i32 %notmask, %y
   %r = or i32 %mx, %my
   ret i32 %r
 }

llvm/test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

; CHECK-XOP-NEXT:    retq
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @out_constant_varx_mone_invmask(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
-; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andnps (%rsi), %xmm1
-; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm0
+; CHECK-SSE1-NEXT:    orps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
-; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: out_constant_varx_mone_invmask:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rdi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovaps (%rdi), %xmm0
+; CHECK-XOP-NEXT:    vorps (%rdx), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %notmask, %x
   %my = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = or <4 x i32> %mx, %my
; CHECK-XOP-NEXT:    retq
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_mone_vary(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: out_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
-; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    orps (%rdx), %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: out_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
-; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0
-; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    orps (%rsi), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: out_constant_mone_vary:
 ; CHECK-XOP:       # %bb.0:
 ; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rsi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vorps (%rsi), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_mone_vary(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
 ; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
-; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0
-; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    orps (%rsi), %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_mone_vary:
 ; CHECK-XOP:       # %bb.0:
 ; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rsi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vorps (%rsi), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r