[DAGCombine][X86][AArch64] Masked merge unfolding: vector edition.

[lebedev.ri]

Summary

This appears to be the last missing piece for the masked merge pattern handling in the backend.

This is PR37104.

PR6773 will introduce an IR canonicalization that is likely bad for the end assembly.
Previously, andps+andnps / bsl would be generated. (see @out)
Now, they would no longer be generated (see @in), and we need to make sure that they are generated.

Before reviewing this, it is best to finish with the scalar part.

[lebedev.ri]

Rebased ontop of updated tests - X86 XOP test added.

[lebedev.ri]

Rebased.

[spatel]

lib/Target/AArch64/AArch64ISelLowering.h

444 ↗

(On Diff #145959)

The name of the function is overly specific if we're including bsl and instructions too. I think we should either generalize that name or add a different hook for "hasBitwiseSelect".

lib/Target/X86/X86ISelLowering.cpp

4773–4776 ↗

(On Diff #145959)

That seems odd. Does something bad happen with other types of 128-bit vectors (eg v8i16) with SSE1 only? I would've thought all element types would get mapped to andnps.

[lebedev.ri]

lib/Target/X86/X86ISelLowering.cpp

4773–4776 ↗

(On Diff #145959)

I'm pretty sure that with SSE1 we only get this vector size.
https://godbolt.org/g/o5CGQt

https://software.intel.com/sites/landingpage/IntrinsicsGuide/#techs=SSE,SSE2&text=andn
http://www.felixcloutier.com/x86/ANDNPS.html
http://www.felixcloutier.com/x86/ANDNPD.html

[lebedev.ri]

lib/Target/AArch64/AArch64ISelLowering.h

444 ↗

(On Diff #145959)

Currently for AArch64, hasAndNot() is not overriden.
We could rename it to that, and make hasAndNotCompare() call hasAndNot().

Or is this about these hooks in general?

[spatel]

lib/Target/AArch64/AArch64ISelLowering.h

444 ↗

(On Diff #145959)

It's a combination:

1. It's not accurate to say here (or in x86) that we have AndNotCompare for vector types. That hook is supposed to indicate that the logic instruction also sets flags (bics vs. bic). It's not causing any problems right now AFAICT, but stretching the truth of that logic could cause trouble some day.

2. bsl (or vpcmov for x86) are not and-not operations either, so some transform may be misled by us claiming those are and-not ops.

For AArch, we have vector bic. So here, I think we should override hasAndNot() and indicate that we have bic for scalar and bic for vector. It's really just a bonus for the masked-merge transform that we can shrink it down to bsl.

Similarly for x86, what we care about is that we can use andnps, and producing vpcmov (who knew) is a bonus. So we want to fix the override of hasAndNot() to indicate that legal vector types are supported with the right dose of SSE.

lib/Target/X86/X86ISelLowering.cpp

4773–4776 ↗

(On Diff #145959)

Hmm...the other types seem to get legalized via scalarization before we have a chance to transform them into the x86-specific logic nodes.

Probably nobody cares about pre-SSE2 codegen at this point, but Craig or Simon may have suggestions about making the tests less noisy in that respect.

[lebedev.ri]

Rework X86/AArch64 hasAndNotCompare() / hasAndNot() hooks a little.

lib/Target/AArch64/AArch64ISelLowering.h

444 ↗

(On Diff #145959)

I have reworked this a little, is this what you had in mind?

[spatel]

lib/Target/AArch64/AArch64ISelLowering.h

453–459 ↗

(On Diff #146136)

No need to change hasAndNotCompare? Seem clearer to just "return true" as we do today.

[spatel]

lib/Target/AArch64/AArch64ISelLowering.h

453–459 ↗

(On Diff #146136)

Oops...the current code isn't correct. It should be:

bool hasAndNotCompare(SDValue V) const override {
  // We can use bics for any scalar.
  return V.getValueType().isVector();
}

[spatel]

lib/Target/AArch64/AArch64ISelLowering.h

453–459 ↗

(On Diff #146136)

Argh...forgot to invert:

bool hasAndNotCompare(SDValue V) const override {
  // We can use bics for any scalar.
  return V.getValueType().isScalarInteger();
}

[lebedev.ri]

Rework AArch64 hasAndNotCompare() / hasAndNot() hooks a little once more,
move them around.

[lebedev.ri]

Rebased.

[lebedev.ri]

Ping.

[spatel]

We may be waiting for the test coverage to be resolved, but this logic change LGTM

lib/Target/AArch64/AArch64ISelLowering.h

455 ↗

(On Diff #146315)

Could just return true here? We could assert that we're not dealing with FP if that's a concern.

[lebedev.ri]

In D46528#1106477, @spatel wrote:

We may be waiting for the test coverage to be resolved, but this logic change LGTM

Thank you for the review!

Waiting on @RKSimon ...

[RKSimon]

I 'm happy with the illegal type tests to go in but the lost ANDNP combine could be a problem.

lib/Target/X86/X86ISelLowering.cpp

4776 ↗

(On Diff #146315)

Its very unlikely we'll ever add MMX support to this - drop this FIXME.

test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

65 ↗

(On Diff #146315)

This regression is rather unfortunate - anyway that we can get the X86ISD::ANDNP back?

[lebedev.ri]

test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

65 ↗

(On Diff #146315)

Uhh, great observation :/
I think this case isn't even supposed to be affected,
since we skip all not xor's...
I guess isAllOnesConstantOrAllOnesSplatConstant() is somehow misbehaving.

[lebedev.ri]

Drop fixme, fix slight regression in dependent differential.

[lebedev.ri]

lib/Target/AArch64/AArch64ISelLowering.h

455 ↗

(On Diff #146315)

I would think we should delegate to hasAndNotCompare(), since it is already there.

test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

65 ↗

(On Diff #146315)

D47156

[RKSimon]

LGTM - thanks.

[lebedev.ri]

In D46528#1106972, @RKSimon wrote:

LGTM - thanks.

Thank you for the review!

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::unfoldMaskedMerge(SDNode *N) {
   assert(N->getOpcode() == ISD::XOR);
 
   // Don't touch 'not' (i.e. where y = -1).
   if (isAllOnesConstantOrAllOnesSplatConstant(N->getOperand(1)))
     return SDValue();
 
   EVT VT = N->getValueType(0);
 
-  // FIXME
-  if (VT.isVector())
-    return SDValue();
-
   // There are 3 commutable operators in the pattern,
   // so we have to deal with 8 possible variants of the basic pattern.
   SDValue X, Y, M;
   auto matchAndXor = [&X, &Y, &M](SDValue And, unsigned XorIdx, SDValue Other) {
     if (And.getOpcode() != ISD::AND || !And.hasOneUse())
       return false;
     SDValue Xor = And.getOperand(XorIdx);
     if (Xor.getOpcode() != ISD::XOR || !Xor.hasOneUse())

llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.h

public:
   }
 
   bool isCheapToSpeculateCtlz() const override {
     return true;
   }
 
   bool isMaskAndCmp0FoldingBeneficial(const Instruction &AndI) const override;
 
-  bool hasAndNotCompare(SDValue) const override {
-    // 'bics'
-    return true;
+  bool hasAndNotCompare(SDValue V) const override {
+    // We can use bics for any scalar.
+    return V.getValueType().isScalarInteger();
+  }
+
+  bool hasAndNot(SDValue Y) const override {
+    EVT VT = Y.getValueType();
+
+    if (!VT.isVector())
+      return hasAndNotCompare(Y);
+
+    return VT.getSizeInBits() >= 64; // vector 'bic'
   }
 
   bool hasBitPreservingFPLogic(EVT VT) const override {
     // FIXME: Is this always true? It should be true for vectors at least.
     return VT == MVT::f32 || VT == MVT::f64;
   }
 
   bool supportSplitCSR(MachineFunction *MF) const override {

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

 }
 
 bool X86TargetLowering::isMaskAndCmp0FoldingBeneficial(
     const Instruction &AndI) const {
   return true;
 }
 
 bool X86TargetLowering::hasAndNotCompare(SDValue Y) const {
-  // A mask and compare against constant is ok for an 'andn' too
-  // even though the BMI instruction doesn't have an immediate form.
+  EVT VT = Y.getValueType();
+
+  if (VT.isVector())
+    return false;
 
   if (!Subtarget.hasBMI())
     return false;
 
   // There are only 32-bit and 64-bit forms for 'andn'.
-  EVT VT = Y.getValueType();
   if (VT != MVT::i32 && VT != MVT::i64)
     return false;
 
+  // A mask and compare against constant is ok for an 'andn' too
+  // even though the BMI instruction doesn't have an immediate form.
+
   return true;
 }
 
 bool X86TargetLowering::hasAndNot(SDValue Y) const {
-  // x86 can't form 'andn' with an immediate.
-  if (isa<ConstantSDNode>(Y))
+  EVT VT = Y.getValueType();
+
+  if (!VT.isVector()) // x86 can't form 'andn' with an immediate.
+    return !isa<ConstantSDNode>(Y) && hasAndNotCompare(Y);
+
+  // Vector.
+
+  if (!Subtarget.hasSSE1() || VT.getSizeInBits() < 128)
     return false;
 
-  return hasAndNotCompare(Y);
+  if (VT == MVT::v4i32)
+    return true;
+
+  return Subtarget.hasSSE2();
 }
 
 MVT X86TargetLowering::hasFastEqualityCompare(unsigned NumBits) const {
   MVT VT = MVT::getIntegerVT(NumBits);
   if (isTypeLegal(VT))
     return VT;
 
   // PMOVMSKB can handle this.

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask-const.ll

; CHECK-NEXT:    ret
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_varx_42(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_varx_42:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42> ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
; CHECK-NEXT:    ret
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_varx_42_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_varx_42_invmask:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    movi v1.4s, #42
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    bic v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bsl v2.16b, v1.16b, v0.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42> ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @out_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: out_constant_mone_vary:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    bic v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    orr v0.16b, v2.16b, v0.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %mx = and <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_mone_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_mone_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    bic v0.16b, v2.16b, v1.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bic v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    orr v0.16b, v2.16b, v0.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
; CHECK-NEXT:    ret
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_mone_vary_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_mone_vary_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mvn v0.16b, v1.16b
-; CHECK-NEXT:    bic v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    and v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    orn v0.16b, v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
; CHECK-NEXT:    ret
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_42_vary(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_42_vary:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    movi v0.4s, #42
-; CHECK-NEXT:    eor v0.16b, v1.16b, v0.16b
-; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    movi v2.4s, #42
+; CHECK-NEXT:    bsl v0.16b, v2.16b, v1.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> <i32 42, i32 42, i32 42, i32 42>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
; CHECK-NEXT:    ret
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_42_vary_invmask(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) {
 ; CHECK-LABEL: in_constant_42_vary_invmask:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    movi v0.4s, #42
-; CHECK-NEXT:    eor v0.16b, v1.16b, v0.16b
-; CHECK-NEXT:    bic v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
+; CHECK-NEXT:    movi v2.4s, #42
+; CHECK-NEXT:    bsl v0.16b, v1.16b, v2.16b
 ; CHECK-NEXT:    ret
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> <i32 42, i32 42, i32 42, i32 42>, %y ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }

llvm/trunk/test/CodeGen/AArch64/unfold-masked-merge-vector-variablemask.ll

 
 ; ============================================================================ ;
 ; 8-bit vector width
 ; ============================================================================ ;
 
 define <1 x i8> @in_v1i8(<1 x i8> %x, <1 x i8> %y, <1 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v1i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i8> %x, %y
   %n1 = and <1 x i8> %n0, %mask
   %r = xor <1 x i8> %n1, %y
   ret <1 x i8> %r
 }
 
 ; ============================================================================ ;
 ; 16-bit vector width
 ; ============================================================================ ;
 
 define <2 x i8> @in_v2i8(<2 x i8> %x, <2 x i8> %y, <2 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v2i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i8> %x, %y
   %n1 = and <2 x i8> %n0, %mask
   %r = xor <2 x i8> %n1, %y
   ret <2 x i8> %r
 }
 
 define <1 x i16> @in_v1i16(<1 x i16> %x, <1 x i16> %y, <1 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v1i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i16> %x, %y
   %n1 = and <1 x i16> %n0, %mask
   %r = xor <1 x i16> %n1, %y
   ret <1 x i16> %r
 }
 
 ; ============================================================================ ;
 ; 32-bit vector width
 ; ============================================================================ ;
 
 define <4 x i8> @in_v4i8(<4 x i8> %x, <4 x i8> %y, <4 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v4i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i8> %x, %y
   %n1 = and <4 x i8> %n0, %mask
   %r = xor <4 x i8> %n1, %y
   ret <4 x i8> %r
 }
 
 define <2 x i16> @in_v2i16(<2 x i16> %x, <2 x i16> %y, <2 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v2i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i16> %x, %y
   %n1 = and <2 x i16> %n0, %mask
   %r = xor <2 x i16> %n1, %y
   ret <2 x i16> %r
 }
 
 define <1 x i32> @in_v1i32(<1 x i32> %x, <1 x i32> %y, <1 x i32> %mask) nounwind {
 ; CHECK-LABEL: in_v1i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i32> %x, %y
   %n1 = and <1 x i32> %n0, %mask
   %r = xor <1 x i32> %n1, %y
   ret <1 x i32> %r
 }
 
 ; ============================================================================ ;
 ; 64-bit vector width
 ; ============================================================================ ;
 
 define <8 x i8> @in_v8i8(<8 x i8> %x, <8 x i8> %y, <8 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v8i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <8 x i8> %x, %y
   %n1 = and <8 x i8> %n0, %mask
   %r = xor <8 x i8> %n1, %y
   ret <8 x i8> %r
 }
 
 define <4 x i16> @in_v4i16(<4 x i16> %x, <4 x i16> %y, <4 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v4i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i16> %x, %y
   %n1 = and <4 x i16> %n0, %mask
   %r = xor <4 x i16> %n1, %y
   ret <4 x i16> %r
 }
 
 define <2 x i32> @in_v2i32(<2 x i32> %x, <2 x i32> %y, <2 x i32> %mask) nounwind {
 ; CHECK-LABEL: in_v2i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i32> %x, %y
   %n1 = and <2 x i32> %n0, %mask
   %r = xor <2 x i32> %n1, %y
   ret <2 x i32> %r
 }
 
 define <1 x i64> @in_v1i64(<1 x i64> %x, <1 x i64> %y, <1 x i64> %mask) nounwind {
 ; CHECK-LABEL: in_v1i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
-; CHECK-NEXT:    and v0.8b, v0.8b, v2.8b
-; CHECK-NEXT:    eor v0.8b, v0.8b, v1.8b
+; CHECK-NEXT:    bsl v2.8b, v0.8b, v1.8b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <1 x i64> %x, %y
   %n1 = and <1 x i64> %n0, %mask
   %r = xor <1 x i64> %n1, %y
   ret <1 x i64> %r
 }
 
 ; ============================================================================ ;
 ; 128-bit vector width
 ; ============================================================================ ;
 
 define <16 x i8> @in_v16i8(<16 x i8> %x, <16 x i8> %y, <16 x i8> %mask) nounwind {
 ; CHECK-LABEL: in_v16i8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <16 x i8> %x, %y
   %n1 = and <16 x i8> %n0, %mask
   %r = xor <16 x i8> %n1, %y
   ret <16 x i8> %r
 }
 
 define <8 x i16> @in_v8i16(<8 x i16> %x, <8 x i16> %y, <8 x i16> %mask) nounwind {
 ; CHECK-LABEL: in_v8i16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <8 x i16> %x, %y
   %n1 = and <8 x i16> %n0, %mask
   %r = xor <8 x i16> %n1, %y
   ret <8 x i16> %r
 }
 
 define <4 x i32> @in_v4i32(<4 x i32> %x, <4 x i32> %y, <4 x i32> %mask) nounwind {
 ; CHECK-LABEL: in_v4i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <4 x i32> %x, %y
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }
 
 define <2 x i64> @in_v2i64(<2 x i64> %x, <2 x i64> %y, <2 x i64> %mask) nounwind {
 ; CHECK-LABEL: in_v2i64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
-; CHECK-NEXT:    and v0.16b, v0.16b, v2.16b
-; CHECK-NEXT:    eor v0.16b, v0.16b, v1.16b
+; CHECK-NEXT:    bsl v2.16b, v0.16b, v1.16b
+; CHECK-NEXT:    mov v0.16b, v2.16b
 ; CHECK-NEXT:    ret
   %n0 = xor <2 x i64> %x, %y
   %n1 = and <2 x i64> %n0, %mask
   %r = xor <2 x i64> %n1, %y
   ret <2 x i64> %r
 }

llvm/trunk/test/CodeGen/X86/machine-cp.ll

 
 ; Check that copy propagation does not kill thing like:
 ; dst = copy src <-- do not kill that.
 ; ... = op1 undef dst
 ; ... = op2 dst <-- this is used here.
 define <16 x float> @foo(<16 x float> %x) {
 ; CHECK-LABEL: foo:
 ; CHECK:       ## %bb.0: ## %bb
-; CHECK-NEXT:    xorps %xmm8, %xmm8
-; CHECK-NEXT:    cvttps2dq %xmm3, %xmm9
-; CHECK-NEXT:    movaps %xmm3, %xmm13
-; CHECK-NEXT:    cmpltps %xmm8, %xmm13
-; CHECK-NEXT:    movaps {{.*#+}} xmm7 = [1,1,1,1]
-; CHECK-NEXT:    movaps %xmm13, %xmm3
-; CHECK-NEXT:    andps %xmm7, %xmm3
-; CHECK-NEXT:    cvttps2dq %xmm2, %xmm10
-; CHECK-NEXT:    movaps %xmm2, %xmm5
-; CHECK-NEXT:    cmpltps %xmm8, %xmm5
-; CHECK-NEXT:    movaps %xmm5, %xmm2
-; CHECK-NEXT:    andps %xmm7, %xmm2
-; CHECK-NEXT:    cvttps2dq %xmm1, %xmm11
+; CHECK-NEXT:    movaps %xmm3, %xmm9
+; CHECK-NEXT:    movaps %xmm2, %xmm8
+; CHECK-NEXT:    movaps %xmm1, %xmm6
+; CHECK-NEXT:    movaps %xmm0, %xmm7
+; CHECK-NEXT:    xorps %xmm0, %xmm0
+; CHECK-NEXT:    movaps %xmm3, %xmm1
+; CHECK-NEXT:    cmpltps %xmm0, %xmm1
 ; CHECK-NEXT:    movaps %xmm1, %xmm4
-; CHECK-NEXT:    cmpltps %xmm8, %xmm4
-; CHECK-NEXT:    movaps %xmm4, %xmm1
-; CHECK-NEXT:    andps %xmm7, %xmm1
-; CHECK-NEXT:    cvttps2dq %xmm0, %xmm12
-; CHECK-NEXT:    movaps %xmm0, %xmm6
-; CHECK-NEXT:    cmpltps %xmm8, %xmm6
-; CHECK-NEXT:    andps %xmm6, %xmm7
-; CHECK-NEXT:    orps {{.*}}(%rip), %xmm6
-; CHECK-NEXT:    movaps {{.*#+}} xmm14 = [5,6,7,8]
-; CHECK-NEXT:    orps %xmm14, %xmm4
-; CHECK-NEXT:    movaps {{.*#+}} xmm15 = [9,10,11,12]
-; CHECK-NEXT:    orps %xmm15, %xmm5
-; CHECK-NEXT:    movaps {{.*#+}} xmm8 = [13,14,15,16]
-; CHECK-NEXT:    orps %xmm8, %xmm13
-; CHECK-NEXT:    cvtdq2ps %xmm12, %xmm0
-; CHECK-NEXT:    cvtdq2ps %xmm11, %xmm11
-; CHECK-NEXT:    cvtdq2ps %xmm10, %xmm10
-; CHECK-NEXT:    cvtdq2ps %xmm9, %xmm9
-; CHECK-NEXT:    andps %xmm8, %xmm9
-; CHECK-NEXT:    andps %xmm15, %xmm10
-; CHECK-NEXT:    andps %xmm14, %xmm11
-; CHECK-NEXT:    andps {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    xorps %xmm7, %xmm0
+; CHECK-NEXT:    orps {{.*}}(%rip), %xmm4
+; CHECK-NEXT:    movaps %xmm4, %xmm10
+; CHECK-NEXT:    andnps %xmm1, %xmm10
+; CHECK-NEXT:    movaps %xmm2, %xmm1
+; CHECK-NEXT:    cmpltps %xmm0, %xmm1
+; CHECK-NEXT:    movaps {{.*#+}} xmm11 = [9,10,11,12]
+; CHECK-NEXT:    movaps %xmm1, %xmm3
+; CHECK-NEXT:    orps %xmm11, %xmm3
+; CHECK-NEXT:    movaps %xmm3, %xmm14
+; CHECK-NEXT:    andnps %xmm1, %xmm14
+; CHECK-NEXT:    cvttps2dq %xmm6, %xmm12
+; CHECK-NEXT:    cmpltps %xmm0, %xmm6
+; CHECK-NEXT:    movaps {{.*#+}} xmm13 = [5,6,7,8]
+; CHECK-NEXT:    movaps %xmm6, %xmm2
+; CHECK-NEXT:    orps %xmm13, %xmm2
+; CHECK-NEXT:    movaps %xmm2, %xmm5
+; CHECK-NEXT:    andnps %xmm6, %xmm5
+; CHECK-NEXT:    cvttps2dq %xmm7, %xmm6
+; CHECK-NEXT:    cmpltps %xmm0, %xmm7
+; CHECK-NEXT:    movaps {{.*#+}} xmm15 = [1,2,3,4]
+; CHECK-NEXT:    movaps %xmm7, %xmm0
+; CHECK-NEXT:    orps %xmm15, %xmm0
+; CHECK-NEXT:    movaps %xmm0, %xmm1
+; CHECK-NEXT:    andnps %xmm7, %xmm1
+; CHECK-NEXT:    andps %xmm15, %xmm0
+; CHECK-NEXT:    cvtdq2ps %xmm6, %xmm6
 ; CHECK-NEXT:    andps %xmm6, %xmm0
-; CHECK-NEXT:    xorps %xmm1, %xmm11
-; CHECK-NEXT:    andps %xmm4, %xmm11
-; CHECK-NEXT:    xorps %xmm2, %xmm10
-; CHECK-NEXT:    andps %xmm5, %xmm10
-; CHECK-NEXT:    xorps %xmm3, %xmm9
-; CHECK-NEXT:    andps %xmm13, %xmm9
-; CHECK-NEXT:    xorps %xmm7, %xmm0
-; CHECK-NEXT:    xorps %xmm11, %xmm1
-; CHECK-NEXT:    xorps %xmm10, %xmm2
-; CHECK-NEXT:    xorps %xmm9, %xmm3
+; CHECK-NEXT:    movaps {{.*#+}} xmm6 = [1,1,1,1]
+; CHECK-NEXT:    andps %xmm6, %xmm1
+; CHECK-NEXT:    orps %xmm1, %xmm0
+; CHECK-NEXT:    andps %xmm13, %xmm2
+; CHECK-NEXT:    cvtdq2ps %xmm12, %xmm1
+; CHECK-NEXT:    andps %xmm1, %xmm2
+; CHECK-NEXT:    andps %xmm6, %xmm5
+; CHECK-NEXT:    orps %xmm5, %xmm2
+; CHECK-NEXT:    andps %xmm11, %xmm3
+; CHECK-NEXT:    cvttps2dq %xmm8, %xmm1
+; CHECK-NEXT:    cvtdq2ps %xmm1, %xmm1
+; CHECK-NEXT:    andps %xmm1, %xmm3
+; CHECK-NEXT:    andps %xmm6, %xmm14
+; CHECK-NEXT:    orps %xmm14, %xmm3
+; CHECK-NEXT:    andps %xmm6, %xmm10
+; CHECK-NEXT:    andps {{.*}}(%rip), %xmm4
+; CHECK-NEXT:    cvttps2dq %xmm9, %xmm1
+; CHECK-NEXT:    cvtdq2ps %xmm1, %xmm1
+; CHECK-NEXT:    andps %xmm1, %xmm4
+; CHECK-NEXT:    orps %xmm10, %xmm4
+; CHECK-NEXT:    movaps %xmm2, %xmm1
+; CHECK-NEXT:    movaps %xmm3, %xmm2
+; CHECK-NEXT:    movaps %xmm4, %xmm3
 ; CHECK-NEXT:    retq
 bb:
   %v3 = icmp slt <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>, zeroinitializer
   %v14 = zext <16 x i1> %v3 to <16 x i32>
   %v16 = fcmp olt <16 x float> %x, zeroinitializer
   %v17 = sext <16 x i1> %v16 to <16 x i32>
   %v18 = zext <16 x i1> %v16 to <16 x i32>
   %v19 = xor <16 x i32> %v14, %v18

llvm/trunk/test/CodeGen/X86/unfold-masked-merge-vector-variablemask-const.ll

 ; CHECK-SSE1-NEXT:    andnps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    xorps {{.*}}(%rip), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_mone:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movdqa (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm0
 ; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
-; CHECK-SSE2-NEXT:    pandn (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    pxor %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    pxor %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    pand (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    por %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_varx_mone:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovdqa (%rdi), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm0
 ; CHECK-XOP-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vpandn (%rdx), %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vpxor %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpxor %xmm1, %xmm0, %xmm1
+; CHECK-XOP-NEXT:    vpand (%rdi), %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpor %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> %x, <i32 -1, i32 -1, i32 -1, i32 -1> ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, <i32 -1, i32 -1, i32 -1, i32 -1>
   ret <4 x i32> %r

 ; CHECK-SSE1-NEXT:    andnps %xmm2, %xmm0
 ; CHECK-SSE1-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_mone_invmask:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movdqa (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
-; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm2
-; CHECK-SSE2-NEXT:    pxor %xmm1, %xmm2
-; CHECK-SSE2-NEXT:    pandn %xmm2, %xmm0
+; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm1
+; CHECK-SSE2-NEXT:    pcmpeqd %xmm0, %xmm0
+; CHECK-SSE2-NEXT:    pxor %xmm0, %xmm0
 ; CHECK-SSE2-NEXT:    pxor %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    pandn (%rdi), %xmm1
+; CHECK-SSE2-NEXT:    por %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_varx_mone_invmask:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovdqa (%rdi), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm0
 ; CHECK-XOP-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vpxor (%rdx), %xmm1, %xmm2
-; CHECK-XOP-NEXT:    vpandn %xmm2, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpandn (%rdi), %xmm0, %xmm2
+; CHECK-XOP-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-XOP-NEXT:    vpxor %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpor %xmm0, %xmm2, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> %x, <i32 -1, i32 -1, i32 -1, i32 -1> ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 -1, i32 -1, i32 -1, i32 -1>
; CHECK-XOP-NEXT:    retq
   %my = and <4 x i32> %notmask, <i32 42, i32 42, i32 42, i32 42>
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_varx_42(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_varx_42:
 ; CHECK-SSE1:       # %bb.0:
-; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm0 = [5.885454e-44,5.885454e-44,5.885454e-44,5.885454e-44]
+; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andps (%rcx), %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andnps {{.*}}(%rip), %xmm0
+; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_42:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps {{.*#+}} xmm1 = [42,42,42,42]
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm1
+; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    andnps {{.*}}(%rip), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_varx_42:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps {{.*#+}} xmm0 = [42,42,42,42]
-; CHECK-XOP-NEXT:    vxorps (%rdi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vxorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdi), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm1
+; CHECK-XOP-NEXT:    vpcmov %xmm1, {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42> ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
   ret <4 x i32> %r
; CHECK-XOP-NEXT:    retq
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_varx_42_invmask(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_varx_42_invmask:
 ; CHECK-SSE1:       # %bb.0:
 ; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
-; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [5.885454e-44,5.885454e-44,5.885454e-44,5.885454e-44]
-; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm2
-; CHECK-SSE1-NEXT:    xorps %xmm1, %xmm2
-; CHECK-SSE1-NEXT:    andnps %xmm2, %xmm0
-; CHECK-SSE1-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andnps (%rsi), %xmm1
+; CHECK-SSE1-NEXT:    andps {{.*}}(%rip), %xmm0
+; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_varx_42_invmask:
 ; CHECK-SSE2:       # %bb.0:
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    movaps {{.*#+}} xmm1 = [42,42,42,42]
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm2
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm2
-; CHECK-SSE2-NEXT:    andnps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    andnps (%rdi), %xmm1
+; CHECK-SSE2-NEXT:    andps {{.*}}(%rip), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_varx_42_invmask:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
-; CHECK-XOP-NEXT:    vmovaps {{.*#+}} xmm1 = [42,42,42,42]
-; CHECK-XOP-NEXT:    vxorps (%rdi), %xmm1, %xmm2
-; CHECK-XOP-NEXT:    vandnps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa {{.*#+}} xmm1 = [42,42,42,42]
+; CHECK-XOP-NEXT:    vpcmov %xmm0, (%rdi), %xmm1, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> %x, <i32 42, i32 42, i32 42, i32 42> ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, <i32 42, i32 42, i32 42, i32 42>
; CHECK-XOP-NEXT:    retq
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_mone_vary(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_mone_vary:
 ; CHECK-SSE1:       # %bb.0:
-; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
 ; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andnps (%rcx), %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
+; CHECK-SSE1-NEXT:    orps %xmm0, %xmm1
 ; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_mone_vary:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm1
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm1
 ; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andnps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_mone_vary:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %xmm0
-; CHECK-XOP-NEXT:    vandnps (%rdx), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vxorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm0
+; CHECK-XOP-NEXT:    vandnps (%rsi), %xmm0, %xmm1
+; CHECK-XOP-NEXT:    vorps %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
; CHECK-XOP-NEXT:    retq
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_mone_vary_invmask(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_mone_vary_invmask:
 ; CHECK-SSE1:       # %bb.0:
-; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
-; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm1
-; CHECK-SSE1-NEXT:    xorps {{.*}}(%rip), %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm2
-; CHECK-SSE1-NEXT:    andnps %xmm1, %xmm2
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm2
-; CHECK-SSE1-NEXT:    movaps %xmm2, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [nan,nan,nan,nan]
+; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andps (%rdx), %xmm0
+; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_mone_vary_invmask:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movdqa (%rsi), %xmm1
-; CHECK-SSE2-NEXT:    pcmpeqd %xmm2, %xmm2
-; CHECK-SSE2-NEXT:    pxor (%rdx), %xmm2
-; CHECK-SSE2-NEXT:    movdqa %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    pandn %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    pxor %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movdqa (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    pcmpeqd %xmm1, %xmm1
+; CHECK-SSE2-NEXT:    pxor %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    pand (%rsi), %xmm0
+; CHECK-SSE2-NEXT:    por %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_mone_vary_invmask:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovdqa (%rsi), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm0
 ; CHECK-XOP-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vpxor (%rdx), %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vpandn %xmm1, %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vpxor %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vpxor %xmm1, %xmm0, %xmm1
+; CHECK-XOP-NEXT:    vpand (%rsi), %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpor %xmm0, %xmm1, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> <i32 -1, i32 -1, i32 -1, i32 -1>, %y ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, %y
; CHECK-XOP-NEXT:    retq
   %my = and <4 x i32> %notmask, %y
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 define <4 x i32> @in_constant_42_vary(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_42_vary:
 ; CHECK-SSE1:       # %bb.0:
-; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
-; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm1 = [5.885454e-44,5.885454e-44,5.885454e-44,5.885454e-44]
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andps (%rcx), %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
+; CHECK-SSE1-NEXT:    andps {{.*}}(%rip), %xmm0
+; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_42_vary:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm1
-; CHECK-SSE2-NEXT:    movaps {{.*#+}} xmm0 = [42,42,42,42]
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm1
+; CHECK-SSE2-NEXT:    andps {{.*}}(%rip), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_42_vary:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %xmm0
-; CHECK-XOP-NEXT:    vxorps {{.*}}(%rip), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vxorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa {{.*#+}} xmm1 = [42,42,42,42]
+; CHECK-XOP-NEXT:    vpcmov %xmm0, (%rsi), %xmm1, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> <i32 42, i32 42, i32 42, i32 42>, %y ; %x
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
; CHECK-XOP-NEXT:    retq
   %r = or <4 x i32> %mx, %my
   ret <4 x i32> %r
 }
 
 ; This is not a canonical form. Testing for completeness only.
 define <4 x i32> @in_constant_42_vary_invmask(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) {
 ; CHECK-SSE1-LABEL: in_constant_42_vary_invmask:
 ; CHECK-SSE1:       # %bb.0:
-; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
-; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm1
-; CHECK-SSE1-NEXT:    movaps {{.*#+}} xmm2 = [5.885454e-44,5.885454e-44,5.885454e-44,5.885454e-44]
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm2
-; CHECK-SSE1-NEXT:    andnps %xmm2, %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm1
+; CHECK-SSE1-NEXT:    andps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andnps {{.*}}(%rip), %xmm0
+; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_constant_42_vary_invmask:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm1
 ; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    movaps {{.*#+}} xmm2 = [42,42,42,42]
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm2
-; CHECK-SSE2-NEXT:    andnps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm1
+; CHECK-SSE2-NEXT:    andps %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    andnps {{.*}}(%rip), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_constant_42_vary_invmask:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %xmm0
-; CHECK-XOP-NEXT:    vmovaps (%rdx), %xmm1
-; CHECK-XOP-NEXT:    vxorps {{.*}}(%rip), %xmm0, %xmm2
-; CHECK-XOP-NEXT:    vandnps %xmm2, %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vxorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rsi), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm1
+; CHECK-XOP-NEXT:    vpcmov %xmm1, {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %notmask = xor <4 x i32> %mask, <i32 -1, i32 -1, i32 -1, i32 -1>
   %n0 = xor <4 x i32> <i32 42, i32 42, i32 42, i32 42>, %y ; %x
   %n1 = and <4 x i32> %n0, %notmask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r
 }

llvm/trunk/test/CodeGen/X86/unfold-masked-merge-vector-variablemask.ll

 ; CHECK-SSE1-NEXT:    xorl %ecx, %esi
 ; CHECK-SSE1-NEXT:    xorl %edx, %edi
 ; CHECK-SSE1-NEXT:    movl %edi, %eax
 ; CHECK-SSE1-NEXT:    movl %esi, %edx
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v2i8:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v2i8:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <2 x i8> %x, %y
   %n1 = and <2 x i8> %n0, %mask
   %r = xor <2 x i8> %n1, %y
   ret <2 x i8> %r
 }
 
 define <1 x i16> @in_v1i16(<1 x i16> %x, <1 x i16> %y, <1 x i16> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    movb %cl, 2(%rdi)
 ; CHECK-SSE1-NEXT:    movb %dl, 1(%rdi)
 ; CHECK-SSE1-NEXT:    movb %sil, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v4i8:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v4i8:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <4 x i8> %x, %y
   %n1 = and <4 x i8> %n0, %mask
   %r = xor <4 x i8> %n1, %y
   ret <4 x i8> %r
 }
 
 define <2 x i16> @in_v2i16(<2 x i16> %x, <2 x i16> %y, <2 x i16> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    xorl %ecx, %esi
 ; CHECK-SSE1-NEXT:    xorl %edx, %edi
 ; CHECK-SSE1-NEXT:    movl %edi, %eax
 ; CHECK-SSE1-NEXT:    movl %esi, %edx
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v2i16:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v2i16:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <2 x i16> %x, %y
   %n1 = and <2 x i16> %n0, %mask
   %r = xor <2 x i16> %n1, %y
   ret <2 x i16> %r
 }
 
 define <1 x i32> @in_v1i32(<1 x i32> %x, <1 x i32> %y, <1 x i32> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    popq %r13
 ; CHECK-SSE1-NEXT:    popq %r14
 ; CHECK-SSE1-NEXT:    popq %r15
 ; CHECK-SSE1-NEXT:    popq %rbp
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v8i8:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v8i8:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <8 x i8> %x, %y
   %n1 = and <8 x i8> %n0, %mask
   %r = xor <8 x i8> %n1, %y
   ret <8 x i8> %r
 }
 
 define <4 x i16> @in_v4i16(<4 x i16> %x, <4 x i16> %y, <4 x i16> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    movw %cx, 4(%rdi)
 ; CHECK-SSE1-NEXT:    movw %dx, 2(%rdi)
 ; CHECK-SSE1-NEXT:    movw %si, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v4i16:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v4i16:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <4 x i16> %x, %y
   %n1 = and <4 x i16> %n0, %mask
   %r = xor <4 x i16> %n1, %y
   ret <4 x i16> %r
 }
 
 define <2 x i32> @in_v2i32(<2 x i32> %x, <2 x i32> %y, <2 x i32> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    xorl %edx, %edi
 ; CHECK-SSE1-NEXT:    xorl %ecx, %esi
 ; CHECK-SSE1-NEXT:    movl %edi, %eax
 ; CHECK-SSE1-NEXT:    movl %esi, %edx
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v2i32:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v2i32:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <2 x i32> %x, %y
   %n1 = and <2 x i32> %n0, %mask
   %r = xor <2 x i32> %n1, %y
   ret <2 x i32> %r
 }
 
 define <1 x i64> @in_v1i64(<1 x i64> %x, <1 x i64> %y, <1 x i64> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    popq %r13
 ; CHECK-SSE1-NEXT:    popq %r14
 ; CHECK-SSE1-NEXT:    popq %r15
 ; CHECK-SSE1-NEXT:    popq %rbp
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v16i8:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v16i8:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <16 x i8> %x, %y
   %n1 = and <16 x i8> %n0, %mask
   %r = xor <16 x i8> %n1, %y
   ret <16 x i8> %r
 }
 
 define <8 x i16> @in_v8i16(<8 x i16> %x, <8 x i16> %y, <8 x i16> %mask) nounwind {

 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    popq %rbx
 ; CHECK-SSE1-NEXT:    popq %r14
 ; CHECK-SSE1-NEXT:    popq %rbp
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v8i16:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v8i16:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <8 x i16> %x, %y
   %n1 = and <8 x i16> %n0, %mask
   %r = xor <8 x i16> %n1, %y
   ret <8 x i16> %r
 }
 
 define <4 x i32> @in_v4i32(<4 x i32> *%px, <4 x i32> *%py, <4 x i32> *%pmask) nounwind {

 ; CHECK-BASELINE-NEXT:    movl %eax, 4(%rdi)
 ; CHECK-BASELINE-NEXT:    movl %edx, (%rdi)
 ; CHECK-BASELINE-NEXT:    movq %rdi, %rax
 ; CHECK-BASELINE-NEXT:    popq %rbx
 ; CHECK-BASELINE-NEXT:    retq
 ;
 ; CHECK-SSE1-LABEL: in_v4i32:
 ; CHECK-SSE1:       # %bb.0:
-; CHECK-SSE1-NEXT:    movaps (%rdx), %xmm0
-; CHECK-SSE1-NEXT:    movaps (%rsi), %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    andps (%rcx), %xmm1
-; CHECK-SSE1-NEXT:    xorps %xmm0, %xmm1
-; CHECK-SSE1-NEXT:    movaps %xmm1, (%rdi)
+; CHECK-SSE1-NEXT:    movaps (%rcx), %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, %xmm1
+; CHECK-SSE1-NEXT:    andnps (%rdx), %xmm1
+; CHECK-SSE1-NEXT:    andps (%rsi), %xmm0
+; CHECK-SSE1-NEXT:    orps %xmm1, %xmm0
+; CHECK-SSE1-NEXT:    movaps %xmm0, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v4i32:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm1
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm1
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm1
+; CHECK-SSE2-NEXT:    andps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v4i32:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %xmm0
-; CHECK-XOP-NEXT:    vxorps (%rdi), %xmm0, %xmm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %xmm1, %xmm1
-; CHECK-XOP-NEXT:    vxorps %xmm0, %xmm1, %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdi), %xmm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %xmm1
+; CHECK-XOP-NEXT:    vpcmov %xmm1, (%rsi), %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i32>, <4 x i32> *%px, align 16
   %y = load <4 x i32>, <4 x i32> *%py, align 16
   %mask = load <4 x i32>, <4 x i32> *%pmask, align 16
   %n0 = xor <4 x i32> %x, %y
   %n1 = and <4 x i32> %n0, %mask
   %r = xor <4 x i32> %n1, %y
   ret <4 x i32> %r

 ; CHECK-SSE1-NEXT:    xorq %rdx, %rdi
 ; CHECK-SSE1-NEXT:    xorq %rcx, %rsi
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    movq %rsi, %rdx
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v2i64:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
 ; CHECK-SSE2-NEXT:    andps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm1, %xmm0
+; CHECK-SSE2-NEXT:    andnps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v2i64:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vandps %xmm2, %xmm0, %xmm0
-; CHECK-XOP-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-XOP-NEXT:    vpcmov %xmm2, %xmm1, %xmm0, %xmm0
 ; CHECK-XOP-NEXT:    retq
   %n0 = xor <2 x i64> %x, %y
   %n1 = and <2 x i64> %n0, %mask
   %r = xor <2 x i64> %n1, %y
   ret <2 x i64> %r
 }
 
 ; ============================================================================ ;

 ; CHECK-SSE1-NEXT:    popq %r13
 ; CHECK-SSE1-NEXT:    popq %r14
 ; CHECK-SSE1-NEXT:    popq %r15
 ; CHECK-SSE1-NEXT:    popq %rbp
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v32i8:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm2
-; CHECK-SSE2-NEXT:    movaps 16(%rsi), %xmm3
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    movaps 16(%rdi), %xmm1
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
-; CHECK-SSE2-NEXT:    andps 16(%rdx), %xmm1
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps 16(%rdx), %xmm1
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm2
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    andnps 16(%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps 16(%rdi), %xmm1
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm1
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v32i8:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %ymm0
-; CHECK-XOP-NEXT:    vxorps (%rdi), %ymm0, %ymm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %ymm1, %ymm1
-; CHECK-XOP-NEXT:    vxorps %ymm0, %ymm1, %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdi), %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %ymm1
+; CHECK-XOP-NEXT:    vpcmov %ymm1, (%rsi), %ymm0, %ymm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <32 x i8>, <32 x i8> *%px, align 32
   %y = load <32 x i8>, <32 x i8> *%py, align 32
   %mask = load <32 x i8>, <32 x i8> *%pmask, align 32
   %n0 = xor <32 x i8> %x, %y
   %n1 = and <32 x i8> %n0, %mask
   %r = xor <32 x i8> %n1, %y
   ret <32 x i8> %r

 ; CHECK-SSE1-NEXT:    popq %r13
 ; CHECK-SSE1-NEXT:    popq %r14
 ; CHECK-SSE1-NEXT:    popq %r15
 ; CHECK-SSE1-NEXT:    popq %rbp
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v16i16:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm2
-; CHECK-SSE2-NEXT:    movaps 16(%rsi), %xmm3
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    movaps 16(%rdi), %xmm1
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
-; CHECK-SSE2-NEXT:    andps 16(%rdx), %xmm1
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps 16(%rdx), %xmm1
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm2
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    andnps 16(%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps 16(%rdi), %xmm1
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm1
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v16i16:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %ymm0
-; CHECK-XOP-NEXT:    vxorps (%rdi), %ymm0, %ymm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %ymm1, %ymm1
-; CHECK-XOP-NEXT:    vxorps %ymm0, %ymm1, %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdi), %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %ymm1
+; CHECK-XOP-NEXT:    vpcmov %ymm1, (%rsi), %ymm0, %ymm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <16 x i16>, <16 x i16> *%px, align 32
   %y = load <16 x i16>, <16 x i16> *%py, align 32
   %mask = load <16 x i16>, <16 x i16> *%pmask, align 32
   %n0 = xor <16 x i16> %x, %y
   %n1 = and <16 x i16> %n0, %mask
   %r = xor <16 x i16> %n1, %y
   ret <16 x i16> %r

 ; CHECK-SSE1-NEXT:    popq %r13
 ; CHECK-SSE1-NEXT:    popq %r14
 ; CHECK-SSE1-NEXT:    popq %r15
 ; CHECK-SSE1-NEXT:    popq %rbp
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v8i32:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm2
-; CHECK-SSE2-NEXT:    movaps 16(%rsi), %xmm3
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    movaps 16(%rdi), %xmm1
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
-; CHECK-SSE2-NEXT:    andps 16(%rdx), %xmm1
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps 16(%rdx), %xmm1
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm2
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    andnps 16(%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps 16(%rdi), %xmm1
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm1
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v8i32:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %ymm0
-; CHECK-XOP-NEXT:    vxorps (%rdi), %ymm0, %ymm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %ymm1, %ymm1
-; CHECK-XOP-NEXT:    vxorps %ymm0, %ymm1, %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdi), %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %ymm1
+; CHECK-XOP-NEXT:    vpcmov %ymm1, (%rsi), %ymm0, %ymm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <8 x i32>, <8 x i32> *%px, align 32
   %y = load <8 x i32>, <8 x i32> *%py, align 32
   %mask = load <8 x i32>, <8 x i32> *%pmask, align 32
   %n0 = xor <8 x i32> %x, %y
   %n1 = and <8 x i32> %n0, %mask
   %r = xor <8 x i32> %n1, %y
   ret <8 x i32> %r

 ; CHECK-SSE1-NEXT:    movq %rax, 8(%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdx, (%rdi)
 ; CHECK-SSE1-NEXT:    movq %rdi, %rax
 ; CHECK-SSE1-NEXT:    popq %rbx
 ; CHECK-SSE1-NEXT:    retq
 ;
 ; CHECK-SSE2-LABEL: in_v4i64:
 ; CHECK-SSE2:       # %bb.0:
-; CHECK-SSE2-NEXT:    movaps (%rsi), %xmm2
-; CHECK-SSE2-NEXT:    movaps 16(%rsi), %xmm3
-; CHECK-SSE2-NEXT:    movaps (%rdi), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    movaps 16(%rdi), %xmm1
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
-; CHECK-SSE2-NEXT:    andps 16(%rdx), %xmm1
-; CHECK-SSE2-NEXT:    andps (%rdx), %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm2, %xmm0
-; CHECK-SSE2-NEXT:    xorps %xmm3, %xmm1
+; CHECK-SSE2-NEXT:    movaps (%rdx), %xmm0
+; CHECK-SSE2-NEXT:    movaps 16(%rdx), %xmm1
+; CHECK-SSE2-NEXT:    movaps %xmm0, %xmm2
+; CHECK-SSE2-NEXT:    andnps (%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps (%rdi), %xmm0
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm0
+; CHECK-SSE2-NEXT:    movaps %xmm1, %xmm2
+; CHECK-SSE2-NEXT:    andnps 16(%rsi), %xmm2
+; CHECK-SSE2-NEXT:    andps 16(%rdi), %xmm1
+; CHECK-SSE2-NEXT:    orps %xmm2, %xmm1
 ; CHECK-SSE2-NEXT:    retq
 ;
 ; CHECK-XOP-LABEL: in_v4i64:
 ; CHECK-XOP:       # %bb.0:
-; CHECK-XOP-NEXT:    vmovaps (%rsi), %ymm0
-; CHECK-XOP-NEXT:    vxorps (%rdi), %ymm0, %ymm1
-; CHECK-XOP-NEXT:    vandps (%rdx), %ymm1, %ymm1
-; CHECK-XOP-NEXT:    vxorps %ymm0, %ymm1, %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdi), %ymm0
+; CHECK-XOP-NEXT:    vmovdqa (%rdx), %ymm1
+; CHECK-XOP-NEXT:    vpcmov %ymm1, (%rsi), %ymm0, %ymm0
 ; CHECK-XOP-NEXT:    retq
   %x = load <4 x i64>, <4 x i64> *%px, align 32
   %y = load <4 x i64>, <4 x i64> *%py, align 32
   %mask = load <4 x i64>, <4 x i64> *%pmask, align 32
   %n0 = xor <4 x i64> %x, %y
   %n1 = and <4 x i64> %n0, %mask
   %r = xor <4 x i64> %n1, %y
   ret <4 x i64> %r
 }