[X86][AVX512] Improve lowering of AVX512 test intrinsics

[uriel.k]

Summary

Added TESTM and TESTNM to the list of instructions that already zeroing unused upper bits and does not need the redundant shift left and shift right instructions afterwards.

Added a pattern for TESTM and TESTNM in iselLowering, so now icmp(neq,and(X,Y), 0) goes folds into TESTM and icmp(eq,and(X,Y), 0) goes folds into TESTNM

This commit is a preparation for lowering the test and testn X86 intrinsics to IR.

[uriel.k]

test/CodeGen/X86/avx512vl-vec-test-testn.ll

1

This test also governs the case that D38689 revision talked about where in i386 machine it would crash.

[RKSimon]

Add the new test files to trunk with current codegen and then rebase to show the diff from this patch.

lib/Target/X86/X86ISelLowering.cpp

17262

Drop this

17265

Use DAG.getBitcast

[RKSimon]

lib/Target/X86/X86ISelDAGToDAG.cpp

452–455

Pull out repeated N->getOpcode()

unsigned Opcode = N->getOpcode();
if (Opcode == X86ISD::PCMPEQM || Opcode == X86ISD::PCMPGTM ||
    Opcode == X86ISD::TESTM || Opcode == X86ISD::TESTNM ||
    Opcode == X86ISD::CMPM || Opcode == X86ISD::CMPMU) {

459–460

EVT OpVT = N->getOperand(0).getValueType();
if (OpVT == MVT::v8i32 || OpVT == MVT::v8f32)

[craig.topper]

lib/Target/X86/X86ISelLowering.cpp

17259

isBuildVectorAllZeros already peeks through bitcasts so you can just check on Op1 directly.

[uriel.k]

In D38732#893224, @RKSimon wrote:

Add the new test files to trunk with current codegen and then rebase to show the diff from this patch.

You mean to show how the 'CHECK' lines would look like if this tests were to run on trunk and pass but without committing them?

thanks

[uriel.k]

fixed requested changes by Simon and Craig and as you can see added the new test files to trunk as NFC commit and showed the diff.

[RKSimon]

lib/Target/X86/X86ISelDAGToDAG.cpp

452–455

Cheers, you should be able to clang-format this on to fewer lines

lib/Target/X86/X86ISelLowering.cpp

17259

Do we need to support X86ISD::FAND as well?

test/CodeGen/X86/setcc-lowering.ll

28

Should the vptest instructions be in the Integer domain? That should force a vpbroadcastd.

Kind of a pity that the broadcast doesn't fold, but since you're messing with subregs it's not that surprising.

[craig.topper]

lib/Target/X86/X86ISelLowering.cpp

17257

Add a space between "if" and opening paren.

[uriel.k]

test/CodeGen/X86/setcc-lowering.ll

28

care to explain more what do you mean by messing with subregs? Isn't the vpbroadcast here is essential to create a <8 x i23> vector?

and yes, vptest is considered to be in the integer domain so I don't know why it chose vbroadcastss.

[RKSimon]

test/CodeGen/X86/setcc-lowering.ll

28

The vptest is being performed on the zmm register, not just the ymm we care about. I guess because KNL doesn't support the VL variants? So usually this means there is ymm <-> zmm subreg manipulations going on that will interfere with other patterns such as broadcast folding (which we should be safe to do, although naturally full size load folding would be a no-no).

[uriel.k]

Added the case of the pattern with X86::FAND. Hopefully this is the last update. Please let me know if there is something else I missed.

[RKSimon]

LGTM with a couple of minors

lib/Target/X86/X86ISelDAGToDAG.cpp

459–460

This is an NFC - commit it separately first.

lib/Target/X86/X86ISelLowering.cpp

17259

Remove empty line

lib/Target/X86/X86ISelDAGToDAG.cpp

private:
     bool isMaskZeroExtended(SDNode *N) const;
   };
 }
 
 
 // Returns true if this masked compare can be implemented legally with this
 // type.
 static bool isLegalMaskCompare(SDNode *N, const X86Subtarget *Subtarget) {
-  if (N->getOpcode() == X86ISD::PCMPEQM ||
-      N->getOpcode() == X86ISD::PCMPGTM ||
-      N->getOpcode() == X86ISD::CMPM ||
-      N->getOpcode() == X86ISD::CMPMU) {
+  unsigned Opcode = N->getOpcode();
+  if (Opcode == X86ISD::PCMPEQM || Opcode == X86ISD::PCMPGTM ||
+      Opcode == X86ISD::CMPM || Opcode == X86ISD::TESTM ||
+      Opcode == X86ISD::TESTNM || Opcode == X86ISD::CMPMU) {
     // We can get 256-bit 8 element types here without VLX being enabled. When
     // this happens we will use 512-bit operations and the mask will not be
     // zero extended.
-    if (N->getOperand(0).getValueType() == MVT::v8i32 ||
-        N->getOperand(0).getValueType() == MVT::v8f32)
+    EVT OpVT = N->getOperand(0).getValueType();
+    if (OpVT == MVT::v8i32 || OpVT == MVT::v8f32)
       return Subtarget->hasVLX();
 
     return true;
   }
 
   return false;
 }
 

lib/Target/X86/X86ISelLowering.cpp

 }
 
 // Return true if the instruction zeroes the unused upper part of the
 // destination and accepts mask.
 static bool isMaskedZeroUpperBitsvXi1(unsigned int Opcode) {
   switch (Opcode) {
   default:
     return false;
+  case X86ISD::TESTM:
+  case X86ISD::TESTNM:
   case X86ISD::PCMPEQM:
   case X86ISD::PCMPGTM:
   case X86ISD::CMPM:
   case X86ISD::CMPMU:
     return true;
   }
 }
 
static SDValue LowerIntVSETCC_AVX512(SDValue Op, SelectionDAG &DAG) {
   case ISD::SETUGE: SSECC = 5; Unsigned = true; break; //NLT
   case ISD::SETGE:  Swap = true; SSECC = 2; break; // LE + swap
   case ISD::SETULE: Unsigned = true; LLVM_FALLTHROUGH;
   case ISD::SETLE:  SSECC = 2; break;
   }
 
   if (Swap)
     std::swap(Op0, Op1);
+
+  //  See if it is the case of CMP(EQ|NEQ,AND(A,B),ZERO) and change it to TESTM|NM.
+  if ((!Opc && SSECC == 4) || Opc == X86ISD::PCMPEQM) {
+    SDValue A = peekThroughBitcasts(Op0);
+    if ((A.getOpcode() == ISD::AND || A.getOpcode() == X86ISD::FAND) &&
+        ISD::isBuildVectorAllZeros(Op1.getNode())) {
+      MVT VT0 = Op0.getSimpleValueType();
+      SDValue RHS = DAG.getBitcast(VT0, A.getOperand(0));
+      SDValue LHS = DAG.getBitcast(VT0, A.getOperand(1));
+      return DAG.getNode(Opc == X86ISD::PCMPEQM ? X86ISD::TESTNM : X86ISD::TESTM,
+                         dl, VT, RHS, LHS);
+    }
+  }
+
   if (Opc)
     return DAG.getNode(Opc, dl, VT, Op0, Op1);
   Opc = Unsigned ? X86ISD::CMPMU: X86ISD::CMPM;
   return DAG.getNode(Opc, dl, VT, Op0, Op1,
                      DAG.getConstant(SSECC, dl, MVT::i8));
 }
 
 /// \brief Try to turn a VSETULT into a VSETULE by modifying its second

test/CodeGen/X86/avx512-skx-insert-subvec.ll

; CHECK-NEXT:    retq
   ret <8 x i1> %res
 }
 
 define <8 x i1> @test3(<4 x i1> %a) {
 ; CHECK-LABEL: test3:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    vpslld $31, %xmm0, %xmm0
 ; CHECK-NEXT:    vptestmd %xmm0, %xmm0, %k0
-; CHECK-NEXT:    kshiftlb $4, %k0, %k0
-; CHECK-NEXT:    kshiftrb $4, %k0, %k0
 ; CHECK-NEXT:    vpmovm2w %k0, %xmm0
 ; CHECK-NEXT:    retq
 
   %res = shufflevector <4 x i1> %a, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   ret <8 x i1> %res
 }
 
 define <8 x i1> @test4(<4 x i1> %a, <4 x i1>%b) {

test/CodeGen/X86/avx512bw-vec-test-testn.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512bw | FileCheck %s
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i32 @TEST_mm512_test_epi16_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_test_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqw %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestmw %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <32 x i16>
   %1 = icmp ne <32 x i16> %0, zeroinitializer
   %2 = bitcast <32 x i1> %1 to i32
   ret i32 %2
 }
 
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i64 @TEST_mm512_test_epi8_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_test_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqb %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestmb %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovq %k0, %rax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <64 x i8>
   %1 = icmp ne <64 x i8> %0, zeroinitializer
   %2 = bitcast <64 x i1> %1 to i64
   ret i64 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i32 @TEST_mm512_mask_test_epi16_mask(i32 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_test_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpneqw %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmw %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <32 x i16>
   %1 = icmp ne <32 x i16> %0, zeroinitializer
   %2 = bitcast i32 %__U to <32 x i1>
   %3 = and <32 x i1> %1, %2
   %4 = bitcast <32 x i1> %3 to i32
   ret i32 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i64 @TEST_mm512_mask_test_epi8_mask(i64 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_test_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovq %rdi, %k1
-; CHECK-NEXT:    vpcmpneqb %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmb %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovq %k0, %rax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <64 x i8>
   %1 = icmp ne <64 x i8> %0, zeroinitializer
   %2 = bitcast i64 %__U to <64 x i1>
   %3 = and <64 x i1> %1, %2
   %4 = bitcast <64 x i1> %3 to i64
   ret i64 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i32 @TEST_mm512_testn_epi16_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_testn_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqw %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestnmw %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <32 x i16>
   %1 = icmp eq <32 x i16> %0, zeroinitializer
   %2 = bitcast <32 x i1> %1 to i32
   ret i32 %2
 }
 
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i64 @TEST_mm512_testn_epi8_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_testn_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestnmb %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovq %k0, %rax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <64 x i8>
   %1 = icmp eq <64 x i8> %0, zeroinitializer
   %2 = bitcast <64 x i1> %1 to i64
   ret i64 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i32 @TEST_mm512_mask_testn_epi16_mask(i32 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_testn_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpeqw %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmw %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <32 x i16>
   %1 = icmp eq <32 x i16> %0, zeroinitializer
   %2 = bitcast i32 %__U to <32 x i1>
   %3 = and <32 x i1> %1, %2
   %4 = bitcast <32 x i1> %3 to i32
   ret i32 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i64 @TEST_mm512_mask_testn_epi8_mask(i64 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_testn_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovq %rdi, %k1
-; CHECK-NEXT:    vpcmpeqb %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmb %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovq %k0, %rax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <64 x i8>
   %1 = icmp eq <64 x i8> %0, zeroinitializer
   %2 = bitcast i64 %__U to <64 x i1>
   %3 = and <64 x i1> %1, %2
   %4 = bitcast <64 x i1> %3 to i64
   ret i64 %4
 }
 

test/CodeGen/X86/avx512bwvl-vec-test-testn.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512bw,+avx512vl | FileCheck %s
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm_test_epi8_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_test_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqb %xmm1, %xmm0, %k0
+; CHECK-NEXT:    vptestmb %xmm0, %xmm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <16 x i8>
   %1 = icmp ne <16 x i8> %0, zeroinitializer
   %2 = bitcast <16 x i1> %1 to i16
   ret i16 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm_mask_test_epi8_mask(i16 zeroext %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_mask_test_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpneqb %xmm1, %xmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmb %xmm0, %xmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <16 x i8>
   %1 = icmp ne <16 x i8> %0, zeroinitializer
   %2 = bitcast i16 %__U to <16 x i1>
   %3 = and <16 x i1> %1, %2
   %4 = bitcast <16 x i1> %3 to i16
   ret i16 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_test_epi16_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_test_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqw %xmm1, %xmm0, %k0
+; CHECK-NEXT:    vptestmw %xmm0, %xmm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <8 x i16>
   %1 = icmp ne <8 x i16> %0, zeroinitializer
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_mask_test_epi16_mask(i8 zeroext %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_mask_test_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpneqw %xmm1, %xmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmw %xmm0, %xmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <8 x i16>
   %1 = icmp ne <8 x i16> %0, zeroinitializer
   %2 = bitcast i8 %__U to <8 x i1>
   %3 = and <8 x i1> %1, %2
   %4 = bitcast <8 x i1> %3 to i8
   ret i8 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm_testn_epi8_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_testn_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqb %xmm1, %xmm0, %k0
+; CHECK-NEXT:    vptestnmb %xmm0, %xmm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <16 x i8>
   %1 = icmp eq <16 x i8> %0, zeroinitializer
   %2 = bitcast <16 x i1> %1 to i16
   ret i16 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm_mask_testn_epi8_mask(i16 zeroext %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_mask_testn_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpeqb %xmm1, %xmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmb %xmm0, %xmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <16 x i8>
   %1 = icmp eq <16 x i8> %0, zeroinitializer
   %2 = bitcast i16 %__U to <16 x i1>
   %3 = and <16 x i1> %1, %2
   %4 = bitcast <16 x i1> %3 to i16
   ret i16 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_testn_epi16_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_testn_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqw %xmm1, %xmm0, %k0
+; CHECK-NEXT:    vptestnmw %xmm0, %xmm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <8 x i16>
   %1 = icmp eq <8 x i16> %0, zeroinitializer
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_mask_testn_epi16_mask(i8 zeroext %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm_mask_testn_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpeqw %xmm1, %xmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmw %xmm0, %xmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <8 x i16>
   %1 = icmp eq <8 x i16> %0, zeroinitializer
   %2 = bitcast i8 %__U to <8 x i1>
   %3 = and <8 x i1> %1, %2
   %4 = bitcast <8 x i1> %3 to i8
   ret i8 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define i32 @TEST_mm256_test_epi8_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_test_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqb %ymm1, %ymm0, %k0
+; CHECK-NEXT:    vptestmb %ymm0, %ymm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <32 x i8>
   %1 = icmp ne <32 x i8> %0, zeroinitializer
   %2 = bitcast <32 x i1> %1 to i32
   ret i32 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define i32 @TEST_mm256_mask_test_epi8_mask(i32 %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_mask_test_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpneqb %ymm1, %ymm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmb %ymm0, %ymm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <32 x i8>
   %1 = icmp ne <32 x i8> %0, zeroinitializer
   %2 = bitcast i32 %__U to <32 x i1>
   %3 = and <32 x i1> %1, %2
   %4 = bitcast <32 x i1> %3 to i32
   ret i32 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm256_test_epi16_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_test_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqw %ymm1, %ymm0, %k0
+; CHECK-NEXT:    vptestmw %ymm0, %ymm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <16 x i16>
   %1 = icmp ne <16 x i16> %0, zeroinitializer
   %2 = bitcast <16 x i1> %1 to i16
   ret i16 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm256_mask_test_epi16_mask(i16 zeroext %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_mask_test_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpneqw %ymm1, %ymm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmw %ymm0, %ymm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <16 x i16>
   %1 = icmp ne <16 x i16> %0, zeroinitializer
   %2 = bitcast i16 %__U to <16 x i1>
   %3 = and <16 x i1> %1, %2
   %4 = bitcast <16 x i1> %3 to i16
   ret i16 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define i32 @TEST_mm256_testn_epi8_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_testn_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqb %ymm1, %ymm0, %k0
+; CHECK-NEXT:    vptestnmb %ymm0, %ymm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <32 x i8>
   %1 = icmp eq <32 x i8> %0, zeroinitializer
   %2 = bitcast <32 x i1> %1 to i32
   ret i32 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define i32 @TEST_mm256_mask_testn_epi8_mask(i32 %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_mask_testn_epi8_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpeqb %ymm1, %ymm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmb %ymm0, %ymm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <32 x i8>
   %1 = icmp eq <32 x i8> %0, zeroinitializer
   %2 = bitcast i32 %__U to <32 x i1>
   %3 = and <32 x i1> %1, %2
   %4 = bitcast <32 x i1> %3 to i32
   ret i32 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm256_testn_epi16_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_testn_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqw %ymm1, %ymm0, %k0
+; CHECK-NEXT:    vptestnmw %ymm0, %ymm1, %k0
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <16 x i16>
   %1 = icmp eq <16 x i16> %0, zeroinitializer
   %2 = bitcast <16 x i1> %1 to i16
   ret i16 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm256_mask_testn_epi16_mask(i16 zeroext %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm256_mask_testn_epi16_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovd %edi, %k1
-; CHECK-NEXT:    vpcmpeqw %ymm1, %ymm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmw %ymm0, %ymm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovd %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <16 x i16>
   %1 = icmp eq <16 x i16> %0, zeroinitializer
   %2 = bitcast i16 %__U to <16 x i1>
   %3 = and <16 x i1> %1, %2
   %4 = bitcast <16 x i1> %3 to i16
   ret i16 %4
 }
 
 

test/CodeGen/X86/avx512f-vec-test-testn.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512f | FileCheck %s
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm512_test_epi64_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_test_epi64_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqq %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestmq %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = icmp ne <8 x i64> %and.i.i, zeroinitializer
   %1 = bitcast <8 x i1> %0 to i8
   ret i8 %1
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm512_test_epi32_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_test_epi32_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpneqd %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestmd %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <16 x i32>
   %1 = icmp ne <16 x i32> %0, zeroinitializer
   %2 = bitcast <16 x i1> %1 to i16
   ret i16 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm512_mask_test_epi64_mask(i8 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_test_epi64_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovw %edi, %k1
-; CHECK-NEXT:    vpcmpneqq %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmq %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = icmp ne <8 x i64> %and.i.i, zeroinitializer
   %1 = bitcast i8 %__U to <8 x i1>
   %2 = and <8 x i1> %0, %1
   %3 = bitcast <8 x i1> %2 to i8
   ret i8 %3
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm512_mask_test_epi32_mask(i16 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_test_epi32_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovw %edi, %k1
-; CHECK-NEXT:    vpcmpneqd %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestmd %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <16 x i32>
   %1 = icmp ne <16 x i32> %0, zeroinitializer
   %2 = bitcast i16 %__U to <16 x i1>
   %3 = and <16 x i1> %1, %2
   %4 = bitcast <16 x i1> %3 to i16
   ret i16 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm512_testn_epi64_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_testn_epi64_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqq %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestnmq %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = icmp eq <8 x i64> %and.i.i, zeroinitializer
   %1 = bitcast <8 x i1> %0 to i8
   ret i8 %1
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm512_testn_epi32_mask(<8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_testn_epi32_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; CHECK-NEXT:    vpcmpeqd %zmm1, %zmm0, %k0
+; CHECK-NEXT:    vptestnmd %zmm0, %zmm1, %k0
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <16 x i32>
   %1 = icmp eq <16 x i32> %0, zeroinitializer
   %2 = bitcast <16 x i1> %1 to i16
   ret i16 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm512_mask_testn_epi64_mask(i8 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_testn_epi64_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovw %edi, %k1
-; CHECK-NEXT:    vpcmpeqq %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmq %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = icmp eq <8 x i64> %and.i.i, zeroinitializer
   %1 = bitcast i8 %__U to <8 x i1>
   %2 = and <8 x i1> %0, %1
   %3 = bitcast <8 x i1> %2 to i8
   ret i8 %3
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i16 @TEST_mm512_mask_testn_epi32_mask(i16 %__U, <8 x i64> %__A, <8 x i64> %__B) local_unnamed_addr #0 {
 ; CHECK-LABEL: TEST_mm512_mask_testn_epi32_mask:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vpandq %zmm0, %zmm1, %zmm0
-; CHECK-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    kmovw %edi, %k1
-; CHECK-NEXT:    vpcmpeqd %zmm1, %zmm0, %k0 {%k1}
+; CHECK-NEXT:    vptestnmd %zmm0, %zmm1, %k0 {%k1}
 ; CHECK-NEXT:    kmovw %k0, %eax
 ; CHECK-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 entry:
   %and.i.i = and <8 x i64> %__B, %__A
   %0 = bitcast <8 x i64> %and.i.i to <16 x i32>
   %1 = icmp eq <16 x i32> %0, zeroinitializer
   %2 = bitcast i16 %__U to <16 x i1>
   %3 = and <16 x i1> %1, %2
   %4 = bitcast <16 x i1> %3 to i16
   ret i16 %4
 }
 

test/CodeGen/X86/avx512vl-vec-masked-cmp.ll

 ; VLX-NEXT:    retq
 ;
 ; NoVLX-LABEL: test_vpcmpeqw_v8i1_v16i1_mask:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpeqw %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp eq <8 x i16> %0, %1

 ; VLX-NEXT:    retq
 ;
 ; NoVLX-LABEL: test_vpcmpeqw_v8i1_v16i1_mask_mem:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpeqw (%rdi), %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ;
 ; NoVLX-LABEL: test_masked_vpcmpeqw_v8i1_v16i1_mask:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpeqw %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp eq <8 x i16> %0, %1

 ;
 ; NoVLX-LABEL: test_masked_vpcmpeqw_v8i1_v16i1_mask_mem:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpeqw (%rsi), %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ; VLX-NEXT:    retq
 ;
 ; NoVLX-LABEL: test_vpcmpsgtw_v8i1_v16i1_mask:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpgtw %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp sgt <8 x i16> %0, %1

 ; VLX-NEXT:    retq
 ;
 ; NoVLX-LABEL: test_vpcmpsgtw_v8i1_v16i1_mask_mem:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpgtw (%rdi), %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ;
 ; NoVLX-LABEL: test_masked_vpcmpsgtw_v8i1_v16i1_mask:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpgtw %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp sgt <8 x i16> %0, %1

 ;
 ; NoVLX-LABEL: test_masked_vpcmpsgtw_v8i1_v16i1_mask_mem:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpgtw (%rsi), %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ; NoVLX-LABEL: test_vpcmpsgew_v8i1_v16i1_mask:
 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
 ; NoVLX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp sge <8 x i16> %0, %1

 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vmovdqa (%rdi), %xmm1
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
 ; NoVLX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
 ; NoVLX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp sge <8 x i16> %0, %1

 ; NoVLX-NEXT:    vmovdqa (%rsi), %xmm1
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
 ; NoVLX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vmovdqa {{.*#+}} xmm2 = [32768,32768,32768,32768,32768,32768,32768,32768]
 ; NoVLX-NEXT:    vpxor %xmm2, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpxor %xmm2, %xmm1, %xmm1
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp ult <8 x i16> %0, %1

 ; NoVLX:       # BB#0: # %entry
 ; NoVLX-NEXT:    vmovdqa {{.*#+}} xmm1 = [32768,32768,32768,32768,32768,32768,32768,32768]
 ; NoVLX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpxor (%rdi), %xmm1, %xmm1
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

 ; NoVLX-NEXT:    vmovdqa {{.*#+}} xmm2 = [32768,32768,32768,32768,32768,32768,32768,32768]
 ; NoVLX-NEXT:    vpxor %xmm2, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpxor %xmm2, %xmm1, %xmm1
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %1 = bitcast <2 x i64> %__b to <8 x i16>
   %2 = icmp ult <8 x i16> %0, %1

 ; NoVLX-NEXT:    vmovdqa {{.*#+}} xmm1 = [32768,32768,32768,32768,32768,32768,32768,32768]
 ; NoVLX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; NoVLX-NEXT:    vpxor (%rsi), %xmm1, %xmm1
 ; NoVLX-NEXT:    vpcmpgtw %xmm0, %xmm1, %xmm0
 ; NoVLX-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; NoVLX-NEXT:    vpsllq $63, %zmm0, %zmm0
 ; NoVLX-NEXT:    kmovw %edi, %k1
 ; NoVLX-NEXT:    vptestmq %zmm0, %zmm0, %k0 {%k1}
-; NoVLX-NEXT:    kxorw %k0, %k0, %k1
-; NoVLX-NEXT:    kunpckbw %k0, %k1, %k0
 ; NoVLX-NEXT:    kmovw %k0, %eax
 ; NoVLX-NEXT:    # kill: %AX<def> %AX<kill> %EAX<kill>
 ; NoVLX-NEXT:    vzeroupper
 ; NoVLX-NEXT:    retq
 entry:
   %0 = bitcast <2 x i64> %__a to <8 x i16>
   %load = load <2 x i64>, <2 x i64>* %__b
   %1 = bitcast <2 x i64> %load to <8 x i16>

test/CodeGen/X86/avx512vl-vec-test-testn.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512vl | FileCheck %s --check-prefix=X86_64
 ; RUN: llc < %s -mtriple=i386-unknown-unknown -mattr=+avx512vl | FileCheck %s --check-prefix=I386
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_test_epi64_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_test_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpneqq %xmm1, %xmm0, %k0
+; X86_64-NEXT:    vptestmq %xmm0, %xmm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_test_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpneqq %xmm1, %xmm0, %k0
+; I386-NEXT:    vptestmq %xmm0, %xmm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = icmp ne <2 x i64> %and.i.i, zeroinitializer
   %1 = shufflevector <2 x i1> %0, <2 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 2, i32 3, i32 2, i32 3>
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_test_epi32_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_test_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpneqd %xmm1, %xmm0, %k0
+; X86_64-NEXT:    vptestmd %xmm0, %xmm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_test_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpneqd %xmm1, %xmm0, %k0
+; I386-NEXT:    vptestmd %xmm0, %xmm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <4 x i32>
   %1 = icmp ne <4 x i32> %0, zeroinitializer
   %2 = shufflevector <4 x i1> %1, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %3 = bitcast <8 x i1> %2 to i8
   ret i8 %3
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_test_epi64_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_test_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpneqq %ymm1, %ymm0, %k0
+; X86_64-NEXT:    vptestmq %ymm0, %ymm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_test_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpneqq %ymm1, %ymm0, %k0
+; I386-NEXT:    vptestmq %ymm0, %ymm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = icmp ne <4 x i64> %and.i.i, zeroinitializer
   %1 = shufflevector <4 x i1> %0, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_test_epi32_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_test_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpneqd %ymm1, %ymm0, %k0
+; X86_64-NEXT:    vptestmd %ymm0, %ymm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_test_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpneqd %ymm1, %ymm0, %k0
+; I386-NEXT:    vptestmd %ymm0, %ymm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <8 x i32>
   %1 = icmp ne <8 x i32> %0, zeroinitializer
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_mask_test_epi64_mask(i8 %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_mask_test_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpneqq %xmm1, %xmm0, %k0 {%k1}
+; X86_64-NEXT:    vptestmq %xmm0, %xmm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_mask_test_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpneqq %xmm1, %xmm0, %k0 {%k1}
+; I386-NEXT:    vptestmq %xmm0, %xmm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = icmp ne <2 x i64> %and.i.i, zeroinitializer
   %1 = bitcast i8 %__U to <8 x i1>
   %2 = shufflevector <8 x i1> %1, <8 x i1> undef, <2 x i32> <i32 0, i32 1>
   %3 = and <2 x i1> %0, %2
   %4 = shufflevector <2 x i1> %3, <2 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 2, i32 3, i32 2, i32 3>
   %5 = bitcast <8 x i1> %4 to i8
   ret i8 %5
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_mask_test_epi32_mask(i8 %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_mask_test_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpneqd %xmm1, %xmm0, %k0 {%k1}
+; X86_64-NEXT:    vptestmd %xmm0, %xmm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_mask_test_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpneqd %xmm1, %xmm0, %k0 {%k1}
+; I386-NEXT:    vptestmd %xmm0, %xmm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <4 x i32>
   %1 = icmp ne <4 x i32> %0, zeroinitializer
   %2 = bitcast i8 %__U to <8 x i1>
   %3 = shufflevector <8 x i1> %2, <8 x i1> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %4 = and <4 x i1> %1, %3
   %5 = shufflevector <4 x i1> %4, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %6 = bitcast <8 x i1> %5 to i8
   ret i8 %6
 }
 
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_mask_test_epi64_mask(i8 %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_mask_test_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpneqq %ymm1, %ymm0, %k0 {%k1}
+; X86_64-NEXT:    vptestmq %ymm0, %ymm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_mask_test_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpneqq %ymm1, %ymm0, %k0 {%k1}
+; I386-NEXT:    vptestmq %ymm0, %ymm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = icmp ne <4 x i64> %and.i.i, zeroinitializer
   %1 = bitcast i8 %__U to <8 x i1>
   %2 = shufflevector <8 x i1> %1, <8 x i1> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %3 = and <4 x i1> %0, %2
   %4 = shufflevector <4 x i1> %3, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %5 = bitcast <8 x i1> %4 to i8
   ret i8 %5
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_mask_test_epi32_mask(i8 %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_mask_test_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpneqd %ymm1, %ymm0, %k0 {%k1}
+; X86_64-NEXT:    vptestmd %ymm0, %ymm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_mask_test_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpneqd %ymm1, %ymm0, %k0 {%k1}
+; I386-NEXT:    vptestmd %ymm0, %ymm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <8 x i32>
   %1 = icmp ne <8 x i32> %0, zeroinitializer
   %2 = bitcast i8 %__U to <8 x i1>
   %3 = and <8 x i1> %1, %2
   %4 = bitcast <8 x i1> %3 to i8
   ret i8 %4
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_testn_epi64_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_testn_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpeqq %xmm1, %xmm0, %k0
+; X86_64-NEXT:    vptestnmq %xmm0, %xmm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_testn_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpeqq %xmm1, %xmm0, %k0
+; I386-NEXT:    vptestnmq %xmm0, %xmm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = icmp eq <2 x i64> %and.i.i, zeroinitializer
   %1 = shufflevector <2 x i1> %0, <2 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 2, i32 3, i32 2, i32 3>
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_testn_epi32_mask(<2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_testn_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpeqd %xmm1, %xmm0, %k0
+; X86_64-NEXT:    vptestnmd %xmm0, %xmm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_testn_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpeqd %xmm1, %xmm0, %k0
+; I386-NEXT:    vptestnmd %xmm0, %xmm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <4 x i32>
   %1 = icmp eq <4 x i32> %0, zeroinitializer
   %2 = shufflevector <4 x i1> %1, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %3 = bitcast <8 x i1> %2 to i8
   ret i8 %3
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_testn_epi64_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_testn_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpeqq %ymm1, %ymm0, %k0
+; X86_64-NEXT:    vptestnmq %ymm0, %ymm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_testn_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpeqq %ymm1, %ymm0, %k0
+; I386-NEXT:    vptestnmq %ymm0, %ymm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = icmp eq <4 x i64> %and.i.i, zeroinitializer
   %1 = shufflevector <4 x i1> %0, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_testn_epi32_mask(<4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_testn_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X86_64-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0
+; X86_64-NEXT:    vptestnmd %ymm0, %ymm1, %k0
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_testn_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; I386-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0
+; I386-NEXT:    vptestnmd %ymm0, %ymm1, %k0
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <8 x i32>
   %1 = icmp eq <8 x i32> %0, zeroinitializer
   %2 = bitcast <8 x i1> %1 to i8
   ret i8 %2
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_mask_testn_epi64_mask(i8 %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_mask_testn_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpeqq %xmm1, %xmm0, %k0 {%k1}
+; X86_64-NEXT:    vptestnmq %xmm0, %xmm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_mask_testn_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpeqq %xmm1, %xmm0, %k0 {%k1}
+; I386-NEXT:    vptestnmq %xmm0, %xmm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = icmp eq <2 x i64> %and.i.i, zeroinitializer
   %1 = bitcast i8 %__U to <8 x i1>
   %2 = shufflevector <8 x i1> %1, <8 x i1> undef, <2 x i32> <i32 0, i32 1>
   %3 = and <2 x i1> %0, %2
   %4 = shufflevector <2 x i1> %3, <2 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 2, i32 3, i32 2, i32 3>
   %5 = bitcast <8 x i1> %4 to i8
   ret i8 %5
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm_mask_testn_epi32_mask(i8 %__U, <2 x i64> %__A, <2 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm_mask_testn_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpeqd %xmm1, %xmm0, %k0 {%k1}
+; X86_64-NEXT:    vptestnmd %xmm0, %xmm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm_mask_testn_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %xmm0, %xmm1, %xmm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpeqd %xmm1, %xmm0, %k0 {%k1}
+; I386-NEXT:    vptestnmd %xmm0, %xmm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <2 x i64> %__B, %__A
   %0 = bitcast <2 x i64> %and.i.i to <4 x i32>
   %1 = icmp eq <4 x i32> %0, zeroinitializer
   %2 = bitcast i8 %__U to <8 x i1>
   %3 = shufflevector <8 x i1> %2, <8 x i1> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %4 = and <4 x i1> %1, %3
   %5 = shufflevector <4 x i1> %4, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %6 = bitcast <8 x i1> %5 to i8
   ret i8 %6
 }
 
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_mask_testn_epi64_mask(i8 %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_mask_testn_epi64_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpeqq %ymm1, %ymm0, %k0 {%k1}
+; X86_64-NEXT:    vptestnmq %ymm0, %ymm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_mask_testn_epi64_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpeqq %ymm1, %ymm0, %k0 {%k1}
+; I386-NEXT:    vptestnmq %ymm0, %ymm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = icmp eq <4 x i64> %and.i.i, zeroinitializer
   %1 = bitcast i8 %__U to <8 x i1>
   %2 = shufflevector <8 x i1> %1, <8 x i1> undef, <4 x i32> <i32 0, i32 1, i32 2, i32 3>
   %3 = and <4 x i1> %0, %2
   %4 = shufflevector <4 x i1> %3, <4 x i1> zeroinitializer, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %5 = bitcast <8 x i1> %4 to i8
   ret i8 %5
 }
 
 ; Function Attrs: norecurse nounwind readnone
 define zeroext i8 @TEST_mm256_mask_testn_epi32_mask(i8 %__U, <4 x i64> %__A, <4 x i64> %__B) local_unnamed_addr #0 {
 ; X86_64-LABEL: TEST_mm256_mask_testn_epi32_mask:
 ; X86_64:       # BB#0: # %entry
-; X86_64-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; X86_64-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X86_64-NEXT:    kmovw %edi, %k1
-; X86_64-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0 {%k1}
+; X86_64-NEXT:    vptestnmd %ymm0, %ymm1, %k0 {%k1}
 ; X86_64-NEXT:    kmovw %k0, %eax
 ; X86_64-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; X86_64-NEXT:    vzeroupper
 ; X86_64-NEXT:    retq
 ;
 ; I386-LABEL: TEST_mm256_mask_testn_epi32_mask:
 ; I386:       # BB#0: # %entry
-; I386-NEXT:    vpand %ymm0, %ymm1, %ymm0
-; I386-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; I386-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; I386-NEXT:    kmovw %eax, %k1
-; I386-NEXT:    vpcmpeqd %ymm1, %ymm0, %k0 {%k1}
+; I386-NEXT:    vptestnmd %ymm0, %ymm1, %k0 {%k1}
 ; I386-NEXT:    kmovw %k0, %eax
 ; I386-NEXT:    # kill: %AL<def> %AL<kill> %EAX<kill>
 ; I386-NEXT:    vzeroupper
 ; I386-NEXT:    retl
 entry:
   %and.i.i = and <4 x i64> %__B, %__A
   %0 = bitcast <4 x i64> %and.i.i to <8 x i32>
   %1 = icmp eq <8 x i32> %0, zeroinitializer
   %2 = bitcast i8 %__U to <8 x i1>
   %3 = and <8 x i1> %1, %2
   %4 = bitcast <8 x i1> %3 to i8
   ret i8 %4
 }
 

test/CodeGen/X86/compress_expand.ll

 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
 ;
 ; KNL-LABEL: test7:
 ; KNL:       # BB#0:
 ; KNL-NEXT:    # kill: %YMM0<def> %YMM0<kill> %ZMM0<def>
 ; KNL-NEXT:    vpmovsxwq %xmm1, %zmm1
 ; KNL-NEXT:    vpsllq $63, %zmm1, %zmm1
-; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k0
-; KNL-NEXT:    kshiftlw $8, %k0, %k0
-; KNL-NEXT:    kshiftrw $8, %k0, %k1
+; KNL-NEXT:    vptestmq %zmm1, %zmm1, %k1
 ; KNL-NEXT:    vcompressps %zmm0, (%rdi) {%k1}
 ; KNL-NEXT:    retq
   call void @llvm.masked.compressstore.v8f32(<8 x float> %V, float* %base, <8 x i1> %mask)
   ret void
 }
 
 define void @test8(double* %base, <8 x double> %V, <8 x i1> %mask) {
 ; SKX-LABEL: test8:

test/CodeGen/X86/masked_gather_scatter.ll

 ; KNL_32-NEXT:    vscatterqps %ymm0, (,%zmm1) {%k1}
 ; KNL_32-NEXT:    vzeroupper
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test20:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    # kill: %XMM1<def> %XMM1<kill> %YMM1<def>
 ; SKX-NEXT:    vpsllq $63, %xmm2, %xmm2
-; SKX-NEXT:    vptestmq %xmm2, %xmm2, %k0
-; SKX-NEXT:    kshiftlb $6, %k0, %k0
-; SKX-NEXT:    kshiftrb $6, %k0, %k1
+; SKX-NEXT:    vptestmq %xmm2, %xmm2, %k1
 ; SKX-NEXT:    vscatterqps %xmm0, (,%ymm1) {%k1}
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test20:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    vpermilps {{.*#+}} xmm1 = xmm1[0,2,2,3]
 ; SKX_32-NEXT:    vpsllq $63, %xmm2, %xmm2
-; SKX_32-NEXT:    vptestmq %xmm2, %xmm2, %k0
-; SKX_32-NEXT:    kshiftlb $6, %k0, %k0
-; SKX_32-NEXT:    kshiftrb $6, %k0, %k1
+; SKX_32-NEXT:    vptestmq %xmm2, %xmm2, %k1
 ; SKX_32-NEXT:    vscatterdps %xmm0, (,%xmm1) {%k1}
 ; SKX_32-NEXT:    retl
   call void @llvm.masked.scatter.v2f32.v2p0f32(<2 x float> %a1, <2 x float*> %ptr, i32 4, <2 x i1> %mask)
   ret void
 }
 
 ; Data type requires promotion
 define void @test21(<2 x i32>%a1, <2 x i32*> %ptr, <2 x i1>%mask) {

 ; KNL_32-NEXT:    vpscatterqd %ymm0, (,%zmm1) {%k1}
 ; KNL_32-NEXT:    vzeroupper
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test21:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    # kill: %XMM1<def> %XMM1<kill> %YMM1<def>
 ; SKX-NEXT:    vpsllq $63, %xmm2, %xmm2
-; SKX-NEXT:    vptestmq %xmm2, %xmm2, %k0
-; SKX-NEXT:    kshiftlb $6, %k0, %k0
-; SKX-NEXT:    kshiftrb $6, %k0, %k1
+; SKX-NEXT:    vptestmq %xmm2, %xmm2, %k1
 ; SKX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SKX-NEXT:    vpscatterqd %xmm0, (,%ymm1) {%k1}
 ; SKX-NEXT:    vzeroupper
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test21:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    # kill: %XMM1<def> %XMM1<kill> %YMM1<def>
 ; SKX_32-NEXT:    vpsllq $63, %xmm2, %xmm2
-; SKX_32-NEXT:    vptestmq %xmm2, %xmm2, %k0
-; SKX_32-NEXT:    kshiftlb $6, %k0, %k0
-; SKX_32-NEXT:    kshiftrb $6, %k0, %k1
+; SKX_32-NEXT:    vptestmq %xmm2, %xmm2, %k1
 ; SKX_32-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SKX_32-NEXT:    vpscatterqd %xmm0, (,%ymm1) {%k1}
 ; SKX_32-NEXT:    vzeroupper
 ; SKX_32-NEXT:    retl
   call void @llvm.masked.scatter.v2i32.v2p0i32(<2 x i32> %a1, <2 x i32*> %ptr, i32 4, <2 x i1> %mask)
   ret void
 }
 

 ; KNL_32-NEXT:    vmovaps %xmm2, %xmm0
 ; KNL_32-NEXT:    vzeroupper
 ; KNL_32-NEXT:    retl
 ;
 ; SKX-LABEL: test22:
 ; SKX:       # BB#0:
 ; SKX-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SKX-NEXT:    vpsllq $63, %xmm1, %xmm1
-; SKX-NEXT:    vptestmq %xmm1, %xmm1, %k0
-; SKX-NEXT:    kshiftlb $6, %k0, %k0
-; SKX-NEXT:    kshiftrb $6, %k0, %k1
+; SKX-NEXT:    vptestmq %xmm1, %xmm1, %k1
 ; SKX-NEXT:    vgatherdps (%rdi,%xmm0,4), %xmm2 {%k1}
 ; SKX-NEXT:    vmovaps %xmm2, %xmm0
 ; SKX-NEXT:    retq
 ;
 ; SKX_32-LABEL: test22:
 ; SKX_32:       # BB#0:
 ; SKX_32-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[0,2,2,3]
 ; SKX_32-NEXT:    vpsllq $63, %xmm1, %xmm1
-; SKX_32-NEXT:    vptestmq %xmm1, %xmm1, %k0
-; SKX_32-NEXT:    kshiftlb $6, %k0, %k0
-; SKX_32-NEXT:    kshiftrb $6, %k0, %k1
+; SKX_32-NEXT:    vptestmq %xmm1, %xmm1, %k1
 ; SKX_32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; SKX_32-NEXT:    vgatherdps (%eax,%xmm0,4), %xmm2 {%k1}
 ; SKX_32-NEXT:    vmovaps %xmm2, %xmm0
 ; SKX_32-NEXT:    retl
   %sext_ind = sext <2 x i32> %ind to <2 x i64>
   %gep.random = getelementptr float, float* %base, <2 x i64> %sext_ind
   %res = call <2 x float> @llvm.masked.gather.v2f32.v2p0f32(<2 x float*> %gep.random, i32 4, <2 x i1> %mask, <2 x float> %src0)
   ret <2 x float>%res

test/CodeGen/X86/masked_memop.ll

 ; AVX2-NEXT:    vblendvps %ymm0, %ymm2, %ymm1, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: test11b:
 ; AVX512F:       ## BB#0:
 ; AVX512F-NEXT:    ## kill: %YMM1<def> %YMM1<kill> %ZMM1<def>
 ; AVX512F-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; AVX512F-NEXT:    vpsllq $63, %zmm0, %zmm0
-; AVX512F-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; AVX512F-NEXT:    kshiftlw $8, %k0, %k0
-; AVX512F-NEXT:    kshiftrw $8, %k0, %k1
+; AVX512F-NEXT:    vptestmq %zmm0, %zmm0, %k1
 ; AVX512F-NEXT:    vpblendmd (%rdi), %zmm1, %zmm0 {%k1}
 ; AVX512F-NEXT:    ## kill: %YMM0<def> %YMM0<kill> %ZMM0<kill>
 ; AVX512F-NEXT:    retq
 ;
 ; SKX-LABEL: test11b:
 ; SKX:       ## BB#0:
 ; SKX-NEXT:    vpsllw $15, %xmm0, %xmm0
 ; SKX-NEXT:    vpmovw2m %xmm0, %k1

 ; AVX2-NEXT:    vpsrad $31, %ymm0, %ymm0
 ; AVX2-NEXT:    vmaskmovps (%rdi), %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: test11c:
 ; AVX512F:       ## BB#0:
 ; AVX512F-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; AVX512F-NEXT:    vpsllq $63, %zmm0, %zmm0
-; AVX512F-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; AVX512F-NEXT:    kshiftlw $8, %k0, %k0
-; AVX512F-NEXT:    kshiftrw $8, %k0, %k1
+; AVX512F-NEXT:    vptestmq %zmm0, %zmm0, %k1
 ; AVX512F-NEXT:    vmovups (%rdi), %zmm0 {%k1} {z}
 ; AVX512F-NEXT:    ## kill: %YMM0<def> %YMM0<kill> %ZMM0<kill>
 ; AVX512F-NEXT:    retq
 ;
 ; SKX-LABEL: test11c:
 ; SKX:       ## BB#0:
 ; SKX-NEXT:    vpsllw $15, %xmm0, %xmm0
 ; SKX-NEXT:    vpmovw2m %xmm0, %k1

 ; AVX2-NEXT:    vpsrad $31, %ymm0, %ymm0
 ; AVX2-NEXT:    vpmaskmovd (%rdi), %ymm0, %ymm0
 ; AVX2-NEXT:    retq
 ;
 ; AVX512F-LABEL: test11d:
 ; AVX512F:       ## BB#0:
 ; AVX512F-NEXT:    vpmovsxwq %xmm0, %zmm0
 ; AVX512F-NEXT:    vpsllq $63, %zmm0, %zmm0
-; AVX512F-NEXT:    vptestmq %zmm0, %zmm0, %k0
-; AVX512F-NEXT:    kshiftlw $8, %k0, %k0
-; AVX512F-NEXT:    kshiftrw $8, %k0, %k1
+; AVX512F-NEXT:    vptestmq %zmm0, %zmm0, %k1
 ; AVX512F-NEXT:    vmovdqu32 (%rdi), %zmm0 {%k1} {z}
 ; AVX512F-NEXT:    ## kill: %YMM0<def> %YMM0<kill> %ZMM0<kill>
 ; AVX512F-NEXT:    retq
 ;
 ; SKX-LABEL: test11d:
 ; SKX:       ## BB#0:
 ; SKX-NEXT:    vpsllw $15, %xmm0, %xmm0
 ; SKX-NEXT:    vpmovw2m %xmm0, %k1

test/CodeGen/X86/setcc-lowering.ll

 ; AVX-NEXT:    vpor {{.*}}(%rip), %xmm0, %xmm0
 ; AVX-NEXT:    vpsllw $15, %xmm0, %xmm0
 ; AVX-NEXT:    vpsraw $15, %xmm0, %xmm0
 ; AVX-NEXT:    vzeroupper
 ; AVX-NEXT:    retq
 ;
 ; KNL-32-LABEL: pr25080:
 ; KNL-32:       # BB#0: # %entry
-; KNL-32-NEXT:    vpbroadcastd {{.*#+}} ymm1 = [8388607,8388607,8388607,8388607,8388607,8388607,8388607,8388607]
-; KNL-32-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; KNL-32-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; KNL-32-NEXT:    vpcmpeqd %zmm1, %zmm0, %k0
+; KNL-32-NEXT:    # kill: %YMM0<def> %YMM0<kill> %ZMM0<def>
+; KNL-32-NEXT:    vbroadcastss {{.*#+}} ymm1 = [8388607,8388607,8388607,8388607,8388607,8388607,8388607,8388607]
+; KNL-32-NEXT:    vptestnmd %zmm1, %zmm0, %k0
 ; KNL-32-NEXT:    movb $15, %al
 ; KNL-32-NEXT:    kmovw %eax, %k1
 ; KNL-32-NEXT:    korw %k1, %k0, %k1
 ; KNL-32-NEXT:    vpternlogq $255, %zmm0, %zmm0, %zmm0 {%k1} {z}
 ; KNL-32-NEXT:    vpmovqw %zmm0, %xmm0
 ; KNL-32-NEXT:    retl
 entry:
   %0 = trunc <8 x i32> %a to <8 x i23>