[X86] Make memcmp() use PTEST if possible and also enable AVX1

[davezarzycki]

Summary

Hi Craig – I'm not an expert in this area. Feedback would be appreciated. In particular, the way PTEST was wired up. Thanks!

[craig.topper]

lib/Target/X86/X86ISelLowering.cpp

20991

Remove this. It doesn't make sense.

42584

Why are these machine opcodes and not ISD::XOR?

42606

PTEST returns an i32 representing EFLAGS. You can't pass it to ISD::SETCC it doesn't mean anything. The DAG.getSetcc call needs to call the X86ISelLowering.cpp copy of getSetcc that creates an X86ISD::SETCC. You'll need to pass it the X86::COND_E/X86::COND_NE here. This will return an MVT::i8 so you'll need to emit an ISD::TRUNCATE to VT after it.

[davezarzycki]

lib/Target/X86/X86ISelLowering.cpp

42584

My first attempt at this patch used ISD::XOR after the vector bitcasts, but scalar instructions were often (but not always) were emitted. I later noticed the comment at the top of the function which suggests/implies that explicit vector instructions are the right solution and indeed that worked reliably.

42606

Thanks for the PTEST feedback. I think I have it working the way you describe.

[davezarzycki]

Incorporated PTEST feedback.

[RKSimon]

In general we don't use auto for anything other than casts or iterators - see the llvm coding style guidelines

lib/Target/X86/X86ISelLowering.cpp

42551

bool HasAVX = Subtarget.hasAVX();

42558

I'd prefer not to nest ternary operators like this - either nest them in brackets or if-else

42575

(style guide) Don't use auto - use unsigned

[craig.topper]

lib/Target/X86/X86ISelLowering.cpp

42606

80 columns

[craig.topper]

lib/Target/X86/X86ISelLowering.cpp

42605

You already used PT as a variable name earlier. The earlier one should probably be UsePTEST

[craig.topper]

test/CodeGen/X86/setcc-wide-types.ll

2–3

Need an sse4.1 command line since we changed that behavior too.

148

The AVX512 checks are missing here?

[craig.topper]

This should fix the scaliarization issue seen with ISD::XOR and ISD::OR

diff --git a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
index 377c608..92d1166 100644

• a/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

+++ b/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
@@ -4344,7 +4344,9 @@ SDValue DAGCombiner::hoistLogicOpWithSameOpcodeHands(SDNode *N) {

if ((HandOpcode == ISD::BITCAST || HandOpcode == ISD::SCALAR_TO_VECTOR) &&
     Level <= AfterLegalizeTypes) {
  // Input types must be integer and the same.

• if (XVT.isInteger() && XVT == Y.getValueType()) {

+ if (XVT.isInteger() && XVT == Y.getValueType() &&
+ !(VT.isVector() && TLI.isTypeLegal(VT) &&
+ !XVT.isVector() && !TLI.isTypeLegal(XVT))) {

  SDValue Logic = DAG.getNode(LogicOpcode, DL, XVT, X, Y);
  return DAG.getNode(HandOpcode, DL, VT, Logic);
}

[davezarzycki]

I believe I've incorporated all of the feedback so far.

[davezarzycki]

Hi @craig.topper – Ping. Is this ready to land? Any other requests?

[RKSimon]

lib/Target/X86/X86ISelLowering.cpp

42604

SDValue Mask = DAG.getAllOnesConstant(DL, MVT::i64)

42607

DAG.getAllOnesConstant(DL, MVT::i16)

[davezarzycki]

Now using DAG.getAllOnesConstant()

[craig.topper]

LGTM

[davezarzycki]

r374922

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::hoistLogicOpWithSameOpcodeHands(SDNode *N) {
   // LegalizeVectorOprs. LegalizeVectorOprs promotes vector operations by
   // adding bitcasts. For example (xor v4i32) is promoted to (v2i64), and
   // we don't want to undo this promotion.
   // We also handle SCALAR_TO_VECTOR because xor/or/and operations are cheaper
   // on scalars.
   if ((HandOpcode == ISD::BITCAST || HandOpcode == ISD::SCALAR_TO_VECTOR) &&
        Level <= AfterLegalizeTypes) {
     // Input types must be integer and the same.
-    if (XVT.isInteger() && XVT == Y.getValueType()) {
+    if (XVT.isInteger() && XVT == Y.getValueType() &&
+        !(VT.isVector() && TLI.isTypeLegal(VT) &&
+          !XVT.isVector() && !TLI.isTypeLegal(XVT))) {
       SDValue Logic = DAG.getNode(LogicOpcode, DL, XVT, X, Y);
       return DAG.getNode(HandOpcode, DL, VT, Logic);
     }
   }
 
   // Xor/and/or are indifferent to the swizzle operation (shuffle of one value).
   // Simplify xor/and/or (shuff(A), shuff(B)) -> shuff(op (A,B))
   // If both shuffles use the same mask, and both shuffle within a single

lib/Target/X86/X86ISelLowering.cpp

SDValue X86TargetLowering::emitFlagsForSetcc(SDValue Op0, SDValue Op1,
   // Try to use PTEST for a tree ORs equality compared with 0.
   // TODO: We could do AND tree with all 1s as well by using the C flag.
   if (Op0.getOpcode() == ISD::OR && isNullConstant(Op1) &&
       (CC == ISD::SETEQ || CC == ISD::SETNE)) {
     if (SDValue PTEST = LowerVectorAllZeroTest(Op0, CC, Subtarget, DAG, X86CC))
       return PTEST;
   }
 
   // Try to lower using KORTEST.
   if (SDValue KORTEST = EmitKORTEST(Op0, Op1, CC, dl, DAG, Subtarget, X86CC))
     return KORTEST;
 
   // Look for X == 0, X == 1, X != 0, or X != 1.  We can simplify some forms of
   // these.
   if ((isOneConstant(Op1) || isNullConstant(Op1)) &&
       (CC == ISD::SETEQ || CC == ISD::SETNE)) {
     // If the input is a setcc, then reuse the input setcc or use a new one with
auto IsVectorBitCastCheap = [](SDValue X) {
     X = peekThroughBitcasts(X);
     return isa<ConstantSDNode>(X) || X.getValueType().isVector() ||
            X.getOpcode() == ISD::LOAD;
   };
   if ((!IsVectorBitCastCheap(X) || !IsVectorBitCastCheap(Y)) &&
       !IsOrXorXorCCZero)
     return SDValue();
 
-  // TODO: Use PXOR + PTEST for SSE4.1 or later?
   EVT VT = SetCC->getValueType(0);
   SDLoc DL(SetCC);
+  bool HasAVX = Subtarget.hasAVX();
+
+  // Use XOR (plus OR) and PTEST after SSE4.1 and before AVX512.
+  // Otherwise use PCMPEQ (plus AND) and mask testing.
   if ((OpSize == 128 && Subtarget.hasSSE2()) ||
-      (OpSize == 256 && Subtarget.hasAVX2()) ||
+      (OpSize == 256 && HasAVX) ||
       (OpSize == 512 && Subtarget.useAVX512Regs())) {
-    auto BW = Subtarget.hasBWI();
-    EVT VecVT = OpSize == 512 ? (BW ? MVT::v64i8 : MVT::v16i32) :
-                OpSize == 256 ? MVT::v32i8 :
-                                MVT::v16i8;
-    EVT CmpVT = OpSize == 512 ? (BW ? MVT::v64i1 : MVT::v16i1) : VecVT;
+    bool HasPT = Subtarget.hasSSE41();
+    EVT VecVT = MVT::v16i8;
+    EVT CmpVT = MVT::v16i8;
+    if (OpSize == 256)
+      VecVT = CmpVT = MVT::v32i8;
+    if (OpSize == 512) {
+      if (Subtarget.hasBWI()) {
+        VecVT = MVT::v64i8;
+        CmpVT = MVT::v64i1;
+      } else {
+        VecVT = MVT::v16i32;
+        CmpVT = MVT::v16i1;
+      }
+    }
 
     SDValue Cmp;
     if (IsOrXorXorCCZero) {
       // This is a bitwise-combined equality comparison of 2 pairs of vectors:
       // setcc i128 (or (xor A, B), (xor C, D)), 0, eq|ne
       // Use 2 vector equality compares and 'and' the results before doing a
       // MOVMSK.
       SDValue A = DAG.getBitcast(VecVT, X.getOperand(0).getOperand(0));
       SDValue B = DAG.getBitcast(VecVT, X.getOperand(0).getOperand(1));
       SDValue C = DAG.getBitcast(VecVT, X.getOperand(1).getOperand(0));
       SDValue D = DAG.getBitcast(VecVT, X.getOperand(1).getOperand(1));
+      if (VecVT == CmpVT && HasPT) {
+        SDValue Cmp1 = DAG.getNode(ISD::XOR, DL, VecVT, A, B);
+        SDValue Cmp2 = DAG.getNode(ISD::XOR, DL, VecVT, C, D);
+        Cmp = DAG.getNode(ISD::OR, DL, VecVT, Cmp1, Cmp2);
+      } else {
         SDValue Cmp1 = DAG.getSetCC(DL, CmpVT, A, B, ISD::SETEQ);
         SDValue Cmp2 = DAG.getSetCC(DL, CmpVT, C, D, ISD::SETEQ);
         Cmp = DAG.getNode(ISD::AND, DL, CmpVT, Cmp1, Cmp2);
+      }
     } else {
       SDValue VecX = DAG.getBitcast(VecVT, X);
       SDValue VecY = DAG.getBitcast(VecVT, Y);
+      if (VecVT == CmpVT && HasPT) {
+        Cmp = DAG.getNode(ISD::XOR, DL, VecVT, VecX, VecY);
+      } else {
         Cmp = DAG.getSetCC(DL, CmpVT, VecX, VecY, ISD::SETEQ);
       }
+    }
     // For 512-bits we want to emit a setcc that will lower to kortest.
-    if (OpSize == 512 && BW)
-      return DAG.getSetCC(DL, VT, DAG.getBitcast(MVT::i64, Cmp),
-                          DAG.getConstant(0xFFFFFFFFFFFFFFFF, DL, MVT::i64), CC);
-    if (OpSize == 512)
-      return DAG.getSetCC(DL, VT, DAG.getBitcast(MVT::i16, Cmp),
-                          DAG.getConstant(0xFFFF, DL, MVT::i16), CC);
+    if (VecVT != CmpVT) {
+      EVT KRegVT = CmpVT == MVT::v64i1 ? MVT::i64 : MVT::i16;
+      SDValue Mask = DAG.getAllOnesConstant(DL, KRegVT);
+      return DAG.getSetCC(DL, VT, DAG.getBitcast(KRegVT, Cmp), Mask, CC);
+    }
+    if (HasPT) {
+      SDValue BCCmp = DAG.getBitcast(OpSize == 256 ? MVT::v4i64 : MVT::v2i64,
+                                     Cmp);
+      SDValue PT = DAG.getNode(X86ISD::PTEST, DL, MVT::i32, BCCmp, BCCmp);
+      X86::CondCode X86CC = CC == ISD::SETEQ ? X86::COND_E : X86::COND_NE;
+      SDValue SetCC = getSETCC(X86CC, PT, DL, DAG);
+      return DAG.getNode(ISD::TRUNCATE, DL, VT, SetCC.getValue(0));
+    }
     // If all bytes match (bitmask is 0x(FFFF)FFFF), that's equality.
     // setcc i128 X, Y, eq --> setcc (pmovmskb (pcmpeqb X, Y)), 0xFFFF, eq
     // setcc i128 X, Y, ne --> setcc (pmovmskb (pcmpeqb X, Y)), 0xFFFF, ne
     // setcc i256 X, Y, eq --> setcc (vpmovmskb (vpcmpeqb X, Y)), 0xFFFFFFFF, eq
     // setcc i256 X, Y, ne --> setcc (vpmovmskb (vpcmpeqb X, Y)), 0xFFFFFFFF, ne
     SDValue MovMsk = DAG.getNode(X86ISD::MOVMSK, DL, MVT::i32, Cmp);
     SDValue FFFFs = DAG.getConstant(OpSize == 128 ? 0xFFFF : 0xFFFFFFFF, DL,
                                     MVT::i32);

test/CodeGen/X86/memcmp-minsize.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=cmov | FileCheck %s --check-prefix=X86 --check-prefix=X86-NOSSE
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefix=X86 --check-prefix=X86-SSE2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefix=X64 --check-prefix=X64-SSE2
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2 | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX --check-prefix=X64-AVX1
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2 | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX --check-prefix=X64-AVX2
 
 ; This tests codegen time inlining/optimization of memcmp
 ; rdar://6480398
 
 @.str = private constant [65 x i8] c"0123456789012345678901234567890123456789012345678901234567890123\00", align 1
 
 declare i32 @memcmp(i8*, i8*, i64)
 

 ; X64-SSE2-NEXT:    movdqu (%rsi), %xmm0
 ; X64-SSE2-NEXT:    movdqu (%rdi), %xmm1
 ; X64-SSE2-NEXT:    pcmpeqb %xmm0, %xmm1
 ; X64-SSE2-NEXT:    pmovmskb %xmm1, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
-; X64-AVX2-LABEL: length16_eq:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
-; X64-AVX2-NEXT:    setne %al
-; X64-AVX2-NEXT:    retq
+; X64-AVX-LABEL: length16_eq:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX-NEXT:    setne %al
+; X64-AVX-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 16) nounwind
   %cmp = icmp ne i32 %call, 0
   ret i1 %cmp
 }
 
 define i1 @length16_eq_const(i8* %X) nounwind minsize {
 ; X86-NOSSE-LABEL: length16_eq_const:
 ; X86-NOSSE:       # %bb.0:

 ; X64-SSE2:       # %bb.0:
 ; X64-SSE2-NEXT:    movdqu (%rdi), %xmm0
 ; X64-SSE2-NEXT:    pcmpeqb {{.*}}(%rip), %xmm0
 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
-; X64-AVX2-LABEL: length16_eq_const:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
-; X64-AVX2-NEXT:    sete %al
-; X64-AVX2-NEXT:    retq
+; X64-AVX-LABEL: length16_eq_const:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX-NEXT:    sete %al
+; X64-AVX-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 16) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 ; PR33914 - https://bugs.llvm.org/show_bug.cgi?id=33914
 
 define i32 @length24(i8* %X, i8* %Y) nounwind minsize {

 ; X64-SSE2-NEXT:    pushq $32
 ; X64-SSE2-NEXT:    popq %rdx
 ; X64-SSE2-NEXT:    callq memcmp
 ; X64-SSE2-NEXT:    testl %eax, %eax
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    popq %rcx
 ; X64-SSE2-NEXT:    retq
 ;
+; X64-AVX1-LABEL: length32_eq:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovups (%rdi), %ymm0
+; X64-AVX1-NEXT:    vxorps (%rsi), %ymm0, %ymm0
+; X64-AVX1-NEXT:    vptest %ymm0, %ymm
+; X64-AVX1-NEXT:    sete %al
+; X64-AVX1-NEXT:    vzeroupper
+; X64-AVX1-NEXT:    retq
+;
 ; X64-AVX2-LABEL: length32_eq:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm
 ; X64-AVX2-NEXT:    sete %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 32) nounwind
   %cmp = icmp eq i32 %call, 0
   ret i1 %cmp
 }
 

 ; X64-SSE2-NEXT:    popq %rdx
 ; X64-SSE2-NEXT:    movl $.L.str, %esi
 ; X64-SSE2-NEXT:    callq memcmp
 ; X64-SSE2-NEXT:    testl %eax, %eax
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    popq %rcx
 ; X64-SSE2-NEXT:    retq
 ;
+; X64-AVX1-LABEL: length32_eq_const:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovups (%rdi), %ymm0
+; X64-AVX1-NEXT:    vxorps {{.*}}(%rip), %ymm0, %ymm0
+; X64-AVX1-NEXT:    vptest %ymm0, %ymm0
+; X64-AVX1-NEXT:    setne %al
+; X64-AVX1-NEXT:    vzeroupper
+; X64-AVX1-NEXT:    retq
+;
 ; X64-AVX2-LABEL: length32_eq_const:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    setne %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 32) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 

test/CodeGen/X86/memcmp-optsize.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=cmov | FileCheck %s --check-prefix=X86 --check-prefix=X86-NOSSE
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefix=X86 --check-prefix=X86-SSE2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefix=X64 --check-prefix=X64-SSE2
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2 | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX --check-prefix=X64-AVX1
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2 | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX --check-prefix=X64-AVX2
 
 ; This tests codegen time inlining/optimization of memcmp
 ; rdar://6480398
 
 @.str = private constant [65 x i8] c"0123456789012345678901234567890123456789012345678901234567890123\00", align 1
 
 declare i32 @memcmp(i8*, i8*, i64)
 declare i32 @bcmp(i8*, i8*, i64)

 ; X64-SSE2-NEXT:    movdqu (%rdi), %xmm0
 ; X64-SSE2-NEXT:    movdqu (%rsi), %xmm1
 ; X64-SSE2-NEXT:    pcmpeqb %xmm0, %xmm1
 ; X64-SSE2-NEXT:    pmovmskb %xmm1, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
-; X64-AVX2-LABEL: length16_eq:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
-; X64-AVX2-NEXT:    setne %al
-; X64-AVX2-NEXT:    retq
+; X64-AVX-LABEL: length16_eq:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX-NEXT:    setne %al
+; X64-AVX-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 16) nounwind
   %cmp = icmp ne i32 %call, 0
   ret i1 %cmp
 }
 
 define i1 @length16_eq_const(i8* %X) nounwind optsize {
 ; X86-NOSSE-LABEL: length16_eq_const:
 ; X86-NOSSE:       # %bb.0:

 ; X64-SSE2:       # %bb.0:
 ; X64-SSE2-NEXT:    movdqu (%rdi), %xmm0
 ; X64-SSE2-NEXT:    pcmpeqb {{.*}}(%rip), %xmm0
 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
-; X64-AVX2-LABEL: length16_eq_const:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
-; X64-AVX2-NEXT:    sete %al
-; X64-AVX2-NEXT:    retq
+; X64-AVX-LABEL: length16_eq_const:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX-NEXT:    sete %al
+; X64-AVX-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 16) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 ; PR33914 - https://bugs.llvm.org/show_bug.cgi?id=33914
 
 define i32 @length24(i8* %X, i8* %Y) nounwind optsize {

 ; X64-SSE2-NEXT:    movq {{.*#+}} xmm2 = mem[0],zero
 ; X64-SSE2-NEXT:    pcmpeqb %xmm0, %xmm2
 ; X64-SSE2-NEXT:    pand %xmm1, %xmm2
 ; X64-SSE2-NEXT:    pmovmskb %xmm2, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
-; X64-AVX2-LABEL: length24_eq:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX2-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
-; X64-AVX2-NEXT:    vmovq {{.*#+}} xmm2 = mem[0],zero
-; X64-AVX2-NEXT:    vpcmpeqb %xmm2, %xmm1, %xmm1
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
-; X64-AVX2-NEXT:    sete %al
-; X64-AVX2-NEXT:    retq
+; X64-AVX-LABEL: length24_eq:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX-NEXT:    vmovq 16(%rdi), %xmm1
+; X64-AVX-NEXT:    vmovq 16(%rsi), %xmm2
+; X64-AVX-NEXT:    vpxor %xmm2, %xmm1, %xmm1
+; X64-AVX-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX-NEXT:    sete %al
+; X64-AVX-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 24) nounwind
   %cmp = icmp eq i32 %call, 0
   ret i1 %cmp
 }
 
 define i1 @length24_eq_const(i8* %X) nounwind optsize {
 ; X86-NOSSE-LABEL: length24_eq_const:
 ; X86-NOSSE:       # %bb.0:

 ; X64-SSE2-NEXT:    pcmpeqb {{.*}}(%rip), %xmm1
 ; X64-SSE2-NEXT:    pcmpeqb {{.*}}(%rip), %xmm0
 ; X64-SSE2-NEXT:    pand %xmm1, %xmm0
 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
-; X64-AVX2-LABEL: length24_eq_const:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX2-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm1, %xmm1
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX2-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
-; X64-AVX2-NEXT:    setne %al
-; X64-AVX2-NEXT:    retq
+; X64-AVX-LABEL: length24_eq_const:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX-NEXT:    vmovq 16(%rdi), %xmm1
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm1, %xmm1
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX-NEXT:    setne %al
+; X64-AVX-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 24) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length32(i8* %X, i8* %Y) nounwind optsize {
 ; X86-LABEL: length32:
 ; X86:       # %bb.0:

 ; X64-SSE2-NEXT:    movdqu 16(%rsi), %xmm0
 ; X64-SSE2-NEXT:    pcmpeqb %xmm1, %xmm0
 ; X64-SSE2-NEXT:    pand %xmm2, %xmm0
 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
+; X64-AVX1-LABEL: length32_eq:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX1-NEXT:    vmovdqu 16(%rdi), %xmm1
+; X64-AVX1-NEXT:    vpxor 16(%rsi), %xmm1, %xmm1
+; X64-AVX1-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX1-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX1-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX1-NEXT:    sete %al
+; X64-AVX1-NEXT:    retq
+;
 ; X64-AVX2-LABEL: length32_eq:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    sete %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 32) nounwind
   %cmp = icmp eq i32 %call, 0
   ret i1 %cmp
 }
 

 ; X64-SSE2-NEXT:    pcmpeqb {{.*}}(%rip), %xmm1
 ; X64-SSE2-NEXT:    pcmpeqb {{.*}}(%rip), %xmm0
 ; X64-SSE2-NEXT:    pand %xmm1, %xmm0
 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
+; X64-AVX1-LABEL: length32_eq_const:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    vmovdqu (%rdi), %xmm0
+; X64-AVX1-NEXT:    vmovdqu 16(%rdi), %xmm1
+; X64-AVX1-NEXT:    vpxor {{.*}}(%rip), %xmm1, %xmm1
+; X64-AVX1-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX1-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX1-NEXT:    vptest %xmm0, %xmm0
+; X64-AVX1-NEXT:    setne %al
+; X64-AVX1-NEXT:    retq
+;
 ; X64-AVX2-LABEL: length32_eq_const:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    setne %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 32) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 

 ; X64-SSE2-NEXT:    pushq %rax
 ; X64-SSE2-NEXT:    movl $64, %edx
 ; X64-SSE2-NEXT:    callq memcmp
 ; X64-SSE2-NEXT:    testl %eax, %eax
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    popq %rcx
 ; X64-SSE2-NEXT:    retq
 ;
+; X64-AVX1-LABEL: length64_eq:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    pushq %rax
+; X64-AVX1-NEXT:    movl $64, %edx
+; X64-AVX1-NEXT:    callq memcmp
+; X64-AVX1-NEXT:    testl %eax, %eax
+; X64-AVX1-NEXT:    setne %al
+; X64-AVX1-NEXT:    popq %rcx
+; X64-AVX1-NEXT:    retq
+;
 ; X64-AVX2-LABEL: length64_eq:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
 ; X64-AVX2-NEXT:    vmovdqu 32(%rdi), %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb 32(%rsi), %ymm1, %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor 32(%rsi), %ymm1, %ymm1
+; X64-AVX2-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    setne %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 64) nounwind
   %cmp = icmp ne i32 %call, 0
   ret i1 %cmp
 }
 

 ; X64-SSE2-NEXT:    movl $.L.str, %esi
 ; X64-SSE2-NEXT:    movl $64, %edx
 ; X64-SSE2-NEXT:    callq memcmp
 ; X64-SSE2-NEXT:    testl %eax, %eax
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    popq %rcx
 ; X64-SSE2-NEXT:    retq
 ;
+; X64-AVX1-LABEL: length64_eq_const:
+; X64-AVX1:       # %bb.0:
+; X64-AVX1-NEXT:    pushq %rax
+; X64-AVX1-NEXT:    movl $.L.str, %esi
+; X64-AVX1-NEXT:    movl $64, %edx
+; X64-AVX1-NEXT:    callq memcmp
+; X64-AVX1-NEXT:    testl %eax, %eax
+; X64-AVX1-NEXT:    sete %al
+; X64-AVX1-NEXT:    popq %rcx
+; X64-AVX1-NEXT:    retq
+;
 ; X64-AVX2-LABEL: length64_eq_const:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
 ; X64-AVX2-NEXT:    vmovdqu 32(%rdi), %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm1, %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm1, %ymm1
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    sete %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 64) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 

test/CodeGen/X86/memcmp.ll

 ; X64-SSE2-NEXT:    pmovmskb %xmm1, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX-LABEL: length16_eq:
 ; X64-AVX:       # %bb.0:
 ; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX-NEXT:    setne %al
 ; X64-AVX-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 16) nounwind
   %cmp = icmp ne i32 %call, 0
   ret i1 %cmp
 }
 
 define i1 @length16_eq_const(i8* %X) nounwind {

 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX-LABEL: length16_eq_const:
 ; X64-AVX:       # %bb.0:
 ; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm0, %xmm0
-; X64-AVX-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX-NEXT:    sete %al
 ; X64-AVX-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 16) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 ; PR33914 - https://bugs.llvm.org/show_bug.cgi?id=33914

 ; X64-SSE2-NEXT:    pmovmskb %xmm2, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX-LABEL: length24_eq:
 ; X64-AVX:       # %bb.0:
 ; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
-; X64-AVX-NEXT:    vmovq {{.*#+}} xmm2 = mem[0],zero
-; X64-AVX-NEXT:    vpcmpeqb %xmm2, %xmm1, %xmm1
-; X64-AVX-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX-NEXT:    vmovq 16(%rdi), %xmm1
+; X64-AVX-NEXT:    vmovq 16(%rsi), %xmm2
+; X64-AVX-NEXT:    vpxor %xmm2, %xmm1, %xmm1
+; X64-AVX-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX-NEXT:    sete %al
 ; X64-AVX-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 24) nounwind
   %cmp = icmp eq i32 %call, 0
   ret i1 %cmp
 }
 
 define i1 @length24_eq_const(i8* %X) nounwind {

 ; X64-SSE2-NEXT:    pmovmskb %xmm0, %eax
 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX-LABEL: length24_eq_const:
 ; X64-AVX:       # %bb.0:
 ; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
-; X64-AVX-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
-; X64-AVX-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm1, %xmm1
-; X64-AVX-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm0, %xmm0
-; X64-AVX-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX-NEXT:    vmovq 16(%rdi), %xmm1
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm1, %xmm1
+; X64-AVX-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX-NEXT:    setne %al
 ; X64-AVX-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 24) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length32(i8* %X, i8* %Y) nounwind {

 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: length32_eq:
 ; X64-AVX1:       # %bb.0:
 ; X64-AVX1-NEXT:    vmovdqu (%rdi), %xmm0
 ; X64-AVX1-NEXT:    vmovdqu 16(%rdi), %xmm1
-; X64-AVX1-NEXT:    vpcmpeqb 16(%rsi), %xmm1, %xmm1
-; X64-AVX1-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX1-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX1-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX1-NEXT:    vpxor 16(%rsi), %xmm1, %xmm1
+; X64-AVX1-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX1-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX1-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX1-NEXT:    sete %al
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: length32_eq:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor   (%rsi), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest  %ymm0, %ymm0
 ; X64-AVX2-NEXT:    sete %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 32) nounwind
   %cmp = icmp eq i32 %call, 0
   ret i1 %cmp
 }
 

 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    sete %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX-LABEL: length32_eq_prefer128:
 ; X64-AVX:       # %bb.0:
 ; X64-AVX-NEXT:    vmovdqu (%rdi), %xmm0
 ; X64-AVX-NEXT:    vmovdqu 16(%rdi), %xmm1
-; X64-AVX-NEXT:    vpcmpeqb 16(%rsi), %xmm1, %xmm1
-; X64-AVX-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; X64-AVX-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX-NEXT:    vpxor 16(%rsi), %xmm1, %xmm1
+; X64-AVX-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; X64-AVX-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX-NEXT:    sete %al
 ; X64-AVX-NEXT:    retq
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 32) nounwind
   %cmp = icmp eq i32 %call, 0
   ret i1 %cmp
 }
 
 define i1 @length32_eq_const(i8* %X) nounwind {

 ; X64-SSE2-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
 ; X64-SSE2-NEXT:    setne %al
 ; X64-SSE2-NEXT:    retq
 ;
 ; X64-AVX1-LABEL: length32_eq_const:
 ; X64-AVX1:       # %bb.0:
 ; X64-AVX1-NEXT:    vmovdqu (%rdi), %xmm0
 ; X64-AVX1-NEXT:    vmovdqu 16(%rdi), %xmm1
-; X64-AVX1-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm1, %xmm1
-; X64-AVX1-NEXT:    vpcmpeqb {{.*}}(%rip), %xmm0, %xmm0
-; X64-AVX1-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; X64-AVX1-NEXT:    vpmovmskb %xmm0, %eax
-; X64-AVX1-NEXT:    cmpl $65535, %eax # imm = 0xFFFF
+; X64-AVX1-NEXT:    vpxor {{.*}}(%rip), %xmm1, %xmm1
+; X64-AVX1-NEXT:    vpxor {{.*}}(%rip), %xmm0, %xmm0
+; X64-AVX1-NEXT:    vpor %xmm1, %xmm0, %xmm0
+; X64-AVX1-NEXT:    vptest %xmm0, %xmm0
 ; X64-AVX1-NEXT:    setne %al
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: length32_eq_const:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    setne %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* getelementptr inbounds ([65 x i8], [65 x i8]* @.str, i32 0, i32 0), i64 32) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 

 ; X64-AVX1-NEXT:    setne %al
 ; X64-AVX1-NEXT:    popq %rcx
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: length64_eq:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
 ; X64-AVX2-NEXT:    vmovdqu 32(%rdi), %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb 32(%rsi), %ymm1, %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor 32(%rsi), %ymm1, %ymm1
+; X64-AVX2-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    setne %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
 ;
 ; X64-AVX512F-LABEL: length64_eq:
 ; X64-AVX512F:       # %bb.0:
 ; X64-AVX512F-NEXT:    vmovdqu64 (%rdi), %zmm0
 ; X64-AVX512F-NEXT:    vpcmpeqd (%rsi), %zmm0, %k0

 ; X64-AVX1-NEXT:    sete %al
 ; X64-AVX1-NEXT:    popq %rcx
 ; X64-AVX1-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: length64_eq_const:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vmovdqu (%rdi), %ymm0
 ; X64-AVX2-NEXT:    vmovdqu 32(%rdi), %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm1, %ymm1
-; X64-AVX2-NEXT:    vpcmpeqb {{.*}}(%rip), %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; X64-AVX2-NEXT:    vpmovmskb %ymm0, %eax
-; X64-AVX2-NEXT:    cmpl $-1, %eax
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm1, %ymm1
+; X64-AVX2-NEXT:    vpxor {{.*}}(%rip), %ymm0, %ymm0
+; X64-AVX2-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; X64-AVX2-NEXT:    vptest %ymm0, %ymm0
 ; X64-AVX2-NEXT:    sete %al
 ; X64-AVX2-NEXT:    vzeroupper
 ; X64-AVX2-NEXT:    retq
 ;
 ; X64-AVX512F-LABEL: length64_eq_const:
 ; X64-AVX512F:       # %bb.0:
 ; X64-AVX512F-NEXT:    vmovdqu64 (%rdi), %zmm0
 ; X64-AVX512F-NEXT:    vpcmpeqd {{.*}}(%rip), %zmm0, %k0

test/CodeGen/X86/setcc-wide-types.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=sse2     | FileCheck %s --check-prefix=ANY --check-prefix=NO512 --check-prefix=SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=sse4.1   | FileCheck %s --check-prefix=ANY --check-prefix=NO512 --check-prefix=SSE41
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx      | FileCheck %s --check-prefix=ANY --check-prefix=NO512 --check-prefix=AVXANY --check-prefix=AVX1
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2     | FileCheck %s --check-prefix=ANY --check-prefix=NO512 --check-prefix=AVXANY --check-prefix=AVX256 --check-prefix=AVX2
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512f  | FileCheck %s --check-prefix=ANY --check-prefix=AVXANY --check-prefix=AVX256 --check-prefix=AVX512 --check-prefix=AVX512F
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512bw | FileCheck %s --check-prefix=ANY --check-prefix=AVXANY --check-prefix=AVX256 --check-prefix=AVX512 --check-prefix=AVX512BW
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2     | FileCheck %s --check-prefix=ANY --check-prefix=NO512 --check-prefix=AVXANY --check-prefix=AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512f  | FileCheck %s --check-prefix=ANY --check-prefix=AVXANY --check-prefix=AVX512 --check-prefix=AVX512F
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx512bw | FileCheck %s --check-prefix=ANY --check-prefix=AVXANY --check-prefix=AVX512 --check-prefix=AVX512BW
 
 ; Equality checks of 128/256-bit values can use PMOVMSK or PTEST to avoid scalarization.
 
 define i32 @ne_i128(<2 x i64> %x, <2 x i64> %y) {
 ; SSE2-LABEL: ne_i128:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pcmpeqb %xmm1, %xmm0
 ; SSE2-NEXT:    pmovmskb %xmm0, %ecx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: ne_i128:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pxor %xmm1, %xmm0
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    ptest %xmm0, %xmm0
+; SSE41-NEXT:    setne %al
+; SSE41-NEXT:    retq
+;
 ; AVXANY-LABEL: ne_i128:
 ; AVXANY:       # %bb.0:
-; AVXANY-NEXT:    vpcmpeqb %xmm1, %xmm0, %xmm0
-; AVXANY-NEXT:    vpmovmskb %xmm0, %ecx
+; AVXANY-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; AVXANY-NEXT:    xorl %eax, %eax
-; AVXANY-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
+; AVXANY-NEXT:    vptest %xmm0, %xmm0
 ; AVXANY-NEXT:    setne %al
 ; AVXANY-NEXT:    retq
   %bcx = bitcast <2 x i64> %x to i128
   %bcy = bitcast <2 x i64> %y to i128
   %cmp = icmp ne i128 %bcx, %bcy
   %zext = zext i1 %cmp to i32
   ret i32 %zext
 }
 
 define i32 @eq_i128(<2 x i64> %x, <2 x i64> %y) {
 ; SSE2-LABEL: eq_i128:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    pcmpeqb %xmm1, %xmm0
 ; SSE2-NEXT:    pmovmskb %xmm0, %ecx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
 ; SSE2-NEXT:    sete %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: eq_i128:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pxor %xmm1, %xmm0
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    ptest %xmm0, %xmm0
+; SSE41-NEXT:    sete %al
+; SSE41-NEXT:    retq
+;
 ; AVXANY-LABEL: eq_i128:
 ; AVXANY:       # %bb.0:
-; AVXANY-NEXT:    vpcmpeqb %xmm1, %xmm0, %xmm0
-; AVXANY-NEXT:    vpmovmskb %xmm0, %ecx
+; AVXANY-NEXT:    vpxor %xmm1, %xmm0, %xmm0
 ; AVXANY-NEXT:    xorl %eax, %eax
-; AVXANY-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
+; AVXANY-NEXT:    vptest %xmm0, %xmm0
 ; AVXANY-NEXT:    sete %al
 ; AVXANY-NEXT:    retq
   %bcx = bitcast <2 x i64> %x to i128
   %bcy = bitcast <2 x i64> %y to i128
   %cmp = icmp eq i128 %bcx, %bcy
   %zext = zext i1 %cmp to i32
   ret i32 %zext
 }

 ; SSE2-NEXT:    movq %xmm3, %rcx
 ; SSE2-NEXT:    xorq %r8, %rcx
 ; SSE2-NEXT:    orq %rax, %rcx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    orq %rsi, %rcx
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: ne_i256:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movq %xmm0, %rax
+; SSE41-NEXT:    movq %xmm1, %rcx
+; SSE41-NEXT:    pextrq $1, %xmm0, %rdx
+; SSE41-NEXT:    pextrq $1, %xmm1, %r8
+; SSE41-NEXT:    movq %xmm2, %rdi
+; SSE41-NEXT:    xorq %rax, %rdi
+; SSE41-NEXT:    movq %xmm3, %rsi
+; SSE41-NEXT:    xorq %rcx, %rsi
+; SSE41-NEXT:    orq %rdi, %rsi
+; SSE41-NEXT:    pextrq $1, %xmm2, %rax
+; SSE41-NEXT:    xorq %rdx, %rax
+; SSE41-NEXT:    pextrq $1, %xmm3, %rcx
+; SSE41-NEXT:    xorq %r8, %rcx
+; SSE41-NEXT:    orq %rax, %rcx
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    orq %rsi, %rcx
+; SSE41-NEXT:    setne %al
+; SSE41-NEXT:    retq
+;
 ; AVX1-LABEL: ne_i256:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vmovq %xmm0, %rax
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm2
-; AVX1-NEXT:    vmovq %xmm2, %rcx
-; AVX1-NEXT:    vpextrq $1, %xmm0, %rdx
-; AVX1-NEXT:    vpextrq $1, %xmm2, %r8
-; AVX1-NEXT:    vmovq %xmm1, %rdi
-; AVX1-NEXT:    xorq %rax, %rdi
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm0
-; AVX1-NEXT:    vmovq %xmm0, %rsi
-; AVX1-NEXT:    xorq %rcx, %rsi
-; AVX1-NEXT:    orq %rdi, %rsi
-; AVX1-NEXT:    vpextrq $1, %xmm1, %rax
-; AVX1-NEXT:    xorq %rdx, %rax
-; AVX1-NEXT:    vpextrq $1, %xmm0, %rcx
-; AVX1-NEXT:    xorq %r8, %rcx
-; AVX1-NEXT:    orq %rax, %rcx
+; AVX1-NEXT:    vxorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    xorl %eax, %eax
-; AVX1-NEXT:    orq %rsi, %rcx
+; AVX1-NEXT:    vptest %ymm0, %ymm0
 ; AVX1-NEXT:    setne %al
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
-; AVX256-LABEL: ne_i256:
-; AVX256:       # %bb.0:
-; AVX256-NEXT:    vpcmpeqb %ymm1, %ymm0, %ymm0
-; AVX256-NEXT:    vpmovmskb %ymm0, %ecx
-; AVX256-NEXT:    xorl %eax, %eax
-; AVX256-NEXT:    cmpl $-1, %ecx
-; AVX256-NEXT:    setne %al
-; AVX256-NEXT:    vzeroupper
-; AVX256-NEXT:    retq
+; AVX2-LABEL: ne_i256:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpxor %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    xorl %eax, %eax
+; AVX2-NEXT:    vptest %ymm0, %ymm0
+; AVX2-NEXT:    setne %al
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: ne_i256:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpxor %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    xorl %eax, %eax
+; AVX512-NEXT:    vptest %ymm0, %ymm0
+; AVX512-NEXT:    setne %al
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %bcx = bitcast <4 x i64> %x to i256
   %bcy = bitcast <4 x i64> %y to i256
   %cmp = icmp ne i256 %bcx, %bcy
   %zext = zext i1 %cmp to i32
   ret i32 %zext
 }
 
 define i32 @eq_i256(<4 x i64> %x, <4 x i64> %y) {
 ; SSE2-LABEL: eq_i256:

 ; SSE2-NEXT:    movq %xmm3, %rcx
 ; SSE2-NEXT:    xorq %r8, %rcx
 ; SSE2-NEXT:    orq %rax, %rcx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    orq %rsi, %rcx
 ; SSE2-NEXT:    sete %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: eq_i256:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movq %xmm0, %rax
+; SSE41-NEXT:    movq %xmm1, %rcx
+; SSE41-NEXT:    pextrq $1, %xmm0, %rdx
+; SSE41-NEXT:    pextrq $1, %xmm1, %r8
+; SSE41-NEXT:    movq %xmm2, %rdi
+; SSE41-NEXT:    xorq %rax, %rdi
+; SSE41-NEXT:    movq %xmm3, %rsi
+; SSE41-NEXT:    xorq %rcx, %rsi
+; SSE41-NEXT:    orq %rdi, %rsi
+; SSE41-NEXT:    pextrq $1, %xmm2, %rax
+; SSE41-NEXT:    xorq %rdx, %rax
+; SSE41-NEXT:    pextrq $1, %xmm3, %rcx
+; SSE41-NEXT:    xorq %r8, %rcx
+; SSE41-NEXT:    orq %rax, %rcx
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    orq %rsi, %rcx
+; SSE41-NEXT:    sete %al
+; SSE41-NEXT:    retq
+;
 ; AVX1-LABEL: eq_i256:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vmovq %xmm0, %rax
-; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm2
-; AVX1-NEXT:    vmovq %xmm2, %rcx
-; AVX1-NEXT:    vpextrq $1, %xmm0, %rdx
-; AVX1-NEXT:    vpextrq $1, %xmm2, %r8
-; AVX1-NEXT:    vmovq %xmm1, %rdi
-; AVX1-NEXT:    xorq %rax, %rdi
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm0
-; AVX1-NEXT:    vmovq %xmm0, %rsi
-; AVX1-NEXT:    xorq %rcx, %rsi
-; AVX1-NEXT:    orq %rdi, %rsi
-; AVX1-NEXT:    vpextrq $1, %xmm1, %rax
-; AVX1-NEXT:    xorq %rdx, %rax
-; AVX1-NEXT:    vpextrq $1, %xmm0, %rcx
-; AVX1-NEXT:    xorq %r8, %rcx
-; AVX1-NEXT:    orq %rax, %rcx
+; AVX1-NEXT:    vxorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    xorl %eax, %eax
-; AVX1-NEXT:    orq %rsi, %rcx
+; AVX1-NEXT:    vptest %ymm0, %ymm0
 ; AVX1-NEXT:    sete %al
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
-; AVX256-LABEL: eq_i256:
-; AVX256:       # %bb.0:
-; AVX256-NEXT:    vpcmpeqb %ymm1, %ymm0, %ymm0
-; AVX256-NEXT:    vpmovmskb %ymm0, %ecx
-; AVX256-NEXT:    xorl %eax, %eax
-; AVX256-NEXT:    cmpl $-1, %ecx
-; AVX256-NEXT:    sete %al
-; AVX256-NEXT:    vzeroupper
-; AVX256-NEXT:    retq
+; AVX2-LABEL: eq_i256:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpxor %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    xorl %eax, %eax
+; AVX2-NEXT:    vptest %ymm0, %ymm0
+; AVX2-NEXT:    sete %al
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: eq_i256:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpxor %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    xorl %eax, %eax
+; AVX512-NEXT:    vptest %ymm0, %ymm0
+; AVX512-NEXT:    sete %al
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %bcx = bitcast <4 x i64> %x to i256
   %bcy = bitcast <4 x i64> %y to i256
   %cmp = icmp eq i256 %bcx, %bcy
   %zext = zext i1 %cmp to i32
   ret i32 %zext
 }
 
 define i32 @ne_i512(<8 x i64> %x, <8 x i64> %y) {

 ; SSE2-NEXT:    orq %rsi, %rdi
 ; SSE2-NEXT:    orq %rcx, %rdi
 ; SSE2-NEXT:    orq %rax, %rdi
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    orq %rdx, %rdi
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: ne_i512:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movq %xmm0, %rax
+; SSE41-NEXT:    movq %xmm2, %rcx
+; SSE41-NEXT:    movq %xmm1, %rdx
+; SSE41-NEXT:    movq %xmm3, %rsi
+; SSE41-NEXT:    pextrq $1, %xmm0, %r11
+; SSE41-NEXT:    pextrq $1, %xmm2, %r8
+; SSE41-NEXT:    pextrq $1, %xmm1, %r9
+; SSE41-NEXT:    pextrq $1, %xmm3, %r10
+; SSE41-NEXT:    movq %xmm4, %rdi
+; SSE41-NEXT:    xorq %rax, %rdi
+; SSE41-NEXT:    movq %xmm6, %rax
+; SSE41-NEXT:    xorq %rcx, %rax
+; SSE41-NEXT:    movq %xmm5, %rcx
+; SSE41-NEXT:    xorq %rdx, %rcx
+; SSE41-NEXT:    movq %xmm7, %rdx
+; SSE41-NEXT:    xorq %rsi, %rdx
+; SSE41-NEXT:    orq %rcx, %rdx
+; SSE41-NEXT:    orq %rax, %rdx
+; SSE41-NEXT:    orq %rdi, %rdx
+; SSE41-NEXT:    pextrq $1, %xmm4, %rax
+; SSE41-NEXT:    xorq %r11, %rax
+; SSE41-NEXT:    pextrq $1, %xmm6, %rcx
+; SSE41-NEXT:    xorq %r8, %rcx
+; SSE41-NEXT:    pextrq $1, %xmm5, %rsi
+; SSE41-NEXT:    xorq %r9, %rsi
+; SSE41-NEXT:    pextrq $1, %xmm7, %rdi
+; SSE41-NEXT:    xorq %r10, %rdi
+; SSE41-NEXT:    orq %rsi, %rdi
+; SSE41-NEXT:    orq %rcx, %rdi
+; SSE41-NEXT:    orq %rax, %rdi
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    orq %rdx, %rdi
+; SSE41-NEXT:    setne %al
+; SSE41-NEXT:    retq
+;
 ; AVX1-LABEL: ne_i512:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vmovq %xmm1, %rcx
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vmovq %xmm4, %rdx
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm5
 ; AVX1-NEXT:    vmovq %xmm5, %rsi

 ; SSE2-NEXT:    orq %rsi, %rdi
 ; SSE2-NEXT:    orq %rcx, %rdi
 ; SSE2-NEXT:    orq %rax, %rdi
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    orq %rdx, %rdi
 ; SSE2-NEXT:    sete %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: eq_i512:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movq %xmm0, %rax
+; SSE41-NEXT:    movq %xmm2, %rcx
+; SSE41-NEXT:    movq %xmm1, %rdx
+; SSE41-NEXT:    movq %xmm3, %rsi
+; SSE41-NEXT:    pextrq $1, %xmm0, %r11
+; SSE41-NEXT:    pextrq $1, %xmm2, %r8
+; SSE41-NEXT:    pextrq $1, %xmm1, %r9
+; SSE41-NEXT:    pextrq $1, %xmm3, %r10
+; SSE41-NEXT:    movq %xmm4, %rdi
+; SSE41-NEXT:    xorq %rax, %rdi
+; SSE41-NEXT:    movq %xmm6, %rax
+; SSE41-NEXT:    xorq %rcx, %rax
+; SSE41-NEXT:    movq %xmm5, %rcx
+; SSE41-NEXT:    xorq %rdx, %rcx
+; SSE41-NEXT:    movq %xmm7, %rdx
+; SSE41-NEXT:    xorq %rsi, %rdx
+; SSE41-NEXT:    orq %rcx, %rdx
+; SSE41-NEXT:    orq %rax, %rdx
+; SSE41-NEXT:    orq %rdi, %rdx
+; SSE41-NEXT:    pextrq $1, %xmm4, %rax
+; SSE41-NEXT:    xorq %r11, %rax
+; SSE41-NEXT:    pextrq $1, %xmm6, %rcx
+; SSE41-NEXT:    xorq %r8, %rcx
+; SSE41-NEXT:    pextrq $1, %xmm5, %rsi
+; SSE41-NEXT:    xorq %r9, %rsi
+; SSE41-NEXT:    pextrq $1, %xmm7, %rdi
+; SSE41-NEXT:    xorq %r10, %rdi
+; SSE41-NEXT:    orq %rsi, %rdi
+; SSE41-NEXT:    orq %rcx, %rdi
+; SSE41-NEXT:    orq %rax, %rdi
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    orq %rdx, %rdi
+; SSE41-NEXT:    sete %al
+; SSE41-NEXT:    retq
+;
 ; AVX1-LABEL: eq_i512:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vmovq %xmm0, %rax
 ; AVX1-NEXT:    vmovq %xmm1, %rcx
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vmovq %xmm4, %rdx
 ; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm5
 ; AVX1-NEXT:    vmovq %xmm5, %rsi

 ; SSE2-NEXT:    pcmpeqb %xmm1, %xmm0
 ; SSE2-NEXT:    pand %xmm2, %xmm0
 ; SSE2-NEXT:    pmovmskb %xmm0, %ecx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: ne_i128_pair:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movdqu (%rdi), %xmm0
+; SSE41-NEXT:    movdqu 16(%rdi), %xmm1
+; SSE41-NEXT:    movdqu (%rsi), %xmm2
+; SSE41-NEXT:    pxor %xmm0, %xmm2
+; SSE41-NEXT:    movdqu 16(%rsi), %xmm0
+; SSE41-NEXT:    pxor %xmm1, %xmm0
+; SSE41-NEXT:    por %xmm2, %xmm0
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    ptest %xmm0, %xmm0
+; SSE41-NEXT:    setne %al
+; SSE41-NEXT:    retq
+;
 ; AVXANY-LABEL: ne_i128_pair:
 ; AVXANY:       # %bb.0:
 ; AVXANY-NEXT:    vmovdqu (%rdi), %xmm0
 ; AVXANY-NEXT:    vmovdqu 16(%rdi), %xmm1
-; AVXANY-NEXT:    vpcmpeqb 16(%rsi), %xmm1, %xmm1
-; AVXANY-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; AVXANY-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; AVXANY-NEXT:    vpmovmskb %xmm0, %ecx
+; AVXANY-NEXT:    vpxor 16(%rsi), %xmm1, %xmm1
+; AVXANY-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; AVXANY-NEXT:    vpor %xmm1, %xmm0, %xmm0
 ; AVXANY-NEXT:    xorl %eax, %eax
-; AVXANY-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
+; AVXANY-NEXT:    vptest %xmm0, %xmm0
 ; AVXANY-NEXT:    setne %al
 ; AVXANY-NEXT:    retq
   %a0 = load i128, i128* %a
   %b0 = load i128, i128* %b
   %xor1 = xor i128 %a0, %b0
   %ap1 = getelementptr i128, i128* %a, i128 1
   %bp1 = getelementptr i128, i128* %b, i128 1
   %a1 = load i128, i128* %ap1

 ; SSE2-NEXT:    pcmpeqb %xmm1, %xmm0
 ; SSE2-NEXT:    pand %xmm2, %xmm0
 ; SSE2-NEXT:    pmovmskb %xmm0, %ecx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
 ; SSE2-NEXT:    sete %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: eq_i128_pair:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movdqu (%rdi), %xmm0
+; SSE41-NEXT:    movdqu 16(%rdi), %xmm1
+; SSE41-NEXT:    movdqu (%rsi), %xmm2
+; SSE41-NEXT:    pxor %xmm0, %xmm2
+; SSE41-NEXT:    movdqu 16(%rsi), %xmm0
+; SSE41-NEXT:    pxor %xmm1, %xmm0
+; SSE41-NEXT:    por %xmm2, %xmm0
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    ptest %xmm0, %xmm0
+; SSE41-NEXT:    sete %al
+; SSE41-NEXT:    retq
+;
 ; AVXANY-LABEL: eq_i128_pair:
 ; AVXANY:       # %bb.0:
 ; AVXANY-NEXT:    vmovdqu (%rdi), %xmm0
 ; AVXANY-NEXT:    vmovdqu 16(%rdi), %xmm1
-; AVXANY-NEXT:    vpcmpeqb 16(%rsi), %xmm1, %xmm1
-; AVXANY-NEXT:    vpcmpeqb (%rsi), %xmm0, %xmm0
-; AVXANY-NEXT:    vpand %xmm1, %xmm0, %xmm0
-; AVXANY-NEXT:    vpmovmskb %xmm0, %ecx
+; AVXANY-NEXT:    vpxor 16(%rsi), %xmm1, %xmm1
+; AVXANY-NEXT:    vpxor (%rsi), %xmm0, %xmm0
+; AVXANY-NEXT:    vpor %xmm1, %xmm0, %xmm0
 ; AVXANY-NEXT:    xorl %eax, %eax
-; AVXANY-NEXT:    cmpl $65535, %ecx # imm = 0xFFFF
+; AVXANY-NEXT:    vptest %xmm0, %xmm0
 ; AVXANY-NEXT:    sete %al
 ; AVXANY-NEXT:    retq
   %a0 = load i128, i128* %a
   %b0 = load i128, i128* %b
   %xor1 = xor i128 %a0, %b0
   %ap1 = getelementptr i128, i128* %a, i128 1
   %bp1 = getelementptr i128, i128* %b, i128 1
   %a1 = load i128, i128* %ap1

 ; SSE2-NEXT:    orq %r9, %rdx
 ; SSE2-NEXT:    orq %rax, %rdx
 ; SSE2-NEXT:    orq %r8, %rdx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    orq %rcx, %rdx
 ; SSE2-NEXT:    setne %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: ne_i256_pair:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movq 16(%rdi), %r9
+; SSE41-NEXT:    movq 24(%rdi), %r11
+; SSE41-NEXT:    movq (%rdi), %r8
+; SSE41-NEXT:    movq 8(%rdi), %r10
+; SSE41-NEXT:    xorq 8(%rsi), %r10
+; SSE41-NEXT:    xorq 24(%rsi), %r11
+; SSE41-NEXT:    xorq (%rsi), %r8
+; SSE41-NEXT:    xorq 16(%rsi), %r9
+; SSE41-NEXT:    movq 48(%rdi), %rdx
+; SSE41-NEXT:    movq 32(%rdi), %rax
+; SSE41-NEXT:    movq 56(%rdi), %rcx
+; SSE41-NEXT:    movq 40(%rdi), %rdi
+; SSE41-NEXT:    xorq 40(%rsi), %rdi
+; SSE41-NEXT:    xorq 56(%rsi), %rcx
+; SSE41-NEXT:    orq %r11, %rcx
+; SSE41-NEXT:    orq %rdi, %rcx
+; SSE41-NEXT:    orq %r10, %rcx
+; SSE41-NEXT:    xorq 32(%rsi), %rax
+; SSE41-NEXT:    xorq 48(%rsi), %rdx
+; SSE41-NEXT:    orq %r9, %rdx
+; SSE41-NEXT:    orq %rax, %rdx
+; SSE41-NEXT:    orq %r8, %rdx
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    orq %rcx, %rdx
+; SSE41-NEXT:    setne %al
+; SSE41-NEXT:    retq
+;
 ; AVX1-LABEL: ne_i256_pair:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    movq 16(%rdi), %r9
-; AVX1-NEXT:    movq 24(%rdi), %r11
-; AVX1-NEXT:    movq (%rdi), %r8
-; AVX1-NEXT:    movq 8(%rdi), %r10
-; AVX1-NEXT:    xorq 8(%rsi), %r10
-; AVX1-NEXT:    xorq 24(%rsi), %r11
-; AVX1-NEXT:    xorq (%rsi), %r8
-; AVX1-NEXT:    xorq 16(%rsi), %r9
-; AVX1-NEXT:    movq 48(%rdi), %rdx
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq 56(%rdi), %rcx
-; AVX1-NEXT:    movq 40(%rdi), %rdi
-; AVX1-NEXT:    xorq 40(%rsi), %rdi
-; AVX1-NEXT:    xorq 56(%rsi), %rcx
-; AVX1-NEXT:    orq %r11, %rcx
-; AVX1-NEXT:    orq %rdi, %rcx
-; AVX1-NEXT:    orq %r10, %rcx
-; AVX1-NEXT:    xorq 32(%rsi), %rax
-; AVX1-NEXT:    xorq 48(%rsi), %rdx
-; AVX1-NEXT:    orq %r9, %rdx
-; AVX1-NEXT:    orq %rax, %rdx
-; AVX1-NEXT:    orq %r8, %rdx
+; AVX1-NEXT:    vmovups (%rdi), %ymm0
+; AVX1-NEXT:    vmovups 32(%rdi), %ymm1
+; AVX1-NEXT:    vxorps 32(%rsi), %ymm1, %ymm1
+; AVX1-NEXT:    vxorps (%rsi), %ymm0, %ymm0
+; AVX1-NEXT:    vorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    xorl %eax, %eax
-; AVX1-NEXT:    orq %rcx, %rdx
+; AVX1-NEXT:    vptest %ymm0, %ymm0
 ; AVX1-NEXT:    setne %al
+; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
-; AVX256-LABEL: ne_i256_pair:
-; AVX256:       # %bb.0:
-; AVX256-NEXT:    vmovdqu (%rdi), %ymm0
-; AVX256-NEXT:    vmovdqu 32(%rdi), %ymm1
-; AVX256-NEXT:    vpcmpeqb 32(%rsi), %ymm1, %ymm1
-; AVX256-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; AVX256-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; AVX256-NEXT:    vpmovmskb %ymm0, %ecx
-; AVX256-NEXT:    xorl %eax, %eax
-; AVX256-NEXT:    cmpl $-1, %ecx
-; AVX256-NEXT:    setne %al
-; AVX256-NEXT:    vzeroupper
-; AVX256-NEXT:    retq
+; AVX2-LABEL: ne_i256_pair:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovdqu (%rdi), %ymm0
+; AVX2-NEXT:    vmovdqu 32(%rdi), %ymm1
+; AVX2-NEXT:    vpxor 32(%rsi), %ymm1, %ymm1
+; AVX2-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; AVX2-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    xorl %eax, %eax
+; AVX2-NEXT:    vptest %ymm0, %ymm0
+; AVX2-NEXT:    setne %al
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: ne_i256_pair:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovdqu (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqu 32(%rdi), %ymm1
+; AVX512-NEXT:    vpxor 32(%rsi), %ymm1, %ymm1
+; AVX512-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; AVX512-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    xorl %eax, %eax
+; AVX512-NEXT:    vptest %ymm0, %ymm0
+; AVX512-NEXT:    setne %al
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %a0 = load i256, i256* %a
   %b0 = load i256, i256* %b
   %xor1 = xor i256 %a0, %b0
   %ap1 = getelementptr i256, i256* %a, i256 1
   %bp1 = getelementptr i256, i256* %b, i256 1
   %a1 = load i256, i256* %ap1
   %b1 = load i256, i256* %bp1
   %xor2 = xor i256 %a1, %b1

 ; SSE2-NEXT:    orq %r9, %rdx
 ; SSE2-NEXT:    orq %rax, %rdx
 ; SSE2-NEXT:    orq %r8, %rdx
 ; SSE2-NEXT:    xorl %eax, %eax
 ; SSE2-NEXT:    orq %rcx, %rdx
 ; SSE2-NEXT:    sete %al
 ; SSE2-NEXT:    retq
 ;
+; SSE41-LABEL: eq_i256_pair:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movq 16(%rdi), %r9
+; SSE41-NEXT:    movq 24(%rdi), %r11
+; SSE41-NEXT:    movq (%rdi), %r8
+; SSE41-NEXT:    movq 8(%rdi), %r10
+; SSE41-NEXT:    xorq 8(%rsi), %r10
+; SSE41-NEXT:    xorq 24(%rsi), %r11
+; SSE41-NEXT:    xorq (%rsi), %r8
+; SSE41-NEXT:    xorq 16(%rsi), %r9
+; SSE41-NEXT:    movq 48(%rdi), %rdx
+; SSE41-NEXT:    movq 32(%rdi), %rax
+; SSE41-NEXT:    movq 56(%rdi), %rcx
+; SSE41-NEXT:    movq 40(%rdi), %rdi
+; SSE41-NEXT:    xorq 40(%rsi), %rdi
+; SSE41-NEXT:    xorq 56(%rsi), %rcx
+; SSE41-NEXT:    orq %r11, %rcx
+; SSE41-NEXT:    orq %rdi, %rcx
+; SSE41-NEXT:    orq %r10, %rcx
+; SSE41-NEXT:    xorq 32(%rsi), %rax
+; SSE41-NEXT:    xorq 48(%rsi), %rdx
+; SSE41-NEXT:    orq %r9, %rdx
+; SSE41-NEXT:    orq %rax, %rdx
+; SSE41-NEXT:    orq %r8, %rdx
+; SSE41-NEXT:    xorl %eax, %eax
+; SSE41-NEXT:    orq %rcx, %rdx
+; SSE41-NEXT:    sete %al
+; SSE41-NEXT:    retq
+;
 ; AVX1-LABEL: eq_i256_pair:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    movq 16(%rdi), %r9
-; AVX1-NEXT:    movq 24(%rdi), %r11
-; AVX1-NEXT:    movq (%rdi), %r8
-; AVX1-NEXT:    movq 8(%rdi), %r10
-; AVX1-NEXT:    xorq 8(%rsi), %r10
-; AVX1-NEXT:    xorq 24(%rsi), %r11
-; AVX1-NEXT:    xorq (%rsi), %r8
-; AVX1-NEXT:    xorq 16(%rsi), %r9
-; AVX1-NEXT:    movq 48(%rdi), %rdx
-; AVX1-NEXT:    movq 32(%rdi), %rax
-; AVX1-NEXT:    movq 56(%rdi), %rcx
-; AVX1-NEXT:    movq 40(%rdi), %rdi
-; AVX1-NEXT:    xorq 40(%rsi), %rdi
-; AVX1-NEXT:    xorq 56(%rsi), %rcx
-; AVX1-NEXT:    orq %r11, %rcx
-; AVX1-NEXT:    orq %rdi, %rcx
-; AVX1-NEXT:    orq %r10, %rcx
-; AVX1-NEXT:    xorq 32(%rsi), %rax
-; AVX1-NEXT:    xorq 48(%rsi), %rdx
-; AVX1-NEXT:    orq %r9, %rdx
-; AVX1-NEXT:    orq %rax, %rdx
-; AVX1-NEXT:    orq %r8, %rdx
+; AVX1-NEXT:    vmovups (%rdi), %ymm0
+; AVX1-NEXT:    vmovups 32(%rdi), %ymm1
+; AVX1-NEXT:    vxorps 32(%rsi), %ymm1, %ymm1
+; AVX1-NEXT:    vxorps (%rsi), %ymm0, %ymm0
+; AVX1-NEXT:    vorps %ymm1, %ymm0, %ymm0
 ; AVX1-NEXT:    xorl %eax, %eax
-; AVX1-NEXT:    orq %rcx, %rdx
+; AVX1-NEXT:    vptest %ymm0, %ymm0
 ; AVX1-NEXT:    sete %al
+; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
-; AVX256-LABEL: eq_i256_pair:
-; AVX256:       # %bb.0:
-; AVX256-NEXT:    vmovdqu (%rdi), %ymm0
-; AVX256-NEXT:    vmovdqu 32(%rdi), %ymm1
-; AVX256-NEXT:    vpcmpeqb 32(%rsi), %ymm1, %ymm1
-; AVX256-NEXT:    vpcmpeqb (%rsi), %ymm0, %ymm0
-; AVX256-NEXT:    vpand %ymm1, %ymm0, %ymm0
-; AVX256-NEXT:    vpmovmskb %ymm0, %ecx
-; AVX256-NEXT:    xorl %eax, %eax
-; AVX256-NEXT:    cmpl $-1, %ecx
-; AVX256-NEXT:    sete %al
-; AVX256-NEXT:    vzeroupper
-; AVX256-NEXT:    retq
+; AVX2-LABEL: eq_i256_pair:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovdqu (%rdi), %ymm0
+; AVX2-NEXT:    vmovdqu 32(%rdi), %ymm1
+; AVX2-NEXT:    vpxor 32(%rsi), %ymm1, %ymm1
+; AVX2-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; AVX2-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    xorl %eax, %eax
+; AVX2-NEXT:    vptest %ymm0, %ymm0
+; AVX2-NEXT:    sete %al
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: eq_i256_pair:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovdqu (%rdi), %ymm0
+; AVX512-NEXT:    vmovdqu 32(%rdi), %ymm1
+; AVX512-NEXT:    vpxor 32(%rsi), %ymm1, %ymm1
+; AVX512-NEXT:    vpxor (%rsi), %ymm0, %ymm0
+; AVX512-NEXT:    vpor %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    xorl %eax, %eax
+; AVX512-NEXT:    vptest %ymm0, %ymm0
+; AVX512-NEXT:    sete %al
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
   %a0 = load i256, i256* %a
   %b0 = load i256, i256* %b
   %xor1 = xor i256 %a0, %b0
   %ap1 = getelementptr i256, i256* %a, i256 1
   %bp1 = getelementptr i256, i256* %b, i256 1
   %a1 = load i256, i256* %ap1
   %b1 = load i256, i256* %bp1
   %xor2 = xor i256 %a1, %b1