[CGP] use subtract of cmps for result of memcmp expansion

[spatel]

Summary

There are some regressions for PPC here (even if we keep the 'subf', the 'extsw' is unnecessary, isn't it?). I'm hoping we can address that as a follow-up rather than make this change dependent on fixing that.

As noted in the code comment, transforming this in the other direction might require a separate transform here in CGP given the block-at-a-time DAG constraint.

[efriedma]

test/CodeGen/X86/memcmp.ll

38 ↗

(On Diff #104912)

Potential alternative sequence: instead of xorl+cmpw+seta+sbbl, just generate a "subl", like we do for length 1.

[spatel]

test/CodeGen/X86/memcmp.ll

38 ↗

(On Diff #104912)

Ah, of course. I've been trying to find a way to use subtract in the larger cases, but this is straightforward for i8/i16. I'll update the patch.

[spatel]

Patch updated:
Bypass the compares for i8/i16:
http://rise4fun.com/Alive/7sl

[nemanjai]

We have patches that will eventually land that improve our handling of zero-extending i1 in the PPC back end so I'm not overly concern about the slightly poorer code here. However, I'd like to make sure I understand the reason for this change.

1. On targets that will expand selects to branches, we want to avoid introducing selects to reduce branching
2. Recreating a select in the DAG is possible and getting rid of the select in the DAG is impossible on targets that expand it to branches
3. Making this choice conditional on something like an IR-level version of canInsertSelect() (maybe something called shouldInsertSelect()...) introduces unnecessary complexity

Does that evaluation roughly cover the motivation for this patch?

test/CodeGen/PowerPC/memcmp.ll

15 ↗

(On Diff #105649)

I wouldn't worry about this sign extension. This is something that happens all over the PPC back end and we have a patch in the pipeline that will eliminate it.

[spatel]

In D34904#804655, @nemanjai wrote:

We have patches that will eventually land that improve our handling of zero-extending i1 in the PPC back end so I'm not overly concern about the slightly poorer code here. However, I'd like to make sure I understand the reason for this change.

1. On targets that will expand selects to branches, we want to avoid introducing selects to reduce branching
2. Recreating a select in the DAG is possible and getting rid of the select in the DAG is impossible on targets that expand it to branches
3. Making this choice conditional on something like an IR-level version of canInsertSelect() (maybe something called shouldInsertSelect()...) introduces unnecessary complexity

Does that evaluation roughly cover the motivation for this patch?

That's correct, although if pre-DAG select-to-branch transforms are a concern, then this is just a tiny example of that.

I admit that when I saw the branchy output on some of the x86 tests, I thought this was happening already, but it turns out those branches are created later. When I realized we could create the memcmp result with a subtract, however, I thought that would translate to better codegen for both PPC and x86 without introducing more DAG transforms.

It turns out we'll have to add those transforms anyway if the related llvm-dev thread ( http://lists.llvm.org/pipermail/llvm-dev/2017-July/114885.html ) continues in its current direction. So the practical motivation is that we're producing better code for memcmp() for x86 (and PPC if we avoid the isels?) without waiting for that extra DAG step to improve codegen.

[spatel]

I almost forgot another hidden motivation that I need to mention in case we ever come back to the idea for x86 or other targets try to hack this: I was looking to use vector-length types in this expansion on x86 for 16-byte and larger memcmp. Because of existing DAG transforms, the x86 codegen would become a strange mix of scalar and vector code with the select-based IR in this expansion...and that would have required several steps to undo. Using subtract-of-cmps avoided that. :)
See: https://bugs.llvm.org/show_bug.cgi?id=33329 for detailed analysis.

[RKSimon]

This makes sense to me if the PowerPC guys are happy.

[RKSimon]

In D34904#807911, @RKSimon wrote:

This makes sense to me if the PowerPC guys are happy.

@nemanjai @echristo @hfinkel Are you guys happy with this approach?

[nemanjai]

In D34904#815919, @RKSimon wrote:

In D34904#807911, @RKSimon wrote:

This makes sense to me if the PowerPC guys are happy.

@nemanjai @echristo @hfinkel Are you guys happy with this approach?

Seems perfectly fine to me. Let's wait and see if Eric or Hal have any further objections.

[nemanjai]

Since there have not been any concerns voiced, I think this patch can proceed.

llvm/trunk/lib/CodeGen/CodeGenPrepare.cpp

Value *MemCmpExpansion::getMemCmpOneBlock(unsigned Size) {
 
   if (DL.isLittleEndian() && Size != 1) {
     Function *Bswap = Intrinsic::getDeclaration(CI->getModule(),
                                                 Intrinsic::bswap, LoadSizeType);
     LoadSrc1 = Builder.CreateCall(Bswap, LoadSrc1);
     LoadSrc2 = Builder.CreateCall(Bswap, LoadSrc2);
   }
 
-  // TODO: Instead of comparing ULT, just subtract and return the difference?
-  Value *CmpNE = Builder.CreateICmpNE(LoadSrc1, LoadSrc2);
+  if (Size < 4) {
+    // The i8 and i16 cases don't need compares. We zext the loaded values and
+    // subtract them to get the suitable negative, zero, or positive i32 result.
+    LoadSrc1 = Builder.CreateZExt(LoadSrc1, Builder.getInt32Ty());
+    LoadSrc2 = Builder.CreateZExt(LoadSrc2, Builder.getInt32Ty());
+    return Builder.CreateSub(LoadSrc1, LoadSrc2);
+  }
+
+  // The result of memcmp is negative, zero, or positive, so produce that by
+  // subtracting 2 extended compare bits: sub (ugt, ult).
+  // If a target prefers to use selects to get -1/0/1, they should be able
+  // to transform this later. The inverse transform (going from selects to math)
+  // may not be possible in the DAG because the selects got converted into
+  // branches before we got there.
+  Value *CmpUGT = Builder.CreateICmpUGT(LoadSrc1, LoadSrc2);
   Value *CmpULT = Builder.CreateICmpULT(LoadSrc1, LoadSrc2);
-  Type *I32 = Builder.getInt32Ty();
-  Value *Sel1 = Builder.CreateSelect(CmpULT, ConstantInt::get(I32, -1),
-                                             ConstantInt::get(I32, 1));
-  return Builder.CreateSelect(CmpNE, Sel1, ConstantInt::get(I32, 0));
+  Value *ZextUGT = Builder.CreateZExt(CmpUGT, Builder.getInt32Ty());
+  Value *ZextULT = Builder.CreateZExt(CmpULT, Builder.getInt32Ty());
+  return Builder.CreateSub(ZextUGT, ZextULT);
 }
 
 // This function expands the memcmp call into an inline expansion and returns
 // the memcmp result.
 Value *MemCmpExpansion::getMemCmpExpansion(uint64_t Size) {
   if (IsUsedForZeroCmp)
     return NumBlocks == 1 ? getMemCmpEqZeroOneBlock(Size) :
                             getMemCmpExpansionZeroCase(Size);

llvm/trunk/test/CodeGen/PowerPC/memcmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mcpu=pwr8 -mtriple=powerpc64le-unknown-gnu-linux  < %s | FileCheck %s -check-prefix=CHECK
 
 define signext i32 @memcmp8(i32* nocapture readonly %buffer1, i32* nocapture readonly %buffer2) {
 ; CHECK-LABEL: memcmp8:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    ldbrx 3, 0, 3
 ; CHECK-NEXT:    ldbrx 4, 0, 4
-; CHECK-NEXT:    li 5, 1
-; CHECK-NEXT:    li 12, -1
+; CHECK-NEXT:    li 5, 0
 ; CHECK-NEXT:    cmpld 3, 4
-; CHECK-NEXT:    isel 3, 12, 5, 0
-; CHECK-NEXT:    isel 3, 0, 3, 2
+; CHECK-NEXT:    li 3, 1
+; CHECK-NEXT:    isel 4, 3, 5, 1
+; CHECK-NEXT:    isel 3, 3, 5, 0
+; CHECK-NEXT:    subf 3, 3, 4
+; CHECK-NEXT:    extsw 3, 3
 ; CHECK-NEXT:    blr
   %t0 = bitcast i32* %buffer1 to i8*
   %t1 = bitcast i32* %buffer2 to i8*
   %call = tail call signext i32 @memcmp(i8* %t0, i8* %t1, i64 8)
   ret i32 %call
 }
 
 define signext i32 @memcmp4(i32* nocapture readonly %buffer1, i32* nocapture readonly %buffer2) {
 ; CHECK-LABEL: memcmp4:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    lwbrx 3, 0, 3
 ; CHECK-NEXT:    lwbrx 4, 0, 4
-; CHECK-NEXT:    li 5, 1
-; CHECK-NEXT:    li 12, -1
-; CHECK-NEXT:    cmplw 3, 4
-; CHECK-NEXT:    isel 3, 12, 5, 0
-; CHECK-NEXT:    isel 3, 0, 3, 2
+; CHECK-NEXT:    sub 5, 4, 3
+; CHECK-NEXT:    sub 3, 3, 4
+; CHECK-NEXT:    rldicl 4, 5, 1, 63
+; CHECK-NEXT:    rldicl 3, 3, 1, 63
+; CHECK-NEXT:    subf 3, 3, 4
+; CHECK-NEXT:    extsw 3, 3
 ; CHECK-NEXT:    blr
   %t0 = bitcast i32* %buffer1 to i8*
   %t1 = bitcast i32* %buffer2 to i8*
   %call = tail call signext i32 @memcmp(i8* %t0, i8* %t1, i64 4)
   ret i32 %call
 }
 
 define signext i32 @memcmp2(i32* nocapture readonly %buffer1, i32* nocapture readonly %buffer2) {
 ; CHECK-LABEL: memcmp2:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    lhbrx 3, 0, 3
 ; CHECK-NEXT:    lhbrx 4, 0, 4
-; CHECK-NEXT:    li 5, 1
-; CHECK-NEXT:    li 12, -1
-; CHECK-NEXT:    cmplw 3, 4
-; CHECK-NEXT:    isel 3, 12, 5, 0
-; CHECK-NEXT:    isel 3, 0, 3, 2
+; CHECK-NEXT:    subf 3, 4, 3
+; CHECK-NEXT:    extsw 3, 3
 ; CHECK-NEXT:    blr
   %t0 = bitcast i32* %buffer1 to i8*
   %t1 = bitcast i32* %buffer2 to i8*
   %call = tail call signext i32 @memcmp(i8* %t0, i8* %t1, i64 2)
   ret i32 %call
 }
 
 define signext i32 @memcmp1(i32* nocapture readonly %buffer1, i32* nocapture readonly %buffer2) {
 ; CHECK-LABEL: memcmp1:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    lbz 3, 0(3)
 ; CHECK-NEXT:    lbz 4, 0(4)
-; CHECK-NEXT:    li 5, 1
-; CHECK-NEXT:    li 12, -1
-; CHECK-NEXT:    cmplw 3, 4
-; CHECK-NEXT:    isel 3, 12, 5, 0
-; CHECK-NEXT:    isel 3, 0, 3, 2
+; CHECK-NEXT:    subf 3, 4, 3
+; CHECK-NEXT:    extsw 3, 3
 ; CHECK-NEXT:    blr
   %t0 = bitcast i32* %buffer1 to i8*
   %t1 = bitcast i32* %buffer2 to i8*
   %call = tail call signext i32 @memcmp(i8* %t0, i8* %t1, i64 1) #2
   ret i32 %call
 }
 
 declare signext i32 @memcmp(i8*, i8*, i64)

llvm/trunk/test/CodeGen/PowerPC/memcmpIR.ll

 declare signext i32 @memcmp(i8* nocapture, i8* nocapture, i64) local_unnamed_addr #1
 
 define signext i32 @test2(i32* nocapture readonly %buffer1, i32* nocapture readonly %buffer2)  {
   ; CHECK-LABEL: @test2(
   ; CHECK: [[LOAD1:%[0-9]+]] = load i32, i32*
   ; CHECK-NEXT: [[LOAD2:%[0-9]+]] = load i32, i32*
   ; CHECK-NEXT: [[BSWAP1:%[0-9]+]] = call i32 @llvm.bswap.i32(i32 [[LOAD1]])
   ; CHECK-NEXT: [[BSWAP2:%[0-9]+]] = call i32 @llvm.bswap.i32(i32 [[LOAD2]])
-  ; CHECK-NEXT: [[CMP1:%[0-9]+]] = icmp ne i32 [[BSWAP1]], [[BSWAP2]]
+  ; CHECK-NEXT: [[CMP1:%[0-9]+]] = icmp ugt i32 [[BSWAP1]], [[BSWAP2]]
   ; CHECK-NEXT: [[CMP2:%[0-9]+]] = icmp ult i32 [[BSWAP1]], [[BSWAP2]]
-  ; CHECK-NEXT: [[SELECT1:%[0-9]+]] = select i1 [[CMP2]], i32 -1, i32 1
-  ; CHECK-NEXT: [[SELECT2:%[0-9]+]] = select i1 [[CMP1]], i32 [[SELECT1]], i32 0
-  ; CHECK-NEXT: ret i32 [[SELECT2]]
+  ; CHECK-NEXT: [[Z1:%[0-9]+]] = zext i1 [[CMP1]] to i32
+  ; CHECK-NEXT: [[Z2:%[0-9]+]] = zext i1 [[CMP2]] to i32
+  ; CHECK-NEXT: [[SUB:%[0-9]+]] = sub i32 [[Z1]], [[Z2]]
+  ; CHECK-NEXT: ret i32 [[SUB]]
 
   ; CHECK-BE-LABEL: @test2(
   ; CHECK-BE: [[LOAD1:%[0-9]+]] = load i32, i32*
   ; CHECK-BE-NEXT: [[LOAD2:%[0-9]+]] = load i32, i32*
-  ; CHECK-BE-NEXT: [[CMP1:%[0-9]+]] = icmp ne i32 [[LOAD1]], [[LOAD2]]
+  ; CHECK-BE-NEXT: [[CMP1:%[0-9]+]] = icmp ugt i32 [[LOAD1]], [[LOAD2]]
   ; CHECK-BE-NEXT: [[CMP2:%[0-9]+]] = icmp ult i32 [[LOAD1]], [[LOAD2]]
-  ; CHECK-BE-NEXT: [[SELECT1:%[0-9]+]] = select i1 [[CMP2]], i32 -1, i32 1
-  ; CHECK-BE-NEXT: [[SELECT2:%[0-9]+]] = select i1 [[CMP1]], i32 [[SELECT1]], i32 0
-  ; CHECK-BE-NEXT: ret i32 [[SELECT2]]
+  ; CHECK-BE-NEXT: [[Z1:%[0-9]+]] = zext i1 [[CMP1]] to i32
+  ; CHECK-BE-NEXT: [[Z2:%[0-9]+]] = zext i1 [[CMP2]] to i32
+  ; CHECK-BE-NEXT: [[SUB:%[0-9]+]] = sub i32 [[Z1]], [[Z2]]
+  ; CHECK-BE-NEXT: ret i32 [[SUB]]
 
 entry:
   %0 = bitcast i32* %buffer1 to i8*
   %1 = bitcast i32* %buffer2 to i8*
   %call = tail call signext i32 @memcmp(i8* %0, i8* %1, i64 4)
   ret i32 %call
 }
 

llvm/trunk/test/CodeGen/X86/memcmp-optsize.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=cmov | FileCheck %s --check-prefix=X86 --check-prefix=X86-NOSSE
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefix=X86 --check-prefix=X86-SSE2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s --check-prefix=X64 --check-prefix=X64-SSE2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx2 | FileCheck %s --check-prefix=X64 --check-prefix=X64-AVX2
 
 ; This tests codegen time inlining/optimization of memcmp
 ; rdar://6480398
 
 @.str = private constant [65 x i8] c"0123456789012345678901234567890123456789012345678901234567890123\00", align 1
 
 declare i32 @memcmp(i8*, i8*, i64)
 
 define i32 @length2(i8* %X, i8* %Y) nounwind optsize {
 ; X86-LABEL: length2:
 ; X86:       # BB#0:
-; X86-NEXT:    pushl %edi
-; X86-NEXT:    pushl %esi
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movzwl (%ecx), %ecx
 ; X86-NEXT:    movzwl (%eax), %edx
 ; X86-NEXT:    rolw $8, %cx
 ; X86-NEXT:    rolw $8, %dx
-; X86-NEXT:    xorl %esi, %esi
-; X86-NEXT:    xorl %edi, %edi
-; X86-NEXT:    incl %edi
-; X86-NEXT:    xorl %eax, %eax
-; X86-NEXT:    decl %eax
-; X86-NEXT:    cmpw %dx, %cx
-; X86-NEXT:    cmovael %edi, %eax
-; X86-NEXT:    cmovel %esi, %eax
-; X86-NEXT:    popl %esi
-; X86-NEXT:    popl %edi
+; X86-NEXT:    movzwl %cx, %eax
+; X86-NEXT:    movzwl %dx, %ecx
+; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: length2:
 ; X64:       # BB#0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzwl (%rsi), %ecx
 ; X64-NEXT:    rolw $8, %ax
 ; X64-NEXT:    rolw $8, %cx
-; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    cmpw %cx, %ax
-; X64-NEXT:    movl $-1, %ecx
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovbl %ecx, %eax
-; X64-NEXT:    cmovel %edx, %eax
+; X64-NEXT:    movzwl %ax, %eax
+; X64-NEXT:    movzwl %cx, %ecx
+; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 2) nounwind
   ret i32 %m
 }
 
 define i1 @length2_eq(i8* %X, i8* %Y) nounwind optsize {
 ; X86-LABEL: length2_eq:
 ; X86:       # BB#0:
; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 3) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length4(i8* %X, i8* %Y) nounwind optsize {
 ; X86-LABEL: length4:
 ; X86:       # BB#0:
-; X86-NEXT:    pushl %edi
-; X86-NEXT:    pushl %esi
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl (%ecx), %ecx
 ; X86-NEXT:    movl (%eax), %edx
 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
-; X86-NEXT:    xorl %esi, %esi
-; X86-NEXT:    xorl %edi, %edi
-; X86-NEXT:    incl %edi
 ; X86-NEXT:    xorl %eax, %eax
-; X86-NEXT:    decl %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    cmovael %edi, %eax
-; X86-NEXT:    cmovel %esi, %eax
-; X86-NEXT:    popl %esi
-; X86-NEXT:    popl %edi
+; X86-NEXT:    seta %al
+; X86-NEXT:    sbbl $0, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: length4:
 ; X64:       # BB#0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    cmpl %ecx, %eax
-; X64-NEXT:    movl $-1, %ecx
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovbl %ecx, %eax
-; X64-NEXT:    cmovel %edx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    seta %al
+; X64-NEXT:    sbbl $0, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 4) nounwind
   ret i32 %m
 }
 
 define i1 @length4_eq(i8* %X, i8* %Y) nounwind optsize {
 ; X86-LABEL: length4_eq:
 ; X86:       # BB#0:

 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    cmovael %esi, %eax
 ; X86-NEXT:  .LBB11_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: length8:
 ; X64:       # BB#0:
-; X64-NEXT:    movq (%rdi), %rax
-; X64-NEXT:    movq (%rsi), %rcx
-; X64-NEXT:    bswapq %rax
+; X64-NEXT:    movq (%rdi), %rcx
+; X64-NEXT:    movq (%rsi), %rdx
 ; X64-NEXT:    bswapq %rcx
-; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    movl $-1, %ecx
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovbl %ecx, %eax
-; X64-NEXT:    cmovel %edx, %eax
+; X64-NEXT:    bswapq %rdx
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpq %rdx, %rcx
+; X64-NEXT:    seta %al
+; X64-NEXT:    sbbl $0, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 8) nounwind
   ret i32 %m
 }
 
 define i1 @length8_eq(i8* %X, i8* %Y) nounwind optsize {
 ; X86-LABEL: length8_eq:
 ; X86:       # BB#0: # %loadbb

llvm/trunk/test/CodeGen/X86/memcmp.ll

 declare i32 @memcmp(i8*, i8*, i64)
 
 define i32 @length2(i8* %X, i8* %Y) nounwind {
 ; X86-LABEL: length2:
 ; X86:       # BB#0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movzwl (%ecx), %ecx
-; X86-NEXT:    movzwl (%eax), %eax
+; X86-NEXT:    movzwl (%eax), %edx
 ; X86-NEXT:    rolw $8, %cx
-; X86-NEXT:    rolw $8, %ax
-; X86-NEXT:    xorl %edx, %edx
-; X86-NEXT:    cmpw %ax, %cx
-; X86-NEXT:    movl $-1, %ecx
-; X86-NEXT:    movl $1, %eax
-; X86-NEXT:    cmovbl %ecx, %eax
-; X86-NEXT:    cmovel %edx, %eax
+; X86-NEXT:    rolw $8, %dx
+; X86-NEXT:    movzwl %cx, %eax
+; X86-NEXT:    movzwl %dx, %ecx
+; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: length2:
 ; X64:       # BB#0:
 ; X64-NEXT:    movzwl (%rdi), %eax
 ; X64-NEXT:    movzwl (%rsi), %ecx
 ; X64-NEXT:    rolw $8, %ax
 ; X64-NEXT:    rolw $8, %cx
-; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    cmpw %cx, %ax
-; X64-NEXT:    movl $-1, %ecx
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovbl %ecx, %eax
-; X64-NEXT:    cmovel %edx, %eax
+; X64-NEXT:    movzwl %ax, %eax
+; X64-NEXT:    movzwl %cx, %ecx
+; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 2) nounwind
   ret i32 %m
 }
 
 define i1 @length2_eq(i8* %X, i8* %Y) nounwind {
 ; X86-LABEL: length2_eq:
 ; X86:       # BB#0:

 }
 
 define i32 @length4(i8* %X, i8* %Y) nounwind {
 ; X86-LABEL: length4:
 ; X86:       # BB#0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl (%ecx), %ecx
-; X86-NEXT:    movl (%eax), %eax
+; X86-NEXT:    movl (%eax), %edx
 ; X86-NEXT:    bswapl %ecx
-; X86-NEXT:    bswapl %eax
-; X86-NEXT:    xorl %edx, %edx
-; X86-NEXT:    cmpl %eax, %ecx
-; X86-NEXT:    movl $-1, %ecx
-; X86-NEXT:    movl $1, %eax
-; X86-NEXT:    cmovbl %ecx, %eax
-; X86-NEXT:    cmovel %edx, %eax
+; X86-NEXT:    bswapl %edx
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %edx, %ecx
+; X86-NEXT:    seta %al
+; X86-NEXT:    sbbl $0, %eax
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: length4:
 ; X64:       # BB#0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    cmpl %ecx, %eax
-; X64-NEXT:    movl $-1, %ecx
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovbl %ecx, %eax
-; X64-NEXT:    cmovel %edx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    seta %al
+; X64-NEXT:    sbbl $0, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 4) nounwind
   ret i32 %m
 }
 
 define i1 @length4_eq(i8* %X, i8* %Y) nounwind {
 ; X86-LABEL: length4_eq:
 ; X86:       # BB#0:

 ; X86-NEXT:    movl $-1, %ecx
 ; X86-NEXT:    movl $1, %eax
 ; X86-NEXT:    cmovbl %ecx, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: length8:
 ; X64:       # BB#0:
-; X64-NEXT:    movq (%rdi), %rax
-; X64-NEXT:    movq (%rsi), %rcx
-; X64-NEXT:    bswapq %rax
+; X64-NEXT:    movq (%rdi), %rcx
+; X64-NEXT:    movq (%rsi), %rdx
 ; X64-NEXT:    bswapq %rcx
-; X64-NEXT:    xorl %edx, %edx
-; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    movl $-1, %ecx
-; X64-NEXT:    movl $1, %eax
-; X64-NEXT:    cmovbl %ecx, %eax
-; X64-NEXT:    cmovel %edx, %eax
+; X64-NEXT:    bswapq %rdx
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpq %rdx, %rcx
+; X64-NEXT:    seta %al
+; X64-NEXT:    sbbl $0, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(i8* %X, i8* %Y, i64 8) nounwind
   ret i32 %m
 }
 
 define i1 @length8_eq(i8* %X, i8* %Y) nounwind {
 ; X86-LABEL: length8_eq:
 ; X86:       # BB#0: # %loadbb

llvm/trunk/test/Transforms/CodeGenPrepare/X86/memcmp.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -codegenprepare -mtriple=i686-unknown-unknown   -data-layout=e-m:o-p:32:32-f64:32:64-f80:128-n8:16:32-S128 < %s | FileCheck %s --check-prefix=ALL --check-prefix=X32
 ; RUN: opt -S -codegenprepare -mtriple=x86_64-unknown-unknown -data-layout=e-m:o-i64:64-f80:128-n8:16:32:64-S128         < %s | FileCheck %s --check-prefix=ALL --check-prefix=X64
 
 declare i32 @memcmp(i8* nocapture, i8* nocapture, i64)
 
 define i32 @cmp2(i8* nocapture readonly %x, i8* nocapture readonly %y)  {
 ; ALL-LABEL: @cmp2(
 ; ALL-NEXT:    [[TMP1:%.*]] = bitcast i8* [[X:%.*]] to i16*
 ; ALL-NEXT:    [[TMP2:%.*]] = bitcast i8* [[Y:%.*]] to i16*
 ; ALL-NEXT:    [[TMP3:%.*]] = load i16, i16* [[TMP1]]
 ; ALL-NEXT:    [[TMP4:%.*]] = load i16, i16* [[TMP2]]
 ; ALL-NEXT:    [[TMP5:%.*]] = call i16 @llvm.bswap.i16(i16 [[TMP3]])
 ; ALL-NEXT:    [[TMP6:%.*]] = call i16 @llvm.bswap.i16(i16 [[TMP4]])
-; ALL-NEXT:    [[TMP7:%.*]] = icmp ne i16 [[TMP5]], [[TMP6]]
-; ALL-NEXT:    [[TMP8:%.*]] = icmp ult i16 [[TMP5]], [[TMP6]]
-; ALL-NEXT:    [[TMP9:%.*]] = select i1 [[TMP8]], i32 -1, i32 1
-; ALL-NEXT:    [[TMP10:%.*]] = select i1 [[TMP7]], i32 [[TMP9]], i32 0
-; ALL-NEXT:    ret i32 [[TMP10]]
+; ALL-NEXT:    [[TMP7:%.*]] = zext i16 [[TMP5]] to i32
+; ALL-NEXT:    [[TMP8:%.*]] = zext i16 [[TMP6]] to i32
+; ALL-NEXT:    [[TMP9:%.*]] = sub i32 [[TMP7]], [[TMP8]]
+; ALL-NEXT:    ret i32 [[TMP9]]
 ;
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 2)
   ret i32 %call
 }
 
 define i32 @cmp3(i8* nocapture readonly %x, i8* nocapture readonly %y)  {
 ; X32-LABEL: @cmp3(
 ; X32-NEXT:  loadbb:

 define i32 @cmp4(i8* nocapture readonly %x, i8* nocapture readonly %y)  {
 ; ALL-LABEL: @cmp4(
 ; ALL-NEXT:    [[TMP1:%.*]] = bitcast i8* [[X:%.*]] to i32*
 ; ALL-NEXT:    [[TMP2:%.*]] = bitcast i8* [[Y:%.*]] to i32*
 ; ALL-NEXT:    [[TMP3:%.*]] = load i32, i32* [[TMP1]]
 ; ALL-NEXT:    [[TMP4:%.*]] = load i32, i32* [[TMP2]]
 ; ALL-NEXT:    [[TMP5:%.*]] = call i32 @llvm.bswap.i32(i32 [[TMP3]])
 ; ALL-NEXT:    [[TMP6:%.*]] = call i32 @llvm.bswap.i32(i32 [[TMP4]])
-; ALL-NEXT:    [[TMP7:%.*]] = icmp ne i32 [[TMP5]], [[TMP6]]
+; ALL-NEXT:    [[TMP7:%.*]] = icmp ugt i32 [[TMP5]], [[TMP6]]
 ; ALL-NEXT:    [[TMP8:%.*]] = icmp ult i32 [[TMP5]], [[TMP6]]
-; ALL-NEXT:    [[TMP9:%.*]] = select i1 [[TMP8]], i32 -1, i32 1
-; ALL-NEXT:    [[TMP10:%.*]] = select i1 [[TMP7]], i32 [[TMP9]], i32 0
-; ALL-NEXT:    ret i32 [[TMP10]]
+; ALL-NEXT:    [[TMP9:%.*]] = zext i1 [[TMP7]] to i32
+; ALL-NEXT:    [[TMP10:%.*]] = zext i1 [[TMP8]] to i32
+; ALL-NEXT:    [[TMP11:%.*]] = sub i32 [[TMP9]], [[TMP10]]
+; ALL-NEXT:    ret i32 [[TMP11]]
 ;
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 4)
   ret i32 %call
 }
 
 define i32 @cmp5(i8* nocapture readonly %x, i8* nocapture readonly %y)  {
 ; X32-LABEL: @cmp5(
 ; X32-NEXT:  loadbb:

 ;
 ; X64-LABEL: @cmp8(
 ; X64-NEXT:    [[TMP1:%.*]] = bitcast i8* [[X:%.*]] to i64*
 ; X64-NEXT:    [[TMP2:%.*]] = bitcast i8* [[Y:%.*]] to i64*
 ; X64-NEXT:    [[TMP3:%.*]] = load i64, i64* [[TMP1]]
 ; X64-NEXT:    [[TMP4:%.*]] = load i64, i64* [[TMP2]]
 ; X64-NEXT:    [[TMP5:%.*]] = call i64 @llvm.bswap.i64(i64 [[TMP3]])
 ; X64-NEXT:    [[TMP6:%.*]] = call i64 @llvm.bswap.i64(i64 [[TMP4]])
-; X64-NEXT:    [[TMP7:%.*]] = icmp ne i64 [[TMP5]], [[TMP6]]
+; X64-NEXT:    [[TMP7:%.*]] = icmp ugt i64 [[TMP5]], [[TMP6]]
 ; X64-NEXT:    [[TMP8:%.*]] = icmp ult i64 [[TMP5]], [[TMP6]]
-; X64-NEXT:    [[TMP9:%.*]] = select i1 [[TMP8]], i32 -1, i32 1
-; X64-NEXT:    [[TMP10:%.*]] = select i1 [[TMP7]], i32 [[TMP9]], i32 0
-; X64-NEXT:    ret i32 [[TMP10]]
+; X64-NEXT:    [[TMP9:%.*]] = zext i1 [[TMP7]] to i32
+; X64-NEXT:    [[TMP10:%.*]] = zext i1 [[TMP8]] to i32
+; X64-NEXT:    [[TMP11:%.*]] = sub i32 [[TMP9]], [[TMP10]]
+; X64-NEXT:    ret i32 [[TMP11]]
 ;
   %call = tail call i32 @memcmp(i8* %x, i8* %y, i64 8)
   ret i32 %call
 }
 
 define i32 @cmp9(i8* nocapture readonly %x, i8* nocapture readonly %y)  {
 ; X32-LABEL: @cmp9(
 ; X32-NEXT:    [[CALL:%.*]] = tail call i32 @memcmp(i8* [[X:%.*]], i8* [[Y:%.*]], i64 9)