[DAG] select Cond, -1, C --> or (sext Cond), C if Cond is MVT::i1

[deadalnix]

Summary

This seems to be beneficial overall, except for midpoint-int.ll .

The X86 backend seems to generate zeroing that are not necesary.

[deadalnix]

llvm/test/CodeGen/X86/memcmp-more-load-pairs-x32.ll

163–166

This seems to be unnecessary.

[pengfei]

llvm/test/CodeGen/X86/memcmp-more-load-pairs-x32.ll

163–166

Looks like dependency breaking for sbb:
https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp#L5874-L5876

[deadalnix]

llvm/test/CodeGen/X86/memcmp-more-load-pairs-x32.ll

163–166

Good catch, thanks.

[deadalnix]

Reabse and variosu tweaks.

[deadalnix]

llvm/test/CodeGen/X86/select_const.ll

290

TODO: (sext Cond) | (Pow2 - 1) -> Pow2 - (zext (not Cond)) ?

Maybe if inverting Cond if for free? Is there a way to check for this?

Or maybe this is best reserved for X86DAGToDAGISel::matchAddressRecursively?

[deadalnix]

llvm/test/CodeGen/X86/select_const.ll

290

I meant (sext Cond) | (Pow2 - 1) -> (zext (not Cond)) * Pow2 - 1 ?

[deadalnix]

llvm/test/CodeGen/X86/select_const.ll

290

Doing it in selectLEAAddr doesn't work as the score doesn't get high enough.

[deadalnix]

Rebase on top of D131358

[deadalnix]

llvm/test/CodeGen/X86/memcmp.ll

371–372

In both cases here, cmpl %edx, %ecx seems to be recomputed for no reason (?) It seems to be able to reuse cmpl %ecx, %eax in length5 and cmpw %dx, %cx in length3, at least before this diff.

Is there a reason why this isn't the case here? I'm not quite sure how memcmp is being lowered, and -debug really isn't useful here. @gchatelet , you were looking for ways to improve memcmp, I think there is a low hanging fruit somewhere in there.

[pengfei]

llvm/test/CodeGen/X86/memcmp.ll

371–372

xor will change EFLAGS, so we need to recomputed to get the correct EFLAGS.

[deadalnix]

llvm/test/CodeGen/X86/memcmp.ll

371–372

Sure, but the xor itself doesn't seem necessary. For instance, length5 does:

setae %al
movzbl %al, %eax

Which does away witth he need for the xor, with in turn does away with the need to recompute the cmpl. Both function do the same thing in that block, there is no reason to have different codegen.

[RKSimon]

please can you rebase this against trunk latest?

[deadalnix]

rebase

[RKSimon]

llvm/test/CodeGen/X86/select_const.ll

204

Comment doesn't match select_lea_2 (but the others are still OK).

[RKSimon]

llvm/test/CodeGen/PowerPC/ppc64-P9-setb.ll

1212 ↗

(On Diff #451591)

regression?

[deadalnix]

llvm/test/CodeGen/PowerPC/ppc64-P9-setb.ll

1212 ↗

(On Diff #451591)

I have no idea, but think so.

[deadalnix]

I restored the select_cc creation, but now it generates some select_cc that the powerpc backend doesn't quite know what to do with.

[RKSimon]

Adding some PPC people who might be able to advise on the remaining regressions

[deadalnix]

Dear PowerPC folks, your help would be greatly appreciated here.

[shchenz]

llvm/test/CodeGen/PowerPC/signbit-shift.ll

142 ↗

(On Diff #455002)

The changes in this file seem regressions, especially for this case, more instructions and more registers. Do we know why?

[deadalnix]

llvm/test/CodeGen/PowerPC/signbit-shift.ll

142 ↗

(On Diff #455002)

Yes. The PowerPC backend request for select_cc instruction. The select_cc instruction is not turned into arythmetic the way regular select instructions are.

This patch changes how select -> arithmetic transformations are done, which enable more transform in general, but specifically on PowerPC, this cases problems because it causes either select_cc to not be generated due select -> arithmetic transformations, or, alternatively, if the promotion to select_cc is prioritized, then the cases where the arithmetic transform was beneficial are not done.

Do we have a good idea when select_cc -> arithmetic is beneficial? If we have something reasonable here, we can add the select_cc transforms and get these regression dealt with.

[deadalnix]

What happens if select -> arithmetic takes priority

[deadalnix]

When select_cc takes precedence

[deadalnix]

llvm/test/CodeGen/PowerPC/signbit-shift.ll

142 ↗

(On Diff #455002)

If you want to see the difference when one takes priority over the other: https://reviews.llvm.org/D131260?vs=458558&id=458561

[shchenz]

llvm/test/CodeGen/PowerPC/signbit-shift.ll

142 ↗

(On Diff #455002)

Hmm, PowerPC sets SELECT_CC as custom for type i32/i64, but it does not mean the customization is better than the arithmetic way here. In fact it is worse as shown in the case. If on other targets, the customization for SELECT_CC is better, then we may need to adjust the current target hook foldSelectOfConstants()(for example, if we know the SELECT can be optimized to SELECT_CC on some target and SELECT_CC is better, we return false?) or we may need to re-implement the SELECT combining logic here on PowerPC for SELECT_CC. IMO, the first one makes more sense. Thoughts?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

10312

Maybe adjust convertSelectOfConstantsToMath to take the Cond opcode as another argument instead?

[RKSimon]

@deadalnix reverse ping

[deadalnix]

@RKSimon I'll try what you suggest tomorrow. I've been sick so things are moving slowly on my end. Apologies.

[deadalnix]

Update the condition in which we transform select to math as to preserve opportunities to optimize select_cc

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

23979

unrelated - pre-commit this?

[deadalnix]

Fix the latest broken test

[deadalnix]

Precommit trivial changes and rebase

[RKSimon]

LGTM @shchenz any more comments?

[shchenz]

In D131260#3817531, @RKSimon wrote:

LGTM @shchenz any more comments?

PPC part looks good!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (CC == ISD::SETLT && isNullOrNullSplat(CondC) && isNullOrNullSplat(C2)) {
     SDLoc DL(N);
     SDValue ShAmtC = DAG.getConstant(X.getScalarValueSizeInBits() - 1, DL, VT);
     SDValue Sra = DAG.getNode(ISD::SRA, DL, VT, X, ShAmtC);
     return DAG.getNode(ISD::AND, DL, VT, Sra, C1);
   }
   return SDValue();
 }
 
+static bool shouldConvertSelectOfConstantsToMath(const SDValue &Cond, EVT VT,
+                                                 const TargetLowering &TLI) {
+  if (!TLI.convertSelectOfConstantsToMath(VT))
+    return false;
+
+  if (Cond.getOpcode() != ISD::SETCC || !Cond->hasOneUse())
+    return true;
+  if (!TLI.isOperationLegalOrCustom(ISD::SELECT_CC, VT))
+    return true;
+
+  ISD::CondCode CC = cast<CondCodeSDNode>(Cond.getOperand(2))->get();
+  if (CC == ISD::SETLT && isNullOrNullSplat(Cond.getOperand(1)))
+    return true;
+  if (CC == ISD::SETGT && isAllOnesOrAllOnesSplat(Cond.getOperand(1)))
+    return true;
+
+  return false;
+}
+
 SDValue DAGCombiner::foldSelectOfConstants(SDNode *N) {
   SDValue Cond = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   SDValue N2 = N->getOperand(2);
   EVT VT = N->getValueType(0);
   EVT CondVT = Cond.getValueType();
   SDLoc DL(N);
 
if (C1->isAllOnes() && C2->isZero()) {
       // select Cond, -1, 0 --> sext (Cond)
       if (VT != MVT::i1)
         Cond = DAG.getNode(ISD::SIGN_EXTEND, DL, VT, Cond);
       return Cond;
     }
 
     // Use a target hook because some targets may prefer to transform in the
     // other direction.
-    if (TLI.convertSelectOfConstantsToMath(VT)) {
-      // For any constants that differ by 1, we can transform the select into an
-      // extend and add.
+    if (shouldConvertSelectOfConstantsToMath(Cond, VT, TLI)) {
+      // For any constants that differ by 1, we can transform the select into
+      // an extend and add.
       const APInt &C1Val = C1->getAPIntValue();
       const APInt &C2Val = C2->getAPIntValue();
       if (C1Val - 1 == C2Val) {
         // select Cond, C1, C1-1 --> add (zext Cond), C1-1
         if (VT != MVT::i1)
           Cond = DAG.getNode(ISD::ZERO_EXTEND, DL, VT, Cond);
         return DAG.getNode(ISD::ADD, DL, VT, Cond, N2);
       }
+
       if (C1Val + 1 == C2Val) {
         // select Cond, C1, C1+1 --> add (sext Cond), C1+1
         if (VT != MVT::i1)
           Cond = DAG.getNode(ISD::SIGN_EXTEND, DL, VT, Cond);
         return DAG.getNode(ISD::ADD, DL, VT, Cond, N2);
       }
 
       // select Cond, Pow2, 0 --> (zext Cond) << log2(Pow2)
       if (C1Val.isPowerOf2() && C2Val.isZero()) {
         if (VT != MVT::i1)
           Cond = DAG.getNode(ISD::ZERO_EXTEND, DL, VT, Cond);
         SDValue ShAmtC =
             DAG.getShiftAmountConstant(C1Val.exactLogBase2(), VT, DL);
         return DAG.getNode(ISD::SHL, DL, VT, Cond, ShAmtC);
       }
 
+      // select Cond, -1, C --> or (sext Cond), C
+      if (C1->isAllOnes()) {
+        Cond = DAG.getSExtOrTrunc(Cond, DL, VT);
+        return DAG.getNode(ISD::OR, DL, VT, Cond, N2);
+      }
+
       if (SDValue V = foldSelectOfConstantsUsingSra(N, DAG))
         return V;
     }
 
     return SDValue();
   }
 
   // fold (select Cond, 0, 1) -> (xor Cond, 1)
SDValue DAGCombiner::visitSELECT(SDNode *N) {
   EVT VT = N->getValueType(0);
   EVT VT0 = N0.getValueType();
   SDLoc DL(N);
   SDNodeFlags Flags = N->getFlags();
 
   if (SDValue V = DAG.simplifySelect(N0, N1, N2))
     return V;
 
-  if (SDValue V = foldSelectOfConstants(N))
+  if (SDValue V = foldBoolSelectToLogic(N, DAG))
     return V;
 
-  if (SDValue V = foldBoolSelectToLogic(N, DAG))
+  // select (not Cond), N1, N2 -> select Cond, N2, N1
+  if (SDValue F = extractBooleanFlip(N0, DAG, TLI, false)) {
+    SDValue SelectOp = DAG.getSelect(DL, VT, F, N2, N1);
+    SelectOp->setFlags(Flags);
+    return SelectOp;
+  }
+
+  if (SDValue V = foldSelectOfConstants(N))
     return V;
 
   // If we can fold this based on the true/false value, do so.
   if (SimplifySelectOps(N, N1, N2))
     return SDValue(N, 0); // Don't revisit N.
 
   if (VT0 == MVT::i1) {
     // The code in this block deals with the following 2 equivalences:
if (N2->getOpcode() == ISD::SELECT && N2->hasOneUse()) {
         // Otherwise see if we can optimize to a better pattern.
         if (SDValue Combined = visitORLike(N0, N2_0, N))
           return DAG.getNode(ISD::SELECT, DL, N1.getValueType(), Combined, N1,
                              N2_2, Flags);
       }
     }
   }
 
-  // select (not Cond), N1, N2 -> select Cond, N2, N1
-  if (SDValue F = extractBooleanFlip(N0, DAG, TLI, false)) {
-    SDValue SelectOp = DAG.getSelect(DL, VT, F, N2, N1);
-    SelectOp->setFlags(Flags);
-    return SelectOp;
-  }
-
   // Fold selects based on a setcc into other things, such as min/max/abs.
   if (N0.getOpcode() == ISD::SETCC) {
     SDValue Cond0 = N0.getOperand(0), Cond1 = N0.getOperand(1);
     ISD::CondCode CC = cast<CondCodeSDNode>(N0.getOperand(2))->get();
 
     // select (fcmp lt x, y), x, y -> fminnum x, y
     // select (fcmp gt x, y), x, y -> fmaxnum x, y
     //

 /// A vector select of 2 constant vectors can be simplified to math/logic to
 /// avoid a variable select instruction and possibly avoid constant loads.
 SDValue DAGCombiner::foldVSelectOfConstants(SDNode *N) {
   SDValue Cond = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   SDValue N2 = N->getOperand(2);
   EVT VT = N->getValueType(0);
   if (!Cond.hasOneUse() || Cond.getScalarValueSizeInBits() != 1 ||
-      !TLI.convertSelectOfConstantsToMath(VT) ||
+      !shouldConvertSelectOfConstantsToMath(Cond, VT, TLI) ||
       !ISD::isBuildVectorOfConstantSDNodes(N1.getNode()) ||
       !ISD::isBuildVectorOfConstantSDNodes(N2.getNode()))
     return SDValue();
 
   // Check if we can use the condition value to increment/decrement a single
   // constant value. This simplifies a select to an add and removes a constant
   // load/materialization from the general case.
   bool AllAddOne = true;
SDValue DAGCombiner::visitSELECT_CC(SDNode *N) {
   SDValue N3 = N->getOperand(3);
   SDValue N4 = N->getOperand(4);
   ISD::CondCode CC = cast<CondCodeSDNode>(N4)->get();
 
   // fold select_cc lhs, rhs, x, x, cc -> x
   if (N2 == N3)
     return N2;
 
+  // select_cc bool, 0, x, y, seteq -> select bool, y, x
+  if (CC == ISD::SETEQ && !LegalTypes && N0.getValueType() == MVT::i1 &&
+      isNullConstant(N1))
+    return DAG.getSelect(SDLoc(N), N2.getValueType(), N0, N3, N2);
+
   // Determine if the condition we're dealing with is constant
   if (SDValue SCC = SimplifySetCC(getSetCCResultType(N0.getValueType()), N0, N1,
                                   CC, SDLoc(N), false)) {
     AddToWorklist(SCC.getNode());
 
     // cond always true -> true val
     // cond always false -> false val
     if (auto *SCCC = dyn_cast<ConstantSDNode>(SCC.getNode()))
SDValue DAGCombiner::foldSextSetcc(SDNode *N) {
   // of the appropriate width.
   SDValue ExtTrueVal = (SetCCWidth == 1)
                            ? DAG.getAllOnesConstant(DL, VT)
                            : DAG.getBoolConstant(true, DL, VT, N00VT);
   SDValue Zero = DAG.getConstant(0, DL, VT);
   if (SDValue SCC = SimplifySelectCC(DL, N00, N01, ExtTrueVal, Zero, CC, true))
     return SCC;
 
-  if (!VT.isVector() && !TLI.convertSelectOfConstantsToMath(VT)) {
+  if (!VT.isVector() && !shouldConvertSelectOfConstantsToMath(N0, VT, TLI)) {
     EVT SetCCVT = getSetCCResultType(N00VT);
     // Don't do this transform for i1 because there's a select transform
     // that would reverse it.
     // TODO: We should not do this transform at all without a target hook
     // because a sext is likely cheaper than a select?
     if (SetCCVT.getScalarSizeInBits() != 1 &&
         (!LegalOperations || TLI.isOperationLegal(ISD::SETCC, N00VT))) {
       SDValue SetCC = DAG.getSetCC(DL, SetCCVT, N00, N01, CC);
if (SDValue V = scalarizeBinOpOfSplats(N, DAG, DL))
     return V;
 
   return SDValue();
 }
 
 SDValue DAGCombiner::SimplifySelect(const SDLoc &DL, SDValue N0, SDValue N1,
                                     SDValue N2) {
   assert(N0.getOpcode() == ISD::SETCC &&
          "First argument must be a SetCC node!");
 
   SDValue SCC = SimplifySelectCC(DL, N0.getOperand(0), N0.getOperand(1), N1, N2,
                                  cast<CondCodeSDNode>(N0.getOperand(2))->get());
 
   // If we got a simplified select_cc node back from SimplifySelectCC, then
   // break it down into a new SETCC node, and a new SELECT node, and then return
   // the SELECT node, since we were called with a SELECT node.
   if (SCC.getNode()) {

llvm/test/CodeGen/PowerPC/crbits.ll

entry:
   ret i32 %cond
 
 }
 
 define signext i32 @exttest7(i32 signext %a) #0 {
 ; CHECK-LABEL: exttest7:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    li 4, 8
-; CHECK-NEXT:    cmpwi 3, 5
+; CHECK-NEXT:    cmplwi 3, 5
 ; CHECK-NEXT:    li 3, 7
 ; CHECK-NEXT:    iseleq 3, 3, 4
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-NO-ISEL-LABEL: exttest7:
 ; CHECK-NO-ISEL:       # %bb.0: # %entry
 ; CHECK-NO-ISEL-NEXT:    li 4, 8
-; CHECK-NO-ISEL-NEXT:    cmpwi 3, 5
+; CHECK-NO-ISEL-NEXT:    cmplwi 3, 5
 ; CHECK-NO-ISEL-NEXT:    li 3, 7
 ; CHECK-NO-ISEL-NEXT:    bclr 12, 2, 0
 ; CHECK-NO-ISEL-NEXT:  # %bb.1: # %entry
 ; CHECK-NO-ISEL-NEXT:    ori 3, 4, 0
 ; CHECK-NO-ISEL-NEXT:    blr
 ;
 ; CHECK-P10-LABEL: exttest7:
 ; CHECK-P10:       # %bb.0: # %entry
-; CHECK-P10-NEXT:    cmpwi r3, 5
+; CHECK-P10-NEXT:    cmplwi r3, 5
 ; CHECK-P10-NEXT:    li r3, 8
 ; CHECK-P10-NEXT:    li r4, 7
 ; CHECK-P10-NEXT:    iseleq r3, r4, r3
 ; CHECK-P10-NEXT:    blr
 entry:
   %cmp = icmp eq i32 %a, 5
   %cond = select i1 %cmp, i32 7, i32 8
   ret i32 %cond

llvm/test/CodeGen/PowerPC/prefer-dqform.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -disable-ppc-instr-form-prep=true -mcpu=pwr9 < %s \
 ; RUN:   -ppc-vsr-nums-as-vr -ppc-asm-full-reg-names | FileCheck %s -check-prefix=CHECK-P9
 ; RUN: llc -verify-machineinstrs -disable-ppc-instr-form-prep=true -mcpu=pwr10 < %s \
 ; RUN:   -ppc-vsr-nums-as-vr -ppc-asm-full-reg-names | FileCheck %s -check-prefix=CHECK-P10
 
 target triple = "powerpc64le-unknown-linux-gnu"
 
 %_elem_type_of_a = type <{ double }>
 %_elem_type_of_x = type <{ double }>
 %_elem_type_of_y = type <{ double }>
 
 define void @test(i32* dereferenceable(4) %.ial, i32* noalias dereferenceable(4) %.m, i32* noalias dereferenceable(4) %.n, [0 x %_elem_type_of_a]*  %.a, i32* noalias dereferenceable(4) %.lda, [0 x %_elem_type_of_x]* noalias %.x, [0 x %_elem_type_of_y]* noalias %.y) {
 ; CHECK-P9-LABEL: test:
 ; CHECK-P9:       # %bb.0: # %test_entry
 ; CHECK-P9-NEXT:    andi. r3, r6, 15
+; CHECK-P9-NEXT:    li r3, 2
+; CHECK-P9-NEXT:    li r10, 1
 ; CHECK-P9-NEXT:    lwz r4, 0(r4)
 ; CHECK-P9-NEXT:    lwz r5, 0(r5)
-; CHECK-P9-NEXT:    li r11, 1
-; CHECK-P9-NEXT:    addic r3, r3, -1
-; CHECK-P9-NEXT:    subfe r10, r3, r3
-; CHECK-P9-NEXT:    li r3, 2
-; CHECK-P9-NEXT:    not r10, r10
-; CHECK-P9-NEXT:    iseleq r3, r11, r3
+; CHECK-P9-NEXT:    iseleq r3, r10, r3
+; CHECK-P9-NEXT:    subfic r10, r3, 1
 ; CHECK-P9-NEXT:    add r4, r10, r4
 ; CHECK-P9-NEXT:    srawi r4, r4, 4
 ; CHECK-P9-NEXT:    addze r4, r4
 ; CHECK-P9-NEXT:    srawi r5, r5, 1
 ; CHECK-P9-NEXT:    slwi r4, r4, 4
 ; CHECK-P9-NEXT:    addze r5, r5
 ; CHECK-P9-NEXT:    sub r4, r4, r10
 ; CHECK-P9-NEXT:    cmpw r3, r4

 ; CHECK-P9-NEXT:    stxv vs2, 0(r5)
 ; CHECK-P9-NEXT:    addi r5, r5, 128
 ; CHECK-P9-NEXT:    bdnz .LBB0_2
 ; CHECK-P9-NEXT:  # %bb.3: # %_return_bb
 ; CHECK-P9-NEXT:    blr
 ;
 ; CHECK-P10-LABEL: test:
 ; CHECK-P10:       # %bb.0: # %test_entry
-; CHECK-P10-NEXT:    lwz r4, 0(r4)
 ; CHECK-P10-NEXT:    andi. r3, r6, 15
 ; CHECK-P10-NEXT:    li r3, 2
 ; CHECK-P10-NEXT:    li r10, 1
+; CHECK-P10-NEXT:    lwz r4, 0(r4)
 ; CHECK-P10-NEXT:    lwz r5, 0(r5)
 ; CHECK-P10-NEXT:    iseleq r3, r10, r3
-; CHECK-P10-NEXT:    setnbc r10, eq
-; CHECK-P10-NEXT:    not r10, r10
+; CHECK-P10-NEXT:    subfic r10, r3, 1
 ; CHECK-P10-NEXT:    add r4, r10, r4
 ; CHECK-P10-NEXT:    srawi r4, r4, 4
 ; CHECK-P10-NEXT:    addze r4, r4
 ; CHECK-P10-NEXT:    srawi r5, r5, 1
 ; CHECK-P10-NEXT:    slwi r4, r4, 4
 ; CHECK-P10-NEXT:    addze r5, r5
 ; CHECK-P10-NEXT:    sub r4, r4, r10
 ; CHECK-P10-NEXT:    cmpw r3, r4

llvm/test/CodeGen/X86/memcmp-more-load-pairs-x32.ll

 ; X86-NEXT:    jne .LBB9_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 2(%eax), %eax
 ; X86-NEXT:    movzbl 2(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
 ; X86-NEXT:  .LBB9_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpw %si, %dx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind {
 ; X86-LABEL: length3_eq:

 ; X86-NEXT:    jne .LBB16_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 4(%eax), %eax
 ; X86-NEXT:    movzbl 4(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
 ; X86-NEXT:  .LBB16_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %esi, %edx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind {
 ; X86-LABEL: length5_eq:

 ; X86-NEXT:    cmpl %esi, %edx
 ; X86-NEXT:    jne .LBB18_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 4(%eax), %eax
 ; X86-NEXT:    movzbl 4(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    jmp .LBB18_2
 ; X86-NEXT:  .LBB18_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %esi, %edx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB18_2: # %endblock
 ; X86-NEXT:    shrl $31, %eax
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 5) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB19_3
 ; X86-NEXT:  .LBB19_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB19_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 7) nounwind
   ret i32 %m
 }
 
 define i1 @length7_eq(ptr %X, ptr %Y) nounwind {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB21_3
 ; X86-NEXT:  .LBB21_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB21_3: # %endblock
 ; X86-NEXT:    shrl $31, %eax
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 7) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB22_3
 ; X86-NEXT:  .LBB22_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB22_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 8) nounwind
   ret i32 %m
 }
 
 define i1 @length8_eq(ptr %X, ptr %Y) nounwind {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB29_4
 ; X86-NEXT:  .LBB29_3: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB29_4: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 12) nounwind
   ret i32 %m
 }
 
 define i1 @length13_eq(ptr %X, ptr %Y) nounwind {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB33_5
 ; X86-NEXT:  .LBB33_4: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB33_5: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 16) nounwind
   ret i32 %m
 }
 
 define i1 @length16_eq(ptr %x, ptr %y) nounwind {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB35_5
 ; X86-NEXT:  .LBB35_4: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB35_5: # %endblock
 ; X86-NEXT:    shrl $31, %eax
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i32 16) nounwind
   %cmp = icmp slt i32 %call, 0
   ret i1 %cmp

 ; X86-NEXT:    bswapl %eax
 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    cmpl %ecx, %eax
 ; X86-NEXT:    je .LBB36_5
 ; X86-NEXT:  .LBB36_4: # %res_block
 ; X86-NEXT:    xorl %edx, %edx
 ; X86-NEXT:    cmpl %ecx, %eax
-; X86-NEXT:    setae %dl
-; X86-NEXT:    leal -1(%edx,%edx), %edx
+; X86-NEXT:    sbbl %edx, %edx
+; X86-NEXT:    orl $1, %edx
 ; X86-NEXT:  .LBB36_5: # %endblock
 ; X86-NEXT:    testl %edx, %edx
 ; X86-NEXT:    setg %al
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i32 16) nounwind
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp

llvm/test/CodeGen/X86/memcmp-more-load-pairs.ll

; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 2) nounwind nobuiltin
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length3(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length3:
 ; X64:       # %bb.0:
-; X64-NEXT:    movzwl (%rdi), %eax
-; X64-NEXT:    movzwl (%rsi), %ecx
-; X64-NEXT:    rolw $8, %ax
+; X64-NEXT:    movzwl (%rdi), %ecx
+; X64-NEXT:    movzwl (%rsi), %edx
 ; X64-NEXT:    rolw $8, %cx
-; X64-NEXT:    cmpw %cx, %ax
+; X64-NEXT:    rolw $8, %dx
+; X64-NEXT:    cmpw %dx, %cx
 ; X64-NEXT:    jne .LBB9_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 2(%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB9_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpw %dx, %cx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length3_eq:
 ; X64:       # %bb.0:
; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr getelementptr inbounds ([513 x i8], ptr @.str, i32 0, i32 1), i64 4) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length5(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length5:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    cmpl %ecx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    jne .LBB16_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 4(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB16_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length5_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl (%rdi), %eax
 ; X64-NEXT:    xorl (%rsi), %eax
 ; X64-NEXT:    movzbl 4(%rdi), %ecx
 ; X64-NEXT:    xorb 4(%rsi), %cl
 ; X64-NEXT:    movzbl %cl, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    setne %al
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 
 define i1 @length5_lt(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length5_lt:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    cmpl %ecx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    jne .LBB18_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 4(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB18_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c
 }
 

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    je .LBB19_3
 ; X64-NEXT:  .LBB19_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB19_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 7) nounwind
   ret i32 %m
 }
 
 define i1 @length7_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length7_eq:

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    je .LBB21_3
 ; X64-NEXT:  .LBB21_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB21_3: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 7) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c
 }

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB29_3
 ; X64-NEXT:  .LBB29_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB29_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 12) nounwind
   ret i32 %m
 }
 
 define i1 @length13_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length13_eq:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB33_3
 ; X64-NEXT:  .LBB33_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB33_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 16) nounwind
   ret i32 %m
 }
 
 define i1 @length16_eq(ptr %x, ptr %y) nounwind {
 ; X64-SSE2-LABEL: length16_eq:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB35_3
 ; X64-NEXT:  .LBB35_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB35_3: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 16) nounwind
   %cmp = icmp slt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rax
 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
 ; X64-NEXT:    je .LBB36_3
 ; X64-NEXT:  .LBB36_2: # %res_block
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    setae %dl
-; X64-NEXT:    leal -1(%rdx,%rdx), %edx
+; X64-NEXT:    sbbl %edx, %edx
+; X64-NEXT:    orl $1, %edx
 ; X64-NEXT:  .LBB36_3: # %endblock
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    setg %al
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 16) nounwind
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB38_4
 ; X64-NEXT:  .LBB38_3: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB38_4: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 24) nounwind
   ret i32 %m
 }
 
 define i1 @length24_eq(ptr %x, ptr %y) nounwind {
 ; X64-SSE2-LABEL: length24_eq:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB40_4
 ; X64-NEXT:  .LBB40_3: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB40_4: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 24) nounwind
   %cmp = icmp slt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rax
 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
 ; X64-NEXT:    je .LBB41_4
 ; X64-NEXT:  .LBB41_3: # %res_block
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    setae %dl
-; X64-NEXT:    leal -1(%rdx,%rdx), %edx
+; X64-NEXT:    sbbl %edx, %edx
+; X64-NEXT:    orl $1, %edx
 ; X64-NEXT:  .LBB41_4: # %endblock
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    setg %al
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 24) nounwind
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB43_5
 ; X64-NEXT:  .LBB43_4: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB43_5: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 31) nounwind
   ret i32 %m
 }
 
 define i1 @length31_eq(ptr %x, ptr %y) nounwind {
 ; X64-SSE2-LABEL: length31_eq:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB45_5
 ; X64-NEXT:  .LBB45_4: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB45_5: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 31) nounwind
   %cmp = icmp slt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rax
 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
 ; X64-NEXT:    je .LBB46_5
 ; X64-NEXT:  .LBB46_4: # %res_block
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    setae %dl
-; X64-NEXT:    leal -1(%rdx,%rdx), %edx
+; X64-NEXT:    sbbl %edx, %edx
+; X64-NEXT:    orl $1, %edx
 ; X64-NEXT:  .LBB46_5: # %endblock
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    setg %al
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 31) nounwind
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB49_5
 ; X64-NEXT:  .LBB49_4: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB49_5: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 32) nounwind
   ret i32 %m
 }
 
 ; PR33325 - https://bugs.llvm.org/show_bug.cgi?id=33325
 

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB51_5
 ; X64-NEXT:  .LBB51_4: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB51_5: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 32) nounwind
   %cmp = icmp slt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rax
 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
 ; X64-NEXT:    je .LBB52_5
 ; X64-NEXT:  .LBB52_4: # %res_block
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    setae %dl
-; X64-NEXT:    leal -1(%rdx,%rdx), %edx
+; X64-NEXT:    sbbl %edx, %edx
+; X64-NEXT:    orl $1, %edx
 ; X64-NEXT:  .LBB52_5: # %endblock
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    setg %al
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 32) nounwind
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp
 }

llvm/test/CodeGen/X86/memcmp-optsize-x32.ll

 ; X86-NEXT:    cmpw %si, %dx
 ; X86-NEXT:    jne .LBB4_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 2(%eax), %eax
 ; X86-NEXT:    movzbl 2(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    jmp .LBB4_2
 ; X86-NEXT:  .LBB4_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpw %si, %dx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB4_2: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind optsize {

 ; X86-NEXT:    cmpl %esi, %edx
 ; X86-NEXT:    jne .LBB9_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 4(%eax), %eax
 ; X86-NEXT:    movzbl 4(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    jmp .LBB9_2
 ; X86-NEXT:  .LBB9_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %esi, %edx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB9_2: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind optsize {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB11_3
 ; X86-NEXT:  .LBB11_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB11_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 8) nounwind
   ret i32 %m
 }
 
 define i1 @length8_eq(ptr %X, ptr %Y) nounwind optsize {

llvm/test/CodeGen/X86/memcmp-optsize.ll

; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 2) nounwind nobuiltin
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length3(ptr %X, ptr %Y) nounwind optsize {
 ; X64-LABEL: length3:
 ; X64:       # %bb.0:
-; X64-NEXT:    movzwl (%rdi), %eax
-; X64-NEXT:    movzwl (%rsi), %ecx
-; X64-NEXT:    rolw $8, %ax
+; X64-NEXT:    movzwl (%rdi), %ecx
+; X64-NEXT:    movzwl (%rsi), %edx
 ; X64-NEXT:    rolw $8, %cx
-; X64-NEXT:    cmpw %cx, %ax
+; X64-NEXT:    rolw $8, %dx
+; X64-NEXT:    cmpw %dx, %cx
 ; X64-NEXT:    jne .LBB4_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 2(%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB4_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpw %dx, %cx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind optsize {
 ; X64-LABEL: length3_eq:
 ; X64:       # %bb.0:
; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr getelementptr inbounds ([65 x i8], ptr @.str, i32 0, i32 1), i64 4) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length5(ptr %X, ptr %Y) nounwind optsize {
 ; X64-LABEL: length5:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    cmpl %ecx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    jne .LBB9_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 4(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB9_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind optsize {
 ; X64-LABEL: length5_eq:
 ; X64:       # %bb.0:

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB15_3
 ; X64-NEXT:  .LBB15_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB15_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 12) nounwind
   ret i32 %m
 }
 
 ; PR33329 - https://bugs.llvm.org/show_bug.cgi?id=33329
 

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB16_3
 ; X64-NEXT:  .LBB16_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB16_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 16) nounwind
   ret i32 %m
 }
 
 define i1 @length16_eq(ptr %x, ptr %y) nounwind optsize {
 ; X64-SSE2-LABEL: length16_eq:

llvm/test/CodeGen/X86/memcmp-pgso-x32.ll

 ; X86-NEXT:    cmpw %si, %dx
 ; X86-NEXT:    jne .LBB4_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 2(%eax), %eax
 ; X86-NEXT:    movzbl 2(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    jmp .LBB4_2
 ; X86-NEXT:  .LBB4_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpw %si, %dx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB4_2: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind !prof !14 {

 ; X86-NEXT:    cmpl %esi, %edx
 ; X86-NEXT:    jne .LBB9_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 4(%eax), %eax
 ; X86-NEXT:    movzbl 4(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    jmp .LBB9_2
 ; X86-NEXT:  .LBB9_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %esi, %edx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB9_2: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind !prof !14 {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB11_3
 ; X86-NEXT:  .LBB11_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB11_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 8) nounwind
   ret i32 %m
 }
 
 define i1 @length8_eq(ptr %X, ptr %Y) nounwind !prof !14 {

llvm/test/CodeGen/X86/memcmp-pgso.ll

; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 2) nounwind nobuiltin
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length3(ptr %X, ptr %Y) nounwind !prof !14 {
 ; X64-LABEL: length3:
 ; X64:       # %bb.0:
-; X64-NEXT:    movzwl (%rdi), %eax
-; X64-NEXT:    movzwl (%rsi), %ecx
-; X64-NEXT:    rolw $8, %ax
+; X64-NEXT:    movzwl (%rdi), %ecx
+; X64-NEXT:    movzwl (%rsi), %edx
 ; X64-NEXT:    rolw $8, %cx
-; X64-NEXT:    cmpw %cx, %ax
+; X64-NEXT:    rolw $8, %dx
+; X64-NEXT:    cmpw %dx, %cx
 ; X64-NEXT:    jne .LBB4_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 2(%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB4_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpw %dx, %cx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind !prof !14 {
 ; X64-LABEL: length3_eq:
 ; X64:       # %bb.0:
; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr getelementptr inbounds ([65 x i8], ptr @.str, i32 0, i32 1), i64 4) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length5(ptr %X, ptr %Y) nounwind !prof !14 {
 ; X64-LABEL: length5:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    cmpl %ecx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    jne .LBB9_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 4(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB9_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind !prof !14 {
 ; X64-LABEL: length5_eq:
 ; X64:       # %bb.0:

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB15_3
 ; X64-NEXT:  .LBB15_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB15_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 12) nounwind
   ret i32 %m
 }
 
 ; PR33329 - https://bugs.llvm.org/show_bug.cgi?id=33329
 

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB16_3
 ; X64-NEXT:  .LBB16_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB16_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 16) nounwind
   ret i32 %m
 }
 
 define i1 @length16_eq(ptr %x, ptr %y) nounwind !prof !14 {
 ; X64-SSE2-LABEL: length16_eq:

llvm/test/CodeGen/X86/memcmp-x32.ll

 ; X86-NEXT:    jne .LBB11_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 2(%eax), %eax
 ; X86-NEXT:    movzbl 2(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
 ; X86-NEXT:  .LBB11_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpw %si, %dx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind {
 ; X86-LABEL: length3_eq:

 ; X86-NEXT:    jne .LBB18_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 4(%eax), %eax
 ; X86-NEXT:    movzbl 4(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
 ; X86-NEXT:  .LBB18_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %esi, %edx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind {
 ; X86-LABEL: length5_eq:

 ; X86-NEXT:    cmpl %esi, %edx
 ; X86-NEXT:    jne .LBB20_3
 ; X86-NEXT:  # %bb.1: # %loadbb1
 ; X86-NEXT:    movzbl 4(%eax), %eax
 ; X86-NEXT:    movzbl 4(%ecx), %ecx
 ; X86-NEXT:    subl %ecx, %eax
 ; X86-NEXT:    jmp .LBB20_2
 ; X86-NEXT:  .LBB20_3: # %res_block
-; X86-NEXT:    setae %al
-; X86-NEXT:    movzbl %al, %eax
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    xorl %eax, %eax
+; X86-NEXT:    cmpl %esi, %edx
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB20_2: # %endblock
 ; X86-NEXT:    shrl $31, %eax
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 5) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB21_3
 ; X86-NEXT:  .LBB21_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB21_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 7) nounwind
   ret i32 %m
 }
 
 define i1 @length7_lt(ptr %X, ptr %Y) nounwind {

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB22_3
 ; X86-NEXT:  .LBB22_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB22_3: # %endblock
 ; X86-NEXT:    shrl $31, %eax
 ; X86-NEXT:    # kill: def $al killed $al killed $eax
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 7) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c

 ; X86-NEXT:    bswapl %ecx
 ; X86-NEXT:    bswapl %edx
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
 ; X86-NEXT:    je .LBB24_3
 ; X86-NEXT:  .LBB24_2: # %res_block
 ; X86-NEXT:    xorl %eax, %eax
 ; X86-NEXT:    cmpl %edx, %ecx
-; X86-NEXT:    setae %al
-; X86-NEXT:    leal -1(%eax,%eax), %eax
+; X86-NEXT:    sbbl %eax, %eax
+; X86-NEXT:    orl $1, %eax
 ; X86-NEXT:  .LBB24_3: # %endblock
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    retl
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i32 8) nounwind
   ret i32 %m
 }
 
 define i1 @length8_eq(ptr %X, ptr %Y) nounwind {

llvm/test/CodeGen/X86/memcmp.ll

; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 2) nounwind nobuiltin
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length3(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length3:
 ; X64:       # %bb.0:
-; X64-NEXT:    movzwl (%rdi), %eax
-; X64-NEXT:    movzwl (%rsi), %ecx
-; X64-NEXT:    rolw $8, %ax
+; X64-NEXT:    movzwl (%rdi), %ecx
+; X64-NEXT:    movzwl (%rsi), %edx
 ; X64-NEXT:    rolw $8, %cx
-; X64-NEXT:    cmpw %cx, %ax
+; X64-NEXT:    rolw $8, %dx
+; X64-NEXT:    cmpw %dx, %cx
 ; X64-NEXT:    jne .LBB11_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 2(%rdi), %eax
 ; X64-NEXT:    movzbl 2(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB11_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpw %dx, %cx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 3) nounwind
   ret i32 %m
 }
 
 define i1 @length3_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length3_eq:
 ; X64:       # %bb.0:
; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr getelementptr inbounds ([513 x i8], ptr @.str, i32 0, i32 1), i64 4) nounwind
   %c = icmp eq i32 %m, 0
   ret i1 %c
 }
 
 define i32 @length5(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length5:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    cmpl %ecx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    jne .LBB18_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 4(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB18_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   ret i32 %m
 }
 
 define i1 @length5_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length5_eq:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl (%rdi), %eax
 ; X64-NEXT:    xorl (%rsi), %eax
 ; X64-NEXT:    movzbl 4(%rdi), %ecx
 ; X64-NEXT:    xorb 4(%rsi), %cl
 ; X64-NEXT:    movzbl %cl, %ecx
 ; X64-NEXT:    orl %eax, %ecx
 ; X64-NEXT:    setne %al
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   %c = icmp ne i32 %m, 0
   ret i1 %c
 }
 
 define i1 @length5_lt(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length5_lt:
 ; X64:       # %bb.0:
-; X64-NEXT:    movl (%rdi), %eax
-; X64-NEXT:    movl (%rsi), %ecx
-; X64-NEXT:    bswapl %eax
+; X64-NEXT:    movl (%rdi), %ecx
+; X64-NEXT:    movl (%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
-; X64-NEXT:    cmpl %ecx, %eax
+; X64-NEXT:    bswapl %edx
+; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    jne .LBB20_3
 ; X64-NEXT:  # %bb.1: # %loadbb1
 ; X64-NEXT:    movzbl 4(%rdi), %eax
 ; X64-NEXT:    movzbl 4(%rsi), %ecx
 ; X64-NEXT:    subl %ecx, %eax
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
 ; X64-NEXT:  .LBB20_3: # %res_block
-; X64-NEXT:    setae %al
-; X64-NEXT:    movzbl %al, %eax
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    xorl %eax, %eax
+; X64-NEXT:    cmpl %edx, %ecx
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 5) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c
 }
 

 ; X64-NEXT:    movl 3(%rsi), %edx
 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    je .LBB21_3
 ; X64-NEXT:  .LBB21_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB21_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 7) nounwind
   ret i32 %m
 }
 
 define i1 @length7_lt(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length7_lt:

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
 ; X64-NEXT:    je .LBB22_3
 ; X64-NEXT:  .LBB22_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpl %edx, %ecx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB22_3: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 7) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c
 }

 ; X64-NEXT:    bswapl %ecx
 ; X64-NEXT:    bswapl %edx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB31_3
 ; X64-NEXT:  .LBB31_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB31_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 12) nounwind
   ret i32 %m
 }
 
 define i1 @length13_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length13_eq:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB34_3
 ; X64-NEXT:  .LBB34_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB34_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 15) nounwind
   ret i32 %m
 }
 
 define i1 @length15_lt(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length15_lt:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB35_3
 ; X64-NEXT:  .LBB35_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB35_3: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 15) nounwind
   %c = icmp slt i32 %m, 0
   ret i1 %c
 }

 ; X64-NEXT:    movq 7(%rdi), %rdx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rcx, %rdx
 ; X64-NEXT:    je .LBB36_3
 ; X64-NEXT:  .LBB36_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rcx, %rdx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB36_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr getelementptr inbounds ([513 x i8], ptr @.str, i32 0, i32 1), i64 15) nounwind
   ret i32 %m
 }
 
 define i1 @length15_eq(ptr %X, ptr %Y) nounwind {
 ; X64-LABEL: length15_eq:

 ; X64-NEXT:    movq 7(%rdi), %rcx
 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rax, %rcx
 ; X64-NEXT:    je .LBB38_3
 ; X64-NEXT:  .LBB38_2: # %res_block
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rax, %rcx
-; X64-NEXT:    setae %dl
-; X64-NEXT:    leal -1(%rdx,%rdx), %edx
+; X64-NEXT:    sbbl %edx, %edx
+; X64-NEXT:    orl $1, %edx
 ; X64-NEXT:  .LBB38_3: # %endblock
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    setg %al
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr getelementptr inbounds ([513 x i8], ptr @.str, i32 0, i32 1), i64 15) nounwind
   %c = icmp sgt i32 %m, 0
   ret i1 %c
 }

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB39_3
 ; X64-NEXT:  .LBB39_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB39_3: # %endblock
 ; X64-NEXT:    retq
   %m = tail call i32 @memcmp(ptr %X, ptr %Y, i64 16) nounwind
   ret i32 %m
 }
 
 define i1 @length16_eq(ptr %x, ptr %y) nounwind {
 ; X64-SSE2-LABEL: length16_eq:

 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    bswapq %rdx
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
 ; X64-NEXT:    je .LBB41_3
 ; X64-NEXT:  .LBB41_2: # %res_block
 ; X64-NEXT:    xorl %eax, %eax
 ; X64-NEXT:    cmpq %rdx, %rcx
-; X64-NEXT:    setae %al
-; X64-NEXT:    leal -1(%rax,%rax), %eax
+; X64-NEXT:    sbbl %eax, %eax
+; X64-NEXT:    orl $1, %eax
 ; X64-NEXT:  .LBB41_3: # %endblock
 ; X64-NEXT:    shrl $31, %eax
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 16) nounwind
   %cmp = icmp slt i32 %call, 0
   ret i1 %cmp
 }

 ; X64-NEXT:    bswapq %rax
 ; X64-NEXT:    bswapq %rcx
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
 ; X64-NEXT:    je .LBB42_3
 ; X64-NEXT:  .LBB42_2: # %res_block
 ; X64-NEXT:    xorl %edx, %edx
 ; X64-NEXT:    cmpq %rcx, %rax
-; X64-NEXT:    setae %dl
-; X64-NEXT:    leal -1(%rdx,%rdx), %edx
+; X64-NEXT:    sbbl %edx, %edx
+; X64-NEXT:    orl $1, %edx
 ; X64-NEXT:  .LBB42_3: # %endblock
 ; X64-NEXT:    testl %edx, %edx
 ; X64-NEXT:    setg %al
 ; X64-NEXT:    retq
   %call = tail call i32 @memcmp(ptr %x, ptr %y, i64 16) nounwind
   %cmp = icmp sgt i32 %call, 0
   ret i1 %cmp
 }

llvm/test/CodeGen/X86/midpoint-int.ll

 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i64_signed_reg_reg:
 ; X86:       # %bb.0:
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    cmpl %ecx, %eax
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %edi, %edx
-; X86-NEXT:    sbbl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl $-1, %ebx
+; X86-NEXT:    sbbl %ebp, %edx
+; X86-NEXT:    setl %dl
+; X86-NEXT:    movzbl %dl, %ebx
 ; X86-NEXT:    jl .LBB5_1
 ; X86-NEXT:  # %bb.2:
-; X86-NEXT:    xorl %ebp, %ebp
-; X86-NEXT:    movl $1, %ebx
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl %ecx, %esi
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movl %ebp, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    jmp .LBB5_3
 ; X86-NEXT:  .LBB5_1:
-; X86-NEXT:    movl $-1, %ebp
 ; X86-NEXT:    movl %edi, %edx
 ; X86-NEXT:    movl %eax, %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movl %ebp, %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:  .LBB5_3:
+; X86-NEXT:    negl %ebx
+; X86-NEXT:    movl %ebx, %ebp
+; X86-NEXT:    orl $1, %ebp
 ; X86-NEXT:    subl %esi, %eax
 ; X86-NEXT:    sbbl %edx, %edi
 ; X86-NEXT:    shrdl $1, %edi, %eax
-; X86-NEXT:    imull %eax, %ebp
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    addl %ebp, %edx
+; X86-NEXT:    imull %eax, %ebx
+; X86-NEXT:    mull %ebp
+; X86-NEXT:    addl %ebx, %edx
 ; X86-NEXT:    shrl %edi
-; X86-NEXT:    imull %ebx, %edi
+; X86-NEXT:    imull %ebp, %edi
 ; X86-NEXT:    addl %edi, %edx
-; X86-NEXT:    addl %ecx, %eax
-; X86-NEXT:    adcl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    addl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    adcl %ecx, %edx
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    popl %ebp
 ; X86-NEXT:    retl
   %t3 = icmp sgt i64 %a1, %a2 ; signed
   %t4 = select i1 %t3, i64 -1, i64 1
   %t5 = select i1 %t3, i64 %a2, i64 %a1

 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i64_unsigned_reg_reg:
 ; X86:       # %bb.0:
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    cmpl %ecx, %eax
+; X86-NEXT:    xorl %ebx, %ebx
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %edi, %edx
-; X86-NEXT:    sbbl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl $-1, %ebx
-; X86-NEXT:    jb .LBB6_1
+; X86-NEXT:    sbbl %ebp, %edx
+; X86-NEXT:    setb %dl
+; X86-NEXT:    sbbl %ebx, %ebx
+; X86-NEXT:    testb %dl, %dl
+; X86-NEXT:    jne .LBB6_1
 ; X86-NEXT:  # %bb.2:
-; X86-NEXT:    xorl %ebp, %ebp
-; X86-NEXT:    movl $1, %ebx
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl %ecx, %esi
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movl %ebp, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    jmp .LBB6_3
 ; X86-NEXT:  .LBB6_1:
-; X86-NEXT:    movl $-1, %ebp
 ; X86-NEXT:    movl %edi, %edx
 ; X86-NEXT:    movl %eax, %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movl %ebp, %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:  .LBB6_3:
+; X86-NEXT:    movl %ebx, %ebp
+; X86-NEXT:    orl $1, %ebp
 ; X86-NEXT:    subl %esi, %eax
 ; X86-NEXT:    sbbl %edx, %edi
 ; X86-NEXT:    shrdl $1, %edi, %eax
-; X86-NEXT:    imull %eax, %ebp
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    addl %ebp, %edx
+; X86-NEXT:    imull %eax, %ebx
+; X86-NEXT:    mull %ebp
+; X86-NEXT:    addl %ebx, %edx
 ; X86-NEXT:    shrl %edi
-; X86-NEXT:    imull %ebx, %edi
+; X86-NEXT:    imull %ebp, %edi
 ; X86-NEXT:    addl %edi, %edx
-; X86-NEXT:    addl %ecx, %eax
-; X86-NEXT:    adcl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    addl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    adcl %ecx, %edx
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    popl %ebp
 ; X86-NEXT:    retl
   %t3 = icmp ugt i64 %a1, %a2
   %t4 = select i1 %t3, i64 -1, i64 1
   %t5 = select i1 %t3, i64 %a2, i64 %a1

 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl (%ecx), %esi
 ; X86-NEXT:    movl 4(%ecx), %ecx
 ; X86-NEXT:    cmpl %esi, %eax
 ; X86-NEXT:    movl %edi, %edx
 ; X86-NEXT:    sbbl %ecx, %edx
-; X86-NEXT:    movl $-1, %ebx
+; X86-NEXT:    setl %dl
+; X86-NEXT:    movzbl %dl, %ebx
 ; X86-NEXT:    jl .LBB7_1
 ; X86-NEXT:  # %bb.2:
-; X86-NEXT:    xorl %ebp, %ebp
-; X86-NEXT:    movl $1, %ebx
 ; X86-NEXT:    movl %ecx, (%esp) # 4-byte Spill
 ; X86-NEXT:    movl %esi, %edx
 ; X86-NEXT:    jmp .LBB7_3
 ; X86-NEXT:  .LBB7_1:
-; X86-NEXT:    movl $-1, %ebp
 ; X86-NEXT:    movl %edi, (%esp) # 4-byte Spill
 ; X86-NEXT:    movl %eax, %edx
 ; X86-NEXT:    movl %ecx, %edi
 ; X86-NEXT:    movl %esi, %eax
 ; X86-NEXT:  .LBB7_3:
+; X86-NEXT:    negl %ebx
+; X86-NEXT:    movl %ebx, %ebp
+; X86-NEXT:    orl $1, %ebp
 ; X86-NEXT:    subl %edx, %eax
 ; X86-NEXT:    sbbl (%esp), %edi # 4-byte Folded Reload
 ; X86-NEXT:    shrdl $1, %edi, %eax
-; X86-NEXT:    imull %eax, %ebp
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    addl %ebp, %edx
+; X86-NEXT:    imull %eax, %ebx
+; X86-NEXT:    mull %ebp
+; X86-NEXT:    addl %ebx, %edx
 ; X86-NEXT:    shrl %edi
-; X86-NEXT:    imull %ebx, %edi
+; X86-NEXT:    imull %ebp, %edi
 ; X86-NEXT:    addl %edi, %edx
 ; X86-NEXT:    addl %esi, %eax
 ; X86-NEXT:    adcl %ecx, %edx
 ; X86-NEXT:    addl $4, %esp
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    popl %ebp

 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i64_signed_reg_mem:
 ; X86:       # %bb.0:
 ; X86-NEXT:    pushl %ebp
 ; X86-NEXT:    pushl %ebx
 ; X86-NEXT:    pushl %edi
 ; X86-NEXT:    pushl %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %ebp
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X86-NEXT:    movl (%edx), %eax
 ; X86-NEXT:    movl 4(%edx), %edi
-; X86-NEXT:    cmpl %ecx, %eax
+; X86-NEXT:    cmpl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl %edi, %edx
-; X86-NEXT:    sbbl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl $-1, %ebx
+; X86-NEXT:    sbbl %ebp, %edx
+; X86-NEXT:    setl %dl
+; X86-NEXT:    movzbl %dl, %ebx
 ; X86-NEXT:    jl .LBB8_1
 ; X86-NEXT:  # %bb.2:
-; X86-NEXT:    xorl %ebp, %ebp
-; X86-NEXT:    movl $1, %ebx
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; X86-NEXT:    movl %ecx, %esi
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movl %ebp, %edx
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X86-NEXT:    jmp .LBB8_3
 ; X86-NEXT:  .LBB8_1:
-; X86-NEXT:    movl $-1, %ebp
 ; X86-NEXT:    movl %edi, %edx
 ; X86-NEXT:    movl %eax, %esi
-; X86-NEXT:    movl {{[0-9]+}}(%esp), %edi
-; X86-NEXT:    movl %ecx, %eax
+; X86-NEXT:    movl %ebp, %ecx
+; X86-NEXT:    movl %ebp, %edi
+; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:  .LBB8_3:
+; X86-NEXT:    negl %ebx
+; X86-NEXT:    movl %ebx, %ebp
+; X86-NEXT:    orl $1, %ebp
 ; X86-NEXT:    subl %esi, %eax
 ; X86-NEXT:    sbbl %edx, %edi
 ; X86-NEXT:    shrdl $1, %edi, %eax
-; X86-NEXT:    imull %eax, %ebp
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    addl %ebp, %edx
+; X86-NEXT:    imull %eax, %ebx
+; X86-NEXT:    mull %ebp
+; X86-NEXT:    addl %ebx, %edx
 ; X86-NEXT:    shrl %edi
-; X86-NEXT:    imull %ebx, %edi
+; X86-NEXT:    imull %ebp, %edi
 ; X86-NEXT:    addl %edi, %edx
-; X86-NEXT:    addl %ecx, %eax
-; X86-NEXT:    adcl {{[0-9]+}}(%esp), %edx
+; X86-NEXT:    addl {{[0-9]+}}(%esp), %eax
+; X86-NEXT:    adcl %ecx, %edx
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    popl %ebp
 ; X86-NEXT:    retl
   %a2 = load i64, ptr %a2_addr
   %t3 = icmp sgt i64 %a1, %a2 ; signed
   %t4 = select i1 %t3, i64 -1, i64 1

 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl (%eax), %esi
 ; X86-NEXT:    movl 4(%eax), %ecx
 ; X86-NEXT:    movl (%edx), %eax
 ; X86-NEXT:    movl 4(%edx), %edi
 ; X86-NEXT:    cmpl %esi, %eax
 ; X86-NEXT:    movl %edi, %edx
 ; X86-NEXT:    sbbl %ecx, %edx
-; X86-NEXT:    movl $-1, %ebx
+; X86-NEXT:    setl %dl
+; X86-NEXT:    movzbl %dl, %ebx
 ; X86-NEXT:    jl .LBB9_1
 ; X86-NEXT:  # %bb.2:
-; X86-NEXT:    xorl %ebp, %ebp
-; X86-NEXT:    movl $1, %ebx
 ; X86-NEXT:    movl %ecx, (%esp) # 4-byte Spill
 ; X86-NEXT:    movl %esi, %edx
 ; X86-NEXT:    jmp .LBB9_3
 ; X86-NEXT:  .LBB9_1:
-; X86-NEXT:    movl $-1, %ebp
 ; X86-NEXT:    movl %edi, (%esp) # 4-byte Spill
 ; X86-NEXT:    movl %eax, %edx
 ; X86-NEXT:    movl %ecx, %edi
 ; X86-NEXT:    movl %esi, %eax
 ; X86-NEXT:  .LBB9_3:
+; X86-NEXT:    negl %ebx
+; X86-NEXT:    movl %ebx, %ebp
+; X86-NEXT:    orl $1, %ebp
 ; X86-NEXT:    subl %edx, %eax
 ; X86-NEXT:    sbbl (%esp), %edi # 4-byte Folded Reload
 ; X86-NEXT:    shrdl $1, %edi, %eax
-; X86-NEXT:    imull %eax, %ebp
-; X86-NEXT:    mull %ebx
-; X86-NEXT:    addl %ebp, %edx
+; X86-NEXT:    imull %eax, %ebx
+; X86-NEXT:    mull %ebp
+; X86-NEXT:    addl %ebx, %edx
 ; X86-NEXT:    shrl %edi
-; X86-NEXT:    imull %ebx, %edi
+; X86-NEXT:    imull %ebp, %edi
 ; X86-NEXT:    addl %edi, %edx
 ; X86-NEXT:    addl %esi, %eax
 ; X86-NEXT:    adcl %ecx, %edx
 ; X86-NEXT:    addl $4, %esp
 ; X86-NEXT:    popl %esi
 ; X86-NEXT:    popl %edi
 ; X86-NEXT:    popl %ebx
 ; X86-NEXT:    popl %ebp

 
 ; Values come from regs
 
 define i8 @scalar_i8_signed_reg_reg(i8 %a1, i8 %a2) nounwind {
 ; X64-LABEL: scalar_i8_signed_reg_reg:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %eax
 ; X64-NEXT:    cmpb %al, %dil
-; X64-NEXT:    setle %cl
+; X64-NEXT:    setg %cl
 ; X64-NEXT:    movl %edi, %edx
 ; X64-NEXT:    cmovgl %esi, %edx
 ; X64-NEXT:    cmovgl %edi, %eax
-; X64-NEXT:    addb %cl, %cl
-; X64-NEXT:    decb %cl
+; X64-NEXT:    negb %cl
+; X64-NEXT:    orb $1, %cl
 ; X64-NEXT:    subb %dl, %al
 ; X64-NEXT:    shrb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    mulb %cl
 ; X64-NEXT:    addb %dil, %al
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i8_signed_reg_reg:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    cmpb %al, %cl
-; X86-NEXT:    setle %dl
+; X86-NEXT:    setg %dl
 ; X86-NEXT:    jg .LBB15_1
 ; X86-NEXT:  # %bb.2:
 ; X86-NEXT:    movb %cl, %ah
 ; X86-NEXT:    jmp .LBB15_3
 ; X86-NEXT:  .LBB15_1:
 ; X86-NEXT:    movb %al, %ah
 ; X86-NEXT:    movb %cl, %al
 ; X86-NEXT:  .LBB15_3:
 ; X86-NEXT:    subb %ah, %al
-; X86-NEXT:    addb %dl, %dl
-; X86-NEXT:    decb %dl
+; X86-NEXT:    negb %dl
+; X86-NEXT:    orb $1, %dl
 ; X86-NEXT:    shrb %al
 ; X86-NEXT:    mulb %dl
 ; X86-NEXT:    addb %cl, %al
 ; X86-NEXT:    retl
   %t3 = icmp sgt i8 %a1, %a2 ; signed
   %t4 = select i1 %t3, i8 -1, i8 1
   %t5 = select i1 %t3, i8 %a2, i8 %a1
   %t6 = select i1 %t3, i8 %a1, i8 %a2
   %t7 = sub i8 %t6, %t5
   %t8 = lshr i8 %t7, 1
   %t9 = mul nsw i8 %t8, %t4 ; signed
   %a10 = add nsw i8 %t9, %a1 ; signed
   ret i8 %a10
 }
 
 define i8 @scalar_i8_unsigned_reg_reg(i8 %a1, i8 %a2) nounwind {
 ; X64-LABEL: scalar_i8_unsigned_reg_reg:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movl %esi, %eax
 ; X64-NEXT:    cmpb %al, %dil
-; X64-NEXT:    setbe %cl
+; X64-NEXT:    seta %cl
 ; X64-NEXT:    movl %edi, %edx
 ; X64-NEXT:    cmoval %esi, %edx
 ; X64-NEXT:    cmoval %edi, %eax
-; X64-NEXT:    addb %cl, %cl
-; X64-NEXT:    decb %cl
+; X64-NEXT:    negb %cl
+; X64-NEXT:    orb $1, %cl
 ; X64-NEXT:    subb %dl, %al
 ; X64-NEXT:    shrb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    mulb %cl
 ; X64-NEXT:    addb %dil, %al
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i8_unsigned_reg_reg:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    cmpb %al, %cl
-; X86-NEXT:    setbe %dl
+; X86-NEXT:    seta %dl
 ; X86-NEXT:    ja .LBB16_1
 ; X86-NEXT:  # %bb.2:
 ; X86-NEXT:    movb %cl, %ah
 ; X86-NEXT:    jmp .LBB16_3
 ; X86-NEXT:  .LBB16_1:
 ; X86-NEXT:    movb %al, %ah
 ; X86-NEXT:    movb %cl, %al
 ; X86-NEXT:  .LBB16_3:
 ; X86-NEXT:    subb %ah, %al
-; X86-NEXT:    addb %dl, %dl
-; X86-NEXT:    decb %dl
+; X86-NEXT:    negb %dl
+; X86-NEXT:    orb $1, %dl
 ; X86-NEXT:    shrb %al
 ; X86-NEXT:    mulb %dl
 ; X86-NEXT:    addb %cl, %al
 ; X86-NEXT:    retl
   %t3 = icmp ugt i8 %a1, %a2
   %t4 = select i1 %t3, i8 -1, i8 1
   %t5 = select i1 %t3, i8 %a2, i8 %a1
   %t6 = select i1 %t3, i8 %a1, i8 %a2
   %t7 = sub i8 %t6, %t5
   %t8 = lshr i8 %t7, 1
   %t9 = mul i8 %t8, %t4
   %a10 = add i8 %t9, %a1
   ret i8 %a10
 }
 
 ; Values are loaded. Only check signed case.
 
 define i8 @scalar_i8_signed_mem_reg(ptr %a1_addr, i8 %a2) nounwind {
 ; X64-LABEL: scalar_i8_signed_mem_reg:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl (%rdi), %ecx
 ; X64-NEXT:    cmpb %sil, %cl
-; X64-NEXT:    setle %dl
+; X64-NEXT:    setg %dl
 ; X64-NEXT:    movl %ecx, %edi
 ; X64-NEXT:    cmovgl %esi, %edi
 ; X64-NEXT:    movl %ecx, %eax
 ; X64-NEXT:    cmovlel %esi, %eax
-; X64-NEXT:    addb %dl, %dl
-; X64-NEXT:    decb %dl
+; X64-NEXT:    negb %dl
+; X64-NEXT:    orb $1, %dl
 ; X64-NEXT:    subb %dil, %al
 ; X64-NEXT:    shrb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    mulb %dl
 ; X64-NEXT:    addb %cl, %al
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i8_signed_mem_reg:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movzbl (%ecx), %ecx
 ; X86-NEXT:    cmpb %al, %cl
-; X86-NEXT:    setle %dl
+; X86-NEXT:    setg %dl
 ; X86-NEXT:    jg .LBB17_1
 ; X86-NEXT:  # %bb.2:
 ; X86-NEXT:    movb %cl, %ah
 ; X86-NEXT:    jmp .LBB17_3
 ; X86-NEXT:  .LBB17_1:
 ; X86-NEXT:    movb %al, %ah
 ; X86-NEXT:    movb %cl, %al
 ; X86-NEXT:  .LBB17_3:
 ; X86-NEXT:    subb %ah, %al
-; X86-NEXT:    addb %dl, %dl
-; X86-NEXT:    decb %dl
+; X86-NEXT:    negb %dl
+; X86-NEXT:    orb $1, %dl
 ; X86-NEXT:    shrb %al
 ; X86-NEXT:    mulb %dl
 ; X86-NEXT:    addb %cl, %al
 ; X86-NEXT:    retl
   %a1 = load i8, ptr %a1_addr
   %t3 = icmp sgt i8 %a1, %a2 ; signed
   %t4 = select i1 %t3, i8 -1, i8 1
   %t5 = select i1 %t3, i8 %a2, i8 %a1
   %t6 = select i1 %t3, i8 %a1, i8 %a2
   %t7 = sub i8 %t6, %t5
   %t8 = lshr i8 %t7, 1
   %t9 = mul nsw i8 %t8, %t4 ; signed
   %a10 = add nsw i8 %t9, %a1 ; signed
   ret i8 %a10
 }
 
 define i8 @scalar_i8_signed_reg_mem(i8 %a1, ptr %a2_addr) nounwind {
 ; X64-LABEL: scalar_i8_signed_reg_mem:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl (%rsi), %eax
 ; X64-NEXT:    cmpb %al, %dil
-; X64-NEXT:    setle %cl
+; X64-NEXT:    setg %cl
 ; X64-NEXT:    movl %edi, %edx
 ; X64-NEXT:    cmovgl %eax, %edx
 ; X64-NEXT:    cmovgl %edi, %eax
-; X64-NEXT:    addb %cl, %cl
-; X64-NEXT:    decb %cl
+; X64-NEXT:    negb %cl
+; X64-NEXT:    orb $1, %cl
 ; X64-NEXT:    subb %dl, %al
 ; X64-NEXT:    shrb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    mulb %cl
 ; X64-NEXT:    addb %dil, %al
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i8_signed_reg_mem:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movzbl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movzbl (%eax), %eax
 ; X86-NEXT:    cmpb %al, %cl
-; X86-NEXT:    setle %dl
+; X86-NEXT:    setg %dl
 ; X86-NEXT:    jg .LBB18_1
 ; X86-NEXT:  # %bb.2:
 ; X86-NEXT:    movb %cl, %ah
 ; X86-NEXT:    jmp .LBB18_3
 ; X86-NEXT:  .LBB18_1:
 ; X86-NEXT:    movb %al, %ah
 ; X86-NEXT:    movb %cl, %al
 ; X86-NEXT:  .LBB18_3:
 ; X86-NEXT:    subb %ah, %al
-; X86-NEXT:    addb %dl, %dl
-; X86-NEXT:    decb %dl
+; X86-NEXT:    negb %dl
+; X86-NEXT:    orb $1, %dl
 ; X86-NEXT:    shrb %al
 ; X86-NEXT:    mulb %dl
 ; X86-NEXT:    addb %cl, %al
 ; X86-NEXT:    retl
   %a2 = load i8, ptr %a2_addr
   %t3 = icmp sgt i8 %a1, %a2 ; signed
   %t4 = select i1 %t3, i8 -1, i8 1
   %t5 = select i1 %t3, i8 %a2, i8 %a1
   %t6 = select i1 %t3, i8 %a1, i8 %a2
   %t7 = sub i8 %t6, %t5
   %t8 = lshr i8 %t7, 1
   %t9 = mul nsw i8 %t8, %t4 ; signed
   %a10 = add nsw i8 %t9, %a1 ; signed
   ret i8 %a10
 }
 
 define i8 @scalar_i8_signed_mem_mem(ptr %a1_addr, ptr %a2_addr) nounwind {
 ; X64-LABEL: scalar_i8_signed_mem_mem:
 ; X64:       # %bb.0:
 ; X64-NEXT:    movzbl (%rdi), %ecx
 ; X64-NEXT:    movzbl (%rsi), %eax
 ; X64-NEXT:    cmpb %al, %cl
-; X64-NEXT:    setle %dl
+; X64-NEXT:    setg %dl
 ; X64-NEXT:    movl %ecx, %esi
 ; X64-NEXT:    cmovgl %eax, %esi
 ; X64-NEXT:    cmovgl %ecx, %eax
-; X64-NEXT:    addb %dl, %dl
-; X64-NEXT:    decb %dl
+; X64-NEXT:    negb %dl
+; X64-NEXT:    orb $1, %dl
 ; X64-NEXT:    subb %sil, %al
 ; X64-NEXT:    shrb %al
 ; X64-NEXT:    # kill: def $al killed $al killed $eax
 ; X64-NEXT:    mulb %dl
 ; X64-NEXT:    addb %cl, %al
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: scalar_i8_signed_mem_mem:
 ; X86:       # %bb.0:
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X86-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X86-NEXT:    movzbl (%ecx), %ecx
 ; X86-NEXT:    movzbl (%eax), %eax
 ; X86-NEXT:    cmpb %al, %cl
-; X86-NEXT:    setle %dl
+; X86-NEXT:    setg %dl
 ; X86-NEXT:    jg .LBB19_1
 ; X86-NEXT:  # %bb.2:
 ; X86-NEXT:    movb %cl, %ah
 ; X86-NEXT:    jmp .LBB19_3
 ; X86-NEXT:  .LBB19_1:
 ; X86-NEXT:    movb %al, %ah
 ; X86-NEXT:    movb %cl, %al
 ; X86-NEXT:  .LBB19_3:
 ; X86-NEXT:    subb %ah, %al
-; X86-NEXT:    addb %dl, %dl
-; X86-NEXT:    decb %dl
+; X86-NEXT:    negb %dl
+; X86-NEXT:    orb $1, %dl
 ; X86-NEXT:    shrb %al
 ; X86-NEXT:    mulb %dl
 ; X86-NEXT:    addb %cl, %al
 ; X86-NEXT:    retl
   %a1 = load i8, ptr %a1_addr
   %a2 = load i8, ptr %a2_addr
   %t3 = icmp sgt i8 %a1, %a2 ; signed
   %t4 = select i1 %t3, i8 -1, i8 1
   %t5 = select i1 %t3, i8 %a2, i8 %a1
   %t6 = select i1 %t3, i8 %a1, i8 %a2
   %t7 = sub i8 %t6, %t5
   %t8 = lshr i8 %t7, 1
   %t9 = mul nsw i8 %t8, %t4 ; signed
   %a10 = add nsw i8 %t9, %a1 ; signed
   ret i8 %a10
 }

llvm/test/CodeGen/X86/select.ll

 ; CHECK-NEXT:    orq $1, %rax
 ; CHECK-NEXT:    retq
 ;
 ; ATHLON-LABEL: test10:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    xorl %edx, %edx
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
-; ATHLON-NEXT:    movl $-1, %ecx
-; ATHLON-NEXT:    movl $1, %eax
-; ATHLON-NEXT:    cmovel %ecx, %eax
-; ATHLON-NEXT:    cmovel %ecx, %edx
+; ATHLON-NEXT:    sete %dl
+; ATHLON-NEXT:    negl %edx
+; ATHLON-NEXT:    movl %edx, %eax
+; ATHLON-NEXT:    orl $1, %eax
 ; ATHLON-NEXT:    retl
 ;
 ; MCU-LABEL: test10:
 ; MCU:       # %bb.0:
-; MCU-NEXT:    orl %edx, %eax
-; MCU-NEXT:    movl $-1, %eax
-; MCU-NEXT:    movl $-1, %edx
-; MCU-NEXT:    je .LBB11_2
-; MCU-NEXT:  # %bb.1:
+; MCU-NEXT:    movl %edx, %ecx
 ; MCU-NEXT:    xorl %edx, %edx
-; MCU-NEXT:    movl $1, %eax
-; MCU-NEXT:  .LBB11_2:
+; MCU-NEXT:    orl %ecx, %eax
+; MCU-NEXT:    sete %dl
+; MCU-NEXT:    negl %edx
+; MCU-NEXT:    movl %edx, %eax
+; MCU-NEXT:    orl $1, %eax
 ; MCU-NEXT:    retl
   %cmp = icmp eq i64 %x, 0
   %cond = select i1 %cmp, i64 -1, i64 1
   ret i64 %cond
 }
 
 define i64 @test11(i64 %x, i64 %y) nounwind readnone ssp noredzone {
 ; CHECK-LABEL: test11:

 ; CHECK-NEXT:    orq $42, %rax
 ; CHECK-NEXT:    retq
 ;
 ; ATHLON-LABEL: eqzero_all_ones_or_const:
 ; ATHLON:       ## %bb.0:
 ; ATHLON-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; ATHLON-NEXT:    xorl %edx, %edx
 ; ATHLON-NEXT:    orl {{[0-9]+}}(%esp), %eax
-; ATHLON-NEXT:    movl $-1, %ecx
-; ATHLON-NEXT:    movl $42, %eax
-; ATHLON-NEXT:    cmovel %ecx, %eax
-; ATHLON-NEXT:    cmovel %ecx, %edx
+; ATHLON-NEXT:    sete %dl
+; ATHLON-NEXT:    negl %edx
+; ATHLON-NEXT:    movl %edx, %eax
+; ATHLON-NEXT:    orl $42, %eax
 ; ATHLON-NEXT:    retl
 ;
 ; MCU-LABEL: eqzero_all_ones_or_const:
 ; MCU:       # %bb.0:
-; MCU-NEXT:    orl %edx, %eax
-; MCU-NEXT:    movl $-1, %eax
-; MCU-NEXT:    movl $-1, %edx
-; MCU-NEXT:    je .LBB16_2
-; MCU-NEXT:  # %bb.1:
+; MCU-NEXT:    movl %edx, %ecx
 ; MCU-NEXT:    xorl %edx, %edx
-; MCU-NEXT:    movl $42, %eax
-; MCU-NEXT:  .LBB16_2:
+; MCU-NEXT:    orl %ecx, %eax
+; MCU-NEXT:    sete %dl
+; MCU-NEXT:    negl %edx
+; MCU-NEXT:    movl %edx, %eax
+; MCU-NEXT:    orl $42, %eax
 ; MCU-NEXT:    retl
   %z = icmp eq i64 %x, 0
   %r = select i1 %z, i64 -1, i64 42
   ret i64 %r
 }
 
 define i8 @nezero_all_ones_or_const(i8 %x) {
 ; CHECK-LABEL: nezero_all_ones_or_const:

llvm/test/CodeGen/X86/select_const.ll

 ; CHECK-NEXT:    # kill: def $edi killed $edi def $rdi
 ; CHECK-NEXT:    leal 42(%rdi), %eax
 ; CHECK-NEXT:    retq
   %sel = select i1 %cond, i32 41, i32 42
   ret i32 %sel
 }
 
 ; If the constants differ by a small multiplier, use LEA.
 ; select Cond, C1, C2 --> add (mul (zext Cond), C1-C2), C2 --> LEA C2(Cond * (C1-C2))
 
 define i32 @select_lea_2(i1 zeroext %cond) {
 ; CHECK-LABEL: select_lea_2:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    xorb $1, %dil
-; CHECK-NEXT:    movzbl %dil, %eax
-; CHECK-NEXT:    leal -1(%rax,%rax), %eax
+; CHECK-NEXT:    movl %edi, %eax
+; CHECK-NEXT:    negl %eax
+; CHECK-NEXT:    orl $1, %eax
 ; CHECK-NEXT:    retq
   %sel = select i1 %cond, i32 -1, i32 1
   ret i32 %sel
 }
 
 define i64 @select_lea_3(i1 zeroext %cond) {
 ; CHECK-LABEL: select_lea_3:
 ; CHECK:       # %bb.0:

 define i16 @sel_neg1_1(i32 %x) {
 ; CHECK-LABEL: sel_neg1_1:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    cmpl $43, %edi
 ; CHECK-NEXT:    setl %al
 ; CHECK-NEXT:    leal -1(,%rax,4), %eax
 ; CHECK-NEXT:    # kill: def $ax killed $ax killed $eax
 ; CHECK-NEXT:    retq
   %cmp = icmp sgt i32 %x, 42
   %sel = select i1 %cmp, i16 -1, i16 3
   ret i16 %sel
 }
 
 ; If the comparison is available, the predicate can be inverted.
 
 define i32 @sel_1_neg1_32(i32 %x) {