[SDAG] commute setcc operands to match a subtract

[spatel]

Summary

If we have:

R = sub X, Y
P = cmp Y, X

...then flipping the operands in the compare instruction can allow using a subtract that sets compare flags.

Motivated by diffs in D58875 - not sure if this changes anything there, but this seems like a good thing independent of that.

I've never seen Lanai code before this, so I don't know whether that change is good, bad, or even correct.

[RKSimon]

Apart from the Lanai code (which like @spatel I know very little about) everything LGTM

[spatel]

In D63958#1564460, @RKSimon wrote:

Apart from the Lanai code (which like @spatel I know very little about) everything LGTM

I'm not finding any docs for this target other than the code in LLVM itself. Discussion about whether this target should be included in LLVM from when it first appeared in trunk:
http://lists.llvm.org/pipermail/llvm-dev/2016-February/095118.html

From commit activity, maybe only @jpienaar can say what is happening on that target?

FWIW, there's a more involved version of this transform already in IR (in instcombine although that seems misplaced to me) - see "swapMayExposeCSEOpportunities()".

[jpienaar]

In D63958#1564692, @spatel wrote:

In D63958#1564460, @RKSimon wrote:

Apart from the Lanai code (which like @spatel I know very little about) everything LGTM

I'm not finding any docs for this target other than the code in LLVM itself. Discussion about whether this target should be included in LLVM from when it first appeared in trunk:
http://lists.llvm.org/pipermail/llvm-dev/2016-February/095118.html

From commit activity, maybe only @jpienaar can say what is happening on that target?

FWIW, there's a more involved version of this transform already in IR (in instcombine although that seems misplaced to me) - see "swapMayExposeCSEOpportunities()".

There is also http://g.co/lanai/isa . I'll try to patch and see what is happening with this change, the test seems suspicious given the change in select op without change to sub.f operands.

llvm/test/CodeGen/Lanai/sub-cmp-peephole.ll

28 ↗

(On Diff #207144)

The select compare changes, but the operands to the subtract doesn't, that seems weird.

[spatel]

llvm/test/CodeGen/Lanai/sub-cmp-peephole.ll

28 ↗

(On Diff #207144)

If I'm reading this asm correctly based on the linked doc:
http://g.co/lanai/isa

We have:
%a == %r6 , %b == %r7
so "sub.f %r7, %r6, %r3" means "b-a" goes into %r3
And "sel.gt %r3, %r0, %rv" means "b-a" goes into "rv" if "b-a" was greater than 0.

That matches the IR logic, so is the code miscompiled without this patch? If we have "sel.lt", then that means the returned value is negative or zero rather than the intended positive or zero?

[spatel]

Ping @jpienaar

[jpienaar]

Lanai side looks good to me, thanks!

llvm/test/CodeGen/Lanai/sub-cmp-peephole.ll

28 ↗

(On Diff #207144)

You are correct.

llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp

SDValue TargetLowering::SimplifySetCC(EVT VT, SDValue N0, SDValue N1,
   // Ensure that the constant occurs on the RHS and fold constant comparisons.
   // TODO: Handle non-splat vector constants. All undef causes trouble.
   ISD::CondCode SwappedCC = ISD::getSetCCSwappedOperands(Cond);
   if (isConstOrConstSplat(N0) &&
       (DCI.isBeforeLegalizeOps() ||
        isCondCodeLegal(SwappedCC, N0.getSimpleValueType())))
     return DAG.getSetCC(dl, VT, N1, N0, SwappedCC);
 
+  // If we have a subtract with the same 2 non-constant operands as this setcc
+  // -- but in reverse order -- then try to commute the operands of this setcc
+  // to match. A matching pair of setcc (cmp) and sub may be combined into 1
+  // instruction on some targets.
+  if (!isConstOrConstSplat(N0) && !isConstOrConstSplat(N1) &&
+      (DCI.isBeforeLegalizeOps() ||
+       isCondCodeLegal(SwappedCC, N0.getSimpleValueType())) &&
+      DAG.getNodeIfExists(ISD::SUB, DAG.getVTList(OpVT), { N1, N0 } ) &&
+      !DAG.getNodeIfExists(ISD::SUB, DAG.getVTList(OpVT), { N0, N1 } ))
+    return DAG.getSetCC(dl, VT, N1, N0, SwappedCC);
+
   if (auto *N1C = dyn_cast<ConstantSDNode>(N1.getNode())) {
     const APInt &C1 = N1C->getAPIntValue();
 
     // If the LHS is '(srl (ctlz x), 5)', the RHS is 0/1, and this is an
     // equality comparison, then we're just comparing whether X itself is
     // zero.
     if (N0.getOpcode() == ISD::SRL && (C1.isNullValue() || C1.isOneValue()) &&
         N0.getOperand(0).getOpcode() == ISD::CTLZ &&

llvm/trunk/test/CodeGen/AArch64/arm64-shrink-wrapping.ll

 ; ENABLE-NEXT:    .cfi_offset w24, -64
 ; ENABLE-NEXT:    .cfi_offset w25, -72
 ; ENABLE-NEXT:    .cfi_offset w26, -80
 ; ENABLE-NEXT:    .cfi_offset w27, -88
 ; ENABLE-NEXT:    .cfi_offset w28, -96
 ; ENABLE-NEXT:    lsl w8, w0, w1
 ; ENABLE-NEXT:    lsl w9, w1, w0
 ; ENABLE-NEXT:    lsr w10, w0, w1
-; ENABLE-NEXT:    lsr w11, w1, w0
-; ENABLE-NEXT:    add w12, w1, w0
-; ENABLE-NEXT:    sub w13, w1, w0
-; ENABLE-NEXT:    cmp w0, w1
-; ENABLE-NEXT:    add w17, w8, w9
-; ENABLE-NEXT:    sub w16, w9, w10
-; ENABLE-NEXT:    add w15, w10, w11
-; ENABLE-NEXT:    add w14, w11, w12
-; ENABLE-NEXT:    b.ge LBB14_2
+; ENABLE-NEXT:    lsr w12, w1, w0
+; ENABLE-NEXT:    add w15, w1, w0
+; ENABLE-NEXT:    subs w17, w1, w0
+; ENABLE-NEXT:    sub w11, w9, w10
+; ENABLE-NEXT:    add w16, w8, w9
+; ENABLE-NEXT:    add w13, w10, w12
+; ENABLE-NEXT:    add w14, w12, w15
+; ENABLE-NEXT:    b.le LBB14_2
 ; ENABLE-NEXT:  ; %bb.1: ; %true
 ; ENABLE-NEXT:    str w0, [sp]
 ; ENABLE-NEXT:    ; InlineAsm Start
 ; ENABLE-NEXT:    nop
 ; ENABLE-NEXT:    ; InlineAsm End
 ; ENABLE-NEXT:  LBB14_2: ; %false
 ; ENABLE-NEXT:    str w8, [x2]
 ; ENABLE-NEXT:    str w9, [x3]
 ; ENABLE-NEXT:    str w10, [x4]
-; ENABLE-NEXT:    str w11, [x5]
-; ENABLE-NEXT:    str w12, [x6]
-; ENABLE-NEXT:    str w13, [x7]
+; ENABLE-NEXT:    str w12, [x5]
+; ENABLE-NEXT:    str w15, [x6]
+; ENABLE-NEXT:    str w17, [x7]
 ; ENABLE-NEXT:    stp w0, w1, [x2, #4]
-; ENABLE-NEXT:    stp w17, w16, [x2, #12]
-; ENABLE-NEXT:    stp w15, w14, [x2, #20]
+; ENABLE-NEXT:    stp w16, w11, [x2, #12]
+; ENABLE-NEXT:    stp w13, w14, [x2, #20]
 ; ENABLE-NEXT:    sub sp, x29, #80 ; =80
 ; ENABLE-NEXT:    ldp x29, x30, [sp, #80] ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ldp x20, x19, [sp, #64] ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ldp x22, x21, [sp, #48] ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ldp x24, x23, [sp, #32] ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ldp x26, x25, [sp, #16] ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ldp x28, x27, [sp], #96 ; 16-byte Folded Reload
 ; ENABLE-NEXT:    ret

 ; DISABLE-NEXT:    .cfi_offset w24, -64
 ; DISABLE-NEXT:    .cfi_offset w25, -72
 ; DISABLE-NEXT:    .cfi_offset w26, -80
 ; DISABLE-NEXT:    .cfi_offset w27, -88
 ; DISABLE-NEXT:    .cfi_offset w28, -96
 ; DISABLE-NEXT:    lsl w8, w0, w1
 ; DISABLE-NEXT:    lsl w9, w1, w0
 ; DISABLE-NEXT:    lsr w10, w0, w1
-; DISABLE-NEXT:    lsr w11, w1, w0
-; DISABLE-NEXT:    add w12, w1, w0
-; DISABLE-NEXT:    sub w13, w1, w0
-; DISABLE-NEXT:    cmp w0, w1
-; DISABLE-NEXT:    add w17, w8, w9
-; DISABLE-NEXT:    sub w16, w9, w10
-; DISABLE-NEXT:    add w15, w10, w11
-; DISABLE-NEXT:    add w14, w11, w12
-; DISABLE-NEXT:    b.ge LBB14_2
+; DISABLE-NEXT:    lsr w12, w1, w0
+; DISABLE-NEXT:    add w15, w1, w0
+; DISABLE-NEXT:    subs w17, w1, w0
+; DISABLE-NEXT:    sub w11, w9, w10
+; DISABLE-NEXT:    add w16, w8, w9
+; DISABLE-NEXT:    add w13, w10, w12
+; DISABLE-NEXT:    add w14, w12, w15
+; DISABLE-NEXT:    b.le LBB14_2
 ; DISABLE-NEXT:  ; %bb.1: ; %true
 ; DISABLE-NEXT:    str w0, [sp]
 ; DISABLE-NEXT:    ; InlineAsm Start
 ; DISABLE-NEXT:    nop
 ; DISABLE-NEXT:    ; InlineAsm End
 ; DISABLE-NEXT:  LBB14_2: ; %false
 ; DISABLE-NEXT:    str w8, [x2]
 ; DISABLE-NEXT:    str w9, [x3]
 ; DISABLE-NEXT:    str w10, [x4]
-; DISABLE-NEXT:    str w11, [x5]
-; DISABLE-NEXT:    str w12, [x6]
-; DISABLE-NEXT:    str w13, [x7]
+; DISABLE-NEXT:    str w12, [x5]
+; DISABLE-NEXT:    str w15, [x6]
+; DISABLE-NEXT:    str w17, [x7]
 ; DISABLE-NEXT:    stp w0, w1, [x2, #4]
-; DISABLE-NEXT:    stp w17, w16, [x2, #12]
-; DISABLE-NEXT:    stp w15, w14, [x2, #20]
+; DISABLE-NEXT:    stp w16, w11, [x2, #12]
+; DISABLE-NEXT:    stp w13, w14, [x2, #20]
 ; DISABLE-NEXT:    sub sp, x29, #80 ; =80
 ; DISABLE-NEXT:    ldp x29, x30, [sp, #80] ; 16-byte Folded Reload
 ; DISABLE-NEXT:    ldp x20, x19, [sp, #64] ; 16-byte Folded Reload
 ; DISABLE-NEXT:    ldp x22, x21, [sp, #48] ; 16-byte Folded Reload
 ; DISABLE-NEXT:    ldp x24, x23, [sp, #32] ; 16-byte Folded Reload
 ; DISABLE-NEXT:    ldp x26, x25, [sp, #16] ; 16-byte Folded Reload
 ; DISABLE-NEXT:    ldp x28, x27, [sp], #96 ; 16-byte Folded Reload
 ; DISABLE-NEXT:    ret

llvm/trunk/test/CodeGen/AArch64/cgp-usubo.ll

; CHECK-NEXT:    ret
   ret i1 %ov
 }
 
 ; Verify insertion point for single-BB. Toggle predicate.
 
 define i1 @usubo_ugt_i32(i32 %x, i32 %y, i32* %p) nounwind {
 ; CHECK-LABEL: usubo_ugt_i32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    cmp w1, w0
-; CHECK-NEXT:    cset w8, hi
-; CHECK-NEXT:    sub w9, w0, w1
-; CHECK-NEXT:    mov w0, w8
-; CHECK-NEXT:    str w9, [x2]
+; CHECK-NEXT:    subs w8, w0, w1
+; CHECK-NEXT:    cset w0, lo
+; CHECK-NEXT:    str w8, [x2]
 ; CHECK-NEXT:    ret
   %ov = icmp ugt i32 %y, %x
   %s = sub i32 %x, %y
   store i32 %s, i32* %p
   ret i1 %ov
 }
 
 ; Constant operand should match.

llvm/trunk/test/CodeGen/Lanai/sub-cmp-peephole.ll

 
 define i32 @g(i32 inreg %a, i32 inreg %b) nounwind ssp {
 ; CHECK-LABEL: g:
 ; CHECK:       ! %bb.0:
 ; CHECK-NEXT:    st %fp, [--%sp]
 ; CHECK-NEXT:    add %sp, 0x8, %fp
 ; CHECK-NEXT:    sub %sp, 0x8, %sp
 ; CHECK-NEXT:    sub.f %r7, %r6, %r3
-; CHECK-NEXT:    sel.lt %r3, %r0, %rv
+; CHECK-NEXT:    sel.gt %r3, %r0, %rv
 ; CHECK-NEXT:    ld -4[%fp], %pc ! return
 ; CHECK-NEXT:    add %fp, 0x0, %sp
 ; CHECK-NEXT:    ld -8[%fp], %fp
   %cmp = icmp slt i32 %a, %b
   %sub = sub nsw i32 %b, %a
   %sub. = select i1 %cmp, i32 %sub, i32 0
   ret i32 %sub.
 }

 
 define i32 @i(i32 inreg %a, i32 inreg %b) nounwind readnone ssp {
 ; CHECK-LABEL: i:
 ; CHECK:       ! %bb.0:
 ; CHECK-NEXT:    st %fp, [--%sp]
 ; CHECK-NEXT:    add %sp, 0x8, %fp
 ; CHECK-NEXT:    sub %sp, 0x8, %sp
 ; CHECK-NEXT:    sub.f %r7, %r6, %r3
-; CHECK-NEXT:    sel.ult %r3, %r0, %rv
+; CHECK-NEXT:    sel.ugt %r3, %r0, %rv
 ; CHECK-NEXT:    ld -4[%fp], %pc ! return
 ; CHECK-NEXT:    add %fp, 0x0, %sp
 ; CHECK-NEXT:    ld -8[%fp], %fp
   %cmp = icmp ult i32 %a, %b
   %sub = sub i32 %b, %a
   %sub. = select i1 %cmp, i32 %sub, i32 0
   ret i32 %sub.
 }
 
 ; If SR is live-out, we can't remove cmp if there exists a swapped sub.
 define i32 @j(i32 inreg %a, i32 inreg %b) nounwind {
 ; CHECK-LABEL: j:
 ; CHECK:       ! %bb.0: ! %entry
 ; CHECK-NEXT:    st %fp, [--%sp]
 ; CHECK-NEXT:    add %sp, 0x8, %fp
-; CHECK-NEXT:    sub %sp, 0x8, %sp
-; CHECK-NEXT:    sub.f %r7, %r6, %r0
+; CHECK-NEXT:    sub.f %r6, %r7, %rv
 ; CHECK-NEXT:    bne .LBB4_2
-; CHECK-NEXT:    sub %r6, %r7, %rv
+; CHECK-NEXT:    sub %sp, 0x8, %sp
 ; CHECK-NEXT:  .LBB4_1: ! %if.then
+; CHECK-NEXT:    sub.f %r7, %r6, %r0
 ; CHECK-NEXT:    sel.gt %rv, %r6, %rv
 ; CHECK-NEXT:  .LBB4_2: ! %if.else
 ; CHECK-NEXT:    ld -4[%fp], %pc ! return
 ; CHECK-NEXT:    add %fp, 0x0, %sp
 ; CHECK-NEXT:    ld -8[%fp], %fp
 entry:
   %cmp = icmp eq i32 %b, %a
   %sub = sub nsw i32 %a, %b

llvm/trunk/test/CodeGen/X86/jump_sign.ll

; CHECK-NEXT:    retl
   %cond = select i1 %cmp, i32 %sub, i32 0
   ret i32 %cond
 }
 
 ; rdar://10734411
 define i32 @func_h(i32 %a, i32 %b) nounwind {
 ; CHECK-LABEL: func_h:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    xorl %edx, %edx
-; CHECK-NEXT:    subl %ecx, %eax
-; CHECK-NEXT:    cmovlel %edx, %eax
+; CHECK-NEXT:    xorl %ecx, %ecx
+; CHECK-NEXT:    subl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    cmovlel %ecx, %eax
 ; CHECK-NEXT:    retl
   %cmp = icmp slt i32 %b, %a
   %sub = sub nsw i32 %a, %b
   %cond = select i1 %cmp, i32 %sub, i32 0
   ret i32 %cond
 }
 
 define i32 @func_i(i32 %a, i32 %b) nounwind {
; CHECK-NEXT:    retl
   %sub = sub i32 %a, %b
   %cond = select i1 %cmp, i32 %sub, i32 0
   ret i32 %cond
 }
 
 define i32 @func_k(i32 %a, i32 %b) nounwind {
 ; CHECK-LABEL: func_k:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    xorl %edx, %edx
-; CHECK-NEXT:    subl %ecx, %eax
-; CHECK-NEXT:    cmovbel %edx, %eax
+; CHECK-NEXT:    xorl %ecx, %ecx
+; CHECK-NEXT:    subl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    cmovbel %ecx, %eax
 ; CHECK-NEXT:    retl
   %cmp = icmp ult i32 %b, %a
   %sub = sub i32 %a, %b
   %cond = select i1 %cmp, i32 %sub, i32 0
   ret i32 %cond
 }
 
 ; redundant cmp instruction
 define i32 @func_l(i32 %a, i32 %b) nounwind {
 ; CHECK-LABEL: func_l:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; CHECK-NEXT:    movl %edx, %eax
-; CHECK-NEXT:    subl %ecx, %eax
-; CHECK-NEXT:    cmovlel %edx, %eax
+; CHECK-NEXT:    movl %ecx, %eax
+; CHECK-NEXT:    subl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    cmovlel %ecx, %eax
 ; CHECK-NEXT:    retl
   %cmp = icmp slt i32 %b, %a
   %sub = sub nsw i32 %a, %b
   %cond = select i1 %cmp, i32 %sub, i32 %a
   ret i32 %cond
 }
 
 define i32 @func_m(i32 %a, i32 %b) nounwind {

 }
 
 ; If EFLAGS is live-out, we can't remove cmp if there exists
 ; a swapped sub.
 define i32 @func_l2(i32 %a, i32 %b) nounwind {
 ; CHECK-LABEL: func_l2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edx
-; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    movl %eax, %ecx
-; CHECK-NEXT:    subl %edx, %ecx
-; CHECK-NEXT:    cmpl %eax, %edx
+; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
+; CHECK-NEXT:    movl %ecx, %eax
+; CHECK-NEXT:    subl %edx, %eax
 ; CHECK-NEXT:    jne .LBB8_2
 ; CHECK-NEXT:  # %bb.1: # %if.then
-; CHECK-NEXT:    cmovgl %ecx, %eax
-; CHECK-NEXT:    retl
+; CHECK-NEXT:    cmpl %ecx, %edx
+; CHECK-NEXT:    cmovlel %ecx, %eax
 ; CHECK-NEXT:  .LBB8_2: # %if.else
-; CHECK-NEXT:    movl %ecx, %eax
 ; CHECK-NEXT:    retl
   %cmp = icmp eq i32 %b, %a
   %sub = sub nsw i32 %a, %b
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:
   %cmp2 = icmp sgt i32 %b, %a
   %sel = select i1 %cmp2, i32 %sub, i32 %a
   ret i32 %sel
 
 if.else:
   ret i32 %sub
 }
 
 define i32 @func_l3(i32 %a, i32 %b) nounwind {
 ; CHECK-LABEL: func_l3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    subl %ecx, %eax
+; CHECK-NEXT:    subl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    jge .LBB9_2
 ; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .LBB9_2: # %if.else
 ; CHECK-NEXT:    incl %eax
 ; CHECK-NEXT:    retl
   %cmp = icmp sgt i32 %b, %a
   %sub = sub nsw i32 %a, %b
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:
   ret i32 %sub
 
 if.else:
   %add = add nsw i32 %sub, 1
   ret i32 %add
 }
 
 ; rdar://11830760
 ; When Movr0 is between sub and cmp, we need to move "Movr0" before sub.
 define i32 @func_l4(i32 %a, i32 %b) nounwind {
 ; CHECK-LABEL: func_l4:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; CHECK-NEXT:    xorl %edx, %edx
-; CHECK-NEXT:    subl %ecx, %eax
-; CHECK-NEXT:    cmovll %edx, %eax
+; CHECK-NEXT:    xorl %ecx, %ecx
+; CHECK-NEXT:    subl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    cmovll %ecx, %eax
 ; CHECK-NEXT:    retl
   %cmp = icmp sgt i32 %b, %a
   %sub = sub i32 %a, %b
   %.sub = select i1 %cmp, i32 0, i32 %sub
   ret i32 %.sub
 }
 
 ; rdar://11540023

llvm/trunk/test/CodeGen/X86/psubus.ll

 ; SSSE3-NEXT:    movdqa %xmm3, %xmm0
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: test16:
 ; SSE41:       # %bb.0: # %vector.ph
 ; SSE41-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
 ; SSE41-NEXT:    pmovzxwd {{.*#+}} xmm3 = xmm3[0],zero,xmm3[1],zero,xmm3[2],zero,xmm3[3],zero
 ; SSE41-NEXT:    pmovzxwd {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; SSE41-NEXT:    movdqa %xmm1, %xmm4
-; SSE41-NEXT:    pmaxud %xmm0, %xmm4
-; SSE41-NEXT:    pcmpeqd %xmm1, %xmm4
+; SSE41-NEXT:    movdqa %xmm0, %xmm4
+; SSE41-NEXT:    pminud %xmm1, %xmm4
+; SSE41-NEXT:    pcmpeqd %xmm0, %xmm4
 ; SSE41-NEXT:    pcmpeqd %xmm5, %xmm5
 ; SSE41-NEXT:    pxor %xmm5, %xmm4
-; SSE41-NEXT:    movdqa %xmm2, %xmm6
-; SSE41-NEXT:    pmaxud %xmm3, %xmm6
-; SSE41-NEXT:    pcmpeqd %xmm2, %xmm6
+; SSE41-NEXT:    movdqa %xmm3, %xmm6
+; SSE41-NEXT:    pminud %xmm2, %xmm6
+; SSE41-NEXT:    pcmpeqd %xmm3, %xmm6
 ; SSE41-NEXT:    pxor %xmm5, %xmm6
 ; SSE41-NEXT:    packssdw %xmm6, %xmm4
 ; SSE41-NEXT:    psubd %xmm2, %xmm3
 ; SSE41-NEXT:    psubd %xmm1, %xmm0
 ; SSE41-NEXT:    movdqa {{.*#+}} xmm1 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
 ; SSE41-NEXT:    pshufb %xmm1, %xmm0
 ; SSE41-NEXT:    pshufb %xmm1, %xmm3
 ; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm3[0]
 ; SSE41-NEXT:    pand %xmm4, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: test16:
 ; AVX1:       # %bb.0: # %vector.ph
 ; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX1-NEXT:    vpunpckhwd {{.*#+}} xmm2 = xmm0[4],xmm2[4],xmm0[5],xmm2[5],xmm0[6],xmm2[6],xmm0[7],xmm2[7]
 ; AVX1-NEXT:    vpmovzxwd {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
-; AVX1-NEXT:    vpmaxud %xmm0, %xmm1, %xmm3
-; AVX1-NEXT:    vpcmpeqd %xmm3, %xmm1, %xmm3
-; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
-; AVX1-NEXT:    vpmaxud %xmm2, %xmm4, %xmm5
-; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm4, %xmm5
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm3
+; AVX1-NEXT:    vpminud %xmm3, %xmm2, %xmm4
+; AVX1-NEXT:    vpcmpeqd %xmm4, %xmm2, %xmm4
+; AVX1-NEXT:    vpminud %xmm1, %xmm0, %xmm5
+; AVX1-NEXT:    vpcmpeqd %xmm5, %xmm0, %xmm5
 ; AVX1-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
-; AVX1-NEXT:    vpandn %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    vpsubd %xmm4, %xmm2, %xmm1
-; AVX1-NEXT:    vpandn %xmm1, %xmm5, %xmm1
+; AVX1-NEXT:    vpandn %xmm0, %xmm5, %xmm0
+; AVX1-NEXT:    vpsubd %xmm3, %xmm2, %xmm1
+; AVX1-NEXT:    vpandn %xmm1, %xmm4, %xmm1
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15]
 ; AVX1-NEXT:    vpshufb %xmm2, %xmm1, %xmm1
 ; AVX1-NEXT:    vpshufb %xmm2, %xmm0, %xmm0
 ; AVX1-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
 ; AVX1-NEXT:    vzeroupper
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: test16:
 ; AVX2:       # %bb.0: # %vector.ph
 ; AVX2-NEXT:    vpmovzxwd {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX2-NEXT:    vpmaxud %ymm0, %ymm1, %ymm2
-; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm1, %ymm2
+; AVX2-NEXT:    vpminud %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpcmpeqd %ymm2, %ymm0, %ymm2
 ; AVX2-NEXT:    vpsubd %ymm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vpandn %ymm0, %ymm2, %ymm0
 ; AVX2-NEXT:    vpshufb {{.*#+}} ymm0 = ymm0[0,1,4,5,8,9,12,13,8,9,12,13,12,13,14,15,16,17,20,21,24,25,28,29,24,25,28,29,28,29,30,31]
 ; AVX2-NEXT:    vpermq {{.*#+}} ymm0 = ymm0[0,2,2,3]
 ; AVX2-NEXT:    # kill: def $xmm0 killed $xmm0 killed $ymm0
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: test16:
 ; AVX512:       # %bb.0: # %vector.ph
 ; AVX512-NEXT:    vpmovzxwd {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
-; AVX512-NEXT:    vpcmpltud %ymm0, %ymm1, %k1
+; AVX512-NEXT:    vpcmpnleud %ymm1, %ymm0, %k1
 ; AVX512-NEXT:    vpsubd %ymm1, %ymm0, %ymm0
 ; AVX512-NEXT:    vpmovdw %ymm0, %xmm0 {%k1} {z}
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
 vector.ph:
   %lhs = zext <8 x i16> %x to <8 x i32>
   %cond = icmp ult <8 x i32> %y, %lhs
   %sub = sub <8 x i32> %lhs, %y