[SelectionDAG] Support some SimplifySetCC cases for comparing against vector splats of constants.

[craig.topper]

Summary

This supports things like

(setcc uge X, 0) -> true
(setcc ugt X, 0) -> (setcc ne X, 0)

The AArch64 test changes are to avoid using things that we can simplify now. I'll probably commit these separately.

[spatel]

Please check in the AArch64 (and MIPS?) test changes ahead of this.

I'd prefer to add some dedicated minimal tests, so we know exactly what the improvements are without digging through the more complicated diffs. That would be stuff like this I think:

define <4 x i32> @lt_min(<4 x i32> %x) {
  %cmp = icmp slt <4 x i32> %x, <i32 2147483648, i32 2147483648, i32 2147483648, i32 2147483648>
  %r = sext <4 x i1> %cmp to <4 x i32>
  ret <4 x i32> %r
}

lib/CodeGen/SelectionDAG/TargetLowering.cpp

1891 ↗

(On Diff #132940)

Is this (and similar diffs below) currently wrong for ZeroOrNegativeOneBooleanContent targets?
I'm having a hard time seeing all the pre-conditions in this 900 line monster of a function...

[sdardis]

In D42948#999225, @spatel wrote:

Please check in the AArch64 (and MIPS?) test changes ahead of this.

We're a ZeroOrNegativeOneBooleanContent target for vectors, so this may be useful.

[craig.topper]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

1891 ↗

(On Diff #132940)

I suspect it is. I believe there are other lines in here that I didn't touch that are probably also wrong.

I can split out the getBooleanTrueConstant function and fix this up for scalar types first if you want?

[spatel]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

1891 ↗

(On Diff #132940)

If it's not too hard, that would be good. A quick check of in-tree targets says AMDGPU, NVPTX, and some flavors of MIPS could hit that bug via scalar ZeroOrNegativeOneBooleanContent.

[spatel]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

1891 ↗

(On Diff #132940)

Oops - I read that wrong. Looks like all MIPS have ZeroOrOneBooleanContent for scalars.

[craig.topper]

lib/CodeGen/SelectionDAG/TargetLowering.cpp

1891 ↗

(On Diff #132940)

In most cases we probably optimize the setcc before type legalization so the result type is i1 making the difference not meaningful. So to hit a bug you'd need to only see the simplifcation after the result type has been promoted.

[craig.topper]

Add directed tests. I believe there are some missing optimizations due to not enabling canonicalization of GE/LE to GT/LT for vectors.

[spatel]

In D42948#1007109, @craig.topper wrote:

Add directed tests. I believe there are some missing optimizations due to not enabling canonicalization of GE/LE to GT/LT for vectors.

We might have to be more careful with vector ISAs because they tend to leave out the full set of predicates. Ie, do we have to check for legal-or-custom if this transform can happen late, or can we assume that all predicates must be handled for all data types?

I checked in the baseline tests at rL325138 - can you rebase here, so we just see the improvements?

[RKSimon]

In D42948#1007581, @spatel wrote:

We might have to be more careful with vector ISAs because they tend to leave out the full set of predicates. Ie, do we have to check for legal-or-custom if this transform can happen late, or can we assume that all predicates must be handled for all data types?

Do you think we should override isCondCodeLegal?

[spatel]

In D42948#1011601, @RKSimon wrote:

In D42948#1007581, @spatel wrote:

We might have to be more careful with vector ISAs because they tend to leave out the full set of predicates. Ie, do we have to check for legal-or-custom if this transform can happen late, or can we assume that all predicates must be handled for all data types?

Do you think we should override isCondCodeLegal?

No - that looks exactly like what I was thinking of. I just couldn't tell from the diffs if we're checking that consistently...so this patch should be good as-is (just rebased).

Refactoring to better organize this blob can be a follow-up. :)

[craig.topper]

We're not checking isCondCodeLegal consistently. The only calls to it are in the two blocks that covered with !isVector. I'm not sure we can use isCondCodeLegal correctly with SSE/AVX and AVX512 having different behaviors.

[craig.topper]

Rebase

[RKSimon]

test/CodeGen/X86/vector-compare-simplify.ll

2 ↗

(On Diff #134852)

Do we need to add SSE/SSE42/AVX2/XOP/AVX512 test targets here to check different integer compare abilities?

[spatel]

test/CodeGen/X86/vector-compare-simplify.ll

29–31 ↗

(On Diff #134852)

Should we account for this regression first (by adding x86-specific folds for vector compares with edge case constants)?

78–80 ↗

(On Diff #134852)

Do we know why this didn't load-fold like the sgt_max test?

[spatel]

test/CodeGen/X86/vector-compare-simplify.ll

78–80 ↗

(On Diff #134852)

Disregard...this is less-than, so we can't commute the memop. But this might still be viewed as a regression.

[craig.topper]

Change to only allow vector adjustments before legalize ops since the isCondCodeLegal is probably not correct for targets today. But this will at least give us a chance to make some simplifications.

[craig.topper]

Ping

[spatel]

LGTM.

llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (Cond == ISD::SETLE || Cond == ISD::SETULE) {
         }
       }
     }
 
     if (Cond == ISD::SETLT || Cond == ISD::SETULT) {
       if (C1 == MinVal)
         return DAG.getBoolConstant(false, dl, VT, OpVT); // X < MIN --> false
 
-      if (!VT.isVector()) { // TODO: Support this for vectors.
+      // TODO: Support this for vectors after legalize ops.
+      if (!VT.isVector() || DCI.isBeforeLegalizeOps()) {
         // Canonicalize setlt X, Max --> setne X, Max
         if (C1 == MaxVal)
           return DAG.getSetCC(dl, VT, N0, N1, ISD::SETNE);
 
         // If we have setult X, 1, turn it into seteq X, 0
         if (C1 == MinVal+1)
           return DAG.getSetCC(dl, VT, N0,
                               DAG.getConstant(MinVal, dl, N0.getValueType()),
                               ISD::SETEQ);
       }
     }
 
     if (Cond == ISD::SETGT || Cond == ISD::SETUGT) {
       if (C1 == MaxVal)
         return DAG.getBoolConstant(false, dl, VT, OpVT); // X > MAX --> false
 
-      if (!VT.isVector()) { // TODO: Support this for vectors.
+      // TODO: Support this for vectors after legalize ops.
+      if (!VT.isVector() || DCI.isBeforeLegalizeOps()) {
         // Canonicalize setgt X, Min --> setne X, Min
         if (C1 == MinVal)
           return DAG.getSetCC(dl, VT, N0, N1, ISD::SETNE);
 
         // If we have setugt X, Max-1, turn it into seteq X, Max
         if ((Cond == ISD::SETGT || Cond == ISD::SETUGT) && C1 == MaxVal-1)
           return DAG.getSetCC(dl, VT, N0,
                               DAG.getConstant(MaxVal, dl, N0.getValueType()),
                               ISD::SETEQ);
       }
     }
 
     // If we have "setcc X, C0", check to see if we can shrink the immediate
     // by changing cc.
-    if (!VT.isVector()) { // TODO: Support this for vectors.
+    // TODO: Support this for vectors after legalize ops.
+    if (!VT.isVector() || DCI.isBeforeLegalizeOps()) {
       // SETUGT X, SINTMAX  -> SETLT X, 0
       if (Cond == ISD::SETUGT &&
           C1 == APInt::getSignedMaxValue(OperandBitSize))
         return DAG.getSetCC(dl, VT, N0,
                             DAG.getConstant(0, dl, N1.getValueType()),
                             ISD::SETLT);
 
       // SETULT X, SINTMIN  -> SETGT X, -1

llvm/trunk/test/CodeGen/X86/avx512-insert-extract.ll

 ; KNL-NEXT:    .cfi_def_cfa_offset 16
 ; KNL-NEXT:    .cfi_offset %rbp, -16
 ; KNL-NEXT:    movq %rsp, %rbp
 ; KNL-NEXT:    .cfi_def_cfa_register %rbp
 ; KNL-NEXT:    andq $-32, %rsp
 ; KNL-NEXT:    subq $64, %rsp
 ; KNL-NEXT:    ## kill: def $esi killed $esi def $rsi
 ; KNL-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; KNL-NEXT:    vpminub %ymm1, %ymm0, %ymm1
 ; KNL-NEXT:    vpcmpeqb %ymm1, %ymm0, %ymm0
 ; KNL-NEXT:    vpternlogq $15, %zmm0, %zmm0, %zmm0
 ; KNL-NEXT:    andl $31, %esi
 ; KNL-NEXT:    testb %dil, %dil
 ; KNL-NEXT:    vmovdqa %ymm0, (%rsp)
 ; KNL-NEXT:    setne (%rsp,%rsi)
 ; KNL-NEXT:    vpmovsxbd (%rsp), %zmm0
 ; KNL-NEXT:    vpslld $31, %zmm0, %zmm0

 ; SKX-NEXT:    pushq %rbp
 ; SKX-NEXT:    .cfi_def_cfa_offset 16
 ; SKX-NEXT:    .cfi_offset %rbp, -16
 ; SKX-NEXT:    movq %rsp, %rbp
 ; SKX-NEXT:    .cfi_def_cfa_register %rbp
 ; SKX-NEXT:    andq $-32, %rsp
 ; SKX-NEXT:    subq $64, %rsp
 ; SKX-NEXT:    ## kill: def $esi killed $esi def $rsi
-; SKX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; SKX-NEXT:    vpcmpnleub %ymm1, %ymm0, %k0
+; SKX-NEXT:    vptestmb %ymm0, %ymm0, %k0
 ; SKX-NEXT:    andl $31, %esi
 ; SKX-NEXT:    testb %dil, %dil
 ; SKX-NEXT:    vpmovm2b %k0, %ymm0
 ; SKX-NEXT:    vmovdqa %ymm0, (%rsp)
 ; SKX-NEXT:    setne (%rsp,%rsi)
 ; SKX-NEXT:    vpsllw $7, (%rsp), %ymm0
 ; SKX-NEXT:    vpmovb2m %ymm0, %k0
 ; SKX-NEXT:    kmovd %k0, %eax

 ; KNL-NEXT:    .cfi_def_cfa_offset 16
 ; KNL-NEXT:    .cfi_offset %rbp, -16
 ; KNL-NEXT:    movq %rsp, %rbp
 ; KNL-NEXT:    .cfi_def_cfa_register %rbp
 ; KNL-NEXT:    andq $-64, %rsp
 ; KNL-NEXT:    subq $128, %rsp
 ; KNL-NEXT:    ## kill: def $esi killed $esi def $rsi
 ; KNL-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; KNL-NEXT:    vpminub %ymm2, %ymm0, %ymm3
-; KNL-NEXT:    vpcmpeqb %ymm3, %ymm0, %ymm0
+; KNL-NEXT:    vpcmpeqb %ymm2, %ymm0, %ymm0
 ; KNL-NEXT:    vpternlogq $15, %zmm0, %zmm0, %zmm0
-; KNL-NEXT:    vpminub %ymm2, %ymm1, %ymm2
 ; KNL-NEXT:    vpcmpeqb %ymm2, %ymm1, %ymm1
 ; KNL-NEXT:    vpternlogq $15, %zmm1, %zmm1, %zmm1
 ; KNL-NEXT:    andl $63, %esi
 ; KNL-NEXT:    testb %dil, %dil
 ; KNL-NEXT:    vmovdqa %ymm1, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm0, (%rsp)
 ; KNL-NEXT:    setne (%rsp,%rsi)
 ; KNL-NEXT:    vmovdqa (%rsp), %ymm0

 ; SKX-NEXT:    pushq %rbp
 ; SKX-NEXT:    .cfi_def_cfa_offset 16
 ; SKX-NEXT:    .cfi_offset %rbp, -16
 ; SKX-NEXT:    movq %rsp, %rbp
 ; SKX-NEXT:    .cfi_def_cfa_register %rbp
 ; SKX-NEXT:    andq $-64, %rsp
 ; SKX-NEXT:    subq $128, %rsp
 ; SKX-NEXT:    ## kill: def $esi killed $esi def $rsi
-; SKX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; SKX-NEXT:    vpcmpnleub %zmm1, %zmm0, %k0
+; SKX-NEXT:    vptestmb %zmm0, %zmm0, %k0
 ; SKX-NEXT:    andl $63, %esi
 ; SKX-NEXT:    testb %dil, %dil
 ; SKX-NEXT:    vpmovm2b %k0, %zmm0
 ; SKX-NEXT:    vmovdqa64 %zmm0, (%rsp)
 ; SKX-NEXT:    setne (%rsp,%rsi)
 ; SKX-NEXT:    vpsllw $7, (%rsp), %zmm0
 ; SKX-NEXT:    vpmovb2m %zmm0, %k0
 ; SKX-NEXT:    kmovq %k0, %rax

 ; KNL-NEXT:    vpinsrb $10, 176(%rbp), %xmm3, %xmm3
 ; KNL-NEXT:    vpinsrb $11, 184(%rbp), %xmm3, %xmm3
 ; KNL-NEXT:    vpinsrb $12, 192(%rbp), %xmm3, %xmm3
 ; KNL-NEXT:    vpinsrb $13, 200(%rbp), %xmm3, %xmm3
 ; KNL-NEXT:    vpinsrb $14, 208(%rbp), %xmm3, %xmm3
 ; KNL-NEXT:    vpinsrb $15, 216(%rbp), %xmm3, %xmm3
 ; KNL-NEXT:    vinserti128 $1, %xmm3, %ymm2, %ymm2
 ; KNL-NEXT:    vpxor %xmm3, %xmm3, %xmm3
-; KNL-NEXT:    vpminub %ymm3, %ymm2, %ymm4
-; KNL-NEXT:    vpcmpeqb %ymm4, %ymm2, %ymm2
+; KNL-NEXT:    vpcmpeqb %ymm3, %ymm2, %ymm2
 ; KNL-NEXT:    vpternlogq $15, %zmm2, %zmm2, %zmm2
-; KNL-NEXT:    vpminub %ymm3, %ymm1, %ymm4
-; KNL-NEXT:    vpcmpeqb %ymm4, %ymm1, %ymm1
+; KNL-NEXT:    vpcmpeqb %ymm3, %ymm1, %ymm1
 ; KNL-NEXT:    vpternlogq $15, %zmm1, %zmm1, %zmm1
-; KNL-NEXT:    vpminub %ymm3, %ymm0, %ymm4
-; KNL-NEXT:    vpcmpeqb %ymm4, %ymm0, %ymm0
+; KNL-NEXT:    vpcmpeqb %ymm3, %ymm0, %ymm0
 ; KNL-NEXT:    vpternlogq $15, %zmm0, %zmm0, %zmm0
 ; KNL-NEXT:    movl 744(%rbp), %eax
 ; KNL-NEXT:    andl $127, %eax
 ; KNL-NEXT:    cmpb $0, 736(%rbp)
 ; KNL-NEXT:    vmovdqa %ymm3, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm0, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm1, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm2, (%rsp)

 ; SKX-NEXT:    vpinsrb $9, 680(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vpinsrb $10, 688(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vpinsrb $11, 696(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vpinsrb $12, 704(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vpinsrb $13, 712(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vpinsrb $14, 720(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vpinsrb $15, 728(%rbp), %xmm2, %xmm2
 ; SKX-NEXT:    vinserti128 $1, %xmm2, %ymm1, %ymm1
-; SKX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; SKX-NEXT:    vpcmpnleub %zmm2, %zmm0, %k0
-; SKX-NEXT:    vpcmpnleub %zmm2, %zmm1, %k1
+; SKX-NEXT:    vptestmb %zmm0, %zmm0, %k0
+; SKX-NEXT:    vptestmb %zmm1, %zmm1, %k1
 ; SKX-NEXT:    movl 744(%rbp), %eax
 ; SKX-NEXT:    andl $127, %eax
 ; SKX-NEXT:    cmpb $0, 736(%rbp)
 ; SKX-NEXT:    vpmovm2b %k1, %zmm0
 ; SKX-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
 ; SKX-NEXT:    vpmovm2b %k0, %zmm0
 ; SKX-NEXT:    vmovdqa64 %zmm0, (%rsp)
 ; SKX-NEXT:    setne (%rsp,%rax)

 ; KNL-NEXT:    .cfi_def_cfa_offset 16
 ; KNL-NEXT:    .cfi_offset %rbp, -16
 ; KNL-NEXT:    movq %rsp, %rbp
 ; KNL-NEXT:    .cfi_def_cfa_register %rbp
 ; KNL-NEXT:    andq $-128, %rsp
 ; KNL-NEXT:    subq $256, %rsp ## imm = 0x100
 ; KNL-NEXT:    ## kill: def $esi killed $esi def $rsi
 ; KNL-NEXT:    vpxor %xmm4, %xmm4, %xmm4
-; KNL-NEXT:    vpminub %ymm4, %ymm0, %ymm5
-; KNL-NEXT:    vpcmpeqb %ymm5, %ymm0, %ymm0
+; KNL-NEXT:    vpcmpeqb %ymm4, %ymm0, %ymm0
 ; KNL-NEXT:    vpternlogq $15, %zmm0, %zmm0, %zmm0
-; KNL-NEXT:    vpminub %ymm4, %ymm1, %ymm5
-; KNL-NEXT:    vpcmpeqb %ymm5, %ymm1, %ymm1
+; KNL-NEXT:    vpcmpeqb %ymm4, %ymm1, %ymm1
 ; KNL-NEXT:    vpternlogq $15, %zmm1, %zmm1, %zmm1
-; KNL-NEXT:    vpminub %ymm4, %ymm2, %ymm5
-; KNL-NEXT:    vpcmpeqb %ymm5, %ymm2, %ymm2
+; KNL-NEXT:    vpcmpeqb %ymm4, %ymm2, %ymm2
 ; KNL-NEXT:    vpternlogq $15, %zmm2, %zmm2, %zmm2
-; KNL-NEXT:    vpminub %ymm4, %ymm3, %ymm4
 ; KNL-NEXT:    vpcmpeqb %ymm4, %ymm3, %ymm3
 ; KNL-NEXT:    vpternlogq $15, %zmm3, %zmm3, %zmm3
 ; KNL-NEXT:    andl $127, %esi
 ; KNL-NEXT:    testb %dil, %dil
 ; KNL-NEXT:    vmovdqa %ymm3, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm2, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm1, {{[0-9]+}}(%rsp)
 ; KNL-NEXT:    vmovdqa %ymm0, (%rsp)

 ; SKX-NEXT:    pushq %rbp
 ; SKX-NEXT:    .cfi_def_cfa_offset 16
 ; SKX-NEXT:    .cfi_offset %rbp, -16
 ; SKX-NEXT:    movq %rsp, %rbp
 ; SKX-NEXT:    .cfi_def_cfa_register %rbp
 ; SKX-NEXT:    andq $-128, %rsp
 ; SKX-NEXT:    subq $256, %rsp ## imm = 0x100
 ; SKX-NEXT:    ## kill: def $esi killed $esi def $rsi
-; SKX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; SKX-NEXT:    vpcmpnleub %zmm2, %zmm0, %k0
-; SKX-NEXT:    vpcmpnleub %zmm2, %zmm1, %k1
+; SKX-NEXT:    vptestmb %zmm0, %zmm0, %k0
+; SKX-NEXT:    vptestmb %zmm1, %zmm1, %k1
 ; SKX-NEXT:    andl $127, %esi
 ; SKX-NEXT:    testb %dil, %dil
 ; SKX-NEXT:    vpmovm2b %k1, %zmm0
 ; SKX-NEXT:    vmovdqa64 %zmm0, {{[0-9]+}}(%rsp)
 ; SKX-NEXT:    vpmovm2b %k0, %zmm0
 ; SKX-NEXT:    vmovdqa64 %zmm0, (%rsp)
 ; SKX-NEXT:    setne (%rsp,%rsi)
 ; SKX-NEXT:    vpsllw $7, {{[0-9]+}}(%rsp), %zmm0

llvm/trunk/test/CodeGen/X86/vector-compare-simplify.ll

; CHECK-NEXT:    retq
   %cmp = icmp uge <4 x i32> %x, zeroinitializer
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ugt_min(<4 x i32> %x) {
 ; CHECK-LABEL: ugt_min:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483648,2147483648,2147483648,2147483648]
+; CHECK-NEXT:    pxor %xmm1, %xmm1
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm0
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
-; CHECK-NEXT:    pcmpgtd %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ugt <4 x i32> %x, zeroinitializer
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ule_min(<4 x i32> %x) {
 ; CHECK-LABEL: ule_min:
; CHECK-NEXT:    retq
   %cmp = icmp ule <4 x i32> %x, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ult_max(<4 x i32> %x) {
 ; CHECK-LABEL: ult_max:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483647,2147483647,2147483647,2147483647]
-; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
-; CHECK-NEXT:    movdqa %xmm1, %xmm0
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm0
+; CHECK-NEXT:    pxor %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ult <4 x i32> %x, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @uge_max(<4 x i32> %x) {
 ; CHECK-LABEL: uge_max:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pcmpeqd %xmm2, %xmm2
 ; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483647,2147483647,2147483647,2147483647]
 ; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
 ; CHECK-NEXT:    pxor %xmm2, %xmm1
 ; CHECK-NEXT:    movdqa %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp uge <4 x i32> %x, <i32 -1, i32 -1, i32 -1, i32 -1>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @slt_min_plus1(<4 x i32> %x) {
 ; CHECK-LABEL: slt_min_plus1:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483649,2147483649,2147483649,2147483649]
-; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
-; CHECK-NEXT:    movdqa %xmm1, %xmm0
+; CHECK-NEXT:    pcmpeqd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp slt <4 x i32> %x, <i32 -2147483647, i32 -2147483647, i32 -2147483647, i32 -2147483647>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @sge_min_plus1(<4 x i32> %x) {
 ; CHECK-LABEL: sge_min_plus1:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483649,2147483649,2147483649,2147483649]
 ; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
 ; CHECK-NEXT:    pcmpeqd %xmm0, %xmm0
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp sge <4 x i32> %x, <i32 -2147483647, i32 -2147483647, i32 -2147483647, i32 -2147483647>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @sgt_max_minus1(<4 x i32> %x) {
 ; CHECK-LABEL: sgt_max_minus1:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pcmpgtd {{.*}}(%rip), %xmm0
+; CHECK-NEXT:    pcmpeqd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp sgt <4 x i32> %x, <i32 2147483646, i32 2147483646, i32 2147483646, i32 2147483646>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @sle_max_minus1(<4 x i32> %x) {
 ; CHECK-LABEL: sle_max_minus1:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pcmpgtd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp sle <4 x i32> %x, <i32 2147483646, i32 2147483646, i32 2147483646, i32 2147483646>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ult_one(<4 x i32> %x) {
 ; CHECK-LABEL: ult_one:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483649,2147483649,2147483649,2147483649]
-; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
-; CHECK-NEXT:    movdqa %xmm1, %xmm0
+; CHECK-NEXT:    pxor %xmm1, %xmm1
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ult <4 x i32> %x, <i32 1, i32 1, i32 1, i32 1>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @uge_one(<4 x i32> %x) {
 ; CHECK-LABEL: uge_one:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    movdqa {{.*#+}} xmm1 = [2147483649,2147483649,2147483649,2147483649]
 ; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
 ; CHECK-NEXT:    pcmpeqd %xmm0, %xmm0
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp uge <4 x i32> %x, <i32 1, i32 1, i32 1, i32 1>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ugt_max_minus1(<4 x i32> %x) {
 ; CHECK-LABEL: ugt_max_minus1:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    pcmpgtd {{.*}}(%rip), %xmm0
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ugt <4 x i32> %x, <i32 -2, i32 -2, i32 -2, i32 -2>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ule_max_minus1(<4 x i32> %x) {
 ; CHECK-LABEL: ule_max_minus1:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    pcmpgtd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ule <4 x i32> %x, <i32 -2, i32 -2, i32 -2, i32 -2>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ugt_smax(<4 x i32> %x) {
 ; CHECK-LABEL: ugt_smax:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
-; CHECK-NEXT:    pcmpgtd %xmm1, %xmm0
+; CHECK-NEXT:    pxor %xmm1, %xmm1
+; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
+; CHECK-NEXT:    movdqa %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ugt <4 x i32> %x, <i32 2147483647, i32 2147483647, i32 2147483647, i32 2147483647>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ule_smax(<4 x i32> %x) {
 ; CHECK-LABEL: ule_smax:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
 ; CHECK-NEXT:    pcmpgtd %xmm1, %xmm0
 ; CHECK-NEXT:    pxor %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ule <4 x i32> %x, <i32 2147483647, i32 2147483647, i32 2147483647, i32 2147483647>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @ult_smin(<4 x i32> %x) {
 ; CHECK-LABEL: ult_smin:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    pxor {{.*}}(%rip), %xmm0
-; CHECK-NEXT:    pxor %xmm1, %xmm1
-; CHECK-NEXT:    pcmpgtd %xmm0, %xmm1
-; CHECK-NEXT:    movdqa %xmm1, %xmm0
+; CHECK-NEXT:    pcmpeqd %xmm1, %xmm1
+; CHECK-NEXT:    pcmpgtd %xmm1, %xmm0
 ; CHECK-NEXT:    retq
   %cmp = icmp ult <4 x i32> %x, <i32 -2147483648, i32 -2147483648, i32 -2147483648, i32 -2147483648>
   %r = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %r
 }
 
 define <4 x i32> @uge_smin(<4 x i32> %x) {
 ; CHECK-LABEL: uge_smin: