[DAGCombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

Summary

This happens when CMP1 and CMP3 have the same predicate (or CMP2 and CMP3 have
the same predicate).

This helps optimizations such as the fololowing one:
CMP(A,C)||CMP(B,C) => CMP(MIN/MAX(A,B), C)
CMP(A,C)&&CMP(B,C) => CMP(MIN/MAX(A,B), C)

[kmitropoulou]

Updating D156215: [DAGcombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

Updating D156215: [DAGcombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

Updating D156215: [DAGcombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

llvm/test/CodeGen/Hexagon/isel/logical.ll

1253

One of the comparisons is ne when we process the nodes of the DAG.

[arsenm]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

1333–1336

Should add a dag comment explaining the transform

1342

no else after return

[kmitropoulou]

Updating D156215: [DAGcombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

Updating D156215: [DAGCombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

Updating D156215: [DAGCombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[foad]

Seems reasonable, but I wonder how often this helps in real code.

Your patch does not handle swapped comparison conditions, e.g. reassociating an expression so that a<b can be combined with b>c. Doing that properly would make it much more complicated and I don't think it is worth it.

Your patch does not help if there is a tree of ANDs or ORs with more than three setcc nodes.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

1333

Typo "N001"

[kmitropoulou]

In D156215#4538242, @foad wrote:

Seems reasonable, but I wonder how often this helps in real code.

I ran the Vulcan game tests. The reassociateOpsCommutative() is called 115168 times and the proposed optimization is triggered 302 times. So, I will abandon the patch.

[arsenm]

In D156215#4552684, @kmitropoulou wrote:

In D156215#4538242, @foad wrote:

Seems reasonable, but I wonder how often this helps in real code.

I ran the Vulcan game tests. The reassociateOpsCommutative() is called 115168 times and the proposed optimization is triggered 302 times. So, I will abandon the patch.

That seems worthwhile? Why abandon it?

[kmitropoulou]

In D156215#4555639, @arsenm wrote:

In D156215#4552684, @kmitropoulou wrote:

In D156215#4538242, @foad wrote:

Seems reasonable, but I wonder how often this helps in real code.

I ran the Vulcan game tests. The reassociateOpsCommutative() is called 115168 times and the proposed optimization is triggered 302 times. So, I will abandon the patch.

That seems worthwhile? Why abandon it?

I am sorry I restored it.

[kmitropoulou]

Updating D156215: [DAGCombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

[kmitropoulou]

ping

[kmitropoulou]

Updating D156215: [DAGCombiner] Reassociate the operands from (OR (OR(CMP1, CMP2)), CMP3) to (OR (OR(CMP1, CMP3)), CMP2)

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (N1 != N00) {
       // Reassociate if (op N01, N1) already exist
       if (SDNode *NE = DAG.getNodeIfExists(Opc, DAG.getVTList(VT), {N01, N1})) {
         // if Op (Op N01, N1), N00 already exist
         // we need to stop reassciate to avoid dead loop
         if (!DAG.doesNodeExist(Opc, DAG.getVTList(VT), {SDValue(NE, 0), N00}))
           return DAG.getNode(Opc, DL, VT, SDValue(NE, 0), N00);
       }
     }
+
+    // Reassociate the operands from (OR/AND (OR/AND(N00, N001)), N1) to (OR/AND
+    // (OR/AND(N00, N1)), N01) when N00 and N1 are comparisons with the same
+    // predicate or to (OR/AND (OR/AND(N1, N01)), N00) when N01 and N1 are
+    // comparisons with the same predicate. This enables optimizations as the
+    // following one:
+    // CMP(A,C)||CMP(B,C) => CMP(MIN/MAX(A,B), C)
+    // CMP(A,C)&&CMP(B,C) => CMP(MIN/MAX(A,B), C)
+    if (Opc == ISD::AND || Opc == ISD::OR) {
+      if (N1->getOpcode() == ISD::SETCC && N00->getOpcode() == ISD::SETCC &&
+          N01->getOpcode() == ISD::SETCC) {
+        ISD::CondCode CC1 = cast<CondCodeSDNode>(N1.getOperand(2))->get();
+        ISD::CondCode CC00 = cast<CondCodeSDNode>(N00.getOperand(2))->get();
+        ISD::CondCode CC01 = cast<CondCodeSDNode>(N01.getOperand(2))->get();
+        if (CC1 == CC00 && CC1 != CC01) {
+          SDValue OpNode = DAG.getNode(Opc, SDLoc(N0), VT, N00, N1, Flags);
+          return DAG.getNode(Opc, DL, VT, OpNode, N01, Flags);
+        }
+        if (CC1 == CC01 && CC1 != CC00) {
+          SDValue OpNode = DAG.getNode(Opc, SDLoc(N0), VT, N01, N1, Flags);
+          return DAG.getNode(Opc, DL, VT, OpNode, N00, Flags);
+        }
+      }
+    }
   }
 
   return SDValue();
 }
 
 // Try to reassociate commutative binops.
 SDValue DAGCombiner::reassociateOps(unsigned Opc, const SDLoc &DL, SDValue N0,
                                     SDValue N1, SDNodeFlags Flags) {

llvm/test/CodeGen/AMDGPU/combine_andor_with_cmps.ll

; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %or4 = or i1 %or2, %or3
   ret i1 %or4
 }
 
 define i1 @test103(i32 %arg1, i32 %arg2, i32 %arg3, i32 %arg4, i32 %arg5, i32 %arg6, i32 %C) {
 ; CHECK-LABEL: test103:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    v_max_u32_e32 v0, v0, v1
-; CHECK-NEXT:    v_max_u32_e32 v1, v2, v3
-; CHECK-NEXT:    v_max_u32_e32 v2, v4, v5
-; CHECK-NEXT:    v_cmp_lt_u32_e32 vcc_lo, v0, v6
-; CHECK-NEXT:    v_cmp_gt_u32_e64 s0, v1, v6
-; CHECK-NEXT:    v_cmp_lt_u32_e64 s1, v2, v6
-; CHECK-NEXT:    s_or_b32 s0, vcc_lo, s0
-; CHECK-NEXT:    s_or_b32 s0, s0, s1
+; CHECK-NEXT:    v_max_u32_e32 v4, v4, v5
+; CHECK-NEXT:    v_max_u32_e32 v2, v2, v3
+; CHECK-NEXT:    v_maxmin_u32 v0, v0, v1, v4
+; CHECK-NEXT:    v_cmp_gt_u32_e32 vcc_lo, v2, v6
+; CHECK-NEXT:    v_cmp_lt_u32_e64 s0, v0, v6
+; CHECK-NEXT:    s_or_b32 s0, s0, vcc_lo
 ; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %cmp1 = icmp ult i32 %arg1, %C
   %cmp2 = icmp ult i32 %arg2, %C
   %cmp3 = icmp ugt i32 %arg3, %C
   %cmp4 = icmp ugt i32 %arg4, %C
   %cmp5 = icmp ult i32 %arg5, %C
   %cmp6 = icmp ult i32 %arg6, %C
   %and1 = and i1 %cmp1, %cmp2
   %or1 = or i1 %cmp3, %cmp4
   %and2 = and i1 %cmp5, %cmp6
   %or2 = or i1 %and1, %or1
   %or3 = or i1 %or2, %and2
   ret i1 %or3
 }
 
 
 define i1 @test104(i32 %arg1, i32 %arg2, i32 %arg3, i32 %arg4, i32 %arg5, i32 %arg6, i32 %arg7, i32 %arg8, i32 %arg9, i32 %arg10, i32 %C) {
 ; CHECK-LABEL: test104:
 ; CHECK:       ; %bb.0:
 ; CHECK-NEXT:    s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)
-; CHECK-NEXT:    v_min_u32_e32 v0, v0, v1
-; CHECK-NEXT:    v_max_u32_e32 v1, v2, v3
-; CHECK-NEXT:    v_min_u32_e32 v2, v4, v5
-; CHECK-NEXT:    v_max_u32_e32 v3, v6, v7
-; CHECK-NEXT:    v_min_u32_e32 v4, v8, v9
-; CHECK-NEXT:    v_cmp_lt_u32_e32 vcc_lo, v0, v10
-; CHECK-NEXT:    v_cmp_gt_u32_e64 s0, v1, v10
-; CHECK-NEXT:    v_cmp_lt_u32_e64 s1, v2, v10
-; CHECK-NEXT:    v_cmp_gt_u32_e64 s2, v3, v10
-; CHECK-NEXT:    v_cmp_lt_u32_e64 s3, v4, v10
-; CHECK-NEXT:    s_or_b32 s0, vcc_lo, s0
-; CHECK-NEXT:    s_or_b32 s1, s1, s2
-; CHECK-NEXT:    s_or_b32 s0, s3, s0
-; CHECK-NEXT:    s_or_b32 s0, s1, s0
+; CHECK-NEXT:    v_min_u32_e32 v8, v8, v9
+; CHECK-NEXT:    v_max_u32_e32 v2, v2, v3
+; CHECK-NEXT:    v_min_u32_e32 v3, v4, v5
+; CHECK-NEXT:    v_max_u32_e32 v4, v6, v7
+; CHECK-NEXT:    v_min3_u32 v0, v0, v1, v8
+; CHECK-NEXT:    v_cmp_gt_u32_e32 vcc_lo, v2, v10
+; CHECK-NEXT:    v_cmp_lt_u32_e64 s0, v3, v10
+; CHECK-NEXT:    v_cmp_gt_u32_e64 s1, v4, v10
+; CHECK-NEXT:    v_cmp_lt_u32_e64 s2, v0, v10
+; CHECK-NEXT:    s_or_b32 s0, s0, s1
+; CHECK-NEXT:    s_or_b32 s1, s2, vcc_lo
+; CHECK-NEXT:    s_or_b32 s0, s0, s1
 ; CHECK-NEXT:    v_cndmask_b32_e64 v0, 0, 1, s0
 ; CHECK-NEXT:    s_setpc_b64 s[30:31]
   %cmp1 = icmp ult i32 %arg1, %C
   %cmp2 = icmp ult i32 %arg2, %C
   %cmp3 = icmp ugt i32 %arg3, %C
   %cmp4 = icmp ugt i32 %arg4, %C
   %cmp5 = icmp ult i32 %arg5, %C
   %cmp6 = icmp ult i32 %arg6, %C

llvm/test/CodeGen/AMDGPU/wave32.ll

 
 define amdgpu_kernel void @test_preserve_condition_undef_flag(float %arg, i32 %arg1, float %arg2) #0 {
 ; GFX1032-LABEL: test_preserve_condition_undef_flag:
 ; GFX1032:       ; %bb.0: ; %bb0
 ; GFX1032-NEXT:    s_clause 0x1
 ; GFX1032-NEXT:    s_load_dword s2, s[0:1], 0x2c
 ; GFX1032-NEXT:    s_load_dword s3, s[0:1], 0x24
 ; GFX1032-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1032-NEXT:    v_cmp_ngt_f32_e64 s0, s2, 0
-; GFX1032-NEXT:    v_cmp_nlt_f32_e64 s1, s2, 1.0
-; GFX1032-NEXT:    v_cmp_nlt_f32_e64 s2, s3, 1.0
+; GFX1032-NEXT:    v_cmp_nlt_f32_e64 s0, s2, 1.0
+; GFX1032-NEXT:    v_cmp_nlt_f32_e64 s1, s3, 1.0
+; GFX1032-NEXT:    v_cmp_ngt_f32_e64 s2, s2, 0
 ; GFX1032-NEXT:    s_or_b32 s0, s0, s1
 ; GFX1032-NEXT:    s_or_b32 s0, s0, s2
 ; GFX1032-NEXT:    s_and_b32 vcc_lo, exec_lo, s0
 ; GFX1032-NEXT:    s_cbranch_vccnz .LBB26_2
 ; GFX1032-NEXT:  ; %bb.1: ; %bb1
 ; GFX1032-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX1032-NEXT:    global_store_dword v[0:1], v0, off
 ; GFX1032-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GFX1032-NEXT:  .LBB26_2: ; %bb2
 ; GFX1032-NEXT:    s_endpgm
 ;
 ; GFX1064-LABEL: test_preserve_condition_undef_flag:
 ; GFX1064:       ; %bb.0: ; %bb0
 ; GFX1064-NEXT:    s_clause 0x1
-; GFX1064-NEXT:    s_load_dword s2, s[0:1], 0x2c
-; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x24
+; GFX1064-NEXT:    s_load_dword s4, s[0:1], 0x2c
+; GFX1064-NEXT:    s_load_dword s2, s[0:1], 0x24
 ; GFX1064-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX1064-NEXT:    v_cmp_ngt_f32_e64 s[0:1], s2, 0
+; GFX1064-NEXT:    v_cmp_nlt_f32_e64 s[0:1], s4, 1.0
 ; GFX1064-NEXT:    v_cmp_nlt_f32_e64 s[2:3], s2, 1.0
-; GFX1064-NEXT:    v_cmp_nlt_f32_e64 s[4:5], s4, 1.0
+; GFX1064-NEXT:    v_cmp_ngt_f32_e64 s[4:5], s4, 0
 ; GFX1064-NEXT:    s_or_b64 s[0:1], s[0:1], s[2:3]
 ; GFX1064-NEXT:    s_or_b64 s[0:1], s[0:1], s[4:5]
 ; GFX1064-NEXT:    s_and_b64 vcc, exec, s[0:1]
 ; GFX1064-NEXT:    s_cbranch_vccnz .LBB26_2
 ; GFX1064-NEXT:  ; %bb.1: ; %bb1
 ; GFX1064-NEXT:    v_mov_b32_e32 v0, 0
 ; GFX1064-NEXT:    global_store_dword v[0:1], v0, off
 ; GFX1064-NEXT:    s_waitcnt_vscnt null, 0x0

llvm/test/CodeGen/Hexagon/isel/logical.ll

 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r7 = ##65537
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     p0 = vcmph.eq(r1:0,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p1 = vcmph.eq(r3:2,r7:6)
+; CHECK-NEXT:     p1 = vcmph.eq(r5:4,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p2 = vcmph.eq(r5:4,r7:6)
+; CHECK-NEXT:     p2 = vcmph.eq(r3:2,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p0 = and(p2,and(p0,!p1))
+; CHECK-NEXT:     p0 = and(p0,p1)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     p0 = and(p0,!p2)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r1:0 = mask(p0)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     jumpr r31
 ; CHECK-NEXT:    }
 b0:

 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r7 = ##65537
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     p0 = vcmph.eq(r1:0,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p1 = vcmph.eq(r3:2,r7:6)
+; CHECK-NEXT:     p1 = vcmph.eq(r5:4,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p2 = vcmph.eq(r5:4,r7:6)
+; CHECK-NEXT:     p2 = vcmph.eq(r3:2,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p0 = or(p2,or(p0,!p1))
+; CHECK-NEXT:     p0 = or(p0,p1)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     p0 = or(p0,!p2)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r1:0 = mask(p0)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     jumpr r31
 ; CHECK-NEXT:    }
 b0:

 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r7 = ##16843009
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     p0 = vcmpb.eq(r1:0,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p1 = vcmpb.eq(r3:2,r7:6)
+; CHECK-NEXT:     p1 = vcmpb.eq(r5:4,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p2 = vcmpb.eq(r5:4,r7:6)
+; CHECK-NEXT:     p2 = vcmpb.eq(r3:2,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p0 = and(p2,and(p0,!p1))
+; CHECK-NEXT:     p0 = and(p0,p1)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     p0 = and(p0,!p2)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r1:0 = mask(p0)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     jumpr r31
 ; CHECK-NEXT:    }
 b0:

 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r7 = ##16843009
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     p0 = vcmpb.eq(r1:0,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p1 = vcmpb.eq(r3:2,r7:6)
+; CHECK-NEXT:     p1 = vcmpb.eq(r5:4,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p2 = vcmpb.eq(r5:4,r7:6)
+; CHECK-NEXT:     p2 = vcmpb.eq(r3:2,r7:6)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
-; CHECK-NEXT:     p0 = or(p2,or(p0,!p1))
+; CHECK-NEXT:     p0 = or(p0,p1)
+; CHECK-NEXT:    }
+; CHECK-NEXT:    {
+; CHECK-NEXT:     p0 = or(p0,!p2)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     r1:0 = mask(p0)
 ; CHECK-NEXT:    }
 ; CHECK-NEXT:    {
 ; CHECK-NEXT:     jumpr r31
 ; CHECK-NEXT:    }
 b0:

llvm/test/CodeGen/X86/v8i1-masks.ll

 define <8 x i32> @six_or_and_xor(<8 x float> %x) {
 ; X86-LABEL: six_or_and_xor:
 ; X86:       ## %bb.0: ## %entry
 ; X86-NEXT:    vbroadcastss {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
 ; X86-NEXT:    vcmpleps %ymm0, %ymm1, %ymm1
 ; X86-NEXT:    vcmpltps {{\.?LCPI[0-9]+_[0-9]+}}, %ymm0, %ymm2
 ; X86-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; X86-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
-; X86-NEXT:    vandps %ymm3, %ymm2, %ymm2
-; X86-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}, %ymm0, %ymm3
-; X86-NEXT:    vandps %ymm3, %ymm2, %ymm2
+; X86-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}, %ymm0, %ymm4
+; X86-NEXT:    vandps %ymm4, %ymm3, %ymm3
+; X86-NEXT:    vandps %ymm2, %ymm3, %ymm2
 ; X86-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}, %ymm0, %ymm3
 ; X86-NEXT:    vxorps %ymm1, %ymm3, %ymm1
 ; X86-NEXT:    vxorps %ymm2, %ymm1, %ymm1
 ; X86-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}, %ymm0, %ymm0
 ; X86-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; X86-NEXT:    retl
 ;
 ; X64-LABEL: six_or_and_xor:
 ; X64:       ## %bb.0: ## %entry
 ; X64-NEXT:    vbroadcastss {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
 ; X64-NEXT:    vcmpleps %ymm0, %ymm1, %ymm1
 ; X64-NEXT:    vcmpltps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm2
 ; X64-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; X64-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
-; X64-NEXT:    vandps %ymm3, %ymm2, %ymm2
-; X64-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm3
-; X64-NEXT:    vandps %ymm3, %ymm2, %ymm2
+; X64-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm4
+; X64-NEXT:    vandps %ymm4, %ymm3, %ymm3
+; X64-NEXT:    vandps %ymm2, %ymm3, %ymm2
 ; X64-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm3
 ; X64-NEXT:    vxorps %ymm1, %ymm3, %ymm1
 ; X64-NEXT:    vxorps %ymm2, %ymm1, %ymm1
 ; X64-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %ymm0, %ymm0
 ; X64-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; X64-NEXT:    retq
 ;
 ; X86-AVX2-LABEL: six_or_and_xor:
 ; X86-AVX2:       ## %bb.0: ## %entry
 ; X86-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
 ; X86-AVX2-NEXT:    vcmpleps %ymm0, %ymm1, %ymm1
 ; X86-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; X86-AVX2-NEXT:    vcmpltps %ymm2, %ymm0, %ymm2
 ; X86-AVX2-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; X86-AVX2-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
-; X86-AVX2-NEXT:    vandps %ymm3, %ymm2, %ymm2
-; X86-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1]
-; X86-AVX2-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
-; X86-AVX2-NEXT:    vandps %ymm3, %ymm2, %ymm2
+; X86-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm4 = [1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1]
+; X86-AVX2-NEXT:    vcmpneqps %ymm4, %ymm0, %ymm4
+; X86-AVX2-NEXT:    vandps %ymm4, %ymm3, %ymm3
+; X86-AVX2-NEXT:    vandps %ymm2, %ymm3, %ymm2
 ; X86-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm3 = [2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1]
 ; X86-AVX2-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
 ; X86-AVX2-NEXT:    vxorps %ymm1, %ymm3, %ymm1
 ; X86-AVX2-NEXT:    vxorps %ymm2, %ymm1, %ymm1
 ; X86-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm2 = [4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1]
 ; X86-AVX2-NEXT:    vcmpneqps %ymm2, %ymm0, %ymm0
 ; X86-AVX2-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; X86-AVX2-NEXT:    retl
 ;
 ; X64-AVX2-LABEL: six_or_and_xor:
 ; X64-AVX2:       ## %bb.0: ## %entry
 ; X64-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
 ; X64-AVX2-NEXT:    vcmpleps %ymm0, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; X64-AVX2-NEXT:    vcmpltps %ymm2, %ymm0, %ymm2
 ; X64-AVX2-NEXT:    vxorps %xmm3, %xmm3, %xmm3
 ; X64-AVX2-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
-; X64-AVX2-NEXT:    vandps %ymm3, %ymm2, %ymm2
-; X64-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1]
-; X64-AVX2-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
-; X64-AVX2-NEXT:    vandps %ymm3, %ymm2, %ymm2
+; X64-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm4 = [1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1,1.00000001E-1]
+; X64-AVX2-NEXT:    vcmpneqps %ymm4, %ymm0, %ymm4
+; X64-AVX2-NEXT:    vandps %ymm4, %ymm3, %ymm3
+; X64-AVX2-NEXT:    vandps %ymm2, %ymm3, %ymm2
 ; X64-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm3 = [2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1,2.00000003E-1]
 ; X64-AVX2-NEXT:    vcmpneqps %ymm3, %ymm0, %ymm3
 ; X64-AVX2-NEXT:    vxorps %ymm1, %ymm3, %ymm1
 ; X64-AVX2-NEXT:    vxorps %ymm2, %ymm1, %ymm1
 ; X64-AVX2-NEXT:    vbroadcastss {{.*#+}} ymm2 = [4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1,4.00000006E-1]
 ; X64-AVX2-NEXT:    vcmpneqps %ymm2, %ymm0, %ymm0
 ; X64-AVX2-NEXT:    vorps %ymm0, %ymm1, %ymm0
 ; X64-AVX2-NEXT:    retq
 ;
 ; X86-AVX512-LABEL: six_or_and_xor:
 ; X86-AVX512:       ## %bb.0: ## %entry
 ; X86-AVX512-NEXT:    vcmpgeps {{\.?LCPI[0-9]+_[0-9]+}}{1to8}, %ymm0, %k0
 ; X86-AVX512-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X86-AVX512-NEXT:    vcmpneqps %ymm1, %ymm0, %k1
-; X86-AVX512-NEXT:    vcmpltps {{\.?LCPI[0-9]+_[0-9]+}}{1to8}, %ymm0, %k1 {%k1}
 ; X86-AVX512-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}{1to8}, %ymm0, %k1 {%k1}
+; X86-AVX512-NEXT:    vcmpltps {{\.?LCPI[0-9]+_[0-9]+}}{1to8}, %ymm0, %k1 {%k1}
 ; X86-AVX512-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}{1to8}, %ymm0, %k2
 ; X86-AVX512-NEXT:    kxorw %k0, %k2, %k0
 ; X86-AVX512-NEXT:    kxorw %k1, %k0, %k0
 ; X86-AVX512-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}{1to8}, %ymm0, %k1
 ; X86-AVX512-NEXT:    korw %k1, %k0, %k1
 ; X86-AVX512-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X86-AVX512-NEXT:    vmovdqa32 %ymm0, %ymm0 {%k1} {z}
 ; X86-AVX512-NEXT:    retl
 ;
 ; X64-AVX512-LABEL: six_or_and_xor:
 ; X64-AVX512:       ## %bb.0: ## %entry
 ; X64-AVX512-NEXT:    vcmpgeps {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to8}, %ymm0, %k0
 ; X64-AVX512-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; X64-AVX512-NEXT:    vcmpneqps %ymm1, %ymm0, %k1
-; X64-AVX512-NEXT:    vcmpltps {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to8}, %ymm0, %k1 {%k1}
 ; X64-AVX512-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to8}, %ymm0, %k1 {%k1}
+; X64-AVX512-NEXT:    vcmpltps {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to8}, %ymm0, %k1 {%k1}
 ; X64-AVX512-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to8}, %ymm0, %k2
 ; X64-AVX512-NEXT:    kxorw %k0, %k2, %k0
 ; X64-AVX512-NEXT:    kxorw %k1, %k0, %k0
 ; X64-AVX512-NEXT:    vcmpneqps {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to8}, %ymm0, %k1
 ; X64-AVX512-NEXT:    korw %k1, %k0, %k1
 ; X64-AVX512-NEXT:    vpcmpeqd %ymm0, %ymm0, %ymm0
 ; X64-AVX512-NEXT:    vmovdqa32 %ymm0, %ymm0 {%k1} {z}
 ; X64-AVX512-NEXT:    retq