[DAGCombiner] Update most ADD combines to support general vector combines

[RKSimon]

Summary

Add a number of helper functions to match scalar or vector equivalent constant/splat values to allow most of the combine patterns to be used by vectors.

[mkuper]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

829

What if we have a BUILD_VECTOR of constants where one of the constants going into it is opaque?

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

829

I see what you're saying. For the current cases it won't have much effect - some of the combines would switch ops around but the expected constant folding will just fail. I'll change isNonOpaqueConstantOrConstantVector to check for opaques in the build vector, and make a separate isConstantOrConstantVector helper as well which we can use in a couple of places. Thorough tests of opaque vector elements might be tricky though.

[RKSimon]

Updated isNonOpaqueConstantOrConstantVector to isNonOpaqueConstantOrNonOpaqueConstantVector to check that no vector elements are opaque constants. Also added isConstantOrConstantVector helper that doesn't care about opaque. Both of these allow undef vector elements.

Updated the helpers to not allow implicit truncation in build vectors - this will be necessary for the reuse of these helpers in other combines (e.g. SUB and MUL). This isn't necessary for the isNullConstantOrNullSplatConstant case.

[mkuper]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

858

Maybe merge isConstantOrConstantVector and isNonOpaqueConstantOrNonOpaqueConstantVector and have a nonOpaque parameter?

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

858

Will do - it'll help reduce the length of those function names!

[RKSimon]

Added NoOpaques optional argument to isConstantOrConstantVector and removed isNonOpaqueConstantOrNonOpaqueConstantVector helper

[mkuper]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

831

"return !(NoOpaques && Const->isOpaque())" ?
(assuming I got the logic right :-) )

[RKSimon]

Fixed opaque typo

[mkuper]

LGTM

Although I'm curious - do you see this in real code?
To the best of my understanding, the major goal of DAG-level constant folding is to get rid of constants that weren't visible on the IR level. This happens with scalars quite a lot (e.g. because of GEP lowering), but I'm not sure I understand where "new" vector constants come from.

[RKSimon]

In D25374#567407, @mkuper wrote:

LGTM

Although I'm curious - do you see this in real code?
To the best of my understanding, the major goal of DAG-level constant folding is to get rid of constants that weren't visible on the IR level. This happens with scalars quite a lot (e.g. because of GEP lowering), but I'm not sure I understand where "new" vector constants come from.

Thanks Michael, the only pattern from visitADD I've seen in the wild is "(a+b) -> (a|b)", but this will hopefully be the first stage in making most of the DAGCombiner combines properly support vector patterns when they do. Both visitSUB and visitAND/visitOR/visitXOR have a lot more cases that do occur but I wanted to start with a relatively easy one! These along with better vector support for computeKnownBits (et al.) should have a greater (accumulative) effect on vector IR.

[adriantong1024]

In D25374#565483, @RKSimon wrote:

Updated isNonOpaqueConstantOrConstantVector to isNonOpaqueConstantOrNonOpaqueConstantVector to check that no vector elements are opaque constants. Also added isConstantOrConstantVector helper that doesn't care about opaque. Both of these allow undef vector elements.

Updated the helpers to not allow implicit truncation in build vectors - this will be necessary for the reuse of these helpers in other combines (e.g. SUB and MUL). This isn't necessary for the isNullConstantOrNullSplatConstant case.

Hi @RKSimon

I know this CR is quite old, I am trying to remove the "not allow implicit truncation" restriction for one optimization. I am somewhat new to this part of the compiler, can you please explain to me why this is necessary for SUB and MUL combines. An example would be great. Thanks !

[RKSimon]

In D25374#3582196, @adriantong1024 wrote:

In D25374#565483, @RKSimon wrote:

Updated isNonOpaqueConstantOrConstantVector to isNonOpaqueConstantOrNonOpaqueConstantVector to check that no vector elements are opaque constants. Also added isConstantOrConstantVector helper that doesn't care about opaque. Both of these allow undef vector elements.

Updated the helpers to not allow implicit truncation in build vectors - this will be necessary for the reuse of these helpers in other combines (e.g. SUB and MUL). This isn't necessary for the isNullConstantOrNullSplatConstant case.

Hi @RKSimon

I know this CR is quite old, I am trying to remove the "not allow implicit truncation" restriction for one optimization. I am somewhat new to this part of the compiler, can you please explain to me why this is necessary for SUB and MUL combines. An example would be great. Thanks !

I can't remember tbh - best way to find out would be for you to remove the restriction and run check-llvm-codegen to see what changes/regresses/asserts/explodes....

[adriantong1024]

In D25374#3582281, @RKSimon wrote:

In D25374#3582196, @adriantong1024 wrote:

In D25374#565483, @RKSimon wrote:

Updated isNonOpaqueConstantOrConstantVector to isNonOpaqueConstantOrNonOpaqueConstantVector to check that no vector elements are opaque constants. Also added isConstantOrConstantVector helper that doesn't care about opaque. Both of these allow undef vector elements.

Updated the helpers to not allow implicit truncation in build vectors - this will be necessary for the reuse of these helpers in other combines (e.g. SUB and MUL). This isn't necessary for the isNullConstantOrNullSplatConstant case.

Hi @RKSimon

I know this CR is quite old, I am trying to remove the "not allow implicit truncation" restriction for one optimization. I am somewhat new to this part of the compiler, can you please explain to me why this is necessary for SUB and MUL combines. An example would be great. Thanks !

I can't remember tbh - best way to find out would be for you to remove the restriction and run check-llvm-codegen to see what changes/regresses/asserts/explodes....

so, I've ran check-llvm-codegen on all available targets and things are fine.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (BuildVectorSDNode *BV = dyn_cast<BuildVectorSDNode>(N)) {
 
     if (CN && UndefElements.none())
       return CN;
   }
 
   return nullptr;
 }
 
+static bool isNonOpaqueConstantOrConstantVector(SDValue N) {
+  if (ConstantSDNode *Const = dyn_cast<ConstantSDNode>(N))
+    return !Const->isOpaque();
+  return ISD::isBuildVectorOfConstantSDNodes(N.getNode());
+}
+
+static bool isNullConstantOrNullSplatConstant(SDValue N) {
+  if (ConstantSDNode *Splat = isConstOrConstSplat(N))
+    return Splat->isNullValue();
+  return false;
+}
+
+static bool isOneConstantOrOneSplatConstant(SDValue N) {
+  if (ConstantSDNode *Splat = isConstOrConstSplat(N))
+    return Splat->isOne();
+  return false;
+}
+
 SDValue DAGCombiner::ReassociateOps(unsigned Opc, const SDLoc &DL, SDValue N0,
                                     SDValue N1) {
   EVT VT = N0.getValueType();
   if (N0.getOpcode() == Opc) {
     if (SDNode *L = DAG.isConstantIntBuildVectorOrConstantInt(N0.getOperand(1))) {
       if (SDNode *R = DAG.isConstantIntBuildVectorOrConstantInt(N1)) {
         // reassoc. (op (op x, c1), c2) -> (op x, (op c1, c2))
         if (SDValue OpNode = DAG.FoldConstantArithmetic(Opc, DL, VT, L, R))
           return DAG.getNode(Opc, DL, VT, N0.getOperand(0), OpNode);
         return SDValue();
       }
       if (N0.hasOneUse()) {
         // reassoc. (op (op x, c1), y) -> (op (op x, y), c1) iff x+c1 has one
         // use
         SDValue OpNode = DAG.getNode(Opc, SDLoc(N0), VT, N0.getOperand(0), N1);
         if (!OpNode.getNode())
           return SDValue();
         AddToWorklist(OpNode.getNode());
         return DAG.getNode(Opc, DL, VT, OpNode, N0.getOperand(1));
       }
     }
   }
 
if (DAG.isConstantIntBuildVectorOrConstantInt(N0)) {
     // fold (add c1, c2) -> c1+c2
     return DAG.FoldConstantArithmetic(ISD::ADD, SDLoc(N), VT,
                                       N0.getNode(), N1.getNode());
   }
   // fold (add x, 0) -> x
   if (isNullConstant(N1))
     return N0;
   // fold ((c1-A)+c2) -> (c1+c2)-A
-  if (ConstantSDNode *N1C = getAsNonOpaqueConstant(N1)) {
+  if (isNonOpaqueConstantOrConstantVector(N1)) {
     if (N0.getOpcode() == ISD::SUB)
-      if (ConstantSDNode *N0C = getAsNonOpaqueConstant(N0.getOperand(0))) {
+      if (isNonOpaqueConstantOrConstantVector(N0.getOperand(0))) {
         SDLoc DL(N);
         return DAG.getNode(ISD::SUB, DL, VT,
-                           DAG.getConstant(N1C->getAPIntValue()+
-                                           N0C->getAPIntValue(), DL, VT),
+                           DAG.getNode(ISD::ADD, DL, VT, N1, N0.getOperand(0)),
                            N0.getOperand(1));
       }
   }
   // reassociate add
   if (SDValue RADD = ReassociateOps(ISD::ADD, SDLoc(N), N0, N1))
     return RADD;
   // fold ((0-A) + B) -> B-A
-  if (N0.getOpcode() == ISD::SUB && isNullConstant(N0.getOperand(0)))
+  if (N0.getOpcode() == ISD::SUB &&
+      isNullConstantOrNullSplatConstant(N0.getOperand(0)))
     return DAG.getNode(ISD::SUB, SDLoc(N), VT, N1, N0.getOperand(1));
   // fold (A + (0-B)) -> A-B
-  if (N1.getOpcode() == ISD::SUB && isNullConstant(N1.getOperand(0)))
+  if (N1.getOpcode() == ISD::SUB &&
+      isNullConstantOrNullSplatConstant(N1.getOperand(0)))
     return DAG.getNode(ISD::SUB, SDLoc(N), VT, N0, N1.getOperand(1));
   // fold (A+(B-A)) -> B
   if (N1.getOpcode() == ISD::SUB && N0 == N1.getOperand(1))
     return N1.getOperand(0);
   // fold ((B-A)+A) -> B
   if (N0.getOpcode() == ISD::SUB && N1 == N0.getOperand(1))
     return N0.getOperand(0);
   // fold (A+(B-(A+C))) to (B-C)
SDValue DAGCombiner::visitADD(SDNode *N) {
 
   // fold (A-B)+(C-D) to (A+C)-(B+D) when A or C is constant
   if (N0.getOpcode() == ISD::SUB && N1.getOpcode() == ISD::SUB) {
     SDValue N00 = N0.getOperand(0);
     SDValue N01 = N0.getOperand(1);
     SDValue N10 = N1.getOperand(0);
     SDValue N11 = N1.getOperand(1);
 
-    if (isa<ConstantSDNode>(N00) || isa<ConstantSDNode>(N10))
+    if (isNonOpaqueConstantOrConstantVector(N00) ||
+        isNonOpaqueConstantOrConstantVector(N10))
       return DAG.getNode(ISD::SUB, SDLoc(N), VT,
                          DAG.getNode(ISD::ADD, SDLoc(N0), VT, N00, N10),
                          DAG.getNode(ISD::ADD, SDLoc(N1), VT, N01, N11));
   }
 
   if (!VT.isVector() && SimplifyDemandedBits(SDValue(N, 0)))
     return SDValue(N, 0);
 
   // fold (a+b) -> (a|b) iff a and b share no bits.
   if ((!LegalOperations || TLI.isOperationLegal(ISD::OR, VT)) &&
-      VT.isInteger() && !VT.isVector() && DAG.haveNoCommonBitsSet(N0, N1))
+      VT.isInteger() && DAG.haveNoCommonBitsSet(N0, N1))
     return DAG.getNode(ISD::OR, SDLoc(N), VT, N0, N1);
 
   // fold (add x, shl(0 - y, n)) -> sub(x, shl(y, n))
   if (N1.getOpcode() == ISD::SHL && N1.getOperand(0).getOpcode() == ISD::SUB &&
-      isNullConstant(N1.getOperand(0).getOperand(0)))
+      isNullConstantOrNullSplatConstant(N1.getOperand(0).getOperand(0)))
     return DAG.getNode(ISD::SUB, SDLoc(N), VT, N0,
                        DAG.getNode(ISD::SHL, SDLoc(N), VT,
                                    N1.getOperand(0).getOperand(1),
                                    N1.getOperand(1)));
   if (N0.getOpcode() == ISD::SHL && N0.getOperand(0).getOpcode() == ISD::SUB &&
-      isNullConstant(N0.getOperand(0).getOperand(0)))
+      isNullConstantOrNullSplatConstant(N0.getOperand(0).getOperand(0)))
     return DAG.getNode(ISD::SUB, SDLoc(N), VT, N1,
                        DAG.getNode(ISD::SHL, SDLoc(N), VT,
                                    N0.getOperand(0).getOperand(1),
                                    N0.getOperand(1)));
 
   if (N1.getOpcode() == ISD::AND) {
     SDValue AndOp0 = N1.getOperand(0);
     unsigned NumSignBits = DAG.ComputeNumSignBits(AndOp0);
     unsigned DestBits = VT.getScalarSizeInBits();
 
     // (add z, (and (sbbl x, x), 1)) -> (sub z, (sbbl x, x))
     // and similar xforms where the inner op is either ~0 or 0.
-    if (NumSignBits == DestBits && isOneConstant(N1->getOperand(1))) {
+    if (NumSignBits == DestBits &&
+        isOneConstantOrOneSplatConstant(N1->getOperand(1))) {
       SDLoc DL(N);
       return DAG.getNode(ISD::SUB, DL, VT, N->getOperand(0), AndOp0);
     }
   }
 
   // add (sext i1), X -> sub X, (zext i1)
   if (N0.getOpcode() == ISD::SIGN_EXTEND &&
       N0.getOperand(0).getValueType() == MVT::i1 &&

test/CodeGen/X86/combine-add.ll

; AVX-NEXT:    retq
   %1 = add <4 x i32> %a, zeroinitializer
   ret <4 x i32> %1
 }
 
 ; fold ((c1-A)+c2) -> (c1+c2)-A
 define <4 x i32> @combine_vec_add_constant_sub(<4 x i32> %a) {
 ; SSE-LABEL: combine_vec_add_constant_sub:
 ; SSE:       # BB#0:
-; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [0,1,2,3]
-; SSE-NEXT:    movdqa %xmm2, %xmm1
+; SSE-NEXT:    movdqa {{.*#+}} xmm1 = [0,2,4,6]
 ; SSE-NEXT:    psubd %xmm0, %xmm1
-; SSE-NEXT:    paddd %xmm2, %xmm1
 ; SSE-NEXT:    movdqa %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_constant_sub:
 ; AVX:       # BB#0:
-; AVX-NEXT:    vmovdqa {{.*#+}} xmm1 = [0,1,2,3]
+; AVX-NEXT:    vmovdqa {{.*#+}} xmm1 = [0,2,4,6]
 ; AVX-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = sub <4 x i32> <i32 0, i32 1, i32 2, i32 3>, %a
   %2 = add <4 x i32> <i32 0, i32 1, i32 2, i32 3>, %1
   ret <4 x i32> %2
 }
 
 ; fold ((0-A) + B) -> B-A
 define <4 x i32> @combine_vec_add_neg0(<4 x i32> %a, <4 x i32> %b) {
 ; SSE-LABEL: combine_vec_add_neg0:
 ; SSE:       # BB#0:
-; SSE-NEXT:    pxor %xmm2, %xmm2
-; SSE-NEXT:    psubd %xmm0, %xmm2
-; SSE-NEXT:    paddd %xmm1, %xmm2
-; SSE-NEXT:    movdqa %xmm2, %xmm0
+; SSE-NEXT:    psubd %xmm0, %xmm1
+; SSE-NEXT:    movdqa %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_neg0:
 ; AVX:       # BB#0:
-; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX-NEXT:    vpsubd %xmm0, %xmm2, %xmm0
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
 ; AVX-NEXT:    retq
   %1 = sub <4 x i32> zeroinitializer, %a
   %2 = add <4 x i32> %1, %b
   ret <4 x i32> %2
 }
 
 ; fold (A + (0-B)) -> A-B
 define <4 x i32> @combine_vec_add_neg1(<4 x i32> %a, <4 x i32> %b) {
 ; SSE-LABEL: combine_vec_add_neg1:
 ; SSE:       # BB#0:
-; SSE-NEXT:    pxor %xmm2, %xmm2
-; SSE-NEXT:    psubd %xmm1, %xmm2
-; SSE-NEXT:    paddd %xmm2, %xmm0
+; SSE-NEXT:    psubd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_neg1:
 ; AVX:       # BB#0:
-; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX-NEXT:    vpsubd %xmm1, %xmm2, %xmm1
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = sub <4 x i32> zeroinitializer, %b
   %2 = add <4 x i32> %a, %1
   ret <4 x i32> %2
 }
 
 ; fold (A+(B-A)) -> B
 define <4 x i32> @combine_vec_add_sub0(<4 x i32> %a, <4 x i32> %b) {
; AVX-NEXT:    retq
   %3 = add <4 x i32> %a, %2
   ret <4 x i32> %3
 }
 
 ; fold (A-B)+(C-D) to (A+C)-(B+D) when A or C is constant
 define <4 x i32> @combine_vec_add_sub_sub(<4 x i32> %a, <4 x i32> %b, <4 x i32> %d) {
 ; SSE-LABEL: combine_vec_add_sub_sub:
 ; SSE:       # BB#0:
+; SSE-NEXT:    paddd {{.*}}(%rip), %xmm0
+; SSE-NEXT:    paddd %xmm2, %xmm1
 ; SSE-NEXT:    psubd %xmm1, %xmm0
-; SSE-NEXT:    movdqa {{.*#+}} xmm1 = [0,1,2,3]
-; SSE-NEXT:    psubd %xmm2, %xmm1
-; SSE-NEXT:    paddd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_sub_sub:
 ; AVX:       # BB#0:
+; AVX-NEXT:    vpaddd {{.*}}(%rip), %xmm0, %xmm0
+; AVX-NEXT:    vpaddd %xmm2, %xmm1, %xmm1
 ; AVX-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
-; AVX-NEXT:    vmovdqa {{.*#+}} xmm1 = [0,1,2,3]
-; AVX-NEXT:    vpsubd %xmm2, %xmm1, %xmm1
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = sub <4 x i32> %a, %b
   %2 = sub <4 x i32> <i32 0, i32 1, i32 2, i32 3>, %d
   %3 = add <4 x i32> %1, %2
   ret <4 x i32> %3
 }
 
 ; fold (a+b) -> (a|b) iff a and b share no bits.
 define <4 x i32> @combine_vec_add_uniquebits(<4 x i32> %a, <4 x i32> %b) {
 ; SSE-LABEL: combine_vec_add_uniquebits:
 ; SSE:       # BB#0:
-; SSE-NEXT:    pand {{.*}}(%rip), %xmm0
-; SSE-NEXT:    pand {{.*}}(%rip), %xmm1
-; SSE-NEXT:    paddd %xmm1, %xmm0
+; SSE-NEXT:    andps {{.*}}(%rip), %xmm0
+; SSE-NEXT:    andps {{.*}}(%rip), %xmm1
+; SSE-NEXT:    orps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_uniquebits:
 ; AVX:       # BB#0:
-; AVX-NEXT:    vpbroadcastd {{.*}}(%rip), %xmm2
-; AVX-NEXT:    vpand %xmm2, %xmm0, %xmm0
-; AVX-NEXT:    vpbroadcastd {{.*}}(%rip), %xmm2
-; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm1
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vbroadcastss {{.*}}(%rip), %xmm2
+; AVX-NEXT:    vandps %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    vbroadcastss {{.*}}(%rip), %xmm2
+; AVX-NEXT:    vandps %xmm2, %xmm1, %xmm1
+; AVX-NEXT:    vorps %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = and <4 x i32> %a, <i32 61680, i32 61680, i32 61680, i32 61680>
   %2 = and <4 x i32> %b, <i32 3855, i32 3855, i32 3855, i32 3855>
   %3 = add <4 x i32> %1, %2
   ret <4 x i32> %3
 }
 
 ; fold (add x, shl(0 - y, n)) -> sub(x, shl(y, n))
 define <4 x i32> @combine_vec_add_shl_neg0(<4 x i32> %x, <4 x i32> %y) {
 ; SSE-LABEL: combine_vec_add_shl_neg0:
 ; SSE:       # BB#0:
-; SSE-NEXT:    pxor %xmm2, %xmm2
-; SSE-NEXT:    psubd %xmm1, %xmm2
-; SSE-NEXT:    pslld $5, %xmm2
-; SSE-NEXT:    paddd %xmm2, %xmm0
+; SSE-NEXT:    pslld $5, %xmm1
+; SSE-NEXT:    psubd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_shl_neg0:
 ; AVX:       # BB#0:
-; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX-NEXT:    vpsubd %xmm1, %xmm2, %xmm1
 ; AVX-NEXT:    vpslld $5, %xmm1, %xmm1
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = sub <4 x i32> zeroinitializer, %y
   %2 = shl <4 x i32> %1, <i32 5, i32 5, i32 5, i32 5>
   %3 = add <4 x i32> %x, %2
   ret <4 x i32> %3
 }
 
 ; fold (add shl(0 - y, n), x) -> sub(x, shl(y, n))
 define <4 x i32> @combine_vec_add_shl_neg1(<4 x i32> %x, <4 x i32> %y) {
 ; SSE-LABEL: combine_vec_add_shl_neg1:
 ; SSE:       # BB#0:
-; SSE-NEXT:    pxor %xmm2, %xmm2
-; SSE-NEXT:    psubd %xmm1, %xmm2
-; SSE-NEXT:    pslld $5, %xmm2
-; SSE-NEXT:    paddd %xmm2, %xmm0
+; SSE-NEXT:    pslld $5, %xmm1
+; SSE-NEXT:    psubd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_shl_neg1:
 ; AVX:       # BB#0:
-; AVX-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX-NEXT:    vpsubd %xmm1, %xmm2, %xmm1
 ; AVX-NEXT:    vpslld $5, %xmm1, %xmm1
-; AVX-NEXT:    vpaddd %xmm0, %xmm1, %xmm0
+; AVX-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = sub <4 x i32> zeroinitializer, %y
   %2 = shl <4 x i32> %1, <i32 5, i32 5, i32 5, i32 5>
   %3 = add <4 x i32> %2, %x
   ret <4 x i32> %3
 }
 
 ; (add z, (and (sbbl x, x), 1)) -> (sub z, (sbbl x, x))
 ; and similar xforms where the inner op is either ~0 or 0.
 define <4 x i32> @combine_vec_add_and_compare(<4 x i32> %a0, <4 x i32> %a1, <4 x i32> %a2) {
 ; SSE-LABEL: combine_vec_add_and_compare:
 ; SSE:       # BB#0:
 ; SSE-NEXT:    pcmpeqd %xmm2, %xmm1
-; SSE-NEXT:    psrld $31, %xmm1
-; SSE-NEXT:    paddd %xmm1, %xmm0
+; SSE-NEXT:    psubd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: combine_vec_add_and_compare:
 ; AVX:       # BB#0:
 ; AVX-NEXT:    vpcmpeqd %xmm2, %xmm1, %xmm1
-; AVX-NEXT:    vpsrld $31, %xmm1, %xmm1
-; AVX-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %1 = icmp eq <4 x i32> %a1, %a2
   %2 = sext <4 x i1> %1 to <4 x i32>
   %3 = and <4 x i32> %2, <i32 1, i32 1, i32 1, i32 1>
   %4 = add <4 x i32> %a0, %3
   ret <4 x i32> %4
 }