[DAG] Add SimplifyDemandedVectorElts binop SimplifyMultipleUseDemandedBits handling

[RKSimon]

Summary

For the supported binops (basic arithmetic, logicals + shifts), if we fail to simplify the demanded vector elts, then call SimplifyMultipleUseDemandedBits and try to peek through ops to remove unnecessary dependencies.

This helps with PR40502.

[RKSimon]

llvm/test/CodeGen/AArch64/mul_by_elt.ll

145

Not sure about this change - it strips the first shufflevector entirely losing the multiply by scalar, but instcombine would have done something similar anyhow:

define <4 x float> @splat0_before_fmul_fmul_constant(<4 x float> %a) {
  %1 = fmul <4 x float> %a, <float 3.000000e+00, float undef, float undef, float undef>
  %2 = fmul <4 x float> %1, <float 6.000000e+00, float undef, float undef, float undef>
  %mul2 = shufflevector <4 x float> %2, <4 x float> undef, <4 x i32> zeroinitializer
  ret <4 x float> %mul2
}

[RKSimon]

ping?

[lebedev.ri]

Seems reasonable to me in general.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

2682–2693

Just wondering, should we not demand elts that are KnownUndef?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

2682–2693

That's a good idea - decent tests might be tricky but should be doable. A slight thing to watch out for is KnownUndef/KnownZero bits are only guaranteed to be correct for demanded elts.

Would it be OK just to add this as a TODO comment for now? It might need some investigation (and improvements to KnownUndef/KnownZero emission which isn't great yet as we don't use it enough).

[lebedev.ri]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

2682–2693

That's a good idea - decent tests might be tricky but should be doable.

Aha. That might even be more generic than this if-block.

Would it be OK just to add this as a TODO comment for now?

Yes, absolutely.

[craig.topper]

llvm/test/CodeGen/AArch64/mul_by_elt.ll

145

Why does "multiple use" come into play here?

[RKSimon]

llvm/test/CodeGen/AArch64/mul_by_elt.ll

145

Because SimplifyMultipleUseDemandedBits will peek through a shuffle entirely but SimplifyDemandedVectorElts is (overly) careful at not simplifying to an identity shuffle - which tbh is something we should get rid of.

I'm not sure why the constant splat isn't being used as the scalar operand as an alternative though.

[RKSimon]

add KnownUndef TODO comments

[RKSimon]

ping - any more comments?

[spatel]

The code duplication is making me itchy. Add a helper like:

if (simplifyDemandedVectorEltsBinop(Op, DemandedElts, TLO, Depth))
  return true;

That could include just the new block that's being created here, or we can dispatch directly on all of the binop cases in the top-level and then switch for the KnownZero/Undef/DemandedBits differences within there.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

2687

Is it correct to recycle the demanded bits on both operands here? That seems wrong for FP ops IIUC.

[RKSimon]

Cheers @spatel - I've pulled out the SimplifyDemandedVectorEltsBinOp helper which makes separate DemandedBits 'all bits' maskes for each op which should also answer the query about the fpops.

[spatel]

LGTM.
I'd sink the isAllOnesValue() check and/or hoist the CombineTo() into the lambda too for still less repeated code, but maybe there's more differentiation from known undef that could happen here?

[RKSimon]

In D79003#2052679, @spatel wrote:

LGTM.
I'd sink the isAllOnesValue() check and/or hoist the CombineTo() into the lambda too for still less repeated code, but maybe there's more differentiation from known undef that could happen here?

I'll move the CombineTo handling into the lambda as well - the isAllOnes() I'm going to leave out until I can work out how to best merge it with KnownUndef as mentioned in the TODOs.

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

bool TargetLowering::SimplifyDemandedVectorElts(
 
   // Limit search depth.
   if (Depth >= SelectionDAG::MaxRecursionDepth)
     return false;
 
   SDLoc DL(Op);
   unsigned EltSizeInBits = VT.getScalarSizeInBits();
 
-  switch (Op.getOpcode()) {
+  // Helper for demanding the specified elements and all the bits of both binary
+  // operands.
+  auto SimplifyDemandedVectorEltsBinOp = [&](SDValue &Op0, SDValue &Op1) {
+    unsigned NumBits0 = Op0.getScalarValueSizeInBits();
+    unsigned NumBits1 = Op1.getScalarValueSizeInBits();
+    APInt DemandedBits0 = APInt::getAllOnesValue(NumBits0);
+    APInt DemandedBits1 = APInt::getAllOnesValue(NumBits1);
+    SDValue NewOp0 = SimplifyMultipleUseDemandedBits(
+        Op0, DemandedBits0, DemandedElts, TLO.DAG, Depth + 1);
+    SDValue NewOp1 = SimplifyMultipleUseDemandedBits(
+        Op1, DemandedBits1, DemandedElts, TLO.DAG, Depth + 1);
+    if (NewOp0 || NewOp1) {
+      Op0 = NewOp0 ? NewOp0 : Op0;
+      Op1 = NewOp1 ? NewOp1 : Op1;
+      return true;
+    }
+    return false;
+  };
+
+  unsigned Opcode = Op.getOpcode();
+  switch (Opcode) {
   case ISD::SCALAR_TO_VECTOR: {
     if (!DemandedElts[0]) {
       KnownUndef.setAllBits();
       return TLO.CombineTo(Op, TLO.DAG.getUNDEF(VT));
     }
     KnownUndef.setHighBits(NumElts - 1);
     break;
   }
if (Op.getOpcode() == ISD::ZERO_EXTEND_VECTOR_INREG) {
       // zext(undef) upper bits are guaranteed to be zero.
       if (DemandedElts.isSubsetOf(KnownUndef))
         return TLO.CombineTo(Op, TLO.DAG.getConstant(0, SDLoc(Op), VT));
       KnownUndef.clearAllBits();
     }
     break;
   }
 
-  // TODO: There are more binop opcodes that could be handled here - MUL, MIN,
+  // TODO: There are more binop opcodes that could be handled here - MIN,
   // MAX, saturated math, etc.
   case ISD::OR:
   case ISD::XOR:
   case ISD::ADD:
   case ISD::SUB:
   case ISD::FADD:
   case ISD::FSUB:
   case ISD::FMUL:
   case ISD::FDIV:
   case ISD::FREM: {
+    SDValue Op0 = Op.getOperand(0);
+    SDValue Op1 = Op.getOperand(1);
+
     APInt UndefRHS, ZeroRHS;
-    if (SimplifyDemandedVectorElts(Op.getOperand(1), DemandedElts, UndefRHS,
-                                   ZeroRHS, TLO, Depth + 1))
+    if (SimplifyDemandedVectorElts(Op1, DemandedElts, UndefRHS, ZeroRHS, TLO,
+                                   Depth + 1))
       return true;
     APInt UndefLHS, ZeroLHS;
-    if (SimplifyDemandedVectorElts(Op.getOperand(0), DemandedElts, UndefLHS,
-                                   ZeroLHS, TLO, Depth + 1))
+    if (SimplifyDemandedVectorElts(Op0, DemandedElts, UndefLHS, ZeroLHS, TLO,
+                                   Depth + 1))
       return true;
 
     KnownZero = ZeroLHS & ZeroRHS;
     KnownUndef = getKnownUndefForVectorBinop(Op, TLO.DAG, UndefLHS, UndefRHS);
+
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    // TODO - use KnownUndef to relax the demandedelts?
+    if (!DemandedElts.isAllOnesValue()) {
+      SDValue NewOp0 = Op0, NewOp1 = Op1;
+      if (SimplifyDemandedVectorEltsBinOp(NewOp0, NewOp1)) {
+        SDValue NewOp = TLO.DAG.getNode(Opcode, SDLoc(Op), VT, NewOp0, NewOp1);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
     break;
   }
   case ISD::SHL:
   case ISD::SRL:
   case ISD::SRA:
   case ISD::ROTL:
   case ISD::ROTR: {
+    SDValue Op0 = Op.getOperand(0);
+    SDValue Op1 = Op.getOperand(1);
+
     APInt UndefRHS, ZeroRHS;
-    if (SimplifyDemandedVectorElts(Op.getOperand(1), DemandedElts, UndefRHS,
-                                   ZeroRHS, TLO, Depth + 1))
+    if (SimplifyDemandedVectorElts(Op1, DemandedElts, UndefRHS, ZeroRHS, TLO,
+                                   Depth + 1))
       return true;
     APInt UndefLHS, ZeroLHS;
-    if (SimplifyDemandedVectorElts(Op.getOperand(0), DemandedElts, UndefLHS,
-                                   ZeroLHS, TLO, Depth + 1))
+    if (SimplifyDemandedVectorElts(Op0, DemandedElts, UndefLHS, ZeroLHS, TLO,
+                                   Depth + 1))
       return true;
 
     KnownZero = ZeroLHS;
     KnownUndef = UndefLHS & UndefRHS; // TODO: use getKnownUndefForVectorBinop?
+
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    // TODO - use KnownUndef to relax the demandedelts?
+    if (!DemandedElts.isAllOnesValue()) {
+      SDValue NewOp0 = Op0, NewOp1 = Op1;
+      if (SimplifyDemandedVectorEltsBinOp(NewOp0, NewOp1)) {
+        SDValue NewOp = TLO.DAG.getNode(Opcode, SDLoc(Op), VT, NewOp0, NewOp1);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
     break;
   }
   case ISD::MUL:
   case ISD::AND: {
+    SDValue Op0 = Op.getOperand(0);
+    SDValue Op1 = Op.getOperand(1);
+
     APInt SrcUndef, SrcZero;
-    if (SimplifyDemandedVectorElts(Op.getOperand(1), DemandedElts, SrcUndef,
-                                   SrcZero, TLO, Depth + 1))
+    if (SimplifyDemandedVectorElts(Op1, DemandedElts, SrcUndef, SrcZero, TLO,
+                                   Depth + 1))
       return true;
-    if (SimplifyDemandedVectorElts(Op.getOperand(0), DemandedElts, KnownUndef,
-                                   KnownZero, TLO, Depth + 1))
+    if (SimplifyDemandedVectorElts(Op0, DemandedElts, KnownUndef, KnownZero,
+                                   TLO, Depth + 1))
       return true;
 
     // If either side has a zero element, then the result element is zero, even
     // if the other is an UNDEF.
     // TODO: Extend getKnownUndefForVectorBinop to also deal with known zeros
     // and then handle 'and' nodes with the rest of the binop opcodes.
     KnownZero |= SrcZero;
     KnownUndef &= SrcUndef;
     KnownUndef &= ~KnownZero;
+
+    // Attempt to avoid multi-use ops if we don't need anything from them.
+    // TODO - use KnownUndef to relax the demandedelts?
+    if (!DemandedElts.isAllOnesValue()) {
+      SDValue NewOp0 = Op0, NewOp1 = Op1;
+      if (SimplifyDemandedVectorEltsBinOp(NewOp0, NewOp1)) {
+        SDValue NewOp = TLO.DAG.getNode(Opcode, SDLoc(Op), VT, NewOp0, NewOp1);
+        return TLO.CombineTo(Op, NewOp);
+      }
+    }
     break;
   }
   case ISD::TRUNCATE:
   case ISD::SIGN_EXTEND:
   case ISD::ZERO_EXTEND:
     if (SimplifyDemandedVectorElts(Op.getOperand(0), DemandedElts, KnownUndef,
                                    KnownZero, TLO, Depth + 1))
       return true;

llvm/test/CodeGen/AArch64/mul_by_elt.ll

 }
 
 ; Multiple multiplies.
 
 define <4 x float> @splat0_before_fmul_fmul_constant(<4 x float> %a) {
 ; CHECK-LABEL: splat0_before_fmul_fmul_constant:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fmov v1.4s, #3.00000000
-; CHECK-NEXT:    fmul v0.4s, v1.4s, v0.s[0]
+; CHECK-NEXT:    fmul v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    fmov v1.4s, #6.00000000
 ; CHECK-NEXT:    fmul v0.4s, v1.4s, v0.s[0]
 ; CHECK-NEXT:    ret
   %splat1 = shufflevector <4 x float> %a, <4 x float> undef, <4 x i32> zeroinitializer
   %mul1 = fmul <4 x float> %splat1, <float 3.0, float 3.0, float 3.0, float 3.0>
   %splat2 = shufflevector <4 x float> %mul1, <4 x float> undef, <4 x i32> zeroinitializer
   %mul2 = fmul <4 x float> %splat2, <float 6.0, float 6.0, float 6.0, float 6.0>
   ret <4 x float> %mul2
 }
 
 define <4 x float> @splat0_after_fmul_fmul_constant(<4 x float> %a) {
 ; CHECK-LABEL: splat0_after_fmul_fmul_constant:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fmov v1.4s, #3.00000000
 ; CHECK-NEXT:    fmul v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    fmov v1.4s, #6.00000000
 ; CHECK-NEXT:    fmul v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    dup v0.4s, v0.s[0]
 ; CHECK-NEXT:    ret
   %mul1 = fmul <4 x float> %a, <float 3.0, float 42.0, float 3.0, float 3.0>
   %mul2 = fmul <4 x float> %mul1, <float 6.0, float 42.0, float 3.0, float 3.0>
   %splat = shufflevector <4 x float> %mul2, <4 x float> undef, <4 x i32> zeroinitializer
   ret <4 x float> %splat
 }

llvm/test/CodeGen/X86/combine-pmuldq.ll

 ; SSE-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[1,1,3,3]
 ; SSE-NEXT:    pmuludq %xmm2, %xmm3
 ; SSE-NEXT:    movdqa %xmm0, %xmm2
 ; SSE-NEXT:    psrld $1, %xmm2
 ; SSE-NEXT:    pblendw {{.*#+}} xmm2 = xmm0[0,1,2,3],xmm2[4,5],xmm0[6,7]
 ; SSE-NEXT:    pmuludq %xmm1, %xmm2
 ; SSE-NEXT:    pshufd {{.*#+}} xmm1 = xmm2[1,1,3,3]
 ; SSE-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1],xmm3[2,3],xmm1[4,5],xmm3[6,7]
-; SSE-NEXT:    psubd %xmm1, %xmm0
+; SSE-NEXT:    psubd %xmm3, %xmm0
 ; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; SSE-NEXT:    pmuludq {{.*}}(%rip), %xmm0
 ; SSE-NEXT:    pxor %xmm2, %xmm2
 ; SSE-NEXT:    pblendw {{.*#+}} xmm2 = xmm2[0,1],xmm0[2,3],xmm2[4,5],xmm0[6,7]
 ; SSE-NEXT:    paddd %xmm1, %xmm2
 ; SSE-NEXT:    movdqa %xmm2, %xmm0
 ; SSE-NEXT:    psrld $7, %xmm0
 ; SSE-NEXT:    psrld $6, %xmm2

 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm1 = [1645975491,344322273,2164392969,1916962805]
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[1,1,3,3]
 ; AVX2-NEXT:    vpsrlvd {{.*}}(%rip), %xmm0, %xmm3
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm4 = xmm3[1,1,3,3]
 ; AVX2-NEXT:    vpmuludq %xmm2, %xmm4, %xmm2
 ; AVX2-NEXT:    vpmuludq %xmm1, %xmm3, %xmm1
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
 ; AVX2-NEXT:    vpblendd {{.*#+}} xmm1 = xmm1[0],xmm2[1],xmm1[2],xmm2[3]
-; AVX2-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpsubd %xmm2, %xmm0, %xmm0
 ; AVX2-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; AVX2-NEXT:    vpbroadcastd {{.*#+}} xmm2 = [2147483648,2147483648,2147483648,2147483648]
 ; AVX2-NEXT:    vpmuludq %xmm2, %xmm0, %xmm0
 ; AVX2-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX2-NEXT:    vpblendd {{.*#+}} xmm0 = xmm2[0],xmm0[1],xmm2[2],xmm0[3]
 ; AVX2-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX2-NEXT:    vpsrlvd {{.*}}(%rip), %xmm0, %xmm0
 ; AVX2-NEXT:    vmovd %xmm0, %edi
 ; AVX2-NEXT:    vpextrd $1, %xmm0, %esi
 ; AVX2-NEXT:    vpextrd $2, %xmm0, %edx
 ; AVX2-NEXT:    vpextrd $3, %xmm0, %ecx
 ; AVX2-NEXT:    jmp foo # TAILCALL
 ;
 ; AVX512VL-LABEL: PR43159:
 ; AVX512VL:       # %bb.0: # %entry
 ; AVX512VL-NEXT:    vmovdqa (%rdi), %xmm0
 ; AVX512VL-NEXT:    vmovdqa {{.*#+}} xmm1 = [1645975491,344322273,2164392969,1916962805]
 ; AVX512VL-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[1,1,3,3]
 ; AVX512VL-NEXT:    vpsrlvd {{.*}}(%rip), %xmm0, %xmm3
 ; AVX512VL-NEXT:    vpshufd {{.*#+}} xmm4 = xmm3[1,1,3,3]
 ; AVX512VL-NEXT:    vpmuludq %xmm2, %xmm4, %xmm2
 ; AVX512VL-NEXT:    vpmuludq %xmm1, %xmm3, %xmm1
 ; AVX512VL-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
 ; AVX512VL-NEXT:    vpblendd {{.*#+}} xmm1 = xmm1[0],xmm2[1],xmm1[2],xmm2[3]
-; AVX512VL-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
+; AVX512VL-NEXT:    vpsubd %xmm2, %xmm0, %xmm0
 ; AVX512VL-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; AVX512VL-NEXT:    vpbroadcastd {{.*#+}} xmm2 = [2147483648,2147483648,2147483648,2147483648]
 ; AVX512VL-NEXT:    vpmuludq %xmm2, %xmm0, %xmm0
 ; AVX512VL-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX512VL-NEXT:    vpblendd {{.*#+}} xmm0 = xmm2[0],xmm0[1],xmm2[2],xmm0[3]
 ; AVX512VL-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX512VL-NEXT:    vpsrlvd {{.*}}(%rip), %xmm0, %xmm0
 ; AVX512VL-NEXT:    vmovd %xmm0, %edi
 ; AVX512VL-NEXT:    vpextrd $1, %xmm0, %esi
 ; AVX512VL-NEXT:    vpextrd $2, %xmm0, %edx
 ; AVX512VL-NEXT:    vpextrd $3, %xmm0, %ecx
 ; AVX512VL-NEXT:    jmp foo # TAILCALL
 ;
 ; AVX512DQVL-LABEL: PR43159:
 ; AVX512DQVL:       # %bb.0: # %entry
 ; AVX512DQVL-NEXT:    vmovdqa (%rdi), %xmm0
 ; AVX512DQVL-NEXT:    vmovdqa {{.*#+}} xmm1 = [1645975491,344322273,2164392969,1916962805]
 ; AVX512DQVL-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[1,1,3,3]
 ; AVX512DQVL-NEXT:    vpsrlvd {{.*}}(%rip), %xmm0, %xmm3
 ; AVX512DQVL-NEXT:    vpshufd {{.*#+}} xmm4 = xmm3[1,1,3,3]
 ; AVX512DQVL-NEXT:    vpmuludq %xmm2, %xmm4, %xmm2
 ; AVX512DQVL-NEXT:    vpmuludq %xmm1, %xmm3, %xmm1
 ; AVX512DQVL-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[1,1,3,3]
 ; AVX512DQVL-NEXT:    vpblendd {{.*#+}} xmm1 = xmm1[0],xmm2[1],xmm1[2],xmm2[3]
-; AVX512DQVL-NEXT:    vpsubd %xmm1, %xmm0, %xmm0
+; AVX512DQVL-NEXT:    vpsubd %xmm2, %xmm0, %xmm0
 ; AVX512DQVL-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,3,3]
 ; AVX512DQVL-NEXT:    vpbroadcastd {{.*#+}} xmm2 = [2147483648,2147483648,2147483648,2147483648]
 ; AVX512DQVL-NEXT:    vpmuludq %xmm2, %xmm0, %xmm0
 ; AVX512DQVL-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; AVX512DQVL-NEXT:    vpblendd {{.*#+}} xmm0 = xmm2[0],xmm0[1],xmm2[2],xmm0[3]
 ; AVX512DQVL-NEXT:    vpaddd %xmm1, %xmm0, %xmm0
 ; AVX512DQVL-NEXT:    vpsrlvd {{.*}}(%rip), %xmm0, %xmm0
 ; AVX512DQVL-NEXT:    vmovd %xmm0, %edi

llvm/test/CodeGen/X86/combine-sdiv.ll

 ; SSE2-NEXT:    paddd %xmm1, %xmm0
 ; SSE2-NEXT:    movdqa %xmm0, %xmm2
 ; SSE2-NEXT:    psrad $4, %xmm2
 ; SSE2-NEXT:    movdqa %xmm0, %xmm3
 ; SSE2-NEXT:    psrad $3, %xmm3
 ; SSE2-NEXT:    punpckhqdq {{.*#+}} xmm3 = xmm3[1],xmm2[1]
 ; SSE2-NEXT:    psrad $2, %xmm0
 ; SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,1],xmm3[0,3]
-; SSE2-NEXT:    movss {{.*#+}} xmm0 = xmm1[0],xmm0[1,2,3]
-; SSE2-NEXT:    pxor %xmm1, %xmm1
-; SSE2-NEXT:    psubd %xmm0, %xmm1
+; SSE2-NEXT:    pxor %xmm2, %xmm2
+; SSE2-NEXT:    psubd %xmm0, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[1,3,2,3]
+; SSE2-NEXT:    movsd {{.*#+}} xmm0 = xmm1[0],xmm0[1]
 ; SSE2-NEXT:    shufps {{.*#+}} xmm0 = xmm0[0,2,2,3]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,3,2,3]
-; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1]
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: combine_vec_sdiv_by_pow2b_PosAndNeg:
 ; SSE41:       # %bb.0:
 ; SSE41-NEXT:    movdqa %xmm0, %xmm1
 ; SSE41-NEXT:    psrad $31, %xmm1
 ; SSE41-NEXT:    movdqa %xmm1, %xmm2
 ; SSE41-NEXT:    psrld $28, %xmm2
 ; SSE41-NEXT:    movdqa %xmm1, %xmm3
 ; SSE41-NEXT:    psrld $30, %xmm3
 ; SSE41-NEXT:    pblendw {{.*#+}} xmm3 = xmm3[0,1,2,3],xmm2[4,5,6,7]
 ; SSE41-NEXT:    psrld $29, %xmm1
 ; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1],xmm3[2,3],xmm1[4,5],xmm3[6,7]
 ; SSE41-NEXT:    paddd %xmm0, %xmm1
 ; SSE41-NEXT:    movdqa %xmm1, %xmm2
 ; SSE41-NEXT:    psrad $4, %xmm2
 ; SSE41-NEXT:    movdqa %xmm1, %xmm3
 ; SSE41-NEXT:    psrad $2, %xmm3
 ; SSE41-NEXT:    pblendw {{.*#+}} xmm3 = xmm3[0,1,2,3],xmm2[4,5,6,7]
+; SSE41-NEXT:    pxor %xmm2, %xmm2
+; SSE41-NEXT:    psubd %xmm3, %xmm2
 ; SSE41-NEXT:    psrad $3, %xmm1
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1],xmm3[2,3],xmm1[4,5],xmm3[6,7]
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm0[0,1],xmm1[2,3,4,5,6,7]
-; SSE41-NEXT:    pxor %xmm0, %xmm0
-; SSE41-NEXT:    psubd %xmm1, %xmm0
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1],xmm0[2,3],xmm1[4,5],xmm0[6,7]
+; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm0[0,1,2,3],xmm1[4,5,6,7]
+; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1],xmm2[2,3],xmm1[4,5],xmm2[6,7]
 ; SSE41-NEXT:    movdqa %xmm1, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: combine_vec_sdiv_by_pow2b_PosAndNeg:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vpsrad $31, %xmm0, %xmm1
 ; AVX1-NEXT:    vpsrld $28, %xmm1, %xmm2
 ; AVX1-NEXT:    vpsrld $30, %xmm1, %xmm3
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm2 = xmm3[0,1,2,3],xmm2[4,5,6,7]
 ; AVX1-NEXT:    vpsrld $29, %xmm1, %xmm1
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0,1],xmm2[2,3],xmm1[4,5],xmm2[6,7]
 ; AVX1-NEXT:    vpaddd %xmm1, %xmm0, %xmm1
 ; AVX1-NEXT:    vpsrad $4, %xmm1, %xmm2
 ; AVX1-NEXT:    vpsrad $2, %xmm1, %xmm3
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm2 = xmm3[0,1,2,3],xmm2[4,5,6,7]
+; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
+; AVX1-NEXT:    vpsubd %xmm2, %xmm3, %xmm2
 ; AVX1-NEXT:    vpsrad $3, %xmm1, %xmm1
-; AVX1-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0,1],xmm2[2,3],xmm1[4,5],xmm2[6,7]
-; AVX1-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1],xmm1[2,3,4,5,6,7]
-; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vpsubd %xmm0, %xmm1, %xmm1
-; AVX1-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1],xmm1[2,3],xmm0[4,5],xmm1[6,7]
+; AVX1-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1,2,3],xmm1[4,5,6,7]
+; AVX1-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1],xmm2[2,3],xmm0[4,5],xmm2[6,7]
 ; AVX1-NEXT:    retq
 ;
 ; AVX2ORLATER-LABEL: combine_vec_sdiv_by_pow2b_PosAndNeg:
 ; AVX2ORLATER:       # %bb.0:
 ; AVX2ORLATER-NEXT:    vpsrad $31, %xmm0, %xmm1
 ; AVX2ORLATER-NEXT:    vpsrlvd {{.*}}(%rip), %xmm1, %xmm1
 ; AVX2ORLATER-NEXT:    vpaddd %xmm1, %xmm0, %xmm1
 ; AVX2ORLATER-NEXT:    vpsravd {{.*}}(%rip), %xmm1, %xmm1
-; AVX2ORLATER-NEXT:    vpblendd {{.*#+}} xmm0 = xmm0[0],xmm1[1,2,3]
-; AVX2ORLATER-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX2ORLATER-NEXT:    vpsubd %xmm0, %xmm1, %xmm1
-; AVX2ORLATER-NEXT:    vpblendd {{.*#+}} xmm0 = xmm0[0],xmm1[1],xmm0[2],xmm1[3]
+; AVX2ORLATER-NEXT:    vpxor %xmm2, %xmm2, %xmm2
+; AVX2ORLATER-NEXT:    vpsubd %xmm1, %xmm2, %xmm2
+; AVX2ORLATER-NEXT:    vpblendd {{.*#+}} xmm0 = xmm0[0,1],xmm1[2,3]
+; AVX2ORLATER-NEXT:    vpblendd {{.*#+}} xmm0 = xmm0[0],xmm2[1],xmm0[2],xmm2[3]
 ; AVX2ORLATER-NEXT:    retq
 ;
 ; XOP-LABEL: combine_vec_sdiv_by_pow2b_PosAndNeg:
 ; XOP:       # %bb.0:
 ; XOP-NEXT:    vpsrad $31, %xmm0, %xmm1
 ; XOP-NEXT:    vpshld {{.*}}(%rip), %xmm1, %xmm1
 ; XOP-NEXT:    vpaddd %xmm1, %xmm0, %xmm1
 ; XOP-NEXT:    vpshad {{.*}}(%rip), %xmm1, %xmm1
-; XOP-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1],xmm1[2,3,4,5,6,7]
-; XOP-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; XOP-NEXT:    vpsubd %xmm0, %xmm1, %xmm1
-; XOP-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1],xmm1[2,3],xmm0[4,5],xmm1[6,7]
+; XOP-NEXT:    vpxor %xmm2, %xmm2, %xmm2
+; XOP-NEXT:    vpsubd %xmm1, %xmm2, %xmm2
+; XOP-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1,2,3],xmm1[4,5,6,7]
+; XOP-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1],xmm2[2,3],xmm0[4,5],xmm2[6,7]
 ; XOP-NEXT:    retq
   %1 = sdiv <4 x i32> %x, <i32 1, i32 -4, i32 8, i32 -16>
   ret <4 x i32> %1
 }
 
 define <4 x i32> @combine_vec_sdiv_by_pow2b_undef1(<4 x i32> %x) {
 ; CHECK-LABEL: combine_vec_sdiv_by_pow2b_undef1:
 ; CHECK:       # %bb.0:

llvm/test/CodeGen/X86/oddsubvector.ll

 define void @PR42833() {
 ; SSE2-LABEL: PR42833:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm1
 ; SSE2-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm0
 ; SSE2-NEXT:    movd %xmm0, %eax
 ; SSE2-NEXT:    addl .Lb${{.*}}(%rip), %eax
 ; SSE2-NEXT:    movd %eax, %xmm2
-; SSE2-NEXT:    movaps {{.*#+}} xmm3 = <u,1,1,1>
-; SSE2-NEXT:    movss {{.*#+}} xmm3 = xmm2[0],xmm3[1,2,3]
-; SSE2-NEXT:    movdqa %xmm0, %xmm4
-; SSE2-NEXT:    paddd %xmm3, %xmm4
-; SSE2-NEXT:    pslld $23, %xmm3
-; SSE2-NEXT:    paddd {{.*}}(%rip), %xmm3
-; SSE2-NEXT:    cvttps2dq %xmm3, %xmm3
-; SSE2-NEXT:    movdqa %xmm0, %xmm5
-; SSE2-NEXT:    pmuludq %xmm3, %xmm5
-; SSE2-NEXT:    pshufd {{.*#+}} xmm5 = xmm5[0,2,2,3]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm3[1,1,3,3]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm6 = xmm0[1,1,3,3]
-; SSE2-NEXT:    pmuludq %xmm3, %xmm6
-; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm6[0,2,2,3]
-; SSE2-NEXT:    punpckldq {{.*#+}} xmm5 = xmm5[0],xmm3[0],xmm5[1],xmm3[1]
-; SSE2-NEXT:    movss {{.*#+}} xmm5 = xmm4[0],xmm5[1,2,3]
-; SSE2-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm3
-; SSE2-NEXT:    psubd %xmm1, %xmm3
+; SSE2-NEXT:    movd %eax, %xmm3
+; SSE2-NEXT:    paddd %xmm0, %xmm3
+; SSE2-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm4
+; SSE2-NEXT:    psubd %xmm1, %xmm4
 ; SSE2-NEXT:    paddd %xmm1, %xmm1
+; SSE2-NEXT:    movdqa %xmm0, %xmm5
+; SSE2-NEXT:    paddd %xmm0, %xmm5
+; SSE2-NEXT:    movss {{.*#+}} xmm5 = xmm3[0],xmm5[1,2,3]
 ; SSE2-NEXT:    movdqa %xmm1, .Lc$local+{{.*}}(%rip)
 ; SSE2-NEXT:    movaps %xmm5, .Lc$local+{{.*}}(%rip)
 ; SSE2-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm1
-; SSE2-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm4
+; SSE2-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm3
 ; SSE2-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm5
 ; SSE2-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm6
 ; SSE2-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm7
 ; SSE2-NEXT:    movss {{.*#+}} xmm0 = xmm2[0],xmm0[1,2,3]
 ; SSE2-NEXT:    psubd %xmm0, %xmm7
-; SSE2-NEXT:    psubd %xmm4, %xmm6
+; SSE2-NEXT:    psubd %xmm3, %xmm6
 ; SSE2-NEXT:    psubd %xmm1, %xmm5
 ; SSE2-NEXT:    movdqa %xmm5, .Ld$local+{{.*}}(%rip)
 ; SSE2-NEXT:    movdqa %xmm6, .Ld$local+{{.*}}(%rip)
-; SSE2-NEXT:    movdqa %xmm3, .Ld$local+{{.*}}(%rip)
+; SSE2-NEXT:    movdqa %xmm4, .Ld$local+{{.*}}(%rip)
 ; SSE2-NEXT:    movdqa %xmm7, .Ld$local+{{.*}}(%rip)
-; SSE2-NEXT:    paddd %xmm4, %xmm4
+; SSE2-NEXT:    paddd %xmm3, %xmm3
 ; SSE2-NEXT:    paddd %xmm1, %xmm1
 ; SSE2-NEXT:    movdqa %xmm1, .Lc$local+{{.*}}(%rip)
-; SSE2-NEXT:    movdqa %xmm4, .Lc$local+{{.*}}(%rip)
+; SSE2-NEXT:    movdqa %xmm3, .Lc$local+{{.*}}(%rip)
 ; SSE2-NEXT:    retq
 ;
 ; SSE42-LABEL: PR42833:
 ; SSE42:       # %bb.0:
-; SSE42-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm1
 ; SSE42-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm0
-; SSE42-NEXT:    movd %xmm0, %eax
+; SSE42-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm1
+; SSE42-NEXT:    movd %xmm1, %eax
 ; SSE42-NEXT:    addl .Lb${{.*}}(%rip), %eax
-; SSE42-NEXT:    movdqa {{.*#+}} xmm2 = <u,1,1,1>
-; SSE42-NEXT:    pinsrd $0, %eax, %xmm2
-; SSE42-NEXT:    movdqa %xmm0, %xmm3
-; SSE42-NEXT:    paddd %xmm2, %xmm3
-; SSE42-NEXT:    pslld $23, %xmm2
-; SSE42-NEXT:    paddd {{.*}}(%rip), %xmm2
-; SSE42-NEXT:    cvttps2dq %xmm2, %xmm2
-; SSE42-NEXT:    pmulld %xmm0, %xmm2
-; SSE42-NEXT:    pblendw {{.*#+}} xmm2 = xmm3[0,1],xmm2[2,3,4,5,6,7]
+; SSE42-NEXT:    movd %eax, %xmm2
+; SSE42-NEXT:    paddd %xmm1, %xmm2
 ; SSE42-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm3
-; SSE42-NEXT:    psubd %xmm1, %xmm3
-; SSE42-NEXT:    paddd %xmm1, %xmm1
-; SSE42-NEXT:    movdqa %xmm1, .Lc$local+{{.*}}(%rip)
-; SSE42-NEXT:    movdqa %xmm2, .Lc$local+{{.*}}(%rip)
-; SSE42-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm1
+; SSE42-NEXT:    psubd %xmm0, %xmm3
+; SSE42-NEXT:    paddd %xmm0, %xmm0
+; SSE42-NEXT:    movdqa %xmm1, %xmm4
+; SSE42-NEXT:    paddd %xmm1, %xmm4
+; SSE42-NEXT:    pblendw {{.*#+}} xmm4 = xmm2[0,1],xmm4[2,3,4,5,6,7]
+; SSE42-NEXT:    movdqa %xmm0, .Lc$local+{{.*}}(%rip)
+; SSE42-NEXT:    movdqa %xmm4, .Lc$local+{{.*}}(%rip)
+; SSE42-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm0
 ; SSE42-NEXT:    movdqa .Lc$local+{{.*}}(%rip), %xmm2
 ; SSE42-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm4
 ; SSE42-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm5
 ; SSE42-NEXT:    movdqa .Ld$local+{{.*}}(%rip), %xmm6
-; SSE42-NEXT:    pinsrd $0, %eax, %xmm0
-; SSE42-NEXT:    psubd %xmm0, %xmm6
+; SSE42-NEXT:    pinsrd $0, %eax, %xmm1
+; SSE42-NEXT:    psubd %xmm1, %xmm6
 ; SSE42-NEXT:    psubd %xmm2, %xmm5
-; SSE42-NEXT:    psubd %xmm1, %xmm4
+; SSE42-NEXT:    psubd %xmm0, %xmm4
 ; SSE42-NEXT:    movdqa %xmm4, .Ld$local+{{.*}}(%rip)
 ; SSE42-NEXT:    movdqa %xmm5, .Ld$local+{{.*}}(%rip)
 ; SSE42-NEXT:    movdqa %xmm3, .Ld$local+{{.*}}(%rip)
 ; SSE42-NEXT:    movdqa %xmm6, .Ld$local+{{.*}}(%rip)
 ; SSE42-NEXT:    paddd %xmm2, %xmm2
-; SSE42-NEXT:    paddd %xmm1, %xmm1
-; SSE42-NEXT:    movdqa %xmm1, .Lc$local+{{.*}}(%rip)
+; SSE42-NEXT:    paddd %xmm0, %xmm0
+; SSE42-NEXT:    movdqa %xmm0, .Lc$local+{{.*}}(%rip)
 ; SSE42-NEXT:    movdqa %xmm2, .Lc$local+{{.*}}(%rip)
 ; SSE42-NEXT:    retq
 ;
 ; AVX1-LABEL: PR42833:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm0
 ; AVX1-NEXT:    vmovd %xmm0, %eax
 ; AVX1-NEXT:    addl .Lb${{.*}}(%rip), %eax
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm1 = <u,1,1,1>
-; AVX1-NEXT:    vpinsrd $0, %eax, %xmm1, %xmm1
-; AVX1-NEXT:    vpaddd %xmm1, %xmm0, %xmm2
+; AVX1-NEXT:    vmovd %eax, %xmm1
+; AVX1-NEXT:    vpaddd %xmm1, %xmm0, %xmm1
+; AVX1-NEXT:    vpaddd %xmm0, %xmm0, %xmm2
 ; AVX1-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm3
-; AVX1-NEXT:    vpslld $23, %xmm1, %xmm1
-; AVX1-NEXT:    vpaddd {{.*}}(%rip), %xmm1, %xmm1
-; AVX1-NEXT:    vcvttps2dq %xmm1, %xmm1
-; AVX1-NEXT:    vpmulld %xmm1, %xmm0, %xmm1
-; AVX1-NEXT:    vpslld $1, %xmm3, %xmm3
-; AVX1-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
-; AVX1-NEXT:    vblendps {{.*#+}} ymm1 = ymm2[0],ymm1[1,2,3,4,5,6,7]
+; AVX1-NEXT:    vpaddd %xmm3, %xmm3, %xmm3
+; AVX1-NEXT:    vinsertf128 $1, %xmm3, %ymm2, %ymm2
+; AVX1-NEXT:    vblendps {{.*#+}} ymm1 = ymm1[0],ymm2[1,2,3,4,5,6,7]
 ; AVX1-NEXT:    vmovdqa .Ld$local+{{.*}}(%rip), %xmm2
 ; AVX1-NEXT:    vpsubd .Lc$local+{{.*}}(%rip), %xmm2, %xmm2
 ; AVX1-NEXT:    vmovups %ymm1, .Lc$local+{{.*}}(%rip)
 ; AVX1-NEXT:    vpinsrd $0, %eax, %xmm0, %xmm0
 ; AVX1-NEXT:    vmovdqa .Ld$local+{{.*}}(%rip), %xmm1
 ; AVX1-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
 ; AVX1-NEXT:    vmovdqa .Ld$local+{{.*}}(%rip), %xmm1
 ; AVX1-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm3

 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: PR42833:
 ; AVX2:       # %bb.0:
 ; AVX2-NEXT:    movl .Lb${{.*}}(%rip), %eax
 ; AVX2-NEXT:    vmovdqu .Lc$local+{{.*}}(%rip), %ymm0
 ; AVX2-NEXT:    addl .Lc$local+{{.*}}(%rip), %eax
 ; AVX2-NEXT:    vmovd %eax, %xmm1
-; AVX2-NEXT:    vpblendd {{.*#+}} ymm2 = ymm1[0],mem[1,2,3,4,5,6,7]
-; AVX2-NEXT:    vpaddd %ymm2, %ymm0, %ymm3
-; AVX2-NEXT:    vpsllvd %ymm2, %ymm0, %ymm2
-; AVX2-NEXT:    vpblendd {{.*#+}} ymm2 = ymm3[0],ymm2[1,2,3,4,5,6,7]
+; AVX2-NEXT:    vpaddd %ymm1, %ymm0, %ymm2
+; AVX2-NEXT:    vpaddd %ymm0, %ymm0, %ymm3
+; AVX2-NEXT:    vpblendd {{.*#+}} ymm2 = ymm2[0],ymm3[1,2,3,4,5,6,7]
 ; AVX2-NEXT:    vmovdqu %ymm2, .Lc$local+{{.*}}(%rip)
 ; AVX2-NEXT:    vmovdqu .Lc$local+{{.*}}(%rip), %ymm2
 ; AVX2-NEXT:    vmovdqu .Ld$local+{{.*}}(%rip), %ymm3
 ; AVX2-NEXT:    vmovdqu .Ld$local+{{.*}}(%rip), %ymm4
 ; AVX2-NEXT:    vpblendd {{.*#+}} ymm0 = ymm1[0],ymm0[1,2,3,4,5,6,7]
 ; AVX2-NEXT:    vpsubd %ymm0, %ymm4, %ymm0
 ; AVX2-NEXT:    vpsubd %ymm2, %ymm3, %ymm1
 ; AVX2-NEXT:    vmovdqu %ymm1, .Ld$local+{{.*}}(%rip)
 ; AVX2-NEXT:    vmovdqu %ymm0, .Ld$local+{{.*}}(%rip)
 ; AVX2-NEXT:    vpaddd %ymm2, %ymm2, %ymm0
 ; AVX2-NEXT:    vmovdqu %ymm0, .Lc$local+{{.*}}(%rip)
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; AVX512-LABEL: PR42833:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    movl .Lb${{.*}}(%rip), %eax
 ; AVX512-NEXT:    vmovdqu .Lc$local+{{.*}}(%rip), %ymm0
 ; AVX512-NEXT:    vmovdqu64 .Lc$local+{{.*}}(%rip), %zmm1
 ; AVX512-NEXT:    addl .Lc$local+{{.*}}(%rip), %eax
 ; AVX512-NEXT:    vmovd %eax, %xmm2
-; AVX512-NEXT:    vpblendd {{.*#+}} ymm2 = ymm2[0],mem[1,2,3,4,5,6,7]
-; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm3
-; AVX512-NEXT:    vpsllvd %ymm2, %ymm0, %ymm0
-; AVX512-NEXT:    vpblendd {{.*#+}} ymm0 = ymm3[0],ymm0[1,2,3,4,5,6,7]
+; AVX512-NEXT:    vpaddd %ymm2, %ymm0, %ymm2
+; AVX512-NEXT:    vpaddd %ymm0, %ymm0, %ymm0
+; AVX512-NEXT:    vpblendd {{.*#+}} ymm0 = ymm2[0],ymm0[1,2,3,4,5,6,7]
 ; AVX512-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm2
 ; AVX512-NEXT:    vmovdqu %ymm0, .Lc$local+{{.*}}(%rip)
 ; AVX512-NEXT:    vmovdqu .Lc$local+{{.*}}(%rip), %ymm0
 ; AVX512-NEXT:    vmovdqu64 .Ld$local+{{.*}}(%rip), %zmm3
 ; AVX512-NEXT:    vpinsrd $0, %eax, %xmm2, %xmm2
 ; AVX512-NEXT:    vinserti32x4 $0, %xmm2, %zmm1, %zmm1
 ; AVX512-NEXT:    vinserti64x4 $1, %ymm0, %zmm1, %zmm1
 ; AVX512-NEXT:    vpsubd %zmm1, %zmm3, %zmm1
 ; AVX512-NEXT:    vmovdqu64 %zmm1, .Ld$local+{{.*}}(%rip)
 ; AVX512-NEXT:    vpaddd %ymm0, %ymm0, %ymm0
 ; AVX512-NEXT:    vmovdqu %ymm0, .Lc$local+{{.*}}(%rip)
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
 ;
 ; XOP-LABEL: PR42833:
 ; XOP:       # %bb.0:
 ; XOP-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm0
 ; XOP-NEXT:    vmovd %xmm0, %eax
 ; XOP-NEXT:    addl .Lb${{.*}}(%rip), %eax
-; XOP-NEXT:    vmovdqa {{.*#+}} xmm1 = <u,1,1,1>
-; XOP-NEXT:    vpinsrd $0, %eax, %xmm1, %xmm1
-; XOP-NEXT:    vpaddd %xmm1, %xmm0, %xmm2
+; XOP-NEXT:    vmovd %eax, %xmm1
+; XOP-NEXT:    vpaddd %xmm1, %xmm0, %xmm1
+; XOP-NEXT:    vpaddd %xmm0, %xmm0, %xmm2
 ; XOP-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm3
-; XOP-NEXT:    vpshld %xmm1, %xmm0, %xmm1
-; XOP-NEXT:    vpslld $1, %xmm3, %xmm3
-; XOP-NEXT:    vinsertf128 $1, %xmm3, %ymm1, %ymm1
-; XOP-NEXT:    vblendps {{.*#+}} ymm1 = ymm2[0],ymm1[1,2,3,4,5,6,7]
+; XOP-NEXT:    vpaddd %xmm3, %xmm3, %xmm3
+; XOP-NEXT:    vinsertf128 $1, %xmm3, %ymm2, %ymm2
+; XOP-NEXT:    vblendps {{.*#+}} ymm1 = ymm1[0],ymm2[1,2,3,4,5,6,7]
 ; XOP-NEXT:    vmovdqa .Ld$local+{{.*}}(%rip), %xmm2
 ; XOP-NEXT:    vpsubd .Lc$local+{{.*}}(%rip), %xmm2, %xmm2
 ; XOP-NEXT:    vmovups %ymm1, .Lc$local+{{.*}}(%rip)
 ; XOP-NEXT:    vpinsrd $0, %eax, %xmm0, %xmm0
 ; XOP-NEXT:    vmovdqa .Ld$local+{{.*}}(%rip), %xmm1
 ; XOP-NEXT:    vpsubd %xmm0, %xmm1, %xmm0
 ; XOP-NEXT:    vmovdqa .Ld$local+{{.*}}(%rip), %xmm1
 ; XOP-NEXT:    vmovdqa .Lc$local+{{.*}}(%rip), %xmm3

llvm/test/CodeGen/X86/vector-fshl-rot-128.ll

 
 ;
 ; Uniform Variable Shifts
 ;
 
 define <2 x i64> @splatvar_funnnel_v2i64(<2 x i64> %x, <2 x i64> %amt) nounwind {
 ; SSE-LABEL: splatvar_funnnel_v2i64:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,1,0,1]
 ; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [63,63]
 ; SSE-NEXT:    pxor %xmm3, %xmm3
 ; SSE-NEXT:    psubq %xmm1, %xmm3
 ; SSE-NEXT:    pand %xmm2, %xmm1
 ; SSE-NEXT:    movdqa %xmm0, %xmm4
 ; SSE-NEXT:    psllq %xmm1, %xmm4
 ; SSE-NEXT:    pand %xmm2, %xmm3
 ; SSE-NEXT:    psrlq %xmm3, %xmm0
 ; SSE-NEXT:    por %xmm4, %xmm0
 ; SSE-NEXT:    retq
 ;
-; AVX1-LABEL: splatvar_funnnel_v2i64:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[0,1,0,1]
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
-; AVX1-NEXT:    vpand %xmm2, %xmm1, %xmm3
-; AVX1-NEXT:    vpsllq %xmm3, %xmm0, %xmm3
-; AVX1-NEXT:    vpxor %xmm4, %xmm4, %xmm4
-; AVX1-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
-; AVX1-NEXT:    vpand %xmm2, %xmm1, %xmm1
-; AVX1-NEXT:    vpsrlq %xmm1, %xmm0, %xmm0
-; AVX1-NEXT:    vpor %xmm0, %xmm3, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: splatvar_funnnel_v2i64:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpbroadcastq %xmm1, %xmm1
-; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
-; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm3
-; AVX2-NEXT:    vpsllq %xmm3, %xmm0, %xmm3
-; AVX2-NEXT:    vpxor %xmm4, %xmm4, %xmm4
-; AVX2-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
-; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm1
-; AVX2-NEXT:    vpsrlq %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpor %xmm0, %xmm3, %xmm0
-; AVX2-NEXT:    retq
+; AVX-LABEL: splatvar_funnnel_v2i64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
+; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm3
+; AVX-NEXT:    vpsllq %xmm3, %xmm0, %xmm3
+; AVX-NEXT:    vpxor %xmm4, %xmm4, %xmm4
+; AVX-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
+; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX-NEXT:    vpsrlq %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpor %xmm0, %xmm3, %xmm0
+; AVX-NEXT:    retq
 ;
 ; AVX512F-LABEL: splatvar_funnnel_v2i64:
 ; AVX512F:       # %bb.0:
 ; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; AVX512F-NEXT:    vpbroadcastq %xmm1, %xmm1
 ; AVX512F-NEXT:    vprolvq %zmm1, %zmm0, %zmm0
 ; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 killed $zmm0
 ; AVX512F-NEXT:    vzeroupper

llvm/test/CodeGen/X86/vector-fshl-rot-256.ll

 
 ;
 ; Uniform Variable Shifts
 ;
 
 define <4 x i64> @splatvar_funnnel_v4i64(<4 x i64> %x, <4 x i64> %amt) nounwind {
 ; AVX1-LABEL: splatvar_funnnel_v4i64:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[0,1,0,1]
-; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX1-NEXT:    vpsubq %xmm1, %xmm2, %xmm2
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[0,1,0,1]
+; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
+; AVX1-NEXT:    vpsubq %xmm2, %xmm3, %xmm2
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = [63,63]
 ; AVX1-NEXT:    vpand %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vpsrlq %xmm2, %xmm4, %xmm5
 ; AVX1-NEXT:    vpshufd {{.*#+}} xmm6 = xmm2[2,3,0,1]
 ; AVX1-NEXT:    vpsrlq %xmm6, %xmm4, %xmm7
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm5 = xmm5[0,1,2,3],xmm7[4,5,6,7]
 ; AVX1-NEXT:    vpsrlq %xmm2, %xmm0, %xmm2
 ; AVX1-NEXT:    vpsrlq %xmm6, %xmm0, %xmm6
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm2 = xmm2[0,1,2,3],xmm6[4,5,6,7]
 ; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm2, %ymm2
 ; AVX1-NEXT:    vpand %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpsllq %xmm1, %xmm4, %xmm3
 ; AVX1-NEXT:    vpsllq %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm3, %ymm0, %ymm0
 ; AVX1-NEXT:    vorps %ymm2, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: splatvar_funnnel_v4i64:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpbroadcastq %xmm1, %xmm1
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
 ; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm3
 ; AVX2-NEXT:    vpsllq %xmm3, %ymm0, %ymm3
 ; AVX2-NEXT:    vpxor %xmm4, %xmm4, %xmm4
 ; AVX2-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
 ; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vpsrlq %xmm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vpor %ymm0, %ymm3, %ymm0

llvm/test/CodeGen/X86/vector-fshr-rot-128.ll

 
 ;
 ; Uniform Variable Shifts
 ;
 
 define <2 x i64> @splatvar_funnnel_v2i64(<2 x i64> %x, <2 x i64> %amt) nounwind {
 ; SSE-LABEL: splatvar_funnnel_v2i64:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,1,0,1]
 ; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [63,63]
 ; SSE-NEXT:    pxor %xmm3, %xmm3
 ; SSE-NEXT:    psubq %xmm1, %xmm3
 ; SSE-NEXT:    pand %xmm2, %xmm1
 ; SSE-NEXT:    movdqa %xmm0, %xmm4
 ; SSE-NEXT:    psrlq %xmm1, %xmm4
 ; SSE-NEXT:    pand %xmm2, %xmm3
 ; SSE-NEXT:    psllq %xmm3, %xmm0
 ; SSE-NEXT:    por %xmm4, %xmm0
 ; SSE-NEXT:    retq
 ;
-; AVX1-LABEL: splatvar_funnnel_v2i64:
-; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[0,1,0,1]
-; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
-; AVX1-NEXT:    vpand %xmm2, %xmm1, %xmm3
-; AVX1-NEXT:    vpsrlq %xmm3, %xmm0, %xmm3
-; AVX1-NEXT:    vpxor %xmm4, %xmm4, %xmm4
-; AVX1-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
-; AVX1-NEXT:    vpand %xmm2, %xmm1, %xmm1
-; AVX1-NEXT:    vpsllq %xmm1, %xmm0, %xmm0
-; AVX1-NEXT:    vpor %xmm3, %xmm0, %xmm0
-; AVX1-NEXT:    retq
-;
-; AVX2-LABEL: splatvar_funnnel_v2i64:
-; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpbroadcastq %xmm1, %xmm1
-; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
-; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm3
-; AVX2-NEXT:    vpsrlq %xmm3, %xmm0, %xmm3
-; AVX2-NEXT:    vpxor %xmm4, %xmm4, %xmm4
-; AVX2-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
-; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm1
-; AVX2-NEXT:    vpsllq %xmm1, %xmm0, %xmm0
-; AVX2-NEXT:    vpor %xmm3, %xmm0, %xmm0
-; AVX2-NEXT:    retq
+; AVX-LABEL: splatvar_funnnel_v2i64:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
+; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm3
+; AVX-NEXT:    vpsrlq %xmm3, %xmm0, %xmm3
+; AVX-NEXT:    vpxor %xmm4, %xmm4, %xmm4
+; AVX-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
+; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX-NEXT:    vpsllq %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpor %xmm3, %xmm0, %xmm0
+; AVX-NEXT:    retq
 ;
 ; AVX512F-LABEL: splatvar_funnnel_v2i64:
 ; AVX512F:       # %bb.0:
 ; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 def $zmm0
 ; AVX512F-NEXT:    vpbroadcastq %xmm1, %xmm1
 ; AVX512F-NEXT:    vprorvq %zmm1, %zmm0, %zmm0
 ; AVX512F-NEXT:    # kill: def $xmm0 killed $xmm0 killed $zmm0
 ; AVX512F-NEXT:    vzeroupper

llvm/test/CodeGen/X86/vector-fshr-rot-256.ll

 
 ;
 ; Uniform Variable Shifts
 ;
 
 define <4 x i64> @splatvar_funnnel_v4i64(<4 x i64> %x, <4 x i64> %amt) nounwind {
 ; AVX1-LABEL: splatvar_funnnel_v4i64:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpshufd {{.*#+}} xmm1 = xmm1[0,1,0,1]
-; AVX1-NEXT:    vpxor %xmm2, %xmm2, %xmm2
-; AVX1-NEXT:    vpsubq %xmm1, %xmm2, %xmm2
+; AVX1-NEXT:    vpshufd {{.*#+}} xmm2 = xmm1[0,1,0,1]
+; AVX1-NEXT:    vpxor %xmm3, %xmm3, %xmm3
+; AVX1-NEXT:    vpsubq %xmm2, %xmm3, %xmm2
 ; AVX1-NEXT:    vmovdqa {{.*#+}} xmm3 = [63,63]
 ; AVX1-NEXT:    vpand %xmm3, %xmm2, %xmm2
 ; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm4
 ; AVX1-NEXT:    vpsllq %xmm2, %xmm4, %xmm5
 ; AVX1-NEXT:    vpshufd {{.*#+}} xmm6 = xmm2[2,3,0,1]
 ; AVX1-NEXT:    vpsllq %xmm6, %xmm4, %xmm7
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm5 = xmm5[0,1,2,3],xmm7[4,5,6,7]
 ; AVX1-NEXT:    vpsllq %xmm2, %xmm0, %xmm2
 ; AVX1-NEXT:    vpsllq %xmm6, %xmm0, %xmm6
 ; AVX1-NEXT:    vpblendw {{.*#+}} xmm2 = xmm2[0,1,2,3],xmm6[4,5,6,7]
 ; AVX1-NEXT:    vinsertf128 $1, %xmm5, %ymm2, %ymm2
 ; AVX1-NEXT:    vpand %xmm3, %xmm1, %xmm1
 ; AVX1-NEXT:    vpsrlq %xmm1, %xmm4, %xmm3
 ; AVX1-NEXT:    vpsrlq %xmm1, %xmm0, %xmm0
 ; AVX1-NEXT:    vinsertf128 $1, %xmm3, %ymm0, %ymm0
 ; AVX1-NEXT:    vorps %ymm0, %ymm2, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: splatvar_funnnel_v4i64:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpbroadcastq %xmm1, %xmm1
 ; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [63,63]
 ; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm3
 ; AVX2-NEXT:    vpsrlq %xmm3, %ymm0, %ymm3
 ; AVX2-NEXT:    vpxor %xmm4, %xmm4, %xmm4
 ; AVX2-NEXT:    vpsubq %xmm1, %xmm4, %xmm1
 ; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm1
 ; AVX2-NEXT:    vpsllq %xmm1, %ymm0, %ymm0
 ; AVX2-NEXT:    vpor %ymm3, %ymm0, %ymm0

llvm/test/CodeGen/X86/vector-narrow-binop.ll

; AVX-NEXT:    retq
   ret <2 x i8> %shuffle
 }
 
 define <4 x double> @fmul_v2f64(<2 x  double> %x, <2 x double> %y) {
 ; SSE-LABEL: fmul_v2f64:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    movapd %xmm1, %xmm2
 ; SSE-NEXT:    unpcklpd {{.*#+}} xmm2 = xmm2[0],xmm0[0]
-; SSE-NEXT:    unpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
-; SSE-NEXT:    mulpd %xmm0, %xmm0
 ; SSE-NEXT:    mulpd %xmm2, %xmm2
-; SSE-NEXT:    addpd %xmm0, %xmm2
-; SSE-NEXT:    unpckhpd {{.*#+}} xmm2 = xmm2[1,1]
-; SSE-NEXT:    movapd %xmm2, %xmm0
+; SSE-NEXT:    mulpd %xmm1, %xmm1
+; SSE-NEXT:    addpd %xmm2, %xmm1
+; SSE-NEXT:    unpckhpd {{.*#+}} xmm1 = xmm1[1,1]
+; SSE-NEXT:    movapd %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX1-LABEL: fmul_v2f64:
 ; AVX1:       # %bb.0:
 ; AVX1-NEXT:    vunpcklpd {{.*#+}} xmm2 = xmm1[0],xmm0[0]
 ; AVX1-NEXT:    vunpckhpd {{.*#+}} xmm0 = xmm0[1],xmm1[1]
 ; AVX1-NEXT:    vmulpd %xmm0, %xmm0, %xmm0
 ; AVX1-NEXT:    vmulpd %xmm2, %xmm2, %xmm1