[X86] decomposeMulByConstant - decompose legal vXi32 multiplies on SlowPMULLD targets and all vXi64 multiplies

[RKSimon]

Summary

X86's decomposeMulByConstant never permits mul decomposition to shift+add/sub if the vector multiply is legal.

Unfortunately this isn't great for SSE41+ targets which have PMULLD for vXi32 multiplies, but is often quite slow. This patch proposes to allow decomposition if the target has the SlowPMULLD flag (i.e. Silvermont). We also always decompose legal vXi64 multiplies - even latest IceLake has really poor latencies for PMULLQ.

[RKSimon]

ping - any thoughts? should we do not do this at all? limit this to SlowPMULLD or always do it for vXi32 and vXi64?

[craig.topper]

In D110588#3036506, @RKSimon wrote:

ping - any thoughts? should we do not do this at all? limit this to SlowPMULLD or always do it for vXi32 and vXi64?

I think it makes sense for SlowPMULLD. I think this makes sense for vXi32 on Haswell and later Intel Core CPUs. That's where it went from one uop to two serialize 5 cycle uops. I think it makes sense for vXi64 on all Intel CPUs with PMULLQ since that instruction is 3 serialized uops.

[RKSimon]

Always decompose vXi64 multiplies

[craig.topper]

LGTM

[RKSimon]

Thanks Craig

[xbolva00]

Can this patch solve first part of PR52039?

https://bugs.llvm.org/show_bug.cgi?id=52039

[RKSimon]

In D110588#3038019, @xbolva00 wrote:

Can this patch solve first part of PR52039?

https://bugs.llvm.org/show_bug.cgi?id=52039

I think this will only help that on Silvermont :)

We might have to redefine SlowPMULLD slightly so we can enable it on Haswell/Broadwell (and maybe Jaguar) (see PR35948). As well as this patch its used in reduceVMULWidth for PMULLD->PMULLW+PMULHW+SHUFFLE expansion of small values which is good for SLM (which has REALLY bad microcoded PMULLD) but not usually for anything else where its just a 2/3uop instruction.

I've been trying to improve PMADDWD combines as an alternative to reduceVMULWidth but there's a still some work to do (PR47437 and PR45897).

llvm/lib/Target/X86/X86ISelLowering.cpp

bool X86TargetLowering::decomposeMulByConstant(LLVMContext &Context, EVT VT,
   // Another choice would be to defer the decision for illegal types until
   // after type legalization. But constant splat vectors of i64 can't make it
   // through type legalization on 32-bit targets so we would need to special
   // case vXi64.
   while (getTypeAction(Context, VT) != TypeLegal)
     VT = getTypeToTransformTo(Context, VT);
 
   // If vector multiply is legal, assume that's faster than shl + add/sub.
-  // TODO: Multiply is a complex op with higher latency and lower throughput in
-  //       most implementations, so this check could be loosened based on type
-  //       and/or a CPU attribute.
-  if (isOperationLegal(ISD::MUL, VT))
+  // Multiply is a complex op with higher latency and lower throughput in
+  // most implementations, sub-vXi32 vector multiplies are always fast,
+  // vXi32 mustn't have a SlowMULLD implementation, and anything larger (vXi64)
+  // is always going to be slow.
+  unsigned EltSizeInBits = VT.getScalarSizeInBits();
+  if (isOperationLegal(ISD::MUL, VT) && EltSizeInBits <= 32 &&
+      (EltSizeInBits != 32 || !Subtarget.isPMULLDSlow()))
     return false;
 
   // shl+add, shl+sub, shl+add+neg
   return (MulC + 1).isPowerOf2() || (MulC - 1).isPowerOf2() ||
          (1 - MulC).isPowerOf2() || (-(MulC + 1)).isPowerOf2();
 }
 
 bool X86TargetLowering::isExtractSubvectorCheap(EVT ResVT, EVT SrcVT,

llvm/test/CodeGen/X86/vector-mul.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse2   | FileCheck %s --check-prefixes=SSE,SSE2,X86-SSE,X86-SSE2
 ; RUN: llc < %s -mtriple=i686-unknown-unknown -mattr=+sse4.2 | FileCheck %s --check-prefixes=SSE,SSE4,X86-SSE,X86-SSE4
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefixes=SSE,SSE2,X64-SSE,X64-SSE2
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2,-slow-pmulld | FileCheck %s --check-prefixes=SSE,SSE4,X64-SSE,X64-SSE4
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2,+slow-pmulld | FileCheck %s --check-prefixes=SSE,SSE4,X64-SSE,X64-SSE4
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2,-slow-pmulld | FileCheck %s --check-prefixes=SSE,SSE4,X64-SSE,X64-SSE4,X64-SSE4-FAST
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2,+slow-pmulld | FileCheck %s --check-prefixes=SSE,SSE4,X64-SSE,X64-SSE4,X64-SSE4-SLOW
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+xop  | FileCheck %s --check-prefixes=X64-AVX,X64-XOP
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=X64-AVX,X64-AVX2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512dq,+avx512vl | FileCheck %s --check-prefixes=X64-AVX,X64-AVX512DQ
 
 ;
 ; PowOf2 (uniform)
 ;
 

 ; SSE-LABEL: mul_v2i64_17:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    movdqa %xmm0, %xmm1
 ; SSE-NEXT:    psllq $4, %xmm1
 ; SSE-NEXT:    paddq %xmm0, %xmm1
 ; SSE-NEXT:    movdqa %xmm1, %xmm0
 ; SSE-NEXT:    ret{{[l|q]}}
 ;
-; X64-XOP-LABEL: mul_v2i64_17:
-; X64-XOP:       # %bb.0:
-; X64-XOP-NEXT:    vpsllq $4, %xmm0, %xmm1
-; X64-XOP-NEXT:    vpaddq %xmm0, %xmm1, %xmm0
-; X64-XOP-NEXT:    retq
-;
-; X64-AVX2-LABEL: mul_v2i64_17:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vpsllq $4, %xmm0, %xmm1
-; X64-AVX2-NEXT:    vpaddq %xmm0, %xmm1, %xmm0
-; X64-AVX2-NEXT:    retq
-;
-; X64-AVX512DQ-LABEL: mul_v2i64_17:
-; X64-AVX512DQ:       # %bb.0:
-; X64-AVX512DQ-NEXT:    vpmullq {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; X64-AVX512DQ-NEXT:    retq
+; X64-AVX-LABEL: mul_v2i64_17:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vpsllq $4, %xmm0, %xmm1
+; X64-AVX-NEXT:    vpaddq %xmm0, %xmm1, %xmm0
+; X64-AVX-NEXT:    retq
   %1 = mul <2 x i64> %a0, <i64 17, i64 17>
   ret <2 x i64> %1
 }
 
 define <4 x i32> @mul_v4i32_17(<4 x i32> %a0) nounwind {
 ; SSE2-LABEL: mul_v4i32_17:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa %xmm0, %xmm1
 ; SSE2-NEXT:    pslld $4, %xmm1
 ; SSE2-NEXT:    paddd %xmm0, %xmm1
 ; SSE2-NEXT:    movdqa %xmm1, %xmm0
 ; SSE2-NEXT:    ret{{[l|q]}}
 ;
 ; X86-SSE4-LABEL: mul_v4i32_17:
 ; X86-SSE4:       # %bb.0:
 ; X86-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}, %xmm0
 ; X86-SSE4-NEXT:    retl
 ;
-; X64-SSE4-LABEL: mul_v4i32_17:
-; X64-SSE4:       # %bb.0:
-; X64-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; X64-SSE4-NEXT:    retq
+; X64-SSE4-FAST-LABEL: mul_v4i32_17:
+; X64-SSE4-FAST:       # %bb.0:
+; X64-SSE4-FAST-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; X64-SSE4-FAST-NEXT:    retq
+;
+; X64-SSE4-SLOW-LABEL: mul_v4i32_17:
+; X64-SSE4-SLOW:       # %bb.0:
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    pslld $4, %xmm1
+; X64-SSE4-SLOW-NEXT:    paddd %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm1, %xmm0
+; X64-SSE4-SLOW-NEXT:    retq
 ;
 ; X64-XOP-LABEL: mul_v4i32_17:
 ; X64-XOP:       # %bb.0:
 ; X64-XOP-NEXT:    vpmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
 ; X64-XOP-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: mul_v4i32_17:
 ; X64-AVX2:       # %bb.0:

 ; X64-AVX2-LABEL: mul_v4i64_17:
 ; X64-AVX2:       # %bb.0:
 ; X64-AVX2-NEXT:    vpsllq $4, %ymm0, %ymm1
 ; X64-AVX2-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; X64-AVX2-NEXT:    retq
 ;
 ; X64-AVX512DQ-LABEL: mul_v4i64_17:
 ; X64-AVX512DQ:       # %bb.0:
-; X64-AVX512DQ-NEXT:    vpmullq {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to4}, %ymm0, %ymm0
+; X64-AVX512DQ-NEXT:    vpsllq $4, %ymm0, %ymm1
+; X64-AVX512DQ-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; X64-AVX512DQ-NEXT:    retq
   %1 = mul <4 x i64> %a0, <i64 17, i64 17, i64 17, i64 17>
   ret <4 x i64> %1
 }
 
 define <8 x i32> @mul_v8i32_17(<8 x i32> %a0) nounwind {
 ; SSE2-LABEL: mul_v8i32_17:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa %xmm0, %xmm2
 ; SSE2-NEXT:    pslld $4, %xmm2
 ; SSE2-NEXT:    paddd %xmm0, %xmm2
 ; SSE2-NEXT:    movdqa %xmm1, %xmm3
 ; SSE2-NEXT:    pslld $4, %xmm3
 ; SSE2-NEXT:    paddd %xmm1, %xmm3
 ; SSE2-NEXT:    movdqa %xmm2, %xmm0
 ; SSE2-NEXT:    movdqa %xmm3, %xmm1
 ; SSE2-NEXT:    ret{{[l|q]}}
 ;
-; SSE4-LABEL: mul_v8i32_17:
-; SSE4:       # %bb.0:
-; SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [17,17,17,17]
-; SSE4-NEXT:    pmulld %xmm2, %xmm0
-; SSE4-NEXT:    pmulld %xmm2, %xmm1
-; SSE4-NEXT:    ret{{[l|q]}}
+; X86-SSE4-LABEL: mul_v8i32_17:
+; X86-SSE4:       # %bb.0:
+; X86-SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [17,17,17,17]
+; X86-SSE4-NEXT:    pmulld %xmm2, %xmm0
+; X86-SSE4-NEXT:    pmulld %xmm2, %xmm1
+; X86-SSE4-NEXT:    retl
+;
+; X64-SSE4-FAST-LABEL: mul_v8i32_17:
+; X64-SSE4-FAST:       # %bb.0:
+; X64-SSE4-FAST-NEXT:    movdqa {{.*#+}} xmm2 = [17,17,17,17]
+; X64-SSE4-FAST-NEXT:    pmulld %xmm2, %xmm0
+; X64-SSE4-FAST-NEXT:    pmulld %xmm2, %xmm1
+; X64-SSE4-FAST-NEXT:    retq
+;
+; X64-SSE4-SLOW-LABEL: mul_v8i32_17:
+; X64-SSE4-SLOW:       # %bb.0:
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm0, %xmm2
+; X64-SSE4-SLOW-NEXT:    pslld $4, %xmm2
+; X64-SSE4-SLOW-NEXT:    paddd %xmm0, %xmm2
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm1, %xmm3
+; X64-SSE4-SLOW-NEXT:    pslld $4, %xmm3
+; X64-SSE4-SLOW-NEXT:    paddd %xmm1, %xmm3
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm2, %xmm0
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm3, %xmm1
+; X64-SSE4-SLOW-NEXT:    retq
 ;
 ; X64-XOP-LABEL: mul_v8i32_17:
 ; X64-XOP:       # %bb.0:
 ; X64-XOP-NEXT:    vextractf128 $1, %ymm0, %xmm1
 ; X64-XOP-NEXT:    vpslld $4, %xmm1, %xmm2
 ; X64-XOP-NEXT:    vpaddd %xmm1, %xmm2, %xmm1
 ; X64-XOP-NEXT:    vpslld $4, %xmm0, %xmm2
 ; X64-XOP-NEXT:    vpaddd %xmm0, %xmm2, %xmm0

 ; SSE:       # %bb.0:
 ; SSE-NEXT:    movdqa %xmm0, %xmm1
 ; SSE-NEXT:    psllq $10, %xmm1
 ; SSE-NEXT:    paddq %xmm0, %xmm1
 ; SSE-NEXT:    pxor %xmm0, %xmm0
 ; SSE-NEXT:    psubq %xmm1, %xmm0
 ; SSE-NEXT:    ret{{[l|q]}}
 ;
-; X64-XOP-LABEL: mul_v2i64_neg1025:
-; X64-XOP:       # %bb.0:
-; X64-XOP-NEXT:    vpsllq $10, %xmm0, %xmm1
-; X64-XOP-NEXT:    vpaddq %xmm0, %xmm1, %xmm0
-; X64-XOP-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X64-XOP-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
-; X64-XOP-NEXT:    retq
-;
-; X64-AVX2-LABEL: mul_v2i64_neg1025:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vpsllq $10, %xmm0, %xmm1
-; X64-AVX2-NEXT:    vpaddq %xmm0, %xmm1, %xmm0
-; X64-AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; X64-AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
-; X64-AVX2-NEXT:    retq
-;
-; X64-AVX512DQ-LABEL: mul_v2i64_neg1025:
-; X64-AVX512DQ:       # %bb.0:
-; X64-AVX512DQ-NEXT:    vpmullq {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; X64-AVX512DQ-NEXT:    retq
+; X64-AVX-LABEL: mul_v2i64_neg1025:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vpsllq $10, %xmm0, %xmm1
+; X64-AVX-NEXT:    vpaddq %xmm0, %xmm1, %xmm0
+; X64-AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; X64-AVX-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
+; X64-AVX-NEXT:    retq
   %1 = mul <2 x i64> %a0, <i64 -1025, i64 -1025>
   ret <2 x i64> %1
 }
 
 define <4 x i32> @mul_v4i32_neg33(<4 x i32> %a0) nounwind {
 ; SSE2-LABEL: mul_v4i32_neg33:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa %xmm0, %xmm1
 ; SSE2-NEXT:    pslld $5, %xmm1
 ; SSE2-NEXT:    paddd %xmm0, %xmm1
 ; SSE2-NEXT:    pxor %xmm0, %xmm0
 ; SSE2-NEXT:    psubd %xmm1, %xmm0
 ; SSE2-NEXT:    ret{{[l|q]}}
 ;
 ; X86-SSE4-LABEL: mul_v4i32_neg33:
 ; X86-SSE4:       # %bb.0:
 ; X86-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}, %xmm0
 ; X86-SSE4-NEXT:    retl
 ;
-; X64-SSE4-LABEL: mul_v4i32_neg33:
-; X64-SSE4:       # %bb.0:
-; X64-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; X64-SSE4-NEXT:    retq
+; X64-SSE4-FAST-LABEL: mul_v4i32_neg33:
+; X64-SSE4-FAST:       # %bb.0:
+; X64-SSE4-FAST-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; X64-SSE4-FAST-NEXT:    retq
+;
+; X64-SSE4-SLOW-LABEL: mul_v4i32_neg33:
+; X64-SSE4-SLOW:       # %bb.0:
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    pslld $5, %xmm1
+; X64-SSE4-SLOW-NEXT:    paddd %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    pxor %xmm0, %xmm0
+; X64-SSE4-SLOW-NEXT:    psubd %xmm1, %xmm0
+; X64-SSE4-SLOW-NEXT:    retq
 ;
 ; X64-XOP-LABEL: mul_v4i32_neg33:
 ; X64-XOP:       # %bb.0:
 ; X64-XOP-NEXT:    vpmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
 ; X64-XOP-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: mul_v4i32_neg33:
 ; X64-AVX2:       # %bb.0:

 ; X64-AVX2-NEXT:    vpsllq $10, %ymm0, %ymm1
 ; X64-AVX2-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
 ; X64-AVX2-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; X64-AVX2-NEXT:    vpsubq %ymm0, %ymm1, %ymm0
 ; X64-AVX2-NEXT:    retq
 ;
 ; X64-AVX512DQ-LABEL: mul_v4i64_neg1025:
 ; X64-AVX512DQ:       # %bb.0:
-; X64-AVX512DQ-NEXT:    vpmullq {{\.?LCPI[0-9]+_[0-9]+}}(%rip){1to4}, %ymm0, %ymm0
+; X64-AVX512DQ-NEXT:    vpsllq $10, %ymm0, %ymm1
+; X64-AVX512DQ-NEXT:    vpaddq %ymm0, %ymm1, %ymm0
+; X64-AVX512DQ-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; X64-AVX512DQ-NEXT:    vpsubq %ymm0, %ymm1, %ymm0
 ; X64-AVX512DQ-NEXT:    retq
   %1 = mul <4 x i64> %a0, <i64 -1025, i64 -1025, i64 -1025, i64 -1025>
   ret <4 x i64> %1
 }
 
 define <8 x i32> @mul_v8i32_neg33(<8 x i32> %a0) nounwind {
 ; SSE2-LABEL: mul_v8i32_neg33:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa %xmm0, %xmm3
 ; SSE2-NEXT:    pslld $5, %xmm3
 ; SSE2-NEXT:    paddd %xmm0, %xmm3
 ; SSE2-NEXT:    pxor %xmm2, %xmm2
 ; SSE2-NEXT:    pxor %xmm0, %xmm0
 ; SSE2-NEXT:    psubd %xmm3, %xmm0
 ; SSE2-NEXT:    movdqa %xmm1, %xmm3
 ; SSE2-NEXT:    pslld $5, %xmm3
 ; SSE2-NEXT:    paddd %xmm1, %xmm3
 ; SSE2-NEXT:    psubd %xmm3, %xmm2
 ; SSE2-NEXT:    movdqa %xmm2, %xmm1
 ; SSE2-NEXT:    ret{{[l|q]}}
 ;
-; SSE4-LABEL: mul_v8i32_neg33:
-; SSE4:       # %bb.0:
-; SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [4294967263,4294967263,4294967263,4294967263]
-; SSE4-NEXT:    pmulld %xmm2, %xmm0
-; SSE4-NEXT:    pmulld %xmm2, %xmm1
-; SSE4-NEXT:    ret{{[l|q]}}
+; X86-SSE4-LABEL: mul_v8i32_neg33:
+; X86-SSE4:       # %bb.0:
+; X86-SSE4-NEXT:    movdqa {{.*#+}} xmm2 = [4294967263,4294967263,4294967263,4294967263]
+; X86-SSE4-NEXT:    pmulld %xmm2, %xmm0
+; X86-SSE4-NEXT:    pmulld %xmm2, %xmm1
+; X86-SSE4-NEXT:    retl
+;
+; X64-SSE4-FAST-LABEL: mul_v8i32_neg33:
+; X64-SSE4-FAST:       # %bb.0:
+; X64-SSE4-FAST-NEXT:    movdqa {{.*#+}} xmm2 = [4294967263,4294967263,4294967263,4294967263]
+; X64-SSE4-FAST-NEXT:    pmulld %xmm2, %xmm0
+; X64-SSE4-FAST-NEXT:    pmulld %xmm2, %xmm1
+; X64-SSE4-FAST-NEXT:    retq
+;
+; X64-SSE4-SLOW-LABEL: mul_v8i32_neg33:
+; X64-SSE4-SLOW:       # %bb.0:
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm0, %xmm3
+; X64-SSE4-SLOW-NEXT:    pslld $5, %xmm3
+; X64-SSE4-SLOW-NEXT:    paddd %xmm0, %xmm3
+; X64-SSE4-SLOW-NEXT:    pxor %xmm2, %xmm2
+; X64-SSE4-SLOW-NEXT:    pxor %xmm0, %xmm0
+; X64-SSE4-SLOW-NEXT:    psubd %xmm3, %xmm0
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm1, %xmm3
+; X64-SSE4-SLOW-NEXT:    pslld $5, %xmm3
+; X64-SSE4-SLOW-NEXT:    paddd %xmm1, %xmm3
+; X64-SSE4-SLOW-NEXT:    psubd %xmm3, %xmm2
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm2, %xmm1
+; X64-SSE4-SLOW-NEXT:    retq
 ;
 ; X64-XOP-LABEL: mul_v8i32_neg33:
 ; X64-XOP:       # %bb.0:
 ; X64-XOP-NEXT:    vextractf128 $1, %ymm0, %xmm1
 ; X64-XOP-NEXT:    vpslld $5, %xmm1, %xmm2
 ; X64-XOP-NEXT:    vpaddd %xmm1, %xmm2, %xmm1
 ; X64-XOP-NEXT:    vpxor %xmm2, %xmm2, %xmm2
 ; X64-XOP-NEXT:    vpsubd %xmm1, %xmm2, %xmm1

 ; SSE-LABEL: mul_v2i64_7:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    movdqa %xmm0, %xmm1
 ; SSE-NEXT:    psllq $3, %xmm1
 ; SSE-NEXT:    psubq %xmm0, %xmm1
 ; SSE-NEXT:    movdqa %xmm1, %xmm0
 ; SSE-NEXT:    ret{{[l|q]}}
 ;
-; X64-XOP-LABEL: mul_v2i64_7:
-; X64-XOP:       # %bb.0:
-; X64-XOP-NEXT:    vpsllq $3, %xmm0, %xmm1
-; X64-XOP-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
-; X64-XOP-NEXT:    retq
-;
-; X64-AVX2-LABEL: mul_v2i64_7:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vpsllq $3, %xmm0, %xmm1
-; X64-AVX2-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
-; X64-AVX2-NEXT:    retq
-;
-; X64-AVX512DQ-LABEL: mul_v2i64_7:
-; X64-AVX512DQ:       # %bb.0:
-; X64-AVX512DQ-NEXT:    vpmullq {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; X64-AVX512DQ-NEXT:    retq
+; X64-AVX-LABEL: mul_v2i64_7:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vpsllq $3, %xmm0, %xmm1
+; X64-AVX-NEXT:    vpsubq %xmm0, %xmm1, %xmm0
+; X64-AVX-NEXT:    retq
   %1 = mul <2 x i64> %a0, <i64 7, i64 7>
   ret <2 x i64> %1
 }
 
 define <4 x i32> @mul_v4i32_7(<4 x i32> %a0) nounwind {
 ; SSE2-LABEL: mul_v4i32_7:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa %xmm0, %xmm1
 ; SSE2-NEXT:    pslld $3, %xmm1
 ; SSE2-NEXT:    psubd %xmm0, %xmm1
 ; SSE2-NEXT:    movdqa %xmm1, %xmm0
 ; SSE2-NEXT:    ret{{[l|q]}}
 ;
 ; X86-SSE4-LABEL: mul_v4i32_7:
 ; X86-SSE4:       # %bb.0:
 ; X86-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}, %xmm0
 ; X86-SSE4-NEXT:    retl
 ;
-; X64-SSE4-LABEL: mul_v4i32_7:
-; X64-SSE4:       # %bb.0:
-; X64-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; X64-SSE4-NEXT:    retq
+; X64-SSE4-FAST-LABEL: mul_v4i32_7:
+; X64-SSE4-FAST:       # %bb.0:
+; X64-SSE4-FAST-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; X64-SSE4-FAST-NEXT:    retq
+;
+; X64-SSE4-SLOW-LABEL: mul_v4i32_7:
+; X64-SSE4-SLOW:       # %bb.0:
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    pslld $3, %xmm1
+; X64-SSE4-SLOW-NEXT:    psubd %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm1, %xmm0
+; X64-SSE4-SLOW-NEXT:    retq
 ;
 ; X64-XOP-LABEL: mul_v4i32_7:
 ; X64-XOP:       # %bb.0:
 ; X64-XOP-NEXT:    vpmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
 ; X64-XOP-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: mul_v4i32_7:
 ; X64-AVX2:       # %bb.0:

 define <2 x i64> @mul_v2i64_neg7(<2 x i64> %a0) nounwind {
 ; SSE-LABEL: mul_v2i64_neg7:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    movdqa %xmm0, %xmm1
 ; SSE-NEXT:    psllq $3, %xmm1
 ; SSE-NEXT:    psubq %xmm1, %xmm0
 ; SSE-NEXT:    ret{{[l|q]}}
 ;
-; X64-XOP-LABEL: mul_v2i64_neg7:
-; X64-XOP:       # %bb.0:
-; X64-XOP-NEXT:    vpsllq $3, %xmm0, %xmm1
-; X64-XOP-NEXT:    vpsubq %xmm1, %xmm0, %xmm0
-; X64-XOP-NEXT:    retq
-;
-; X64-AVX2-LABEL: mul_v2i64_neg7:
-; X64-AVX2:       # %bb.0:
-; X64-AVX2-NEXT:    vpsllq $3, %xmm0, %xmm1
-; X64-AVX2-NEXT:    vpsubq %xmm1, %xmm0, %xmm0
-; X64-AVX2-NEXT:    retq
-;
-; X64-AVX512DQ-LABEL: mul_v2i64_neg7:
-; X64-AVX512DQ:       # %bb.0:
-; X64-AVX512DQ-NEXT:    vpmullq {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
-; X64-AVX512DQ-NEXT:    retq
+; X64-AVX-LABEL: mul_v2i64_neg7:
+; X64-AVX:       # %bb.0:
+; X64-AVX-NEXT:    vpsllq $3, %xmm0, %xmm1
+; X64-AVX-NEXT:    vpsubq %xmm1, %xmm0, %xmm0
+; X64-AVX-NEXT:    retq
   %1 = mul <2 x i64> %a0, <i64 -7, i64 -7>
   ret <2 x i64> %1
 }
 
 define <4 x i32> @mul_v4i32_neg63(<4 x i32> %a0) nounwind {
 ; SSE2-LABEL: mul_v4i32_neg63:
 ; SSE2:       # %bb.0:
 ; SSE2-NEXT:    movdqa %xmm0, %xmm1
 ; SSE2-NEXT:    pslld $6, %xmm1
 ; SSE2-NEXT:    psubd %xmm1, %xmm0
 ; SSE2-NEXT:    ret{{[l|q]}}
 ;
 ; X86-SSE4-LABEL: mul_v4i32_neg63:
 ; X86-SSE4:       # %bb.0:
 ; X86-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}, %xmm0
 ; X86-SSE4-NEXT:    retl
 ;
-; X64-SSE4-LABEL: mul_v4i32_neg63:
-; X64-SSE4:       # %bb.0:
-; X64-SSE4-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
-; X64-SSE4-NEXT:    retq
+; X64-SSE4-FAST-LABEL: mul_v4i32_neg63:
+; X64-SSE4-FAST:       # %bb.0:
+; X64-SSE4-FAST-NEXT:    pmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0
+; X64-SSE4-FAST-NEXT:    retq
+;
+; X64-SSE4-SLOW-LABEL: mul_v4i32_neg63:
+; X64-SSE4-SLOW:       # %bb.0:
+; X64-SSE4-SLOW-NEXT:    movdqa %xmm0, %xmm1
+; X64-SSE4-SLOW-NEXT:    pslld $6, %xmm1
+; X64-SSE4-SLOW-NEXT:    psubd %xmm1, %xmm0
+; X64-SSE4-SLOW-NEXT:    retq
 ;
 ; X64-XOP-LABEL: mul_v4i32_neg63:
 ; X64-XOP:       # %bb.0:
 ; X64-XOP-NEXT:    vpmulld {{\.?LCPI[0-9]+_[0-9]+}}(%rip), %xmm0, %xmm0
 ; X64-XOP-NEXT:    retq
 ;
 ; X64-AVX2-LABEL: mul_v4i32_neg63:
 ; X64-AVX2:       # %bb.0: