Fix for bug 41512: lower INSERT_VECTOR_ELT(ZeroVec, 0, Elt) to SCALAR_TO_VECTOR(Elt) for all SSE flavors

[Serge_Preis]

Summary

Current LLVM uses pxor+pinsrb on SSE4+ for INSERT_VECTOR_ELT(ZeroVec, 0, Elt) insead of much simpler movd.
INSERT_VECTOR_ELT(ZeroVec, 0, Elt) is idiomatic construct which is used e.g. for _mm_cvtsi32_si128(Elt) and for lowest element initialization in _mm_set_epi32.
So such inefficient lowering leads to significant performance digradations in ceratin cases switching from SSSE3 to SSE4.
https://bugs.llvm.org/show_bug.cgi?id=41512

Here INSERT_VECTOR_ELT(ZeroVec, 0, Elt) is simply converted to SCALAR_TO_VECTOR(Elt) when applicable since latter is closer match to desired behavior and always efficiently lowered to movd and alike.

[RKSimon]

Your test cases need to be a lot simpler - I'd recommend looking at buildvec-insertvec.ll and possibly adding your tests to that file instead of adding this new file.

You will need to use the utils/update_llc_test_checks.py script to autogenerate your codegen checks as well.

[Serge_Preis]

In D60852#1471378, @RKSimon wrote:

Your test cases need to be a lot simpler - I'd recommend looking at buildvec-insertvec.ll and possibly adding your tests to that file instead of adding this new file.

The problem is that in tiny kernel llvm behaves as I expect it to, while in a loop it underperforms: https://gcc.godbolt.org/z/PljujX -- compare Sum2 and Loop() code generation.
I will do my best to minimize the case.

You will need to use the utils/update_llc_test_checks.py script to autogenerate your codegen checks as well.

Ok, I will look into this.

[spatel]

In D60852#1471492, @Serge_Preis wrote:

In D60852#1471378, @RKSimon wrote:

Your test cases need to be a lot simpler - I'd recommend looking at buildvec-insertvec.ll and possibly adding your tests to that file instead of adding this new file.

The problem is that in tiny kernel llvm behaves as I expect it to, while in a loop it underperforms: https://gcc.godbolt.org/z/PljujX -- compare Sum2 and Loop() code generation.
I will do my best to minimize the case.

If we are getting this right sometimes, then we might already have the transform that we want, but it is limited in some way that prevents getting the larger case.

I doubt that the loop itself is needed to demonstrate the problem because I see 'movd' codegen even with a loop as long as it is not unrolled.

[Serge_Preis]

If we are getting this right sometimes, then we might already have the transform that we want, but it is limited in some way that prevents getting the larger case.
I doubt that the loop itself is needed to demonstrate the problem because I see 'movd' codegen even with a loop as long as it is not unrolled.

After more experimentation I tend to agree. Also the most basic case produces pinsrd even in a small kernel (https://gcc.godbolt.org/z/HAmNha), so will just create test out of it.

[spatel]

In D60852#1471661, @Serge_Preis wrote:

If we are getting this right sometimes, then we might already have the transform that we want, but it is limited in some way that prevents getting the larger case.
I doubt that the loop itself is needed to demonstrate the problem because I see 'movd' codegen even with a loop as long as it is not unrolled.

After more experimentation I tend to agree. Also the most basic case produces pinsrd even in a small kernel (https://gcc.godbolt.org/z/HAmNha), so will just create test out of it.

Not sure if this is minimal, but this seems to show the problem:

define <2 x i64> @pinsr(i32 %x, i32 %y) {
  %ins1 = insertelement <4 x i32> <i32 undef, i32 0, i32 undef, i32 undef>, i32 %x, i32 0
  %ins2 = insertelement <4 x i32> <i32 undef, i32 0, i32 undef, i32 undef>, i32 %y, i32 0
  %b1 = bitcast <4 x i32> %ins1 to <2 x i64>
  %b2 = bitcast <4 x i32> %ins2 to <2 x i64>
  %r = shufflevector <2 x i64> %b1, <2 x i64> %b2, <2 x i32> <i32 0, i32 2>
  ret <2 x i64> %r
}

$ llc -o - pinsr.ll -mattr=sse4.2
	pxor	%xmm1, %xmm1
	pxor	%xmm0, %xmm0
	pinsrd	$0, %edi, %xmm0
	pinsrd	$0, %esi, %xmm1
	punpcklqdq	%xmm1, %xmm0    ## xmm0 = xmm0[0],xmm1[0]
	retq

[Serge_Preis]

Fixed test according to review comments

[spatel]

I added some baseline tests here:
rL358687

Please rebase/update against those and upload a new patch.

[Serge_Preis]

With updated insertelement-zero.ll test

[spatel]

This looks like the expected improvement to me, but given the 'blend' comment/test, I'd like someone else to confirm that I'm not overlooking some weird corner case.

llvm/test/CodeGen/X86/insertelement-zero.ll

503 ↗

(On Diff #195795)

Remove 'FIXME' from this line.

633–635 ↗

(On Diff #195795)

I looked at this case, and it is intentional because there's a comment in X86InstrSSE.td:
"these were changed to use blends because blends have better throughput on sandybridge and haswell"

[Serge_Preis]

Deleted FIXME

[Serge_Preis]

llvm/test/CodeGen/X86/insertelement-zero.ll

633–635 ↗

(On Diff #195795)

My change itself doen't dictate any particular instruction selection. it just makes more obvious to ISel the idiom it deals with. If some instructions are better than others for 1st element intialization they should be naturaly used after the fix. So I'm just happy to see confirmation here.

[Serge_Preis]

Friendly ping: how should I proceed with this review?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

16980 ↗

(On Diff #195858)

Subtarget.is64Bit() should be superfluous

[Serge_Preis]

llvm/lib/Target/X86/X86ISelLowering.cpp

16980 ↗

(On Diff #195858)

This is a good question, On one (conservative) hand I am aiming at movq instruction, whch is only available on 64bit subtarget (and similar places have same set of checks). On other hand the logic behind the conversion is pretty generic and doesn't depend on subtarget.. So the question is "are we sure that 64bit SCALAR_TO_VECTOR will be handled efficiently now on 32bit subtarget or such generalization will require further patching?"

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

16980 ↗

(On Diff #195858)

Type legalization would have guaranteed there are no i64 values in the DAG if this is not a 64-bit target before we get to this code.

[Serge_Preis]

Removed subtarget check in response to review comments

[Serge_Preis]

llvm/lib/Target/X86/X86ISelLowering.cpp

16980 ↗

(On Diff #195858)

Ok. uploaded new iteration without subtarget check

[RKSimon]

LGTM - @Serge_Preis do you have commit access?

[Serge_Preis]

In D60852#1483730, @RKSimon wrote:

@Serge_Preis do you have commit access?

No, unfortunately don't.

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

if (VT.is256BitVector() || VT.is512BitVector()) {
     V = DAG.getNode(ISD::INSERT_VECTOR_ELT, dl, V.getValueType(), V, N1,
                     DAG.getConstant(IdxIn128, dl, MVT::i32));
 
     // Insert the changed part back into the bigger vector
     return insert128BitVector(N0, V, IdxVal, DAG, dl);
   }
   assert(VT.is128BitVector() && "Only 128-bit vector types should be left!");
 
+  // This will be just movd/movq/movss/movsd.
+  if (IdxVal == 0 && ISD::isBuildVectorAllZeros(N0.getNode()) &&
+      (EltVT == MVT::i32 || EltVT == MVT::f32 || EltVT == MVT::f64 ||
+       EltVT == MVT::i64)) {
+    N1 = DAG.getNode(ISD::SCALAR_TO_VECTOR, dl, VT, N1);
+    return getShuffleVectorZeroOrUndef(N1, 0, true, Subtarget, DAG);
+  }
+
   // Transform it so it match pinsr{b,w} which expects a GR32 as its second
   // argument. SSE41 required for pinsrb.
   if (VT == MVT::v8i16 || (VT == MVT::v16i8 && Subtarget.hasSSE41())) {
     unsigned Opc;
     if (VT == MVT::v8i16) {
       assert(Subtarget.hasSSE2() && "SSE2 required for PINSRW");
       Opc = X86ISD::PINSRW;
     } else {

llvm/trunk/test/CodeGen/X86/insertelement-zero.ll

 ; AVX2-FAST-NEXT:    retq
   %1 = insertelement <32 x i8> %a, i8 0, i32 0
   %2 = insertelement <32 x i8> %1, i8 0, i32 15
   %3 = insertelement <32 x i8> %2, i8 0, i32 30
   %4 = insertelement <32 x i8> %3, i8 0, i32 31
   ret <32 x i8> %4
 }
 
-; FIXME: Prefer 'movd' over 'pinsr' to element 0.
-
 define <4 x i32> @PR41512(i32 %x, i32 %y) {
-; SSE2-LABEL: PR41512:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    movd %edi, %xmm0
-; SSE2-NEXT:    movd %esi, %xmm1
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE2-NEXT:    retq
-;
-; SSE3-LABEL: PR41512:
-; SSE3:       # %bb.0:
-; SSE3-NEXT:    movd %edi, %xmm0
-; SSE3-NEXT:    movd %esi, %xmm1
-; SSE3-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE3-NEXT:    retq
-;
-; SSSE3-LABEL: PR41512:
-; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    movd %edi, %xmm0
-; SSSE3-NEXT:    movd %esi, %xmm1
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSSE3-NEXT:    retq
-;
-; SSE41-LABEL: PR41512:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    pxor %xmm1, %xmm1
-; SSE41-NEXT:    pxor %xmm0, %xmm0
-; SSE41-NEXT:    pinsrd $0, %edi, %xmm0
-; SSE41-NEXT:    pinsrd $0, %esi, %xmm1
-; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE41-NEXT:    retq
+; SSE-LABEL: PR41512:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movd %edi, %xmm0
+; SSE-NEXT:    movd %esi, %xmm1
+; SSE-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: PR41512:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; AVX-NEXT:    vpinsrd $0, %edi, %xmm0, %xmm1
-; AVX-NEXT:    vpinsrd $0, %esi, %xmm0, %xmm0
-; AVX-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm1[0],xmm0[0]
+; AVX-NEXT:    vmovd %edi, %xmm0
+; AVX-NEXT:    vmovd %esi, %xmm1
+; AVX-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
 ; AVX-NEXT:    retq
   %ins1 = insertelement <4 x i32> <i32 undef, i32 0, i32 undef, i32 undef>, i32 %x, i32 0
   %ins2 = insertelement <4 x i32> <i32 undef, i32 0, i32 undef, i32 undef>, i32 %y, i32 0
   %r = shufflevector <4 x i32> %ins1, <4 x i32> %ins2, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
   ret <4 x i32> %r
 }
 
 define <4 x i64> @PR41512_v4i64(i64 %x, i64 %y) {
-; SSE2-LABEL: PR41512_v4i64:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    movq %rdi, %xmm0
-; SSE2-NEXT:    movq %rsi, %xmm1
-; SSE2-NEXT:    retq
-;
-; SSE3-LABEL: PR41512_v4i64:
-; SSE3:       # %bb.0:
-; SSE3-NEXT:    movq %rdi, %xmm0
-; SSE3-NEXT:    movq %rsi, %xmm1
-; SSE3-NEXT:    retq
-;
-; SSSE3-LABEL: PR41512_v4i64:
-; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    movq %rdi, %xmm0
-; SSSE3-NEXT:    movq %rsi, %xmm1
-; SSSE3-NEXT:    retq
-;
-; SSE41-LABEL: PR41512_v4i64:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    pxor %xmm1, %xmm1
-; SSE41-NEXT:    pxor %xmm0, %xmm0
-; SSE41-NEXT:    pinsrq $0, %rdi, %xmm0
-; SSE41-NEXT:    pinsrq $0, %rsi, %xmm1
-; SSE41-NEXT:    retq
+; SSE-LABEL: PR41512_v4i64:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq %rdi, %xmm0
+; SSE-NEXT:    movq %rsi, %xmm1
+; SSE-NEXT:    retq
 ;
 ; AVX1-LABEL: PR41512_v4i64:
 ; AVX1:       # %bb.0:
-; AVX1-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; AVX1-NEXT:    vpinsrq $0, %rdi, %xmm0, %xmm1
-; AVX1-NEXT:    vpinsrq $0, %rsi, %xmm0, %xmm0
-; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
+; AVX1-NEXT:    vmovq %rdi, %xmm0
+; AVX1-NEXT:    vmovq %rsi, %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: PR41512_v4i64:
 ; AVX2:       # %bb.0:
-; AVX2-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; AVX2-NEXT:    vpinsrq $0, %rdi, %xmm0, %xmm1
-; AVX2-NEXT:    vpinsrq $0, %rsi, %xmm0, %xmm0
-; AVX2-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
+; AVX2-NEXT:    vmovq %rdi, %xmm0
+; AVX2-NEXT:    vmovq %rsi, %xmm1
+; AVX2-NEXT:    vinserti128 $1, %xmm1, %ymm0, %ymm0
 ; AVX2-NEXT:    retq
   %ins1 = insertelement <4 x i64> <i64 undef, i64 0, i64 undef, i64 undef>, i64 %x, i32 0
   %ins2 = insertelement <4 x i64> <i64 undef, i64 0, i64 undef, i64 undef>, i64 %y, i32 0
   %r = shufflevector <4 x i64> %ins1, <4 x i64> %ins2, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
   ret <4 x i64> %r
 }
 
 define <8 x float> @PR41512_v8f32(float %x, float %y) {

 ; AVX-NEXT:    retq
   %ins1 = insertelement <8 x float> zeroinitializer, float %x, i32 0
   %ins2 = insertelement <8 x float> zeroinitializer, float %y, i32 0
   %r = shufflevector <8 x float> %ins1, <8 x float> %ins2, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 8, i32 9, i32 10, i32 11>
   ret <8 x float> %r
 }
 
 define <4 x i32> @PR41512_loads(i32* %p1, i32* %p2) {
-; SSE2-LABEL: PR41512_loads:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE2-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; SSE2-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE2-NEXT:    retq
-;
-; SSE3-LABEL: PR41512_loads:
-; SSE3:       # %bb.0:
-; SSE3-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSE3-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; SSE3-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE3-NEXT:    retq
-;
-; SSSE3-LABEL: PR41512_loads:
-; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; SSSE3-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; SSSE3-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSSE3-NEXT:    retq
-;
-; SSE41-LABEL: PR41512_loads:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    pxor %xmm1, %xmm1
-; SSE41-NEXT:    pxor %xmm0, %xmm0
-; SSE41-NEXT:    pinsrd $0, (%rdi), %xmm0
-; SSE41-NEXT:    pinsrd $0, (%rsi), %xmm1
-; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm1[0]
-; SSE41-NEXT:    retq
+; SSE-LABEL: PR41512_loads:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE-NEXT:    movlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: PR41512_loads:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpxor %xmm0, %xmm0, %xmm0
-; AVX-NEXT:    vpinsrd $0, (%rdi), %xmm0, %xmm1
-; AVX-NEXT:    vpinsrd $0, (%rsi), %xmm0, %xmm0
-; AVX-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm1[0],xmm0[0]
+; AVX-NEXT:    vmovss {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX-NEXT:    vmovlhps {{.*#+}} xmm0 = xmm0[0],xmm1[0]
 ; AVX-NEXT:    retq
   %x = load i32, i32* %p1
   %y = load i32, i32* %p2
   %ins1 = insertelement <4 x i32> <i32 undef, i32 0, i32 undef, i32 undef>, i32 %x, i32 0
   %ins2 = insertelement <4 x i32> <i32 undef, i32 0, i32 undef, i32 undef>, i32 %y, i32 0
   %r = shufflevector <4 x i32> %ins1, <4 x i32> %ins2, <4 x i32> <i32 0, i32 1, i32 4, i32 5>
   ret <4 x i32> %r
 }