[X86][SSE] Add cpu feature for aggressive combining to variable shuffles

[RKSimon]

Summary

As mentioned in D38318 and D40865, modern Intel processors prefer to combine multiple shuffles to a variable shuffle mask (PSHUFB/VPERMPS etc.) instead of having multiple stage 'fixed' shuffles which put more pressure on Port 5 (at the expense of extra shuffle mask loads).

As discussed, this patch provides a FeatureFastVariableShuffle target flag for Haswell+ CPUs that prefers combining 2 or more fixed shuffles to a single variable shuffle (default is 3 shuffles).

If everybody is happy with this approach I will refactor some of the vector-shuffle-* tests to run with -fast-variable-shuffle enabled to compare shuffles.

The long term aim is to drive more of this from schedule data (probably via the MC) but we're not close to being ready for that yet.

[spatel]

LGTM.

The only question is whether we prefer a 'Fast' feature or a 'Slow' feature to distinguish these cases. I don't have a preference, so you might want to wait for more feedback before committing.

[zvi]

LGTM. Thanks

[zvi]

On second thought, i think we need to update the tests with a -mattr=+fast-variable-shuffle configuration, right?

[RKSimon]

In D41323#958329, @zvi wrote:

On second thought, i think we need to update the tests with a -mattr=+fast-variable-shuffle configuration, right?

Yup, that's what I'm working on now.

[RKSimon]

Added slow/fast passes for AVX2/AVX512 shuffle lowering tests - did this for v8i16/v16i8 and all 256-bit types as these have useful tests to show the diffs. I can add them for the others if you wish but these will mostly not cause any codegen diffs.

[zvi]

Here's the full list of tests that are affected by setting AllowVariableMask for Depth=2. I think that we should have the full list covered with the new configuration.
I would be happy to assist with the work involved.

CodeGen/X86/avx-intrinsics-fast-isel.ll
CodeGen/X86/avx-splat.ll
CodeGen/X86/avx2-conversions.ll
CodeGen/X86/avx2-vector-shifts.ll
CodeGen/X86/avx512-shuffles/broadcast-vector-int.ll
CodeGen/X86/avx512-shuffles/partial_permute.ll
CodeGen/X86/avx512-trunc.ll
CodeGen/X86/bitcast-and-setcc-512.ll
CodeGen/X86/bitcast-int-to-vector-bool-sext.ll
CodeGen/X86/bitcast-int-to-vector-bool-zext.ll
CodeGen/X86/bitcast-int-to-vector-bool.ll
CodeGen/X86/cast-vsel.ll
CodeGen/X86/combine-or.ll
CodeGen/X86/combine-shl.ll
CodeGen/X86/combine-sra.ll
CodeGen/X86/combine-srl.ll
CodeGen/X86/insertelement-duplicates.ll
CodeGen/X86/insertelement-zero.ll
CodeGen/X86/known-signbits-vector.ll
CodeGen/X86/merge-consecutive-loads-128.ll
CodeGen/X86/oddshuffles.ll
CodeGen/X86/psubus.ll
CodeGen/X86/reduce-trunc-shl.ll
CodeGen/X86/select.ll
CodeGen/X86/shuffle-of-splat-multiuses.ll
CodeGen/X86/shuffle-strided-with-offset-128.ll
CodeGen/X86/shuffle-strided-with-offset-256.ll
CodeGen/X86/shuffle-strided-with-offset-512.ll
CodeGen/X86/shuffle-vs-trunc-128.ll
CodeGen/X86/shuffle-vs-trunc-256.ll
CodeGen/X86/shuffle-vs-trunc-512.ll
CodeGen/X86/sse41.ll
CodeGen/X86/swizzle-2.ll
CodeGen/X86/trunc-ext-ld-st.ll
CodeGen/X86/vec_cast2.ll
CodeGen/X86/vec_insert-5.ll
CodeGen/X86/vec_insert-mmx.ll
CodeGen/X86/vec_set-3.ll
CodeGen/X86/vec_trunc_sext.ll
CodeGen/X86/vector-compare-results.ll
CodeGen/X86/vector-half-conversions.ll
CodeGen/X86/vector-rotate-128.ll
CodeGen/X86/vector-rotate-256.ll
CodeGen/X86/vector-sext.ll
CodeGen/X86/vector-shuffle-128-v16.ll
CodeGen/X86/vector-shuffle-128-v4.ll
CodeGen/X86/vector-shuffle-128-v8.ll

[RKSimon]

In D41323#958558, @zvi wrote:

Here's the full list of tests that are affected by setting AllowVariableMask for Depth=2. I think that we should have the full list covered with the new configuration.
I would be happy to assist with the work involved.

Sure - as long as we're testing with the fast and slow cases for them all. Are you happy with this patch with its tests changes as it is and you just update the remaining tests as followups?

[zvi]

In D41323#958561, @RKSimon wrote:

In D41323#958558, @zvi wrote:

Here's the full list of tests that are affected by setting AllowVariableMask for Depth=2. I think that we should have the full list covered with the new configuration.
I would be happy to assist with the work involved.

Sure - as long as we're testing with the fast and slow cases for them all. Are you happy with this patch with its tests changes as it is and you just update the remaining tests as followups?

Ok, let's do that.

[zvi]

Since all known processors with AVX512 will prefer this new feature turned on, can we make AVX512 imply Fast-var-shuffles?

[RKSimon]

In D41323#960589, @zvi wrote:

Since all known processors with AVX512 will prefer this new feature turned on, can we make AVX512 imply Fast-var-shuffles?

I've created D41436 - the main issue is whether KNL prefers variable shuffles the same as SkylakeServer

[zvi]

In D41323#960715, @RKSimon wrote:

I've created D41436 - the main issue is whether KNL prefers variable shuffles the same as SkylakeServer

The Intel Optimization Reference Manual is the only source of information i have for KNL. It says that of the two core's VPUs, one can execute shuffles. It says that single-source shuffles have a reciprocal throughput of 1 cycle, dual-source shuffles have a reciprocal throughput of 2 cycles. So i don't have any information indicating that KNL differs from SKX in preference of variable-shuffles.

lib/Target/X86/X86.td

def FeatureSlowLEA : SubtargetFeature<"slow-lea", "SlowLEA", "true",
                                    "LEA instruction with certain arguments is slow">;
 def FeatureSlow3OpsLEA : SubtargetFeature<"slow-3ops-lea", "Slow3OpsLEA", "true",
                                    "LEA instruction with 3 ops or certain registers is slow">;
 def FeatureSlowIncDec : SubtargetFeature<"slow-incdec", "SlowIncDec", "true",
                                    "INC and DEC instructions are slower than ADD and SUB">;
 def FeatureSoftFloat
     : SubtargetFeature<"soft-float", "UseSoftFloat", "true",
                        "Use software floating point features.">;
+// On recent X86 (port bound) processors, its preferable to combine to a single shuffle
+// using a variable mask over multiple fixed shuffles.
+def FeatureFastVariableShuffle
+    : SubtargetFeature<"fast-variable-shuffle",
+                       "HasFastVariableShuffle",
+                       "true", "Shuffles with variable masks are fast">;
 // On some X86 processors, there is no performance hazard to writing only the
 // lower parts of a YMM or ZMM register without clearing the upper part.
 def FeatureFastPartialYMMorZMMWrite
     : SubtargetFeature<"fast-partial-ymm-or-zmm-write",
                        "HasFastPartialYMMorZMMWrite",
                        "true", "Partial writes to YMM/ZMM registers are fast">;
 // FeatureFastScalarFSQRT should be enabled if scalar FSQRT has shorter latency
 // than the corresponding NR code. FeatureFastVectorFSQRT should be enabled if

 
 def HSWFeatures : ProcessorFeatures<IVBFeatures.Value, [
   FeatureAVX2,
   FeatureBMI,
   FeatureBMI2,
   FeatureERMSB,
   FeatureFMA,
   FeatureLZCNT,
-  FeatureMOVBE
+  FeatureMOVBE,
+  FeatureFastVariableShuffle
 ]>;
 
 class HaswellProc<string Name> : ProcModel<Name, HaswellModel,
                                            HSWFeatures.Value, [
   ProcIntelHSW
 ]>;
 def : HaswellProc<"haswell">;
 def : HaswellProc<"core-avx2">; // Legacy alias.

lib/Target/X86/X86ISelLowering.cpp

static SDValue combineX86ShuffleChain(ArrayRef<SDValue> Inputs, SDValue Root,
   }
 
   // Don't try to re-form single instruction chains under any circumstances now
   // that we've done encoding canonicalization for them.
   if (Depth < 2)
     return SDValue();
 
   // Depth threshold above which we can efficiently use variable mask shuffles.
-  // TODO This should probably be target specific.
-  bool AllowVariableMask = (Depth >= 3) || HasVariableMask;
+  int VariableShuffleDepth = Subtarget.hasFastVariableShuffle() ? 2 : 3;
+  bool AllowVariableMask = (Depth >= VariableShuffleDepth) || HasVariableMask;
 
   bool MaskContainsZeros =
       any_of(Mask, [](int M) { return M == SM_SentinelZero; });
 
   if (is128BitLaneCrossingShuffleMask(MaskVT, Mask)) {
     // If we have a single input lane-crossing shuffle then lower to VPERMV.
     if (UnaryShuffle && AllowVariableMask && !MaskContainsZeros &&
         ((Subtarget.hasAVX2() &&

lib/Target/X86/X86Subtarget.h

protected:
   /// True if this processor has the CMPXCHG16B instruction;
   /// this is true for most x86-64 chips, but not the first AMD chips.
   bool HasCmpxchg16b;
 
   /// True if the LEA instruction should be used for adjusting
   /// the stack pointer. This is an optimization for Intel Atom processors.
   bool UseLeaForSP;
 
+  /// True if its preferable to combine to a single shuffle using a variable
+  /// mask over multiple fixed shuffles.
+  bool HasFastVariableShuffle;
+
   /// True if there is no performance penalty to writing only the lower parts
   /// of a YMM or ZMM register without clearing the upper part.
   bool HasFastPartialYMMorZMMWrite;
 
   /// True if gather is reasonably fast. This is true for Skylake client and
   /// all AVX-512 CPUs.
   bool HasFastGather;
 
public:
   bool isPMULLDSlow() const { return IsPMULLDSlow; }
   bool isUnalignedMem16Slow() const { return IsUAMem16Slow; }
   bool isUnalignedMem32Slow() const { return IsUAMem32Slow; }
   int getGatherOverhead() const { return GatherOverhead; }
   int getScatterOverhead() const { return ScatterOverhead; }
   bool hasSSEUnalignedMem() const { return HasSSEUnalignedMem; }
   bool hasCmpxchg16b() const { return HasCmpxchg16b; }
   bool useLeaForSP() const { return UseLeaForSP; }
+  bool hasFastVariableShuffle() const {
+    return HasFastVariableShuffle;
+  }
   bool hasFastPartialYMMorZMMWrite() const {
     return HasFastPartialYMMorZMMWrite;
   }
   bool hasFastGather() const { return HasFastGather; }
   bool hasFastScalarFSQRT() const { return HasFastScalarFSQRT; }
   bool hasFastVectorFSQRT() const { return HasFastVectorFSQRT; }
   bool hasFastLZCNT() const { return HasFastLZCNT; }
   bool hasFastSHLDRotate() const { return HasFastSHLDRotate; }

test/CodeGen/X86/vector-shuffle-512-v32.ll

 ; KNL-NEXT:    vpshuflw {{.*#+}} ymm0 = ymm0[1,1,0,0,4,5,6,7,9,9,8,8,12,13,14,15]
 ; KNL-NEXT:    vpshufhw {{.*#+}} ymm0 = ymm0[0,1,2,3,5,5,4,4,8,9,10,11,13,13,12,12]
 ; KNL-NEXT:    vpshuflw {{.*#+}} ymm1 = ymm1[1,1,0,0,4,5,6,7,9,9,8,8,12,13,14,15]
 ; KNL-NEXT:    vpshufhw {{.*#+}} ymm1 = ymm1[0,1,2,3,5,5,4,4,8,9,10,11,13,13,12,12]
 ; KNL-NEXT:    retq
 ;
 ; SKX-LABEL: shuffle_v32i16_1_1_0_0_5_5_4_4_9_9_11_11_13_13_12_12_17_17_19_19_21_21_20_20_25_25_27_27_29_29_28_28:
 ; SKX:       ## %bb.0:
-; SKX-NEXT:    vpshuflw {{.*#+}} zmm0 = zmm0[1,1,0,0,4,5,6,7,9,9,8,8,12,13,14,15,17,17,16,16,20,21,22,23,25,25,24,24,28,29,30,31]
-; SKX-NEXT:    vpshufhw {{.*#+}} zmm0 = zmm0[0,1,2,3,5,5,4,4,8,9,10,11,13,13,12,12,16,17,18,19,21,21,20,20,24,25,26,27,29,29,28,28]
+; SKX-NEXT:    vpshufb {{.*#+}} zmm0 = zmm0[2,3,2,3,0,1,0,1,10,11,10,11,8,9,8,9,18,19,18,19,16,17,16,17,26,27,26,27,24,25,24,25,34,35,34,35,32,33,32,33,42,43,42,43,40,41,40,41,50,51,50,51,48,49,48,49,58,59,58,59,56,57,56,57]
 ; SKX-NEXT:    retq
   %c = shufflevector <32 x i16> %a, <32 x i16> zeroinitializer, <32 x i32> <i32 1, i32 1, i32 0, i32 0, i32 5, i32 5, i32 4, i32 4, i32 9, i32 9, i32 8, i32 8, i32 13, i32 13, i32 12, i32 12, i32 17, i32 17, i32 16, i32 16, i32 21, i32 21, i32 20, i32 20, i32 25, i32 25, i32 24, i32 24, i32 29, i32 29, i32 28, i32 28>
   ret <32 x i16> %c
 }
 
 define <32 x i16> @shuffle_v32i16_0zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz(<32 x i16> %a) {
 ; KNL-LABEL: shuffle_v32i16_0zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz:
 ; KNL:       ## %bb.0: