[X86] Prefer vmovmsk instead of vtest for alderlake.

[LuoYuanke]

Summary

On alderlake E-core, the latency of VMOVMSKPS is 5 for YMM/XMM. The
latency of VPTESTPS is 7 for YMM and is 5 for XMM. Since alderlake use
the P-core schedule model, we can't determine which one better based on
the latency information of schedule model. Alternatively we add an
tuning feature for alderlake and select VMOVMSKPS with the indication
for the tuning feature. In the case of "vmovmskps + test + jcc", the
test and jcc can be fused, while vtest and jcc can't.

[LuoYuanke]

Rebase and update test case.

[goldstein.w.n]

llvm/lib/Target/X86/X86.td

426

Personally think slowvtest is a kind of confusing name b.c vtest also has a perf dropoff from SnB -> HSW.
Maybe "PreferMovmskOverVTest" would be clearer?

[LuoYuanke]

Rebase and update test cases.

[LuoYuanke]

llvm/lib/Target/X86/X86.td

426

I just follow the previous naming convention. I'm open to "PreferMovmskOverVTest". @RKSimon and @pengfei, what's your opinion?

[pengfei]

llvm/lib/Target/X86/X86.td

425

Should be Is?

426

I think we should start from Tuning, but TuningPreferMovmskOverVTest looks verbose..

[LuoYuanke]

Address Phoebe's comments.

[RKSimon]

llvm/lib/Target/X86/X86.td

426

+1 TuningPreferMovmskOverVTest explains the purpose of the tuning flag better

llvm/test/CodeGen/X86/combine-movmsk.ll

6

Add a variant that tests the tuning flag directly
; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+slow-vtest | FileCheck %s --check-prefix=ADL

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

47913

What about these folds?

47973

What about these folds?

[LuoYuanke]

Address Simon's comments.

[LuoYuanke]

llvm/lib/Target/X86/X86ISelLowering.cpp

47913

I'll take a look at it and I prefer to implement it in a separate patch if MOVMSK is better.

47973

Ditto

[LuoYuanke]

llvm/lib/Target/X86/X86ISelLowering.cpp

47913

It seems this transform is good. If I revert the transform, I get below lit test change.

-; AVX-LABEL: movmsk_or_v2i64:
-; AVX:       # %bb.0:
-; AVX-NEXT:    vpxor %xmm1, %xmm0, %xmm0
-; AVX-NEXT:    vptest %xmm0, %xmm0
-; AVX-NEXT:    setne %al
-; AVX-NEXT:    retq
+; AVX1OR2-LABEL: movmsk_or_v2i64:
+; AVX1OR2:       # %bb.0:
+; AVX1OR2-NEXT:    vpcmpeqq %xmm1, %xmm0, %xmm0
+; AVX1OR2-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
+; AVX1OR2-NEXT:    vtestpd %xmm1, %xmm0
+; AVX1OR2-NEXT:    setae %al
+; AVX1OR2-NEXT:    retq

[LuoYuanke]

llvm/lib/Target/X86/X86ISelLowering.cpp

47973

It seems there is not lit test failure if I disable this code.

[LuoYuanke]

Hi,
Is there anything else that I need to update?

[pengfei]

LGTM. Please wait for one more day for other reviewers.

[goldstein.w.n]

LGTM as well.

[RKSimon]

LGTM - cheers

llvm/lib/Target/X86/X86.td

 def FeatureHardenSlsIJmp
     : SubtargetFeature<
           "harden-sls-ijmp", "HardenSlsIJmp", "true",
           "Harden against straight line speculation across indirect JMP instructions.">;
 
 //===----------------------------------------------------------------------===//
 // X86 Subtarget Tuning features
 //===----------------------------------------------------------------------===//
+def TuningPreferMovmskOverVTest : SubtargetFeature<"prefer-movmsk-over-vtest",
+                                       "PreferMovmskOverVTest", "true",
+                                       "Prefer movmsk over vtest instruction">;
 
 def TuningSlowSHLD : SubtargetFeature<"slow-shld", "IsSHLDSlow", "true",
                                        "SHLD instruction is slow">;
 
 def TuningSlowPMULLD : SubtargetFeature<"slow-pmulld", "IsPMULLDSlow", "true",
                                         "PMULLD instruction is slow (compared to PMULLW/PMULHW and PMULUDQ)">;
 
 def TuningSlowPMADDWD : SubtargetFeature<"slow-pmaddwd", "IsPMADDWDSlow",
list<SubtargetFeature> ADLAdditionalFeatures = [FeatureSERIALIZE,
                                                   FeatureLZCNT,
                                                   FeatureAVXVNNI,
                                                   FeaturePKU,
                                                   FeatureHRESET,
                                                   FeatureCLDEMOTE,
                                                   FeatureMOVDIRI,
                                                   FeatureMOVDIR64B,
                                                   FeatureWAITPKG];
-  list<SubtargetFeature> ADLAdditionalTuning = [TuningPERMFalseDeps];
+  list<SubtargetFeature> ADLAdditionalTuning = [TuningPERMFalseDeps,
+                                                TuningPreferMovmskOverVTest];
   list<SubtargetFeature> ADLTuning = !listconcat(SKLTuning, ADLAdditionalTuning);
   list<SubtargetFeature> ADLFeatures =
     !listconcat(TRMFeatures, ADLAdditionalFeatures);
 
   // Sierraforest
   list<SubtargetFeature> SRFAdditionalFeatures = [FeatureCMPCCXADD,
                                                   FeatureAVXIFMA,
                                                   FeatureAVXNECONVERT,

llvm/lib/Target/X86/X86ISelLowering.cpp

                          DAG.getConstant(CmpMask, DL, MVT::i32));
     }
   }
 
   // MOVMSK(PCMPEQ(X,0)) == -1 -> PTESTZ(X,X).
   // MOVMSK(PCMPEQ(X,0)) != -1 -> !PTESTZ(X,X).
   // MOVMSK(PCMPEQ(X,Y)) == -1 -> PTESTZ(XOR(X,Y),XOR(X,Y)).
   // MOVMSK(PCMPEQ(X,Y)) != -1 -> !PTESTZ(XOR(X,Y),XOR(X,Y)).
   if (IsAllOf && Subtarget.hasSSE41() && IsOneUse) {
     MVT TestVT = VecVT.is128BitVector() ? MVT::v2i64 : MVT::v4i64;
     SDValue BC = peekThroughBitcasts(Vec);
     // Ensure MOVMSK was testing every signbit of BC.
     if (BC.getValueType().getVectorNumElements() <= NumElts) {
       if (BC.getOpcode() == X86ISD::PCMPEQ) {
         SDValue V = DAG.getNode(ISD::XOR, SDLoc(BC), BC.getValueType(),
                                 BC.getOperand(0), BC.getOperand(1));
         V = DAG.getBitcast(TestVT, V);

     // PMOVMSKB(PACKSSBW(LO(X), HI(X)))
     // -> PMOVMSKB(BITCAST_v32i8(X)) & 0xAAAAAAAA.
     if (CmpBits >= 16 && Subtarget.hasInt256() &&
         (IsAnyOf || (SignExt0 && SignExt1))) {
       if (SDValue Src = getSplitVectorSrc(VecOp0, VecOp1, true)) {
         SDLoc DL(EFLAGS);
         SDValue Result = peekThroughBitcasts(Src);
         if (IsAllOf && Result.getOpcode() == X86ISD::PCMPEQ &&
             Result.getValueType().getVectorNumElements() <= NumElts) {
           SDValue V = DAG.getNode(ISD::XOR, DL, Result.getValueType(),
                                   Result.getOperand(0), Result.getOperand(1));
           V = DAG.getBitcast(MVT::v4i64, V);
           return DAG.getNode(X86ISD::PTEST, SDLoc(EFLAGS), MVT::i32, V, V);
         }
         Result = DAG.getBitcast(MVT::v32i8, Result);
         Result = DAG.getNode(X86ISD::MOVMSK, DL, MVT::i32, Result);
         unsigned CmpMask = IsAnyOf ? 0 : 0xFFFFFFFF;

     }
   }
 
   // MOVMSKPS(V) !=/== 0 -> TESTPS(V,V)
   // MOVMSKPD(V) !=/== 0 -> TESTPD(V,V)
   // MOVMSKPS(V) !=/== -1 -> TESTPS(V,V)
   // MOVMSKPD(V) !=/== -1 -> TESTPD(V,V)
   // iff every element is referenced.
-  if (NumElts <= CmpBits && Subtarget.hasAVX() && IsOneUse &&
+  if (NumElts <= CmpBits && Subtarget.hasAVX() &&
+      !Subtarget.preferMovmskOverVTest() && IsOneUse &&
       (NumEltBits == 32 || NumEltBits == 64)) {
     SDLoc DL(EFLAGS);
     MVT FloatSVT = MVT::getFloatingPointVT(NumEltBits);
     MVT FloatVT = MVT::getVectorVT(FloatSVT, NumElts);
     MVT IntVT = FloatVT.changeVectorElementTypeToInteger();
     SDValue LHS = Vec;
     SDValue RHS = IsAnyOf ? Vec : DAG.getAllOnesConstant(DL, IntVT);
     CC = IsAnyOf ? CC : (CC == X86::COND_E ? X86::COND_B : X86::COND_AE);

llvm/test/CodeGen/X86/combine-movmsk-avx.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefixes=CHECK,AVX1
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=CHECK,AVX2
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=alderlake | FileCheck %s --check-prefixes=ADL
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+prefer-movmsk-over-vtest | FileCheck %s --check-prefixes=ADL
 
 declare i32 @llvm.x86.avx.movmsk.pd.256(<4 x double>)
 declare i32 @llvm.x86.avx.movmsk.ps.256(<8 x float>)
 
 ; Use widest possible vector for movmsk comparisons (PR37087)
 
 define i1 @movmskps_noneof_bitcast_v4f64(<4 x double> %a0) {
 ; CHECK-LABEL: movmskps_noneof_bitcast_v4f64:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; CHECK-NEXT:    vcmpeqpd %ymm1, %ymm0, %ymm0
 ; CHECK-NEXT:    vtestpd %ymm0, %ymm0
 ; CHECK-NEXT:    sete %al
 ; CHECK-NEXT:    vzeroupper
 ; CHECK-NEXT:    retq
 ;
 ; ADL-LABEL: movmskps_noneof_bitcast_v4f64:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vcmpeqpd %ymm1, %ymm0, %ymm0
-; ADL-NEXT:    vtestpd %ymm0, %ymm0
+; ADL-NEXT:    vmovmskpd %ymm0, %eax
+; ADL-NEXT:    testl %eax, %eax
 ; ADL-NEXT:    sete %al
 ; ADL-NEXT:    vzeroupper
 ; ADL-NEXT:    retq
   %1 = fcmp oeq <4 x double> %a0, zeroinitializer
   %2 = sext <4 x i1> %1 to <4 x i64>
   %3 = bitcast <4 x i64> %2 to <8 x float>
   %4 = tail call i32 @llvm.x86.avx.movmsk.ps.256(<8 x float> %3)
   %5 = icmp eq i32 %4, 0

 ; AVX2-NEXT:    setb %al
 ; AVX2-NEXT:    vzeroupper
 ; AVX2-NEXT:    retq
 ;
 ; ADL-LABEL: movmskps_allof_bitcast_v4f64:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vcmpeqpd %ymm1, %ymm0, %ymm0
-; ADL-NEXT:    vpcmpeqd %ymm1, %ymm1, %ymm1
-; ADL-NEXT:    vtestpd %ymm1, %ymm0
-; ADL-NEXT:    setb %al
+; ADL-NEXT:    vmovmskpd %ymm0, %eax
+; ADL-NEXT:    cmpl $15, %eax
+; ADL-NEXT:    sete %al
 ; ADL-NEXT:    vzeroupper
 ; ADL-NEXT:    retq
   %1 = fcmp oeq <4 x double> %a0, zeroinitializer
   %2 = sext <4 x i1> %1 to <4 x i64>
   %3 = bitcast <4 x i64> %2 to <8 x float>
   %4 = tail call i32 @llvm.x86.avx.movmsk.ps.256(<8 x float> %3)
   %5 = icmp eq i32 %4, 255
   ret i1 %5

 ; CHECK-NEXT:    vtestps %xmm0, %xmm0
 ; CHECK-NEXT:    setne %al
 ; CHECK-NEXT:    negl %eax
 ; CHECK-NEXT:    retq
 ;
 ; ADL-LABEL: movmskps_concat_v4f32:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vorps %xmm1, %xmm0, %xmm0
+; ADL-NEXT:    vmovmskps %xmm0, %ecx
 ; ADL-NEXT:    xorl %eax, %eax
-; ADL-NEXT:    vtestps %xmm0, %xmm0
-; ADL-NEXT:    setne %al
-; ADL-NEXT:    negl %eax
+; ADL-NEXT:    negl %ecx
+; ADL-NEXT:    sbbl %eax, %eax
 ; ADL-NEXT:    retq
   %1 = shufflevector <4 x float> %a0, <4 x float> %a1, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
   %2 = tail call i32 @llvm.x86.avx.movmsk.ps.256(<8 x float> %1)
   %3 = icmp ne i32 %2, 0
   %4 = sext i1 %3 to i32
   ret i32 %4
 }
 

llvm/test/CodeGen/X86/combine-movmsk.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefixes=SSE,SSE2
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.2 | FileCheck %s --check-prefixes=SSE,SSE42
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefix=AVX
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefix=AVX
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mcpu=alderlake | FileCheck %s --check-prefixes=ADL
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2,+prefer-movmsk-over-vtest | FileCheck %s --check-prefixes=ADL
 
 declare i32 @llvm.x86.sse.movmsk.ps(<4 x float>)
 declare i32 @llvm.x86.sse2.movmsk.pd(<2 x double>)
 declare i32 @llvm.x86.sse2.pmovmskb.128(<16 x i8>)
 
 ; Use widest possible vector for movmsk comparisons (PR37087)
 
 define i1 @movmskps_noneof_bitcast_v2f64(<2 x double> %a0) {

 ; AVX-NEXT:    vtestpd %xmm0, %xmm0
 ; AVX-NEXT:    sete %al
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: movmskps_noneof_bitcast_v2f64:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vcmpeqpd %xmm0, %xmm1, %xmm0
-; ADL-NEXT:    vtestpd %xmm0, %xmm0
+; ADL-NEXT:    vmovmskpd %xmm0, %eax
+; ADL-NEXT:    testl %eax, %eax
 ; ADL-NEXT:    sete %al
 ; ADL-NEXT:    retq
   %1 = fcmp oeq <2 x double> zeroinitializer, %a0
   %2 = sext <2 x i1> %1 to <2 x i64>
   %3 = bitcast <2 x i64> %2 to <4 x float>
   %4 = tail call i32 @llvm.x86.sse.movmsk.ps(<4 x float> %3)
   %5 = icmp eq i32 %4, 0
   ret i1 %5

 ; AVX-NEXT:    vtestpd %xmm1, %xmm0
 ; AVX-NEXT:    setb %al
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: movmskps_allof_bitcast_v2f64:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vxorpd %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vcmpeqpd %xmm0, %xmm1, %xmm0
-; ADL-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; ADL-NEXT:    vtestpd %xmm1, %xmm0
-; ADL-NEXT:    setb %al
+; ADL-NEXT:    vmovmskpd %xmm0, %eax
+; ADL-NEXT:    cmpl $3, %eax
+; ADL-NEXT:    sete %al
 ; ADL-NEXT:    retq
   %1 = fcmp oeq <2 x double> zeroinitializer, %a0
   %2 = sext <2 x i1> %1 to <2 x i64>
   %3 = bitcast <2 x i64> %2 to <4 x float>
   %4 = tail call i32 @llvm.x86.sse.movmsk.ps(<4 x float> %3)
   %5 = icmp eq i32 %4, 15
   ret i1 %5
 }

 ; AVX-LABEL: pmovmskb_noneof_bitcast_v2i64:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vtestpd %xmm0, %xmm0
 ; AVX-NEXT:    sete %al
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: pmovmskb_noneof_bitcast_v2i64:
 ; ADL:       # %bb.0:
-; ADL-NEXT:    vtestpd %xmm0, %xmm0
+; ADL-NEXT:    vmovmskpd %xmm0, %eax
+; ADL-NEXT:    testl %eax, %eax
 ; ADL-NEXT:    sete %al
 ; ADL-NEXT:    retq
   %1 = icmp sgt <2 x i64> zeroinitializer, %a0
   %2 = sext <2 x i1> %1 to <2 x i64>
   %3 = bitcast <2 x i64> %2 to <16 x i8>
   %4 = tail call i32 @llvm.x86.sse2.pmovmskb.128(<16 x i8> %3)
   %5 = icmp eq i32 %4, 0
   ret i1 %5

 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
 ; AVX-NEXT:    vtestpd %xmm1, %xmm0
 ; AVX-NEXT:    setb %al
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: pmovmskb_allof_bitcast_v2i64:
 ; ADL:       # %bb.0:
-; ADL-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; ADL-NEXT:    vtestpd %xmm1, %xmm0
-; ADL-NEXT:    setb %al
+; ADL-NEXT:    vmovmskpd %xmm0, %eax
+; ADL-NEXT:    cmpl $3, %eax
+; ADL-NEXT:    sete %al
 ; ADL-NEXT:    retq
   %1 = icmp sgt <2 x i64> zeroinitializer, %a0
   %2 = sext <2 x i1> %1 to <2 x i64>
   %3 = bitcast <2 x i64> %2 to <16 x i8>
   %4 = tail call i32 @llvm.x86.sse2.pmovmskb.128(<16 x i8> %3)
   %5 = icmp eq i32 %4, 65535
   ret i1 %5
 }

 ; AVX-NEXT:    vtestps %xmm0, %xmm0
 ; AVX-NEXT:    sete %al
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: pmovmskb_noneof_bitcast_v4f32:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vcmpeqps %xmm1, %xmm0, %xmm0
-; ADL-NEXT:    vtestps %xmm0, %xmm0
+; ADL-NEXT:    vmovmskps %xmm0, %eax
+; ADL-NEXT:    testl %eax, %eax
 ; ADL-NEXT:    sete %al
 ; ADL-NEXT:    retq
   %1 = fcmp oeq <4 x float> %a0, zeroinitializer
   %2 = sext <4 x i1> %1 to <4 x i32>
   %3 = bitcast <4 x i32> %2 to <16 x i8>
   %4 = tail call i32 @llvm.x86.sse2.pmovmskb.128(<16 x i8> %3)
   %5 = icmp eq i32 %4, 0
   ret i1 %5

 ; AVX-NEXT:    vtestps %xmm1, %xmm0
 ; AVX-NEXT:    setb %al
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: pmovmskb_allof_bitcast_v4f32:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vxorps %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vcmpeqps %xmm1, %xmm0, %xmm0
-; ADL-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
-; ADL-NEXT:    vtestps %xmm1, %xmm0
-; ADL-NEXT:    setb %al
+; ADL-NEXT:    vmovmskps %xmm0, %eax
+; ADL-NEXT:    cmpl $15, %eax
+; ADL-NEXT:    sete %al
 ; ADL-NEXT:    retq
   %1 = fcmp oeq <4 x float> %a0, zeroinitializer
   %2 = sext <4 x i1> %1 to <4 x i32>
   %3 = bitcast <4 x i32> %2 to <16 x i8>
   %4 = tail call i32 @llvm.x86.sse2.pmovmskb.128(<16 x i8> %3)
   %5 = icmp eq i32 %4, 65535
   ret i1 %5
 }

 ; AVX-NEXT:    vtestps %xmm1, %xmm0
 ; AVX-NEXT:    sbbl %eax, %eax
 ; AVX-NEXT:    retq
 ;
 ; ADL-LABEL: movmskps_ptest_numelts_mismatch:
 ; ADL:       # %bb.0:
 ; ADL-NEXT:    vpxor %xmm1, %xmm1, %xmm1
 ; ADL-NEXT:    vpcmpeqb %xmm1, %xmm0, %xmm0
-; ADL-NEXT:    vpcmpeqd %xmm1, %xmm1, %xmm1
+; ADL-NEXT:    vmovmskps %xmm0, %ecx
 ; ADL-NEXT:    xorl %eax, %eax
-; ADL-NEXT:    vtestps %xmm1, %xmm0
-; ADL-NEXT:    sbbl %eax, %eax
+; ADL-NEXT:    cmpl $15, %ecx
+; ADL-NEXT:    sete %al
+; ADL-NEXT:    negl %eax
 ; ADL-NEXT:    retq
   %1 = icmp eq <16 x i8> %a0, zeroinitializer
   %2 = sext <16 x i1> %1 to <16 x i8>
   %3 = bitcast <16 x i8> %2 to <4 x float>
   %4 = tail call i32 @llvm.x86.sse.movmsk.ps(<4 x float> %3)
   %5 = icmp eq i32 %4, 15
   %6 = sext i1 %5 to i32
   ret i32 %6
 }