[X86] Support arch=x86-64{,-v2,-v3,-v4} for target_clones attribute

[MaskRay]

Summary

GCC 12 (https://gcc.gnu.org/PR101696) allows arch=x86-64
arch=x86-64-v2 arch=x86-64-v3 arch=x86-64-v4 in the
target_clones function attribute. This patch ports the feature.

• Set KeyFeature to x86-64{,-v2,-v3,-v4} in Processors[], to be used by X86TargetInfo::multiVersionSortPriority
• builtins: change __cpu_features2 to an array like libgcc. Define FEATURE_X86_64_{BASELINE,V2,V3,V4} and depended ISA feature bits.
• CGBuiltin.cpp: update EmitX86CpuSupports to handle arch=x86-64*.

Close https://github.com/llvm/llvm-project/issues/55830

[erichkeane]

clang/lib/CodeGen/CGBuiltin.cpp

13334

Can this be a std::array instead? The C array is pretty meaningless here.

13371

Why is this in the loop? Is there a good reason to not just always initialize it?

clang/lib/CodeGen/CodeGenFunction.cpp

2688

Can you clarify the comments here on these? I'm a little confused by their meaning.

[pengfei]

clang/lib/CodeGen/CGBuiltin.cpp

13329

Use Lo_32(Mask), Hi_32(Mask)?

13375–13377

Do we have problem when linking with old version of libgcc?

compiler-rt/lib/builtins/cpu_model.c

810

Can we reuse bit defination in cpuid.h, e.g., if (ECX & bit_LAHF_LM)? The same below.

810

Can we reuse bit defination in cpuid.h, e.g., if (ECX & bit_LAHF_LM)? The same below.

[MaskRay]

address comments

clang/lib/CodeGen/CGBuiltin.cpp

13329

Lo_32/Hi_32 are currently unused in clang/. Using this needs MathExtras.h. I am unsure it's worthwhile..

13375–13377

If we don't use __cpu_features2[1..3], no problem with older libgcc.

The way GCC upgraded __cpu_features2 to an array is compatible with scalar __cpu_features2.

compiler-rt/lib/builtins/cpu_model.c

810

Thanks for the cpuid.h idea, but I think likely no. MSVC does not provide cpuid.h. It would be nice if we don't allow MSVC to build the file (we can say that LLVM_ENABLE_PROJECTS='...compiler-rt' is not allowed and LLVM_ENABLE_RUNTIMES or Clang should be used, but I think we are not there yet).

Unfortunately I am also unsure whether anyone uses MSVC to build this file, but there are Windows related macros like FIXME: For MSVC, we should make a function pointer global in in this file...

[erichkeane]

Cant help with the RT/LLVM parts, but the clang parts are pretty much right.

clang/lib/CodeGen/CGBuiltin.cpp

13329

I was previously using Lo_32/Hi_32 here (see the removed parts of EmitX86CpuSupports). Were we using those transitively?

13334

Hmm... I guess size-wise this is on the edge of "const ref vs pass by value". I think its fine now, but 'next time' this grows we'll have to think about making this a const-ref.

13367

This won't double-create this if used more than 1x, right? There doesn't need to be something like GetOrCreate... here?

[MaskRay]

Use Lo_32/Hi_32

clang/lib/CodeGen/CGBuiltin.cpp

13334

Yes. Right now passing 2 registers on a 64-bit target is more efficient.
'next time' may be quite a while from now:)

13367

CodeGenModule::CreateRuntimeVariable calls GetOrCreate internally. It's fine to be called multiple times.

[pengfei]

clang/lib/CodeGen/CGBuiltin.cpp

13375–13377

I assume GCC seldom links to older libgcc, but there's no guarantee Clang can link to latest libgcc.
And we cannot assume user won't use it once the method provided.
I don't have a good idea for it, but I think we should write the requirement done in release note or somewhere if we cannot find a better way.

[FreddyYe]

compiler-rt/lib/builtins/cpu_model.c

167

= 57 redundant

[MaskRay]

clang/lib/CodeGen/CGBuiltin.cpp

13375–13377

The way we treat __cpu_features2 as an array is compatible with older libgcc providing scalar __cpu_features2, as long as we don't access the high bits (FEATURE_X86_64_{BASELINE,V2,V3,V4}).

• When arch=x86-64 is used, this patch changes a compiler crash to (if newer libgcc, work; if older libgcc, runtime crash)
• When arch=x86-64* is unused, the same as before.

compiler-rt/lib/builtins/cpu_model.c

167

Thanks. Minor issue, I've fixed it locally and will only upload a new diff if other things need changing.

[MaskRay]

clang/lib/CodeGen/CGBuiltin.cpp

13375–13377

I think a release note isn't really necessary as the compatibility story with this patch is strictly better than the current state.

[FreddyYe]

clang/lib/CodeGen/CGBuiltin.cpp

13375

Will function multi version also be fixed in this patch? https://gcc.godbolt.org/z/cafhs9qbG If so, need to add test in clang/test/CodeGen/attr-target-mv.c

[FreddyYe]

clang/lib/CodeGen/CGBuiltin.cpp

13373

Don't quite understand why we need to change from a i32* into <4 x i32>* and the first element here to be always zero.

[MaskRay]

<4 x i32> => <3 x i32> (though no difference at runtime)
remove a redundant = 57

clang/lib/CodeGen/CGBuiltin.cpp

13373

Sorry, this global variable (array) should be <3 x i32>. Fixed.

This (strange) scheme is for GCC/libgcc compatibility: bits 0~31 are in __cpu_model.__cpufeatures[0] while bits 32~max are in __cpu_features2[0..2]

13375

The target attribute has strange semantics for overloading:

int __attribute__((target("arch=skylake"))) foo(void) {return 0;}
int __attribute__((target("arch=x86-64"))) foo(void) {return 1;}

is not allowed in C mode of GCC.

I think such use cases are not recommended in C++.

If we don't use overloading, int __attribute__((target("arch=x86-64"))) foo(void) {return 1;} is supported by Clang today and this patch does not intend to change anything about it.

[FreddyYe]

clang/lib/CodeGen/CGBuiltin.cpp

13375

I think behavior for target attribute is not only overloading but also function multiversioning for redefined functions. And seems like C model of gcc haven't supported is due to it will target C23 standard: https://clang.llvm.org/docs/AttributeReference.html#target
Comparing to gcc, clang misses not only target attribute but also other cpuname/feature related features for "x86-64*". See https://gcc.godbolt.org/z/arhne35GG (Seems gcc defined x86-64* as not only cpu name but also feature name.) Anyway, this patch is targeting for target_clones attribute only. So no problem here.

[MaskRay]

clang/lib/CodeGen/CGBuiltin.cpp

13375

So no problem here.

Right. Not supporting C and requiring duplicate definitions makes target not really useful.
https://gcc.gnu.org/pipermail/gcc-patches/2015-October/430585.html Jeff Law said: "I wasn't aware that multi-versioning was only implemented for C++, that seems fairly lame. I hope I didn't approve that :-)"

[MaskRay]

(I think there is no action item on my side and this patch is good for review)

[pengfei]

LGTM.

clang/lib/CodeGen/CGBuiltin.cpp

 #include "llvm/IR/IntrinsicsRISCV.h"
 #include "llvm/IR/IntrinsicsS390.h"
 #include "llvm/IR/IntrinsicsVE.h"
 #include "llvm/IR/IntrinsicsWebAssembly.h"
 #include "llvm/IR/IntrinsicsX86.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/IR/MatrixBuilder.h"
 #include "llvm/Support/ConvertUTF.h"
+#include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/TargetParser/AArch64TargetParser.h"
 #include "llvm/TargetParser/X86TargetParser.h"
 #include <optional>
 #include <sstream>
 
 using namespace clang;
 using namespace CodeGen;

 
 Value *CodeGenFunction::EmitX86CpuSupports(const CallExpr *E) {
   const Expr *FeatureExpr = E->getArg(0)->IgnoreParenCasts();
   StringRef FeatureStr = cast<StringLiteral>(FeatureExpr)->getString();
   return EmitX86CpuSupports(FeatureStr);
 }
 
 Value *CodeGenFunction::EmitX86CpuSupports(ArrayRef<StringRef> FeatureStrs) {
-  return EmitX86CpuSupports(llvm::X86::getCpuSupportsMask(FeatureStrs));
+  uint64_t Mask = llvm::X86::getCpuSupportsMask(FeatureStrs);
+  std::array<uint32_t, 4> FeatureMask{Lo_32(Mask), Hi_32(Mask), 0, 0};
+  return EmitX86CpuSupports(FeatureMask);
 }
 
-llvm::Value *CodeGenFunction::EmitX86CpuSupports(uint64_t FeaturesMask) {
-  uint32_t Features1 = Lo_32(FeaturesMask);
-  uint32_t Features2 = Hi_32(FeaturesMask);
-
+llvm::Value *
+CodeGenFunction::EmitX86CpuSupports(std::array<uint32_t, 4> FeatureMask) {
   Value *Result = Builder.getTrue();
-
-  if (Features1 != 0) {
+  if (FeatureMask[0] != 0) {
     // Matching the struct layout from the compiler-rt/libgcc structure that is
     // filled in:
     // unsigned int __cpu_vendor;
     // unsigned int __cpu_type;
     // unsigned int __cpu_subtype;
     // unsigned int __cpu_features[1];
     llvm::Type *STy = llvm::StructType::get(Int32Ty, Int32Ty, Int32Ty,
                                             llvm::ArrayType::get(Int32Ty, 1));
 
     // Grab the global __cpu_model.
     llvm::Constant *CpuModel = CGM.CreateRuntimeVariable(STy, "__cpu_model");
     cast<llvm::GlobalValue>(CpuModel)->setDSOLocal(true);
 
     // Grab the first (0th) element from the field __cpu_features off of the
     // global in the struct STy.
     Value *Idxs[] = {Builder.getInt32(0), Builder.getInt32(3),
                      Builder.getInt32(0)};
     Value *CpuFeatures = Builder.CreateGEP(STy, CpuModel, Idxs);
     Value *Features = Builder.CreateAlignedLoad(Int32Ty, CpuFeatures,
                                                 CharUnits::fromQuantity(4));
 
     // Check the value of the bit corresponding to the feature requested.
-    Value *Mask = Builder.getInt32(Features1);
+    Value *Mask = Builder.getInt32(FeatureMask[0]);
     Value *Bitset = Builder.CreateAnd(Features, Mask);
     Value *Cmp = Builder.CreateICmpEQ(Bitset, Mask);
     Result = Builder.CreateAnd(Result, Cmp);
   }
 
-  if (Features2 != 0) {
-    llvm::Constant *CpuFeatures2 = CGM.CreateRuntimeVariable(Int32Ty,
-                                                             "__cpu_features2");
-    cast<llvm::GlobalValue>(CpuFeatures2)->setDSOLocal(true);
-
-    Value *Features = Builder.CreateAlignedLoad(Int32Ty, CpuFeatures2,
-                                                CharUnits::fromQuantity(4));
-
+  llvm::Type *ATy = llvm::ArrayType::get(Int32Ty, 3);
+  llvm::Constant *CpuFeatures2 =
+      CGM.CreateRuntimeVariable(ATy, "__cpu_features2");
+  for (int i = 1; i != 4; ++i) {
+    const uint32_t M = FeatureMask[i];
+    if (!M)
+      continue;
+    Value *Idxs[] = {Builder.getInt32(0), Builder.getInt32(i - 1)};
+    Value *Features = Builder.CreateAlignedLoad(
+        Int32Ty, Builder.CreateGEP(ATy, CpuFeatures2, Idxs),
     // Check the value of the bit corresponding to the feature requested.
-    Value *Mask = Builder.getInt32(Features2);
+    Value *Mask = Builder.getInt32(M);
     Value *Bitset = Builder.CreateAnd(Features, Mask);
     Value *Cmp = Builder.CreateICmpEQ(Bitset, Mask);
     Result = Builder.CreateAnd(Result, Cmp);
   }
 
   return Result;
 }
 

clang/lib/CodeGen/CodeGenFunction.h

private:
 
   void AddObjCARCExceptionMetadata(llvm::Instruction *Inst);
 
   llvm::Value *GetValueForARMHint(unsigned BuiltinID);
   llvm::Value *EmitX86CpuIs(const CallExpr *E);
   llvm::Value *EmitX86CpuIs(StringRef CPUStr);
   llvm::Value *EmitX86CpuSupports(const CallExpr *E);
   llvm::Value *EmitX86CpuSupports(ArrayRef<StringRef> FeatureStrs);
-  llvm::Value *EmitX86CpuSupports(uint64_t Mask);
+  llvm::Value *EmitX86CpuSupports(std::array<uint32_t, 4> FeatureMask);
   llvm::Value *EmitX86CpuInit();
   llvm::Value *FormX86ResolverCondition(const MultiVersionResolverOption &RO);
   llvm::Value *EmitAArch64CpuInit();
   llvm::Value *
   FormAArch64ResolverCondition(const MultiVersionResolverOption &RO);
   llvm::Value *EmitAArch64CpuSupports(ArrayRef<StringRef> FeatureStrs);
 };
 

clang/lib/CodeGen/CodeGenFunction.cpp

llvm::Value *CodeGenFunction::FormAArch64ResolverCondition(
   }
   return nullptr;
 }
 
 llvm::Value *CodeGenFunction::FormX86ResolverCondition(
     const MultiVersionResolverOption &RO) {
   llvm::Value *Condition = nullptr;
 
-  if (!RO.Conditions.Architecture.empty())
-    Condition = EmitX86CpuIs(RO.Conditions.Architecture);
+  if (!RO.Conditions.Architecture.empty()) {
+    StringRef Arch = RO.Conditions.Architecture;
+    std::array<uint32_t, 4> Mask{};
+    // If arch= specifies an x86-64 micro-architecture level, test a special
+    // feature named FEATURE_X86_64_*, otherwise we use __builtin_cpu_is.
+    if (Arch.consume_front("x86-64")) {
+      if (Arch.empty()) // FEATURE_X86_64_BASELINE 95=2*32+31
+        Mask[2] = 1u << 31;
+      else if (Arch == "-v2") // FEATURE_X86_64_V2 96==3*32+0
+        Mask[3] = 1u << 0;
+      else if (Arch == "-v3") // FEATURE_X86_64_V3 97==3*32+1
+        Mask[3] = 1u << 1;
+      else if (Arch == "-v4") // FEATURE_X86_64_V3 98==3*32+2
+        Mask[3] = 1u << 2;
+      else
+        llvm_unreachable("invalid x86-64 micro-architecture level");
+      Condition = EmitX86CpuSupports(Mask);
+    } else {
+      Condition = EmitX86CpuIs(Arch);
+    }
+  }
 
   if (!RO.Conditions.Features.empty()) {
     llvm::Value *FeatureCond = EmitX86CpuSupports(RO.Conditions.Features);
     Condition =
         Condition ? Builder.CreateAnd(Condition, FeatureCond) : FeatureCond;
   }
   return Condition;
 }

clang/test/CodeGen/attr-target-clones.c

 // RUN: %clang_cc1 -triple x86_64-linux-gnu -emit-llvm %s -o - | FileCheck %s --check-prefixes=LINUX,CHECK
 // RUN: %clang_cc1 -triple x86_64-windows-pc -emit-llvm %s -o - | FileCheck %s --check-prefixes=WINDOWS,CHECK
 
 // LINUX: $foo.resolver = comdat any
 // LINUX: $foo_dupes.resolver = comdat any
 // LINUX: $unused.resolver = comdat any
 // LINUX: $foo_inline.resolver = comdat any
 // LINUX: $foo_inline2.resolver = comdat any
 
 // WINDOWS: $foo = comdat any
 // WINDOWS: $foo_dupes = comdat any
 // WINDOWS: $unused = comdat any
 // WINDOWS: $foo_inline = comdat any
 // WINDOWS: $foo_inline2 = comdat any
 
+// LINUX: @__cpu_model = external dso_local global { i32, i32, i32, [1 x i32] }
+// LINUX: @__cpu_features2 = external dso_local global [3 x i32]
+
 // LINUX: @foo.ifunc = weak_odr ifunc i32 (), ptr @foo.resolver
 // LINUX: @foo_dupes.ifunc = weak_odr ifunc void (), ptr @foo_dupes.resolver
 // LINUX: @unused.ifunc = weak_odr ifunc void (), ptr @unused.resolver
 // LINUX: @foo_inline.ifunc = weak_odr ifunc i32 (), ptr @foo_inline.resolver
 // LINUX: @foo_inline2.ifunc = weak_odr ifunc i32 (), ptr @foo_inline2.resolver
 // LINUX: @foo_used_no_defn.ifunc = weak_odr ifunc i32 (), ptr @foo_used_no_defn.resolver
 
 int __attribute__((target_clones("sse4.2, default"))) foo(void) { return 0; }

 // LINUX: define weak_odr ptr @foo_used_no_defn.resolver() comdat
 // LINUX: ret ptr @foo_used_no_defn.sse4.2.0
 // LINUX: ret ptr @foo_used_no_defn.default.1
 
 // WINDOWS: define weak_odr dso_local i32 @foo_used_no_defn() comdat
 // WINDOWS: musttail call i32 @foo_used_no_defn.sse4.2.0
 // WINDOWS: musttail call i32 @foo_used_no_defn.default.1
 
+__attribute__((target_clones("default", "arch=x86-64", "arch=x86-64-v2", "arch=x86-64-v3", "arch=x86-64-v4")))
+int isa_level(int) { return 0; }
+// LINUX:      define{{.*}} i32 @isa_level.default.4(
+// LINUX:      define{{.*}} i32 @isa_level.arch_x86-64.0(
+// LINUX:      define{{.*}} i32 @isa_level.arch_x86-64-v2.1(
+// LINUX:      define{{.*}} i32 @isa_level.arch_x86-64-v3.2(
+// LINUX:      define{{.*}} i32 @isa_level.arch_x86-64-v4.3(
+// LINUX:      define weak_odr ptr @isa_level.resolver() comdat
+// LINUX:        call void @__cpu_indicator_init()
+// LINUX-NEXT:   load i32, ptr getelementptr inbounds ([3 x i32], ptr @__cpu_features2, i32 0, i32 2)
+// LINUX-NEXT:   and i32 %[[#]], 4
+// LINUX:        ret ptr @isa_level.arch_x86-64-v4.3
+// LINUX:        load i32, ptr getelementptr inbounds ([3 x i32], ptr @__cpu_features2, i32 0, i32 2)
+// LINUX-NEXT:   and i32 %[[#]], 2
+// LINUX:        ret ptr @isa_level.arch_x86-64-v3.2
+// LINUX:        load i32, ptr getelementptr inbounds ([3 x i32], ptr @__cpu_features2, i32 0, i32 2)
+// LINUX-NEXT:   and i32 %[[#]], 1
+// LINUX:        ret ptr @isa_level.arch_x86-64-v2.1
+// LINUX:        load i32, ptr getelementptr inbounds ([3 x i32], ptr @__cpu_features2, i32 0, i32 1)
+// LINUX-NEXT:   and i32 %[[#]], -2147483648
+// LINUX:        ret ptr @isa_level.arch_x86-64.0
+// LINUX:        ret ptr @isa_level.default.4
 
 // Deferred emission of inline definitions.
 
 // LINUX: define linkonce i32 @foo_inline.arch_sandybridge.0() #[[SB:[0-9]+]]
 // LINUX: define linkonce i32 @foo_inline.default.2() #[[DEF:[0-9]+]]
 // LINUX: define linkonce i32 @foo_inline.sse4.2.1() #[[SSE42:[0-9]+]]
 
 // WINDOWS: define linkonce_odr dso_local i32 @foo_inline.arch_sandybridge.0() #[[SB:[0-9]+]]

clang/test/CodeGen/builtin-cpu-supports.c

 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -emit-llvm < %s| FileCheck %s
 
 // Test that we have the structure definition, the gep offsets, the name of the
 // global, the bit grab, and the icmp correct.
 extern void a(const char *);
 
 // CHECK: @__cpu_model = external dso_local global { i32, i32, i32, [1 x i32] }
-// CHECK: @__cpu_features2 = external dso_local global i32
+// CHECK: @__cpu_features2 = external dso_local global [3 x i32]
 
 int main(void) {
   __builtin_cpu_init();
 
   // CHECK: call void @__cpu_indicator_init
 
   if (__builtin_cpu_supports("sse4.2"))
     a("sse4.2");

clang/test/Sema/attr-target-clones.c

 
 void good_overload4(void) __attribute__((target_clones("mmx", "sse4.2", "default")));
 // expected-error@+1 {{attribute 'target_clones' multiversioning cannot be combined with attribute 'overloadable'}}
 void __attribute__((__overloadable__)) good_overload4(int) __attribute__((target_clones("mmx", "sse4.2", "default")));
 
 // expected-error@+1 {{attribute 'target_clones' multiversioning cannot be combined with attribute 'overloadable'}}
 void __attribute__((__overloadable__)) good_overload5(void) __attribute__((target_clones("mmx", "sse4.2", "default")));
 void good_overload5(int) __attribute__((target_clones("mmx", "sse4.2", "default")));
+
+void good_isa_level(int) __attribute__((target_clones("default", "arch=x86-64", "arch=x86-64-v2", "arch=x86-64-v3", "arch=x86-64-v4")));
+// expected-warning@+1 {{unsupported CPU 'x86-64-v5' in the 'target_clones' attribute string; 'target_clones' attribute ignored}}
+void bad_isa_level(int) __attribute__((target_clones("default", "arch=x86-64-v5")));

compiler-rt/lib/builtins/cpu_model.c

enum ProcessorFeatures {
   FEATURE_AVX512VPOPCNTDQ,
   FEATURE_AVX512VBMI2,
   FEATURE_GFNI,
   FEATURE_VPCLMULQDQ,
   FEATURE_AVX512VNNI,
   FEATURE_AVX512BITALG,
   FEATURE_AVX512BF16,
   FEATURE_AVX512VP2INTERSECT,
+
+  FEATURE_CMPXCHG16B = 46,
+  FEATURE_F16C = 49,
+  FEATURE_LAHF_LM = 54,
+  FEATURE_LM,
+  FEATURE_WP,
+  FEATURE_LZCNT,
+  FEATURE_MOVBE,
+
+  FEATURE_X86_64_BASELINE = 95,
+  FEATURE_X86_64_V2,
+  FEATURE_X86_64_V3,
+  FEATURE_X86_64_V4,
   CPU_FEATURE_MAX
 };
 
 // The check below for i386 was copied from clang's cpuid.h (__get_cpuid_max).
 // Check motivated by bug reports for OpenSSL crashing on CPUs without CPUID
 // support. Consequently, for i386, the presence of CPUID is checked first
 // via the corresponding eflags bit.
 static bool isCpuIdSupported(void) {
getAMDProcessorTypeAndSubtype(unsigned Family, unsigned Model,
 
   return CPU;
 }
 
 static void getAvailableFeatures(unsigned ECX, unsigned EDX, unsigned MaxLeaf,
                                  unsigned *Features) {
   unsigned EAX = 0, EBX = 0;
 
+#define hasFeature(F) ((Features[F / 32] >> (F % 32)) & 1)
 #define setFeature(F)                                                          \
   Features[F / 32] |= 1U << (F % 32)
 
   if ((EDX >> 15) & 1)
     setFeature(FEATURE_CMOV);
   if ((EDX >> 23) & 1)
     setFeature(FEATURE_MMX);
   if ((EDX >> 25) & 1)
     setFeature(FEATURE_SSE);
   if ((EDX >> 26) & 1)
     setFeature(FEATURE_SSE2);
 
   if ((ECX >> 0) & 1)
     setFeature(FEATURE_SSE3);
   if ((ECX >> 1) & 1)
     setFeature(FEATURE_PCLMUL);
   if ((ECX >> 9) & 1)
     setFeature(FEATURE_SSSE3);
   if ((ECX >> 12) & 1)
     setFeature(FEATURE_FMA);
+  if ((ECX >> 13) & 1)
+    setFeature(FEATURE_CMPXCHG16B);
   if ((ECX >> 19) & 1)
     setFeature(FEATURE_SSE4_1);
   if ((ECX >> 20) & 1)
     setFeature(FEATURE_SSE4_2);
+  if ((ECX >> 22) & 1)
+    setFeature(FEATURE_MOVBE);
   if ((ECX >> 23) & 1)
     setFeature(FEATURE_POPCNT);
   if ((ECX >> 25) & 1)
     setFeature(FEATURE_AES);
+  if ((ECX >> 29) & 1)
+    setFeature(FEATURE_F16C);
 
   // If CPUID indicates support for XSAVE, XRESTORE and AVX, and XGETBV
   // indicates that the AVX registers will be saved and restored on context
   // switch, then we have full AVX support.
   const unsigned AVXBits = (1 << 27) | (1 << 28);
   bool HasAVX = ((ECX & AVXBits) == AVXBits) && !getX86XCR0(&EAX, &EDX) &&
                 ((EAX & 0x6) == 0x6);
 #if defined(__APPLE__)
#endif
   if (HasLeaf7Subleaf1 && ((EAX >> 5) & 1) && HasAVX512Save)
     setFeature(FEATURE_AVX512BF16);
 
   unsigned MaxExtLevel;
   getX86CpuIDAndInfo(0x80000000, &MaxExtLevel, &EBX, &ECX, &EDX);
 
   bool HasExtLeaf1 = MaxExtLevel >= 0x80000001 &&
                      !getX86CpuIDAndInfo(0x80000001, &EAX, &EBX, &ECX, &EDX);
-  if (HasExtLeaf1 && ((ECX >> 6) & 1))
+  if (HasExtLeaf1) {
+    if (ECX & 1)
+      setFeature(FEATURE_LAHF_LM);
+    if ((ECX >> 5) & 1)
+      setFeature(FEATURE_LZCNT);
+    if (((ECX >> 6) & 1))
       setFeature(FEATURE_SSE4_A);
-  if (HasExtLeaf1 && ((ECX >> 11) & 1))
+    if (((ECX >> 11) & 1))
       setFeature(FEATURE_XOP);
-  if (HasExtLeaf1 && ((ECX >> 16) & 1))
+    if (((ECX >> 16) & 1))
       setFeature(FEATURE_FMA4);
+    if (((EDX >> 29) & 1))
+      setFeature(FEATURE_LM);
+  }
+
+  if (hasFeature(FEATURE_LM) && hasFeature(FEATURE_SSE2)) {
+    setFeature(FEATURE_X86_64_BASELINE);
+    if (hasFeature(FEATURE_CMPXCHG16B) && hasFeature(FEATURE_POPCNT) &&
+        hasFeature(FEATURE_LAHF_LM) && hasFeature(FEATURE_SSE4_2)) {
+      setFeature(FEATURE_X86_64_V2);
+      if (hasFeature(FEATURE_AVX2) && hasFeature(FEATURE_BMI) &&
+          hasFeature(FEATURE_BMI2) && hasFeature(FEATURE_F16C) &&
+          hasFeature(FEATURE_FMA) && hasFeature(FEATURE_LZCNT) &&
+          hasFeature(FEATURE_MOVBE)) {
+        setFeature(FEATURE_X86_64_V3);
+        if (hasFeature(FEATURE_AVX512BW) && hasFeature(FEATURE_AVX512CD) &&
+            hasFeature(FEATURE_AVX512DQ) && hasFeature(FEATURE_AVX512VL))
+          setFeature(FEATURE_X86_64_V4);
+      }
+    }
+  }
+
+#undef hasFeature
 #undef setFeature
 }
 
 #ifndef _WIN32
 __attribute__((visibility("hidden")))
 #endif
 int __cpu_indicator_init(void) CONSTRUCTOR_ATTRIBUTE;
 
 #ifndef _WIN32
 __attribute__((visibility("hidden")))
 #endif
 struct __processor_model {
   unsigned int __cpu_vendor;
   unsigned int __cpu_type;
   unsigned int __cpu_subtype;
   unsigned int __cpu_features[1];
 } __cpu_model = {0, 0, 0, {0}};
 
 #ifndef _WIN32
 __attribute__((visibility("hidden")))
 #endif
-unsigned int __cpu_features2 = 0;
+unsigned __cpu_features2[(CPU_FEATURE_MAX - 1) / 32];
 
 // A constructor function that is sets __cpu_model and __cpu_features2 with
 // the right values.  This needs to run only once.  This constructor is
 // given the highest priority and it should run before constructors without
 // the priority set.  However, it still runs after ifunc initializers and
 // needs to be called explicitly there.
 
 int CONSTRUCTOR_ATTRIBUTE __cpu_indicator_init(void) {
   unsigned EAX, EBX, ECX, EDX;
   unsigned MaxLeaf = 5;
   unsigned Vendor;
   unsigned Model, Family;
   unsigned Features[(CPU_FEATURE_MAX + 31) / 32] = {0};
+  static_assert(sizeof(Features) / sizeof(Features[0]) == 4, "");
+  static_assert(sizeof(__cpu_features2) / sizeof(__cpu_features2[0]) == 3, "");
 
   // This function needs to run just once.
   if (__cpu_model.__cpu_vendor)
     return 0;
 
   if (!isCpuIdSupported() ||
       getX86CpuIDAndInfo(0, &MaxLeaf, &Vendor, &ECX, &EDX) || MaxLeaf < 1) {
     __cpu_model.__cpu_vendor = VENDOR_OTHER;
     return -1;
   }
 
   getX86CpuIDAndInfo(1, &EAX, &EBX, &ECX, &EDX);
   detectX86FamilyModel(EAX, &Family, &Model);
 
   // Find available features.
   getAvailableFeatures(ECX, EDX, MaxLeaf, &Features[0]);
 
-  assert((sizeof(Features)/sizeof(Features[0])) == 2);
   __cpu_model.__cpu_features[0] = Features[0];
-  __cpu_features2 = Features[1];
+  __cpu_features2[0] = Features[1];
+  __cpu_features2[1] = Features[2];
+  __cpu_features2[2] = Features[3];
 
   if (Vendor == SIG_INTEL) {
     // Get CPU type.
     getIntelProcessorTypeAndSubtype(Family, Model, &Features[0],
                                     &(__cpu_model.__cpu_type),
                                     &(__cpu_model.__cpu_subtype));
     __cpu_model.__cpu_vendor = VENDOR_INTEL;
   } else if (Vendor == SIG_AMD) {

llvm/lib/TargetParser/X86TargetParser.cpp

 // D151696 tranplanted Mangling and OnlyForCPUDispatchSpecific from
 // X86TargetParser.def to here. They are assigned by following ways:
 // 1. Copy the mangling from the original CPU_SPEICIFC MACROs. If no, assign
 // to '\0' by default, which means not support cpu_specific/dispatch feature.
 // 2. set OnlyForCPUDispatchSpecific as true if this cpu name was not
 // listed here before, which means it doesn't support -march, -mtune and so on.
 // FIXME: Remove OnlyForCPUDispatchSpecific after all CPUs here support both
 // cpu_dispatch/specific() feature and -march, -mtune, and so on.
+// clang-format off
 constexpr ProcInfo Processors[] = {
  // Empty processor. Include X87 and CMPXCHG8 for backwards compatibility.
   { {""}, CK_None, ~0U, FeatureX87 | FeatureCMPXCHG8B, '\0', false },
   { {"generic"}, CK_None, ~0U, FeatureX87 | FeatureCMPXCHG8B | Feature64BIT, 'A', true },
   // i386-generation processors.
   { {"i386"}, CK_i386, ~0U, FeatureX87, '\0', false },
   // i486-generation processors.
   { {"i486"}, CK_i486, ~0U, FeatureX87, '\0', false },
// Empty processor. Include X87 and CMPXCHG8 for backwards compatibility.
   { {"bdver3"}, CK_BDVER3, FEATURE_FMA, FeaturesBDVER3, '\0', false },
   { {"bdver4"}, CK_BDVER4, FEATURE_AVX2, FeaturesBDVER4, '\0', false },
   // Zen architecture processors.
   { {"znver1"}, CK_ZNVER1, FEATURE_AVX2, FeaturesZNVER1, '\0', false },
   { {"znver2"}, CK_ZNVER2, FEATURE_AVX2, FeaturesZNVER2, '\0', false },
   { {"znver3"}, CK_ZNVER3, FEATURE_AVX2, FeaturesZNVER3, '\0', false },
   { {"znver4"}, CK_ZNVER4, FEATURE_AVX512VBMI2, FeaturesZNVER4, '\0', false },
   // Generic 64-bit processor.
-  { {"x86-64"}, CK_x86_64, ~0U, FeaturesX86_64, '\0', false },
-  { {"x86-64-v2"}, CK_x86_64_v2, ~0U, FeaturesX86_64_V2, '\0', false },
-  { {"x86-64-v3"}, CK_x86_64_v3, ~0U, FeaturesX86_64_V3, '\0', false },
-  { {"x86-64-v4"}, CK_x86_64_v4, ~0U, FeaturesX86_64_V4, '\0', false },
+  { {"x86-64"}, CK_x86_64, FEATURE_SSE2 , FeaturesX86_64, '\0', false },
+  { {"x86-64-v2"}, CK_x86_64_v2, FEATURE_SSE4_2 , FeaturesX86_64_V2, '\0', false },
+  { {"x86-64-v3"}, CK_x86_64_v3, FEATURE_AVX2, FeaturesX86_64_V3, '\0', false },
+  { {"x86-64-v4"}, CK_x86_64_v4, FEATURE_AVX512VL, FeaturesX86_64_V4, '\0', false },
   // Geode processors.
   { {"geode"}, CK_Geode, ~0U, FeaturesGeode, '\0', false },
 };
+// clang-format on
 
 constexpr const char *NoTuneList[] = {"x86-64-v2", "x86-64-v3", "x86-64-v4"};
 
 X86::CPUKind llvm::X86::parseArchX86(StringRef CPU, bool Only64Bit) {
   for (const auto &P : Processors)
     if (!P.OnlyForCPUDispatchSpecific && P.Name == CPU &&
         (P.Features[FEATURE_64BIT] || !Only64Bit))
       return P.Kind;