Normalize GlobalDecls when used with CPUDispatch

[erichkeane]

Summary

Previously, CPUDispatch functionality did some wacky things with
GlobalDecls, which resulted in multiple GlobalDecls having the same
mangled name. This patch corrects this in a few ways:

First, it Moves the emission of CPUDispatch resolvers to the end of the
translation unit. This is akin to how 'target' multiversioning works,
and permits the resolver emission mechanism to see all possible
FunctionDecls in the TU.

Second, it changes the meaning of the GlobalDecl MultiVersionIndex to
MultiVersionOrdinal. In the case of CPUSpecific, '0' now means "A call
to a non-present CPUDispatch function". Other ordinals are the order of
the CPU Name in the attribute source. In the case of CPUDispatch, '0'
is the resolver function, and other ordinals represent the non-present
CPU-Specific version.

Because of this, the 1:1 relationship between a GlobalDecl and a
Mangled Name is restored. Additionally, it results in functions only
being emitted when necessary, rather than use a variety of hackery to
emit in cases we presume they would be used.

[aaron.ballman]

include/clang/Basic/Attr.td

896–910

gets -> Gets

901

if (II->isStr(Name))

908

Can this return a const IdentifierInfo * instead?

lib/CodeGen/CodeGenModule.cpp

2150

Rather than use hasAttr<> followed by getAttr<>, just get the attribute and test it for null.

Also, const auto *.

2464

const auto *

2584

const auto *

2611

const auto *

2634–2641

const auto *

[erichkeane]

Thanks @aaron.ballman . This should fix all of your complaints.

[rsmith]

Thanks, this makes a lot of sense to me.

lib/CodeGen/CodeGenModule.cpp

907

Missing indentation on this line.

918

Hmm, it looks like we don't have a unique name for a GlobalDecl naming the CPUOrdinal == 0 case of an overloaded cpu_specific function:

__attribute__((cpu_specific(atom)))
void f() {} // CPUOrdinal 0 GD mangled as _Z1fv.ifunc
__attribute__((cpu_specific(haswell)))
void f() {} // CPUOrdinal 0 GD also mangled as _Z1fv.ifunc

Maybe we could append the CPU-specific manglings for all named CPUs before the .ifunc?

2142–2155

Instead of adding a special-case list of MultiVersionFuncs, could we instead change MayBeEmittedEagerly to return false for CPUDispatch functions and use the normal DeferredDecls mechanism for this?

2656–2657

I think the first part of this should now be equivalent to getMangledName(...ImplDecl...), which should means that these 6 lines can be replaced by GetAddrOfFunction(ImplDecl, DeclTy, false, false, NotForDefinition).

(Also I think you should pass NotForDefinition here because you just want to reference the symbol, you don't want to define it yourself.)

2657–2663

This too should just be the same GetAddrOfFunction call, I think.

2659

I think we should have a comment somewhere explaining the subtle use of the multiversion ordinal:

• (Resolver, 0) refers to the resolver itself
• (Resolver, N) refers to an external (outside this module) symbol for the N'th CPU in Resolver's cpu_dispatch list
• (Specific, 0) is a placeholder used to refer to the set of function symbols defined by a cpu_specific function declaration; no corresponding symbol is ever emitted (right?)
• (Specific, N) refers to an internal (within this module) symbol for the N'th CPU in Specific's cpu_specific list

[erichkeane]

lib/CodeGen/CodeGenModule.cpp

918

Well shoot, thats true, since you can call the function at different points and have overload resolution 'choose' a different one (since it is just by ordering).

If we were to give it a different name, we would either have to replace it later, or make sure a call to 'not the ifunc' doesn't happen.

2142–2155

That list already would have to exist for GCC Multiversioning, since there isn't a single declaration that corresponds to the resolver. I could definitely remove CPUDispatch from this mechanism and return it to a separate one, but I'm not sure what that buys us.

Unless you think this is something we can do that will fix the GCC Multiversioning as well?

2656–2657

I remember 'DontDefer=true' being set for a good reason, since we need this function emitted.

At the moment, this doesn't work because GetOrCreateLLVMFunction (called by GetAddressOfFunction) replaces NotForDefinition with the ifunc instead of the function, so calling that here ends up getting the ifunc/resolver instead of the function itself.

I'll continue looking into that, but if you have alternative design decisions, I'd love to hear them.

2659

(Specific,0) is a placeholder for a not-yet-existing CPU-Dispatch version of the function. Otherwise, I think documenting that somewhere is a good idea. I'll look for a good spot for it.

[erichkeane]

I got a couple of @rsmith's requests done. Most importantly I suspect is MultiVersionFuncs, though NotForDefinition vs ForDefinition is perhpas something you'll find important.

lib/CodeGen/CodeGenModule.cpp

918

I thought about it more last night, if the purpose is simply unique-ness, we can generate it as a unique name. The intent is that it should never make it into the IR, so I think appending the names and .ifunc is fine, simply so that it is unique.

2142–2155

I tried for a bit, and couldn't come up with a way that this would work for Target as well. IMO, since MultiVersionFuncs already has to exist, this fits better there.

Additionally, I noticed that when I switched Dispatch over to this version that I was having to emit more functions during resolver emission, since they were in the deferred list. The deferred list then attempted to emit them, but no longer needed to (though no problems were caused by this).

2656–2657

I replaced both of these as you stated, though I DID need to do ForDefinition. I couldn't come up with a not-super-intrusive way to differentiate between "Not for definition, but for resolver use" and "Not for definition, give me the IFunc".

2659

I rephrased this slightly and put it in "EmitMultiVersionFunctionDefinition", since it seems to fit there best IMO.

[erichkeane]

As mentioned, the @rsmith comments that I thought were doable without feedback.

include/clang/AST/GlobalDecl.h

 namespace clang {
 
 /// GlobalDecl - represents a global declaration. This can either be a
 /// CXXConstructorDecl and the constructor type (Base, Complete).
 /// a CXXDestructorDecl and the destructor type (Base, Complete) or
 /// a VarDecl, a FunctionDecl or a BlockDecl.
 class GlobalDecl {
   llvm::PointerIntPair<const Decl *, 2> Value;
-  unsigned MultiVersionIndex = 0;
+  unsigned MultiVersionOrdinal = 0;
 
   void Init(const Decl *D) {
     assert(!isa<CXXConstructorDecl>(D) && "Use other ctor with ctor decls!");
     assert(!isa<CXXDestructorDecl>(D) && "Use other ctor with dtor decls!");
 
     Value.setPointer(D);
   }
 
 public:
   GlobalDecl() = default;
   GlobalDecl(const VarDecl *D) { Init(D);}
-  GlobalDecl(const FunctionDecl *D, unsigned MVIndex = 0)
-      : MultiVersionIndex(MVIndex) {
+  GlobalDecl(const FunctionDecl *D, unsigned MVOrd = 0)
+      : MultiVersionOrdinal(MVOrd) {
     Init(D);
   }
   GlobalDecl(const BlockDecl *D) { Init(D); }
   GlobalDecl(const CapturedDecl *D) { Init(D); }
   GlobalDecl(const ObjCMethodDecl *D) { Init(D); }
   GlobalDecl(const OMPDeclareReductionDecl *D) { Init(D); }
   GlobalDecl(const CXXConstructorDecl *D, CXXCtorType Type) : Value(D, Type) {}
   GlobalDecl(const CXXDestructorDecl *D, CXXDtorType Type) : Value(D, Type) {}
 
   GlobalDecl getCanonicalDecl() const {
     GlobalDecl CanonGD;
     CanonGD.Value.setPointer(Value.getPointer()->getCanonicalDecl());
     CanonGD.Value.setInt(Value.getInt());
-    CanonGD.MultiVersionIndex = MultiVersionIndex;
+    CanonGD.MultiVersionOrdinal = MultiVersionOrdinal;
 
     return CanonGD;
   }
 
   const Decl *getDecl() const { return Value.getPointer(); }
 
   CXXCtorType getCtorType() const {
     assert(isa<CXXConstructorDecl>(getDecl()) && "Decl is not a ctor!");
     return static_cast<CXXCtorType>(Value.getInt());
   }
 
   CXXDtorType getDtorType() const {
     assert(isa<CXXDestructorDecl>(getDecl()) && "Decl is not a dtor!");
     return static_cast<CXXDtorType>(Value.getInt());
   }
 
-  unsigned getMultiVersionIndex() const {
+  unsigned getMultiVersionOrdinal() const {
     assert(isa<FunctionDecl>(getDecl()) &&
            !isa<CXXConstructorDecl>(getDecl()) &&
            !isa<CXXDestructorDecl>(getDecl()) &&
            "Decl is not a plain FunctionDecl!");
-    return MultiVersionIndex;
+    return MultiVersionOrdinal;
   }
 
   friend bool operator==(const GlobalDecl &LHS, const GlobalDecl &RHS) {
     return LHS.Value == RHS.Value &&
-           LHS.MultiVersionIndex == RHS.MultiVersionIndex;
+           LHS.MultiVersionOrdinal== RHS.MultiVersionOrdinal;
   }
 
   void *getAsOpaquePtr() const { return Value.getOpaqueValue(); }
 
   static GlobalDecl getFromOpaquePtr(void *P) {
     GlobalDecl GD;
     GD.Value.setFromOpaqueValue(P);
     return GD;
   }
 
   GlobalDecl getWithDecl(const Decl *D) {
     GlobalDecl Result(*this);
     Result.Value.setPointer(D);
     return Result;
   }
 
-  GlobalDecl getWithMultiVersionIndex(unsigned Index) {
+  GlobalDecl getWithMultiVersionOrdinal(unsigned Ordinal) {
     assert(isa<FunctionDecl>(getDecl()) &&
            !isa<CXXConstructorDecl>(getDecl()) &&
            !isa<CXXDestructorDecl>(getDecl()) &&
            "Decl is not a plain FunctionDecl!");
     GlobalDecl Result(*this);
-    Result.MultiVersionIndex = Index;
+    Result.MultiVersionOrdinal = Ordinal;
     return Result;
   }
 };
 
 } // namespace clang
 
 namespace llvm {
 

include/clang/Basic/Attr.td

 }
 
 def CPUSpecific : InheritableAttr {
   let Spellings = [Clang<"cpu_specific">, Declspec<"cpu_specific">];
   let Args = [VariadicIdentifierArgument<"Cpus">];
   let Subjects = SubjectList<[Function]>;
   let Documentation = [CPUSpecificCPUDispatchDocs];
   let AdditionalMembers = [{
-    IdentifierInfo *getCPUName(unsigned Index) const {
-      return *(cpus_begin() + Index);
+    // Gets the ordinal of the requested CPU name, or 0 if it isn't in the cpu
+    // list.
+    unsigned getCPUNameOrdinal(StringRef Name) const {
+      unsigned Ord = 1;
+      for (const IdentifierInfo *II : cpus()) {
+        if (II->isStr(Name))
+          return Ord;
+        ++Ord;
+      }
+      return 0;
+    }
+
+    const IdentifierInfo *getCPUName(unsigned Ordinal) const {
+      assert(Ordinal > 0 && "Invalid Ordinal");
+      return *(cpus_begin() + Ordinal - 1);
     }
   }];
 }
 
 def CPUDispatch : InheritableAttr {
   let Spellings = [Clang<"cpu_dispatch">, Declspec<"cpu_dispatch">];
   let Args = [VariadicIdentifierArgument<"Cpus">];
   let Subjects = SubjectList<[Function]>;
   let Documentation = [CPUSpecificCPUDispatchDocs];
+  let AdditionalMembers = [{
+    const IdentifierInfo *getCPUName(unsigned Ordinal) const {
+      assert(Ordinal > 0 && "Invalid Ordinal");
+      return *(cpus_begin() + Ordinal - 1);
+    }
+  }];
 }
 
 // CUDA attributes are spelled __attribute__((attr)) or __declspec(__attr__),
 // and they do not receive a [[]] spelling.
 def CUDAConstant : InheritableAttr {
   let Spellings = [GNU<"constant">, Declspec<"__constant__">];
   let Subjects = SubjectList<[Var]>;
   let LangOpts = [CUDA];

lib/CodeGen/CodeGenModule.h

private:
   void EmitGlobalDefinition(GlobalDecl D, llvm::GlobalValue *GV = nullptr);
 
   void EmitGlobalFunctionDefinition(GlobalDecl GD, llvm::GlobalValue *GV);
   void EmitMultiVersionFunctionDefinition(GlobalDecl GD, llvm::GlobalValue *GV);
 
   void EmitGlobalVarDefinition(const VarDecl *D, bool IsTentative = false);
   void EmitAliasDefinition(GlobalDecl GD);
   void emitIFuncDefinition(GlobalDecl GD);
-  void emitCPUDispatchDefinition(GlobalDecl GD);
+  void emitTargetMultiVersionResolver(GlobalDecl GD);
+  llvm::Function *getCPUDispatchResolverFunction(GlobalDecl GD,
+                                                 llvm::Type *&DeclTy);
+  void emitCPUDispatchResolver(GlobalDecl GD);
   void EmitObjCPropertyImplementations(const ObjCImplementationDecl *D);
   void EmitObjCIvarInitializations(ObjCImplementationDecl *D);
 
   // C++ related functions.
 
   void EmitDeclContext(const DeclContext *DC);
   void EmitLinkageSpec(const LinkageSpecDecl *D);
 

lib/CodeGen/CodeGenModule.cpp

 }
 
 static std::string getCPUSpecificMangling(const CodeGenModule &CGM,
                                           StringRef Name) {
   const TargetInfo &Target = CGM.getTarget();
   return (Twine('.') + Twine(Target.CPUSpecificManglingCharacter(Name))).str();
 }
 
-static void AppendCPUSpecificCPUDispatchMangling(const CodeGenModule &CGM,
-                                                 const CPUSpecificAttr *Attr,
-                                                 unsigned CPUIndex,
-                                                 raw_ostream &Out) {
+static void AppendCPUSpecificCPUDispatchMangling(
+    const CodeGenModule &CGM, const CPUSpecificAttr *SpecAttr,
+    const CPUDispatchAttr *DispAttr, unsigned CPUOrdinal, raw_ostream &Out) {
+  assert((!SpecAttr || !DispAttr) && "Cannot be both dispatch and specific");
   // cpu_specific gets the current name, dispatch gets the resolver if IFunc is
   // supported.
-  if (Attr)
-    Out << getCPUSpecificMangling(CGM, Attr->getCPUName(CPUIndex)->getName());
-  else if (CGM.getTarget().supportsIFunc())
+  if (SpecAttr && CPUOrdinal)
+    Out << getCPUSpecificMangling(CGM,
+                                  SpecAttr->getCPUName(CPUOrdinal)->getName());
+  else if (SpecAttr && !CPUOrdinal) {
+    // This name should never be emitted, however it should have a unique name.
+    // In order to accomplish that, we simply append all of the CPUs and .ifunc
+    // to it so it is more easily identified and is unique.
+    for (const IdentifierInfo *II : SpecAttr->cpus())
+      Out << "." << getCPUSpecificMangling(CGM, II->getName());
+    Out << ".ifunc";
+  } else if (CPUOrdinal) {
+    // This should be the materialized version of an externally declared version
+    // of this function.
+    Out << getCPUSpecificMangling(CGM,
+                                  DispAttr->getCPUName(CPUOrdinal)->getName());
+  } else if (CGM.getTarget().supportsIFunc())
     Out << ".resolver";
 }
 
 static void AppendTargetMangling(const CodeGenModule &CGM,
                                  const TargetAttr *Attr, raw_ostream &Out) {
   if (Attr->isDefaultVersion())
     return;
 
   Out << '.';
   const TargetInfo &Target = CGM.getTarget();
static std::string getMangledNameImpl(const CodeGenModule &CGM, GlobalDecl GD,
 
   if (const auto *FD = dyn_cast<FunctionDecl>(ND))
     if (FD->isMultiVersion() && !OmitMultiVersionMangling) {
       switch (FD->getMultiVersionKind()) {
       case MultiVersionKind::CPUDispatch:
       case MultiVersionKind::CPUSpecific:
         AppendCPUSpecificCPUDispatchMangling(CGM,
                                              FD->getAttr<CPUSpecificAttr>(),
-                                             GD.getMultiVersionIndex(), Out);
+                                             FD->getAttr<CPUDispatchAttr>(),
+                                             GD.getMultiVersionOrdinal(), Out);
         break;
       case MultiVersionKind::Target:
         AppendTargetMangling(CGM, FD->getAttr<TargetAttr>(), Out);
         break;
       case MultiVersionKind::None:
         llvm_unreachable("None multiversion type isn't valid here");
       }
     }
void CodeGenModule::EmitGlobal(GlobalDecl GD) {
   // emit it now.
   if (Global->hasAttr<AliasAttr>())
     return EmitAliasDefinition(GD);
 
   // IFunc like an alias whose value is resolved at runtime by calling resolver.
   if (Global->hasAttr<IFuncAttr>())
     return emitIFuncDefinition(GD);
 
-  // If this is a cpu_dispatch multiversion function, emit the resolver.
-  if (Global->hasAttr<CPUDispatchAttr>())
-    return emitCPUDispatchDefinition(GD);
+  // If this is a cpu_dispatch multiversion function, designate it for emission
+  // at the end of the Translation Unit.
+  if (Global->hasAttr<CPUDispatchAttr>()) {
+    MultiVersionFuncs.push_back(GD);
+    return;
+  }
+
+  const auto *SpecAttr = Global->getAttr<CPUSpecificAttr>();
+  if (SpecAttr && GD.getMultiVersionOrdinal() == 0) {
+    for (unsigned I = 1; I <= SpecAttr->cpus_size(); ++I)
+      EmitGlobal(GD.getWithMultiVersionOrdinal(I));
+    return;
+  }
 
   // If this is CUDA, be selective about which declarations we emit.
   if (LangOpts.CUDA) {
     if (LangOpts.CUDAIsDevice) {
       if (!Global->hasAttr<CUDADeviceAttr>() &&
           !Global->hasAttr<CUDAGlobalAttr>() &&
           !Global->hasAttr<CUDAConstantAttr>() &&
           !Global->hasAttr<CUDASharedAttr>())
         return;

 }
 
 bool CodeGenModule::shouldOpportunisticallyEmitVTables() {
   return CodeGenOpts.OptimizationLevel > 0;
 }
 
 void CodeGenModule::EmitMultiVersionFunctionDefinition(GlobalDecl GD,
                                                        llvm::GlobalValue *GV) {
+  // The usage of GlobalDecls containing a CPUDispatch/CPUSpecific function are
+  // a touch subtle. The value of the Ordinal is significant in differentiating
+  // each version. The meaning of each of the 4 combinations are as follows:
+  // (CPDispatchFunc, 0): This refers to the resolver itself.
+  // (CPUDispatchFunc, N): refers to a an externally declared (outside of this
+  //   Module) symbol corresponding to the Nth CPU in the CPUDispatchAttr's CPU
+  //   list.
+  // (CPUSpecificFunc, 0): This is a placeholder used to refer to the collection
+  //   collection of function versions represented by the function declaration.
+  //   These symbols should never be emitted, since usages will be replaced by
+  //   the call to the resolver/ifunc directly.
+  // (CPUSpecificFunc, N): refers to the internal (inside this Module) symbol
+  // for the Nth CPU in the CPUSpecificAttr's CPU list.
   const auto *FD = cast<FunctionDecl>(GD.getDecl());
 
   if (FD->isCPUSpecificMultiVersion()) {
-    auto *Spec = FD->getAttr<CPUSpecificAttr>();
-    for (unsigned I = 0; I < Spec->cpus_size(); ++I)
-      EmitGlobalFunctionDefinition(GD.getWithMultiVersionIndex(I), nullptr);
     // Requires multiple emits.
+    const auto *Spec = FD->getAttr<CPUSpecificAttr>();
+    for (unsigned I = 1; I <= Spec->cpus_size(); ++I)
+      EmitGlobalFunctionDefinition(GD.getWithMultiVersionOrdinal(I), nullptr);
   } else
     EmitGlobalFunctionDefinition(GD, GV);
 }
 
 void CodeGenModule::EmitGlobalDefinition(GlobalDecl GD, llvm::GlobalValue *GV) {
   const auto *D = cast<ValueDecl>(GD.getDecl());
 
   PrettyStackTraceDecl CrashInfo(const_cast<ValueDecl *>(D), D->getLocation(),
for (StringRef Feat : RO.Conditions.Features)
     Priority = std::max(Priority, TI.multiVersionSortPriority(Feat));
 
   if (!RO.Conditions.Architecture.empty())
     Priority = std::max(
         Priority, TI.multiVersionSortPriority(RO.Conditions.Architecture));
   return Priority;
 }
 
-void CodeGenModule::emitMultiVersionFunctions() {
-  for (GlobalDecl GD : MultiVersionFuncs) {
-    SmallVector<CodeGenFunction::MultiVersionResolverOption, 10> Options;
-    const FunctionDecl *FD = cast<FunctionDecl>(GD.getDecl());
-    getContext().forEachMultiversionedFunctionVersion(
-        FD, [this, &GD, &Options](const FunctionDecl *CurFD) {
-          GlobalDecl CurGD{
-              (CurFD->isDefined() ? CurFD->getDefinition() : CurFD)};
-          StringRef MangledName = getMangledName(CurGD);
-          llvm::Constant *Func = GetGlobalValue(MangledName);
-          if (!Func) {
-            if (CurFD->isDefined()) {
-              EmitGlobalFunctionDefinition(CurGD, nullptr);
-              Func = GetGlobalValue(MangledName);
-            } else {
-              const CGFunctionInfo &FI =
-                  getTypes().arrangeGlobalDeclaration(GD);
-              llvm::FunctionType *Ty = getTypes().GetFunctionType(FI);
-              Func = GetAddrOfFunction(CurGD, Ty, /*ForVTable=*/false,
-                                       /*DontDefer=*/false, ForDefinition);
-            }
-            assert(Func && "This should have just been created");
-          }
+void CodeGenModule::emitTargetMultiVersionResolver(GlobalDecl GD) {
+  assert(FD->isTargetMultiVersion() && "Not a target multiversion function?");
+
+        GlobalDecl CurGD{(CurFD->isDefined() ? CurFD->getDefinition() : CurFD)};
+            const CGFunctionInfo &FI = getTypes().arrangeGlobalDeclaration(GD);
 
-          const auto *TA = CurFD->getAttr<TargetAttr>();
-          llvm::SmallVector<StringRef, 8> Feats;
-          TA->getAddedFeatures(Feats);
 
-          Options.emplace_back(cast<llvm::Function>(Func),
-                               TA->getArchitecture(), Feats);
-        });
-
-    llvm::Function *ResolverFunc;
-    const TargetInfo &TI = getTarget();
+        Options.emplace_back(cast<llvm::Function>(Func), TA->getArchitecture(),
+                             Feats);
 
-    if (TI.supportsIFunc() || FD->isTargetMultiVersion())
-      ResolverFunc = cast<llvm::Function>(
-          GetGlobalValue((getMangledName(GD) + ".resolver").str()));
-    else
-      ResolverFunc = cast<llvm::Function>(GetGlobalValue(getMangledName(GD)));
 
-    if (supportsCOMDAT())
-      ResolverFunc->setComdat(
-          getModule().getOrInsertComdat(ResolverFunc->getName()));
 
-    std::stable_sort(
-        Options.begin(), Options.end(),
-        [&TI](const CodeGenFunction::MultiVersionResolverOption &LHS,
-              const CodeGenFunction::MultiVersionResolverOption &RHS) {
-          return TargetMVPriority(TI, LHS) > TargetMVPriority(TI, RHS);
-        });
-    CodeGenFunction CGF(*this);
-    CGF.EmitMultiVersionResolver(ResolverFunc, Options);
-  }
+
+void CodeGenModule::emitMultiVersionFunctions() {
+  for (GlobalDecl GD : MultiVersionFuncs) {
+    const auto *FD = cast<FunctionDecl>(GD.getDecl());
+    assert(FD && FD->isMultiVersion() && "Not a multiversion function?");
+
+    switch (FD->getMultiVersionKind()) {
+    case MultiVersionKind::Target:
+      emitTargetMultiVersionResolver(GD);
+      break;
+    case MultiVersionKind::CPUDispatch:
+      emitCPUDispatchResolver(GD);
+      break;
+    case MultiVersionKind::CPUSpecific:
+    case MultiVersionKind::None:
+      llvm_unreachable("Function does not cause a multiversion resolver.");
+    }
+  }
 }
 
-void CodeGenModule::emitCPUDispatchDefinition(GlobalDecl GD) {
+// Helper function to get the llvm::Function object for the resolver so that
+// it can be filled in.
+llvm::Function *
+CodeGenModule::getCPUDispatchResolverFunction(GlobalDecl GD,
+                                              llvm::Type *&DeclTy) {
   const auto *FD = cast<FunctionDecl>(GD.getDecl());
-  assert(FD && "Not a FunctionDecl?");
-  const auto *DD = FD->getAttr<CPUDispatchAttr>();
-  assert(DD && "Not a cpu_dispatch Function?");
   QualType CanonTy = Context.getCanonicalType(FD->getType());
-  llvm::Type *DeclTy = getTypes().ConvertFunctionType(CanonTy, FD);
-
+  DeclTy = getTypes().ConvertFunctionType(CanonTy, FD);
   if (const auto *CXXFD = dyn_cast<CXXMethodDecl>(FD)) {
-    const CGFunctionInfo &FInfo = getTypes().arrangeCXXMethodDeclaration(CXXFD);
+    const CGFunctionInfo &FInfo =
+        getTypes().arrangeCXXMethodDeclaration(CXXFD);
     DeclTy = getTypes().GetFunctionType(FInfo);
   }
-
   StringRef ResolverName = getMangledName(GD);
-
   llvm::Type *ResolverType;
   GlobalDecl ResolverGD;
   if (getTarget().supportsIFunc())
     ResolverType = llvm::FunctionType::get(
-        llvm::PointerType::get(DeclTy,
-                               Context.getTargetAddressSpace(FD->getType())),
+        llvm::PointerType::get(
+            DeclTy, getContext().getTargetAddressSpace(FD->getType())),
         false);
   else {
     ResolverType = DeclTy;
     ResolverGD = GD;
   }
 
-  auto *ResolverFunc = cast<llvm::Function>(GetOrCreateLLVMFunction(
+  return cast<llvm::Function>(GetOrCreateLLVMFunction(
       ResolverName, ResolverType, ResolverGD, /*ForVTable=*/false));
+}
 
+void CodeGenModule::emitCPUDispatchResolver(GlobalDecl GD) {
   SmallVector<CodeGenFunction::MultiVersionResolverOption, 10> Options;
   const TargetInfo &Target = getTarget();
-  unsigned Index = 0;
-  for (const IdentifierInfo *II : DD->cpus()) {
-    // Get the name of the target function so we can look it up/create it.
-    std::string MangledName = getMangledNameImpl(*this, GD, FD, true) +
-                              getCPUSpecificMangling(*this, II->getName());
-
-    llvm::Constant *Func = GetGlobalValue(MangledName);
+  const auto *FD = cast<FunctionDecl>(GD.getDecl());
+  assert(FD->isCPUDispatchMultiVersion() &&
+         "Not a target multiversion function?");
+  const auto *DD = FD->getAttr<CPUDispatchAttr>();
+  llvm::Type *DeclTy = nullptr;
+  llvm::Function *ResolverFunc = getCPUDispatchResolverFunction(GD, DeclTy);
 
-    if (!Func) {
-      GlobalDecl ExistingDecl = Manglings.lookup(MangledName);
-      if (ExistingDecl.getDecl() &&
-          ExistingDecl.getDecl()->getAsFunction()->isDefined()) {
-        EmitGlobalFunctionDefinition(ExistingDecl, nullptr);
-        Func = GetGlobalValue(MangledName);
-      } else {
-        if (!ExistingDecl.getDecl())
-          ExistingDecl = GD.getWithMultiVersionIndex(Index);
+  unsigned Ordinal = 1;
+  for (const IdentifierInfo *II : DD->cpus()) {
+    GlobalDecl ImplDecl;
+    // If the function version was declared/defined in the translation unit, use
+    // that one.
+    getContext().forEachMultiversionedFunctionVersion(
+        FD, [&ImplDecl, II](const FunctionDecl *CurFD) {
+          CPUSpecificAttr *SA = CurFD->getAttr<CPUSpecificAttr>();
+          unsigned Ord;
+          if (SA && (Ord = SA->getCPUNameOrdinal(II->getName())))
+            ImplDecl =
+                GlobalDecl(CurFD->isDefined() ? CurFD->getDefinition()
+                                              : CurFD->getCanonicalDecl(),
+                           Ord);
+        });
 
-      Func = GetOrCreateLLVMFunction(
-          MangledName, DeclTy, ExistingDecl,
-          /*ForVTable=*/false, /*DontDefer=*/true,
-          /*IsThunk=*/false, llvm::AttributeList(), ForDefinition);
-      }
-    }
+    if (!ImplDecl.getDecl())
+      ImplDecl = GD.getWithMultiVersionOrdinal(Ordinal);
+    else if (ImplDecl.getDecl()->getAsFunction()->isDefined())
+        EmitGlobalFunctionDefinition(ImplDecl, nullptr);
+
+    auto *Func = cast<llvm::Function>(
+        GetAddrOfFunction(ImplDecl, DeclTy, /*ForVTable=*/false,
+                          /*DontDefer*/ false, ForDefinition));
+    assert(Func && "This should have just been created");
 
     llvm::SmallVector<StringRef, 32> Features;
     Target.getCPUSpecificCPUDispatchFeatures(II->getName(), Features);
     llvm::transform(Features, Features.begin(),
                     [](StringRef Str) { return Str.substr(1); });
-    Features.erase(std::remove_if(
-        Features.begin(), Features.end(), [&Target](StringRef Feat) {
+    Features.erase(std::remove_if(Features.begin(), Features.end(),
+                                  [&Target](StringRef Feat) {
                                     return !Target.validateCpuSupports(Feat);
-        }), Features.end());
+                                  }),
+                   Features.end());
     Options.emplace_back(cast<llvm::Function>(Func), StringRef{}, Features);
-    ++Index;
+    ++Ordinal;
   }
 
   llvm::sort(
       Options, [](const CodeGenFunction::MultiVersionResolverOption &LHS,
                   const CodeGenFunction::MultiVersionResolverOption &RHS) {
         return CodeGenFunction::GetX86CpuSupportsMask(LHS.Conditions.Features) >
                CodeGenFunction::GetX86CpuSupportsMask(RHS.Conditions.Features);
       });
-
   // If the list contains multiple 'default' versions, such as when it contains
   // 'pentium' and 'generic', don't emit the call to the generic one (since we
   // always run on at least a 'pentium'). We do this by deleting the 'least
   // advanced' (read, lowest mangling letter).
   while (Options.size() > 1 &&
          CodeGenFunction::GetX86CpuSupportsMask(
              (Options.end() - 2)->Conditions.Features) == 0) {
     StringRef LHSName = (Options.end() - 2)->Function->getName();
if (const auto *TD = FD->getAttr<TargetAttr>()) {
     // the default or a new one from the target attribute string. Then we'll use
     // the passed in features (FeaturesAsWritten) along with the new ones from
     // the attribute.
     Target.initFeatureMap(FeatureMap, getDiags(), TargetCPU,
                           ParsedAttr.Features);
   } else if (const auto *SD = FD->getAttr<CPUSpecificAttr>()) {
     llvm::SmallVector<StringRef, 32> FeaturesTmp;
     Target.getCPUSpecificCPUDispatchFeatures(
-        SD->getCPUName(GD.getMultiVersionIndex())->getName(), FeaturesTmp);
+        SD->getCPUName(GD.getMultiVersionOrdinal())->getName(), FeaturesTmp);
     std::vector<std::string> Features(FeaturesTmp.begin(), FeaturesTmp.end());
     Target.initFeatureMap(FeatureMap, getDiags(), TargetCPU, Features);
   } else {
     Target.initFeatureMap(FeatureMap, getDiags(), TargetCPU,
                           Target.getTargetOpts().Features);
   }
 }
 

test/CodeGen/attr-cpuspecific.c

 // WINDOWS: define dso_local void @NotCalled.S() #[[S:[0-9]+]]
 
 // Done before any of the implementations.  Also has an undecorated forward
 // declaration.
 void TwoVersions(void);
 
 ATTR(cpu_dispatch(ivybridge, knl))
 void TwoVersions(void);
-// LINUX: define void ()* @TwoVersions.resolver()
-// LINUX: call void @__cpu_indicator_init
-// LINUX: ret void ()* @TwoVersions.Z
-// LINUX: ret void ()* @TwoVersions.S
-// LINUX: call void @llvm.trap
-// LINUX: unreachable
-
-// WINDOWS: define dso_local void @TwoVersions()
-// WINDOWS: call void @__cpu_indicator_init()
-// WINDOWS: call void @TwoVersions.Z()
-// WINDOWS-NEXT: ret void
-// WINDOWS: call void @TwoVersions.S()
-// WINDOWS-NEXT: ret void
-// WINDOWS: call void @llvm.trap
-// WINDOWS: unreachable
+// Resolvers are emitted at the end, so the check lines are at the bottom.
 
 ATTR(cpu_specific(ivybridge))
 void TwoVersions(void){}
 // CHECK: define {{.*}}void @TwoVersions.S() #[[S]]
 
 ATTR(cpu_specific(knl))
 void TwoVersions(void){}
 // CHECK: define {{.*}}void @TwoVersions.Z() #[[K:[0-9]+]]
void usages() {
   ThreeVersionsSameAttr();
   // LINUX: @ThreeVersionsSameAttr.ifunc()
   // WINDOWS: @ThreeVersionsSameAttr()
 }
 
 // has an extra config to emit!
 ATTR(cpu_dispatch(ivybridge, knl, atom))
 void TwoVersionsSameAttr(void);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_dispatch(atom, ivybridge, knl))
+void ThreeVersionsSameAttr(void){}
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+// No Cpu Specific options.
+ATTR(cpu_dispatch(atom, ivybridge, knl))
+void NoSpecifics(void);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_dispatch(atom, generic, ivybridge, knl))
+void HasGeneric(void);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_dispatch(atom, generic, ivybridge, knl))
+void HasParams(int i, double d);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_dispatch(atom, generic, ivybridge, knl))
+int HasParamsAndReturn(int i, double d);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_dispatch(atom, generic, pentium))
+int GenericAndPentium(int i, double d);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_dispatch(atom, pentium))
+int DispatchFirst(void);
+// Resolvers are emitted at the end, so the check lines are at the bottom.
+
+ATTR(cpu_specific(atom))
+int DispatchFirst(void) {return 0;}
+ATTR(cpu_specific(pentium))
+int DispatchFirst(void) {return 1;}
+// Resolver emit causes these to be emited, so they happen later.
+
+// LINUX: define void ()* @TwoVersions.resolver()
+// LINUX: call void @__cpu_indicator_init
+// LINUX: ret void ()* @TwoVersions.Z
+// LINUX: ret void ()* @TwoVersions.S
+// LINUX: call void @llvm.trap
+// LINUX: unreachable
+
+// WINDOWS: define dso_local void @TwoVersions()
+// WINDOWS: call void @__cpu_indicator_init()
+// WINDOWS: call void @TwoVersions.Z()
+// WINDOWS-NEXT: ret void
+// WINDOWS: call void @TwoVersions.S()
+// WINDOWS-NEXT: ret void
+// WINDOWS: call void @llvm.trap
+// WINDOWS: unreachable
+
 // LINUX: define void ()* @TwoVersionsSameAttr.resolver()
 // LINUX: ret void ()* @TwoVersionsSameAttr.Z
 // LINUX: ret void ()* @TwoVersionsSameAttr.S
 // LINUX: ret void ()* @TwoVersionsSameAttr.O
 // LINUX: call void @llvm.trap
 // LINUX: unreachable
 
 // WINDOWS: define dso_local void @TwoVersionsSameAttr()
 // WINDOWS: call void @TwoVersionsSameAttr.Z
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @TwoVersionsSameAttr.S
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @TwoVersionsSameAttr.O
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @llvm.trap
 // WINDOWS: unreachable
 
-ATTR(cpu_dispatch(atom, ivybridge, knl))
-void ThreeVersionsSameAttr(void){}
 // LINUX: define void ()* @ThreeVersionsSameAttr.resolver()
 // LINUX: call void @__cpu_indicator_init
 // LINUX: ret void ()* @ThreeVersionsSameAttr.Z
 // LINUX: ret void ()* @ThreeVersionsSameAttr.S
 // LINUX: ret void ()* @ThreeVersionsSameAttr.O
 // LINUX: call void @llvm.trap
 // LINUX: unreachable
 
 // WINDOWS: define dso_local void @ThreeVersionsSameAttr()
 // WINDOWS: call void @__cpu_indicator_init
 // WINDOWS: call void @ThreeVersionsSameAttr.Z
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @ThreeVersionsSameAttr.S
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @ThreeVersionsSameAttr.O
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @llvm.trap
 // WINDOWS: unreachable
 
-// No Cpu Specific options.
-ATTR(cpu_dispatch(atom, ivybridge, knl))
-void NoSpecifics(void);
+// LINUX: define i32 @DispatchFirst.O
+// LINUX: ret i32 0
+// LINUX: define i32 @DispatchFirst.B
+// LINUX: ret i32 1
+
+// WINDOWS: define dso_local i32 @DispatchFirst.O()
+// WINDOWS: ret i32 0
+// WINDOWS: define dso_local i32 @DispatchFirst.B
+// WINDOWS: ret i32 1
+
 // LINUX: define void ()* @NoSpecifics.resolver()
 // LINUX: call void @__cpu_indicator_init
 // LINUX: ret void ()* @NoSpecifics.Z
 // LINUX: ret void ()* @NoSpecifics.S
 // LINUX: ret void ()* @NoSpecifics.O
 // LINUX: call void @llvm.trap
 // LINUX: unreachable
 
 // WINDOWS: define dso_local void @NoSpecifics()
 // WINDOWS: call void @__cpu_indicator_init
 // WINDOWS: call void @NoSpecifics.Z
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @NoSpecifics.S
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @NoSpecifics.O
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @llvm.trap
 // WINDOWS: unreachable
 
-ATTR(cpu_dispatch(atom, generic, ivybridge, knl))
-void HasGeneric(void);
 // LINUX: define void ()* @HasGeneric.resolver()
 // LINUX: call void @__cpu_indicator_init
 // LINUX: ret void ()* @HasGeneric.Z
 // LINUX: ret void ()* @HasGeneric.S
 // LINUX: ret void ()* @HasGeneric.O
 // LINUX: ret void ()* @HasGeneric.A
 // LINUX-NOT: call void @llvm.trap
 
 // WINDOWS: define dso_local void @HasGeneric()
 // WINDOWS: call void @__cpu_indicator_init
 // WINDOWS: call void @HasGeneric.Z
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @HasGeneric.S
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @HasGeneric.O
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @HasGeneric.A
 // WINDOWS-NEXT: ret void
 // WINDOWS-NOT: call void @llvm.trap
 
-ATTR(cpu_dispatch(atom, generic, ivybridge, knl))
-void HasParams(int i, double d);
 // LINUX: define void (i32, double)* @HasParams.resolver()
 // LINUX: call void @__cpu_indicator_init
 // LINUX: ret void (i32, double)* @HasParams.Z
 // LINUX: ret void (i32, double)* @HasParams.S
 // LINUX: ret void (i32, double)* @HasParams.O
 // LINUX: ret void (i32, double)* @HasParams.A
 // LINUX-NOT: call void @llvm.trap
 
 // WINDOWS: define dso_local void @HasParams(i32, double)
 // WINDOWS: call void @__cpu_indicator_init
 // WINDOWS: call void @HasParams.Z(i32 %0, double %1)
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @HasParams.S(i32 %0, double %1)
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @HasParams.O(i32 %0, double %1)
 // WINDOWS-NEXT: ret void
 // WINDOWS: call void @HasParams.A(i32 %0, double %1)
 // WINDOWS-NEXT: ret void
 // WINDOWS-NOT: call void @llvm.trap
 
-ATTR(cpu_dispatch(atom, generic, ivybridge, knl))
-int HasParamsAndReturn(int i, double d);
 // LINUX: define i32 (i32, double)* @HasParamsAndReturn.resolver()
 // LINUX: call void @__cpu_indicator_init
 // LINUX: ret i32 (i32, double)* @HasParamsAndReturn.Z
 // LINUX: ret i32 (i32, double)* @HasParamsAndReturn.S
 // LINUX: ret i32 (i32, double)* @HasParamsAndReturn.O
 // LINUX: ret i32 (i32, double)* @HasParamsAndReturn.A
 // LINUX-NOT: call void @llvm.trap
 
 // WINDOWS: define dso_local i32 @HasParamsAndReturn(i32, double)
 // WINDOWS: call void @__cpu_indicator_init
 // WINDOWS: %[[RET:.+]] = musttail call i32 @HasParamsAndReturn.Z(i32 %0, double %1)
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS: %[[RET:.+]] = musttail call i32 @HasParamsAndReturn.S(i32 %0, double %1)
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS: %[[RET:.+]] = musttail call i32 @HasParamsAndReturn.O(i32 %0, double %1)
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS: %[[RET:.+]] = musttail call i32 @HasParamsAndReturn.A(i32 %0, double %1)
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS-NOT: call void @llvm.trap
 
-ATTR(cpu_dispatch(atom, generic, pentium))
-int GenericAndPentium(int i, double d);
 // LINUX: define i32 (i32, double)* @GenericAndPentium.resolver()
 // LINUX: call void @__cpu_indicator_init
 // LINUX: ret i32 (i32, double)* @GenericAndPentium.O
 // LINUX: ret i32 (i32, double)* @GenericAndPentium.B
 // LINUX-NOT: ret i32 (i32, double)* @GenericAndPentium.A
 // LINUX-NOT: call void @llvm.trap
 
 // WINDOWS: define dso_local i32 @GenericAndPentium(i32, double)
 // WINDOWS: call void @__cpu_indicator_init
 // WINDOWS: %[[RET:.+]] = musttail call i32 @GenericAndPentium.O(i32 %0, double %1)
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS: %[[RET:.+]] = musttail call i32 @GenericAndPentium.B(i32 %0, double %1)
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS-NOT: call i32 @GenericAndPentium.A
 // WINDOWS-NOT: call void @llvm.trap
 
-ATTR(cpu_dispatch(atom, pentium))
-int DispatchFirst(void);
 // LINUX: define i32 ()* @DispatchFirst.resolver
 // LINUX: ret i32 ()* @DispatchFirst.O
 // LINUX: ret i32 ()* @DispatchFirst.B
 
 // WINDOWS: define dso_local i32 @DispatchFirst()
 // WINDOWS: %[[RET:.+]] = musttail call i32 @DispatchFirst.O()
 // WINDOWS-NEXT: ret i32 %[[RET]]
 // WINDOWS: %[[RET:.+]] = musttail call i32 @DispatchFirst.B()
 // WINDOWS-NEXT: ret i32 %[[RET]]
 
-ATTR(cpu_specific(atom))
-int DispatchFirst(void) {return 0;}
-// LINUX: define i32 @DispatchFirst.O
-// LINUX: ret i32 0
-
-// WINDOWS: define dso_local i32 @DispatchFirst.O()
-// WINDOWS: ret i32 0
-
-ATTR(cpu_specific(pentium))
-int DispatchFirst(void) {return 1;}
-// LINUX: define i32 @DispatchFirst.B
-// LINUX: ret i32 1
-
-// WINDOWS: define dso_local i32 @DispatchFirst.B
-// WINDOWS: ret i32 1
-
 // CHECK: attributes #[[S]] = {{.*}}"target-features"="+avx,+cmov,+f16c,+mmx,+popcnt,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave"
 // CHECK: attributes #[[K]] = {{.*}}"target-features"="+adx,+avx,+avx2,+avx512cd,+avx512er,+avx512f,+avx512pf,+bmi,+cmov,+f16c,+fma,+lzcnt,+mmx,+movbe,+popcnt,+sse,+sse2,+sse3,+sse4.1,+sse4.2,+ssse3,+x87,+xsave"
 // CHECK: attributes #[[O]] = {{.*}}"target-features"="+cmov,+mmx,+movbe,+sse,+sse2,+sse3,+ssse3,+x87"