[WIP] [ThinLTO] Importing function by function specialization

[ChuanqiXu]

Summary

This diff is the successive for D105524, which extract the analysis part of function specialization as an independent analysis.

The intention of this diff is to enable ThinLTO to import functions by heuristics for function specialization.
Now the ThinLTO imports functions from other CU by the heuristics for inlining only.
Simply, ThinLTO would import functions whose lines of codes are less than 100 lines by default to enable potential inlining.
This makes sense due to the inlining is the main root cause for IPO in fact.

And function specialization is another important IPO technique which got implemented recently.
My goal was to make ThinLTO to import functions to enable function specialization.
The key point here is that we need to record the information needed for the imported to judge whether it is beneficial to import a function whose LoC is more than 100 lines.

Here is the extra information I planned to add:

• Specialize Function Cost. An unsigned number, which is an estimation for the cost to clone one function.
• Base Bonus for specializing specific argument. A map from ArgNo to the corresponding base bonus. Here the base bonus means the bonus we could calculate by visiting the function body itself only (in other words, doesn't need to see the call site).
• ArgUsage. A map from ArgNo to the extra bonus for each call site. The extra bonus means the bonus we could calculate for specific call site. For example, if we passed a function in the call, we could calculate the possibility that the function could get inlined at the specific call site only.

Before in https://lists.llvm.org/pipermail/llvm-dev/2021-May/150443.html, I raised some problems we may met:

• We can't see the funciton body before we import it.
• It would repeat traversing the call graph in each translation unit, which is very redundant.
• It may specialize functions with the same version, which could make the code size get larger and redundant.

Here the first problem would be solved by D105524 and this patch.
Then the second problem seems not so noisy. Except 502.gcc_r, which CT time increased 30%, we didn't observe any significant CT change in SPEC2017 int.
Finally, the third problem. The average enlarged code size in spec2017 int is 6.7%. And the maximum is 19%.

The numbers may not be so satisfying. And the function specialization pass is not so mature. It's the reason that I marked this patch as [WIP].

The reason why I update the patch is that I want to make sure that I am on the right direction.

Finally, there are some TODOs in this patch:

• Didn't implement writer and parser for .ll files.
• Need to add and fix tests.

Test Plan: SPEC2017 int rate.

llvm/include/llvm/Analysis/FuncSpecCost.h

 //===------------------------------------------------------------------------===//
 
 #ifndef LLVM_ANALYSIS_FUNCSPECCOST_H
 #define LLVM_ANALYSIS_FUNCSPECCOST_H
 
 #include "llvm/ADT/MapVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/IR/Argument.h"
+#include "llvm/IR/InstrTypes.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/InstructionCost.h"
 
 namespace llvm {
 
 class AssumptionCache;
 class TargetTransformInfo;
 class TargetLibraryInfo;
+class ArgUsage;
 
 /// Cost/Bonus information for specialize a function with
 /// each argument.
 class FuncSpecCostInfo {
   /// The cost to specialize the function.
   InstructionCost Cost;
   /// Map from the number of the argument, to the base bonuss for specialize it.
   /// Base bonus stands for the bonus we could get by the function body.
   MapVector<unsigned, unsigned> SpecBonusBaseMap;
 
   unsigned getBonusBase(Argument *Arg) const;
   unsigned getBonusBase(unsigned Index) const;
 
 public:
   InstructionCost getCost() const { return Cost; }
   unsigned
   getBonus(Argument *Arg, Constant *C,
            function_ref<TargetTransformInfo &(Function &)> GetTTI,
            function_ref<AssumptionCache &(Function &)> GetAC,
            function_ref<const TargetLibraryInfo &(Function &)> GetTLI) const;
 
+  /// Given an ArgUsage, estimating if we should import corresponding function.
+  /// We should **only** call this when importing.
+  ///
+  /// If there is one argument marks function, we would think it would be
+  /// inlined if its lines of codes is less than a specific threshold.
+  ///
+  /// TOOD: Add profiling infomation.
+  bool shouldImport(const ArgUsage &) const;
+
+  ArrayRef<std::pair<unsigned, unsigned>> getSpecBonusBaseMap() const {
+    return  makeArrayRef(&SpecBonusBaseMap.front(),
+                         SpecBonusBaseMap.size());
+  }
+
   FuncSpecCostInfo() {}
   FuncSpecCostInfo(FuncSpecCostInfo &&Other)
       : FuncSpecCostInfo(std::move(Other.Cost),
                          std::move(Other.SpecBonusBaseMap)) {}
   FuncSpecCostInfo(InstructionCost Cost,
                    MapVector<unsigned, unsigned> &&BonusBaseMap)
       : Cost(std::move(Cost)) {
     SpecBonusBaseMap.swap(BonusBaseMap);
public:
 
   bool runOnFunction(Function &F) override;
 
   void getAnalysisUsage(AnalysisUsage &AU) const override;
 
   StringRef getPassName() const override;
 };
 
+/// Represent the usage of args at the callsite used in module summary.
+/// We should keep it as small as possible.
+///
+/// Now we mainly cares if the argument is a function. If yes, it implies
+/// a chance to hoist an indirect call to a direct call by function specialize
+/// pass.
+///
+/// It should be easy to add value infomation about the constantness or value
+/// range.
+class ArgUsage {
+private:
+  static unsigned ConstantMarker;
+  /// Map from ArgNo to the lines of codes if the corresponding argument refer
+  /// to a function. If the corresponding argument is a constant other than
+  /// function, we would set the value to ConstantMarker(0xffffffff).
+  ///
+  /// For example, the value of LinesOfArgs for following example:
+  /// ```
+  ///     foo(var, 1, bar); // bar is a function; var is a variable
+  /// ```
+  /// should be `[<1, 0xffffffff>, <2, lines of bar>]`.
+  SmallVector<std::pair<unsigned, unsigned>, 4> LinesOfArgs;
+
+  friend class FuncSpecCostInfo;
+  friend class CalleeInfo;
+
+public:
+  ArgUsage() {}
+  ArgUsage(const CallBase &);
+  ArgUsage(SmallVectorImpl<std::pair<unsigned, unsigned>> &&Uses)
+      : LinesOfArgs(std::move(Uses)) {}
+
+  ArgUsage(const ArgUsage &AU) : LinesOfArgs(AU.LinesOfArgs) {}
+  ArgUsage(ArgUsage &&AU) : LinesOfArgs(std::move(AU.LinesOfArgs)) {}
+  ArgUsage &operator=(ArgUsage &&AU) {
+    LinesOfArgs = std::move(AU.LinesOfArgs);
+    return *this;
+  }
+  ArgUsage &operator=(const ArgUsage &AU) {
+    LinesOfArgs = AU.LinesOfArgs;
+    return *this;
+  }
+
+  static bool isConstant(unsigned value) { return value == ConstantMarker; }
+};
 } // namespace llvm
 
 #endif

llvm/include/llvm/Analysis/ModuleSummaryAnalysis.h

 ///
 /// If operating within a pass manager which has defined ways to compute the \c
 /// BlockFrequencyInfo for a given function, that can be provided via
 /// a std::function callback. Otherwise, this routine will manually construct
 /// that information.
 ModuleSummaryIndex buildModuleSummaryIndex(
     const Module &M,
     std::function<BlockFrequencyInfo *(const Function &F)> GetBFICallback,
+    std::function<FuncSpecCostInfo *(const Function &F)> GetFSCICallback,
     ProfileSummaryInfo *PSI,
     std::function<const StackSafetyInfo *(const Function &F)> GetSSICallback =
         [](const Function &F) -> const StackSafetyInfo * { return nullptr; });
 
 /// Analysis pass to provide the ModuleSummaryIndex object.
 class ModuleSummaryIndexAnalysis
     : public AnalysisInfoMixin<ModuleSummaryIndexAnalysis> {
   friend AnalysisInfoMixin<ModuleSummaryIndexAnalysis>;

llvm/include/llvm/IR/ModuleSummaryIndex.h

 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/TinyPtrVector.h"
+#include "llvm/Analysis/FuncSpecCost.h"
 #include "llvm/IR/ConstantRange.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/Module.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ScaledNumber.h"
 #include "llvm/Support/StringSaver.h"
 #include "llvm/Support/raw_ostream.h"
struct CalleeInfo {
 
   // The size of the bit-field might need to be adjusted if more values are
   // added to HotnessType enum.
   uint32_t Hotness : 3;
 
   /// The value stored in RelBlockFreq has to be interpreted as the digits of
   /// a scaled number with a scale of \p -ScaleShift.
   uint32_t RelBlockFreq : 29;
+  /// The infomation about how arguments are used at the callsite.
+  /// NOTE: This may enlarge the size of CalleeInfo.
+  ArgUsage Usages;
   static constexpr int32_t ScaleShift = 8;
   static constexpr uint64_t MaxRelBlockFreq = (1 << 29) - 1;
 
   CalleeInfo()
-      : Hotness(static_cast<uint32_t>(HotnessType::Unknown)), RelBlockFreq(0) {}
-  explicit CalleeInfo(HotnessType Hotness, uint64_t RelBF)
-      : Hotness(static_cast<uint32_t>(Hotness)), RelBlockFreq(RelBF) {}
+      : Hotness(static_cast<uint32_t>(HotnessType::Unknown)), RelBlockFreq(0),
+        Usages() {}
+  explicit CalleeInfo(HotnessType Hotness, uint64_t RelBF, ArgUsage &&Usages)
+      : Hotness(static_cast<uint32_t>(Hotness)), RelBlockFreq(RelBF),
+        Usages(std::move(Usages)) {}
+
+  void updateUsages(ArgUsage &&OtherUsages) { Usages = std::move(OtherUsages); }
+
+  size_t getUsagesSize() const { return Usages.LinesOfArgs.size(); }
+
+  ArrayRef<std::pair<unsigned, unsigned>> getUsages() const {
+    return makeArrayRef(Usages.LinesOfArgs.begin(), Usages.LinesOfArgs.end());
+  }
 
   void updateHotness(const HotnessType OtherHotness) {
     Hotness = std::max(Hotness, static_cast<uint32_t>(OtherHotness));
   }
 
   HotnessType getHotness() const { return HotnessType(Hotness); }
 
   /// Update \p RelBlockFreq from \p BlockFreq and \p EntryFreq
return FunctionSummary(
             /*CanAutoHide=*/false),
         /*NumInsts=*/0, FunctionSummary::FFlags{}, /*EntryCount=*/0,
         std::vector<ValueInfo>(), std::move(Edges),
         std::vector<GlobalValue::GUID>(),
         std::vector<FunctionSummary::VFuncId>(),
         std::vector<FunctionSummary::VFuncId>(),
         std::vector<FunctionSummary::ConstVCall>(),
         std::vector<FunctionSummary::ConstVCall>(),
-        std::vector<FunctionSummary::ParamAccess>());
+        std::vector<FunctionSummary::ParamAccess>(),
+        FuncSpecCostInfo());
   }
 
   /// A dummy node to reference external functions that aren't in the index
   static FunctionSummary ExternalNode;
 
 private:
   /// Number of instructions (ignoring debug instructions, e.g.) computed
   /// during the initial compile step when the summary index is first built.
private:
   std::vector<EdgeTy> CallGraphEdgeList;
 
   std::unique_ptr<TypeIdInfo> TIdInfo;
 
   /// Uses for every parameter to this function.
   using ParamAccessesTy = std::vector<ParamAccess>;
   std::unique_ptr<ParamAccessesTy> ParamAccesses;
 
+  FuncSpecCostInfo SpecCostInfo;
+
 public:
   FunctionSummary(GVFlags Flags, unsigned NumInsts, FFlags FunFlags,
                   uint64_t EntryCount, std::vector<ValueInfo> Refs,
                   std::vector<EdgeTy> CGEdges,
                   std::vector<GlobalValue::GUID> TypeTests,
                   std::vector<VFuncId> TypeTestAssumeVCalls,
                   std::vector<VFuncId> TypeCheckedLoadVCalls,
                   std::vector<ConstVCall> TypeTestAssumeConstVCalls,
                   std::vector<ConstVCall> TypeCheckedLoadConstVCalls,
-                  std::vector<ParamAccess> Params)
+                  std::vector<ParamAccess> Params,
+                  FuncSpecCostInfo SpecCostInfo)
       : GlobalValueSummary(FunctionKind, Flags, std::move(Refs)),
         InstCount(NumInsts), FunFlags(FunFlags), EntryCount(EntryCount),
-        CallGraphEdgeList(std::move(CGEdges)) {
+        CallGraphEdgeList(std::move(CGEdges)),
+        SpecCostInfo(std::move(SpecCostInfo)) {
     if (!TypeTests.empty() || !TypeTestAssumeVCalls.empty() ||
         !TypeCheckedLoadVCalls.empty() || !TypeTestAssumeConstVCalls.empty() ||
         !TypeCheckedLoadConstVCalls.empty())
       TIdInfo = std::make_unique<TypeIdInfo>(
           TypeIdInfo{std::move(TypeTests), std::move(TypeTestAssumeVCalls),
                      std::move(TypeCheckedLoadVCalls),
                      std::move(TypeTestAssumeConstVCalls),
                      std::move(TypeCheckedLoadConstVCalls)});
public:
   void addTypeTest(GlobalValue::GUID Guid) {
     if (!TIdInfo)
       TIdInfo = std::make_unique<TypeIdInfo>();
     TIdInfo->TypeTests.push_back(Guid);
   }
 
   const TypeIdInfo *getTypeIdInfo() const { return TIdInfo.get(); };
 
+  unsigned getSpecializeCost() const {
+    InstructionCost Cost = SpecCostInfo.getCost();
+    if (Cost.isValid())
+      return *Cost.getValue();
+    return std::numeric_limits<unsigned>::max();
+  }
+
+  ArrayRef<std::pair<unsigned, unsigned>> getSpecBonusBase() const {
+    return SpecCostInfo.getSpecBonusBaseMap();
+  }
+
+  bool shouldImport(const CalleeInfo &CI) const {
+    return SpecCostInfo.shouldImport(CI.Usages);
+  }
+
   friend struct GraphTraits<ValueInfo>;
 };
 
 template <> struct DenseMapInfo<FunctionSummary::VFuncId> {
   static FunctionSummary::VFuncId getEmptyKey() { return {0, uint64_t(-1)}; }
 
   static FunctionSummary::VFuncId getTombstoneKey() {
     return {0, uint64_t(-2)};
ModuleSummaryIndex(bool HaveGVs, bool EnableSplitLTOUnit = false)
       : HaveGVs(HaveGVs), EnableSplitLTOUnit(EnableSplitLTOUnit), Saver(Alloc),
         BlockCount(0) {}
 
   // Current version for the module summary in bitcode files.
   // The BitcodeSummaryVersion should be bumped whenever we introduce changes
   // in the way some record are interpreted, like flags for instance.
   // Note that incrementing this may require changes in both BitcodeReader.cpp
   // and BitcodeWriter.cpp.
-  static constexpr uint64_t BitcodeSummaryVersion = 9;
+  // ChangeLog: 9->10. Add ArgUsage in CalleeInfo and SpecCostInfo in 
+  // Function Summary.
+  static constexpr uint64_t BitcodeSummaryVersion = 10;
 
   // Regular LTO module name for ASM writer
   static constexpr const char *getRegularLTOModuleName() {
     return "[Regular LTO]";
   }
 
   bool haveGVs() const { return HaveGVs; }
 

llvm/include/llvm/IR/ModuleSummaryIndexYAML.h

for (auto &FSum : FSums) {
               FSum.NotEligibleToImport, FSum.Live, FSum.IsLocal,
               FSum.CanAutoHide),
           /*NumInsts=*/0, FunctionSummary::FFlags{}, /*EntryCount=*/0, Refs,
           ArrayRef<FunctionSummary::EdgeTy>{}, std::move(FSum.TypeTests),
           std::move(FSum.TypeTestAssumeVCalls),
           std::move(FSum.TypeCheckedLoadVCalls),
           std::move(FSum.TypeTestAssumeConstVCalls),
           std::move(FSum.TypeCheckedLoadConstVCalls),
-          ArrayRef<FunctionSummary::ParamAccess>{}));
+          ArrayRef<FunctionSummary::ParamAccess>{}, FuncSpecCostInfo()));
     }
   }
   static void output(IO &io, GlobalValueSummaryMapTy &V) {
     for (auto &P : V) {
       std::vector<FunctionSummaryYaml> FSums;
       for (auto &Sum : P.second.SummaryList) {
         if (auto *FSum = dyn_cast<FunctionSummary>(Sum.get())) {
           std::vector<uint64_t> Refs;

llvm/lib/Analysis/FuncSpecCost.cpp

 #include "llvm/Support/CommandLine.h"
 #include <cassert>
 #include <cmath>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "func-spec-cost"
 
+/// Limit on instruction count of imported functions.
+/// The function whose number of instruction below this argument
+/// is considered to be inlined during function importing.
+static cl::opt<unsigned> PotentialInlingLimit(
+    "funcspec-potential-inlininglimit", cl::init(100), cl::Hidden,
+    cl::value_desc("N"),
+    cl::desc("Only functions with less than N instructions are considered to "
+             "be inlined during analysis of func spec."));
+
+static cl::opt<double>
+    BonusFactorFromInlining("func-spec-bonus-from-inlining-factor", cl::Hidden,
+                            cl::desc("Factor for bonus from potetial inling."),
+                            cl::init(10));
+
 static cl::opt<unsigned>
     AvgLoopIterationCount("func-specialization-avg-iters-cost", cl::Hidden,
                           cl::desc("Average loop iteration count cost"),
                           cl::init(10));
 
 static Function *getFunction(Value *CalledValue) {
   // Since the argument is a function pointer, its incoming constant values
   // should be functions or constant expressions. The code below attempts to
bool FunctionSpecializationWrapperPass::runOnFunction(Function &F) {
   AssumptionCache &AC =
       getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
   Info.emplace(CreateFuncSpecCostInfo(F, AC, TTI, LI));
   return false;
 }
 
 StringRef FunctionSpecializationWrapperPass::getPassName() const {
   return "Function Specialization Cost Analysis";
+}
+
+unsigned ArgUsage::ConstantMarker = ~(unsigned)0;
+
+ArgUsage::ArgUsage(const CallBase &CB) {
+  for (auto &U : CB.args()) {
+    Function *F = getFunction(U.get());
+    LinesOfArgs.push_back({CB.getArgOperandNo(&U), ConstantMarker});
+    if (!F)
+      continue;
+
+    LinesOfArgs.push_back({CB.getArgOperandNo(&U), F->getInstructionCount()});
+  }
+}
+
+bool FuncSpecCostInfo::shouldImport(const ArgUsage &AU) const {
+  if (!Cost.isValid())
+    return false;
+  unsigned SpecCost = *Cost.getValue();
+  /// FIXME: It would consider the first argument who fits
+  /// the condition. It should be fixed after Function Specialization
+  /// pass fix this.
+  for (auto &IndexValuePair : AU.LinesOfArgs) {
+    unsigned Index = IndexValuePair.first;
+    unsigned Value = IndexValuePair.second;
+    unsigned BonusBase = getBonusBase(Index);
+    // Should we consider attributes like `noinline` and `always_inline` here?
+    if (!ArgUsage::isConstant(Value) && Value < PotentialInlingLimit)
+      BonusBase += BonusFactorFromInlining * (PotentialInlingLimit - Value);
+
+    if (BonusBase > SpecCost)
+      return true;
+  }
+  return false;
 }
 No newline at end of file

llvm/lib/Analysis/ModuleSummaryAnalysis.cpp

static bool isNonVolatileStore(const Instruction *I) {
   return false;
 }
 
 static void computeFunctionSummary(
     ModuleSummaryIndex &Index, const Module &M, const Function &F,
     BlockFrequencyInfo *BFI, ProfileSummaryInfo *PSI, DominatorTree &DT,
     bool HasLocalsInUsedOrAsm, DenseSet<GlobalValue::GUID> &CantBePromoted,
     bool IsThinLTO,
-    std::function<const StackSafetyInfo *(const Function &F)> GetSSICallback) {
+    std::function<const StackSafetyInfo *(const Function &F)> GetSSICallback,
+    std::function<FuncSpecCostInfo *(const Function &F)> GetFSCICallback) {
   // Summary not currently supported for anonymous functions, they should
   // have been named.
   assert(F.hasName());
 
   unsigned NumInsts = 0;
   // Map from callee ValueId to profile count. Used to accumulate profile
   // counts for all static calls to a given callee.
   MapVector<ValueInfo, CalleeInfo> CallGraphEdges;
for (const Instruction &I : BB) {
         ValueInfo.updateHotness(Hotness);
         // Add the relative block frequency to CalleeInfo if there is no profile
         // information.
         if (BFI != nullptr && Hotness == CalleeInfo::HotnessType::Unknown) {
           uint64_t BBFreq = BFI->getBlockFreq(&BB).getFrequency();
           uint64_t EntryFreq = BFI->getEntryFreq();
           ValueInfo.updateRelBlockFreq(BBFreq, EntryFreq);
         }
+
+        ValueInfo.updateUsages(ArgUsage(*CB));
       } else {
         // Skip inline assembly calls.
         if (CI && CI->isInlineAsm())
           continue;
         // Skip direct calls.
         if (!CalledValue || isa<Constant>(CalledValue))
           continue;
 
FunctionSummary::FFlags FunFlags{
       F.hasFnAttribute(Attribute::NoRecurse), F.returnDoesNotAlias(),
       // FIXME: refactor this to use the same code that inliner is using.
       // Don't try to import functions with noinline attribute.
       F.getAttributes().hasFnAttribute(Attribute::NoInline),
       F.hasFnAttribute(Attribute::AlwaysInline)};
   std::vector<FunctionSummary::ParamAccess> ParamAccesses;
   if (auto *SSI = GetSSICallback(F))
     ParamAccesses = SSI->getParamAccesses(Index);
+  FuncSpecCostInfo *FSCI = nullptr;
+  if (GetFSCICallback)
+    FSCI = GetFSCICallback(F);
   auto FuncSummary = std::make_unique<FunctionSummary>(
       Flags, NumInsts, FunFlags, /*EntryCount=*/0, std::move(Refs),
       CallGraphEdges.takeVector(), TypeTests.takeVector(),
       TypeTestAssumeVCalls.takeVector(), TypeCheckedLoadVCalls.takeVector(),
       TypeTestAssumeConstVCalls.takeVector(),
-      TypeCheckedLoadConstVCalls.takeVector(), std::move(ParamAccesses));
+      TypeCheckedLoadConstVCalls.takeVector(), std::move(ParamAccesses),
+      FSCI ? std::move(*FSCI) : FuncSpecCostInfo());
   if (NonRenamableLocal)
     CantBePromoted.insert(F.getGUID());
   Index.addGlobalValueSummary(F, std::move(FuncSummary));
 }
 
 /// Find function pointers referenced within the given vtable initializer
 /// (or subset of an initializer) \p I. The starting offset of \p I within
 /// the vtable initializer is \p StartingOffset. Any discovered function
static void setLiveRoot(ModuleSummaryIndex &Index, StringRef Name) {
   if (ValueInfo VI = Index.getValueInfo(GlobalValue::getGUID(Name)))
     for (auto &Summary : VI.getSummaryList())
       Summary->setLive(true);
 }
 
 ModuleSummaryIndex llvm::buildModuleSummaryIndex(
     const Module &M,
     std::function<BlockFrequencyInfo *(const Function &F)> GetBFICallback,
+    std::function<FuncSpecCostInfo *(const Function &F)> GetFSCICallback,
     ProfileSummaryInfo *PSI,
     std::function<const StackSafetyInfo *(const Function &F)> GetSSICallback) {
   assert(PSI);
   bool EnableSplitLTOUnit = false;
   if (auto *MD = mdconst::extract_or_null<ConstantInt>(
           M.getModuleFlag("EnableSplitLTOUnit")))
     EnableSplitLTOUnit = MD->getZExtValue();
   ModuleSummaryIndex Index(/*HaveGVs=*/true, EnableSplitLTOUnit);
ModuleSymbolTable::CollectAsmSymbols(
                         F->hasFnAttribute(Attribute::AlwaysInline)},
                     /*EntryCount=*/0, ArrayRef<ValueInfo>{},
                     ArrayRef<FunctionSummary::EdgeTy>{},
                     ArrayRef<GlobalValue::GUID>{},
                     ArrayRef<FunctionSummary::VFuncId>{},
                     ArrayRef<FunctionSummary::VFuncId>{},
                     ArrayRef<FunctionSummary::ConstVCall>{},
                     ArrayRef<FunctionSummary::ConstVCall>{},
-                    ArrayRef<FunctionSummary::ParamAccess>{});
+                    ArrayRef<FunctionSummary::ParamAccess>{},
+                    FuncSpecCostInfo());
             Index.addGlobalValueSummary(*GV, std::move(Summary));
           } else {
             std::unique_ptr<GlobalVarSummary> Summary =
                 std::make_unique<GlobalVarSummary>(
                     GVFlags,
                     GlobalVarSummary::GVarFlags(
                         false, false, cast<GlobalVariable>(GV)->isConstant(),
                         GlobalObject::VCallVisibilityPublic),
else if (F.hasProfileData()) {
       LoopInfo LI{DT};
       BranchProbabilityInfo BPI{F, LI};
       BFIPtr = std::make_unique<BlockFrequencyInfo>(F, BPI, LI);
       BFI = BFIPtr.get();
     }
 
     computeFunctionSummary(Index, M, F, BFI, PSI, DT,
                            !LocalsUsed.empty() || HasLocalInlineAsmSymbol,
-                           CantBePromoted, IsThinLTO, GetSSICallback);
+                           CantBePromoted, IsThinLTO, GetSSICallback, GetFSCICallback);
   }
 
   // Compute summaries for all variables defined in module, and save in the
   // index.
   SmallVector<MDNode *, 2> Types;
   for (const GlobalVariable &G : M.globals()) {
     if (G.isDeclaration())
       continue;
ModuleSummaryIndexAnalysis::run(Module &M, ModuleAnalysisManager &AM) {
   auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   bool NeedSSI = needsParamAccessSummary(M);
   return buildModuleSummaryIndex(
       M,
       [&FAM](const Function &F) {
         return &FAM.getResult<BlockFrequencyAnalysis>(
             *const_cast<Function *>(&F));
       },
+      [&FAM](const Function &F) {
+        return &FAM.getResult<FunctionSpecializationAnalysis>(
+            *const_cast<Function *>(&F));
+      },
       &PSI,
       [&FAM, NeedSSI](const Function &F) -> const StackSafetyInfo * {
         return NeedSSI ? &FAM.getResult<StackSafetyAnalysis>(
                              const_cast<Function &>(F))
                        : nullptr;
       });
 }
 
 char ModuleSummaryIndexWrapperPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(ModuleSummaryIndexWrapperPass, "module-summary-analysis",
                       "Module Summary Analysis", false, true)
 INITIALIZE_PASS_DEPENDENCY(BlockFrequencyInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(StackSafetyInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(FunctionSpecializationWrapperPass)
 INITIALIZE_PASS_END(ModuleSummaryIndexWrapperPass, "module-summary-analysis",
                     "Module Summary Analysis", false, true)
 
 ModulePass *llvm::createModuleSummaryIndexWrapperPass() {
   return new ModuleSummaryIndexWrapperPass();
 }
 
 ModuleSummaryIndexWrapperPass::ModuleSummaryIndexWrapperPass()
     : ModulePass(ID) {
   initializeModuleSummaryIndexWrapperPassPass(*PassRegistry::getPassRegistry());
 }
 
 bool ModuleSummaryIndexWrapperPass::runOnModule(Module &M) {
   auto *PSI = &getAnalysis<ProfileSummaryInfoWrapperPass>().getPSI();
   bool NeedSSI = needsParamAccessSummary(M);
   Index.emplace(buildModuleSummaryIndex(
       M,
       [this](const Function &F) {
         return &(this->getAnalysis<BlockFrequencyInfoWrapperPass>(
                          *const_cast<Function *>(&F))
                      .getBFI());
       },
+      [&](const Function &F) {
+        return &(this->getAnalysis<FunctionSpecializationWrapperPass>(
+                         *const_cast<Function *>(&F))
+                     .getFuncSpecCost());
+      },
       PSI,
       [&](const Function &F) -> const StackSafetyInfo * {
         return NeedSSI ? &getAnalysis<StackSafetyInfoWrapperPass>(
                               const_cast<Function &>(F))
                               .getResult()
                        : nullptr;
       }));
   return false;
 }
 
 bool ModuleSummaryIndexWrapperPass::doFinalization(Module &M) {
   Index.reset();
   return false;
 }
 
 void ModuleSummaryIndexWrapperPass::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.setPreservesAll();
   AU.addRequired<BlockFrequencyInfoWrapperPass>();
   AU.addRequired<ProfileSummaryInfoWrapperPass>();
   AU.addRequired<StackSafetyInfoWrapperPass>();
+  AU.addRequired<FunctionSpecializationWrapperPass>();
 }
 
 char ImmutableModuleSummaryIndexWrapperPass::ID = 0;
 
 ImmutableModuleSummaryIndexWrapperPass::ImmutableModuleSummaryIndexWrapperPass(
     const ModuleSummaryIndex *Index)
     : ImmutablePass(ID), Index(Index) {
   initializeImmutableModuleSummaryIndexWrapperPassPass(

llvm/lib/AsmParser/LLParser.cpp

bool LLParser::parseFunctionSummary(std::string Name, GlobalValue::GUID GUID,
 
   auto FS = std::make_unique<FunctionSummary>(
       GVFlags, InstCount, FFlags, /*EntryCount=*/0, std::move(Refs),
       std::move(Calls), std::move(TypeIdInfo.TypeTests),
       std::move(TypeIdInfo.TypeTestAssumeVCalls),
       std::move(TypeIdInfo.TypeCheckedLoadVCalls),
       std::move(TypeIdInfo.TypeTestAssumeConstVCalls),
       std::move(TypeIdInfo.TypeCheckedLoadConstVCalls),
-      std::move(ParamAccesses));
+      std::move(ParamAccesses),
+      /// FIXME: Implement actually parser for FuncSpecCostInfo.
+      FuncSpecCostInfo());
 
   FS->setModulePath(ModulePath);
 
   addGlobalValueToIndex(Name, GUID, (GlobalValue::LinkageTypes)GVFlags.Linkage,
                         ID, std::move(FS));
 
   return false;
 }
if (EatIfPresent(lltok::comma)) {
           return true;
       }
     }
     // Keep track of the Call array index needing a forward reference.
     // We will save the location of the ValueInfo needing an update, but
     // can only do so once the std::vector is finalized.
     if (VI.getRef() == FwdVIRef)
       IdToIndexMap[GVId].push_back(std::make_pair(Calls.size(), Loc));
-    Calls.push_back(FunctionSummary::EdgeTy{VI, CalleeInfo(Hotness, RelBF)});
+
+    /// FIXME: Implement parser for new added infomation.
+    Calls.push_back(
+        FunctionSummary::EdgeTy{VI, CalleeInfo(Hotness, RelBF, ArgUsage())});
 
     if (parseToken(lltok::rparen, "expected ')' in call"))
       return true;
   } while (EatIfPresent(lltok::comma));
 
   // Now that the Calls vector is finalized, it is safe to save the locations
   // of any forward GV references that need updating later.
   for (auto I : IdToIndexMap) {

llvm/lib/Bitcode/Reader/BitcodeReader.cpp

 private:
   void setValueGUID(uint64_t ValueID, StringRef ValueName,
                     GlobalValue::LinkageTypes Linkage,
                     StringRef SourceFileName);
   Error parseValueSymbolTable(
       uint64_t Offset,
       DenseMap<unsigned, GlobalValue::LinkageTypes> &ValueIdToLinkageMap);
   std::vector<ValueInfo> makeRefList(ArrayRef<uint64_t> Record);
+  FuncSpecCostInfo makeSpecCostInfo(ArrayRef<uint64_t> Record,
+                                    unsigned SpecCost);
   std::vector<FunctionSummary::EdgeTy> makeCallList(ArrayRef<uint64_t> Record,
+                                                    uint64_t Version,
                                                     bool IsOldProfileFormat,
                                                     bool HasProfile,
                                                     bool HasRelBF);
+  ArgUsage makeArgUsages(ArrayRef<uint64_t> Record, unsigned &I);
   Error parseEntireSummary(unsigned ID);
   Error parseModuleStringTable();
   void parseTypeIdCompatibleVtableSummaryRecord(ArrayRef<uint64_t> Record);
   void parseTypeIdCompatibleVtableInfo(ArrayRef<uint64_t> Record, size_t &Slot,
                                        TypeIdCompatibleVtableInfo &TypeId);
   std::vector<FunctionSummary::ParamAccess>
   parseParamAccesses(ArrayRef<uint64_t> Record);
 

 ModuleSummaryIndexBitcodeReader::makeRefList(ArrayRef<uint64_t> Record) {
   std::vector<ValueInfo> Ret;
   Ret.reserve(Record.size());
   for (uint64_t RefValueId : Record)
     Ret.push_back(getValueInfoFromValueId(RefValueId).first);
   return Ret;
 }
 
+FuncSpecCostInfo
+ModuleSummaryIndexBitcodeReader::makeSpecCostInfo(ArrayRef<uint64_t> Record,
+                                                  unsigned SpecCost) {
+  MapVector<unsigned, unsigned> BonusBaseMap;
+  auto size = Record.size();
+  assert(size % 2 == 0 && "The size of base bonus should be even!");
+  for (unsigned i = 0; i < size; i += 2)
+    BonusBaseMap.insert({Record[i], Record[i + 1]});
+  return FuncSpecCostInfo(SpecCost, std::move(BonusBaseMap));
+}
+
+ArgUsage
+ModuleSummaryIndexBitcodeReader::makeArgUsages(ArrayRef<uint64_t> Record,
+                                               unsigned &I) {
+  SmallVector<std::pair<unsigned, unsigned>, 4> Uses;
+  size_t Size = Record[I++];
+  assert(Size % 2 == 0);
+  assert(I + Size <= Record.size());
+  Size += I;
+  for (; I != Size; I += 2)
+    Uses.push_back({Record[I], Record[I + 1]});
+  return ArgUsage(std::move(Uses));
+}
+
 std::vector<FunctionSummary::EdgeTy>
 ModuleSummaryIndexBitcodeReader::makeCallList(ArrayRef<uint64_t> Record,
+                                              uint64_t Version,
                                               bool IsOldProfileFormat,
                                               bool HasProfile, bool HasRelBF) {
   std::vector<FunctionSummary::EdgeTy> Ret;
   Ret.reserve(Record.size());
   for (unsigned I = 0, E = Record.size(); I != E; ++I) {
     CalleeInfo::HotnessType Hotness = CalleeInfo::HotnessType::Unknown;
     uint64_t RelBF = 0;
     ValueInfo Callee = getValueInfoFromValueId(Record[I]).first;
+    ArgUsage Uses;
+    if (Version >= 10) {
+      Uses = makeArgUsages(Record, ++I);
+      I -= 1; // Match original order.
+    }
     if (IsOldProfileFormat) {
       I += 1; // Skip old callsitecount field
       if (HasProfile)
         I += 1; // Skip old profilecount field
     } else if (HasProfile)
       Hotness = static_cast<CalleeInfo::HotnessType>(Record[++I]);
     else if (HasRelBF)
       RelBF = Record[++I];
-    Ret.push_back(FunctionSummary::EdgeTy{Callee, CalleeInfo(Hotness, RelBF)});
+    Ret.push_back(FunctionSummary::EdgeTy{
+        Callee, CalleeInfo(Hotness, RelBF, std::move(Uses))});
   }
   return Ret;
 }
 
 static void
 parseWholeProgramDevirtResolutionByArg(ArrayRef<uint64_t> Record, size_t &Slot,
                                        WholeProgramDevirtResolution &Wpd) {
   uint64_t ArgNum = Record[Slot++];
while (true) {
     case bitc::FS_PERMODULE_PROFILE: {
       unsigned ValueID = Record[0];
       uint64_t RawFlags = Record[1];
       unsigned InstCount = Record[2];
       uint64_t RawFunFlags = 0;
       unsigned NumRefs = Record[3];
       unsigned NumRORefs = 0, NumWORefs = 0;
       int RefListStartIndex = 4;
+      unsigned SpecCost = ~(unsigned)0;
+      unsigned NumBaseBonus = 0;
       if (Version >= 4) {
         RawFunFlags = Record[3];
         NumRefs = Record[4];
         RefListStartIndex = 5;
         if (Version >= 5) {
           NumRORefs = Record[5];
           RefListStartIndex = 6;
           if (Version >= 7) {
             NumWORefs = Record[6];
             RefListStartIndex = 7;
+            if (Version >= 10) {
+              SpecCost = Record[7];
+              NumBaseBonus = Record[8];
+              RefListStartIndex = 9;
+            }
           }
         }
       }
 
       auto Flags = getDecodedGVSummaryFlags(RawFlags, Version);
       // The module path string ref set in the summary must be owned by the
       // index's module string table. Since we don't have a module path
       // string table section in the per-module index, we create a single
       // module path string table entry with an empty (0) ID to take
       // ownership.
-      int CallGraphEdgeStartIndex = RefListStartIndex + NumRefs;
-      assert(Record.size() >= RefListStartIndex + NumRefs &&
+      unsigned FuncSpecStartIndex = RefListStartIndex + NumRefs;
+      unsigned CallGraphEdgeStartIndex = FuncSpecStartIndex + NumBaseBonus;
+      assert(Record.size() >= CallGraphEdgeStartIndex &&
              "Record size inconsistent with number of references");
       std::vector<ValueInfo> Refs = makeRefList(
           ArrayRef<uint64_t>(Record).slice(RefListStartIndex, NumRefs));
+      FuncSpecCostInfo FSCI = makeSpecCostInfo(
+          ArrayRef<uint64_t>(Record).slice(FuncSpecStartIndex, NumBaseBonus),
+          SpecCost);
       bool HasProfile = (BitCode == bitc::FS_PERMODULE_PROFILE);
       bool HasRelBF = (BitCode == bitc::FS_PERMODULE_RELBF);
       std::vector<FunctionSummary::EdgeTy> Calls = makeCallList(
           ArrayRef<uint64_t>(Record).slice(CallGraphEdgeStartIndex),
-          IsOldProfileFormat, HasProfile, HasRelBF);
+          Version, IsOldProfileFormat, HasProfile, HasRelBF);
       setSpecialRefs(Refs, NumRORefs, NumWORefs);
       auto FS = std::make_unique<FunctionSummary>(
           Flags, InstCount, getDecodedFFlags(RawFunFlags), /*EntryCount=*/0,
           std::move(Refs), std::move(Calls), std::move(PendingTypeTests),
           std::move(PendingTypeTestAssumeVCalls),
           std::move(PendingTypeCheckedLoadVCalls),
           std::move(PendingTypeTestAssumeConstVCalls),
           std::move(PendingTypeCheckedLoadConstVCalls),
-          std::move(PendingParamAccesses));
+          std::move(PendingParamAccesses), std::move(FSCI));
       auto VIAndOriginalGUID = getValueInfoFromValueId(ValueID);
       FS->setModulePath(getThisModule()->first());
       FS->setOriginalName(VIAndOriginalGUID.second);
       TheIndex.addGlobalValueSummary(VIAndOriginalGUID.first, std::move(FS));
       break;
     }
     // FS_ALIAS: [valueid, flags, valueid]
     // Aliases must be emitted (and parsed) after all FS_PERMODULE entries, as
case bitc::FS_COMBINED_PROFILE: {
       uint64_t ModuleId = Record[1];
       uint64_t RawFlags = Record[2];
       unsigned InstCount = Record[3];
       uint64_t RawFunFlags = 0;
       uint64_t EntryCount = 0;
       unsigned NumRefs = Record[4];
       unsigned NumRORefs = 0, NumWORefs = 0;
       int RefListStartIndex = 5;
+      unsigned SpecCost = ~(unsigned)0;
+      unsigned NumBaseBonus = 0;
 
       if (Version >= 4) {
         RawFunFlags = Record[4];
         RefListStartIndex = 6;
         size_t NumRefsIndex = 5;
         if (Version >= 5) {
           unsigned NumRORefsOffset = 1;
           RefListStartIndex = 7;
           if (Version >= 6) {
             NumRefsIndex = 6;
             EntryCount = Record[5];
             RefListStartIndex = 8;
             if (Version >= 7) {
               RefListStartIndex = 9;
               NumWORefs = Record[8];
               NumRORefsOffset = 2;
+
+              if (Version >= 10) {
+                SpecCost = Record[9];
+                NumBaseBonus = Record[10];
+                RefListStartIndex = 11;
+                NumRORefsOffset = 4;
+              }
             }
           }
           NumRORefs = Record[RefListStartIndex - NumRORefsOffset];
         }
         NumRefs = Record[NumRefsIndex];
       }
 
       auto Flags = getDecodedGVSummaryFlags(RawFlags, Version);
-      int CallGraphEdgeStartIndex = RefListStartIndex + NumRefs;
-      assert(Record.size() >= RefListStartIndex + NumRefs &&
+
+      uint64_t FuncSpecStartIndex = RefListStartIndex + NumRefs;
+      uint64_t CallGraphEdgeStartIndex = FuncSpecStartIndex + NumBaseBonus;
+      assert(Record.size() >= CallGraphEdgeStartIndex &&
              "Record size inconsistent with number of references");
       std::vector<ValueInfo> Refs = makeRefList(
           ArrayRef<uint64_t>(Record).slice(RefListStartIndex, NumRefs));
+      FuncSpecCostInfo FSCI = makeSpecCostInfo(
+          ArrayRef<uint64_t>(Record).slice(FuncSpecStartIndex, NumBaseBonus),
+          SpecCost);
       bool HasProfile = (BitCode == bitc::FS_COMBINED_PROFILE);
       std::vector<FunctionSummary::EdgeTy> Edges = makeCallList(
           ArrayRef<uint64_t>(Record).slice(CallGraphEdgeStartIndex),
-          IsOldProfileFormat, HasProfile, false);
+          Version, IsOldProfileFormat, HasProfile, false);
       ValueInfo VI = getValueInfoFromValueId(ValueID).first;
       setSpecialRefs(Refs, NumRORefs, NumWORefs);
       auto FS = std::make_unique<FunctionSummary>(
           Flags, InstCount, getDecodedFFlags(RawFunFlags), EntryCount,
           std::move(Refs), std::move(Edges), std::move(PendingTypeTests),
           std::move(PendingTypeTestAssumeVCalls),
           std::move(PendingTypeCheckedLoadVCalls),
           std::move(PendingTypeTestAssumeConstVCalls),
           std::move(PendingTypeCheckedLoadConstVCalls),
-          std::move(PendingParamAccesses));
+          std::move(PendingParamAccesses), std::move(FSCI));
       LastSeenSummary = FS.get();
       LastSeenGUID = VI.getGUID();
       FS->setModulePath(ModuleIdMap[ModuleId]);
       TheIndex.addGlobalValueSummary(VI, std::move(FS));
       break;
     }
     // FS_COMBINED_ALIAS: [valueid, modid, flags, valueid]
     // Aliases must be emitted (and parsed) after all FS_COMBINED entries, as

llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

void ModuleBitcodeWriterBase::writePerModuleFunctionSummaryRecord(
 
   auto SpecialRefCnts = FS->specialRefCounts();
   NameVals.push_back(getEncodedGVSummaryFlags(FS->flags()));
   NameVals.push_back(FS->instCount());
   NameVals.push_back(getEncodedFFlags(FS->fflags()));
   NameVals.push_back(FS->refs().size());
   NameVals.push_back(SpecialRefCnts.first);  // rorefcnt
   NameVals.push_back(SpecialRefCnts.second); // worefcnt
+  NameVals.push_back(FS->getSpecializeCost());
+  auto BonusBases = FS->getSpecBonusBase();
+  NameVals.push_back(BonusBases.size() * 2);
 
   for (auto &RI : FS->refs())
     NameVals.push_back(VE.getValueID(RI.getValue()));
 
+  for (auto &I : BonusBases) {
+    NameVals.push_back(I.first);
+    NameVals.push_back(I.second);
+  }
+
   bool HasProfileData =
       F.hasProfileData() || ForceSummaryEdgesCold != FunctionSummary::FSHT_None;
   for (auto &ECI : FS->calls()) {
     NameVals.push_back(getValueId(ECI.first));
+    NameVals.push_back(ECI.second.getUsagesSize() * 2);
+    auto Uses = ECI.second.getUsages();
+    for (auto &IndexValuePair : Uses) {
+      NameVals.push_back(IndexValuePair.first);
+      NameVals.push_back(IndexValuePair.second);
+    }
     if (HasProfileData)
       NameVals.push_back(static_cast<uint8_t>(ECI.second.Hotness));
     else if (WriteRelBFToSummary)
       NameVals.push_back(ECI.second.RelBlockFreq);
   }
 
   unsigned FSAbbrev = (HasProfileData ? FSCallsProfileAbbrev : FSCallsAbbrev);
   unsigned Code =
forEachSummary([&](GVInfo I, bool IsAliasee) {
     NameVals.push_back(FS->instCount());
     NameVals.push_back(getEncodedFFlags(FS->fflags()));
     NameVals.push_back(FS->entryCount());
 
     // Fill in below
     NameVals.push_back(0); // numrefs
     NameVals.push_back(0); // rorefcnt
     NameVals.push_back(0); // worefcnt
+    NameVals.push_back(FS->getSpecializeCost());
+    auto BonusBases = FS->getSpecBonusBase();
+    NameVals.push_back(BonusBases.size() * 2);
 
     unsigned Count = 0, RORefCnt = 0, WORefCnt = 0;
     for (auto &RI : FS->refs()) {
       auto RefValueId = getValueId(RI.getGUID());
       if (!RefValueId)
         continue;
       NameVals.push_back(*RefValueId);
       if (RI.isReadOnly())
         RORefCnt++;
       else if (RI.isWriteOnly())
         WORefCnt++;
       Count++;
     }
     NameVals[6] = Count;
     NameVals[7] = RORefCnt;
     NameVals[8] = WORefCnt;
 
+    for (auto &I : BonusBases) {
+      NameVals.push_back(I.first);
+      NameVals.push_back(I.second);
+    }
+
     bool HasProfileData = false;
     for (auto &EI : FS->calls()) {
       HasProfileData |=
           EI.second.getHotness() != CalleeInfo::HotnessType::Unknown;
       if (HasProfileData)
         break;
     }
 
     for (auto &EI : FS->calls()) {
       // If this GUID doesn't have a value id, it doesn't have a function
       // summary and we don't need to record any calls to it.
       Optional<unsigned> CallValueId = GetValueId(EI.first);
       if (!CallValueId)
         continue;
       NameVals.push_back(*CallValueId);
+      NameVals.push_back(EI.second.getUsagesSize() * 2);
+      auto Uses = EI.second.getUsages();
+      for (auto &IndexValuePair : Uses) {
+        NameVals.push_back(IndexValuePair.first);
+        NameVals.push_back(IndexValuePair.second);
+      }
       if (HasProfileData)
         NameVals.push_back(static_cast<uint8_t>(EI.second.Hotness));
     }
 
     unsigned FSAbbrev = (HasProfileData ? FSCallsProfileAbbrev : FSCallsAbbrev);
     unsigned Code =
         (HasProfileData ? bitc::FS_COMBINED_PROFILE : bitc::FS_COMBINED);
 

llvm/lib/IR/AsmWriter.cpp

if (FFlags.ReadNone | FFlags.ReadOnly | FFlags.NoRecurse |
     Out << "readNone: " << FFlags.ReadNone;
     Out << ", readOnly: " << FFlags.ReadOnly;
     Out << ", noRecurse: " << FFlags.NoRecurse;
     Out << ", returnDoesNotAlias: " << FFlags.ReturnDoesNotAlias;
     Out << ", noInline: " << FFlags.NoInline;
     Out << ", alwaysInline: " << FFlags.AlwaysInline;
     Out << ")";
   }
+
+  /// FIXME: Add writer for new infomation added for function specialization.
+
   if (!FS->calls().empty()) {
     Out << ", calls: (";
     FieldSeparator IFS;
     for (auto &Call : FS->calls()) {
       Out << IFS;
       Out << "(callee: ^" << Machine.getGUIDSlot(Call.first.getGUID());
       if (Call.second.getHotness() != CalleeInfo::HotnessType::Unknown)
         Out << ", hotness: " << getHotnessName(Call.second.getHotness());

llvm/lib/LTO/ThinLTOCodeGenerator.cpp

ProcessThinLTOModule(Module &TheModule, ModuleSummaryIndex &Index,
   saveTempBitcode(TheModule, SaveTempsDir, count, ".4.opt.bc");
 
   if (DisableCodeGen) {
     // Configured to stop before CodeGen, serialize the bitcode and return.
     SmallVector<char, 128> OutputBuffer;
     {
       raw_svector_ostream OS(OutputBuffer);
       ProfileSummaryInfo PSI(TheModule);
-      auto Index = buildModuleSummaryIndex(TheModule, nullptr, &PSI);
+      auto Index = buildModuleSummaryIndex(TheModule, nullptr, nullptr, &PSI);
       WriteBitcodeToFile(TheModule, OS, true, &Index);
     }
     return std::make_unique<SmallVectorMemoryBuffer>(std::move(OutputBuffer));
   }
 
   return codegenModule(TheModule, TM);
 }
 

llvm/lib/Transforms/IPO/FunctionImport.cpp

SummaryFile("summary-file",
                 cl::desc("The summary file to use for function importing."));
 
 /// Used when testing importing from distributed indexes via opt
 // -function-import.
 static cl::opt<bool>
     ImportAllIndex("import-all-index",
                    cl::desc("Import all external functions in index."));
 
+static cl::opt<bool>
+    UseFuncSpecCostInfo("import-use-func-spec-info",
+                        cl::desc("Considering function specialization cost "
+                                 "infomation when importing functions.\n"),
+                        cl::Hidden, cl::init(true));
+
 // Load lazily a module from \p FileName in \p Context.
 static std::unique_ptr<Module> loadFile(const std::string &FileName,
                                         LLVMContext &Context) {
   SMDiagnostic Err;
   LLVM_DEBUG(dbgs() << "Loading '" << FileName << "'\n");
   // Metadata isn't loaded until functions are imported, to minimize
   // the memory overhead.
   std::unique_ptr<Module> Result =

 ///
 /// FIXME: select "best" instead of first that fits. But what is "best"?
 /// - The smallest: more likely to be inlined.
 /// - The one with the least outgoing edges (already well optimized).
 /// - One from a module already being imported from in order to reduce the
 ///   number of source modules parsed/linked.
 /// - One that has PGO data attached.
 /// - [insert you fancy metric here]
-static const GlobalValueSummary *
-selectCallee(const ModuleSummaryIndex &Index,
-             ArrayRef<std::unique_ptr<GlobalValueSummary>> CalleeSummaryList,
-             unsigned Threshold, StringRef CallerModulePath,
-             FunctionImporter::ImportFailureReason &Reason,
-             GlobalValue::GUID GUID) {
+static const GlobalValueSummary *selectCallee(
+    const ModuleSummaryIndex &Index,
+    const CalleeInfo &CI, unsigned Threshold, StringRef CallerModulePath,
+    FunctionImporter::ImportFailureReason &Reason, GlobalValue::GUID GUID) {
   Reason = FunctionImporter::ImportFailureReason::None;
   auto It = llvm::find_if(
       CalleeSummaryList,
       [&](const std::unique_ptr<GlobalValueSummary> &SummaryPtr) {
         auto *GVSummary = SummaryPtr.get();
         if (!Index.isGlobalValueLive(GVSummary)) {
           Reason = FunctionImporter::ImportFailureReason::NotLive;
           return false;
auto It = llvm::find_if(
         if (GlobalValue::isLocalLinkage(Summary->linkage()) &&
             CalleeSummaryList.size() > 1 &&
             Summary->modulePath() != CallerModulePath) {
           Reason =
               FunctionImporter::ImportFailureReason::LocalLinkageNotInModule;
           return false;
         }
 
+        Reason = [&](FunctionSummary *Summary) {
           if ((Summary->instCount() > Threshold) &&
-            !Summary->fflags().AlwaysInline && !ForceImportAll) {
-          Reason = FunctionImporter::ImportFailureReason::TooLarge;
-          return false;
-        }
+              !Summary->fflags().AlwaysInline && !ForceImportAll)
+            return FunctionImporter::ImportFailureReason::TooLarge;
 
           // Skip if it isn't legal to import (e.g. may reference unpromotable
           // locals).
-        if (Summary->notEligibleToImport()) {
-          Reason = FunctionImporter::ImportFailureReason::NotEligible;
-          return false;
-        }
+          if (Summary->notEligibleToImport())
+            return FunctionImporter::ImportFailureReason::NotEligible;
 
           // Don't bother importing if we can't inline it anyway.
-        if (Summary->fflags().NoInline && !ForceImportAll) {
-          Reason = FunctionImporter::ImportFailureReason::NoInline;
+          if (Summary->fflags().NoInline && !ForceImportAll)
+            return FunctionImporter::ImportFailureReason::NoInline;
+
+          return FunctionImporter::ImportFailureReason::None;
+        }(Summary);
+
+        if (Reason == FunctionImporter::ImportFailureReason::None)
+          return true;
+
+        if (!UseFuncSpecCostInfo)
+          return false;
+
+        if (!Summary->shouldImport(CI)) {
+          /// FIXME: add new failure type.
+          Reason = FunctionImporter::ImportFailureReason::NotEligible;
           return false;
         }
 
         return true;
       });
   if (It == CalleeSummaryList.end())
     return nullptr;
 
if (CalleeSummary) {
           assert(FailureInfo &&
                  "Expected FailureInfo for previously rejected candidate");
           FailureInfo->Attempts++;
         }
         continue;
       }
 
       FunctionImporter::ImportFailureReason Reason;
-      CalleeSummary = selectCallee(Index, VI.getSummaryList(), NewThreshold,
-                                   Summary.modulePath(), Reason, VI.getGUID());
+      CalleeSummary =
+          selectCallee(Index, VI.getSummaryList(), Edge.second, NewThreshold,
       if (!CalleeSummary) {
         // Update with new larger threshold if this was a retry (otherwise
         // we would have already inserted with NewThreshold above). Also
         // update failure info if requested.
         if (PreviouslyVisited) {
           ProcessedThreshold = NewThreshold;
           if (PrintImportFailures) {
             assert(FailureInfo &&
if (CalleeSummary) {
           continue;
         }
       }
 
       // "Resolve" the summary
       CalleeSummary = CalleeSummary->getBaseObject();
       ResolvedCalleeSummary = cast<FunctionSummary>(CalleeSummary);
 
+      /// FIXME: add cheap condition for assert function specialization.
       assert((ResolvedCalleeSummary->fflags().AlwaysInline || ForceImportAll ||
               (ResolvedCalleeSummary->instCount() <= NewThreshold)) &&
              "selectCallee() didn't honor the threshold");
 
       auto ExportModulePath = ResolvedCalleeSummary->modulePath();
       auto ILI = ImportList[ExportModulePath].insert(VI.getGUID());
       // We previously decided to import this GUID definition if it was already
       // inserted in the set of imports from the exporting module.

llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp

void splitAndWriteThinLTOBitcode(
     raw_ostream &OS, raw_ostream *ThinLinkOS,
     function_ref<AAResults &(Function &)> AARGetter, Module &M) {
   std::string ModuleId = getUniqueModuleId(&M);
   if (ModuleId.empty()) {
     // We couldn't generate a module ID for this module, write it out as a
     // regular LTO module with an index for summary-based dead stripping.
     ProfileSummaryInfo PSI(M);
     M.addModuleFlag(Module::Error, "ThinLTO", uint32_t(0));
-    ModuleSummaryIndex Index = buildModuleSummaryIndex(M, nullptr, &PSI);
+    ModuleSummaryIndex Index =
+        buildModuleSummaryIndex(M, nullptr, nullptr, &PSI);
     WriteBitcodeToFile(M, OS, /*ShouldPreserveUseListOrder=*/false, &Index);
 
     if (ThinLinkOS)
       // We don't have a ThinLTO part, but still write the module to the
       // ThinLinkOS if requested so that the expected output file is produced.
       WriteBitcodeToFile(M, *ThinLinkOS, /*ShouldPreserveUseListOrder=*/false,
                          &Index);
 
if (!Symvers.empty()) {
     for (auto MD : Symvers)
       NMD->addOperand(MD);
   }
 
   simplifyExternals(*MergedM);
 
   // FIXME: Try to re-use BSI and PFI from the original module here.
   ProfileSummaryInfo PSI(M);
-  ModuleSummaryIndex Index = buildModuleSummaryIndex(M, nullptr, &PSI);
+  ModuleSummaryIndex Index = buildModuleSummaryIndex(M, nullptr, nullptr, &PSI);
 
   // Mark the merged module as requiring full LTO. We still want an index for
   // it though, so that it can participate in summary-based dead stripping.
   MergedM->addModuleFlag(Module::Error, "ThinLTO", uint32_t(0));
   ModuleSummaryIndex MergedMIndex =
-      buildModuleSummaryIndex(*MergedM, nullptr, &PSI);
+      buildModuleSummaryIndex(*MergedM, nullptr, nullptr, &PSI);
 
   SmallVector<char, 0> Buffer;
 
   BitcodeWriter W(Buffer);
   // Save the module hash produced for the full bitcode, which will
   // be used in the backends, and use that in the minimized bitcode
   // produced for the full link.
   ModuleHash ModHash = {{0}};
if (!ModuleId.empty()) {
       // FIXME: Probably should not bother building the index at all
       // in the caller of writeThinLTOBitcode (which does so via the
       // ModuleSummaryIndexAnalysis pass), since we have to rebuild it
       // anyway whenever there is type metadata (here or in
       // splitAndWriteThinLTOBitcode). Just always build it once via the
       // buildModuleSummaryIndex when Module(s) are ready.
       ProfileSummaryInfo PSI(M);
       NewIndex = std::make_unique<ModuleSummaryIndex>(
-          buildModuleSummaryIndex(M, nullptr, &PSI));
+          buildModuleSummaryIndex(M, nullptr, nullptr, &PSI));
       Index = NewIndex.get();
     }
   }
 
   // Write it out as an unsplit ThinLTO module.
 
   // Save the module hash produced for the full bitcode, which will
   // be used in the backends, and use that in the minimized bitcode

llvm/lib/Transforms/IPO/WholeProgramDevirt.cpp

if (Callee.getSummaryList().empty())
     return false;
 
   // Insert calls into the summary index so that the devirtualized targets
   // are eligible for import.
   // FIXME: Annotate type tests with hotness. For now, mark these as hot
   // to better ensure we have the opportunity to inline them.
   bool IsExported = false;
   auto &S = Callee.getSummaryList()[0];
-  CalleeInfo CI(CalleeInfo::HotnessType::Hot, /* RelBF = */ 0);
+  CalleeInfo CI(CalleeInfo::HotnessType::Hot, /* RelBF = */ 0, ArgUsage());
   auto AddCalls = [&](CallSiteInfo &CSInfo) {
     for (auto *FS : CSInfo.SummaryTypeCheckedLoadUsers) {
       FS->addCall({Callee, CI});
       IsExported |= S->modulePath() != FS->modulePath();
     }
     for (auto *FS : CSInfo.SummaryTypeTestAssumeUsers) {
       FS->addCall({Callee, CI});
       IsExported |= S->modulePath() != FS->modulePath();

llvm/test/Bitcode/summary_version.ll

 ; Check summary versioning
 ; RUN: opt  -module-summary  %s -o - | llvm-bcanalyzer -dump | FileCheck %s
 
 ; CHECK: <GLOBALVAL_SUMMARY_BLOCK
-; CHECK: <VERSION op0=9/>
+; CHECK: <VERSION op0=10/>
 
 
 
 ; Need a function for the summary to be populated.
 define void @foo() {
     ret void
 }

llvm/test/Bitcode/thinlto-alias.ll

 ; CHECK-NEXT: <FUNCTION op0=0 op1=4
 ; "analias"
 ; CHECK-NEXT: <FUNCTION op0=4 op1=7
 ; CHECK:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
 ; See if the call to func is registered.
 ; The value id 1 matches the second FUNCTION record above.
-; CHECK-NEXT:    <PERMODULE {{.*}} op7=1/>
+; CHECK-NEXT:    <PERMODULE {{.*}} op9=1 op10=0/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=1/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; CHECK: <STRTAB_BLOCK
 ; CHECK-NEXT: blob data = 'mainanalias{{.*}}'
 
 ; COMBINED:       <GLOBALVAL_SUMMARY_BLOCK
 ; COMBINED-NEXT:    <VERSION
 ; COMBINED-NEXT:    <FLAGS
 ; See if the call to analias is registered, using the expected value id.
 ; COMBINED-NEXT:    <VALUE_GUID op0=[[ALIASID:[0-9]+]] op1=-5751648690987223394/>
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID op0=[[ALIASEEID:[0-9]+]] op1=-1039159065113703048/>
-; COMBINED-NEXT:    <COMBINED {{.*}} op9=[[ALIASID]]/>
+; COMBINED-NEXT:    <COMBINED {{.*}} op11=[[ALIASID]] op12=0/>
 ; COMBINED-NEXT:    <COMBINED {{.*}}
 ; COMBINED-NEXT:    <COMBINED_ALIAS  {{.*}} op3=[[ALIASEEID]]
 ; COMBINED-NEXT:    <BLOCK_COUNT op0=2/>
 ; COMBINED-NEXT:  </GLOBALVAL_SUMMARY_BLOCK
 
 ; ModuleID = 'thinlto-function-summary-callgraph.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

llvm/test/Bitcode/thinlto-alias2.ll

 ; Test to check the callgraph for call to alias in module.
 ; RUN: opt -module-summary %s -o %t.o
 ; RUN: llvm-bcanalyzer -dump %t.o | FileCheck %s
 
 ; CHECK:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
-; CHECK-NEXT:    <PERMODULE {{.*}} op4=0 op5=0 op6=0 op7=[[ALIASID:[0-9]+]]/>
+; CHECK-NEXT:    <PERMODULE {{.*}} op9=[[ALIASID:[0-9]+]] op10=0/>
 ; CHECK-NEXT:    <PERMODULE {{.*}} op0=[[ALIASEEID:[0-9]+]]
 ; CHECK-NEXT:    <ALIAS {{.*}} op0=[[ALIASID]] {{.*}} op2=[[ALIASEEID]]/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=2/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; ModuleID = 'thinlto-alias2.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

llvm/test/Bitcode/thinlto-function-summary-callgraph-cast.ll

 ; Test to check the callgraph for calls to casts.
 ; RUN: opt -module-summary %s -o %t.o
 ; RUN: llvm-bcanalyzer -dump %t.o | FileCheck %s
 ; PR34966
 
 ; CHECK:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
 ; "op7" is a call to "callee" function.
-; CHECK-NEXT:    <PERMODULE {{.*}} op9=3 op10=[[ALIASID:[0-9]+]]/>
+; CHECK-NEXT:    <PERMODULE {{.*}} op9=3 op10=[[ALIASID:[0-9]+]]
 ; "another_caller" has only references but no calls.
-; CHECK-NEXT:    <PERMODULE {{.*}} op4=3 {{.*}} op9={{[0-9]+}}/>
+; CHECK-NEXT:    <PERMODULE {{.*}} op4=3 {{.*}} op9={{[0-9]+}}
 ; CHECK-NEXT:    <PERMODULE {{.*}} op0=[[ALIASEEID:[0-9]+]]
 ; CHECK-NEXT:    <ALIAS {{.*}} op0=[[ALIASID]] {{.*}} op2=[[ALIASEEID]]/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=3/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; ModuleID = 'thinlto-function-summary-callgraph-cast.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

llvm/test/Bitcode/thinlto-function-summary-callgraph-pgo.ll

 ; CHECK: <SOURCE_FILENAME
 ; CHECK-NEXT: <FUNCTION
 ; "func"
 ; CHECK-NEXT: <FUNCTION op0=4 op1=4
 ; CHECK:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
 ; See if the call to func is registered, using the expected hotness type.
-; CHECK-NEXT:    <PERMODULE_PROFILE {{.*}} op7=1 op8=2/>
+; CHECK-NEXT:    <PERMODULE_PROFILE {{.*}} op9=1 op10=0 op11=2/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=1/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 ; CHECK: <STRTAB_BLOCK
 ; CHECK-NEXT: blob data = 'mainfunc{{.*}}'
 
 ; COMBINED:       <GLOBALVAL_SUMMARY_BLOCK
 ; COMBINED-NEXT:    <VERSION
 ; COMBINED-NEXT:    <FLAGS
 ; COMBINED-NEXT:    <VALUE_GUID op0=[[FUNCID:[0-9]+]] op1=7289175272376759421/>
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <COMBINED
 ; See if the call to func is registered, using the expected hotness type.
 ; op6=2 which is hotnessType::None.
-; COMBINED-NEXT:    <COMBINED_PROFILE {{.*}} op9=[[FUNCID]] op10=2/>
+; COMBINED-NEXT:    <COMBINED_PROFILE {{.*}} op11=[[FUNCID]] op12=0 op13=2/>
 ; COMBINED-NEXT:    <BLOCK_COUNT op0=2/>
 ; COMBINED-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; ModuleID = 'thinlto-function-summary-callgraph.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; Function Attrs: nounwind uwtable

llvm/test/Bitcode/thinlto-function-summary-callgraph-profile-summary.ll

 ; CHECK-NEXT: <FUNCTION op0=37 op1=5
 ; "none3"
 ; CHECK-NEXT: <FUNCTION op0=42 op1=5
 ; CHECK-LABEL:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
 ; CHECK-NEXT:    <VALUE_GUID op0=25 op1=123/>
 ; op4=hot1 op6=cold op8=hot2 op10=hot4 op12=none1 op14=hot3 op16=none2 op18=none3 op20=123
-; CHECK-NEXT:    <PERMODULE_PROFILE {{.*}} op7=1 op8=3 op9=5 op10=1 op11=2 op12=3 op13=4 op14=1 op15=6 op16=2 op17=3 op18=3 op19=7 op20=2 op21=8 op22=2 op23=25 op24=4/>
+; CHECK-NEXT:    <PERMODULE_PROFILE {{.*}} op13=1 op14=0 op15=3 op16=5 op17=0 op18=1 op19=2 op20=0 op21=3 op22=4 op23=0 op24=1 op25=6 op26=0 op27=2 op28=3 op29=0 op30=3 op31=7 op32=0 op33=2 op34=8 op35=0 op36=2 op37=25 op38=0 op39=4/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=6/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; CHECK: <STRTAB_BLOCK
 ; CHECK-NEXT: blob data = 'hot_functionhot1hot2hot3hot4coldnone1none2none3{{.*}}'
 
 ; COMBINED:       <GLOBALVAL_SUMMARY_BLOCK
 ; COMBINED-NEXT:    <VERSION
 ; COMBINED-NEXT:    <FLAGS
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
-; COMBINED-NEXT:    <COMBINED_PROFILE {{.*}} op9=[[HOT1:.*]] op10=3 op11=[[COLD:.*]] op12=1 op13=[[HOT2:.*]] op14=3 op15=[[NONE1:.*]] op16=2 op17=[[HOT3:.*]] op18=3 op19=[[NONE2:.*]] op20=2 op21=[[NONE3:.*]] op22=2/>
+; COMBINED-NEXT:    <COMBINED_PROFILE {{.*}} op16=[[HOT1:.*]] op17=3 op18=[[COLD:.*]] op20=1 op21=[[HOT2:.*]] op23=3 op24=[[NONE1:.*]] op26=2 op27=[[HOT3:.*]] op29=3 op30=[[NONE2:.*]] op32=2 op33=[[NONE3:.*]] op35=2/>
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <BLOCK_COUNT op0=13/>
 ; COMBINED-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 
 ; ModuleID = 'thinlto-function-summary-callgraph.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

llvm/test/Bitcode/thinlto-function-summary-callgraph-relbf.ll

 ; Test to check the callgraph in summary
 ; RUN: opt -write-relbf-to-summary -module-summary %s -o %t.o
 ; RUN: llvm-bcanalyzer -dump %t.o | FileCheck %s
 ; RUN: llvm-dis -o - %t.o | FileCheck %s --check-prefix=DIS
 ; Round trip it through llvm-as
 ; RUN: llvm-dis -o - %t.o | llvm-as -write-relbf-to-summary -o - | llvm-dis -o - | FileCheck %s --check-prefix=DIS
 
 ; CHECK: <SOURCE_FILENAME
 ; CHECK-NEXT: <GLOBALVAR
 ; CHECK-NEXT: <FUNCTION
 ; "func"
 ; CHECK-NEXT: <FUNCTION op0=17 op1=4
 ; CHECK:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
 ; See if the call to func is registered.
-; CHECK-NEXT:    <PERMODULE_RELBF {{.*}} op4=1 {{.*}} op9=256
+; CHECK-NEXT:    <PERMODULE_RELBF {{.*}} op4=1 {{.*}} op12=256
 ; CHECK-NEXT:    <BLOCK_COUNT op0=1/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 ; CHECK: <STRTAB_BLOCK
 ; CHECK-NEXT: blob data = 'undefinedglobmainfunc{{.*}}'
 
 
 ; ModuleID = 'thinlto-function-summary-callgraph.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"

llvm/test/Bitcode/thinlto-function-summary-callgraph-sample-profile-summary.ll

 ; CHECK-NEXT: <FUNCTION op0=44 op1=5
 ; CHECK-NEXT: <FUNCTION op0=49 op1=5
 
 ; CHECK-LABEL:       <GLOBALVAL_SUMMARY_BLOCK
 ; CHECK-NEXT:    <VERSION
 ; CHECK-NEXT:    <FLAGS
 ; CHECK-NEXT:    <VALUE_GUID op0=26 op1=123/>
 ; op4=none1 op6=hot1 op8=cold1 op10=none2 op12=hot2 op14=cold2 op16=none3 op18=hot3 op20=cold3 op22=123
-; CHECK-NEXT:    <PERMODULE_PROFILE {{.*}} op7=7 op8=0 op9=1 op10=3 op11=4 op12=1 op13=8 op14=0 op15=2 op16=3 op17=5 op18=1 op19=9 op20=0 op21=3 op22=3 op23=6 op24=1 op25=26 op26=4/>
+; CHECK-NEXT:    <PERMODULE_PROFILE {{.*}} op13=7 op14=0 op15=0 op16=1 op17=0 op18=3 op19=4 op20=0 op21=1 op22=8 op23=0 op24=0 op25=2 op26=0 op27=3 op28=5 op29=0 op30=1 op31=9 op32=0 op33=0 op34=3 op35=0 op36=3 op37=6 op38=0 op39=1 op40=26 op41=0 op42=4/>
 ; CHECK-NEXT:    <BLOCK_COUNT op0=4/>
 ; CHECK-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; CHECK: <STRTAB_BLOCK
 ; CHECK-NEXT: blob data = 'hot_functionhot1hot2hot3cold1cold2cold3none1none2none3{{.*}}'
 
 ; COMBINED:       <GLOBALVAL_SUMMARY_BLOCK
 ; COMBINED-NEXT:    <VERSION

 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <COMBINED abbrevid=
-; COMBINED-NEXT:    <COMBINED_PROFILE {{.*}} op9=[[NONE1:.*]] op10=0 op11=[[HOT1:.*]] op12=3 op13=[[COLD1:.*]] op14=1 op15=[[NONE2:.*]] op16=0 op17=[[HOT2:.*]] op18=3 op19=[[COLD2:.*]] op20=1 op21=[[NONE3:.*]] op22=0 op23=[[HOT3:.*]] op24=3 op25=[[COLD3:.*]] op26=1/>
+; COMBINED-NEXT:    <COMBINED_PROFILE {{.*}} op20=3 op21=[[COLD1:.*]] op23=1 op24=[[NONE2:.*]] op26=0 op27=[[HOT2:.*]] op29=3 op30=[[COLD2:.*]] op32=1 op33=[[NONE3:.*]] op34=0 op35=[[HOT3:.*]] op38=3 op39=[[COLD3:.*]] op41=1/>
 ; COMBINED-NEXT:    <COMBINED abbrevid=
 ; COMBINED-NEXT:    <BLOCK_COUNT op0=13/>
 ; COMBINED-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 
 ; ModuleID = 'thinlto-function-summary-callgraph.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"

llvm/test/Bitcode/thinlto-function-summary-callgraph.ll

 ; COMBINED-NEXT:    <VERSION
 ; COMBINED-NEXT:    <FLAGS
 ; Only 2 VALUE_GUID since reference to undefinedglob should not be included in
 ; combined index.
 ; COMBINED-NEXT:    <VALUE_GUID op0=[[FUNCID:[0-9]+]] op1=7289175272376759421/>
 ; COMBINED-NEXT:    <VALUE_GUID
 ; COMBINED-NEXT:    <COMBINED
 ; See if the call to func is registered.
-; COMBINED-NEXT:    <COMBINED {{.*}} op9=[[FUNCID]]/>
+; COMBINED-NEXT:    <COMBINED {{.*}} op11=[[FUNCID]] op12=0/>
 ; COMBINED-NEXT:    <BLOCK_COUNT op0=2/>
 ; COMBINED-NEXT:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; ModuleID = 'thinlto-function-summary-callgraph.ll'
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; Function Attrs: nounwind uwtable

llvm/test/Bitcode/thinlto-function-summary-refgraph.ll

 ; for calls). Use different linkage types for the various test cases to
 ; distinguish the test cases here (op1 contains the linkage type).
 ; Note that op3 contains the # non-call references.
 ; This also ensures that we didn't include a call or reference to intrinsic
 ; llvm.ctpop.i8.
 ; CHECK:       <GLOBALVAL_SUMMARY_BLOCK
 ; Function main contains call to func, as well as address reference to func:
 ; op0=main op4=func op5=func
-; CHECK-DAG:    <PERMODULE {{.*}} op0=11 op1=0 {{.*}} op4=1 op5=0 op6=0 op7=2 op8=2/>
+; CHECK-DAG:    <PERMODULE {{.*}} op0=11 op1=0 {{.*}} op4=1 op5=0 op6=0 {{.*}} op9=2 op10=2 op11=0/>
 ; Function W contains a call to func3 as well as a reference to globalvar:
 ; op0=W op4=globalvar op5=func3
-; CHECK-DAG:    <PERMODULE {{.*}} op0=6 op1=5 {{.*}} op4=1 op5=0 op6=0 op7=1 op8=5/>
+; CHECK-DAG:    <PERMODULE {{.*}} op0=6 op1=5 {{.*}} op4=1 op5=0 {{.*}} op8=0 op9=1 op10=5
 ; Function X contains call to foo, as well as address reference to foo
 ; which is in the same instruction as the call:
 ; op0=X op4=foo op5=foo
-; CHECK-DAG:    <PERMODULE {{.*}} op0=7 op1=1 {{.*}} op4=1 op5=0 op6=0 op7=4 op8=4/>
+; CHECK-DAG:    <PERMODULE {{.*}} op0=7 op1=1 {{.*}} op9=4 op10=4
 ; Function Y contains call to func2, and ensures we don't incorrectly add
 ; a reference to it when reached while earlier analyzing the phi using its
 ; return value:
 ; op0=Y op4=func2
-; CHECK-DAG:    <PERMODULE {{.*}} op0=8 op1=72 {{.*}} op4=0 op5=0 op6=0 op7=3/>
+; CHECK-DAG:    <PERMODULE {{.*}} op0=8 op1=72 {{.*}} op9=0 op10=8 op11=3 op12=0/>
 ; Function Z contains call to func2, and ensures we don't incorrectly add
 ; a reference to it when reached while analyzing subsequent use of its return
 ; value:
 ; op0=Z op4=func2
-; CHECK-DAG:    <PERMODULE {{.*}} op0=9 op1=3 {{.*}} op4=0 op5=0 op6=0 op7=3/>
+; CHECK-DAG:    <PERMODULE {{.*}} op0=9 op1=3 {{.*}} op6=0 op7=10 op8=0 op9=3 op10=0/>
 ; Variable bar initialization contains address reference to func:
 ; op0=bar op2=func
 ; CHECK-DAG:    <PERMODULE_GLOBALVAR_INIT_REFS {{.*}} op0=0 op1=0 op2=3 op3=2/>
 ; CHECK:  </GLOBALVAL_SUMMARY_BLOCK>
 
 ; CHECK: <STRTAB_BLOCK
 ; CHECK-NEXT: blob data = 'barglobalvarfuncfunc2foofunc3WXYZllvm.ctpop.i8main{{.*}}'