Refactor indirect call promotion profitability analysis (NFC)

[tejohnson]

Summary

Refactored the profitability analysis out of the IC promotion pass and
into lib/Analysis so that it can be accessed by the summary index
builder in a follow-on patch to enable IC promotion in ThinLTO (D21932).

[davidxl]

The new header ICallPromotionAnalysis.h seems missing.

[tejohnson]

Added missing new header file to patch:
include/llvm/Analysis/IndirectCallPromotionAnalysis.h

[tejohnson]

• Add another file missed from refactoring

[davidxl]

include/llvm/Analysis/IndirectCallPromotionAnalysis.h

61

This analysis does not check legality, why is that?

lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

149

Document the parameters including the new one.

262

I think there is a bug in the original code. This check should be moved after legality check below. The update of NumOfPGOICallPromotion should also be done here instead of in tryToPromote.

But this can be fixed in a different patch.

[xur]

lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

262

Which check are you referring to?
These early exits seem good to me: for the *Only check, the icall type only depends on the instruction. it has nothing to do with legality.
For the cutoff check, only it reaches, we can safely exit without doing the other checks. why you think this is a bug?

595

is the last argument correct? I think it should be NumCandidates.

[xur]

include/llvm/Analysis/IndirectCallPromotionAnalysis.h

61

I guess this can be a super set of the real transformations. this just to add edges.

lib/Analysis/IndirectCallPromotionAnalysis.cpp

70

icp-call-only and -icp-invoke-only are mainly for debug. I think it's better to keep together with the transformation code.

[tejohnson]

include/llvm/Analysis/IndirectCallPromotionAnalysis.h

61

Because the legality check is based on the target function, and this is being refactored so that it can be called when the target function is not yet known (when building the summary in the ThinLTO compile step where it is an inter-module indirect call).

[tejohnson]

include/llvm/Analysis/IndirectCallPromotionAnalysis.h

61

I guess this can be a super set of the real transformations. this just to add edges.

Right

lib/Analysis/IndirectCallPromotionAnalysis.cpp

70

Why not have them here and prevent adding the ThinLTO edges in the debug case?

lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

149

Ok

595

Yes I think you are right, let me fix that.

[davidxl]

include/llvm/Analysis/IndirectCallPromotionAnalysis.h

61

ok.

lib/Analysis/IndirectCallPromotionAnalysis.cpp

70

sounds ok to put them here.

[tejohnson]

I am testing the fixed patch, should upload it shortly.

lib/Analysis/IndirectCallPromotionAnalysis.cpp

70

Actually, after discussing with Rong, I have moved these back.

[tejohnson]

• Address review comments.

[xur]

lgtm.

include/llvm/Analysis/IndirectCallPromotionAnalysis.h

+//===- IndirectCallPromotionAnalysis.h - Indirect call analysis -*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+/// \file
+/// Interface to identify indirect call promotion candidates.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_ANALYSIS_INDIRECTCALLPROMOTIONANALYSIS_H
+#define LLVM_ANALYSIS_INDIRECTCALLPROMOTIONANALYSIS_H
+
+#include "llvm/ProfileData/InstrProf.h"
+
+namespace llvm {
+
+class Instruction;
+
+// Class for identifying profitable indirect call promotion candidates when
+// the indirect-call value profile metadata is available.
+class ICallPromotionAnalysis {
+private:
+  // Allocate space to read the profile annotation.
+  std::unique_ptr<InstrProfValueData[]> ValueDataArray;
+
+  // Count is the call count for the direct-call target and
+  // TotalCount is the call count for the indirect-call callsite.
+  // Return true we should promote this indirect-call target.
+  bool isPromotionProfitable(uint64_t Count, uint64_t TotalCount);
+
+  // Returns the number of profitable candidates to promote for the
+  // current ValueDataArray and the given \p Inst.
+  uint32_t getProfitablePromotionCandidates(const Instruction *Inst,
+                                            uint32_t NumVals,
+                                            uint64_t TotalCount);
+
+  // Noncopyable
+  ICallPromotionAnalysis(const ICallPromotionAnalysis &other) = delete;
+  ICallPromotionAnalysis &
+  operator=(const ICallPromotionAnalysis &other) = delete;
+
+public:
+  ICallPromotionAnalysis();
+
+  /// \brief Returns reference to array of InstrProfValueData for the given
+  /// instruction \p I.
+  ///
+  /// The \p NumVals, \p TotalCount and \p NumCandidates
+  /// are set to the number of values in the array, the total profile count
+  /// of the indirect call \p I, and the number of profitable candidates
+  /// in the given array (which is sorted in reverse order of profitability).
+  ///
+  /// The returned array space is owned by this class, and overwritten on
+  /// subsequent calls.
+  ArrayRef<InstrProfValueData>
+  getPromotionCandidatesForInstruction(const Instruction *I, uint32_t &NumVals,
+                                       uint64_t &TotalCount,
+                                       uint32_t &NumCandidates);
+};
+
+} // end namespace llvm
+
+#endif

include/llvm/Analysis/IndirectCallSiteVisitor.h

 //===-- IndirectCallSiteVisitor.h - indirect call-sites visitor -----------===//
 //
 //                      The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements defines a visitor class and a helper function that find
 // all indirect call-sites in a function.
 
 #include "llvm/IR/InstVisitor.h"
 #include <vector>
 
 namespace llvm {
 // Visitor class that finds all indirect call sites.
 struct PGOIndirectCallSiteVisitor
     : public InstVisitor<PGOIndirectCallSiteVisitor> {
   std::vector<Instruction *> IndirectCallInsts;
   PGOIndirectCallSiteVisitor() {}
 
   void visitCallSite(CallSite CS) {
     if (CS.getCalledFunction() || !CS.getCalledValue())
       return;
     Instruction *I = CS.getInstruction();
     if (CallInst *CI = dyn_cast<CallInst>(I)) {
       if (CI->isInlineAsm())
         return;
     }
     if (isa<Constant>(CS.getCalledValue()))
       return;
     IndirectCallInsts.push_back(I);
   }
 };
 
 // Helper function that finds all indirect call sites.
 static inline std::vector<Instruction *> findIndirectCallSites(Function &F) {
   PGOIndirectCallSiteVisitor ICV;
   ICV.visit(F);
   return ICV.IndirectCallInsts;
 }
 }

lib/Analysis/CMakeLists.txt

add_llvm_library(LLVMAnalysis
   DemandedBits.cpp
   DependenceAnalysis.cpp
   DivergenceAnalysis.cpp
   DomPrinter.cpp
   DominanceFrontier.cpp
   EHPersonalities.cpp
   GlobalsModRef.cpp
   IVUsers.cpp
+  IndirectCallPromotionAnalysis.cpp
   InlineCost.cpp
   InstCount.cpp
   InstructionSimplify.cpp
   Interval.cpp
   IntervalPartition.cpp
   IteratedDominanceFrontier.cpp
   LazyCallGraph.cpp
   LazyValueInfo.cpp

lib/Analysis/IndirectCallPromotionAnalysis.cpp

+//===-- IndirectCallPromotionAnalysis.cpp - Find promotion candidates ===//
+//
+//                      The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Helper methods for identifying profitable indirect call promotion
+// candidates for an instruction when the indirect-call value profile metadata
+// is available.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Analysis/IndirectCallPromotionAnalysis.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Analysis/IndirectCallSiteVisitor.h"
+#include "llvm/IR/CallSite.h"
+#include "llvm/IR/DiagnosticInfo.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/InstVisitor.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/ProfileData/InstrProf.h"
+#include "llvm/Support/Debug.h"
+#include <string>
+#include <utility>
+#include <vector>
+
+using namespace llvm;
+
+#define DEBUG_TYPE "pgo-icall-prom-analysis"
+
+// The minimum call count for the direct-call target to be considered as the
+// promotion candidate.
+static cl::opt<unsigned>
+    ICPCountThreshold("icp-count-threshold", cl::Hidden, cl::ZeroOrMore,
+                      cl::init(1000),
+                      cl::desc("The minimum count to the direct call target "
+                               "for the promotion"));
+
+// The percent threshold for the direct-call target (this call site vs the
+// total call count) for it to be considered as the promotion target.
+static cl::opt<unsigned>
+    ICPPercentThreshold("icp-percent-threshold", cl::init(33), cl::Hidden,
+                        cl::ZeroOrMore,
+                        cl::desc("The percentage threshold for the promotion"));
+
+// Set the maximum number of targets to promote for a single indirect-call
+// callsite.
+static cl::opt<unsigned>
+    MaxNumPromotions("icp-max-prom", cl::init(2), cl::Hidden, cl::ZeroOrMore,
+                     cl::desc("Max number of promotions for a single indirect "
+                              "call callsite"));
+
+// If the option is set to true, only call instructions will be considered for
+// transformation -- invoke instructions will be ignored.
+static cl::opt<bool>
+    ICPCallOnly("icp-call-only", cl::init(false), cl::Hidden,
+                cl::desc("Run indirect-call promotion for call instructions "
+                         "only"));
+
+// If the option is set to true, only invoke instructions will be considered for
+// transformation -- call instructions will be ignored.
+static cl::opt<bool> ICPInvokeOnly("icp-invoke-only", cl::init(false),
+                                   cl::Hidden,
+                                   cl::desc("Run indirect-call promotion for "
+                                            "invoke instruction only"));
+
+ICallPromotionAnalysis::ICallPromotionAnalysis() {
+  ValueDataArray = llvm::make_unique<InstrProfValueData[]>(MaxNumPromotions);
+}
+
+bool ICallPromotionAnalysis::isPromotionProfitable(uint64_t Count,
+                                                   uint64_t TotalCount) {
+  if (Count < ICPCountThreshold)
+    return false;
+
+  unsigned Percentage = (Count * 100) / TotalCount;
+  return (Percentage >= ICPPercentThreshold);
+}
+
+// Indirect-call promotion heuristic. The direct targets are sorted based on
+// the count. Stop at the first target that is not promoted. Returns the
+// number of candidates deemed profitable.
+uint32_t ICallPromotionAnalysis::getProfitablePromotionCandidates(
+    const Instruction *Inst, uint32_t NumVals, uint64_t TotalCount) {
+  ArrayRef<InstrProfValueData> ValueDataRef(ValueDataArray.get(), NumVals);
+
+  DEBUG(dbgs() << " \nWork on callsite " << *Inst << " Num_targets: " << NumVals
+               << "\n");
+
+  uint32_t I = 0;
+  for (; I < MaxNumPromotions && I < NumVals; I++) {
+    uint64_t Count = ValueDataRef[I].Count;
+    assert(Count <= TotalCount);
+    uint64_t Target = ValueDataRef[I].Value;
+    DEBUG(dbgs() << " Candidate " << I << " Count=" << Count
+                 << "  Target_func: " << Target << "\n");
+
+    if (ICPInvokeOnly && dyn_cast<CallInst>(Inst)) {
+      DEBUG(dbgs() << " Not promote: User options.\n");
+      return I;
+    }
+    if (ICPCallOnly && dyn_cast<InvokeInst>(Inst)) {
+      DEBUG(dbgs() << " Not promote: User option.\n");
+      return I;
+    }
+    if (!isPromotionProfitable(Count, TotalCount)) {
+      DEBUG(dbgs() << " Not promote: Cold target.\n");
+      return I;
+    }
+    TotalCount -= Count;
+  }
+  return I;
+}
+
+ArrayRef<InstrProfValueData>
+ICallPromotionAnalysis::getPromotionCandidatesForInstruction(
+    const Instruction *I, uint32_t &NumVals, uint64_t &TotalCount,
+    uint32_t &NumCandidates) {
+  bool Res =
+      getValueProfDataFromInst(*I, IPVK_IndirectCallTarget, MaxNumPromotions,
+                               ValueDataArray.get(), NumVals, TotalCount);
+  if (!Res) {
+    NumCandidates = 0;
+    return ArrayRef<InstrProfValueData>();
+  }
+  NumCandidates = getProfitablePromotionCandidates(I, NumVals, TotalCount);
+  return ArrayRef<InstrProfValueData>(ValueDataArray.get(), NumVals);
+}

lib/Transforms/Instrumentation/IndirectCallPromotion.cpp

 //===-- IndirectCallPromotion.cpp - Promote indirect calls to direct calls ===//
 //
 //                      The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the transformation that promotes indirect calls to
 // conditional direct calls when the indirect-call value profile metadata is
 // available.
 //
 //===----------------------------------------------------------------------===//
 
-#include "IndirectCallSiteVisitor.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Analysis/CFG.h"
+#include "llvm/Analysis/IndirectCallPromotionAnalysis.h"
+#include "llvm/Analysis/IndirectCallSiteVisitor.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/IR/InstVisitor.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/MDBuilder.h"

 STATISTIC(NumOfPGOICallPromotion, "Number of indirect call promotions.");
 STATISTIC(NumOfPGOICallsites, "Number of indirect call candidate sites.");
 
 // Command line option to disable indirect-call promotion with the default as
 // false. This is for debug purpose.
 static cl::opt<bool> DisableICP("disable-icp", cl::init(false), cl::Hidden,
                                 cl::desc("Disable indirect call promotion"));
 
-// The minimum call count for the direct-call target to be considered as the
-// promotion candidate.
-static cl::opt<unsigned>
-    ICPCountThreshold("icp-count-threshold", cl::Hidden, cl::ZeroOrMore,
-                      cl::init(1000),
-                      cl::desc("The minimum count to the direct call target "
-                               "for the promotion"));
-
-// The percent threshold for the direct-call target (this call site vs the
-// total call count) for it to be considered as the promotion target.
-static cl::opt<unsigned>
-    ICPPercentThreshold("icp-percent-threshold", cl::init(33), cl::Hidden,
-                        cl::ZeroOrMore,
-                        cl::desc("The percentage threshold for the promotion"));
-
-// Set the maximum number of targets to promote for a single indirect-call
-// callsite.
-static cl::opt<unsigned>
-    MaxNumPromotions("icp-max-prom", cl::init(2), cl::Hidden, cl::ZeroOrMore,
-                     cl::desc("Max number of promotions for a single indirect "
-                              "call callsite"));
-
 // Set the cutoff value for the promotion. If the value is other than 0, we
 // stop the transformation once the total number of promotions equals the cutoff
 // value.
 // For debug use only.
 static cl::opt<unsigned>
     ICPCutOff("icp-cutoff", cl::init(0), cl::Hidden, cl::ZeroOrMore,
               cl::desc("Max number of promotions for this compilaiton"));
 
 // If ICPCSSkip is non zero, the first ICPCSSkip callsites will be skipped.
 // For debug use only.
 static cl::opt<unsigned>
     ICPCSSkip("icp-csskip", cl::init(0), cl::Hidden, cl::ZeroOrMore,
               cl::desc("Skip Callsite up to this number for this compilaiton"));
 
 // Set if the pass is called in LTO optimization. The difference for LTO mode
 // is the pass won't prefix the source module name to the internal linkage
 // symbols.
 static cl::opt<bool> ICPLTOMode("icp-lto", cl::init(false), cl::Hidden,
                                 cl::desc("Run indirect-call promotion in LTO "
                                          "mode"));
-// If the option is set to true, only call instructions will be considered for
-// transformation -- invoke instructions will be ignored.
-static cl::opt<bool>
-    ICPCallOnly("icp-call-only", cl::init(false), cl::Hidden,
-                cl::desc("Run indirect-call promotion for call instructions "
-                         "only"));
-
-// If the option is set to true, only invoke instructions will be considered for
-// transformation -- call instructions will be ignored.
-static cl::opt<bool> ICPInvokeOnly("icp-invoke-only", cl::init(false),
-                                   cl::Hidden,
-                                   cl::desc("Run indirect-call promotion for "
-                                            "invoke instruction only"));
 
 // Dump the function level IR if the transformation happened in this
 // function. For debug use only.
 static cl::opt<bool>
     ICPDUMPAFTER("icp-dumpafter", cl::init(false), cl::Hidden,
                  cl::desc("Dump IR after transformation happens"));
 
 namespace {

 private:
   Function &F;
   Module *M;
 
   // Symtab that maps indirect call profile values to function names and
   // defines.
   InstrProfSymtab *Symtab;
 
-  // Allocate space to read the profile annotation.
-  std::unique_ptr<InstrProfValueData[]> ValueDataArray;
-
-  // Count is the call count for the direct-call target and
-  // TotalCount is the call count for the indirect-call callsite.
-  // Return true we should promote this indirect-call target.
-  bool isPromotionProfitable(uint64_t Count, uint64_t TotalCount);
-
   enum TargetStatus {
     OK,                   // Should be able to promote.
     NotAvailableInModule, // Cannot find the target in current module.
     ReturnTypeMismatch,   // Return type mismatch b/w target and indirect-call.
     NumArgsMismatch,      // Number of arguments does not match.
     ArgTypeMismatch       // Type mismatch in the arguments (cannot bitcast).
   };
 
   // Test if we can legally promote this direct-call of Target.
   TargetStatus isPromotionLegal(Instruction *Inst, uint64_t Target,
                                 Function *&F);
 
   // A struct that records the direct target and it's call count.
   struct PromotionCandidate {
     Function *TargetFunction;
     uint64_t Count;
     PromotionCandidate(Function *F, uint64_t C) : TargetFunction(F), Count(C) {}
   };
 
   // Check if the indirect-call call site should be promoted. Return the number
   // of promotions.
   std::vector<PromotionCandidate> getPromotionCandidatesForCallSite(
       Instruction *Inst, const ArrayRef<InstrProfValueData> &ValueDataRef,
-      uint64_t TotalCount);
+      uint64_t TotalCount, uint32_t NumCandidates);
 
   // Main function that transforms Inst (either a indirect-call instruction, or
   // an invoke instruction , to a conditional call to F. This is like:
   //     if (Inst.CalledValue == F)
   //        F(...);
   //     else
   //        Inst(...);
   //     end
private:
 
   // Noncopyable
   ICallPromotionFunc(const ICallPromotionFunc &other) = delete;
   ICallPromotionFunc &operator=(const ICallPromotionFunc &other) = delete;
 
 public:
   ICallPromotionFunc(Function &Func, Module *Modu, InstrProfSymtab *Symtab)
       : F(Func), M(Modu), Symtab(Symtab) {
-    ValueDataArray = llvm::make_unique<InstrProfValueData[]>(MaxNumPromotions);
   }
   bool processFunction();
 };
 } // end anonymous namespace
 
-bool ICallPromotionFunc::isPromotionProfitable(uint64_t Count,
-                                               uint64_t TotalCount) {
-  if (Count < ICPCountThreshold)
-    return false;
-
-  unsigned Percentage = (Count * 100) / TotalCount;
-  return (Percentage >= ICPPercentThreshold);
-}
-
 ICallPromotionFunc::TargetStatus
 ICallPromotionFunc::isPromotionLegal(Instruction *Inst, uint64_t Target,
                                      Function *&TargetFunction) {
   Function *DirectCallee = Symtab->getFunction(Target);
   if (DirectCallee == nullptr)
     return NotAvailableInModule;
   // Check the return type.
   Type *CallRetType = Inst->getType();
ICallPromotionFunc::isPromotionLegal(Instruction *Inst, uint64_t Target,
   return OK;
 }
 
 // Indirect-call promotion heuristic. The direct targets are sorted based on
 // the count. Stop at the first target that is not promoted.
 std::vector<ICallPromotionFunc::PromotionCandidate>
 ICallPromotionFunc::getPromotionCandidatesForCallSite(
     Instruction *Inst, const ArrayRef<InstrProfValueData> &ValueDataRef,
-    uint64_t TotalCount) {
+    uint64_t TotalCount, uint32_t NumCandidates) {
   uint32_t NumVals = ValueDataRef.size();
   std::vector<PromotionCandidate> Ret;
 
   DEBUG(dbgs() << " \nWork on callsite #" << NumOfPGOICallsites << *Inst
-               << " Num_targets: " << NumVals << "\n");
+               << " Num_targets: " << NumVals
+               << " Num_candidates: " << NumCandidates << "\n");
   NumOfPGOICallsites++;
   if (ICPCSSkip != 0 && NumOfPGOICallsites <= ICPCSSkip) {
     DEBUG(dbgs() << " Skip: User options.\n");
     return Ret;
   }
 
-  for (uint32_t I = 0; I < MaxNumPromotions && I < NumVals; I++) {
+  for (uint32_t I = 0; I < NumCandidates; I++) {
     uint64_t Count = ValueDataRef[I].Count;
     assert(Count <= TotalCount);
     uint64_t Target = ValueDataRef[I].Value;
     DEBUG(dbgs() << " Candidate " << I << " Count=" << Count
                  << "  Target_func: " << Target << "\n");
 
-    if (ICPInvokeOnly && dyn_cast<CallInst>(Inst)) {
-      DEBUG(dbgs() << " Not promote: User options.\n");
-      break;
-    }
-    if (ICPCallOnly && dyn_cast<InvokeInst>(Inst)) {
-      DEBUG(dbgs() << " Not promote: User option.\n");
-      break;
-    }
     if (ICPCutOff != 0 && NumOfPGOICallPromotion >= ICPCutOff) {
       DEBUG(dbgs() << " Not promote: Cutoff reached.\n");
       break;
     }
-    if (!isPromotionProfitable(Count, TotalCount)) {
-      DEBUG(dbgs() << " Not promote: Cold target.\n");
-      break;
-    }
     Function *TargetFunction = nullptr;
     TargetStatus Status = isPromotionLegal(Inst, Target, TargetFunction);
     if (Status != OK) {
       StringRef TargetFuncName = Symtab->getFuncName(Target);
       const char *Reason = StatusToString(Status);
       DEBUG(dbgs() << " Not promote: " << Reason << "\n");
       emitOptimizationRemarkMissed(
           F.getContext(), "pgo-icall-prom", F, Inst->getDebugLoc(),
uint32_t ICallPromotionFunc::tryToPromote(
   }
   return NumPromoted;
 }
 
 // Traverse all the indirect-call callsite and get the value profile
 // annotation to perform indirect-call promotion.
 bool ICallPromotionFunc::processFunction() {
   bool Changed = false;
+  ICallPromotionAnalysis ICallAnalysis;
   for (auto &I : findIndirectCallSites(F)) {
-    uint32_t NumVals;
+    uint32_t NumVals, NumCandidates;
     uint64_t TotalCount;
-    bool Res =
-        getValueProfDataFromInst(*I, IPVK_IndirectCallTarget, MaxNumPromotions,
-                                 ValueDataArray.get(), NumVals, TotalCount);
-    if (!Res)
-      continue;
-    ArrayRef<InstrProfValueData> ValueDataArrayRef(ValueDataArray.get(),
-                                                   NumVals);
-    auto PromotionCandidates =
-        getPromotionCandidatesForCallSite(I, ValueDataArrayRef, TotalCount);
+    auto ICallProfDataRef = ICallAnalysis.getPromotionCandidatesForInstruction(
+        I, NumVals, TotalCount, NumCandidates);
+    if (!NumCandidates)
+      continue;
+    auto PromotionCandidates = getPromotionCandidatesForCallSite(
+        I, ICallProfDataRef, TotalCount, NumCandidates);
     uint32_t NumPromoted = tryToPromote(I, PromotionCandidates, TotalCount);
     if (NumPromoted == 0)
       continue;
 
     Changed = true;
     // Adjust the MD.prof metadata. First delete the old one.
     I->setMetadata(LLVMContext::MD_prof, 0);
     // If all promoted, we don't need the MD.prof metadata.
     if (TotalCount == 0 || NumPromoted == NumVals)
       continue;
     // Otherwise we need update with the un-promoted records back.
-    annotateValueSite(*M, *I, ValueDataArrayRef.slice(NumPromoted), TotalCount,
-                      IPVK_IndirectCallTarget, MaxNumPromotions);
+    annotateValueSite(*M, *I, ICallProfDataRef.slice(NumPromoted), TotalCount,
+                      IPVK_IndirectCallTarget, NumPromoted);
   }
   return Changed;
 }
 
 // A wrapper function that does the actual work.
 static bool promoteIndirectCalls(Module &M, bool InLTO) {
   if (DisableICP)
     return false;

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

 // class of PGOEdge and BBInfo, respectively. They contains extra data structure
 // used in populating profile counters.
 // The MST implementation is in Class CFGMST (CFGMST.h).
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Transforms/PGOInstrumentation.h"
 #include "CFGMST.h"
-#include "IndirectCallSiteVisitor.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/CFG.h"
+#include "llvm/Analysis/IndirectCallSiteVisitor.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/IR/Module.h"