[NFC] FunctionSamples::getEntrySamples -> getHeadSamplesEstimate

[mtrofin]

Summary

The name getEntrySamples was misleading for 2 reasons. One, it's close in name to Function::getEntryCount, but the equivalent here is getHeadSamples; second, as opposed to the other get* APIs in FunctionSamples, it performs an estimate/heuristic rather than just retrieving raw data (or a non-heuristic derivate off that data, like getMaxCountInside)

The new name should more clearly communicate its intent; and, being close (in name) to getHeadSamples, it should allow the reader discover the relation between them.

Also updated the doc comments for both getHeadSamples[Estimate] so a reader may better understand the relation between them.

[hoy]

Thanks for making the change. Agreed that getEntrySamples is confusing when used together with getHeadSamples.

llvm/include/llvm/ProfileData/SampleProf.h

889

nit: how about just name this getEntrySamplesEstimate ? BB is somehow implicated in the contexts where the API is used.

[wenlei]

Maybe getEntrySampleEstimate? the heuristic doesn't guarantee we get the counts for entry BB.

[mtrofin]

In D130281#3669191, @wenlei wrote:

Maybe getEntrySampleEstimate? the heuristic doesn't guarantee we get the counts for entry BB.

It does try to offer an estimate corresponding to that, though, if that's missing - hmmm... how about getHeadSamplesEstimate? It corresponds to what we call "head samples" in FunctionSamples (we're really not using "entry" in these APIs); it's discoverable right next to the getHeadSamples() API, and the "estimate" suffix should offer sufficient of a hint that it's a heuristic, and one would then be incentivized to read the docstring. I'd also, then, highlight in the comment the distinction between the two (i.e. one is raw data, while the "estimate" is, well, trying to produce a number using heuristic, etc)

WDYT?

llvm/include/llvm/ProfileData/SampleProf.h

889

Right, but my goal is improving readability by making the name hint sufficiently well to what the function is about, without requiring the engineer wishing to use it also have to go see how it's used (and understand that code, etc).

[hoy]

In D130281#3669528, @mtrofin wrote:

In D130281#3669191, @wenlei wrote:

Maybe getEntrySampleEstimate? the heuristic doesn't guarantee we get the counts for entry BB.

It does try to offer an estimate corresponding to that, though, if that's missing - hmmm... how about getHeadSamplesEstimate? It corresponds to what we call "head samples" in FunctionSamples (we're really not using "entry" in these APIs); it's discoverable right next to the getHeadSamples() API, and the "estimate" suffix should offer sufficient of a hint that it's a heuristic, and one would then be incentivized to read the docstring. I'd also, then, highlight in the comment the distinction between the two (i.e. one is raw data, while the "estimate" is, well, trying to produce a number using heuristic, etc)

WDYT?

getHeadSamplesEstimate sounds good to me. I don't see a big difference between "head" and "entry" so using one term only should be more clear.

[wenlei]

In D130281#3669528, @mtrofin wrote:

In D130281#3669191, @wenlei wrote:

Maybe getEntrySampleEstimate? the heuristic doesn't guarantee we get the counts for entry BB.

It does try to offer an estimate corresponding to that, though, if that's missing - hmmm... how about getHeadSamplesEstimate? It corresponds to what we call "head samples" in FunctionSamples (we're really not using "entry" in these APIs); it's discoverable right next to the getHeadSamples() API, and the "estimate" suffix should offer sufficient of a hint that it's a heuristic, and one would then be incentivized to read the docstring. I'd also, then, highlight in the comment the distinction between the two (i.e. one is raw data, while the "estimate" is, well, trying to produce a number using heuristic, etc)

WDYT?

Good point, I agreed that's better, for the consistency with term Head. Thanks!

[mtrofin]

updated with agreed name

[wenlei]

lgtm, thanks.

llvm/include/llvm/ProfileData/SampleProf.h

public:
   findFunctionSamplesAt(const LineLocation &Loc, StringRef CalleeName,
                         SampleProfileReaderItaniumRemapper *Remapper) const;
 
   bool empty() const { return TotalSamples == 0; }
 
   /// Return the total number of samples collected inside the function.
   uint64_t getTotalSamples() const { return TotalSamples; }
 
-  /// Return the total number of branch samples that have the function as the
-  /// branch target. This should be equivalent to the sample of the first
-  /// instruction of the symbol. But as we directly get this info for raw
-  /// profile without referring to potentially inaccurate debug info, this
+  /// For top-level functions, return the total number of branch samples that
+  /// have the function as the branch target (or 0 otherwise). This is the raw
+  /// data fetched from the profile. This should be equivalent to the sample of
+  /// the first instruction of the symbol. But as we directly get this info for
+  /// raw profile without referring to potentially inaccurate debug info, this
   /// gives more accurate profile data and is preferred for standalone symbols.
   uint64_t getHeadSamples() const { return TotalHeadSamples; }
 
-  /// Return the sample count of the first instruction of the function.
+  /// Return an estimate of the sample count of the function entry basic block.
   /// The function can be either a standalone symbol or an inlined function.
-  uint64_t getEntrySamples() const {
+  /// For Context-Sensitive profiles, this will prefer returning the head
+  /// samples (i.e. getHeadSamples()), if non-zero. Otherwise it estimates from
+  /// the function body's samples or callsite samples.
+  uint64_t getHeadSamplesEstimate() const {
     if (FunctionSamples::ProfileIsCS && getHeadSamples()) {
       // For CS profile, if we already have more accurate head samples
       // counted by branch sample from caller, use them as entry samples.
       return getHeadSamples();
     }
     uint64_t Count = 0;
     // Use either BodySamples or CallsiteSamples which ever has the smaller
     // lineno.
     if (!BodySamples.empty() &&
         (CallsiteSamples.empty() ||
          BodySamples.begin()->first < CallsiteSamples.begin()->first))
       Count = BodySamples.begin()->second.getSamples();
     else if (!CallsiteSamples.empty()) {
       // An indirect callsite may be promoted to several inlined direct calls.
       // We need to get the sum of them.
       for (const auto &N_FS : CallsiteSamples.begin()->second)
-        Count += N_FS.second.getEntrySamples();
+        Count += N_FS.second.getHeadSamplesEstimate();
     }
     // Return at least 1 if total sample is not 0.
     return Count ? Count : TotalSamples > 0;
   }
 
   /// Return all the samples collected in the body of the function.
   const BodySampleMap &getBodySamples() const { return BodySamples; }
 

llvm/include/llvm/Transforms/IPO/ProfiledCallGraph.h

while (!Queue.empty()) {
         Queue.push(Callee);
 
         // Fetch edge weight from the profile.
         uint64_t Weight;
         FunctionSamples *CalleeSamples = Callee->getFunctionSamples();
         if (!CalleeSamples || !CallerSamples) {
           Weight = 0;
         } else {
-          uint64_t CalleeEntryCount = CalleeSamples->getEntrySamples();
+          uint64_t CalleeEntryCount = CalleeSamples->getHeadSamplesEstimate();
           uint64_t CallsiteCount = 0;
           LineLocation Callsite = Callee->getCallSiteLoc();
           if (auto CallTargets = CallerSamples->findCallTargetMapAt(Callsite)) {
             SampleRecord::CallTargetMap &TargetCounts = CallTargets.get();
             auto It = TargetCounts.find(CalleeSamples->getName());
             if (It != TargetCounts.end())
               CallsiteCount = It->second;
           }
for (const auto &Sample : Samples.getBodySamples()) {
         addProfiledCall(Samples.getFuncName(), Target.first(), Target.second);
       }
     }
 
     for (const auto &CallsiteSamples : Samples.getCallsiteSamples()) {
       for (const auto &InlinedSamples : CallsiteSamples.second) {
         addProfiledFunction(InlinedSamples.first);
         addProfiledCall(Samples.getFuncName(), InlinedSamples.first,
-                        InlinedSamples.second.getEntrySamples());
+                        InlinedSamples.second.getHeadSamplesEstimate());
         addProfiledCalls(InlinedSamples.second);
       }
     }
   }
 
   ProfiledCallGraphNode Root;
   StringMap<ProfiledCallGraphNode> ProfiledFunctions;
 };

llvm/lib/Transforms/IPO/SampleProfile.cpp

SampleProfileLoader::findIndirectCallFunctionSamples(
   std::vector<const FunctionSamples *> R;
 
   if (!DIL) {
     return R;
   }
 
   auto FSCompare = [](const FunctionSamples *L, const FunctionSamples *R) {
     assert(L && R && "Expect non-null FunctionSamples");
-    if (L->getEntrySamples() != R->getEntrySamples())
-      return L->getEntrySamples() > R->getEntrySamples();
+    if (L->getHeadSamplesEstimate() != R->getHeadSamplesEstimate())
+      return L->getHeadSamplesEstimate() > R->getHeadSamplesEstimate();
     return FunctionSamples::getGUID(L->getName()) <
            FunctionSamples::getGUID(R->getName());
   };
 
   if (FunctionSamples::ProfileIsCS) {
     auto CalleeSamples =
         ContextTracker->getIndirectCalleeContextSamplesFor(DIL);
     if (CalleeSamples.empty())
       return R;
 
     // For CSSPGO, we only use target context profile's entry count
     // as that already includes both inlined callee and non-inlined ones..
     Sum = 0;
     for (const auto *const FS : CalleeSamples) {
-      Sum += FS->getEntrySamples();
+      Sum += FS->getHeadSamplesEstimate();
       R.push_back(FS);
     }
     llvm::sort(R, FSCompare);
     return R;
   }
 
   const FunctionSamples *FS = findFunctionSamples(Inst);
   if (FS == nullptr)
     return R;
 
   auto CallSite = FunctionSamples::getCallSiteIdentifier(DIL);
   auto T = FS->findCallTargetMapAt(CallSite);
   Sum = 0;
   if (T)
     for (const auto &T_C : T.get())
       Sum += T_C.second;
   if (const FunctionSamplesMap *M = FS->findFunctionSamplesMapAt(CallSite)) {
     if (M->empty())
       return R;
     for (const auto &NameFS : *M) {
-      Sum += NameFS.second.getEntrySamples();
+      Sum += NameFS.second.getHeadSamplesEstimate();
       R.push_back(&NameFS.second);
     }
     llvm::sort(R, FSCompare);
   }
   return R;
 }
 
 const FunctionSamples *
while (!CalleeList.empty()) {
     // even if callee doesn't have a corresponding context profile.
     if (!CalleeSample)
       continue;
 
     // If pre-inliner decision is used, honor that for importing as well.
     bool PreInline =
         UsePreInlinerDecision &&
         CalleeSample->getContext().hasAttribute(ContextShouldBeInlined);
-    if (!PreInline && CalleeSample->getEntrySamples() < Threshold)
+    if (!PreInline && CalleeSample->getHeadSamplesEstimate() < Threshold)
       continue;
 
     StringRef Name = CalleeSample->getFuncName();
     Function *Func = SymbolMap.lookup(Name);
     // Add to the import list only when it's defined out of module.
     if (!Func || Func->isDeclaration())
       InlinedGUIDs.insert(FunctionSamples::getGUID(CalleeSample->getName()));
 
for (auto &BB : F) {
       for (auto &I : BB.getInstList()) {
         const FunctionSamples *FS = nullptr;
         if (auto *CB = dyn_cast<CallBase>(&I)) {
           if (!isa<IntrinsicInst>(I)) {
             if ((FS = findCalleeFunctionSamples(*CB))) {
               assert((!FunctionSamples::UseMD5 || FS->GUIDToFuncNameMap) &&
                      "GUIDToFuncNameMap has to be populated");
               AllCandidates.push_back(CB);
-              if (FS->getEntrySamples() > 0 || FunctionSamples::ProfileIsCS)
+              if (FS->getHeadSamplesEstimate() > 0 ||
+                  FunctionSamples::ProfileIsCS)
                 LocalNotInlinedCallSites.try_emplace(CB, FS);
               if (callsiteIsHot(FS, PSI, ProfAccForSymsInList))
                 Hot = true;
               else if (shouldInlineColdCallee(*CB))
                 ColdCandidates.push_back(CB);
             } else if (getExternalInlineAdvisorShouldInline(*CB)) {
               AllCandidates.push_back(CB);
             }
for (CallBase *I : CIS) {
           if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
             findExternalInlineCandidate(I, FS, InlinedGUIDs, SymbolMap,
                                         PSI->getOrCompHotCountThreshold());
             continue;
           }
           if (!callsiteIsHot(FS, PSI, ProfAccForSymsInList))
             continue;
 
-          Candidate = {I, FS, FS->getEntrySamples(), 1.0};
+          Candidate = {I, FS, FS->getHeadSamplesEstimate(), 1.0};
           if (tryPromoteAndInlineCandidate(F, Candidate, SumOrigin, Sum)) {
             LocalNotInlinedCallSites.erase(I);
             LocalChanged = true;
           }
         }
       } else if (CalledFunction && CalledFunction->getSubprogram() &&
                  !CalledFunction->isDeclaration()) {
         if (tryInlineCandidate(Candidate)) {
bool SampleProfileLoader::getInlineCandidate(InlineCandidate *NewCandidate,
   if (!CalleeSamples && !getExternalInlineAdvisorShouldInline(*CB))
     return false;
 
   float Factor = 1.0;
   if (Optional<PseudoProbe> Probe = extractProbe(*CB))
     Factor = Probe->Factor;
 
   uint64_t CallsiteCount =
-      CalleeSamples ? CalleeSamples->getEntrySamples() * Factor : 0;
+      CalleeSamples ? CalleeSamples->getHeadSamplesEstimate() * Factor : 0;
   *NewCandidate = {CB, CalleeSamples, CallsiteCount, Factor};
   return true;
 }
 
 Optional<InlineCost>
 SampleProfileLoader::getExternalInlineAdvisorCost(CallBase &CB) {
   std::unique_ptr<InlineAdvice> Advice = nullptr;
   if (ExternalInlineAdvisor) {
if (I->isIndirectCall()) {
       for (const auto *FS : CalleeSamples) {
         // TODO: Consider disable pre-lTO ICP for MonoLTO as well
         if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
           findExternalInlineCandidate(I, FS, InlinedGUIDs, SymbolMap,
                                       PSI->getOrCompHotCountThreshold());
           continue;
         }
         uint64_t EntryCountDistributed =
-            FS->getEntrySamples() * Candidate.CallsiteDistribution;
+            FS->getHeadSamplesEstimate() * Candidate.CallsiteDistribution;
         // In addition to regular inline cost check, we also need to make sure
         // ICP isn't introducing excessive speculative checks even if individual
         // target looks beneficial to promote and inline. That means we should
         // only do ICP when there's a small number dominant targets.
         if (ICPCount >= ProfileICPRelativeHotnessSkip &&
             EntryCountDistributed * 100 < SumOrigin * ProfileICPRelativeHotness)
           break;
         // TODO: Fix CallAnalyzer to handle all indirect calls.
for (const auto &Pair : NonInlinedCallSites) {
     ORE->emit(
         OptimizationRemarkAnalysis(getAnnotatedRemarkPassName(), "NotInline",
                                    I->getDebugLoc(), I->getParent())
         << "previous inlining not repeated: '" << ore::NV("Callee", Callee)
         << "' into '" << ore::NV("Caller", &F) << "'");
 
     ++NumCSNotInlined;
     const FunctionSamples *FS = Pair.getSecond();
-    if (FS->getTotalSamples() == 0 && FS->getEntrySamples() == 0) {
+    if (FS->getTotalSamples() == 0 && FS->getHeadSamplesEstimate() == 0) {
       continue;
     }
 
     // Do not merge a context that is already duplicated into the base profile.
     if (FS->getContext().hasAttribute(sampleprof::ContextDuplicatedIntoBase))
       continue;
 
     if (ProfileMergeInlinee) {
       // A function call can be replicated by optimizations like callsite
       // splitting or jump threading and the replicates end up sharing the
       // sample nested callee profile instead of slicing the original
       // inlinee's profile. We want to do merge exactly once by filtering out
       // callee profiles with a non-zero head sample count.
       if (FS->getHeadSamples() == 0) {
         // Use entry samples as head samples during the merge, as inlinees
         // don't have head samples.
         const_cast<FunctionSamples *>(FS)->addHeadSamples(
-            FS->getEntrySamples());
+            FS->getHeadSamplesEstimate());
 
         // Note that we have to do the merge right after processing function.
         // This allows OutlineFS's profile to be used for annotation during
         // top-down processing of functions' annotation.
         FunctionSamples *OutlineFS = Reader->getOrCreateSamplesFor(*Callee);
         OutlineFS->merge(*FS, 1);
         // Set outlined profile to be synthetic to not bias the inliner.
         OutlineFS->SetContextSynthetic();
       }
     } else {
       auto pair =
           notInlinedCallInfo.try_emplace(Callee, NotInlinedProfileInfo{0});
-      pair.first->second.entryCount += FS->getEntrySamples();
+      pair.first->second.entryCount += FS->getHeadSamplesEstimate();
     }
   }
 }
 
 /// Returns the sorted CallTargetMap \p M by count in descending order.
 static SmallVector<InstrProfValueData, 2>
 GetSortedValueDataFromCallTargets(const SampleRecord::CallTargetMap &M) {
   SmallVector<InstrProfValueData, 2> R;
if (BlockWeights[BB]) {
             Sum += C.second;
           // With CSSPGO all indirect call targets are counted torwards the
           // original indirect call site in the profile, including both
           // inlined and non-inlined targets.
           if (!FunctionSamples::ProfileIsCS) {
             if (const FunctionSamplesMap *M =
                     FS->findFunctionSamplesMapAt(CallSite)) {
               for (const auto &NameFS : *M)
-                Sum += NameFS.second.getEntrySamples();
+                Sum += NameFS.second.getHeadSamplesEstimate();
             }
           }
           if (Sum)
             updateIDTMetaData(I, SortedCallTargets, Sum);
           else if (OverwriteExistingWeights)
             I.setMetadata(LLVMContext::MD_prof, nullptr);
         } else if (!isa<IntrinsicInst>(&I)) {
           I.setMetadata(LLVMContext::MD_prof,

llvm/tools/llvm-profdata/llvm-profdata.cpp

std::string Metric =
       std::string("max sample >= ") + std::to_string(MinCountThreshold);
   std::vector<HotFuncInfo> PrintValues;
   for (const auto &FuncPair : HotFunc) {
     const FunctionSamples &Func = *FuncPair.second.first;
     double TotalSamplePercent =
         (ProfileTotalSample > 0)
             ? (Func.getTotalSamples() * 100.0) / ProfileTotalSample
             : 0;
-    PrintValues.emplace_back(HotFuncInfo(
-        Func.getContext().toString(), Func.getTotalSamples(),
-        TotalSamplePercent, FuncPair.second.second, Func.getEntrySamples()));
+    PrintValues.emplace_back(
+        HotFuncInfo(Func.getContext().toString(), Func.getTotalSamples(),
+                    TotalSamplePercent, FuncPair.second.second,
+                    Func.getHeadSamplesEstimate()));
   }
   dumpHotFunctionList(ColumnTitle, ColumnOffset, PrintValues, HotFuncCount,
                       Profiles.size(), HotFuncSample, ProfileTotalSample,
                       Metric, TopN, OS);
 
   return 0;
 }
 

llvm/tools/llvm-profgen/CSPreInliner.cpp

bool CSPreInliner::getInlineCandidates(ProfiledCandidateQueue &CQueue,
   for (auto &Child : CallerNode->getAllChildContext()) {
     ContextTrieNode *CalleeNode = &Child.second;
     FunctionSamples *CalleeSamples = CalleeNode->getFunctionSamples();
     if (!CalleeSamples)
       continue;
 
     // Call site count is more reliable, so we look up the corresponding call
     // target profile in caller's context profile to retrieve call site count.
-    uint64_t CalleeEntryCount = CalleeSamples->getEntrySamples();
+    uint64_t CalleeEntryCount = CalleeSamples->getHeadSamplesEstimate();
     uint64_t CallsiteCount = 0;
     LineLocation Callsite = CalleeNode->getCallSiteLoc();
     if (auto CallTargets = CallerSamples->findCallTargetMapAt(Callsite)) {
       SampleRecord::CallTargetMap &TargetCounts = CallTargets.get();
       auto It = TargetCounts.find(CalleeSamples->getName());
       if (It != TargetCounts.end())
         CallsiteCount = It->second;
     }

llvm/tools/llvm-profgen/ProfileGenerator.cpp

void CSProfileGenerator::populateInferredFunctionSamples(
   // This could happen for entry point function.
   if (CallerNode == &getRootContext())
     return;
 
   LineLocation CallerLeafFrameLoc = Node.getCallSiteLoc();
   FunctionSamples &CallerProfile = *getOrCreateFunctionSamples(CallerNode);
   // Since we don't have call count for inlined functions, we
   // estimate it from inlinee's profile using entry body sample.
-  uint64_t EstimatedCallCount = CalleeProfile->getEntrySamples();
+  uint64_t EstimatedCallCount = CalleeProfile->getHeadSamplesEstimate();
   // If we don't have samples with location, use 1 to indicate live.
   if (!EstimatedCallCount && !CalleeProfile->getBodySamples().size())
     EstimatedCallCount = 1;
   CallerProfile.addCalledTargetSamples(CallerLeafFrameLoc.LineOffset,
                                        CallerLeafFrameLoc.Discriminator,
                                        Node.getFuncName(), EstimatedCallCount);
   CallerProfile.addBodySamples(CallerLeafFrameLoc.LineOffset,
                                CallerLeafFrameLoc.Discriminator,