[SampleProfile] Add all callsites to AllCandidates if InlineReplay is in effect

[modimo]

Summary

Replay in sample profiling needs to be asked on candidates that may not have counts or below the threshold. If replay is in effect for a function make sure these are captured and also imported during thinLTO.

Testing:
ninja check-all

[hoy]

Good catch, thanks for working on this.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

36 ↗

(On Diff #380519)

Wondering if CallersToReplay could contain function names with those deduplicating suffixes, like .llvm. and if we should exclude those suffixes from F.getName using FunctionSamples::getCanonicalFnName. If the replay file from dwarf dump or Rpass?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1112

Nit: This code doesn't exist on the inlineHotFunctionsWithPriority path which mainly serves csspgo. We should bring up the support there, could be in a separate change.

[modimo]

Use getCanonicalFnName, add same path to inlineHotFunctionsWithPriority

[modimo]

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

36 ↗

(On Diff #380519)

Both dwarf (from llvm-symbolizer) and Rpass will emit .llvm. suffixes. Added getCanonicalFnName to uses.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1112

Makes sense. I added the code path but not sure how to generate a csspgo profile for function_metadata.ll to test it out. Is there a simple way to do that or should I create a test case from scratch?

[hoy]

Add a test case for the inlining (non-importing) path? to the existing test llvm/test/Transforms/SampleProfile/inline-replay.ll ?

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

36 ↗

(On Diff #380519)

Thanks. How about move the getCanonicalFnName calls here?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1112

Thanks for working on the csspgo support. To test thinlto function import, here is an existing test case
llvm/test/Transforms/SampleProfile/csspgo-import-list.ll. It can be mocked to take in a replay file. I'm fine with make a separate change for csspgo work.

[wenlei]

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

34 ↗

(On Diff #380569)

nit: the name is not accurate. it's more like hasInlineAdvice? When scope is module, this can return true while there's no remark for the function.

36 ↗

(On Diff #380519)

How does this work with current version where uses have suffix stripped by getCanonicalFnName? CallersToReplay can still have the original names with suffixes from remarks so they won't match.

Moving FunctionSamples::getCanonicalFnName causes a weird dependency from InlineAdvisor to FunctionSamples. But not moving it here make things inconsistent between sample loader replay and cgscc replay. I actually don't think handling of suffix is critical for reply - consider the case with funique-internal-linkage-name. So we can leave out the canonicalization if possible.

Both dwarf (from llvm-symbolizer) and Rpass will emit .llvm. suffixes.

I thought dwarf names won't have the llvm suffix, but only Rpass will have it since it's using symbol names which is changed by ThinLTO local to global promotion.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1106

Can we move the following into a helper SampleProfileLoader::ExternalInlineAdviceAvailableForFunction(..), it's used in a few places.

ExternalInlineAdvisor &&
    ExternalInlineAdvisor->hasRemarksForFunction(F)

1112

Good point Hongtao. I think it's better if we include the change for inlineHotFunctionsWithPriority in this patch - it feels incomplete otherwise.

1161

When no recommendation is available from replay advisor, this should be a no-op. However with the way it's currently implemented, this will actually fall back to run call analyzer for regular inline decision in shouldInlineCandidate? May the checks above guarantees the advice will always be available here, but the way it's setup seems a bit fragile and not fail-safe..

1162–1163

There're two issues:

• If we rely on this path to influence importing using replay input, we would lose that influence completely for non-FDO path, but replay is not meant for FDO only.
• On the other hand, for FDO, we have the opportunity to look deeper into the call tree and do better, i.e following findExternalInlineCandidate to import lower level candidates too.

For #2, with current implementation, even if the inline replay provides the exact same decision as the native inline decision, we could miss out inlining due to missing importing here.

And this added complexity logically still belongs to findExternalInlineCandidate, I suggest we move the logic there to keep the main flow cleaner here.

All above applies to inlineHotFunctionsWithPriority as well.

1285–1288

Add a comment please.

1888

With the only callsite for getReplayInlineAdvisor removed, the function and all stuff around HasReplayRemarks can be removed too? We now assume ReplayInlineAdvisor always have remarks loaded, right?

[modimo]

Refactor uses of replay advisor into getExternalInlineAdvisorCost and getExternalInlineAdvisorShouldInline. Move importing code to findExternalInlineCandidate

[modimo]

@wenlei I refactored the implementation so that all queries to the external advisor route to getExternalInlineAdvisorCost or getExternalInlineAdvisorShouldInline depending on if we want the actual cost or just if inlining occurs, respectively. This unifies every location that queries for the existence of remarks (the original one in hasInlineAdvice and the new ones that add sites as candidates) and also means I don't need to expose a hasRemarksForFunction from the ReplayAdvisor and can do everything using the original InlineAdvisor interface. I also edited how importing is done:

1. If samples are not available then we can't transitively import but can at least import the symbol itself
2. If samples are present and external advisor wants to inline drop the import threshold to 0 so we have the opportunity to replay post-link with imported call sites

Does it make sense to drop the threshold to allow importing and are there any potential recursive issues with that?

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

36 ↗

(On Diff #380519)

I thought dwarf names won't have the llvm suffix, but only Rpass will have it since it's using symbol names which is changed by ThinLTO local to global promotion.

I looked into it more since the behavior is odd, when I query llvm-symbolizer on an address I get the .llvm. suffixes even though AFAIK it should use DWARF information:

llvm-symbolizer --inlining -e llvm-objdump 0x998400 --no-demangle
_ZL14getSegmentNamePKN4llvm6object15MachOObjectFileERKNS0_10SectionRefE.llvm.5974130588735922841

However, using GDB which directly queries dwarf:

(gdb) disas/s 0x998400
Dump of assembler code for function getSegmentName(llvm::object::MachOObjectFile const*, llvm::object::SectionRef const&):

Which doesn't contain any suffixes. Looking into it, llvm-symbolizer by default actually prefers using the symbol table over the dwarf information: https://reviews.llvm.org/D83530#inline-789802 and the option --use-symbol-table is now a no-op (D87067) so it always uses symbol table information. I manually edited Opts.UseSymbolTable = false; and confirm that the suffix does get removed by llvm-symbolizer.

What that means is that both sources I'm using (llvm-symbolizer and Rpass) will have these suffixes. Canonicalization isn't necessary though, implementing it when we need it makes sense rather than anticipating its usage. I don't particularly like the weird dependency of InlineAdvisor to FunctionSamples as well.

[wenlei]

In D112033#3079444, @modimo wrote:

@wenlei I refactored the implementation so that all queries to the external advisor route to getExternalInlineAdvisorCost or getExternalInlineAdvisorShouldInline depending on if we want the actual cost or just if inlining occurs, respectively. This unifies every location that queries for the existence of remarks (the original one in hasInlineAdvice and the new ones that add sites as candidates) and also means I don't need to expose a hasRemarksForFunction from the ReplayAdvisor and can do everything using the original InlineAdvisor interface. I also edited how importing is done:

1. If samples are not available then we can't transitively import but can at least import the symbol itself
2. If samples are present and external advisor wants to inline drop the import threshold to 0 so we have the opportunity to replay post-link with imported call sites

Does it make sense to drop the threshold to allow importing and are there any potential recursive issues with that?

That sounds good to me. It makes the implementation cleaner. Setting threshold to 0 may lead to more importing than needed which can slow down build a bit, but I don't think that's a blocker for inline replay. Lgtm now.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1285–1288

Maybe you missed this one, a comment will be helpful. When sample loader inliner is used for reply, we want to inline callee if replay advisor say so even when the callee doesn't have profile.

One thing I'm not exactly sure is how robust is the sample loader inliner for handling candidate without a profile. For testing, maybe you want to try removing the check altogether for a regular (non-replay) build for a larger component, and see if there's any trouble.

[modimo]

Add comment to SampleProfileLoader::getInlineCandidate. Add inline-topdown-missing.prof file that wasn't part of the diff before.

llvm/lib/Transforms/IPO/SampleProfile.cpp

1285–1288

Maybe you missed this one, a comment will be helpful. When sample loader inliner is used for reply, we want to inline callee if replay advisor say so even when the callee doesn't have profile.

Added

One thing I'm not exactly sure is how robust is the sample loader inliner for handling candidate without a profile. For testing, maybe you want to try removing the check altogether for a regular (non-replay) build for a larger component, and see if there's any trouble.

That's a good idea to validate robustness, I'm run with this check removed on CSSPGO clang build and see what shakes loose.

llvm/test/Transforms/SampleProfile/Inputs/csspgo-import-list.prof

3

FYI I was debugging through this and found that sample associated with CallBase expected a discriminator for _Z5funcAi. I'm assuming these were meant to fully match against the source. I added csspgo-import-list-no-funca.prof as an explicit test case where _Z5funcAi samples are missing.

[modimo]

llvm/lib/Transforms/IPO/SampleProfile.cpp

1285–1288

Building Clang with this early return commented out was successful.

llvm/lib/Transforms/IPO/SampleProfile.cpp

protected:
   bool emitAnnotations(Function &F);
   ErrorOr<uint64_t> getInstWeight(const Instruction &I) override;
   ErrorOr<uint64_t> getProbeWeight(const Instruction &I);
   const FunctionSamples *findCalleeFunctionSamples(const CallBase &I) const;
   const FunctionSamples *
   findFunctionSamples(const Instruction &I) const override;
   std::vector<const FunctionSamples *>
   findIndirectCallFunctionSamples(const Instruction &I, uint64_t &Sum) const;
-  void findExternalInlineCandidate(const FunctionSamples *Samples,
+  void findExternalInlineCandidate(CallBase *CB, const FunctionSamples *Samples,
                                    DenseSet<GlobalValue::GUID> &InlinedGUIDs,
                                    const StringMap<Function *> &SymbolMap,
                                    uint64_t Threshold);
   // Attempt to promote indirect call and also inline the promoted call
   bool tryPromoteAndInlineCandidate(
       Function &F, InlineCandidate &Candidate, uint64_t SumOrigin,
       uint64_t &Sum, SmallVector<CallBase *, 8> *InlinedCallSites = nullptr);
+
   bool inlineHotFunctions(Function &F,
                           DenseSet<GlobalValue::GUID> &InlinedGUIDs);
+  Optional<InlineCost> getExternalInlineAdvisorCost(CallBase &CB);
+  bool getExternalInlineAdvisorShouldInline(CallBase &CB);
   InlineCost shouldInlineCandidate(InlineCandidate &Candidate);
   bool getInlineCandidate(InlineCandidate *NewCandidate, CallBase *CB);
   bool
   tryInlineCandidate(InlineCandidate &Candidate,
                      SmallVector<CallBase *, 8> *InlinedCallSites = nullptr);
   bool
   inlineHotFunctionsWithPriority(Function &F,
                                  DenseSet<GlobalValue::GUID> &InlinedGUIDs);
if (CalledFunction) {
                 << (Hot ? "hotness: '" : "size: '")
                 << ore::NV("Callee", CalledFunction) << "' into '"
                 << ore::NV("Caller", &F) << "'");
     }
   }
 }
 
 void SampleProfileLoader::findExternalInlineCandidate(
-    const FunctionSamples *Samples, DenseSet<GlobalValue::GUID> &InlinedGUIDs,
+    CallBase *CB, const FunctionSamples *Samples,
+    DenseSet<GlobalValue::GUID> &InlinedGUIDs,
     const StringMap<Function *> &SymbolMap, uint64_t Threshold) {
+
+  // If ExternalInlineAdvisor wants to inline an external function
+  // make sure it's imported
+  if (CB && getExternalInlineAdvisorShouldInline(*CB)) {
+    // Samples may not exist for replayed function, if so
+    // just add the direct GUID and move on
+    if (!Samples) {
+      InlinedGUIDs.insert(
+          FunctionSamples::getGUID(CB->getCalledFunction()->getName()));
+      return;
+    }
+    // Otherwise, drop the threshold to import everything that we can
+    Threshold = 0;
+  }
+
   assert(Samples && "expect non-null caller profile");
 
   // For AutoFDO profile, retrieve candidate profiles by walking over
   // the nested inlinee profiles.
   if (!ProfileIsCS) {
     Samples->findInlinedFunctions(InlinedGUIDs, SymbolMap, Threshold);
     return;
   }
while (LocalChanged) {
     SmallVector<CallBase *, 10> CIS;
     for (auto &BB : F) {
       bool Hot = false;
       SmallVector<CallBase *, 10> AllCandidates;
       SmallVector<CallBase *, 10> ColdCandidates;
       for (auto &I : BB.getInstList()) {
         const FunctionSamples *FS = nullptr;
         if (auto *CB = dyn_cast<CallBase>(&I)) {
-          if (!isa<IntrinsicInst>(I) && (FS = findCalleeFunctionSamples(*CB))) {
+          if (!isa<IntrinsicInst>(I)) {
+            if ((FS = findCalleeFunctionSamples(*CB))) {
               assert((!FunctionSamples::UseMD5 || FS->GUIDToFuncNameMap) &&
                      "GUIDToFuncNameMap has to be populated");
               AllCandidates.push_back(CB);
               if (FS->getEntrySamples() > 0 || ProfileIsCS)
                 LocalNotInlinedCallSites.try_emplace(CB, FS);
               if (callsiteIsHot(FS, PSI, ProfAccForSymsInList))
                 Hot = true;
               else if (shouldInlineColdCallee(*CB))
                 ColdCandidates.push_back(CB);
+            } else if (getExternalInlineAdvisorShouldInline(*CB)) {
+              AllCandidates.push_back(CB);
+            }
           }
         }
       }
       if (Hot || ExternalInlineAdvisor) {
         CIS.insert(CIS.begin(), AllCandidates.begin(), AllCandidates.end());
         emitOptimizationRemarksForInlineCandidates(AllCandidates, F, true);
       } else {
         CIS.insert(CIS.begin(), ColdCandidates.begin(), ColdCandidates.end());
         emitOptimizationRemarksForInlineCandidates(ColdCandidates, F, false);
       }
     }
     for (CallBase *I : CIS) {
       Function *CalledFunction = I->getCalledFunction();
       InlineCandidate Candidate = {
           I,
           LocalNotInlinedCallSites.count(I) ? LocalNotInlinedCallSites[I]
                                             : nullptr,
           0 /* dummy count */, 1.0 /* dummy distribution factor */};
       // Do not inline recursive calls.
       if (CalledFunction == &F)
         continue;
       if (I->isIndirectCall()) {
         uint64_t Sum;
         for (const auto *FS : findIndirectCallFunctionSamples(*I, Sum)) {
           uint64_t SumOrigin = Sum;
           if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
-            findExternalInlineCandidate(FS, InlinedGUIDs, SymbolMap,
+            findExternalInlineCandidate(I, FS, InlinedGUIDs, SymbolMap,
                                         PSI->getOrCompHotCountThreshold());
             continue;
           }
           if (!callsiteIsHot(FS, PSI, ProfAccForSymsInList))
             continue;
 
           Candidate = {I, FS, FS->getEntrySamples(), 1.0};
           if (tryPromoteAndInlineCandidate(F, Candidate, SumOrigin, Sum)) {
             LocalNotInlinedCallSites.erase(I);
             LocalChanged = true;
           }
         }
       } else if (CalledFunction && CalledFunction->getSubprogram() &&
                  !CalledFunction->isDeclaration()) {
         if (tryInlineCandidate(Candidate)) {
           LocalNotInlinedCallSites.erase(I);
           LocalChanged = true;
         }
       } else if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
-        findExternalInlineCandidate(findCalleeFunctionSamples(*I), InlinedGUIDs,
-                                    SymbolMap,
+        findExternalInlineCandidate(I, findCalleeFunctionSamples(*I),
+                                    InlinedGUIDs, SymbolMap,
                                     PSI->getOrCompHotCountThreshold());
       }
     }
     Changed |= LocalChanged;
   }
 
   // For CS profile, profile for not inlined context will be merged when
   // base profile is being trieved
   if (ProfileIsCS)
     return Changed;
 
   // Accumulate not inlined callsite information into notInlinedSamples
   for (const auto &Pair : LocalNotInlinedCallSites) {
     CallBase *I = Pair.getFirst();
     Function *Callee = I->getCalledFunction();
bool SampleProfileLoader::getInlineCandidate(InlineCandidate *NewCandidate,
                                              CallBase *CB) {
   assert(CB && "Expect non-null call instruction");
 
   if (isa<IntrinsicInst>(CB))
     return false;
 
   // Find the callee's profile. For indirect call, find hottest target profile.
   const FunctionSamples *CalleeSamples = findCalleeFunctionSamples(*CB);
-  if (!CalleeSamples)
+  // If ExternalInlineAdvisor wants to inline this site, do so even
+  // if Samples are not present.
+  if (!CalleeSamples && !getExternalInlineAdvisorShouldInline(*CB))
     return false;
 
   float Factor = 1.0;
   if (Optional<PseudoProbe> Probe = extractProbe(*CB))
     Factor = Probe->Factor;
 
   uint64_t CallsiteCount = 0;
   ErrorOr<uint64_t> Weight = getBlockWeight(CB->getParent());
   if (Weight)
     CallsiteCount = Weight.get();
   if (CalleeSamples)
     CallsiteCount = std::max(
         CallsiteCount, uint64_t(CalleeSamples->getEntrySamples() * Factor));
 
   *NewCandidate = {CB, CalleeSamples, CallsiteCount, Factor};
   return true;
 }
 
-InlineCost
-SampleProfileLoader::shouldInlineCandidate(InlineCandidate &Candidate) {
+Optional<InlineCost>
+SampleProfileLoader::getExternalInlineAdvisorCost(CallBase &CB) {
   std::unique_ptr<InlineAdvice> Advice = nullptr;
   if (ExternalInlineAdvisor) {
-    Advice = ExternalInlineAdvisor->getAdvice(*Candidate.CallInstr);
+    Advice = ExternalInlineAdvisor->getAdvice(CB);
     if (Advice) {
       if (!Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         return InlineCost::getNever("not previously inlined");
       }
       Advice->recordInlining();
       return InlineCost::getAlways("previously inlined");
     }
   }
 
+  return {};
+}
+
+bool SampleProfileLoader::getExternalInlineAdvisorShouldInline(CallBase &CB) {
+  Optional<InlineCost> Cost = getExternalInlineAdvisorCost(CB);
+  return Cost ? !!Cost.getValue() : false;
+}
+
+InlineCost
+SampleProfileLoader::shouldInlineCandidate(InlineCandidate &Candidate) {
+  if (Optional<InlineCost> ReplayCost =
+          getExternalInlineAdvisorCost(*Candidate.CallInstr))
+    return ReplayCost.getValue();
   // Adjust threshold based on call site hotness, only do this for callsite
   // prioritized inliner because otherwise cost-benefit check is done earlier.
   int SampleThreshold = SampleColdCallSiteThreshold;
   if (CallsitePrioritizedInline) {
     if (Candidate.CallsiteCount > PSI->getHotCountThreshold())
       SampleThreshold = SampleHotCallSiteThreshold;
     else if (!ProfileSizeInline)
       return InlineCost::getNever("cold callsite");
if (I->isIndirectCall()) {
       uint64_t Sum = 0;
       auto CalleeSamples = findIndirectCallFunctionSamples(*I, Sum);
       uint64_t SumOrigin = Sum;
       Sum *= Candidate.CallsiteDistribution;
       unsigned ICPCount = 0;
       for (const auto *FS : CalleeSamples) {
         // TODO: Consider disable pre-lTO ICP for MonoLTO as well
         if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
-          findExternalInlineCandidate(FS, InlinedGUIDs, SymbolMap,
+          findExternalInlineCandidate(I, FS, InlinedGUIDs, SymbolMap,
                                       PSI->getOrCompHotCountThreshold());
           continue;
         }
         uint64_t EntryCountDistributed =
             FS->getEntrySamples() * Candidate.CallsiteDistribution;
         // In addition to regular inline cost check, we also need to make sure
         // ICP isn't introducing excessive speculative checks even if individual
         // target looks beneficial to promote and inline. That means we should
if (I->isIndirectCall()) {
       if (tryInlineCandidate(Candidate, &InlinedCallSites)) {
         for (auto *CB : InlinedCallSites) {
           if (getInlineCandidate(&NewCandidate, CB))
             CQueue.emplace(NewCandidate);
         }
         Changed = true;
       }
     } else if (LTOPhase == ThinOrFullLTOPhase::ThinLTOPreLink) {
-      findExternalInlineCandidate(Candidate.CalleeSamples, InlinedGUIDs,
-                                  SymbolMap, PSI->getOrCompHotCountThreshold());
+      findExternalInlineCandidate(I, findCalleeFunctionSamples(*I),
+                                  InlinedGUIDs, SymbolMap,
+                                  PSI->getOrCompHotCountThreshold());
     }
   }
 
   if (!CQueue.empty()) {
     if (SizeLimit == (unsigned)ProfileInlineLimitMax)
       ++NumCSInlinedHitMaxLimit;
     else if (SizeLimit == (unsigned)ProfileInlineLimitMin)
       ++NumCSInlinedHitMinLimit;
bool SampleProfileLoader::doInitialization(Module &M,
   if (ProfAccForSymsInList) {
     NamesInProfile.clear();
     if (auto NameTable = Reader->getNameTable())
       NamesInProfile.insert(NameTable->begin(), NameTable->end());
     CoverageTracker.setProfAccForSymsInList(true);
   }
 
   if (FAM && !ProfileInlineReplayFile.empty()) {
     ExternalInlineAdvisor = getReplayInlineAdvisor(
         M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr, ProfileInlineReplayFile,
         ProfileInlineReplayScope, /*EmitRemarks=*/false);
   }
 
   // Apply tweaks if context-sensitive profile is available.
   if (Reader->profileIsCS()) {
     ProfileIsCS = true;
     FunctionSamples::ProfileIsCS = true;

llvm/test/Transforms/SampleProfile/Inputs/csspgo-import-list-no-funca.prof

 [main]:154:2
  2: 12
- 3: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
 [main:3.1 @ _Z5funcBi]:120:7040
  0: 7001
  1: 19 _Z8funcLeafi:9999
  3: 12
 [main:3.1 @ _Z5funcBi @ _Z5funcBiLeaf2]:1:9010
  0: 7001
  1: 19 _Z8funcLeafi3:9999
  3: 12
-[main:2 @ _Z5funcAi]:99:11
- 0: 10
- 1: 10 _Z8funcLeafi:11
- 2: 287864 _Z3fibi:315608
- 3: 24
 [main:3 @ _Z5funcCi]:23254:11
  0: 10
  1: 23250
 [main:3 @ _Z5funcDi]:23:45201
  0: 10
  1: 23250
-[main:2 @ _Z5funcAi:2 @ _Z3fibi]:120:101
- 0: 99
- 1: 6
- 3: 97

llvm/test/Transforms/SampleProfile/Inputs/csspgo-import-list-replay.txt

+remark: merged.cpp:14:21.1: '_Z5funcBi' inlined into 'main' to match profiling context with (cost=0, threshold=337) at callsite main:3:21.1
+remark: merged.cpp:13:10.1: '_Z5funcAi' inlined into 'main' to match profiling context with (cost=0, threshold=337) at callsite main:2:10.1
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/csspgo-import-list.prof

 [main]:154:2
  2: 12
- 3: 18 _Z5funcAi:11
+ 2.1: 18 _Z5funcAi:11
  3.1: 18 _Z5funcBi:19
 [main:3.1 @ _Z5funcBi]:120:7040
  0: 7001
  1: 19 _Z8funcLeafi:9999
  3: 12
 [main:3.1 @ _Z5funcBi @ _Z5funcBiLeaf2]:1:9010
  0: 7001
  1: 19 _Z8funcLeafi3:9999
  3: 12
-[main:2 @ _Z5funcAi]:99:11
+[main:2.1 @ _Z5funcAi]:99:11
  0: 10
  1: 10 _Z8funcLeafi:11
  2: 287864 _Z3fibi:315608
  3: 24
 [main:3 @ _Z5funcCi]:23254:11
  0: 10
  1: 23250
 [main:3 @ _Z5funcDi]:23:45201
  0: 10
  1: 23250
-[main:2 @ _Z5funcAi:2 @ _Z3fibi]:120:101
+[main:2.1 @ _Z5funcAi:2 @ _Z3fibi]:120:101
  0: 99
  1: 6
  3: 97

llvm/test/Transforms/SampleProfile/Inputs/function_metadata_replay.txt

+remark: calls.cc:8:0: 'foo' inlined into 'test_liveness' to match profiling context with (cost=0, threshold=337) at callsite test_liveness:1:0;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-topdown-missing.prof

+main:225715:0
+ 2.1: 5553
+ 3: 5391
+
+_Z3sumii:6010:50000
 No newline at end of file

llvm/test/Transforms/SampleProfile/csspgo-import-list.ll

 ; Make sure Import GUID list for ThinLTO properly set for CSSPGO
 ; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -sample-profile-file=%S/Inputs/csspgo-import-list.prof -S | FileCheck %s
 ; RUN: llvm-profdata merge --sample --extbinary %S/Inputs/csspgo-import-list.prof -o %t.prof
 ; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -sample-profile-file=%t.prof -S | FileCheck %s
 ; RUN: llvm-profdata show --sample -show-sec-info-only %t.prof | FileCheck %s --check-prefix=CHECK-ORDERED
 ; RUN: llvm-profdata merge --sample --extbinary --use-md5 %S/Inputs/csspgo-import-list.prof -o %t.md5
 ; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -sample-profile-file=%t.md5 -S | FileCheck %s
 ; RUN: llvm-profdata show --sample -show-sec-info-only %t.md5 | FileCheck %s --check-prefix=CHECK-ORDERED
 
+;; Validate that with replay in effect, we import call sites even if they are below the threshold
+;; Baseline import decisions
+; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -sample-profile-file=%S/Inputs/csspgo-import-list.prof -profile-summary-hot-count=10000 -S | FileCheck %s --check-prefix=THRESHOLD
+;; With replay
+; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -sample-profile-file=%S/Inputs/csspgo-import-list.prof -sample-profile-inline-replay=%S/Inputs/csspgo-import-list-replay.txt -sample-profile-inline-replay-scope=Module -profile-summary-hot-count=10000 -S | FileCheck %s --check-prefix=THRESHOLD-REPLAY
+;; With replay but no profile information for call to _Z5funcAi. We import _Z5funcAi because it's explicitly in the replay but don't go further to its callee (_Z3fibi) because we lack samples
+; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -sample-profile-file=%S/Inputs/csspgo-import-list-no-funca.prof -sample-profile-inline-replay=%S/Inputs/csspgo-import-list-replay.txt -sample-profile-inline-replay-scope=Module -profile-summary-hot-count=10000 -S | FileCheck %s --check-prefix=THRESHOLD-REPLAY-NO-FUNCA
 
 declare i32 @_Z5funcBi(i32 %x)
 declare i32 @_Z5funcAi(i32 %x)
 
 define dso_local i32 @main() local_unnamed_addr #0 !dbg !18 {
 entry:
   br label %for.body, !dbg !25
 
for.body:                                         ; preds = %for.body, %entry
   %add2 = add i32 %call, %r.010, !dbg !34
   %add3 = add i32 %add2, %call1, !dbg !35
   %dec = add nsw i32 %x.011, -1, !dbg !36
   %cmp = icmp eq i32 %x.011, 0, !dbg !38
   br i1 %cmp, label %for.cond.cleanup, label %for.body, !dbg !25
 }
 
 ; Make sure the ImportGUID stays with entry count metadata for ThinLTO-PreLink
+; For reference, the GUIDs of the functions are:
+; _Z5funcBi:446061515086924981
+; _Z8funcLeafi:7102633082150537521
+; _Z5funcBiLeaf2:3815895320998406042
+; _Z8funcLeafi3:-2862076748587597320
+; _Z5funcAi:6309742469962978389
+; _Z3fibi:-2016976694713209516
+
 ; CHECK: distinct !DISubprogram(name: "main"
 ; CHECK: !{!"function_entry_count", i64 3, i64 446061515086924981, i64 3815895320998406042, i64 7102633082150537521, i64 -2862076748587597320}
 
 ; CHECK-ORDERED: FuncOffsetTableSection {{.*}} {ordered}
 
+; THRESHOLD: !{!"function_entry_count", i64 3}
+; THRESHOLD-REPLAY: !{!"function_entry_count", i64 3, i64 446061515086924981, i64 3815895320998406042, i64 6309742469962978389, i64 7102633082150537521, i64 -2862076748587597320, i64 -2016976694713209516}
+; THRESHOLD-REPLAY-NO-FUNCA: !{!"function_entry_count", i64 3, i64 446061515086924981, i64 3815895320998406042, i64 6309742469962978389, i64 7102633082150537521, i64 -2862076748587597320}
+
 attributes #0 = { nofree noinline norecurse nounwind uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "disable-tail-calls"="false" "frame-pointer"="none" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math"="false" "use-soft-float"="false" "use-sample-profile" }
 
 !llvm.dbg.cu = !{!2}
 !llvm.module.flags = !{!14, !15, !16}
 !llvm.ident = !{!17}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "factor", scope: !2, file: !3, line: 21, type: !13, isLocal: false, isDefinition: true)

llvm/test/Transforms/SampleProfile/function_metadata.ll

 ; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%S/Inputs/function_metadata.prof -S | FileCheck %s
 ; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%S/Inputs/function_metadata.compact.afdo -S | FileCheck %s
 ; RUN: opt < %s -passes='pseudo-probe,thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-file=%S/Inputs/pseudo-probe-func-metadata.prof -S | FileCheck %s
 
+;; Validate that with replay in effect, we import call sites even if they are below the threshold
+;; Baseline import decisions
+; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-summary-hot-count=2000 -profile-file=%S/Inputs/function_metadata.prof -S | FileCheck -check-prefix=THRESHOLD %s
+;; With replay decisions
+; RUN: opt < %s -passes='thinlto-pre-link<O2>' -pgo-kind=pgo-sample-use-pipeline -profile-summary-hot-count=2000 -profile-file=%S/Inputs/function_metadata.prof -S -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay=%S/Inputs/function_metadata_replay.txt | FileCheck -check-prefix=THRESHOLD-REPLAY %s
+
+
 ; Tests whether the functions in the inline stack are added to the
 ; function_entry_count metadata.
 
 declare void @foo()
 
 declare void @bar()
 
 declare !dbg !13 void @bar_dbg()
 
 define void @bar_available() #0 !dbg !14 {
   ret void
 }
 
 ; CHECK: define void @test({{.*}} !prof ![[ENTRY_TEST:[0-9]+]]
+; THRESHOLD: define void @test({{.*}} !prof ![[ENTRY_THRESHOLD:[0-9]+]]
+; THRESHOLD-REPLAY: define void @test({{.*}} !prof ![[ENTRY_REPLAY_TEST:[0-9]+]]
 define void @test(void ()*) #0 !dbg !7 {
   %2 = alloca void ()*
   store void ()* %0, void ()** %2
   %3 = load void ()*, void ()** %2
   ; CHECK: call {{.*}}, !prof ![[PROF:[0-9]+]]
   call void @foo(), !dbg !18
   call void %3(), !dbg !19
   ret void
 }
 
 ; CHECK: define void @test_liveness({{.*}} !prof ![[ENTRY_TEST_LIVENESS:[0-9]+]]
+; THRESHOLD: define void @test_liveness({{.*}} !prof ![[ENTRY_THRESHOLD:[0-9]+]]
+; THRESHOLD-REPLAY: define void @test_liveness({{.*}} !prof ![[ENTRY_REPLAY_TEST_LIVENESS:[0-9]+]]
 define void @test_liveness() #0 !dbg !12 {
   call void @foo(), !dbg !20
   ret void
 }
 
 ; GUIDs of foo, bar, foo1, foo2 and foo3 should be included in the metadata to
 ; make sure hot inline stacks are imported. The total count of baz is lower
 ; than the hot cutoff threshold and its GUID should not be included in the
 ; metadata.
 ; CHECK: ![[ENTRY_TEST]] = !{!"function_entry_count", i64 1, i64 2494702099028631698, i64 6699318081062747564, i64 7682762345278052905,  i64 -7908226060800700466, i64 -2012135647395072713}
 
 ; Check GUIDs for foo, bar and bar_dbg are included in the metadata to
 ; make sure the liveness analysis can capture the dependency from test_liveness
 ; to bar. bar_available should not be included as it's within the same module.
 ; CHECK: ![[ENTRY_TEST_LIVENESS]] = !{!"function_entry_count", i64 1, i64 6699318081062747564, i64 -2012135647395072713, i64 -1522495160813492905}
 
+; With high threshold, nothing should be imported
+; THRESHOLD: ![[ENTRY_THRESHOLD]] = !{!"function_entry_count", i64 1}
+
+; With high threshold and replay, sites that are in the replay (foo, and transitively bar and bar_dbg in function test_liveness) should be imported
+; THRESHOLD-REPLAY: ![[ENTRY_REPLAY_TEST]] = !{!"function_entry_count", i64 1}
+; THRESHOLD-REPLAY: ![[ENTRY_REPLAY_TEST_LIVENESS]] = !{!"function_entry_count", i64 1, i64 6699318081062747564, i64 -2012135647395072713, i64 -1522495160813492905}
+
 attributes #0 = {"use-sample-profile"}
 
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!8, !9}
 !llvm.ident = !{!10}
 
 !0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, producer: "clang version 3.5 ", isOptimized: false, emissionKind: NoDebug, file: !1, enums: !2, retainedTypes: !2, globals: !2, imports: !2)
 !1 = !DIFile(filename: "calls.cc", directory: ".")

llvm/test/Transforms/SampleProfile/inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-sample-profile-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
 ;; Check replay inline decisions
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module  -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+
+;; Check baseline inline decisions with all callee counts missing
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-missing.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT-NOINLINING -allow-empty %s
+
+;; Check replay inline decisions with all callee counts missing. The call sites should still be passed to the replay advisor and successfully get inlined as before
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-missing.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+
+;; Check baseline inline decisions with high threshold
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -profile-summary-hot-count=500000 -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT-NOINLINING -allow-empty %s
+
+;; Check replay inline decisions with high threshold. The call sites should still be passed to the replay advisor and successfully get inlined as before
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -profile-summary-hot-count=500000 -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY -allow-empty %s
 
 ;; Check baseline inline decisions with "inline-topdown-inline-all.prof" which inlines all sites
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT-ALL %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT-ALL %s
 
 ;; Check function scope replay inline decisions with "inline-topdown-inline-all.prof" and "inline-topdown-function-scope.txt" which only contains: '_Z3sumii' inlined into 'main'
 ;; 1. _Z3sumii is inlined into main, but all other inline candidates in main (e.g. _Z3subii) are not inlined
 ;; 2. Inline decisions made in other functions match default sample inlining, in this case _Z3subii is inlined into _Z3sumii
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function-scope.txt -sample-profile-inline-replay-scope=Function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function-scope.txt -sample-profile-inline-replay-scope=Function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION %s
 
 ;; Check behavior on non-existent replay file
-; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/non-existent-dummy.txt -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/non-existent-dummy.txt -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
 
 ;; Check scope inlining errors out on non <Module|Function> inputs
-; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
 
 ; DEFAULT: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-NOT: '_Z3subii' inlined into 'main'
 
 ; REPLAY: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
 ; REPLAY: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
 ; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'
 
+; DEFAULT-NOINLINING-NOT: inlined into
+
 ; DEFAULT-ALL: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-ALL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-ALL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 
 ; REPLAY-ALL-FUNCTION : _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
 ; REPLAY-ALL-FUNCTION-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; REPLAY-ALL-FUNCTION: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}