[InlineAdvisor] Add -inline-replay-scope=<Function|Module> to control replay scope

[modimo]

Summary

The goal is to allow grafting an inline tree from Clang or GCC into a new compilation without affecting other functions. For GCC, we're doing this by extracting the inline tree from dwarf information and generating the equivalent remarks.

This allows easier side-by-side asm analysis and a trial way to see if a particular inlining setup provides benefits by itself.

Testing:
ninja check-all

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

844

Since getAdvice returns a unique_ptr, returning a null seems like a good way to indicate "no advice". Does that also make sense for the ML inliner @mtrofin?

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

844

The design intent was for the advice to be clear, i.e. either inline or not. You probably want to delegate the decision to some other advisor if you don't have one? i.e. the remarks advisor could delegate to the default one - and return a non-null Advice. WDYT?

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

844

It does delegate it in the CGSCC case because we have a nicely nested system with the default one taking over.

Unfortunately the SampleProfile inliner currently isn't using the InlineAdvisor setup. That being said, perhaps the better approach is to adapt the SampleProfile inliner to an InlineAdvisor so the replay advisor will always return a non-null.

[modimo]

Formatting

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

844

The SampleProfile inliner needs additional information beyond CallBase to make its decision. Namely, InlineCandidate which contains sampling information. Does it make sense to extend getAdvice to take additional information?

[mtrofin]

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

76

nit: probably no need for curly brackets.

llvm/lib/Transforms/IPO/Inliner.cpp

844

I don't follow why, but to the scope of this patch, couldn't the ReplayInlineAdvisor return an Advice that says "no" when it has no advice to provide?

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

844

In SampleProfile, it needs 3 states:

1. Yes when it matches replay
2. No when it doesn't match replay
3. HasNoAdvice when in strict mode and we want the SampleProfile inliner to make the decision

ATM there's only (1) and (2), I'm using null to represent (3). An alternative could be to have a HasNoAdvice state that maps to no inline but can be queried to differentiate between (2) and (3)

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

844

I think I see now. OK, let's allow a null return then from getAdvice; to answer your original question, I don't think it affects advisors that want to be categorical (i.e. an advisor can always return yes/no), but it affects consumers, and your patch handles that. The ML stuff doesn't consume, it just implements the advisor interfaces.

We can explore afterwards if/how to adapt SampleProfile to an advisor design; we'd definitely not want to introduce that InlineCandidate in the advisor interface, the former is too specific to the sample profiler.

Could you leave a comment on the getAdvice() abstract API re. the fact that it can return nothing, a pointer to this review, and a TODO that maybe we can tighten it?

Thanks!

[modimo]

Add comments on getAdvice abstract API. Remove curly brackets in ReplayInlineAdvisor destructor

[wenlei]

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

42

nit: InlineCallersFromRemarks -> InlinerFromRemarks

llvm/lib/Analysis/InlineAdvisor.cpp

49

Sample loader replay was independent of cgscc replay as they use different switches for taking different inputs. Now the "mode" switch is global, so both replays are tied to a single mode even if they use different input files. I think it's more flexible if they are still kept independent (the hot/cold thresholds etc are all kept separate for the two inliners).

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

48–49

nits:

• Pair.first.split(" inlined into ") is duplicated string search and can be merged.
• If you do Pair.first.split("' inlined into '"), the drop_back() and drop_front() can be removed, and it helps readability..

54

Pair.second.drop_back() works? or you expect something after ;?

55–56

Check Caller.epmty() too?

I would also add a warning if any of the three is empty - invalid remarks, can't be replayed etc.. instead of silently swallow it.

72–75

I'm not sure whether printing such debug dumps from dtor is a good idea. Lifetime of objects could potentially be somewhat arbitrary, but logs better be printed in more controlled/organized fashion. Looking at the test case, it also seems like these prints are added for testing purpose? See the comments in the test case - we better avoid using debug prints for testing.

87–88

Perhaps we can remove this for simplicity. The shortcut isn't helping much.

117

nit:

if (!Strict || InlineCallersFromRemarks.count(CB.getFunction()->getName())) {
  ...
}
else if (Strict) {
  if (OriginalAdvisor)
    return OriginalAdvisor->getAdvice(CB);
  return {};
}

This way we don't need a lambda - since it's only called once it can be inlined.

llvm/lib/Transforms/IPO/Inliner.cpp

844

@mtrofin I added InlineCandidate stuff in sample loader in https://reviews.llvm.org/D94001 and https://reviews.llvm.org/D95024, but the use of extra info for inline decision predates these changes. As to why we can't just use CallBase there - the inlining happens before profile is annotated on IR as branch metadata (it's the sample *loader*), so none of the BPI, BFI stuff is available, hence we have to use the raw context profile representation alongside CallBase to make inline decisions.

I agree that the InlineCandidate there is very specific and not good for advisor API. However I think having three state yes/no/unknown is more flexible, and some advisors can still choose to not use the unknown state. We can use NULL for unknown, or a different representation.

You probably want to delegate the decision to some other advisor if you don't have one

Conceptually there's a local "unknown" for some advisor already. Forcing a yes/no decision and not having a representation for unknown makes the inline advisor system "closed" instead of "open" in the sense that everything needs to reach a decision within the advisor hierarchy without any external help. In ideal world, it's clean, and sure we can try to convert everything into the system, but practically it might be too restrictive?

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

17

This would fail ninja check with release builds. Usually we try to avoid testing that requires debug builds unless it's critical and no other way to observe.

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

844

Maybe. Right now though I think a good chunk of the discussion stems from the fact that SampleProfile does 2 things in one: populates profile info; and inlines. We should probably split this, but that's both very much out of the scope of this patch, and orthogonal to the tri/bi-state advisor topic.

[modimo]

Address feedback and change over to replay-scope

[modimo]

In D110658#3035652, @wenlei wrote:

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

I think using scope to capture the functionality here makes sense. For your definition of strict I see it as modifier to positive/negative. A strict+negative replay would not inline the remarks present but will attempt to inline everything else and strict+positive would inline the remarks but not inline anything else. Non-strict+negative would make sure the remarks are not inlined but defer the rest of the decisions to the original advisor and non-strict+positive will inline the remarks but defer the rest to the original advisor.

Strict+negative is useful because one way we can capture replay is by looking at all call-sites that remain in the assembly and eliminate all the rest (i.e. making sure they're inlined) which has the advantage that inlined sites no longer present in the assembly (e.g. simple accessors whose bodies completely get folded) will properly be inlined while in strict+positive mode because the dwarf information is gone no remarks are generated and the original call remains.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

42

This is to indicate which callers should have replay enforced in function scope, don't think InlinerFromRemarks is enough to capture that.

llvm/lib/Analysis/InlineAdvisor.cpp

49

Makes sense, split

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

54

If the remarks are generated via -Rpass there can be more after:

inline.cpp:8:12: remark: _Z3foov inlined into main with (cost=0, threshold=375) at callsite main:2:12; [-Rpass=inline]
    return foo();

72–75

This was originally intended as a way to easily check which remark was not processed to give a hint to the user that an unexpected inlining difference could be caused by it. In practice I haven't found it to be too useful because other reasons more often led to replay mismatching:

1. Source line mismatch between GCC and LLVM, seen on destructor placement
2. Pre-inline IR difference, GCC performs tail recursion elimination pre-inline which changes what call-sites remain post-inline compared to Clang.

So these end up being red herrings. Removed.

117

I like it, changed.

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

17

Removed

[wenlei]

In D110658#3057147, @modimo wrote:

In D110658#3035652, @wenlei wrote:

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

I think using scope to capture the functionality here makes sense. For your definition of strict I see it as modifier to positive/negative. A strict+negative replay would not inline the remarks present but will attempt to inline everything else and strict+positive would inline the remarks but not inline anything else. Non-strict+negative would make sure the remarks are not inlined but defer the rest of the decisions to the original advisor and non-strict+positive will inline the remarks but defer the rest to the original advisor.

Strict+negative is useful because one way we can capture replay is by looking at all call-sites that remain in the assembly and eliminate all the rest (i.e. making sure they're inlined) which has the advantage that inlined sites no longer present in the assembly (e.g. simple accessors whose bodies completely get folded) will properly be inlined while in strict+positive mode because the dwarf information is gone no remarks are generated and the original call remains.

Using strict as modifier would make it even more flexible. I didn't think strict+negative would be useful, so didn't give it much thought. Actually it's a bit weird to represent negative decision using the same remark format (... inlined into ...) but relying on flag to tell positive vs negative, perhaps negative-ness can be represented with the remark line using ... not inlined into ... (same as missed remarks). This eliminates the need to tell positive/negative from command line and also enable mixing of positive and negative within a file (for non-strict mode). But we can defer that discussion/decision.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

42

What I meant is Inliner is equivalent to InlineCaller, but more of a concise and canonical term. I guess you intend to emphasize on top level inliner, but InlinerCaller doesn't really carry that info either. Not a big deal though.

llvm/lib/Transforms/IPO/Inliner.cpp

103

Please use enum directly to avoid parsing strings and error checking in ReplayInlineAdvisor's ctor. Search for cl::opt<SampleProfileFormat> OutputFormat or cl::opt<enum PassDebugLevel>PassDebugging. Description needs to be updated to be per value too.

llvm/test/Transforms/SampleProfile/inline-replay.ll

48

inline-replay-strict.txt does't seem to be included here? And let's rename it accordingly.

[modimo]

Review feedback. InlineCallersFromRemarks->CallersToReplay. Refactor creating a ReplayInlineAdvisor to use getReplayInlineAdvisor.

[modimo]

In D110658#3058700, @wenlei wrote:

In D110658#3057147, @modimo wrote:

In D110658#3035652, @wenlei wrote:

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

I think using scope to capture the functionality here makes sense. For your definition of strict I see it as modifier to positive/negative. A strict+negative replay would not inline the remarks present but will attempt to inline everything else and strict+positive would inline the remarks but not inline anything else. Non-strict+negative would make sure the remarks are not inlined but defer the rest of the decisions to the original advisor and non-strict+positive will inline the remarks but defer the rest to the original advisor.

Strict+negative is useful because one way we can capture replay is by looking at all call-sites that remain in the assembly and eliminate all the rest (i.e. making sure they're inlined) which has the advantage that inlined sites no longer present in the assembly (e.g. simple accessors whose bodies completely get folded) will properly be inlined while in strict+positive mode because the dwarf information is gone no remarks are generated and the original call remains.

Using strict as modifier would make it even more flexible. I didn't think strict+negative would be useful, so didn't give it much thought. Actually it's a bit weird to represent negative decision using the same remark format (... inlined into ...) but relying on flag to tell positive vs negative, perhaps negative-ness can be represented with the remark line using ... not inlined into ... (same as missed remarks). This eliminates the need to tell positive/negative from command line and also enable mixing of positive and negative within a file (for non-strict mode). But we can defer that discussion/decision.

Looks like the remarks format for not inlining is "will not be inlined into" (see OptimizationRemarkMissed in Inliner.cpp) which can be used to distinguish negative decisions. That being said, negative mode itself is different in that it assumes all other legal sites will be inlined. Remark format is for when replay has a decision, mode format is what happens to all the other inline sites.

Negative mode only really works in function-scope since module scope would likely lead to never-ending compilation. In my testing, however, it has an opportunity to be more faithful to replay a full inline tree than positive/strict since the outcome is directly going after having identical call-sites at the end. Positive can deviate since non-matching sites get heuristics applied and strict loses information if no assembly instructions remain detailing that an inlining has occurred. Negative OTOH will leave all call sites that existed earlier intact while generally ensuring every other call site gets inlined since the call didn't exist in inlining that generated the replay. Regardless, separate discussion for separate patch.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

42

Hmm maybe CallersToReplay? The key point is that this is a list of callers to replay on.

llvm/lib/Transforms/IPO/Inliner.cpp

103

Ah nice, I was looking for functionality like that but couldn't find it. Changed.

[modimo]

@mtrofin Adding the enum ReplayInlineScope revealed that I was adding #include "llvm/Analysis/ReplayInlineAdvisor.h" to InlineAdvisor.cpp which seems wrong from a dependency perspective. Looking at how the other advisors interact it seems the interface is to declare a getReplayInlineAdvisor in InlineAdvisor.h similar to getReleaseModeAdvisor and also define ReplayInlineScope there so including InlineAdvisor.h by itself is enough to be able to create a ReplayInlineAdvisor. Does this look right?

[wenlei]

lgtm, thanks.

That being said, negative mode itself is different in that it assumes all other legal sites will be inlined.

Depends on how we define it. This is how I think about it now: in non-strict mode, positive/negative means do or don't do certain inlining, but leave the rest to default heuristic. With strict mode, anything unspecific gets the opposite replay decision (not inline for positive, inline for negative). With that definition, negative is no different in that it can have both strict and non-strict mode.

Negative mode only really works in function-scope since module scope would likely lead to never-ending compilation.

I think negative + strict only works in function-scope; but negative + non-strict should be fine. On the other hand, people can spell out all negative decision in input, same as positive ones, then negative + strict works for whole program too.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

42

sounds good.

[modimo]

In D110658#3060164, @wenlei wrote:

lgtm, thanks.

That being said, negative mode itself is different in that it assumes all other legal sites will be inlined.

Depends on how we define it. This is how I think about it now: in non-strict mode, positive/negative means do or don't do certain inlining, but leave the rest to default heuristic. With strict mode, anything unspecific gets the opposite replay decision (not inline for positive, inline for negative). With that definition, negative is no different in that it can have both strict and non-strict mode.

Ah, I'm thinking ""will not be inlined into" vs. "inlined into" in the remarks is the differentiation for positive/negative as you suggested before. Thinking further, in that setup only strict-negative and strict-positive would exist as modifiers. If implemented we would drop the strict naming completely: unspecified would mean leave it to the heuristic and positive/negative would mean always not inline/inline call sites without remarks associated with them.

Negative mode only really works in function-scope since module scope would likely lead to never-ending compilation.

I think negative + strict only works in function-scope; but negative + non-strict should be fine. On the other hand, people can spell out all negative decision in input, same as positive ones, then negative + strict works for whole program too.

True, what I described is actually strict-negative mode under your definition.

[modimo]

Fix up description so to be consistent in that Function scope is default.

llvm/include/llvm/Analysis/InlineAdvisor.h

 ///
 /// - Development mode, for training new models.
 /// In this mode, we trade off runtime performance for flexibility. This mode
 /// requires the full C Tensorflow API library, and evaluates models
 /// dynamically. This mode also permits generating training logs, for offline
 /// training.
 enum class InliningAdvisorMode : int { Default, Release, Development };
 
+/// For Replay Inliner initialization
+enum class ReplayInlineScope : int { Function, Module };
+
 class InlineAdvisor;
 /// Capture state between an inlining decision having had been made, and
 /// its impact being observable. When collecting model training data, this
 /// allows recording features/decisions/partial reward data sets.
 ///
 /// Derivations of this type are expected to be tightly coupled with their
 /// InliningAdvisors. The base type implements the minimal contractual
 /// obligations.
public:
   InlineAdvisor(InlineAdvisor &&) = delete;
   virtual ~InlineAdvisor();
 
   /// Get an InlineAdvice containing a recommendation on whether to
   /// inline or not. \p CB is assumed to be a direct call. \p FAM is assumed to
   /// be up-to-date wrt previous inlining decisions. \p MandatoryOnly indicates
   /// only mandatory (always-inline) call sites should be recommended - this
   /// allows the InlineAdvisor track such inlininings.
-  /// Returns an InlineAdvice with the inlining recommendation.
+  /// Returns:
+  /// - An InlineAdvice with the inlining recommendation.
+  /// - Null when no recommendation is made (https://reviews.llvm.org/D110658).
+  /// TODO: Consider removing the Null return scenario by incorporating the
+  /// SampleProfile inliner into an InlineAdvisor
   std::unique_ptr<InlineAdvice> getAdvice(CallBase &CB,
                                           bool MandatoryOnly = false);
 
   /// This must be called when the Inliner pass is entered, to allow the
   /// InlineAdvisor update internal state, as result of function passes run
   /// between Inliner pass runs (for the same module).
   virtual void onPassEntry() {}
 
struct Result {
     bool invalidate(Module &, const PreservedAnalyses &PA,
                     ModuleAnalysisManager::Invalidator &) {
       // Check whether the analysis has been explicitly invalidated. Otherwise,
       // it's stateless and remains preserved.
       auto PAC = PA.getChecker<InlineAdvisorAnalysis>();
       return !PAC.preservedWhenStateless();
     }
     bool tryCreate(InlineParams Params, InliningAdvisorMode Mode,
-                   StringRef ReplayFile);
+                   StringRef ReplayFile, ReplayInlineScope ReplayScope);
     InlineAdvisor *getAdvisor() const { return Advisor.get(); }
 
   private:
     Module &M;
     ModuleAnalysisManager &MAM;
     std::unique_ptr<InlineAdvisor> Advisor;
   };
 
   Result run(Module &M, ModuleAnalysisManager &MAM) { return Result(M, MAM); }
 };
 
 #ifdef LLVM_HAVE_TF_AOT
 std::unique_ptr<InlineAdvisor>
 getReleaseModeAdvisor(Module &M, ModuleAnalysisManager &MAM);
 #endif
 
 #ifdef LLVM_HAVE_TF_API
 std::unique_ptr<InlineAdvisor>
 getDevelopmentModeAdvisor(Module &M, ModuleAnalysisManager &MAM,
                           std::function<bool(CallBase &)> GetDefaultAdvice);
 #endif
 
+std::unique_ptr<InlineAdvisor> getReplayInlineAdvisor(
+    Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
+    ReplayInlineScope Scope, bool EmitRemarks);
+
 // Default (manual policy) decision making helper APIs. Shared with the legacy
 // pass manager inliner.
 
 /// Return the cost only if the inliner should attempt to inline at the given
 /// CallSite. If we return the cost, we will emit an optimisation remark later
 /// using that cost, so we won't do so from this function. Return None if
 /// inlining should not be attempted.
 Optional<InlineCost>

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

 
 /// Replay inline advisor that uses optimization remarks from inlining of
 /// previous build to guide current inlining. This is useful for inliner tuning.
 class ReplayInlineAdvisor : public InlineAdvisor {
 public:
   ReplayInlineAdvisor(Module &M, FunctionAnalysisManager &FAM,
                       LLVMContext &Context,
                       std::unique_ptr<InlineAdvisor> OriginalAdvisor,
-                      StringRef RemarksFile, bool EmitRemarks);
+                      StringRef RemarksFile, ReplayInlineScope Scope,
+                      bool EmitRemarks);
   std::unique_ptr<InlineAdvice> getAdviceImpl(CallBase &CB) override;
   bool areReplayRemarksLoaded() const { return HasReplayRemarks; }
 
 private:
-  StringSet<> InlineSitesFromRemarks;
   std::unique_ptr<InlineAdvisor> OriginalAdvisor;
   bool HasReplayRemarks = false;
+  ReplayInlineScope Scope;
   bool EmitRemarks = false;
+
+  StringMap<bool> InlineSitesFromRemarks;
+  StringSet<> CallersToReplay;
 };
 } // namespace llvm
 #endif // LLVM_ANALYSIS_REPLAYINLINEADVISOR_H

llvm/include/llvm/Transforms/IPO/Inliner.h

 //===- Inliner.h - Inliner pass and infrastructure --------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_IPO_INLINER_H
 #define LLVM_TRANSFORMS_IPO_INLINER_H
 
 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/LazyCallGraph.h"
-#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/Utils/ImportedFunctionsInliningStatistics.h"
 #include "llvm/IR/PassManager.h"
 #include <utility>
 
 namespace llvm {
 
 class AssumptionCacheTracker;
 class CallGraph;

llvm/lib/Analysis/InlineAdvisor.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
-#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;

 
 // An integer used to limit the cost of inline deferral.  The default negative
 // number tells shouldBeDeferred to only take the secondary cost into account.
 static cl::opt<int>
     InlineDeferralScale("inline-deferral-scale",
                         cl::desc("Scale to limit the cost of inline deferral"),
                         cl::init(2), cl::Hidden);
 
 extern cl::opt<InlinerFunctionImportStatsOpts> InlinerFunctionImportStats;
 
 namespace {
 using namespace llvm::ore;
 class MandatoryInlineAdvice : public InlineAdvice {
 public:
   MandatoryInlineAdvice(InlineAdvisor *Advisor, CallBase &CB,
                         OptimizationRemarkEmitter &ORE,
                         bool IsInliningMandatory)
void InlineAdvice::recordInliningWithCalleeDeleted() {
   Advisor->markFunctionAsDeleted(Callee);
   recordInliningWithCalleeDeletedImpl();
 }
 
 AnalysisKey InlineAdvisorAnalysis::Key;
 
 bool InlineAdvisorAnalysis::Result::tryCreate(InlineParams Params,
                                               InliningAdvisorMode Mode,
-                                              StringRef ReplayFile) {
+                                              StringRef ReplayFile,
+                                              ReplayInlineScope ReplayScope) {
   auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   switch (Mode) {
   case InliningAdvisorMode::Default:
     LLVM_DEBUG(dbgs() << "Using default inliner heuristic.\n");
     Advisor.reset(new DefaultInlineAdvisor(M, FAM, Params));
     // Restrict replay to default advisor, ML advisors are stateful so
     // replay will need augmentations to interleave with them correctly.
     if (!ReplayFile.empty()) {
-      Advisor = std::make_unique<ReplayInlineAdvisor>(
-          M, FAM, M.getContext(), std::move(Advisor), ReplayFile,
+      Advisor = llvm::getReplayInlineAdvisor(
+          M, FAM, M.getContext(), std::move(Advisor), ReplayFile, ReplayScope,
           /* EmitRemarks =*/true);
     }
     break;
   case InliningAdvisorMode::Development:
 #ifdef LLVM_HAVE_TF_API
     LLVM_DEBUG(dbgs() << "Using development-mode inliner policy.\n");
     Advisor =
         llvm::getDevelopmentModeAdvisor(M, MAM, [&FAM, Params](CallBase &CB) {

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

 // inlining steady.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/Support/LineIterator.h"
+#include <memory>
 
 using namespace llvm;
 
-#define DEBUG_TYPE "inline-replay"
+#define DEBUG_TYPE "replay-inline"
 
 ReplayInlineAdvisor::ReplayInlineAdvisor(
     Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
     std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    bool EmitRemarks)
+    ReplayInlineScope Scope, bool EmitRemarks)
     : InlineAdvisor(M, FAM), OriginalAdvisor(std::move(OriginalAdvisor)),
-      HasReplayRemarks(false), EmitRemarks(EmitRemarks) {
+      HasReplayRemarks(false), Scope(Scope), EmitRemarks(EmitRemarks) {
+
   auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(RemarksFile);
   std::error_code EC = BufferOrErr.getError();
   if (EC) {
     Context.emitError("Could not open remarks file: " + EC.message());
     return;
   }
 
   // Example for inline remarks to parse:
-  //   main:3:1.1: '_Z3subii' inlined into 'main' at callsite sum:1 @ main:3:1.1
+  //   main:3:1.1: '_Z3subii' inlined into 'main' at callsite sum:1 @
+  //   main:3:1.1;
   // We use the callsite string after `at callsite` to replay inlining.
   line_iterator LineIt(*BufferOrErr.get(), /*SkipBlanks=*/true);
   for (; !LineIt.is_at_eof(); ++LineIt) {
     StringRef Line = *LineIt;
     auto Pair = Line.split(" at callsite ");
 
-    StringRef Callee = Pair.first.split(" inlined into")
-                           .first.rsplit(": '")
-                           .second.drop_back();
+    auto CalleeCaller = Pair.first.split("' inlined into '");
+
+    StringRef Callee = CalleeCaller.first.rsplit(": '").second;
+    StringRef Caller = CalleeCaller.second.rsplit("'").first;
+
     auto CallSite = Pair.second.split(";").first;
 
-    if (Callee.empty() || CallSite.empty())
-      continue;
+    if (Callee.empty() || Caller.empty() || CallSite.empty()) {
+      Context.emitError("Invalid remark format: " + Line);
+      return;
+    }
 
     std::string Combined = (Callee + CallSite).str();
-    InlineSitesFromRemarks.insert(Combined);
+    InlineSitesFromRemarks[Combined] = false;
+    if (Scope == ReplayInlineScope::Function)
+      CallersToReplay.insert(Caller);
   }
 
   HasReplayRemarks = true;
 }
 
+std::unique_ptr<InlineAdvisor> llvm::getReplayInlineAdvisor(
+    Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
+    ReplayInlineScope Scope, bool EmitRemarks) {
+  auto Advisor = std::make_unique<ReplayInlineAdvisor>(
+      M, FAM, Context, std::move(OriginalAdvisor), RemarksFile, Scope,
+      EmitRemarks);
+  if (!Advisor->areReplayRemarksLoaded())
+    Advisor.reset();
+  return Advisor;
+}
+
 std::unique_ptr<InlineAdvice> ReplayInlineAdvisor::getAdviceImpl(CallBase &CB) {
   assert(HasReplayRemarks);
 
   Function &Caller = *CB.getCaller();
   auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(Caller);
 
-  if (InlineSitesFromRemarks.empty())
-    return std::make_unique<DefaultInlineAdvice>(this, CB, None, ORE,
-                                                 EmitRemarks);
+  Optional<InlineCost> InlineRecommended;
 
+  if (Scope == ReplayInlineScope::Module ||
+      CallersToReplay.count(CB.getFunction()->getName())) {
     std::string CallSiteLoc = getCallSiteLocation(CB.getDebugLoc());
     StringRef Callee = CB.getCalledFunction()->getName();
     std::string Combined = (Callee + CallSiteLoc).str();
-  auto Iter = InlineSitesFromRemarks.find(Combined);
 
-  Optional<InlineCost> InlineRecommended = None;
+    auto Iter = InlineSitesFromRemarks.find(Combined);
     if (Iter != InlineSitesFromRemarks.end()) {
-    InlineRecommended = llvm::InlineCost::getAlways("found in replay");
+      InlineSitesFromRemarks[Combined] = true;
+      InlineRecommended = llvm::InlineCost::getAlways("previously inlined");
+    }
+  } else if (Scope == ReplayInlineScope::Function) {
+    if (OriginalAdvisor)
+      return OriginalAdvisor->getAdvice(CB);
+    return {};
   }
 
   return std::make_unique<DefaultInlineAdvice>(this, CB, InlineRecommended, ORE,
                                                EmitRemarks);
 }

llvm/lib/Transforms/IPO/Inliner.cpp

DisableInlinedAllocaMerging("disable-inlined-alloca-merging",
                                 cl::init(false), cl::Hidden);
 
 extern cl::opt<InlinerFunctionImportStatsOpts> InlinerFunctionImportStats;
 
 static cl::opt<std::string> CGSCCInlineReplayFile(
     "cgscc-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
-        "by inlining from cgscc inline remarks."),
+        "by cgscc inlining."),
+    cl::Hidden);
+
+static cl::opt<ReplayInlineScope> CGSCCInlineReplayScope(
+    "cgscc-inline-replay-scope", cl::init(ReplayInlineScope::Function),
+    cl::values(clEnumValN(ReplayInlineScope::Function, "Function",
+                          "Replay on functions that have remarks associated "
+                          "with them (default)"),
+               clEnumValN(ReplayInlineScope::Module, "Module",
+                          "Replay on the entire module")),
+    cl::desc("Whether inline replay should be applied to the entire "
+             "Module or just the Functions (default) that are present as "
+             "callers in remarks during cgscc inlining."),
     cl::Hidden);
 
 static cl::opt<bool> InlineEnablePriorityOrder(
     "inline-enable-priority-order", cl::Hidden, cl::init(false),
     cl::desc("Enable the priority inline order for the inliner"));
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID) : CallGraphSCCPass(ID) {}
 
if (!IAA) {
     // In this case, we need to use the provided FAM, which is valid for the
     // duration of the inliner pass, and thus the lifetime of the owned advisor.
     // The one we would get from the MAM can be invalidated as a result of the
     // inliner's activity.
     OwnedAdvisor =
         std::make_unique<DefaultInlineAdvisor>(M, FAM, getInlineParams());
 
     if (!CGSCCInlineReplayFile.empty())
-      OwnedAdvisor = std::make_unique<ReplayInlineAdvisor>(
+      OwnedAdvisor = getReplayInlineAdvisor(
           M, FAM, M.getContext(), std::move(OwnedAdvisor),
-          CGSCCInlineReplayFile,
+          CGSCCInlineReplayFile, CGSCCInlineReplayScope,
           /*EmitRemarks=*/true);
 
     return *OwnedAdvisor;
   }
   assert(IAA->getAdvisor() &&
          "Expected a present InlineAdvisorAnalysis also have an "
          "InlineAdvisor initialized");
   return *IAA->getAdvisor();
while (!Calls->empty() && Calls->front().first->getCaller() == &F) {
         LLVM_DEBUG(dbgs() << "Skipping inlining internal SCC edge from a node "
                              "previously split out of this SCC by inlining: "
                           << F.getName() << " -> " << Callee.getName() << "\n");
         setInlineRemark(*CB, "recursive SCC split");
         continue;
       }
 
       auto Advice = Advisor.getAdvice(*CB, OnlyMandatory);
+
       // Check whether we want to inline this callsite.
-      if (!Advice->isInliningRecommended()) {
+      if (!Advice || !Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         continue;
       }
 
       // Setup the data structure used to plumb customization into the
       // `InlineFunction` routine.
       InlineFunctionInfo IFI(
           /*cg=*/nullptr, GetAssumptionCache, PSI,
ModuleInlinerWrapperPass::ModuleInlinerWrapperPass(InlineParams Params,
   if (MandatoryFirst)
     PM.addPass(InlinerPass(/*OnlyMandatory*/ true));
   PM.addPass(InlinerPass());
 }
 
 PreservedAnalyses ModuleInlinerWrapperPass::run(Module &M,
                                                 ModuleAnalysisManager &MAM) {
   auto &IAA = MAM.getResult<InlineAdvisorAnalysis>(M);
-  if (!IAA.tryCreate(Params, Mode, CGSCCInlineReplayFile)) {
+  if (!IAA.tryCreate(Params, Mode, CGSCCInlineReplayFile,
+                     CGSCCInlineReplayScope)) {
     M.getContext().emitError(
         "Could not setup Inlining Advisor for the requested "
         "mode and/or options");
     return PreservedAnalyses::all();
   }
 
   // We wrap the CGSCC pipeline in a devirtualization repeater. This will try
   // to detect when we devirtualize indirect calls and iterate the SCC passes

llvm/lib/Transforms/IPO/SampleProfile.cpp

 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/PostDominators.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
-#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Dominators.h"

 
 static cl::opt<std::string> ProfileInlineReplayFile(
     "sample-profile-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by inlining from sample profile loader."),
     cl::Hidden);
 
+static cl::opt<ReplayInlineScope> ProfileInlineReplayScope(
+    "sample-profile-inline-replay-scope", cl::init(ReplayInlineScope::Function),
+    cl::values(clEnumValN(ReplayInlineScope::Function, "Function",
+                          "Replay on functions that have remarks associated "
+                          "with them (default)"),
+               clEnumValN(ReplayInlineScope::Module, "Module",
+                          "Replay on the entire module")),
+    cl::desc("Whether inline replay should be applied to the entire "
+             "Module or just the Functions (default) that are present as "
+             "callers in remarks during sample profile inlining."),
+    cl::Hidden);
+
 static cl::opt<unsigned>
     MaxNumPromotions("sample-profile-icp-max-prom", cl::init(3), cl::Hidden,
                      cl::ZeroOrMore,
                      cl::desc("Max number of promotions for a single indirect "
                               "call callsite in sample profile loader"));
 
 static cl::opt<bool> OverwriteExistingWeights(
     "overwrite-existing-weights", cl::Hidden, cl::init(false),
protected:
   // For symbol in profile symbol list, whether to regard their profiles
   // to be accurate. It is mainly decided by existance of profile symbol
   // list and -profile-accurate-for-symsinlist flag, but it can be
   // overriden by -profile-sample-accurate or profile-sample-accurate
   // attribute.
   bool ProfAccForSymsInList;
 
   // External inline advisor used to replay inline decision from remarks.
-  std::unique_ptr<ReplayInlineAdvisor> ExternalInlineAdvisor;
+  std::unique_ptr<InlineAdvisor> ExternalInlineAdvisor;
 
   // A pseudo probe helper to correlate the imported sample counts.
   std::unique_ptr<PseudoProbeManager> ProbeManager;
 };
 
 class SampleProfileLoaderLegacyPass : public ModulePass {
 public:
   // Class identification, replacement for typeinfo
bool SampleProfileLoader::getInlineCandidate(InlineCandidate *NewCandidate,
   return true;
 }
 
 InlineCost
 SampleProfileLoader::shouldInlineCandidate(InlineCandidate &Candidate) {
   std::unique_ptr<InlineAdvice> Advice = nullptr;
   if (ExternalInlineAdvisor) {
     Advice = ExternalInlineAdvisor->getAdvice(*Candidate.CallInstr);
+    if (Advice) {
       if (!Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         return InlineCost::getNever("not previously inlined");
       }
       Advice->recordInlining();
       return InlineCost::getAlways("previously inlined");
     }
+  }
 
   // Adjust threshold based on call site hotness, only do this for callsite
   // prioritized inliner because otherwise cost-benefit check is done earlier.
   int SampleThreshold = SampleColdCallSiteThreshold;
   if (CallsitePrioritizedInline) {
     if (Candidate.CallsiteCount > PSI->getHotCountThreshold())
       SampleThreshold = SampleHotCallSiteThreshold;
     else if (!ProfileSizeInline)
bool SampleProfileLoader::doInitialization(Module &M,
   if (ProfAccForSymsInList) {
     NamesInProfile.clear();
     if (auto NameTable = Reader->getNameTable())
       NamesInProfile.insert(NameTable->begin(), NameTable->end());
     CoverageTracker.setProfAccForSymsInList(true);
   }
 
   if (FAM && !ProfileInlineReplayFile.empty()) {
-    ExternalInlineAdvisor = std::make_unique<ReplayInlineAdvisor>(
+    ExternalInlineAdvisor = getReplayInlineAdvisor(
         M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr, ProfileInlineReplayFile,
-        /*EmitRemarks=*/false);
-    if (!ExternalInlineAdvisor->areReplayRemarksLoaded())
-      ExternalInlineAdvisor.reset();
+        ProfileInlineReplayScope, /*EmitRemarks=*/false);
   }
 
   // Apply tweaks if context-sensitive profile is available.
   if (Reader->profileIsCS()) {
     ProfileIsCS = true;
     FunctionSamples::ProfileIsCS = true;
 
     // Enable priority-base inliner and size inline by default for CSSPGO.

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-cgscc-inline-replay` to legacy pass manager is a no-op.
 
-;; Check replay inline decisions
-; RUN: opt < %s -passes=inline -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
-; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+;; Check baseline inline decisions
+; RUN: opt < %s -passes=inline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT %s
+
+;; Check module-scope replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+
+;; Check function-scope inline replay decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Function -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION %s
+
+;; Check behavior on non-existent replay file
+; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
+
+;; Check scope inlining errors out on non <Module|Function> inputs
+; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=function -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
+
+; DEFAULT: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; DEFAULT: '_Z3sumii' inlined into 'main' with (cost={{[-0-9]+}}
+; DEFAULT-NOT: '_Z3subii' inlined into 'main'
+
+; REPLAY: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY: '_Z3subii' inlined into 'main' with (cost=always)
+
+; REPLAY-FUNCTION: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION: '_Z3sumii' inlined into 'main' with (cost=always)
+
+; REPLAY-ERROR: error: Could not open remarks file: Is a directory
+; REPLAY-ERROR-SCOPE: opt: for the --cgscc-inline-replay-scope option: Cannot find option named 'function'!
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4

 !19 = !DILocation(line: 10, scope: !20)
 !20 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 2)
 !21 = !DILocation(line: 10, scope: !22)
 !22 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 4)
 !23 = !DILocation(line: 10, scope: !24)
 !24 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 6)
 !25 = !DILocation(line: 11, scope: !12)
 !26 = !DILocation(line: 12, scope: !12)
-
-; DEFAULT: '_Z3subii' inlined into '_Z3sumii'
-; DEFAULT: '_Z3sumii' inlined into 'main'
-; DEFAULT-NOT: '_Z3subii' inlined into 'main'
-
-; REPLAY: '_Z3sumii' inlined into 'main'
-; REPLAY: '_Z3subii' inlined into 'main'
-; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-function-scope.txt

+remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

 remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
-remark: calls.cc:4:0: '_Z3subii' inlined into 'main' to match profiling context with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;
+remark: calls.cc:4:0: '_Z3subii' inlined into 'main' to match profiling context with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-topdown-inline-all.prof

+main:225715:0
+ 2.1: 5553
+ 3: 5391
+ 3.1: _Z3sumii:50000
+  1: _Z3subii:50000
+   1: 0
+
+_Z3sumii:6010:50000
+ 1: _Z3subii:60000
+   1: 9
 No newline at end of file

llvm/test/Transforms/SampleProfile/inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-sample-profile-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
 ;; Check replay inline decisions
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module  -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+
+;; Check baseline inline decisions with "inline-topdown-inline-all.prof" which inlines all sites
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT-ALL %s
+
+;; Check function scope replay inline decisions with "inline-topdown-inline-all.prof" and "inline-topdown-function-scope.txt" which only contains: '_Z3sumii' inlined into 'main'
+;; 1. _Z3sumii is inlined into main, but all other inline candidates in main (e.g. _Z3subii) are not inlined
+;; 2. Inline decisions made in other functions match default sample inlining, in this case _Z3subii is inlined into _Z3sumii
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function-scope.txt -sample-profile-inline-replay-scope=Function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION %s
+
+;; Check behavior on non-existent replay file
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
+
+;; Check scope inlining errors out on non <Module|Function> inputs
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
+
+; DEFAULT: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT-NOT: '_Z3subii' inlined into 'main'
+
+; REPLAY: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'
+
+; DEFAULT-ALL: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT-ALL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT-ALL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ALL-FUNCTION : _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ERROR: error: Could not open remarks file: Is a directory
+; REPLAY-ERROR-SCOPE: opt: for the --sample-profile-inline-replay-scope option: Cannot find option named 'function'!
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4
   store i32 %y, i32* %y.addr, align 4
   %tmp = load i32, i32* %x.addr, align 4, !dbg !8
   %tmp1 = load i32, i32* %y.addr, align 4, !dbg !8
   %add = add nsw i32 %tmp, %tmp1, !dbg !8
   %tmp2 = load i32, i32* %x.addr, align 4, !dbg !8
   %tmp3 = load i32, i32* %y.addr, align 4, !dbg !8
   %call = call i32 @_Z3subii(i32 %tmp2, i32 %tmp3), !dbg !8
   ret i32 %add, !dbg !8

 !19 = !DILocation(line: 10, scope: !20)
 !20 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 2)
 !21 = !DILocation(line: 10, scope: !22)
 !22 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 4)
 !23 = !DILocation(line: 10, scope: !24)
 !24 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 6)
 !25 = !DILocation(line: 11, scope: !12)
 !26 = !DILocation(line: 12, scope: !12)
-
-
-; DEFAULT: '_Z3sumii' inlined into 'main'
-; DEFAULT: '_Z3subii' inlined into '_Z3sumii'
-; DEFAULT-NOT: '_Z3subii' inlined into 'main'
-
-; REPLAY: '_Z3sumii' inlined into 'main'
-; REPLAY: '_Z3subii' inlined into 'main'
-; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'