[InlineAdvisor] Add -inline-replay-scope=<Function|Module> to control replay scope

[modimo]

Summary

The goal is to allow grafting an inline tree from Clang or GCC into a new compilation without affecting other functions. For GCC, we're doing this by extracting the inline tree from dwarf information and generating the equivalent remarks.

This allows easier side-by-side asm analysis and a trial way to see if a particular inlining setup provides benefits by itself.

Testing:
ninja check-all

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

835

Since getAdvice returns a unique_ptr, returning a null seems like a good way to indicate "no advice". Does that also make sense for the ML inliner @mtrofin?

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

835

The design intent was for the advice to be clear, i.e. either inline or not. You probably want to delegate the decision to some other advisor if you don't have one? i.e. the remarks advisor could delegate to the default one - and return a non-null Advice. WDYT?

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

835

It does delegate it in the CGSCC case because we have a nicely nested system with the default one taking over.

Unfortunately the SampleProfile inliner currently isn't using the InlineAdvisor setup. That being said, perhaps the better approach is to adapt the SampleProfile inliner to an InlineAdvisor so the replay advisor will always return a non-null.

[modimo]

Formatting

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

835

The SampleProfile inliner needs additional information beyond CallBase to make its decision. Namely, InlineCandidate which contains sampling information. Does it make sense to extend getAdvice to take additional information?

[mtrofin]

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

72

nit: probably no need for curly brackets.

llvm/lib/Transforms/IPO/Inliner.cpp

835

I don't follow why, but to the scope of this patch, couldn't the ReplayInlineAdvisor return an Advice that says "no" when it has no advice to provide?

[modimo]

llvm/lib/Transforms/IPO/Inliner.cpp

835

In SampleProfile, it needs 3 states:

1. Yes when it matches replay
2. No when it doesn't match replay
3. HasNoAdvice when in strict mode and we want the SampleProfile inliner to make the decision

ATM there's only (1) and (2), I'm using null to represent (3). An alternative could be to have a HasNoAdvice state that maps to no inline but can be queried to differentiate between (2) and (3)

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

835

I think I see now. OK, let's allow a null return then from getAdvice; to answer your original question, I don't think it affects advisors that want to be categorical (i.e. an advisor can always return yes/no), but it affects consumers, and your patch handles that. The ML stuff doesn't consume, it just implements the advisor interfaces.

We can explore afterwards if/how to adapt SampleProfile to an advisor design; we'd definitely not want to introduce that InlineCandidate in the advisor interface, the former is too specific to the sample profiler.

Could you leave a comment on the getAdvice() abstract API re. the fact that it can return nothing, a pointer to this review, and a TODO that maybe we can tighten it?

Thanks!

[modimo]

Add comments on getAdvice abstract API. Remove curly brackets in ReplayInlineAdvisor destructor

[wenlei]

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

43

nit: InlineCallersFromRemarks -> InlinerFromRemarks

llvm/lib/Analysis/InlineAdvisor.cpp

50

Sample loader replay was independent of cgscc replay as they use different switches for taking different inputs. Now the "mode" switch is global, so both replays are tied to a single mode even if they use different input files. I think it's more flexible if they are still kept independent (the hot/cold thresholds etc are all kept separate for the two inliners).

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

46

nits:

• Pair.first.split(" inlined into ") is duplicated string search and can be merged.
• If you do Pair.first.split("' inlined into '"), the drop_back() and drop_front() can be removed, and it helps readability..

52

Pair.second.drop_back() works? or you expect something after ;?

54

Check Caller.epmty() too?

I would also add a warning if any of the three is empty - invalid remarks, can't be replayed etc.. instead of silently swallow it.

68–71

I'm not sure whether printing such debug dumps from dtor is a good idea. Lifetime of objects could potentially be somewhat arbitrary, but logs better be printed in more controlled/organized fashion. Looking at the test case, it also seems like these prints are added for testing purpose? See the comments in the test case - we better avoid using debug prints for testing.

81–82

Perhaps we can remove this for simplicity. The shortcut isn't helping much.

120

nit:

if (!Strict || InlineCallersFromRemarks.count(CB.getFunction()->getName())) {
  ...
}
else if (Strict) {
  if (OriginalAdvisor)
    return OriginalAdvisor->getAdvice(CB);
  return {};
}

This way we don't need a lambda - since it's only called once it can be inlined.

llvm/lib/Transforms/IPO/Inliner.cpp

835

@mtrofin I added InlineCandidate stuff in sample loader in https://reviews.llvm.org/D94001 and https://reviews.llvm.org/D95024, but the use of extra info for inline decision predates these changes. As to why we can't just use CallBase there - the inlining happens before profile is annotated on IR as branch metadata (it's the sample *loader*), so none of the BPI, BFI stuff is available, hence we have to use the raw context profile representation alongside CallBase to make inline decisions.

I agree that the InlineCandidate there is very specific and not good for advisor API. However I think having three state yes/no/unknown is more flexible, and some advisors can still choose to not use the unknown state. We can use NULL for unknown, or a different representation.

You probably want to delegate the decision to some other advisor if you don't have one

Conceptually there's a local "unknown" for some advisor already. Forcing a yes/no decision and not having a representation for unknown makes the inline advisor system "closed" instead of "open" in the sense that everything needs to reach a decision within the advisor hierarchy without any external help. In ideal world, it's clean, and sure we can try to convert everything into the system, but practically it might be too restrictive?

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

21

This would fail ninja check with release builds. Usually we try to avoid testing that requires debug builds unless it's critical and no other way to observe.

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

835

Maybe. Right now though I think a good chunk of the discussion stems from the fact that SampleProfile does 2 things in one: populates profile info; and inlines. We should probably split this, but that's both very much out of the scope of this patch, and orthogonal to the tri/bi-state advisor topic.

[modimo]

Address feedback and change over to replay-scope

[modimo]

In D110658#3035652, @wenlei wrote:

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

I think using scope to capture the functionality here makes sense. For your definition of strict I see it as modifier to positive/negative. A strict+negative replay would not inline the remarks present but will attempt to inline everything else and strict+positive would inline the remarks but not inline anything else. Non-strict+negative would make sure the remarks are not inlined but defer the rest of the decisions to the original advisor and non-strict+positive will inline the remarks but defer the rest to the original advisor.

Strict+negative is useful because one way we can capture replay is by looking at all call-sites that remain in the assembly and eliminate all the rest (i.e. making sure they're inlined) which has the advantage that inlined sites no longer present in the assembly (e.g. simple accessors whose bodies completely get folded) will properly be inlined while in strict+positive mode because the dwarf information is gone no remarks are generated and the original call remains.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

43

This is to indicate which callers should have replay enforced in function scope, don't think InlinerFromRemarks is enough to capture that.

llvm/lib/Analysis/InlineAdvisor.cpp

50

Makes sense, split

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

52

If the remarks are generated via -Rpass there can be more after:

inline.cpp:8:12: remark: _Z3foov inlined into main with (cost=0, threshold=375) at callsite main:2:12; [-Rpass=inline]
    return foo();

68–71

This was originally intended as a way to easily check which remark was not processed to give a hint to the user that an unexpected inlining difference could be caused by it. In practice I haven't found it to be too useful because other reasons more often led to replay mismatching:

1. Source line mismatch between GCC and LLVM, seen on destructor placement
2. Pre-inline IR difference, GCC performs tail recursion elimination pre-inline which changes what call-sites remain post-inline compared to Clang.

So these end up being red herrings. Removed.

120

I like it, changed.

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

21

Removed

[wenlei]

In D110658#3057147, @modimo wrote:

In D110658#3035652, @wenlei wrote:

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

I think using scope to capture the functionality here makes sense. For your definition of strict I see it as modifier to positive/negative. A strict+negative replay would not inline the remarks present but will attempt to inline everything else and strict+positive would inline the remarks but not inline anything else. Non-strict+negative would make sure the remarks are not inlined but defer the rest of the decisions to the original advisor and non-strict+positive will inline the remarks but defer the rest to the original advisor.

Strict+negative is useful because one way we can capture replay is by looking at all call-sites that remain in the assembly and eliminate all the rest (i.e. making sure they're inlined) which has the advantage that inlined sites no longer present in the assembly (e.g. simple accessors whose bodies completely get folded) will properly be inlined while in strict+positive mode because the dwarf information is gone no remarks are generated and the original call remains.

Using strict as modifier would make it even more flexible. I didn't think strict+negative would be useful, so didn't give it much thought. Actually it's a bit weird to represent negative decision using the same remark format (... inlined into ...) but relying on flag to tell positive vs negative, perhaps negative-ness can be represented with the remark line using ... not inlined into ... (same as missed remarks). This eliminates the need to tell positive/negative from command line and also enable mixing of positive and negative within a file (for non-strict mode). But we can defer that discussion/decision.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

43

What I meant is Inliner is equivalent to InlineCaller, but more of a concise and canonical term. I guess you intend to emphasize on top level inliner, but InlinerCaller doesn't really carry that info either. Not a big deal though.

llvm/lib/Transforms/IPO/Inliner.cpp

103

Please use enum directly to avoid parsing strings and error checking in ReplayInlineAdvisor's ctor. Search for cl::opt<SampleProfileFormat> OutputFormat or cl::opt<enum PassDebugLevel>PassDebugging. Description needs to be updated to be per value too.

llvm/test/Transforms/SampleProfile/inline-replay.ll

15

inline-replay-strict.txt does't seem to be included here? And let's rename it accordingly.

[modimo]

Review feedback. InlineCallersFromRemarks->CallersToReplay. Refactor creating a ReplayInlineAdvisor to use getReplayInlineAdvisor.

[modimo]

In D110658#3058700, @wenlei wrote:

In D110658#3057147, @modimo wrote:

In D110658#3035652, @wenlei wrote:

Thanks for working on this. Though perhaps there's some misunderstanding on what "strict" mode means (sorry I should have clarified more beforehand). In the original inline replay patch (https://reviews.llvm.org/D83743), I mentioned strict/positive/negative mode.

The only mode we have before and after this patch is the strict mode. "strict" means existing remarks indicate positive decision, and the rest are all considered negative decision. "positive" means existing remarks indicate positive decision, but the rest is left to other heuristics (or fall back advisors). Accordingly, negative means the input only specifies negative decision.

The change is limiting the scope of replay to function, as opposed to module or whole program. A powerful inline replay framework would support scope={module|function} x mode={strict|positive|negative}.

So I would suggest adding a flag -inline-replay-scope=[function|module] (with a default value), instead of -inline-replay-strict. Later on if we add positive/negative mode, we could add flag -inline-replay-mode=[strict|positive|negative], but for now we don't need it. Thoughts?

I think using scope to capture the functionality here makes sense. For your definition of strict I see it as modifier to positive/negative. A strict+negative replay would not inline the remarks present but will attempt to inline everything else and strict+positive would inline the remarks but not inline anything else. Non-strict+negative would make sure the remarks are not inlined but defer the rest of the decisions to the original advisor and non-strict+positive will inline the remarks but defer the rest to the original advisor.

Strict+negative is useful because one way we can capture replay is by looking at all call-sites that remain in the assembly and eliminate all the rest (i.e. making sure they're inlined) which has the advantage that inlined sites no longer present in the assembly (e.g. simple accessors whose bodies completely get folded) will properly be inlined while in strict+positive mode because the dwarf information is gone no remarks are generated and the original call remains.

Using strict as modifier would make it even more flexible. I didn't think strict+negative would be useful, so didn't give it much thought. Actually it's a bit weird to represent negative decision using the same remark format (... inlined into ...) but relying on flag to tell positive vs negative, perhaps negative-ness can be represented with the remark line using ... not inlined into ... (same as missed remarks). This eliminates the need to tell positive/negative from command line and also enable mixing of positive and negative within a file (for non-strict mode). But we can defer that discussion/decision.

Looks like the remarks format for not inlining is "will not be inlined into" (see OptimizationRemarkMissed in Inliner.cpp) which can be used to distinguish negative decisions. That being said, negative mode itself is different in that it assumes all other legal sites will be inlined. Remark format is for when replay has a decision, mode format is what happens to all the other inline sites.

Negative mode only really works in function-scope since module scope would likely lead to never-ending compilation. In my testing, however, it has an opportunity to be more faithful to replay a full inline tree than positive/strict since the outcome is directly going after having identical call-sites at the end. Positive can deviate since non-matching sites get heuristics applied and strict loses information if no assembly instructions remain detailing that an inlining has occurred. Negative OTOH will leave all call sites that existed earlier intact while generally ensuring every other call site gets inlined since the call didn't exist in inlining that generated the replay. Regardless, separate discussion for separate patch.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

43

Hmm maybe CallersToReplay? The key point is that this is a list of callers to replay on.

llvm/lib/Transforms/IPO/Inliner.cpp

103

Ah nice, I was looking for functionality like that but couldn't find it. Changed.

[modimo]

@mtrofin Adding the enum ReplayInlineScope revealed that I was adding #include "llvm/Analysis/ReplayInlineAdvisor.h" to InlineAdvisor.cpp which seems wrong from a dependency perspective. Looking at how the other advisors interact it seems the interface is to declare a getReplayInlineAdvisor in InlineAdvisor.h similar to getReleaseModeAdvisor and also define ReplayInlineScope there so including InlineAdvisor.h by itself is enough to be able to create a ReplayInlineAdvisor. Does this look right?

[wenlei]

lgtm, thanks.

That being said, negative mode itself is different in that it assumes all other legal sites will be inlined.

Depends on how we define it. This is how I think about it now: in non-strict mode, positive/negative means do or don't do certain inlining, but leave the rest to default heuristic. With strict mode, anything unspecific gets the opposite replay decision (not inline for positive, inline for negative). With that definition, negative is no different in that it can have both strict and non-strict mode.

Negative mode only really works in function-scope since module scope would likely lead to never-ending compilation.

I think negative + strict only works in function-scope; but negative + non-strict should be fine. On the other hand, people can spell out all negative decision in input, same as positive ones, then negative + strict works for whole program too.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

43

sounds good.

[modimo]

In D110658#3060164, @wenlei wrote:

lgtm, thanks.

That being said, negative mode itself is different in that it assumes all other legal sites will be inlined.

Depends on how we define it. This is how I think about it now: in non-strict mode, positive/negative means do or don't do certain inlining, but leave the rest to default heuristic. With strict mode, anything unspecific gets the opposite replay decision (not inline for positive, inline for negative). With that definition, negative is no different in that it can have both strict and non-strict mode.

Ah, I'm thinking ""will not be inlined into" vs. "inlined into" in the remarks is the differentiation for positive/negative as you suggested before. Thinking further, in that setup only strict-negative and strict-positive would exist as modifiers. If implemented we would drop the strict naming completely: unspecified would mean leave it to the heuristic and positive/negative would mean always not inline/inline call sites without remarks associated with them.

Negative mode only really works in function-scope since module scope would likely lead to never-ending compilation.

I think negative + strict only works in function-scope; but negative + non-strict should be fine. On the other hand, people can spell out all negative decision in input, same as positive ones, then negative + strict works for whole program too.

True, what I described is actually strict-negative mode under your definition.

[modimo]

Fix up description so to be consistent in that Function scope is default.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

 
 /// Replay inline advisor that uses optimization remarks from inlining of
 /// previous build to guide current inlining. This is useful for inliner tuning.
 class ReplayInlineAdvisor : public InlineAdvisor {
 public:
   ReplayInlineAdvisor(Module &M, FunctionAnalysisManager &FAM,
                       LLVMContext &Context,
                       std::unique_ptr<InlineAdvisor> OriginalAdvisor,
-                      StringRef RemarksFile, bool EmitRemarks);
+                      StringRef RemarksFile, bool Strict, bool EmitRemarks);
   std::unique_ptr<InlineAdvice> getAdviceImpl(CallBase &CB) override;
   bool areReplayRemarksLoaded() const { return HasReplayRemarks; }
 
+  ~ReplayInlineAdvisor();
+
 private:
-  StringSet<> InlineSitesFromRemarks;
   std::unique_ptr<InlineAdvisor> OriginalAdvisor;
   bool HasReplayRemarks = false;
+  bool Strict = false;
   bool EmitRemarks = false;
+
+  StringMap<bool> InlineSitesFromRemarks;
+  StringSet<> InlineCallersFromRemarks;
 };
 } // namespace llvm
 #endif // LLVM_ANALYSIS_REPLAYINLINEADVISOR_H

llvm/lib/Analysis/InlineAdvisor.cpp

 
 // An integer used to limit the cost of inline deferral.  The default negative
 // number tells shouldBeDeferred to only take the secondary cost into account.
 static cl::opt<int>
     InlineDeferralScale("inline-deferral-scale",
                         cl::desc("Scale to limit the cost of inline deferral"),
                         cl::init(2), cl::Hidden);
 
+cl::opt<bool> InlineReplayStrict(
+    "inline-replay-strict", cl::init(false),
+    cl::desc(
+        "Inline remarks are only replayed on called functions, otherwise defaults to original advisor."),
+    cl::Hidden);
+
 extern cl::opt<InlinerFunctionImportStatsOpts> InlinerFunctionImportStats;
 
 void DefaultInlineAdvice::recordUnsuccessfulInliningImpl(
     const InlineResult &Result) {
   using namespace ore;
   llvm::setInlineRemark(*OriginalCB, std::string(Result.getFailureReason()) +
                                          "; " + inlineCostStr(*OIC));
   ORE.emit([&]() {
bool InlineAdvisorAnalysis::Result::tryCreate(InlineParams Params,
   case InliningAdvisorMode::Default:
     LLVM_DEBUG(dbgs() << "Using default inliner heuristic.\n");
     Advisor.reset(new DefaultInlineAdvisor(M, FAM, Params));
     // Restrict replay to default advisor, ML advisors are stateful so
     // replay will need augmentations to interleave with them correctly.
     if (!ReplayFile.empty()) {
       Advisor = std::make_unique<ReplayInlineAdvisor>(
           M, FAM, M.getContext(), std::move(Advisor), ReplayFile,
-          /* EmitRemarks =*/true);
+          InlineReplayStrict, /* EmitRemarks =*/true);
     }
     break;
   case InliningAdvisorMode::Development:
 #ifdef LLVM_HAVE_TF_API
     LLVM_DEBUG(dbgs() << "Using development-mode inliner policy.\n");
     Advisor =
         llvm::getDevelopmentModeAdvisor(M, MAM, [&FAM, Params](CallBase &CB) {
           auto OIC = getDefaultInlineAdvice(CB, FAM, Params);

 }
 
 InlineAdvisor::InlineAdvisor(Module &M, FunctionAnalysisManager &FAM)
     : M(M), FAM(FAM) {
   if (InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No) {
     ImportedFunctionsStats =
         std::make_unique<ImportedFunctionsInliningStatistics>();
     ImportedFunctionsStats->setModuleInfo(M);
-  }
 }
 
 InlineAdvisor::~InlineAdvisor() {
   if (ImportedFunctionsStats) {
     assert(InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No);
     ImportedFunctionsStats->dump(InlinerFunctionImportStats ==
                                  InlinerFunctionImportStatsOpts::Verbose);
   }

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

 
 using namespace llvm;
 
 #define DEBUG_TYPE "inline-replay"
 
 ReplayInlineAdvisor::ReplayInlineAdvisor(
     Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
     std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    bool EmitRemarks)
+    bool Strict, bool EmitRemarks)
     : InlineAdvisor(M, FAM), OriginalAdvisor(std::move(OriginalAdvisor)),
-      HasReplayRemarks(false), EmitRemarks(EmitRemarks) {
+      HasReplayRemarks(false), Strict(Strict), EmitRemarks(EmitRemarks) {
   auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(RemarksFile);
   std::error_code EC = BufferOrErr.getError();
   if (EC) {
     Context.emitError("Could not open remarks file: " + EC.message());
     return;
   }
 
   // Example for inline remarks to parse:
-  //   main:3:1.1: '_Z3subii' inlined into 'main' at callsite sum:1 @ main:3:1.1
+  //   main:3:1.1: '_Z3subii' inlined into 'main' at callsite sum:1 @ main:3:1.1;
   // We use the callsite string after `at callsite` to replay inlining.
   line_iterator LineIt(*BufferOrErr.get(), /*SkipBlanks=*/true);
   for (; !LineIt.is_at_eof(); ++LineIt) {
     StringRef Line = *LineIt;
     auto Pair = Line.split(" at callsite ");
 
-    StringRef Callee = Pair.first.split(" inlined into")
+    StringRef Callee = Pair.first.split(" inlined into ")
                            .first.rsplit(": '")
                            .second.drop_back();
+    StringRef Caller = Pair.first.split(" inlined into ")
+                           .second.rsplit("'")
+                           .first.drop_front();
     auto CallSite = Pair.second.split(";").first;
 
     if (Callee.empty() || CallSite.empty())
       continue;
 
     std::string Combined = (Callee + CallSite).str();
-    InlineSitesFromRemarks.insert(Combined);
+    InlineSitesFromRemarks[Combined] = false;
+    if (Strict)
+        InlineCallersFromRemarks.insert(Caller);
   }
 
   HasReplayRemarks = true;
 }
 
+ReplayInlineAdvisor::~ReplayInlineAdvisor() {
+    if (HasReplayRemarks && Strict) {
+        for (const auto & Remark : InlineSitesFromRemarks) {
+            if (!Remark.second) {
+            LLVM_DEBUG(dbgs() << "Inline Replay Strict: Did not apply " << Remark.first() << "\n");
+            }
+        }
+    }
+}
+
 std::unique_ptr<InlineAdvice> ReplayInlineAdvisor::getAdviceImpl(CallBase &CB) {
   assert(HasReplayRemarks);
 
   Function &Caller = *CB.getCaller();
   auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(Caller);
 
-  if (InlineSitesFromRemarks.empty())
-    return std::make_unique<DefaultInlineAdvice>(this, CB, None, ORE,
-                                                 EmitRemarks);
+  if (!Strict && InlineSitesFromRemarks.empty())
+    return std::make_unique<DefaultInlineAdvice>(this, CB, None, ORE, EmitRemarks);
+
+  Optional<InlineCost> InlineRecommended;
 
+  auto GetReplayAdvice = [&](){
     std::string CallSiteLoc = getCallSiteLocation(CB.getDebugLoc());
     StringRef Callee = CB.getCalledFunction()->getName();
     std::string Combined = (Callee + CallSiteLoc).str();
-  auto Iter = InlineSitesFromRemarks.find(Combined);
     
-  Optional<InlineCost> InlineRecommended = None;
+    auto Iter = InlineSitesFromRemarks.find(Combined);
     if (Iter != InlineSitesFromRemarks.end()) {
-    InlineRecommended = llvm::InlineCost::getAlways("found in replay");
+     InlineSitesFromRemarks[Combined] = true;
+     InlineRecommended = llvm::InlineCost::getAlways("previously inlined");
+    }
+  };
+
+  if (Strict) {
+      if (InlineCallersFromRemarks.count(CB.getFunction()->getName()))
+        GetReplayAdvice();
+      else {
+          if (OriginalAdvisor)
+            return OriginalAdvisor->getAdvice(CB);
+        return {};
+      }
+        
   }
+  else
+    GetReplayAdvice();
 
-  return std::make_unique<DefaultInlineAdvice>(this, CB, InlineRecommended, ORE,
-                                               EmitRemarks);
+  return std::make_unique<DefaultInlineAdvice>(this, CB, InlineRecommended, ORE, EmitRemarks);
 }

llvm/lib/Transforms/IPO/Inliner.cpp

 
 static cl::opt<std::string> CGSCCInlineReplayFile(
     "cgscc-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by inlining from cgscc inline remarks."),
     cl::Hidden);
 
+extern cl::opt<bool> InlineReplayStrict;
+
 static cl::opt<bool> InlineEnablePriorityOrder(
     "inline-enable-priority-order", cl::Hidden, cl::init(false),
     cl::desc("Enable the priority inline order for the inliner"));
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID) : CallGraphSCCPass(ID) {}
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID, bool InsertLifetime)
     : CallGraphSCCPass(ID), InsertLifetime(InsertLifetime) {}
if (!IAA) {
     // The one we would get from the MAM can be invalidated as a result of the
     // inliner's activity.
     OwnedAdvisor =
         std::make_unique<DefaultInlineAdvisor>(M, FAM, getInlineParams());
 
     if (!CGSCCInlineReplayFile.empty())
       OwnedAdvisor = std::make_unique<ReplayInlineAdvisor>(
           M, FAM, M.getContext(), std::move(OwnedAdvisor),
           CGSCCInlineReplayFile,
+          InlineReplayStrict,
           /*EmitRemarks=*/true);
 
     return *OwnedAdvisor;
   }
   assert(IAA->getAdvisor() &&
          "Expected a present InlineAdvisorAnalysis also have an "
          "InlineAdvisor initialized");
   return *IAA->getAdvisor();
while (!Calls->empty() && Calls->front().first->getCaller() == &F) {
         LLVM_DEBUG(dbgs() << "Skipping inlining internal SCC edge from a node "
                              "previously split out of this SCC by inlining: "
                           << F.getName() << " -> " << Callee.getName() << "\n");
         setInlineRemark(*CB, "recursive SCC split");
         continue;
       }
 
       auto Advice = Advisor.getAdvice(*CB, OnlyMandatory);
+      
       // Check whether we want to inline this callsite.
-      if (!Advice->isInliningRecommended()) {
+      if (!Advice || !Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         continue;
       }
 
       // Setup the data structure used to plumb customization into the
       // `InlineFunction` routine.
       InlineFunctionInfo IFI(
           /*cg=*/nullptr, GetAssumptionCache, PSI,

llvm/lib/Transforms/IPO/SampleProfile.cpp

 
 static cl::opt<std::string> ProfileInlineReplayFile(
     "sample-profile-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by inlining from sample profile loader."),
     cl::Hidden);
 
+extern cl::opt<bool> InlineReplayStrict;
+
 static cl::opt<unsigned>
     MaxNumPromotions("sample-profile-icp-max-prom", cl::init(3), cl::Hidden,
                      cl::ZeroOrMore,
                      cl::desc("Max number of promotions for a single indirect "
                               "call callsite in sample profile loader"));
 
 static cl::opt<bool> OverwriteExistingWeights(
     "overwrite-existing-weights", cl::Hidden, cl::init(false),
bool SampleProfileLoader::getInlineCandidate(InlineCandidate *NewCandidate,
   return true;
 }
 
 InlineCost
 SampleProfileLoader::shouldInlineCandidate(InlineCandidate &Candidate) {
   std::unique_ptr<InlineAdvice> Advice = nullptr;
   if (ExternalInlineAdvisor) {
     Advice = ExternalInlineAdvisor->getAdvice(*Candidate.CallInstr);
+    if (Advice) {
         if (!Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         return InlineCost::getNever("not previously inlined");
         }
         Advice->recordInlining();
         return InlineCost::getAlways("previously inlined");
     }
+  }
 
   // Adjust threshold based on call site hotness, only do this for callsite
   // prioritized inliner because otherwise cost-benefit check is done earlier.
   int SampleThreshold = SampleColdCallSiteThreshold;
   if (CallsitePrioritizedInline) {
     if (Candidate.CallsiteCount > PSI->getHotCountThreshold())
       SampleThreshold = SampleHotCallSiteThreshold;
     else if (!ProfileSizeInline)
if (ProfAccForSymsInList) {
     if (auto NameTable = Reader->getNameTable())
       NamesInProfile.insert(NameTable->begin(), NameTable->end());
     CoverageTracker.setProfAccForSymsInList(true);
   }
 
   if (FAM && !ProfileInlineReplayFile.empty()) {
     ExternalInlineAdvisor = std::make_unique<ReplayInlineAdvisor>(
         M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr, ProfileInlineReplayFile,
-        /*EmitRemarks=*/false);
+        InlineReplayStrict, /*EmitRemarks=*/false);
     if (!ExternalInlineAdvisor->areReplayRemarksLoaded())
       ExternalInlineAdvisor.reset();
   }
 
   // Apply tweaks if context-sensitive profile is available.
   if (Reader->profileIsCS()) {
     ProfileIsCS = true;
     FunctionSamples::ProfileIsCS = true;

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-cgscc-inline-replay` to legacy pass manager is a no-op.
 
+;; Check baseline inline decisions
+; RUN: opt < %s -passes=inline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT %s
+
 ;; Check replay inline decisions
-; RUN: opt < %s -passes=inline -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
-; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+
+;; Check strict replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -inline-replay-strict -debug-only=inline-replay -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-STRICT %s
+
+; DEFAULT: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; DEFAULT: '_Z3sumii' inlined into 'main' with (cost={{[-0-9]+}}
+; DEFAULT-NOT: '_Z3subii' inlined into 'main'
+
+; REPLAY: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY: '_Z3subii' inlined into 'main' with (cost=always)
+
+; REPLAY-STRICT: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-STRICT: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-STRICT: Inline Replay Strict: Did not apply _Z3subii_Z3sumii:1:0 @ main:3:0.1
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4

 !19 = !DILocation(line: 10, scope: !20)
 !20 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 2)
 !21 = !DILocation(line: 10, scope: !22)
 !22 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 4)
 !23 = !DILocation(line: 10, scope: !24)
 !24 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 6)
 !25 = !DILocation(line: 11, scope: !12)
 !26 = !DILocation(line: 12, scope: !12)
-
-; DEFAULT: '_Z3subii' inlined into '_Z3sumii'
-; DEFAULT: '_Z3sumii' inlined into 'main'
-; DEFAULT-NOT: '_Z3subii' inlined into 'main'
-
-; REPLAY: '_Z3sumii' inlined into 'main'
-; REPLAY: '_Z3subii' inlined into 'main'
-; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-strict.txt

+remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

 remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
-remark: calls.cc:4:0: '_Z3subii' inlined into 'main' to match profiling context with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;
+remark: calls.cc:4:0: '_Z3subii' inlined into 'main' to match profiling context with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-topdown-inline-all.prof

+main:225715:0
+ 2.1: 5553
+ 3: 5391
+ 3.1: _Z3sumii:50000
+  1: _Z3subii:50000
+   1: 0
+
+_Z3sumii:6010:50000
+ 1: _Z3subii:60000
+   1: 9
 No newline at end of file

llvm/test/Transforms/SampleProfile/inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-sample-profile-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
 ;; Check replay inline decisions
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
 
+;; Check baseline inline decisions with "inline-topdown-inline-all.prof" which inlines all sites
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT-ALL %s
+
+;; Check strict replay inline decisions with "inline-topdown-inline-all.prof" which only contains: '_Z3sumii' inlined into 'main'
+;; 1. _Z3sumii is inlined into main, but all other inline candidates in main (e.g. _Z3subii) are not inlined
+;; 2. Inline decisions made in other functions match default sample inlining, in this case _Z3subii is inlined into _Z3sumii
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-strict.txt -inline-replay-strict -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-STRICT %s
+
+
+; DEFAULT: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT-NOT: '_Z3subii' inlined into 'main'
+
+; REPLAY: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'
+
+; DEFAULT-ALL: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT-ALL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; DEFAULT-ALL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ALL-STRICT : _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-STRICT-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-STRICT: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4
   store i32 %y, i32* %y.addr, align 4

 !19 = !DILocation(line: 10, scope: !20)
 !20 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 2)
 !21 = !DILocation(line: 10, scope: !22)
 !22 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 4)
 !23 = !DILocation(line: 10, scope: !24)
 !24 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 6)
 !25 = !DILocation(line: 11, scope: !12)
 !26 = !DILocation(line: 12, scope: !12)
-
-
-; DEFAULT: '_Z3sumii' inlined into 'main'
-; DEFAULT: '_Z3subii' inlined into '_Z3sumii'
-; DEFAULT-NOT: '_Z3subii' inlined into 'main'
-
-; REPLAY: '_Z3sumii' inlined into 'main'
-; REPLAY: '_Z3subii' inlined into 'main'
-; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'