[InlineAdvisor] New inliner advisor to replay inlining from optimization remarks

[wenlei]

Summary

This change added a new inline advisor that takes optimization remarks from previous inlining as input, and provides the decision as advice so current inlining can replay inline decisions of a different compilation. Dwarf inline stack with line and discriminator is used as anchor for call sites including call context. The change can be useful for Inliner tuning as it provides a channel to allow external input for tweaking inline decisions. Existing alternatives like alwaysinline attribute is per-function, not per-callsite. Per-callsite inline intrinsic can be another solution (not yet existing), but it's intrusive to implement and also does not differentiate call context.

A switch -sample-profile-inline-replay=<inline_remarks_file> is added to hook up the new inline advisor with SampleProfileLoader's inline decision for replay. Since SampleProfileLoader does top-down inlining, inline decision can be specialized for each call context, hence we should be able to replay inlining accurately. However with a bottom-up inliner like CGSCC inlining, the replay can be limited due to lack of specialization for different call context. Apart from that limitation, the new inline advisor can still be used by regular CGSCC inliner later if needed for tuning purpose.

[mtrofin]

llvm/include/llvm/Analysis/InlineAdvisor.h

184 ↗

(On Diff #277666)

Nit: could this be factored in its own .h file (and .cpp)?

189 ↗

(On Diff #277666)

nit: lower case first letter, also verb form (e.g. areReplayRemarksLoaded)

193 ↗

(On Diff #277666)

Nit: initialize HasReplayRemarks here, then the state of the object is deterministic.

llvm/lib/Analysis/InlineAdvisor.cpp

162 ↗

(On Diff #277666)

this assumes DIL->getLine() >= DIL->getScope()->getSubprogram()->getLine(). Perhaps an assert before, or Offset could be int, and check it's non-negative?

llvm/lib/Transforms/IPO/SampleProfile.cpp

330

I'm curious, is there a case when FAM would be not passed? (i.e. why not just require it?)

912

why not define Advice within the if below?

[wenlei]

llvm/lib/Analysis/InlineAdvisor.cpp

162 ↗

(On Diff #277666)

This is consistent with addLocationToRemarks where we construct the location string for remarks, so the output can be consumed without change. Negative line offset is possible, we've seen that in FDO profile, and they're encoded with unsigned int there too - I don't know why that's the case, but seems there's convention and functionality-wise it works as long as everything is consistent..

llvm/lib/Transforms/IPO/SampleProfile.cpp

330

There's another call site from old pass manager pipeline, where we don't have FAM.

[wenlei]

Address feedbacks.

[mtrofin]

llvm/lib/Analysis/InlineAdvisor.cpp

162 ↗

(On Diff #277666)

I don't think that makes it right, though. I'd suggest at least having a test here, then, i.e. bailing out if the this is greater than lhs. I'll look into the other API.

[wenlei]

llvm/lib/Analysis/InlineAdvisor.cpp

162 ↗

(On Diff #277666)

I'm not sure if we want to bail out on negative offsets. I added addLocationToRemarks couple weeks back. Unsigned int was used there so we can match line offset from remarks to the line offset from FDO profile for negative offsets, which is important from tooling perspective.

Ideally we shouldn't have negative offset, but practically, we see it. If we put an assertion there, it will fire. Due to macro, there could be cases where LHS and RHS come from different files, hence the subtraction can lead to negative offset; and there could be other cases due to not having a line number on LHS so 0 is retrieved for LHS. We've been living with that for a while (see FunctionSamples::getOffset), and here's an example of FDO profile where negative offset is encoded using unsigned int (the last three lines).

Not advocating for that, but just thought asserting non-negative offset or fixing what's breaking that assertion is a separate work. And being consistent with what we've been doing (e.g. FunctionSamples::getOffset) should be good. What do you think?

ZSTD_decompressSequences_bmi2:736988668:1677
 0: 1557
 6: 93
 5: ZSTD_decompressSequences_body:736954111
  8: 1557
  9: 1557
  10: 1557
  11: 1557
  53: 109
  55: 109
  58: 0
  64693: 1520
  64940: 1564
  65080: 1264073

[wenlei]

lint

[davidxl]

Can this be extended to the SCC inliner?

[wenlei]

In D83743#2150703, @davidxl wrote:

Can this be extended to the SCC inliner?

Yes, we can use it in SCC inliner as well. We just need extra plumbing there. We can do that in a separate change if needed.

[wmi]

In D83743#2150725, @wenlei wrote:

In D83743#2150703, @davidxl wrote:

Can this be extended to the SCC inliner?

Yes, we can use it in SCC inliner as well. We just need extra plumbing there. We can do that in a separate change if needed.

Thanks for the work, using inline advisor to replay inline is exactly something we want too.

Currently every inline remark message has only one level: one caller, one callee and one callsite (maybe with multiple levels of inline stack associated with the callsite). One problem I am thinking of is: the inline advisor may only work with top-down inlining. In bottom-up inlining, if we decide to inline a specific callsite, it will actually be inlined at many places after its caller also being inlined into multiple places in the caller's callers. Since SCC inliner uses bottom-up inlining, current format of inline advise may not provide precise enough information to specify an exact location where inlining is expected to happen.

Have you ever considered to use a callpath to specify the inlining location?

[mtrofin]

llvm/lib/Analysis/InlineAdvisor.cpp

162 ↗

(On Diff #277666)

So I make sure I understand:

• remarks already output unsigned ints
• negative offsets are possible, due to cases like you mentioned

In this case, I agree that the choice of supporting negative offsets is a separate problem from this CL.

Could you add a comment explaining the current motivation for unsigned it? (i.e. match what remarks do)

Also, could it be uint32_t, to ensure 32 bit-ness?

[mtrofin]

In D83743#2150725, @wenlei wrote:

In D83743#2150703, @davidxl wrote:

Can this be extended to the SCC inliner?

Yes, we can use it in SCC inliner as well. We just need extra plumbing there. We can do that in a separate change if needed.

I'd be curious what the scenario in the SCC case would be. IIUC, here, the value is that you can replay decisions made with a profile, and use a different profile for everything else in the compiler. Hmm... I suppose maybe a similar scenario could be articulated for the SCC case?

On this - could you please add in the patch description the motivating scenario (helps with understanding) - thanks!

[wenlei]

In D83743#2151020, @wmi wrote:

In D83743#2150725, @wenlei wrote:

In D83743#2150703, @davidxl wrote:

Can this be extended to the SCC inliner?

Yes, we can use it in SCC inliner as well. We just need extra plumbing there. We can do that in a separate change if needed.

Thanks for the work, using inline advisor to replay inline is exactly something we want too.

Currently every inline remark message has only one level: one caller, one callee and one callsite (maybe with multiple levels of inline stack associated with the callsite). One problem I am thinking of is: the inline advisor may only work with top-down inlining. In bottom-up inlining, if we decide to inline a specific callsite, it will actually be inlined at many places after its caller also being inlined into multiple places in the caller's callers. Since SCC inliner uses bottom-up inlining, current format of inline advise may not provide precise enough information to specify an exact location where inlining is expected to happen.

Have you ever considered to use a callpath to specify the inlining location?

Glad to know that it will be useful for your case too. You're right that with a bottom-up CGSCC inliner, we won't be able to replay arbitrary inline decisions. However I think that's a limitation of the inliner, not how inline decisions are represented. E.g. even if we have a full inline tree (subset of call graph), we still won't be able to replay that with CGSCC inliner if there's context-sensitive inline (specialization) in it.

The use case we had was for tuning full context-sensitive AutoFDO's early inlining - we can replay baseline AutoFDO early inlining there, or the other way around. We could also replay CGSCC inline of one build in another. The way it's implemented would work for those scenarios. But replay FDO early inline in CGSCC inline or the opposite may not get us what we wanted for the reasons you mentioned.

[wenlei]

In D83743#2151983, @mtrofin wrote:

In D83743#2150725, @wenlei wrote:

In D83743#2150703, @davidxl wrote:

Can this be extended to the SCC inliner?

Yes, we can use it in SCC inliner as well. We just need extra plumbing there. We can do that in a separate change if needed.

I'd be curious what the scenario in the SCC case would be. IIUC, here, the value is that you can replay decisions made with a profile, and use a different profile for everything else in the compiler. Hmm... I suppose maybe a similar scenario could be articulated for the SCC case?

On this - could you please add in the patch description the motivating scenario (helps with understanding) - thanks!

Our current use case is not in the SCC inliner. But yeah, I image it could be useful there too. In general I feel some mechanism like this one to allow external input for tweaking inlining decision can be useful, mostly for tuning and experimental purpose. We currently don't have a good way of doing that, existing attributes like alwaysinline is function level, not call site level. And per-callsite inline intrinsic like https://reviews.llvm.org/D51200 is intrusive and hard to push through either. I thought this is a relatively easy and clean way of getting that functionality for tuning. I will update the description to include the motivation.

[wenlei]

Address feedback, update comment.

[wmi]

Thanks for the explanation.

We could also replay CGSCC inline of one build in another. The way it's implemented would work for those scenarios.

When you replay CGSCC, the inline decision in replay file should have order? If it is unordered, I cannot see how it can replay CGSCC inline from another build exactly.

Our current use case is not in the SCC inliner. But yeah, I image it could be useful there too. In general I feel some mechanism like this one to allow external input for tweaking inlining decision can be useful, mostly for tuning and experimental purpose. We currently don't have a good way of doing that, existing attributes like alwaysinline is function level, not call site level. And per-callsite inline intrinsic like https://reviews.llvm.org/D51200 is intrusive and hard to push through either. I thought this is a relatively easy and clean way of getting that functionality for tuning.

That is the scenario I am interested -- using it for tweaking inlining decision for tuning/experimental purpose. I understand it could need a lot more work than this patch and we may not need to address them currently. I just want to know how it should work in your mind. Could you explain it in more detail?

[wenlei]

In D83743#2153532, @wmi wrote:

Thanks for the explanation.

We could also replay CGSCC inline of one build in another. The way it's implemented would work for those scenarios.

When you replay CGSCC, the inline decision in replay file should have order? If it is unordered, I cannot see how it can replay CGSCC inline from another build exactly.

I might be missing something, but here's how I think about this. For a given inline decision, say D on A->B->C path, we would only evaluate once and make that decision once. Then for replay, assuming we follow the same bottom-up order, for each call site, we can retrieve that exact decision from input remarks, and as long as we make the same decisions, the sequence of call sites that gets exposed for evaluation will also be the same. Because we evaluate call site for each path only once, any decision recorded in remarks should be unique, then I'm not sure how order of the inline remarks interfere with replay.

Our current use case is not in the SCC inliner. But yeah, I image it could be useful there too. In general I feel some mechanism like this one to allow external input for tweaking inlining decision can be useful, mostly for tuning and experimental purpose. We currently don't have a good way of doing that, existing attributes like alwaysinline is function level, not call site level. And per-callsite inline intrinsic like https://reviews.llvm.org/D51200 is intrusive and hard to push through either. I thought this is a relatively easy and clean way of getting that functionality for tuning.

That is the scenario I am interested -- using it for tweaking inlining decision for tuning/experimental purpose. I understand it could need a lot more work than this patch and we may not need to address them currently. I just want to know how it should work in your mind. Could you explain it in more detail?

I am considering different modes for replay in the future, the first is strict mode like what we have here - we simply replay everything faithfully. The second mode is positive replay, e.g. we only apply extra inlining using the input. A third mode can be negative replay, that is we prohibit inlining specified in input without affecting others. Orthogonally, we sometime may also want to focus on a specific inline tree, so it may be useful to have an unknown decision from advisor, then we can fall back to other advisors for decisions outside of the inline tree specified in input.

[wenlei]

rebase

[wmi]

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

60–80

Can we extract this part to a function? I think it can be reused by other types of InlineAdvisor, for tuning purpose for example.

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

3

Why it is sum:1 instead of _Z3sumii:1 @ main:3.1?

[wenlei]

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

60–80

Sure, done.

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

3

Oh, this has to do with the contrived !dbg metadata where linkageName was missing. Changed both the remarks here and !dbg to include mangle names now.

[wenlei]

address feedbacks.

[wenlei]

llvm/lib/Analysis/InlineAdvisor.cpp

162 ↗

(On Diff #277666)

Comment added, and also changed to use uint32_t.

[wmi]

LGTM. Better wait another day to see if other reviewers have further comments.

[mtrofin]

lgtm - a small nit, could you also add in the description the current scope (the top down, SampleProfile case) - for clarity. Thanks!

[wenlei]

In D83743#2156697, @mtrofin wrote:

lgtm - a small nit, could you also add in the description the current scope (the top down, SampleProfile case) - for clarity. Thanks!

Sure, updated. Thank you all for feedbacks and quick review!

[stellaraccident]

The inline-replay.ll test that this added appears to be broken (as was noted by the pre-merge check https://reviews.llvm.org/harbormaster/unit/view/120108/) and is failing in our build.

[wenlei]

In D83743#2159588, @stellaraccident wrote:

The inline-replay.ll test that this added appears to be broken (as was noted by the pre-merge check https://reviews.llvm.org/harbormaster/unit/view/120108/) and is failing in our build.

Sorry for the breakage, this wasn't caught in my local build. Taking a look.. Feel free to revert to get unblock.

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

+//===- ReplayInlineAdvisor.h - Replay Inline Advisor interface -*- C++ --*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+#ifndef LLVM_REPLAYINLINEADVISOR_H_
+#define LLVM_REPLAYINLINEADVISOR_H_
+
+#include "llvm/ADT/StringSet.h"
+#include "llvm/Analysis/InlineAdvisor.h"
+#include "llvm/IR/LLVMContext.h"
+
+namespace llvm {
+class BasicBlock;
+class CallBase;
+class Function;
+class Module;
+class OptimizationRemarkEmitter;
+
+/// Replay inline advisor that uses optimization remarks from inlining of
+/// previous build to guide current inlining. This is useful for inliner tuning.
+class ReplayInlineAdvisor : public InlineAdvisor {
+public:
+  ReplayInlineAdvisor(FunctionAnalysisManager &FAM, LLVMContext &Context,
+                      StringRef RemarksFile);
+  std::unique_ptr<InlineAdvice> getAdvice(CallBase &CB) override;
+  bool areReplayRemarksLoaded() const { return HasReplayRemarks; }
+
+private:
+  StringSet<> InlineSitesFromRemarks;
+  bool HasReplayRemarks = false;
+};
+} // namespace llvm
+#endif // LLVM_REPLAYINLINEADVISOR_H_
 No newline at end of file

llvm/lib/Analysis/CMakeLists.txt

add_llvm_component_library(LLVMAnalysis
   PHITransAddr.cpp
   PhiValues.cpp
   PostDominators.cpp
   ProfileSummaryInfo.cpp
   PtrUseVisitor.cpp
   RegionInfo.cpp
   RegionPass.cpp
   RegionPrinter.cpp
+  ReplayInlineAdvisor.cpp
   ScalarEvolution.cpp
   ScalarEvolutionAliasAnalysis.cpp
   ScalarEvolutionDivision.cpp
   ScalarEvolutionNormalization.cpp
   StackLifetime.cpp
   StackSafetyAnalysis.cpp
   SyncDependenceAnalysis.cpp
   SyntheticCountsUtils.cpp

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

+//===- ReplayInlineAdvisor.cpp - Replay InlineAdvisor ---------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements ReplayInlineAdvisor that replays inline decision based
+// on previous inline remarks from optimization remark log.
+//
+//===----------------------------------------------------------------------===//
+#include <sstream>
+
+#include "llvm/Analysis/InlineAdvisor.h"
+#include "llvm/Analysis/ReplayInlineAdvisor.h"
+#include "llvm/IR/DebugInfoMetadata.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/Support/LineIterator.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "inline-replay"
+
+ReplayInlineAdvisor::ReplayInlineAdvisor(FunctionAnalysisManager &FAM,
+                                         LLVMContext &Context,
+                                         StringRef RemarksFile)
+    : InlineAdvisor(FAM), HasReplayRemarks(false) {
+  auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(RemarksFile);
+  std::error_code EC = BufferOrErr.getError();
+  if (EC) {
+    Context.emitError("Could not open remarks file: " + EC.message());
+    return;
+  }
+
+  line_iterator LineIt(*BufferOrErr.get(), /*SkipBlanks=*/true);
+  for (; !LineIt.is_at_eof(); ++LineIt) {
+    StringRef Line = *LineIt;
+    auto Pair = Line.split(" at callsite ");
+    if (Pair.second.empty())
+      continue;
+    InlineSitesFromRemarks.insert(Pair.second);
+  }
+  HasReplayRemarks = true;
+}
+
+std::unique_ptr<InlineAdvice> ReplayInlineAdvisor::getAdvice(CallBase &CB) {
+  assert(HasReplayRemarks);
+
+  Function &Caller = *CB.getCaller();
+  auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(Caller);
+
+  if (InlineSitesFromRemarks.empty())
+    return std::make_unique<InlineAdvice>(this, CB, ORE, false);
+
+  // Example for inline remarks to parse:
+  //   _Z3subii inlined into main [details] at callsite sum:1 @ main:3.1
+  // We use the callsite string after `at callsite` to replay inlining.
+  std::ostringstream CallSiteLoc;
+  auto DLoc = CB.getDebugLoc();
+  bool First = true;
+  for (DILocation *DIL = DLoc.get(); DIL; DIL = DIL->getInlinedAt()) {
+    if (!First)
+      CallSiteLoc << " @ ";
+    unsigned int Offset =
+        DIL->getLine() - DIL->getScope()->getSubprogram()->getLine();
+    unsigned int Discriminator = DIL->getBaseDiscriminator();
+    StringRef Name = DIL->getScope()->getSubprogram()->getLinkageName();
+    if (Name.empty())
+      Name = DIL->getScope()->getSubprogram()->getName();
+    CallSiteLoc << Name.str() << ":" << llvm::utostr(Offset);
+    if (Discriminator) {
+      CallSiteLoc << "." << llvm::utostr(Discriminator);
+    }
+    First = false;
+  }
+
+  bool InlineRecommended = InlineSitesFromRemarks.count(CallSiteLoc.str()) > 0;
+  return std::make_unique<InlineAdvice>(this, CB, ORE, InlineRecommended);
+}

llvm/lib/Transforms/IPO/SampleProfile.cpp

 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/PostDominators.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Dominators.h"
static cl::opt<bool> ProfileSizeInline(
     "sample-profile-inline-size", cl::Hidden, cl::init(false),
     cl::desc("Inline cold call sites in profile loader if it's beneficial "
              "for code size."));
 
 static cl::opt<int> SampleColdCallSiteThreshold(
     "sample-profile-cold-inline-threshold", cl::Hidden, cl::init(45),
     cl::desc("Threshold for inlining cold callsites"));
 
+static cl::opt<std::string> ProfileInlineReplayFile(
+    "sample-profile-inline-replay", cl::init(""), cl::value_desc("filename"),
+    cl::desc(
+        "Optimization remarks file containing inline remarks to be replayed "
+        "by inlining from sample profile loader."),
+    cl::Hidden);
+
 namespace {
 
 using BlockWeightMap = DenseMap<const BasicBlock *, uint64_t>;
 using EquivalenceClassMap = DenseMap<const BasicBlock *, const BasicBlock *>;
 using Edge = std::pair<const BasicBlock *, const BasicBlock *>;
 using EdgeWeightMap = DenseMap<Edge, uint64_t>;
 using BlockEdgeMap =
     DenseMap<const BasicBlock *, SmallVector<const BasicBlock *, 8>>;
SampleProfileLoader(
       std::function<TargetTransformInfo &(Function &)> GetTargetTransformInfo,
       std::function<const TargetLibraryInfo &(Function &)> GetTLI)
       : GetAC(std::move(GetAssumptionCache)),
         GetTTI(std::move(GetTargetTransformInfo)), GetTLI(std::move(GetTLI)),
         CoverageTracker(*this), Filename(std::string(Name)),
         RemappingFilename(std::string(RemapName)),
         IsThinLTOPreLink(IsThinLTOPreLink) {}
 
-  bool doInitialization(Module &M);
+  bool doInitialization(Module &M, FunctionAnalysisManager *FAM = nullptr);
   bool runOnModule(Module &M, ModuleAnalysisManager *AM,
                    ProfileSummaryInfo *_PSI, CallGraph *CG);
 
   void dump() { Reader->dump(); }
 
 protected:
   friend class SampleCoverageTracker;
 
protected:
   StringSet<> NamesInProfile;
 
   // For symbol in profile symbol list, whether to regard their profiles
   // to be accurate. It is mainly decided by existance of profile symbol
   // list and -profile-accurate-for-symsinlist flag, but it can be
   // overriden by -profile-sample-accurate or profile-sample-accurate
   // attribute.
   bool ProfAccForSymsInList;
+
+  // External inline advisor used to replay inline decision from remarks.
+  std::unique_ptr<ReplayInlineAdvisor> ExternalInlineAdvisor;
 };
 
 class SampleProfileLoaderLegacyPass : public ModulePass {
 public:
   // Class identification, replacement for typeinfo
   static char ID;
 
   SampleProfileLoaderLegacyPass(StringRef Name = SampleProfileFile,
SampleProfileLoader::findFunctionSamples(const Instruction &Inst) const {
 
   auto it = DILocation2SampleMap.try_emplace(DIL,nullptr);
   if (it.second)
     it.first->second = Samples->findFunctionSamples(DIL);
   return it.first->second;
 }
 
 bool SampleProfileLoader::inlineCallInstruction(CallBase &CB) {
+  if (ExternalInlineAdvisor) {
+    auto Advice = ExternalInlineAdvisor->getAdvice(CB);
+    if (!Advice->isInliningRecommended()) {
+      Advice->recordUnattemptedInlining();
+      return false;
+    }
+    // Dummy record, we don't use it for replay.
+    Advice->recordInlining();
+  }
+
   Function *CalledFunction = CB.getCalledFunction();
   assert(CalledFunction);
   DebugLoc DLoc = CB.getDebugLoc();
   BasicBlock *BB = CB.getParent();
   InlineParams Params = getInlineParams();
   Params.ComputeFullInlineCost = true;
   // Checks if there is anything in the reachable portion of the callee at
   // this callsite that makes this inlining potentially illegal. Need to
for (auto &BB : F) {
               localNotInlinedCallSites.try_emplace(CB, FS);
             if (callsiteIsHot(FS, PSI))
               Hot = true;
             else if (shouldInlineColdCallee(*CB))
               ColdCandidates.push_back(CB);
           }
         }
       }
-      if (Hot) {
+      if (Hot || ExternalInlineAdvisor) {
         CIS.insert(CIS.begin(), AllCandidates.begin(), AllCandidates.end());
         emitOptimizationRemarksForInlineCandidates(AllCandidates, F, true);
       } else {
         CIS.insert(CIS.begin(), ColdCandidates.begin(), ColdCandidates.end());
         emitOptimizationRemarksForInlineCandidates(ColdCandidates, F, false);
       }
     }
     for (CallBase *I : CIS) {
while (!CGI.isAtEnd()) {
     }
     ++CGI;
   }
 
   std::reverse(FunctionOrderList.begin(), FunctionOrderList.end());
   return FunctionOrderList;
 }
 
-bool SampleProfileLoader::doInitialization(Module &M) {
+bool SampleProfileLoader::doInitialization(Module &M,
+                                           FunctionAnalysisManager *FAM) {
   auto &Ctx = M.getContext();
 
   std::unique_ptr<SampleProfileReaderItaniumRemapper> RemapReader;
   auto ReaderOrErr =
       SampleProfileReader::create(Filename, Ctx, RemappingFilename);
   if (std::error_code EC = ReaderOrErr.getError()) {
     std::string Msg = "Could not open profile: " + EC.message();
     Ctx.diagnose(DiagnosticInfoSampleProfile(Filename, Msg));
     return false;
   }
   Reader = std::move(ReaderOrErr.get());
   Reader->collectFuncsFrom(M);
   ProfileIsValid = (Reader->read() == sampleprof_error::success);
   PSL = Reader->getProfileSymbolList();
 
   // While profile-sample-accurate is on, ignore symbol list.
   ProfAccForSymsInList =
       ProfileAccurateForSymsInList && PSL && !ProfileSampleAccurate;
   if (ProfAccForSymsInList) {
     NamesInProfile.clear();
     if (auto NameTable = Reader->getNameTable())
       NamesInProfile.insert(NameTable->begin(), NameTable->end());
   }
 
+  if (FAM && !ProfileInlineReplayFile.empty()) {
+    ExternalInlineAdvisor = std::make_unique<ReplayInlineAdvisor>(
+        *FAM, Ctx, ProfileInlineReplayFile);
+    if (!ExternalInlineAdvisor->areReplayRemarksLoaded())
+      ExternalInlineAdvisor.reset();
+  }
+
   return true;
 }
 
 ModulePass *llvm::createSampleProfileLoaderPass() {
   return new SampleProfileLoaderLegacyPass();
 }
 
 ModulePass *llvm::createSampleProfileLoaderPass(StringRef Name) {
PreservedAnalyses SampleProfileLoaderPass::run(Module &M,
   };
 
   SampleProfileLoader SampleLoader(
       ProfileFileName.empty() ? SampleProfileFile : ProfileFileName,
       ProfileRemappingFileName.empty() ? SampleProfileRemappingFile
                                        : ProfileRemappingFileName,
       IsThinLTOPreLink, GetAssumptionCache, GetTTI, GetTLI);
 
-  if (!SampleLoader.doInitialization(M))
+  if (!SampleLoader.doInitialization(M, &FAM))
     return PreservedAnalyses::all();
 
   ProfileSummaryInfo *PSI = &AM.getResult<ProfileSummaryAnalysis>(M);
   CallGraph &CG = AM.getResult<CallGraphAnalysis>(M);
   if (!SampleLoader.runOnModule(M, &AM, PSI, &CG))
     return PreservedAnalyses::all();
 
   return PreservedAnalyses::none();
 }

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

+remark: calls.cc:10:0: _Z3sumii inlined into main to match profiling context with (cost=45, threshold=337) at callsite main:3.1
+remark: calls.cc:4:0: _Z3subii inlined into main to match profiling context with (cost=-5, threshold=337) at callsite sum:1 @ main:3.1

llvm/test/Transforms/SampleProfile/inline-replay.ll

+;; Note that this needs new pass manager for now. Passing `-sample-profile-inline-replay` to legacy pass manager is a no-op.
+
+;; Check baseline inline decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
+
+;; Check replay inline decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+
+@.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
+
+define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
+entry:
+  %x.addr = alloca i32, align 4
+  %y.addr = alloca i32, align 4
+  store i32 %x, i32* %x.addr, align 4
+  store i32 %y, i32* %y.addr, align 4
+  %tmp = load i32, i32* %x.addr, align 4, !dbg !8
+  %tmp1 = load i32, i32* %y.addr, align 4, !dbg !8
+  %add = add nsw i32 %tmp, %tmp1, !dbg !8
+  %tmp2 = load i32, i32* %x.addr, align 4, !dbg !8
+  %tmp3 = load i32, i32* %y.addr, align 4, !dbg !8
+  %call = call i32 @_Z3subii(i32 %tmp2, i32 %tmp3), !dbg !8
+  ret i32 %add, !dbg !8
+}
+
+define i32 @_Z3subii(i32 %x, i32 %y) #0 !dbg !9 {
+entry:
+  %x.addr = alloca i32, align 4
+  %y.addr = alloca i32, align 4
+  store i32 %x, i32* %x.addr, align 4
+  store i32 %y, i32* %y.addr, align 4
+  %tmp = load i32, i32* %x.addr, align 4, !dbg !10
+  %tmp1 = load i32, i32* %y.addr, align 4, !dbg !10
+  %add = sub nsw i32 %tmp, %tmp1, !dbg !10
+  ret i32 %add, !dbg !11
+}
+
+define i32 @main() #0 !dbg !12 {
+entry:
+  %retval = alloca i32, align 4
+  %s = alloca i32, align 4
+  %i = alloca i32, align 4
+  store i32 0, i32* %retval
+  store i32 0, i32* %i, align 4, !dbg !13
+  br label %while.cond, !dbg !14
+
+while.cond:                                       ; preds = %if.end, %entry
+  %tmp = load i32, i32* %i, align 4, !dbg !15
+  %inc = add nsw i32 %tmp, 1, !dbg !15
+  store i32 %inc, i32* %i, align 4, !dbg !15
+  %cmp = icmp slt i32 %tmp, 400000000, !dbg !15
+  br i1 %cmp, label %while.body, label %while.end, !dbg !15
+
+while.body:                                       ; preds = %while.cond
+  %tmp1 = load i32, i32* %i, align 4, !dbg !17
+  %cmp1 = icmp ne i32 %tmp1, 100, !dbg !17
+  br i1 %cmp1, label %if.then, label %if.else, !dbg !17
+
+if.then:                                          ; preds = %while.body
+  %tmp2 = load i32, i32* %i, align 4, !dbg !19
+  %tmp3 = load i32, i32* %s, align 4, !dbg !19
+  %call = call i32 @_Z3sumii(i32 %tmp2, i32 %tmp3), !dbg !19
+  store i32 %call, i32* %s, align 4, !dbg !19
+  br label %if.end, !dbg !19
+
+if.else:                                          ; preds = %while.body
+  store i32 30, i32* %s, align 4, !dbg !21
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %if.then
+  br label %while.cond, !dbg !23
+
+while.end:                                        ; preds = %while.cond
+  %tmp4 = load i32, i32* %s, align 4, !dbg !25
+  %call2 = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([11 x i8], [11 x i8]* @.str, i32 0, i32 0), i32 %tmp4), !dbg !25
+  ret i32 0, !dbg !26
+}
+
+declare i32 @printf(i8*, ...)
+
+attributes #0 = { "use-sample-profile" }
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+!llvm.ident = !{!5}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 3.5 ", isOptimized: false, runtimeVersion: 0, emissionKind: NoDebug, enums: !2, retainedTypes: !2, globals: !2, imports: !2)
+!1 = !DIFile(filename: "calls.cc", directory: ".")
+!2 = !{}
+!3 = !{i32 2, !"Dwarf Version", i32 4}
+!4 = !{i32 1, !"Debug Info Version", i32 3}
+!5 = !{!"clang version 3.5 "}
+!6 = distinct !DISubprogram(name: "sum", scope: !1, file: !1, line: 3, type: !7, scopeLine: 3, virtualIndex: 6, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
+!7 = !DISubroutineType(types: !2)
+!8 = !DILocation(line: 4, scope: !6)
+!9 = distinct !DISubprogram(name: "sub", scope: !1, file: !1, line: 20, type: !7, scopeLine: 20, virtualIndex: 6, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
+!10 = !DILocation(line: 20, scope: !9)
+!11 = !DILocation(line: 21, scope: !9)
+!12 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 7, type: !7, scopeLine: 7, virtualIndex: 6, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !2)
+!13 = !DILocation(line: 8, scope: !12)
+!14 = !DILocation(line: 9, scope: !12)
+!15 = !DILocation(line: 9, scope: !16)
+!16 = !DILexicalBlockFile(scope: !12, file: !1, discriminator: 2)
+!17 = !DILocation(line: 10, scope: !18)
+!18 = distinct !DILexicalBlock(scope: !12, file: !1, line: 10)
+!19 = !DILocation(line: 10, scope: !20)
+!20 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 2)
+!21 = !DILocation(line: 10, scope: !22)
+!22 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 4)
+!23 = !DILocation(line: 10, scope: !24)
+!24 = !DILexicalBlockFile(scope: !18, file: !1, discriminator: 6)
+!25 = !DILocation(line: 11, scope: !12)
+!26 = !DILocation(line: 12, scope: !12)
+
+
+; DEFAULT: _Z3sumii inlined into main
+; DEFAULT: _Z3subii inlined into _Z3sumii
+; DEFAULT-NOT: _Z3subii inlined into main 
+
+; REPLAY: _Z3sumii inlined into main
+; REPLAY: _Z3subii inlined into main 
+; REPLA-NOT: _Z3subii inlined into _Z3sumii