[InlineAdvisor] Add fallback/format switches and negative remark processing to Replay Inliner

[modimo]

Summary

Adds the following switches:

1. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback: controls what the replay advisor does for inline sites that are not present in the replay. Options are:
   1. Original: defers to original advisor
   2. AlwaysInline: inline all sites not in replay
   3. NeverInline: inline no sites not in replay

2. --sample-profile-inline-replay-format/--cgscc-inline-replay-format: controls what format should be generated to match against the replay remarks. Options are:
   1. Line
   2. LineColumn
   3. LineDiscriminator
   4. LineColumnDiscriminator

Adds support for negative inlining decisions. These are denoted by "will not be inlined into" as compared to the positive "inlined into" in the remarks.

All of these together with the previous --sample-profile-inline-replay-scope/--cgscc-inline-replay-scope allow tweaking in how to apply replay. In my testing, I'm using:

1. --sample-profile-inline-replay-scope/--cgscc-inline-replay-scope = Function to only replay on a function
2. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback = NeverInline since I'm feeding in only positive remarks to the replay system
3. --sample-profile-inline-replay-format/--cgscc-inline-replay-format = Line since I'm generating the remarks from DWARF information from GCC which can conflict quite heavily in column number compared to Clang

An alternative configuration could be to do Function, AlwaysInline, Line fallback with negative remarks which closer matches the final call-sites. Note that this can lead to unbounded inlining if a negative remark doesn't match/exist for one reason or another.

Updated various tests to cover the new switches and negative remarks

Testing:
ninja check-all

[wenlei]

llvm/include/llvm/Analysis/InlineAdvisor.h

49

nit: OutputColumn -> outputColumn, same for OutputDiscriminator

65

I think this now reflects a narrower aspect of the settings than mode. Specifically, this is some kind of fall back inline strategy. I suggest we rename this field as well as the flag names because "AlwaysInline" reply mode doesn't really convey the right message, and can be confusing.

We can rename Mode -> Fallback, and xxx-inline-replay-mode -> xxx-inline-replay-fallback.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

99–137

always inline, never inline and original advisor are parallel, I think we can restructure the logic and add some comments to help readability:

// We have a decision from replay
if (hasInlineAdvice(..)) {

  return ...
}

// We need to fall back ..
if (always inline)

else if (never inline)

else {
  assert(original);
  
}

return ..

100–101

Use hasInlineAdvice (hasRemarksForFunction)?

109–110

Should we also return a negative decision (never inline) if InlineSitesFromRemarks[Combined] is false?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1896–1902

Hmm.. now I'm a bit confused why the call to getReplayInlineAdvisor is removed in the other diff.

llvm/test/Transforms/SampleProfile/inline-replay.ll

42

AlwayLineColumnsInline -> LineColumn

74–77

These are probably not very necessary as they're testing cl::opt/cl::value/clEnumValN, though there's no harm in having them. On the other hand, it might be good have a case to cover LineColumn
or LineDiscriminator.

[mtrofin]

llvm/include/llvm/Analysis/InlineAdvisor.h

41

I think we shold start moving the replay stuff in its header (with possibly some refactoring of how we get to obtain an advisor)

[modimo]

Review feedback

llvm/include/llvm/Analysis/InlineAdvisor.h

41

Moved ReplayInlinerSettings to ReplayInlineAdvisor.h and forward declared it like OptimizationRemarkEmitter in InlineAdvisor.h

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

109–110

Good catch, yes we should

llvm/lib/Transforms/IPO/SampleProfile.cpp

1896–1902

TL;DR getReplayInlineAdvisor is the correct way to do this.

That came about due to how we query if the replay inliner has remarks. Originally (including v1 here) I used hasRemarksForFunction which doesn't exist in the base InlineAdvisor class so the advisor needed to specifically be a ReplayInlineAdvisor. After I made the changes in D112033 to just rely on a decision being returned this was no longer necessary.

llvm/test/Transforms/SampleProfile/inline-replay.ll

74–77

Added them above

[wenlei]

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

30

We can rename Mode -> Fallback, and xxx-inline-replay-mode -> xxx-inline-replay-fallback.

I meant to say we should rename all "Mode" to be Fallback, beyond the switch.

41

I thought I commented somewhere - hasInlineAdvice is a better name. When scope is module, we may not have remark for this function, and yet it can return true.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

99–137

Actually what I suggested earlier was wrong based on current implementation of hasRemarksForFunction/hasInlineAdvice. That was assuming hasRemarksForFunction only returns true for explicit replay decision, and false for fall back cases.

Perhaps something like this would be a clean structure:

// This inline decision is out of scope for replay advisor
if (!hasInlineAdvice(..)) {
  return {}
}

// Use explicit replay decision if available
Optional<bool> ShouldInline = getAdviceFromRemarks();
if (ShouldInline.hasValue()) {
  if (ShouldInline.getValue()) {
    ..
  }
  else {
   ..
  }
}

// We need to use fall back decisions ..
if (fall back is always inline)

else if (fall back is never inline)

else 
   assert(fall back is original)

return ..

getAdviceFromRemarks looks up InlineSitesFromRemarks and returns three-way value.

140–143

Why is this a Nona instead of llvm::InlineCost::getNever(..)? Same for the other instance above.

llvm/lib/Transforms/IPO/Inliner.cpp

878–879

Why we do we skip recordUnattemptedInlining when not not inlining due to not having an advice?

[wenlei]

Change summary needs to be updated as well.

[modimo]

ReplayMode->ReplayFallback, refactoring

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

30

I missed that one, thanks.

41

You commented that in the other change (D112033). Still makes sense, changed.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

99–137

With the nesting of the CGSCC inliner inside this, no advice still defers to the original advisor if it exists. That above is cleaner though, changed.

140–143

DefaultInlineAdvice(InlineAdvisor *Advisor, CallBase &CB,
                    Optional<InlineCost> OIC, OptimizationRemarkEmitter &ORE,
                    bool EmitRemarks = true)
    : InlineAdvice(Advisor, CB, ORE, /*IsInliningRecommended=*/OIC.hasValue())

If the Optional cost has a value, IsInliningRecommended gets set to true. This looks intended. I'll add comments to that effect.

llvm/lib/Transforms/IPO/Inliner.cpp

878–879

Advice is a std::unique_ptr<InlineAdvice> so being negative means it doesn't contain a value. Calling recordUnattemptedInlining on that leads to an AV. This path is untested ATM because the cgscc advisor is always nested inside the replay one so there will always be an InlineAdvice returned. I came upon this when one of my intermediate edits didn't forward the decision to the original advisor.

I made the auto explicit to make this clearer.

[wenlei]

lgtm, thanks.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

99–137

Yes, it was functionally correct, but logically wrong..

[modimo]

Fix up comment

[mtrofin]

llvm/include/llvm/Analysis/InlineAdvisor.h

266

should ReplayInlinerSettings be passed as const &?

289–293

can this declaration also move to ReplayInlineAdvisor.h?

320

can formatting (so callsiteformat and getCallSiteLocation) be moved to ReplayInlineAdvisor.h?

also: s/getCallSiteLocation/formatCallSiteLocation.

llvm/lib/Transforms/IPO/Inliner.cpp

875

unrelated change?

[modimo]

Move more things to ReplayInlineAdvisor.h and other feedback

llvm/include/llvm/Analysis/InlineAdvisor.h

320

Yes, this was originally added for replay and only that consumes this atm.

llvm/lib/Transforms/IPO/Inliner.cpp

875

The comment below is that !Advice isn't very clear with auto and also tripped me up: the previous change
if (!Advice || !Advice->isInliningRecommended()) {

treated !Advice as having "negative" advice rather than being empty. Clearing this up seems valuable.

[mtrofin]

lgtm

llvm/lib/Transforms/IPO/Inliner.cpp

875

ah, good point, thanks!

[modimo]

Rebase and add -disable-output consistently to inline-replay.ll

[modimo]

Thanks for the reviews @mtrofin !

llvm/include/llvm/Analysis/InlineAdvisor.h

 ///
 /// - Development mode, for training new models.
 /// In this mode, we trade off runtime performance for flexibility. This mode
 /// requires the full C Tensorflow API library, and evaluates models
 /// dynamically. This mode also permits generating training logs, for offline
 /// training.
 enum class InliningAdvisorMode : int { Default, Release, Development };
 
-/// For Replay Inliner initialization
-enum class ReplayInlineScope : int { Function, Module };
+struct CallSiteFormat {
+  enum class Format : int {
+    Line,
+    LineColumn,
+    LineDiscriminator,
+    LineColumnDiscriminator
+  };
+
+  bool OutputColumn() const {
+    return OutputFormat == Format::LineColumn ||
+           OutputFormat == Format::LineColumnDiscriminator;
+  }
+
+  bool OutputDiscriminator() const {
+    return OutputFormat == Format::LineDiscriminator ||
+           OutputFormat == Format::LineColumnDiscriminator;
+  }
+
+  Format OutputFormat;
+};
+
+/// Replay Inliner Setup
+struct ReplayInlinerSettings {
+  enum class Scope : int { Function, Module };
+  enum class Mode : int { Original, AlwaysInline, NeverInline };
+
+  StringRef ReplayFile;
+  Scope ReplayScope;
+  Mode ReplayMode;
+  CallSiteFormat ReplayFormat;
+};
 
 class InlineAdvisor;
 /// Capture state between an inlining decision having had been made, and
 /// its impact being observable. When collecting model training data, this
 /// allows recording features/decisions/partial reward data sets.
 ///
 /// Derivations of this type are expected to be tightly coupled with their
 /// InliningAdvisors. The base type implements the minimal contractual
struct Result {
     bool invalidate(Module &, const PreservedAnalyses &PA,
                     ModuleAnalysisManager::Invalidator &) {
       // Check whether the analysis has been explicitly invalidated. Otherwise,
       // it's stateless and remains preserved.
       auto PAC = PA.getChecker<InlineAdvisorAnalysis>();
       return !PAC.preservedWhenStateless();
     }
     bool tryCreate(InlineParams Params, InliningAdvisorMode Mode,
-                   StringRef ReplayFile, ReplayInlineScope ReplayScope);
+                   ReplayInlinerSettings ReplaySettings);
     InlineAdvisor *getAdvisor() const { return Advisor.get(); }
 
   private:
     Module &M;
     ModuleAnalysisManager &MAM;
     std::unique_ptr<InlineAdvisor> Advisor;
   };
 
   Result run(Module &M, ModuleAnalysisManager &MAM) { return Result(M, MAM); }
 };
 
 #ifdef LLVM_HAVE_TF_AOT
 std::unique_ptr<InlineAdvisor>
 getReleaseModeAdvisor(Module &M, ModuleAnalysisManager &MAM);
 #endif
 
 #ifdef LLVM_HAVE_TF_API
 std::unique_ptr<InlineAdvisor>
 getDevelopmentModeAdvisor(Module &M, ModuleAnalysisManager &MAM,
                           std::function<bool(CallBase &)> GetDefaultAdvice);
 #endif
 
-std::unique_ptr<InlineAdvisor> getReplayInlineAdvisor(
-    Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
-    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    ReplayInlineScope Scope, bool EmitRemarks);
+std::unique_ptr<InlineAdvisor>
+getReplayInlineAdvisor(Module &M, FunctionAnalysisManager &FAM,
+                       LLVMContext &Context,
+                       std::unique_ptr<InlineAdvisor> OriginalAdvisor,
+                       ReplayInlinerSettings ReplaySettings, bool EmitRemarks);
 
 // Default (manual policy) decision making helper APIs. Shared with the legacy
 // pass manager inliner.
 
 /// Return the cost only if the inliner should attempt to inline at the given
 /// CallSite. If we return the cost, we will emit an optimisation remark later
 /// using that cost, so we won't do so from this function. Return None if
 /// inlining should not be attempted.

 
 /// Emit ORE message based in cost (default heuristic).
 void emitInlinedIntoBasedOnCost(OptimizationRemarkEmitter &ORE, DebugLoc DLoc,
                                 const BasicBlock *Block, const Function &Callee,
                                 const Function &Caller, const InlineCost &IC,
                                 bool ForProfileContext = false,
                                 const char *PassName = nullptr);
 
 /// get call site location as string
-std::string getCallSiteLocation(DebugLoc DLoc);
+std::string getCallSiteLocation(DebugLoc DLoc, const CallSiteFormat &Format);
 
 /// Add location info to ORE message.
 void addLocationToRemarks(OptimizationRemark &Remark, DebugLoc DLoc);
 
 /// Set the inline-remark attribute.
 void setInlineRemark(CallBase &CB, StringRef Message);
 
 /// Utility for extracting the inline cost message to a string.
 std::string inlineCostStr(const InlineCost &IC);
 } // namespace llvm
 #endif // LLVM_ANALYSIS_INLINEADVISOR_H

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

 
 /// Replay inline advisor that uses optimization remarks from inlining of
 /// previous build to guide current inlining. This is useful for inliner tuning.
 class ReplayInlineAdvisor : public InlineAdvisor {
 public:
   ReplayInlineAdvisor(Module &M, FunctionAnalysisManager &FAM,
                       LLVMContext &Context,
                       std::unique_ptr<InlineAdvisor> OriginalAdvisor,
-                      StringRef RemarksFile, ReplayInlineScope Scope,
+                      const ReplayInlinerSettings ReplaySettings,
                       bool EmitRemarks);
   std::unique_ptr<InlineAdvice> getAdviceImpl(CallBase &CB) override;
   bool areReplayRemarksLoaded() const { return HasReplayRemarks; }
   bool hasRemarksForFunction(Function &F) const {
-    return (Scope == ReplayInlineScope::Module) ||
+    return (ReplaySettings.ReplayScope ==
+            ReplayInlinerSettings::Scope::Module) ||
            CallersToReplay.contains(F.getName());
   }
 
 private:
   std::unique_ptr<InlineAdvisor> OriginalAdvisor;
   bool HasReplayRemarks = false;
-  ReplayInlineScope Scope;
+  ReplayInlinerSettings ReplaySettings;
   bool EmitRemarks = false;
 
   StringMap<bool> InlineSitesFromRemarks;
   StringSet<> CallersToReplay;
 };
 } // namespace llvm
 #endif // LLVM_ANALYSIS_REPLAYINLINEADVISOR_H

llvm/lib/Analysis/InlineAdvisor.cpp

void InlineAdvice::recordInliningWithCalleeDeleted() {
   markRecorded();
   recordInlineStatsIfNeeded();
   Advisor->markFunctionAsDeleted(Callee);
   recordInliningWithCalleeDeletedImpl();
 }
 
 AnalysisKey InlineAdvisorAnalysis::Key;
 
-bool InlineAdvisorAnalysis::Result::tryCreate(InlineParams Params,
-                                              InliningAdvisorMode Mode,
-                                              StringRef ReplayFile,
-                                              ReplayInlineScope ReplayScope) {
+bool InlineAdvisorAnalysis::Result::tryCreate(
+    InlineParams Params, InliningAdvisorMode Mode,
+    ReplayInlinerSettings ReplaySettings) {
   auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   switch (Mode) {
   case InliningAdvisorMode::Default:
     LLVM_DEBUG(dbgs() << "Using default inliner heuristic.\n");
     Advisor.reset(new DefaultInlineAdvisor(M, FAM, Params));
     // Restrict replay to default advisor, ML advisors are stateful so
     // replay will need augmentations to interleave with them correctly.
-    if (!ReplayFile.empty()) {
-      Advisor = llvm::getReplayInlineAdvisor(
-          M, FAM, M.getContext(), std::move(Advisor), ReplayFile, ReplayScope,
+    if (!ReplaySettings.ReplayFile.empty()) {
+      Advisor = llvm::getReplayInlineAdvisor(M, FAM, M.getContext(),
+                                             std::move(Advisor), ReplaySettings,
                                              /* EmitRemarks =*/true);
     }
     break;
   case InliningAdvisorMode::Development:
 #ifdef LLVM_HAVE_TF_API
     LLVM_DEBUG(dbgs() << "Using development-mode inliner policy.\n");
     Advisor =
         llvm::getDevelopmentModeAdvisor(M, MAM, [&FAM, Params](CallBase &CB) {
           auto OIC = getDefaultInlineAdvice(CB, FAM, Params);
if (EnableDeferral &&
     return None;
   }
 
   LLVM_DEBUG(dbgs() << "    Inlining " << inlineCostStr(IC) << ", Call: " << CB
                     << '\n');
   return IC;
 }
 
-std::string llvm::getCallSiteLocation(DebugLoc DLoc) {
+std::string llvm::getCallSiteLocation(DebugLoc DLoc,
+                                      const CallSiteFormat &Format) {
   std::string Buffer;
   raw_string_ostream CallSiteLoc(Buffer);
   bool First = true;
   for (DILocation *DIL = DLoc.get(); DIL; DIL = DIL->getInlinedAt()) {
     if (!First)
       CallSiteLoc << " @ ";
     // Note that negative line offset is actually possible, but we use
     // unsigned int to match line offset representation in remarks so
     // it's directly consumable by relay advisor.
     uint32_t Offset =
         DIL->getLine() - DIL->getScope()->getSubprogram()->getLine();
     uint32_t Discriminator = DIL->getBaseDiscriminator();
     StringRef Name = DIL->getScope()->getSubprogram()->getLinkageName();
     if (Name.empty())
       Name = DIL->getScope()->getSubprogram()->getName();
-    CallSiteLoc << Name.str() << ":" << llvm::utostr(Offset) << ":"
-                << llvm::utostr(DIL->getColumn());
-    if (Discriminator)
+    CallSiteLoc << Name.str() << ":" << llvm::utostr(Offset);
+    if (Format.OutputColumn())
+      CallSiteLoc << ":" << llvm::utostr(DIL->getColumn());
+    if (Format.OutputDiscriminator() && Discriminator)
       CallSiteLoc << "." << llvm::utostr(Discriminator);
     First = false;
   }
 
   return CallSiteLoc.str();
 }
 
 void llvm::addLocationToRemarks(OptimizationRemark &Remark, DebugLoc DLoc) {

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

 #include <memory>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "replay-inline"
 
 ReplayInlineAdvisor::ReplayInlineAdvisor(
     Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
-    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    ReplayInlineScope Scope, bool EmitRemarks)
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor,
+    const ReplayInlinerSettings ReplaySettings, bool EmitRemarks)
     : InlineAdvisor(M, FAM), OriginalAdvisor(std::move(OriginalAdvisor)),
-      HasReplayRemarks(false), Scope(Scope), EmitRemarks(EmitRemarks) {
+      HasReplayRemarks(false), ReplaySettings(ReplaySettings),
+      EmitRemarks(EmitRemarks) {
 
-  auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(RemarksFile);
+  auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(ReplaySettings.ReplayFile);
   std::error_code EC = BufferOrErr.getError();
   if (EC) {
     Context.emitError("Could not open remarks file: " + EC.message());
     return;
   }
 
   // Example for inline remarks to parse:
   //   main:3:1.1: '_Z3subii' inlined into 'main' at callsite sum:1 @
   //   main:3:1.1;
   // We use the callsite string after `at callsite` to replay inlining.
   line_iterator LineIt(*BufferOrErr.get(), /*SkipBlanks=*/true);
+  const std::string PositiveRemark = "' inlined into '";
+  const std::string NegativeRemark = "' will not be inlined into '";
+
   for (; !LineIt.is_at_eof(); ++LineIt) {
     StringRef Line = *LineIt;
     auto Pair = Line.split(" at callsite ");
 
-    auto CalleeCaller = Pair.first.split("' inlined into '");
+    bool IsPositiveRemark = true;
+    if (Pair.first.contains(NegativeRemark))
+      IsPositiveRemark = false;
+
+    auto CalleeCaller =
+        Pair.first.split(IsPositiveRemark ? PositiveRemark : NegativeRemark);
 
     StringRef Callee = CalleeCaller.first.rsplit(": '").second;
     StringRef Caller = CalleeCaller.second.rsplit("'").first;
 
     auto CallSite = Pair.second.split(";").first;
 
     if (Callee.empty() || Caller.empty() || CallSite.empty()) {
       Context.emitError("Invalid remark format: " + Line);
       return;
     }
 
     std::string Combined = (Callee + CallSite).str();
-    InlineSitesFromRemarks[Combined] = false;
-    if (Scope == ReplayInlineScope::Function)
+    InlineSitesFromRemarks[Combined] = IsPositiveRemark;
+    if (ReplaySettings.ReplayScope == ReplayInlinerSettings::Scope::Function)
       CallersToReplay.insert(Caller);
   }
 
   HasReplayRemarks = true;
 }
 
 std::unique_ptr<InlineAdvisor> llvm::getReplayInlineAdvisor(
     Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
-    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    ReplayInlineScope Scope, bool EmitRemarks) {
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor,
+    ReplayInlinerSettings ReplaySettings, bool EmitRemarks) {
   auto Advisor = std::make_unique<ReplayInlineAdvisor>(
-      M, FAM, Context, std::move(OriginalAdvisor), RemarksFile, Scope,
-      EmitRemarks);
+      M, FAM, Context, std::move(OriginalAdvisor), ReplaySettings, EmitRemarks);
   if (!Advisor->areReplayRemarksLoaded())
     Advisor.reset();
   return Advisor;
 }
 
 std::unique_ptr<InlineAdvice> ReplayInlineAdvisor::getAdviceImpl(CallBase &CB) {
   assert(HasReplayRemarks);
 
   Function &Caller = *CB.getCaller();
   auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(Caller);
 
   Optional<InlineCost> InlineRecommended;
+  bool ReplayMadeDecision = false;
 
-  if (Scope == ReplayInlineScope::Module ||
+  // Decision could be made by replay system
+  if (ReplaySettings.ReplayScope == ReplayInlinerSettings::Scope::Module ||
       CallersToReplay.count(CB.getFunction()->getName())) {
-    std::string CallSiteLoc = getCallSiteLocation(CB.getDebugLoc());
+    std::string CallSiteLoc =
+        getCallSiteLocation(CB.getDebugLoc(), ReplaySettings.ReplayFormat);
     StringRef Callee = CB.getCalledFunction()->getName();
     std::string Combined = (Callee + CallSiteLoc).str();
 
     auto Iter = InlineSitesFromRemarks.find(Combined);
     if (Iter != InlineSitesFromRemarks.end()) {
-      InlineSitesFromRemarks[Combined] = true;
+      if (InlineSitesFromRemarks[Combined])
         InlineRecommended = llvm::InlineCost::getAlways("previously inlined");
-    }
-  } else if (Scope == ReplayInlineScope::Function) {
+      ReplayMadeDecision = true;
+    } else if (ReplaySettings.ReplayMode ==
+               ReplayInlinerSettings::Mode::AlwaysInline) {
+      InlineRecommended =
+          llvm::InlineCost::getAlways("AlwaysInline replay mode");
+      ReplayMadeDecision = true;
+    } else if (ReplaySettings.ReplayMode ==
+               ReplayInlinerSettings::Mode::NeverInline)
+      ReplayMadeDecision = true;
+
+    LLVM_DEBUG(dbgs() << "Replay Inliner: "
+                      << (InlineRecommended ? "Inlined " : "Not Inlined ")
+                      << Callee << " @ " << CallSiteLoc << "\n");
+  }
+
+  // No decisions from replay inliner, see if we defer to original inliner or
+  // return a non-decision
+  if (!ReplayMadeDecision &&
+      ((ReplaySettings.ReplayScope == ReplayInlinerSettings::Scope::Function) ||
+       (ReplaySettings.ReplayMode == ReplayInlinerSettings::Mode::Original))) {
     if (OriginalAdvisor)
       return OriginalAdvisor->getAdvice(CB);
     return {};
   }
 
   return std::make_unique<DefaultInlineAdvice>(this, CB, InlineRecommended, ORE,
                                                EmitRemarks);
 }

llvm/lib/Transforms/IPO/Inliner.cpp

 
 static cl::opt<std::string> CGSCCInlineReplayFile(
     "cgscc-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by cgscc inlining."),
     cl::Hidden);
 
-static cl::opt<ReplayInlineScope> CGSCCInlineReplayScope(
-    "cgscc-inline-replay-scope", cl::init(ReplayInlineScope::Function),
-    cl::values(clEnumValN(ReplayInlineScope::Function, "Function",
+static cl::opt<ReplayInlinerSettings::Scope> CGSCCInlineReplayScope(
+    "cgscc-inline-replay-scope",
+    cl::init(ReplayInlinerSettings::Scope::Function),
+    cl::values(clEnumValN(ReplayInlinerSettings::Scope::Function, "Function",
                           "Replay on functions that have remarks associated "
                           "with them (default)"),
-               clEnumValN(ReplayInlineScope::Module, "Module",
+               clEnumValN(ReplayInlinerSettings::Scope::Module, "Module",
                           "Replay on the entire module")),
     cl::desc("Whether inline replay should be applied to the entire "
              "Module or just the Functions (default) that are present as "
              "callers in remarks during cgscc inlining."),
     cl::Hidden);
 
+static cl::opt<ReplayInlinerSettings::Mode> CGSCCInlineReplayMode(
+    "cgscc-inline-replay-mode", cl::init(ReplayInlinerSettings::Mode::Original),
+    cl::values(
+        clEnumValN(
+            ReplayInlinerSettings::Mode::Original, "Original",
+            "All decisions not in replay send to original advisor (default)"),
+        clEnumValN(ReplayInlinerSettings::Mode::AlwaysInline, "AlwaysInline",
+                   "All decisions not in replay are inlined"),
+        clEnumValN(ReplayInlinerSettings::Mode::NeverInline, "NeverInline",
+                   "All decisions not in replay are not inlined")),
+    cl::desc(
+        "How cgscc inline replay treats sites that don't come from the replay. "
+        "Original: defers to original advisor, AlwaysInline: inline all sites "
+        "not in replay, NeverInline: inline no sites not in replay"),
+    cl::Hidden);
+
+static cl::opt<CallSiteFormat::Format> CGSCCInlineReplayFormat(
+    "cgscc-inline-replay-format",
+    cl::init(CallSiteFormat::Format::LineColumnDiscriminator),
+    cl::values(
+        clEnumValN(CallSiteFormat::Format::Line, "Line", "<Line Number>"),
+        clEnumValN(CallSiteFormat::Format::LineColumn, "LineColumn",
+                   "<Line Number>:<Column Number>"),
+        clEnumValN(CallSiteFormat::Format::LineDiscriminator,
+                   "LineDiscriminator", "<Line Number>.<Discriminator>"),
+        clEnumValN(CallSiteFormat::Format::LineColumnDiscriminator,
+                   "LineColumnDiscriminator",
+                   "<Line Number>:<Column Number>.<Discriminator> (default)")),
+    cl::desc("How cgscc inline replay file is formatted"), cl::Hidden);
+
 static cl::opt<bool> InlineEnablePriorityOrder(
     "inline-enable-priority-order", cl::Hidden, cl::init(false),
     cl::desc("Enable the priority inline order for the inliner"));
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID) : CallGraphSCCPass(ID) {}
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID, bool InsertLifetime)
     : CallGraphSCCPass(ID), InsertLifetime(InsertLifetime) {}
if (!IAA) {
     // The one we would get from the MAM can be invalidated as a result of the
     // inliner's activity.
     OwnedAdvisor =
         std::make_unique<DefaultInlineAdvisor>(M, FAM, getInlineParams());
 
     if (!CGSCCInlineReplayFile.empty())
       OwnedAdvisor = getReplayInlineAdvisor(
           M, FAM, M.getContext(), std::move(OwnedAdvisor),
-          CGSCCInlineReplayFile, CGSCCInlineReplayScope,
+          ReplayInlinerSettings{CGSCCInlineReplayFile,
+                                CGSCCInlineReplayScope,
+                                CGSCCInlineReplayMode,
+                                {CGSCCInlineReplayFormat}},
           /*EmitRemarks=*/true);
 
     return *OwnedAdvisor;
   }
   assert(IAA->getAdvisor() &&
          "Expected a present InlineAdvisorAnalysis also have an "
          "InlineAdvisor initialized");
   return *IAA->getAdvisor();
while (!Calls->empty() && Calls->front().first->getCaller() == &F) {
           UR.InlinedInternalEdges.count({&N, C})) {
         LLVM_DEBUG(dbgs() << "Skipping inlining internal SCC edge from a node "
                              "previously split out of this SCC by inlining: "
                           << F.getName() << " -> " << Callee.getName() << "\n");
         setInlineRemark(*CB, "recursive SCC split");
         continue;
       }
 
       auto Advice = Advisor.getAdvice(*CB, OnlyMandatory);
 
       // Check whether we want to inline this callsite.
       if (!Advice || !Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         continue;
       }
 
       // Setup the data structure used to plumb customization into the
       // `InlineFunction` routine.
       InlineFunctionInfo IFI(
           /*cg=*/nullptr, GetAssumptionCache, PSI,
           &FAM.getResult<BlockFrequencyAnalysis>(*(CB->getCaller())),
ModuleInlinerWrapperPass::ModuleInlinerWrapperPass(InlineParams Params,
   if (MandatoryFirst)
     PM.addPass(InlinerPass(/*OnlyMandatory*/ true));
   PM.addPass(InlinerPass());
 }
 
 PreservedAnalyses ModuleInlinerWrapperPass::run(Module &M,
                                                 ModuleAnalysisManager &MAM) {
   auto &IAA = MAM.getResult<InlineAdvisorAnalysis>(M);
-  if (!IAA.tryCreate(Params, Mode, CGSCCInlineReplayFile,
-                     CGSCCInlineReplayScope)) {
+  if (!IAA.tryCreate(Params, Mode,
+                     {CGSCCInlineReplayFile,
+                      CGSCCInlineReplayScope,
+                      CGSCCInlineReplayMode,
+                      {CGSCCInlineReplayFormat}})) {
     M.getContext().emitError(
         "Could not setup Inlining Advisor for the requested "
         "mode and/or options");
     return PreservedAnalyses::all();
   }
 
   // We wrap the CGSCC pipeline in a devirtualization repeater. This will try
   // to detect when we devirtualize indirect calls and iterate the SCC passes

llvm/lib/Transforms/IPO/SampleProfile.cpp

 
 static cl::opt<std::string> ProfileInlineReplayFile(
     "sample-profile-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by inlining from sample profile loader."),
     cl::Hidden);
 
-static cl::opt<ReplayInlineScope> ProfileInlineReplayScope(
-    "sample-profile-inline-replay-scope", cl::init(ReplayInlineScope::Function),
-    cl::values(clEnumValN(ReplayInlineScope::Function, "Function",
+static cl::opt<ReplayInlinerSettings::Scope> ProfileInlineReplayScope(
+    "sample-profile-inline-replay-scope",
+    cl::init(ReplayInlinerSettings::Scope::Function),
+    cl::values(clEnumValN(ReplayInlinerSettings::Scope::Function, "Function",
                           "Replay on functions that have remarks associated "
                           "with them (default)"),
-               clEnumValN(ReplayInlineScope::Module, "Module",
+               clEnumValN(ReplayInlinerSettings::Scope::Module, "Module",
                           "Replay on the entire module")),
     cl::desc("Whether inline replay should be applied to the entire "
              "Module or just the Functions (default) that are present as "
              "callers in remarks during sample profile inlining."),
     cl::Hidden);
 
+static cl::opt<ReplayInlinerSettings::Mode> ProfileInlineReplayMode(
+    "sample-profile-inline-replay-mode",
+    cl::init(ReplayInlinerSettings::Mode::Original),
+    cl::values(
+        clEnumValN(
+            ReplayInlinerSettings::Mode::Original, "Original",
+            "All decisions not in replay send to original advisor (default)"),
+        clEnumValN(ReplayInlinerSettings::Mode::AlwaysInline, "AlwaysInline",
+                   "All decisions not in replay are inlined"),
+        clEnumValN(ReplayInlinerSettings::Mode::NeverInline, "NeverInline",
+                   "All decisions not in replay are not inlined")),
+    cl::desc("How sample profile inline replay treats sites that don't come "
+             "from the replay. Original: defers to original advisor, "
+             "AlwaysInline: inline all sites not in replay, NeverInline: "
+             "inline no sites not in replay"),
+    cl::Hidden);
+
+static cl::opt<CallSiteFormat::Format> ProfileInlineReplayFormat(
+    "sample-profile-inline-replay-format",
+    cl::init(CallSiteFormat::Format::LineColumnDiscriminator),
+    cl::values(
+        clEnumValN(CallSiteFormat::Format::Line, "Line", "<Line Number>"),
+        clEnumValN(CallSiteFormat::Format::LineColumn, "LineColumn",
+                   "<Line Number>:<Column Number>"),
+        clEnumValN(CallSiteFormat::Format::LineDiscriminator,
+                   "LineDiscriminator", "<Line Number>.<Discriminator>"),
+        clEnumValN(CallSiteFormat::Format::LineColumnDiscriminator,
+                   "LineColumnDiscriminator",
+                   "<Line Number>:<Column Number>.<Discriminator> (default)")),
+    cl::desc("How sample profile inline replay file is formatted"), cl::Hidden);
+
 static cl::opt<unsigned>
     MaxNumPromotions("sample-profile-icp-max-prom", cl::init(3), cl::Hidden,
                      cl::ZeroOrMore,
                      cl::desc("Max number of promotions for a single indirect "
                               "call callsite in sample profile loader"));
 
 static cl::opt<bool> OverwriteExistingWeights(
     "overwrite-existing-weights", cl::Hidden, cl::init(false),
bool SampleProfileLoader::doInitialization(Module &M,
   if (ProfAccForSymsInList) {
     NamesInProfile.clear();
     if (auto NameTable = Reader->getNameTable())
       NamesInProfile.insert(NameTable->begin(), NameTable->end());
     CoverageTracker.setProfAccForSymsInList(true);
   }
 
   if (FAM && !ProfileInlineReplayFile.empty()) {
     ExternalInlineAdvisor = std::make_unique<ReplayInlineAdvisor>(
-        M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr, ProfileInlineReplayFile,
-        ProfileInlineReplayScope, /*EmitRemarks=*/false);
+        M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr,
+        ReplayInlinerSettings{ProfileInlineReplayFile,
+                              ProfileInlineReplayScope,
+                              ProfileInlineReplayMode,
+                              {ProfileInlineReplayFormat}},
+        /*EmitRemarks=*/false);
   }
 
   // Apply tweaks if context-sensitive profile is available.
   if (Reader->profileIsCS()) {
     ProfileIsCS = true;
     FunctionSamples::ProfileIsCS = true;
 
     // Enable priority-base inliner and size inline by default for CSSPGO.

llvm/test/Transforms/Inline/Inputs/cgscc-inline-replay-function.txt

+remark: calls.cc:4:0: '_Z3subii' will not be inlined into 'main' with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;

llvm/test/Transforms/Inline/Inputs/cgscc-inline-replay-line.txt

+remark: calls.cc:4: '_Z3subii' will not be inlined into '_Z3sumii' with (cost=Never) at callsite _Z3sumii:1;
+remark: calls.cc:10: '_Z3sumii' inlined into 'main' with (cost=45, threshold=337) at callsite main:3;

llvm/test/Transforms/Inline/Inputs/cgscc-inline-replay.txt

+remark: calls.cc:4:0: '_Z3subii' will not be inlined into '_Z3sumii' with (cost=Never) at callsite _Z3sumii:1:0;
 remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' with (cost=45, threshold=337) at callsite main:3:0.1;
-remark: calls.cc:4:0: '_Z3subii' inlined into 'main' with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-cgscc-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
 ; RUN: opt < %s -passes=inline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
-;; Check module-scope replay inline decisions
-; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+;; Check Module scope Original mode replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-mode=Original -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
 
-;; Check function-scope inline replay decisions
-; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Function -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION %s
+;; Check Module scope Original mode replay inline with 'Line' format decisions
+;; The results are not different than REPLAY-MODULE-ORIGINAL, but the replay input only contains line numbers rather than line:column.discriminator
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-line.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-mode=Original -cgscc-inline-replay-format=Line -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope AlwaysInline mode replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-mode=AlwaysInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ALWAYS %s
+
+;; Check Module scope NeverInline mode replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-mode=NeverInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-NEVER %s
+
+;; Check Function scope Original mode replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-function.txt -cgscc-inline-replay-scope=Function -cgscc-inline-replay-mode=Original -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION-ORIGINAL %s
+
+;; Check Function scope AlwaysInline mode replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-function.txt -cgscc-inline-replay-scope=Function -cgscc-inline-replay-mode=AlwaysInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION-ALWAYS %s
+
+;; Check Function scope NeverInline mode replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-function.txt -cgscc-inline-replay-scope=Function -cgscc-inline-replay-mode=NeverInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION-NEVER %s
 
 ;; Check behavior on non-existent replay file
 ; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
 
 ;; Check scope inlining errors out on non <Module|Function> inputs
 ; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=function -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
 
+;; Check mode inlining errors out on non <Original|AlwaysInline|NeverInline> inputs
+; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-mode=original -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-MODE %s
+
+;; Check format inlining errors out on non <Line|AlwayLineColumnsInline|LineDiscriminator|LineColumnDiscriminator> inputs
+; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-format=line -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-FORMAT %s
+
 ; DEFAULT: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
 ; DEFAULT: '_Z3sumii' inlined into 'main' with (cost={{[-0-9]+}}
 ; DEFAULT-NOT: '_Z3subii' inlined into 'main'
 
-; REPLAY: '_Z3sumii' inlined into 'main' with (cost=always)
-; REPLAY: '_Z3subii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-ORIGINAL: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-ORIGINAL: '_Z3subii' inlined into 'main' with (cost={{[-0-9]+}}
+
+; REPLAY-MODULE-ALWAYS: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-ALWAYS: '_Z3subii' inlined into 'main' with (cost=always)
+
+; REPLAY-MODULE-NEVER: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-NEVER-NOT: '_Z3subii' inlined into 'main'
+
+; REPLAY-FUNCTION-ORIGINAL: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION-ORIGINAL: '_Z3sumii' inlined into 'main' with (cost={{[-0-9]+}}
+
+; REPLAY-FUNCTION-ALWAYS: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION-ALWAYS: '_Z3sumii' inlined into 'main' with (cost=always)
 
-; REPLAY-FUNCTION: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
-; REPLAY-FUNCTION: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-FUNCTION-NEVER: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION-NEVER-NOT: '_Z3sumii' inlined into 'main'
 
 ; REPLAY-ERROR: error: Could not open remarks file:
 ; REPLAY-ERROR-SCOPE: for the --cgscc-inline-replay-scope option: Cannot find option named 'function'!
+; REPLAY-ERROR-MODE: for the --cgscc-inline-replay-mode option: Cannot find option named 'original'!
+; REPLAY-ERROR-FORMAT: for the --cgscc-inline-replay-format option: Cannot find option named 'line'!
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-function.txt

 remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-line.txt

+remark: calls.cc:4: '_Z3subii' will not be inlined into '_Z3sumii' to match profiling context with (cost=Never) at callsite _Z3sumii:1;
+remark: calls.cc:10: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

-remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
-remark: calls.cc:4:0: '_Z3subii' inlined into 'main' to match profiling context with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;
+remark: calls.cc:4:0: '_Z3subii' will not be inlined into '_Z3sumii' to match profiling context with (cost=Never) at callsite _Z3sumii:1:0;
+remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-sample-profile-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
-;; Check replay inline decisions
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module  -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY %s
+;; Check Module scope Original mode replay inline decisions 
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-mode=Original -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope Original mode replay inline with 'Line' format decisions 
+;; The results are not different than REPLAY-MODULE-ORIGINAL, but the replay input only contains line numbers rather than line:column.discriminator
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-line.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-mode=Original -sample-profile-inline-replay-format=Line -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope AlwaysInline mode replay inline decisions 
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-mode=AlwaysInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ALWAYS %s
+
+;; Check Module scope NeverInline mode replay inline decisions 
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-mode=NeverInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-NEVER %s
 
 ;; Check baseline inline decisions with "inline-topdown-inline-all.prof" which inlines all sites
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=DEFAULT-ALL %s
 
 ;; Check function scope replay inline decisions with "inline-topdown-inline-all.prof" and "inline-topdown-function-scope.txt" which only contains: '_Z3sumii' inlined into 'main'
 ;; 1. _Z3sumii is inlined into main, but all other inline candidates in main (e.g. _Z3subii) are not inlined
 ;; 2. Inline decisions made in other functions match default sample inlining, in this case _Z3subii is inlined into _Z3sumii
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function-scope.txt -sample-profile-inline-replay-scope=Function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function.txt -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay-mode=NeverInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION-NEVER %s
+
+;; Function scope Original mode
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function.txt -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay-mode=Original -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION-ORIGINAL %s
+
+;; Function scope AlwaysInline mode
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function.txt -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay-mode=AlwaysInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION-ALWAYS %s
 
 ;; Check behavior on non-existent replay file
 ; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
 
 ;; Check scope inlining errors out on non <Module|Function> inputs
 ; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
 
+;; Check mode inlining errors out on non <Original|AlwaysInline|NeverInline> inputs
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-mode=original -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-MODE %s
+
+;; Check format inlining errors out on non <Line|AlwayLineColumnsInline|LineDiscriminator|LineColumnDiscriminator> inputs
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-format=line -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-FORMAT %s
+
 ; DEFAULT: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-NOT: '_Z3subii' inlined into 'main'
 
-; REPLAY: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
-; REPLAY: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
-; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'
+; REPLAY-MODULE-ORIGINAL: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-ORIGINAL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-MODULE-ALWAYS: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-ALWAYS: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
+
+; REPLAY-MODULE-NEVER: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-NEVER-NOT: '_Z3subii' inlined into 'main'
 
 ; DEFAULT-ALL: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-ALL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-ALL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 
-; REPLAY-ALL-FUNCTION : _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
-; REPLAY-ALL-FUNCTION-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
-; REPLAY-ALL-FUNCTION: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION-NEVER: _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-NEVER-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION-NEVER: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ALL-FUNCTION-ORIGINAL: _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-ORIGINAL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION-ORIGINAL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ALL-FUNCTION-ALWAYS: _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-ALWAYS: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-ALWAYS: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 
 ; REPLAY-ERROR: error: Could not open remarks file:
 ; REPLAY-ERROR-SCOPE: for the --sample-profile-inline-replay-scope option: Cannot find option named 'function'!
+; REPLAY-ERROR-MODE: for the --sample-profile-inline-replay-mode option: Cannot find option named 'original'!
+; REPLAY-ERROR-FORMAT: for the --sample-profile-inline-replay-format option: Cannot find option named 'line'!
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4