[InlineAdvisor] Add fallback/format switches and negative remark processing to Replay Inliner

[modimo]

Summary

Adds the following switches:

1. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback: controls what the replay advisor does for inline sites that are not present in the replay. Options are:
   1. Original: defers to original advisor
   2. AlwaysInline: inline all sites not in replay
   3. NeverInline: inline no sites not in replay

2. --sample-profile-inline-replay-format/--cgscc-inline-replay-format: controls what format should be generated to match against the replay remarks. Options are:
   1. Line
   2. LineColumn
   3. LineDiscriminator
   4. LineColumnDiscriminator

Adds support for negative inlining decisions. These are denoted by "will not be inlined into" as compared to the positive "inlined into" in the remarks.

All of these together with the previous --sample-profile-inline-replay-scope/--cgscc-inline-replay-scope allow tweaking in how to apply replay. In my testing, I'm using:

1. --sample-profile-inline-replay-scope/--cgscc-inline-replay-scope = Function to only replay on a function
2. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback = NeverInline since I'm feeding in only positive remarks to the replay system
3. --sample-profile-inline-replay-format/--cgscc-inline-replay-format = Line since I'm generating the remarks from DWARF information from GCC which can conflict quite heavily in column number compared to Clang

An alternative configuration could be to do Function, AlwaysInline, Line fallback with negative remarks which closer matches the final call-sites. Note that this can lead to unbounded inlining if a negative remark doesn't match/exist for one reason or another.

Updated various tests to cover the new switches and negative remarks

Testing:
ninja check-all

[wenlei]

llvm/include/llvm/Analysis/InlineAdvisor.h

49

nit: OutputColumn -> outputColumn, same for OutputDiscriminator

65

I think this now reflects a narrower aspect of the settings than mode. Specifically, this is some kind of fall back inline strategy. I suggest we rename this field as well as the flag names because "AlwaysInline" reply mode doesn't really convey the right message, and can be confusing.

We can rename Mode -> Fallback, and xxx-inline-replay-mode -> xxx-inline-replay-fallback.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

101–149

always inline, never inline and original advisor are parallel, I think we can restructure the logic and add some comments to help readability:

// We have a decision from replay
if (hasInlineAdvice(..)) {

  return ...
}

// We need to fall back ..
if (always inline)

else if (never inline)

else {
  assert(original);
  
}

return ..

102–103

Use hasInlineAdvice (hasRemarksForFunction)?

127–144

Should we also return a negative decision (never inline) if InlineSitesFromRemarks[Combined] is false?

llvm/lib/Transforms/IPO/SampleProfile.cpp

1921–1928

Hmm.. now I'm a bit confused why the call to getReplayInlineAdvisor is removed in the other diff.

llvm/test/Transforms/SampleProfile/inline-replay.ll

63

AlwayLineColumnsInline -> LineColumn

101–105

These are probably not very necessary as they're testing cl::opt/cl::value/clEnumValN, though there's no harm in having them. On the other hand, it might be good have a case to cover LineColumn
or LineDiscriminator.

[mtrofin]

llvm/include/llvm/Analysis/InlineAdvisor.h

41–42

I think we shold start moving the replay stuff in its header (with possibly some refactoring of how we get to obtain an advisor)

[modimo]

Review feedback

llvm/include/llvm/Analysis/InlineAdvisor.h

41–42

Moved ReplayInlinerSettings to ReplayInlineAdvisor.h and forward declared it like OptimizationRemarkEmitter in InlineAdvisor.h

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

127–144

Good catch, yes we should

llvm/lib/Transforms/IPO/SampleProfile.cpp

1921–1928

TL;DR getReplayInlineAdvisor is the correct way to do this.

That came about due to how we query if the replay inliner has remarks. Originally (including v1 here) I used hasRemarksForFunction which doesn't exist in the base InlineAdvisor class so the advisor needed to specifically be a ReplayInlineAdvisor. After I made the changes in D112033 to just rely on a decision being returned this was no longer necessary.

llvm/test/Transforms/SampleProfile/inline-replay.ll

101–105

Added them above

[wenlei]

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

30

We can rename Mode -> Fallback, and xxx-inline-replay-mode -> xxx-inline-replay-fallback.

I meant to say we should rename all "Mode" to be Fallback, beyond the switch.

76

I thought I commented somewhere - hasInlineAdvice is a better name. When scope is module, we may not have remark for this function, and yet it can return true.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

101–149

Actually what I suggested earlier was wrong based on current implementation of hasRemarksForFunction/hasInlineAdvice. That was assuming hasRemarksForFunction only returns true for explicit replay decision, and false for fall back cases.

Perhaps something like this would be a clean structure:

// This inline decision is out of scope for replay advisor
if (!hasInlineAdvice(..)) {
  return {}
}

// Use explicit replay decision if available
Optional<bool> ShouldInline = getAdviceFromRemarks();
if (ShouldInline.hasValue()) {
  if (ShouldInline.getValue()) {
    ..
  }
  else {
   ..
  }
}

// We need to use fall back decisions ..
if (fall back is always inline)

else if (fall back is never inline)

else 
   assert(fall back is original)

return ..

getAdviceFromRemarks looks up InlineSitesFromRemarks and returns three-way value.

152–155

Why is this a Nona instead of llvm::InlineCost::getNever(..)? Same for the other instance above.

llvm/lib/Transforms/IPO/Inliner.cpp

881–885

Why we do we skip recordUnattemptedInlining when not not inlining due to not having an advice?

[wenlei]

Change summary needs to be updated as well.

[modimo]

ReplayMode->ReplayFallback, refactoring

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

30

I missed that one, thanks.

76

You commented that in the other change (D112033). Still makes sense, changed.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

101–149

With the nesting of the CGSCC inliner inside this, no advice still defers to the original advisor if it exists. That above is cleaner though, changed.

152–155

DefaultInlineAdvice(InlineAdvisor *Advisor, CallBase &CB,
                    Optional<InlineCost> OIC, OptimizationRemarkEmitter &ORE,
                    bool EmitRemarks = true)
    : InlineAdvice(Advisor, CB, ORE, /*IsInliningRecommended=*/OIC.hasValue())

If the Optional cost has a value, IsInliningRecommended gets set to true. This looks intended. I'll add comments to that effect.

llvm/lib/Transforms/IPO/Inliner.cpp

881–885

Advice is a std::unique_ptr<InlineAdvice> so being negative means it doesn't contain a value. Calling recordUnattemptedInlining on that leads to an AV. This path is untested ATM because the cgscc advisor is always nested inside the replay one so there will always be an InlineAdvice returned. I came upon this when one of my intermediate edits didn't forward the decision to the original advisor.

I made the auto explicit to make this clearer.

[wenlei]

lgtm, thanks.

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

101–149

Yes, it was functionally correct, but logically wrong..

[modimo]

Fix up comment

[mtrofin]

llvm/include/llvm/Analysis/InlineAdvisor.h

235

should ReplayInlinerSettings be passed as const &?

257–258

can this declaration also move to ReplayInlineAdvisor.h?

283

can formatting (so callsiteformat and getCallSiteLocation) be moved to ReplayInlineAdvisor.h?

also: s/getCallSiteLocation/formatCallSiteLocation.

llvm/lib/Transforms/IPO/Inliner.cpp

877–878

unrelated change?

[modimo]

Move more things to ReplayInlineAdvisor.h and other feedback

llvm/include/llvm/Analysis/InlineAdvisor.h

283

Yes, this was originally added for replay and only that consumes this atm.

llvm/lib/Transforms/IPO/Inliner.cpp

877–878

The comment below is that !Advice isn't very clear with auto and also tripped me up: the previous change
if (!Advice || !Advice->isInliningRecommended()) {

treated !Advice as having "negative" advice rather than being empty. Clearing this up seems valuable.

[mtrofin]

lgtm

llvm/lib/Transforms/IPO/Inliner.cpp

877–878

ah, good point, thanks!

[modimo]

Rebase and add -disable-output consistently to inline-replay.ll

[modimo]

Thanks for the reviews @mtrofin !

llvm/include/llvm/Analysis/InlineAdvisor.h

 #include <unordered_set>
 
 namespace llvm {
 class BasicBlock;
 class CallBase;
 class Function;
 class Module;
 class OptimizationRemarkEmitter;
+struct ReplayInlinerSettings;
 
 /// There are 3 scenarios we can use the InlineAdvisor:
 /// - Default - use manual heuristics.
 ///
 /// - Release mode, the expected mode for production, day to day deployments.
 /// In this mode, when building the compiler, we also compile a pre-trained ML
 /// model to native code, and link it as a static library. This mode has low
 /// overhead and no additional dependencies for the compiler runtime.
 ///
 /// - Development mode, for training new models.
 /// In this mode, we trade off runtime performance for flexibility. This mode
 /// requires the full C Tensorflow API library, and evaluates models
 /// dynamically. This mode also permits generating training logs, for offline
 /// training.
 enum class InliningAdvisorMode : int { Default, Release, Development };
 
-/// For Replay Inliner initialization
-enum class ReplayInlineScope : int { Function, Module };
-
 class InlineAdvisor;
 /// Capture state between an inlining decision having had been made, and
 /// its impact being observable. When collecting model training data, this
 /// allows recording features/decisions/partial reward data sets.
 ///
 /// Derivations of this type are expected to be tightly coupled with their
 /// InliningAdvisors. The base type implements the minimal contractual
 /// obligations.
 class InlineAdvice {
 public:
   InlineAdvice(InlineAdvisor *Advisor, CallBase &CB,
                OptimizationRemarkEmitter &ORE, bool IsInliningRecommended);
 
   InlineAdvice(InlineAdvice &&) = delete;
   InlineAdvice(const InlineAdvice &) = delete;
   virtual ~InlineAdvice() {
     assert(Recorded && "InlineAdvice should have been informed of the "
                        "inliner's decision in all cases");
   }
 
   /// Exactly one of the record* APIs must be called. Implementers may extend
   /// behavior by implementing the corresponding record*Impl.
   ///
   /// Call after inlining succeeded, and did not result in deleting the callee.
   void recordInlining();
 
   /// Call after inlining succeeded, and resulted in deleting the callee.
   void recordInliningWithCalleeDeleted();
 
   /// Call after the decision for a call site was to not inline.
   void recordUnsuccessfulInlining(const InlineResult &Result) {
     markRecorded();
struct Result {
     bool invalidate(Module &, const PreservedAnalyses &PA,
                     ModuleAnalysisManager::Invalidator &) {
       // Check whether the analysis has been explicitly invalidated. Otherwise,
       // it's stateless and remains preserved.
       auto PAC = PA.getChecker<InlineAdvisorAnalysis>();
       return !PAC.preservedWhenStateless();
     }
     bool tryCreate(InlineParams Params, InliningAdvisorMode Mode,
-                   StringRef ReplayFile, ReplayInlineScope ReplayScope);
+                   const ReplayInlinerSettings &ReplaySettings);
     InlineAdvisor *getAdvisor() const { return Advisor.get(); }
 
   private:
     Module &M;
     ModuleAnalysisManager &MAM;
     std::unique_ptr<InlineAdvisor> Advisor;
   };
 
   Result run(Module &M, ModuleAnalysisManager &MAM) { return Result(M, MAM); }
 };
 
 #ifdef LLVM_HAVE_TF_AOT
 std::unique_ptr<InlineAdvisor>
 getReleaseModeAdvisor(Module &M, ModuleAnalysisManager &MAM);
 #endif
 
 #ifdef LLVM_HAVE_TF_API
 std::unique_ptr<InlineAdvisor>
 getDevelopmentModeAdvisor(Module &M, ModuleAnalysisManager &MAM,
                           std::function<bool(CallBase &)> GetDefaultAdvice);
 #endif
 
-std::unique_ptr<InlineAdvisor> getReplayInlineAdvisor(
-    Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
-    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    ReplayInlineScope Scope, bool EmitRemarks);
-
 // Default (manual policy) decision making helper APIs. Shared with the legacy
 // pass manager inliner.
 
 /// Return the cost only if the inliner should attempt to inline at the given
 /// CallSite. If we return the cost, we will emit an optimisation remark later
 /// using that cost, so we won't do so from this function. Return None if
 /// inlining should not be attempted.
 Optional<InlineCost>
 shouldInline(CallBase &CB, function_ref<InlineCost(CallBase &CB)> GetInlineCost,
              OptimizationRemarkEmitter &ORE, bool EnableDeferral = true);
 
 /// Emit ORE message.
 void emitInlinedInto(OptimizationRemarkEmitter &ORE, DebugLoc DLoc,
                      const BasicBlock *Block, const Function &Callee,
                      const Function &Caller, bool IsMandatory,
                      function_ref<void(OptimizationRemark &)> ExtraContext = {},
                      const char *PassName = nullptr);
 
 /// Emit ORE message based in cost (default heuristic).
 void emitInlinedIntoBasedOnCost(OptimizationRemarkEmitter &ORE, DebugLoc DLoc,
                                 const BasicBlock *Block, const Function &Callee,
                                 const Function &Caller, const InlineCost &IC,
                                 bool ForProfileContext = false,
                                 const char *PassName = nullptr);
 
-/// get call site location as string
-std::string getCallSiteLocation(DebugLoc DLoc);
-
 /// Add location info to ORE message.
 void addLocationToRemarks(OptimizationRemark &Remark, DebugLoc DLoc);
 
 /// Set the inline-remark attribute.
 void setInlineRemark(CallBase &CB, StringRef Message);
 
 /// Utility for extracting the inline cost message to a string.
 std::string inlineCostStr(const InlineCost &IC);
 } // namespace llvm
 #endif // LLVM_ANALYSIS_INLINEADVISOR_H

llvm/include/llvm/Analysis/ReplayInlineAdvisor.h

 
 namespace llvm {
 class BasicBlock;
 class CallBase;
 class Function;
 class Module;
 class OptimizationRemarkEmitter;
 
+struct CallSiteFormat {
+  enum class Format : int {
+    Line,
+    LineColumn,
+    LineDiscriminator,
+    LineColumnDiscriminator
+  };
+
+  bool outputColumn() const {
+    return OutputFormat == Format::LineColumn ||
+           OutputFormat == Format::LineColumnDiscriminator;
+  }
+
+  bool outputDiscriminator() const {
+    return OutputFormat == Format::LineDiscriminator ||
+           OutputFormat == Format::LineColumnDiscriminator;
+  }
+
+  Format OutputFormat;
+};
+
+/// Replay Inliner Setup
+struct ReplayInlinerSettings {
+  enum class Scope : int { Function, Module };
+  enum class Fallback : int { Original, AlwaysInline, NeverInline };
+
+  StringRef ReplayFile;
+  Scope ReplayScope;
+  Fallback ReplayFallback;
+  CallSiteFormat ReplayFormat;
+};
+
+/// Get call site location as a string with the given format
+std::string formatCallSiteLocation(DebugLoc DLoc, const CallSiteFormat &Format);
+
+std::unique_ptr<InlineAdvisor> getReplayInlineAdvisor(
+    Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor,
+    const ReplayInlinerSettings &ReplaySettings, bool EmitRemarks);
+
 /// Replay inline advisor that uses optimization remarks from inlining of
 /// previous build to guide current inlining. This is useful for inliner tuning.
 class ReplayInlineAdvisor : public InlineAdvisor {
 public:
   ReplayInlineAdvisor(Module &M, FunctionAnalysisManager &FAM,
                       LLVMContext &Context,
                       std::unique_ptr<InlineAdvisor> OriginalAdvisor,
-                      StringRef RemarksFile, ReplayInlineScope Scope,
+                      const ReplayInlinerSettings &ReplaySettings,
                       bool EmitRemarks);
   std::unique_ptr<InlineAdvice> getAdviceImpl(CallBase &CB) override;
   bool areReplayRemarksLoaded() const { return HasReplayRemarks; }
 
 private:
+  bool hasInlineAdvice(Function &F) const {
+    return (ReplaySettings.ReplayScope ==
+            ReplayInlinerSettings::Scope::Module) ||
+           CallersToReplay.contains(F.getName());
+  }
   std::unique_ptr<InlineAdvisor> OriginalAdvisor;
   bool HasReplayRemarks = false;
-  ReplayInlineScope Scope;
+  const ReplayInlinerSettings ReplaySettings;
   bool EmitRemarks = false;
 
   StringMap<bool> InlineSitesFromRemarks;
   StringSet<> CallersToReplay;
 };
 } // namespace llvm
 #endif // LLVM_ANALYSIS_REPLAYINLINEADVISOR_H

llvm/lib/Analysis/InlineAdvisor.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
void InlineAdvice::recordInliningWithCalleeDeleted() {
   markRecorded();
   recordInlineStatsIfNeeded();
   Advisor->markFunctionAsDeleted(Callee);
   recordInliningWithCalleeDeletedImpl();
 }
 
 AnalysisKey InlineAdvisorAnalysis::Key;
 
-bool InlineAdvisorAnalysis::Result::tryCreate(InlineParams Params,
-                                              InliningAdvisorMode Mode,
-                                              StringRef ReplayFile,
-                                              ReplayInlineScope ReplayScope) {
+bool InlineAdvisorAnalysis::Result::tryCreate(
+    InlineParams Params, InliningAdvisorMode Mode,
+    const ReplayInlinerSettings &ReplaySettings) {
   auto &FAM = MAM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   switch (Mode) {
   case InliningAdvisorMode::Default:
     LLVM_DEBUG(dbgs() << "Using default inliner heuristic.\n");
     Advisor.reset(new DefaultInlineAdvisor(M, FAM, Params));
     // Restrict replay to default advisor, ML advisors are stateful so
     // replay will need augmentations to interleave with them correctly.
-    if (!ReplayFile.empty()) {
-      Advisor = llvm::getReplayInlineAdvisor(
-          M, FAM, M.getContext(), std::move(Advisor), ReplayFile, ReplayScope,
+    if (!ReplaySettings.ReplayFile.empty()) {
+      Advisor = llvm::getReplayInlineAdvisor(M, FAM, M.getContext(),
+                                             std::move(Advisor), ReplaySettings,
                                              /* EmitRemarks =*/true);
     }
     break;
   case InliningAdvisorMode::Development:
 #ifdef LLVM_HAVE_TF_API
     LLVM_DEBUG(dbgs() << "Using development-mode inliner policy.\n");
     Advisor =
         llvm::getDevelopmentModeAdvisor(M, MAM, [&FAM, Params](CallBase &CB) {
           auto OIC = getDefaultInlineAdvice(CB, FAM, Params);
if (EnableDeferral &&
     return None;
   }
 
   LLVM_DEBUG(dbgs() << "    Inlining " << inlineCostStr(IC) << ", Call: " << CB
                     << '\n');
   return IC;
 }
 
-std::string llvm::getCallSiteLocation(DebugLoc DLoc) {
+std::string llvm::formatCallSiteLocation(DebugLoc DLoc,
+                                         const CallSiteFormat &Format) {
   std::string Buffer;
   raw_string_ostream CallSiteLoc(Buffer);
   bool First = true;
   for (DILocation *DIL = DLoc.get(); DIL; DIL = DIL->getInlinedAt()) {
     if (!First)
       CallSiteLoc << " @ ";
     // Note that negative line offset is actually possible, but we use
     // unsigned int to match line offset representation in remarks so
     // it's directly consumable by relay advisor.
     uint32_t Offset =
         DIL->getLine() - DIL->getScope()->getSubprogram()->getLine();
     uint32_t Discriminator = DIL->getBaseDiscriminator();
     StringRef Name = DIL->getScope()->getSubprogram()->getLinkageName();
     if (Name.empty())
       Name = DIL->getScope()->getSubprogram()->getName();
-    CallSiteLoc << Name.str() << ":" << llvm::utostr(Offset) << ":"
-                << llvm::utostr(DIL->getColumn());
-    if (Discriminator)
+    CallSiteLoc << Name.str() << ":" << llvm::utostr(Offset);
+    if (Format.outputColumn())
+      CallSiteLoc << ":" << llvm::utostr(DIL->getColumn());
+    if (Format.outputDiscriminator() && Discriminator)
       CallSiteLoc << "." << llvm::utostr(Discriminator);
     First = false;
   }
 
   return CallSiteLoc.str();
 }
 
 void llvm::addLocationToRemarks(OptimizationRemark &Remark, DebugLoc DLoc) {

llvm/lib/Analysis/ReplayInlineAdvisor.cpp

 #include <memory>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "replay-inline"
 
 ReplayInlineAdvisor::ReplayInlineAdvisor(
     Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
-    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    ReplayInlineScope Scope, bool EmitRemarks)
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor,
+    const ReplayInlinerSettings &ReplaySettings, bool EmitRemarks)
     : InlineAdvisor(M, FAM), OriginalAdvisor(std::move(OriginalAdvisor)),
-      HasReplayRemarks(false), Scope(Scope), EmitRemarks(EmitRemarks) {
+      HasReplayRemarks(false), ReplaySettings(ReplaySettings),
+      EmitRemarks(EmitRemarks) {
 
-  auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(RemarksFile);
+  auto BufferOrErr = MemoryBuffer::getFileOrSTDIN(ReplaySettings.ReplayFile);
   std::error_code EC = BufferOrErr.getError();
   if (EC) {
     Context.emitError("Could not open remarks file: " + EC.message());
     return;
   }
 
   // Example for inline remarks to parse:
   //   main:3:1.1: '_Z3subii' inlined into 'main' at callsite sum:1 @
   //   main:3:1.1;
   // We use the callsite string after `at callsite` to replay inlining.
   line_iterator LineIt(*BufferOrErr.get(), /*SkipBlanks=*/true);
+  const std::string PositiveRemark = "' inlined into '";
+  const std::string NegativeRemark = "' will not be inlined into '";
+
   for (; !LineIt.is_at_eof(); ++LineIt) {
     StringRef Line = *LineIt;
     auto Pair = Line.split(" at callsite ");
 
-    auto CalleeCaller = Pair.first.split("' inlined into '");
+    bool IsPositiveRemark = true;
+    if (Pair.first.contains(NegativeRemark))
+      IsPositiveRemark = false;
+
+    auto CalleeCaller =
+        Pair.first.split(IsPositiveRemark ? PositiveRemark : NegativeRemark);
 
     StringRef Callee = CalleeCaller.first.rsplit(": '").second;
     StringRef Caller = CalleeCaller.second.rsplit("'").first;
 
     auto CallSite = Pair.second.split(";").first;
 
     if (Callee.empty() || Caller.empty() || CallSite.empty()) {
       Context.emitError("Invalid remark format: " + Line);
       return;
     }
 
     std::string Combined = (Callee + CallSite).str();
-    InlineSitesFromRemarks[Combined] = false;
-    if (Scope == ReplayInlineScope::Function)
+    InlineSitesFromRemarks[Combined] = IsPositiveRemark;
+    if (ReplaySettings.ReplayScope == ReplayInlinerSettings::Scope::Function)
       CallersToReplay.insert(Caller);
   }
 
   HasReplayRemarks = true;
 }
 
 std::unique_ptr<InlineAdvisor> llvm::getReplayInlineAdvisor(
     Module &M, FunctionAnalysisManager &FAM, LLVMContext &Context,
-    std::unique_ptr<InlineAdvisor> OriginalAdvisor, StringRef RemarksFile,
-    ReplayInlineScope Scope, bool EmitRemarks) {
+    std::unique_ptr<InlineAdvisor> OriginalAdvisor,
+    const ReplayInlinerSettings &ReplaySettings, bool EmitRemarks) {
   auto Advisor = std::make_unique<ReplayInlineAdvisor>(
-      M, FAM, Context, std::move(OriginalAdvisor), RemarksFile, Scope,
-      EmitRemarks);
+      M, FAM, Context, std::move(OriginalAdvisor), ReplaySettings, EmitRemarks);
   if (!Advisor->areReplayRemarksLoaded())
     Advisor.reset();
   return Advisor;
 }
 
 std::unique_ptr<InlineAdvice> ReplayInlineAdvisor::getAdviceImpl(CallBase &CB) {
   assert(HasReplayRemarks);
 
   Function &Caller = *CB.getCaller();
   auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(Caller);
 
-  Optional<InlineCost> InlineRecommended;
+  // Decision not made by replay system
+  if (!hasInlineAdvice(*CB.getFunction())) {
+    // If there's a registered original advisor, return its decision
+    if (OriginalAdvisor)
+      return OriginalAdvisor->getAdvice(CB);
 
-  if (Scope == ReplayInlineScope::Module ||
-      CallersToReplay.count(CB.getFunction()->getName())) {
-    std::string CallSiteLoc = getCallSiteLocation(CB.getDebugLoc());
-    StringRef Callee = CB.getCalledFunction()->getName();
-    std::string Combined = (Callee + CallSiteLoc).str();
+    // If no decision is made above, return non-decision
+    return {};
+  }
+
+  std::string CallSiteLoc =
+      formatCallSiteLocation(CB.getDebugLoc(), ReplaySettings.ReplayFormat);
 
-    auto Iter = InlineSitesFromRemarks.find(Combined);
-    if (Iter != InlineSitesFromRemarks.end()) {
-      InlineSitesFromRemarks[Combined] = true;
-      InlineRecommended = llvm::InlineCost::getAlways("previously inlined");
-    }
-  } else if (Scope == ReplayInlineScope::Function) {
+  // Replay decision, if it has one
+    if (InlineSitesFromRemarks[Combined]) {
+      LLVM_DEBUG(dbgs() << "Replay Inliner: Inlined " << Callee << " @ "
+                        << CallSiteLoc << "\n");
+      return std::make_unique<DefaultInlineAdvice>(
+          this, CB, llvm::InlineCost::getAlways("previously inlined"), ORE,
+          EmitRemarks);
+    } else {
+      LLVM_DEBUG(dbgs() << "Replay Inliner: Not Inlined " << Callee << " @ "
+                        << CallSiteLoc << "\n");
+      // A negative inline is conveyed by "None" Optional<InlineCost>
+      return std::make_unique<DefaultInlineAdvice>(this, CB, None, ORE,
+                                                   EmitRemarks);
+    }
+
+  // Fallback decisions
+  if (ReplaySettings.ReplayFallback ==
+      ReplayInlinerSettings::Fallback::AlwaysInline)
+    return std::make_unique<DefaultInlineAdvice>(
+        this, CB, llvm::InlineCost::getAlways("AlwaysInline Fallback"), ORE,
+        EmitRemarks);
+  else if (ReplaySettings.ReplayFallback ==
+           ReplayInlinerSettings::Fallback::NeverInline)
+    // A negative inline is conveyed by "None" Optional<InlineCost>
+    return std::make_unique<DefaultInlineAdvice>(this, CB, None, ORE,
+                                                 EmitRemarks);
+  else {
+    assert(ReplaySettings.ReplayFallback ==
+           ReplayInlinerSettings::Fallback::Original);
+    // If there's a registered original advisor, return its decision
     if (OriginalAdvisor)
       return OriginalAdvisor->getAdvice(CB);
-    return {};
   }
 
-  return std::make_unique<DefaultInlineAdvice>(this, CB, InlineRecommended, ORE,
-                                               EmitRemarks);
+  // If no decision is made above, return non-decision
+  return {};
 }

llvm/lib/Transforms/IPO/Inliner.cpp

 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/InlineOrder.h"
 #include "llvm/Analysis/LazyCallGraph.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Analysis/Utils/ImportedFunctionsInliningStatistics.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/DerivedTypes.h"

 
 static cl::opt<std::string> CGSCCInlineReplayFile(
     "cgscc-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by cgscc inlining."),
     cl::Hidden);
 
-static cl::opt<ReplayInlineScope> CGSCCInlineReplayScope(
-    "cgscc-inline-replay-scope", cl::init(ReplayInlineScope::Function),
-    cl::values(clEnumValN(ReplayInlineScope::Function, "Function",
+static cl::opt<ReplayInlinerSettings::Scope> CGSCCInlineReplayScope(
+    "cgscc-inline-replay-scope",
+    cl::init(ReplayInlinerSettings::Scope::Function),
+    cl::values(clEnumValN(ReplayInlinerSettings::Scope::Function, "Function",
                           "Replay on functions that have remarks associated "
                           "with them (default)"),
-               clEnumValN(ReplayInlineScope::Module, "Module",
+               clEnumValN(ReplayInlinerSettings::Scope::Module, "Module",
                           "Replay on the entire module")),
     cl::desc("Whether inline replay should be applied to the entire "
              "Module or just the Functions (default) that are present as "
              "callers in remarks during cgscc inlining."),
     cl::Hidden);
 
+static cl::opt<ReplayInlinerSettings::Fallback> CGSCCInlineReplayFallback(
+    "cgscc-inline-replay-fallback",
+    cl::init(ReplayInlinerSettings::Fallback::Original),
+    cl::values(
+        clEnumValN(
+            ReplayInlinerSettings::Fallback::Original, "Original",
+            "All decisions not in replay send to original advisor (default)"),
+        clEnumValN(ReplayInlinerSettings::Fallback::AlwaysInline,
+                   "AlwaysInline", "All decisions not in replay are inlined"),
+        clEnumValN(ReplayInlinerSettings::Fallback::NeverInline, "NeverInline",
+                   "All decisions not in replay are not inlined")),
+    cl::desc(
+        "How cgscc inline replay treats sites that don't come from the replay. "
+        "Original: defers to original advisor, AlwaysInline: inline all sites "
+        "not in replay, NeverInline: inline no sites not in replay"),
+    cl::Hidden);
+
+static cl::opt<CallSiteFormat::Format> CGSCCInlineReplayFormat(
+    "cgscc-inline-replay-format",
+    cl::init(CallSiteFormat::Format::LineColumnDiscriminator),
+    cl::values(
+        clEnumValN(CallSiteFormat::Format::Line, "Line", "<Line Number>"),
+        clEnumValN(CallSiteFormat::Format::LineColumn, "LineColumn",
+                   "<Line Number>:<Column Number>"),
+        clEnumValN(CallSiteFormat::Format::LineDiscriminator,
+                   "LineDiscriminator", "<Line Number>.<Discriminator>"),
+        clEnumValN(CallSiteFormat::Format::LineColumnDiscriminator,
+                   "LineColumnDiscriminator",
+                   "<Line Number>:<Column Number>.<Discriminator> (default)")),
+    cl::desc("How cgscc inline replay file is formatted"), cl::Hidden);
+
 static cl::opt<bool> InlineEnablePriorityOrder(
     "inline-enable-priority-order", cl::Hidden, cl::init(false),
     cl::desc("Enable the priority inline order for the inliner"));
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID) : CallGraphSCCPass(ID) {}
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID, bool InsertLifetime)
     : CallGraphSCCPass(ID), InsertLifetime(InsertLifetime) {}
if (!IAA) {
     // The one we would get from the MAM can be invalidated as a result of the
     // inliner's activity.
     OwnedAdvisor =
         std::make_unique<DefaultInlineAdvisor>(M, FAM, getInlineParams());
 
     if (!CGSCCInlineReplayFile.empty())
       OwnedAdvisor = getReplayInlineAdvisor(
           M, FAM, M.getContext(), std::move(OwnedAdvisor),
-          CGSCCInlineReplayFile, CGSCCInlineReplayScope,
+          ReplayInlinerSettings{CGSCCInlineReplayFile,
+                                CGSCCInlineReplayScope,
+                                CGSCCInlineReplayFallback,
+                                {CGSCCInlineReplayFormat}},
           /*EmitRemarks=*/true);
 
     return *OwnedAdvisor;
   }
   assert(IAA->getAdvisor() &&
          "Expected a present InlineAdvisorAnalysis also have an "
          "InlineAdvisor initialized");
   return *IAA->getAdvisor();
while (!Calls->empty() && Calls->front().first->getCaller() == &F) {
           UR.InlinedInternalEdges.count({&N, C})) {
         LLVM_DEBUG(dbgs() << "Skipping inlining internal SCC edge from a node "
                              "previously split out of this SCC by inlining: "
                           << F.getName() << " -> " << Callee.getName() << "\n");
         setInlineRemark(*CB, "recursive SCC split");
         continue;
       }
 
-      auto Advice = Advisor.getAdvice(*CB, OnlyMandatory);
+      std::unique_ptr<InlineAdvice> Advice =
+          Advisor.getAdvice(*CB, OnlyMandatory);
 
       // Check whether we want to inline this callsite.
-      if (!Advice || !Advice->isInliningRecommended()) {
+      if (!Advice)
+        continue;
+
+      if (!Advice->isInliningRecommended()) {
         Advice->recordUnattemptedInlining();
         continue;
       }
 
       // Setup the data structure used to plumb customization into the
       // `InlineFunction` routine.
       InlineFunctionInfo IFI(
           /*cg=*/nullptr, GetAssumptionCache, PSI,
           &FAM.getResult<BlockFrequencyAnalysis>(*(CB->getCaller())),
ModuleInlinerWrapperPass::ModuleInlinerWrapperPass(InlineParams Params,
   if (MandatoryFirst)
     PM.addPass(InlinerPass(/*OnlyMandatory*/ true));
   PM.addPass(InlinerPass());
 }
 
 PreservedAnalyses ModuleInlinerWrapperPass::run(Module &M,
                                                 ModuleAnalysisManager &MAM) {
   auto &IAA = MAM.getResult<InlineAdvisorAnalysis>(M);
-  if (!IAA.tryCreate(Params, Mode, CGSCCInlineReplayFile,
-                     CGSCCInlineReplayScope)) {
+  if (!IAA.tryCreate(Params, Mode,
+                     {CGSCCInlineReplayFile,
+                      CGSCCInlineReplayScope,
+                      CGSCCInlineReplayFallback,
+                      {CGSCCInlineReplayFormat}})) {
     M.getContext().emitError(
         "Could not setup Inlining Advisor for the requested "
         "mode and/or options");
     return PreservedAnalyses::all();
   }
 
   // We wrap the CGSCC pipeline in a devirtualization repeater. This will try
   // to detect when we devirtualize indirect calls and iterate the SCC passes

llvm/lib/Transforms/IPO/SampleProfile.cpp

 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/Analysis/InlineAdvisor.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/PostDominators.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/ReplayInlineAdvisor.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/CFG.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/DiagnosticInfo.h"
 #include "llvm/IR/Dominators.h"

 
 static cl::opt<std::string> ProfileInlineReplayFile(
     "sample-profile-inline-replay", cl::init(""), cl::value_desc("filename"),
     cl::desc(
         "Optimization remarks file containing inline remarks to be replayed "
         "by inlining from sample profile loader."),
     cl::Hidden);
 
-static cl::opt<ReplayInlineScope> ProfileInlineReplayScope(
-    "sample-profile-inline-replay-scope", cl::init(ReplayInlineScope::Function),
-    cl::values(clEnumValN(ReplayInlineScope::Function, "Function",
+static cl::opt<ReplayInlinerSettings::Scope> ProfileInlineReplayScope(
+    "sample-profile-inline-replay-scope",
+    cl::init(ReplayInlinerSettings::Scope::Function),
+    cl::values(clEnumValN(ReplayInlinerSettings::Scope::Function, "Function",
                           "Replay on functions that have remarks associated "
                           "with them (default)"),
-               clEnumValN(ReplayInlineScope::Module, "Module",
+               clEnumValN(ReplayInlinerSettings::Scope::Module, "Module",
                           "Replay on the entire module")),
     cl::desc("Whether inline replay should be applied to the entire "
              "Module or just the Functions (default) that are present as "
              "callers in remarks during sample profile inlining."),
     cl::Hidden);
 
+static cl::opt<ReplayInlinerSettings::Fallback> ProfileInlineReplayFallback(
+    "sample-profile-inline-replay-fallback",
+    cl::init(ReplayInlinerSettings::Fallback::Original),
+    cl::values(
+        clEnumValN(
+            ReplayInlinerSettings::Fallback::Original, "Original",
+            "All decisions not in replay send to original advisor (default)"),
+        clEnumValN(ReplayInlinerSettings::Fallback::AlwaysInline,
+                   "AlwaysInline", "All decisions not in replay are inlined"),
+        clEnumValN(ReplayInlinerSettings::Fallback::NeverInline, "NeverInline",
+                   "All decisions not in replay are not inlined")),
+    cl::desc("How sample profile inline replay treats sites that don't come "
+             "from the replay. Original: defers to original advisor, "
+             "AlwaysInline: inline all sites not in replay, NeverInline: "
+             "inline no sites not in replay"),
+    cl::Hidden);
+
+static cl::opt<CallSiteFormat::Format> ProfileInlineReplayFormat(
+    "sample-profile-inline-replay-format",
+    cl::init(CallSiteFormat::Format::LineColumnDiscriminator),
+    cl::values(
+        clEnumValN(CallSiteFormat::Format::Line, "Line", "<Line Number>"),
+        clEnumValN(CallSiteFormat::Format::LineColumn, "LineColumn",
+                   "<Line Number>:<Column Number>"),
+        clEnumValN(CallSiteFormat::Format::LineDiscriminator,
+                   "LineDiscriminator", "<Line Number>.<Discriminator>"),
+        clEnumValN(CallSiteFormat::Format::LineColumnDiscriminator,
+                   "LineColumnDiscriminator",
+                   "<Line Number>:<Column Number>.<Discriminator> (default)")),
+    cl::desc("How sample profile inline replay file is formatted"), cl::Hidden);
+
 static cl::opt<unsigned>
     MaxNumPromotions("sample-profile-icp-max-prom", cl::init(3), cl::Hidden,
                      cl::ZeroOrMore,
                      cl::desc("Max number of promotions for a single indirect "
                               "call callsite in sample profile loader"));
 
 static cl::opt<bool> OverwriteExistingWeights(
     "overwrite-existing-weights", cl::Hidden, cl::init(false),
bool SampleProfileLoader::doInitialization(Module &M,
   if (ProfAccForSymsInList) {
     NamesInProfile.clear();
     if (auto NameTable = Reader->getNameTable())
       NamesInProfile.insert(NameTable->begin(), NameTable->end());
     CoverageTracker.setProfAccForSymsInList(true);
   }
 
   if (FAM && !ProfileInlineReplayFile.empty()) {
     ExternalInlineAdvisor = getReplayInlineAdvisor(
-        M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr, ProfileInlineReplayFile,
-        ProfileInlineReplayScope, /*EmitRemarks=*/false);
+        M, *FAM, Ctx, /*OriginalAdvisor=*/nullptr,
+        ReplayInlinerSettings{ProfileInlineReplayFile,
+                              ProfileInlineReplayScope,
+                              ProfileInlineReplayFallback,
+                              {ProfileInlineReplayFormat}},
+        /*EmitRemarks=*/false);
   }
 
   // Apply tweaks if context-sensitive profile is available.
   if (Reader->profileIsCS()) {
     ProfileIsCS = true;
     FunctionSamples::ProfileIsCS = true;
 
     // Enable priority-base inliner and size inline by default for CSSPGO.
     if (!ProfileSizeInline.getNumOccurrences())

llvm/test/Transforms/Inline/Inputs/cgscc-inline-replay-function.txt

+remark: calls.cc:4:0: '_Z3subii' will not be inlined into 'main' with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;

llvm/test/Transforms/Inline/Inputs/cgscc-inline-replay-line.txt

+remark: calls.cc:4: '_Z3subii' will not be inlined into '_Z3sumii' with (cost=Never) at callsite _Z3sumii:1;
+remark: calls.cc:10: '_Z3sumii' inlined into 'main' with (cost=45, threshold=337) at callsite main:3;

llvm/test/Transforms/Inline/Inputs/cgscc-inline-replay.txt

+remark: calls.cc:4:0: '_Z3subii' will not be inlined into '_Z3sumii' with (cost=Never) at callsite _Z3sumii:1:0;
 remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' with (cost=45, threshold=337) at callsite main:3:0.1;
-remark: calls.cc:4:0: '_Z3subii' inlined into 'main' with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;

llvm/test/Transforms/Inline/cgscc-inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-cgscc-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
 ; RUN: opt < %s -passes=inline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
-;; Check module-scope replay inline decisions
-; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+;; Check Module scope Original fallback replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-fallback=Original -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
 
-;; Check function-scope inline replay decisions
-; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Function -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION %s
+;; Check Module scope Original fallback replay inline with 'Line' format decisions
+;; The results are not different than REPLAY-MODULE-ORIGINAL, but the replay input only contains line numbers rather than line:column.discriminator
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-line.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-fallback=Original -cgscc-inline-replay-format=Line -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope AlwaysInline fallback replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-fallback=AlwaysInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ALWAYS %s
+
+;; Check Module scope NeverInline fallback replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=Module -cgscc-inline-replay-fallback=NeverInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-NEVER %s
+
+;; Check Function scope Original fallback replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-function.txt -cgscc-inline-replay-scope=Function -cgscc-inline-replay-fallback=Original -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION-ORIGINAL %s
+
+;; Check Function scope AlwaysInline fallback replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-function.txt -cgscc-inline-replay-scope=Function -cgscc-inline-replay-fallback=AlwaysInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION-ALWAYS %s
+
+;; Check Function scope NeverInline fallback replay inline decisions
+; RUN: opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay-function.txt -cgscc-inline-replay-scope=Function -cgscc-inline-replay-fallback=NeverInline -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-FUNCTION-NEVER %s
 
 ;; Check behavior on non-existent replay file
 ; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
 
 ;; Check scope inlining errors out on non <Module|Function> inputs
 ; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-scope=function -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
 
+;; Check fallback inlining errors out on non <Original|AlwaysInline|NeverInline> inputs
+; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-fallback=original -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-FALLBACK %s
+
+;; Check format inlining errors out on non <Line|AlwayLineColumnsInline|LineDiscriminator|LineColumnDiscriminator> inputs
+; RUN: not opt < %s -passes=inline -cgscc-inline-replay=%S/Inputs/cgscc-inline-replay.txt -cgscc-inline-replay-format=line -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-FORMAT %s
+
 ; DEFAULT: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
 ; DEFAULT: '_Z3sumii' inlined into 'main' with (cost={{[-0-9]+}}
 ; DEFAULT-NOT: '_Z3subii' inlined into 'main'
 
-; REPLAY: '_Z3sumii' inlined into 'main' with (cost=always)
-; REPLAY: '_Z3subii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-ORIGINAL: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-ORIGINAL: '_Z3subii' inlined into 'main' with (cost={{[-0-9]+}}
+
+; REPLAY-MODULE-ALWAYS: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-ALWAYS: '_Z3subii' inlined into 'main' with (cost=always)
+
+; REPLAY-MODULE-NEVER: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-MODULE-NEVER-NOT: '_Z3subii' inlined into 'main'
+
+; REPLAY-FUNCTION-ORIGINAL: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION-ORIGINAL: '_Z3sumii' inlined into 'main' with (cost={{[-0-9]+}}
+
+; REPLAY-FUNCTION-ALWAYS: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION-ALWAYS: '_Z3sumii' inlined into 'main' with (cost=always)
 
-; REPLAY-FUNCTION: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
-; REPLAY-FUNCTION: '_Z3sumii' inlined into 'main' with (cost=always)
+; REPLAY-FUNCTION-NEVER: '_Z3subii' inlined into '_Z3sumii' with (cost={{[-0-9]+}}
+; REPLAY-FUNCTION-NEVER-NOT: '_Z3sumii' inlined into 'main'
 
 ; REPLAY-ERROR: error: Could not open remarks file:
 ; REPLAY-ERROR-SCOPE: for the --cgscc-inline-replay-scope option: Cannot find option named 'function'!
+; REPLAY-ERROR-FALLBACK: for the --cgscc-inline-replay-fallback option: Cannot find option named 'original'!
+; REPLAY-ERROR-FORMAT: for the --cgscc-inline-replay-format option: Cannot find option named 'line'!
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-function.txt

 remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-line.txt

+remark: calls.cc:4: '_Z3subii' will not be inlined into '_Z3sumii' to match profiling context with (cost=Never) at callsite _Z3sumii:1;
+remark: calls.cc:10: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-linecolumn.txt

+remark: calls.cc:4:0: '_Z3subii' will not be inlined into '_Z3sumii' to match profiling context with (cost=Never) at callsite _Z3sumii:1:0;
+remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay-linediscriminator.txt

+remark: calls.cc:4: '_Z3subii' will not be inlined into '_Z3sumii' to match profiling context with (cost=Never) at callsite _Z3sumii:1;
+remark: calls.cc:10: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/Inputs/inline-replay.txt

-remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
-remark: calls.cc:4:0: '_Z3subii' inlined into 'main' to match profiling context with (cost=-5, threshold=337) at callsite _Z3sumii:1:0 @ main:3:0.1;
+remark: calls.cc:4:0: '_Z3subii' will not be inlined into '_Z3sumii' to match profiling context with (cost=Never) at callsite _Z3sumii:1:0;
+remark: calls.cc:10:0: '_Z3sumii' inlined into 'main' to match profiling context with (cost=45, threshold=337) at callsite main:3:0.1;
 No newline at end of file

llvm/test/Transforms/SampleProfile/inline-replay.ll

 ;; Note that this needs new pass manager for now. Passing `-sample-profile-inline-replay` to legacy pass manager is a no-op.
 
 ;; Check baseline inline decisions
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT %s
 
 ;; Check replay inline decisions
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
 
 ;; Check baseline inline decisions with all callee counts missing
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-missing.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT-NOINLINING -allow-empty %s
 
 ;; Check replay inline decisions with all callee counts missing. The call sites should still be passed to the replay advisor and successfully get inlined as before
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-missing.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-missing.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-MISSING %s
 
 ;; Check baseline inline decisions with high threshold
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -profile-summary-hot-count=500000 -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT-NOINLINING -allow-empty %s
 
 ;; Check replay inline decisions with high threshold. The call sites should still be passed to the replay advisor and successfully get inlined as before
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -profile-summary-hot-count=500000 -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY -allow-empty %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -profile-summary-hot-count=500000 -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL -allow-empty %s
+
+;; Check Module scope Original fallback replay inline decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-fallback=Original -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope Original fallback replay inline with 'Line' format decisions
+;; The results are not different than REPLAY-MODULE-ORIGINAL, but the replay input only contains line numbers rather than line:column.discriminator
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-line.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-fallback=Original -sample-profile-inline-replay-format=Line -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope Original fallback replay inline with 'LineColumn' format decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-linecolumn.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-fallback=Original -sample-profile-inline-replay-format=LineColumn -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope Original fallback replay inline with 'LineDiscriminator' format decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-linediscriminator.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-fallback=Original -sample-profile-inline-replay-format=LineDiscriminator -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ORIGINAL %s
+
+;; Check Module scope AlwaysInline fallback replay inline decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-fallback=AlwaysInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-ALWAYS %s
+
+;; Check Module scope NeverInline fallback replay inline decisions
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=Module -sample-profile-inline-replay-fallback=NeverInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-MODULE-NEVER %s
 
 ;; Check baseline inline decisions with "inline-topdown-inline-all.prof" which inlines all sites
 ; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=DEFAULT-ALL %s
 
 ;; Check function scope replay inline decisions with "inline-topdown-inline-all.prof" and "inline-topdown-function-scope.txt" which only contains: '_Z3sumii' inlined into 'main'
 ;; 1. _Z3sumii is inlined into main, but all other inline candidates in main (e.g. _Z3subii) are not inlined
 ;; 2. Inline decisions made in other functions match default sample inlining, in this case _Z3subii is inlined into _Z3sumii
-; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function-scope.txt -sample-profile-inline-replay-scope=Function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION %s
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function.txt -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay-fallback=NeverInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION-NEVER %s
+
+;; Function scope Original fallback
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function.txt -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay-fallback=Original -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION-ORIGINAL %s
+
+;; Function scope AlwaysInline fallback
+; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown-inline-all.prof -sample-profile-inline-replay=%S/Inputs/inline-replay-function.txt -sample-profile-inline-replay-scope=Function -sample-profile-inline-replay-fallback=AlwaysInline -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ALL-FUNCTION-ALWAYS %s
 
 ;; Check behavior on non-existent replay file
 ; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR %s
 
 ;; Check scope inlining errors out on non <Module|Function> inputs
 ; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-scope=function -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline --disable-output 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-SCOPE %s
 
+;; Check fallback inlining errors out on non <Original|AlwaysInline|NeverInline> inputs
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-fallback=original -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-FALLBACK %s
+
+;; Check format inlining errors out on non <Line|LineColumn|LineDiscriminator|LineColumnDiscriminator> inputs
+; RUN: not opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/inline-topdown.prof -sample-profile-inline-replay=%S/Inputs/inline-replay.txt -sample-profile-inline-replay-format=line -sample-profile-merge-inlinee -sample-profile-top-down-load -pass-remarks=inline -S 2>&1 | FileCheck -check-prefix=REPLAY-ERROR-FORMAT %s
+
+
 ; DEFAULT: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-NOT: '_Z3subii' inlined into 'main'
 
-; REPLAY: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
-; REPLAY: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
-; REPLAY-NOT: '_Z3subii' inlined into '_Z3sumii'
+; REPLAY-MODULE-MISSING: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-MISSING-NOT: inlined into
+
+; REPLAY-MODULE-ORIGINAL: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-ORIGINAL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-MODULE-ALWAYS: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-ALWAYS: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
+
+; REPLAY-MODULE-NEVER: '_Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-MODULE-NEVER-NOT: '_Z3subii' inlined into 'main'
 
 ; DEFAULT-NOINLINING-NOT: inlined into
 
 ; DEFAULT-ALL: '_Z3sumii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-ALL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
 ; DEFAULT-ALL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 
-; REPLAY-ALL-FUNCTION : _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
-; REPLAY-ALL-FUNCTION-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
-; REPLAY-ALL-FUNCTION: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION-NEVER: _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-NEVER-NOT: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION-NEVER: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ALL-FUNCTION-ORIGINAL: _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-ORIGINAL: '_Z3subii' inlined into 'main' to match profiling context with (cost={{[-0-9]+}}
+; REPLAY-ALL-FUNCTION-ORIGINAL: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
+
+; REPLAY-ALL-FUNCTION-ALWAYS: _Z3sumii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-ALWAYS: '_Z3subii' inlined into 'main' to match profiling context with (cost=always)
+; REPLAY-ALL-FUNCTION-ALWAYS: '_Z3subii' inlined into '_Z3sumii' to match profiling context with (cost={{[-0-9]+}}
 
 ; REPLAY-ERROR: error: Could not open remarks file:
 ; REPLAY-ERROR-SCOPE: for the --sample-profile-inline-replay-scope option: Cannot find option named 'function'!
+; REPLAY-ERROR-FALLBACK: for the --sample-profile-inline-replay-fallback option: Cannot find option named 'original'!
+; REPLAY-ERROR-FORMAT: for the --sample-profile-inline-replay-format option: Cannot find option named 'line'!
 
 @.str = private unnamed_addr constant [11 x i8] c"sum is %d\0A\00", align 1
 
 define i32 @_Z3sumii(i32 %x, i32 %y) #0 !dbg !6 {
 entry:
   %x.addr = alloca i32, align 4
   %y.addr = alloca i32, align 4
   store i32 %x, i32* %x.addr, align 4
   store i32 %y, i32* %y.addr, align 4