[Inlining] Introduce -enable-npm-pgo-inline-deferral

[kazu]

Summary

Experiments show that inline deferral past pre-inlining slightly
pessimizes the performance.

This patch introduces an option to control inline deferral during PGO.
The option defaults to true for now (that is, NFC).

[davidxl]

llvm/include/llvm/Analysis/InlineAdvisor.h

202

give it a default value (true)?

[kazu]

Made the last parameter of shouldInline optional.

[kazu]

PTAL Thanks!

[davidxl]

lgtm

[wenlei]

Experiments show that inline deferral past pre-inlining slightly pessimizes the performance.

What are the performance numbers in terms of % you saw when deferral is disabled, if it's ok to share? Was it from Instr. PGO or Sample FDO? Asking because the new switch seems to cover both.

[kazu]

In D80776#2073762, @wenlei wrote:

Experiments show that inline deferral past pre-inlining slightly pessimizes the performance.

What are the performance numbers in terms of % you saw when deferral is disabled, if it's ok to share? Was it from Instr. PGO or Sample FDO? Asking because the new switch seems to cover both.

I am seeing 0.2% to 0.3% improvement in one of our internal benchmarks. Please see https://reviews.llvm.org/D81216 for more numbers. My experiment is from Instrumentation-based PGO.

llvm/include/llvm/Analysis/InlineAdvisor.h

 // pass manager inliner.
 
 /// Return the cost only if the inliner should attempt to inline at the given
 /// CallSite. If we return the cost, we will emit an optimisation remark later
 /// using that cost, so we won't do so from this function. Return None if
 /// inlining should not be attempted.
 Optional<InlineCost>
 shouldInline(CallBase &CB, function_ref<InlineCost(CallBase &CB)> GetInlineCost,
-             OptimizationRemarkEmitter &ORE);
+             OptimizationRemarkEmitter &ORE, bool EnableDeferral);
 
 /// Emit ORE message.
 void emitInlinedInto(OptimizationRemarkEmitter &ORE, DebugLoc DLoc,
                      const BasicBlock *Block, const Function &Callee,
                      const Function &Caller, const InlineCost &IC);
 
 /// Set the inline-remark attribute.
 void setInlineRemark(CallBase &CB, StringRef Message);
 
 /// Utility for extracting the inline cost message to a string.
 std::string inlineCostStr(const InlineCost &IC);
 } // namespace llvm
 #endif // LLVM_INLINEADVISOR_H_

llvm/include/llvm/Analysis/InlineCost.h

struct InlineParams {
   /// entry.
   Optional<int> LocallyHotCallSiteThreshold;
 
   /// Threshold to use when the callsite is considered cold.
   Optional<int> ColdCallSiteThreshold;
 
   /// Compute inline cost even when the cost has exceeded the threshold.
   Optional<bool> ComputeFullInlineCost;
+
+  /// Indicate whether we should allow inline deferral.
+  Optional<bool> EnableDeferral = true;
 };
 
 /// Generate the parameters to tune the inline cost analysis based only on the
 /// commandline options.
 InlineParams getInlineParams();
 
 /// Generate the parameters to tune the inline cost analysis based on command
 /// line options. If -inline-threshold option is not explicitly passed,

llvm/lib/Analysis/InlineAdvisor.cpp

auto GetInlineCost = [&](CallBase &CB) {
     Function &Callee = *CB.getCalledFunction();
     auto &CalleeTTI = FAM.getResult<TargetIRAnalysis>(Callee);
     bool RemarksEnabled =
         Callee.getContext().getDiagHandlerPtr()->isMissedOptRemarkEnabled(
             DEBUG_TYPE);
     return getInlineCost(CB, Params, CalleeTTI, GetAssumptionCache, GetTLI,
                          GetBFI, PSI, RemarksEnabled ? &ORE : nullptr);
   };
-  auto OIC = llvm::shouldInline(CB, GetInlineCost, ORE);
+  auto OIC = llvm::shouldInline(CB, GetInlineCost, ORE,
+                                Params.EnableDeferral.hasValue() &&
+                                    Params.EnableDeferral.getValue());
   return std::make_unique<DefaultInlineAdvice>(this, CB, OIC, ORE);
 }
 
 InlineAdvice::InlineAdvice(InlineAdvisor *Advisor, CallBase &CB,
                            bool IsInliningRecommended)
     : Advisor(Advisor), Caller(CB.getCaller()), Callee(CB.getCalledFunction()),
       IsInliningRecommended(IsInliningRecommended) {}
 

 
 /// Return the cost only if the inliner should attempt to inline at the given
 /// CallSite. If we return the cost, we will emit an optimisation remark later
 /// using that cost, so we won't do so from this function. Return None if
 /// inlining should not be attempted.
 Optional<InlineCost>
 llvm::shouldInline(CallBase &CB,
                    function_ref<InlineCost(CallBase &CB)> GetInlineCost,
-                   OptimizationRemarkEmitter &ORE) {
+                   OptimizationRemarkEmitter &ORE, bool EnableDeferral) {
   using namespace ore;
 
   InlineCost IC = GetInlineCost(CB);
   Instruction *Call = &CB;
   Function *Callee = CB.getCalledFunction();
   Function *Caller = CB.getCaller();
 
   if (IC.isAlways()) {
if (IC.isNever()) {
                << IC;
       });
     }
     setInlineRemark(CB, inlineCostStr(IC));
     return None;
   }
 
   int TotalSecondaryCost = 0;
-  if (shouldBeDeferred(Caller, IC, TotalSecondaryCost, GetInlineCost)) {
+  if (EnableDeferral &&
+      shouldBeDeferred(Caller, IC, TotalSecondaryCost, GetInlineCost)) {
     LLVM_DEBUG(dbgs() << "    NOT Inlining: " << CB
                       << " Cost = " << IC.getCost()
                       << ", outer Cost = " << TotalSecondaryCost << '\n');
     ORE.emit([&]() {
       return OptimizationRemarkMissed(DEBUG_TYPE, "IncreaseCostInOtherContexts",
                                       Call)
              << "Not inlining. Cost of inlining " << NV("Callee", Callee)
              << " increases the cost of inlining " << NV("Caller", Caller)

llvm/lib/Analysis/InlineCost.cpp

Optional<int> llvm::getInliningCostEstimate(
   const InlineParams Params = {/* DefaultThreshold*/ 0,
                                /*HintThreshold*/ {},
                                /*ColdThreshold*/ {},
                                /*OptSizeThreshold*/ {},
                                /*OptMinSizeThreshold*/ {},
                                /*HotCallSiteThreshold*/ {},
                                /*LocallyHotCallSiteThreshold*/ {},
                                /*ColdCallSiteThreshold*/ {},
-                               /* ComputeFullInlineCost*/ true};
+                               /*ComputeFullInlineCost*/ true,
+                               /*EnableDeferral*/ true};
 
   InlineCostCallAnalyzer CA(*Call.getCalledFunction(), Call, Params, CalleeTTI,
                             GetAssumptionCache, GetBFI, PSI, ORE, true,
                             /*IgnoreThreshold*/ true);
   auto R = CA.analyze();
   if (!R.isSuccess())
     return None;
   return CA.getCost();

llvm/lib/Passes/PassBuilder.cpp

 static cl::opt<bool>
     EnableCHR("enable-chr-npm", cl::init(true), cl::Hidden,
               cl::desc("Enable control height reduction optimization (CHR)"));
 
 static cl::opt<bool> EnableCallGraphProfile(
     "enable-npm-call-graph-profile", cl::init(true), cl::Hidden,
     cl::desc("Enable call graph profile pass for the new PM (default = on)"));
 
+/// Flag to enable inline deferral during PGO.
+static cl::opt<bool>
+    EnablePGOInlineDeferral("enable-npm-pgo-inline-deferral", cl::init(true),
+                            cl::Hidden,
+                            cl::desc("Enable inline deferral during PGO"));
+
 PipelineTuningOptions::PipelineTuningOptions() {
   LoopInterleaving = true;
   LoopVectorization = true;
   SLPVectorization = false;
   LoopUnrolling = true;
   ForgetAllSCEVInLoopUnroll = ForgetSCEVInLoopUnroll;
   Coroutines = false;
   LicmMssaOptCap = SetLicmMssaOptCap;

 ModuleInlinerWrapperPass
 PassBuilder::buildInlinerPipeline(OptimizationLevel Level, ThinLTOPhase Phase,
                                   bool DebugLogging) {
   InlineParams IP = getInlineParamsFromOptLevel(Level);
   if (Phase == PassBuilder::ThinLTOPhase::PreLink && PGOOpt &&
       PGOOpt->Action == PGOOptions::SampleUse)
     IP.HotCallSiteThreshold = 0;
 
+  if (PGOOpt)
+    IP.EnableDeferral = EnablePGOInlineDeferral;
+
   ModuleInlinerWrapperPass MIWP(IP, DebugLogging, UseInlineAdvisor,
                                 MaxDevirtIterations);
 
   // Require the GlobalsAA analysis for the module so we can query it within
   // the CGSCC pipeline.
   MIWP.addRequiredModuleAnalysis<GlobalsAA>();
 
   // Require the ProfileSummaryAnalysis for the module so we can query it within

llvm/lib/Transforms/IPO/Inliner.cpp

for (unsigned CSi = 0; CSi != CallSites.size(); ++CSi) {
         }
       }
 
       // FIXME for new PM: because of the old PM we currently generate ORE and
       // in turn BFI on demand.  With the new PM, the ORE dependency should
       // just become a regular analysis dependency.
       OptimizationRemarkEmitter ORE(Caller);
 
-      auto OIC = shouldInline(CB, GetInlineCost, ORE);
+      auto OIC = shouldInline(CB, GetInlineCost, ORE, true);
       // If the policy determines that we should inline this function,
       // delete the call instead.
       if (!OIC)
         continue;
 
       // If this call site is dead and it is to a readonly function, we should
       // just delete the call instead of trying to inline it, regardless of
       // size.  This happens because IPSCCP propagates the result out of the