[InlineCost] Relax bonus restrictions on uninlinable functions

[george.burgess.iv]

Summary

We currently ignore inlining bonuses sometimes (importantly to me, LastCallToStaticBonus) in order to make the caller of a function more inlinable. This appears suboptimal if the caller can't be inlined in the first place.

This is an attempt to fix that suboptimality (which showed up for me most prominently with the new PM), though it's not quite finished. Before I plumb everything through, I'd like to know if anyone's strongly against a change like this.

Importantly, I can see the following downsides:

• This patch makes our inline cost depend on state of the caller, which is generally discouraged, since we try to cache these things.
• This patch basically causes the inliner to inline differently depending on whether we're LTO'ing or not, which AFAICT is new.

The last bullet is the case because, in a traditional build, I don't think we care about how inlineable exportedFunction is below, assuming it has no users in its TU:

static void helper() {
  // large body
}

void exportedFunction(bool i) {
  if (UNLIKELY(i))
    helper();
}

OTOH, with ThinLTO or LTO enabled, we might discover uses of exportedFunction during our link, so the inlinability of it starts to matter.

Thoughts appreciated. :)

[mtrofin]

On the first bullet, I'm trying to find where InlineCost values are cached. Are we doing that?

[george.burgess.iv]

On the first bullet, I'm trying to find where InlineCost values are cached. Are we doing that?

I don't actually know myself; I got that from above the comment for CandidateCall: "The candidate callsite being analyzed. Please do not use this to do analysis in the caller function; we want the inline cost query to be easily cacheable. Instead, use the cover function paramHasAttr."

[davidxl]

This is sort of extension of the 'shouldBeDeferred' check in Inliner.cpp, except that here the caller may not actually be inlined. Blindly eliminating bonus may preventing the callee from being inlined while not actually enabling the caller to be inlined.

[mtrofin]

In D73994#1858006, @george.burgess.iv wrote:

On the first bullet, I'm trying to find where InlineCost values are cached. Are we doing that?

I don't actually know myself; I got that from above the comment for CandidateCall: "The candidate callsite being analyzed. Please do not use this to do analysis in the caller function; we want the inline cost query to be easily cacheable. Instead, use the cover function paramHasAttr."

Yup - I also don't know of any location caching the cost. I wonder if this was a nerver-realized goal, and whether it is abandoned. May be worth checking with the author of the comment?

[george.burgess.iv]

Yup - I also don't know of any location caching the cost. I wonder if this was a nerver-realized goal, and whether it is abandoned. May be worth checking with the author of the comment?

Good call -- looks like the comment was added in 9b5c9580e384 around 5 years ago. The commit message says "We're not currently [caching inline costs], but initial results on LTO indicate this will quickly become important," so this sounds like a nice-to-have that was never had.

Removed the cacheable bit of that comment, since I think it's a bit misleading if there's no motion on it.

This is sort of extension of the 'shouldBeDeferred' check in Inliner.cpp, except that here the caller may not actually be inlined. Blindly eliminating bonus may preventing the callee from being inlined while not actually enabling the caller to be inlined.

Oof, yeah, these look similar, but it's not clear to me if there's a good way to unify them as a part of this patch (or lift this out to there). Lemme poke it a bit and see :)

[eraman]

DisallowAllBonuses is called only when the callsite or the callee entry is cold. The only reason to make it conditional on caller's inlinability would be for code size improvement. Among the 3 bonuses disabled in DisallowAllBonuses, only the LastCallToStaticBonus is useful to allow for code size reduction. You don't want to give a single BB bonus or a vector bonus for a cold callee (or a callee at a cold callsite) whose caller is unlikely to be inlined.

[george.burgess.iv]

Updated to plumb through the "are we doing ThinLTO || prelink LTO" bit in a way that makes sense to me. Suggestions for potentially better approaches is welcome :)

Among the 3 bonuses disabled in DisallowAllBonuses, only the LastCallToStaticBonus is useful to allow for code size reduction [...].

Fixed; thanks!

[george.burgess.iv]

Also, RE

Oof, yeah, these look similar, but it's not clear to me if there's a good way to unify them as a part of this patch (or lift this out to there). Lemme poke it a bit and see :)

Unfortunately, I couldn't figure out a clean way to do this, so I added a FIXME for now

[george.burgess.iv]

ping :)

[davidxl]

I have concerns on so much plumbing changes to make the small heuristic change.

On the other hand, can you first extract the pass manager builder changes (i.e, passing Phase kind) into a separate patch? That one seems independent.

[Gerolf]

Especially for inliner changes I suggest a) sharing performance numbers on standard benchmarks with O2 & O2 (thin) LTO, and b) offering a flag to toggle individual changes. Performance here includes run-time performance, compile-time, and code size. A simple change can have detrimental impact on any of key metrics folks might be watching, and that that impact could be different depending on the targets.

[george.burgess.iv]

I have concerns on so much plumbing changes to make the small heuristic change.

Agreed that that's discouraging. Lemme take numbers and see what this does for us. If this still looks promising, I'll split as requested.

sharing performance numbers on standard benchmarks with O2 & O2 (thin) LTO,

Thanks for the feedback! I'm not sure what benchmarks are considered standard. Do we have documentation on this, or might you have specific ones in mind I can try this on?

[george.burgess.iv]

Friendly ping Gerolf (or anyone else who has opinions on standardish benchmarks) :)

[davidxl]

SPEC2006/2017 will be a good public benchmarks. Clang performance/size (with self build with and without PGO) is another one. Internal benchmark results are also meaningful.

clang/lib/CodeGen/BackendUtil.cpp

if (CodeGenOpts.OptimizationLevel <= 1) {
     PMBuilder.Inliner = createAlwaysInlinerLegacyPass(InsertLifetimeIntrinsics);
   } else {
     // We do not want to inline hot callsites for SamplePGO module-summary build
     // because profile annotation will happen again in ThinLTO backend, and we
     // want the IR of the hot path to match the profile.
     PMBuilder.Inliner = createFunctionInliningPass(
         CodeGenOpts.OptimizationLevel, CodeGenOpts.OptimizeSize,
         (!CodeGenOpts.SampleProfileFile.empty() &&
-         CodeGenOpts.PrepareForThinLTO));
+         CodeGenOpts.PrepareForThinLTO),
+        /*ExternalFunctionUseListsAreIncomplete=*/
+        CodeGenOpts.PrepareForThinLTO || CodeGenOpts.PrepareForLTO);
   }
 
   PMBuilder.OptLevel = CodeGenOpts.OptimizationLevel;
   PMBuilder.SizeLevel = CodeGenOpts.OptimizeSize;
   PMBuilder.SLPVectorize = CodeGenOpts.VectorizeSLP;
   PMBuilder.LoopVectorize = CodeGenOpts.VectorizeLoop;
 
   PMBuilder.DisableUnrollLoops = !CodeGenOpts.UnrollLoops;

llvm/include/llvm/Analysis/InlineCost.h

 /// Note that a default threshold is passed into this function. This threshold
 /// could be modified based on callsite's properties and only costs below this
 /// new threshold are computed with any accuracy. The new threshold can be
 /// used to bound the computation necessary to determine whether the cost is
 /// sufficiently low to warrant inlining.
 ///
 /// Also note that calling this function *dynamically* computes the cost of
 /// inlining the callsite. It is an expensive, heavyweight call.
-InlineCost getInlineCost(
-    CallBase &Call, const InlineParams &Params, TargetTransformInfo &CalleeTTI,
+InlineCost
+getInlineCost(CallBase &Call, const InlineParams &Params,
+              TargetTransformInfo &CalleeTTI,
               std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
               Optional<function_ref<BlockFrequencyInfo &(Function &)>> GetBFI,
-    ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE = nullptr);
+              ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE = nullptr,
+              bool CallerInlinabilityMatters = true);
 
 /// Get an InlineCost with the callee explicitly specified.
 /// This allows you to calculate the cost of inlining a function via a
 /// pointer. This behaves exactly as the version with no explicit callee
 /// parameter in all other respects.
 //
 InlineCost
 getInlineCost(CallBase &Call, Function *Callee, const InlineParams &Params,
               TargetTransformInfo &CalleeTTI,
               std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
               Optional<function_ref<BlockFrequencyInfo &(Function &)>> GetBFI,
-              ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE);
+              ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE,
+              bool CallerInlinabilityMatters = true);
 
 /// Minimal filter to detect invalid constructs for inlining.
 InlineResult isInlineViable(Function &Callee);
 } // namespace llvm
 
 #endif

llvm/include/llvm/Passes/PassBuilder.h

public:
   /// Note that \p Level cannot be `O0` here. The pipelines produced are
   /// only intended for use when attempting to optimize code. If frontends
   /// require some transformations for semantic reasons, they should explicitly
   /// build them.
   ///
   /// \p Phase indicates the current ThinLTO phase.
   ModulePassManager
   buildModuleSimplificationPipeline(OptimizationLevel Level,
-                                    ThinLTOPhase Phase,
-                                    bool DebugLogging = false);
+                                    bool DebugLogging = false,
+                                    ThinLTOPhase Phase = ThinLTOPhase::None);
 
   /// Construct the core LLVM module optimization pipeline.
   ///
   /// This pipeline focuses on optimizing the execution speed of the IR. It
   /// uses cost modeling and thresholds to balance code growth against runtime
   /// improvements. It includes vectorization and other information destroying
   /// transformations. It also cannot generally be run repeatedly on a module
   /// without potentially seriously regressing either runtime performance of
   /// the code or serious code size growth.
   ///
   /// Note that \p Level cannot be `O0` here. The pipelines produced are
   /// only intended for use when attempting to optimize code. If frontends
   /// require some transformations for semantic reasons, they should explicitly
   /// build them.
-  ModulePassManager buildModuleOptimizationPipeline(OptimizationLevel Level,
-                                                    bool DebugLogging = false,
-                                                    bool LTOPreLink = false);
+  ModulePassManager
+  buildModuleOptimizationPipeline(OptimizationLevel Level,
+                                  ThinLTOPhase Phase = ThinLTOPhase::None);
 
   /// Build a per-module default optimization pipeline.
   ///
   /// This provides a good default optimization pipeline for per-module
   /// optimization and code generation without any link-time optimization. It
   /// typically correspond to frontend "-O[123]" options for optimization
   /// levels \c O1, \c O2 and \c O3 resp.
   ///
   /// Note that \p Level cannot be `O0` here. The pipelines produced are
   /// only intended for use when attempting to optimize code. If frontends
   /// require some transformations for semantic reasons, they should explicitly
   /// build them.
-  ModulePassManager buildPerModuleDefaultPipeline(OptimizationLevel Level,
-                                                  bool DebugLogging = false,
-                                                  bool LTOPreLink = false);
+  ModulePassManager
+  buildPerModuleDefaultPipeline(OptimizationLevel Level,
+                                ThinLTOPhase Phase = ThinLTOPhase::None);
 
   /// Build a pre-link, ThinLTO-targeting default optimization pipeline to
   /// a pass manager.
   ///
   /// This adds the pre-link optimizations tuned to prepare a module for
   /// a ThinLTO run. It works to minimize the IR which needs to be analyzed
   /// without making irreversible decisions which could be made better during
   /// the LTO run.
Error parseCGSCCPassPipeline(CGSCCPassManager &CGPM,
                                bool VerifyEachPass, bool DebugLogging);
   Error parseModulePassPipeline(ModulePassManager &MPM,
                                 ArrayRef<PipelineElement> Pipeline,
                                 bool VerifyEachPass, bool DebugLogging);
 
   void addPGOInstrPasses(ModulePassManager &MPM, bool DebugLogging,
                          OptimizationLevel Level, bool RunProfileGen, bool IsCS,
                          std::string ProfileFile,
-                         std::string ProfileRemappingFile);
+                         std::string ProfileRemappingFile,
+                         bool ExternalFunctionUseListsAreIncomplete);
   void invokePeepholeEPCallbacks(FunctionPassManager &, OptimizationLevel);
 
   // Extension Point callbacks
   SmallVector<std::function<void(FunctionPassManager &, OptimizationLevel)>, 2>
       PeepholeEPCallbacks;
   SmallVector<std::function<void(LoopPassManager &, OptimizationLevel)>, 2>
       LateLoopOptimizationsEPCallbacks;
   SmallVector<std::function<void(LoopPassManager &, OptimizationLevel)>, 2>

llvm/include/llvm/Transforms/IPO.h

 /// to inline direct function calls to small functions.
 ///
 /// The Threshold can be passed directly, or asked to be computed from the
 /// given optimization and size optimization arguments.
 ///
 /// The -inline-threshold command line option takes precedence over the
 /// threshold given here.
 Pass *createFunctionInliningPass();
-Pass *createFunctionInliningPass(int Threshold);
-Pass *createFunctionInliningPass(unsigned OptLevel, unsigned SizeOptLevel,
-                                 bool DisableInlineHotCallSite);
-Pass *createFunctionInliningPass(InlineParams &Params);
+Pass *
+createFunctionInliningPass(int Threshold,
+                           bool ExternalFunctionUseListsAreIncomplete = false);
+Pass *
+createFunctionInliningPass(unsigned OptLevel, unsigned SizeOptLevel,
+                           bool DisableInlineHotCallSite,
+                           bool ExternalFunctionUseListsAreIncomplete = false);
+Pass *
+createFunctionInliningPass(InlineParams &Params,
+                           bool ExternalFunctionUseListsAreIncomplete = false);
 
 //===----------------------------------------------------------------------===//
 /// createPruneEHPass - Return a new pass object which transforms invoke
 /// instructions into calls, if the callee can _not_ unwind the stack.
 ///
 Pass *createPruneEHPass();
 
 //===----------------------------------------------------------------------===//

llvm/include/llvm/Transforms/IPO/Inliner.h

 /// It should be noted that the legacy inliners do considerably more than this
 /// inliner pass does. They provide logic for manually merging allocas, and
 /// doing considerable DCE including the DCE of dead functions. This pass makes
 /// every attempt to be simpler. DCE of functions requires complex reasoning
 /// about comdat groups, etc. Instead, it is expected that other more focused
 /// passes be composed to achieve the same end result.
 class InlinerPass : public PassInfoMixin<InlinerPass> {
 public:
-  InlinerPass(InlineParams Params = getInlineParams())
-      : Params(std::move(Params)) {}
+  InlinerPass(InlineParams Params = getInlineParams(),
+              bool ExternalFunctionUseListsAreIncomplete = false)
+      : Params(std::move(Params)), ExternalFunctionUseListsAreIncomplete(
+                                       ExternalFunctionUseListsAreIncomplete) {}
   ~InlinerPass();
-  InlinerPass(InlinerPass &&Arg)
-      : Params(std::move(Arg.Params)),
-        ImportedFunctionsStats(std::move(Arg.ImportedFunctionsStats)) {}
+  InlinerPass(InlinerPass &&Arg) = default;
 
   PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
                         LazyCallGraph &CG, CGSCCUpdateResult &UR);
 
 private:
   InlineParams Params;
   std::unique_ptr<ImportedFunctionsInliningStatistics> ImportedFunctionsStats;
+  // Assume that new references can't appear on otherwise-unreferenced `extern`
+  // functions. This isn't the case in e.g., ThinLTO.
+  bool ExternalFunctionUseListsAreIncomplete;
 };
 
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_IPO_INLINER_H

llvm/lib/Analysis/InlineCost.cpp

protected:
   Function &F;
 
   // Cache the DataLayout since we use it a lot.
   const DataLayout &DL;
 
   /// The OptimizationRemarkEmitter available for this compilation.
   OptimizationRemarkEmitter *ORE;
 
-  /// The candidate callsite being analyzed. Please do not use this to do
-  /// analysis in the caller function; we want the inline cost query to be
-  /// easily cacheable. Instead, use the cover function paramHasAttr.
+  /// The candidate callsite being analyzed.
   CallBase &CandidateCall;
 
   /// Extension points for handling callsite features.
   /// Called after a basic block was analyzed.
   virtual void onBlockAnalyzed(const BasicBlock *BB) {}
 
   /// Called at the end of the analysis of the callsite. Return the outcome of
   /// the analysis, i.e. 'InlineResult(true)' if the inlining may happen, or
class InlineCostCallAnalyzer final : public CallAnalyzer {
 
   /// Upper bound for the inlining cost. Bonuses are being applied to account
   /// for speculative "expected profit" of the inlining decision.
   int Threshold = 0;
 
   /// Attempt to evaluate indirect calls to boost its inline cost.
   const bool BoostIndirectCalls;
 
+  /// If true, inlining may be more conservative to take the caller's
+  /// inlineability into account.
+  const bool CallerInlinabilityMatters;
+
   /// Inlining cost measured in abstract units, accounts for all the
   /// instructions expected to be executed for a given function invocation.
   /// Instructions that are statically proven to be dead based on call-site
   /// arguments are not counted here.
   int Cost = 0;
 
   bool SingleBB = true;
 
class InlineCostCallAnalyzer final : public CallAnalyzer {
   }
 
 public:
   InlineCostCallAnalyzer(
       const TargetTransformInfo &TTI,
       std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
       Optional<function_ref<BlockFrequencyInfo &(Function &)>> &GetBFI,
       ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE, Function &Callee,
-      CallBase &Call, const InlineParams &Params, bool BoostIndirect = true)
+      CallBase &Call, const InlineParams &Params, bool BoostIndirect = true,
+      bool CallerInlinabilityMatters = true)
       : CallAnalyzer(TTI, GetAssumptionCache, GetBFI, PSI, ORE, Callee, Call),
         ComputeFullInlineCost(OptComputeFullInlineCost ||
                               Params.ComputeFullInlineCost || ORE),
         Params(Params), Threshold(Params.DefaultThreshold),
-        BoostIndirectCalls(BoostIndirect) {}
+        BoostIndirectCalls(BoostIndirect),
+        CallerInlinabilityMatters(CallerInlinabilityMatters) {}
   void dump();
 
   virtual ~InlineCostCallAnalyzer() {}
   int getThreshold() { return Threshold; }
   int getCost() { return Cost; }
 };
 } // namespace
 
void InlineCostCallAnalyzer::updateThreshold(CallBase &Call, Function &Callee) {
   // guaranteed to reduce code size.
   //
   // These bonus percentages may be set to 0 based on properties of the caller
   // and the callsite.
   int SingleBBBonusPercent = 50;
   int VectorBonusPercent = TTI.getInlinerVectorBonusPercent();
   int LastCallToStaticBonus = InlineConstants::LastCallToStaticBonus;
 
-  // Lambda to set all the above bonus and bonus percentages to 0.
-  auto DisallowAllBonuses = [&]() {
+  // Lambda to set all the above bonus and bonus percentages to 0, if doing so
+  // might help us inline the caller elsewhere.
+  auto SetBonusesForColdCalee = [&]() {
+    // Cold callsites should be kept small; disallow their bonuses.
     SingleBBBonusPercent = 0;
     VectorBonusPercent = 0;
+
+    // While keeping LastCallToStaticBonus might result in code size reduction,
+    // it can cause the size of the caller to increase, which may prevent it
+    // from being inlined.
+    //
+    // FIXME: This is logically a part of our `shouldBeDeferred` logic in the
+    // main `Inliner` pass. Figuring out how to hoist some of this there (or
+    // sink it here, as the note there mentions) might be nice.
+    if (CallerInlinabilityMatters)
       LastCallToStaticBonus = 0;
   };
 
   // Use the OptMinSizeThreshold or OptSizeThreshold knob if they are available
   // and reduce the threshold if the caller has the necessary attribute.
   if (Caller->hasMinSize()) {
     Threshold = MinIfValid(Threshold, Params.OptMinSizeThreshold);
     // For minsize, we want to disable the single BB bonus and the vector
     // bonuses, but not the last-call-to-static bonus. Inlining the last call to
if (!Caller->hasOptSize() && HotCallSiteThreshold) {
       LLVM_DEBUG(dbgs() << "Hot callsite.\n");
       // FIXME: This should update the threshold only if it exceeds the
       // current threshold, but AutoFDO + ThinLTO currently relies on this
       // behavior to prevent inlining of hot callsites during ThinLTO
       // compile phase.
       Threshold = HotCallSiteThreshold.getValue();
     } else if (isColdCallSite(Call, CallerBFI)) {
       LLVM_DEBUG(dbgs() << "Cold callsite.\n");
-      // Do not apply bonuses for a cold callsite including the
-      // LastCallToStatic bonus. While this bonus might result in code size
-      // reduction, it can cause the size of a non-cold caller to increase
-      // preventing it from being inlined.
-      DisallowAllBonuses();
+      SetBonusesForColdCalee();
       Threshold = MinIfValid(Threshold, Params.ColdCallSiteThreshold);
     } else if (PSI) {
       // Use callee's global profile information only if we have no way of
       // determining this via callsite information.
       if (PSI->isFunctionEntryHot(&Callee)) {
         LLVM_DEBUG(dbgs() << "Hot callee.\n");
         // If callsite hotness can not be determined, we may still know
         // that the callee is hot and treat it as a weaker hint for threshold
         // increase.
         Threshold = MaxIfValid(Threshold, Params.HintThreshold);
       } else if (PSI->isFunctionEntryCold(&Callee)) {
         LLVM_DEBUG(dbgs() << "Cold callee.\n");
-        // Do not apply bonuses for a cold callee including the
-        // LastCallToStatic bonus. While this bonus might result in code size
-        // reduction, it can cause the size of a non-cold caller to increase
-        // preventing it from being inlined.
-        DisallowAllBonuses();
+        SetBonusesForColdCalee();
         Threshold = MinIfValid(Threshold, Params.ColdThreshold);
       }
     }
   }
 
   // Finally, take the target-specific inlining threshold multiplier into
   // account.
   Threshold *= TTI.getInliningThresholdMultiplier();
int llvm::getCallsiteCost(CallBase &Call, const DataLayout &DL) {
   Cost += InlineConstants::InstrCost + InlineConstants::CallPenalty;
   return Cost;
 }
 
 InlineCost llvm::getInlineCost(
     CallBase &Call, const InlineParams &Params, TargetTransformInfo &CalleeTTI,
     std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
     Optional<function_ref<BlockFrequencyInfo &(Function &)>> GetBFI,
-    ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE) {
+    ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE,
+    bool CallerInlinabilityMatters) {
   return getInlineCost(Call, Call.getCalledFunction(), Params, CalleeTTI,
-                       GetAssumptionCache, GetBFI, PSI, ORE);
+                       GetAssumptionCache, GetBFI, PSI, ORE,
+                       CallerInlinabilityMatters);
 }
 
 InlineCost llvm::getInlineCost(
     CallBase &Call, Function *Callee, const InlineParams &Params,
     TargetTransformInfo &CalleeTTI,
     std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
     Optional<function_ref<BlockFrequencyInfo &(Function &)>> GetBFI,
-    ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE) {
+    ProfileSummaryInfo *PSI, OptimizationRemarkEmitter *ORE,
+    bool CallerInlinabilityMatters) {
 
   // Cannot inline indirect calls.
   if (!Callee)
     return llvm::InlineCost::getNever("indirect call");
 
   // Never inline calls with byval arguments that does not have the alloca
   // address space. Since byval arguments can be replaced with a copy to an
   // alloca, the inlined code would need to be adjusted to handle that the
InlineCost llvm::getInlineCost(
   // Don't inline call sites marked noinline.
   if (Call.isNoInline())
     return llvm::InlineCost::getNever("noinline call site attribute");
 
   LLVM_DEBUG(llvm::dbgs() << "      Analyzing call of " << Callee->getName()
                           << "... (caller:" << Caller->getName() << ")\n");
 
   InlineCostCallAnalyzer CA(CalleeTTI, GetAssumptionCache, GetBFI, PSI, ORE,
-                            *Callee, Call, Params);
+                            *Callee, Call, Params, /*BoostIndirect=*/true,
+                            CallerInlinabilityMatters);
   InlineResult ShouldInline = CA.analyze();
 
   LLVM_DEBUG(CA.dump());
 
   // Check if there was a reason to force inlining or no inlining.
   if (!ShouldInline.isSuccess() && CA.getCost() < CA.getThreshold())
     return InlineCost::getNever(ShouldInline.getFailureReason());
   if (ShouldInline.isSuccess() && CA.getCost() >= CA.getThreshold())

llvm/lib/Passes/PassBuilder.cpp

PassBuilder::buildFunctionSimplificationPipeline(OptimizationLevel Level,
   if (EnableCHR && Level == OptimizationLevel::O3 && PGOOpt &&
       (PGOOpt->Action == PGOOptions::IRUse ||
        PGOOpt->Action == PGOOptions::SampleUse))
     FPM.addPass(ControlHeightReductionPass());
 
   return FPM;
 }
 
-void PassBuilder::addPGOInstrPasses(ModulePassManager &MPM, bool DebugLogging,
-                                    PassBuilder::OptimizationLevel Level,
-                                    bool RunProfileGen, bool IsCS,
-                                    std::string ProfileFile,
-                                    std::string ProfileRemappingFile) {
+void PassBuilder::addPGOInstrPasses(
+    ModulePassManager &MPM, bool DebugLogging,
+    PassBuilder::OptimizationLevel Level, bool RunProfileGen, bool IsCS,
+    std::string ProfileFile, std::string ProfileRemappingFile,
+    bool ExternalFunctionUseListsAreIncomplete) {
   assert(Level != OptimizationLevel::O0 && "Not expecting O0 here!");
   // Generally running simplification passes and the inliner with an high
   // threshold results in smaller executables, but there may be cases where
   // the size grows, so let's be conservative here and skip this simplification
   // at -Os/Oz. We will not do this  inline for context sensistive PGO (when
   // IsCS is true).
   if (!Level.isOptimizingForSize() && !IsCS) {
     InlineParams IP;
 
     IP.DefaultThreshold = PreInlineThreshold;
 
     // FIXME: The hint threshold has the same value used by the regular inliner.
     // This should probably be lowered after performance testing.
     // FIXME: this comment is cargo culted from the old pass manager, revisit).
     IP.HintThreshold = 325;
 
     CGSCCPassManager CGPipeline(DebugLogging);
 
-    CGPipeline.addPass(InlinerPass(IP));
+    CGPipeline.addPass(InlinerPass(IP, ExternalFunctionUseListsAreIncomplete));
 
     FunctionPassManager FPM;
     FPM.addPass(SROA());
     FPM.addPass(EarlyCSEPass());    // Catch trivial redundancies.
     FPM.addPass(SimplifyCFGPass()); // Merge & remove basic blocks.
     FPM.addPass(InstCombinePass()); // Combine silly sequences.
     invokePeepholeEPCallbacks(FPM, Level);
 
void PassBuilder::addPGOInstrPassesForO0(ModulePassManager &MPM,
   MPM.addPass(InstrProfiling(Options, IsCS));
 }
 
 static InlineParams
 getInlineParamsFromOptLevel(PassBuilder::OptimizationLevel Level) {
   return getInlineParams(Level.getSpeedupLevel(), Level.getSizeLevel());
 }
 
-ModulePassManager
-PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
-                                               ThinLTOPhase Phase,
-                                               bool DebugLogging) {
+static bool areFuncUseListsIncompleteDuring(PassBuilder::ThinLTOPhase Phase) {
+  return Phase != PassBuilder::ThinLTOPhase::None;
+}
+
+ModulePassManager PassBuilder::buildModuleSimplificationPipeline(
+    OptimizationLevel Level, bool DebugLogging, ThinLTOPhase Phase) {
   ModulePassManager MPM(DebugLogging);
 
   bool HasSampleProfile = PGOOpt && (PGOOpt->Action == PGOOptions::SampleUse);
 
   // In ThinLTO mode, when flattened profile is used, all the available
   // profile information will be annotated in PreLink phase so there is
   // no need to load the profile again in PostLink.
   bool LoadSampleProfile =
ModulePassManager PassBuilder::buildModuleSimplificationPipeline(
 
   // Add all the requested passes for instrumentation PGO, if requested.
   if (PGOOpt && Phase != ThinLTOPhase::PostLink &&
       (PGOOpt->Action == PGOOptions::IRInstr ||
        PGOOpt->Action == PGOOptions::IRUse)) {
     addPGOInstrPasses(MPM, DebugLogging, Level,
                       /* RunProfileGen */ PGOOpt->Action == PGOOptions::IRInstr,
                       /* IsCS */ false, PGOOpt->ProfileFile,
-                      PGOOpt->ProfileRemappingFile);
+                      PGOOpt->ProfileRemappingFile,
+                      areFuncUseListsIncompleteDuring(Phase));
     MPM.addPass(PGOIndirectCallPromotion(false, false));
   }
   if (PGOOpt && Phase != ThinLTOPhase::PostLink &&
       PGOOpt->CSAction == PGOOptions::CSIRInstr)
     MPM.addPass(PGOInstrumentationGenCreateVar(PGOOpt->CSProfileGenFile));
 
   // Synthesize function entry counts for non-PGO compilation.
   if (EnableSyntheticCounts && !PGOOpt)
ModulePassManager PassBuilder::buildModuleSimplificationPipeline(
   // the callees have already been fully optimized, and we want to inline them
   // into the callers so that our optimizations can reflect that.
   // For PreLinkThinLTO pass, we disable hot-caller heuristic for sample PGO
   // because it makes profile annotation in the backend inaccurate.
   InlineParams IP = getInlineParamsFromOptLevel(Level);
   if (Phase == ThinLTOPhase::PreLink && PGOOpt &&
       PGOOpt->Action == PGOOptions::SampleUse)
     IP.HotCallSiteThreshold = 0;
-  MainCGPipeline.addPass(InlinerPass(IP));
+  MainCGPipeline.addPass(
+      InlinerPass(IP, areFuncUseListsIncompleteDuring(Phase)));
 
   // Now deduce any function attributes based in the current code.
   MainCGPipeline.addPass(PostOrderFunctionAttrsPass());
 
   // When at O3 add argument promotion to the pass pipeline.
   // FIXME: It isn't at all clear why this should be limited to O3.
   if (Level == OptimizationLevel::O3)
     MainCGPipeline.addPass(ArgumentPromotionPass());
ModulePassManager PassBuilder::buildModuleSimplificationPipeline(
   MPM.addPass(
       createModuleToPostOrderCGSCCPassAdaptor(createDevirtSCCRepeatedPass(
           std::move(MainCGPipeline), MaxDevirtIterations)));
 
   return MPM;
 }
 
 ModulePassManager PassBuilder::buildModuleOptimizationPipeline(
-    OptimizationLevel Level, bool DebugLogging, bool LTOPreLink) {
+    OptimizationLevel Level, bool DebugLogging, ThinLTOPhase Phase) {
   ModulePassManager MPM(DebugLogging);
 
   // Optimize globals now that the module is fully simplified.
   MPM.addPass(GlobalOptPass());
   MPM.addPass(GlobalDCEPass());
 
   // Run partial inlining pass to partially inline functions that have
   // large bodies.
   if (RunPartialInlining)
     MPM.addPass(PartialInlinerPass());
 
   // Remove avail extern fns and globals definitions since we aren't compiling
   // an object file for later LTO. For LTO we want to preserve these so they
   // are eligible for inlining at link-time. Note if they are unreferenced they
   // will be removed by GlobalDCE later, so this only impacts referenced
   // available externally globals. Eventually they will be suppressed during
   // codegen, but eliminating here enables more opportunity for GlobalDCE as it
   // may make globals referenced by available external functions dead and saves
   // running remaining passes on the eliminated functions. These should be
   // preserved during prelinking for link-time inlining decisions.
-  if (!LTOPreLink)
+  if (Phase != ThinLTOPhase::PreLink)
     MPM.addPass(EliminateAvailableExternallyPass());
 
   if (EnableOrderFileInstrumentation)
     MPM.addPass(InstrOrderFilePass());
 
   // Do RPO function attribute inference across the module to forward-propagate
   // attributes where applicable.
   // FIXME: Is this really an optimization rather than a canonicalization?
   MPM.addPass(ReversePostOrderFunctionAttrsPass());
 
   // Do a post inline PGO instrumentation and use pass. This is a context
   // sensitive PGO pass. We don't want to do this in LTOPreLink phrase as
   // cross-module inline has not been done yet. The context sensitive
   // instrumentation is after all the inlines are done.
-  if (!LTOPreLink && PGOOpt) {
+  if (Phase != ThinLTOPhase::PreLink && PGOOpt) {
     if (PGOOpt->CSAction == PGOOptions::CSIRInstr)
       addPGOInstrPasses(MPM, DebugLogging, Level, /* RunProfileGen */ true,
                         /* IsCS */ true, PGOOpt->CSProfileGenFile,
-                        PGOOpt->ProfileRemappingFile);
+                        PGOOpt->ProfileRemappingFile,
+                        areFuncUseListsIncompleteDuring(Phase));
     else if (PGOOpt->CSAction == PGOOptions::CSIRUse)
       addPGOInstrPasses(MPM, DebugLogging, Level, /* RunProfileGen */ false,
                         /* IsCS */ true, PGOOpt->ProfileFile,
-                        PGOOpt->ProfileRemappingFile);
+                        PGOOpt->ProfileRemappingFile,
+                        areFuncUseListsIncompleteDuring(Phase));
   }
 
   // Re-require GloblasAA here prior to function passes. This is particularly
   // useful as the above will have inlined, DCE'ed, and function-attr
   // propagated everything. We should at this point have a reasonably minimal
   // and richly annotated call graph. By computing aliasing and mod/ref
   // information for all local globals here, the late loop passes and notably
   // the vectorizer will be able to use them to help recognize vectorizable
ModulePassManager PassBuilder::buildModuleOptimizationPipeline(
 
   // Now that we've vectorized and unrolled loops, we may have more refined
   // alignment information, try to re-derive it here.
   OptimizePM.addPass(AlignmentFromAssumptionsPass());
 
   // Split out cold code. Splitting is done late to avoid hiding context from
   // other optimizations and inadvertently regressing performance. The tradeoff
   // is that this has a higher code size cost than splitting early.
-  if (EnableHotColdSplit && !LTOPreLink)
+  if (EnableHotColdSplit && Phase != ThinLTOPhase::PreLink)
     MPM.addPass(HotColdSplittingPass());
 
   // LoopSink pass sinks instructions hoisted by LICM, which serves as a
   // canonicalization pass that enables other optimizations. As a result,
   // LoopSink pass needs to be a very late IR pass to avoid undoing LICM
   // result too early.
   OptimizePM.addPass(LoopSinkPass());
 
ModulePassManager PassBuilder::buildModuleOptimizationPipeline(
   // pipeline and maybe be the bottom of the canonicalization pipeline? Weird
   // ordering here.
   MPM.addPass(GlobalDCEPass());
   MPM.addPass(ConstantMergePass());
 
   return MPM;
 }
 
-ModulePassManager
-PassBuilder::buildPerModuleDefaultPipeline(OptimizationLevel Level,
-                                           bool DebugLogging, bool LTOPreLink) {
+ModulePassManager PassBuilder::buildPerModuleDefaultPipeline(
+    OptimizationLevel Level, bool DebugLogging, ThinLTOPhase Phase) {
   assert(Level != OptimizationLevel::O0 &&
          "Must request optimizations for the default pipeline!");
 
   ModulePassManager MPM(DebugLogging);
 
   // Force any function attributes we want the rest of the pipeline to observe.
   MPM.addPass(ForceFunctionAttrsPass());
 
   // Apply module pipeline start EP callback.
   for (auto &C : PipelineStartEPCallbacks)
     C(MPM);
 
   if (PGOOpt && PGOOpt->SamplePGOSupport)
     MPM.addPass(createModuleToFunctionPassAdaptor(AddDiscriminatorsPass()));
 
   // Add the core simplification pipeline.
-  MPM.addPass(buildModuleSimplificationPipeline(Level, ThinLTOPhase::None,
-                                                DebugLogging));
+  MPM.addPass(buildModuleSimplificationPipeline(Level, DebugLogging, Phase));
 
   // Now add the optimization pipeline.
-  MPM.addPass(buildModuleOptimizationPipeline(Level, DebugLogging, LTOPreLink));
+  MPM.addPass(buildModuleOptimizationPipeline(Level, DebugLogging, Phase));
 
   return MPM;
 }
 
 ModulePassManager
 PassBuilder::buildThinLTOPreLinkDefaultPipeline(OptimizationLevel Level,
                                                 bool DebugLogging) {
   assert(Level != OptimizationLevel::O0 &&
PassBuilder::buildThinLTOPreLinkDefaultPipeline(OptimizationLevel Level,
 
   // Apply module pipeline start EP callback.
   for (auto &C : PipelineStartEPCallbacks)
     C(MPM);
 
   // If we are planning to perform ThinLTO later, we don't bloat the code with
   // unrolling/vectorization/... now. Just simplify the module as much as we
   // can.
-  MPM.addPass(buildModuleSimplificationPipeline(Level, ThinLTOPhase::PreLink,
-                                                DebugLogging));
+  MPM.addPass(buildModuleSimplificationPipeline(Level, DebugLogging,
+                                                ThinLTOPhase::PreLink));
 
   // Run partial inlining pass to partially inline functions that have
   // large bodies.
   // FIXME: It isn't clear whether this is really the right place to run this
   // in ThinLTO. Because there is another canonicalization and simplification
   // phase that will run after the thin link, running this here ends up with
   // less information than will be available later and it may grow functions in
   // ways that aren't beneficial.
ModulePassManager PassBuilder::buildThinLTODefaultPipeline(
 
   if (Level == OptimizationLevel::O0)
     return MPM;
 
   // Force any function attributes we want the rest of the pipeline to observe.
   MPM.addPass(ForceFunctionAttrsPass());
 
   // Add the core simplification pipeline.
-  MPM.addPass(buildModuleSimplificationPipeline(Level, ThinLTOPhase::PostLink,
-                                                DebugLogging));
+  MPM.addPass(buildModuleSimplificationPipeline(Level, DebugLogging,
+                                                ThinLTOPhase::PostLink));
 
   // Now add the optimization pipeline.
-  MPM.addPass(buildModuleOptimizationPipeline(Level, DebugLogging));
+  MPM.addPass(buildModuleOptimizationPipeline(Level, DebugLogging,
+                                              ThinLTOPhase::PostLink));
 
   return MPM;
 }
 
 ModulePassManager
 PassBuilder::buildLTOPreLinkDefaultPipeline(OptimizationLevel Level,
                                             bool DebugLogging) {
   assert(Level != OptimizationLevel::O0 &&
          "Must request optimizations for the default pipeline!");
   // FIXME: We should use a customized pre-link pipeline!
   return buildPerModuleDefaultPipeline(Level, DebugLogging,
-                                       /* LTOPreLink */ true);
+                                       ThinLTOPhase::PreLink);
 }
 
 ModulePassManager
 PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level, bool DebugLogging,
                                      ModuleSummaryIndex *ExportSummary) {
   ModulePassManager MPM(DebugLogging);
 
   if (Level == OptimizationLevel::O0) {
PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level, bool DebugLogging,
   FPM.addPass(JumpThreadingPass());
 
   // Do a post inline PGO instrumentation and use pass. This is a context
   // sensitive PGO pass.
   if (PGOOpt) {
     if (PGOOpt->CSAction == PGOOptions::CSIRInstr)
       addPGOInstrPasses(MPM, DebugLogging, Level, /* RunProfileGen */ true,
                         /* IsCS */ true, PGOOpt->CSProfileGenFile,
-                        PGOOpt->ProfileRemappingFile);
+                        PGOOpt->ProfileRemappingFile,
+                        /*ExternalFunctionUseListsAreIncomplete=*/false);
     else if (PGOOpt->CSAction == PGOOptions::CSIRUse)
       addPGOInstrPasses(MPM, DebugLogging, Level, /* RunProfileGen */ false,
                         /* IsCS */ true, PGOOpt->ProfileFile,
-                        PGOOpt->ProfileRemappingFile);
+                        PGOOpt->ProfileRemappingFile,
+                        /*ExternalFunctionUseListsAreIncomplete=*/false);
   }
 
   // Break up allocas
   FPM.addPass(SROA());
 
   // LTO provides additional opportunities for tailcall elimination due to
   // link-time inlining, and visibility of nocapture attribute.
   FPM.addPass(TailCallElimPass());

llvm/lib/Transforms/IPO/InlineSimple.cpp

class SimpleInliner : public LegacyInlinerBase {
 
   InlineParams Params;
 
 public:
   SimpleInliner() : LegacyInlinerBase(ID), Params(llvm::getInlineParams()) {
     initializeSimpleInlinerPass(*PassRegistry::getPassRegistry());
   }
 
-  explicit SimpleInliner(InlineParams Params)
-      : LegacyInlinerBase(ID), Params(std::move(Params)) {
+  explicit SimpleInliner(InlineParams Params,
+                         bool ExternalFunctionUseListsAreIncomplete)
+      : LegacyInlinerBase(ID), Params(std::move(Params)),
+        ExternalFunctionUseListsAreIncomplete(
+            ExternalFunctionUseListsAreIncomplete) {
     initializeSimpleInlinerPass(*PassRegistry::getPassRegistry());
   }
 
   static char ID; // Pass identification, replacement for typeid
 
   InlineCost getInlineCost(CallSite CS) override {
     Function *Callee = CS.getCalledFunction();
     TargetTransformInfo &TTI = TTIWP->getTTI(*Callee);
 
     bool RemarksEnabled = false;
     const auto &BBs = CS.getCaller()->getBasicBlockList();
     if (!BBs.empty()) {
       auto DI = OptimizationRemark(DEBUG_TYPE, "", DebugLoc(), &BBs.front());
       if (DI.isEnabled())
         RemarksEnabled = true;
     }
     OptimizationRemarkEmitter ORE(CS.getCaller());
 
     std::function<AssumptionCache &(Function &)> GetAssumptionCache =
         [&](Function &F) -> AssumptionCache & {
       return ACT->getAssumptionCache(F);
     };
+
+    Function &Caller = *CS.getCaller();
+    bool CallerMayHaveUses =
+        ExternalFunctionUseListsAreIncomplete || !Caller.use_empty();
+    bool CallerMayBeInlined =
+        CallerMayHaveUses && !Caller.hasFnAttribute(Attribute::NoInline);
+
     return llvm::getInlineCost(
         cast<CallBase>(*CS.getInstruction()), Params, TTI, GetAssumptionCache,
-        /*GetBFI=*/None, PSI, RemarksEnabled ? &ORE : nullptr);
+        /*GetBFI=*/None, PSI, RemarksEnabled ? &ORE : nullptr,
+        /*CallerInlineabilityMatters=*/CallerMayBeInlined);
   }
 
   bool runOnSCC(CallGraphSCC &SCC) override;
   void getAnalysisUsage(AnalysisUsage &AU) const override;
 
 private:
   TargetTransformInfoWrapperPass *TTIWP;
-
+  // Assume that new references can't appear on otherwise-unreferenced `extern`
+  // functions. This isn't the case in e.g., ThinLTO.
+  bool ExternalFunctionUseListsAreIncomplete = true;
 };
 
 } // end anonymous namespace
 
 char SimpleInliner::ID = 0;
 INITIALIZE_PASS_BEGIN(SimpleInliner, "inline", "Function Integration/Inlining",
                       false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_DEPENDENCY(CallGraphWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
 INITIALIZE_PASS_END(SimpleInliner, "inline", "Function Integration/Inlining",
                     false, false)
 
 Pass *llvm::createFunctionInliningPass() { return new SimpleInliner(); }
 
-Pass *llvm::createFunctionInliningPass(int Threshold) {
-  return new SimpleInliner(llvm::getInlineParams(Threshold));
+Pass *
+llvm::createFunctionInliningPass(int Threshold,
+                                 bool ExternalFunctionUseListsAreIncomplete) {
+  return new SimpleInliner(llvm::getInlineParams(Threshold),
+                           ExternalFunctionUseListsAreIncomplete);
 }
 
-Pass *llvm::createFunctionInliningPass(unsigned OptLevel,
-                                       unsigned SizeOptLevel,
-                                       bool DisableInlineHotCallSite) {
+Pass *
+llvm::createFunctionInliningPass(unsigned OptLevel, unsigned SizeOptLevel,
+                                 bool DisableInlineHotCallSite,
+                                 bool ExternalFunctionUseListsAreIncomplete) {
   auto Param = llvm::getInlineParams(OptLevel, SizeOptLevel);
   if (DisableInlineHotCallSite)
     Param.HotCallSiteThreshold = 0;
-  return new SimpleInliner(Param);
+  return new SimpleInliner(Param, ExternalFunctionUseListsAreIncomplete);
 }
 
-Pass *llvm::createFunctionInliningPass(InlineParams &Params) {
-  return new SimpleInliner(Params);
+Pass *
+llvm::createFunctionInliningPass(InlineParams &Params,
+                                 bool ExternalFunctionUseListsAreIncomplete) {
+  return new SimpleInliner(Params, ExternalFunctionUseListsAreIncomplete);
 }
 
 bool SimpleInliner::runOnSCC(CallGraphSCC &SCC) {
   TTIWP = &getAnalysis<TargetTransformInfoWrapperPass>();
   return LegacyInlinerBase::runOnSCC(SCC);
 }
 
 void SimpleInliner::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addRequired<TargetTransformInfoWrapperPass>();
   LegacyInlinerBase::getAnalysisUsage(AU);
 }

llvm/lib/Transforms/IPO/Inliner.cpp

for (int i = 0; i < (int)Calls.size(); ++i) {
     };
 
     auto GetInlineCost = [&](CallSite CS) {
       Function &Callee = *CS.getCalledFunction();
       auto &CalleeTTI = FAM.getResult<TargetIRAnalysis>(Callee);
       bool RemarksEnabled =
           Callee.getContext().getDiagHandlerPtr()->isMissedOptRemarkEnabled(
               DEBUG_TYPE);
+
+      Function &Caller = *CS.getCaller();
+      bool CallerMayHaveUses =
+          ExternalFunctionUseListsAreIncomplete || !Caller.use_empty();
+      bool CallerMayBeInlined =
+          CallerMayHaveUses && !Caller.hasFnAttribute(Attribute::NoInline);
+
       return getInlineCost(cast<CallBase>(*CS.getInstruction()), Params,
                            CalleeTTI, GetAssumptionCache, {GetBFI}, PSI,
-                           RemarksEnabled ? &ORE : nullptr);
+                           RemarksEnabled ? &ORE : nullptr,
+                           /*CallerInlineabilityMatters=*/CallerMayBeInlined);
     };
 
     // Now process as many calls as we have within this caller in the sequnece.
     // We bail out as soon as the caller has to change so we can update the
     // call graph and prepare the context of that new caller.
     bool DidInline = false;
     for (; i < (int)Calls.size() && Calls[i].first.getCaller() == &F; ++i) {
       int InlineHistoryID;

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

if (OptLevel > 0 && SizeLevel == 0 && !DisablePreInliner &&
     // inliner to influence pre-inlining. The only fields of InlineParams we
     // care about are DefaultThreshold and HintThreshold.
     InlineParams IP;
     IP.DefaultThreshold = PreInlineThreshold;
     // FIXME: The hint threshold has the same value used by the regular inliner.
     // This should probably be lowered after performance testing.
     IP.HintThreshold = 325;
 
-    MPM.add(createFunctionInliningPass(IP));
+    MPM.add(createFunctionInliningPass(
+        IP,
+        /*ExternalFunctionUseListsAreIncomplete=*/PrepareForLTO ||
+            PrepareForThinLTO || PerformThinLTO));
     MPM.add(createSROAPass());
     MPM.add(createEarlyCSEPass());             // Catch trivial redundancies
     MPM.add(createCFGSimplificationPass());    // Merge & remove BBs
     MPM.add(createInstructionCombiningPass()); // Combine silly seq's
     addExtensionsToPM(EP_Peephole, MPM);
   }
   if ((EnablePGOInstrGen && !IsCS) || (EnablePGOCSInstrGen && IsCS)) {
     MPM.add(createPGOInstrumentationGenLegacyPass(IsCS));

llvm/test/Transforms/Inline/bpi-cold-inlining.ll

+; RUN: opt < %s -passes=inline -inline-cold-callsite-threshold=0 -S | FileCheck %s
+
+declare void @foo(i32)
+
+@a = external global i1
+
+define internal void @callee1() {
+  call void @foo(i32 1)
+  ret void
+}
+
+; CHECK-LABEL: define void @not_inlined_if_cold
+define void @not_inlined_if_cold() {
+entry:
+  %a = load i1, i1* @a
+  br i1 %a, label %if.then, label %if.end, !prof !0
+
+if.then:
+  ; CHECK: call void @callee1()
+  call void @callee1()
+  br label %if.end
+
+if.end:
+  ; CHECK: ret void
+  ret void
+}
+
+@not_inlined_if_cold_addr = global void ()* @not_inlined_if_cold
+
+define internal void @callee2() {
+  call void @foo(i32 2)
+  ret void
+}
+
+; CHECK-LABEL: define void @gets_inlined_noinline_and_cold
+define void @gets_inlined_noinline_and_cold() #0 {
+entry:
+  %a = load i1, i1* @a
+  br i1 %a, label %if.then, label %if.end, !prof !0
+
+if.then:
+  ; CHECK: call void @foo(i32 2)
+  call void @callee2()
+  br label %if.end
+
+if.end:
+  ; CHECK: ret void
+  ret void
+}
+
+@gets_inlined_noinline_and_cold_addr = global void ()* @gets_inlined_noinline_and_cold
+
+define internal void @callee3() {
+  call void @foo(i32 3)
+  ret void
+}
+
+; CHECK-LABEL: define void @gets_inlined_no_uses_and_cold
+define void @gets_inlined_no_uses_and_cold() {
+entry:
+  %a = load i1, i1* @a
+  br i1 %a, label %if.then, label %if.end, !prof !0
+
+if.then:
+  ; CHECK: call void @foo(i32 3)
+  call void @callee3()
+  br label %if.end
+
+if.end:
+  ; CHECK: ret void
+  ret void
+}
+
+attributes #0 = { noinline }
+
+!0 = !{!"branch_weights", i32 1, i32 2000}