[llvm] Add interface to drive inlining decision using ML model

[mtrofin]

Summary

This patch adds the analysis pass and the hook-ups into the inliner. The
implementation of the analysis pass requires optional dependencies be
present, and will be introduced subsequently.

RFC: http://lists.llvm.org/pipermail/llvm-dev/2020-April/140763.html

[mtrofin]

added comments

[mtrofin]

Removed unrelated changes

[davidxl]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

28 ↗

(On Diff #260773)

No other interfaces in the class?

33 ↗

(On Diff #260773)

two bools are confusing. Perhaps split them? SetWasDeleted, and SetWasInlined?

Also document this interface.

42 ↗

(On Diff #260773)

desided --> decided

50 ↗

(On Diff #260773)

nit: shouldInline --> getInlineAdvise?

68 ↗

(On Diff #260773)

This comments belongs somewhere else -- perhaps in implementation ?

[mtrofin]

feedback

[mtrofin]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

28 ↗

(On Diff #260773)

Not sure I follow. Maybe I should place InliningAdvisor first in the file, followed by PendingInliningRecord (forward - declaring the latter - the current order is just to avoid that fwd decl)?

33 ↗

(On Diff #260773)

The idea was to atomically communicate the conclusion of the inlining operation.

I think the liability of setWasDeleted and setWasInlined would be that:

• we'd need a 'finish()' method to say we're done, which adds complexity and more statefulness to the design (harder to track); or
• just rely on the dtor to do that, but that doesn't address the statefulness
• need to rely on user to remember to call both setWas{Deleted|Inlined}

I realize the current design requires a reader look up and see what the 2 params meant. I think that's a 'lesser evil' than those outlined above.

How about either:

• I decorate the call sites for recordInlining with comments indicating what the parameters mean, or
• add 3 public APIs, as follows (new API -> current API)

recordInlining() -> recordInlining(/*CalleeWasDeleted*/false, /*SiteWasInlined*/ true)
recordInliningWithCalleeDeleted() -> recordInlining(/*CalleeWasDeleted*/true, /*SiteWasInlined*/ true)
recordUnsuccessfulInlining() -> recordInlining(/*CalleeWasDeleted*/false, /*SiteWasInlined*/ false)

wdyt?

42 ↗

(On Diff #260773)

thanks - I meant 'desired' :)

68 ↗

(On Diff #260773)

Reworded.

[mtrofin]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

50 ↗

(On Diff #260773)

sure - I'll leave this open until the review gets closer to the end, since it's easy to refactor, and I'd just want to do it once.

[davidxl]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

28 ↗

(On Diff #260773)

what I meant is that this class defined here has no other accessor interfaces (other than one setter).

33 ↗

(On Diff #260773)

Making recordInlining private and add wrappers to them sounds good.

[mtrofin]

feedback. Simplified interaction between inliner pass, removed 1 todo

[mtrofin]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

28 ↗

(On Diff #260773)

Ah, yes. That's intended - well, 3 setters, with the recent changes.

[davidxl]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

23 ↗

(On Diff #261060)

add documentation on MLMode.

84 ↗

(On Diff #261060)

Perhaps rename 'AlternativeRecommendation' to 'MLDecision'? or 'MLRecommendation'?

llvm/lib/Analysis/InliningAdvisorAnalysis.cpp

33 ↗

(On Diff #261060)

Can this happen, given the option parser filters out other values?

llvm/lib/Passes/PassBuilder.cpp

220

Invalid mode --> Null mode?

llvm/lib/Transforms/IPO/Inliner.cpp

684

why not use it to handle Entry too? If null, does nothing.

[mtrofin]

Feedback

[mtrofin]

llvm/lib/Analysis/InliningAdvisorAnalysis.cpp

33 ↗

(On Diff #261060)

This happens if we pass Dev (for example) and neither LLVM_HAVE_TF_{API|AOT} are defined. It's what the test tests, too.

llvm/lib/Passes/PassBuilder.cpp

220

How about 'NoML' or 'None'?

[echristo]

This is pretty cool. I've got one high level question and then a few minor things in the patch.

High level: Would it be better to split this out even more rather than the if/elses in the Inliner? i.e. rather than conditionalizing it just have N paths through. It may mean a little bit more duplicated or factored code, but would probably be a little easier to read through.

Thoughts?

-eric

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

33 ↗

(On Diff #261114)

Might be nice to just spell them out?

46 ↗

(On Diff #261114)

Is there any way we could use an enum(s) instead for these calls rather than boolean arguments?

llvm/lib/Passes/PassBuilder.cpp

724

Extra braces.

752

Can we get a better description of what the cleanup pass is supposed to do? We don't typically have such things and it seems worth calling out in a bit more detail.

[davidxl]

llvm/lib/Passes/PassBuilder.cpp

220

NoML or None or just No sound fine.

llvm/lib/Transforms/IPO/Inliner.cpp

857–858

This should be better computed under

if (Advisor) {

  TrivialDecision = ...
  if (noinline)
     continue;
   Mandatory = ...
}

llvm/test/Transforms/Inline/inlining-advisor-default.ll

10

I wonder if it is useful to have a mock mode for testing purpose where the ML code is barebone but not fully statically compiled out.

[mtrofin]

In D79042#2011802, @echristo wrote:

This is pretty cool. I've got one high level question and then a few minor things in the patch.

High level: Would it be better to split this out even more rather than the if/elses in the Inliner? i.e. rather than conditionalizing it just have N paths through. It may mean a little bit more duplicated or factored code, but would probably be a little easier to read through.

Thoughts?

-eric

I agree - I'll send first some NFC patches cleaning up a bit the current code; then, refactor along the lines you suggested. I think there'll end up being very little, if any, duplication, too!

[mtrofin]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

33 ↗

(On Diff #261114)

I.e. 'Release' and 'Development'? I like that. (will send and updated patch once Inliner.cpp refactoring is done, so leaving this and others marked as not 'done')

46 ↗

(On Diff #261114)

You mean instead of the 3 members that call into this protected member?

so we have it fleshed out:

enum class InliningOutcome {
  Failed,
  Succeeded,
  SucceededWithCalleeRemoval
}

and then we only have the one virtual, and no protected:

virtual void recordInlining(InliningOutcome Outcome)

Does that capture it?

I'm fine with whatever we all feel is the more usable API. One thing to consider - looking at how the APIs are used, I'd actually prefer 2 methods: 1 method for "it failed", and 1 for "it succeeded", with a bool saying whether the callee will be removed or not (see Inliner.cpp:1213 and below, in this patch)

wdyt?

llvm/test/Transforms/Inline/inlining-advisor-default.ll

10

If we only use that for test, then the 'dev' mode where we just capture logs does exactly that - and I plan on getting a build bot setup for this; and it has a test, of course (and doesn't add complexity)

[davidxl]

llvm/test/Transforms/Inline/inlining-advisor-default.ll

10

it might be better to exercise some tests by default (without relying on bot). When there is a breakage, it is also easier for folks to reproduce.

[dblaikie]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

30–31 ↗

(On Diff #260773)

You can leave these off if you like - they're implied by the existence of a user-declared destructor. (that'll automatically delete/remove the copy/move operations)

36–37 ↗

(On Diff #260773)

This is the default (well, it'd be public by default, but it can't be called directly because of the presence of a pure virtual function in the class) so you could omit it.

llvm/lib/Analysis/InliningAdvisorAnalysis.cpp

25 ↗

(On Diff #260773)

Remove dead code after an unreachable here.

28–35 ↗

(On Diff #260773)

I'd probably picture this not using the #ifdef stuff here - InliningAdvisor.h has to be unconditionally compiled anyway (since it provides the IniliningAdvisor base class)... oh - but you weren't planning on unconditionally compiling a .cpp file that implements it? That would be good to avoid - it'd be unfortunate to have a header we have to include with silent contracts about which parts of the header you can use (& we shouldn't be in a position where certain member functions has to be defined in a header or must be defined in the .cc file or there will be subtle problems - that'd be surprising/confusing for maintenance). I'd probably go further and say maybe the whole header should live outside the ML directory? And either it'd provide the InliningAdvisor::create function, and use #ifdefs in its own implementation (seems fair - it's already going to need to test TF_API and TF_AOT to decide which kind of implementations it has available) so it can be called unconditionally - and just returns nullptr if neither API is available.

That could be implemented in llvm/Analysis (non-ML), InliningAdvisorFactory, just a header with one function declared in it, and a single source file implementing it that's just "ifdef TF_API return new TFAPIThing #elseif TF_AOT ... etc")

llvm/lib/Transforms/IPO/Inliner.cpp

684–694

I'm surprised we don't have a scope-guard like device in LLVM already, but one other way to address this (perhaps that's what everyone else has done in LLVM so far) would be to pull the body of the function out into a separate function, so you can easily ensure the end operation is done.

Otherwise, if you prefer to keep it this way it seems like a bit of overkill to make it a class - probably a struct, used {} init rather than a ctor, etc to keep it simple, given it's only used once/immediately.

[mtrofin]

llvm/lib/Analysis/InliningAdvisorAnalysis.cpp

28–35 ↗

(On Diff #260773)

I don't follow. The ifdef is in the InliningAdvisorAnalysis.cpp, which is unconditionally compiled.

[dblaikie]

llvm/include/llvm/Analysis/ML/InliningAdvisor.h

30–31 ↗

(On Diff #260773)

Yeah, looks like this comment ended up out-of-place, was writing it based on a previous version of the code. (the email had the right quoted text, FWIW)

& in any case, my comment was incorrect - the user-declared dtor doesn't disable copy/move, so makes sense to leave them.

36–37 ↗

(On Diff #260773)

This comment was about the protected defaulted default ctor - well, you still need to define it (because of the deleted copy/move ctors) but you could define it as public as much as as protected - either way no one can instantiate this class because it has pure virtual functions. Not a huge deal either way I guess.

llvm/lib/Analysis/InliningAdvisorAnalysis.cpp

25 ↗

(On Diff #260773)

(the "return nullptr;" after "llvm_unreachable" is dead - llvm_unreachable is trap/abort/UB-in-optimized builds, the return is meaningless/would be DCE'd anyway, best to remove it)

28–35 ↗

(On Diff #260773)

Ah, sorry, that was a bit rambly.

What I mean is, why is the call to InliningAdvisor::create conditionalized? I assume it's because, while the declaration of InliningAdvisor::create is always available in InliningAdvisor.h, its implementation is not (because, I assume, everything in Analysis/ML is conditionally compiled only if TF is available?). I think it's problematic to have a header that only works because certain parts of it must remain in the header & can't be moved to an implementation file (because there is no unconditionally compiled implementation file to move them to).

Hmm, now I'm confused - the header has parts defined in lib/Analysis (like InliningAdvisorAnalysis::* here) and parts defined in lib/Analysis/MC (like InliningAdvisor::create - well, that isn't defined anywhere right now, it seems - but I assume the idea was to implement it in lib/Analysis/MC?)).

I think it might be tidier to have the lib/Analysis/MC entry point be very small - just the InliningAdvisorAnalysis::create - and if the goal is to have everything in lib/Analysis/MC be conditionally compiled using CMake and the presence of TF, then maybe having a header in lib/Analysis that declares (& defines inline, or out-of-line in its own .cpp file in lib/Analysis, either way) and uses macros to define the implementation as either a no-op (because it can't call into lib/Analysis/MC because it doesn't exist) or as the functional implementation that instantiates the desired class in lib/Analysis/MC).

I think having the factory function always be available, but it either is implemented as a no-op (returning a null unique_ptr) when the library isn't available keeps the boundary in a very clear/small part of the code, not amongst any other code (not that InliningAdvisorAnalysis.cpp is especially long by any means).

I wouldn't put any of the #ifdefs in if they're always false when this patch is committed - that's dead/untested code, which is best not to commit (a comment describing what else might go there is OK though).

I think there's some discussion of having a stub implementation for testing? I think that'd be appropriate to go in with this patch - otherwise there's a lot of dead code (not just these #ifdefs, but all the InliningAdvisor results, etc aren't tested because there's never a non-null implementation in this patch). (I'm not sure how that stub would be powered - either a hardcoded general implementation (does something silly like forcibly inline every second function) with DEBUG outputs to indicate how it's being called - or perhaps reading from an external file to dictate what paths to go down... or unit tested, but that's not usually how optimizations are tested (maybe there's a reasonable amount of unit testing for analyses though? I'm not sure - I haven't looked))

[mtrofin]

llvm/lib/Analysis/InliningAdvisorAnalysis.cpp

28–35 ↗

(On Diff #260773)

Initially, I was planning to implement InliningAdvisor::create in lib/Analysis/ML/InliningAdvisor.cpp. To summarize your proposal, also factoring in the stub implementation proposal:

• declare a "llvm::createInliningAdvisor" in include/Analysis/ML/InliningAdvisor.h, implement it in lib/Analysis/ML/InliningAdvisor.cpp (instead of the static InliningAdvisor::create)
• in lib/Analysis/InliningAdvisorAnalysis.cpp, include include/Analysis/ML/InliningAdvisor.h conditionally
• add the conditional include and the conditional call to llvm::createInliningAdvisor in the patch following this one, when we add the ML stuff

• for the stub implementation, I see 2 options, with all the above staying the same :
  1. there'll be a 4th MLMode, 'Test' or something
     1. • the implementation of InliningAdvisor will be a no-op here, in InliningAdvisorAnalysis.cpp
        • the case statement will just new-up that no-op directly (i.e. it won't be going through the llvm::createInliningAdvisor when it gets introduced)
        • the noop would be mimic the codepath for 'Development' mode, when we want to learn starting from the current behavior, and parrot back the cost-based result (i.e. whether shouldInline in Inliner.cpp says "yes" or "no"). That still means we don't test the 'Release' code path in Inliner.cpp, which skips calling shouldInline.
     2. no 4th mode for MLMode, instead:
        • we always have an InliningAdvisor, i.e. even without ML in the picture
        • the default InliningAdvisor performs today's shouldInline
        • Inliner.cpp's code gets simplified - no conditional 'if (Advisor)' or anything. All the InlineCost-based reporting happens in the default InliningAdvisor's corresponding PendingInliningRecord implementation.
        • the Development mode behavior of calling today's decision making process is encapsulated in it, and thus tested there

I'd pitch for option 2 - simpler code, I think

What do folks think? (Probably it'd help the discussion if I made the update to the patch)

[mtrofin]

Simplified Inliner control flow by factoring out the inlining decision from the pass. This also means we exercise the Inliner.cpp control flow in both ML and non-ML cases.

Removed the 'cleanup' pass, and, instead, created a module pass wrapper for the inliner that manages the necessary state internally.

[google-llvm-upstream-contributions]

llvm/lib/Passes/PassBuilder.cpp

220

Replaced with 'Default', as we always use an InliningAdvisor.

752

Removed the cleanup pass altogether.

llvm/lib/Transforms/IPO/Inliner.cpp

857–858

Simplified control flow. The reasoning around mandatory and trivial case will go in the ML-specific implementation.

llvm/test/Transforms/Inline/inlining-advisor-default.ll

10

Done. We always use an InliningAdvisor. Also added tests side-by-side the inliner tests that were using the new pass manager, to test the ModuleInlinerPassWrapper. So the control flow in the inliner is tested in all cases, and these 'duplicate' cases test the wrapper and the logic getting the advisor from it.

[davidxl]

llvm/include/llvm/Analysis/InliningAdvisor.h

32

perhaps use compile time instead of runtime

[davidxl]

llvm/include/llvm/Analysis/InliningAdvisor.h

204

should this refactoring be done seperately?

llvm/lib/Transforms/IPO/Inliner.cpp

679

Put the advisor computation code into its own helper method?

679

OwnedAdvisor --> DefaultAdvisor or NullAdvisor or NopAdvisor?

[mtrofin]

incorporated feedback

llvm/include/llvm/Analysis/InliningAdvisor.h

32

reworded, I think it's more clear now.

204

Yes, if we're OK with this direction, I'll add first the InliningAdvisor.{h|cpp} with just these functions.

llvm/lib/Passes/PassBuilder.cpp

724

This probably got out of sync with the source, or got addressed by the latest patch.

llvm/lib/Transforms/IPO/Inliner.cpp

679

It's not a Null/Nop advisor, it's today's behavior. Also, you can get a "DefaultAdvisor" through the analysis pass. The key difference here is lifetime management: when InlinerPass is used stand-alone as a SCC pass, we make and own the DefaultInliningAdvisor instance.

684

Un-did the entry handling, because we're using the scope_exit API.

684–694

There is a scope-guarding API! Thanks for the pointer. Addressed.

llvm/test/Transforms/Inline/inlining-advisor-default.ll

10

Done, as we always use an InliningAdvisor. All the bots will test is ML-specific implementation.

[mtrofin]

updated after related patch landed

[davidxl]

llvm/include/llvm/Analysis/InlineAdvisor.h

50 ↗

(On Diff #263205)

To make the naming more consistent, suggest changing the name to

InlineAdvise or InlineRecord

111 ↗

(On Diff #263205)

InliningAdviser --> InlineAdviser

121 ↗

(On Diff #263205)

perhaps shorten it : getAdvice

158 ↗

(On Diff #263205)

DefaultInliningAdvisor --> DefaultInlineAdvisor

172 ↗

(On Diff #263205)

->InlineAdviserAnalysis

llvm/lib/Analysis/InlineAdvisor.cpp

51 ↗

(On Diff #263205)

DefaultInlineRecord

llvm/lib/Passes/PassBuilder.cpp

625–626

This is for InstrPGO early inlining. I don't see the need to hook up advisor here -- even there is need in the future, it may need a different model. recommend leave this alone and add some comments.

llvm/lib/Transforms/IPO/Inliner.cpp

1079

Is this guard needed?

[mtrofin]

renames.

will update the names in pass pipeline tests once we close on whether to use the wrapper for InstPGO early inlining.

llvm/lib/Passes/PassBuilder.cpp

625–626

It uses the default advisor. I can see it either way, at the moment it's behaviorally equivalent, the reason I'd use the wrapper here is for uniformity in the pass builder.

llvm/lib/Transforms/IPO/Inliner.cpp

1079

Well... only to avoid creating a wrapper if one isn't needed?

[mtrofin]

also fixed tests

[davidxl]

llvm/lib/Transforms/IPO/Inliner.cpp

1079

While this is a good change, it seems irrelevant to this patch though.

1083

so this pass injects the inliner pass into the pipeline in a lazy way (on the fly as it runs)?

[davidxl]

+tejohnson for a closer look at pass changes.

[mtrofin]

integration. Added API to register module analyses.

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

1079

Fair enough - removed. The devirt wrapper is noop with MaxDevirtIterations == 0 anyway.

1083

It's not lazily injected, in the sense that the parent PM is unaffected. We just use a PM as an implementation detail.

[tejohnson]

llvm/test/Other/new-pm-lto-defaults.ll

79

What causes this one to get added?

[mtrofin]

llvm/test/Other/new-pm-lto-defaults.ll

79

See PassBuilder.cpp:1329 - the patch uses the ModuleInlinerWrapper, so that we always inject, canonically, the InlineAdvisor outside the inline pass.

[tejohnson]

llvm/test/Other/new-pm-lto-defaults.ll

79

Right, I understand why the InlineAdvisorAnalysis gets added. But why does LazyCallGraph get added when it wasn't before?

[mtrofin]

clarified new-pm-lto-defaults.ll new pass

[mtrofin]

llvm/test/Other/new-pm-lto-defaults.ll

79

Oh! It's actually the devirt pass, which should be a no-op here. If we prefer, for simplicity/clarity, I can bypass using it in Inliner.cpp:1076

[tejohnson]

llvm/test/Other/new-pm-lto-defaults.ll

79

Oh I see. It is a noop because in the regular LTO case there is no simplification pass added to the CGSCC PM, and the inliner pass itself cannot create devirtualization opportunities? The regular LTO pipeline is going to be more sensitive to compile time, what is the overhead of adding this even when it is a noop? Might be better to avoid it.

[dblaikie]

(just some minor stylistic/formatting things from me)

llvm/lib/Analysis/InlineAdvisor.cpp

103–112 ↗

(On Diff #263488)

Any particular reason that GetAssumptionCache is a std::function, but GetBFI/TLI/InlineCost are all lambdas without the std::function wrapper? (probably good to make GetAssumptionCache match the others, I guess)

124–125 ↗

(On Diff #263488)

Prefer std::make_unique if/where possible. (probably written as auto Ret = std::make_unique<DefaultInlineAdvice>(this, CB, OIC, ORE) or just return std::make_unique...;)

142 ↗

(On Diff #263488)

Drop the () around F here - they're not needed. (also - elsewhere in this patch, there's a bunch of unnecessary ; on function definitions (eg: virtual void OnPassExit(){};) - be good to remove those too)

llvm/lib/Transforms/IPO/Inliner.cpp

676

Usually I'd prefer/suggest using * rather than getValue() on an Optional.

[mtrofin]

opt-in to devirt

[mtrofin]

correction

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

1079

Undid this - see tejohnson's observation about the LTO pipeline sensitivity to compiletime.

llvm/test/Other/new-pm-lto-defaults.ll

79

It's no-op because the MaxDevirtIterations parameter of the ModuleInlinerWrapperPass ctor is 0. I also prefer not instantiating it if not needed - addressed.

I looked deeper, here's what happens:

• originally, we had this:

MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(
      InlinerPass(getInlineParamsFromOptLevel(Level))));

• with the change, we have:

MPM.add(ModuleInlinerWrapper)

Now, ModuleInlinerWrapper has a ModuleToPostOrderCGSCCPassAdaptor wrapping a CGSCCPassManager, which inside has the InlinerPass (or, just before, the ModuleInlinerWrapper had a ModuleToPostOrderCGSCCPassAdaptor wrapping a devirt pass wrapping the CGSCCPassManager wrapping the InlinerPass)

When we used the devirt wrapper, the ModuleToPostOrderCGSCCPassAdaptor's template type parameters included the devirt wrapper type - hence my initially (and incompletely) observing that as a reason for the diff. Now, it includes PassManager as type parameters (and LazyCallGraph, but probably a more apt pattern match is on PassManager though)

I'd argue the CGSCCPassManager doesn't add much overhead in this case (as opposed to the devirt wrapper) - its run method basically goes straight to running the contained pass.

Alternatively, if we want to completely avoid any overhead, we could do some customization in the ModuleInlinerWrapper to avoid using the CGSCCPassManager if there's just the inliner.

A third option I'd prefer not to do is to just instantiate (in this case) the InlinerPass like before. I prefer not going that way because the inliner parameters are now a concept owned by the InlineAdvisor.

WDYT?

[tejohnson]

The PM changes lgtm

llvm/test/Other/new-pm-lto-defaults.ll

79

I think I agree that wrapping in the additional CGSCC PM shouldn't be a problem. So I think this is fine now with the change to avoid the devirt wrapper.

[mtrofin]

feedback

llvm/lib/Analysis/InlineAdvisor.cpp

103–112 ↗

(On Diff #263488)

This is how it was originally, in Inliner.cpp (and still is, it's needed there still). There seems to be a cast issue. I'll investigate separately, as it may involve slightly changing the types of getInlineCost parameters. Left a fixme.

[davidxl]

LGTM (assuming all pending comments are addressed).

llvm/lib/Transforms/IPO/Inliner.cpp

855

PendingRecord -- IA or Advise

[mtrofin]

small rename

[mtrofin]

llvm/lib/Transforms/IPO/Inliner.cpp

855

done

[leonardchan]

Hello, a bisect seems to show this breaks our 2 stage toolchain build from assertion errors:

[12/3658] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o
FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o 
/b/s/w/ir/cache/goma/client/gomacc /b/s/w/ir/k/recipe_cleanup/clangy7u70P/llvm_build_dir/./bin/clang++ --sysroot=/b/s/w/ir/k/cipd/linux-amd64  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/b/s/w/ir/k/llvm-project/llvm/lib/Support -I/b/s/w/ir/k/recipe_cleanup/clangy7u70P/libxml2_install/include/libxml2 -Iinclude -I/b/s/w/ir/k/llvm-project/llvm/include -I/b/s/w/ir/k/recipe_cleanup/clangy7u70P/zlib_install/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wno-comment -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -flto -ffile-prefix-map=/b/s/w/ir/k/recipe_cleanup/clangy7u70P/llvm_build_dir/tools/clang/stage2-bins=../recipe_cleanup/clangy7u70P/llvm_build_dir/tools/clang/stage2-bins -ffile-prefix-map=/b/s/w/ir/k/llvm-project/= -no-canonical-prefixes -O3 -DNDEBUG   -std=c++14  -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o -c /b/s/w/ir/k/llvm-project/llvm/lib/Support/APFloat.cpp
clang++: /b/s/w/ir/k/llvm-project/llvm/lib/Analysis/CodeMetrics.cpp:104: static void llvm::CodeMetrics::collectEphemeralValues(const llvm::Function *, llvm::AssumptionCache *, SmallPtrSetImpl<const llvm::Value *> &): Assertion `I->getParent()->getParent() == F && "Found assumption for the wrong function!"' failed.
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
Fuchsia clang version 11.0.0 (https://fuchsia.googlesource.com/a/third_party/llvm-project 9ed9860d8774b364412ce50bf5d59a8eaee1adb7)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /b/s/w/ir/k/recipe_cleanup/clangy7u70P/llvm_build_dir/./bin
clang++: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang++: note: diagnostic msg: /tmp/APFloat-fb32b4.cpp
clang++: note: diagnostic msg: /tmp/APFloat-fb32b4.sh
clang++: note: diagnostic msg: 

********************

Would you mind taking a look and sending out a fix or reverting? Thanks.

Builder: https://luci-milo.appspot.com/p/fuchsia/builders/prod/clang-linux-x64/b8880367188790280224

[mtrofin]

In D79042#2037194, @leonardchan wrote:

Hello, a bisect seems to show this breaks our 2 stage toolchain build from assertion errors:

[12/3658] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o
FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o 
/b/s/w/ir/cache/goma/client/gomacc /b/s/w/ir/k/recipe_cleanup/clangy7u70P/llvm_build_dir/./bin/clang++ --sysroot=/b/s/w/ir/k/cipd/linux-amd64  -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/b/s/w/ir/k/llvm-project/llvm/lib/Support -I/b/s/w/ir/k/recipe_cleanup/clangy7u70P/libxml2_install/include/libxml2 -Iinclude -I/b/s/w/ir/k/llvm-project/llvm/include -I/b/s/w/ir/k/recipe_cleanup/clangy7u70P/zlib_install/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wno-comment -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -flto -ffile-prefix-map=/b/s/w/ir/k/recipe_cleanup/clangy7u70P/llvm_build_dir/tools/clang/stage2-bins=../recipe_cleanup/clangy7u70P/llvm_build_dir/tools/clang/stage2-bins -ffile-prefix-map=/b/s/w/ir/k/llvm-project/= -no-canonical-prefixes -O3 -DNDEBUG   -std=c++14  -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o -c /b/s/w/ir/k/llvm-project/llvm/lib/Support/APFloat.cpp
clang++: /b/s/w/ir/k/llvm-project/llvm/lib/Analysis/CodeMetrics.cpp:104: static void llvm::CodeMetrics::collectEphemeralValues(const llvm::Function *, llvm::AssumptionCache *, SmallPtrSetImpl<const llvm::Value *> &): Assertion `I->getParent()->getParent() == F && "Found assumption for the wrong function!"' failed.
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
Fuchsia clang version 11.0.0 (https://fuchsia.googlesource.com/a/third_party/llvm-project 9ed9860d8774b364412ce50bf5d59a8eaee1adb7)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /b/s/w/ir/k/recipe_cleanup/clangy7u70P/llvm_build_dir/./bin
clang++: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang++: note: diagnostic msg: /tmp/APFloat-fb32b4.cpp
clang++: note: diagnostic msg: /tmp/APFloat-fb32b4.sh
clang++: note: diagnostic msg: 

********************

Would you mind taking a look and sending out a fix or reverting? Thanks.

Builder: https://luci-milo.appspot.com/p/fuchsia/builders/prod/clang-linux-x64/b8880367188790280224

Looking right now - thanks!

llvm/include/llvm/Analysis/InliningAdvisor.h

+//===- InliningAdvisor.h - Inlining decision making abstraction -*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+#ifndef LLVM_INLININGADVISOR_H_
+#define LLVM_INLININGADVISOR_H_
+
+#include <memory>
+#include <unordered_set>
+#include <vector>
+
+#include "llvm/Analysis/InlineCost.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+class BasicBlock;
+class CallBase;
+class Function;
+class Module;
+class PreservedAnalyses;
+class OptimizationRemarkEmitter;
+
+/// There are 3 scenarios we can use the InliningAdvisor:
+/// - Default - use manual heuristics.
+///
+/// - Release mode, the expected mode for production, day to day deployments.
+/// In this mode, we compile ahead of time a pre-trained ML model. This mode has
+/// low runtime overhead, and no additional runtime dependencies.
+///
+/// - Development mode, for training new models.
+/// In this mode, we trade off runtime performance for flexibility. This mode
+/// requires the full C Tensorflow API library, and evaluates models
+/// dynamically. This mode also permits generating training logs, for offline
+/// training.
+enum class InliningAdvisorMode : int { Default, Release, Development };
+
+class InliningAdvisor;
+/// Capture state between an inlining decision having had been made, and
+/// its impact being observable. When collecting model training data, this
+/// allows recording features/decisions/partial reward data sets.
+///
+/// Derivations of this type are expected to be tightly coupled with their
+/// InliningAdvisors. The base type implements the minimal contractual
+/// obligations.
+class PendingInliningRecord {
+public:
+  PendingInliningRecord(PendingInliningRecord &&) = delete;
+  PendingInliningRecord(const PendingInliningRecord &) = delete;
+  virtual ~PendingInliningRecord() {
+    assert(Recorded && "PendingInliningRecord should have been informed of the "
+                       "inliner's decision in all cases");
+  }
+
+  /// Exactly one of the record* APIs must be called. Implementers may extend
+  /// behavior by implementing the corresponding record*Impl.
+  ///
+  /// Call after inlining succeeded, and did not result in deleting the callee.
+  void recordInlining() {
+    markRecorded();
+    recordInliningImpl();
+  }
+
+  /// Call after inlining succeeded, and resulted in deleting the callee.
+  void recordInliningWithCalleeDeleted();
+
+  /// Call after the decision for a call site was to not inline.
+  void recordUnsuccessfulInlining(const InlineResult &Result) {
+    markRecorded();
+    recordUnsuccessfulInliningImpl(Result);
+  }
+
+  /// Call to indicate inlining was not attempted.
+  void recordUnattemptedInlining() {
+    markRecorded();
+    recordUnattemptedInliningImpl();
+  }
+
+  /// Get the inlining recommendation.
+  bool isInliningRecommended() const { return IsInliningRecommended; }
+
+protected:
+  PendingInliningRecord(InliningAdvisor *Advisor, CallBase &CB,
+                        bool IsInliningRecommended);
+
+  virtual void recordInliningImpl(){};
+  virtual void recordInliningWithCalleeDeletedImpl() {}
+  virtual void recordUnsuccessfulInliningImpl(const InlineResult &Result) {}
+  virtual void recordUnattemptedInliningImpl() {}
+
+  InliningAdvisor *const Advisor;
+  /// Caller and Callee are pre-inlining.
+  Function *const Caller;
+  Function *const Callee;
+  const bool IsInliningRecommended;
+
+private:
+  void markRecorded() {
+    assert(!Recorded && "Recording should happen exactly once");
+    Recorded = true;
+  }
+
+  bool Recorded = false;
+};
+
+/// Interface for deciding whether to inline a call site or not.
+class InliningAdvisor {
+public:
+  InliningAdvisor(InliningAdvisor &&) = delete;
+  virtual ~InliningAdvisor() { freeDeletedFunctions(); };
+
+  /// Get a PendingInliningRecord containing a recommendation on whether to
+  /// inline or not. \p CB is assumed to be a direct call. \p FAM is assumed to
+  /// be up-to-date wrt previous inlining decisions.
+  /// Returns a PendingInliningRecord with the inlining recommendation.
+  virtual std::unique_ptr<PendingInliningRecord>
+  getInliningAdvice(CallBase &CB, FunctionAnalysisManager &FAM) = 0;
+
+  /// This must be called when the Inliner pass is entered, to allow the
+  /// InliningAdvisor update internal state, as result of function passes run
+  /// between Inliner pass runs (for the same module).
+  virtual void OnPassEntry(){};
+
+  /// This must be called when the Inliner pass is exited, as function passes
+  /// may be run subsequently. This allows an implementation of InliningAdvisor
+  /// to prepare for a partial update.
+  virtual void OnPassExit(){};
+
+protected:
+  InliningAdvisor() = default;
+
+  /// We may want to defer deleting functions to after the inlining for a whole
+  /// module has finished. This allows us to reliably use function pointers as
+  /// unique identifiers, as an efficient implementation detail of the
+  /// InliningAdvisor. Otherwise, it is possible the memory allocator
+  /// re-allocate Function objects at the same address of a deleted Function;
+  /// and Functions are potentially created during the function passes called
+  /// after each SCC inlining (e.g. argument promotion does that).
+  void freeDeletedFunctions();
+
+  bool isFunctionDeleted(Function *F) const {
+    return DeletedFunctions.count(F);
+  }
+
+private:
+  friend class PendingInliningRecord;
+  void markFunctionAsDeleted(Function *F);
+  std::unordered_set<Function *> DeletedFunctions;
+};
+
+/// The default (manual heuristics) implementation of the InliningAdvisor. This
+/// implementation does not need to keep state between inliner pass runs, and is
+/// reusable as-is for inliner pass test scenarios, as well as for regular use.
+class DefaultInliningAdvisor : public InliningAdvisor {
+public:
+  DefaultInliningAdvisor(InlineParams Params) : Params(Params) {}
+
+private:
+  std::unique_ptr<PendingInliningRecord>
+  getInliningAdvice(CallBase &CB, FunctionAnalysisManager &FAM) override;
+
+  void OnPassExit() override { freeDeletedFunctions(); }
+  InlineParams Params;
+};
+
+/// The InliningAdvisorAnalysis is a module pass because the InliningAdvisor
+/// needs to capture state right before inlining commences over a module.
+class InliningAdvisorAnalysis
+    : public AnalysisInfoMixin<InliningAdvisorAnalysis> {
+public:
+  static AnalysisKey Key;
+  InliningAdvisorAnalysis() = default;
+  struct Result {
+    Result(Module &M, ModuleAnalysisManager &MAM) : M(M), MAM(MAM) {}
+    bool invalidate(Module &, const PreservedAnalyses &,
+                    ModuleAnalysisManager::Invalidator &) {
+      // InliningAdvisor must be preserved across analysis invalidations.
+      return false;
+    }
+    bool tryCreate(InlineParams Params, InliningAdvisorMode Mode);
+    InliningAdvisor *getAdvisor() const { return Advisor.get(); }
+    void clear() { Advisor.reset(); }
+
+  private:
+    Module &M;
+    ModuleAnalysisManager &MAM;
+    std::unique_ptr<InliningAdvisor> Advisor;
+  };
+
+  Result run(Module &M, ModuleAnalysisManager &MAM) { return Result(M, MAM); }
+};
+
+// Default (manual policy) decision making helper APIs. Shared with the legacy
+// pass manager inliner.
+
+/// Return true if inlining of CB can block the caller from being
+/// inlined which is proved to be more beneficial. \p IC is the
+/// estimated inline cost associated with callsite \p CB.
+/// \p TotalSecondaryCost will be set to the estimated cost of inlining the
+/// caller if \p CB is suppressed for inlining.
+bool shouldBeDeferred(Function *Caller, InlineCost IC, int &TotalSecondaryCost,
+                      function_ref<InlineCost(CallBase &CB)> GetInlineCost);
+
+/// Return the cost only if the inliner should attempt to inline at the given
+/// CallSite. If we return the cost, we will emit an optimisation remark later
+/// using that cost, so we won't do so from this function. Return None if
+/// inlining should not be attempted.
+Optional<InlineCost>
+shouldInline(CallBase &CB, function_ref<InlineCost(CallBase &CB)> GetInlineCost,
+             OptimizationRemarkEmitter &ORE);
+
+/// Emit ORE message.
+void emitInlinedInto(OptimizationRemarkEmitter &ORE, DebugLoc DLoc,
+                     const BasicBlock *Block, const Function &Callee,
+                     const Function &Caller, const InlineCost &IC);
+
+/// Set the inline-remark attribute.
+void setInlineRemark(CallBase &CB, StringRef Message);
+
+/// Utility for extracting the inline cost message to a string.
+std::string inlineCostStr(const InlineCost &IC);
+} // namespace llvm
+#endif // LLVM_INLININGADVISOR_H_

llvm/include/llvm/Passes/PassBuilder.h

 
 #ifndef LLVM_PASSES_PASSBUILDER_H
 #define LLVM_PASSES_PASSBUILDER_H
 
 #include "llvm/ADT/Optional.h"
 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Transforms/IPO/Inliner.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/Scalar/LoopPassManager.h"
 #include <vector>
 
 namespace llvm {
 class StringRef;
 class AAManager;
 class TargetMachine;
public:
   /// \p Phase indicates the current ThinLTO phase.
   ModulePassManager
   buildModuleSimplificationPipeline(OptimizationLevel Level,
                                     ThinLTOPhase Phase,
                                     bool DebugLogging = false);
 
   /// Construct the module pipeline that performs inlining as well as
   /// the inlining-driven cleanups.
-  ModulePassManager buildInlinerPipeline(OptimizationLevel Level,
+  ModuleInlinerWrapperPass buildInlinerPipeline(OptimizationLevel Level,
                                                 ThinLTOPhase Phase,
                                                 bool DebugLogging = false);
 
   /// Construct the core LLVM module optimization pipeline.
   ///
   /// This pipeline focuses on optimizing the execution speed of the IR. It
   /// uses cost modeling and thresholds to balance code growth against runtime
   /// improvements. It includes vectorization and other information destroying
   /// transformations. It also cannot generally be run repeatedly on a module
   /// without potentially seriously regressing either runtime performance of

llvm/include/llvm/Transforms/IPO/Inliner.h

 //===- Inliner.h - Inliner pass and infrastructure --------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_IPO_INLINER_H
 #define LLVM_TRANSFORMS_IPO_INLINER_H
 
 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/Analysis/InlineCost.h"
+#include "llvm/Analysis/InliningAdvisor.h"
 #include "llvm/Analysis/LazyCallGraph.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Transforms/Utils/ImportedFunctionsInliningStatistics.h"
 #include <utility>
 
 namespace llvm {
 
 class AssumptionCacheTracker;

 /// It should be noted that the legacy inliners do considerably more than this
 /// inliner pass does. They provide logic for manually merging allocas, and
 /// doing considerable DCE including the DCE of dead functions. This pass makes
 /// every attempt to be simpler. DCE of functions requires complex reasoning
 /// about comdat groups, etc. Instead, it is expected that other more focused
 /// passes be composed to achieve the same end result.
 class InlinerPass : public PassInfoMixin<InlinerPass> {
 public:
-  InlinerPass(InlineParams Params = getInlineParams())
-      : Params(std::move(Params)) {}
+  InlinerPass() = default;
   ~InlinerPass();
   InlinerPass(InlinerPass &&Arg)
-      : Params(std::move(Arg.Params)),
-        ImportedFunctionsStats(std::move(Arg.ImportedFunctionsStats)) {}
+      : ImportedFunctionsStats(std::move(Arg.ImportedFunctionsStats)) {}
 
   PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
                         LazyCallGraph &CG, CGSCCUpdateResult &UR);
 
 private:
-  InlineParams Params;
   std::unique_ptr<ImportedFunctionsInliningStatistics> ImportedFunctionsStats;
 };
 
+/// Module pass, wrapping the inliner pass. This works in conjunction with the
+/// InliningAdvisorAnalysis to facilitate inlining decisions taking into account
+/// module-wide state, that need to keep track of inter-inliner pass runs, for
+/// a given module. An InliningAdvisor is configured and kept alive for the
+/// duration of the ModuleInlinerWrapperPass::run.
+class ModuleInlinerWrapperPass
+    : public PassInfoMixin<ModuleInlinerWrapperPass> {
+public:
+  ModuleInlinerWrapperPass(
+      InlineParams Params = getInlineParams(), bool Debugging = false,
+      InliningAdvisorMode Mode = InliningAdvisorMode::Default,
+      unsigned MaxDevirtIterations = 0);
+  ModuleInlinerWrapperPass(ModuleInlinerWrapperPass &&Arg) = default;
+
+  PreservedAnalyses run(Module &, ModuleAnalysisManager &);
+
+  /// Allow adding more CGSCC passes, besides inlining. This should be called
+  /// before run is called, as part of pass pipeline building.
+  CGSCCPassManager &getPM() { return PM; }
+
+private:
+  const InlineParams Params;
+  const InliningAdvisorMode Mode;
+  const unsigned MaxDevirtIterations;
+  const bool Debugging;
+  CGSCCPassManager PM;
+};
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_IPO_INLINER_H

llvm/lib/Analysis/CMakeLists.txt

add_llvm_component_library(LLVMAnalysis
   EHPersonalities.cpp
   GlobalsModRef.cpp
   GuardUtils.cpp
   HeatUtils.cpp
   IVDescriptors.cpp
   IVUsers.cpp
   IndirectCallPromotionAnalysis.cpp
   InlineCost.cpp
+  InliningAdvisor.cpp
   InstCount.cpp
   InstructionPrecedenceTracking.cpp
   InstructionSimplify.cpp
   Interval.cpp
   IntervalPartition.cpp
   LazyBranchProbabilityInfo.cpp
   LazyBlockFrequencyInfo.cpp
   LazyCallGraph.cpp

llvm/lib/Analysis/InliningAdvisor.cpp

+//===- InliningAdvisor.cpp - analysis pass implementation -----------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements InliningAdvisorAnalysis and DefaultInliningAdvisor, and
+// related types.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Analysis/InliningAdvisor.h"
+#include "llvm/ADT/Statistic.h"
+#include "llvm/Analysis/InlineCost.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <sstream>
+
+using namespace llvm;
+#define DEBUG_TYPE "inline"
+
+// This weirdly named statistic tracks the number of times that, when attempting
+// to inline a function A into B, we analyze the callers of B in order to see
+// if those would be more profitable and blocked inline steps.
+STATISTIC(NumCallerCallersAnalyzed, "Number of caller-callers analyzed");
+
+/// Flag to add inline messages as callsite attributes 'inline-remark'.
+static cl::opt<bool>
+    InlineRemarkAttribute("inline-remark-attribute", cl::init(false),
+                          cl::Hidden,
+                          cl::desc("Enable adding inline-remark attribute to"
+                                   " callsites processed by inliner but decided"
+                                   " to be not inlined"));
+
+// An integer used to limit the cost of inline deferral.  The default negative
+// number tells shouldBeDeferred to only take the secondary cost into account.
+static cl::opt<int>
+    InlineDeferralScale("inline-deferral-scale",
+                        cl::desc("Scale to limit the cost of inline deferral"),
+                        cl::init(-1), cl::Hidden);
+
+namespace llvm {
+std::basic_ostream<char> &operator<<(std::basic_ostream<char> &R,
+                                     const ore::NV &Arg) {
+  return R << Arg.Val;
+}
+
+template <class RemarkT>
+RemarkT &operator<<(RemarkT &&R, const InlineCost &IC) {
+  using namespace ore;
+  if (IC.isAlways()) {
+    R << "(cost=always)";
+  } else if (IC.isNever()) {
+    R << "(cost=never)";
+  } else {
+    R << "(cost=" << ore::NV("Cost", IC.getCost())
+      << ", threshold=" << ore::NV("Threshold", IC.getThreshold()) << ")";
+  }
+  if (const char *Reason = IC.getReason())
+    R << ": " << ore::NV("Reason", Reason);
+  return R;
+}
+} // namespace llvm
+
+namespace {
+class DefaultPendingRecord : public PendingInliningRecord {
+public:
+  DefaultPendingRecord(DefaultInliningAdvisor *Advisor, CallBase &CB,
+                       Optional<InlineCost> OIC, OptimizationRemarkEmitter &ORE)
+      : PendingInliningRecord(Advisor, CB, OIC.hasValue()), OriginalCB(&CB),
+        OIC(OIC), ORE(ORE), DLoc(CB.getDebugLoc()), Block(CB.getParent()) {}
+
+private:
+  void recordUnsuccessfulInliningImpl(const InlineResult &Result) override {
+    using namespace ore;
+    llvm::setInlineRemark(*OriginalCB, std::string(Result.getFailureReason()) +
+                                           "; " + inlineCostStr(*OIC));
+    ORE.emit([&]() {
+      return OptimizationRemarkMissed(DEBUG_TYPE, "NotInlined", DLoc, Block)
+             << NV("Callee", Callee) << " will not be inlined into "
+             << NV("Caller", Caller) << ": "
+             << NV("Reason", Result.getFailureReason());
+    });
+  }
+
+  void recordInliningWithCalleeDeletedImpl() override {
+    emitInlinedInto(ORE, DLoc, Block, *Callee, *Caller, *OIC);
+  }
+
+  void recordInliningImpl() override {
+    emitInlinedInto(ORE, DLoc, Block, *Callee, *Caller, *OIC);
+  }
+
+private:
+  CallBase *const OriginalCB;
+  Optional<InlineCost> OIC;
+  OptimizationRemarkEmitter &ORE;
+
+  // Capture the context of CB before inlining, as a successful inlining may
+  // change that context, and we want to report success or failure in the
+  // original context.
+  const DebugLoc DLoc;
+  const BasicBlock *const Block;
+};
+
+} // namespace
+
+std::unique_ptr<PendingInliningRecord>
+DefaultInliningAdvisor::getInliningAdvice(CallBase &CB,
+                                          FunctionAnalysisManager &FAM) {
+  Function &Callee = *CB.getCalledFunction();
+  Function &F = *CB.getCaller();
+  ProfileSummaryInfo *PSI = FAM.getResult<ModuleAnalysisManagerFunctionProxy>(F)
+                                .getManager()
+                                .getCachedResult<ProfileSummaryAnalysis>(
+                                    *CB.getParent()->getParent()->getParent());
+
+  auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(F);
+  std::function<AssumptionCache &(Function &)> GetAssumptionCache =
+      [&](Function &F) -> AssumptionCache & {
+    return FAM.getResult<AssumptionAnalysis>(Callee);
+  };
+  auto GetBFI = [&](Function &F) -> BlockFrequencyInfo & {
+    return FAM.getResult<BlockFrequencyAnalysis>(F);
+  };
+  auto GetTLI = [&](Function &F) -> const TargetLibraryInfo & {
+    return FAM.getResult<TargetLibraryAnalysis>(F);
+  };
+
+  auto GetInlineCost = [&](CallBase &CB) {
+    Function &Callee = *CB.getCalledFunction();
+    auto &CalleeTTI = FAM.getResult<TargetIRAnalysis>(Callee);
+    bool RemarksEnabled =
+        Callee.getContext().getDiagHandlerPtr()->isMissedOptRemarkEnabled(
+            DEBUG_TYPE);
+    return getInlineCost(CB, Params, CalleeTTI, GetAssumptionCache, {GetBFI},
+                         GetTLI, PSI, RemarksEnabled ? &ORE : nullptr);
+  };
+  auto OIC = llvm::shouldInline(CB, GetInlineCost, ORE);
+  std::unique_ptr<PendingInliningRecord> Ret(
+      new DefaultPendingRecord(this, CB, OIC, ORE));
+  return Ret;
+}
+
+PendingInliningRecord::PendingInliningRecord(InliningAdvisor *Advisor,
+                                             CallBase &CB,
+                                             bool IsInliningRecommended)
+    : Advisor(Advisor), Caller(CB.getCaller()), Callee(CB.getCalledFunction()),
+      IsInliningRecommended(IsInliningRecommended) {}
+
+void InliningAdvisor::markFunctionAsDeleted(Function *F) {
+  assert((!DeletedFunctions.count(F)) &&
+         "Cannot put cause a function to become dead twice!");
+  DeletedFunctions.insert(F);
+}
+
+void InliningAdvisor::freeDeletedFunctions() {
+  for (auto *F : DeletedFunctions)
+    delete (F);
+  DeletedFunctions.clear();
+}
+
+void PendingInliningRecord::recordInliningWithCalleeDeleted() {
+  markRecorded();
+  Advisor->markFunctionAsDeleted(Callee);
+  recordInliningWithCalleeDeletedImpl();
+}
+
+AnalysisKey InliningAdvisorAnalysis::Key;
+
+bool InliningAdvisorAnalysis::Result::tryCreate(InlineParams Params,
+                                                InliningAdvisorMode Mode) {
+  switch (Mode) {
+  case InliningAdvisorMode::Default:
+    Advisor.reset(new DefaultInliningAdvisor(Params));
+    break;
+  case InliningAdvisorMode::Development:
+    // To be added subsequently under conditional compilation.
+    break;
+  case InliningAdvisorMode::Release:
+    // To be added subsequently under conditional compilation.
+    break;
+  }
+  return !!Advisor;
+}
+
+void llvm::emitInlinedInto(OptimizationRemarkEmitter &ORE, DebugLoc DLoc,
+                           const BasicBlock *Block, const Function &Callee,
+                           const Function &Caller, const InlineCost &IC) {
+  using namespace ore;
+  ORE.emit([&]() {
+    bool AlwaysInline = IC.isAlways();
+    StringRef RemarkName = AlwaysInline ? "AlwaysInline" : "Inlined";
+    return OptimizationRemark(DEBUG_TYPE, RemarkName, DLoc, Block)
+           << ore::NV("Callee", &Callee) << " inlined into "
+           << ore::NV("Caller", &Caller) << " with " << IC;
+  });
+}
+
+/// Return true if inlining of CB can block the caller from being
+/// inlined which is proved to be more beneficial. \p IC is the
+/// estimated inline cost associated with callsite \p CB.
+/// \p TotalSecondaryCost will be set to the estimated cost of inlining the
+/// caller if \p CB is suppressed for inlining.
+bool llvm::shouldBeDeferred(
+    Function *Caller, InlineCost IC, int &TotalSecondaryCost,
+    function_ref<InlineCost(CallBase &CB)> GetInlineCost) {
+  // For now we only handle local or inline functions.
+  if (!Caller->hasLocalLinkage() && !Caller->hasLinkOnceODRLinkage())
+    return false;
+  // If the cost of inlining CB is non-positive, it is not going to prevent the
+  // caller from being inlined into its callers and hence we don't need to
+  // defer.
+  if (IC.getCost() <= 0)
+    return false;
+  // Try to detect the case where the current inlining candidate caller (call
+  // it B) is a static or linkonce-ODR function and is an inlining candidate
+  // elsewhere, and the current candidate callee (call it C) is large enough
+  // that inlining it into B would make B too big to inline later. In these
+  // circumstances it may be best not to inline C into B, but to inline B into
+  // its callers.
+  //
+  // This only applies to static and linkonce-ODR functions because those are
+  // expected to be available for inlining in the translation units where they
+  // are used. Thus we will always have the opportunity to make local inlining
+  // decisions. Importantly the linkonce-ODR linkage covers inline functions
+  // and templates in C++.
+  //
+  // FIXME: All of this logic should be sunk into getInlineCost. It relies on
+  // the internal implementation of the inline cost metrics rather than
+  // treating them as truly abstract units etc.
+  TotalSecondaryCost = 0;
+  // The candidate cost to be imposed upon the current function.
+  int CandidateCost = IC.getCost() - 1;
+  // If the caller has local linkage and can be inlined to all its callers, we
+  // can apply a huge negative bonus to TotalSecondaryCost.
+  bool ApplyLastCallBonus = Caller->hasLocalLinkage() && !Caller->hasOneUse();
+  // This bool tracks what happens if we DO inline C into B.
+  bool InliningPreventsSomeOuterInline = false;
+  unsigned NumCallerUsers = 0;
+  for (User *U : Caller->users()) {
+    CallBase *CS2 = dyn_cast<CallBase>(U);
+
+    // If this isn't a call to Caller (it could be some other sort
+    // of reference) skip it.  Such references will prevent the caller
+    // from being removed.
+    if (!CS2 || CS2->getCalledFunction() != Caller) {
+      ApplyLastCallBonus = false;
+      continue;
+    }
+
+    InlineCost IC2 = GetInlineCost(*CS2);
+    ++NumCallerCallersAnalyzed;
+    if (!IC2) {
+      ApplyLastCallBonus = false;
+      continue;
+    }
+    if (IC2.isAlways())
+      continue;
+
+    // See if inlining of the original callsite would erase the cost delta of
+    // this callsite. We subtract off the penalty for the call instruction,
+    // which we would be deleting.
+    if (IC2.getCostDelta() <= CandidateCost) {
+      InliningPreventsSomeOuterInline = true;
+      TotalSecondaryCost += IC2.getCost();
+      NumCallerUsers++;
+    }
+  }
+
+  if (!InliningPreventsSomeOuterInline)
+    return false;
+
+  // If all outer calls to Caller would get inlined, the cost for the last
+  // one is set very low by getInlineCost, in anticipation that Caller will
+  // be removed entirely.  We did not account for this above unless there
+  // is only one caller of Caller.
+  if (ApplyLastCallBonus)
+    TotalSecondaryCost -= InlineConstants::LastCallToStaticBonus;
+
+  // If InlineDeferralScale is negative, then ignore the cost of primary
+  // inlining -- IC.getCost() multiplied by the number of callers to Caller.
+  if (InlineDeferralScale < 0)
+    return TotalSecondaryCost < IC.getCost();
+
+  int TotalCost = TotalSecondaryCost + IC.getCost() * NumCallerUsers;
+  int Allowance = IC.getCost() * InlineDeferralScale;
+  return TotalCost < Allowance;
+}
+
+void llvm::setInlineRemark(CallBase &CB, StringRef Message) {
+  if (!InlineRemarkAttribute)
+    return;
+
+  Attribute Attr = Attribute::get(CB.getContext(), "inline-remark", Message);
+  CB.addAttribute(AttributeList::FunctionIndex, Attr);
+}
+
+/// Return the cost only if the inliner should attempt to inline at the given
+/// CallSite. If we return the cost, we will emit an optimisation remark later
+/// using that cost, so we won't do so from this function. Return None if
+/// inlining should not be attempted.
+Optional<InlineCost>
+llvm::shouldInline(CallBase &CB,
+                   function_ref<InlineCost(CallBase &CB)> GetInlineCost,
+                   OptimizationRemarkEmitter &ORE) {
+  using namespace ore;
+
+  InlineCost IC = GetInlineCost(CB);
+  Instruction *Call = &CB;
+  Function *Callee = CB.getCalledFunction();
+  Function *Caller = CB.getCaller();
+
+  if (IC.isAlways()) {
+    LLVM_DEBUG(dbgs() << "    Inlining " << inlineCostStr(IC)
+                      << ", Call: " << CB << "\n");
+    return IC;
+  }
+
+  if (!IC) {
+    LLVM_DEBUG(dbgs() << "    NOT Inlining " << inlineCostStr(IC)
+                      << ", Call: " << CB << "\n");
+    if (IC.isNever()) {
+      ORE.emit([&]() {
+        return OptimizationRemarkMissed(DEBUG_TYPE, "NeverInline", Call)
+               << NV("Callee", Callee) << " not inlined into "
+               << NV("Caller", Caller) << " because it should never be inlined "
+               << IC;
+      });
+    } else {
+      ORE.emit([&]() {
+        return OptimizationRemarkMissed(DEBUG_TYPE, "TooCostly", Call)
+               << NV("Callee", Callee) << " not inlined into "
+               << NV("Caller", Caller) << " because too costly to inline "
+               << IC;
+      });
+    }
+    setInlineRemark(CB, inlineCostStr(IC));
+    return None;
+  }
+
+  int TotalSecondaryCost = 0;
+  if (shouldBeDeferred(Caller, IC, TotalSecondaryCost, GetInlineCost)) {
+    LLVM_DEBUG(dbgs() << "    NOT Inlining: " << CB
+                      << " Cost = " << IC.getCost()
+                      << ", outer Cost = " << TotalSecondaryCost << '\n');
+    ORE.emit([&]() {
+      return OptimizationRemarkMissed(DEBUG_TYPE, "IncreaseCostInOtherContexts",
+                                      Call)
+             << "Not inlining. Cost of inlining " << NV("Callee", Callee)
+             << " increases the cost of inlining " << NV("Caller", Caller)
+             << " in other contexts";
+    });
+    setInlineRemark(CB, "deferred");
+    // IC does not bool() to false, so get an InlineCost that will.
+    // This will not be inspected to make an error message.
+    return None;
+  }
+
+  LLVM_DEBUG(dbgs() << "    Inlining " << inlineCostStr(IC) << ", Call: " << CB
+                    << '\n');
+  return IC;
+}
+
+std::string llvm::inlineCostStr(const InlineCost &IC) {
+  std::stringstream Remark;
+  Remark << IC;
+  return Remark.str();
+}

llvm/lib/Passes/PassBuilder.cpp

 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/DDG.h"
 #include "llvm/Analysis/DemandedBits.h"
 #include "llvm/Analysis/DependenceAnalysis.h"
 #include "llvm/Analysis/DominanceFrontier.h"
 #include "llvm/Analysis/GlobalsModRef.h"
 #include "llvm/Analysis/IVUsers.h"
+#include "llvm/Analysis/InliningAdvisor.h"
 #include "llvm/Analysis/LazyCallGraph.h"
 #include "llvm/Analysis/LazyValueInfo.h"
 #include "llvm/Analysis/LoopAccessAnalysis.h"
 #include "llvm/Analysis/LoopCacheAnalysis.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/LoopNestAnalysis.h"
 #include "llvm/Analysis/MemoryDependenceAnalysis.h"
 #include "llvm/Analysis/MemorySSA.h"
static cl::opt<bool>
     RunNewGVN("enable-npm-newgvn", cl::init(false),
               cl::Hidden, cl::ZeroOrMore,
               cl::desc("Run NewGVN instead of GVN"));
 
 static cl::opt<bool> EnableGVNHoist(
     "enable-npm-gvn-hoist", cl::init(false), cl::Hidden,
     cl::desc("Enable the GVN hoisting pass for the new PM (default = off)"));
 
+static cl::opt<InliningAdvisorMode> UseInliningAdvisor(
+    "enable-ml-inliner", cl::init(InliningAdvisorMode::Default), cl::Hidden,
+    cl::desc("Enable ML policy for inliner. Currently trained for -Oz only"),
+    cl::values(clEnumValN(InliningAdvisorMode::Default, "default",
+                          "Heuristics-based inliner version."),
+               clEnumValN(InliningAdvisorMode::Development, "development",
+                          "Use development mode (runtime-loadable model)."),
+               clEnumValN(InliningAdvisorMode::Release, "release",
+                          "Use release mode (AOT-compiled model).")));
+
 static cl::opt<bool> EnableGVNSink(
     "enable-npm-gvn-sink", cl::init(false), cl::Hidden,
     cl::desc("Enable the GVN hoisting pass for the new PM (default = off)"));
 
 static cl::opt<bool> EnableUnrollAndJam(
     "enable-npm-unroll-and-jam", cl::init(false), cl::Hidden,
     cl::desc("Enable the Unroll and Jam pass for the new PM (default = off)"));
 
if (!Level.isOptimizingForSize() && !IsCS) {
     InlineParams IP;
 
     IP.DefaultThreshold = PreInlineThreshold;
 
     // FIXME: The hint threshold has the same value used by the regular inliner.
     // This should probably be lowered after performance testing.
     // FIXME: this comment is cargo culted from the old pass manager, revisit).
     IP.HintThreshold = 325;
-
-    CGSCCPassManager CGPipeline(DebugLogging);
-
-    CGPipeline.addPass(InlinerPass(IP));
+    ModuleInlinerWrapperPass MIWP(IP, DebugLogging);
+    CGSCCPassManager &CGPipeline = MIWP.getPM();
 
     FunctionPassManager FPM;
     FPM.addPass(SROA());
     FPM.addPass(EarlyCSEPass());    // Catch trivial redundancies.
     FPM.addPass(SimplifyCFGPass()); // Merge & remove basic blocks.
     FPM.addPass(InstCombinePass()); // Combine silly sequences.
     invokePeepholeEPCallbacks(FPM, Level);
 
     CGPipeline.addPass(createCGSCCToFunctionPassAdaptor(std::move(FPM)));
 
-    MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(std::move(CGPipeline)));
+    MPM.addPass(std::move(MIWP));
 
     // Delete anything that is now dead to make sure that we don't instrument
     // dead code. Instrumentation can end up keeping dead code around and
     // dramatically increase code size.
     MPM.addPass(GlobalDCEPass());
   }
 
   if (!RunProfileGen) {
void PassBuilder::addPGOInstrPassesForO0(ModulePassManager &MPM,
   MPM.addPass(InstrProfiling(Options, IsCS));
 }
 
 static InlineParams
 getInlineParamsFromOptLevel(PassBuilder::OptimizationLevel Level) {
   return getInlineParams(Level.getSpeedupLevel(), Level.getSizeLevel());
 }
 
-ModulePassManager PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
-                                                    ThinLTOPhase Phase,
-                                                    bool DebugLogging) {
-  ModulePassManager MPM(DebugLogging);
+ModuleInlinerWrapperPass
+PassBuilder::buildInlinerPipeline(OptimizationLevel Level, ThinLTOPhase Phase,
+  InlineParams IP = getInlineParamsFromOptLevel(Level);
+  if (Phase == PassBuilder::ThinLTOPhase::PreLink && PGOOpt &&
+      PGOOpt->Action == PGOOptions::SampleUse)
+    IP.HotCallSiteThreshold = 0;
+
+  ModuleInlinerWrapperPass MIWP(IP, DebugLogging, UseInliningAdvisor,
+                                MaxDevirtIterations);
 
   // Now begin the main postorder CGSCC pipeline.
   // FIXME: The current CGSCC pipeline has its origins in the legacy pass
   // manager and trying to emulate its precise behavior. Much of this doesn't
   // make a lot of sense and we should revisit the core CGSCC structure.
-  CGSCCPassManager MainCGPipeline(DebugLogging);
+  CGSCCPassManager &MainCGPipeline = MIWP.getPM();
 
   // Note: historically, the PruneEH pass was run first to deduce nounwind and
   // generally clean up exception handling overhead. It isn't clear this is
   // valuable as the inliner doesn't currently care whether it is inlining an
   // invoke or a call.
 
-  // Run the inliner first. The theory is that we are walking bottom-up and so
-  // the callees have already been fully optimized, and we want to inline them
-  // into the callers so that our optimizations can reflect that.
-  // For PreLinkThinLTO pass, we disable hot-caller heuristic for sample PGO
-  // because it makes profile annotation in the backend inaccurate.
-  InlineParams IP = getInlineParamsFromOptLevel(Level);
-  if (Phase == ThinLTOPhase::PreLink && PGOOpt &&
-      PGOOpt->Action == PGOOptions::SampleUse)
-    IP.HotCallSiteThreshold = 0;
-  MainCGPipeline.addPass(InlinerPass(IP));
-
   if (AttributorRun & AttributorRunOption::CGSCC)
     MainCGPipeline.addPass(AttributorCGSCCPass());
 
   if (PTO.Coroutines)
     MainCGPipeline.addPass(CoroSplitPass());
 
   // Now deduce any function attributes based in the current code.
   MainCGPipeline.addPass(PostOrderFunctionAttrsPass());
 
PassBuilder::buildInlinerPipeline(OptimizationLevel Level, ThinLTOPhase Phase,
   // Lastly, add the core function simplification pipeline nested inside the
   // CGSCC walk.
   MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
       buildFunctionSimplificationPipeline(Level, Phase, DebugLogging)));
 
   for (auto &C : CGSCCOptimizerLateEPCallbacks)
     C(MainCGPipeline, Level);
 
-  // We wrap the CGSCC pipeline in a devirtualization repeater. This will try
-  // to detect when we devirtualize indirect calls and iterate the SCC passes
-  // in that case to try and catch knock-on inlining or function attrs
-  // opportunities. Then we add it to the module pipeline by walking the SCCs
-  // in postorder (or bottom-up).
-  MPM.addPass(
-      createModuleToPostOrderCGSCCPassAdaptor(createDevirtSCCRepeatedPass(
-          std::move(MainCGPipeline), MaxDevirtIterations)));
-  return MPM;
+  return MIWP;
 }
 
 ModulePassManager PassBuilder::buildModuleSimplificationPipeline(
     OptimizationLevel Level, ThinLTOPhase Phase, bool DebugLogging) {
   ModulePassManager MPM(DebugLogging);
 
   bool HasSampleProfile = PGOOpt && (PGOOpt->Action == PGOOptions::SampleUse);
 
   // In ThinLTO mode, when flattened profile is used, all the available
PassBuilder::buildLTODefaultPipeline(OptimizationLevel Level, bool DebugLogging,
 
   MPM.addPass(createModuleToFunctionPassAdaptor(std::move(PeepholeFPM)));
 
   // Note: historically, the PruneEH pass was run first to deduce nounwind and
   // generally clean up exception handling overhead. It isn't clear this is
   // valuable as the inliner doesn't currently care whether it is inlining an
   // invoke or a call.
   // Run the inliner now.
-  MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(
-      InlinerPass(getInlineParamsFromOptLevel(Level))));
+  MPM.addPass(ModuleInlinerWrapperPass(getInlineParamsFromOptLevel(Level),
+                                       DebugLogging));
 
   // Optimize globals again after we ran the inliner.
   MPM.addPass(GlobalOptPass());
 
   // Garbage collect dead functions.
   // FIXME: Add ArgumentPromotion pass after once it's ported.
   MPM.addPass(GlobalDCEPass());
 

llvm/lib/Passes/PassRegistry.def

 MODULE_ANALYSIS("lcg", LazyCallGraphAnalysis())
 MODULE_ANALYSIS("module-summary", ModuleSummaryIndexAnalysis())
 MODULE_ANALYSIS("no-op-module", NoOpModuleAnalysis())
 MODULE_ANALYSIS("profile-summary", ProfileSummaryAnalysis())
 MODULE_ANALYSIS("stack-safety", StackSafetyGlobalAnalysis())
 MODULE_ANALYSIS("verify", VerifierAnalysis())
 MODULE_ANALYSIS("pass-instrumentation", PassInstrumentationAnalysis(PIC))
 MODULE_ANALYSIS("asan-globals-md", ASanGlobalsMetadataAnalysis())
+MODULE_ANALYSIS("inlining-advisor", InliningAdvisorAnalysis())
 
 #ifndef MODULE_ALIAS_ANALYSIS
 #define MODULE_ALIAS_ANALYSIS(NAME, CREATE_PASS)                               \
   MODULE_ANALYSIS(NAME, CREATE_PASS)
 #endif
 MODULE_ALIAS_ANALYSIS("globals-aa", GlobalsAA())
 #undef MODULE_ALIAS_ANALYSIS
 #undef MODULE_ANALYSIS

 MODULE_PASS("function-import", FunctionImportPass())
 MODULE_PASS("globaldce", GlobalDCEPass())
 MODULE_PASS("globalopt", GlobalOptPass())
 MODULE_PASS("globalsplit", GlobalSplitPass())
 MODULE_PASS("hotcoldsplit", HotColdSplittingPass())
 MODULE_PASS("hwasan", HWAddressSanitizerPass(false, false))
 MODULE_PASS("khwasan", HWAddressSanitizerPass(true, true))
 MODULE_PASS("inferattrs", InferFunctionAttrsPass())
+MODULE_PASS("inliner-wrapper", ModuleInlinerWrapperPass())
 MODULE_PASS("insert-gcov-profiling", GCOVProfilerPass())
 MODULE_PASS("instrorderfile", InstrOrderFilePass())
 MODULE_PASS("instrprof", InstrProfiling())
 MODULE_PASS("internalize", InternalizePass())
 MODULE_PASS("invalidate<all>", InvalidateAllAnalysesPass())
 MODULE_PASS("ipsccp", IPSCCPPass())
 MODULE_PASS("lowertypetests", LowerTypeTestsPass(nullptr, nullptr))
 MODULE_PASS("mergefunc", MergeFunctionsPass())

llvm/lib/Transforms/IPO/Inliner.cpp

 #include "llvm/ADT/StringRef.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/BasicAliasAnalysis.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/Analysis/CallGraph.h"
 #include "llvm/Analysis/InlineCost.h"
+#include "llvm/Analysis/InliningAdvisor.h"
 #include "llvm/Analysis/LazyCallGraph.h"
 #include "llvm/Analysis/OptimizationRemarkEmitter.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/DataLayout.h"

 
 #define DEBUG_TYPE "inline"
 
 STATISTIC(NumInlined, "Number of functions inlined");
 STATISTIC(NumCallsDeleted, "Number of call sites deleted, not inlined");
 STATISTIC(NumDeleted, "Number of functions deleted because all callers found");
 STATISTIC(NumMergedAllocas, "Number of allocas merged together");
 
-// This weirdly named statistic tracks the number of times that, when attempting
-// to inline a function A into B, we analyze the callers of B in order to see
-// if those would be more profitable and blocked inline steps.
-STATISTIC(NumCallerCallersAnalyzed, "Number of caller-callers analyzed");
-
 /// Flag to disable manual alloca merging.
 ///
 /// Merging of allocas was originally done as a stack-size saving technique
 /// prior to LLVM's code generator having support for stack coloring based on
 /// lifetime markers. It is now in the process of being removed. To experiment
 /// with disabling it and relying fully on lifetime marker based stack
 /// coloring, you can pass this flag to LLVM.
 static cl::opt<bool>
     DisableInlinedAllocaMerging("disable-inlined-alloca-merging",
                                 cl::init(false), cl::Hidden);
 
-// An integer used to limit the cost of inline deferral.  The default negative
-// number tells shouldBeDeferred to only take the secondary cost into account.
-static cl::opt<int>
-    InlineDeferralScale("inline-deferral-scale",
-                        cl::desc("Scale to limit the cost of inline deferral"),
-                        cl::init(-1), cl::Hidden);
-
 namespace {
 
 enum class InlinerFunctionImportStatsOpts {
   No = 0,
   Basic = 1,
   Verbose = 2,
 };
 
 } // end anonymous namespace
 
 static cl::opt<InlinerFunctionImportStatsOpts> InlinerFunctionImportStats(
     "inliner-function-import-stats",
     cl::init(InlinerFunctionImportStatsOpts::No),
     cl::values(clEnumValN(InlinerFunctionImportStatsOpts::Basic, "basic",
                           "basic statistics"),
                clEnumValN(InlinerFunctionImportStatsOpts::Verbose, "verbose",
                           "printing of statistics for each inlined function")),
     cl::Hidden, cl::desc("Enable inliner stats for imported functions"));
 
-/// Flag to add inline messages as callsite attributes 'inline-remark'.
-static cl::opt<bool>
-    InlineRemarkAttribute("inline-remark-attribute", cl::init(false),
-                          cl::Hidden,
-                          cl::desc("Enable adding inline-remark attribute to"
-                                   " callsites processed by inliner but decided"
-                                   " to be not inlined"));
-
 LegacyInlinerBase::LegacyInlinerBase(char &ID) : CallGraphSCCPass(ID) {}
 
 LegacyInlinerBase::LegacyInlinerBase(char &ID, bool InsertLifetime)
     : CallGraphSCCPass(ID), InsertLifetime(InsertLifetime) {}
 
 /// For this class, we declare that we require and preserve the call graph.
 /// If the derived class implements this method, it should
 /// always explicitly call the implementation here.
static InlineResult inlineCallIfPossible(
   AttributeFuncs::mergeAttributesForInlining(*Caller, *Callee);
 
   if (!DisableInlinedAllocaMerging)
     mergeInlinedArrayAllocas(Caller, IFI, InlinedArrayAllocas, InlineHistory);
 
   return IR; // success
 }
 
-/// Return true if inlining of CB can block the caller from being
-/// inlined which is proved to be more beneficial. \p IC is the
-/// estimated inline cost associated with callsite \p CB.
-/// \p TotalSecondaryCost will be set to the estimated cost of inlining the
-/// caller if \p CB is suppressed for inlining.
-static bool
-shouldBeDeferred(Function *Caller, InlineCost IC, int &TotalSecondaryCost,
-                 function_ref<InlineCost(CallBase &CB)> GetInlineCost) {
-  // For now we only handle local or inline functions.
-  if (!Caller->hasLocalLinkage() && !Caller->hasLinkOnceODRLinkage())
-    return false;
-  // If the cost of inlining CB is non-positive, it is not going to prevent the
-  // caller from being inlined into its callers and hence we don't need to
-  // defer.
-  if (IC.getCost() <= 0)
-    return false;
-  // Try to detect the case where the current inlining candidate caller (call
-  // it B) is a static or linkonce-ODR function and is an inlining candidate
-  // elsewhere, and the current candidate callee (call it C) is large enough
-  // that inlining it into B would make B too big to inline later. In these
-  // circumstances it may be best not to inline C into B, but to inline B into
-  // its callers.
-  //
-  // This only applies to static and linkonce-ODR functions because those are
-  // expected to be available for inlining in the translation units where they
-  // are used. Thus we will always have the opportunity to make local inlining
-  // decisions. Importantly the linkonce-ODR linkage covers inline functions
-  // and templates in C++.
-  //
-  // FIXME: All of this logic should be sunk into getInlineCost. It relies on
-  // the internal implementation of the inline cost metrics rather than
-  // treating them as truly abstract units etc.
-  TotalSecondaryCost = 0;
-  // The candidate cost to be imposed upon the current function.
-  int CandidateCost = IC.getCost() - 1;
-  // If the caller has local linkage and can be inlined to all its callers, we
-  // can apply a huge negative bonus to TotalSecondaryCost.
-  bool ApplyLastCallBonus = Caller->hasLocalLinkage() && !Caller->hasOneUse();
-  // This bool tracks what happens if we DO inline C into B.
-  bool InliningPreventsSomeOuterInline = false;
-  unsigned NumCallerUsers = 0;
-  for (User *U : Caller->users()) {
-    CallBase *CS2 = dyn_cast<CallBase>(U);
-
-    // If this isn't a call to Caller (it could be some other sort
-    // of reference) skip it.  Such references will prevent the caller
-    // from being removed.
-    if (!CS2 || CS2->getCalledFunction() != Caller) {
-      ApplyLastCallBonus = false;
-      continue;
-    }
-
-    InlineCost IC2 = GetInlineCost(*CS2);
-    ++NumCallerCallersAnalyzed;
-    if (!IC2) {
-      ApplyLastCallBonus = false;
-      continue;
-    }
-    if (IC2.isAlways())
-      continue;
-
-    // See if inlining of the original callsite would erase the cost delta of
-    // this callsite. We subtract off the penalty for the call instruction,
-    // which we would be deleting.
-    if (IC2.getCostDelta() <= CandidateCost) {
-      InliningPreventsSomeOuterInline = true;
-      TotalSecondaryCost += IC2.getCost();
-      NumCallerUsers++;
-    }
-  }
-
-  if (!InliningPreventsSomeOuterInline)
-    return false;
-
-  // If all outer calls to Caller would get inlined, the cost for the last
-  // one is set very low by getInlineCost, in anticipation that Caller will
-  // be removed entirely.  We did not account for this above unless there
-  // is only one caller of Caller.
-  if (ApplyLastCallBonus)
-    TotalSecondaryCost -= InlineConstants::LastCallToStaticBonus;
-
-  // If InlineDeferralScale is negative, then ignore the cost of primary
-  // inlining -- IC.getCost() multiplied by the number of callers to Caller.
-  if (InlineDeferralScale < 0)
-    return TotalSecondaryCost < IC.getCost();
-
-  int TotalCost = TotalSecondaryCost + IC.getCost() * NumCallerUsers;
-  int Allowance = IC.getCost() * InlineDeferralScale;
-  return TotalCost < Allowance;
-}
-
-static std::basic_ostream<char> &operator<<(std::basic_ostream<char> &R,
-                                            const ore::NV &Arg) {
-  return R << Arg.Val;
-}
-
-template <class RemarkT>
-RemarkT &operator<<(RemarkT &&R, const InlineCost &IC) {
-  using namespace ore;
-  if (IC.isAlways()) {
-    R << "(cost=always)";
-  } else if (IC.isNever()) {
-    R << "(cost=never)";
-  } else {
-    R << "(cost=" << ore::NV("Cost", IC.getCost())
-      << ", threshold=" << ore::NV("Threshold", IC.getThreshold()) << ")";
-  }
-  if (const char *Reason = IC.getReason())
-    R << ": " << ore::NV("Reason", Reason);
-  return R;
-}
-
-static std::string inlineCostStr(const InlineCost &IC) {
-  std::stringstream Remark;
-  Remark << IC;
-  return Remark.str();
-}
-
-static void setInlineRemark(CallBase &CB, StringRef Message) {
-  if (!InlineRemarkAttribute)
-    return;
-
-  Attribute Attr = Attribute::get(CB.getContext(), "inline-remark", Message);
-  CB.addAttribute(AttributeList::FunctionIndex, Attr);
-}
-
-/// Return the cost only if the inliner should attempt to inline at the given
-/// CallSite. If we return the cost, we will emit an optimisation remark later
-/// using that cost, so we won't do so from this function. Return None if
-/// inlining should not be attempted.
-static Optional<InlineCost>
-shouldInline(CallBase &CB, function_ref<InlineCost(CallBase &CB)> GetInlineCost,
-             OptimizationRemarkEmitter &ORE) {
-  using namespace ore;
-
-  InlineCost IC = GetInlineCost(CB);
-  Instruction *Call = &CB;
-  Function *Callee = CB.getCalledFunction();
-  Function *Caller = CB.getCaller();
-
-  if (IC.isAlways()) {
-    LLVM_DEBUG(dbgs() << "    Inlining " << inlineCostStr(IC)
-                      << ", Call: " << CB << "\n");
-    return IC;
-  }
-
-  if (!IC) {
-    LLVM_DEBUG(dbgs() << "    NOT Inlining " << inlineCostStr(IC)
-                      << ", Call: " << CB << "\n");
-    if (IC.isNever()) {
-      ORE.emit([&]() {
-        return OptimizationRemarkMissed(DEBUG_TYPE, "NeverInline", Call)
-               << NV("Callee", Callee) << " not inlined into "
-               << NV("Caller", Caller) << " because it should never be inlined "
-               << IC;
-      });
-    } else {
-      ORE.emit([&]() {
-        return OptimizationRemarkMissed(DEBUG_TYPE, "TooCostly", Call)
-               << NV("Callee", Callee) << " not inlined into "
-               << NV("Caller", Caller) << " because too costly to inline "
-               << IC;
-      });
-    }
-    setInlineRemark(CB, inlineCostStr(IC));
-    return None;
-  }
-
-  int TotalSecondaryCost = 0;
-  if (shouldBeDeferred(Caller, IC, TotalSecondaryCost, GetInlineCost)) {
-    LLVM_DEBUG(dbgs() << "    NOT Inlining: " << CB
-                      << " Cost = " << IC.getCost()
-                      << ", outer Cost = " << TotalSecondaryCost << '\n');
-    ORE.emit([&]() {
-      return OptimizationRemarkMissed(DEBUG_TYPE, "IncreaseCostInOtherContexts",
-                                      Call)
-             << "Not inlining. Cost of inlining " << NV("Callee", Callee)
-             << " increases the cost of inlining " << NV("Caller", Caller)
-             << " in other contexts";
-    });
-    setInlineRemark(CB, "deferred");
-    // IC does not bool() to false, so get an InlineCost that will.
-    // This will not be inspected to make an error message.
-    return None;
-  }
-
-  LLVM_DEBUG(dbgs() << "    Inlining " << inlineCostStr(IC) << ", Call: " << CB
-                    << '\n');
-  return IC;
-}
-
 /// Return true if the specified inline history ID
 /// indicates an inline history that includes the specified function.
 static bool inlineHistoryIncludes(
     Function *F, int InlineHistoryID,
     const SmallVectorImpl<std::pair<Function *, int>> &InlineHistory) {
   while (InlineHistoryID != -1) {
     assert(unsigned(InlineHistoryID) < InlineHistory.size() &&
            "Invalid inline history ID");

 }
 
 bool LegacyInlinerBase::runOnSCC(CallGraphSCC &SCC) {
   if (skipSCC(SCC))
     return false;
   return inlineCalls(SCC);
 }
 
-static void emitInlinedInto(OptimizationRemarkEmitter &ORE, DebugLoc &DLoc,
-                            const BasicBlock *Block, const Function &Callee,
-                            const Function &Caller, const InlineCost &IC) {
-  ORE.emit([&]() {
-    bool AlwaysInline = IC.isAlways();
-    StringRef RemarkName = AlwaysInline ? "AlwaysInline" : "Inlined";
-    return OptimizationRemark(DEBUG_TYPE, RemarkName, DLoc, Block)
-           << ore::NV("Callee", &Callee) << " inlined into "
-           << ore::NV("Caller", &Caller) << " with " << IC;
-  });
-}
-
 static bool
 inlineCallsImpl(CallGraphSCC &SCC, CallGraph &CG,
                 std::function<AssumptionCache &(Function &)> GetAssumptionCache,
                 ProfileSummaryInfo *PSI,
                 std::function<const TargetLibraryInfo &(Function &)> GetTLI,
                 bool InsertLifetime,
                 function_ref<InlineCost(CallBase &CB)> GetInlineCost,
                 function_ref<AAResults &(Function &)> AARGetter,
PreservedAnalyses InlinerPass::run(LazyCallGraph::SCC &InitialC,
                                    CGSCCAnalysisManager &AM, LazyCallGraph &CG,
                                    CGSCCUpdateResult &UR) {
   const ModuleAnalysisManager &MAM =
       AM.getResult<ModuleAnalysisManagerCGSCCProxy>(InitialC, CG).getManager();
   bool Changed = false;
 
   assert(InitialC.size() > 0 && "Cannot handle an empty SCC!");
   Module &M = *InitialC.begin()->getFunction().getParent();
   ProfileSummaryInfo *PSI = MAM.getCachedResult<ProfileSummaryAnalysis>(M);
+  auto *IAA = MAM.getCachedResult<InliningAdvisorAnalysis>(M);
+
+  Optional<DefaultInliningAdvisor> OwnedAdvisor;
+  InliningAdvisor *Advisor = nullptr;
+  if (!IAA) {
+    // It should still be possible to run the inliner as a stand-alone SCC pass,
+    // for test scenarios. In that case, we default to the
+    // DefaultInliningAdvisor, which doesn't need to keep state between SCC pass
+    // runs. It also uses just the default InlineParams.
+    OwnedAdvisor.emplace(getInlineParams());
+    Advisor = &OwnedAdvisor.getValue();
+  } else {
+    Advisor = IAA->getAdvisor();
+    assert(Advisor && "Expected a present InliningAdvisorAnalysis also have an "
+                      "InliningAdvisor initialized");
+  }
+  // Avoid subtle bugs due to alternative exits from this method - if we have
+  // an advisor, ensure it is always informed when we're done with a scc.
+  class AdvisorExitCapture final {
+    InliningAdvisor *const Advisor;
+
+  public:
+    AdvisorExitCapture(InliningAdvisor *A) : Advisor(A) {
+      Advisor->OnPassEntry();
+    }
+    ~AdvisorExitCapture() { Advisor->OnPassExit(); }
+  };
+  AdvisorExitCapture Capturer(Advisor);
 
   if (!ImportedFunctionsStats &&
       InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No) {
     ImportedFunctionsStats =
         std::make_unique<ImportedFunctionsInliningStatistics>();
     ImportedFunctionsStats->setModuleInfo(M);
   }
 
for (int I = 0; I < (int)Calls.size(); ++I) {
     // we're going to mutate this particular function we want to make sure the
     // proxy is in place to forward any invalidation events. We can use the
     // manager we get here for looking up results for functions other than this
     // node however because those functions aren't going to be mutated by this
     // pass.
     FunctionAnalysisManager &FAM =
         AM.getResult<FunctionAnalysisManagerCGSCCProxy>(*C, CG).getManager();
 
-    // Get the remarks emission analysis for the caller.
-    auto &ORE = FAM.getResult<OptimizationRemarkEmitterAnalysis>(F);
-
     std::function<AssumptionCache &(Function &)> GetAssumptionCache =
         [&](Function &F) -> AssumptionCache & {
       return FAM.getResult<AssumptionAnalysis>(F);
     };
-    auto GetBFI = [&](Function &F) -> BlockFrequencyInfo & {
-      return FAM.getResult<BlockFrequencyAnalysis>(F);
-    };
-    auto GetTLI = [&](Function &F) -> const TargetLibraryInfo & {
-      return FAM.getResult<TargetLibraryAnalysis>(F);
-    };
-
-    auto GetInlineCost = [&](CallBase &CB) {
-      Function &Callee = *CB.getCalledFunction();
-      auto &CalleeTTI = FAM.getResult<TargetIRAnalysis>(Callee);
-      bool RemarksEnabled =
-          Callee.getContext().getDiagHandlerPtr()->isMissedOptRemarkEnabled(
-              DEBUG_TYPE);
-      return getInlineCost(CB, Params, CalleeTTI, GetAssumptionCache, {GetBFI},
-                           GetTLI, PSI, RemarksEnabled ? &ORE : nullptr);
-    };
 
     // Now process as many calls as we have within this caller in the sequnece.
     // We bail out as soon as the caller has to change so we can update the
     // call graph and prepare the context of that new caller.
     bool DidInline = false;
     for (; I < (int)Calls.size() && Calls[I].first->getCaller() == &F; ++I) {
       auto &P = Calls[I];
       CallBase *CB = P.first;
for (; I < (int)Calls.size() && Calls[I].first->getCaller() == &F; ++I) {
           UR.InlinedInternalEdges.count({&N, C})) {
         LLVM_DEBUG(dbgs() << "Skipping inlining internal SCC edge from a node "
                              "previously split out of this SCC by inlining: "
                           << F.getName() << " -> " << Callee.getName() << "\n");
         setInlineRemark(*CB, "recursive SCC split");
         continue;
       }
 
-      auto OIC = shouldInline(*CB, GetInlineCost, ORE);
+      auto PendingRecord = Advisor->getInliningAdvice(*CB, FAM);
       // Check whether we want to inline this callsite.
-      if (!OIC)
+      if (!PendingRecord->isInliningRecommended()) {
+        PendingRecord->recordUnattemptedInlining();
         continue;
-      auto DoInline = [&]() -> InlineResult {
-        // Setup the data structure used to plumb customization into the
-        // `InlineFunction` routine.
-        InlineFunctionInfo IFI(
-            /*cg=*/nullptr, &GetAssumptionCache, PSI,
-            &FAM.getResult<BlockFrequencyAnalysis>(*(CB->getCaller())),
-            &FAM.getResult<BlockFrequencyAnalysis>(Callee));
+      }
+
 
-        InlineResult IR = InlineFunction(*CB, IFI);
-        if (!IR.isSuccess())
-          return IR;
+      if (!IR.isSuccess()) {
+        PendingRecord->recordUnsuccessfulInlining(IR);
+        continue;
+      }
 
-        DidInline = true;
-        InlinedCallees.insert(&Callee);
-        ++NumInlined;
 
-        // Add any new callsites to defined functions to the worklist.
-        if (!IFI.InlinedCallSites.empty()) {
-          int NewHistoryID = InlineHistory.size();
-          InlineHistory.push_back({&Callee, InlineHistoryID});
 
-          for (CallBase *ICB : reverse(IFI.InlinedCallSites)) {
-            Function *NewCallee = ICB->getCalledFunction();
-            if (!NewCallee) {
-              // Try to promote an indirect (virtual) call without waiting for
-              // the post-inline cleanup and the next DevirtSCCRepeatedPass
-              // iteration because the next iteration may not happen and we may
-              // miss inlining it.
-              if (tryPromoteCall(*ICB))
-                NewCallee = ICB->getCalledFunction();
-            }
-            if (NewCallee)
-              if (!NewCallee->isDeclaration())
-                Calls.push_back({ICB, NewHistoryID});
-          }
-        }
 
-        if (InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No)
-          ImportedFunctionsStats->recordInline(F, Callee);
 
-        // Merge the attributes based on the inlining.
-        AttributeFuncs::mergeAttributesForInlining(F, Callee);
 
-        // For local functions, check whether this makes the callee trivially
-        // dead. In that case, we can drop the body of the function eagerly
-        // which may reduce the number of callers of other functions to one,
-        // changing inline cost thresholds.
-        if (Callee.hasLocalLinkage()) {
-          // To check this we also need to nuke any dead constant uses (perhaps
-          // made dead by this operation on other functions).
-          Callee.removeDeadConstantUsers();
-          if (Callee.use_empty() && !CG.isLibFunction(Callee)) {
-            Calls.erase(
-                std::remove_if(Calls.begin() + I + 1, Calls.end(),
-                               [&](const std::pair<CallBase *, int> &Call) {
-                                 return Call.first->getCaller() == &Callee;
-                               }),
-                Calls.end());
-            // Clear the body and queue the function itself for deletion when we
-            // finish inlining and call graph updates.
-            // Note that after this point, it is an error to do anything other
-            // than use the callee's address or delete it.
-            Callee.dropAllReferences();
-            assert(find(DeadFunctions, &Callee) == DeadFunctions.end() &&
-                   "Cannot put cause a function to become dead twice!");
-            DeadFunctions.push_back(&Callee);
-          }
-        }
-        return IR;
-      };
-      // Capture the context of CB before inlining, as a successful inlining may
-      // change that context, and we want to report success or failure in the
-      // original context.
-      auto DLoc = CB->getDebugLoc();
-      auto *Block = CB->getParent();
-
-      auto Outcome = DoInline();
-      if (!Outcome.isSuccess()) {
-        using namespace ore;
-        setInlineRemark(*CB, std::string(Outcome.getFailureReason()) + "; " +
-                                 inlineCostStr(*OIC));
-        ORE.emit([&]() {
-          return OptimizationRemarkMissed(DEBUG_TYPE, "NotInlined", DLoc, Block)
-                 << NV("Callee", &Callee) << " will not be inlined into "
-                 << NV("Caller", &F) << ": "
-                 << NV("Reason", Outcome.getFailureReason());
-        });
-        continue;
-      }
-
-      emitInlinedInto(ORE, DLoc, Block, Callee, F, *OIC);
+      bool CalleeWasDeleted = false;
+          CalleeWasDeleted = true;
+      if (CalleeWasDeleted)
+        PendingRecord->recordInliningWithCalleeDeleted();
+      else
+        PendingRecord->recordInlining();
     }
 
     // Back the call index up by one to put us in a good position to go around
     // the outer loop.
     --I;
 
     if (!DidInline)
       continue;
PreservedAnalyses InlinerPass::run(LazyCallGraph::SCC &InitialC,
   // all of the trivially dead functions, updating the call graph and the CGSCC
   // pass manager in the process.
   //
   // Note that this walks a pointer set which has non-deterministic order but
   // that is OK as all we do is delete things and add pointers to unordered
   // sets.
   for (Function *DeadF : DeadFunctions) {
     // Get the necessary information out of the call graph and nuke the
-    // function there. Also, cclear out any cached analyses.
+    // function there. Also, clear out any cached analyses.
     auto &DeadC = *CG.lookupSCC(*CG.lookup(*DeadF));
     FunctionAnalysisManager &FAM =
         AM.getResult<FunctionAnalysisManagerCGSCCProxy>(DeadC, CG).getManager();
     FAM.clear(*DeadF, DeadF->getName());
     AM.clear(DeadC, DeadC.getName());
     auto &DeadRC = DeadC.getOuterRefSCC();
     CG.removeDeadFunction(*DeadF);
 
     // Mark the relevant parts of the call graph as invalid so we don't visit
     // them.
     UR.InvalidatedSCCs.insert(&DeadC);
     UR.InvalidatedRefSCCs.insert(&DeadRC);
 
     // And delete the actual function from the module.
-    M.getFunctionList().erase(DeadF);
+    // The Advisor may use Function pointers to efficiently index various
+    // internal maps, e.g. for memoization. Function cleanup passes like
+    // argument promotion create new functions. It is possible for a new
+    // function to be allocated at the address of a deleted function. We could
+    // index using names, but that's inefficient. Alternatively, we let the
+    // Advisor free the functions when it sees fit.
+    DeadF->getBasicBlockList().clear();
+    M.getFunctionList().remove(DeadF);
+
     ++NumDeleted;
   }
 
   if (!Changed)
     return PreservedAnalyses::all();
 
   // Even if we change the IR, we update the core CGSCC data structures and so
   // can preserve the proxy to the function analysis manager.
   PreservedAnalyses PA;
   PA.preserve<FunctionAnalysisManagerCGSCCProxy>();
   return PA;
 }
+
+ModuleInlinerWrapperPass::ModuleInlinerWrapperPass(InlineParams Params,
+                                                   bool Debugging,
+                                                   InliningAdvisorMode Mode,
+                                                   unsigned MaxDevirtIterations)
+    : Params(Params), Mode(Mode), MaxDevirtIterations(MaxDevirtIterations),
+      Debugging(Debugging), PM(Debugging) {
+  // Run the inliner first. The theory is that we are walking bottom-up and so
+  // the callees have already been fully optimized, and we want to inline them
+  // into the callers so that our optimizations can reflect that.
+  // For PreLinkThinLTO pass, we disable hot-caller heuristic for sample PGO
+  // because it makes profile annotation in the backend inaccurate.
+  PM.addPass(InlinerPass());
+}
+
+PreservedAnalyses ModuleInlinerWrapperPass::run(Module &M,
+                                                ModuleAnalysisManager &MAM) {
+  auto &IAA = MAM.getResult<InliningAdvisorAnalysis>(M);
+  if (!IAA.tryCreate(Params, Mode)) {
+    M.getContext().emitError(
+        "Could not setup Inlining Advisor for the requested "
+        "mode and/or options");
+    return PreservedAnalyses::all();
+  }
+
+  ModulePassManager MPM(Debugging);
+  // If devirtualization iterations are requested, we wrap the CGSCC pipeline in
+  // a devirtualization repeater. This will try to detect when we devirtualize
+  // indirect calls and iterate the SCC passes in that case to try and catch
+  // knock-on inlining or function attrs opportunities. Then we add it to the
+  // module pipeline by walking the SCCs in postorder (or bottom-up).
+  if (MaxDevirtIterations > 0)
+    MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(
+        createDevirtSCCRepeatedPass(std::move(PM), MaxDevirtIterations)));
+  else
+    MPM.addPass(createModuleToPostOrderCGSCCPassAdaptor(std::move(PM)));
+  auto Ret = MPM.run(M, MAM);
+
+  IAA.clear();
+  return Ret;
+}

llvm/test/Other/new-pm-defaults.ll

 ; CHECK-EP-PEEPHOLE-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Finished llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running analysis: GlobalsAA
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
 ; CHECK-O-NEXT: Running analysis: ProfileSummaryAnalysis
-; CHECK-O-NEXT: Running pass: PassManager<{{.*}}Module{{.*}}>
+; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O-NEXT: Running analysis: InliningAdvisorAnalysis
 ; CHECK-O-NEXT: Starting llvm::Module pass manager run.
 ; CHECK-O-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>
 ; CHECK-O-NEXT: Starting CGSCC pass manager run.

llvm/test/Other/new-pm-lto-defaults.ll

 ; CHECK-O2-NEXT: Running pass: DeadArgumentEliminationPass
 ; CHECK-O2-NEXT: Running pass: ModuleToFunctionPassAdaptor<{{.*}}PassManager{{.*}}>
 ; CHECK-O2-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O3-NEXT: Running pass: AggressiveInstCombinePass
 ; CHECK-O2-NEXT: Running pass: InstCombinePass
 ; CHECK-O2-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O2-NEXT: Finished llvm::Function pass manager run.
-; CHECK-O2-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}InlinerPass>
+; CHECK-O2-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O2-NEXT: Running analysis: InliningAdvisorAnalysis
+; CHECK-O2-NEXT: Starting llvm::Module pass manager run.
+; CHECK-O2-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
+; CHECK-O2-NEXT: Starting CGSCC pass manager run.
+; CHECK-O2-NEXT: Running pass: InlinerPass
+; CHECK-O2-NEXT: Finished CGSCC pass manager run.
+; CHECK-O2-NEXT: Finished llvm::Module pass manager run.
 ; CHECK-O2-NEXT: Running pass: GlobalOptPass
 ; CHECK-O2-NEXT: Running pass: GlobalDCEPass
 ; CHECK-O2-NEXT: Running pass: ModuleToFunctionPassAdaptor<{{.*}}PassManager{{.*}}>
 ; CHECK-O2-NEXT: Starting llvm::Function pass manager run.
 ; CHECK-O2-NEXT: Running pass: InstCombinePass
 ; CHECK-EP-Peephole-NEXT: Running pass: NoOpFunctionPass
 ; CHECK-O2-NEXT: Running pass: JumpThreadingPass
 ; CHECK-O2-NEXT: Running analysis: LazyValueAnalysis

llvm/test/Other/new-pm-thinlto-defaults.ll

 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Finished llvm::Function pass manager run.
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running analysis: GlobalsAA
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
 ; CHECK-PRELINK-O-NEXT: Running analysis: ProfileSummaryAnalysis
-; CHECK-O-NEXT: Running pass: PassManager<{{.*}}Module{{.*}}>
+; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O-NEXT: Running analysis: InliningAdvisorAnalysis
 ; CHECK-O-NEXT: Starting llvm::Module pass manager run.
 ; CHECK-O-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>
 ; CHECK-O-NEXT: Starting CGSCC pass manager run.

llvm/test/Other/new-pm-thinlto-postlink-pgo-defaults.ll

 ; CHECK-O-DAG: Running analysis: BranchProbabilityAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis on foo
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run.
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running analysis: GlobalsAA
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
-; CHECK-O-NEXT: Running pass: PassManager<{{.*}}Module{{.*}}>
+; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O-NEXT: Running analysis: InliningAdvisorAnalysis
 ; CHECK-O-NEXT: Starting {{.*}}Module pass manager run.
 ; CHECK-O-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>
 ; CHECK-O-NEXT: Starting CGSCC pass manager run.

llvm/test/Other/new-pm-thinlto-postlink-samplepgo-defaults.ll

 ; CHECK-O-DAG: Running analysis: LoopAnalysis on foo
 ; CHECK-O-DAG: Running analysis: BranchProbabilityAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis on foo
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass on foo
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running analysis: GlobalsAA
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
-; CHECK-O-NEXT: Running pass: PassManager<{{.*}}Module{{.*}}>
+; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O-NEXT: Running analysis: InliningAdvisorAnalysis
 ; CHECK-O-NEXT: Starting {{.*}}Module pass manager run.
 ; CHECK-O-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>
 ; CHECK-O-NEXT: Starting CGSCC pass manager run.

llvm/test/Other/new-pm-thinlto-prelink-pgo-defaults.ll

 ; CHECK-O-NEXT: Running pass: ModuleToFunctionPassAdaptor<{{.*}}PassManager{{.*}}>
 ; CHECK-O-NEXT: Starting {{.*}}Function pass manager run.
 ; CHECK-O-NEXT: Running pass: InstCombinePass
 ; CHECK-O-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis
 ; CHECK-O-NEXT: Running analysis: AAManager
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run.
+; CHECK-O123-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O123-NEXT: Running analysis: InliningAdvisorAnalysis
+; CHECK-O123-NEXT: Starting llvm::Module pass manager run.
 ; CHECK-O123-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}PassManager<{{.*}}LazyCallGraph::SCC
 ; CHECK-O123-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O123-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O123-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy on (foo)
 ; CHECK-O123-NEXT: Running analysis: PassInstrumentationAnalysis on (foo)
 ; CHECK-O123-NEXT: Running analysis: OuterAnalysisManagerProxy
 ; CHECK-O123-NEXT: Starting CGSCC pass manager run.
 ; CHECK-O123-NEXT: Running pass: InlinerPass on (foo)
 ; CHECK-O123-NEXT: Running pass: CGSCCToFunctionPassAdaptor<{{.*}}PassManager{{.*}}>
 ; CHECK-O123-NEXT: Finished CGSCC pass manager run.
+; CHECK-O123-NEXT: Finished {{.*}}Module pass manager run.
 ; CHECK-O123-NEXT: Running pass: GlobalDCEPass
 ; CHECK-O-NEXT: Running pass: PGOInstrumentationUse
 ; CHECK-O-NEXT: Running analysis: ProfileSummaryAnalysis
 ; These next two can appear in any order since they are accessed as parameters
 ; on the same call to BlockFrequencyInfo::calculate.
 ; CHECK-O-DAG: Running analysis: BranchProbabilityAnalysis on foo
 ; CHECK-O-DAG: Running analysis: PostDominatorTreeAnalysis on foo
 ; CHECK-O-DAG: Running analysis: LoopAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: BlockFrequencyAnalysis on foo
 ; CHECK-O-NEXT: Invalidating all non-preserved analyses for:
 ; CHECK-O-NEXT: Invalidating analysis: InnerAnalysisManagerProxy
 ; CHECK-O123-NEXT: Invalidating analysis: LazyCallGraphAnalysis on
 ; CHECK-O123-NEXT: Invalidating analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
 ; CHECK-O-NEXT: Running pass: PGOIndirectCallPromotion on
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: OptimizationRemarkEmitterAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis on foo
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running analysis: GlobalsAA
 ; CHECK-O-NEXT: Running analysis: CallGraphAnalysis
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
-; CHECK-O-NEXT: Running pass: PassManager<{{.*}}Module{{.*}}>
+; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-Os-NEXT: Running analysis: InliningAdvisorAnalysis
+; CHECK-Oz-NEXT: Running analysis: InliningAdvisorAnalysis
 ; CHECK-O-NEXT: Starting {{.*}}Module pass manager run.
 ; CHECK-O-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O-NEXT: Running analysis: TargetLibraryAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>

llvm/test/Other/new-pm-thinlto-prelink-samplepgo-defaults.ll

 ; CHECK-O-DAG: Running analysis: LoopAnalysis on foo
 ; CHECK-O-DAG: Running analysis: BranchProbabilityAnalysis on foo
 ; CHECK-O-NEXT: Running analysis: PostDominatorTreeAnalysis on foo
 ; CHECK-O-NEXT: Running pass: SimplifyCFGPass on foo
 ; CHECK-O-NEXT: Finished {{.*}}Function pass manager run
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}GlobalsAA
 ; CHECK-O-NEXT: Running analysis: GlobalsAA
 ; CHECK-O-NEXT: Running pass: RequireAnalysisPass<{{.*}}ProfileSummaryAnalysis
-; CHECK-O-NEXT: Running pass: PassManager<{{.*}}Module{{.*}}>
+; CHECK-O-NEXT: Running pass: ModuleInlinerWrapperPass
+; CHECK-O-NEXT: Running analysis: InliningAdvisorAnalysis
 ; CHECK-O-NEXT: Starting {{.*}}Module pass manager run.
 ; CHECK-O-NEXT: Running pass: ModuleToPostOrderCGSCCPassAdaptor<{{.*}}LazyCallGraph{{.*}}>
 ; CHECK-O-NEXT: Running analysis: InnerAnalysisManagerProxy
 ; CHECK-O-NEXT: Running analysis: LazyCallGraphAnalysis
 ; CHECK-O-NEXT: Running analysis: FunctionAnalysisManagerCGSCCProxy
 ; CHECK-O-NEXT: Running analysis: PassInstrumentationAnalysis
 ; CHECK-O-NEXT: Running analysis: OuterAnalysisManagerProxy<{{.*}}LazyCallGraph::SCC{{.*}}>
 ; CHECK-O-NEXT: Starting CGSCC pass manager run.

llvm/test/Other/scc-deleted-printer.ll

 ; RUN: opt < %s 2>&1 -disable-output \
 ; RUN:     -passes=inline -print-before-all -print-after-all | FileCheck %s -check-prefix=INL
 ; RUN: opt < %s 2>&1 -disable-output \
 ; RUN:     -passes=inline -print-before-all -print-after-all -print-module-scope | FileCheck %s -check-prefix=INL-MOD
 
+; RUN: opt < %s 2>&1 -disable-output \
+; RUN:     -passes=inliner-wrapper -print-before-all -print-after-all | FileCheck %s -check-prefix=INL
+; RUN: opt < %s 2>&1 -disable-output \
+; RUN:     -passes=inliner-wrapper -print-before-all -print-after-all -print-module-scope | FileCheck %s -check-prefix=INL-MOD
+
 ; INL: IR Dump Before {{InlinerPass .*scc: .tester, foo}}
 ; INL-NOT: IR Dump After {{InlinerPass}}
 ; INL: IR Dump Before {{InlinerPass .*scc: .tester}}
 ; INL: IR Dump After {{InlinerPass .*scc: .tester}}
 
 ; INL-MOD: IR Dump Before {{InlinerPass .*scc: .tester, foo}}
 ; INL-MOD: IR Dump After {{InlinerPass .*invalidated: .*scc: .tester, foo}}
 ; INL-MOD: IR Dump Before {{InlinerPass .*scc: .tester}}

llvm/test/Other/scc-pass-printer.ll

 ; RUN: opt < %s 2>&1 -disable-output \
 ; RUN:     -inline -print-after-all | FileCheck %s -check-prefix=INL
 ; RUN: opt < %s 2>&1 -disable-output \
 ; RUN:     -passes=inline -print-after-all | FileCheck %s -check-prefix=INL
 ; RUN: opt < %s 2>&1 -disable-output \
+; RUN:     -passes=inliner-wrapper -print-after-all | FileCheck %s -check-prefix=INL
+; RUN: opt < %s 2>&1 -disable-output \
 ; RUN:     -inline -print-after-all -print-module-scope | FileCheck %s -check-prefix=INL-MOD
 ; RUN: opt < %s 2>&1 -disable-output \
 ; RUN:     -passes=inline -print-after-all -print-module-scope | FileCheck %s -check-prefix=INL-MOD
+; RUN: opt < %s 2>&1 -disable-output \
+; RUN:     -passes=inliner-wrapper -print-after-all -print-module-scope | FileCheck %s -check-prefix=INL-MOD
 
 ; INL: IR Dump After {{Function Integration/Inlining|InlinerPass .*scc: .bar, foo}}
 ; INL: define void @bar()
 ; INL-NEXT:  call void @foo()
 ; INL: define void @foo()
 ; INL-NEXT:   call void @bar()
 ; INL: IR Dump After {{Function Integration/Inlining|InlinerPass .*scc: .tester}}
 ; INL: define void @tester()

llvm/test/Transforms/Inline/inline_stats.ll

 ; First with legacy PM
 ; RUN: opt -S -inline -inliner-function-import-stats=basic < %s 2>&1 | FileCheck %s -check-prefix=CHECK-BASIC -check-prefix=CHECK
 ; RUN: opt -S -inline -inliner-function-import-stats=verbose < %s 2>&1 | FileCheck %s -check-prefix="CHECK-VERBOSE" -check-prefix=CHECK
 
 ; Do again with new PM
 ; RUN: opt -S -passes=inline -inliner-function-import-stats=basic < %s 2>&1 | FileCheck %s -check-prefix=CHECK-BASIC -check-prefix=CHECK
 ; RUN: opt -S -passes=inline -inliner-function-import-stats=verbose < %s 2>&1 | FileCheck %s -check-prefix="CHECK-VERBOSE" -check-prefix=CHECK
 
+; RUN: opt -S -passes=inliner-wrapper -inliner-function-import-stats=basic < %s 2>&1 | FileCheck %s -check-prefix=CHECK-BASIC -check-prefix=CHECK
+; RUN: opt -S -passes=inliner-wrapper -inliner-function-import-stats=verbose < %s 2>&1 | FileCheck %s -check-prefix="CHECK-VERBOSE" -check-prefix=CHECK
+
 ; CHECK: ------- Dumping inliner stats for [<stdin>] -------
 ; CHECK-BASIC-NOT: -- List of inlined functions:
 ; CHECK-BASIC-NOT: -- Inlined not imported function
 ; CHECK-VERBOSE: -- List of inlined functions:
 ; CHECK-VERBOSE: Inlined not imported function [internal2]: #inlines = 6, #inlines_to_importing_module = 2
 ; CHECK-VERBOSE: Inlined imported function [external2]: #inlines = 4, #inlines_to_importing_module = 1
 ; CHECK-VERBOSE: Inlined imported function [external1]: #inlines = 3, #inlines_to_importing_module = 2
 ; CHECK-VERBOSE: Inlined imported function [external5]: #inlines = 1, #inlines_to_importing_module = 1

llvm/test/Transforms/Inline/inlining-advisor-default.ll

+; Check that, in the absence of dependencies, we emit an error message when
+; trying to use ML-driven inlining.
+;
+; RUN: not opt -passes=scc-oz-module-inliner -enable-ml-inliner=development -S < %s 2>&1 | FileCheck %s
+; RUN: not opt -passes=scc-oz-module-inliner -enable-ml-inliner=release -S < %s 2>&1 | FileCheck %s
+
+declare i64 @f1()
+
+; CHECK: Could not setup Inlining Advisor for the requested mode and/or options
 No newline at end of file

llvm/test/Transforms/Inline/internal-scc-members.ll

 ; Test that the inliner can handle deleting functions within an SCC while still
 ; processing the calls in that SCC.
 ;
 ; RUN: opt < %s -S -inline | FileCheck %s
 ; RUN: opt < %s -S -passes=inline | FileCheck %s
+; RUN: opt < %s -S -passes=inliner-wrapper | FileCheck %s
 
 ; CHECK-LABEL: define internal void @test1_scc0()
 ; CHECK-NOT: call
 ; CHECK: call void @test1_scc0()
 ; CHECK-NOT: call
 ; CHECK: ret
 define internal void @test1_scc0() {
 entry:

llvm/test/Transforms/Inline/module-inlining.ll

 ; modify_value will be inlined into main. With just the inliner pass, at most 
 ; some trivial DCE would happen, which in this case doesn't modify post-inlined
 ; main much.
 ; In contrast, with the full set of module inliner-related passes, at the end of
 ; inlining (incl. function cleanups ran after inlining), main will be reduced to
 ; a 'ret 10'
 ;
 ; RUN: opt -passes=inline -S < %s | FileCheck %s --check-prefix=INLINE --check-prefix=CHECK
+; RUN: opt -passes=inliner-wrapper -S < %s | FileCheck %s --check-prefix=INLINE --check-prefix=CHECK
 ; RUN: opt -passes=scc-oz-module-inliner -S < %s | FileCheck %s --check-prefix=MODULE --check-prefix=CHECK
 
 define void @modify_value({i32, float}* %v) {
     %f = getelementptr { i32, float }, { i32, float }* %v, i64 0, i32 0
     store i32 10, i32* %f
     ret void
 }
 
 define i32 @main() {
     %my_val = alloca {i32, float}
     call void @modify_value({i32, float}* %my_val)
     %f = getelementptr { i32, float }, { i32, float }* %my_val, i64 0, i32 0
     %ret = load i32, i32* %f
     ret i32 %ret
 }
 
 ; CHECK-LABEL: @main
 ; INLINE-NEXT: %my_val = alloca
 ; MODULE-NEXT: ret i32 10
 No newline at end of file

llvm/test/Transforms/Inline/monster_scc.ll

 ;   }
 ;   template <> void f<false, MAX>(bool *B, bool *E) { return f<false, 0>(B, E); }
 ;   template <> void f<true, MAX>(bool *B, bool *E) { return f<true, 0>(B, E); }
 ;
 ;   void test(bool *B, bool *E) { f<false, 0>(B, E); }
 ;
 ; RUN: opt -S < %s -inline -inline-threshold=150 | FileCheck %s --check-prefixes=CHECK,OLD
 ; RUN: opt -S < %s -passes=inline -inline-threshold=150 | FileCheck %s --check-prefixes=CHECK,NEW
+; RUN: opt -S < %s -passes=inliner-wrapper -inline-threshold=150 | FileCheck %s --check-prefixes=CHECK,NEW
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 
 declare void @_Z1gi(i32)
 
 ; CHECK-LABEL: define void @_Z1fILb0ELi0EEvPbS0_(
 ; OLD-NOT: call
 ; OLD: call void @_Z1gi(

llvm/test/Transforms/Inline/optimization-remarks-hotness-threshold.ll

 ; RUN: opt < %s -S -inline -pass-remarks=inline \
 ; RUN:    -pass-remarks-with-hotness 2>&1 | FileCheck %s
 
 ; RUN: opt < %s -S -passes=inline -pass-remarks-output=%t -pass-remarks=inline \
 ; RUN:    -pass-remarks-with-hotness -pass-remarks-hotness-threshold=1 2>&1 | \
 ; RUN:    FileCheck -allow-empty -check-prefix=THRESHOLD %s
 
+; RUN: opt < %s -S -passes=inliner-wrapper -pass-remarks-output=%t -pass-remarks=inline \
+; RUN:    -pass-remarks-with-hotness -pass-remarks-hotness-threshold=1 2>&1 | \
+; RUN:    FileCheck -allow-empty -check-prefix=THRESHOLD %s
+
 ; Check that when any threshold is specified we ignore remarks with no
 ; hotness -- these are blocks that have not been executed during training.
 
 ;  1     int foo() { return 1; }
 ;  2
 ;  3     int bar() {
 ;  4       return foo();
 ;  5     }

llvm/test/Transforms/Inline/optimization-remarks-passed-yaml.ll

 ; RUN: opt < %s -S -inline -pass-remarks-output=%t -pass-remarks=inline \
 ; RUN:    -pass-remarks-missed=inline -pass-remarks-analysis=inline \
 ; RUN:    -pass-remarks-with-hotness 2>&1 | FileCheck %s
 ; RUN: cat %t | FileCheck -check-prefix=YAML %s
 
 ; RUN: opt < %s -S -passes=inline -pass-remarks-output=%t -pass-remarks=inline \
 ; RUN:    -pass-remarks-missed=inline -pass-remarks-analysis=inline \
 ; RUN:    -pass-remarks-with-hotness 2>&1 | FileCheck %s
 ; RUN: cat %t | FileCheck -check-prefix=YAML %s
 
+; RUN: opt < %s -S -passes=inliner-wrapper -pass-remarks-output=%t -pass-remarks=inline \
+; RUN:    -pass-remarks-missed=inline -pass-remarks-analysis=inline \
+; RUN:    -pass-remarks-with-hotness 2>&1 | FileCheck %s
+; RUN: cat %t | FileCheck -check-prefix=YAML %s
+
 ; Check the YAML file for inliner-generated passed and analysis remarks.  This
 ; is the input:
 
 ;  1     int foo() { return 1; }
 ;  2
 ;  3     int bar() {
 ;  4       return foo();
 ;  5     }

llvm/test/Transforms/Inline/optimization-remarks-with-hotness.ll

 ; RUN: opt < %s -inline -pass-remarks=inline -pass-remarks-missed=inline \
 ; RUN:     -pass-remarks-analysis=inline -pass-remarks-with-hotness -S 2>&1 \
 ; RUN:     | FileCheck %s
 ; RUN: opt < %s -passes=inline -pass-remarks=inline -pass-remarks-missed=inline \
 ; RUN:     -pass-remarks-analysis=inline -pass-remarks-with-hotness -S 2>&1 \
 ; RUN:     | FileCheck %s
+; RUN: opt < %s -passes=inliner-wrapper -pass-remarks=inline -pass-remarks-missed=inline \
+; RUN:     -pass-remarks-analysis=inline -pass-remarks-with-hotness -S 2>&1 \
+; RUN:     | FileCheck %s
 
 ; CHECK: foo inlined into bar with (cost=always): always inline attribute (hotness: 30)
 ; CHECK: foz not inlined into bar because it should never be inlined (cost=never): noinline function attribute (hotness: 30)
 
 ; Function Attrs: alwaysinline nounwind uwtable
 define i32 @foo() #0 !prof !1 {
 entry:
   ret i32 4

llvm/test/Transforms/Inline/optimization-remarks-yaml.ll

 ; RUN: opt < %s -S -passes=inline -pass-remarks-missed=inline \
 ; RUN:     -pass-remarks-with-hotness -pass-remarks-hotness-threshold 100 \
 ; RUN:     -pass-remarks-output=%t.threshold 2>&1 | \
 ; RUN:     FileCheck -check-prefix=THRESHOLD %s
 ; RUN: test ! -s %t.threshold
 ; RUN: opt < %s -S -passes=inline \
 ; RUN:     -pass-remarks-with-hotness -pass-remarks-hotness-threshold 100 \
 ; RUN:     -pass-remarks-output=%t.threshold
+
+; Inliner - Module Wrapper
+; RUN: opt < %s -S -passes=inliner-wrapper -pass-remarks-missed=inline \
+; RUN:     -pass-remarks-with-hotness -pass-remarks-hotness-threshold 15 \
+; RUN:     -pass-remarks-output=%t 2>&1 | FileCheck %s
+; RUN: cat %t | FileCheck -check-prefix=YAML %s
+; RUN: opt < %s -S -passes=inliner-wrapper -pass-remarks-with-hotness -pass-remarks-output=%t
+; RUN: cat %t | FileCheck -check-prefix=YAML %s
+;
+; Verify that remarks that don't meet the hotness threshold are not output.
+; RUN: opt < %s -S -passes=inliner-wrapper -pass-remarks-missed=inline \
+; RUN:     -pass-remarks-with-hotness -pass-remarks-hotness-threshold 100 \
+; RUN:     -pass-remarks-output=%t.threshold 2>&1 | \
+; RUN:     FileCheck -check-prefix=THRESHOLD %s
+; RUN: test ! -s %t.threshold
+; RUN: opt < %s -S -passes=inliner-wrapper \
+; RUN:     -pass-remarks-with-hotness -pass-remarks-hotness-threshold 100 \
+; RUN:     -pass-remarks-output=%t.threshold
+
 ; The remarks output file should be empty.
 ; RUN: test ! -s %t.threshold
 
 ; Check the YAML file generated for inliner remarks for this program:
 ;
 ;   1  int foo();
 ;   2  int bar();
 ;   3

llvm/test/Transforms/Inline/optimization-remarks.ll

 ; RUN: opt < %s -inline -pass-remarks=inline -pass-remarks-missed=inline \
 ; RUN:       -pass-remarks-analysis=inline -S 2>&1 | \
 ; RUN:       FileCheck -check-prefix=CHECK -check-prefix=NO_HOTNESS %s
 ; RUN: opt < %s -inline -pass-remarks=inline -pass-remarks-missed=inline \
 ; RUN:       -pass-remarks-analysis=inline -pass-remarks-with-hotness -S 2>&1 | \
 ; RUN:       FileCheck -check-prefix=CHECK -check-prefix=HOTNESS %s
 
 ; RUN: opt < %s -passes=inline -pass-remarks=inline -pass-remarks-missed=inline \
 ; RUN:       -pass-remarks-analysis=inline -S 2>&1 | \
 ; RUN:       FileCheck -check-prefix=CHECK -check-prefix=NO_HOTNESS %s
 ; RUN: opt < %s -passes=inline -pass-remarks=inline -pass-remarks-missed=inline \
 ; RUN:       -pass-remarks-analysis=inline -pass-remarks-with-hotness -S 2>&1 | \
 ; RUN:       FileCheck -check-prefix=CHECK -check-prefix=HOTNESS %s
 
+; RUN: opt < %s -passes=inliner-wrapper -pass-remarks=inline -pass-remarks-missed=inline \
+; RUN:       -pass-remarks-analysis=inline -S 2>&1 | \
+; RUN:       FileCheck -check-prefix=CHECK -check-prefix=NO_HOTNESS %s
+; RUN: opt < %s -passes=inliner-wrapper -pass-remarks=inline -pass-remarks-missed=inline \
+; RUN:       -pass-remarks-analysis=inline -pass-remarks-with-hotness -S 2>&1 | \
+; RUN:       FileCheck -check-prefix=CHECK -check-prefix=HOTNESS %s
+
 ; HOTNESS: fox will not be inlined into bar because its definition is unavailable
 ; NO_HOTNESS-NOT: fox will not be inlined into bar because its definition is unavailable
 ; CHECK: foo inlined into bar with (cost=always): always inline attribute
 ; CHECK: foz not inlined into bar because it should never be inlined (cost=never): noinline function attribute
 
 ; Function Attrs: alwaysinline nounwind uwtable
 define i32 @foo(i32 %x, i32 %y) #0 !prof !1 {
 entry: