[FuncSpec] Make the Function Specializer part of the IPSCCP pass.

[labrinea]

Summary

The aim of this patch is to minimize the compilation time overhead of running Function Specialization. It is about 40% slower to run as a standalone pass (IPSCCP + FuncSpec vs IPSCCP with FuncSpec) according to my measurements. I compiled the llvm testsuite with NewPM-O3 + LTO and measured single threaded [user + system] time of IPSCCP and FuncSpec by passing the '-time-passes' option to lld. Then I compared the two configurations in terms of Instruction Count of the total compilation (not of the individual passes) as in https://llvm-compile-time-tracker.com. Geomean for non-LTO builds is -0.25% and LTO is -0.5% approximately.

You can find more info below:
https://discourse.llvm.org/t/rfc-should-we-enable-function-specialization/61518

[labrinea]

This is a proof of concept for RFC: Should we enable Function Specialization?, not ready for review.

[labrinea]

rebased + fixed tests

[sinan]

tryToReplaceWithConstant method in SCCP does not update the lattice value map at SCCPSolver, and it might lead to a problem that

if

%arg = getelementptr %struct, %struct* @Global, i32 0, i32 3
%tmp0 = call i64 @func2(i64* %arg)

is folded into

%tmp0 = call i64 @func2(i64* getelementptr inbounds %struct, %struct* @Global, i32 0, i32 3)

a new callbase argument appears, but it is not recorded at SCCPSolver, and this leads to problems such as D128822.

Suggestion:
FunctionSpecializer::tryToReplaceWithConstant use the code piece below to update the propagated argument, and maybe we need such a change for tryToReplaceWithConstant as well.

for (auto *I : UseInsts)
  Solver.visit(I);

[labrinea]

In D126455#3644307, @sinan wrote:

tryToReplaceWithConstant method in SCCP does not update the lattice value map at SCCPSolver, and it might lead to a problem that

if

%arg = getelementptr %struct, %struct* @Global, i32 0, i32 3
%tmp0 = call i64 @func2(i64* %arg)

is folded into

%tmp0 = call i64 @func2(i64* getelementptr inbounds %struct, %struct* @Global, i32 0, i32 3)

a new callbase argument appears, but it is not recorded at SCCPSolver, and this leads to problems such as D128822.

Suggestion:
FunctionSpecializer::tryToReplaceWithConstant use the code piece below to update the propagated argument, and maybe we need such a change for tryToReplaceWithConstant as well.

for (auto *I : UseInsts)
  Solver.visit(I);

I am doing this in a later patch (see D126456) as I wanted to keep this one as close as possible to the original implementation.

[labrinea]

ping

[fhahn]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

301

can you fix the indentation here in a NFC?

764

IIUC this is done during in between solver runs, right? Is this needed? Isn't it sufficient to continue with the constant value in the value mapping? This would probably remove the need to tell the solver to forget instructions/values.

[labrinea]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

301

How? It seems ok.

764

This is not the only invocation of tryToReplaceWithConstant in FuncSpec. On this instance we try to replace the arguments of cloned functions. There's another invocation inside the functor RunSCCPSolver. On that instance we try to replace the instructions of cloned functions. Both calls occur as many times as FuncSpecializationMaxIters is set to.

Moreover, the SCCP pass itself does the same thing on arguments of tracked functions and on instructions of executable blocks (with tryToReplaceWithConstant and simplifyInstsInBlock accordingly). This happens after the Solver runs and before the Function Specializer is invoked. Therefore, I think we still need to tell the Solver to forget instructions/values if we want to merge the two passes.

[chill]

llvm/lib/Transforms/Scalar/SCCP.cpp

484

IMHO, the invocation of the FunctionSpecialization pass ought to happen in this place. The general flow would be like:

1. Initialise solver
2. Run solver once (Solver.solve() + resolvedUndefsIn loop)
3. Run function specialisation
4. Run solver again
5. Optionally go to 2.
6. Do replacements (from line 512 on)

At no point before the last step the passes ought to replace or delete anything (well, except called function operand for cloned functions). If an operand/argument is determined to be a constant, it does not need to be replaced right away, because the passes should consult its lattice value.

Yeah, the devil is in the details, but this is the approach to merging the tow passes, as I see it.

[labrinea]

I have moved the invocation of the specializer earlier in the ipsccp pass, such that no instructions get deleted until all the solving is done. This essentially makes all of D128822, D128823, D128824, D128825, D126456 and D128827 obsolete. There is one thing I couldn't get working, which is to update the lattice value of the callsites to specialized functions. Unfortunately the semantics of the solver do not allow lattices to move from a generic to a more specific state (i.e. from a wider to a narrower constant range). That said the zapping of returned values won't work on specialized functions, neither we can propagate a constant returned value to the function body where the callsite resides. On another note, the function analysis information which is provided to the pass (predication info, dominator tree) cannot be used on the specialized functions; not sure if that's a problem though.

[labrinea]

llvm/lib/Transforms/Scalar/SCCP.cpp

143–154

Just found that we need to do the same inside replaceSignedInst() too. I will move this code a function.

[chill]

llvm/lib/Transforms/Scalar/CMakeLists.txt

97

Why add IPO here ?

llvm/lib/Transforms/Scalar/SCCP.cpp

143–154

Would it be possible to call markUsersAsChanged here ?

147–148

Is there a specific reason to remove the instruction from the block?
If not, I'd suggest doing deletion in a single place, as opposed to spreading parts of it all over.

160

Likewise.

[labrinea]

llvm/lib/Transforms/Scalar/CMakeLists.txt

97

I vaguely remember a link time error without this change. See also llvm/utils/gn/secondary/llvm/lib/Transforms/Scalar/BUILD.gn at the bottom of this diff. The IPSCCP pass now depends on the FunctionSpecializer whose cpp file is under the IPO directory.

llvm/lib/Transforms/Scalar/SCCP.cpp

143–154

I think we can't because if we replace the uses first then the users of the old value will be empty. Can we markUsersAsChanged before we replaceAllUsesWith the new value? Btw markUsersAsChanged is private for the SCCPInstVisitor, but I suppose I could make it public if need be.

147–148

I am not entirely sure. I wanted to avoid revisiting this instruction accidentally in either of simplifyInstsInBlock(), solve(), or resolvedUndefsIn(). For simplifyInstsInBlock() I could skip the instruction if it's present in ToDelete. For the others I don't know what the consequences of revisitng would be. I need to run some tests first.

[labrinea]

llvm/lib/Transforms/Scalar/SCCP.cpp

143–154

Actually I could call markUsersAsChanged on the new Instrcution after replacing the uses of the old Instruction with it.

[chill]

llvm/lib/Transforms/Scalar/CMakeLists.txt

97

IPO already depends on Scalar, i.e. in IPO/CMakeLists.txt we have

...

  COMPONENT_NAME
  IPO

  LINK_COMPONENTS
...
  Scalar
...

Looks like a circular dependency.

Perhaps FunctionSpecialization needs to go to Utils (alongside SCCPSolver).
Or runIPSCCP needs to go to IPO/SCCP.cpp.
Or both.

llvm/lib/Transforms/Scalar/SCCP.cpp

143–154

OK, let's leave it hanging for now, until I can take a look on top of the latest trunk.

Ideally, we are trying to avoid changing code until the Solver is done.
Here we have found that an instruction has constant lattice value - we should not replace the users' operands
right away, but notify the Solver. The Solver in turn would add the instructions that need reexamining to the instructions worklist and update their lattice values the next time we invoke Solvet.solve().

Most likely SCCPSolver::visit should become private, the Solver (and the SCCP algorithm in general) is
driven by its worklists, we should stick to this design: want something done - add it to the worklist.

147–148

I can't see why would anything go wrong if the instruction is revisited.
Do we know if the instruction is safe to remove? It could be SDiv/SRem with a zero divisor.

[chill]

llvm/lib/Transforms/Scalar/SCCP.cpp

147–148

Actually, never mind, we're not replacing the instruction with a constant but with another instruction.

[labrinea]

llvm/lib/Transforms/Scalar/SCCP.cpp

143–154

Update: I tried this. It works for 'some' cases. Instead of replacing values with constants I create mappings from the old to the new value and only after all the solving is done then I replace the uses. The specialization of recursive functions doesn't work because it relies on finding allocas of constant integers. Also the rewriting of callsites doesn't work either if the actual arguments have been constant propagated prior to specialization, but the old value hasn't been replaced yet. In theory I could pass on the mappings from sccp to the specializer but it seems overly complicated to do so.

[labrinea]

Changes from last revision:

• rebased on top of main; that required adjusting a couple of pass-manager tests as the LoopInfo analysis is now used in the default pipelines by the sccp pass
• lazily eraseFromParent instructions which have been replaced instead of removeFromParent and deleteValue later as suggested by @chill
• used markUsersAsChanged instead of visit as suggested by @chill (required exposing it to the public interface)
• moved/renamed solveWhileResolvedUndefsIn to the solver as it is required from the specializer too (fixes a bug I have added a new testcase for)
• changed createSpecialization to return a pointer to the cloned function

[chill]

llvm/lib/Transforms/IPO/SCCP.cpp

41

This part was added for the FunctionSpecialization, if func spec is disabled maybe not pass along the LoopAnalysis?

[chill]

llvm/lib/Transforms/IPO/SCCP.cpp

48

Now that we added LoopAnalysis we may well preserve it too. (I should have included it in the patch which introduced the LoopAnalysis here)

[labrinea]

Good point. Running the LoopInfo analysis when we do not specialize adds unnecessary compile time overhead. After some investigation I found that notifying the users after replacing a value (instead of notifying only the old users), as well as not removing replaced instructions, both add significant compile time overhead. I am inclined to revert the latter and rework on the former so that markUsersAsChanged can accept a list of users.

[labrinea]

Changes from last revision:

• Predicated the LoopInfo analysis on the cmd-line option that enables function specialization. As a result we no longer need to adjust the pass manager unit-tests.
• Reverted the eraseFromParent to removeFromParent/deleteValue for replaced instructions to save compilation time.
• Modified markUsersAsChanged to accept a UserList so that we avoid revisiting irrelevant users after replacing a value (saves compilation time).

Note: The instruction count delta for CTMark with NewPM-O3+LTO between baseline (da5ded4fc9d8c8edfd4a79fa0e75c2ac9165fa7b) and this patch (with funcspec disabled) is about 0.05 % geomean.

[labrinea]

llvm/lib/Transforms/IPO/SCCP.cpp

48

I tried this but the compiler crashes. Probably because SCCP deletes dead basic blocks.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

273

WorkList is a too generic name for a parameter and it's not a worklist per se anyway.

388

This is a little bit fragile in the sense the caller may forget to clear the list.
It would be nicer if this function itself clears the list first thing when it starts execution.

WorkList could also be made a return, taking advantage of move semantics, which looks nicer on paper, but
may cause a few allocations/deallocations if we iterate.

llvm/lib/Transforms/Scalar/SCCP.cpp

99

Wouldn't it work without the temporary vector?

markUsersAsChanged would go over each user, look at the user's operands (including Old),
and find the New (which is some constant) form the lattice values map.

Thus we would maybe get just:

Solver.markUsersAsChanged(Old);
Old->replaceAllUsesWith(New);

143–154

Instead of replacing values with constants I create mappings from the old to the new value ..

But isn't this what the ValueState already contains?

Also the rewriting of callsites doesn't work either if the actual arguments have been constant propagated prior to specialization, but the old value hasn't been replaced yet.

Well, FunctionSpecializer::rewriteCallSites and everything else should lookup lattice values, not work directly with operands.

But OK, let's not make too many changes at once and revisit it later.

489

I would suggest not creating a vector of all the functions in the module as they could be quite a lot (e.g. in LTO)
and thus trigger several heap allocations for WorkList.

solveWhileResolvedUndefIn is quite small and could be overloaded for a Module * parameter.

I considered making this function a template along the lines of:

template<typename RangeT>
void printNames(RangeT &&R) {
    for (auto &F : R)
      llvm::dbgs() << magic(F)->getName();
}

std::vector<llvm::Function *> v;

llvm::Module *M;

int main() {
    printNames(M->functions());
    printNames(v);
}

but couldn't come up with magic.

As for propagateConstants it could be done with a few overloads as well:

static bool propagateConstants(SCCPSolver &Solver, Function *F,
                               SmallPtrSetImpl<Instruction *> &ToDelete);

static bool propagateConstants(SCCPSolver &Solver,
                               SmallVectorImpl<Function *> &WorkList,
                               SmallPtrSetImpl<Instruction *> &ToDelete) {
  for (Function *F : WorkList)
      propagateConstants(Solve, F, ToDelete);
}

static bool propagateConstants(SCCPSolver &Solver, Module *M,
                               SmallPtrSetImpl<Instruction *> &ToDelete) {
  for (auto &F : Module)
      propagateConstants(Solve, &F, ToDelete);
}

llvm/lib/Transforms/Utils/SCCPSolver.cpp

1577 ↗

(On Diff #472022)

All the functions here forward to the Visitor, this one should also just be forwarding.

(Not sure why this proxy class exists at all, but I guess we can address it later).

[labrinea]

This revision is somewhat different from the previous ones because we no longer replace instructions/arguments whilst in the main specialization loop of the SCCP pass. Instead we use the lattice value when rewritting callsites and when promoting constant stack values. Last time I checked this was even more lightweight from the previous revision. Builds successfully the llvm-test-suite with aggressive funcspec options; haven't tried clang bootstrap yet. Also improves one of the unit tests with recursive functions.

[labrinea]

llvm/include/llvm/Transforms/IPO/FunctionSpecialization.h

70

I've removed an unused typedef from here ;)

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

224–225

I am not sure whether this is necessary. The unit tests which exercise recursion are passing without it at least.

329

We now call this function once, not for every clone as it used to be.

821–825

There might be more call sites to rewrite than those in the CallSpecBinding that we have already found, therefore we need to repeat this look up of users here, but at least it now happens once for F compared to Clones.size() times which was the case before.

825

no need to examine lattices of arguments if it's the key of the CallSpecBinding

832–833

We are modifying the list whist traversing it, so we swap the current element with the last one and reduce the iteration range by one.

835

the condition was different before, but I think this is correct

llvm/lib/Transforms/Utils/SCCPSolver.cpp

465–483 ↗

(On Diff #473696)

I couldn't template these two. The main reason was that one iterates over Function * whereas the other over Function &.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

826–835

You can swap the order of the loops and get rid of CallSiteToRewrite.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

826–835

Maybe I'm getting a bit ahead of me, but you can change the function to rewrite just a single call site and factor out the iteration over call sites. The benefit is the function becomes more reusable in as you can independently choose the set of call sites it operates upon. (Incidently, I'm planning to use it that way, but it generally a good change, even if what I have in mind turns out non-working).

[labrinea]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

826–835

You mean to traverse F's users here instead? We are iterating over them while modifying them, which is the reason why CallSiteToRewrite existed in the first place I believe. Also the dynamic cast and other checks we do: CS->getCalledFunction() == F && Solver.isBlockExecutable(CS->getParent()) (note: this one is missing from the current revision) don't need to be repeated on every iteration of the outer loop which walks the specializaions.

826–835

If I change it the way you suggest it will regress the current behaviour. Qsort() from SPEC's mcf won't specialize (it's a recursive function) and it's been quite a drive for this work. What if you alter it once you refactor how we rewrite callsites?

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

826–835

What we have now is:

forall S : Specialisations {
  forall  C : CallSites {
   do_stuff(S, C);
 }
}

What I'm suggesting is to reorder the loops

forall C: call sites {
   forall S: specialisations {
    do_stuff(S, C);
  }
}

That'll avoid the swaps/pop_back. The vector itself stays, good point.

And then I'm suggesting to move the outer loop out of the function:

func foo() {
 ...
  forall C: CallSites {
    rewriteCallSite(C)
  }
...
}

func rewriteCallSite(C) {
    forall S : Specialisations {
     do_stuff(S, C);
   }
}

Both are NFC.

Also the dynamic cast and other checks we do: CS->getCalledFunction() == F && Solver.isBlockExecutable(CS->getParent()) (note: this one is missing from the current revision) don't need to be repeated on every iteration of the outer loop which walks the specializaions.

I don't understand this. We do nothing between loops, so interchanging them will execute exactly the same operations in the loop body.

If you add these checks somewhere, it's another argument to move the iteration over call sites to the outer loop.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

830

This is better placed outside of rewriteCallSites, perhaps just after the call to rewriteCallSites.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

266

I believe the LLVM convention for these kinds of classes/methods is run, e.g. Vectorizer::run(), EarlyCSE::run(), etc.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

830

Or a better idea: get the initial size of CallSitesToRewrite, decrement that number every time you update a call site. At the end if this number drops to zero mark the function unreachable.

[chill]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

821

Use braces around the for, since there are more than two levels of nesting.

[labrinea]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

830

that won't work for dead recursive functions

[labrinea]

Changes from last revision:

• renamed specialize() to run() as suggested
• renamed rewriteCallSites to updateCallSites
• interchanged the loops in updateCallSites as suggested
• update callsites of specializations before anything else (changes the test identical-specializations.ll)
• used braces for nested loop
• clang formated

[labrinea]

Found a bug in the last revision. The swapping idiom violates the expected order of traversing the call sites to update. They are supposed to be sorted by gain.

[labrinea]

Changes from last revison:

• Used a counter for updated callsites instead of revising them at the end to identify dead functions.

[labrinea]

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

815

I'll rename this and add a comment to explain what it is used for.

[labrinea]

Changes from last revision:

• rebased
• renamed the variable which determines whether we decrement the counter of left callsites to replace
• cached the cloned function pointer to the SpecializationInfo to avoid keeping the Clone vector in sync with the Specialization vector when replacing callsites

[labrinea]

Also removed the immediate replacement of known callsites. It's better to lookup the lattice value instead. This fixed a compile time regression possibly caused by otherwise dead duplicate specializations that had to be processed by later passes.

[chill]

Would you, please, mark as done the no longer relevant comments?
I think the only issue left is with the circular dependency between libraries.

[labrinea]

In D126455#3936611, @chill wrote:

Would you, please, mark as done the no longer relevant comments?
I think the only issue left is with the circular dependency between libraries.

Ack

[labrinea]

Removed the cyclic dependency between LLVMipo and LLVMScalarOpts and rebased on top of D138654.

[chill]

LGTM, but let's give a chance for other people to have a look too. @sinan @fhahn

[labrinea]

• rebase
• migrated all funcspec tests to use the -passes= cmdline option

[fhahn]

llvm/lib/Transforms/IPO/SCCP.cpp

25

move this to the the loop below, which uses it

llvm/lib/Transforms/Utils/SCCPSolver.cpp

266 ↗

(On Diff #479303)

I think DomTreeUpdater provides a constructor that doesn't take a DT which could be used unconditionally instead of having all those if (DTU) checks spread out across various functions.

[labrinea]

• Moved a flag close to its use.
• Used a DomTreeUpdater without DT/PDT for cloned functions.

llvm/include/llvm/InitializePasses.h

 void initializeFixIrreduciblePass(PassRegistry &);
 void initializeFixupStatepointCallerSavedPass(PassRegistry&);
 void initializeFlattenCFGLegacyPassPass(PassRegistry &);
 void initializeFloat2IntLegacyPassPass(PassRegistry&);
 void initializeForceFunctionAttrsLegacyPassPass(PassRegistry&);
 void initializeForwardControlFlowIntegrityPass(PassRegistry&);
 void initializeFuncletLayoutPass(PassRegistry&);
 void initializeFunctionImportLegacyPassPass(PassRegistry&);
-void initializeFunctionSpecializationLegacyPassPass(PassRegistry &);
 void initializeGCMachineCodeAnalysisPass(PassRegistry&);
 void initializeGCModuleInfoPass(PassRegistry&);
 void initializeGVNHoistLegacyPassPass(PassRegistry&);
 void initializeGVNLegacyPassPass(PassRegistry&);
 void initializeGVNSinkLegacyPassPass(PassRegistry&);
 void initializeGlobalDCELegacyPassPass(PassRegistry&);
 void initializeGlobalMergePass(PassRegistry&);
 void initializeGlobalOptLegacyPassPass(PassRegistry&);

llvm/include/llvm/LinkAllPasses.h

ForcePassLinking() {
       (void) llvm::createFloat2IntPass();
       (void) llvm::createEliminateAvailableExternallyPass();
       (void)llvm::createScalarizeMaskedMemIntrinLegacyPass();
       (void) llvm::createWarnMissedTransformationsPass();
       (void) llvm::createHardwareLoopsPass();
       (void) llvm::createInjectTLIMappingsLegacyPass();
       (void) llvm::createUnifyLoopExitsPass();
       (void) llvm::createFixIrreduciblePass();
-      (void)llvm::createFunctionSpecializationPass();
       (void)llvm::createSelectOptimizePass();
 
       (void)new llvm::IntervalPartition();
       (void)new llvm::ScalarEvolutionWrapperPass();
       llvm::Function::Create(nullptr, llvm::GlobalValue::ExternalLinkage)->viewCFGOnly();
       llvm::RGPassManager RGM;
       llvm::TargetLibraryInfoImpl TLII;
       llvm::TargetLibraryInfo TLI(TLII);

llvm/include/llvm/Transforms/IPO.h

 //===----------------------------------------------------------------------===//
 /// createIPSCCPPass - This pass propagates constants from call sites into the
 /// bodies of functions, and keeps track of whether basic blocks are executable
 /// in the process.
 ///
 ModulePass *createIPSCCPPass();
 
 //===----------------------------------------------------------------------===//
-/// createFunctionSpecializationPass - This pass propagates constants from call
-/// sites to the specialized version of the callee function.
-ModulePass *createFunctionSpecializationPass();
-
-//===----------------------------------------------------------------------===//
 //
 /// createLoopExtractorPass - This pass extracts all natural loops from the
 /// program into a function if it can.
 ///
 Pass *createLoopExtractorPass();
 
 /// createSingleLoopExtractorPass - This pass extracts one natural loop from the
 /// program into a function if it can.  This is used by bugpoint.

llvm/include/llvm/Transforms/IPO/FunctionSpecialization.h

+//===- FunctionSpecialization.h - Function Specialization -----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This specialises functions with constant parameters. Constant parameters
+// like function pointers and constant globals are propagated to the callee by
+// specializing the function. The main benefit of this pass at the moment is
+// that indirect calls are transformed into direct calls, which provides inline
+// opportunities that the inliner would not have been able to achieve. That's
+// why function specialisation is run before the inliner in the optimisation
+// pipeline; that is by design. Otherwise, we would only benefit from constant
+// passing, which is a valid use-case too, but hasn't been explored much in
+// terms of performance uplifts, cost-model and compile-time impact.
+//
+// Current limitations:
+// - It does not yet handle integer ranges. We do support "literal constants",
+//   but that's off by default under an option.
+// - The cost-model could be further looked into (it mainly focuses on inlining
+//   benefits),
+//
+// Ideas:
+// - With a function specialization attribute for arguments, we could have
+//   a direct way to steer function specialization, avoiding the cost-model,
+//   and thus control compile-times / code-size.
+//
+// Todos:
+// - Specializing recursive functions relies on running the transformation a
+//   number of times, which is controlled by option
+//   `func-specialization-max-iters`. Thus, increasing this value and the
+//   number of iterations, will linearly increase the number of times recursive
+//   functions get specialized, see also the discussion in
+//   https://reviews.llvm.org/D106426 for details. Perhaps there is a
+//   compile-time friendlier way to control/limit the number of specialisations
+//   for recursive functions.
+// - Don't transform the function if function specialization does not trigger;
+//   the SCCPSolver may make IR changes.
+//
+// References:
+// - 2021 LLVM Dev Mtg “Introducing function specialisation, and can we enable
+//   it by default?”, https://www.youtube.com/watch?v=zJiCjeXgV5Q
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_IPO_FUNCTIONSPECIALIZATION_H
+#define LLVM_TRANSFORMS_IPO_FUNCTIONSPECIALIZATION_H
+
+#include "llvm/Analysis/CodeMetrics.h"
+#include "llvm/Analysis/InlineCost.h"
+#include "llvm/Analysis/LoopInfo.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
+#include "llvm/Transforms/Scalar/SCCP.h"
+#include "llvm/Transforms/Utils/Cloning.h"
+#include "llvm/Transforms/Utils/SCCPSolver.h"
+#include "llvm/Transforms/Utils/SizeOpts.h"
+
+using namespace llvm;
+
+namespace llvm {
+// Bookkeeping struct to pass data from the analysis and profitability phase
+// to the actual transform helper functions.
+struct SpecializationInfo {
+  SmallVector<ArgInfo, 8> Args; // Stores the {formal,actual} argument pairs.
+  InstructionCost Gain;         // Profitability: Gain = Bonus - Cost.
+};
+
+using FuncList = SmallVectorImpl<Function *>;
+using CallArgBinding = std::pair<CallBase *, Constant *>;
+using CallSpecBinding = std::pair<CallBase *, SpecializationInfo>;
+// We are using MapVector because it guarantees deterministic iteration
+// order across executions.
+using SpecializationMap = SmallMapVector<CallBase *, SpecializationInfo, 8>;
+
+class FunctionSpecializer {
+
+  /// The IPSCCP Solver.
+  SCCPSolver &Solver;
+
+  Module &M;
+
+  /// Analyses used to help determine if a function should be specialized.
+  std::function<AssumptionCache &(Function &)> GetAC;
+  std::function<TargetTransformInfo &(Function &)> GetTTI;
+  std::function<const TargetLibraryInfo &(Function &)> GetTLI;
+
+  // The number of functions specialised, used for collecting statistics and
+  // also in the cost model.
+  unsigned NbFunctionsSpecialized = 0;
+
+  SmallPtrSet<Function *, 4> SpecializedFuncs;
+  SmallPtrSet<Function *, 4> FullySpecialized;
+  DenseMap<Function *, CodeMetrics> FunctionMetrics;
+
+public:
+  FunctionSpecializer(SCCPSolver &Solver, Module &M,
+      std::function<AssumptionCache &(Function &)> GetAC,
+      std::function<TargetTransformInfo &(Function &)> GetTTI,
+      std::function<const TargetLibraryInfo &(Function &)> GetTLI)
+    : Solver(Solver), M(M), GetAC(GetAC), GetTTI(GetTTI), GetTLI(GetTLI) {}
+
+  ~FunctionSpecializer() { removeDeadFunctions(); }
+
+  bool specialize(FuncList &FuncDecls);
+
+private:
+  /// Iterate over the argument tracked functions see if there
+  /// are any new constant values for the call instruction via
+  /// stack variables.
+  void propagateConstantArgs(FuncList &WorkList);
+
+  /// Attempt to specialize functions in the module to enable constant
+  /// propagation across function boundaries.
+  ///
+  /// \returns true if at least one function is specialized.
+  bool specializeFunctions(FuncList &Candidates, FuncList &WorkList);
+
+  /// Clean up fully specialized functions.
+  void removeDeadFunctions();
+
+  // Compute the code metrics for function \p F.
+  CodeMetrics &analyzeFunction(Function *F);
+
+  /// This function decides whether it's worthwhile to specialize function
+  /// \p F based on the known constant values its arguments can take on. It
+  /// only discovers potential specialization opportunities without actually
+  /// applying them.
+  ///
+  /// \returns true if any specializations have been found.
+  bool calculateGains(Function *F, InstructionCost Cost,
+                      SmallVectorImpl<CallSpecBinding> &WorkList);
+
+  bool isCandidateFunction(Function *F);
+
+  void specializeFunction(Function *F, SpecializationInfo &S,
+                          FuncList &WorkList);
+
+  /// Compute and return the cost of specializing function \p F.
+  InstructionCost getSpecializationCost(Function *F);
+
+  /// Compute a bonus for replacing argument \p A with constant \p C.
+  InstructionCost getSpecializationBonus(Argument *A, Constant *C);
+
+  /// Determine if we should specialize a function based on the incoming values
+  /// of the given argument.
+  ///
+  /// This function implements the goal-directed heuristic. It determines if
+  /// specializing the function based on the incoming values of argument \p A
+  /// would result in any significant optimization opportunities. If
+  /// optimization opportunities exist, the constant values of \p A on which to
+  /// specialize the function are collected in \p Constants.
+  ///
+  /// \returns true if the function should be specialized on the given
+  /// argument.
+  bool isArgumentInteresting(Argument *A,
+                             SmallVectorImpl<CallArgBinding> &Constants);
+
+  /// Collect in \p Constants all the constant values that argument \p A can
+  /// take on.
+  void getPossibleConstants(Argument *A,
+                            SmallVectorImpl<CallArgBinding> &Constants);
+
+  /// Rewrite calls to function \p F to call function \p Clone instead.
+  ///
+  /// This function modifies calls to function \p F as long as the actual
+  /// arguments match those in \p Args. Note that for recursive calls we
+  /// need to compare against the cloned formal arguments.
+  ///
+  /// Callsites that have been marked with the MinSize function attribute won't
+  /// be specialized and rewritten.
+  void rewriteCallSites(Function *Clone, const SmallVectorImpl<ArgInfo> &Args,
+                        ValueToValueMapTy &Mappings);
+
+  void updateSpecializedFuncs(FuncList &Candidates, FuncList &WorkList);
+};
+} // namespace
+
+#endif // LLVM_TRANSFORMS_IPO_FUNCTIONSPECIALIZATION_H

llvm/include/llvm/Transforms/IPO/SCCP.h

 class Module;
 
 /// Pass to perform interprocedural constant propagation.
 class IPSCCPPass : public PassInfoMixin<IPSCCPPass> {
 public:
   PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
 };
 
-/// Pass to perform interprocedural constant propagation by specializing
-/// functions
-class FunctionSpecializationPass
-    : public PassInfoMixin<FunctionSpecializationPass> {
-public:
-  PreservedAnalyses run(Module &M, ModuleAnalysisManager &AM);
-};
-
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_IPO_SCCP_H

llvm/include/llvm/Transforms/Scalar/SCCP.h

 /// This pass performs function-level constant propagation and merging.
 class SCCPPass : public PassInfoMixin<SCCPPass> {
 public:
   PreservedAnalyses run(Function &F, FunctionAnalysisManager &AM);
 };
 
 bool runIPSCCP(Module &M, const DataLayout &DL,
                std::function<const TargetLibraryInfo &(Function &)> GetTLI,
-               function_ref<AnalysisResultsForFn(Function &)> getAnalysis);
-
-bool runFunctionSpecialization(
-    Module &M, const DataLayout &DL,
-    std::function<TargetLibraryInfo &(Function &)> GetTLI,
                std::function<TargetTransformInfo &(Function &)> GetTTI,
                std::function<AssumptionCache &(Function &)> GetAC,
-    function_ref<AnalysisResultsForFn(Function &)> GetAnalysis);
+               function_ref<AnalysisResultsForFn(Function &)> getAnalysis);
 } // end namespace llvm
 
 #endif // LLVM_TRANSFORMS_SCALAR_SCCP_H

llvm/lib/Passes/PassBuilderPipelines.cpp

PassBuilder::buildModuleSimplificationPipeline(OptimizationLevel Level,
   // post link pipeline after ICP. This is to enable usage of the type
   // tests in ICP sequences.
   if (Phase == ThinOrFullLTOPhase::ThinLTOPostLink)
     MPM.addPass(LowerTypeTestsPass(nullptr, nullptr, true));
 
   for (auto &C : PipelineEarlySimplificationEPCallbacks)
     C(MPM, Level);
 
-  // Specialize functions with IPSCCP.
-  if (EnableFunctionSpecialization && Level == OptimizationLevel::O3)
-    MPM.addPass(FunctionSpecializationPass());
-
   // Interprocedural constant propagation now that basic cleanup has occurred
   // and prior to optimizing globals.
   // FIXME: This position in the pipeline hasn't been carefully considered in
   // years, it should be re-analyzed.
   MPM.addPass(IPSCCPPass());
 
   // Attach metadata to indirect call sites indicating the set of functions
   // they may target at run-time. This should follow IPSCCP.
if (Level.getSpeedupLevel() > 1) {
 
     // Indirect call promotion. This should promote all the targets that are
     // left by the earlier promotion pass that promotes intra-module targets.
     // This two-step promotion is to save the compile time. For LTO, it should
     // produce the same result as if we only do promotion here.
     MPM.addPass(PGOIndirectCallPromotion(
         true /* InLTO */, PGOOpt && PGOOpt->Action == PGOOptions::SampleUse));
 
-    if (EnableFunctionSpecialization && Level == OptimizationLevel::O3)
-      MPM.addPass(FunctionSpecializationPass());
     // Propagate constants at call sites into the functions they call.  This
     // opens opportunities for globalopt (and inlining) by substituting function
     // pointers passed as arguments to direct uses of functions.
     MPM.addPass(IPSCCPPass());
 
     // Attach metadata to indirect call sites indicating the set of functions
     // they may target at run-time. This should follow IPSCCP.
     MPM.addPass(CalledValuePropagationPass());

llvm/lib/Passes/PassRegistry.def

 MODULE_PASS("cross-dso-cfi", CrossDSOCFIPass())
 MODULE_PASS("deadargelim", DeadArgumentEliminationPass())
 MODULE_PASS("debugify", NewPMDebugifyPass())
 MODULE_PASS("dot-callgraph", CallGraphDOTPrinterPass())
 MODULE_PASS("elim-avail-extern", EliminateAvailableExternallyPass())
 MODULE_PASS("extract-blocks", BlockExtractorPass())
 MODULE_PASS("forceattrs", ForceFunctionAttrsPass())
 MODULE_PASS("function-import", FunctionImportPass())
-MODULE_PASS("function-specialization", FunctionSpecializationPass())
 MODULE_PASS("globaldce", GlobalDCEPass())
 MODULE_PASS("globalopt", GlobalOptPass())
 MODULE_PASS("globalsplit", GlobalSplitPass())
 MODULE_PASS("hotcoldsplit", HotColdSplittingPass())
 MODULE_PASS("inferattrs", InferFunctionAttrsPass())
 MODULE_PASS("inliner-wrapper", ModuleInlinerWrapperPass())
 MODULE_PASS("print<inline-advisor>", InlineAdvisorAnalysisPrinterPass(dbgs()))
 MODULE_PASS("inliner-wrapper-no-mandatory-first", ModuleInlinerWrapperPass(

llvm/lib/Transforms/IPO/FunctionSpecialization.cpp

 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/CodeMetrics.h"
 #include "llvm/Analysis/InlineCost.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Analysis/ValueLattice.h"
 #include "llvm/Analysis/ValueLatticeUtils.h"
 #include "llvm/IR/IntrinsicInst.h"
+#include "llvm/Transforms/IPO/FunctionSpecialization.h"
 #include "llvm/Transforms/Scalar/SCCP.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include "llvm/Transforms/Utils/SCCPSolver.h"
 #include "llvm/Transforms/Utils/SizeOpts.h"
 #include <cmath>
 
 using namespace llvm;
 

 //
 // https://llvm-compile-time-tracker.com
 // https://github.com/nikic/llvm-compile-time-tracker
 static cl::opt<bool> EnableSpecializationForLiteralConstant(
     "function-specialization-for-literal-constant", cl::init(false), cl::Hidden,
     cl::desc("Enable specialization of functions that take a literal constant "
              "as an argument."));
 
-namespace {
-// Bookkeeping struct to pass data from the analysis and profitability phase
-// to the actual transform helper functions.
-struct SpecializationInfo {
-  SmallVector<ArgInfo, 8> Args; // Stores the {formal,actual} argument pairs.
-  InstructionCost Gain;         // Profitability: Gain = Bonus - Cost.
-};
-} // Anonymous namespace
-
-using FuncList = SmallVectorImpl<Function *>;
-using CallArgBinding = std::pair<CallBase *, Constant *>;
-using CallSpecBinding = std::pair<CallBase *, SpecializationInfo>;
-// We are using MapVector because it guarantees deterministic iteration
-// order across executions.
-using SpecializationMap = SmallMapVector<CallBase *, SpecializationInfo, 8>;
-
 // Helper to check if \p LV is either a constant or a constant
 // range with a single element. This should cover exactly the same cases as the
 // old ValueLatticeElement::isConstant() and is intended to be used in the
 // transition to ValueLatticeElement.
 static bool isConstant(const ValueLatticeElement &LV) {
   return LV.isConstant() ||
          (LV.isConstantRange() && LV.getConstantRange().isSingleElement());
 }
for (auto *User : Alloca->users()) {
     return nullptr;
   }
   return dyn_cast_or_null<Constant>(StoreValue);
 }
 
 // A constant stack value is an AllocaInst that has a single constant
 // value stored to it. Return this constant if such an alloca stack value
 // is a function argument.
-static Constant *getConstantStackValue(CallInst *Call, Value *Val,
-                                       SCCPSolver &Solver) {
+static Constant *getConstantStackValue(CallInst *Call, Value *Val) {
   if (!Val)
     return nullptr;
   Val = Val->stripPointerCasts();
   if (auto *ConstVal = dyn_cast<ConstantInt>(Val))
     return ConstVal;
   auto *Alloca = dyn_cast<AllocaInst>(Val);
   if (!Alloca || !Alloca->getAllocatedType()->isIntegerTy())
     return nullptr;

 //
 //     @funcspec.arg = internal constant i32 2
 //
 //     define internal void @someFunc(i32* arg1) {
 //       call void @otherFunc(i32* nonnull @funcspec.arg)
 //       ret void
 //     }
 //
-static void constantArgPropagation(FuncList &WorkList, Module &M,
-                                   SCCPSolver &Solver) {
+void FunctionSpecializer::propagateConstantArgs(FuncList &WorkList) {
   // Iterate over the argument tracked functions see if there
   // are any new constant values for the call instruction via
   // stack variables.
   for (auto *F : WorkList) {
 
     for (auto *User : F->users()) {
 
       auto *Call = dyn_cast<CallInst>(User);
       if (!Call)
         continue;
 
       bool Changed = false;
       for (const Use &U : Call->args()) {
         unsigned Idx = Call->getArgOperandNo(&U);
         Value *ArgOp = Call->getArgOperand(Idx);
         Type *ArgOpType = ArgOp->getType();
 
         if (!Call->onlyReadsMemory(Idx) || !ArgOpType->isPointerTy())
           continue;
 
-        auto *ConstVal = getConstantStackValue(Call, ArgOp, Solver);
+        auto *ConstVal = getConstantStackValue(Call, ArgOp);
         if (!ConstVal)
           continue;
 
         Value *GV = new GlobalVariable(M, ConstVal->getType(), true,
                                        GlobalValue::InternalLinkage, ConstVal,
                                        "funcspec.arg");
         if (ArgOpType != ConstVal->getType())
           GV = ConstantExpr::getBitCast(cast<Constant>(GV), ArgOpType);
 
         Call->setArgOperand(Idx, GV);
         Changed = true;
       }
 
       // Add the changed CallInst to Solver Worklist
       if (Changed)
         Solver.visitCall(*Call);
     }
   }
 }
 
 // ssa_copy intrinsics are introduced by the SCCP solver. These intrinsics
 // interfere with the constantArgPropagation optimization.
 static void removeSSACopy(Function &F) {
   for (BasicBlock &BB : F) {
static void removeSSACopy(Function &F) {
   }
 }
 
 static void removeSSACopy(Module &M) {
   for (Function &F : M)
     removeSSACopy(F);
 }
 
-namespace {
-class FunctionSpecializer {
-
-  /// The IPSCCP Solver.
-  SCCPSolver &Solver;
-
-  /// Analyses used to help determine if a function should be specialized.
-  std::function<AssumptionCache &(Function &)> GetAC;
-  std::function<TargetTransformInfo &(Function &)> GetTTI;
-  std::function<TargetLibraryInfo &(Function &)> GetTLI;
-
-  SmallPtrSet<Function *, 4> SpecializedFuncs;
-  SmallPtrSet<Function *, 4> FullySpecialized;
-  SmallVector<Instruction *> ReplacedWithConstant;
-  DenseMap<Function *, CodeMetrics> FunctionMetrics;
-
-public:
-  FunctionSpecializer(SCCPSolver &Solver,
-                      std::function<AssumptionCache &(Function &)> GetAC,
-                      std::function<TargetTransformInfo &(Function &)> GetTTI,
-                      std::function<TargetLibraryInfo &(Function &)> GetTLI)
-      : Solver(Solver), GetAC(GetAC), GetTTI(GetTTI), GetTLI(GetTLI) {}
-
-  ~FunctionSpecializer() {
-    // Eliminate dead code.
-    removeDeadInstructions();
-    removeDeadFunctions();
-  }
-
-  /// Attempt to specialize functions in the module to enable constant
-  /// propagation across function boundaries.
-  ///
-  /// \returns true if at least one function is specialized.
-  bool specializeFunctions(FuncList &Candidates, FuncList &WorkList) {
-    bool Changed = false;
-    for (auto *F : Candidates) {
-      if (!isCandidateFunction(F))
-        continue;
+bool FunctionSpecializer::specializeFunctions(FuncList &Candidates,
+                                              FuncList &WorkList) {
 
-      auto Cost = getSpecializationCost(F);
-      if (!Cost.isValid()) {
-        LLVM_DEBUG(
-            dbgs() << "FnSpecialization: Invalid specialization cost.\n");
-        continue;
-      }
 
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Specialization cost for "
-                        << F->getName() << " is " << Cost << "\n");
 
-      SmallVector<CallSpecBinding, 8> Specializations;
-      if (!calculateGains(F, Cost, Specializations)) {
-        LLVM_DEBUG(dbgs() << "FnSpecialization: No possible constants found\n");
-        continue;
-      }
 
-      Changed = true;
-      for (auto &Entry : Specializations)
-        specializeFunction(F, Entry.second, WorkList);
-    }
 
-    updateSpecializedFuncs(Candidates, WorkList);
-    NumFuncSpecialized += NbFunctionsSpecialized;
-    return Changed;
-  }
 
-  void removeDeadInstructions() {
-    for (auto *I : ReplacedWithConstant) {
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Removing dead instruction " << *I
-                        << "\n");
-      I->eraseFromParent();
-    }
-    ReplacedWithConstant.clear();
-  }
-
-  void removeDeadFunctions() {
-    for (auto *F : FullySpecialized) {
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Removing dead function "
-                        << F->getName() << "\n");
-      F->eraseFromParent();
-    }
-    FullySpecialized.clear();
-  }
+void FunctionSpecializer::removeDeadFunctions() {
 
-  bool tryToReplaceWithConstant(Value *V) {
-    if (!V->getType()->isSingleValueType() || isa<CallBase>(V) ||
-        V->user_empty())
-      return false;
+static bool tryToReplaceWithConstant(SCCPSolver &Solver, Value *V) {
 
-    const ValueLatticeElement &IV = Solver.getLatticeValueFor(V);
-    if (isOverdefined(IV))
-      return false;
-    auto *Const =
-        isConstant(IV) ? Solver.getConstant(IV) : UndefValue::get(V->getType());
 
-    LLVM_DEBUG(dbgs() << "FnSpecialization: Replacing " << *V
-                      << "\nFnSpecialization: with " << *Const << "\n");
 
-    // Record uses of V to avoid visiting irrelevant uses of const later.
-    SmallVector<Instruction *> UseInsts;
-    for (auto *U : V->users())
-      if (auto *I = dyn_cast<Instruction>(U))
-        if (Solver.isBlockExecutable(I->getParent()))
-          UseInsts.push_back(I);
 
-    V->replaceAllUsesWith(Const);
 
-    for (auto *I : UseInsts)
-      Solver.visit(I);
 
-    // Remove the instruction from Block and Solver.
-    if (auto *I = dyn_cast<Instruction>(V)) {
-      if (I->isSafeToRemove()) {
-        ReplacedWithConstant.push_back(I);
-        Solver.removeLatticeValueFor(I);
-      }
-    }
-    return true;
-  }
+      I->eraseFromParent();
 
-private:
-  // The number of functions specialised, used for collecting statistics and
-  // also in the cost model.
-  unsigned NbFunctionsSpecialized = 0;
-
-  // Compute the code metrics for function \p F.
-  CodeMetrics &analyzeFunction(Function *F) {
-    auto I = FunctionMetrics.insert({F, CodeMetrics()});
-    CodeMetrics &Metrics = I.first->second;
-    if (I.second) {
-      // The code metrics were not cached.
-      SmallPtrSet<const Value *, 32> EphValues;
-      CodeMetrics::collectEphemeralValues(F, &(GetAC)(*F), EphValues);
-      for (BasicBlock &BB : *F)
-        Metrics.analyzeBasicBlock(&BB, (GetTTI)(*F), EphValues);
+CodeMetrics &FunctionSpecializer::analyzeFunction(Function *F) {
 
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Code size of function "
-                        << F->getName() << " is " << Metrics.NumInsts
-                        << " instructions\n");
-    }
-    return Metrics;
-  }
 
-  /// Clone the function \p F and remove the ssa_copy intrinsics added by
-  /// the SCCPSolver in the cloned version.
-  Function *cloneCandidateFunction(Function *F, ValueToValueMapTy &Mappings) {
-    Function *Clone = CloneFunction(F, Mappings);
-    removeSSACopy(*Clone);
-    return Clone;
-  }
+static Function *cloneCandidateFunction(Function *F,
+                                        ValueToValueMapTy &Mappings) {
 
-  /// This function decides whether it's worthwhile to specialize function
-  /// \p F based on the known constant values its arguments can take on. It
-  /// only discovers potential specialization opportunities without actually
-  /// applying them.
-  ///
-  /// \returns true if any specializations have been found.
-  bool calculateGains(Function *F, InstructionCost Cost,
+bool FunctionSpecializer::calculateGains(Function *F, InstructionCost Cost,
                               SmallVectorImpl<CallSpecBinding> &WorkList) {
-    SpecializationMap Specializations;
-    // Determine if we should specialize the function based on the values the
-    // argument can take on. If specialization is not profitable, we continue
-    // on to the next argument.
-    for (Argument &FormalArg : F->args()) {
-      // Determine if this argument is interesting. If we know the argument can
-      // take on any constant values, they are collected in Constants.
-      SmallVector<CallArgBinding, 8> ActualArgs;
-      if (!isArgumentInteresting(&FormalArg, ActualArgs)) {
-        LLVM_DEBUG(dbgs() << "FnSpecialization: Argument "
-                          << FormalArg.getNameOrAsOperand()
-                          << " is not interesting\n");
-        continue;
-      }
 
-      for (const auto &Entry : ActualArgs) {
-        CallBase *Call = Entry.first;
-        Constant *ActualArg = Entry.second;
 
-        auto I = Specializations.insert({Call, SpecializationInfo()});
-        SpecializationInfo &S = I.first->second;
 
-        if (I.second)
-          S.Gain = ForceFunctionSpecialization ? 1 : 0 - Cost;
-        if (!ForceFunctionSpecialization)
-          S.Gain += getSpecializationBonus(&FormalArg, ActualArg);
-        S.Args.push_back({&FormalArg, ActualArg});
-      }
-    }
 
-    // Remove unprofitable specializations.
-    Specializations.remove_if(
-        [](const auto &Entry) { return Entry.second.Gain <= 0; });
 
-    // Clear the MapVector and return the underlying vector.
-    WorkList = Specializations.takeVector();
 
-    // Sort the candidates in descending order.
-    llvm::stable_sort(WorkList, [](const auto &L, const auto &R) {
-      return L.second.Gain > R.second.Gain;
-    });
 
-    // Truncate the worklist to 'MaxClonesThreshold' candidates if necessary.
-    if (WorkList.size() > MaxClonesThreshold) {
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Number of candidates exceed "
-                        << "the maximum number of clones threshold.\n"
-                        << "FnSpecialization: Truncating worklist to "
-                        << MaxClonesThreshold << " candidates.\n");
-      WorkList.erase(WorkList.begin() + MaxClonesThreshold, WorkList.end());
-    }
 
-    LLVM_DEBUG(dbgs() << "FnSpecialization: Specializations for function "
-                      << F->getName() << "\n";
-               for (const auto &Entry
-                    : WorkList) {
-                 dbgs() << "FnSpecialization:   Gain = " << Entry.second.Gain
-                        << "\n";
-                 for (const ArgInfo &Arg : Entry.second.Args)
-                   dbgs() << "FnSpecialization:   FormalArg = "
-                          << Arg.Formal->getNameOrAsOperand()
-                          << ", ActualArg = "
-                          << Arg.Actual->getNameOrAsOperand() << "\n";
-               });
 
-    return !WorkList.empty();
-  }
 
-  bool isCandidateFunction(Function *F) {
-    // Do not specialize the cloned function again.
-    if (SpecializedFuncs.contains(F))
-      return false;
+bool FunctionSpecializer::isCandidateFunction(Function *F) {
 
-    // If we're optimizing the function for size, we shouldn't specialize it.
-    if (F->hasOptSize() ||
-        shouldOptimizeForSize(F, nullptr, nullptr, PGSOQueryType::IRPass))
-      return false;
 
-    // Exit if the function is not executable. There's no point in specializing
-    // a dead function.
-    if (!Solver.isBlockExecutable(&F->getEntryBlock()))
-      return false;
 
-    // It wastes time to specialize a function which would get inlined finally.
-    if (F->hasFnAttribute(Attribute::AlwaysInline))
-      return false;
 
-    LLVM_DEBUG(dbgs() << "FnSpecialization: Try function: " << F->getName()
-                      << "\n");
-    return true;
-  }
 
-  void specializeFunction(Function *F, SpecializationInfo &S,
+void FunctionSpecializer::specializeFunction(Function *F, SpecializationInfo &S,
                                              FuncList &WorkList) {
-    ValueToValueMapTy Mappings;
-    Function *Clone = cloneCandidateFunction(F, Mappings);
 
-    // Rewrite calls to the function so that they call the clone instead.
-    rewriteCallSites(Clone, S.Args, Mappings);
 
-    // Initialize the lattice state of the arguments of the function clone,
-    // marking the argument on which we specialized the function constant
-    // with the given value.
-    Solver.markArgInFuncSpecialization(Clone, S.Args);
 
-    // Mark all the specialized functions
-    WorkList.push_back(Clone);
-    NbFunctionsSpecialized++;
 
-    // If the function has been completely specialized, the original function
-    // is no longer needed. Mark it unreachable.
-    if (F->getNumUses() == 0 || all_of(F->users(), [F](User *U) {
-          if (auto *CS = dyn_cast<CallBase>(U))
-            return CS->getFunction() == F;
-          return false;
-        })) {
-      Solver.markFunctionUnreachable(F);
-      FullySpecialized.insert(F);
-    }
-  }
 
-  /// Compute and return the cost of specializing function \p F.
-  InstructionCost getSpecializationCost(Function *F) {
-    CodeMetrics &Metrics = analyzeFunction(F);
-    // If the code metrics reveal that we shouldn't duplicate the function, we
-    // shouldn't specialize it. Set the specialization cost to Invalid.
-    // Or if the lines of codes implies that this function is easy to get
-    // inlined so that we shouldn't specialize it.
-    if (Metrics.notDuplicatable ||
-        (!ForceFunctionSpecialization &&
-         Metrics.NumInsts < SmallFunctionThreshold)) {
-      InstructionCost C{};
-      C.setInvalid();
-      return C;
-    }
+InstructionCost FunctionSpecializer::getSpecializationCost(Function *F) {
 
-    // Otherwise, set the specialization cost to be the cost of all the
-    // instructions in the function and penalty for specializing more functions.
-    unsigned Penalty = NbFunctionsSpecialized + 1;
-    return Metrics.NumInsts * InlineConstants::InstrCost * Penalty;
-  }
 
-  InstructionCost getUserBonus(User *U, llvm::TargetTransformInfo &TTI,
+static InstructionCost getUserBonus(User *U, llvm::TargetTransformInfo &TTI,
                                     LoopInfo &LI) {
-    auto *I = dyn_cast_or_null<Instruction>(U);
-    // If not an instruction we do not know how to evaluate.
-    // Keep minimum possible cost for now so that it doesnt affect
-    // specialization.
-    if (!I)
-      return std::numeric_limits<unsigned>::min();
 
-    auto Cost = TTI.getUserCost(U, TargetTransformInfo::TCK_SizeAndLatency);
 
-    // Traverse recursively if there are more uses.
-    // TODO: Any other instructions to be added here?
-    if (I->mayReadFromMemory() || I->isCast())
-      for (auto *User : I->users())
-        Cost += getUserBonus(User, TTI, LI);
 
-    // Increase the cost if it is inside the loop.
-    auto LoopDepth = LI.getLoopDepth(I->getParent());
-    Cost *= std::pow((double)AvgLoopIterationCount, LoopDepth);
-    return Cost;
-  }
 
-  /// Compute a bonus for replacing argument \p A with constant \p C.
-  InstructionCost getSpecializationBonus(Argument *A, Constant *C) {
-    Function *F = A->getParent();
-    DominatorTree DT(*F);
-    LoopInfo LI(DT);
-    auto &TTI = (GetTTI)(*F);
-    LLVM_DEBUG(dbgs() << "FnSpecialization: Analysing bonus for constant: "
-                      << C->getNameOrAsOperand() << "\n");
+InstructionCost FunctionSpecializer::getSpecializationBonus(Argument *A,
+                                                            Constant *C) {
 
-    InstructionCost TotalCost = 0;
-    for (auto *U : A->users()) {
-      TotalCost += getUserBonus(U, TTI, LI);
-      LLVM_DEBUG(dbgs() << "FnSpecialization:   User cost ";
-                 TotalCost.print(dbgs()); dbgs() << " for: " << *U << "\n");
-    }
 
-    // The below heuristic is only concerned with exposing inlining
-    // opportunities via indirect call promotion. If the argument is not a
-    // (potentially casted) function pointer, give up.
-    Function *CalledFunction = dyn_cast<Function>(C->stripPointerCasts());
-    if (!CalledFunction)
-      return TotalCost;
 
-    // Get TTI for the called function (used for the inline cost).
-    auto &CalleeTTI = (GetTTI)(*CalledFunction);
 
-    // Look at all the call sites whose called value is the argument.
-    // Specializing the function on the argument would allow these indirect
-    // calls to be promoted to direct calls. If the indirect call promotion
-    // would likely enable the called function to be inlined, specializing is a
-    // good idea.
-    int Bonus = 0;
-    for (User *U : A->users()) {
-      if (!isa<CallInst>(U) && !isa<InvokeInst>(U))
-        continue;
-      auto *CS = cast<CallBase>(U);
-      if (CS->getCalledOperand() != A)
-        continue;
 
-      // Get the cost of inlining the called function at this call site. Note
-      // that this is only an estimate. The called function may eventually
-      // change in a way that leads to it not being inlined here, even though
-      // inlining looks profitable now. For example, one of its called
-      // functions may be inlined into it, making the called function too large
-      // to be inlined into this call site.
-      //
-      // We apply a boost for performing indirect call promotion by increasing
-      // the default threshold by the threshold for indirect calls.
-      auto Params = getInlineParams();
-      Params.DefaultThreshold += InlineConstants::IndirectCallThreshold;
-      InlineCost IC =
-          getInlineCost(*CS, CalledFunction, Params, CalleeTTI, GetAC, GetTLI);
 
-      // We clamp the bonus for this call to be between zero and the default
-      // threshold.
-      if (IC.isAlways())
-        Bonus += Params.DefaultThreshold;
-      else if (IC.isVariable() && IC.getCostDelta() > 0)
-        Bonus += IC.getCostDelta();
 
-      LLVM_DEBUG(dbgs() << "FnSpecialization:   Inlining bonus " << Bonus
-                        << " for user " << *U << "\n");
-    }
 
-    return TotalCost + Bonus;
-  }
 
-  /// Determine if we should specialize a function based on the incoming values
-  /// of the given argument.
-  ///
-  /// This function implements the goal-directed heuristic. It determines if
-  /// specializing the function based on the incoming values of argument \p A
-  /// would result in any significant optimization opportunities. If
-  /// optimization opportunities exist, the constant values of \p A on which to
-  /// specialize the function are collected in \p Constants.
-  ///
-  /// \returns true if the function should be specialized on the given
-  /// argument.
-  bool isArgumentInteresting(Argument *A,
+bool FunctionSpecializer::isArgumentInteresting(Argument *A,
                               SmallVectorImpl<CallArgBinding> &Constants) {
-    // For now, don't attempt to specialize functions based on the values of
-    // composite types.
-    if (!A->getType()->isSingleValueType() || A->user_empty())
-      return false;
 
-    // If the argument isn't overdefined, there's nothing to do. It should
-    // already be constant.
-    if (!Solver.getLatticeValueFor(A).isOverdefined()) {
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Nothing to do, argument "
-                        << A->getNameOrAsOperand()
-                        << " is already constant?\n");
-      return false;
-    }
 
-    // Collect the constant values that the argument can take on. If the
-    // argument can't take on any constant values, we aren't going to
-    // specialize the function. While it's possible to specialize the function
-    // based on non-constant arguments, there's likely not much benefit to
-    // constant propagation in doing so.
-    //
-    // TODO 1: currently it won't specialize if there are over the threshold of
-    // calls using the same argument, e.g foo(a) x 4 and foo(b) x 1, but it
-    // might be beneficial to take the occurrences into account in the cost
-    // model, so we would need to find the unique constants.
-    //
-    // TODO 2: this currently does not support constants, i.e. integer ranges.
-    //
-    getPossibleConstants(A, Constants);
 
-    if (Constants.empty())
-      return false;
 
-    LLVM_DEBUG(dbgs() << "FnSpecialization: Found interesting argument "
-                      << A->getNameOrAsOperand() << "\n");
-    return true;
-  }
 
-  /// Collect in \p Constants all the constant values that argument \p A can
-  /// take on.
-  void getPossibleConstants(Argument *A,
+void FunctionSpecializer::getPossibleConstants(Argument *A,
                               SmallVectorImpl<CallArgBinding> &Constants) {
-    Function *F = A->getParent();
 
-    // Iterate over all the call sites of the argument's parent function.
-    for (User *U : F->users()) {
-      if (!isa<CallInst>(U) && !isa<InvokeInst>(U))
-        continue;
-      auto &CS = *cast<CallBase>(U);
-      // If the call site has attribute minsize set, that callsite won't be
-      // specialized.
-      if (CS.hasFnAttr(Attribute::MinSize))
-        continue;
 
-      // If the parent of the call site will never be executed, we don't need
-      // to worry about the passed value.
-      if (!Solver.isBlockExecutable(CS.getParent()))
-        continue;
 
-      auto *V = CS.getArgOperand(A->getArgNo());
-      if (isa<PoisonValue>(V))
-        return;
 
-      // For now, constant expressions are fine but only if they are function
-      // calls.
-      if (auto *CE = dyn_cast<ConstantExpr>(V))
-        if (!isa<Function>(CE->getOperand(0)))
-          return;
 
-      // TrackValueOfGlobalVariable only tracks scalar global variables.
-      if (auto *GV = dyn_cast<GlobalVariable>(V)) {
-        // Check if we want to specialize on the address of non-constant
-        // global values.
-        if (!GV->isConstant())
-          if (!SpecializeOnAddresses)
-            return;
 
-        if (!GV->getValueType()->isSingleValueType())
-          return;
-      }
 
-      if (isa<Constant>(V) && (Solver.getLatticeValueFor(V).isConstant() ||
-                               EnableSpecializationForLiteralConstant))
-        Constants.push_back({&CS, cast<Constant>(V)});
-    }
-  }
 
-  /// Rewrite calls to function \p F to call function \p Clone instead.
-  ///
-  /// This function modifies calls to function \p F as long as the actual
-  /// arguments match those in \p Args. Note that for recursive calls we
-  /// need to compare against the cloned formal arguments.
-  ///
-  /// Callsites that have been marked with the MinSize function attribute won't
-  /// be specialized and rewritten.
-  void rewriteCallSites(Function *Clone, const SmallVectorImpl<ArgInfo> &Args,
+void FunctionSpecializer::rewriteCallSites(Function *Clone,
+                                           const SmallVectorImpl<ArgInfo> &Args,
                                            ValueToValueMapTy &Mappings) {
-    assert(!Args.empty() && "Specialization without arguments");
-    Function *F = Args[0].Formal->getParent();
 
-    SmallVector<CallBase *, 8> CallSitesToRewrite;
-    for (auto *U : F->users()) {
-      if (!isa<CallInst>(U) && !isa<InvokeInst>(U))
-        continue;
-      auto &CS = *cast<CallBase>(U);
-      if (!CS.getCalledFunction() || CS.getCalledFunction() != F)
-        continue;
-      CallSitesToRewrite.push_back(&CS);
-    }
 
-    LLVM_DEBUG(dbgs() << "FnSpecialization: Replacing call sites of "
-                      << F->getName() << " with " << Clone->getName() << "\n");
 
-    for (auto *CS : CallSitesToRewrite) {
-      LLVM_DEBUG(dbgs() << "FnSpecialization:   "
-                        << CS->getFunction()->getName() << " ->" << *CS
-                        << "\n");
-      if (/* recursive call */
-          (CS->getFunction() == Clone &&
-           all_of(Args,
-                  [CS, &Mappings](const ArgInfo &Arg) {
-                    unsigned ArgNo = Arg.Formal->getArgNo();
-                    return CS->getArgOperand(ArgNo) == Mappings[Arg.Formal];
-                  })) ||
-          /* normal call */
-          all_of(Args, [CS](const ArgInfo &Arg) {
-            unsigned ArgNo = Arg.Formal->getArgNo();
-            return CS->getArgOperand(ArgNo) == Arg.Actual;
-          })) {
-        CS->setCalledFunction(Clone);
-        Solver.markOverdefined(CS);
-      }
-    }
-  }
 
-  void updateSpecializedFuncs(FuncList &Candidates, FuncList &WorkList) {
-    for (auto *F : WorkList) {
-      SpecializedFuncs.insert(F);
+void FunctionSpecializer::updateSpecializedFuncs(FuncList &Candidates,
+                                                 FuncList &WorkList) {
 
-      // Initialize the state of the newly created functions, marking them
-      // argument-tracked and executable.
-      if (F->hasExactDefinition() && !F->hasFnAttribute(Attribute::Naked))
-        Solver.addTrackedFunction(F);
 
-      Solver.addArgumentTrackedFunction(F);
-      Candidates.push_back(F);
-      Solver.markBlockExecutable(&F->front());
 
-      // Replace the function arguments for the specialized functions.
-      for (Argument &Arg : F->args())
-        if (!Arg.use_empty() && tryToReplaceWithConstant(&Arg))
-          LLVM_DEBUG(dbgs() << "FnSpecialization: Replaced constant argument: "
-                            << Arg.getNameOrAsOperand() << "\n");
-    }
-  }
-};
-} // namespace
+      if (!Arg.use_empty() && tryToReplaceWithConstant(Solver, &Arg))
 
-bool llvm::runFunctionSpecialization(
-    Module &M, const DataLayout &DL,
-    std::function<TargetLibraryInfo &(Function &)> GetTLI,
-    std::function<TargetTransformInfo &(Function &)> GetTTI,
-    std::function<AssumptionCache &(Function &)> GetAC,
-    function_ref<AnalysisResultsForFn(Function &)> GetAnalysis) {
-  SCCPSolver Solver(DL, GetTLI, M.getContext());
-  FunctionSpecializer FS(Solver, GetAC, GetTTI, GetTLI);
+bool FunctionSpecializer::specialize(FuncList &FuncDecls) {
   bool Changed = false;
 
-  // Loop over all functions, marking arguments to those with their addresses
-  // taken or that are external as overdefined.
-  for (Function &F : M) {
-    if (F.isDeclaration())
-      continue;
-    if (F.hasFnAttribute(Attribute::NoDuplicate))
-      continue;
-
-    LLVM_DEBUG(dbgs() << "\nFnSpecialization: Analysing decl: " << F.getName()
-                      << "\n");
-    Solver.addAnalysis(F, GetAnalysis(F));
-
-    // Determine if we can track the function's arguments. If so, add the
-    // function to the solver's set of argument-tracked functions.
-    if (canTrackArgumentsInterprocedurally(&F)) {
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Can track arguments\n");
-      Solver.addArgumentTrackedFunction(&F);
-      continue;
-    } else {
-      LLVM_DEBUG(dbgs() << "FnSpecialization: Can't track arguments!\n"
-                        << "FnSpecialization: Doesn't have local linkage, or "
-                        << "has its address taken\n");
-    }
-
-    // Assume the function is called.
-    Solver.markBlockExecutable(&F.front());
-
-    // Assume nothing about the incoming arguments.
-    for (Argument &AI : F.args())
-      Solver.markOverdefined(&AI);
-  }
-
-  // Determine if we can track any of the module's global variables. If so, add
-  // the global variables we can track to the solver's set of tracked global
-  // variables.
-  for (GlobalVariable &G : M.globals()) {
-    G.removeDeadConstantUsers();
-    if (canTrackGlobalVariableInterprocedurally(&G))
-      Solver.trackValueOfGlobalVariable(&G);
-  }
-
-  auto &TrackedFuncs = Solver.getArgumentTrackedFunctions();
-  SmallVector<Function *, 16> FuncDecls(TrackedFuncs.begin(),
-                                        TrackedFuncs.end());
-
-  // No tracked functions, so nothing to do: don't run the solver and remove
-  // the ssa_copy intrinsics that may have been introduced.
-  if (TrackedFuncs.empty()) {
-    removeSSACopy(M);
-    return false;
-  }
-
   // Solve for constants.
   auto RunSCCPSolver = [&](auto &WorkList) {
     bool ResolvedUndefs = true;
 
     while (ResolvedUndefs) {
       // Not running the solver unnecessary is checked in regression test
       // nothing-to-do.ll, so if this debug message is changed, this regression
       // test needs updating too.
auto RunSCCPSolver = [&](auto &WorkList) {
 
     for (auto *F : WorkList) {
       for (BasicBlock &BB : *F) {
         if (!Solver.isBlockExecutable(&BB))
           continue;
         // FIXME: The solver may make changes to the function here, so set
         // Changed, even if later function specialization does not trigger.
         for (auto &I : make_early_inc_range(BB))
-          Changed |= FS.tryToReplaceWithConstant(&I);
+          Changed |= tryToReplaceWithConstant(Solver, &I);
       }
     }
   };
 
 #ifndef NDEBUG
   LLVM_DEBUG(dbgs() << "FnSpecialization: Worklist fn decls:\n");
   for (auto *F : FuncDecls)
     LLVM_DEBUG(dbgs() << "FnSpecialization: *) " << F->getName() << "\n");
 #endif
 
-  // Initially resolve the constants in all the argument tracked functions.
-  RunSCCPSolver(FuncDecls);
-
   SmallVector<Function *, 8> WorkList;
   unsigned I = 0;
   while (FuncSpecializationMaxIters != I++ &&
-         FS.specializeFunctions(FuncDecls, WorkList)) {
+         specializeFunctions(FuncDecls, WorkList)) {
     LLVM_DEBUG(dbgs() << "FnSpecialization: Finished iteration " << I << "\n");
 
     // Run the solver for the specialized functions.
     RunSCCPSolver(WorkList);
 
     // Replace some unresolved constant arguments.
-    constantArgPropagation(FuncDecls, M, Solver);
+    propagateConstantArgs(FuncDecls);
 
     WorkList.clear();
     Changed = true;
   }
 
   LLVM_DEBUG(dbgs() << "FnSpecialization: Number of specializations = "
                     << NumFuncSpecialized << "\n");
 
   // Remove any ssa_copy intrinsics that may have been introduced.
   removeSSACopy(M);
   return Changed;
 }

llvm/lib/Transforms/IPO/IPO.cpp

void llvm::initializeIPO(PassRegistry &Registry) {
   initializeArgPromotionPass(Registry);
   initializeAnnotation2MetadataLegacyPass(Registry);
   initializeCalledValuePropagationLegacyPassPass(Registry);
   initializeConstantMergeLegacyPassPass(Registry);
   initializeCrossDSOCFIPass(Registry);
   initializeDAEPass(Registry);
   initializeDAHPass(Registry);
   initializeForceFunctionAttrsLegacyPassPass(Registry);
-  initializeFunctionSpecializationLegacyPassPass(Registry);
   initializeGlobalDCELegacyPassPass(Registry);
   initializeGlobalOptLegacyPassPass(Registry);
   initializeGlobalSplitPass(Registry);
   initializeHotColdSplittingLegacyPassPass(Registry);
   initializeIROutlinerLegacyPassPass(Registry);
   initializeAlwaysInlinerLegacyPassPass(Registry);
   initializeSimpleInlinerPass(Registry);
   initializeInferFunctionAttrsLegacyPassPass(Registry);

llvm/lib/Transforms/IPO/PassManagerBuilder.cpp

void PassManagerBuilder::populateModulePassManager(
   if (AttributorRun & AttributorRunOption::MODULE)
     MPM.add(createAttributorLegacyPass());
 
   addExtensionsToPM(EP_ModuleOptimizerEarly, MPM);
 
   if (OptLevel > 2)
     MPM.add(createCallSiteSplittingPass());
 
-  // Propage constant function arguments by specializing the functions.
-  if (OptLevel > 2 && EnableFunctionSpecialization)
-    MPM.add(createFunctionSpecializationPass());
-
   MPM.add(createIPSCCPPass());          // IP SCCP
   MPM.add(createCalledValuePropagationPass());
 
   MPM.add(createGlobalOptimizerPass()); // Optimize out global vars
   // Promote any localized global vars.
   MPM.add(createPromoteMemoryToRegisterPass());
 
   MPM.add(createDeadArgEliminationPass()); // Dead argument elimination
void PassManagerBuilder::addLTOOptimizationPasses(legacy::PassManagerBase &PM) {
 
   // Infer attributes about declarations if possible.
   PM.add(createInferFunctionAttrsLegacyPass());
 
   if (OptLevel > 1) {
     // Split call-site with more constrained arguments.
     PM.add(createCallSiteSplittingPass());
 
-    // Propage constant function arguments by specializing the functions.
-    if (EnableFunctionSpecialization && OptLevel > 2)
-      PM.add(createFunctionSpecializationPass());
-
     // Propagate constants at call sites into the functions they call.  This
     // opens opportunities for globalopt (and inlining) by substituting function
     // pointers passed as arguments to direct uses of functions.
     PM.add(createIPSCCPPass());
 
     // Attach metadata to indirect call sites indicating the set of functions
     // they may target at run-time. This should follow IPSCCP.
     PM.add(createCalledValuePropagationPass());

llvm/lib/Transforms/IPO/SCCP.cpp

 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/Scalar/SCCP.h"
 #include "llvm/Transforms/Utils/SCCPSolver.h"
 
 using namespace llvm;
 
 PreservedAnalyses IPSCCPPass::run(Module &M, ModuleAnalysisManager &AM) {
   const DataLayout &DL = M.getDataLayout();
   auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
   auto GetTLI = [&FAM](Function &F) -> const TargetLibraryInfo & {
     return FAM.getResult<TargetLibraryAnalysis>(F);
   };
+  auto GetTTI = [&FAM](Function &F) -> TargetTransformInfo & {
+    return FAM.getResult<TargetIRAnalysis>(F);
+  };
+  auto GetAC = [&FAM](Function &F) -> AssumptionCache & {
+    return FAM.getResult<AssumptionAnalysis>(F);
+  };
   auto getAnalysis = [&FAM](Function &F) -> AnalysisResultsForFn {
     DominatorTree &DT = FAM.getResult<DominatorTreeAnalysis>(F);
     return {
         std::make_unique<PredicateInfo>(F, DT, FAM.getResult<AssumptionAnalysis>(F)),
         &DT, FAM.getCachedResult<PostDominatorTreeAnalysis>(F)};
   };
 
-  if (!runIPSCCP(M, DL, GetTLI, getAnalysis))
+  if (!runIPSCCP(M, DL, GetTLI, GetTTI, GetAC, getAnalysis))
     return PreservedAnalyses::all();
 
   PreservedAnalyses PA;
   PA.preserve<DominatorTreeAnalysis>();
   PA.preserve<PostDominatorTreeAnalysis>();
   PA.preserve<FunctionAnalysisManagerModuleProxy>();
   return PA;
 }
 
 namespace {
 
 //===--------------------------------------------------------------------===//
 //
 /// IPSCCP Class - This class implements interprocedural Sparse Conditional
 /// Constant Propagation.
 ///
 class IPSCCPLegacyPass : public ModulePass {
 public:
   static char ID;
 
   IPSCCPLegacyPass() : ModulePass(ID) {
     initializeIPSCCPLegacyPassPass(*PassRegistry::getPassRegistry());
   }
 
   bool runOnModule(Module &M) override {
     if (skipModule(M))
       return false;
+
     const DataLayout &DL = M.getDataLayout();
     auto GetTLI = [this](Function &F) -> const TargetLibraryInfo & {
       return this->getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
     };
+    auto GetTTI = [this](Function &F) -> TargetTransformInfo & {
+      return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
+    };
+    auto GetAC = [this](Function &F) -> AssumptionCache & {
+      return this->getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
+    };
     auto getAnalysis = [this](Function &F) -> AnalysisResultsForFn {
       DominatorTree &DT =
           this->getAnalysis<DominatorTreeWrapperPass>(F).getDomTree();
       return {
           std::make_unique<PredicateInfo>(
               F, DT,
               this->getAnalysis<AssumptionCacheTracker>().getAssumptionCache(
                   F)),
           nullptr,  // We cannot preserve the DT or PDT with the legacy pass
           nullptr}; // manager, so set them to nullptr.
     };
 
-    return runIPSCCP(M, DL, GetTLI, getAnalysis);
+    return runIPSCCP(M, DL, GetTLI, GetTTI, GetAC, getAnalysis);
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.addRequired<AssumptionCacheTracker>();
     AU.addRequired<DominatorTreeWrapperPass>();
     AU.addRequired<TargetLibraryInfoWrapperPass>();
+    AU.addRequired<TargetTransformInfoWrapperPass>();
   }
 };
 
 } // end anonymous namespace
 
 char IPSCCPLegacyPass::ID = 0;
 
 INITIALIZE_PASS_BEGIN(IPSCCPLegacyPass, "ipsccp",
                       "Interprocedural Sparse Conditional Constant Propagation",
                       false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
 INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
 INITIALIZE_PASS_END(IPSCCPLegacyPass, "ipsccp",
                     "Interprocedural Sparse Conditional Constant Propagation",
                     false, false)
 
 // createIPSCCPPass - This is the public interface to this file.
 ModulePass *llvm::createIPSCCPPass() { return new IPSCCPLegacyPass(); }
 
-PreservedAnalyses FunctionSpecializationPass::run(Module &M,
-                                                  ModuleAnalysisManager &AM) {
-  const DataLayout &DL = M.getDataLayout();
-  auto &FAM = AM.getResult<FunctionAnalysisManagerModuleProxy>(M).getManager();
-  auto GetTLI = [&FAM](Function &F) -> TargetLibraryInfo & {
-    return FAM.getResult<TargetLibraryAnalysis>(F);
-  };
-  auto GetTTI = [&FAM](Function &F) -> TargetTransformInfo & {
-    return FAM.getResult<TargetIRAnalysis>(F);
-  };
-  auto GetAC = [&FAM](Function &F) -> AssumptionCache & {
-    return FAM.getResult<AssumptionAnalysis>(F);
-  };
-  auto GetAnalysis = [&FAM](Function &F) -> AnalysisResultsForFn {
-    DominatorTree &DT = FAM.getResult<DominatorTreeAnalysis>(F);
-    return {std::make_unique<PredicateInfo>(
-                F, DT, FAM.getResult<AssumptionAnalysis>(F)),
-            &DT, FAM.getCachedResult<PostDominatorTreeAnalysis>(F)};
-  };
-
-  if (!runFunctionSpecialization(M, DL, GetTLI, GetTTI, GetAC, GetAnalysis))
-    return PreservedAnalyses::all();
-
-  PreservedAnalyses PA;
-  PA.preserve<DominatorTreeAnalysis>();
-  PA.preserve<PostDominatorTreeAnalysis>();
-  PA.preserve<FunctionAnalysisManagerModuleProxy>();
-  return PA;
-}
-
-namespace {
-struct FunctionSpecializationLegacyPass : public ModulePass {
-  static char ID; // Pass identification, replacement for typeid
-  FunctionSpecializationLegacyPass() : ModulePass(ID) {}
-
-  void getAnalysisUsage(AnalysisUsage &AU) const override {
-    AU.addRequired<AssumptionCacheTracker>();
-    AU.addRequired<DominatorTreeWrapperPass>();
-    AU.addRequired<TargetLibraryInfoWrapperPass>();
-    AU.addRequired<TargetTransformInfoWrapperPass>();
-  }
-
-  virtual bool runOnModule(Module &M) override {
-    if (skipModule(M))
-      return false;
-
-    const DataLayout &DL = M.getDataLayout();
-    auto GetTLI = [this](Function &F) -> TargetLibraryInfo & {
-      return this->getAnalysis<TargetLibraryInfoWrapperPass>().getTLI(F);
-    };
-    auto GetTTI = [this](Function &F) -> TargetTransformInfo & {
-      return this->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
-    };
-    auto GetAC = [this](Function &F) -> AssumptionCache & {
-      return this->getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
-    };
-
-    auto GetAnalysis = [this](Function &F) -> AnalysisResultsForFn {
-      DominatorTree &DT =
-          this->getAnalysis<DominatorTreeWrapperPass>(F).getDomTree();
-      return {
-          std::make_unique<PredicateInfo>(
-              F, DT,
-              this->getAnalysis<AssumptionCacheTracker>().getAssumptionCache(
-                  F)),
-          nullptr,  // We cannot preserve the DT or PDT with the legacy pass
-          nullptr}; // manager, so set them to nullptr.
-    };
-    return runFunctionSpecialization(M, DL, GetTLI, GetTTI, GetAC, GetAnalysis);
-  }
-};
-} // namespace
-
-char FunctionSpecializationLegacyPass::ID = 0;
-
-INITIALIZE_PASS_BEGIN(
-    FunctionSpecializationLegacyPass, "function-specialization",
-    "Propagate constant arguments by specializing the function", false, false)
-
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_END(FunctionSpecializationLegacyPass, "function-specialization",
-                    "Propagate constant arguments by specializing the function",
-                    false, false)
-
-ModulePass *llvm::createFunctionSpecializationPass() {
-  return new FunctionSpecializationLegacyPass();
-}

llvm/lib/Transforms/Scalar/CMakeLists.txt

add_llvm_component_library(LLVMScalarOpts
   COMPONENT_NAME
   Scalar
 
   LINK_COMPONENTS
   AggressiveInstCombine
   Analysis
   Core
   InstCombine
+  IPO
   Support
   TransformUtils
   )

llvm/lib/Transforms/Scalar/SCCP.cpp

 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Transforms/IPO/FunctionSpecialization.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Utils/Local.h"
 #include "llvm/Transforms/Utils/SCCPSolver.h"
 #include <cassert>
 #include <utility>
 #include <vector>
 
 using namespace llvm;
 
 #define DEBUG_TYPE "sccp"
 
 STATISTIC(NumInstRemoved, "Number of instructions removed");
 STATISTIC(NumDeadBlocks , "Number of basic blocks unreachable");
 STATISTIC(NumInstReplaced,
           "Number of instructions replaced with (simpler) instruction");
 
 STATISTIC(IPNumInstRemoved, "Number of instructions removed by IPSCCP");
 STATISTIC(IPNumArgsElimed ,"Number of arguments constant propagated by IPSCCP");
 STATISTIC(IPNumGlobalConst, "Number of globals found to be constant by IPSCCP");
 STATISTIC(
     IPNumInstReplaced,
     "Number of instructions replaced with (simpler) instruction by IPSCCP");
 
+static cl::opt<bool> SpecializeFunctions("specialize-functions", cl::init(false),
+    cl::Hidden, cl::desc("Enable function specialization"));
+
 // Helper to check if \p LV is either a constant or a constant
 // range with a single element. This should cover exactly the same cases as the
 // old ValueLatticeElement::isConstant() and is intended to be used in the
 // transition to ValueLatticeElement.
 static bool isConstant(const ValueLatticeElement &LV) {
   return LV.isConstant() ||
          (LV.isConstantRange() && LV.getConstantRange().isSingleElement());
 }
 
 // Helper to check if \p LV is either overdefined or a constant range with more
 // than a single element. This should cover exactly the same cases as the old
 // ValueLatticeElement::isOverdefined() and is intended to be used in the
 // transition to ValueLatticeElement.
 static bool isOverdefined(const ValueLatticeElement &LV) {
   return !LV.isUnknownOrUndef() && !isConstant(LV);
 }
 
 static bool tryToReplaceWithConstant(SCCPSolver &Solver, Value *V) {
   Constant *Const = nullptr;
   if (V->getType()->isStructTy()) {
     std::vector<ValueLatticeElement> IVs = Solver.getStructLatticeValueFor(V);
     if (llvm::any_of(IVs, isOverdefined))
       return false;
     std::vector<Constant *> ConstVals;
     auto *ST = cast<StructType>(V->getType());
     for (unsigned i = 0, e = ST->getNumElements(); i != e; ++i) {
       ValueLatticeElement V = IVs[i];
if (CB && ((CB->isMustTailCall() && !CB->isSafeToRemove()) ||
 
     LLVM_DEBUG(dbgs() << "  Can\'t treat the result of call " << *CB
                       << " as a constant\n");
     return false;
   }
 
   LLVM_DEBUG(dbgs() << "  Constant: " << *Const << " = " << *V << '\n');
 
   // Replaces all of the uses of a variable with uses of the constant.
   V->replaceAllUsesWith(Const);
   return true;
 }
 
 static bool simplifyInstsInBlock(SCCPSolver &Solver, BasicBlock &BB,
                                  SmallPtrSetImpl<Value *> &InsertedValues,
                                  Statistic &InstRemovedStat,
                                  Statistic &InstReplacedStat) {
   bool MadeChanges = false;
   for (Instruction &Inst : make_early_inc_range(BB)) {
     if (Inst.getType()->isVoidTy())
       continue;
     if (tryToReplaceWithConstant(Solver, &Inst)) {
       if (Inst.isSafeToRemove())
         Inst.eraseFromParent();
 
       MadeChanges = true;
       ++InstRemovedStat;
     } else if (isa<SExtInst>(&Inst)) {
       Value *ExtOp = Inst.getOperand(0);
       if (isa<Constant>(ExtOp) || InsertedValues.count(ExtOp))
         continue;
       const ValueLatticeElement &IV = Solver.getLatticeValueFor(ExtOp);
       if (!IV.isConstantRange(/*UndefAllowed=*/false))
         continue;
if (FeasibleSuccessors.size() == 1) {
     llvm_unreachable("Must have at least one feasible successor");
   }
   return true;
 }
 
 bool llvm::runIPSCCP(
     Module &M, const DataLayout &DL,
     std::function<const TargetLibraryInfo &(Function &)> GetTLI,
+    std::function<TargetTransformInfo &(Function &)> GetTTI,
+    std::function<AssumptionCache &(Function &)> GetAC,
     function_ref<AnalysisResultsForFn(Function &)> getAnalysis) {
   SCCPSolver Solver(DL, GetTLI, M.getContext());
+  FunctionSpecializer Specializer(Solver, M, GetAC, GetTTI, GetTLI);
 
   // Loop over all functions, marking arguments to those with their addresses
   // taken or that are external as overdefined.
   for (Function &F : M) {
     if (F.isDeclaration())
       continue;
 
     Solver.addAnalysis(F, getAnalysis(F));
while (ResolvedUndefs) {
     ResolvedUndefs = false;
     for (Function &F : M) {
       if (Solver.resolvedUndefsIn(F))
         ResolvedUndefs = true;
     }
     if (ResolvedUndefs)
       Solver.solve();
   }
 
   bool MadeChanges = false;
 
   // Iterate over all of the instructions in the module, replacing them with
   // constants if we have found them to be of constant values.
 
   for (Function &F : M) {
     if (F.isDeclaration())
       continue;
 
     SmallVector<BasicBlock *, 512> BlocksToErase;
 
     if (Solver.isBlockExecutable(&F.front())) {
       bool ReplacedPointerArg = false;
for (Function &F : M) {
     BasicBlock *NewUnreachableBB = nullptr;
     for (BasicBlock &BB : F)
       MadeChanges |= removeNonFeasibleEdges(Solver, &BB, DTU, NewUnreachableBB);
 
     for (BasicBlock *DeadBB : BlocksToErase)
       if (!DeadBB->hasAddressTaken())
         DTU.deleteBB(DeadBB);
 
+    if (!SpecializeFunctions) {
+      // The Function Specializer will delete those after completion.
       for (BasicBlock &BB : F) {
         for (Instruction &Inst : llvm::make_early_inc_range(BB)) {
           if (Solver.getPredicateInfoFor(&Inst)) {
             if (auto *II = dyn_cast<IntrinsicInst>(&Inst)) {
               if (II->getIntrinsicID() == Intrinsic::ssa_copy) {
                 Value *Op = II->getOperand(0);
                 Inst.replaceAllUsesWith(Op);
                 Inst.eraseFromParent();
               }
             }
           }
         }
       }
     }
+  }
 
   // If we inferred constant or undef return values for a function, we replaced
   // all call uses with the inferred value.  This means we don't need to bother
   // actually returning anything from the function.  Replace all return
   // instructions with return undef.
   //
   // Do this in two stages: first identify the functions we should process, then
   // actually zap their returns.  This is important because we can only do this
while (!GV->use_empty()) {
       StoreInst *SI = cast<StoreInst>(GV->user_back());
       SI->eraseFromParent();
       MadeChanges = true;
     }
     M.getGlobalList().erase(GV);
     ++IPNumGlobalConst;
   }
 
+  if (SpecializeFunctions) {
+    SmallVector<Function *> Candidates;
+    for (Function *F : Solver.getArgumentTrackedFunctions())
+      if (!F->hasFnAttribute(Attribute::NoDuplicate))
+        Candidates.push_back(F);
+
+    MadeChanges |= Specializer.specialize(Candidates);
+  }
+
   return MadeChanges;
 }

llvm/utils/gn/secondary/llvm/lib/Transforms/Scalar/BUILD.gn

 static_library("Scalar") {
   output_name = "LLVMScalarOpts"
   deps = [
     "//llvm/include/llvm/Config:llvm-config",
     "//llvm/lib/Analysis",
     "//llvm/lib/IR",
     "//llvm/lib/Support",
     "//llvm/lib/Transforms/AggressiveInstCombine",
+    "//llvm/lib/Transforms/IPO",
     "//llvm/lib/Transforms/InstCombine",
     "//llvm/lib/Transforms/Utils",
   ]
   sources = [
     "ADCE.cpp",
     "AlignmentFromAssumptions.cpp",
     "AnnotationRemarks.cpp",
     "BDCE.cpp",