[PM] Port the always inliner to the new pass manager in a much more minimal and boring form than the old pass manager's version.

[chandlerc]

Summary

This pass does the very minimal amount of work necessary to inline
functions declared as always-inline. It doesn't support a wide array of
things that the legacy pass manager did support, but is alse ... about
20 lines of code. So it has that going for it. Notably things this
doesn't support:

• Array alloca merging
  • To support the above, bottom-up inlining with careful history tracking and call graph updates
• DCE of the functions that become dead after this inlining.
• Inlining through call instructions with the always_inline attribute. Instead, it focuses on inlining functions with that attribute.

The first I've omitted because I'm hoping to just turn it off for the
primary pass manager. If that doesn't pan out, I can add it here but it
will be reasonably expensive to do so.

The second should really be handled by running global-dce after the
inliner. I don't want to re-implement the non-trivial logic necessary to
do comdat-correct DCE of functions. This means the -O0 pipeline will
have to be at least 'always-inline,global-dce', but that seems
reasonable to me. If others are seriously worried about this I'd like to
heard and understand why. Again, this is all solveable by factoring that
logic into a utility and calling it here, but I'd like to wait to do
that until there is a clear reason why the existing pass-based factoring
won't work.

The final point is a serious one. I can fairly easily add support for
this, but it seems both costly and a confusing construct for the use
case of the always inliner running at O0. This attribute can of course
still impact the normal inliner easily (although I find that
a questionable re-use of the same attribute). I've started a discussion
to sort out what semantics we want here and based on that can figure out
if it makes sense ta have this complexity at O0 or not.

One other advantage of this design is that it should be quite a bit
faster due to checking for whether the function is a viable candidate
for inlining exactly once per function instead of doing it for each call
site.

Anyways, hopefully a reasonable starting point for this pass.

[davide]

Some comments.

lib/Passes/PassRegistry.def

40 ↗

(On Diff #67299)

Is there a particular reason why you renamed always-inline to always-inliner?
I think we should try to keep the names of the passes consistent between old and new PM (unless there's a reason not to)/

lib/Transforms/IPO/AlwaysInliner.cpp

53 ↗

(On Diff #67299)

Changed |= InlineFunction(CS, IFI); , no?

94–97 ↗

(On Diff #67299)

I'm not entirely sure here, but are all these dependencies actually needed?

test/Transforms/Inline/always-inline.ll

4 ↗

(On Diff #67299)

These tests modification (at least part of them) can be committed separately, probably.

[chandlerc]

Thanks for the review. See some responses inline, but largely good catches! Updated patch momentarily.

lib/Passes/PassRegistry.def

40 ↗

(On Diff #67299)

No actually... I'm not sure what the best thing to do in this particular case is...

The class is named AlwaysInliner and that seems the right thing -- it should be a noun, etc. I've adjusted the file names to match which seems the right thing.

I had adjusted the pass name to match as well, but I agree it is a bit vague whether we should do this or not. The pass names don't actually have the noun pattern necessarily. For now, I'll take this back to the old name. We can always pursue some correspondence between pass name and class name later if that's desirable. Currently (as this diff hunk illustrates) we're no where near anyways.

lib/Transforms/IPO/AlwaysInliner.cpp

53 ↗

(On Diff #67299)

Well, the FIXME was about potentially asserting on this inlined variable. But yea, I guess the variable isn't serving any purpose until then. Nuked.

94–97 ↗

(On Diff #67299)

For the legacy one, I think so, because of the inliner base class. I'm not changing any of it in this patch at least...

test/Transforms/Inline/always-inline.ll

4 ↗

(On Diff #67299)

I don't think that really makes sense...

The only changes here are to allow one of the test cases to be omitted with the new pass manager's pass because it doesn't support that case. But none of these changes would be required without that, so it seems best to put them into this patch?

[chandlerc]

Update with fixes suggested in review!

[davide]

+ @davidxl

I have a general concern about changing the behavior of a pass during the transition to the new PM.
I don't think it's entirely unreasonable but, dependently on how many people care about this pass you may want to confirm that:

1. The behavioral changes you're proposing don't regress compile time/runtime performance
2. Checking that this design you're proposing is actually faster

[eraman]

include/llvm/Transforms/IPO/AlwaysInliner.h

28 ↗

(On Diff #67427)

Incomplete comment.

lib/Transforms/IPO/AlwaysInliner.cpp

37 ↗

(On Diff #67427)

InlineFunction makes use of IFI.GetAssumptionCache in AddAlignmentAssumptions.

[chandlerc]

Thanks!

include/llvm/Transforms/IPO/AlwaysInliner.h

28 ↗

(On Diff #67427)

Hah, not only incomplete, very stale. I totally changed my mind about how to do this after writing the comment. I've updated it to match the implementation and the patch description. Thanks!

lib/Transforms/IPO/AlwaysInliner.cpp

37 ↗

(On Diff #67427)

Good catch. I thought that AddAlignmentAssumptions was disabled in the face of a null assumption cache, but I see it isn't. =[ I've just added a check to that routine. For -O0 it seems better to not spend time adding assume intrinsics here, and this is really intended to be an -O0 style pass.

(Of course, we can add some of this back very easily if there are good reasons, I was just starting simple.)

Perhaps more importantly, I've added a test case that would have caught this bug. =]

[chandlerc]

Address review comments.

[davidxl]

test/Transforms/Inline/always-inline.ll

137 ↗

(On Diff #67450)

Some code may depend on the always-inline to be done for correctness (e.g not expecting calls at runtime for some functions). I forgot the details, but I remember cases like that.

[davidxl]

Why is this pass implemented as a module pass (instead of cgscc pass as before)? Is there any compile time concerns (at O0) ?

[chandlerc]

In D23299#512749, @davidxl wrote:

Why is this pass implemented as a module pass (instead of cgscc pass as before)? Is there any compile time concerns (at O0) ?

As I tried to explain in the patch description (and let me know if I should improve it): because a module pass is simpler, and avoids computing the call graph when we don't need it.

It might help compile time, but that wasn't my primary concern.

[davidxl]

yes, I understand the module pass is quite simple and straightforward, but it seems to me the legacy always inliner is even simpler -- as it is simply one 'instantiation' of a common inliner implementation with one inline cost hook provided :) Have we compared the pros-and-cons of the two approaches?

The pro of using module pass include : avoid building CG so it might be a compile time win (depending on how expensive GlobalDCE is).

The cons I can see:

1. need to insert life time marker at O0 and depend on stack coloring to be turned on at O0 (which can be expensive)
2. need to run GlobalDCE
3. It is likely in the future CG is needed at O0 for other reasons, then the benefit of avoiding CG will be gone.
4. less code sharing.

Did I miss others?

[davidxl]

For the record, I am all for a more general Module-pass based inliner. The motivation is that it allows us to do a very quick round of priority based inlining (e.g, wrapper call inliing or other top-down heuristics) to avoid the limitation of bottom-up inlining. However the general framework still needs CG (and update) though.

[eraman]

In D23299#512857, @chandlerc wrote:

In D23299#512749, @davidxl wrote:

Why is this pass implemented as a module pass (instead of cgscc pass as before)? Is there any compile time concerns (at O0) ?

As I tried to explain in the patch description (and let me know if I should improve it): because a module pass is simpler, and avoids computing the call graph when we don't need it.

It might help compile time, but that wasn't my primary concern.

In D23299#512879, @davidxl wrote:

yes, I understand the module pass is quite simple and straightforward, but it seems to me the legacy always inliner is even simpler -- as it is simply one 'instantiation' of a common inliner implementation with one inline cost hook provided :) Have we compared the pros-and-cons of the two approaches?

The pro of using module pass include : avoid building CG so it might be a compile time win (depending on how expensive GlobalDCE is).

The cons I can see:

1. need to insert life time marker at O0 and depend on stack coloring to be turned on at O0 (which can be expensive)
2. need to run GlobalDCE
3. It is likely in the future CG is needed at O0 for other reasons, then the benefit of avoiding CG will be gone.
4. less code sharing.

Did I miss others?

When used in the -O1 pipeline, we might end up inlining less with the new AlwaysInliner. This is because the isInlineViable call scans the callee for conditions that disallow inlining and it matters whether some function pass gets rid of them (if they are in unreachable paths, for example). In the case of existing CGSCC AlwaysInliner pass, since we optimize a CGSCC node with function passes before moving to its parent, isInlineViable could get more precise result. No such cleanup happens in the module pass (unless we run some cleanup passes before the module pass). I don't think this difference matters much in practice though.

[chandlerc]

In D23299#512879, @davidxl wrote:

yes, I understand the module pass is quite simple and straightforward, but it seems to me the legacy always inliner is even simpler -- as it is simply one 'instantiation' of a common inliner implementation with one inline cost hook provided :) Have we compared the pros-and-cons of the two approaches?

I mean, yes. ;] I didn't do this without some careful thought.

The pro of using module pass include : avoid building CG so it might be a compile time win (depending on how expensive GlobalDCE is).

• Avoid coupling the always inliner which needs no cost model to an inliner built entirely around cost modeling
• Avoid abstractions between the always cost model (or lack there of) and a concrete cost model
• Avoid the complexity of rigging up and potentially emitting remarks when the decision of whether or not to inline is completely predictable from source.

The cons I can see:

1. need to insert life time marker at O0 and depend on stack coloring to be turned on at O0 (which can be expensive)

If we need to do this at all. It isn't clear that we do. I've specifically said we can add alloca merging as a follow-up if it proves necessary.

2. need to run GlobalDCE

This adds no complexity though. The goal is to de-couple things which should make it overall more simple.

3. It is likely in the future CG is needed at O0 for other reasons, then the benefit of avoiding CG will be gone.

I don't see how this can be a con... It seems circular.

4. less code sharing.

I disagree. This shares all relevant logic with the existing inliner via the InlineFunction routine.

Did I miss others?

I outlined all of the ones I saw in the patch description already (and it does include a couple you didn't mention) but I also described why I don't find them compelling to use a more complex approach.

It is probably worth highlighting that this new pass requires *less code* than the old version does even if we don't count any of the common inliner code or logic shared with other inliners. Despite the fact that some of that code only exists to support this pass's use case.

In D23299#512903, @davidxl wrote:

For the record, I am all for a more general Module-pass based inliner. The motivation is that it allows us to do a very quick round of priority based inlining (e.g, wrapper call inliing or other top-down heuristics) to avoid the limitation of bottom-up inlining. However the general framework still needs CG (and update) though.

I think that is a completely different discussion. My goal is for the always inliner to be the simplest thing possible that merely inlines function bodies based on a single signal: the alwaysinline attribute.

In D23299#512909, @eraman wrote:

In D23299#512749, @davidxl wrote:

Did I miss others?

When used in the -O1 pipeline, we might end up inlining less with the new AlwaysInliner.

The fact that -O1 uses the always inliner ... doesn't make much sense IMO. In fact, I suspect most don't realize that this is the case. It dates from r89464 in 2009 when this logic was added as part of some code dump of option parsing -- I suspect ported from the python driver wrapper. I don't see any particular logic for this model. I strongly suspect we should do something closer to -Os's inlining logic at -O1.

And I think we have the flexibility to do exactly this with the new PM. I'm not changing the old pass manager in any way.

This is because the isInlineViable call scans the callee for conditions that disallow inlining and it matters whether some function pass gets rid of them (if they are in unreachable paths, for example). In the case of existing CGSCC AlwaysInliner pass, since we optimize a CGSCC node with function passes before moving to its parent, isInlineViable could get more precise result. No such cleanup happens in the module pass (unless we run some cleanup passes before the module pass). I don't think this difference matters much in practice though.

Yea, this is a difference, but I strongly agree it doesn't matter in practice.

Notably, we'd really like the alwaysinline to not be a hint but a guarantee. As such, it seems like a much bigger problem if some code has this attribute and relies on DCE or something else to be viable for inlining.

[davidxl]

I don't know why it is a bad thing to use a super simple cost model for always-inliner (as done by Old pass manager).

Anyway, I don't think this patch should be held because of the difference in opinions here: it is not yet on by default and if we see problems, it can always be revisited (assuming implementing this as regular CG based inliner in new PM is possible) later.

I do think a follow up is needed to handle always-inline callsite attribute.

lgtm

[chandlerc]

In D23299#512879, @davidxl wrote:

yes, I understand the module pass is quite simple and straightforward, but it seems to me the legacy always inliner is even simpler -- as it is simply one 'instantiation' of a common inliner implementation with one inline cost hook provided :) Have we compared the pros-and-cons of the two approaches?

I mean, yes. ;] I didn't do this without some careful thought.

The pro of using module pass include : avoid building CG so it might be a compile time win (depending on how expensive GlobalDCE is).

• Avoid coupling the always inliner which needs no cost model to an inliner built entirely around cost modeling
• Avoid abstractions between the always cost model (or lack there of) and a concrete cost model
• Avoid the complexity of rigging up and potentially emitting remarks when the decision of whether or not to inline is completely predictable from source.

The cons I can see:

1. need to insert life time marker at O0 and depend on stack coloring to be turned on at O0 (which can be expensive)

If we need to do this at all. It isn't clear that we do. I've specifically said we can add alloca merging as a follow-up if it proves necessary.

2. need to run GlobalDCE

This adds no complexity though. The goal is to de-couple things which should make it overall more simple.

3. It is likely in the future CG is needed at O0 for other reasons, then the benefit of avoiding CG will be gone.

I don't see how this can be a con... It seems circular.

4. less code sharing.

I disagree. This shares all relevant logic with the existing inliner via the InlineFunction routine.

Did I miss others?

I outlined all of the ones I saw in the patch description already (and it does include a couple you didn't mention) but I also described why I don't find them compelling to use a more complex approach.

It is probably worth highlighting that this new pass requires *less code* than the old version does even if we don't count any of the common inliner code or logic shared with other inliners. Despite the fact that some of that code only exists to support this pass's use case.

In D23299#512903, @davidxl wrote:

For the record, I am all for a more general Module-pass based inliner. The motivation is that it allows us to do a very quick round of priority based inlining (e.g, wrapper call inliing or other top-down heuristics) to avoid the limitation of bottom-up inlining. However the general framework still needs CG (and update) though.

I think that is a completely different discussion. My goal is for the always inliner to be the simplest thing possible that merely inlines function bodies based on a single signal: the alwaysinline attribute.

In D23299#512909, @eraman wrote:

In D23299#512749, @davidxl wrote:

Did I miss others?

When used in the -O1 pipeline, we might end up inlining less with the new AlwaysInliner.

The fact that -O1 uses the always inliner ... doesn't make much sense IMO. In fact, I suspect most don't realize that this is the case. It dates from r89464 in 2009 when this logic was added as part of some code dump of option parsing -- I suspect ported from the python driver wrapper. I don't see any particular logic for this model. I strongly suspect we should do something closer to -Os's inlining logic at -O1.

And I think we have the flexibility to do exactly this with the new PM. I'm not changing the old pass manager in any way.

This is because the isInlineViable call scans the callee for conditions that disallow inlining and it matters whether some function pass gets rid of them (if they are in unreachable paths, for example). In the case of existing CGSCC AlwaysInliner pass, since we optimize a CGSCC node with function passes before moving to its parent, isInlineViable could get more precise result. No such cleanup happens in the module pass (unless we run some cleanup passes before the module pass). I don't think this difference matters much in practice though.

Yea, this is a difference, but I strongly agree it doesn't matter in practice.

Notably, we'd really like the alwaysinline to not be a hint but a guarantee. As such, it seems like a much bigger problem if some code has this attribute and relies on DCE or something else to be viable for inlining.

In D23299#513854, @davidxl wrote:

I don't know why it is a bad thing to use a super simple cost model for always-inliner (as done by Old pass manager).

Anyway, I don't think this patch should be held because of the difference in opinions here: it is not yet on by default and if we see problems, it can always be revisited (assuming implementing this as regular CG based inliner in new PM is possible) later.

I do think a follow up is needed to handle always-inline callsite attribute.

lgtm

Thanks. I'll try to re-invigorate the always-inline refinement that was already underway.

[leonardchan]

Hi, I'm attempting to fix all clang tests that fail when enabling the new PM by default and one of the failing tests is CodeGen/flatten.c which tests the flatten attribute. According to the docs, this attribute attempts to inline function calls in functions marked with flatten. But if I understand this patch correctly, one of the intentions is only inlining for functions and not calls.

Is it set in stone that the new PM will not inline calls? Otherwise, it seems to break flatten for -O0 cases with new PM. At a quick glance, it seems that we can enable this on callsites by adding a check for AlwaysInline here and removing the F.hasFnAttribute(Attribute::AlwaysInline) here.

llvm/trunk/include/llvm/InitializePasses.h

 void initializeAAEvalLegacyPassPass(PassRegistry&);
 void initializeAAResultsWrapperPassPass(PassRegistry &);
 void initializeADCELegacyPassPass(PassRegistry&);
 void initializeAddDiscriminatorsLegacyPassPass(PassRegistry&);
 void initializeAddressSanitizerModulePass(PassRegistry&);
 void initializeAddressSanitizerPass(PassRegistry&);
 void initializeAliasSetPrinterPass(PassRegistry&);
 void initializeAlignmentFromAssumptionsPass(PassRegistry&);
-void initializeAlwaysInlinerPass(PassRegistry&);
+void initializeAlwaysInlinerLegacyPassPass(PassRegistry&);
 void initializeArgPromotionPass(PassRegistry&);
 void initializeAssumptionCacheTrackerPass(PassRegistry &);
 void initializeAtomicExpandPass(PassRegistry&);
 void initializeBBVectorizePass(PassRegistry&);
 void initializeBDCELegacyPassPass(PassRegistry &);
 void initializeBarrierNoopPass(PassRegistry&);
 void initializeBasicAAWrapperPassPass(PassRegistry&);
 void initializeBlockExtractorPassPass(PassRegistry&);

llvm/trunk/include/llvm/LinkAllPasses.h

 #include "llvm/Analysis/ScalarEvolutionAliasAnalysis.h"
 #include "llvm/Analysis/ScopedNoAliasAA.h"
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TypeBasedAliasAnalysis.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/IRPrintingPasses.h"
 #include "llvm/Transforms/IPO.h"
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/FunctionAttrs.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/ObjCARC.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Scalar/GVN.h"
 #include "llvm/Transforms/Utils/SymbolRewriter.h"
 #include "llvm/Transforms/Utils/UnifyFunctionExitNodes.h"
 #include "llvm/Transforms/Vectorize.h"
ForcePassLinking() {
       (void) llvm::createDomViewerPass();
       (void) llvm::createGCOVProfilerPass();
       (void) llvm::createPGOInstrumentationGenLegacyPass();
       (void) llvm::createPGOInstrumentationUseLegacyPass();
       (void) llvm::createPGOIndirectCallPromotionLegacyPass();
       (void) llvm::createInstrProfilingLegacyPass();
       (void) llvm::createFunctionImportPass();
       (void) llvm::createFunctionInliningPass();
-      (void) llvm::createAlwaysInlinerPass();
+      (void) llvm::createAlwaysInlinerLegacyPass();
       (void) llvm::createGlobalDCEPass();
       (void) llvm::createGlobalOptimizerPass();
       (void) llvm::createGlobalsAAWrapperPass();
       (void) llvm::createGuardWideningPass();
       (void) llvm::createIPConstantPropagationPass();
       (void) llvm::createIPSCCPPass();
       (void) llvm::createInductiveRangeCheckEliminationPass();
       (void) llvm::createIndVarSimplifyPass();

llvm/trunk/include/llvm/Transforms/IPO.h

 /// The -inline-threshold command line option takes precedence over the
 /// threshold given here.
 Pass *createFunctionInliningPass();
 Pass *createFunctionInliningPass(int Threshold);
 Pass *createFunctionInliningPass(unsigned OptLevel, unsigned SizeOptLevel);
 Pass *createFunctionInliningPass(InlineParams &Params);
 
 //===----------------------------------------------------------------------===//
-/// createAlwaysInlinerPass - Return a new pass object that inlines only
-/// functions that are marked as "always_inline".
-Pass *createAlwaysInlinerPass();
-Pass *createAlwaysInlinerPass(bool InsertLifetime);
-
-//===----------------------------------------------------------------------===//
 /// createPruneEHPass - Return a new pass object which transforms invoke
 /// instructions into calls, if the callee can _not_ unwind the stack.
 ///
 Pass *createPruneEHPass();
 
 //===----------------------------------------------------------------------===//
 /// createInternalizePass - This pass loops over all of the functions in the
 /// input module, internalizing all globals (functions and variables) it can.

llvm/trunk/include/llvm/Transforms/IPO/AlwaysInliner.h

+//===-- AlwaysInliner.h - Pass to inline "always_inline" functions --------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Provides passes to inlining "always_inline" functions.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TRANSFORMS_IPO_ALWAYSINLINER_H
+#define LLVM_TRANSFORMS_IPO_ALWAYSINLINER_H
+
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+/// Inlines functions marked as "always_inline".
+///
+/// Note that this does not inline call sites marked as always_inline and does
+/// not delete the functions even when all users are inlined. The normal
+/// inliner should be used to handle call site inlining, this pass's goal is to
+/// be the simplest possible pass to remove always_inline function definitions'
+/// uses by inlining them. The \c GlobalDCE pass can be used to remove these
+/// functions once all users are gone.
+struct AlwaysInlinerPass : PassInfoMixin<AlwaysInlinerPass> {
+  PreservedAnalyses run(Module &M, ModuleAnalysisManager &);
+};
+
+/// Create a legacy pass manager instance of a pass to inline and remove
+/// functions marked as "always_inline".
+Pass *createAlwaysInlinerLegacyPass(bool InsertLifetime = true);
+
+}
+
+#endif // LLVM_TRANSFORMS_IPO_ALWAYSINLINER_H

llvm/trunk/lib/Passes/PassBuilder.cpp

 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/IRPrintingPasses.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/Verifier.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/GCOVProfiler.h"
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/ConstantMerge.h"
 #include "llvm/Transforms/IPO/CrossDSOCFI.h"
 #include "llvm/Transforms/IPO/DeadArgumentElimination.h"
 #include "llvm/Transforms/IPO/ElimAvailExtern.h"
 #include "llvm/Transforms/IPO/ForceFunctionAttrs.h"
 #include "llvm/Transforms/IPO/FunctionAttrs.h"
 #include "llvm/Transforms/IPO/FunctionImport.h"
 #include "llvm/Transforms/IPO/GlobalDCE.h"

llvm/trunk/lib/Passes/PassRegistry.def

 #endif
 MODULE_ALIAS_ANALYSIS("globals-aa", GlobalsAA())
 #undef MODULE_ALIAS_ANALYSIS
 #undef MODULE_ANALYSIS
 
 #ifndef MODULE_PASS
 #define MODULE_PASS(NAME, CREATE_PASS)
 #endif
+MODULE_PASS("always-inline", AlwaysInlinerPass())
 MODULE_PASS("constmerge", ConstantMergePass())
 MODULE_PASS("cross-dso-cfi", CrossDSOCFIPass())
 MODULE_PASS("deadargelim", DeadArgumentEliminationPass())
 MODULE_PASS("elim-avail-extern", EliminateAvailableExternallyPass())
 MODULE_PASS("forceattrs", ForceFunctionAttrsPass())
 MODULE_PASS("function-import", FunctionImportPass())
 MODULE_PASS("globaldce", GlobalDCEPass())
 MODULE_PASS("globalopt", GlobalOptPass())

llvm/trunk/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

 #include "SIISelLowering.h"
 #include "SIInstrInfo.h"
 #include "SIMachineScheduler.h"
 #include "llvm/CodeGen/GlobalISel/IRTranslator.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/CodeGen/TargetPassConfig.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Transforms/IPO.h"
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/Scalar.h"
 #include "llvm/Transforms/Scalar/GVN.h"
 #include "llvm/Transforms/Vectorize.h"
 
 using namespace llvm;
 
 static cl::opt<bool> EnableR600StructurizeCFG(
   "r600-ir-structurize",

 void AMDGPUPassConfig::addIRPasses() {
   // There is no reason to run these.
   disablePass(&StackMapLivenessID);
   disablePass(&FuncletLayoutID);
   disablePass(&PatchableFunctionID);
 
   // Function calls are not supported, so make sure we inline everything.
   addPass(createAMDGPUAlwaysInlinePass());
-  addPass(createAlwaysInlinerPass());
+  addPass(createAlwaysInlinerLegacyPass());
   // We need to add the barrier noop pass, otherwise adding the function
   // inlining pass will cause all of the PassConfigs passes to be run
   // one function at a time, which means if we have a nodule with two
   // functions, then we will generate code for the first function
   // without ever running any passes on the second.
   addPass(createBarrierNoopPass());
 
   // Handle uses of OpenCL image2d_t, image3d_t and sampler_t arguments.

llvm/trunk/lib/Transforms/IPO/AlwaysInliner.cpp

+//===- InlineAlways.cpp - Code to inline always_inline functions ----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a custom inliner that handles only functions that
+// are marked as "always inline".
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
+#include "llvm/ADT/SetVector.h"
+#include "llvm/Analysis/AssumptionCache.h"
+#include "llvm/Analysis/CallGraph.h"
+#include "llvm/Analysis/InlineCost.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/IR/CallSite.h"
+#include "llvm/IR/CallingConv.h"
+#include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/Type.h"
+#include "llvm/Transforms/IPO/InlinerPass.h"
+#include "llvm/Transforms/Utils/Cloning.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "inline"
+
+PreservedAnalyses AlwaysInlinerPass::run(Module &M, ModuleAnalysisManager &) {
+  InlineFunctionInfo IFI;
+  SmallSetVector<CallSite, 16> Calls;
+  bool Changed = false;
+  for (Function &F : M)
+    if (!F.isDeclaration() && F.hasFnAttribute(Attribute::AlwaysInline) &&
+        isInlineViable(F)) {
+      Calls.clear();
+
+      for (User *U : F.users())
+        if (auto CS = CallSite(U))
+          if (CS.getCalledFunction() == &F)
+            Calls.insert(CS);
+
+      for (CallSite CS : Calls)
+        // FIXME: We really shouldn't be able to fail to inline at this point!
+        // We should do something to log or check the inline failures here.
+        Changed |= InlineFunction(CS, IFI);
+    }
+
+  return Changed ? PreservedAnalyses::none() : PreservedAnalyses::all();
+}
+
+namespace {
+
+/// Inliner pass which only handles "always inline" functions.
+///
+/// Unlike the \c AlwaysInlinerPass, this uses the more heavyweight \c Inliner
+/// base class to provide several facilities such as array alloca merging.
+class AlwaysInlinerLegacyPass : public Inliner {
+
+public:
+  AlwaysInlinerLegacyPass() : Inliner(ID, /*InsertLifetime*/ true) {
+    initializeAlwaysInlinerLegacyPassPass(*PassRegistry::getPassRegistry());
+  }
+
+  AlwaysInlinerLegacyPass(bool InsertLifetime) : Inliner(ID, InsertLifetime) {
+    initializeAlwaysInlinerLegacyPassPass(*PassRegistry::getPassRegistry());
+  }
+
+  /// Main run interface method.  We override here to avoid calling skipSCC().
+  bool runOnSCC(CallGraphSCC &SCC) override { return inlineCalls(SCC); }
+
+  static char ID; // Pass identification, replacement for typeid
+
+  InlineCost getInlineCost(CallSite CS) override;
+
+  using llvm::Pass::doFinalization;
+  bool doFinalization(CallGraph &CG) override {
+    return removeDeadFunctions(CG, /*AlwaysInlineOnly=*/true);
+  }
+};
+}
+
+char AlwaysInlinerLegacyPass::ID = 0;
+INITIALIZE_PASS_BEGIN(AlwaysInlinerLegacyPass, "always-inline",
+                      "Inliner for always_inline functions", false, false)
+INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
+INITIALIZE_PASS_DEPENDENCY(CallGraphWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
+INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
+INITIALIZE_PASS_END(AlwaysInlinerLegacyPass, "always-inline",
+                    "Inliner for always_inline functions", false, false)
+
+Pass *llvm::createAlwaysInlinerLegacyPass(bool InsertLifetime) {
+  return new AlwaysInlinerLegacyPass(InsertLifetime);
+}
+
+/// \brief Get the inline cost for the always-inliner.
+///
+/// The always inliner *only* handles functions which are marked with the
+/// attribute to force inlining. As such, it is dramatically simpler and avoids
+/// using the powerful (but expensive) inline cost analysis. Instead it uses
+/// a very simple and boring direct walk of the instructions looking for
+/// impossible-to-inline constructs.
+///
+/// Note, it would be possible to go to some lengths to cache the information
+/// computed here, but as we only expect to do this for relatively few and
+/// small functions which have the explicit attribute to force inlining, it is
+/// likely not worth it in practice.
+InlineCost AlwaysInlinerLegacyPass::getInlineCost(CallSite CS) {
+  Function *Callee = CS.getCalledFunction();
+
+  // Only inline direct calls to functions with always-inline attributes
+  // that are viable for inlining. FIXME: We shouldn't even get here for
+  // declarations.
+  if (Callee && !Callee->isDeclaration() &&
+      CS.hasFnAttr(Attribute::AlwaysInline) && isInlineViable(*Callee))
+    return InlineCost::getAlways();
+
+  return InlineCost::getNever();
+}

llvm/trunk/lib/Transforms/IPO/CMakeLists.txt

 add_llvm_library(LLVMipo
+  AlwaysInliner.cpp
   ArgumentPromotion.cpp
   BarrierNoopPass.cpp
   ConstantMerge.cpp
   CrossDSOCFI.cpp
   DeadArgumentElimination.cpp
   ElimAvailExtern.cpp
   ExtractGV.cpp
   ForceFunctionAttrs.cpp
   FunctionAttrs.cpp
   FunctionImport.cpp
   GlobalDCE.cpp
   GlobalOpt.cpp
   IPConstantPropagation.cpp
   IPO.cpp
   InferFunctionAttrs.cpp
-  InlineAlways.cpp
   InlineSimple.cpp
   Inliner.cpp
   Internalize.cpp
   LoopExtractor.cpp
   LowerTypeTests.cpp
   MergeFunctions.cpp
   PartialInlining.cpp
   PassManagerBuilder.cpp

llvm/trunk/lib/Transforms/IPO/IPO.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm-c/Initialization.h"
 #include "llvm-c/Transforms/IPO.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/IR/LegacyPassManager.h"
 #include "llvm/Transforms/IPO.h"
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/FunctionAttrs.h"
 
 using namespace llvm;
 
 void llvm::initializeIPO(PassRegistry &Registry) {
   initializeArgPromotionPass(Registry);
   initializeConstantMergeLegacyPassPass(Registry);
   initializeCrossDSOCFIPass(Registry);
   initializeDAEPass(Registry);
   initializeDAHPass(Registry);
   initializeForceFunctionAttrsLegacyPassPass(Registry);
   initializeGlobalDCELegacyPassPass(Registry);
   initializeGlobalOptLegacyPassPass(Registry);
   initializeIPCPPass(Registry);
-  initializeAlwaysInlinerPass(Registry);
+  initializeAlwaysInlinerLegacyPassPass(Registry);
   initializeSimpleInlinerPass(Registry);
   initializeInferFunctionAttrsLegacyPassPass(Registry);
   initializeInternalizeLegacyPassPass(Registry);
   initializeLoopExtractorPass(Registry);
   initializeBlockExtractorPassPass(Registry);
   initializeSingleLoopExtractorPass(Registry);
   initializeLowerTypeTestsPass(Registry);
   initializeMergeFunctionsPass(Registry);
void LLVMAddFunctionAttrsPass(LLVMPassManagerRef PM) {
   unwrap(PM)->add(createPostOrderFunctionAttrsLegacyPass());
 }
 
 void LLVMAddFunctionInliningPass(LLVMPassManagerRef PM) {
   unwrap(PM)->add(createFunctionInliningPass());
 }
 
 void LLVMAddAlwaysInlinerPass(LLVMPassManagerRef PM) {
-  unwrap(PM)->add(llvm::createAlwaysInlinerPass());
+  unwrap(PM)->add(llvm::createAlwaysInlinerLegacyPass());
 }
 
 void LLVMAddGlobalDCEPass(LLVMPassManagerRef PM) {
   unwrap(PM)->add(createGlobalDCEPass());
 }
 
 void LLVMAddGlobalOptimizerPass(LLVMPassManagerRef PM) {
   unwrap(PM)->add(createGlobalOptimizerPass());

llvm/trunk/lib/Transforms/IPO/InlineAlways.cpp

-//===- InlineAlways.cpp - Code to inline always_inline functions ----------===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements a custom inliner that handles only functions that
-// are marked as "always inline".
-//
-//===----------------------------------------------------------------------===//
-
-#include "llvm/ADT/SmallPtrSet.h"
-#include "llvm/Analysis/AssumptionCache.h"
-#include "llvm/Analysis/CallGraph.h"
-#include "llvm/Analysis/InlineCost.h"
-#include "llvm/Analysis/ProfileSummaryInfo.h"
-#include "llvm/Analysis/TargetLibraryInfo.h"
-#include "llvm/IR/CallSite.h"
-#include "llvm/IR/CallingConv.h"
-#include "llvm/IR/DataLayout.h"
-#include "llvm/IR/Instructions.h"
-#include "llvm/IR/IntrinsicInst.h"
-#include "llvm/IR/Module.h"
-#include "llvm/IR/Type.h"
-#include "llvm/Transforms/IPO.h"
-#include "llvm/Transforms/IPO/InlinerPass.h"
-
-using namespace llvm;
-
-#define DEBUG_TYPE "inline"
-
-namespace {
-
-/// \brief Inliner pass which only handles "always inline" functions.
-class AlwaysInliner : public Inliner {
-
-public:
-  AlwaysInliner() : Inliner(ID, /*InsertLifetime*/ true) {
-    initializeAlwaysInlinerPass(*PassRegistry::getPassRegistry());
-  }
-
-  AlwaysInliner(bool InsertLifetime) : Inliner(ID, InsertLifetime) {
-    initializeAlwaysInlinerPass(*PassRegistry::getPassRegistry());
-  }
-
-  /// Main run interface method.  We override here to avoid calling skipSCC().
-  bool runOnSCC(CallGraphSCC &SCC) override { return inlineCalls(SCC); }
-
-  static char ID; // Pass identification, replacement for typeid
-
-  InlineCost getInlineCost(CallSite CS) override;
-
-  using llvm::Pass::doFinalization;
-  bool doFinalization(CallGraph &CG) override {
-    return removeDeadFunctions(CG, /*AlwaysInlineOnly=*/true);
-  }
-};
-}
-
-char AlwaysInliner::ID = 0;
-INITIALIZE_PASS_BEGIN(AlwaysInliner, "always-inline",
-                      "Inliner for always_inline functions", false, false)
-INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
-INITIALIZE_PASS_DEPENDENCY(CallGraphWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(ProfileSummaryInfoWrapperPass)
-INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
-INITIALIZE_PASS_END(AlwaysInliner, "always-inline",
-                    "Inliner for always_inline functions", false, false)
-
-Pass *llvm::createAlwaysInlinerPass() { return new AlwaysInliner(); }
-
-Pass *llvm::createAlwaysInlinerPass(bool InsertLifetime) {
-  return new AlwaysInliner(InsertLifetime);
-}
-
-/// \brief Get the inline cost for the always-inliner.
-///
-/// The always inliner *only* handles functions which are marked with the
-/// attribute to force inlining. As such, it is dramatically simpler and avoids
-/// using the powerful (but expensive) inline cost analysis. Instead it uses
-/// a very simple and boring direct walk of the instructions looking for
-/// impossible-to-inline constructs.
-///
-/// Note, it would be possible to go to some lengths to cache the information
-/// computed here, but as we only expect to do this for relatively few and
-/// small functions which have the explicit attribute to force inlining, it is
-/// likely not worth it in practice.
-InlineCost AlwaysInliner::getInlineCost(CallSite CS) {
-  Function *Callee = CS.getCalledFunction();
-
-  // Only inline direct calls to functions with always-inline attributes
-  // that are viable for inlining. FIXME: We shouldn't even get here for
-  // declarations.
-  if (Callee && !Callee->isDeclaration() &&
-      CS.hasFnAttr(Attribute::AlwaysInline) && isInlineViable(*Callee))
-    return InlineCost::getAlways();
-
-  return InlineCost::getNever();
-}

llvm/trunk/lib/Transforms/Utils/InlineFunction.cpp

if (const Instruction *I = dyn_cast<Instruction>(VMI->first)) {
                                 MDNode::get(CalledFunc->getContext(), Scopes)));
     }
   }
 }
 
 /// If the inlined function has non-byval align arguments, then
 /// add @llvm.assume-based alignment assumptions to preserve this information.
 static void AddAlignmentAssumptions(CallSite CS, InlineFunctionInfo &IFI) {
-  if (!PreserveAlignmentAssumptions)
+  if (!PreserveAlignmentAssumptions || !IFI.GetAssumptionCache)
     return;
   AssumptionCache *AC = IFI.GetAssumptionCache
                             ? &(*IFI.GetAssumptionCache)(*CS.getCaller())
                             : nullptr;
   auto &DL = CS.getCaller()->getParent()->getDataLayout();
 
   // To avoid inserting redundant assumptions, we should check for assumptions
   // already in the caller. To do this, we might need a DT of the caller.

llvm/trunk/test/Transforms/Inline/always-inline.ll

-; RUN: opt < %s -inline-threshold=0 -always-inline -S | FileCheck %s
+; RUN: opt < %s -inline-threshold=0 -always-inline -S | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-CALL
 ;
 ; Ensure the threshold has no impact on these decisions.
-; RUN: opt < %s -inline-threshold=20000000 -always-inline -S | FileCheck %s
-; RUN: opt < %s -inline-threshold=-20000000 -always-inline -S | FileCheck %s
+; RUN: opt < %s -inline-threshold=20000000 -always-inline -S | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-CALL
+; RUN: opt < %s -inline-threshold=-20000000 -always-inline -S | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-CALL
+;
+; The new pass manager doesn't re-use any threshold based infrastructure for
+; the always inliner, but test that we get the correct result.
+; RUN: opt < %s -passes=always-inline -S | FileCheck %s --check-prefix=CHECK
 
 define i32 @inner1() alwaysinline {
   ret i32 1
 }
 define i32 @outer1() {
 ; CHECK-LABEL: @outer1(
 ; CHECK-NOT: call
 ; CHECK: ret
entry:
   call void @inner6(i32 42)
   ret void
 }
 
 define i32 @inner7() {
   ret i32 1
 }
 define i32 @outer7() {
-; CHECK-LABEL: @outer7(
-; CHECK-NOT: call
-; CHECK: ret
+; CHECK-CALL-LABEL: @outer7(
+; CHECK-CALL-NOT: call
+; CHECK-CALL: ret
 
    %r = call i32 @inner7() alwaysinline
    ret i32 %r
 }
+
+define float* @inner8(float* nocapture align 128 %a) alwaysinline {
+  ret float* %a
+}
+define float @outer8(float* nocapture %a) {
+; CHECK-LABEL: @outer8(
+; CHECK-NOT: call float* @inner8
+; CHECK: ret
+
+  %inner_a = call float* @inner8(float* %a)
+  %f = load float, float* %inner_a, align 4
+  ret float %f
+}

llvm/trunk/tools/bugpoint/bugpoint.cpp

 #include "llvm/LinkAllPasses.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/PluginLoader.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/Valgrind.h"
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 
 //Enable this macro to debug bugpoint itself.
 //#define DEBUG_BUGPOINT 1
 
 using namespace llvm;
 
 static cl::opt<bool>
if (StandardLinkOpts) {
     PassManagerBuilder Builder;
     Builder.Inliner = createFunctionInliningPass();
     Builder.populateLTOPassManager(PM);
   }
 
   if (OptLevelO1 || OptLevelO2 || OptLevelO3) {
     PassManagerBuilder Builder;
     if (OptLevelO1)
-      Builder.Inliner = createAlwaysInlinerPass();
+      Builder.Inliner = createAlwaysInlinerLegacyPass();
     else if (OptLevelOs || OptLevelO2)
       Builder.Inliner = createFunctionInliningPass(2, OptLevelOs ? 1 : 0);
     else
       Builder.Inliner = createFunctionInliningPass(275);
     Builder.populateFunctionPassManager(PM);
     Builder.populateModulePassManager(PM);
   }
 

llvm/trunk/tools/opt/opt.cpp

 #include "llvm/Support/Signals.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Support/SystemUtils.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/TargetSelect.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Transforms/Coroutines.h"
+#include "llvm/Transforms/IPO/AlwaysInliner.h"
 #include "llvm/Transforms/IPO/PassManagerBuilder.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include <algorithm>
 #include <memory>
 using namespace llvm;
 using namespace opt_tool;
 
 // The OptimizationList is automatically populated with registered Passes by the
static void AddOptimizationPasses(legacy::PassManagerBase &MPM,
   Builder.OptLevel = OptLevel;
   Builder.SizeLevel = SizeLevel;
 
   if (DisableInline) {
     // No inlining pass
   } else if (OptLevel > 1) {
     Builder.Inliner = createFunctionInliningPass(OptLevel, SizeLevel);
   } else {
-    Builder.Inliner = createAlwaysInlinerPass();
+    Builder.Inliner = createAlwaysInlinerLegacyPass();
   }
   Builder.DisableUnitAtATime = !UnitAtATime;
   Builder.DisableUnrollLoops = (DisableLoopUnrolling.getNumOccurrences() > 0) ?
                                DisableLoopUnrolling : OptLevel == 0;
 
   // This is final, unless there is a #pragma vectorize enable
   if (DisableLoopVectorization)
     Builder.LoopVectorize = false;