[Inliner] Inlining to enable more loop unrolling

[rcox2]

Summary

This patch extends the inliner to give a stronger preference to inline functions which allow trip counts of inner loops to become small constants in the inlined code.

For example:

 extern int foo(int m, int n, float b[n], float a[m][n])  { 
    for (i = 0; I < n; i++) 
      for (j = 0; j < m; j++) 
        a[i][j] = a[i][j] + b[i]
} 

 extern int main() { 
 float b[4], a[3,4]; 
 …
 foo(3, 4, b, a); 
 …
 }

Inlining foo will expose the inner loop bound of 3. Such loops can be good candidates for complete unrolling.

The function convolve() in the geekbench benchmark sharpen_filter is another such example.

The max trip count value chosen is defined in include/llvm/Analysis/InlineCost.h as

const int SmallTripCountMax = 10;

to match the default value of UnrollMaxIterationsCountToAnalyze in lib/Transforms/Scalar/LoopUnrollPass.cpp.

The patch makes use of DominatorTree, LoopInfo, and ScalarExpansion to find loops and potential constant loop trip counts in the potentially inlined functions. An instance of InliningLoopInfoCache is used to cache results for functions and so that compile time is not spent recomputing the DominatorTree, LoopInfo, and ScalarExpansion info for functions into which no inlining has occurred. Our experiments show that this cache is an effective and essential mechanism to keep the inlining compile time low.

Patch by Robert Cox

[rcox2]

No comments on this review, yet. Any chance the reviewers I picked will be able to look at it in the near future? Should I add some other names? -- Thanks, Robert

[reames]

Resigning from a stale review (2016). Feel free to re-add if thread ever revived.

include/llvm/Analysis/InlineCost.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_ANALYSIS_INLINECOST_H
 #define LLVM_ANALYSIS_INLINECOST_H
 
 #include "llvm/Analysis/CallGraphSCCPass.h"
 #include "llvm/Analysis/AssumptionCache.h"
+#include "llvm/Analysis/LoopInfo.h"
+#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
 #include <cassert>
 #include <climits>
 
 namespace llvm {
 class AssumptionCacheTracker;
 class CallSite;
 class DataLayout;
 class Function;

 const int IndirectCallThreshold = 100;
 const int CallPenalty = 25;
 const int LastCallToStaticBonus = 15000;
 const int ColdccPenalty = 2000;
 const int NoreturnPenalty = 10000;
 /// Do not inline functions which allocate this many bytes on the stack
 /// when the caller is recursive.
 const unsigned TotalAllocaSizeRecursiveCaller = 1024;
+// We give preference (by reducing the cost by 
+// StaticSmallConstantTripCountBonus) to inlining of routines which make
+// a loop in the called routine's code go from being unknown to a small
+// constant (less than or equal to SmallTripCountMax).  This was motivated 
+// by seeing such loops getting profitably unrolled. The value is equal to
+// UnrollMaxIterationsCountToAnalyze from 
+//   llvm/lib/Transforms/Scalar/LoopUnrollPass.cpp.
+const int SmallTripCountMax = 10;
+const int StaticSmallConstantTripCountBonus = 410;
 }
 
+/// \brief A cache to save loop related info during inlining of an SCC.
+///
+/// This cache is used to store the DominatorTree, LoopInfo, and
+/// ScalarEvolution for called functions which are candidates for
+/// inlining.  Unnecessarily recomputing them can add significantly to
+/// compile time and memory consumption.
+///
+/// The DominatorTree, LoopInfo, and ScalarEvolution for a function is
+/// invalidated when someother function is inlined into it.
+///
+typedef std::map<Function*, DominatorTree*> DTMap;
+typedef std::map<Function*, LoopInfo*> LIMap;
+typedef std::map<Function*, ScalarEvolution*> SEMap;
+
+class InliningLoopInfoCache {
+
+  DTMap DTMapSCC;
+  LIMap LIMapSCC;
+  SEMap SEMapSCC;
+
+public:
+
+  InliningLoopInfoCache() {}
+
+  ~InliningLoopInfoCache();
+
+  DominatorTree* getDT(Function *F);
+  LoopInfo* getLI(Function* F);
+  ScalarEvolution* getSE(Function *F, TargetLibraryInfo& TLI,
+    std::function<AssumptionCache &(Function &)> &GetAssumptionCache);
+  void invalidateFunction(Function *F);
+
+};
+
 /// \brief Represents the cost of inlining a function.
 ///
 /// This supports special values for functions which should "always" or
 /// "never" be inlined. Otherwise, the cost represents a unitless amount;
 /// smaller values increase the likelihood of the function being inlined.
 ///
 /// Objects of this type also provide the adjusted threshold for inlining
 /// based on the information available for a particular callsite. They can be

 /// used to bound the computation necessary to determine whether the cost is
 /// sufficiently low to warrant inlining.
 ///
 /// Also note that calling this function *dynamically* computes the cost of
 /// inlining the callsite. It is an expensive, heavyweight call.
 InlineCost
 getInlineCost(CallSite CS, const InlineParams &Params,
               TargetTransformInfo &CalleeTTI,
+              TargetLibraryInfo &TLI,
               std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
+              InliningLoopInfoCache *ILIC,
               ProfileSummaryInfo *PSI);
 
 /// \brief Get an InlineCost with the callee explicitly specified.
 /// This allows you to calculate the cost of inlining a function via a
 /// pointer. This behaves exactly as the version with no explicit callee
 /// parameter in all other respects.
 //
 InlineCost
 getInlineCost(CallSite CS, Function *Callee, const InlineParams &Params,
               TargetTransformInfo &CalleeTTI,
+              TargetLibraryInfo &TLI,
               std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
+              InliningLoopInfoCache *ILIC,
               ProfileSummaryInfo *PSI);
 
 /// \brief Minimal filter to detect invalid constructs for inlining.
 bool isInlineViable(Function &Callee);
 }
 
 #endif

include/llvm/Transforms/IPO/InlinerPass.h

 
 private:
   // Insert @llvm.lifetime intrinsics.
   bool InsertLifetime;
 
 protected:
   AssumptionCacheTracker *ACT;
   ProfileSummaryInfo *PSI;
+  InliningLoopInfoCache *ILIC;
   ImportedFunctionsInliningStatistics ImportedFunctionsStats;
 };
 
 } // End llvm namespace
 
 #endif

lib/Analysis/InlineCost.cpp

 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AssumptionCache.h"
 #include "llvm/Analysis/CodeMetrics.h"
 #include "llvm/Analysis/ConstantFolding.h"
 #include "llvm/Analysis/InstructionSimplify.h"
+#include "llvm/Analysis/LoopInfo.h"
 #include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/Analysis/TargetLibraryInfo.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
+#include "llvm/Analysis/ScalarEvolution.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/IR/CallSite.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/DataLayout.h"
+#include "llvm/IR/Dominators.h"
 #include "llvm/IR/GetElementPtrTypeIterator.h"
 #include "llvm/IR/GlobalAlias.h"
 #include "llvm/IR/InstVisitor.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Operator.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/raw_ostream.h"
+#include "llvm/Transforms/Utils/ValueMapper.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "inline-cost"
 
 STATISTIC(NumCallsAnalyzed, "Number of call sites analyzed");
 
 static cl::opt<int> InlineThreshold(

 
 class CallAnalyzer : public InstVisitor<CallAnalyzer, bool> {
   typedef InstVisitor<CallAnalyzer, bool> Base;
   friend class InstVisitor<CallAnalyzer, bool>;
 
   /// The TargetTransformInfo available for this compilation.
   const TargetTransformInfo &TTI;
 
+  TargetLibraryInfo &TLI;
+
+  InliningLoopInfoCache* ILIC;
+
   /// Getter for the cache of @llvm.assume intrinsics.
   std::function<AssumptionCache &(Function &)> &GetAssumptionCache;
 
   /// Profile summary information.
   ProfileSummaryInfo *PSI;
 
   /// The called function.
   Function &F;
class CallAnalyzer : public InstVisitor<CallAnalyzer, bool> {
   bool visitIndirectBrInst(IndirectBrInst &IBI);
   bool visitResumeInst(ResumeInst &RI);
   bool visitCleanupReturnInst(CleanupReturnInst &RI);
   bool visitCatchReturnInst(CatchReturnInst &RI);
   bool visitUnreachableInst(UnreachableInst &I);
 
 public:
   CallAnalyzer(const TargetTransformInfo &TTI,
+               TargetLibraryInfo &TLI, InliningLoopInfoCache *ILIC,
                std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
                ProfileSummaryInfo *PSI, Function &Callee, CallSite CSArg,
                const InlineParams &Params)
-      : TTI(TTI), GetAssumptionCache(GetAssumptionCache), PSI(PSI), F(Callee),
-        CandidateCS(CSArg), Params(Params), Threshold(Params.DefaultThreshold),
+      : TTI(TTI), TLI(TLI), ILIC(ILIC), GetAssumptionCache(GetAssumptionCache),
+        PSI(PSI), F(Callee), CandidateCS(CSArg), Params(Params),
+        Threshold(Params.DefaultThreshold),
         Cost(0), IsCallerRecursive(false), IsRecursiveCall(false),
         ExposesReturnsTwice(false), HasDynamicAlloca(false),
         ContainsNoDuplicateCall(false), HasReturn(false), HasIndirectBr(false),
         HasFrameEscape(false), AllocatedSize(0), NumInstructions(0),
         NumVectorInstructions(0), FiftyPercentVectorBonus(0),
         TenPercentVectorBonus(0), VectorBonus(0), NumConstantArgs(0),
         NumConstantOffsetPtrArgs(0), NumAllocaArgs(0), NumConstantPtrCmps(0),
         NumConstantPtrDiffs(0), NumInstructionsSimplified(0),
         SROACostSavings(0), SROACostSavingsLost(0) {}
 
   bool analyzeCall(CallSite CS);
-
+  bool propagatesConstantInnerLoopTripCount(CallSite &CS);
   int getThreshold() { return Threshold; }
   int getCost() { return Cost; }
 
   // Keep a bunch of stats about the cost savings found so we can print them
   // out when debugging.
   unsigned NumConstantArgs;
   unsigned NumConstantOffsetPtrArgs;
   unsigned NumAllocaArgs;
bool CallAnalyzer::visitCallSite(CallSite CS) {
 
   // If we have a constant that we are calling as a function, we can peer
   // through it and see the function target. This happens not infrequently
   // during devirtualization and so we want to give it a hefty bonus for
   // inlining, but cap that bonus in the event that inlining wouldn't pan
   // out. Pretend to inline the function, with a custom threshold.
   auto IndirectCallParams = Params;
   IndirectCallParams.DefaultThreshold = InlineConstants::IndirectCallThreshold;
-  CallAnalyzer CA(TTI, GetAssumptionCache, PSI, *F, CS, IndirectCallParams);
+  CallAnalyzer CA(TTI, TLI, ILIC, GetAssumptionCache, PSI, *F, CS,
+    IndirectCallParams);
   if (CA.analyzeCall(CS)) {
     // We were able to inline the indirect call! Subtract the cost from the
     // threshold to get the bonus we want to apply, but don't go below zero.
     Cost -= std::max(0, CA.getThreshold() - CA.getCost());
   }
 
   return Base::visitCallSite(CS);
 }
do {
     }
     assert(V->getType()->isPointerTy() && "Unexpected operand type!");
   } while (Visited.insert(V).second);
 
   Type *IntPtrTy = DL.getIntPtrType(V->getContext());
   return cast<ConstantInt>(ConstantInt::get(IntPtrTy, Offset));
 }
 
+/// \brief Accessor functions for the InliningLoopCacheInfo
+///
+/// Note the dependence: ScalarEvolution -> LoopInfo -> DominatorTree
+/// All dependent parts are built on demand if needed.
+///
+
+DominatorTree* InliningLoopInfoCache::getDT(Function* F) {
+  auto It = DTMapSCC.find(F);
+  if (It != DTMapSCC.end())
+    return It->second; 
+  auto ret = new DominatorTree(*F);
+  DTMapSCC.insert(std::make_pair(F, ret));
+  return ret; 
+}
+
+LoopInfo* InliningLoopInfoCache::getLI(Function* F) {
+  auto It = LIMapSCC.find(F);
+  if (It != LIMapSCC.end())
+    return It->second; 
+  DominatorTree* DT = getDT(F);
+  assert(DT != nullptr); 
+  auto ret = new LoopInfo(*DT); 
+  LIMapSCC.insert(std::make_pair(F, ret)); 
+  return ret; 
+} 
+  
+ScalarEvolution* InliningLoopInfoCache::getSE(Function* F,
+  TargetLibraryInfo& TLI,
+  std::function<AssumptionCache &(Function &)> &GetAssumptionCache) {
+  auto It = SEMapSCC.find(F); 
+  if (It != SEMapSCC.end()) 
+    return It->second; 
+  DominatorTree* DT = getDT(F);
+  assert(DT != nullptr);
+  LoopInfo* LI = getLI(F); 
+  assert(LI != nullptr);
+  AssumptionCache AC = GetAssumptionCache(*F);
+  ScalarEvolution* ret = new ScalarEvolution(*F, TLI, AC, *DT, *LI);
+  SEMapSCC.insert(std::make_pair(F, ret)); 
+  return ret; 
+} 
+
+void InliningLoopInfoCache::invalidateFunction(Function* F) {
+  auto DTit = DTMapSCC.find(F);
+  if (DTit != DTMapSCC.end()) { 
+    delete DTit->second; 
+    DTMapSCC.erase(DTit); 
+  }
+  auto LIit = LIMapSCC.find(F);
+  if (LIit != LIMapSCC.end()) { 
+    delete LIit->second; 
+    LIMapSCC.erase(LIit); 
+  } 
+  auto SEit = SEMapSCC.find(F);
+  if (SEit != SEMapSCC.end()) { 
+    delete SEit->second; 
+    SEMapSCC.erase(SEit); 
+  } 
+} 
+
+InliningLoopInfoCache::~InliningLoopInfoCache() {
+  for (auto &DTI: DTMapSCC)
+    delete DTI.second;
+  DTMapSCC.clear();
+  for (auto &LTI: LIMapSCC)
+    delete LTI.second;
+  LIMapSCC.clear();
+  for (auto &STI: SEMapSCC)
+    delete STI.second;
+  SEMapSCC.clear();
+}
+
+/// \brief Heuristic for preferring inlining that exposes complete unrolling
+///
+/// Return true if inlining CS will propagate a constant into the called
+/// function F, which then lets us determine that an inner loop in F is
+/// a small constant (and hence a good candidate for unrolling).
+///
+bool CallAnalyzer::propagatesConstantInnerLoopTripCount(CallSite &CS) {
+  Function *F = CS.getCalledFunction();
+  auto PI = F->arg_begin(), PE = F->arg_end();
+  auto AI = CS.arg_begin(), AE = CS.arg_end();
+
+  for (; AI != AE; ++AI) {
+    Value *Actual = *AI;
+    if (PI == PE)
+      break;
+    ConstantInt *UB = dyn_cast<ConstantInt>(Actual);
+    if (UB == nullptr) {
+      PI++;
+      continue;
+    }
+    Argument *Formal = &*PI++;
+    for (Use &U : Formal->uses()) {
+       auto ICmp = dyn_cast<ICmpInst>(U.getUser());
+       if (ICmp == nullptr)
+         continue;
+       BasicBlock *LatchBlock = ICmp->getParent();
+       Loop *L = ILIC->getLI(F)->getLoopFor(LatchBlock);
+       if (L == nullptr || !L->empty() || L->getLoopLatch() != LatchBlock)
+         continue;
+       for (Use &UU : ICmp->uses()) {
+         if (UU.getUser() != LatchBlock->getTerminator())
+           continue;
+         ScalarEvolution* SE = ILIC->getSE(F, TLI, GetAssumptionCache);
+         const SCEV *FBackCount = SE->getBackedgeTakenCount(L);
+         ValueToValueMap FormalToConstantActualParameterMap;
+         FormalToConstantActualParameterMap.insert(
+           std::make_pair(Formal, Actual));
+         const SCEV *ABackCount
+           = SCEVParameterRewriter::rewrite(FBackCount, *SE, 
+           FormalToConstantActualParameterMap, true);
+         const SCEVConstant *ABackCountConst
+           = dyn_cast<SCEVConstant>(ABackCount);
+         if (ABackCountConst == nullptr)
+           continue;
+         uint64_t BackCount = ABackCountConst->getValue()->getZExtValue();
+         if (BackCount >= InlineConstants::SmallTripCountMax)
+           continue;
+         return true;
+       }
+    }
+  }
+  return false;
+}
+
 /// \brief Analyze a call site for potential inlining.
 ///
 /// Returns true if inlining this call is viable, and false if it is not
 /// viable. It computes the cost and adjusts the threshold based on numerous
 /// factors and heuristics. If this method returns false but the computed cost
 /// is below the computed threshold, then inlining was forcibly disabled by
 /// some artifact of the routine.
 bool CallAnalyzer::analyzeCall(CallSite CS) {
bool CallAnalyzer::analyzeCall(CallSite CS) {
 
   // If there is only one call of the function, and it has internal linkage,
   // the cost of inlining it drops dramatically.
   bool OnlyOneCallAndLocalLinkage =
       F.hasLocalLinkage() && F.hasOneUse() && &F == CS.getCalledFunction();
   if (OnlyOneCallAndLocalLinkage)
     Cost -= InlineConstants::LastCallToStaticBonus;
 
+  // If a function is preferred for inlining, and may expose an unrolling
+  // opportunity, drop the cost somewhat.
+  bool StaticConstantTripCount
+    = (F.hasLocalLinkage() || F.hasLinkOnceODRLinkage()) &&
+    (&F == CS.getCalledFunction()) && F.hasFnAttribute(Attribute::InlineHint) &&
+    propagatesConstantInnerLoopTripCount(CS);
+  if (StaticConstantTripCount)
+    Cost -= InlineConstants::StaticSmallConstantTripCountBonus;
+
   // If this function uses the coldcc calling convention, prefer not to inline
   // it.
   if (F.getCallingConv() == CallingConv::Cold)
     Cost += InlineConstants::ColdccPenalty;
 
   // Check if we're done. This can happen due to bonuses and penalties.
   if (Cost > Threshold)
     return false;

 static bool functionsHaveCompatibleAttributes(Function *Caller,
                                               Function *Callee,
                                               TargetTransformInfo &TTI) {
   return TTI.areInlineCompatible(Caller, Callee) &&
          AttributeFuncs::areInlineCompatible(*Caller, *Callee);
 }
 
 InlineCost llvm::getInlineCost(
-    CallSite CS, const InlineParams &Params, TargetTransformInfo &CalleeTTI,
+    CallSite CS, const InlineParams &Params,
+    TargetTransformInfo &CalleeTTI, TargetLibraryInfo &TLI,
     std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
+    InliningLoopInfoCache *ILIC,
     ProfileSummaryInfo *PSI) {
   return getInlineCost(CS, CS.getCalledFunction(), Params, CalleeTTI,
-                       GetAssumptionCache, PSI);
+                       TLI, GetAssumptionCache, ILIC, PSI);
 }
 
 InlineCost llvm::getInlineCost(
     CallSite CS, Function *Callee, const InlineParams &Params,
     TargetTransformInfo &CalleeTTI,
+    TargetLibraryInfo &TLI,
     std::function<AssumptionCache &(Function &)> &GetAssumptionCache,
+    InliningLoopInfoCache *ILIC,
     ProfileSummaryInfo *PSI) {
 
   // Cannot inline indirect calls.
   if (!Callee)
     return llvm::InlineCost::getNever();
 
   // Calls to functions with always-inline attributes should be inlined
   // whenever possible.
InlineCost llvm::getInlineCost(
   // we have *a* correct implementation of the source level function.
   if (Callee->isInterposable() || Callee->hasFnAttribute(Attribute::NoInline) ||
       CS.isNoInline())
     return llvm::InlineCost::getNever();
 
   DEBUG(llvm::dbgs() << "      Analyzing call of " << Callee->getName()
                      << "...\n");
 
-  CallAnalyzer CA(CalleeTTI, GetAssumptionCache, PSI, *Callee, CS, Params);
+  CallAnalyzer CA(CalleeTTI, TLI, ILIC, GetAssumptionCache, PSI, *Callee,
+    CS, Params);
   bool ShouldInline = CA.analyzeCall(CS);
 
   DEBUG(CA.dump());
 
   // Check if there was a reason to force inlining or no inlining.
   if (!ShouldInline && CA.getCost() < CA.getThreshold())
     return InlineCost::getNever();
   if (ShouldInline && CA.getCost() >= CA.getThreshold())

lib/Transforms/IPO/InlineSimple.cpp

explicit SimpleInliner(InlineParams Params) : Inliner(ID), Params(Params) {
     initializeSimpleInlinerPass(*PassRegistry::getPassRegistry());
   }
 
   static char ID; // Pass identification, replacement for typeid
 
   InlineCost getInlineCost(CallSite CS) override {
     Function *Callee = CS.getCalledFunction();
     TargetTransformInfo &TTI = TTIWP->getTTI(*Callee);
+    TargetLibraryInfo &TLI = TLIWP->getTLI();
     std::function<AssumptionCache &(Function &)> GetAssumptionCache =
         [&](Function &F) -> AssumptionCache & {
       return ACT->getAssumptionCache(F);
     };
-    return llvm::getInlineCost(CS, Params, TTI, GetAssumptionCache, PSI);
+    return llvm::getInlineCost(CS, Params, TTI, TLI, GetAssumptionCache,
+      ILIC, PSI);
   }
 
   bool runOnSCC(CallGraphSCC &SCC) override;
   void getAnalysisUsage(AnalysisUsage &AU) const override;
 
 private:
   TargetTransformInfoWrapperPass *TTIWP;
-
+  TargetLibraryInfoWrapperPass *TLIWP;
 };
 
 } // end anonymous namespace
 
 char SimpleInliner::ID = 0;
 INITIALIZE_PASS_BEGIN(SimpleInliner, "inline", "Function Integration/Inlining",
                       false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)

 }
 
 Pass *llvm::createFunctionInliningPass(InlineParams &Params) {
   return new SimpleInliner(Params);
 }
 
 bool SimpleInliner::runOnSCC(CallGraphSCC &SCC) {
   TTIWP = &getAnalysis<TargetTransformInfoWrapperPass>();
+  TLIWP = &getAnalysis<TargetLibraryInfoWrapperPass>();
   return Inliner::runOnSCC(SCC);
 }
 
 void SimpleInliner::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.addRequired<TargetTransformInfoWrapperPass>();
+  AU.addRequired<TargetLibraryInfoWrapperPass>();
   Inliner::getAnalysisUsage(AU);
 }

lib/Transforms/IPO/Inliner.cpp

if (skipSCC(SCC))
     return false;
   return inlineCalls(SCC);
 }
 
 static bool
 inlineCallsImpl(CallGraphSCC &SCC, CallGraph &CG,
                 std::function<AssumptionCache &(Function &)> GetAssumptionCache,
                 ProfileSummaryInfo *PSI, TargetLibraryInfo &TLI,
+                InliningLoopInfoCache *ILIC,
                 bool InsertLifetime,
                 function_ref<InlineCost(CallSite CS)> GetInlineCost,
                 function_ref<AAResults &(Function &)> AARGetter,
                 ImportedFunctionsInliningStatistics &ImportedFunctionsStats) {
   SmallPtrSet<Function *, 8> SCCFunctions;
   DEBUG(dbgs() << "Inliner visiting SCC:");
   for (CallGraphNode *Node : SCC) {
     Function *F = Node->getFunction();
for (unsigned CSi = 0; CSi != CallSites.size(); ++CSi) {
                                   ImportedFunctionsStats)) {
           ORE.emit(
               OptimizationRemarkMissed(DEBUG_TYPE, "NotInlined", DLoc, Block)
               << NV("Callee", Callee) << " will not be inlined into "
               << NV("Caller", Caller));
           continue;
         }
         ++NumInlined;
-
+        ILIC->invalidateFunction(Caller);
         // Report the inline decision.
         ORE.emit(OptimizationRemark(DEBUG_TYPE, "Inlined", DLoc, Block)
                  << NV("Callee", Callee) << " inlined into "
                  << NV("Caller", Caller));
 
         // If inlining this function gave us any new call sites, throw them
         // onto our worklist to process.  They are useful inline candidates.
         if (!InlineInfo.InlinedCalls.empty()) {
for (unsigned CSi = 0; CSi != CallSites.size(); ++CSi) {
         DEBUG(dbgs() << "    -> Deleting dead function: " << Callee->getName()
                      << "\n");
         CallGraphNode *CalleeNode = CG[Callee];
 
         // Remove any call graph edges from the callee to its callees.
         CalleeNode->removeAllCalledFunctions();
 
         // Removing the node for callee from the call graph and delete it.
+        ILIC->invalidateFunction(Callee);
         delete CG.removeFunctionFromModule(CalleeNode);
         ++NumDeleted;
       }
 
       // Remove this call site from the list.  If possible, use
       // swap/pop_back for efficiency, but do not use it if doing so would
       // move a call site to a function in this SCC before the
       // 'FirstCallInSCC' barrier.
bool Inliner::inlineCalls(CallGraphSCC &SCC) {
   auto AARGetter = [&](Function &F) -> AAResults & {
     BAR.emplace(createLegacyPMBasicAAResult(*this, F));
     AAR.emplace(createLegacyPMAAResults(*this, F, *BAR));
     return *AAR;
   };
   auto GetAssumptionCache = [&](Function &F) -> AssumptionCache & {
     return ACT->getAssumptionCache(F);
   };
-  return inlineCallsImpl(SCC, CG, GetAssumptionCache, PSI, TLI, InsertLifetime,
+  ILIC = new InliningLoopInfoCache();
+  bool rv = inlineCallsImpl(SCC, CG, GetAssumptionCache, PSI, TLI, ILIC,
+                         InsertLifetime,
                          [this](CallSite CS) { return getInlineCost(CS); },
                          AARGetter, ImportedFunctionsStats);
+  delete ILIC;
+  ILIC = nullptr;
+  return rv;
 }
 
 /// Remove now-dead linkonce functions at the end of
 /// processing to avoid breaking the SCC traversal.
 bool Inliner::doFinalization(CallGraph &CG) {
   if (InlinerFunctionImportStats != InlinerFunctionImportStatsOpts::No)
     ImportedFunctionsStats.dump(InlinerFunctionImportStats ==
                                 InlinerFunctionImportStatsOpts::Verbose);