[InlineCost] CallPenalty knob to provide custom values

[eastig]

Summary

At the moment CallPenalty is fixed to the value 25.
The patch introduces a knob '-call-penalty' to provide custom values to CallPenalty.

[efriedma]

include/llvm/Analysis/InlineCost.h

80

It seems kind of weird to return the CallPenalty here... but I guess you need it in lib/Transforms/IPO/Inliner.cpp?

92

Using INT_MIN and INT_MAX seems like it's just asking for trouble with overflow... but I guess we don't actually use it in that case? Seems better to just use 0.

lib/Analysis/InlineCost.cpp

1235

The call penalty should be the same for every callsite; we shouldn't need to update it here... unless you're planning to change that?

[eastig]

include/llvm/Analysis/InlineCost.h

80

Yes, this is because of Inliner.cpp.
Should CallPenalty be a part of InlineParams?

1. If no, then we know its value either default or cmd-option provided. In such case we can have a function accessing to CallPenalty defined in InlineCost.cpp.
2. If yes, its value can be whatever a caller of llvm::getInlineCost has been provided. In such case we need to save the value somewhere. I chose this option. As InlineCost object is only way of communication I stored used CallPenalty in it as it's done for Threshold.

What about having it in TTI?

92

I used zeros in the initial version. Then I thought if I could use some values which could protect from incorrect uses. Like -infinity penalty means no penalty and no inline. +infinity means a lot of penalty and always inline.
I have nothing against using zeros.

lib/Analysis/InlineCost.cpp

1235

As I wrote above if the penalty is a part of InlineParams it can be different per a call. So we need to update the current value based on the value passed through InlineParams.

[eraman]

lib/Transforms/IPO/Inliner.cpp

292

r286814 subtracts CallPenalty while computing the inline cost (which is returned by IC.getCost()), but this code is not updated to reflect that. I think CandidateCost should be IC.getCost() - 1, which will eliminate the need for IC.getCallPenalty().

[eastig]

lib/Transforms/IPO/Inliner.cpp

292

Thank you for information. It makes things simpler. I'll have a look at this.

include/llvm/Analysis/InlineCost.h

 const int OptMinSizeThreshold = 5;
 
 /// Use when -O3 is specified.
 const int OptAggressiveThreshold = 250;
 
 // Various magic constants used to adjust heuristics.
 const int InstrCost = 5;
 const int IndirectCallThreshold = 100;
-const int CallPenalty = 25;
 const int LastCallToStaticBonus = 15000;
 const int ColdccPenalty = 2000;
 const int NoreturnPenalty = 10000;
 /// Do not inline functions which allocate this many bytes on the stack
 /// when the caller is recursive.
 const unsigned TotalAllocaSizeRecursiveCaller = 1024;
 }
 
class InlineCost {
   };
 
   /// \brief The estimated cost of inlining this callsite.
   const int Cost;
 
   /// \brief The adjusted threshold against which this cost was computed.
   const int Threshold;
 
+  /// \brief The call penalty which was used to calculate the cost.
+  const int CallPenalty;
+
   // Trivial constructor, interesting logic in the factory functions below.
-  InlineCost(int Cost, int Threshold) : Cost(Cost), Threshold(Threshold) {}
+  InlineCost(int Cost, int Threshold, int CallPenalty)
+      : Cost(Cost), Threshold(Threshold), CallPenalty(CallPenalty) {}
 
 public:
-  static InlineCost get(int Cost, int Threshold) {
+  static InlineCost get(int Cost, int Threshold, int CallPenalty) {
     assert(Cost > AlwaysInlineCost && "Cost crosses sentinel value");
     assert(Cost < NeverInlineCost && "Cost crosses sentinel value");
-    return InlineCost(Cost, Threshold);
+    return InlineCost(Cost, Threshold, CallPenalty);
   }
   static InlineCost getAlways() {
-    return InlineCost(AlwaysInlineCost, 0);
+    return InlineCost(AlwaysInlineCost, 0, INT_MAX);
   }
   static InlineCost getNever() {
-    return InlineCost(NeverInlineCost, 0);
+    return InlineCost(NeverInlineCost, 0, INT_MIN);
   }
 
   /// \brief Test whether the inline cost is low enough for inlining.
   explicit operator bool() const {
     return Cost < Threshold;
   }
 
   bool isAlways() const { return Cost == AlwaysInlineCost; }
   bool isNever() const { return Cost == NeverInlineCost; }
   bool isVariable() const { return !isAlways() && !isNever(); }
 
   /// \brief Get the inline cost estimate.
   /// It is an error to call this on an "always" or "never" InlineCost.
   int getCost() const {
     assert(isVariable() && "Invalid access of InlineCost");
     return Cost;
   }
 
+  /// \brief Get the call penalty.
+  /// It is an error to call this on an "always" or "never" InlineCost.
+  int getCallPenalty() const {
+    assert(isVariable() && "Invalid access of InlineCost");
+    return CallPenalty;
+  }
+
   /// \brief Get the cost delta from the threshold for inlining.
   /// Only valid if the cost is of the variable kind. Returns a negative
   /// value if the cost is too high to inline.
   int getCostDelta() const { return Threshold - getCost(); }
 };
 
 /// Thresholds to tune inline cost analysis. The inline cost analysis decides
 /// the condition to apply a threshold and applies it. Otherwise,
struct InlineParams {
   /// Threshold to use when the caller is optimized for minsize.
   Optional<int> OptMinSizeThreshold;
 
   /// Threshold to use when the callsite is considered hot.
   Optional<int> HotCallSiteThreshold;
 
   /// Threshold to use when the callsite is considered cold.
   Optional<int> ColdCallSiteThreshold;
+
+  /// The value of penalty to use for call penalties.
+  Optional<int> CallPenalty;
 };
 
 /// Generate the parameters to tune the inline cost analysis based only on the
 /// commandline options.
 InlineParams getInlineParams();
 
 /// Generate the parameters to tune the inline cost analysis based on command
 /// line options. If -inline-threshold option is not explicitly passed,

lib/Analysis/InlineCost.cpp

static cl::opt<int> ColdThreshold(
     "inlinecold-threshold", cl::Hidden, cl::init(225),
     cl::desc("Threshold for inlining functions with cold attribute"));
 
 static cl::opt<int>
     HotCallSiteThreshold("hot-callsite-threshold", cl::Hidden, cl::init(3000),
                          cl::ZeroOrMore,
                          cl::desc("Threshold for hot callsites "));
 
+static cl::opt<int> CallPenalty("call-penalty", cl::Hidden, cl::init(25),
+                                cl::desc("The penalty of a function call"));
+
 namespace {
 
 class CallAnalyzer : public InstVisitor<CallAnalyzer, bool> {
   typedef InstVisitor<CallAnalyzer, bool> Base;
   friend class InstVisitor<CallAnalyzer, bool>;
 
   /// The TargetTransformInfo available for this compilation.
   const TargetTransformInfo &TTI;
class CallAnalyzer : public InstVisitor<CallAnalyzer, bool> {
   /// easily cacheable. Instead, use the cover function paramHasAttr.
   CallSite CandidateCS;
 
   /// Tunable parameters that control the analysis.
   const InlineParams &Params;
 
   int Threshold;
   int Cost;
+  int CallPenalty;
 
   bool IsCallerRecursive;
   bool IsRecursiveCall;
   bool ExposesReturnsTwice;
   bool HasDynamicAlloca;
   bool ContainsNoDuplicateCall;
   bool HasReturn;
   bool HasIndirectBr;
class CallAnalyzer : public InstVisitor<CallAnalyzer, bool> {
   bool isKnownNonNullInCallee(Value *V);
 
   /// Update Threshold based on callsite properties such as callee
   /// attributes and callee hotness for PGO builds. The Callee is explicitly
   /// passed to support analyzing indirect calls whose target is inferred by
   /// analysis.
   void updateThreshold(CallSite CS, Function &Callee);
 
+  /// Update the value of CallPenalty if its value is passed via Params
+  void updateCallPenaltyValue();
+
   /// Return true if size growth is allowed when inlining the callee at CS.
   bool allowSizeGrowth(CallSite CS);
 
   // Custom analysis routines.
   bool analyzeBlock(BasicBlock *BB, SmallPtrSetImpl<const Value *> &EphValues);
 
   // Disable several entry points to the visitor so we don't accidentally use
   // them by declaring but not defining them here.
CallAnalyzer(const TargetTransformInfo &TTI,
         NumConstantPtrCmps(0), NumConstantPtrDiffs(0),
         NumInstructionsSimplified(0), SROACostSavings(0),
         SROACostSavingsLost(0) {}
 
   bool analyzeCall(CallSite CS);
 
   int getThreshold() { return Threshold; }
   int getCost() { return Cost; }
+  int getCallPenalty() const { return CallPenalty; }
 
   // Keep a bunch of stats about the cost savings found so we can print them
   // out when debugging.
   unsigned NumConstantArgs;
   unsigned NumConstantOffsetPtrArgs;
   unsigned NumAllocaArgs;
   unsigned NumConstantPtrCmps;
   unsigned NumConstantPtrDiffs;
if (Function *F = CS.getCalledFunction()) {
     if (TTI.isLoweredToCall(F)) {
       // We account for the average 1 instruction per call argument setup
       // here.
       Cost += CS.arg_size() * InlineConstants::InstrCost;
 
       // Everything other than inline ASM will also have a significant cost
       // merely from making the call.
       if (!isa<InlineAsm>(CS.getCalledValue()))
-        Cost += InlineConstants::CallPenalty;
+        Cost += CallPenalty;
     }
 
     return Base::visitCallSite(CS);
   }
 
   // Otherwise we're in a very special case -- an indirect function call. See
   // if we can be particularly clever about this.
   Value *Callee = CS.getCalledValue();
if (I->getType()->isFloatingPointTy()) {
         Attribute Attr = F.getFnAttribute("use-soft-float");
         StringRef Val = Attr.getValueAsString();
         if (Val == "true")
           hasSoftFloatAttr = true;
       }
 
       if (TTI.getFPOpCost(I->getType()) == TargetTransformInfo::TCC_Expensive ||
           hasSoftFloatAttr)
-        Cost += InlineConstants::CallPenalty;
+        Cost += CallPenalty;
     }
 
     // If the instruction simplified to a constant, there is no cost to this
     // instruction. Visit the instructions using our InstVisitor to account for
     // all of the per-instruction logic. The visit tree returns true if we
     // consumed the instruction in any way, and false if the instruction's base
     // cost should count against inlining.
     if (Base::visit(&*I))
do {
     }
     assert(V->getType()->isPointerTy() && "Unexpected operand type!");
   } while (Visited.insert(V).second);
 
   Type *IntPtrTy = DL.getIntPtrType(V->getContext());
   return cast<ConstantInt>(ConstantInt::get(IntPtrTy, Offset));
 }
 
+/// \brief Update the value of CallPenalty if its value is passed via Params
+void CallAnalyzer::updateCallPenaltyValue() {
+  CallPenalty = (Params.CallPenalty) ? Params.CallPenalty.getValue()
+                                     : ::CallPenalty.getDefault().getValue();
+}
+
 /// \brief Analyze a call site for potential inlining.
 ///
 /// Returns true if inlining this call is viable, and false if it is not
 /// viable. It computes the cost and adjusts the threshold based on numerous
 /// factors and heuristics. If this method returns false but the computed cost
 /// is below the computed threshold, then inlining was forcibly disabled by
 /// some artifact of the routine.
 bool CallAnalyzer::analyzeCall(CallSite CS) {
bool CallAnalyzer::analyzeCall(CallSite CS) {
   // FIXME: It would be nice to remove all such bonuses. At least it would be
   // nice to base the bonus values on something more scientific.
   assert(NumInstructions == 0);
   assert(NumVectorInstructions == 0);
 
   // Update the threshold based on callsite properties
   updateThreshold(CS, F);
 
+  updateCallPenaltyValue();
+
   FiftyPercentVectorBonus = 3 * Threshold / 2;
   TenPercentVectorBonus = 3 * Threshold / 4;
   const DataLayout &DL = F.getParent()->getDataLayout();
 
   // Track whether the post-inlining function would have more than one basic
   // block. A single basic block is often intended for inlining. Balloon the
   // threshold by 50% until we pass the single-BB phase.
   bool SingleBB = true;
if (CS.isByValArgument(I)) {
       Cost -= 2 * NumStores * InlineConstants::InstrCost;
     } else {
       // For non-byval arguments subtract off one instruction per call
       // argument.
       Cost -= InlineConstants::InstrCost;
     }
   }
   // The call instruction also disappears after inlining.
-  Cost -= InlineConstants::InstrCost + InlineConstants::CallPenalty;
+  Cost -= InlineConstants::InstrCost + CallPenalty;
   
   // If there is only one call of the function, and it has internal linkage,
   // the cost of inlining it drops dramatically.
   bool OnlyOneCallAndLocalLinkage =
       F.hasLocalLinkage() && F.hasOneUse() && &F == CS.getCalledFunction();
   if (OnlyOneCallAndLocalLinkage)
     Cost -= InlineConstants::LastCallToStaticBonus;
 
InlineCost llvm::getInlineCost(
   DEBUG(CA.dump());
 
   // Check if there was a reason to force inlining or no inlining.
   if (!ShouldInline && CA.getCost() < CA.getThreshold())
     return InlineCost::getNever();
   if (ShouldInline && CA.getCost() >= CA.getThreshold())
     return InlineCost::getAlways();
 
-  return llvm::InlineCost::get(CA.getCost(), CA.getThreshold());
+  return llvm::InlineCost::get(CA.getCost(), CA.getThreshold(),
+                               CA.getCallPenalty());
 }
 
 bool llvm::isInlineViable(Function &F) {
   bool ReturnsTwice = F.hasFnAttribute(Attribute::ReturnsTwice);
   for (Function::iterator BI = F.begin(), BE = F.end(); BI != BE; ++BI) {
     // Disallow inlining of functions which contain indirect branches or
     // blockaddresses.
     if (isa<IndirectBrInst>(BI->getTerminator()) || BI->hasAddressTaken())
InlineParams llvm::getInlineParams(int Threshold) {
   Params.HintThreshold = HintThreshold;
 
   // Set the HotCallSiteThreshold knob from the -hot-callsite-threshold.
   Params.HotCallSiteThreshold = HotCallSiteThreshold;
 
   // Set the ColdCallSiteThreshold knob from the -inline-cold-callsite-threshold.
   Params.ColdCallSiteThreshold = ColdCallSiteThreshold;
 
+  // Set the CallPenalty knob from the -call-penalty.
+  Params.CallPenalty = CallPenalty;
+
   // Set the OptMinSizeThreshold and OptSizeThreshold params only if the
   // Set the OptMinSizeThreshold and OptSizeThreshold params only if the
   // -inlinehint-threshold commandline option is not explicitly given. If that
   // option is present, then its value applies even for callees with size and
   // minsize attributes.
   // If the -inline-threshold is not specified, set the ColdThreshold from the
   // -inlinecold-threshold even if it is not explicitly passed. If
   // -inline-threshold is specified, then -inlinecold-threshold needs to be

lib/Transforms/IPO/Inliner.cpp

shouldBeDeferred(Function *Caller, CallSite CS, InlineCost IC,
   // decisions. Importantly the linkonce-ODR linkage covers inline functions
   // and templates in C++.
   //
   // FIXME: All of this logic should be sunk into getInlineCost. It relies on
   // the internal implementation of the inline cost metrics rather than
   // treating them as truly abstract units etc.
   TotalSecondaryCost = 0;
   // The candidate cost to be imposed upon the current function.
-  int CandidateCost = IC.getCost() - (InlineConstants::CallPenalty + 1);
+  int CandidateCost = IC.getCost() - (IC.getCallPenalty() + 1);
   // This bool tracks what happens if we do NOT inline C into B.
   bool callerWillBeRemoved = Caller->hasLocalLinkage();
   // This bool tracks what happens if we DO inline C into B.
   bool inliningPreventsSomeOuterInline = false;
   for (User *U : Caller->users()) {
     CallSite CS2(U);
 
     // If this isn't a call to Caller (it could be some other sort

test/Transforms/Inline/call-penalty-knob.ll

+; Check that calls are not inlined if the call penalty is low. The value of the
+; call penalty is provided with the '-call-penalty' option.
+;
+; RUN: opt < %s -inline -call-penalty=0 -inline-threshold=5 -S | FileCheck %s
+; RUN: opt < %s -inline -inline-threshold=5 -S | FileCheck %s -check-prefix=DEFAULT_CALL_PENALTY
+
+define i32 @X9(i32 %x) nounwind {
+  %x2 = add i32 %x, %x
+  %x3 = add i32 %x2, %x
+  %x4 = add i32 %x3, %x
+  %x5 = add i32 %x4, %x
+  %x6 = add i32 %x5, %x
+  %x7 = add i32 %x6, %x
+  %x8 = add i32 %x7, %x
+  %x9 = add i32 %x8, %x
+
+  ret i32 %x9
+}
+
+define i32 @f1(i32 %x) nounwind {
+  %res = call i32 @X9(i32 %x)
+  ret i32 %res
+; CHECK-LABEL: @f1(
+; CHECK: %res = call i32 @X9
+
+; DEFAULT_CALL_PENALTY-LABEL: @f1(
+; DEFAULT_CALL_PENALTY-NOT: call
+}
+