Adding InlineCostAnnotationPrinterPass for Inline Cost Analysis

[knaumov]

Summary

For testing purposes, it is more efficient to use a pass which performs the Inline Cost Analysis and displays all the information about instructions. The functionality of the flag "-print-instruction-deltas" (which is renamed to "-print-inline-cost-instruction-annotations" for clarity) remains.
We need this pass because several tests, existing and upcoming, will require this functionality, but for now, it is limited only to debug builds.

[knaumov]

• Refactoring change

[mtrofin]

Could you split this into 2 changes:

• one that enables printing per-instruction inline cost analysis grouped together (i.e. the reason InstructionCostDetailMap is introduced)
• the other that introduces the new pass.

llvm/lib/Analysis/InlineCost.cpp

59

I think 'Annotate' refers to adding metadata, is that the intent? Should the rewording happen when that's implemented, rather (for clarity)

409

Consider InlineCostCallAnalyzer *const ICCA, to force the initialization of the pointer in the ctor. This helps maintainability - we won't risk ICCA bein uninitialized; it also captures design intent - ICCA, once set, isn't re-set (IIUC).

413

drop virtual, add override, so the compiler would check for you that you're actually overriding the base class' emitInstructionAnnot, not just adding another virtual (easier to maintain code - if later there's a refactoring in the base class, for instance, it'd be caught)

741

the move of the code in the file seems unrelated to this change - I'd propose doing it in a separate patch instead (if it's important to do).

2524

This won't give you the Params used for inlining, it'll give you ad default Params, is that intended?

[knaumov]

Addressed @mtrofin 's comments:

• • Before the change, we had CostAnnotationWriter that was used just to display the cost difference per instruction, hence the CostThresholdMap (which now became InstructionCostDetailMap) was in the printer itself. Now, the change changed that by putting the map into InlineCostCallAnalyzer to keep all the data in this class.
• The move of code was unnecessary, changed it

[knaumov]

Ping

[mtrofin]

In D81016#2086070, @knaumov wrote:

Ping

Oh - sorry, I saw you made changes for the comments on lines 420, 424 and 742, but didn't see anything on 59 and 2525, so I assumed you were looking to address those. Also, would it make sense to split off the pass in its own change (the other comment).

Could you also mark 'done' the addressed comments (it's easier to keep track for anyone looking at the CL)

Thanks!

[knaumov]

All of the preparational patches were approved, but there was an issue with integration which is being resolved in D82205

[knaumov]

The change landed: https://github.com/llvm/llvm-project/commit/6a5d7d498c0b16b13ace802f422b223eb510c303

llvm/include/llvm/Analysis/InlineCost.h

Optional<int> getInliningCostEstimate(
     CallBase &Call, TargetTransformInfo &CalleeTTI,
     function_ref<AssumptionCache &(Function &)> GetAssumptionCache,
     function_ref<BlockFrequencyInfo &(Function &)> GetBFI = nullptr,
     ProfileSummaryInfo *PSI = nullptr,
     OptimizationRemarkEmitter *ORE = nullptr);
 
 /// Minimal filter to detect invalid constructs for inlining.
 InlineResult isInlineViable(Function &Callee);
+
+// This pass is used to annotate instructions during the inline process for
+// debugging and analysis
+struct InlineCostAnnotationPrinterPass
+    : PassInfoMixin<InlineCostAnnotationPrinterPass> {
+   raw_ostream &OS;
+
+  public:
+    explicit InlineCostAnnotationPrinterPass(raw_ostream &OS) : OS(OS) {}
+  PreservedAnalyses run(Function &F, FunctionAnalysisManager &FAM);
+};
 } // namespace llvm
 
 #endif

llvm/lib/Analysis/InlineCost.cpp

 
 STATISTIC(NumCallsAnalyzed, "Number of call sites analyzed");
 
 static cl::opt<int>
     DefaultThreshold("inlinedefault-threshold", cl::Hidden, cl::init(225),
                      cl::ZeroOrMore,
                      cl::desc("Default amount of inlining to perform"));
 
-static cl::opt<bool> PrintDebugInstructionDeltas(
-    "print-instruction-deltas", cl::Hidden, cl::init(false),
-    cl::desc("Prints deltas of cost and threshold per instruction"));
+static cl::opt<bool> PrintInstructionAnnotations(
+    "print-inline-cost-instruction-annotations", cl::Hidden, cl::init(false),
+    cl::desc("Annotate instructions based on inline analysis"));
 
 static cl::opt<int> InlineThreshold(
     "inline-threshold", cl::Hidden, cl::init(225), cl::ZeroOrMore,
     cl::desc("Control the amount of inlining to perform (default = 225)"));
 
 static cl::opt<int> HintThreshold(
     "inlinehint-threshold", cl::Hidden, cl::init(325), cl::ZeroOrMore,
     cl::desc("Threshold for inlining functions with inline hint"));
struct InstructionCostDetail {
 
   int getThresholdDelta() const { return ThresholdAfter - ThresholdBefore; }
 
   int getCostDelta() const { return CostAfter - CostBefore; }
 
   bool hasThresholdChanged() const { return ThresholdAfter != ThresholdBefore; }
 };
 
-class CostAnnotationWriter : public AssemblyAnnotationWriter {
-public:
-  // This DenseMap stores the delta change in cost and threshold after
-  // accounting for the given instruction.
-  DenseMap<const Instruction *, InstructionCostDetail> CostThresholdMap;
-
-  virtual void emitInstructionAnnot(const Instruction *I,
-                                    formatted_raw_ostream &OS);
-};
-
 /// Carry out call site analysis, in order to evaluate inlinability.
 /// NOTE: the type is currently used as implementation detail of functions such
 /// as llvm::getInlineCost. Note the function_ref constructor parameters - the
 /// expectation is that they come from the outer scope, from the wrapper
 /// functions. If we want to support constructing CallAnalyzer objects where
 /// lambdas are provided inline at construction, or where the object needs to
 /// otherwise survive past the scope of the provided functions, we need to
 /// revisit the argument types.
public:
   unsigned NumAllocaArgs = 0;
   unsigned NumConstantPtrCmps = 0;
   unsigned NumConstantPtrDiffs = 0;
   unsigned NumInstructionsSimplified = 0;
 
   void dump();
 };
 
+class InlineCostAnnotationWriter : public AssemblyAnnotationWriter {
+private:
+  InlineCostCallAnalyzer *ICCA;
+
+public:
+  InlineCostAnnotationWriter(InlineCostCallAnalyzer *ICCA) : ICCA(ICCA) {}
+  virtual void emitInstructionAnnot(const Instruction *I,
+                                    formatted_raw_ostream &OS);
+};
+
 /// FIXME: if it is necessary to derive from InlineCostCallAnalyzer, note
 /// the FIXME in onLoweredCall, when instantiating an InlineCostCallAnalyzer
 class InlineCostCallAnalyzer final : public CallAnalyzer {
   const int CostUpperBound = INT_MAX - InlineConstants::InstrCost - 1;
   const bool ComputeFullInlineCost;
   int LoadEliminationCost = 0;
   /// Bonus to be applied when percentage of vector instructions in callee is
   /// high (see more details in updateThreshold).
   int VectorBonus = 0;
   /// Bonus to be applied when the callee has only one reachable basic block.
   int SingleBBBonus = 0;
 
   /// Tunable parameters that control the analysis.
   const InlineParams &Params;
 
+  // This DenseMap stores the delta change in cost and threshold after
+  // accounting for the given instruction. The map is filled only with the
+  // flag PrintInstructionAnnotations on.
+  DenseMap<const Instruction *, InstructionCostDetail> InstructionCostDetailMap;
+
   /// Upper bound for the inlining cost. Bonuses are being applied to account
   /// for speculative "expected profit" of the inlining decision.
   int Threshold = 0;
 
   /// Attempt to evaluate indirect calls to boost its inline cost.
   const bool BoostIndirectCalls;
 
   /// Ignore the threshold when finalizing analysis.
if (SingleBB && TI->getNumSuccessors() > 1) {
       Threshold -= SingleBBBonus;
       SingleBB = false;
     }
   }
 
   void onInstructionAnalysisStart(const Instruction *I) override {
     // This function is called to store the initial cost of inlining before
     // the given instruction was assessed.
-    if (!PrintDebugInstructionDeltas)
+    if (!PrintInstructionAnnotations)
       return;
-    Writer.CostThresholdMap[I].CostBefore = Cost;
-    Writer.CostThresholdMap[I].ThresholdBefore = Threshold;
+    InstructionCostDetailMap[I].CostBefore = Cost;
+    InstructionCostDetailMap[I].ThresholdBefore = Threshold;
   }
 
   void onInstructionAnalysisFinish(const Instruction *I) override {
     // This function is called to find new values of cost and threshold after
     // the instruction has been assessed.
-    if (!PrintDebugInstructionDeltas)
+    if (!PrintInstructionAnnotations)
       return;
-    Writer.CostThresholdMap[I].CostAfter = Cost;
-    Writer.CostThresholdMap[I].ThresholdAfter = Threshold;
+    InstructionCostDetailMap[I].CostAfter = Cost;
+    InstructionCostDetailMap[I].ThresholdAfter = Threshold;
   }
 
   InlineResult finalizeAnalysis() override {
     // Loops generally act a lot like calls in that they act like barriers to
     // movement, require a certain amount of setup, etc. So when optimising for
     // size, we penalise any call sites that perform loops. We do this after all
     // other costs here, so will likely only be dealing with relatively small
     // functions (and hence DT and LI will hopefully be cheap).
InlineCostCallAnalyzer(
       function_ref<BlockFrequencyInfo &(Function &)> GetBFI = nullptr,
       ProfileSummaryInfo *PSI = nullptr,
       OptimizationRemarkEmitter *ORE = nullptr, bool BoostIndirect = true,
       bool IgnoreThreshold = false)
       : CallAnalyzer(Callee, Call, TTI, GetAssumptionCache, GetBFI, PSI, ORE),
         ComputeFullInlineCost(OptComputeFullInlineCost ||
                               Params.ComputeFullInlineCost || ORE),
         Params(Params), Threshold(Params.DefaultThreshold),
-        BoostIndirectCalls(BoostIndirect), IgnoreThreshold(IgnoreThreshold) {}
+        BoostIndirectCalls(BoostIndirect), IgnoreThreshold(IgnoreThreshold),
+        ICAWriter(this) {}
 
-  /// Annotation Writer for cost annotation
-  CostAnnotationWriter Writer;
+  /// Annotation Writer for instruction details
+  InlineCostAnnotationWriter ICAWriter;
 
   void dump();
 
+  Optional<InstructionCostDetail> getCostDetails(const Instruction *I) {
+    if (InstructionCostDetailMap.find(I) != InstructionCostDetailMap.end())
+      return InstructionCostDetailMap[I];
+    return None;
+  }
+
   virtual ~InlineCostCallAnalyzer() {}
   int getThreshold() { return Threshold; }
   int getCost() { return Cost; }
 };
 } // namespace
 
+void InlineCostAnnotationWriter::emitInstructionAnnot(
+    const Instruction *I, formatted_raw_ostream &OS) {
+  // The cost of inlining of the given instruction is printed always.
+  // The threshold delta is printed only when it is non-zero. It happens
+  // when we decided to give a bonus at a particular instruction.
+  Optional<InstructionCostDetail> Record = ICCA->getCostDetails(I);
+  if (!Record)
+    OS << "; No cost analysis for the instruction";
+  else {
+    OS << "; cost before = " << Record->CostBefore
+       << ", cost after = " << Record->CostAfter
+       << ", threshold before = " << Record->ThresholdBefore
+       << ", threshold after = " << Record->ThresholdAfter << ", ";
+    OS << "cost delta = " << Record->getCostDelta();
+    if (Record->hasThresholdChanged())
+      OS << ", threshold delta = " << Record->getThresholdDelta();
+  }
+  OS << "\n";
+}
+
 /// Test whether the given value is an Alloca-derived function argument.
 bool CallAnalyzer::isAllocaDerivedArg(Value *V) {
   return SROAArgValues.count(V);
 }
 
 void CallAnalyzer::disableSROAForArg(AllocaInst *SROAArg) {
   onDisableSROA(SROAArg);
   EnabledSROAAllocas.erase(SROAArg);
   disableLoadElimination();
 }
 
-void CostAnnotationWriter::emitInstructionAnnot(const Instruction *I,
-                                                formatted_raw_ostream &OS) {
-  // The cost of inlining of the given instruction is printed always.
-  // The threshold delta is printed only when it is non-zero. It happens
-  // when we decided to give a bonus at a particular instruction.
-  if (CostThresholdMap.count(I) == 0) {
-    OS << "; No analysis for the instruction\n";
-    return;
-  }
-  const auto &Record = CostThresholdMap[I];
-  OS << "; cost before = " << Record.CostBefore
-     << ", cost after = " << Record.CostAfter
-     << ", threshold before = " << Record.ThresholdBefore
-     << ", threshold after = " << Record.ThresholdAfter << ", ";
-  OS << "cost delta = " << Record.getCostDelta();
-  if (Record.hasThresholdChanged())
-    OS << ", threshold delta = " << Record.getThresholdDelta();
-  OS << "\n";
-}
-
 /// If 'V' maps to a SROA candidate, disable SROA for it.
 void CallAnalyzer::disableSROA(Value *V) {
   if (auto *SROAArg = getSROAArgForValueOrNull(V)) {
     disableSROAForArg(SROAArg);
   }
 }
 
 void CallAnalyzer::disableLoadElimination() {
InlineResult CallAnalyzer::analyze() {
 
   return finalizeAnalysis();
 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 /// Dump stats about this call's analysis.
 LLVM_DUMP_METHOD void InlineCostCallAnalyzer::dump() {
 #define DEBUG_PRINT_STAT(x) dbgs() << "      " #x ": " << x << "\n"
-  if (PrintDebugInstructionDeltas)
-    F.print(dbgs(), &Writer);
+  if (PrintInstructionAnnotations)
+    F.print(dbgs(), &ICAWriter);
   DEBUG_PRINT_STAT(NumConstantArgs);
   DEBUG_PRINT_STAT(NumConstantOffsetPtrArgs);
   DEBUG_PRINT_STAT(NumAllocaArgs);
   DEBUG_PRINT_STAT(NumConstantPtrCmps);
   DEBUG_PRINT_STAT(NumConstantPtrDiffs);
   DEBUG_PRINT_STAT(NumInstructionsSimplified);
   DEBUG_PRINT_STAT(NumInstructions);
   DEBUG_PRINT_STAT(SROACostSavings);
auto Params =
       getInlineParams(computeThresholdFromOptLevels(OptLevel, SizeOptLevel));
   // At O3, use the value of -locally-hot-callsite-threshold option to populate
   // Params.LocallyHotCallSiteThreshold. Below O3, this flag has effect only
   // when it is specified explicitly.
   if (OptLevel > 2)
     Params.LocallyHotCallSiteThreshold = LocallyHotCallSiteThreshold;
   return Params;
 }
+
+PreservedAnalyses
+InlineCostAnnotationPrinterPass::run(Function &F,
+                                     FunctionAnalysisManager &FAM) {
+  PrintInstructionAnnotations = true;
+  std::function<AssumptionCache &(Function &)> GetAssumptionCache = [&](
+      Function &F) -> AssumptionCache & {
+    return FAM.getResult<AssumptionAnalysis>(F);
+  };
+  function_ref<BlockFrequencyInfo &(Function &)> GetBFI = [&](
+      Function &F) -> BlockFrequencyInfo & {
+    return FAM.getResult<BlockFrequencyAnalysis>(F);
+  };
+  Module *M = F.getParent();
+  ProfileSummaryInfo PSI(*M);
+  DataLayout DL(M);
+  TargetTransformInfo TTI(DL);
+  const InlineParams Params = llvm::getInlineParams();
+  for (BasicBlock &BB : F) {
+    for (Instruction &I : BB) {
+      if (CallInst *CI = dyn_cast<CallInst>(&I)) {
+        Function *CalledFunction = CI->getCalledFunction();
+        if (!CalledFunction || CalledFunction->isDeclaration())
+          continue;
+        OptimizationRemarkEmitter ORE(CalledFunction);
+        InlineCostCallAnalyzer ICCA(*CalledFunction, *CI, Params, TTI,
+                                    GetAssumptionCache, nullptr, &PSI, &ORE);
+        ICCA.analyze();
+        OS << "      Analyzing call of " << CalledFunction->getName()
+           << "... (caller:" << CI->getCaller()->getName() << ")\n";
+        ICCA.dump();
+      }
+    }
+  }
+  return PreservedAnalyses::all();
+}

llvm/lib/Passes/PassRegistry.def

 FUNCTION_PASS("print<assumptions>", AssumptionPrinterPass(dbgs()))
 FUNCTION_PASS("print<block-freq>", BlockFrequencyPrinterPass(dbgs()))
 FUNCTION_PASS("print<branch-prob>", BranchProbabilityPrinterPass(dbgs()))
 FUNCTION_PASS("print<da>", DependenceAnalysisPrinterPass(dbgs()))
 FUNCTION_PASS("print<domtree>", DominatorTreePrinterPass(dbgs()))
 FUNCTION_PASS("print<postdomtree>", PostDominatorTreePrinterPass(dbgs()))
 FUNCTION_PASS("print<demanded-bits>", DemandedBitsPrinterPass(dbgs()))
 FUNCTION_PASS("print<domfrontier>", DominanceFrontierPrinterPass(dbgs()))
+FUNCTION_PASS("print<inline-cost>", InlineCostAnnotationPrinterPass(dbgs()))
 FUNCTION_PASS("print<loops>", LoopPrinterPass(dbgs()))
 FUNCTION_PASS("print<memoryssa>", MemorySSAPrinterPass(dbgs()))
 FUNCTION_PASS("print<phi-values>", PhiValuesPrinterPass(dbgs()))
 FUNCTION_PASS("print<regions>", RegionInfoPrinterPass(dbgs()))
 FUNCTION_PASS("print<scalar-evolution>", ScalarEvolutionPrinterPass(dbgs()))
 FUNCTION_PASS("print<stack-safety-local>", StackSafetyPrinterPass(dbgs()))
 FUNCTION_PASS("reassociate", ReassociatePass())
 FUNCTION_PASS("scalarizer", ScalarizerPass())

llvm/test/Transforms/Inline/debuginline-cost-delta.ll

-; Require asserts for -debug-only
-; REQUIRES: asserts
-
-; RUN: opt < %s -inline -debug-only=inline-cost -disable-output -print-instruction-deltas 2>&1 | FileCheck %s
+; RUN: opt < %s -passes="print<inline-cost>" 2>&1 | FileCheck %s
 
 ; CHECK:       Analyzing call of callee1... (caller:foo)
-; CHECK: define i32 @callee1(i32 %x) {
-; CHECK: ; cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = 5
-; CHECK:   %x1 = add i32 %x, 1
-; CHECK: ; cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = 5
-; CHECK:   %x2 = add i32 %x1, 1
-; CHECK: ; cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = 5
-; CHECK:   %x3 = add i32 %x2, 1
-; CHECK: ; cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = 0
-; CHECK:   ret i32 %x3
-; CHECK: }
-; CHECK:      NumConstantArgs: 0
-; CHECK:      NumConstantOffsetPtrArgs: 0
-; CHECK:      NumAllocaArgs: 0
-; CHECK:      NumConstantPtrCmps: 0
-; CHECK:      NumConstantPtrDiffs: 0
-; CHECK:      NumInstructionsSimplified: 1
-; CHECK:      NumInstructions: 4
-; CHECK:      SROACostSavings: 0
-; CHECK:      SROACostSavingsLost: 0
-; CHECK:      LoadEliminationCost: 0
-; CHECK:      ContainsNoDuplicateCall: 0
-; CHECK:      Cost: {{.*}}
-; CHECK:      Threshold: {{.*}}
+; CHECK-NEXT: define i32 @callee1(i32 %x) {
+; CHECK-NEXT: cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = {{.*}}
+; CHECK-NEXT:   %x1 = add i32 %x, 1
+; CHECK-NEXT: cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = {{.*}}
+; CHECK-NEXT:   %x2 = add i32 %x1, 1
+; CHECK-NEXT: cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = {{.*}}
+; CHECK-NEXT:   %x3 = add i32 %x2, 1
+; CHECK-NEXT: cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = {{.*}}
+; CHECK-NEXT:   ret i32 %x3
+; CHECK-NEXT: }
 
 define i32 @foo(i32 %y) {
   %x = call i32 @callee1(i32 %y)
   ret i32 %x
 }
 
 define i32 @callee1(i32 %x) {
   %x1 = add i32 %x, 1
   %x2 = add i32 %x1, 1
   %x3 = add i32 %x2, 1
   ret i32 %x3
 }

llvm/test/Transforms/Inline/inline-cost-annotation-pass.ll

+; RUN: opt < %s -passes="print<inline-cost>" 2>&1 | FileCheck %s
+
+; CHECK:       Analyzing call of foo... (caller:main)
+; CHECK: define i8 addrspace(1)** @foo() {
+; CHECK:  cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = {{.*}}
+; CHECK:  %1 = inttoptr i64 754974720 to i8 addrspace(1)**
+; CHECK:  cost before = {{.*}}, cost after = {{.*}}, threshold before = {{.*}}, threshold after = {{.*}}, cost delta = {{.*}}
+; CHECK:  ret i8 addrspace(1)** %1
+; CHECK: }
+; CHECK:       NumConstantArgs: {{.*}}
+; CHECK:       NumConstantOffsetPtrArgs: {{.*}}
+; CHECK:       NumAllocaArgs: {{.*}}
+; CHECK:       NumConstantPtrCmps: {{.*}}
+; CHECK:       NumConstantPtrDiffs: {{.*}}
+; CHECK:       NumInstructionsSimplified: {{.*}}
+; CHECK:       NumInstructions: {{.*}}
+; CHECK:       SROACostSavings: {{.*}}
+; CHECK:       SROACostSavingsLost: {{.*}}
+; CHECK:       LoadEliminationCost: {{.*}}
+; CHECK:       ContainsNoDuplicateCall: {{.*}}
+; CHECK:       Cost: {{.*}}
+; CHECK:       Threshold: {{.*}}
+
+define i8 addrspace(1)** @foo() {
+  %1 = inttoptr i64 754974720 to i8 addrspace(1)**
+  ret i8 addrspace(1)** %1
+}
+
+define i8 addrspace(1)** @main() {
+  %1 = call i8 addrspace(1)** @foo()
+  ret i8 addrspace(1)** %1
+}

llvm/test/Transforms/Inline/print-instructions-deltas-unfinished.ll

-; Require asserts for -debug-only
-; REQUIRES: asserts
-
-; This test ensures that the hadling of instructions which were not analyzed by
-; '-print-instruction-deltas' flag due to the early exit was done correctly.
-
-; RUN: opt < %s -inline -debug-only=inline-cost -disable-output -print-instruction-deltas -inline-threshold=0 2>&1 | FileCheck %s
-
-; CHECK: No analysis for the instruction
-; CHECK:   ret void
-
-declare void @callee1()
-
-define void @bar() {
-  call void @callee1()
-  ret void
-}
-
-define void @foo() {
-  call void @bar()
-  ret void
-}