[CSSPGO][Preinliner] Always inline zero-sized functions.

[hoy]

Summary

Zero-sized functions should be cost-free in term of size budget, so they should be considered during inlining even if we run out of size budget.

This appears to give 0.5% win for one of our internal services.

[wenlei]

Can this be achieved by simply tweaking ProfiledCandidateComparer so zero SizeCost candidate always comes before non-zero ones?

[hoy]

In D153820#4453775, @wenlei wrote:

Can this be achieved by simply tweaking ProfiledCandidateComparer so zero SizeCost candidate always comes before non-zero ones?

Good idea!

[hoy]

Updating D153820: [CSSPGO][Preinliner] Always inline zero-sized functions.

[wenlei]

lgtm, thanks.

llvm/tools/llvm-profgen/CSPreInliner.h

47–48 ↗

(On Diff #535189)

nit: merge the two ifs into a single line?

if ((LHS.SizeCost == 0 || RHS.SizeCost == 0) && LHS.SizeCost != RHS.SizeCost)
    return return RHS.SizeCost == 0;

[hoy]

Updating D153820: [CSSPGO][Preinliner] Always inline zero-sized functions.

llvm/tools/llvm-profgen/CSPreInliner.cpp

void CSPreInliner::processFunction(const StringRef Name) {
   SizeLimit = std::max(SizeLimit, (unsigned)ProfileInlineLimitMin);
 
   LLVM_DEBUG(dbgs() << "Process " << Name
                     << " for context-sensitive pre-inlining (pre-inline size: "
                     << FuncSize << ", size limit: " << SizeLimit << ")\n");
 
   ProfiledCandidateQueue CQueue;
   getInlineCandidates(CQueue, FSamples);
+  std::vector<ProfiledInlineCandidate> IngoredDuetoSizeLimit;
 
-  while (!CQueue.empty() && FuncFinalSize < SizeLimit) {
+  while (!CQueue.empty()) {
     ProfiledInlineCandidate Candidate = CQueue.top();
     CQueue.pop();
-    bool ShouldInline = false;
-    if ((ShouldInline = shouldInline(Candidate))) {
+    // Always continue processing zero-sized functions even if the size budget
+    // is hit. This could happen when all of the callee's code is gone and only
+    // pseudo probes are left.
+    bool ShouldInline = (!Candidate.SizeCost ||
+                         Candidate.SizeCost + FuncFinalSize < SizeLimit) &&
+                        shouldInline(Candidate);
+
+    if (ShouldInline) {
       // We mark context as inlined as the corresponding context profile
       // won't be merged into that function's base profile.
       ++PreInlNumCSInlined;
       ContextTracker.markContextSamplesInlined(Candidate.CalleeSamples);
       Candidate.CalleeSamples->getContext().setAttribute(
           ContextShouldBeInlined);
       FuncFinalSize += Candidate.SizeCost;
       getInlineCandidates(CQueue, Candidate.CalleeSamples);
     } else {
       ++PreInlNumCSNotInlined;
+      if (Candidate.SizeCost + FuncFinalSize >= SizeLimit)
+        IngoredDuetoSizeLimit.push_back(Candidate);
     }
     LLVM_DEBUG(
         dbgs() << (ShouldInline ? "  Inlined" : "  Outlined")
                << " context profile for: "
                << ContextTracker.getContextString(*Candidate.CalleeSamples)
                << " (callee size: " << Candidate.SizeCost
                << ", call count:" << Candidate.CallsiteCount << ")\n");
   }
 
-  if (!CQueue.empty()) {
+  if (!IngoredDuetoSizeLimit.empty()) {
     if (SizeLimit == (unsigned)ProfileInlineLimitMax)
       ++PreInlNumCSInlinedHitMaxLimit;
     else if (SizeLimit == (unsigned)ProfileInlineLimitMin)
       ++PreInlNumCSInlinedHitMinLimit;
     else
       ++PreInlNumCSInlinedHitGrowthLimit;
   }
 
   LLVM_DEBUG({
-    if (!CQueue.empty())
+    if (!IngoredDuetoSizeLimit.empty())
       dbgs() << "  Inline candidates ignored due to size limit (inliner "
                 "original size: "
              << FuncSize << ", inliner final size: " << FuncFinalSize
              << ", size limit: " << SizeLimit << ")\n";
 
-    while (!CQueue.empty()) {
+    while (!IngoredDuetoSizeLimit.empty()) {
       ProfiledInlineCandidate Candidate = CQueue.top();
       CQueue.pop();
       bool WasInlined =
           Candidate.CalleeSamples->getContext().hasAttribute(ContextWasInlined);
       dbgs() << "    "
              << ContextTracker.getContextString(*Candidate.CalleeSamples)
              << " (candidate size:" << Candidate.SizeCost
              << ", call count: " << Candidate.CallsiteCount << ", previously "