Use call target count to derive the call instruction weight

[danielcdh]

Summary

The call target count profile is directly derived from LBR branch->target data. This is more reliable than instruction frequency profiles that could be moved across basic block boundaries. This patches uses call target count profile to annotate call instructions.

[dnovillo]

OK, so instead of looking at the number of samples collected at the call site, it looks at the number of calls the target had. But what if the target is called disproportionately more from one call site than the other?

Like:

if (this_is_true_95_percent_of_the_time())

A();

else

A();

The example is ridiculous, I know, but wouldn't we end up inlining A() in both branches of the if()? Is that a code growth issue in general?

[danielcdh]

In D24410#545823, @dnovillo wrote:

OK, so instead of looking at the number of samples collected at the call site, it looks at the number of calls the target had. But what if the target is called disproportionately more from one call site than the other?

Like:

if (this_is_true_95_percent_of_the_time())

A();

else

A();

The example is ridiculous, I know, but wouldn't we end up inlining A() in both branches of the if()? Is that a code growth issue in general?

No because we use lineno+discriminator to distinguish a callsite, so we can precisely locate both callsites to A and only inline the hot one.

[danielcdh]

format

include/llvm/ProfileData/SampleProf.h

ErrorOr<uint64_t> findSamplesAt(uint32_t LineOffset,
                                   uint32_t Discriminator) const {
     const auto &ret = BodySamples.find(LineLocation(LineOffset, Discriminator));
     if (ret == BodySamples.end())
       return std::error_code();
     else
       return ret->second.getSamples();
   }
 
+  /// Return the total number of call target samples collected at a given
+  /// location. Each location is specified by \p LineOffset and
+  /// \p Discriminator. If the location is not found in profile, return error.
+  ErrorOr<uint64_t> findCallSamplesAt(uint32_t LineOffset,
+                                      uint32_t Discriminator) const {
+    const auto &ret = BodySamples.find(LineLocation(LineOffset, Discriminator));
+    if (ret == BodySamples.end())
+      return std::error_code();
+    uint64_t T = 0;
+    for (const auto &t_c : ret->second.getCallTargets()) {
+      T += t_c.second;
+    }
+    return T;
+  }
+
   /// Return the function samples at the given callsite location.
   FunctionSamples &functionSamplesAt(const LineLocation &Loc) {
     return CallsiteSamples[Loc];
   }
 
   /// Return a pointer to function samples at the given callsite location.
   const FunctionSamples *findFunctionSamplesAt(const LineLocation &Loc) const {
     auto iter = CallsiteSamples.find(Loc);

lib/Transforms/IPO/SampleProfile.cpp

if (IsCall && findCalleeFunctionSamples(Inst))
     return 0;
 
   const DILocation *DIL = DLoc;
   unsigned Lineno = DLoc.getLine();
   unsigned HeaderLineno = DIL->getScope()->getSubprogram()->getLine();
 
   uint32_t LineOffset = getOffset(Lineno, HeaderLineno);
   uint32_t Discriminator = DIL->getDiscriminator();
-  ErrorOr<uint64_t> R = FS->findSamplesAt(LineOffset, Discriminator);
+  ErrorOr<uint64_t> R = IsCall
+                            ? FS->findCallSamplesAt(LineOffset, Discriminator)
+                            : FS->findSamplesAt(LineOffset, Discriminator);
   if (R) {
     bool FirstMark =
         CoverageTracker.markSamplesUsed(FS, LineOffset, Discriminator, R.get());
     if (FirstMark) {
       const Function *F = Inst.getParent()->getParent();
       LLVMContext &Ctx = F->getContext();
       emitOptimizationRemark(
           Ctx, DEBUG_TYPE, *F, DLoc,
if (Coverage < SampleProfileSampleCoverage) {
           DS_Warning));
     }
   }
   return Changed;
 }
 
 char SampleProfileLoaderLegacyPass::ID = 0;
 INITIALIZE_PASS_BEGIN(SampleProfileLoaderLegacyPass, "sample-profile",
                       "Sample Profile loader", false, false)
 INITIALIZE_PASS_DEPENDENCY(AssumptionCacheTracker)
 INITIALIZE_PASS_END(SampleProfileLoaderLegacyPass, "sample-profile",
                     "Sample Profile loader", false, false)
 
 bool SampleProfileLoader::doInitialization(Module &M) {
   auto &Ctx = M.getContext();
   auto ReaderOrErr = SampleProfileReader::create(Filename, Ctx);
   if (std::error_code EC = ReaderOrErr.getError()) {
     std::string Msg = "Could not open profile: " + EC.message();
     Ctx.diagnose(DiagnosticInfoSampleProfile(Filename, Msg));
     return false;

test/Transforms/SampleProfile/Inputs/inline-coverage.prof

 main:501438:0
  2.1: 23478
  3: 23478
  4: 0
  0: 0
  3: _Z3fool:172746
-  1: 31878
+  1: 31878 rand:31878