[llvm-profgen] Fill zero count for all function ranges

[wlei]

Summary

Allow filling zero count for all the function ranges even there is no samples hitting that function. Add a switch for this.

[wlei]

Updating D112858: [llvm-profgen] Add a switch to fill zero count for all function ranges

[wenlei]

lgtm, thanks!

[hoy]

llvm/test/tools/llvm-profgen/inline-noprobe.test

7

Does it work when given an empty raw profile? Can it also work without giving a profile input? That can be a real scenario.

[modimo]

llvm/test/tools/llvm-profgen/inline-noprobe.test

7

No profile here would be great!

[wenlei]

llvm/test/tools/llvm-profgen/inline-noprobe.test

7

I think asking user to provide an (empty) profile is reasonable.. Otherwise it feels a bit stretched for llvm-profgen as a profile generation tool.

[wlei]

fix typo and test

llvm/test/tools/llvm-profgen/inline-noprobe.test

7

Currently we only have PerfScriptReader and UnsymbolizedProfileReader which both need an input. To support no input profile, we might need lots of if (FillZeroForAllFuncs) else ... in validateCommandLine, PerfReader or an UnknownFileReader. Those seem make the code a little messy.

Our internal tools also require an input which can be empty(two zero on the top). What do you think?

[modimo]

llvm/test/tools/llvm-profgen/inline-noprobe.test

7

Having an empty raw profile is enough, do you mind adding a test case for it?

[wlei]

llvm/test/tools/llvm-profgen/inline-noprobe.test

7

Good point, changed the test case with only two zero input.

[wlei]

add empty file test case

[wlei]

update test case

[hoy]

Lgtm, thanks.

llvm/test/tools/llvm-profgen/inline-noprobe.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-noprobe.perfscript --binary=%S/Inputs/inline-noprobe.perfbin --output=%t --skip-symbolization
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE
 ; RUN: llvm-profgen --format=text --use-dwarf-correlation --perfscript=%S/Inputs/inline-noprobe.perfscript --binary=%S/Inputs/inline-noprobe.perfbin --output=%t --skip-symbolization
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-RAW-PROFILE
 ; RUN: llvm-profgen --format=text --unsymbolized-profile=%t --binary=%S/Inputs/inline-noprobe.perfbin --output=%t1
 ; RUN: FileCheck %s --input-file %t1 --check-prefix=CHECK
 ; RUN: llvm-profgen --format=text --use-dwarf-correlation --perfscript=%S/Inputs/inline-noprobe.perfscript --binary=%S/Inputs/inline-noprobe.perfbin --output=%t
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK
+; RUN: echo -e "0\n0" > %t
+; RUN: llvm-profgen --format=text --unsymbolized-profile=%t --binary=%S/Inputs/inline-noprobe.perfbin --output=%t1 --fill-zero-for-all-funcs
+; RUN: FileCheck %s --input-file %t1 --check-prefix=CHECK-ALL-ZERO
 
 CHECK: main:188:0
 CHECK:  0: 0
 CHECK:  2: 0
 CHECK:  1: foo:188
 CHECK:   2.1: 42
 CHECK:   3: 62
 CHECK:   3.2: 21
 CHECK:   4: 0
 CHECK:   3.1: bar:42
 CHECK:    1: 42
 CHECK:   3.2: bar:21
 CHECK:    1: 21
 
+CHECK-ALL-ZERO: bar:0:0
+CHECK-ALL-ZERO:  1: 0
+CHECK-ALL-ZERO:  5: 0
+CHECK-ALL-ZERO: foo:0:0
+CHECK-ALL-ZERO:  0: 0
+CHECK-ALL-ZERO:  2.1: 0
+CHECK-ALL-ZERO:  3: 0
+CHECK-ALL-ZERO:  3.2: 0
+CHECK-ALL-ZERO:  4: 0
+CHECK-ALL-ZERO:  3.1: bar:0
+CHECK-ALL-ZERO:   1: 0
+CHECK-ALL-ZERO:  3.2: bar:0
+CHECK-ALL-ZERO:   1: 0
+CHECK-ALL-ZERO:   7: 0
+CHECK-ALL-ZERO: main:0:0
+CHECK-ALL-ZERO:  0: 0
+CHECK-ALL-ZERO:  2: 0
+CHECK-ALL-ZERO:  1: foo:0
+CHECK-ALL-ZERO:   2.1: 0
+CHECK-ALL-ZERO:   3: 0
+CHECK-ALL-ZERO:   3.2: 0
+CHECK-ALL-ZERO:   4: 0
+CHECK-ALL-ZERO:   3.1: bar:0
+CHECK-ALL-ZERO:    1: 0
+CHECK-ALL-ZERO:   3.2: bar:0
+CHECK-ALL-ZERO:    1: 0
+
 CHECK-RAW-PROFILE:      3
 CHECK-RAW-PROFILE-NEXT: 650-691:21
 CHECK-RAW-PROFILE-NEXT: 669-677:20
 CHECK-RAW-PROFILE-NEXT: 669-691:21
 CHECK-RAW-PROFILE-NEXT: 2
 CHECK-RAW-PROFILE-NEXT: 677->650:21
 CHECK-RAW-PROFILE-NEXT: 691->669:43
 

llvm/tools/llvm-profgen/ProfileGenerator.cpp

cl::opt<bool> UseMD5(
     "use-md5", cl::init(false), cl::Hidden,
     cl::desc("Use md5 to represent function names in the output profile (only "
              "meaningful for -extbinary)"));
 
 static cl::opt<bool> PopulateProfileSymbolList(
     "populate-profile-symbol-list", cl::init(false), cl::Hidden,
     cl::desc("Populate profile symbol list (only meaningful for -extbinary)"));
 
+static cl::opt<bool> FillZeroForAllFuncs(
+    "fill-zero-for-all-funcs", cl::init(false), cl::Hidden,
+    cl::desc("Attribute all functions' range with zero count "
+             "even it's not hit by any samples."));
+
 static cl::opt<int32_t, true> RecursionCompression(
     "compress-recursion",
     cl::desc("Compressing recursion by deduplicating adjacent frame "
              "sequences up to the specified size. -1 means no size limit."),
     cl::Hidden,
     cl::location(llvm::sampleprof::CSProfileGenerator::MaxCompressionSize));
 
 static cl::opt<bool> CSProfMergeColdContext(
FunctionSamples &ProfileGenerator::getLeafFrameProfile(
   }
 
   return *FunctionProfile;
 }
 
 RangeSample
 ProfileGenerator::preprocessRangeCounter(const RangeSample &RangeCounter) {
   RangeSample Ranges(RangeCounter.begin(), RangeCounter.end());
-  // For each range, we search for all ranges of the function it belongs to and
-  // initialize it with zero count, so it remains zero if doesn't hit any
+  if (FillZeroForAllFuncs) {
+    for (auto &FuncI : Binary->getAllBinaryFunctions()) {
+      for (auto &R : FuncI.second.Ranges) {
+        Ranges[{R.first, R.second}] += 0;
+      }
+    }
+  } else {
+    // For each range, we search for all ranges of the function it belongs to
+    // and initialize it with zero count, so it remains zero if doesn't hit any
     // samples. This is to be consistent with compiler that interpret zero count
     // as unexecuted(cold).
     for (auto I : RangeCounter) {
       uint64_t StartOffset = I.first.first;
       for (const auto &Range : Binary->getRangesForOffset(StartOffset))
         Ranges[{Range.first, Range.second - 1}] += 0;
     }
+  }
   RangeSample DisjointRanges;
   findDisjointRanges(DisjointRanges, Ranges);
   return DisjointRanges;
 }
 
 void ProfileGenerator::populateBodySamplesForAllFunctions(
     const RangeSample &RangeCounter) {
   for (auto Range : preprocessRangeCounter(RangeCounter)) {
void CSProfileGenerator::postProcessProfiles() {
   if (EnableCSPreInliner) {
     CSPreInliner(ProfileMap, *Binary, HotCountThreshold, ColdCountThreshold)
         .run();
     // Turn off the profile merger by default unless it is explicitly enabled.
     if (!CSProfMergeColdContext.getNumOccurrences())
       CSProfMergeColdContext = false;
   }
 
-  // Trim and merge cold context profile using cold threshold above. 
+  // Trim and merge cold context profile using cold threshold above.
   if (CSProfTrimColdContext || CSProfMergeColdContext) {
     SampleContextTrimmer(ProfileMap)
         .trimAndMergeColdContextProfiles(
             HotCountThreshold, CSProfTrimColdContext, CSProfMergeColdContext,
             CSProfMaxColdContextDepth, EnableCSPreInliner);
   }
 }
 

llvm/tools/llvm-profgen/ProfiledBinary.h

RangesTy getRangesForOffset(uint64_t Offset) {
     auto *FRange = findFuncRangeForOffset(Offset);
     // Ignore the range which falls into plt section or system lib.
     if (!FRange)
       return RangesTy();
 
     return FRange->Func->Ranges;
   }
 
+  const std::unordered_map<std::string, BinaryFunction> &
+  getAllBinaryFunctions() {
+    return BinaryFunctions;
+  }
+
   uint32_t getFuncSizeForContext(SampleContext &Context) {
     return FuncSizeTracker.getFuncSizeForContext(Context);
   }
 
   // Load the symbols from debug table and populate into symbol list.
   void populateSymbolListFromDWARF(ProfileSymbolList &SymbolList);
 
   const SampleContextFrameVector &