[llvm-profgen][CSSPGO] Support count based aggregated type of hybrid perf script

[wlei]

Summary

This change tried to integrate a new count based aggregated type of perf script. The only difference of the format is that an aggregated count is added at the head of the original sample which means the same samples are repeated to the given count times. This is used to reduce the perf script size.
e.g.

2
	          4005dc
	          400634
	          400684
	    7f68c5788793
 0x4005c8/0x4005dc/P/-/-/0  ....

Implemented by a dedicated PerfReader AggregatedHybridPerfReader.

[wenlei]

llvm/tools/llvm-profgen/PerfReader.h

67

How about we make the aggregation form an "extension" applicable to any perf script type instead of a new type? This is because we could have a count for LBR perf script too (when we use llvm-profgen for line-based AutoFDO too in the future). If we make aggregation a separate type, we would potentially need to double the number of types..

[hoy]

llvm/test/tools/llvm-profgen/Inputs/noinline-cs-noprobe.aggperfscript

7

Can you remove the /P/-/-/0 part and see if the existing parser can handle both formats? That part is unnecessary.

llvm/tools/llvm-profgen/PerfReader.h

67

+1, this is a good idea.

[wlei]

Addressing feedbacks, change to not use a seperated class for the aggregated script

[wenlei]

Along the lines of supporting aggregation as an extension, perhaps we could handle parseAggregatedCount in the base class inside parseSample for anything other than stuff like PERF_RECORD_MMAP2. What do you think?

[wlei]

put parseAggregatedCount into base class

[wlei]

In D107192#2920779, @wenlei wrote:

Along the lines of supporting aggregation as an extension, perhaps we could handle parseAggregatedCount in the base class inside parseSample for anything other than stuff like PERF_RECORD_MMAP2. What do you think?

That's good point, so we can share this with the non-cs profile.

[wlei]

lint

[wenlei]

There seems to have some test failures related to unwinder checks. Otherwise lgtm, thanks.

llvm/tools/llvm-profgen/PerfReader.cpp

681

assert that Count >= 1 here?

[hoy]

Looks like the test failures are due to missing mmap events. Rebasing the current diff on top of previous patch should fix it.

[wlei]

add assertion

[wlei]

There seems to have some test failures related to unwinder checks. Otherwise lgtm, thanks.
Looks like the test failures are due to missing mmap events. Rebasing the current diff on top of previous patch should fix it.

Yeah, see the recent build, it passed. Seems because I didn't add the dependance of the two patches and the build system doesn't catch this automatically.

llvm/test/tools/llvm-profgen/Inputs/noinline-cs-noprobe.aggperfscript

+2
+            4005dc
+            400634
+            400684
+      7f68c5788793
+ 0x4005c8/0x4005dc 0x40062f/0x4005b0 0x400645/0x4005ff 0x400637/0x400645 0x4005e9/0x400634 0x4005d7/0x4005e5 0x40062f/0x4005b0 0x400645/0x4005ff 0x400637/0x400645 0x4005e9/0x400634 0x4005d7/0x4005e5 0x40062f/0x4005b0 0x400645/0x4005ff 0x400637/0x400645 0x4005e9/0x400634 0x4005c8/0x4005dc
+2
+            4005b0
+            400684
+      7f68c5788793
+ 0x40062f/0x4005b0 0x400645/0x4005ff 0x400637/0x400645 0x4005e9/0x400634 0x4005c8/0x4005dc 0x40062f/0x4005b0 0x400645/0x4005ff 0x400637/0x400645 0x4005e9/0x400634 0x4005d7/0x4005e5 0x40062f/0x4005b0 0x400645/0x4005ff 0x400637/0x400645 0x4005e9/0x400634 0x4005d7/0x4005e5 0x40062f/0x4005b0

llvm/test/tools/llvm-profgen/Inputs/noinline-cs-pseudoprobe.aggperfscript

+1
+            20179e
+            2017f9
+      7f83e84e7793
+  5541f689495641d7
+ 0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0
+1
+            2017c4
+            2017f9
+      7f83e84e7793
+  5541f689495641d7
+ 0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0
+1
+            2017c4
+            2017f9
+      7f83e84e7793
+  5541f689495641d7
+ 0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0  0x2017bf/0x201760/P/-/-/0  0x2017cf/0x20179e/P/-/-/0  0x20177f/0x2017c4/P/-/-/0

llvm/test/tools/llvm-profgen/noinline-cs-noprobe.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.perfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --show-unwinder-output --profile-summary-cold-count=0 | FileCheck %s --check-prefix=CHECK-UNWINDER
 ; RUN: FileCheck %s --input-file %t
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.aggperfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --show-unwinder-output --profile-summary-cold-count=0 | FileCheck %s --check-prefix=CHECK-AGG-UNWINDER
+; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-AGG
+
+; CHECK-AGG:[main:1 @ foo]:108:0
+; CHECK-AGG: 2: 6
+; CHECK-AGG: 3: 6 bar:6
+; CHECK-AGG:[main:1 @ foo:3 @ bar]:100:6
+; CHECK-AGG: 0: 6
+; CHECK-AGG: 1: 6
+; CHECK-AGG: 2: 4
+; CHECK-AGG: 4: 2
+; CHECK-AGG: 5: 6
+
+; CHECK-AGG-UNWINDER: Binary(noinline-cs-noprobe.perfbin)'s Range Counter:
+; CHECK-AGG-UNWINDER: main:1 @ foo
+; CHECK-AGG-UNWINDER:   (5ff, 62f): 6
+; CHECK-AGG-UNWINDER:   (634, 637): 6
+; CHECK-AGG-UNWINDER:   (645, 645): 6
+; CHECK-AGG-UNWINDER: main:1 @ foo:3 @ bar
+; CHECK-AGG-UNWINDER:   (5b0, 5c8): 2
+; CHECK-AGG-UNWINDER:   (5b0, 5d7): 4
+; CHECK-AGG-UNWINDER:   (5dc, 5e9): 2
+; CHECK-AGG-UNWINDER:   (5e5, 5e9): 4
+
+; CHECK-AGG-UNWINDER: Binary(noinline-cs-noprobe.perfbin)'s Branch Counter:
+; CHECK-AGG-UNWINDER: main:1 @ foo
+; CHECK-AGG-UNWINDER:   (62f, 5b0): 6
+; CHECK-AGG-UNWINDER:   (637, 645): 6
+; CHECK-AGG-UNWINDER:   (645, 5ff): 6
+; CHECK-AGG-UNWINDER: main:1 @ foo:3 @ bar
+; CHECK-AGG-UNWINDER:   (5c8, 5dc): 4
+; CHECK-AGG-UNWINDER:   (5d7, 5e5): 4
+; CHECK-AGG-UNWINDER:   (5e9, 634): 6
+
+
 
 ; CHECK:[main:1 @ foo]:54:0
 ; CHECK: 2: 3
 ; CHECK: 3: 3 bar:3
 ; CHECK:[main:1 @ foo:3 @ bar]:50:3
 ; CHECK: 0: 3
 ; CHECK: 1: 3
 ; CHECK: 2: 2

 ; CHECK-UNWINDER:   (5d7, 5e5): 2
 ; CHECK-UNWINDER:   (5e9, 634): 3
 
 
 
 
 
 
-
 ; original code:
 ; clang -O0 -g test.c -o a.out
 #include <stdio.h>
 
 int bar(int x, int y) {
   if (x % 3) {
     return x - y;
   }

llvm/test/tools/llvm-profgen/noinline-cs-pseudoprobe.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-pseudoprobe.perfscript --binary=%S/Inputs/noinline-cs-pseudoprobe.perfbin --output=%t --show-unwinder-output --profile-summary-cold-count=0 | FileCheck %s --check-prefix=CHECK-UNWINDER
 ; RUN: FileCheck %s --input-file %t
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-pseudoprobe.aggperfscript --binary=%S/Inputs/noinline-cs-pseudoprobe.perfbin --output=%t --show-unwinder-output --profile-summary-cold-count=0 | FileCheck %s --check-prefix=CHECK-UNWINDER
+; RUN: FileCheck %s --input-file %t
+
 
 ; CHECK:     [main:2 @ foo]:75:0
 ; CHECK-NEXT: 1: 0
 ; CHECK-NEXT: 2: 15
 ; CHECK-NEXT: 3: 15
 ; CHECK-NEXT: 4: 15
 ; CHECK-NEXT: 5: 0
 ; CHECK-NEXT: 6: 15

llvm/tools/llvm-profgen/PerfReader.h

 
 // The type of perfscript
 enum PerfScriptType {
   PERF_UNKNOWN = 0,
   PERF_INVALID = 1,
   PERF_LBR = 2,       // Only LBR sample
   PERF_LBR_STACK = 3, // Hybrid sample including call stack and LBR stack.
 };
 
 // The parsed LBR sample entry.
 struct LBREntry {
   uint64_t Source = 0;
   uint64_t Target = 0;
   // An artificial branch stands for a series of consecutive branches starting
   // from the current binary with a transition through external code and
   // eventually landing back in the current binary.
   bool IsArtificial = false;
protected:
   void parseEventOrSample(TraceStream &TraceIt);
   // Extract call stack from the perf trace lines
   bool extractCallstack(TraceStream &TraceIt,
                         SmallVectorImpl<uint64_t> &CallStack);
   // Extract LBR stack from one perf trace line
   bool extractLBRStack(TraceStream &TraceIt,
                        SmallVectorImpl<LBREntry> &LBRStack,
                        ProfiledBinary *Binary);
+  uint64_t parseAggregatedCount(TraceStream &TraceIt);
   // Parse one sample from multiple perf lines, override this for different
   // sample type
-  virtual void parseSample(TraceStream &TraceIt) = 0;
+  void parseSample(TraceStream &TraceIt);
+  // An aggregated count is given to indicate how many times the sample is
+  // repeated.
+  virtual void parseSample(TraceStream &TraceIt, uint64_t Count) = 0;
   // Post process the profile after trace aggregation, we will do simple range
   // overlap computation for AutoFDO, or unwind for CSSPGO(hybrid sample).
   virtual void generateRawProfile() = 0;
   // Helper function for looking up binary in AddressBinaryMap
   ProfiledBinary *getBinary(uint64_t Address);
 
   BinaryMap BinaryTable;
   AddressBinaryMap AddrToBinaryMap; // Used by address-based lookup.

 */
 class HybridPerfReader : public PerfReaderBase {
 public:
   HybridPerfReader(cl::list<std::string> &BinaryFilenames)
       : PerfReaderBase(BinaryFilenames) {
     PerfType = PERF_LBR_STACK;
   };
   // Parse the hybrid sample including the call and LBR line
-  void parseSample(TraceStream &TraceIt) override;
+  void parseSample(TraceStream &TraceIt, uint64_t Count) override;
   void generateRawProfile() override;
 
 private:
   // Unwind the hybrid samples after aggregration
   void unwindSamples();
   void printUnwinderOutput();
 };
 
 } // end namespace sampleprof
 } // end namespace llvm
 
 #endif

llvm/tools/llvm-profgen/PerfReader.cpp

bool PerfReaderBase::extractCallstack(TraceStream &TraceIt,
   // out to reduce the number of different calling contexts. One instance
   // of such case - when sample landed in prolog/epilog, somehow stack
   // walking will be broken in an unexpected way that higher frames will be
   // missing.
   return !CallStack.empty() &&
          !Binary->addressInPrologEpilog(CallStack.front());
 }
 
-void HybridPerfReader::parseSample(TraceStream &TraceIt) {
+void HybridPerfReader::parseSample(TraceStream &TraceIt, uint64_t Count) {
   // The raw hybird sample started with call stack in FILO order and followed
   // intermediately by LBR sample
   // e.g.
   //            4005dc    # call stack leaf
   //            400634
   //            400684    # call stack root
   // 0x4005c8/0x4005dc/P/-/-/0   0x40062f/0x4005b0/P/-/-/0 ...
   //          ... 0x4005c8/0x4005dc/P/-/-/0    # LBR Entries
void HybridPerfReader::parseSample(TraceStream &TraceIt, uint64_t Count) {
   if (!TraceIt.isAtEoF() && TraceIt.getCurrentLine().startswith(" 0x")) {
     // Parsing LBR stack and populate into HybridSample.LBRStack
     if (extractLBRStack(TraceIt, Sample->LBRStack, Sample->Binary)) {
       // Canonicalize stack leaf to avoid 'random' IP from leaf frame skew LBR
       // ranges
       Sample->CallStack.front() = Sample->LBRStack[0].Target;
       // Record samples by aggregation
       Sample->genHashCode();
-      AggregatedSamples[Hashable<PerfSample>(Sample)]++;
+      AggregatedSamples[Hashable<PerfSample>(Sample)] += Count;
     }
   } else {
     // LBR sample is encoded in single line after stack sample
     exitWithError("'Hybrid perf sample is corrupted, No LBR sample line");
   }
 }
 
+uint64_t PerfReaderBase::parseAggregatedCount(TraceStream &TraceIt) {
+  // The aggregated count is optional, so do not skip the line and return 1 if
+  // it's unmatched
+  uint64_t Count = 1;
+  if (!TraceIt.getCurrentLine().getAsInteger(10, Count))
+    TraceIt.advance();
+  return Count;
+}
+
+void PerfReaderBase::parseSample(TraceStream &TraceIt) {
+  uint64_t Count = parseAggregatedCount(TraceIt);
+  assert(Count >= 1 && "Aggregated count should be >= 1!");
+  parseSample(TraceIt, Count);
+}
+
 void PerfReaderBase::parseMMap2Event(TraceStream &TraceIt) {
   // Parse a line like:
   //  PERF_RECORD_MMAP2 2113428/2113428: [0x7fd4efb57000(0x204000) @ 0
   //  08:04 19532229 3585508847]: r-xp /usr/lib64/libdl-2.17.so
   constexpr static const char *const Pattern =
       "PERF_RECORD_MMAP2 ([0-9]+)/[0-9]+: "
       "\\[(0x[a-f0-9]+)\\((0x[a-f0-9]+)\\) @ "
       "(0x[a-f0-9]+|0) .*\\]: [-a-z]+ (.*)";