[llvm-profgen] Strip context to support non-CS profile generation for hybrid sample

[wlei]

Summary

The patch introduces strip-context mode for generating non-CS profiles for hybrid samples. After perf reader of hybrid sample, it strips the call stack(context) sample. With this, we can generate both CS and non-CS profiles using the same perf script input.

[wlei]

rebase

[wenlei]

llvm/tools/llvm-profgen/PerfReader.cpp

25

Use ignore-stack-samples?

680–687

assert SampleCounters.size is zero then?

785

Where do we check PerfType later? If PerfType is meant to be input type, can we not set it to PERF_LBR since input type which is indeed hybrid?

If we meant to use PerfType to indicate "processing mode", perhaps we should then set it to PERF_LBR earlier in extractPerfType, then create LBRPerfReader and teach it to ignore stack samples. That also avoids changing to have HybridPerfReader derive from LBRPerfReader..

Change the type half way seems inconsistent.

[wlei]

addressing feedbacks

refactor to move the generateRawProfile code out of parsePerfTraces and introduce a new flag(ProfileIsCS) for the profile generation creator

llvm/tools/llvm-profgen/PerfReader.cpp

25

Sounds good!

680–687

added!

785

Agree with you except I still prefer LBRPerfReader only responsible for PERF_LBR and HybridPerfReader only for PERF_LBR_STACK otherwise we need to maintain two part of code for the hybrid sample parsing.

On second thought, I think we already know whether it's a CS/Non-CS [raw] profile in PerfReader, so we can move the genRawProfiles logic out of parsePerfTraces. In parsePerfTraces , we only do populate perfSample. Then for genRawProfiles based on the perf type or ignore-stack-samples , it can decide to generate a CS raw profile or a non-CS raw profile. With this, we now pass a new flag ProfileIsCS to the generator instead of the perf_type.

Updated the code here, see if this looks good to you?

[wenlei]

llvm/tools/llvm-profgen/PerfReader.cpp

785

I still prefer LBRPerfReader only responsible for PERF_LBR and HybridPerfReader only for PERF_LBR_STACK otherwise we need to maintain two part of code for the hybrid sample parsing.

Yeah, we don't want to duplicate the code for sure.

I'm not sure about the refactoring to move unwind etc all into the base class though. I feel the unwind and hybrid sample parsing belong to HybridPerfReader, otherwise why do we need a HybridPerfReader..

Can we call LBRPerfReader::generateRawProfile() here in HybridPerfReader::generateRawProfile() just like what you had before, but without setting PerfType to PERF_LBR?

[hoy]

llvm/tools/llvm-profgen/PerfReader.cpp

26

nit: sample -> samples

785

Can we call LBRPerfReader::generateRawProfile() here in HybridPerfReader::generateRawProfile() just like what you had before, but without setting PerfType to PERF_LBR?

This sounds good to me. We still need to parse stack samples even with IgnoreStackSamples , but we don't need to unwind them.

[wlei]

Updating D109769: [llvm-profgen] Strip context to support non-CS profile generation for hybrid sample

llvm/tools/llvm-profgen/PerfReader.cpp

26

fixed!

785

Sounds good to make unwinder into HybridPerfReader .

[wenlei]

llvm/tools/llvm-profgen/PerfReader.cpp

784

nit: make sure we update ProfileIsCS properly for both cases and not relying on current value.

ProfileIsCS = !IgnoreStackSamples;
if (ProfileIsCS)
  unwindSamples();
else
  LBRPerfReader::generateRawProfile();

llvm/tools/llvm-profgen/PerfReader.h

614

This is now defined both in LBRPerfReader and PerfReaderBase? Is that intended? And it's not virtual, and yet declared as protected..

llvm/tools/llvm-profgen/llvm-profgen.cpp

92

Can we revert this change too and move generateRawProfile back into parsePerfTraces?

[wlei]

llvm/tools/llvm-profgen/PerfReader.cpp

784

Good idea!

llvm/tools/llvm-profgen/PerfReader.h

614

removed

llvm/tools/llvm-profgen/llvm-profgen.cpp

92

fixed!

[wlei]

Updating D109769: [llvm-profgen] Strip context to support non-CS profile generation for hybrid sample

[wenlei]

llvm/tools/llvm-profgen/PerfReader.h

646

Can this be private just like before?

[wlei]

protected --> private

[wenlei]

lgtm, thanks!

[hoy]

lgtm.

llvm/test/tools/llvm-profgen/inline-cs-noprobe.test

 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --skip-symbolization --profile-summary-cold-count=0
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-UNWINDER
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --profile-summary-cold-count=0
 ; RUN: FileCheck %s --input-file %t
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/inline-cs-noprobe.perfscript --binary=%S/Inputs/inline-cs-noprobe.perfbin --output=%t --profile-summary-cold-count=0 --ignore-stack-samples
+; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-STRIP-CTX
 
 ; CHECK:[main:1 @ foo]:309:0
 ; CHECK: 2.1: 14
 ; CHECK: 3: 15
 ; CHECK: 3.1: 14 bar:14
 ; CHECK: 3.2: 1
 ; CHECK:[main:1 @ foo:3.1 @ bar]:84:0
 ; CHECK: 1: 14
 
+; CHECK-STRIP-CTX: main:379:0
+; CHECK-STRIP-CTX:  0: 0
+; CHECK-STRIP-CTX:  2: 0
+; CHECK-STRIP-CTX:  1: foo:379
+; CHECK-STRIP-CTX:   2.1: 14
+; CHECK-STRIP-CTX:   3: 15
+; CHECK-STRIP-CTX:   3.2: 1
+; CHECK-STRIP-CTX:   4: 0
+; CHECK-STRIP-CTX:   7: 0
+; CHECK-STRIP-CTX:   3.1: bar:84
+; CHECK-STRIP-CTX:    1: 14
+
 ; CHECK-UNWINDER: [main:1 @ foo]
 ; CHECK-UNWINDER:   4
 ; CHECK-UNWINDER:   670-6ad:1
 ; CHECK-UNWINDER:   67e-69b:1
 ; CHECK-UNWINDER:   67e-6ad:13
 ; CHECK-UNWINDER:   6bd-6c8:14
 ; CHECK-UNWINDER:   2
 ; CHECK-UNWINDER:   69b->670:1

llvm/test/tools/llvm-profgen/noinline-cs-noprobe.test

 ;; The test fails on Windows. Fix it before removing the following requirement.
 ; REQUIRES: x86_64-linux
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.perfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --skip-symbolization --profile-summary-cold-count=0
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-UNWINDER
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.perfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --profile-summary-cold-count=0
 ; RUN: FileCheck %s --input-file %t
+; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.perfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --profile-summary-cold-count=0 --ignore-stack-samples
+; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-STRIP-CTX
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.aggperfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --skip-symbolization --profile-summary-cold-count=0
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-AGG-UNWINDER
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.aggperfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t --profile-summary-cold-count=0
 ; RUN: FileCheck %s --input-file %t --check-prefix=CHECK-AGG
 
 ; RUN: llvm-profgen --format=text --perfscript=%S/Inputs/noinline-cs-noprobe.perfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t1 --profile-summary-cold-count=0
 ; RUN: llvm-profgen --format=extbinary --perfscript=%S/Inputs/noinline-cs-noprobe.perfscript --binary=%S/Inputs/noinline-cs-noprobe.perfbin --output=%t2 --profile-summary-cold-count=0 -use-md5
 ; RUN: llvm-profdata show --sample -show-sec-info-only %t2 | FileCheck %s --check-prefix=CHECK-MD5

 ; CHECK: 3: 3 bar:3
 ; CHECK:[main:1 @ foo:3 @ bar]:50:3
 ; CHECK: 0: 3
 ; CHECK: 1: 3
 ; CHECK: 2: 2
 ; CHECK: 4: 1
 ; CHECK: 5: 3
 
+; CHECK-STRIP-CTX: foo:57:0
+; CHECK-STRIP-CTX:  0: 0
+; CHECK-STRIP-CTX:  1: 0
+; CHECK-STRIP-CTX:  2: 3
+; CHECK-STRIP-CTX:  3: 3 bar:3
+; CHECK-STRIP-CTX:  4: 0
+; CHECK-STRIP-CTX:  5: 0
+; CHECK-STRIP-CTX: bar:50:3
+; CHECK-STRIP-CTX:  0: 3
+; CHECK-STRIP-CTX:  1: 3
+; CHECK-STRIP-CTX:  2: 2
+; CHECK-STRIP-CTX:  4: 1
+; CHECK-STRIP-CTX:  5: 3
 
 ; CHECK-UNWINDER:      [main:1 @ foo]
 ; CHECK-UNWINDER-NEXT:   3
 ; CHECK-UNWINDER-NEXT:   5ff-62f:3
 ; CHECK-UNWINDER-NEXT:   634-637:3
 ; CHECK-UNWINDER-NEXT:   645-645:3
 ; CHECK-UNWINDER-NEXT:   3
 ; CHECK-UNWINDER-NEXT:   62f->5b0:3

llvm/tools/llvm-profgen/PerfReader.h

struct MMapEvent {
     uint64_t PID = 0;
     uint64_t Address = 0;
     uint64_t Size = 0;
     uint64_t Offset = 0;
     StringRef BinaryPath;
   };
 
   void updateBinaryAddress(const MMapEvent &Event);
-  PerfScriptType getPerfScriptType() const { return PerfType; }
   // Entry of the reader to parse multiple perf traces
   void parsePerfTraces(cl::list<std::string> &PerfTraceFilenames);
   const ContextSampleCounterMap &getSampleCounters() const {
     return SampleCounters;
   }
+  bool profileIsCS() { return ProfileIsCS; }
 
 protected:
   static PerfScriptType
   extractPerfType(cl::list<std::string> &PerfTraceFilenames);
   /// Parse a single line of a PERF_RECORD_MMAP2 event looking for a
   /// mapping between the binary name and its memory layout.
   ///
   void parseMMap2Event(TraceStream &TraceIt);
protected:
   // Parse one sample from multiple perf lines, override this for different
   // sample type
   void parseSample(TraceStream &TraceIt);
   // An aggregated count is given to indicate how many times the sample is
   // repeated.
   virtual void parseSample(TraceStream &TraceIt, uint64_t Count) = 0;
   // Post process the profile after trace aggregation, we will do simple range
   // overlap computation for AutoFDO, or unwind for CSSPGO(hybrid sample).
   virtual void generateRawProfile() = 0;
   void writeRawProfile(StringRef Filename);
   void writeRawProfile(raw_fd_ostream &OS);
 
   ProfiledBinary *Binary = nullptr;
 
   ContextSampleCounterMap SampleCounters;
   // Samples with the repeating time generated by the perf reader
   AggregatedCounter AggregatedSamples;
   PerfScriptType PerfType = PERF_UNKNOWN;
   // Keep track of all invalid return addresses
   std::set<uint64_t> InvalidReturnAddresses;
+
+  bool ProfileIsCS = false;
+};
+
+/*
+  The reader of LBR only perf script.
+  A typical LBR sample is like:
+    40062f 0x4005c8/0x4005dc/P/-/-/0   0x40062f/0x4005b0/P/-/-/0 ...
+          ... 0x4005c8/0x4005dc/P/-/-/0
+*/
+class LBRPerfReader : public PerfReaderBase {
+public:
+  LBRPerfReader(ProfiledBinary *Binary) : PerfReaderBase(Binary) {
+    PerfType = PERF_LBR;
+  };
+  // Parse the LBR only sample.
+  virtual void parseSample(TraceStream &TraceIt, uint64_t Count) override;
+  virtual void generateRawProfile() override;
+
+private:
+  void computeCounterFromLBR(const PerfSample *Sample, uint64_t Repeat);
 };
 
 /*
   Hybrid perf script includes a group of hybrid samples(LBRs + call stack),
   which is used to generate CS profile. An example of hybrid sample:
     4005dc    # call stack leaf
     400634
     400684    # call stack root
     0x4005c8/0x4005dc/P/-/-/0   0x40062f/0x4005b0/P/-/-/0 ...
           ... 0x4005c8/0x4005dc/P/-/-/0    # LBR Entries
 */
-class HybridPerfReader : public PerfReaderBase {
+class HybridPerfReader : public LBRPerfReader {
 public:
-  HybridPerfReader(ProfiledBinary *Binary) : PerfReaderBase(Binary) {
+  HybridPerfReader(ProfiledBinary *Binary) : LBRPerfReader(Binary) {
     PerfType = PERF_LBR_STACK;
   };
   // Parse the hybrid sample including the call and LBR line
   void parseSample(TraceStream &TraceIt, uint64_t Count) override;
   void generateRawProfile() override;
 
 private:
   // Unwind the hybrid samples after aggregration
   void unwindSamples();
 };
 
-/*
-  The reader of LBR only perf script.
-  A typical LBR sample is like:
-    40062f 0x4005c8/0x4005dc/P/-/-/0   0x40062f/0x4005b0/P/-/-/0 ...
-          ... 0x4005c8/0x4005dc/P/-/-/0
-*/
-class LBRPerfReader : public PerfReaderBase {
-public:
-  LBRPerfReader(ProfiledBinary *Binary) : PerfReaderBase(Binary) {
-    // There is no context for LBR only sample, so initialize one entry with
-    // fake "empty" context key.
-    std::shared_ptr<StringBasedCtxKey> Key =
-        std::make_shared<StringBasedCtxKey>();
-    Key->genHashCode();
-    SampleCounters.emplace(Hashable<ContextKey>(Key), SampleCounter());
-    PerfType = PERF_LBR;
-  };
-
-  // Parse the LBR only sample.
-  void parseSample(TraceStream &TraceIt, uint64_t Count) override;
-  void generateRawProfile() override;
-
-private:
-  void computeCounterFromLBR(const PerfSample *Sample, uint64_t Repeat);
-};
-
 } // end namespace sampleprof
 } // end namespace llvm
 
 #endif

llvm/tools/llvm-profgen/PerfReader.cpp

 
 cl::opt<bool> SkipSymbolization("skip-symbolization", cl::ReallyHidden,
                                 cl::init(false), cl::ZeroOrMore,
                                 cl::desc("Dump the unsymbolized profile to the "
                                          "output file. It will show unwinder "
                                          "output for CS profile generation."));
 cl::opt<bool> UseOffset("use-offset", cl::ReallyHidden, cl::init(true),
                         cl::ZeroOrMore,
                         cl::desc("Work with `--skip-symbolization` to dump the "
                                  "offset instead of virtual address."));
+cl::opt<bool>
+    IgnoreStackSamples("ignore-stack-samples", cl::ReallyHidden,
+                       cl::init(false), cl::ZeroOrMore,
+                       cl::desc("Ignore call stack samples for hybrid samples "
+                                "and produce context-insensitive profile."));
 
 extern cl::opt<bool> ShowDisassemblyOnly;
 extern cl::opt<bool> ShowSourceLocations;
 extern cl::opt<std::string> OutputFilename;
 
 namespace llvm {
 namespace sampleprof {
 
AllUntrackedCallsites.insert(CurrUntrackedCallsites.begin(),
                                  CurrUntrackedCallsites.end());
   }
 
   // Warn about untracked frames due to missing probes.
   for (auto Address : AllUntrackedCallsites)
     WithColor::warning() << "Profile context truncated due to missing probe "
                          << "for call instruction at "
                          << format("%" PRIx64, Address) << "\n";
-
-  if (SkipSymbolization)
-    writeRawProfile(OutputFilename);
 }
 
 bool PerfReaderBase::extractLBRStack(TraceStream &TraceIt,
                                      SmallVectorImpl<LBREntry> &LBRStack) {
   // The raw format of LBR stack is like:
   // 0x4005c8/0x4005dc/P/-/-/0 0x40062f/0x4005b0/P/-/-/0 ...
   //                           ... 0x4005c8/0x4005dc/P/-/-/0
   // It's in FIFO order and seperated by whitespace.
void LBRPerfReader::parseSample(TraceStream &TraceIt, uint64_t Count) {
   if (extractLBRStack(TraceIt, Sample->LBRStack)) {
     warnIfMissingMMap();
     // Record LBR only samples by aggregation
     AggregatedSamples[Hashable<PerfSample>(Sample)] += Count;
   }
 }
 
 void LBRPerfReader::generateRawProfile() {
-  assert(SampleCounters.size() == 1 && "Must have one entry of sample counter");
+  // There is no context for LBR only sample, so initialize one entry with
+  // fake "empty" context key.
+  assert(SampleCounters.empty() &&
+         "Sample counter map should be empty before raw profile generation");
+  std::shared_ptr<StringBasedCtxKey> Key =
+      std::make_shared<StringBasedCtxKey>();
+  Key->genHashCode();
+  SampleCounters.emplace(Hashable<ContextKey>(Key), SampleCounter());
   for (const auto &Item : AggregatedSamples) {
     const PerfSample *Sample = Item.first.getPtr();
     computeCounterFromLBR(Sample, Item.second);
   }
-
-  if (SkipSymbolization)
-    PerfReaderBase::writeRawProfile(OutputFilename);
 }
 
 uint64_t PerfReaderBase::parseAggregatedCount(TraceStream &TraceIt) {
   // The aggregated count is optional, so do not skip the line and return 1 if
   // it's unmatched
   uint64_t Count = 1;
   if (!TraceIt.getCurrentLine().getAsInteger(10, Count))
     TraceIt.advance();
if (Type == PERF_INVALID)
       exitWithError("Invalid perf script input!");
     if (PerfType != PERF_UNKNOWN && PerfType != Type)
       exitWithError("Inconsistent sample among different perf scripts");
     PerfType = Type;
   }
   return PerfType;
 }
 
-void HybridPerfReader::generateRawProfile() { unwindSamples(); }
+void HybridPerfReader::generateRawProfile() {
+  ProfileIsCS = !IgnoreStackSamples;
+  if (ProfileIsCS)
+    unwindSamples();
+  else
+    LBRPerfReader::generateRawProfile();
+}
 
 void PerfReaderBase::warnTruncatedStack() {
   for (auto Address : InvalidReturnAddresses) {
     WithColor::warning()
         << "Truncated stack sample due to invalid return address at "
         << format("0x%" PRIx64, Address)
         << ", likely caused by frame pointer omission\n";
   }
 }
 
 void PerfReaderBase::parsePerfTraces(
     cl::list<std::string> &PerfTraceFilenames) {
   // Parse perf traces and do aggregation.
   for (auto Filename : PerfTraceFilenames)
     parseAndAggregateTrace(Filename);
 
   warnTruncatedStack();
   generateRawProfile();
+
+  if (SkipSymbolization)
+    writeRawProfile(OutputFilename);
 }
 
 } // end namespace sampleprof
 } // end namespace llvm

llvm/tools/llvm-profgen/ProfileGenerator.h

 
 public:
   ProfileGeneratorBase(ProfiledBinary *Binary,
                        const ContextSampleCounterMap &Counters)
       : Binary(Binary), SampleCounters(Counters){};
   virtual ~ProfileGeneratorBase() = default;
   static std::unique_ptr<ProfileGeneratorBase>
   create(ProfiledBinary *Binary, const ContextSampleCounterMap &SampleCounters,
-         enum PerfScriptType SampleType);
+         bool ProfileIsCS);
   virtual void generateProfile() = 0;
   void write();
 
 protected:
   // Use SampleProfileWriter to serialize profile map
   void write(std::unique_ptr<SampleProfileWriter> Writer,
              SampleProfileMap &ProfileMap);
   /*

llvm/tools/llvm-profgen/ProfileGenerator.cpp

 // Initialize the MaxCompressionSize to -1 which means no size limit
 int32_t CSProfileGenerator::MaxCompressionSize = -1;
 
 int CSProfileGenerator::MaxContextDepth = -1;
 
 std::unique_ptr<ProfileGeneratorBase>
 ProfileGeneratorBase::create(ProfiledBinary *Binary,
                              const ContextSampleCounterMap &SampleCounters,
-                             enum PerfScriptType SampleType) {
+                             bool ProfileIsCS) {
   std::unique_ptr<ProfileGeneratorBase> Generator;
-  if (SampleType == PERF_LBR) {
-    // TODO: Support probe based profile generation
-    Generator.reset(new ProfileGenerator(Binary, SampleCounters));
-  } else if (SampleType == PERF_LBR_STACK) {
+  if (ProfileIsCS) {
     Generator.reset(new CSProfileGenerator(Binary, SampleCounters));
   } else {
-    llvm_unreachable("Unsupported perfscript!");
+    Generator.reset(new ProfileGenerator(Binary, SampleCounters));
   }
 
   return Generator;
 }
 
 void ProfileGeneratorBase::write(std::unique_ptr<SampleProfileWriter> Writer,
                                  SampleProfileMap &ProfileMap) {
   if (std::error_code EC = Writer->write(ProfileMap))

llvm/tools/llvm-profgen/llvm-profgen.cpp

std::unique_ptr<ProfiledBinary> Binary =
       std::make_unique<ProfiledBinary>(BinaryPath);
   if (ShowDisassemblyOnly)
     return EXIT_SUCCESS;
 
   // Parse perf events and samples
   std::unique_ptr<PerfReaderBase> Reader =
       PerfReaderBase::create(Binary.get(), PerfTraceFilenames);
   Reader->parsePerfTraces(PerfTraceFilenames);
 
   if (SkipSymbolization)
     return EXIT_SUCCESS;
 
   std::unique_ptr<ProfileGeneratorBase> Generator =
       ProfileGeneratorBase::create(Binary.get(), Reader->getSampleCounters(),
-                                   Reader->getPerfScriptType());
+                                   Reader->profileIsCS());
   Generator->generateProfile();
   Generator->write();
 
   return EXIT_SUCCESS;
 }