[SampleFDO] Expose an interface to return the size of a section or the size of the profile for profile in ExtBinary format

[wmi]

Summary

Sometimes we want to limit the size of the profile by stripping some functions with low sample count or by stripping some function names with small text size from profile symbol list. That requires the profile reader to have the interfaces returning the size of a section or the size of total profile. The patch add those interfaces.

At the same time, add some dump facility to show the size of each section.

[davidxl]

Missing tests.

[wmi]

Add test to verify the size of the profile matching the size shown by llvm-profdata and add an assertion to verify the size of individual section and header adds up to total file size.

[davidxl]

Is the intended workflow to create a profile with full symbol list and then have an option to trim symbol list according to size impact? Why not do the trimming at the time of the profile creation?

[wmi]

In D67726#1675481, @davidxl wrote:

Is the intended workflow to create a profile with full symbol list and then have an option to trim symbol list according to size impact? Why not do the trimming at the time of the profile creation?

Yes, that is the intended workflow. Because before creating the profile, we don't know how much size the profile will have especially after compression. For the size of profile symbol list section, we can estimate the size after compression, but since we already trim the profile iteratively by dropping symbols in FunctionSamples, regenerating the profile and reevaluating the size, I tend to use the same way to handle symbol list. The cons include longer profile generation time. The pros include preciseness -- when we limit the profile size to be 10M, the profile generated will be close to it but won't exceed it. If we estimate the size of the profile, there may be outliers.

We can have some methods to reduce the iterations of cutting the profile in the future. Currently it is not a problem.

[davidxl]

Sounds a tricky situation to handle. Suppose the profile size increases some limit, what criterial should be used to trim the size? Trim the symbol list or drop more samples?

[wmi]

In D67726#1675508, @davidxl wrote:

Sounds a tricky situation to handle. Suppose the profile size increases some limit, what criterial should be used to trim the size? Trim the symbol list or drop more samples?

We plan to use separate limits for symbol list and the rest. If size of symbol list is over the limit, trim symbol list, otherwise drop samples.

[davidxl]

lib/ProfileData/SampleProfReader.cpp

683

Perhaps makes this a virtual function and let the base impl return 'false' (so that caller can warn).

[wmi]

Address David's comment.

[davidxl]

lgtm

[wmi]

Committed at rL372439. Sorry I missed to add Differential Revision msg in the commit so close it manually.

include/llvm/ProfileData/SampleProf.h

enum SecType {
   SecProfSummary = 1,
   SecNameTable = 2,
   SecProfileSymbolList = 3,
   // marker for the first type of profile.
   SecFuncProfileFirst = 32,
   SecLBRProfile = SecFuncProfileFirst
 };
 
+static inline std::string getSecName(SecType Type) {
+  switch (Type) {
+  case SecInValid:
+    return "InvalidSection";
+  case SecProfSummary:
+    return "ProfileSummarySection";
+  case SecNameTable:
+    return "NameTableSection";
+  case SecProfileSymbolList:
+    return "ProfileSymbolListSection";
+  case SecLBRProfile:
+    return "LBRProfileSection";
+  }
+  llvm_unreachable("A SecType has no name for output");
+}
+
 // Entry type of section header table used by SampleProfileExtBinaryBaseReader
 // and SampleProfileExtBinaryBaseWriter.
 struct SecHdrTableEntry {
   SecType Type;
   uint64_t Flag;
   uint64_t Offset;
   uint64_t Size;
 };

include/llvm/ProfileData/SampleProfReader.h

public:
 
   virtual std::unique_ptr<ProfileSymbolList> getProfileSymbolList() {
     return nullptr;
   };
 
   /// It includes all the names that have samples either in outline instance
   /// or inline instance.
   virtual std::vector<StringRef> *getNameTable() { return nullptr; }
+  virtual bool dumpSectionInfo(raw_ostream &OS = dbgs()) { return false; };
 
 protected:
   /// Map every function to its associated profile.
   ///
   /// The profile of every function executed at runtime is collected
   /// in the structure FunctionSamples. This maps function objects
   /// to their corresponding profiles.
   StringMap<FunctionSamples> Profiles;

 
 public:
   SampleProfileReaderExtBinaryBase(std::unique_ptr<MemoryBuffer> B,
                                    LLVMContext &C, SampleProfileFormat Format)
       : SampleProfileReaderBinary(std::move(B), C, Format) {}
 
   /// Read sample profiles in extensible format from the associated file.
   std::error_code read() override;
+
+  /// Get the total size of all \p Type sections.
+  uint64_t getSectionSize(SecType Type);
+  /// Get the total size of header and all sections.
+  uint64_t getFileSize();
+  virtual bool dumpSectionInfo(raw_ostream &OS = dbgs()) override;
 };
 
 class SampleProfileReaderExtBinary : public SampleProfileReaderExtBinaryBase {
 private:
   virtual std::error_code verifySPMagic(uint64_t Magic) override;
   virtual std::error_code readOneSection(const uint8_t *Start, uint64_t Size,
                                          SecType Type) override;
   std::error_code readProfileSymbolList();

lib/ProfileData/SampleProfReader.cpp

if (std::error_code EC = readMagicIdent())
     return EC;
 
   if (std::error_code EC = readSecHdrTable())
     return EC;
 
   return sampleprof_error::success;
 }
 
+uint64_t SampleProfileReaderExtBinaryBase::getSectionSize(SecType Type) {
+  for (auto &Entry : SecHdrTable) {
+    if (Entry.Type == Type)
+      return Entry.Size;
+  }
+  return 0;
+}
+
+uint64_t SampleProfileReaderExtBinaryBase::getFileSize() {
+  auto &LastEntry = SecHdrTable.back();
+  return LastEntry.Offset + LastEntry.Size;
+}
+
+bool SampleProfileReaderExtBinaryBase::dumpSectionInfo(raw_ostream &OS) {
+  uint64_t TotalSecsSize = 0;
+  for (auto &Entry : SecHdrTable) {
+    OS << getSecName(Entry.Type) << " - Offset: " << Entry.Offset
+       << ", Size: " << Entry.Size << "\n";
+    TotalSecsSize += getSectionSize(Entry.Type);
+  }
+  uint64_t HeaderSize = SecHdrTable.front().Offset;
+  assert(HeaderSize + TotalSecsSize == getFileSize() &&
+         "Size of 'header + sections' doesn't match the total size of profile");
+
+  OS << "Header Size: " << HeaderSize << "\n";
+  OS << "Total Sections Size: " << TotalSecsSize << "\n";
+  OS << "File Size: " << getFileSize() << "\n";
+  return true;
+}
+
 std::error_code SampleProfileReaderBinary::readMagicIdent() {
   // Read and check the magic identifier.
   auto Magic = readNumber<uint64_t>();
   if (std::error_code EC = Magic.getError())
     return EC;
   else if (std::error_code EC = verifySPMagic(*Magic))
     return EC;
 

test/tools/llvm-profdata/show-prof-size.test

+; RUN: llvm-profdata merge -sample -extbinary -prof-sym-list=%S/Inputs/profile-symbol-list-1.text %S/Inputs/sample-profile.proftext -o %t.1.output
+; RUN: ls -l %t.1.output |cut -f5 -d ' ' > %t.txt
+; RUN: llvm-profdata show -sample -show-sec-info-only %t.1.output >> %t.txt
+; RUN: FileCheck %s --input-file=%t.txt
+; Check llvm-profdata shows the correct file size.
+; CHECK: [[FILESIZE:.*]]
+; CHECK: [[FILESIZE]]

tools/llvm-profdata/llvm-profdata.cpp

for (auto Entry : PS->getDetailedSummary()) {
          << " account for "
          << format("%0.6g", (float)Entry.Cutoff / ProfileSummary::Scale * 100)
          << " percentage of the total counts.\n";
     }
   }
   return 0;
 }
 
+static void showSectionInfo(sampleprof::SampleProfileReader *Reader,
+                            raw_fd_ostream &OS) {
+  if (!Reader->dumpSectionInfo(OS)) {
+    WithColor::warning() << "-show-sec-info-only is only supported for "
+                         << "sample profile in extbinary format and is "
+                         << "ignored for other formats.\n";
+    return;
+  }
+}
+
 static int showSampleProfile(const std::string &Filename, bool ShowCounts,
                              bool ShowAllFunctions,
                              const std::string &ShowFunction,
-                             bool ShowProfileSymbolList, raw_fd_ostream &OS) {
+                             bool ShowProfileSymbolList,
+                             bool ShowSectionInfoOnly, raw_fd_ostream &OS) {
   using namespace sampleprof;
   LLVMContext Context;
   auto ReaderOrErr = SampleProfileReader::create(Filename, Context);
   if (std::error_code EC = ReaderOrErr.getError())
     exitWithErrorCode(EC, Filename);
 
   auto Reader = std::move(ReaderOrErr.get());
+
+  if (ShowSectionInfoOnly) {
+    showSectionInfo(Reader.get(), OS);
+    return 0;
+  }
+
   if (std::error_code EC = Reader->read())
     exitWithErrorCode(EC, Filename);
 
   if (ShowAllFunctions || ShowFunction.empty())
     Reader->dump(OS);
   else
     Reader->dumpFunctionProfile(ShowFunction, OS);
 
cl::opt<uint32_t> ValueCutoff(
                "less than this value will not be printed out. (Default is 0)"));
   cl::opt<bool> OnlyListBelow(
       "list-below-cutoff", cl::init(false),
       cl::desc("Only output names of functions whose max count values are "
                "below the cutoff value"));
   cl::opt<bool> ShowProfileSymbolList(
       "show-prof-sym-list", cl::init(false),
       cl::desc("Show profile symbol list if it exists in the profile. "));
+  cl::opt<bool> ShowSectionInfoOnly(
+      "show-sec-info-only", cl::init(false),
+      cl::desc("Show the information of each section in the sample profile. "
+               "The flag is only usable when the sample profile is in "
+               "extbinary format"));
 
   cl::ParseCommandLineOptions(argc, argv, "LLVM profile data summary\n");
 
   if (OutputFilename.empty())
     OutputFilename = "-";
 
   if (!Filename.compare(OutputFilename)) {
     errs() << sys::path::filename(argv[0])
static int show_main(int argc, const char *argv[]) {
   if (ProfileKind == instr)
     return showInstrProfile(Filename, ShowCounts, TopNFunctions,
                             ShowIndirectCallTargets, ShowMemOPSizes,
                             ShowDetailedSummary, DetailedSummaryCutoffs,
                             ShowAllFunctions, ShowCS, ValueCutoff,
                             OnlyListBelow, ShowFunction, TextFormat, OS);
   else
     return showSampleProfile(Filename, ShowCounts, ShowAllFunctions,
-                             ShowFunction, ShowProfileSymbolList, OS);
+                             ShowFunction, ShowProfileSymbolList,
+                             ShowSectionInfoOnly, OS);
 }
 
 int main(int argc, const char *argv[]) {
   InitLLVM X(argc, argv);
 
   StringRef ProgName(sys::path::filename(argv[0]));
   if (argc > 1) {
     int (*func)(int, const char *[]) = nullptr;