[PGO] Profile Summary Index profile writer and reader support

[davidxl]

Summary

Profile summary can be expensive to recompute, so it is better to compute it once during profile write and save the summary in the indexed profile data. This patch adds the support for summary writing and reading.

For older versions of the profile data, the reader is taught to re-synthesize the summary data on the fly. Since profileSummary includes MaxFunctionCount already, existing fields in InstrProfReader and Writer are removed. The MaxFunctionCount computation code is also removed (redundant). The Header still includes MaxFunctionCount field to minimize layout change. This field is deprecated and can be repurposed for something else in the future.

[vsk]

Thanks, comments inline --

include/llvm/ProfileData/InstrProf.h

673 ↗

(On Diff #45072)

Tiny nit, could you move one line up?

709 ↗

(On Diff #45072)

Nit, do you mind keeping the original comment ('The maximal execution count among all functions.')?

Wdyt of changing getMaxFunctionCount() to return an Optional<uint64_t> in a later commit?

714 ↗

(On Diff #45072)

Missing comment.

726 ↗

(On Diff #45072)

Why not use a SmallVector<uint64_t, N>? Then there's no need for NumEntries, getSize(), allocSummary() etc.

lib/ProfileData/InstrProf.cpp

642 ↗

(On Diff #45072)

A range-based for loop would be nicer here.

644 ↗

(On Diff #45072)

If you add a constructor to ProfileSummaryEntry, you can take advantage of emplace_back() here.

lib/ProfileData/InstrProfReader.cpp

567 ↗

(On Diff #45072)

In this scenario I'd prefer using a different name to avoid confusion and extra 'this->'s. Maybe 'SummaryInHE'?

582 ↗

(On Diff #45072)

As written, constructing the Cutoffs vector takes linear time. Passing it into the ProfileSummary constructor is also linear time. ISTM the ProfileSummary constructor shouldn't accept a vector of cutoffs, it should use ArrayRef.

If you agree, I can take care of it in a separate patch.

583 ↗

(On Diff #45072)

Ignoring the result of a getter is a bit confusing. Maybe you could call computeDetailedSummary() directly?

lib/ProfileData/InstrProfWriter.cpp

78 ↗

(On Diff #45072)

These static variables break the use-case of running llvm in a multi-threaded environment! If you know of any other instances like this, they need to be fixed.

Please pass a summary pointer into EmitData directly.

194 ↗

(On Diff #45072)

Using a vector-like container for Entries turns this loop into a resize() + std::copy.

[eraman]

include/llvm/ProfileData/InstrProf.h

698 ↗

(On Diff #45072)

it's useful to add a cutoff of 999999 to identify cold blocks. IMO these are more useful than the cutoffs in the 1-80% range.

[davidxl]

include/llvm/ProfileData/InstrProf.h

726 ↗

(On Diff #45072)

This data structure is used for serialization describing on disk/buffer data. The in-memory data structure is ProfileSummary

lib/ProfileData/InstrProf.cpp

642 ↗

(On Diff #45072)

The number of entries is unknown statically.

lib/ProfileData/InstrProfReader.cpp

582 ↗

(On Diff #45072)

I did not notice -- I think making the ProfileSummary to take a const reference to vector should do. Or even better to avoid the the copy from Cutoffs to DetailedSummaryCutoffs member, a unique_ptr to the cutoffs can be created by the creator of ProfileSummary and the ownership of the cutoffs can be passed in.

Feel free to improve this part.

lib/ProfileData/InstrProfWriter.cpp

78 ↗

(On Diff #45072)

Fixed.

194 ↗

(On Diff #45072)

The target data structure is serialization/persistence so it is in 'raw' form.

[davidxl]

Update patch by incorporating vsk and eraman's comments.

[davidxl]

Misc updates of the patch:

1. renamed unused header field
2. ensure summary data to be zeroalized
3. added more comments
4. added reserved fields in summary for future extension.

[davidxl]

Any more comments on this patch?

[eraman]

include/llvm/ProfileData/InstrProf.h

743 ↗

(On Diff #45881)

This is not very useful. In practice we should expect MaxInternalBlockCount == MaxBlockCount

include/llvm/ProfileData/InstrProfReader.h

345 ↗

(On Diff #45881)

Nit: pass -> past

[davidxl]

include/llvm/ProfileData/InstrProf.h

743 ↗

(On Diff #45881)

This field is introduced to match the corresponding field in ProfileSummary -- it is used by the current dumper. I prefer keeping it (just 8 byte space).

include/llvm/ProfileData/InstrProfReader.h

345 ↗

(On Diff #45881)

will change.

[davidxl]

Fix a typo.

[davidxl]

ping

[vsk]

include/llvm/ProfileData/InstrProf.h

751 ↗

(On Diff #46351)

I'm not sure about this Reserved field. IMO any new future functionality should come with a format version bump, even if the on-disk layout stays the same. That means we can teach the reader to skip straight to NumEntries for v4 files.

lib/ProfileData/InstrProfWriter.cpp

197 ↗

(On Diff #46351)

I see. Is it possible to make the Entries array an array of ProfileSummaryEntry? We'd have to make the Cutoff field 64-bit, but it would help keep things in sync. That also lets us use the default copy-constructor here, i.e TheSummary->Entries[I] = Res[I].

[xur]

include/llvm/ProfileData/InstrProf.h

751 ↗

(On Diff #46351)

Do we need to use uint64_t for TotalNumFunctions and TotalNumBlocks. Will uint32_t enough?

[davidxl]

include/llvm/ProfileData/InstrProf.h

751 ↗

(On Diff #46351)

Yes it is true that the version number will always bump up when new functionality is added -- however keeping the same layout can greatly simplify the reader change.

To remove the reserved field, I think it might be better to split the summary into two parts: summary header plus the Summary entry array. I will make that change and remove the reserved fields.

lib/ProfileData/InstrProfWriter.cpp

197 ↗

(On Diff #46351)

It is tempting to do that -- but ProfileSummaryEntry data structure is intended to also be shared with sample PGO and its contents may be subject to unrelated changes in the future which may affect layout. I would like to avoid it.

[davidxl]

include/llvm/ProfileData/InstrProf.h

751 ↗

(On Diff #46351)

uint64_t is consistently used in all fields in the header.

[davidxl]

Rebase and address vsk's comment (no more reserved fields)

[vsk]

Thanks, lgtm

llvm/trunk/include/llvm/ProfileData/InstrProf.h

/* Return the size of ValueProfData structure that can be used to store
    the value profile data collected at runtime. */
 uint32_t getValueProfDataSizeRT(const ValueProfRuntimeRecord *Record);
 
 /* Return a ValueProfData instance that stores the data collected at runtime. */
 ValueProfData *
 serializeValueProfDataFromRT(const ValueProfRuntimeRecord *Record,
                              ValueProfData *Dst);
 
+namespace IndexedInstrProf {
+struct Summary;
+}
+
 ///// Profile summary computation ////
 // The 'show' command displays richer summary of the profile data. The profile
 // summary is one or more (Cutoff, MinBlockCount, NumBlocks) triplets. Given a
 // target execution count percentile, we compute the minimum number of blocks
 // needed to reach this target and the minimum execution count of these blocks.
 struct ProfileSummaryEntry {
   uint32_t Cutoff;        ///< The required percentile of total execution count.
   uint64_t MinBlockCount; ///< The minimum execution count for this percentile.
   uint64_t NumBlocks;     ///< Number of blocks >= the minumum execution count.
+  ProfileSummaryEntry(uint32_t TheCutoff, uint64_t TheMinBlockCount,
+                      uint64_t TheNumBlocks)
+      : Cutoff(TheCutoff), MinBlockCount(TheMinBlockCount),
+        NumBlocks(TheNumBlocks) {}
 };
 
 class ProfileSummary {
   // We keep track of the number of times a count appears in the profile and
   // keep the map sorted in the descending order of counts.
   std::map<uint64_t, uint32_t, std::greater<uint64_t>> CountFrequencies;
   std::vector<ProfileSummaryEntry> DetailedSummary;
   std::vector<uint32_t> DetailedSummaryCutoffs;
   // Sum of all counts.
   uint64_t TotalCount;
   uint64_t MaxBlockCount, MaxInternalBlockCount, MaxFunctionCount;
   uint32_t NumBlocks, NumFunctions;
   inline void addCount(uint64_t Count, bool IsEntry);
-  void computeDetailedSummary();
 
 public:
   static const int Scale = 1000000;
   ProfileSummary(std::vector<uint32_t> Cutoffs)
       : DetailedSummaryCutoffs(Cutoffs), TotalCount(0), MaxBlockCount(0),
-        MaxInternalBlockCount(0), MaxFunctionCount(0), NumBlocks(0), NumFunctions(0) {}
+        MaxInternalBlockCount(0), MaxFunctionCount(0), NumBlocks(0),
+        NumFunctions(0) {}
+  ProfileSummary(const IndexedInstrProf::Summary &S);
   inline void addRecord(const InstrProfRecord &);
   inline std::vector<ProfileSummaryEntry> &getDetailedSummary();
+  void computeDetailedSummary();
   uint32_t getNumBlocks() { return NumBlocks; }
   uint64_t getTotalCount() { return TotalCount; }
   uint32_t getNumFunctions() { return NumFunctions; }
   uint64_t getMaxFunctionCount() { return MaxFunctionCount; }
   uint64_t getMaxBlockCount() { return MaxBlockCount; }
   uint64_t getMaxInternalBlockCount() { return MaxInternalBlockCount; }
 };
 
enum ProfVersion {
   // The version 2 format supports recording profile data of multiple
   // functions which share the same key in one value field. To support this,
   // the number block counters is recorded as an uint64_t field right after the
   // function structural hash.
   Version2 = 2,
   // Version 3 supports value profile data. The value profile data is expected
   // to follow the block counter profile data.
   Version3 = 3,
-  // The current version is 3.
+  // In this version, profile summary data \c IndexedInstrProf::Summary is
+  // stored after the profile header.
+  Version4 = 4,
+  // The current version is 4.
   CurrentVersion = INSTR_PROF_INDEX_VERSION
 };
 const uint64_t Version = ProfVersion::CurrentVersion;
 
 const HashT HashType = HashT::MD5;
 
 inline uint64_t ComputeHash(StringRef K) { return ComputeHash(HashType, K); }
 
 // This structure defines the file header of the LLVM profile
 // data file in indexed-format.
 struct Header {
   uint64_t Magic;
   uint64_t Version;
-  uint64_t MaxFunctionCount;
+  uint64_t Unused; // Becomes unused since version 4
   uint64_t HashType;
   uint64_t HashOffset;
 };
 
+static const uint32_t SummaryCutoffs[] = {
+    10000,  /*  1% */
+    100000, /* 10% */
+    200000, 300000, 400000, 500000, 600000, 500000, 600000, 700000,
+    800000, 900000, 950000, 990000, 999000, 999900, 999990, 999999};
+static const uint32_t NumSummaryCutoffs =
+    sizeof(SummaryCutoffs) / sizeof(*SummaryCutoffs);
+
+// Profile summary data recorded in the profile data file in indexed
+// format. It is introduced in version 4. The summary data follows
+// right after the profile file header.
+struct Summary {
+
+  struct Entry {
+    uint64_t Cutoff; ///< The required percentile of total execution count.
+    uint64_t
+        MinBlockCount;  ///< The minimum execution count for this percentile.
+    uint64_t NumBlocks; ///< Number of blocks >= the minumum execution count.
+  };
+  // New field kind to existing enum value mapping should remain unchanged
+  // when new kind is added in the future.
+  enum SummaryFieldKind {
+    /// The total number of functions instrumented.
+    TotalNumFunctions = 0,
+    /// Total number of instrumented blocks/edges.
+    TotalNumBlocks = 1,
+    /// The maximal execution count among all functions.
+    /// This field does not exist for profile data from IR based
+    /// instrumentation.
+    MaxFunctionCount = 2,
+    /// Max block count of the program.
+    MaxBlockCount = 3,
+    /// Max internal block count of the program (excluding entry blocks).
+    MaxInternalBlockCount = 4,
+    /// The sum of all instrumented block counts.
+    TotalBlockCount = 5,
+    NumKinds = TotalBlockCount + 1
+  };
+
+  // The number of summmary fields following the summary header.
+  uint64_t NumSummaryFields;
+  // The number of Cutoff Entries (Summary::Entry) following summary fields.
+  uint64_t NumCutoffEntries;
+
+  static uint32_t getSize(uint32_t NumSumFields, uint32_t NumCutoffEntries) {
+    return sizeof(Summary) + NumCutoffEntries * sizeof(Entry) +
+           NumSumFields * sizeof(uint64_t);
+  }
+
+  const uint64_t *getSummaryDataBase() const {
+    return reinterpret_cast<const uint64_t *>(this + 1);
+  }
+  uint64_t *getSummaryDataBase() {
+    return reinterpret_cast<uint64_t *>(this + 1);
+  }
+  const Entry *getCutoffEntryBase() const {
+    return reinterpret_cast<const Entry *>(
+        &getSummaryDataBase()[NumSummaryFields]);
+  }
+  Entry *getCutoffEntryBase() {
+    return reinterpret_cast<Entry *>(&getSummaryDataBase()[NumSummaryFields]);
+  }
+
+  uint64_t get(SummaryFieldKind K) const {
+    return getSummaryDataBase()[K];
+  }
+
+  void set(SummaryFieldKind K, uint64_t V) {
+    getSummaryDataBase()[K] = V;
+  }
+
+  const Entry &getEntry(uint32_t I) const { return getCutoffEntryBase()[I]; }
+  void setEntry(uint32_t I, const ProfileSummaryEntry &E) {
+    Entry &ER = getCutoffEntryBase()[I];
+    ER.Cutoff = E.Cutoff;
+    ER.MinBlockCount = E.MinBlockCount;
+    ER.NumBlocks = E.NumBlocks;
+  }
+
+  Summary(uint32_t Size) { memset(this, 0, Size); }
+  void operator delete(void *ptr) { ::operator delete(ptr); }
+
+  Summary() = delete;
+};
+
+inline std::unique_ptr<Summary> allocSummary(uint32_t TotalSize) {
+  return std::unique_ptr<Summary>(new (::operator new(TotalSize))
+                                      Summary(TotalSize));
+}
 } // end namespace IndexedInstrProf
 
 namespace RawInstrProf {
 
 const uint64_t Version = INSTR_PROF_RAW_VERSION;
 
 template <class IntPtrT> inline uint64_t getMagic();
 template <> inline uint64_t getMagic<uint64_t>() {

llvm/trunk/include/llvm/ProfileData/InstrProfData.inc

#define INSTR_PROF_RAW_MAGIC_64 (uint64_t)255 << 56 | (uint64_t)'l' << 48 | \
        (uint64_t)'p' << 40 | (uint64_t)'r' << 32 | (uint64_t)'o' << 24 |  \
         (uint64_t)'f' << 16 | (uint64_t)'r' << 8 | (uint64_t)129
 #define INSTR_PROF_RAW_MAGIC_32 (uint64_t)255 << 56 | (uint64_t)'l' << 48 | \
        (uint64_t)'p' << 40 | (uint64_t)'r' << 32 | (uint64_t)'o' << 24 |  \
         (uint64_t)'f' << 16 | (uint64_t)'R' << 8 | (uint64_t)129
 
 /* Raw profile format version. */
 #define INSTR_PROF_RAW_VERSION 2
-#define INSTR_PROF_INDEX_VERSION 3
+#define INSTR_PROF_INDEX_VERSION 4
 #define INSTR_PROF_COVMAP_VERSION 0
 
 /* Profile version is always of type uint64_t. Reserve the upper 8 bits in the
  * version for other variants of profile. We set the lowest bit of the upper 8
  * bits (i.e. bit 56) to 1 to indicate if this is an IR-level instrumentaiton
  * generated profile, and 0 if this is a Clang FE generated profile.
 */
 #define VARIANT_MASKS_ALL 0xff00000000000000ULL

llvm/trunk/include/llvm/ProfileData/InstrProfReader.h

 
 /// Reader for the indexed binary instrprof format.
 class IndexedInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   /// The index into the profile data.
   std::unique_ptr<InstrProfReaderIndexBase> Index;
-  /// The maximal execution count among all functions.
-  uint64_t MaxFunctionCount;
+  /// Profile summary data.
+  std::unique_ptr<ProfileSummary> Summary;
 
   IndexedInstrProfReader(const IndexedInstrProfReader &) = delete;
   IndexedInstrProfReader &operator=(const IndexedInstrProfReader &) = delete;
 
+  // Read the profile summary. Return a pointer pointing to one byte past the
+  // end of the summary data if it exists or the input \c Cur.
+  const unsigned char *readSummary(IndexedInstrProf::ProfVersion Version,
+                                   const unsigned char *Cur);
+
 public:
   uint64_t getVersion() const { return Index->getVersion(); }
   IndexedInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer)
       : DataBuffer(std::move(DataBuffer)), Index(nullptr) {}
 
   /// Return true if the given buffer is in an indexed instrprof format.
   static bool hasFormat(const MemoryBuffer &DataBuffer);
 
   /// Read the file header.
   std::error_code readHeader() override;
   /// Read a single record.
   std::error_code readNextRecord(InstrProfRecord &Record) override;
 
   /// Return the pointer to InstrProfRecord associated with FuncName
   /// and FuncHash
   ErrorOr<InstrProfRecord> getInstrProfRecord(StringRef FuncName,
                                               uint64_t FuncHash);
 
   /// Fill Counts with the profile data for the given function name.
   std::error_code getFunctionCounts(StringRef FuncName, uint64_t FuncHash,
                                     std::vector<uint64_t> &Counts);
 
   /// Return the maximum of all known function counts.
-  uint64_t getMaximumFunctionCount() { return MaxFunctionCount; }
+  uint64_t getMaximumFunctionCount() { return Summary->getMaxFunctionCount(); }
 
   /// Factory method to create an indexed reader.
   static ErrorOr<std::unique_ptr<IndexedInstrProfReader>>
   create(std::string Path);
 
   static ErrorOr<std::unique_ptr<IndexedInstrProfReader>>
   create(std::unique_ptr<MemoryBuffer> Buffer);
 
   // Used for testing purpose only.
   void setValueProfDataEndianness(support::endianness Endianness) {
     Index->setValueProfDataEndianness(Endianness);
   }
 
   // See description in the base class. This interface is designed
   // to be used by llvm-profdata (for dumping). Avoid using this when
   // the client is the compiler.
   InstrProfSymtab &getSymtab() override;
+  ProfileSummary &getSummary() { return *(Summary.get()); }
 };
 
 } // end namespace llvm
 
 #endif

llvm/trunk/include/llvm/ProfileData/InstrProfWriter.h

 
 class InstrProfWriter {
 public:
   typedef SmallDenseMap<uint64_t, InstrProfRecord, 1> ProfilingData;
 
 private:
   bool Sparse;
   StringMap<ProfilingData> FunctionData;
-  uint64_t MaxFunctionCount;
   // Use raw pointer here for the incomplete type object.
   InstrProfRecordWriterTrait *InfoObj;
 
 public:
   InstrProfWriter(bool Sparse = false);
   ~InstrProfWriter();
 
   /// Add function counts for the given function. If there are already counts

llvm/trunk/lib/ProfileData/InstrProf.cpp

while (CurrSum < DesiredCount && Iter != End) {
       Iter++;
     }
     assert(CurrSum >= DesiredCount);
     ProfileSummaryEntry PSE = {Cutoff, Count, BlocksSeen};
     DetailedSummary.push_back(PSE);
   }
 }
 
+ProfileSummary::ProfileSummary(const IndexedInstrProf::Summary &S)
+    : TotalCount(S.get(IndexedInstrProf::Summary::TotalBlockCount)),
+      MaxBlockCount(S.get(IndexedInstrProf::Summary::MaxBlockCount)),
+      MaxInternalBlockCount(
+          S.get(IndexedInstrProf::Summary::MaxInternalBlockCount)),
+      MaxFunctionCount(S.get(IndexedInstrProf::Summary::MaxFunctionCount)),
+      NumBlocks(S.get(IndexedInstrProf::Summary::TotalNumBlocks)),
+      NumFunctions(S.get(IndexedInstrProf::Summary::TotalNumFunctions)) {
+  for (unsigned I = 0; I < S.NumCutoffEntries; I++) {
+    const IndexedInstrProf::Summary::Entry &Ent = S.getEntry(I);
+    DetailedSummary.emplace_back((uint32_t)Ent.Cutoff, Ent.MinBlockCount,
+                                 Ent.NumBlocks);
+  }
+}
+
 } // end namespace llvm

llvm/trunk/lib/ProfileData/InstrProfReader.cpp

if (DataBuffer.getBufferSize() < 8)
     return false;
   using namespace support;
   uint64_t Magic =
       endian::read<uint64_t, little, aligned>(DataBuffer.getBufferStart());
   // Verify that it's magical.
   return Magic == IndexedInstrProf::Magic;
 }
 
+const unsigned char *
+IndexedInstrProfReader::readSummary(IndexedInstrProf::ProfVersion Version,
+                                    const unsigned char *Cur) {
+  using namespace support;
+  if (Version >= IndexedInstrProf::Version4) {
+    const IndexedInstrProf::Summary *SummaryInLE =
+        reinterpret_cast<const IndexedInstrProf::Summary *>(Cur);
+    uint64_t NFields =
+        endian::byte_swap<uint64_t, little>(SummaryInLE->NumSummaryFields);
+    uint64_t NEntries =
+        endian::byte_swap<uint64_t, little>(SummaryInLE->NumCutoffEntries);
+    uint32_t SummarySize =
+        IndexedInstrProf::Summary::getSize(NFields, NEntries);
+    std::unique_ptr<IndexedInstrProf::Summary> SummaryData =
+        IndexedInstrProf::allocSummary(SummarySize);
+
+    const uint64_t *Src = reinterpret_cast<const uint64_t *>(SummaryInLE);
+    uint64_t *Dst = reinterpret_cast<uint64_t *>(SummaryData.get());
+    for (unsigned I = 0; I < SummarySize / sizeof(uint64_t); I++)
+      Dst[I] = endian::byte_swap<uint64_t, little>(Src[I]);
+
+    // initialize ProfileSummary using the SummaryData from disk.
+    this->Summary = llvm::make_unique<ProfileSummary>(*(SummaryData.get()));
+    return Cur + SummarySize;
+  } else {
+    // For older version of profile data, we need to compute on the fly:
+    using namespace IndexedInstrProf;
+    std::vector<uint32_t> Cutoffs(&SummaryCutoffs[0],
+                                  &SummaryCutoffs[NumSummaryCutoffs]);
+    this->Summary = llvm::make_unique<ProfileSummary>(Cutoffs);
+    this->Summary->computeDetailedSummary();
+    return Cur;
+  }
+}
+
 std::error_code IndexedInstrProfReader::readHeader() {
   const unsigned char *Start =
       (const unsigned char *)DataBuffer->getBufferStart();
   const unsigned char *Cur = Start;
   if ((const unsigned char *)DataBuffer->getBufferEnd() - Cur < 24)
     return error(instrprof_error::truncated);
 
   using namespace support;
 
   auto *Header = reinterpret_cast<const IndexedInstrProf::Header *>(Cur);
   Cur += sizeof(IndexedInstrProf::Header);
 
   // Check the magic number.
   uint64_t Magic = endian::byte_swap<uint64_t, little>(Header->Magic);
   if (Magic != IndexedInstrProf::Magic)
     return error(instrprof_error::bad_magic);
 
   // Read the version.
   uint64_t FormatVersion = endian::byte_swap<uint64_t, little>(Header->Version);
   if (FormatVersion > IndexedInstrProf::ProfVersion::CurrentVersion)
     return error(instrprof_error::unsupported_version);
 
-  // Read the maximal function count.
-  MaxFunctionCount =
-      endian::byte_swap<uint64_t, little>(Header->MaxFunctionCount);
+  Cur = readSummary((IndexedInstrProf::ProfVersion)FormatVersion, Cur);
 
   // Read the hash type and start offset.
   IndexedInstrProf::HashT HashType = static_cast<IndexedInstrProf::HashT>(
       endian::byte_swap<uint64_t, little>(Header->HashType));
   if (HashType > IndexedInstrProf::HashT::Last)
     return error(instrprof_error::unsupported_hash_type);
 
   uint64_t HashOffset = endian::byte_swap<uint64_t, little>(Header->HashOffset);

llvm/trunk/lib/ProfileData/InstrProfWriter.cpp

public:
 
   typedef const InstrProfWriter::ProfilingData *const data_type;
   typedef const InstrProfWriter::ProfilingData *const data_type_ref;
 
   typedef uint64_t hash_value_type;
   typedef uint64_t offset_type;
 
   support::endianness ValueProfDataEndianness;
+  ProfileSummary *TheProfileSummary;
 
   InstrProfRecordWriterTrait() : ValueProfDataEndianness(support::little) {}
   static hash_value_type ComputeHash(key_type_ref K) {
     return IndexedInstrProf::ComputeHash(K);
   }
 
   static std::pair<offset_type, offset_type>
   EmitKeyDataLength(raw_ostream &Out, key_type_ref K, data_type_ref V) {
void EmitKey(raw_ostream &Out, key_type_ref K, offset_type N) {
     Out.write(K.data(), N);
   }
 
   void EmitData(raw_ostream &Out, key_type_ref, data_type_ref V, offset_type) {
     using namespace llvm::support;
     endian::Writer<little> LE(Out);
     for (const auto &ProfileData : *V) {
       const InstrProfRecord &ProfRecord = ProfileData.second;
+      TheProfileSummary->addRecord(ProfRecord);
 
       LE.write<uint64_t>(ProfileData.first); // Function hash
       LE.write<uint64_t>(ProfRecord.Counts.size());
       for (uint64_t I : ProfRecord.Counts)
         LE.write<uint64_t>(I);
 
       // Write value data
       std::unique_ptr<ValueProfData> VDataPtr =
           ValueProfData::serializeFrom(ProfileData.second);
       uint32_t S = VDataPtr->getSize();
       VDataPtr->swapBytesFromHost(ValueProfDataEndianness);
       Out.write((const char *)VDataPtr.get(), S);
     }
   }
 };
 }
 
 InstrProfWriter::InstrProfWriter(bool Sparse)
-    : Sparse(Sparse), FunctionData(), MaxFunctionCount(0),
+    : Sparse(Sparse), FunctionData(),
       InfoObj(new InstrProfRecordWriterTrait()) {}
 
 InstrProfWriter::~InstrProfWriter() { delete InfoObj; }
 
 // Internal interface for testing purpose only.
 void InstrProfWriter::setValueProfDataEndianness(
     support::endianness Endianness) {
   InfoObj->ValueProfDataEndianness = Endianness;
if (Weight > 1)
       Result = Dest.scale(Weight);
   } else {
     // We're updating a function we've seen before.
     Result = Dest.merge(I, Weight);
   }
 
   Dest.sortValueData();
 
-  // We keep track of the max function count as we go for simplicity.
-  // Update this statistic no matter the result of the merge.
-  if (Dest.Counts[0] > MaxFunctionCount)
-    MaxFunctionCount = Dest.Counts[0];
-
   return Result;
 }
 
 bool InstrProfWriter::shouldEncodeData(const ProfilingData &PD) {
   if (!Sparse)
     return true;
   for (const auto &Func : PD) {
     const InstrProfRecord &IPR = Func.second;
     if (std::any_of(IPR.Counts.begin(), IPR.Counts.end(),
                     [](uint64_t Count) { return Count > 0; }))
       return true;
   }
   return false;
 }
 
+static void setSummary(IndexedInstrProf::Summary *TheSummary,
+                       ProfileSummary &PS) {
+  using namespace IndexedInstrProf;
+  std::vector<ProfileSummaryEntry> &Res = PS.getDetailedSummary();
+  TheSummary->NumSummaryFields = Summary::NumKinds;
+  TheSummary->NumCutoffEntries = Res.size();
+  TheSummary->set(Summary::MaxFunctionCount, PS.getMaxFunctionCount());
+  TheSummary->set(Summary::MaxBlockCount, PS.getMaxBlockCount());
+  TheSummary->set(Summary::MaxInternalBlockCount,
+                  PS.getMaxInternalBlockCount());
+  TheSummary->set(Summary::TotalBlockCount, PS.getTotalCount());
+  TheSummary->set(Summary::TotalNumBlocks, PS.getNumBlocks());
+  TheSummary->set(Summary::TotalNumFunctions, PS.getNumFunctions());
+  for (unsigned I = 0; I < Res.size(); I++)
+    TheSummary->setEntry(I, Res[I]);
+}
+
 void InstrProfWriter::writeImpl(ProfOStream &OS) {
   OnDiskChainedHashTableGenerator<InstrProfRecordWriterTrait> Generator;
+
+  using namespace IndexedInstrProf;
+  std::vector<uint32_t> Cutoffs(&SummaryCutoffs[0],
+                                &SummaryCutoffs[NumSummaryCutoffs]);
+  ProfileSummary PS(Cutoffs);
+  InfoObj->TheProfileSummary = &PS;
+
   // Populate the hash table generator.
   for (const auto &I : FunctionData)
     if (shouldEncodeData(I.getValue()))
       Generator.insert(I.getKey(), &I.getValue());
   // Write the header.
   IndexedInstrProf::Header Header;
   Header.Magic = IndexedInstrProf::Magic;
   Header.Version = IndexedInstrProf::ProfVersion::CurrentVersion;
-  Header.MaxFunctionCount = MaxFunctionCount;
+  Header.Unused = 0;
   Header.HashType = static_cast<uint64_t>(IndexedInstrProf::HashType);
   Header.HashOffset = 0;
   int N = sizeof(IndexedInstrProf::Header) / sizeof(uint64_t);
 
   // Only write out all the fields execpt 'HashOffset'. We need
   // to remember the offset of that field to allow back patching
   // later.
   for (int I = 0; I < N - 1; I++)
     OS.write(reinterpret_cast<uint64_t *>(&Header)[I]);
 
-  // Save a space to write the hash table start location.
-  uint64_t HashTableStartLoc = OS.tell();
+  // Save the location of Header.HashOffset field in \c OS.
+  uint64_t HashTableStartFieldOffset = OS.tell();
   // Reserve the space for HashOffset field.
   OS.write(0);
+
+  // Reserve space to write profile summary data.
+  uint32_t NumEntries = Cutoffs.size();
+  uint32_t SummarySize = Summary::getSize(Summary::NumKinds, NumEntries);
+  // Remember the summary offset.
+  uint64_t SummaryOffset = OS.tell();
+  for (unsigned I = 0; I < SummarySize / sizeof(uint64_t); I++)
+    OS.write(0);
+
   // Write the hash table.
   uint64_t HashTableStart = Generator.Emit(OS.OS, *InfoObj);
 
+  // Allocate space for data to be serialized out.
+  std::unique_ptr<IndexedInstrProf::Summary> TheSummary =
+      IndexedInstrProf::allocSummary(SummarySize);
+  // Compute the Summary and copy the data to the data
+  // structure to be serialized out (to disk or buffer).
+  setSummary(TheSummary.get(), PS);
+  InfoObj->TheProfileSummary = 0;
+
   // Now do the final patch:
-  PatchItem PatchItems[1] = {{HashTableStartLoc, &HashTableStart, 1}};
+  PatchItem PatchItems[] = {
+      // Patch the Header.HashOffset field.
+      {HashTableStartFieldOffset, &HashTableStart, 1},
+      // Patch the summary data.
+      {SummaryOffset, reinterpret_cast<uint64_t *>(TheSummary.get()),
+       (int)(SummarySize / sizeof(uint64_t))}};
   OS.patch(PatchItems, sizeof(PatchItems) / sizeof(*PatchItems));
 }
 
 void InstrProfWriter::write(raw_fd_ostream &OS) {
   // Write the hash table.
   ProfOStream POS(OS);
   writeImpl(POS);
 }

llvm/trunk/unittests/ProfileData/InstrProfTest.cpp

TEST_P(MaybeSparseInstrProfTest, get_function_counts) {
   std::error_code EC;
   EC = Reader->getFunctionCounts("foo", 0x5678, Counts);
   ASSERT_TRUE(ErrorEquals(instrprof_error::hash_mismatch, EC));
 
   EC = Reader->getFunctionCounts("bar", 0x1234, Counts);
   ASSERT_TRUE(ErrorEquals(instrprof_error::unknown_function, EC));
 }
 
+// Profile data is copied from general.proftext
+TEST_F(InstrProfTest, get_profile_summary) {
+  InstrProfRecord Record1("func1", 0x1234, {97531});
+  InstrProfRecord Record2("func2", 0x1234, {0, 0});
+  InstrProfRecord Record3("func3", 0x1234,
+                          {2305843009213693952, 1152921504606846976,
+                           576460752303423488, 288230376151711744,
+                           144115188075855872, 72057594037927936});
+  InstrProfRecord Record4("func4", 0x1234, {0});
+  Writer.addRecord(std::move(Record1));
+  Writer.addRecord(std::move(Record2));
+  Writer.addRecord(std::move(Record3));
+  Writer.addRecord(std::move(Record4));
+  auto Profile = Writer.writeBuffer();
+  readProfile(std::move(Profile));
+
+  ProfileSummary &PS = Reader->getSummary();
+  ASSERT_EQ(2305843009213693952U, PS.getMaxFunctionCount());
+  ASSERT_EQ(2305843009213693952U, PS.getMaxBlockCount());
+  ASSERT_EQ(10U, PS.getNumBlocks());
+  ASSERT_EQ(4539628424389557499U, PS.getTotalCount());
+  std::vector<ProfileSummaryEntry> &Details = PS.getDetailedSummary();
+  uint32_t Cutoff = 800000;
+  auto Predicate = [&Cutoff](const ProfileSummaryEntry &PE) {
+    return PE.Cutoff == Cutoff;
+  };
+  auto EightyPerc = std::find_if(Details.begin(), Details.end(), Predicate);
+  Cutoff = 900000;
+  auto NinetyPerc = std::find_if(Details.begin(), Details.end(), Predicate);
+  Cutoff = 950000;
+  auto NinetyFivePerc = std::find_if(Details.begin(), Details.end(), Predicate);
+  Cutoff = 990000;
+  auto NinetyNinePerc = std::find_if(Details.begin(), Details.end(), Predicate);
+  ASSERT_EQ(576460752303423488U, EightyPerc->MinBlockCount);
+  ASSERT_EQ(288230376151711744U, NinetyPerc->MinBlockCount);
+  ASSERT_EQ(288230376151711744U, NinetyFivePerc->MinBlockCount);
+  ASSERT_EQ(72057594037927936U, NinetyNinePerc->MinBlockCount);
+}
+
 TEST_P(MaybeSparseInstrProfTest, get_icall_data_read_write) {
   InstrProfRecord Record1("caller", 0x1234, {1, 2});
   InstrProfRecord Record2("callee1", 0x1235, {3, 4});
   InstrProfRecord Record3("callee2", 0x1235, {3, 4});
   InstrProfRecord Record4("callee3", 0x1235, {3, 4});
 
   // 4 value sites.
   Record1.reserveSites(IPVK_IndirectCallTarget, 4);