[PGO] differentiate FE instrumentation and IR level instrumentation profiles

[xur]

Summary

This patch uses one bit in profile version to differentiate FE instrumentation and IR level instrumentation profiles. It is based on the suggestion in RFC: PGO Late instrumentation for LLVM
(http://lists.llvm.org/pipermail/llvm-dev/2015-August/089058.html)

PGOInstrumenation now checks this bit to make sure it's IR level instrumentation profiles. PGOInstrumenation also creates a new COMDAT variable __llvm_profile_ir_level and sets the value to 1. This variable tells the profile runtime to set the right version in the generated profiles.

llvm-profdata also handles the profiles differently: mainly for the MaxFunctionCount. For FE profile, it only needs to find the max of the entry count (the first count in the function). For IR level profile, the entry count might not be available, we will set it as the maximum block count in the profile.

[davidxl]

include/llvm/ProfileData/InstrProfData.inc

681

uint64_t

682

Why bit 60?

Also I think we should reserve perhaps 8 bits in the version so that we can use the same format for other variants of profile (for instance completely strips-off name if the user does not care about it). For that, we need to define a set of masks:

#define VARIANT_MASKS_ALL 0xf000000000000000ULL
#define VARIANT_LATE_INTR_MASK 0x1ULL
#define GET_VERSION(V) ((V)&~VARIANT_MASKS)

687

Why do you need to unset it?

689

Also remember to sync up the copy in compiler-rt of this file

include/llvm/ProfileData/InstrProfReader.h

109

Text format does not have version. Make this a boolean flag.

128

Do not introduce this interface in the base class -- Version means different things across Raw and Indexed reader.

Simply add an interface to query if it is for late instr.

146

Ok to add this field in implementation -- add a comment that Version value here is a OR of mask and real version.

172

Do not expose version at interface level. Just 'isIRLevelProfile'

339–343

This interface needs to be changed to mask out the flag bit -- otherwise you change the semantics of the old interface (which fortunately is not yet used).

{ return GET_VERSION(Index->getVersion()); }

Add a new interface for isIRLevelProfiling.

include/llvm/ProfileData/InstrProfWriter.h

44–45

make IsIRInstr a member field of the writer so that you don't need to change interfaces here.

lib/ProfileData/InstrProfReader.cpp

112

Missing documentation.

121

Report error if V is not 0 nor 1.

297

Do not use 'UNSET'..., but use GET_VERSION macro I suggested.

596

GET_VERSION(...)

lib/ProfileData/InstrProfWriter.cpp

137

Check Dest's Counts with merged values.

159

It is actually more readable with direct IndexedInstrProf::version | IR_PROF_VARIANT_MASK

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

683

may be a helper function for this.

689

As it is shared, put the string macro in InstrProfData.inc -- see other section strings.

Also add an interface in InstrProf.h to retrieve the string -- do not hard code string here.

691

Why align 8?

693

No literal string

tools/llvm-profdata/llvm-profdata.cpp

130

Simplify the code like this:

1. introduce a interface in writer:

setIsIRLevelProfile (bool true|false); On writer construction, the value is 'unknown'. When the set interface is called, it will check if it is set and compare previous value with current value -- reports Error when incosistence is detected.

2. the code will be just: if (std::error_code EC = Writer.setIsIRLevelProfile(Reader->isIRLevelProfile()) exitWithError("....);

[silvas]

llvm-profdata also handles the profiles differently: mainly for the MaxFunctionCount. For FE profile, it only needs to find the max of the entry count (the first count in the function). For IR level profile, the entry count might not be available, we will set it as the maximum block count in the profile.

If I understand correctly, the motivation is that some functions might have already been inlined when IR-level instrumentation happens, and so we assume that any individual basic block could have been from an inlined function, so to recover that "function"'s count, we take the highest basic block count. This doesn't make sense. I expect MaxFunctionCount to be the maximum function count *at the point where the instrumentation was done*. And the instrumentation (whether Clang's stuff or the IR-level stuff) puts in an entry count on every function it sees at the point where the instrumentation was done.

An example of why this matters is that many of my customers have some extremely hot functions like operator+ on a Vec4 class. A big advantage of the advantage of the IR-level instrumentation (especially early-inlining) is to be able to get rid of these functions before the instrumentation happens. I want MaxFunctionCount to represent the maximum function count *after* early inlining has cleaned up all the trivial inlinable functions. Or, to look at it from a different perspective, my clients use Vec4 as much as int; we do not want to count operator+ on Vec4 for the same reason we do not keep a count for the built-in operator+ on int, and we do not want MaxFunctionCount to be skewed by it. Early inlining is an elegant solution. I consider the current behavior a feature.

Or to put it another way, I think that my suggestion of "simply do not instrument the top 1% hottest functions" was a potential "quick fix", but the early inlining (+ other early optimizations) is a much more elegant and complete solution. Once we have characterized the benefits of early optimizations we will be able to more clearly evaluate whether we need to avoid instrumenting the hottest functions.

Right now, I think we should focus on integration into clang, since that is necessary for characterizing the effects of early optimizations so that we can decide on a good set of "early" passes to run to minimize instrumentation overhead. (I am glad to help out once it has been integrated into clang)

[xur]

Integrated David's comments.

Thanks,

-Rong

[davidxl]

include/llvm/ProfileData/InstrProfData.inc

681

typo : uint64_t, not uint_64_t.

691

Use existing macro: INSTR_PROF_QUOTE, no need for a new helper macro.

include/llvm/ProfileData/InstrProfReader.h

146

Raw --> raw

322

do not remove override here -- note that getVersion here is for indexed format class hierarchy (to support different indexed formats ).

Note that Version really means format version, so the masking needs to be applied here too.

Add another interface to check isIRLevelProfiling.

322

mask out the variant bits

339

Remove 'Note that is a unmasked version' -- it is confusing.

340

No need for GET_VERSION here -- it needs to be applied by the callee

342

return Index->isIRLevelProfile();

include/llvm/ProfileData/InstrProfWriter.h

30

PF_None --> PF_Unknown

lib/ProfileData/InstrProfReader.cpp

117

Why not skipping comments and empty lines first:

// Skip empty lines and comments.

while (!Line.is_at_end() && (Line->empty() || Line->startswith("#")))
  ++Line;

119

if (!Line->startsWith(":")) {

isIRLevelProfile = false;
return success();

}

tools/llvm-profdata/llvm-profdata.cpp

132

Add a test case for the error.

[xur]

lib/ProfileData/InstrProfReader.cpp

117

current code allows multiple instances of ":" line and the mixing of comments in between (late ones overrides the earlier ones).
if you think we should only allow one. We can simplify the code like you suggested.

119

same as above. current code allows multiple ":" lines.

[davidxl]

lib/ProfileData/InstrProfReader.cpp

117

We don't want that kind of flexibility for the header structure. I would even suggest making it more strict -- the header lines (if there are any) must start at the first line of the file -- but I won't insist on that.

[xur]

updated the patch with David's new comments.

[davidxl]

As a follow up, there should also be a change in Clang FE that checks IR level profile is not misused by FE.

include/llvm/ProfileData/InstrProfReader.h

120

Missing new field initialization.

include/llvm/ProfileData/InstrProfWriter.h

30

Fix format.

[slingn]

include/llvm/ProfileData/InstrProfWriter.h

57

Is it possible to determine ProfileKind (and IsIRLevel) at construction time? If it is, that seems like it would be less error prone than inferring it from IsIRLevel / leaving as PF_Unknown if setIsIRLevelProfile().

lib/ProfileData/InstrProfReader.cpp

113

If the text format is intended to be mostly human readable it would be nice to use something like ':irlevel' rather than a number since that requires referring to the code.

130

Does this handle the case where V could not be parsed as an integer - i.e. getAsInteger() returns true to indicate an error?

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

668

minor: 'to let runtime' -> 'to make the runtime'

test/Transforms/PGOProfile/Inputs/branch1.proftext

2

typo: flat -> flag

A few more of the same in the other *.proftext files.

[xur]

include/llvm/ProfileData/InstrProfWriter.h

57

in current profile merge, we generate the writer before reading the profile. So it's difficult to set the ProfileKind in the construct. We could sink the code of creation of the writer into the loop that reading the profiles (after reading the first one). But I'm not sure that coding style is good.

lib/ProfileData/InstrProfReader.cpp

113

The advantage of integer is that we can encode other profile variant into one number. As of now, this is the only variant but it's likely we will have more. Also note that usually the text format is generated by llvm-profdata in which case, we do have a comment to explain the the meaning of ":1".

130

Good catch. Fixed in the new patch.

[xur]

Integrated the latest review comments from David and Nathan.

Thanks!

-Rong

[davidxl]

This version looks good to me, but please wait for Justin's feedback on the approach in general.

[xur]

rebased the patch with recent change. Also have the following changes:
(1) directly generate __llvm_profile_raw_version variable so that no compile_rt change is needed.
(2) use ":ir or :fe" instead of a number as a ProfileKind flag in text profile.

[xur]

Ping.

[davidxl]

After this patch is done, the FE also needs to be changed to recognize IR profile and do the right handling.

include/llvm/ProfileData/InstrProfData.inc

680

Please rebase the patch. This part is already in tree.

include/llvm/ProfileData/InstrProfReader.h

146

Change the comments to:

The value of the version field of the raw profile data header. The lower 56 bits specifies the format version and the most significant 8 bits specify the variant types of the profile.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

723

This warning message needs to be revisited later once the clang option is finalized. Please add a TODO here.

[xur]

rebased the patch and integrated David's most recent comments.

Thanks,

-Rong

[davidxl]

lib/ProfileData/InstrProfWriter.cpp

134

MaxFunctionCount field is now removed (replaced by profileSummary) -- so this change can now be removed.

[xur]

updated patch after synced to MaxFunctionCount cleanup commits by David

[davidxl]

Add a negative test case with error when FE based profile is used in IR level profile-use.

[xur]

remove one unused function in InstrProfWrite in last revision.

[davidxl]

The latest patch uploaded does not look complete .

[xur]

re-upload the patch.

[davidxl]

LGTM

include/llvm/ProfileData/InstrProfData.inc

#define INSTR_PROF_RAW_MAGIC_64 (uint64_t)255 << 56 | (uint64_t)'l' << 48 | \
        (uint64_t)'p' << 40 | (uint64_t)'r' << 32 | (uint64_t)'o' << 24 |  \
         (uint64_t)'f' << 16 | (uint64_t)'r' << 8 | (uint64_t)129
 #define INSTR_PROF_RAW_MAGIC_32 (uint64_t)255 << 56 | (uint64_t)'l' << 48 | \
        (uint64_t)'p' << 40 | (uint64_t)'r' << 32 | (uint64_t)'o' << 24 |  \
         (uint64_t)'f' << 16 | (uint64_t)'R' << 8 | (uint64_t)129
 
 /* Raw profile format version. */
 #define INSTR_PROF_RAW_VERSION 2
 
+/* Profile version is always of type uint_64_t. Reserve the upper 8 bits in the
+ * version for other variants of profile. We set the lowest bit of the upper 8
+ * bits (i.e. bit 56) to 1 to indicate if this is an IR-level instrumentaiton
+ * generated profile, and 0 if this is a Clang FE generated profile.
+ */
+#define VARIANT_MASKS_ALL 0xff00000000000000ULL
+#define VARIANT_MASK_IR_PROF (0x1ULL << 56)
+#define GET_VERSION(V) ((V)&~VARIANT_MASKS_ALL)
+#define IR_LEVEL_PROF_VARNAME __llvm_profile_ir_level
+#define IR_LEVEL_PROF_STRINGFY(s) #s
+#define IR_LEVEL_PROF_STRING(s) IR_LEVEL_PROF_STRINGFY(s)
+
 /* Runtime section names and name strings.  */
 #define INSTR_PROF_DATA_SECT_NAME __llvm_prf_data
 #define INSTR_PROF_NAME_SECT_NAME __llvm_prf_names
 #define INSTR_PROF_CNTS_SECT_NAME __llvm_prf_cnts
 
 #define INSTR_PROF_DATA_SECT_NAME_STR \
         INSTR_PROF_QUOTE(INSTR_PROF_DATA_SECT_NAME)
 #define INSTR_PROF_NAME_SECT_NAME_STR \

include/llvm/ProfileData/InstrProfReader.h

public:
 
   /// Read the header.  Required before reading first record.
   virtual std::error_code readHeader() = 0;
   /// Read a single record.
   virtual std::error_code readNextRecord(InstrProfRecord &Record) = 0;
   /// Iterator over profile data.
   InstrProfIterator begin() { return InstrProfIterator(this); }
   InstrProfIterator end() { return InstrProfIterator(); }
+  virtual bool isIRLevelProfile() const = 0;
 
  protected:
   /// Set the current std::error_code and return same.
   std::error_code error(std::error_code EC) {
     LastError = EC;
     return EC;
   }
 

 /// Each record consists of a function name, a function hash, a number of
 /// counters, and then each counter value, in that order.
 class TextInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   /// Iterator over the profile data.
   line_iterator Line;
+  bool IsIRLevelProfile;
 
   // String table for holding a unique copy of all the strings in the profile.
   InstrProfStringTable StringTable;
 
   TextInstrProfReader(const TextInstrProfReader &) = delete;
   TextInstrProfReader &operator=(const TextInstrProfReader &) = delete;
   std::error_code readValueProfileData(InstrProfRecord &Record);
 
 public:
   TextInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer_)
       : DataBuffer(std::move(DataBuffer_)), Line(*DataBuffer, true, '#') {}
 
   /// Return true if the given buffer is in text instrprof format.
   static bool hasFormat(const MemoryBuffer &Buffer);
 
+  bool isIRLevelProfile() const override { return IsIRLevelProfile; };
+
   /// Read the header.
-  std::error_code readHeader() override { return success(); }
+  std::error_code readHeader() override;
   /// Read a single record.
   std::error_code readNextRecord(InstrProfRecord &Record) override;
 };
 
 /// Reader for the raw instrprof binary format from runtime.
 ///
 /// This format is a raw memory dump of the instrumentation-baed profiling data
 /// from the runtime.  It has no index.
 ///
 /// Templated on the unsigned type whose size matches pointers on the platform
 /// that wrote the profile.
 template <class IntPtrT>
 class RawInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   bool ShouldSwapBytes;
+  // Version of the Raw profile. It contains the variant profile information.
+  uint64_t Version;
   uint64_t CountersDelta;
   uint64_t NamesDelta;
   const RawInstrProf::ProfileData<IntPtrT> *Data;
   const RawInstrProf::ProfileData<IntPtrT> *DataEnd;
   const uint64_t *CountersStart;
   const char *NamesStart;
   const uint8_t *ValueDataStart;
   const char *ProfileEnd;
   uint32_t ValueKindLast;
   uint32_t CurValueDataSize;
 
   // String table for holding a unique copy of all the strings in the profile.
   InstrProfStringTable StringTable;
   InstrProfRecord::ValueMapType FunctionPtrToNameMap;
 
   RawInstrProfReader(const RawInstrProfReader &) = delete;
   RawInstrProfReader &operator=(const RawInstrProfReader &) = delete;
 public:
   RawInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer)
       : DataBuffer(std::move(DataBuffer)) { }
 
   static bool hasFormat(const MemoryBuffer &DataBuffer);
   std::error_code readHeader() override;
   std::error_code readNextRecord(InstrProfRecord &Record) override;
+  bool isIRLevelProfile() const override {
+    return (Version & VARIANT_MASK_IR_PROF) != 0;
+  }
 
 private:
   std::error_code readNextHeader(const char *CurrentPos);
   std::error_code readHeader(const RawInstrProf::Header &Header);
   template <class IntT> IntT swap(IntT Int) const {
     return ShouldSwapBytes ? sys::getSwappedBytes(Int) : Int;
   }
   support::endianness getDataEndianness() const {
public:
   void advanceToNextKey() override { RecordIterator++; }
   bool atEnd() const override {
     return RecordIterator == HashTable->data_end();
   }
   void setValueProfDataEndianness(support::endianness Endianness) override {
     HashTable->getInfoObj().setValueProfDataEndianness(Endianness);
   }
   ~InstrProfReaderIndex() override {}
-  uint64_t getVersion() const override { return FormatVersion; }
+  uint64_t getVersion() const { return FormatVersion; }
 };
 
 /// Reader for the indexed binary instrprof format.
 class IndexedInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   /// The index into the profile data.
   std::unique_ptr<InstrProfReaderIndexBase> Index;
   /// The maximal execution count among all functions.
   uint64_t MaxFunctionCount;
 
   IndexedInstrProfReader(const IndexedInstrProfReader &) = delete;
   IndexedInstrProfReader &operator=(const IndexedInstrProfReader &) = delete;
 
 public:
-  uint64_t getVersion() const { return Index->getVersion(); }
+  /// Return the profile version. Note that is a unmasked version.
+  uint64_t getVersion() const { return GET_VERSION(Index->getVersion()); }
+  bool isIRLevelProfile() const override {
+    return (Index->getVersion() & VARIANT_MASK_IR_PROF) !=0;
+  }
   IndexedInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer)
       : DataBuffer(std::move(DataBuffer)), Index(nullptr) {}
 
   /// Return true if the given buffer is in an indexed instrprof format.
   static bool hasFormat(const MemoryBuffer &DataBuffer);
 
   /// Read the file header.
   std::error_code readHeader() override;

include/llvm/ProfileData/InstrProfWriter.h

 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
 
 /// Writer for instrumentation based profile data.
 class InstrProfWriter {
 public:
   typedef SmallDenseMap<uint64_t, InstrProfRecord, 1> ProfilingData;
+  enum ProfKind { PF_None = 0, PF_FE, PF_IRLevel };
 
 private:
   InstrProfStringTable StringTable;
   StringMap<ProfilingData> FunctionData;
   uint64_t MaxFunctionCount;
+  ProfKind ProfileKind;
 public:
-  InstrProfWriter() : MaxFunctionCount(0) {}
+  InstrProfWriter() : MaxFunctionCount(0), ProfileKind(PF_None) {}
 
   /// Update string entries in profile data with references to StringTable.
   void updateStringTableReferences(InstrProfRecord &I);
   /// Add function counts for the given function. If there are already counts
   /// for this function and the hash and number of counts match, each counter is
   /// summed. Optionally scale counts by \p Weight.
   std::error_code addRecord(InstrProfRecord &&I, uint64_t Weight = 1);
   /// Write the profile to \c OS
   void write(raw_fd_ostream &OS);
   /// Write the profile in text format to \c OS
   void writeText(raw_fd_ostream &OS);
   /// Write \c Record in text format to \c OS
   static void writeRecordInText(const InstrProfRecord &Record,
                                 raw_fd_ostream &OS);
   /// Write the profile, returning the raw data. For testing.
   std::unique_ptr<MemoryBuffer> writeBuffer();
 
+  /// Set the ProfileKind. Report error if mixing FE and IR level profiles.
+  std::error_code setIsIRLevelProfile(bool IsIRLevel) {
+    if (ProfileKind == PF_None) {
+      ProfileKind = IsIRLevel ? PF_IRLevel: PF_FE;
+      return instrprof_error::success;
+    }
+    return (IsIRLevel == (ProfileKind == PF_IRLevel)) ?
+           instrprof_error::success : instrprof_error::unsupported_version;
+  }
+
   // Internal interface for testing purpose only.
   void setValueProfDataEndianness(support::endianness Endianness);
 
 private:
   std::pair<uint64_t, uint64_t> writeImpl(raw_ostream &OS);
 };
 
 } // end namespace llvm
 
 #endif

lib/ProfileData/InstrProfReader.cpp

bool TextInstrProfReader::hasFormat(const MemoryBuffer &Buffer) {
   // 'reasonable' number of characters (up to profile magic size).
   size_t count = std::min(Buffer.getBufferSize(), sizeof(uint64_t));
   StringRef buffer = Buffer.getBufferStart();
   return count == 0 ||
          std::all_of(buffer.begin(), buffer.begin() + count,
                      [](char c) { return ::isprint(c) || ::isspace(c); });
 }
 
+// Read the profile variant flag from the header: ":0" means this is a FE
+// generated profile. ":1" means this is an IR level profile. Other strings
+// with a leading ':' will be reported an error format.
+std::error_code TextInstrProfReader::readHeader() {
+  bool IsIRInstr = false;
+  while (!Line.is_at_end() &&
+         (Line->empty() || Line->startswith("#") || Line->startswith(":"))) {
+    if (Line->startswith(":")) {
+      StringRef Str = (Line)->substr(1);
+      int32_t V = 0;
+      if (!Str.getAsInteger(0, V)) {
+        if (V == 1) 
+          IsIRInstr = true;
+        else if (V == 0)
+          IsIRInstr = false;
+        else
+          return instrprof_error::bad_header;
+      }
+    }
+    ++Line;
+  }
+  IsIRLevelProfile = IsIRInstr;
+  return success();
+}
+
 std::error_code
 TextInstrProfReader::readValueProfileData(InstrProfRecord &Record) {
 
 #define CHECK_LINE_END(Line)                                                   \
   if (Line.is_at_end())                                                        \
     return error(instrprof_error::truncated);
 #define READ_NUM(Str, Dst)                                                     \
   if ((Str).getAsInteger(10, (Dst)))                                           \
RawInstrProfReader<IntPtrT>::readNextHeader(const char *CurrentPos) {
   // There's another profile to read, so we need to process the header.
   auto *Header = reinterpret_cast<const RawInstrProf::Header *>(CurrentPos);
   return readHeader(*Header);
 }
 
 template <class IntPtrT>
 std::error_code RawInstrProfReader<IntPtrT>::readHeader(
     const RawInstrProf::Header &Header) {
-  if (swap(Header.Version) != RawInstrProf::Version)
+  Version = swap(Header.Version);
+  if (GET_VERSION(Version) != RawInstrProf::Version)
     return error(instrprof_error::unsupported_version);
 
   CountersDelta = swap(Header.CountersDelta);
   NamesDelta = swap(Header.NamesDelta);
   auto DataSize = swap(Header.DataSize);
   auto CountersSize = swap(Header.CountersSize);
   auto NamesSize = swap(Header.NamesSize);
   auto ValueDataSize = swap(Header.ValueDataSize);
data_type InstrProfLookupTrait::ReadData(StringRef K, const unsigned char *D,
   using namespace support;
   const unsigned char *End = D + N;
   while (D < End) {
     // Read hash.
     if (D + sizeof(uint64_t) >= End)
       return data_type();
     uint64_t Hash = endian::readNext<uint64_t, little, unaligned>(D);
 
-    // Initialize number of counters for FormatVersion == 1.
+    // Initialize number of counters for GET_VERSION(FormatVersion) == 1.
     uint64_t CountsSize = N / sizeof(uint64_t) - 1;
     // If format version is different then read the number of counters.
-    if (FormatVersion != 1) {
+    if (GET_VERSION(FormatVersion) != 1) {
       if (D + sizeof(uint64_t) > End)
         return data_type();
       CountsSize = endian::readNext<uint64_t, little, unaligned>(D);
     }
     // Read counter values.
     if (D + CountsSize * sizeof(uint64_t) > End)
       return data_type();
 
     CounterBuffer.clear();
     CounterBuffer.reserve(CountsSize);
     for (uint64_t J = 0; J < CountsSize; ++J)
       CounterBuffer.push_back(endian::readNext<uint64_t, little, unaligned>(D));
 
     DataBuffer.emplace_back(K, Hash, std::move(CounterBuffer));
 
     // Read value profiling data.
-    if (FormatVersion > 2 && !readValueProfilingData(D, End)) {
+    if (GET_VERSION(FormatVersion) > 2 && !readValueProfilingData(D, End)) {
       DataBuffer.clear();
       return data_type();
     }
   }
   return DataBuffer;
 }
 
 template <typename HashTableImpl>
std::error_code IndexedInstrProfReader::readHeader() {
 
   // Check the magic number.
   uint64_t Magic = endian::byte_swap<uint64_t, little>(Header->Magic);
   if (Magic != IndexedInstrProf::Magic)
     return error(instrprof_error::bad_magic);
 
   // Read the version.
   uint64_t FormatVersion = endian::byte_swap<uint64_t, little>(Header->Version);
-  if (FormatVersion > IndexedInstrProf::Version)
+  if (GET_VERSION(FormatVersion) > IndexedInstrProf::Version)
     return error(instrprof_error::unsupported_version);
 
   // Read the maximal function count.
   MaxFunctionCount =
       endian::byte_swap<uint64_t, little>(Header->MaxFunctionCount);
 
   // Read the hash type and start offset.
   IndexedInstrProf::HashT HashType = static_cast<IndexedInstrProf::HashT>(

lib/ProfileData/InstrProfWriter.cpp

if (NewFunc) {
     }
   } else {
     // We're updating a function we've seen before.
     Result = Dest.merge(I, Weight);
   }
 
   // We keep track of the max function count as we go for simplicity.
   // Update this statistic no matter the result of the merge.
+  if (ProfileKind != PF_IRLevel) {
     if (Dest.Counts[0] > MaxFunctionCount)
       MaxFunctionCount = Dest.Counts[0];
+  } else {
+      for (auto &IT : Dest.Counts)
+        if (IT > MaxFunctionCount)
+          MaxFunctionCount = IT;
+  }
 
   return Result;
 }
 
 std::pair<uint64_t, uint64_t> InstrProfWriter::writeImpl(raw_ostream &OS) {
   OnDiskChainedHashTableGenerator<InstrProfRecordTrait> Generator;
 
   // Populate the hash table generator.
   for (const auto &I : FunctionData)
     Generator.insert(I.getKey(), &I.getValue());
 
   using namespace llvm::support;
   endian::Writer<little> LE(OS);
 
   // Write the header.
   IndexedInstrProf::Header Header;
   Header.Magic = IndexedInstrProf::Magic;
+  if (ProfileKind == PF_IRLevel)
+    Header.Version = IndexedInstrProf::Version | VARIANT_MASK_IR_PROF;
+  else
     Header.Version = IndexedInstrProf::Version;
   Header.MaxFunctionCount = MaxFunctionCount;
   Header.HashType = static_cast<uint64_t>(IndexedInstrProf::HashType);
   Header.HashOffset = 0;
   int N = sizeof(IndexedInstrProf::Header) / sizeof(uint64_t);
 
   // Only write out all the fields execpt 'HashOffset'. We need
   // to remember the offset of that field to allow back patching
   // later.
for (uint32_t S = 0; S < NS; S++) {
       }
     }
   }
 
   OS << "\n";
 }
 
 void InstrProfWriter::writeText(raw_fd_ostream &OS) {
+  if (ProfileKind == PF_IRLevel)
+    OS << "# IR level Instrumentation Flag:\n:" << 1 << "\n";
   for (const auto &I : FunctionData)
     for (const auto &Func : I.getValue())
       writeRecordInText(Func.second, OS);
 }
 
 std::unique_ptr<MemoryBuffer> InstrProfWriter::writeBuffer() {
   std::string Data;
   llvm::raw_string_ostream OS(Data);

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

TI->setMetadata(llvm::LLVMContext::MD_prof,
                     MDB.createBranchWeights(Weights));
     DEBUG(dbgs() << "Weight is: ";
           for (const auto &W : Weights) { dbgs() << W << " "; }
           dbgs() << "\n";);
   }
 }
 } // end anonymous namespace
 
+// Create a COMDAT variable IR_LEVEL_PROF_VARNAME to let runtime
+// aware this is an ir_level profile so it can set the version flag.
+static void createIRLevelProfileFlagVariable(Module &M) {
+  Type *IntTy32 = Type::getInt32Ty(M.getContext());
+  auto IRLevelFlagVariable =
+      new GlobalVariable(M, IntTy32, true, GlobalVariable::ExternalLinkage,
+                         Constant::getIntegerValue(IntTy32, APInt(32, 1)),
+                         IR_LEVEL_PROF_STRING(IR_LEVEL_PROF_VARNAME));
+  IRLevelFlagVariable->setVisibility(GlobalValue::DefaultVisibility);
+  IRLevelFlagVariable->setComdat(
+      M.getOrInsertComdat(StringRef(
+          IR_LEVEL_PROF_STRING(IR_LEVEL_PROF_VARNAME))));
+}
+
 bool PGOInstrumentationGen::runOnModule(Module &M) {
+  createIRLevelProfileFlagVariable(M);
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
     BranchProbabilityInfo *BPI =
         &(getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI());
     BlockFrequencyInfo *BFI =
         &(getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI());
     instrumentOneFunc(F, &M, BPI, BFI);
   }
   return true;
 }
 
 static void setPGOCountOnFunc(PGOUseFunc &Func,
                               IndexedInstrProfReader *PGOReader) {
   if (Func.readCounters(PGOReader)) {
     Func.populateCounters();
     Func.setBranchWeights();
   }
bool PGOInstrumentationUse::runOnModule(Module &M) {
   }
 
   PGOReader = std::move(ReaderOrErr.get());
   if (!PGOReader) {
     Ctx.diagnose(DiagnosticInfoPGOProfile(ProfileFileName.data(),
                                           "Cannot get PGOReader"));
     return false;
   }
+  if (!PGOReader->isIRLevelProfile()) {
+    Ctx.diagnose(DiagnosticInfoPGOProfile(
+        ProfileFileName.data(), "Not an IR level instrumentation profile"));
+    return false;
+  }
+
 
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
     BranchProbabilityInfo *BPI =
         &(getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI());
     BlockFrequencyInfo *BFI =
         &(getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI());
     PGOUseFunc Func(F, &M, BPI, BFI);
     setPGOCountOnFunc(Func, PGOReader.get());
   }
   return true;
 }

test/Transforms/PGOProfile/Inputs/branch1.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 test_br_1
 25571299074
 2
 3
 2
 

test/Transforms/PGOProfile/Inputs/branch2.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 test_br_2
 29667547796
 2
 1
 1
 

test/Transforms/PGOProfile/Inputs/criticaledge.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 test_criticalEdge
 82323253069
 8
 2
 1
 2
 2
 0

test/Transforms/PGOProfile/Inputs/diag.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 foo
 12884999999
 1
 1
 

test/Transforms/PGOProfile/Inputs/landingpad.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 foo
 59130013419
 4
 3
 1
 2
 0
 
 bar
 24868915205
 2
 1
 2
 

test/Transforms/PGOProfile/Inputs/loop1.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 test_simple_for
 34137660316
 2
 96
 4
 

test/Transforms/PGOProfile/Inputs/loop2.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 test_nested_for
 53929068288
 3
 33
 10
 6
 

test/Transforms/PGOProfile/Inputs/switch.proftext

+# :1 is the flat to indicate this is IR level profile.
+:1
 test_switch
 46200943743
 4
 0
 5
 2
 3
 

tools/llvm-profdata/llvm-profdata.cpp

static void mergeInstrProfile(const WeightedFileVector &Inputs,
   InstrProfWriter Writer;
   SmallSet<std::error_code, 4> WriterErrorCodes;
   for (const auto &Input : Inputs) {
     auto ReaderOrErr = InstrProfReader::create(Input.Filename);
     if (std::error_code ec = ReaderOrErr.getError())
       exitWithErrorCode(ec, Input.Filename);
 
     auto Reader = std::move(ReaderOrErr.get());
+    bool IsIRProfile = Reader->isIRLevelProfile();
+    if (Writer.setIsIRLevelProfile(IsIRProfile))
+      exitWithError("Merge IR generated profile with Clang generated profile.");
+
     for (auto &I : *Reader) {
       if (std::error_code EC = Writer.addRecord(std::move(I), Input.Weight)) {
         // Only show hint the first time an error occurs.
         bool firstTime = WriterErrorCodes.insert(EC).second;
         handleMergeWriterError(EC, Input.Filename, I.Name, firstTime);
       }
     }
     if (Reader->hasError())
static int showInstrProfile(std::string Filename, bool ShowCounts,
                             bool ShowIndirectCallTargets, bool ShowAllFunctions,
                             std::string ShowFunction, bool TextFormat,
                             raw_fd_ostream &OS) {
   auto ReaderOrErr = InstrProfReader::create(Filename);
   if (std::error_code EC = ReaderOrErr.getError())
     exitWithErrorCode(EC, Filename);
 
   auto Reader = std::move(ReaderOrErr.get());
+  bool IsIRInstr = Reader->isIRLevelProfile();
   uint64_t MaxFunctionCount = 0, MaxBlockCount = 0;
   size_t ShownFunctions = 0, TotalFunctions = 0;
   for (const auto &Func : *Reader) {
     bool Show =
         ShowAllFunctions || (!ShowFunction.empty() &&
                              Func.Name.find(ShowFunction) != Func.Name.npos);
 
     bool doTextFormatDump = (Show && ShowCounts && TextFormat);
 
     if (doTextFormatDump) {
       InstrProfWriter::writeRecordInText(Func, OS);
       continue;
     }
 
     ++TotalFunctions;
     assert(Func.Counts.size() > 0 && "function missing entry counter");
+    if (!IsIRInstr) {
       if (Func.Counts[0] > MaxFunctionCount)
         MaxFunctionCount = Func.Counts[0];
+    }
 
     for (size_t I = 1, E = Func.Counts.size(); I < E; ++I) {
       if (Func.Counts[I] > MaxBlockCount)
         MaxBlockCount = Func.Counts[I];
     }
 
     if (Show) {
 
       if (!ShownFunctions)
         OS << "Counters:\n";
 
       ++ShownFunctions;
 
       OS << "  " << Func.Name << ":\n"
          << "    Hash: " << format("0x%016" PRIx64, Func.Hash) << "\n"
-         << "    Counters: " << Func.Counts.size() << "\n"
-         << "    Function count: " << Func.Counts[0] << "\n";
+         << "    Counters: " << Func.Counts.size() << "\n";
+      if (!IsIRInstr)
+        OS << "    Function count: " << Func.Counts[0] << "\n";
 
       if (ShowIndirectCallTargets)
         OS << "    Indirect Call Site Count: "
            << Func.getNumValueSites(IPVK_IndirectCallTarget) << "\n";
 
       if (ShowCounts) {
         OS << "    Block counts: [";
-        for (size_t I = 1, E = Func.Counts.size(); I < E; ++I) {
-          OS << (I == 1 ? "" : ", ") << Func.Counts[I];
+        size_t Start = (IsIRInstr ? 0 : 1);
+        for (size_t I = Start, E = Func.Counts.size(); I < E; ++I) {
+          OS << (I == Start ? "" : ", ") << Func.Counts[I];
         }
         OS << "]\n";
       }
 
       if (ShowIndirectCallTargets) {
         uint32_t NS = Func.getNumValueSites(IPVK_IndirectCallTarget);
         OS << "    Indirect Target Results: \n";
         for (size_t I = 0; I < NS; ++I) {
if (Reader->hasError())
     exitWithErrorCode(Reader->getError(), Filename);
 
   if (ShowCounts && TextFormat)
     return 0;
 
   if (ShowAllFunctions || !ShowFunction.empty())
     OS << "Functions shown: " << ShownFunctions << "\n";
   OS << "Total functions: " << TotalFunctions << "\n";
+  if (!IsIRInstr) {
     OS << "Maximum function count: " << MaxFunctionCount << "\n";
     OS << "Maximum internal block count: " << MaxBlockCount << "\n";
+  } else {
+    OS << "Maximum count: " << MaxBlockCount << "\n";
+  }
   return 0;
 }
 
 static int showSampleProfile(std::string Filename, bool ShowCounts,
                              bool ShowAllFunctions, std::string ShowFunction,
                              raw_fd_ostream &OS) {
   using namespace sampleprof;
   auto ReaderOrErr = SampleProfileReader::create(Filename, getGlobalContext());