[PGO] differentiate FE instrumentation and IR level instrumentation profiles

[xur]

Summary

This patch uses one bit in profile version to differentiate FE instrumentation and IR level instrumentation profiles. It is based on the suggestion in RFC: PGO Late instrumentation for LLVM
(http://lists.llvm.org/pipermail/llvm-dev/2015-August/089058.html)

PGOInstrumenation now checks this bit to make sure it's IR level instrumentation profiles. PGOInstrumenation also creates a new COMDAT variable __llvm_profile_ir_level and sets the value to 1. This variable tells the profile runtime to set the right version in the generated profiles.

llvm-profdata also handles the profiles differently: mainly for the MaxFunctionCount. For FE profile, it only needs to find the max of the entry count (the first count in the function). For IR level profile, the entry count might not be available, we will set it as the maximum block count in the profile.

[davidxl]

include/llvm/ProfileData/InstrProfData.inc

709

uint64_t

710

Why bit 60?

Also I think we should reserve perhaps 8 bits in the version so that we can use the same format for other variants of profile (for instance completely strips-off name if the user does not care about it). For that, we need to define a set of masks:

#define VARIANT_MASKS_ALL 0xf000000000000000ULL
#define VARIANT_LATE_INTR_MASK 0x1ULL
#define GET_VERSION(V) ((V)&~VARIANT_MASKS)

715

Why do you need to unset it?

717

Also remember to sync up the copy in compiler-rt of this file

include/llvm/ProfileData/InstrProfReader.h

122

Text format does not have version. Make this a boolean flag.

139

Do not introduce this interface in the base class -- Version means different things across Raw and Indexed reader.

Simply add an interface to query if it is for late instr.

162

Ok to add this field in implementation -- add a comment that Version value here is a OR of mask and real version.

188

Do not expose version at interface level. Just 'isIRLevelProfile'

368–369

This interface needs to be changed to mask out the flag bit -- otherwise you change the semantics of the old interface (which fortunately is not yet used).

{ return GET_VERSION(Index->getVersion()); }

Add a new interface for isIRLevelProfiling.

include/llvm/ProfileData/InstrProfWriter.h

48–49

make IsIRInstr a member field of the writer so that you don't need to change interfaces here.

lib/ProfileData/InstrProfReader.cpp

112

Missing documentation.

121

Report error if V is not 0 nor 1.

316

Do not use 'UNSET'..., but use GET_VERSION macro I suggested.

630–631

GET_VERSION(...)

lib/ProfileData/InstrProfWriter.cpp

186

Check Dest's Counts with merged values.

232

It is actually more readable with direct IndexedInstrProf::version | IR_PROF_VARIANT_MASK

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

731

may be a helper function for this.

737

As it is shared, put the string macro in InstrProfData.inc -- see other section strings.

Also add an interface in InstrProf.h to retrieve the string -- do not hard code string here.

739

Why align 8?

741

No literal string

tools/llvm-profdata/llvm-profdata.cpp

130

Simplify the code like this:

1. introduce a interface in writer:

setIsIRLevelProfile (bool true|false); On writer construction, the value is 'unknown'. When the set interface is called, it will check if it is set and compare previous value with current value -- reports Error when incosistence is detected.

2. the code will be just: if (std::error_code EC = Writer.setIsIRLevelProfile(Reader->isIRLevelProfile()) exitWithError("....);

[silvas]

llvm-profdata also handles the profiles differently: mainly for the MaxFunctionCount. For FE profile, it only needs to find the max of the entry count (the first count in the function). For IR level profile, the entry count might not be available, we will set it as the maximum block count in the profile.

If I understand correctly, the motivation is that some functions might have already been inlined when IR-level instrumentation happens, and so we assume that any individual basic block could have been from an inlined function, so to recover that "function"'s count, we take the highest basic block count. This doesn't make sense. I expect MaxFunctionCount to be the maximum function count *at the point where the instrumentation was done*. And the instrumentation (whether Clang's stuff or the IR-level stuff) puts in an entry count on every function it sees at the point where the instrumentation was done.

An example of why this matters is that many of my customers have some extremely hot functions like operator+ on a Vec4 class. A big advantage of the advantage of the IR-level instrumentation (especially early-inlining) is to be able to get rid of these functions before the instrumentation happens. I want MaxFunctionCount to represent the maximum function count *after* early inlining has cleaned up all the trivial inlinable functions. Or, to look at it from a different perspective, my clients use Vec4 as much as int; we do not want to count operator+ on Vec4 for the same reason we do not keep a count for the built-in operator+ on int, and we do not want MaxFunctionCount to be skewed by it. Early inlining is an elegant solution. I consider the current behavior a feature.

Or to put it another way, I think that my suggestion of "simply do not instrument the top 1% hottest functions" was a potential "quick fix", but the early inlining (+ other early optimizations) is a much more elegant and complete solution. Once we have characterized the benefits of early optimizations we will be able to more clearly evaluate whether we need to avoid instrumenting the hottest functions.

Right now, I think we should focus on integration into clang, since that is necessary for characterizing the effects of early optimizations so that we can decide on a good set of "early" passes to run to minimize instrumentation overhead. (I am glad to help out once it has been integrated into clang)

[xur]

Integrated David's comments.

Thanks,

-Rong

[davidxl]

include/llvm/ProfileData/InstrProfData.inc

709

typo : uint64_t, not uint_64_t.

719

Use existing macro: INSTR_PROF_QUOTE, no need for a new helper macro.

include/llvm/ProfileData/InstrProfReader.h

162

Raw --> raw

339–345

do not remove override here -- note that getVersion here is for indexed format class hierarchy (to support different indexed formats ).

Note that Version really means format version, so the masking needs to be applied here too.

Add another interface to check isIRLevelProfiling.

339–345

mask out the variant bits

368–369

Remove 'Note that is a unmasked version' -- it is confusing.

369

No need for GET_VERSION here -- it needs to be applied by the callee

371

return Index->isIRLevelProfile();

include/llvm/ProfileData/InstrProfWriter.h

33

PF_None --> PF_Unknown

lib/ProfileData/InstrProfReader.cpp

117

Why not skipping comments and empty lines first:

// Skip empty lines and comments.

while (!Line.is_at_end() && (Line->empty() || Line->startswith("#")))
  ++Line;

119

if (!Line->startsWith(":")) {

isIRLevelProfile = false;
return success();

}

tools/llvm-profdata/llvm-profdata.cpp

132

Add a test case for the error.

[xur]

lib/ProfileData/InstrProfReader.cpp

117

current code allows multiple instances of ":" line and the mixing of comments in between (late ones overrides the earlier ones).
if you think we should only allow one. We can simplify the code like you suggested.

119

same as above. current code allows multiple ":" lines.

[davidxl]

lib/ProfileData/InstrProfReader.cpp

117

We don't want that kind of flexibility for the header structure. I would even suggest making it more strict -- the header lines (if there are any) must start at the first line of the file -- but I won't insist on that.

[xur]

updated the patch with David's new comments.

[davidxl]

As a follow up, there should also be a change in Clang FE that checks IR level profile is not misused by FE.

include/llvm/ProfileData/InstrProfReader.h

130–131

Missing new field initialization.

include/llvm/ProfileData/InstrProfWriter.h

33

Fix format.

[slingn]

include/llvm/ProfileData/InstrProfWriter.h

61

Is it possible to determine ProfileKind (and IsIRLevel) at construction time? If it is, that seems like it would be less error prone than inferring it from IsIRLevel / leaving as PF_Unknown if setIsIRLevelProfile().

lib/ProfileData/InstrProfReader.cpp

113

If the text format is intended to be mostly human readable it would be nice to use something like ':irlevel' rather than a number since that requires referring to the code.

130

Does this handle the case where V could not be parsed as an integer - i.e. getAsInteger() returns true to indicate an error?

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

716

minor: 'to let runtime' -> 'to make the runtime'

test/Transforms/PGOProfile/Inputs/branch1.proftext

2

typo: flat -> flag

A few more of the same in the other *.proftext files.

[xur]

include/llvm/ProfileData/InstrProfWriter.h

61

in current profile merge, we generate the writer before reading the profile. So it's difficult to set the ProfileKind in the construct. We could sink the code of creation of the writer into the loop that reading the profiles (after reading the first one). But I'm not sure that coding style is good.

lib/ProfileData/InstrProfReader.cpp

113

The advantage of integer is that we can encode other profile variant into one number. As of now, this is the only variant but it's likely we will have more. Also note that usually the text format is generated by llvm-profdata in which case, we do have a comment to explain the the meaning of ":1".

130

Good catch. Fixed in the new patch.

[xur]

Integrated the latest review comments from David and Nathan.

Thanks!

-Rong

[davidxl]

This version looks good to me, but please wait for Justin's feedback on the approach in general.

[xur]

rebased the patch with recent change. Also have the following changes:
(1) directly generate __llvm_profile_raw_version variable so that no compile_rt change is needed.
(2) use ":ir or :fe" instead of a number as a ProfileKind flag in text profile.

[xur]

Ping.

[davidxl]

After this patch is done, the FE also needs to be changed to recognize IR profile and do the right handling.

include/llvm/ProfileData/InstrProfData.inc

706

Please rebase the patch. This part is already in tree.

include/llvm/ProfileData/InstrProfReader.h

162

Change the comments to:

The value of the version field of the raw profile data header. The lower 56 bits specifies the format version and the most significant 8 bits specify the variant types of the profile.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

771

This warning message needs to be revisited later once the clang option is finalized. Please add a TODO here.

[xur]

rebased the patch and integrated David's most recent comments.

Thanks,

-Rong

[davidxl]

lib/ProfileData/InstrProfWriter.cpp

184

MaxFunctionCount field is now removed (replaced by profileSummary) -- so this change can now be removed.

[xur]

updated patch after synced to MaxFunctionCount cleanup commits by David

[davidxl]

Add a negative test case with error when FE based profile is used in IR level profile-use.

[xur]

remove one unused function in InstrProfWrite in last revision.

[davidxl]

The latest patch uploaded does not look complete .

[xur]

re-upload the patch.

[davidxl]

LGTM

include/llvm/ProfileData/InstrProfData.inc

 #define INSTR_PROF_RAW_VERSION 2
 #define INSTR_PROF_INDEX_VERSION 4
 #define INSTR_PROF_COVMAP_VERSION 0
 
 /* Profile version is always of type uint64_t. Reserve the upper 8 bits in the
  * version for other variants of profile. We set the lowest bit of the upper 8
  * bits (i.e. bit 56) to 1 to indicate if this is an IR-level instrumentaiton
  * generated profile, and 0 if this is a Clang FE generated profile.
  */
 #define VARIANT_MASKS_ALL 0xff00000000000000ULL
 #define GET_VERSION(V) ((V) & ~VARIANT_MASKS_ALL)
+#define VARIANT_MASK_IR_PROF (0x1ULL << 56)
+#define IR_LEVEL_PROF_VERSION_VAR __llvm_profile_raw_version
 
 /* Runtime section names and name strings.  */
 #define INSTR_PROF_DATA_SECT_NAME __llvm_prf_data
 #define INSTR_PROF_NAME_SECT_NAME __llvm_prf_names
 #define INSTR_PROF_CNTS_SECT_NAME __llvm_prf_cnts
 #define INSTR_PROF_COVMAP_SECT_NAME __llvm_covmap
 
 #define INSTR_PROF_DATA_SECT_NAME_STR                                          \
   INSTR_PROF_QUOTE(INSTR_PROF_DATA_SECT_NAME)
 #define INSTR_PROF_NAME_SECT_NAME_STR                                          \
   INSTR_PROF_QUOTE(INSTR_PROF_NAME_SECT_NAME)
 #define INSTR_PROF_CNTS_SECT_NAME_STR                                          \
   INSTR_PROF_QUOTE(INSTR_PROF_CNTS_SECT_NAME)
 #define INSTR_PROF_COVMAP_SECT_NAME_STR                                        \
   INSTR_PROF_QUOTE(INSTR_PROF_COVMAP_SECT_NAME)
 
 /* Macros to define start/stop section symbol for a given
  * section on Linux. For instance
  * INSTR_PROF_SECT_START(INSTR_PROF_DATA_SECT_NAME) will
  * expand to __start___llvm_prof_data

include/llvm/ProfileData/InstrProfReader.h

public:
 
   /// Read the header.  Required before reading first record.
   virtual std::error_code readHeader() = 0;
   /// Read a single record.
   virtual std::error_code readNextRecord(InstrProfRecord &Record) = 0;
   /// Iterator over profile data.
   InstrProfIterator begin() { return InstrProfIterator(this); }
   InstrProfIterator end() { return InstrProfIterator(); }
+  virtual bool isIRLevelProfile() const = 0;
 
   /// Return the PGO symtab. There are three different readers:
   /// Raw, Text, and Indexed profile readers. The first two types
   /// of readers are used only by llvm-profdata tool, while the indexed
   /// profile reader is also used by llvm-cov tool and the compiler (
   /// backend or frontend). Since creating PGO symtab can create
   /// significant runtime and memory overhead (as it touches data
   /// for the whole program), InstrProfSymtab for the indexed profile

 /// Each record consists of a function name, a function hash, a number of
 /// counters, and then each counter value, in that order.
 class TextInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   /// Iterator over the profile data.
   line_iterator Line;
+  bool IsIRLevelProfile;
 
   TextInstrProfReader(const TextInstrProfReader &) = delete;
   TextInstrProfReader &operator=(const TextInstrProfReader &) = delete;
   std::error_code readValueProfileData(InstrProfRecord &Record);
 
 public:
   TextInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer_)
-      : DataBuffer(std::move(DataBuffer_)), Line(*DataBuffer, true, '#') {}
+      : DataBuffer(std::move(DataBuffer_)), Line(*DataBuffer, true, '#'),
+        IsIRLevelProfile(false) {}
 
   /// Return true if the given buffer is in text instrprof format.
   static bool hasFormat(const MemoryBuffer &Buffer);
 
+  bool isIRLevelProfile() const override { return IsIRLevelProfile; }
+
   /// Read the header.
   std::error_code readHeader() override;
   /// Read a single record.
   std::error_code readNextRecord(InstrProfRecord &Record) override;
 
   InstrProfSymtab &getSymtab() override {
     assert(Symtab.get());
     return *Symtab.get();
   }
 };
 
 /// Reader for the raw instrprof binary format from runtime.
 ///
 /// This format is a raw memory dump of the instrumentation-baed profiling data
 /// from the runtime.  It has no index.
 ///
 /// Templated on the unsigned type whose size matches pointers on the platform
 /// that wrote the profile.
 template <class IntPtrT>
 class RawInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   bool ShouldSwapBytes;
+  // The value of the version field of the raw profile data header. The lower 56
+  // bits specifies the format version and the most significant 8 bits specify
+  // the variant types of the profile.
+  uint64_t Version;
   uint64_t CountersDelta;
   uint64_t NamesDelta;
   const RawInstrProf::ProfileData<IntPtrT> *Data;
   const RawInstrProf::ProfileData<IntPtrT> *DataEnd;
   const uint64_t *CountersStart;
   const char *NamesStart;
   const uint8_t *ValueDataStart;
   const char *ProfileEnd;
   uint32_t ValueKindLast;
   uint32_t CurValueDataSize;
 
   InstrProfRecord::ValueMapType FunctionPtrToNameMap;
 
   RawInstrProfReader(const RawInstrProfReader &) = delete;
   RawInstrProfReader &operator=(const RawInstrProfReader &) = delete;
 public:
   RawInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer)
       : DataBuffer(std::move(DataBuffer)) { }
 
   static bool hasFormat(const MemoryBuffer &DataBuffer);
   std::error_code readHeader() override;
   std::error_code readNextRecord(InstrProfRecord &Record) override;
+  bool isIRLevelProfile() const override {
+    return (Version & VARIANT_MASK_IR_PROF) != 0;
+  }
 
   InstrProfSymtab &getSymtab() override {
     assert(Symtab.get());
     return *Symtab.get();
   }
 
 private:
   void createSymtab(InstrProfSymtab &Symtab);
struct InstrProfReaderIndexBase {
   // Read all the profile records with the key equal to FuncName
   virtual std::error_code getRecords(StringRef FuncName,
                                      ArrayRef<InstrProfRecord> &Data) = 0;
   virtual void advanceToNextKey() = 0;
   virtual bool atEnd() const = 0;
   virtual void setValueProfDataEndianness(support::endianness Endianness) = 0;
   virtual ~InstrProfReaderIndexBase() {}
   virtual uint64_t getVersion() const = 0;
+  virtual bool isIRLevelProfile() const = 0;
   virtual void populateSymtab(InstrProfSymtab &) = 0;
 };
 
 typedef OnDiskIterableChainedHashTable<InstrProfLookupTrait>
     OnDiskHashTableImplV3;
 
 template <typename HashTableImpl>
 class InstrProfReaderIndex : public InstrProfReaderIndexBase {
public:
   void advanceToNextKey() override { RecordIterator++; }
   bool atEnd() const override {
     return RecordIterator == HashTable->data_end();
   }
   void setValueProfDataEndianness(support::endianness Endianness) override {
     HashTable->getInfoObj().setValueProfDataEndianness(Endianness);
   }
   ~InstrProfReaderIndex() override {}
-  uint64_t getVersion() const override { return FormatVersion; }
+  uint64_t getVersion() const override { return GET_VERSION(FormatVersion); }
+  bool isIRLevelProfile() const override {
+    return (FormatVersion & VARIANT_MASK_IR_PROF) != 0;
+  }
   void populateSymtab(InstrProfSymtab &Symtab) override {
     Symtab.create(HashTable->keys());
   }
 };
 
 /// Reader for the indexed binary instrprof format.
 class IndexedInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
   /// The index into the profile data.
   std::unique_ptr<InstrProfReaderIndexBase> Index;
   /// Profile summary data.
   std::unique_ptr<ProfileSummary> Summary;
 
   IndexedInstrProfReader(const IndexedInstrProfReader &) = delete;
   IndexedInstrProfReader &operator=(const IndexedInstrProfReader &) = delete;
 
   // Read the profile summary. Return a pointer pointing to one byte past the
   // end of the summary data if it exists or the input \c Cur.
   const unsigned char *readSummary(IndexedInstrProf::ProfVersion Version,
                                    const unsigned char *Cur);
 
 public:
+  /// Return the profile version.
   uint64_t getVersion() const { return Index->getVersion(); }
+  bool isIRLevelProfile() const override { return Index->isIRLevelProfile(); }
   IndexedInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer)
       : DataBuffer(std::move(DataBuffer)), Index(nullptr) {}
 
   /// Return true if the given buffer is in an indexed instrprof format.
   static bool hasFormat(const MemoryBuffer &DataBuffer);
 
   /// Read the file header.
   std::error_code readHeader() override;
   /// Read a single record.
   std::error_code readNextRecord(InstrProfRecord &Record) override;

include/llvm/ProfileData/InstrProfWriter.h

 
 /// Writer for instrumentation based profile data.
 class ProfOStream;
 class InstrProfRecordWriterTrait;
 
 class InstrProfWriter {
 public:
   typedef SmallDenseMap<uint64_t, InstrProfRecord, 1> ProfilingData;
+  enum ProfKind { PF_Unknown = 0, PF_FE, PF_IRLevel };
 
 private:
   bool Sparse;
   StringMap<ProfilingData> FunctionData;
+  ProfKind ProfileKind;
   // Use raw pointer here for the incomplete type object.
   InstrProfRecordWriterTrait *InfoObj;
 
 public:
   InstrProfWriter(bool Sparse = false);
   ~InstrProfWriter();
 
   /// Add function counts for the given function. If there are already counts
   /// for this function and the hash and number of counts match, each counter is
   /// summed. Optionally scale counts by \p Weight.
   std::error_code addRecord(InstrProfRecord &&I, uint64_t Weight = 1);
   /// Write the profile to \c OS
   void write(raw_fd_ostream &OS);
   /// Write the profile in text format to \c OS
   void writeText(raw_fd_ostream &OS);
   /// Write \c Record in text format to \c OS
   static void writeRecordInText(const InstrProfRecord &Record,
                                 InstrProfSymtab &Symtab, raw_fd_ostream &OS);
   /// Write the profile, returning the raw data. For testing.
   std::unique_ptr<MemoryBuffer> writeBuffer();
 
+  /// Set the ProfileKind. Report error if mixing FE and IR level profiles.
+  std::error_code setIsIRLevelProfile(bool IsIRLevel) {
+    if (ProfileKind == PF_Unknown) {
+      ProfileKind = IsIRLevel ? PF_IRLevel: PF_FE;
+      return instrprof_error::success;
+    }
+    return (IsIRLevel == (ProfileKind == PF_IRLevel)) ?
+           instrprof_error::success : instrprof_error::unsupported_version;
+  }
+
+  /// Return if the profile is generated by IR level instrumentation.
+  bool IsIRLevelProfile() const { return ProfileKind == PF_IRLevel; }
+
   // Internal interface for testing purpose only.
   void setValueProfDataEndianness(support::endianness Endianness);
   void setOutputSparse(bool Sparse);
 
 private:
   bool shouldEncodeData(const ProfilingData &PD);
   void writeImpl(ProfOStream &OS);
 };
 
 } // end namespace llvm
 
 #endif

lib/ProfileData/InstrProfReader.cpp

bool TextInstrProfReader::hasFormat(const MemoryBuffer &Buffer) {
   // 'reasonable' number of characters (up to profile magic size).
   size_t count = std::min(Buffer.getBufferSize(), sizeof(uint64_t));
   StringRef buffer = Buffer.getBufferStart();
   return count == 0 ||
          std::all_of(buffer.begin(), buffer.begin() + count,
                      [](char c) { return ::isprint(c) || ::isspace(c); });
 }
 
+// Read the profile variant flag from the header: ":FE" means this is a FE
+// generated profile. ":IR" means this is an IR level profile. Other strings
+// with a leading ':' will be reported an error format.
 std::error_code TextInstrProfReader::readHeader() {
   Symtab.reset(new InstrProfSymtab());
+  bool IsIRInstr = false;
+  if (!Line->startswith(":")) {
+    IsIRLevelProfile = false;
+    return success();
+  }
+  StringRef Str = (Line)->substr(1);
+  if (Str.equals_lower("ir"))
+    IsIRInstr = true;
+  else if (Str.equals_lower("fe"))
+    IsIRInstr = false;
+  else
+    return instrprof_error::bad_header;
+
+  ++Line;
+  IsIRLevelProfile = IsIRInstr;
   return success();
 }
 
 std::error_code
 TextInstrProfReader::readValueProfileData(InstrProfRecord &Record) {
 
 #define CHECK_LINE_END(Line)                                                   \
   if (Line.is_at_end())                                                        \
for (const RawInstrProf::ProfileData<IntPtrT> *I = Data; I != DataEnd; ++I) {
     Symtab.mapAddress(FPtr, IndexedInstrProf::ComputeHash(FunctionName));
   }
   Symtab.finalizeSymtab();
 }
 
 template <class IntPtrT>
 std::error_code
 RawInstrProfReader<IntPtrT>::readHeader(const RawInstrProf::Header &Header) {
-  if (swap(Header.Version) != RawInstrProf::Version)
+  Version = swap(Header.Version);
+  if (GET_VERSION(Version) != RawInstrProf::Version)
     return error(instrprof_error::unsupported_version);
 
   CountersDelta = swap(Header.CountersDelta);
   NamesDelta = swap(Header.NamesDelta);
   auto DataSize = swap(Header.DataSize);
   auto CountersSize = swap(Header.CountersSize);
   auto NamesSize = swap(Header.NamesSize);
   auto ValueDataSize = swap(Header.ValueDataSize);
data_type InstrProfLookupTrait::ReadData(StringRef K, const unsigned char *D,
   using namespace support;
   const unsigned char *End = D + N;
   while (D < End) {
     // Read hash.
     if (D + sizeof(uint64_t) >= End)
       return data_type();
     uint64_t Hash = endian::readNext<uint64_t, little, unaligned>(D);
 
-    // Initialize number of counters for FormatVersion == 1.
+    // Initialize number of counters for GET_VERSION(FormatVersion) == 1.
     uint64_t CountsSize = N / sizeof(uint64_t) - 1;
     // If format version is different then read the number of counters.
-    if (FormatVersion != IndexedInstrProf::ProfVersion::Version1) {
+    if (GET_VERSION(FormatVersion) != IndexedInstrProf::ProfVersion::Version1) {
       if (D + sizeof(uint64_t) > End)
         return data_type();
       CountsSize = endian::readNext<uint64_t, little, unaligned>(D);
     }
     // Read counter values.
     if (D + CountsSize * sizeof(uint64_t) > End)
       return data_type();
 
     CounterBuffer.clear();
     CounterBuffer.reserve(CountsSize);
     for (uint64_t J = 0; J < CountsSize; ++J)
       CounterBuffer.push_back(endian::readNext<uint64_t, little, unaligned>(D));
 
     DataBuffer.emplace_back(K, Hash, std::move(CounterBuffer));
 
     // Read value profiling data.
-    if (FormatVersion > IndexedInstrProf::ProfVersion::Version2 &&
+    if (GET_VERSION(FormatVersion) > IndexedInstrProf::ProfVersion::Version2 &&
         !readValueProfilingData(D, End)) {
       DataBuffer.clear();
       return data_type();
     }
   }
   return DataBuffer;
 }
 
std::error_code IndexedInstrProfReader::readHeader() {
 
   // Check the magic number.
   uint64_t Magic = endian::byte_swap<uint64_t, little>(Header->Magic);
   if (Magic != IndexedInstrProf::Magic)
     return error(instrprof_error::bad_magic);
 
   // Read the version.
   uint64_t FormatVersion = endian::byte_swap<uint64_t, little>(Header->Version);
-  if (FormatVersion > IndexedInstrProf::ProfVersion::CurrentVersion)
+  if (GET_VERSION(FormatVersion) >
+      IndexedInstrProf::ProfVersion::CurrentVersion)
     return error(instrprof_error::unsupported_version);
 
   Cur = readSummary((IndexedInstrProf::ProfVersion)FormatVersion, Cur);
 
   // Read the hash type and start offset.
   IndexedInstrProf::HashT HashType = static_cast<IndexedInstrProf::HashT>(
       endian::byte_swap<uint64_t, little>(Header->HashType));
   if (HashType > IndexedInstrProf::HashT::Last)

lib/ProfileData/InstrProfWriter.cpp

for (const auto &ProfileData : *V) {
       VDataPtr->swapBytesFromHost(ValueProfDataEndianness);
       Out.write((const char *)VDataPtr.get(), S);
     }
   }
 };
 }
 
 InstrProfWriter::InstrProfWriter(bool Sparse)
-    : Sparse(Sparse), FunctionData(),
+    : Sparse(Sparse), FunctionData(), ProfileKind(PF_Unknown),
       InfoObj(new InstrProfRecordWriterTrait()) {}
 
 InstrProfWriter::~InstrProfWriter() { delete InfoObj; }
 
 // Internal interface for testing purpose only.
 void InstrProfWriter::setValueProfDataEndianness(
     support::endianness Endianness) {
   InfoObj->ValueProfDataEndianness = Endianness;
if (Weight > 1)
       Result = Dest.scale(Weight);
   } else {
     // We're updating a function we've seen before.
     Result = Dest.merge(I, Weight);
   }
 
   Dest.sortValueData();
 
   return Result;
 }
 
 bool InstrProfWriter::shouldEncodeData(const ProfilingData &PD) {
   if (!Sparse)
     return true;
   for (const auto &Func : PD) {
     const InstrProfRecord &IPR = Func.second;
     if (std::any_of(IPR.Counts.begin(), IPR.Counts.end(),
                     [](uint64_t Count) { return Count > 0; }))
       return true;
void InstrProfWriter::writeImpl(ProfOStream &OS) {
 
   // Populate the hash table generator.
   for (const auto &I : FunctionData)
     if (shouldEncodeData(I.getValue()))
       Generator.insert(I.getKey(), &I.getValue());
   // Write the header.
   IndexedInstrProf::Header Header;
   Header.Magic = IndexedInstrProf::Magic;
   Header.Version = IndexedInstrProf::ProfVersion::CurrentVersion;
+  if (ProfileKind == PF_IRLevel)
+    Header.Version |= VARIANT_MASK_IR_PROF;
   Header.Unused = 0;
   Header.HashType = static_cast<uint64_t>(IndexedInstrProf::HashType);
   Header.HashOffset = 0;
   int N = sizeof(IndexedInstrProf::Header) / sizeof(uint64_t);
 
   // Only write out all the fields except 'HashOffset'. We need
   // to remember the offset of that field to allow back patching
   // later.
for (uint32_t S = 0; S < NS; S++) {
       }
     }
   }
 
   OS << "\n";
 }
 
 void InstrProfWriter::writeText(raw_fd_ostream &OS) {
+  if (ProfileKind == PF_IRLevel)
+    OS << "# IR level Instrumentation Flag\n:ir\n";
   InstrProfSymtab Symtab;
   for (const auto &I : FunctionData)
     if (shouldEncodeData(I.getValue()))
       Symtab.addFuncName(I.getKey());
   Symtab.finalizeSymtab();
 
   for (const auto &I : FunctionData)
     if (shouldEncodeData(I.getValue()))
       for (const auto &Func : I.getValue())
         writeRecordInText(Func.second, Symtab, OS);
 }

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

TI->setMetadata(llvm::LLVMContext::MD_prof,
                     MDB.createBranchWeights(Weights));
     DEBUG(dbgs() << "Weight is: ";
           for (const auto &W : Weights) { dbgs() << W << " "; }
           dbgs() << "\n";);
   }
 }
 } // end anonymous namespace
 
+// Create a COMDAT variable IR_LEVEL_PROF_VARNAME to make the runtime
+// aware this is an ir_level profile so it can set the version flag.
+static void createIRLevelProfileFlagVariable(Module &M) {
+  Type *IntTy64 = Type::getInt64Ty(M.getContext());
+  uint64_t ProfileVersion = (INSTR_PROF_RAW_VERSION | VARIANT_MASK_IR_PROF);
+  auto IRLevelVersionVariable =
+      new GlobalVariable(M, IntTy64, true, GlobalVariable::ExternalLinkage,
+                         Constant::getIntegerValue(IntTy64, APInt(64, ProfileVersion)),
+                         INSTR_PROF_QUOTE(IR_LEVEL_PROF_VERSION_VAR));
+  IRLevelVersionVariable->setVisibility(GlobalValue::DefaultVisibility);
+  IRLevelVersionVariable->setComdat(
+      M.getOrInsertComdat(StringRef(INSTR_PROF_QUOTE(IR_LEVEL_PROF_VERSION_VAR))));
+}
+
 bool PGOInstrumentationGen::runOnModule(Module &M) {
+  createIRLevelProfileFlagVariable(M);
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
     BranchProbabilityInfo *BPI =
         &(getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI());
     BlockFrequencyInfo *BFI =
         &(getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI());
     instrumentOneFunc(F, &M, BPI, BFI);
   }
   return true;
 }
 
 static void setPGOCountOnFunc(PGOUseFunc &Func,
                               IndexedInstrProfReader *PGOReader) {
   if (Func.readCounters(PGOReader)) {
     Func.populateCounters();
     Func.setBranchWeights();
   }
bool PGOInstrumentationUse::runOnModule(Module &M) {
   }
 
   PGOReader = std::move(ReaderOrErr.get());
   if (!PGOReader) {
     Ctx.diagnose(DiagnosticInfoPGOProfile(ProfileFileName.data(),
                                           "Cannot get PGOReader"));
     return false;
   }
+  // TODO: might need to change the warning once the clang option is finalized.
+  if (!PGOReader->isIRLevelProfile()) {
+    Ctx.diagnose(DiagnosticInfoPGOProfile(
+        ProfileFileName.data(), "Not an IR level instrumentation profile"));
+    return false;
+  }
+
 
   for (auto &F : M) {
     if (F.isDeclaration())
       continue;
     BranchProbabilityInfo *BPI =
         &(getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI());
     BlockFrequencyInfo *BFI =
         &(getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI());
     PGOUseFunc Func(F, &M, BPI, BFI);
     setPGOCountOnFunc(Func, PGOReader.get());
   }
   return true;
 }

test/Transforms/PGOProfile/Inputs/branch1.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 test_br_1
 25571299074
 2
 3
 2
 

test/Transforms/PGOProfile/Inputs/branch2.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 test_br_2
 29667547796
 2
 1
 1
 

test/Transforms/PGOProfile/Inputs/criticaledge.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 test_criticalEdge
 82323253069
 8
 2
 1
 2
 2
 0

test/Transforms/PGOProfile/Inputs/diag.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 foo
 12884999999
 1
 1
 

test/Transforms/PGOProfile/Inputs/landingpad.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 foo
 59130013419
 4
 3
 1
 2
 0
 
 bar
 24868915205
 2
 1
 2
 

test/Transforms/PGOProfile/Inputs/loop1.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 test_simple_for
 34137660316
 2
 96
 4
 

test/Transforms/PGOProfile/Inputs/loop2.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 test_nested_for
 53929068288
 3
 33
 10
 6
 

test/Transforms/PGOProfile/Inputs/switch.proftext

+# :ir is the flag to indicate this is IR level profile.
+:ir
 test_switch
 46200943743
 4
 0
 5
 2
 3
 

test/Transforms/PGOProfile/branch1.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/branch1.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_test_br_1 = private constant [9 x i8] c"test_br_1"
 
 define i32 @test_br_1(i32 %i) {
 entry:
 ; GEN: entry:
 ; GEN-NOT: llvm.instrprof.increment
   %cmp = icmp sgt i32 %i, 0
   br i1 %cmp, label %if.then, label %if.end

test/Transforms/PGOProfile/branch2.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/branch2.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_test_br_2 = private constant [9 x i8] c"test_br_2"
 
 define i32 @test_br_2(i32 %i) {
 entry:
 ; GEN: entry:
 ; GEN-NOT: llvm.instrprof.increment
   %cmp = icmp sgt i32 %i, 0
   br i1 %cmp, label %if.then, label %if.else

test/Transforms/PGOProfile/criticaledge.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/criticaledge.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_test_criticalEdge = private constant [17 x i8] c"test_criticalEdge"
 ; GEN: @__profn__stdin__bar = private constant [11 x i8] c"<stdin>:bar"
 
 define i32 @test_criticalEdge(i32 %i, i32 %j) {
 entry:
 ; CHECK: entry:
 ; GEN-NOT: call void @llvm.instrprof.increment
   switch i32 %i, label %sw.default [

test/Transforms/PGOProfile/landingpad.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/landingpad.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @val = global i32 0, align 4
 @_ZTIi = external constant i8*
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_bar = private constant [3 x i8] c"bar"
 ; GEN: @__profn_foo = private constant [3 x i8] c"foo"
 
 define i32 @bar(i32 %i) {
 entry:
 ; GEN: entry:
 ; GEN-NOT: call void @llvm.instrprof.increment
   %rem = srem i32 %i, 3

test/Transforms/PGOProfile/loop1.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/loop1.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_test_simple_for = private constant [15 x i8] c"test_simple_for"
 
 define i32 @test_simple_for(i32 %n) {
 entry:
 ; GEN: entry:
 ; GEN-NOT: call void @llvm.instrprof.increment
   br label %for.cond
 

test/Transforms/PGOProfile/loop2.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/loop2.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_test_nested_for = private constant [15 x i8] c"test_nested_for"
 
 define i32 @test_nested_for(i32 %r, i32 %s) {
 entry:
 ; GEN: entry:
 ; GEN-NOT: call void @llvm.instrprof.increment
   br label %for.cond.outer
 

test/Transforms/PGOProfile/single_bb.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_single_bb = private constant [9 x i8] c"single_bb"
 
 define i32 @single_bb() {
 entry:
 ; GEN: entry:
 ; GEN: call void @llvm.instrprof.increment(i8* getelementptr inbounds ([9 x i8], [9 x i8]* @__profn_single_bb, i32 0, i32 0), i64 12884901887, i32 1, i32 0)
   ret i32 0
 }

test/Transforms/PGOProfile/switch.ll

 ; RUN: opt < %s -pgo-instr-gen -S | FileCheck %s --check-prefix=GEN
 ; RUN: llvm-profdata merge %S/Inputs/switch.proftext -o %t.profdata
 ; RUN: opt < %s -pgo-instr-use -pgo-test-profile-file=%t.profdata -S | FileCheck %s --check-prefix=USE
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
+; GEN: $__llvm_profile_raw_version = comdat any
+; GEN: @__llvm_profile_raw_version = constant i64 72057594037927938, comdat
 ; GEN: @__profn_test_switch = private constant [11 x i8] c"test_switch"
 
 define void @test_switch(i32 %i) {
 entry:
 ; GEN: entry:
 ; GEN-NOT: call void @llvm.instrprof.increment
   switch i32 %i, label %sw.default [
     i32 1, label %sw.bb

tools/llvm-profdata/llvm-profdata.cpp

static void mergeInstrProfile(const WeightedFileVector &Inputs,
   InstrProfWriter Writer(OutputSparse);
   SmallSet<std::error_code, 4> WriterErrorCodes;
   for (const auto &Input : Inputs) {
     auto ReaderOrErr = InstrProfReader::create(Input.Filename);
     if (std::error_code ec = ReaderOrErr.getError())
       exitWithErrorCode(ec, Input.Filename);
 
     auto Reader = std::move(ReaderOrErr.get());
+    bool IsIRProfile = Reader->isIRLevelProfile();
+    if (Writer.setIsIRLevelProfile(IsIRProfile))
+      exitWithError("Merge IR generated profile with Clang generated profile.");
+
     for (auto &I : *Reader) {
       if (std::error_code EC = Writer.addRecord(std::move(I), Input.Weight)) {
         // Only show hint the first time an error occurs.
         bool firstTime = WriterErrorCodes.insert(EC).second;
         handleMergeWriterError(EC, Input.Filename, I.Name, firstTime);
       }
     }
     if (Reader->hasError())
static int showInstrProfile(std::string Filename, bool ShowCounts,
   if (ShowDetailedSummary && DetailedSummaryCutoffs.empty()) {
     Cutoffs = {800000, 900000, 950000, 990000, 999000, 999900, 999990};
   }
   ProfileSummary PS(Cutoffs);
   if (std::error_code EC = ReaderOrErr.getError())
     exitWithErrorCode(EC, Filename);
 
   auto Reader = std::move(ReaderOrErr.get());
+  bool IsIRInstr = Reader->isIRLevelProfile();
   size_t ShownFunctions = 0;
   for (const auto &Func : *Reader) {
     bool Show =
         ShowAllFunctions || (!ShowFunction.empty() &&
                              Func.Name.find(ShowFunction) != Func.Name.npos);
 
     bool doTextFormatDump = (Show && ShowCounts && TextFormat);
 
if (Show) {
 
       if (!ShownFunctions)
         OS << "Counters:\n";
 
       ++ShownFunctions;
 
       OS << "  " << Func.Name << ":\n"
          << "    Hash: " << format("0x%016" PRIx64, Func.Hash) << "\n"
-         << "    Counters: " << Func.Counts.size() << "\n"
-         << "    Function count: " << Func.Counts[0] << "\n";
+         << "    Counters: " << Func.Counts.size() << "\n";
+      if (!IsIRInstr)
+        OS << "    Function count: " << Func.Counts[0] << "\n";
 
       if (ShowIndirectCallTargets)
         OS << "    Indirect Call Site Count: "
            << Func.getNumValueSites(IPVK_IndirectCallTarget) << "\n";
 
       if (ShowCounts) {
         OS << "    Block counts: [";
-        for (size_t I = 1, E = Func.Counts.size(); I < E; ++I) {
-          OS << (I == 1 ? "" : ", ") << Func.Counts[I];
+        size_t Start = (IsIRInstr ? 0 : 1);
+        for (size_t I = Start, E = Func.Counts.size(); I < E; ++I) {
+          OS << (I == Start ? "" : ", ") << Func.Counts[I];
         }
         OS << "]\n";
       }
 
       if (ShowIndirectCallTargets) {
         InstrProfSymtab &Symtab = Reader->getSymtab();
         uint32_t NS = Func.getNumValueSites(IPVK_IndirectCallTarget);
         OS << "    Indirect Target Results: \n";