[CSSPGO] Consume pseudo-probe-based AutoFDO profile

[hoy]

Summary

This change enables pseudo-probe-based sample counts to be consumed by the sample profile loader under the regular -fprofile-sample-use switch with minimal adjustments to the existing sample file formats. After the counts are imported, a probe helper, aka, a PseudoProbeManager object, is automatically launched to verify the CFG checksum of every function in the current compilation against the corresponding checksum from the profile. Mismatched checksums will cause a function profile to be slipped. A SampleProfileProber pass is scheduled before any of the SampleProfileLoader instances so that the CFG checksums as well as probe mappings are available during the profile loading time. The PseudoProbeManager object is set up right after the profile reading is done. In the future a CFG-based fuzzy matching could be done in PseudoProbeManager.

Samples will be applied only to pseudo probe instructions as well as probed callsites once the checksum verification goes through. Those instructions are processed in the same way that regular instructions would be processed in the line-number-based scenario. In other words, a function is processed in a regular way as if it was reduced to just containing pseudo probes (block probes and callsites).

Adjustment to profile format

A CFG checksum field is being added to the existing AutoFDO profile formats. So far only the text format and the extended binary format are supported. For the text format, a new line like

!CFGChecksum: 12345

is added to the end of the body sample lines. For the extended binary profile format, we introduce a metadata section to store the checksum map from function names to their CFG checksums.

[hoy]

Rebasing

[hoy]

Updating test.

[wmi]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

I guess it makes sense to treat missing entry to have zero count instead of unknown because the function profiles with unmatched CFGchecksum has been dropped, is that right?

If it is, can we check CFGchecksum not zero here instead of ProfileIsProbeBased or ProfileIsCS? When CFGchecksum is not zero and matched, we are confident the profile is accurate no matter what kind of profile it is.

Another thing is: I think debug info based profile can have a similar checksum then it can use a similar strategy here as probe based profile. It may not be a CFGchecksum because debug info based profile has no idea about CFG, so it is better to rename CFGchecksum to something more general.

675–677

Is it correct that for probe based profile all the FunctionHash should already be initialized to a non-zero value?

681–682

Item1 won't be an issue if -funique-internal-linkage-names becomes default (https://reviews.llvm.org/rGad1b9daa4bf40c1907794fd5de7807aad1f0553c).

llvm/lib/IR/PseudoProbe.cpp

38–44

Can you wrap them into a function like extractProbeFromDiscriminator?

llvm/lib/ProfileData/SampleProf.cpp

135

Should we output CFGchecksum if it is supported? It will affect the output of 'llvm-profdata show -sample'

llvm/lib/ProfileData/SampleProfReader.cpp

870

Nit: if (!ProfileIsProbeBased) return sampleprof_error::success;

llvm/lib/ProfileData/SampleProfWriter.cpp

171

Nit: if (!FunctionSamples::ProfileIsProbeBased) return sampleprof_error::success;

174

What about the case that the profile use MD5 to represent name?

llvm/tools/llvm-profdata/llvm-profdata.cpp

684–686

This should use exitWithError instead of assertion for user to get the error message in release mode, same as you did for compare below.

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

I think the comment I put here is a bit confusing. A missing entry for a probe here really means a missing counter. When a probe counter doesn't appear in a function profile, we are sure that the probe has not been executed (virtually). From our experiment, giving it a zero count is a bit more accurate instead of having the counts inference algorithm to figure out a count for it.

We use CFGchecksum to identify if a profile is debug-info-based or pseudo-probe-based. A probe-based profile will likely have a non-zero checksum due to the way CFGchecksum is computed. As you pointed out, when CFGchecksum is matched, we are confident the profile is accurate (even if there are missing counters for some probes). If not matched, the profile will be completely dropped and no counters will be applied to the function..

It's a good suggestion to apply a similar checksum for debug-info-based profile to tell how close the pervious source is to the current IR. We can discuss more about this.

675–677

A FunctionHash is SampleProfileProber::computeCFGHash which in theory could have zero value, though in practice we haven't seen that even with a single-block function.

681–682

Thanks for pointing this out! The switch will be helpful. Comment updated.

llvm/lib/IR/PseudoProbe.cpp

38–44

Done.

llvm/lib/ProfileData/SampleProf.cpp

135

Good point, added printing for CFGchecksum.

llvm/lib/ProfileData/SampleProfWriter.cpp

174

Good question. I thought this is orthogonal from MD5 but correct me if I'm wrong. Does a name index point to the name table that are filled with MD5 codes?

llvm/tools/llvm-profdata/llvm-profdata.cpp

684–686

Sounds good.

[hoy]

Addressing Wei's feedbacks.

[wmi]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

A missing entry for a probe here really means a missing counter.

Could you list the possible cases you have known where probe could be missing?

From our experiment, giving it a zero count is a bit more accurate instead of having the counts inference algorithm to figure out a count for it.

Yes, I agree with that. I just want to know whether it makes sense to check CFGChecksum match instead of checking the profile is probe based or context sensitive.

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

Yeah. A probe is deleted literally only if the corresponding block is really dead. Otherwise if the probe counter is not sampled, it means the block is there but just was not run. We trust this and assign a zero weight to the probe. There is a third case where a probe is sampled through its adjacent physical instruction but the samples are not trusted because the next physical instruction starts a new block. We call that case a "dangling" probe. This usually reflects that all instructions of a block are moved out of that block and only that probe is left there. For dangling probes we intentionally record a zero count in the profile instead of a missing counter. This helps the counts inference algorithm. Will have a patch to deal with optimizations that result in dangling probes.

I see. So far the check against probe-based is equivalent to checking against CFGChecksum, since that's the way to identify if a profile is probe-based or debug-info based. If we'd like to also employ a CFGChecksum for the debug-info scenario in future, it's reasonable to check CFGChecksum here.

[wmi]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

Is there case that optimizations after probe insertion create new BBs and the new BBs somehow don't have probes?

So far the check against probe-based is equivalent to checking against CFGChecksum, since that's the way to identify if a profile is probe-based or debug-info based.

Ok, it is fine to leave the change later.

llvm/lib/ProfileData/SampleProfWriter.cpp

174

Ah, sorry, never mind. It is writing name index intead of name here so it is fine.

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

Given a second thought, I'm not sure how we'd like to make a CFGChecksum for the debug-info profile. Maybe just check ProfileIsProbeBased for now and change it to checking CFGChecksum when it comes to extending debug-info profile? What do you think?

[wmi]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

Yes, I agree with it. It is fine to adjust it later.

554–556

Is there case that optimizations after probe insertion create new BBs and the new BBs somehow don't have probes?

I still curious about the answer to the question above. Could you share your thought?

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

Sorry for missing the comment.

That's a good question. Yes, there will be blocks created after probe insertion that won't get a probe. I hope that's not too common if the new blocks really contain code from the original IR. And if that happens, we could rely on the counts inference pass to mitigate that. The worst case is that optimizer creates a conditional structure with none of the new blocks copied from the original IR and places some user code in them. Since there's no probe in the new blocks, we won't know anything about how frequent that code is executed during sampling. I guess in that case we would need to fix the optimizer to copy the original probe along with the code, if the optimization is not done on block level.

[hoy]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

I could write a pass to validate that every block with real instructions gets a probe and run it after each optimization. That should give us a view of how common the missing probe cases are. Then we can start fixing the offending passes from there. What do you think? Thanks for raising the issue.

[wmi]

llvm/include/llvm/ProfileData/SampleProf.h

554–556

That is a good plan. Thanks. And I think it is fine to return zero count for BB without probe here, and revisit it later.

llvm/lib/IR/PseudoProbe.cpp

25

Add a message for it.

llvm/lib/Transforms/IPO/SampleProfile.cpp

790

Add a message for it.

llvm/lib/Transforms/IPO/SampleProfileProbe.cpp

46

Change to emplace_back to avoid build issue of old GCC?

llvm/test/Transforms/SampleProfile/pseudo-probe-profile.ll

2

Could you also add a test similar to test/Transforms/SampleProfile/function_metadata.ll? That is to verify pseudo probe works with thinlto -- hot inline instance will be added to function_entry_count metadata so that thinlto can import them.

[hoy]

llvm/lib/IR/PseudoProbe.cpp

25

Done.

llvm/lib/Transforms/IPO/SampleProfileProbe.cpp

46

Good catch, thanks!

llvm/test/Transforms/SampleProfile/pseudo-probe-profile.ll

2

Thanks for the suggestion. Adjusted function_metadata.ll for pseudo probe.

[hoy]

Addressing Wei's feedbacks.

[wmi]

LGTM.

llvm/include/llvm/IR/PseudoProbe.h

 //===- PseudoProbe.h - Pseudo Probe IR Helpers ------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // Pseudo probe IR intrinsic and dwarf discriminator manipulation routines.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_IR_PSEUDOPROBE_H
 #define LLVM_IR_PSEUDOPROBE_H
 
+#include "llvm/ADT/Optional.h"
 #include <cassert>
 #include <cstdint>
 
 namespace llvm {
 
+class Instruction;
+
 constexpr const char *PseudoProbeDescMetadataName = "llvm.pseudo_probe_desc";
 
 enum class PseudoProbeType { Block = 0, IndirectCall, DirectCall };
 
 struct PseudoProbeDwarfDiscriminator {
   // The following APIs encodes/decodes per-probe information to/from a
   // 32-bit integer which is organized as:
   //  [2:0] - 0x7, this is reserved for regular discriminator,
struct PseudoProbeDwarfDiscriminator {
   static uint32_t extractProbeType(uint32_t Value) {
     return (Value >> 26) & 0x7;
   }
 
   static uint32_t extractProbeAttributes(uint32_t Value) {
     return (Value >> 29) & 0x7;
   }
 };
+
+struct PseudoProbe {
+  uint32_t Id;
+  uint32_t Type;
+  uint32_t Attr;
+};
+
+Optional<PseudoProbe> extractProbe(const Instruction &Inst);
+
 } // end namespace llvm
 
 #endif // LLVM_IR_PSEUDOPROBE_H

llvm/include/llvm/ProfileData/SampleProf.h

enum class sampleprof_error {
   unrecognized_format,
   unsupported_writing_format,
   truncated_name_table,
   not_implemented,
   counter_overflow,
   ostream_seek_unsupported,
   compress_failed,
   uncompress_failed,
-  zlib_unavailable
+  zlib_unavailable,
+  hash_mismatch
 };
 
 inline std::error_code make_error_code(sampleprof_error E) {
   return std::error_code(static_cast<int>(E), sampleprof_category());
 }
 
 inline sampleprof_error MergeResult(sampleprof_error &Accumulator,
                                     sampleprof_error Result) {

 // SampleProfileExtBinaryBaseWriter. Never change the existing
 // value of enum. Only append new ones.
 enum SecType {
   SecInValid = 0,
   SecProfSummary = 1,
   SecNameTable = 2,
   SecProfileSymbolList = 3,
   SecFuncOffsetTable = 4,
+  SecFuncMetadata = 5,
   // marker for the first type of profile.
   SecFuncProfileFirst = 32,
   SecLBRProfile = SecFuncProfileFirst
 };
 
 static inline std::string getSecName(SecType Type) {
   switch (Type) {
   case SecInValid:
     return "InvalidSection";
   case SecProfSummary:
     return "ProfileSummarySection";
   case SecNameTable:
     return "NameTableSection";
   case SecProfileSymbolList:
     return "ProfileSymbolListSection";
   case SecFuncOffsetTable:
     return "FuncOffsetTableSection";
+  case SecFuncMetadata:
+    return "FunctionMetadata";
   case SecLBRProfile:
     return "LBRProfileSection";
   }
   llvm_unreachable("A SecType has no name for output");
 }
 
 // Entry type of section header table used by SampleProfileExtBinaryBaseReader
 // and SampleProfileExtBinaryBaseWriter.

 enum class SecProfSummaryFlags : uint32_t {
   SecFlagInValid = 0,
   /// SecFlagPartial means the profile is for common/shared code.
   /// The common profile is usually merged from profiles collected
   /// from running other targets.
   SecFlagPartial = (1 << 0)
 };
 
+enum class SecFuncMetadataFlags : uint32_t {
+  SecFlagInvalid = 0,
+  SecFlagIsProbeBased = (1 << 0),
+};
+
 // Verify section specific flag is used for the correct section.
 template <class SecFlagType>
 static inline void verifySecFlag(SecType Type, SecFlagType Flag) {
   // No verification is needed for common flags.
   if (std::is_same<SecCommonFlags, SecFlagType>())
     return;
 
   // Verification starts here for section specific flag.
   bool IsFlagLegal = false;
   switch (Type) {
   case SecNameTable:
     IsFlagLegal = std::is_same<SecNameTableFlags, SecFlagType>();
     break;
   case SecProfSummary:
     IsFlagLegal = std::is_same<SecProfSummaryFlags, SecFlagType>();
     break;
+  case SecFuncMetadata:
+    IsFlagLegal = std::is_same<SecFuncMetadataFlags, SecFlagType>();
+    break;
   default:
     break;
   }
   if (!IsFlagLegal)
     llvm_unreachable("Misuse of a flag in an incompatible section");
 }
 
 template <class SecFlagType>
public:
   sampleprof_error addTotalSamples(uint64_t Num, uint64_t Weight = 1) {
     bool Overflowed;
     TotalSamples =
         SaturatingMultiplyAdd(Num, Weight, TotalSamples, &Overflowed);
     return Overflowed ? sampleprof_error::counter_overflow
                       : sampleprof_error::success;
   }
 
+  void setTotalSamples(uint64_t Num) { TotalSamples = Num; }
+
   sampleprof_error addHeadSamples(uint64_t Num, uint64_t Weight = 1) {
     bool Overflowed;
     TotalHeadSamples =
         SaturatingMultiplyAdd(Num, Weight, TotalHeadSamples, &Overflowed);
     return Overflowed ? sampleprof_error::counter_overflow
                       : sampleprof_error::success;
   }
 
ErrorOr<uint64_t> findSamplesAt(uint32_t LineOffset,
     const auto &ret = BodySamples.find(LineLocation(LineOffset, Discriminator));
     if (ret == BodySamples.end()) {
       // For CSSPGO, in order to conserve profile size, we no longer write out
       // locations profile for those not hit during training, so we need to
       // treat them as zero instead of error here.
       if (ProfileIsCS)
         return 0;
       return std::error_code();
+      // A missing entry for a probe likely means the probe was not executed.
+      // Treat it as a zero count instead of an unknown count to help edge
+      // weight inference.
+      if (FunctionSamples::ProfileIsProbeBased)
+        return 0;
+      return std::error_code();
     } else {
       return ret->second.getSamples();
     }
   }
 
   /// Returns the call target map collected at a given location.
   /// Each location is specified by \p LineOffset and \p Discriminator.
   /// If the location is not found in profile, return error.
   ErrorOr<SampleRecord::CallTargetMap>
   findCallTargetMapAt(uint32_t LineOffset, uint32_t Discriminator) const {
     const auto &ret = BodySamples.find(LineLocation(LineOffset, Discriminator));
     if (ret == BodySamples.end())
       return std::error_code();
     return ret->second.getCallTargets();
   }
 
+  /// Returns the call target map collected at a given location specified by \p
+  /// CallSite. If the location is not found in profile, return error.
+  ErrorOr<SampleRecord::CallTargetMap>
+  findCallTargetMapAt(const LineLocation &CallSite) const {
+    const auto &ret = BodySamples.find(CallSite);
+    if (ret == BodySamples.end())
+      return std::error_code();
+    return ret->second.getCallTargets();
+  }
+
   /// Return the function samples at the given callsite location.
   FunctionSamplesMap &functionSamplesAt(const LineLocation &Loc) {
     return CallsiteSamples[Loc];
   }
 
   /// Returns the FunctionSamplesMap at the given \p Loc.
   const FunctionSamplesMap *
   findFunctionSamplesMapAt(const LineLocation &Loc) const {
public:
 
   /// Merge the samples in \p Other into this one.
   /// Optionally scale samples by \p Weight.
   sampleprof_error merge(const FunctionSamples &Other, uint64_t Weight = 1) {
     sampleprof_error Result = sampleprof_error::success;
     Name = Other.getName();
     if (!GUIDToFuncNameMap)
       GUIDToFuncNameMap = Other.GUIDToFuncNameMap;
+
+    if (FunctionHash == 0) {
+      // Set the function hash code for the target profile.
+      FunctionHash = Other.getFunctionHash();
+    } else if (FunctionHash != Other.getFunctionHash()) {
+      // The two profiles coming with different valid hash codes indicates
+      // either:
+      // 1. They are same-named static functions from different compilation
+      // units, or
+      // 2. They are really the same function but from different compilations.
+      // Let's bail out in either case for now, which means one profile is
+      // dropped.
+      return sampleprof_error::hash_mismatch;
+    }
+
     MergeResult(Result, addTotalSamples(Other.getTotalSamples(), Weight));
     MergeResult(Result, addHeadSamples(Other.getHeadSamples(), Weight));
     for (const auto &I : Other.getBodySamples()) {
       const LineLocation &Loc = I.first;
       const SampleRecord &Rec = I.second;
       MergeResult(Result, BodySamples[Loc].merge(Rec, Weight));
     }
     for (const auto &I : Other.getCallsiteSamples()) {
public:
   /// Return function name with context.
   StringRef getNameWithContext() const {
     return FunctionSamples::ProfileIsCS ? Context.getNameWithContext() : Name;
   }
 
   /// Return the original function name.
   StringRef getFuncName() const { return getFuncName(Name); }
 
+  void setFunctionHash(uint64_t Hash) { FunctionHash = Hash; }
+
+  uint64_t getFunctionHash() const { return FunctionHash; }
+
   /// Return the canonical name for a function, taking into account
   /// suffix elision policy attributes.
   static StringRef getCanonicalFnName(const Function &F) {
     auto AttrName = "sample-profile-suffix-elision-policy";
     auto Attr = F.getFnAttribute(AttrName).getValueAsString();
     return getCanonicalFnName(F.getName(), Attr);
   }
 
if (iter == GUIDToFuncNameMap->end())
       return StringRef();
     return iter->second;
   }
 
   /// Returns the line offset to the start line of the subprogram.
   /// We assume that a single function will not exceed 65535 LOC.
   static unsigned getOffset(const DILocation *DIL);
 
+  /// Returns a unique call site identifier for a given debug location of a call
+  /// instruction. This is wrapper of two sceanarios, the hybrid profile and
+  /// regular profile, to hide implementation details from the sample loader and
+  /// the context tracker.
+  static LineLocation getCallSiteIdentifier(const DILocation *DIL);
+
   /// Get the FunctionSamples of the inline instance where DIL originates
   /// from.
   ///
   /// The FunctionSamples of the instruction (Machine or IR) associated to
   /// \p DIL is the inlined instance in which that instruction is coming from.
   /// We traverse the inline stack of that instruction, and match it with the
   /// tree nodes in the profile.
   ///
   /// \returns the FunctionSamples pointer to the inlined instance.
   /// If \p Remapper is not nullptr, it will be used to find matching
   /// FunctionSamples with not exactly the same but equivalent name.
   const FunctionSamples *findFunctionSamples(
       const DILocation *DIL,
       SampleProfileReaderItaniumRemapper *Remapper = nullptr) const;
 
+  static bool ProfileIsProbeBased;
+
   static bool ProfileIsCS;
 
   SampleContext &getContext() const { return Context; }
 
   void setContext(const SampleContext &FContext) { Context = FContext; }
 
   static SampleProfileFormat Format;
 
public:
   // Find all the names in the current FunctionSamples including names in
   // all the inline instances and names of call targets.
   void findAllNames(DenseSet<StringRef> &NameSet) const;
 
 private:
   /// Mangled name of the function.
   StringRef Name;
 
+  /// CFG hash value for the function.
+  uint64_t FunctionHash = 0;
+
   /// Calling context for function profile
   mutable SampleContext Context;
 
   /// Total number of samples collected inside this function.
   ///
   /// Samples are cumulative, they include all the samples collected
   /// inside this function and all its inlined callees.
   uint64_t TotalSamples = 0;

llvm/include/llvm/ProfileData/SampleProfReader.h

 //     function1:total_samples:total_head_samples
 //      offset1[.discriminator]: number_of_samples [fn1:num fn2:num ... ]
 //      offset2[.discriminator]: number_of_samples [fn3:num fn4:num ... ]
 //      ...
 //      offsetN[.discriminator]: number_of_samples [fn5:num fn6:num ... ]
 //      offsetA[.discriminator]: fnA:num_of_total_samples
 //       offsetA1[.discriminator]: number_of_samples [fn7:num fn8:num ... ]
 //       ...
+//      !CFGChecksum: num
 //
-// This is a nested tree in which the identation represents the nesting level
+// This is a nested tree in which the indentation represents the nesting level
 // of the inline stack. There are no blank lines in the file. And the spacing
 // within a single line is fixed. Additional spaces will result in an error
 // while reading the file.
 //
 // Any line starting with the '#' character is completely ignored.
 //
 // Inlined calls are represented with indentation. The Inline stack is a
 // stack of source locations in which the top of the stack represents the
 // leaf function, and the bottom of the stack represents the actual
 // symbol to which the instruction belongs.
 //
 // Function names must be mangled in order for the profile loader to
 // match them in the current translation unit. The two numbers in the
 // function header specify how many total samples were accumulated in the
 // function (first number), and the total number of samples accumulated
 // in the prologue of the function (second number). This head sample
 // count provides an indicator of how frequently the function is invoked.
 //
-// There are two types of lines in the function body.
+// There are three types of lines in the function body.
 //
 // * Sampled line represents the profile information of a source location.
 // * Callsite line represents the profile information of a callsite.
+// * Metadata line represents extra metadata of the function.
 //
 // Each sampled line may contain several items. Some are optional (marked
 // below):
 //
 // a. Source line offset. This number represents the line number
 //    in the function where the sample was collected. The line number is
 //    always relative to the line where symbol of the function is
 //    defined. So, if the function has its header at line 280, the offset

 //    callsite that is inlined in the profiled binary.
 //
 // b. [OPTIONAL] Discriminator. Same as the discriminator for sampled line.
 //
 // c. Number of samples. This is an integer quantity representing the
 //    total number of samples collected for the inlined instance at this
 //    callsite
 //
+// Metadata line can occur in lines with one indent only, containing extra
+// information for the top-level function. Furthermore, metadata can only
+// occur after all the body samples and callsite samples.
+// Each metadata line may contain a particular type of metadata, marked by
+// the starting characters annotated with !. We process each metadata line
+// independently, hence each metadata line has to form an independent piece
+// of information that does not require cross-line reference.
+// We support the following types of metadata:
+//
+// a. CFG Checksum (a.k.a. function hash):
+//   !CFGChecksum: 12345
+//
 //
 // Binary format
 // -------------
 //
 // This is a more compact encoding. Numbers are encoded as ULEB128 values
 // and all strings are encoded in a name table. The file is organized in
 // the following sections:
 //
public:
   /// Return the profile summary.
   ProfileSummary &getSummary() const { return *(Summary.get()); }
 
   MemoryBuffer *getBuffer() const { return Buffer.get(); }
 
   /// \brief Return the profile format.
   SampleProfileFormat getFormat() const { return Format; }
 
-  /// Whether input profile is fully context-sensitie
+  /// Whether input profile is based on pseudo probes.
+  bool profileIsProbeBased() const { return ProfileIsProbeBased; }
+
+  /// Whether input profile is fully context-sensitive
   bool profileIsCS() const { return ProfileIsCS; }
 
   virtual std::unique_ptr<ProfileSymbolList> getProfileSymbolList() {
     return nullptr;
   };
 
   /// It includes all the names that have samples either in outline instance
   /// or inline instance.
takeSummary(SampleProfileReader &Reader) {
     return std::move(Reader.Summary);
   }
 
   /// Compute summary for this profile.
   void computeSummary();
 
   std::unique_ptr<SampleProfileReaderItaniumRemapper> Remapper;
 
+  /// \brief Whether samples are collected based on pseudo probes.
+  bool ProfileIsProbeBased = false;
+
   bool ProfileIsCS = false;
 
   /// \brief The format of sample.
   SampleProfileFormat Format = SPF_None;
 };
 
 class SampleProfileReaderText : public SampleProfileReader {
 public:
private:
 
   BumpPtrAllocator Allocator;
 
 protected:
   std::vector<SecHdrTableEntry> SecHdrTable;
   std::error_code readSecHdrTableEntry();
   std::error_code readSecHdrTable();
 
+  std::error_code readFuncMetadata();
   std::error_code readFuncOffsetTable();
   std::error_code readFuncProfiles();
   std::error_code readMD5NameTable();
   std::error_code readNameTableSec(bool IsMD5);
   std::error_code readProfileSymbolList();
 
   virtual std::error_code readHeader() override;
   virtual std::error_code verifySPMagic(uint64_t Magic) override = 0;

llvm/include/llvm/ProfileData/SampleProfWriter.h

protected:
 
   // Dispatch section writer for each section.
   virtual std::error_code
   writeOneSection(SecType Type, const StringMap<FunctionSamples> &ProfileMap);
 
   // Helper function to write name table.
   virtual std::error_code writeNameTable() override;
 
+  std::error_code writeFuncMetadata(const StringMap<FunctionSamples> &Profiles);
+
   // Functions to write various kinds of sections.
   std::error_code
   writeNameTableSection(const StringMap<FunctionSamples> &ProfileMap);
   std::error_code writeFuncOffsetTable();
   std::error_code writeProfileSymbolListSection();
 
   // Specifiy the order of sections in section header table. Note
   // the order of sections in the profile may be different that the
virtual void initSectionHdrLayout() override {
     //
     // This is because sample reader follows the order of SectionHdrLayout to
     // read each section, to read function profiles on demand sample reader
     // need to get the offset of each function profile first.
     //
     // SecFuncOffsetTable section is written after SecLBRProfile in the
     // profile because FuncOffsetTable needs to be populated while section
     // SecLBRProfile is written.
-    SectionHdrLayout = {{SecProfSummary, 0, 0, 0},
-                        {SecNameTable, 0, 0, 0},
-                        {SecFuncOffsetTable, 0, 0, 0},
-                        {SecLBRProfile, 0, 0, 0},
-                        {SecProfileSymbolList, 0, 0, 0}};
+    SectionHdrLayout = {
+        {SecProfSummary, 0, 0, 0},       {SecNameTable, 0, 0, 0},
+        {SecFuncOffsetTable, 0, 0, 0},   {SecLBRProfile, 0, 0, 0},
+        {SecProfileSymbolList, 0, 0, 0}, {SecFuncMetadata, 0, 0, 0}};
   };
   virtual std::error_code
   writeSections(const StringMap<FunctionSamples> &ProfileMap) override;
 
   virtual std::error_code writeCustomSection(SecType Type) override {
     return sampleprof_error::success;
   };
 };

llvm/include/llvm/Transforms/IPO/SampleProfileProbe.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TRANSFORMS_IPO_SAMPLEPROFILEPROBE_H
 #define LLVM_TRANSFORMS_IPO_SAMPLEPROFILEPROBE_H
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/IR/PseudoProbe.h"
+#include "llvm/ProfileData/SampleProf.h"
 #include "llvm/Target/TargetMachine.h"
 #include <unordered_map>
 
 namespace llvm {
 
 class Module;
 
+using namespace sampleprof;
 using BlockIdMap = std::unordered_map<BasicBlock *, uint32_t>;
 using InstructionIdMap = std::unordered_map<Instruction *, uint32_t>;
 
 enum class PseudoProbeReservedId { Invalid = 0, Last = Invalid };
 
+class PseudoProbeDescriptor {
+  uint64_t FunctionGUID;
+  uint64_t FunctionHash;
+
+public:
+  PseudoProbeDescriptor(uint64_t GUID, uint64_t Hash)
+      : FunctionGUID(GUID), FunctionHash(Hash) {}
+  uint64_t getFunctionGUID() const { return FunctionGUID; }
+  uint64_t getFunctionHash() const { return FunctionHash; }
+};
+
+// This class serves sample counts correlation for SampleProfileLoader by
+// analyzing pseudo probes and their function descriptors injected by
+// SampleProfileProber.
+class PseudoProbeManager {
+  DenseMap<uint64_t, PseudoProbeDescriptor> GUIDToProbeDescMap;
+
+  const PseudoProbeDescriptor *getDesc(const Function &F) const;
+
+public:
+  PseudoProbeManager(const Module &M);
+  bool moduleIsProbed(const Module &M) const;
+  bool profileIsValid(const Function &F, const FunctionSamples &Samples) const;
+};
 
 /// Sample profile pseudo prober.
 ///
 /// Insert pseudo probes for block sampling and value sampling.
 class SampleProfileProber {
 public:
   // Give an empty module id when the prober is not used for instrumentation.
   SampleProfileProber(Function &F, const std::string &CurModuleUniqueId);

llvm/lib/IR/CMakeLists.txt

add_llvm_component_library(LLVMCore
   Pass.cpp
   PassInstrumentation.cpp
   PassManager.cpp
   PassRegistry.cpp
   PassTimingInfo.cpp
   PrintPasses.cpp
   SafepointIRVerifier.cpp
   ProfileSummary.cpp
+  PseudoProbe.cpp
   Statepoint.cpp
   StructuralHash.cpp
   Type.cpp
   TypeFinder.cpp
   Use.cpp
   User.cpp
   Value.cpp
   ValueSymbolTable.cpp

llvm/lib/IR/PseudoProbe.cpp

+//===- PseudoProbe.cpp - Pseudo Probe Helpers -----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the helpers to manipulate pseudo probe IR intrinsic
+// calls.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/IR/PseudoProbe.h"
+#include "llvm/IR/DebugInfoMetadata.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/Instruction.h"
+
+using namespace llvm;
+
+namespace llvm {
+
+Optional<PseudoProbe> extractProbe(const Instruction &Inst) {
+  if (auto *II = dyn_cast<PseudoProbeInst>(&Inst)) {
+    PseudoProbe Probe;
+    Probe.Id = II->getIndex()->getZExtValue();
+    Probe.Type = (uint32_t)PseudoProbeType::Block;
+    Probe.Attr = II->getAttributes()->getZExtValue();
+    return Probe;
+  }
+
+  if (isa<CallBase>(&Inst) && !isa<IntrinsicInst>(&Inst)) {
+    if (const DebugLoc &DLoc = Inst.getDebugLoc()) {
+      const DILocation *DIL = DLoc;
+      auto Discriminator = DIL->getDiscriminator();
+      if (DILocation::isPseudoProbeDiscriminator(Discriminator)) {
+        PseudoProbe Probe;
+        Probe.Id =
+            PseudoProbeDwarfDiscriminator::extractProbeIndex(Discriminator);
+        Probe.Type =
+            PseudoProbeDwarfDiscriminator::extractProbeType(Discriminator);
+        Probe.Attr = PseudoProbeDwarfDiscriminator::extractProbeAttributes(
+            Discriminator);
+        return Probe;
+      }
+    }
+  }
+  return None;
+}
+} // namespace llvm

llvm/lib/ProfileData/SampleProf.cpp

 //=-- SampleProf.cpp - Sample profiling format support --------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains common definitions used in the reading and writing of
 // sample profile data.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ProfileData/SampleProf.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/IR/DebugInfoMetadata.h"
+#include "llvm/IR/PseudoProbe.h"
 #include "llvm/ProfileData/SampleProfReader.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/raw_ostream.h"
 #include <string>
 #include <system_error>
 
 using namespace llvm;
 using namespace sampleprof;
 
 namespace llvm {
 namespace sampleprof {
 SampleProfileFormat FunctionSamples::Format;
+bool FunctionSamples::ProfileIsProbeBased = false;
 bool FunctionSamples::ProfileIsCS = false;
 bool FunctionSamples::UseMD5;
 } // namespace sampleprof
 } // namespace llvm
 
 namespace {
 
 // FIXME: This class is only here to support the transition to llvm::Error. It
std::string message(int IE) const override {
     case sampleprof_error::ostream_seek_unsupported:
       return "Ostream does not support seek";
     case sampleprof_error::compress_failed:
       return "Compress failure";
     case sampleprof_error::uncompress_failed:
       return "Uncompress failure";
     case sampleprof_error::zlib_unavailable:
       return "Zlib is unavailable";
+    case sampleprof_error::hash_mismatch:
+      return "Function hash mismatch";
     }
     llvm_unreachable("A value of sampleprof_error has no message.");
   }
 };
 
 } // end anonymous namespace
 
 static ManagedStatic<SampleProfErrorCategoryType> ErrorCategory;

 
 raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS,
                                           const SampleRecord &Sample) {
   Sample.print(OS, 0);
   return OS;
 }
 
 /// Print the samples collected for a function on stream \p OS.
 void FunctionSamples::print(raw_ostream &OS, unsigned Indent) const {
   OS << TotalSamples << ", " << TotalHeadSamples << ", " << BodySamples.size()
      << " sampled lines\n";
 
   OS.indent(Indent);
   if (!BodySamples.empty()) {
     OS << "Samples collected in the function's body {\n";
     SampleSorter<LineLocation, SampleRecord> SortedBodySamples(BodySamples);
     for (const auto &SI : SortedBodySamples.get()) {
raw_ostream &llvm::sampleprof::operator<<(raw_ostream &OS,
   return OS;
 }
 
 unsigned FunctionSamples::getOffset(const DILocation *DIL) {
   return (DIL->getLine() - DIL->getScope()->getSubprogram()->getLine()) &
       0xffff;
 }
 
+LineLocation FunctionSamples::getCallSiteIdentifier(const DILocation *DIL) {
+  if (FunctionSamples::ProfileIsProbeBased)
+    // In a pseudo-probe based profile, a callsite is simply represented by the
+    // ID of the probe associated with the call instruction. The probe ID is
+    // encoded in the Discriminator field of the call instruction's debug
+    // metadata.
+    return LineLocation(PseudoProbeDwarfDiscriminator::extractProbeIndex(
+                            DIL->getDiscriminator()),
+                        0);
+  else
+    return LineLocation(FunctionSamples::getOffset(DIL),
+                        DIL->getBaseDiscriminator());
+}
+
 const FunctionSamples *FunctionSamples::findFunctionSamples(
     const DILocation *DIL, SampleProfileReaderItaniumRemapper *Remapper) const {
   assert(DIL);
   SmallVector<std::pair<LineLocation, StringRef>, 10> S;
 
   const DILocation *PrevDIL = DIL;
   for (DIL = DIL->getInlinedAt(); DIL; DIL = DIL->getInlinedAt()) {
     S.push_back(std::make_pair(

llvm/lib/ProfileData/SampleProfReader.cpp

static bool ParseHead(const StringRef &Input, StringRef &FName,
   if (Input.substr(n2 + 1).getAsInteger(10, NumHeadSamples))
     return false;
   return true;
 }
 
 /// Returns true if line offset \p L is legal (only has 16 bits).
 static bool isOffsetLegal(unsigned L) { return (L & 0xffff) == L; }
 
+/// Parse \p Input that contains metadata.
+/// Possible metadata:
+/// - CFG Checksum information:
+///     !CFGChecksum: 12345
+/// Stores the FunctionHash (a.k.a. CFG Checksum) into \p FunctionHash.
+static bool ParseMetadata(const StringRef &Input, uint64_t &FunctionHash) {
+  if (!Input.startswith("!CFGChecksum:"))
+    return false;
+
+  StringRef CFGInfo = Input.substr(strlen("!CFGChecksum:")).trim();
+  return !CFGInfo.getAsInteger(10, FunctionHash);
+}
+
+enum class LineType {
+  CallSiteProfile,
+  BodyProfile,
+  Metadata,
+};
+
 /// Parse \p Input as line sample.
 ///
 /// \param Input input line.
-/// \param IsCallsite true if the line represents an inlined callsite.
+/// \param LineTy Type of this line.
 /// \param Depth the depth of the inline stack.
 /// \param NumSamples total samples of the line/inlined callsite.
 /// \param LineOffset line offset to the start of the function.
 /// \param Discriminator discriminator of the line.
 /// \param TargetCountMap map from indirect call target to count.
+/// \param FunctionHash the function's CFG hash, used by pseudo probe.
 ///
 /// returns true if parsing is successful.
-static bool ParseLine(const StringRef &Input, bool &IsCallsite, uint32_t &Depth,
+static bool ParseLine(const StringRef &Input, LineType &LineTy, uint32_t &Depth,
                       uint64_t &NumSamples, uint32_t &LineOffset,
                       uint32_t &Discriminator, StringRef &CalleeName,
-                      DenseMap<StringRef, uint64_t> &TargetCountMap) {
+                      DenseMap<StringRef, uint64_t> &TargetCountMap,
+                      uint64_t &FunctionHash) {
   for (Depth = 0; Input[Depth] == ' '; Depth++)
     ;
   if (Depth == 0)
     return false;
 
+  if (Depth == 1 && Input[Depth] == '!') {
+    LineTy = LineType::Metadata;
+    return ParseMetadata(Input.substr(Depth), FunctionHash);
+  }
+
   size_t n1 = Input.find(':');
   StringRef Loc = Input.substr(Depth, n1 - Depth);
   size_t n2 = Loc.find('.');
   if (n2 == StringRef::npos) {
     if (Loc.getAsInteger(10, LineOffset) || !isOffsetLegal(LineOffset))
       return false;
     Discriminator = 0;
   } else {
     if (Loc.substr(0, n2).getAsInteger(10, LineOffset))
       return false;
     if (Loc.substr(n2 + 1).getAsInteger(10, Discriminator))
       return false;
   }
 
   StringRef Rest = Input.substr(n1 + 2);
   if (Rest[0] >= '0' && Rest[0] <= '9') {
-    IsCallsite = false;
+    LineTy = LineType::BodyProfile;
     size_t n3 = Rest.find(' ');
     if (n3 == StringRef::npos) {
       if (Rest.getAsInteger(10, NumSamples))
         return false;
     } else {
       if (Rest.substr(0, n3).getAsInteger(10, NumSamples))
         return false;
     }
while (n3 != StringRef::npos) {
       // An anchor point is found. Save the {target, count} pair
       TargetCountMap[Target] = count;
       if (n4 == Rest.size())
         break;
       // Change n3 to the next blank space after colon + integer pair.
       n3 = n4;
     }
   } else {
-    IsCallsite = true;
+    LineTy = LineType::CallSiteProfile;
     size_t n3 = Rest.find_last_of(':');
     CalleeName = Rest.substr(0, n3);
     if (Rest.substr(n3 + 1).getAsInteger(10, NumSamples))
       return false;
   }
   return true;
 }
 
 /// Load samples from a text file.
 ///
 /// See the documentation at the top of the file for an explanation of
 /// the expected format.
 ///
 /// \returns true if the file was loaded successfully, false otherwise.
 std::error_code SampleProfileReaderText::readImpl() {
   line_iterator LineIt(*Buffer, /*SkipBlanks=*/true, '#');
   sampleprof_error Result = sampleprof_error::success;
 
   InlineCallStack InlineStack;
   int CSProfileCount = 0;
   int RegularProfileCount = 0;
+  uint32_t ProbeProfileCount = 0;
+
+  // SeenMetadata tracks whether we have processed metadata for the current
+  // top-level function profile.
+  bool SeenMetadata = false;
 
   for (; !LineIt.is_at_eof(); ++LineIt) {
     if ((*LineIt)[(*LineIt).find_first_not_of(' ')] == '#')
       continue;
     // Read the header of each function.
     //
     // Note that for function identifiers we are actually expecting
     // mangled names, but we may not always get them. This happens when
     // the compiler decides not to emit the function (e.g., it was inlined
     // and removed). In this case, the binary will not have the linkage
     // name for the function, so the profiler will emit the function's
     // unmangled name, which may contain characters like ':' and '>' in its
     // name (member functions, templates, etc).
     //
     // The only requirement we place on the identifier, then, is that it
     // should not begin with a number.
     if ((*LineIt)[0] != ' ') {
       uint64_t NumSamples, NumHeadSamples;
       StringRef FName;
       if (!ParseHead(*LineIt, FName, NumSamples, NumHeadSamples)) {
         reportError(LineIt.line_number(),
                     "Expected 'mangled_name:NUM:NUM', found " + *LineIt);
         return sampleprof_error::malformed;
       }
+      SeenMetadata = false;
       SampleContext FContext(FName);
       if (FContext.hasContext())
         ++CSProfileCount;
       else
         ++RegularProfileCount;
       Profiles[FContext] = FunctionSamples();
       FunctionSamples &FProfile = Profiles[FContext];
       FProfile.setName(FContext.getName());
       FProfile.setContext(FContext);
       MergeResult(Result, FProfile.addTotalSamples(NumSamples));
       MergeResult(Result, FProfile.addHeadSamples(NumHeadSamples));
       InlineStack.clear();
       InlineStack.push_back(&FProfile);
     } else {
       uint64_t NumSamples;
       StringRef FName;
       DenseMap<StringRef, uint64_t> TargetCountMap;
-      bool IsCallsite;
       uint32_t Depth, LineOffset, Discriminator;
-      if (!ParseLine(*LineIt, IsCallsite, Depth, NumSamples, LineOffset,
-                     Discriminator, FName, TargetCountMap)) {
+      LineType LineTy;
+      uint64_t FunctionHash;
+      if (!ParseLine(*LineIt, LineTy, Depth, NumSamples, LineOffset,
+                     Discriminator, FName, TargetCountMap, FunctionHash)) {
         reportError(LineIt.line_number(),
                     "Expected 'NUM[.NUM]: NUM[ mangled_name:NUM]*', found " +
                         *LineIt);
         return sampleprof_error::malformed;
       }
-      if (IsCallsite) {
-        while (InlineStack.size() > Depth) {
-          InlineStack.pop_back();
-        }
+      if (SeenMetadata && LineTy != LineType::Metadata) {
+        // Metadata must be put at the end of a function profile.
+        reportError(LineIt.line_number(),
+                    "Found non-metadata after metadata: " + *LineIt);
+        return sampleprof_error::malformed;
+      }
+      switch (LineTy) {
+      case LineType::CallSiteProfile: {
         FunctionSamples &FSamples = InlineStack.back()->functionSamplesAt(
             LineLocation(LineOffset, Discriminator))[std::string(FName)];
         FSamples.setName(FName);
         MergeResult(Result, FSamples.addTotalSamples(NumSamples));
         InlineStack.push_back(&FSamples);
-      } else {
+        break;
+      }
+      case LineType::BodyProfile: {
         while (InlineStack.size() > Depth) {
           InlineStack.pop_back();
         }
         FunctionSamples &FProfile = *InlineStack.back();
         for (const auto &name_count : TargetCountMap) {
           MergeResult(Result, FProfile.addCalledTargetSamples(
                                   LineOffset, Discriminator, name_count.first,
                                   name_count.second));
         }
         MergeResult(Result, FProfile.addBodySamples(LineOffset, Discriminator,
                                                     NumSamples));
+        break;
+      }
+      case LineType::Metadata: {
+        FunctionSamples &FProfile = *InlineStack.back();
+        FProfile.setFunctionHash(FunctionHash);
+        ++ProbeProfileCount;
+        SeenMetadata = true;
+        break;
+      }
       }
     }
   }
 
   assert((RegularProfileCount == 0 || CSProfileCount == 0) &&
          "Cannot have both context-sensitive and regular profile");
   ProfileIsCS = (CSProfileCount > 0);
+  assert((ProbeProfileCount == 0 || ProbeProfileCount == Profiles.size()) &&
+         "Cannot have both probe-based profiles and regular profiles");
+  ProfileIsProbeBased = (ProbeProfileCount > 0);
+  FunctionSamples::ProfileIsProbeBased = ProfileIsProbeBased;
 
   if (Result == sampleprof_error::success)
     computeSummary();
 
   return Result;
 }
 
 bool SampleProfileReaderText::hasFormat(const MemoryBuffer &Buffer) {
std::error_code SampleProfileReaderExtBinaryBase::readOneSection(
   case SecLBRProfile:
     if (std::error_code EC = readFuncProfiles())
       return EC;
     break;
   case SecFuncOffsetTable:
     if (std::error_code EC = readFuncOffsetTable())
       return EC;
     break;
+  case SecFuncMetadata:
+    ProfileIsProbeBased =
+        hasSecFlag(Entry, SecFuncMetadataFlags::SecFlagIsProbeBased);
+    FunctionSamples::ProfileIsProbeBased = ProfileIsProbeBased;
+    if (std::error_code EC = readFuncMetadata())
+      return EC;
+    break;
   case SecProfileSymbolList:
     if (std::error_code EC = readProfileSymbolList())
       return EC;
     break;
   default:
     if (std::error_code EC = readCustomSection(Entry))
       return EC;
     break;

 }
 
 std::error_code SampleProfileReaderExtBinaryBase::readNameTableSec(bool IsMD5) {
   if (IsMD5)
     return readMD5NameTable();
   return SampleProfileReaderBinary::readNameTable();
 }
 
+std::error_code SampleProfileReaderExtBinaryBase::readFuncMetadata() {
+  if (ProfileIsProbeBased) {
+    for (unsigned i = 0; i < Profiles.size(); ++i) {
+      auto FName(readStringFromTable());
+      if (std::error_code EC = FName.getError())
+        return EC;
+
+      auto Checksum = readNumber<uint64_t>();
+      if (std::error_code EC = Checksum.getError())
+        return EC;
+
+      Profiles[*FName].setFunctionHash(*Checksum);
+    }
+  }
+
+  return sampleprof_error::success;
+}
+
 std::error_code SampleProfileReaderCompactBinary::readNameTable() {
   auto Size = readNumber<uint64_t>();
   if (std::error_code EC = Size.getError())
     return EC;
   NameTable.reserve(*Size);
   for (uint32_t I = 0; I < *Size; ++I) {
     auto FID = readNumber<uint64_t>();
     if (std::error_code EC = FID.getError())

llvm/lib/ProfileData/SampleProfWriter.cpp

std::error_code SampleProfileWriterExtBinaryBase::writeFuncOffsetTable() {
   // Write out FuncOffsetTable.
   for (auto entry : FuncOffsetTable) {
     writeNameIdx(entry.first);
     encodeULEB128(entry.second, OS);
   }
   return sampleprof_error::success;
 }
 
+std::error_code SampleProfileWriterExtBinaryBase::writeFuncMetadata(
+    const StringMap<FunctionSamples> &Profiles) {
+  if (FunctionSamples::ProfileIsProbeBased) {
+    auto &OS = *OutputStream;
+    for (const auto &Entry : Profiles) {
+      writeNameIdx(Entry.first());
+      encodeULEB128(Entry.second.getFunctionHash(), OS);
+    }
+  }
+  return sampleprof_error::success;
+}
+
 std::error_code SampleProfileWriterExtBinaryBase::writeNameTable() {
   if (!UseMD5)
     return SampleProfileWriterBinary::writeNameTable();
 
   auto &OS = *OutputStream;
   std::set<StringRef> V;
   stablizeNameTable(V);
 
SampleProfileWriterExtBinaryBase::writeProfileSymbolListSection() {
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterExtBinaryBase::writeOneSection(
     SecType Type, const StringMap<FunctionSamples> &ProfileMap) {
   // The setting of SecFlagCompress should happen before markSectionStart.
   if (Type == SecProfileSymbolList && ProfSymList && ProfSymList->toCompress())
     setToCompressSection(SecProfileSymbolList);
+  if (Type == SecFuncMetadata && FunctionSamples::ProfileIsProbeBased)
+    addSectionFlag(SecFuncMetadata, SecFuncMetadataFlags::SecFlagIsProbeBased);
 
   uint64_t SectionStart = markSectionStart(Type);
   switch (Type) {
   case SecProfSummary:
     computeSummary(ProfileMap);
     if (auto EC = writeSummary())
       return EC;
     break;
   case SecNameTable:
     if (auto EC = writeNameTableSection(ProfileMap))
       return EC;
     break;
   case SecLBRProfile:
     SecLBRProfileStart = OutputStream->tell();
     if (std::error_code EC = writeFuncProfiles(ProfileMap))
       return EC;
     break;
   case SecFuncOffsetTable:
     if (auto EC = writeFuncOffsetTable())
       return EC;
     break;
+  case SecFuncMetadata:
+    if (std::error_code EC = writeFuncMetadata(ProfileMap))
+      return EC;
+    break;
   case SecProfileSymbolList:
     if (auto EC = writeProfileSymbolListSection())
       return EC;
     break;
   default:
     if (auto EC = writeCustomSection(Type))
       return EC;
     break;
std::error_code SampleProfileWriterExtBinary::writeSections(
   if (auto EC = writeOneSection(SecNameTable, ProfileMap))
     return EC;
   if (auto EC = writeOneSection(SecLBRProfile, ProfileMap))
     return EC;
   if (auto EC = writeOneSection(SecProfileSymbolList, ProfileMap))
     return EC;
   if (auto EC = writeOneSection(SecFuncOffsetTable, ProfileMap))
     return EC;
+  if (auto EC = writeOneSection(SecFuncMetadata, ProfileMap))
+    return EC;
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterCompactBinary::write(
     const StringMap<FunctionSamples> &ProfileMap) {
   if (std::error_code EC = SampleProfileWriter::write(ProfileMap))
     return EC;
   if (std::error_code EC = writeFuncOffsetTable())
for (const auto &FS : I->second) {
         OS << Loc.LineOffset << ": ";
       else
         OS << Loc.LineOffset << "." << Loc.Discriminator << ": ";
       if (std::error_code EC = writeSample(CalleeSamples))
         return EC;
     }
   Indent -= 1;
 
+  if (Indent == 0) {
+    if (FunctionSamples::ProfileIsProbeBased) {
+      OS.indent(Indent + 1);
+      OS << "!CFGChecksum: " << S.getFunctionHash() << "\n";
+    }
+  }
+
   return sampleprof_error::success;
 }
 
 std::error_code SampleProfileWriterBinary::writeNameIdx(StringRef FName) {
   const auto &ret = NameTable.find(FName);
   if (ret == NameTable.end())
     return sampleprof_error::truncated_name_table;
   encodeULEB128(ret->second, *OutputStream);

llvm/lib/Transforms/IPO/SampleProfile.cpp

 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/GenericDomTree.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/IPO.h"
 #include "llvm/Transforms/IPO/SampleContextTracker.h"
+#include "llvm/Transforms/IPO/SampleProfileProbe.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/Utils/CallPromotionUtils.h"
 #include "llvm/Transforms/Utils/Cloning.h"
 #include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <functional>
 #include <limits>

 using ProfileCount = Function::ProfileCount;
 #define DEBUG_TYPE "sample-profile"
 #define CSINLINE_DEBUG DEBUG_TYPE "-inline"
 
 STATISTIC(NumCSInlined,
           "Number of functions inlined with context sensitive profile");
 STATISTIC(NumCSNotInlined,
           "Number of functions not inlined with context sensitive profile");
+STATISTIC(NumMismatchedProfile,
+          "Number of functions with CFG mismatched profile");
+STATISTIC(NumMatchedProfile, "Number of functions with CFG matched profile");
 
 // Command line option to specify the file to read samples from. This is
 // mainly used for debugging.
 static cl::opt<std::string> SampleProfileFile(
     "sample-profile-file", cl::init(""), cl::value_desc("filename"),
     cl::desc("Profile file loaded by -sample-profile"), cl::Hidden);
 
 // The named file contains a set of transformations that may have been applied

 
 protected:
   friend class SampleCoverageTracker;
 
   bool runOnFunction(Function &F, ModuleAnalysisManager *AM);
   unsigned getFunctionLoc(Function &F);
   bool emitAnnotations(Function &F);
   ErrorOr<uint64_t> getInstWeight(const Instruction &I);
+  ErrorOr<uint64_t> getProbeWeight(const Instruction &I);
   ErrorOr<uint64_t> getBlockWeight(const BasicBlock *BB);
   const FunctionSamples *findCalleeFunctionSamples(const CallBase &I) const;
   std::vector<const FunctionSamples *>
   findIndirectCallFunctionSamples(const Instruction &I, uint64_t &Sum) const;
   mutable DenseMap<const DILocation *, const FunctionSamples *> DILocation2SampleMap;
   const FunctionSamples *findFunctionSamples(const Instruction &I) const;
   bool inlineCallInstruction(CallBase &CB);
   bool inlineHotFunctions(Function &F,
protected:
   // to be accurate. It is mainly decided by existance of profile symbol
   // list and -profile-accurate-for-symsinlist flag, but it can be
   // overriden by -profile-sample-accurate or profile-sample-accurate
   // attribute.
   bool ProfAccForSymsInList;
 
   // External inline advisor used to replay inline decision from remarks.
   std::unique_ptr<ReplayInlineAdvisor> ExternalInlineAdvisor;
+
+  // A pseudo probe helper to correlate the imported sample counts.
+  std::unique_ptr<PseudoProbeManager> ProbeManager;
 };
 
 class SampleProfileLoaderLegacyPass : public ModulePass {
 public:
   // Class identification, replacement for typeinfo
   static char ID;
 
   SampleProfileLoaderLegacyPass(StringRef Name = SampleProfileFile,

 /// need to compute the line number of \p Inst relative to the start of its
 /// function. We use HeaderLineno to compute the offset. We then
 /// look up the samples collected for \p Inst using BodySamples.
 ///
 /// \param Inst Instruction to query.
 ///
 /// \returns the weight of \p Inst.
 ErrorOr<uint64_t> SampleProfileLoader::getInstWeight(const Instruction &Inst) {
+  if (FunctionSamples::ProfileIsProbeBased)
+    return getProbeWeight(Inst);
+
   const DebugLoc &DLoc = Inst.getDebugLoc();
   if (!DLoc)
     return std::error_code();
 
   const FunctionSamples *FS = findFunctionSamples(Inst);
   if (!FS)
     return std::error_code();
 
LLVM_DEBUG(dbgs() << "    " << DLoc.getLine() << "."
                       << DIL->getBaseDiscriminator() << ":" << Inst
                       << " (line offset: " << LineOffset << "."
                       << DIL->getBaseDiscriminator() << " - weight: " << R.get()
                       << ")\n");
   }
   return R;
 }
 
+ErrorOr<uint64_t> SampleProfileLoader::getProbeWeight(const Instruction &Inst) {
+  assert(FunctionSamples::ProfileIsProbeBased);
+  Optional<PseudoProbe> Probe = extractProbe(Inst);
+  if (!Probe)
+    return std::error_code();
+
+  const FunctionSamples *FS = findFunctionSamples(Inst);
+  if (!FS)
+    return std::error_code();
+
+  // If a direct call/invoke instruction is inlined in profile
+  // (findCalleeFunctionSamples returns non-empty result), but not inlined here,
+  // it means that the inlined callsite has no sample, thus the call
+  // instruction should have 0 count.
+  if (auto *CB = dyn_cast<CallBase>(&Inst))
+    if (!CB->isIndirectCall() && findCalleeFunctionSamples(*CB))
+      return 0;
+
+  const ErrorOr<uint64_t> &R = FS->findSamplesAt(Probe->Id, 0);
+  if (R) {
+    uint64_t Samples = R.get();
+    bool FirstMark = CoverageTracker.markSamplesUsed(FS, Probe->Id, 0, Samples);
+    if (FirstMark) {
+      ORE->emit([&]() {
+        OptimizationRemarkAnalysis Remark(DEBUG_TYPE, "AppliedSamples", &Inst);
+        Remark << "Applied " << ore::NV("NumSamples", Samples);
+        Remark << " samples from profile (ProbeId=";
+        Remark << ore::NV("ProbeId", Probe->Id);
+        Remark << ")";
+        return Remark;
+      });
+    }
+
+    LLVM_DEBUG(dbgs() << "    " << Probe->Id << ":" << Inst
+                      << " - weight: " << R.get() << ")\n");
+    return Samples;
+  }
+  return R;
+}
+
 /// Compute the weight of a basic block.
 ///
 /// The weight of basic block \p BB is the maximum weight of all the
 /// instructions in BB.
 ///
 /// \param BB The basic block to query.
 ///
 /// \returns the weight for \p BB.
SampleProfileLoader::findCalleeFunctionSamples(const CallBase &Inst) const {
 
   if (ProfileIsCS)
     return ContextTracker->getCalleeContextSamplesFor(Inst, CalleeName);
 
   const FunctionSamples *FS = findFunctionSamples(Inst);
   if (FS == nullptr)
     return nullptr;
 
-  return FS->findFunctionSamplesAt(LineLocation(FunctionSamples::getOffset(DIL),
-                                                DIL->getBaseDiscriminator()),
+  return FS->findFunctionSamplesAt(FunctionSamples::getCallSiteIdentifier(DIL),
                                    CalleeName, Reader->getRemapper());
 }
 
 /// Returns a vector of FunctionSamples that are the indirect call targets
 /// of \p Inst. The vector is sorted by the total number of samples. Stores
 /// the total call count of the indirect call in \p Sum.
 std::vector<const FunctionSamples *>
 SampleProfileLoader::findIndirectCallFunctionSamples(
     const Instruction &Inst, uint64_t &Sum) const {
   const DILocation *DIL = Inst.getDebugLoc();
   std::vector<const FunctionSamples *> R;
 
   if (!DIL) {
     return R;
   }
 
   const FunctionSamples *FS = findFunctionSamples(Inst);
   if (FS == nullptr)
     return R;
 
-  uint32_t LineOffset = FunctionSamples::getOffset(DIL);
-  uint32_t Discriminator = DIL->getBaseDiscriminator();
-
-  auto T = FS->findCallTargetMapAt(LineOffset, Discriminator);
+  auto CallSite = FunctionSamples::getCallSiteIdentifier(DIL);
+  auto T = FS->findCallTargetMapAt(CallSite);
   Sum = 0;
   if (T)
     for (const auto &T_C : T.get())
       Sum += T_C.second;
-  if (const FunctionSamplesMap *M = FS->findFunctionSamplesMapAt(LineLocation(
-          FunctionSamples::getOffset(DIL), DIL->getBaseDiscriminator()))) {
+  if (const FunctionSamplesMap *M = FS->findFunctionSamplesMapAt(CallSite)) {
     if (M->empty())
       return R;
     for (const auto &NameFS : *M) {
       Sum += NameFS.second.getEntrySamples();
       R.push_back(&NameFS.second);
     }
     llvm::sort(R, [](const FunctionSamples *L, const FunctionSamples *R) {
       if (L->getEntrySamples() != R->getEntrySamples())

 /// in which that instruction is coming from. We traverse the inline stack
 /// of that instruction, and match it with the tree nodes in the profile.
 ///
 /// \param Inst Instruction to query.
 ///
 /// \returns the FunctionSamples pointer to the inlined instance.
 const FunctionSamples *
 SampleProfileLoader::findFunctionSamples(const Instruction &Inst) const {
+  if (FunctionSamples::ProfileIsProbeBased) {
+    Optional<PseudoProbe> Probe = extractProbe(Inst);
+    if (!Probe)
+      return nullptr;
+  }
+
   const DILocation *DIL = Inst.getDebugLoc();
   if (!DIL)
     return Samples;
 
   auto it = DILocation2SampleMap.try_emplace(DIL,nullptr);
   if (it.second) {
     if (ProfileIsCS)
       it.first->second = ContextTracker->getContextSamplesFor(DIL);
if (BlockWeights[BB]) {
       for (auto &I : BB->getInstList()) {
         if (!isa<CallInst>(I) && !isa<InvokeInst>(I))
           continue;
         if (!cast<CallBase>(I).getCalledFunction()) {
           const DebugLoc &DLoc = I.getDebugLoc();
           if (!DLoc)
             continue;
           const DILocation *DIL = DLoc;
-          uint32_t LineOffset = FunctionSamples::getOffset(DIL);
-          uint32_t Discriminator = DIL->getBaseDiscriminator();
-
           const FunctionSamples *FS = findFunctionSamples(I);
           if (!FS)
             continue;
-          auto T = FS->findCallTargetMapAt(LineOffset, Discriminator);
+          auto CallSite = FunctionSamples::getCallSiteIdentifier(DIL);
+          auto T = FS->findCallTargetMapAt(CallSite);
           if (!T || T.get().empty())
             continue;
           SmallVector<InstrProfValueData, 2> SortedCallTargets =
               GetSortedValueDataFromCallTargets(T.get());
           uint64_t Sum;
           findIndirectCallFunctionSamples(I, Sum);
           annotateValueSite(*I.getParent()->getParent()->getParent(), I,
                             SortedCallTargets, Sum, IPVK_IndirectCallTarget,

 /// metadata on BB using the computed values for each of its branches.
 ///
 /// \param F The function to query.
 ///
 /// \returns true if \p F was modified. Returns false, otherwise.
 bool SampleProfileLoader::emitAnnotations(Function &F) {
   bool Changed = false;
 
+  if (FunctionSamples::ProfileIsProbeBased) {
+    if (!ProbeManager->profileIsValid(F, *Samples)) {
+      LLVM_DEBUG(
+          dbgs() << "Profile is invalid due to CFG mismatch for Function "
+                 << F.getName());
+      ++NumMismatchedProfile;
+      return false;
+    }
+    ++NumMatchedProfile;
+  } else {
     if (getFunctionLoc(F) == 0)
       return false;
 
     LLVM_DEBUG(dbgs() << "Line number for the first instruction in "
                       << F.getName() << ": " << getFunctionLoc(F) << "\n");
+  }
 
   DenseSet<GlobalValue::GUID> InlinedGUIDs;
   Changed |= inlineHotFunctions(F, InlinedGUIDs);
 
   // Compute basic block weights.
   Changed |= computeBlockWeights(F);
 
   if (Changed) {
if (Reader->profileIsCS()) {
     ProfileIsCS = true;
     FunctionSamples::ProfileIsCS = true;
 
     // Tracker for profiles under different context
     ContextTracker =
         std::make_unique<SampleContextTracker>(Reader->getProfiles());
   }
 
+  // Load pseudo probe descriptors for hybrid function samples.
+  if (Reader->profileIsProbeBased()) {
+    ProbeManager = std::make_unique<PseudoProbeManager>(M);
+    if (!ProbeManager->moduleIsProbed(M)) {
+      const char *Msg = "Hybrid profile requires SampleProfileProbePass";
+      Ctx.diagnose(DiagnosticInfoSampleProfile(Filename, Msg));
+      return false;
+    }
+  }
+
   return true;
 }
 
 ModulePass *llvm::createSampleProfileLoaderPass() {
   return new SampleProfileLoaderLegacyPass();
 }
 
 ModulePass *llvm::createSampleProfileLoaderPass(StringRef Name) {

llvm/lib/Transforms/IPO/SampleProfileProbe.cpp

 #include <vector>
 
 using namespace llvm;
 #define DEBUG_TYPE "sample-profile-probe"
 
 STATISTIC(ArtificialDbgLine,
           "Number of probes that have an artificial debug line");
 
+PseudoProbeManager::PseudoProbeManager(const Module &M) {
+  if (NamedMDNode *FuncInfo = M.getNamedMetadata(PseudoProbeDescMetadataName)) {
+    for (const auto *Operand : FuncInfo->operands()) {
+      const auto *MD = cast<MDNode>(Operand);
+      auto GUID =
+          mdconst::dyn_extract<ConstantInt>(MD->getOperand(0))->getZExtValue();
+      auto Hash =
+          mdconst::dyn_extract<ConstantInt>(MD->getOperand(1))->getZExtValue();
+      GUIDToProbeDescMap.insert({GUID, PseudoProbeDescriptor(GUID, Hash)});
+    }
+  }
+}
+
+const PseudoProbeDescriptor *
+PseudoProbeManager::getDesc(const Function &F) const {
+  auto I = GUIDToProbeDescMap.find(
+      Function::getGUID(FunctionSamples::getCanonicalFnName(F)));
+  return I == GUIDToProbeDescMap.end() ? nullptr : &I->second;
+}
+
+bool PseudoProbeManager::moduleIsProbed(const Module &M) const {
+  return M.getNamedMetadata(PseudoProbeDescMetadataName);
+}
+
+bool PseudoProbeManager::profileIsValid(const Function &F,
+                                        const FunctionSamples &Samples) const {
+  auto Desc = getDesc(F);
+  if (!Desc) {
+    LLVM_DEBUG(dbgs() << "Probe descriptor missing for Function " << F.getName()
+                      << "\n");
+    return false;
+  } else {
+    if (Desc->getFunctionHash() != Samples.getFunctionHash()) {
+      LLVM_DEBUG(dbgs() << "Hash mismatch for Function " << F.getName()
+                        << "\n");
+      return false;
+    }
+  }
+  return true;
+}
+
 SampleProfileProber::SampleProfileProber(Function &Func,
                                          const std::string &CurModuleUniqueId)
     : F(&Func), CurModuleUniqueId(CurModuleUniqueId) {
   BlockProbeIds.clear();
   CallProbeIds.clear();
   LastProbeId = (uint32_t)PseudoProbeReservedId::Last;
   computeProbeIdForBlocks();
   computeProbeIdForCallsites();

llvm/test/Transforms/SampleProfile/Inputs/pseudo-probe-profile.prof

+foo:3200:13
+ 1: 13
+ 2: 7
+ 3: 6
+ 4: 13
+ 5: 7 _Z3barv:2 _Z3foov:5
+ 6: 6 _Z3barv:4 _Z3foov:2
+ !CFGChecksum: 563022570642068

llvm/test/Transforms/SampleProfile/pseudo-probe-profile.ll

+; RUN: opt < %s -passes=pseudo-probe,sample-profile -sample-profile-file=%S/Inputs/pseudo-probe-profile.prof -pass-remarks=sample-profile -pass-remarks-output=%t.opt.yaml -S | FileCheck %s
+; RUN: FileCheck %s -check-prefix=YAML < %t.opt.yaml
+
+define dso_local i32 @foo(i32 %x, void (i32)* %f) #0 !dbg !4 {
+entry:
+  %retval = alloca i32, align 4
+  %x.addr = alloca i32, align 4
+  store i32 %x, i32* %x.addr, align 4
+  %0 = load i32, i32* %x.addr, align 4
+  %cmp = icmp eq i32 %0, 0
+  br i1 %cmp, label %if.then, label %if.else
+  ; CHECK: br i1 %cmp, label %if.then, label %if.else, !prof ![[PD1:[0-9]+]]
+
+if.then:
+  ; CHECK: call {{.*}}, !prof ![[PROF1:[0-9]+]]
+  call void %f(i32 1)
+  store i32 1, i32* %retval, align 4
+  br label %return
+
+if.else:
+  ; CHECK: call {{.*}}, !prof ![[PROF2:[0-9]+]]
+  call void %f(i32 2)
+  store i32 2, i32* %retval, align 4
+  br label %return
+
+return:
+  %1 = load i32, i32* %retval, align 4
+  ret i32 %1
+}
+
+attributes #0 = {"use-sample-profile"}
+
+; CHECK: ![[PD1]] = !{!"branch_weights", i32 8, i32 7}
+; CHECK: ![[PROF1]] = !{!"VP", i32 0, i64 7, i64 9191153033785521275, i64 5, i64 -1069303473483922844, i64 2}
+; CHECK: ![[PROF2]] = !{!"VP", i32 0, i64 6, i64 -1069303473483922844, i64 4, i64 9191153033785521275, i64 2}
+
+!llvm.module.flags = !{!9, !10}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1)
+!1 = !DIFile(filename: "test.c", directory: "")
+!2 = !{}
+!4 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !5, unit: !0, retainedNodes: !2)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
+!9 = !{i32 2, !"Dwarf Version", i32 4}
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+
+; Checking to see if YAML file is generated and contains remarks
+;YAML:  --- !Analysis
+;YAML-NEXT:  Pass:            sample-profile
+;YAML-NEXT:  Name:            AppliedSamples
+;YAML-NEXT:  DebugLoc:        { File: test.c, Line: 0, Column: 0 }
+;YAML-NEXT:  Function:        foo
+;YAML-NEXT:  Args:
+;YAML-NEXT:    - String:          'Applied '
+;YAML-NEXT:    - NumSamples:      '13'
+;YAML-NEXT:    - String:          ' samples from profile (ProbeId='
+;YAML-NEXT:    - ProbeId:         '1'
+;YAML-NEXT:    - String:          ')'
+;YAML:  --- !Analysis
+;YAML-NEXT:  Pass:            sample-profile
+;YAML-NEXT:  Name:            AppliedSamples
+;YAML-NEXT:  DebugLoc:        { File: test.c, Line: 0, Column: 0 }
+;YAML-NEXT:  Function:        foo
+;YAML-NEXT:  Args:
+;YAML-NEXT:    - String:          'Applied '
+;YAML-NEXT:    - NumSamples:      '7'
+;YAML-NEXT:    - String:          ' samples from profile (ProbeId='
+;YAML-NEXT:    - ProbeId:         '5'
+;YAML-NEXT:    - String:          ')'
+;YAML:  --- !Analysis
+;YAML-NEXT:  Pass:            sample-profile
+;YAML-NEXT:  Name:            AppliedSamples
+;YAML-NEXT:  DebugLoc:        { File: test.c, Line: 0, Column: 0 }
+;YAML-NEXT:  Function:        foo
+;YAML-NEXT:  Args:
+;YAML-NEXT:    - String:          'Applied '
+;YAML-NEXT:    - NumSamples:      '7'
+;YAML-NEXT:    - String:          ' samples from profile (ProbeId='
+;YAML-NEXT:    - ProbeId:         '2'
+;YAML-NEXT:    - String:          ')'
+;YAML:  --- !Analysis
+;YAML-NEXT:  Pass:            sample-profile
+;YAML-NEXT:  Name:            AppliedSamples
+;YAML-NEXT:  DebugLoc:        { File: test.c, Line: 0, Column: 0 }
+;YAML-NEXT:  Function:        foo
+;YAML-NEXT:  Args:
+;YAML-NEXT:    - String:          'Applied '
+;YAML-NEXT:    - NumSamples:      '6'
+;YAML-NEXT:    - String:          ' samples from profile (ProbeId='
+;YAML-NEXT:    - ProbeId:         '6'
+;YAML-NEXT:    - String:          ')'
+;YAML:  --- !Analysis
+;YAML-NEXT:  Pass:            sample-profile
+;YAML-NEXT:  Name:            AppliedSamples
+;YAML-NEXT:  DebugLoc:        { File: test.c, Line: 0, Column: 0 }
+;YAML-NEXT:  Function:        foo
+;YAML-NEXT:  Args:
+;YAML-NEXT:    - String:          'Applied '
+;YAML-NEXT:    - NumSamples:      '6'
+;YAML-NEXT:    - String:          ' samples from profile (ProbeId='
+;YAML-NEXT:    - ProbeId:         '3'
+;YAML-NEXT:    - String:          ')'
+;YAML:  --- !Analysis
+;YAML-NEXT:  Pass:            sample-profile
+;YAML-NEXT:  Name:            AppliedSamples
+;YAML-NEXT:  DebugLoc:        { File: test.c, Line: 0, Column: 0 }
+;YAML-NEXT:  Function:        foo
+;YAML-NEXT:  Args:
+;YAML-NEXT:    - String:          'Applied '
+;YAML-NEXT:    - NumSamples:      '13'
+;YAML-NEXT:    - String:          ' samples from profile (ProbeId='
+;YAML-NEXT:    - ProbeId:         '4'
+;YAML-NEXT:    - String:          ')'

llvm/test/tools/llvm-profdata/Inputs/pseudo-probe-profile.proftext

+foo:3200:13
+ 1: 13
+ 2: 7
+ 3: 6
+ 4: 13
+ 5: 7 _Z3foov:5 _Z3barv:2
+ 6: 6 _Z3barv:4 _Z3foov:2
+ !CFGChecksum: 563022570642068

llvm/test/tools/llvm-profdata/merge-probe-profile.test

+# Tests for merge of hybrid profile files.
+
+RUN: llvm-profdata merge --sample --text %p/Inputs/pseudo-probe-profile.proftext -o - | FileCheck %s --check-prefix=MERGE1
+RUN: llvm-profdata merge --sample --extbinary %p/Inputs/pseudo-probe-profile.proftext -o %t && llvm-profdata merge --sample --text %t -o - | FileCheck %s --check-prefix=MERGE1
+MERGE1: foo:3200:13
+MERGE1:  1: 13
+MERGE1:  2: 7
+MERGE1:  3: 6
+MERGE1:  4: 13
+MERGE1:  5: 7 _Z3foov:5 _Z3barv:2
+MERGE1:  6: 6 _Z3barv:4 _Z3foov:2
+MERGE1:  !CFGChecksum: 563022570642068
+
+RUN: llvm-profdata merge --sample --text %p/Inputs/pseudo-probe-profile.proftext %p/Inputs/pseudo-probe-profile.proftext -o - | FileCheck %s --check-prefix=MERGE2
+RUN: llvm-profdata merge --sample --extbinary %p/Inputs/pseudo-probe-profile.proftext %p/Inputs/pseudo-probe-profile.proftext -o %t && llvm-profdata merge --sample --text %t -o - | FileCheck %s --check-prefix=MERGE2
+MERGE2: foo:6400:26
+MERGE2:  1: 26
+MERGE2:  2: 14
+MERGE2:  3: 12
+MERGE2:  4: 26
+MERGE2:  5: 14 _Z3foov:10 _Z3barv:4
+MERGE2:  6: 12 _Z3barv:8 _Z3foov:4
+MERGE2:  !CFGChecksum: 563022570642068

llvm/tools/llvm-profdata/llvm-profdata.cpp

mergeSampleProfile(const WeightedFileVector &Inputs, SymbolRemapper *Remapper,
                    StringRef OutputFilename, ProfileFormat OutputFormat,
                    StringRef ProfileSymbolListFile, bool CompressAllSections,
                    bool UseMD5, bool GenPartialProfile, FailureMode FailMode) {
   using namespace sampleprof;
   StringMap<FunctionSamples> ProfileMap;
   SmallVector<std::unique_ptr<sampleprof::SampleProfileReader>, 5> Readers;
   LLVMContext Context;
   sampleprof::ProfileSymbolList WriterList;
+  Optional<bool> ProfileIsProbeBased;
   for (const auto &Input : Inputs) {
     auto ReaderOrErr = SampleProfileReader::create(Input.Filename, Context);
     if (std::error_code EC = ReaderOrErr.getError()) {
       warnOrExitGivenError(FailMode, EC, Input.Filename);
       continue;
     }
 
     // We need to keep the readers around until after all the files are
     // read so that we do not lose the function names stored in each
     // reader's memory. The function names are needed to write out the
     // merged profile map.
     Readers.push_back(std::move(ReaderOrErr.get()));
     const auto Reader = Readers.back().get();
     if (std::error_code EC = Reader->read()) {
       warnOrExitGivenError(FailMode, EC, Input.Filename);
       Readers.pop_back();
       continue;
     }
 
     StringMap<FunctionSamples> &Profiles = Reader->getProfiles();
+    assert(!ProfileIsProbeBased ||
+           ProfileIsProbeBased == FunctionSamples::ProfileIsProbeBased);
+    ProfileIsProbeBased = FunctionSamples::ProfileIsProbeBased;
     for (StringMap<FunctionSamples>::iterator I = Profiles.begin(),
                                               E = Profiles.end();
          I != E; ++I) {
       sampleprof_error Result = sampleprof_error::success;
       FunctionSamples Remapped =
           Remapper ? remapSamples(I->second, *Remapper, Result)
                    : FunctionSamples();
       FunctionSamples &Samples = Remapper ? Remapped : I->second;
std::error_code SampleOverlapAggregator::loadProfiles() {
 
   BaseReader = std::move(BaseReaderOrErr.get());
   TestReader = std::move(TestReaderOrErr.get());
 
   if (std::error_code EC = BaseReader->read())
     exitWithErrorCode(EC, BaseFilename);
   if (std::error_code EC = TestReader->read())
     exitWithErrorCode(EC, TestFilename);
+  if (BaseReader->profileIsProbeBased() != TestReader->profileIsProbeBased())
+    exitWithError(
+        "cannot compare probe-based profile with non-probe-based profile");
 
   // Load BaseHotThreshold and TestHotThreshold as 99-percentile threshold in
   // profile summary.
   const uint64_t HotCutoff = 990000;
   ProfileSummary &BasePS = BaseReader->getSummary();
   for (const auto &SummaryEntry : BasePS.getDetailedSummary()) {
     if (SummaryEntry.Cutoff == HotCutoff) {
       BaseHotThreshold = SummaryEntry.MinCount;