[InstrProf] Add Correlator class to read debug info

[ellis]

Summary

Extend llvm-profdata to read in a .proflite file and also a debug info file to generate a normal .profdata profile. This reduces the binary size by 8.4% when building an instrumented Clang binary without value profiling (164 MB vs 179 MB).

This work is part of the "lightweight instrumentation" RFC: https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4

[ellis]

Fix build error.

[kyulee]

llvm/lib/ProfileData/InstrProfCorrelator.cpp

91

It's nor blocking but it might be a more relevant error message when it fails for COFF -- not yet implemented or supported case.

155

It's concise to have it here but should we check this outside so that we don't need to come here after creating unnecessary DWARFDie?

162

IIRC, this die is attached to a function, so we try to get the function address from its parent?

183

My understanding is that FunctionName, CFGHash, and NumCounters are attached to this correlated die always under this flag.
Should we assert them instead? If they are null, I wonder what situation we might run into.

[ellis]

Rebase.

[ellis]

llvm/lib/ProfileData/InstrProfCorrelator.cpp

155

I'm not sure how I would move this since it needs a DWARFDie to read the variable name.

162

The DIE we are looking at is assumed to be for a static variable (the probe). The parent DIE should be the function that owns the variable and that is the address we want.

183

isDIEOfProbe() only checks if the variable name starts with __profc_, so it's not unlikely that we run into this case. We could add the artificial flag or other attributes that make it more likely that this is a probe, then we could change this to an assert.

[ellis]

Add more checks to isDIEOfProbe().

[ellis]

Fix MachO case.

[ellis]

I'm planning to improve MachO support by expanding .dSYMs. I also want to first refactor readRawCounts() in a separate diff.

[MaskRay]

(While I am happy to study/review lightweight instrumentation, I am out of town til Dec 11, so would not have time reading the patches. No need to wait for me :) )

[ellis]

Expand .dSYM paths to their objects and add a separate test for MachO. Also a bit of refactoring.

[kyulee]

Thanks for fixing MachO.
It appears there is still an issue but it seems in dwarflinker (dsymutil). As below with clang -g -mllvm -disable-vp=true -mllvm -debug-info-correlate=true -fprofile-generate -Oz -mllvm -disable-preinline, when the function is fully inlined, dsymutil seems to optimize this static data in the final dSYM.
I was trying to use -keep-function-for-static to keep this, but still has a missing location info (DW_AT_location) at dSYM, and thus we cannot correlate foo and thus miss the coverage for it.
But I think missing location info for the function static variables seems a known issue in dsymutil, and we may follow this up. cc @clayborg

static 
int foo() {
 return 1;
}
int main() {
 return foo();
}

llvm/lib/ProfileData/InstrProfCorrelator.cpp

37

Hmm. This code seems repeated 3 times in dwarfdump and gsymutil, and now here.
I think this helper might be moved to some dsym related support file. But certainly this can be followed up.

206

Die or ParentDie are references. Can they be null?

220

Can you add a high-level (example) dwarf layout for this function to match as the comment?

228

Can we use getLowAndHighPC or make one getLow?
I think 0 address (no existent of function address) is a valid case when the function is completely inlined (and thus the body is elided) -- only this static profile data remains.

llvm/lib/ProfileData/InstrProfReader.cpp

435

Can we set these value for Correlator once instead of overwriting them, which may be out-of-bound or invalid?
I guess DataSize and NameSize etc are 0, and we can assert these values under this flag.

llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp

956

I was thinking whether we need another INSTR_PROF_DATA or a similar one for DebugInfoCorrelate in the header and include them in order so that we can ensure sync these data in order in both a writer, here and a reader, correlateProfileDataImpl.

llvm/tools/llvm-profdata/llvm-profdata.cpp

151

Why is this moved? Is this relevant to this change?

[ellis]

llvm/lib/ProfileData/InstrProfCorrelator.cpp

37

Absolutely! I'll follow up in a separate diff.

206

DWARFDie has a bool operator that just calls .isValid(). It's probably safer and more clear to call .isValid() directly.

220

Good idea. I've added a comment in the header.

228

IIUC FunctionPtr is only used for value profiling to match callsite dispatches to their callees. If the function is completely inlined then there won't be value profiling for that callee.

I've changed this so that if we can't find an address then we emit a warning so we can investigate and possible fixup debug info. So for the .profdata file, an address of 0 means either "we can't find the address" or "the function is not concrete (it is inlined)".

llvm/lib/ProfileData/InstrProfReader.cpp

435

Yeah I've change this up a bit to only set them once. It's confusing because DataSize is zero but there is actual data in the Correlator.

llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp

956

I don't think adding a macro in InstrProfData.inc is the right solution because it makes it unnecessary complicated and hard to read. IIUC the worry is that someone may change the writer and forget to change the reader. If this code is changed without the right reader change then the compiler-rt test will break and hopefully the change won't land. I would like to leave this how it is, but I'm open to discussion.

[ellis]

Add requested changes.

[kyulee]

This looks good to me. Given this works for ELF while having a minor issue with the fully inlined case for MachO, could you file a bug to follow this up?
I also let other folks review this code.

[ellis]

In D114566#3185418, @kyulee wrote:

This looks good to me. Given this works for ELF while having a minor issue with the fully inlined case for MachO, could you file a bug to follow this up?
I also let other folks review this code.

I believe the MachO bug is fixed by https://reviews.llvm.org/D115565
I'm not currently aware of any more bugs, but I'll file a bug report if I find them.

[ellis]

Rebase and test.

[ellis]

Use findDsymObjectMembers() that was added in https://reviews.llvm.org/D115418

compiler-rt/test/profile/Darwin/instrprof-debug-info-correlate.c

+// Value profiling is currently not supported in lightweight mode.
+// RUN: %clang_pgogen -o %t.normal -mllvm --disable-vp=true %s
+// RUN: env LLVM_PROFILE_FILE=%t.profraw %run %t.normal
+// RUN: llvm-profdata merge -o %t.normal.profdata %t.profraw
+
+// RUN: %clang_pgogen -o %t -g -mllvm --debug-info-correlate -mllvm --disable-vp=true %s
+// RUN: env LLVM_PROFILE_FILE=%t.proflite %run %t
+// RUN: llvm-profdata merge -o %t.profdata --debug-info=%t.dSYM %t.proflite
+
+// RUN: diff %t.normal.profdata %t.profdata
+
+int foo(int a) {
+  if (a % 2)
+    return 4 * a + 1;
+  return 0;
+}
+
+int bar(int a) {
+  while (a > 100)
+    a /= 2;
+  return a;
+}
+
+typedef int (*FP)(int);
+FP Fps[3] = {foo, bar};
+
+int main() {
+  for (int i = 0; i < 5; i++)
+    Fps[i % 2](i);
+  return 0;
+}

compiler-rt/test/profile/Linux/instrprof-debug-info-correlate.c

+// Value profiling is currently not supported in lightweight mode.
+// RUN: %clang_pgogen -o %t.normal -mllvm --disable-vp=true %s
+// RUN: env LLVM_PROFILE_FILE=%t.profraw %run %t.normal
+// RUN: llvm-profdata merge -o %t.normal.profdata %t.profraw
+
+// RUN: %clang_pgogen -o %t -g -mllvm --debug-info-correlate -mllvm --disable-vp=true %s
+// RUN: env LLVM_PROFILE_FILE=%t.proflite %run %t
+// RUN: llvm-profdata merge -o %t.profdata --debug-info=%t %t.proflite
+
+// RUN: diff %t.normal.profdata %t.profdata
+
+int foo(int a) {
+  if (a % 2)
+    return 4 * a + 1;
+  return 0;
+}
+
+int bar(int a) {
+  while (a > 100)
+    a /= 2;
+  return a;
+}
+
+typedef int (*FP)(int);
+FP Fps[3] = {foo, bar};
+
+int main() {
+  for (int i = 0; i < 5; i++)
+    Fps[i % 2](i);
+  return 0;
+}

llvm/include/llvm/ProfileData/InstrProf.h

enum class instrprof_error {
   unrecognized_format,
   bad_magic,
   bad_header,
   unsupported_version,
   unsupported_hash_type,
   too_large,
   truncated,
   malformed,
+  missing_debug_info_for_correlation,
+  unexpected_debug_info_for_correlation,
+  unable_to_correlate_profile,
+  unsupported_debug_format,
   unknown_function,
   invalid_prof,
   hash_mismatch,
   count_mismatch,
   counter_overflow,
   value_site_count_mismatch,
   compress_failed,
   uncompress_failed,

llvm/include/llvm/ProfileData/InstrProfCorrelator.h

+//===- InstrProfCorrelator.h ------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+// This file defines InstrProfCorrelator used to generate PGO profiles from
+// raw profile data and debug info.
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_PROFILEDATA_INSTRPROFCORRELATOR_H
+#define LLVM_PROFILEDATA_INSTRPROFCORRELATOR_H
+
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
+#include "llvm/Object/Binary.h"
+#include "llvm/Object/ObjectFile.h"
+#include "llvm/ProfileData/InstrProf.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include <vector>
+
+namespace llvm {
+
+/// InstrProfCorrelator - A base class used to create raw instrumentation data
+/// to their functions.
+class InstrProfCorrelator {
+public:
+  static llvm::Expected<std::unique_ptr<InstrProfCorrelator>>
+  get(StringRef DebugInfoFilename);
+
+  /// Construct a ProfileData vector used to correlate raw instrumentation data
+  /// to their functions.
+  virtual Error correlateProfileData() = 0;
+
+  static const char *FunctionNameAttributeName;
+  static const char *CFGHashAttributeName;
+  static const char *NumCountersAttributeName;
+
+  enum InstrProfCorrelatorKind { CK_32Bit, CK_64Bit };
+  InstrProfCorrelatorKind getKind() const { return Kind; }
+  virtual ~InstrProfCorrelator() {}
+
+protected:
+  struct Context {
+    static llvm::Expected<std::unique_ptr<Context>>
+    get(std::unique_ptr<MemoryBuffer> Buffer, const object::ObjectFile &Obj);
+    std::unique_ptr<MemoryBuffer> Buffer;
+    /// The address range of the __llvm_prf_cnts section.
+    uint64_t CountersSectionStart;
+    uint64_t CountersSectionEnd;
+    /// True if target and host have different endian orders.
+    bool ShouldSwapBytes;
+  };
+  const std::unique_ptr<InstrProfCorrelator::Context> Ctx;
+
+  InstrProfCorrelator(InstrProfCorrelatorKind K, std::unique_ptr<Context> Ctx)
+      : Ctx(std::move(Ctx)), Kind(K) {}
+
+private:
+  static llvm::Expected<std::unique_ptr<InstrProfCorrelator>>
+  get(std::unique_ptr<MemoryBuffer> Buffer);
+
+  const InstrProfCorrelatorKind Kind;
+};
+
+/// InstrProfCorrelatorImpl - A child of InstrProfCorrelator with a template
+/// pointer type so that the ProfileData vector can be materialized.
+template <class IntPtrT>
+class InstrProfCorrelatorImpl : public InstrProfCorrelator {
+public:
+  InstrProfCorrelatorImpl(std::unique_ptr<InstrProfCorrelator::Context> Ctx);
+  static bool classof(const InstrProfCorrelator *C);
+
+  /// Return a pointer to the underlying ProfileData vector that this class
+  /// constructs.
+  const RawInstrProf::ProfileData<IntPtrT> *getDataPointer() const {
+    return Data.empty() ? nullptr : Data.data();
+  }
+
+  /// Return the number of ProfileData elements.
+  size_t getDataSize() const { return Data.size(); }
+
+  /// Return a pointer to the compressed names string that this class
+  /// constructs.
+  const char *getCompressedNamesPointer() const {
+    return CompressedNames.c_str();
+  }
+
+  /// Return the number of bytes in the compressed names string.
+  size_t getCompressedNamesSize() const { return CompressedNames.size(); }
+
+  static llvm::Expected<std::unique_ptr<InstrProfCorrelatorImpl<IntPtrT>>>
+  get(std::unique_ptr<InstrProfCorrelator::Context> Ctx,
+      const object::ObjectFile &Obj);
+
+protected:
+  std::vector<RawInstrProf::ProfileData<IntPtrT>> Data;
+  std::string CompressedNames;
+
+  Error correlateProfileData() override;
+  virtual void correlateProfileDataImpl() = 0;
+
+  void addProbe(StringRef FunctionName, uint64_t CFGHash, IntPtrT CounterOffset,
+                IntPtrT FunctionPtr, uint32_t NumCounters);
+
+private:
+  InstrProfCorrelatorImpl(InstrProfCorrelatorKind Kind,
+                          std::unique_ptr<InstrProfCorrelator::Context> Ctx)
+      : InstrProfCorrelator(Kind, std::move(Ctx)){};
+  std::vector<std::string> Names;
+
+  // Byte-swap the value if necessary.
+  template <class T> T maybeSwap(T Value) const {
+    return Ctx->ShouldSwapBytes ? sys::getSwappedBytes(Value) : Value;
+  }
+};
+
+/// DwarfInstrProfCorrelator - A child of InstrProfCorrelatorImpl that takes
+/// DWARF debug info as input to correlate profiles.
+template <class IntPtrT>
+class DwarfInstrProfCorrelator : public InstrProfCorrelatorImpl<IntPtrT> {
+public:
+  DwarfInstrProfCorrelator(std::unique_ptr<DWARFContext> DICtx,
+                           std::unique_ptr<InstrProfCorrelator::Context> Ctx)
+      : InstrProfCorrelatorImpl<IntPtrT>(std::move(Ctx)),
+        DICtx(std::move(DICtx)) {}
+
+private:
+  std::unique_ptr<DWARFContext> DICtx;
+
+  /// Return the address of the object that the provided DIE symbolizes.
+  llvm::Optional<uint64_t> getLocation(const DWARFDie &Die) const;
+
+  /// Returns true if the provided DIE symbolizes an instrumentation probe
+  /// symbol.
+  static bool isDIEOfProbe(const DWARFDie &Die);
+
+  /// Iterate over DWARF DIEs to find those that symbolize instrumentation
+  /// probes and construct the ProfileData vector and CompressedNames string.
+  ///
+  /// Here is some example DWARF for an instrumentation probe we are looking
+  /// for:
+  /// \code
+  ///   DW_TAG_subprogram
+  ///   DW_AT_low_pc  (0x0000000000000000)
+  ///   DW_AT_high_pc (0x0000000000000014)
+  ///   DW_AT_name  ("foo")
+  ///     DW_TAG_variable
+  ///       DW_AT_name  ("__profc_foo")
+  ///       DW_AT_location  (DW_OP_addr 0x0)
+  ///       DW_TAG_LLVM_annotation
+  ///         DW_AT_name  ("Function Name")
+  ///         DW_AT_const_value ("foo")
+  ///       DW_TAG_LLVM_annotation
+  ///         DW_AT_name  ("CFG Hash")
+  ///         DW_AT_const_value (12345678)
+  ///       DW_TAG_LLVM_annotation
+  ///         DW_AT_name  ("Num Counters")
+  ///         DW_AT_const_value (2)
+  ///       NULL
+  ///     NULL
+  /// \endcode
+  void correlateProfileDataImpl() override;
+};
+
+} // end namespace llvm
+
+#endif // LLVM_PROFILEDATA_INSTRPROFCORRELATOR_H

llvm/include/llvm/ProfileData/InstrProfReader.h

 
 #ifndef LLVM_PROFILEDATA_INSTRPROFREADER_H
 #define LLVM_PROFILEDATA_INSTRPROFREADER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/ProfileSummary.h"
 #include "llvm/ProfileData/InstrProf.h"
+#include "llvm/ProfileData/InstrProfCorrelator.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/LineIterator.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/OnDiskHashTable.h"
 #include "llvm/Support/SwapByteOrder.h"
 #include <algorithm>
 #include <cassert>
public:
   InstrProfIterator end() { return InstrProfIterator(); }
 
   virtual bool isIRLevelProfile() const = 0;
 
   virtual bool hasCSIRLevelProfile() const = 0;
 
   virtual bool instrEntryBBEnabled() const = 0;
 
+  /// Return true if we must provide debug info to create PGO profiles.
+  virtual bool useDebugInfoCorrelate() const { return false; }
+
   /// Return the PGO symtab. There are three different readers:
   /// Raw, Text, and Indexed profile readers. The first two types
   /// of readers are used only by llvm-profdata tool, while the indexed
   /// profile reader is also used by llvm-cov tool and the compiler (
   /// backend or frontend). Since creating PGO symtab can create
   /// significant runtime and memory overhead (as it touches data
   /// for the whole program), InstrProfSymtab for the indexed profile
   /// reader should be created on demand and it is recommended to be
public:
   Error getError() {
     if (hasError())
       return make_error<InstrProfError>(LastError, LastErrorMsg);
     return Error::success();
   }
 
   /// Factory method to create an appropriately typed reader for the given
   /// instrprof file.
-  static Expected<std::unique_ptr<InstrProfReader>> create(const Twine &Path);
+  static Expected<std::unique_ptr<InstrProfReader>>
+  create(const Twine &Path, const InstrProfCorrelator *Correlator = nullptr);
 
   static Expected<std::unique_ptr<InstrProfReader>>
-  create(std::unique_ptr<MemoryBuffer> Buffer);
+  create(std::unique_ptr<MemoryBuffer> Buffer,
+         const InstrProfCorrelator *Correlator = nullptr);
 };
 
 /// Reader for the simple text based instrprof format.
 ///
 /// This format is a simple text format that's suitable for test data. Records
 /// are separated by one or more blank lines, and record fields are separated by
 /// new lines.
 ///

 ///
 /// Templated on the unsigned type whose size matches pointers on the platform
 /// that wrote the profile.
 template <class IntPtrT>
 class RawInstrProfReader : public InstrProfReader {
 private:
   /// The profile data file contents.
   std::unique_ptr<MemoryBuffer> DataBuffer;
+  /// If available, this hold the ProfileData array used to correlate raw
+  /// instrumentation data to their functions.
+  const InstrProfCorrelatorImpl<IntPtrT> *Correlator;
   bool ShouldSwapBytes;
   // The value of the version field of the raw profile data header. The lower 56
   // bits specifies the format version and the most significant 8 bits specify
   // the variant types of the profile.
   uint64_t Version;
   uint64_t CountersDelta;
   uint64_t NamesDelta;
   const RawInstrProf::ProfileData<IntPtrT> *Data;
   const RawInstrProf::ProfileData<IntPtrT> *DataEnd;
   const uint64_t *CountersStart;
   const char *NamesStart;
-  uint64_t NamesSize;
+  const char *NamesEnd;
   // After value profile is all read, this pointer points to
   // the header of next profile data (if exists)
   const uint8_t *ValueDataStart;
   uint32_t ValueKindLast;
   uint32_t CurValueDataSize;
 
   uint64_t BinaryIdsSize;
   const uint8_t *BinaryIdsStart;
 
 public:
-  RawInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer)
-      : DataBuffer(std::move(DataBuffer)) {}
+  RawInstrProfReader(std::unique_ptr<MemoryBuffer> DataBuffer,
+                     const InstrProfCorrelator *Correlator)
+      : DataBuffer(std::move(DataBuffer)),
+        Correlator(dyn_cast_or_null<const InstrProfCorrelatorImpl<IntPtrT>>(
+            Correlator)) {}
   RawInstrProfReader(const RawInstrProfReader &) = delete;
   RawInstrProfReader &operator=(const RawInstrProfReader &) = delete;
 
   static bool hasFormat(const MemoryBuffer &DataBuffer);
   Error readHeader() override;
   Error readNextRecord(NamedInstrProfRecord &Record) override;
   Error printBinaryIds(raw_ostream &OS) override;
 
   bool isIRLevelProfile() const override {
     return (Version & VARIANT_MASK_IR_PROF) != 0;
   }
 
   bool hasCSIRLevelProfile() const override {
     return (Version & VARIANT_MASK_CSIR_PROF) != 0;
   }
 
   bool instrEntryBBEnabled() const override {
     return (Version & VARIANT_MASK_INSTR_ENTRY) != 0;
   }
 
+  bool useDebugInfoCorrelate() const override {
+    return (Version & VARIANT_MASK_DBG_CORRELATE) != 0;
+  }
+
   InstrProfSymtab &getSymtab() override {
     assert(Symtab.get());
     return *Symtab.get();
   }
 
 private:
   Error createSymtab(InstrProfSymtab &Symtab);
   Error readNextHeader(const char *CurrentPos);

llvm/lib/ProfileData/CMakeLists.txt

 add_llvm_component_library(LLVMProfileData
   GCOV.cpp
   InstrProf.cpp
+  InstrProfCorrelator.cpp
   InstrProfReader.cpp
   InstrProfWriter.cpp
   ProfileSummaryBuilder.cpp
   SampleProf.cpp
   SampleProfReader.cpp
   SampleProfWriter.cpp
   RawMemProfReader.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/ProfileData
 
   DEPENDS
   intrinsics_gen
 
   LINK_COMPONENTS
   Core
   Support
   Demangle
+  Object
+  DebugInfoDWARF
   )
 
 add_subdirectory(Coverage)

llvm/lib/ProfileData/InstrProf.cpp

case instrprof_error::too_large:
     OS << "too much profile data";
     break;
   case instrprof_error::truncated:
     OS << "truncated profile data";
     break;
   case instrprof_error::malformed:
     OS << "malformed instrumentation profile data";
     break;
+  case instrprof_error::missing_debug_info_for_correlation:
+    OS << "debug info for correlation is required";
+    break;
+  case instrprof_error::unexpected_debug_info_for_correlation:
+    OS << "debug info for correlation is not necessary";
+    break;
+  case instrprof_error::unable_to_correlate_profile:
+    OS << "unable to correlate profile";
+    break;
+  case instrprof_error::unsupported_debug_format:
+    OS << "unsupported debug info format (only DWARF is supported)";
+    break;
   case instrprof_error::invalid_prof:
     OS << "invalid profile created. Please file a bug "
           "at: " BUG_REPORT_URL
           " and include the profraw files that caused this error.";
     break;
   case instrprof_error::unknown_function:
     OS << "no profile data available for function";
     break;

llvm/lib/ProfileData/InstrProfCorrelator.cpp

+//===-- InstrProfCorrelator.cpp -------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ProfileData/InstrProfCorrelator.h"
+#include "llvm/Object/MachO.h"
+#include "llvm/Support/Debug.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/Path.h"
+
+#define DEBUG_TYPE "correlator"
+
+using namespace llvm;
+
+/// Get the __llvm_prf_cnts section.
+Expected<object::SectionRef> getCountersSection(const object::ObjectFile &Obj) {
+  for (auto &Section : Obj.sections())
+    if (auto SectionName = Section.getName())
+      if (SectionName.get() == INSTR_PROF_CNTS_SECT_NAME)
+        return Section;
+  return make_error<InstrProfError>(
+      instrprof_error::unable_to_correlate_profile);
+}
+
+const char *InstrProfCorrelator::FunctionNameAttributeName = "Function Name";
+const char *InstrProfCorrelator::CFGHashAttributeName = "CFG Hash";
+const char *InstrProfCorrelator::NumCountersAttributeName = "Num Counters";
+
+llvm::Expected<std::unique_ptr<InstrProfCorrelator::Context>>
+InstrProfCorrelator::Context::get(std::unique_ptr<MemoryBuffer> Buffer,
+                                  const object::ObjectFile &Obj) {
+  auto CountersSection = getCountersSection(Obj);
+  if (auto Err = CountersSection.takeError())
+    return std::move(Err);
+  auto C = std::make_unique<Context>();
+  C->Buffer = std::move(Buffer);
+  C->CountersSectionStart = CountersSection->getAddress();
+  C->CountersSectionEnd = C->CountersSectionStart + CountersSection->getSize();
+  C->ShouldSwapBytes = Obj.isLittleEndian() != sys::IsLittleEndianHost;
+  return C;
+}
+
+llvm::Expected<std::unique_ptr<InstrProfCorrelator>>
+InstrProfCorrelator::get(StringRef DebugInfoFilename) {
+  auto DsymObjectsOrErr =
+      object::MachOObjectFile::findDsymObjectMembers(DebugInfoFilename);
+  if (auto Err = DsymObjectsOrErr.takeError())
+    return std::move(Err);
+  if (!DsymObjectsOrErr->empty()) {
+    // TODO: Enable profile correlation when there are multiple objects in a
+    // dSYM bundle.
+    if (DsymObjectsOrErr->size() > 1)
+      return createStringError(
+          std::error_code(),
+          "Profile correlation using multiple objects is not yet supported");
+    DebugInfoFilename = *DsymObjectsOrErr->begin();
+  }
+  auto BufferOrErr =
+      errorOrToExpected(MemoryBuffer::getFile(DebugInfoFilename));
+  if (auto Err = BufferOrErr.takeError())
+    return std::move(Err);
+
+  return get(std::move(*BufferOrErr));
+}
+
+llvm::Expected<std::unique_ptr<InstrProfCorrelator>>
+InstrProfCorrelator::get(std::unique_ptr<MemoryBuffer> Buffer) {
+  auto BinOrErr = object::createBinary(*Buffer);
+  if (auto Err = BinOrErr.takeError())
+    return std::move(Err);
+
+  if (auto *Obj = dyn_cast<object::ObjectFile>(BinOrErr->get())) {
+    auto CtxOrErr = Context::get(std::move(Buffer), *Obj);
+    if (auto Err = CtxOrErr.takeError())
+      return std::move(Err);
+    auto T = Obj->makeTriple();
+    if (T.isArch64Bit())
+      return InstrProfCorrelatorImpl<uint64_t>::get(std::move(*CtxOrErr), *Obj);
+    if (T.isArch32Bit())
+      return InstrProfCorrelatorImpl<uint32_t>::get(std::move(*CtxOrErr), *Obj);
+  }
+  return make_error<InstrProfError>(
+      instrprof_error::unable_to_correlate_profile);
+}
+
+template <>
+InstrProfCorrelatorImpl<uint32_t>::InstrProfCorrelatorImpl(
+    std::unique_ptr<InstrProfCorrelator::Context> Ctx)
+    : InstrProfCorrelatorImpl(InstrProfCorrelatorKind::CK_32Bit,
+                              std::move(Ctx)) {}
+template <>
+InstrProfCorrelatorImpl<uint64_t>::InstrProfCorrelatorImpl(
+    std::unique_ptr<InstrProfCorrelator::Context> Ctx)
+    : InstrProfCorrelatorImpl(InstrProfCorrelatorKind::CK_64Bit,
+                              std::move(Ctx)) {}
+template <>
+bool InstrProfCorrelatorImpl<uint32_t>::classof(const InstrProfCorrelator *C) {
+  return C->getKind() == InstrProfCorrelatorKind::CK_32Bit;
+}
+template <>
+bool InstrProfCorrelatorImpl<uint64_t>::classof(const InstrProfCorrelator *C) {
+  return C->getKind() == InstrProfCorrelatorKind::CK_64Bit;
+}
+
+template <class IntPtrT>
+llvm::Expected<std::unique_ptr<InstrProfCorrelatorImpl<IntPtrT>>>
+InstrProfCorrelatorImpl<IntPtrT>::get(
+    std::unique_ptr<InstrProfCorrelator::Context> Ctx,
+    const object::ObjectFile &Obj) {
+  if (Obj.isELF() || Obj.isMachO()) {
+    auto DICtx = DWARFContext::create(Obj);
+    return std::make_unique<DwarfInstrProfCorrelator<IntPtrT>>(std::move(DICtx),
+                                                               std::move(Ctx));
+  }
+  return make_error<InstrProfError>(instrprof_error::unsupported_debug_format);
+}
+
+template <class IntPtrT>
+Error InstrProfCorrelatorImpl<IntPtrT>::correlateProfileData() {
+  assert(Data.empty() && CompressedNames.empty() && Names.empty());
+  correlateProfileDataImpl();
+  auto Result =
+      collectPGOFuncNameStrings(Names, /*doCompression=*/true, CompressedNames);
+  Names.clear();
+  return Result;
+}
+
+template <class IntPtrT>
+void InstrProfCorrelatorImpl<IntPtrT>::addProbe(StringRef FunctionName,
+                                                uint64_t CFGHash,
+                                                IntPtrT CounterOffset,
+                                                IntPtrT FunctionPtr,
+                                                uint32_t NumCounters) {
+  Data.push_back({
+      maybeSwap<uint64_t>(IndexedInstrProf::ComputeHash(FunctionName)),
+      maybeSwap<uint64_t>(CFGHash),
+      // In this mode, CounterPtr actually stores the section relative address
+      // of the counter.
+      maybeSwap<IntPtrT>(CounterOffset),
+      maybeSwap<IntPtrT>(FunctionPtr),
+      // TODO: Value profiling is not yet supported.
+      /*ValuesPtr=*/maybeSwap<IntPtrT>(0),
+      maybeSwap<uint32_t>(NumCounters),
+      /*NumValueSites=*/{maybeSwap<uint16_t>(0), maybeSwap<uint16_t>(0)},
+  });
+  Names.push_back(FunctionName.str());
+}
+
+template <class IntPtrT>
+llvm::Optional<uint64_t>
+DwarfInstrProfCorrelator<IntPtrT>::getLocation(const DWARFDie &Die) const {
+  auto Locations = Die.getLocations(dwarf::DW_AT_location);
+  if (!Locations) {
+    consumeError(Locations.takeError());
+    return {};
+  }
+  auto &DU = *Die.getDwarfUnit();
+  for (auto &Location : *Locations) {
+    auto AddressSize = DU.getAddressByteSize();
+    DataExtractor Data(Location.Expr, DICtx->isLittleEndian(), AddressSize);
+    DWARFExpression Expr(Data, AddressSize);
+    for (auto &Op : Expr)
+      if (Op.getCode() == dwarf::DW_OP_addr)
+        return Op.getRawOperand(0);
+  }
+  return {};
+}
+
+template <class IntPtrT>
+bool DwarfInstrProfCorrelator<IntPtrT>::isDIEOfProbe(const DWARFDie &Die) {
+  const auto &ParentDie = Die.getParent();
+  if (!Die.isValid() || !ParentDie.isValid() || Die.isNULL())
+    return false;
+  if (Die.getTag() != dwarf::DW_TAG_variable)
+    return false;
+  if (!ParentDie.isSubprogramDIE())
+    return false;
+  if (!Die.hasChildren())
+    return false;
+  if (const char *Name = Die.getName(DINameKind::ShortName))
+    return StringRef(Name).startswith(getInstrProfCountersVarPrefix());
+  return false;
+}
+
+template <class IntPtrT>
+void DwarfInstrProfCorrelator<IntPtrT>::correlateProfileDataImpl() {
+  auto maybeAddProbe = [&](DWARFDie Die) {
+    if (!isDIEOfProbe(Die))
+      return;
+    Optional<const char *> FunctionName;
+    Optional<uint64_t> CFGHash;
+    Optional<uint64_t> CounterPtr = getLocation(Die);
+    auto FunctionPtr =
+        dwarf::toAddress(Die.getParent().find(dwarf::DW_AT_low_pc));
+    Optional<uint64_t> NumCounters;
+    for (const DWARFDie &Child : Die.children()) {
+      if (Child.getTag() != dwarf::DW_TAG_LLVM_annotation)
+        continue;
+      auto AnnotationFormName = Child.find(dwarf::DW_AT_name);
+      auto AnnotationFormValue = Child.find(dwarf::DW_AT_const_value);
+      if (!AnnotationFormName || !AnnotationFormValue)
+        continue;
+      auto AnnotationNameOrErr = AnnotationFormName->getAsCString();
+      if (auto Err = AnnotationNameOrErr.takeError()) {
+        consumeError(std::move(Err));
+        continue;
+      }
+      StringRef AnnotationName = *AnnotationNameOrErr;
+      if (AnnotationName.compare(
+              InstrProfCorrelator::FunctionNameAttributeName) == 0) {
+        if (auto EC =
+                AnnotationFormValue->getAsCString().moveInto(FunctionName))
+          consumeError(std::move(EC));
+      } else if (AnnotationName.compare(
+                     InstrProfCorrelator::CFGHashAttributeName) == 0) {
+        CFGHash = AnnotationFormValue->getAsUnsignedConstant();
+      } else if (AnnotationName.compare(
+                     InstrProfCorrelator::NumCountersAttributeName) == 0) {
+        NumCounters = AnnotationFormValue->getAsUnsignedConstant();
+      }
+    }
+    if (!FunctionName || !CFGHash || !CounterPtr || !NumCounters) {
+      LLVM_DEBUG(dbgs() << "Incomplete DIE for probe\n\tFunctionName: "
+                        << FunctionName << "\n\tCFGHash: " << CFGHash
+                        << "\n\tCounterPtr: " << CounterPtr
+                        << "\n\tNumCounters: " << NumCounters);
+      LLVM_DEBUG(Die.dump(dbgs()));
+      return;
+    }
+    uint64_t CountersStart = this->Ctx->CountersSectionStart;
+    uint64_t CountersEnd = this->Ctx->CountersSectionEnd;
+    if (*CounterPtr < CountersStart || *CounterPtr >= CountersEnd) {
+      LLVM_DEBUG(
+          dbgs() << "CounterPtr out of range for probe\n\tFunction Name: "
+                 << FunctionName << "\n\tExpected: [0x"
+                 << Twine::utohexstr(CountersStart) << ", 0x"
+                 << Twine::utohexstr(CountersEnd) << ")\n\tActual: 0x"
+                 << Twine::utohexstr(*CounterPtr));
+      LLVM_DEBUG(Die.dump(dbgs()));
+      return;
+    }
+    if (!FunctionPtr) {
+      LLVM_DEBUG(dbgs() << "Could not find address of " << *FunctionName
+                        << "\n");
+      LLVM_DEBUG(Die.dump(dbgs()));
+    }
+    this->addProbe(*FunctionName, *CFGHash, *CounterPtr - CountersStart,
+                   FunctionPtr.getValueOr(0), *NumCounters);
+  };
+  for (auto &CU : DICtx->normal_units())
+    for (const auto &Entry : CU->dies())
+      maybeAddProbe(DWARFDie(CU.get(), &Entry));
+  for (auto &CU : DICtx->dwo_units())
+    for (const auto &Entry : CU->dies())
+      maybeAddProbe(DWARFDie(CU.get(), &Entry));
+}

llvm/lib/ProfileData/InstrProfReader.cpp

setupMemoryBuffer(const Twine &Path) {
   return std::move(BufferOrErr.get());
 }
 
 static Error initializeReader(InstrProfReader &Reader) {
   return Reader.readHeader();
 }
 
 Expected<std::unique_ptr<InstrProfReader>>
-InstrProfReader::create(const Twine &Path) {
+InstrProfReader::create(const Twine &Path,
+                        const InstrProfCorrelator *Correlator) {
   // Set up the buffer to read.
   auto BufferOrError = setupMemoryBuffer(Path);
   if (Error E = BufferOrError.takeError())
     return std::move(E);
-  return InstrProfReader::create(std::move(BufferOrError.get()));
+  return InstrProfReader::create(std::move(BufferOrError.get()), Correlator);
 }
 
 Expected<std::unique_ptr<InstrProfReader>>
-InstrProfReader::create(std::unique_ptr<MemoryBuffer> Buffer) {
+InstrProfReader::create(std::unique_ptr<MemoryBuffer> Buffer,
+                        const InstrProfCorrelator *Correlator) {
+  // Sanity check the buffer.
   if (uint64_t(Buffer->getBufferSize()) > std::numeric_limits<uint64_t>::max())
     return make_error<InstrProfError>(instrprof_error::too_large);
 
   if (Buffer->getBufferSize() == 0)
     return make_error<InstrProfError>(instrprof_error::empty_raw_profile);
 
   std::unique_ptr<InstrProfReader> Result;
   // Create the reader.
   if (IndexedInstrProfReader::hasFormat(*Buffer))
     Result.reset(new IndexedInstrProfReader(std::move(Buffer)));
   else if (RawInstrProfReader64::hasFormat(*Buffer))
-    Result.reset(new RawInstrProfReader64(std::move(Buffer)));
+    Result.reset(new RawInstrProfReader64(std::move(Buffer), Correlator));
   else if (RawInstrProfReader32::hasFormat(*Buffer))
-    Result.reset(new RawInstrProfReader32(std::move(Buffer)));
+    Result.reset(new RawInstrProfReader32(std::move(Buffer), Correlator));
   else if (TextInstrProfReader::hasFormat(*Buffer))
     Result.reset(new TextInstrProfReader(std::move(Buffer)));
   else
     return make_error<InstrProfError>(instrprof_error::unrecognized_format);
 
   // Initialize the reader and return the result.
   if (Error E = initializeReader(*Result))
     return std::move(E);
Error RawInstrProfReader<IntPtrT>::readNextHeader(const char *CurrentPos) {
 
   // There's another profile to read, so we need to process the header.
   auto *Header = reinterpret_cast<const RawInstrProf::Header *>(CurrentPos);
   return readHeader(*Header);
 }
 
 template <class IntPtrT>
 Error RawInstrProfReader<IntPtrT>::createSymtab(InstrProfSymtab &Symtab) {
-  if (Error E = Symtab.create(StringRef(NamesStart, NamesSize)))
+  if (Error E = Symtab.create(StringRef(NamesStart, NamesEnd - NamesStart)))
     return error(std::move(E));
   for (const RawInstrProf::ProfileData<IntPtrT> *I = Data; I != DataEnd; ++I) {
     const IntPtrT FPtr = swap(I->FunctionPointer);
     if (!FPtr)
       continue;
     Symtab.mapAddress(FPtr, I->NameRef);
   }
   return success();
 }
 
 template <class IntPtrT>
 Error RawInstrProfReader<IntPtrT>::readHeader(
     const RawInstrProf::Header &Header) {
   Version = swap(Header.Version);
   if (GET_VERSION(Version) != RawInstrProf::Version)
     return error(instrprof_error::unsupported_version);
+  if (useDebugInfoCorrelate() && !Correlator)
+    return error(instrprof_error::missing_debug_info_for_correlation);
+  if (!useDebugInfoCorrelate() && Correlator)
+    return error(instrprof_error::unexpected_debug_info_for_correlation);
 
   BinaryIdsSize = swap(Header.BinaryIdsSize);
   if (BinaryIdsSize % sizeof(uint64_t))
     return error(instrprof_error::bad_header);
 
   CountersDelta = swap(Header.CountersDelta);
   NamesDelta = swap(Header.NamesDelta);
   auto DataSize = swap(Header.DataSize);
   auto PaddingBytesBeforeCounters = swap(Header.PaddingBytesBeforeCounters);
   auto CountersSize = swap(Header.CountersSize);
   auto PaddingBytesAfterCounters = swap(Header.PaddingBytesAfterCounters);
-  NamesSize = swap(Header.NamesSize);
+  auto NamesSize = swap(Header.NamesSize);
   ValueKindLast = swap(Header.ValueKindLast);
 
   auto DataSizeInBytes = DataSize * sizeof(RawInstrProf::ProfileData<IntPtrT>);
   auto PaddingSize = getNumPaddingBytes(NamesSize);
 
   // Profile data starts after profile header and binary ids if exist.
   ptrdiff_t DataOffset = sizeof(RawInstrProf::Header) + BinaryIdsSize;
   ptrdiff_t CountersOffset =
       DataOffset + DataSizeInBytes + PaddingBytesBeforeCounters;
   ptrdiff_t NamesOffset = CountersOffset + (sizeof(uint64_t) * CountersSize) +
                           PaddingBytesAfterCounters;
   ptrdiff_t ValueDataOffset = NamesOffset + NamesSize + PaddingSize;
 
   auto *Start = reinterpret_cast<const char *>(&Header);
   if (Start + ValueDataOffset > DataBuffer->getBufferEnd())
     return error(instrprof_error::bad_header);
 
+  if (Correlator) {
+    // These sizes in the raw file are zero because we constructed them in the
+    // Correlator.
+    assert(DataSize == 0 && NamesSize == 0);
+    assert(CountersDelta == 0 && NamesDelta == 0);
+    Data = Correlator->getDataPointer();
+    DataEnd = Data + Correlator->getDataSize();
+    NamesStart = Correlator->getCompressedNamesPointer();
+    NamesEnd = NamesStart + Correlator->getCompressedNamesSize();
+  } else {
     Data = reinterpret_cast<const RawInstrProf::ProfileData<IntPtrT> *>(
         Start + DataOffset);
     DataEnd = Data + DataSize;
+    NamesStart = Start + NamesOffset;
+    NamesEnd = NamesStart + NamesSize;
+  }
 
   // Binary ids start just after the header.
   BinaryIdsStart =
       reinterpret_cast<const uint8_t *>(&Header) + sizeof(RawInstrProf::Header);
   CountersStart = reinterpret_cast<const uint64_t *>(Start + CountersOffset);
-  NamesStart = Start + NamesOffset;
   ValueDataStart = reinterpret_cast<const uint8_t *>(Start + ValueDataOffset);
 
   const uint8_t *BufferEnd = (const uint8_t *)DataBuffer->getBufferEnd();
   if (BinaryIdsStart + BinaryIdsSize > BufferEnd)
     return error(instrprof_error::bad_header);
 
   std::unique_ptr<InstrProfSymtab> NewSymtab = std::make_unique<InstrProfSymtab>();
   if (Error E = createSymtab(*NewSymtab.get()))
     return E;
 
   Symtab = std::move(NewSymtab);
   return success();
 }
 
 template <class IntPtrT>

 
 template <class IntPtrT>
 Error RawInstrProfReader<IntPtrT>::readRawCounts(
     InstrProfRecord &Record) {
   uint32_t NumCounters = swap(Data->NumCounters);
   if (NumCounters == 0)
     return error(instrprof_error::malformed, "number of counters is zero");
 
+  ArrayRef<uint64_t> RawCounts;
+  if (Correlator) {
+    uint64_t CounterOffset = swap<IntPtrT>(Data->CounterPtr) / sizeof(uint64_t);
+    RawCounts =
+        makeArrayRef<uint64_t>(CountersStart + CounterOffset, NumCounters);
+  } else {
     IntPtrT CounterPtr = Data->CounterPtr;
-  auto *NamesStartAsCounter = reinterpret_cast<const uint64_t *>(NamesStart);
-  ptrdiff_t MaxNumCounters = NamesStartAsCounter - CountersStart;
-
-  // Check bounds. Note that the counter pointer embedded in the data record
-  // may itself be corrupt.
-  if (MaxNumCounters < 0 || NumCounters > (uint32_t)MaxNumCounters)
-    return error(instrprof_error::malformed,
-                 "counter pointer is out of bounds");
-
-  // We need to compute the in-buffer counter offset from the in-memory address
-  // distance. The initial CountersDelta is the in-memory address difference
-  // start(__llvm_prf_cnts)-start(__llvm_prf_data), so SrcData->CounterPtr -
-  // CountersDelta computes the offset into the in-buffer counter section.
-  //
-  // CountersDelta decreases as we advance to the next data record.
     ptrdiff_t CounterOffset = getCounterOffset(CounterPtr);
-  CountersDelta -= sizeof(*Data);
     if (CounterOffset < 0)
       return error(
           instrprof_error::malformed,
           ("counter offset " + Twine(CounterOffset) + " is negative").str());
 
+    // Check bounds. Note that the counter pointer embedded in the data record
+    // may itself be corrupt.
+    auto *NamesStartAsCounter = reinterpret_cast<const uint64_t *>(NamesStart);
+    ptrdiff_t MaxNumCounters = NamesStartAsCounter - CountersStart;
+    if (MaxNumCounters < 0 || NumCounters > (uint32_t)MaxNumCounters)
+      return error(instrprof_error::malformed,
+                   "counter pointer is out of bounds");
+    // We need to compute the in-buffer counter offset from the in-memory
+    // address distance. The initial CountersDelta is the in-memory address
+    // difference start(__llvm_prf_cnts)-start(__llvm_prf_data), so
+    // SrcData->CounterPtr - CountersDelta computes the offset into the
+    // in-buffer counter section.
     if (CounterOffset > MaxNumCounters)
       return error(instrprof_error::malformed,
                    ("counter offset " + Twine(CounterOffset) +
                     " is greater than the maximum number of counters " +
                     Twine((uint32_t)MaxNumCounters))
                        .str());
 
     if (((uint32_t)CounterOffset + NumCounters) > (uint32_t)MaxNumCounters)
       return error(instrprof_error::malformed,
                    ("number of counters " +
                     Twine(((uint32_t)CounterOffset + NumCounters)) +
                     " is greater than the maximum number of counters " +
                     Twine((uint32_t)MaxNumCounters))
                        .str());
+    // CountersDelta decreases as we advance to the next data record.
+    CountersDelta -= sizeof(*Data);
 
-  auto RawCounts = makeArrayRef(getCounter(CounterOffset), NumCounters);
+    RawCounts = makeArrayRef(getCounter(CounterOffset), NumCounters);
+  }
 
   if (ShouldSwapBytes) {
     Record.Counts.clear();
     Record.Counts.reserve(RawCounts.size());
     for (uint64_t Count : RawCounts)
       Record.Counts.push_back(swap(Count));
   } else
     Record.Counts = RawCounts;

llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp

 #include "llvm/IR/Instruction.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/Module.h"
 #include "llvm/IR/Type.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/ProfileData/InstrProf.h"
+#include "llvm/ProfileData/InstrProfCorrelator.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 #include "llvm/Transforms/Utils/SSAUpdater.h"
 #include <algorithm>
InstrProfiling::getOrCreateRegionCounters(InstrProfIncrementInst *Inc) {
   CounterPtr->setAlignment(Align(8));
   MaybeSetComdat(CounterPtr);
   CounterPtr->setLinkage(Linkage);
   PD.RegionCounters = CounterPtr;
   if (DebugInfoCorrelate) {
     if (auto *SP = Fn->getSubprogram()) {
       DIBuilder DB(*M, true, SP->getUnit());
       Metadata *FunctionNameAnnotation[] = {
-          MDString::get(Ctx, "Function Name"),
+          MDString::get(Ctx, InstrProfCorrelator::FunctionNameAttributeName),
           MDString::get(Ctx, getPGOFuncNameVarInitializer(NamePtr)),
       };
       Metadata *CFGHashAnnotation[] = {
-          MDString::get(Ctx, "CFG Hash"),
+          MDString::get(Ctx, InstrProfCorrelator::CFGHashAttributeName),
           ConstantAsMetadata::get(Inc->getHash()),
       };
       Metadata *NumCountersAnnotation[] = {
-          MDString::get(Ctx, "Num Counters"),
+          MDString::get(Ctx, InstrProfCorrelator::NumCountersAttributeName),
           ConstantAsMetadata::get(Inc->getNumCounters()),
       };
       auto Annotations = DB.getOrCreateArray({
           MDNode::get(Ctx, FunctionNameAnnotation),
           MDNode::get(Ctx, CFGHashAnnotation),
           MDNode::get(Ctx, NumCountersAnnotation),
       });
       auto *DICounter = DB.createGlobalVariableExpression(
           SP, CounterPtr->getName(), /*LinkageName=*/StringRef(), SP->getFile(),
           /*LineNo=*/0, DB.createUnspecifiedType("Profile Data Type"),
           CounterPtr->hasLocalLinkage(), /*IsDefined=*/true, /*Expr=*/nullptr,

llvm/tools/llvm-profdata/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   Core
   ProfileData
+  Object
+  DebugInfoDWARF
   Support
   )
 
 add_llvm_tool(llvm-profdata
   llvm-profdata.cpp
 
   DEPENDS
   intrinsics_gen
   )

llvm/tools/llvm-profdata/llvm-profdata.cpp

 //===- llvm-profdata.cpp - LLVM profile data tool -------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // llvm-profdata merges .profdata files.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/IR/LLVMContext.h"
+#include "llvm/Object/Binary.h"
+#include "llvm/ProfileData/InstrProfCorrelator.h"
 #include "llvm/ProfileData/InstrProfReader.h"
 #include "llvm/ProfileData/InstrProfWriter.h"
 #include "llvm/ProfileData/ProfileCommon.h"
 #include "llvm/ProfileData/RawMemProfReader.h"
 #include "llvm/ProfileData/SampleProfReader.h"
 #include "llvm/ProfileData/SampleProfWriter.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Discriminator.h"
if (ShowHint) {
     }
 
     if (!Hint.empty())
       errs() << Hint << "\n";
   }
 }
 
 namespace {
 /// A remapper from original symbol names to new symbol names based on a file
 /// containing a list of mappings from old name to new name.
 class SymbolRemapper {
   std::unique_ptr<MemoryBuffer> File;
   DenseMap<StringRef, StringRef> RemappingTable;
 
 public:
   /// Build a SymbolRemapper from a file containing a list of old/new symbols.
   static std::unique_ptr<SymbolRemapper> create(StringRef InputFile) {
for (auto &I : *Reader) {
 
     WC->Writer.overlapRecord(std::move(I), Overlap, FuncOverlap, FuncFilter);
     FuncOverlap.dump(OS);
   }
 }
 
 /// Load an input into a writer context.
 static void loadInput(const WeightedFile &Input, SymbolRemapper *Remapper,
+                      const InstrProfCorrelator *Correlator,
                       WriterContext *WC) {
   std::unique_lock<std::mutex> CtxGuard{WC->Lock};
 
   // Copy the filename, because llvm::ThreadPool copied the input "const
   // WeightedFile &" by value, making a reference to the filename within it
   // invalid outside of this packaged task.
   std::string Filename = Input.Filename;
 
-  auto ReaderOrErr = InstrProfReader::create(Input.Filename);
+  auto ReaderOrErr = InstrProfReader::create(Input.Filename, Correlator);
   if (Error E = ReaderOrErr.takeError()) {
     // Skip the empty profiles by returning sliently.
     instrprof_error IPE = InstrProfError::take(std::move(E));
     if (IPE != instrprof_error::empty_raw_profile)
       WC->Errors.emplace_back(make_error<InstrProfError>(IPE), Filename);
     return;
   }
 
if (OutputFormat == PF_Text) {
     if (Output.is_displayed())
       exitWithError("cannot write a non-text format profile to the terminal");
     if (Error E = Writer.write(Output))
       warn(std::move(E));
   }
 }
 
 static void mergeInstrProfile(const WeightedFileVector &Inputs,
+                              StringRef DebugInfoFilename,
                               SymbolRemapper *Remapper,
                               StringRef OutputFilename,
                               ProfileFormat OutputFormat, bool OutputSparse,
                               unsigned NumThreads, FailureMode FailMode) {
   if (OutputFormat != PF_Binary && OutputFormat != PF_Compact_Binary &&
       OutputFormat != PF_Ext_Binary && OutputFormat != PF_Text)
     exitWithError("unknown format is specified");
 
+  std::unique_ptr<InstrProfCorrelator> Correlator;
+  if (!DebugInfoFilename.empty()) {
+    if (auto Err =
+            InstrProfCorrelator::get(DebugInfoFilename).moveInto(Correlator))
+      exitWithError(std::move(Err), DebugInfoFilename);
+    if (auto Err = Correlator->correlateProfileData())
+      exitWithError(std::move(Err), DebugInfoFilename);
+  }
+
   std::mutex ErrorLock;
   SmallSet<instrprof_error, 4> WriterErrorCodes;
 
   // If NumThreads is not specified, auto-detect a good default.
   if (NumThreads == 0)
     NumThreads = std::min(hardware_concurrency().compute_thread_count(),
                           unsigned((Inputs.size() + 1) / 2));
   // FIXME: There's a bug here, where setting NumThreads = Inputs.size() fails
   // the merge_empty_profile.test because the InstrProfWriter.ProfileKind isn't
   // merged, thus the emitted file ends up with a PF_Unknown kind.
 
   // Initialize the writer contexts.
   SmallVector<std::unique_ptr<WriterContext>, 4> Contexts;
   for (unsigned I = 0; I < NumThreads; ++I)
     Contexts.emplace_back(std::make_unique<WriterContext>(
         OutputSparse, ErrorLock, WriterErrorCodes));
 
   if (NumThreads == 1) {
     for (const auto &Input : Inputs)
-      loadInput(Input, Remapper, Contexts[0].get());
+      loadInput(Input, Remapper, Correlator.get(), Contexts[0].get());
   } else {
     ThreadPool Pool(hardware_concurrency(NumThreads));
 
     // Load the inputs in parallel (N/NumThreads serial steps).
     unsigned Ctx = 0;
     for (const auto &Input : Inputs) {
-      Pool.async(loadInput, Input, Remapper, Contexts[Ctx].get());
+      Pool.async(loadInput, Input, Remapper, Correlator.get(),
+                 Contexts[Ctx].get());
       Ctx = (Ctx + 1) % NumThreads;
     }
     Pool.wait();
 
     // Merge the writer contexts together (~ lg(NumThreads) serial steps).
     unsigned Mid = Contexts.size() / 2;
     unsigned End = Contexts.size();
     assert(Mid > 0 && "Expected more than one context");
static void supplementInstrProfile(
   if (std::error_code EC = Reader->read())
     exitWithErrorCode(EC, SampleFilename);
 
   // Read instr profile.
   std::mutex ErrorLock;
   SmallSet<instrprof_error, 4> WriterErrorCodes;
   auto WC = std::make_unique<WriterContext>(OutputSparse, ErrorLock,
                                             WriterErrorCodes);
-  loadInput(Inputs[0], nullptr, WC.get());
+  loadInput(Inputs[0], nullptr, nullptr, WC.get());
   if (WC->Errors.size() > 0)
     exitWithError(std::move(WC->Errors[0].first), InstrFilename);
 
   adjustInstrProfile(WC, Reader, SupplMinSizeThreshold, ZeroCounterThreshold,
                      InstrProfColdThreshold);
   writeInstrProfile(OutputFilename, OutputFormat, WC->Writer);
 }
 
cl::opt<unsigned> SupplMinSizeThreshold(
                "be adjusted based on sample profile."));
   cl::opt<unsigned> InstrProfColdThreshold(
       "instr-prof-cold-threshold", cl::init(0), cl::Hidden,
       cl::desc("User specified cold threshold for instr profile which will "
                "override the cold threshold got from profile summary. "));
   cl::opt<bool> GenCSNestedProfile(
       "gen-cs-nested-profile", cl::Hidden, cl::init(false),
       cl::desc("Generate nested function profiles for CSSPGO"));
+  cl::opt<std::string> DebugInfoFilename(
+      "debug-info", cl::init(""), cl::Hidden,
+      cl::desc("Use the provided debug info to correlate the raw profile."));
 
   cl::ParseCommandLineOptions(argc, argv, "LLVM profile data merger\n");
 
   WeightedFileVector WeightedInputs;
   for (StringRef Filename : InputFilenames)
     addWeightedInput(WeightedInputs, {std::string(Filename), 1});
   for (StringRef WeightedFilename : WeightedInputFilenames)
     addWeightedInput(WeightedInputs, parseWeightedFile(WeightedFilename));
if (!SupplInstrWithSample.empty()) {
 
     supplementInstrProfile(WeightedInputs, SupplInstrWithSample, OutputFilename,
                            OutputFormat, OutputSparse, SupplMinSizeThreshold,
                            ZeroCounterThreshold, InstrProfColdThreshold);
     return 0;
   }
 
   if (ProfileKind == instr)
-    mergeInstrProfile(WeightedInputs, Remapper.get(), OutputFilename,
-                      OutputFormat, OutputSparse, NumThreads, FailureMode);
+    mergeInstrProfile(WeightedInputs, DebugInfoFilename, Remapper.get(),
+                      OutputFilename, OutputFormat, OutputSparse, NumThreads,
+                      FailureMode);
   else
     mergeSampleProfile(WeightedInputs, Remapper.get(), OutputFilename,
                        OutputFormat, ProfileSymbolListFile, CompressAllSections,
                        UseMD5, GenPartialProfile, GenCSNestedProfile,
                        SampleMergeColdContext, SampleTrimColdContext,
                        SampleColdContextFrameDepth, FailureMode);
   return 0;
 }
static void overlapInstrProfile(const std::string &BaseFilename,
   if (Overlap.Base.CountSum < 1.0f) {
     OS << "Sum of edge counts for profile " << BaseFilename << " is 0.\n";
     exit(0);
   }
   if (Overlap.Test.CountSum < 1.0f) {
     OS << "Sum of edge counts for profile " << TestFilename << " is 0.\n";
     exit(0);
   }
-  loadInput(WeightedInput, nullptr, &Context);
+  loadInput(WeightedInput, nullptr, nullptr, &Context);
   overlapInput(BaseFilename, TestFilename, &Context, Overlap, FuncFilter, OS,
                IsCS);
   Overlap.dump(OS);
 }
 
 namespace {
 struct SampleOverlapStats {
   SampleContext BaseName;