Add flag to llvm-profdata to allow symbols in profile data to be remapped, andadd a tool to generate symbol remapping files.

[rsmith]

Summary

The new tool llvm-cxxmap builds a symbol mapping table from a file containing
a description of partial equivalences to apply to mangled names and files
containing old and new symbol tables.

[davidxl]

Some test case for llvm-profdata new option is also needed (under test/tools/llvm-profdata)

tools/llvm-profdata/llvm-profdata.cpp

543

The new option needs to be documented in docs/CommandGuide/llvm-profdata.rst

[rsmith]

• Update documentation for llvm-profdata and add documentation for llvm-cxxmap.
• Add tests for new llvm-profdata functionality.
• Add tests for llvm-cxxmap.

[davidxl]

lgtm

Wei may have more comments on the samplePGO remapping code.

[wmi]

LGTM.

[riccibruno]

This commit made the reverse-iteration bot (http://lab.llvm.org:8011/builders/reverse-iteration) red (red since 6 months now) :

FAIL: LLVM :: tools/llvm-profdata/instr-remap.test (42755 of 44261)
******************** TEST 'LLVM :: tools/llvm-profdata/instr-remap.test' FAILED ********************
Script:
--
: 'RUN: at line 1';   /local/buildbot/slaves/hexagon-build-02/reverse-iteration/llvm.obj/bin/llvm-profdata merge -text /local/buildbot/slaves/hexagon-build-02/reverse-iteration/llvm.src/test/tools/llvm-profdata/Inputs/instr-remap.proftext -r /local/buildbot/slaves/hexagon-build-02/reverse-iteration/llvm.src/test/tools/llvm-profdata/Inputs/instr-remap.remap -o /local/buildbot/slaves/hexagon-build-02/reverse-iteration/llvm.obj/test/tools/llvm-profdata/Output/instr-remap.test.tmp.output
: 'RUN: at line 2';   diff /local/buildbot/slaves/hexagon-build-02/reverse-iteration/llvm.src/test/tools/llvm-profdata/Inputs/instr-remap.expected /local/buildbot/slaves/hexagon-build-02/reverse-iteration/llvm.obj/test/tools/llvm-profdata/Output/instr-remap.test.tmp.output
--
Exit Code: 1

Command Output (stdout):
--
5c5
< 1234
---
> 5678
9,10c9,10
< 31
< 42
---
> 500
> 600
14c14
< 5678
---
> 1234
18,19c18,19
< 500
< 600
---
> 31
> 42

--

********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 

1 warning(s) in tests.
Testing Time: 184.64s
********************
Failing Tests (1):
    LLVM :: tools/llvm-profdata/instr-remap.test

  Expected Passes    : 43374
  Expected Failures  : 195
  Unsupported Tests  : 691
  Unexpected Failures: 1

[riccibruno]

@davidxl @wmi Ping. The reverse-iteration bot is still red because of this patch.
(http://lab.llvm.org:8011/builders/reverse-iteration)

[Hahnfeld]

In D51470#1351254, @riccibruno wrote:

This commit made the reverse-iteration bot (http://lab.llvm.org:8011/builders/reverse-iteration) red (red since 6 months now) :

It's a pity that nobody cared. I've submitted D58385 to fix this, PTAL.

[riccibruno]

In D51470#1402322, @Hahnfeld wrote:

In D51470#1351254, @riccibruno wrote:

This commit made the reverse-iteration bot (http://lab.llvm.org:8011/builders/reverse-iteration) red (red since 6 months now) :

It's a pity that nobody cared. I've submitted D58385 to fix this, PTAL.

Thanks a lot. I am afraid I cannot comment on the changes since I am not at all familiar with it, but I am glad that this issue is being looked at !

tools/llvm-cxxmap/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Core
+  Support
+  )
+
+add_llvm_tool(llvm-cxxmap
+  llvm-cxxmap.cpp
+  )

tools/llvm-cxxmap/LLVMBuild.txt

+;===- ./tools/llvm-cxxmap/LLVMBuild.txt ------------------------*- Conf -*--===;
+;
+;                     The LLVM Compiler Infrastructure
+;
+; This file is distributed under the University of Illinois Open Source
+; License. See LICENSE.TXT for details.
+;
+;===------------------------------------------------------------------------===;
+;
+; This is an LLVMBuild description file for the components in this subdirectory.
+;
+; For more information on the LLVMBuild system, please see:
+;
+;   http://llvm.org/docs/LLVMBuild.html
+;
+;===------------------------------------------------------------------------===;
+
+[component_0]
+type = Tool
+name = llvm-cxxmap
+parent = Tools
+required_libraries = Support

tools/llvm-cxxmap/llvm-cxxmap.cpp

+//===- llvm-cxxmap.cpp ----------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// llvm-cxxmap computes a correspondence between old symbol names and new
+// symbol names based on a symbol equivalence file.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/LineIterator.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/SymbolRemappingReader.h"
+#include "llvm/Support/WithColor.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace llvm;
+
+cl::opt<std::string> OldSymbolFile(cl::Positional, cl::Required,
+                                   cl::desc("<symbol-file>"));
+cl::opt<std::string> NewSymbolFile(cl::Positional, cl::Required,
+                                   cl::desc("<symbol-file>"));
+cl::opt<std::string> RemappingFile("remapping-file", cl::Required,
+                                   cl::desc("Remapping file"));
+cl::alias RemappingFileA("r", cl::aliasopt(RemappingFile));
+cl::opt<std::string> OutputFilename("output", cl::value_desc("output"),
+                                    cl::init("-"), cl::desc("Output file"));
+cl::alias OutputFilenameA("o", cl::aliasopt(OutputFilename));
+
+cl::opt<bool> WarnAmbiguous(
+    "Wambiguous",
+    cl::desc("Warn on equivalent symbols in the output symbol list"));
+cl::opt<bool> WarnIncomplete(
+    "Wincomplete",
+    cl::desc("Warn on input symbols missing from output symbol list"));
+cl::opt<bool> WarnInvalidMangling(
+    "Winvalid-mangling",
+    cl::desc("Warn on mangled names that cannot be demangled"));
+
+static void warn(Twine Message, Twine Whence = "",
+                 std::string Hint = "") {
+  WithColor::warning();
+  std::string WhenceStr = Whence.str();
+  if (!WhenceStr.empty())
+    errs() << WhenceStr << ": ";
+  errs() << Message << "\n";
+  if (!Hint.empty())
+    WithColor::note() << Hint << "\n";
+}
+
+static void exitWithError(Twine Message, Twine Whence = "",
+                          std::string Hint = "") {
+  WithColor::error();
+  std::string WhenceStr = Whence.str();
+  if (!WhenceStr.empty())
+    errs() << WhenceStr << ": ";
+  errs() << Message << "\n";
+  if (!Hint.empty())
+    WithColor::note() << Hint << "\n";
+  ::exit(1);
+}
+
+static void exitWithError(Error E, StringRef Whence = "") {
+  exitWithError(toString(std::move(E)), Whence);
+}
+
+static void exitWithErrorCode(std::error_code EC, StringRef Whence = "") {
+  exitWithError(EC.message(), Whence);
+}
+
+static void remapSymbols(MemoryBuffer &OldSymbolFile,
+                         MemoryBuffer &NewSymbolFile,
+                         MemoryBuffer &RemappingFile,
+                         raw_ostream &Out) {
+  // Load the remapping file and prepare to canonicalize symbols.
+  SymbolRemappingReader Reader;
+  if (Error E = Reader.read(RemappingFile))
+    exitWithError(std::move(E));
+
+  // Canonicalize the new symbols.
+  DenseMap<SymbolRemappingReader::Key, StringRef> MappedNames;
+  DenseSet<StringRef> UnparseableSymbols;
+  for (line_iterator LineIt(NewSymbolFile, /*SkipBlanks=*/true, '#');
+       !LineIt.is_at_eof(); ++LineIt) {
+    StringRef Symbol = *LineIt;
+
+    auto K = Reader.insert(Symbol);
+    if (!K) {
+      if (WarnInvalidMangling && Symbol.startswith("_Z")) {
+        warn("could not demangle symbol " + Symbol,
+             NewSymbolFile.getBufferIdentifier() + ":" +
+                 Twine(LineIt.line_number()));
+      }
+      UnparseableSymbols.insert(Symbol);
+      continue;
+    }
+
+    auto ItAndIsNew = MappedNames.insert({K, Symbol});
+    if (WarnAmbiguous && !ItAndIsNew.second &&
+        ItAndIsNew.first->second != Symbol) {
+      warn("symbol " + Symbol + " is equivalent to earlier symbol " +
+               ItAndIsNew.first->second,
+           NewSymbolFile.getBufferIdentifier() + ":" +
+               Twine(LineIt.line_number()),
+           "later symbol will not be the target of any remappings");
+    }
+  }
+
+  // Figure out which new symbol each old symbol is equivalent to.
+  for (line_iterator LineIt(OldSymbolFile, /*SkipBlanks=*/true, '#');
+       !LineIt.is_at_eof(); ++LineIt) {
+    StringRef Symbol = *LineIt;
+
+    auto K = Reader.lookup(Symbol);
+    StringRef NewSymbol = MappedNames.lookup(K);
+
+    if (NewSymbol.empty()) {
+      if (WarnIncomplete && !UnparseableSymbols.count(Symbol)) {
+        warn("no new symbol matches old symbol " + Symbol,
+             OldSymbolFile.getBufferIdentifier() + ":" +
+                 Twine(LineIt.line_number()));
+      }
+      continue;
+    }
+
+    Out << Symbol << " " << NewSymbol << "\n";
+  }
+}
+
+int main(int argc, const char *argv[]) {
+  InitLLVM X(argc, argv);
+
+  cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
+
+  auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
+  if (!OldSymbolBufOrError)
+    exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
+
+  auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
+  if (!NewSymbolBufOrError)
+    exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
+
+  auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
+  if (!RemappingBufOrError)
+    exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
+
+  std::error_code EC;
+  raw_fd_ostream OS(OutputFilename.data(), EC, sys::fs::F_Text);
+  if (EC)
+    exitWithErrorCode(EC, OutputFilename);
+
+  remapSymbols(*OldSymbolBufOrError.get(), *NewSymbolBufOrError.get(),
+               *RemappingBufOrError.get(), OS);
+}

tools/llvm-profdata/llvm-profdata.cpp

if (IPE != instrprof_error::success) {
       }
     }
 
     if (!Hint.empty())
       errs() << Hint << "\n";
   }
 }
 
+namespace {
+/// A remapper from original symbol names to new symbol names based on a file
+/// containing a list of mappings from old name to new name.
+class SymbolRemapper {
+  std::unique_ptr<MemoryBuffer> File;
+  DenseMap<StringRef, StringRef> RemappingTable;
+
+public:
+  /// Build a SymbolRemapper from a file containing a list of old/new symbols.
+  static std::unique_ptr<SymbolRemapper> create(StringRef InputFile) {
+    auto BufOrError = MemoryBuffer::getFileOrSTDIN(InputFile);
+    if (!BufOrError)
+      exitWithErrorCode(BufOrError.getError(), InputFile);
+
+    auto Remapper = llvm::make_unique<SymbolRemapper>();
+    Remapper->File = std::move(BufOrError.get());
+
+    for (line_iterator LineIt(*Remapper->File, /*SkipBlanks=*/true, '#');
+         !LineIt.is_at_eof(); ++LineIt) {
+      std::pair<StringRef, StringRef> Parts = LineIt->split(' ');
+      if (Parts.first.empty() || Parts.second.empty() ||
+          Parts.second.count(' ')) {
+        exitWithError("unexpected line in remapping file",
+                      (InputFile + ":" + Twine(LineIt.line_number())).str(),
+                      "expected 'old_symbol new_symbol'");
+      }
+      Remapper->RemappingTable.insert(Parts);
+    }
+    return Remapper;
+  }
+
+  /// Attempt to map the given old symbol into a new symbol.
+  ///
+  /// \return The new symbol, or an empty StringRef if no such symbol was
+  /// found.
+  StringRef operator()(StringRef Name) { return RemappingTable.lookup(Name); }
+};
+}
+
 struct WeightedFile {
   std::string Filename;
   uint64_t Weight;
 };
 typedef SmallVector<WeightedFile, 5> WeightedFileVector;
 
 /// Keep track of merged data and reported errors.
 struct WriterContext {
static bool isFatalError(instrprof_error IPE) {
   case instrprof_error::count_mismatch:
   case instrprof_error::counter_overflow:
   case instrprof_error::value_site_count_mismatch:
     return false;
   }
 }
 
 /// Load an input into a writer context.
-static void loadInput(const WeightedFile &Input, WriterContext *WC) {
+static void loadInput(const WeightedFile &Input, SymbolRemapper *Remapper,
+                      WriterContext *WC) {
   std::unique_lock<std::mutex> CtxGuard{WC->Lock};
 
   // If there's a pending hard error, don't do more work.
   if (WC->Err)
     return;
 
   // Copy the filename, because llvm::ThreadPool copied the input "const
   // WeightedFile &" by value, making a reference to the filename within it
static void loadInput(const WeightedFile &Input, SymbolRemapper *Remapper,
   if (WC->Writer.setIsIRLevelProfile(IsIRProfile)) {
     WC->Err = make_error<StringError>(
         "Merge IR generated profile with Clang generated profile.",
         std::error_code());
     return;
   }
 
   for (auto &I : *Reader) {
+    if (Remapper)
+      I.Name = (*Remapper)(I.Name);
     const StringRef FuncName = I.Name;
     bool Reported = false;
     WC->Writer.addRecord(std::move(I), Input.Weight, [&](Error E) {
       if (Reported) {
         consumeError(std::move(E));
         return;
       }
       Reported = true;
if (Reported) {
       return;
     }
     Reported = true;
     Dst->Err = std::move(E);
   });
 }
 
 static void mergeInstrProfile(const WeightedFileVector &Inputs,
+                              SymbolRemapper *Remapper,
                               StringRef OutputFilename,
                               ProfileFormat OutputFormat, bool OutputSparse,
                               unsigned NumThreads) {
   if (OutputFilename.compare("-") == 0)
     exitWithError("Cannot write indexed profdata format to stdout.");
 
   if (OutputFormat != PF_Binary && OutputFormat != PF_Compact_Binary &&
       OutputFormat != PF_Text)
static void mergeInstrProfile(const WeightedFileVector &Inputs,
   // Initialize the writer contexts.
   SmallVector<std::unique_ptr<WriterContext>, 4> Contexts;
   for (unsigned I = 0; I < NumThreads; ++I)
     Contexts.emplace_back(llvm::make_unique<WriterContext>(
         OutputSparse, ErrorLock, WriterErrorCodes));
 
   if (NumThreads == 1) {
     for (const auto &Input : Inputs)
-      loadInput(Input, Contexts[0].get());
+      loadInput(Input, Remapper, Contexts[0].get());
   } else {
     ThreadPool Pool(NumThreads);
 
     // Load the inputs in parallel (N/NumThreads serial steps).
     unsigned Ctx = 0;
     for (const auto &Input : Inputs) {
-      Pool.async(loadInput, Input, Contexts[Ctx].get());
+      Pool.async(loadInput, Input, Remapper, Contexts[Ctx].get());
       Ctx = (Ctx + 1) % NumThreads;
     }
     Pool.wait();
 
     // Merge the writer contexts together (~ lg(NumThreads) serial steps).
     unsigned Mid = Contexts.size() / 2;
     unsigned End = Contexts.size();
     assert(Mid > 0 && "Expected more than one context");
static void mergeInstrProfile(const WeightedFileVector &Inputs,
   if (OutputFormat == PF_Text) {
     if (Error E = Writer.writeText(Output))
       exitWithError(std::move(E));
   } else {
     Writer.write(Output);
   }
 }
 
+/// Make a copy of the given function samples with all symbol names remapped
+/// by the provided symbol remapper.
+static sampleprof::FunctionSamples
+remapSamples(const sampleprof::FunctionSamples &Samples,
+             SymbolRemapper &Remapper, sampleprof_error &Error) {
+  sampleprof::FunctionSamples Result;
+  Result.setName(Remapper(Samples.getName()));
+  Result.addTotalSamples(Samples.getTotalSamples());
+  Result.addHeadSamples(Samples.getHeadSamples());
+  for (const auto &BodySample : Samples.getBodySamples()) {
+    Result.addBodySamples(BodySample.first.LineOffset,
+                          BodySample.first.Discriminator,
+                          BodySample.second.getSamples());
+    for (const auto &Target : BodySample.second.getCallTargets()) {
+      Result.addCalledTargetSamples(BodySample.first.LineOffset,
+                                    BodySample.first.Discriminator,
+                                    Remapper(Target.first()), Target.second);
+    }
+  }
+  for (const auto &CallsiteSamples : Samples.getCallsiteSamples()) {
+    sampleprof::FunctionSamplesMap &Target =
+        Result.functionSamplesAt(CallsiteSamples.first);
+    for (const auto &Callsite : CallsiteSamples.second) {
+      sampleprof::FunctionSamples Remapped =
+          remapSamples(Callsite.second, Remapper, Error);
+      MergeResult(Error, Target[Remapped.getName()].merge(Remapped));
+    }
+  }
+  return Result;
+}
+
 static sampleprof::SampleProfileFormat FormatMap[] = {
     sampleprof::SPF_None, sampleprof::SPF_Text, sampleprof::SPF_Compact_Binary,
     sampleprof::SPF_GCC, sampleprof::SPF_Binary};
 
 static void mergeSampleProfile(const WeightedFileVector &Inputs,
+                               SymbolRemapper *Remapper,
                                StringRef OutputFilename,
                                ProfileFormat OutputFormat) {
   using namespace sampleprof;
   auto WriterOrErr =
       SampleProfileWriter::create(OutputFilename, FormatMap[OutputFormat]);
   if (std::error_code EC = WriterOrErr.getError())
     exitWithErrorCode(EC, OutputFilename);
 
for (const auto &Input : Inputs) {
     const auto Reader = Readers.back().get();
     if (std::error_code EC = Reader->read())
       exitWithErrorCode(EC, Input.Filename);
 
     StringMap<FunctionSamples> &Profiles = Reader->getProfiles();
     for (StringMap<FunctionSamples>::iterator I = Profiles.begin(),
                                               E = Profiles.end();
          I != E; ++I) {
-      StringRef FName = I->first();
-      FunctionSamples &Samples = I->second;
-      sampleprof_error Result = ProfileMap[FName].merge(Samples, Input.Weight);
+      sampleprof_error Result = sampleprof_error::success;
+      FunctionSamples Remapped =
+          Remapper ? remapSamples(I->second, *Remapper, Result)
+                   : FunctionSamples();
+      FunctionSamples &Samples = Remapper ? Remapped : I->second;
+      StringRef FName = Samples.getName();
+      MergeResult(Result, ProfileMap[FName].merge(Samples, Input.Weight));
       if (Result != sampleprof_error::success) {
         std::error_code EC = make_error_code(Result);
         handleMergeWriterError(errorCodeToError(EC), Input.Filename, FName);
       }
     }
   }
   Writer->write(ProfileMap);
 }
cl::opt<std::string> InputFilenamesFile(
       "input-files", cl::init(""),
       cl::desc("Path to file containing newline-separated "
                "[<weight>,]<filename> entries"));
   cl::alias InputFilenamesFileA("f", cl::desc("Alias for --input-files"),
                                 cl::aliasopt(InputFilenamesFile));
   cl::opt<bool> DumpInputFileList(
       "dump-input-file-list", cl::init(false), cl::Hidden,
       cl::desc("Dump the list of input files and their weights, then exit"));
+  cl::opt<std::string> RemappingFile("remapping-file", cl::value_desc("file"),
+                                     cl::desc("Symbol remapping file"));
   cl::opt<std::string> OutputFilename("output", cl::value_desc("output"),
                                       cl::init("-"), cl::Required,
                                       cl::desc("Output file"));
   cl::alias OutputFilenameA("o", cl::desc("Alias for --output"),
                             cl::aliasopt(OutputFilename));
   cl::opt<ProfileKinds> ProfileKind(
       cl::desc("Profile kind:"), cl::init(instr),
       cl::values(clEnumVal(instr, "Instrumentation profile (default)"),
exitWithError("No input files specified. See " +
                   sys::path::filename(argv[0]) + " -help");
 
   if (DumpInputFileList) {
     for (auto &WF : WeightedInputs)
       outs() << WF.Weight << "," << WF.Filename << "\n";
     return 0;
   }
 
+  std::unique_ptr<SymbolRemapper> Remapper;
+  if (!RemappingFile.empty())
+    Remapper = SymbolRemapper::create(RemappingFile);
+
   if (ProfileKind == instr)
-    mergeInstrProfile(WeightedInputs, OutputFilename, OutputFormat,
-                      OutputSparse, NumThreads);
+    mergeInstrProfile(WeightedInputs, Remapper.get(), OutputFilename,
+                      OutputFormat, OutputSparse, NumThreads);
   else
-    mergeSampleProfile(WeightedInputs, OutputFilename, OutputFormat);
+    mergeSampleProfile(WeightedInputs, Remapper.get(), OutputFilename,
+                       OutputFormat);
 
   return 0;
 }
 
 typedef struct ValueSitesStats {
   ValueSitesStats()
       : TotalNumValueSites(0), TotalNumValueSitesWithValueProfile(0),
         TotalNumValues(0) {}