[PGO] Add --text-format option for llvm-profdata show|merge commands

[davidxl]

Summary

The new option allows user to

1. dump raw/indexed format into text profile format
2. merge the profile and output into text profile format.

Note that Value Profiling data text format is not yet designed. That functionality will be added later.

[davidxl]

1. Make the merge to text option consistent with sample-profile.
2. updated documentation.

[vsk]

Looks good with minor nits

docs/CommandGuide/llvm-profdata.rst

132

Please move your description under the .. option:: -text-format heading. The rest of the descriptions here start with an imperative, so maybe we can use "Enables ..."?

lib/ProfileData/InstrProfWriter.cpp

177

for (uint64_t Count : Func.Counts)?

tools/llvm-profdata/llvm-profdata.cpp

35

Could you handle this in a follow-up commit?

[dnovillo]

lib/ProfileData/InstrProfWriter.cpp

188

I would move this functionality into lib/ProfileData itself. See how the sample writers are organized. From llvm-profdata, you can then just select the kind of writer to use.

tools/llvm-profdata/llvm-profdata.cpp

280

Some of the changes in here seem unrelated to the text format changes. Could you send them separately?

[davidxl]

lib/ProfileData/InstrProfWriter.cpp

188

The change is in lib/ProfileData. There is no need to create a new InstrProfWriter subtype text -- it is much simpler to just add a new writer interface for text.

tools/llvm-profdata/llvm-profdata.cpp

35

clang-format did this for me. will revert this part.

[davidxl]

Addressed feedbacks from review comments. Also make the show|merge text dump option the same (for consistency).

docs/CommandGuide/llvm-profdata.rst

 
 .. option:: -output=output, -o=output
 
  Specify the output file name.  *Output* cannot be ``-`` as the resulting
  indexed profile data can't be written to standard output.
 
 .. option:: -instr (default)
 
- Specify that the input profile is an instrumentation-based profile.
+ Specify that the input profile is an instrumentation-based profile. When
+ using instrumentation-based profiles, the format of the generated file
+ can be generated in one of the two ways:
+
+ .. option:: -binary (default)
+
+ Emit the profile using a binary encoding in indexed profile format.
+
+ .. option:: -text
+
+ Emit the profile in text format.
 
 .. option:: -sample
 
  Specify that the input profile is a sample-based profile. When using
  sample-based profiles, the format of the generated file can be generated
  in one of three ways:
 
  .. option:: -binary (default)

 
 .. option:: -output=output, -o=output
 
  Specify the output file name.  If *output* is ``-`` or it isn't specified,
  then the output is sent to standard output.
 
 .. option:: -instr (default)
 
- Specify that the input profile is an instrumentation-based profile.
+ Specify that the input profile is an instrumentation-based profile. When
+ instrumentation-based profile is specified, the profile counters can be
+ dumped in more human readable form with annotations or dumped in text
+ profile format. To enable dumping in text-profile format, use
+
+.. option:: -text-format
 
 .. option:: -sample
 
  Specify that the input profile is a sample-based profile.
 
 EXIT STATUS
 -----------
 
 :program:`llvm-profdata` returns 1 if the command is omitted or is invalid,
 if it cannot read input files, or if there is a mismatch between their data.

include/llvm/ProfileData/InstrProfWriter.h

public:
   /// Update string entries in profile data with references to StringTable.
   void updateStringTableReferences(InstrProfRecord &I);
   /// Add function counts for the given function. If there are already counts
   /// for this function and the hash and number of counts match, each counter is
   /// summed.
   std::error_code addRecord(InstrProfRecord &&I);
   /// Write the profile to \c OS
   void write(raw_fd_ostream &OS);
+  /// Write the profile in text format to \c OS
+  void writeText(raw_fd_ostream &OS);
+  /// Write \c Record in text format to \c OS
+  static void writeRecordInText(const InstrProfRecord &Record,
+                                raw_fd_ostream &OS);
   /// Write the profile, returning the raw data. For testing.
   std::unique_ptr<MemoryBuffer> writeBuffer();
 
   // Internal interface for testing purpose only.
   void setValueProfDataEndianness(support::endianness Endianness);
 
 private:
   std::pair<uint64_t, uint64_t> writeImpl(raw_ostream &OS);
 };
 
 } // end namespace llvm
 
 #endif

lib/ProfileData/InstrProfWriter.cpp

void InstrProfWriter::write(raw_fd_ostream &OS) {
 
   // Go back and fill in the hash table start.
   using namespace support;
   OS.seek(TableStart.first);
   // Now patch the HashOffset field previously reserved.
   endian::Writer<little>(OS).write<uint64_t>(TableStart.second);
 }
 
+void InstrProfWriter::writeRecordInText(const InstrProfRecord &Func,
+                                        raw_fd_ostream &OS) {
+  OS << Func.Name << "\n" << Func.Hash << "\n" << Func.Counts.size() << "\n";
+  for (size_t I = 0, E = Func.Counts.size(); I < E; ++I)
+    OS << Func.Counts[I] << "\n";
+
+  OS << "\n";
+}
+
+void InstrProfWriter::writeText(raw_fd_ostream &OS) {
+  for (const auto &I : FunctionData)
+    for (const auto &Func : I.getValue())
+      writeRecordInText(Func.second, OS);
+}
+
 std::unique_ptr<MemoryBuffer> InstrProfWriter::writeBuffer() {
   std::string Data;
   llvm::raw_string_ostream OS(Data);
   // Write the hash table.
   auto TableStart = writeImpl(OS);
   OS.flush();
 
   // Go back and fill in the hash table start.
   using namespace support;
   uint64_t Bytes = endian::byte_swap<uint64_t, little>(TableStart.second);
   Data.replace(TableStart.first, sizeof(uint64_t), (const char *)&Bytes,
                sizeof(uint64_t));
 
   // Return this in an aligned memory buffer.
   return MemoryBuffer::getMemBufferCopy(Data);
 }

test/tools/llvm-profdata/Inputs/basic.proftext

+foo
+10
+2
+499500
+179900
+
+main
+16650
+4
+1
+1000
+1000000
+499500
+
+foo2
+10
+2
+500500
+180100
+

test/tools/llvm-profdata/text-dump.test

+Basic tests for testing text dump functions.
+
+RUN: llvm-profdata show --all-functions -counts --text-format %p/Inputs/basic.proftext > %t-basic.proftext1
+RUN: llvm-profdata merge -o %t-basic.proftext2 --text  %p/Inputs/basic.proftext
+
+RUN: llvm-profdata merge -binary -o %t-basic.profdata1 %t-basic.proftext1
+RUN: llvm-profdata merge -o %t-basic.profdata2 %t-basic.proftext2
+
+RUN: llvm-profdata show --all-functions -counts  %t-basic.profdata1 > %t-basic.dump3
+RUN: llvm-profdata show --all-functions -counts  %t-basic.profdata2 > %t-basic.dump4
+
+RUN: llvm-profdata merge -text -o %t-basic.proftext5 %t-basic.profdata1
+RUN: llvm-profdata merge -text -o %t-basic.proftext6 %t-basic.profdata2
+
+RUN: diff %t-basic.dump3 %t-basic.dump4
+RUN: diff %t-basic.proftext5 %t-basic.proftext6
+
+RUN: not llvm-profdata merge -gcc -o %t-basic-profdata3 %t-basic.proftext2 2>&1 | FileCheck %s --check-prefix=UNKNOWN
+UNKNOWN: Unknown
+
+

tools/llvm-profdata/llvm-profdata.cpp

 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PrettyStackTrace.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/raw_ostream.h"
 
 using namespace llvm;
 
-static void exitWithError(const Twine &Message,
-                          StringRef Whence = "",
+enum ProfileFormat { PF_None = 0, PF_Text, PF_Binary, PF_GCC };
+
+static void exitWithError(const Twine &Message, StringRef Whence = "",
                           StringRef Hint = "") {
   errs() << "error: ";
   if (!Whence.empty())
     errs() << Whence << ": ";
   errs() << Message << "\n";
   if (!Hint.empty())
     errs() << Hint << "\n";
   ::exit(1);
if (ShowHint) {
     }
 
     if (!Hint.empty())
       errs() << Hint << "\n";
   }
 }
 
 static void mergeInstrProfile(const cl::list<std::string> &Inputs,
-                              StringRef OutputFilename) {
+                              StringRef OutputFilename,
+                              ProfileFormat OutputFormat) {
   if (OutputFilename.compare("-") == 0)
     exitWithError("Cannot write indexed profdata format to stdout.");
 
+  if (OutputFormat != PF_Binary && OutputFormat != PF_Text)
+    exitWithError("Unknown format is specified.");
+
   std::error_code EC;
   raw_fd_ostream Output(OutputFilename.data(), EC, sys::fs::F_None);
   if (EC)
     exitWithErrorCode(EC, OutputFilename);
 
   InstrProfWriter Writer;
   SmallSet<std::error_code, 4> WriterErrorCodes;
   for (const auto &Filename : Inputs) {
     auto ReaderOrErr = InstrProfReader::create(Filename);
     if (std::error_code ec = ReaderOrErr.getError())
       exitWithErrorCode(ec, Filename);
 
     auto Reader = std::move(ReaderOrErr.get());
     for (auto &I : *Reader) {
       if (std::error_code EC = Writer.addRecord(std::move(I))) {
         // Only show hint the first time an error occurs.
         bool firstTime = WriterErrorCodes.insert(EC).second;
         handleMergeWriterError(EC, Filename, I.Name, firstTime);
       }
     }
     if (Reader->hasError())
       exitWithErrorCode(Reader->getError(), Filename);
   }
+  if (OutputFormat == PF_Text)
+    Writer.writeText(Output);
+  else
     Writer.write(Output);
 }
 
+static sampleprof::SampleProfileFormat FormatMap[] = {
+    sampleprof::SPF_None, sampleprof::SPF_Text, sampleprof::SPF_Binary,
+    sampleprof::SPF_GCC};
+
 static void mergeSampleProfile(const cl::list<std::string> &Inputs,
                                StringRef OutputFilename,
-                               sampleprof::SampleProfileFormat OutputFormat) {
+                               ProfileFormat OutputFormat) {
   using namespace sampleprof;
-  auto WriterOrErr = SampleProfileWriter::create(OutputFilename, OutputFormat);
+  auto WriterOrErr =
+      SampleProfileWriter::create(OutputFilename, FormatMap[OutputFormat]);
   if (std::error_code EC = WriterOrErr.getError())
     exitWithErrorCode(EC, OutputFilename);
 
   auto Writer = std::move(WriterOrErr.get());
   StringMap<FunctionSamples> ProfileMap;
   SmallVector<std::unique_ptr<sampleprof::SampleProfileReader>, 5> Readers;
   for (const auto &Filename : Inputs) {
     auto ReaderOrErr =
cl::opt<std::string> OutputFilename("output", cl::value_desc("output"),
                                       cl::desc("Output file"));
   cl::alias OutputFilenameA("o", cl::desc("Alias for --output"),
                             cl::aliasopt(OutputFilename));
   cl::opt<ProfileKinds> ProfileKind(
       cl::desc("Profile kind:"), cl::init(instr),
       cl::values(clEnumVal(instr, "Instrumentation profile (default)"),
                  clEnumVal(sample, "Sample profile"), clEnumValEnd));
 
-  cl::opt<sampleprof::SampleProfileFormat> OutputFormat(
-      cl::desc("Format of output profile (only meaningful with --sample)"),
-      cl::init(sampleprof::SPF_Binary),
-      cl::values(clEnumValN(sampleprof::SPF_Binary, "binary",
-                            "Binary encoding (default)"),
-                 clEnumValN(sampleprof::SPF_Text, "text", "Text encoding"),
-                 clEnumValN(sampleprof::SPF_GCC, "gcc", "GCC encoding"),
+  cl::opt<ProfileFormat> OutputFormat(
+      cl::desc("Format of output profile"), cl::init(PF_Binary),
+      cl::values(clEnumValN(PF_Binary, "binary", "Binary encoding (default)"),
+                 clEnumValN(PF_Text, "text", "Text encoding"),
+                 clEnumValN(PF_GCC, "gcc",
+                            "GCC encoding (only meaningful for -sample)"),
                  clEnumValEnd));
 
   cl::ParseCommandLineOptions(argc, argv, "LLVM profile data merger\n");
 
   if (ProfileKind == instr)
-    mergeInstrProfile(Inputs, OutputFilename);
+    mergeInstrProfile(Inputs, OutputFilename, OutputFormat);
   else
     mergeSampleProfile(Inputs, OutputFilename, OutputFormat);
 
   return 0;
 }
 
 static int showInstrProfile(std::string Filename, bool ShowCounts,
                             bool ShowIndirectCallTargets, bool ShowAllFunctions,
-                            std::string ShowFunction, raw_fd_ostream &OS) {
+                            std::string ShowFunction, bool TextFormat,
+                            raw_fd_ostream &OS) {
   auto ReaderOrErr = InstrProfReader::create(Filename);
   if (std::error_code EC = ReaderOrErr.getError())
     exitWithErrorCode(EC, Filename);
 
   auto Reader = std::move(ReaderOrErr.get());
   uint64_t MaxFunctionCount = 0, MaxBlockCount = 0;
   size_t ShownFunctions = 0, TotalFunctions = 0;
   for (const auto &Func : *Reader) {
     bool Show =
         ShowAllFunctions || (!ShowFunction.empty() &&
                              Func.Name.find(ShowFunction) != Func.Name.npos);
 
+    bool doTextFormatDump = (Show && ShowCounts && TextFormat);
+
+    if (doTextFormatDump) {
+      InstrProfWriter::writeRecordInText(Func, OS);
+      continue;
+    }
+
     ++TotalFunctions;
     assert(Func.Counts.size() > 0 && "function missing entry counter");
     if (Func.Counts[0] > MaxFunctionCount)
       MaxFunctionCount = Func.Counts[0];
 
+    for (size_t I = 1, E = Func.Counts.size(); I < E; ++I) {
+      if (Func.Counts[I] > MaxBlockCount)
+        MaxBlockCount = Func.Counts[I];
+    }
+
     if (Show) {
+
       if (!ShownFunctions)
         OS << "Counters:\n";
+
       ++ShownFunctions;
 
       OS << "  " << Func.Name << ":\n"
          << "    Hash: " << format("0x%016" PRIx64, Func.Hash) << "\n"
          << "    Counters: " << Func.Counts.size() << "\n"
          << "    Function count: " << Func.Counts[0] << "\n";
+
       if (ShowIndirectCallTargets)
         OS << "    Indirect Call Site Count: "
            << Func.getNumValueSites(IPVK_IndirectCallTarget) << "\n";
-    }
 
-    if (Show && ShowCounts)
+      if (ShowCounts) {
         OS << "    Block counts: [";
         for (size_t I = 1, E = Func.Counts.size(); I < E; ++I) {
-      if (Func.Counts[I] > MaxBlockCount)
-        MaxBlockCount = Func.Counts[I];
-      if (Show && ShowCounts)
           OS << (I == 1 ? "" : ", ") << Func.Counts[I];
         }
-    if (Show && ShowCounts)
         OS << "]\n";
+      }
 
-    if (Show && ShowIndirectCallTargets) {
+      if (ShowIndirectCallTargets) {
         uint32_t NS = Func.getNumValueSites(IPVK_IndirectCallTarget);
         OS << "    Indirect Target Results: \n";
         for (size_t I = 0; I < NS; ++I) {
           uint32_t NV = Func.getNumValueDataForSite(IPVK_IndirectCallTarget, I);
           std::unique_ptr<InstrProfValueData[]> VD =
               Func.getValueForSite(IPVK_IndirectCallTarget, I);
           for (uint32_t V = 0; V < NV; V++) {
             OS << "\t[ " << I << ", ";
             OS << (const char *)VD[V].Value << ", " << VD[V].Count << " ]\n";
           }
         }
       }
     }
+  }
+
   if (Reader->hasError())
     exitWithErrorCode(Reader->getError(), Filename);
 
+  if (ShowCounts && TextFormat)
+    return 0;
+
   if (ShowAllFunctions || !ShowFunction.empty())
     OS << "Functions shown: " << ShownFunctions << "\n";
   OS << "Total functions: " << TotalFunctions << "\n";
   OS << "Maximum function count: " << MaxFunctionCount << "\n";
   OS << "Maximum internal block count: " << MaxBlockCount << "\n";
   return 0;
 }
 

 }
 
 static int show_main(int argc, const char *argv[]) {
   cl::opt<std::string> Filename(cl::Positional, cl::Required,
                                 cl::desc("<profdata-file>"));
 
   cl::opt<bool> ShowCounts("counts", cl::init(false),
                            cl::desc("Show counter values for shown functions"));
+  cl::opt<bool> TextFormat(
+      "text-format", cl::init(false),
+      cl::desc("Show instr profile data in text dump format"));
   cl::opt<bool> ShowIndirectCallTargets(
       "ic-targets", cl::init(false),
       cl::desc("Show indirect call site target values for shown functions"));
   cl::opt<bool> ShowAllFunctions("all-functions", cl::init(false),
                                  cl::desc("Details for every function"));
   cl::opt<std::string> ShowFunction("function",
                                     cl::desc("Details for matching functions"));
 
static int show_main(int argc, const char *argv[]) {
   cl::ParseCommandLineOptions(argc, argv, "LLVM profile data summary\n");
 
   if (OutputFilename.empty())
     OutputFilename = "-";
 
   std::error_code EC;
   raw_fd_ostream OS(OutputFilename.data(), EC, sys::fs::F_Text);
   if (EC)
-      exitWithErrorCode(EC, OutputFilename);
 
   if (ShowAllFunctions && !ShowFunction.empty())
     errs() << "warning: -function argument ignored: showing all functions\n";
 
   if (ProfileKind == instr)
     return showInstrProfile(Filename, ShowCounts, ShowIndirectCallTargets,
-                            ShowAllFunctions, ShowFunction, OS);
+                            ShowAllFunctions, ShowFunction, TextFormat, OS);
   else
     return showSampleProfile(Filename, ShowCounts, ShowAllFunctions,
                              ShowFunction, OS);
 }
 
 int main(int argc, const char *argv[]) {
   // Print a stack trace if we signal out.
   sys::PrintStackTraceOnErrorSignal();