[BOLT] Compact legacy profiles

[kongyi]

Summary

Merging multiple legacy profiles (produced by BOLT instrumentation) can easily reach multiple GiBs. Let merge-fdata compact the profiles to significantly reduce space usage.

[maksfb]

Thanks for the improvement. Could you please share before and after numbers you are seeing?

bolt/tools/merge-fdata/merge-fdata.cpp

271

277

[kongyi]

Could you please share before and after numbers you are seeing?

Before: merged profile size grows linearly; for a profile from 30 invocations of Clang, it takes more than 3GiB in disk space.

After: merged profile is constant size; the same profile now only takes 100 MiB.

[maksfb]

LGTM

bolt/tools/merge-fdata/merge-fdata.cpp

if (Buffer.startswith("---\n"))
     return true;
   return false;
 }
 
 void mergeLegacyProfiles(const cl::list<std::string> &Filenames) {
   errs() << "Using legacy profile format.\n";
   bool BoltedCollection = false;
   bool First = true;
+  StringMap<uint64_t> Entries;
   for (const std::string &Filename : Filenames) {
     if (isYAML(Filename))
       report_error(Filename, "cannot mix YAML and legacy formats");
     ErrorOr<std::unique_ptr<MemoryBuffer>> MB =
         MemoryBuffer::getFileOrSTDIN(Filename);
     if (std::error_code EC = MB.getError())
       report_error(Filename, EC);
     errs() << "Merging data from " << Filename << "...\n";
if (Buf.startswith("boltedcollection\n")) {
         Buf = Buf.drop_front(17);
     } else {
       if (BoltedCollection)
         report_error(
             Filename,
             "cannot mix profile collected in BOLT and non-BOLT deployments");
     }
 
-    outs() << Buf;
+    SmallVector<StringRef, 10> Lines;
+    SplitString(Buf, Lines, "\n");
+    for (StringRef Line : Lines) {
+      size_t Pos = Line.rfind(" ");
+      if (Pos == StringRef::npos)
+        report_error(Filename, "Malformed / corrupted profile");
+      StringRef Signature = Line.substr(0, Pos);
+      uint64_t Count;
+      if (Line.substr(Pos + 1, Line.size() - Pos).getAsInteger(10, Count))
+        report_error(Filename, "Malformed / corrupted profile counter");
+      Count += Entries.lookup(Signature);
+      Entries.insert_or_assign(Signature, Count);
+    }
     First = false;
   }
+
+  for (const auto &Entry : Entries)
+    outs() << Entry.getKey() << " " << Entry.getValue() << "\n";
+
   errs() << "Profile from " << Filenames.size() << " files merged.\n";
 }
 
 } // anonymous namespace
 
 int main(int argc, char **argv) {
   // Print a stack trace if we signal out.
   sys::PrintStackTraceOnErrorSignal(argv[0]);