[PGO] Add option to dot-dump raw profile counts as computed by profile annotator

[davidxl]

Summary

This is a follow up patch of D28967 which dumps counts computed by BFI.

[davide]

This one is very useful. I guess we may want to add documentation for it (and maybe the other flag you just added?)?

[davidxl]

I won't mind adding documentation of the flags. Is there a known place to do so (for internal options)?

[davide]

In D29045#653812, @davidxl wrote:

I won't mind adding documentation of the flags. Is there a known place to do so (for internal options)?

I personally wouldn't mind them to be added to https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization (maybe in a separate subsection). It's generally nicer having these flags documented somewhere than looking at the code, IMHO

[silvas]

In D29045#653868, @davide wrote:

In D29045#653812, @davidxl wrote:

I won't mind adding documentation of the flags. Is there a known place to do so (for internal options)?

I personally wouldn't mind them to be added to https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization (maybe in a separate subsection). It's generally nicer having these flags documented somewhere than looking at the code, IMHO

This is an internal flag. That page is end-user documentation. Officially, users are not allowed to use -mllvm flags and we should not encourage them to do so. I'm not aware of a good place to document this flag unfortunately.

[xur]

This gonna be very useful in debug PGO related issues -- we don't need to use the instrumentation debug dumps to track the counts (which is not available in debug builds).
Overall, the patch looks good to me except the minor issues (see inline comments).

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1240

ViewGraph is interactive. Is WriteGraph better?
I tried the patch without specified the PGOViewFunction, it just pops too many windows at once.

1331

Check if BI is null. If the Node is not reachable from entry. We don't have a UseBBInfo.

1332

CountValid for the internal use. After count prorogation, all the reachable BBs should have a valid count.

[davidxl]

Address review feedback from xur.

[davidxl]

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1240

make it display only there is one function specified. Also add function name in filenames.

1331

Added check. How can that happen?

1332

That is why unknown is displayed so that errors like this can be spotted.

[xur]

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

1331

We construct the Minimum spanning tree starting the entry node. An reachable BB (not clean up) will be left out without being touched. This is race, but does happen in some constructed .ll files.

lib/Transforms/Instrumentation/PGOInstrumentation.cpp

 #include "llvm/ADT/Statistic.h"
 #include "llvm/ADT/Triple.h"
 #include "llvm/Analysis/BlockFrequencyInfo.h"
 #include "llvm/Analysis/BranchProbabilityInfo.h"
 #include "llvm/Analysis/CFG.h"
 #include "llvm/Analysis/IndirectCallSiteVisitor.h"
 #include "llvm/Analysis/LoopInfo.h"
 #include "llvm/IR/CallSite.h"
-#include "llvm/IR/Dominators.h"
 #include "llvm/IR/DiagnosticInfo.h"
+#include "llvm/IR/Dominators.h"
 #include "llvm/IR/GlobalValue.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/MDBuilder.h"
 #include "llvm/IR/Module.h"
 #include "llvm/Pass.h"
 #include "llvm/ProfileData/InstrProfReader.h"
 #include "llvm/ProfileData/ProfileCommon.h"
 #include "llvm/Support/BranchProbability.h"
+#include "llvm/Support/DOTGraphTraits.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/GraphWriter.h"
 #include "llvm/Support/JamCRC.h"
 #include "llvm/Transforms/Instrumentation.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include <algorithm>
 #include <string>
 #include <unordered_map>
 #include <utility>
 #include <vector>

 
 // Command line option to specify the name of the function for CFG dump
 static cl::opt<std::string>
     PGOViewFunction("pgo-view-function", cl::Hidden,
                     cl::desc("The option to specify "
                              "the name of the function "
                              "whose CFG will be displayed."));
 
+// Command line option to turn on CFG dot dump of raw profile counts
+static cl::opt<bool> PGOViewRawCounts("pgo-view-raw-counts", cl::init(false),
+                                      cl::Hidden);
+
 // Command line option to turn on CFG dot dump after profile annotation.
 extern cl::opt<bool> PGOViewCounts;
 
 namespace {
 
 /// The select instruction visitor plays three roles specified
 /// by the mode. In \c VM_counting mode, it simply counts the number of
 /// select instructions. In \c VM_instrument mode, it inserts code to count

     return FuncInfo.getBBInfo(BB);
   }
 
   // Return the auxiliary BB information if available.
   UseBBInfo *findBBInfo(const BasicBlock *BB) const {
     return FuncInfo.findBBInfo(BB);
   }
 
+  Function &getFunc() const { return F; }
+
 private:
   Function &F;
   Module *M;
   // This member stores the shared information with class PGOGenFunc.
   FuncPGOInstrumentation<PGOUseEdge, UseBBInfo> FuncInfo;
 
   // The maximum count value in the profile. This is only used in PGO use
   // compilation.

       std::unique_ptr<BranchProbabilityInfo> NewBPI =
           llvm::make_unique<BranchProbabilityInfo>(F, LI);
       std::unique_ptr<BlockFrequencyInfo> NewBFI =
           llvm::make_unique<BlockFrequencyInfo>(F, *NewBPI, LI);
 
       NewBFI->view();
     }
 #endif
+    if (PGOViewRawCounts &&
+        (PGOViewFunction.empty() || F.getName().equals(PGOViewFunction))) {
+      if (PGOViewFunction.empty())
+        WriteGraph(&Func, Twine("PGORawCounts_") + Func.getFunc().getName());
+      else
+        ViewGraph(&Func, Twine("PGORawCounts_") + Func.getFunc().getName());
+    }
   }
   M.setProfileSummary(PGOReader->getSummary().getMD(M.getContext()));
   // Set function hotness attribute from the profile.
   // We have to apply these attributes at the end because their presence
   // can affect the BranchProbabilityInfo of any callers, resulting in an
   // inconsistent MST between prof-gen and prof-use.
   for (auto &F : HotFunctions) {
     F->addFnAttr(llvm::Attribute::InlineHint);

     return &this->getAnalysis<BranchProbabilityInfoWrapperPass>(F).getBPI();
   };
   auto LookupBFI = [this](Function &F) {
     return &this->getAnalysis<BlockFrequencyInfoWrapperPass>(F).getBFI();
   };
 
   return annotateAllFunctions(M, ProfileFileName, LookupBPI, LookupBFI);
 }
+
+namespace llvm {
+template <> struct GraphTraits<PGOUseFunc *> {
+  typedef const BasicBlock *NodeRef;
+  typedef succ_const_iterator ChildIteratorType;
+  typedef pointer_iterator<Function::const_iterator> nodes_iterator;
+
+  static NodeRef getEntryNode(const PGOUseFunc *G) {
+    return &G->getFunc().front();
+  }
+  static ChildIteratorType child_begin(const NodeRef N) {
+    return succ_begin(N);
+  }
+  static ChildIteratorType child_end(const NodeRef N) { return succ_end(N); }
+  static nodes_iterator nodes_begin(const PGOUseFunc *G) {
+    return nodes_iterator(G->getFunc().begin());
+  }
+  static nodes_iterator nodes_end(const PGOUseFunc *G) {
+    return nodes_iterator(G->getFunc().end());
+  }
+};
+
+template <> struct DOTGraphTraits<PGOUseFunc *> : DefaultDOTGraphTraits {
+  explicit DOTGraphTraits(bool isSimple = false)
+      : DefaultDOTGraphTraits(isSimple) {}
+
+  static std::string getGraphName(const PGOUseFunc *G) {
+    return G->getFunc().getName();
+  }
+
+  std::string getNodeLabel(const BasicBlock *Node, const PGOUseFunc *Graph) {
+    std::string Result;
+    raw_string_ostream OS(Result);
+    OS << Node->getName().str() << " : ";
+    UseBBInfo *BI = Graph->findBBInfo(Node);
+    if (BI && BI->CountValid)
+      OS << BI->CountValue;
+    else
+      OS << "Unknown";
+    return Result;
+  }
+};
+}