[llvm-cov][WIP] Add Per Directory coverage to llvm-cov

[ashpande]

Summary

Per Directory Coverage: by default, llvm-cov reports are generated on a per file basis. While this works for small applications with few executable files, it can become cumbersome for very large projects, such as LLVM itself. The resulting HTML file can be large, and it can be hard to find any file. Moreover, directory structure is not preserved, so it can be difficult to know the coverage of one particular directory in a large report (unless you build it selectively beforehand).

The gcov code coverage frontend lcov defaults to a per directory structure. An example is here: LCOV LLVM Report. We would like to go one step further and default the report to output a report which preserves directory structure. For example, if a user builds LLVM with code coverage, the resulting report should replicate the llvm-monorepo. The user should be able to click on the directory and see the coverage for all the subdirectories, and so on until they can see the source files and their coverage info.

Implementation Details:
The function buildCoverageTree takes a vector of source files and builds the tree from it.
This is called by createIndexFile when llvm-cov is run.
CreateIndexFile calls itself recursively for every node in the tree that is not a leaf node.

Note: This is a very rough draft. I have to get the HTML generation working. Will clean this up and send more updates over the next couple of days.

[phosek]

For future revisions, please use larger context as described in https://llvm.org/docs/Phabricator.html#requesting-a-review-via-the-web-interface

llvm/tools/llvm-cov/SourceCoverageViewHTML.cpp

71

This looks like an unrelated change.

357

For debugging output, we usually use the LLVM_DEBUG macro and the dbgs() stream, see https://llvm.org/docs/ProgrammersManual.html#the-llvm-debug-macro-and-debug-option

[ashpande]

Updated with Debug log.
In the CoverageTreeNode struct, changed children from an array to an unordered set. This prevents the multiple child problem while building up the tree.
Tried to increase the context with -U999999, not sure if its correct now.

The patch is generating the HTML for folders but not the files. It's throwing an error saying "too many open files".

This might be because we are recursing over the tree and opening a new file each time. Node->Name is relative path, while Node->FCS.Name is the absolute path, FCS.Name is what llvm-cov is using. Chopping strings seems to be hacky, so have to figure out a way to be relative to repository root/ user specified DIR root. This should avoid the problem of too many files being open.

[gulfem]

I have a few suggestions:

1. In the commit message, please add the link to https://llvm.org/reports/coverage/index.html after "An example is here".
2. I think you can introduce a flag into llvm-cov like -per-directory-coverage and enable this functionality when this flag is enabled.

llvm/tools/llvm-cov/SourceCoverageViewHTML.cpp

343

I think one potential problem here is that we can have the same folder name, e.g. src in separate sub-projects like in https://lab.llvm.org/coverage/coverage-reports/index.html:

libcxx/src/memory.cpp
libcxxabi/src/cxa_vector.cpp

Maybe, you can use DirPath in Dirs instead of DirName to make sure that it works in such cases.

[ashpande]

Block scoped the stream so that it is closed before calling the builder for children. This is more efficient than keeping the stream open

llvm/tools/llvm-cov/SourceCoverageViewHTML.h

 //===- SourceCoverageViewHTML.h - A html code coverage view ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file This file defines the interface to the html coverage renderer.
 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_COV_SOURCECOVERAGEVIEWHTML_H
 #define LLVM_COV_SOURCECOVERAGEVIEWHTML_H
 
 #include "SourceCoverageView.h"
+#include "llvm/ADT/ArrayRef.h"
+#include <unordered_set>
 
 namespace llvm {
 
 using namespace coverage;
 
 struct FileCoverageSummary;
 
+struct CoverageTreeNode {
+  StringRef Name;
+  std::unordered_set<CoverageTreeNode *> Children;
+  FileCoverageSummary FCS;
+
+  CoverageTreeNode(StringRef Name, FileCoverageSummary FCS)
+      : Name(Name), FCS(FCS) {}
+};
+
 /// A coverage printer for html output.
 class CoveragePrinterHTML : public CoveragePrinter {
 public:
   Expected<OwnedStream> createViewFile(StringRef Path,
                                        bool InToplevel) override;
 
   void closeViewFile(OwnedStream OS) override;
 
   Error createIndexFile(ArrayRef<std::string> SourceFiles,
                         const coverage::CoverageMapping &Coverage,
                         const CoverageFiltersMatchAll &Filters) override;
 
+  Error createIndexFile(CoverageTreeNode *Root,
+                        const CoverageFiltersMatchAll &Filters);
+
   CoveragePrinterHTML(const CoverageViewOptions &Opts)
       : CoveragePrinter(Opts) {}
 
 private:
+  CoverageTreeNode *buildCoverageTree(ArrayRef<std::string> &SourceFiles,
+                                      const coverage::CoverageMapping &Coverage,
+                                      const CoverageFiltersMatchAll &Filters);
   void emitFileSummary(raw_ostream &OS, StringRef SF,
-                       const FileCoverageSummary &FCS,
-                       bool IsTotals = false) const;
-  std::string buildLinkToFile(StringRef SF,
-                              const FileCoverageSummary &FCS) const;
+                       const FileCoverageSummary &FCS, bool IsTotals = false,
+                       bool IsDir = false) const;
+  std::string buildLinkToFile(StringRef SF, const FileCoverageSummary &FCS,
+                              bool InTopLevel = false) const;
 };
 
 /// A code coverage view which supports html-based rendering.
 class SourceCoverageViewHTML : public SourceCoverageView {
   void renderViewHeader(raw_ostream &OS) override;
 
   void renderViewFooter(raw_ostream &OS) override;
 

llvm/tools/llvm-cov/SourceCoverageViewHTML.cpp

 //===- SourceCoverageViewHTML.cpp - A html code coverage view -------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 ///
 /// \file This file implements the html coverage renderer.
 ///
 //===----------------------------------------------------------------------===//
 
-#include "CoverageReport.h"
 #include "SourceCoverageViewHTML.h"
+#include "CoverageReport.h"
+#include "CoverageSummaryInfo.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringMap.h"
+#include "llvm/Support/Debug.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/Path.h"
 
 using namespace llvm;
 
 namespace {
 
 // Return a string with the special characters in \p Str escaped.

 // Create an anchor to \p Link with the label \p Str.
 std::string a(const std::string &Link, const std::string &Str,
               const std::string &TargetName = "") {
   std::string Name = TargetName.empty() ? "" : ("name='" + TargetName + "' ");
   return "<a " + Name + "href='" + Link + "'>" + Str + "</a>";
 }
 
 const char *BeginHeader =
     "<head>"
     "<meta name='viewport' content='width=device-width,initial-scale=1'>"
     "<meta charset='UTF-8'>";
 
 const char *CSSForCoverage =
     R"(.red {
   background-color: #ffd0d0;
 }
 .cyan {
static void emitColumnLabelsForIndex(raw_ostream &OS,
   Columns.emplace_back(tag("td", "Line Coverage", "column-entry-bold"));
   if (Opts.ShowRegionSummary)
     Columns.emplace_back(tag("td", "Region Coverage", "column-entry-bold"));
   if (Opts.ShowBranchSummary)
     Columns.emplace_back(tag("td", "Branch Coverage", "column-entry-bold"));
   OS << tag("tr", join(Columns.begin(), Columns.end(), ""));
 }
 
-std::string
-CoveragePrinterHTML::buildLinkToFile(StringRef SF,
-                                     const FileCoverageSummary &FCS) const {
+void printTree(CoverageTreeNode *N) {
+  LLVM_DEBUG(dbgs() << "\nParent: ");
+  LLVM_DEBUG(dbgs() << N->Name);
+  for (auto *I : N->Children) {
+    printTree(I);
+  }
+}
+
+CoverageTreeNode *CoveragePrinterHTML::buildCoverageTree(
+    ArrayRef<std::string> &SourceFiles,
+    const coverage::CoverageMapping &Coverage,
+    const CoverageFiltersMatchAll &Filters) {
+  // Generate file reports
+  FileCoverageSummary Totals("TOTALS");
+  auto FileReports = CoverageReport::prepareFileReports(
+      Coverage, Totals, SourceFiles, Opts, Filters);
+  StringMap<CoverageTreeNode *> Dirs;
+  CoverageTreeNode *Root = new CoverageTreeNode{"/", Totals};
+  Dirs.insert_or_assign("/", Root);
+
+  // Build the tree
+  for (unsigned I = 0, E = SourceFiles.size(); I < E; I++) {
+    StringRef DirPath = llvm::sys::path::parent_path(SourceFiles[I]);
+    StringRef DirName = llvm::sys::path::filename(DirPath);
+    StringRef FCSPath = llvm::sys::path::parent_path(FileReports[I].Name);
+    llvm::CoverageTreeNode *ParentDir =
+        Dirs.try_emplace(
+                DirName,
+                new CoverageTreeNode{DirName, FileCoverageSummary{FCSPath}})
+            .first->getValue();
+    ParentDir->Children.emplace(
+        new CoverageTreeNode{SourceFiles[I], FileReports[I]});
+    ParentDir->FCS += FileReports[I];
+    DirPath = llvm::sys::path::parent_path(DirPath);
+    DirName = llvm::sys::path::filename(DirPath);
+    FCSPath = llvm::sys::path::parent_path(FCSPath);
+    // Maintain a reference to the current directory
+    llvm::CoverageTreeNode *CurrentDir = ParentDir;
+    while (!DirName.equals("llvm-project")) {
+      ParentDir = Dirs.try_emplace(DirName,
+                                   new CoverageTreeNode{
+                                       DirName, FileCoverageSummary{FCSPath}})
+                      .first->getValue();
+      ParentDir->Children.emplace(CurrentDir);
+      ParentDir->FCS += FileReports[I];
+      CurrentDir = ParentDir;
+      DirPath = llvm::sys::path::parent_path(DirPath);
+      DirName = llvm::sys::path::filename(DirPath);
+      FCSPath = llvm::sys::path::parent_path(FCSPath);
+    }
+    ParentDir = Dirs.find("/")->getValue();
+    ParentDir->Children.emplace(CurrentDir);
+  }
+
+  printTree(Root);
+  return Root;
+}
+
+std::string CoveragePrinterHTML::buildLinkToFile(StringRef SF,
+                                                 const FileCoverageSummary &FCS,
+                                                 bool InTopLevel) const {
   SmallString<128> LinkTextStr(sys::path::relative_path(FCS.Name));
   sys::path::remove_dots(LinkTextStr, /*remove_dot_dot=*/true);
   sys::path::native(LinkTextStr);
   std::string LinkText = escape(LinkTextStr, Opts);
-  std::string LinkTarget =
-      escape(getOutputPath(SF, "html", /*InToplevel=*/false), Opts);
+  std::string LinkTarget = escape(getOutputPath(SF, "html", InTopLevel), Opts);
   return a(LinkTarget, LinkText);
 }
 
 /// Render a file coverage summary (\p FCS) in a table row. If \p IsTotals is
 /// false, link the summary to \p SF.
 void CoveragePrinterHTML::emitFileSummary(raw_ostream &OS, StringRef SF,
                                           const FileCoverageSummary &FCS,
-                                          bool IsTotals) const {
+                                          bool IsTotals, bool IsDir) const {
   SmallVector<std::string, 8> Columns;
 
   // Format a coverage triple and add the result to the list of columns.
   auto AddCoverageTripleToColumn =
       [&Columns, this](unsigned Hit, unsigned Total, float Pctg) {
         std::string S;
         {
           raw_string_ostream RSO{S};
auto AddCoverageTripleToColumn =
           CellClass = "column-entry-red";
         Columns.emplace_back(tag("td", tag("pre", S), CellClass));
       };
 
   // Simplify the display file path, and wrap it in a link if requested.
   std::string Filename;
   if (IsTotals) {
     Filename = std::string(SF);
+  } else if (IsDir) {
+    Filename = buildLinkToFile(SF, FCS, true);
   } else {
     Filename = buildLinkToFile(SF, FCS);
   }
 
   Columns.emplace_back(tag("td", tag("pre", Filename)));
   AddCoverageTripleToColumn(FCS.FunctionCoverage.getExecuted(),
                             FCS.FunctionCoverage.getNumFunctions(),
                             FCS.FunctionCoverage.getPercentCovered());
void CoveragePrinterHTML::emitFileSummary(raw_ostream &OS, StringRef SF,
 
   if (IsTotals)
     OS << tag("tr", join(Columns.begin(), Columns.end(), ""), "light-row-bold");
   else
     OS << tag("tr", join(Columns.begin(), Columns.end(), ""), "light-row");
 }
 
 Error CoveragePrinterHTML::createIndexFile(
-    ArrayRef<std::string> SourceFiles, const CoverageMapping &Coverage,
-    const CoverageFiltersMatchAll &Filters) {
-  // Emit the default stylesheet.
-  auto CSSOrErr = createOutputStream("style", "css", /*InToplevel=*/true);
-  if (Error E = CSSOrErr.takeError())
-    return E;
-
-  OwnedStream CSS = std::move(CSSOrErr.get());
-  CSS->operator<<(CSSForCoverage);
-
+    CoverageTreeNode *Root, const CoverageFiltersMatchAll &Filters) {
+  {
     // Emit a file index along with some coverage statistics.
-  auto OSOrErr = createOutputStream("index", "html", /*InToplevel=*/true);
+    auto OSOrErr = createOutputStream(Root->FCS.Name.str(), "html", false);
     if (Error E = OSOrErr.takeError())
       return E;
     auto OS = std::move(OSOrErr.get());
     raw_ostream &OSRef = *OS.get();
 
     assert(Opts.hasOutputDirectory() && "No output directory for index file");
-  emitPrelude(OSRef, Opts, getPathToStyle(""));
+    emitPrelude(OSRef, Opts, getPathToStyle(Root->FCS.Name));
 
     // Emit some basic information about the coverage report.
     if (Opts.hasProjectTitle())
       OSRef << tag(ProjectTitleTag, escape(Opts.ProjectTitle, Opts));
     OSRef << tag(ReportTitleTag, "Coverage Report");
     if (Opts.hasCreatedTime())
       OSRef << tag(CreatedTimeTag, escape(Opts.CreatedTimeStr, Opts));
 
     // Emit a link to some documentation.
     OSRef << tag("p", "Click " +
                           a("http://clang.llvm.org/docs/"
                             "SourceBasedCodeCoverage.html#interpreting-reports",
                             "here") +
                           " for information about interpreting this report.");
 
     // Emit a table containing links to reports for each file in the covmapping.
     // Exclude files which don't contain any regions.
     OSRef << BeginCenteredDiv << BeginTable;
     emitColumnLabelsForIndex(OSRef, Opts);
-  FileCoverageSummary Totals("TOTALS");
-  auto FileReports = CoverageReport::prepareFileReports(
-      Coverage, Totals, SourceFiles, Opts, Filters);
     bool EmptyFiles = false;
-  for (unsigned I = 0, E = FileReports.size(); I < E; ++I) {
-    if (FileReports[I].FunctionCoverage.getNumFunctions())
-      emitFileSummary(OSRef, SourceFiles[I], FileReports[I]);
+
+    for (auto Child = Root->Children.begin(), E = Root->Children.end();
+         Child != E; Child++) {
+      if (!(*Child)->Children.empty()) {
+        if (Error E = createIndexFile((*Child), Filters)) {
+          return E;
+        }
+        emitFileSummary(OSRef, (*Child)->Name, (*Child)->FCS,
+                        /*IsTotals=*/false,
+                        /*IsDir=*/true);
+      } else {
+        if ((*Child)->FCS.FunctionCoverage.getNumFunctions())
+          emitFileSummary(OSRef, (*Child)->Name, (*Child)->FCS);
         else
           EmptyFiles = true;
       }
-  emitFileSummary(OSRef, "Totals", Totals, /*IsTotals=*/true);
+    }
+    emitFileSummary(OSRef, "Totals", Root->FCS, /*IsTotals=*/true);
     OSRef << EndTable << EndCenteredDiv;
 
     // Emit links to files which don't contain any functions. These are normally
     // not very useful, but could be relevant for code which abuses the
     // preprocessor.
     if (EmptyFiles && Filters.empty()) {
       OSRef << tag("p", "Files which contain no functions. (These "
                         "files contain code pulled into other files "
                         "by the preprocessor.)\n");
       OSRef << BeginCenteredDiv << BeginTable;
-    for (unsigned I = 0, E = FileReports.size(); I < E; ++I)
-      if (!FileReports[I].FunctionCoverage.getNumFunctions()) {
-        std::string Link = buildLinkToFile(SourceFiles[I], FileReports[I]);
+      for (auto Child = Root->Children.begin(), E = Root->Children.end();
+           Child != E; Child++) {
+        if (!(*Child)->FCS.FunctionCoverage.getNumFunctions()) {
+          std::string Link = buildLinkToFile((*Child)->Name, (*Child)->FCS);
           OSRef << tag("tr", tag("td", tag("pre", Link)), "light-row") << '\n';
         }
+      }
       OSRef << EndTable << EndCenteredDiv;
     }
 
     OSRef << tag("h5", escape(Opts.getLLVMVersionString(), Opts));
     emitEpilog(OSRef);
+  }
+  for (auto Child = Root->Children.begin(), E = Root->Children.end();
+       Child != E; Child++) {
+    if (!(*Child)->Children.empty()) {
+      if (Error E = createIndexFile((*Child), Filters)) {
+        return E;
+      }
+    }
+  }
+  return Error::success();
+}
+
+Error CoveragePrinterHTML::createIndexFile(
+    ArrayRef<std::string> SourceFiles, const CoverageMapping &Coverage,
+    const CoverageFiltersMatchAll &Filters) {
+  // Emit the default stylesheet.
+  auto CSSOrErr = createOutputStream("style", "css", true);
+  if (Error E = CSSOrErr.takeError())
+    return E;
+
+  OwnedStream CSS = std::move(CSSOrErr.get());
+  CSS->operator<<(CSSForCoverage);
 
+  // Traverse the directory tree and emit the index files
+  CoverageTreeNode *IndexTree =
+      buildCoverageTree(SourceFiles, Coverage, Filters);
+  if (Error E = createIndexFile(IndexTree, Filters))
+    return E;
   return Error::success();
 }
 
 void SourceCoverageViewHTML::renderViewHeader(raw_ostream &OS) {
   OS << BeginCenteredDiv << BeginTable;
 }
 
 void SourceCoverageViewHTML::renderViewFooter(raw_ostream &OS) {
if (shouldRenderRegionMarkers(LCS)) {
     for (unsigned I = 0, E = Segments.size() - 1; I < E; ++I) {
       const auto *CurSeg = Segments[I];
       if (!CurSeg->IsRegionEntry)
         continue;
       if (CurSeg->Count == LCS.getExecutionCount())
         continue;
 
       Snippets[I + 1] =
-          tag("div", Snippets[I + 1] + tag("span", formatCount(CurSeg->Count),
-                                           "tooltip-content"),
+          tag("div",
+              Snippets[I + 1] +
+                  tag("span", formatCount(CurSeg->Count), "tooltip-content"),
               "tooltip");
 
       if (getOptions().Debug)
         errs() << "Marker at " << CurSeg->Line << ":" << CurSeg->Col << " = "
                << formatCount(CurSeg->Count) << "\n";
     }
   }