[llvm-cov] Introduce -path-equivalence-regex

[tunz]

Summary

Introduce a new option -path-equivalence-regex=<from>,<to> for llvm-cov show so that it can group the same file represented as multiple different paths in the coverage profile data. Unlike -path-equivalence, this new option takes <from> as a regular expression, remaps matching paths into the given <to> path, and generates outputs with the remapped paths.

More discussions and explanations are https://lists.llvm.org/pipermail/llvm-dev/2021-February/148560.html

[tunz]

Add a test and do not break existing tests.

[tunz]

rebase

[vsk]

Hi @tunz - mind adding a patch summary, and some documentation that describes this feature?

[tunz]

Hi @vsk. Sorry for submitting this patch without any explanation. I was learning and testing this new environment, and was writing an email to llvm-dev.
I've just sent the email to the mailing group. https://lists.llvm.org/pipermail/llvm-dev/2021-February/148560.html

[tunz]

Add -path-equivalence-regex to the llvm-cov doc.

[tunz]

Abandon this since https://reviews.llvm.org/D95753 addresses our issue.

llvm/docs/CommandGuide/llvm-cov.rst

  threshold.
 
 .. option:: -path-equivalence=<from>,<to>
 
  Map the paths in the coverage data to local source file paths. This allows you
  to generate the coverage data on one machine, and then use llvm-cov on a
  different machine where you have the same files on a different path.
 
+.. option:: -path-equivalence-regex=<from>,<to>
+
+ Map the paths in the coverage data to local source file paths. Unlike
+ `-path-equivalence`, this takes <from> as a regular expression. Any string
+ matched by the regular expression in file paths will be replaced to the
+ given <to> string. If there is a captured string in the regular expression,
+ the captured string will be used as a prefix of the new remapped path.
+
 .. program:: llvm-cov report
 
 .. _llvm-cov-report:
 
 REPORT COMMAND
 --------------
 
 SYNOPSIS

llvm/test/tools/llvm-cov/sources-specified.test

 RUN: llvm-cov report -instr-profile %S/Inputs/sources_specified/main.profdata \
 RUN:   -path-equivalence=/tmp,%S/Inputs \
 RUN:   %S/Inputs/sources_specified/main.covmapping \
 RUN:   %S/Inputs/sources_specified/main.cc %S/Inputs/sources_specified/extra \
 RUN:   | FileCheck -check-prefix=REPORT %s
 
 RUN: llvm-cov show -instr-profile %S/Inputs/sources_specified/main.profdata \
 RUN:   -path-equivalence=/tmp,%S/Inputs \
 RUN:   %S/Inputs/sources_specified/main.covmapping \
 RUN:   %S/Inputs/sources_specified/main.cc %S/Inputs/sources_specified/extra \
 RUN:   | FileCheck -check-prefix=SHOW %s
 
+RUN: llvm-cov show -instr-profile %S/Inputs/sources_specified/main.profdata \
+RUN:   -path-equivalence-regex=/tmp,%S/Inputs \
+RUN:   %S/Inputs/sources_specified/main.covmapping \
+RUN:   %S/Inputs/sources_specified/main.cc %S/Inputs/sources_specified/extra \
+RUN:   | FileCheck -check-prefix=SHOWREGEX %s
+
 # Don't include all source files when provided source files are filtered out.
 RUN: llvm-cov show -instr-profile %S/Inputs/sources_specified/main.profdata \
 RUN:   -path-equivalence=/tmp,%S/Inputs \
 RUN:   %S/Inputs/sources_specified/main.covmapping \
 RUN:   --ignore-filename-regex='.*' \
 RUN:   %S/Inputs/sources_specified/main.cc %S/Inputs/sources_specified/extra \
 RUN:   | FileCheck -allow-empty -check-prefix=IGNORE %s
 
 IGNORE-NOT: {{.*}}main.cc{{.*}}
 IGNORE-NOT: {{.*}}inc.h{{.*}}
 IGNORE-NOT: {{.*}}dec.h{{.*}}
 
 # Order of files may differ, check the total values calculated.
 REPORT-NOT: {{.*}}abs.h{{.*}}
 REPORT: {{^}}TOTAL 3{{.*}}72.73%
 
 # Order of files may differ, check that there are 3 files and not abs.h.
 SHOW-NOT: {{.*}}abs.h{{.*}}
 SHOW: {{.*}}sources_specified{{.*}}
 SHOW: {{.*}}sources_specified{{.*}}
 SHOW: {{.*}}sources_specified{{.*}}
 
+# Order of files may differ, check that there are 3 files and not abs.h.
+SHOWREGEX-NOT: {{.*}}abs.h{{.*}}
+SHOWREGEX: {{.*}}sources_specified{{.*}}
+SHOWREGEX: {{.*}}sources_specified{{.*}}
+SHOWREGEX: {{.*}}sources_specified{{.*}}
+
 
 # Test "export" command. Use a temp .json file as output is a single line.
 RUN: llvm-cov export -instr-profile %S/Inputs/sources_specified/main.profdata \
 RUN:   -path-equivalence=/tmp,%S/Inputs \
 RUN:   %S/Inputs/sources_specified/main.covmapping \
 RUN:   %S/Inputs/sources_specified/main.cc %S/Inputs/sources_specified/extra \
 RUN:   > %t.export.json
 

llvm/tools/llvm-cov/CodeCoverage.cpp

 #include "llvm/ProfileData/InstrProfReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
+#include "llvm/Support/Regex.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/Support/SpecialCaseList.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/Threading.h"
 #include "llvm/Support/ToolOutputFile.h"
 #include "llvm/Support/VirtualFileSystem.h"
 
 #include <functional>
private:
   /// Convert \p Path into an absolute path and append it to the list
   /// of collected paths.
   void addCollectedPath(const std::string &Path);
 
   /// If \p Path is a regular file, collect the path. If it's a
   /// directory, recursively collect all of the paths within the directory.
   void collectPaths(const std::string &Path);
 
+  /// Return the actual file path of a given source file
+  StringRef getSourceFilePath(StringRef SourceFile);
+
   /// Return a memory buffer for the given source file.
   ErrorOr<const MemoryBuffer &> getSourceFile(StringRef SourceFile);
 
   /// Create source views for the expansions of the view.
   void attachExpansionSubViews(SourceCoverageView &View,
                                ArrayRef<ExpansionRecord> Expansions,
                                const CoverageMapping &Coverage);
 
private:
 
   /// Create the main source view of a particular source file.
   std::unique_ptr<SourceCoverageView>
   createSourceFileView(StringRef SourceFile, const CoverageMapping &Coverage);
 
   /// Load the coverage mapping data. Return nullptr if an error occurred.
   std::unique_ptr<CoverageMapping> load();
 
+  // Convert remapping paths to native paths with trailing seperators.
+  std::string nativeWithTrailing(StringRef Path, bool RemoveDots);
+
+  // Initialize RemappedFilenames by a regular expression of the coverage data
+  // path (path-equivalence-regex).
+  void remapPathsFromRegex(const CoverageMapping &Coverage);
+
+  // Initialize RemappedFilenames by a string of the coverage data path
+  // (path-equivalence).
+  void remapPathsFromString(const CoverageMapping &Coverage);
+
   /// Create a mapping from files in the Coverage data to local copies
   /// (path-equivalence).
   void remapPathNames(const CoverageMapping &Coverage);
 
   /// Remove input source files which aren't mapped by \p Coverage.
   void removeUnmappedInputs(const CoverageMapping &Coverage);
 
   /// If a demangler is available, demangle all symbol names.
private:
   /// In -path-equivalence mode, this maps the absolute paths from the coverage
   /// mapping data to the input source files.
   StringMap<std::string> RemappedFilenames;
 
   /// The coverage data path to be remapped from, and the source path to be
   /// remapped to, when using -path-equivalence.
   Optional<std::pair<std::string, std::string>> PathRemapping;
 
+  // True if the first value of PathRemapping is a regular expression.
+  bool PathRemappingRegex = false;
+
   /// The architecture the coverage mapping data targets.
   std::vector<StringRef> CoverageArches;
 
   /// A cache for demangled symbols.
   DemangleCache DC;
 
   /// A lock which guards printing to stderr.
   std::mutex ErrsLock;
for (llvm::sys::fs::recursive_directory_iterator F(Path, EC), E;
       }
 
       if (Status->type() == llvm::sys::fs::file_type::regular_file)
         addCollectedPath(F->path());
     }
   }
 }
 
+StringRef CodeCoverageTool::getSourceFilePath(StringRef SourceFile) {
+  if (!RemappedFilenames.empty()) {
+    auto LocalFile = RemappedFilenames.find(SourceFile);
+    if (LocalFile != RemappedFilenames.end())
+      SourceFile = LocalFile->second;
+  }
+
+  return SourceFile;
+}
+
 ErrorOr<const MemoryBuffer &>
 CodeCoverageTool::getSourceFile(StringRef SourceFile) {
   // If we've remapped filenames, look up the real location for this file.
   std::unique_lock<std::mutex> Guard{LoadedSourceFilesLock};
-  if (!RemappedFilenames.empty()) {
-    auto Loc = RemappedFilenames.find(SourceFile);
-    if (Loc != RemappedFilenames.end())
-      SourceFile = Loc->second;
-  }
+  SourceFile = getSourceFilePath(SourceFile);
   for (const auto &Files : LoadedSourceFiles)
-    if (sys::fs::equivalent(SourceFile, Files.first))
+    if ((!PathRemappingRegex && sys::fs::equivalent(SourceFile, Files.first)) ||
+        (PathRemappingRegex && SourceFile == Files.first))
       return *Files.second;
   auto Buffer = MemoryBuffer::getFile(SourceFile);
   if (auto EC = Buffer.getError()) {
     error(EC.message(), SourceFile);
     return EC;
   }
   LoadedSourceFiles.emplace_back(std::string(SourceFile),
                                  std::move(Buffer.get()));
attachBranchSubViews(*View, DC.demangle(Function.Name), Branches,
                        SourceBuffer.get(), FunctionCoverage);
 
   return View;
 }
 
 std::unique_ptr<SourceCoverageView>
 CodeCoverageTool::createSourceFileView(StringRef SourceFile,
                                        const CoverageMapping &Coverage) {
+  auto RemappedSourceFile = SourceFile;
+  if (PathRemappingRegex)
+    RemappedSourceFile = getSourceFilePath(SourceFile);
+
   auto SourceBuffer = getSourceFile(SourceFile);
   if (!SourceBuffer)
     return nullptr;
   auto FileCoverage = Coverage.getCoverageForFile(SourceFile);
   if (FileCoverage.empty())
     return nullptr;
 
   auto Branches = FileCoverage.getBranches();
   auto Expansions = FileCoverage.getExpansions();
-  auto View = SourceCoverageView::create(SourceFile, SourceBuffer.get(),
+  auto View = SourceCoverageView::create(RemappedSourceFile, SourceBuffer.get(),
                                          ViewOpts, std::move(FileCoverage));
   attachExpansionSubViews(*View, Expansions, Coverage);
-  attachBranchSubViews(*View, SourceFile, Branches, SourceBuffer.get(),
+  attachBranchSubViews(*View, RemappedSourceFile, Branches, SourceBuffer.get(),
                        FileCoverage);
   if (!ViewOpts.ShowFunctionInstantiations)
     return View;
 
   for (const auto &Group : Coverage.getInstantiationGroups(SourceFile)) {
     // Skip functions which have a single instantiation.
     if (Group.size() < 2)
       continue;
std::unique_ptr<CoverageMapping> CodeCoverageTool::load() {
   if (!SourceFiles.empty())
     removeUnmappedInputs(*Coverage);
 
   demangleSymbols(*Coverage);
 
   return Coverage;
 }
 
-void CodeCoverageTool::remapPathNames(const CoverageMapping &Coverage) {
-  if (!PathRemapping)
-    return;
-
-  // Convert remapping paths to native paths with trailing seperators.
-  auto nativeWithTrailing = [](StringRef Path) -> std::string {
-    if (Path.empty())
-      return "";
-    SmallString<128> NativePath;
-    sys::path::native(Path, NativePath);
+std::string CodeCoverageTool::nativeWithTrailing(StringRef Path,
+                                                 bool RemoveDots) {
+  SmallString<256> NativePath;
+  if (RemoveDots)
     sys::path::remove_dots(NativePath, true);
-    if (!sys::path::is_separator(NativePath.back()))
-      NativePath += sys::path::get_separator();
-    return NativePath.c_str();
-  };
-  std::string RemapFrom = nativeWithTrailing(PathRemapping->first);
-  std::string RemapTo = nativeWithTrailing(PathRemapping->second);
+
+void CodeCoverageTool::remapPathsFromRegex(const CoverageMapping &Coverage) {
+  llvm::Regex RemapFrom = llvm::Regex(PathRemapping->first);
+  std::string RemapTo =
+      llvm::Regex::escape(nativeWithTrailing(PathRemapping->second, false));
+
+  // Create a mapping from coverage data file paths to local paths.
+  for (StringRef Filename : Coverage.getUniqueSourceFiles()) {
+    SmallVector<StringRef, 1> Matches;
+    if (RemapFrom.match(Filename.str(), &Matches)) {
+      SmallString<256> RemappedNativeFilename;
+      if (Matches.size() == 2)
+        sys::path::native(Matches[1].str() +
+                              RemapFrom.sub(RemapTo, Filename.str()),
+                          RemappedNativeFilename);
+      else
+        sys::path::native(RemapFrom.sub(RemapTo, Filename.str()),
+                          RemappedNativeFilename);
+
+      sys::path::remove_dots(RemappedNativeFilename, true);
+      RemappedFilenames[Filename] = std::string(RemappedNativeFilename.str());
+    }
+  }
+}
+
+void CodeCoverageTool::remapPathsFromString(const CoverageMapping &Coverage) {
+  std::string RemapFrom = nativeWithTrailing(PathRemapping->first, true);
+  std::string RemapTo = nativeWithTrailing(PathRemapping->second, true);
 
   // Create a mapping from coverage data file paths to local paths.
   for (StringRef Filename : Coverage.getUniqueSourceFiles()) {
     SmallString<128> NativeFilename;
     sys::path::native(Filename, NativeFilename);
     sys::path::remove_dots(NativeFilename, true);
     if (NativeFilename.startswith(RemapFrom)) {
       RemappedFilenames[Filename] =
           RemapTo + NativeFilename.substr(RemapFrom.size()).str();
     }
   }
+}
+
+void CodeCoverageTool::remapPathNames(const CoverageMapping &Coverage) {
+  if (!PathRemapping)
+    return;
+
+  if (PathRemappingRegex)
+    remapPathsFromRegex(Coverage);
+  else
+    remapPathsFromString(Coverage);
 
   // Convert input files from local paths to coverage data file paths.
   StringMap<std::string> InvRemappedFilenames;
   for (const auto &RemappedFilename : RemappedFilenames)
     InvRemappedFilenames[RemappedFilename.getValue()] =
         std::string(RemappedFilename.getKey());
 
   for (std::string &Filename : SourceFiles) {
-    SmallString<128> NativeFilename;
+    SmallString<256> NativeFilename;
     sys::path::native(Filename, NativeFilename);
     auto CovFileName = InvRemappedFilenames.find(NativeFilename);
     if (CovFileName != InvRemappedFilenames.end())
       Filename = CovFileName->second;
   }
 }
 
 void CodeCoverageTool::removeUnmappedInputs(const CoverageMapping &Coverage) {
void CodeCoverageTool::writeSourceFileView(StringRef SourceFile,
                                            CoveragePrinter *Printer,
                                            bool ShowFilenames) {
   auto View = createSourceFileView(SourceFile, *Coverage);
   if (!View) {
     warning("The file '" + SourceFile + "' isn't covered.");
     return;
   }
 
+  if (PathRemappingRegex)
+    SourceFile = getSourceFilePath(SourceFile);
   auto OSOrErr = Printer->createViewFile(SourceFile, /*InToplevel=*/false);
   if (Error E = OSOrErr.takeError()) {
     error("Could not create view file!", toString(std::move(E)));
     return;
   }
   auto OS = std::move(OSOrErr.get());
 
   View->print(*OS.get(), /*Wholefile=*/true,
cl::opt<CoverageViewOptions::OutputFormat> Format(
                             "lcov tracefile output")),
       cl::init(CoverageViewOptions::OutputFormat::Text));
 
   cl::opt<std::string> PathRemap(
       "path-equivalence", cl::Optional,
       cl::desc("<from>,<to> Map coverage data paths to local source file "
                "paths"));
 
+  cl::opt<std::string> PathRemapRegex(
+      "path-equivalence-regex", cl::Optional,
+      cl::desc("<from>,<to> Map regular expression of coverage data paths "
+               "to local source file paths."));
+
   cl::OptionCategory FilteringCategory("Function filtering options");
 
   cl::list<std::string> NameFilters(
       "name", cl::Optional,
       cl::desc("Show code coverage only for functions with the given name"),
       cl::ZeroOrMore, cl::cat(FilteringCategory));
 
   cl::list<std::string> NameFilterFiles(
auto commandLineParser = [&, this](int argc, const char **argv) -> int {
 
     // If path-equivalence was given and is a comma seperated pair then set
     // PathRemapping.
     auto EquivPair = StringRef(PathRemap).split(',');
     if (!(EquivPair.first.empty() && EquivPair.second.empty()))
       PathRemapping = {std::string(EquivPair.first),
                        std::string(EquivPair.second)};
 
+    auto EquivRegexPair = StringRef(PathRemapRegex).split(',');
+    if (!(EquivRegexPair.first.empty() && EquivRegexPair.second.empty())) {
+      PathRemapping = {std::string(EquivRegexPair.first),
+                       std::string(EquivRegexPair.second)};
+      PathRemappingRegex = true;
+    }
+
     // If a demangler is supplied, check if it exists and register it.
     if (!DemanglerOpts.empty()) {
       auto DemanglerPathOrErr = sys::findProgramByName(DemanglerOpts[0]);
       if (!DemanglerPathOrErr) {
         error("Could not find the demangler!",
               DemanglerPathOrErr.getError().message());
         return 1;
       }