Sancov in C++.

[aizatsky]

Summary

First batch of sancov.py rewrite in C++.
Supports "-print" and "-coveredfns" commands.

[aizatsky]

• cleanup

[kcc]

is a test possible here?

lib/sanitizer_common/sancov.cc

86

I wonder if you can reuse functionality from clang/Basic/FileManager.h
It's rather sad that we need to re-implement such basic stuff here.

[silvas]

lib/sanitizer_common/sancov.cc

86

There's a FIXME on FileManager::removeDotPaths saying FIXME: Move this to llvm::sys::path. so I think now is a perfect time to address that FIXME.

[aizatsky]

CLs with path utility are out for review:

http://reviews.llvm.org/D14393
http://reviews.llvm.org/D14394

[aizatsky]

merge

[aizatsky]

Updated to use remove_dots utility fn.

[kcc]

+krasin

[kcc]

more to follow

lib/sanitizer_common/sancov.cc

45

use full words, please, e.g. covered_functions

97

no {} in such code.

155

size_t?

157

<optional> I am not a huge fan of lambda functions when they don't save code or make it more readable.
Will it not be shorter *and* simpler to write regular loop here? </optional>

164

won't it be simpler to insert everything into a set, and they copy into a vector?

[kcc]

lib/sanitizer_common/sancov.cc

193

printCoveredFunctions

218

not necessary in this change, but we'll need an option for demangling, similar to "-functions" option of llvm-symbolizer

227

no {}

245

no {}

[aizatsky]

review

[aizatsky]

All done. PTAL.

lib/sanitizer_common/sancov.cc

164

Wouldn't that be slower?

218

why do we need not-demangled output?

[kcc]

lib/sanitizer_common/sancov.cc

165

First, speed is not important here, code simplicity is much more so.

Second, I actually think it might be faster in case you need to merge lots of coverage files.
Your current code reserves vector for the total combined size of all the files.
Using set you will only need memory proportional to the result (yes, with a large multiplicative constant).

219

Sometimes you may need to pass this further to some scripts and there is no way to un-demangle function names back.
Sometimes the fully demangled names are too long and we need "short names"

[aizatsky]

review

[aizatsky]

added flag & using set.

[kcc]

lib/sanitizer_common/sancov.cc

158

did you mean unordered_set?

163

I wouldn't bother with reserve, but I would actually use set, not an unordered_set so that the resulting PCs are sorted (just in case).

164

no {}

[aizatsky]

review

[aizatsky]

all done.

[aizatsky]

review

[kcc]

LGTM

lib/sanitizer_common/CMakeLists.txt

add_compiler_rt_object_libraries(RTSanitizerCommonLibc
   SOURCES ${SANITIZER_LIBCDEP_SOURCES}
   CFLAGS ${SANITIZER_CFLAGS}
   DEFS ${SANITIZER_COMMON_DEFINITIONS})
 
 # Unit tests for common sanitizer runtime.
 if(COMPILER_RT_INCLUDE_TESTS)
   add_subdirectory(tests)
 endif()
+
+add_llvm_tool(sancov sancov.cc)
+target_link_libraries(sancov LLVMSupport LLVMSymbolize LLVMObject LLVMDebugInfoDWARF LLVMDebugInfoPDB)

lib/sanitizer_common/sancov.cc

+//===-- sancov.cc --------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a command-line tool for reading and analyzing sanitizer
+// coverage.
+//===----------------------------------------------------------------------===//
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/DebugInfo/Symbolize/Symbolize.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/LineIterator.h"
+#include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/PrettyStackTrace.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <cxxabi.h>
+#include <set>
+#include <stdio.h>
+#include <vector>
+
+using namespace llvm;
+
+namespace {
+
+// --------- COMMAND LINE FLAGS ---------
+
+enum ActionType { PrintAction, CoveredFunctionsAction };
+
+cl::opt<ActionType> Action(
+    cl::desc("Action (required)"), cl::Required,
+    cl::values(clEnumValN(PrintAction, "print", "Print coverage addresses"),
+               clEnumValN(CoveredFunctionsAction, "coveredfns",
+                          "Print all covered funcions."),
+               clEnumValEnd));
+
+static cl::list<std::string> ClInputFiles(cl::Positional, cl::OneOrMore,
+                                          cl::desc("<filenames...>"));
+
+static cl::opt<std::string>
+    ClBinaryName("obj", cl::Required,
+                 cl::desc("Path to object file to be symbolized"));
+
+// --------- FORMAT SPECIFICATION ---------
+
+struct FileHeader {
+  uint32_t Bitness;
+  uint32_t Magic;
+};
+
+static const uint32_t BinCoverageMagic = 0xC0BFFFFF;
+static const uint32_t Bitness32 = 0xFFFFFF32;
+static const uint32_t Bitness64 = 0xFFFFFF64;
+
+// ---------
+
+template <typename T> static void FailIfError(const ErrorOr<T> &E) {
+  if (E)
+    return;
+
+  auto Error = E.getError();
+  errs() << "Error: " << Error.message() << "(" << Error.value() << ")\n";
+  exit(-2);
+}
+
+template <typename T>
+static void readInts(const char *Start, const char *End,
+                     std::vector<uint64_t> *V) {
+  const T *S = reinterpret_cast<const T *>(Start);
+  const T *E = reinterpret_cast<const T *>(End);
+  V->reserve(E - S);
+  std::copy(S, E, std::back_inserter(*V));
+}
+
+static bool removeDotPaths(SmallVectorImpl<char> *Path) {
+  using namespace llvm::sys;
+
+  SmallVector<StringRef, 16> ComponentStack;
+  StringRef P(Path->data(), Path->size());
+
+  // Skip the root path, then look for traversal in the components.
+  StringRef Rel = path::relative_path(P);
+  for (StringRef C : llvm::make_range(path::begin(Rel), path::end(Rel))) {
+    if (C == ".")
+      continue;
+    if (C == "..") {
+      if (!ComponentStack.empty())
+        ComponentStack.pop_back();
+      continue;
+    }
+    ComponentStack.push_back(C);
+  }
+
+  SmallString<256> Buffer = path::root_path(P);
+  for (StringRef C : ComponentStack)
+    path::append(Buffer, C);
+
+  bool Changed = (*Path != Buffer);
+  Path->swap(Buffer);
+  return Changed;
+}
+
+static std::string CommonPrefix(std::string A, std::string B) {
+  if (A.size() > B.size())
+    return std::string(B.begin(),
+                       std::mismatch(B.begin(), B.end(), A.begin()).first);
+  else
+    return std::string(A.begin(),
+                       std::mismatch(A.begin(), A.end(), B.begin()).first);
+}
+
+static std::string demangle(std::string Name) {
+  if (Name.substr(0, 2) != "_Z") {
+    return Name;
+  }
+
+  int status = 0;
+  char *DemangledName =
+      abi::__cxa_demangle(Name.c_str(), nullptr, nullptr, &status);
+  if (status != 0)
+    return Name;
+  std::string Result = DemangledName;
+  free(DemangledName);
+  return Result;
+}
+
+class CoverageData {
+public:
+  // Read single file coverage data.
+  static ErrorOr<std::unique_ptr<CoverageData>> read(std::string FileName) {
+    ErrorOr<std::unique_ptr<MemoryBuffer>> BufOrErr =
+        MemoryBuffer::getFile(FileName);
+    if (!BufOrErr)
+      return BufOrErr.getError();
+    std::unique_ptr<MemoryBuffer> Buf = std::move(BufOrErr.get());
+    if (Buf->getBufferSize() < 8) {
+      errs() << "File too small (<8): " << Buf->getBufferSize();
+      return make_error_code(errc::illegal_byte_sequence);
+    }
+    const FileHeader *Header =
+        reinterpret_cast<const FileHeader *>(Buf->getBufferStart());
+
+    if (Header->Magic != BinCoverageMagic) {
+      errs() << "Wrong magic: " << Header->Magic;
+      return make_error_code(errc::illegal_byte_sequence);
+    }
+
+    std::unique_ptr<std::vector<uint64_t>> Addrs(new std::vector<uint64_t>());
+
+    switch (Header->Bitness) {
+    case Bitness64:
+      readInts<uint64_t>(Buf->getBufferStart() + 8, Buf->getBufferEnd(),
+                         Addrs.get());
+      break;
+    case Bitness32:
+      readInts<uint32_t>(Buf->getBufferStart() + 8, Buf->getBufferEnd(),
+                         Addrs.get());
+      break;
+    default:
+      errs() << "Unsupported bitness: " << Header->Bitness;
+      return make_error_code(errc::illegal_byte_sequence);
+    }
+
+    return std::unique_ptr<CoverageData>(new CoverageData(std::move(Addrs)));
+  }
+
+  // Merge multiple coverage data together.
+  static std::unique_ptr<CoverageData>
+  merge(const std::vector<std::unique_ptr<CoverageData>> &Covs) {
+    std::unique_ptr<std::vector<uint64_t>> Addrs(new std::vector<uint64_t>());
+    int Sz = std::accumulate(
+        Covs.begin(), Covs.end(), 0,
+        [](const int &A, const std::unique_ptr<CoverageData> &B) {
+          return A + B->Addrs->size();
+        });
+    Addrs->reserve(Sz);
+    for (const auto &Cov : Covs) {
+      Addrs->insert(Addrs->end(), Cov->Addrs->begin(), Cov->Addrs->end());
+    }
+    std::sort(Addrs->begin(), Addrs->end());
+    Addrs->erase(std::unique(Addrs->begin(), Addrs->end()), Addrs->end());
+    return std::unique_ptr<CoverageData>(new CoverageData(std::move(Addrs)));
+  }
+
+  // Read list of files and merges their coverage info.
+  static ErrorOr<std::unique_ptr<CoverageData>>
+  readAndMerge(const std::vector<std::string> &FileNames) {
+    std::vector<std::unique_ptr<CoverageData>> Covs;
+    for (const auto &FileName : FileNames) {
+      auto Cov = read(FileName);
+      if (!Cov)
+        return Cov.getError();
+      Covs.push_back(std::move(Cov.get()));
+    }
+    return merge(Covs);
+  }
+
+  // Print coverage addresses.
+  void printAddrs(raw_ostream &out) {
+    for (auto Addr : *Addrs) {
+      out << "0x";
+      out.write_hex(Addr);
+      out << "\n";
+    }
+  }
+
+  // Print list of covered functions.
+  // Line format: <file_name>:<line> <function_name>
+  void printCoveredFns(raw_ostream &out) {
+    if (Addrs->empty())
+      return;
+    symbolize::LLVMSymbolizer Symbolizer;
+
+    struct FileLoc {
+      std::string FileName;
+      uint32_t Line;
+      bool operator<(const FileLoc &Rhs) const {
+        return std::tie(FileName, Line) < std::tie(Rhs.FileName, Rhs.Line);
+      }
+    };
+
+    // FileLoc -> FunctionName
+    std::map<FileLoc, std::string> Fns;
+
+    // Fill in Fns map.
+    for (auto Addr : *Addrs) {
+      auto InliningInfo = Symbolizer.symbolizeInlinedCode(ClBinaryName, Addr);
+      FailIfError(InliningInfo);
+      for (uint32_t i = 0; i < InliningInfo->getNumberOfFrames(); ++i) {
+        auto FrameInfo = InliningInfo->getFrame(i);
+        SmallString<256> FileName(FrameInfo.FileName);
+        removeDotPaths(&FileName);
+        uint32_t Line = FrameInfo.Line;
+        std::string FunctionName = demangle(FrameInfo.FunctionName);
+
+        FileLoc Loc{FileName.str(), Line};
+        Fns[Loc] = FunctionName;
+      }
+    }
+
+    // Compute file names common prefix.
+    std::string FilePrefix = Fns.begin()->first.FileName;
+    for (const auto &P : Fns) {
+      FilePrefix = CommonPrefix(FilePrefix, P.first.FileName);
+    }
+
+    // Print first function occurence in a file.
+    {
+      std::string LastFileName;
+      std::set<std::string> ProcessedFunctions;
+
+      for (const auto &P : Fns) {
+        std::string FileName = P.first.FileName;
+        std::string FunctionName = P.second;
+        uint32_t Line = P.first.Line;
+
+        if (LastFileName != FileName)
+          ProcessedFunctions.clear();
+        LastFileName = FileName;
+
+        if (ProcessedFunctions.find(FunctionName) != ProcessedFunctions.end()) {
+          continue;
+        }
+        ProcessedFunctions.insert(FunctionName);
+
+        out << FileName.substr(FilePrefix.size()) << ":" << Line << " "
+            << FunctionName << "\n";
+      }
+    }
+  }
+
+private:
+  explicit CoverageData(std::unique_ptr<std::vector<uint64_t>> Addrs)
+      : Addrs(std::move(Addrs)) {}
+
+  std::unique_ptr<std::vector<uint64_t>> Addrs;
+};
+} // namespace
+
+int main(int argc, char **argv) {
+  // Print stack trace if we signal out.
+  sys::PrintStackTraceOnErrorSignal();
+  PrettyStackTraceProgram X(argc, argv);
+  llvm_shutdown_obj Y; // Call llvm_shutdown() on exit.
+
+  cl::ParseCommandLineOptions(argc, argv, "Sanitizer Coverage Processing Tool");
+
+  auto CovData = CoverageData::readAndMerge(ClInputFiles);
+  FailIfError(CovData);
+
+  switch (Action) {
+  case PrintAction: {
+    CovData.get()->printAddrs(outs());
+    return 0;
+  }
+  case CoveredFunctionsAction: {
+    CovData.get()->printCoveredFns(outs());
+    return 0;
+  }
+  }
+}