Sancov in C++.

[aizatsky]

Summary

First batch of sancov.py rewrite in C++.
Supports "-print" and "-coveredfns" commands.

[aizatsky]

• cleanup

[kcc]

is a test possible here?

lib/sanitizer_common/sancov.cc

86 ↗

(On Diff #39284)

I wonder if you can reuse functionality from clang/Basic/FileManager.h
It's rather sad that we need to re-implement such basic stuff here.

[silvas]

lib/sanitizer_common/sancov.cc

86 ↗

(On Diff #39284)

There's a FIXME on FileManager::removeDotPaths saying FIXME: Move this to llvm::sys::path. so I think now is a perfect time to address that FIXME.

[aizatsky]

CLs with path utility are out for review:

http://reviews.llvm.org/D14393
http://reviews.llvm.org/D14394

[aizatsky]

merge

[aizatsky]

Updated to use remove_dots utility fn.

[kcc]

+krasin

[kcc]

more to follow

lib/sanitizer_common/sancov.cc

44 ↗

(On Diff #39732)

use full words, please, e.g. covered_functions

96 ↗

(On Diff #39732)

no {} in such code.

154 ↗

(On Diff #39732)

size_t?

156 ↗

(On Diff #39732)

<optional> I am not a huge fan of lambda functions when they don't save code or make it more readable.
Will it not be shorter *and* simpler to write regular loop here? </optional>

163 ↗

(On Diff #39732)

won't it be simpler to insert everything into a set, and they copy into a vector?

[kcc]

lib/sanitizer_common/sancov.cc

192 ↗

(On Diff #39732)

printCoveredFunctions

217 ↗

(On Diff #39732)

not necessary in this change, but we'll need an option for demangling, similar to "-functions" option of llvm-symbolizer

226 ↗

(On Diff #39732)

no {}

244 ↗

(On Diff #39732)

no {}

[aizatsky]

review

[aizatsky]

All done. PTAL.

lib/sanitizer_common/sancov.cc

163 ↗

(On Diff #39732)

Wouldn't that be slower?

217 ↗

(On Diff #39732)

why do we need not-demangled output?

[kcc]

lib/sanitizer_common/sancov.cc

164 ↗

(On Diff #39818)

First, speed is not important here, code simplicity is much more so.

Second, I actually think it might be faster in case you need to merge lots of coverage files.
Your current code reserves vector for the total combined size of all the files.
Using set you will only need memory proportional to the result (yes, with a large multiplicative constant).

218 ↗

(On Diff #39818)

Sometimes you may need to pass this further to some scripts and there is no way to un-demangle function names back.
Sometimes the fully demangled names are too long and we need "short names"

[aizatsky]

review

[aizatsky]

added flag & using set.

[kcc]

lib/sanitizer_common/sancov.cc

157 ↗

(On Diff #39854)

did you mean unordered_set?

162 ↗

(On Diff #39854)

I wouldn't bother with reserve, but I would actually use set, not an unordered_set so that the resulting PCs are sorted (just in case).

163 ↗

(On Diff #39854)

no {}

[aizatsky]

review

[aizatsky]

all done.

[aizatsky]

review

[kcc]

LGTM

compiler-rt/trunk/lib/sanitizer_common/CMakeLists.txt

add_compiler_rt_object_libraries(RTSanitizerCommonLibc
   SOURCES ${SANITIZER_LIBCDEP_SOURCES}
   CFLAGS ${SANITIZER_CFLAGS}
   DEFS ${SANITIZER_COMMON_DEFINITIONS})
 
 # Unit tests for common sanitizer runtime.
 if(COMPILER_RT_INCLUDE_TESTS)
   add_subdirectory(tests)
 endif()
+
+add_llvm_tool(sancov sancov.cc)
+target_link_libraries(sancov LLVMSupport LLVMSymbolize LLVMObject LLVMDebugInfoDWARF LLVMDebugInfoPDB)

compiler-rt/trunk/lib/sanitizer_common/sancov.cc

+//===-- sancov.cc --------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is a command-line tool for reading and analyzing sanitizer
+// coverage.
+//===----------------------------------------------------------------------===//
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/DebugInfo/Symbolize/Symbolize.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/LineIterator.h"
+#include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/PrettyStackTrace.h"
+#include "llvm/Support/Signals.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include <cxxabi.h>
+#include <set>
+#include <stdio.h>
+#include <vector>
+
+using namespace llvm;
+
+namespace {
+
+// --------- COMMAND LINE FLAGS ---------
+
+enum ActionType { PrintAction, CoveredFunctionsAction };
+
+cl::opt<ActionType> Action(
+    cl::desc("Action (required)"), cl::Required,
+    cl::values(clEnumValN(PrintAction, "print", "Print coverage addresses"),
+               clEnumValN(CoveredFunctionsAction, "covered_functions",
+                          "Print all covered funcions."),
+               clEnumValEnd));
+
+static cl::list<std::string> ClInputFiles(cl::Positional, cl::OneOrMore,
+                                          cl::desc("<filenames...>"));
+
+static cl::opt<std::string>
+    ClBinaryName("obj", cl::Required,
+                 cl::desc("Path to object file to be symbolized"));
+
+static cl::opt<bool>
+    ClDemangle("demangle", cl::init(true),
+        cl::desc("Print demangled function name."));
+
+// --------- FORMAT SPECIFICATION ---------
+
+struct FileHeader {
+  uint32_t Bitness;
+  uint32_t Magic;
+};
+
+static const uint32_t BinCoverageMagic = 0xC0BFFFFF;
+static const uint32_t Bitness32 = 0xFFFFFF32;
+static const uint32_t Bitness64 = 0xFFFFFF64;
+
+// ---------
+
+template <typename T> static void FailIfError(const ErrorOr<T> &E) {
+  if (E)
+    return;
+
+  auto Error = E.getError();
+  errs() << "Error: " << Error.message() << "(" << Error.value() << ")\n";
+  exit(-2);
+}
+
+template <typename T>
+static void readInts(const char *Start, const char *End,
+                     std::vector<uint64_t> *V) {
+  const T *S = reinterpret_cast<const T *>(Start);
+  const T *E = reinterpret_cast<const T *>(End);
+  V->reserve(E - S);
+  std::copy(S, E, std::back_inserter(*V));
+}
+
+static std::string CommonPrefix(std::string A, std::string B) {
+  if (A.size() > B.size())
+    return std::string(B.begin(),
+                       std::mismatch(B.begin(), B.end(), A.begin()).first);
+  else
+    return std::string(A.begin(),
+                       std::mismatch(A.begin(), A.end(), B.begin()).first);
+}
+
+static std::string demangle(std::string Name) {
+  if (Name.substr(0, 2) != "_Z")
+    return Name;
+
+  int status = 0;
+  char *DemangledName =
+      abi::__cxa_demangle(Name.c_str(), nullptr, nullptr, &status);
+  if (status != 0)
+    return Name;
+  std::string Result = DemangledName;
+  free(DemangledName);
+  return Result;
+}
+
+class CoverageData {
+ public:
+  // Read single file coverage data.
+  static ErrorOr<std::unique_ptr<CoverageData>> read(std::string FileName) {
+    ErrorOr<std::unique_ptr<MemoryBuffer>> BufOrErr =
+        MemoryBuffer::getFile(FileName);
+    if (!BufOrErr)
+      return BufOrErr.getError();
+    std::unique_ptr<MemoryBuffer> Buf = std::move(BufOrErr.get());
+    if (Buf->getBufferSize() < 8) {
+      errs() << "File too small (<8): " << Buf->getBufferSize();
+      return make_error_code(errc::illegal_byte_sequence);
+    }
+    const FileHeader *Header =
+        reinterpret_cast<const FileHeader *>(Buf->getBufferStart());
+
+    if (Header->Magic != BinCoverageMagic) {
+      errs() << "Wrong magic: " << Header->Magic;
+      return make_error_code(errc::illegal_byte_sequence);
+    }
+
+    std::unique_ptr<std::vector<uint64_t>> Addrs(new std::vector<uint64_t>());
+
+    switch (Header->Bitness) {
+    case Bitness64:
+      readInts<uint64_t>(Buf->getBufferStart() + 8, Buf->getBufferEnd(),
+                         Addrs.get());
+      break;
+    case Bitness32:
+      readInts<uint32_t>(Buf->getBufferStart() + 8, Buf->getBufferEnd(),
+                         Addrs.get());
+      break;
+    default:
+      errs() << "Unsupported bitness: " << Header->Bitness;
+      return make_error_code(errc::illegal_byte_sequence);
+    }
+
+    return std::unique_ptr<CoverageData>(new CoverageData(std::move(Addrs)));
+  }
+
+  // Merge multiple coverage data together.
+  static std::unique_ptr<CoverageData>
+  merge(const std::vector<std::unique_ptr<CoverageData>> &Covs) {
+    std::set<uint64_t> Addrs;
+
+    for (const auto &Cov : Covs)
+      Addrs.insert(Cov->Addrs->begin(), Cov->Addrs->end());
+
+    std::unique_ptr<std::vector<uint64_t>> AddrsVector(
+        new std::vector<uint64_t>(Addrs.begin(), Addrs.end()));
+    return std::unique_ptr<CoverageData>(
+        new CoverageData(std::move(AddrsVector)));
+  }
+
+  // Read list of files and merges their coverage info.
+  static ErrorOr<std::unique_ptr<CoverageData>>
+  readAndMerge(const std::vector<std::string> &FileNames) {
+    std::vector<std::unique_ptr<CoverageData>> Covs;
+    for (const auto &FileName : FileNames) {
+      auto Cov = read(FileName);
+      if (!Cov)
+        return Cov.getError();
+      Covs.push_back(std::move(Cov.get()));
+    }
+    return merge(Covs);
+  }
+
+  // Print coverage addresses.
+  void printAddrs(raw_ostream &out) { // NOLINT(runtime/references)
+    for (auto Addr : *Addrs) {
+      out << "0x";
+      out.write_hex(Addr);
+      out << "\n";
+    }
+  }
+
+  // Print list of covered functions.
+  // Line format: <file_name>:<line> <function_name>
+  void printCoveredFunctions(raw_ostream &out) { // NOLINT(runtime/references)
+    if (Addrs->empty())
+      return;
+    symbolize::LLVMSymbolizer Symbolizer;
+
+    struct FileLoc {
+      std::string FileName;
+      uint32_t Line;
+      bool operator<(const FileLoc &Rhs) const {
+        return std::tie(FileName, Line) < std::tie(Rhs.FileName, Rhs.Line);
+      }
+    };
+
+    // FileLoc -> FunctionName
+    std::map<FileLoc, std::string> Fns;
+
+    // Fill in Fns map.
+    for (auto Addr : *Addrs) {
+      auto InliningInfo = Symbolizer.symbolizeInlinedCode(ClBinaryName, Addr);
+      FailIfError(InliningInfo);
+      for (uint32_t i = 0; i < InliningInfo->getNumberOfFrames(); ++i) {
+        auto FrameInfo = InliningInfo->getFrame(i);
+        SmallString<256> FileName(FrameInfo.FileName);
+        sys::path::remove_dots(FileName, /* remove_dot_dot */ true);
+        uint32_t Line = FrameInfo.Line;
+        std::string FunctionName = FrameInfo.FunctionName;
+        if (ClDemangle)
+          FunctionName = demangle(FrameInfo.FunctionName);
+
+        FileLoc Loc = { FileName.str(), Line };
+        Fns[Loc] = FunctionName;
+      }
+    }
+
+    // Compute file names common prefix.
+    std::string FilePrefix = Fns.begin()->first.FileName;
+    for (const auto &P : Fns)
+      FilePrefix = CommonPrefix(FilePrefix, P.first.FileName);
+
+    // Print first function occurence in a file.
+    {
+      std::string LastFileName;
+      std::set<std::string> ProcessedFunctions;
+
+      for (const auto &P : Fns) {
+        std::string FileName = P.first.FileName;
+        std::string FunctionName = P.second;
+        uint32_t Line = P.first.Line;
+
+        if (LastFileName != FileName)
+          ProcessedFunctions.clear();
+        LastFileName = FileName;
+
+        if (ProcessedFunctions.find(FunctionName) != ProcessedFunctions.end())
+          continue;
+        ProcessedFunctions.insert(FunctionName);
+
+        out << FileName.substr(FilePrefix.size()) << ":" << Line << " "
+            << FunctionName << "\n";
+      }
+    }
+  }
+
+ private:
+  explicit CoverageData(std::unique_ptr<std::vector<uint64_t>> Addrs)
+      : Addrs(std::move(Addrs)) {}
+
+  std::unique_ptr<std::vector<uint64_t>> Addrs;
+};
+} // namespace
+
+int main(int argc, char **argv) {
+  // Print stack trace if we signal out.
+  sys::PrintStackTraceOnErrorSignal();
+  PrettyStackTraceProgram X(argc, argv);
+  llvm_shutdown_obj Y; // Call llvm_shutdown() on exit.
+
+  cl::ParseCommandLineOptions(argc, argv, "Sanitizer Coverage Processing Tool");
+
+  auto CovData = CoverageData::readAndMerge(ClInputFiles);
+  FailIfError(CovData);
+
+  switch (Action) {
+  case PrintAction: {
+    CovData.get()->printAddrs(outs());
+    return 0;
+  }
+  case CoveredFunctionsAction: {
+    CovData.get()->printCoveredFunctions(outs());
+    return 0;
+  }
+  }
+}