Index: docs/SourceBasedCodeCoverage.rst =================================================================== --- /dev/null +++ docs/SourceBasedCodeCoverage.rst @@ -0,0 +1,174 @@ +========================== +Source-based Code Coverage +========================== + +.. contents:: + :local: + +Introduction +============ + +This document explains how to use clang's source-based code coverage feature. +It's called "source-based" because it operates on AST and preprocessor +information directly. This allows it to generate very precise coverage data. + +Clang ships two other code coverage implementations: + +* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the + various sanitizers. It can provide up to edge-level coverage. + +* gcov - A GCC-compatible coverage implementation which operates on DebugInfo. + +From this point onwards "code coverage" will refer to the source-based kind. + +The code coverage workflow +========================== + +The code coverage workflow consists of three main steps: + +1. Compile with coverage enabled. + +2. Run the program. + +3. Create a report out of the generated profile. + +The next few sections work through a complete, copy-'n-paste friendly example +based on this program: + +.. code-block:: console + + % cat < foo.cc + #define BAR(x) ((x) || (x)) + template void foo(T x) { + for (unsigned I = 0; I < 10; ++I) { BAR(I); } + } + int main() { + foo(0); + foo(0); + return 0; + } + EOF + +Compiling with coverage enabled +=============================== + +To compile code with coverage enabled pass ``-fprofile-instr-generate +-fcoverage-mapping`` to the compiler: + +.. code-block:: console + + # Step 1: Compile with coverage enabled. + % cc -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo + +Note that linking together code with and without coverage instrumentation is +supported: any uninstrumented code simply won't be accounted for. + +Running the instrumented program +================================ + +The next step is to run the instrumented program. When the program exits it +will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE`` +environment variable. If that variable does not exist the profile is written to +``./default.profraw``. + +If ``LLVM_PROFILE_FILE`` contains a path to a non-existent directory the +missing directory structure will be created. Additionally, the following +special **pattern strings** are replaced: + +* "%p" expands out to the PID. + +* "%h" expands out to the hostname of the machine running the program. + +.. code-block:: console + + # Step 2: Run the program. + % LLVM_PROFILE_FILE="foo.profraw" ./foo + +Creating coverage reports +========================= + +Raw profiles have to be **indexed** before they can be used to generated +coverage reports: + +.. code-block:: console + + # Step 3(a): Index the raw profile. + % llvm-profdata merge -sparse foo.profraw -o foo.profdata + +There are multiple different ways to render coverage reports. One option is to +generate a line-oriented report: + +.. code-block:: console + + # Step 3(b): Create a line-oriented coverage report. + % llvm-cov show ./foo -instr-profile=foo.profdata + +This report includes a summary view as well as dedicated sub-views for +templated functions and their instantiations. For our example program, we get +distinct views for ``foo(...)`` and ``foo(...)``. If +``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays nested +region counts (even in macro expansions): + +.. code-block:: console + + 20| 1|#define BAR(x) ((x) || (x)) + ^20 ^2 + 2| 2|template void foo(T x) { + 22| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); } + ^22 ^20 ^20^20 + 1| 4|} + ------------------ + | _Z3fooIiEvT_: + | 1| 2|template void foo(T x) { + | 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); } + | ^11 ^10 ^10^10 + | 1| 4|} + ------------------ + | _Z3fooIfEvT_: + | 1| 2|template void foo(T x) { + | 11| 3| for (unsigned I = 0; I < 10; ++I) { BAR(I); } + | ^11 ^10 ^10^10 + | 1| 4|} + ------------------ + +It's possible to generate a file-level summary of coverage statistics (instead +of a line-oriented report) with: + +.. code-block:: console + + # Step 3(c): Create a coverage summary. + % llvm-cov report ./foo -instr-profile=foo.profdata + +A few final notes: + +* The ``-sparse`` flag is optional but can result in dramatically smaller + indexed profiles. This option should not be used if the indexed profile will + be reused for PGO. + +* Raw profiles can be discarded after they are indexed. It **is** possible for + instrumented programs to merge profiling information directly into an existing + raw profile on disk. The details are out of this document's scope. + +* The ``llvm-profdata`` tool can be used to merge together multiple raw or + indexed profiles. To combine profiling data from multiple runs of a program, + try e.g: + +.. code-block:: console + + % llvm-profdata merge -sparse sort1.profraw sort2.profdata -o sort.profdata + +Format compatibility guarantees +=============================== + +* There are no backwards or forwards compatibility guarantees whatsoever for + the raw profile format. Raw profiles may be dependent on the specific + compiler revision used to generate them. It's inadvisable to store raw + profiles for long periods of time. + +* Tools must retain backwards compatibility with indexed profile formats. These + formats are not forwards-compatible: i.e, a tool which uses format version X + will not be able to understand format version (X+k). + +* There is a third format in play: the format of the coverage mappings emitted + into instrumented binaries. Tools must retain backwards compatibility with + these formats. These formats are not forwards-compatible.