[CMake] Add targets for generating coverage reports

[beanz]

Summary

This is a pretty small bit of CMake goop to generate code coverage
reports. I always forget the right script invocation and end up
fumbling around too much.

Wouldn't it be great to have targets that "Just Work"?

Well, I thought so.

At present this only really works correctly for LLVM, but I'll extend
it in subsequent patches to work for subprojects.

[compnerd]

llvm/cmake/modules/CoverageReport.cmake

3

LLVM tends to use if(...) as the style, please uniformly use that across the file.

8

Do you really expect users to provide an alternate implementation of this?

[beanz]

llvm/cmake/modules/CoverageReport.cmake

8

Probably not. I was thinking about some changes, but I suspect it is probably easier to modify the existing tool than make a replacement. I can make this a hard-coded value.

[beanz]

Updates based on feedback from @compnerd

[phosek]

LGTM

llvm/cmake/modules/CoverageReport.cmake

19

Have you considered using SEND_ERROR? I'm worried that people might miss the warning and then become surprised when they don't get coverage so I'd slightly prefer error over warning.

llvm/docs/CMake.rst

322

327

[beanz]

llvm/cmake/modules/CoverageReport.cmake

19

LLVM_BUILD_INSTRUMENTED_COVERAGE works with gcc and gcov, but the python script in our repo doesn't. So I wanted to only generate these targets if you have llvm-cov and llvm-profdata, but I don't want it to be fatal if you don't.

I felt like a warning would get a little more attention than a note or generic message if something went wrong, but not fail the build configuration.

Does that make sense?

[beanz]

I can't spell... Thanks @phosek!

[phosek]

llvm/cmake/modules/CoverageReport.cmake

19

That makes sense, I wasn't aware that gcc and gcov is also supported.

llvm/CMakeLists.txt

if(LLVM_LIBS)
     endforeach()
   endif()
 endif()
 
 # This must be at the end of the LLVM root CMakeLists file because it must run
 # after all targets are created.
 llvm_distribution_add_targets()
 process_llvm_pass_plugins(GEN_CONFIG)
+include(CoverageReport)
 
 # This allows us to deploy the Universal CRT DLLs by passing -DCMAKE_INSTALL_UCRT_LIBRARIES=ON to CMake
 if (MSVC AND CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows" AND CMAKE_INSTALL_UCRT_LIBRARIES)
   include(InstallRequiredSystemLibraries)
 endif()
 
 if (LLVM_INCLUDE_BENCHMARKS)
   # Override benchmark defaults so that when the library itself is updated these

llvm/cmake/modules/CoverageReport.cmake

+# if coverage reports are not enabled, skip all of this
+if(NOT LLVM_BUILD_INSTRUMENTED_COVERAGE)
+  return()
+endif()
+
+file(TO_NATIVE_PATH
+     "${LLVM_SOURCE_DIR}/utils/prepare-code-coverage-artifact.py"
+     PREPARE_CODE_COV_ARTIFACT)
+
+# llvm-cov and llvm-profdata need to match the host compiler. They can either be
+# explicitly provided by the user, or we will look them up based on the install
+# location of the C++ compiler.
+get_filename_component(COMPILER_DIRECTORY ${CMAKE_CXX_COMPILER} DIRECTORY)
+find_program(LLVM_COV "llvm-cov" ${COMPILER_DIRECTORY} NO_DEFAULT_PATH)
+find_program(LLVM_PROFDATA "llvm-profdata" ${COMPILER_DIRECTORY} NO_DEFAULT_PATH)
+
+if(NOT LLVM_COV OR NOT LLVM_PROFDATA)
+  message(WARNING "Could not find code coverage tools, skipping generating targets. You may explicitly specify LLVM_COV and LLVM_PROFDATA to work around this warning.")
+  return()
+endif()
+
+set(LLVM_CODE_COVERAGE_TARGETS "" CACHE STRING "Targets to run code coverage on (defaults to all exported targets if empty)")
+mark_as_advanced(LLVM_CODE_COVERAGE_TARGETS)
+
+if(NOT LLVM_CODE_COVERAGE_TARGETS)
+  # by default run the coverage report across all the exports provided
+  get_property(COV_TARGETS GLOBAL PROPERTY LLVM_EXPORTS)
+endif()
+
+file(TO_NATIVE_PATH
+     "${CMAKE_BINARY_DIR}/report/"
+     REPORT_DIR)
+
+foreach(target ${LLVM_CODE_COVERAGE_TARGETS} ${COV_TARGETS})
+  get_target_property(target_type ${target} TYPE)
+  if("${target_type}" STREQUAL "SHARED_LIBRARY" OR "${target_type}" STREQUAL "EXECUTABLE")
+    list(APPEND coverage_binaries $<TARGET_FILE:${target}>)
+  endif()
+endforeach()
+
+set(LLVM_COVERAGE_SOURCE_DIRS "" CACHE STRING "Source directories to restrict coverage reports to.")
+mark_as_advanced(LLVM_COVERAGE_SOURCE_DIRS)
+
+foreach(dir ${LLVM_COVERAGE_SOURCE_DIRS})
+  list(APPEND restrict_flags -restrict ${dir})
+endforeach()
+
+# Utility target to clear out profile data.
+# This isn't connected to any dependencies because it is a bit finacky to get
+# working exactly how a user might want.
+add_custom_target(clear-profile-data
+                  COMMAND ${CMAKE_COMMAND} -E
+                          remove_directory ${LLVM_PROFILE_DATA_DIR})
+
+# This currently only works for LLVM, but could be expanded to work for all
+# sub-projects. The current limitation is based on not having a good way to
+# automaticall plumb through the targets that we want to run coverage against.
+add_custom_target(generate-coverage-report
+                  COMMAND ${Python3_EXECUTABLE} ${PREPARE_CODE_COV_ARTIFACT}
+                          ${LLVM_PROFDATA} ${LLVM_COV} ${LLVM_PROFILE_DATA_DIR}
+                          ${REPORT_DIR} ${coverage_binaries}
+                          --unified-report ${restrict_flags}
+                  WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
+                  DEPENDS check-llvm) # Run tests

llvm/docs/CMake.rst

 **LLVM_BUILD_EXAMPLES**:BOOL
   Build LLVM examples. Defaults to OFF. Targets for building each example are
   generated in any case. See documentation for *LLVM_BUILD_TOOLS* above for more
   details.
 
 **LLVM_BUILD_INSTRUMENTED_COVERAGE**:BOOL
   If enabled, `source-based code coverage
   <https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>`_ instrumentation
-  is enabled while building llvm.
+  is enabled while building llvm. If CMake can locate the code coverage
+  scripts and the llvm-cov and llvm-profdata tools that pair to your compiler,
+  the build will also generate the `generate-coverage-report` target to generate
+  the code coverage report for LLVM, and the `clear-profile-data` utility target
+  to delete captured profile data. See documentation for
+  *LLVM_CODE_COVERAGE_TARGETS* and *LLVM_COVERAGE_SOURCE_DIRS* for more
+  information on configuring code coverage reports.
+
+**LLVM_CODE_COVERAGE_TARGETS**:STRING
+  If set to a semicolon separated list of targets, those targets will be used
+  to drive the code coverage reports. If unset, the target list will be
+  constructed using the LLVM build's CMake export list.
+
+**LLVM_COVERAGE_SOURCE_DIRS**:STRING
+  If set to a semicolon separated list of directories, the coverage reports
+  will limit code coverage summaries to just the listed directories. If unset,
+  coverage reports will include all sources identified by the tooling.
 
 **LLVM_BUILD_LLVM_DYLIB**:BOOL
   If enabled, the target for building the libLLVM shared library is added.
   This library contains all of LLVM's components in a single shared library.
   Defaults to OFF. This cannot be used in conjunction with BUILD_SHARED_LIBS.
   Tools will only be linked to the libLLVM shared library if LLVM_LINK_LLVM_DYLIB
   is also ON.
   The components in the library can be customised by setting LLVM_DYLIB_COMPONENTS