This patch is a prototype implementation of the clang-scan-deps tool. This tool computes the set of file dependencies for a particular compiler invocation using some optimizations that are outlined below. This tool makes the non-modular dependency scanning up to 10 times faster for particular workloads (e.g. llc target, 1542 C++ files) on one of our machines, when compared to parallel invocations of clang with -Eonly. We are still in the early stages of proper modules support, but our initial crude prototype can get up to 4x when run on the first 1000 files from clang’s compilation database for a build of LLVM with modules turned on.
We run the full Clang preprocessor. Here’s what we do to reduce its workload:
- Minimize sources by stripping away unused tokens. We keep only the interesting PP directives (#define, #if, #include, etc.), i.e. those that might impact the set of dependencies. The patch includes an additional standalone clang-filter-to-includes tool that can be used to perform this minimization.
- Assume the filesystem is immutable for one run of the service, and cache the files and their minimized contents in memory in a global cache.
- Skip over excluded preprocessor ranges by bumping up the buffer pointer in the lexer instead of lexing the skipped tokens.
Right now we still haven't integrated the explicit module experiments with this tool, and at the moment we are using the tool to:
- benchmark the performance of the fast scanner.
- compare the accuracy to the regular -Eonly invocation.
In the future it might be interesting to extend this tool to provide an interface that the build system can use for the pre scanning phase for the explicit modules.