Page MenuHomePhabricator

[clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization

Authored by arphaman on Sep 3 2019, 2:22 PM.



This patch adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive. The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens.

On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on the compilation database for a recent LLVM and Clang (3511 files).

Diff Detail


Event Timeline

arphaman created this revision.Sep 3 2019, 2:22 PM
Herald added a project: Restricted Project. · View Herald TranscriptSep 3 2019, 2:22 PM
Bigcheese added inline comments.Sep 3 2019, 5:19 PM
256 ↗(On Diff #218531)

This kinda sounds like it can be called multiple times. Is there any way you can use chained pp callbacks?

171–177 ↗(On Diff #218531)

Is there any reason for this to be configurable other than for perf testing?

arphaman updated this revision to Diff 218778.Sep 4 2019, 1:11 PM
arphaman marked 2 inline comments as done.

Add support for chained PP callbacks in setAdditionalPPCallbacks.

256 ↗(On Diff #218531)

Good idea, I implemented support for adding multiple ones with chained callbacks.

171–177 ↗(On Diff #218531)

It's just for perf-testing / verification. No other reason.

Looking more into this I'm not sure you need the PPCallbacks. The Preprocessor should just own the PreprocessorSkippedMappings data structure and the dependency scanner can update it live.

arphaman updated this revision to Diff 218821.Sep 4 2019, 5:54 PM

restructure the code to perform skipping directly in the preprocessor

Bigcheese accepted this revision.Sep 11 2019, 1:04 PM
This revision is now accepted and ready to land.Sep 11 2019, 1:04 PM
This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptSep 11 2019, 1:41 PM