Page MenuHomePhabricator

[Preprocessor] Fix incorrect token caching that occurs when lexing _Pragma in macro argument pre-expansion mode when skipping a function body

Authored by arphaman on Jan 16 2017, 7:29 AM.



This patch fixes a token caching problem that currently occurs when clang is skipping a function body (e.g. when looking for a code completion token) and at the same time caching the tokens for _Pragma when lexing it in macro argument pre-expansion mode.

When _Pragma is being lexed in macro argument pre-expansion mode, it caches the tokens so that it can avoid interpreting the pragma immediately (as the macro argument may not be used in the macro body), and then either backtracks over or commits these tokens. The problem is that, when we're backtracking/committing in such a scenario, there's already a previous backtracking position stored in BacktrackPositions (as we're skipping the function body), and this leads to a situation where the cached tokens from the pragma (like '(' 'string_literal' and ')') will remain in the cached tokens array incorrectly even after they're consumed (in the case of backtracking) or just ignored (in the case when they're committed). Furthermore, what makes it even worse, is that because of a previous backtracking position, the logic that deals with when should we call ExitCachingLexMode in CachingLex no longer works for us in this situation, and more tokens in the macro argument get cached, to the point where the EOF token that corresponds to the macro argument EOF is cached. This problem leads to all sorts of issues in code completion mode, where incorrect errors get presented and code completion completely fails to produce completion results.

Thanks for taking a look

Diff Detail


Event Timeline

arphaman updated this revision to Diff 84561.Jan 16 2017, 7:29 AM
arphaman retitled this revision from to [Preprocessor] Fix incorrect token caching that occurs when lexing _Pragma in macro argument pre-expansion mode when skipping a function body.
arphaman updated this object.
arphaman added reviewers: bruno, rsmith, akyrtzi.
arphaman set the repository for this revision to rL LLVM.
arphaman added a subscriber: cfe-commits.
rsmith edited edge metadata.Jan 23 2017, 3:42 PM

Can we instead address this locally in _Pragma handling, by getting it to clear out the junk it inserted into the token stream when it's done (if backtracking is enabled)?

arphaman updated this revision to Diff 87801.Feb 9 2017, 7:00 AM

Sorry about the delay.
As per Richard's suggestion, the updated patch now makes the _Pragma parser responsible for initiating the removal of cached tokens.

akyrtzi accepted this revision.Feb 23 2017, 5:22 PM


This revision is now accepted and ready to land.Feb 23 2017, 5:22 PM
This revision was automatically updated to reflect the committed changes.