Page MenuHomePhabricator

[clangd] Collect tokens of main files when building the AST
ClosedPublic

Authored by ilya-biryukov on Jun 6 2019, 7:25 AM.

Details

Summary

The first use of this is a code tweak to expand macro calls.
Will later be used to build syntax trees.

The memory overhead is small as we only store tokens of the main file.

Diff Detail

Repository
rL LLVM

Event Timeline

ilya-biryukov created this revision.Jun 6 2019, 7:25 AM
Herald added a project: Restricted Project. · View Herald TranscriptJun 6 2019, 7:25 AM
ilya-biryukov edited the summary of this revision. (Show Details)Jun 6 2019, 7:30 AM
sammccall accepted this revision.Jun 18 2019, 1:56 AM

Can we add a test using TestTU that does a very basic verification of expanded/spelled tokens (first after preamble, last token in file)?

clang-tools-extra/clangd/ClangdUnit.h
140 ↗(On Diff #203357)

Not sure "expanded" is accurate here - it has both.
(And it'd be worth spelling out if expanded tokens or spelled tokens or both are missing for the preamble)

This revision is now accepted and ready to land.Jun 18 2019, 1:56 AM
ilya-biryukov marked an inline comment as done.
  • Rename tokens() to getTokens() for consistency with the rest of ParsedAST.
  • Update comments.
  • Add a test.

Can we add a test using TestTU that does a very basic verification of expanded/spelled tokens (first after preamble, last token in file)?

Done. Will land this tomorrow, want to run it over all of LLVM first to give the token-building code some coverage.

This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptJun 19 2019, 7:01 AM