This is an archive of the discontinued LLVM Phabricator instance.

[pseudo] Strip comments for TokenStream.
ClosedPublic

Authored by hokein on Mar 7 2022, 1:53 AM.

Details

Summary

Add a utility function to strip comments from a "raw" tokenstream. The
derived stream will be fed to the GLR parser (for early testing).

Diff Detail

Event Timeline

hokein created this revision.Mar 7 2022, 1:53 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 7 2022, 1:53 AM
hokein requested review of this revision.Mar 7 2022, 1:53 AM
Herald added a project: Restricted Project. · View Herald TranscriptMar 7 2022, 1:53 AM
sammccall accepted this revision.Mar 7 2022, 4:55 AM
sammccall added inline comments.
clang/include/clang/Tooling/Syntax/Pseudo/Preprocess.h
144

I think this fits more naturally into Token.h next to cook().

It produces and consumes a token stream, and its implementation is a trivial use of those APIs.

While formally we might see comment stripping as part of the preprocessor, this file is more to do with directive structure. (Maybe it should be renamed?)

clang/unittests/Tooling/Syntax/Pseudo/PreprocessTest.cpp
158

also include a /*comment*/ between int and ;?

This revision is now accepted and ready to land.Mar 7 2022, 4:55 AM
hokein added inline comments.Mar 7 2022, 5:19 AM
clang/include/clang/Tooling/Syntax/Pseudo/Preprocess.h
144

moving to Token.h sounds good to me -- I'd treat this is a trivial implementation (I just wrote it to test the GLR parser). And yeah, stripping comment is just part of preprocessing the code.

Maybe it should be renamed?

The current Preprocess.h file name indicates that everything about preprocessing of TokenStream should live in it.

Since this file defines PPStructure, just call it PPStructure.h/cpp for better fine grained granularity?

hokein updated this revision to Diff 413446.Mar 7 2022, 6:37 AM
hokein marked an inline comment as done.

address comment.

This revision was landed with ongoing or failed builds.Mar 7 2022, 11:24 AM
This revision was automatically updated to reflect the committed changes.