[pseudo] Strip comments for TokenStream.

[hokein]

Summary

Add a utility function to strip comments from a "raw" tokenstream. The
derived stream will be fed to the GLR parser (for early testing).

[sammccall]

clang/include/clang/Tooling/Syntax/Pseudo/Preprocess.h

144 ↗

(On Diff #413382)

I think this fits more naturally into Token.h next to cook().

It produces and consumes a token stream, and its implementation is a trivial use of those APIs.

While formally we might see comment stripping as part of the preprocessor, this file is more to do with directive structure. (Maybe it should be renamed?)

clang/unittests/Tooling/Syntax/Pseudo/PreprocessTest.cpp

158 ↗

(On Diff #413382)

also include a /*comment*/ between int and ;?

[hokein]

clang/include/clang/Tooling/Syntax/Pseudo/Preprocess.h

144 ↗

(On Diff #413382)

moving to Token.h sounds good to me -- I'd treat this is a trivial implementation (I just wrote it to test the GLR parser). And yeah, stripping comment is just part of preprocessing the code.

Maybe it should be renamed?

The current Preprocess.h file name indicates that everything about preprocessing of TokenStream should live in it.

Since this file defines PPStructure, just call it PPStructure.h/cpp for better fine grained granularity?

[hokein]

address comment.

clang/include/clang/Tooling/Syntax/Pseudo/Token.h

 /// The StartsPPLine flag is preserved.
 ///
 /// Formally the identifier correctly happens before preprocessing, while we
 /// should only cook raw_identifiers that survive preprocessing.
 /// However, ignoring the Token::Kind of tokens in directives achieves the same.
 /// (And having cooked token kinds in PP-disabled sections is useful for us).
 TokenStream cook(const TokenStream &, const clang::LangOptions &);
 
+/// Drops comment tokens.
+TokenStream stripComments(const TokenStream &);
+
 } // namespace pseudo
 } // namespace syntax
 } // namespace clang
 
 #endif

clang/lib/Tooling/Syntax/Pseudo/Token.cpp

for (const auto &T : tokens()) {
     OS << Text;
     LastLine = T.Line;
     LastText = Text;
   }
   if (!FirstToken)
     OS << '\n';
 }
 
+TokenStream stripComments(const TokenStream &Input) {
+  TokenStream Out;
+  for (const Token &T : Input.tokens()) {
+    if (T.Kind == tok::comment)
+      continue;
+    Out.push(T);
+  }
+  Out.finalize();
+  return Out;
+}
+
 } // namespace pseudo
 } // namespace syntax
 } // namespace clang

clang/unittests/Tooling/Syntax/Pseudo/TokenTest.cpp

)cpp";
   EXPECT_THAT(Raw.tokens(), ElementsAreArray({
                                 lineIndent(0, 3), // hello
                                 lineIndent(0, 3), // world
                                 lineIndent(1, 0), // no_indent
                                 lineIndent(2, 2), // line_was_continued
                             }));
 }
 
+TEST(TokenTest, DropComments) {
+  LangOptions Opts;
+  std::string Code = R"cpp(
+  // comment
+  int /*abc*/;
+)cpp";
+  TokenStream Raw = cook(lex(Code, Opts), Opts);
+  TokenStream Stripped = stripComments(Raw);
+  EXPECT_THAT(Raw.tokens(),
+              ElementsAreArray(
+                  {token("// comment", tok::comment), token("int", tok::kw_int),
+                   token("/*abc*/", tok::comment), token(";", tok::semi)}));
+
+  EXPECT_THAT(Stripped.tokens(), ElementsAreArray({token("int", tok::kw_int),
+                                                   token(";", tok::semi)}));
+}
+
 } // namespace
 } // namespace pseudo
 } // namespace syntax
 } // namespace clang