[Lex] Handle repl_input_end in Preprocessor::LexTokensUntilEOF()

[Hahnfeld]

Summary

This fixes many unit tests when trying to enable IncrementalExtensions by default for testing purposes.

[Hahnfeld]

ping :)

[v.g.vassilev]

Looks reasonable to me.

[Hahnfeld]

ping!

[v.g.vassilev]

@Hahnfeld, let's move forward and rely on a post-commit review here.

[Hahnfeld]

This revision cannot land without https://reviews.llvm.org/D158413 and for that one I really want to have feedback from somebody working in that area...

[aaron.ballman]

The idea LGTM when D158413 lands, even if that lands with a slightly different design. (IOW, once D158413 lands, feel free to push this change without further review unless you would like more feedback.)

[Hahnfeld]

Rebase on D158413

clang/lib/Lex/Preprocessor.cpp

if (OnToken)
       OnToken(Result);
   }
 }
 
 void Preprocessor::LexTokensUntilEOF(std::vector<Token> *Tokens) {
   while (1) {
     Token Tok;
     Lex(Tok);
-    if (Tok.isOneOf(tok::unknown, tok::eof, tok::eod))
+    if (Tok.isOneOf(tok::unknown, tok::eof, tok::eod,
+                    tok::annot_repl_input_end))
       break;
     if (Tokens != nullptr)
       Tokens->push_back(Tok);
   }
 }
 
 /// Lex a header-name token (including one formed from header-name-tokens if
 /// \p AllowConcatenation is \c true).