[clang] Add a new annotation token: annot_repl_input_end

[junaire]

Summary

This patch is the first part of the below RFC:
https://discourse.llvm.org/t/rfc-handle-execution-results-in-clang-repl/68493

It adds an annotation token that will replace the original EOF token
when we are in incremental C++ mode. In addition, when we're
parsing an ExprStmt and there's a missing semicolon after the
expression, we set a marker in the annotation token and continue
parsing.

Eventually, we propagate this info in ParseTopLevelStmtDecl and are able
to mark this Decl as something we want to do value printing. Below is a
example:

clang-repl> int x = 42;
clang-repl> x
x is a TopLevelStmtDecl and without a semicolon, we should set
it's IsSemiMissing bit so we can do something interesting in
ASTConsumer::HandleTopLevelDecl.

The idea of the annotation token is proposed by Richard Smith, thanks!

Signed-off-by: Jun Zhang <[e-mail]>

[junaire]

This patch is part of D141215. I split it as some jit people requested. I don't know if there's a good way to test the patch, please leave a comment if you have an idea. Thanks.

clang/lib/CodeGen/CodeGenModule.cpp

7219 ↗

(On Diff #516079)

I'll removed the change after https://reviews.llvm.org/D148435 landed.

[junaire]

.

[rsmith]

This looks like a nice improvement to me.

clang/lib/Interpreter/IncrementalParser.cpp

162

254–270

Not really related to this patch, but in DelayedTemplateParsing mode this code appears to expect two annot_repl_input_end tokens in a row.

clang/lib/Parse/ParseStmt.cpp

547–563

I don't think we need to check whether incremental processing is enabled here -- if not, we shouldn't see the "end of REPL input" token at all.

clang/lib/Parse/Parser.cpp

617–618

Do we need to do this here? IncrementalParser already seems to take care of this, and the logic here would be easier to reason about if Parser never steps past an annot_repl_input_end token, and such tokens instead are only ever consumed by the REPL.

Are there other users of incremental processing mode, other than the REPL / IncrementalParser?

[junaire]

Address comments, thanks!

clang/lib/Interpreter/IncrementalParser.cpp

162

Sorry but that's a private member function so I can't do that. Should I make it public?

clang/lib/Parse/Parser.cpp

617–618

Make sense to me, I'll remove it.

[v.g.vassilev]

clang/include/clang/AST/Decl.h

4345

We should change the accessors to the new field name accordingly.

[v.g.vassilev]

@rsmith, any ideas how to support value printing of constructs such as int i = 12, eg, that's not a top-level statement declaration but worth printing its value? It is the shorthand of int i = 12; i.

clang/lib/CodeGen/CodeGenModule.cpp

7219 ↗

(On Diff #516079)

That can probably go away now.

[junaire]

Update + Rebase

[v.g.vassilev]

LGTM! Thanks for putting so much efforts into this!

[bnbarham]

Are there other users of incremental processing mode, other than the REPL / IncrementalParser?

It seems Swift's clang importer also uses incremental processing mode, I'm assuming to keep the TUScope and CurLexer alive after EOF. We also end up using the same context with various actions, which leads to a few hangs as there's various checks for eof only, eg. ReadPCHAndPreprocessAction, PreprocessOnlyAction, and RewriteIncludesAction. There's also quite a few places in the parser that only check for eof as well (I just grepped for Tok.isNot(tok::eof)).

Should these all be updated to handle annot_repl_input_end or should we instead have a different flag that we set for this purpose instead of co-opting isIncrementalProcessingEnabled?

[v.g.vassilev]

In D148997#4559646, @bnbarham wrote:

Are there other users of incremental processing mode, other than the REPL / IncrementalParser?

It seems Swift's clang importer also uses incremental processing mode, I'm assuming to keep the TUScope and CurLexer alive after EOF. We also end up using the same context with various actions, which leads to a few hangs as there's various checks for eof only, eg. ReadPCHAndPreprocessAction, PreprocessOnlyAction, and RewriteIncludesAction. There's also quite a few places in the parser that only check for eof as well (I just grepped for Tok.isNot(tok::eof)).

Should these all be updated to handle annot_repl_input_end or should we instead have a different flag that we set for this purpose instead of co-opting isIncrementalProcessingEnabled?

I'd prefer to avoid adding a new flag. Is there a way to see how does the diff looks like? Maybe it would make more sense to use the annot_repl_input_end token? If the token name does not capture well the generic use-case I am happy to change it to something better.

[bnbarham]

In D148997#4559788, @v.g.vassilev wrote:

I'd prefer to avoid adding a new flag. Is there a way to see how does the diff looks like?

You mean for a new flag? I don't have one prepared, but it would basically just be adding an extra check where isIncrementalProcessingEnabled is currently used to skip resetting CurLexer and TUScope. I don't believe we'd want any difference in parsing in Swift's clang importer use case.

Maybe it would make more sense to use the annot_repl_input_end token? If the token name does not capture well the generic use-case I am happy to change it to something better.

The issue is that all these actions (and the parser checks) can run with and without isIncrementalProcessingEnabled, so they would need to check both eof and annot_repl_input_end. For some concrete locations (but no where near complete):
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L82
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L955
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/Rewrite/InclusionRewriter.cpp#L542
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Parse/ParseExprCXX.cpp#L4070

[v.g.vassilev]

In D148997#4561015, @bnbarham wrote:

In D148997#4559788, @v.g.vassilev wrote:

I'd prefer to avoid adding a new flag. Is there a way to see how does the diff looks like?

You mean for a new flag? I don't have one prepared, but it would basically just be adding an extra check where isIncrementalProcessingEnabled is currently used to skip resetting CurLexer and TUScope. I don't believe we'd want any difference in parsing in Swift's clang importer use case.

I meant that I'd like to figure out if we could use the annot_repl_input_end before considering a new flag.

Maybe it would make more sense to use the annot_repl_input_end token? If the token name does not capture well the generic use-case I am happy to change it to something better.

The issue is that all these actions (and the parser checks) can run with and without isIncrementalProcessingEnabled, so they would need to check both eof and annot_repl_input_end. For some concrete locations (but no where near complete):
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L82
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L955
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/Rewrite/InclusionRewriter.cpp#L542

These three seem to be useful for clang-repl too, so we might want to extend it with like !(eof || annot_repl_input_end)

https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Parse/ParseExprCXX.cpp#L4070

That looks a bit obscure to me. Looks like we are trying to reach some error recovery anchor but do you happen to have some use case at hand? In principle we could do the same as for the other 3.

[bnbarham]

In D148997#4561022, @v.g.vassilev wrote:

I meant that I'd like to figure out if we could use the annot_repl_input_end before considering a new flag.

Oh sure, that's why I'm here :). Just trying to figure out what we think the best way forward is.

The issue is that all these actions (and the parser checks) can run with and without isIncrementalProcessingEnabled, so they would need to check both eof and annot_repl_input_end. For some concrete locations (but no where near complete):
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L82
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L955
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/Rewrite/InclusionRewriter.cpp#L542

These three seem to be useful for clang-repl too, so we might want to extend it with like !(eof || annot_repl_input_end)

Sure, though to be clear this isn't all of them, there's also the dump actions too (though not sure that makes sense for clang-repl) and a few in other files (VerifyDiagnosticConsumer.cpp, PrintPreprocessedOutput.cpp).

https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Parse/ParseExprCXX.cpp#L4070

That looks a bit obscure to me. Looks like we are trying to reach some error recovery anchor but do you happen to have some use case at hand? In principle we could do the same as for the other 3.

This was just one I picked at random from my grep over the parser. It's unclear how often this would be hit, but I have to assume it (and the other references) can, even if they're obscure/unlikely. My main point is that !eof is being used somewhat frequently to end loops, but with this change they will now all never end.

[v.g.vassilev]

In D148997#4561068, @bnbarham wrote:

In D148997#4561022, @v.g.vassilev wrote:

I meant that I'd like to figure out if we could use the annot_repl_input_end before considering a new flag.

Oh sure, that's why I'm here :). Just trying to figure out what we think the best way forward is.

The issue is that all these actions (and the parser checks) can run with and without isIncrementalProcessingEnabled, so they would need to check both eof and annot_repl_input_end. For some concrete locations (but no where near complete):
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L82
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/FrontendActions.cpp#L955
https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Frontend/Rewrite/InclusionRewriter.cpp#L542

These three seem to be useful for clang-repl too, so we might want to extend it with like !(eof || annot_repl_input_end)

Sure, though to be clear this isn't all of them, there's also the dump actions too (though not sure that makes sense for clang-repl)

The dump actions are also somewhat interesting because in that case we will dump the delta that was accumulated and repeat.

and a few in other files (VerifyDiagnosticConsumer.cpp, PrintPreprocessedOutput.cpp).

These ones are also interesting.

https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Parse/ParseExprCXX.cpp#L4070

That looks a bit obscure to me. Looks like we are trying to reach some error recovery anchor but do you happen to have some use case at hand? In principle we could do the same as for the other 3.

This was just one I picked at random from my grep over the parser. It's unclear how often this would be hit, but I have to assume it (and the other references) can, even if they're obscure/unlikely. My main point is that !eof is being used somewhat frequently to end loops, but with this change they will now all never end.

Since eof there were several eof-like tokens that were added: annot_module_begin, annot_module_end, tok::annot_module_include which are commonly handled in isEofOrEom. We could update these uses with isEofOrEom but I am pretty sure that @rsmith will point out cases where that's not a good very good idea :) We could either experiment with using isEofOrEom or have a similar utility function only for the two tokens (say isEoI -- end of input whatever that means).

[bnbarham]

In D148997#4561211, @v.g.vassilev wrote:

https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Parse/ParseExprCXX.cpp#L4070

That looks a bit obscure to me. Looks like we are trying to reach some error recovery anchor but do you happen to have some use case at hand? In principle we could do the same as for the other 3.

This was just one I picked at random from my grep over the parser. It's unclear how often this would be hit, but I have to assume it (and the other references) can, even if they're obscure/unlikely. My main point is that !eof is being used somewhat frequently to end loops, but with this change they will now all never end.

Since eof there were several eof-like tokens that were added: annot_module_begin, annot_module_end, tok::annot_module_include which are commonly handled in isEofOrEom. We could update these uses with isEofOrEom but I am pretty sure that @rsmith will point out cases where that's not a good very good idea :) We could either experiment with using isEofOrEom or have a similar utility function only for the two tokens (say isEoI -- end of input whatever that means).

My other concern here is that it seems our use cases are somewhat different, eg. we wouldn't want any parsing differences and while I don't know why this is yet, the removal of:

// Skip over the EOF token, flagging end of previous input for incremental
// processing
if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
  ConsumeToken();

is causing issues for us as well.

[v.g.vassilev]

In D148997#4561516, @bnbarham wrote:

In D148997#4561211, @v.g.vassilev wrote:

https://github.com/llvm/llvm-project/blob/df6b35e329ebecad6dc3bfb83183e482eb7a0020/clang/lib/Parse/ParseExprCXX.cpp#L4070

That looks a bit obscure to me. Looks like we are trying to reach some error recovery anchor but do you happen to have some use case at hand? In principle we could do the same as for the other 3.

This was just one I picked at random from my grep over the parser. It's unclear how often this would be hit, but I have to assume it (and the other references) can, even if they're obscure/unlikely. My main point is that !eof is being used somewhat frequently to end loops, but with this change they will now all never end.

Since eof there were several eof-like tokens that were added: annot_module_begin, annot_module_end, tok::annot_module_include which are commonly handled in isEofOrEom. We could update these uses with isEofOrEom but I am pretty sure that @rsmith will point out cases where that's not a good very good idea :) We could either experiment with using isEofOrEom or have a similar utility function only for the two tokens (say isEoI -- end of input whatever that means).

My other concern here is that it seems our use cases are somewhat different, eg. we wouldn't want any parsing differences and while I don't know why this is yet, the removal of:

I think I understand now. We basically want a mode which keeps some of the clang objects alive and ready to process more input. And on top, for clang-repl we want to enable some special parsing mode for the top-level statement declarations.

// Skip over the EOF token, flagging end of previous input for incremental
// processing
if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
  ConsumeToken();

is causing issues for us as well.

Yes, that now seems to be moved on the user side as we thought it would be dead code. We can bring that one back if that's the only issue you see with that approach.

[bnbarham]

In D148997#4561577, @v.g.vassilev wrote:

My other concern here is that it seems our use cases are somewhat different, eg. we wouldn't want any parsing differences and while I don't know why this is yet, the removal of:

I think I understand now. We basically want a mode which keeps some of the clang objects alive and ready to process more input. And on top, for clang-repl we want to enable some special parsing mode for the top-level statement declarations.

Yeah, I'd say that sums it up pretty well.

// Skip over the EOF token, flagging end of previous input for incremental
// processing
if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
  ConsumeToken();

is causing issues for us as well.

Yes, that now seems to be moved on the user side as we thought it would be dead code. We can bring that one back if that's the only issue you see with that approach.

I think it would make sense to bring it back under the mode you spoke about above, yeah 👍

[v.g.vassilev]

In D148997#4561614, @bnbarham wrote:

In D148997#4561577, @v.g.vassilev wrote:

My other concern here is that it seems our use cases are somewhat different, eg. we wouldn't want any parsing differences and while I don't know why this is yet, the removal of:

I think I understand now. We basically want a mode which keeps some of the clang objects alive and ready to process more input. And on top, for clang-repl we want to enable some special parsing mode for the top-level statement declarations.

Yeah, I'd say that sums it up pretty well.

// Skip over the EOF token, flagging end of previous input for incremental
// processing
if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
  ConsumeToken();

is causing issues for us as well.

Yes, that now seems to be moved on the user side as we thought it would be dead code. We can bring that one back if that's the only issue you see with that approach.

I think it would make sense to bring it back under the mode you spoke about above, yeah 👍

So, in that case we should bring back the boolean flag for incremental processing and keep the IncrementalExtensions LanguageOption separate. In that case IncrementalExtensions would mean that we must turn on incremental processing for managing lifetime and only use the language option when extending the parsing logic. However, I think the problem would be what to do with the tok::eof and tok::annot_repl_input_end? I'd probably need @aaron.ballman or @rsmith here...

[bnbarham]

In D148997#4561620, @v.g.vassilev wrote:

So, in that case we should bring back the boolean flag for incremental processing and keep the IncrementalExtensions LanguageOption separate. In that case IncrementalExtensions would mean that we must turn on incremental processing for managing lifetime and only use the language option when extending the parsing logic. However, I think the problem would be what to do with the tok::eof and tok::annot_repl_input_end? I'd probably need @aaron.ballman or @rsmith here...

Would you be happy to make that change, or should I put it up? Separating the options and what to do about the token in general could be separate PRs.

[v.g.vassilev]

In D148997#4607833, @bnbarham wrote:

In D148997#4561620, @v.g.vassilev wrote:

So, in that case we should bring back the boolean flag for incremental processing and keep the IncrementalExtensions LanguageOption separate. In that case IncrementalExtensions would mean that we must turn on incremental processing for managing lifetime and only use the language option when extending the parsing logic. However, I think the problem would be what to do with the tok::eof and tok::annot_repl_input_end? I'd probably need @aaron.ballman or @rsmith here...

Would you be happy to make that change, or should I put it up? Separating the options and what to do about the token in general could be separate PRs.

If you could do it would be better as it would capture better your concrete use case.

[aaron.ballman]

In D148997#4561620, @v.g.vassilev wrote:

In D148997#4561614, @bnbarham wrote:

In D148997#4561577, @v.g.vassilev wrote:

My other concern here is that it seems our use cases are somewhat different, eg. we wouldn't want any parsing differences and while I don't know why this is yet, the removal of:

I think I understand now. We basically want a mode which keeps some of the clang objects alive and ready to process more input. And on top, for clang-repl we want to enable some special parsing mode for the top-level statement declarations.

Yeah, I'd say that sums it up pretty well.

// Skip over the EOF token, flagging end of previous input for incremental
// processing
if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
  ConsumeToken();

is causing issues for us as well.

Yes, that now seems to be moved on the user side as we thought it would be dead code. We can bring that one back if that's the only issue you see with that approach.

I think it would make sense to bring it back under the mode you spoke about above, yeah 👍

So, in that case we should bring back the boolean flag for incremental processing and keep the IncrementalExtensions LanguageOption separate. In that case IncrementalExtensions would mean that we must turn on incremental processing for managing lifetime and only use the language option when extending the parsing logic. However, I think the problem would be what to do with the tok::eof and tok::annot_repl_input_end? I'd probably need @aaron.ballman or @rsmith here...

I missed this ping earlier in the month, sorry for that! My intuition is that this is handled by isEofOrEom() as done in this patch -- if incremental extensions is not enabled, then we wouldn't see an annot_repl_input_end token in the first place, right?

clang/include/clang/AST/Decl.h

 ///
 /// \note This is used in libInterpreter, clang -cc1 -fincremental-extensions
 /// and in tools such as clang-repl.
 class TopLevelStmtDecl : public Decl {
   friend class ASTDeclReader;
   friend class ASTDeclWriter;
 
   Stmt *Statement = nullptr;
+  bool IsSemiMissing = false;
 
   TopLevelStmtDecl(DeclContext *DC, SourceLocation L, Stmt *S)
       : Decl(TopLevelStmt, DC, L), Statement(S) {}
 
   virtual void anchor();
 
 public:
   static TopLevelStmtDecl *Create(ASTContext &C, Stmt *Statement);
   static TopLevelStmtDecl *CreateDeserialized(ASTContext &C, unsigned ID);
 
   SourceRange getSourceRange() const override LLVM_READONLY;
   Stmt *getStmt() { return Statement; }
   const Stmt *getStmt() const { return Statement; }
+  void setStmt(Stmt *S) {
+    assert(IsSemiMissing && "Operation supported for printing values only!");
+    Statement = S;
+  }
+  bool isSemiMissing() const { return IsSemiMissing; }
+  void setSemiMissing(bool Missing = true) { IsSemiMissing = Missing; }
 
   static bool classof(const Decl *D) { return classofKind(D->getKind()); }
   static bool classofKind(Kind K) { return K == TopLevelStmt; }
 };
 
 /// Represents a block literal declaration, which is like an
 /// unnamed FunctionDecl.  For example:
 /// ^{ statement-body }   or   ^(int arg1, float arg2){ statement-body }

clang/include/clang/Basic/TokenKinds.def

 ANNOTATION(module_include)
 ANNOTATION(module_begin)
 ANNOTATION(module_end)
 
 // Annotation for a header_name token that has been looked up and transformed
 // into the name of a header unit.
 ANNOTATION(header_unit)
 
+// Annotation for end of input in clang-repl.
+ANNOTATION(repl_input_end)
+
 #undef PRAGMA_ANNOTATION
 #undef ANNOTATION
 #undef TESTING_KEYWORD
 #undef OBJC_AT_KEYWORD
 #undef CXX_KEYWORD_OPERATOR
 #undef PPKEYWORD
 #undef ALIAS
 #undef EXPRESSION_TRAIT

clang/include/clang/Parse/Parser.h

 #ifndef LLVM_CLANG_PARSE_PARSER_H
 #define LLVM_CLANG_PARSE_PARSER_H
 
 #include "clang/AST/Availability.h"
 #include "clang/Basic/BitmaskEnum.h"
 #include "clang/Basic/OpenMPKinds.h"
 #include "clang/Basic/OperatorPrecedence.h"
 #include "clang/Basic/Specifiers.h"
+#include "clang/Basic/TokenKinds.h"
 #include "clang/Lex/CodeCompletionHandler.h"
 #include "clang/Lex/Preprocessor.h"
 #include "clang/Sema/DeclSpec.h"
 #include "clang/Sema/Sema.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Frontend/OpenMP/OMPContext.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/PrettyStackTrace.h"
void cutOffParsing() {
     Tok.setKind(tok::eof);
   }
 
   /// Determine if we're at the end of the file or at a transition
   /// between modules.
   bool isEofOrEom() {
     tok::TokenKind Kind = Tok.getKind();
     return Kind == tok::eof || Kind == tok::annot_module_begin ||
-           Kind == tok::annot_module_end || Kind == tok::annot_module_include;
+           Kind == tok::annot_module_end || Kind == tok::annot_module_include ||
+           Kind == tok::annot_repl_input_end;
   }
 
   /// Checks if the \p Level is valid for use in a fold expression.
   bool isFoldOperator(prec::Level Level) const;
 
   /// Checks if the \p Kind is a valid operator for fold expressions.
   bool isFoldOperator(tok::TokenKind Kind) const;
 

clang/lib/Frontend/PrintPreprocessedOutput.cpp

 
 void PrintPPOutputPPCallbacks::HandleWhitespaceBeforeTok(const Token &Tok,
                                                          bool RequireSpace,
                                                          bool RequireSameLine) {
   // These tokens are not expanded to anything and don't need whitespace before
   // them.
   if (Tok.is(tok::eof) ||
       (Tok.isAnnotation() && !Tok.is(tok::annot_header_unit) &&
-       !Tok.is(tok::annot_module_begin) && !Tok.is(tok::annot_module_end)))
+       !Tok.is(tok::annot_module_begin) && !Tok.is(tok::annot_module_end) &&
+       !Tok.is(tok::annot_repl_input_end)))
     return;
 
   // EmittedDirectiveOnThisLine takes priority over RequireSameLine.
   if ((!RequireSameLine || EmittedDirectiveOnThisLine) &&
       MoveToLine(Tok, /*RequireStartOfLine=*/EmittedDirectiveOnThisLine)) {
     if (MinimizeWhitespace) {
       // Avoid interpreting hash as a directive under -fpreprocessed.
       if (Tok.is(tok::hash))
Callbacks->HandleWhitespaceBeforeTok(Tok, /*RequireSpace=*/false,
                                          /*RequireSameLine=*/!IsStartOfLine);
 
     if (DropComments && Tok.is(tok::comment)) {
       // Skip comments. Normally the preprocessor does not generate
       // tok::comment nodes at all when not keeping comments, but under
       // -traditional-cpp the lexer keeps /all/ whitespace, including comments.
       PP.Lex(Tok);
       continue;
+    } else if (Tok.is(tok::annot_repl_input_end)) {
+      PP.Lex(Tok);
+      continue;
     } else if (Tok.is(tok::eod)) {
       // Don't print end of directive tokens, since they are typically newlines
       // that mess up our line tracking. These come from unknown pre-processor
       // directives or hash-prefixed comments in standalone assembly files.
       PP.Lex(Tok);
       // FIXME: The token on the next line after #include should have
       // Tok.isAtStartOfLine() set.
       IsStartOfLine = true;

clang/lib/Interpreter/IncrementalParser.cpp

IncrementalParser::ParseOrWrapTopLevelDecl() {
   PTUs.emplace_back(PartialTranslationUnit());
   PartialTranslationUnit &LastPTU = PTUs.back();
   // Add a new PTU.
   ASTContext &C = S.getASTContext();
   C.addTranslationUnitDecl();
   LastPTU.TUPart = C.getTranslationUnitDecl();
 
   // Skip previous eof due to last incremental input.
-  if (P->getCurToken().is(tok::eof)) {
-    P->ConsumeToken();
+  if (P->getCurToken().is(tok::annot_repl_input_end)) {
+    P->ConsumeAnyToken();
     // FIXME: Clang does not call ExitScope on finalizing the regular TU, we
     // might want to do that around HandleEndOfTranslationUnit.
     P->ExitScope();
     S.CurContext = nullptr;
     // Start a new PTU.
     P->EnterScope(Scope::DeclScope);
     S.ActOnTranslationUnitScope(P->getCurScope());
   }
if (PP.EnterSourceFile(FID, /*DirLookup=*/nullptr, NewLoc))
     return llvm::make_error<llvm::StringError>("Parsing failed. "
                                                "Cannot enter source file.",
                                                std::error_code());
 
   auto PTU = ParseOrWrapTopLevelDecl();
   if (!PTU)
     return PTU.takeError();
 
   if (PP.getLangOpts().DelayedTemplateParsing) {
     // Microsoft-specific:
     // Late parsed templates can leave unswallowed "macro"-like tokens.
     // They will seriously confuse the Parser when entering the next
     // source file. So lex until we are EOF.
     Token Tok;
     do {
       PP.Lex(Tok);
-    } while (Tok.isNot(tok::eof));
-  }
-
+    } while (Tok.isNot(tok::annot_repl_input_end));
+  } else {
     Token AssertTok;
     PP.Lex(AssertTok);
-  assert(AssertTok.is(tok::eof) &&
+    assert(AssertTok.is(tok::annot_repl_input_end) &&
            "Lexer must be EOF when starting incremental parse!");
+  }
 
   if (CodeGenerator *CG = getCodeGen(Act.get())) {
     std::unique_ptr<llvm::Module> M(CG->ReleaseModule());
     CG->StartModule("incr_module_" + std::to_string(PTUs.size()),
                     M->getContext());
 
     PTU->TheModule = std::move(M);
   }
 
   return PTU;

clang/lib/Lex/PPLexerChange.cpp

if (!isEndOfMacro && CurPPLexer && FoundPCHThroughHeader &&
       // Leaving the through header. Continue directly to end of main file
       // processing.
       LeavingPCHThroughHeader = true;
     } else {
       // Client should lex another token unless we generated an EOM.
       return LeavingSubmodule;
     }
   }
-
   // If this is the end of the main file, form an EOF token.
   assert(CurLexer && "Got EOF but no current lexer set!");
   const char *EndPos = getCurLexerEndPos();
   Result.startToken();
   CurLexer->BufferPtr = EndPos;
+
+  if (isIncrementalProcessingEnabled()) {
+    CurLexer->FormTokenWithChars(Result, EndPos, tok::annot_repl_input_end);
+    Result.setAnnotationEndLoc(Result.getLocation());
+    Result.setAnnotationValue(nullptr);
+  } else {
     CurLexer->FormTokenWithChars(Result, EndPos, tok::eof);
+  }
 
   if (isCodeCompletionEnabled()) {
     // Inserting the code-completion point increases the source buffer by 1,
     // but the main FileID was created before inserting the point.
     // Compensate by reducing the EOF location by 1, otherwise the location
     // will point to the next FileID.
     // FIXME: This is hacky, the code-completion point should probably be
     // inserted before the main FileID is created.

clang/lib/Parse/ParseCXXInlineMethods.cpp

if (Tok.is(T1) || Tok.is(T2)) {
       return true;
     }
 
     switch (Tok.getKind()) {
     case tok::eof:
     case tok::annot_module_begin:
     case tok::annot_module_end:
     case tok::annot_module_include:
+    case tok::annot_repl_input_end:
       // Ran out of tokens.
       return false;
 
     case tok::l_paren:
       // Recursively consume properly-nested parens.
       Toks.push_back(Tok);
       ConsumeParen();
       ConsumeAndStoreUntil(tok::r_paren, Toks, /*StopAtSemi=*/false);
case tok::comma:
       // Keep going. We know we're inside a template argument list now.
       ++KnownTemplateCount;
       goto consume_token;
 
     case tok::eof:
     case tok::annot_module_begin:
     case tok::annot_module_end:
     case tok::annot_module_include:
+    case tok::annot_repl_input_end:
       // Ran out of tokens.
       return false;
 
     case tok::less:
       // FIXME: A '<' can only start a template-id if it's preceded by an
       // identifier, an operator-function-id, or a literal-operator-id.
       ++AngleCount;
       goto consume_token;

clang/lib/Parse/ParseDecl.cpp

case tok::plus:
       if (Tok.isAtStartOfLine() && ParsingInObjCContainer)
         return;
       break;
 
     case tok::eof:
     case tok::annot_module_begin:
     case tok::annot_module_end:
     case tok::annot_module_include:
+    case tok::annot_repl_input_end:
       return;
 
     default:
       break;
     }
 
     ConsumeAnyToken();
   }
Parser::DeclGroupPtrTy Parser::ParseTopLevelStmtDecl() {
   Actions.PushFunctionScope();
   StmtResult R = ParseStatementOrDeclaration(Stmts, SubStmtCtx);
   Actions.PopFunctionScopeInfo();
   if (!R.isUsable())
     return nullptr;
 
   SmallVector<Decl *, 2> DeclsInGroup;
   DeclsInGroup.push_back(Actions.ActOnTopLevelStmtDecl(R.get()));
+
+  if (Tok.is(tok::annot_repl_input_end) &&
+      Tok.getAnnotationValue() != nullptr) {
+    ConsumeAnnotationToken();
+    cast<TopLevelStmtDecl>(DeclsInGroup.back())->setSemiMissing();
+  }
+
   // Currently happens for things like  -fms-extensions and use `__if_exists`.
   for (Stmt *S : Stmts)
     DeclsInGroup.push_back(Actions.ActOnTopLevelStmtDecl(S));
 
   return Actions.BuildDeclaratorGroup(DeclsInGroup);
 }
 
 /// isDeclarationSpecifier() - Return true if the current token is part of a

clang/lib/Parse/ParseStmt.cpp

if (Tok.is(tok::colon) && getCurScope()->isSwitchScope() &&
     // suggest a missing case keyword.
     Diag(OldToken, diag::err_expected_case_before_expression)
       << FixItHint::CreateInsertion(OldToken.getLocation(), "case ");
 
     // Recover parsing as a case statement.
     return ParseCaseStatement(StmtCtx, /*MissingCase=*/true, Expr);
   }
 
+  Token *CurTok = nullptr;
+  // If the semicolon is missing at the end of REPL input, consider if
+  // we want to do value printing. Note this is only enabled in C++ mode
+  // since part of the implementation requires C++ language features.
+  // Note we shouldn't eat the token since the callback needs it.
+  if (Tok.is(tok::annot_repl_input_end) && Actions.getLangOpts().CPlusPlus)
+    CurTok = &Tok;
+  else
     // Otherwise, eat the semicolon.
     ExpectAndConsumeSemi(diag::err_expected_semi_after_expr);
-  return handleExprStmt(Expr, StmtCtx);
+
+  StmtResult R = handleExprStmt(Expr, StmtCtx);
+  if (CurTok && !R.isInvalid())
+    CurTok->setAnnotationValue(R.get());
+
+  return R;
 }
 
 /// ParseSEHTryBlockCommon
 ///
 /// seh-try-block:
 ///   '__try' compound-statement seh-handler
 ///
 /// seh-handler:
 ///   seh-except-block

clang/lib/Parse/Parser.cpp

case tok::annot_pragma_openmp_end:
       // Stop before an OpenMP pragma boundary.
       if (OpenMPDirectiveParsing)
         return false;
       ConsumeAnnotationToken();
       break;
     case tok::annot_module_begin:
     case tok::annot_module_end:
     case tok::annot_module_include:
+    case tok::annot_repl_input_end:
       // Stop before we change submodules. They generally indicate a "good"
       // place to pick up parsing again (except in the special case where
       // we're trying to skip to EOF).
       return false;
 
     case tok::code_completion:
       if (!HasFlagsSet(Flags, StopAtCodeCompletion))
         handleUnexpectedCodeCompletionToken();

 /// action tells us to.  This returns true if the EOF was encountered.
 ///
 ///   top-level-declaration:
 ///           declaration
 /// [C++20]   module-import-declaration
 bool Parser::ParseTopLevelDecl(DeclGroupPtrTy &Result,
                                Sema::ModuleImportState &ImportState) {
   DestroyTemplateIdAnnotationsRAIIObj CleanupRAII(*this);
 
-  // Skip over the EOF token, flagging end of previous input for incremental
-  // processing
-  if (PP.isIncrementalProcessingEnabled() && Tok.is(tok::eof))
-    ConsumeToken();
-
   Result = nullptr;
   switch (Tok.getKind()) {
   case tok::annot_pragma_unused:
     HandlePragmaUnused();
     return false;
 
   case tok::kw_export:
     switch (NextToken().getKind()) {
     case tok::kw_module:
bool Parser::ParseTopLevelDecl(DeclGroupPtrTy &Result,
   case tok::annot_module_end:
     Actions.ActOnModuleEnd(Tok.getLocation(), reinterpret_cast<Module *>(
                                                   Tok.getAnnotationValue()));
     ConsumeAnnotationToken();
     ImportState = Sema::ModuleImportState::NotACXX20Module;
     return false;
 
   case tok::eof:
+  case tok::annot_repl_input_end:
     // Check whether -fmax-tokens= was reached.
     if (PP.getMaxTokens() != 0 && PP.getTokenCount() > PP.getMaxTokens()) {
       PP.Diag(Tok.getLocation(), diag::warn_max_tokens_total)
           << PP.getTokenCount() << PP.getMaxTokens();
       SourceLocation OverrideLoc = PP.getMaxTokensOverrideLoc();
       if (OverrideLoc.isValid()) {
         PP.Diag(OverrideLoc, diag::note_max_tokens_total_override);
       }