Change Details

Add TryGrowLexerBuffer to Lexer and SourceFileGrower to Preprocessor that can be used to grow the source code buffer when Lexer reaches the eof of the buffer. Since clang-repl receive source code incrementally line by line, we need to grow source code buffer. This change adds the interfaces to Lexer/Preprocessor in order to accommodate this need. If SourceFileGrower interface is registered to Preprocessor, it will call TryGrowFile method when Lexer reached eof where the user can grow the file and return true to request Lexer to continue the lexing from the last point. When Lexer reaches eof it will call TryGrowLexerBuffer callback specified in the constructor to try growing buffer and if it got new buffer, it will continue the lexing from the last point. Preprocessor registers TryGrowLexerBuffer callback in order to implement handling of SourceFileGrower. Note that all the code changes will not really affect the AOT clang world since they are all disabled when Preprocessor doesn't have SourceFileGrower instance. NOTE: This is part 2 and 3 of https://discourse.llvm.org/t/rfc-flexible-lexer-buffering-for-handling-incomplete-input-in-interactive-c-c/64180

Add TryGrowLexerBuffer to Lexer and SourceFileGrower to Preprocessor that can be used to grow the source code buffer when Lexer reaches the eof of the buffer. Since clang-repl receive source code incrementally line by line, we need to grow source code buffer. This change adds the interfaces to Lexer/Preprocessor in order to accommodate this need. If SourceFileGrower interface is registered to Preprocessor, it will call TryGrowFile method when Lexer reached eof. Inside this method, the user can grow the file and return true to request Lexer to continue the lexing from the last point. When Lexer reaches eof it will call TryGrowLexerBuffer callback specified in the constructor to try growing buffer and if it got new buffer, it will continue the lexing from the last point. Preprocessor registers TryGrowLexerBuffer callback in order to implement handling of SourceFileGrower. Note that all the code changes will not really affect the AOT clang world since they are all disabled when Preprocessor doesn't have SourceFileGrower instance. NOTE: This is part 2 and 3 of https://discourse.llvm.org/t/rfc-flexible-lexer-buffering-for-handling-incomplete-input-in-interactive-c-c/64180

Add TryGrowLexerBuffer to Lexer and SourceFileGrower to Preprocessor that can be used to grow the source code buffer when Lexer reaches the eof of the buffer. Since clang-repl receive source code incrementally line by line, we need to grow source code buffer. This change adds the interfaces to Lexer/Preprocessor in order to accommodate this need. If SourceFileGrower interface is registered to Preprocessor, it will call TryGrowFile method when Lexer reached eof whereeof. Inside this method, the user can grow the file and return true to request Lexer to continue the lexing from the last point. When Lexer reaches eof it will call TryGrowLexerBuffer callback specified in the constructor to try growing buffer and if it got new buffer, it will continue the lexing from the last point. Preprocessor registers TryGrowLexerBuffer callback in order to implement handling of SourceFileGrower. Note that all the code changes will not really affect the AOT clang world since they are all disabled when Preprocessor doesn't have SourceFileGrower instance. NOTE: This is part 2 and 3 of https://discourse.llvm.org/t/rfc-flexible-lexer-buffering-for-handling-incomplete-input-in-interactive-c-c/64180