[ClangRepl] support code completion at a REPL

[capfredf]

Summary

This patch enabled code completion for ClangREPL. The feature was built upon three
existing Clang components: a list completer for LineEditor,
a CompletionConsumer from SemaCodeCompletion, and the ASTUnit::codeComplete method.
with the first component serving as the main entry point of handling interactive inputs.

Because a completion point for a compiler instance has to be unchanged once it
is set, an incremental compiler instance is created for each code
completion. Such a compiler instance carries over AST context source from the
main interpreter compiler in order to obtain declarations or bindings from
previous input in the same REPL session.

The most important API codeComplete in Interpreter/CodeCompletion is a thin
wrapper that calls with ASTUnit::codeComplete with necessary arguments, such as
a code completion point and a ReplCompletionConsumer, which communicates
completion results from SemaCodeCompletion back to the list completer for the
REPL.

In addition, PCC_TopLevelOrExpression and CCC_TopLevelOrExpression` top levels
were added so that SemaCodeCompletion can treat top level statements like
expression statements at the REPL. For example,

clang-repl> int foo = 42;
clang-repl> f<tab>

From a parser's persective, the cursor is at a top level. If we used code
completion without any changes, PCC_Namespace would be supplied to
Sema::CodeCompleteOrdinaryName, and thus the completion results would not
include foo.

Currently, the way we use PCC_TopLevelOrExpression and
CCC_TopLevelOrExpression is no different from the way we use PCC_Statement
and CCC_Statement respectively.

[v.g.vassilev]

Can you extend the commit log to include a description of how things are currently done? For example, it would be good to read about design and technical decisions, etc.

clang/lib/Interpreter/CodeCompletion.cpp

80

Why we drop the error on the floor?

85

I am not sure why we need to wrap this code. Can we teach the TopLevelStmtDecl to work with the code completion logic?

[capfredf]

clang/lib/Interpreter/CodeCompletion.cpp

85

I think the problem has less to do with the TopLevelStmtDecl. Usually, a TopLevelStmtDecl instance is a successful result from ParseXXXXX, but when CodeCompletion is triggered during parsing, nothing meaning comes out from ParseXXXX.

We need to wrap our input because we need Sema::CodeCompleteOrdinaryName to work on code from the REPL in a different completion context than the same code from a regular source file.

In a regular C++ file,

int foo = 42;
f_

Since top-level expressions are not supported, foo should not be an option.

But in a REPL session,

clang-repl> int foo = 42;
clang-repl> f_

we are allowed to use foo to make a statement like foo + 84;.

Since the argument for CompletionContext to CodeCompleteOrdinaryName is controlled by the parsing process, ie. there is no way to configure it from outside, I think faking context is a straightforward solution.

Alternatively, we can check if the compiler is in incremental mode and call CodeCompleteOrdinaryName with the context we want in parsing. Something like:

if (PP.isIncrementalProcessingEnabled() {
  Actions.CodeCompleteOrdinaryName(getCurScope(), Sema::PCC_Statement);
} else {
  Actions.CodeCompleteOrdinaryName(getCurScope(), Sema::PCC_Namespace);
}

I personally think both solutions are extensionally equivalent, but the alternative is a bit intrusive.

[capfredf]

changes per discussions

[capfredf]

update

[capfredf]

move the comment up

[capfredf]

move the comment up

[capfredf]

remove an unused decl

[capfredf]

clang/lib/Interpreter/CodeCompletion.cpp

85

I think the combination of checking isIncrementalProcessingEnabled and adding a new PCC might be better since it is really a different context at the REPL

[capfredf]

get rid of the wrapping

[v.g.vassilev]

@capfredf this seems to be heading in a good direction. Please find another round of comments.

@aaron.ballman, I could not find a person that's active in that area. Could you help with finding reviewers for this type of patch?

clang/include/clang/Interpreter/Interpreter.h

117 ↗

(On Diff #537933)

I could not find where this interface was used. If it is unused, let'd drop it.

clang/include/clang/Sema/Sema.h

13455

clang/lib/Interpreter/IncrementalParser.cpp

318

Seems that we have stray debug fragments? Can you remove them?

330

Why we had to comment out this line?

clang/lib/Interpreter/Interpreter.cpp

245

IIUC, setCodeCompletionConsumer takes the ownership of CConsumer. I'd suggest to drop CConsumer member.

[capfredf]

clang/include/clang/Interpreter/Interpreter.h

117 ↗

(On Diff #537933)

it is used at L86 of CodeCompletion.cpp

clang/lib/Interpreter/IncrementalParser.cpp

330

This is old code. FID is created in a different way. See below.
I will remove the commented code

[capfredf]

• rename
• Do not export ReplCompletionConsumer
• remove commented code

[capfredf]

@v.g.vassilev I moved the ReplCodeCompletionConsumer class out of the header file, since it is not needed elsewhere.

[v.g.vassilev]

clang/include/clang/Sema/CodeCompleteConsumer.h

342

clang/lib/Interpreter/CodeCompletion.cpp

103

Let's move this in Interpreter::createForCodeCompletion.

119

Please remove the braces around single statement blocks.

clang/lib/Interpreter/Interpreter.cpp

291

I still do not entirely understand why we can "just" ask the codecompletion infrastructure for the possible options at the current position. I know that's probably easier set than done but I'd like to entertain that idea for a while..

[capfredf]

clang/lib/Interpreter/Interpreter.cpp

291

I'm a bit confused. Can you expand on the suggestion?

[capfredf]

clang/lib/Interpreter/CodeCompletion.cpp

103

The reasons I moved this out of Interpreter::createForCodeCompletion are

1. Design-wise, we don't need to expose ReplCompletionConsumer to Interpreter.cpp. All createForCodeCompletion needs is a subclass instance of CodeCompletionConsumer.
2. ReplCompletionConsumer has more data members on the type-directed branch than the one on this branch. I'd rather create an instance here than pass all the necessary arguments to one layer down and do that there.

[capfredf]

Remove unnecessary braces

[capfredf]

handle cases where code completion in a function declaration

[capfredf]

clang/unittests/Interpreter/CodeCompletionTest.cpp

72

@v.g.vassilev The way I fixed this is a bit hacky. Do you have a better idea?

[v.g.vassilev]

clang/unittests/Interpreter/CodeCompletionTest.cpp

72

Now I think I understand what you were after. Instead of handling the error in the constructor we should handle it in the caller. Here is some information how we could do that: https://llvm.org/docs/ProgrammersManual.html#fallible-constructors

[capfredf]

changes per @v.g.vassilev's comments

[v.g.vassilev]

I think we are getting there. I added a few suggestions how to improve the current design.

clang/include/clang/Interpreter/CodeCompletion.h

16

I still am a bit uncomfortable with relying on a line editor api to address a generic question one could as the interpreter such as "What's visible at that code position"?

Can we not make the overloaded operators outside of the ReplListCompleter class. That means moving them in ClangRepl.cpp?

clang/include/clang/Interpreter/Interpreter.h

42 ↗

(On Diff #543000)

We should put that in its alphabetical order.

clang/lib/Interpreter/CodeCompletion.cpp

112

I do not believe we need a spe

clang/lib/Interpreter/IncrementalParser.cpp

305

Is there any other particular reason for this routine to be different rather than PP.SetCodeCompletionPoint(*FE, Line, Col)? That is, can we merge back the old code and only set the point of completion externally?

[v.g.vassilev]

clang/lib/Interpreter/Interpreter.cpp

291

I do not believe we need a special case for creating a CodeCompletionConsumer. I think we could rely on the logic in CompilerInstance::createCodeCompletionConsumer which can be triggered in IncrementalAction::ExecuteAction just like in ASTFrontendAction::ExecuteAction. We will need to copy some of the code over but then you could start the same interpreter infrastructure with slightly different flags.

clang/lib/Parse/ParseDecl.cpp

6656

Can we test this change separately outside of clang-repl? That would mean creating a tests and run it with clang -Xclang -fincremental-extensions

clang/lib/Parse/Parser.cpp

934

Likewise, it would be great if we have a test in clang-only mode.

[capfredf]

clang/lib/Interpreter/CodeCompletion.cpp

112

I am not sure I am following.
For a regular interpreter, create Interpreter -> create Incremental Parser -> IncrmentalASTAction triggered -> set a default code completion consumer(PrintCodeCompletionConsumer) to CI if no code consumer has been set -> Sema initialized.
Based on this logic, we need to provide our CodeConsumer so that IncrementalASTAction does not create the default, plus we need to a parent CI to a new code completion interpreter. I don't see how we can avoid a special code path for creation

clang/lib/Interpreter/IncrementalParser.cpp

305

The regular Parse does some lexing after parsing, which I don't think is necessary for code completion.

Also, this is more or less because, as a functional programmer, I prefer building things using small functions with explicit arguments.

I'll try if I can use the old parse with the completion point set and tested.

[capfredf]

changes per discussions

[capfredf]

use ASTUnit::codeComplete

[capfredf]

clang/include/clang/Frontend/ASTUnit.h

906

@v.g.vassilev Not sure if this is the right solution or idiom to extend this method.

we can make this high order function more general, like std::unique_ptr<SyntaxOnlyAction*><()>

[capfredf]

changes per discussions

[v.g.vassilev]

clang/lib/Interpreter/IncrementalParser.h

90

Can we not move this class definition locally to the CodeComplete interface?

[v.g.vassilev]

This looks mostly good to me. I noticed that @sammccall has done some work in that area in clangd and I was wondering if he has some thoughts about this patch.

[capfredf]

up

[capfredf]

up

[capfredf]

use auto

[v.g.vassilev]

clang/lib/Interpreter/CodeCompletion.cpp

2

I would propose to rename this file to InterpreterCodeCompletion.cpp and implement the Interpreter::codeComplete interface. Then we can move all of the content of ExternalSource.{h,cpp}, CodeCompletion.h and ExternalSource.h in it. That will increase the encapsulation of the code.

[capfredf]

move everything into InterpreterCodeCompletion.

the entry point for code completion is now a function, codeCompletion

[capfredf]

format

[sammccall]

This mostly looks good.

My main concern is adding code to the parser to suppress errors when code completing: this is unlikely to be the only such place. Existing consumers seem happy to ignore errors, if this doesn't work for clang-repl it'd be good to understand why.

clang/include/clang/Frontend/ASTUnit.h

891

This comment doesn't really say anything beyond echoing the name/type, either remove it or add some separate explanation?

e.g. "If Act is specified, it will be used to parse the file. This allows customizing the parse by overriding FrontendAction's lifecycle methods."

906

There's no need for this to be a SyntaxOnlyAction, you can take FrontendAction here.

I don't think there's any need to provide a factory, it's safe to assume it will always create one action. (We have FrontendActionFactory in tooling, and it's a pain.)

clang/include/clang/Interpreter/InterpreterCodeCompletion.h

30 ↗

(On Diff #549760)

nit: lowercase initial letter for functions

30 ↗

(On Diff #549760)

you might consider vector<string> instead.

this signature doesn't provide any way to return owned names, and so requires that the results are already allocated somewhere as contiguous strings.

This will prevents completing names like operator int, as well as some non-names that might prove useful.

This is an interactive operation, copying the strings is unlikely to add significant latency on human scale.

clang/include/clang/Sema/CodeCompleteConsumer.h

342

I don't think this name fits with the others, it describes the client rather than the grammatical/semantic context.

I would suggest maybe CCC_TopLevelOrExpression?

clang/lib/Interpreter/InterpreterCodeCompletion.cpp

72 ↗

(On Diff #549760)

(up to you, but default between cases seems surprising)

clang/lib/Parse/ParseDecl.cpp

6648

AIUI, all code completion actions should be ignoring errors, and we should always be happy to cut off parsing and give up as soon as the completion token is reached.

So why do we need special handling for incremental processing here?

(i.e. what behavior do non-repl users see here - if it's correct can clang-repl do the same, if it's wrong why don't we fix both?)

[capfredf]

address @sammccall 's comments

clang/include/clang/Frontend/ASTUnit.h

906

Ideally, this method requires a FrontendAction with hasCodeCompletionSupport returning true. The reason I chose SyntaxOnlyAction are 1. the old code used it 2. SyntaxOnlyAction::hasCodeCompletionSupport returns true. Certainly, right now SyntaxOnlyAction does not prevent its subclasses from overriding this method.

clang/include/clang/Sema/CodeCompleteConsumer.h

342

Nice. This one sounds better indeed.

[capfredf]

@sammccall Thank you very much for your valuable input.

[capfredf]

up

[v.g.vassilev]

@sammccall could you take another look? It seems quite ready to me to land.

[sammccall]

LGTM, thanks!

clang/include/clang/Sema/CodeCompleteConsumer.h

341

I think this is non-obvious & worthy of more explanation

/// Completion in a namespace or global scope, but also accept expressions.
/// This occurs in REPL inputs, which may declare things  or print expression
/// values.

[v.g.vassilev]

Thanks for working on this @capfredf. Let me know if I should land that for you.

[capfredf]

update the commit message

[capfredf]

fix code in Clangd

[v.g.vassilev]

clang/unittests/Interpreter/CodeCompletionTest.cpp

39

We should make this an out parameter instead of returning it by copy.

43

[sammccall]

clang-tools-extra/clangd/CodeComplete.cpp

845 ↗

(On Diff #552736)

This should be true rather than false, since both TopLevel and Expression are completed using the index.

(Though I realize that for now we'll never hit this context)

[capfredf]

fix potential memory issues

[capfredf]

fix memory issues

clang/include/clang/Frontend/ASTUnit.h

 class InputKind;
 class InMemoryModuleCache;
 class PCHContainerOperations;
 class PCHContainerReader;
 class Preprocessor;
 class PreprocessorOptions;
 class Sema;
 class TargetInfo;
+class SyntaxOnlyAction;
 
 /// \brief Enumerates the available scopes for skipping function bodies.
 enum class SkipFunctionBodiesScope { None, Preamble, PreambleAndMainFile };
 
 /// \brief Enumerates the available kinds for capturing diagnostics.
 enum class CaptureDiagsKind { None, All, AllWithoutNonErrorsFromIncludes };
 
 /// Utility class for loading a ASTContext from an AST file.
public:
   /// results.
   ///
   /// \param IncludeCodePatterns Whether to include code patterns (such as a
   /// for loop) in the code-completion results.
   ///
   /// \param IncludeBriefComments Whether to include brief documentation within
   /// the set of code completions returned.
   ///
+  /// \param Act If supplied, this argument is used to parse the input file,
+  /// allowing customized parsing by overriding SyntaxOnlyAction lifecycle
+  /// methods.
+  ///
   /// FIXME: The Diag, LangOpts, SourceMgr, FileMgr, StoredDiagnostics, and
   /// OwnedBuffers parameters are all disgusting hacks. They will go away.
   void CodeComplete(StringRef File, unsigned Line, unsigned Column,
                     ArrayRef<RemappedFile> RemappedFiles, bool IncludeMacros,
                     bool IncludeCodePatterns, bool IncludeBriefComments,
                     CodeCompleteConsumer &Consumer,
                     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
                     DiagnosticsEngine &Diag, LangOptions &LangOpts,
                     SourceManager &SourceMgr, FileManager &FileMgr,
                     SmallVectorImpl<StoredDiagnostic> &StoredDiagnostics,
-                    SmallVectorImpl<const llvm::MemoryBuffer *> &OwnedBuffers);
+                    SmallVectorImpl<const llvm::MemoryBuffer *> &OwnedBuffers,
+                    std::unique_ptr<SyntaxOnlyAction> Act = nullptr);
 
   /// Save this translation unit to a file with the given name.
   ///
   /// \returns true if there was a file error or false if the save was
   /// successful.
   bool Save(StringRef File);
 
   /// Serialize this translation unit with the given output stream.
   ///
   /// \returns True if an error occurred, false otherwise.
   bool serialize(raw_ostream &OS);
 };
 
 } // namespace clang
 
 #endif // LLVM_CLANG_FRONTEND_ASTUNIT_H

clang/include/clang/Interpreter/CodeCompletion.h

+//===----- CodeCompletion.h - Code Completion for ClangRepl ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the classes which performs code completion at the REPL.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_INTERPRETER_CODE_COMPLETION_H
+#define LLVM_CLANG_INTERPRETER_CODE_COMPLETION_H
+#include <string>
+#include <vector>
+
+namespace llvm {
+class StringRef;
+} // namespace llvm
+
+namespace clang {
+class CodeCompletionResult;
+class CompilerInstance;
+
+void codeComplete(CompilerInstance *InterpCI, llvm::StringRef Content,
+                  unsigned Line, unsigned Col, const CompilerInstance *ParentCI,
+                  std::vector<CodeCompletionResult> &CCResults);
+
+std::vector<std::string>
+convertToCodeCompleteStrings(const std::vector<CodeCompletionResult> &Results);
+} // namespace clang
+#endif

clang/include/clang/Sema/CodeCompleteConsumer.h

enum Kind {
     /// Code completion of an attribute name.
     CCC_Attribute,
 
     /// An unknown context, in which we are recovering from a parsing
     /// error and don't know which completions we should give.
     CCC_Recovery,
 
     /// Code completion in a @class forward declaration.
-    CCC_ObjCClassForwardDecl
+    CCC_ObjCClassForwardDecl,
+
+    /// Code completion at a top level in a REPL session.
+    CCC_TopLevelOrExpression,
   };
 
   using VisitedContextSet = llvm::SmallPtrSet<DeclContext *, 8>;
 
 private:
   Kind CCKind;
 
   /// Indicates whether we are completing a name of a using declaration, e.g.

clang/include/clang/Sema/Sema.h

enum ParserCompletionContext {
     PCC_RecoveryInFunction,
     /// Code completion occurs where only a type is permitted.
     PCC_Type,
     /// Code completion occurs in a parenthesized expression, which
     /// might also be a type cast.
     PCC_ParenthesizedExpression,
     /// Code completion occurs within a sequence of declaration
     /// specifiers within a function, method, or block.
-    PCC_LocalDeclarationSpecifiers
+    PCC_LocalDeclarationSpecifiers,
+    /// Code completion occurs at top-level in a REPL session
+    PCC_TopLevelOrExpression,
   };
 
   void CodeCompleteModuleImport(SourceLocation ImportLoc, ModuleIdPath Path);
   void CodeCompleteOrdinaryName(Scope *S,
                                 ParserCompletionContext CompletionContext);
   void CodeCompleteDeclSpec(Scope *S, DeclSpec &DS,
                             bool AllowNonIdentifiers,
                             bool AllowNestedNameSpecifiers);

clang/lib/Frontend/ASTUnit.cpp

static void CalculateHiddenNames(const CodeCompletionContext &Context,
   case CodeCompletionContext::CCC_ArrowMemberAccess:
   case CodeCompletionContext::CCC_ObjCPropertyAccess:
   case CodeCompletionContext::CCC_Namespace:
   case CodeCompletionContext::CCC_Type:
   case CodeCompletionContext::CCC_Symbol:
   case CodeCompletionContext::CCC_SymbolOrNewName:
   case CodeCompletionContext::CCC_ParenthesizedExpression:
   case CodeCompletionContext::CCC_ObjCInterfaceName:
+  case CodeCompletionContext::CCC_TopLevelOrExpression:
       break;
 
   case CodeCompletionContext::CCC_EnumTag:
   case CodeCompletionContext::CCC_UnionTag:
   case CodeCompletionContext::CCC_ClassOrStructTag:
     OnlyTagNames = true;
     break;
 
   case CodeCompletionContext::CCC_ObjCProtocolName:

 void ASTUnit::CodeComplete(
     StringRef File, unsigned Line, unsigned Column,
     ArrayRef<RemappedFile> RemappedFiles, bool IncludeMacros,
     bool IncludeCodePatterns, bool IncludeBriefComments,
     CodeCompleteConsumer &Consumer,
     std::shared_ptr<PCHContainerOperations> PCHContainerOps,
     DiagnosticsEngine &Diag, LangOptions &LangOpts, SourceManager &SourceMgr,
     FileManager &FileMgr, SmallVectorImpl<StoredDiagnostic> &StoredDiagnostics,
-    SmallVectorImpl<const llvm::MemoryBuffer *> &OwnedBuffers) {
+    SmallVectorImpl<const llvm::MemoryBuffer *> &OwnedBuffers,
+    std::unique_ptr<SyntaxOnlyAction> Act) {
   if (!Invocation)
     return;
 
   SimpleTimer CompletionTimer(WantTiming);
   CompletionTimer.setOutput("Code completion @ " + File + ":" +
                             Twine(Line) + ":" + Twine(Column));
 
   auto CCInvocation = std::make_shared<CompilerInvocation>(*Invocation);
if (OverrideMainBuffer) {
     PreprocessorOpts.PrecompiledPreambleBytes.first = 0;
     PreprocessorOpts.PrecompiledPreambleBytes.second = false;
   }
 
   // Disable the preprocessing record if modules are not enabled.
   if (!Clang->getLangOpts().Modules)
     PreprocessorOpts.DetailedRecord = false;
 
-  std::unique_ptr<SyntaxOnlyAction> Act;
+  if (!Act)
     Act.reset(new SyntaxOnlyAction);
+
   if (Act->BeginSourceFile(*Clang.get(), Clang->getFrontendOpts().Inputs[0])) {
     if (llvm::Error Err = Act->Execute()) {
       consumeError(std::move(Err)); // FIXME this drops errors on the floor.
     }
     Act->EndSourceFile();
   }
 }
 

clang/lib/Interpreter/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
    core
    native
    MC
    Option
    OrcJit
    OrcShared
    OrcTargetProcess
    Support
    Target
    TargetParser
   )
 
 add_clang_library(clangInterpreter
   DeviceOffload.cpp
+  CodeCompletion.cpp
   IncrementalExecutor.cpp
   IncrementalParser.cpp
   Interpreter.cpp
   InterpreterUtils.cpp
   Value.cpp
 
   DEPENDS
   intrinsics_gen

clang/lib/Interpreter/CodeCompletion.cpp

+//===------ CodeCompletion.cpp - Code Completion for ClangRepl -------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements the classes which performs code completion at the REPL.
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Interpreter/CodeCompletion.h"
+#include "clang/AST/ASTImporter.h"
+#include "clang/AST/DeclarationName.h"
+#include "clang/AST/ExternalASTSource.h"
+#include "clang/Basic/IdentifierTable.h"
+#include "clang/Frontend/ASTUnit.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Interpreter/Interpreter.h"
+#include "clang/Lex/PreprocessorOptions.h"
+#include "clang/Sema/CodeCompleteConsumer.h"
+#include "clang/Sema/CodeCompleteOptions.h"
+#include "clang/Sema/Sema.h"
+
+namespace clang {
+
+const std::string CodeCompletionFileName = "input_line_[Completion]";
+
+clang::CodeCompleteOptions getClangCompleteOpts() {
+  clang::CodeCompleteOptions Opts;
+  Opts.IncludeCodePatterns = true;
+  Opts.IncludeMacros = true;
+  Opts.IncludeGlobals = true;
+  Opts.IncludeBriefComments = true;
+  return Opts;
+}
+
+class ReplCompletionConsumer : public CodeCompleteConsumer {
+public:
+  ReplCompletionConsumer(std::vector<CodeCompletionResult> &Results)
+      : CodeCompleteConsumer(getClangCompleteOpts()),
+        CCAllocator(std::make_shared<GlobalCodeCompletionAllocator>()),
+        CCTUInfo(CCAllocator), Results(Results){};
+
+  void ProcessCodeCompleteResults(class Sema &S, CodeCompletionContext Context,
+                                  CodeCompletionResult *InResults,
+                                  unsigned NumResults) final;
+
+  CodeCompletionAllocator &getAllocator() override { return *CCAllocator; }
+
+  CodeCompletionTUInfo &getCodeCompletionTUInfo() override { return CCTUInfo; }
+
+private:
+  std::shared_ptr<GlobalCodeCompletionAllocator> CCAllocator;
+  CodeCompletionTUInfo CCTUInfo;
+  std::vector<CodeCompletionResult> &Results;
+};
+
+void ReplCompletionConsumer::ProcessCodeCompleteResults(
+    class Sema &S, CodeCompletionContext Context,
+    CodeCompletionResult *InResults, unsigned NumResults) {
+  for (unsigned I = 0; I < NumResults; ++I) {
+    auto &Result = InResults[I];
+    switch (Result.Kind) {
+    case CodeCompletionResult::RK_Declaration:
+      if (Result.Declaration->getIdentifier()) {
+        Results.push_back(Result);
+      }
+      break;
+    case CodeCompletionResult::RK_Keyword:
+      Results.push_back(Result);
+      break;
+    default:
+      break;
+    }
+  }
+};
+
+std::vector<std::string> convertToCodeCompleteStrings(
+    const std::vector<clang::CodeCompletionResult> &Results) {
+  std::vector<std::string> CompletionStrings;
+  for (auto Res : Results) {
+    switch (Res.Kind) {
+    case clang::CodeCompletionResult::RK_Declaration:
+      if (auto *ID = Res.Declaration->getIdentifier()) {
+        CompletionStrings.push_back(ID->getName().str());
+      }
+      break;
+    case clang::CodeCompletionResult::RK_Keyword:
+      CompletionStrings.push_back(Res.Keyword);
+      break;
+    default:
+      break;
+    }
+  }
+  return CompletionStrings;
+}
+
+class IncrementalSyntaxOnlyAction : public SyntaxOnlyAction {
+  const CompilerInstance *ParentCI;
+
+public:
+  IncrementalSyntaxOnlyAction(const CompilerInstance *ParentCI)
+      : ParentCI(ParentCI) {}
+
+protected:
+  void ExecuteAction() override;
+};
+
+class ExternalSource : public clang::ExternalASTSource {
+  ASTContext &ChildASTCtxt;
+  TranslationUnitDecl *ChildTUDeclCtxt;
+  ASTContext &ParentASTCtxt;
+  TranslationUnitDecl *ParentTUDeclCtxt;
+
+  std::unique_ptr<ASTImporter> Importer;
+
+public:
+  ExternalSource(ASTContext &ChildASTCtxt, FileManager &ChildFM,
+                 ASTContext &ParentASTCtxt, FileManager &ParentFM);
+  bool FindExternalVisibleDeclsByName(const DeclContext *DC,
+                                      DeclarationName Name) override;
+  void
+  completeVisibleDeclsMap(const clang::DeclContext *childDeclContext) override;
+};
+
+// This method is intended to set up `ExternalASTSource` to the running
+// compiler instance before the super `ExecuteAction` triggers parsing
+void IncrementalSyntaxOnlyAction::ExecuteAction() {
+  CompilerInstance &CI = getCompilerInstance();
+  ExternalSource *myExternalSource =
+      new ExternalSource(CI.getASTContext(), CI.getFileManager(),
+                         ParentCI->getASTContext(), ParentCI->getFileManager());
+  llvm::IntrusiveRefCntPtr<clang::ExternalASTSource> astContextExternalSource(
+      myExternalSource);
+  CI.getASTContext().setExternalSource(astContextExternalSource);
+  CI.getASTContext().getTranslationUnitDecl()->setHasExternalVisibleStorage(
+      true);
+
+  SyntaxOnlyAction::ExecuteAction();
+}
+
+ExternalSource::ExternalSource(ASTContext &ChildASTCtxt, FileManager &ChildFM,
+                               ASTContext &ParentASTCtxt, FileManager &ParentFM)
+    : ChildASTCtxt(ChildASTCtxt),
+      ChildTUDeclCtxt(ChildASTCtxt.getTranslationUnitDecl()),
+      ParentASTCtxt(ParentASTCtxt),
+      ParentTUDeclCtxt(ParentASTCtxt.getTranslationUnitDecl()) {
+  ASTImporter *importer =
+      new ASTImporter(ChildASTCtxt, ChildFM, ParentASTCtxt, ParentFM,
+                      /*MinimalImport : ON*/ true);
+  Importer.reset(importer);
+}
+
+bool ExternalSource::FindExternalVisibleDeclsByName(const DeclContext *DC,
+                                                    DeclarationName Name) {
+  IdentifierTable &ParentIdTable = ParentASTCtxt.Idents;
+
+  auto ParentDeclName =
+      DeclarationName(&(ParentIdTable.get(Name.getAsString())));
+
+  DeclContext::lookup_result lookup_result =
+      ParentTUDeclCtxt->lookup(ParentDeclName);
+
+  if (!lookup_result.empty()) {
+    return true;
+  }
+  return false;
+}
+
+void ExternalSource::completeVisibleDeclsMap(
+    const DeclContext *ChildDeclContext) {
+  assert(ChildDeclContext && ChildDeclContext == ChildTUDeclCtxt &&
+         "No child decl context!");
+
+  if (!ChildDeclContext->hasExternalVisibleStorage())
+    return;
+
+  for (auto *DeclCtxt = ParentTUDeclCtxt; DeclCtxt != nullptr;
+       DeclCtxt = DeclCtxt->getPreviousDecl()) {
+    for (auto &IDeclContext : DeclCtxt->decls()) {
+      if (NamedDecl *Decl = llvm::dyn_cast<NamedDecl>(IDeclContext)) {
+        if (auto DeclOrErr = Importer->Import(Decl)) {
+          if (NamedDecl *importedNamedDecl =
+                  llvm::dyn_cast<NamedDecl>(*DeclOrErr)) {
+            SetExternalVisibleDeclsForName(ChildDeclContext,
+                                           importedNamedDecl->getDeclName(),
+                                           importedNamedDecl);
+          }
+
+        } else {
+          llvm::consumeError(DeclOrErr.takeError());
+        }
+      }
+    }
+    ChildDeclContext->setHasExternalLexicalStorage(false);
+  }
+}
+
+void codeComplete(CompilerInstance *InterpCI, llvm::StringRef Content,
+                  unsigned Line, unsigned Col, const CompilerInstance *ParentCI,
+                  std::vector<CodeCompletionResult> &CCResults) {
+  std::unique_ptr<llvm::MemoryBuffer> MB =
+      llvm::MemoryBuffer::getMemBufferCopy(Content, CodeCompletionFileName);
+  llvm::SmallVector<ASTUnit::RemappedFile, 4> RemappedFiles;
+
+  RemappedFiles.push_back(std::make_pair(CodeCompletionFileName, MB.release()));
+
+  auto DiagOpts = DiagnosticOptions();
+  auto consumer = ReplCompletionConsumer(CCResults);
+
+  auto diag = InterpCI->getDiagnosticsPtr();
+  ASTUnit *AU = ASTUnit::LoadFromCompilerInvocationAction(
+      InterpCI->getInvocationPtr(), std::make_shared<PCHContainerOperations>(),
+      diag);
+  llvm::SmallVector<clang::StoredDiagnostic, 8> sd = {};
+  llvm::SmallVector<const llvm::MemoryBuffer *, 1> tb = {};
+  InterpCI->getFrontendOpts().Inputs[0] = FrontendInputFile(
+      CodeCompletionFileName, Language::CXX, InputKind::Source);
+  auto Act = std::unique_ptr<IncrementalSyntaxOnlyAction>(
+      new IncrementalSyntaxOnlyAction(ParentCI));
+  AU->CodeComplete(CodeCompletionFileName, 1, Col, RemappedFiles, false, false,
+                   false, consumer,
+                   std::make_shared<clang::PCHContainerOperations>(), *diag,
+                   InterpCI->getLangOpts(), InterpCI->getSourceManager(),
+                   InterpCI->getFileManager(), sd, tb, std::move(Act));
+}
+
+} // namespace clang

clang/lib/Interpreter/IncrementalParser.h

 //===--- IncrementalParser.h - Incremental Compilation ----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the class which performs incremental code compilation.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H
 #define LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H
 
+#include "clang/AST/GlobalDecl.h"
 #include "clang/Interpreter/PartialTranslationUnit.h"
 
-#include "clang/AST/GlobalDecl.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Error.h"
 
 #include <list>
 #include <memory>
 namespace llvm {
 class LLVMContext;
-}
+} // namespace llvm
 
 namespace clang {
 class ASTConsumer;
 class CodeGenerator;
 class CompilerInstance;
 class IncrementalAction;
 class Interpreter;
 class Parser;
public:
   std::list<PartialTranslationUnit> &getPTUs() { return PTUs; }
 
   std::unique_ptr<llvm::Module> GenModule();
 
 private:
   llvm::Expected<PartialTranslationUnit &> ParseOrWrapTopLevelDecl();
 };
 } // end namespace clang
 
 #endif // LLVM_CLANG_LIB_INTERPRETER_INCREMENTALPARSER_H

clang/lib/Interpreter/IncrementalParser.cpp

 //===--------- IncrementalParser.cpp - Incremental Compilation  -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the class which performs incremental code compilation.
 //
 //===----------------------------------------------------------------------===//
 
 #include "IncrementalParser.h"
+
 #include "clang/AST/DeclContextInternals.h"
 #include "clang/CodeGen/BackendUtil.h"
 #include "clang/CodeGen/CodeGenAction.h"
 #include "clang/CodeGen/ModuleBuilder.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/FrontendTool/Utils.h"
 #include "clang/Interpreter/Interpreter.h"
IncrementalAction(CompilerInstance &CI, llvm::LLVMContext &LLVMCtx,
             break;
           }
           return Act;
         }()) {}
   FrontendAction *getWrapped() const { return WrappedAction.get(); }
   TranslationUnitKind getTranslationUnitKind() override {
     return TU_Incremental;
   }
+
   void ExecuteAction() override {
     CompilerInstance &CI = getCompilerInstance();
     assert(CI.hasPreprocessor() && "No PP!");
 
-    // FIXME: Move the truncation aspect of this into Sema, we delayed this till
-    // here so the source manager would be initialized.
-    if (hasCodeCompletionSupport() &&
-        !CI.getFrontendOpts().CodeCompletionAt.FileName.empty())
-      CI.createCodeCompletionConsumer();
-
     // Use a code completion consumer?
     CodeCompleteConsumer *CompletionConsumer = nullptr;
     if (CI.hasCodeCompletionConsumer())
       CompletionConsumer = &CI.getCodeCompletionConsumer();
 
     Preprocessor &PP = CI.getPreprocessor();
     PP.EnterMainSourceFile();
 
IncrementalParser::ParseOrWrapTopLevelDecl() {
   GlobalInstantiations.perform();
 
   Consumer->HandleTranslationUnit(C);
 
   return LastPTU;
 }
 
 llvm::Expected<PartialTranslationUnit &>
 IncrementalParser::Parse(llvm::StringRef input) {
   Preprocessor &PP = CI->getPreprocessor();
   assert(PP.isIncrementalProcessingEnabled() && "Not in incremental mode!?");
 
   std::ostringstream SourceName;
   SourceName << "input_line_" << InputCount++;
 
   // Create an uninitialized memory buffer, copy code in and append "\n"
   size_t InputSize = input.size(); // don't include trailing 0
   // MemBuffer size should *not* include terminating zero
   std::unique_ptr<llvm::MemoryBuffer> MB(
       llvm::WritableMemoryBuffer::getNewUninitMemBuffer(InputSize + 1,
                                                         SourceName.str()));
   char *MBStart = const_cast<char *>(MB->getBufferStart());
   memcpy(MBStart, input.data(), InputSize);
   MBStart[InputSize] = '\n';
 
   SourceManager &SM = CI->getSourceManager();
 
   // FIXME: Create SourceLocation, which will allow clang to order the overload
   // candidates for example
   SourceLocation NewLoc = SM.getLocForStartOfFile(SM.getMainFileID());
 
   // Create FileID for the current buffer.
   FileID FID = SM.createFileID(std::move(MB), SrcMgr::C_User, /*LoadedID=*/0,
                                /*LoadedOffset=*/0, NewLoc);
 
   // NewLoc only used for diags.
   if (PP.EnterSourceFile(FID, /*DirLookup=*/nullptr, NewLoc))
     return llvm::make_error<llvm::StringError>("Parsing failed. "
                                                "Cannot enter source file.",
                                                std::error_code());
 
   auto PTU = ParseOrWrapTopLevelDecl();
void IncrementalParser::CleanUpPTU(PartialTranslationUnit &PTU) {
   }
 }
 
 llvm::StringRef IncrementalParser::GetMangledName(GlobalDecl GD) const {
   CodeGenerator *CG = getCodeGen();
   assert(CG);
   return CG->GetMangledName(GD);
 }
-
 } // end namespace clang

clang/lib/Interpreter/Interpreter.cpp

 //===------ Interpreter.cpp - Incremental Compilation and Execution -------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the component which performs incremental code
 // compilation and execution.
 //
 //===----------------------------------------------------------------------===//
 
-#include "clang/Interpreter/Interpreter.h"
-
 #include "DeviceOffload.h"
 #include "IncrementalExecutor.h"
 #include "IncrementalParser.h"
-
 #include "InterpreterUtils.h"
+
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/Mangle.h"
 #include "clang/AST/TypeVisitor.h"
 #include "clang/Basic/DiagnosticSema.h"
 #include "clang/Basic/TargetInfo.h"
 #include "clang/CodeGen/CodeGenAction.h"
 #include "clang/CodeGen/ModuleBuilder.h"
 #include "clang/CodeGen/ObjectFilePCHContainerOperations.h"
 #include "clang/Driver/Compilation.h"
 #include "clang/Driver/Driver.h"
 #include "clang/Driver/Job.h"
 #include "clang/Driver/Options.h"
 #include "clang/Driver/Tool.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/TextDiagnosticBuffer.h"
+#include "clang/Interpreter/Interpreter.h"
 #include "clang/Interpreter/Value.h"
 #include "clang/Lex/PreprocessorOptions.h"
 #include "clang/Sema/Lookup.h"
 #include "llvm/ExecutionEngine/JITSymbol.h"
 #include "llvm/ExecutionEngine/Orc/LLJIT.h"
 #include "llvm/IR/Module.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/ErrorHandling.h"
CreateCI(const llvm::opt::ArgStringList &Argv) {
   Clang->getTarget().adjust(Clang->getDiagnostics(), Clang->getLangOpts());
 
   // Don't clear the AST before backend codegen since we do codegen multiple
   // times, reusing the same AST.
   Clang->getCodeGenOpts().ClearASTBeforeBackend = false;
 
   Clang->getFrontendOpts().DisableFree = false;
   Clang->getCodeGenOpts().DisableFree = false;
-
   return std::move(Clang);
 }
 
 } // anonymous namespace
 
 llvm::Expected<std::unique_ptr<CompilerInstance>>
 IncrementalCompilerBuilder::create(std::vector<const char *> &ClangArgv) {
 

 
 Interpreter::~Interpreter() {
   if (IncrExecutor) {
     if (llvm::Error Err = IncrExecutor->cleanUp())
       llvm::report_fatal_error(
           llvm::Twine("Failed to clean up IncrementalExecutor: ") +
           toString(std::move(Err)));
   }
 }
 
 // These better to put in a runtime header but we can't. This is because we
 // can't find the precise resource directory in unittests so we have to hard
 // code them.
 const char *const Runtimes = R"(
     void* operator new(__SIZE_TYPE__, void* __p) noexcept;
     void *__clang_Interpreter_SetValueWithAlloc(void*, void*, void*);
     void __clang_Interpreter_SetValueNoAlloc(void*, void*, void*);

 
 llvm::Expected<std::unique_ptr<Interpreter>>
 Interpreter::create(std::unique_ptr<CompilerInstance> CI) {
   llvm::Error Err = llvm::Error::success();
   auto Interp =
       std::unique_ptr<Interpreter>(new Interpreter(std::move(CI), Err));
   if (Err)
     return std::move(Err);
+
   auto PTU = Interp->Parse(Runtimes);
   if (!PTU)
     return PTU.takeError();
 
   Interp->ValuePrintingInfo.resize(3);
   // FIXME: This is a ugly hack. Undo command checks its availability by looking
   // at the size of the PTU list. However we have parsed something in the
   // beginning of the REPL so we have to mark them as 'Irrevocable'.
   Interp->InitPTUSize = Interp->IncrParser->getPTUs().size();
   return std::move(Interp);
 }
 
 llvm::Expected<std::unique_ptr<Interpreter>>
 Interpreter::createWithCUDA(std::unique_ptr<CompilerInstance> CI,
                             std::unique_ptr<CompilerInstance> DCI) {
   // avoid writing fat binary to disk using an in-memory virtual file system
   llvm::IntrusiveRefCntPtr<llvm::vfs::InMemoryFileSystem> IMVFS =
       std::make_unique<llvm::vfs::InMemoryFileSystem>();
   llvm::IntrusiveRefCntPtr<llvm::vfs::OverlayFileSystem> OverlayVFS =
       std::make_unique<llvm::vfs::OverlayFileSystem>(
           llvm::vfs::getRealFileSystem());
   OverlayVFS->pushOverlay(IMVFS);

clang/lib/Parse/ParseDecl.cpp

 #include "clang/AST/ASTContext.h"
 #include "clang/AST/DeclTemplate.h"
 #include "clang/AST/PrettyDeclStackTrace.h"
 #include "clang/Basic/AddressSpaces.h"
 #include "clang/Basic/AttributeCommonInfo.h"
 #include "clang/Basic/Attributes.h"
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/TargetInfo.h"
+#include "clang/Basic/TokenKinds.h"
 #include "clang/Parse/ParseDiagnostic.h"
 #include "clang/Parse/Parser.h"
 #include "clang/Parse/RAIIObjectsForParser.h"
 #include "clang/Sema/EnterExpressionEvaluationContext.h"
 #include "clang/Sema/Lookup.h"
 #include "clang/Sema/ParsedTemplate.h"
 #include "clang/Sema/Scope.h"
 #include "clang/Sema/SemaDiagnostic.h"
assert(D.isPastIdentifier() &&
          "Haven't past the location of the identifier yet?");
 
   // Don't parse attributes unless we have parsed an unparenthesized name.
   if (D.hasName() && !D.getNumTypeObjects())
     MaybeParseCXX11Attributes(D);
 
   while (true) {
     if (Tok.is(tok::l_paren)) {
       bool IsFunctionDeclaration = D.isFunctionDeclaratorAFunctionDeclaration();
       // Enter function-declaration scope, limiting any declarators to the
       // function prototype scope, including parameter declarators.
       ParseScope PrototypeScope(this,
                                 Scope::FunctionPrototypeScope|Scope::DeclScope|
                                 (IsFunctionDeclaration
                                    ? Scope::FunctionDeclarationScope : 0));
 
       // The paren may be part of a C++ direct initializer, eg. "int x(1);".
       // In such a case, check if we actually have a function declarator; if it
       // is not, the declarator has been fully parsed.
       bool IsAmbiguous = false;
       if (getLangOpts().CPlusPlus && D.mayBeFollowedByCXXDirectInit()) {
         // C++2a [temp.res]p5
         // A qualified-id is assumed to name a type if
         //   - [...]
         //   - it is a decl-specifier of the decl-specifier-seq of a

clang/lib/Parse/Parser.cpp

Parser::ParseExternalDeclaration(ParsedAttributes &Attrs,
   case tok::code_completion:
     cutOffParsing();
     if (CurParsedObjCImpl) {
       // Code-complete Objective-C methods even without leading '-'/'+' prefix.
       Actions.CodeCompleteObjCMethodDecl(getCurScope(),
                                          /*IsInstanceMethod=*/std::nullopt,
                                          /*ReturnType=*/nullptr);
     }
-    Actions.CodeCompleteOrdinaryName(
-        getCurScope(),
-        CurParsedObjCImpl ? Sema::PCC_ObjCImplementation : Sema::PCC_Namespace);
+
+    Sema::ParserCompletionContext PCC;
+    if (CurParsedObjCImpl) {
+      PCC = Sema::PCC_ObjCImplementation;
+    } else if (PP.isIncrementalProcessingEnabled()) {
+      PCC = Sema::PCC_TopLevelOrExpression;
+    } else {
+      PCC = Sema::PCC_Namespace;
+    };
+    Actions.CodeCompleteOrdinaryName(getCurScope(), PCC);
     return nullptr;
   case tok::kw_import: {
     Sema::ModuleImportState IS = Sema::ModuleImportState::NotACXX20Module;
     if (getLangOpts().CPlusPlusModules) {
       llvm_unreachable("not expecting a c++20 import here");
       ProhibitAttributes(Attrs);
     }
     SingleDecl = ParseModuleImport(SourceLocation(), IS);

clang/lib/Sema/CodeCompleteConsumer.cpp

bool CodeCompletionContext::wantConstructorResults() const {
   switch (CCKind) {
   case CCC_Recovery:
   case CCC_Statement:
   case CCC_Expression:
   case CCC_ObjCMessageReceiver:
   case CCC_ParenthesizedExpression:
   case CCC_Symbol:
   case CCC_SymbolOrNewName:
+  case CCC_TopLevelOrExpression:
     return true;
 
   case CCC_TopLevel:
   case CCC_ObjCInterface:
   case CCC_ObjCImplementation:
   case CCC_ObjCIvarList:
   case CCC_ClassStructUnion:
   case CCC_DotMemberAccess:
StringRef clang::getCompletionKindString(CodeCompletionContext::Kind Kind) {
   case CCKind::CCC_IncludedFile:
     return "IncludedFile";
   case CCKind::CCC_Attribute:
     return "Attribute";
   case CCKind::CCC_Recovery:
     return "Recovery";
   case CCKind::CCC_ObjCClassForwardDecl:
     return "ObjCClassForwardDecl";
+  case CCKind::CCC_TopLevelOrExpression:
+    return "ReplTopLevel";
   }
   llvm_unreachable("Invalid CodeCompletionContext::Kind!");
 }
 
 //===----------------------------------------------------------------------===//
 // Code completion string implementation
 //===----------------------------------------------------------------------===//
 

clang/lib/Sema/SemaCodeComplete.cpp

explicit ResultBuilder(Sema &SemaRef, CodeCompletionAllocator &Allocator,
         ObjCImplementation(nullptr) {
     // If this is an Objective-C instance method definition, dig out the
     // corresponding implementation.
     switch (CompletionContext.getKind()) {
     case CodeCompletionContext::CCC_Expression:
     case CodeCompletionContext::CCC_ObjCMessageReceiver:
     case CodeCompletionContext::CCC_ParenthesizedExpression:
     case CodeCompletionContext::CCC_Statement:
+    case CodeCompletionContext::CCC_TopLevelOrExpression:
     case CodeCompletionContext::CCC_Recovery:
       if (ObjCMethodDecl *Method = SemaRef.getCurMethodDecl())
         if (Method->isInstanceMethod())
           if (ObjCInterfaceDecl *Interface = Method->getClassInterface())
             ObjCImplementation = Interface->getImplementation();
       break;
 
     default:
static void AddFunctionSpecifiers(Sema::ParserCompletionContext CCC,
   case Sema::PCC_Template:
     if (LangOpts.CPlusPlus || LangOpts.C99)
       Results.AddResult(Result("inline"));
     break;
 
   case Sema::PCC_ObjCInstanceVariableList:
   case Sema::PCC_Expression:
   case Sema::PCC_Statement:
+  case Sema::PCC_TopLevelOrExpression:
   case Sema::PCC_ForInit:
   case Sema::PCC_Condition:
   case Sema::PCC_RecoveryInFunction:
   case Sema::PCC_Type:
   case Sema::PCC_ParenthesizedExpression:
   case Sema::PCC_LocalDeclarationSpecifiers:
     break;
   }
static bool WantTypesInContext(Sema::ParserCompletionContext CCC,
   case Sema::PCC_ObjCInstanceVariableList:
   case Sema::PCC_Template:
   case Sema::PCC_MemberTemplate:
   case Sema::PCC_Statement:
   case Sema::PCC_RecoveryInFunction:
   case Sema::PCC_Type:
   case Sema::PCC_ParenthesizedExpression:
   case Sema::PCC_LocalDeclarationSpecifiers:
+  case Sema::PCC_TopLevelOrExpression:
     return true;
 
   case Sema::PCC_Expression:
   case Sema::PCC_Condition:
     return LangOpts.CPlusPlus;
 
   case Sema::PCC_ObjCInterface:
   case Sema::PCC_ObjCImplementation:
case Sema::PCC_ObjCImplementation:
     AddFunctionSpecifiers(CCC, SemaRef.getLangOpts(), Results);
     break;
 
   case Sema::PCC_ObjCInstanceVariableList:
     AddObjCVisibilityResults(SemaRef.getLangOpts(), Results, true);
     break;
 
   case Sema::PCC_RecoveryInFunction:
+  case Sema::PCC_TopLevelOrExpression:
   case Sema::PCC_Statement: {
     if (SemaRef.getLangOpts().CPlusPlus11)
       AddUsingAliasResult(Builder, Results);
 
     AddTypedefResult(Results);
 
     if (SemaRef.getLangOpts().CPlusPlus && Results.includeCodePatterns() &&
         SemaRef.getLangOpts().CXXExceptions) {
mapCodeCompletionContext(Sema &S, Sema::ParserCompletionContext PCC) {
   case Sema::PCC_Type:
     return CodeCompletionContext::CCC_Type;
 
   case Sema::PCC_ParenthesizedExpression:
     return CodeCompletionContext::CCC_ParenthesizedExpression;
 
   case Sema::PCC_LocalDeclarationSpecifiers:
     return CodeCompletionContext::CCC_Type;
+  case Sema::PCC_TopLevelOrExpression:
+    return CodeCompletionContext::CCC_TopLevelOrExpression;
   }
 
   llvm_unreachable("Invalid ParserCompletionContext!");
 }
 
 /// If we're in a C++ virtual member function, add completion results
 /// that invoke the functions we override, since it's common to invoke the
 /// overridden function as well as adding new functionality.
void Sema::CodeCompleteOrdinaryName(Scope *S,
   case PCC_Template:
   case PCC_MemberTemplate:
   case PCC_Type:
   case PCC_LocalDeclarationSpecifiers:
     Results.setFilter(&ResultBuilder::IsOrdinaryNonValueName);
     break;
 
   case PCC_Statement:
+  case PCC_TopLevelOrExpression:
   case PCC_ParenthesizedExpression:
   case PCC_Expression:
   case PCC_ForInit:
   case PCC_Condition:
     if (WantTypesInContext(CompletionContext, getLangOpts()))
       Results.setFilter(&ResultBuilder::IsOrdinaryName);
     else
       Results.setFilter(&ResultBuilder::IsOrdinaryNonTypeName);
void Sema::CodeCompleteOrdinaryName(Scope *S,
 
   AddOrdinaryNameResults(CompletionContext, S, *this, Results);
   Results.ExitScope();
 
   switch (CompletionContext) {
   case PCC_ParenthesizedExpression:
   case PCC_Expression:
   case PCC_Statement:
+  case PCC_TopLevelOrExpression:
   case PCC_RecoveryInFunction:
     if (S->getFnParent())
       AddPrettyFunctionResults(getLangOpts(), Results);
     break;
 
   case PCC_Namespace:
   case PCC_Class:
   case PCC_ObjCInterface:

clang/test/CodeCompletion/incrememal-mode-completion-no-error.cpp

+void foo(
+// RUN: %clang_cc1 -fincremental-extensions -fsyntax-only -code-completion-at=%s:%(line-1):9 %s | wc -c | FileCheck %s
+// CHECK: 0

clang/test/CodeCompletion/incremental-top-level.cpp

+int foo = 10;
+f
+// RUN: %clang_cc1 -fincremental-extensions -fsyntax-only -code-completion-at=%s:%(line-1):1 %s | FileCheck %s
+// CHECK: COMPLETION: foo : [#int#]foo

clang/tools/clang-repl/ClangRepl.cpp

 //===--- tools/clang-repl/ClangRepl.cpp - clang-repl - the Clang REPL -----===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file implements a REPL tool on top of clang.
 //
 //===----------------------------------------------------------------------===//
 
 #include "clang/Basic/Diagnostic.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendDiagnostic.h"
+#include "clang/Interpreter/CodeCompletion.h"
 #include "clang/Interpreter/Interpreter.h"
+#include "clang/Sema/CodeCompleteConsumer.h"
 
 #include "llvm/ExecutionEngine/Orc/LLJIT.h"
 #include "llvm/LineEditor/LineEditor.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ManagedStatic.h" // llvm_shutdown
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/TargetSelect.h"
 #include <optional>
if (CI->getDiagnosticOpts().VerifyDiagnostics) {
     Errs = Client->getNumErrors();
 
     // The interpreter expects BeginSourceFile/EndSourceFiles to be balanced.
     Client->BeginSourceFile(CI->getLangOpts(), &CI->getPreprocessor());
   }
   return (Errs || HasError) ? EXIT_FAILURE : EXIT_SUCCESS;
 }
 
+struct ReplListCompleter {
+  clang::IncrementalCompilerBuilder &CB;
+  clang::Interpreter &MainInterp;
+  ReplListCompleter(clang::IncrementalCompilerBuilder &CB,
+                    clang::Interpreter &Interp)
+      : CB(CB), MainInterp(Interp){};
+
+  std::vector<llvm::LineEditor::Completion> operator()(llvm::StringRef Buffer,
+                                                       size_t Pos) const;
+  std::vector<llvm::LineEditor::Completion>
+  operator()(llvm::StringRef Buffer, size_t Pos, llvm::Error &ErrRes) const;
+};
+
+std::vector<llvm::LineEditor::Completion>
+ReplListCompleter::operator()(llvm::StringRef Buffer, size_t Pos) const {
+  auto Err = llvm::Error::success();
+  auto res = (*this)(Buffer, Pos, Err);
+  if (Err)
+    llvm::logAllUnhandledErrors(std::move(Err), llvm::errs(), "error: ");
+  return res;
+}
+
+std::vector<llvm::LineEditor::Completion>
+ReplListCompleter::operator()(llvm::StringRef Buffer, size_t Pos,
+                              llvm::Error &ErrRes) const {
+  std::vector<llvm::LineEditor::Completion> Comps;
+  std::vector<clang::CodeCompletionResult> Results;
+
+  auto CI = CB.CreateCpp();
+  if (auto Err = CI.takeError()) {
+    ErrRes = std::move(Err);
+    return {};
+  }
+
+  size_t Lines =
+      std::count(Buffer.begin(), std::next(Buffer.begin(), Pos), '\n') + 1;
+  auto Interp = clang::Interpreter::create(std::move(*CI));
+
+  if (auto Err = Interp.takeError()) {
+    // log the error and returns an empty vector;
+    ErrRes = std::move(Err);
+
+    return {};
+  }
+
+  codeComplete(
+      const_cast<clang::CompilerInstance *>((*Interp)->getCompilerInstance()),
+      Buffer, Lines, Pos + 1, MainInterp.getCompilerInstance(), Results);
+
+  size_t space_pos = Buffer.rfind(" ");
+  llvm::StringRef s;
+  if (space_pos == llvm::StringRef::npos) {
+    s = Buffer;
+  } else {
+    s = Buffer.substr(space_pos + 1);
+  }
+
+  for (auto c : convertToCodeCompleteStrings(Results)) {
+    if (c.find(s) == 0)
+      Comps.push_back(llvm::LineEditor::Completion(c.substr(s.size()), c));
+  }
+  return Comps;
+}
+
 llvm::ExitOnError ExitOnErr;
 int main(int argc, const char **argv) {
   ExitOnErr.setBanner("clang-repl: ");
   llvm::cl::ParseCommandLineOptions(argc, argv);
 
   llvm::llvm_shutdown_obj Y; // Call llvm_shutdown() on exit.
 
   std::vector<const char *> ClangArgv(ClangArgs.size());
llvm::install_fatal_error_handler(LLVMErrorHandler,
                                     static_cast<void *>(&CI->getDiagnostics()));
 
   // Load any requested plugins.
   CI->LoadRequestedPlugins();
   if (CudaEnabled)
     DeviceCI->LoadRequestedPlugins();
 
   std::unique_ptr<clang::Interpreter> Interp;
+
   if (CudaEnabled) {
     Interp = ExitOnErr(
         clang::Interpreter::createWithCUDA(std::move(CI), std::move(DeviceCI)));
 
     if (CudaPath.empty()) {
       ExitOnErr(Interp->LoadDynamicLibrary("libcudart.so"));
     } else {
       auto CudaRuntimeLibPath = CudaPath + "/lib/libcudart.so";
       ExitOnErr(Interp->LoadDynamicLibrary(CudaRuntimeLibPath.c_str()));
     }
   } else
     Interp = ExitOnErr(clang::Interpreter::create(std::move(CI)));
 
   for (const std::string &input : OptInputs) {
     if (auto Err = Interp->ParseAndExecute(input))
       llvm::logAllUnhandledErrors(std::move(Err), llvm::errs(), "error: ");
   }
 
   bool HasError = false;
 
   if (OptInputs.empty()) {
     llvm::LineEditor LE("clang-repl");
-    // FIXME: Add LE.setListCompleter
     std::string Input;
+    LE.setListCompleter(ReplListCompleter(CB, *Interp));
     while (std::optional<std::string> Line = LE.readLine()) {
       llvm::StringRef L = *Line;
       L = L.trim();
       if (L.endswith("\\")) {
         // FIXME: Support #ifdef X \ ...
         Input += L.drop_back(1);
         LE.setPrompt("clang-repl...   ");
         continue;
       }
 
       Input += L;
-
       if (Input == R"(%quit)") {
         break;
-      } else if (Input == R"(%undo)") {
+      }
+      if (Input == R"(%undo)") {
         if (auto Err = Interp->Undo()) {
           llvm::logAllUnhandledErrors(std::move(Err), llvm::errs(), "error: ");
           HasError = true;
         }
       } else if (Input.rfind("%lib ", 0) == 0) {
         if (auto Err = Interp->LoadDynamicLibrary(Input.data() + 5)) {
           llvm::logAllUnhandledErrors(std::move(Err), llvm::errs(), "error: ");
           HasError = true;

clang/tools/libclang/CIndexCodeCompletion.cpp

 //===- CIndexCodeCompletion.cpp - Code Completion API hooks ---------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements the Clang-C Source Indexing library hooks for
 // code completion.
 //
 //===----------------------------------------------------------------------===//
 
-#include "CIndexer.h"
 #include "CIndexDiagnostic.h"
+#include "CIndexer.h"
 #include "CLog.h"
 #include "CXCursor.h"
 #include "CXSourceLocation.h"
 #include "CXString.h"
 #include "CXTranslationUnit.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclObjC.h"
 #include "clang/AST/Type.h"
 #include "clang/Basic/FileManager.h"
 #include "clang/Basic/SourceManager.h"
 #include "clang/Frontend/ASTUnit.h"
 #include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/FrontendActions.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Sema/Sema.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/Timer.h"
 #include "llvm/Support/raw_ostream.h"
 #include <atomic>
 #include <cstdio>
 #include <cstdlib>
 #include <string>
 
-
 #ifdef UDP_CODE_COMPLETION_LOGGER
 #include "clang/Basic/Version.h"
 #include <arpa/inet.h>
 #include <sys/socket.h>
 #include <sys/types.h>
 #include <unistd.h>
 #endif
 
switch (kind) {
     case CodeCompletionContext::CCC_ObjCInterface:
     case CodeCompletionContext::CCC_ObjCImplementation:
     case CodeCompletionContext::CCC_ObjCClassForwardDecl:
     case CodeCompletionContext::CCC_NewName:
     case CodeCompletionContext::CCC_MacroName:
     case CodeCompletionContext::CCC_PreprocessorExpression:
     case CodeCompletionContext::CCC_PreprocessorDirective:
     case CodeCompletionContext::CCC_Attribute:
+    case CodeCompletionContext::CCC_TopLevelOrExpression:
     case CodeCompletionContext::CCC_TypeQualifiers: {
       //Only Clang results should be accepted, so we'll set all of the other
       //context bits to 0 (i.e. the empty set)
       contexts = CXCompletionContext_Unexposed;
       break;
     }
     case CodeCompletionContext::CCC_Recovery: {
       //We don't know what the current context is, so we'll return unknown

clang/unittests/Interpreter/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   ${LLVM_TARGETS_TO_BUILD}
   Core
   OrcJIT
   Support
   TargetParser
   )
 
 add_clang_unittest(ClangReplInterpreterTests
   IncrementalProcessingTest.cpp
   InterpreterTest.cpp
+  CodeCompletionTest.cpp
   )
 target_link_libraries(ClangReplInterpreterTests PUBLIC
   clangAST
   clangBasic
   clangInterpreter
   clangFrontend
   clangSema
   )
 
 # Exceptions on Windows are not yet supported.
 if(NOT WIN32)
   add_subdirectory(ExceptionTests)
 endif()
 
 export_executable_symbols(ClangReplInterpreterTests)

clang/unittests/Interpreter/CodeCompletionTest.cpp

+#include "clang/Interpreter/CodeCompletion.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Interpreter/Interpreter.h"
+#include "clang/Sema/CodeCompleteConsumer.h"
+#include "llvm/LineEditor/LineEditor.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+using namespace clang;
+namespace {
+auto CB = clang::IncrementalCompilerBuilder();
+
+static std::unique_ptr<Interpreter> createInterpreter() {
+  auto CI = cantFail(CB.CreateCpp());
+  return cantFail(clang::Interpreter::create(std::move(CI)));
+}
+
+static std::vector<std::string> runComp(clang::Interpreter &MainInterp,
+                                        llvm::StringRef Prefix,
+                                        llvm::Error &ErrR) {
+  auto CI = CB.CreateCpp();
+  if (auto Err = CI.takeError()) {
+    ErrR = std::move(Err);
+    return {};
+  }
+
+  auto Interp = clang::Interpreter::create(std::move(*CI));
+  if (auto Err = Interp.takeError()) {
+    // log the error and returns an empty vector;
+    ErrR = std::move(Err);
+
+    return {};
+  }
+
+  std::vector<clang::CodeCompletionResult> Results;
+
+  codeComplete(
+      const_cast<clang::CompilerInstance *>((*Interp)->getCompilerInstance()),
+      Prefix, 1, Prefix.size(), MainInterp.getCompilerInstance(), Results);
+
+  std::vector<std::string> Comps;
+  for (auto c : convertToCodeCompleteStrings(Results)) {
+    if (c.find(Prefix) == 0)
+      Comps.push_back(c.substr(Prefix.size()));
+  }
+
+  return Comps;
+}
+
+TEST(CodeCompletionTest, Sanity) {
+  auto Interp = createInterpreter();
+  if (auto R = Interp->ParseAndExecute("int foo = 12;")) {
+    consumeError(std::move(R));
+    return;
+  }
+  auto Err = llvm::Error::success();
+  auto comps = runComp(*Interp, "f", Err);
+  EXPECT_EQ((size_t)2, comps.size()); // foo and float
+  EXPECT_EQ(comps[0], std::string("oo"));
+  EXPECT_EQ((bool)Err, false);
+}
+
+TEST(CodeCompletionTest, SanityNoneValid) {
+  auto Interp = createInterpreter();
+  if (auto R = Interp->ParseAndExecute("int foo = 12;")) {
+    consumeError(std::move(R));
+    return;
+  }
+  auto Err = llvm::Error::success();
+  auto comps = runComp(*Interp, "babanana", Err);
+  EXPECT_EQ((size_t)0, comps.size()); // foo and float
+  EXPECT_EQ((bool)Err, false);
+}
+
+TEST(CodeCompletionTest, TwoDecls) {
+  auto Interp = createInterpreter();
+  if (auto R = Interp->ParseAndExecute("int application = 12;")) {
+    consumeError(std::move(R));
+    return;
+  }
+  if (auto R = Interp->ParseAndExecute("int apple = 12;")) {
+    consumeError(std::move(R));
+    return;
+  }
+  auto Err = llvm::Error::success();
+  auto comps = runComp(*Interp, "app", Err);
+  EXPECT_EQ((size_t)2, comps.size());
+  EXPECT_EQ((bool)Err, false);
+}
+
+TEST(CodeCompletionTest, CompFunDeclsNoError) {
+  auto Interp = createInterpreter();
+  auto Err = llvm::Error::success();
+  auto comps = runComp(*Interp, "void app(", Err);
+  EXPECT_EQ((bool)Err, false);
+}
+
+} // anonymous namespace