[clangd] Use identifiers in file as completion candidates when build is not ready.

[ioeric]

Summary

o Lex the code to get the identifiers and put them into a "symbol" index.
o Adds a new completion mode without compilation/sema into code completion workflow.
o Make IncludeInserter work even when no compile command is present, by avoiding
inserting non-verbatim headers.

[sammccall]

clangd/CodeComplete.cpp

1141

as discussed offline, I love the lexer approach but returning it as an index seems like the wrong level of abstraction.
These are just strings, and if we want to tweak e.g. the scoring or merging of these we'll be in a better position to do so if we don't pretend they have Symbol semantics (their own signals, ids, etc).
So I'd suggest this fn should return something like StringSet<> or StringMap<unsigned> for counts.

Can we move it to SourceCode.h too?

1141

I don't think we need FileName as a parameter - it shouldn't affect the return value, so we can just use a dummy value.

1188

I'm not sure these changes (mostly to comment formatting) improve clarity.
I think it would be enough to add a comment inside the top of runWithoutCompile saying // Fill in fields normally set by runWithSema() or so

1191

ReplacedRange?

1288

runWithoutSema maybe? - compile is a fine name but this file calls this concept sema for whatever reason

1290

hmm, this function should likely also move to SourceCode, use Offset instead of Position, and move to SourceCode.h. But probably a different patch.

1311

you could just erase the typed filter from the suggestion list.
(It may be a valid word spelled elsewhere, but there's no point suggesting it)

1327

I suspect you want to pass the filtered strings through to mergeResults and have it/CompletionCandidate do the scoring.

clangd/CodeComplete.h

248

It seems a little odd to have a different entrypoint here.
Could we simply indicate this with Preamble=nullptr?

It seems it might even make sense to use the same speculative fuzzyfind?
So I think PCHContainerOperations would be the only odd one out. I'm going to send a patch to remove that anyway :-)

[ioeric]

• address review comments

[ioeric]

• minor cleanup

[ioeric]

Thanks for the review!

clangd/CodeComplete.cpp

1290

ack

1311

This is following conventions of other sources. Both index and sema provide results for fully-typed names. Considering that users might be already used to this, and completion results tend to give reassurance for correctly typed names, I am inclined to keep the typed filter if it's seen somewhere else in the file.

[sammccall]

Very nice! D60503 will conflict, feel free to stall that until this is landed.
On the other hand it will simplify some things, e.g. the prefix is already calculated, and typed scope is available if you want that (no enclosing namespaces though).

clangd/CodeComplete.cpp

185

This is a bit vague - most results are for identifiers in some sense.
Maybe RawIdentifier?

281

move this comment to the member?
The invariant is important to readers, not just callers of the constructor.

1177

rename NBoth -> NSemaAndIndex?

1305

it doesn't make sense to set this here, as long as we're not querying the index.

1311

OK, but we should now be able to just decrement the count for the right entry in the collected identifiers then? rather than carving up the string

1353

add this to the non-sema case too (though the if is always false for now), or add a fixme?
Worried about forgetting this.

1514

why are we not initializing fileproximity in the fallback case?
(To only a single source)

1556

can we have a SymbolOrigin bit for identifiers, and use that instead?

clangd/Headers.cpp

182

nit: i'd move this above the previous case, it seems both more fundamental and cheaper

196

Do we expect this code path to ever be called?
We may want to assert or elog here, depending on how this can be hit.

clangd/SourceCode.cpp

403

why isn't Add just ++Identifiers[Id]?

[ioeric]

• address review comments

clangd/CodeComplete.cpp

1353

Added a FIXME.

clangd/Headers.cpp

196

This can be hit when we start serving index symbols in fallback mode completion. shouldInsertInclude will return false, but the calculation will be attempted regardless. We can probably elog this, but the number of log messages would be O(# of index symbols), which I worry might be too spammy.

[sammccall]

clangd/CodeComplete.cpp

345

Completion.Kind = CompletionItemKind::Text

1317

comment should explain why, rather than what :-)

[ioeric]

• address comments

clangd/ClangdServer.cpp

 #include "index/CanonicalIncludes.h"
 #include "index/FileIndex.h"
 #include "index/Merge.h"
 #include "refactor/Tweak.h"
 #include "clang/Format/Format.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/CompilerInvocation.h"
 #include "clang/Lex/Preprocessor.h"
-#include "clang/Sema/CodeCompleteConsumer.h"
 #include "clang/Tooling/CompilationDatabase.h"
 #include "clang/Tooling/Core/Replacement.h"
 #include "clang/Tooling/Refactoring/RefactoringResultConsumer.h"
 #include "clang/Tooling/Refactoring/Rename/RenamingAction.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/StringRef.h"
void ClangdServer::codeComplete(PathRef File, Position Pos,
   auto FS = FSProvider.getFileSystem();
   auto Task = [Pos, FS, CodeCompleteOpts,
                this](Path File, Callback<CodeCompleteResult> CB,
                      llvm::Expected<InputsAndPreamble> IP) {
     if (!IP)
       return CB(IP.takeError());
     if (isCancelled())
       return CB(llvm::make_error<CancelledError>());
-    if (!IP->Preamble) {
-      vlog("File {0} is not ready for code completion. Enter fallback mode.",
-           File);
-      CodeCompleteResult CCR;
-      CCR.Context = CodeCompletionContext::CCC_Recovery;
-
-      // FIXME: perform simple completion e.g. using identifiers in the current
-      // file and symbols in the index.
-      // FIXME: let clients know that we've entered fallback mode.
-
-      return CB(std::move(CCR));
-    }
 
     llvm::Optional<SpeculativeFuzzyFind> SpecFuzzyFind;
+    if (!IP->Preamble) {
+      // No speculation in Fallback mode, as it's supposed to be much faster
+      // without compiling.
+      vlog("Build for file {0} is not ready. Enter fallback mode.", File);
+    } else {
       if (CodeCompleteOpts.Index && CodeCompleteOpts.SpeculativeIndexRequest) {
         SpecFuzzyFind.emplace();
         {
-        std::lock_guard<std::mutex> Lock(CachedCompletionFuzzyFindRequestMutex);
-        SpecFuzzyFind->CachedReq = CachedCompletionFuzzyFindRequestByFile[File];
+          std::lock_guard<std::mutex> Lock(
+              CachedCompletionFuzzyFindRequestMutex);
+          SpecFuzzyFind->CachedReq =
+              CachedCompletionFuzzyFindRequestByFile[File];
+        }
       }
     }
-
     // FIXME(ibiryukov): even if Preamble is non-null, we may want to check
     // both the old and the new version in case only one of them matches.
     CodeCompleteResult Result = clangd::codeComplete(
         File, IP->Command, IP->Preamble, IP->Contents, Pos, FS,
         CodeCompleteOpts, SpecFuzzyFind ? SpecFuzzyFind.getPointer() : nullptr);
     {
       clang::clangd::trace::Span Tracer("Completion results callback");
       CB(std::move(Result));

clangd/ClangdUnit.cpp

ParsedAST::build(std::unique_ptr<CompilerInvocation> CI,
   // Add IncludeFixer which can recorver diagnostics caused by missing includes
   // (e.g. incomplete type) and attach include insertion fixes to diagnostics.
   llvm::Optional<IncludeFixer> FixIncludes;
   auto BuildDir = VFS->getCurrentWorkingDirectory();
   if (Opts.SuggestMissingIncludes && Index && !BuildDir.getError()) {
     auto Style = getFormatStyleForFile(MainInput.getFile(), Content, VFS.get());
     auto Inserter = std::make_shared<IncludeInserter>(
         MainInput.getFile(), Content, Style, BuildDir.get(),
-        Clang->getPreprocessor().getHeaderSearchInfo());
+        &Clang->getPreprocessor().getHeaderSearchInfo());
     if (Preamble) {
       for (const auto &Inc : Preamble->Includes.MainFileIncludes)
         Inserter->addExisting(Inc);
     }
     FixIncludes.emplace(MainInput.getFile(), Inserter, *Index,
                         /*IndexRequestLimit=*/5);
     ASTDiags.contributeFixes([&FixIncludes](DiagnosticsEngine::Level DiagLevl,
                                             const clang::Diagnostic &Info) {

clangd/CodeComplete.h

struct SpeculativeFuzzyFind {
   /// The actual request used by `codeComplete()`.
   /// Set by `codeComplete()`. This can be used by callers to update cache.
   llvm::Optional<FuzzyFindRequest> NewReq;
   /// The result is consumed by `codeComplete()` if speculation succeeded.
   /// NOTE: the destructor will wait for the async call to finish.
   std::future<SymbolSlab> Result;
 };
 
-/// Get code completions at a specified \p Pos in \p FileName.
+/// Gets code completions at a specified \p Pos in \p FileName.
+///
+/// If \p Preamble is nullptr, this runs code completion without compiling the
+/// code.
+///
 /// If \p SpecFuzzyFind is set, a speculative and asynchronous fuzzy find index
 /// request (based on cached request) will be run before parsing sema. In case
 /// the speculative result is used by code completion (e.g. speculation failed),
 /// the speculative result is not consumed, and `SpecFuzzyFind` is only
 /// destroyed when the async request finishes.
 CodeCompleteResult codeComplete(PathRef FileName,
                                 const tooling::CompileCommand &Command,
                                 const PreambleData *Preamble,
                                 StringRef Contents, Position Pos,
                                 IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS,
                                 CodeCompleteOptions Opts,
                                 SpeculativeFuzzyFind *SpecFuzzyFind = nullptr);
 
 /// Get signature help at a specified \p Pos in \p FileName.
 SignatureHelp signatureHelp(PathRef FileName,
                             const tooling::CompileCommand &Command,
                             const PreambleData *Preamble, StringRef Contents,
                             Position Pos,
                             IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS,
                             const SymbolIndex *Index);
 
 // For index-based completion, we only consider:
 //   * symbols in namespaces or translation unit scopes (e.g. no class
 //     members, no locals)

clangd/CodeComplete.cpp

 #include "Logger.h"
 #include "Quality.h"
 #include "SourceCode.h"
 #include "TUScheduler.h"
 #include "Trace.h"
 #include "URI.h"
 #include "index/Index.h"
 #include "index/Symbol.h"
+#include "index/SymbolOrigin.h"
 #include "clang/AST/Decl.h"
 #include "clang/AST/DeclBase.h"
 #include "clang/Basic/CharInfo.h"
 #include "clang/Basic/LangOptions.h"
 #include "clang/Basic/SourceLocation.h"
 #include "clang/Format/Format.h"
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendActions.h"
for (const auto &Chunk : CCS) {
     default:
       Result += Chunk.Text;
       break;
     }
   }
   return Result;
 }
 
+// Identifier code completion result.
+struct RawIdentifier {
+  llvm::StringRef Name;
+  unsigned References; // # of usages in file.
+};
+
 /// A code completion result, in clang-native form.
 /// It may be promoted to a CompletionItem if it's among the top-ranked results.
 struct CompletionCandidate {
   llvm::StringRef Name; // Used for filtering and sorting.
   // We may have a result from Sema, from the index, or both.
   const CodeCompletionResult *SemaResult = nullptr;
   const Symbol *IndexResult = nullptr;
+  const RawIdentifier *IdentifierResult = nullptr;
   llvm::SmallVector<llvm::StringRef, 1> RankedIncludeHeaders;
 
   // Returns a token identifying the overload set this is part of.
   // 0 indicates it's not part of any overload set.
   size_t overloadSet(const CodeCompleteOptions &Opts) const {
     if (!Opts.BundleOverloads)
       return 0;
     llvm::SmallString<256> Scratch;
#endif
         // This could break #include insertion.
         return llvm::hash_combine(
             (IndexResult->Scope + IndexResult->Name).toStringRef(Scratch),
             headerToInsertIfAllowed(Opts).getValueOr(""));
       default:
         return 0;
       }
     }
-    assert(SemaResult);
+    if (SemaResult) {
       // We need to make sure we're consistent with the IndexResult case!
       const NamedDecl *D = SemaResult->Declaration;
       if (!D || !D->isFunctionOrFunctionTemplate())
         return 0;
       {
         llvm::raw_svector_ostream OS(Scratch);
         D->printQualifiedName(OS);
       }
       return llvm::hash_combine(Scratch,
                                 headerToInsertIfAllowed(Opts).getValueOr(""));
     }
+    assert(IdentifierResult);
+    return 0;
+  }
 
   // The best header to include if include insertion is allowed.
   llvm::Optional<llvm::StringRef>
   headerToInsertIfAllowed(const CodeCompleteOptions &Opts) const {
     if (Opts.InsertIncludes == CodeCompleteOptions::NeverInsert ||
         RankedIncludeHeaders.empty())
       return None;
     if (SemaResult && SemaResult->Declaration) {

 // Assembles a code completion out of a bundle of >=1 completion candidates.
 // Many of the expensive strings are only computed at this point, once we know
 // the candidate bundle is going to be returned.
 //
 // Many fields are the same for all candidates in a bundle (e.g. name), and are
 // computed from the first candidate, in the constructor.
 // Others vary per candidate, so add() must be called for remaining candidates.
 struct CodeCompletionBuilder {
-  CodeCompletionBuilder(ASTContext &ASTCtx, const CompletionCandidate &C,
+  CodeCompletionBuilder(ASTContext *ASTCtx, const CompletionCandidate &C,
                         CodeCompletionString *SemaCCS,
                         llvm::ArrayRef<std::string> QueryScopes,
                         const IncludeInserter &Includes,
                         llvm::StringRef FileName,
                         CodeCompletionContext::Kind ContextKind,
                         const CodeCompleteOptions &Opts)
       : ASTCtx(ASTCtx), ExtractDocumentation(Opts.IncludeComments),
         EnableFunctionArgSnippets(Opts.EnableFunctionArgSnippets) {
     add(C, SemaCCS);
     if (C.SemaResult) {
+      assert(ASTCtx);
       Completion.Origin |= SymbolOrigin::AST;
       Completion.Name = llvm::StringRef(SemaCCS->getTypedText());
       if (Completion.Scope.empty()) {
         if ((C.SemaResult->Kind == CodeCompletionResult::RK_Declaration) ||
             (C.SemaResult->Kind == CodeCompletionResult::RK_Pattern))
           if (const auto *D = C.SemaResult->getDeclaration())
             if (const auto *ND = dyn_cast<NamedDecl>(D))
               Completion.Scope =
                   splitQualifiedName(printQualifiedName(*ND)).first;
       }
       Completion.Kind = toCompletionItemKind(
           C.SemaResult->Kind, C.SemaResult->Declaration, ContextKind);
       // Sema could provide more info on whether the completion was a file or
       // folder.
       if (Completion.Kind == CompletionItemKind::File &&
           Completion.Name.back() == '/')
         Completion.Kind = CompletionItemKind::Folder;
       for (const auto &FixIt : C.SemaResult->FixIts) {
-        Completion.FixIts.push_back(
-            toTextEdit(FixIt, ASTCtx.getSourceManager(), ASTCtx.getLangOpts()));
+        Completion.FixIts.push_back(toTextEdit(
+            FixIt, ASTCtx->getSourceManager(), ASTCtx->getLangOpts()));
       }
       llvm::sort(Completion.FixIts, [](const TextEdit &X, const TextEdit &Y) {
         return std::tie(X.range.start.line, X.range.start.character) <
                std::tie(Y.range.start.line, Y.range.start.character);
       });
       Completion.Deprecated |=
           (C.SemaResult->Availability == CXAvailability_Deprecated);
     }
if (C.IndexResult) {
           if (Qualifier.consume_front(Scope) &&
               Qualifier.size() < ShortestQualifier.size())
             ShortestQualifier = Qualifier;
         }
         Completion.RequiredQualifier = ShortestQualifier;
       }
       Completion.Deprecated |= (C.IndexResult->Flags & Symbol::Deprecated);
     }
+    if (C.IdentifierResult) {
+      Completion.Origin |= SymbolOrigin::Identifier;
+      Completion.Name = C.IdentifierResult->Name;
+    }
 
     // Turn absolute path into a literal string that can be #included.
     auto Inserted = [&](llvm::StringRef Header)
         -> llvm::Expected<std::pair<std::string, bool>> {
       auto ResolvedDeclaring =
           toHeaderFile(C.IndexResult->CanonicalDeclaration.FileURI, FileName);
       if (!ResolvedDeclaring)
         return ResolvedDeclaring.takeError();
if (C.SemaResult) {
       S.Signature = C.IndexResult->Signature;
       S.SnippetSuffix = C.IndexResult->CompletionSnippetSuffix;
       S.ReturnType = C.IndexResult->ReturnType;
     }
     if (ExtractDocumentation && Completion.Documentation.empty()) {
       if (C.IndexResult)
         Completion.Documentation = C.IndexResult->Documentation;
       else if (C.SemaResult)
-        Completion.Documentation = getDocComment(ASTCtx, *C.SemaResult,
+        Completion.Documentation = getDocComment(*ASTCtx, *C.SemaResult,
                                                  /*CommentsFromHeader=*/false);
     }
   }
 
   CodeCompletion build() {
     Completion.ReturnType = summarizeReturnType();
     Completion.Signature = summarizeSignature();
     Completion.SnippetSuffix = summarizeSnippet();
private:
 
   std::string summarizeSignature() const {
     if (auto *Signature = onlyValue<&BundledEntry::Signature>())
       return *Signature;
     // All bundles are function calls.
     return "(…)";
   }
 
-  ASTContext &ASTCtx;
+  // ASTCtx can be nullptr if not run with sema.
+  ASTContext *ASTCtx;
   CodeCompletion Completion;
   llvm::SmallVector<BundledEntry, 1> Bundled;
   bool ExtractDocumentation;
   bool EnableFunctionArgSnippets;
 };
 
 // Determine the symbol ID for a Sema code completion result, if possible.
 llvm::Optional<SymbolID> getSymbolID(const CodeCompletionResult &R,
FuzzyFindRequest speculativeFuzzyFindRequestForCompletion(
     FuzzyFindRequest CachedReq, const CompletionPrefix &HeuristicPrefix) {
   CachedReq.Query = HeuristicPrefix.Name;
   return CachedReq;
 }
 
 // Runs Sema-based (AST) and Index-based completion, returns merged results.
 //
 // There are a few tricky considerations:
 //   - the AST provides information needed for the index query (e.g. which
 //     namespaces to search in). So Sema must start first.
 //   - we only want to return the top results (Opts.Limit).
 //     Building CompletionItems for everything else is wasteful, so we want to
 //     preserve the "native" format until we're done with scoring.
 //   - the data underlying Sema completion items is owned by the AST and various
 //     other arenas, which must stay alive for us to build CompletionItems.
 //   - we may get duplicate results from Sema and the Index, we need to merge.
 //

 class CodeCompleteFlow {
   PathRef FileName;
   IncludeStructure Includes;           // Complete once the compiler runs.
   SpeculativeFuzzyFind *SpecFuzzyFind; // Can be nullptr.
   const CodeCompleteOptions &Opts;
 
   // Sema takes ownership of Recorder. Recorder is valid until Sema cleanup.
   CompletionRecorder *Recorder = nullptr;
-  int NSema = 0, NIndex = 0, NBoth = 0; // Counters for logging.
-  bool Incomplete = false;       // Would more be available with a higher limit?
+  CodeCompletionContext::Kind CCContextKind = CodeCompletionContext::CCC_Other;
+  // Counters for logging.
+  int NSema = 0, NIndex = 0, NSemaAndIndex = 0, NIdent = 0;
+  bool Incomplete = false; // Would more be available with a higher limit?
   CompletionPrefix HeuristicPrefix;
   llvm::Optional<FuzzyMatcher> Filter;  // Initialized once Sema runs.
+  Range ReplacedRange;
   std::vector<std::string> QueryScopes; // Initialized once Sema runs.
   // Initialized once QueryScopes is initialized, if there are scopes.
   llvm::Optional<ScopeDistance> ScopeProximity;
   llvm::Optional<OpaqueType> PreferredType; // Initialized once Sema runs.
   // Whether to query symbols from any scope. Initialized once Sema runs.
   bool AllScopes = false;
   // Include-insertion and proximity scoring rely on the include structure.
   // This is available after Sema has run.
   llvm::Optional<IncludeInserter> Inserter;  // Available during runWithSema.
   llvm::Optional<URIDistance> FileProximity; // Initialized once Sema runs.
   /// Speculative request based on the cached request and the filter text before
   /// the cursor.
   /// Initialized right before sema run. This is only set if `SpecFuzzyFind` is
   /// set and contains a cached request.
   llvm::Optional<FuzzyFindRequest> SpecReq;
 
 public:
   // A CodeCompleteFlow object is only useful for calling run() exactly once.
CodeCompleteResult run(const SemaCompleteInput &SemaCCInput) && {
     }
 
     // We run Sema code completion first. It builds an AST and calculates:
     //   - completion results based on the AST.
     //   - partial identifier and context. We need these for the index query.
     CodeCompleteResult Output;
     auto RecorderOwner = llvm::make_unique<CompletionRecorder>(Opts, [&]() {
       assert(Recorder && "Recorder is not set");
+      CCContextKind = Recorder->CCContext.getKind();
       auto Style = getFormatStyleForFile(
           SemaCCInput.FileName, SemaCCInput.Contents, SemaCCInput.VFS.get());
       // If preprocessor was run, inclusions from preprocessor callback should
       // already be added to Includes.
       Inserter.emplace(
           SemaCCInput.FileName, SemaCCInput.Contents, Style,
           SemaCCInput.Command.Directory,
-          Recorder->CCSema->getPreprocessor().getHeaderSearchInfo());
+          &Recorder->CCSema->getPreprocessor().getHeaderSearchInfo());
       for (const auto &Inc : Includes.MainFileIncludes)
         Inserter->addExisting(Inc);
 
       // Most of the cost of file proximity is in initializing the FileDistance
       // structures based on the observed includes, once per query. Conceptually
       // that happens here (though the per-URI-scheme initialization is lazy).
       // The per-result proximity scoring is (amortized) very cheap.
       FileDistanceOptions ProxOpts{}; // Use defaults.
auto RecorderOwner = llvm::make_unique<CompletionRecorder>(Opts, [&]() {
         if (Entry.getValue() > 0)
           Source.MaxUpTraversals = 1;
       }
       FileProximity.emplace(ProxSources, ProxOpts);
 
       Output = runWithSema();
       Inserter.reset(); // Make sure this doesn't out-live Clang.
       SPAN_ATTACH(Tracer, "sema_completion_kind",
-                  getCompletionKindString(Recorder->CCContext.getKind()));
+                  getCompletionKindString(CCContextKind));
       log("Code complete: sema context {0}, query scopes [{1}] (AnyScope={2}), "
           "expected type {3}",
-          getCompletionKindString(Recorder->CCContext.getKind()),
+          getCompletionKindString(CCContextKind),
           llvm::join(QueryScopes.begin(), QueryScopes.end(), ","), AllScopes,
           PreferredType ? Recorder->CCContext.getPreferredType().getAsString()
                         : "<none>");
     });
 
     Recorder = RecorderOwner.get();
 
     semaCodeComplete(std::move(RecorderOwner), Opts.getClangCompleteOpts(),
                      SemaCCInput, &Includes);
 
     SPAN_ATTACH(Tracer, "sema_results", NSema);
     SPAN_ATTACH(Tracer, "index_results", NIndex);
-    SPAN_ATTACH(Tracer, "merged_results", NBoth);
+    SPAN_ATTACH(Tracer, "merged_results", NSemaAndIndex);
+    SPAN_ATTACH(Tracer, "identifier_results", NIdent);
     SPAN_ATTACH(Tracer, "returned_results", int64_t(Output.Completions.size()));
     SPAN_ATTACH(Tracer, "incomplete", Output.HasMore);
     log("Code complete: {0} results from Sema, {1} from Index, "
-        "{2} matched, {3} returned{4}.",
-        NSema, NIndex, NBoth, Output.Completions.size(),
+        "{2} matched, {3} from identifiers, {4} returned{5}.",
+        NSema, NIndex, NSemaAndIndex, NIdent, Output.Completions.size(),
         Output.HasMore ? " (incomplete)" : "");
     assert(!Opts.Limit || Output.Completions.size() <= Opts.Limit);
     // We don't assert that isIncomplete means we hit a limit.
     // Indexes may choose to impose their own limits even if we don't have one.
     return Output;
   }
 
+  CodeCompleteResult
+  runWithoutSema(llvm::StringRef Content, size_t Offset,
+                 llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS) && {
+    auto CCPrefix = guessCompletionPrefix(Content, Offset);
+    // Fill in fields normally set by runWithSema()
+    CCContextKind = CodeCompletionContext::CCC_Recovery;
+    Filter = FuzzyMatcher(CCPrefix.Name);
+    auto Pos = offsetToPosition(Content, Offset);
+    ReplacedRange.start = ReplacedRange.end = Pos;
+    ReplacedRange.start.character -= CCPrefix.Name.size();
+
+    llvm::StringMap<SourceParams> ProxSources;
+    ProxSources[FileName].Cost = 0;
+    FileProximity.emplace(ProxSources);
+
+    // FIXME: collect typed scope specifier and potentially parse the enclosing
+    // namespaces.
+    // FIXME: initialize ScopeProximity when scopes are added.
+
+    auto Style = getFormatStyleForFile(FileName, Content, VFS.get());
+    // This will only insert verbatim headers.
+    Inserter.emplace(FileName, Content, Style,
+                     /*BuildDir=*/"", /*HeaderSearchInfo=*/nullptr);
+
+    auto Identifiers = collectIdentifiers(Content, Style);
+    std::vector<RawIdentifier> IdentifierResults;
+    for (const auto &IDAndCount : Identifiers) {
+      RawIdentifier ID;
+      ID.Name = IDAndCount.first();
+      ID.References = IDAndCount.second;
+      if (ID.Name == CCPrefix.Name) // Decrement typed filter count.
+        --ID.References;
+      if (ID.References > 0)
+        IdentifierResults.push_back(std::move(ID));
+    }
+
+    // FIXME: add results from Opts.Index when we know more about scopes (e.g.
+    // typed scope specifier).
+    return toCodeCompleteResult(mergeResults(
+        /*SemaResults=*/{}, /*IndexResults*/ {}, IdentifierResults));
+  }
+
 private:
   // This is called by run() once Sema code completion is done, but before the
   // Sema data structures are torn down. It does all the real work.
   CodeCompleteResult runWithSema() {
     const auto &CodeCompletionRange = CharSourceRange::getCharRange(
         Recorder->CCSema->getPreprocessor().getCodeCompletionTokenRange());
-    Range TextEditRange;
     // When we are getting completions with an empty identifier, for example
     //    std::vector<int> asdf;
     //    asdf.^;
     // Then the range will be invalid and we will be doing insertion, use
     // current cursor position in such cases as range.
     if (CodeCompletionRange.isValid()) {
-      TextEditRange = halfOpenToRange(Recorder->CCSema->getSourceManager(),
+      ReplacedRange = halfOpenToRange(Recorder->CCSema->getSourceManager(),
                                       CodeCompletionRange);
     } else {
       const auto &Pos = sourceLocToPosition(
           Recorder->CCSema->getSourceManager(),
           Recorder->CCSema->getPreprocessor().getCodeCompletionLoc());
-      TextEditRange.start = TextEditRange.end = Pos;
+      ReplacedRange.start = ReplacedRange.end = Pos;
     }
     Filter = FuzzyMatcher(
         Recorder->CCSema->getPreprocessor().getCodeCompletionFilter());
     std::tie(QueryScopes, AllScopes) = getQueryScopes(
         Recorder->CCContext, *Recorder->CCSema, HeuristicPrefix, Opts);
     if (!QueryScopes.empty())
       ScopeProximity.emplace(QueryScopes);
     PreferredType =
         OpaqueType::fromType(Recorder->CCSema->getASTContext(),
                              Recorder->CCContext.getPreferredType());
     // Sema provides the needed context to query the index.
     // FIXME: in addition to querying for extra/overlapping symbols, we should
     //        explicitly request symbols corresponding to Sema results.
     //        We can use their signals even if the index can't suggest them.
     // We must copy index results to preserve them, but there are at most Limit.
     auto IndexResults = (Opts.Index && allowIndex(Recorder->CCContext))
                             ? queryIndex()
                             : SymbolSlab();
     trace::Span Tracer("Populate CodeCompleteResult");
     // Merge Sema and Index results, score them, and pick the winners.
-    auto Top = mergeResults(Recorder->Results, IndexResults);
+    auto Top =
+        mergeResults(Recorder->Results, IndexResults, /*Identifiers*/ {});
+    return toCodeCompleteResult(Top);
+  }
+
+  CodeCompleteResult
+  toCodeCompleteResult(const std::vector<ScoredBundle> &Scored) {
     CodeCompleteResult Output;
 
     // Convert the results to final form, assembling the expensive strings.
-    for (auto &C : Top) {
+    for (auto &C : Scored) {
       Output.Completions.push_back(toCodeCompletion(C.first));
       Output.Completions.back().Score = C.second;
-      Output.Completions.back().CompletionTokenRange = TextEditRange;
+      Output.Completions.back().CompletionTokenRange = ReplacedRange;
     }
     Output.HasMore = Incomplete;
-    Output.Context = Recorder->CCContext.getKind();
-
+    Output.Context = CCContextKind;
     return Output;
   }
 
   SymbolSlab queryIndex() {
     trace::Span Tracer("Query index");
     SPAN_ATTACH(Tracer, "limit", int64_t(Opts.Limit));
 
     // Build the query.
SymbolSlab queryIndex() {
     SymbolSlab::Builder ResultsBuilder;
     if (Opts.Index->fuzzyFind(
             Req, [&](const Symbol &Sym) { ResultsBuilder.insert(Sym); }))
       Incomplete = true;
     return std::move(ResultsBuilder).build();
   }
 
   // Merges Sema and Index results where possible, to form CompletionCandidates.
+  // \p Identifiers is raw idenfiers that can also be completion condidates.
+  // Identifiers are not merged with results from index or sema.
   // Groups overloads if desired, to form CompletionCandidate::Bundles. The
   // bundles are scored and top results are returned, best to worst.
   std::vector<ScoredBundle>
   mergeResults(const std::vector<CodeCompletionResult> &SemaResults,
-               const SymbolSlab &IndexResults) {
+               const SymbolSlab &IndexResults,
+               const std::vector<RawIdentifier> &IdentifierResults) {
     trace::Span Tracer("Merge and score results");
     std::vector<CompletionCandidate::Bundle> Bundles;
     llvm::DenseMap<size_t, size_t> BundleLookup;
     auto AddToBundles = [&](const CodeCompletionResult *SemaResult,
-                            const Symbol *IndexResult) {
+                            const Symbol *IndexResult,
+                            const RawIdentifier *IdentifierResult = nullptr) {
       CompletionCandidate C;
       C.SemaResult = SemaResult;
       C.IndexResult = IndexResult;
-      if (C.IndexResult)
+      C.IdentifierResult = IdentifierResult;
+      if (C.IndexResult) {
+        C.Name = IndexResult->Name;
         C.RankedIncludeHeaders = getRankedIncludes(*C.IndexResult);
-      C.Name = IndexResult ? IndexResult->Name : Recorder->getName(*SemaResult);
+      } else if (C.SemaResult) {
+        C.Name = Recorder->getName(*SemaResult);
+      } else {
+        assert(IdentifierResult);
+        C.Name = IdentifierResult->Name;
+      }
       if (auto OverloadSet = C.overloadSet(Opts)) {
         auto Ret = BundleLookup.try_emplace(OverloadSet, Bundles.size());
         if (Ret.second)
           Bundles.emplace_back();
         Bundles[Ret.first->second].push_back(std::move(C));
       } else {
         Bundles.emplace_back();
         Bundles.back().push_back(std::move(C));
       }
     };
     llvm::DenseSet<const Symbol *> UsedIndexResults;
     auto CorrespondingIndexResult =
         [&](const CodeCompletionResult &SemaResult) -> const Symbol * {
       if (auto SymID =
               getSymbolID(SemaResult, Recorder->CCSema->getSourceManager())) {
         auto I = IndexResults.find(*SymID);
         if (I != IndexResults.end()) {
           UsedIndexResults.insert(&*I);
           return &*I;
         }
       }
       return nullptr;
     };
     // Emit all Sema results, merging them with Index results if possible.
-    for (auto &SemaResult : Recorder->Results)
+    for (auto &SemaResult : SemaResults)
       AddToBundles(&SemaResult, CorrespondingIndexResult(SemaResult));
     // Now emit any Index-only results.
     for (const auto &IndexResult : IndexResults) {
       if (UsedIndexResults.count(&IndexResult))
         continue;
       AddToBundles(/*SemaResult=*/nullptr, &IndexResult);
     }
+    // Emit identifier results.
+    for (const auto &Ident : IdentifierResults)
+      AddToBundles(/*SemaResult=*/nullptr, /*IndexResult=*/nullptr, &Ident);
     // We only keep the best N results at any time, in "native" format.
     TopN<ScoredBundle, ScoredBundleGreater> Top(
         Opts.Limit == 0 ? std::numeric_limits<size_t>::max() : Opts.Limit);
     for (auto &Bundle : Bundles)
       addCandidate(Top, std::move(Bundle));
     return std::move(Top).items();
   }
 
   llvm::Optional<float> fuzzyScore(const CompletionCandidate &C) {
     // Macros can be very spammy, so we only support prefix completion.
     // We won't end up with underfull index results, as macros are sema-only.
     if (C.SemaResult && C.SemaResult->Kind == CodeCompletionResult::RK_Macro &&
         !C.Name.startswith_lower(Filter->pattern()))
       return None;
     return Filter->match(C.Name);
   }
 
   // Scores a candidate and adds it to the TopN structure.
   void addCandidate(TopN<ScoredBundle, ScoredBundleGreater> &Candidates,
                     CompletionCandidate::Bundle Bundle) {
     SymbolQualitySignals Quality;
     SymbolRelevanceSignals Relevance;
-    Relevance.Context = Recorder->CCContext.getKind();
+    Relevance.Context = CCContextKind;
     Relevance.Query = SymbolRelevanceSignals::CodeComplete;
     Relevance.FileProximityMatch = FileProximity.getPointer();
     if (ScopeProximity)
       Relevance.ScopeProximityMatch = ScopeProximity.getPointer();
     if (PreferredType)
       Relevance.HadContextType = true;
 
     auto &First = Bundle.front();
     if (auto FuzzyScore = fuzzyScore(First))
       Relevance.NameMatch = *FuzzyScore;
for (const auto &Candidate : Bundle) {
                   Recorder->CCSema->getASTContext(), *Candidate.SemaResult)) {
             Relevance.HadSymbolType |= true;
             if (PreferredType == CompletionType)
               Relevance.TypeMatchesPreferred = true;
           }
         }
         Origin |= SymbolOrigin::AST;
       }
+      if (Candidate.IdentifierResult) {
+        Quality.References = Candidate.IdentifierResult->References;
+        Relevance.Scope = SymbolRelevanceSignals::FileScope;
+        Origin |= SymbolOrigin::Identifier;
+      }
     }
 
     CodeCompletion::Scores Scores;
     Scores.Quality = Quality.evaluate();
     Scores.Relevance = Relevance.evaluate();
     Scores.Total = evaluateSymbolAndRelevance(Scores.Quality, Scores.Relevance);
     // NameMatch is in fact a multiplier on total score, so rescoring is sound.
     Scores.ExcludingName = Relevance.NameMatch
                                ? Scores.Total / Relevance.NameMatch
                                : Scores.Quality;
 
     dlog("CodeComplete: {0} ({1}) = {2}\n{3}{4}\n", First.Name,
          llvm::to_string(Origin), Scores.Total, llvm::to_string(Quality),
          llvm::to_string(Relevance));
 
     NSema += bool(Origin & SymbolOrigin::AST);
     NIndex += FromIndex;
-    NBoth += bool(Origin & SymbolOrigin::AST) && FromIndex;
+    NSemaAndIndex += bool(Origin & SymbolOrigin::AST) && FromIndex;
+    NIdent += bool(Origin & SymbolOrigin::Identifier);
     if (Candidates.push({std::move(Bundle), Scores}))
       Incomplete = true;
   }
 
   CodeCompletion toCodeCompletion(const CompletionCandidate::Bundle &Bundle) {
     llvm::Optional<CodeCompletionBuilder> Builder;
     for (const auto &Item : Bundle) {
       CodeCompletionString *SemaCCS =
           Item.SemaResult ? Recorder->codeCompletionString(*Item.SemaResult)
                           : nullptr;
       if (!Builder)
-        Builder.emplace(Recorder->CCSema->getASTContext(), Item, SemaCCS,
-                        QueryScopes, *Inserter, FileName,
-                        Recorder->CCContext.getKind(), Opts);
+        Builder.emplace(Recorder ? &Recorder->CCSema->getASTContext() : nullptr,
+                        Item, SemaCCS, QueryScopes, *Inserter, FileName,
+                        CCContextKind, Opts);
       else
         Builder->add(Item, SemaCCS);
     }
     return Builder->build();
   }
 };
 
 template <class T> bool isExplicitTemplateSpecialization(const NamedDecl &ND) {
codeComplete(PathRef FileName, const tooling::CompileCommand &Command,
              const PreambleData *Preamble, llvm::StringRef Contents,
              Position Pos, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS,
              CodeCompleteOptions Opts, SpeculativeFuzzyFind *SpecFuzzyFind) {
   auto Offset = positionToOffset(Contents, Pos);
   if (!Offset) {
     elog("Code completion position was invalid {0}", Offset.takeError());
     return CodeCompleteResult();
   }
-  return CodeCompleteFlow(FileName,
-                          Preamble ? Preamble->Includes : IncludeStructure(),
-                          SpecFuzzyFind, Opts)
-      .run({FileName, Command, Preamble, Contents, *Offset, VFS});
+  auto Flow = CodeCompleteFlow(
+      FileName, Preamble ? Preamble->Includes : IncludeStructure(),
+      SpecFuzzyFind, Opts);
+  return Preamble ? std::move(Flow).run(
+                        {FileName, Command, Preamble, Contents, *Offset, VFS})
+                  : std::move(Flow).runWithoutSema(Contents, *Offset, VFS);
 }
 
 SignatureHelp signatureHelp(PathRef FileName,
                             const tooling::CompileCommand &Command,
                             const PreambleData *Preamble,
                             llvm::StringRef Contents, Position Pos,
                             llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> VFS,
                             const SymbolIndex *Index) {

clangd/Headers.h

 
 /// Returns a PPCallback that visits all inclusions in the main file.
 std::unique_ptr<PPCallbacks>
 collectIncludeStructureCallback(const SourceManager &SM, IncludeStructure *Out);
 
 // Calculates insertion edit for including a new header in a file.
 class IncludeInserter {
 public:
+  // If \p HeaderSearchInfo is nullptr (e.g. when compile command is
+  // infeasible), this will only try to insert verbatim headers, and
+  // include path of non-verbatim header will not be shortened.
   IncludeInserter(StringRef FileName, StringRef Code,
                   const format::FormatStyle &Style, StringRef BuildDir,
-                  HeaderSearch &HeaderSearchInfo)
+                  HeaderSearch *HeaderSearchInfo)
       : FileName(FileName), Code(Code), BuildDir(BuildDir),
         HeaderSearchInfo(HeaderSearchInfo),
         Inserter(FileName, Code, Style.IncludeStyle) {}
 
   void addExisting(const Inclusion &Inc);
 
   /// Checks whether to add an #include of the header into \p File.
   /// An #include will not be added if:
public:
   /// Calculates an edit that inserts \p VerbatimHeader into code. If the header
   /// is already included, this returns None.
   llvm::Optional<TextEdit> insert(llvm::StringRef VerbatimHeader) const;
 
 private:
   StringRef FileName;
   StringRef Code;
   StringRef BuildDir;
-  HeaderSearch &HeaderSearchInfo;
+  HeaderSearch *HeaderSearchInfo = nullptr;
   llvm::StringSet<> IncludedHeaders; // Both written and resolved.
   tooling::HeaderIncludes Inserter;  // Computers insertion replacement.
 };
 
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_HEADERS_H

clangd/Headers.cpp

if (!Inc.Resolved.empty())
     IncludedHeaders.insert(Inc.Resolved);
 }
 
 /// FIXME(ioeric): we might not want to insert an absolute include path if the
 /// path is not shortened.
 bool IncludeInserter::shouldInsertInclude(
     const HeaderFile &DeclaringHeader, const HeaderFile &InsertedHeader) const {
   assert(DeclaringHeader.valid() && InsertedHeader.valid());
+  if (!HeaderSearchInfo && !InsertedHeader.Verbatim)
+    return false;
   if (FileName == DeclaringHeader.File || FileName == InsertedHeader.File)
     return false;
   auto Included = [&](llvm::StringRef Header) {
     return IncludedHeaders.find(Header) != IncludedHeaders.end();
   };
   return !Included(DeclaringHeader.File) && !Included(InsertedHeader.File);
 }
 
 std::string
 IncludeInserter::calculateIncludePath(const HeaderFile &DeclaringHeader,
                                       const HeaderFile &InsertedHeader) const {
   assert(DeclaringHeader.valid() && InsertedHeader.valid());
   if (InsertedHeader.Verbatim)
     return InsertedHeader.File;
   bool IsSystem = false;
-  std::string Suggested = HeaderSearchInfo.suggestPathToFileForDiagnostics(
+  if (!HeaderSearchInfo)
+    return "\"" + InsertedHeader.File + "\"";
+  std::string Suggested = HeaderSearchInfo->suggestPathToFileForDiagnostics(
       InsertedHeader.File, BuildDir, &IsSystem);
   if (IsSystem)
     Suggested = "<" + Suggested + ">";
   else
     Suggested = "\"" + Suggested + "\"";
   return Suggested;
 }
 

clangd/SourceCode.h

format::FormatStyle getFormatStyleForFile(llvm::StringRef File,
                                           llvm::StringRef Content,
                                           llvm::vfs::FileSystem *FS);
 
 // Cleanup and format the given replacements.
 llvm::Expected<tooling::Replacements>
 cleanupAndFormat(StringRef Code, const tooling::Replacements &Replaces,
                  const format::FormatStyle &Style);
 
+/// Collects identifiers with counts in the source code.
+llvm::StringMap<unsigned> collectIdentifiers(llvm::StringRef Content,
+                                             const format::FormatStyle &Style);
+
 } // namespace clangd
 } // namespace clang
 #endif

clangd/SourceCode.cpp

 cleanupAndFormat(StringRef Code, const tooling::Replacements &Replaces,
                  const format::FormatStyle &Style) {
   auto CleanReplaces = cleanupAroundReplacements(Code, Replaces, Style);
   if (!CleanReplaces)
     return CleanReplaces;
   return formatReplacements(Code, std::move(*CleanReplaces), Style);
 }
 
+llvm::StringMap<unsigned> collectIdentifiers(llvm::StringRef Content,
+                                             const format::FormatStyle &Style) {
+  SourceManagerForFile FileSM("dummy.cpp", Content);
+  auto &SM = FileSM.get();
+  auto FID = SM.getMainFileID();
+  Lexer Lex(FID, SM.getBuffer(FID), SM, format::getFormattingLangOpts(Style));
+  Token Tok;
+
+  llvm::StringMap<unsigned> Identifiers;
+  while (!Lex.LexFromRawLexer(Tok)) {
+    switch (Tok.getKind()) {
+    case tok::identifier:
+      ++Identifiers[Tok.getIdentifierInfo()->getName()];
+      break;
+    case tok::raw_identifier:
+      ++Identifiers[Tok.getRawIdentifier()];
+      break;
+    default:
+      continue;
+    }
+  }
+  return Identifiers;
+}
+
 } // namespace clangd
 } // namespace clang

clangd/index/SymbolOrigin.h

 namespace clang {
 namespace clangd {
 
 // Describes the source of information about a symbol.
 // Mainly useful for debugging, e.g. understanding code completion reuslts.
 // This is a bitfield as information can be combined from several sources.
 enum class SymbolOrigin : uint8_t {
   Unknown = 0,
-  AST = 1 << 0,     // Directly from the AST (indexes should not set this).
-  Dynamic = 1 << 1, // From the dynamic index of opened files.
-  Static = 1 << 2,  // From the static, externally-built index.
-  Merge = 1 << 3,   // A non-trivial index merge was performed.
+  AST = 1 << 0,        // Directly from the AST (indexes should not set this).
+  Dynamic = 1 << 1,    // From the dynamic index of opened files.
+  Static = 1 << 2,     // From the static, externally-built index.
+  Merge = 1 << 3,      // A non-trivial index merge was performed.
+  Identifier = 1 << 4, // Raw identifiers in file.
   // Remaining bits reserved for index implementations.
 };
 
 inline SymbolOrigin operator|(SymbolOrigin A, SymbolOrigin B) {
   return static_cast<SymbolOrigin>(static_cast<uint8_t>(A) |
                                    static_cast<uint8_t>(B));
 }
 inline SymbolOrigin &operator|=(SymbolOrigin &A, SymbolOrigin B) {

clangd/index/SymbolOrigin.cpp

 //===--- SymbolOrigin.cpp ----------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "SymbolOrigin.h"
 
 namespace clang {
 namespace clangd {
 
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, SymbolOrigin O) {
   if (O == SymbolOrigin::Unknown)
     return OS << "unknown";
-  constexpr static char Sigils[] = "ADSM4567";
+  constexpr static char Sigils[] = "ADSMI567";
   for (unsigned I = 0; I < sizeof(Sigils); ++I)
     if (static_cast<uint8_t>(O) & 1u << I)
       OS << Sigils[I];
   return OS;
 }
 
 } // namespace clangd
 } // namespace clang

unittests/clangd/ClangdTests.cpp

TEST_F(ClangdVFSTest, InvalidCompileCommand) {
 
   // Clang can't parse command args in that case, but we shouldn't crash.
   runAddDocument(Server, FooCpp, "int main() {}");
 
   EXPECT_EQ(runDumpAST(Server, FooCpp), "<no-ast>");
   EXPECT_ERROR(runLocateSymbolAt(Server, FooCpp, Position()));
   EXPECT_ERROR(runFindDocumentHighlights(Server, FooCpp, Position()));
   EXPECT_ERROR(runRename(Server, FooCpp, Position(), "new_name"));
-  // FIXME: codeComplete and signatureHelp should also return errors when they
-  // can't parse the file.
+  // Identifier-based fallback completion.
   EXPECT_THAT(cantFail(runCodeComplete(Server, FooCpp, Position(),
                                        clangd::CodeCompleteOptions()))
                   .Completions,
-              IsEmpty());
+              ElementsAre(Field(&CodeCompletion::Name, "int"),
+                          Field(&CodeCompletion::Name, "main")));
   auto SigHelp = runSignatureHelp(Server, FooCpp, Position());
   ASSERT_TRUE(bool(SigHelp)) << "signatureHelp returned an error";
   EXPECT_THAT(SigHelp->signatures, IsEmpty());
 }
 
 class ClangdThreadingTest : public ClangdVFSTest {};
 
 TEST_F(ClangdThreadingTest, StressTest) {

 
 TEST_F(ClangdVFSTest, FallbackWhenPreambleIsNotReady) {
   MockFSProvider FS;
   ErrorCheckingDiagConsumer DiagConsumer;
   MockCompilationDatabase CDB;
   ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
 
   auto FooCpp = testPath("foo.cpp");
    Annotations Code(R"cpp(
+    namespace ns { int xyz; }
+    using namespace ns;
     int main() {
-      int xyz;
        xy^
     })cpp");
   FS.Files[FooCpp] = FooCpp;
 
   auto Opts = clangd::CodeCompleteOptions();
   Opts.AllowFallback = true;
 
   // This will make compile command broken and preamble absent.
   CDB.ExtraClangFlags = {"yolo.cc"};
   Server.addDocument(FooCpp, Code.code());
   ASSERT_TRUE(Server.blockUntilIdleForTest());
   auto Res = cantFail(runCodeComplete(Server, FooCpp, Code.point(), Opts));
-  EXPECT_THAT(Res.Completions, IsEmpty());
   EXPECT_EQ(Res.Context, CodeCompletionContext::CCC_Recovery);
+  // Identifier-based fallback completion doesn't know about "symbol" scope.
+  EXPECT_THAT(Res.Completions,
+              ElementsAre(AllOf(Field(&CodeCompletion::Name, "xyz"),
+                                Field(&CodeCompletion::Scope, ""))));
 
   // Make the compile command work again.
   CDB.ExtraClangFlags = {"-std=c++11"};
   Server.addDocument(FooCpp, Code.code());
   ASSERT_TRUE(Server.blockUntilIdleForTest());
   EXPECT_THAT(cantFail(runCodeComplete(Server, FooCpp, Code.point(),
-                                       Opts))
+                                       clangd::CodeCompleteOptions()))
                   .Completions,
-              ElementsAre(Field(&CodeCompletion::Name, "xyz")));
+              ElementsAre(AllOf(Field(&CodeCompletion::Name, "xyz"),
+                                Field(&CodeCompletion::Scope, "ns::"))));
 }
 
 } // namespace
 } // namespace clangd
 } // namespace clang

unittests/clangd/CodeCompleteTests.cpp

 #include "Quality.h"
 #include "SourceCode.h"
 #include "SyncAPI.h"
 #include "TestFS.h"
 #include "TestIndex.h"
 #include "TestTU.h"
 #include "index/MemIndex.h"
 #include "clang/Sema/CodeCompleteConsumer.h"
+#include "clang/Tooling/CompilationDatabase.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Testing/Support/Error.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 namespace clang {
 namespace clangd {
 
CodeCompleteResult completions(llvm::StringRef Text,
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   IgnoreDiagnostics DiagConsumer;
   ClangdServer Server(CDB, FS, DiagConsumer, ClangdServer::optsForTest());
   return completions(Server, Text, std::move(IndexSymbols), std::move(Opts),
                      FilePath);
 }
 
+// Builds a server and runs code completion.
+// If IndexSymbols is non-empty, an index will be built and passed to opts.
+CodeCompleteResult completionsNoCompile(llvm::StringRef Text,
+                                        std::vector<Symbol> IndexSymbols = {},
+                                        clangd::CodeCompleteOptions Opts = {},
+                                        PathRef FilePath = "foo.cpp") {
+  std::unique_ptr<SymbolIndex> OverrideIndex;
+  if (!IndexSymbols.empty()) {
+    assert(!Opts.Index && "both Index and IndexSymbols given!");
+    OverrideIndex = memIndex(std::move(IndexSymbols));
+    Opts.Index = OverrideIndex.get();
+  }
+
+  MockFSProvider FS;
+  Annotations Test(Text);
+  return codeComplete(FilePath, tooling::CompileCommand(), /*Preamble=*/nullptr,
+                      Test.code(), Test.point(), FS.getFileSystem(), Opts);
+}
+
 Symbol withReferences(int N, Symbol S) {
   S.References = N;
   return S;
 }
 
 TEST(CompletionTest, Limit) {
   clangd::CodeCompleteOptions Opts;
   Opts.Limit = 2;
auto Results = completions(R"cpp(
     M(ns::ABC^
     }
   )cpp",
                              {cls("foo::ns::ABCDE")}, Opts);
   EXPECT_THAT(Results.Completions,
               UnorderedElementsAre(AllOf(Qualifier(""), Named("ABCDE"))));
 }
 
+TEST(NoCompileCompletionTest, Basic) {
+  auto Results = completionsNoCompile(R"cpp(
+    void func() {
+      int xyz;
+      int abc;
+      ^
+    }
+  )cpp");
+  EXPECT_THAT(Results.Completions,
+              UnorderedElementsAre(Named("void"), Named("func"), Named("int"),
+                                   Named("xyz"), Named("abc")));
+}
+
+TEST(NoCompileCompletionTest, WithFilter) {
+  auto Results = completionsNoCompile(R"cpp(
+    void func() {
+      int sym1;
+      int sym2;
+      int xyz1;
+      int xyz2;
+      sy^
+    }
+  )cpp");
+  EXPECT_THAT(Results.Completions,
+              UnorderedElementsAre(Named("sym1"), Named("sym2")));
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang

unittests/clangd/HeadersTests.cpp

if (Preferred.empty())
       Preferred = Original;
     auto ToHeaderFile = [](llvm::StringRef Header) {
       return HeaderFile{Header,
                         /*Verbatim=*/!llvm::sys::path::is_absolute(Header)};
     };
 
     IncludeInserter Inserter(MainFile, /*Code=*/"", format::getLLVMStyle(),
                              CDB.getCompileCommand(MainFile)->Directory,
-                             Clang->getPreprocessor().getHeaderSearchInfo());
+                             &Clang->getPreprocessor().getHeaderSearchInfo());
     for (const auto &Inc : Inclusions)
       Inserter.addExisting(Inc);
     auto Declaring = ToHeaderFile(Original);
     auto Inserted = ToHeaderFile(Preferred);
     if (!Inserter.shouldInsertInclude(Declaring, Inserted))
       return "";
     std::string Path = Inserter.calculateIncludePath(Declaring, Inserted);
     Action.EndSourceFile();
     return Path;
   }
 
   llvm::Optional<TextEdit> insert(llvm::StringRef VerbatimHeader) {
     auto Clang = setupClang();
     PreprocessOnlyAction Action;
     EXPECT_TRUE(
         Action.BeginSourceFile(*Clang, Clang->getFrontendOpts().Inputs[0]));
 
     IncludeInserter Inserter(MainFile, /*Code=*/"", format::getLLVMStyle(),
                              CDB.getCompileCommand(MainFile)->Directory,
-                             Clang->getPreprocessor().getHeaderSearchInfo());
+                             &Clang->getPreprocessor().getHeaderSearchInfo());
     auto Edit = Inserter.insert(VerbatimHeader);
     Action.EndSourceFile();
     return Edit;
   }
 
   MockFSProvider FS;
   MockCompilationDatabase CDB;
   std::string MainFile = testPath("main.cpp");

 }
 
 TEST_F(HeadersTest, PreferInserted) {
   auto Edit = insert("<y>");
   EXPECT_TRUE(Edit.hasValue());
   EXPECT_TRUE(StringRef(Edit->newText).contains("<y>"));
 }
 
+TEST(Headers, NoHeaderSearchInfo) {
+  std::string MainFile = testPath("main.cpp");
+  IncludeInserter Inserter(MainFile, /*Code=*/"", format::getLLVMStyle(),
+                           /*BuildDir=*/"", /*HeaderSearchInfo=*/nullptr);
+
+  auto HeaderPath = testPath("sub/bar.h");
+  auto Declaring = HeaderFile{HeaderPath, /*Verbatim=*/false};
+  auto Inserting = HeaderFile{HeaderPath, /*Verbatim=*/false};
+  auto Verbatim = HeaderFile{"<x>", /*Verbatim=*/true};
+
+  EXPECT_EQ(Inserter.calculateIncludePath(Declaring, Inserting),
+            "\"" + HeaderPath + "\"");
+  EXPECT_EQ(Inserter.shouldInsertInclude(Declaring, Inserting), false);
+
+  EXPECT_EQ(Inserter.calculateIncludePath(Declaring, Verbatim), "<x>");
+  EXPECT_EQ(Inserter.shouldInsertInclude(Declaring, Verbatim), true);
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang

unittests/clangd/SourceCodeTests.cpp

 //===-- SourceCodeTests.cpp  ------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 #include "Annotations.h"
 #include "Context.h"
 #include "Protocol.h"
 #include "SourceCode.h"
+#include "clang/Format/Format.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/raw_os_ostream.h"
 #include "llvm/Testing/Support/Error.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 namespace clang {
 namespace clangd {
)cpp");
   // Check all positions mentioned in the test return valid results.
   for (auto P : Source.points()) {
     size_t Offset = llvm::cantFail(positionToOffset(Source.code(), P));
     EXPECT_THAT_EXPECTED(sourceLocationInMainFile(SM, P),
                          HasValue(StartOfFile.getLocWithOffset(Offset)));
   }
 }
 
+TEST(SourceCodeTests, CollectIdentifiers) {
+  auto Style = format::getLLVMStyle();
+  auto IDs = collectIdentifiers(R"cpp(
+  #include "a.h"
+  void foo() { int xyz; int abc = xyz; return foo(); }
+  )cpp",
+                                Style);
+  EXPECT_EQ(IDs.size(), 7u);
+  EXPECT_EQ(IDs["include"], 1u);
+  EXPECT_EQ(IDs["void"], 1u);
+  EXPECT_EQ(IDs["int"], 2u);
+  EXPECT_EQ(IDs["xyz"], 2u);
+  EXPECT_EQ(IDs["abc"], 1u);
+  EXPECT_EQ(IDs["return"], 1u);
+  EXPECT_EQ(IDs["foo"], 2u);
+}
+
 } // namespace
 } // namespace clangd
 } // namespace clang