[clangd] Extend dexp to support remote index

[kbobyrev]

Summary

• Merge clangd-remote-client into dexp
• Implement clangd::remote::IndexClient that is derived from SymbolIndex
• Upgrade remote mode-related CMake infrastructure

[kbobyrev]

Fix build.

[sammccall]

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

14

this include and the stuff depending on it needs to be ifdef'd

clang-tools-extra/clangd/index/remote/CMakeLists.txt

1–4

why is this extra param needed?

13

let's avoid propagating the clangDaemon name any further. clangdRemoteIndex? or clangdRemoteIndexClient?

clang-tools-extra/clangd/index/remote/Index.h

21 ↗

(On Diff #258853)

the class doesn't need to be exposed here, as the SymbolIndex interface seems to do everything we need. Just expose a factory?

Actually, I think that means we can expose this header whether grpc is enabled or not. If no grpc, then we just link in an implementation of the factory that always returns an error. WDYT?

clang-tools-extra/clangd/index/remote/Index.proto

16

should also return a stream. has_more is annoying, but you can keep it in the message and set it only on the last element of the stream.

67

confused... why not use Symbol now?

clang-tools-extra/clangd/index/remote/server/Server.cpp

46

move all the conversions into a file, marshalling.cpp or whatever?

[kbobyrev]

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

14

Ah, right, I forgot about this. Was present in the previous patch where I had custom definitions, but forgot about it for some reason.

41

@sammccall do you know why this opt is not being set for some reason? When I try to run dexp --help it is displayed in the possible arguments and when I try to pass anything invalid for boolean flags this also fails, but the value is not changed regardless of the values I put here :( Must be something simple I've missed.

clang-tools-extra/clangd/index/remote/CMakeLists.txt

1–4

Because this needs to be included both here and for dexp binary.

clang-tools-extra/clangd/index/remote/Index.proto

16

Is this an idiomatic way of using gRPC + Protobuf? Seems counter-intuitive to me since repeated stream of symbols in the message + single has_more seems quite logical.

67

Couldn't put Symbols into FuzzyFindReply for some reason. Clangd behaves really weird with Protobuf inclusions for me... Need to figure out why that happens, but might be me doing something wrong.

[kbobyrev]

Implement refs(), address some comments.

[kbobyrev]

Add correct definition and fix build.

[kbobyrev]

Move type conversions to Marshalling.(h|cpp) and hide IndexClient behind a factory function.

[kbobyrev]

Resolve a couple of FIXMEs. The patch is review-ready now.

[kbobyrev]

Make sure the value of --remote flag is stored.

[kbobyrev]

Stream responses in refs() and fuzzyFind()

[sammccall]

clang-tools-extra/clangd/CMakeLists.txt

161

Can we use features.inc.in instead for this?
(I don't have a strong opinion on which one is better, but I'd rather not do things two ways)

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

41

I think you need to read it before we ResetCommandLineParser(). That part is a bit of a hack :-(

clang-tools-extra/clangd/index/remote/CMakeLists.txt

1–4

still? I thought the idea was stuff outside this directory could include the client header, whether it's compiled in or not, and that header doesn't require protos

13

clang -> clangd though :-)

21

do protobuf/grpc++ not get linked in transitively by linking against RemoteIndexProtos? Sad :-(

clang-tools-extra/clangd/index/remote/Index.h

1 ↗

(On Diff #259257)

Suggest calling this Client.h

12 ↗

(On Diff #259257)

no implementation headers should be needed now

21 ↗

(On Diff #259257)

this needs docs:

• what it does
• error conditions
• what happens if support isn't compiled in

21 ↗

(On Diff #259257)

is the address resolved synchronously? is there a timeout?
does this block until the channel is ready, or will that happen on the first request?

(My sense is you're probably going to want a timeout param and to describe what it does, and *maybe* return different error types)

[sammccall]

clang-tools-extra/clangd/index/remote/Index.cpp

36 ↗

(On Diff #259257)

error-handling: what if the symbol is invalid? (log and skip)

36 ↗

(On Diff #259257)

this should call a function in marshalling, which just does the YAML thing for now

38 ↗

(On Diff #259257)

use log()/vlog if you want to log.
But this should probably be a trace::Span instead so you get the latency.

47 ↗

(On Diff #259257)

I'd consider pulling out a streamRPC template, these tend to attract cross-cutting code (setting deadlines, tracing/logging, and such)

97 ↗

(On Diff #259257)

if remote is disabled we can't compile the rest of this file.
Rather than ifdefs here, I'd suggest doing it at the cmake level:

if(CLANGD_ENABLE_REMOTE)
  add_clang_library(... Index.cpp)
else()
  add_clang_library(... IndexUnimplemented.cpp)
endif()

clang-tools-extra/clangd/index/remote/Index.proto

13

Just realized: we should probably call this SymbolIndex to match clangd::SymbolIndex (or rename the latter to match this)

21

nit: I'd suggest grouping requests with corresponding replies, and all the common vocab types either above or below.

51–64

"Actual messages" is a bit vague.
Maybe "The response is a stream of symbol messages, and one terminating has_more message."

54

I think a generic name like kind would be more readable here.

If the stream/final-result pattern is going to be our common one, you could consider giving them *all* generic names, like:

message FuzzyFindReply {
  oneof kind {
    Symbol stream_result;
    bool final_result; // HasMore
  }
}

Then they conform to a Stream<StreamT, FinalT> concept and you can abstract the handling in a template.

This makes sense if we want to lean into the index service being a mechanical translation of clangd::SymbolIndex. But I'm not sure whether that's a good idea, e.g. this one needs to be backwards-compatible, so they may drift.

67

Did you find out? Fixing this isn't a backwards-compatible change, and "this code is broken and I don't know why" isn't a great state for checkin.

Happy to help but please provide some details.

76

Ref should be a message with ref_yaml for now?
(Consistency with Symbol)

clang-tools-extra/clangd/index/remote/server/Server.cpp

79

new and set_allocated_* are scary :-)

NextMessage.mutable_symbol()->set_yaml_serialization(...)

92

fromProtobuf(Request)?

[kbobyrev]

Just a checkpoint: resolve several comments.

clang-tools-extra/clangd/index/remote/Index.proto

67

Yes, it has to do with different code generated by Protobuf and I learned how to discover which API is becoming removed and which is accessible.

[kbobyrev]

Address the remaining comments.

clang-tools-extra/clangd/index/remote/Index.cpp

97 ↗

(On Diff #259257)

I would argue that including index/remote/Client.h without CLANGD_ENABLE_REMOTE is not correct. We would still have to put #ifdefs in the user code regardless of whether what you proposed is implemented. I don't see any benefits in allowing users to include index/remote/Client.h, link clangdRemoteIndex and getting a runtime error. All of those steps have _remote_ in them and if _remote mode_ is not enabled, something certainly went wrong.

Also, this will complicate CMake structure as I can't put files that are conditionally added/excluded from the clang library and I would either have to make a separate directory with an "empty" library, or put a bunch of #ifdefs in here. Either is not optimal and I think it'd be better to leave it like this. WDYT?

[kbobyrev]

Convert plain function pointers to std::function in templates.

[kbobyrev]

Implement (B) for now

[kbobyrev]

Resolve the last comment.

[sammccall]

clang-tools-extra/clangd/Features.inc.in

2

nit: can we use the same name for the cmake variable and the preprocessor define?

There's a lot of potential for conceptual confusion between them, but I don't think giving them subtly different names helps, I'd rather make it just work for confused people :-)

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

14

Features.inc - that's the version that's been cmake-substituted
(I'm not sure why this isn't a regular include-guarded header, let's fix that separately)

18

Being able to include this header but not call the function in it doesn't make sense - linker errors aren't a friendly way to surface this constraint.

If the goal is to forbid use of the API without it compiled in, the header should #ifndef #CLANGD_ENABLE_REMOTE #error "Client.h included but CLANGD_ENABLE_REMOTE is off" #endif, and all the include-sites should be #ifdef-guarded.

(But please see other comments first)

40

I'd consider rather making this a string prefix on IndexLocation, so you'd invoke as dexp remote:localhost:4004 or whatever.
This encapsulates more easily, e.g. we can eventually move this into openIndex so all the callers transparently support the remote index.
(Though we shouldn't push that dependency into clangd in *this* patch, it just makes it easier in future)

clang-tools-extra/clangd/index/remote/Client.h

22

I can't follow this sentence - it's not clear what "SymbolIndex request" or "the actual connection" are.

I guess you mean something like "This method blocks to resolve the address, but does not wait for the channel to become ready".

22

The blocking behaviour is interesting:

• clangd should keep working while offline or in bad connectivity (thus no timeout on RPCs is a bug, we shouldn't rely on the channel outright detecting failure)
• this implies you don't want to return null if the connection can't be established, and so there's indeed not much point blocking. But blocking on resolving the name removes a lot of the benefit here.

I think GRPC actually has a pretty good model, it's documented here: https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md
(may be worth adding a link).
I'm not sure if this is actually blocked on name resolution, but it seems like it's at most blocked on *one attempt* at name resolution: the channel can still be created if resolution hasn't succeeded yet.

23

RPCs have no timeout should be a FIXME

30

I'm not sure what this is saying or why a caller would want to know it.

clang-tools-extra/clangd/index/remote/Index.cpp

1 ↗

(On Diff #259604)

This implements Client.h, should be Client.cpp

14 ↗

(On Diff #259604)

I'd consider using `<> for this include, or at least splitting it into a different section - non-LLVM non-stdlib includes are really rare.

26 ↗

(On Diff #259604)

no need to pass the stub if you make this a member func

27 ↗

(On Diff #259604)

Hmm, I think member pointers are totally the right thing here. And all the params should be deducible.

Have a look at https://godbolt.org/z/AnprJ- (simplified but self-contained and I think shows all the bits)

The thing I can't work out how to do is make the member pointer a template param *and* have it be deduced...

31 ↗

(On Diff #259604)

SpanName can be RPCRequestT::descriptor()->name() (e.g. "LookupRequest") rather than having to pass it explicitly

46 ↗

(On Diff #259604)

not even attempting to open the connection on startup seems to introduce more latency on first request than necessary. Call Channel->GetState(true)?
That won't block, but it will cause the connection to establish in the background.

68 ↗

(On Diff #259604)

this callback is almost the same - if you added a dummy stream result to Lookup you could move it into streamrpc :-)

97 ↗

(On Diff #259257)

I would argue that including index/remote/Client.h without CLANGD_ENABLE_REMOTE is not correct. We would still have to put #ifdefs in the user code regardless of whether what you proposed is implemented

This is begging the question - what I'm proposing is to give that header/library defined behavior when CLANGD_ENABLE_REMOTE is off, and so you wouldn't need #ifdefs.

I don't see any benefits in allowing users to include index/remote/Client.h, link clangdRemoteIndex and getting a runtime error.

The benefits are:

• no ifdefs and fewer conditional cmake rules, which are both hard to read and result in untested constructs without even basic syntax checking
• fewer cases to consider in clients, as this falls into the regular "can't connect to index" codepath
• readers trying to follow the possible behaviours end up at the header documentation of a function that is called, which is an easy and familiar workflow

All of those steps have _remote_ in them and if _remote mode_ is not enabled, something certainly went wrong

Again, this is begging the question. If the header says "if GRPC mode is not enabled, always returns nullptr", then everything is working as designed.

Also, this will complicate CMake structure as I can't put files that are conditionally added/excluded from the clang library

Why not? (This sounds like a fair technical objection, but may be surmountable)

clang-tools-extra/clangd/index/remote/Marshalling.h

25 ↗

(On Diff #259604)

You could also just log the error and return Optional at this level... there's no recovery other than logging anyway. I kind of hate Expected/Error because of their asserting and weird non-constness so I'd be tempted by that.
Up to you though

clang-tools-extra/clangd/index/remote/server/CMakeLists.txt

17

why does the server depend on the client?

[kbobyrev]

Resolve comments that refer to the outdated patch and address few small issues.

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

18

Should be resolved in the newer version with (A) implementation.

clang-tools-extra/clangd/index/remote/Client.h

23

streamRPC already has a FIXME for that.

clang-tools-extra/clangd/index/remote/Index.cpp

97 ↗

(On Diff #259257)

I believe this is resolved in new version.

Also, this will complicate CMake structure as I can't put files that are conditionally added/excluded from the clang library

Why not? (This sounds like a fair technical objection, but may be surmountable)

As discussed before, there is a CMake module in LLVM that errors out in case some files are not included in any library. In case of conditional compilation, either UnimplementedClient.cpp or Client.cpp will be left out (or this has to be the single file with #ifdefs). See new version for the exact layout I came up with.

This is especially unfortunate because I actually need two libraries - one for Client implementation and one for Marshalling. And because of that I have to have three separate directories :( Suboptimal, but probably OK.

clang-tools-extra/clangd/index/remote/server/CMakeLists.txt

17

Fixed in a newer diff.

[kbobyrev]

Resolve couple more comments.

clang-tools-extra/clangd/index/remote/Index.cpp

27 ↗

(On Diff #259604)

Uh, I had the pointers but decided against it because of rather... bizarre syntax :D

And yes, just like you said, the calls contained too many explicit template parameters, so I decided the implementation should be more verbose than "user code". I'm not against pointers, but I think the current version looks simpler, WDYT?

[kbobyrev]

Resolve the last comments.

[kbobyrev]

Don't use callback on the last message.

[sammccall]

LG, nits apart from the llvm_unreachable in unimplemented, let's discuss further if you disagree on that.

We do have to go back and add unit tests for this stuff. I'm OK with doing that once the marshalling is sorted out (not YAML), but we should make sure we remember!

clang-tools-extra/clangd/index/YAMLSerialization.cpp

536

we should still log (or at least vlog)

(Actually personally I'd return Expected here and then log and drop to optional in Marshalling, but it doesn't matter)

clang-tools-extra/clangd/index/dex/dexp/CMakeLists.txt

2

This comment is likely to get stale. I'd drop it: if we're including the clangd source dir, also including the clangd binary dir is appropriate and doesn't really require an explanation.

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

32

Dex -> index file
(dex is the in-memory implementation, not the data format)

44

dex -> dexp (and in the example command)

331

no longer needs to be ifdef'd: if the user passes remote:foo and it's not compiled in, they'll get an appropriate error

clang-tools-extra/clangd/index/remote/CMakeLists.txt

26

Maybe "# provides a dummy implementation of clangdRemoteIndex"

clang-tools-extra/clangd/index/remote/Client.cpp

30

whether this is a member pointer or a function wrapper, it should be in a typedef or readability.
e.g.
template <typename ReqT, typename RspT>
using StreamingCall = std::function<unique_ptr<ClientReader<RspT>>>(Stub*, ClientContext*, const RspT&);

48

nit: I'd suggest dropping "YAML" from this message so we don't forget later. Rather "Received invalid {0}: {1}", ReplyT::descriptor()->name(), Sym.takeError()

96

nit: for slightly cleaner separation, I'd suggest creating the channel here and passing it into the IndexClient constructor.

The channel is where various configuration (auth, etc) might live, and it's nice if the client class stays doesn't need to deal with that level of abstraction.
e.g. a test version that doesn't use the real network would pass in a different channel and the IndexClient wouldn't care. (Not sure if that's how testing is actually done in grpc these days, but you get the idea...)

clang-tools-extra/clangd/index/remote/Client.h

2

Client.h - Connect to a remote index via gRPC

clang-tools-extra/clangd/index/remote/Index.cpp

27 ↗

(On Diff #259604)

There are advantages of member function pointers here:

• the fact that it's a member function on the stub *is* significant information here I think
• passing a std::function allows deduction of RPCRequestT and ReplyT as the compiler can pattern-match, while that doesn't work for function<> because there's an implicit conversion in the way.
• (std::function is slow, but that shouldn't matter here. I guess function_ref doesn't have support for member functions?)

And yes, just like you said, the calls contained too many explicit template parameters

I think this is backwards, the function pointer version should require no explicit type template parameters, where the function<> version requires 2. See the godbolt example above for deduction.

(The thing I couldn't get working was making the member fptr a *non-type-template-param* while keeping the type template params deduced, but that's not a disadvantage vs function<>)

I think the current version looks simpler, WDYT?

After extracting an alias template for the type (which should happen anyway) I don't think there's a significant difference really:

template <typename ReqT, RspT>
using StreamingCall = std::function<std::unique_ptr<grpc::ClientReader<ReplyT>>(Stub*, ClientContext*, const ReqT&)>;
// vs
template <typename ReqT, RspT>
using StreamingCall = std::unique_ptr<grpc::ClientReader<ReplyT>>(Stub::*)(ClientContext*, const ReqT&);

the member fptr syntax is more unusual/exotic, but shorter and less nested. Neither is really readable, and the readability of this line isn't that significant.

clang-tools-extra/clangd/index/remote/server/Server.cpp

64

*NextMessage.mutable_stream_result() = toProtobuf(Sym)

and move the trivial toProtobuf logic to Marshalling (so it's trivial to replace later)

clang-tools-extra/clangd/index/remote/unimplemented/CMakeLists.txt

2

This is unusual, include a comment for what it's doing.

clang-tools-extra/clangd/index/remote/unimplemented/Client.cpp

1 ↗

(On Diff #259839)

Probably call this UnimplementedClient.cpp, no need to have the source filenames unneccesarily coincide? Makes it clear the other one is the primary.

19 ↗

(On Diff #259839)

no llvm_unreachable, we shouldn't crash. Just elog and continue...

[kbobyrev]

Address the last round of comments.

[sammccall]

clang-tools-extra/clangd/index/remote/Client.cpp

65

nit: LookupRequest and LookupReply should be deducible now I think

clang-tools-extra/clangd/CMakeLists.txt

 
 add_subdirectory(refactor/tweaks)
 if (${CMAKE_SYSTEM_NAME} STREQUAL "Linux")
   # FIXME: Make fuzzer not use linux-specific APIs, build it everywhere.
   add_subdirectory(fuzzer)
 endif()
 add_subdirectory(tool)
 add_subdirectory(indexer)
-add_subdirectory(index/dex/dexp)
 
 if (LLVM_INCLUDE_BENCHMARKS)
   add_subdirectory(benchmarks)
 endif()
 if ( CLANGD_BUILD_XPC )
   add_subdirectory(xpc)
 endif ()
 
 if(CLANG_INCLUDE_TESTS)
 add_subdirectory(test)
 add_subdirectory(unittests)
 endif()
 
 # FIXME(kirillbobyrev): Document this in the LLVM docs once remote index is stable.
 option(CLANGD_ENABLE_REMOTE "Use gRPC library to enable remote index support for Clangd" OFF)
 set(GRPC_INSTALL_PATH "" CACHE PATH "Path to gRPC library manual installation.")
 
 if (CLANGD_ENABLE_REMOTE)
   include(FindGRPC)
-  add_subdirectory(index/remote)
 endif()
+add_subdirectory(index/remote)
+add_subdirectory(index/dex/dexp)

clang-tools-extra/clangd/Features.inc.in

 #define CLANGD_BUILD_XPC @CLANGD_BUILD_XPC@
+#define CLANGD_ENABLE_REMOTE @CLANGD_ENABLE_REMOTE@

clang-tools-extra/clangd/index/Serialization.h

 };
 // Serializes an index file.
 llvm::raw_ostream &operator<<(llvm::raw_ostream &OS, const IndexFileOut &O);
 
 // Convert a single symbol to YAML, a nice debug representation.
 std::string toYAML(const Symbol &);
 std::string toYAML(const std::pair<SymbolID, ArrayRef<Ref>> &);
 std::string toYAML(const Relation &);
+std::string toYAML(const Ref &);
+
+// Deserialize a single symbol from YAML.
+llvm::Expected<clangd::Symbol> symbolFromYAML(StringRef YAML,
+                                              llvm::UniqueStringSaver *Strings);
+llvm::Expected<clangd::Ref> refFromYAML(StringRef YAML,
+                                        llvm::UniqueStringSaver *Strings);
 
 // Build an in-memory static index from an index file.
 // The size should be relatively small, so data can be managed in memory.
 std::unique_ptr<SymbolIndex> loadIndex(llvm::StringRef Filename,
                                        bool UseDex = true);
 
 } // namespace clangd
 } // namespace clang
 
 #endif

clang-tools-extra/clangd/index/YAMLSerialization.cpp

std::string Buf;
     llvm::raw_string_ostream OS(Buf);
     llvm::yaml::Output Yout(OS);
     Relation Rel = R; // copy: Yout<< requires mutability.
     Yout << Rel;
   }
   return Buf;
 }
 
+std::string toYAML(const Ref &R) {
+  std::string Buf;
+  {
+    llvm::raw_string_ostream OS(Buf);
+    llvm::yaml::Output Yout(OS);
+    Ref Reference = R; // copy: Yout<< requires mutability.
+    Yout << Reference;
+  }
+  return Buf;
+}
+
+llvm::Expected<clangd::Symbol>
+symbolFromYAML(StringRef YAML, llvm::UniqueStringSaver *Strings) {
+  clangd::Symbol Deserialized;
+  llvm::yaml::Input YAMLInput(YAML, Strings);
+  if (YAMLInput.error())
+    return llvm::make_error<llvm::StringError>(
+        llvm::formatv("Unable to deserialize Symbol from YAML: {0}", YAML),
+        llvm::inconvertibleErrorCode());
+  YAMLInput >> Deserialized;
+  return Deserialized;
+}
+
+llvm::Expected<clangd::Ref> refFromYAML(StringRef YAML,
+                                        llvm::UniqueStringSaver *Strings) {
+  clangd::Ref Deserialized;
+  llvm::yaml::Input YAMLInput(YAML, Strings);
+  if (YAMLInput.error())
+    return llvm::make_error<llvm::StringError>(
+        llvm::formatv("Unable to deserialize Symbol from YAML: {0}", YAML),
+        llvm::inconvertibleErrorCode());
+  YAMLInput >> Deserialized;
+  return Deserialized;
+}
+
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/dex/dexp/CMakeLists.txt

 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../../../)
+include_directories(${CMAKE_CURRENT_BINARY_DIR}/../../../)
 
 set(LLVM_LINK_COMPONENTS
   LineEditor
   Support
   )
 
 add_clang_executable(dexp
   Dexp.cpp
   )
 
 clang_target_link_libraries(dexp
   PRIVATE
   clangBasic
   )
 target_link_libraries(dexp
   PRIVATE
   clangDaemon
+  clangdRemoteIndex
   )

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

 //===--- Dexp.cpp - Dex EXPloration tool ------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file implements a simple interactive tool which can be used to manually
 // evaluate symbol search quality of Clangd index.
 //
 //===----------------------------------------------------------------------===//
 
+#include "Features.inc"
 #include "SourceCode.h"
 #include "index/Serialization.h"
 #include "index/dex/Dex.h"
+#include "index/remote/Client.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/LineEditor/LineEditor.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Signals.h"
 
 namespace clang {
 namespace clangd {
 namespace {
 
-llvm::cl::opt<std::string> IndexPath(llvm::cl::desc("<INDEX FILE>"),
-                                     llvm::cl::Positional, llvm::cl::Required);
+llvm::cl::opt<std::string> IndexLocation(
+    llvm::cl::desc("<path to index file | remote:server.address>"),
+    llvm::cl::Positional);
 
 llvm::cl::opt<std::string>
     ExecCommand("c", llvm::cl::desc("Command to execute and then exit"));
 
 static const std::string Overview = R"(
 This is an **experimental** interactive tool to process user-provided search
 queries over given symbol collection obtained via clangd-indexer. The
 tool can be used to evaluate search quality of existing index implementations
 and manually construct non-trivial test cases.
 
+You can connect to remote index by passing remote:address to dexp. Example:
+
+$ dexp remote:0.0.0.0:9000
+
 Type use "help" request to get information about the details.
 )";
 
 void reportTime(llvm::StringRef Name, llvm::function_ref<void()> F) {
   const auto TimerStart = std::chrono::high_resolution_clock::now();
   F();
   const auto TimerStop = std::chrono::high_resolution_clock::now();
   const auto Duration = std::chrono::duration_cast<std::chrono::milliseconds>(
void run() override {
     Request.Query = Query;
     if (Scopes.getNumOccurrences() > 0) {
       llvm::SmallVector<llvm::StringRef, 8> Scopes;
       llvm::StringRef(this->Scopes).split(Scopes, ',');
       Request.Scopes = {Scopes.begin(), Scopes.end()};
     }
     Request.AnyScope = Request.Scopes.empty();
     // FIXME(kbobyrev): Print symbol final scores to see the distribution.
-    static const auto OutputFormat = "{0,-4} | {1,-40} | {2,-25}\n";
+    static const auto *OutputFormat = "{0,-4} | {1,-40} | {2,-25}\n";
     llvm::outs() << llvm::formatv(OutputFormat, "Rank", "Symbol ID",
                                   "Symbol Name");
     size_t Rank = 0;
     Index->fuzzyFind(Request, [&](const Symbol &Sym) {
       llvm::outs() << llvm::formatv(OutputFormat, Rank++, Sym.ID.str(),
                                     Sym.Scope + Sym.Name);
     });
   }

 struct {
   const char *Name;
   const char *Description;
   std::function<std::unique_ptr<Command>()> Implementation;
 } CommandInfo[] = {
     {"find", "Search for symbols with fuzzyFind", std::make_unique<FuzzyFind>},
     {"lookup", "Dump symbol details by ID or qualified name",
      std::make_unique<Lookup>},
-    {"refs", "Find references by ID or qualified name",
-     std::make_unique<Refs>},
+    {"refs", "Find references by ID or qualified name", std::make_unique<Refs>},
     {"export", "Export index", std::make_unique<Export>},
 };
 
 std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
-  return loadIndex(Index, /*UseDex=*/true);
+  return Index.startswith("remote:")
+             ? remote::getClient(Index.drop_front(strlen("remote:")))
+             : loadIndex(Index, /*UseDex=*/true);
 }
 
 bool runCommand(std::string Request, const SymbolIndex &Index) {
   // Split on spaces and add required null-termination.
   std::replace(Request.begin(), Request.end(), ' ', '\0');
   llvm::SmallVector<llvm::StringRef, 8> Args;
   llvm::StringRef(Request).split(Args, '\0', /*MaxSplit=*/-1,
                                  /*KeepEmpty=*/false);

 int main(int argc, const char *argv[]) {
   using namespace clang::clangd;
 
   llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
   llvm::cl::ResetCommandLineParser(); // We reuse it for REPL commands.
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
 
   std::unique_ptr<SymbolIndex> Index;
-  reportTime("Dex build", [&]() {
-    Index = openIndex(IndexPath);
-  });
+  reportTime(llvm::StringRef(IndexLocation).startswith("remote:")
+                 ? "Remote index client creation"
+                 : "Dex build",
+             [&]() { Index = openIndex(IndexLocation); });
 
   if (!Index) {
     llvm::outs() << "Failed to open the index.\n";
     return -1;
   }
 
   if (!ExecCommand.empty())
     return runCommand(ExecCommand, *Index) ? 0 : 1;
 
   llvm::LineEditor LE("dexp");
   while (llvm::Optional<std::string> Request = LE.readLine())
     runCommand(std::move(*Request), *Index);
 }

clang-tools-extra/clangd/index/remote/CMakeLists.txt

+if (CLANGD_ENABLE_REMOTE)
   generate_grpc_protos(RemoteIndexProtos "Index.proto")
-
-include_directories("${CMAKE_CURRENT_BINARY_DIR}")
+  include_directories(${CMAKE_CURRENT_BINARY_DIR})
   include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../../)
 
-add_subdirectory(client)
+  # FIXME(kirillbobyrev): target_compile_definitions is not working with
+  # add_clang_library for some reason. Is there any way to make this
+  # target-local?
+  add_definitions(-DGOOGLE_PROTOBUF_NO_RTTI=1)
+
+  add_clang_library(clangdRemoteIndex
+    Client.cpp
+
+    LINK_LIBS
+    RemoteIndexProtos
+    clangdRemoteMarshalling
+
+    protobuf
+    grpc++
+    clangDaemon
+    )
+
+  add_subdirectory(marshalling)
   add_subdirectory(server)
+else()
+  # Provides a dummy implementation of clangdRemoteIndex.
+  add_subdirectory(unimplemented)
+endif()

clang-tools-extra/clangd/index/remote/Client.h

+//===--- Client.h - Connect to a remote index via gRPC -----------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H
+
+#include "index/Index.h"
+
+namespace clang {
+namespace clangd {
+namespace remote {
+
+/// Returns an SymbolIndex client that passes requests to remote index located
+/// at \p Address. The client allows synchronous RPC calls.
+///
+/// This method attempts to resolve the address and establish the connection.
+///
+/// \returns nullptr if the address is not resolved during the function call or
+/// if the project was compiled without Remote Index support.
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address);
+
+} // namespace remote
+} // namespace clangd
+} // namespace clang
+
+#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H

clang-tools-extra/clangd/index/remote/Client.cpp

+//===--- Client.cpp ----------------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include <grpcpp/grpcpp.h>
+
+#include "Client.h"
+#include "Index.grpc.pb.h"
+#include "Logger.h"
+#include "Trace.h"
+#include "index/Serialization.h"
+#include "marshalling/Marshalling.h"
+#include "llvm/Support/YAMLTraits.h"
+
+namespace clang {
+namespace clangd {
+namespace remote {
+namespace {
+
+class IndexClient : public clangd::SymbolIndex {
+  template <typename RequestT, typename ReplyT>
+  using StreamingCall = std::unique_ptr<grpc::ClientReader<ReplyT>> (
+      remote::SymbolIndex::Stub::*)(grpc::ClientContext *, const RequestT &);
+
+  // FIXME(kirillbobyrev): Set deadlines for requests.
+  template <typename RequestT, typename ReplyT, typename ClangdRequestT,
+            typename CallbackT>
+  bool streamRPC(ClangdRequestT Request,
+                 StreamingCall<RequestT, ReplyT> RPCCall,
+                 CallbackT Callback) const {
+    bool FinalResult = false;
+    trace::Span Tracer(RequestT::descriptor()->name());
+    const auto RPCRequest = toProtobuf(Request);
+    grpc::ClientContext Context;
+    auto Reader = (Stub.get()->*RPCCall)(&Context, RPCRequest);
+    llvm::BumpPtrAllocator Arena;
+    llvm::UniqueStringSaver Strings(Arena);
+    ReplyT Reply;
+    while (Reader->Read(&Reply)) {
+      if (!Reply.has_stream_result()) {
+        FinalResult = Reply.final_result();
+        continue;
+      }
+      auto Sym = fromProtobuf(Reply.stream_result(), &Strings);
+      if (!Sym)
+        elog("Received invalid {0}: {1}", ReplyT::descriptor()->name(),
+             Reply.stream_result().yaml_serialization());
+      Callback(*Sym);
+    }
+    SPAN_ATTACH(Tracer, "status", Reader->Finish().ok());
+    return FinalResult;
+  }
+
+public:
+  IndexClient(std::shared_ptr<grpc::Channel> Channel)
+      : Stub(remote::SymbolIndex::NewStub(Channel)) {}
+
+  void lookup(const clangd::LookupRequest &Request,
+              llvm::function_ref<void(const clangd::Symbol &)> Callback) const {
+    streamRPC<LookupRequest, LookupReply>(
+        Request, &remote::SymbolIndex::Stub::Lookup, Callback);
+  }
+
+  bool
+  fuzzyFind(const clangd::FuzzyFindRequest &Request,
+            llvm::function_ref<void(const clangd::Symbol &)> Callback) const {
+    return streamRPC<FuzzyFindRequest, FuzzyFindReply>(
+        Request, &remote::SymbolIndex::Stub::FuzzyFind, Callback);
+  }
+
+  bool refs(const clangd::RefsRequest &Request,
+            llvm::function_ref<void(const clangd::Ref &)> Callback) const {
+    return streamRPC<RefsRequest, RefsReply>(
+        Request, &remote::SymbolIndex::Stub::Refs, Callback);
+  }
+
+  // FIXME(kirillbobyrev): Implement this.
+  void
+  relations(const clangd::RelationsRequest &,
+            llvm::function_ref<void(const SymbolID &, const clangd::Symbol &)>)
+      const {}
+
+  // IndexClient does not take any space since the data is stored on the server.
+  size_t estimateMemoryUsage() const { return 0; }
+
+private:
+  std::unique_ptr<remote::SymbolIndex::Stub> Stub;
+};
+
+} // namespace
+
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address) {
+  const auto Channel =
+      grpc::CreateChannel(Address.str(), grpc::InsecureChannelCredentials());
+  Channel->GetState(true);
+  return std::unique_ptr<clangd::SymbolIndex>(new IndexClient(Channel));
+}
+
+} // namespace remote
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/index/remote/Index.proto

 //===--- Index.proto - Remote index Protocol Buffers definition -----------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 syntax = "proto3";
 
 package clang.clangd.remote;
 
-service Index {
+service SymbolIndex {
   rpc Lookup(LookupRequest) returns (stream LookupReply) {}
+
+  rpc FuzzyFind(FuzzyFindRequest) returns (stream FuzzyFindReply) {}
+
+  rpc Refs(RefsRequest) returns (stream RefsReply) {}
+}
+
+message LookupRequest { repeated string ids = 1; }
+
+// The response is a stream of symbol messages and the terminating message
+// indicating the end of stream.
+message LookupReply {
+  oneof kind {
+    Symbol stream_result = 1;
+    bool final_result = 2;
+  }
+}
+
+message FuzzyFindRequest {
+  string query = 1;
+  repeated string scopes = 2;
+  bool any_scope = 3;
+  uint32 limit = 4;
+  bool resricted_for_code_completion = 5;
+  repeated string proximity_paths = 6;
+  repeated string preferred_types = 7;
 }
 
-message LookupRequest { string id = 1; }
+// The response is a stream of symbol messages, and one terminating has_more
+// message.
+message FuzzyFindReply {
+  oneof kind {
+    Symbol stream_result = 1;
+    bool final_result = 2; // HasMore
+  }
+}
+
+message RefsRequest {
+  repeated string ids = 1;
+  uint32 filter = 2;
+  uint32 limit = 3;
+}
+
+// The response is a stream of reference messages, and one terminating has_more
+// message.
+message RefsReply {
+  oneof kind {
+    Ref stream_result = 1;
+    bool final_result = 2; // HasMore
+  }
+}
 
-message LookupReply { string symbol_yaml = 1; }
+// FIXME(kirillbobyrev): Properly serialize symbols and refs instead of passing
+// YAML.
+message Ref { string yaml_serialization = 1; }
+message Symbol { string yaml_serialization = 1; }

clang-tools-extra/clangd/index/remote/client/CMakeLists.txt

-set(LLVM_LINK_COMPONENTS
-  LineEditor
-  Support
-  )
-add_clang_executable(clangd-index-client
-  Client.cpp
-  )
-target_compile_definitions(clangd-index-client PRIVATE -DGOOGLE_PROTOBUF_NO_RTTI=1)
-clang_target_link_libraries(clangd-index-client
-  PRIVATE
-  clangDaemon
-  )
-target_link_libraries(clangd-index-client
-  PRIVATE
-  RemoteIndexProtos
-
-  protobuf
-  grpc++
-  )

clang-tools-extra/clangd/index/remote/client/Client.cpp

-//===--- Client.cpp - Remote Index Client -----------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements a simple interactive tool which can be used to manually
-// evaluate symbol search quality of Clangd index.
-//
-//===----------------------------------------------------------------------===//
-
-#include "SourceCode.h"
-#include "index/Serialization.h"
-#include "index/dex/Dex.h"
-#include "llvm/ADT/ScopeExit.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/ADT/StringRef.h"
-#include "llvm/ADT/StringSwitch.h"
-#include "llvm/LineEditor/LineEditor.h"
-#include "llvm/Support/CommandLine.h"
-#include "llvm/Support/Signals.h"
-
-#include "grpcpp/grpcpp.h"
-
-#include "Index.grpc.pb.h"
-
-namespace clang {
-namespace clangd {
-namespace {
-
-llvm::cl::opt<std::string>
-    ServerAddress("server-address",
-                  llvm::cl::desc("Address of remote index server to use."),
-                  llvm::cl::init("0.0.0.0:50051"));
-
-static const std::string Overview = R"(
-This is an **experimental** interactive tool to process user-provided search
-queries over given symbol collection obtained via clangd-indexer with the help
-of remote index server. The client will connect to remote index server and pass
-it lookup queries.
-)";
-
-class RemoteIndexClient {
-public:
-  RemoteIndexClient(std::shared_ptr<grpc::Channel> Channel)
-      : Stub(remote::Index::NewStub(Channel)) {}
-
-  void lookup(llvm::StringRef ID) {
-    llvm::outs() << "Lookup of symbol with ID " << ID << '\n';
-    remote::LookupRequest Proto;
-    Proto.set_id(ID.str());
-
-    grpc::ClientContext Context;
-    remote::LookupReply Reply;
-    std::unique_ptr<grpc::ClientReader<remote::LookupReply>> Reader(
-        Stub->Lookup(&Context, Proto));
-    while (Reader->Read(&Reply)) {
-      llvm::outs() << Reply.symbol_yaml();
-    }
-    grpc::Status Status = Reader->Finish();
-    if (Status.ok()) {
-      llvm::outs() << "lookupRequest rpc succeeded.\n";
-    } else {
-      llvm::outs() << "lookupRequest rpc failed.\n";
-    }
-  }
-
-private:
-  std::unique_ptr<remote::Index::Stub> Stub;
-};
-
-} // namespace
-} // namespace clangd
-} // namespace clang
-
-int main(int argc, const char *argv[]) {
-  using namespace clang::clangd;
-
-  llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
-  llvm::cl::ResetCommandLineParser(); // We reuse it for REPL commands.
-  llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
-
-  RemoteIndexClient IndexClient(
-      grpc::CreateChannel(ServerAddress, grpc::InsecureChannelCredentials()));
-
-  llvm::LineEditor LE("remote-index-client");
-  while (llvm::Optional<std::string> Request = LE.readLine())
-    IndexClient.lookup(std::move(*Request));
-}

clang-tools-extra/clangd/index/remote/marshalling/CMakeLists.txt

+add_clang_library(clangdRemoteMarshalling
+  Marshalling.cpp
+
+  LINK_LIBS
+  RemoteIndexProtos
+
+  protobuf
+  clangDaemon
+  )

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

+//===--- Marshalling.h -------------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Transformations between native Clangd types and Protobuf-generated classes.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H
+
+#include "Index.grpc.pb.h"
+#include "index/Index.h"
+#include "llvm/Support/StringSaver.h"
+
+namespace clang {
+namespace clangd {
+namespace remote {
+
+clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request);
+llvm::Optional<clangd::Symbol> fromProtobuf(const Symbol &Message,
+                                            llvm::UniqueStringSaver *Strings);
+llvm::Optional<clangd::Ref> fromProtobuf(const Ref &Message,
+                                         llvm::UniqueStringSaver *Strings);
+
+LookupRequest toProtobuf(const clangd::LookupRequest &From);
+FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From);
+RefsRequest toProtobuf(const clangd::RefsRequest &From);
+
+Ref toProtobuf(const clangd::Ref &From);
+Symbol toProtobuf(const clangd::Symbol &From);
+
+} // namespace remote
+} // namespace clangd
+} // namespace clang
+
+#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

+//===--- Marshalling.cpp -----------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "Marshalling.h"
+#include "Logger.h"
+#include "index/Serialization.h"
+
+namespace clang {
+namespace clangd {
+namespace remote {
+
+clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request) {
+  clangd::FuzzyFindRequest Result;
+  Result.Query = Request->query();
+  for (const auto &Scope : Request->scopes())
+    Result.Scopes.push_back(Scope);
+  Result.AnyScope = Request->any_scope();
+  if (Request->limit())
+    Result.Limit = Request->limit();
+  Result.RestrictForCodeCompletion = Request->resricted_for_code_completion();
+  for (const auto &Path : Request->proximity_paths())
+    Result.ProximityPaths.push_back(Path);
+  for (const auto &Type : Request->preferred_types())
+    Result.ProximityPaths.push_back(Type);
+  return Result;
+}
+
+llvm::Optional<clangd::Symbol> fromProtobuf(const Symbol &Message,
+                                            llvm::UniqueStringSaver *Strings) {
+  auto Result = symbolFromYAML(Message.yaml_serialization(), Strings);
+  if (!Result) {
+    elog("Cannot convert Symbol from Protobuf: {}", Result.takeError());
+    return llvm::None;
+  }
+  return *Result;
+}
+llvm::Optional<clangd::Ref> fromProtobuf(const Ref &Message,
+                                         llvm::UniqueStringSaver *Strings) {
+  auto Result = refFromYAML(Message.yaml_serialization(), Strings);
+  if (!Result) {
+    elog("Cannot convert Ref from Protobuf: {}", Result.takeError());
+    return llvm::None;
+  }
+  return *Result;
+}
+
+LookupRequest toProtobuf(const clangd::LookupRequest &From) {
+  LookupRequest RPCRequest;
+  for (const auto &SymbolID : From.IDs)
+    RPCRequest.add_ids(SymbolID.str());
+  return RPCRequest;
+}
+
+FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From) {
+  FuzzyFindRequest RPCRequest;
+  RPCRequest.set_query(From.Query);
+  for (const auto &Scope : From.Scopes)
+    RPCRequest.add_scopes(Scope);
+  RPCRequest.set_any_scope(From.AnyScope);
+  if (From.Limit)
+    RPCRequest.set_limit(*From.Limit);
+  RPCRequest.set_resricted_for_code_completion(From.RestrictForCodeCompletion);
+  for (const auto &Path : From.ProximityPaths)
+    RPCRequest.add_proximity_paths(Path);
+  for (const auto &Type : From.PreferredTypes)
+    RPCRequest.add_preferred_types(Type);
+  return RPCRequest;
+}
+
+RefsRequest toProtobuf(const clangd::RefsRequest &From) {
+  RefsRequest RPCRequest;
+  for (const auto &ID : From.IDs)
+    RPCRequest.add_ids(ID.str());
+  RPCRequest.set_filter(static_cast<uint32_t>(From.Filter));
+  if (From.Limit)
+    RPCRequest.set_limit(*From.Limit);
+  return RPCRequest;
+}
+
+Symbol toProtobuf(const clangd::Symbol &From) {
+  Symbol Result;
+  Result.set_yaml_serialization(toYAML(From));
+  return Result;
+}
+
+Ref toProtobuf(const clangd::Ref &From) {
+  Ref Result;
+  Result.set_yaml_serialization(toYAML(From));
+  return Result;
+}
+
+} // namespace remote
+} // namespace clangd
+} // namespace clang

clang-tools-extra/clangd/index/remote/server/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   LineEditor
   Support
   )
 add_clang_executable(clangd-index-server
   Server.cpp
   )
-target_compile_definitions(clangd-index-server PRIVATE -DGOOGLE_PROTOBUF_NO_RTTI=1)
+target_compile_definitions(clangd-index-server PRIVATE -D GOOGLE_PROTOBUF_NO_RTTI=1)
 clang_target_link_libraries(clangd-index-server
   PRIVATE
   clangDaemon
   )
 target_link_libraries(clangd-index-server
   PRIVATE
   RemoteIndexProtos
 
-  protobuf
-  grpc++
-  clangDaemon
+  clangdRemoteMarshalling
   )

clang-tools-extra/clangd/index/remote/server/Server.cpp

 //===--- Server.cpp - gRPC-based Remote Index Server  ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "index/Index.h"
 #include "index/Serialization.h"
+#include "index/remote/marshalling/Marshalling.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/LineEditor/LineEditor.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Signals.h"
 
-#include "grpcpp/grpcpp.h"
-#include "grpcpp/health_check_service_interface.h"
+#include <grpcpp/grpcpp.h>
+#include <grpcpp/health_check_service_interface.h>
 
 #include "Index.grpc.pb.h"
 
 namespace clang {
 namespace clangd {
+namespace remote {
 namespace {
 
 static const std::string Overview = R"(
 This is an experimental remote index implementation. The server opens Dex and
 awaits gRPC lookup requests from the client.
 )";
 
 llvm::cl::opt<std::string> IndexPath(llvm::cl::desc("<INDEX FILE>"),
                                      llvm::cl::Positional, llvm::cl::Required);
 
-llvm::cl::opt<std::string> ServerAddress("server-address",
-                                         llvm::cl::init("0.0.0.0:50051"));
+llvm::cl::opt<std::string> ServerAddress(
+    "server-address", llvm::cl::init("0.0.0.0:50051"),
+    llvm::cl::desc("Address of the invoked server. Defaults to 0.0.0.0:50051"));
 
-std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
+std::unique_ptr<clangd::SymbolIndex> openIndex(llvm::StringRef Index) {
   return loadIndex(Index, /*UseIndex=*/true);
 }
 
-class RemoteIndexServer final : public remote::Index::Service {
+class RemoteIndexServer final : public SymbolIndex::Service {
 public:
-  RemoteIndexServer(std::unique_ptr<SymbolIndex> Index)
+  RemoteIndexServer(std::unique_ptr<clangd::SymbolIndex> Index)
       : Index(std::move(Index)) {}
 
 private:
   grpc::Status Lookup(grpc::ServerContext *Context,
-                      const remote::LookupRequest *Request,
-                      grpc::ServerWriter<remote::LookupReply> *Reply) override {
-    llvm::outs() << "Lookup of symbol with ID " << Request->id() << '\n';
-    LookupRequest Req;
-    auto SID = SymbolID::fromStr(Request->id());
-    if (!SID) {
-      llvm::outs() << llvm::toString(SID.takeError()) << "\n";
+                      const LookupRequest *Request,
+                      grpc::ServerWriter<LookupReply> *Reply) override {
+    clangd::LookupRequest Req;
+    for (const auto &ID : Request->ids()) {
+      auto SID = SymbolID::fromStr(StringRef(ID));
+      if (!SID)
         return grpc::Status::CANCELLED;
-    }
+      Req.IDs.insert(*SID);
+    }
+    Index->lookup(Req, [&](const clangd::Symbol &Sym) {
+      LookupReply NextMessage;
+      *NextMessage.mutable_stream_result() = toProtobuf(Sym);
+      Reply->Write(NextMessage);
+    });
+    LookupReply LastMessage;
+    LastMessage.set_final_result(true);
+    Reply->Write(LastMessage);
+    return grpc::Status::OK;
+
+  grpc::Status FuzzyFind(grpc::ServerContext *Context,
+                         const FuzzyFindRequest *Request,
+                         grpc::ServerWriter<FuzzyFindReply> *Reply) override {
+    const auto Req = fromProtobuf(Request);
+    bool HasMore = Index->fuzzyFind(Req, [&](const clangd::Symbol &Sym) {
+      FuzzyFindReply NextMessage;
+      *NextMessage.mutable_stream_result() = toProtobuf(Sym);
+      Reply->Write(NextMessage);
+    });
+    FuzzyFindReply LastMessage;
+    LastMessage.set_final_result(HasMore);
+    Reply->Write(LastMessage);
+    return grpc::Status::OK;
+  }
+
+  grpc::Status Refs(grpc::ServerContext *Context, const RefsRequest *Request,
+                    grpc::ServerWriter<RefsReply> *Reply) override {
+    clangd::RefsRequest Req;
+    for (const auto &ID : Request->ids()) {
+      auto SID = SymbolID::fromStr(StringRef(ID));
+      if (!SID)
+        return grpc::Status::CANCELLED;
       Req.IDs.insert(*SID);
-    Index->lookup(Req, [&](const Symbol &Sym) {
-      remote::LookupReply NextSymbol;
-      NextSymbol.set_symbol_yaml(toYAML(Sym));
-      Reply->Write(NextSymbol);
+    }
+    bool HasMore = Index->refs(Req, [&](const clangd::Ref &Reference) {
+      RefsReply NextMessage;
+      *NextMessage.mutable_stream_result() = toProtobuf(Reference);
+      Reply->Write(NextMessage);
     });
+    RefsReply LastMessage;
+    LastMessage.set_final_result(HasMore);
+    Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
-  std::unique_ptr<SymbolIndex> Index;
+  std::unique_ptr<clangd::SymbolIndex> Index;
 };
 
-void runServer(std::unique_ptr<SymbolIndex> Index,
+void runServer(std::unique_ptr<clangd::SymbolIndex> Index,
                const std::string &ServerAddress) {
   RemoteIndexServer Service(std::move(Index));
 
   grpc::EnableDefaultHealthCheckService(true);
   grpc::ServerBuilder Builder;
   Builder.AddListeningPort(ServerAddress, grpc::InsecureServerCredentials());
   Builder.RegisterService(&Service);
   std::unique_ptr<grpc::Server> Server(Builder.BuildAndStart());
   llvm::outs() << "Server listening on " << ServerAddress << '\n';
 
   Server->Wait();
 }
 
 } // namespace
+} // namespace remote
 } // namespace clangd
 } // namespace clang
 
 int main(int argc, char *argv[]) {
-  using namespace clang::clangd;
-  llvm::cl::ParseCommandLineOptions(argc, argv, clang::clangd::Overview);
+  using namespace clang::clangd::remote;
+  llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
 
-  std::unique_ptr<SymbolIndex> Index = openIndex(IndexPath);
+  std::unique_ptr<clang::clangd::SymbolIndex> Index = openIndex(IndexPath);
 
   if (!Index) {
     llvm::outs() << "Failed to open the index.\n";
     return -1;
   }
 
   runServer(std::move(Index), ServerAddress);
 }

clang-tools-extra/clangd/index/remote/unimplemented/CMakeLists.txt

+include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../../../)
+# When compiled without Remote Index support, the real implementation index
+# client is not present. Users will get a notification about this when trying
+# to connect to remote index server instance.
+add_clang_library(clangdRemoteIndex
+  UnimplementedClient.cpp
+
+  LINK_LIBS
+  clangDaemon
+  )

clang-tools-extra/clangd/index/remote/unimplemented/UnimplementedClient.cpp

+//===--- UnimplementedClient.cpp ---------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "Logger.h"
+#include "index/remote/Client.h"
+
+namespace clang {
+namespace clangd {
+namespace remote {
+
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address) {
+  elog("Can't create SymbolIndex client without Remote Index support.");
+  return nullptr;
+}
+
+} // namespace remote
+} // namespace clangd
+} // namespace clang