[clangd] Implement path and URI translation for remote index

[kbobyrev]

%s/PrefixPath/ProjectRoot/g

[sammccall]

Generally I think this is missing high-level documentation describing the translation that happens at each level - I had to refer back to the notes from our meeting to understand what the scheme was.
I think Marshalling.h is probably a good place to document how this works. (Not exactly what prefixes get stripped by which function, but rather what the backing index contains, what goes over the wire, and what the clangd on the client side sees).

clang-tools-extra/clangd/index/remote/Client.cpp

101

this should be a std::string - no need to rely on the caller to keep the value alive

clang-tools-extra/clangd/index/remote/Client.h

21

This is a bit low-level - it describes what is being done but not really why. As a caller of this function it doesn't really tell me what I should pass.

Suggest something like "ProjectRoot is an absolute path on the local machine to the source tree described by the remote index. Paths returned by the index will be treated as relative to this directory."

I'd also suggest calling this IndexRoot instead of ProjectRoot, because the latter is a fairly overloaded concept and they need not coincide.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

34

mention somewhere that this is code used on the *client* side to do translation, and that RelativePath came from the server

38

scheme is always "file", no need to make it a parameter?

41

20 is too small. Suggest 256 or so (SmallString is on the stack, so it's cheap)

43

you can use URI::createFile which does less work and never fails

49

this looks like a no-op cast

52

Again this is a bit low level - "translates a URI from the server's backing index to a relative path suitable to send over the wire to the client"?

55

uriToRelativePath for consistency?

61

you're crashing here with an unhandled Expected. Add a test?
(I'd suggest elog or log, too)

64

static_cast is a pretty surprising/uncommon way to write this conversion....
I'd suggest either Result.str.str() or std::string(result)

65

need to handle the case where this returns false. Probably log and return ""?

Again, we should have a test for this.

141–142

this needs to do path translation in the case where the header is a URI

174

and here

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

46

nit: Clangd -> clangd

59–60

what does "serializes clangd" mean?

clang-tools-extra/clangd/index/remote/server/Server.cpp

35

nit: call this variable IndexRoot rather than IndexPrefixPath, for consistency?

69–70

for hygiene, I think we should pass this in as a constructor parameter rather than accessing the flag directly.

You probably also want to add a little validation: this needs to be an absolute path, and you may need to convert slashes to unix-style, too.

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

102

this is testPath("unittest"), isn't it?

I mean, forward vs backslashes are going to differ, but the current expression is going to produce C:\test/unittest which doesn't seem better.

186

test name is backwards - TEST(RemoteMarshallingTest, URITranslation) ?

[sammccall]

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

39

we should also bail out if RelativePath is an absolute path.

65

we really should ensure the result has unix slashes.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

59–60

The Ref marshaller isn't an appropriate place to put the detailed documentation on how path translation works.
It could go in the file documentation, or we could wrap the marshalling methods into a class we could put it there.
(The latter is somewhat appealing because it means you don't need to explicitly pass the root around everywhere, you can do normalization and have a place to store it, etc)

60

Somewhere we need to describe what the format of ProjectRoot is: trailing slash or not, unix slashes or native, etc.
(Even if the current code works in any variation, it's easier to reason about if we have a tightly "normalized" form)

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

100–101

Is passing the empty string here actually valid? I think you probably want to pass testRoot() or some unixified equivalent

186

I think this test is written in a way where it won't work on windows.
I'd suggest making heavier use of testPath() instead.

191

this is not (lexically) a relative path, it's an absolute path.
For the way it's used in this test it doesn't matter that much (just confusing)

196

OK, this assertion does actually show that we're sending absolute paths over the wire and calling them relative.
The confusion here is more serious because it's in the wire format and harder to address later.
There are other technical reasons: true relative paths still leave us the absolute-path namespace to represent paths outside the source tree if we want that for some reason.

[kbobyrev]

Store the progress so that I don't lose it. Still WIP.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

64

Okay, sure! I just thought .str().str() looks really weird, but std::string is probably a good choice.

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

100–101

Why testroot? That's what I'm stripping from URI body, the URI is unittest:///TestTU.h, hence I'm stripping from /TestTU.h, there is no /clangd-test there. Could you elaborate?

102

I don't understand: std::string(testRoot()) + '/' is /clangd-test/ on unix and testPath("unittest") is /clangd-test/unittest. I understand the slashes argument, but could you please elaborate on the former?

[kbobyrev]

Resolve the rest of patch comments.

[kbobyrev]

Remove IndexRoot conversion to UNIX slashes from the Marshalling hot path.

[kbobyrev]

Use std::string() instead of static_cast<std::string>()

[kbobyrev]

Don't convert test paths to system native, UNIX native is good enough.

[kbobyrev]

Convert prefix paths to native slash format for _serialization_ calls
(deserializations should use UNIX slashes).

[kbobyrev]

Add assertions for ensuring paths have UNIX slashes during deserialization.

[sammccall]

This looks better to me but:

• i think we need to consider what happens when paths are not under the expected path, and handle that in some appropriate way
• I find it hard to see what the tests are actually testing - would be good to make them more explicit, using a few minimal helpers, and avoiding fancy abstractions like URI schemes

clang-tools-extra/clangd/index/remote/Index.proto

102–105

storees -> stores

isn't it conceptually simpler to talk about a "relative path" rather than a "part of a URI body"?
(apart from anything else, no need to worry about whether it's escaped or not...)

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

38

mention error return (or return Optional)

54

this should check the scheme too - it must be file
(we can't reasonably treat IndexedProjectRoot as scheme-relative... but not to any *particular* scheme!)

127

nit: indexedprojectroot -> indexroot, unless there's some reason for the asymmetry? it seems like the same concept in the opposite direction

134–135

how do you want to handle the case where the path is in fact not relative stored under the root?

142

Don't try to parse if it's not a URI (starts with < or ").
As written this code will crash in debug mode if it's not a URI, due to unhandled error. Please add a test!

166

you seem to have this check in a few places - I'd suggest it should be in the relativePathToURI function and there should be some plan for handling errors.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

15

nit: llvm -> llvm-project

46

again, one particular overload is not really the place for this comment, I think. (File comment looks good)

55–56

this also needs the index root, for the proximity path (I'm not sure why they're not URIs, that's unfortunate...)

clang-tools-extra/clangd/index/remote/server/Server.cpp

49

please pass this in as a constructor param rather than accessing the flag directly

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

31

The File scheme is special. Using a different URI scheme here when we only support file in production is confusing and seems like a last resort to be used when we can't make it work any other way.
(For example we use it in lit tests because we must specify input filenames and the presence/absence of drive letters caused problems)
What's being solved there that the much smaller hammer of testPath() plus a local testPathURI() helper can't solve?

84

At this point I really think we should be writing these symbols out instead of trying to parse them from C++ - it's pretty hard to see exactly what fields are set and what is tested when the test is so highly factored.

If you like this approach think everything new is covered in other tests, I'd just remove the changes to this one.

102

sorry, I guess I meant testRoot("/").

190

auto -> std::string (here and in other places where the type is neither overly unreadable nor spelled in the function name)

190

it's not great practice to be assembling the test data from prefix + relative if the code we're testing is splitting it into prefix + relative - easy for the same bugs to cancel each other out.

Is testPath("remote/dir/MarshallingTests.cpp") directly too long?

194

(I'd suggest a helper testPathURI("remote/dir/MarshallingTests.cpp") for this)

198

hmm, I don't think that's the reason.
Either that's the contract of fromProtobuf, or it's not required and we shouldn't do it.

214

nit: use a file path rather than directory?

[kbobyrev]

Store progress. Still WIP.

[kbobyrev]

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

31

This is dealing with the consequences of test infrastructure being set up in a way that makes all URIs in Symbols and Refs use "unittest" scheme, this helper simply translates those URIs into "file" URIs which are the ones being supported in production. I can not understand why this is not desirable but I'll just clear all URIs and leave those two tests as sanity checks.

198

Ah, llvm::sys::path::append already does that for me. Okay, makes sense to remove this then.

[kbobyrev]

Address the rest of the comments and refactor code.

Patch is ready for the review again.

[kbobyrev]

Fix formatting on one line.

[kbobyrev]

Rebase on top of master

[kbobyrev]

Replace Sym in Client response with a more generic name (it can be a reference).

[sammccall]

Rest is just nits, thanks!

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

38

these functions are nice. It might be clearer to expose them and test them directly, particularly for error cases. Up to you.

39

can we add an assert that the index root is valid? (ends with a platform-appropriate slash)

45

message needs a bit more context. Remote index client got absolute path from server: {0}?

50

nit: lots of const auto throughout - we tend not to use const for local values (as opposed to references or pointers) so this is a bit confusing, particularly when combined with auto. Up to you though.

57

can we add an assert that IndexRoot is valid?

60

Similarly: Remote index got bad URI from client: ...

64

I think it's fine to fold this into the previous check: "bad URI" covers it

68

This isn't a valid IndexRoot, we should assert rather than handle this case I think.

131

shouldn't this return llvm::None, too? a SymbolLocation isn't valid without a path.

145

nit: isLiteralInclude in Headers.h (sorry, had forgotten about this func).
Calling that function probably avoids the need for the comment too.

149

isn't the rest of this uriToRelativePath?

210

{} with no index is a nice idea but I think not actually supported

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

19

nit "is passed passed to the remote server".

20

"It contains absolute path to the project root... relative path has unix slashes"

can we pull this out into a separate paragraph at the end? The rest is describing the flow, and this describes an API detail.

20

nit: "*the* absolute path", "*the* relative path" etc.

[kbobyrev]

Address post-LGTM round of comments.

[kbobyrev]

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

64

But it has different message :( ParsedURI.takeError wouldn't make sense here.

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

 
 llvm::cl::opt<std::string> IndexLocation(
     llvm::cl::desc("<path to index file | remote:server.address>"),
     llvm::cl::Positional);
 
 llvm::cl::opt<std::string>
     ExecCommand("c", llvm::cl::desc("Command to execute and then exit"));
 
+llvm::cl::opt<std::string> ProjectRoot("project-root",
+                                       llvm::cl::desc("Path to the project"));
+
 static constexpr char Overview[] = R"(
 This is an **experimental** interactive tool to process user-provided search
 queries over given symbol collection obtained via clangd-indexer. The
 tool can be used to evaluate search quality of existing index implementations
 and manually construct non-trivial test cases.
 
 You can connect to remote index by passing remote:address to dexp. Example:
 
std::function<std::unique_ptr<Command>()> Implementation;
     {"lookup", "Dump symbol details by ID or qualified name",
      std::make_unique<Lookup>},
     {"refs", "Find references by ID or qualified name", std::make_unique<Refs>},
     {"export", "Export index", std::make_unique<Export>},
 };
 
 std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
   return Index.startswith("remote:")
-             ? remote::getClient(Index.drop_front(strlen("remote:")))
+             ? remote::getClient(Index.drop_front(strlen("remote:")),
+                                 ProjectRoot)
              : loadIndex(Index, /*UseDex=*/true);
 }
 
 bool runCommand(std::string Request, const SymbolIndex &Index) {
   // Split on spaces and add required null-termination.
   std::replace(Request.begin(), Request.end(), ' ', '\0');
   llvm::SmallVector<llvm::StringRef, 8> Args;
   llvm::StringRef(Request).split(Args, '\0', /*MaxSplit=*/-1,

clang-tools-extra/clangd/index/remote/Client.h

 //===--- Client.h - Connect to a remote index via gRPC -----------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H
 
 #include "index/Index.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
 /// Returns an SymbolIndex client that passes requests to remote index located
 /// at \p Address. The client allows synchronous RPC calls.
+/// \p IndexRoot is an absolute path on the local machine to the source tree
+/// described by the remote index. Paths returned by the index will be treated
+/// as relative to this directory.
 ///
 /// This method attempts to resolve the address and establish the connection.
 ///
 /// \returns nullptr if the address is not resolved during the function call or
 /// if the project was compiled without Remote Index support.
-std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address);
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address,
+                                               llvm::StringRef IndexRoot);
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H

clang-tools-extra/clangd/index/remote/Client.cpp

 //===--- Client.cpp ----------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include <grpc++/grpc++.h>
 
 #include "Client.h"
 #include "Index.grpc.pb.h"
+#include "index/Index.h"
 #include "index/Serialization.h"
 #include "marshalling/Marshalling.h"
 #include "support/Logger.h"
 #include "support/Trace.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 namespace {
 
 class IndexClient : public clangd::SymbolIndex {
   template <typename RequestT, typename ReplyT>
   using StreamingCall = std::unique_ptr<grpc::ClientReader<ReplyT>> (
       remote::SymbolIndex::Stub::*)(grpc::ClientContext *, const RequestT &);
 
+  template <typename ClangdRequestT, typename RequestT>
+  RequestT serializeRequest(ClangdRequestT Request) const {
+    return toProtobuf(Request);
+  }
+
+  template <>
+  FuzzyFindRequest serializeRequest(clangd::FuzzyFindRequest Request) const {
+    return toProtobuf(Request, ProjectRoot);
+  }
+
   // FIXME(kirillbobyrev): Set deadlines for requests.
   template <typename RequestT, typename ReplyT, typename ClangdRequestT,
             typename CallbackT>
   bool streamRPC(ClangdRequestT Request,
                  StreamingCall<RequestT, ReplyT> RPCCall,
                  CallbackT Callback) const {
     bool FinalResult = false;
     trace::Span Tracer(RequestT::descriptor()->name());
-    const auto RPCRequest = toProtobuf(Request);
+    const auto RPCRequest = serializeRequest<ClangdRequestT, RequestT>(Request);
     grpc::ClientContext Context;
     auto Reader = (Stub.get()->*RPCCall)(&Context, RPCRequest);
     llvm::BumpPtrAllocator Arena;
     llvm::UniqueStringSaver Strings(Arena);
     ReplyT Reply;
     while (Reader->Read(&Reply)) {
       if (!Reply.has_stream_result()) {
         FinalResult = Reply.final_result();
         continue;
       }
-      auto Sym = fromProtobuf(Reply.stream_result(), &Strings);
+      auto Sym = fromProtobuf(Reply.stream_result(), &Strings, ProjectRoot);
       if (!Sym)
         elog("Received invalid {0}", ReplyT::descriptor()->name());
       Callback(*Sym);
     }
     SPAN_ATTACH(Tracer, "status", Reader->Finish().ok());
     return FinalResult;
   }
 
 public:
-  IndexClient(std::shared_ptr<grpc::Channel> Channel)
-      : Stub(remote::SymbolIndex::NewStub(Channel)) {}
+  IndexClient(std::shared_ptr<grpc::Channel> Channel,
+              llvm::StringRef ProjectRoot)
+      : Stub(remote::SymbolIndex::NewStub(Channel)), ProjectRoot(ProjectRoot) {}
 
   void lookup(const clangd::LookupRequest &Request,
               llvm::function_ref<void(const clangd::Symbol &)> Callback) const {
     streamRPC(Request, &remote::SymbolIndex::Stub::Lookup, Callback);
   }
 
   bool
   fuzzyFind(const clangd::FuzzyFindRequest &Request,
             llvm::function_ref<void(const clangd::Symbol &)> Callback) const {
     return streamRPC(Request, &remote::SymbolIndex::Stub::FuzzyFind, Callback);
   }
 
   bool refs(const clangd::RefsRequest &Request,
             llvm::function_ref<void(const clangd::Ref &)> Callback) const {
     return streamRPC(Request, &remote::SymbolIndex::Stub::Refs, Callback);
   }
 
   // FIXME(kirillbobyrev): Implement this.
   void
   relations(const clangd::RelationsRequest &,
             llvm::function_ref<void(const SymbolID &, const clangd::Symbol &)>)
       const {}
 
-  // IndexClient does not take any space since the data is stored on the server.
+  // IndexClient does not take any space since the data is stored on the
+  // server.
   size_t estimateMemoryUsage() const { return 0; }
 
 private:
   std::unique_ptr<remote::SymbolIndex::Stub> Stub;
+  std::string ProjectRoot;
 };
 
 } // namespace
 
-std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address) {
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address,
+                                               llvm::StringRef ProjectRoot) {
   const auto Channel =
       grpc::CreateChannel(Address.str(), grpc::InsecureChannelCredentials());
   Channel->GetState(true);
-  return std::unique_ptr<clangd::SymbolIndex>(new IndexClient(Channel));
+  return std::unique_ptr<clangd::SymbolIndex>(
+      new IndexClient(Channel, ProjectRoot));
 }
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/remote/Index.proto

 }
 
 message Symbol {
   string id = 1;
   SymbolInfo info = 2;
   string name = 3;
   SymbolLocation definition = 4;
   string scope = 5;
-  SymbolLocation canonical_declarattion = 6;
+  SymbolLocation canonical_declaration = 6;
   int32 references = 7;
   uint32 origin = 8;
   string signature = 9;
   string template_specialization_args = 10;
   string completion_snippet_suffix = 11;
   string documentation = 12;
   string return_type = 13;
   string type = 14;
message SymbolInfo {
   uint32 subkind = 2;
   uint32 language = 3;
   uint32 properties = 4;
 }
 
 message SymbolLocation {
   Position start = 1;
   Position end = 2;
-  string file_uri = 3;
+  // clangd::SymbolLocation stores FileURI, but the protocol transmits a the
+  // relative path. Because paths are different on the remote and local machines
+  // they will be translated in the marshalling layer.
+  string file_path = 3;
 }
 
 message Position {
   uint32 line = 1;
   uint32 column = 2;
 }
 
 message HeaderWithReferences {
   string header = 1;
   uint32 references = 2;
 }

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

 //===--- Marshalling.h -------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// Transformations between native Clangd types and Protobuf-generated classes.
+// Marshalling provides translation between native clangd types into the
+// Protobuf-generated classes. Most translations are 1-to-1 and wrap variables
+// into appropriate Protobuf types.
+//
+// A notable exception is URI translation. Because paths to files are different
+// on indexing machine and client machine
+// ("/remote/machine/projects/llvm-project/llvm/include/HelloWorld.h" versus
+// "/usr/local/username/llvm-project/llvm/include/HelloWorld.h"), they need to
+// be converted appropriately. Remote machine strips the prefix from an absolute
+// path and passes paths relative to the project root over the wire
+// ("include/HelloWorld.h" in this example). The indexed project root is passed
+// passed to the remote server. It contains absolute path to the project root
+// and includes a trailing slash. Relative path passed over the wire has unix
+// slashes. Client receives this relative path and constructs a URI that points
+// to the relevant file in the filesystem. Relative path is appended to
+// IndexRoot to construct the full path and build the final URI.
+//
+// toProtobuf() functions serialize native clangd types and strip IndexRoot from
+// the file paths specific to indexing machine. fromProtobuf() functions
+// deserialize clangd types and translate relative paths into machine-native
+// URIs.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H
 
 #include "Index.pb.h"
 #include "index/Index.h"
+#include "llvm/ADT/StringRef.h"
 #include "llvm/Support/StringSaver.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
-clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request);
+clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request,
+                                      llvm::StringRef IndexRoot);
 llvm::Optional<clangd::Symbol> fromProtobuf(const Symbol &Message,
-                                            llvm::UniqueStringSaver *Strings);
+                                            llvm::UniqueStringSaver *Strings,
+                                            llvm::StringRef IndexRoot);
 llvm::Optional<clangd::Ref> fromProtobuf(const Ref &Message,
-                                         llvm::UniqueStringSaver *Strings);
+                                         llvm::UniqueStringSaver *Strings,
+                                         llvm::StringRef IndexRoot);
 
 LookupRequest toProtobuf(const clangd::LookupRequest &From);
-FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From);
+FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From,
+                            llvm::StringRef IndexRoot);
 RefsRequest toProtobuf(const clangd::RefsRequest &From);
 
-Ref toProtobuf(const clangd::Ref &From);
-Symbol toProtobuf(const clangd::Symbol &From);
+Ref toProtobuf(const clangd::Ref &From, llvm::StringRef IndexRoot);
+Symbol toProtobuf(const clangd::Symbol &From, llvm::StringRef IndexRoot);
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

 #include "Protocol.h"
 #include "index/Serialization.h"
 #include "index/Symbol.h"
 #include "index/SymbolID.h"
 #include "index/SymbolLocation.h"
 #include "index/SymbolOrigin.h"
 #include "support/Logger.h"
 #include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/None.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
 namespace {
 
+/// Translates \p RelativePath into the absolute path and builds URI for the
+/// user machine. This translation happens on the client side with the
+/// \p RelativePath received from remote index server and \p IndexRoot is
+/// provided by the client.
+llvm::Optional<std::string> relativePathToURI(llvm::StringRef RelativePath,
+                                              llvm::StringRef IndexRoot) {
+  assert(RelativePath == llvm::sys::path::convert_to_slash(
+                             RelativePath, llvm::sys::path::Style::posix));
+  if (RelativePath.empty())
+    return std::string();
+  if (llvm::sys::path::is_absolute(RelativePath)) {
+    elog("{0} is not relative path", RelativePath);
+    return llvm::None;
+  }
+  llvm::SmallString<256> FullPath = IndexRoot;
+  llvm::sys::path::append(FullPath, RelativePath);
+  const auto Result = URI::createFile(FullPath);
+  return Result.toString();
+}
+
+/// Translates a URI from the server's backing index to a relative path suitable
+/// to send over the wire to the client.
+llvm::Optional<std::string> uriToRelativePath(llvm::StringRef URI,
+                                              llvm::StringRef IndexRoot) {
+  auto ParsedURI = URI::parse(URI);
+  if (!ParsedURI) {
+    elog("Can not parse URI {0}: {1}", URI, ParsedURI.takeError());
+    return llvm::None;
+  }
+  if (ParsedURI->scheme() != "file") {
+    elog("Can not parse URI with scheme other than \"file\" {0}", URI);
+    return llvm::None;
+  }
+  llvm::SmallString<256> Result = ParsedURI->body();
+  if (IndexRoot.empty())
+    return std::string(Result);
+  if (!llvm::sys::path::replace_path_prefix(Result, IndexRoot, "")) {
+    elog("Can not get relative path from the URI {0} given the index root {1}",
+         URI, IndexRoot);
+    return llvm::None;
+  }
+  // Make sure the result has UNIX slashes.
+  return llvm::sys::path::convert_to_slash(Result,
+                                           llvm::sys::path::Style::posix);
+}
+
 clangd::SymbolLocation::Position fromProtobuf(const Position &Message) {
   clangd::SymbolLocation::Position Result;
   Result.setColumn(static_cast<uint32_t>(Message.column()));
   Result.setLine(static_cast<uint32_t>(Message.line()));
   return Result;
 }
 
 Position toProtobuf(const clangd::SymbolLocation::Position &Position) {
SymbolInfo toProtobuf(const clang::index::SymbolInfo &Info) {
   SymbolInfo Result;
   Result.set_kind(static_cast<uint32_t>(Info.Kind));
   Result.set_subkind(static_cast<uint32_t>(Info.SubKind));
   Result.set_language(static_cast<uint32_t>(Info.Lang));
   Result.set_properties(static_cast<uint32_t>(Info.Properties));
   return Result;
 }
 
-clangd::SymbolLocation fromProtobuf(const SymbolLocation &Message,
-                                    llvm::UniqueStringSaver *Strings) {
+llvm::Optional<clangd::SymbolLocation>
+fromProtobuf(const SymbolLocation &Message, llvm::UniqueStringSaver *Strings,
+             llvm::StringRef IndexRoot) {
   clangd::SymbolLocation Location;
+  const auto URIString = relativePathToURI(Message.file_path(), IndexRoot);
+  if (!URIString)
+    return llvm::None;
+  Location.FileURI = Strings->save(*URIString).begin();
   Location.Start = fromProtobuf(Message.start());
   Location.End = fromProtobuf(Message.end());
-  Location.FileURI = Strings->save(Message.file_uri()).begin();
   return Location;
 }
 
-SymbolLocation toProtobuf(const clangd::SymbolLocation &Location) {
+SymbolLocation toProtobuf(const clangd::SymbolLocation &Location,
+                          llvm::StringRef IndexRoot) {
   remote::SymbolLocation Result;
+  const auto RelativePath = uriToRelativePath(Location.FileURI, IndexRoot);
+  if (!RelativePath)
+    return Result;
+  *Result.mutable_file_path() = *RelativePath;
   *Result.mutable_start() = toProtobuf(Location.Start);
   *Result.mutable_end() = toProtobuf(Location.End);
-  *Result.mutable_file_uri() = Location.FileURI;
   return Result;
 }
 
-HeaderWithReferences
-toProtobuf(const clangd::Symbol::IncludeHeaderWithReferences &IncludeHeader) {
+llvm::Optional<HeaderWithReferences>
+toProtobuf(const clangd::Symbol::IncludeHeaderWithReferences &IncludeHeader,
+           llvm::StringRef IndexRoot) {
   HeaderWithReferences Result;
-  Result.set_header(IncludeHeader.IncludeHeader.str());
   Result.set_references(IncludeHeader.References);
+  const std::string Header = IncludeHeader.IncludeHeader.str();
+  // If header starts with < or " it is not a URI: do not attempt to parse it.
+  if (Header.front() == '<' || Header.front() == '"') {
+    Result.set_header(Header);
+    return Result;
+  }
+  auto URI = URI::parse(Header);
+  if (!URI) {
+    elog("Can not parse header URI {0} from HeaderWithReferences: {1}", Header,
+         URI.takeError());
+    return llvm::None;
+  }
+  const auto RelativePath =
+      uriToRelativePath(IncludeHeader.IncludeHeader.str(), IndexRoot);
+  if (!RelativePath)
+    return llvm::None;
+  Result.set_header(*RelativePath);
   return Result;
 }
 
-clangd::Symbol::IncludeHeaderWithReferences
-fromProtobuf(const HeaderWithReferences &Message) {
-  return clangd::Symbol::IncludeHeaderWithReferences{Message.header(),
+llvm::Optional<clangd::Symbol::IncludeHeaderWithReferences>
+fromProtobuf(const HeaderWithReferences &Message,
+             llvm::UniqueStringSaver *Strings, llvm::StringRef IndexRoot) {
+  std::string Header = Message.header();
+  if (Header.front() != '<' && Header.front() != '"') {
+    const auto URIString = relativePathToURI(Header, IndexRoot);
+    if (!URIString)
+      return llvm::None;
+    Header = *URIString;
+  }
+  return clangd::Symbol::IncludeHeaderWithReferences{Strings->save(Header),
                                                      Message.references()};
 }
 
 } // namespace
 
-clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request) {
+clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request,
+                                      llvm::StringRef IndexRoot) {
   clangd::FuzzyFindRequest Result;
   Result.Query = Request->query();
   for (const auto &Scope : Request->scopes())
     Result.Scopes.push_back(Scope);
   Result.AnyScope = Request->any_scope();
   if (Request->limit())
     Result.Limit = Request->limit();
   Result.RestrictForCodeCompletion = Request->restricted_for_code_completion();
-  for (const auto &Path : Request->proximity_paths())
-    Result.ProximityPaths.push_back(Path);
+  for (const auto &Path : Request->proximity_paths()) {
+    llvm::SmallString<256> LocalPath = llvm::StringRef(IndexRoot);
+    llvm::sys::path::append(LocalPath, Path);
+    Result.ProximityPaths.push_back(std::string(LocalPath));
+  }
   for (const auto &Type : Request->preferred_types())
     Result.ProximityPaths.push_back(Type);
   return Result;
 }
 
 llvm::Optional<clangd::Symbol> fromProtobuf(const Symbol &Message,
-                                            llvm::UniqueStringSaver *Strings) {
+                                            llvm::UniqueStringSaver *Strings,
+                                            llvm::StringRef IndexRoot) {
   if (!Message.has_info() || !Message.has_definition() ||
-      !Message.has_canonical_declarattion()) {
+      !Message.has_canonical_declaration()) {
     elog("Cannot convert Symbol from Protobuf: {}", Message.ShortDebugString());
     return llvm::None;
   }
   clangd::Symbol Result;
   auto ID = SymbolID::fromStr(Message.id());
   if (!ID) {
-    elog("Cannot convert parse SymbolID {} from Protobuf: {}", ID.takeError(),
+    elog("Cannot parse SymbolID {} given Protobuf: {}", ID.takeError(),
          Message.ShortDebugString());
     return llvm::None;
   }
   Result.ID = *ID;
   Result.SymInfo = fromProtobuf(Message.info());
   Result.Name = Message.name();
   Result.Scope = Message.scope();
-  Result.Definition = fromProtobuf(Message.definition(), Strings);
-  Result.CanonicalDeclaration =
-      fromProtobuf(Message.canonical_declarattion(), Strings);
+  const auto Definition =
+      fromProtobuf(Message.definition(), Strings, IndexRoot);
+  if (!Definition)
+    return llvm::None;
+  Result.Definition = *Definition;
+  const auto Declaration =
+      fromProtobuf(Message.canonical_declaration(), Strings, IndexRoot);
+  if (!Declaration)
+    return llvm::None;
+  Result.CanonicalDeclaration = *Declaration;
   Result.References = Message.references();
   Result.Origin = static_cast<clangd::SymbolOrigin>(Message.origin());
   Result.Signature = Message.signature();
   Result.TemplateSpecializationArgs = Message.template_specialization_args();
   Result.CompletionSnippetSuffix = Message.completion_snippet_suffix();
   Result.Documentation = Message.documentation();
   Result.ReturnType = Message.return_type();
   Result.Type = Message.type();
   for (const auto &Header : Message.headers()) {
-    Result.IncludeHeaders.push_back(fromProtobuf(Header));
+    const auto SerializedHeader = fromProtobuf(Header, Strings, IndexRoot);
+    if (SerializedHeader)
+      Result.IncludeHeaders.push_back(*SerializedHeader);
   }
   Result.Flags = static_cast<clangd::Symbol::SymbolFlag>(Message.flags());
   return Result;
 }
 
 llvm::Optional<clangd::Ref> fromProtobuf(const Ref &Message,
-                                         llvm::UniqueStringSaver *Strings) {
+                                         llvm::UniqueStringSaver *Strings,
+                                         llvm::StringRef IndexRoot) {
   if (!Message.has_location()) {
     elog("Cannot convert Ref from Protobuf: {}", Message.ShortDebugString());
     return llvm::None;
   }
   clangd::Ref Result;
-  Result.Location = fromProtobuf(Message.location(), Strings);
+  const auto Location = fromProtobuf(Message.location(), Strings, IndexRoot);
+  if (!Location)
+    return llvm::None;
+  Result.Location = *Location;
   Result.Kind = static_cast<clangd::RefKind>(Message.kind());
   return Result;
 }
 
 LookupRequest toProtobuf(const clangd::LookupRequest &From) {
   LookupRequest RPCRequest;
   for (const auto &SymbolID : From.IDs)
     RPCRequest.add_ids(SymbolID.str());
   return RPCRequest;
 }
 
-FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From) {
+FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From,
+                            llvm::StringRef IndexRoot) {
   FuzzyFindRequest RPCRequest;
   RPCRequest.set_query(From.Query);
   for (const auto &Scope : From.Scopes)
     RPCRequest.add_scopes(Scope);
   RPCRequest.set_any_scope(From.AnyScope);
   if (From.Limit)
     RPCRequest.set_limit(*From.Limit);
   RPCRequest.set_restricted_for_code_completion(From.RestrictForCodeCompletion);
-  for (const auto &Path : From.ProximityPaths)
-    RPCRequest.add_proximity_paths(Path);
+  for (const auto &Path : From.ProximityPaths) {
+    llvm::SmallString<256> RelativePath = llvm::StringRef(Path);
+    if (llvm::sys::path::replace_path_prefix(RelativePath, IndexRoot, ""))
+      RPCRequest.add_proximity_paths(llvm::sys::path::convert_to_slash(
+          RelativePath, llvm::sys::path::Style::posix));
+  }
   for (const auto &Type : From.PreferredTypes)
     RPCRequest.add_preferred_types(Type);
   return RPCRequest;
 }
 
 RefsRequest toProtobuf(const clangd::RefsRequest &From) {
   RefsRequest RPCRequest;
   for (const auto &ID : From.IDs)
     RPCRequest.add_ids(ID.str());
   RPCRequest.set_filter(static_cast<uint32_t>(From.Filter));
   if (From.Limit)
     RPCRequest.set_limit(*From.Limit);
   return RPCRequest;
 }
 
-Symbol toProtobuf(const clangd::Symbol &From) {
+Symbol toProtobuf(const clangd::Symbol &From, llvm::StringRef IndexRoot) {
   Symbol Result;
   Result.set_id(From.ID.str());
   *Result.mutable_info() = toProtobuf(From.SymInfo);
   Result.set_name(From.Name.str());
-  *Result.mutable_definition() = toProtobuf(From.Definition);
+  *Result.mutable_definition() = toProtobuf(From.Definition, IndexRoot);
   Result.set_scope(From.Scope.str());
-  *Result.mutable_canonical_declarattion() =
-      toProtobuf(From.CanonicalDeclaration);
+  *Result.mutable_canonical_declaration() =
+      toProtobuf(From.CanonicalDeclaration, IndexRoot);
   Result.set_references(From.References);
   Result.set_origin(static_cast<uint32_t>(From.Origin));
   Result.set_signature(From.Signature.str());
   Result.set_template_specialization_args(
       From.TemplateSpecializationArgs.str());
   Result.set_completion_snippet_suffix(From.CompletionSnippetSuffix.str());
   Result.set_documentation(From.Documentation.str());
   Result.set_return_type(From.ReturnType.str());
   Result.set_type(From.Type.str());
   for (const auto &Header : From.IncludeHeaders) {
+    const auto Serialized = toProtobuf(Header, IndexRoot);
+    if (!Serialized)
+      continue;
     auto *NextHeader = Result.add_headers();
-    *NextHeader = toProtobuf(Header);
+    *NextHeader = *Serialized;
   }
   Result.set_flags(static_cast<uint32_t>(From.Flags));
   return Result;
 }
 
-Ref toProtobuf(const clangd::Ref &From) {
+Ref toProtobuf(const clangd::Ref &From, llvm::StringRef IndexRoot) {
   Ref Result;
   Result.set_kind(static_cast<uint32_t>(From.Kind));
-  *Result.mutable_location() = toProtobuf(From.Location);
+  *Result.mutable_location() = toProtobuf(From.Location, IndexRoot);
   return Result;
 }
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/remote/server/Server.cpp

 //===--- Server.cpp - gRPC-based Remote Index Server  ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "index/Index.h"
 #include "index/Serialization.h"
 #include "index/remote/marshalling/Marshalling.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/Signals.h"
 
 #include <grpc++/grpc++.h>
 #include <grpc++/health_check_service_interface.h>
 
 #include "Index.grpc.pb.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 namespace {
 
 static constexpr char Overview[] = R"(
 This is an experimental remote index implementation. The server opens Dex and
 awaits gRPC lookup requests from the client.
 )";
 
 llvm::cl::opt<std::string> IndexPath(llvm::cl::desc("<INDEX FILE>"),
                                      llvm::cl::Positional, llvm::cl::Required);
 
+llvm::cl::opt<std::string> IndexRoot(llvm::cl::desc("<PROJECT ROOT>"),
+                                     llvm::cl::Positional, llvm::cl::Required);
+
 llvm::cl::opt<std::string> ServerAddress(
     "server-address", llvm::cl::init("0.0.0.0:50051"),
     llvm::cl::desc("Address of the invoked server. Defaults to 0.0.0.0:50051"));
 
 std::unique_ptr<clangd::SymbolIndex> openIndex(llvm::StringRef Index) {
   return loadIndex(Index, /*UseIndex=*/true);
 }
 
 class RemoteIndexServer final : public SymbolIndex::Service {
 public:
-  RemoteIndexServer(std::unique_ptr<clangd::SymbolIndex> Index)
-      : Index(std::move(Index)) {}
+  RemoteIndexServer(std::unique_ptr<clangd::SymbolIndex> Index,
+                    llvm::StringRef IndexRoot)
+      : Index(std::move(Index)) {
+    llvm::SmallString<256> NativePath = IndexRoot;
+    llvm::sys::path::native(NativePath);
+    IndexedProjectRoot = std::string(NativePath);
+  }
 
 private:
   grpc::Status Lookup(grpc::ServerContext *Context,
                       const LookupRequest *Request,
                       grpc::ServerWriter<LookupReply> *Reply) override {
     clangd::LookupRequest Req;
     for (const auto &ID : Request->ids()) {
       auto SID = SymbolID::fromStr(StringRef(ID));
       if (!SID)
         return grpc::Status::CANCELLED;
       Req.IDs.insert(*SID);
     }
     Index->lookup(Req, [&](const clangd::Symbol &Sym) {
       LookupReply NextMessage;
-      *NextMessage.mutable_stream_result() = toProtobuf(Sym);
+      *NextMessage.mutable_stream_result() =
+          toProtobuf(Sym, IndexedProjectRoot);
       Reply->Write(NextMessage);
     });
     LookupReply LastMessage;
     LastMessage.set_final_result(true);
     Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
   grpc::Status FuzzyFind(grpc::ServerContext *Context,
                          const FuzzyFindRequest *Request,
                          grpc::ServerWriter<FuzzyFindReply> *Reply) override {
-    const auto Req = fromProtobuf(Request);
+    const auto Req = fromProtobuf(Request, IndexedProjectRoot);
     bool HasMore = Index->fuzzyFind(Req, [&](const clangd::Symbol &Sym) {
       FuzzyFindReply NextMessage;
-      *NextMessage.mutable_stream_result() = toProtobuf(Sym);
+      *NextMessage.mutable_stream_result() =
+          toProtobuf(Sym, IndexedProjectRoot);
       Reply->Write(NextMessage);
     });
     FuzzyFindReply LastMessage;
     LastMessage.set_final_result(HasMore);
     Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
   grpc::Status Refs(grpc::ServerContext *Context, const RefsRequest *Request,
                     grpc::ServerWriter<RefsReply> *Reply) override {
     clangd::RefsRequest Req;
     for (const auto &ID : Request->ids()) {
       auto SID = SymbolID::fromStr(StringRef(ID));
       if (!SID)
         return grpc::Status::CANCELLED;
       Req.IDs.insert(*SID);
     }
     bool HasMore = Index->refs(Req, [&](const clangd::Ref &Reference) {
       RefsReply NextMessage;
-      *NextMessage.mutable_stream_result() = toProtobuf(Reference);
+      *NextMessage.mutable_stream_result() = toProtobuf(Reference, IndexRoot);
       Reply->Write(NextMessage);
     });
     RefsReply LastMessage;
     LastMessage.set_final_result(HasMore);
     Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
   std::unique_ptr<clangd::SymbolIndex> Index;
+  std::string IndexedProjectRoot;
 };
 
 void runServer(std::unique_ptr<clangd::SymbolIndex> Index,
                const std::string &ServerAddress) {
-  RemoteIndexServer Service(std::move(Index));
+  RemoteIndexServer Service(std::move(Index), IndexRoot);
 
   grpc::EnableDefaultHealthCheckService(true);
   grpc::ServerBuilder Builder;
   Builder.AddListeningPort(ServerAddress, grpc::InsecureServerCredentials());
   Builder.RegisterService(&Service);
   std::unique_ptr<grpc::Server> Server(Builder.BuildAndStart());
   llvm::outs() << "Server listening on " << ServerAddress << '\n';
 
   Server->Wait();
 }
 
 } // namespace
 } // namespace remote
 } // namespace clangd
 } // namespace clang
 
 int main(int argc, char *argv[]) {
   using namespace clang::clangd::remote;
   llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
 
+  if (!llvm::sys::path::is_absolute(IndexRoot)) {
+    llvm::outs() << "Index root should be an absolute path.\n";
+    return -1;
+  }
+
   std::unique_ptr<clang::clangd::SymbolIndex> Index = openIndex(IndexPath);
 
   if (!Index) {
     llvm::outs() << "Failed to open the index.\n";
     return -1;
   }
 
   runServer(std::move(Index), ServerAddress);
 }

clang-tools-extra/clangd/index/remote/unimplemented/UnimplementedClient.cpp

 //===--- UnimplementedClient.cpp ---------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "index/remote/Client.h"
 #include "support/Logger.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
-std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address) {
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address,
+                                               llvm::StringRef IndexRoot) {
   elog("Can't create SymbolIndex client without Remote Index support.");
   return nullptr;
 }
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

 //===--- MarshallingTests.cpp ------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "../TestTU.h"
+#include "TestFS.h"
+#include "index/Index.h"
+#include "index/Ref.h"
 #include "index/Serialization.h"
+#include "index/Symbol.h"
+#include "index/SymbolID.h"
 #include "index/remote/marshalling/Marshalling.h"
+#include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
+#include "gmock/gmock.h"
 #include "gtest/gtest.h"
+#include <cstring>
 
 namespace clang {
 namespace clangd {
 namespace remote {
 namespace {
 
-TEST(RemoteMarshallingTest, SymbolSerialization) {
-  const auto *Header = R"(
-  // This is a class.
-  class Foo {
-  public:
-    Foo();
-
-    int Bar;
-  private:
-    double Number;
-  };
-  /// This is a function.
-  char baz();
-  template <typename T>
-  T getT ();
-  )";
-  const auto TU = TestTU::withHeaderCode(Header);
-  const auto Symbols = TU.headerSymbols();
-  // Sanity check: there are more than 5 symbols available.
-  EXPECT_GE(Symbols.size(), 5UL);
+using llvm::sys::path::convert_to_slash;
+
+const char *testPathURI(llvm::StringRef Path,
+                        llvm::UniqueStringSaver &Strings) {
+  const auto URI = URI::createFile(testPath(Path));
+  return Strings.save(URI.toString()).begin();
+}
+
+TEST(RemoteMarshallingTest, URITranslation) {
   llvm::BumpPtrAllocator Arena;
   llvm::UniqueStringSaver Strings(Arena);
-  for (auto &Sym : Symbols) {
-    const auto ProtobufMeessage = toProtobuf(Sym);
-    const auto SymToProtobufAndBack = fromProtobuf(ProtobufMeessage, &Strings);
-    EXPECT_TRUE(SymToProtobufAndBack.hasValue());
-    EXPECT_EQ(toYAML(Sym), toYAML(*SymToProtobufAndBack));
-  }
+  clangd::Ref Original;
+  Original.Location.FileURI =
+      testPathURI("remote/machine/projects/llvm-project/clang-tools-extra/"
+                  "clangd/unittests/remote/MarshallingTests.cpp",
+                  Strings);
+  auto Serialized =
+      toProtobuf(Original, testPath("remote/machine/projects/llvm-project/"));
+  EXPECT_EQ(Serialized.location().file_path(),
+            "clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp");
+  const std::string LocalIndexPrefix = testPath("local/machine/project/");
+  auto Deserialized = fromProtobuf(Serialized, &Strings,
+                                   testPath("home/my-projects/llvm-project"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(Deserialized->Location.FileURI,
+            testPathURI("home/my-projects/llvm-project/clang-tools-extra/"
+                        "clangd/unittests/remote/MarshallingTests.cpp",
+                        Strings));
+
+  clangd::Ref WithInvalidURI;
+  // Invalid URI results in empty path.
+  WithInvalidURI.Location.FileURI = "This is not a URI";
+  Serialized = toProtobuf(WithInvalidURI, testPath("home/"));
+  EXPECT_EQ(Serialized.location().file_path(), "");
+
+  // Can not use URIs with scheme different from "file".
+  auto UnittestURI =
+      URI::create(testPath("project/lib/HelloWorld.cpp"), "unittest");
+  EXPECT_TRUE(bool(UnittestURI));
+  WithInvalidURI.Location.FileURI =
+      Strings.save(UnittestURI->toString()).begin();
+  Serialized = toProtobuf(WithInvalidURI, testPath("project/lib/"));
+  EXPECT_EQ(Serialized.location().file_path(), "");
+
+  Ref WithAbsolutePath;
+  *WithAbsolutePath.mutable_location()->mutable_file_path() =
+      "/usr/local/user/home/HelloWorld.cpp";
+  Deserialized = fromProtobuf(WithAbsolutePath, &Strings, LocalIndexPrefix);
+  // Paths transmitted over the wire can not be absolute, they have to be
+  // relative.
+  EXPECT_FALSE(Deserialized);
 }
 
-TEST(RemoteMarshallingTest, ReferenceSerialization) {
-  TestTU TU;
-  TU.HeaderCode = R"(
-  int foo();
-  int GlobalVariable = 42;
-  class Foo {
-  public:
-    Foo();
-
-    char Symbol = 'S';
-  };
-  template <typename T>
-  T getT() { return T(); }
-  )";
-  TU.Code = R"(
-  int foo() {
-    ++GlobalVariable;
-
-    Foo foo = Foo();
-    if (foo.Symbol - 'a' == 42) {
-      foo.Symbol = 'b';
-    }
+TEST(RemoteMarshallingTest, SymbolSerialization) {
+  clangd::Symbol Sym;
+
+  auto ID = SymbolID::fromStr("057557CEBF6E6B2D");
+  EXPECT_TRUE(bool(ID));
+  Sym.ID = *ID;
+
+  index::SymbolInfo Info;
+  Info.Kind = index::SymbolKind::Function;
+  Info.SubKind = index::SymbolSubKind::AccessorGetter;
+  Info.Lang = index::SymbolLanguage::CXX;
+  Info.Properties = static_cast<index::SymbolPropertySet>(
+      index::SymbolProperty::TemplateSpecialization);
+  Sym.SymInfo = Info;
+
+  llvm::BumpPtrAllocator Arena;
+  llvm::UniqueStringSaver Strings(Arena);
+
+  Sym.Name = Strings.save("Foo");
+  Sym.Scope = Strings.save("llvm::foo::bar::");
+
+  clangd::SymbolLocation Location;
+  Location.Start.setLine(1);
+  Location.Start.setColumn(15);
+  Location.End.setLine(3);
+  Location.End.setColumn(121);
+  Location.FileURI = testPathURI("home/Definition.cpp", Strings);
+  Sym.Definition = Location;
+
+  Location.Start.setLine(42);
+  Location.Start.setColumn(31);
+  Location.End.setLine(20);
+  Location.End.setColumn(400);
+  Location.FileURI = testPathURI("home/Declaration.h", Strings);
+  Sym.CanonicalDeclaration = Location;
+
+  Sym.References = 9000;
+  Sym.Origin = clangd::SymbolOrigin::Static;
+  Sym.Signature = Strings.save("(int X, char Y, Type T)");
+  Sym.TemplateSpecializationArgs = Strings.save("<int, char, bool, Type>");
+  Sym.CompletionSnippetSuffix =
+      Strings.save("({1: int X}, {2: char Y}, {3: Type T})");
+  Sym.Documentation = Strings.save("This is my amazing Foo constructor!");
+  Sym.ReturnType = Strings.save("Foo");
+
+  Sym.Flags = clangd::Symbol::SymbolFlag::IndexedForCodeCompletion;
+
+  // Check that symbols are exactly the same if the path to indexed project is
+  // the same on indexing machine and the client.
+  auto Serialized = toProtobuf(Sym, testPath("home/"));
+  auto Deserialized = fromProtobuf(Serialized, &Strings, testPath("home/"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(toYAML(Sym), toYAML(*Deserialized));
+  // Serialized paths are relative and have UNIX slashes.
+  EXPECT_EQ(convert_to_slash(Serialized.definition().file_path(),
+                             llvm::sys::path::Style::posix),
+            Serialized.definition().file_path());
+  EXPECT_TRUE(
+      llvm::sys::path::is_relative(Serialized.definition().file_path()));
+
+  // Invalid URI: Definition will be blank in deserialized object.
+  Location.FileURI = "Not A URI";
+  Sym.Definition = Location;
+  Serialized = toProtobuf(Sym, testPath("home/"));
+  Deserialized = fromProtobuf(Serialized, &Strings, testPath("home/"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(Deserialized->Definition, clangd::SymbolLocation());
+
+  // Schemes other than "file" can not be used.
+  auto UnittestURI = URI::create(testPath("home/SomePath.h"), "unittest");
+  EXPECT_TRUE(bool(UnittestURI));
+  Location.FileURI = Strings.save(UnittestURI->toString()).begin();
+  Sym.Definition = Location;
+  Serialized = toProtobuf(Sym, testPath("home/"));
+  Deserialized = fromProtobuf(Serialized, &Strings, testPath("home/"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(Deserialized->Definition, clangd::SymbolLocation());
+
+  // Passing root that is not prefix of the original file path.
+  Location.FileURI = testPathURI("home/File.h", Strings);
+  Sym.Definition = Location;
+  // Check that the symbol is valid and passing the correct path works.
+  Serialized = toProtobuf(Sym, testPath("home/"));
+  Deserialized = fromProtobuf(Serialized, &Strings, testPath("home/"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(Deserialized->Definition.FileURI,
+            testPathURI("home/File.h", Strings));
+  // Passing wrong root will leave definition blank.
+  Serialized = toProtobuf(Sym, testPath("nothome/"));
+  Deserialized = fromProtobuf(Serialized, &Strings, testPath("home/"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(Deserialized->Definition, clangd::SymbolLocation());
 
-    const auto bar = getT<Foo>();
-  }
-  )";
-  const auto References = TU.headerRefs();
+TEST(RemoteMarshallingTest, RefSerialization) {
+  clangd::Ref Ref;
+  Ref.Kind = clangd::RefKind::Spelled | clangd::RefKind::Declaration;
+
+  llvm::BumpPtrAllocator Arena;
+  llvm::UniqueStringSaver Strings(Arena);
+
+  clangd::SymbolLocation Location;
+  Location.Start.setLine(124);
+  Location.Start.setColumn(21);
+  Location.End.setLine(3213);
+  Location.End.setColumn(541);
+  Location.FileURI = testPathURI(
+      "llvm-project/llvm/clang-tools-extra/clangd/Protocol.h", Strings);
+  Ref.Location = Location;
+
+  auto Serialized = toProtobuf(Ref, testPath("llvm-project/"));
+  auto Deserialized =
+      fromProtobuf(Serialized, &Strings, testPath("llvm-project/"));
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(toYAML(Ref), toYAML(*Deserialized));
+
+TEST(RemoteMarshallingTest, IncludeHeaderURIs) {
   llvm::BumpPtrAllocator Arena;
   llvm::UniqueStringSaver Strings(Arena);
-  // Sanity check: there are more than 5 references available.
-  EXPECT_GE(References.numRefs(), 5UL);
-  for (const auto &SymbolWithRefs : References) {
-    for (const auto &Ref : SymbolWithRefs.second) {
-      const auto RefToProtobufAndBack = fromProtobuf(toProtobuf(Ref), &Strings);
-      EXPECT_TRUE(RefToProtobufAndBack.hasValue());
-      EXPECT_EQ(toYAML(Ref), toYAML(*RefToProtobufAndBack));
-    }
-  }
-} // namespace
+
+  llvm::SmallVector<clangd::Symbol::IncludeHeaderWithReferences, 1>
+      ValidHeaders;
+  clangd::Symbol::IncludeHeaderWithReferences Header;
+  Header.IncludeHeader = Strings.save(
+      URI::createFile("/usr/local/user/home/project/Header.h").toString());
+  Header.References = 21;
+  ValidHeaders.push_back(Header);
+  Header.IncludeHeader = Strings.save("<iostream>");
+  Header.References = 100;
+  ValidHeaders.push_back(Header);
+  Header.IncludeHeader = Strings.save("\"cstdio\"");
+  Header.References = 200;
+  ValidHeaders.push_back(Header);
+
+  llvm::SmallVector<clangd::Symbol::IncludeHeaderWithReferences, 1>
+      InvalidHeaders;
+  // This is an absolute path to a header: can not be transmitted over the wire.
+  Header.IncludeHeader = Strings.save(testPath("project/include/Common.h"));
+  Header.References = 42;
+  InvalidHeaders.push_back(Header);
+  // This is not a valid header: can not be transmitted over the wire;
+  Header.IncludeHeader = Strings.save("NotAHeader");
+  Header.References = 5;
+  InvalidHeaders.push_back(Header);
+
+  clangd::Symbol Sym;
+
+  // Try to serialize all headers but only valid ones will end up in Protobuf
+  // message.
+  auto AllHeaders = ValidHeaders;
+  AllHeaders.insert(AllHeaders.end(), InvalidHeaders.begin(),
+                    InvalidHeaders.end());
+  Sym.IncludeHeaders = AllHeaders;
+
+  auto Serialized = toProtobuf(Sym, convert_to_slash("/"));
+  EXPECT_EQ(static_cast<size_t>(Serialized.headers_size()),
+            ValidHeaders.size());
+  auto Deserialized = fromProtobuf(Serialized, &Strings, "/");
+  EXPECT_TRUE(Deserialized);
+
+  Sym.IncludeHeaders = ValidHeaders;
+  EXPECT_EQ(toYAML(Sym), toYAML(*Deserialized));
+
+TEST(RemoteMarshallingTest, FuzzyFindRequestSerialization) {
+  clangd::FuzzyFindRequest Request;
+  Request.ProximityPaths = {testPath("remote/Header.h"),
+                            testPath("remote/subdir/OtherHeader.h"),
+                            testPath("notremote/File.h"), "Not a Path."};
+  const auto Serialized = toProtobuf(Request, testPath("remote/"));
+  EXPECT_EQ(Serialized.proximity_paths_size(), 2);
+  const auto Deserialized = fromProtobuf(&Serialized, testPath("home/"));
+  EXPECT_THAT(Deserialized.ProximityPaths,
+              testing::ElementsAre(testPath("home/Header.h"),
+                                   testPath("home/subdir/OtherHeader.h")));
 
 } // namespace
 } // namespace remote
 } // namespace clangd
 } // namespace clang