[clangd] Implement path and URI translation for remote index

[kbobyrev]

%s/PrefixPath/ProjectRoot/g

[sammccall]

Generally I think this is missing high-level documentation describing the translation that happens at each level - I had to refer back to the notes from our meeting to understand what the scheme was.
I think Marshalling.h is probably a good place to document how this works. (Not exactly what prefixes get stripped by which function, but rather what the backing index contains, what goes over the wire, and what the clangd on the client side sees).

clang-tools-extra/clangd/index/remote/Client.cpp

89

this should be a std::string - no need to rely on the caller to keep the value alive

clang-tools-extra/clangd/index/remote/Client.h

21

This is a bit low-level - it describes what is being done but not really why. As a caller of this function it doesn't really tell me what I should pass.

Suggest something like "ProjectRoot is an absolute path on the local machine to the source tree described by the remote index. Paths returned by the index will be treated as relative to this directory."

I'd also suggest calling this IndexRoot instead of ProjectRoot, because the latter is a fairly overloaded concept and they need not coincide.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

32

mention somewhere that this is code used on the *client* side to do translation, and that RelativePath came from the server

36

scheme is always "file", no need to make it a parameter?

39

20 is too small. Suggest 256 or so (SmallString is on the stack, so it's cheap)

41

you can use URI::createFile which does less work and never fails

47

this looks like a no-op cast

50

Again this is a bit low level - "translates a URI from the server's backing index to a relative path suitable to send over the wire to the client"?

53

uriToRelativePath for consistency?

59

you're crashing here with an unhandled Expected. Add a test?
(I'd suggest elog or log, too)

62

static_cast is a pretty surprising/uncommon way to write this conversion....
I'd suggest either Result.str.str() or std::string(result)

63

need to handle the case where this returns false. Probably log and return ""?

Again, we should have a test for this.

132–136

this needs to do path translation in the case where the header is a URI

148

and here

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

42

nit: Clangd -> clangd

55–59

what does "serializes clangd" mean?

clang-tools-extra/clangd/index/remote/server/Server.cpp

35

nit: call this variable IndexRoot rather than IndexPrefixPath, for consistency?

68–69

for hygiene, I think we should pass this in as a constructor parameter rather than accessing the flag directly.

You probably also want to add a little validation: this needs to be an absolute path, and you may need to convert slashes to unix-style, too.

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

58

this is testPath("unittest"), isn't it?

I mean, forward vs backslashes are going to differ, but the current expression is going to produce C:\test/unittest which doesn't seem better.

120

test name is backwards - TEST(RemoteMarshallingTest, URITranslation) ?

[sammccall]

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

37

we should also bail out if RelativePath is an absolute path.

63

we really should ensure the result has unix slashes.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

55–59

The Ref marshaller isn't an appropriate place to put the detailed documentation on how path translation works.
It could go in the file documentation, or we could wrap the marshalling methods into a class we could put it there.
(The latter is somewhat appealing because it means you don't need to explicitly pass the root around everywhere, you can do normalization and have a place to store it, etc)

56

Somewhere we need to describe what the format of ProjectRoot is: trailing slash or not, unix slashes or native, etc.
(Even if the current code works in any variation, it's easier to reason about if we have a tightly "normalized" form)

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

56–69

Is passing the empty string here actually valid? I think you probably want to pass testRoot() or some unixified equivalent

120

I think this test is written in a way where it won't work on windows.
I'd suggest making heavier use of testPath() instead.

125

this is not (lexically) a relative path, it's an absolute path.
For the way it's used in this test it doesn't matter that much (just confusing)

130

OK, this assertion does actually show that we're sending absolute paths over the wire and calling them relative.
The confusion here is more serious because it's in the wire format and harder to address later.
There are other technical reasons: true relative paths still leave us the absolute-path namespace to represent paths outside the source tree if we want that for some reason.

[kbobyrev]

Store the progress so that I don't lose it. Still WIP.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

62

Okay, sure! I just thought .str().str() looks really weird, but std::string is probably a good choice.

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

56–69

Why testroot? That's what I'm stripping from URI body, the URI is unittest:///TestTU.h, hence I'm stripping from /TestTU.h, there is no /clangd-test there. Could you elaborate?

58

I don't understand: std::string(testRoot()) + '/' is /clangd-test/ on unix and testPath("unittest") is /clangd-test/unittest. I understand the slashes argument, but could you please elaborate on the former?

[kbobyrev]

Resolve the rest of patch comments.

[kbobyrev]

Remove IndexRoot conversion to UNIX slashes from the Marshalling hot path.

[kbobyrev]

Use std::string() instead of static_cast<std::string>()

[kbobyrev]

Don't convert test paths to system native, UNIX native is good enough.

[kbobyrev]

Convert prefix paths to native slash format for _serialization_ calls
(deserializations should use UNIX slashes).

[kbobyrev]

Add assertions for ensuring paths have UNIX slashes during deserialization.

[sammccall]

This looks better to me but:

• i think we need to consider what happens when paths are not under the expected path, and handle that in some appropriate way
• I find it hard to see what the tests are actually testing - would be good to make them more explicit, using a few minimal helpers, and avoiding fancy abstractions like URI schemes

clang-tools-extra/clangd/index/remote/Index.proto

102–105

storees -> stores

isn't it conceptually simpler to talk about a "relative path" rather than a "part of a URI body"?
(apart from anything else, no need to worry about whether it's escaped or not...)

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

36

mention error return (or return Optional)

52

this should check the scheme too - it must be file
(we can't reasonably treat IndexedProjectRoot as scheme-relative... but not to any *particular* scheme!)

119

nit: indexedprojectroot -> indexroot, unless there's some reason for the asymmetry? it seems like the same concept in the opposite direction

123–124

how do you want to handle the case where the path is in fact not relative stored under the root?

133

Don't try to parse if it's not a URI (starts with < or ").
As written this code will crash in debug mode if it's not a URI, due to unhandled error. Please add a test!

145

you seem to have this check in a few places - I'd suggest it should be in the relativePathToURI function and there should be some plan for handling errors.

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

15

nit: llvm -> llvm-project

42

again, one particular overload is not really the place for this comment, I think. (File comment looks good)

52

this also needs the index root, for the proximity path (I'm not sure why they're not URIs, that's unfortunate...)

clang-tools-extra/clangd/index/remote/server/Server.cpp

50

please pass this in as a constructor param rather than accessing the flag directly

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

25

The File scheme is special. Using a different URI scheme here when we only support file in production is confusing and seems like a last resort to be used when we can't make it work any other way.
(For example we use it in lit tests because we must specify input filenames and the presence/absence of drive letters caused problems)
What's being solved there that the much smaller hammer of testPath() plus a local testPathURI() helper can't solve?

34

At this point I really think we should be writing these symbols out instead of trying to parse them from C++ - it's pretty hard to see exactly what fields are set and what is tested when the test is so highly factored.

If you like this approach think everything new is covered in other tests, I'd just remove the changes to this one.

58

sorry, I guess I meant testRoot("/").

124

auto -> std::string (here and in other places where the type is neither overly unreadable nor spelled in the function name)

124

it's not great practice to be assembling the test data from prefix + relative if the code we're testing is splitting it into prefix + relative - easy for the same bugs to cancel each other out.

Is testPath("remote/dir/MarshallingTests.cpp") directly too long?

128

(I'd suggest a helper testPathURI("remote/dir/MarshallingTests.cpp") for this)

132

hmm, I don't think that's the reason.
Either that's the contract of fromProtobuf, or it's not required and we shouldn't do it.

148

nit: use a file path rather than directory?

[kbobyrev]

Store progress. Still WIP.

[kbobyrev]

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

25

This is dealing with the consequences of test infrastructure being set up in a way that makes all URIs in Symbols and Refs use "unittest" scheme, this helper simply translates those URIs into "file" URIs which are the ones being supported in production. I can not understand why this is not desirable but I'll just clear all URIs and leave those two tests as sanity checks.

132

Ah, llvm::sys::path::append already does that for me. Okay, makes sense to remove this then.

[kbobyrev]

Address the rest of the comments and refactor code.

Patch is ready for the review again.

[kbobyrev]

Fix formatting on one line.

[kbobyrev]

Rebase on top of master

[kbobyrev]

Replace Sym in Client response with a more generic name (it can be a reference).

[sammccall]

Rest is just nits, thanks!

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

36

these functions are nice. It might be clearer to expose them and test them directly, particularly for error cases. Up to you.

37

can we add an assert that the index root is valid? (ends with a platform-appropriate slash)

43

message needs a bit more context. Remote index client got absolute path from server: {0}?

48

nit: lots of const auto throughout - we tend not to use const for local values (as opposed to references or pointers) so this is a bit confusing, particularly when combined with auto. Up to you though.

55

can we add an assert that IndexRoot is valid?

58

Similarly: Remote index got bad URI from client: ...

62

I think it's fine to fold this into the previous check: "bad URI" covers it

66

This isn't a valid IndexRoot, we should assert rather than handle this case I think.

123

shouldn't this return llvm::None, too? a SymbolLocation isn't valid without a path.

140

nit: isLiteralInclude in Headers.h (sorry, had forgotten about this func).
Calling that function probably avoids the need for the comment too.

144

isn't the rest of this uriToRelativePath?

180

{} with no index is a nice idea but I think not actually supported

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

19

nit "is passed passed to the remote server".

20

"It contains absolute path to the project root... relative path has unix slashes"

can we pull this out into a separate paragraph at the end? The rest is describing the flow, and this describes an API detail.

20

nit: "*the* absolute path", "*the* relative path" etc.

[kbobyrev]

Address post-LGTM round of comments.

[kbobyrev]

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

62

But it has different message :( ParsedURI.takeError wouldn't make sense here.

clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp

 
 llvm::cl::opt<std::string> IndexLocation(
     llvm::cl::desc("<path to index file | remote:server.address>"),
     llvm::cl::Positional);
 
 llvm::cl::opt<std::string>
     ExecCommand("c", llvm::cl::desc("Command to execute and then exit"));
 
+llvm::cl::opt<std::string> ProjectRoot("project-root",
+                                       llvm::cl::desc("Path to the project"));
+
 static constexpr char Overview[] = R"(
 This is an **experimental** interactive tool to process user-provided search
 queries over given symbol collection obtained via clangd-indexer. The
 tool can be used to evaluate search quality of existing index implementations
 and manually construct non-trivial test cases.
 
 You can connect to remote index by passing remote:address to dexp. Example:
 
std::function<std::unique_ptr<Command>()> Implementation;
     {"lookup", "Dump symbol details by ID or qualified name",
      std::make_unique<Lookup>},
     {"refs", "Find references by ID or qualified name", std::make_unique<Refs>},
     {"export", "Export index", std::make_unique<Export>},
 };
 
 std::unique_ptr<SymbolIndex> openIndex(llvm::StringRef Index) {
   return Index.startswith("remote:")
-             ? remote::getClient(Index.drop_front(strlen("remote:")))
+             ? remote::getClient(Index.drop_front(strlen("remote:")),
+                                 ProjectRoot)
              : loadIndex(Index, /*UseDex=*/true);
 }
 
 bool runCommand(std::string Request, const SymbolIndex &Index) {
   // Split on spaces and add required null-termination.
   std::replace(Request.begin(), Request.end(), ' ', '\0');
   llvm::SmallVector<llvm::StringRef, 8> Args;
   llvm::StringRef(Request).split(Args, '\0', /*MaxSplit=*/-1,

clang-tools-extra/clangd/index/remote/Client.h

 //===--- Client.h - Connect to a remote index via gRPC -----------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H
 
 #include "index/Index.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
 /// Returns an SymbolIndex client that passes requests to remote index located
 /// at \p Address. The client allows synchronous RPC calls.
+/// \p IndexRoot is an absolute path on the local machine to the source tree
+/// described by the remote index. Paths returned by the index will be treated
+/// as relative to this directory.
 ///
 /// This method attempts to resolve the address and establish the connection.
 ///
 /// \returns nullptr if the address is not resolved during the function call or
 /// if the project was compiled without Remote Index support.
-std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address);
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address,
+                                               llvm::StringRef IndexRoot);
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_INDEX_H

clang-tools-extra/clangd/index/remote/Client.cpp

 //===--- Client.cpp ----------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include <grpc++/grpc++.h>
 
 #include "Client.h"
 #include "Index.grpc.pb.h"
 #include "index/Serialization.h"
 #include "marshalling/Marshalling.h"
 #include "support/Logger.h"
 #include "support/Trace.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 namespace {
 
 class IndexClient : public clangd::SymbolIndex {
   template <typename RequestT, typename ReplyT>
bool streamRPC(ClangdRequestT Request,
     llvm::BumpPtrAllocator Arena;
     llvm::UniqueStringSaver Strings(Arena);
     ReplyT Reply;
     while (Reader->Read(&Reply)) {
       if (!Reply.has_stream_result()) {
         FinalResult = Reply.final_result();
         continue;
       }
-      auto Sym = fromProtobuf(Reply.stream_result(), &Strings);
+      auto Sym = fromProtobuf(Reply.stream_result(), &Strings, ProjectRoot);
       if (!Sym)
         elog("Received invalid {0}", ReplyT::descriptor()->name());
       Callback(*Sym);
     }
     SPAN_ATTACH(Tracer, "status", Reader->Finish().ok());
     return FinalResult;
   }
 
 public:
-  IndexClient(std::shared_ptr<grpc::Channel> Channel)
-      : Stub(remote::SymbolIndex::NewStub(Channel)) {}
+  IndexClient(std::shared_ptr<grpc::Channel> Channel,
+              llvm::StringRef ProjectRoot)
+      : Stub(remote::SymbolIndex::NewStub(Channel)), ProjectRoot(ProjectRoot) {}
 
   void lookup(const clangd::LookupRequest &Request,
               llvm::function_ref<void(const clangd::Symbol &)> Callback) const {
     streamRPC(Request, &remote::SymbolIndex::Stub::Lookup, Callback);
   }
 
   bool
   fuzzyFind(const clangd::FuzzyFindRequest &Request,
relations(const clangd::RelationsRequest &,
             llvm::function_ref<void(const SymbolID &, const clangd::Symbol &)>)
       const {}
 
   // IndexClient does not take any space since the data is stored on the server.
   size_t estimateMemoryUsage() const { return 0; }
 
 private:
   std::unique_ptr<remote::SymbolIndex::Stub> Stub;
+  std::string ProjectRoot;
 };
 
 } // namespace
 
-std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address) {
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address,
+                                               llvm::StringRef ProjectRoot) {
   const auto Channel =
       grpc::CreateChannel(Address.str(), grpc::InsecureChannelCredentials());
   Channel->GetState(true);
-  return std::unique_ptr<clangd::SymbolIndex>(new IndexClient(Channel));
+  return std::unique_ptr<clangd::SymbolIndex>(
+      new IndexClient(Channel, ProjectRoot));
 }
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/remote/Index.proto

message SymbolInfo {
   uint32 subkind = 2;
   uint32 language = 3;
   uint32 properties = 4;
 }
 
 message SymbolLocation {
   Position start = 1;
   Position end = 2;
-  string file_uri = 3;
+  // clangd::SymbolLocation storees FileURI, but the protocol transmits only a
+  // part of its body. Because paths are different on the remote and local
+  // machines they will be translated in the marshalling layer.
+  string file_path = 3;
 }
 
 message Position {
   uint32 line = 1;
   uint32 column = 2;
 }
 
 message HeaderWithReferences {
   string header = 1;
   uint32 references = 2;
 }

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.h

 //===--- Marshalling.h -------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// Transformations between native Clangd types and Protobuf-generated classes.
+// Marshalling provides translation between native clangd types into the
+// Protobuf-generated classes. Most translations are 1-to-1 and wrap variables
+// into appropriate Protobuf types.
+//
+// A notable exception is URI translation. Because paths to files are different
+// on indexing machine and client machine
+// ("/remote/machine/projects/llvm/include/HelloWorld.h" versus
+// "/usr/local/username/llvm/include/HelloWorld.h"), they need to be converted
+// appropriately. Remote machine strips the prefix from an absolute path and
+// passes paths relative to the project root over the wire
+// ("include/HelloWorld.h" in this example). The indexed project root is passed
+// passed to the remote server. It contains absolute path to the project root
+// and includes a trailing slash. Relative path passed over the wire has unix
+// slashes. Client receives this relative path and constructs a URI that points
+// to the relevant file in the filesystem. Relative path is appended to
+// IndexRoot (expected to have UNIX slashes) to construct the full path and
+// build the final URI.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H
 #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H
 
 #include "Index.pb.h"
 #include "index/Index.h"
+#include "llvm/ADT/StringRef.h"
 #include "llvm/Support/StringSaver.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
 clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request);
+/// Deserializes clangd types and translates relative paths into
+/// machine-native URIs.
 llvm::Optional<clangd::Symbol> fromProtobuf(const Symbol &Message,
-                                            llvm::UniqueStringSaver *Strings);
+                                            llvm::UniqueStringSaver *Strings,
+                                            llvm::StringRef IndexRoot);
 llvm::Optional<clangd::Ref> fromProtobuf(const Ref &Message,
-                                         llvm::UniqueStringSaver *Strings);
+                                         llvm::UniqueStringSaver *Strings,
+                                         llvm::StringRef IndexRoot);
 
 LookupRequest toProtobuf(const clangd::LookupRequest &From);
 FuzzyFindRequest toProtobuf(const clangd::FuzzyFindRequest &From);
 RefsRequest toProtobuf(const clangd::RefsRequest &From);
 
-Ref toProtobuf(const clangd::Ref &From);
-Symbol toProtobuf(const clangd::Symbol &From);
+/// Serializes native clangd types and strips \p IndexedProjectRoot from the
+/// file paths because they are specific to the indexing machine.
+Ref toProtobuf(const clangd::Ref &From, llvm::StringRef IndexedProjectRoot);
+Symbol toProtobuf(const clangd::Symbol &From,
+                  llvm::StringRef IndexedProjectRoot);
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang
 
 #endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_INDEX_REMOTE_MARSHALLING_H

clang-tools-extra/clangd/index/remote/marshalling/Marshalling.cpp

 #include "Protocol.h"
 #include "index/Serialization.h"
 #include "index/Symbol.h"
 #include "index/SymbolID.h"
 #include "index/SymbolLocation.h"
 #include "index/SymbolOrigin.h"
 #include "support/Logger.h"
 #include "clang/Index/IndexSymbol.h"
+#include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
 namespace {
 
+/// Translates \p RelativePath into the absolute path and builds URI for the
+/// user machine. This translation happens on the client side with the
+/// \p RelativePath received from remote index server and \p IndexRoot is
+/// provided by the client. \p IndexRoot is expected to have UNIX slashes.
+std::string relativePathToURI(llvm::StringRef RelativePath,
+                              llvm::StringRef IndexRoot) {
+  if (RelativePath.empty())
+    return "";
+  if (llvm::sys::path::is_absolute(RelativePath)) {
+    elog("{0} is not relative path", RelativePath);
+    return "";
+  }
+  llvm::SmallString<256> FullPath = IndexRoot;
+  llvm::sys::path::append(FullPath, RelativePath);
+  const auto Result = URI::createFile(FullPath);
+  return Result.toString();
+}
+
+/// Translates a URI from the server's backing index to a relative path suitable
+/// to send over the wire to the client.
+std::string uriToRelativePath(llvm::StringRef URI,
+                              llvm::StringRef IndexedProjectRoot) {
+  if (URI.empty())
+    return "";
+  auto ParsedURI = URI::parse(URI);
+  if (!ParsedURI) {
+    elog("Can not parse URI {0}: {1}", URI, ParsedURI.takeError());
+    return "";
+  }
+  llvm::SmallString<256> Result = ParsedURI->body();
+  if (IndexedProjectRoot.empty())
+    return std::string(Result);
+  if (!llvm::sys::path::replace_path_prefix(Result, IndexedProjectRoot, "")) {
+    elog("Can not get relative path from the URI {0} given the index root {1}",
+         URI, IndexedProjectRoot);
+    return "";
+  }
+  // Make sure the result has UNIX slashes.
+  return llvm::sys::path::convert_to_slash(Result,
+                                           llvm::sys::path::Style::posix);
+}
+
 clangd::SymbolLocation::Position fromProtobuf(const Position &Message) {
   clangd::SymbolLocation::Position Result;
   Result.setColumn(static_cast<uint32_t>(Message.column()));
   Result.setLine(static_cast<uint32_t>(Message.line()));
   return Result;
 }
 
 Position toProtobuf(const clangd::SymbolLocation::Position &Position) {
SymbolInfo toProtobuf(const clang::index::SymbolInfo &Info) {
   Result.set_kind(static_cast<uint32_t>(Info.Kind));
   Result.set_subkind(static_cast<uint32_t>(Info.SubKind));
   Result.set_language(static_cast<uint32_t>(Info.Lang));
   Result.set_properties(static_cast<uint32_t>(Info.Properties));
   return Result;
 }
 
 clangd::SymbolLocation fromProtobuf(const SymbolLocation &Message,
-                                    llvm::UniqueStringSaver *Strings) {
+                                    llvm::UniqueStringSaver *Strings,
+                                    llvm::StringRef IndexRoot) {
   clangd::SymbolLocation Location;
   Location.Start = fromProtobuf(Message.start());
   Location.End = fromProtobuf(Message.end());
-  Location.FileURI = Strings->save(Message.file_uri()).begin();
+  Location.FileURI =
+      Strings->save(relativePathToURI(Message.file_path(), IndexRoot)).begin();
   return Location;
 }
 
-SymbolLocation toProtobuf(const clangd::SymbolLocation &Location) {
+SymbolLocation toProtobuf(const clangd::SymbolLocation &Location,
+                          llvm::StringRef IndexedProjectRoot) {
   remote::SymbolLocation Result;
   *Result.mutable_start() = toProtobuf(Location.Start);
   *Result.mutable_end() = toProtobuf(Location.End);
-  *Result.mutable_file_uri() = Location.FileURI;
+  *Result.mutable_file_path() =
+      uriToRelativePath(Location.FileURI, IndexedProjectRoot);
   return Result;
 }
 
 HeaderWithReferences
-toProtobuf(const clangd::Symbol::IncludeHeaderWithReferences &IncludeHeader) {
+toProtobuf(const clangd::Symbol::IncludeHeaderWithReferences &IncludeHeader,
+           llvm::StringRef IndexedProjectRoot) {
   HeaderWithReferences Result;
-  Result.set_header(IncludeHeader.IncludeHeader.str());
+  const std::string Header = IncludeHeader.IncludeHeader.str();
+  auto URI = URI::parse(Header);
+  Result.set_header(URI ? uriToRelativePath(IncludeHeader.IncludeHeader.str(),
+                                            IndexedProjectRoot)
+                        : Header);
   Result.set_references(IncludeHeader.References);
   return Result;
 }
 
 clangd::Symbol::IncludeHeaderWithReferences
-fromProtobuf(const HeaderWithReferences &Message) {
-  return clangd::Symbol::IncludeHeaderWithReferences{Message.header(),
+fromProtobuf(const HeaderWithReferences &Message,
+             llvm::UniqueStringSaver *Strings, llvm::StringRef IndexRoot) {
+  std::string Header = Message.header();
+  if (llvm::sys::path::is_relative(Header))
+    Header = relativePathToURI(Header, IndexRoot);
+  return clangd::Symbol::IncludeHeaderWithReferences{Strings->save(Header),
                                                      Message.references()};
 }
 
 } // namespace
 
 clangd::FuzzyFindRequest fromProtobuf(const FuzzyFindRequest *Request) {
   clangd::FuzzyFindRequest Result;
   Result.Query = Request->query();
   for (const auto &Scope : Request->scopes())
     Result.Scopes.push_back(Scope);
   Result.AnyScope = Request->any_scope();
   if (Request->limit())
     Result.Limit = Request->limit();
   Result.RestrictForCodeCompletion = Request->restricted_for_code_completion();
   for (const auto &Path : Request->proximity_paths())
     Result.ProximityPaths.push_back(Path);
   for (const auto &Type : Request->preferred_types())
     Result.ProximityPaths.push_back(Type);
   return Result;
 }
 
 llvm::Optional<clangd::Symbol> fromProtobuf(const Symbol &Message,
-                                            llvm::UniqueStringSaver *Strings) {
+                                            llvm::UniqueStringSaver *Strings,
+                                            llvm::StringRef IndexRoot) {
   if (!Message.has_info() || !Message.has_definition() ||
       !Message.has_canonical_declarattion()) {
     elog("Cannot convert Symbol from Protobuf: {}", Message.ShortDebugString());
     return llvm::None;
   }
   clangd::Symbol Result;
   auto ID = SymbolID::fromStr(Message.id());
   if (!ID) {
-    elog("Cannot convert parse SymbolID {} from Protobuf: {}", ID.takeError(),
+    elog("Cannot parse SymbolID {} given Protobuf: {}", ID.takeError(),
          Message.ShortDebugString());
     return llvm::None;
   }
   Result.ID = *ID;
   Result.SymInfo = fromProtobuf(Message.info());
   Result.Name = Message.name();
   Result.Scope = Message.scope();
-  Result.Definition = fromProtobuf(Message.definition(), Strings);
+  Result.Definition = fromProtobuf(Message.definition(), Strings, IndexRoot);
   Result.CanonicalDeclaration =
-      fromProtobuf(Message.canonical_declarattion(), Strings);
+      fromProtobuf(Message.canonical_declarattion(), Strings, IndexRoot);
   Result.References = Message.references();
   Result.Origin = static_cast<clangd::SymbolOrigin>(Message.origin());
   Result.Signature = Message.signature();
   Result.TemplateSpecializationArgs = Message.template_specialization_args();
   Result.CompletionSnippetSuffix = Message.completion_snippet_suffix();
   Result.Documentation = Message.documentation();
   Result.ReturnType = Message.return_type();
   Result.Type = Message.type();
   for (const auto &Header : Message.headers()) {
-    Result.IncludeHeaders.push_back(fromProtobuf(Header));
+    Result.IncludeHeaders.push_back(fromProtobuf(Header, Strings, IndexRoot));
   }
   Result.Flags = static_cast<clangd::Symbol::SymbolFlag>(Message.flags());
   return Result;
 }
 
 llvm::Optional<clangd::Ref> fromProtobuf(const Ref &Message,
-                                         llvm::UniqueStringSaver *Strings) {
+                                         llvm::UniqueStringSaver *Strings,
+                                         llvm::StringRef IndexRoot) {
   if (!Message.has_location()) {
     elog("Cannot convert Ref from Protobuf: {}", Message.ShortDebugString());
     return llvm::None;
   }
   clangd::Ref Result;
-  Result.Location = fromProtobuf(Message.location(), Strings);
+  Result.Location = fromProtobuf(Message.location(), Strings, IndexRoot);
   Result.Kind = static_cast<clangd::RefKind>(Message.kind());
   return Result;
 }
 
 LookupRequest toProtobuf(const clangd::LookupRequest &From) {
   LookupRequest RPCRequest;
   for (const auto &SymbolID : From.IDs)
     RPCRequest.add_ids(SymbolID.str());
RefsRequest toProtobuf(const clangd::RefsRequest &From) {
   for (const auto &ID : From.IDs)
     RPCRequest.add_ids(ID.str());
   RPCRequest.set_filter(static_cast<uint32_t>(From.Filter));
   if (From.Limit)
     RPCRequest.set_limit(*From.Limit);
   return RPCRequest;
 }
 
-Symbol toProtobuf(const clangd::Symbol &From) {
+Symbol toProtobuf(const clangd::Symbol &From,
+                  llvm::StringRef IndexedProjectRoot) {
   Symbol Result;
   Result.set_id(From.ID.str());
   *Result.mutable_info() = toProtobuf(From.SymInfo);
   Result.set_name(From.Name.str());
-  *Result.mutable_definition() = toProtobuf(From.Definition);
+  *Result.mutable_definition() =
+      toProtobuf(From.Definition, IndexedProjectRoot);
   Result.set_scope(From.Scope.str());
   *Result.mutable_canonical_declarattion() =
-      toProtobuf(From.CanonicalDeclaration);
+      toProtobuf(From.CanonicalDeclaration, IndexedProjectRoot);
   Result.set_references(From.References);
   Result.set_origin(static_cast<uint32_t>(From.Origin));
   Result.set_signature(From.Signature.str());
   Result.set_template_specialization_args(
       From.TemplateSpecializationArgs.str());
   Result.set_completion_snippet_suffix(From.CompletionSnippetSuffix.str());
   Result.set_documentation(From.Documentation.str());
   Result.set_return_type(From.ReturnType.str());
   Result.set_type(From.Type.str());
   for (const auto &Header : From.IncludeHeaders) {
     auto *NextHeader = Result.add_headers();
-    *NextHeader = toProtobuf(Header);
+    *NextHeader = toProtobuf(Header, IndexedProjectRoot);
   }
   Result.set_flags(static_cast<uint32_t>(From.Flags));
   return Result;
 }
 
-Ref toProtobuf(const clangd::Ref &From) {
+Ref toProtobuf(const clangd::Ref &From, llvm::StringRef IndexedProjectRoot) {
   Ref Result;
   Result.set_kind(static_cast<uint32_t>(From.Kind));
-  *Result.mutable_location() = toProtobuf(From.Location);
+  *Result.mutable_location() = toProtobuf(From.Location, IndexedProjectRoot);
   return Result;
 }
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/index/remote/server/Server.cpp

 //===--- Server.cpp - gRPC-based Remote Index Server  ---------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "index/Index.h"
 #include "index/Serialization.h"
 #include "index/remote/marshalling/Marshalling.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/Signals.h"
 
 #include <grpc++/grpc++.h>
 #include <grpc++/health_check_service_interface.h>
 
 #include "Index.grpc.pb.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 namespace {
 
 static constexpr char Overview[] = R"(
 This is an experimental remote index implementation. The server opens Dex and
 awaits gRPC lookup requests from the client.
 )";
 
 llvm::cl::opt<std::string> IndexPath(llvm::cl::desc("<INDEX FILE>"),
                                      llvm::cl::Positional, llvm::cl::Required);
 
+llvm::cl::opt<std::string> IndexRoot(llvm::cl::desc("<PROJECT ROOT>"),
+                                     llvm::cl::Positional, llvm::cl::Required);
+
 llvm::cl::opt<std::string> ServerAddress(
     "server-address", llvm::cl::init("0.0.0.0:50051"),
     llvm::cl::desc("Address of the invoked server. Defaults to 0.0.0.0:50051"));
 
 std::unique_ptr<clangd::SymbolIndex> openIndex(llvm::StringRef Index) {
   return loadIndex(Index, /*UseIndex=*/true);
 }
 
 class RemoteIndexServer final : public SymbolIndex::Service {
 public:
   RemoteIndexServer(std::unique_ptr<clangd::SymbolIndex> Index)
-      : Index(std::move(Index)) {}
+      : Index(std::move(Index)) {
+    llvm::SmallString<256> NativePath = llvm::StringRef(IndexRoot);
+    llvm::sys::path::native(NativePath);
+    IndexedProjectRoot = std::string(NativePath);
+  }
 
 private:
   grpc::Status Lookup(grpc::ServerContext *Context,
                       const LookupRequest *Request,
                       grpc::ServerWriter<LookupReply> *Reply) override {
     clangd::LookupRequest Req;
     for (const auto &ID : Request->ids()) {
       auto SID = SymbolID::fromStr(StringRef(ID));
       if (!SID)
         return grpc::Status::CANCELLED;
       Req.IDs.insert(*SID);
     }
     Index->lookup(Req, [&](const clangd::Symbol &Sym) {
       LookupReply NextMessage;
-      *NextMessage.mutable_stream_result() = toProtobuf(Sym);
+      *NextMessage.mutable_stream_result() =
+          toProtobuf(Sym, IndexedProjectRoot);
       Reply->Write(NextMessage);
     });
     LookupReply LastMessage;
     LastMessage.set_final_result(true);
     Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
   grpc::Status FuzzyFind(grpc::ServerContext *Context,
                          const FuzzyFindRequest *Request,
                          grpc::ServerWriter<FuzzyFindReply> *Reply) override {
     const auto Req = fromProtobuf(Request);
     bool HasMore = Index->fuzzyFind(Req, [&](const clangd::Symbol &Sym) {
       FuzzyFindReply NextMessage;
-      *NextMessage.mutable_stream_result() = toProtobuf(Sym);
+      *NextMessage.mutable_stream_result() =
+          toProtobuf(Sym, IndexedProjectRoot);
       Reply->Write(NextMessage);
     });
     FuzzyFindReply LastMessage;
     LastMessage.set_final_result(HasMore);
     Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
   grpc::Status Refs(grpc::ServerContext *Context, const RefsRequest *Request,
                     grpc::ServerWriter<RefsReply> *Reply) override {
     clangd::RefsRequest Req;
     for (const auto &ID : Request->ids()) {
       auto SID = SymbolID::fromStr(StringRef(ID));
       if (!SID)
         return grpc::Status::CANCELLED;
       Req.IDs.insert(*SID);
     }
     bool HasMore = Index->refs(Req, [&](const clangd::Ref &Reference) {
       RefsReply NextMessage;
-      *NextMessage.mutable_stream_result() = toProtobuf(Reference);
+      *NextMessage.mutable_stream_result() = toProtobuf(Reference, IndexRoot);
       Reply->Write(NextMessage);
     });
     RefsReply LastMessage;
     LastMessage.set_final_result(HasMore);
     Reply->Write(LastMessage);
     return grpc::Status::OK;
   }
 
   std::unique_ptr<clangd::SymbolIndex> Index;
+  std::string IndexedProjectRoot;
 };
 
 void runServer(std::unique_ptr<clangd::SymbolIndex> Index,
                const std::string &ServerAddress) {
   RemoteIndexServer Service(std::move(Index));
 
   grpc::EnableDefaultHealthCheckService(true);
   grpc::ServerBuilder Builder;

 } // namespace clangd
 } // namespace clang
 
 int main(int argc, char *argv[]) {
   using namespace clang::clangd::remote;
   llvm::cl::ParseCommandLineOptions(argc, argv, Overview);
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
 
+  if (!llvm::sys::path::is_absolute(IndexRoot)) {
+    llvm::outs() << "Index root should be an absolute path.\n";
+    return -1;
+  }
+
   std::unique_ptr<clang::clangd::SymbolIndex> Index = openIndex(IndexPath);
 
   if (!Index) {
     llvm::outs() << "Failed to open the index.\n";
     return -1;
   }
 
   runServer(std::move(Index), ServerAddress);
 }

clang-tools-extra/clangd/index/remote/unimplemented/UnimplementedClient.cpp

 //===--- UnimplementedClient.cpp ---------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "index/remote/Client.h"
 #include "support/Logger.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace clang {
 namespace clangd {
 namespace remote {
 
-std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address) {
+std::unique_ptr<clangd::SymbolIndex> getClient(llvm::StringRef Address,
+                                               llvm::StringRef IndexRoot) {
   elog("Can't create SymbolIndex client without Remote Index support.");
   return nullptr;
 }
 
 } // namespace remote
 } // namespace clangd
 } // namespace clang

clang-tools-extra/clangd/unittests/remote/MarshallingTests.cpp

 //===--- MarshallingTests.cpp ------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "../TestTU.h"
+#include "TestFS.h"
 #include "index/Serialization.h"
 #include "index/remote/marshalling/Marshalling.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Path.h"
 #include "llvm/Support/StringSaver.h"
 #include "gtest/gtest.h"
+#include <cstring>
 
 namespace clang {
 namespace clangd {
 namespace remote {
 namespace {
 
+const char *unittestURIToFilesystem(const char *UnittestURI,
+                                    llvm::UniqueStringSaver &Strings) {
+  auto URI = URI::parse(UnittestURI);
+  EXPECT_TRUE(bool(URI));
+  const auto FileSystemURI =
+      URI::createFile(testPath(llvm::sys::path::relative_path(URI->body())));
+  return Strings.save(FileSystemURI.toString()).begin();
+}
+
 TEST(RemoteMarshallingTest, SymbolSerialization) {
   const auto *Header = R"(
   // This is a class.
   class Foo {
   public:
     Foo();
 
     int Bar;
   private:
     double Number;
   };
   /// This is a function.
   char baz();
   template <typename T>
   T getT ();
   )";
   const auto TU = TestTU::withHeaderCode(Header);
   const auto Symbols = TU.headerSymbols();
   // Sanity check: there are more than 5 symbols available.
   EXPECT_GE(Symbols.size(), 5UL);
   llvm::BumpPtrAllocator Arena;
   llvm::UniqueStringSaver Strings(Arena);
-  for (auto &Sym : Symbols) {
-    const auto ProtobufMeessage = toProtobuf(Sym);
-    const auto SymToProtobufAndBack = fromProtobuf(ProtobufMeessage, &Strings);
+  llvm::SmallString<256> RootPath = llvm::StringRef("/");
+  llvm::sys::path::native(RootPath);
+  for (auto Sym : Symbols) {
+    Sym.CanonicalDeclaration.FileURI =
+        unittestURIToFilesystem(Sym.CanonicalDeclaration.FileURI, Strings);
+    if (std::strlen(Sym.Definition.FileURI))
+      Sym.Definition.FileURI =
+          unittestURIToFilesystem(Sym.Definition.FileURI, Strings);
+    for (auto &Header : Sym.IncludeHeaders)
+      Header.IncludeHeader =
+          unittestURIToFilesystem(Header.IncludeHeader.str().c_str(), Strings);
+    auto ProtobufMessage = toProtobuf(Sym, RootPath);
+    const auto SymToProtobufAndBack =
+        fromProtobuf(ProtobufMessage, &Strings, RootPath);
     EXPECT_TRUE(SymToProtobufAndBack.hasValue());
     EXPECT_EQ(toYAML(Sym), toYAML(*SymToProtobufAndBack));
   }
 }
 
 TEST(RemoteMarshallingTest, ReferenceSerialization) {
   TestTU TU;
   TU.HeaderCode = R"(
int foo() {
     }
 
     const auto bar = getT<Foo>();
   }
   )";
   const auto References = TU.headerRefs();
   llvm::BumpPtrAllocator Arena;
   llvm::UniqueStringSaver Strings(Arena);
+  llvm::SmallString<256> RootPath = llvm::StringRef("/");
+  llvm::sys::path::native(RootPath);
   // Sanity check: there are more than 5 references available.
   EXPECT_GE(References.numRefs(), 5UL);
   for (const auto &SymbolWithRefs : References) {
-    for (const auto &Ref : SymbolWithRefs.second) {
-      const auto RefToProtobufAndBack = fromProtobuf(toProtobuf(Ref), &Strings);
+    for (auto Ref : SymbolWithRefs.second) {
+      Ref.Location.FileURI =
+          unittestURIToFilesystem(Ref.Location.FileURI, Strings);
+      const auto RefToProtobufAndBack =
+          fromProtobuf(toProtobuf(Ref, RootPath), &Strings, RootPath);
       EXPECT_TRUE(RefToProtobufAndBack.hasValue());
       EXPECT_EQ(toYAML(Ref), toYAML(*RefToProtobufAndBack));
     }
   }
-} // namespace
+}
+
+TEST(RemoteMarshallingTest, URITranslation) {
+  llvm::BumpPtrAllocator Arena;
+  llvm::UniqueStringSaver Strings(Arena);
+  clangd::Ref Original;
+  const std::string RemoteIndexPrefix = testPath("remote/machine/project/");
+  std::string RelativePath = "llvm-project/clang-tools-extra/clangd/"
+                             "unittests/remote/MarshallingTests.cpp";
+  llvm::SmallString<256> NativePath = StringRef(RelativePath);
+  llvm::sys::path::native(NativePath);
+  RelativePath = std::string(NativePath);
+  const auto URI = URI::createFile(RemoteIndexPrefix + RelativePath);
+  Original.Location.FileURI = Strings.save(URI.toString()).begin();
+  Ref Serialized = toProtobuf(Original, RemoteIndexPrefix);
+  EXPECT_EQ(Serialized.location().file_path(), RelativePath);
+  const std::string LocalIndexPrefix = testPath("local/machine/project/");
+  auto Deserialized = fromProtobuf(Serialized, &Strings, LocalIndexPrefix);
+  EXPECT_TRUE(Deserialized);
+  EXPECT_EQ(Deserialized->Location.FileURI,
+            URI::createFile(LocalIndexPrefix + RelativePath).toString());
+
+  clangd::Ref WithInvalidURI;
+  WithInvalidURI.Location.FileURI = "This is not a URI";
+  Serialized = toProtobuf(WithInvalidURI, RemoteIndexPrefix);
+  // Invalid URI results in empty path.
+  EXPECT_EQ(Serialized.location().file_path(), "");
+
+  Ref WithAbsolutePath;
+  *WithAbsolutePath.mutable_location()->mutable_file_path() = "/usr/local/bin/";
+  Deserialized = fromProtobuf(WithAbsolutePath, &Strings, LocalIndexPrefix);
+  // Can not build URI given absolute path.
+  EXPECT_EQ(std::string(Deserialized->Location.FileURI), "");
+}
 
 } // namespace
 } // namespace remote
 } // namespace clangd
 } // namespace clang