[clangd] XPC transport layer, framework, test-client

[jkorous]

Summary

Thanks for looking!

[sammccall]

A question about the high-level build target setup (I don't know much about XPC services/frameworks, bear with me...):

This is set up so that the clangd binary (ClangdMain) can run unmodified as an XPC service, all flags and options are still respected etc.
At the same time, the framework plist seems like it is designed to invoke the clangd binary in a very specific configuration (no flags will be plumbed through).

Is it actually important that the clangd binary itself can be used with XPC, rather than defining another binary that spawns a ClangdServer+XPCTransport with the right config? If you strip out all of ClangdMain that's not related to flag parsing and options, there's almost nothing left...

tool/ClangdMain.cpp

30 ↗

(On Diff #173674)

WDYT about putting newXPCTransport() in Transport.h behind an ifdef?

That will be consistent with JSON, allow the XPCTransport.h to be removed entirely, and remove one #ifdef here.

I find the #ifdefs easier to understand if they surround functional code, rather than #includes - might be just me.

328 ↗

(On Diff #173674)

I'd consider putting this outside the #ifdef, so we can print an error and exit if it's requested but not built.

In fact, can this just be a regular flag instead? -transport={xpc|json}

329 ↗

(On Diff #173674)

no support for -input-mirror or pretty-printing in the logs - deliberate?

xpc/CMakeLists.txt

21 ↗

(On Diff #173674)

why is conversions a separate library from transport?

No problem with fine-grained targets per se, but it's inconsistent with much of the rest of clang-tools-extra.

xpc/Conversion.cpp

16 ↗

(On Diff #173674)

nit:

using namespace llvm;
namespace clang {
namespace clangd {

(we're finally consistent about this)

22 ↗

(On Diff #173674)

Key, PayloadString, etc

23 ↗

(On Diff #173674)

nit: you can #include ScopedPrinter, and then this is just llvm::to_string(json)

28 ↗

(On Diff #173674)

ah, this encoding is a little sad vs the "native" encoding of object -> xpc_dictionary, array -> xpc_array etc which looked promising...

Totally your call, but out of curiosity - why the change?

xpc/Conversion.h

19 ↗

(On Diff #173674)

param name json -> JSON

xpc/XPCTransport.cpp

142 ↗

(On Diff #173674)

lowercase namespace?

177 ↗

(On Diff #173674)

I'm not clear on the relationship between XPC services and connections - if there are multiple connections, what happens? Do you want to check and close any subsequent ones? Can connections and/or messages arrive on multiple threads?

192 ↗

(On Diff #173674)

you could consider asserting that they were previously null

xpc/XPCTransport.h

19 ↗

(On Diff #173674)

as mentioned above, I think this may be more discoverable in Transport.h, with ifdef.
But if the layering/separate libraries are important, here seems OK too.

[jkorous]

In D54428#1297147, @sammccall wrote:

A question about the high-level build target setup (I don't know much about XPC services/frameworks, bear with me...):

This is set up so that the clangd binary (ClangdMain) can run unmodified as an XPC service, all flags and options are still respected etc.
At the same time, the framework plist seems like it is designed to invoke the clangd binary in a very specific configuration (no flags will be plumbed through).

Is it actually important that the clangd binary itself can be used with XPC, rather than defining another binary that spawns a ClangdServer+XPCTransport with the right config? If you strip out all of ClangdMain that's not related to flag parsing and options, there's almost nothing left...

I don't really expect you to be familiar with XPC - feel free to ask about anything and I'll do my best explaining.

It's more like that configuration isn't implemented in this initial patch but as we can't use command line options (launchd doesn't pass these to XPC service) we'll have to duplicate these as environment variables. We discusse internally and we prefer to have just single binary as that would make integration and testing easier for us.

tool/ClangdMain.cpp

30 ↗

(On Diff #173674)

Ok, sounds reasonable.

328 ↗

(On Diff #173674)

Unfortunately it's not possible to have launchd pass flags when spawning XPC services (otherwise it would be my first choice too).

329 ↗

(On Diff #173674)

I was thinking this could be added later.

xpc/CMakeLists.txt

21 ↗

(On Diff #173674)

Guilty of YAGNI violation here - was thinking about eventual use by XPC clients. I'll stick to consistency for now and would separate it if needed.

xpc/Conversion.cpp

16 ↗

(On Diff #173674)

Ok. Thanks.

22 ↗

(On Diff #173674)

Thanks. Somehow I was always working on projects with different naming conventions and still struggle with this.

23 ↗

(On Diff #173674)

I'll take a look at ScopedPrinter. Thanks.

28 ↗

(On Diff #173674)

I know. At first we didn't even think of anything else than 1:1 mapping between JSON and XPC dictionary as it's just natural. But later we learned about performance issues with construction of XPC dictionaries with lots of elements which would be a problem for code completion.

xpc/Conversion.h

19 ↗

(On Diff #173674)

Good catch. Thanks.

xpc/XPCTransport.cpp

142 ↗

(On Diff #173674)

Right. I should probably start looking for some tool checking this...

177 ↗

(On Diff #173674)

Our use-case is an XPC service which is bundled to an application. It could be described as a "private daemon" - the service is launched on-demand for the application. There should be either zero or one connection at a time.

As for messages - we didn't plan for concurrent use. I'll add explicit synchronization.

192 ↗

(On Diff #173674)

Good idea.

xpc/XPCTransport.h

19 ↗

(On Diff #173674)

I agree, will move it.

[jkorous]

Addressed most of the comments.

[jkorous]

Since AFAIK Sam is off until the end of the year I am adding more reviewers.

tool/ClangdMain.cpp

329 ↗

(On Diff #173674)

Just to clarify - since we'd like to have just a single binary for XPC and stdio transport and we sadly can't use command line options we'd have to duplicate probably all of them as env variables.
Would that be fine with you?

xpc/CMakeLists.txt

21 ↗

(On Diff #173674)

Now that I actually took a better look we already have one client - our xpc-test-client. I assume that means - it's fine as it is, right?

xpc/XPCTransport.cpp

177 ↗

(On Diff #173674)

I want to make the synchronous operation explicit here. Still need to work on this.

[jkorous]

added one more assert

[jkorous]

Fixed the synchronous handling of events.

[jkorous]

@arphaman or @sammccall, could you please take the final look?

[arphaman]

LGTM

[tamur]

This patch seems to have broken the compilation. I get the following error on a linux platform:
[12/14] Linking CXX executable bin/clangd
FAILED: bin/clangd
: && /usr/local/google/home/tamur/src/llvm/2018_nov_12/llvm/Stable/bin/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -fuse-ld=lld -Wl,--color-diagnostics -Wl,-allow-shlib-undefined -Wl,-O3 -Wl,--gc-sections tools/clang/tools/extra/clangd/tool/CMakeFiles/clangd.dir/ClangdMain.cpp.o -o bin/clangd -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.so.8svn -lpthread lib/libclangBasic.so.8svn lib/libclangDaemon.so.8svn lib/libclangFormat.so.8svn lib/libclangFrontend.so.8svn lib/libclangSema.so.8svn lib/libclangTooling.so.8svn lib/libclangToolingCore.so.8svn && :
ld.lld: error: undefined symbol: clang::clangd::newXPCTransport()

[MaskRay]

In D54428#1359333, @tamur wrote:

This patch seems to have broken the compilation. I get the following error on a linux platform:
[12/14] Linking CXX executable bin/clangd
FAILED: bin/clangd
: && /usr/local/google/home/tamur/src/llvm/2018_nov_12/llvm/Stable/bin/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -fuse-ld=lld -Wl,--color-diagnostics -Wl,-allow-shlib-undefined -Wl,-O3 -Wl,--gc-sections tools/clang/tools/extra/clangd/tool/CMakeFiles/clangd.dir/ClangdMain.cpp.o -o bin/clangd -Wl,-rpath,"\$ORIGIN/../lib" lib/libLLVMSupport.so.8svn -lpthread lib/libclangBasic.so.8svn lib/libclangDaemon.so.8svn lib/libclangFormat.so.8svn lib/libclangFrontend.so.8svn lib/libclangSema.so.8svn lib/libclangTooling.so.8svn lib/libclangToolingCore.so.8svn && :
ld.lld: error: undefined symbol: clang::clangd::newXPCTransport()

Fixed by rCTE351306

[sammccall]

Sorry for not getting back to this, but it looks good.

I'm really glad about how minimal the changes are outside xpc/, because it makes it less likely we'll accidentally break something that can only be tested on mac.

CMakeLists.txt

+option(CLANGD_BUILD_XPC "Build XPC Support For Clangd." OFF)
+if (${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
+  set(CLANGD_BUILD_XPC ON CACHE BOOL "" FORCE)
+endif ()
+
 add_subdirectory(clang-apply-replacements)
 add_subdirectory(clang-reorder-fields)
 add_subdirectory(modularize)
 add_subdirectory(clang-tidy)
 add_subdirectory(clang-tidy-vs)
 
 add_subdirectory(change-namespace)
 add_subdirectory(clang-doc)

clangd/CMakeLists.txt

+# Configure the Features.inc file.
+llvm_canonicalize_cmake_booleans(
+  CLANGD_BUILD_XPC)
+configure_file(
+  ${CMAKE_CURRENT_SOURCE_DIR}/Features.inc.in
+  ${CMAKE_CURRENT_BINARY_DIR}/Features.inc
+)
+
 set(LLVM_LINK_COMPONENTS
   Support
   )
 
 if(CLANG_BUILT_STANDALONE)
   # needed to get HAVE_CXX_ATOMICS64_WITHOUT_LIB defined
   include(CheckAtomic)
 endif()

 endif()
 add_subdirectory(tool)
 add_subdirectory(indexer)
 add_subdirectory(index/dex/dexp)
 
 if (LLVM_INCLUDE_BENCHMARKS)
   add_subdirectory(benchmarks)
 endif()
+if ( CLANGD_BUILD_XPC )
+  add_subdirectory(xpc)
+endif ()

clangd/Features.inc.in

+#define CLANGD_BUILD_XPC @CLANGD_BUILD_XPC@

clangd/Transport.h

 // sometimes hang rather than exit on other OSes. The interaction between
 // istreams and signals isn't well-specified, so it's hard to get this right.
 // The C APIs seem to be clearer in this respect.
 std::unique_ptr<Transport>
 newJSONTransport(std::FILE *In, llvm::raw_ostream &Out,
                  llvm::raw_ostream *InMirror, bool Pretty,
                  JSONStreamStyle = JSONStreamStyle::Standard);
 
+#ifdef CLANGD_BUILD_XPC
+// Returns a Transport for macOS based on XPC.
+// Clangd with this transport is meant to be run as bundled XPC service.
+std::unique_ptr<Transport> newXPCTransport();
+#endif
+
 } // namespace clangd
 } // namespace clang
 
 #endif

clangd/tool/CMakeLists.txt

 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/..)
+include_directories(${CMAKE_CURRENT_BINARY_DIR}/..)
 
 add_clang_tool(clangd
   ClangdMain.cpp
   )
 
 set(LLVM_LINK_COMPONENTS
   support
   )
 
+set(CLANGD_XPC_LIBS "")
+if(CLANGD_BUILD_XPC)
+  list(APPEND CLANGD_XPC_LIBS "clangdXpcJsonConversions" "clangdXpcTransport")
+endif()
+
 target_link_libraries(clangd
   PRIVATE
   clangBasic
   clangDaemon
   clangFormat
   clangFrontend
   clangSema
   clangTooling
   clangToolingCore
+  ${CLANGD_XPC_LIBS}
   )

clangd/tool/ClangdMain.cpp

 //===--- ClangdMain.cpp - clangd server loop ------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
+#include "Features.inc"
 #include "ClangdLSPServer.h"
 #include "Path.h"
 #include "Trace.h"
 #include "Transport.h"
 #include "index/Serialization.h"
 #include "clang/Basic/Version.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"

 #else
 const char TestScheme::TestDir[] = "/clangd-test";
 #endif
 
 } // namespace
 } // namespace clangd
 } // namespace clang
 
+enum class ErrorResultCode : int {
+  NoShutdownRequest = 1,
+  CantRunAsXPCService = 2
+};
+
 int main(int argc, char *argv[]) {
   using namespace clang;
   using namespace clang::clangd;
 
   llvm::sys::PrintStackTraceOnErrorSignal(argv[0]);
   llvm::cl::SetVersionPrinter([](llvm::raw_ostream &OS) {
     OS << clang::getClangToolFullVersion("clangd") << "\n";
   });
int main(int argc, char *argv[]) {
   }
   CCOpts.SpeculativeIndexRequest = Opts.StaticIndex;
   CCOpts.EnableFunctionArgSnippets = EnableFunctionArgSnippets;
   CCOpts.AllScopes = AllScopesCompletion;
 
   // Initialize and run ClangdLSPServer.
   // Change stdin to binary to not lose \r\n on windows.
   llvm::sys::ChangeStdinToBinary();
-  auto Transport = newJSONTransport(
+
+  std::unique_ptr<Transport> TransportLayer;
+  if (getenv("CLANGD_AS_XPC_SERVICE")) {
+#ifdef CLANGD_BUILD_XPC
+    TransportLayer = newXPCTransport();
+#else
+    errs() << "This clangd binary wasn't built with XPC support.\n";
+    return ErrorResultCode::CantRunAsXPCService;
+#endif
+  } else {
+    TransportLayer = newJSONTransport(
         stdin, llvm::outs(),
-      InputMirrorStream ? InputMirrorStream.getPointer() : nullptr, PrettyPrint,
-      InputStyle);
+        InputMirrorStream ? InputMirrorStream.getPointer() : nullptr,
+        PrettyPrint, InputStyle);
+  }
+
   ClangdLSPServer LSPServer(
-      *Transport, CCOpts, CompileCommandsDirPath,
+      *TransportLayer, CCOpts, CompileCommandsDirPath,
       /*UseDirBasedCDB=*/CompileArgsFrom == FilesystemCompileArgs, Opts);
-  constexpr int NoShutdownRequestErrorCode = 1;
   llvm::set_thread_name("clangd.main");
-  return LSPServer.run() ? 0 : NoShutdownRequestErrorCode;
+  return LSPServer.run() ? 0
+                         : static_cast<int>(ErrorResultCode::NoShutdownRequest);
 }

clangd/xpc/CMakeLists.txt

+set(CLANGD_XPC_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}")
+set(CLANGD_XPC_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}")
+
+list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")
+include(CreateClangdXPCFramework)
+
+add_subdirectory(framework)
+add_subdirectory(test-client)
+
+include_directories(
+  ${CMAKE_CURRENT_SOURCE_DIR}/../
+)
+
+set(LLVM_LINK_COMPONENTS
+  Support
+  )
+
+# Needed by LLVM's CMake checks because this file defines multiple targets.
+set(LLVM_OPTIONAL_SOURCES Conversion.cpp XPCTransport.cpp)
+
+add_clang_library(clangdXpcJsonConversions
+  Conversion.cpp
+  )
+
+add_clang_library(clangdXpcTransport
+  XPCTransport.cpp
+  DEPENDS clangdXpcJsonConversions
+  LINK_LIBS clangdXpcJsonConversions
+  )

clangd/xpc/Conversion.h

+//===--- Conversion.h - LSP data (de-)serialization through XPC -*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_XPC_XPCJSONCONVERSIONS_H
+#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_XPC_XPCJSONCONVERSIONS_H
+
+#include "llvm/Support/JSON.h"
+#include <xpc/xpc.h>
+
+namespace clang {
+namespace clangd {
+
+xpc_object_t jsonToXpc(const llvm::json::Value &JSON);
+llvm::json::Value xpcToJson(const xpc_object_t &XPCObject);
+
+} // namespace clangd
+} // namespace clang
+
+#endif

clangd/xpc/Conversion.cpp

+//===--- Conversion.cpp - LSP data (de-)serialization through XPC - C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "xpc/Conversion.h"
+#include "Logger.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/ScopedPrinter.h"
+#include <string>
+#include <vector>
+
+using namespace llvm;
+namespace clang {
+namespace clangd {
+
+xpc_object_t jsonToXpc(const json::Value &JSON) {
+  const char *const Key = "LSP";
+  xpc_object_t PayloadObj = xpc_string_create(llvm::to_string(JSON).c_str());
+  return xpc_dictionary_create(&Key, &PayloadObj, 1);
+}
+
+json::Value xpcToJson(const xpc_object_t &XPCObject) {
+  if (xpc_get_type(XPCObject) == XPC_TYPE_DICTIONARY) {
+    const char *const LSP = xpc_dictionary_get_string(XPCObject, "LSP");
+    auto Json = json::parse(llvm::StringRef(LSP));
+    if (Json)
+      return *Json;
+    else
+      elog("JSON parse error: {0}", toString(Json.takeError()));
+  }
+  return json::Value(nullptr);
+}
+
+} // namespace clangd
+} // namespace clang

clangd/xpc/README.txt

+This directory contains:
+- the XPC transport layer (alternative transport layer to JSON-RPC)
+- XPC framework wrapper that wraps around Clangd to make it a valid XPC service
+- XPC test-client
+
+MacOS only. Feature is guarded by CLANGD_BUILD_XPC, including whole xpc/ dir.
 No newline at end of file

clangd/xpc/XPCTransport.cpp

+//===--- XPCTransport.cpp - sending and receiving LSP messages over XPC ---===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#include "Conversion.h"
+#include "Logger.h"
+#include "Protocol.h" // For LSPError
+#include "Transport.h"
+#include "llvm/Support/Errno.h"
+
+#include <xpc/xpc.h>
+
+using namespace llvm;
+using namespace clang;
+using namespace clangd;
+
+namespace {
+
+json::Object encodeError(Error E) {
+  std::string Message;
+  ErrorCode Code = ErrorCode::UnknownErrorCode;
+  if (Error Unhandled =
+          handleErrors(std::move(E), [&](const LSPError &L) -> Error {
+            Message = L.Message;
+            Code = L.Code;
+            return Error::success();
+          }))
+    Message = toString(std::move(Unhandled));
+
+  return json::Object{
+      {"message", std::move(Message)},
+      {"code", int64_t(Code)},
+  };
+}
+
+Error decodeError(const json::Object &O) {
+  std::string Msg = O.getString("message").getValueOr("Unspecified error");
+  if (auto Code = O.getInteger("code"))
+    return make_error<LSPError>(std::move(Msg), ErrorCode(*Code));
+  return make_error<StringError>(std::move(Msg), inconvertibleErrorCode());
+}
+
+// C "closure" for XPCTransport::loop() method
+namespace xpcClosure {
+void connection_handler(xpc_connection_t clientConnection);
+}
+
+class XPCTransport : public Transport {
+public:
+  XPCTransport() {}
+
+  void notify(StringRef Method, json::Value Params) override {
+    sendMessage(json::Object{
+        {"jsonrpc", "2.0"},
+        {"method", Method},
+        {"params", std::move(Params)},
+    });
+  }
+  void call(StringRef Method, json::Value Params, json::Value ID) override {
+    sendMessage(json::Object{
+        {"jsonrpc", "2.0"},
+        {"id", std::move(ID)},
+        {"method", Method},
+        {"params", std::move(Params)},
+    });
+  }
+  void reply(json::Value ID, Expected<json::Value> Result) override {
+    if (Result) {
+      sendMessage(json::Object{
+          {"jsonrpc", "2.0"},
+          {"id", std::move(ID)},
+          {"result", std::move(*Result)},
+      });
+    } else {
+      sendMessage(json::Object{
+          {"jsonrpc", "2.0"},
+          {"id", std::move(ID)},
+          {"error", encodeError(Result.takeError())},
+      });
+    }
+  }
+
+  Error loop(MessageHandler &Handler) override;
+
+private:
+  // Needs access to handleMessage() and resetClientConnection()
+  friend void xpcClosure::connection_handler(xpc_connection_t clientConnection);
+
+  // Dispatches incoming message to Handler onNotify/onCall/onReply.
+  bool handleMessage(json::Value Message, MessageHandler &Handler);
+  void sendMessage(json::Value Message) {
+    xpc_object_t response = jsonToXpc(Message);
+    xpc_connection_send_message(clientConnection, response);
+    xpc_release(response);
+  }
+  void resetClientConnection(xpc_connection_t newClientConnection) {
+    clientConnection = newClientConnection;
+  }
+  xpc_connection_t clientConnection;
+};
+
+bool XPCTransport::handleMessage(json::Value Message, MessageHandler &Handler) {
+  // Message must be an object with "jsonrpc":"2.0".
+  auto *Object = Message.getAsObject();
+  if (!Object || Object->getString("jsonrpc") != Optional<StringRef>("2.0")) {
+    elog("Not a JSON-RPC 2.0 message: {0:2}", Message);
+    return false;
+  }
+  // ID may be any JSON value. If absent, this is a notification.
+  Optional<json::Value> ID;
+  if (auto *I = Object->get("id"))
+    ID = std::move(*I);
+  auto Method = Object->getString("method");
+  if (!Method) { // This is a response.
+    if (!ID) {
+      elog("No method and no response ID: {0:2}", Message);
+      return false;
+    }
+    if (auto *Err = Object->getObject("error"))
+      return Handler.onReply(std::move(*ID), decodeError(*Err));
+    // Result should be given, use null if not.
+    json::Value Result = nullptr;
+    if (auto *R = Object->get("result"))
+      Result = std::move(*R);
+    return Handler.onReply(std::move(*ID), std::move(Result));
+  }
+  // Params should be given, use null if not.
+  json::Value Params = nullptr;
+  if (auto *P = Object->get("params"))
+    Params = std::move(*P);
+
+  if (ID)
+    return Handler.onCall(*Method, std::move(Params), std::move(*ID));
+  else
+    return Handler.onNotify(*Method, std::move(Params));
+}
+
+namespace xpcClosure {
+// "owner" of this "closure object" - necessary for propagating connection to
+// XPCTransport so it can send messages to the client.
+XPCTransport *TransportObject = nullptr;
+Transport::MessageHandler *HandlerPtr = nullptr;
+
+void connection_handler(xpc_connection_t clientConnection) {
+  xpc_connection_set_target_queue(clientConnection, dispatch_get_main_queue());
+
+  xpc_transaction_begin();
+
+  TransportObject->resetClientConnection(clientConnection);
+
+  xpc_connection_set_event_handler(clientConnection, ^(xpc_object_t message) {
+    if (message == XPC_ERROR_CONNECTION_INVALID) {
+      // connection is being terminated
+      log("Received XPC_ERROR_CONNECTION_INVALID message - returning from the "
+          "event_handler.");
+      return;
+    }
+
+    if (xpc_get_type(message) != XPC_TYPE_DICTIONARY) {
+      log("Received XPC message of unknown type - returning from the "
+          "event_handler.");
+      return;
+    }
+
+    const json::Value Doc = xpcToJson(message);
+    if (Doc == json::Value(nullptr)) {
+      log("XPC message was converted to Null JSON message - returning from the "
+          "event_handler.");
+      return;
+    }
+
+    vlog("<<< {0}\n", Doc);
+
+    if (!TransportObject->handleMessage(std::move(Doc), *HandlerPtr)) {
+      log("Received exit notification - cancelling connection.");
+      xpc_connection_cancel(xpc_dictionary_get_remote_connection(message));
+      xpc_transaction_end();
+    }
+  });
+
+  xpc_connection_resume(clientConnection);
+}
+} // namespace xpcClosure
+
+Error XPCTransport::loop(MessageHandler &Handler) {
+  assert(xpcClosure::TransportObject == nullptr &&
+         "TransportObject has already been set.");
+  // This looks scary since lifetime of this (or any) XPCTransport object has
+  // to fully contain lifetime of any XPC connection. In practise any Transport
+  // object is destroyed only at the end of main() which is always after
+  // exit of xpc_main().
+  xpcClosure::TransportObject = this;
+
+  assert(xpcClosure::HandlerPtr == nullptr &&
+         "HandlerPtr has already been set.");
+  xpcClosure::HandlerPtr = &Handler;
+
+  xpc_main(xpcClosure::connection_handler);
+  // xpc_main doesn't ever return
+  return errorCodeToError(std::make_error_code(std::errc::io_error));
+}
+
+} // namespace
+
+namespace clang {
+namespace clangd {
+
+std::unique_ptr<Transport> newXPCTransport() {
+  return llvm::make_unique<XPCTransport>();
+}
+
+} // namespace clangd
+} // namespace clang

clangd/xpc/cmake/Info.plist.in

+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>CFBundleDevelopmentRegion</key>
+  <string>English</string>
+  <key>CFBundleExecutable</key>
+  <string>${CLANGD_XPC_FRAMEWORK_NAME}</string>
+  <key>CFBundleIconFile</key>
+  <string></string>
+  <key>CFBundleIdentifier</key>
+  <string>org.llvm.${CLANGD_XPC_FRAMEWORK_NAME}</string>
+  <key>CFBundleInfoDictionaryVersion</key>
+  <string>6.0</string>
+  <key>CFBundleName</key>
+  <string>${CLANGD_XPC_FRAMEWORK_NAME}</string>
+  <key>CFBundlePackageType</key>
+  <string>FMWK</string>
+  <key>CFBundleSignature</key>
+  <string>????</string>
+  <key>CFBundleVersion</key>
+  <string></string>
+  <key>CFBundleShortVersionString</key>
+  <string>1.0</string>
+  <key>CSResourcesFileMapped</key>
+  <true/>
+</dict>
+</plist>

clangd/xpc/cmake/XPCServiceInfo.plist.in

+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+  <key>CFBundleExecutable</key>
+  <string>${CLANGD_XPC_SERVICE_NAME}</string>
+  <key>CFBundleIdentifier</key>
+  <string>${CLANGD_XPC_SERVICE_BUNDLE_NAME}</string>
+  <key>CFBundleInfoDictionaryVersion</key>
+  <string>6.0</string>
+  <key>CFBundleName</key>
+  <string>${CLANGD_XPC_SERVICE_NAME}</string>
+  <key>CFBundlePackageType</key>
+  <string>XPC!</string>
+  <key>CFBundleVersion</key>
+  <string></string>
+  <key>CFBundleShortVersionString</key>
+  <string>1.0</string>
+  <key>XPCService</key>
+  <dict>
+    <key>ServiceType</key>
+    <string>Application</string>
+    <key>EnvironmentVariables</key>
+    <dict>
+        <key>CLANGD_AS_XPC_SERVICE</key>
+        <string>1</string>
+    </dict>
+  </dict>
+</dict>
+</plist>

clangd/xpc/cmake/modules/CreateClangdXPCFramework.cmake

+# Creates the ClangdXPC framework.
+macro(create_clangd_xpc_framework target name)
+  set(CLANGD_FRAMEWORK_LOCATION "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/${name}.framework")
+  set(CLANGD_FRAMEWORK_OUT_LOCATION "${CLANGD_FRAMEWORK_LOCATION}/Versions/A")
+
+  # Create the framework info PLIST.
+  set(CLANGD_XPC_FRAMEWORK_NAME "${name}")
+  configure_file(
+    "${CLANGD_XPC_SOURCE_DIR}/cmake/Info.plist.in"
+    "${CLANGD_XPC_BINARY_DIR}/${name}.Info.plist")
+
+  set(CLANGD_XPC_SERVICE_NAME "clangd")
+  set(CLANGD_XPC_SERVICE_OUT_LOCATION
+      "${CLANGD_FRAMEWORK_OUT_LOCATION}/XPCServices/${CLANGD_XPC_SERVICE_NAME}.xpc/Contents")
+
+  # Create the XPC service info PLIST.
+  set(CLANGD_XPC_SERVICE_BUNDLE_NAME "org.llvm.${CLANGD_XPC_SERVICE_NAME}")
+  configure_file(
+    "${CLANGD_XPC_SOURCE_DIR}/cmake/XPCServiceInfo.plist.in"
+    "${CLANGD_XPC_BINARY_DIR}/${name}Service.Info.plist")
+
+  # Create the custom command
+  add_custom_command(OUTPUT ${CLANGD_FRAMEWORK_LOCATION}
+    # Copy the PLIST.
+    COMMAND ${CMAKE_COMMAND} -E copy
+      "${CLANGD_XPC_BINARY_DIR}/${name}.Info.plist"
+      "${CLANGD_FRAMEWORK_OUT_LOCATION}/Resources/Info.plist"
+
+    # Copy the framework binary.
+    COMMAND ${CMAKE_COMMAND} -E copy
+       "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}/lib${target}.dylib"
+       "${CLANGD_FRAMEWORK_OUT_LOCATION}/${name}"
+
+    # Copy the XPC Service PLIST.
+    COMMAND ${CMAKE_COMMAND} -E copy
+      "${CLANGD_XPC_BINARY_DIR}/${name}Service.Info.plist"
+      "${CLANGD_XPC_SERVICE_OUT_LOCATION}/Info.plist"
+
+    # Copy the Clangd binary.
+    COMMAND ${CMAKE_COMMAND} -E copy
+      "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/clangd"
+      "${CLANGD_XPC_SERVICE_OUT_LOCATION}/MacOS/clangd"
+
+     COMMAND ${CMAKE_COMMAND} -E create_symlink "A"
+     "${CLANGD_FRAMEWORK_LOCATION}/Versions/Current"
+
+     COMMAND ${CMAKE_COMMAND} -E create_symlink
+     "Versions/Current/Resources"
+     "${CLANGD_FRAMEWORK_LOCATION}/Resources"
+
+     COMMAND ${CMAKE_COMMAND} -E create_symlink
+     "Versions/Current/XPCServices"
+     "${CLANGD_FRAMEWORK_LOCATION}/XPCServices"
+
+     COMMAND ${CMAKE_COMMAND} -E create_symlink
+     "Versions/Current/${name}"
+     "${CLANGD_FRAMEWORK_LOCATION}/${name}"
+
+    DEPENDS
+      "${CLANGD_XPC_BINARY_DIR}/${name}.Info.plist"
+      "${CLANGD_XPC_BINARY_DIR}/${name}Service.Info.plist"
+      clangd
+    COMMENT "Creating ClangdXPC framework"
+    VERBATIM
+  )
+
+  add_custom_target(
+    ClangdXPC
+    DEPENDS
+    ${target}
+    ${CLANGD_FRAMEWORK_LOCATION}
+  )
+endmacro(create_clangd_xpc_framework)

clangd/xpc/framework/CMakeLists.txt

+
+set(SOURCES
+    ClangdXPC.cpp)
+add_clang_library(ClangdXPCLib SHARED
+  ${SOURCES}
+  DEPENDS
+  clangd
+)
+create_clangd_xpc_framework(ClangdXPCLib "ClangdXPC")

clangd/xpc/framework/ClangdXPC.cpp

+
+/// Returns the bundle identifier of the Clangd XPC service.
+extern "C" const char *clangd_xpc_get_bundle_identifier() {
+  return "org.llvm.clangd";
+}

clangd/xpc/test-client/CMakeLists.txt

+include_directories(
+  ${CMAKE_CURRENT_SOURCE_DIR}/../../
+)
+
+add_clang_tool(
+  clangd-xpc-test-client
+  ClangdXPCTestClient.cpp
+
+  DEPENDS ClangdXPC
+)
+
+set(LLVM_LINK_COMPONENTS
+    support
+)
+
+target_link_libraries(clangd-xpc-test-client
+  PRIVATE
+  clangBasic
+  clangDaemon
+  clangFormat
+  clangFrontend
+  clangSema
+  clangTooling
+  clangToolingCore
+  clangdXpcJsonConversions
+)

clangd/xpc/test-client/ClangdXPCTestClient.cpp

+#include "xpc/Conversion.h"
+#include "clang/Basic/LLVM.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/LineIterator.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/raw_ostream.h"
+#include <dlfcn.h>
+#include <stdio.h>
+#include <string>
+#include <xpc/xpc.h>
+
+typedef const char *(*clangd_xpc_get_bundle_identifier_t)(void);
+
+using namespace llvm;
+using namespace clang;
+
+std::string getLibraryPath() {
+  Dl_info info;
+  if (dladdr((void *)(uintptr_t)getLibraryPath, &info) == 0)
+    llvm_unreachable("Call to dladdr() failed");
+  llvm::SmallString<128> LibClangPath;
+  LibClangPath = llvm::sys::path::parent_path(
+      llvm::sys::path::parent_path(info.dli_fname));
+  llvm::sys::path::append(LibClangPath, "lib", "ClangdXPC.framework",
+                          "ClangdXPC");
+  return LibClangPath.str();
+}
+
+static void dumpXPCObject(xpc_object_t Object, llvm::raw_ostream &OS) {
+  xpc_type_t Type = xpc_get_type(Object);
+  if (Type == XPC_TYPE_DICTIONARY) {
+    json::Value Json = clang::clangd::xpcToJson(Object);
+    OS << Json;
+  } else {
+    OS << "<UNKNOWN>";
+  }
+}
+
+int main(int argc, char *argv[]) {
+  // Open the ClangdXPC dylib in the framework.
+  std::string LibPath = getLibraryPath();
+  void *dlHandle = dlopen(LibPath.c_str(), RTLD_LOCAL | RTLD_FIRST);
+  if (!dlHandle)
+    return 1;
+
+  // Lookup the XPC service bundle name, and launch it.
+  clangd_xpc_get_bundle_identifier_t clangd_xpc_get_bundle_identifier =
+      (clangd_xpc_get_bundle_identifier_t)dlsym(
+          dlHandle, "clangd_xpc_get_bundle_identifier");
+  xpc_connection_t conn = xpc_connection_create(
+      clangd_xpc_get_bundle_identifier(), dispatch_get_main_queue());
+
+  // Dump the XPC events.
+  xpc_connection_set_event_handler(conn, ^(xpc_object_t event) {
+    if (event == XPC_ERROR_CONNECTION_INVALID) {
+      llvm::errs() << "Received XPC_ERROR_CONNECTION_INVALID.";
+      exit(EXIT_SUCCESS);
+    }
+    if (event == XPC_ERROR_CONNECTION_INTERRUPTED) {
+      llvm::errs() << "Received XPC_ERROR_CONNECTION_INTERRUPTED.";
+      exit(EXIT_SUCCESS);
+    }
+
+    dumpXPCObject(event, llvm::outs());
+    llvm::outs() << "\n";
+  });
+
+  xpc_connection_resume(conn);
+
+  // Read the input to determine the things to send to clangd.
+  llvm::ErrorOr<std::unique_ptr<MemoryBuffer>> Stdin =
+      llvm::MemoryBuffer::getSTDIN();
+  if (!Stdin) {
+    llvm::errs() << "Failed to get STDIN!\n";
+    return 1;
+  }
+  for (llvm::line_iterator It(**Stdin, /*SkipBlanks=*/true,
+                              /*CommentMarker=*/'#');
+       !It.is_at_eof(); ++It) {
+    StringRef Line = *It;
+    if (auto Request = json::parse(Line)) {
+      xpc_object_t Object = clangd::jsonToXpc(*Request);
+      xpc_connection_send_message(conn, Object);
+    } else {
+      llvm::errs() << llvm::Twine("JSON parse error: ")
+                   << llvm::toString(Request.takeError());
+      return 1;
+    }
+  }
+
+  dispatch_main();
+
+  // dispatch_main() doesn't return
+  return EXIT_FAILURE;
+}

test/CMakeLists.txt

if (CMAKE_CFG_INTDIR STREQUAL ".")
   set(LLVM_BUILD_MODE ".")
 else ()
   set(LLVM_BUILD_MODE "%(build_mode)s")
 endif ()
 
 string(REPLACE ${CMAKE_CFG_INTDIR} ${LLVM_BUILD_MODE} CLANG_TOOLS_DIR ${LLVM_RUNTIME_OUTPUT_INTDIR})
 
 llvm_canonicalize_cmake_booleans(
-  CLANG_ENABLE_STATIC_ANALYZER)
+  CLANG_ENABLE_STATIC_ANALYZER
+  CLANGD_BUILD_XPC_SUPPORT)
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
   )
 
 configure_lit_site_cfg(
   ${CMAKE_CURRENT_SOURCE_DIR}/Unit/lit.site.cfg.in
set(CLANG_TOOLS_TEST_DEPS
   ExtraToolsUnitTests
 
   # For the clang-tidy libclang integration test.
   c-index-test
   # clang-tidy tests require it.
   clang-headers
 
   clang-tidy
-  )
 
+if(CLANGD_BUILD_XPC_SUPPORT)
+  list(APPEND CLANG_TOOLS_TEST_DEPS clangd-xpc-test-client)
+endif()
+
 set(CLANGD_TEST_DEPS
   clangd
   ClangdTests
   # clangd-related tools which don't have tests, add them to the test to make
   # sure we don't introduce new changes that break their compilations.
   clangd-indexer
   dexp
   )

test/clangd/xpc/initialize.test

+# RUN: clangd-xpc-test-client < %s | FileCheck %s
+# REQUIRES: clangd-xpc-support
+
+{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"processId":123,"rootUri":"test:///workspace","capabilities":{},"trace":"off"}}
+# CHECK: {"id":0,"jsonrpc":"2.0","result":{"capabilities"
+
+{"jsonrpc":"2.0","id":3,"method":"shutdown"}
+# CHECK: {"id":3,"jsonrpc":"2.0","result":null}
+
+{"jsonrpc":"2.0","method":"exit"}

test/lit.cfg

 # Exclude MSYS due to transforming '/' to 'X:/mingwroot/'.
 if not platform.system() in ['Windows'] or not execute_external:
     config.available_features.add('shell-preserves-root')
 
 # ANSI escape sequences in non-dumb terminal
 if platform.system() not in ['Windows']:
     config.available_features.add('ansi-escape-sequences')
 
+# XPC support for Clangd.
+if config.clangd_xpc_support:
+    config.available_features.add('clangd-xpc-support')
+
 if config.clang_staticanalyzer:
     config.available_features.add('static-analyzer')
 
 check_clang_tidy = os.path.join(
     config.test_source_root, "clang-tidy", "check_clang_tidy.py")
 config.substitutions.append(
     ('%check_clang_tidy',
      '%s %s' % (config.python_executable, check_clang_tidy)) )

test/lit.site.cfg.in

 @LIT_SITE_CFG_IN_HEADER@
 
 import sys
 
 config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
 config.llvm_libs_dir = "@LLVM_LIBS_DIR@"
 config.lit_tools_dir = "@LLVM_LIT_TOOLS_DIR@"
 config.clang_tools_binary_dir = "@CLANG_TOOLS_BINARY_DIR@"
 config.clang_tools_dir = "@CLANG_TOOLS_DIR@"
 config.clang_libs_dir = "@SHLIBDIR@"
 config.python_executable = "@PYTHON_EXECUTABLE@"
 config.target_triple = "@TARGET_TRIPLE@"
 config.clang_staticanalyzer = @CLANG_ENABLE_STATIC_ANALYZER@
+config.clangd_xpc_support = @CLANGD_BUILD_XPC_SUPPORT@
 
 # Support substitution of the tools and libs dirs with user parameters. This is
 # used when we can't determine the tool dir at configuration time.
 try:
     config.clang_tools_dir = config.clang_tools_dir % lit_config.params
     config.llvm_tools_dir = config.llvm_tools_dir % lit_config.params
     config.llvm_libs_dir = config.llvm_libs_dir % lit_config.params
 except KeyError:

unittests/clangd/CMakeLists.txt

target_link_libraries(ClangdTests
   clangSema
   clangSerialization
   clangTooling
   clangToolingCore
   clangToolingInclusions
   LLVMSupport
   LLVMTestingSupport
   )
+
+if (CLANGD_BUILD_XPC)
+  add_subdirectory(xpc)
+endif ()

unittests/clangd/xpc/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  support
+  )
+
+get_filename_component(CLANGD_SOURCE_DIR
+  ${CMAKE_CURRENT_SOURCE_DIR}/../../clangd REALPATH)
+include_directories(
+  ${CLANGD_SOURCE_DIR}
+  )
+
+add_extra_unittest(ClangdXpcTests
+  ConversionTests.cpp
+  )
+
+target_link_libraries(ClangdXpcTests
+  PRIVATE
+  clangdXpcJsonConversions
+  clangDaemon
+  LLVMSupport
+  LLVMTestingSupport
+  )

unittests/clangd/xpc/ConversionTests.cpp

+//===-- ConversionTests.cpp  --------------------------*- C++ -*-----------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "xpc/Conversion.h"
+#include "gtest/gtest.h"
+
+#include <limits>
+
+namespace clang {
+namespace clangd {
+namespace {
+
+using namespace llvm;
+
+TEST(JsonXpcConversionTest, JsonToXpcToJson) {
+
+  for (auto &testcase :
+       {json::Value(false), json::Value(3.14), json::Value(42),
+        json::Value(-100), json::Value("foo"), json::Value(""),
+        json::Value("123"), json::Value(" "),
+        json::Value{true, "foo", nullptr, 42},
+        json::Value(json::Object{
+            {"a", true}, {"b", "foo"}, {"c", nullptr}, {"d", 42}})}) {
+    EXPECT_TRUE(testcase == xpcToJson(jsonToXpc(testcase))) << testcase;
+  }
+}
+
+} // namespace
+} // namespace clangd
+} // namespace clang