[llvm] [Debuginfod] Add HTTP Server to Debuginfod library.

[noajshu]

Summary

This provides a minimal HTTP server interface and an implementation wrapping cpp-httplib in the Debuginfod library. If the Curl HTTP client is available (D112753) the server is tested by pinging it with the client.

[phosek]

llvm/include/llvm/Support/HTTPServer.h

1 ↗

(On Diff #389094)

This file is missing LLVM header.

llvm/lib/Support/HTTPServer.cpp

1 ↗

(On Diff #389094)

This file is missing LLVM header.

19 ↗

(On Diff #389094)

I wonder if we should perhaps move this to a separate method (like get) and take the path as an argument so we can potentially support multiple handlers.

23 ↗

(On Diff #389094)

Having both Response and Resp is confusing, can we use a different name here?

57 ↗

(On Diff #389094)

Calling listen on an unbound server should probably return an error rather than success.

[noajshu]

Incorporate feedback, refactor, add unit tests, and rebase against D113218.

[noajshu]

llvm/lib/Support/HTTPServer.cpp

19 ↗

(On Diff #389094)

My initial thought was to support multiple "handlers" using one big handler that would parse the URL and dispatch to the appropriate sub-handler. However it does seem more user-friendly to expose the same interface as cpp-httplib, letting the HTTPServer deal with the url parsing.

[noajshu]

Implement streaming responses and file streaming helper; add unit tests of client timeouts and streaming string / file responses.

[dblaikie]

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

[phosek]

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

[dblaikie]

In D114415#3159988, @phosek wrote:

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

Oh, fair enough - just checking it wasn't only being implemented for testing. Would the python script from the llvm-symbolizer test be adequate for the production/local developer scenarios you have in mind? (I don't mind the C++ too much, but just trying to understand the landscape, tradeoffs, etc)

[noajshu]

In D114415#3159995, @dblaikie wrote:

In D114415#3159988, @phosek wrote:

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

Oh, fair enough - just checking it wasn't only being implemented for testing. Would the python script from the llvm-symbolizer test be adequate for the production/local developer scenarios you have in mind? (I don't mind the C++ too much, but just trying to understand the landscape, tradeoffs, etc)

The python script used in D113717 (which might also get used to test D112759) only takes care of the HTTP static file serving needed for debuginfod. There is another component, which is to search the filesystem for debug binaries and assemble the collection of artifacts to serve. I'll be publishing a diff shortly which implements this functionality using LLVM's object parsing and filesystem utilities. Then we will have all the pieces needed for a basic C++ debuginfod in LLVM.

There are workarounds we could use to get a simple debuginfod server working without this LLVM HTTP server. E.g., we could use CGI and FastCGI, or we could create symlinks / copies to the discovered debug data in a single static file serving path. These workarounds came with trade offs, and there are other tools in LLVM that could take advantage of a cross-platform HTTP server such as bisectd (D113030). This is what led to the goal of getting an HTTP server in LLVM's supporting libraries. (A side benefit is that we can now thoroughly unit test the HTTP client by letting it communicate with the server, but this isn't the motivation :) .)

[dblaikie]

In D114415#3162737, @noajshu wrote:

In D114415#3159995, @dblaikie wrote:

In D114415#3159988, @phosek wrote:

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

Oh, fair enough - just checking it wasn't only being implemented for testing. Would the python script from the llvm-symbolizer test be adequate for the production/local developer scenarios you have in mind? (I don't mind the C++ too much, but just trying to understand the landscape, tradeoffs, etc)

The python script used in D113717 (which might also get used to test D112759) only takes care of the HTTP static file serving needed for debuginfod. There is another component, which is to search the filesystem for debug binaries and assemble the collection of artifacts to serve. I'll be publishing a diff shortly which implements this functionality using LLVM's object parsing and filesystem utilities. Then we will have all the pieces needed for a basic C++ debuginfod in LLVM.

There are workarounds we could use to get a simple debuginfod server working without this LLVM HTTP server. E.g., we could use CGI and FastCGI, or we could create symlinks / copies to the discovered debug data in a single static file serving path. These workarounds came with trade offs, and there are other tools in LLVM that could take advantage of a cross-platform HTTP server such as bisectd (D113030). This is what led to the goal of getting an HTTP server in LLVM's supporting libraries. (A side benefit is that we can now thoroughly unit test the HTTP client by letting it communicate with the server, but this isn't the motivation :) .)

Fair enough - thanks for the context!

llvm/include/llvm/Support/HTTPServer.h

29–31 ↗

(On Diff #390457)

Should the first element instead be a separate member, since it seems it's not like the rest of the elements/will be handled differently?

llvm/lib/Support/HTTPServer.cpp

58 ↗

(On Diff #390457)

Could this parameter (the last lambda, the "CompletionHandler" member in StreamingHTTPResponse) be move-only (non-copyable) in that case it could capture-by-move the std::unique_ptr<MemoryBuffer> and be a bit more robust in terms of memory management?

67 ↗

(On Diff #390457)

Perhaps HTTPRequestHandler should be an llvm::function_ref since it doesn't appear to outlive this call, if I'm understanding the code correctly?

94 ↗

(On Diff #390457)

should this be const auto&? (I'm not sure how big/expensive to copy these objects are, perhaps it's suitable to copy them)

101–102 ↗

(On Diff #390457)

Looks like this could use StringRef instead of std::string, to avoid copying the contents? Or are there cases where a provider might want to return content by value, rather than from some underlying/longer-lived storage? (worth providing something that could be optimal for both cases somehow - like providing an optional std::string out parameter that the provider could populate if it doesn't have its own long-lived backing store?)

125–130 ↗

(On Diff #390457)

If the underlying API requires a c-style string, maybe simpler to expose that up through the caller layers? Rather than allocating a buffer to create a null terminated string when the caller might already have one to pass down anyway? (alternatively, I guess, if you use Twine as the parameter type rather than StringRef, then at least in the cases where the Twine does refer to a null terminated string it won't have to copy the string again - currently since the HostInterface is StringRef, it'd always have to copy into the buffer and append a null terminator)

142–144 ↗

(On Diff #390457)

Follow-up work to use httplib set_error_handler to improve these error messages, perhaps?

158–162 ↗

(On Diff #390457)

This looks out of date, presumably it doesn't compile? (HTTPServer doesn't declare any ctors, so I guess this would fail to compile)

llvm/unittests/Support/HTTPServer.cpp

73 ↗

(On Diff #390457)

probably make this by-value?

77 ↗

(On Diff #390457)

Is some corresponding de-initialize/teardown required/desirable? (perhaps some scoped initialize/de-initialize) or put the initialize in a global ctor (or test fixture (SetUp? Whatever the one is that's called before any test (but not before every test individually)) in the test file if there's no expectation/desire to initialize/teardown around each test?

82 ↗

(On Diff #390457)

The MemoryBuffer:: probably isn't needed here, I'd have thought?

184–189 ↗

(On Diff #390457)

timing tests like this can be sensitive to machine load - perhaps setting the acceptable timeout much higher than the actual timeout would be good to increase test stability? (eg: the 40 is probably fine - the actual 50ms delay shouldn't ever be shorter than requested, and so it probably shouldn't tickle that test (though it's possible - if the test code were delayed in reaching the timeout call while the delay was already running, it could appear as though the delay was shorter than expected) - but changing the 60 up to something well above the threshold might be good).

Though also, it's not your job to test that HTTPClient implements timeouts correctly, only that you pass them down through the API - so if there's a more robust way to test that property without actually having to run real timeouts (or at least not testing them so precisely) it might be helpful.

206 ↗

(On Diff #390457)

I think there's probably a matcher that matches all the elements in a more terse format than having to check each separately? ( https://stackoverflow.com/q/1460703 )

215–217 ↗

(On Diff #390457)

Prefer llvm_unreachable over assert(false)

[noajshu]

llvm/lib/Support/HTTPServer.cpp

58 ↗

(On Diff #390457)

That's a great idea! Cpp-httplib converts the content provider to a ContentProvider which is a std::function. This ends up copying the lambda.

I was thinking we might refactor the interface, so instead of returning the response (or response provider) you can interact with a "HTTPServerRequest" object similar to how cpp-httplib's own API works.
Their API is more flexible. For example, you can decide at runtime whether you want to do streaming response handling or return a single response string. In the wrapper implementation here you have to either return a StreamingHTTPResponse or an HTTPResponse, fixed at compile time.

Even if we mirror the httplib interface, the problem remains that we cannot pass a non-copyable lambda directly to httplib's set_content_provider. I'm not sure if there is a nice way around this problem.

67 ↗

(On Diff #390457)

Ah, I think the nomenclature is misleading. HTTPServer::get should be called something like HTTPServer::registerGetHandler. So it returns, and then you can later call bind() etc., but the function provided in the argument may be called much later after listen when there is a request to be handled.

llvm/unittests/Support/HTTPServer.cpp

77 ↗

(On Diff #390457)

There is a discussion about the need for a "manual" global HTTPClient::initialize() here. Briefly, Curl's global initialization is not thread-safe and may load other libs; the loader lock on windows prevents use of a static ctor to initialize. We added HTTPClient::initialize() to InitLLVM for convenience, but we had to remove it when HTTPClient got moved to the Debuginfod lib. So now the HTTPClient::initialize must be called "manually". By the way, there is ongoing discussion within libcurl community on making curl global init thread safe.

On the other hand, we can safely deinitialize with a static destructor as you suggest.
This is a great idea which jhenderson also suggested on D113717, and you can see the implementation here:
https://reviews.llvm.org/D113717#change-LO0j60bMM9xj
So there is no need for the de-initialize / teardown here, pending that diff.

[dblaikie]

llvm/lib/Support/HTTPServer.cpp

58 ↗

(On Diff #390457)

Ah, oh well. Thanks for the context!

[noajshu]

Refactor HTTPServerRequest interface to be more close to cpphttplib, and address all of @dblaikie 's comments.

[noajshu]

Hi @dblaikie thank you so much for the review! I believe I have addressed all your comments, however they disappeared because I moved the HTTPServer into the Debuginfod library. You can see them on the previous revision.

llvm/lib/Support/HTTPServer.cpp

101–102 ↗

(On Diff #390457)

I've switched it to take a StringRef -- for the only place this is used currently (streaming chunks of a file) you have a persistent MemoryBuffer which is responsible for the storage.

125–130 ↗

(On Diff #390457)

It's a good point that no matter whether we use cpp-httplib or something else, ultimately the system call will need a c-style string. I've changed it to a const char* and I could switch to a Twine if it's better.

142–144 ↗

(On Diff #390457)

Good idea!
It looks like httplib's set_error_handler is used to set the default response content (e.g. an error page) when the user's handler returns an HTTP error code (>400). More detailed error information is available to us by checking errno, but cpp-httplib doesn't check it other than to see if the bind / listen was successful.

In a future TCPSocket implementation for LLVM it would be great to parse the errno better to give more helpful messages to the user.

llvm/unittests/Support/HTTPServer.cpp

82 ↗

(On Diff #390457)

Ah, sadly it is required, unless you can see a better way around the following.
The Body is a WritableMemoryBuffer, whose override of getBuffer returns an MutableArrayRef<char> (as opposed to the parent's virtual method MemoryBuffer::getBuffer which returns a StringRef). For some reason there is no operator== implemented between MutableArrayRef<char> and std::string. So we have to manually specify use of the parent method

184–189 ↗

(On Diff #390457)

This is a really good observation. I'm concerned that even with a lot of extra wait time, it might flake under some extreme system load. Let's remove it for now, leaving just the test in which a timeout is expected.
HTTPClient::setTimeout is just directly calling curl_easy_setopt so I think that the libcurl tests should cover it for the case where a (nonzero) timeout does not expire.

[dblaikie]

I think this is mostly OK. (still, might be worth initializing/deinitializing in test fixtures, to isolate the tests a bit more - but probably not a big deal/not the sort of bugs we're likely to see (eg: where one test taints the libcurl state and another fails in weird ways because of that taint))

llvm/unittests/Support/HTTPServer.cpp

82 ↗

(On Diff #390457)

Seems like addressing the missing comparison would be good as a separate patch.

[noajshu]

Use test fixtures to perform HTTP Client initialize and teardown for the client-server tests.

[noajshu]

@dblaikie Thanks for taking another look and the LGTM! Per your suggestion I've added a test fixture to call the HTTPClient initialization and teardown methods. Thanks for all the comments on this patch.

It seems at first llvm-debuginfod will be the sole user tool. So, my inclination is to hold off on committing this until the llvm-debuginfod server (D114845 and D114846) is ready to land.

[dblaikie]

In D114415#3250033, @noajshu wrote:

@dblaikie Thanks for taking another look and the LGTM! Per your suggestion I've added a test fixture to call the HTTPClient initialization and teardown methods. Thanks for all the comments on this patch.

It seems at first llvm-debuginfod will be the sole user tool. So, my inclination is to hold off on committing this until the llvm-debuginfod server (D114845 and D114846) is ready to land.

Yep, sounds fair.

[noajshu]

Replace caputure-by-reference of request handler with capture-by-copy, and add a unit test to verify temporary handlers can be registered and work correctly.

[noajshu]

Fix typo in unit tests.

[noajshu]

Rebase against main

[noajshu]

Rebase against main

[noajshu]

Fix HTTP Server / Client unit tests that were broken after rebase

llvm/include/llvm/Debuginfod/HTTPServer.h

+//===-- llvm/Debuginfod/HTTPServer.h - HTTP server library ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains the declarations of the HTTPServer and HTTPServerRequest
+/// classes, the HTTPResponse, and StreamingHTTPResponse structs, and the
+/// streamFile function.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_HTTP_SERVER_H
+#define LLVM_SUPPORT_HTTP_SERVER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+
+#ifdef LLVM_ENABLE_HTTPLIB
+// forward declarations
+namespace httplib {
+class Request;
+class Response;
+class Server;
+} // namespace httplib
+#endif
+
+namespace llvm {
+
+struct HTTPResponse;
+struct StreamingHTTPResponse;
+class HTTPServer;
+
+class HTTPServerRequest {
+  friend HTTPServer;
+
+#ifdef LLVM_ENABLE_HTTPLIB
+private:
+  HTTPServerRequest(const httplib::Request &HTTPLibRequest,
+                    httplib::Response &HTTPLibResponse);
+  httplib::Response &HTTPLibResponse;
+#endif
+
+public:
+  std::string UrlPath;
+  /// The elements correspond to match groups in the url path matching regex.
+  SmallVector<std::string, 1> UrlPathMatches;
+
+  // TODO bring in HTTP headers
+
+  void setResponse(StreamingHTTPResponse Response);
+  void setResponse(HTTPResponse Response);
+};
+
+struct HTTPResponse {
+  unsigned Code;
+  const char *ContentType;
+  StringRef Body;
+};
+
+typedef std::function<void(HTTPServerRequest &)> HTTPRequestHandler;
+
+/// An HTTPContentProvider is called by the HTTPServer to obtain chunks of the
+/// streaming response body. The returned chunk should be located at Offset
+/// bytes and have Length bytes.
+typedef std::function<StringRef(size_t /*Offset*/, size_t /*Length*/)>
+    HTTPContentProvider;
+
+/// Wraps the content provider with HTTP Status code and headers.
+struct StreamingHTTPResponse {
+  unsigned Code;
+  const char *ContentType;
+  size_t ContentLength;
+  HTTPContentProvider Provider;
+  /// Called after the response transfer is complete with the success value of
+  /// the transfer.
+  std::function<void(bool)> CompletionHandler = [](bool Success) {};
+};
+
+/// Sets the response to stream the file at FilePath, if available, and
+/// otherwise an HTTP 404 error response.
+bool streamFile(HTTPServerRequest &Request, StringRef FilePath);
+
+/// An HTTP server which can listen on a single TCP/IP port for HTTP
+/// requests and delgate them to the appropriate registered handler.
+class HTTPServer {
+#ifdef LLVM_ENABLE_HTTPLIB
+  std::unique_ptr<httplib::Server> Server;
+  unsigned Port = 0;
+#endif
+public:
+  HTTPServer();
+  ~HTTPServer();
+
+  /// Returns true only if LLVM has been compiled with a working HTTPServer.
+  static bool isAvailable();
+
+  /// Registers a URL pattern routing rule. When the server is listening, each
+  /// request is dispatched to the first registered handler whose UrlPathPattern
+  /// matches the UrlPath.
+  Error get(StringRef UrlPathPattern, HTTPRequestHandler Handler);
+
+  /// Attempts to assign the requested port and interface, returning an Error
+  /// upon failure.
+  Error bind(unsigned Port, const char *HostInterface = "0.0.0.0");
+
+  /// Attempts to assign any available port and interface, returning either the
+  /// port number or an Error upon failure.
+  Expected<unsigned> bind(const char *HostInterface = "0.0.0.0");
+
+  /// Attempts to listen for requests on the bound port. Returns an Error if
+  /// called before binding a port.
+  Error listen();
+
+  /// If the server is listening, stop and unbind the socket.
+  void stop();
+};
+} // end namespace llvm
+
+#endif // LLVM_SUPPORT_HTTP_SERVER_H

llvm/lib/Debuginfod/CMakeLists.txt

 # Link LibCURL if the user wants it
 if (LLVM_ENABLE_CURL)
   set(imported_libs CURL::libcurl)
 endif()
 
+# Link cpp-httplib if the user wants it
+if (LLVM_ENABLE_HTTPLIB)
+  set(imported_libs ${imported_libs} httplib::httplib)
+endif()
+
 # Note: This isn't a component, since that could potentially add a libcurl
 # dependency to libLLVM.
 add_llvm_library(LLVMDebuginfod
   Debuginfod.cpp
   DIFetcher.cpp
   HTTPClient.cpp
+  HTTPServer.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/Debuginfod
 
   LINK_LIBS
   ${imported_libs}
 
   LINK_COMPONENTS
   Support
   Symbolize
   )

llvm/lib/Debuginfod/HTTPServer.cpp

+//===-- llvm/Debuginfod/HTTPServer.cpp - HTTP server library -----*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+///
+/// This file defines the methods of the HTTPServer class and the streamFile
+/// function.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Debuginfod/HTTPServer.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Regex.h"
+
+#ifdef LLVM_ENABLE_HTTPLIB
+#include "httplib.h"
+#endif
+
+using namespace llvm;
+
+bool llvm::streamFile(HTTPServerRequest &Request, StringRef FilePath) {
+  Expected<sys::fs::file_t> FDOrErr = sys::fs::openNativeFileForRead(FilePath);
+  if (Error Err = FDOrErr.takeError()) {
+    consumeError(std::move(Err));
+    Request.setResponse({404u, "text/plain", "Could not open file to read.\n"});
+    return false;
+  }
+  ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
+      MemoryBuffer::getOpenFile(*FDOrErr, FilePath,
+                                /*FileSize=*/-1,
+                                /*RequiresNullTerminator=*/false);
+  sys::fs::closeFile(*FDOrErr);
+  if (Error Err = errorCodeToError(MBOrErr.getError())) {
+    consumeError(std::move(Err));
+    Request.setResponse({404u, "text/plain", "Could not memory-map file.\n"});
+    return false;
+  }
+  // Lambdas are copied on conversion to to std::function, preventing use of
+  // smart pointers.
+  MemoryBuffer *MB = MBOrErr->release();
+  Request.setResponse({200u, "application/octet-stream", MB->getBufferSize(),
+                       [=](size_t Offset, size_t Length) -> StringRef {
+                         return MB->getBuffer().substr(Offset, Length);
+                       },
+                       [=](bool Success) { delete MB; }});
+  return true;
+}
+
+#ifdef LLVM_ENABLE_HTTPLIB
+
+bool HTTPServer::isAvailable() { return true; }
+
+HTTPServer::HTTPServer() { Server = std::make_unique<httplib::Server>(); }
+
+HTTPServer::~HTTPServer() { stop(); }
+
+static void expandUrlPathMatches(const std::smatch &Matches,
+                                 HTTPServerRequest &Request) {
+  bool UrlPathSet = false;
+  for (const auto &it : Matches) {
+    if (UrlPathSet)
+      Request.UrlPathMatches.push_back(it);
+    else {
+      Request.UrlPath = it;
+      UrlPathSet = true;
+    }
+  }
+}
+
+HTTPServerRequest::HTTPServerRequest(const httplib::Request &HTTPLibRequest,
+                                     httplib::Response &HTTPLibResponse)
+    : HTTPLibResponse(HTTPLibResponse) {
+  expandUrlPathMatches(HTTPLibRequest.matches, *this);
+}
+
+void HTTPServerRequest::setResponse(HTTPResponse Response) {
+  HTTPLibResponse.set_content(Response.Body.begin(), Response.Body.size(),
+                              Response.ContentType);
+  HTTPLibResponse.status = Response.Code;
+}
+
+void HTTPServerRequest::setResponse(StreamingHTTPResponse Response) {
+  HTTPLibResponse.set_content_provider(
+      Response.ContentLength, Response.ContentType,
+      [=](size_t Offset, size_t Length, httplib::DataSink &Sink) {
+        if (Offset < Response.ContentLength) {
+          StringRef Chunk = Response.Provider(Offset, Length);
+          Sink.write(Chunk.begin(), Chunk.size());
+        }
+        return true;
+      },
+      [=](bool Success) { Response.CompletionHandler(Success); });
+
+  HTTPLibResponse.status = Response.Code;
+}
+
+Error HTTPServer::get(StringRef UrlPathPattern, HTTPRequestHandler Handler) {
+  std::string ErrorMessage;
+  if (!Regex(UrlPathPattern).isValid(ErrorMessage))
+    return createStringError(errc::argument_out_of_domain, ErrorMessage);
+  Server->Get(std::string(UrlPathPattern),
+              [Handler](const httplib::Request &HTTPLibRequest,
+                        httplib::Response &HTTPLibResponse) {
+                HTTPServerRequest Request(HTTPLibRequest, HTTPLibResponse);
+                Handler(Request);
+              });
+  return Error::success();
+}
+
+Error HTTPServer::bind(unsigned ListenPort, const char *HostInterface) {
+  if (!Server->bind_to_port(HostInterface, ListenPort))
+    return createStringError(errc::io_error,
+                             "Could not assign requested address.");
+  Port = ListenPort;
+  return Error::success();
+}
+
+Expected<unsigned> HTTPServer::bind(const char *HostInterface) {
+  int ListenPort = Server->bind_to_any_port(HostInterface);
+  if (ListenPort < 0)
+    return createStringError(errc::io_error,
+                             "Could not assign any port on requested address.");
+  return Port = ListenPort;
+}
+
+Error HTTPServer::listen() {
+  if (!Port)
+    return createStringError(errc::io_error,
+                             "Cannot listen without first binding to a port.");
+  if (!Server->listen_after_bind())
+    return createStringError(
+        errc::io_error,
+        "An unknown error occurred when cpp-httplib attempted to listen.");
+  return Error::success();
+}
+
+void HTTPServer::stop() {
+  Server->stop();
+  Port = 0;
+}
+
+#else
+
+// TODO: Implement barebones standalone HTTP server implementation.
+bool HTTPServer::isAvailable() { return false; }
+
+HTTPServer::HTTPServer() = default;
+
+HTTPServer::~HTTPServer() = default;
+
+void HTTPServerRequest::setResponse(HTTPResponse Response) {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+void HTTPServerRequest::setResponse(StreamingHTTPResponse Response) {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+Error HTTPServer::get(StringRef UrlPathPattern, HTTPRequestHandler Handler) {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+Error HTTPServer::bind(unsigned ListenPort, const char *HostInterface) {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+Expected<unsigned> HTTPServer::bind(const char *HostInterface) {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+Error HTTPServer::listen() {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+void HTTPServer::stop() {
+  llvm_unreachable("No HTTP server implementation available");
+}
+
+#endif // LLVM_ENABLE_HTTPLIB

llvm/unittests/Debuginfod/CMakeLists.txt

 add_llvm_unittest(DebuginfodTests
+  HTTPServerTests.cpp
   DebuginfodTests.cpp
   )
 
 target_link_libraries(DebuginfodTests PRIVATE
   LLVMDebuginfod
   LLVMTestingSupport
   )

llvm/unittests/Debuginfod/HTTPServerTests.cpp

+//===-- llvm/unittest/Support/HTTPServer.cpp - unit tests -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Debuginfod/HTTPClient.h"
+#include "llvm/Debuginfod/HTTPServer.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Testing/Support/Error.h"
+#include "gmock/gmock.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+
+#ifdef LLVM_ENABLE_HTTPLIB
+
+TEST(HTTPServer, IsAvailable) { EXPECT_TRUE(HTTPServer::isAvailable()); }
+
+HTTPResponse Response = {200u, "text/plain", "hello, world\n"};
+std::string UrlPathPattern = R"(/(.*))";
+std::string InvalidUrlPathPattern = R"(/(.*)";
+
+HTTPRequestHandler Handler = [](HTTPServerRequest &Request) {
+  Request.setResponse(Response);
+};
+
+HTTPRequestHandler DelayHandler = [](HTTPServerRequest &Request) {
+  std::this_thread::sleep_for(std::chrono::milliseconds(50));
+  Request.setResponse(Response);
+};
+
+HTTPRequestHandler StreamingHandler = [](HTTPServerRequest &Request) {
+  Request.setResponse({200, "text/plain", Response.Body.size(),
+                       [=](size_t Offset, size_t Length) -> StringRef {
+                         return Response.Body.substr(Offset, Length);
+                       }});
+};
+
+TEST(HTTPServer, InvalidUrlPath) {
+  // test that we can bind to any address
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(InvalidUrlPathPattern, Handler),
+                    Failed<StringError>());
+  EXPECT_THAT_EXPECTED(Server.bind(), Succeeded());
+}
+
+TEST(HTTPServer, bind) {
+  // test that we can bind to any address
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, Handler), Succeeded());
+  EXPECT_THAT_EXPECTED(Server.bind(), Succeeded());
+}
+
+TEST(HTTPServer, ListenBeforeBind) {
+  // test that we can bind to any address
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, Handler), Succeeded());
+  EXPECT_THAT_ERROR(Server.listen(), Failed<StringError>());
+}
+
+#ifdef LLVM_ENABLE_CURL
+// Test the client and server against each other.
+
+// Test fixture to initialize and teardown the HTTP client for each
+// client-server test
+class HTTPClientServerTest : public ::testing::Test {
+protected:
+  void SetUp() override { HTTPClient::initialize(); }
+  void TearDown() override { HTTPClient::cleanup(); }
+};
+
+/// A simple handler which writes returned data to a string.
+struct StringHTTPResponseHandler final : public HTTPResponseHandler {
+  std::string ResponseBody = "";
+  /// These callbacks store the body and status code in an HTTPResponseBuffer
+  /// allocated based on Content-Length. The Content-Length header must be
+  /// handled by handleHeaderLine before any calls to handleBodyChunk.
+  Error handleBodyChunk(StringRef BodyChunk) override {
+    ResponseBody = ResponseBody + BodyChunk.str();
+    return Error::success();
+  }
+};
+
+TEST_F(HTTPClientServerTest, Hello) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, Handler), Succeeded());
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Handler.ResponseBody, Response.Body);
+  EXPECT_EQ(Client.responseCode(), Response.Code);
+  Server.stop();
+}
+
+TEST_F(HTTPClientServerTest, LambdaHandlerHello) {
+  HTTPServer Server;
+  HTTPResponse LambdaResponse = {200u, "text/plain",
+                                 "hello, world from a lambda\n"};
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern,
+                               [LambdaResponse](HTTPServerRequest &Request) {
+                                 Request.setResponse(LambdaResponse);
+                               }),
+                    Succeeded());
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Handler.ResponseBody, LambdaResponse.Body);
+  EXPECT_EQ(Client.responseCode(), LambdaResponse.Code);
+  Server.stop();
+}
+
+// Test the streaming response.
+TEST_F(HTTPClientServerTest, StreamingHello) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, StreamingHandler), Succeeded());
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Handler.ResponseBody, Response.Body);
+  EXPECT_EQ(Client.responseCode(), Response.Code);
+  Server.stop();
+}
+
+// Writes a temporary file and streams it back using streamFile.
+HTTPRequestHandler TempFileStreamingHandler = [](HTTPServerRequest Request) {
+  int FD;
+  SmallString<64> TempFilePath;
+  sys::fs::createTemporaryFile("http-stream-file-test", "temp", FD,
+                               TempFilePath);
+  raw_fd_ostream OS(FD, true, /*unbuffered=*/true);
+  OS << Response.Body;
+  OS.close();
+  streamFile(Request, TempFilePath);
+};
+
+// Test streaming back chunks of a file.
+TEST_F(HTTPClientServerTest, StreamingFileResponse) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, TempFileStreamingHandler),
+                    Succeeded());
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Handler.ResponseBody, Response.Body);
+  EXPECT_EQ(Client.responseCode(), Response.Code);
+  Server.stop();
+}
+
+// Deletes the temporary file before streaming it back, should give a 404 not
+// found status code.
+HTTPRequestHandler MissingTempFileStreamingHandler =
+    [](HTTPServerRequest Request) {
+      int FD;
+      SmallString<64> TempFilePath;
+      sys::fs::createTemporaryFile("http-stream-file-test", "temp", FD,
+                                   TempFilePath);
+      raw_fd_ostream OS(FD, true, /*unbuffered=*/true);
+      OS << Response.Body;
+      OS.close();
+      // delete the file
+      sys::fs::remove(TempFilePath);
+      streamFile(Request, TempFilePath);
+    };
+
+// Streaming a missing file should give a 404.
+TEST_F(HTTPClientServerTest, StreamingMissingFileResponse) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, MissingTempFileStreamingHandler),
+                    Succeeded());
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Client.responseCode(), 404u);
+  Server.stop();
+}
+
+TEST_F(HTTPClientServerTest, ClientTimeout) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, DelayHandler), Succeeded());
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPClient Client;
+  // Timeout below 50ms, request should fail
+  Client.setTimeout(std::chrono::milliseconds(40));
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Failed<StringError>());
+  Server.stop();
+}
+
+// Check that Url paths are dispatched to the first matching handler and provide
+// the correct path pattern match components.
+TEST_F(HTTPClientServerTest, PathMatching) {
+  HTTPServer Server;
+
+  EXPECT_THAT_ERROR(
+      Server.get(R"(/abc/(.*)/(.*))",
+                 [&](HTTPServerRequest &Request) {
+                   EXPECT_EQ(Request.UrlPath, "/abc/1/2");
+                   ASSERT_THAT(Request.UrlPathMatches,
+                               testing::ElementsAre("1", "2"));
+                   Request.setResponse({200u, "text/plain", Request.UrlPath});
+                 }),
+      Succeeded());
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern,
+                               [&](HTTPServerRequest &Request) {
+                                 llvm_unreachable(
+                                     "Should not reach this handler");
+                                 Handler(Request);
+                               }),
+                    Succeeded());
+
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port) + "/abc/1/2";
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Handler.ResponseBody, "/abc/1/2");
+  EXPECT_EQ(Client.responseCode(), 200u);
+  Server.stop();
+}
+
+TEST_F(HTTPClientServerTest, FirstPathMatched) {
+  HTTPServer Server;
+
+  EXPECT_THAT_ERROR(
+      Server.get(UrlPathPattern,
+                 [&](HTTPServerRequest Request) { Handler(Request); }),
+      Succeeded());
+
+  EXPECT_THAT_ERROR(
+      Server.get(R"(/abc/(.*)/(.*))",
+                 [&](HTTPServerRequest Request) {
+                   EXPECT_EQ(Request.UrlPathMatches.size(), 2u);
+                   llvm_unreachable("Should not reach this handler");
+                   Request.setResponse({200u, "text/plain", Request.UrlPath});
+                 }),
+      Succeeded());
+
+  Expected<unsigned> PortOrErr = Server.bind();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port) + "/abc/1/2";
+  HTTPRequest Request(Url);
+  StringHTTPResponseHandler Handler;
+  HTTPClient Client;
+  EXPECT_THAT_ERROR(Client.perform(Request, Handler), Succeeded());
+  EXPECT_EQ(Handler.ResponseBody, Response.Body);
+  EXPECT_EQ(Client.responseCode(), Response.Code);
+  Server.stop();
+}
+
+#endif
+
+#else
+
+TEST(HTTPServer, IsAvailable) { EXPECT_FALSE(HTTPServer::isAvailable()); }
+
+#endif // LLVM_ENABLE_HTTPLIB