[llvm] [Debuginfod] Add HTTP Server to Debuginfod library.

[noajshu]

Summary

This provides a minimal HTTP server interface and an implementation wrapping cpp-httplib in the Debuginfod library. If the Curl HTTP client is available (D112753) the server is tested by pinging it with the client.

[phosek]

llvm/include/llvm/Support/HTTPServer.h

2

This file is missing LLVM header.

llvm/lib/Support/HTTPServer.cpp

2

This file is missing LLVM header.

20

I wonder if we should perhaps move this to a separate method (like get) and take the path as an argument so we can potentially support multiple handlers.

24

Having both Response and Resp is confusing, can we use a different name here?

58

Calling listen on an unbound server should probably return an error rather than success.

[noajshu]

Incorporate feedback, refactor, add unit tests, and rebase against D113218.

[noajshu]

llvm/lib/Support/HTTPServer.cpp

20

My initial thought was to support multiple "handlers" using one big handler that would parse the URL and dispatch to the appropriate sub-handler. However it does seem more user-friendly to expose the same interface as cpp-httplib, letting the HTTPServer deal with the url parsing.

[noajshu]

Implement streaming responses and file streaming helper; add unit tests of client timeouts and streaming string / file responses.

[dblaikie]

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

[phosek]

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

[dblaikie]

In D114415#3159988, @phosek wrote:

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

Oh, fair enough - just checking it wasn't only being implemented for testing. Would the python script from the llvm-symbolizer test be adequate for the production/local developer scenarios you have in mind? (I don't mind the C++ too much, but just trying to understand the landscape, tradeoffs, etc)

[noajshu]

In D114415#3159995, @dblaikie wrote:

In D114415#3159988, @phosek wrote:

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

Oh, fair enough - just checking it wasn't only being implemented for testing. Would the python script from the llvm-symbolizer test be adequate for the production/local developer scenarios you have in mind? (I don't mind the C++ too much, but just trying to understand the landscape, tradeoffs, etc)

The python script used in D113717 (which might also get used to test D112759) only takes care of the HTTP static file serving needed for debuginfod. There is another component, which is to search the filesystem for debug binaries and assemble the collection of artifacts to serve. I'll be publishing a diff shortly which implements this functionality using LLVM's object parsing and filesystem utilities. Then we will have all the pieces needed for a basic C++ debuginfod in LLVM.

There are workarounds we could use to get a simple debuginfod server working without this LLVM HTTP server. E.g., we could use CGI and FastCGI, or we could create symlinks / copies to the discovered debug data in a single static file serving path. These workarounds came with trade offs, and there are other tools in LLVM that could take advantage of a cross-platform HTTP server such as bisectd (D113030). This is what led to the goal of getting an HTTP server in LLVM's supporting libraries. (A side benefit is that we can now thoroughly unit test the HTTP client by letting it communicate with the server, but this isn't the motivation :) .)

[dblaikie]

In D114415#3162737, @noajshu wrote:

In D114415#3159995, @dblaikie wrote:

In D114415#3159988, @phosek wrote:

In D114415#3159905, @dblaikie wrote:

What's the ultimate use intended for this? For testing the debuginfod client functionality (the llvm-symbolizer functionality is tested with a smaller(?) python http server - perhaps that could be used more & we could avoid having this C++ HTTP server implementation?)?

It's for the server part of the LLVM debuginfod implementation. elfutils debuginfod has two parts: (1) debuginfod-find client and the corresponding library that could be integrated into other tools and (2) debuginfod which is a small daemon that periodically scans a set of directories, indexes any debugging information it finds and serves it over the builtin HTTP server using the debuginfod protocol. We have been focusing on #1 so far but we would also like to implement #2 which is really important for local development. https://groups.google.com/g/llvm-dev/c/jFdq0qYtKqM/m/1dLcYUGBBAAJ has more details.

Oh, fair enough - just checking it wasn't only being implemented for testing. Would the python script from the llvm-symbolizer test be adequate for the production/local developer scenarios you have in mind? (I don't mind the C++ too much, but just trying to understand the landscape, tradeoffs, etc)

The python script used in D113717 (which might also get used to test D112759) only takes care of the HTTP static file serving needed for debuginfod. There is another component, which is to search the filesystem for debug binaries and assemble the collection of artifacts to serve. I'll be publishing a diff shortly which implements this functionality using LLVM's object parsing and filesystem utilities. Then we will have all the pieces needed for a basic C++ debuginfod in LLVM.

There are workarounds we could use to get a simple debuginfod server working without this LLVM HTTP server. E.g., we could use CGI and FastCGI, or we could create symlinks / copies to the discovered debug data in a single static file serving path. These workarounds came with trade offs, and there are other tools in LLVM that could take advantage of a cross-platform HTTP server such as bisectd (D113030). This is what led to the goal of getting an HTTP server in LLVM's supporting libraries. (A side benefit is that we can now thoroughly unit test the HTTP client by letting it communicate with the server, but this isn't the motivation :) .)

Fair enough - thanks for the context!

llvm/include/llvm/Support/HTTPServer.h

29–31

Should the first element instead be a separate member, since it seems it's not like the rest of the elements/will be handled differently?

llvm/lib/Support/HTTPServer.cpp

58

Could this parameter (the last lambda, the "CompletionHandler" member in StreamingHTTPResponse) be move-only (non-copyable) in that case it could capture-by-move the std::unique_ptr<MemoryBuffer> and be a bit more robust in terms of memory management?

67

Perhaps HTTPRequestHandler should be an llvm::function_ref since it doesn't appear to outlive this call, if I'm understanding the code correctly?

94

should this be const auto&? (I'm not sure how big/expensive to copy these objects are, perhaps it's suitable to copy them)

101–102

Looks like this could use StringRef instead of std::string, to avoid copying the contents? Or are there cases where a provider might want to return content by value, rather than from some underlying/longer-lived storage? (worth providing something that could be optimal for both cases somehow - like providing an optional std::string out parameter that the provider could populate if it doesn't have its own long-lived backing store?)

125–130

If the underlying API requires a c-style string, maybe simpler to expose that up through the caller layers? Rather than allocating a buffer to create a null terminated string when the caller might already have one to pass down anyway? (alternatively, I guess, if you use Twine as the parameter type rather than StringRef, then at least in the cases where the Twine does refer to a null terminated string it won't have to copy the string again - currently since the HostInterface is StringRef, it'd always have to copy into the buffer and append a null terminator)

142–144

Follow-up work to use httplib set_error_handler to improve these error messages, perhaps?

158–162

This looks out of date, presumably it doesn't compile? (HTTPServer doesn't declare any ctors, so I guess this would fail to compile)

llvm/unittests/Support/HTTPServer.cpp

73

probably make this by-value?

77

Is some corresponding de-initialize/teardown required/desirable? (perhaps some scoped initialize/de-initialize) or put the initialize in a global ctor (or test fixture (SetUp? Whatever the one is that's called before any test (but not before every test individually)) in the test file if there's no expectation/desire to initialize/teardown around each test?

82

The MemoryBuffer:: probably isn't needed here, I'd have thought?

184–189

timing tests like this can be sensitive to machine load - perhaps setting the acceptable timeout much higher than the actual timeout would be good to increase test stability? (eg: the 40 is probably fine - the actual 50ms delay shouldn't ever be shorter than requested, and so it probably shouldn't tickle that test (though it's possible - if the test code were delayed in reaching the timeout call while the delay was already running, it could appear as though the delay was shorter than expected) - but changing the 60 up to something well above the threshold might be good).

Though also, it's not your job to test that HTTPClient implements timeouts correctly, only that you pass them down through the API - so if there's a more robust way to test that property without actually having to run real timeouts (or at least not testing them so precisely) it might be helpful.

206

I think there's probably a matcher that matches all the elements in a more terse format than having to check each separately? ( https://stackoverflow.com/q/1460703 )

215–217

Prefer llvm_unreachable over assert(false)

[noajshu]

llvm/lib/Support/HTTPServer.cpp

58

That's a great idea! Cpp-httplib converts the content provider to a ContentProvider which is a std::function. This ends up copying the lambda.

I was thinking we might refactor the interface, so instead of returning the response (or response provider) you can interact with a "HTTPServerRequest" object similar to how cpp-httplib's own API works.
Their API is more flexible. For example, you can decide at runtime whether you want to do streaming response handling or return a single response string. In the wrapper implementation here you have to either return a StreamingHTTPResponse or an HTTPResponse, fixed at compile time.

Even if we mirror the httplib interface, the problem remains that we cannot pass a non-copyable lambda directly to httplib's set_content_provider. I'm not sure if there is a nice way around this problem.

67

Ah, I think the nomenclature is misleading. HTTPServer::get should be called something like HTTPServer::registerGetHandler. So it returns, and then you can later call bind() etc., but the function provided in the argument may be called much later after listen when there is a request to be handled.

llvm/unittests/Support/HTTPServer.cpp

77

There is a discussion about the need for a "manual" global HTTPClient::initialize() here. Briefly, Curl's global initialization is not thread-safe and may load other libs; the loader lock on windows prevents use of a static ctor to initialize. We added HTTPClient::initialize() to InitLLVM for convenience, but we had to remove it when HTTPClient got moved to the Debuginfod lib. So now the HTTPClient::initialize must be called "manually". By the way, there is ongoing discussion within libcurl community on making curl global init thread safe.

On the other hand, we can safely deinitialize with a static destructor as you suggest.
This is a great idea which jhenderson also suggested on D113717, and you can see the implementation here:
https://reviews.llvm.org/D113717#change-LO0j60bMM9xj
So there is no need for the de-initialize / teardown here, pending that diff.

[dblaikie]

llvm/lib/Support/HTTPServer.cpp

58

Ah, oh well. Thanks for the context!

[noajshu]

Refactor HTTPServerRequest interface to be more close to cpphttplib, and address all of @dblaikie 's comments.

[noajshu]

Hi @dblaikie thank you so much for the review! I believe I have addressed all your comments, however they disappeared because I moved the HTTPServer into the Debuginfod library. You can see them on the previous revision.

llvm/lib/Support/HTTPServer.cpp

101–102

I've switched it to take a StringRef -- for the only place this is used currently (streaming chunks of a file) you have a persistent MemoryBuffer which is responsible for the storage.

125–130

It's a good point that no matter whether we use cpp-httplib or something else, ultimately the system call will need a c-style string. I've changed it to a const char* and I could switch to a Twine if it's better.

142–144

Good idea!
It looks like httplib's set_error_handler is used to set the default response content (e.g. an error page) when the user's handler returns an HTTP error code (>400). More detailed error information is available to us by checking errno, but cpp-httplib doesn't check it other than to see if the bind / listen was successful.

In a future TCPSocket implementation for LLVM it would be great to parse the errno better to give more helpful messages to the user.

llvm/unittests/Support/HTTPServer.cpp

82

Ah, sadly it is required, unless you can see a better way around the following.
The Body is a WritableMemoryBuffer, whose override of getBuffer returns an MutableArrayRef<char> (as opposed to the parent's virtual method MemoryBuffer::getBuffer which returns a StringRef). For some reason there is no operator== implemented between MutableArrayRef<char> and std::string. So we have to manually specify use of the parent method

184–189

This is a really good observation. I'm concerned that even with a lot of extra wait time, it might flake under some extreme system load. Let's remove it for now, leaving just the test in which a timeout is expected.
HTTPClient::setTimeout is just directly calling curl_easy_setopt so I think that the libcurl tests should cover it for the case where a (nonzero) timeout does not expire.

[dblaikie]

I think this is mostly OK. (still, might be worth initializing/deinitializing in test fixtures, to isolate the tests a bit more - but probably not a big deal/not the sort of bugs we're likely to see (eg: where one test taints the libcurl state and another fails in weird ways because of that taint))

llvm/unittests/Support/HTTPServer.cpp

82

Seems like addressing the missing comparison would be good as a separate patch.

[noajshu]

Use test fixtures to perform HTTP Client initialize and teardown for the client-server tests.

[noajshu]

@dblaikie Thanks for taking another look and the LGTM! Per your suggestion I've added a test fixture to call the HTTPClient initialization and teardown methods. Thanks for all the comments on this patch.

It seems at first llvm-debuginfod will be the sole user tool. So, my inclination is to hold off on committing this until the llvm-debuginfod server (D114845 and D114846) is ready to land.

[dblaikie]

In D114415#3250033, @noajshu wrote:

@dblaikie Thanks for taking another look and the LGTM! Per your suggestion I've added a test fixture to call the HTTPClient initialization and teardown methods. Thanks for all the comments on this patch.

It seems at first llvm-debuginfod will be the sole user tool. So, my inclination is to hold off on committing this until the llvm-debuginfod server (D114845 and D114846) is ready to land.

Yep, sounds fair.

[noajshu]

Replace caputure-by-reference of request handler with capture-by-copy, and add a unit test to verify temporary handlers can be registered and work correctly.

[noajshu]

Fix typo in unit tests.

[noajshu]

Rebase against main

[noajshu]

Rebase against main

[noajshu]

Fix HTTP Server / Client unit tests that were broken after rebase

llvm/include/llvm/Support/HTTPServer.h

+//===-- llvm/Support/HTTPServer.h - HTTP server library ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains the declarations of the HTTPServer class, the
+/// HTTPServerRequest, HTTPResponse, and StreamingHTTPResponse structs, and the
+/// streamFileResponse function.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_HTTP_SERVER_H
+#define LLVM_SUPPORT_HTTP_SERVER_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Error.h"
+
+#ifdef LLVM_ENABLE_HTTPLIB
+#include "httplib.h"
+#endif
+
+namespace llvm {
+
+struct HTTPServerRequest {
+  /// The first element is the entire url path, and the rest of the elements
+  /// correspond to match groups in the url path matching regex.
+  SmallVector<std::string, 2> UrlPathMatches;
+
+  // TODO bring in HTTP headers
+};
+
+struct HTTPResponse {
+  unsigned Code;
+  std::string ContentType;
+  std::string Body;
+};
+
+typedef std::function<HTTPResponse(HTTPServerRequest)> HTTPRequestHandler;
+
+/// An HTTPContentProvider is called by the HTTPServer to obtain chunks of the
+/// streaming response body. The returned chunk should be located at Offset
+/// bytes and have Length bytes.
+typedef std::function<std::string(size_t /*Offset*/, size_t /*Length*/)>
+    HTTPContentProvider;
+
+/// Wraps the content provider with HTTP Status code and headers.
+struct StreamingHTTPResponse {
+  unsigned Code;
+  std::string ContentType;
+  size_t ContentLength;
+  HTTPContentProvider Provider;
+  /// Called after the response transfer is complete with the success value of
+  /// the transfer.
+  std::function<void(bool)> CompletionHandler = [](bool Success) {};
+};
+
+/// Returns a simple streaming response which streams chunks of the file at
+/// FilePath, if available, and otherwise returns an HTTP 404 error status code.
+StreamingHTTPResponse streamFileResponse(StringRef FilePath);
+
+typedef std::function<StreamingHTTPResponse(HTTPServerRequest)>
+    StreamingHTTPRequestHandler;
+
+/// An HTTP server which can listen on a single TCP/IP port for HTTP requests
+/// and delgate them to the appropriate registered handler.
+class HTTPServer {
+#ifdef LLVM_ENABLE_HTTPLIB
+  httplib::Server Server;
+  unsigned Port = 0;
+#endif
+public:
+  ~HTTPServer();
+
+  /// Returns true only if LLVM has been compiled with a working HTTPServer.
+  static bool isAvailable();
+
+  /// Registers a URL pattern routing rule. When the server is listening, each
+  /// request is dispatched to the first registered handler whose UrlPathPattern
+  /// matches the UrlPath.
+  Error get(StringRef UrlPathPattern, HTTPRequestHandler Handler);
+  Error get(StringRef UrlPathPattern, StreamingHTTPRequestHandler Handler);
+
+  /// Attempts to assign the requested port and interface, returning an Error
+  /// upon failure.
+  Error bind(unsigned Port, StringRef HostInterface = "0.0.0.0");
+
+  /// Attempts to assign any available port and interface, returning either the
+  /// port number or an Error upon failure.
+  Expected<unsigned> bindAny(StringRef HostInterface = "0.0.0.0");
+
+  /// Attempts to listen for requests on the bound port. Returns an Error if
+  /// called before binding a port.
+  Error listen();
+
+  /// If the server is listening, stop and unbind the socket.
+  void stop();
+};
+} // end namespace llvm
+
+#endif // LLVM_SUPPORT_HTTP_SERVER_H

llvm/lib/Support/CMakeLists.txt

add_llvm_component_library(LLVMSupport
   FileOutputBuffer.cpp
   FoldingSet.cpp
   FormattedStream.cpp
   FormatVariadic.cpp
   GlobPattern.cpp
   GraphWriter.cpp
   Hashing.cpp
   HTTPClient.cpp
+  HTTPServer.cpp
   InitLLVM.cpp
   InstructionCost.cpp
   IntEqClasses.cpp
   IntervalMap.cpp
   ItaniumManglingCanonicalizer.cpp
   JSON.cpp
   KnownBits.cpp
   LEB128.cpp

llvm/lib/Support/HTTPServer.cpp

+//===-- llvm/Support/HTTPServer.cpp - HTTP server library -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+///
+/// This file defines the methods of the HTTPServer class.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/HTTPServer.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Errc.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Regex.h"
+
+using namespace llvm;
+
+StreamingHTTPResponse llvm::streamFileResponse(StringRef FilePath) {
+  Expected<sys::fs::file_t> FDOrErr = sys::fs::openNativeFileForRead(FilePath);
+  if (Error Err = FDOrErr.takeError()) {
+    std::string ResponseStr = "Could not open file to read.\n";
+    consumeError(std::move(Err));
+    return {404u, "text/plain", ResponseStr.size(),
+            [=](size_t Offset, size_t Length) {
+              return ResponseStr.substr(Offset, Length);
+            }};
+  }
+  ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr =
+      MemoryBuffer::getOpenFile(*FDOrErr, FilePath,
+                                /*FileSize=*/-1,
+                                /*RequiresNullTerminator=*/false);
+  sys::fs::closeFile(*FDOrErr);
+  if (Error Err = errorCodeToError(MBOrErr.getError())) {
+    std::string ResponseStr = "Could not memory-map file.\n";
+    consumeError(std::move(Err));
+    return {404u, "text/plain", ResponseStr.size(),
+            [=](size_t Offset, size_t Length) {
+              return ResponseStr.substr(Offset, Length);
+            }};
+  }
+  // Lambdas are copied on conversion to to std::function, preventing use of
+  // smart pointers.
+  MemoryBuffer *MB = MBOrErr->release();
+  return {200u, "application/octet-stream", MB->getBufferSize(),
+          [=](size_t Offset, size_t Length) -> std::string {
+            std::string Chunk =
+                std::string(MB->getBuffer().substr(Offset, Length));
+            return Chunk;
+          },
+          [=](bool Success) { delete MB; }};
+}
+
+#ifdef LLVM_ENABLE_HTTPLIB
+
+bool HTTPServer::isAvailable() { return true; }
+
+HTTPServer::~HTTPServer() { stop(); }
+
+Error HTTPServer::get(StringRef UrlPathPattern, HTTPRequestHandler Handler) {
+  std::string ErrorMessage;
+  if (!Regex(UrlPathPattern).isValid(ErrorMessage))
+    return createStringError(errc::argument_out_of_domain, ErrorMessage);
+  Server.Get(std::string(UrlPathPattern),
+             [&](const httplib::Request &HTTPLibRequest,
+                 httplib::Response &HTTPLibResponse) {
+               HTTPServerRequest Request;
+               for (auto PathComponent : HTTPLibRequest.matches)
+                 Request.UrlPathMatches.push_back(std::string(PathComponent));
+               HTTPResponse Response = Handler(Request);
+               HTTPLibResponse.set_content(Response.Body.c_str(),
+                                           Response.ContentType.c_str());
+               return HTTPLibResponse.status = Response.Code;
+             });
+  return Error::success();
+}
+
+Error HTTPServer::get(StringRef UrlPathPattern,
+                      StreamingHTTPRequestHandler Handler) {
+  std::string ErrorMessage;
+  if (!Regex(UrlPathPattern).isValid(ErrorMessage))
+    return createStringError(errc::argument_out_of_domain, ErrorMessage);
+  Server.Get(std::string(UrlPathPattern),
+             [&](const httplib::Request &HTTPLibRequest,
+                 httplib::Response &HTTPLibResponse) {
+               HTTPServerRequest Request;
+               for (auto PathComponent : HTTPLibRequest.matches)
+                 Request.UrlPathMatches.push_back(std::string(PathComponent));
+               StreamingHTTPResponse Response = Handler(Request);
+               HTTPLibResponse.set_content_provider(
+                   Response.ContentLength, Response.ContentType.c_str(),
+                   [=](size_t Offset, size_t Length, httplib::DataSink &Sink) {
+                     if (Offset < Response.ContentLength) {
+                       std::string Chunk = Response.Provider(Offset, Length);
+                       Sink.write(Chunk.c_str(), Chunk.size());
+                     }
+                     return true;
+                   },
+                   [=](bool Success) { Response.CompletionHandler(Success); });
+               return HTTPLibResponse.status = Response.Code;
+             });
+  return Error::success();
+}
+
+Error HTTPServer::bind(unsigned ListenPort, StringRef HostInterface) {
+  SmallString<16> HostInterfaceStorage;
+  StringRef S =
+      Twine(HostInterface).toNullTerminatedStringRef(HostInterfaceStorage);
+
+  if (!Server.bind_to_port(S.begin(), ListenPort))
+    return createStringError(errc::io_error,
+                             "Could not assign requested address.");
+
+  Port = ListenPort;
+  return Error::success();
+}
+
+Expected<unsigned> HTTPServer::bindAny(StringRef HostInterface) {
+  SmallString<16> HostInterfaceStorage;
+  StringRef S =
+      Twine(HostInterface).toNullTerminatedStringRef(HostInterfaceStorage);
+
+  int ListenPort = Server.bind_to_any_port(S.begin());
+  if (ListenPort < 0)
+    return createStringError(errc::io_error,
+                             "Could not assign any port on requested address.");
+  return Port = ListenPort;
+}
+
+Error HTTPServer::listen() {
+  if (!Port)
+    return createStringError(errc::io_error,
+                             "Cannot listen without first binding to a port.");
+  if (!Server.listen_after_bind())
+    return createStringError(
+        errc::io_error,
+        "An unknown error occurred when cpp-httplib attempted to listen.");
+  return Error::success();
+}
+
+void HTTPServer::stop() {
+  Server.stop();
+  Port = 0;
+}
+
+#else
+
+// TODO: Implement barebones standalone HTTP server implementation.
+bool HTTPServer::isAvailable() { return false; }
+
+HTTPServer::HTTPServer(
+    std::function<StreamingHTTPResponse(StringRef)> StreamingRequestHandler) {
+  llvm_unreachable(
+      "Attempt to instantiate HTTPServer when no implementation is available.");
+}
+#endif

llvm/unittests/Support/CMakeLists.txt

add_llvm_unittest(SupportTests
   FileOutputBufferTest.cpp
   FileUtilitiesTest.cpp
   FormatVariadicTest.cpp
   FSUniqueIDTest.cpp
   GlobPatternTest.cpp
   HashBuilderTest.cpp
   Host.cpp
   HTTPClient.cpp
+  HTTPServer.cpp
   IndexedAccessorTest.cpp
   InstructionCostTest.cpp
   ItaniumManglingCanonicalizerTest.cpp
   JSONTest.cpp
   KnownBitsTest.cpp
   LEB128Test.cpp
   LinearPolyBaseTest.cpp
   LineIteratorTest.cpp

llvm/unittests/Support/HTTPServer.cpp

+//===-- llvm/unittest/Support/HTTPServer.cpp - unit tests -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/HTTPServer.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/HTTPClient.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Testing/Support/Error.h"
+#include "gtest/gtest.h"
+
+using namespace llvm;
+
+#ifdef LLVM_ENABLE_HTTPLIB
+
+TEST(HTTPServer, IsAvailable) { EXPECT_TRUE(HTTPServer::isAvailable()); }
+
+HTTPResponse Response = {200u, "text/plain", "hello, world\n"};
+std::string UrlPathPattern = R"(/(.*))";
+std::string InvalidUrlPathPattern = R"(/(.*)";
+
+HTTPRequestHandler Handler = [](HTTPServerRequest) -> HTTPResponse {
+  return Response;
+};
+
+HTTPRequestHandler DelayHandler = [](HTTPServerRequest) -> HTTPResponse {
+  std::this_thread::sleep_for(std::chrono::milliseconds(50));
+  return Response;
+};
+
+StreamingHTTPRequestHandler StreamingHandler =
+    [](HTTPServerRequest Request) -> StreamingHTTPResponse {
+  return {200, "text/plain", Response.Body.size(),
+          [=](size_t Offset, size_t Length) -> std::string {
+            return Response.Body.substr(Offset, Length);
+          }};
+};
+
+TEST(HTTPServer, InvalidUrlPath) {
+  // test that we can bind to any address
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(InvalidUrlPathPattern, Handler),
+                    Failed<StringError>());
+  EXPECT_THAT_EXPECTED(Server.bindAny(), Succeeded());
+}
+
+TEST(HTTPServer, BindAny) {
+  // test that we can bind to any address
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, Handler), Succeeded());
+  EXPECT_THAT_EXPECTED(Server.bindAny(), Succeeded());
+}
+
+TEST(HTTPServer, ListenBeforeBind) {
+  // test that we can bind to any address
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, Handler), Succeeded());
+  EXPECT_THAT_ERROR(Server.listen(), Failed<StringError>());
+}
+
+#ifdef LLVM_ENABLE_CURL
+// Test the client and server against each other.
+
+TEST(HTTPClientServer, Hello) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, Handler), Succeeded());
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPClient::initialize();
+  Expected<HTTPResponseBuffer> BufferOrErr = HTTPClient().get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, Response.Code);
+  EXPECT_EQ(Buffer.Body->MemoryBuffer::getBuffer(), Response.Body);
+  Server.stop();
+}
+
+// Test the streaming response.
+TEST(HTTPClientServer, StreamingHello) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, StreamingHandler), Succeeded());
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPClient::initialize();
+  Expected<HTTPResponseBuffer> BufferOrErr = HTTPClient().get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, Response.Code);
+  EXPECT_EQ(Buffer.Body->MemoryBuffer::getBuffer(), Response.Body);
+  Server.stop();
+}
+
+// Writes a temporary file and streams it back using streamFileResponse.
+StreamingHTTPRequestHandler TempFileStreamingHandler =
+    [](HTTPServerRequest Request) -> StreamingHTTPResponse {
+  int FD;
+  SmallString<64> TempFilePath;
+  sys::fs::createTemporaryFile("http-stream-file-test", "temp", FD,
+                               TempFilePath);
+  raw_fd_ostream OF(FD, true, /*unbuffered=*/true);
+  OF << Response.Body;
+  OF.close();
+  return streamFileResponse(TempFilePath);
+};
+
+// Test streaming back chunks of a file.
+TEST(HTTPClientServer, StreamingFileResponse) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, TempFileStreamingHandler),
+                    Succeeded());
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPClient::initialize();
+  Expected<HTTPResponseBuffer> BufferOrErr = HTTPClient().get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, Response.Code);
+  EXPECT_EQ(Buffer.Body->MemoryBuffer::getBuffer(), Response.Body);
+  Server.stop();
+}
+
+// Deletes the temporary file before streaming it back, should give a 404 not
+// found status code.
+StreamingHTTPRequestHandler MissingTempFileStreamingHandler =
+    [](HTTPServerRequest Request) -> StreamingHTTPResponse {
+  int FD;
+  SmallString<64> TempFilePath;
+  sys::fs::createTemporaryFile("http-stream-file-test", "temp", FD,
+                               TempFilePath);
+  raw_fd_ostream OF(FD, true, /*unbuffered=*/true);
+  OF << Response.Body;
+  OF.close();
+  // delete the file
+  sys::fs::remove(TempFilePath);
+  return streamFileResponse(TempFilePath);
+};
+
+// Streaming a missing file should give a 404.
+TEST(HTTPClientServer, StreamingMissingFileResponse) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, MissingTempFileStreamingHandler),
+                    Succeeded());
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPClient::initialize();
+  Expected<HTTPResponseBuffer> BufferOrErr = HTTPClient().get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, 404u);
+  Server.stop();
+}
+
+TEST(HTTPClientServer, ClientTimeout) {
+  HTTPServer Server;
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern, DelayHandler), Succeeded());
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port);
+  HTTPClient::initialize();
+  HTTPClient Client;
+  // Timeout below 50ms, request should fail
+  Client.setTimeout(40);
+  Expected<HTTPResponseBuffer> BufferOrErr = Client.get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Failed<StringError>());
+  // Timeout above 50ms, request should succeed
+  Client.setTimeout(60);
+  BufferOrErr = Client.get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, Response.Code);
+  EXPECT_EQ(Buffer.Body->MemoryBuffer::getBuffer(), Response.Body);
+  Server.stop();
+}
+
+// Check that Url paths are dispatched to the first matching handler and provide
+// the correct path pattern match components.
+TEST(HTTPClientServer, PathMatching) {
+  HTTPServer Server;
+
+  EXPECT_THAT_ERROR(
+      Server.get(R"(/abc/(.*)/(.*))",
+                 [&](HTTPServerRequest Request) -> HTTPResponse {
+                   EXPECT_EQ(Request.UrlPathMatches.size(), 3u);
+                   EXPECT_EQ(Request.UrlPathMatches[0], "/abc/1/2");
+                   EXPECT_EQ(Request.UrlPathMatches[1], "1");
+                   EXPECT_EQ(Request.UrlPathMatches[2], "2");
+                   return {200u, "text/plain", Request.UrlPathMatches[0]};
+                 }),
+      Succeeded());
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern,
+                               [&](HTTPServerRequest Request) -> HTTPResponse {
+                                 assert(false &&
+                                        "Should not reach this handler");
+                                 return Handler(Request);
+                               }),
+                    Succeeded());
+
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port) + "/abc/1/2";
+  HTTPClient::initialize();
+  Expected<HTTPResponseBuffer> BufferOrErr = HTTPClient().get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, Response.Code);
+  EXPECT_EQ(Buffer.Body->MemoryBuffer::getBuffer(), "/abc/1/2");
+  Server.stop();
+}
+
+TEST(HTTPClientServer, FirstPathMatched) {
+  HTTPServer Server;
+
+  EXPECT_THAT_ERROR(Server.get(UrlPathPattern,
+                               [&](HTTPServerRequest Request) -> HTTPResponse {
+                                 return Handler(Request);
+                               }),
+                    Succeeded());
+
+  EXPECT_THAT_ERROR(
+      Server.get(R"(/abc/(.*)/(.*))",
+                 [&](HTTPServerRequest Request) -> HTTPResponse {
+                   EXPECT_EQ(Request.UrlPathMatches.size(), 3u);
+                   assert(false && "Should not reach this handler");
+                   return {200u, "text/plain", Request.UrlPathMatches[0]};
+                 }),
+      Succeeded());
+
+  Expected<unsigned> PortOrErr = Server.bindAny();
+  EXPECT_THAT_EXPECTED(PortOrErr, Succeeded());
+  unsigned &Port = *PortOrErr;
+  ThreadPool Pool(hardware_concurrency(1));
+  Pool.async([&]() { EXPECT_THAT_ERROR(Server.listen(), Succeeded()); });
+  std::string Url = "http://localhost:" + utostr(Port) + "/abc/1/2";
+  HTTPClient::initialize();
+  Expected<HTTPResponseBuffer> BufferOrErr = HTTPClient().get(Url);
+  EXPECT_THAT_EXPECTED(BufferOrErr, Succeeded());
+  HTTPResponseBuffer &Buffer = *BufferOrErr;
+  EXPECT_EQ(Buffer.Code, Response.Code);
+  EXPECT_EQ(Buffer.Body->MemoryBuffer::getBuffer(), Response.Body);
+  Server.stop();
+}
+
+#endif
+
+#endif