[Debuginfod] DEBUGINFOD_HEADERS_FILE environment variable

[mysterymath]

Summary

This change adds a DEBUGINFOD_HEADERS_FILE environment variable provides
a file containing HTTP headers to attach to outgoing HTTP requests, one
per line. This allows a file permissioned with OS access control
mechanisms to supply bearer credentials for Debuginfod requests.

This matches the mechanism recently added to elfutils' libdebuginfod.

[phosek]

llvm/test/tools/llvm-debuginfod-find/Inputs/capture_req.py

16

Wouldn't this fail if something was already running on a port 8000 (which is pretty common)? Is it possible to instead choose first available?

[mysterymath]

Let OS automatically assign port.

[mysterymath]

llvm/test/tools/llvm-debuginfod-find/Inputs/capture_req.py

16

Ah, good call. Done.

[mysterymath]

Note that this flag just landed in the elfutils libdebuginfod client.

[MaskRay]

I see a 2022-10-28 elfutils commit adding DEBUGINFOD_HEADERS_FILE. The message should mention that the feature matches debuginfod.
I'd use llvm-debuginfod instead of Debuginfod as the tag to make it clear the one in llvm-project is an alternative implementation.

Also, I'd remove the trailing dot in the subject line.

[MaskRay]

Thanks. Generally LG

llvm/lib/Debuginfod/Debuginfod.cpp

195

For StringExtras STLExtras, llvm:: is common, llvm::split

196

Should there be a warning for non-header lines? If comments are useful, we can allow # comment markers.

llvm/test/tools/llvm-debuginfod-find/Inputs/capture_req.py

21

Single quotes are more common

llvm/test/tools/llvm-debuginfod-find/headers.test

14

The check can be made stricter

HEADERS:      Accept: */*
HEADERS-NEXT: A: B
HEADERS-NEXT: ...
HEADERS-NOT:  {{.}}

Then NO-HEADERS-NOT can be removed

[mysterymath]

This comment has been deleted.

llvm/lib/Debuginfod/Debuginfod.cpp

196

Added a warning; this should help keep the semantics of this file very simple.

llvm/test/tools/llvm-debuginfod-find/headers.test

14

Mostly done, but I think we still do want a check of some kind in the NO-HEADERS case.

[mysterymath]

In D136303#3910194, @MaskRay wrote:

I see a 2022-10-28 elfutils commit adding DEBUGINFOD_HEADERS_FILE. The message should mention that the feature matches debuginfod.

Done.

I'd use llvm-debuginfod instead of Debuginfod as the tag to make it clear the one in llvm-project is an alternative implementation.

I would, but there's already a llvm-debuginfod utility in LLVM that makes this ambiguous. The library is named Debuginfod, so that's what I've been using to tag CLs against the client library.

Also, I'd remove the trailing dot in the subject line.

Done.

[mysterymath]

Address review comments.

[mysterymath]

Fix formatting.

[MaskRay]

llvm/test/tools/llvm-debuginfod-find/headers.test

18

Align HEADERS's value when HEADER-NEXT is present.

26

Add -NEXT if applicable

[mysterymath]

Add -NEXT to tests.

llvm/include/llvm/Debuginfod/HTTPClient.h

 
 namespace llvm {
 
 enum class HTTPMethod { GET };
 
 /// A stateless description of an outbound HTTP request.
 struct HTTPRequest {
   SmallString<128> Url;
+  SmallVector<std::string, 0> Headers;
   HTTPMethod Method = HTTPMethod::GET;
   bool FollowRedirects = true;
   HTTPRequest(StringRef Url);
 };
 
 bool operator==(const HTTPRequest &A, const HTTPRequest &B);
 
 /// A handler for state updates occurring while an HTTPRequest is performed.

llvm/lib/Debuginfod/Debuginfod.cpp

 /// If a source file is to be fetched, its absolute path must be specified in
 /// the Description argument to fetchInfo. The DebuginfodLogEntry,
 /// DebuginfodLog, and DebuginfodCollection are used by the DebuginfodServer to
 /// scan the local filesystem for binaries and serve the debuginfod protocol.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Debuginfod/Debuginfod.h"
+#include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/BinaryFormat/Magic.h"
 #include "llvm/DebugInfo/DWARF/DWARFContext.h"
 #include "llvm/DebugInfo/Symbolize/Symbolize.h"
 #include "llvm/Debuginfod/HTTPClient.h"
 #include "llvm/Object/BuildID.h"
 #include "llvm/Object/ELFObjectFile.h"
 #include "llvm/Support/CachePruning.h"
 #include "llvm/Support/Caching.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileUtilities.h"
+#include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ThreadPool.h"
 #include "llvm/Support/xxhash.h"
 
 #include <atomic>
 #include <thread>
 
 namespace llvm {
if (!FileStream) {
     if (!FileStreamOrError)
       return FileStreamOrError.takeError();
     FileStream = std::move(*FileStreamOrError);
   }
   *FileStream->OS << BodyChunk;
   return Error::success();
 }
 
+// An over-accepting simplification of the HTTP RFC 7230 spec.
+static bool isHeader(StringRef S) {
+  StringRef Name;
+  StringRef Value;
+  std::tie(Name, Value) = S.split(':');
+  if (Name.empty() || Value.empty())
+    return false;
+  return all_of(Name, [](char C) { return llvm::isPrint(C) && C != ' '; }) &&
+         all_of(Value, [](char C) { return llvm::isPrint(C) || C == '\t'; });
+}
+
+static SmallVector<std::string, 0> getHeaders() {
+  const char *Filename = getenv("DEBUGINFOD_HEADERS_FILE");
+  if (!Filename)
+    return {};
+  ErrorOr<std::unique_ptr<MemoryBuffer>> HeadersFile =
+      MemoryBuffer::getFile(Filename, /*IsText=*/true);
+  if (!HeadersFile)
+    return {};
+
+  SmallVector<std::string, 0> Headers;
+  uint64_t LineNumber = 0;
+  for (StringRef Line : llvm::split((*HeadersFile)->getBuffer(), '\n')) {
+    LineNumber++;
+    if (!isHeader(Line)) {
+      if (!all_of(Line, llvm::isSpace))
+        WithColor::warning()
+            << "could not parse debuginfod header: " << Filename << ':'
+            << LineNumber << '\n';
+      continue;
+    }
+    if (Line.back() == '\r')
+      Line = Line.drop_back();
+    Headers.emplace_back(Line);
+  }
+  return Headers;
+}
+
 Expected<std::string> getCachedOrDownloadArtifact(
     StringRef UniqueKey, StringRef UrlPath, StringRef CacheDirectoryPath,
     ArrayRef<StringRef> DebuginfodUrls, std::chrono::milliseconds Timeout) {
   SmallString<64> AbsCachedArtifactPath;
   sys::path::append(AbsCachedArtifactPath, CacheDirectoryPath,
                     "llvmcache-" + UniqueKey);
 
   Expected<FileCache> CacheOrErr =
for (StringRef ServerUrl : DebuginfodUrls) {
     SmallString<64> ArtifactUrl;
     sys::path::append(ArtifactUrl, sys::path::Style::posix, ServerUrl, UrlPath);
 
     // Perform the HTTP request and if successful, write the response body to
     // the cache.
     StreamedHTTPResponseHandler Handler([&]() { return CacheAddStream(Task); },
                                         Client);
     HTTPRequest Request(ArtifactUrl);
+    Request.Headers = getHeaders();
     Error Err = Client.perform(Request, Handler);
     if (Err)
       return std::move(Err);
 
     if (Client.responseCode() != 200)
       continue;
 
     // Return the path to the artifact on disk.

llvm/lib/Debuginfod/HTTPClient.cpp

Error HTTPClient::perform(const HTTPRequest &Request,
   if (Request.Method != HTTPMethod::GET)
     return createStringError(errc::invalid_argument,
                              "Unsupported CURL request method.");
 
   SmallString<128> Url = Request.Url;
   curl_easy_setopt(Curl, CURLOPT_URL, Url.c_str());
   curl_easy_setopt(Curl, CURLOPT_FOLLOWLOCATION, Request.FollowRedirects);
 
+  curl_slist *Headers = nullptr;
+  for (const std::string &Header : Request.Headers)
+    Headers = curl_slist_append(Headers, Header.c_str());
+  curl_easy_setopt(Curl, CURLOPT_HTTPHEADER, Headers);
+
   CurlHTTPRequest CurlRequest(Handler);
   curl_easy_setopt(Curl, CURLOPT_WRITEDATA, &CurlRequest);
   CURLcode CurlRes = curl_easy_perform(Curl);
+  curl_slist_free_all(Headers);
   if (CurlRes != CURLE_OK)
     return joinErrors(std::move(CurlRequest.ErrorState),
                       createStringError(errc::io_error,
                                         "curl_easy_perform() failed: %s\n",
                                         curl_easy_strerror(CurlRes)));
   return std::move(CurlRequest.ErrorState);
 }
 

llvm/test/tools/llvm-debuginfod-find/Inputs/capture_req.py

+import http.server
+import os
+import subprocess
+import sys
+import threading
+
+class TrivialHandler(http.server.BaseHTTPRequestHandler):
+  def do_GET(self):
+    self.send_response(501)
+
+  def log_request(self, *args, **kwargs):
+    print(self.requestline)
+    print(self.headers)
+
+httpd = http.server.HTTPServer(('', 0),  TrivialHandler)
+port = httpd.socket.getsockname()[1]
+
+try:
+  t = threading.Thread(target=httpd.serve_forever).start()
+  os.environ['DEBUGINFOD_URLS'] =f'http://localhost:{port}'
+  subprocess.run(sys.argv[1:], capture_output = True)
+finally:
+  httpd.shutdown()

llvm/test/tools/llvm-debuginfod-find/Inputs/headers

+
+
+A:
+:A
+:
+A :B
+
+A:B
+C:  D
+E:F  
+hi!$:   j     k
+

llvm/test/tools/llvm-debuginfod-find/headers.test

+REQUIRES: curl
+
+RUN: %python %S/Inputs/capture_req.py llvm-debuginfod-find --debuginfo 0 \
+RUN:   | FileCheck --check-prefix NO-HEADERS %s
+RUN: DEBUGINFOD_HEADERS_FILE=bad %python %S/Inputs/capture_req.py \
+RUN:   llvm-debuginfod-find --debuginfo 0 \
+RUN:   | FileCheck --check-prefix NO-HEADERS %s
+RUN: DEBUGINFOD_HEADERS_FILE=%S/Inputs/headers %python %S/Inputs/capture_req.py \
+RUN:   llvm-debuginfod-find --debuginfo 0 \
+RUN:   | FileCheck --check-prefix HEADERS %s
+RUN: DEBUGINFOD_HEADERS_FILE=%S/Inputs/headers DEBUGINFOD_URLS=fake not llvm-debuginfod-find --debuginfo 0 2>&1 \
+RUN:   | FileCheck --check-prefix ERR -DHEADER_FILE=%S/Inputs/headers %s
+
+NO-HEADERS: Accept: */*
+NO-HEADERS-NOT: {{.}}
+
+HEADERS:      Accept: */*
+HEADERS-NEXT: A: B
+HEADERS-NEXT: C: D
+HEADERS-NEXT: E: F
+HEADERS-NEXT: hi!$:   j     k
+HEADERS-NOT: {{.}}
+
+ERR:      warning: could not parse debuginfod header: [[HEADER_FILE]]:3
+ERR-NEXT: warning: could not parse debuginfod header: [[HEADER_FILE]]:4
+ERR-NEXT: warning: could not parse debuginfod header: [[HEADER_FILE]]:5
+ERR-NEXT: warning: could not parse debuginfod header: [[HEADER_FILE]]:6