[llvm] [Debuginfod] LLVM debuginfod server.

[noajshu]

Summary

This implements a debuginfod server in llvm using the DebuginfodCollection and DebuginfodServer classes. This is tested with lit tests against the debuginfod-find client.

The server scans 0 or more local directories for artifacts. It serves the debuginfod protocol over HTTP. Only the executable and debuginfo endpoints are supported (no /source endpoint).
The server also uses the debuginfod client as a fallback, so it can hit the local debuginfod cache or federate to other known debuginfod servers.
The client behavior is controllable through the standard environment variables (DEBUGINFOD_URLS, DEBUGINFOD_CACHE_PATH, DEBUGINFOD_TIMEOUT)

The server implements on-demand collection updates as follows:
If the build-id is not found by a local lookup, rescan immediately and look up the build-id again before returning 404. To protect against DoS attacks, do not rescan more frequently than once per N seconds (specified by -m).

Lit tests are provided which test the llvm-debuginfod-find client against the llvm-debuginfod server.

[phosek]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

122

This should be controllable through a command line argument, elfutils' debuginfod supports -t SECONDS and defaults to 300 seconds.

[noajshu]

Add -c concurrency limit and -t scan interval command-line opts. Refactor almost all server logic out of llvm-debuginfod.cpp.

[noajshu]

Remove extra copy of the LLVM_ENABLE_HTTPLIB option

[noajshu]

Remove extraneous #includes from llvm-debuginfod.cpp and suppress unrelated cl options.

[noajshu]

Set llvm-debuginfod to use the hardware concurrency by default or if -c 0 is passed.

[noajshu]

Add end-to-end tests.

[noajshu]

rebase

[pcc]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

11

Looks like this comment should be updated to describe llvm-debuginfod instead of llvm-debuginfod-find.

[pcc]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

70

Could we make the bind interface configurable please? This will allow running a local symbol server without exposing it to other machines on the network.

[noajshu]

Incorporate DebuginfodLog. Fix tests by adding binary files and enabling concurrency. Make threading a requirement for the tests.

[noajshu]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

70

Great idea, this has been added as the -i command line option.

[noajshu]

Add -v flag to enable verbose logging, suppress verbosity by default.

[noajshu]

Updated comments in llvm-debuginfod.cpp.

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

11

Thank you for catching this!

[phosek]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

44

10000s is a long time between scans, elfutils defaults to 300s which I think would be better.

[pcc]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

44

If the server implements the following behavior, do we need a scan interval?

• If the build-id is not found by a local lookup, rescan immediately and look up the build-id again before returning 404.

• To protect against DoS attacks, do not rescan more frequently than once per N seconds (e.g. N = 30).

debuginfod does not have this behavior and it was somewhat of a pain point when using it for local development.

[noajshu]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

44

10000s is a long time between scans, elfutils defaults to 300s which I think would be better.

Thanks, will update this.

If the server implements the following behavior, do we need a scan interval?

Thanks for this suggestion, this would be really nice for local development. I've just updated D114845 (which implements DebuginfodCollection) to support this behavior via updateIfStale(), though we still need to add the option here. It's not much trouble to support both periodic and on-demand update triggers, so perhaps we could allow the when-to-update behavior to be customized by the cl::opts.

44

10000s is a long time between scans, elfutils defaults to 300s which I think would be better.

[mysterymath]

llvm/test/tools/llvm-debuginfod/llvm-debuginfod.test

26–28

It doesn't seem like the python script does anything for these two invocations; the DEBUGINFOD_CACHE_PATH is set manually, and there's nothing to wait for. It'd be clearer just to use llvm-debuginfod-find directly. This should also help remove conditional logic from the Python script.

52

Consider just env=os.environ, since env is otherwise unused.

55

59

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

44

It looks like this hasn't yet been updated as of the most recent patch.

98

Consider adding an llvm_unreachable() and a comment after this for documentation purposes; it was surprising to discover that the structure of the pool means it will never finish, but that there's still a wait call.

[noajshu]

Reduce default directory scan wait period to 300s, incorporate on-demand update using timer.

[mysterymath]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

101

std::chrono::seconds? It looks like the implicit conversion should automatically convert the durations.

[noajshu]

Update the -t delay parameter to require integer delay time in seconds between scans.

[noajshu]

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

101

The goal of this was to allow someone to specify non-integer arguments, which would be rounded to the nearest millisecond. However I think it is more canonical to require an integer, so I've updated the argument and used your suggested conversion. Thanks!

[noajshu]

Rebase against main

[noajshu]

rebase against main

[fche2]

requesting lldb followup: https://github.com/llvm/llvm-project/issues/61547

[noajshu]

In D114846#4206434, @fche2 wrote:

requesting lldb followup: https://github.com/llvm/llvm-project/issues/61547

Hi @fche2 thanks for the ping.

D75750 is a great starting point that looked up source files with elfutils' debuginfod.

Integrating llvm-debuginfod into lldb may be pretty simple as the library comes for free with llvm if built with libcurl.

I am curious if others are already reviving this effort? I could work on it if not.

[mysterymath]

In D114846#4214141, @noajshu wrote:

In D114846#4206434, @fche2 wrote:

requesting lldb followup: https://github.com/llvm/llvm-project/issues/61547

Hi @fche2 thanks for the ping.

D75750 is a great starting point that looked up source files with elfutils' debuginfod.

Integrating llvm-debuginfod into lldb may be pretty simple as the library comes for free with llvm if built with libcurl.

I am curious if others are already reviving this effort? I could work on it if not.

I'm planning to pick this up in the vaguely nearish future (if noone else does), but I don't have a clear ETA for when I'll get around to it.

llvm/test/CMakeLists.txt

 llvm_canonicalize_cmake_booleans(
   BUILD_SHARED_LIBS
   LLVM_HAVE_LIBXAR
   HAVE_OCAMLOPT
   LLVM_ENABLE_DIA_SDK
   LLVM_ENABLE_FFI
   LLVM_ENABLE_THREADS
   LLVM_ENABLE_CURL
+  LLVM_ENABLE_HTTPLIB
   LLVM_ENABLE_ZLIB
   LLVM_ENABLE_LIBXML2
   LLVM_INCLUDE_GO_TESTS
   LLVM_LINK_LLVM_DYLIB
   LLVM_TOOL_LTO_BUILD
   LLVM_USE_INTEL_JITEVENTS
   LLVM_BUILD_EXAMPLES
   LLVM_ENABLE_PLUGINS
set(LLVM_TEST_DEPENDS
           llvm-cfi-verify
           llvm-config
           llvm-cov
           llvm-cvtres
           llvm-cxxdump
           llvm-cxxfilt
           llvm-cxxmap
           llvm-debuginfod-find
+          llvm-debuginfod
           llvm-diff
           llvm-dis
           llvm-dlltool
           dsymutil
           llvm-dwarfdump
           llvm-dwp
           llvm-exegesis
           llvm-extract

llvm/test/lit.cfg.py

tools = [
     ToolSubst('%llvm-bitcode-strip', FindTool('llvm-bitcode-strip')),
     ToolSubst('%split-file', FindTool('split-file')),
 ]
 
 # FIXME: Why do we have both `lli` and `%lli` that do slightly different things?
 tools.extend([
     'dsymutil', 'lli', 'lli-child-target', 'llvm-ar', 'llvm-as',
     'llvm-addr2line', 'llvm-bcanalyzer', 'llvm-bitcode-strip', 'llvm-config',
-    'llvm-cov', 'llvm-cxxdump', 'llvm-cvtres', 'llvm-debuginfod-find',
+    'llvm-cov', 'llvm-cxxdump', 'llvm-cvtres', 'llvm-debuginfod-find', 'llvm-debuginfod',
     'llvm-diff', 'llvm-dis', 'llvm-dwarfdump', 'llvm-dlltool', 'llvm-exegesis',
     'llvm-extract', 'llvm-isel-fuzzer', 'llvm-ifs',
     'llvm-install-name-tool', 'llvm-jitlink', 'llvm-opt-fuzzer', 'llvm-lib',
     'llvm-link', 'llvm-lto', 'llvm-lto2', 'llvm-mc', 'llvm-mca',
     'llvm-modextract', 'llvm-nm', 'llvm-objcopy', 'llvm-objdump', 'llvm-otool',
     'llvm-pdbutil', 'llvm-profdata', 'llvm-profgen', 'llvm-ranlib', 'llvm-rc', 'llvm-readelf',
     'llvm-readobj', 'llvm-remark-size-diff', 'llvm-rtdyld', 'llvm-sim',
     'llvm-size', 'llvm-split', 'llvm-stress', 'llvm-strings', 'llvm-strip',
if config.enable_threads:
     config.available_features.add('thread_support')
 
 if config.have_libxml2:
     config.available_features.add('libxml2')
 
 if config.have_curl:
     config.available_features.add('curl')
 
+if config.have_httplib:
+    config.available_features.add('httplib')
+
 if config.have_opt_viewer_modules:
     config.available_features.add('have_opt_viewer_modules')
 
 if config.expensive_checks:
     config.available_features.add('expensive_checks')
 
 if "MemoryWithOrigins" in config.llvm_use_sanitizer:
     config.available_features.add('use_msan_with_origins')
for filename in os.listdir(dirname):
           if ('-emit-obj' in data or '-filetype=obj' in data) and '64' in config.target_triple:
             config.excludes += [ filename ]
        finally:
           f.close()
 
 if 'aix' in config.target_triple:
     for directory in ('/CodeGen/X86', '/DebugInfo', '/DebugInfo/X86', '/DebugInfo/Generic', '/LTO/X86', '/Linker'):
         exclude_unsupported_files_for_aix(config.test_source_root + directory)
-

llvm/test/lit.site.cfg.py.in

 # Note: ldflags can contain double-quoted paths, so must use single quotes here.
 config.host_ldflags = '@HOST_LDFLAGS@'
 config.llvm_use_intel_jitevents = @LLVM_USE_INTEL_JITEVENTS@
 config.llvm_use_sanitizer = "@LLVM_USE_SANITIZER@"
 config.have_zlib = @LLVM_ENABLE_ZLIB@
 config.have_libxar = @LLVM_HAVE_LIBXAR@
 config.have_libxml2 = @LLVM_ENABLE_LIBXML2@
 config.have_curl = @LLVM_ENABLE_CURL@
+config.have_httplib = @LLVM_ENABLE_HTTPLIB@
 config.have_dia_sdk = @LLVM_ENABLE_DIA_SDK@
 config.enable_ffi = @LLVM_ENABLE_FFI@
 config.build_examples = @LLVM_BUILD_EXAMPLES@
 config.enable_threads = @LLVM_ENABLE_THREADS@
 config.build_shared_libs = @BUILD_SHARED_LIBS@
 config.link_llvm_dylib = @LLVM_LINK_LLVM_DYLIB@
 config.llvm_dylib_version = "@LLVM_VERSION_MAJOR@@LLVM_VERSION_SUFFIX@"
 config.llvm_host_triple = '@LLVM_HOST_TRIPLE@'

llvm/test/tools/llvm-debuginfod/llvm-debuginfod.test

+# REQUIRES: curl, httplib, thread_support
+
+#int main () {
+#  int x = 1;
+#  return x;
+#}
+#
+#Build as : clang -g main.c -o main-debug.exe
+#Then run : cp main-debug.exe main.exe && strip main.exe
+#resulting buildid: 2c39b7557c50162aaeb5a3148c9f76e6e46012e3
+
+# RUN: rm -rf %t
+# RUN: mkdir %t
+# # Query the debuginfod server for artifacts
+# RUN: DEBUGINFOD_CACHE_PATH=%t %python %s --server-cmd 'llvm-debuginfod -v -c 3 %S/Inputs' \
+# RUN:   --tool-cmd 'llvm-debuginfod-find --dump --executable 2c39b7557c50162aaeb5a3148c9f76e6e46012e3' | \
+# RUN:   diff - %S/Inputs/main.exe
+# RUN: DEBUGINFOD_CACHE_PATH=%t %python %s --server-cmd 'llvm-debuginfod -v -c 3 %S/Inputs' \
+# RUN:   --tool-cmd 'llvm-debuginfod-find --dump --debuginfo 2c39b7557c50162aaeb5a3148c9f76e6e46012e3' | \
+# RUN:   diff - %S/Inputs/main-debug.exe
+# Debuginfod server does not yet support source files
+
+# # The artifacts should still be present in the cache without needing to query
+# # the server.
+# RUN: DEBUGINFOD_CACHE_PATH=%t llvm-debuginfod-find --dump \
+# RUN:   --executable 2c39b7557c50162aaeb5a3148c9f76e6e46012e3 | \
+# RUN:   diff - %S/Inputs/main.exe
+# RUN: DEBUGINFOD_CACHE_PATH=%t llvm-debuginfod-find --dump \
+# RUN:   --debuginfo 2c39b7557c50162aaeb5a3148c9f76e6e46012e3 | \
+# RUN:   diff - %S/Inputs/main-debug.exe
+
+
+
+# This script is used to test the debuginfod client within a host tool against
+# the debuginfod server.
+# It first stands up the debuginfod server and then executes the tool.
+# This way the tool can make debuginfod HTTP requests to the debuginfod server.
+import argparse
+import threading
+import subprocess
+import sys
+import os
+import io
+
+# Starts the server and obtains the port number from the first line of stdout.
+# Waits until the server has completed one full directory scan before returning.
+def start_debuginfod_server(server_args):
+    process = subprocess.Popen(
+        server_args,
+        env=os.environ,
+        stdout=subprocess.PIPE)
+    port = -1
+    # Obtain the port.
+    stdout_reader = io.TextIOWrapper(process.stdout, encoding='ascii')
+    stdout_line = stdout_reader.readline()
+    port = int(stdout_line.split()[-1])
+    # Wait until a directory scan is completed.
+    while True:
+        stdout_line = stdout_reader.readline().strip()
+        print(stdout_line, file=sys.stderr)
+        if stdout_line == 'Updated collection':
+            break
+    return (process, port)
+
+# Starts the server with the specified args (if nonempty), then runs the tool
+# with specified args.
+# Sets the DEBUGINFOD_CACHE_PATH env var to point at the given cache_directory.
+# Sets the DEBUGINFOD_URLS env var to point at the local server.
+def test_tool(server_args, tool_args):
+    server_process = None
+    client_process = None
+    port = None
+    server_process, port = start_debuginfod_server(server_args)
+    try:
+        env = os.environ
+        if port is not None:
+            env['DEBUGINFOD_URLS'] = 'http://localhost:%s' % port
+        client_process = subprocess.Popen(
+            tool_args, env=os.environ)
+        client_code = client_process.wait()
+        if client_code != 0:
+            print('nontrivial client return code %s' % client_code, file=sys.stderr)
+            return 1
+        if server_process is not None:
+            server_process.terminate()
+            server_code = server_process.wait()
+            if server_code != -15:
+                print('nontrivial server return code %s' % server_code, file=sys.stderr)
+                return 1
+
+    finally:
+        if server_process is not None:
+            server_process.terminate()
+        if client_process is not None:
+            client_process.terminate()
+    return 0
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--server-cmd', default='', help='Command to start the server. If not present, no server is started.', type=str)
+    parser.add_argument('--tool-cmd', required=True, type=str)
+    args = parser.parse_args()
+    result = test_tool(args.server_cmd.split(),
+        args.tool_cmd.split())
+    sys.exit(result)
+
+if __name__ == '__main__':
+    main()

llvm/tools/llvm-debuginfod/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Debuginfod
+  Support
+  )
+add_llvm_tool(llvm-debuginfod
+  llvm-debuginfod.cpp
+  )
+if(LLVM_INSTALL_BINUTILS_SYMLINKS)
+  add_llvm_tool_symlink(debuginfod llvm-debuginfod)
+endif()

llvm/tools/llvm-debuginfod/llvm-debuginfod.cpp

+//===-- llvm-debuginfod.cpp - federating debuginfod server ----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains the llvm-debuginfod tool, which serves the debuginfod
+/// protocol over HTTP. The tool periodically scans zero or more filesystem
+/// directories for ELF binaries to serve, and federates requests for unknown
+/// build IDs to the debuginfod servers set in the DEBUGINFOD_URLS environment
+/// variable.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Debuginfod/Debuginfod.h"
+#include "llvm/Debuginfod/HTTPClient.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/ThreadPool.h"
+
+using namespace llvm;
+
+cl::OptionCategory DebuginfodCategory("llvm-debuginfod Options");
+
+static cl::list<std::string> ScanPaths(cl::Positional,
+                                       cl::desc("<Directories to scan>"),
+                                       cl::cat(DebuginfodCategory));
+
+static cl::opt<unsigned>
+    Port("p", cl::init(0),
+         cl::desc("Port to listen on. Set to 0 to bind to any available port."),
+         cl::cat(DebuginfodCategory));
+
+static cl::opt<std::string>
+    HostInterface("i", cl::init("0.0.0.0"),
+                  cl::desc("Host interface to bind to."),
+                  cl::cat(DebuginfodCategory));
+
+static cl::opt<double>
+    ScanInterval("t", cl::init(300),
+                 cl::desc("Number of seconds to wait between subsequent "
+                          "automated scans of the filesystem."),
+                 cl::cat(DebuginfodCategory));
+
+static cl::opt<double> MinInterval(
+    "m", cl::init(10),
+    cl::desc(
+        "Minimum number of seconds to wait before an on-demand update can be "
+        "triggered by a request for a buildid which is not in the collection."),
+    cl::cat(DebuginfodCategory));
+
+static cl::opt<size_t>
+    MaxConcurrency("c", cl::init(0),
+                   cl::desc("Maximum number of files to scan concurrently. If "
+                            "0, use the hardware concurrency."),
+                   cl::cat(DebuginfodCategory));
+
+static cl::opt<bool> VerboseLogging("v", cl::init(false),
+                                    cl::desc("Enable verbose logging."),
+                                    cl::cat(DebuginfodCategory));
+
+ExitOnError ExitOnErr;
+
+int main(int argc, char **argv) {
+  InitLLVM X(argc, argv);
+  HTTPClient::initialize();
+  cl::HideUnrelatedOptions({&DebuginfodCategory});
+  cl::ParseCommandLineOptions(argc, argv);
+
+  SmallVector<StringRef, 1> Paths;
+  for (const std::string &Path : ScanPaths)
+    Paths.push_back(Path);
+
+  ThreadPool Pool(hardware_concurrency(MaxConcurrency));
+  DebuginfodLog Log;
+  DebuginfodCollection Collection(Paths, Log, Pool, MinInterval);
+  DebuginfodServer Server(Log, Collection);
+
+  if (!Port)
+    Port = ExitOnErr(Server.Server.bind(HostInterface.c_str()));
+  else
+    ExitOnErr(Server.Server.bind(Port, HostInterface.c_str()));
+
+  Log.push("Listening on port " + Twine(Port).str());
+
+  Pool.async([&]() { ExitOnErr(Server.Server.listen()); });
+  Pool.async([&]() {
+    while (1) {
+      DebuginfodLogEntry Entry = Log.pop();
+      if (VerboseLogging) {
+        outs() << Entry.Message << "\n";
+        outs().flush();
+      }
+    }
+  });
+  if (Paths.size())
+    ExitOnErr(Collection.updateForever(
+        std::chrono::milliseconds(static_cast<int>(ScanInterval * 1000))));
+  Pool.wait();
+  llvm_unreachable("The ThreadPool should never finish running its tasks.");
+}