[Symbolize][DebugInfoD] Try looking up failed paths as Build IDs.

[mysterymath]

Summary

If a path given to the symbolizer cannot be loaded as a binary file, and
the path string is a 160 character hex, attempt to look up the binary
via DebugInfoD. This will first try cache directories, then an HTTP
lookup if built with libcurl.

This allows Build IDs to be provided to llvm-symbolizer anywhere a path
could be provided: in stdin, via flag, via args, or via environment
variables.

[phosek]

llvm/docs/CommandGuide/llvm-symbolizer.rst

31

Nit: spelling.

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

522

Build ID doesn't have any prescribed length, and linkers typically support a number of hashing algorithms (see https://github.com/llvm/llvm-project/blob/3704abaa166bec865f56f48337a1732eab77dc68/lld/ELF/Driver.cpp#L764) so this isn't going to work.

[mysterymath]

• Bail on small hex strings, not exact size.
• Reworded comment.
• Fixed typo.

[mysterymath]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

522

From the link, it looks like the smallest presently supported would be MD5 and UUID, both at 128 bits.
Given that Build IDs are intended to be globally unique (right?), that seems like a reasonable minimum for this check.

It should be fairly unlikely to have a filename of >= 16 characters that is a valid hex identifier unless it's some kind of ID; we'd be into
aBadBeefCafeBabe territory.

[mysterymath]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

522

Ah, but the --build_id flag can be used to stamp binaries with an arbitrary hex string as part of the build process.

So, the possibilities that immediately occur to me:

1. Keep this as a heuristic check that captures "the usual" build IDs, but then you simply couldn't use this feature in llvm-symbolizer with your own custom build ID schema.
2. Remove this check entirely, and do lookups on anything that could be a build ID. This could issue spurious lookups on paths like "bad", but only in the case where "bad" isn't a valid binary.
3. Provide an in-band mechanism in the syntax of llvm-symbolizer for specifying whether the "path" component of a symoblization request is intended to be a path or a build ID. This would need some further design.

[mcgrathr]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

522

Build IDs can be of any size, and you've already overlooked one of the common hash algorithms (the one lld uses by default) in attempting to enumerate "expected" sizes, which should be a pretty strong sign that an enumeration of expected sizes might not be the best idea. It's for reasons like this that I don't think it's the best CLI here to conflate file names with build ID hex strings. It doesn't seem at all taxing to require a command-line switch to indicate a build ID just as the --obj / --exe / -e switch indicates a file name. For the stdin syntax, I think a new explicit syntax makes sense as well. It already accepts the optional "CODE", "DATA", or "FRAME" prefix before "filename address". So I think it should be fine enough to accept an optional "BUILDID" prefix between the optional "command" prefix and the filename (which given the prefix is interpreted as a build ID). Or something similar.

AFAICT the only environment variable case is using LLVM_SYMBOLIZER_OPTS to pass command-line switches. If you can use -e there today, then using the new build ID switch should be just as easy and require no additional work to support.

[phosek]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

522

I agree with @mcgrathr, having an explicit command-line option and syntax is less error-prone and should be fairly straightforward to implement.

[mysterymath]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

522

SGTM; I'll abandon this PR then, as the proposed mechanisms wouldn't look anything like this.

Adding another prefix to the syntax may take a bit of refactoring (I'll just have to try it.), but I already have a patch more-or-less ready for a command line switch. I'll send PRs that reference this one.

llvm/docs/CommandGuide/llvm-symbolizer.rst

 "CODE" to indicate that the address should be symbolized as data or executable
 code respectively. If neither is specified, "CODE" is assumed. DATA is
 symbolized as address and symbol size rather than line number.
 
 Object files can be specified together with the addresses either on standard
 input or as positional arguments on the command-line, following any "DATA" or
 "CODE" prefix.
 
+Wherever a binary filename is accepted, a Build ID can be provided instead to
+search for the binary. Various cache directories are searched first, followed by
+any DebugInfoD servers configured via the ``DEBUGINFOD_URLS`` environment
+variable. For server lookups to work, the tool needs must have been built with
+``LLVM_ENABLE_CURL`` enabled.
+
+
 :program:`llvm-symbolizer` parses options from the environment variable
 ``LLVM_SYMBOLIZER_OPTS`` after parsing options from the command line.
 ``LLVM_SYMBOLIZER_OPTS`` is primarily useful for supplementing the command-line
 options when :program:`llvm-symbolizer` is invoked by another program or
 runtime.
 
 EXAMPLES
 --------

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

 LLVMSymbolizer::getOrCreateObject(const std::string &Path,
                                   const std::string &ArchName) {
   Binary *Bin;
   auto Pair = BinaryForPath.emplace(Path, OwningBinary<Binary>());
   if (!Pair.second) {
     Bin = Pair.first->second.getBinary();
   } else {
     Expected<OwningBinary<Binary>> BinOrErr = createBinary(Path);
-    if (!BinOrErr)
-      return BinOrErr.takeError();
+    if (BinOrErr) {
       Pair.first->second = std::move(BinOrErr.get());
+    } else {
+      ArrayRef<uint8_t> BuildID;
+
+      // Since this isn't a path to a valid binary, maybe it's a Build ID.
+      std::string Bytes;
+      if (!tryGetFromHex(Path, Bytes))
+        return BinOrErr.takeError();
+      BuildID = ArrayRef<uint8_t>(
+          reinterpret_cast<const uint8_t *>(Bytes.data()), Bytes.size());
+      // There's no fixed size for Build IDs, but if it's fewer than 128 bits,
+      // it's unlikely to be one.
+      if (BuildID.size() < 16)
+        return BinOrErr.takeError();
+
+      std::string PathFromBuildID;
+      if (!findDebugBinary(Opts.DebugFileDirectory, BuildID, PathFromBuildID))
+        return BinOrErr.takeError();
+      Expected<OwningBinary<Binary>> BuildIDBinOrErr =
+          createBinary(PathFromBuildID);
+      if (!BuildIDBinOrErr)
+        return joinErrors(BinOrErr.takeError(), BuildIDBinOrErr.takeError());
+      consumeError(BinOrErr.takeError());
+      Pair.first->second = std::move(BuildIDBinOrErr.get());
+    }
     Bin = Pair.first->second.getBinary();
   }
 
   if (!Bin)
     return static_cast<ObjectFile *>(nullptr);
 
   if (MachOUniversalBinary *UB = dyn_cast_or_null<MachOUniversalBinary>(Bin)) {
     auto I = ObjectForUBPathAndArch.find(std::make_pair(Path, ArchName));

llvm/test/tools/llvm-symbolizer/debuginfod.test

 RUN: llvm-objcopy --keep-section=.debug_info %p/Inputs/addr.exe \
 RUN:   %t/llvmcache-9800707741016212219
 
 # The symbolizer should call the debuginfod client library, which finds the
 # debuginfo placed in the cache, enabling symbolization of the address.
 RUN: env DEBUGINFOD_CACHE_PATH=%t llvm-symbolizer \
 RUN:   --obj=%t/addr.exe 0x40054d | FileCheck %s --check-prefix=FOUND
 FOUND: {{[/\]+}}tmp{{[/\]+}}x.c:14:0
+
+# This should also work if the Build ID is provided in lieu of a path.
+RUN: env DEBUGINFOD_CACHE_PATH=%t llvm-symbolizer \
+RUN:   --obj=127da749021c1fc1a58cba734a1f542cbe2b7ce4 0x40054d | \
+RUN:   FileCheck %s --check-prefix=FOUND