[llvm-symbolizer] Support debug file lookup using build ID

[phosek]

Summary

Build ID is a protocol for looking up debug files that's already
supported by various tools including debuggers. For example, when
locating debug files, gdb would check the following directories:

• /usr/lib/debug/.build-id/ab/cdef1234.debug
• /usr/bin/ls.debug
• /usr/bin/.debug/ls.debug
• /usr/lib/debug/usr/bin/ls.debug

llvm-symbolizer currently consults all of these except for build ID
based one. This patch implements support for build ID lookup. The
set of debug directories to search is specified by the new option:
--debug-file-directory, whose name matches the debug-file-directory
variable used by gdb for the same purpose.

[phosek]

@rupprecht added the --fallback-debug-path option in rL353730 which is AFAIK used for similar purpose as --debug-file-directory, except that it supports only a single directory, whereas in our use case we'd like to support searching through multiple directories.

[dblaikie]

Test coverage? (since the directory can be specified, I imagine a small checked in binary example (as with other llvm-symbolizer tests) would be feasible?)

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

308–315

Avoid else-after-return: https://llvm.org/docs/CodingStandards.html#don-t-use-else-after-a-return

[phosek]

Test added.

[MaskRay]

llvm/test/DebugInfo/symbolize-build-id.test

7

Can the binary artifact Inputs/dwarfdump-test.elf-x86-64.build-id be replaced with a YAML file that specifies .note.gnu.build-id?

[dblaikie]

llvm/test/DebugInfo/symbolize-build-id.test

4

Perhaps this should have a --debug-file-directory that's specifically non-existent rather than (if I'm reading the code correctly) falling back to the system directories and potentially interacting with other paths on the system that are outside the test's control?

[MaskRay]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

328

Super nit: /*LowerCase=*/ may be more common. At least clang-format recognizes this form and deletes the space before true.

llvm/test/DebugInfo/symbolize-build-id.test

2

In binary utilities tests, we use -o to avoid shell redirection.

16

Machine: is the longest key. The indentation of values can be reduced to the minimum.

Machine: EM_X86_64

24

ditto. Content: 040000000800000003000000474e5500abb50d82b6bdc861

[vsk]

Hey @phosek, @delcypher and I noticed that a Swift+ASan test is failing due to llvm-symbolizer crashing like this:

(gdb) bt
#0  0x00007f6c3adbae97 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f6c3adbc801 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00000000004228d7 in llvm::Error::fatalUncheckedError() const ()
#3  0x000000000046d515 in llvm::symbolize::LLVMSymbolizer::lookUpBuildIDObject(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, llvm::object::ELFObjectFileBase const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#4  0x000000000046dffc in llvm::symbolize::LLVMSymbolizer::getOrCreateObjectPair(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#5  0x000000000046a473 in llvm::symbolize::LLVMSymbolizer::getOrCreateModuleInfo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#6  0x000000000046a9c7 in llvm::symbolize::LLVMSymbolizer::symbolizeInlinedCode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, llvm::object::SectionedAddress) ()
#7  0x000000000040dcd6 in symbolizeInput(llvm::StringRef, llvm::symbolize::LLVMSymbolizer&, llvm::symbolize::DIPrinter&) ()
#8  0x000000000040cff2 in main ()

Based on the history I thought I'd ping this patch (although I'm not sure I spot an unchecked error). Any help appreciated.

[delcypher]

In D70759#1839133, @vsk wrote:

Hey @phosek, @delcypher and I noticed that a Swift+ASan test is failing due to llvm-symbolizer crashing like this:

@phosek @vsk

To add more context without gdb attached the error message is Program aborted due to an unhandled Error: ELF note overflows container

[vsk]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

303

Does Err need to be checked here, after the loop?

From notes_begin:

/// \param Err [out] an error to support fallible iteration, which should
///  be checked after iteration ends.

[dblaikie]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

303

Looks like it to me.

A test case would be good/necessary, though - @lhames - you wouldn't happen to know a good way to produce an object file with this problem, by chance?

[dblaikie]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

303

Also is there anything to be done about this idiom to make it more robust like other uses of llvm::Error? So that this usage would fail even in Success-only executions.

[lhames]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

303

I think it should already fail if you don't check Err, even on success. The caveat is that you would have to exit the bottom of the loop: If you always leave via that return statement (which you might have if that's what you were testing) then you aren't forced to check the error, because returns from inside the loop body are guaranteed to be safe. Making safe returns not require checking was actually one of the main reasons we developed the fallible iterator wrapper in the first place, if memory serves. :)

[dblaikie]

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

303

That does sound vaguely familiar - though whichever case it was (that we designed it this way intentionally or accidentally) - reckon it might be worth revisiting given how easy it is to write code like this that sort of defeats the "must always check" kind of design of Error?

Hmm - I was going to say that I think I'd be OK saying you can't return early from a fallible iteration - but given range-based-for loops make it hard to add new conditions into your loop to exit early when you've found the thing you want, maybe that's not ideal either. Then I end up back at the functor based iteration schemes, rather than exposing a full iterator/range?

BUt yeah, I guess for this code, a test case that doesn't have a BUILD_ID should suffice... - any ideas if yaml2obj is the right tool for that? Or other ways to construct such objects for testing this sort of thing?

[phosek]

Sorry about the breakage, D73438 should address this issue and also includes a test.

llvm/include/llvm/DebugInfo/Symbolize/Symbolize.h

 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_SYMBOLIZE_SYMBOLIZE_H
 #define LLVM_DEBUGINFO_SYMBOLIZE_SYMBOLIZE_H
 
 #include "llvm/DebugInfo/Symbolize/SymbolizableModule.h"
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/ObjectFile.h"
+#include "llvm/Object/ELFObjectFile.h"
 #include "llvm/Support/Error.h"
 #include <algorithm>
 #include <cstdint>
 #include <map>
 #include <memory>
 #include <string>
 #include <utility>
 #include <vector>
struct Options {
     bool UseSymbolTable = true;
     bool Demangle = true;
     bool RelativeAddresses = false;
     bool UntagAddresses = false;
     std::string DefaultArch;
     std::vector<std::string> DsymHints;
     std::string FallbackDebugPath;
     std::string DWPName;
+    std::vector<std::string> DebugFileDirectory;
   };
 
   LLVMSymbolizer() = default;
   LLVMSymbolizer(const Options &Opts) : Opts(Opts) {}
 
   ~LLVMSymbolizer() {
     flush();
   }
createModuleInfo(const ObjectFile *Obj,
                    StringRef ModuleName);
 
   ObjectFile *lookUpDsymFile(const std::string &Path,
                              const MachOObjectFile *ExeObj,
                              const std::string &ArchName);
   ObjectFile *lookUpDebuglinkObject(const std::string &Path,
                                     const ObjectFile *Obj,
                                     const std::string &ArchName);
+  ObjectFile *lookUpBuildIDObject(const std::string &Path,
+                                  const ELFObjectFileBase *Obj,
+                                  const std::string &ArchName);
 
   /// Returns pair of pointers to object and debug object.
   Expected<ObjectPair> getOrCreateObjectPair(const std::string &Path,
                                             const std::string &ArchName);
 
   /// Return a pointer to object file at specified path, for a specified
   /// architecture (e.g. if path refers to a Mach-O universal binary, only one
   /// object file from it will be returned).

llvm/lib/DebugInfo/Symbolize/Symbolize.cpp

bool darwinDsymMatchesBinary(const MachOObjectFile *DbgObj,
                              const MachOObjectFile *Obj) {
   ArrayRef<uint8_t> dbg_uuid = DbgObj->getUuid();
   ArrayRef<uint8_t> bin_uuid = Obj->getUuid();
   if (dbg_uuid.empty() || bin_uuid.empty())
     return false;
   return !memcmp(dbg_uuid.data(), bin_uuid.data(), dbg_uuid.size());
 }
 
+template <typename ELFT>
+Optional<ArrayRef<uint8_t>> getBuildID(const ELFFile<ELFT> *Obj) {
+  if (!Obj)
+    return {};
+  auto PhdrsOrErr = Obj->program_headers();
+  if (!PhdrsOrErr) {
+    consumeError(PhdrsOrErr.takeError());
+    return {};
+  }
+  for (const auto &P : *PhdrsOrErr) {
+    if (P.p_type != ELF::PT_NOTE)
+      continue;
+    Error Err = Error::success();
+    for (const auto &N : Obj->notes(P, Err))
+      if (N.getType() == ELF::NT_GNU_BUILD_ID && N.getName() == ELF::ELF_NOTE_GNU)
+        return N.getDesc();
+  }
+  return {};
+}
+
+Optional<ArrayRef<uint8_t>> getBuildID(const ELFObjectFileBase *Obj) {
+  Optional<ArrayRef<uint8_t>> BuildID;
+  if (auto *O = dyn_cast<ELFObjectFile<ELF32LE>>(Obj))
+    BuildID = getBuildID(O->getELFFile());
+  else if (auto *O = dyn_cast<ELFObjectFile<ELF32BE>>(Obj))
+    BuildID = getBuildID(O->getELFFile());
+  else if (auto *O = dyn_cast<ELFObjectFile<ELF64LE>>(Obj))
+    BuildID = getBuildID(O->getELFFile());
+  else if (auto *O = dyn_cast<ELFObjectFile<ELF64BE>>(Obj))
+    BuildID = getBuildID(O->getELFFile());
+  else
+    llvm_unreachable("unsupported file format");
+  return BuildID;
+}
+
+bool findDebugBinary(const std::vector<std::string> &DebugFileDirectory,
+                     const ArrayRef<uint8_t> BuildID,
+                     std::string &Result) {
+  auto getDebugPath = [&](StringRef Directory) {
+    SmallString<128> Path{Directory};
+    sys::path::append(Path, ".build-id",
+                      llvm::toHex(BuildID[0], /*LowerCase=*/true),
+                      llvm::toHex(BuildID.slice(1), /*LowerCase=*/true));
+    Path += ".debug";
+    return Path;
+  };
+  if (DebugFileDirectory.empty()) {
+    SmallString<128> Path = getDebugPath(
+#if defined(__NetBSD__)
+      // Try /usr/libdata/debug/.build-id/../...
+      "/usr/libdata/debug"
+#else
+      // Try /usr/lib/debug/.build-id/../...
+      "/usr/lib/debug"
+#endif
+    );
+    if (llvm::sys::fs::exists(Path)) {
+      Result = Path.str();
+      return true;
+    }
+  } else {
+    for (const auto &Directory : DebugFileDirectory) {
+      // Try <debug-file-directory>/.build-id/../...
+      SmallString<128> Path = getDebugPath(Directory);
+      if (llvm::sys::fs::exists(Path)) {
+        Result = Path.str();
+        return true;
+      }
+    }
+  }
+  return false;
+}
+
 } // end anonymous namespace
 
 ObjectFile *LLVMSymbolizer::lookUpDsymFile(const std::string &ExePath,
     const MachOObjectFile *MachExeObj, const std::string &ArchName) {
   // On Darwin we may find DWARF in separate object file in
   // resource directory.
   std::vector<std::string> DsymPaths;
   StringRef Filename = sys::path::filename(ExePath);
ObjectFile *LLVMSymbolizer::lookUpDebuglinkObject(const std::string &Path,
   if (!DbgObjOrErr) {
     // Ignore errors, the file might not exist.
     consumeError(DbgObjOrErr.takeError());
     return nullptr;
   }
   return DbgObjOrErr.get();
 }
 
+ObjectFile *LLVMSymbolizer::lookUpBuildIDObject(const std::string &Path,
+                                                const ELFObjectFileBase *Obj,
+                                                const std::string &ArchName) {
+  auto BuildID = getBuildID(Obj);
+  if (!BuildID)
+    return nullptr;
+  if (BuildID->size() < 2)
+    return nullptr;
+  std::string DebugBinaryPath;
+  if (!findDebugBinary(Opts.DebugFileDirectory, *BuildID, DebugBinaryPath))
+    return nullptr;
+  auto DbgObjOrErr = getOrCreateObject(DebugBinaryPath, ArchName);
+  if (!DbgObjOrErr) {
+    consumeError(DbgObjOrErr.takeError());
+    return nullptr;
+  }
+  return DbgObjOrErr.get();
+}
+
 Expected<LLVMSymbolizer::ObjectPair>
 LLVMSymbolizer::getOrCreateObjectPair(const std::string &Path,
                                       const std::string &ArchName) {
   auto I = ObjectPairForPathArch.find(std::make_pair(Path, ArchName));
   if (I != ObjectPairForPathArch.end())
     return I->second;
 
   auto ObjOrErr = getOrCreateObject(Path, ArchName);
   if (!ObjOrErr) {
     ObjectPairForPathArch.emplace(std::make_pair(Path, ArchName),
                                   ObjectPair(nullptr, nullptr));
     return ObjOrErr.takeError();
   }
 
   ObjectFile *Obj = ObjOrErr.get();
   assert(Obj != nullptr);
   ObjectFile *DbgObj = nullptr;
 
   if (auto MachObj = dyn_cast<const MachOObjectFile>(Obj))
     DbgObj = lookUpDsymFile(Path, MachObj, ArchName);
+  else if (auto ELFObj = dyn_cast<const ELFObjectFileBase>(Obj))
+    DbgObj = lookUpBuildIDObject(Path, ELFObj, ArchName);
   if (!DbgObj)
     DbgObj = lookUpDebuglinkObject(Path, Obj, ArchName);
   if (!DbgObj)
     DbgObj = Obj;
   ObjectPair Res = std::make_pair(Obj, DbgObj);
   ObjectPairForPathArch.emplace(std::make_pair(Path, ArchName), Res);
   return Res;
 }

llvm/test/DebugInfo/symbolize-build-id.test

+# RUN: yaml2obj %s -o %t
+
+# RUN: llvm-symbolizer --debug-file-directory=/non-existent --obj=%t 0x20112f | FileCheck --check-prefix=UNKNOWN %s
+
+# UNKNOWN:      ??
+# UNKNOWN-NEXT: ??:0:0
+
+# RUN: llvm-symbolizer --debug-file-directory=%p/Inputs --obj=%t 0x20112f | FileCheck --check-prefix=FOUND %s
+
+# FOUND:      main
+# FOUND-NEXT: /tmp/dbginfo{{[/\\]}}dwarfdump-test.cc:16
+
+--- !ELF
+FileHeader:
+  Class:   ELFCLASS64
+  Data:    ELFDATA2LSB
+  Type:    ET_EXEC
+  Machine: EM_X86_64
+Sections:
+  - Name:    .note.gnu.build-id
+    Type:    SHT_NOTE
+    Flags:   [ SHF_ALLOC ]
+    Content: 040000000800000003000000474e5500abb50d82b6bdc861
+ProgramHeaders:
+  - Type: PT_NOTE
+    Flags: [ PF_R ]
+    Sections:
+      - Section: .note.gnu.build-id

llvm/tools/llvm-symbolizer/llvm-symbolizer.cpp

 static cl::list<std::string> ClInputAddresses(cl::Positional,
                                               cl::desc("<input addresses>..."),
                                               cl::ZeroOrMore);
 
 static cl::opt<std::string>
     ClFallbackDebugPath("fallback-debug-path", cl::init(""),
                         cl::desc("Fallback path for debug binaries."));
 
+static cl::list<std::string>
+    ClDebugFileDirectory("debug-file-directory", cl::ZeroOrMore,
+                         cl::value_desc("dir"),
+                         cl::desc("Path to directory where to look for debug "
+                                  "files."));
+
 static cl::opt<DIPrinter::OutputStyle>
     ClOutputStyle("output-style", cl::init(DIPrinter::OutputStyle::LLVM),
                   cl::desc("Specify print style"),
                   cl::values(clEnumValN(DIPrinter::OutputStyle::LLVM, "LLVM",
                                         "LLVM default style"),
                              clEnumValN(DIPrinter::OutputStyle::GNU, "GNU",
                                         "GNU addr2line style")));
 
int main(int argc, char **argv) {
   Opts.PrintFunctions = ClPrintFunctions;
   Opts.UseSymbolTable = ClUseSymbolTable;
   Opts.Demangle = ClDemangle;
   Opts.RelativeAddresses = ClUseRelativeAddress;
   Opts.UntagAddresses = ClUntagAddresses;
   Opts.DefaultArch = ClDefaultArch;
   Opts.FallbackDebugPath = ClFallbackDebugPath;
   Opts.DWPName = ClDwpName;
+  Opts.DebugFileDirectory = ClDebugFileDirectory;
 
   for (const auto &hint : ClDsymHint) {
     if (sys::path::extension(hint) == ".dSYM") {
       Opts.DsymHints.push_back(hint);
     } else {
       errs() << "Warning: invalid dSYM hint: \"" << hint <<
                 "\" (must have the '.dSYM' extension).\n";
     }