[llvm-cov] Look up object files using debuginfod

[mysterymath]

Remove error condition when no files specified.

[mysterymath]

Allow mixing object files and fetched build IDs.

[mysterymath]

Extract the build IDs from the given files.

[mysterymath]

Fix logic and add tests and documentation.

[mysterymath]

Clean the patch a bit.

[jhenderson]

Couple of drive-by comments, as my Herald rule triggered due to the llvm-objdump/llvm-symbolizer changes (which look fine to me).

llvm/lib/Debuginfod/Debuginfod.cpp

61

In other parts of the codebase, consumeError without any explanation is generally a code smell. A comment explaining why it's appropriate instead of returning the error up the stack would be worthwhile here.

llvm/tools/llvm-cov/CodeCoverage.cpp

657

There seems to be an unfortunate mixture of help descriptions ending with full stops and others that don't. It seems no full stop in this file is more prevalent, so please remove it.

[mysterymath]

Address review comments.
Refactor getDefaultDebuginfodUrls.

llvm/lib/Debuginfod/Debuginfod.cpp

61

Several of the debuginfod functions return Expected even though they have no possibility of failure, which seems like the root of the smell. I've refactored this call to just return a value.

[jhenderson]

Thanks for the minor changes. I'll leave reviewing the rest of this to someone else with more applicable knowledge.

llvm/lib/Debuginfod/Debuginfod.cpp

61

Cool, looks good.

[ellis]

llvm/docs/CommandGuide/llvm-cov.rst

352

Can we add a test that uses this option? Maybe a test to make sure we can disable it without problem.

llvm/tools/llvm-cov/CodeCoverage.cpp

656

I personally prefer cl::ZeroOrMore rather than cl::Optional because I believe cl::Optional will fail if this option occurs twice. Does that make sense for these options?

[mysterymath]

Addressed review comments.

Testing this patch using file:// URLs uncovered an minor issue in the
Debuginfod client. Fixed this, allowing file URLs to work.

[mysterymath]

llvm/tools/llvm-cov/CodeCoverage.cpp

656

SGTM; I'm generally a fan of the the last-flag-wins semantics as well.

[mysterymath]

Integrate previous patch.
Require curl for debuginfod test.

[mysterymath]

Fix have_curl configuration in compiler-rt lit site config.

[gulfem]

compiler-rt/test/profile/Linux/binary-id-debuginfod.c

14

Should this be -debuginfod instead of -debuginfod=0? Are you trying to enable debuginfod option here?

compiler-rt/test/profile/Linux/binary-id-lookup.c

15

Can you extend both tests to check the line and coverage numbers instead of just checking the function names?

llvm/docs/CommandGuide/llvm-cov.rst

356

Maybe "but its object file is not provided on the command line"?

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

448

What happens when -debug-file-directory option is provided and there is no debug file that includes coverage data? For example, what happens if an empty directory or a directory with uninstrumented binaries is provided? When the object file that is provided does not contain any coverage instrumentation, this line should catch that it is a no_data_found error. With the new changes, I think this line would not be an error because ObjectFilenames will be empty if we only provide -debug-file-directory. So, I think it won't be able report any coverage, but it won't report an error. That's why I suggest adding a test with empty directory and check the expected behavior?

llvm/tools/llvm-cov/CodeCoverage.cpp

766

Should we only create a BIDFetcher when -debuginfod or -debug-file-directory options are provided? This seems to create a BIDFetcher by default even if those options are not provided.

[gulfem]

Can you remove the dot at the end of the commit message? ([llvm-cov] Look up object files using debuginfod.)

[mysterymath]

Address review comments.

[mysterymath]

In D136702#4053086, @gulfem wrote:

Can you remove the dot at the end of the commit message? ([llvm-cov] Look up object files using debuginfod.

Done.

compiler-rt/test/profile/Linux/binary-id-debuginfod.c

14

Changed this to -debuginfod=false and renamed the CHECK to NODEBUGINFOD to clarify. debuginfod should default on when compiled with curl and if DEBUGINFOD_URLS is specified.

llvm/docs/CommandGuide/llvm-cov.rst

356

Tightened up the wording here along those lines. Done.

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

448

I was originally worried about changing the behavior of this line WRT when ObjectFilenames were empty, but looking at the broader codebase, it doesn't look like the empty ObjectFilename case was ever possible. An early exit occured on empty ObjectFilenames before this line on every path that called CoverageMapping::load.

Accordingly, I've just changed this to emit an error if no data was found. I've also added a check for this error.

llvm/tools/llvm-cov/CodeCoverage.cpp

766

That's intentional, to parallel how the other utilities in llvm work. There are system default directories provided by Linux distributions that contain DWARF object files whenever debug packages are installed; these directories are implicitly present in the -debug-file-directory list, and the default BIDFetcher will pick these up.

[mysterymath]

Correct description of -debug-file-directory.

[dyung]

@mysterymath this change is causing problems on several bots related to your detection on whether curl is present or not. Can you take a look and revert if you need time to investigate?

• https://lab.llvm.org/buildbot/#/builders/247/builds/884
• https://lab.llvm.org/buildbot/#/builders/231/builds/7688
• https://lab.llvm.org/buildbot/#/builders/121/builds/27389
• https://lab.llvm.org/buildbot/#/builders/230/builds/8464
• https://lab.llvm.org/buildbot/#/builders/57/builds/24209
• https://lab.llvm.org/buildbot/#/builders/127/builds/42722

[dyung]

@mysterymath I reverted this change as well as follow-up change you made in order to unblock the build bots.

[phosek]

compiler-rt/test/profile/lit.site.cfg.py.in

7

You need to pythonize this variable. For this to work with the runtimes build, we also need to export LLVM_ENABLE_CURL in LLVMConfig.cmake, see https://github.com/llvm/llvm-project/blob/7f48154ca11fee6579fe29a51ab2a329424d31c4/llvm/cmake/modules/LLVMConfig.cmake.in#L81

There's a broader question of whether we should be passing the variables like LLVM_ENABLE_CURL to the runtimes in the first build since they are only relevant to the LLVM build. A better approach would be to provide some indication whether debuginfod is available in the llvm-cov --help output (or perhaps have a dedicated flag) and then parse that information when running compiler-rt CMake build and use the result here.

[mysterymath]

In D136702#4081713, @dyung wrote:

@mysterymath I reverted this change as well as follow-up change you made in order to unblock the build bots.

Thanks, sorry about that.

[mysterymath]

compiler-rt/test/profile/lit.site.cfg.py.in

7

You need to pythonize this variable. For this to work with the runtimes build, we also need to export LLVM_ENABLE_CURL in LLVMConfig.cmake, see https://github.com/llvm/llvm-project/blob/7f48154ca11fee6579fe29a51ab2a329424d31c4/llvm/cmake/modules/LLVMConfig.cmake.in#L81

It looks like compiler-rt does this directly in cmake; I'll add this there for consistency.

There's a broader question of whether we should be passing the variables like LLVM_ENABLE_CURL to the runtimes in the first build since they are only relevant to the LLVM build. A better approach would be to provide some indication whether debuginfod is available in the llvm-cov --help output (or perhaps have a dedicated flag) and then parse that information when running compiler-rt CMake build and use the result here.

Seems like the capabilities of the LLVM build are relevant to the compiler-rt tests, since they control the capabilities of the tools used in those tests. We could add optional capability advertisement generally to utilities, but I think one would probably want a principled way to do this; either something centralized like llvm-config, or a common flag to extract out optional capabilities in some format or another.

[jgorbe]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

We're getting build errors here (and the similar call to std::unique on line 426) with -Wunused-result.

This looks like a bug: std::unique returns the new end of the range, and typically you need to call container.erase(new_end, container.end()) after the call to std::unique to reduce the container size to match the new logical size. See https://en.cppreference.com/w/cpp/algorithm/unique.

[mysterymath]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

Thanks; did a quick fix forward for this. This uncovered a mistake with Compare as well; std::unique requires a == comparator, while sort and set_difference a < comparator.

[dblaikie]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

Is there test coverage for these bugs/fixes?

[mysterymath]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

Partially; the unused result lint caught the issue with std::unique; fixing that caused the original test cases to fail due to the comparator issue.

[dblaikie]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

Could you see about adding a test case that would've failed with the code as originally committed? (something that demonstrates the need for the unique call in the first place)

[mysterymath]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

I took a closer look at the gulfem's preceding change (https://reviews.llvm.org/D135929), and it appears that the std::unique here is superfluous; there's no effective way to produce duplicate build IDs in ProfileBinaryIDs, and in that case, they're harmless in FoundBinaryIDs, since std::max(1-k, 0) is 0 for any k >= 1. The ProfileBinaryIDs sort is also superfluous.

That does mean there's intrinsically no way to test it, since it makes no observable difference. I've thus removed the uniquing logic completely.

[dblaikie]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

ah, excellent - the best code is no code. Could you provide a link here to the commit hash/etc that removed it, for the record?

[mysterymath]

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

422

That's https://reviews.llvm.org/rG0a51eeda1ef43d0f9a73ed2ac15f602262943ea1

compiler-rt/test/profile/Linux/binary-id-debuginfod.c

+// REQUIRES: curl, linux
+// RUN: split-file %s %t
+// RUN: %clang_profgen -Wl,--build-id=0x12345678 -fcoverage-mapping -O2 -shared %t/foo.c -o %t/libfoo.so
+// RUN: %clang_profgen -Wl,--build-id=0xabcd1234 -fcoverage-mapping -O2 %t/main.c -L%t -lfoo -o %t.main
+// RUN: env LLVM_PROFILE_FILE=%t.profdir/default_%m.profraw LD_LIBRARY_PATH=%t %run %t.main
+// RUN: mkdir -p %t/buildid/12345678 %t/buildid/abcd1234
+// RUN: cp %t/libfoo.so %t/buildid/12345678/debuginfo
+// RUN: cp %t.main %t/buildid/abcd1234/debuginfo
+// RUN: llvm-profdata merge -o %t.profdata %t.profdir/default_*.profraw
+// RUN: env DEBUGINFOD_URLS=file://%t llvm-cov show -instr-profile %t.profdata | FileCheck %s
+// RUN: echo "bad" > %t/libfoo.so %t/buildid/12345678/debuginfo
+// RUN: echo "bad" > %t/buildid/abcd1234/debuginfo
+// RUN: env DEBUGINFOD_URLS=file://%t llvm-cov show -instr-profile %t.profdata -debuginfod=0 %t.main | FileCheck %s --check-prefix=MAINONLY
+
+// CHECK: foo(void) {}
+// CHECK: bar
+// CHECK: main
+
+// MAINONLY-NOT: foo(void) {}
+// MAINONLY: main
+
+//--- foo.c
+void foo(void) {}
+
+//--- main.c
+void foo(void);
+void bar(void) {}
+int main() {
+  foo();
+  bar();
+  return 0;
+}

compiler-rt/test/profile/Linux/binary-id-lookup.c

+// REQUIRES: linux
+// RUN: split-file %s %t
+// RUN: %clang_profgen -Wl,--build-id=0x12345678 -fcoverage-mapping -O2 -shared %t/foo.c -o %t/libfoo.so
+// RUN: %clang_profgen -Wl,--build-id=0xabcd1234 -fcoverage-mapping -O2 %t/main.c -L%t -lfoo -o %t.main
+// RUN: env LLVM_PROFILE_FILE=%t.profdir/default_%m.profraw LD_LIBRARY_PATH=%t %run %t.main
+// RUN: mkdir -p %t/.build-id/12 %t/.build-id/ab
+// RUN: cp %t/libfoo.so %t/.build-id/12/345678.debug
+// RUN: cp %t.main %t/.build-id/ab/cd1234.debug
+// RUN: llvm-profdata merge -o %t.profdata %t.profdir/default_*.profraw
+// RUN: llvm-cov show -instr-profile %t.profdata -debug-file-directory %t | FileCheck %s
+// RUN: echo "bad" > %t/.build-id/ab/cd1234.debug
+// RUN: llvm-cov show -instr-profile %t.profdata -debug-file-directory %t %t.main | FileCheck %s
+
+// CHECK: foo
+// CHECK: bar
+// CHECK: main
+
+//--- foo.c
+void foo(void) {}
+
+//--- main.c
+void foo(void);
+void bar(void) {}
+int main() {
+  foo();
+  bar();
+  return 0;
+}

compiler-rt/test/profile/Linux/lit.local.cfg.py

if not 'invalid linker' in clang_err:
     return True
 
   return False
 
 root = getRoot(config)
 
 if root.host_os not in ['Linux'] or not is_gold_linker_available():
   config.unsupported = True
+
+if config.have_curl:
+    config.available_features.add('curl')

compiler-rt/test/profile/lit.site.cfg.py.in

 @LIT_SITE_CFG_IN_HEADER@
 
 # Tool-specific config options.
 config.profile_lit_binary_dir = "@PROFILE_LIT_BINARY_DIR@"
 config.target_cflags = "@PROFILE_TEST_TARGET_CFLAGS@"
 config.target_arch = "@PROFILE_TEST_TARGET_ARCH@"
+config.have_curl = "@LLVM_ENABLE_CURL@"
 
 # Load common config for all compiler-rt lit tests.
 lit_config.load_config(config, "@COMPILER_RT_BINARY_DIR@/test/lit.common.configured")
 
 # Load tool-specific config that would do the real work.
 lit_config.load_config(config, "@PROFILE_LIT_SOURCE_DIR@/lit.cfg.py")

llvm/docs/CommandGuide/llvm-cov.rst

 
 .. option:: -coverage-watermark=<high>,<low>
 
  Set high and low watermarks for coverage in html format output. This allows you
  to set the high and low watermark of coverage as desired, green when
  coverage >= high, red when coverage < low, and yellow otherwise. Both high and
  low should be between 0-100 and high > low.
 
+.. option:: -debuginfod
+
+Attempt to look up coverage mapping from objects using debuginfod. This is
+attempted by default for binary IDs present in the profile but not provided on
+the command line, so long as debuginfod is compiled in and configured via
+DEBUGINFOD_URLS.
+
+.. option:: -debug-file-directory=<dir>
+
+Provides a directory to search for objects corresponding to binary IDs in the
+profile.
+
 .. program:: llvm-cov report
 
 .. _llvm-cov-report:
 
 REPORT COMMAND
 --------------
 
 SYNOPSIS

  Skip source code files with file paths that match the given regular expression.
 
 .. option:: -compilation-dir=<dir>
 
  Directory used as a base for relative coverage mapping paths. Only applicable
  when binaries have been compiled with one of `-fcoverage-prefix-map`
  `-fcoverage-compilation-dir`, or `-ffile-compilation-dir`.
 
+.. option:: -debuginfod
+
+Attempt to look up coverage mapping from objects using debuginfod. This is
+attempted by default for binary IDs present in the profile but not provided on
+the command line, so long as debuginfod is compiled in and configured via
+DEBUGINFOD_URLS.
+
+.. option:: -debug-file-directory=<dir>
+
+Provides a directory to search for objects corresponding to binary IDs in the
+profile.
+
 .. program:: llvm-cov export
 
 .. _llvm-cov-export:
 
 EXPORT COMMAND
 --------------
 
 SYNOPSIS

  Use N threads to export coverage data. When N=0, llvm-cov auto-detects an
  appropriate number of threads to use. This is the default.
 
 .. option:: -compilation-dir=<dir>
 
  Directory used as a base for relative coverage mapping paths. Only applicable
  when binaries have been compiled with one of `-fcoverage-prefix-map`
  `-fcoverage-compilation-dir`, or `-ffile-compilation-dir`.
+
+.. option:: -debuginfod
+
+Attempt to look up coverage mapping from objects using debuginfod. This is
+attempted by default for binary IDs present in the profile but not provided on
+the command line, so long as debuginfod is compiled in and configured via
+DEBUGINFOD_URLS.
+
+.. option:: -debug-file-directory=<dir>
+
+Provides a directory to search for objects corresponding to binary IDs in the
+profile.

llvm/include/llvm/Debuginfod/Debuginfod.h

 #include "llvm/Support/Timer.h"
 
 #include <chrono>
 #include <condition_variable>
 #include <queue>
 
 namespace llvm {
 
+/// Returns false if a debuginfod lookup can be determined to have no chance of
+/// succeeding.
+bool canUseDebuginfod();
+
 /// Finds default array of Debuginfod server URLs by checking DEBUGINFOD_URLS
 /// environment variable.
-Expected<SmallVector<StringRef>> getDefaultDebuginfodUrls();
+SmallVector<StringRef> getDefaultDebuginfodUrls();
 
 /// Finds a default local file caching directory for the debuginfod client,
 /// first checking DEBUGINFOD_CACHE_PATH.
 Expected<std::string> getDefaultDebuginfodCacheDirectory();
 
 /// Finds a default timeout for debuginfod HTTP requests. Checks
 /// DEBUGINFOD_TIMEOUT environment variable, default is 90 seconds (90000 ms).
 std::chrono::milliseconds getDefaultDebuginfodTimeout();

llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h

 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/Hashing.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/iterator.h"
 #include "llvm/ADT/iterator_range.h"
+#include "llvm/Object/BuildID.h"
 #include "llvm/ProfileData/InstrProf.h"
 #include "llvm/Support/Alignment.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <cstdint>
 #include <iterator>
 #include <memory>
 #include <string>
 #include <system_error>
 #include <tuple>
 #include <utility>
 #include <vector>
 
 namespace llvm {
 
 class IndexedInstrProfReader;
 
+namespace object {
+class BuildIDFetcher;
+} // namespace object
+
 namespace coverage {
 
 class CoverageMappingReader;
 struct CoverageMappingRecord;
 
 enum class coveragemap_error {
   success = 0,
   eof,
class CoverageMapping {
 
   CoverageMapping() = default;
 
   // Load coverage records from readers.
   static Error loadFromReaders(
       ArrayRef<std::unique_ptr<CoverageMappingReader>> CoverageReaders,
       IndexedInstrProfReader &ProfileReader, CoverageMapping &Coverage);
 
+  // Load coverage records from file.
+  static Error
+  loadFromFile(StringRef Filename, StringRef Arch, StringRef CompilationDir,
+               IndexedInstrProfReader &ProfileReader, CoverageMapping &Coverage,
+               bool &DataFound,
+               SmallVectorImpl<object::BuildID> *FoundBinaryIDs = nullptr);
+
   /// Add a function record corresponding to \p Record.
   Error loadFunctionRecord(const CoverageMappingRecord &Record,
                            IndexedInstrProfReader &ProfileReader);
 
   /// Look up the indices for function records which are at least partially
   /// defined in the specified file. This is guaranteed to return a superset of
   /// such records: extra records not in the file may be included if there is
   /// a hash collision on the filename. Clients must be robust to collisions.
public:
   load(ArrayRef<std::unique_ptr<CoverageMappingReader>> CoverageReaders,
        IndexedInstrProfReader &ProfileReader);
 
   /// Load the coverage mapping from the given object files and profile. If
   /// \p Arches is non-empty, it must specify an architecture for each object.
   /// Ignores non-instrumented object files unless all are not instrumented.
   static Expected<std::unique_ptr<CoverageMapping>>
   load(ArrayRef<StringRef> ObjectFilenames, StringRef ProfileFilename,
-       ArrayRef<StringRef> Arches = None, StringRef CompilationDir = "");
+       ArrayRef<StringRef> Arches = None, StringRef CompilationDir = "",
+       const object::BuildIDFetcher* BIDFetcher = nullptr);
 
   /// The number of functions that couldn't have their profiles mapped.
   ///
   /// This is a count of functions whose profile is out of date or otherwise
   /// can't be associated with any coverage information.
   unsigned getMismatchedCount() const { return FuncHashMismatches.size(); }
 
   /// A hash mismatch occurs when a profile record for a symbol does not have

llvm/include/llvm/ProfileData/Coverage/CoverageMappingReader.h

 
 public:
   BinaryCoverageReader(const BinaryCoverageReader &) = delete;
   BinaryCoverageReader &operator=(const BinaryCoverageReader &) = delete;
 
   static Expected<std::vector<std::unique_ptr<BinaryCoverageReader>>>
   create(MemoryBufferRef ObjectBuffer, StringRef Arch,
          SmallVectorImpl<std::unique_ptr<MemoryBuffer>> &ObjectFileBuffers,
-         StringRef CompilationDir = "");
+         StringRef CompilationDir = "",
+         SmallVectorImpl<object::BuildIDRef> *BinaryIDs = nullptr);
 
   static Expected<std::unique_ptr<BinaryCoverageReader>>
   createCoverageReaderFromBuffer(StringRef Coverage,
                                  FuncRecordsStorage &&FuncRecords,
                                  InstrProfSymtab &&ProfileNames,
                                  uint8_t BytesInAddress,
                                  support::endianness Endian,
                                  StringRef CompilationDir = "");

llvm/lib/Debuginfod/Debuginfod.cpp

 static std::string uniqueKey(llvm::StringRef S) { return utostr(xxHash64(S)); }
 
 // Returns a binary BuildID as a normalized hex string.
 // Uses lowercase for compatibility with common debuginfod servers.
 static std::string buildIDToString(BuildIDRef ID) {
   return llvm::toHex(ID, /*LowerCase=*/true);
 }
 
-Expected<SmallVector<StringRef>> getDefaultDebuginfodUrls() {
+bool canUseDebuginfod() {
+  return HTTPClient::isAvailable() && !getDefaultDebuginfodUrls().empty();
+}
+
+SmallVector<StringRef> getDefaultDebuginfodUrls() {
   const char *DebuginfodUrlsEnv = std::getenv("DEBUGINFOD_URLS");
   if (DebuginfodUrlsEnv == nullptr)
     return SmallVector<StringRef>();
 
   SmallVector<StringRef> DebuginfodUrls;
   StringRef(DebuginfodUrlsEnv).split(DebuginfodUrls, " ");
   return DebuginfodUrls;
 }
 
Expected<std::string> getCachedOrDownloadArtifact(StringRef UniqueKey,
                                                   StringRef UrlPath) {
   SmallString<10> CacheDir;
 
   Expected<std::string> CacheDirOrErr = getDefaultDebuginfodCacheDirectory();
   if (!CacheDirOrErr)
     return CacheDirOrErr.takeError();
   CacheDir = *CacheDirOrErr;
 
-  Expected<SmallVector<StringRef>> DebuginfodUrlsOrErr =
-      getDefaultDebuginfodUrls();
-  if (!DebuginfodUrlsOrErr)
-    return DebuginfodUrlsOrErr.takeError();
-  SmallVector<StringRef> &DebuginfodUrls = *DebuginfodUrlsOrErr;
   return getCachedOrDownloadArtifact(UniqueKey, UrlPath, CacheDir,
-                                     DebuginfodUrls,
+                                     getDefaultDebuginfodUrls(),
                                      getDefaultDebuginfodTimeout());
 }
 
 namespace {
 
 /// A simple handler which streams the returned data to a cache file. The cache
 /// file is only created if a 200 OK status is observed.
 class StreamedHTTPResponseHandler : public HTTPResponseHandler {
public:
 
   Error handleBodyChunk(StringRef BodyChunk) override;
 };
 
 } // namespace
 
 Error StreamedHTTPResponseHandler::handleBodyChunk(StringRef BodyChunk) {
   if (!FileStream) {
-    if (Client.responseCode() != 200)
+    unsigned Code = Client.responseCode();
+    if (Code && Code != 200)
       return Error::success();
     Expected<std::unique_ptr<CachedFileStream>> FileStreamOrError =
         CreateStream();
     if (!FileStreamOrError)
       return FileStreamOrError.takeError();
     FileStream = std::move(*FileStreamOrError);
   }
   *FileStream->OS << BodyChunk;
for (StringRef ServerUrl : DebuginfodUrls) {
     // the cache.
     StreamedHTTPResponseHandler Handler([&]() { return CacheAddStream(Task); },
                                         Client);
     HTTPRequest Request(ArtifactUrl);
     Error Err = Client.perform(Request, Handler);
     if (Err)
       return std::move(Err);
 
-    if (Client.responseCode() != 200)
+    unsigned Code = Client.responseCode();
+    if (Code && Code != 200)
       continue;
 
     // Return the path to the artifact on disk.
     return std::string(AbsCachedArtifactPath);
   }
 
   return createStringError(errc::argument_out_of_domain, "build id not found");
 }

llvm/lib/ProfileData/Coverage/CoverageMapping.cpp

 #include "llvm/ProfileData/Coverage/CoverageMapping.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/None.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallBitVector.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Object/BuildID.h"
 #include "llvm/ProfileData/Coverage/CoverageMappingReader.h"
 #include "llvm/ProfileData/InstrProfReader.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/raw_ostream.h"
static Error handleMaybeNoDataFoundError(Error E) {
   return handleErrors(
       std::move(E), [](const CoverageMapError &CME) {
         if (CME.get() == coveragemap_error::no_data_found)
           return static_cast<Error>(Error::success());
         return make_error<CoverageMapError>(CME.get());
       });
 }
 
-Expected<std::unique_ptr<CoverageMapping>>
-CoverageMapping::load(ArrayRef<StringRef> ObjectFilenames,
-                      StringRef ProfileFilename, ArrayRef<StringRef> Arches,
-                      StringRef CompilationDir) {
-  auto ProfileReaderOrErr = IndexedInstrProfReader::create(ProfileFilename);
-  if (Error E = ProfileReaderOrErr.takeError())
-    return createFileError(ProfileFilename, std::move(E));
-  auto ProfileReader = std::move(ProfileReaderOrErr.get());
-  auto Coverage = std::unique_ptr<CoverageMapping>(new CoverageMapping());
-  bool DataFound = false;
-
-  for (const auto &File : llvm::enumerate(ObjectFilenames)) {
-    auto CovMappingBufOrErr = MemoryBuffer::getFileOrSTDIN(
-        File.value(), /*IsText=*/false, /*RequiresNullTerminator=*/false);
-    if (std::error_code EC = CovMappingBufOrErr.getError())
-      return createFileError(File.value(), errorCodeToError(EC));
-    StringRef Arch = Arches.empty() ? StringRef() : Arches[File.index()];
-    MemoryBufferRef CovMappingBufRef =
-        CovMappingBufOrErr.get()->getMemBufferRef();
-    SmallVector<std::unique_ptr<MemoryBuffer>, 4> Buffers;
-    auto CoverageReadersOrErr = BinaryCoverageReader::create(
-        CovMappingBufRef, Arch, Buffers, CompilationDir);
-    if (Error E = CoverageReadersOrErr.takeError()) {
-      E = handleMaybeNoDataFoundError(std::move(E));
-      if (E)
-        return createFileError(File.value(), std::move(E));
-      // E == success (originally a no_data_found error).
-      continue;
-    }
+Error CoverageMapping::loadFromFile(
+    StringRef Filename, StringRef Arch, StringRef CompilationDir,
+    IndexedInstrProfReader &ProfileReader, CoverageMapping &Coverage,
+    bool &DataFound, SmallVectorImpl<object::BuildID> *FoundBinaryIDs) {
+      Filename, /*IsText=*/false, /*RequiresNullTerminator=*/false);
+    return createFileError(Filename, errorCodeToError(EC));
+
+  SmallVector<object::BuildIDRef> BinaryIDs;
+      CovMappingBufRef, Arch, Buffers, CompilationDir,
+      FoundBinaryIDs ? &BinaryIDs : nullptr);
+      return createFileError(Filename, std::move(E));
+    return E;
 
-    SmallVector<std::unique_ptr<CoverageMappingReader>, 4> Readers;
-    for (auto &Reader : CoverageReadersOrErr.get())
-      Readers.push_back(std::move(Reader));
-    DataFound |= !Readers.empty();
-    if (Error E = loadFromReaders(Readers, *ProfileReader, *Coverage))
-      return createFileError(File.value(), std::move(E));
-  }
+  if (FoundBinaryIDs && !Readers.empty()) {
+    llvm::append_range(*FoundBinaryIDs,
+                       llvm::map_range(BinaryIDs, [](object::BuildIDRef BID) {
+                         return object::BuildID(BID);
+                       }));
+  }
+  if (Error E = loadFromReaders(Readers, ProfileReader, Coverage))
+    return createFileError(Filename, std::move(E));
+  return Error::success();
+
+Expected<std::unique_ptr<CoverageMapping>>
+CoverageMapping::load(ArrayRef<StringRef> ObjectFilenames,
+                      StringRef ProfileFilename, ArrayRef<StringRef> Arches,
+                      StringRef CompilationDir,
+                      const object::BuildIDFetcher* BIDFetcher) {
+  auto ProfileReaderOrErr = IndexedInstrProfReader::create(ProfileFilename);
+  if (Error E = ProfileReaderOrErr.takeError())
+    return createFileError(ProfileFilename, std::move(E));
+  auto ProfileReader = std::move(ProfileReaderOrErr.get());
+  auto Coverage = std::unique_ptr<CoverageMapping>(new CoverageMapping());
+  bool DataFound = false;
+
+  auto GetArch = [&](size_t Idx) {
+    if (Arches.empty())
+      return StringRef();
+    if (Arches.size() == 1)
+      return Arches.front();
+    return Arches[Idx];
+  };
+
+  SmallVector<object::BuildID> FoundBinaryIDs;
+  for (const auto &File : llvm::enumerate(ObjectFilenames)) {
+    if (Error E =
+            loadFromFile(File.value(), GetArch(File.index()), CompilationDir,
+                         *ProfileReader, *Coverage, DataFound, &FoundBinaryIDs))
+      return E;
+  }
+
+  if (BIDFetcher) {
+    const auto &Compare = [](object::BuildIDRef A, object::BuildIDRef B) {
+      return StringRef(reinterpret_cast<const char *>(A.data()), A.size()) <
+             StringRef(reinterpret_cast<const char *>(B.data()), B.size());
+    };
+    std::vector<object::BuildIDRef> ProfileBinaryIDs;
+    if (Error E = ProfileReader->readBinaryIds(ProfileBinaryIDs))
+      return createFileError(ProfileFilename, std::move(E));
+    llvm::sort(ProfileBinaryIDs, Compare);
+    std::unique(ProfileBinaryIDs.begin(), ProfileBinaryIDs.end(), Compare);
+
+    SmallVector<object::BuildIDRef> BinaryIDsToFetch;
+    if (!ProfileBinaryIDs.empty()) {
+      llvm::sort(FoundBinaryIDs, Compare);
+      std::unique(FoundBinaryIDs.begin(), FoundBinaryIDs.end(), Compare);
+      std::set_difference(
+          ProfileBinaryIDs.begin(), ProfileBinaryIDs.end(),
+          FoundBinaryIDs.begin(), FoundBinaryIDs.end(),
+          std::inserter(BinaryIDsToFetch, BinaryIDsToFetch.end()), Compare);
+    }
+
+    for (object::BuildIDRef BinaryID : BinaryIDsToFetch) {
+      Optional<std::string> PathOpt = BIDFetcher->fetch(BinaryID);
+      if (!PathOpt)
+        continue;
+      std::string Path = std::move(*PathOpt);
+      StringRef Arch = Arches.size() == 1 ? Arches.front() : StringRef();
+      if (Error E = loadFromFile(Path, Arch, CompilationDir, *ProfileReader,
+                                 *Coverage, DataFound))
+        return E;
+    }
+  }
+
   // If no readers were created, either no objects were provided or none of them
   // had coverage data. Return an error in the latter case.
   if (!DataFound && !ObjectFilenames.empty())
     return createFileError(
         join(ObjectFilenames.begin(), ObjectFilenames.end(), ", "),
         make_error<CoverageMapError>(coveragemap_error::no_data_found));
   return std::move(Coverage);
 }
 
 namespace {
 

llvm/lib/ProfileData/Coverage/CoverageMappingReader.cpp

static Expected<std::vector<SectionRef>> lookupSections(ObjectFile &OF,
   }
   if (Sections.empty())
     return make_error<CoverageMapError>(coveragemap_error::no_data_found);
   return Sections;
 }
 
 static Expected<std::unique_ptr<BinaryCoverageReader>>
 loadBinaryFormat(std::unique_ptr<Binary> Bin, StringRef Arch,
-                 StringRef CompilationDir = "") {
+                 StringRef CompilationDir = "",
+                 Optional<object::BuildIDRef> *BinaryID = nullptr) {
   std::unique_ptr<ObjectFile> OF;
   if (auto *Universal = dyn_cast<MachOUniversalBinary>(Bin.get())) {
     // If we have a universal binary, try to look up the object for the
     // appropriate architecture.
     auto ObjectFileOrErr = Universal->getMachOObjectForArch(Arch);
     if (!ObjectFileOrErr)
       return ObjectFileOrErr.takeError();
     OF = std::move(ObjectFileOrErr.get());
for (SectionRef Section : *CoverageRecordsSections) {
                           (uintptr_t)FuncRecordsBuffer,
                       '\0');
     }
     assert(FuncRecordsBuffer == WritableBuffer->getBufferEnd() &&
            "consistent init");
     FuncRecords = std::move(WritableBuffer);
   }
 
+  if (BinaryID)
+    *BinaryID = getBuildID(OF.get());
+
   return BinaryCoverageReader::createCoverageReaderFromBuffer(
       CoverageMapping, std::move(FuncRecords), std::move(ProfileNames),
       BytesInAddress, Endian, CompilationDir);
 }
 
 /// Determine whether \p Arch is invalid or empty, given \p Bin.
 static bool isArchSpecifierInvalidOrMissing(Binary *Bin, StringRef Arch) {
   // If we have a universal binary and Arch doesn't identify any of its slices,
   // it's user error.
   if (auto *Universal = dyn_cast<MachOUniversalBinary>(Bin)) {
     for (auto &ObjForArch : Universal->objects())
       if (Arch == ObjForArch.getArchFlagName())
         return false;
     return true;
   }
   return false;
 }
 
 Expected<std::vector<std::unique_ptr<BinaryCoverageReader>>>
 BinaryCoverageReader::create(
     MemoryBufferRef ObjectBuffer, StringRef Arch,
     SmallVectorImpl<std::unique_ptr<MemoryBuffer>> &ObjectFileBuffers,
-    StringRef CompilationDir) {
+    StringRef CompilationDir,
+    SmallVectorImpl<object::BuildIDRef> *BinaryIDs) {
   std::vector<std::unique_ptr<BinaryCoverageReader>> Readers;
 
   if (ObjectBuffer.getBuffer().startswith(TestingFormatMagic)) {
     // This is a special format used for testing.
     auto ReaderOrErr =
         loadTestingFormat(ObjectBuffer.getBuffer(), CompilationDir);
     if (!ReaderOrErr)
       return ReaderOrErr.takeError();
for (auto &ObjForArch : Universal->objects()) {
       if (!ArchiveOrErr) {
         // If this is not an archive, try treating it as a regular object.
         consumeError(ArchiveOrErr.takeError());
         break;
       }
 
       return BinaryCoverageReader::create(
           ArchiveOrErr.get()->getMemoryBufferRef(), Arch, ObjectFileBuffers,
-          CompilationDir);
+          CompilationDir, BinaryIDs);
     }
   }
 
   // Load coverage out of archive members.
   if (auto *Ar = dyn_cast<Archive>(Bin.get())) {
     Error Err = Error::success();
     for (auto &Child : Ar->children(Err)) {
       Expected<MemoryBufferRef> ChildBufOrErr = Child.getMemoryBufferRef();
       if (!ChildBufOrErr)
         return ChildBufOrErr.takeError();
 
       auto ChildReadersOrErr = BinaryCoverageReader::create(
-          ChildBufOrErr.get(), Arch, ObjectFileBuffers, CompilationDir);
+          ChildBufOrErr.get(), Arch, ObjectFileBuffers, CompilationDir,
+          BinaryIDs);
       if (!ChildReadersOrErr)
         return ChildReadersOrErr.takeError();
       for (auto &Reader : ChildReadersOrErr.get())
         Readers.push_back(std::move(Reader));
     }
     if (Err)
       return std::move(Err);
 
     // Thin archives reference object files outside of the archive file, i.e.
     // files which reside in memory not owned by the caller. Transfer ownership
     // to the caller.
     if (Ar->isThin())
       for (auto &Buffer : Ar->takeThinBuffers())
         ObjectFileBuffers.push_back(std::move(Buffer));
 
     return std::move(Readers);
   }
 
-  auto ReaderOrErr = loadBinaryFormat(std::move(Bin), Arch, CompilationDir);
+  Optional<object::BuildIDRef> BinaryID;
+  auto ReaderOrErr = loadBinaryFormat(std::move(Bin), Arch, CompilationDir,
+                                      BinaryIDs ? &BinaryID : nullptr);
   if (!ReaderOrErr)
     return ReaderOrErr.takeError();
   Readers.push_back(std::move(ReaderOrErr.get()));
+  if (BinaryID)
+    BinaryIDs->push_back(*BinaryID);
   return std::move(Readers);
 }
 
 Error BinaryCoverageReader::readNextRecord(CoverageMappingRecord &Record) {
   if (CurrentRecord >= MappingRecords.size())
     return make_error<CoverageMapError>(coveragemap_error::eof);
 
   FunctionsFilenames.clear();

llvm/tools/llvm-cov/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS core support object coverage profiledata)
 
 add_llvm_tool(llvm-cov
   llvm-cov.cpp
   gcov.cpp
   CodeCoverage.cpp
   CoverageExporterJson.cpp
   CoverageExporterLcov.cpp
   CoverageFilters.cpp
   CoverageReport.cpp
   CoverageSummaryInfo.cpp
   SourceCoverageView.cpp
   SourceCoverageViewHTML.cpp
   SourceCoverageViewText.cpp
   TestingSupport.cpp
   )
+
+target_link_libraries(llvm-cov PRIVATE LLVMDebuginfod)

llvm/tools/llvm-cov/CodeCoverage.cpp

 #include "CoverageReport.h"
 #include "CoverageSummaryInfo.h"
 #include "CoverageViewOptions.h"
 #include "RenderingSupport.h"
 #include "SourceCoverageView.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/Triple.h"
+#include "llvm/Debuginfod/BuildIDFetcher.h"
+#include "llvm/Debuginfod/Debuginfod.h"
+#include "llvm/Debuginfod/HTTPClient.h"
+#include "llvm/Object/BuildID.h"
 #include "llvm/ProfileData/Coverage/CoverageMapping.h"
 #include "llvm/ProfileData/InstrProfReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Process.h"
private:
 
   /// A container for input source file buffers.
   std::mutex LoadedSourceFilesLock;
   std::vector<std::pair<std::string, std::unique_ptr<MemoryBuffer>>>
       LoadedSourceFiles;
 
   /// Allowlist from -name-allowlist to be used for filtering.
   std::unique_ptr<SpecialCaseList> NameAllowlist;
+
+  std::unique_ptr<object::BuildIDFetcher> BIDFetcher;
 };
 }
 
 static std::string getErrorString(const Twine &Message, StringRef Whence,
                                   bool Warning) {
   std::string Str = (Warning ? "warning" : "error");
   Str += ": ";
   if (!Whence.empty())

 
 std::unique_ptr<CoverageMapping> CodeCoverageTool::load() {
   for (StringRef ObjectFilename : ObjectFilenames)
     if (modifiedTimeGT(ObjectFilename, PGOFilename))
       warning("profile data may be out of date - object is newer",
               ObjectFilename);
   auto CoverageOrErr =
       CoverageMapping::load(ObjectFilenames, PGOFilename, CoverageArches,
-                            ViewOpts.CompilationDirectory);
+                            ViewOpts.CompilationDirectory, BIDFetcher.get());
   if (Error E = CoverageOrErr.takeError()) {
     error("Failed to load coverage: " + toString(std::move(E)));
     return nullptr;
   }
   auto Coverage = std::move(CoverageOrErr.get());
   unsigned Mismatched = Coverage->getMismatchedCount();
   if (Mismatched) {
     warning(Twine(Mismatched) + " functions have mismatched data");
cl::opt<std::string, true> PGOFilename(
           "File with the profile data obtained after an instrumented run"));
 
   cl::list<std::string> Arches(
       "arch", cl::desc("architectures of the coverage mapping binaries"));
 
   cl::opt<bool> DebugDump("dump", cl::Optional,
                           cl::desc("Show internal debug dump"));
 
+  cl::list<std::string> DebugFileDirectory(
+      "debug-file-directory",
+      cl::desc("Directories to search for object files by build ID"));
+  cl::opt<bool> Debuginfod(
+      "debuginfod", cl::ZeroOrMore,
+      cl::desc("Use debuginfod to look up object files from profile"),
+      cl::init(canUseDebuginfod()));
+
   cl::opt<CoverageViewOptions::OutputFormat> Format(
       "format", cl::desc("Output format for line-based coverage reports"),
       cl::values(clEnumValN(CoverageViewOptions::OutputFormat::Text, "text",
                             "Text output"),
                  clEnumValN(CoverageViewOptions::OutputFormat::HTML, "html",
                             "HTML output"),
                  clEnumValN(CoverageViewOptions::OutputFormat::Lcov, "lcov",
                             "lcov tracefile output")),
int CodeCoverageTool::run(Command Cmd, int argc, const char **argv) {
 
   cl::opt<std::string> CompilationDirectory(
       "compilation-dir", cl::init(""),
       cl::desc("Directory used as a base for relative coverage mapping paths"));
 
   auto commandLineParser = [&, this](int argc, const char **argv) -> int {
     cl::ParseCommandLineOptions(argc, argv, "LLVM code coverage tool\n");
     ViewOpts.Debug = DebugDump;
+    if (Debuginfod) {
+      HTTPClient::initialize();
+      BIDFetcher = std::make_unique<DebuginfodFetcher>(DebugFileDirectory);
+    } else {
+      BIDFetcher = std::make_unique<object::BuildIDFetcher>(DebugFileDirectory);
+    }
 
     if (!CovFilename.empty())
       ObjectFilenames.emplace_back(CovFilename);
     for (const std::string &Filename : CovFilenames)
       ObjectFilenames.emplace_back(Filename);
-    if (ObjectFilenames.empty()) {
+    if (ObjectFilenames.empty() && !Debuginfod && DebugFileDirectory.empty()) {
       errs() << "No filenames specified!\n";
       ::exit(1);
     }
 
     if (DebugDumpCollectedObjects) {
       for (StringRef OF : ObjectFilenames)
         outs() << OF << '\n';
       ::exit(0);
auto commandLineParser = [&, this](int argc, const char **argv) -> int {
     if (!Arches.empty()) {
       for (const std::string &Arch : Arches) {
         if (Triple(Arch).getArch() == llvm::Triple::ArchType::UnknownArch) {
           error("Unknown architecture: " + Arch);
           return 1;
         }
         CoverageArches.emplace_back(Arch);
       }
-      if (CoverageArches.size() == 1)
-        CoverageArches.insert(CoverageArches.end(), ObjectFilenames.size() - 1,
-                              CoverageArches[0]);
-      if (CoverageArches.size() != ObjectFilenames.size()) {
+      if (CoverageArches.size() != 1 &&
+          CoverageArches.size() != ObjectFilenames.size()) {
         error("Number of architectures doesn't match the number of objects");
         return 1;
       }
     }
 
     // IgnoreFilenameFilters are applied even when InputSourceFiles specified.
     for (const std::string &File : InputSourceFiles)
       collectPaths(File);

llvm/tools/llvm-objdump/llvm-objdump.cpp

if (!Is("otool")) {
       outs() << '\n';
       TargetRegistry::printRegisteredTargetsForVersion(outs());
     }
     return 0;
   }
 
   // Initialize debuginfod.
   const bool ShouldUseDebuginfodByDefault =
-      InputArgs.hasArg(OBJDUMP_build_id) ||
-      (HTTPClient::isAvailable() &&
-       !ExitOnErr(getDefaultDebuginfodUrls()).empty());
+      InputArgs.hasArg(OBJDUMP_build_id) || canUseDebuginfod();
   std::vector<std::string> DebugFileDirectories =
       InputArgs.getAllArgValues(OBJDUMP_debug_file_directory);
   if (InputArgs.hasFlag(OBJDUMP_debuginfod, OBJDUMP_no_debuginfod,
                         ShouldUseDebuginfodByDefault)) {
     HTTPClient::initialize();
     BIDFetcher =
         std::make_unique<DebuginfodFetcher>(std::move(DebugFileDirectories));
   } else {

llvm/tools/llvm-symbolizer/llvm-symbolizer.cpp

for (const opt::Arg *A : Args.filtered(OPT_dsym_hint_EQ)) {
     } else {
       errs() << "Warning: invalid dSYM hint: \"" << Hint
              << "\" (must have the '.dSYM' extension).\n";
     }
   }
 
   LLVMSymbolizer Symbolizer(Opts);
 
-  // A debuginfod lookup could succeed if a HTTP client is available and at
-  // least one backing URL is configured.
-  bool ShouldUseDebuginfodByDefault =
-      HTTPClient::isAvailable() &&
-      !ExitOnErr(getDefaultDebuginfodUrls()).empty();
-  if (Args.hasFlag(OPT_debuginfod, OPT_no_debuginfod,
-                   ShouldUseDebuginfodByDefault))
+  if (Args.hasFlag(OPT_debuginfod, OPT_no_debuginfod, canUseDebuginfod()))
     enableDebuginfod(Symbolizer, Args);
 
   if (Args.hasArg(OPT_filter_markup)) {
     filterMarkup(Args, Symbolizer);
     return 0;
   }
 
   auto Style = IsAddr2Line ? OutputStyle::GNU : OutputStyle::LLVM;