[llvm] [Debuginfo] Add llvm-debuginfod-find tool and end-to-end-tests.

[noajshu]

Summary

This implements the llvm-debuginfod-find tool, which wraps the Debuginfod library (D112758) to query debuginfod servers for artifacts according to the specification.

[noajshu]

Clean up unrelated changes.

[noajshu]

Use latest debuginfod client library (D112758).
Also condition debuginfod-find lit test execution on availability of libcurl.

[noajshu]

Build llvm-debuginfod-find even if curl is disabled. HTTP fetch will be a no-op.

[noajshu]

Add more help info for client tool and update lit tests.

[labath]

llvm/test/tools/llvm-debuginfod-find/debuginfod-find.py

2–3 ↗

(On Diff #390209)

you don't have to chain the commands with &&. The script is automatically aborted when any command fails.

This would be much clearer if you just put every command on a separate line instead of peppering it with && and \'s.

36 ↗

(On Diff #390209)

It may be helpful to print what it was.

38 ↗

(On Diff #390209)

I am somewhat surprised by the need to peek inside the cache directory. What is the actual interface of the tool, or how is the user expected to know where to locate the file that it has downloaded? It seems like the elfutils debuginfod-find prints the path to the file on stdout. Should this do the same?

(I mean, it may be useful to hardcode the cache path so you can verify the file has been downloaded to the right place, but i don't think that every user should need to know the precise hash algorithm just to be able to locate the downloaded file.)

41 ↗

(On Diff #390209)

or even "does not", for maximum formality.

45–46 ↗

(On Diff #390209)

Instead of doing the checks here, it may be better to just print the relevant data (exit code, file contents, whatever) and then verify that through an external FileCheck command.

The reason I'm proposing this is to improve the error message in case this fails. Like, imagine this test fails on a bot somewhere. "file contents does not match" will not tell you much, as it does not even say which file it refers to. OTOH, with FileCheck you get the "i expected <this> but got <that>" functionality for free. (You can print repr(file.read()) and use FileCheck --strict-whitespace, if you want to catch any issues with newlines or unprintable characters.)

[noajshu]

Add --dump option to llvm-debuginfod-find and simplify lit tests using FileCheck.

[noajshu]

Print exit code of process when lit test fails.

[noajshu]

Hi @labath thanks for all of your comments on debuginfod-find.py. I agree it is cleaner to use FileCheck for the lit tests, and I rewrote these tests into debuginfod.test incorporating your suggestions and also some feedback on the tests in D113717.

Regarding the output behavior, Elfutils debuginfod-find prints the path of the fetched/cached artifact file to stdout. By default, llvm-debuginfod-find does the same. I've also added a command line flag llvm-debuginfod-find --dump which dumps the contents of the artifact to stdout instead of its file path.
If you can only print the path to the cached file, you have to read the stdout and then use that path as the argument to another program (e.g. cat). This is easy to do in a full shell using xargs, or a subshell or variable, but inconvenient in LLVM Lit scripts which use a sequence of LLVM commands chained by pipes.

Does this output behavior of llvm-debuginfod-find and the --dump flag seem reasonable to you? If it's more clear we could change the name to --print-contents or something else. Thanks!

[labath]

In D112759#3172488, @noajshu wrote:

Regarding the output behavior, Elfutils debuginfod-find prints the path of the fetched/cached artifact file to stdout. By default, llvm-debuginfod-find does the same. I've also added a command line flag llvm-debuginfod-find --dump which dumps the contents of the artifact to stdout instead of its file path.
If you can only print the path to the cached file, you have to read the stdout and then use that path as the argument to another program (e.g. cat). This is easy to do in a full shell using xargs, or a subshell or variable, but inconvenient in LLVM Lit scripts which use a sequence of LLVM commands chained by pipes.

Does this output behavior of llvm-debuginfod-find and the --dump flag seem reasonable to you? If it's more clear we could change the name to --print-contents or something else. Thanks!

Yeah, this isn't easy to do from a lit test. My idea was (since you have a python script anyway) to implement the read-the-filename-and-dump-contents functionality in python. *However*, having the ability for the tool to dump the file contents seems like it could potentially be useful as well, so I have no problem with doing it this way. I have no opinion on the option name.

llvm/tools/llvm-debuginfod-find/llvm-debuginfod-find.cpp

32

Are you sure you're (still) using all of these headers?

[noajshu]

Remove unnecessary header includes, and add HTTPClient::initialize() call to llvm-debuginfod-find.cpp.

[noajshu]

llvm/tools/llvm-debuginfod-find/llvm-debuginfod-find.cpp

32

No I'm not, thanks for pointing this out!

[noajshu]

Broke this build: https://lab.llvm.org/buildbot/#/builders/91/builds/387

[thakis]

llvm/test/CMakeLists.txt

81

you have to add llvm-debuginfod-find here too so that it's guaranteed the executable is build before tests are run when target check-llvm is built

[noajshu]

Add llvm-debuginfod-find to list of required binaries for lit tests and update llvm-debuginfod-find test script to be more compatible with older pythons. (see buildbot failure due to python usage below 3.6: https://lab.llvm.org/buildbot/#/builders/164/builds/6751/steps/6/logs/FAIL__LLVM__debuginfod_test)

[noajshu]

It seems like some buildbots use python that is below 3.6 (see buildbot failure due to missing python3.6 features)
This seems to contradict LLVM's testing infra docs which say to use python3.6 or later.
Either way I've backported most parts of the python testing script to be more compatible with earlier python.

llvm/test/CMakeLists.txt

81

Thanks so much!

[noajshu]

Hide unrelated options printed by llvm-debuginfod-find.

[uabelho]

Hi,

I get the following error when I run tests (check-all) on this commit:

FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (32121 of 80857)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --executable abcdef' |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --executable abcdef |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 2

Command Output (stderr):
--
  File "/repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test", line 55
    env['DEBUGINFOD_URLS'] = f'http://localhost:%s' % port
                                                  ^
SyntaxError: invalid syntax
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE

--

Any idea what is going on?

[noajshu]

Hi Mikael,

Could you check which version of python you are running? (python --version)

Thanks,
Noah

[noajshu]

@uabelho does D115814 fix this for you? I think it might be caused by the python3 f-string.

[vitalybuka]

I have one like this

[1/2] Running the LLVM regression tests
FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (15361 of 46561)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --executable abcdef' |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --executable abcdef |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 1

Command Output (stderr):
--
127.0.0.1 - - [15/Dec/2021 12:58:47] "GET /buildid/abcdef/executable HTTP/1.1" 200 -
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --executable abcdef
 #0 0x0000000000252043 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x252043)
 #1 0x000000000024ff7e llvm::sys::RunSignalHandlers() (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x24ff7e)
 #2 0x000000000025264f SignalHandler(int) Signals.cpp:0:0
 #3 0x00007fd255a7a8e0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x138e0)
 #4 0x00007fd2554c3d3c free ./malloc/malloc.c:3124:7
 #5 0x000000000021dac3 llvm::HTTPClient::perform(llvm::HTTPRequest const&, llvm::HTTPResponseHandler&) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21dac3)
 #6 0x000000000021dc84 llvm::HTTPClient::get(llvm::StringRef) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21dc84)
 #7 0x000000000021c471 llvm::getCachedOrDownloadArtifact[abi:cxx11](llvm::StringRef, llvm::StringRef, llvm::StringRef, llvm::ArrayRef<llvm::StringRef>, std::chrono::duration<long, std::ratio<1l, 1000l> >) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21c471)
 #8 0x000000000021bb2c llvm::getCachedOrDownloadArtifact[abi:cxx11](llvm::StringRef, llvm::StringRef) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21bb2c)
 #9 0x000000000021be0e llvm::getCachedOrDownloadExecutable[abi:cxx11](llvm::ArrayRef<unsigned char>) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21be0e)
#10 0x0000000000219b68 main (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x219b68)
#11 0x00007fd255460e4a __libc_start_main ./csu/../csu/libc-start.c:314:16
#12 0x00000000002195da _start (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x2195da)
/usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test:15:15: error: EXECUTABLE: expected string not found in input
# EXECUTABLE: fake_executable
              ^
<stdin>:1:1: note: scanning from here
nontrivial return code -11
^
<stdin>:1:8: note: possible intended match here
nontrivial return code -11
       ^

Input file: <stdin>
Check file: /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: nontrivial return code -11 
check:15'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:15'1            ?                    possible intended match
>>>>>>

--

********************
********************
Failed Tests (1):
  LLVM :: tools/llvm-debuginfod-find/debuginfod.test

[noajshu]

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

[uabelho]

In D112759#3195363, @noajshu wrote:

Hi Mikael,

Could you check which version of python you are running? (python --version)

Thanks,
Noah

Hi,

We're using python 3.8:

-- Found Python3: /app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 (found suitable version "3.8.0", minimum required is "3.6") found components: Interpreter

and

> app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 --version
Python 3.8.0

However, I noticed now that it passes if I do a clean build, so I have no idea what actually happened when I got the failure yesterday.
Could there be some problem so binaries are not rebuilt as they should?

Apologies for sending you on a wild-goose chase. :/

The test passes for me also with https://reviews.llvm.org/D115814 applied, but as I said, that patch doesn't seem to be needed by us at least.

[noajshu]

In D112759#3196640, @uabelho wrote:

In D112759#3195363, @noajshu wrote:

Hi Mikael,

Could you check which version of python you are running? (python --version)

Thanks,
Noah

Hi,

We're using python 3.8:

-- Found Python3: /app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 (found suitable version "3.8.0", minimum required is "3.6") found components: Interpreter

and

> app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 --version
Python 3.8.0

However, I noticed now that it passes if I do a clean build, so I have no idea what actually happened when I got the failure yesterday.
Could there be some problem so binaries are not rebuilt as they should?

Apologies for sending you on a wild-goose chase. :/

The test passes for me also with https://reviews.llvm.org/D115814 applied, but as I said, that patch doesn't seem to be needed by us at least.

@uabelho thanks so much for confirming it works after you clean your build folder, and no worries about the flake.
Although the cmake caching behavior is mostly correct, I have seen at least one exception (e.g. when implicit tools are added).
Glad to hear it now passes the test, and thanks for running it!

[vitalybuka]

In D112759#3195791, @noajshu wrote:

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

I am on 3aece7995cb63a536a15df35ebe107d04a8acc86 and my llvm/lib/Debuginfod/HTTPClient.cpp is exactly as 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44
git diff 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44 -- llvm/lib/Debuginfod/HTTPClient.cpp
no output

[vitalybuka]

In D112759#3198133, @vitalybuka wrote:

In D112759#3195791, @noajshu wrote:

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

I am on 3aece7995cb63a536a15df35ebe107d04a8acc86 and my llvm/lib/Debuginfod/HTTPClient.cpp is exactly as 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44
git diff 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44 -- llvm/lib/Debuginfod/HTTPClient.cpp
no output

I just did full clean rebuild and it's gone.

[noajshu]

In D112759#3198180, @vitalybuka wrote:

In D112759#3198133, @vitalybuka wrote:

In D112759#3195791, @noajshu wrote:

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

I am on 3aece7995cb63a536a15df35ebe107d04a8acc86 and my llvm/lib/Debuginfod/HTTPClient.cpp is exactly as 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44
git diff 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44 -- llvm/lib/Debuginfod/HTTPClient.cpp
no output

I just did full clean rebuild and it's gone.

I'm glad the issue has disappeared. Although it's mysterious the build system caching didn't work as intended.

[lebedev.ri]

Hello.
The test added here is failing for me (and i think i'm not the only one) on debian:

[ 99% 3987/3988][ 98% 0:10:34 + 0:00:08] Running the LLVM regression tests
FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (913 of 46565)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 1

Command Output (stderr):
--
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/abcdef/executable HTTP/1.1" 200 -
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef
127.0.0.1 - - [20/Dec/2021 17:01:26] code 404, message File not found
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/811bf0bfb413c00e/debuginfo HTTP/1.1" 404 -
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamEi+0x23)[0x7f20d9f65353]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys17RunSignalHandlersEv+0xf0)[0x7f20d9f63010]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(+0x26585a)[0x7f20d9f6585a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13200)[0x7f20d9cc8200]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x25)[0x7f20d980b945]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestERNS_19HTTPResponseHandlerE+0x262)[0x7f20d9f8a042]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestE+0x41)[0x7f20d9f89d41]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient3getENS_9StringRefE+0x31)[0x7f20d9f8a2a1]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_S0_NS_8ArrayRefIS0_EENSt6chrono8durationIlSt5ratioILl1ELl1000EEEE+0x475)[0x7f20d9f88135]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_+0x154)[0x7f20d9f876a4]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm29getCachedOrDownloadExecutableB5cxx11ENS_8ArrayRefIhEE+0x1c2)[0x7f20d9f87a02]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x20558e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f20d97a87ed]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x2048ea]
/repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test:15:15: error: EXECUTABLE: expected string not found in input
# EXECUTABLE: fake_executable
              ^
<stdin>:1:1: note: scanning from here
nontrivial return code -11
^
<stdin>:1:8: note: possible intended match here
nontrivial return code -11
       ^

Input file: <stdin>
Check file: /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: nontrivial return code -11 
check:15'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:15'1            ?                    possible intended match
>>>>>>

--

********************
********************
Failed Tests (1):
  LLVM :: tools/llvm-debuginfod-find/debuginfod.test


Testing Time: 277.28s
  Unsupported      :   344
  Passed           : 46059
  Expectedly Failed:   161
  Failed           :     1
FAILED: test/CMakeFiles/check-llvm /builddirs/llvm-project/build-Clang13/test/CMakeFiles/check-llvm 
cd /builddirs/llvm-project/build-Clang13/test && /usr/bin/python3.9 /builddirs/llvm-project/build-Clang13/./bin/llvm-lit -sv /builddirs/llvm-project/build-Clang13/test
ninja: build stopped: subcommand failed.

Could you please resolve the problem?

[noajshu]

In D112759#3202690, @lebedev.ri wrote:

Hello.
The test added here is failing for me (and i think i'm not the only one) on debian:

[ 99% 3987/3988][ 98% 0:10:34 + 0:00:08] Running the LLVM regression tests
FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (913 of 46565)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 1

Command Output (stderr):
--
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/abcdef/executable HTTP/1.1" 200 -
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef
127.0.0.1 - - [20/Dec/2021 17:01:26] code 404, message File not found
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/811bf0bfb413c00e/debuginfo HTTP/1.1" 404 -
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamEi+0x23)[0x7f20d9f65353]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys17RunSignalHandlersEv+0xf0)[0x7f20d9f63010]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(+0x26585a)[0x7f20d9f6585a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13200)[0x7f20d9cc8200]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x25)[0x7f20d980b945]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestERNS_19HTTPResponseHandlerE+0x262)[0x7f20d9f8a042]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestE+0x41)[0x7f20d9f89d41]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient3getENS_9StringRefE+0x31)[0x7f20d9f8a2a1]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_S0_NS_8ArrayRefIS0_EENSt6chrono8durationIlSt5ratioILl1ELl1000EEEE+0x475)[0x7f20d9f88135]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_+0x154)[0x7f20d9f876a4]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm29getCachedOrDownloadExecutableB5cxx11ENS_8ArrayRefIhEE+0x1c2)[0x7f20d9f87a02]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x20558e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f20d97a87ed]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x2048ea]
/repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test:15:15: error: EXECUTABLE: expected string not found in input
# EXECUTABLE: fake_executable
              ^
<stdin>:1:1: note: scanning from here
nontrivial return code -11
^
<stdin>:1:8: note: possible intended match here
nontrivial return code -11
       ^

Input file: <stdin>
Check file: /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: nontrivial return code -11 
check:15'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:15'1            ?                    possible intended match
>>>>>>

--

********************
********************
Failed Tests (1):
  LLVM :: tools/llvm-debuginfod-find/debuginfod.test


Testing Time: 277.28s
  Unsupported      :   344
  Passed           : 46059
  Expectedly Failed:   161
  Failed           :     1
FAILED: test/CMakeFiles/check-llvm /builddirs/llvm-project/build-Clang13/test/CMakeFiles/check-llvm 
cd /builddirs/llvm-project/build-Clang13/test && /usr/bin/python3.9 /builddirs/llvm-project/build-Clang13/./bin/llvm-lit -sv /builddirs/llvm-project/build-Clang13/test
ninja: build stopped: subcommand failed.

Could you please resolve the problem?

Hi @lebedev.ri thanks for sharing this error. Based on other anecdotal reports I expect that clearing and regenerating your build folder will fix the issue. It would be helpful if you could please save a backup copy of the build folder first, so that if clearing and re-generating works, we can diagnose and fix the underlying problem with the build system caching.

Then, if that clear-and-regenerate works, would be possibly be able to share your build folder? If you just compress it and upload somewhere I can take a look.

[lebedev.ri]

In D112759#3203362, @noajshu wrote:

In D112759#3202690, @lebedev.ri wrote:

Hello.
The test added here is failing for me (and i think i'm not the only one) on debian:

<...>

Could you please resolve the problem?

Hi @lebedev.ri thanks for sharing this error. Based on other anecdotal reports I expect that clearing and regenerating your build folder will fix the issue. It would be helpful if you could please save a backup copy of the build folder first, so that if clearing and re-generating works, we can diagnose and fix the underlying problem with the build system caching.

Then, if that clear-and-regenerate works, would be possibly be able to share your build folder? If you just compress it and upload somewhere I can take a look.

Ok, thanks, removing build directory helped. CMake caching indeed breaks sometimes with things like this, so i don't think it's worthwhile trying to fix this then.

[Hahnfeld]

The test still doesn't work for me on Python 3.6 because ThreadingHTTPServer was apparently introduced with Python 3.7.

[noajshu]

In D112759#3271718, @Hahnfeld wrote:

The test still doesn't work for me on Python 3.6 because ThreadingHTTPServer was apparently introduced with Python 3.7.

Thanks @Hahnfeld for pointing this out!
I think I can make it python 3.6 compatible soon.
Also, I'll add some lit tests to the WIP debuginfod server patch (D114846) to ping our own (real) debuginfod server with llvm-debuginfod-find.

[MaskRay]

llvm/test/tools/llvm-debuginfod-find/debuginfod.test

5

This should use %python lit substitution. python may mean python2 on some systems. The test needs python3.

I fixed it in 42f9ca55dd487580cfd32d4b803315d789442442

[noajshu]

In D112759#3271718, @Hahnfeld wrote:

The test still doesn't work for me on Python 3.6 because ThreadingHTTPServer was apparently introduced with Python 3.7.

Sorry again about this!
This will be fixed by D119096.

[Hahnfeld]

In D112759#3299743, @noajshu wrote:

In D112759#3271718, @Hahnfeld wrote:

The test still doesn't work for me on Python 3.6 because ThreadingHTTPServer was apparently introduced with Python 3.7.

Sorry again about this!
This will be fixed by D119096.

This doesn't seem to be going anywhere, the test is still failing...

llvm/test/CMakeLists.txt

 llvm_canonicalize_cmake_booleans(
   BUILD_SHARED_LIBS
   LLVM_HAVE_LIBXAR
   HAVE_OCAMLOPT
   HAVE_OCAML_OUNIT
   LLVM_ENABLE_DIA_SDK
   LLVM_ENABLE_FFI
   LLVM_ENABLE_THREADS
+  LLVM_ENABLE_CURL
   LLVM_ENABLE_ZLIB
   LLVM_ENABLE_LIBXML2
   LLVM_INCLUDE_GO_TESTS
   LLVM_LINK_LLVM_DYLIB
   LLVM_TOOL_LTO_BUILD
   LLVM_USE_INTEL_JITEVENTS
   LLVM_BUILD_EXAMPLES
   LLVM_ENABLE_PLUGINS
set(LLVM_TEST_DEPENDS
           llvm-cat
           llvm-cfi-verify
           llvm-config
           llvm-cov
           llvm-cvtres
           llvm-cxxdump
           llvm-cxxfilt
           llvm-cxxmap
+          llvm-debuginfod-find
           llvm-diff
           llvm-dis
           llvm-dlltool
           dsymutil
           llvm-dwarfdump
           llvm-dwp
           llvm-exegesis
           llvm-extract
           llvm-gsymutil
           llvm-isel-fuzzer
           llvm-ifs
           llvm-install-name-tool
           llvm-jitlink

llvm/test/lit.cfg.py

tools = [
     ToolSubst('%llvm-bitcode-strip', FindTool('llvm-bitcode-strip')),
     ToolSubst('%split-file', FindTool('split-file')),
 ]
 
 # FIXME: Why do we have both `lli` and `%lli` that do slightly different things?
 tools.extend([
     'dsymutil', 'lli', 'lli-child-target', 'llvm-ar', 'llvm-as',
     'llvm-addr2line', 'llvm-bcanalyzer', 'llvm-bitcode-strip', 'llvm-config',
-    'llvm-cov', 'llvm-cxxdump', 'llvm-cvtres', 'llvm-diff', 'llvm-dis',
-    'llvm-dwarfdump', 'llvm-dlltool', 'llvm-exegesis', 'llvm-extract',
-    'llvm-isel-fuzzer', 'llvm-ifs',
+    'llvm-cov', 'llvm-cxxdump', 'llvm-cvtres', 'llvm-debuginfod-find',
+    'llvm-diff', 'llvm-dis', 'llvm-dwarfdump', 'llvm-dlltool', 'llvm-exegesis',
+    'llvm-extract', 'llvm-isel-fuzzer', 'llvm-ifs',
     'llvm-install-name-tool', 'llvm-jitlink', 'llvm-opt-fuzzer', 'llvm-lib',
     'llvm-link', 'llvm-lto', 'llvm-lto2', 'llvm-mc', 'llvm-mca',
     'llvm-modextract', 'llvm-nm', 'llvm-objcopy', 'llvm-objdump', 'llvm-otool',
     'llvm-pdbutil', 'llvm-profdata', 'llvm-profgen', 'llvm-ranlib', 'llvm-rc', 'llvm-readelf',
     'llvm-readobj', 'llvm-rtdyld', 'llvm-sim', 'llvm-size', 'llvm-split',
     'llvm-stress', 'llvm-strings', 'llvm-strip', 'llvm-tblgen', 'llvm-tapi-diff',
     'llvm-undname', 'llvm-windres', 'llvm-c-test', 'llvm-cxxfilt',
     'llvm-xray', 'yaml2obj', 'obj2yaml', 'yaml-bench', 'verify-uselistorder',
if config.have_libxar:
     config.available_features.add('xar')
 
 if config.enable_threads:
     config.available_features.add('thread_support')
 
 if config.have_libxml2:
     config.available_features.add('libxml2')
 
+if config.have_curl:
+    config.available_features.add('curl')
+
 if config.have_opt_viewer_modules:
     config.available_features.add('have_opt_viewer_modules')
 
 if config.expensive_checks:
     config.available_features.add('expensive_checks')
 
 if "MemoryWithOrigins" in config.llvm_use_sanitizer:
     config.available_features.add('use_msan_with_origins')

llvm/test/lit.site.cfg.py.in

 config.host_cxx = "@HOST_CXX@"
 # Note: ldflags can contain double-quoted paths, so must use single quotes here.
 config.host_ldflags = '@HOST_LDFLAGS@'
 config.llvm_use_intel_jitevents = @LLVM_USE_INTEL_JITEVENTS@
 config.llvm_use_sanitizer = "@LLVM_USE_SANITIZER@"
 config.have_zlib = @LLVM_ENABLE_ZLIB@
 config.have_libxar = @LLVM_HAVE_LIBXAR@
 config.have_libxml2 = @LLVM_ENABLE_LIBXML2@
+config.have_curl = @LLVM_ENABLE_CURL@
 config.have_dia_sdk = @LLVM_ENABLE_DIA_SDK@
 config.enable_ffi = @LLVM_ENABLE_FFI@
 config.build_examples = @LLVM_BUILD_EXAMPLES@
 config.enable_threads = @LLVM_ENABLE_THREADS@
 config.build_shared_libs = @BUILD_SHARED_LIBS@
 config.link_llvm_dylib = @LLVM_LINK_LLVM_DYLIB@
 config.llvm_dylib_version = "@LLVM_VERSION_MAJOR@@LLVM_VERSION_SUFFIX@"
 config.llvm_host_triple = '@LLVM_HOST_TRIPLE@'

llvm/test/tools/llvm-debuginfod-find/Inputs/buildid/abcdef/debuginfo

+fake_debuginfo

llvm/test/tools/llvm-debuginfod-find/Inputs/buildid/abcdef/executable

+fake_executable

llvm/test/tools/llvm-debuginfod-find/Inputs/buildid/abcdef/source/directory/file.c

+int foo = 0;

llvm/test/tools/llvm-debuginfod-find/debuginfod.test

+# REQUIRES: curl
+# RUN: rm -rf %t
+# RUN: mkdir %t
+# # Query the python server for artifacts
+# RUN: DEBUGINFOD_CACHE_PATH=%t python %s --server-path %S/Inputs \
+# RUN:   --tool-cmd 'llvm-debuginfod-find --dump --executable abcdef' | \
+# RUN:   FileCheck %s --check-prefix=EXECUTABLE
+# RUN: DEBUGINFOD_CACHE_PATH=%t python %s --server-path %S/Inputs \
+# RUN:   --tool-cmd 'llvm-debuginfod-find --dump --source=/directory/file.c abcdef' | \
+# RUN:   FileCheck %s --check-prefix=SOURCE
+# RUN: DEBUGINFOD_CACHE_PATH=%t python %s --server-path %S/Inputs \
+# RUN:   --tool-cmd 'llvm-debuginfod-find --dump --debuginfo abcdef' | \
+# RUN:   FileCheck %s --check-prefix=DEBUGINFO
+
+# EXECUTABLE: fake_executable
+# SOURCE: int foo = 0;
+# DEBUGINFO: fake_debuginfo
+
+# # The artifacts should still be present in the cache without needing to query
+# # the server.
+# RUN: DEBUGINFOD_CACHE_PATH=%t llvm-debuginfod-find --dump --executable abcdef | \
+# RUN:   FileCheck %s --check-prefix=EXECUTABLE
+# RUN: DEBUGINFOD_CACHE_PATH=%t llvm-debuginfod-find --dump \
+# RUN:   --source=/directory/file.c abcdef | \
+# RUN:   FileCheck %s --check-prefix=SOURCE
+# RUN: DEBUGINFOD_CACHE_PATH=%t llvm-debuginfod-find --dump --debuginfo abcdef | \
+# RUN:   FileCheck %s --check-prefix=DEBUGINFO
+
+
+# This script is used to test the debuginfod client within a host tool.
+# It first stands up a Python HTTP static file server and then executes the tool.
+# This way the tool can make debuginfod HTTP requests to the static file server.
+import argparse
+import threading
+import http.server
+import functools
+import subprocess
+import sys
+import os
+
+
+# Serves files at the server_path, then runs the tool with specified args.
+# Sets the DEBUGINFOD_CACHE_PATH env var to point at the given cache_directory.
+# Sets the DEBUGINFOD_URLS env var to point at the local server.
+def test_tool(server_path, tool_args):
+    httpd = http.server.ThreadingHTTPServer(
+        ('',0), functools.partial(
+            http.server.SimpleHTTPRequestHandler,
+            directory=server_path))
+    port = httpd.server_port
+    thread = threading.Thread(target=httpd.serve_forever)
+    try:
+        thread.start()
+        env = os.environ
+        env['DEBUGINFOD_URLS'] = f'http://localhost:%s' % port
+        process = subprocess.Popen(
+            tool_args, env=env)
+        code = process.wait()
+        if code != 0:
+            print('nontrivial return code %s' % code)
+            return 1
+    finally:
+        httpd.shutdown()
+        thread.join()
+    return 0
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--server-path', default='./')
+    parser.add_argument('--tool-cmd', required=True, type=str)
+    args = parser.parse_args()
+    result = test_tool(args.server_path,
+        args.tool_cmd.split())
+    sys.exit(result)
+
+if __name__ == '__main__':
+    main()

llvm/tools/llvm-debuginfod-find/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Debuginfod
+  Support
+  )
+add_llvm_tool(llvm-debuginfod-find
+  llvm-debuginfod-find.cpp
+  )
+if(LLVM_INSTALL_BINUTILS_SYMLINKS)
+  add_llvm_tool_symlink(debuginfod-find llvm-debuginfod-find)
+endif()

llvm/tools/llvm-debuginfod-find/llvm-debuginfod-find.cpp

+//===-- llvm-debuginfod-find.cpp - Simple CLI for libdebuginfod-client ----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// This file contains the llvm-debuginfod-find tool. This tool
+/// queries the debuginfod servers in the DEBUGINFOD_URLS environment
+/// variable (delimited by space (" ")) for the executable,
+/// debuginfo, or specified source file of the binary matching the
+/// given build-id.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Debuginfod/Debuginfod.h"
+#include "llvm/Debuginfod/HTTPClient.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/InitLLVM.h"
+
+using namespace llvm;
+
+cl::OptionCategory DebuginfodFindCategory("llvm-debuginfod-find Options");
+
+cl::opt<std::string> InputBuildID(cl::Positional, cl::Required,
+                                  cl::desc("<input build_id>"), cl::init("-"),
+                                  cl::cat(DebuginfodFindCategory));
+
+static cl::opt<bool>
+    FetchExecutable("executable", cl::init(false),
+                    cl::desc("If set, fetch a binary file associated with this "
+                             "build id, containing the executable sections."),
+                    cl::cat(DebuginfodFindCategory));
+
+static cl::opt<bool>
+    FetchDebuginfo("debuginfo", cl::init(false),
+                   cl::desc("If set, fetch a binary file associated with this "
+                            "build id, containing the debuginfo sections."),
+                   cl::cat(DebuginfodFindCategory));
+
+static cl::opt<std::string> FetchSource(
+    "source", cl::init(""),
+    cl::desc("Fetch a source file associated with this build id, which is at "
+             "this relative path relative to the compilation directory."),
+    cl::cat(DebuginfodFindCategory));
+
+static cl::opt<bool>
+    DumpToStdout("dump", cl::init(false),
+                 cl::desc("If set, dumps the contents of the fetched artifact "
+                          "to standard output. Otherwise, dumps the absolute "
+                          "path to the cached artifact on disk."),
+                 cl::cat(DebuginfodFindCategory));
+
+[[noreturn]] static void helpExit() {
+  errs() << "Must specify exactly one of --executable, "
+            "--source=/path/to/file, or --debuginfo.";
+  exit(1);
+}
+
+ExitOnError ExitOnErr;
+
+int main(int argc, char **argv) {
+  InitLLVM X(argc, argv);
+  HTTPClient::initialize();
+
+  cl::HideUnrelatedOptions({&DebuginfodFindCategory});
+  cl::ParseCommandLineOptions(
+      argc, argv,
+      "llvm-debuginfod-find: Fetch debuginfod artifacts\n\n"
+      "This program is a frontend to the debuginfod client library. The cache "
+      "directory, request timeout (in seconds), and debuginfod server urls are "
+      "set by these environment variables:\n"
+      "DEBUGINFOD_CACHE_PATH (default set by sys::path::cache_directory)\n"
+      "DEBUGINFOD_TIMEOUT (defaults to 90s)\n"
+      "DEBUGINFOD_URLS=[comma separated URLs] (defaults to empty)\n");
+
+  if (FetchExecutable + FetchDebuginfo + (FetchSource != "") != 1)
+    helpExit();
+
+  std::string IDString;
+  if (!tryGetFromHex(InputBuildID, IDString)) {
+    errs() << "Build ID " << InputBuildID << " is not a hex string.\n";
+    exit(1);
+  }
+  BuildID ID(IDString.begin(), IDString.end());
+
+  std::string Path;
+  if (FetchSource != "")
+    Path = ExitOnErr(getCachedOrDownloadSource(ID, FetchSource));
+  else if (FetchExecutable)
+    Path = ExitOnErr(getCachedOrDownloadExecutable(ID));
+  else if (FetchDebuginfo)
+    Path = ExitOnErr(getCachedOrDownloadDebuginfo(ID));
+  else
+    llvm_unreachable("We have already checked that exactly one of the above "
+                     "conditions is true.");
+
+  if (DumpToStdout) {
+    // Print the contents of the artifact.
+    ErrorOr<std::unique_ptr<MemoryBuffer>> Buf = MemoryBuffer::getFile(
+        Path, /*IsText=*/false, /*RequiresNullTerminator=*/false);
+    ExitOnErr(errorCodeToError(Buf.getError()));
+    outs() << Buf.get()->getBuffer();
+  } else
+    // Print the path to the cached artifact file.
+    outs() << Path << "\n";
+}