Page MenuHomePhabricator

[llvm] [Debuginfo] Add llvm-debuginfod-find tool and end-to-end-tests.
ClosedPublic

Authored by noajshu on Oct 28 2021, 2:31 PM.

Details

Summary

This implements the llvm-debuginfod-find tool, which wraps the Debuginfod library (D112758) to query debuginfod servers for artifacts according to the specification.

Diff Detail

Unit TestsFailed

TimeTest
80 msx64 debian > LLVM.Bindings/Go::go.test
Script: -- : 'RUN: at line 1'; /var/lib/buildkite-agent/builds/llvm-project/build/bin/llvm-go go=/usr/bin/go test llvm.org/llvm/bindings/go/llvm

Event Timeline

noajshu created this revision.Oct 28 2021, 2:31 PM
noajshu requested review of this revision.Oct 28 2021, 2:31 PM
Herald added a project: Restricted Project. · View Herald TranscriptOct 28 2021, 2:31 PM
noajshu updated this revision to Diff 384317.Nov 2 2021, 8:01 PM

Clean up unrelated changes.

noajshu updated this revision to Diff 388801.Nov 21 2021, 11:21 PM

Use latest debuginfod client library (D112758).
Also condition debuginfod-find lit test execution on availability of libcurl.

noajshu updated this revision to Diff 388804.Nov 21 2021, 11:42 PM

Build llvm-debuginfod-find even if curl is disabled. HTTP fetch will be a no-op.

noajshu updated this revision to Diff 390209.Nov 28 2021, 9:01 AM

Add more help info for client tool and update lit tests.

labath added inline comments.Nov 29 2021, 1:20 AM
llvm/test/tools/llvm-debuginfod-find/debuginfod-find.py
2–3 ↗(On Diff #390209)

you don't have to chain the commands with &&. The script is automatically aborted when any command fails.

This would be much clearer if you just put every command on a separate line instead of peppering it with && and \'s.

36 ↗(On Diff #390209)

It may be helpful to print what it was.

38 ↗(On Diff #390209)

I am somewhat surprised by the need to peek inside the cache directory. What is the actual interface of the tool, or how is the user expected to know where to locate the file that it has downloaded? It seems like the elfutils debuginfod-find prints the path to the file on stdout. Should this do the same?

(I mean, it may be useful to hardcode the cache path so you can verify the file has been downloaded to the right place, but i don't think that every user should need to know the precise hash algorithm just to be able to locate the downloaded file.)

41 ↗(On Diff #390209)

or even "does not", for maximum formality.

45–46 ↗(On Diff #390209)

Instead of doing the checks here, it may be better to just print the relevant data (exit code, file contents, whatever) and then verify that through an external FileCheck command.

The reason I'm proposing this is to improve the error message in case this fails. Like, imagine this test fails on a bot somewhere. "file contents does not match" will not tell you much, as it does not even say which file it refers to. OTOH, with FileCheck you get the "i expected <this> but got <that>" functionality for free. (You can print repr(file.read()) and use FileCheck --strict-whitespace, if you want to catch any issues with newlines or unprintable characters.)

noajshu updated this revision to Diff 391950.Dec 5 2021, 6:42 PM

Add --dump option to llvm-debuginfod-find and simplify lit tests using FileCheck.

noajshu updated this revision to Diff 391953.Dec 5 2021, 7:35 PM

Print exit code of process when lit test fails.

Hi @labath thanks for all of your comments on debuginfod-find.py. I agree it is cleaner to use FileCheck for the lit tests, and I rewrote these tests into debuginfod.test incorporating your suggestions and also some feedback on the tests in D113717.

Regarding the output behavior, Elfutils debuginfod-find prints the path of the fetched/cached artifact file to stdout. By default, llvm-debuginfod-find does the same. I've also added a command line flag llvm-debuginfod-find --dump which dumps the contents of the artifact to stdout instead of its file path.
If you can only print the path to the cached file, you have to read the stdout and then use that path as the argument to another program (e.g. cat). This is easy to do in a full shell using xargs, or a subshell or variable, but inconvenient in LLVM Lit scripts which use a sequence of LLVM commands chained by pipes.

Does this output behavior of llvm-debuginfod-find and the --dump flag seem reasonable to you? If it's more clear we could change the name to --print-contents or something else. Thanks!

labath accepted this revision.Dec 6 2021, 4:55 AM

Regarding the output behavior, Elfutils debuginfod-find prints the path of the fetched/cached artifact file to stdout. By default, llvm-debuginfod-find does the same. I've also added a command line flag llvm-debuginfod-find --dump which dumps the contents of the artifact to stdout instead of its file path.
If you can only print the path to the cached file, you have to read the stdout and then use that path as the argument to another program (e.g. cat). This is easy to do in a full shell using xargs, or a subshell or variable, but inconvenient in LLVM Lit scripts which use a sequence of LLVM commands chained by pipes.

Does this output behavior of llvm-debuginfod-find and the --dump flag seem reasonable to you? If it's more clear we could change the name to --print-contents or something else. Thanks!

Yeah, this isn't easy to do from a lit test. My idea was (since you have a python script anyway) to implement the read-the-filename-and-dump-contents functionality in python. *However*, having the ability for the tool to dump the file contents seems like it could potentially be useful as well, so I have no problem with doing it this way. I have no opinion on the option name.

llvm/tools/llvm-debuginfod-find/llvm-debuginfod-find.cpp
32

Are you sure you're (still) using all of these headers?

This revision is now accepted and ready to land.Dec 6 2021, 4:55 AM
noajshu updated this revision to Diff 393367.Dec 9 2021, 7:21 PM
noajshu marked an inline comment as done.

Remove unnecessary header includes, and add HTTPClient::initialize() call to llvm-debuginfod-find.cpp.

noajshu added inline comments.Dec 9 2021, 7:22 PM
llvm/tools/llvm-debuginfod-find/llvm-debuginfod-find.cpp
32

No I'm not, thanks for pointing this out!

noajshu edited the summary of this revision. (Show Details)Dec 10 2021, 2:33 AM
noajshu edited the summary of this revision. (Show Details)
This revision was landed with ongoing or failed builds.Dec 10 2021, 2:34 AM
This revision was automatically updated to reflect the committed changes.
thakis added inline comments.
llvm/test/CMakeLists.txt
81

you have to add llvm-debuginfod-find here too so that it's guaranteed the executable is build before tests are run when target check-llvm is built

noajshu updated this revision to Diff 393693.Dec 11 2021, 1:18 PM
noajshu marked an inline comment as done.

Add llvm-debuginfod-find to list of required binaries for lit tests and update llvm-debuginfod-find test script to be more compatible with older pythons. (see buildbot failure due to python usage below 3.6: https://lab.llvm.org/buildbot/#/builders/164/builds/6751/steps/6/logs/FAIL__LLVM__debuginfod_test)

It seems like some buildbots use python that is below 3.6 (see buildbot failure due to missing python3.6 features)
This seems to contradict LLVM's testing infra docs which say to use python3.6 or later.
Either way I've backported most parts of the python testing script to be more compatible with earlier python.

llvm/test/CMakeLists.txt
81

Thanks so much!

noajshu updated this revision to Diff 394080.Dec 13 2021, 4:22 PM

Hide unrelated options printed by llvm-debuginfod-find.

Hi,

I get the following error when I run tests (check-all) on this commit:

FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (32121 of 80857)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --executable abcdef' |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --executable abcdef |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/repo/uabelho/master-github/llvm/build-all-builtins/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /repo/uabelho/master-github/llvm/build-all-builtins/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 2

Command Output (stderr):
--
  File "/repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test", line 55
    env['DEBUGINFOD_URLS'] = f'http://localhost:%s' % port
                                                  ^
SyntaxError: invalid syntax
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /repo/uabelho/master-github/llvm/build-all-builtins/bin/FileCheck /repo/uabelho/master-github/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE

--

Any idea what is going on?

Hi Mikael,

Could you check which version of python you are running? (python --version)

Thanks,
Noah

@uabelho does D115814 fix this for you? I think it might be caused by the python3 f-string.

vitalybuka added a subscriber: vitalybuka.EditedDec 15 2021, 1:01 PM

I have one like this

[1/2] Running the LLVM regression tests
FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (15361 of 46561)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --executable abcdef' |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --executable abcdef |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/usr/local/google/home/vitalybuka/src/llvm.git/out/z/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/FileCheck /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 1

Command Output (stderr):
--
127.0.0.1 - - [15/Dec/2021 12:58:47] "GET /buildid/abcdef/executable HTTP/1.1" 200 -
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find --dump --executable abcdef
 #0 0x0000000000252043 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x252043)
 #1 0x000000000024ff7e llvm::sys::RunSignalHandlers() (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x24ff7e)
 #2 0x000000000025264f SignalHandler(int) Signals.cpp:0:0
 #3 0x00007fd255a7a8e0 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x138e0)
 #4 0x00007fd2554c3d3c free ./malloc/malloc.c:3124:7
 #5 0x000000000021dac3 llvm::HTTPClient::perform(llvm::HTTPRequest const&, llvm::HTTPResponseHandler&) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21dac3)
 #6 0x000000000021dc84 llvm::HTTPClient::get(llvm::StringRef) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21dc84)
 #7 0x000000000021c471 llvm::getCachedOrDownloadArtifact[abi:cxx11](llvm::StringRef, llvm::StringRef, llvm::StringRef, llvm::ArrayRef<llvm::StringRef>, std::chrono::duration<long, std::ratio<1l, 1000l> >) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21c471)
 #8 0x000000000021bb2c llvm::getCachedOrDownloadArtifact[abi:cxx11](llvm::StringRef, llvm::StringRef) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21bb2c)
 #9 0x000000000021be0e llvm::getCachedOrDownloadExecutable[abi:cxx11](llvm::ArrayRef<unsigned char>) (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x21be0e)
#10 0x0000000000219b68 main (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x219b68)
#11 0x00007fd255460e4a __libc_start_main ./csu/../csu/libc-start.c:314:16
#12 0x00000000002195da _start (/usr/local/google/home/vitalybuka/src/llvm.git/out/z/bin/llvm-debuginfod-find+0x2195da)
/usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test:15:15: error: EXECUTABLE: expected string not found in input
# EXECUTABLE: fake_executable
              ^
<stdin>:1:1: note: scanning from here
nontrivial return code -11
^
<stdin>:1:8: note: possible intended match here
nontrivial return code -11
       ^

Input file: <stdin>
Check file: /usr/local/google/home/vitalybuka/src/llvm.git/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: nontrivial return code -11 
check:15'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:15'1            ?                    possible intended match
>>>>>>

--

********************
********************
Failed Tests (1):
  LLVM :: tools/llvm-debuginfod-find/debuginfod.test

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

uabelho added a comment.EditedDec 15 2021, 10:52 PM

Hi Mikael,

Could you check which version of python you are running? (python --version)

Thanks,
Noah

Hi,

We're using python 3.8:

-- Found Python3: /app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 (found suitable version "3.8.0", minimum required is "3.6") found components: Interpreter

and

> app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 --version
Python 3.8.0

However, I noticed now that it passes if I do a clean build, so I have no idea what actually happened when I got the failure yesterday.
Could there be some problem so binaries are not rebuilt as they should?

Apologies for sending you on a wild-goose chase. :/

The test passes for me also with https://reviews.llvm.org/D115814 applied, but as I said, that patch doesn't seem to be needed by us at least.

Hi Mikael,

Could you check which version of python you are running? (python --version)

Thanks,
Noah

Hi,

We're using python 3.8:

-- Found Python3: /app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 (found suitable version "3.8.0", minimum required is "3.6") found components: Interpreter

and

> app/vbuild/RHEL7-x86_64/python/3.8.0/bin/python3.8 --version
Python 3.8.0

However, I noticed now that it passes if I do a clean build, so I have no idea what actually happened when I got the failure yesterday.
Could there be some problem so binaries are not rebuilt as they should?

Apologies for sending you on a wild-goose chase. :/

The test passes for me also with https://reviews.llvm.org/D115814 applied, but as I said, that patch doesn't seem to be needed by us at least.

@uabelho thanks so much for confirming it works after you clean your build folder, and no worries about the flake.
Although the cmake caching behavior is mostly correct, I have seen at least one exception (e.g. when implicit tools are added).
Glad to hear it now passes the test, and thanks for running it!

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

I am on 3aece7995cb63a536a15df35ebe107d04a8acc86 and my llvm/lib/Debuginfod/HTTPClient.cpp is exactly as 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44
git diff 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44 -- llvm/lib/Debuginfod/HTTPClient.cpp
no output

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

I am on 3aece7995cb63a536a15df35ebe107d04a8acc86 and my llvm/lib/Debuginfod/HTTPClient.cpp is exactly as 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44
git diff 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44 -- llvm/lib/Debuginfod/HTTPClient.cpp
no output

I just did full clean rebuild and it's gone.

Hi @vitalybuka thanks for reporting this. It looks similar to the bug in HTTPClient.cpp that was fixed by 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44.
For simplicity, could you please post the contents of your local copy of HTTPClient.cpp?
This way I can confirm the fix hasn't been cherry-picked out somehow.
Thanks!

I am on 3aece7995cb63a536a15df35ebe107d04a8acc86 and my llvm/lib/Debuginfod/HTTPClient.cpp is exactly as 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44
git diff 34491ca7291ceb3d4e7ad65288b9b48c6d7eeb44 -- llvm/lib/Debuginfod/HTTPClient.cpp
no output

I just did full clean rebuild and it's gone.

I'm glad the issue has disappeared. Although it's mysterious the build system caching didn't work as intended.

Hello.
The test added here is failing for me (and i think i'm not the only one) on debian:

[ 99% 3987/3988][ 98% 0:10:34 + 0:00:08] Running the LLVM regression tests
FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (913 of 46565)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 1

Command Output (stderr):
--
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/abcdef/executable HTTP/1.1" 200 -
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef
127.0.0.1 - - [20/Dec/2021 17:01:26] code 404, message File not found
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/811bf0bfb413c00e/debuginfo HTTP/1.1" 404 -
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamEi+0x23)[0x7f20d9f65353]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys17RunSignalHandlersEv+0xf0)[0x7f20d9f63010]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(+0x26585a)[0x7f20d9f6585a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13200)[0x7f20d9cc8200]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x25)[0x7f20d980b945]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestERNS_19HTTPResponseHandlerE+0x262)[0x7f20d9f8a042]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestE+0x41)[0x7f20d9f89d41]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient3getENS_9StringRefE+0x31)[0x7f20d9f8a2a1]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_S0_NS_8ArrayRefIS0_EENSt6chrono8durationIlSt5ratioILl1ELl1000EEEE+0x475)[0x7f20d9f88135]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_+0x154)[0x7f20d9f876a4]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm29getCachedOrDownloadExecutableB5cxx11ENS_8ArrayRefIhEE+0x1c2)[0x7f20d9f87a02]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x20558e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f20d97a87ed]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x2048ea]
/repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test:15:15: error: EXECUTABLE: expected string not found in input
# EXECUTABLE: fake_executable
              ^
<stdin>:1:1: note: scanning from here
nontrivial return code -11
^
<stdin>:1:8: note: possible intended match here
nontrivial return code -11
       ^

Input file: <stdin>
Check file: /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: nontrivial return code -11 
check:15'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:15'1            ?                    possible intended match
>>>>>>

--

********************
********************
Failed Tests (1):
  LLVM :: tools/llvm-debuginfod-find/debuginfod.test


Testing Time: 277.28s
  Unsupported      :   344
  Passed           : 46059
  Expectedly Failed:   161
  Failed           :     1
FAILED: test/CMakeFiles/check-llvm /builddirs/llvm-project/build-Clang13/test/CMakeFiles/check-llvm 
cd /builddirs/llvm-project/build-Clang13/test && /usr/bin/python3.9 /builddirs/llvm-project/build-Clang13/./bin/llvm-lit -sv /builddirs/llvm-project/build-Clang13/test
ninja: build stopped: subcommand failed.

Could you please resolve the problem?

Hello.
The test added here is failing for me (and i think i'm not the only one) on debian:

[ 99% 3987/3988][ 98% 0:10:34 + 0:00:08] Running the LLVM regression tests
FAIL: LLVM :: tools/llvm-debuginfod-find/debuginfod.test (913 of 46565)
******************** TEST 'LLVM :: tools/llvm-debuginfod-find/debuginfod.test' FAILED ********************
Script:
--
: 'RUN: at line 2';   rm -rf /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 3';   mkdir /builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp
: 'RUN: at line 5';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 8';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --source=/directory/file.c abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 11';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp python /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --server-path /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/Inputs    --tool-cmd '/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef' |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
: 'RUN: at line 21';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=EXECUTABLE
: 'RUN: at line 23';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump    --source=/directory/file.c abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=SOURCE
: 'RUN: at line 26';   DEBUGINFOD_CACHE_PATH=/builddirs/llvm-project/build-Clang13/test/tools/llvm-debuginfod-find/Output/debuginfod.test.tmp /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --debuginfo abcdef |    /builddirs/llvm-project/build-Clang13/bin/FileCheck /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test --check-prefix=DEBUGINFO
--
Exit Code: 1

Command Output (stderr):
--
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/abcdef/executable HTTP/1.1" 200 -
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find --dump --executable abcdef
127.0.0.1 - - [20/Dec/2021 17:01:26] code 404, message File not found
127.0.0.1 - - [20/Dec/2021 17:01:26] "GET /buildid/811bf0bfb413c00e/debuginfo HTTP/1.1" 404 -
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamEi+0x23)[0x7f20d9f65353]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(_ZN4llvm3sys17RunSignalHandlersEv+0xf0)[0x7f20d9f63010]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMSupport.so.14git(+0x26585a)[0x7f20d9f6585a]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13200)[0x7f20d9cc8200]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x25)[0x7f20d980b945]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestERNS_19HTTPResponseHandlerE+0x262)[0x7f20d9f8a042]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient7performERKNS_11HTTPRequestE+0x41)[0x7f20d9f89d41]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm10HTTPClient3getENS_9StringRefE+0x31)[0x7f20d9f8a2a1]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_S0_NS_8ArrayRefIS0_EENSt6chrono8durationIlSt5ratioILl1ELl1000EEEE+0x475)[0x7f20d9f88135]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm27getCachedOrDownloadArtifactB5cxx11ENS_9StringRefES0_+0x154)[0x7f20d9f876a4]
/builddirs/llvm-project/build-Clang13/bin/../lib/libLLVMDebuginfod.so.14git(_ZN4llvm29getCachedOrDownloadExecutableB5cxx11ENS_8ArrayRefIhEE+0x1c2)[0x7f20d9f87a02]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x20558e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f20d97a87ed]
/builddirs/llvm-project/build-Clang13/bin/llvm-debuginfod-find[0x2048ea]
/repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test:15:15: error: EXECUTABLE: expected string not found in input
# EXECUTABLE: fake_executable
              ^
<stdin>:1:1: note: scanning from here
nontrivial return code -11
^
<stdin>:1:8: note: possible intended match here
nontrivial return code -11
       ^

Input file: <stdin>
Check file: /repositories/llvm-project/llvm/test/tools/llvm-debuginfod-find/debuginfod.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: nontrivial return code -11 
check:15'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:15'1            ?                    possible intended match
>>>>>>

--

********************
********************
Failed Tests (1):
  LLVM :: tools/llvm-debuginfod-find/debuginfod.test


Testing Time: 277.28s
  Unsupported      :   344
  Passed           : 46059
  Expectedly Failed:   161
  Failed           :     1
FAILED: test/CMakeFiles/check-llvm /builddirs/llvm-project/build-Clang13/test/CMakeFiles/check-llvm 
cd /builddirs/llvm-project/build-Clang13/test && /usr/bin/python3.9 /builddirs/llvm-project/build-Clang13/./bin/llvm-lit -sv /builddirs/llvm-project/build-Clang13/test
ninja: build stopped: subcommand failed.

Could you please resolve the problem?

Hi @lebedev.ri thanks for sharing this error. Based on other anecdotal reports I expect that clearing and regenerating your build folder will fix the issue. It would be helpful if you could please save a backup copy of the build folder first, so that if clearing and re-generating works, we can diagnose and fix the underlying problem with the build system caching.

Then, if that clear-and-regenerate works, would be possibly be able to share your build folder? If you just compress it and upload somewhere I can take a look.

Hello.
The test added here is failing for me (and i think i'm not the only one) on debian:

<...>

Could you please resolve the problem?

Hi @lebedev.ri thanks for sharing this error. Based on other anecdotal reports I expect that clearing and regenerating your build folder will fix the issue. It would be helpful if you could please save a backup copy of the build folder first, so that if clearing and re-generating works, we can diagnose and fix the underlying problem with the build system caching.

Then, if that clear-and-regenerate works, would be possibly be able to share your build folder? If you just compress it and upload somewhere I can take a look.

Ok, thanks, removing build directory helped. CMake caching indeed breaks sometimes with things like this, so i don't think it's worthwhile trying to fix this then.