Page MenuHomePhabricator

DWZ 05/06: DWZ test mode
Changes PlannedPublic

Authored by jankratochvil on Nov 26 2017, 5:20 AM.

Details

Reviewers
clayborg
labath
Summary

With the new lit mode any DWZ test mode regression will make the whole testcase to fail. So one can regression test LLDB against disabled DWZ (such as mv /usr/bin/dwz /usr/bin/dwz-x, that could be improved).

All DWZ patches are also applied in: git clone -b dwz git://git.jankratochvil.net/lldb

Diff Detail

Event Timeline

jankratochvil created this revision.Nov 26 2017, 5:20 AM
labath edited edge metadata.Nov 27 2017, 3:04 AM

The thing to be aware of with this testing strategy is that you will probably be the only person who will be able to run these tests and verify dwz functionality. If you don't setup a buildbot testing this then other developers will never even know if they have broken any dwz functionality in the future. While end-to-end tests are great and we need them, I think it would be worthwhile to think about other testing strategies. These can range from checking in a couple of siple dwz files and making sure we can extract information from them (see unittests/SymbolFile/PDB for code that does something similar) to reimplementing/upstreaming the utilities necessary to build dwz files ourselves (I am not sure what the dwz tool does, but the eu-strip and sepdebugcrcfix tools sound like they would be fairly easy to implement on top of existing llvm libraries).

The other thing that worries me is that your comments seem to indicate that not all dwz tests pass after this. Could you elaborate on the "Many ERRORs are correct" part and how you plan to rectify that?

The third thing I want to say is not really related to your patchset, but your comments seem to indicate that not all regular dwarf tests were passing for you to begin with. Is that true? Would you be willing to help me (probably in a separate thread) to identify the tests that are failing and why? I'd like to move us towards a state where the result of the tests does not depend on the system you're running on and people don't have to resort to differential comparisons.

packages/Python/lldbsuite/test/test_categories.py
43

The way we do these sorts of things is that we explicitly add the category to the list of skipped categories (take a look at checkLibcxxSupport in dotest.py).

you will probably be the only person who will be able to run these tests and verify dwz functionality.

OK; really nobody runs RHEL/CentOS/Fedora? :-)

If you don't setup a buildbot testing

OK, I will try that. I was even running a test bot for GDB.

These can range from checking in a couple of siple dwz files

OK; although during development various bugs required various testfiles, single testfiles does not catch it all but sure better than nothing.

reimplementing/upstreaming the utilities necessary to build dwz files ourselves (I am not sure what the dwz tool does, but the eu-strip and sepdebugcrcfix tools sound like they would be fairly easy to implement on top of existing llvm libraries).

DWZ has its upstream sources.
eu-strip is just a more optimized (and more ELF-details preserving) variant of GNU binutils strip. It should be easy to slightly extend llvm-objcopy (as there is no llvm-strip). Or is it OK to use GNU binutils strip instead?
sepdebugcrcfix is a small utility currently inside upstream rpm package so that could be reimplemented in LLVM.
Although I admit both eu-strip and sepdebugcrcfix are used in the LLDB testsuite only to workaround some DWZ bug it fails for some reason to process unsplit debug info. I did not find DWZ code convenient enough to bugfix it there and its author Jakub Jelínek does not fix its reported bugs in recent years.

The other thing that worries me is that your comments seem to indicate that not all dwz tests pass after this. Could you elaborate on the "Many ERRORs are correct" part and how you plan to rectify that?

There is never any Success->Failure case for _dwarf->_dwz (there were such cases when I had bugs in my LLDB DWZ code). But there are Success->Error cases because in some cases DWZ tool cannot be used such as:

os command: make MAKE_DSYM=NO DWZ=YES ARCH=x86_64 CC="/quad/home/jkratoch/redhat/llvm-git-build-release/bin/clang-6.0"
with pid: 14709
stdout:
stderr: dwz: a.out.debug.dwz: .debug_info section not present
make[4]: *** [../../make/Makefile.rules:538: a.out] Error 1
retcode: 2
ERROR
Error when building test subject.

Unaware what to do with it. It could run the testcase without DWZ but then the _dwarf testcase would be needlessly run twice. Or there could be a list of tests to skip in DWZ mode. Or maybe some list of "ExpectedError" testcases (I guess one cannot defined "ExpectedError" now). This also means make check-lldb prints with DWZ a lot of error messages due to these "expected Errors".

your comments seem to indicate that not all regular dwarf tests were passing for you to begin with. Is that true?

Yes, I haven't investigated why yet. I am used from GDB to only diff results against regressions. RESULT-clean/ or RESULT-clean.tar.xz
BTW with installed system debug info files (/usr/lib/debug/**.debug) the LLDB testsuite times out and fails completely. LLDB could have some option like: gdb -iex 'set debug-file-directory /notexists'

Would you be willing to help me (probably in a separate thread) to identify the tests that are failing and why?

Yes but I would like to put more attention at the DWZ support upstreaming first as it is a LLDB blocker for Fedora/RHEL/CentOS.

you will probably be the only person who will be able to run these tests and verify dwz functionality.

OK; really nobody runs RHEL/CentOS/Fedora? :-)

There aren't that many lldb developers... I wouldn't be surprised if the number of them having access to a red-hat machine is 1 (you).

If you don't setup a buildbot testing

OK, I will try that. I was even running a test bot for GDB.

These can range from checking in a couple of siple dwz files

OK; although during development various bugs required various testfiles, single testfiles does not catch it all but sure better than nothing.

reimplementing/upstreaming the utilities necessary to build dwz files ourselves (I am not sure what the dwz tool does, but the eu-strip and sepdebugcrcfix tools sound like they would be fairly easy to implement on top of existing llvm libraries).

DWZ has its upstream sources.
eu-strip is just a more optimized (and more ELF-details preserving) variant of GNU binutils strip. It should be easy to slightly extend llvm-objcopy (as there is no llvm-strip). Or is it OK to use GNU binutils strip instead?
sepdebugcrcfix is a small utility currently inside upstream rpm package so that could be reimplemented in LLVM.
Although I admit both eu-strip and sepdebugcrcfix are used in the LLDB testsuite only to workaround some DWZ bug it fails for some reason to process unsplit debug info. I did not find DWZ code convenient enough to bugfix it there and its author Jakub Jelínek does not fix its reported bugs in recent years.

It looks like it could be a fun project to reimplement dwz on top of llvm dwarf library, but I understand that would be a considerable detour for you.

The other thing that worries me is that your comments seem to indicate that not all dwz tests pass after this. Could you elaborate on the "Many ERRORs are correct" part and how you plan to rectify that?

There is never any Success->Failure case for _dwarf->_dwz (there were such cases when I had bugs in my LLDB DWZ code). But there are Success->Error cases because in some cases DWZ tool cannot be used such as:

os command: make MAKE_DSYM=NO DWZ=YES ARCH=x86_64 CC="/quad/home/jkratoch/redhat/llvm-git-build-release/bin/clang-6.0"
with pid: 14709
stdout:
stderr: dwz: a.out.debug.dwz: .debug_info section not present
make[4]: *** [../../make/Makefile.rules:538: a.out] Error 1
retcode: 2
ERROR
Error when building test subject.

Unaware what to do with it. It could run the testcase without DWZ but then the _dwarf testcase would be needlessly run twice. Or there could be a list of tests to skip in DWZ mode. Or maybe some list of "ExpectedError" testcases (I guess one cannot defined "ExpectedError" now). This also means make check-lldb prints with DWZ a lot of error messages due to these "expected Errors".

How many tests are we talking about here? Could you list them here? If these really don't produce any debug info, then we can probably just mark them as @no_debug_info_testcase (that prevents generating N versions of the same test) and be done with it. There is also the ability to skip a test for dwz explicitly.

Also, be aware that we don't really do a good job of distinguising errors and failures. Compilation problems are a common source of Errors, but other issues can come up as errors as well -- it just depends on details of how test is written.

your comments seem to indicate that not all regular dwarf tests were passing for you to begin with. Is that true?

Yes, I haven't investigated why yet. I am used from GDB to only diff results against regressions. RESULT-clean/ or RESULT-clean.tar.xz

Yea, sometimes you have to do that here as well, but it's not a place where we'd like to be.
Looking at the results, they don't seem that bad, really. Most of the failures are in data formatters -- I think a well-placed -fno-limit-debug-info would get those working. The other category is something in the system library confusing the test (e.g. can't find environ or an extra global variable c). I think we could get those resolved fairly quickly.

BTW, is this on current master? I was hoping that some of these (such as TestTopLevelExpressions) would be resolved already.

BTW with installed system debug info files (/usr/lib/debug/**.debug) the LLDB testsuite times out and fails completely. LLDB could have some option like: gdb -iex 'set debug-file-directory /notexists'

Yea, that would be useful, particularly as we want to isolate the test from system specifics. It would also be useful to figure out why does it hang, as it will probably affect your users as well. :)

davide added a subscriber: davide.Nov 30 2017, 11:06 AM

@jankratochvil I run Fedora for now on my desktop, but I don't have time and resources to devote to a buildbot.
IMHO, having a fedora bot would be a great thing for two reasons:

  1. I found out Fedora exposes bugs that ubuntu doesn't (e.g. in symbol resolution, due to slightly different naming for globals in shared libraries)
  2. We can improve coverage for Linux for lldb.

I think the first step to get this to work is to get the build green there, and then we can iterate, if you're interested.

The thing to be aware of with this testing strategy is that you will probably be the only person who will be able to run these tests and verify dwz functionality. If you don't setup a buildbot testing this then other developers will never even know if they have broken any dwz functionality in the future. While end-to-end tests are great and we need them, I think it would be worthwhile to think about other testing strategies. These can range from checking in a couple of siple dwz files and making sure we can extract information from them (see unittests/SymbolFile/PDB for code that does something similar) to reimplementing/upstreaming the utilities necessary to build dwz files ourselves (I am not sure what the dwz tool does, but the eu-strip and sepdebugcrcfix tools sound like they would be fairly easy to implement on top of existing llvm libraries).

The other thing that worries me is that your comments seem to indicate that not all dwz tests pass after this. Could you elaborate on the "Many ERRORs are correct" part and how you plan to rectify that?

The third thing I want to say is not really related to your patchset, but your comments seem to indicate that not all regular dwarf tests were passing for you to begin with. Is that true? Would you be willing to help me (probably in a separate thread) to identify the tests that are failing and why? I'd like to move us towards a state where the result of the tests does not depend on the system you're running on and people don't have to resort to differential comparisons.

With the forthcoming introduction of lldb-test, why don't we piggyback off of that? We already have a compiler, so use it to produce a dwz, and then use lldb-test to exercise something about it. It won't be as comprehensive as a full scale test, but it would be a nice complement. We'd have basic test coverage on all platforms right out of the gate.

It looks like it could be a fun project to reimplement dwz on top of llvm dwarf library, but I understand that would be a considerable detour for you.

I think it would be best to drop DWZ, IIRC its gain is not much bigger than -fdebug-types-section and it is incompatible with some 3rd party debug info tools as DWZ did not really became a standard. I just want to make LLDB compatible with it.

How many tests are we talking about here? Could you list them here?

These tests ERROR due to dwz: a.out.debug.dwz: .debug_info section not present:

functionalities/breakpoint/breakpoint_ignore_count/TestBreakpointIgnoreCount.py
functionalities/data-formatter/data-formatter-ptr-to-array/TestPtrToArrayFormatting.py
functionalities/data-formatter/ptr_ref_typedef/TestPtrRef2Typedef.py
functionalities/data-formatter/typedef_array/TestTypedefArray.py
functionalities/frame_var_scope/TestFrameVariableScope.py
functionalities/signal/TestSendSignal.py
lang/cpp/bool/TestCPPBool.py
lang/cpp/frame-var-anon-unions/TestFrameVariableAnonymousUnions.py
lang/cpp/lambdas/TestLambdas.py
lang/cpp/nsimport/TestCppNsImport.py
linux/builtin_trap/TestBuiltinTrap.py
python_api/rdar-12481949/Test-rdar-12481949.py
sample_test/TestSampleInlineTest.py

If these really don't produce any debug info, then we can probably just mark them as @no_debug_info_testcase

I do not see this keyword in the tree.

Looking at the results, they don't seem that bad, really. Most of the failures are in data formatters -- I think a well-placed -fno-limit-debug-info would get those working. The other category is something in the system library confusing the test (e.g. can't find environ or an extra global variable c). I think we could get those resolved fairly quickly.

BTW, is this on current master? I was hoping that some of these (such as TestTopLevelExpressions) would be resolved already.

No, it was from 26th. I have refreshed the files with today's GIT hash 6513119854292f1992c95073e99d55bf88456adb = SVN 319402.

It would also be useful to figure out why does it hang, as it will probably affect your users as well. :)

I expect because LLDB does not have any Linux DWARF index support, it has only some Apple OSX index support. GDB's .debug_index is not usable for LLDB as it does not contain DIE offsets. I would like to implement DWARF-5 .debug_names for it later.

jankratochvil retitled this revision from DWZ 12/12: DWZ test mode to DWZ 10/11: DWZ test mode.

It now includes also unittests/SymbolFile/DWZ/ which should PASS even on systems without DWZ tool(s).

The way we do these sorts of things is that we explicitly add the category to the list of skipped categories (take a look at checkLibcxxSupport in dotest.py).

I have implemented dwz category like gmodules but even when checkDWZSupport makes configuration.skipCategories.append("dwz") the DWZ-variants of tests are still being run. I will investigate it more.

fwiw, you don't need unit tests or python tests to implement this.
If I understand the feature correctly you can probably extend lldb-test (which is run as part of check-lldb-lit).
Up to you though.

fwiw, you don't need unit tests or python tests to implement this.
If I understand the feature correctly you can probably extend lldb-test (which is run as part of check-lldb-lit).

I do not understand much yet all the testing frameworks LLDB implements. Could you suggest what you find redundant? So far I do want to run the whole testsuite also in dwz mode.
I understand unittests/SymbolFile/DWZ/SymbolFileDWZTests.cpp could be written also as an extension of tools/lldb-test. But then I do not see why. The unit test of mine looks fine to me.

I have implemented dwz category like gmodules but even when checkDWZSupport makes configuration.skipCategories.append("dwz") the DWZ-variants of tests are still being run. I will investigate it more.

This is fixed now (forgotten update of some items copied from gmodules).

jankratochvil planned changes to this revision.Apr 14 2018, 4:22 AM
jankratochvil retitled this revision from DWZ 10/11: DWZ test mode to DWZ 06/07: DWZ test mode.
jankratochvil planned changes to this revision.Apr 27 2018, 2:34 AM

With the new testsuite run it currently reports failures due to missing .debug_info sections in some testcases, I need to make it false-regression-free.

jankratochvil planned changes to this revision.Jul 3 2018, 10:43 AM
jankratochvil updated this revision to Diff 153941.
jankratochvil planned changes to this revision.Jul 3 2018, 10:45 AM
jankratochvil updated this revision to Diff 153942.
jankratochvil retitled this revision from DWZ 06/07: DWZ test mode to DWZ 05/06: DWZ test mode.
jankratochvil edited the summary of this revision. (Show Details)

There is no DWZ-tool wrapping script as discussed at: D48782

jankratochvil set the repository for this revision to rLLDB LLDB.Aug 2 2018, 9:26 AM
jankratochvil added a project: Restricted Project.Aug 30 2018, 7:15 AM
jankratochvil planned changes to this revision.Jun 27 2019, 3:45 AM