I gave a short presentation internally with an overview of our test suite. Adrian suggested converting (part of it) for the lldb website, which I thought was a great idea.
This is great!
what does "which is binary" mean?
We should also mention that the API tests will build the inferior for many different configurations
mention that these are end-to-end tests that compile programs from source, run them, and debug the processes?
mention that there is an an actual, literal, example test, too?
I would find most easy the link: https://github.com/llvm/llvm-project/tree/master/lldb/test/API/sample_test
s/fictive tests/fictive test/
There are multiple lldbutil* files, there could be a link: https://github.com/llvm/llvm-project/blob/master/lldb/packages/Python/lldbsuite/test/lldbutil.py
Some example: @expectedFailureAll(archs=["aarch64"], oslist=["linux"])
This is great, thanks for doing this. You point people at lldbutils.py as a source for utility functions (like print_stacktrace, etc...) But it is maybe also worth pointing them at the lldbtest.py for extensions runCmd and some of the other testing primitives we've added to the unittest2 framework.
I think this is a really good writeup. Thanks for doing this.
As we usually maintain our own tests, I would say "maintenance time" is a subset of "developer time". I guess you wanted to say something like that it's easy to set up, but increases the runtime of the tests and has a large ongoing cost.
s/prevent/remove (obviate) ?
Another interesting aspect of the "shell" tests is that there you can often get away with a broken/incomplete binary, whereas the "api" tests almost always require a fully functional executable. This enables testing of (some) aspects of handling of binaries with non-native architectures or operating systems. I see this as a big advantage of the shell tests. And a bit of a missed opportunity too -- we don't have many tests like this right now, which means is fairly hard to contribute something to lldb without having access to mac, linux, and windows machines (arm boxes are useful too, at times)...
This looks good overall. I would add a section describing which test suite to use when you're interested in a particular DWARF feature for example. I heard from my GDB colleagues that the don't use a compiler, because that might change and produce a different DWARF. Instead they use a DWARF assembler (IIRC). I guess the analogy to LLDB would be yaml2obj which I see not mentioned here. Does it make sense to at least reference it in a section?
I forgot to mention that I'd love to have this at hand when I began developing for LLDB. So thank you very much for writing this up!!!
I personally struggled with the way inferiors are being build. The Makefile includes another Makefile and was more like a framework than simple make. You had to set special variables in order for the included Makefile to pick it up. That level of indirection made it quite complicated for me to get what I wanted. To put it differently, it would be nice if you could describe what the Makefile should look like or what is expected.
Maybe link XFAIL to https://ftp.gnu.org/old-gnu/Manuals/dejagnu-1.3/html_node/dejagnu_6.html ?
Ah, there you're mentioning it.
The diff wasn't updated with the committed changes because I forgot the Differential Revision: trailer. If you look at https://reviews.llvm.org/rG3a6b60fa623da6e311b61c812932917085067cd3 you'll see they're fixed :-)