[lldb/Docs] Add some more info about the test suite layout

[JDevlieghere]

Summary

I gave a short presentation internally with an overview of our test suite. Adrian suggested converting (part of it) for the lldb website, which I thought was a great idea.

[aprantl]

This is great!

lldb/docs/resources/test.rst

15

th*r*ough

what does "which is binary" mean?

32

located

70

We should also mention that the API tests will build the inferior for many different configurations

90

mention that these are end-to-end tests that compile programs from source, run them, and debug the processes?

99

mention that there is an an actual, literal, example test, too?

147

double space

[JDevlieghere]

Thanks for the review!

[jankratochvil]

lldb/docs/resources/test.rst

18

https://llvm.org/docs/CommandGuide/FileCheck.html

21

s/are are/are/

22

https://docs.python.org/2/library/unittest.html

43

s#lldb/test/shell#lldb/test/Shell#

44

s/build/built/

45

https://llvm.org/docs/CommandGuide/FileCheck.html

58

s/```/``/

59

https://lldb.llvm.org/python_reference/lldb.SBTarget-class.html#BreakpointCreateByName

99

I would find most easy the link: https://github.com/llvm/llvm-project/tree/master/lldb/test/API/sample_test

104

s/fictive tests/fictive test/

112

There are multiple lldbutil* files, there could be a link: https://github.com/llvm/llvm-project/blob/master/lldb/packages/Python/lldbsuite/test/lldbutil.py

117

Some example: @expectedFailureAll(archs=["aarch64"], oslist=["linux"])

122

Probably: @expectedFailure(checking_function_name)

126

s/tests/test/

131

s/things/thing/

151

expressivity?

[JDevlieghere]

Thanks for the review, Jan!

lldb/docs/resources/test.rst

45

You can't combine preformatted text with links, so I've added the link above and kept it preformatted here.

59

Same issue as mentioned earlier. Let me know if you prefer the link over the preformatted text.

[jankratochvil]

lldb/docs/resources/test.rst

45

OK, sure I am fine with that, thanks.

59

Up to you but I find it convenient for lazy readers to have the links there to get more involved into the framework.

[jingham]

This is great, thanks for doing this. You point people at lldbutils.py as a source for utility functions (like print_stacktrace, etc...) But it is maybe also worth pointing them at the lldbtest.py for extensions runCmd and some of the other testing primitives we've added to the unittest2 framework.

lldb/docs/resources/test.rst

116

doubly

[labath]

I think this is a really good writeup. Thanks for doing this.

lldb/docs/resources/test.rst

153–154

As we usually maintain our own tests, I would say "maintenance time" is a subset of "developer time". I guess you wanted to say something like that it's easy to set up, but increases the runtime of the tests and has a large ongoing cost.

157

s/prevent/remove (obviate) ?

[labath]

Another interesting aspect of the "shell" tests is that there you can often get away with a broken/incomplete binary, whereas the "api" tests almost always require a fully functional executable. This enables testing of (some) aspects of handling of binaries with non-native architectures or operating systems. I see this as a big advantage of the shell tests. And a bit of a missed opportunity too -- we don't have many tests like this right now, which means is fairly hard to contribute something to lldb without having access to mac, linux, and windows machines (arm boxes are useful too, at times)...

[kwk]

This looks good overall. I would add a section describing which test suite to use when you're interested in a particular DWARF feature for example. I heard from my GDB colleagues that the don't use a compiler, because that might change and produce a different DWARF. Instead they use a DWARF assembler (IIRC). I guess the analogy to LLDB would be yaml2obj which I see not mentioned here. Does it make sense to at least reference it in a section?

I forgot to mention that I'd love to have this at hand when I began developing for LLDB. So thank you very much for writing this up!!!

lldb/docs/resources/test.rst

86

s/Thy/They

88

I personally struggled with the way inferiors are being build. The Makefile includes another Makefile and was more like a framework than simple make. You had to set special variables in order for the included Makefile to pick it up. That level of indirection made it quite complicated for me to get what I wanted. To put it differently, it would be nice if you could describe what the Makefile should look like or what is expected.

94

s/begin/being

95

s/sources/source
s/file/files

119

Maybe link XFAIL to https://ftp.gnu.org/old-gnu/Manuals/dejagnu-1.3/html_node/dejagnu_6.html ?

139

Ah, there you're mentioning it.

[JDevlieghere]

Thanks a lot for the feedback everyone!

[JDevlieghere]

Closed by 3a6b60fa623da6e311b61c812932917085067cd3

[kwk]

I've marked some mistakes that were not addressed yet but are marked as "Done".

lldb/docs/resources/test.rst

86

Not done yet.

94

Not done yet.

95

Not done yet.

[JDevlieghere]

In D78242#1993781, @kwk wrote:

I've marked some mistakes that were not addressed yet but are marked as "Done".

The diff wasn't updated with the committed changes because I forgot the Differential Revision: trailer. If you look at https://reviews.llvm.org/rG3a6b60fa623da6e311b61c812932917085067cd3 you'll see they're fixed :-)

lldb/docs/resources/test.rst

 Testing
 =======
 
+.. contents::
+   :local:
+
+Test Suite Structure
+--------------------
+
 The LLDB test suite consists of three different kinds of test:
 
-* Unit test. These are located under ``lldb/unittests`` and are written in C++
-  using googletest.
-* Integration tests that test the debugger through the SB API. These are
-  located under ``lldb/packages/Python/lldbsuite`` and are written in Python
-  using ``dotest`` (LLDB's custom testing framework on top of unittest2).
-* Integration tests that test the debugger through the command line. These are
-  located under `lldb/test/Shell` and are written in a shell-style format
-  using FileCheck to verify its output.
-
-All three test suites use the `LLVM Integrated Tester
-<https://llvm.org/docs/CommandGuide/lit.html>`_ (lit) as their test driver. The
-test suites can be run as a whole or separately.
-
-Many of the tests are accompanied by a C (C++, ObjC, etc.) source file. Each
-test first compiles the source file and then uses LLDB to debug the resulting
-executable.
+* **Unit tests**: written in C++ using the googletest unit testing library.
+* **Shell tests**: Integration tests that test the debugger through the command
+  line. These tests interact with the debugger either through the command line
+  driver or through ``lldb-test`` which is a tool that exposes the internal
+  data structures in an easy-to-parse way for testing. Most people will know
+  these as *lit tests* in LLVM, although lit is the test driver and ShellTest
+  is the test format that uses ``RUN:`` lines. ``FileCheck`` is used to verify
+  the output.
+* **API tests**: Integration tests that interact with the debugger through the
+  SB API. These are are written in Python and use LLDB's ``dotest.py`` testing
+  framework on top of Python's unittest2.
+
+All three test suites use ``lit`` (`LLVM Integrated Tester
+<https://llvm.org/docs/CommandGuide/lit.html>`_ ) as the test driver. The test
+suites can be run as a whole or separately.
+
+
+Unit Tests
+``````````
+
+Unit tests are located under ``lldb/unittests``. If it's possible to test
+something in isolation or as a single unit, you should make it a unit test.
+
+Often you need instances of the core objects such as a debugger, target or
+process, in order to test something meaningful. We already have a handful of
+tests that have the necessary boiler plate, but this is something we could
+abstract away and make it more user friendly.
+
+Shell Tests
+```````````
+
+Shell tests are located under ``lldb/test/shell``. These tests are generally
+build around checking the output of ``lldb`` (the command line driver) or
+``lldb-test`` using ``FileCheck``. Shell tests are generally small and fast to
+write because they require little boilerplate.
+
+``lldb-test`` is a relatively new addition to the test suite. It was the first
+tool that was added that is designed for testing. Since then it has been
+continuously extended with new subcommands, improving our test coverage. Among
+other things you can use it to query lldb for symbol files, for object files
+and breakpoints.
+
+Obviously shell tests are great for testing the command line driver itself or
+the subcomponents already exposed by lldb-test. But when it comes to LLDB's
+vast functionality, most things can be tested both through the driver as well
+as the Python API. For example, to test setting a breakpoint, you could do it
+from the command line driver with ``b main``` or you could use the SB API and do
+something like ``target.BreakpointCreateByName``.
+
+A good rule of thumb is to prefer shell tests when what is being tested is
+relatively simple. Expressivity is limited compared to the API tests, which
+means that you have to have a well-defined test scenario that you can easily
+match with ``FileCheck``.
+
+Another thing to consider are the binaries being debugged, which we call
+inferiors. For shell tests, they have to be relatively simple. The
+``dotest.py`` test framework has extensive support for complex build scenarios
+and different variants, which is described in more detail below, while shell
+tests are limited to single lines of shell commands with compiler and linker
+invocations.
+
+Finally, the shell tests always run in batch mode. You start with some input
+and the test verifies the output. The debugger can be sensitive to its
+environment, such as the the platform it runs on. It can be hard to express
+that the same test might behave slightly differently on macOS and Linux.
+Additionally, the debugger is an interactive tool, and the shell test provide
+no good way of testing those interactive aspects, such as tab completion for
+example.
+
+API Tests
+`````````
+
+API tests are located under ``lldb/test/API``. Thy are run with the
+``dotest.py``. Tests are written in Python and test binaries (inferiors) are
+compiled with Make. The majority of API tests are end-to-end tests that compile
+programs from source, run them, and debug the processes.
+
+Below is the directory layout of the example API test. The test directory will
+always contain a python file, starting with ``Test``.  Most of the tests are
+structured as a binary begin debugged, so there will be one or more sources
+file and a Makefile.
+
+::
+
+  sample_test
+  ├── Makefile
+  ├── TestSampleTest.py
+  └── main.c
+
+Let's start with the Python test file. Every test is its own class and can have
+one or more test methods, that start with ``test_``.  Many tests define
+multiple test methods and share a bunch of common code. For example, for a
+fictive tests that makes sure we can set breakpoints we might have one test
+method that ensures we can set a breakpoint by address, on that sets a
+breakpoint by name and another that sets the same breakpoint by file and line
+number. The setup, teardown and everything else other than setting the
+breakpoint could be shared.
+
+Our testing framework also has a bunch of utilities that abstract common
+operations, such as creating targets, setting breakpoints etc. When code is
+shared across tests, we extract it into a utility in ``lldbutil``. It's always
+worth taking a look at  lldbutil to see if there's a utility to simplify some
+of the testing boiler plate. Because we can't always audit every existing test,
+this is double true when looking at an existing test for inspiration.
+
+It's possible to skip or XFAIL tests using decorators. You'll see them a lot.
+The debugger can be sensitive to things like the architecture, the host and
+target platform, the compiler version etc. LLDB comes with a range of
+predefined decorators for these configurations. Another great thing about these
+decorators is that they're very easy to extend, it's even possible to define a
+function in a test case that determines whether the test should be run or not.
+
+In addition to providing a lot more flexibility when it comes to writing the
+test, the API test also allow for much more complex scenarios when it comes to
+building inferiors. Every tests has its own Makefile, most of them only a few
+lines long. A shared Makefile (``Makefile.rules``) with about a thousand lines
+of rules takes care of most if not all of the boiler plate, while individual
+make files can be used to build more advanced tests. 

+
+Another things this enables is having different variants for the same test
+case. By default, we run every test for all 3 debug info formats, so once with
+DWARF from the object files, once with gmodules and finally with a dSYM on
+macOS or split DWARF (DWO) on Linux. But there are many more things we can test
+that are orthogonal to the test itself. On GreenDragon we have a matrix bot
+that runs the test suite under different configurations, with older host
+compilers and different DWARF versions.
+
+As you can imagine, this quickly lead to combinatorial explosion in the number
+of variants. It's very tempting to add more variants because it's an easy way
+to increase test coverage. It doesn't scale. It's cheap in developer time, but
+extremely expensive in terms of runtime and maintenance time because it
+multiplies the overhead.
+
+The key take away is that the different variants don't prevent the need for
+focused tests. So relying on it to test say DWARF5 is a really bad idea.
+Instead you should write tests that check the specific DWARF5 feature, and have
+the variant as a nice-to-have.
+
+In conclusion, you'll want to opt for an API test to test the API itself or
+when you need the expressively, either for the test case itself or for the
+program being debugged. The fact that the API tests work with different
+variants mean that more general tests should be API tests, so that they can be
+run against the different variants.
 
-.. contents::
-   :local:
 
 Running The Tests
 -----------------
 
 .. note::
 
    On Windows any invocations of python should be replaced with python_d, the
    debug interpreter, when running the test suite against a debug version of
    LLDB.