Index: docs/DeveloperPolicy.rst =================================================================== --- docs/DeveloperPolicy.rst +++ docs/DeveloperPolicy.rst @@ -221,6 +221,9 @@ entire failing program into ``llvm/test`` as this creates a *time-to-test* burden on all developers. Please keep them short. +* More testing details and discussion can be found in + :doc:`Testing Policy `. + Note that llvm/test and clang/test are designed for regression and small feature tests only. More extensive test cases (e.g., entire applications, benchmarks, etc) should be added to the ``llvm-test`` test suite. The llvm-test suite is Index: docs/TestingPolicy.rst =================================================================== --- /dev/null +++ docs/TestingPolicy.rst @@ -0,0 +1,127 @@ +=================== +LLVM Testing Policy +=================== + +.. contents:: + :local: + +Introduction +============ + +LLVM was designed from the beginning to enable fine grained testing of compiler +passes and code emission. While the project has expanded to include compilers, +assemblers, linkers, and debuggers the general testing philosophy continues: + +**“Test fine grained function rather than full integration tests wherever +possible.”** + +This means that tests involving clang and the backend are decoupled from each +other to make sure that we're testing the right part each time. + +Example +======= + +Let’s take as an example a C++ file that you’d like to check the assembly output +in order to get an idea of the probable performance impact of an individual +function. + +You could add something in llvm/tools/clang/test/CodeGen that looks like: + +``clang -S -O3 -target my_favorite_cpu-linux-gnu a.c -o - | FileCheck %s`` + +that checks the output of your favorite test compiled by clang with all of the +optimization you desired. This is just the kind of test we’d like to avoid. +Instead look at adding tests in all of these places: + +* ``llvm/tools/clang/test/CodeGen`` +* ``llvm/test/Transforms/`` +* ``llvm/test/CodeGen/my_favorite_cpu`` + +The first test will look something like: + +``clang -O0 -mllvm -disable-llvm-optzns -S a.c -o - -emit-llvm | FileCheck %s`` + +And verify that the IR emitted out of clang is both correct and a reasonable +place to begin optimization. + +The second test (or set of tests) will look something like: + +``opt -O3 a.ll -S -o - | FileCheck %s`` + +or even better: + +``opt -instcombine a.ll -S -o - | FileCheck %s`` + +This second set of tests takes the llvm IR generated by the first test and +runs some amount of optimization on it - making sure that the optimization +passes are able to take the expected incoming IR patterns and generate +appropriate IR for the code generator. The first test will run a lot of +optimizations, the second one is even more targeted to a single run of a pass. +The second form is preferable here if you can arrange it, we’ll discuss why +more later. + +The third test will then look something like: + +``llc -O3 a.ll -S -o - | FileCheck %s`` + +or even: + +``llc a.ll -S -o - | FileCheck %s`` + +This last test (or set of tests) takes IR coming out of the mid-level optimizers +that you’ve saved and tidied up by hand and runs it through the backend to +generate assembly code that you can then check for particular set of assembly +instructions you want to make sure are generated. + +Look at how all of these tests work as unit tests on different parts of the llvm +infrastructure. + +* .cc tests in clang (that produce IR) are used to test the correctness of the + IR that comes out of clang. +* opt tests (in llvm) that produce IR are designed to test the effectiveness of + the optimizer. +* .ll (or MIR) tests are used to test the effectiveness of the code generator. + +Each of these has its part in how we want to test code generation from a .cc +file in llvm. + +FAQ/Additional Comments +======================= + +#. Passes can differ between clang and opt. + + ``clang++ -O0 -mllvm -disable-llvm-optzns -S a.cc -o - -emit-llvm | opt -O3 + -S | llc -O3`` + + Can be different from + + ``clang++ -O3 -S a.cc`` + + Due to differences in the set of passes and additional infrastructure that + can be difficult to figure out when invoking clang versus the developer + tools. The developer list (or IRC) is useful for trying to analyze this, but + sometimes you’ll need to modify the intermediate steps in your test by hand + or add additional command line flags you weren’t aware of the ensure the + behavior you were expecting. + +#. “But I just want to check that my integration all worked!” + + You’re a compiler developer now! We want to be more able to pinpoint bugs + when something fails under testing. + +#. “... but how can I ensure that the integration test actually works?” + + Comments in test files are great! One thing that has proven useful over the + years is taking the original source and adding it as a comment with + reproduction instructions on how you generated the reduced testcase so that + when someone wants to verify that the original test works it makes it easier, + and they’ve still got a reduced testcase to make development easier. You + can also add an execution test to ``llvm/projects/test-suite`` to ensure the + runtime correctness of your testcase. + +#. “OK, this sounds great, but I have a lot of tests?” + + There are a number of tools in the repository that can help you with scripts + to make updating and writing your tests much easier. Take a look at + ``update_test_checks.py`` and the similar ones for llc and mir to make your + life as a compiler developer much easier.