This is the initial commit for the benchmark testing framework I plan to use in libc++. It functions similarly to the existing LIT setup.
The benchmarks use the Google benchmark library found here: http://github.com/google/benchmark.
To enable building the benchmark library use the cmake option -DLIBCXX_ENABLE_BENCHMARK=ON. This option will checkout the library from github and build it in the build/external directory.
Once the library is built the benchmarks can be run in one of two ways.
- Standalone (without baseline comparison): This simply runs the benchmarks and generates the output. Use -o /path/to/OUTPUT to save the results of the benchmarks.
lit -v -o /path/to/baseline.txt /path/to/benchmarks
- Comparison against baseline: This runs the benchmarks and compares the results to a baseline file specified. If the current results are slower by more than the "allowed difference" the test fails and the results are reported.
lit -sv -o /path/to/current_results.txt --param=baseline=/path/to/baseline.txt --param=allowed_difference=2.5 /path/to/benchmarks
The allowed_difference parameter takes the percentage which the results are allowed to differ from the baseline. The default is 5%.
The benchmark tests are not run as part of the regular test suite. They are too time consuming and do not provide much value unless they are compared to a baseline. They are instead an entirely separate test suite.