Page MenuHomePhabricator

[libcxx] Check in LIT benchmark testing framework.

Authored by EricWF on Mar 6 2015, 7:57 AM.



Hi All,

This is the initial commit for the benchmark testing framework I plan to use in libc++. It functions similarly to the existing LIT setup.

The benchmarks use the Google benchmark library found here:
To enable building the benchmark library use the cmake option -DLIBCXX_ENABLE_BENCHMARK=ON. This option will checkout the library from github and build it in the build/external directory.

Once the library is built the benchmarks can be run in one of two ways.

  1. Standalone (without baseline comparison): This simply runs the benchmarks and generates the output. Use -o /path/to/OUTPUT to save the results of the benchmarks.

Example usage:

lit -v -o /path/to/baseline.txt /path/to/benchmarks
  1. Comparison against baseline: This runs the benchmarks and compares the results to a baseline file specified. If the current results are slower by more than the "allowed difference" the test fails and the results are reported.

Example usage:

lit -sv -o /path/to/current_results.txt --param=baseline=/path/to/baseline.txt --param=allowed_difference=2.5 /path/to/benchmarks

The allowed_difference parameter takes the percentage which the results are allowed to differ from the baseline. The default is 5%.

The benchmark tests are not run as part of the regular test suite. They are too time consuming and do not provide much value unless they are compared to a baseline. They are instead an entirely separate test suite.

Diff Detail

Event Timeline

EricWF updated this revision to Diff 21358.Mar 6 2015, 7:57 AM
EricWF retitled this revision from to [libcxx] Checking LIT benchmark testing framework..
EricWF updated this object.
EricWF edited the test plan for this revision. (Show Details)
EricWF added a subscriber: Unknown Object (MLST).
EricWF updated this revision to Diff 21361.Mar 6 2015, 7:59 AM

Fix initial issues with patch.

EricWF updated this revision to Diff 21362.Mar 6 2015, 8:05 AM

Add missing changes to test/lit.cfg. Sorry for the noise.

Almost none of my benchmarking machines can initiate outgoing connections, so I'd need to download/checkout the Google 'benchmark' library manually. Can you please add some comments about exactly what should be retrieved, and where it should be put.

EricWF updated this revision to Diff 21375.Mar 6 2015, 10:43 AM

Address @hfinkle's comment about building from a local source directory.

You can now use -DLIBCXX_BENCHMARK_SOURCE_PATH=/path/to/benchmark/ to specify a local source directory and prevent SVN from being used.

EricWF retitled this revision from [libcxx] Checking LIT benchmark testing framework. to [libcxx] Check in LIT benchmark testing framework..Mar 6 2015, 1:00 PM
EricWF added a reviewer: titus.
mclow.lists edited edge metadata.Mar 17 2015, 7:54 AM

This all looks fine to me, but I'd like to see a page in "www" that describes how to run the benchmarks, what the options are, etc.

I'm not actually going to move forward with this patch. There are a lot of people who want to see the library in the LLVM repo.

However I am going keep this patch up for a while before abandoning in order to collect any extra comments.


I will flip this to OFF before committing.

EricWF abandoned this revision.Aug 16 2016, 2:21 AM