[test-suite] Add list of programs we might add.

[Meinersbur]

Summary

Add a list of benchmarks, applications and algorithms which are under discussion to be added to the test-suite.

I added all the benchmarks mentioned under https://llvm.org/PR34216, missing SPEC benchmarks, some image processing algorithms and a few others.

The list at https://llvm.org/PR34216 only allows adding to the discussion, not removing, commenting, adding details to individual benchmarks.
The file includes a comment mentioning a regular review to edit this file is not required (which would add a lot of churn). This review a discussion for the general format of the file, or whether to include such a file at all.

Suggested-by: Hal Finkel

[rengolin]

We can't add SPEC, as it's commercial. I'm not sure about others, but please make sure they are open source.

[MatzeB]

It's odd to have this in the repository, but admittedly we don't really have a wiki or similar in LLVM so I may be ok.

As we are on the topic: I think we should start discussions on breaking up the test-suite into multiple pieces/repositories.
From the technical side we can already do this today (at least with the cmake/lit mode), but we probably will need some rounds of discussions on how exactly to split things apart.

[lebedev.ri]

some image processing algorithms

I wonder if it would be of any interest to add a raw image decoding library? (the images produced by digital cameras, DSLRs)?
https://github.com/darktable-org/rawspeed

The downside is that it require the actual images to work on.
The upside to that downside is that there is a maintained set of such images exactly for this purpose already.
https://raw.pixls.us/data-unique/

[Meinersbur]

In D46714#1095014, @rengolin wrote:

We can't add SPEC, as it's commercial. I'm not sure about others, but please make sure they are open source.

I should have clarified: Regarding SPEC, I meant adding CMakeLists in the External directory.

[rengolin]

In D46714#1095029, @MatzeB wrote:

It's odd to have this in the repository, but admittedly we don't really have a wiki or similar in LLVM so I may be ok.

It's also in bugzilla (which is also odd, but ok).

As we are on the topic: I think we should start discussions on breaking up the test-suite into multiple pieces/repositories.
From the technical side we can already do this today (at least with the cmake/lit mode), but we probably will need some rounds of discussions on how exactly to split things apart.

+1

We don't do a good job at separating benchmarks in test mode and benchmark mode, and right now, they're mostly independent runs, with independent buildbots anyway.

[rengolin]

In D46714#1095034, @Meinersbur wrote:

I should have clarified: Regarding SPEC, I meant adding CMakeLists in the External directory.

SPEC can be very sensitive on how you run it, so it may be a losing battle, but I'm not against doing this, as long as it doesn't break existing downstream scripts (which there are loads).

[Meinersbur]

In D46714#1095030, @lebedev.ri wrote:

some image processing algorithms

I wonder if it would be of any interest to add a raw image decoding library? (the images produced by digital cameras, DSLRs)?
https://github.com/darktable-org/rawspeed

The downside is that it require the actual images to work on.
The upside to that downside is that there is a maintained set of such images exactly for this purpose already.
https://raw.pixls.us/data-unique/

I think it is a good candidate to be added to the file; we can remark the reason why it has not (yet) been added.

[Meinersbur]

In D46714#1095046, @rengolin wrote:

SPEC can be very sensitive on how you run it, so it may be a losing battle, but I'm not against doing this, as long as it doesn't break existing downstream scripts (which there are loads).

We already have SPEC CPU 2000/2006/2017 compile definitions in External (I myself added SPEC CPU 2017 CMakeLists.txt).
The results are not official anyway, i.e. not suitable to be submitted to https://www.spec.org/cpu2017/results/. I use them to not require configuring and invoking multiple benchmark suites.

[kristof.beyls]

It does seem like a wiki would be nice to maintain this kind of information. In the absence of that, I think that a file in the test-suite repository, or a page in www are about equally easy/hard to maintain: it requires commit access to make any changes.
A file in www in theory could be more visible as it becomes part of the llvm.org web pages. That being said, source code is also viewable online, so it's easy to browse this text too.

Next to listing future potential extensions to the test-suite, it might make sense to also have a section somewhere on test-suite design/philosophy and where we'd want the design to evolve to (e.g. a place where we can document in a bit more detail on what "breaking up the test-suite into multiple repositories" means?)

On the contents of the file as is: I wonder if it would be possible to group the proposed benchmarks by application domain, e.g. "HPC", "image processing", ...? That way it would help to identify an over-representation of some application domains and under-representation of other application domains.

TODO.txt

1–2

It might be worthwhile to also state why we want to add more applications/benchmarks/algorithms to the test-suite.
My personal take on this is roughly:
"For benchmarking, many have observed that there isn't much overlap between performance regressions observed in programs or benchmarks not included in the test-suite and the benchmarks that are in the test-suite. This an indication that the test-suite doesn't have great coverage of 'typical' performance critical code. It is also an indication that a few hundred kernels doesn't seem to be enough to be able to cover most 'typical' performance critical codes. The hope is that adding a lot more and a lot more diverse code kernels will result in more coverage."

[rengolin]

In D46714#1098954, @kristof.beyls wrote:

It does seem like a wiki would be nice to maintain this kind of information. In the absence of that, I think that a file in the test-suite repository, or a page in www are about equally easy/hard to maintain: it requires commit access to make any changes.
A file in www in theory could be more visible as it becomes part of the llvm.org web pages. That being said, source code is also viewable online, so it's easy to browse this text too.

That's actually a good point. We have the directory http://llvm.org/docs/Proposals/ for that reason.

Next to listing future potential extensions to the test-suite, it might make sense to also have a section somewhere on test-suite design/philosophy and where we'd want the design to evolve to (e.g. a place where we can document in a bit more detail on what "breaking up the test-suite into multiple repositories" means?)

This would also go into the testing docs we already have in www.

On the contents of the file as is: I wonder if it would be possible to group the proposed benchmarks by application domain, e.g. "HPC", "image processing", ...? That way it would help to identify an over-representation of some application domains and under-representation of other application domains.

+1

[lebedev.ri]

This kinda stalled, i think?

[Meinersbur]

Sorry for the delay, I haven't forgotten this patch, but did not prioritize it.

As suggested by @rengolin, I moved the document to LLVM repository's docs/Proposals. I also added a few more benchmarks.

[rengolin]

Awesome, thanks! LGTM.

I also have a list somewhere, that I will add once I find it.

[Meinersbur]

@rengolin Tanks

TODO.txt

+This file contains applications, benchmarks and algorithms
+that could be added to this test-suite.
+
+No review required for committing changes to this file such as
+ - proposing another benchmark/application/algorithms
+ - adding additional details (license, compatibility, ...)
+ - removing items that are already present in this test-suite
+
+SPEC CPU 2017
+=============
+https://www.spec.org/cpu2017/
+
+ * 503.bwaves_r/603.bwaves_s
+   Reason for non-inclusion: Fortran
+
+ * 507.cactuBSSN_r
+   Reason for non-inclusion: partially Fortran
+
+ * 521.wrf_r/621.wrf_s
+   Reason for non-inclusion: Fortran
+
+ * 527.cam4_r/627.cam4_s
+   Reason for non-inclusion: Fortran
+
+ * 628.pop2_s
+   Reason for non-inclusion: Fortran
+
+ * 548.exchange2_r/648.exchange2_s
+   Reason for non-inclusion: Fortran
+
+ * 549.fotonik3d_r/649.fotonik3d_s
+   Reason for non-inclusion: Fortran
+
+ * 554.roms_r/654.roms_s
+   Reason for non-inclusion: Fortran
+
+
+SPEC OMP2012
+============
+https://www.spec.org/omp2012/
+
+ * 350.md
+
+ * 351.bwaves
+
+ * 352.nab
+
+ * 357.bt331
+
+ * 358.botsalgn
+
+ * 359.botsspar
+
+ * 360.ilbdc
+
+ * 362.fma3d
+
+ * 363.swim
+
+ * 367.imagick
+
+ * 370.mgrid331
+
+ * 371.applu331
+
+ * 372.smithwa
+
+ * 376.kdtree
+
+
+OpenCV
+======
+https://opencv.org/
+
+
+OpenMP 4.x SIMD Benchmarks
+==========================
+https://github.com/flwende/simd_benchmarks
+
+
+PWM-benchmarking
+================
+https://github.com/tbepler/PWM-benchmarking
+
+
+SLAMBench
+=========
+https://github.com/pamela-project/slambench
+
+
+FireHose
+===========================
+http://firehose.sandia.gov/
+
+
+A Benchmark for the C/C++ Standard Library
+==========================================
+https://github.com/hiraditya/std-benchmark
+
+
+OpenBenchmarking.org CPU / Processor Suite
+==========================================
+https://openbenchmarking.org/suite/pts/cpu
+
+Subset of Phoronix Test Suite
+
+Itself a collection of benchmark suites
+
+
+Parboil Benchmarks
+==================
+http://impact.crhc.illinois.edu/parboil/parboil.aspx
+
+
+MachSuite
+=========
+https://breagen.github.io/MachSuite/
+
+
+Rodinia
+=======
+http://lava.cs.virginia.edu/Rodinia/download_links.htm
+
+
+vecmathlib rests harness
+========================
+https://bitbucket.org/eschnett/vecmathlib/wiki/Home
+
+
+PARSEC
+======
+http://parsec.cs.princeton.edu/
+
+
+Graph500 reference implementations
+===============================================
+https://github.com/graph500/graph500/tree/v2-spec
+
+
+NAS Parallel Benchmarks
+=======================
+https://github.com/benchmark-subsetting/NPB3.0-omp-C
+
+
+DARPA HPCS SSCA#2 C/OpenMP reference implementation
+===================================================
+http://www.highproductivity.org/SSCABmks.htm
+
+This web site does not exist any more, but there seems to be a copy of
+some of the benchmarks
+https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2
+
+
+Kokkos
+======
+https://github.com/kokkos/kokkos-kernels/tree/master/perf_test
+https://github.com/kokkos/kokkos/tree/master/benchmarks
+
+
+PolyMage
+========
+https://github.com/bondhugula/polymage-benchmarks
+
+
+PolyBench 
+=========
+https://sourceforge.net/projects/polybench/
+
+A modified version of Polybench 3.2 is already presented in
+SingleSource/Benchmarks/Polybench. A newer version 4.2.1 is available.
+
+
+Image processing
+================
+
+Resampling
+----------
+
+ * Bilinear
+
+ * Bicubic
+
+ * Lanczos
+
+Dither
+------
+
+ * Threshold
+
+ * Random
+
+ * Halftone
+
+ * Bayer
+
+ * Floyd–Steinberg
+
+ * Jarvis
+
+ * Stucki
+
+ * Burkes
+
+ * Sierra
+
+ * Atkinson
+
+ * Gradient-based
+
+Feature detection
+-----------------
+
+ * Harris
+
+ * Histogram of Oriented Gradients
+
+Color conversion
+----------------
+
+ * RGB to grayscale
+
+ * HSL to RGB