[test-suite] Add list of programs we might add.

[Meinersbur]

Summary

Add a list of benchmarks, applications and algorithms which are under discussion to be added to the test-suite.

I added all the benchmarks mentioned under https://llvm.org/PR34216, missing SPEC benchmarks, some image processing algorithms and a few others.

The list at https://llvm.org/PR34216 only allows adding to the discussion, not removing, commenting, adding details to individual benchmarks.
The file includes a comment mentioning a regular review to edit this file is not required (which would add a lot of churn). This review a discussion for the general format of the file, or whether to include such a file at all.

Suggested-by: Hal Finkel

[rengolin]

We can't add SPEC, as it's commercial. I'm not sure about others, but please make sure they are open source.

[MatzeB]

It's odd to have this in the repository, but admittedly we don't really have a wiki or similar in LLVM so I may be ok.

As we are on the topic: I think we should start discussions on breaking up the test-suite into multiple pieces/repositories.
From the technical side we can already do this today (at least with the cmake/lit mode), but we probably will need some rounds of discussions on how exactly to split things apart.

[lebedev.ri]

some image processing algorithms

I wonder if it would be of any interest to add a raw image decoding library? (the images produced by digital cameras, DSLRs)?
https://github.com/darktable-org/rawspeed

The downside is that it require the actual images to work on.
The upside to that downside is that there is a maintained set of such images exactly for this purpose already.
https://raw.pixls.us/data-unique/

[Meinersbur]

In D46714#1095014, @rengolin wrote:

We can't add SPEC, as it's commercial. I'm not sure about others, but please make sure they are open source.

I should have clarified: Regarding SPEC, I meant adding CMakeLists in the External directory.

[rengolin]

In D46714#1095029, @MatzeB wrote:

It's odd to have this in the repository, but admittedly we don't really have a wiki or similar in LLVM so I may be ok.

It's also in bugzilla (which is also odd, but ok).

As we are on the topic: I think we should start discussions on breaking up the test-suite into multiple pieces/repositories.
From the technical side we can already do this today (at least with the cmake/lit mode), but we probably will need some rounds of discussions on how exactly to split things apart.

+1

We don't do a good job at separating benchmarks in test mode and benchmark mode, and right now, they're mostly independent runs, with independent buildbots anyway.

[rengolin]

In D46714#1095034, @Meinersbur wrote:

I should have clarified: Regarding SPEC, I meant adding CMakeLists in the External directory.

SPEC can be very sensitive on how you run it, so it may be a losing battle, but I'm not against doing this, as long as it doesn't break existing downstream scripts (which there are loads).

[Meinersbur]

In D46714#1095030, @lebedev.ri wrote:

some image processing algorithms

I wonder if it would be of any interest to add a raw image decoding library? (the images produced by digital cameras, DSLRs)?
https://github.com/darktable-org/rawspeed

The downside is that it require the actual images to work on.
The upside to that downside is that there is a maintained set of such images exactly for this purpose already.
https://raw.pixls.us/data-unique/

I think it is a good candidate to be added to the file; we can remark the reason why it has not (yet) been added.

[Meinersbur]

In D46714#1095046, @rengolin wrote:

SPEC can be very sensitive on how you run it, so it may be a losing battle, but I'm not against doing this, as long as it doesn't break existing downstream scripts (which there are loads).

We already have SPEC CPU 2000/2006/2017 compile definitions in External (I myself added SPEC CPU 2017 CMakeLists.txt).
The results are not official anyway, i.e. not suitable to be submitted to https://www.spec.org/cpu2017/results/. I use them to not require configuring and invoking multiple benchmark suites.

[kristof.beyls]

It does seem like a wiki would be nice to maintain this kind of information. In the absence of that, I think that a file in the test-suite repository, or a page in www are about equally easy/hard to maintain: it requires commit access to make any changes.
A file in www in theory could be more visible as it becomes part of the llvm.org web pages. That being said, source code is also viewable online, so it's easy to browse this text too.

Next to listing future potential extensions to the test-suite, it might make sense to also have a section somewhere on test-suite design/philosophy and where we'd want the design to evolve to (e.g. a place where we can document in a bit more detail on what "breaking up the test-suite into multiple repositories" means?)

On the contents of the file as is: I wonder if it would be possible to group the proposed benchmarks by application domain, e.g. "HPC", "image processing", ...? That way it would help to identify an over-representation of some application domains and under-representation of other application domains.

TODO.txt

1–2 ↗

(On Diff #146188)

It might be worthwhile to also state why we want to add more applications/benchmarks/algorithms to the test-suite.
My personal take on this is roughly:
"For benchmarking, many have observed that there isn't much overlap between performance regressions observed in programs or benchmarks not included in the test-suite and the benchmarks that are in the test-suite. This an indication that the test-suite doesn't have great coverage of 'typical' performance critical code. It is also an indication that a few hundred kernels doesn't seem to be enough to be able to cover most 'typical' performance critical codes. The hope is that adding a lot more and a lot more diverse code kernels will result in more coverage."

[rengolin]

In D46714#1098954, @kristof.beyls wrote:

It does seem like a wiki would be nice to maintain this kind of information. In the absence of that, I think that a file in the test-suite repository, or a page in www are about equally easy/hard to maintain: it requires commit access to make any changes.
A file in www in theory could be more visible as it becomes part of the llvm.org web pages. That being said, source code is also viewable online, so it's easy to browse this text too.

That's actually a good point. We have the directory http://llvm.org/docs/Proposals/ for that reason.

Next to listing future potential extensions to the test-suite, it might make sense to also have a section somewhere on test-suite design/philosophy and where we'd want the design to evolve to (e.g. a place where we can document in a bit more detail on what "breaking up the test-suite into multiple repositories" means?)

This would also go into the testing docs we already have in www.

On the contents of the file as is: I wonder if it would be possible to group the proposed benchmarks by application domain, e.g. "HPC", "image processing", ...? That way it would help to identify an over-representation of some application domains and under-representation of other application domains.

+1

[lebedev.ri]

This kinda stalled, i think?

[Meinersbur]

Sorry for the delay, I haven't forgotten this patch, but did not prioritize it.

As suggested by @rengolin, I moved the document to LLVM repository's docs/Proposals. I also added a few more benchmarks.

[rengolin]

Awesome, thanks! LGTM.

I also have a list somewhere, that I will add once I find it.

[Meinersbur]

@rengolin Tanks

docs/Proposals/TestSuite.rst

+=====================
+Test-Suite Extentions
+=====================
+
+.. contents::
+   :depth: 1
+   :local:
+
+Abstract
+========
+
+These are ideas for additional programs, benchmarks, applications and
+algorithms that could be added to the LLVM Test-Suite.
+The test-suite could be much larger than it is now, which would help us
+detecting compiler errors (crashes, miscompiles) during development.
+
+Most probably, the reason why the programs below have not been added to
+the test-suite yet is that nobody has found time to do it. But there
+might be other issues as well, such as 
+
+ * Licensing (Support can still be added as external module,
+              like for the SPEC benchmarks)
+
+ * Language (in particular, there is no official LLVM frontend
+             for FORTRAN yet)
+
+ * Parallelism (currently, all programs in test-suite use
+                one thread only)
+
+Benchmarks
+==========
+
+SPEC CPU 2017
+-------------
+https://www.spec.org/cpu2017/
+
+The following have not been included yet because they contain Fortran
+code.
+
+In case of cactuBSSN only a small portion is Fortran. The hosts's
+Fortran compiler could be used for these parts.
+
+Note that CMake's Ninja generator has difficulties with Fortran. See the
+`CMake documentation <https://cmake.org/cmake/help/v3.13/generator/Ninja.html#fortran-support>`_
+for details.
+
+ * 503.bwaves_r/603.bwaves_s
+ * 507.cactuBSSN_r
+ * 521.wrf_r/621.wrf_s
+ * 527.cam4_r/627.cam4_s
+ * 628.pop2_s
+ * 548.exchange2_r/648.exchange2_s
+ * 549.fotonik3d_r/649.fotonik3d_s
+ * 554.roms_r/654.roms_s
+
+SPEC OMP2012
+------------
+https://www.spec.org/omp2012/
+
+ * 350.md
+ * 351.bwaves
+ * 352.nab
+ * 357.bt331
+ * 358.botsalgn
+ * 359.botsspar
+ * 360.ilbdc
+ * 362.fma3d
+ * 363.swim
+ * 367.imagick
+ * 370.mgrid331
+ * 371.applu331
+ * 372.smithwa
+ * 376.kdtree
+
+OpenCV
+------
+https://opencv.org/
+
+OpenMP 4.x SIMD Benchmarks
+--------------------------
+https://github.com/flwende/simd_benchmarks
+
+PWM-benchmarking
+----------------
+https://github.com/tbepler/PWM-benchmarking
+
+SLAMBench
+---------
+https://github.com/pamela-project/slambench
+
+FireHose
+--------
+http://firehose.sandia.gov/
+
+A Benchmark for the C/C++ Standard Library
+------------------------------------------
+https://github.com/hiraditya/std-benchmark
+
+OpenBenchmarking.org CPU / Processor Suite
+------------------------------------------
+https://openbenchmarking.org/suite/pts/cpu
+
+This is a subset of the
+`Phoronix Test Suite <https://github.com/phoronix-test-suite/phoronix-test-suite/>`_
+and is itself a collection of benchmark suites
+
+Parboil Benchmarks
+------------------
+http://impact.crhc.illinois.edu/parboil/parboil.aspx
+
+MachSuite
+---------
+https://breagen.github.io/MachSuite/
+
+Rodinia
+-------
+http://lava.cs.virginia.edu/Rodinia/download_links.htm
+
+Rodinia has already been partially included in
+MultiSource/Benchmarks/Rodinia. Benchmarks still missing are:
+
+ * streamcluster
+ * particlefilter
+ * nw
+ * nn
+ * myocyte
+ * mummergpu
+ * lud
+ * leukocyte
+ * lavaMD
+ * kmeans
+ * hotspot3D
+ * heartwall
+ * cfd
+ * bfs
+ * b+tree
+
+vecmathlib tests harness
+------------------------
+https://bitbucket.org/eschnett/vecmathlib/wiki/Home
+
+PARSEC
+------
+http://parsec.cs.princeton.edu/
+
+Graph500 reference implementations
+----------------------------------
+https://github.com/graph500/graph500/tree/v2-spec
+
+NAS Parallel Benchmarks
+-----------------------
+https://www.nas.nasa.gov/publications/npb.html
+
+The official benchmark is written in Fortran, but an unofficial
+C-translation is available as well:
+https://github.com/benchmark-subsetting/NPB3.0-omp-C
+
+DARPA HPCS SSCA#2 C/OpenMP reference implementation
+---------------------------------------------------
+http://www.highproductivity.org/SSCABmks.htm
+
+This web site does not exist any more, but there seems to be a copy of
+some of the benchmarks
+https://github.com/gtcasl/hpc-benchmarks/tree/master/SSCA2v2.2
+
+Kokkos
+------
+https://github.com/kokkos/kokkos-kernels/tree/master/perf_test
+https://github.com/kokkos/kokkos/tree/master/benchmarks
+
+PolyMage
+--------
+https://github.com/bondhugula/polymage-benchmarks
+
+PolyBench 
+---------
+https://sourceforge.net/projects/polybench/
+
+A modified version of Polybench 3.2 is already presented in
+SingleSource/Benchmarks/Polybench. A newer version 4.2.1 is available.
+
+High Performance Geometric Multigrid
+------------------------------------
+https://crd.lbl.gov/departments/computer-science/PAR/research/hpgmg/
+
+RAJA Performance Suite
+----------------------
+https://github.com/LLNL/RAJAPerf
+
+CORAL-2 Benchmarks
+------------------
+https://asc.llnl.gov/coral-2-benchmarks/
+
+Many of its programs have already been integreated in
+MultiSource/Benchmarks/DOE-ProxyApps-C and
+MultiSource/Benchmarks/DOE-ProxyApps-C++.
+
+ * Nekbone
+ * QMCPack
+ * LAMMPS
+ * Kripke
+ * Quicksilver
+ * PENNANT
+ * Big Data Analytic Suite
+ * Deep Learning Suite
+ * Stream
+ * Stride
+ * ML/DL micro-benchmark
+ * Pynamic 
+ * ACME
+ * VPIC
+ * Laghos
+ * Parallel Integer Sort
+ * Havoq
+ 
+NWChem
+------
+http://www.nwchem-sw.org/index.php/Benchmarks
+
+TVM
+----
+https://github.com/dmlc/tvm/tree/master/apps/benchmark
+
+HydroBench
+----------
+https://github.com/HydroBench/Hydro
+
+ParRes
+------
+https://github.com/ParRes/Kernels/tree/master/Cxx11
+
+Applications/Libraries
+======================
+
+GnuPG
+-----
+https://gnupg.org/
+
+Blitz++
+-------
+https://sourceforge.net/projects/blitz/
+
+FFmpeg
+------
+https://ffmpeg.org/
+
+FreePOOMA
+---------
+http://www.nongnu.org/freepooma/
+
+FTensors
+--------
+http://www.wlandry.net/Projects/FTensor
+
+Generic Algorithms
+==================
+
+Image processing
+----------------
+
+Resampling
+``````````
+
+ * Bilinear
+ * Bicubic
+ * Lanczos
+
+Dither
+``````
+
+ * Threshold
+ * Random
+ * Halftone
+ * Bayer
+ * Floyd-Steinberg
+ * Jarvis
+ * Stucki
+ * Burkes
+ * Sierra
+ * Atkinson
+ * Gradient-based
+
+Feature detection
+`````````````````
+
+ * Harris
+ * Histogram of Oriented Gradients
+
+Color conversion
+````````````````
+
+ * RGB to grayscale
+ * HSL to RGB
+ 
+Graph
+-----
+
+Search Algorithms
+`````````````````
+
+ * Breadth-First-Search
+ * Depth-First-Search
+ * Dijkstra's algorithm
+ * A-Star
+
+Spanning Tree
+`````````````
+
+ * Kruskal's algorithm 
+ * Prim's algorithm

docs/index.rst

 Proposals for massive changes in how the community behaves and how the work flow
 can be better.
 
 .. toctree::
    :hidden:
 
    CodeOfConduct
    Proposals/GitHubMove
+   Proposals/TestSuite
    Proposals/VectorizationPlan
 
 :doc:`CodeOfConduct`
    Proposal to adopt a code of conduct on the LLVM social spaces (lists, events,
    IRC, etc).
 
 :doc:`Proposals/GitHubMove`
    Proposal to move from SVN/Git to GitHub.
 
+:doc:`Proposals/TestSuite`
+   Proposals for additional benchmarks/programs for llvm's test-suite.
+
 :doc:`Proposals/VectorizationPlan`
    Proposal to model the process and upgrade the infrastructure of LLVM's Loop Vectorizer.
 
 Indices and tables
 ==================
 
 * :ref:`genindex`
 * :ref:`search`