Add some tips on how to benchhmark

[inouehrs]

docs/Benchmarking.rst

48

Do you mean SMT (e.g. HyperThreding in Intel)? If you mean CPUs in the same socket, core_siblings_list can be used (but stopping sibling SMT threads is more important.)

56

Maybe it is nice to mention about numactl command in case for benchmarking parallel programs.

58

Simple explanation on the meaning or -r option will be valuable, i.e. -r 10 will increase the execution time 10x.

[emaste]

docs/Benchmarking.rst

15–17

Should we have a "General" section with this note? Certainly this is true for FreeBSD as well, but it seems like it will apply everywhere.

The advice could also be in a general section, with just the specific commands specific to the OS perhaps. For example, the tmpfs explanation also applies to FreeBSD, and even on other operating systems the general advice is sound. (Actually, the command is also identical on FreeBSD for tmpfs, but probably not elsewhere.)

[• rafael]

Split out suggestions that are not linux specific.

[emaste]

Thanks! I will try to add an equivalent section for FreeBSD shortly after this goes in.

[kristof.beyls]

docs/Benchmarking.rst

2–4

I think that at the moment, these are mainly tips specifically for how to set up a system to reduce noisiness in benchmark results.
Therefore, I'm wondering if the title should be a bit more specific than "Benchmarking tips". But I also can't immediately come up with a title that's much better...

10

nitpick: s/to be in control of all the possible sources of noise/reduce all possible sources of noise as much as possible/?
I think that's slightly better, but don't feel strongly about this change.

26

At http://lnt.llvm.org/quickstart.html#running-tests, there are a few hints under number 3 on how to reduce noise and cope with the remaining noise better when you use LNT for benchmarking. Would it be useful to have a pointer here to there?
Maybe some of the hints there might also make sense to have here?
Or maybe some of them should just be moved here, and there can be a pointer from there to here?

30–32

Results will be less noisy with address space randomization turned off, but I continue to think that incorrectly skews experiment results.
If you recommend this, maybe you also need to recommend to align all functions to a reasonably large offset so that small code changes in one function have less of a probability of affecting code layout and the associated performance impact in another function?

[kristof.beyls]

Looks like a reasonable start to me (with one more nit-pick).
Let's get this in as it's valuable as it is.
It can be improved further incrementally.

docs/Benchmarking.rst

48–51

This one seems linux x86 specific. Maybe worthwhile to move the linux-x86 specific ones into a sub-section of linux?

[• rafael]

r303769.

docs/Benchmarking.rst

+==================================
+Benchmarking tips
+==================================
+
+
+Introduction
+============
+
+For benchmarking a patch we want to reduce all possible sources of
+noise as much as possible. How to do that is very OS dependent.
+
+Note that low noise is required, but not sufficient. It does not
+exclude measurement bias. See
+https://www.cis.upenn.edu/~cis501/papers/producing-wrong-data.pdf for
+example.
+
+General
+================================
+
+* Use a high resolution timer, e.g. perf under linux.
+
+* Run the benchmark multiple times to be able to recognize noise.
+
+* Disable as many processes or services as possible on the target system.
+
+* Disable frequency scaling, turbo boost and address space
+  randomization (see OS specific section).
+
+* Static link if the OS supports it. That avoids any variation that
+  might be introduced by loading dynamic libraries. This can be done
+  by passing ``-DLLVM_BUILD_STATIC=ON`` to cmake.
+
+* Try to avoid storage. On some systems you can use tmpfs. Putting the
+  program, inputs and outputs on tmpfs avoids touching a real storage
+  system, which can have a pretty big variability.
+
+  To mount it (on linux and freebsd at least)::
+
+    mount -t tmpfs -o size=<XX>g none dir_to_mount
+
+Linux
+================================
+
+* Disable address space randomization::
+
+    echo 0 > /proc/sys/kernel/randomize_va_space
+
+* Disable turbo mode::
+
+    echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
+
+* Set scaling_governor to performance::
+
+   for i in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+   do
+     echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+   done
+
+* Use https://github.com/lpechacek/cpuset to reserve cpus for just the
+  program you are benchmarking. If using perf, leave at least 2 cores
+  so that perf runs in one and your program in another::
+
+    cset shield -c N1,N2 -k on
+
+  This will move all threads out of N1 and N2. The ``-k on`` means
+  that even kernel threads are moved out.
+
+* Disable the SMT pair of the cpus you will use for the benchmark. The
+  pair of cpu N can be found in
+  ``/sys/devices/system/cpu/cpuN/topology/thread_siblings_list`` and
+  disabled with::
+
+    echo 0 > /sys/devices/system/cpu/cpuX/online
+
+
+* Run the program with::
+
+    cset shield --exec -- perf stat -r 10 <cmd>
+
+  This will run the command after ``--`` in the isolated cpus. The
+  particular perf command runs the ``<cmd>`` 10 times and reports
+  statistics.
+
+With these in place you can expect perf variations of less than 0.1%.

docs/index.rst

.. toctree::
    Passes
    YamlIO
    GetElementPtr
    Frontend/PerformanceTips
    MCJITDesignAndImplementation
    CodeOfConduct
    CompileCudaWithLLVM
    ReportingGuide
+   Benchmarking
 
 :doc:`GettingStarted`
    Discusses how to get up and running quickly with the LLVM infrastructure.
    Everything from unpacking and compilation of the distribution to execution
    of some tools.
 
 :doc:`CMake`
    An addendum to the main Getting Started guide for those using the `CMake