Add some tips on how to benchhmark
ClosedPublic

Authored by rafael on May 18 2017, 6:06 PM.

Details

Diff Detail

rafael created this revision.May 18 2017, 6:06 PM
inouehrs added inline comments.
docs/Benchmarking.rst
48

Do you mean SMT (e.g. HyperThreding in Intel)? If you mean CPUs in the same socket, core_siblings_list can be used (but stopping sibling SMT threads is more important.)

56

Maybe it is nice to mention about numactl command in case for benchmarking parallel programs.

58

Simple explanation on the meaning or -r option will be valuable, i.e. -r 10 will increase the execution time 10x.

emaste added a subscriber: emaste.May 19 2017, 9:42 AM
rafael updated this revision to Diff 99581.May 19 2017, 9:44 AM
emaste added inline comments.May 19 2017, 9:46 AM
docs/Benchmarking.rst
15–17

Should we have a "General" section with this note? Certainly this is true for FreeBSD as well, but it seems like it will apply everywhere.

The advice could also be in a general section, with just the specific commands specific to the OS perhaps. For example, the tmpfs explanation also applies to FreeBSD, and even on other operating systems the general advice is sound. (Actually, the command is also identical on FreeBSD for tmpfs, but probably not elsewhere.)

rafael updated this revision to Diff 99593.May 19 2017, 10:25 AM

Split out suggestions that are not linux specific.

Thanks! I will try to add an equivalent section for FreeBSD shortly after this goes in.

kristof.beyls added inline comments.
docs/Benchmarking.rst
2–4

I think that at the moment, these are mainly tips specifically for how to set up a system to reduce noisiness in benchmark results.
Therefore, I'm wondering if the title should be a bit more specific than "Benchmarking tips". But I also can't immediately come up with a title that's much better...

10

nitpick: s/to be in control of all the possible sources of noise/reduce all possible sources of noise as much as possible/?
I think that's slightly better, but don't feel strongly about this change.

26

At http://lnt.llvm.org/quickstart.html#running-tests, there are a few hints under number 3 on how to reduce noise and cope with the remaining noise better when you use LNT for benchmarking. Would it be useful to have a pointer here to there?
Maybe some of the hints there might also make sense to have here?
Or maybe some of them should just be moved here, and there can be a pointer from there to here?

30–32

Results will be less noisy with address space randomization turned off, but I continue to think that incorrectly skews experiment results.
If you recommend this, maybe you also need to recommend to align all functions to a reasonably large offset so that small code changes in one function have less of a probability of affecting code layout and the associated performance impact in another function?

rafael updated this revision to Diff 99630.May 19 2017, 3:01 PM
grimar added a subscriber: grimar.May 20 2017, 3:42 AM
sanjoy added a subscriber: sanjoy.May 21 2017, 9:32 PM
rafael updated this revision to Diff 100007.May 23 2017, 3:00 PM
rafael added a reviewer: kristof.beyls.
kristof.beyls accepted this revision.May 24 2017, 2:39 AM

Looks like a reasonable start to me (with one more nit-pick).
Let's get this in as it's valuable as it is.
It can be improved further incrementally.

docs/Benchmarking.rst
48–51

This one seems linux x86 specific. Maybe worthwhile to move the linux-x86 specific ones into a sub-section of linux?

This revision is now accepted and ready to land.May 24 2017, 2:39 AM
rafael closed this revision.May 24 2017, 9:40 AM

r303769.