Add some tips on how to benchhmark
Needs ReviewPublic

Authored by rafael on Thu, May 18, 6:06 PM.



Diff Detail

rafael created this revision.Thu, May 18, 6:06 PM
inouehrs added inline comments.

Do you mean SMT (e.g. HyperThreding in Intel)? If you mean CPUs in the same socket, core_siblings_list can be used (but stopping sibling SMT threads is more important.)


Maybe it is nice to mention about numactl command in case for benchmarking parallel programs.


Simple explanation on the meaning or -r option will be valuable, i.e. -r 10 will increase the execution time 10x.

emaste added a subscriber: emaste.Fri, May 19, 9:42 AM
rafael updated this revision to Diff 99581.Fri, May 19, 9:44 AM
emaste added inline comments.Fri, May 19, 9:46 AM

Should we have a "General" section with this note? Certainly this is true for FreeBSD as well, but it seems like it will apply everywhere.

The advice could also be in a general section, with just the specific commands specific to the OS perhaps. For example, the tmpfs explanation also applies to FreeBSD, and even on other operating systems the general advice is sound. (Actually, the command is also identical on FreeBSD for tmpfs, but probably not elsewhere.)

rafael updated this revision to Diff 99593.Fri, May 19, 10:25 AM

Split out suggestions that are not linux specific.

Thanks! I will try to add an equivalent section for FreeBSD shortly after this goes in.

kristof.beyls added inline comments.

I think that at the moment, these are mainly tips specifically for how to set up a system to reduce noisiness in benchmark results.
Therefore, I'm wondering if the title should be a bit more specific than "Benchmarking tips". But I also can't immediately come up with a title that's much better...


nitpick: s/to be in control of all the possible sources of noise/reduce all possible sources of noise as much as possible/?
I think that's slightly better, but don't feel strongly about this change.


At, there are a few hints under number 3 on how to reduce noise and cope with the remaining noise better when you use LNT for benchmarking. Would it be useful to have a pointer here to there?
Maybe some of the hints there might also make sense to have here?
Or maybe some of them should just be moved here, and there can be a pointer from there to here?


Results will be less noisy with address space randomization turned off, but I continue to think that incorrectly skews experiment results.
If you recommend this, maybe you also need to recommend to align all functions to a reasonably large offset so that small code changes in one function have less of a probability of affecting code layout and the associated performance impact in another function?

rafael updated this revision to Diff 99630.Fri, May 19, 3:01 PM
grimar added a subscriber: grimar.Sat, May 20, 3:42 AM
sanjoy added a subscriber: sanjoy.Sun, May 21, 9:32 PM