7

I'm writing a program similar to Python's timeit module. The idea is to time a function by executing it anywhere from 10 to 100,000 times depending on how long it takes and then report results.

I've read that the most important number is the minimum execution time because this is the number that best reflects how fast the machine can run the code in question in the absence of other programs competing for processor time and memory. This argument makes sense to me.

Would you be happy with this?

Would you want to know the average time or the standard deviation? Is there some other measure that you consider more important?

aaronasterling
  • 385

2 Answers2

3

The first question, is does your function do the same difficulty of computation every time? Or are some inputs intrinsically harder than others? Then what matters depends upon the environment it is to be used in. If it is a realtime application, there may be a hard upper bound on acceptable execution time. For most purposes average execution time would be relevant. Note that even for an unloaded machine -say you are only running a single thread, even processes that have the same formal compuational complexity (and even execute exactly the same instructions), might have greatly differing execution times, depending upon the pre-existing state of the cache memory, state of the TLB, etc. Performance of modern machines is not simple at all. Even the execution time of a chunk of code is no longer a good metric. For every chunk of code that is executed, the state of cache, the TLB, etc are altered, so the execution speed of whatever process follows, depends in a non trivial way upon what went on before.

Omega Centauri
  • 1,048
2

You definitely need more than a single value returned.

I would see Mean, Min, Max and Standard Deviation as a minimum to use the information in a statistically meaningful manner.

There are a lot of issues that can affect performance that have variances even on a 'clean machine', even more so now that we are moving into a world with increasingly parallel processing and all the complexities with scheduling, etc that it brings.

Dan McGrath
  • 11,181
  • 6
  • 57
  • 82