Optimization: How much time saved is considered worth it?

Question

I was reading up about branch predictions and cache misses, and have decided to read what every programmer should know about memory. But I'm not sure how much low level knowledge do I need to have to write performant code.

One classic example is accessing the memory in the wrong way, accessing column by column when it's row-major would cause a cache miss on every lookup.

I ran a benchmark of 10 000 by 10 000 matrix, and the correct order runs in 97ms while the wrong order runs in 1535ms.

While it's evident that accessing the memory in the right order is much faster, but by how much? In this example, it's only faster by 1.44s. So a program written by a programmer without such memory knowledge is probably only penalized by up to a few seconds.

I have 2 questions.

• How much low level knowledge does a competent programmer need to have?
• How much time saved through optimization is considered worthwhile? At least a minute shaved? Half a minute? Or any amount of time saved?

Some people think that it's totally not worth it to optimize just to save a few seconds or even milliseconds. But what do you all think?

Answer 1

Its not only a question of how long a piece of string is, but what that string is used for.

Sure, spending a month to optimise a second off your code execution time is probably worthless. You could have found some slower code that was easier to optimise, or add more features. But sometimes you know something is slow and you need to modify it to make it faster and you have to experiment a bit to achieve this - for example a colleague years ago spent several months implementing a caching mechanism to our product (that was far too slow at the time). It added a 10% increase to overall speed. Then we discovered the MS_DTS setup we were using was incredibly inefficient, changed it to DCOM and achieved a 100% speed increase. What was certain was that our product was far too slow and we all needed to spend a lot of time making it faster in every way.

The other aspect is how much time is saved cumulatively - shaving a fraction of a second off a query might sound useless, but if that query is used repeatedly and often by every user, those fractions are going to add up to a fairly worthwhile server performance improvement.

So its difficult to answer your question without knowing your specific circumstances. You just need to trade off the performance impact against the cost of development.It can be useful to give a "cost to the user" to make this trade off easier to calculate - saving a second on how long it takes the start menu to show might not sound much, but if you have a billion users it might be worth the effort.