Wednesday, February 16, 2011

On Premature Optimization

There has been a lot of discussion in my Twitter peer group lately about what's the right level of optimizing website performance and how to go about it.
Premature optimization is the root of all evil.
This sentence is true but it might also seem like an excuse to never optimize, so lets elaborate a little on the condition when and how to optimize your code.
Lets start by defining optimization for the purpose of this blog entry:
When optimizing for performance one chooses between two or more implementation strategies and picks the fastest one based on the reason that it is faster then the other strategies.
Now that we have defined performance optimization as a choice between multiple implementation strategies is becomes clear that it is just a subset of the more general concept of software design. When picking between implementation strategies in software design one has to regularly pick the worst of all evils to achieve the best possible outcome. The trade offs are:
  • Maintainability
  • Scope
  • Quality
  • Performance
You never get more than 3 of those right with limited ressources. This also means that if we want great performance we will have to be cool with worse maintainability, reduced scope and/or worse quality. As a software designer we need to be aware of this and should make a very conscious decision when we opt for great performance.
Now Steve Souders and friends have proven to great length that in the special case of website and web application load time performance can have a great positive impact on the overall success of a project, thus is will often be the right thing to opt for performance and e.g. decide to reduce scope.
Now if performance optimization can be a good thing, then why is premature optimization always evil: Because the premature refers to the fact that the performance optimization was not done because of a conscious design decision but for other reasons: E.g. because we can or because when writing that ray tracer the other day something proved to be faster.
To differentiate mature from premature optimization I already late down that conscious decisions are the key factor. What is needed for a conscious decision:
Data
When thinking about data and performance profiling comes to mind. Yes, profiling is very, very important, but I was also referring to data in terms of how the optimization will impact the scope, quality and maintainability of your application.
Unfortunately profiling is not the solution to all performance problems. Often you have to make decision early in the software design process when little code is written and you cannot really profile anything (you could micro profile but you don't know if that code is relevant to your overall performance yet). Thus besides profiling you also want to use experience to get some things right in the first place. If you don't have that experience, buy some books. Experience often also is the only way to assess how an optimization will impact things like quality or maintainability. I guess, you need a lot of experience to write good software :)
The good thing is: Computers are fast. This leads to an important rule:
When things aren't too slow, optimizing them will always be a bad idea because the the trade offs in quality, maintainability and scope will result in a net loss.
When designing your software, you should never pick an implementation strategy because it is faster unless you have data (through profiling or experience) to support your decision.
In concrete terms for JavaScript when picking between
  1. C-style loop and forEach -> use forEach.
  2. String builder and concatenation -> use concatenation
  3. Script tag VS. script loader -> use a script tag
  4. etc.
If you ever pick the faster alternative that is cool as well, just make sure that when you work on my team that you are able to back up your decision with data or I'll make you change it to the slower code with a really embarrassing commit statement :)
PS: Point 1+2 in the list above are examples where you will only ever opt to the faster version after profiling. Point 3 is a good example where your experience might tell you that the tradeoff of using a script loader is well worth the effort.

18 comments:

Michael Mahemoff said...

Good point. I went to the school of YAGNI and favour agile code above performance when beginning a project.

Just to make it interesting, would you:

w = $("#x").width();
h = $("#x").css("height");

or would you pre-cache the DOM value:

$x = $("#x");
w = $x.width();
h = $x.css("height");

On the basis of keeping code concise, I prefer the former in the first instance, but I think many developers would beg to differ.

Malte said...

I would write the latter because it is clearer that $x is the same thing in both cases. If this is the intend the code is much more maintainable and DRY. So in this case the faster code is also better in at least one other dimension.

Now it becomes a different matter if you cache $x in closure scope. Now the trade off becomes much less clear and it is much harder to say without data which one is the better alternative in several dimensions.

Peter van der Zee said...

Always treat $ queries like you would treat dom queries: hostile. That advice is library agnostic.

Nice post Malte

Tim Caswell said...

I would go on to even say that premature optimization (especially the kind without any concrete data) eventually leads to terrible performance in the long run. When a project gets large and complex, all those unneeded micro "optimizations" make the code so hard to reason about that the overall architecture gets fundamentally flawed and ends up being slow, buggy, and unmaintainable all at the same time.

Anonymous said...

Re: Mahemoff — Definitely cache the $('#x'). The overhead can be huge (depending on the library we’re talking about) and it’s more readable/maintainable that way IMHO.

Marcus Westin said...

Related thoughts on what types of metrics we should be looking for when evaluating web performance: http://marcuswest.in/read/four-performance-elements/

Oleg Slobodskoi said...

Nice article.

The bad thing is even knowing this will not help to make right decisions. As you already said the perfect balance between performance and maintainability can be only reached throughout own experience. I don't even believe this can be replaced by any smart books.

Michael Mahemoff said...

For those who say to avoid DOM lookup at all cost in my example, fine if the argument is readability/maintenability, but if it's because of performance, then it's an example of premature optimisation. An extreme one that may well be justified, but one that shows this isn't black and white for anyone.

Everyone's unique - in my case, for every 100 lines of code I write, here's my estimate of where it goes:

* 75 lines will be refactored into something else
* 10 lines will be ultimately posted a pastebin somewhere or uploaded in some obscure part of my homepage. Few people will ever try it out or see it.
* 10 lines will become a somewhat useful proof-of-concept or demo for people mainly using fast, modern, browsers, but never go beyond that.
* 5 lines will make it into a useful library or web app. This is the only code where speed - such as an extra DOM lookup - *might* matter, and even then, it will probably be unnoticeable to most users.

Of course, blatant estimate and maybe it seems weird to mix refactored-out code with "live" code, but my point is 5% of the code I write ends up being used anywhere. That's why I'm heavily biased towards making it quick and simple, and getting it out there.

To take the opposite of me, if someone was churning out code that has to run fast on IE6, and not experimenting much with new concepts, maybe 50% of their lines of code will become production code, and production code where speed matters a lot. So they might be more inclined to think about "performance in the small" (ie small idioms like this DOM lookup or Malte's for-loop example).

Malte said...

Some related advice by Brendan Eich in "A minute with..." http://www.aminutewithbrendan.com/pages/20110216

mike said...

As to your comment on string builder vs concatination concatenation is faster until string size is > 25 characters to bad you didn't benchmark this one

Malte said...

@mike: I'm not sure I follow. I wasn't suggesting that string builders are faster than concatenation or the other way around. I was merely suggesting that normally mutable strings are more maintainable than immutable strings so that you should go with mutable strings when you don't care about performance (because you haven't measured that you have to care).

Michael Mahemoff said...

To put it in perspective, a quick benchmark - looking for a specific attribute on 1000 DOM elements takes ~3ms on Chrome on a 5-year old Macbook Pro.

I'm harping on my example, but I'm also making the broader point that what people sometimes consider "evil" is more of a sentimental frustration that it's 100x slower than it could be (and therefore is sometimes called "being lazy", a quality I strive for), rather than something that actually has the slightest impact on the only thing that really matters: user experience.

Oisín said...

Good article. Optimisation decisions are not completely black and white - as you note it's a set of trade-offs.
Experience also gives us a good idea of when the cost (in other regards) of an optimisation is probably very low, in which case it's often better to write the optimised version first rather than a slow implementation to be revisited later.
When I read about premature optimisation, the message can sometimes seem like "write the slowest code that comes to mind"; of course in reality we code a little more opportunistically.

Ken said...

All the books say to use string builder over concatinization. I wanted to find out how important that really was. I built the code using concatinization. It ran in just under an hour. Put in string builder, it ran in 20 minutes. Then rewrote the code to only create the string when it was needed. Down to 7 minutes. Played around with other things, got it to 4 minutes. Got a new computer that was dual core and twice as fast. Original was running 100%, new computer ran in 2 minutes at 50%. Could change the code to multithread it, but agreed with you,it isn't worth optimizing it.

Using string builder makes the code slightly longer and in my opinion as easy to read. Taking out the redundancy did increase the complexity and therefore made it less maintainable.

I agree that optimizing to improve the performance by 2 milliseconds just isn't worth it.

I had a case where I changed a 6 tenth of a second process to 2 milliseconds. That was on a daily maintenence process that wasn't running daily because each time it ran, it took longer and longer to finish and the "daily" job was running over 48 hours.

Anything in JavaScript should NOT be CPU intensive. I agree, use concatination there. Don't optimize if you don't have a good business reason to do so.

(My business reason in the first case was just me, playing around with software. I did it to find out what impact it would have in a case where I knew there was a lot of string manipulation going on. I wanted to see what kind of impacts optimization could do for it.)

Malte said...

I made this little experiment, to see where the threshold between fast and slow is:
http://pastehtml.com/view/1dal064.html
(the implementation is wrong. should probably not use setTimeout but rather a busy wait, but it is close enough).

I agree with Michael. A mantra of "touching the DOM is evil" is not the right way. They are also the kind of problems that are easily benchmarked and optimized.

In jQuery-land I have seen much worse. People often have super optimized code that is still slow. Why? Because it didn't actually do anything on the current page. Everything started with a selector that didn't match anything. A good dependency management system need like overhead (some people see all abstractions as overhead) but would have really sped things up.

Michael Mahemoff said...

There's also an issue of not seing the forest for the trees - cluttering code with small optimisations makes it harder to make substantial (2x/10x) algorithmic improvements.

My "favourite" example in the Java world is the horrendous logger.isDebugEnabled() check, which triples or quadruples line count for a debug statement, and therefore degrades code quality. In one project, developers were blindly applying to an admin system which would be used by 2 or 3 people max at any time.

Miller Medeiros said...

I wrote something similar a couple months ago: http://blog.millermedeiros.com/2010/10/the-performance-dogma/ - I agree that it is important to do some profiling and test on the production code, sometimes benchmarks and common knowledge can be misleading and sometimes bottlenecks are hard to spot. Cheers.

Alexander said...

I am currently working on a Project that has grown over 6 months without basic jQuery optimization, or as you like to call it "premature optimization". This resulted in 4 jsFiles(each over 600 Lines long) where 85% of the lines started with a jQuery("#" + something).
This Project banned IE6, but not IE7, and after some more features, it just crashed on startup cause these scripts did so much lookups and where so unoptimized in every regard. It took hours to make those files a) easy readable(is this really the same selector?) b) easy maintainable( Try to change a selector based on 4 strings that where typed out at 90 different positions in a script vs change one variable) and have at least basic performance optimization.

I would like see "performance awareness". I agree that you should not use a negative while loop just because some perf test say its 0.5% faster then just a for. But i also think that basic maintainability is way way more important than some premature optimization.
DRY > Premature Optimization * 1e10