April 2011 browsers benchmarks

I’m getting better at this ^^. Here are new benchmarks with updated versions of the browsers, run at the beginning of April 2011. Compared to the previous browser speed test and its complement, this test features the final versions of Firefox 4 (instead of RC1), the final version of MSIE 9 (instead of the beta), and Safari 5.0.4 (instead of 5.0.3). I also added a combined score, which averages all the benchmarks I ran into a single score. Once again I remind you that browser speed benchmarks are meaningless, so read them only if you have some spare time to waste: you’d be better off trying those browsers yourself – except maybe Safari which IS really slow indeed :D.

Materials used

I used the same old hardware which is: AMD Phenom 2 X4 Mobile N930 (4x2GHz), 8 GiB DDR3 RAM, ATI Mobility Radeon HD 5650 1 GiB VRAM.
The computer was running on Windows 7 x64 Enterprise with ATI Catalyst 11.3.
For the benchmarks, I used the current versions of:

• SunSpider (version 0.9.1)
• Futuremark Peacekeeper
• Microsoft’s Mr Potato Gun
• Mozilla’s Kraken 1.0
• Google V8 (version 6)

The exact browser versions were:

• SRWare Iron Portable 10.0.650.0 (should perform more or less the same as Chrome 10)
• Opera 11.50 labs (build 24661)
• Firefox 4.0
• MSIE 9.0.8112.16421 (MSIE 9 final)
• Safari 5.0.4

Results

Combined score

The formula I used to obtain this score was the sum of relative scores where higher is better plus the sum of 1/relative scores where lower is better. So, the maximum possible combined score is 5 (which would mean that the browser beat all others in all tests). Here are the combined scores:

1. Iron 10: 4.062
2. Firefox 4: 3.678
3. Opera 11.50 nightly: 3.396
4. MSIE 9: 2.942
5. Safari 5.0.4: 2.107

Pictures!

Here are graphs with relative scores for all benchmarks. Only one picture for easier handling 😉

Score details

SunSpider

That’s the results I found most surprising: MSIE 9 really improved a lot compared to its beta, and is now first. Safari 5.0.4 on the contrary performs a lot worse than Safari 5.0.3. Firefox 4 improved pretty well too since its RC1. So the results are much different from a month ago: MSIE 9 beta was last, MSIE 9 final is first; Firefox 4 RC1 was next to last, Firefox 4 is second. Safari 5.0.3 was third but with a score close to the first, Safari 5.0.4 is last by a large margin… Confidence intervals are < +/- 0.8% except for Iron, with which even by running the test several times I couldn't achieve a confidence interval smaller than +/- 2.2%. Lower is better (the score is the time in ms to complete the test, averaged over several runs)

1. MSIE 9: 396.5
2. Firefox 4.0: 415.2
3. SRWare Iron 10: 431.3
4. Opera 11.50: 438.4
5. Safari 5.0.4: 562.4

Futuremark Peacekeeper

Nothing really new here, except that now MSIE 9 beats Firefox 4 by a tiny margin (since the results have a rather important variability, maybe the difference isn’t significant). Higher is better.

1. SRWare Iron 10: 6436
2. Opera 11.50: 5538
3. Safari 5.0.4: 3616
4. MSIE 9: 3353
5. Firefox 4.0: 3159

Microsoft’s Mr Potato Gun

There are some small changes, but this test has an even greater variability than Peacekeeper… So let’s say there’s nothing new compared to both our March tests. Higher is better.

1. Firefox 4.0: 13747
2. Opera 11.50: 10471
3. MSIE 9: 9349
4. Safari 5.0.4: 2895
5. SRWare Iron 10: 1966

Kraken 1.0

It’s funny how Iron/Chrome beats Firefox at its own game (just about the same as Firefox beats MSIE at Mr Potato ^^). Like SunSpider, the result is the time in ms to complete the test, averaged on several runs, so lower is better

1. SRWare Iron 10: 11707
2. Firefox 4.0: 15409.4
3. Opera 11.50: 25352.2
4. MSIE 9: 26024.2
5. Safari 5.0.4: 37721.1

Google V8 (v6)

My only comment goes to the poor naming choice, which will be even more confusing once they reach version 8 of their “V8” test… Like SunSpider and Kraken, Google V8 is vendor-specific, and this time there’s no surprise: Iron/Chrome indeed beats everyone else, and by a fair margin (more than twice the speed). Higher is better.

1. SRWare Iron 10: 5117
2. Firefox 4.0: 2418
3. Opera 11.50: 2087
4. Safari 5.0.4: 1635
5. MSIE 9: 1487

Conclusion

Did I already tell you that browser speed benchmarks are meaningless? When we average those 5 ones, the first thing that come to mind is that globally there isn’t a big difference between Iron 10 and Firefox 4 (just 9%), and even between Iron 10 and Opera 11.50 (16%). Even though within each particular benchmarks, differences between those browsers are usually rather large. MSIE 9, although a bit behind (28% lower than Iron 10), is now definitely not as lame as its previous versions… Even though it still “feels” somewhat slow in real life conditions (you know, the thing that matters, as opposed to benchmark conditions). Safari 5.0.4 really stands out: last in 2 tests, next to last in 2 other tests, and third (=second to last) in 1 test, it’s globally almost twice slower than Iron 10… Which is coherent with the way it feels in real use cases. I still use it (very) occasionally because sometimes it manages to display very badly programmed sites that no other browser seem to be able to run properly, but that’s getting rarer and rarer. Safari might be a good browser on Mac OS (I didn’t check that, but apparently that’s what Mac users often say), but on Windows it just has nothing for it.