Video: Windows Browser Speed Shootout - IE7, Firefox2, Opera9, Safari for Windows Beta 3
Sponsored By
So, I decided over lunch to do a non-scientific (but using common sense) test. I started by loading up each browser and resizing them to exactly 800x600. I used Camtasia to record them, independently, loading hanselman.com, apple.com, cnn.com, msdn.com, sun.com and craigslist.org.
I then downloaded a free version of Sony Vegas and made a 1600x1200 (four quadrants of 800x600) and put each site's loading in their appropriate quadrant. I started the timer (and start frame) when I press Enter in the Address Bar and stopped it when I saw a "done" indication from the browser. For some that was the word "done" in the Status Bar, for others it was a progress bar finishing and disappearing.
I did the whole thing, capture and edit in Raw 30fps and didn't make it smaller until the final Render. I deleted browser caches and pre-cached DNS for each. Each browser's file was about 1.2 gigs each.
Here's a table of the results:
| Desc | FF2 | IE7 | Opera9 | Safari3 | |
| hanselman.com | heavy, css, javascript | 3.604 | 5.939 | 4.872 | 4.872 |
| apple.com | light, lots of images | 3.670 | 4.337 | 4.27 | 1.134 |
| cnn.com | heavy, javascript, many ads | 5.339 | 2.936 | 3.337 | 2.002 |
| msdn.com | very heavy | 6.74 | 11.946 | 12.88 | 2.67 |
| sun.com | light, big image | 4.938 | 1.868 | 3.336 | 1.801 |
| craigslist.org | very light, nearly all text | 1.602 | 0.868 | 0.668 | 0.668 |
You can check the video for yourself, but the nutshell conclusion for the non-Safari browsers is "it depends on the site," but on my computer, Safari on Windows is not slow. The question is, are you willing to put up with a Mac-like interface, wonky text anti-aliasing and some weird rendering, without your FireFox plugins? I'm going to give it a try and see how it goes. It's a lot to ask.
Also, Craigslist just generally rocks. Enjoy.
About Scott
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
About Newsletter
Hosting By
June 13, 2007 2:58
Wow. 12 seconds for IE to load MSDN, and Safari does it in about 3. Crazy.
John Batdorf
Are we looking at the same video? Safari is the fastest or tied for faster on all but your site. Saying it "depends on the site" doesn't seem far to how fast Safari is right now. Also, a table with all the results would be nice.
it would be cool to get the veritest ibench suite to do our own tests on our own sites and networks -- i couldn't find it...anyone know where it is?
In IE7, did you have the phishing filter on? Any other add-ons turned on in any of the browsers?
One point that occurs to me though, unfortunately this is hard to do a truly empirical test with everything equal... you can't even be sure the same code paths are being executed (on server or client side). In other words, I could write some server side code that generated my HTML and then called Thread.Sleep() for a few seconds depending on the browser. I'm not saying this is what's happening, I'm saying we don't know. We can't tell if everything we are observing is happening within the browser process itself, and even if it is (i.e., JavaScript) the same functionality may not necessarily be being used across all browsers.
One point that occurs to me though, unfortunately this is hard to do a truly empirical test with everything equal... you can't even be sure the same code paths are being executed (on server or client side). In other words, I could write some server side code that generated my HTML and then called Thread.Sleep() for a few seconds depending on the browser. I'm not saying this is what's happening, I'm saying we don't know. We can't tell if everything we are observing is happening within the browser process itself, and even if it is (i.e., JavaScript) the same functionality may not necessarily be being used across all browsers.
Tim K - The IE7 Phishing filter was OFF in this test.
I hear what you're saying about the server-side path, but I just loaded the home pages, and they all looked the same, but yes, different ads might have been served.
I hear what you're saying about the server-side path, but I just loaded the home pages, and they all looked the same, but yes, different ads might have been served.
More specifically, I am wondering what kind of "up-level" HTML is rendered on MSDN. Safari on Windows should be a down-level browser by default (I don't know that--I'm guessing). My basic point is that in general, MSDN should be the slowest in IE because Microsoft is going to use the browser to the fullest extent possible.
Excellent point...of these sites MSDN would be the most likely to do just that. Possibly not the best site to choose, but there you go.
I believe you should also include the time spent to fire up the browser in the first place. Got some data on the load times for each?
I don't agree with the "wonky text anti-aliasing", implying bad. Jeff Atwood was on about that yesterday. I think it is personal preference. Back when I started using OS X in 2003, that was one of the first things I noticed, fonts were better on OS X than XP to my eyes/taste, no matter how much ClearType tweaking I did. I haven't spent a tremendous amount of time in Vista to see if my opinion changes, maybe that's what you guys are talking about, but the time I have spent in Vista hasn't changed my opinion.
What you could do for a bit more thorough test would be to run it again with a script (Selenium, NUnitASP, Watir) and have it browse a few pages on each site. It could run each of the websites in sequence, perhaps multiple times to even out the variations due to ads, etc. But I would be interested in recording the timings and page sizes with the test script today, in 3 months, 6 months, etc. If I were to do it I would use Selenium with the SeleniumIDE extension for FireFox but the Watir framework may have a few additional features that would help gather more detail.
I would also like to know how you used Sony Vegas to piece this together. I have it downloaded and I will see if I can put something together.
I would also like to know how you used Sony Vegas to piece this together. I have it downloaded and I will see if I can put something together.
Based on your informal test, you've done a good job proving Safari isn't slow, but I'd be more convinced if the tests were executed multiple times and averages for each site were used, particularly for MSDN. In my experience, MSDN's performance fluctuates wildly throughout the day. Safari may very well be faster, but a wider sample is needed to make up for possible statistical anomalies.
Once we've answered the question about the relative speed of Safari, the more important question is whether or not that's a compelling enough reason to use it. Mac fanboys don't even use it (at least, none of the Mac fanboys I know...).
Once we've answered the question about the relative speed of Safari, the more important question is whether or not that's a compelling enough reason to use it. Mac fanboys don't even use it (at least, none of the Mac fanboys I know...).
Flawed test, sorry.
I just tested the loading of msdn.com with FF2 and Opera9 (latest versions). The results: both load the page in 9 seconds.
I really don't know where did you get the 6 seconds difference. I tested several times and cleared the cache and all.
I just tested the loading of msdn.com with FF2 and Opera9 (latest versions). The results: both load the page in 9 seconds.
I really don't know where did you get the 6 seconds difference. I tested several times and cleared the cache and all.
Toby - I just report the news. It's true, though, that MANY folks are having huge problems with Safari not working well, or not at all, on their machines. I just filmed my experience. Not sure why you're seeing what you're seeing.
Ars Technica posted a good comparison of Safari vs. IE vs. Firefox, but they didn't do any in-depth speed tests beyond saying that Safari "offers slightly faster page loading."
I'm not sure on what you're saying here... i mean, even if it's faster, it's clear that it can't render many pages correctly. For instance, it looks like it has lots of problems with anchors. what's the purpose of being quick it it doesn't render the pages correctly?
Luis - I'm just saying it's fast. If the they get the rendering right, then it raises all of our expectations of quickness. I have a Core2Duo that takes 5 seconds to "draw" a page. How much of that is network time? DNS? Rendering? Let's get rendering to ~0 seconds. If they do that, it'll make other browsers faster.
Scott,
I am curious how Safari is setup on your machine. For some reason and it might just be me, but in the sites that I visit frequently are usually always really slow. A good example of this is slashdot, at least on my computer.
I am curious how Safari is setup on your machine. For some reason and it might just be me, but in the sites that I visit frequently are usually always really slow. A good example of this is slashdot, at least on my computer.
Something has to be wrong here, Scott. I just duplicated your msdn.com and apple.com test with IE7 and Safari beta here at home on a pristine, clean install of Windows Vista with caches and history cleared. There is nowhere close to a delta of 11.496 vs. 2.67 and 4.337 vs. 1.134 respectively. They are equivalent as far as my naked eye is concerned.
You might also want to take a look at http://msdn2.microsoft.com/en-us/library/default.aspx in both browsers as well. Safari exhibits the same result on my Mac, so can't be because it's a beta.
You might also want to take a look at http://msdn2.microsoft.com/en-us/library/default.aspx in both browsers as well. Safari exhibits the same result on my Mac, so can't be because it's a beta.
@Charlie - yes, it is weird. Of course, it's beta, and it's pretty clear that everyone is seeing VERY different behaviors one their machines. Could have been a (consistent) DNS glitch on my side.
@Brandon - I didn't change any of the Safari defaults.
@Brandon - I didn't change any of the Safari defaults.
Scott,
Thanks, there must be something really screwy then on my machine. So far I haven't been able to see anything that is really impressive with Safari. The one site that I think is really odd is how www.codezone.com looks on Safari when compared to FF2 and IE7. Also how difficult was it for you to use Sony Vegas?
Thanks, there must be something really screwy then on my machine. So far I haven't been able to see anything that is really impressive with Safari. The one site that I think is really odd is how www.codezone.com looks on Safari when compared to FF2 and IE7. Also how difficult was it for you to use Sony Vegas?
Did you tweak any of the FF2 about:config settings, or was this a vanilla install?
Thanks for putting this together. Tim's comment about ads is perhaps the most relevant. Downlevel html probably had a lot to do with the major lag you're seeing on IE for any news site.
I work on the software team at Newsweek.com, and there are some major difference between the JavaScript and ad packages that we serve for Opera and Safari when compared to IE or FF. I think the the major winner of this study is really FF. All of those sites play much more fairly with IE and FF then they do with Safari to the extent of not serving Flash for ad packages or not using Ajax for interface manipulation. Handing down those libraries and instantiating DOM objects when the page loads is going to hold open the loading state much longer.
I wonder what the benchmarks would look like if you downloaded the site html locally and ran all these test locally. That would ensure that your content was exactly the same. However, it would require a much more accurate, automatic way of stopping and starting the timer because they would the content would load much faster. That would also get rid of any unfair abnomolies with DNS or network time.
------------------
Jeff Devinian
My web site
My non profit software
I work on the software team at Newsweek.com, and there are some major difference between the JavaScript and ad packages that we serve for Opera and Safari when compared to IE or FF. I think the the major winner of this study is really FF. All of those sites play much more fairly with IE and FF then they do with Safari to the extent of not serving Flash for ad packages or not using Ajax for interface manipulation. Handing down those libraries and instantiating DOM objects when the page loads is going to hold open the loading state much longer.
I wonder what the benchmarks would look like if you downloaded the site html locally and ran all these test locally. That would ensure that your content was exactly the same. However, it would require a much more accurate, automatic way of stopping and starting the timer because they would the content would load much faster. That would also get rid of any unfair abnomolies with DNS or network time.
------------------
Jeff Devinian
My web site
My non profit software
I use IE and FF. Even if Safari is faster, I am not planning to use a third browser. Two is enough. As a web developer, developing sites that work well with IE and FF is already a pain in itself. I really hope I don't have to deal with Safari in the future
I agree MSDN is a bad choice as it will most probably serve different (usable) content to different browsers (additionally Opera uses BrowserJS to make msdn to work properly). Even worse is the Wired test using just Gmail and Google Calender. I don't know about the rest of the pages you use in your benchmark but your benchmark is the best benchmark with Safari for Windows I know until now.
For the flaws of iBench used by Apple I recommend "Safari and page load timing" by Mark "Tarquin" Wilton-Jones (in short: iBench and other Tools use JS onload as sign that the page is completly rendered while in Safari "onload fires before the page has been displayed, before layout has been calculated, before any costly reflows have taken place"). You can find some hints about browser benchmarking there as well as in Tarquin's Browser speed comparisons (although the tests themselves are said to be retired).
For the flaws of iBench used by Apple I recommend "Safari and page load timing" by Mark "Tarquin" Wilton-Jones (in short: iBench and other Tools use JS onload as sign that the page is completly rendered while in Safari "onload fires before the page has been displayed, before layout has been calculated, before any costly reflows have taken place"). You can find some hints about browser benchmarking there as well as in Tarquin's Browser speed comparisons (although the tests themselves are said to be retired).
Comments are closed.