Reader Results/Notes: (later mails first)

Browser Javascript Compatibility: One of the readers that had previously sent notes/tests with nightly Webkit updates dramatically improving performance (JS benchmark scores) sent a FYI about compatibility improvements with them as well:

(added 9/8/2008)
"While everyone is marveling at how fast Chrome is at passing its own Javascript tests, I decided to see how compatible it is with the rest of the web, by running it against the Acid3 test, at http://acid3.acidtests.org/.
WebKit nightly build (any of them in the last 3 months) gets 100/100 compatibility with the web standards, and thus passes the tests.

Chrome (0.2.149.29, build 1798) gets 79/100, and fails the test.
Safari 3.1.2 gets 75 (and fails)
Firefox gets 71 (but doesn't fail)
Opera 9.5.4 gets 84 and fails

AFAIK, the only browser that passes the test, on any platform, is the nightly build of WebKit for the Mac.
Cordially, Tracy V."

A reader replied to this post saying that Safari 4.0 Developer Preview (5528.1) also passed the test with a 100/100. (There's a couple reports from last week below from users of Safari 4.0 dev. preview/beta).

Another comment on performance increases with later Webkit builds from Apple:

(added 9/8/2008)
"Just a quick tip on Webkit's javascript performance. The squirrelfish-extreme branch was just merged to the trunk (http://trac.webkit.org/changeset/36244), and has led to a dramatic improvement in the Webkit nightly builds performance. (often a nearly 3x increase and some noting almost 5x increase using later Webkit builds in reports below.-Mike) As far as I can tell, the squirrelfish-extreme add native code compilation to Webkit's javascript implementation, but that's based on me looking at the code, and I'm not a professional programmer, so take that with a grain of salt. Comparing r36130 against r36247 shows a 1.47 fold speed up, which is huge. This was tested on a MacPro with 2x2.66GHz dual core intel xeons, with 2GB RAM on 10.5.4. The results below speak for themselves. When V8 (Chrome for Mac) is easily usable on OSX, it will be interesting to see how V8 and Webkit's squirrelfish-extreme JavaScriptCore stack up against one another.
Cheers, Gavin"

There's several previous reports on Webkit update performance below as well as some Chrome results under Bootcamp/XP, Fusion and VirtualPC. (And as noted above, JS compatibility is also improved with later Webkit builds.)

(added 9/8/2008)
"I have a 2x3GHz Quad Core intel xeon Mac Pro w/10GB of ram and Leopard 10.5.4. Here are my results which seem to be really higher than anything already reported on your page (at least for webkit): (Note reports below were from late last week, using earlier builds of Webkit.-Mike)

webkit nightly (r36120 dloaded saturday 9/6/2008):
Score: 674
Richards: 756
DeltaBlue: 538
Crypto: 821
RayTrace: 466
EarleyBoyer: 895

Safari 3 Version 3.1.1 (5525.20)
(not 3.1.2 (5525.20.1) which is latest public release)
Score: 255
Richards: 142
DeltaBlue: 194
Crypto: 216
RayTrace: 372
EarleyBoyer: 493

Firefox 3.0 (not 3.0.1):
Score: 231
Richards: 234
DeltaBlue: 252
Crypto: 224
RayTrace: 174
EarleyBoyer: 288

I haven't tested with the 4.0 dev beta of safari since I had to uninstall it because of a bug with password autofill.
-Nicolas B."

Mac Pro results - OS X, WinXP/Parallels, WinXP/Bootcamp: (includes Chrome beta results as well as webkit updates for OS X and WinXP.)

(from 9/4/2008 mail - comments updated on Parallels/Chrome scores)
" Saw the (V8) browser benchmarks and compared safari to webkit to chrome on my Mac Pro. (I asked what CPU speed and to confirm OS X version was 10.5.4.-Mike)
The System I'm using is the (first model) MacPro1,1 Dual Xeon, 2.66 GHz, 5GB Ram (Win 3GB), Radeon X1900, OS X.5.4 (9E17). I'm using Parallels Desktop 3.0 (5608) English Version which is using the Bootcamp partition, so both systems should be identical.

All in all it looks like Googles V8 engine is really awesome, at least running on XP, can't check in Vista...
Have a nice weekend and thanks a lot for all the guides and tests on your site, helped me out quite a few times.
Here are the results. (I created a spreadsheet of the results for easier comparison.-Mike)

(regarding the insanely high Parallels Chrome score)
Maybe the scores are so extremely high, because it takes about 5 times as long to run as using bootcamp....
(He later wrote with more notes on Parallels tests of Chrome, noting scores/time to complete varied wildly)
Did some more testing, this time I let a stopwatch do the job and took every tests time:
Bootcamp (and any other browser): 8 seconds.
But I noticed some really strange behaviour in Parallels now:

Chrome under Parallels
Time Running - Score
12 seconds - 2399
13 seconds - 2829
20 seconds - 3846
28 seconds - 8018
30 seconds - 8013
32 seconds - 8121
33 seconds - 10004
35 seconds - 9710

So it looks like my first thought of "5 times as long means five times the score" is quite irrelevant, because it always takes a different time and the score varies too much.
(I asked if the browser was quit/restarted before each run in that last Parallels/Chrome series.-Mike)
yes, both reloading and restarting the (Chrome) browser (in Parallels) produces completely random runtimes. Checked again, can't see any pattern:

V8 test times for Chome under Parallels 3.0 (5608)
initial start
first run: 23sec
second run: 36sec
third run: 43sec

restarted chrome
first run: 32sec
second run: 34sec
third run: 33sec

rebooted windows
first run: 38sec
second run: 32sec
third run: 29sec

restarted chrome
first run: 39sec
second run: 34sec
third run: 48sec (awesome 14k score ;)

-Michael."

Safari v4 Beta, Webkit nightly builds, Chrome Beta under VirtualPC 7:

(from 9/4/2008 mail)
" I saw the article about Safari 3's V8 performance. I'm currently one of the beta testers of Safari 4. (I asked for build number and he said 5528.1) I have to say that it doesn't seem like it's a beta at all... it never crashes. Anyway, the benchmark blew me away:

(Overall) Score: 169
Richards: 131
DeltaBlue: 167
Crypto: 145
RayTrace: 142
EarleyBoyer: 301

I just tried the V8 benchmark using the latest build of webkit (r36108).
Here are the results:

Score: 229
Richards: 312
DeltaBlue: 222
Crypto: 189
RayTrace: 147
EarleyBoyer: 326

The configuration of my machine:
Power Mac G5 Dual 1.8ghz, 3GB RAM, WD 640GB HD, Nvidia 6800GT Graphics Card running OS 10.5.4.
(He later wrote)
I installed Google Chrome (beta) under emulation (not virtualization) with Virtual PC 7. Remember, emulation is much slower then anything achieved with virtualization. Running Windows XP with 512MB of memory allocated to Virtual PC 7, the results were impressive:

Score: 144
Richards: 112
DeltaBlue: 149
Crypto: 117
RayTrace: 134
EarleyBoyer: 235

Considering Virtual PC is set up to emulate a 550Mhz Pentium CPU, this is impressive.
-Brian R."

And per an article I linked in the other news on Wednesday, Chrome running natively under Windows scored over 2000 (humbling). There's also some impressive Chrome scores here from Bootcamp/XP users and VirtualMachine (Fusion, Parallels) users - although the parallels score for Chrome was insanely high (unrealistic).

MacBook Pro 2.2GHz:

" Here are my results from the Google JS benchmarks for recent browsers on my 2.2GHz MacBook Pro and 10.5.4:

FF 3.0.1
Score: 172
Richards: 176
DeltaBlue: 193
Crypto: 165
RayTrace: 134
EarleyBoyer: 200

FF 3.1Alpha (jit=true) (nightly 2008-09-04)
Score: 140
Richards: 105
DeltaBlue: 117
Crypto: 142
RayTrace: 125
EarleyBoyer: 250

FF 3.1Alpha (jit=false) (nightly 2008-09-04)
Score: 162
Richards: 127
DeltaBlue: 134
Crypto: 188
RayTrace: 141
EarleyBoyer: 248

Webkit 2008-09-04
Score: 490
Richards: 558
DeltaBlue: 393
Crypto: 603
RayTrace: 323
EarleyBoyer: 662

Looks like the new Webkit is still the fastest by far on a Mac according to this benchmark...
-Jason"

G5 2.5GHz Quad/OS X 10.5.4 (w/Webkit nightly build updates):

" WebKit (for Safari) nightly build is here: http://nightly.webkit.org/. (Just download it, drop into apps, doubleclick. It will launch Safari, but Safari will be running the nightly build, not the built-in version of WebKit.)
latest nightly build results on V8 Benchmark Suite:

(Webkit build #?)
Score: 305
Richards: 414
DeltaBlue: 296
Crypto: 263
RayTrace: 188
EarleyBoyer: 439

(he had previously wrote with earlier results (webkit users please include build number, as it changes frequently. Thanks))
And... using the latest WebKit (which I've been doing for months now) the scores are:

using previous Webkit build)
Score: 312
Richards: 439
DeltaBlue: 315
Crypto: 262
RayTrace: 183
EarleyBoyer: 445

...pretty much blowing everything else out of the water...(PPC Quad, 10.5.4) (Assuming G5 tower 2.5GHz Quad)

FireFox 3.01
Score: 115
Richards: 140
DeltaBlue: 147
Crypto: 71
RayTrace: 98
EarleyBoyer: 141

Opera 9.5
Score: 118
Richards: 68
DeltaBlue: 80
Crypto: 58
RayTrace: 207
EarleyBoyer: 351

Safari (3.1.2):
Score: 106
Richards: 64
DeltaBlue: 90
Crypto: 65
RayTrace: 158
EarleyBoyer: 221

Cordially, Tracy V."

OS X and VMware Fusion (XP): (updated to note Mac model/speed and OS X version)

"I found the browser benchmarks interesting - so I did some comparisons of my own. Using the same v8 test page as Lars did, here are some interesting statistics.
In Mac OS X 10.5 (10.5.4 he later said) and latest versions of all software.
(What is your Mac Model/CPU speed, etc.? (good to include as speeds/cpu types, etc. influence scores - although I understand your point is to show how various browsers compared on the same machine.-Mike)
(2007) AL iMac 2.8GHz core2 Duo running OS X 10.5.4.

These provided to give some reference for the windows versions - the score listed is the "total" score reported by the V8 test page. These just included for comparison to the windows equivalent:

• Safari 3.1.2 - 170
• Firefox 3.0.1 - 194
• Camino 1.6.3 - 121

Running Windows XP via VMWare Fusion on the same computer to get Chrome comparisons

• Firefox 3.0.1 - 145
• Safari 3.1.2 - 104
• IE 7.0.5730.11 - 18 (not a typo, eighteen - IE actually stopped running the tests halfway through asking me if I wanted to continue because it was slowing the browser down, although the tests before the dialog were scored in the teens as well so I don't know that it affected the total result)
• Chrome 0.2.149.7 - 1684 (also not a typo, one thousand six hundred eighty four)

The time that it took to actually run the script tests was not really different, the tests were all completed in all browsers in approximately 10 seconds - I didn't time it with a stopwatch, but none of the browsers were noticeably slower or faster than the other (other than the IE, so I don't know if the virtualization is throwing the scores off (which I don't think so, the safari/firefox scores are pretty much inline), or if there is that much difference between the browsers, or if the test site has some specific browser optimizations to skew the results for specific browsers...
Regards, Brian B."

3GHz C2D/Windows XP, G5 1.8GHz/OS X 10.4.11:

(updated with later Webkit build results)
"V8 results running various browsers on my PMac G5 1.8GHz and my home made Intel PC running a 3GHz E8400 Core 2 Duo, with configuration data on both. It may be simpler to just link to two .jpg summary pictures I've created at:
Intel Core2 Duo 3GHz, 4GB Rram - Win XP Pro SP2
(Here's a summary of overall scores on the 3GHz C2D/WinXP system)

• Chrome - 2564
• Opera 9.52 - 371
• Netscape 9.0.0.6 - 127
• Safari 3.1 - 251
• IE 8 beta2 - 63
• Firefox 3.0.1 - 281

PPC G5 1.8GHz, 5.5GB RAM, OS X 10.4.11
(Here's a summary of overall scores on the G5 1.8GHz/5.5GB Ram/OS X 10.4.11 system)

• Safari 3.1.2 - 68
• Camino 1.6.3 - 50
• OmniWeb 5.7 - 33
• iCab 4.1.1b62 - 69
• Opera 9.52 - 82
• DeerPark G5 1.5 - 40
• Netscape 9.0.0.6 - 48

(Late Friday (9/5) He wrote)
I totally forgot about the WebKit nightly builds and it does make a big difference in Safari's speed, tested with V8. Here's the results on my G5 launching the nightly build R36108:

Safari 3.1.2 ----- Safari/WebKit r36108

Score:        68 --- 296
Richards:     42 --- 225
DeltaBlue:    61 --- 182
RayTrace:    100 --- 136
EarleyBoyer: 132 --- 296

Warm regards, Rod"

2GHz iMac C2D/Safari 4.0 Preview/Firefox 3.0.1:

" Just did the test on a iMac 2Ghz Core 2 Duo with Safari 4.0 Developer Preview. (OS X 10.5.4, 2GB RAM)

Safari 4.0 dev. preview:
Score: 252
Richards: 156
DeltaBlue: 216
Crypto: 213
RayTrace: 257
EarleyBoyer: 554

Firefox 3.0.1:
Score: 152
Richards: 136
DeltaBlue: 176
Crypto: 144
RayTrace: 122
EarleyBoyer: 197

-Ruben "

PB G4 1.67GHz running OS X Tiger:

(from 9/4/2008 news page - updated 9/6 with webkit update scores)
" While us Mac users can't use Chome yet, we can test out current Mac browsers on Google's own JavaScript suite that's linked to in one of the Chrome articles you had linked on the news page (yesterday). (V8 Benchmark Suite v1 and also some (windows) Browser speed comparisons with Chrome-Mike)
Tests run are:

Richards (OS kernel simulation benchmark. 539 lines)

DeltaBlue (One-way constraint solver. 880 lines)

Crypto (Encryption/decryption benchmark. 1689 lines)

RayTrace (Ray tracer benchmark. 3418 lines)

EarleyBoyer (Classic Scheme benchmarks. 4682 lines)

Here are my results from a 1.67GHz G4 Powerbook running 10.4.11 with Better Performance energy setting. (Minefield 3.1b1pre was latest from Mozilla as of yesterday) (I asked if he enabled javascript.options.jit.content in the About:config of FF 3.1beta. They promised improved JS performance with FF 3.1, but his results showed a lower overall score with the 3.1beta vs FF 3.0.1. Lars later wrote he hadn't changed that setting from the default.-Mike)
Higher scores are better:

... I'm now really curious how the Chrome beta, Safari 4.0, and Firefox 3.1 will do. If I can get access to my sister's dual core Vista notebook, I will try out the Chrome beta and Firefox 3.0.1 and maybe Safari with the suite.
(he later sent results with the Sept. 5th webkit build - note the huge boost in scores compared to previous results above.)

Score: 172
Richards: 251
DeltaBlue: 166
Crypto: 180
RayTrace: 90
EarleyBoyer: 222

Regards, Lars "