Lies, Damn Lies, and Statistics

One of my biggest pet peeves are benchmarks. While the intention is usually noble, the end result is almost always to mislead and misinform. In the early 90’s, as the whole video acceleration scene began to take hold in the PC industry, every manufacturer needed a way to prove the superiority of their product. What they began to do is develop drivers that were specifically attuned to a particular benchmark, and publish the ludicrous results claiming their product was the best thing since sliced bread. Some got caught. Some said everybody does it. In the end, the informed consumer could not rely on any benchmarking results. It became such a PR debacle for the graphic accelerator manufacturers that for a while no one even published a benchmark score for fear of being labeled a cheat.

As smart as we are as people, we always somehow manage to repeat history.

The whole marketing campaign Apple developed around an obscure benchmark called Bytemark shows that marketing people never learn anything about the value of truth, but rather on how they can misrepresent truths without getting caught. Bytemark is a rather ineffectual test the smart folks at Byte developed some years ago. They discovered, in fairly short order, that this benchmark is wholly inaccurate and easily skewed by compiling in different environments, so they stopped using it. [Note that one PC magazine test showed that a recompiled ByteMark with MS C++ resulted in higher than Mac scores. Bytemark is probably the farthest removed from real world (OS/App) performance benchmarks available.-Mike] But the Apple marketing folks, plagued by years of terrible, ineffectual marketing needed some marketing proof that their product is worth the extra money and nonstandard existence in a PC world- so the grasped at what was available to them. FUD is better then nothing, right? In the meantime, Mac detractors used this as proof that Apple has nothing VALID to show that their product is faster and therefore is either slower/inferior/just plain sucks/you name it. And the question still remains unanswered- how does the MacOS/PPC architecture measure up to Windows/X86?

Many different organizations have already shown that the Mac can out-Photoshop the fastest PCs. Many others showed that typical office apps run faster on PCs, and a million excuses on why that is on both. The simple truth is that there are some fundamental differences between Windows/MacOS result in a completely different user experience- some good and some bad for both platforms. On the processor side, the G3 is probably the single best implementation of data pipelining, due to its small uniform word size and its superior 2nd level cache. [Note - the Pentium II has a similar backside cache design and current Celerons and Mobile PII CPUs have 1:1 ratio caches, albeit 128K/256K resp. in size. The horribly expensive Xeon PII/PIIIs have larger 1:1 caches but are priced out of the consumer market.-Mike]. Due to this, the G3’s raw integer speed is a magnitude greater then same MHz P2 (P3 has no real improvements on Integer speed than a P2.) On the flipside, the 603 FPU (which is mostly unchanged in the G3) is inferior to even the 604 FPU, much less a P2. The cache helps this somewhat, but that’s why Quake on P2 PCs runs circles around even the fastest B/W Macs with the Rage 128.

The other quagmire is the OS itself. MacOS has a much lesser overhead processing GUI and file system operations, as its been designed as a GUI OS from the start. PPC optimizations, while STILL not complete after 7 years, have also brought significant efficiencies. When comparing equal single process applications side by side on both platforms, the Mac usually wins. [A big exception is MS Office - due I think to the visual basic/interpreted code on the Mac, which often makes Mac Excel 10x slower - see my PB G3 vs Gateway Solo Apps tests page for examples-Mike]

However, in its nature, MacOS is a single tasking OS. Since much of what we do in real life involves multiple processes, the Windows platform breaks away in general usability. A good example of this would be browsing the net, cracking RC5, playing MP3s and working on a data document (Word Processor, Spreadsheets, or anything else for that matter) simultaneously would FEEL infinitely faster and more responsive on a Windows PC, even though one-one comparisons would usually show the Mac to be faster. Microsoft Apps are very modular, which means there are quite a few libraries that are attempting to obtain the processor's attention, which is why the Mac Office apps are traditionally slower then their PC counterparts. Also, Microsoft has concentrated on different areas on Windows than Apple did with Macs. While the Apple GDI (graphic display interface) has not been significantly improved in years, Microsoft has been tirelessly working on improving the Direct-X interface to the point where its really useful to developers, allowing quicker development of better games, etc. With Apple adopting OpenGL and moving towards OS-X, these areas of PC superiority might be fading.

Ultimately, I view the Mac to be a superior work machine (which is why it is the primary production box in the studio) while PCs make much better general purpose machines (which is why thats the Everquest machine ;) This does not imply that either machine is incapable of any and all use- far from it. Today’s computer are enjoying the best user experience of all times, and it appears to be getting better- There is no better time to begin using a computer then today, regardless of what any manufacturer will have you believe with stupid benchmark scores.