Driver/Rom Tweaking for Benchmarks - a short history
(1997)
In the past there have been cases where claims have been made that a vendor has modified their driver especially to produce high scores for a particular benchmark or series of tests. Obviously all vendors do this to a certain extent - as many cards are sold on the basis of benchmark scores alone, and in many cases optimizing for the benchmark results in better real world performance.

However the cases I'm talking about are not just simple driver optimization, but actual tweaking of drivers and in some cases card ROMs specifically to produce unrealistic benchmark scores. In general this is more rampant in the PC market - where stories exist of a certain big name vendor (I'll call them "Cubic Zirconium" for legal reasons) actually modified their driver and rom to detect that Winbench was running and had special rom code embedded just to produce very high scores. Their advantage vanished with the next version of Winbench, which used real world application profiling ( vs strictly graphics primitives tests or functions) to make such tweaks less likely to succeed.

To this day, many card vendors on the PC side (probably the Mac also) bypass standard graphics API calls in an attempt to deliver higher performance. However recent Winbench scores have proven that with the newer API's this in many cases no longer provides any advantage and can cause compatibility issues. Some card vendors on the PC now allow you to turn on or off these tweaks in the display setup control panels.

MacBench 4.0 uses a type of real world application profiling to attempt to thwart any attempts at this type of trickery. They went to extreme measures to emulate actual application use of the display and disk - taking into account that standard graphics function bypassing may be used. For an excellent description of this - read the MacBench FAQ from Ziff Davis. MacBench is not perfect - but those that critisize it's results are often wrong, and I've been amazed at how accurate its results equate to real world preformance.

Vendors will always try to optimize their drivers for benchmark performance, that's to their credit. The issue we're addressing here is one of attempting to unfairly skew the results of a particular test. Since the newer benchmarks are good gauges of actual applications, it makes sense for the vendor to use them as a indicator of their driver efficiency and as a guide to improving overall performance.

Note: Ziff Davis moved the MacBench page - so I now have to change all my links. As of this date (8/15/97) the page is at:

http://www1.zdnet.com/zdbop/macbench/macbench.html

I was frustrated that it took me 5 minutes to find the MacBench link at Zdnet - and then it took me to a PC only page. I was able to find it but it was not easy - even the search failed to find it.

I hear MacBench 5 is underway - and will address issues such as 8-bit graphics being used in the current tests. I also hope they consider a Mac version of 3D Bench, and excellent 3D graphics benchmark program for the PC.



Copyright © 1997. All Rights Reserved
Brand/product names mentioned here are properties of their respective companies.
Legal: Users of this web site must read
& are bound by the terms & conditions of use.

For more recent video card reviews - see www.xlr8yourmac.com/video.html.