Could Toms Hardware Guide Get Its Soul Back?

Toms Hardware Guide sold its soul back in 2002 when Intel released the Northwood version of P4. The reviews and testing in THG have been slanted in Intel's favor ever since. Strangely though, a recent article in TGDaily actually questions Intel's ethics.

There is no doubt that the "testing" at Toms Hardware Guide has been biased since 2002. Toms once commendable scepticism of Intel vanished and they've been little more than a PR site for Intel ever since. But recently, this article, Did Intel rig its integrated graphics demo against AMD? was published in TGDaily.

"but we were puzzled by how bad AMD fared. Our personal experiences have shown that the X1600 card is very capable of video playback, when configured correctly. The graphics guys at Tom's Hardware Guide are now testing out their X1600 cards and will try to duplicate Intel's results.

But we aren't the only ones who were puzzled at the side-by-side comparison because Intel showed the same demo to other journalists, journalists who have told us that they will also run their own tests. Scott Wasson, editor in chief of The Tech Report, emailed us saying that his lab guys are testing out the card and will publish the results in a future article."

I haven't seen Tom's Hardware Guide question Intel's integrity like this since Intel tried to push the flaky 1.13Ghz PIII on the market. Pressure from THG and other review sites caused Intel to pull the chips until stable versions were available six months later. But, it's been many years now since THG showed that kind of concern for consumers. If indeed THG is finally becoming serious about fair testing and reviews again it would require a number of changes from the past practices at THG.

1. Use the Portland Group Compiler. Using Intel's compiler to test its and its competitor's products is an obvious conflict of interest.


2. Use proper DIMMs. THG has a habit of using high speed but high latency DIMMs in its tests. This works great for Intel, especially when the FSB is overclocked. However, it works against AMD because additional DIMM speed doesn't help but with the Integrated Memory Controller AMD's processors can make good use of low latency. When THG does this you can typically find lower speed, lower latency DIMMs on NewEgg for the same or lower price than what THG used. Match each processor to the DIMMs that they need rather than putting AMD at a disadvantage by using the DIMMs that Intel needs.


3. Compare stock with stock and overclock with overclock. It is very annoying when THG publishes a general review and the only overclocked chips are Intel. There have been many times in the past when AMD would have won nearly every benchmark but THG threw in an overclocked Intel chip. General reviews should only be stock Intel versus stock AMD because this is the way that the vast majority of these chips will be run by consumers. Then in a separate overclocking article you crank the chips up and whoever wins wins. You don't put nitrous injected, supercharged machines up against stock street machines on the drag strip and you shouldn't do it in a review.


4. Properly load the cores. When reviewing multi-core processors there is a lot of room for confusion and THG's testing methods have only added to the confusion. Typically THG uses almost all single threaded benchmarks and then throws in some token multi-core loading at the end. This is like comparing touring buses by having them carry six passengers. The proper way of testing is to fully load every core and compare them multi-core to multi-core. Then give a comparison with single core in a separate section. This should go along with commentary of whether there is even any reason to buy a multi-core.


5. Publish the CPU activity meters. THG should follow Tech Report's example and publish the CPU activity meters so that the readers can tell that all of the cores are fully loaded. This would make it clear at a glance just how many cores a given benchmark was using.


6. Do both interleaved and non-interleaved testing. AMD needs NUMA for dual and quad socket systems. When the OS is non-NUMA this can impose a severe penalty on AMD processors. The best way to demonstrate the quality of the NUMA support is to do both interleaved and non-interleaved testing. If the interleaved testing is faster then the NUMA support is poor or non-existent. If the interleaved testing is slower then the OS has at least some NUMA support. As long as THG gives some initial results with both interleaved and non-interleaved then the rest of the tests can be done with whichever is better.

It is unethical to throw overclocked chips in with standard reviews because the process of overclocking is neither simple nor without risks. It is entirely possible that a less experienced user could destroy the processor trying to duplicate THG's results. Overclocking is no different from putting street mods on your car to boost horsepower. These can be done safely and give a boost in performance but these can also blow the engine. Overclocking should be done in a separate review. This would clearly divide the results into different categories for typical users and overclocking enthusiasts.

Having multi-core reviews with mostly single threaded code is clearly unethical. Although some have tried to argue that THG is merely using benchmarks that reflect typical applications even this argument is bogus. If typical applications cannot make use of the extra cores then rather than hiding this fact in the review by using single threaded code, the conclusions at the end (and probably the first page remarks) should clearly state that there is no reason to buy a multi-core. However, I have yet to see this conclusion in a THG review. You cannot insist that a multi-core CPU is worth buying while simultaneously under-testing the chips. If a given dual core or quad core processor is worth buying then it will perform better than the competition when all cores are loaded.

Let's hope that THG will change its behavior and will start doing real reviews that properly and fairly test the chips. Let's hope that THG starts caring about buyers of computer hardware again rather than the best source of advertising dollars. A new Tom's Hardware Guide would be a great benefit to the computing community; we can hope.

Addendum: I made these same suggestions to Wolfgang Gruener after he suggested that the testing at Tom's Hardware was unbiased. TGDaily -- Opinion: Is Intel copying AMD? . Let's see what happens.