Could Toms Hardware Guide Get Its Soul Back?

Toms Hardware Guide sold its soul back in 2002 when Intel released the Northwood version of P4. The reviews and testing in THG have been slanted in Intel's favor ever since. Strangely though, a recent article in TGDaily actually questions Intel's ethics.

There is no doubt that the "testing" at Toms Hardware Guide has been biased since 2002. Toms once commendable scepticism of Intel vanished and they've been little more than a PR site for Intel ever since. But recently, this article, Did Intel rig its integrated graphics demo against AMD? was published in TGDaily.

"but we were puzzled by how bad AMD fared. Our personal experiences have shown that the X1600 card is very capable of video playback, when configured correctly. The graphics guys at Tom's Hardware Guide are now testing out their X1600 cards and will try to duplicate Intel's results.

But we aren't the only ones who were puzzled at the side-by-side comparison because Intel showed the same demo to other journalists, journalists who have told us that they will also run their own tests. Scott Wasson, editor in chief of The Tech Report, emailed us saying that his lab guys are testing out the card and will publish the results in a future article."

I haven't seen Tom's Hardware Guide question Intel's integrity like this since Intel tried to push the flaky 1.13Ghz PIII on the market. Pressure from THG and other review sites caused Intel to pull the chips until stable versions were available six months later. But, it's been many years now since THG showed that kind of concern for consumers. If indeed THG is finally becoming serious about fair testing and reviews again it would require a number of changes from the past practices at THG.

1. Use the Portland Group Compiler. Using Intel's compiler to test its and its competitor's products is an obvious conflict of interest.


2. Use proper DIMMs. THG has a habit of using high speed but high latency DIMMs in its tests. This works great for Intel, especially when the FSB is overclocked. However, it works against AMD because additional DIMM speed doesn't help but with the Integrated Memory Controller AMD's processors can make good use of low latency. When THG does this you can typically find lower speed, lower latency DIMMs on NewEgg for the same or lower price than what THG used. Match each processor to the DIMMs that they need rather than putting AMD at a disadvantage by using the DIMMs that Intel needs.


3. Compare stock with stock and overclock with overclock. It is very annoying when THG publishes a general review and the only overclocked chips are Intel. There have been many times in the past when AMD would have won nearly every benchmark but THG threw in an overclocked Intel chip. General reviews should only be stock Intel versus stock AMD because this is the way that the vast majority of these chips will be run by consumers. Then in a separate overclocking article you crank the chips up and whoever wins wins. You don't put nitrous injected, supercharged machines up against stock street machines on the drag strip and you shouldn't do it in a review.


4. Properly load the cores. When reviewing multi-core processors there is a lot of room for confusion and THG's testing methods have only added to the confusion. Typically THG uses almost all single threaded benchmarks and then throws in some token multi-core loading at the end. This is like comparing touring buses by having them carry six passengers. The proper way of testing is to fully load every core and compare them multi-core to multi-core. Then give a comparison with single core in a separate section. This should go along with commentary of whether there is even any reason to buy a multi-core.


5. Publish the CPU activity meters. THG should follow Tech Report's example and publish the CPU activity meters so that the readers can tell that all of the cores are fully loaded. This would make it clear at a glance just how many cores a given benchmark was using.


6. Do both interleaved and non-interleaved testing. AMD needs NUMA for dual and quad socket systems. When the OS is non-NUMA this can impose a severe penalty on AMD processors. The best way to demonstrate the quality of the NUMA support is to do both interleaved and non-interleaved testing. If the interleaved testing is faster then the NUMA support is poor or non-existent. If the interleaved testing is slower then the OS has at least some NUMA support. As long as THG gives some initial results with both interleaved and non-interleaved then the rest of the tests can be done with whichever is better.

It is unethical to throw overclocked chips in with standard reviews because the process of overclocking is neither simple nor without risks. It is entirely possible that a less experienced user could destroy the processor trying to duplicate THG's results. Overclocking is no different from putting street mods on your car to boost horsepower. These can be done safely and give a boost in performance but these can also blow the engine. Overclocking should be done in a separate review. This would clearly divide the results into different categories for typical users and overclocking enthusiasts.

Having multi-core reviews with mostly single threaded code is clearly unethical. Although some have tried to argue that THG is merely using benchmarks that reflect typical applications even this argument is bogus. If typical applications cannot make use of the extra cores then rather than hiding this fact in the review by using single threaded code, the conclusions at the end (and probably the first page remarks) should clearly state that there is no reason to buy a multi-core. However, I have yet to see this conclusion in a THG review. You cannot insist that a multi-core CPU is worth buying while simultaneously under-testing the chips. If a given dual core or quad core processor is worth buying then it will perform better than the competition when all cores are loaded.

Let's hope that THG will change its behavior and will start doing real reviews that properly and fairly test the chips. Let's hope that THG starts caring about buyers of computer hardware again rather than the best source of advertising dollars. A new Tom's Hardware Guide would be a great benefit to the computing community; we can hope.

Addendum: I made these same suggestions to Wolfgang Gruener after he suggested that the testing at Tom's Hardware was unbiased. TGDaily -- Opinion: Is Intel copying AMD? . Let's see what happens.

2007: Where Are The Clock Speeds?

The most significant CPU event in 2006 was, without doubt, Intel's release of Core 2 Duo. This will be followed this year by AMD's equally significant release of K10. With a major release from both companies the world of microprocessors would be rosy indeed . . . if not for the nagging question of clock speeds.

Before Core 2 Duo was actually released in mid 2006 there were announcements that Intel would release 3.2Ghz C2D's later in the year. There were even persistent rumors by Intel enthusiasts that 3.2Ghz chips would be available at launch. The latest roadmaps seem to indicate however that even 18 months later, Intel will still be stuck at 3.0Ghz. This however doesn't seem to have dented the enthusiasm of the Intel faithful who were certain that 3.2Ghz chips would be available before the end of 2006 and that at least 3.4Ghz chips would be available before the end of 2007. Most have now simply resorted to chattering about 45nm (Penryn). Supposedly, Intel 45nm is going to allow phenomenal speeds at low power draw. And, these eager souls are hoping fervently that Intel will somehow be able to release 45nm in Q3 07, presumably to compete with AMD's K10. However, Q4 07 looks to be the earliest for 45nm from Intel and there is as yet no indication of higher clock speeds. AMD, not to be outdone, seems equally determined to deliver sub 3.0Ghz clock speeds. AMD gives no indication of getting faster than 2.9Ghz with dual core K10 on 65nm and only 2.5Ghz with quad core K10 in 2007.

So, where are the clocks? Why are AMD and Intel unable to keep increasing clock speed with 65nm and push through 3.0Ghz? One consideration is that AMD has had a history of second wave clocks. For example, the highest clock achieved on 130nm exceeded the initial clock speed of 90nm. And, the high clock on 90nm exceeded the initial speed for 65nm. If this pattern is followed then it would be logical that AMD could reach a higher clock on 65nm by the time 45nm is released. Perhaps this will be the case. The picture is much less clear with Intel since Intel has almost no track record of one architecture on multiple processes. The only crossover has been Smithfield/Presler. With these two we see no evidence at all of second wave clocks. While Smithfield topped out at 3.2Ghz on 90nm, Presler launched at 3.6Ghz.

This has, of course, happened before. In late 2002 Intel was stuck at 3.0Ghz while AMD was stuck at 2.0Ghz. AMD did manage in early 2003 to bump the Barton K7 up to 2.2Ghz and, at about the same time, Intel bumped the Northwood P4 up to 3.2Ghz. Intel eventually managed 3.46Ghz with Northwood on 90nm with its Extreme Edition in 2004. However, it certainly looks now like the clock ceiling has returned with both Intel and AMD stuck below 3.0Ghz. I suppose AMD enthusiasts might point out that AMD did have increasing clocks and only recently hit 3.0Ghz with FX-74 and 6000+ (on 90nm). The problem is that we see no indication of clocks beyond 2.9Ghz on 65nm during all of 2007 and up to Q2 08. Strangely, none of the Intel enthusiasts seem to be discussing the lack of a 3.2Ghz Conroe. I suppose if someone can forget that Intel promised a 5.0Ghz Tejas in 2005 and then promised Whitefield with CSI in 2006 then forgetting the lack of a 3.2Ghz C2D clock would be relatively easy. So, ignoring Intel's 18 month span with no clock increases, I suppose it is possible that 45nm really is much better and that Intel could deliver clock rates as high as 3.6Ghz in 2008. However, nothing like this has been indicated yet. And, with a history of second wave clocks I suppose AMD could hit 3.1Ghz on 65nm by mid 2008.

This still does not answer the question though. Since K8 is currently hitting 3.0Ghz on 90nm one would assume that 3.2Ghz would be possible on 65nm. So, why is K10 only hitting 2.9Ghz? I suppose with AMD the problem is the process. Presumably AMD can't soon get over 2.9Ghz without going above 120 watts. Looking at history for AMD this probably matches the release of K8 in 2003. K8 Sledgehammer was released at 1.8Ghz which was roughly equal to the 2.2Ghz Barton K7. This could suggest that the initial 2.3Ghz Barcelona will be roughly equal to the 2.8Ghz Santa Rosa Opteron. The speeds would also parallel the introductory speeds of dual core X2 which were down from the fastest single cores. This also suggests that the later release of 2.5Ghz Barcelona will match the current speed of 3.0Ghz K8. This would put the Q2 08 speed of 2.6Ghz ahead of the current fastest K8. However, it appears that QFX will likewise be stuck at 2.5Ghz. This does seem a bit slow to compete with Kentsfield, however, QFX is a dual socket board. Given Opteron's current near equality with Xeon at the same clock it seems somewhat possible that a dual socket 2.5Ghz QFX could keep up with a dual socket 3.0Ghz Clovertown competitor. However, this is not entirely clear since a true competitor would not use FBDIMM but DDR2 like QFX. So, perhaps Intel could stay ahead of QFX.

And, why is Intel still stuck at 3.0Ghz? In spite of claims that Intel has superior process technology the conclusion would have to be the same, that Intel is unable to clock higher. And, there is no similar history for Intel as there is with AMD. There has simply never been a time when Intel was unable to increase clock speed for 18 months. This is a year and a half with no improvement on the process. Very strange. It is possible that the early results for 45nm were so promising that Intel stopped developing 65nm. This could be the case. I also know that some will claim that Intel has lots of headroom because their chips overclock well. Unfortunately, bulk silicon chips are more prone to failure than SOI chips so Intel has to maintain significantly more margin than AMD. Simply put, careful overclocking by enthusiasts is not comparable to the same speed with a stock heatsink and fan in a standard case in a house with no air conditioning and high ambient temperatures in the summer. Still, I'm sure this myth of headroom will stay alive until Intel moves to SOI as well. If you can conveniently forget the 5.0Ghz Tejas then presumably you can also forget that Dothan, Yonah, and Prescott all overclocked significantly higher than the speeds Intel released. I suppose if Intel fans are guilty of amnesia then AMD fans would be guilty of over-optimism. I've seen people insist that K10 will have a 40% increase in Integer IPC which is purely ridiculous; 20% is much closer to reality.

In terms of single socket in 2007, AMD's competitiveness is somewhat split. For dual core, this would be a 2.9Ghz Kuma versus a 2.93Ghz Conroe. This is a trivial 1% difference in clock, so probably even. However, for quad core this would be a 2.5Ghz Agena versus a 3.0Ghz Kentsfield. This is a not so trivial 20% difference in clock. I'm guessing Intel will stay ahead on this one although I suppose being competitive with the second fastest clock (2.6Ghz) is a lot better than having no quad cores at all. All in all this should be sufficient to halt any gains in share by Intel. Additional gains in share by AMD would then probably depend on other factors like price, availability, and chipset quality. So, it does appear that 2006/2007 mirrors 2002/2003 in several ways. Perhaps we will once again see standard clocks above 3.0Ghz in 2008.

Chipsets And Chess Pieces

Coming into 2007 the chipset market has shifted dramatically with AMD's acquisition of ATI. This not only makes AMD's position considerably stronger but will have effects on Intel's chipset direction as well.

While nVidia and ATI have been slugging it out in the discrete graphics market, Intel has enjoyed a fairly secure position in terms of desktop and mobile integrated chipsets. Intel's nearest integrated competitor was ATI, however ATI was not only a distant second but often depended on orders from Intel to have significant sales. Intel has enjoyed its ability to flex its muscle in this area as it did when it stopped purchases from VIA when they would not pay bus licensing fees. This has reduced VIA from being a strong chipset and graphics player to its current status as not much more than a supplier in the budget niche market. However, with AMD's purchase of ATI Intel's position has weakened dramatically.

In one stroke, ATI not only gains secure funding for R&D to compete with nVidia but it also gains early notification of strategic direction as well as close cooperation with AMD's CPU designers. This gives the ATI division a definite advantage over nVidia and puts ATI in a category that was the sole domain of Intel's chipset and graphics division. While most computer enthusiasts will see this as competition in the discrete area this is not the big fight. I do expect ATI to gain back some of the graphics share that it lost to nVidia however this won't be as dramatic as it will be for Intel. ATI and nVidia have been competing for a long time and are very used to this; the company that isn't used to this is Intel. Having lost its integrated factory supplied chipset monopoly, Intel can expect competition to greatly increase for corporate and commercial sales.

The commercial sales area is defined much less by performance and more by cost and reliability. OEM's want the systems to be easy to configure and maintain and as low of cost as possible. This is the stronghold of the integrated graphics chipset where the graphics chip is already attached to the motherboard. ATI's presence in this market just got a big boost since ATI chipsets can now be shipped with AMD's name and official endorsement clearly stenciled on each package. There is no doubt that AMD is deadly serious about this market and is diving in with a headstart over Intel. The reason for this headstart has to do with cost.

Every chip on a motherboard costs something, and every chip connects to other chips and makes motherboard design more complex. The best way to reduce cost is to reduce the chip count. AMD already has a huge leg up on Intel because AMD motherboards don't need a Northbridge chip as all Intel motherboards do. This reduces both cost and complexity. AMD is continuing down the chip count reduction road with its plan to put the graphics chip (GPU) on the same die as the CPU. This sort of thing has been done in the past as when the MMU and FPU were integrated on the CPU die in the same fashion. AMD has went further by including the memory controller as well. So, in 2008 there should be plenty of smiles from motherboard designers who can leave off the Northbridge and GPU and design a simpler, cheaper motherboard. These smiles should be passed along to commercial OEM's and finally to commercial customers. Simple, inexpensive, easy to validate and configure. This is exactly what the market needs.

Intel however seems to have been caught off guard in this area. Presumably, it has been financing the added cost of the Northbridge with the money it has been getting from bus licenses. However, it may now not have this revenue from ATI. Intel had talked about putting the GPU inside the Northbridge which would indeed reduce the chip count. It appears that Intel has done this with the iG965. However, one has to wonder if this move is very bright when presumably the Northbridge will disappear in 2009 when Intel releases the new X86 socket with CSI and the memory controller included on the CPU die. In other words, when Intel tries to follow AMD by dropping the Northbridge then whatever work it has put into having the GPU on the Northbridge will get tossed out. This certainly isn't additive development but Intel probably has the money to waste. However, since the iG965 still has to connect to memory and the FSB of the CPU even adding the GPU doesn't reduce the complexity as much as it does on AMD motherboards. Also, it has been suggested that intel will need a switching hub to connect CSI among the processors so this would be adding a new chip on the motherboard. This too creates doubt that Intel will be able to match AMD in terms of reducing chip count and motherboard cost.

Intel may try to follow AMD by putting the GPU on the CPU die however I'm certain it will be at least a year behind AMD. Whereas AMD's K8 uses a crossbar to communicate among the HyperTransport links, the Memory Controller, and the CPU Cache, Intel's chips have no such functionality. AMD can easily hook up a GPU by adding another port to the crossbar. However, since Intel chips have no crossbar they will need to invent some method of communication. In the past this was done via Northbridge translation between the GPU and FSB of the CPU. Now that the FSB will be dropped along with the Northbridge Intel will need something entirely new. I suppose the quick and dirty approach would be to use CSI. This could work but CSI is a long way from HyperTransport's track record of robust, reliable communication. In fact, it is perhaps a somewhat embarassing secret that some 3rd party Intel motherboards use AMD's HyperTransport to talk between the North and Southbridge chips. In the BIOS this is referred to by the original name, Lightning Data Transport or LDT.

So, while Intel currently has the stronger position in terms of processors they will have to scramble to keep from having the chipset rug pulled out from under them in a determined and focused assault by AMD. AMD's drive to reduce cost and complexity by reducing chip count will be hard to match, not only by Intel but by other 3rd party chipset makers like VIA. This does leave the question of whether low end computers could be reduced in cost even further. And, since any further reductions in low end system cost along with potential increases in power could overlap with the set-top market one has to wonder if XBOX and Playstation will still be around in 2009. However, I have little doubt that by 2009 ATI's share of the discrete graphics market will be much closer to nVidia's than it is now and ATI's share of the desktop and mobile integrated markets will be much closer to Intel's.