Wednesday, March 21, 2007

Could Toms Hardware Guide Get Its Soul Back?

Toms Hardware Guide sold its soul back in 2002 when Intel released the Northwood version of P4. The reviews and testing in THG have been slanted in Intel's favor ever since. Strangely though, a recent article in TGDaily actually questions Intel's ethics.

There is no doubt that the "testing" at Toms Hardware Guide has been biased since 2002. Toms once commendable scepticism of Intel vanished and they've been little more than a PR site for Intel ever since. But recently, this article, Did Intel rig its integrated graphics demo against AMD? was published in TGDaily.

"but we were puzzled by how bad AMD fared. Our personal experiences have shown that the X1600 card is very capable of video playback, when configured correctly. The graphics guys at Tom's Hardware Guide are now testing out their X1600 cards and will try to duplicate Intel's results.

But we aren't the only ones who were puzzled at the side-by-side comparison because Intel showed the same demo to other journalists, journalists who have told us that they will also run their own tests. Scott Wasson, editor in chief of The Tech Report, emailed us saying that his lab guys are testing out the card and will publish the results in a future article."


I haven't seen Tom's Hardware Guide question Intel's integrity like this since Intel tried to push the flaky 1.13Ghz PIII on the market. Pressure from THG and other review sites caused Intel to pull the chips until stable versions were available six months later. But, it's been many years now since THG showed that kind of concern for consumers. If indeed THG is finally becoming serious about fair testing and reviews again it would require a number of changes from the past practices at THG.

  1. Use the Portland Group Compiler. Using Intel's compiler to test its and its competitor's products is an obvious conflict of interest.

  2. Use proper DIMMs. THG has a habit of using high speed but high latency DIMMs in its tests. This works great for Intel, especially when the FSB is overclocked. However, it works against AMD because additional DIMM speed doesn't help but with the Integrated Memory Controller AMD's processors can make good use of low latency. When THG does this you can typically find lower speed, lower latency DIMMs on NewEgg for the same or lower price than what THG used. Match each processor to the DIMMs that they need rather than putting AMD at a disadvantage by using the DIMMs that Intel needs.

  3. Compare stock with stock and overclock with overclock. It is very annoying when THG publishes a general review and the only overclocked chips are Intel. There have been many times in the past when AMD would have won nearly every benchmark but THG threw in an overclocked Intel chip. General reviews should only be stock Intel versus stock AMD because this is the way that the vast majority of these chips will be run by consumers. Then in a separate overclocking article you crank the chips up and whoever wins wins. You don't put nitrous injected, supercharged machines up against stock street machines on the drag strip and you shouldn't do it in a review.

  4. Properly load the cores. When reviewing multi-core processors there is a lot of room for confusion and THG's testing methods have only added to the confusion. Typically THG uses almost all single threaded benchmarks and then throws in some token multi-core loading at the end. This is like comparing touring buses by having them carry six passengers. The proper way of testing is to fully load every core and compare them multi-core to multi-core. Then give a comparison with single core in a separate section. This should go along with commentary of whether there is even any reason to buy a multi-core.

  5. Publish the CPU activity meters. THG should follow Tech Report's example and publish the CPU activity meters so that the readers can tell that all of the cores are fully loaded. This would make it clear at a glance just how many cores a given benchmark was using.

  6. Do both interleaved and non-interleaved testing. AMD needs NUMA for dual and quad socket systems. When the OS is non-NUMA this can impose a severe penalty on AMD processors. The best way to demonstrate the quality of the NUMA support is to do both interleaved and non-interleaved testing. If the interleaved testing is faster then the NUMA support is poor or non-existent. If the interleaved testing is slower then the OS has at least some NUMA support. As long as THG gives some initial results with both interleaved and non-interleaved then the rest of the tests can be done with whichever is better.
It is unethical to throw overclocked chips in with standard reviews because the process of overclocking is neither simple nor without risks. It is entirely possible that a less experienced user could destroy the processor trying to duplicate THG's results. Overclocking is no different from putting street mods on your car to boost horsepower. These can be done safely and give a boost in performance but these can also blow the engine. Overclocking should be done in a separate review. This would clearly divide the results into different categories for typical users and overclocking enthusiasts.

Having multi-core reviews with mostly single threaded code is clearly unethical. Although some have tried to argue that THG is merely using benchmarks that reflect typical applications even this argument is bogus. If typical applications cannot make use of the extra cores then rather than hiding this fact in the review by using single threaded code, the conclusions at the end (and probably the first page remarks) should clearly state that there is no reason to buy a multi-core. However, I have yet to see this conclusion in a THG review. You cannot insist that a multi-core CPU is worth buying while simultaneously under-testing the chips. If a given dual core or quad core processor is worth buying then it will perform better than the competition when all cores are loaded.

Let's hope that THG will change its behavior and will start doing real reviews that properly and fairly test the chips. Let's hope that THG starts caring about buyers of computer hardware again rather than the best source of advertising dollars. A new Tom's Hardware Guide would be a great benefit to the computing community; we can hope.

Addendum: I made these same suggestions to Wolfgang Gruener after he suggested that the testing at Tom's Hardware was unbiased. TGDaily -- Opinion: Is Intel copying AMD? . Let's see what happens.

24 comments:

Ho Ho said...

scientia
"Use the Portland Group Compiler"

That one I don't understand. What is wrong with GCC? The newest version is even better than ICC and as it is free and multiplatform, everyone can duplicate/check the results. Also is that Portland compiler any good compared to GCC, ICC or MSVC? Have there been any benchmarks made on the generated code efficiency?


"General reviews should only be stock Intel versus stock AMD"

I don't remember them comparing only OC'd intels against default AMD's. Though I haven't actually read their reviews during the last year or so. Nobody sais you may not ignore the OC results and just see how the stock ones fare.

Also wouldn't that mean when you make an additional OC review you'd simply copy the original article and add the results of OC'd CPUs.

"Publish the CPU activity meters"

It would be better than nothing but I'm sure you can have considerably better statistics than task manager screenshots. It is trivial in Linux to log CPU load and tens of other system parameters, it should be doable in Windows too.


"When the OS is non-NUMA this can impose a severe penalty on AMD processors"

Vista was supposed to have NUMA support, though I don't remember 4x4 benchmark showing big improvements. Should they start using Linux instead?

Also I don't remember THG making any 4P benchmarks, at least not from >1 year ago. Have they done any this year?


"It is entirely possible that a less experienced user could destroy the processor trying to duplicate THG's results"

I have yet to hear about someone frying his CPU without using voltage modifications soldered to motherboard.


I don't say they are doing fine, they sure would need a gigantic update on their benchmarking methods. They should also have more text than advertising on their pages, not the opposite. Just that some of your points are not exactly clear and well backed up in my oppinion.

Scientia from AMDZone said...

ho ho
That one I don't understand. What is wrong with GCC?

GCC is fine if you are using Linux (although PG also works with Linux). If you are compiling for Windows then you'll need to use something else. I'm sorry if you don't understand the term "conflict of interest".

I don't remember them comparing only OC'd intels against default AMD's.

Again, this is your poor reading skills; I never said "only" overclocked Intels against stock AMDs.

Nobody sais you may not ignore the OC results and just see how the stock ones fare.

Again, I've already made this point; I'm sorry you didn't understand it.

Also wouldn't that mean when you make an additional OC review you'd simply copy the original article and add the results of OC'd CPUs.

Why would you? You can have all new values for both Intel and AMD.

It would be better than nothing but I'm sure you can have considerably better

THG does not currently do even this; better would be fine.

I have yet to hear about someone frying his CPU without using voltage modifications soldered to motherboard.

It doesn't matter. Damaging the chip, making the chip unstable, or making the system slower would all be bad. All of these are possible.

I don't say they are doing fine, they sure would need a gigantic update on their benchmarking methods.

They would to have proper testing.

They should also have more text than advertising on their pages, not the opposite. Just that some of your points are not exactly clear and well backed up in my oppinion.

You didn't seem to understand several of my points so I'm not surprised you would think this.

Erlindo said...

I'd like to add that not only Toms Hardware is plagued of bias, also their forums is infested with it.

People go over there asking for advice on setting up a PC with an AMD chip and in a sudden they are attacked by the fact of not going the intel route. People over there bash AMD the whole day. Even, I can be sure that THG is the hardware site with the most presence of intel employees of all other hardware sites. :D

All the site is bias.
I'm still sceptical about any positive change in their way of thinking.

People will go over there asking for advice and instead they'll get misleaded with lies and FUD.

This has to stop!!

Wirmish said...

This is why you can't trust Intel compiler:

http://www.swallowtail.org/naughty-intel.html

gdp77 said...

Use the Portland Group Compiler. Using Intel's compiler to test its and its competitor's products is an obvious conflict of interest.

What's the industry standard? Because if it id PGC and they don't use it, then they are wrong.

Use proper DIMMs

agree

Compare stock with stock and overclock with overclock

agree. But I think they do that, unless u have a link showing otherwise. Also, the problem is that current top of the line AMD chips can't be overclocked not even by a margin of 50MHz, while Intel chips can be overclocked 1 - 1.5 GHz out of the box. Maybe u don't like that, but please get used to it.

Properly load the cores

They use what programs they consider that people use more and that they will show performance differences. I am sorry, that the majority of the programs and games people use is still single core, but that's how computer industry is. I want to see benchmarks for the programs I USE and not see benchmarks for multi-core programs i WOULD in an IDEAL world use. Anyway, whatever benchmarks they use, there is not even one benchmark that AMD would win over Intel, so what's the point?

Do both interleaved and non-interleaved testing. AMD needs NUMA for dual and quad socket systems

since when? Is this mentioned anywhere in the official AMD site? Please give us a link? If NUMA is required for AMD cpus then by all means, AMD should say so. Anyway (as usual) there in not one OS out there that AMD would have better performance comapred to C2D.


About overclocking, 16 year old kids, buy computers using their "dady" money and they want to overclock out of the box. They may fail miserably, but the point is that overclocking ability has become a test parameter when benchmarking cpu whether u like it or not. It is like something crash test with cars. In the near past, nobody was testing cars in crash tests, but now all review magazines, do. And of course no driver will go and crash test his/her car. (As some cpu buyers may never overclock).

I can see that overclocking testing "feels" unethical for AMD because they have ZERO overclocking ability now at high end, but this doesn't mean that review sites shouldn't review overcocking. Overclocking ability should ALWAYS be tested when reviewing a cpu, not in a different article, as u want.

Ho Ho said...

scientia
"If you are compiling for Windows then you'll need to use something else"

Please explain. I've used GCC under both OSes (mingw) and haven't seen a difference. Exactly whose interests are conflicting?

Also anyone can read the whole source of GCC line by line if they want to, you can't do same kind of verification on the PGC. Also PGC is no where near "industry standard". In fact, I first heard about it just a few weeks ago. I'm quite sure that GCC is more used than PGC. I think the compiler popularity goes something like MSVC>ICC>GCC>PGC. Interesting is, that the worst perfroming one is quite likely the most used one.


"Again, this is your poor reading skills; I never said "only" overclocked Intels against stock AMDs."

So where is the problem? You simply ignore the results from the OC'd systems. That was my whole point. When you want to see how fast x2 4400+ performs and you see a review with all the other x2's then you don't say that you can't use that review since it had other CPUs with different clock speeds listed besides the 4400+.


"Why would you? You can have all new values for both Intel and AMD."

You need to compare them with the base systems. If they do their tests half-decently they should get similar results for the non-OC'd systems. Where would the new values come from? Or should they run a whole new set of benchmarks? Wouldn't that make it kind of pointles when you can't directly compared the OC results with the base systems?


"Damaging the chip, making the chip unstable, or making the system slower would all be bad. All of these are possible."

All except the damagning part. You will need insane voltages (that most motherboards can't provide) or physical damage to damage the chip. When OC'ing and swapping coolers it is quite simple to damage the chip or the socket, actually.

Even older CPUs like P4 can easily survive insane situations. E.g I had one running with slight OC at full load for 24h before I noticed that heat sink was loose and CPU was in constant throttle at >100C. Interesting was there were no signs of instability, it was just slower than usual.

Azary Omega said...

Well for one i think that Tom's Hardware misGuide will never, or at least not for a long time, change their intel alignment. And the reason for this is simple, people who post there, people who read their reviews, they are pro-intel and as any business THmG will try to satisfy the customers they have. There wont be no change in how THG does their reviews and one should be pretty fullish to think otherwise.

Also i agree with HoHo's GCC remark, i don't know if GCC is better but thats what I've been using for a Long time.

Aguia said...

Erlindo:
People go over there asking for advice on setting up a PC with an AMD chip and in a sudden they are attacked by the fact of not going the intel route.

You are very right with your post.

Here is one very good example:
-One guy wanted some X2 4200+.
-Two Intel guys said to that guy the Core 2 Duo is a much better buy. (Nothing to discuss here)
-The funny part. The two Intel guys talking to each other also said they top out their CPU price at 150$. Well I ask what Core 2 Duo you can buy with 150$?

Strange isn’t it?

Greg said...

The comments about ignoring the overclocking potential of the chips, or the overclocking potential being a necessary component of chip performance are, to say the least, weak.

These articles are being read by people who don't know the specifics of what they're reading. When you look at it this way, saying a processor is better because of how it runs at a clock speed that a large number of people wont use is wrong. Also, using your own logic about what applications to use, you can't have an overclocked chip in a review. Being that no stock machine can be overclocked, and that most enthusiasts still don't overclock their machines (and there are statistics that show this), you have an extremely small minority of enthusiasts who overclock.

gdp, your logic concerning multi-core is extremely poor. Most people buy nicer machines in order to try to "future proof" their purpose. No matter how futile such a term is, making sure you get the most life out of your computer for the amount of money you spend is, hands down, the most important statistic anyone is looking at. Being that there are a lot of games that are multi-threaded (stalker, quake, supreme commander, and COD2 to name a few) and that all games to be released will be multi-threaded (alan wake, ut2k7, crysis, etc.) using single threaded gaming performance in any way is disingenous to THG's largest constituency.

Also, scientia is talking about the server benchmarks they "try" to write. These are the most pathetic out of any of their benchmarks, and I'm fairly certain no one reads them (even my balefully incompetent IT guy knew to stay away from them).

I have to agree with hoho on this one. GCC needs to be the standard they used, however, this still means that their use of Intel's compiler is a very poor and disingenous choice.

Ho ho, you'd be surprised what a poor power supply and cheap motherboard will do to a chip when you try to overclock it. I've had 2 friends fry Intel CPUs when trying to overclock their p4s. These weren't even overheating problems, but simply the power supply running into too much demand for that specific rail and sending ridiculous amount of rippling that ended up killing the proc.

Ho Ho said...

greg
"Ho ho, you'd be surprised what a poor power supply and cheap motherboard will do to a chip when you try to overclock it"

Bad PSU can kill your motherboard when you put a too big load on it. It can damage CPU only after the motherboard is (near) dead or when some catastrophic failure occurs like massive overvoltage.

Also if you have too bad PSU for your PC then it's either the fault of retailer or when you built it yourself, your own fault.

Scientia from AMDZone said...

I'm going to try one more time to explain these concepts because clearly some here are missing them completely.

1. Compiler

Using the Intel Compiler is a conflict of interest. This cannot be justified under any circumstances. If you were testing with a limited budget then GCC would be fine since it is 3rd party. However, no actual software developer is going to use GCC for Windows code. So, PGC is better since it costs almost exactly the same as the Intel Compiler and is designed to produce good code with both brands.

2. Overclocking

Overclocking is NOT a subject for casual users. Overclocking requires a bit of knowledge and is similar to putting after market enhancements on your car. There is not one car magazine in existence that reviews production cars after they've been modified. Since the VAST majority of users will run the chips stock this is the best way to review. Put the overclocking in a separate performance article; I'm fine with that even if the Intel chips clobber the AMD chips.

3. Core Loading

The argument that its okay to use almost all single threaded benchmarks is absurd. If this is all the testing you are going to do then the title of your review should be "Quad Cores a Big Waste of Money". You can't treat multi-cores like single cores and then claim that you've done any kind of real testing. Load the cores and test them. You can do single threaded testing as well but these CANNOT be just a token addition at the end of the testing. Again, if you do proper loaded testing then you can run as many single threaded benchmarks as you want and if Intel wins every single one of the single threaded benchmarks I'm fine with that. You will at least know what these chips can do under real loads which is the whole point of buying a multi-core.

4. NUMA

AMD Developer

the SMP (symmetric multiprocessing architecture) used in Intel's x86/x84 processors, memory access is via a single Northbridge memory controller for the whole computer . . .

By contrast, AMD uses a NUMA design for its multi-processor AMD Opteron™ servers.


Publish at least a couple of interleaved and non-interleaved tests to show that the review used the best method.

5. Spin

If a review site is trying to present itself as unbiased then it definitely needs to avoid doing things that work as advertizing for one brand or another.

There are literally hundreds of ways to spin a review toward one company or the other by throwing in artificial conditions. Curiously though the conditions that show up all seem to favor Intel.

For example, code with lots of cache thrashing would favor AMD because of its dedicated L2 but running single threaded code favors Intel because of its large shared cache. The testing carefully avoids situations where cache thrashing would be an issue.

It has been established that AMD chips perform well under load while Intel chips do not. The testing all but ignores loading. Legacy FP instructions are much faster on AMD chips while SSE is much faster on Intel. The testing is SSE heavy.

The VAST majority of chips run stock. Overclockers make up about 1/10th of 1% of all computer buyers. Setting up a review for this group is absurd unless you are specifically calling your site a performance/enthusiast site. Neither Anandtech nor Toms Hardware Guide labels itself this way. Yet, overclocking is a very prominent part of testing which again favors Intel.

None of these by itself would be important but when the testing conditions favor Intel again and again on each item the question of bias becomes more than suspicious.

It is clear to me that Barcelona will cancel out most of these advantages. For example it has the larger shared L3 which should negate Intel's large shared L2. K10 has much better SSE and better IPC than K8. However, review sites could still cheat by using the Intel compiler, using high latency DIMMs, by throwing in overclocked Intel chips, and avoiding loading the cores.

Jeach! said...

wirmish
http://www.swallowtail.org/naughty-intel.html

I saw that a few years back, but it's great that you post it again. What does Intelers have to say about that?

Scientia, it's great that you point it out with specifics. I think that as of late, people are starting to realize Intel's deception... including Intelers.

As for Toms Hardware, it wouldn't surprise me if he was paid by Intel. I know many believes this to be far fetch, or even impossible. But the general 80:20 rule, usually applies. This means that 80% of all viewers go to the same 20% of each site. So even if Intel was to target the top of the 20%, and pay them off, they would hit a very large audience.

But just you writing about it is a step in denouncing sites like that.

When it comes to Intel's behavior with the compiler, I think the SEC should get involved. Users should write to the SEC and complain about such abusive behavior.

But what I don't understand is if there are Intel pro benchmark sites, why isn't there the AMD pro benchmark sites? AMD sure doesn't want to play Intel's game, but why isn't there anyone else? Are they waiting for AMD to be able to 'pay them'?

Ho Ho said...

scientia
"Using the Intel Compiler is a conflict of interest"

Yes, nobody has ever said anything different.


"If you were testing with a limited budget then GCC would be fine since it is 3rd party"

You know you can get MSVC for free? I bet PGC will be given to benchmarkers for free too, it is a free advertising afterall.


"However, no actual software developer is going to use GCC for Windows code."

I wonder then who are all those people looking for help in GCC mailinglist. I haven't heard lots of indie groups having problems with porting their >100k loc projects from, say, 2.9.5 to newer GCC versions. Can you quote anyone who could back your claims up?


"So, PGC is better since it costs almost exactly the same as the Intel Compiler and is designed to produce good code with both brands"

Wow, those have to be the worst reasons I've heard not to use GCC.

FYI, ICC is designed after GCC, at least on Linux. On Windows it mimics MSVC. GCC on the other hand acts the same on all platforms and is in fact considered industry standard and it produces very good code on all platforms. Though there is one thing GCC got from Intel and that is their object file format Intel designed for Itanium. It will also give you the ability to compare how the code would work on different platforms.

Though perhaps this link is the reason why you want to see PGC used in benchmarking:

""AMD has worked closely with The Portland Group for more than two years to help ensure its leading-edge compiler and tools solutions are optimized for the AMD Opteron processor with Direct Connect Architecture," said Ben Williams, vice president, Enterprise and Server/Workstation Business, AMD's Microprocessor Business Unit, CPG"

Wouldn't you say that using PGC is as bad as using ICC? Kind of makes me doubt in your unbiasedness.


"Overclocking is NOT a subject for casual users."

Though there are programs that allow you to OC your CPU using your mouse under Windows I agree, most people don't OC their CPUs. Also most people don't buy the highest-end CPUs after which the reviewers make their conclusions.

To reiterate once again (for the fourth time?), OC results do not affect general public that buys lower end CPUs much more than benchmarks with conclusions that are based on the highest-end CPUs. Don't you agree?

Ho Ho said...

I should read my posts before I post them.


The thing about price was to show that it is irrevelant in real world.

That GCC version should have been 2.95 instead. Too many dots there :)

Scientia from AMDZone said...

ho ho
"Also PGC is no where near "industry standard". In fact, I first heard about it just a few weeks ago. I'm quite sure that GCC is more used than PGC."

Industrial Newsroom Sept. 2006
The latest release of the PGI Workstation compilers adds native support for 32-bit Windows platforms to an existing tool suite that already supports 64-bit Windows and is the reference standard on both 32-bit and 64-bit Linux systems.

ho ho
"FYI, ICC is designed after GCC, at least on Linux. On Windows it mimics MSVC."

The Portland Group Compiler dates back to 1989. Linux 1.0 was released in 1994. Intel didn't have a Linux compiler until 2000.

ICC does not mimic MSVC, it is designed to replace the compiler in MSVC and run as a plug-in.

ho ho
"GCC on the other hand acts the same on all platforms"

So does PGI.

PGI Workstation 6.2 deliver complete, uniform, optimizing, parallel C/C++ and Fortran application development environment across 32-bit and 64-bit systems based on AMD or Intel multi-core processors running Linux or Windows.

I've already said that GCC would be acceptable because it is 3rd party. That's quite enough about GCC.

ho ho
"Though perhaps this link is the reason why you want to see PGC used in benchmarking:

'AMD has worked closely with The Portland Group for more than two years to help ensure its leading-edge compiler and tools solutions are optimized for the AMD Opteron processor with Direct Connect Architecture'"


I see. So, that is what you think. Hmmm.

ho ho
"Wouldn't you say that using PGC is as bad as using ICC?"

No. I would say that you've jumped to the wrong conclusion without really knowing anything about Portland Group.

1. The link you pointed to is 3 years old; that was 2004.

2. The reason for closely working with AMD was not to optimize the compiler to favor AMD but to insure proper 64 bit support.

The Portland Group was among the first commercial compiler vendors to support x64 processors, and has aggressively optimized its compilers for successive generations of AMD processors and the recently introduced x64 processors from Intel.

In addition to extending support to native 32-bit Windows, the PGI 6.2 release includes enhancements to the unique PGI Unified Binary(TM) feature that enables a single x64 binary executable containing code sequences optimized for both AMD and Intel x64 processors, ensuring correct function and optimal performance regardless of the type of x64 processor on which applications are deployed. The PGI Unified Binary enables developers to leverage the latest processor innovations from both AMD and Intel while treating x64 as a single platform, maximizing flexibility and eliminating the need to target and optimize for two separate processor platforms.


ho ho
Kind of makes me doubt in your unbiasedness.

I'll accept an apology.

Scientia from AMDZone said...

ho ho

You implied that I wanted the Portland Group compiler because it would favor AMD. I've shown that Portland Group does not favor either Intel or AMD. You implied that I wanted the testing to be biased in AMD's favor. If that is your opinion then your opinion is no better than Red's was. If you can't apologize then don't post here.

Scientia from AMDZone said...

BTW, I made these same suggestions in a comment to a WolfGang Gruener. So far there has been no response.

Pop Catalin Sever said...

http://forumz.tomshardware.com/hardware/Open-Letter-Omid-Lost-Frakkin-Mind-ftopict231257.html

Seems that even on forumz there are people that are unhappy with TH, this was an interesting thread but there are allot more that share similar thoughts.

ashenman said...

ho ho, my point about overclocking is, most people won't figure out when that's been applied to the conlclusion, which in Tom's articles, is always the case. As such, they'll be buying a processor based on a conclusion that was, in part, based on overclockability. I don't see how this, in any way, make THG any more respectable.

Scientia from AMDZone said...

I think leaving the overclocks out of the general reviews would make the results clearer for regular users. And, putting overclocks in an Enthusiast article would make that article much more concise for OC'ers. To me it looks like a win for everyone so I still don't understand why anyone would object.

Also, I checked my mail and realized that Mr. Gruener had already emailed me with his phone number and an invitation to call. I'll see if I can get ahold of him tomorrow.

Dr. Yield, PhD, MBA said...

"Use the Portland Group Compiler"

Instead of specifying PGC, why not specify whatever the most commonly used compiler is for commercially available code? If we are talking about "typical" use cases, then we should be talking about "typical" software. Chances are, it's either MSVC or GCC. I highly doubt it is PGC or ICC.

"When the OS is non-NUMA this can impose a severe penalty on AMD processors"

Again, let's look at typical use case. If most users are not using a NUMA OS, then so be it. If AMD is selling ahead of the infrastructure they need to shine, then it sucks for them, but that's the market reality. On the other hand, if the mainstream OS for new purchases is NUMA, bully for AMD. It should be used- but not because it helps AMD or hurts Intel.

Use proper DIMMs. THG has a habit of using high speed but high latency DIMMs in its tests.
Again, most PCs are purchased with lower than best-spec memory. Stinks, doesn't it? If you want the fastest rig possible, you could cherry-pick meomory for that system. That's ok- as long as both systems in the comparison get the memory best for them. Otherwise, the shoe would be on the other foot, and Intel can claim "No fair, we're hamstrung with low speed, low latency memory!"


Properly load the cores.
Valid claim- except that for the fact that most software is still single-threaded. Again, both Intel and AMD are out ahead of the typical user here. A good test includes commercially available SW, with both single and multi-threaded apps. It might drive sales of the MT-SW if the performance is that much better, which would drive more MT development.

My problem with this article is that you are mixing and matching what you want. On one hand, you don't want anything that isn't typical user config. On the other hand, you want high-end configs with cherry-picked SW and compilers for max performance. Both are valid test configs- but they aren't the same. And the latter test config is likely to be a system that gets O/C'd- which is a whole other can of worms.

Scientia from AMDZone said...

dr. yield
Instead of specifying PGC, why not specify whatever the most commonly used compiler is for commercially available code?

GCC is fine. MSVC is fine. The problem is that some will then complain that these compilers don't produce good quality code and don't make use of advanced cpu features. This is why ICC is used. BGC is only compiler I know of that produces good quality code for both and would be in the same class as ICC.

Again, let's look at typical use case.

Most users don't have dual or quad socket systems so using the word "typical" is immediately wrong. The point is that the OS can be set to interleaved or non-interleaved and obviously you would use whichever was better.

Again, most PCs are purchased with lower than best-spec memory.

What are you talking about? This has nothing to do with typical machine memory. This is specifically about THG using faster DIMMs to simultaneously help Intel and hurt AMD. If THG wants to test with typical speed and latency DIMMs for both I'm fine with that too.

If you want the fastest rig possible, you could cherry-pick meomory for that system. That's ok- as long as both systems in the comparison get the memory best for them.

That's what I said. Use the best DIMMs for each. Why put high speed DIMMs on an AMD processor running with a stock memory clock?

My problem with this article is that you are mixing and matching what you want.

These are all common sense ideas. However, if you think that is a problem then you would have to explain why the current routine testing always favors Intel.

On one hand, you don't want anything that isn't typical user config. On the other hand, you want high-end configs with cherry-picked SW and compilers for max performance.

You've lost me. I've never talked about high end configs. I've only talked about using something equivalent to what is already being used. Also, if you have a problem with using proper DIMMs and interleaving or non-interleaving then explain why it was okay to test P4 both with HyperThreading turned on and off for best result. What I've talked about is far less effort.

Roborat, Ph. D. said...

I've always though of Tom's benchmarks to be consistent and fair. Before C2D, AMD was winning and it was the same conclusion from all other sites.
When C2D arrived it showed that Intel won the performance crown back and it was consistent with the rest of the internet benchmarks.

It's the people that turn against THG or Anand whenever their favorite CPU loses that I find inconsistent. They go and trivialize benchmark when they lose but never do so when they win. And i'm talking about both camps here.

Really, get over it.

Scientia from AMDZone said...

roborat

I've always though of Tom's benchmarks to be consistent and fair.

Which only shows that you are not an expert on benchmarking.

Before C2D, AMD was winning and it was the same conclusion from all other sites.

This had no effect on my criticism of THG. They've been doing the same bad testing for the past 4 years including when AMD was doing better in the scores.

When C2D arrived it showed that Intel won the performance crown back and it was consistent with the rest of the internet benchmarks.

There is no doubt that C2D is faster than K8 at the same clock. The problem is that because of the poor testing you can't tell exactly how much. Is C2D 10%, 15%, 20%, or 25%? Without proper testing, it's just a guess.

It's the people that turn against THG or Anand whenever their favorite CPU loses

My opinion has been against both THG and Anandtech since 2002. This has nothing to do with which processor wins or loses. Bad testing is bad testing even if AMD wins.

that I find inconsistent. They go and trivialize benchmark when they lose but never do so when they win. And i'm talking about both camps here.

I don't know exactly who are talking about but it defintely isn't me.