Supercomputing -- Catching The Last Wave

I had been waiting for the 11/06 Top 500 figures for Supercomputers. I had been expecting AMD to do well and this is what the figures show. However, the figures also show a worse situation for both Intel and IBM than I had expected. Since 2003 we've seen cycles of replacement in HPC but I think X86-64 will be the final cycle. And, the clear leader of that cycle is AMD.

We can see that traditional supercomputing architectures from Cray, NEC, Hitachi, and Fujitsu have been on a slow but steady decline since 2003 as shown with the dashed line marked "Super". We can also see that the once dominant RISC systems of Alpha, Pa-RISC, MIPS, and SPARC have plummeted during the same time frame as shown with the brown line marked "RISC Other". This is to distinguish the other RISC's from IBM's Power series. We can see the rise and fall of Itanium. Power seems to be a shallower mirror image of Itanium with a smaller dip and then a rise. However, the real story is the trend for Intel X86.

It is clear that the Intel 32 bit systems of the old workhorse P4 Xeon swept in to replace the vanishing RISC architectures. It is common sense that when Intel 32 began to fall in 06/05 that Intel 64 would rise to replace it. And, by the graph it is clear that Intel 64 systems shot up at an incredible rate. However, 11/05 doesn't follow common sense at all. Intel 32 systems should still be falling and Intel 64 should be rising, but we don't see that. It can't be a general trend because we see that AMD 64 is moving up nicely. What appears to have happened is that having sampled the Intel 64 systems in 06/05 the market somewhat recoiled and fell back on the older Intel 32 bit systems. Again, it appears that the market sampled Intel 64 in 06/06 but once again found it lacking in some fashion and this shows as a decline in 11/06. The most likely culprit is the high power draw of the Prescott based Intel 64 systems.

With AMD the picture is quite different. In 06/04 the older RISC based systems were falling rapidly. This share was picked up mostly by Intel 32 and AMD 64 with Itanium taking share from traditional supercomputing systems and IBM's Power. Six months later, AMD is flat while Intel 32 continues to absorb share from the still falling RISC systems and Itanium continues to take Power share. Six months later in 06/05 AMD is still flat as Intel 64 takes off like a bullet taking share from the falling Intel 32 and RISC systems. Power appears to have reversed the trend and is taking back share from Itanium. Just six months later this all changes. In 11/05 Intel 64 has lost momentum and Itanium plummets even as RISC and traditional SC systems continue to decline.

Clearly when we look at the last 18 months of supercomputing AMD 64 stands out as the leader of the pack. Intel 32 has peaked and is falling; there are no indications of a revival from traditional SC, RISC, or Itanium. This only leaves Power, Intel 64, and AMD 64 to take up the slack from the fading Intel 32 systems. For the past 18 months AMD has risen steadily. Intel had appeared to be a contender but the fall in the last six months along with the earlier slowdown casts serious doubt on this. AMD's position is also bolstered looking forward by three announced large scale HPC wins while there have been no similar wins for Intel. With AMD's dominance in 4-way systems and its one year lead with Torrenza over Intel's Geneseo initiative it doesn't appear that this trend will change anytime soon. Intel won't have a real quad system for its C2D architecture until it releases a new chipset in late 2007. However, this chipset won't be suitable for HPC. This will put Intel at a disadvantage until perhaps mid 2008 to early 2009. However, AMD could be the dominant player in HPC long before then. AMD could top 40% by 2008 duplicating Intel's once strong position with Intel 32.

The cycle of waves is even easier to see if we look at the graphs for total power and total processors.


With the % of Total HPC power we can see that the first wave to replace the older RISC systems was Intel 32. The second wave to replace Intel 32 appeared at first to be both IBM's Power and Intel's Itanium however now it is clear that Itanium peaked in 11/04 and the second wave was Power alone. Now too it is clear that Power peaked in 11/05 after never having reached quite the performance dominance of Intel 32 and has been declining for the past year. There doesn't seem to be much hope for a resurgence of Power based HPC since IBM has been using lightweight embedded derivatives of Power for Blue Gene which are not as good for general computing. IBM does have Cell but Cell is even worse for general computing than the embedded derivatives. This seems to be the reason why IBM is pairing Cell with Opteron. The % power graph also shows a much closer race between Intel 64 and AMD 64 with AMD pulling strongly ahead in the last six months. Also, the second graph tells a different story in terms of Intel's position with 64 bit systems. Although there have been more numbers of systems of Intel 64 apparently these were more smalller systems as the total performance is much closer.

The graph for numbers of processors is similar to the power graph.


However, one difference we can see is that in terms of numbers of processors used in HPC AMD was never behind Intel. They reached rough parity in 06/05 but AMD has been in the lead ever since. Both graphs suggest that within a year AMD will have surpassed Power both in terms of total performance and in terms of numbers of processors. This would appear to be good news for both Cray and Sun with mixed news for IBM. As IBM loses share with Power they do stand to gain with AMD HPC systems. However, Intel clearly has to do something if it plans to avoid being third in HPC behind both AMD and IBM. In terms of processors used for HPC Itanium's numbers are only slightly higher than the old RISC systems which is pretty dismal to say the least while Intel's 64 bit numbers are just a little over half (54%) of AMD's. By almost any standard Intel show's no current hope of being able to regain dominance in HPC. This seems to be a serious issue for Itanium as its market gets pinched ever smaller. Opteron has already driven Itanium out of the 4-way market and it appears that Opteron is also keeping Itanium out of the HPC market. It does not seem realistic for Itanium to survive in the sliver of the server market from 8-way to low end HPC. This also brings up the question of whether Intel is deliberately crippling Woodcrest in 4-way to keep it from rolling over Itanium. For Intel to turn things around it will take far more than they current have and more than they have announced for the next year. However, whether the delay with 4-way is deliberate or just due to poor engineering, AMD is not waiting. Nor does it appear that AMD is waiting to become the final wave in HPC.

That X86-64 will be the last wave seems almost a certainty. There is no chance now for the older RISC systems to make a comeback and there are only two of the traditional supercomputer makers left: Cray and NEC. Itanium does not appear to be capable of challenging the current systems and is likely to fade further in the coming years. While Cray and NEC are working hard it is most likely that they will simply take back some small share that they have lost since 2003. The only real player left is IBM with Power. However IBM's use of lightweight Power derivatives which are only 32 bit and have limited memory addressing in Blue Gene suggests that this architecture too is unsuitable. This is further bolstered by Cell which is even more constrained than the lightweight derivatives. IBM is rapidly becoming a co-processor player rather than a main processor supplier for HPC. In fact, IBM may have to work hard to avoid being completely marginalized by GPU processing systems.

Intel should continue to share in the rise of X86-64 but the lack of announced large scale HPC systems is disturbing. Similarly disturbing is Intel's trend of doubling FSB's on its Northbridge chips. Essentially, the Intel Northbridge is like a mythical Hydra which survives by growing back two FSB's to replace the old one. Intel seems to be getting by with two FSB's now. However, there is some doubt that Intel can deliver quad FSB systems at a competitive price and it will be nearly impossible for Intel to move any higher. This leaves Intel once again facing a deadend while AMD simply beefs up its existing structure. If Intel doesn't do something quickly it will end up giving away nearly all of the top share of servers all the way from 4-way to large scale HPC to AMD. X86-64 seems to be the final wave in HPC with AMD in the lead now and most likely into 2008.