Beyond FBDIMM

It is no secret that as processors have gotten faster, they've outpaced memory speed. There was a time when memory was actually faster than the processor but this has slowly changed. Over time, processor designs began including Level 1 (L1) cache and then Level 2 (L2). And, now it appears that most processors will soon need L3 as well. These caches are smaller and faster with each level. So, L3 is smaller than main memory but faster; L2 is smaller, but faster than L3, and L1 is smaller again but faster than L2. Without these caches, the processor would spend a lot of time waiting on main memory.

Memory has valiantly tried to keep up. It has gone from regular DRAM to DDR to DDR2 and soon DDR3. However, as speed has gone up, fanout has decreased. Therefore, it is not possible to put as many DDR3 DIMM's on a board as DDR DIMMs. The number of usable DIMMs has been traded for speed. The only reason why this is somewhat acceptable is that memory chip capacity is going up as well so the greater memory on each DIMM somewhat makes up for having fewer DIMMs.

Intel has introduced what it feels is a way to solve this problem, FBDIMM. Unfortunately, FBDIMM creates just as many problems as it solves. Although FDBIMM is fairly fast and has good fanout, it introduces very large latencies and draws much more power. An FBDIMM is basically a DDR2 DIMM with an extra chip on it, an AMB. This AMB chip provides two serial ports that allow an upstream channel and a downstream channel. These connect from the memory controller to the first FBDIMM. Additional FBDIMMs are simply daisychained to the first one. The problem is that each FBDIMM in the chain introduces latency. It is difficult to imagine how poor the performance of the 8th FBDIMM in a chain would be after the data request has had to hop over the previous seven FBDIMMs and then the data has to make the same seven hops going back up. This can drastically slow down the memory speed and make FBDIMMs impractical at maximum fanout. Therefore, at large fanout, we've simply come full circle and traded away speed again to increase capacity. Secondly, the AMB chip draws lots of power. This chip all by itself draws as much power as the rest of the DIMM combined, thereby doubling the power draw. This is not a good tradeoff, especially in a time when large scale computers are becoming sensitive to the high cost of electric power for these systems. Processors themselves have become much more energy efficient and Intel likes to brag about the low power draw of its newer Core 2 Duo designs verus its older Pentium D designs, however, these gains will easily be lost with the extra power needed for Intel's FBDIMM design. So, power draw for the processor is simply replaced by power draw from memory and no real advancement is made.

Currently, there is nothing better, but there could be. Suppose we replace the AMB chip with a HyperTransport chip with three 16 bit links. We'll call this HTDIMM or just HTD for short. One link goes back to the processor and the other two provide a fanout tree. The first HTD connects to two HTD's. These two then connect to two more, making seven total (1 + 2 + 4). A fanout of seven would be almost as good as the maximum eight fanout of FBDIMM. However, whereas FBDIMM would need seven hops to reach the last DIMM, HTD would only need two hops. Power draw and cost could be an issue but the last four HTD's only need a single link so they could be cheaper and draw less power. Cost is also not likely to be an issue since HT was designed for low cost and current FBDIMMs cost about the same as regular DDR2 DIMMs. Three links might seem complicated compared to FBDIMM but all of these combined would be narrower than the current DIMM bus, so it would reduce motherboard circuit trace complexity. Also, each 16 bit path is independent so this greatly reduces the complexity of the current very wide serpentine circuit traces.

A fanout of seven would be a good compromise for most server configurations however this is not an actual limit. There is actually no reason you couldn't add another hop and another fanout of two. So then you would have eight more DIMMs for a total of fifteen (1 + 2 + 4 + 8). This might be necessary for some high powered servers. Even with three hops we would still be less than half of the seven hops maximum for FBDIMM but at about double the fanout. And, again, the last eight HTD's would only need a single link.

AMD's K8 now uses an onboard dual channel controller. The processor could use four HT channels instead of the current memory controller which would greatly reduce both the pincount and the number of circuit traces for memory access. This would reduce both motherboard and processor complexity.

The reason why HT would work better than FBDIMM is several factors. HT was designed long ago for use on motherboards and is rooted in technology from the DEC Alpha. As it became an open standard, it was designed for low cost, low complexity, low latency, and high bandwidth. It has been successfully used by AMD, IBM, Apple, and all of the 3rd party chipset makers like Nvidia, ATI, and VIA. In fact, there are motherboards today for Intel processors that use HT to communicate between the Northbridge and Southbridge chips. HT is so robust that it is a superset of PCI, USB, AGP, PCI-X, PCI-E, and even Intel's proposed CSI standard. HT is capable of transporting data from all of these protocols. HT has an advantage over the AMB chip in terms of flexibility and speed. HT is capable of handling a tree structure whereas AMB uses a ring structure. HT is also much faster. Whereas FBDIMM can move 6.4 GB/sec, HTDIMM would be capable of moving 20.6 GB/sec. This speed is about as fast as the fastest proposed DDR3 memory. However, HTDIMM would be capable of handling multiple simultaneous DIMM accesses and data transfers. With four HT channels, the processor would have a peak memory bandwidth of 83 GB/sec. Since this is well beyond what current processors can use, desktop systems would probably only use two channels while budget systems used one. The reason for this speed is that the base technology for HT comes from the very fast DEC Alpha chip. This technology has continued to be developed since it was used on DEC's Alpha and AMD's Athlon MP processors. Today, HyperTransport is a mature protocol in version 3.0. The speed has doubled with each version.

This type of memory would work for Intel as well. HyperTransport is an open protocol and can be used royalty free. Replacing the current FBDIMM ports on the 5000 Northbridge with HT ports would be fairly simple for Intel to do and it could pick up additional desktop chipsets from Nvidia since they already use HT. This would require no change to the processor or FSB design if Intel wanted to retain these. Using HyperTransport with current DIMM design would be the best way to reverse the lag in memory speed and increase fanout. This would also reverse the trend toward more and more cache. The DIMM design itself would be superior to FBDIMM in every way. HTDIMM is what the next Jedec standard should be.