Architecture backs up all local I/Os

It's important to note that local interconnects such as PCI Express, PCI-X 2.0 and HyperTransport are chip-to-chip or chip-to-slot links inside the box. Infiniband is representative of another category: an interconnect designed to expand the capabilities of system-to-system communications, whether server-to-server or server-to-storage. In fact, 10-Gbit/second Infiniband is driving the need for the new local interconnects in servers since existing local interconnects cannot handle the 20-Gbit/s full-duplex bandwidth of Infiniband.

In theory, it should be possible to develop a topside bus module with the flexibility to work with all the new local interconnects, since at some level of abstraction each of the local interconnects is compatible with the PCI architecture. The challenge is to develop an architecture that implements the specific nuances of each of the local interconnects but encapsulates them in such a way that a module that provides a common interface can be upgraded easily without affecting the rest of the design.

A realization of one such m odular architecture is for Mellanox's InfiniHost dual 10-Gbit/s port Infiniband HCA. Critical to achieving the design goal is the development of a module that, independently of the specific external local interconnect, presents a uniform interface to the internal device subsystems. In this way, the local interconnect module can be modified, and derivative devices developed, with little or no impact on the rest of the design.

Furthermore, it is critical that the architecture be absolutely transparent to system software so that no changes to the software will be required in order to support whichever local interconnect succeeds.

The solution developed by Mellanox relies on a common High Speed Internal Interconnect (HSII) to connect the different external local I/O interconnects to the internal modules. The flexible local interconnect module abstracts the functionality of the specific local interconnect and maps this to the HSII high-performance packet-based interface with embedded flow control .

There are many challenges to effectuating such a module. These include forward designing to ensure adequate future bandwidth capabilities, accommodating ordering rules and providing for local interconnect specific functions. This is complicated by the fact that certain features such as quality-of-service (QoS) may be implemented in one local interconnect and not in another.

For example, PCI Express includes an optional virtual channel mechanism for QoS, while other interconnects may not. It is necessary to support specific features like this without affecting functionality or compromising performance for I/Os that do not support similar mechanisms.

Fortunately, careful architectural design was able to overcome these challenges, and the flexible HSII module was developed with no significant impact on die size. In fact, executing the design using the HSII module actually slightly decreased the die size and shortened the design time compared with an implementation using a fixed-function module.

While these results are somewhat counterintuitive, the HSII module actually simplified the design by breaking one big problem into several contained and independent ones. The net result is a cleaner, faster and cheaper design. The only impact is a slight increase in latency (approximately 1 percent), caused by the cross-module communications, but that trade-off is clearly outweighed by the ability to adapt rapidly to whichever local interconnect the market demands The flexibility delivered by the HSII implementation has proved to be well worth the additional effort, yielding a highly modular design that enables rapid adaptability to any interconnect.

There are strong arguments to be made for PCI Express, PCI-X 2.0 and HyperTransport. Given these circumstances the ideal strategy would be not to bet on any one technology but rather to have the flexi bility to rapidly adapt to changing market demands. Thus, it is vital that system-on-chip designers follow a strategy that allows a rapid adaptation all of the local I/O interconnects that succeed in the marketplace.