PCI Express has been around for a decade and until recently has not been viewed as a viable general-purpose fabric, but that is changing. I believe the ideal datacenter has PCI Express inside the rack and either Ethernet or InfiniBand connecting the racks together.
Ethernet and InfiniBand are both excellent solutions for connecting racks together in a datacenter with the interconnect choice being guided by the application's cost/performance requirements. Within the rack, all three interconnects will do the job. But even though 10G Ethernet and InfiniBand have an established track record, neither is well-suited to the task.
Ethernet is not ideal for the tightly knit connections within the rack. It tends to be expensive for the amount of bandwidth provided, doesn't scale easily or elegantly, and dissipates more power than it should for the bounded distances between blades inside the rack. Its feature-set is also limited for popular rack-level usage.
Click here to read more ...