Peering into RapidIO's Move to the Data Plane

The logical layer specification defines the overall protocol and packet formats. This is the information necessary for end points to initiate and complete a transaction. The logical layer defines the presentation to the application.

The common transport specification provides the necessary routing information for a packet to move from end point to end point. The common transport unifies all the traffic so any RapidIO switch can carry traffic from all logical and physical layers.

The physical layer specifications contain the device level interface, such as the packet transport mechanisms, flow control, electrical characteristics, and low-level error management. Physical specifications include 8- and 16-bit LVDS parallel interfaces for low latency as well as 1X/4X multi-gigabit serial interfaces for low pin counts and longer transmission distances.

The RapidIO interconnect architecture is also extensible at both the logical and physical layers. For example, the trade assoc iation has recently completed a specification that defines end-to-end flow control enhancements to improve congestion management. The association is also currently defining data streaming logical layer will define a logical transaction layer optimized for the variable sized payloads typical in networking workloads. Let's look at both of these extensions in more detail starting with the flow control mechanism.

Controlling Flows
All RapidIO physical layers have a link-based flow-control mechanism for flow management between two devices. This ensures that individual link receivers will not be overrun with data, but doesn't address the situation where multiple traffic sources might individually or collectively create congestion scenarios two or more switches removed from the traffic sources. The recently released flow-control specification defines a mechanism to deal with these scenarios and provide end-to-end flow control within a system.

End-to-end flow control allows systems to manage co ngestion. Congestion can occur naturally in larger diameter fabrics. The statistical nature of traffic can cause short term over utilization of resources. Without the ability to regulate traffic at the source, this can lead to a cascading congestion effect, lowering overall fabric performance.

As fabric utilization exceeds 60%, congestion can significantly affect traffic latency. For applications where overprovisioning of bandwidth is the method used to insure performance, the recently completed flow-control extensions can expand the effectiveness of the available bandwidth and reduce latency. Another feature provided by the flow control extensions is the ability to managing traffic flow back to the originating endpoints. This is essential for fault management. Without a means to shut down traffic sources in a system, a single failed endpoint can bring an entire fabric and system down by spewing traffic into the system.

To review, the two flow control mechanisms in RapidIO are each optimized for sol ving different but related traffic congestion problems in a system. Link-based flow control is designed to prevent traffic overruns between two directly connected devices in a system. An example of this would be a situation where a receiving device has run out of buffer space to place incoming data. Upon detecting this situation the receiver can send a message to the sender to either stop or slow its rate of transmission.

End-to-end flow control is used when multiple devices have created a traffic congestion scenario downstream in a system. This congestion may be severe enough to cause unrelated traffic flows in the system to be adversely affected as well. Unlike the link-based flow control mechanism, for which signaling is confined to between two adjacent devices, the end-to-end flow control can span any number of devices in the system. It also offers flow-control management based on flow priorities, allowing higher priority traffic to continue to flow through the system while lower priority tra ffic is throttled.

Data Streaming at the Logical Layer
Network traffic is different in character than that of most computing applications. Network traffic usually consists of a stream of data, consisting of multiple or even continuous transactions unique to specific destinations. These "flows" can range from very small to very large data grams. Network traffic today uses may different protocols. It is important for today's telecom equipment to be protocol agnostic.

Telecom traffic adds an additional requirement to manage quality of service (QoS). Traffic behavior can range from requiring highly deterministic delivery of all traffic classes to solely requiring best effort. Traffic regulation may be required on a per flow basis. This gives network service providers the ability to regulate and charge for services. This finely grained traffic classification model is an important characteristic that distinguishes the data streaming logical layer from the other logical layers.

Through the n ew data streaming specifications, network traffic, upon entering a managed system, is classified based on specific fields in the protocol. This application specific process creates a Stream ID that when combined with a class of service field and the source and destination IDs, forms a unique key. This virtual Stream ID can be subdivided to support virtual queuing at the "on ramp" to the fabric. Further use of the class field by the transport layer assists the fabric in prioritizing traffic (Figure 2).