Build the next generation of telecom systems with open interfaces (Part 2)

Ajay Kamalvanshi and Timo Jokiaho, Nokia Corp., and members of the Service Availability Forum
May 26, 2005 (5:00 AM)

The Telecom Equipment Manufacturer's (TEMs) need for low-cost modular building blocks to build next generation telecom systems has led to the adoption of standard-based hardware and software interfaces. In the past, software architectures were often tied to proprietary hardware and applications. In fact, the TEMs' product names frequently identified a specific hardware, processor architecture, ASICs, operating system (OS), and middleware instead of functions. Moving to a newer hardware platform mandated partial or complete overhaul of the software architecture. Developing new products meant developing new hardware and adopting a new OS, porting middleware and applications, resulting in two to five years to develop a product.

With a growing market for quickly developing new services, a robust modular software architecture is needed. A unified telecom software architecture addresses this need (Fig. 1). The architecture consists of three layers with well-defined interfaces. Modules in each layer can be replaced with minimumal impact on the other layers. In addition, the modules in a layer provide well-defined interfaces and could be replaced with more competitive, better performing modules or building blocks with low integration and verification cost.

1. Common software architecture for telecom systems.

The lowest layer is the hardware platform representing telecom hardware, such as an ATCA chassis or blade server, with different processor architecture or hardware dependencies. The middle layer (system platform software) consists of several software building blocks with standard interfaces. The carrier-grade OS provides the basic services such as scheduling, memory protection, file-system services, and low-level interprocess communication, such as shared memory support. In addition, the carrier-grade OS supports serviceability, scalability, and availability features described later. The platform drivers, such as a HPI driver, provide an interface for managing hardware components of equipment such as PEMs, fan trays, and blades.

The network-processing framework handles the user-plane packet processing and encapsulates functionality of network processing elements, such as network and communication processors. The cluster middleware provides the distributed fault-tolerance features such as monitoring software failure, handing failover, and defining redundancy models. The system management modules provide a generic administrative interface for configuring and provisioning of the overall system. The top (application) layer consists of management applications providing a specific interface, such as CLI and SNMP; distributed telecom applications like GGSN and SGSN; and standard routing applications such as OSPF and BGP.

Carrier-grade OSs
A carrier-grade OS is a key constituent of a telecom systems' software architecture. Traditionally, the commercial real-time operating systems (RTOS) like VxWorks, QNX, and OSE dominated the telecom OS landscape. However, the TEMs were required to add several features to support availability, scalability, and serviceability requirements. As Linux gained acceptance in the server market, the telecom equipment vendors also started seriously considering Linux for telecom systems.

The Open Source Development Lab's carrier-grade Linux working group was established to collect requirements from network and telecom equipment providers, and independent software vendors specifically for Linux. The working group published carrier-grade Linux requirements to help Linux distributors demonstrate readiness for adoption in telecom systems. The latest requirements, version 3.0, are a super-set of version 2.0.2. The requirements are split into seven documents, with each describing an area of carrier-grade OSs. The main requirements are support for the following features:

On-line diagnosis and repair operations such as forced file system unmount for file system repair.
Redundant communication paths and failover capabilities like Ethernet bonding.
Enhance monitoring mechanism for quickly detecting hardware and software failures such as non-intrusive process monitoring and memory over-commit actions. The non-intrusive monitoring detects events such as abnormal process termination, excessive CPU cycles usage, and infinite loops, and takes corrective action such as restarting a new process. The memory over-commit feature detects a violation when a process exceeds its threshold limits and takes corrective actions such as terminating the process.
System-downtime reduction with fast restarts by bypassing the BIOS (using kexec) and live patches.
Remote management standards such as SNMP, Common Information Model (CIM), and Web-based element management.
Run-time configuration of latency and scheduling parameters, including the scheduling quantum configurable to 1 ms or less.
Robust mutexes with priority inheritance and protection traditionally available in an RTOS. Robustness includes passing ownership if the owner dies.
Asynchronous event management for a large number of events to enable traditional event-driven telecom applications.
High-resolution timers based on a 1-ms tick.
Scheduling priority for certain important network protocols over others.
Stream-control transfer protocol, Internet protocols (Ipv4/IPv6), and mobile Internet protocol (MIPv6).

The Linux distributors have started including these features in their carrier-grade editions, although many still support version 2.0.2. This support testifies that Linux will soon be a viable alternative for next-generation telecom systems as the cost cuts continue. The cost of an OS includes training, support, and licensing. These are all much lower with Linux.

Cluster middleware
The telecom application software must be robust and designed to handle failures. Designing each application from scratch to handle failure is time consuming and makes the overall system unpredictable. This led to evolution of middlewares that abstract the common tasks, such as monitoring failure, defining a redundancy model, and handling failovers, for making any application highly available. Many telecom vendors had a proprietary cluster middleware implementation as part of the platform software. This middleware solved many issues, but the cost to integrate an application with the cluster was high. Hence, the independent telecom applications vendors still struggled to support their applications across clusters.

The Service Availability Forum (SAF), a consortium of TEMs, middleware software vendors, and application software vendors, addresses this challenge of defining two open interface specs between application and cluster middleware: the application interface specification (AIS) and the hardware platform interface (HPI). The AIS is focused on middleware interfaces for applications and defining redundancy models, and the HPI is focused on providing a uniform interface for system management application.

The AIS logical model is designed to support applications ranging from as web-servers to telecom applications (like GGSN). In this model, the application software provides a service, a set of actions to satisfy a request from a user or other system. A collection of components implements a service. For example, a web-server may have one component that handles HTTP protocol and another for handling transactions. A component can be mapped to a Unix process, although the spec doesn't enforce this mapping.

A collection of components for a service on a node forms a service unit. The logical model also defines component service as an abstraction for the workload handled by components and service respectively. The service units are associated to form a service group providing protection from failure of a service unit. The component service instances are also associated to provide a protection group.

The set of cluster interfaces that the SAF's AIS defines are:

Availability management framework (AMF): defines APIs for managing the lifecycle of application software such as registration and de-registration; health-monitoring; managing availability states; and reporting errors. The middleware cluster software that provides this interface will coordinate with redundant resources within a cluster to provide a fault-tolerant system and perform functions such as determining and assigning readiness and HA states, assigning workloads, and handling failovers.
Cluster membership service interfaces: provides an application with a consistent view of dynamic membership information in a cluster, which consist of a set of nodes, each with a unique identifier.
Checkpoint services: specifies an interface for a facility to incrementally record and retrieve checkpoint data. This data is used for failure recovery or switchover.
Messaging service: defines a low-level communication interface for processes in the same or different nodes (a logical entity that runs one instance of the OS).
Event service: defines a publish-subscribe communication mechanism. When an event is published, all subscribed clients receive the message.
Lock service interface: for distributed lock implementation.

Making a telecom application highly available using the AIS spec is not complex. An application like GGSN can be made highly available by:

Choosing the application or service redundancy model: 2N, N+M, N-way, or N-way active.
Mapping application entities to SAF entities like components, component service instance, service instance, service units, service group, and protection group. For example, the application may be mapped to service units. Various subsystems implemented as separate processes are mapped as components.
Implementing lifecycle APIs and registering the application as one or more components with AMF along with callback handlers. If additional services, like event or lock are required, the application may need to implement initialization functions along with callbacks.
Implementing checkpointing schemes for synchronization between active and standby components based on component service instances. The checkpointing scheme depends on the warmness of standby components. The standby components could choose to use the checkpoints when there's a failure or for continuously updating its states to be in lock step with the active component. In some cases, an application can use multiple checkpoint schemes.
Handling failover and validating after switchover when the standby component becomes active.

A Gateway GPRS support node (GGSN) application can be implemented as one process or as a component to form a service unit (Fig. 2). A service group is an association of two units and a protection group is an association of two processes on a different node. The service and component service instances represent the workload, and are managed by the AMF.

2. A sample mapping of a 2N redundancy model telecom application.

Data-plane software interface
As the requirement for speed increased, specialized ASICs were developed to handle the forwarding task, particularly in routing. However, as more was learned about ASICs and processor technology improved, general-purpose processors were developed (known as network processors.) The interface between the network processor became complex and proprietary. Hence, the Network Processing Forum (NPF), a consortium of hardware component vendors, system providers, service vendors, and software vendors, was formed standardize the interface to network processors and provide interoperability.

The NPF architecture uses a layered approach (Fig. 3). The first layer above the network processing elements is the interconnect layer, like forCes, that provides the low-level messaging services used to send packets across various network processors. The upper layers use this service to provide location transparency. Above the interconnect layers are three important functional constituents: NPF functional APIs (NFAPI), NPF service APIs (NSAPI), and operational APIs.

3. The layered NPF architecture.

The NSAPIs expose a vendor independent interface to the control-plane application, such as routing, and encapsulate an underlying distribution of the NPE. These APIs control service specific functionality, such as IP routing and MPLS of network processors, and are used by protocol stack and software vendors.

The NFAPIs are vendor-independent interfaces to individual NPE and define different stages in packet processing known as logical functional blocks (LFB). For example, NFAPIs are defined for IP forwarding, classification, and QoS. A processing packet path may be established by chaining LFBs in a specific order to achieve the desired processing. The application discovers LFBs using APIs and then selects building blocks based on NPF FAPI and NPF SAPI defined for select LFBs to realize the required system behavior.

As network processors are increasingly becoming application specific, the NPF is planning to define service specific protocols such as TCP and stream offloading. In addition, application-specific interfaces like firewall and SSL may soon evolve.

While the NPF's primary focus is to provide interfaces for network processing elements, high availability is also a key consideration. The NP Forum is aligned with the SAF and uses its complementary services such as the availability management framework. The NFAPIs could be viewed as service areas in addition to the SAF's services, like event, lock, and membership. It's recommended that NFAPI and FSAPI providers use the SAF's services to making their interfaces highly available.

System management
The most important aspect of a telecom system from operator's perspective is ease of managing the system. The operators select the TEMs based on the ability to monitor and control the system as well as the different components that constitute the system. The management plane is layered too. The lower layers include specialized hardware, such as IPMI, and a generic software interface such as HPI. The next layer in system management includes agents that provide a common information model (CIM), and exchange protocols like SNMP and Web-based element management (WBEM). The Software Management Group has defined a management information base (MIB) and document type definition (DTD) for its services.

These management agents define interfaces for the entire equipment including hardware, cluster middleware, and applications. For example, MIBs are defined for the SAF's HPI, AIS management framework, and service areas. The last system-management layer consists of management applications like command-line interface or network-management adaptors. Newer interfaces, such as software management architecture for server hardware (SMASH), may also be used in future, particularly for managing multiple heterogeneous clusters.

An important aspect of system management for highly available systems is software management. In particular, the in-service software upgrades are becoming a necessity, rather than a "nice-to-have" feature. Many middleware vendors support this feature, but so far this area continues to be complex. The SAF is exploring a standards-based software upgrade.

Editor's Note: In part 1, we looked at standardization efforts in equipment hardware and low-level system management. We also discussed data plane and cover details of a common modular profile for telecom equipment vendors. To view Part 1 of this article.

References:
1. Carrier Grade Linux Requirement Definitions, Version 3.0, www.osdl.org.
2. SAF Application Interface Specification AIS B.01.01, www.saforum.org.
3. SAF Hardware Platform Interface Specification HPI B.01.01, www.saforum.org.
4. PICMG AdvancedTCA Base Specification document (PICMG 3.0 R1.0) and as amended by ECN 3.0-1.0-001.
5. Modularizing HA Part 1: The Hardware Platform Interface, Manfred Reitenspiess and Louise Moser, June 09, 2004, www.commsdesign.com.
6. Modularizing HA Part 2: Application Interface Specification, Manfred Reitenspiess and Louise Moser, June 08, 2004, www.commsdesign.com.
7. Framework Brings HA Capabilities to Net Processing Designs, By Ram Gopal and T. Sridhar, Aug 04, 2004, www.commsdesign.com.

About the authors
Ajay Kamalvanshi is a technology manager in the system technologies group of the networks division at Nokia Corp. He holds a master's degree in computer science and automation from the Indian Institute of Science, Bangalore, India, and can be contacted at ajay.kamalvanshi@nokia.com .
Timo Jokiaho is the director of technology at Nokia, working with carrier-grade platforms. He holds a master's degree in computer science from the University of Helsinki, Finland, and can be contacted at timo.jokiaho@nokia.com.

Click here to read more ...