The fabric of the star

StarFabric is a switched interconnect technology that is currently being designed into systems ranging from Voice-over-Packet media gateways and optical edge routers, to video servers and medical imaging platforms.

Today’s designers of embedded distributed processing systems are under pressure to meet aggressive time-to-market requirements. Competition is fierce and the pace of technology continues to accelerate. Getting the right product to market quickly means success or failure. A broad trend is to use off-the-shelf building blocks as starting points. The rapid growth in the use of the CompactPCI platform as a starting point for many embedded distributed processing system designs is evidence of this trend. Open standard-based technology lets OEMs focus scarce engineering resources on areas of the system design where they add significant value and differentiation.

Off-the-shelf open technology available today is built on bus-based architectures, such as PCI. These architectures are simply running out of gas when asked to meet the needs of next-generation systems. They lack the scalability required, constrain physical form-factor options, don’t support cost effective high availability features, and aren’t designed to support multiple traffic classes.

A new open technology approach is required. One not based on a bus architecture, but on switched interconnect architecture. Properly defined, this fabric can provide the system level functionality next generation equipment requires and provide an easy to adopt migration path from existing open platforms technologies such as PCI.

StarFabric is a switched interconnect technology that enables embedded distributed processing applications. It supports multiple classes of traffic adding further flexibility and enhancements to systems while leveraging existing standards-based software and hardware investments.

It is currently being designed into systems ranging from Voice-over-Packet media gateways and optical edge routers, to video servers and medical imaging platforms. The protocol-agnostic architecture can be incorporated into many types of systems. It has hardware-based support for multiple classes of traffic, including native TDM voice traffic and hardware-based high availability features. This makes it suitable for scalable communication equipment. It incorporates support for loosely coupled distributed computing and uses memory-to-memory communications. This makes it suited to many compute and data-intensive applications.

StarFabric’s switched architecture can generally be described as a multi-queued (simultaneous input and output queuing), non-blocking switch fabric with shared memory. The switch fabric uses point-to-point LVDS connections with clocks embedded in the data stream. Clock and data recovery are performed on each differential pair at the receiver side of the point-to-point connections. This feature eliminates the need for tight skew control, which burdens technologies that use a clock separate from the data. Because of this, the technology is ideally suited for chip-to-chip, through backplane connectors and chassis-to-chassis interconnects up to 10 metres with standard PCB construction, 2mm existing connectors and CAT5 twisted-pair cable.

The StarFabric architecture was designed to support seven traffic classes, including asynchronous, isochronous, multicast and high-priority. It is suited for communication applications with converged voice, video and data requirements, meeting each one’s unique service requirements. Asynchronous traffic is traditional data-oriented traffic, with large, bursty bandwidth requirements but without real-time delivery requirements. Control and signaling traffic are typically asynchronous. Isochronous traffic, including voice and video, requires deterministic real-time delivery. StarFabric allows the unification of traditionally separate interconnects for control traffic and data payload traffic. This simplifies design and lowers cost. Also, in many voice or video applications, separate infrastructures are maintained for the real-time and non-real-time traffic. With StarGen’s technology, these infrastructures can be collapsed into a single unified interconnect architecture. Designers will now be able to produce systems that carry multiple traffic types, including data, voice and video via a single switch fabric interconnect.

There are several design trade-offs to be considered by engineers as they develop their next platform designs. The primary areas are the use of transparent and non-transparent bridging functionality and the use of pure address routing versus path routing for communicating across the fabric.

The SG2010 PCI-to-StarFabric Bridge is a multi-function device. It has two primary modes of operation. The first is the familiar transparent bridge function. The second mode is a fabric-native gateway function. When operating in bridge mode, the SG2010 looks like a standard PCI-to-PCI bridge. All existing legacy software including operating systems, BIOSs and drivers can be used without any modification. This mode of operation is also called address routing. Here, a single flat global address space is supported. Each device within the system is visible and can be accessed by its unique address range.

When initializing a system, one processing node must be established as the root. This can be accomplished by either hardwiring the device or by an election process. At power up, the hardware automatically begins a series of procedures. The initial activity takes place as StarFabric devices attempt to synchronize with all of their link partners. This is achieved by initiating traffic on all active ports. Once synchronization is achieved, an enumeration ‘storm’ occurs, which is sometimes referred to as a ‘FID Storm’ (FID stands for Fabric Identification). When the storm is over, each StarFabric device has a unique fabric identification number and has stored the path from itself back to the root of the fabric. All of this occurs within the hardware.

Now the BIOS and operating system begin their device discovery process. As noted earlier, when using address routing, all StarFabric devices, bridges and switches, look like PCI-to-PCI bridges. After the discovery process, the system map is that of a hierarchy of PCI bus segments. <a href=’’ TARGET=’_top’>Figure 1</a> illustrates a fabric topology and the resultant PCI hierarchy. Once the system map is discovered, normal device resource allocation can be accomplished. Intra-system communication is through normal PCI addressing techniques.

All this is accomplished without changes to the software. In this example, the 5 PCI bus segments could each have 5 to 8 devices creating one large system. All of the devices could communicate through a single address space. This mode has a variety of practical real world applications. Many desktops and servers are factory configured with a limited number of PCI expansion slots. By having a StarFabric Bridge card in one of these slots and having another one in a remote chassis, the system can reach new levels of performance and reliability. StarFabric links can be upwards of 10 metres long. A designer could use this feature and separate the compute chassis from its noisy disk drive chassis, even putting it in an adjacent room.

This distance feature has application in many embedded systems. Many of these systems are physically quite large. For example, a medical imaging system, an industrial controller and an automated tester can all take advantage of this. Many of these systems use PCI due to its ubiquity and low cost. By removing the current distance limitations, new mechanical architectures are possible.

While legacy address routing has many applications in the real world, it is not without limits. Referring to <a href=’’ TARGET=’_top’>Figure 1</a>, the hierarchical system map requires that all traffic flow through the root node of the system. This results in a practical upper limit in terms of scale and performance. The actual topology also has a number of links that cannot be effectively used. To overcome these inherent limits in tree topologies, StarFabric devices have a gateway mode. With this mode, system designers can take advantage of a wide range of StarFabric advanced features, including support for multiple classes of traffic, high availability and distributed computing capabilities.

The gateway mode within the PCI-to-StarFabric Bridge is analogous to an embedded PCI-to-PCI bridge. In this mode, devices and processors can be isolated behind the bridge. Once isolated, each processor can have its own address space and its own dedicated devices. Hardware-based initialization sets up the connections across the fabric. Here, each device can store multiple paths to the same destination device. The primary would be used at first. If there were a failure on that path, the device would automatically, in silicon, switch to the secondary path. In some cases, software intervention could also force a failover. In both cases, the event handler receives an event notifying of the path change.

Gateway mode uses path routing. After the initialization process described earlier for address routing, the system boot-up process continues. After the FID storm has created unique IDs for each node and the BIOS and operating system has created the hierarchical system map, additional steps are initiated. Now a software object called the Fabric Primitives Library (FPL) is invoked, setting up the path-routed connections engineered by the designer.

The FPL is a set of utilities that enables the range of advanced features in StarFabric. The utilities are built in C and C++, endian-agnostic and easily portable. StarGen provides the FPL in source code to all customers. The power of the FPL is that it provides a layer of abstraction from the hardware register details. Developers can take advantage of the advanced features, including bandwidth allocation, class of service weighting, redundant path routing, etc.

Complex distributed processing systems can be built using path routing. Devices can be controlled from anywhere in the system, which would use sophisticated mechanisms for handling system events and interrupts. Interrupts, for example, could be configured to be handled by any processor in the system. Similarly, all of the processors can be programmed to field interrupts from any number of devices. Given the potential complexity of an embedded distributed processing system, handling a high volume of interrupts is a primary concern. With StarFabric, an interrupt source is identified by a local memory read. This system knows exactly the source of the interrupt and in many cases, how to handle it. In a large system, this reduces the latency of interrupt handling by two to three orders of magnitude over a traditional system. Additionally, StarFabric enables developers to structure priorities queues for I/O interrupts. The combined ability to map and to route interrupt priorities enables interrupts to be using in many innovative ways.

StarFabric can use in-band interrupt techniques where the interrupt is sent in-band with the data. In a traditional PCI out-of-band approach, an interrupt may occur before the data has cleared the bus. With StarFabric, a designer could send the data ahead of the interrupt where it could be used immediately by the processor. With this, mechanism processors may not have to read devices at all. A memory write would clear the interrupt.

The architecture can support hybrid systems using both address routing and path routing simultaneously. Parts of the system could use the benefits of legacy address routing, while more complex parts of the system could take advantage of the advanced features of path routing.

With StarFabric, designers can start with address routing, re-use existing software and add the robust features of path routing as required. Additional advanced features could be added to the system over time.