Configurable switch fabric interface bandwidth system and method

This application is a continuation-in-part of a application entitled, SYSTEM AND METHOD FOR SWITCHING VARIABLY SIZED INFORMATION GROUPS, invented by Yun et al., Ser. No. 10/023,266, filed Dec. 14, 2001, now abandoned.

This application is a continuation-in-part of a application entitled, SYSTEM AND METHOD FOR SIMULTANEOUS DEFICIT ROUND ROBIN PRIORITIZATION, invented by Yun et al., Ser. No. 10/022,673, filed Dec. 17, 2001, now U.S. Pat. No. 7,079,545.

This application is a continuation-in-part of a pending application entitled, SYSTEM AND METHOD FOR GRANTING ARBITRATED BIDS IN THE SWITCHING OF INFORMATION, invented by Yun et al., Ser. No. 10/029,581, filed Dec. 20, 2001.

This application is a continuation-in-part of a application entitled, SYSTEM AND METHOD FOR HIERARCHIAL SWITCHING, invented by Yun et al., Ser. No. 10/035,835, filed Dec. 24, 2001, now U.S. Pat. No. 7,020,131.

This application is a continuation-in-part of a pending application entitled, SYSTEM AND METHOD FOR TOLERATING DATA LINK FAULTS IN A PACKET COMMUNICATIONS SWITCH FABRIC, invented by Clovis et al., Ser. No. 10/373,139, filed Feb. 24, 2003, which is incorporated herein by reference.

1. Field of the Invention

This invention generally relates to packet communications switching and, more particularly, to a system and method for configuring the ingress and egress interface bandwidths of a switch fabric.

2. Description of the Related Art

There is industry demand for integrated circuits (ICs) switching systems that can be easily scaled for Network Access, the Network Edge, or a Core Switch Router, at the high end. SCSA (Signal Computing System Architecture) defines a switch fabric to be the facility for connecting any two (or more) transmitting or receiving Service Providers.

FIG. 5 is a schematic block diagram of a general switch fabric system (prior art). Traffic enters the system on the ingress side (the left side of the figure). Traffic exits the system on the egress side (the right side of the figure). The port card ingress and egress structures are sometimes subdivided into channels and subchannels. The packet headers specify a required destination port card, channel, and subchannel. The system must synchronously route each ingress packet to the correct egress destination.

Packets are converted into frames by ingress traffic managers (iTMs). A frame is a logical unit of data, which is often a small piece of a much larger data set such as a file or image. The iTMs feed the ingress side of the fabric. The switch fabric might convert the frame format to a “native” format, and then on egress, convert the data back into the TM frame format before sending the data to the egress traffic managers (eTMs). If the frames are fixed size (for example: 53 bytes, 64 bytes, or 80 bytes), the frames are often called cells.

Protocol Specific vs. Protocol Agnostic

A switch fabric can be protocol specific or protocol agnostic. An example of a protocol specific switch fabric would be a system designed and optimized specifically for asynchronous transfer mode (ATM) traffic. Another example would be a switch that handles only TCP/IP traffic. The obvious disadvantage of a switch fabric hardwired to handle a specific protocol is lack of flexibility. Service providers want to sell services to a variety of customers in various industries. Protocols vary from industry to industry.

Even within one protocol type, there can be protocol upgrades. For example, TCP/IP, the machine language of Internet routers, now runs primarily “IPv4.” This protocol, successful as it has been, has a lack of available addresses, poor security features, and no “quality of service” (QoS) provisions. The next generation Internet protocol is “IPv6.” It provides solutions to these limitations.

A protocol agnostic switch fabric works equally well with all protocol types, however, the traffic manager must be responsible for recognizing specific protocol types. The disadvantage of a protocol agnostic switch fabric is that it may be more complicated, and perhaps slower than a switch fabric dedicated to a particular protocol.

Packet Striping vs. Single Link per Packet

The simplest way for a traffic manager to transmit a packet into a switch fabric is to transmit the packet serially along one line. Striping is a way of achieving higher bandwidth by transmitting a single packet across multiple ingress/egress lines. For example, a TM can transmit a packet into a switch fabric eight times as fast if the packet is sliced into eight pieces (stripes) by the TM, and conveyed into the fabric along 8 parallel lines simultaneously. The fabric captures the packet in memory, routes it to the required egress destination, and slices the packet into 8 parallel lines before transmitting the packet to the egress TM.

The upside to packet striping is the potential for lower latency. There are several negative aspects of packet striping:

if one of the links is damaged (1 to 8 in the example above), the entire channel is out of service, degrading fault tolerance; and,

the interface between the TM and switch fabric is more complicated. Circuitry must be used to slice the packet into stripes and reassemble it into packets.

Single-Cell packet vs. Multi-Cell Packets

Many switch fabrics now deployed and passing revenue traffic, especially in wide area networks (WANs), use asynchronous transfer mode (ATM) packets. ATM packets are single-cell packets, with a fixed cell size of 53 bytes, 48 bytes of which is the information payload. The ATM specification evolved in the 1980s and early 1990s. It was then believed that variable length (multi-cell) packets would be too difficult to implement at rates needed for wire-speed traffic. The single-cell solution was a compromise that would work for voice data, video data, multimedia data, email data, file data, etc. With a fixed frame size, switch designs are simplified and ultimately faster.

However, files are better sent in large frames. Voice switching performs better with small bursts of data, corresponding to analog-to-digital converter time slots. Large frames and concomitant switching latencies can render a switch useless for most 2-way voice applications. Voice communications require low latency (time delay). Since ATM had to work for all network data, the small payload (voice) requirements prevailed at the expense of the large frame applications.

For large frames or packets it is much more efficient to establish a path from an ingress port card to the required egress destination, and then leave this path undisturbed until the packet is completed. With single cell packets, the ingress port card must bid for, accept grants, and then schedule each cell of a multi-cell packet.

There are also Frame Relay switch fabrics. Frame Relay is a single-cell protocol, albeit with frame size ranging from 6 to 4096 bytes. Such switch fabrics have been deployed and passing revenue traffic since the early 1990s. These switch fabrics have generally not been used for voice data because of the large latency. Only highly compressed voice traffic works well over frame relay. Voice-over-frame relay was added as an afterthought by network engineers. Frame relay excels at data communications such as local area network internetworking (LAN-to-LAN). Such communications are very high speed and bursty, with non-critical latency constraints.

Cut-Through vs. Store-and-Forward

The conventional method of switch fabric packet routing is called Store-and-Forward. In this method, the switch fabric accepts an input packet and buffers the packet on the ingress side of the fabric, making sure the packet was received intact, knowing the exact number of cells in the packet. The problem with the store-and-forward method is the added latency of buffering the packet. In Cut-through Packet Routing (CPR), a switch fabric is able to send the incoming packet cells to the correct egress port as soon as the destination address is known.

Memory-Based vs. Arbitrated Crossbar

A switch fabric can use memory-based crossbars or arbitrated crossbars. A memory-based crossbar is sometimes called a “shared memory switch.” Ingress packets flow from the port cards into a huge memory bank, which serve as the switch. From the memory bank, the destination address is determined from egress port destination information in the cell headers. The problem with these switch fabrics is that they become prohibitively expensive and large from all the required high-speed memory. Such fabrics cannot be used to reach terabit total switching speeds with technology available today.

It would be advantageous if a switch fabric could be configurable to communicate packet information at a number of interface bandwidths.

It would be advantageous if a switch fabric could communicate packet information to and from port cards at a plurality of different bandwidths.

It would be advantageous if switch fabric could communicate packet information to and from any port card in the system at a plurality of bandwidths.

The present invention describes a switch fabric with configurable bandwidth ingress and egress interfaces. That is, the fabric can communicate with a number of networks (traffic managers), regardless of the network bandwidths. The bandwidths may vary across different port cards in the fabric, or even across a single port card. Further the ingress bandwidths are independent of the egress bandwidths.

Accordingly, a method is provided for configuring interface bandwidths in a packet communications switch fabric. The method comprises: interfacing data links with a first plurality of traffic managers (TMs); differentiating port card interface ports into a first plurality of subchannels associated with the first plurality of TMs; and, communicating packets information with the TMs at a first plurality of data rates corresponding to the first plurality of subchannels.

More specifically, interfacing data links with a first plurality of TMs means that a second plurality of ingress data links are interfaced to input port card (IPC) ingress ports from a third plurality of ingress TMs (iTMs). Further, a fourth plurality of egress data links are interfaced from output port card (OPC) egress ports to a fifth plurality of egress TMs (eTMs). Then, differentiating port card interface ports into a first plurality of subchannels associated with the first plurality of TMs includes: differentiating the second plurality of ingress data links into a third plurality of ingress subchannels associated with the third plurality of iTMs; and, differentiating the fourth plurality of egress data links into a fifth plurality of egress subchannels associated with the fifth plurality of eTMs.

The other aspects, communicating packets information with the TMs at a first plurality of data rates corresponding to the first plurality of subchannels includes: the IPC receiving packet information from the iTMs at a third plurality of data rates; and, the OPC transmitting packet information to the eTMs at a fifth plurality of data rates.

Additional details of the above-described method and a packet communications switch fabric system for configuring interface bandwidths are provided below.

FIG. 1 is a schematic block diagram of the present invention packet communications switch fabric system for configuring interface bandwidths.

FIG. 2 is a more detailed depiction of the switch fabric of FIG. 1 .

FIG. 3 illustrates an example of WAN-to-MAN switching.

FIG. 4 is a flowchart illustrating the present invention method for configuring interface bandwidths in a packet communications switch fabric.

FIG. 5 is a schematic block diagram of a general switch fabric system (prior art).

FIG. 1 is a schematic block diagram of the present invention packet communications switch fabric system for configuring interface bandwidths. The system 100 comprises a switch fabric port card 102 , including data link interface ports differentiated into a first plurality of subchannels 1 through a. The subchannels communicate packet information at a first plurality of data rates corresponding to the first plurality of subchannels. The value of a is not limited to any particular value. Typically, the system 100 includes a plurality of port cards. To illustrate the invention, a first port card 102 may have interface ports differentiated into a first plurality of subchannels 1 through a, as discussed above. A second port card 104 may have interface ports differentiated into a second plurality of subchannels 1 through m, different than the first plurality. That is, the value of a need not equal the value of m.

In another aspect of the invention, independent of the consideration of bandwidth, the first port card 102 may differentiate interface ports into a first subchannel ( 1 ) for serial format packet communication and a second subchannel (a) for striped format packet communication.

From the perspective of a broader packet communications system, the system 100 may further comprise a plurality of traffic managers (TMs) 1 ( 106 ) through a ( 108 ) with interface ports for communicating packet information on data links. Then, a switch fabric port card, such as the first port card 102 , has ports for interface data links with the TMs. The interface ports are differentiated into a plurality of subchannels 1 through a, associated with the plurality of TMs 1 through a. The first port card 102 communicates packets information at a plurality of data rates (a) corresponding to the plurality of subchannels 1 through a.

FIG. 2 is a more detailed depiction of the switch fabric of FIG. 1 . Typically, the system 100 includes a plurality of input port cards. For simplicity, just a single input and a single output port card are shown. As shown, an input port card 200 (IPC) has a second plurality of ingress data link ingress ports 1 through b, on lines 202 through 204 , differentiated into a third plurality of subchannels (SCs) 1 through c, where the values of b and c are not limited to any particular value.

Likewise, an output port card 206 (OPC) has a fourth plurality of egress data link egress ports 1 through d, on lines 208 through 210 , differentiated into a fifth plurality of subchannels 1 through e, where d and e are not limited to any particular values. Note that it would be typical for each port card to consist of an IPC section and an OPC section.

The IPC 200 communicates packet information at a third plurality of data rates corresponding to the third plurality of subchannels 1 through c. The OPC 206 communicates packet information at a fifth plurality of data rates corresponding to the fifth plurality of subchannels 1 through e. Note, there is no predetermined relationship between the values of c and e.

The system 100 typically includes a plurality of switch cards, where a single switch card may include a plurality of crossbars. For simplicity a single switch card, with a single crossbar, is shown. As shown, a switch card 212 includes a crossbar 214 . The crossbar 214 has inputs 1 through f, on lines 216 through 218 , selectively connectable to crossbar outputs 1 through g, on lines 220 through 222 , for transferring packets between the IPC 200 and the OPC 206 . The IPC 200 has ingress backplane data links connected to the crossbar inputs 1 through f. Likewise, the OPC 206 has egress backplane data links connected to the crossbar outpus 1 through g. Note that it would be typical for both the IPC 200 and the OPC 206 to have backplane data links to a plurality of crossbars (not shown). Also note that it would be typical for the crossbar 214 to be connected to a plurality of IPCs and OPCs (not shown).

The switch card crossbar 214 transfers packet information between IPCs and OPCs through backplane data links at bandwidths independent of the ingress and egress bandwidths. That is, the backplane data link bandwidths are independent of the third plurality of data rates ( 1 through c) at which the IPC 200 is receiving packet information, and independent of the fifth plurality of data rates ( 1 through e) at which the OPC 206 is transmitting packet information.

Alternately stated, the switch card crossbar 214 transfers packet information between IPCs and OPCs through backplane data links in backplane channels independent of the third plurality of IPC 200 ingress subchannels and the fifth plurality of OPC 206 egress subchannels. The switch fabric makes a best effort to communicate packets through the backplane.

The system considers the destination egress subchannel in making the best effort as follows: The input priority queue (iPQ) scheduler in each IPC considers class of service (COS) in prioritizing the transmitting of packets from the iMS to a crossbar to a given egress subchannel. The OPC also considers COS in prioritizing transmit of packets from the eMS to the eTMs. The OPC uses an output priority queue (oPQ), or an earliest deadline first queue (EDFQ) as the egress scheduler. Either scheduler considers COS in prioritizing packet transmission. The EDFQ also considers the N-bit “packet flow ID” in the packet cell header. The iTM would assign flow IDs on packets such that higher priority packets and higher priority output subchannels get more bandwidth than lower priority subchannels. The switch cards themselves do not consider the subchannels on which the packets are being received or transmitted.

Therefore, the OPC 206 accepts packet information on egress backplane data links connected to the crossbar outputs on lines 220 through 222 , and differentiates the packet information into the fifth plurality of egress subchannels ( 1 through e).

For example, in some aspects of the system 100 , the IPC 200 differentiates a first ingress subchannel (a) with a first number of links ( 202 a, 202 b, and 202 c, for example) from a second ingress subchannel (b) with a second number of links ( 204 a, 204 b, 204 c, and 204 d), greater than the first number. The IPC 200 accepts packet information on the first ingress subchannel at a first data rate and accepts packet information on the second ingress subchannel at a second data rate, greater than the first data rate.

Likewise, the OPC 206 may differentiate a third egress subchannel ( 1 ) with a third number of links ( 208 a and 208 b) from a fourth egress succhannel (e) with a fourth number of links ( 210 a, 210 b, 210 c, and 210 d), greater than the third number. The OPC 206 transmits packet information on the fourth egress subchannel at a fourth data rate and transmits packet information on the fifth egress subchannel at a fifth data rate, greater than the fourth data rate.

Further, the IPC first ingress subchannel ( 1 ) is connected to a first ingress traffic manager 250 (iTM) and the second ingress subchannel (b) is connected to a second iTM 252 . The IPC 200 receives control traffic from the first iTM 250 on any of the first number of links ( 202 a, 202 b, or 202 c). The IPC 200 receives control traffic from the second iTM 252 on any of the second number of links ( 204 a, 204 b, 204 c, or 204 d).

Likewise, the OPC 206 third egress subchannel ( 1 ) is connected to a first egress traffic manager (eTM) 254 on the third number of links ( 208 a and 208 b) and the fourth egress subchannel (e) is connected to a second eTM 256 . The OPC 206 transmits control traffic to the first eTM 254 on any of the third number of links ( 208 a or 208 b) and transmits control traffic to the second eTM 256 on any of the fourth number of links ( 210 a, 210 b, 210 c, or 210 d).

Functional Description

FIG. 3 illustrates an example of WAN-to-MAN switching. This example shows how the invention might be used for a port card WAN-to-MAN switch fabric. Input “pipes” into the fabric are from a Wide Area Network (WAN). A typical WAN might extend across a country or even internationally. The connections are high speed “fat pipes.” A Metropolitan Area Network (MAN) is a public network that covers a metropolitan area or perhaps several metropolitan areas. The connections are typically not as fast as those of the WAN.

In the example, there are two input channels. Input channel 1 has two subchannels, each with four links. Each subchannel can handle 5 Gbps of revenue traffic. Input channel 2 has only one subchannel, comprising 8 links, capable of accepting 10 Gbps of revenue traffic. The output channels, to the egress traffic managers, are tailored to the bandwidth needs of the different destinations. Egress channel 0 has four subchannels, each with two links, each capable of 2.5 Gbps of revenue traffic. Egress channel 1 is divided up into three, one, one, and three links each for subchannels 9 , 1 , 2 , and 3 , respectively. These subchannels can handle 3.75, 1.25, 1.25, and 3.75 Gbps of revenue traffic each respectively.

Each port card in a switch fabric can have a different allocation. Other port cards in the fabric might be configured to perform the opposite function—that is aggregate small pipes from the MAN into fatter pipes for the WAN.

FIG. 4 is a flowchart illustrating the present invention method for configuring interface bandwidths in a packet communications switch fabric. Although the method is depicted as a sequence of numbered steps for clarity, no order should be inferred from the numbering unless explicitly stated. It should be understood that some of these steps may be skipped, performed in parallel, or performed without the requirement of maintaining a strict order of sequence. The method starts at Step 400 .

Step 402 interfaces data links with a first plurality of traffic managers (TMs). Step 404 differentiates port card interface ports into a first plurality of subchannels associated with the first plurality of TMs. Step 406 communicates packets information with the TMs at a first plurality of data rates corresponding to the first plurality of subchannels.

In some aspects, interfacing data links with a first plurality of TMs in Step 402 includes substeps. Step 402 a interfaces a second plurality of ingress data links to input port card (IPC) ingress ports from a third plurality of ingress TMs (iTMs). Step 402 b interfaces a fourth plurality of egress data links from output port card (OPC) egress ports to a fifth plurality of egress TMs (eTMs). Then, differentiating port card interface ports into a first plurality of subchannels associated with the first plurality of TMs in Step 404 includes substeps. Step 404 a differentiates the second plurality of ingress data links into a third plurality of ingress subchannels (SCs) associated with the third plurality of iTMs. Step 404 b differentiates the fourth plurality of egress data links into a fifth plurality of egress subchannels associated with the fifth plurality of eTMs.

Further, communicating packets information with the TMs at a first plurality of data rates corresponding to the first plurality of subchannels in Step 406 includes substeps. In Step 406 a the IPC receives packet information from the iTMs at a third plurality of data rates. In Step 406 b the OPC transmits packet information to the eTMs at a fifth plurality of data rates.

Some aspects of the method include a further step. Step 408 transfers packet information between IPCs and OPCs through backplane data links. In some aspects, Step 408 includes communicating packet information at bandwidths independent of the third plurality of data rates at which the IPC is receiving packet information from the iTMs; and, the fifth plurality of data rates at which the OPC is transmitting packet information to the eTMs.

In other aspects, transferring packet information between IPCs and OPCs through backplane data links in Step 408 includes communicating packet information in backplane channels independent of: the third plurality of IPC subchannels associated with the third plurality of iTMs; and, the fifth plurality of OPC subchannels associated with the fifth plurality of eTMs.

In some aspects, differentiating the fourth plurality of egress data links into a fifth plurality of egress subchannels associated with the fifth plurality of eTMs in Step 404 b includes substeps (not shown). Step 404 b 1 accepts packet information from backplane data links connected to crossbar outputs. Step 404 b 2 organizes the packet information into the fifth plurality of egress subchannels.

In some aspects, differentiating the second plurality of ingress data links into a third plurality of subchannels associated with the third plurality of iTMs (Step 404 a ) includes differentiating a first ingress subchannel with a first number of links from a second ingress subchannel with a second number of links, greater than the first number. Then, the IPC receiving packet information from the iTMs at a third plurality of data rates in Step 406 a includes accepting packet information on the first ingress subchannel at a first data rate and accepting packet information on the second ingress subchannel at a second data rate, greater than the first data rate.

In other aspects, differentiating a first ingress subchannel with a first number of links from a second ingress subchannel with a second number of links (Step 404 a ) includes associating the first ingress subchannel with a first iTM and the second ingress subchannel with a second iTM. Then, the method comprises further steps. Step 410 receives control traffic from the first iTM at the IPC on any of the first number of links. Step 412 receives control traffic from the second iTM at the IPC on any of the second number of links.

In other aspects, differentiating the fourth plurality of egress data links into a fifth plurality of subchannels associated with the fifth plurality of eTMs in Step 404 b includes differentiating a third egress subchannel with a third number of links from a fourth egress channel with a fourth number of links, greater than the third number. Then, the OPC transmitting packet information to the eTMs at a fifth plurality of data rates in Step 406 b includes transmitting packet information on the fourth egress subchannel at a fourth data rate and transmitting packet information on the fifth egress subchannel at a fifth data rate, greater than the fourth data rate.

In some aspects, differentiating a third egress subchannel with a third number of links from a fourth egress subchannel with a fourth number of links in Step 404 b includes associating the third egress subchannel with a first eTM and the fourth egress subchannel with a second eTM. Then, Step 414 sends control traffic from the OPC to the first eTM on any of the third number of links. Step 416 sends control traffic from the OPC to the second eTM on any of the fourth number of links.

In other aspects, differentiating a first plurality of port card interface ports into subchannels associated with the first plurality of TMs in Step 404 includes additional substeps (not shown). Step 404 c differentiates interface ports into a sixth plurality of subchannels on a first port card. Step 404 d differentiates interface ports into a seventh plurality of subchannels, different than the sixth plurality, on a second port card.

In other aspects, differentiating a first plurality of port card interface ports into subchannels associated with the first plurality of TMs in Step 404 includes differentiating interface ports into a first subchannel for serial format packet communication and a second subchannel for striped format packet communication.

A system and method have been provided for configuring a switch fabric interface bandwidths. A few simple examples have been given to illustrate the invention. However, the invention is not limited to merely these examples. Other variations and embodiments of the invention will occur to those skilled in the art.