Fault-tolerant maintenance bus protocol and method for using the same

1. Field of the Invention

This invention relates to fault-tolerant computer systems and more particularly to a protocol transmitted over dedicated maintenance bus for use with such computer systems.

2. Background Information

Fault-tolerant computer systems are employed in situations and environments that demand high reliability and minimal downtime. Such computer systems may be employed in the tracking of financial markets, the control and routing of telecommunications and in other mission-critical functions such as air traffic control.

A common technique for incorporating fault-tolerance into a computer system is to provide a degree of redundancy to various components. In other words, important components are often paired with one or more backup components of the same type. As such, two or more components may operate in a so-called lockstep mode in which each component performs the same task at the same time, while only one is typically called upon for delivery of information. Where data collisions, race conditions and other complications may limit the use of lockstep architecture, redundant components may be employed in a failover mode. In failover mode, one component is selected as a primary component that operates under normal circumstances. If a failure in the primary component is detected, then the primary component is bypassed and the secondary (or tertiary) redundant component is brought on line. A variety of initialization and switchover techniques are employed to make a transition from one component to another during runtime of the computer system. A primary goal of these techniques is to minimize downtime and corresponding loss of function and/or data.

Fault-tolerant computer systems are often costly to implement since many commercially available components are not specifically designed for use in redundant systems. It is desirable to adapt conventional components and their built-in architecture whenever possible. All modem computer systems have particular capabilities directed to control and monitoring of functions. For example, large microprocessor chips such as the Pentium III™, available from Intel Corporation of Santa Clara, Calif., are designed to operate within a specific temperature range that is monitored by a commercially available environmental/temperature-sensing chip. One technique for interconnecting such an environmental monitor or other monitoring and control devices is to utilize a dedicated maintenance bus. The maintenance bus is typically separate system's main data and control bus structure. The maintenance bus generally connects to a single, centralized point of control, often implemented as a peripheral component interconnect (PCI) device.

However, as discussed above, conventional maintenance bus architecture is not specifically designed for redundant operation. Accordingly, prior fault-tolerant systems have utilized a customized architecture for transmitting monitor and control signals over the system's main buses (or dedicated proprietary buses) using, for example, a series of application specific integrated circuits (ASICs) mounted on each circuit board being monitored. To take advantage of current, commercially available maintenance bus architecture in a fault tolerant computing environment, a more comprehensive and cost-effective approach is needed.

Accordingly, it is an object of this invention to provide a protocol for use with a maintenance bus architecture that displays a high-degree of fault-tolerance. This maintenance bus architecture and associated protocol should be interoperable with commercially available components and should allow a fairly high degree of versatility in terms of monitoring and control of important computer system components.

This invention overcomes the disadvantages of the prior art by providing a protocol that is instantiated on a fault-tolerant maintenance bus architecture. The architecture includes two maintenance buses interconnecting each of a plurality of printed circuit boards, termed “parent” circuit boards. The two maintenance buses are each connected to a pair of system management modules (SMMs) that are configured to perform a variety of maintenance bus activities. The SMM can comprise any acceptable device for driving commands according to the protocol on the maintenance bus arrangement. The SMM has general knowledge of the circuit boards and their components. According to a preferred embodiment, the protocol is formatted to operate in accordance with Philips Semiconductors' I 2 C maintenance bus standard. Other standards are expressly contemplated. Within each parent board are a pair of redundant bridges both having a unique address. One bridge is connected to the first maintenance bus while a second bridge is connected to the second maintenance bus of the pair. A child maintenance bus interconnects the two bridges through a “child” printed circuit board. The introduction of a separate board to implement the child maintenance bus can be useful, but is not essential according to this invention. The child maintenance bus is itself interconnected with a variety of monitor and control functions on maintenance bus-compatible subsystem components. Using the protocol, the SMMs can address components on each child printed circuit board individually and receive appropriate responses therefrom based upon appropriate response identifiers within the protocol. In the event of a bus or bridge failure, the SMM can still communicate with the child subsystem components via the redundant bus and bridge.

The protocol includes a unique data packet structure. The command message initiated by the SMM includes a target bridge header, a command byte (wherein a non-zero byte code designates the message as a command rather than a response), the message size and a unique originator tag value. The command message further includes one or more bytes of forwarding data for subordinate bridges on the child bus (leading to and from remote components/circuitry). Next the command message has a response byte code to direct responses on the return trip through the bridge. The command message also includes one or more bytes of data to identify, and be used by, subsystem components. Finally the command message includes one checksum byte meant to sum up all previous message bytes.

A similar message packet is provided by the bridge in response to the command message. The response includes an SMM address byte and a zero-value command byte (indicating a response). Also provided is a byte indicating the overall message size in bytes and the identical tag originally provided in the command packet. The tags allow the SMM to verify that the response is to a particular transmitted command. A one-byte status code field and one-byte error message field are also provided. Unique status codes and error messages are generated by the bridge if a formatted message is incorrect or commanded action was not (or may not have been) taken by the subsystem. One or more bytes of response data delivered from the subsystem component or bridge is also provided in the response message. Finally, a checksum byte is provided for error checking.

Command message/data packets are transmitted by the SMM to be received by an appropriate component within a given time frame. If an expected response message/data packet is not returned from the component as expected, the SMM “times-out” and performs various error procedures that may include an alarm condition, system shut-down and/or retransmission of the packet.

The bridge can include an interconnection to a further bridge. This remote bridge can, itself, be interconnected to additional microprocessors and associated memory. The remote bridge is addressed through one of the parent board's bridges so the communication to and from the SMM can occur. The forwarding data of the command packet enables the packets to be transferred through these further bridges, while stored response data in each subordinate bridge is used to route the return of a response back to the originating SMM.

The SMM can be interconnected with a variety of other computer system peripherals and components, and can be accessed over a local network or through an Internet-based communication network.

The foregoing and other objects and advantages of the invention will become more clear with reference to the following detailed description as illustrated by the drawings in which:

FIG. 1 is a block diagram showing an overview of a fault-tolerant maintenance bus architecture utilizing the maintenance bus protocol according to this invention;

FIG. 2 is a more detailed block diagram showing one parent and child printed circuit board implementing a fault-tolerant maintenance bus according to this invention;

FIG. 3 is the board of FIG. 2 including a bridge for accessing a remote microprocessor board according to an alternate embodiment;

FIG. 4 is a block diagram of the protocol's command message/data packet according to an embodiment of this invention; and

FIG. 5 is a block diagram of the protocol's response message/data packet according to an embodiment of this invention.

FIG. 1 details a fault-tolerant maintenance bus architecture adapted to use a protocol according to a preferred embodiment of this invention. Before discussing the protocol in detail, the underlying maintenance bus architecture will be explained. The architecture and associated hardware according to an embodiment of this invention is shown and described individually in the co-pending, commonly owned US Patent Application Serial No. 09/548,536, entitled Fault-Tolerant Maintenance Bus Architecture by A. Charles Suffin, et al, which is expressly incorporated herein by reference.

A pair of parent maintenance buses MBA and MBB are shown. These maintenance buses are identical in architecture and can be implemented as a combination of cables, circuitry and circuit board traces. The buses MBA and MBB interconnect with a plurality of input/output (I/O) slots and pin locations within a cabinet that may contain a plurality of circuit boards. The parent maintenance buses MBA and MBB can also jump between cabinets in a larger computer system. It is generally contemplated that the buses are implemented in a multi-cabinet fault-tolerant server system, but the architecture according to this invention can be utilized in a variety of fault-tolerant computing configurations. According to a preferred embodiment, the buses MBA and MBB are each two-wire buses designed to take advantage of integrated circuit components utilizing the I 2 C bus standard. The I 2 C bus is a proprietary design of Philips Semiconductors of the Netherlands. This standard has become widely adopted for consumer electronics and various circuit applications, and is now supported by a large number of commercially available monitoring and control devices. Details on the use of the I 2 C bus can be found in the I 2 C-bus and how to use it (including specifications), April 1995 update, Chapter 3 by Philips Semiconductors. Typically, the bus is clocked at a speed of approximately 10 Kbytes/Sec. While I 2 C is employed as the bus standard according to a preferred embodiment of this invention, it is expressly contemplated that other maintenance bus standards can be utilized according to the teachings of this invention with appropriate modifications.

To avoid signal loss over long distances, the parent buses MBA and MBB are amplified to generate a signal at 5 V×30 mA. A variety of bus amplification circuits can be used. In particular, commercially available hardware bus extenders are employed. The amplified bus operates at a gain that is ten times the normal operating range for an I 2 C bus (5 V×30 mA). As described further below, this difference in operating level between the parent bus and various circuit components is compensated-for (on both sides) using the extender hardware.

In FIG. 1, three circuit board assemblies 102 , 104 and 106 are shown. Each of these circuit board assemblies is interconnected with the bus pair (MBA and MBB). These board assemblies can represent a variety of computer system components. For example, the boards can together comprise a set of redundant identical boards or a set of separate functions including a central processing unit (CPU) board, “front panel” board and input/output (I/O) board. Each board assembly 102 , 104 and 106 is defined functionally as a parent printed circuit board 112 , 114 and 116 and an associated child printed circuit board 122 , 124 and 126 . As discussed above, while a separate board to implement the child maintenance bus can be useful according to an embodiment of this invention, it is not required. In general, the division between the parent board and child board is somewhat arbitrary, and the actual physical structure for one or more boards can be implemented as a single plug-in printed circuit card residing in a connector socket or slot on a larger cabinet-based motherboard.

Also interconnected with the parent bus pair MBA and MBB are a pair of redundant system management modules (SMMs) identified herein as SMMA 128 and SMMB 130 . Each SMM is a microprocessor-based component. The SMMs 128 and 130 each reside on a PCI bus 132 and 134 . The SMM performs a variety of functions and includes both Ethernet and modem capabilities allowing it to interconnect with the computer operating system and other network communication structures (block 136 ). The SMM may also include other unrelated system components such as a video driver chip. The SMM is particularly based around a Motorola Power PC™860T microprocessor utilizing the is VxWorks real-time operating system available from Wind River Systems, Inc. of Alameda, Calif. The term “system management module” (or SMM) is defined broadly to include any acceptable device for driving commands on the maintenance bus arrangement. While a microcontroller described above is used in a preferred embodiment, the SMM can be an application specific integrated circuit (ASIC), a programmable logic array, a microprocessor unit or any other command originator interconnected with the maintenance bus arrangement. The term “command module” is also used to define the SMM in its various possible embodiments.

Each SMM includes a pair of I 2 C buses 138 and 140 . Each pair is connected with a respective bus from the parent bus pair MBA and MBB. The SMMs are configured to operate in failover mode. In other words, SMMA operates under normal circumstances. In this mode SMMB monitors and communicates with SMMA over the shared I 2 C bus, ready to take over for SMMA if it fails. If a failure is detected, then SMMB takes over operation but is otherwise largely idle, during normal run time. The function of the SMMs is described in further detail below. In summary, the SMMs carry information about components on each of the board assemblies 102 , 104 and 106 . The SMMs use this information to monitor and generally control the board assemblies. This information may be transferred to other parts of the computer system and over a network via the PCI bus.

Each parent board 112 , 114 and 116 includes various data processing, display and communication capabilities in accordance with its purpose. Each board 112 , 114 and 116 also includes a respective CPU (CPU 1 , CPU 2 and CPU 3 ) 152 , 154 and 156 , respectively. Each CPU can comprise an Intel Pentium III™, Xeon™ or any other acceptable microprocessor having I 2 C or equivalent maintenance bus architecture. Each board 112 , 114 and 116 is interconnected with the parent buses MBA and MBB at various interconnection points, where appropriate. Since the parent bus is amplified by a gain by approximately ten times the normal I 2 C operating level, interconnections with the parent buses can be made via bus extenders to be described further below.

Each parent board 112 , 114 and 116 also includes a pair of interconnections 160 and 162 with each of the respective parent maintenance buses MBA and MBB. The interconnections 160 and 162 link to respective bridges 192 and 194 . These bridges, in turn interconnect with respective child maintenance buses CB 1 , CB 2 and CB 3 to interconnect child board components. On each child board 122 , 124 and 126 resides various control and monitoring subsystem components 172 , 174 and 176 , respectively. The subsystem components are described in further detail below. These components are each in communication with the maintenance bus using the preferred I 2 C standard. The interconnection between each parent board and child board occurs via a pair of bridges 192 and 194 . Each bridge is essentially identical in architecture, each pair of bridges 192 and 194 has the same different address for communication with the SMMs. The address of the bridge pair on each board, however, differs so that the SMM can uniquely address a specific board. Addresses are established based upon the pin and socket arrangement for the respective bridge. It is useful to assign the same address to both bridges 192 , 194 in the pair since they each reside on a different bus (MBA or MBB). For a given transaction, the SMM utilizes only one of the two bridges on a parent board to accomplish a task. The other, unused bridge in the pair can be used if the SMM cannot complete the transaction with the original bridge. The SMM uses the other parent maintenance bus to access the other, previously unused bridge. As noted above, in order provide an amplified parent bus signal and an appropriate signal level to the subsystem bridges, bus extender hardware is in employed. The bus extender hardware is available from Phillip Semiconductors under part number 82B715. Using amplified parent buses, approximately thirty or more loads can be carried. The extender acts as a buffer for signals traversing the extender hardware providing the necessary amplification and deamplification. Extender components 196 are provided between the parent buses MBA and MBB and corresponding bridge interconnections 160 and 162 . While not shown, interconnections 160 and 162 can also include appropriate series resistors and FET triggers in line with the extender components 196 in accordance with the 82B715 hardware manufacturer's data sheet.

The bridges 192 and 194 each act as store-forward devices in the transfer of I 2 C signals into and out of the child board subsystem. In other words, the bridges receive packetized signals from the SMMs and transfer them to appropriate I 2 C-compatable maintenance bus ports on subsystem components. Likewise, the bridges receive signal from subsystem components and transfers them back to the SMMs. In order to provide desired fault-tolerance, two bridges 192 and 194 are employed, each communicating with one of the dual parent buses MBA and MBB. In this manner, the failure of a single bridge or parent bus does not cause a lost of connection between the subsystem components and SMMs. This is because each child bus CB 1 , CB 2 and CB 3 is interconnected with both bridges simultaneously. The subsystem components are accessed via the child bus. A reset connection (R 1 , R 2 and R 3 ) and power connection (P 1 , P 2 AND P 3 ) extend from each bridge in the pair. A reset and/or power command from an SMM to the active bridge in the pair is used to power-up or reset the underlying board assembly. The SMMs are configured to provide independent reset and power commands to the bridges 192 and 194 , to allow powering and reset of each underlying board through the maintenance bus arrangement. In general the active bridge performs power-up. However, the bridges are configured to handshake, or otherwise communicate, to ensure that the board hardware is functioning properly before power-up occurs generally within the board.

According to a preferred embodiment, each bridge 192 , 194 comprises a commercially available Intel 87C54 microcontroller. This circuit package includes a built-in programmable storage device (an erasable programmable read-only memory EPROM) and 256 bytes of random access memory (RAM). This package is relatively low-cost and complete. Data traveling over the I 2 C bus is buffered in the RAM while basic routing and power control functions are preprogrammed into the bridge microcontroller EPROM. Though the 87C54 is the preferred embodiment, any microcontroller with sufficient I/O ports to drive both parent and child maintenance buses could instantiate the bridges 192 and 194 .

With further reference to FIG. 2, the exemplary parent board assembly 102 is shown in further detail. Particularly, the subsystem components 172 , interconnected with the I 2 C bus are illustrated. The electrically erasable programmable read only memory (EEPROM) carrying the board identification (ID), generally termed the IDEEPROM 202 , is provided on the bus. In addition, a light-emitting diode (LED) monitor 204 is provided. This LED provides a visible indication of the status of the board for an operator of the board. In addition, an environmental monitor chip 206 having I 2 C compatibility is provided. This chip typically monitors temperature and other important functions and transmits appropriate data and/or alarms regarding environment. Microprocessor information from CPU 1 ( 152 ) is also interconnected with the bus CB 1 vi an I 2 C interconnection. The CPU support information 208 is transmitted over the I 2 C bus, as well as other important status data. I 2 C interconnections with the dual inline memory module sockets (DIMMS) 210 of the board assembly are also provided by the child bus CB 1 . In addition, other I/O ports 212 with I 2 C capabilities may be serviced by the child bus CB 1 .

Having described the architecture of the maintenance bus arrangement, the protocol operating thereon is now described in detail. It is contemplated that the SMMs communicate with the bridges 192 and 194 via data packets, sent over the I 2 C bus with appropriate destination addresses. In general, each bridge and subsystem component contains its own unique address on the maintenance bus that make it identifiable by the SMMs. The SMMs have knowledge of the subsystem components on each board. Packets sent to and from the SMM have the bridge identification and the data within the packet is used to identify the particular subsystem device. A variety of protocols and communication techniques can be used according to this invention. Heretofore, I 2 C connections have operated using a highly simplified communication scheme without the benefit of addressing and protocol techniques. Because of the fault-tolerant nature of the bus arrangement and bridge system, addressing of control and monitor functions between the SMMs and the appropriate bridge are highly desirable.

Referring to FIG. 4, a command message/data packet structure 400 is shown schematically. While the illustrated message 400 is a command message initiated by the SMM, the packet structure to be described is generally a two-way message structure in which command versus response messages are differentiated by a unique command designator/identifier byte within the message header.

The command data packet 400 , which is transferred between the SMM and the various subsystem components, includes an address header 402 . This header is typically limited to one byte of information. In general, the architecture is arranged so that one byte (six address bits, one parity bit and one read/write bit) is sufficient to direct the packet to an appropriate bridge and corresponding subcomponent. In particular, the address specifies the target bridge through which data is transferred. As described further below, the final delivery of a command to a specific subcomponent is facilitated by the data bytes of the packet.

Following the address header 402 , a command byte 404 is provided. Specifically, if the transferred packet is a command packet from the SMM, the command byte is enabled with a specific recognized command byte code. The command bytes direct a particular subsystem component to perform a particular action, cause a bridge to perform a power or reset function, or to cause a subordinate bridge to perform a forwarding operation (described further below). The following is an exemplary list of command codes:

The foregoing list of commands is only exemplary, and is specifically adapted to an I 2 C standard application. Where the standard and available subcomponents vary from those described herein a different set of commands may be appropriate.

Following the command byte, is a one-byte number designating the overall message size 406 . The overall message size is indicated as a number of bytes by the one-byte string.

Next, a tag byte 408 is provided to the header packet. This tag byte is generated by the originator (the SMM) of the data packet. It is a unique one-byte number. The tag byte is repeated in a response message, as described below. In other words, when a message is originated and transferred to a subsystem, the appropriate response sent by the subsystem component should include the same tag byte, indicating that the message was received properly and acted upon. If the tag byte is not received, then an error has occurred an appropriate action is taken by the SMM.

Next, one or more bytes of forwarding data 410 are provided. The forwarding data enables a hierarchy of bridge structures to be established within the child bus. In connection with the forwarding data 410 , reference is also made to FIG. 3, which again illustrates the exemplary board assembly 102 . The subsystem 172 of this board includes the set subsystem components described above with reference to FIG. 2 . In addition, another bridge 302 (FIG. 3) is interconnected with the child bus CB 1 . This bridge is similar in configuration to the bridges 192 and 194 , and can be constructed from the same type of microcontroller circuit. The bridge 302 includes another discrete address that is recognized by the SMM so that data is transferred via the bridge 192 , 194 to the subordinate bridge 302 as if it were any travelling to any other subsystem component. The packet structure according to this embodiment enables a large number of remote components to be accessed notwithstanding the relatively small (one-byte) address 402 .

With reference again to FIG. 3, once a command message packet is received, the subordinate bridge 302 stores and forwards the message to the I 2 C-compatable ports on further computer circuitry 304 . Note that the CPU information block 208 is connected through the subordinate bridge 302 according to FIG. 3 . According to this embodiment, the processor information is located behind the child bridge, accounting for the depicted arrangement. In this example, the circuitry 304 includes another microprocessor (such as an Intel Xeon™) and/or associated memory and other peripherals. The above-described protocol enables messages to be transferred from the child bus through bridges to additional, subordinate bridges (such as bridge 302 ). Further components, such as circuitry 304 , can be accessed through these subordinate bridges. The subordinate bridge acts as a hierarchy to access the remote microprocessor circuitry 304 given the limited addressing available for the protocol. In this sense, the circuitry 304 is invisible to the child bus CB 1 . Based upon the above-described structure a series of subordinate bridges can be chained in series. Each subordinate bridge uses response data in the command message to route a received response back up the hierarchy of bridges to the originating SMM.

Each bridge in the bridge hierarchy stores knowledge of the message transferred therefrom. The response field 412 is stripped and stored by each bridge along the pathway, and the bridge's own field is substituted. This enables responses to command messages to return to their source (SMM) after passing back from the subordinate location through the bridge system.

Referring again to the command data packet 400 of FIG. 4, the structure includes zero or more bytes of data 414 , following the response byte field 412 . In general, data is provided as part of a particular transaction under the I 2 C protocol. This data is subcomponent-specific. In addition, the data provides the identification of a particular component on the child bridge, or in a remote, subordinate bridge hierarchy (such as the circuitry 304 ).

Finally, the command packet 400 includes a checksum 416 comprising a one-byte number. The checksum 416 indicates the sum of previous bytes in the message. If either the bridge, or a SMM discovers an incorrect checksum byte in a message, then the message is discarded. An erroneous message can trigger a variety of actions, including resending of the message, an alarm condition or shut down as appropriate. Note that at least two error modes for the SMM exist: (1) where it receives no response following the transmission of a message, resulting in “time-out” state; and (2) an error state in which a message is returned that is properly formatted, but is not understood by the bridge or the SMM as applicable. One possible corrective action is to retry sending of the message for a predetermined number of times. Certain subsystem devices that must return date, require a positive response, and thus, a full-way communication is required for an error not to occur. Conversely, where a power-on command is sent by the SMM or another non-response-dependent action occurs within the subsystem, then failure to return a response may cause an indication that an error may be present, but the action is possibly completed. These conditions are described further with respect to the response phase that is now discussed in detail.

The response message/data packet 500 is shown in FIG. 5 . It includes a one-byte address 502 designating the SMM. Where the command packet contains a command byte ( 404 ), the response packet contains a reserve value 504 , generally equal to zero. This, in fact, indicates that a response is being transferred over the maintenance bus.

The response packet 500 also includes an overall message size 506 similar to the command message size 406 . The tag 508 of the packet 500 is the original tag ( 408 ) in the command message, now being returned by subsystem components. Next is provided a one-byte status code 510 . The status code, as described above, is an indicator that a problem may exist, but the system does not have confirmation. For example, power may have been turned on, but no response is given. Different potential problems may be indicated by different status codes. In general, the status code is appended to the response by the active bridge.

If the command is, however, misunderstood or improperly formatted, then an appropriate error code 512 is generated. The absence of an error is generally indicated by a zero value. Conversely, the specific errors are indicated by particular byte codes. Exemplary error codes are listed in the following table:

Referring further to FIG. 5, the response packet 500 next includes zero or more bytes of response data 514 . This data can comprise environmental telemetry or variety of other required response data from the selected subsystem components. Finally, a check-sum byte 516 is provided. This checksum, again indicates the sum of the previous bytes in the response message. If the SMM or a bridge discovers an incorrect checksum, then the response is discarded and the command-response cycle is generally retried (or other appropriate action is taken).

It should now be clear that the foregoing architecture and accompanying protocol enables an effective and low-cost technique for implementing a fault-tolerant maintenance bus within a number of separate computer components.

The foregoing has been a detailed description of a preferred embodiment. Various modifications and additions can be made without departing from the spirit and scope of the invention. For example, while the maintenance bus is implemented as an I 2 C standard, it can be implemented in any other acceptable standard and the number of lines in the bus can be varied from the two lines shown. While a serial maintenance bus is utilized, it is contemplated that a parallel bus can be employed according to an alternate embodiment. Various components such as bridges and SMMs can be implemented using a variety of commercially available and customized circuits. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of the invention.