Failing over a virtual machine

1. Field of the Invention

This invention is related to the field of highly available computer systems and, more particularly, to the failing over of applications in computer systems, including clustered computer systems.

2. Description of the Related Art

Certain applications are often required to be available virtually uninterrupted, either 24 hours a day or at least during working hours. Various efforts have been undertaken to provide high availability services to support the high availability of such applications. Such highly-available applications may include email servers, web servers, database servers, etc.

Typically, efforts to provide high availability for a given application have focused on detecting that the application has failed and getting the application re-started. An application may fail due to an internal coding error in the application, an error in the operating system on which the application is running, an error in the hardware of the computer system on which the application is running, or a combination of any of the above errors. The errors may cause the application, or the operating system, to cease executing (e.g. a crash) or to stop functioning (e.g. a hang).

In some cases, each application for which high availability is desired may be assigned to a separate computer system. In this configuration, a failure of one application may not affect the operation of the other applications on the other computer systems. Additionally, this configuration allows for variations in the operating system on which the applications are run (e.g. different versions of the same operating system, or different operating systems). The cost of obtaining and maintaining separate computer systems for each application may be considerable.

Another method is to cluster a group of computer systems using specialized software (referred to as a cluster server) to control the group of computer systems. A given application may be executed on a first computer system of the group. The cluster server monitors the operation of the application and, if the cluster server detects that the application has failed, the cluster server may close the application on the first computer system and restart the application on another computer system. Typically, such cluster servers involve identifying, for each application supported by the cluster server, all of the state in the computer system that is needed to restart the application. In practice, such identification may be problematic and frequently involves making use of undocumented features of the application. Additionally, some applications may not function correctly when restarted on another machine. For example, the Exchange2000 application from Microsoft Corporation may not access a mailbox database used when the application was executing on another machine because Microsoft's Active Directory may identify that other machine as the owner of the database.

A computer system, carrier medium, and method for failing over a virtual machine are provided. A first computer system may be configured to execute a first application in a first virtual machine. A second computer system may be coupled to the first computer system. In response to a failure, the first computer system is configured to failover the first virtual machine to the second computer system. In one embodiment, failing over the first virtual machine may be accomplished using instructions comprising a program carried on a carrier medium. In one specific implementation, the program may include instructions which cause an image of a first virtual machine in which a first application is executing to be stored to a storage accessible to two or more computer systems in response to a failure of the first application on a first computer system of the two or more computer systems. Additionally, the instructions may activate a second virtual machine on the first computer system from the storage in response to receiving a message to failover the second virtual machine from another one of the computer systems.

In one particular implementation, cluster server software may be executing on the computer systems. The first virtual machine may be defined as a resource monitored by the cluster server software. The first virtual machine resource may be failed over in response to detection of a failure.

The following detailed description makes reference to the accompanying drawings, which are now briefly described.

FIG. 1 is a block diagram of one embodiment of a cluster of computer systems.

FIG. 2 is a block diagram of another embodiment of a cluster of computer systems.

FIG. 3 is a state machine diagram illustrating operation of one embodiment of certain components shown in FIG. 2 .

FIGS. 4–10 illustrate one embodiment of the cluster shown in FIG. 2 at various points during a failover of an application.

FIG. 11 is a block diagram of a carrier medium.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims.

Turning now to FIG. 1 , a block diagram is shown illustrating one embodiment of a cluster of computer systems. Other embodiments are possible and contemplated. In the embodiment of FIG. 1 , the cluster includes computer systems 10 A– 10 N, a network 12 , and a shared storage device 14 . The computer systems 10 A– 10 N are each coupled to the network 12 , and to the shared storage device 14 . Each of the computer systems 10 A– 10 N includes one or more virtual machines (e.g. virtual machines 16 A– 16 B on the computer system 10 A, virtual machines 16 C– 16 D on the computer system 10 B, and virtual machine 16 E on the computer system 10 N). The virtual machines are controlled by a virtual machine (VM) kernel (e.g. kernels 18 A, 18 B, and 18 N in the computer systems 10 A, 10 B, and 10 N, respectively). The virtual machines 16 A– 16 E and the VM kernels 18 A– 18 N may comprise software and/or data structures. The software may be executed on the underlying hardware in the computer systems 10 A– 10 N (e.g. the hardware 20 A, 20 B, and 20 N in the computer systems 10 A, 10 B, and 10 N). The hardware may include any desired circuitry. For example, the hardware 20 A is shown to include a processor, or central processing unit (CPU) 22 , a storage 24 , and input/output (I/O) circuitry 26 . In some embodiments, a computer system may include multiple CPUs 22 . Similarly, a virtual machine may comprise multiple virtual CPUs, in some embodiments.

As shown in FIG. 1 , each application executing on the computer systems 10 A– 10 N executes within a virtual machine 16 A– 16 E. Generally, a virtual machine comprises any combination of software, one or more data structures in memory, and/or one or more files stored on the shared storage device 14 . The virtual machine mimics the hardware used during execution of a given application. For example, in the virtual machine 16 A, an application 28 is shown. The application 28 is designed to execute within the operating system (O/S) 30 . Both the application 28 and the O/S 30 are coded with instructions executed by the virtual CPU 32 . Additionally, the application 28 and/or the O/S 30 may make use of various virtual storage and virtual I/O devices 34 . The virtual storage may represent any type of storage, such as memory, disk storage, tape storage, etc. The disk storage may be any type of disk (e.g. fixed disk, removable disk, compact disc read-only memory (CD-ROM), rewriteable or read/write CD, digital versatile disk (DVD) ROM, etc.). Each disk storage in the virtual machine may be mapped to a file on the shared storage device 14 , to a file on any storage device accessible to the hardware 20 A, or directly to a storage device accessible to the hardware 20 A. A storage device may be accessible to the hardware 20 A if the storage device is connected to the hardware 20 A or if the storage device is coupled to a network to which the hardware 20 A is coupled. For example, storage devices attached to a network may include network attached storage (NAS) or storage area network (SAN) devices, or IP-based storage devices of any type may be used. The virtual I/O devices may include any type of I/O devices, including modems, audio devices, video devices, network interface cards (NICs), universal serial bus (USB) ports, firewire (IEEE 1394) ports, serial ports, parallel ports, etc. Generally, each virtual I/O device may be mapped to a corresponding I/O device in the underlying hardware (e.g. the I/O circuitry 26 ) or may be emulated in software if no corresponding I/O device is included in the I/O circuitry 26 .

The virtual machine in which an application is executing encompasses the entire system state associated with an application. Generally, when a virtual machine is active (i.e. the application within the virtual machine is executing), the virtual machine may be stored in the memory of the computer system on which the virtual machine is executing (although the VM kernel may support a paging system in which various pages of the memory storing the virtual machine may be paged out to local storage in the computer system) and in the files on the shared storage device 14 which are mapped to the virtual storage devices in the virtual machine. The VM kernel may support a command to suspend the virtual machine. In response to the command, the VM kernel may write an image of the virtual machine to the shared storage device 14 , thus capturing the current state of the executing application. The image may include one or more files written in response to the suspend command, capturing the state of the virtual machine that was in memory in the computer system, as well as the files stored on the shared storage device 14 that represent the virtual disks included in the virtual machine. The state may include not only files written by the application, but uncommitted changes to files which may still be in the memory within the virtual machine, the state of the hardware (including the processor 32 , the memory in the virtual machine, etc.) within the virtual machine, etc. Thus, the image may be a snapshot of the state of the executing application.

A suspended virtual machine may be resumed using a resume command supported by the VM kernel. In response to the resume command, the VM kernel may read the image of the suspended virtual machine from disk and may activate the virtual machine in the computer system.

An application may be failed over in the cluster of computer systems 10 A– 10 N. If an application is detected as failing, the suspend command may be used to suspend the virtual machine in which the application is executing. Thus, the image of the virtual machine may be written to the shared storage device 14 . Another computer system 10 A– 10 N in the cluster may be selected to execute the application. The resume command may be used in the selected computer system 10 A– 10 N to resume the virtual machine from the image on the shared storage device 14 . In this manner, the application may begin executing on the selected computer system 10 A– 10 N after experiencing a failure on the previous computer system 10 A– 10 N. In this context, an application may fail due to an internal coding error in the application, an error in the operating system on which the application is running, an error in the virtual machine or the VM kernel on which the virtual machine is executing, an error in the hardware of the computer system on which the application is running, or a combination of any of the above errors. The errors may cause the application, or the operating system, to cease executing (e.g. a crash) or to stop functioning (e.g. a hang). It is noted that, in some embodiments, the computer systems 10 A– 10 N may be configured to periodically checkpoint the virtual machines executing thereon, thus providing a set of possible images from which to resume when a failover occurs.

Since the state of the virtual machine is failed over from one computer system to another, the process of identifying which machine state is to be failed over with a given application may be avoided. Additionally, applications which may be dependent on the particular computer system on which they are executing (e.g. applications using Microsoft's Active Directory) may be failed over. Since the virtual machine in which the application executes is the same on any computer system on which it executes, the application appears to be executing on the same computer system. This appearance is independent of which actual computer system 10 A– 10 N is executing the virtual machine including the application.

In the illustrated embodiment, cluster server software is used to manage the cluster and to failover applications (by failing over virtual machines). The cluster server software may be installed on each computer system 10 A– 10 N in the cluster. For example, the cluster server software 36 A may be installed on the computer system 10 A, the cluster server software 36 B may be installed on the computer system 10 B, and the cluster server software 36 N may be installed on the computer system 10 N. The cluster server software on each computer system 10 A– 10 N communicates with the other cluster server software on the other machines to manage the cluster, select computer systems to execute applications, monitor operation of the applications and underlying hardware to detect failures, and failover applications when a failure is detected. In one embodiment, the cluster server software also executes in a virtual machine. For example, in the embodiment of FIG. 1 , the cluster server 36 A executes in the virtual machine 16 B, the cluster server 36 B executes in the virtual machine 16 C, and the cluster server 36 N executes in the virtual machine 16 E. In one implementation, the cluster server may operate similar to the Veritas Cluster Server™ product available from Veritas Software Corporation (Mountain View, Calif.).

The cluster server software may define various resources that it monitors and fails over a resource (or a group of dependent resources) if one of the resources are detected as failing. Resources may include software (e.g. applications, operating system software components, operating system services, etc.) and hardware (e.g. disk drives, network interface circuitry, etc.). In such embodiments, the virtual machine may be considered as a resource which may be failed over by the cluster server software. The number of resources associated with a given application may be reduced, since the virtual machine encompasses many software and hardware components that may have otherwise been tracked as separate resources by the cluster server software. The failing over of applications within virtual machines may be simpler than tracking the various resources separately.

The virtual hardware in the virtual machine 16 A (and other virtual machines such as virtual machines 16 B– 16 E) may be similar to the hardware 20 A included in the computer system 10 A. For example, the virtual CPU 32 may implement the same instruction set architecture as the processor 22 . In such cases, the virtual CPU 32 may be one or more data structures storing the processor state for the virtual machine 16 A. The application and O/S software instructions may execute on the CPU 22 when the virtual machine 16 A is scheduled for execution by the VM kernel 18 A. When the VM kernel 18 A schedules another virtual machine for execution (e.g. the virtual machine 16 B), the VM kernel 18 A may write the state of the processor into the virtual CPU 32 data structure. Alternatively, the virtual CPU 32 may be different from the CPU 22 . For example, the virtual CPU 32 may comprise software coded using instructions from the instruction set supported by the CPU 22 to emulate instruction execution according to the instruction set architecture of the virtual CPU 32 . Alternatively, the VM kernel 18 A may emulate the operation of the hardware in the virtual machine. Similarly, other virtual hardware may be emulated in software if not included in the hardware 20 A.

Different virtual machines which execute on the same computer system 10 A may differ. For example, the O/S 30 included in each virtual machine may differ. Different virtual machines may employ different versions of the same O/S (e.g. Microsoft Windows NT with different service packs installed), different versions of the same O/S family (e.g. Microsoft Windows NT and Microsoft Windows2000), or different O/Ss (e.g. Microsoft Windows NT, Linux, Sun Solaris, etc.).

Generally, the VM kernel may be responsible for managing the virtual machines on a given computer system. The VM kernel may schedule virtual machines for execution on the underlying hardware, using any scheduling scheme. For example, a time division multiplexed scheme may be used to assign time slots to each virtual machine. Additionally, the VM kernel may handle the suspending and resuming of virtual machines responsive to suspend and resume commands. The commands may be received from a virtual machine (e.g. the virtual machine in which the cluster server is executing). In one embodiment, the VM kernel may be the ESX product available from VMWare, Inc. (Palo Alto, Calif.).

In the illustrated embodiment, the VM kernel may execute directly on the underlying hardware (i.e. without an underlying operating system). In other embodiments, the VM kernel may be designed to execute within an operating system. For example, the GSX product available from VMWare, Inc. may execute under various versions of Microsoft's Windows operating system and/or the Linux operating systems.

The network 12 may comprise any network technology in various embodiments. The network 12 may be a local area network, wide area network, intranet network, Internet network, or any other type of network. The network 12 may be designed to be continuously available (although network outages may occur), or may be intermittent (e.g. a modem connection made between a computer system in a user's home and a computer system in a user's workplace). Any network protocol may be used. For example, the network 12 may be an Ethernet network. Alternatively, the network may be a token ring network, etc. Additionally, the network 12 may comprise shared storage or shared memory for communicating between the computer systems, in some embodiments.

The shared storage device 14 may be any type of storage device to which the computer systems 10 A– 10 N may be coupled. For example, the shared storage device 14 may comprise one or more small computer system interface (SCSI) drives attached to a SCSI bus shared by the computer systems 10 A– 10 N. Alternatively, the shared storage device 14 may couple to the network 12 (e.g. network attached storage (NAS) or storage area network (SAN) technologies may be used). The shared storage device may also comprise memory. Generally, the shared storage device 14 may be any device capable of storing data.

Turning now to FIG. 2 , a block diagram of a second embodiment of a cluster of computer systems is shown. Other embodiments are possible and contemplated. In the embodiment of FIG. 2 , the cluster includes computer systems 10 A and 10 B. Other computer systems may be included, as desired, in the cluster. The cluster shown in FIG. 2 will be used as an example to highlight the failover of an application. As used herein, a failover refers to the starting of application execution on a different computer system in response to a failure of the application on a given computer system. In the illustrated embodiment, the computer system 10 A includes the virtual machines 16 A and 16 B, the VM kernel 18 A, and the hardware 20 A, similar to the embodiment of FIG. 1 . The virtual machine 16 A includes the application 28 , the O/S 30 , and storage 34 A (part of the storage and I/O 34 ). The storage 34 A is mapped, by the VM kernel 18 A, to the shared storage device 14 . Additionally, the virtual machine 16 A includes a proxy agent 40 configured to communicate with the cluster server 36 A in the virtual machine 16 B. The cluster server 36 A may be designed to execute within an O/S 30 B. Additionally, the virtual machine 16 B may include a pair of virtual NICs 42 A– 42 B. In the illustrated embodiment, the hardware 20 A includes NICs 44 A– 44 E. The virtual machine 16 B is configured such that each of the virtual NICs 42 A– 42 B are mapped by the VM kernel 18 A to one of the NICs 44 A– 44 E (e.g. the NICs 44 D– 44 E, respectively, in this embodiment). The computer system 10 B includes the virtual machine 16 C, the virtual machine 16 A with the same components as the computer system 10 A includes, the VM kernel 18 B, and the hardware 20 B. The hardware 20 B includes NICs 44 F– 44 J. The virtual machine 16 C includes the cluster server 36 B, an O/S 30 C, and virtual NICs 42 C– 42 D which are mapped to two of the NICs 44 F– 44 J (e.g. the NICs 44 I and 44 J, in this embodiment). The NICs 44 E and 44 J are coupled to communicate, as are the NICs 44 D and 44 I and the NICS 44 C and 44 H.

The virtual machine 16 A may be active at any given time in only one of the computer systems 10 A– 10 B. Since the image of the virtual machine 16 A is stored on the shared storage device 14 in response to a suspend command, the virtual machine 16 A may be executed on either computer system 10 A– 10 B. If, for example, the virtual machine 16 A is executing on the computer system 10 A and a failure is detected, the virtual machine 16 A may be failed over to the computer system 10 B. Likewise, if the virtual machine 16 A is executing on the computer system 10 B and a failure is detected, the virtual machine 16 A may be failed over to the computer system 10 A. As mentioned previously, in some embodiments, multiple suspended images of the virtual machine may be maintained on the shared storage 14 , and one of the checkpoint images may be selected for resuming the virtual machine in a failover. Additionally, in some embodiments, a failover may occur to the same computer system 10 A– 10 B on which the failure occurred, if desired.

While the embodiment shown in FIG. 2 includes two computer systems, other embodiments may include more than two computer systems. If more than two computer systems are included, a computer system may be selected to be the receiver of a given failover. Various selection criteria may be used. For example, the cluster server may monitor the load on each computer system and select the computer system with the lowest current load for the failover. Alternatively, a fixed priority scheme or a round-robin scheme may be used.

The proxy agent 40 may be used to monitor the application 28 , to detect any errors that may occur in the execution of the application. The proxy agent 40 may monitor the processes comprising the application 28 to see that the processes are active and consuming/releasing system resources properly. The proxy agent 40 may be configured to periodically make requests to the application 28 to determine if the application 28 responds properly. The proxy agent 40 may be configured to connect to defined ports of the application 28 to request status information on the application 28 . In embodiments running on Microsoft's Windows operating system as the O/S 30 , the proxy agent 40 may replicate the registry entries corresponding to the application 28 to the shared storage device 14 . Alternatively, the registry entries may be copied from the virtual storage device of the failed virtual machine after the virtual machine is failed over.

The cluster server 36 A may be configured to monitor the virtual machine 16 A to detect any failures in the virtual machine as a whole. Particularly, in one embodiment, the cluster server 36 A may ensure that the virtual machine 16 A is active and may ping the IP address of the virtual machine 16 A (e.g. the IP address of a virtual NIC in the virtual machine 16 A). If the ping is successful, the cluster server 36 A may connect to the proxy agent 40 to collect status information regarding the application 28 . In some embodiments, the cluster server 36 A may also be configured to reserve access to the shared storage device 14 when such access is desired.

While the illustrated embodiment includes the proxy agent 40 running within the virtual machine 16 A, other embodiments may not include the proxy agent 40 . For example, an application may be designed to communicate status information to the cluster server 36 A directly.

The cluster server 36 A and the cluster server 36 B communicate using the virtual NICs 42 A– 42 D. Two NICs may be used for each cluster server to provide redundancy, to allow the cluster servers to distinguish between a network error between the computer systems and an error within one of the computer systems. Other embodiments may use one NIC for each cluster server if redundancy is not desired. The communication between the cluster servers 36 A– 36 B may include a “heartbeat” communication indicating that the sending computer system 10 A– 10 B is still running. The heartbeat may be sent and checked at predetermined intervals. Alternatively, heartbeat information may be stored by each cluster server 36 A– 36 B on a shared storage device or in shared memory, and the heartbeat information may be accessed from the shared storage device or memory as desired. Any mechanism for communicating heartbeat information among cluster servers 36 A– 36 B may be used. Additionally, the communication may include status indicating the state of the system. A failover message may also be communicated, requesting the failover of a virtual machine in which an application experiences a failure.

The cluster server software in each computer system may monitor for the heartbeat communications from each other computer system. If the heartbeat communications from a given computer system cease, the cluster server software may conclude that the given computer system has failed. The cluster server software may attempt to suspend the virtual machines executing on the given computer system remotely. If successful, the suspended virtual machines may be resumed on another computer system. If not successful, the virtual machines on the given computer system may be restarted on other computer systems from a clean boot (or from one of the checkpoint images, in embodiments in which multiple images are maintained for a virtual machine on the shared storage device 14 ). While some state may be lost in such a case, the applications which experienced failure on the given computer system may be available for use.

The NICs 44 C and 44 H may be used for other communications between the computer systems 10 A– 10 B. The remaining NICS 44 A– 44 B and 44 F– 44 G may be provided for use by the applications executing on the virtual machines within a given computer system 10 A– 10 B. For example, an email server like Microsoft's Exchange2000 may use one or more NICs to receive requests to check email accounts.

Turning next to FIG. 3 , a state machine diagram is shown illustrating various states of one embodiment of the software illustrated in FIG. 2 . Other embodiments are possible and contemplated. Thick lines (e.g. reference numeral 50 ) represent transitions between states. Thin lines (e.g. reference numeral 52 ) represent communication between software. Dashed lines (e.g. reference numeral 54 ) indicate state changes due to an external event or a state change in the source of the dashed line. The state machine will be described below with reference to the virtual machine 16 A, the proxy agent 40 , the application 28 , and the cluster server 36 A on the computer system 10 A. Similar state machines may be used in other computer systems in the cluster. The virtual machine 16 A has two states in the illustrated diagram: a powered on state 56 and a powered off state 58 . The proxy agent 40 has three states in the illustrated diagram: a monitor state 60 , a shutdown state 62 , and a tell-CS state 64 . The application 28 has three states in the illustrated diagram: a running state 66 , a failed state 68 , and a closed state 70 . The cluster server 36 A has four states in the illustrated diagram: an online state 72 , a cleared state 74 , a faulted state 76 , and an offline state 78 .

During normal operation, the virtual machine 16 A is in the powered on state 56 , the proxy agent 40 is in the monitor state 60 , and the application 28 is in the running state 66 . Generally, when the virtual machine 16 A transitions to the powered on state, the proxy agent 40 is initialized in the monitor state 60 and the application is started (the running state 66 ). In the monitor state 60 , the proxy agent monitors the application 28 to detect failures.

The application 28 transitions from the running state 66 to the failed state 68 in response to the occurrence of a failure. In the failed state 68 , the application 28 is no longer running correctly. The application may be hung, may have crashed, or may have one or more services that have ceased operating. The proxy agent 40 detects the failure in the application 28 , and transitions to the shutdown state 62 .

In the shutdown state 62 , the proxy agent 40 transmits a close message to the application 28 . The application 28 transitions to the closed state 70 in response to the close message. Additionally, the proxy agent 40 transitions to the tell-CS state 64 . The proxy agent remains in the tell-CS state 64 until contacted by the cluster server 36 A (which is in the online state 72 ). The proxy agent 40 transmits a status of faulted to the cluster server 36 A. The status may optionally include information identifying the fault, if desired.

In response to the faulted status from the proxy agent 40 , the cluster server 36 A transmits a power off message to the virtual machine 16 A, causing the virtual machine 16 A to transition from the powered on state 56 to the powered off state 58 . The virtual machine 16 A may be suspended (writing the image of the virtual machine 16 A to the shared storage device 14 ) to allow failover of the virtual machine 16 A to another computer system. Additionally, the cluster server 36 A transitions to the cleared state 74 .

In the cleared state 74 , the cluster server 36 A transmits a failover message to the cluster server in the computer system selected to execute the failed application (the “new node” in FIG. 3 ). The failover message may include the name of the virtual machine to be failed over, which may indicate the file or files on the shared storage device 14 that are used to establish the virtual machine in the new node. The cluster server 36 A then transitions to the faulted state 76 . While the cluster server 36 A is in the faulted state 76 , the state of the computer system 10 A may remain frozen. New applications may not be started on the computer system 10 A through the cluster server 36 A. If the computer system 10 A itself is the source of the failure, preventing new applications from running on the computer system 10 A may prevent additional failures until the failure can be corrected. Additionally, by freezing the state of the computer system 10 A, the state may be analyzed to determine the failure and if any correcting measures may be indicated.

The cluster server 36 A transitions from the faulted state 76 to the offline state 78 in response to an external clear message. The clear message may be transmitted by a user, for example, after examining the computer system 10 A to determine the cause of the failure. Alternatively, the clear message may be transmitted by the user without first examining the computer system to bring the computer system 10 A back into eligibility for executing applications within the cluster.

In the offline state 78 , the cluster server 36 A may receive a failover message from another computer system in which a failure of an application has occurred (the “old node” in FIG. 3 ). In response to the failover message, the cluster server 36 A transitions to the online state 72 . From the online state 72 , the cluster server transmits a power on message to the virtual machine 16 A to cause the virtual machine 16 A to be resumed. The power on may include transmitting the resume command to the VM kernel 18 A. The virtual machine 16 A powers up and is loaded with the state from the image of the virtual machine 16 A on the shared storage device 14 (or from a selected image, if multiple checkpoint images are maintained). The proxy agent 40 is initialized to the monitor state 60 . The application 28 is started (the running state 66 ).

It is noted that, while a state machine is shown in FIG. 3 with various states for various software included in the computer system 10 A, the software may not formally be divided into states. Instead, the software may have various code sequences, delimited by branches which test for the causes of transitions shown in FIG. 3 , and may progress to the next code sequence based on the results of the branches. Alternatively, each state may be a different routine with the code for executing that state and determining state transitions. Still further, the states of the application 28 may not describe different sections of the application 28 , but rather its state of execution.

Turning now to FIGS. 4–10 , a failover of an application from the computer system 10 A to the computer system 10 B is shown at various points in time during the failover. For each piece of software, the name is listed followed by its state (for the embodiment of FIG. 3 ) in parentheses.

In FIG. 4 , the computer system 10 A is executing the virtual machine 16 A including the application 28 and the proxy agent 40 . The application 28 is in the running state 66 , the proxy agent 40 is in the monitor state 60 , and the virtual machine 16 A is in the powered on state 56 . The virtual machines 16 B and 16 C are in the powered on state as well. The cluster server 36 A, executing within the virtual machine 16 B, is in the online state 72 . The cluster server 36 B, executing within the virtual machine 16 C, is in the offline state 78 .

As illustrated in FIG. 5 , the application 28 fails on the computer system 10 A. Thus, the application 28 is in the failed state 68 , and the proxy agent 40 detects the failure. In response to the failure, the proxy agent 40 transitions to the shutdown state 62 ( FIG. 6 ). The proxy agent 40 transmits a close message to the application 28 . As shown in FIG. 7 , the application 28 transitions to the closed state 70 in response to the close message and the proxy agent 40 transitions to the tell-CS state 64 . The cluster server 36 A receives the faulted status from the proxy agent 40 , and transmits a power off message to the virtual machine 16 A in response.

FIG. 8 illustrates the virtual machine 16 A powered off, and an image 80 of the virtual machine 16 A stored on the shared storage device 14 . The cluster server 36 A has transitioned to the cleared state, and transmits a failover message to the cluster server 36 B. The cluster server 36 A then transitions to the faulted state 76 ( FIG. 9 ), and the cluster server 36 B transitions to the online state 72 . The cluster server 36 B transmits a power on message to the virtual machine 16 A (which is shown in a powered off state in FIG. 9 , but may generally not exist in the computer system 10 B yet). As part of powering on the virtual machine 16 A on the computer system 10 B, the computer system 10 B reads the image 80 of the virtual machine 16 A from the shared storage device 14 ( FIG. 10 ) and resumes the virtual machine 16 A. As mentioned previously, the image 80 may be selected from one of several images of the virtual machine 16 A that may be stored on the shared storage device 14 , in some embodiments. The virtual machine 16 A activates on the computer system 10 B, with the application 28 in the running state 66 and the proxy agent 40 in the monitor state 60 .

It is noted that, while the above description has referred to a cluster of computer systems managed by cluster server software, other embodiments may failover virtual machines containing applications from one computer system to another without configuring the computer systems into clusters. For example, software may be executed on a first computer system including a first virtual machine in which a first application executes. The software may monitor the first application and the first virtual machine to detect failures, and may cause the failover to another computer system. Alternatively, the software may executed on a second computer system separate from the first computer system. Still further, the failover may be a manual process performed by a user.

It is further noted that, while the above examples have shown one application executing in each virtual machine, a given virtual machine may include one or more applications, as desired.

Turning next to FIG. 11 , a block diagram of a carrier medium 300 including one or pieces of software described above is shown. Generally speaking, a carrier medium may include storage media such as magnetic or optical media, e.g., disk or CD-ROM, volatile or non-volatile memory media such as RAM (e.g. SDRAM, RDRAM, SRAM, etc.), ROM, etc., as well as transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link.

As illustrated in FIG. 11 , the carrier medium 300 may store one or more of the cluster server 36 A, the VM kernel 18 A, or the proxy agent 40 . In other embodiments, the application 28 may also be stored, as may be the O/S 30 and any other software. Generally, as used herein, software comprises one or more instructions which, when executed, performs the function described for the software. The instructions may be machine level instructions from an instruction set implemented in the virtual CPU 32 or the CPU 22 , or may be higher level instructions (e.g. shell scripts, interpretive languages, etc.).

Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.