System and method of lowering overhead and latency needed to service operating system interrupts

The present invention relates generally to servicing of interrupts by an operating system (“OS”), and more particularly, to a method of achieving low overhead and low latency for OS interrupts.

On the majority of computer hardware systems hosting commercial operating systems, devices exist whose function it is to generate hardware interrupts to signal the occurrence of asynchronous events such as a packet arriving on a wire for a network interface card (“NIC”) or an event signaling the completion of a disk I/O request. The handling of these kinds of asynchronous events has traditionally been accomplished in one of two ways, including polling and interrupt-driven. Each of these methods has advantages, as well as disadvantages.

In particular, with regard to interrupt-driven handling of asynchronous events, during normal operation, when a device on an I/O bus, such as a storage device, NIC, or I/O device, has work that needs to be done, it asserts an interrupt to the processor. If interrupts are enabled at the processor, receipt of an interrupt causes the processor to enter a special state to service the interrupt. First, the processor must save context, which means saving the state of the registers to memory so the processor can use the registers during handling of the interrupt without destroying the state of the interrupted process. The processor then vectors to a designated location in memory and runs an interrupt service routine, or interrupt handler to service the interrupt, after which the state, or context, is restored and a return from interrupt is executed. The processor then continues executing from the position in the code at which it was interrupted.

In today's computing environment, the bit rate of a packet on the wire is steadily increasing. Hardware will traditionally deliver one interrupt to the OS for each packet of information received on the wire. As the transfer rate increases, the amount of interrupt processing required by the CPU also increases, reducing the amount of processing that can be performed by the CPU for other purposes. Interrupts are inherently inefficient, as they require some state of the interrupted process to be saved before interrupt handling can begin. This state must later be restored when interrupt handling is completed. This saving and restoring on each interrupt is a waste of CPU cycles. This problem is only compounded as the number of interrupts increases in the system.

On Intel's 32-bit architecture, the context to be saved is small, and therefore interrupt-driven drivers are not a serious problem. However, Intel's new 64-bit architecture requires a significant amount of information to be saved and restored each time an interrupt is taken. This enormous expense is not tolerable for a highly performant OS. On the other hand, the advantage of using an interrupt-driven method is that there is almost no latency between the occurrence of the asynchronous event and the time in which the OS services that event.

In contrast, in the polling mechanism, the OS informs the hardware devices to not generate interrupts. Instead, the OS will poll the devices' drivers at regular intervals to determine whether there is work that needs to be performed. This method clearly has the advantage of completely avoiding the costly overhead of interrupt-driven drivers; however, it suffers from two notable deficiencies. First, the OS must constantly be polling the drivers to determine whether there is work to be done. This act itself is a waste of the CPU's processing time and should be avoided. Secondly, there is some latency between the time the driver would have normally interrupted processing and the time the OS actually polls the device. This latency, if too great, can also result in unacceptable performance.

Therefore, what is needed is a method of handling asynchronous events in a manner that results in both lower latency and lower overhead than the aforementioned traditional methods of handling the same.

In one embodiment, a system and method of method for achieving low overhead and latency in servicing operating system (“OS”) interrupts takes advantage of the availability of a “lightweight interrupt” available on various processors, including a 64-bit processor commercially available from Intel Corporation of Santa Clara, Calif., as “Itanium” (hereinafter “IA-64”). In a preferred embodiment, when a deferrable interrupt occurs, a lightweight interrupt handler, in combination with banked and special control registers, is used to inform the OS that an interrupt occurred and prevent the CPU and associated hardware from fully acknowledging or servicing the interrupt from a hardware standpoint until a designated future time. The OS sets to TRUE a per CPU software flag, referred to as a

“Deferred_Interrupt_Pending” flag, indicating that an interrupt has been received but is still pending. A return from the lightweight interrupt handler is then executed. In this manner, the interrupt is partially acknowledged by the CPU, but the device that caused the interrupt is still awaiting service; i.e., from the device's point of view, the interrupt remains pending. To guarantee low latency, a heavyweight (“non-deferrable”), time-based interrupt that flushes all deferrable interrupts is scheduled to occur within a specified time.

At a later time, for example, when drivers would normally be polled for work, the per CPU Deferred_Interrupt_Pending flag is checked to see if there is interrupt work to be performed. If so, with all interrupts disabled at the CPU, the pending interrupts are unmasked to determine which ones need to be serviced. In one embodiment, this is accomplished by, examining a special register in the CPU designated for this purpose. Subsequently, the deferred interrupts are batch processed sequentially by calling their service routines in order of priority without any context save required.

A technical advantage achieved with the invention is that it incorporates the most desirable attributes (low latency, low overhead), while avoiding the undesirable aspects (high latency, high overhead), of both interrupt-driven and polled driver architectures.

FIG. 1 is a system block diagram of a processing environment for implementing a preferred embodiment of the invention.

FIG. 2 is a high level state diagram of a preferred embodiment of the invention.

FIG. 3 is a more detailed state diagram of a preferred embodiment of the invention.

FIG. 4 is a flowchart illustrating the operation of a hardware interrupt context of a preferred embodiment of the invention.

FIG. 5 is a flowchart illustrating the operation of a deferred interrupt poll routine of a preferred embodiment of the invention.

FIG. 1 illustrates a portion of a computer system 100 embodying features of a preferred embodiment. The computer system 100 includes an operating system (“OS”) 102 comprising instructions executed by a CPU 104 . A plurality of devices 106 individually generate interrupts to the CPU 104 when the corresponding device has work to be performed by the CPU. As will be described in detail below, for purposes of the preferred embodiment, the OS 102 operates in three main states, including an Operating System Miscellaneous Activities state, a Hardware Interrupt Context state, and a Deferred Interrupt Poll Routine state. In addition, the OS 102 includes a per CPU Deferred_Interrupt_Pending flag 108 , the purpose and function of which will be described in greater detail below.

In a preferred embodiment, and for the purposes of example, the CPU 104 comprises an IA-64, although it will be recognized that other types of processors may be used and the following description of the preferred embodiment with reference to the IA-64 is not intended to limit application of the invention to that particular architecture. The CPU 104 includes a plurality of important registers, including a PSR.i/PSR.bn register 112 , a TPR.mmi/TPR.mic register 114 , a an IVR register 116 , an IVA register 118 , an EOI register 120 , an ITC register 122 , an ITM register 124 , a plurality of banked registers GR 16 -GR 31 , collectively designated by a reference numeral 126 , and four interrupt request registers IRR 0 -IRR 3 , collectively designated by a reference numeral 128 .

FIG. 2 illustrates a high level state diagram of a preferred embodiment of the present invention for servicing OS interrupts. As shown in FIG. 2, the CPU 104 begins, for purposes of example, in first state 200 , designated “Operating System Miscellaneous Activities”. As will be described in greater details in FIGS. 3-5, the CPU 104 transitions between this first state 200 and second and third states 202 , 204 , respectively designated “Deferred Interrupt Poll routine” and “Hardware Interrupt Context”.

FIG. 3 illustrates a more detailed state diagram of the preferred embodiment of the present invention for servicing OS interrupts illustrated in FIG. 2 . In particular, FIG. 3 illustrates several “sub-states” of the second and third states 202 , 204 . Beginning in the Operating System Miscellaneous Activities state 200 , receipt of a hardware interrupt causes a transition 300 to a “Lightweight Interrupt Handler” sub-state 204 a . The lightweight interrupt is supported on IA-64 by the presence of an interrupt vector table pointed to by the CPU IVA register 118 , a series of general scratch registers, in this case, banked registers GR 16 -GR 31 126 and the PRS.bn register 112 , which is used to manually switch between banks 0 and 1 for banked registers GR 16 -GR 31 126 . From this sub-state 204 a , if the hardware interrupt is determined to be deferrable by examining the interrupt request registers IRR 0 -IRR 3 128 , the deferrable interrupts are held pending by a modification to the TPR.mic register 114 , the Deferred_Interrupt_Pending flag 108 is set to TRUE, and a transition 302 is made back to the Operating System Miscellaneous Activities state 200 responsive to a return from interrupt. If, however, the hardware interrupt is determined to be non-deferrable, a transition 304 is made to a “Heavyweight Context Save” substate 204 b . Transition 304 may also occur if a time quantum determined by the OS scheduler, for previously deferred interrupts, has expired. Expiration of this time quantum may be determined by a number of methods, including, for example, the OS examining the contents of the IA-64 timer registers ITC register 122 and the ITM register 124 . Ensuring that previously deferred interrupts are not held pending longer than the prescribed time quantum guarantees an overall low latency for interrupt processing. Once the context is saved in a conventional fashion, a transition 306 is made to a “Service All Non-Deferred and Deferred Interrupts” sub-state 204 c , in which all interrupts are serviced sequentially in order of priority. During interrupt processing, the EOI register 120 is used to indicate end-of-interrupt processing for each interrupt vector serviced. Once interrupt processing is complete, the Deferred_Interrupt_Pending flag 108 is set to FALSE and a transition 308 is made to a “Heavyweight Context Restore” sub-state 204 d . Once context is restored, a transition 310 is made back to the Operating System Miscellaneous Activities state 200 responsive to a return from interrupt. Sub-states 204 a - 204 d comprise substates of the Hardware Interrupt Context state 204 of FIG. 2 .

From the Operating System Miscellaneous Activities state 200 , a call is made to a Deferred Interrupt Poll routine at regular intervals. The frequency with which the Deferred Interrupt Poll routine is called exceeds the expiration time quantum for deferrable interrupts. This prevents subsequent interrupts that would normally have been deferred from taking transition 304 from state 202 a as a result of the expiration of the time limit or quantum for servicing a previously deferred interrupt. A call to the Deferred Interrupt Poll routine causes a transition 312 to a “Mask All Processor Interrupts and Test the Deferred_Interrupt_Pending Flag” sub-state 202 a . If it is determined that the Deferred_Interrupt_Pending flag 108 is set to TRUE, a transition 314 is made to a “Service All Non-Deferred and Deferred Interrupts” sub-state 202 b , in which all interrupts are serviced sequentially in order of priority. During interrupt processing, the EOI register 120 is used to indicate end-of-interrupt processing for each interrupt vector serviced. Once all of the interrupts have been serviced, the Deferred_Interrupt_Pending flag 108 is set to FALSE and a transition 316 is made to an “Unmask Processor Interrupts” sub-state 202 c . Returning to the sub-state 202 a , if it is determined that the Deferred_Interrupt_Pending flag 108 is set to FALSE, a transition 318 is made directly to the Unmask Processor Interrupts sub-state 202 c . From the Unmask Processor Interrupts sub-state 202 c , a transition 320 is made back to the Operating System Miscellaneous Activities state 200 when the Deferred Interrupt Poll routine completes. Sub-states 202 a - 202 c comprise sub-states of the Deferred Interrupt Poll routine state 202 of FIG. 2 .

Referring now to FIG. 4, operation of the Hardware Interrupt Context will be described in greater detail. Again, processing begins at an Operating System Process Time step 400 . Responsive to receipt of a hardware interrupt, processing proceeds to step 402 , in which a determination is made whether an interrupt is pending that cannot be deferred. In other words, a determination is made whether a non-deferrable interrupt is pending. In the preferred embodiment, this is accomplished by examining the four interrupt request registers IRR 0 -IRR 3 128 . For example, all external interrupts represented in IRR 0 and IRR 1 may have been designated deferrable. Therefore, any bit set in IRR 2 or IRR 3 indicates that a non-deferrable interrupt is pending. If all pending interrupts are deferrable, execution proceeds to step 404 , in which all deferrable interrupts are disabled. In the preferred embodiment, this is accomplished by “masking” all deferrable interrupts using the TPR.mic register 114 . Execution then proceeds to step 406 , in which the Deferred_Interrupt_Pending flag 108 is set to TRUE. In step 408 , a return from interrupt is executed, returning processing to step 400 . As a result of steps 404 - 408 , all subsequent deferrable interrupts will be held as pending at the CPU 104 and will not be serviced immediately.

Returning to step 402 , if it is determined that there is an interrupt pending that cannot be deferred, execution proceeds to step 410 , in which a heavyweight interrupt context save is performed. This step encompasses the context save that is typically necessary in connection with a hardware interrupt; i.e., saving the state of the registers to memory. Execution then proceeds to step 412 , in which all deferrable interrupts are enabled. Specifically, at this point, all pending deferrable interrupts are unmasked, using the TPR.mic register 114 , so that they can be serviced. In step 414 , all non-deferred and deferred interrupts are serviced sequentially in order of priority. In step 416 , the Deferred_Interrupt_Pending flag 108 is set to FALSE. In step 418 , a heavyweight interrupt context restore is performed, resulting in the state of the registers being restored so that the processor can continue executing the interrupted process, and in step 420 , a return from interrupt is executed, returning processing to step 400 .

Comparing FIGS. 3 and 4, it will be recognized that steps 402 - 408 correspond to sub-state 204 a ; step 410 corresponds to sub-state 204 b ; steps 412 - 416 correspond to sub-state 204 c ; and step 418 corresponds to sub-state 204 d.

Referring now to FIG. 5, operation of the Deferred Interrupt Poll routine will be described in greater detail. Again, processing begins at an Operating System Process Time step 500 . Responsive to a call to the Deferred Interrupt Poll routine, which occurs at regular time intervals, processing proceeds to step 502 , in which all processor interrupts are masked at the CPU 104 . This step can be accomplished using either the PSR.i register 112 or the TPR.mmi register 114 . Step 502 is performed to prevent the state of the Deferred_Interrupt_Pending flag 108 from possibly changing state while it is being examined as described below. In step 504 , a determination is made whether the Deferred_Interrupt_Pending flag 108 is set to TRUE. If so, indicating that there is at least one deferrable interrupt awaiting servicing, execution proceeds to step 506 , in which all deferrable interrupts are enabled, or unmasked. In the preferred embodiment, this is accomplished by clearing the TPR.mic register 114 . Execution then proceeds to step 508 , in which all non-deferred and deferred interrupts are serviced sequentially in order of priority. In the preferred embodiment, this is accomplished by reading the IVR register 116 . In step 510 , the Deferred_Interrupt_Pending flag 108 is set to FALSE, indicating that there are no deferrable interrupts awaiting servicing. In step 512 , the processor interrupts, which were masked in step 502 , are unmasked, as the state of the Deferred_Interrupt_Pending flag 108 is now free to change. Returning to step 504 , if it is determined that the Deferred_Interrupt_Pending flag is not TRUE (i.e., it is FALSE), indicating that there are no deferred interrupts pending, execution proceeds directly to step 512 . Upon completion of step 512 , execution returns to step 500 .

Comparing FIGS. 3 and 5, it will be recognized that steps 502 and 504 correspond to the sub-state 202 a ; steps 506 - 510 correspond to sub-state 202 b ; and step 512 corresponds to substate 202 c.

As previously indicated, although a preferred embodiment of the invention has been described with reference to the IA-64 architecture, it is not necessary that this particular architecture be employed to implement the invention. All that is required is hardware support for several activities, including (1) masking and unmasking, at the processor, all interrupts; (2) masking an unmasking specific interrupt vectors to create a distinction between deferrable and non-deferrable interrupts; (3) detecting of pending interrupts within those classes of deferrable and non-deferrable interrupts; (4) the ability to so service hardware interrupts without the traditional hardware interrupt context save and restore; (5) the ability to defer and culminate interrupt processing in a hardware state outside the processor's formal interrupt state, commonly referred to as “interrupt time;” and (6) supporting the notion of a lightweight interrupt so that full context save is not absolutely required to perform minimal operations, such as setting the Deferred_Interrupt_Pending flag 108 and masking deferrable interrupt vectors. As it were, the IA-64 architecture provides hardware support for each of the above-noted activities; however, it is anticipated that other processor architectures, both current and future, will also support these activities in some form or fashion.

Moreover, it is not necessary that the above-noted activities be supported in exactly the manner described herein. For example, the Deferred_Interrupt_Pending flag 108 could always be set to TRUE and never changed and all deferrable interrupts could always be masked except when being serviced as a result of a call to the deferred interrupt poll routine or as a result of the occurrence of a non-deferred “heavyweight interrupt.” It is also anticipated that a “lightweight interrupt” could be effected using commonly available “scratch registers,” rather than special banked registers designed for that purpose. For “lightweight interrupt” processing, all that is needed is that the processor architecture not “force” heavyweight interrupts and that there be sufficient space for a small amount of work to be performed in the context of an interrupt without requiring context to be saved.

Although illustrative embodiments have been shown and described, a wide range of modification change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.