Memory controller for improved read port selection in a memory mirrored system

1. Field of the Invention

This invention relates generally to improving performance of highly reliable computer systems. In particular, read scheduling in a memory mirroring environment is provided.

2. Description of the Related Art

“Memory mirroring” is the practice of creating and maintaining an exact replica of original data on system memory. Memory mirroring has been used in highly reliable systems for years. It requires that a portion of the user's physical memory be allocated as mirrored memory, thus resulting in reduced system memory capacity. In effect, memory mirroring would reduce memory capacity by half. Although memory mirroring offers increased system reliability, the reduction in system memory capacity may generally result in reduced performance. Current techniques for addressing the performance tradeoff are few and inadequate.

One technique for improving performance on mirrored memory systems has been to split memory access requests between a primary memory port and a mirrored (secondary) memory port. In a mirrored memory system, write requests are executed by both a primary memory port and a secondary memory port. In contrast, read requests need only be executed by either the primary port or the secondary port. When all read requests are dispatched to the primary port, the read bandwidth of the secondary memory port goes unused. In an effort to reclaim the wasted bandwidth, prior mirrored memory computer system have used a “toggle” mode, alternating between the primary and secondary port, to direct each read request to the opposite port to which the previous read request was directed. Although the “toggle” method is a definite improvement over directing all reads to the primary port, there are still significant drawbacks.

Therefore, there is a need for a mirrored memory system that schedules read commands such that both overall memory latency is reduced and read bandwidth on the mirrored memory port is utilized.

The present invention provides for the scheduling of read commands on a mirrored memory system by utilizing information about in-flight and pending memory access requests. A conflict queue is configured to track commands associated with each of a plurality of memory ports on the mirrored memory system. The conflict queue determines a predicted latency on each memory port based on the commands associated with each of the plurality of memory ports. A compare logic unit is coupled to the conflict queue, wherein the compare logic unit compares a predicted latency of a primary memory and a mirrored memory and schedules read commands to the memory port with the lowest predicted latency.

FIG. 1 is a method embodiment of a system that assigns read memory commands between a primary memory and a mirrored memory based on shortest predicted latency.

FIG. 2A illustrates an embodiment of a mirrored memory computer system implementing the process illustrated in FIG. 1 that uses the address translation unit to assign read memory commands to either of a primary or mirrored memory port.

FIG. 2B illustrates a detailed embodiment of the mirrored memory computer system illustrated in FIG. 2A .

FIG. 2C illustrates one method of determining a predicted latency with the conflict queue.

FIG. 3A illustrates a second detailed embodiment of a mirrored memory computer system implementing the process illustrated in FIG. 1 that uses a compare logic unit in a memory port to delete a duplicate read command based on a longest predicted latency.

FIG. 3B illustrates a detailed embodiment of the mirrored memory computer system illustrated in FIG. 3A .

In the following detailed description of embodiments of the invention, reference is made to the accompanying drawings, which form a part hereof, and within which are shown by way of illustration specific embodiments by which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the invention.

FIG. 1 is a flow chart 100 illustrating in general the process of a system that assigns read memory commands between a primary memory and a mirrored memory. In step 102 , the system determines a predicted latency of a read command on the primary memory and the mirrored memory. In step 103 , the system compares the predicted latency of the read command on the primary memory and the mirrored memory to determine if primary memory has a shorter access time. If YES, the system in block 104 schedules the read command to the primary memory 105 . If NO, the system in block 104 schedules the read command to mirrored memory 106 .

FIG. 2A shows a high level block diagram of an embodiment of a computer system 200 implementing the process illustrated in FIG. 1 . One skilled in the art will appreciate that the computer system 200 may be, for example, a mainframe computer, a server, a personal computer system, or a similar system. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations employing multiprocessor design, minicomputers, programmable electronics and the like.

Referring to FIG. 2A , shown is a computer system 200 that comprises a central processing unit (CPU) 230 coupled to a mirrored memory controller 270 having mirrored memory management and error recovery functions (not shown). Some embodiments may include error recovery functions that maintain system reliability while reducing the system bandwidth, such as that described in U.S. Patent Application Publication 2008/0052568 A1, which is herein incorporated by reference in its entirety. The computer system 200 further comprises an input/output (I/O) controller 240 and levels of memory hierarchy including a cache module 250 communicatively coupled to the CPU 230 and mirrored memory controller 270 via a system interconnect 260 .

The mirrored memory controller 270 comprises an address translation unit 201 coupled to a primary memory port, hereinafter memory port 0 202 A, and a mirrored memory port, hereinafter memory port 1 202 B. Those skilled in the art will appreciate alternative implementations that include additional memory for increased redundancy and reliability. The address translation unit 201 takes as input a plurality of requests 222 and outputs, along a read/write command bus 205 A and 205 B, read and write commands. Both memory port 0 202 A and memory port 1 202 B queue read commands and transmit latency information to the address translation unit 201 via 204 A and 204 B as provided in more detail in FIG. 2B . Memory port 0 202 A and memory port 1 202 B communicate via a memory bus 207 A and memory bus 207 B respectively to a first DIMM (dual in-line memory modules) 206 A and a second DIMM 206 B respectively. Each DIMM is comprised of DRAM modules or similar memory modules. Memory port 0 202 A and memory port 1 202 B output data via data output bus 220 A and 220 B respectively.

FIG. 2B illustrates a detailed embodiment of the mirrored memory controller 270 illustrated in FIG. 2A . Memory port 0 202 A takes as input write commands 205 A 1 and read commands 205 A 2 . A write command queue 209 and a read command queue 210 maintain read and write commands to be executed in memory. The write command queue 209 and read command queue 210 each have multiple commands to choose from and are configured to select the oldest non-conflicting command. A conflict may arise when two consecutive commands attempt to access the same section of memory, referred to as the memory bank, resulting in increased latency. Conflicts may also arise when alternating between different ranks, wherein a rank is an independently addressable 64-bit data area of a memory module, because additional clock cycles (clk(s)) are required to avoid data driver contention and also to avoid changes in On Die Termination (ODT).

A MUX (multiplexer) 212 is coupled to the write command queue 209 and read command queue 210 and alternatively selects between each queue. The MUX outputs the alternating read and write commands to the DIMM interface 215 for transmission to the first DIMM 206 A via the memory bus 207 A. A conflict queue 214 A keeps a history of read and write commands transmitted to the first DIMM 206 A. Data retrieved from the first DIMM 206 A are transmitted into the DIMM interface 215 for error correction, then into a read data queue 208 , and subsequently out of memory port 0 202 A. Memory port 1 202 B operates in a similar manner as memory port 0 202 A mentioned above.

In one embodiment, the address translation unit 201 outputs a read command to either of memory port 0 202 A or memory port 1 202 B based on the predicted latency of each memory port as determined by the conflict queue 214 A. Thus, read commands may be consecutively assigned to a primary memory port if, for example, predicted latency on the primary memory port is shorter than predicted latency on a mirrored memory port. Similarly, read commands may be consecutively assigned to the mirrored memory port.

In general, the conflict queue 214 A is configured to predict latency based on various factors including memory bank and memory rank access. In a particular embodiment, the conflict queue 214 A may be configured to include a list of memory access steps and related memory access times. The memory access times may be based on manufacturer specification or independent third-party benchmark as an example. Generally, memory manufacturers specify the number of memory cycles required to perform different steps in memory. For example, a memory chip may have a CAS (Column Access Strobe) of two (2), indicating that two (2) memory cycles pass between the time a column is requested from an active page and the time the data is ready to send across a bus. The conflict queue 214 A may be configured to evaluate the steps required to execute a particular in-flight command (e.g., column access, row access, data access, etc.) based on memory bank location and memory rank location. Generally, bank and rank locations are embedded in the memory address associated with the in-flight command.

The total number of memory cycles associated with executing a command may vary depending on prior in-flight commands. For example, if data is active on a row as a result of a prior memory access then there is no delay in memory cycles between the time a row is activated and the time the data within the row can be requested (referred to as “tRCD” delay or “RAS to CAS” delay). Thus, if memory port 0 202 A has a read command in the read command queue designated to the same row of memory as read command 216 , the read command may be scheduled more efficiently because there will be no tRCD delay.

Similarly, if one of the memory ports 202 A has a read command in the read command queue to the same memory rank as a read command and a different internal bank and no conflicts against commands already in-flight, the read command might be scheduled more efficiently because it can be dispatched back-to-back with no cycle gap.

Thus, the conflict queue 214 A is configured to estimate the steps necessary to execute a command based on current and prior in-flight commands. A conflict queue 214 B is configured to operate in a similar manner. The first conflict queue 214 A and the second conflict queue 214 B output predicted latency to a compare logic unit 213 located in the address translation unit 201 .

FIG. 2C illustrates a flow chart overview 280 of the operation of conflict queue 214 A and 214 B as described in detail above. In step 281 , the conflict queue takes as input in-flight commands and pending read commands. In step 282 , the conflict queue identifies areas of memory that will be impacted by the in-flight and pending commands. In step 283 , the conflict queue evaluates the number of memory access steps involved in accessing each area of memory impacted by the commands. In other words, the memory access steps (e.g., column access, row access, data access, etc.) needed to execute the in-flight and pending commands. In step 284 , the conflict queue calculates the total memory access time to complete all memory access steps for each command. In step 285 , the conflict queue outputs to the compare logic unit the memory access times (i.e., predicted latency) for each impacted area of memory.

Referring back to FIG. 2B , the compare logic unit 213 takes as input a read command 216 and compares the read command 216 to a first memory port predicted latency 204 A and to a second memory port predicted latency 204 B. The compare logic unit 213 outputs the read command 216 to the memory port ( 202 A or 202 B) with the shortest predicted latency, wherein shortest predicted latency depends on the area of memory impacted by read command 216 .

In one embodiment, if the predicted latency for both memory port 0 202 A and memory port 1 202 B are equal, the compare logic unit 213 outputs the read command 216 to the read command queue with the fewest pending read commands. In an alternative embodiment, if predicted latency for both memory port 0 202 A and memory port 1 202 B are equal, the compare logic unit 213 alternates between each memory port.

Predicted latency may be equal if the memory address associated with read command 216 references an area of memory not impacted by in-flight commands on either of the first DIMM 206 A or the second DIMM 206 B. Alternatively, predicted latency may be equal if the memory address associated with read command 216 references an area of memory equally impacted by in-flight commands on both the first DIMM 206 A and the second DIMM 206 B. Alternatively, predicted latency may be equal if neither of the first DIMM 206 A or the second DIMM 206 B have in-flight commands that impact memory access times.

In an alternative embodiment, the first conflict queue 214 A and second conflict queue 214 B keep a history of pending read commands located in the read command queues of memory port 0 202 A and memory port 1 202 B in addition to in-flight read and write commands. Thus, the first conflict queue 214 A and the second conflict queue 214 B output the predicted latency associated with pending read queue commands in addition to the predicted latency associated with in-flight read and write commands.

In another alternative embodiment, the compare logic unit 213 keeps a history of read commands 216 assigned to memory port 0 202 A or memory port 1 202 B and determines potential bank conflicts between the read command 216 input to the compare logic unit 213 and the read commands in the read command queues 210 of memory port 0 202 A and memory port 1 202 B. If a pending read command in one of memory port 0 202 A or memory port 1 202 B impacts an area of memory that conflicts with an area of memory to be accessed by read command 216 , the compare logic unit 213 outputs the read command to the non-conflicting memory port. Thus, the compare logic unit 213 accounts for pending read commands and in-flight read commands.

FIG. 3A shows a high level block diagram of a second embodiment of a computer system 300 implementing the process illustrated in FIG. 1 . In contrast to FIG. 2A , each memory port is interconnected for communication via 304 A and 308 A as explained in more detail below. Otherwise, the computer system 300 functions, in general, in a similar way to that described in FIG. 2A with an input/output controller 340 and levels of memory hierarchy including a cache module 350 communicatively coupled to a CPU 330 and a mirrored memory controller 370 via a system interconnect 360 . As previously described, the computer system may be, for example, a mainframe computer, a server, a personal computer system, or a similar system with a variety of configurations including multiprocessor design, minicomputers and the like.

Referring to FIG. 3A , an address translation unit 301 is coupled to a primary memory port, hereinafter memory port 0 302 A, and a mirrored memory port, hereinafter memory port 1 302 A. The address translation unit 301 takes a plurality of requests 322 as input and outputs read and write commands along a read/write command bus 305 A and 305 B. Both memory port 0 302 A and memory port 1 302 B queue read commands and transmit latency information between ports as provided in more detail in FIG. 3B . Memory port 0 302 A and memory port 1 302 B communicate via a memory bus 307 A and memory bus 307 B respectively to a first DIMM (dual in-line memory modules) 306 A and a second DIMM 306 B respectively. Each DIMM is comprised of DRAM modules or similar memory modules. Memory port 0 302 A and memory port 1 302 B output data via data output bus 320 A and 320 B respectively.

FIG. 3B illustrates a detailed embodiment of the mirrored memory controller 370 illustrated in FIG. 3A in which the address translation unit 301 transmits duplicate read commands 305 to BOTH memory port 0 302 A and memory port 1 302 B, unlike mirrored memory computer systems that toggle read commands between a primary memory and a mirrored memory. The first conflict queue 316 A and the second conflict queue 316 B keep a history of pending read commands located in the read command queues of memory port 0 302 A and memory port 1 302 B in addition to in-flight read and write commands. The first conflict queue 316 A and the second conflict queue 316 B output to a selected memory port the predicted latency associated with pending read queue commands in addition to the predicted latency associated with in-flight read and write commands. While FIG. 3B depicts the selected memory port as memory port 1 302 B, those of ordinary skill in the art will appreciate that logic for scheduling read commands can be located in either of the primary (memory port 0 302 A) or mirrored memory ports (memory port 1 302 B). In an alternative embodiment, the compare logic unit 315 is located in between the primary memory port (memory port 0 302 A) and the mirrored memory port (memory port 1 302 B).

The compare logic unit 315 identifies the memory bank and rank associated with a read command 317 , transmitted from the address translation unit to both read command queues, and compares a first memory port predicted latency 304 A to a second memory port predicted latency 304 B based on pending read queue commands and in-flight commands associated with each memory port. The compare logic unit 315 selects a memory port with a shortest predicted latency and outputs a read command delete request to a non-selected memory port via 308 A or via 308 B. If the predicted latency for both memory port 0 302 A and memory port 1 302 B are equal, the compare logic unit 315 selects a read command queue with the fewest pending read commands and sends a delete read command request to non-selected memory ports. In an alternative embodiment, if bank access times for both memory port 0 302 A and memory port 1 302 B are equal, the compare logic unit 315 alternates between each memory port.

Those skilled in the art will appreciate a more sophisticated memory port system with command arbitration logic that chooses not only between read and write commands but also other DRAM commands such as scrub, refresh, ZQ calibration, mirror failover read, retry read, and maintenance commands. In such a system, the compare logic unit assigns read commands to the memory port with the shortest memory bank access time as described in the embodiments above.