Method and apparatus for reordering memory accesses to reduce power consumption in computer systems

1. Field of the Invention

This invention relates to memory subsystems and, more particularly, to memory controller design.

2. Description of the Related Art

Computer systems include one or more processors that execute instructions from a memory medium. The memory medium is typically one or more levels of cache and a main memory. The main memory may be made up of one or more memory modules, e.g., Dual In-Line Memory Modules (DIMMs) or Fully-Buffered DIMMs (FB-DIMMs). A memory subsystem of a computer system typically includes a memory controller, which is connected to the main memory. The main function of the memory controller is to schedule and coordinate the execution of read and write requests (i.e., memory accesses) to main memory. The memory controller may include at least one queue to schedule the read and write requests. In most cases, the memory controller includes a queue for the read requests and a separate queue for the write requests.

Memory subsystems are typically configured to place memory modules that are not being used into a low power mode to save power. This is usually the case because in typical systems a large portion of the power dissipation is in the main memory. In most cases, memory accesses are typically executed in the order that they are received at the appropriate queue of the memory controller. The memory controller may receive several read commands in a row and each command may access a different memory module. For example, if the memory subsystem includes eight memory modules, the memory controller may receive eight read commands each accessing a different memory module. In this situation, since typical memory controllers execute the read commands in a row, the powers savings would be minimal if some of the memory modules entered into a low power mode because all the memory modules would be accessed within a short period of time. Since read and write requests are received randomly and are executed in the order they are received, the memory modules may be constantly switching from a normal power mode to a low power mode, which may result in minimal or no power savings in most situations. In some cases, the constant switching from the two different power modes may actually increase the power consumption in the system.

Various embodiments of a reordering command queue are disclosed. The reordering command queue may be comprised in a memory controller of a computer system. The computer system may also include one or more memory modules (i.e., main memory), which may be coupled to the memory controller. The reordering command queue may reduce the power that is typically used up in a computer system when performing accesses to the main memory by improving the scheduling of memory accesses with a pattern that is optimized for power and which has no (or negligible) impacting on performance.

The reordering command queue may include a plurality of storage locations, a plurality of comparators, and a plurality of reordering logic. The storage locations may store commands received at the reordering command queue in a particular order. Each of received commands may include an address corresponding to at least one of the memory modules in the system. For example, the address corresponding to each of the commands may include a memory module number and a rank number.

In one embodiment, each of the comparators of the reordering command queue may be connected between adjacent storage locations and may perform compare operations. Also, the reordering logic may be connected to each of the storage locations and may perform reorder operations. During a compare operation, each of a subset of the comparators may compare the address corresponding to the command stored in a current storage location to the address corresponding to the command stored in an adjacent storage location. In one embodiment, the comparators may compare the rank number and the memory module number of each address. The results derived from the compare operation may indicate whether the corresponding commands are in a desired order. The desired order for the commands in the reordering command queue may be from the lowest to the highest rank number and then from the lowest to the highest memory module number, i.e., the commands having the lowest rank number and then having the lowest memory module number are stored in the storage locations closer to the output of the reordering command queue. In this way, the ranks that are located physically closer to the memory controller are accessed first and all the accesses to a particular rank are grouped together to conserve power.

In response to one or more of the commands not being in the desired order, which may be determined from the results of the compare operation, the reordering logic may perform a reorder operation. During the reorder operation, each of the one or more commands that are not in the desired order may be reordered from a current storage location to an adjacent storage location. The comparators may continually perform compare operations to provide results to the reordering logic, and in response the reordering logic may reorder one or more of the commands from a current storage location to an adjacent storage location depending upon the results derived from the compare operations to improve the scheduling of memory accesses.

In one embodiment, the memory controller may include a power control unit to manage the power mode of each of the memory modules depending upon an order of the commands within the queue after one or more reorder operations to reduce power consumption in the computer system. The power control unit may change the power mode of each of the memory modules from a normal power mode to a low power mode or vice versa depending upon an order of access of the memory modules indicated by the order of the commands after one or more reorder operations.

In one embodiment, after a programmable amount of time, the reordering command queue may perform a drain operation to prevent any commands from being stuck in the reordering command queue. The drain operation may be performed with respect to one or more original commands that were stored in the reordering command queue at a time when the drain operation is started. During the drain operation, the comparators and the reordering logic may stop the compare and reorder operations with respect to the one or more original commands. If new commands are received at the reordering command queue during the drain operation, the comparators and the reordering logic may perform the compare and reorder operations with respect to the new commands even while the drain operation is being performed with respect to the original commands.

FIG. 1 is a block diagram of one embodiment of a memory subsystem;

FIG. 2 is a block diagram of one embodiment of a reordering command queue;

FIG. 3 is a flow diagram illustrating a method for reordering commands in the reordering command queue, according to one embodiment;

FIG. 4 is a block diagram of one embodiment of a memory subsystem;

FIG. 5A is a drawing illustrating storage locations of a reordering command queue and the implementation of a first compare operation with respect to the commands stored in the storage locations, according to one embodiment;

FIG. 5B is a drawing illustrating storage locations of a reordering command queue and the implementation of a second compare operation with respect to the commands stored in the storage locations, according to one embodiment;

FIG. 5C is a drawing illustrating storage locations of a reordering command queue and the order of the commands after the first and second compare operations and the corresponding first and second reordering operations, according to one embodiment; and

FIG. 6 is a block diagram of one embodiment of a computer system.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present invention as defined by the appended claims. Note, the headings are for organizational purposes only and are not meant to be used to limit or interpret the description or claims. Furthermore, note that the word “may” is used throughout this application in a permissive sense (i.e., having the potential to, being able to), not a mandatory sense (i.e., must). The term “include”, and derivations thereof, mean “including, but not limited to”. The term “coupled” means “directly or indirectly connected”.

Memory Subsystem

Turning now to FIG. 1 , a block diagram of one embodiment of a memory subsystem 100 is shown. The memory subsystem 100 may be comprised in a computer system, e.g., a workstation, a personal computer, and a portable computer system, among others, as will be further described below with reference to FIG. 6 . The memory subsystem 100 may include a memory controller 150 and one or more memory modules 125 . For example, in the illustrated embodiments the memory subsystem 100 includes N memory modules, i.e., memory modules 125 A- 125 X.

The memory controller 150 typically performs the scheduling of the read and writes to the memory modules 125 . In the illustrated embodiment, the memory controller 150 is coupled to the memory modules 125 via two links. The link 110 may be referred to as a “downstream” or “southbound” link, since the first memory module (e.g., memory module 125 A) receives data and/or commands from the memory controller 150 via the link 110 . The link 120 may be referred to as an “upstream” or “northbound” link, since data is conveyed from the memory module 125 A to the memory controller 150 via the link 120 . The remainder of the memory modules 125 (i.e., memory modules 125 B- 125 X) in the embodiment shown are coupled to each other through a plurality of upstream and downstream links as illustrated. In general, a link may be referred to as an upstream link if information conveyed through the link is flowing towards the memory controller 150 , while a link may be referred to as a downstream link if information conveyed through the link is flowing away from the memory controller 150 . In the illustrated embodiment, the memory modules 125 are coupled to each other in what is commonly referred to as a “daisy-chain” arrangement. It is noted however that in other embodiments the memory modules 125 may be coupled to each other in other arrangements.

The memory modules 125 may be configured as Dual In-Line Memory Modules (DIMMs) or Fully-Buffered DIMMs (FB-DIMMs) and may collectively form the main memory of the computer system. It is noted however that in other embodiments the memory modules 125 may have other configurations. The memory modules 125 may include memory devices in the Dynamic Random Access Memory (DRAM) family of devices. For example, a plurality of banks of Synchronous DRAM (SDRAM), Double Data Rate (DDR) SDRAM, DRR2 SDRAM, or Rambus DRAM (RDRAM) devices may be include in each of the memory modules 125 . It is noted however that in other embodiments the memory modules 125 may include other types of memory devices.

The memory controller 150 may include a reordering command queue (e.g., the reordering command queue shown below in FIG. 2 ). The reordering command queue may reduce the power that is typically used up in a computer system when performing accesses to the main memory by improving the scheduling of memory accesses with a pattern that is optimized for power and which has no (or negligible) impacting on performance. The reordering command queue may improve the scheduling of memory accesses by reordering the commands within the reordering command queue based on rank numbers and memory module numbers, as will be further described below with reference to FIGS. 2-5 .

It should be noted that the components described with reference to FIG. 1 are meant to be exemplary only, and are not intended to limit the invention to any specific set of components or configurations. For example, in various embodiments, one or more of the components described may be omitted, combined, modified, or additional components included, as desired. For instance, in one embodiment, the arrangement of the memory modules 125 may vary.

Reordering Command Queue

FIG. 2 is a block diagram of one embodiment of a reordering command queue 250 . The reordering command queue 250 may be comprised in an integrated circuit (IC), for example, a digital IC. In one embodiment, the IC may be an application-specific IC (ASIC). The reordering command queue 250 may be comprised in a memory controller of a memory subsystem, e.g., the memory controller 150 of the memory subsystem 100 of FIG. 1 , or the reordering command queue 250 may be coupled to other components which collectively function as the memory controller of a system. It is also noted that in some embodiments the reordering command queue 250 may be included within other subsystems of a computer system. Furthermore, it is noted that in other embodiments the reordering command queue 250 may be any type of storage mechanism that includes the reordering functionality described below.

In the illustrated embodiment shown in FIG. 2 , the reordering command queue 250 (e.g., the memory controller 150 of FIG. 1 ) includes an input line 205 , a plurality of storage locations 210 A- 210 C, a plurality of comparators 220 A- 220 C, reordering logic 230 A- 230 C, and control unit 240 . As illustrated, the reordering command queue 250 may include any number (i.e., N) of storage locations and the corresponding comparators and reordering logic. For example, in one embodiment, the reordering command queue 250 may include eight storage locations, seven comparators, and eight reordering logic. In the illustrated embodiment, the output terminal of the reordering logic 230 B is coupled to the input terminal of the storage location 210 B. Also, the input line 205 , the output terminal of the storage location 210 C (i.e., one of the adjacent storage locations), the output terminal of the storage location 210 B, and the output terminal of the storage location 210 A (i.e., the other adjacent storage location) are connected to the input terminals of the reordering logic 230 B. The input and output terminals of the reordering logic 230 A and 230 C are similarly connected. It is noted that one of the inputs to the reordering logic 230 A may be connected to the output terminal of a storage location (not shown) that precedes the storage location 210 A. Also, it is noted that one of the inputs to the reordering logic 230 C may be connected to the output terminal of a storage location (not shown) that follows the storage location 210 C. The configuration shown on FIG. 2 may continue for the rest of the reordering logic, storage locations, and comparators included within embodiments of the reordering command queue 250 .

Additionally, in the illustrated embodiment, each of the plurality of comparators 220 are coupled between adjacent storage locations 210 . For example, the comparator 220 B is coupled between storage location 210 B and storage location 210 C, and the comparator 220 A is coupled between the storage location 210 A and the storage location 210 B. Each of the comparators 210 is also coupled to the control unit 240 . Furthermore, the control unit 240 is coupled to each of the reordering logic 230 .

It should be noted that the components described with reference to FIG. 2 are meant to be exemplary only, and are not intended to limit the invention to any specific set of components or configurations. For example, in various embodiments, one or more of the components described may be omitted, combined, modified, or additional components included, as desired. For instance, in some embodiments, each of the reordering logic 230 may include any number of input terminals, e.g., the reordering logic 230 may include three input terminals or five input terminals. Also, in other embodiments, some of the components, e.g., the control unit 240 , may be coupled to the reordering command queue 240 but may be physically located outside the IC including the queue 250 .

The reordering command queue 250 may reduce the power that is typically used up in a computer system when performing accesses to the main memory without (or negligible) impacting performance. In previous design, memory accesses are typically scheduled such that different DIMMs are accessed in successive cycles or DIMMs are accessed randomly; therefore, the DIMMs typically do not switch to a low power mode or they remain in a low power mode for an insignificant amount of time. For example, in some previous designs, commands to perform memory accesses may be received in a queue and may be executed in the order they were received. Unlike previous designs, the reordering command queue 250 may reduce power consumption by improving the scheduling of memory accesses with a pattern that is optimized for power and which maintains the same level of bandwidth (performance).

In one embodiment, the reordering command queue 250 may receive commands to perform memory accesses. The reordering command queue 250 may improve the scheduling of memory accesses by reordering the commands within the queue 250 based on rank numbers and memory module (e.g., DIMM) numbers, as will be described further below. By reordering the commands within the queue 250 , the memory controller (e.g., the memory controller 150 of FIG. 1 ) or another controlling device may predict which memory modules (e.g., FB-DIMMs) are going to be accessed next and will wake up only these memory modules, i.e., place (or maintain) the memory modules in a normal power mode. The remaining memory modules may remain (or may be placed) in a low power mode to conserve power.

FIG. 3 is a flow diagram illustrating a method for reordering commands in the reordering command queue 250 , according to one embodiment. It should be noted that in various embodiments, some of the steps shown may be performed concurrently, in a different order than shown, or omitted. Additional steps may also be performed as desired.

Referring collectively to FIG. 3 and FIG. 2 , the reordering command queue 250 may receive a plurality of commands to perform memory accesses (block 305 ), e.g., from the memory controller (e.g., memory controller 150 of FIG. 1 ). As described above, in some embodiments the reordering command queue 250 may receive the commands from any type of device within any type of subsystem. The commands may be stored within one or more of the storage locations 230 in a particular order. Each of the commands may include one or more bits specifying an address corresponding to one or more of the memory modules (e.g., memory modules 125 of FIG. 1 ). For example, the command may include one or more bits for each of the memory module number, the rank number, the internal bank (ibank) number, and the row and column number. It is noted however that in other embodiments each of the commands may include one or more bits specifying an address corresponding to any type of device that the system is trying to access. In one embodiment, the commands may include bits for storing miscellaneous information.

The comparators 220 of the reordering command queue 250 may then perform compare operations, which may compare the address corresponding to the command stored in each of the storage locations to the address corresponding to the command stored in an adjacent storage location to determine whether the commands are in a desired order. In one embodiment, as shown in the illustrated embodiment of FIG. 4 , from the direction of memory controller 150 , the memory modules 125 A- 125 X may be arranged from the lowest to the highest memory module number and, within each memory module 125 , from the lowest to the highest rank number. For example, memory modules 125 A- 125 X may be designated as memory module # 0 —memory module #N, respectively, and each of the memory modules 125 may include a rank # 0 and a rank # 1 . The number designations may be based on the physical locations from the memory controller 150 , i.e., the lower numbers may be given to the memory modules and ranks that are physically closer to the memory controller 150 . In this embodiment, the desired order for the commands in the reordering command queue 250 is based on the corresponding memory module number and the rank number. It is noted however that in other embodiments the memory modules may be connected in other arrangements and/or other kinds of number designations may be used. For example, larger numbers may be given to the memory modules and/or ranks that are physically closer to the memory controller 150 . It is also noted that in some embodiments each of the memory modules 125 may include any number of ranks, e.g., three or more ranks.

As shown in FIG. 4 , in one embodiment the memory controller 150 may access the memory modules 125 (e.g., FB-DIMMs) in a round robin fashion. For example, the memory controller 150 may access the rank # 0 of each of the memory modules in successive order starting with memory module # 0 , and then the memory controller 150 may access the rank # 1 of each of the memory modules in successive order starting with memory module # 0 . In this embodiment, the desired order for the commands in the reordering command queue 250 is from the lowest to the highest rank number and then from the lowest to the highest memory module number, i.e., the commands having the lowest rank number and then having the lowest memory module number are stored in the storage locations closer to the output of the reordering command queue 250 . In this way, the ranks that are located physically closer to the memory controller 150 are accessed first and all the accesses to a particular rank are grouped together to conserve power, as will be described further below. It is noted however that in other embodiments the memory controller 150 may access the memory modules 125 in different ways besides a round robin fashion.

Referring collectively to FIGS. 2-4 , after receiving the commands, each of a first subset of the comparators 220 may then perform a first compare operation, which may compare the rank number and the memory module number of the address stored in each of one or more current storage locations to the rank number and the memory module number of the address stored in a first adjacent storage location to determine whether the commands are in a desired order (block 310 ). Then, the reordering logic 230 may perform a first reorder operation depending on the results from the first compare operation, i.e., whether the commands where in the desired order (block 320 ). If one or more commands are not in the desired order, the corresponding reordering logic 230 may perform a first reorder operation on these commands. It is noted that in one embodiment the desired order for the commands may be from the lowest to the highest rank number and then from the lowest to the highest memory module number, i.e., the commands having the lowest rank number and then having the lowest memory module number are stored in the storage locations closer to the output of the reordering command queue 250 . In the first reorder operation, the corresponding reordering logic 230 reorders the one or more of the commands from a current storage location to the first adjacent storage location (block 330 ). For example, if the first compare operation performed by comparator 220 A determines that the commands stored in storage locations 210 A and 210 B (shown in FIG. 2 ) are not in the desired order, the reordering logic 230 A and 230 B may perform a first reorder operation to swap the order of the commands stored in storage locations 210 A and 210 B. It is noted that one or more first reorder operations may be performed depending on the results from the one or more first compare operations. It is also noted that in some embodiments the desired order for the commands may be any order determined to save power and maintain (or improve) performance, for example, the desired order may be from the lowest to the highest memory module number.

If the results from the first compare operation indicate that all the commands are in the desired order, then a first reorder operation may not have to be performed. The reordering command queue 250 may then receive additional commands (block 325 ). Also, after performing the first reorder operation, the reordering command queue 250 may receive additional commands (block 335 ). It is noted that the reordering command queue 250 may receive additional commands at any time during the process, e.g., before or after a compare operation and/or before or after a reorder operation. It is also noted that one or more commands may be executed before the compare operation and/or reordering operation is fully completed.

Then, each of a second subset of the comparators 220 may perform a second compare operation, which may compare the rank number and the memory module number of the address stored in each of one or more current storage locations to the rank number and the memory module number of the address stored in a second adjacent storage location to determine whether the commands are in a desired order (block 340 ). Depending on the results from the second compare operation, i.e., whether the commands where in the desired order (block 350 ), the reordering logic 230 may then perform a second reorder operation. If one or more commands are not in the desired order (e.g., from the lowest to the highest rank number and then from the lowest to the highest memory module number), the corresponding reordering logic 230 may perform a second reorder operation on these commands. In the second reorder operation, the corresponding reordering logic 230 reorders the one or more of the commands from a current storage location to the second adjacent storage location (block 360 ). For example, if the second compare operation performed by comparator 220 B determines that the commands stored in storage locations 210 C and 210 B (shown in FIG. 2 ) are not in the desired order, the reordering logic 230 C and 230 B may perform a second reorder operation to swap the order of the commands stored in storage locations 210 C and 210 B. It is noted that one or more second reorder operations may be performed depending on the results from the one or more second compare operations. After block 350 or block 360 , the operation may restart and new commands may be received at the reordering command queue 250 (block 305 ). The comparators 220 may continually perform one compare operation after another to provide results to the reordering logic 230 and reorder the commands, especially since at any given point in time some of the commands in the queue 250 may be executed and new commands may be stored in the queue 250 .

In one embodiment, the control unit 240 may receive the results from the comparators 220 after the compare operations and may control the reordering logic 230 accordingly. For example, after the first compare operation, the control unit 240 may receive the results indicating that the commands in storage locations 210 A and 210 B are not in the desired order. In response to receive the results, the control unit 240 may send control signals to the reordering logic 230 A and 230 B to select the appropriate inputs to perform the first reorder operation. In the embodiment illustrated in FIG. 2 , the reordering logic 230 A may select the fourth input from the top to select the output from the storage location 210 B and store this command into storage location 210 A. Also, the reordering logic 230 B may select the second input from the top to select the output from the storage location 210 A and to store this command into storage location 210 B. The control unit 240 may similarly receive results from other compare operations and control the implementation of other reorder operations. It is noted that in other embodiments the functionality of the control unit 240 may be performed by other components outside of the reordering command queue 250 .

FIGS. 5A-5C illustrate one example of compare and reordering operations that may be performed in one embodiment of the reordering command queue 250 . In this example, as shown in FIG. 5A , the reordering command queue 250 may receive eight commands and may store the received commands in eight of its storage locations 510 A- 510 H, which may correspond to the storage locations 210 of FIG. 2 . In one embodiment, the storage location 510 H may be the storage location that is physically the closest to the output of the queue 250 . After storing the commands, each of a first subset of the comparators (e.g., comparators 220 of FIG. 2 ) may perform a first compare operation, which may compare the rank number and the memory module number of the address stored in each of the storage locations 510 to the rank number and the memory module number of the address stored in a first adjacent storage location to determine whether the commands are in a desired order. As shown in FIG. 5A , the first adjacent storage location of 510 A is 510 B, of 510 B is 510 A, of 510 C is 510 D, and so on. The first compare operation between storage locations 510 A and 510 B and the first compare operation between storage locations 510 E and 510 F do not result in a first reorder operation for those storage locations because these commands are in the desired order. The first compare operations between storage locations 510 C and 510 D and the first compare operation between storage locations 510 G and 510 H do result in a first reorder operation because these commands are not in the desired order. Note that the commands in storage locations 510 C and 510 D are swapped even though the command in 510 C corresponds to a higher memory module number than 510 D. The reason is because the desired order (as described above) in one embodiment may be arranging the commands from the lowest to the highest rank number and then from the lowest to the highest memory module number. In this example, storage location 510 C has a lower rank number than 510 D, therefore the commands switch locations during the first reorder operation.

FIG. 5B illustrates the location of the commands after the first reorder operation described in FIG. 5A . After the first reorder operation, each of a second subset of the comparators (e.g., comparators 220 of FIG. 2 ) may perform a second compare operation, which may compare the rank number and the memory module number of the address stored in each of one or more of the storage locations 510 to the rank number and the memory module number of the address stored in a second adjacent storage location to determine whether the commands are in the desired order. Note that in this embodiment storage locations 510 A and 510 H may be the storage locations at the ends of the queue 250 and therefore may not participate in the second compare operation (and a second reorder operation). It is noted however that in other embodiments the storage locations at the end of the queue 250 (e.g., 510 A and 510 H) may not participate in the first compare operation but may participate in the second compare operation. As shown in FIG. 5B , the second adjacent storage location of 510 B is 510 C, of 510 C is 510 B, of 510 D is 510 E, and so on.

FIG. 5C illustrates the location of the commands after the second reorder operation described in FIG. 5B . Note that after comparing the location of the commands shown in FIG. 5A and FIG. 5C several of the commands have swapped location during the first and second reorder operations in the examples described above. After the operations, the commands are now close to the desired order; however, the commands at storage locations 510 C and 510 D may still need to be swapped. The process starts again and a new first compare operation is performed. It is noted that execution of a command and/or receiving of new commands is not accounted for in the example described above. However it is noted that some of the commands illustrated in FIGS. 5A-5C (e.g., the commands at storage locations 510 G and 510 H) may be executed at any point during the process, and new commands may be received by the queue 250 also at any point during the process. In some cases, even though the commands stored in the queue 250 may be executed faster than the command and reordering operations may be performed and therefore the desired order may not be achieved, an improved order that is close to the desired order may be achieved.

By rearranging the commands stored within the reordering command queue 250 to the desired order or to an improved order that is close to the desired order, the memory accesses corresponding to the stored commands may be predictable since several memory accesses to a particular memory module or to a few memory modules are performed in a row. Since memory accesses may be predictable, several memory modules that will not be accessed for a particular number of cycles may be put in low power mode. Therefore, more memory modules may be placed in a lower power mode at any particular time, which may lead to improved power consumption. In the embodiment illustrated in FIG. 4 , the memory controller 150 includes a power control unit 450 to manage the power mode of each of the memory modules depending upon an order of the commands within the queue 250 after one or more reorder operations. The power control unit 450 may change the power mode of each of the memory modules from a normal power mode to a low power mode or vice versa depending upon an order of access of the memory modules indicated by the order of the commands after one or more reorder operations. For example, with reference to FIG. 4 , if the order of the commands indicate that the next several memory accesses are to rank # 0 and to memory modules # 0 - 2 , then memory modules # 3 -N (or memory modules # 4 -N) may be placed in a low power mode. Also, since the memory accesses may be predictable, the system may also have improved wake-up/sleep scheduling. For example, when it is determined that memory modules # 3 - 4 will be accessed in the next few cycles, only memory modules # 3 - 4 may be switched to a normal power mode and the remaining modules # 5 -N may remain in a low power mode. Therefore, the reordering of commands may lead to improved power consumption without sacrificing performance. In one embodiment, the power control unit 450 may determine the order of the commands from the compare operations and/or the reordering operations.

It is noted that in other embodiments the compare operations may compare one or more of a memory module number, a rank number, an internal bank number, and a row and column number corresponding to each of the commands in the storage locations (e.g., storage locations 210 ). In some embodiments the reordering command queue 250 may store and reorder read commands. It is noted however that in other embodiments the reordering command queue 250 may store and reorder write commands or both read and write commands. In one embodiment, if the reordering command queue 250 is configured to store and reorder read commands, the execution of write operations may be dependent on which memory modules are turned on (e.g., in a normal power mode). For example, if during a period of time only memory modules # 0 - 2 are turned on due to the reordering of the read commands, only the write operations corresponding to the memory modules # 0 - 2 may be performed during that period of time.

As shown in the illustrated embodiment of FIG. 2 , the control unit 240 may also control which storage location 210 receives a new command from the input line 205 of the reordering command queue 250 . For example, if storage location 210 C is free, the control unit 240 may send a control signal to the reordering logic 230 C to select the third input from the top and store the new command. Additionally, when a comparator (e.g., comparator 220 A) determines that the adjacent commands (commands stored within storage locations 230 A and 230 B) are in the desired order, the control unit 240 may notify the reordering logic (e.g., reordering logic 230 A and reordering logic 230 B) that the commands are in the desired order or the control unit 240 may do nothing. In one embodiment, if the control unit 240 sends a notification (e.g., a control signal), both the reordering logic 230 A and 230 B may select the first input from the top to keep the current command within the corresponding storage location. In another embodiment, if the control unit 240 sends a notification (e.g., a control signal), both the reordering logic 230 A and 230 B may do nothing and keep the current command within the corresponding storage location. Furthermore, the control unit 240 may be coupled to the power control unit 450 to send results from the compare and/or reordering operations so that the power control unit 450 may determine the order of the commands within the queue 250 to place certain memory modules in a normal power mode and other memory modules in a low power mode.

Due to the compare and reordering operations, the components in the reordering command queue 250 may prevent some of the commands from being executed, i.e., to perform a memory access. In other words, some commands may get stuck in the queue 250 because new commands continue to have higher priority based on the compare and reorder operations. For example, in the illustrated embodiment of FIG. 5C , the commands in storage locations 510 A and 510 B may get stuck in their respective storage locations for a period of time. To prevent commands from being stuck for a significant amount of time, the reordering command queue 250 may drain all the commands stored in the storage locations 210 periodically.

During a drain operation, the compare and reordering operations are stopped for the commands stored in the queue 250 at the time the drain operation is started (i.e., the original commands) and these commands are executed. Therefore, the drain operation may execute all the commands that are stuck in the queue 250 . As these original commands are executed, new commands may be received and stored in the queue 250 . The compare and reordering operations may be performed with respect to the new commands, even while the drain operation is being performed for the original commands. For example, there may be eight commands stored in the queue 250 when the drain operation is started. In this example, if five commands are drained and four new commands are received, the drain operation may be performed only for the three remaining original commands and the compare and reorder operations may be performed for the four new commands. In one embodiment, the original command are tagged to distinguish the original commands from any new commands that may be received and stored in the queue 250 . It is noted however that in other embodiments the compare and reordering operations are stopped for all of the commands (whether original or new commands) until the drain operation is completed. The drain operation may be performed periodically at regular intervals. In one embodiment, the interval for performing the drain operation (i.e., the drain window) is programmable. It is noted however that in other embodiments the drain operation may be performed at any time, i.e., the intervals for performing the drain operation may not be regular. It is also noted that in some embodiments the queue 250 may detect whether one or more commands are stuck and in response perform drain operations.

As described above, in one embodiment the memory modules illustrated in FIG. 4 (or in FIG. 1 ) may be Fully-Buffered DIMMs (FB-DIMMs). Memory controllers that are used with FB-DIMMs (or other types of memory modules) may be configured to implement either fixed latency or variable latency. A fixed latency implementation is when the memory controller functions with the assumption that all memory modules have the same latency, even though the latency may vary between memory modules depending at least on the physical distance from the memory controller. A variable latency implementation is when the memory controller functions with the assumption that a memory module that is physically closer to the memory controller has less latency than a memory controller that is physically farther way.

The scheduling of memory accesses is much more complicated in a variable latency implementation because the memory controller has to account for the various latencies to avoid conflicts on the data, therefore the design of these memory controllers are typically very complex. Since the compare and reordering operations performed by components of the reordering command queue 250 make memory accesses predictable, the scheduling of memory accesses may be easier in a variable latency implementation when memory controllers that include the reordering command queue 250 are used. In other words, the reordering command queue 250 may facilitate the implementation of variable latency in memory subsystems, e.g., memory systems including FB-DIMMs. The implementation of variable latency may be further facilitated by performing memory accesses in a round robin fashion as described with reference to FIG. 4 . Therefore, by using the reordering command queue 250 (as described above with reference to FIGS. 1-5 ), the design of the memory controller may not need to be as complex to implement a variable latency scheme.

Computer System

Turning now to FIG. 6 , one embodiment of a system which incorporates a memory subsystem as described above is depicted. In the illustrated embodiment, system 600 includes a processor 610 coupled to the memory controller 150 , a boot device 630 , a peripheral storage device 620 , and a network 640 . The network 640 is in turn coupled to another computer system 650 .

In some embodiments, system 600 may include more than one instance of the devices shown, such as more than one processor 610 , for example. In various embodiments, system 600 may be configured as a rack-mountable server system, a standalone system, or in any other suitable form factor. In some embodiments, system 600 may be configured as a client system rather than a server system. While the memory controller 150 is depicted as a separate device, in other embodiments the memory controller 150 may be integrated with, or part of, the processor 610 . In one embodiment, as described above, the memory controller 150 may include a reordering command queue (e.g., queue 250 of FIG. 2 ). System memory 125 may comprises memory modules (e.g., FB-DIMMs) as described above.

Peripheral storage device 620 , in various embodiments, may include support for magnetic, optical, or solid-state storage media such as hard drives, optical disks, nonvolatile RAM devices, etc. In some embodiments, peripheral storage device 620 may include more complex storage devices such as disk arrays or storage area networks (SANs), which may be coupled to processor 610 via a standard Small Computer System Interface (SCSI), a Fibre Channel interface, an IEEE 1394 interface, or another suitable interface. Additionally, it is contemplated that in other embodiments, any other suitable peripheral devices may be coupled to processor 610 , such as multimedia devices, graphics/display devices, standard input/output devices, etc.

Boot device 630 may include a device such as an FPGA or ASIC configured to coordinate initialization and boot of processor 610 , such as from a power-on reset state. Additionally, in some embodiments boot device 630 may include a secondary computer system configured to allow access to administrative functions such as debug or test modes of processor 610 .

Network 640 may include any suitable devices, media and/or protocol for interconnecting computer systems, such as wired or wireless Ethernet, for example. In various embodiments, network 640 may include local area networks (LANs), wide area networks (WANs), telecommunication networks, or other suitable types of networks. In some embodiments, computer system 650 may be similar to or identical in configuration to illustrated system 600 , whereas in other embodiments, computer system 650 may be substantially differently configured. For example, computer system 650 may be a server system, a processor-based client system, a stateless “thin” client system, a mobile device, etc.

Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.