Apparatus with a memory controller configured to control access to randomly accessible non-volatile memory

This application is a continuation of U.S. patent application Ser. No. 14/475,398, filed Sep. 2, 2014, which application is a continuation of U.S. patent application Ser. No. 13/747,455, filed Jan. 22, 2013, which application is a continuation of U.S. patent application Ser. No. 12/832,921, filed Jul. 8, 2010, which application is a divisional of U.S. patent application Ser. No. 11/848,083, filed Aug. 30, 2007, which application claims the benefit of U.S. Provisional Patent Application No. 60/827,421, filed on Sep. 28, 2006 and U. S. Provisional Patent Application No. 60/862,597 filed on Oct. 23, 2006. Additionally, U.S. patent application Ser. No. 13/747,455, filed Jan. 22, 2013, is a divisional of U.S. patent application Ser. No. 12/831,233, filed Jul. 6, 2010, which application is a divisional of U.S. patent application Ser. No. 11/847,986, filed Aug. 30, 2007, which application claims the benefit of U.S. Provisional Patent Application No. 60/827,421, filed on Sep. 28, 2006 and U.S. Provisional Patent Application No. 60/862,597 filed on Oct. 23, 2006. Each of the aforementioned applications are herein incorporated by reference.

This document generally relates to memory controllers and memory modules.

A computing system may include dynamic random access memory (DRAM) integrated circuits (ICs) as part of its main memory. DRAM ICs retain data information by storing a certain amount of charge on a capacitor in each memory cell to store a logical one or alternatively, a logical zero. Over time, and because of read operations, the stored charge on the capacitor dissipates, in a process often referred to as leaking off. To preserve the stored charge on a DRAM capacitor, and thus maintain the ability of the DRAM to maintain its memory contents, the stored charge in the memory cell may be increased through refresh cycles, which sometimes are performed periodically.

FIG. 1 is a functional block diagram of a computer system with only DRAM DIMMS wherein the memory controllers are physically separate from the processors.

FIG. 1A is a functional block diagram of an alternate configuration of a memory controller through the use of subsidiary memory controller chips.

FIG. 2A is a functional block diagram of a computer system with integrated memory controllers collocated within the processors.

FIG. 2B is a functional block diagram of another computer system with integrated memory controllers collocated within the processors.

FIG. 3A is a functional block diagram of a card with a memory controller to control non-DRAM type of memory DIMMS capable of plugging into an expansion slot.

FIG. 3B is a functional block diagram of a card with a memory controller and non-DRAM type of memory mounted thereto.

FIG. 4 is a functional block diagram of a non-DRAM type of memory module.

FIG. 5 is a functional block diagram of an internet server coupled to the internet.

FIGS. 6A-6B are functional block diagrams of a support integrated circuit.

FIG. 7A illustrates a flow chart of a method for upgrading a computing system.

FIG. 7B illustrates a flow chart of a method for assembling a printed circuit board assembly with a memory controller and non-DRAM type of memory integrated circuits mounted thereto.

In the following detailed description, numerous examples of specific implementations are set forth. However, implementations may include configurations that include less than all of or alternatives for the detailed features and combinations set forth in these examples.

For similar memory capacity, dynamic random access memory (DRAM) integrated circuits (ICs) typically consume more power than non-volatile memory integrated circuits, particularly when data is read. Non-volatile memory integrated circuits typically do not require refresh cycles and thus conserve power. To reduce power consumption in system applications with a main memory, a non-volatile memory integrated circuit may be used in place of or as a supplement to a DRAM integrated circuit.

Typically, a write access to non-volatile memory integrated circuits takes more time than a write access to DRAM integrated circuits. Some types of non-volatile memory integrated circuits, such as NOR FLASH EEPROM integrated circuits, may be configured with improved read access times (e.g., twice that of DRAM integrated circuits). In order to address differences between read and write performance, a data communication protocol may be used that accesses the non-volatile memory modules in a different manner than DRAM memory modules.

In one configuration, a non-DRAM memory controller and non-volatile memory modules may be introduced into a computer system. The technologies used by non-volatile memory integrated circuits differ from dynamic random access memory (DRAM) integrated circuits (ICs) in the structure of their memory cell and in how they store information within the cell. These differences may help the resultant computer system achieve relatively low power consumption characteristics. For example, non-volatile memory integrated circuits typically do not require refresh cycles and thus conserve power. Alternately or in addition, they may help expand the capacity of main memory in the system.

Referring now to FIG. 1 , a functional block diagram of a computer system is illustrated with dual in-line memory modules (DIMMS). The computer system includes a multiprocessor mother-board 100 A. Mounted to the mother-board 100 A is a plurality of processor sockets 101 A- 101 N. Processors 122 A- 122 N may be inserted into these processor sockets 101 A- 101 N. The processor sockets are connected to the interconnect fabric 103 via traces 102 A- 102 N. The interconnect fabric 103 may consist of printed circuit board traces alone or it may include other integrated circuits. The interconnect fabric 103 may be configured to connect the various processors, memory, and I/O together within the mother-board. Portions of the interconnect fabric logic may be embedded within the processors and memory controllers.

Additionally mounted to the mother-board 100 A are one or more memory controllers 107 A- 107 N, 117 connected to the interconnect fabric 103 via traces 106 A- 106 N. The memory controllers 107 A- 107 N, 117 respectively control each of the memory channels 123 A- 123 N, 133 A- 133 N. Additional printed circuit board traces 110 A- 110 N in each of the memory channels 123 A- 123 N, 133 A- 133 N are coupled between the memory module sockets 108 A- 108 N and the memory controllers 107 A- 107 N, 117 .

One or more DRAM memory DIMMS 109 A- 109 N may be accommodated by the sockets 108 A- 108 N in the memory channels 123 A- 123 N. One or more non-DRAM DIMMS 119 A- 119 N, such as non-volatile random access memory (NVRAM) DIMMS, may be accommodated by the sockets 108 A- 108 N in the memory channels 133 A- 133 N.

Additionally mounted to the mother-board 100 A are one or more I/O subsystems 105 A- 105 N that are connected to the interconnect fabric 103 via traces 104 A- 104 N.

Additionally mounted to the mother-board 100 A may be one or more expansion (EXP) connectors 121 A- 121 N that may be connected to the interconnect fabric by traces 120 A- 120 N. In one configuration, one or more of the expansion connectors 121 A- 121 N are used to upgrade the main memory of the mother-board 100 A.

In FIG. 1 , the memory controllers 107 A- 107 N, 117 are directly coupled to the sockets 108 A- 108 N in each respective memory channel 123 A- 123 N, 133 A- 133 N using the PCB traces 110 A- 110 N. However, memory controllers may also indirectly couple to the sockets 108 A- 108 N in each memory channel through secondary memory controllers.

Referring now to FIG. 1A , an alternate memory controller system is used instead of the direct memory control provided by the memory controllers 107 A- 107 N, 117 . The alternate memory controller system includes one or more primary memory controllers (PMC) 113 A for each of several or all memory control channels, and one or more secondary memory controllers (SMC) 112 A- 112 N, (SNVMC) 132 for each of the one or more primary memory controllers 113 A.

The one or more primary memory controllers (PMC) 113 A connect to the interconnect fabric 103 via traces 106 A- 106 N. In this configuration, the one or more primary memory controllers 113 A are indirectly coupled to the memory channels 123 A- 123 N, 133 A- 133 N. Each of the one or more primary memory controllers 113 A connects to the one or more secondary memory controllers (SMC) 112 A- 112 N, (SNVMC) 132 via interconnect traces 111 A- 111 N.

Each of the one or more secondary memory controllers 112 A- 112 N, 132 are coupled to the DIMM sockets 108 A- 108 N via the printed circuit board traces 110 A- 110 N. The one or more secondary memory controllers 112 A- 112 N may couple to one or more of the DRAM DIMMS 109 A- 109 N inserted into the DIMM sockets 108 A- 108 N within the memory channels 123 A- 123 N to control the read and write access to DRAM memory modules. One or more secondary non-DRAM memory controllers (SNVMC) 132 , such as a secondary non-volatile memory controller, may couple to one or more of the non-DRAM DIMMS 119 A- 119 N (such as non-volatile memory (NVRAM) DIMMS) inserted into the DIMM sockets 108 A- 108 N within the memory channels 133 A- 133 N to control the read and write access to non-DRAM memory modules. In some implementations, the secondary non-DRAM memory controller 132 is a secondary non-volatile memory controller to control read and write access to non-volatile memory modules.

Referring now to FIG. 2A , an alternate multiprocessor system and mother-board 100 B is illustrated. In FIG. 2A , the external memory controllers 107 A- 107 N, 117 of FIG. 1 are moved into the processor packages 200 A- 200 N, 200 P′ as part of a processor with one or more integrated memory controllers.

In the multiprocessor system 100 B, processor sockets 204 A- 204 P are connected to the interconnect fabric 103 via the traces 102 A- 102 N. The processor sockets 204 A- 204 N are also connected to the memory channels 123 A- 123 N, 133 A- 133 N via traces 110 A- 110 N. In this construction, the processor package 200 A- 200 N, 200 P′ includes both one or more processor elements (MP) 201 A- 201 P and an integrated memory controller (IMC) 202 A- 202 N, (INVMC) 212 .

The integrated memory controllers (IMC) 202 A- 202 N control read and write accesses to DRAM memory modules 109 A- 109 N plugged into the DIMM sockets 108 A- 108 N within the memory channels 123 A- 123 N. The integrated memory controller (INVMC) 212 controls read and write accesses to non-DRAM memory modules 119 A- 119 N (such as non-volatile memory modules) plugged into the DIMM sockets 108 A- 108 N within the memory channels 133 A- 133 N.

There may be one or more memory controllers within a processor package.

Referring now to FIG. 2B , another alternate multiprocessor system and mother-board 100 B′ is illustrated. The multiprocessor system and mother-board 100 B′ is similar to the multiprocessor system and mother-board 100 B. However in FIG. 2B , the processor package 200 P″ includes one or more processor elements (MP) 201 P with two integrated memory controllers 202 P and 212 .

The integrated memory controller (IMC) 202 P controls read and write accesses to DRAM memory modules 109 A- 109 N plugged into the DIMM sockets 108 A- 108 N within the memory channels 123 N coupled to the processor 200 P″. The integrated memory controller (INVMC) 212 controls read and write accesses to non-DRAM memory modules 119 A- 119 N (such as non-volatile memory modules) plugged into the DIMM sockets 108 A- 108 N within the memory channels 133 N coupled to the processor 200 P″.

As previously discussed, constructions of the multiprocessor systems 100 A, 100 B, and 100 B′ may have one or more expansion connectors 121 A- 121 N. These connectors may be used to improve system performance by increasing memory capacity with a daughter card. In some cases, they may help reduce the power consumption of the main memory of the computer system.

In one configuration, the one or more expansion connectors 121 A- 121 N may be used to upgrade and expand the main memory of the mother-boards 100 A, 100 B, 100 B′.

Referring now to FIG. 3A , a daughter card 300 is illustrated to upgrade the main memory in the computer systems having the mother-boards 100 A, 100 B, 100 B′. The daughter card 300 includes a memory controller 302 to control non-DRAM type of memory DIMMS. The daughter card 300 may also be referred to herein as an expansion board.

One or more expansion boards 300 are respectively connected to the one or more expansion mother-board connectors 121 A- 121 N in the mother-boards 100 A, 100 B, 100 B′ via the edge connector 301 . The non-DRAM memory controller 302 is coupled to the edge connector 301 via traces 306 A- 306 N on the printed circuit board 300 A. The non-DRAM memory controller 302 is connected to each of the non-DRAM memory channels 307 A- 307 N via traces 303 A- 303 N, which, in turn, connect to sockets 304 A- 304 N. Non-DRAM DIMMS 305 A- 305 N are then inserted into the sockets 304 A- 304 N to expand the main memory to include non-DRAM type of memory therein. That is, the main memory in the computer system is expanded to be a heterogeneous main memory with different types of memory integrated circuits therein.

Referring now to FIG. 4 , a diagram of a non-DRAM type of memory module 305 is illustrated. The non-DRAM type of memory module 305 may be plugged into the memory module sockets 304 A- 304 N in the one or more non-DRAM memory channels 307 A- 307 N of the expansion board 300 A illustrated in FIG. 3A .

In one implementation, the non-DRAM type of memory module 305 is a non-volatile type of memory module. In this case, the non-DRAM memory controller 302 is a non-volatile memory controller. In particular, the non-volatile type of memory module may include at least one NOR-gate flash electrically erasable programmable read only memory (EEPROM) integrated circuit in one implementation.

In FIG. 4 , the non-DRAM type of memory module 305 includes a printed circuit board 400 having pads of edge connectors 401 (one on each side for a DIMM) formed thereon, a plurality of non-DRAM memory chips 402 A- 402 N, and a plurality of support chips 403 A- 403 N. The memory module 305 further includes a plurality of printed circuit board traces (e.g., printed wires) 404 A- 404 N and 406 A- 406 L formed on the PCB 400 coupling between the non-DRAM memory chips 402 A- 402 N and the support chips 403 A- 403 N and between the support chips 403 A- 403 N and the pads of the edge connectors 401 .

In one implementation, the memory module 305 is a dual in-line memory module (DIMM) and the printed circuit board (PCB) 400 is a DIMM PCB. The non-DRAM memory chips 402 A- 402 N may be NOR FLASH EEPROM integrated circuit chips or some other kind of non-DRAM type of memory integrated circuit chips.

The plurality of support chips 403 A- 403 N may be used to buffer addresses, and/or multiplex and de-multiplex data to and from the non-DRAM memory chips 402 A- 403 N. The plurality of support chips 403 A- 403 N may also be referred to herein as a plurality of buffer integrated circuits 403 .

In an alternate implementation, non-DRAM type of memory integrated circuits 402 A- 402 N (e.g., NOR Flash EEPROM) and support chips 403 A- 403 N may be directly mounted onto the printed circuit board 300 A of the expansion board 300 in each memory channel 307 A- 307 N and coupled to the traces 303 A- 303 N without the circuit boards 400 , edge connectors 401 , and sockets 304 A- 304 N.

Referring now to FIG. 3B , instead of non-DRAM type of memory modules being coupled to sockets of a printed circuit board, non-DRAM type of memory integrated circuits 312 AA- 312 MN (e.g., NOR Flash EEPROM) and support chips 313 A- 313 N are directly mounted onto the printed circuit board 300 B as shown. The non-DRAM memory controller 302 is also mounted on the printed circuit board 300 B. The support chips 313 A- 313 N are coupled between the memory controller 302 and the non-DRAM type of memory integrated circuits 312 AA- 312 MN.

In one implementation, the non-DRAM type of memory integrated circuits 312 AA- 312 MN are non-volatile memory integrated circuits, such as NOR Flash EEPROM integrated circuits. In one configuration, read and write accesses to a non-volatile memory integrated circuit is asymmetric. In this case, a write access to non-volatile memory integrated circuits takes more time than a read access to non-volatile memory integrated circuits. Some types of non-volatile memory integrated circuits, such as NOR FLASH EEPROM integrated circuits, may be configured so that read access times may be reduced to levels sufficient for use in main memory.

If the system 300 ′ is an expansion board, the printed circuit board 300 B includes an edge connector 301 to plug into an expansion socket on a mother-board. In this case, the memory controller 302 may couple to the edge connector 301 .

While an expansion board may be used to increase the capacity of main memory with non-volatile memory as discussed previously, DRAM memory modules may be swapped out for non-volatile memory modules and DRAM memory controllers may be swapped out for non-volatile memory controllers.

In accordance with the teachings of U.S. provisional patent application 60/827,421 filed on Sep. 28, 2006 by inventors Kumar Ganapathy et al., the main memory of the motherboard board 100 A of FIG. 1 may be upgraded to swap out DRAM memory modules with non-volatile memory modules in one or more memory channels. In doing so, relatively low power consumption characteristics may be attained by the resultant computer system. In this case (see FIG. 1 ), non-volatile memory modules 119 A- 119 N are plugged into sockets 108 A- 108 N of the respective memory channel 133 A- 133 N.

In one implementation (see FIG. 1 ), one or more of the respective memory controllers of the memory channel with the non-volatile memory modules 119 A- 119 N is a non-volatile memory controller 117 to control read and write access to the non-volatile memory modules 119 A- 119 N. In another implementation (see FIG. 1A ), one or more secondary memory controllers of the memory channel may be a non-volatile memory controller 132 to control read and write access to the non-volatile memory modules 119 A- 119 N in the respective memory channel 133 A- 133 N. In yet another implementation (see FIGS. 2A-2B ), one or more processors in the multiprocessor system may be replaced with processor packages 200 P′, 200 P″ having one or more integrated memory controllers 212 , 212 and 202 P, one or more of which may be a non-volatile memory controller 212 to control read and write access to the non-volatile memory modules 119 A- 119 N in a respective memory channel 133 A- 133 N.

Referring now to FIG. 6A in accordance with one implementation, a support integrated circuit chip 313 is illustrated as an instance of each of the support chips 313 A- 313 N illustrated in FIG. 3B or support chips 403 A- 403 N illustrated in FIG. 4 . The support integrated circuit chips support (i.e., facilitate) read and write data accesses by the non-volatile memory controllers to non-volatile type of memory modules and to non-volatile memory integrated circuits.

The support chip 313 may include a many-to-one bus multiplexer 602 and a one-to-many bus demultiplexer 604 . The many-to-one bus multiplexer 602 is used to write data onto a data bus 316 such as data buses 316 A- 316 N illustrated in FIG. 3B or printed circuit board traces 406 A- 406 L illustrated in FIG. 4 . The one-to-many bus demultiplexer 604 may be used to read data from the data bus 316 , such as data buses 316 A- 316 N or the printed circuit board traces 406 A- 406 L, onto one of many data buses such as data buses 314 A- 314 N or printed circuit board traces 404 A- 404 N coupled to the memory integrated circuits.

Referring now to FIG. 6B in accordance with another implementation, a support chip 313 ′ is illustrated as an instance of each of the support chips 313 A- 313 N illustrated in FIG. 3B or support chips 403 A- 403 N illustrated in FIG. 4 . The support chip 313 ′ may instead include a cross-bar switch 606 coupled between the plurality of data buses 314 A- 314 N in FIG. 3B or printed circuit board traces 404 A- 404 N in FIG. 4 connected to the memory integrated circuits and the data bus 316 such as data buses 316 A- 316 N illustrated in FIG. 3B or printed circuit board traces 406 A- 406 L illustrated in FIG. 4 . The cross bar switch 606 is used to write data onto the data bus 316 or printed circuit board traces 406 A- 406 L from the memory integrated circuits. The cross bar switch 616 is used further to read data from the data bus 316 or printed circuit board traces 406 A- 406 L and couple the data onto one of data buses 314 A- 314 N or a plurality of printed circuit board traces 404 A- 404 N connected to the memory integrated circuits.

Referring now to FIG. 7A , a flow chart of a method for upgrading a computing system using an expansion board of the form shown in FIG. 3A or 3B is illustrated.

At block 702 , an expansion board having a memory controller is plugged into an expansion socket on a mother-board. The memory controller is a non-DRAM memory controller to control read and write accesses to non-DRAM type of memory modules. In one implementation, the non-DRAM type of memory modules are non-volatile type of memory modules, such as NOR flash electrically erasable programmable read only memory (EEPROM) for example.

At block 704 , a plurality of non-DRAM type of memory modules are plugged into memory sockets of the expansion board. The memory sockets are coupled to the memory controller by way of printed circuit board traces on the expansion board.

At block 706 , to access the non-DRAM type of memory modules in the expansion board, a data communication protocol is used for the non-DRAM type of memory modules. The data communication protocol to access the non-DRAM type of memory modules may differ from the data communication protocol to access DRAM type of memory modules. If a non-volatile memory module is plugged into a memory module socket a data communication protocol for accessing non-volatile memory modules may be used to address the asymmetry between read and write performance.

In the data communication protocol for accessing non-volatile memory modules, a feedback status control signal is communicated from a non-volatile memory module to the memory controller to alleviate the non-deterministic nature of the erase and write operations in the non-volatile memory modules. With a feedback status control signal, the memory controller can avoid constantly polling the memory module as to when an erase or write operation is completed. Each feedback status control signal indicates whether or not a rank of memory in a memory module is busy or ready for another access to alleviate the non-deterministic nature of erase and write operations to non-volatile memory modules.

Referring now to FIG. 7B , a method of assembling a printed circuit board assembly is illustrated.

At block 712 , non-DRAM type of memory integrated circuits (e.g., NOR Flash EEPROM) are mounted to a printed circuit board and coupled to the printed circuit board traces of the printed circuit board.

At block 714 , support chips are also mounted onto the printed circuit board coupled to printed circuit board traces of the printed circuit board.

At block 716 , a non-DRAM memory controller is also mounted on the printed circuit board and coupled to printed circuit board traces of the printed circuit board.

The support chips are coupled between the memory controller and the non-DRAM type of memory integrated circuits. In one implementation, the non-DRAM type of memory integrated circuits are non-volatile memory integrated circuits, such as NOR Flash EEPROM integrated circuits.

If the printed circuit board is manufactured as an expansion board, the method of assembly of the printed circuit board may further include coupling or forming an edge connector onto or in the printed circuit board. In this case, the non-DRAM memory controller may couple to the edge connector.

Referring now to FIG. 5 , a block diagram of an internet server 500 and a remote client 501 coupled to the internet 502 is illustrated. The internet server 500 includes the mother-board 100 A, 100 B, 100 B′ that has been upgraded by the one or more of the expansion boards 300 A or 300 B plugged into the one or more mother-board expansion sockets 121 A- 121 N. Each of the expansion boards 300 A include non-volatile memory modules 305 plugged into the memory module sockets of one or more added memory channels to upgrade main memory to include non-volatile memory.

An example of the use of non-volatile memory modules in main memory is now described. The remote client 501 executes a search query 510 against a search engine running on the internet server 500 to search for data. In this case, the main memory 512 on the mother-board 100 A, 100 B associated with internet server 500 may be more often read than it is written. This application behavior permits the use of non-volatile memory modules in lieu of DRAM memory modules. With the mother-board 100 A, 100 B, 100 B′ upgraded to include non-volatile memory modules in its main memory 512 , power is conserved over that of a main memory solely having DRAM memory modules. Additionally, because of the cost advantage of non-volatile memory integrated circuits over DRAM integrated circuits, the internet server 500 can be configured with the same main memory capacity for less money, or alternately, can benefit from higher main memory capacity for the same cost.

While this specification includes many specifics, these should not be construed as limitations on the scope of the disclosure or of what may be claimed, but rather as descriptions of features specific to particular implementations of the disclosure. Certain features that are described in this specification in the context of separate implementations also may be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation also may be implemented in multiple implementations separately or in sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Other implementations are within the scope of the following claims. For example, the memory modules and the memory sockets have been described as being dual in-line memory modules (DIMM) and DIMM sockets. However, the memory modules and memory sockets may have other types of form factors such as single in-line memory modules (SIMM), for example.