Method and apparatus for maintaining compliant storage

A patent application for “Method and Apparatus for Mirroring Non-Compliant Storage to Compliant Storage”, filed on Sep. 1, 2004, having the application Ser. No. 10/932,683, is hereby incorporated by reference in its entirety.

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

The invention generally relates to data storage, and specifically to maintaining compliant data storage of unstructured data.

There are a number of regulations that require a variety of data records be available for retrieval, for a specified period of time, from non-modifiable, non-erasable archives. For example, Securities and Exchange Commission (SEC) Rule 17a-4 (i.e., 17 C.F.R. §240.17a-4) requires that certain stock exchange members, brokers, and dealers maintain certain records for a period of time (typically three or seven years). Rule 17a-4 (hereinafter “the Rule,” which may also encompass any other data permanence regulation) encompasses computerized records, such as e-mail and application documents (such as documents produced by Microsoft® Word® or Excel®). This data must therefore be archived for the period of time specified by the Rule in order to comply with the Rule.

Compliant storage is used to store the data required by the Rule. “Compliant” storage refers to data storage that complies with the Rule. “Non-compliant” storage refers to data storage that does not comply with the Rule. Compliance generally requires that the data must be archived and cannot be deleted or modified until the end of the period for which it must be retained. The data must also be retrievable in a reasonable period of time.

There are generally two types of computer records—those that are static and those that are frequently modified. E-mail is an example of static computer records. An e-mail sender composes and distributes an e-mail to one or more e-mail recipients. The recipients can either retain or delete the original e-mail. The recipients cannot modify the original e-mail. One or more recipients may reply to the e-mail, but the reply constitutes a discrete new record. Microsoft® Word® or Excel® documents are examples of frequently modified data. A user may work on the same document over a period of time. That user may choose to rewrite entire sections of the document. Over the course of its existence, a user may create hundreds or thousands of unique versions of that file that can be printed, viewed, or analyzed. The primary difference between static and modifiable documents is the notion of publication. An email recipient has received a published record. It has been distributed in a completed form. A Microsoft® Word® document, however, does not undergo such a publication event. Therefore, since a large percentage of computer data can easily be modified, several different versions of each document may need to be archived in order to comply with the Rule. Compliant storage therefore generates a copy of each document at a predetermined “reasonable” interval, for example, once a day. Compliant storage also requires that every copy must be non-deletable before the expiration of the period for which the copy must be maintained. Typical compliant storage includes optical and magnetic (tape) media.

FIG. 1 illustrates compliant storage of e-mail data. A system 100 includes an exchange server 102 , an application 104 , and compliant storage 106 . The exchange server 102 includes a database 108 containing e-mail data. The application 104 extracts the e-mail data from the database 108 and stores the data on the compliant storage 106 . The application may be software produced by Legato Systems, KVS, etc. The compliant storage 106 typically includes optical or tape media, and is stored for the period of time required by the Rule.

Databases are considered structured data, and e-mail data is considered semi-structured. Databases and other structured data can easily be stored using the application 104 , and the compliant storage 106 . The system 100 searches for new e-mail messages, and archives them. Since the e-mail database is semi-structured, and the e-mail data can easily be organized by the date of creation, the application 104 can easily determine the changes made since the last archive was created. Unstructured data, such as application files including word processing or spreadsheet files, cannot be archived using the system 100 , because the application is unable to determine what changes have been made to the documents. As a result, unstructured data is typically archived in a compliant manner by performing full system backups to the compliant storage 106 on a regular basis.

Performing and maintaining frequent full system backups for several years, is both resource intensive and expensive. A storage server that requires archival may include tens of terabytes (TB) of data. If compliant backups are performed daily, then more than one thousand multi-TB backups would have to be maintained concurrently to comply with the Rule. The backups consume thousands of pieces of discrete media to be tracked, maintained, and kept available for retrieval.

Furthermore, during an investigation, the SEC may also require the stock exchange member, broker, or dealer under investigation to produce the records made on a specific date. System administrators managing the backups must search through the compliant media to locate the requisite data. The scale of search makes it difficult to fulfill the requirement that the data must be obtained within a short period of time to comply with the investigation.

What is needed is a method and apparatus for easily storing and retrieving archival documents to comply with the Rule.

The present invention includes methods and apparatuses for maintaining compliant storage. In one method, a request to delete a point-in-time image is received. A data structure including a retention date for the point-in-time image is then referenced. The point-in-time image is deleted if a current date is later than the retention date.

Other aspects of the invention will be apparent from the accompanying figures and from the detailed description which follows.

One or more embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:

FIG. 1 illustrates compliant storage of e-mail data;

FIG. 2 illustrates an example of a configuration to facilitate data backup using mirroring;

FIG. 3 shows the architecture of a filer, representative of the storage server or the mirroring server, according to certain embodiments of the invention;

FIG. 4 illustrates the operating system of the filer, according to certain embodiments of the invention;

FIG. 5 illustrates a technique for creating point-in-time images according to an embodiment of the invention;

FIG. 6A is a flowchart describing a process for modifying a delete command in the file system;

FIG. 6B illustrates a table that is referenced by the file system when a delete command is issued; and

FIG. 7 illustrates a control panel for setting retention times according to one embodiment of the invention.

Described herein are methods and apparatuses for maintaining compliant storage. Note that in this description, references to “one embodiment” or “an embodiment” mean that the feature being referred to is included in at least one embodiment of the present invention. Further, separate references to “one embodiment” or “an embodiment” in this description do not necessarily refer to the same embodiment. However, such embodiments are also not mutually exclusive unless so stated, and except as will be readily apparent to those skilled in the art from the description. For example, a feature, structure, act, etc. described in one embodiment may also be included in other embodiments. Thus, the present invention can include a variety of combinations and/or integrations of the embodiments described herein.

According to an embodiment of the invention, a storage server is mirrored onto a mirroring server. The mirroring server maintains point-in-time images of data stored on the storage server. The point-in-time images have a retention period. During the retention period, the point-in-time images must be maintained to comply with 17 C.F.R. 240.17a-4 (the “Rule”). It should be appreciated that other rules such as 17 C.F.R. 240.17a-3 or other regulations may be used with embodiments of the invention. When a command to delete a point-in-time image is issued, the file system of the mirroring server references a table including a list of point-in-time images and their retention dates. If the current date is greater than the retention date, the point-in-time image is deleted. If the current date is less than the retention date, the point-in-time image cannot be deleted.

FIG. 2 illustrates an example of a configuration to facilitate data backup using mirroring. The system 200 includes a storage server 202 , and a mirroring server 204 . The storage server 202 stores data, which is mirrored on the mirroring server 204 . Mirroring describes an arrangement between a source and a destination. In this case, the storage server 202 is the source, and the mirroring server 204 is the destination. In the mirroring relationship, the mirroring server 204 maintains a copy of a volume stored by the storage server 202 . A volume is a logical collection of data that may include one or more physical data storage devices, such as magnetic or optical storage.

Mirroring may be synchronous or asynchronous. In a synchronous mirroring relationship, every change that is made to the volume maintained by the source is written to the mirrored volume on the destination before the source considers the data to be written. Using an asynchronous relationship, the destination periodically updates the mirror using changes made to the source since the last update. Embodiments of this invention are described using asynchronous mirroring. However, it is understood that synchronous mirroring may also be used.

Several clients 206 are connected to the storage server 202 . The storage server 202 and mirroring server 204 are connected through a network connection 212 . The clients 206 are individual workstations that store data on the storage server 202 . The storage server 202 may be a file server, or filer, such as those manufactured by Network Appliance, Inc., located in Sunnyvale, Calif. (NetApp®). The storage server 202 maintains centralized data storage for the several clients 206 . The clients 206 may use several different interfaces, such as the Small Computer System Interface, (iSCSI), Transmission Control Protocol (TCP), Network File System (NFS), and Common Internet File System (CMS) to communicate with the storage server 202 . The storage server 202 may maintain one or more volumes 208 that include data such as Microsoft Word® documents 210 a and other documents 210 b that may include unstructured data stored by the storage server 202 . According to one embodiment, the volumes 208 and the documents 210 a and 210 b do not comply with the Rule or other regulation. The documents 210 a and 210 b can generally be modified or deleted, since the storage server 202 is used as storage for the clients 206 .

The mirroring server 204 may be maintained at a different physical location than the storage server 202 . The network 212 may be a local area network (LAN), or a wide area network (WAN) such as the Internet. If the mirroring server 204 is maintained at a separate location physically remote to the storage server 202 certain conditions that affect the storage server 202 (such as power outages) may not affect the mirroring server 204 .

The mirroring server 204 may also be a file server such as a filer. According to one embodiment of the invention, the mirroring server 204 includes less expensive physical storage than the storage used for the storage server 202 to reduce the cost of the mirroring server 204 and increase the amount of storage available. The mirroring server 204 maintains a mirror 214 of the volume 208 maintained by the storage server 202 . The mirroring server 204 also maintains a volume 216 of point-in-time images 218 . The point-in-time images 218 , as will be explained below, are images of the mirrored volume 214 that can be used to recreate the state of the mirrored system at a specific point-in-time. For example, a point-in-time image 218 may be able to recreate every file in the mirrored volume 214 as it was on Apr. 15, 2004. According to an embodiment of the invention, the point-in-time images 218 comply with the Rule or other regulation. According to this embodiment, the system 200 generates compliant point-in-time images 218 .

FIG. 3 shows the architecture of a filer 300 , representative of the storage server 202 or the mirroring server 204 , according to certain embodiments of the invention. A filer is a type of storage server used to store file data. Note that certain standard and well-known components which are not germane to the present invention are not shown. The filer 300 includes a processor 302 and main memory 304 , coupled together by a bus system 306 . The bus system 306 in FIG. 3 is an abstraction that represents any one or more separate physical buses and/or point-to-point connections, connected by appropriate bridges, adapters and/or controllers. The bus system 306 , therefore, may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (sometimes referred to as “Firewire”).

The processor 302 is the central processing unit (CPU) of the filer 300 and, thus, controls the overall operation of the filer 300 . In certain embodiments, the processor 302 accomplishes this by executing software stored in main memory 304 . The processor 302 may be, or may include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or the like, or a combination of such devices.

The main memory 304 , which is generally some form of random access memory (RAM), stores at least a portion of the operating system 308 of the filer 300 . Techniques of the present invention may be implemented within the operating system 308 , as described further below. The operating system 308 may be, for example, the ONTAP operating system by NetApp®. Also connected to the processor 302 through the bus system 306 are a network adapter 310 and a storage adapter 312 . The network adapter 310 provides the filer 300 with the ability to communicate with remote devices, such as clients and/or another filer, over a network and may be, for example, an Ethernet adapter. The storage adapter 312 allows the filer to access the external mass storage devices and may be, for example, a Fibre Channel (FC) adapter or SCSI adapter.

FIG. 4 illustrates the operating system 308 of the filer 300 , according to certain embodiments of the invention. As can be seen, the operating system 308 includes a number of layers. The core of the operating system 308 is the file system 402 . The file system 402 is a programmatic entity that imposes structure on an address space of one or more physical or virtual storage devices, such as disks, so that the operating system 308 may conveniently read and write data containers, such as files and blocks, and related metadata. The file system 402 , among other responsibilities, executes read and write operations on the mass storage devices in response to client requests, maintains directories, and manages consistency point operations. An example of the file system 402 suitable for this purpose is the Write Anywhere File Layout to (WAFL) file system from Network Appliance, such as used in the NetApp® Filers. The file system 402 in certain embodiments operates on blocks of data of a predetermined exemplary size, such as 4 Kbytes. Also shown in FIG. 4 is the logical data path 404 from clients to mass storage devices, through the file system 402 .

The operating system 308 also includes a user interface 406 , through which a network administrator or other user can control and/or configure the filer 300 of FIG. 3 (e.g., remotely from a management station). The user interface 406 may generate a command line interface and/or a graphical user interface for this purpose.

The client side of the operating system 308 includes a network access layer 408 and, at the lowest level, a media access layer 410 . The network access layer 408 implements any of various protocols used to communicate with client devices, such as network file system (NFS), common Internet file system (CIFS) and/or hypertext transport protocol (HTTP). The media access layer 410 includes one or more drivers which implement the protocols used to communicate over the network, such as Ethernet.

On the storage device side, the operating system 308 includes a storage access layer 412 and, at the lowest level, a driver layer 414 . The storage access layer 412 implements a disk storage protocol such as RAID, while the driver layer 414 implements a lower-level storage device access protocol, such as Fibre Channel or SCSI.

The operating system of 308 also includes a mirroring module 416 , which is operatively coupled to the file system 402 and the storage access layer 412 . The mirroring module 416 controls the synchronization of data at the remote secondary site with data stored at the primary site. The techniques introduced herein may be implemented at least partially using the mirroring module 416 .

FIG. 5 illustrates a technique for creating point-in-time images 218 according to an embodiment of the invention. The point-in-time image 218 (see FIG. 2 ) is a data structure stored by the mirroring server. The point-in-time image 218 is a data structure that recreates a file structure of a volume including pointers that point to blocks including the data stored by that file structure. The point-in-time 218 image may be recalled at a later time to recreate the state of a volume managed by the storage server 202 of FIG. 2 at the time the point-in-time image was created. When the volume 214 of FIG. 2 is recreated, the pointers stored by the point-in-time image 218 point to blocks created by a baseline image or by previous point-in-time images. For example, if one wanted to know the contents of a file on a certain date, they could recall the point-in-time image 218 for that date.

FIG. 5 illustrates a file 500 being stored by the point-in-time image. Files and other data are typically stored in the form of one or more blocks. A block is a logical unit of data storage on a storage device. A file may include pointers referencing one or more blocks that store the data comprising the file. The point-in-time image 218 (see FIG. 2 ) typically includes pointers to new blocks that have changed since the last point-in-time image 218 . For example, if only one file had been modified since the last point-in-time image 218 , the current point-in-time image 218 would include the new blocks of the file, and the pointers to the new blocks, as well as pointers to the old blocks that had not changed. When the new blocks are copied onto the mirroring server 204 of FIG. 2 , the original blocks are not modified. In this way, both the new version of the file, and the file as it originally was, can be recreated.

When the first point-in-time image 218 (see FIG. 2 ) in the volume 216 of FIG. 2 is created, all of the blocks corresponding to the files that are to be archived are copied into the volume 216 . This set of blocks is known as the baseline. The original point-in-time image points to these data blocks. When blocks are subsequently modified, the new point-in-time images will point to copies of the original blocks, which cannot be deleted in order to maintain compliance.

The point-in-time image includes pointers to recreate the several files stored by the storage server 202 of FIG. 2 . The file 500 includes several pointers 502 to data stored in four blocks 504 , 506 , 508 , and 510 . Each reference to the file 500 in a point-in-time image includes the pointers to the blocks 504 - 510 . For example, if, on April 15, the file 500 includes the data in the blocks 504 - 510 , the point-in-time image created on April 15 will include pointers to the blocks 504 - 510 representing the file 500 . If, after the April 15 point-in-time image is created, the block 510 is modified, when the April 16 point-in-time image is created, a copy 512 of the block 510 is created. In the April 16 point-in-time image, the pointers 502 point to the block 512 rather than the block 510 . In this way, the mirroring server 204 of FIG. 2 avoids copying redundant information.

When a point-in-time image is made, the blocks to which the point-in-time image refers are “locked.” When the blocks are locked, according to embodiments of the invention, they cannot be deleted until the retention period has expired. Further, they also cannot be modified. When a block is modified on the storage server 202 of FIG. 2 , the mirroring server 204 of FIG. 2 creates a copy of the block, to which the new version of the file in the new point-in-time image points. As soon as the copied block 512 is created, it is also locked, and cannot be modified.

The point-in-time images 218 are made according to a specified scheduling criterion. For example, the criterion may be a predetermined or regular interval, such as once daily. The criterion may be determined so as to comply with the Rule. For example, the criterion may be chosen so that the point-in-time images 218 are created at a “reasonable” interval.

FIGS. 6A and 6B illustrate a technique for implementing a retention date for point-in-time images. FIG. 6A is a flowchart describing a process for modifying a delete command in the file system 402 (see FIG. 4 ). FIG. 6B illustrates a table 650 that is referenced by the file system 402 when a delete command is issued. The techniques described herein are used when a system command to delete a point-in-time image is issued. In order to comply with the Rule, the point-in-time images must be retained for the stated retention period (typically three or seven years).

The process 600 describes the actions taken by the file system 402 (see FIG. 4 ) when a delete command is issued. In block 602 , a delete command to delete a point-in-time image is received by the file system 402 . The delete command may have been issued by an automated process, a user, an application, etc. The delete command is directed toward one or more point-in-time images 218 (see FIG. 2 ) in the volume 216 (see FIG. 2 ) of point-in-time images 218 .

In block 604 , the process 600 determines which point-in-time image 218 (see FIG. 2 ) is to be deleted. The point-in-time images 218 may be organized using any of several naming conventions. For example, a point-in-time image 218 made on Apr. 15, 2004, may be given the identifier “Apr.15.2004.” This identifier corresponds to an image number. Each point-in-time image 218 , as it is created, is also assigned an image number, whose order may not be consistent with the identifier issued above. For example, the Apr.15.2004 image may have the image number 14 . The Apr.16.2004 image may have an image number 21 . The file system 402 (see FIG. 4 ) reconciles the identifiers and the image numbers.

In block 606 , the process 600 calls the table 650 , as shown in FIG. 6B . The table 650 includes a column 652 including image numbers and a column 654 including retention dates for those image numbers. The retention date corresponds to a retention period for the image. For example, if an image is created on Apr. 15, 2004, and the retention period is three years, then the retention date would be Apr. 15, 2007, as shown in the second column 654 .

In block 608 , it is determined whether the current date is greater than the retention date. If it is later than the retention date found in the column 654 , then, in block 610 , the point-in-time image is deleted. If the current date is prior to the retention date, then in block 612 , the point-in-time image 218 (see FIG. 2 ) may not be deleted.

The process 600 can be used to ensure that the point-in-time images 218 (see FIG. 2 ) stored in the volume 216 are compliant with the Rule. The file system 402 (see FIG. 4 ) incorporates the process 600 into the delete command, requiring that the table 650 be referenced before a delete command can be executed. This ensures that a point-in-time image cannot be deleted before the end of the retention period.

FIG. 7 illustrates a control panel 700 for setting retention times according to one embodiment of the invention. The control panel may be issued for an individual image, according to the image's identifier. A pull-down menu 702 or other selection device can be used to choose different point-in-time images 218 (see FIG. 2 ).

According to one embodiment of the invention, the retention period may be chosen using several pull-down menus 704 . According to the embodiment, in order to maintain compliance with the Rule, the control panel 700 may disallow any retention time less than would be necessary for compliance. For example, if a system administrator chose a two-year retention time where three were necessary, the control panel 700 could prevent the system administrator from selecting that retention time and issue an error message.

The control panel 700 may also include several radio buttons 706 that can be used to select times according to the type of data. For example, a point-in-time image may point to data subject to Rule 17a-4(a)(10). If the system administrator were to choose the radio button corresponding to that rule, the control panel 700 would automatically choose the correct retention period. The system administrator could then lengthen the retention period if necessary.

The control panel 700 may also include a button 708 that can freeze the point-in-image 218 (see FIG. 2 ), preventing it from being deleted. A system administrator may choose to freeze a particular point-in-time image 218 if it is of interest. For example, if the SEC or other government agency is investigating the actions that occurred on the day the image was written, the image may be “frozen” to prevent it from being deleted, even after the original retention period.

It is understood that the control panel 700 may be have several configurations other than the one shown. For example, the control panel 700 may be implemented using a text-based parser or through a physical interface. The control panel 700 may also include other commands or choices.

The techniques introduced above have been described in the context of a network attached storage (NAS) environment. However, these techniques can also be applied in various other contexts. For example, the techniques introduced above can be applied in a storage area network (SAN) environment. A SAN is a highly efficient network of interconnected, shared storage devices. One difference between NAS and SAN is that in a SAN, the storage server (which may be an appliance) provides a remote host with block-level access to stored data, whereas in a NAS configuration, the storage server provides clients with file-level access to stored data. Thus, the techniques introduced above are not limited to use in a file server or in a NAS environment.

This invention has been described with reference to specific exemplary embodiments thereof. It will, however, be evident to persons having the benefit of this disclosure that various modifications changes may be made to these embodiments without departing from the broader spirit and scope of the invention. The specification and drawings are accordingly to be regarded in an illustrative rather than in a restrictive sense.