Methods for seamless storage importing and devices thereof

This application claims the benefit of U.S. Provisional Patent Application Ser. No. 61/302,864, filed Feb. 9, 2010, which is hereby incorporated by reference in its entirety.

This technology generally relates to methods and devices for appartuses for importing file systems and, more particularly, to methods for seamless storage importing of file systems and devices thereof.

In a virtualized storage environment, the virtualization device maintains metadata comprising all file system entries for all virtualized storage devices. The initial inventory scan (storage import) of the backend file systems builds the metadata. This storage import involves the virtualization device walking the entire directory structure to enumerate every file and directory in the backend file systems. It is critical that the file system entries are accurate and no incorrect entries exist in the metadata, and that no entries are missed. Depending on the dataset, storage import can take from minutes to several days or even weeks to complete.

During this time period, client querying and client operations that may result in file system changes are sent to the virtualization device and are expected to be handled and responses to in a timely manner. However, due to the nature and requirements of the storage import process described above, certain restrictions have been imposed on the types of client operations that are permitted during the data storage import to ensure that the client modifications do not compromise the integrity of the metadata.

Currently, directory modifications, such as directory renames or directory deletions, as well as certain file operations, are not allowed during the storage import scan. This is non-optimal at best and non-workable at worst in certain environments where clients may lose access to view or modify their files for long periods of time during the storage import process, which as mentioned earlier may take anywhere from hours to days/weeks. Additionally, in case of metadata failure where a rebuild of the metadata is required, these restrictions could easily interrupt the normal client workflow and impact operations with potential substantial costs.

A method for seamless storage importing includes importing with a storage management computing device file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained at the storage management computing device during the importing. An import status of the requested one of the entries is determined with the storage management computing device. The obtained request is processed with the storage management computing device based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored with the storage management computing device.

A non-transitory computer readable medium has stored thereon instructions for seamless storage importing comprising machine executable code which when executed by at least one processor, causes the processor to perform steps including importing file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained during the importing. An import status of the requested one of the entries is determined. The obtained request is processed based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored.

A storage management computing device includes a memory coupled to the one or more processors which are configured to execute programmed instructions stored in the memory including importing file system structure entries stored in one or more storage devices. A request for an operation on one of the entries in the one or more storage devices is obtained during the importing. An import status of the requested one of the entries is determined. The obtained request is processed based on the determined import status of the requested one of the entries without interrupting the importing. The requested one of the entries is updated with the storage management computing device based on the processing. An aggregation of the imported entries with the updating of the requested one of the entries is stored.

This technology provides a number of advantages including providing more effective and efficient methods and devices for seamless storage importing of file systems. With this technology, file system operations are permitted during the storage import process. As a result, clients do not lose access to view or modify their files during the storage import process which could potentially take anywhere from hours to days/weeks. Additionally, while permitting files system operations during storage import, this technology still ensures that all imported file system and metadata entries are accurate.

FIG. 1 is an environment with an exemplary storage management computing device for seamless storage import;

FIG. 2 is a flowchart of an exemplary method for seamless storage importing;

FIG. 3 is an example of a directory to import;

FIG. 4 is a flowchart of an exemplary method for processing of a directory;

FIG. 5 is a flowchart of an exemplary method for managing the start of processing of a directory;

FIG. 6 is a flowchart of an exemplary method for managing the completion of processing of a directory;

FIG. 7 is a flowchart of an exemplary method for updating a scan based on one or more client changes during importing;

FIG. 8 is a flowchart of an exemplary method for processing each entry in a directory;

FIG. 9 is a flowchart of an exemplary method for managing importation of a directory during a client operation; and

FIG. 10 is a flowchart of an exemplary method for processing the next component.

An environment 10 with an exemplary storage management computing device 14 for seamless storage import is illustrated in FIG. 1 . The environment 10 includes the storage management computing device 14 , a client computing device 12 , data storage devices 16 ( 1 )- 16 ( 2 ), and a metadata storage device 18 which are all coupled together by one or more communication networks 20 , although this environment can include other numbers and types of systems, devices, components, and elements in other configurations. This technology provides a number of advantages including providing methods and devices for seamlessly importing file systems.

The storage management computing device 14 provides seamless storage import from the data storage devices 16 ( 1 )- 16 ( 2 ) and the metadata storage device 18 , although other numbers and types of systems could be used and other numbers and types of functions could be performed. The storage management computing device 14 includes a central processing unit (CPU) or processor 22 , a memory 24 , and an interface system 26 which are coupled together by a bus or other link, although other numbers and types of systems, devices, components, and elements in other configurations and locations can be used. The processor 22 in the storage management computing device 14 executes a program of stored instructions for one or more aspects of the present technology as described and illustrated by way of the examples herein, although other types and numbers of processing devices and logic could be used and the processor could execute other numbers and types of programmed instructions.

The memory 24 in the storage management computing device 14 stores these programmed instructions for one or more aspects of the present technology as described and illustrated herein, although some or all of the programmed instructions could be stored and executed elsewhere. A variety of different types of memory storage devices, such as a random access memory (RAM) or a read only memory (ROM) in the system or a floppy disk, hard disk, CD ROM, DVD ROM, or other computer readable medium which is read from and written to by a magnetic, optical, or other reading and writing system that is coupled to the processor 22 in the storage management computing device 14 , can be used for the memory 24 in the storage management computing device 14 .

The interface system 26 in the storage management computing device 14 is used to operatively couple and communicate between the storage management computing device 14 and the client computing device 12 as well as the data storage devices 16 ( 1 )- 16 ( 2 ) and the metadata storage device 18 via one or more of the communications networks 20 , although other types and numbers of communication networks or systems with other types and numbers of connections and configurations can be used. By way of example only, the one or more communications networks can use TCP/IP over Ethernet and industry-standard protocols, including NFS, CIFS, SOAP, XML, LDAP, and SNMP, although other types and numbers of communication networks, such as a direct connection, a local area network, a wide area network, modems and phone lines, e-mail, and wireless communication technology, each having their own communications protocols, can be used.

The client computing device 12 utilizes the storage management computing device 14 to conduct one or more operations with one or more of the data storage devices 16 ( 1 )- 16 ( 2 ) and the metadata storage device 18 , such as to store a file or directory, delete a file or directory, create a file or directory, or rename a file or directory by way of example only, although other numbers and types of systems could be utilizing these resources and other types and numbers of functions utilizing other types of protocols could be performed. Although one client computing device 12 is shown, other numbers and types of devices could be used to interact with the storage management computing device 14 , the data storage devices 16 ( 1 )- 16 ( 2 ) and the metadata storage device 18 . Similarly, other numbers and types of data storage devices and metadata storage devices could be used. Each of the data storage devices 16 ( 1 )- 16 ( 2 ) stores content, such as files and directories, although other numbers and types of storage systems which could have other numbers and types of functions and store other data could be used. The metadata storage server 18 stores metadata relating to the stored content, such as files and directories, on the data storage devices 16 ( 1 )- 16 ( 2 ).

The client computing device 12 , the data storage devices 16 ( 1 )- 16 ( 2 ), and the metadata storage device 18 each include a central processing unit (CPU) or processor, a memory, and an interface or I/O system, which are coupled together by a bus or other link, although each could comprise other numbers and types of elements and component, such as control logic. The client computing device 12 , in this example, may make requests for and send data to different data storage devices 16 ( 1 )- 16 ( 2 ) and interact with metadata storage device 18 via the storage management computing device 14 . Generally, the storage management computing device 14 process requests received from the requesting client computing device 12 for files or directories on one or more of the data storage devices 16 ( 1 )- 16 ( 2 ) using metadata stored in metadata storage server 18 . The data storage devices 16 ( 1 )- 16 ( 2 ) may provide data or receive data in response to the received request.

Although examples of the storage management computing device 14 , the client computing device 12 , the data storage devices 16 ( 1 )- 16 ( 2 ), and the metadata storage device 18 are described herein, each of these systems can be implemented on any suitable computer system or computing device. It is to be understood that the devices and systems of the examples described herein are for exemplary purposes, as many variations of the specific hardware and software used to implement the examples are possible, as will be appreciated by those skilled in the relevant art(s).

Furthermore, each of the systems of the examples may be conveniently implemented using one or more general purpose computer systems, microprocessors, digital signal processors, and micro-controllers, programmed according to the teachings of the examples, as described and illustrated herein, and as will be appreciated by those ordinary skill in the art.

In addition, two or more computing systems or devices can be substituted for any one of the systems in any embodiment of the examples. Accordingly, principles and advantages of distributed processing, such as redundancy and replication also can be implemented, as desired, to increase the robustness and performance of the devices and systems of the examples. The examples may also be implemented on computer system or systems that extend across any suitable network using any suitable interface mechanisms and communications technologies, including by way of example only telecommunications in any suitable form (e.g., voice and modem), wireless communications media, wireless communications networks, cellular communications networks, G3 communications networks, Public Switched Telephone Network (PSTNs), Packet Data Networks (PDNs), the Internet, intranets, and combinations thereof.

The examples may also be embodied as a computer readable medium having instructions stored thereon for one or more aspects of the present technology as described and illustrated by way of the examples herein, as described herein, which when executed by a processor, cause the processor to carry out the steps necessary to implement the methods of the examples, as described and illustrated herein. Exemplary methods for seamless storage import will now be described herein.

An exemplary method for seamless storage importing will now be described with reference to FIGS. 1-10 . Referring more specifically to FIG. 2 , in step 200 the storage management computing device 14 begins to process an exemplary directory illustrated in FIG. 3 . This exemplary directory includes a plurality of files (file 1 -fileN) and a plurality of directories (dir 1 -dirN), although the directory could comprise other numbers and types of structures.

The exemplary processing of the directory in step 200 of FIG. 2 is described in greater detail with reference to FIGS. 4-6 . Referring more specifically to FIG. 4 , in step 202 the processing of the directory by the storage management computing device 14 . Next, in step 204 the storage management computing device 14 determines whether processing of a new directory has started. If in step 204 the storage management computing device 14 determines processing of a new directory has been started, then the Yes branch is taken to B at the start of the flowchart shown in FIG. 5 .

Referring to FIG. 5 , in step 212 the storage management computing device 14 determines whether the directory is already being processed. If in step 212 the storage management computing device 14 determines the directory currently being imported is already being processed, then the Yes branch is taken to step 214 . In step 214 , the storage management computing device 14 skips initiating any new processing with respect to this directory. Next, in step 222 this portion of the processing the directory is done and the storage management computing device 14 returns to step 210 in FIG. 4 .

If in step 212 the storage management computing device 14 determines the directory currently being imported is not already being processed, then the No branch is taken to step 216 . In step 216 , the storage management computing device 14 indicates that the directory currently being imported is now being processed and initiates tracking all changes to this directory, such as by a file operation received from the client device 12 by way of example only.

In step 218 , the storage management computing device 14 determines whether a processing structure for this directory currently being imported exists. If in step 218 the storage management computing device 14 determines a processing structure for this directory currently being imported does not exist, then the No branch is taken to step 220 . In step 220 , the storage management computing device 14 creates a processing structure for this directory currently being imported and then proceeds to step 222 where this portion of the step of processing the directory in step 200 in FIG. 2 is done. If in step 218 the storage management computing device 14 determines a processing structure for this directory currently being imported does exist, then the Yes branch is taken to step 222 . Next, in step 222 this portion of the processing the directory is done and the storage management computing device 14 returns to step 210 in FIG. 4 .

Referring back to FIG. 4 , if in step 204 the storage management computing device 14 determines processing of a new directory has not been started, then the No branch is taken to step 206 . In step 206 , the storage management computing device 14 determines whether the directory currently being processed is restarting processing. If in step 206 the storage management computing device 14 determines the directory currently being processed is restarting, then the Yes branch is taken to step 208 . In step 208 , the storage management computing device 14 resets the processing back to the beginning of the directory and returns to step 204 .

If in step 206 the storage management computing device 14 determines the directory currently being processed is not restarting, then the No branch is taken to step 210 . In step 210 , the storage management computing device 14 determines whether processing of the directory currently being imported is done. If in step 210 , the storage management computing device 14 determines that processing of the directory currently being imported is done, then the Yes branch is taken to C at the start of the flowchart shown in FIG. 6 .

Referring to FIG. 6 , in step 240 , the storage management computing device 14 sets the pending processing notification for the directory currently being processed to a done state. In step 242 , the storage management computing device 14 determines whether the processing structure established for the directory currently being process is referenced by any other work threads for other directories. If in step 242 , the storage management computing device 14 determines the processing structure is referenced by another work thread, then the Yes branch is taken to step 246 where this portion of the step of processing the directory in step 200 in FIG. 2 is done.

If in step 242 , the storage management computing device 14 determines the processing structure is not referenced by another work thread, then the No branch is taken to step 244 . In step 244 , the storage management computing device 14 deletes this processing structure for the directory which was currently being processed and then proceeds to step 246 where this portion of the step of processing the directory in step 200 in FIG. 2 is done.

Referring back to FIG. 4 , if in step 210 , the storage management computing device 14 determines processing of the directory currently being imported is not done, then the No branch is taken to A back at the start of the flowchart shown in FIG. 2 .

Referring back to FIG. 2 , after the processing of the directory in step 200 and as described in greater detail with reference to FIGS. 4-6 , the storage management computing device 14 proceeds to step 300 . In step 300 , the storage management computing device 14 updates the scan based on any changes resulting from one or more file operations initiated by the client computing device 12 , although the changes can occur in other manners from operations initiated by other devices.

Referring more specifically to FIG. 7 , in step 302 the storage management computing device 12 determines whether any changes have been made to the directory which currently is being processed by the client device 12 or other device or system. If in step 302 , the storage management computing device 12 determines one or more changes have not been made to the directory which currently is being processed, then the No branch is taken to step 308 where this portion of the step of processing the directory in step 300 in FIG. 2 is done.

If in step 302 , the storage management computing device 12 determines one or more changes have been made to the directory which currently is being processed, then the Yes branch is taken to step 304 . In step 304 , the storage management computing device 14 determines whether this directory currently being processed has been marked or otherwise identified as requiring a restart because of a change by the client computing device 12 or other device. If in step 304 , the storage management computing device 14 determines this directory currently being processed has been marked or otherwise identified as requiring a restart, then the Yes branch is taken to D in FIG. 4 where a directory restart process starts in step 206 .

If in step 304 , the storage management computing device 14 determines this directory currently being processed has not been marked or otherwise identified as requiring a restart, then the No branch is taken to step 306 . In step 306 , the storage management computing device 14 determines whether any of the parent directories to the directory currently being processed have any changes that require a restart of this directory. If in step 306 , the storage management computing device 14 determines none of the parent directories has any changes that requires a restart of this directory, then the No branch is taken to step 308 where this portion of the step of processing the directory in step 300 in FIG. 2 is done.

If in step 306 , the storage management computing device 14 determines one or more of the parent directories has one or more changes that requires a restart of this directory, then the Yes branch is taken to D in FIG. 4 where a directory restart process starts in step 206 .

Referring back to FIG. 2 , after updating the scan based on any changes resulting from one or more file operations initiated by the client computing device 12 in step 300 and as described in greater detail with reference to FIG. 7 , the storage management computing device 14 proceeds to step 400 . In step 400 , the storage management computing device 14 processes each entry in the directory.

Referring more specifically to FIG. 8 , in step 402 , the storage management computing device 14 begins to process each entry in the directory currently being processed for importation. In step 404 , the storage management computing device 14 determines whether there is an importlock currently being held on the entry in the directory being processed. If in step 404 , the storage management computing device 14 determines there is an importlock currently being held on the entry in the directory being processed, then the Yes branch is taken to step 404 until the importlock is released or a programmed period of time expires.

If in step 404 , the storage management computing device 14 determines there is a not importlock currently being held on the entry in the directory being processed, then the No branch is taken to step 406 . In step 406 , the storage management computing device 14 sets an importlock for the entry currently being processed.

In step 408 , the storage management computing device 14 determines whether the client device 12 or other device or system has modified this entry or created a change list entry, although other manners for indicating a change to this entry can be used. If in step 408 , the storage management computing device 14 determines this entry has been modified or a change list entry has been created, then the Yes branch is taken to step 410 . In step 410 , the storage management computing device skips this entry.

If in step 408 , the storage management computing device 14 determines this entry has not been modified and a change list entry has not been created, then the No branch is taken to step 412 . In step 412 , the storage management computing device 12 creates metadata for this entry for the importation process. In step 414 , the storage management computing device 14 releases the importlock on this entry.

In step 416 , the storage management computing device 14 determines whether there are any additional entries in this directory to be processed. If in step 416 , the storage management computing device 14 determines there is an additional entry in this directory to be processed, then the Yes branch is taken back to step 402 . If in step 416 , the storage management computing device 14 determines there is not an additional entry in this directory to be processed, then the No branch is taken back to step 418 where this portion of the step of processing the directory in step 400 in FIG. 2 is done.

Referring back to FIG. 2 , after the client computing device 12 processes each entry in the directory in step 400 and as described in greater detail with reference to FIG. 8 , the storage management computing device 14 proceeds to step 500 . In step 500 , the storage management computing device 14 determines whether the directory has been fully imported. If in step 500 , the storage management computing device 14 determines the directory has not been fully imported, then the No branch is taken back to step 200 .

If in step 500 , the storage management computing device 14 determines the directory has been fully imported, then the Yes branch is taken back to step 600 . In step 600 , the storage management computing device 14 signals that the seamless storage import process has been completed.

Referring to FIG. 9 , an exemplary method for managing importation of a directory during a client operation is illustrated. In step 702 , the storage management computing device 14 monitors for and identifies a file operation on a path in a directory by the client device 12 .

In step 704 , the storage management computing device 14 determines whether the file operation is on a directory which has already completed an import scan. If in step 704 , the storage management computing device 14 determines the file operation is on a directory which has already completed an import scan, then the Yes branch is taken to step 716 . In step 716 , the storage management computing device 14 is ready to service the file operation requested by the client device 12 . If in step 704 , the storage management computing device 14 determines the file operation is on a directory which has not already completed an import scan, then the No branch is taken to step 706 .

In step 706 , the storage management computing device 14 determines whether all path and entry components exist for the requested file operation. If in step 706 , the storage management computing device 14 determines all path and entry components exist for the requested file operation, then the Yes branch is taken to step 716 as described earlier. If in step 706 , the storage management computing device 14 determines all path and entry components do not yet exist for the requested file operation, then the No branch is taken to step 708 .

In step 708 , the storage management computing device 14 determines whether all necessary components have been processed for the requested file operation. If in step 708 , the storage management computing device 14 determines all necessary components have been processed for the requested file operation, then the Yes branch is taken to step 716 as described earlier. If in step 708 , the storage management computing device 14 determines all necessary components have not been processed for the requested file operation, then the No branch is taken to step 710 .

Referring to FIG. 10 , an exemplary method for processing all necessary components in step 710 in FIG. 9 is illustrated. In step 718 , the storage management computing device 14 conducts a lookup of the entry for the requested file operation on all shares.

In step 720 , the storage management computing device 14 determines whether the entry exists in any share. If in step 720 , the storage management computing device 14 determines the entry does not exist on any share, then the No branch is taken back to step 712 in FIG. 9 . If back in step 720 in FIG. 10 , the storage management computing device 14 determines the entry does exist on one or more shares, then the Yes branch is taken to step 722 .

In step 722 , the storage management computing device 14 determines whether the entry exists in metadata for all shares the entry was found on. If in step 722 , the storage management computing device 14 determines the entry exists in metadata for all shares the entry was found on, then the Yes branch is taken back to step 712 in FIG. 9 . If in step 722 , the storage management computing device 14 determines the entry does not exist in metadata for all shares the entry was found on, then the No branch is taken back to step 724 . In step 724 , the storage management computing device 14 imports the file system entry and then proceeds back to step 712 in FIG. 9 .

In step 712 , the storage management computing device 14 determines whether the importation of the entry was successful. If in step 712 , the storage management computing device 14 determines the importation of the entry was not successful then the No branch is taken to step 714 . In step 714 , the storage management computing device 14 generates an error message which is returned to the client device 12 . If in step 712 , the storage management computing device 14 determines the importation of the entry was successful then the Yes branch is taken back to step 708 as described in earlier.

Accordingly, as illustrated and described herein this technology provides a number of advantages including providing more effective and efficient methods and devices for seamless storage importing of file systems. With this technology, file system operations are permitted during the storage import process. As a result, clients do not lose access to view or modify their files during the storage import process which could potentially take anywhere from hours to days/weeks. Additionally, while permitting files system operations during storage import, this technology still ensures that all imported file system and metadata entries are accurate.

Having thus described the basic concept of the invention, it will be rather apparent to those skilled in the art that the foregoing detailed disclosure is intended to be presented by way of example only, and is not limiting. Various alterations, improvements, and modifications will occur and are intended to those skilled in the art, though not expressly stated herein. These alterations, improvements, and modifications are intended to be suggested hereby, and are within the spirit and scope of the invention. Additionally, the recited order of processing elements or sequences, or the use of numbers, letters, or other designations therefore, is not intended to limit the claimed processes to any order except as may be specified in the claims. Accordingly, the invention is limited only by the following claims and equivalents thereto.