Migrating data using multiple assets

Computer data is vital to today's organizations and a significant part of protection against disasters is focused on data protection. As solid-state memory has advanced to the point where cost of storage has become a relatively insignificant factor, organizations can afford to operate with systems that store and process terabytes of data.

Conventional data protection systems include tape backup drives, for storing organizational production site data on a periodic basis. Another conventional data protection system uses data replication, by creating a copy of production site data of an organization on a secondary backup storage system, and updating the backup with changes. The backup storage system may be situated in the same physical location as the production storage system, or in a physically remote location. Data replication systems generally operate either at the application level, at the file system level, or at the data block level.

In one aspect, a method includes migrating a first device on a first array to a second device on a second storage array. The migrating includes instructing an asset to scan the first array and the second array for the first device and the second device, reading a work buffer on the second array, sending a request to the work buffer for a chunk of the first device to copy to the second device, reading assigned chunk from the first device, copying the assigned chunk to the second device and notifying the work buffer that the copying of the assigned chunk was successful.

In another aspect, an apparatus, includes electronic hardware circuitry configured to migrate a first device on a first array to a second device on a second storage array. The circuitry configured to migrate the first device on the first array to the second device on the second storage array includes circuitry configured to instruct an asset to scan the first array and the second array for the first device and the second device, read a work buffer on the second array, send a request to the work buffer for a chunk of the first device to copy to the second device, read assigned chunk from the first device, copy the assigned chunk to the second device and notify the work buffer that the copying of the assigned chunk was successful.

In a further example, an article includes a non-transitory computer-readable medium that stores computer-executable instructions to migrate a first device on a first array to a second device on a second storage array. The instructions cause a machine to instruct assets to scan the first array and the second array for the first device and the second device, read a work buffer on the second array, send a request to the work buffer for a chunk of the first device to copy to the second device, read assigned chunk from the first device, copy the assigned chunk to the second device and notify the work buffer that the copying of the assigned chunk was successful.

FIG. 1 is a block diagram of an example of a system to migrate data from one device to another device.

FIG. 2 is an illustration of an example of storage device to migrate and an example, of a work buffer used to migrate the storage device.

FIG. 3 is a flowchart of an example of a process to migrate a device.

FIG. 4 is a flowchart of an example of a process to assign assets for migration based on asset performance.

FIG. 5 is an illustration of another example of a work buffer used to migrate the storage device that records asset performance.

FIG. 6 is a computer on which any of the processes of FIGS. 3 and 4 may be implemented.

Described herein are techniques to migrate data from one device to another device using multiple assets. In particular, instead of having one asset involved in the migration, other assets having bandwidth also contribute to the migration. As will be also described herein the selection of assets may be based on machine learning (e.g., based on the performance of each asset). While the description herein describes migrating one device to another device using multiple hosts, one of ordinary skill in the art can apply these techniques to other assets such as storage arrays, virtual hosts, virtual storage (e.g., EMC® VPLEX®), and so forth.

Referring to FIG. 1 , a system 100 includes hosts 102 a - 102 c and arrays 104 a , 104 b . Each of the hosts 102 a - 102 c includes a respective migration module 120 a - 120 c , each configured to migrate a device to another device. The host 102 a also includes an application 122 . The array 104 a includes storage devices 112 a , 112 c , 112 e and the array 104 b includes storage devices 112 b , 112 d , 112 f and a work buffer 140 .

The host 102 a is configured to see the device 112 a and the device 112 b but not see (e.g., have no knowledge) the devices 112 c - 112 f . The host 102 b is configured to see the devices 112 c and the device 112 d but not the devices 112 a , 112 b , 112 e , 112 f . The host 102 c is configured to see the device 112 e and the device 112 f but not the devices 112 a - 112 d.

As will be described further herein the migration modules 120 a - 120 c will be configured to assist in the migration of data from one device to another device using hosts that initially do not see either device. For example, this description will focus on an example migration of the device 112 a to the device 112 b . Devices 112 a , 112 b are not seen by the hosts 102 b , 102 c (even though hosts 102 b , 102 c can see other devices on the array 104 a , 104 b ), yet the techniques described herein will enable hosts 102 b , 102 c to assist in the migration. One of ordinary skill in the art would recognize that even though device 112 a is on storage array 104 a and the migration target, device 112 b is on a storage array 104 b , the techniques described will work even if the devices (e.g., 112 a , 112 b ) are located on the same storage array.

Referring to FIG. 2 , as will be shown further herein the work buffer 140 will be used. In one example, the work buffer 140 is placed on the storage array of the target device (e.g., device 112 b ) which is storage array 104 b.

The work buffer 140 includes a migration field 142 which indicates that device 1 is being migrated to device 2 . The work buffer 140 also includes asset-chunk fields (e.g., asset-chunk fields 144 a - 144 f ) indicating which chunks of device 1 are being moved by an asset. For example, the device 112 a may be divided into chucks (of storage) (e.g., chunk 0 220 a , chunk 1 220 b , chunk 2 220 c , chunk 3 220 d , chunk 4 220 e and chunk 5 220 f ). In this particular example, the asset-chunk field 144 a indicates that host 2 will move chunk 0 220 a , the asset-chunk field 144 b indicates that host 2 will move chunk 1 220 b , the asset-chunk field 144 c indicates that host 1 will move chunk 2 220 c , the asset-chunk field 144 d indicates that host 3 will move chunk 3 220 d , the asset-chunk field 144 e indicates that host 3 will move chunk 4 220 e and the asset-chunk field 144 f indicates that host 1 will move chunk 5 220 f.

Each asset-chunk field 144 a - 144 f has a corresponding success field 146 a - 146 f respectively. If the migration of a chunk is successful then the corresponding success field 146 a - 146 f will indicate successful transfer (e.g., as shown by the “X” in FIG. 2 ). For example, the success field 146 c is marked with an “X” to indicate that the transfer of chunk 2 by host 1 was successful, the success field 146 d is marked with an “X” to indicate that the transfer of chunk 3 by host 3 was successful, the success field 146 e is marked with an “X” to indicate that the transfer of chunk 4 by host 3 was successful and the success field 146 f is marked with an “X” to indicate that the transfer of chunk 5 by host 1 was successful.

Referring to FIG. 3 , a process 300 is an example of a process to migrate a device (e.g., migrating the device 112 a to the device 112 b ). Process 300 configures migration module at first host to read from device 1 and write to device 2 ( 302 ). For example, the first host, in this example, is the host 102 a . The migration module 120 a is configured to read from the device 112 a and to write to the device 112 b.

Process 300 enables application writes to both device 1 and device 2 . For example, the application 122 may still write to the device 112 a , but the migration module 120 a will also intercept the writes from the application 122 to device 112 a and write them to the device 112 b.

Process 300 reads from device 1 and saves to device 2 ( 308 ). For example, the migration module 120 a reads from the device 112 a and writes to the device 112 b.

Process 300 instructs assets to scan for device 1 and device 2 , but not to mount device 1 and device 2 ( 312 ). For example, the migration module 120 b and the migration module 120 c are instructed to scan the arrays 104 a , 104 b for the devices 112 a , 112 b , but not to mount (access) the devices 112 a , 112 b . In one example, processing block 312 is initiated by a user. In another example, processing block 312 is initiated by the array 104 b when the migration is started by the migration module 120 a.

Process 300 reads work buffer ( 314 ). For example, the migration modules 120 b , 120 c read the work buffer 140 . The migration modules 120 b , 120 c are able to access the work buffer 140 after the devices 112 a , 112 b can be seen. By reading the work buffer 140 , the hosts 102 b , 102 c read the field 142 and recognize that the host 102 a need help in the migration of device 112 a to device 112 b.

Process 300 sends requests to work buffer requesting a chunk of device 1 ( 316 ). For example, the hosts 102 b , 102 c (e.g., using migration modules 120 b , 120 c , respectively) send a SCSI command to the work buffer 114 asking for a chunk of device 112 a to copy to the device 112 b.

Process 300 reads assigned chunk from device 1 ( 324 ) and writes assigned chunk to the device 2 ( 328 ). For example, hosts 102 b , 102 c (e.g., using migration modules 120 b , 120 c , respectively) read the assigned chunk from the device 112 a and writes the assigned chunk to the device 112 b.

Process 300 notifies work buffer copy was successful ( 330 ). For example, hosts 102 b , 102 c (e.g., using migration modules 120 b , 120 c , respectively) notify the work buffer 140 that the assigned chunk was written successfully to the device 112 b . A field in column 146 of the work buffer is changed to get the successful transfer of the assigned chunk to the device 112 b.

Process 300 determines if there are more chunks assigned ( 334 ) and if more chunks are assigned repeats processing blocks 316 , 324 , 328 , 330 and 334 .

If there are no more chunks to assign, process 300 scans work buffer for non-copied chunks ( 338 ). For example, if host 102 b crashes before it can complete the transfer of its assigned chunk to the device 112 b . This failed assigned chunk is a non-copied chunk. The remaining host 102 c (e.g., using migration module 120 c ) reads the work buffer 140 for fields in the column 146 that have not been annotated to indicate that a chunk has been successfully transferred to the device 112 b.

Process 300 reads non-copied chunks from device 1 ( 346 ), writes the non-copied chunks to device 2 ( 348 ) and notifies work buffer that non-copied chunks were copied on to device 2 ( 350 ). For example, the remaining host reads the non-copied chunk from the device 112 a , writes the non-copied chunk to the device 112 b and notifies the work buffer 140 that the non-copied chunk has been successfully delivered to the device 112 b.

Referring to FIGS. 4 and 5 , a process 400 is an example of a process to assign assets for migration of device 1 to device 2 based on asset performance. In one example, if device 112 a is being migrated to device 112 b , then the migration module 120 a of the host 102 a that sees both devices performs process 400 . As shown in FIG. 2 , the asset-chunk fields 144 a - 144 f of the work buffer 140 are completely prefilled, for example, by the migration module 120 . As will be further described herein in conjunction with FIGS. 4 and 5 , the asset-chunk fields may be partially prefilled while the remaining asset-chunk fields are filled in based on the performance of the assets during migration.

Process 400 assigns chunks of device 1 to each of the assets to copy to device 2 ( 402 ) and records parameters for each asset ( 404 ). For example, a work buffer 140 ′ assigns host 102 a chunk 0 (as indicated in an asset-chunk field 144 a ′ of the work buffer 140 ′), host 102 b chunk 1 (as indicated in an asset-chunk field 144 b ′ of the work buffer 140 ′) and host 102 c chunk 2 (as indicated in an asset-chunk field 144 c ′ of the work buffer 140 ′) to copy from the device 112 a to the device 112 b . The work buffer 140 ′ indicates that the copying of chunks 0 , 1 , 2 is successful as indicated in fields 146 a ′- 146 c ′ respectively. The work buffer 140 ′ also records how long it took each asset to copy their respective chunk to device 112 b . For example, it took host 102 a 10 milliseconds (as indicated in a time field 148 a of the work buffer 140 ′), host 102 b 1 millisecond (as indicated in a time field 148 b of the work buffer 140 ′) and host 102 c 50 milliseconds (as indicated in a time field 148 c of the work buffer 140 ′).

Process 400 evaluates performance of each asset ( 408 ) and assigns additional chunks of device 1 to copy based on the performance ( 412 ). Since host 102 c performed the slowest (50 milliseconds), host 102 c will be assigned less chunks than the other hosts 102 a , 102 b and since host 102 b performed the fastest (1 millisecond) host 102 b will be assigned more chunks than the other host 102 a , 102 c . In this example, host 102 a will copy chunks 3 to 5 (three chunks) (as indicated in an asset-chunk field 144 d ′ of the work buffer 140 ′), host 102 b will copy chunks 6 to 12 (7 chunks) (as indicated in an asset-chunk field 144 e ′ of the work buffer 140 ′) and host 102 c will copy chunk 13 (1 chunk) (as indicated in an asset-chunk field 144 f ′ of the work buffer 140 ′).

Process 400 records parameters for each asset ( 414 ). For example, the work buffer 140 ′ indicates that the copying of chunks 3 to 5 , 6 to 12 and 13 is successful as indicated in fields 146 d ′- 146 f ′, respectively. The work buffer 140 records how long it took each host to copy their respective chunk to device 112 b . In this particular example, it took host 102 a 25 milliseconds to copy three chunks (as indicated in a time field 148 d of the work buffer 140 ′), host 102 b 2 seconds to copy seven chunks (as indicated in a time field 148 e of the work buffer 140 ′) and host 102 c 10 milliseconds to copy one chunk 13 (as indicated in a time field 148 f of the work buffer 140 ′) from device 112 a to device 112 b.

Process 400 determines if there are more chunks of device to assign ( 416 ) and if there are more chunks of device 1 to assign repeats processing blocks 408 , 412 , 414 and 416 . If there are no chunks of device 1 left to assign, process 400 ends.

Referring to FIG. 6 , in one example, a computer 600 includes a processor 602 , a volatile memory 604 , a non-volatile memory 606 (e.g., hard disk) and the user interface (UI) 608 (e.g., a graphical user interface, a mouse, a keyboard, a display, touch screen and so forth). The non-volatile memory 606 stores computer instructions 612 , an operating system 616 and data 618 . In one example, the computer instructions 612 are executed by the processor 602 out of volatile memory 604 to perform all or part of the processes described herein (e.g., processes 300 and 400 ).

The processes described herein (e.g., processes 300 and 400 ) are not limited to use with the hardware and software of FIG. 6 ; they may find applicability in any computing or processing environment and with any type of machine or set of machines that is capable of running a computer program. The processes described herein may be implemented in hardware, software, or a combination of the two. The processes described herein may be implemented in computer programs executed on programmable computers/machines that each includes a processor, a non-transitory machine-readable medium or other article of manufacture that is readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Program code may be applied to data entered using an input device to perform any of the processes described herein and to generate output information.

The system may be implemented, at least in part, via a computer program product, (e.g., in a non-transitory machine-readable storage medium such as, for example, a non-transitory computer-readable medium), for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers)). Each such program may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs may be implemented in assembly or machine language. The language may be a compiled or an interpreted language and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. A computer program may be stored on a non-transitory machine-readable medium that is readable by a general or special purpose programmable computer for configuring and operating the computer when the non-transitory machine-readable medium is read by the computer to perform the processes described herein. For example, the processes described herein may also be implemented as a non-transitory machine-readable storage medium, configured with a computer program, where upon execution, instructions in the computer program cause the computer to operate in accordance with the processes. A non-transitory machine-readable medium may include but is not limited to a hard drive, compact disc, flash memory, non-volatile memory, volatile memory, magnetic diskette and so forth but does not include a transitory signal per se.

The processes described herein are not limited to the specific examples described. For example, the processes 300 and 400 are not limited to the specific processing order of FIGS. 3 and 4 . Rather, any of the processing blocks of FIGS. 3 and 4 may be re-ordered, combined or removed, performed in parallel or in serial, as necessary, to achieve the results set forth above.

The processing blocks (for example, in the processes 300 and 400 ) associated with implementing the system may be performed by one or more programmable processors executing one or more computer programs to perform the functions of the system. All or part of the system may be implemented as, special purpose logic circuitry (e.g., an FPGA (field-programmable gate array) and/or an ASIC (application-specific integrated circuit)). All or part of the system may be implemented using electronic hardware circuitry that include electronic devices such as, for example, at least one of a processor, a memory, a programmable logic device or a logic gate.

Elements of different embodiments described herein may be combined to form other embodiments not specifically set forth above. Other embodiments not specifically described herein are also within the scope of the following claims.