Method and apparatus for encoding erasure code for storing data

This application is a National Phase Application of PCT International Application No. PCT/KR2018/008427, which was filed on Jul. 25, 2018, and which claims priority from Korean Patent Application No. 10-2017-0096303 filed with the Korean Intellectual Property Office on Jul. 28, 2017. The disclosures of the above patent applications are incorporated herein by reference in their entirety.

1. Technical Field

The present invention relates to a method and an apparatus for encoding erasure code for storing data.

2. Description of the Related Art

As big data becomes ever more important, there is a growing need for an efficient system for storing and managing data. A storage system for handling big data must be able to provide expandability for adapting to data sizes in a flexible manner as well as ensure safe storage of the data. As such, the concept of utilizing error correction code for storing data safely is increasingly gaining attention.

To resolve the problems found in the related art above, an aspect of the invention provides a method and an apparatus for encoding erasure code that are capable of storing and recovering data in a safe and efficient manner.

To achieve the objective above, an embodiment of the invention provides a method of encoding erasure code for storing data that includes: (a) generating a first local parity group that is composed of two or more local parity nodes for data nodes; (b) generating at least one global parity node for the data nodes; (c) generating at least one second local parity group that is composed of two or more local parity nodes for the data nodes; and (d) storing the data nodes, the first local parity group, the second local parity groups, and the global parity nodes.

The local parity nodes of the second local parity groups and the local parity nodes of the first local parity group may be generated without overlap.

The same number of data nodes may be used for generating each local parity node.

The number of second local parity groups may be smaller than the number of data nodes used for generating a local parity node of the first local parity group.

Another embodiment of the invention provides a method of encoding erasure code for storing data that includes: (a) generating a first local parity group that is composed of two or more local parity nodes for data nodes; (b) generating a multiple number of global parity nodes for the data nodes; (c) storing the data nodes, the first local parity group, and the global parity nodes; (d) calculating an average erasure probability by monitoring the stored nodes; (e) converting at least one global parity node from among the generated multiple number of global parity nodes into a second local parity group composed of two or more local parity nodes for the data nodes if the average erasure probability exceeds a threshold value; and (f) re-storing the data nodes, the first local parity group, the second local parity group, and global parity nodes unconverted in step (e).

The local parity nodes of the second local parity group and the local parity nodes of the first local parity group may be generated without overlap.

The same number of data nodes may be used for generating each local parity node.

The number of second local parity groups may be smaller than the number of data nodes used for generating a local parity node of the first local parity group.

The average erasure probability may be calculated by the equation below:

where P e is the average erasure probability, n is the number of nodes, P i is an average erasure probability of each node, N m,i is a number of periodical monitoring of each node, and f i is the number of node failures of each node occurring during monitoring.

Yet another embodiment of the invention provides an apparatus for encoding erasure code for storing data that includes: a first local parity node generation unit configured to generate a first local parity group that is composed of two or more local parity nodes for data nodes; a global parity node generation unit configured to generate at least one global parity node for the data nodes; a second local parity node generation unit configured to generate a second local parity group that is composed of two or more local parity nodes for data nodes; and a control unit configured to store the data nodes, the first local parity group, the second local parity groups, and the global parity nodes.

The local parity nodes of the second local parity group and the local parity nodes of the first local parity group may be generated without overlap.

The same number of data nodes may be used for generating each local parity node.

The number of second local parity groups may be smaller than the number of data nodes used for generating a local parity node of the first local parity group.

An embodiment of the invention may provide the advantageous effect of allowing safe and efficient storage and recovery.

FIG. 1 is a diagram for explaining an LRC.

FIG. 2A , FIG. 2B , and FIG. 2C are diagrams for describing cases where node failures occur in the LRC illustrated in FIG. 1 .

FIG. 3 is a flowchart illustrating a method of encoding erasure code for storing data according to an embodiment of the invention, shown according to time.

FIG. 4 is a diagram for explaining an E-LRC.

FIG. 5 is a diagram illustrating an apparatus for encoding erasure code for storing data according to an embodiment of the invention.

FIG. 6 is a graph illustrating the probability of recovery for various cases of node failures for LRC and E-LRC.

FIG. 7 is a block diagram illustrating an algorithm for a method of encoding erasure code for storing data according to an embodiment of the invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to particular modes of practice, and it is to be appreciated that all changes, equivalents, and substitutes that do not depart from the spirit and technical scope of the present invention are encompassed in the present invention. In describing the drawings, similar reference numerals are used for similar elements.

While such terms as “first” and “second,” etc., may be used to describe various elements, such elements must not be limited to the above terms. The above terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element without departing from the scope of rights of the present invention, and likewise a second element may be referred to as a first element. Certain embodiments of the present invention are described below in more detail with reference to the accompanying drawings.

In a distributed storage device, information can be lost due to a variety of factors as information stored distributed over multiple nodes is retrieved. An embodiment of the invention presents a method of encoding erasure code for recovering such lost information.

An example of a typically used encoding method for erasure code is the RS (Reed-Solomon) code, which is a type of MDS (maximum distance separable) code. An RS code expressed as (n,k)-RS represents an RS code having a total of n nodes, including a k number of data nodes and an (n−k) number of parity nodes. The RS code, which is an MDS code, can recover an arbitrary (n−k) number of node failures and is characterized in that the average reconstruction cost C is the constant at k. Here, the reconstruction cost refers to the number of other nodes required when recovering a node failure, and the average reconstruction cost refers to the average of reconstruction costs for all cases of arbitrary node failures. Also, the storage overhead S can be expressed as a ratio of the number of data nodes k and the total number of nodes n as s=(n/k).

An embodiment of the invention may basically use an LRC (locally repairable code), which is efficient when recovering a single node failure. That is, when comparing the RS code with the LRC, the LRC may be very efficient when recovering a single node failure rather than recovering multiple node failures. An LRC expressed as (k,l,g)-LRC represents an LRC having a total of (k+l+g) nodes, including a k number of data nodes, an l number of local parity nodes, and a g number of global parity nodes. The number of data nodes used in generating one local parity node may be referred to as locality, where locality L=(k/l).

FIG. 1 shows the structure of an (8,2,2)-LRC. x 0 , x 1 , . . . , x 7 represents the eight data nodes, P L,0 and P L,1 represent the two local parity nodes, and P G,0 and P G,1 represent the two global parity nodes. Also, the local parity node P L,0 may be generated with x 0 , x 1 , x 2 , and x 3 , the local parity node P L,1 may be generated with x 4 , x 5 , x 6 , and x 7 , and the global parity nodes P G,0 , P G,1 may each be generated with all data nodes.

Therefore, the procedure for generating the local parity and global parity nodes of a (k,l,g)-LRC can be expressed by the equation below.

( x 0 ,x 1 , . . . ,x k−1 )× G =( x 0 ,x 1 , . . . ,x k−1 ,p L,0 ,p L,1 , . . . ,p L,l−1 ,p G,0 ,p G,1 , . . . ,p G,g−1 )  [Equation 1]

In Equation 1, x 0 , x 1 , . . . , x k−1 are the k number of data nodes, P L,0 , P L,1 , . . . , P L,l−1 are the l number of local parity nodes, P G,0 , P G,1 , . . . , P G,g−1 are the g number of global parity nodes, and G is a generator matrix of dimension (k×n) having finite field elements.

For instance, the generator matrix G in the (8,2,2)-LRC illustrated in FIG. 1 is as the equation shown below.

In Equation 2, α i represents a finite field element.

FIG. 2A , FIG. 2B , and FIG. 2C are diagrams for describing cases where node failures occur in the LRC illustrated in FIG. 1 .

FIG. 2A shows a case in which a node failure has occurred in a single data node x 1 . In such a case, x 1 can be recovered by using four nodes x 0 , x 2 , x 3 , P L,0 , and the reconstruction cost is 4.

FIG. 2B shows a case in which a node failure has occurred in a local parity node P L,1 . In such a case, P L,1 can be recovered by using four nodes x 4 , x 5 , x 6 , x 7 , and the reconstruction cost is 4.

FIG. 2C shows a case in which a node failure has occurred in a global parity node P G,0 . In such a case, P G,0 can be recovered by using eight nodes x 0 , . . . , x 7 , and the reconstruction cost is 8.

Therefore, the average reconstruction cost for a single node failure of the (8,2,2)-LRC is

Also, an LRC allows a 100% recovery for an (n-k-l) number of node failures and allows probabilistic recovery for an (n-k) number of node failures. For instance, with the (8,2,2)-LRC illustrated in FIG. 1 , 100% recovery is possible for three or fewer node failures, and 85.86% recovery is possible for four node failures.

With the LRC, whereas cases of two or more recoverable node failures may yield an average reconstruction cost C that is constant at k, cases of a single node failure may yield an average reconstruction cost that is lower than k. Therefore, the LRC may be more efficient than the RS code for recovering a single node failure.

FIG. 3 is a flowchart illustrating a method of encoding erasure code for storing data according to an embodiment of the invention, shown according to time.

Referring to FIG. 3 , a method of encoding erasure code for storing data according to an embodiment of the invention can include generating a first local parity group (operation S 110 ), generating global parity nodes (operation S 120 ), storing nodes (operation S 130 ), calculating an average erasure probability (operation S 140 ), generating a second local parity group (operation S 150 ), and re-storing nodes (operation S 160 ).

Generating the first local parity group (operation S 110 ) and generating the global parity nodes (operation S 120 ) can be performed in the same manner as for the procedures associated with generating the local parity nodes and global parity nodes of the LRC described above.

The first local parity group may be a group composed of local parity nodes in an LRC. Therefore, the first local parity group and the global parity nodes can be generated by using Equation 1.

Storing the nodes (operation S 130 ) may be an operation of storing data by using the LRC generated in operation S 110 and operation S 120 . The nodes stored by using the LRC may be monitored in operation S 140 .

Calculating the average erasure probability (operation S 140 ) may be an operation of monitoring the nodes stored in operation S 130 and calculating the average erasure probability of the nodes.

The average erasure probability P e is the average of the erasure probability values P i of n nodes and can be calculated by the equation below.

In Equation 3, N m,i is the number of periodical monitoring of each node, and f i is the number of node failures of each node that occurred during monitoring.

A method of encoding erasure code for storing data according to an embodiment of the invention may use an E-LRC instead of the conventional LRC when the average erasure probability P e , of the nodes exceeds a threshold P e,th . Here, the E-LRC is a novel method of encoding erasure code presented by the present invention.

In the E-LRC, a portion of the global parity nodes of the LRC can be generated to be divided into local parities. Although the E-LRC provides a larger storage overhead S compared to the LRC, the reconstruction costs for a single node failure and multiple node failures may be reduced compared to the LRC, and the number of cases of node failures that can be recovered is also higher.

In a conventional (k,l,g)-LRC, if an m number of global parity nodes are each divided into an l 0 , l 1 , . . . , l m−1 number of local parity nodes, then a (k, l, l 0 , l 1 , . . . , l m−1 , g-m)E-LRC can be generated. Here, sets having the l, l 0 , l 1 , . . . , l m−1 local parity nodes may be defined as local parity groups. In particular, in the present invention, the group having an 1 number of local parity nodes composed of the local parity nodes of the conventional (k,l,g)-LRC may be referred to as the first local parity group, and the groups having the other l 0 , l 1 , . . . , l m−1 local parity nodes may be referred to as second local parity groups. The parameters of the (k, l, l 0 , l 1 , . . . , l m−1 , g-m)E-LRC mean that the number of data nodes is k, the numbers of local parity nodes of the local parity groups are l, l 0 , l 1 , . . . , l m−1 , respectively, and the number of global parity nodes is (g-m).

In the (k, l, l 0 , l 1 , . . . , l m−1 , g-m)E-LRC, the number of second local parity groups m is smaller than the locality L, which is the number of data nodes used in generating one local parity node. Therefore, the local parity nodes of all local parity groups can be generated without overlap. Also, the number of data nodes used for generating all of the local parity nodes can be the same as L.

That is, the first local parity nodes P L 0, 0 , P L 1, 0 , . . . , P L m−1, 0 of the first local parity group may be generated with an L number of data nodes x 1 , x i+1 , . . . , x i+L−1 , where i=0. Also, the first local parity nodes of each second local parity group may be generated with an L number of data nodes x 1 , x i+1 , . . . , x i+L−1 , where i is an element of {1, 2, 3, . . . , L−1} without overlapping.

FIG. 4 is a diagram for explaining an E-LRC.

The E-LRC illustrated in FIG. 4 represents an (8,2,2,1)E-LRC generated by dividing a global parity node in the (8,2,1)-LRC illustrated in FIG. 1 .

Referring to FIG. 1 and FIG. 4 , it can be seen that in the (8,2,2,1)E-LRC, m=1, L=4, k=8, l=2, l 0 =2, and g=1. That is, it can be seen that the second local parity group of the (8,2,2,1)E-LRC was generated by dividing the global parity node P G,1 of the (8,2,1)-LRC.

Therefore, the procedures for generating the first local parity group, global parity nodes, and second local parity group for the (8,2,2,1)E-LRC illustrated in FIG. 4 can be expressed as the equations shown below.

p L,0 =x 0 +x 1 +x 2 +x 3 ,

p L,1 =x 4 +x 5 +x 6 +x 7 ,

p L 0, 0 =α 2 2 x 2 +α 3 2 x 3 +α 4 2 x 4 +α 5 2 x 5 ,

p L 0, 1 =α 0 2 x 0 +α 1 2 x 1 +α 6 2 x 6 +α 7 2 x 7 ,

p G,0 =α 0 x 0 +α 1 x 1 + . . . +α 7 x 7 ,  [Equation 4]

In Equation 4, α i represents a finite field element.

Generating a second local parity group (operation S 150 ) may be an operation of generating a second local parity group by using the E-LRC described above.

As described above, the second local parity group can be generated by dividing a global parity node of operation S 120 . The number of second local parity groups may be smaller than the number of data nodes L used for generating a local parity node, and the number of data nodes used for generating each local parity node may be the same. Also, the local parity nodes of the second local parity group and the local parity nodes of the first local parity group may be generated without overlapping one another.

Re-storing the nodes (operation S 160 ) may be an operation of storing the nodes generated by using the E-LRC.

The global parity nodes that were not used in the generating of second local parity groups in operation S 150 from among the global parity nodes generated in operation S 120 , the data nodes, the local parity nodes of the first local parity group generated in operation S 110 , and the local parity nodes of the second local parity groups generated in operation S 150 can be stored.

FIG. 5 is a diagram illustrating an apparatus for encoding erasure code for storing data according to an embodiment of the invention.

Referring to FIG. 5 , an apparatus for encoding erasure code for storing data according to an embodiment of the invention can include a first local parity node generation unit 210 , a global parity node generation unit 220 , a second local parity node generation unit 230 , and a control unit 240 .

The apparatus for encoding erasure code for storing data according to an embodiment of the invention may store data nodes, local parity nodes, and global parity nodes according to the E-LRC procedures described above.

The first local parity node generation unit 210 can generate a first local parity group in accordance with operation S 110 .

The global parity node generation unit 220 can generate global parity nodes in accordance with operation S 120 .

The second local parity node generation unit 230 can generate a second local parity group in accordance with operation S 150 .

The control unit 240 can store the nodes generated by the first local parity node generation unit 210 , global parity node generation unit 220 , and second local parity node generation unit 230 in accordance with operation S 160 .

FIG. 6 is a graph illustrating the probability of recovery for various cases of node failures for LRC and E-LRC.

Referring to FIG. 6 , the LRC and the E-LRC both provide a recovery probability of 100% up to three node failures, but when the number of node failures reaches four, the recovery probability of the LRC is 86%, whereas the recovery probability of the E-LRC is superior at 97%. Also, whereas the LRC is incapable of providing recovery when the number of node failures is five, it can be seen that the E-LRC provides a recovery probability of 80%.

Thus, an embodiment of the invention can calculate the average erasure probability of the nodes to employ the LRC, which entails a smaller overhead, in sections where the number of node failures is three or fewer and employ the E-LRC, which entails a higher recovery probability, in sections where the number of node failures is four or higher, thereby enabling data storage in an efficient and stable manner.

FIG. 7 is a block diagram illustrating an algorithm for a method of encoding erasure code for storing data according to an embodiment of the invention.

Referring to FIG. 7 , an embodiment of the invention may first monitor each node by using the LRC, which entails a smaller overhead, and calculate the average erasure probability of the nodes. If node failures occur more frequently so that the average erasure probability exceeds the threshold, an embodiment of the invention may use E-LRC, which entails a higher recovery probability, instead of the LRC. As an embodiment of the invention may thus apply the LRC and the E-LRC adaptively according to circumstances, data can be stored in a stable and efficient manner.

While the spirit of the invention has been described in detail with reference to particular embodiments, the embodiments are for illustrative purposes only and do not limit the invention. It is to be appreciated that those skilled in the art can change or modify the embodiments without departing from the scope and spirit of the invention. As such, the spirit of the present invention must not be interpreted as being limited only to the embodiments described above but as encompassing all equivalents and variations of the scope of claims set forth below.