Method of multi-frame image noise reduction

This application claims the priority benefit of Taiwan application serial no. 100114933, filed on Apr. 28, 2011. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.

1. Field of the Invention

The invention generally relates to an image noise reduction method, and more particularly, to a method of multi-frame image noise reduction.

2. Description of Related Art

Usually, image noise can be primarily divided into impulse noise and Gaussian noise. Gaussian noise is usually processed with finite impulse response method (FIR method), i.e., convolution-based method. In terms of processing impulse noise, median filter method is a more effective way. However, the most problematic issue of the median filter method rests in it easily destroys integrity of an edge and also easily makes the edge look indistinct.

The conventional median filter method is applied in three-dimensional noise reduction (3D NR) and has roughly following two schemes. One of the schemes is to set a window with a certain size, followed by performing median filter operation on the pixels in each window of all temporal frames. Another one is to set a window with a certain size, followed by weighting current pixels and then, performing median filter operation on the pixels in each window of all temporal frames. The above-mentioned schemes that performing median filter operation on the pixels in each window of all temporal frames without any judgement on edges would destroy the detail of an edge, and at the boundary between a smooth region and an edge, it possibly occur that the detail of an edge replaces a part of a smooth region; that is, a weird situation may occur that the edge pixels present themselves on a smooth region. Although the above-mentioned second scheme increases the weights of the current pixels, but under the situation without judgement on edge direction and the presence of edge, the risks of destroying the detail and wrongly replacing a part of a smooth region with the edge pixels are still unavoidable.

Accordingly, the invention is directed to a method of multi-frame image noise reduction capable of keeping the detail of an edge in a frame.

The invention provides a method of multi-frame image noise reduction suitable for an image-capturing device. The method includes following steps. First, a current frame and a plurality of reference frames are obtained. Next, a mask and a target point in the mask are defined. Then, whether or not the pixel of the target point of the current frame is located on an edge is judged according to an edge map of the current frame. When the pixel of the target point is located on the edge, the pixels in the reference frames located on the edge are used to calculate a replacement result; when the pixel of the target point is not located on the edge, the pixels in the reference frames surrounding the target point are used to calculate a replacement result, in which the pixels surrounding the target point are located at a same side of the edge. After that, a pixel corresponding to the position of the target point in an output image is generated according to the replacement result. Further, the mask is moved and the method goes back to the step of judging whether or not the pixel of the target point of the current frame is located on the edge.

In an embodiment of the present invention, the step of calculating a replacement result by using the pixels in the reference frames located on the edge includes following steps: first, determining a representing direction of the edge among a plurality of edge directions according to the contrast of the pixel of the target point over the adjacent pixels; next, respectively calculating a median value of the pixels of the current frame on the representing direction and the pixels of each of the reference frames on the representing direction; then, determining the replacement result according to the median values.

In an embodiment of the present invention, the method of multi-frame image noise reduction further includes determining in auxiliary way the representing direction according to the contrast of a plurality of background pixels in the mask over the pixels on the edge directions in the mask.

In an embodiment of the present invention, the step of determining the replacement result according to the median values includes following steps: first, respectively calculating a background average value of the pixels outside the representing direction of the current frame and the reference frames; next, respectively calculating a difference value between the median values and the background average values; then, respectively calculating a weighting value of the current frame and the reference frames according to the difference values; after that, weighting the median values according to the weighting values so as to determine the replacement result.

In an embodiment of the present invention, the difference values and the weighting values have positive-going relationship with each other.

In an embodiment of the present invention, the step of determining the replacement result according to the median values includes finding out the median of the median values to determine the replacement result.

In an embodiment of the present invention, the step of respectively calculating the median values includes following steps: first, finding out a plurality of effective points on the representing direction according to the edge map; next, respectively calculating the pixels at the effective points of the current frame and the reference frames to serve as the median values.

In an embodiment of the present invention, the step of calculating the replacement result by using the pixels in the reference frames surrounding the target point includes following steps: first, respectively calculating a median value of the pixels surrounding the target point in the reference frames; next, determining the replacement result according to the median values.

In an embodiment of the present invention, the step of determining the replacement result according to the median values includes averaging the median values to determine the replacement result.

In an embodiment of the present invention, the method of multi-frame image noise reduction further includes following steps: first, judging whether or not the pixels of the target point of the reference frames are located on the edge according to the edge maps of the frames; then, discarding the frame if the pixel of the target point of a frame among the reference frames is not located on the edge.

Based on the above depiction, the invention can effectively remove the impulse noise on an edge and the noise of the smooth region and can still keep the detail of an edge.

Other objectives, features and advantages of the present invention will be further understood from the further technological features disclosed by the embodiments of the present invention wherein there are shown and described preferred embodiments of this invention, simply by way of illustration of modes best suited to carry out the invention.

The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

FIG. 1 is a flowchart diagram of the method of multi-frame image noise reduction according to an embodiment of the invention.

FIG. 2 is a diagram of a plurality of frames according to an embodiment of the invention.

FIG. 3 is a diagram showing target points are located on the edges.

FIG. 4 is a diagram showing a target point is not located on the edge.

FIG. 5 is a flowchart diagram of the method of multi-frame image noise reduction according to another embodiment of the invention.

FIG. 6 is a diagram showing target points are located on local edges.

FIG. 7 is a diagram showing a plurality of edge directions.

FIG. 8 is a diagram of mask templates respectively corresponding to the edge directions of FIG. 7 .

FIG. 1 is a flowchart diagram of the method of multi-frame image noise reduction according to an embodiment of the invention. In the embodiment, the flowchart of FIG. 1 is suitable for image-capturing devices such as digital camera or digital video camera (not shown). FIG. 2 is a diagram of a plurality of frames according to an embodiment of the invention. Referring to FIGS. 1 and 2 , first, step S 110 is performed that a current frame F 3 and a plurality of reference frames F 1 , F 2 , F 4 and F 5 are obtained. Next, step S 120 is performed that a mask M and a target point T in the mask M are defined, in which the mask M has a size, for example, 5×5 and the target point T can be the center point of the mask M, which the invention is not limited to. In addition, in the current frame F 3 and the reference frames F 1 , F 2 , F 4 and F 5 , the mask M can mark out the same corresponding positions of the five frames F 1 -F 5 after motion compensations.

Then, step S 130 is performed that it is judged whether or not the pixel of the target point T of the current frame F 3 is located on an edge according to the edge map (not shown) of the current frame F 3 . FIG. 3 is a diagram showing target points are located on the edges. Referring to FIG. 3 , the edges herein respectively occupy, for example, five pixels of the centre part of the mask M. When the pixel of the target point T of the current frame F 3 is located on an edge, step S 140 is performed that the pixels in the reference frames F 1 , F 2 , F 4 and F 5 located on the edges are used to calculate a replacement result, that is, the five pixels of the centre part respectively in the reference frames F 1 , F 2 , F 4 and F 5 are used to calculate a replacement result. Then, step S 150 is performed that a pixel corresponding to the position of the target point T in an output image is generated according to the replacement result. Further, step S 170 is performed that the mask M is moved and the method goes back to step S 130 .

FIG. 4 is a diagram showing a target point is not located on the edge. Referring to FIG. 4 , contrary to step S 140 , when the pixel of the target point T of the current frame F 3 is not located on the edge E, step S 160 is performed that the pixels in the reference frames F 1 , F 2 , F 4 surrounding the target point T are used to calculate a replacement result, in which the pixels surrounding the target point T are located at a same side P 2 of the edge E. That is to say the pixels located at the other side of the edge are not used so as to avoid a wrong replacement result.

Then, step S 150 is performed that a pixel corresponding to the position of the target point T in an output image is generated according to the replacement result. Further, step S 170 is performed that the mask is moved and the method goes back to step S 130 . It should be noted that when the target point T of the mask M in the current frame F 3 is judged out to belong to the edge, only the pixels on the edge are used to compensate the pixel of the target point T; when the target point T of the mask M in the current frame F 3 is judged out not to belong to the edge, only the pixels surrounding the target point T are used to compensate the pixel of the target point T. In this way, the invention not only can reduce image noise, but also can avoid destroying the edge and the surrounding pixels adjacent to the edge. As a result, the detail of the edge is kept.

FIG. 5 is a flowchart diagram of the method of multi-frame image noise reduction according to another embodiment of the invention. For simplicity, the flowchart of FIG. 5 is depicted in association with the multiple frames of FIG. 2 . First, step S 210 is performed that a current frame F 3 and a plurality of reference frames F 1 , F 2 , F 4 and F 5 are obtained. Next, step S 220 is performed that a mask M and a target point T in the mask M are defined, in which the mask M has a size, for example, 5×5 and the target point T can be the centre point of the mask M, which the invention is not limited to. In addition, in the current frame F 3 and the reference frames F 1 , F 2 , F 4 and F 5 , the mask M after motion compensations can be located at different positions.

Then, step S 230 is performed that it is judged whether or not the pixel of the target point T of the current frame F 3 is located on an edge according to the edge map (not shown) of the current frame F 3 . FIG. 6 is a diagram showing target points are located on local edges. In comparison with FIG. 3 where the edge occupies all the five pixels of the centre part of the mask M, the edge of FIG. 6 occupies however, for example, three pixels of the centre part of the mask M only. In the embodiment, when the pixel of the target point T of the current frame F 3 is located on the edge, it can be firstly affirmed whether or not the motion compensations of the reference frames F 1 , F 2 , F 4 and F 5 meet the requirement. In more details, step S 280 is performed that it is judged that whether or not the pixels of the target point T of the reference frames F 1 , F 2 , F 4 and F 5 are located at the edges according to the edge maps of the reference frames F 1 , F 2 , F 4 and F 5 . If the pixel of the target point T of a frame among the reference frames F 1 , F 2 , F 4 and F 5 is not located on the edge, step S 290 is performed that the frame where the pixel of the target point T is not located on the edge is discarded. For example, in FIG. 6 , the reference frame F 5 where the pixel of the target point T is not located on the edge, so that the frame F 5 is discarded. In this way, it can ensure each of the reference frames F 1 , F 2 , F 4 and F 5 has a relatively correct motion compensation to advance the effect of the following noise reduction operation.

Then step S 240 is performed that a replacement result is calculated by using the pixels on the edges in the reference frames F 1 , F 2 and F 4 . In more details, step S 240 includes a plurality of sub steps S 242 -S 248 . FIG. 7 is a diagram showing a plurality of edge directions. Referring to FIG. 7 , first, step S 242 is performed that a representing direction of the edge is determined among a plurality of edge directions M 1 -M 8 according to the contrast of the pixel of the target point T over the adjacent pixels. FIG. 8 is a diagram of mask templates respectively corresponding to the edge directions of FIG. 7 . Referring to FIG. 8 , eight mask templates are respectively corresponding to the edge directions M 1 -M 8 . After the pixels of the mask M are respectively brought into the eight mask templates, eight gradient values can be calculated, in which the edge direction corresponding to the mask template with the minimum gradient value is the representing direction.

In terms of the horizontal edges of FIG. 3 or FIG. 6 , the differences between the pixels on the horizontal direction in the masks M are minimal, so that the minimum gradient value can be calculated out for the mask template corresponding to the edge direction M 1 . At the time, the contrast of the edge in the mask M on the edge direction M 1 is minimal and the edge direction M 1 is determined as the representing direction.

In order to increase the reliability of searching the representing direction of edges, it can further affirm the contrast of the edge over the background. When the contrast of the pixels on an edge direction over the background pixels in a mask M is larger, the edge direction is more likely the representing direction. Based on the reason, the embodiment further has step S 244 that the representing direction is in auxiliary way determined according to the contrast of a plurality of background pixels over the pixels on the edge directions M 1 -M 8 . In more strict methodology, only an edge direction found out by meeting both the two conditions of steps S 242 and S 244 can serve as one for the following noise reduction processing so as to reduce the possibility of destroying the detail due to noise reduction processing.

For example, the average value of all pixels in a mask M is calculated and the result serves as background average value. Then, the average values of five pixels on each of the edge directions M 1 -M 8 are calculated. After that, the average values of each of the edge directions M 1 -M 8 is respectively compared with the background average value so as to find out an edge direction with a larger difference to determine in auxiliary way the representing direction. In another embodiment, the representing value of the background pixels can be the average value of eight pixels outside the eight edge directions M 1 -M 8 . Moreover, a threshold value can be set to determine which difference between the edge and the background is large enough to be considered.

Further step S 246 is performed that median values of the pixels of the current frame F 3 and the reference frames F 1 , F 2 and F 4 on the representing direction M 1 . In more details, first step S 246 a is performed that a plurality of effective points on the representing direction M 1 are found out according to the edge map. For example, the number of the effective points in FIG. 3 is five and the number in FIG. 6 is three. Next, step S 246 b is performed that the pixels corresponding to the effective points of the current frame F 3 and the reference frames F 1 , F 2 and F 4 are respectively calculated and the results serve as the directional median filtering value of each the frame. Taking FIG. 6 as an example, each the result is the median value of three pixels on the horizontal direction, in which the median values of the current frame F 3 and the reference frames F 1 , F 2 and F 4 are respectively notated as Median 3 , Median 1 , Median 2 and Median 4 .

Then, step S 248 is performed that the replacement result is determined according to the median values Median 1 , Median 2 , Median 3 and Median 4 . In more details, first, step S 248 a is performed that a background average value of the pixels outside the representing direction M 1 of each of the current frame F 3 and the reference frames F 1 , F 2 and F 4 is respectively calculated. Taking FIG. 6 as an example, the background average value is calculated by using 23 pixels among all the pixels except for the three pixels on the representing direction M 1 . The background average values of the current frame F 3 and the reference frames F 1 , F 2 and F 4 are respectively notated as Mean 3 , Mean 1 , Mean 2 and Mean 4 . For example, in FIG. 3 , 20 pixels in total except for the five pixels on the representing direction M 1 are used to calculate the background average value of a frame, and analogically for the others.

After that, step S 248 b is performed that a difference value between the median values Median 1 , Median 2 , Median 3 and Median 4 and the background average values Mean 1 , Mean 2 , Mean 3 and Mean 4 are respectively calculated, and the difference values of the current frame F 3 and the reference frames F 1 , F 2 and F 4 are notated as Diff 3 , Diff 1 , Diff 2 and Diff 4 :

Diff1=abs(Mean1−Median1);

Diff2=abs(Mean2−Median2);

Diff3=abs(Mean3−Median3);

Diff4=abs(Mean4−Median4).

Further, step S 248 c is performed that a weighting value of each of the current frame F 3 and the reference frames F 1 , F 2 and F 4 is respectively calculated according to the difference values Diff 3 , Diff 1 , Diff 2 and Diff 4 . An edge with a higher contrast and more distinct look should be assigned with a larger weighting value, so that the larger the difference value, the higher the weight is during weighting average operation; that is, the difference values and the weighting values have positive proportional relationship with each other. The weighting values of the current frame F 3 and the reference frames F 1 , F 2 and F 4 are notated as W 3 , W 1 , W 2 and W 4 .

Further, step S 248 d is performed that the median values Median 1 , Median 2 , Median 3 and Median 4 are weighted with the weighting values W 1 , W 2 , W 3 and W 4 so as to determine the replacement result. For example, the replacement result=(W 1 *Median 1 +W 2 *Median 2 +W 3 *Median 3 +W 4 *Median 4 /(W 1 +W 2 +W 3 +W 4 ). In another embodiment, if more-distinct pixels are desired but without the weighting processing, then the replacement result can be determined by the median of the median values Median 3 , Median 1 , Median 2 and Median 4 . For example, the replacement result=Median(Median 1 , Median 2 , Median 3 , Median 4 ).

Further, step S 250 is performed that a pixel corresponding to the position of the target point T in an output image is generated according to the replacement result. For example, a blank image is firstly generated, followed by generating a pixel corresponding to the position of the target point T in the blank image according to the replacement result. Further, step S 270 is performed that the mask M is moved and the method goes back to step S 230 .

Referring to FIG. 4 again, in contrary, when the pixel of the target point T of the current frame F 3 is not located on the edge, step S 260 is performed that a replacement result is calculated by using the pixels surrounding the target point T in the reference frames F 1 , F 2 , F 4 and F 5 , in which the pixels surrounding the target point T are located at a same side of the edge. In more details, a median value of the pixels surrounding the target point T in each of the reference frames F 1 , F 2 , F 4 and F 5 is respectively calculated. Then, the replacement result is determined according to the calculated median values of the current frame F 3 and the reference frames F 1 , F 2 , F 4 and F 5 . That is to say, if the pulse noise to be processed is not on the edge map, a general median filter operation is performed. It should be noted that, during sequencing, only the pixels located at a same side as the target point T are considered, but the pixels located at the other side of the edge E are not adopted so as to avoid a wrong replacement result. At the time the replacement result is calculated directly by using a same weighting value assigned to every median value, i.e., the replacement result=(Median 1 +Median 2 +Median 3 +Median 4 +Median 5 )/5.

Further, step S 250 is performed that a pixel corresponding to the position of the target point T in an output image is generated according to the replacement result. Further, step S 270 is performed that the mask M is moved and the method goes back to step S 230 .

In summary, the invention is able to effectively eliminate the impulse noise on the edge and the smooth region. In addition, the invention is different from the conventional noise reduction method without judging the edge profile. The invention uses different strategy for dealing with the edge and the smooth region, which can achieve a better noise reduction result and keep the true detail instead of obtaining an indistinct image.

The foregoing description of the preferred embodiments of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form or to exemplary embodiments disclosed. Accordingly, the foregoing description should be regarded as illustrative rather than restrictive. Obviously, many modifications and variations will be apparent to practitioners skilled in this art. The embodiments are chosen and described in order to best explain the principles of the invention and its best mode practical application, thereby to enable persons skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use or implementation contemplated. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents in which all terms are meant in their broadest reasonable sense unless otherwise indicated. Therefore, the term “the invention”, “the invention” or the like does not necessarily limit the claim scope to a specific embodiment, and the reference to particularly preferred exemplary embodiments of the invention does not imply a limitation on the invention, and no such limitation is to be inferred. The invention is limited only by the spirit and scope of the appended claims. The abstract of the disclosure is provided to comply with the rules requiring an abstract, which will allow a searcher to quickly ascertain the subject matter of the technical disclosure of any patent issued from this disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Any advantages and benefits described may not apply to all embodiments of the invention. It should be appreciated that variations may be made in the embodiments described by persons skilled in the art without departing from the scope of the invention as defined by the following claims. Moreover, no element and component in the present disclosure is intended to be dedicated to the public regardless of whether the element or component is explicitly recited in the following claims.