Method for demosaicking

1. Field of the Invention

The invention relates to digital image processing, and more particularly to demosaicking a digital image captured by an image sensor overlaid with a color filter array.

2. Description of the Related Art

Most digital cameras or image sensors use a color filter array, such as the well-known Bayer color filter array, to capture a digital image in order to reduce costs. In this way, each pixel in the captured image records a value of only one color. This kind of image is called a mosaicked image. FIG. 1 illustrates a structure of a Bayer pattern, wherein G is green, R is red, and B is blue. To reconstruct a full color image, a mosaicked image is demosaicked to obtain values of two missing colors for each pixel of the mosaicked image.

Some demosaicking processes can be roughly divided into two steps. The first step is to decide the interpolating direction of each pixel. The second step is to interpolate the values of two missing colors for each pixel along the decided interpolating direction. For example, some demosaicking processes use bilinear interpolation to estimate information of two missing colors for each pixel. In bilinear interpolation, the values of the two missing colors are calculated based on an average of the values of the neighboring pixels, in a vertical, horizontal and/or diagonal direction. However, artifacts, such as zipper effect or false color, might occur in the demosaicked image when the values of the two missing colors for each pixel are not interpolated along a proper interpolating direction, reducing image quality. The zipper effect makes a straight edge in the image looks like a zipper. Effects on a part of the artifacts can be diminished by preventing interpolation across the edge.

In view of the above, the invention provides a method for demosaicking a digital image captured by an image sensor overlaid with a color filter array, wherein each of pixels of the digital image has one of a first color, a second color and a third color, and the pixels of the digital image are processed in an order from the upper-left to lower-right of the digital image. The method comprises: if a current-processed pixel is the second color or the third color, implementing a first process to the current-processed pixel to obtain a value of the first color. The first process comprises: generating a first sub-window centered at the current-processed pixel; using a first mask to select a first plurality of prior-processed pixels; calculating a set of direction weightings of the current-processed pixel according to stored direction weightings of the first plurality of prior-processed pixels; determining an interpolating direction of the current-processed pixel according to the set of direction weightings of the current-processed pixel; calculating a value of the first color for the current-processed pixel based on the set of direction weightings of the current-processed pixel and values of the first color of pixels of the first color in the first sub-window; and storing the largest one of the set of direction weightings, the interpolating direction and the value of the first color of the current-processed pixel.

In another embodiment, the invention provides a non-transitory machine-readable storage medium, having an encoded program code, wherein when the program code is loaded into and executed by a machine, the machine implements a method for demosaicking a digital image captured by an image sensor overlaid with a color filter array, wherein each of pixels of the digital image has one of a first color, a second color and a third color, and the pixels of the digital image are processed in an order from the upper-left to lower-right of the digital image, and the method comprises: if a current-processed pixel is the second color or the third color, implementing a first process to the current-processed pixel to obtain a value of the first color. The first process comprises: generating a first sub-window centered at the current-processed pixel; using a first mask to select a first plurality of prior-processed pixels; calculating a set of direction weightings of the current-processed pixel according to stored direction weightings of the first plurality of prior-processed pixels; determining an interpolating direction of the current-processed pixel according to the set of direction weightings of the current-processed pixel; calculating a value of the first color for the current-processed pixel based on the set of direction weightings of the current-processed pixel and values of the first color of pixels of the first color in the first sub-window; and storing the largest one of the set of direction weightings, the interpolating direction and the value of the first color of the current-processed pixel.

In still another embodiment, the invention provides an image signal processor for processing a digital image captured by an image sensor overlaid with a color filter array, wherein each of pixels of the digital image has one of a first color, a second color and a third color, and the pixels of the digital image are processed in an order from the upper-left to lower-right of the digital image. The image signal processor comprises a buffer module, receiving the digital image; a demosaicking module coupled to the buffer module, demosaicking each of the pixels of the digital image to reconstruct values of two other missing colors; a memory module coupled to the demosaicking module; an image processing module coupled to the demosaicking module, receiving values of the first color, the second color and the third color of each of the pixels to implement an image processing process to each of the pixels based on the values. If a current-processed pixel currently processed by the demosaicking module is the second color or the third color, the demosaicking module generates a first sub-window centered at the current-processed pixel, wherein a first mask is used to select a first plurality of prior-processed pixels. Also, a set of direction weightings of the current-processed pixel is calculated according to direction weightings of the first plurality of prior-processed pixels stored at the memory module, and an interpolating direction of the current-processed pixel is determined according to the set of direction weightings of the current-processed pixel and values of the first color of pixels of the first color in the first sub-window. A value of the first color for the current-processed pixel is determined based on the set of direction weightings of the current-processed pixel and values of the first color of pixels of the first color in the first sub-window. Next, the largest one of the set of direction weightings, the interpolating direction and the value of the first color of the current-processed pixel are stored at the memory module, and the value of the first color of the current-processed pixel is transmitted to the image processing module.

A detailed description is given in the following embodiments with reference to the accompanying drawings.

The invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein:

FIG. 1 illustrates a structure of Bayer pattern;

FIG. 2 a ˜ 2 c illustrate a flow chart of a method for demosaicking a digital image in accordance with an embodiment of the invention;

FIG. 3 a illustrates an example of demosaicking a pixel which is not green;

FIG. 3 b illustrates an example of data of prior-processed pixels;

FIG. 3 c illustrates an example of estimating an interpolating direction of a non-processed pixel;

FIG. 3 d - 1 and FIG. 3 d - 2 illustrate an example of data of pixels;

FIG. 4 illustrates an example of demosaicking a green pixel;

FIG. 5 illustrates an image signal processor for demosaicking a digital image in accordance with an embodiment of the invention.

The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims.

FIG. 2 a ˜ 2 c illustrate a flow chart of a method for demosaicking a digital image in accordance with an embodiment of the invention. In this embodiment, the digital image is captured by an image sensor overlaid with a Bayer color filter array. Each pixel of the digital image has only a green value, a red value or a blue value. Note that, a person having ordinary skill in the art can use other types of color filter arrays instead of the Bayer color filter array described herein. The method is implemented by an image signal processor. If the image signal processor uses a line-based scanning scheme to output the processed digital image without more full-color buffers, it may need to generate a sub window centered at a current-processed pixel to implement demosaicking or other image processing in other image pipelines, such as sharpening, smoothing or/and denoising. Pixels of the digital image are processed in an order from the upper-left to lower-right of the digital image, that is the scan scheme is upper-left to lower-right.

In step S 201 , it is determined whether a current-processed pixel is green or not. If the current-processed pixel is not green (step S 201 : No), a first process (steps S 202 ˜S 206 ), a re-estimating process (step S 207 ) and a third process (steps S 208 ˜S 210 ) are implemented to the current-processed pixel to obtain values of two missing colors of the current-processed pixel. If the current-processed pixel is green (step S 201 : Yes), a second process (steps S 211 ˜S 219 ) is implemented to the current-processed pixel to obtain a red value and a blue value of the current-processed pixel. In the first process, a first sub-window SW1 centered at the current-processed pixel is generated at step S 202 , as shown in FIGS. 3 a and 3 c . FIGS. 3 a and 3 c show a part of a digital image captured by an image sensor overlaid with a Bayer color filter array. Therefore, G0˜G12 denote green pixels, R0˜R9 denote red pixels, and B0˜B4 denote blue pixels. In this embodiment the first sub-window is a 5×5 window, but the size of the first sub-window is not limited. In step S 203 , a first mask M1 is used to select prior-processed pixels. For example, if the current-processed pixel is a pixel R4, the selected prior-processed pixels are pixels R0, R1, R2, B0, and B1, since pixels R0, R1, R2, B0, and B1 are processed before the pixel R4 according to the scan scheme. Each of the selected prior-processed pixels is red or blue.

In step S 204 , a set of direction weightings of the current-processed pixel is calculated. In this embodiment, the set of direction weightings comprises a horizontal weighting, a vertical weighting and a flat weighting. The number of the set of direction weightings is not limited to three, for example, in another embodiment, the set of direction weightings can further comprise a northwest-southeast weighting and a northeast-southwest weighting. In demosaicking, “flat” can mean no particular interpolating direction is used when interpolating. Also, an interpolating direction of “flat” can be realized as a combination of all interpolating directions. Note that, in the specification, horizontal is equal to east-west and vertical is equal to north-south. The horizontal weighting W H , the vertical weighting W V and the flat weighting W F of the current-processed pixel are calculated respectively from equations illustrated as following:

wherein R denotes a plurality of pixels considered when calculating direction weightings, W(P) is a weighting of a pixel P of the plurality of pixels, W recorded (P) is a stored direction weighting of the pixel P of the plurality of pixel, P X,H denotes pixels with horizontal interpolating directions in the plurality of pixels, P Y,V denotes pixels with vertical interpolating directions in the plurality of pixels, and the P Z,F denotes pixels with flat interpolating directions in the plurality of pixels. In one example, weightings W of all pixels of the digital image are all 1. FIG. 3 b shows an example of data of prior-processed pixels R0, R1, R2, B0, and B1. The prior-processed pixels R0, R1, R2, B0, and B1 are considered when the set of direction weightings of the current-processed pixel R4 is calculated. That is, the prior-processed pixels R0, R1, R2, B0, and B1 make a contribution to calculation of the current-processed pixel R4. Each pixel of the prior-processed pixels stores an interpolating direction and a value of a corresponding stored direction weighting. In FIG. 3 b , H denotes horizontal, V denotes vertical, and F denotes flat. According to formulas of direction weightings described above, if weightings of the pixels of the digital image are all 1, the horizontal weighting W H , the vertical weighting W V and the flat weighting W F of the current-processed pixel R4 are calculated respectively from equations illustrated as following:

W H ( R 4)= W H ( R 0)+ W H ( R 1),

W V ( R 4)= W V ( R 2)+ W V ( B 1), and

W F ( R 4)= W F ( B 0).

Though in the above example red and blue prior-processed pixels are considered, green prior-processed pixels can also be consider as well in other examples.

In step S 205 , an interpolating direction of the current-processed pixel is determined according to the set of direction weightings of the current-processed pixel. For example, an interpolating direction of the current-processed pixel R4 is a direction of the largest one of W H (R4), W V (R4) and W F (R4). For example, if the vertical weighting W V (R4) is larger than the horizontal weighting W H (R4) and the flat weighting W F (R4), the interpolating of the current-processed pixel R4 is vertical. In this way, the interpolating of the current-processed pixel R4 is voted by neighboring pixels R0, R1, R2, B0, and B1.

If all three direction weightings are equal, a weighting W of each pixel of the plurality of pixels considered when the set of direction weightings of the current-processed pixel is calculated can be set according to a distance between each pixel and the current-processed pixel. That is, the weighting W of a pixel P N is:

wherein P C denotes the current-processed pixel.

For example, according to the formula of the weighting W described above, the horizontal weighting W H , the vertical weighting W V and the flat weighting W F of the current-processed pixel R4 are calculated respectively from equations illustrated as following:

In another example, if the interpolating of the current-processed pixel R4 cannot be determined by vote, the set of direction weightings can be re-calculated according to difference (gradient) in green values of corresponding direction, such as following:

wherein DG H denotes difference in green values of horizontal and DG V denotes difference in green values of vertical. Alternatively, the set of direction weightings can be re-calculated as following:

wherein N is a positive integer to make (W H (R4)+N), (W V (R4)+N) and (W F (R4)+N) not zero.

If all three direction weightings are still equal, the interpolating direction of the current-processed pixel R4 is flat and the stored direction weighting is W F (R4).

In step S 206 , a green value of the current-processed pixel is calculated based on the set of direction weightings of the current-processed pixel and green values of green pixels in the first sub-window SW1. In this embodiment, the green value G of the current-processed pixel is calculated based on the set of direction weightings of the current-processed pixel, a horizontal green gradient G H , a vertical green gradient G V and a flat green gradient G F as the following formula:

For example, the green value G(R4) of the current-processed pixel R4 is calculated as follows:

After step S 206 , the interpolating direction, the direction weighting corresponding to the interpolating direction and the green value of the current-processed pixel are obtained and stored. For example, if the interpolating direction of the current-processed pixel R4 is vertical, the stored data for the current-processed pixel R4 is the vertical interpolating direction, the vertical weighting W V (R4) and the green value G(R4).

In step S 207 , the green value of the current-processed pixel is re-estimated according to not only prior-processed pixel but also non-processed pixels. In the example shown in the FIG. 3 a , when the pixel R4 is processed, pixels G6˜G11, R5˜R8 and B2˜B4 are non-processed pixels. In order to consider non-processed pixels, interpolating directions of the non-processed pixels can be estimated according to at least the current-processed pixel. FIG. 3 c shows an example of estimating an interpolating direction of a non-processed pixel R6. The first mask M1 is used to select pixels R3, R4, R9, B2 and B4 for the pixel R6. Noted, though the mask used in FIG. 3 c is the same as the mask used in FIG. 3 a , the mask used in FIG. 3 c may be different from the mask used in FIG. 3 a in other examples. The interpolating direction of the pixel R6 is voted by pixels R3, R4, R9, B2 and B4, in the same manner of the example for determining the interpolation direction of the pixel R4 as described above in step S 204 and S 205 . Interpolating directions of pixels B2 and B4 and other non-processed pixels can also be voted in the same manner of the example for determining the interpolation direction of the pixel R4 as described above in step S 204 and S 205 . After interpolating directions of the non-processed pixels R5˜R8 and B2˜B4 are estimated, the interpolating direction of the current-processed pixel R4 is re-calculated according to interpolating directions and corresponding direction weightings of the pixels R0˜R8 and B0˜B3 in the first sub window SW1. That is, the current-processed pixel R4 is re-voted by the pixels R0˜R8 and B0˜B3. FIG. 3 d - 1 and FIG. 3 d - 2 show an example of data of pixels R0˜R8 and B0˜B3, wherein the data of the pixel R4 is obtained from steps S 202 to S 205 . If the weightings of the pixels are all 1, the set of direction weightings of the pixel R4 is re-calculated according to the pixels R0˜R8 and B0˜B3 respectively from equations illustrated as following:

W H ( R 4)= W H ( R 0)+ W H ( R 1)+ W H ( R 7)+ W H ( R 8),

W V ( R 4)= W V ( R 2)+ W V ( R 4)+ W V ( R 5)+ W V ( R 6)+ W V ( B 1)+ W V ( B 3), and

W F ( R 4)= W F ( R 3)+ W F ( B 0)+ W F ( B 2).

In another example, if a weighting of each pixel is decided according to a distance between each pixel and the current-processed pixel, the set of direction weightings of the current-processed pixel R4 is re-calculated as follows:

wherein because the Euclidean distance from R4 to R4 itself is 0, the weighting of the pixel R4 is specially designed as any user-defined parameter a. In one example, a is 1.

After the set of direction weightings of the pixel R4 is re-calculated, the interpolating direction of the current-processed pixel R4 is re-determined according to the set of direction weightings of the pixel R4. For example, the interpolating direction of the current-processed pixel R4 is a direction of the largest one of re-calculated W H (R4), W V (R4) and W F (R4). Then the green value of the current-processed pixel R4 is re-determined by the following formula:

wherein G H , G V and G F are the horizontal green gradient, the vertical green gradient and the flat green gradient, respectively, as described above.

After the step S 207 , the re-determined interpolating direction, the re-calculated direction weighting corresponding to the re-determined interpolating direction and the re-calculated green value of the current-processed pixel is re-stored.

In another embodiment, the step S 207 can be eliminated; that is, the green value of the current-processed pixel may not be re-estimated.

After the green value of the current-processed pixel is obtained, the third process (steps S 208 ˜S 210 ) is implemented to the current-processed pixel to obtain a blue value or a red value of the current-processed pixel. In step S 208 , the color of the current-processed pixel is determined. If the current-processed pixel is red, a blue value of the current-processed pixel is calculated in step S 209 . If the current-processed pixel is blue, a red value of the current-processed pixel is calculated in step S 210 .

In step S 209 , the blue value B of the current-processed pixel is calculated based on the green value of the current-processed pixel and stored direction weightings and difference in values between green and blue of blue pixels neighboring the current-processed pixel, wherein each of the blue pixels neighboring the current-processed pixel has the same stored interpolating direction as the current-processed pixel, as the equation illustrated below:

wherein P B denotes pixels in the first sub-window SW1 that have the same interpolating direction N as the current-processed pixel, and N is one of horizontal, vertical and flat. Take FIG. 3 d - 1 and FIG. 3 d - 2 as an example. The stored interpolating direction of the current-processed pixel R4 is vertical. Since interpolating directions of pixels B1 and B3 are also vertical (the interpolating direction of the non-processed pixel B3 can be estimated in the same manner as that described in step S 207 ), the blue value of the current-processed pixel R4 is:

In another example, if there is no pixel in the first sub-window SW1 that has the same interpolating direction N as the current-processed pixel, the blue value B of the current-processed pixel can be calculated as following:

wherein P B′ denotes all blue pixels in the first sub-window SW1 and Num(P B′ ) denotes the number of all blue pixels in the first sub-window SW1. For example, assume the stored interpolating direction of the current-processed pixel R4 is horizontal, then the blue value of the current-processed pixel R4 is:

If the current-processed pixel is blue, the red value of the current-processed pixel is calculated in step S 210 . The algorithm of calculating the red value of the blue current-processed pixel is similar to calculating the blue value of the red current-processed pixel in step S 209 described above, since a Bayer picture with a red pixel that is centered is symmetrical to a Bayer picture with a blue pixel that is centered.

If the current-processed pixel is green, a second process (steps S 211 ˜S 219 ) is implemented to the current-processed pixel to obtain a red value and a blue value of the current-processed pixel. In step S 211 , a second sub-window SW2 centered at the current-processed pixel is generated, as shown in FIG. 4 . FIG. 4 shows a part of a digital image captured by an image sensor overlaid with a Bayer color filter array. In this embodiment the second sub-window is a 3×3 window, but the size of the second sub-window is not limited, for example, the second sub-window can be a 5×5 window. In step S 212 , a second mask M2 is used to select prior-processed blue and red pixels. In the example of FIG. 4 , the current-processed pixel is pixel G6, and the selected prior-processed green pixels are pixels B1, R4 and R5.

In step S 213 , a set of direction weightings of the green current-processed pixel is calculated as described in step S 204 . In another embodiment, since the set of direction weightings of the green current-processed pixel is previously estimated, as described in step S 207 , the green current-processed pixel may continue to use the previously estimated set of direction weightings. In step S 214 , an interpolating direction of the green current-processed pixel is determined according to the set of direction weightings of the green current-processed pixel. For example, if the horizontal weighting W H (G6) is larger than the vertical weighting W V (G6) and the flat weighting W F (G6) the interpolating of the green current-processed pixel G6 is horizontal. In step S 215 , green values of red or blue non-processed pixels in the second sub-window are estimated. For example, green values of a blue non-processed pixel B3 and a red non-processed pixel R5 are estimated as described in steps S 202 to S 207 . After step S 215 , all green values of pixels neighboring the green current-processed pixel are known. For example, green values of pixels B1, B3, R4 and R5 are known. In step S 216 , red values of blue non-processed pixels in the second sub-window are estimated. For example, a red value of the blue non-processed pixel B3 is estimated as described in step S 210 . After step S 216 , all red values of the pixels neighboring the green current-processed pixel are known. For example, red values of the pixels B1, B3, R4 and R5 are known. In step S 217 , a red value of the green current-processed pixel is calculated based on the red values of the pixels neighboring the current-processed pixel and the set of direction weightings of the current-processed pixel. For example, a red value of the green current-processed pixel G6 is calculated based on the set of direction weightings of the pixel G6 and the red values of neighboring pixels B1, B3, R4 and R5 such as:

In step S 218 , blue values of red non-processed pixels in the second sub-window are estimated. For example, a blue value of the red non-processed pixel R5 is estimated as described in step S 209 . After step S 218 , all blue values of the pixels neighboring the green current-processed pixel are known. For example, blue values of the pixels B1, B3, R4 and R5 are known. In step S 219 , a blue value of the green current-processed pixel is calculated based on the blue values of the pixels neighboring the current-processed pixel and the set of direction weightings of the current-processed pixel. For example, a blue value of the green current-processed pixel G6 is calculated based on the set of direction weightings of the pixel G6 and the blue values of the neighboring pixels B1, B3, R4 and R5 such as:

After the step S 219 , the red value and the blue value of the green current-processed pixel are known. In one embodiment, after steps S 201 to S 219 , values of three colors of the current-processed and some neighboring pixels can be used in other image processing in other image pipelines, such as sharpening, smoothing or/and denoising. Steps S 201 to S 219 are repeated to all the pixels of the digital image (steps S 220 ˜S 221 ).

The method for demosaicking described above may take the form of a program code (i.e., instructions) embodied on a non-transitory machine-readable storage medium such as floppy diskettes, CD-ROMS, hard drives, firmware, or any other machine-readable storage medium. When the program code is loaded into and executed by a machine, such as a digital imaging device or a computer, the machine implements the method for demosaicking.

FIG. 5 illustrates an image signal processor 50 for demosaicking a digital image in accordance with an embodiment of the invention. In this embodiment, the digital image IMG is captured by an image sensor overlaid with a Bayer color filter array. Each pixel of the digital image has only a green value, a red value or a blue value. The image signal processor 50 comprises a buffer module 510 , a demosaicking module 520 , a memory module 530 and an image processing module 540 .

The buffer module 510 receives the digital image IMG. The demosaicking module 520 is coupled to the buffer module 510 and implements demosaicking to each pixel of the digital image IMG to reconstruct values of two other missing colors. The memory module 530 is coupled to the demosaicking module 520 and stores the interpolating direction, the direction weighting corresponding to the interpolating direction and data used when the demosaicking module 520 implements steps S 201 to S 219 of the demosaicking method as described above to each pixel. The image processing module 540 is coupled to the demosaicking module 520 , receives a green value, a red value and a blue value of each pixels to implement a image processing process on each pixel based on the values, such as sharpening, smoothing, denoising or/and other image processing processes. After being processed, a processed image P_IMG is output.

Methods and apparatuses of the present disclosure, or certain aspects or portions of embodiments thereof, may take the form of a program code (i.e., instructions) embodied in media, such as floppy diskettes, CD-ROMS, hard drives, firmware, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing embodiments of the disclosure. The methods and apparatus of the present disclosure may also be embodied in the form of a program code transmitted over some transmission medium, such as electrical wiring or cabling, through fiber optics, or via any other form of transmission, wherein, when the program code is received and loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing and embodiment of the disclosure. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates analogously to specific logic circuits.

While the invention has been described by way of example and in terms of preferred embodiment, it is to be understood that the invention is not limited thereto. To the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.