Color interpolation system and method thereof

1. Field of the Invention

The present invention generally relates to digital image processing, and more particularly to a color interpolation system and method thereof.

2. Description of Related Art

As an image sensor is commonly equipped with a color filter array (CFA), for example arranged as Bayer pattern, Raw image data retrieved from the image sensor therefore need be subjected to demosaicing to reconstruct a full color image. The demosaicing is normally performed by interpolation technique.

Image enhancement is another digital image processing commonly used to enhance one or more image parameters such as contrast, sharpness, etc. The image enhancement may be performed according to the results of edge detection or gradient.

In a conventional imaging system, the raw image data retrieved from the image sensor are firstly subjected to interpolation operation, followed by image enhancement process. One disadvantage of the conventional imaging system is its inherent drawback that red data and blue data are sampled at a rate substantially lower than green data, such that the imaging system generates distorted image with low sharpness.

For the foregoing reasons, a need has arisen to propose a novel imaging system with improved quality.

In view of the foregoing, it is an object of the embodiment of the present invention to provide a color interpolation system and method that are capable of simultaneously performing interpolation and enhancement with improved sharpness.

According to one embodiment, the color interpolation system includes an enhancement unit, a first interpolation unit and a gain generator. The enhancement unit is coupled to receive a number of raw signals from an image sensor, wherein, the raw signals include at least a raw first signal and a raw second signal, and the enhancement unit processes the raw signals to output an enhanced first signal. The first interpolation unit is coupled to receive and process the raw first signal and accordingly output an interpolated first signal. The gain generator is configured to generate an enhancement gain according to the enhanced first signal and the interpolated first signal.

According to another embodiment, the color interpolation system includes an enhancement unit, a first interpolation unit, a blender and a gain generator. The enhancement unit is coupled to receive a number of raw signals from an image sensor, wherein, the raw signals include at least a raw first signal and a raw second signal, and the enhancement unit processes the raw signals to output an enhanced first signal. The first interpolation unit is coupled to receive and process the raw first signal and accordingly output a plurality of interpolated first signals. The blender is configured to give weighting to the enhanced first signal with respect to one of the interpolated first signals, thereby outputting a weighted enhanced first signal. The gain generator is configured to generate an enhancement gain according to the weighted enhanced first signal and the interpolated first signals except the interpolated first signal fed to the blender.

FIG. 1A shows a block diagram illustrative of a color interpolation system according to one embodiment of the present invention;

FIG. 1B shows a block diagram illustrative of a color interpolation system according to an embodiment alternative to FIG. 1A ;

FIG. 2 shows a process window of image pixels with the raw blue signal at its center;

FIG. 3 shows another process window of image pixels with the raw green signal at its center, which has raw blue signals to its right and left;

FIG. 4A shows a block diagram illustrative of a color interpolation system according to another embodiment of the present invention; and

FIG. 4B shows a block diagram illustrative of a color interpolation system according to an embodiment alternative to FIG. 4A .

FIG. 1 shows a block diagram illustrative of a color interpolation system 1 A according to one embodiment of the present invention. Each block of the illustrated system 1 A may be implemented by hardware, software or their combination.

In the embodiment, the system 1 A is coupled to receive a raw green signal G_raw, a raw red signal R_raw and a raw blue signal B_raw, which are in a raw format and are outputted from an image sensor (not shown in the figure) equipped with a color filter array (CFA) having a pattern such as Bayer pattern. Other possible patterns may, for example, be RGB pattern or CMY pattern. The color filter array of Bayer pattern may have plural of color filters, 50% of which are green, 25% of which are red and 25% of which are blue. Generally speaking, there are more green elements than red elements or blue elements in the Bayer pattern or other similar pattern. In this specification, the green signal, the red signal and the blue signal may be interchangeably named the first signal, the second signal and the third signal respectively.

FIG. 2 shows a process window of image pixels (with the raw blue signal B_raw at its center), which constitute a portion of the image sensor. An enhancement unit 10 ( FIG. 1A ) is coupled to receive and process the raw signals G_raw, R_raw and B_raw, thereby outputting an enhanced green signal G22_en, which is located, at the center of the process window. The enhancement of the enhancement unit 10 may be performed by an enhancing mask, which may generally be a high-boost filter, as follows:

G 22 — en=W 01 *P 01+ W 03* P 03+ W 10 *P 10+ W 12 *P 12+ W 14 *P 14+ W 21 *P 21+ W 23 *P 23+ W 30 *P 30+ W 32 *P 32+ W 34 *P 34+ W 41 *P 41+ W 43 *P 43;

//W01-W43: Enhancing Mask Coefficient.

A first interpolation unit 11 receives and processes the raw green signal G_raw, and then outputs an interpolated green signal G22_sm at the center of the process window. A conventional interpolation technique, such as bi-linear interpolation, bi-cubic interpolation, spline interpolation, or the like, may be adopted by the first interpolation unit 11 .

Similarly, a second interpolation unit 12 receives and processes the raw red signal R_raw, and then outputs an interpolated red signal R22_sm at the center of the process window. A conventional interpolation technique, such as bi-linear interpolation, bi-cubic interpolation, spline interpolation, or the like, may be adopted by the second interpolation unit 12 .

Similarly, a third interpolation unit 13 receives and processes the raw blue signal B_raw, and then outputs an interpolated blue signal B22_sm at the center of the process window. A conventional interpolation technique, such as bi-linear interpolation, bi-cubic interpolation, spline interpolation, or the like, may be adopted by the third interpolation unit 13 .

In one embodiment, a blender 14 may be selectively used to generate a weighted enhanced green signal WG22 according to the enhanced green signal G22_en and the interpolated, green signal G22_sm. Specifically, the blender 14 is used to give weighting to the enhanced green signal G22_en with respect to the interpolated green signal G22_sm. The weighted enhanced green signal WG22 may be obtained, for example, according to the following expression:

WG 22=( G 22 — en *weight+ G 22 — sm *(1−weight));

//weight: Weight of enhancing mask, not greater than 1 and not less than 0.

The expressions demonstrated above are with respect to the process window with the raw blue signal B_raw at its center. Similar expressions may be likewise obtained with respect to a process window with the raw red signal R_raw at its center.

FIG. 3 shows another process window of image pixels with the raw green signal G_raw at its center, which has raw blue signals B_raw to its right and left.

Regarding this process window, the associated enhanced green signal G22_en and the weighted enhanced green signal WG22 may be respectively expressed as follows:

G 22 — en=W 00 *P 00+ W 02 *P 02+ W 04 *P 04+ W 11 *P 11+ W 13 *P 13+ W 20 *P 20+ W 22 *P 22+ W 24 *P 24+ W 31 *P 31+ W 33 *P 33+ W 40 *P 40+ W 42 *P 42+ W 44 *P 44;

//W00-W44: Enhancing Mask Coefficient

WG 22=( G 22 — en *weight+ G 22 — sm *(1−weight));

//weight: Weight of Enhancement Mask, not greater than 1 and not less than 0.

The expressions demonstrated above are with respect to the process window with the raw green signal G_raw at its center, which has raw blue signals B_raw to its right and left. Similar expressions may be likewise obtained with respect to a process window with the raw green signal G_raw at its center, which has raw red signals R_raw to its right and left.

The enhanced green signal G22_en (or the weighted enhanced green signal WG22 if the blender 14 is used) and the interpolated green signal G22_sm are fed to a gain generator 16 generates an enhancement gain D according to the enhanced green signal G22_en (or the weighted enhanced green signal WG22) and the interpolated green signal G22_sm. In one example, the enhancement gain D is generated according to a difference between the weighted enhanced green signal WG22 and the interpolated green signal G22_sm. In another example, the enhancement gain D is generated according to a ratio between the weighted enhanced green signal WG22 and the interpolated green signal G22_sm.

In one embodiment, if the absolute value of the difference between the weighted enhanced green signal WG22 and the interpolated green signal G22_sm is not greater than a predetermined noise threshold, the enhancement gain D has a value of “1.” Otherwise, the enhancement gain D is determined according to the value of the difference between the weighted enhanced green signal WG22 and the interpolated green signal G22_sm and may be adjusted by a predetermined value.

Finally, a gain multiplier 17 is configured to multiply the interpolated, green signal G22_sm, the interpolated red signal R22_sm and the interpolated blue signal B22_sm with the enhancement gain D respectively as follow:

G 22 ′=G 22 — sm*D;

R 22 ′=R 22 — sm*D;

B 22 ′=B 22 — sm*D.

FIG. 1B shows a block diagram illustrative of a color interpolation system 1 B according to an embodiment alternative to FIG. 1A . The system 1 B of FIG. 1B is similar to the system 1 A of FIG. 1A with the distinctness being described below. In the present embodiment, only the raw green signal G_raw is processed by an interpolation unit (i.e., the first interpolation unit 11 ). The gain multiplier 17 is configured to multiply the interpolated green signal G22_sm, the raw red signal R_raw and the raw blue signal B_raw with the enhancement gain D respectively.

FIG. 4A shows a block diagram illustrative of a color interpolation system 4 A according to another embodiment of the present invention. Each block of the illustrated system 4 A may be implemented by hardware, software or their combination.

The system 4 A of FIG. 4A is similar to the system 1 A of FIG. 1A with the distinctness being described below. In the present embodiment, a first interpolation unit 11 B receives and processes the raw green signal G_raw, and then outputs interpolated green signals G11-G33 (e.g., 3×3 pixels) that constitute a process sub-window (i.e., a subset of the original process window) centered at the process window as shown in FIG. 2 . For example, the interpolated green signals G11-G33 may be obtained as follows:

G 11 =f ( P 01 ,P 10 ,P 21 ,P 12);

G 12 =f ( P 01 ,P 03 ,P 21 ,P 23 ,P 12);

G 13 =f ( P 03 ,P 12 ,P 14 ,P 23);

G 21 =f ( P 10 ,P 12 ,P 30 ,P 32 ,P 21);

G 22 =f ( P 12 ,P 21 ,P 23 ,P 32);

G 23 =f ( P 14 ,P 12 ,P 34 ,P 32 ,P 23);

G 31 =f ( P 21 ,P 30 ,P 41 ,P 32);

G 32= f ( P 21 ,P 23 ,P 41 ,P 43 ,P 32);

G 33= f ( P 23 ,P 32 ,P 34 ,P 43);

G22=G22; //Output Green pixel.

A blender 14 B is used to give weighting to the enhanced green signal G22_en with respect to the center interpolated green signal G22 (from the first interpolation unit 11 B). The weighted enhanced green signal WG22 may be obtained, for example, according to the following expression:

WG 22=( G 22 — en *alpha+ G 22*(1−alpha));

//0=<alpha<1.

In the embodiment, the gain generator 16 may perform low-pass filtering (LPF) on the weighted enhanced green signal WG22 and the interpolated green signals, for example G11-G33 except G22, thereby resulting in an interpolated green signal G22_sm.

Similar to the system 1 A of FIG. 1A , the weighted enhanced green signal WG22 and the interpolated green signal G22 are fed to the gain generator 16 , which generates an enhancement gain D according to a difference or a ration between the weighted enhanced green signal WG22 and the interpolated green signal G22. The gain multiplier 17 is configured to multiply the interpolated green signal G22_sm, the interpolated red signal R22_sm and the interpolated blue signal B22_sm with the enhancement gain D respectively.

The expressions demonstrated above are with respect to the process window with the raw blue signal B_raw at its center ( FIG. 2 ). Similar expressions may be likewise obtained with respect to a process window with the raw red signal R_raw at its center.

FIG. 4B shows a block diagram illustrative of a color interpolation system 4 B according to an embodiment alternative to FIG. 4A . The system 4 B of FIG. 4B is similar to the system 4 A of FIG. 4A with the distinctness being described below. In the present embodiment, only the raw green signal G_raw is processed by an interpolation unit (i.e., the first interpolation unit 11 B). The gain multiplier 17 is configured to multiply the interpolated green signal G22_sm, the raw red signal R_raw and the raw blue signal B_raw with the enhancement gain D respectively.

Although specific embodiments have been illustrated and described, it will be appreciated by those skilled in the art that various modifications may be made without departing from the scope of the present invention, which is intended to be limited solely by the appended claims.