Method of color processing using a color and white filter array

1. Field of the Invention

The present invention generally relates to a color and white filter array, and more particularly to a method of color processing using an RGBW filter array.

2. Description of Related Art

A Bayer filter, as depicted in FIG. 1 , is a color filter array composed of red (R), green (G) and blue (B) color filters disposed on a grid of photo-sensors, and specifically configured with a pattern of 50% green, 25% red and 25% blue. The Bayer filter with the photo-sensors forms an image sensor that is commonly used in electronic devices such as digital cameras or camcorders.

As each composing color filter of the Bayer filter passes only one color, information of other colors at each pixel position need be calculated by interpolation. The image sensor with the Bayer filter generally suffers from low sensitivity and low signal noise ratio (SNR). A modified Bayer filter with transparent white (W) filter (commonly called RGBW filter array) is thus introduced to improve sensitivity and SNR. The conventional image sensor with RGBW filter array, however, does not make effective use of the color and white information, and therefore cannot effectively improve sensitivity and SNR while preserving resolution and minimizing color artifacts.

A need has thus arisen to propose a novel scheme of processing the color and white information obtained from the RGBW filter array in order to overcome the disadvantages mentioned above.

In view of the foregoing, it is an object of the embodiment of the present invention to provide a method of color processing using a color and white filter array that performs pixel adaptive edge-sensing interpolation of white signal, and effectively utilizes white and white-chroma difference for color space transformation, therefore substantially improving sensitivity and signal noise ratio while preserving resolution and minimizing color artifacts. Moreover, the embodiment may provide flexible sharpness control and/or low light auto-exposure adjustment.

According to one embodiment, a color and white filter array is placed over pixels sensors of an image sensor, thereby generating converted color signals and converted white signals. The converted white signals are interpolated to generate interpolated white signals. White-chroma differences between the converted color signals and the converted white signals are interpolated to generate interpolated white-chroma difference signals. The interpolated white signal and the interpolated white-chroma difference signals are transformed into a color space, thereby generating transformed color signals.

FIG. 1 schematically shows a Bayer filter;

FIG. 2 shows a block diagram illustrated of color processing according to one embodiment of the present invention;

FIG. 3 shows an exemplary array pattern of a portion of the RGBW array of FIG. 2 ;

FIG. 4 exemplarily shows a portion of the RGBW array of FIG. 2 with a working window centered at a current center pixel;

FIG. 5 illustrates a flow diagram of the white interpolation block of FIG. 2 according to one embodiment of the present invention;

FIG. 6 exemplarily shows a portion of the RGBW array of FIG. 2 with a current white center pixel;

FIG. 7 exemplarily shows a portion of the RGBW array of FIG. 2 with a current green center pixel;

FIG. 8 exemplarily shows a portion of the RGBW array of FIG. 2 with a current blue center pixel; and

FIG. 9 illustrates a flow diagram of the white-chroma interpolation block of FIG. 2 according to one embodiment of the present invention.

FIG. 2 shows a block diagram illustrated of color processing according to one embodiment of the present invention. In the embodiment, the color processing may be performed by a processor such as a digital image processor.

According to the embodiment, a color and white filter array (filter array hereinafter) 20 is used to filter light. Specifically, the filter array 20 is composed of color filters, such as red (R) filters, green (G) filters and/or blue (B) filters, which filter light by wavelength range. The filter array 20 further includes white (W) filters that pass, or is transparent to, light in visible wavelength. In the embodiment, a filter array 20 composed of red filter, green filters, blue filters and white filters is adopted, and the filter array 20 of the embodiment may sometimes be referred to as an RGBW array instead.

FIG. 3 shows an exemplary array pattern (4×4 array is shown here) of a portion of the RGBW array 20 of FIG. 2 . The array pattern of the RGBW array 20 is configured of 50% white, 25% green, 12.5% red and 12.5% blue in a manner similar to a Bayer pattern as exemplified in FIG. 1 .

The RGBW array 20 of the embodiment is generally placed over pixels sensors (or photo-sensors) of an image sensor, such as complementary metal-oxide-semiconductor (CMOS) image sensor, which converts an optical image captured by the RGBW array 20 into electrical signals. Specifically, the image sensor along with an associated circuitry 21 converts red light captured by the red filter into a red signal, converts green light captured by the green filter into a green signal, converts blue light captured by the blue filter into a blue signal, and converts white light captured by the white filter into a white signal.

The circuitry 21 associated with the image sensor may include, for example, an amplifier 211 with gains and a bad-pixel corrector (BPC) 212 . The amplifier 211 and the BPC 212 may be implemented by conventional technique, details of which are omitted for brevity.

In the embodiment, as shown in FIG. 2 , the converted white signals are then subjected to an interpolation 22 , particularly an adaptive edge-sensing interpolation. FIG. 4 exemplarily shows a portion of the RGBW array 20 (5×5 array is shown) with a (e.g., 3×3) working window 41 centered at a current center pixel (P 13 in this example).

FIG. 5 illustrates a flow diagram of the white interpolation block 22 of FIG. 2 according to one embodiment of the present invention. In step 221 , a filtered white (FW) signal for each pixel in the working window 41 may be generated by filtering, such as low-pass filtering, neighboring converted white signals according to the following exemplary expressions:

FW 7 =(W 2 +W 6 +W 8 +W 12 )/4;

FW 8 =0.5*W 8 +(W 2 +W 4 +W 12 +W 14 )/8;

FW 9 =(W 4 +W 8 +W 10 +W 14 )/4;

FW 12 =0.5*W 12 +(W 6 +W 8 +W 16 +W 18 )/8;

FW 13 =(W 8 +W 12 +W 14 +W 18 )/4;

FW 14 =0.5*W 14 +(W 8 +W 10 +W 18 +W 20 )/8;

FW 17 =(W 12 +W 16 +W 18 +W 22 )/4;

FW 18 =0.5*W 18 +(W 12 +W 14 +W 22 +W 24 )/8;

FW 19 =(W 14 +W 18 +W 20 +W 24 )/4

For example, a filtered white signal FW 7 at the pixel P 7 may be generated by averaging neighboring converted white signals, that is, W 2 , W 6 , W 8 and W 12 . Similarly, for example, a filtered white signal FW 8 at the pixel P 8 may be generated by taking weighted average of neighboring converted white signals (i.e., W 2 , W 4 , W 12 and W 14 ) with a weight of ⅛ and the converted white signal at the pixel P 8 itself with a (greater) weight of 0.5.

Subsequently, in step 222 , edge-detection parameters are generated using the filtered white signals in the working window 41 according to the following exemplary expressions:

V_l=abs(FW 12 -FW 7 )+abs(FW 12 -FW 17 );

V_c=abs(FW 13 -FW 8 )+abs(FW 13 -FW 18 );

V_r=abs(FW 14 -FW 9 )+abs(FW 14 -FW 19 );

V_con=V_l<FW 12 *Eth&&V_c<FW 13 *Eth&&V—r<FW 14 *Eth;

H_t=abs(FW 8 -FW 7 )+abs(FW 8 -FW 9 );

H_c=abs(FW 13 -FW 12 )+abs(FW 13 -FW 14 );

H_b=abs(FW 18 -FW 17 )+abs(FW 18 -FW 19 );

H_con=H_t<FW 8 *Eth&&H_c<FW 13 *Eth&&H_b<FW 18 *Eth;

EV=V_l+V_c+V_r+H_t+H_c+H_b

For example, a sum V_l of absolute values of pairs of adjacent pixels (e.g., a pair of P 12 and P 7 , and a pair of P 12 and P 17 ) along a vertically left direction may be generated to indicate the extent of detecting an edge along that direction (e.g., the less the value is, the more probably an edge exists). Similarly, for example, a sum H_t of absolute values of pairs of adjacent pixels (e.g., a pair of P 8 and P 7 , and a pair of P 8 and P 9 ) along a horizontally top direction may be generated to indicate the extent of detecting an edge along that direction.

Afterwards, in step 223 , edge detection is performed based on the edge-detection parameters generated from step 222 . In the embodiment, one of three edge-detection decisions is determined: an edge detected, no edge detected, and an in-between. Finally, in step 224 A, 224 B or 224 C (depending on the determined edge-detection decision), an interpolated white signal (e.g., W 13 ) at the current center pixel (e.g., P 13 in the example) may be generated according to the converted white signals in the working window 41 in accordant with the determined edge-detection decision. Steps 223 and 224 A-C may be carried out according to the following exemplary expressions, where ev_thL and ev_thH are threshold values:

(1) Edge Detected:

If EV>FW 13 *ev_thH

W ⁢ ⁢ 13 = d H = { ( W ⁢ ⁢ 8 + W ⁢ ⁢ 18 ) / 2 ifV_con && H_con && V_l + V_c + V_r < H_t + H_c + H_b ( W ⁢ ⁢ 12 + W ⁢ ⁢ 14 ) / 2 ifV_con && H_con && V_l + V_c + V_r > H_t + H_c + H_b ( W ⁢ ⁢ 8 + W ⁢ ⁢ 18 ) / 2 ifV_con && ~ H_con ( W ⁢ ⁢ 12 + W ⁢ ⁢ 14 ) / 2 ifH_con && ~ V_con ( W ⁢ ⁢ 8 + W ⁢ ⁢ 18 ⁢ + W ⁢ ⁢ 12 + W ⁢ ⁢ 14 ) / 4 else }

(2) No Edge Detected

If EV<FW 13 *ev_thL

W 13= d L =( W 8+ W 18+ W 12+ W 14)/4

(3) In-Between

If FW 13 *ev_thL≦EV≦FW 13 *ev_thH

where

Summarily speaking, in case a vertical edge has been detected, the interpolated white signal is generated by averaging neighboring converted white signals (e.g., W 8 and W 18 ) along the edge. In case a horizontal edge has been detected, the interpolated white signal is generated by averaging neighboring converted white signals (e.g., W 12 and W 14 ) along the edge. In case no edge has been detected, the interpolated white signal is generated by averaging all neighboring converted white signals (e.g., W 8 , W 18 , W 12 and W 14 ). In case an in-between has been detected, the interpolated white signal is a weighted sum of the interpolated white signal with an edge detected, and the interpolated white signal without an edge detected.

Referring back to FIG. 2 , sharpness enhancement 23 may be further performed on the interpolated white signal. Specifically, scaled differences between the interpolated signal of the current white center pixel (e.g., P 13 in FIG. 4 ) and the converted white signals of neighboring pixels (e.g., P 8 , P 12 , P 14 and P 18 ) may be added back to the current white center pixel to generate a weighted average value with enhanced sharpness. Other conventional technique may be adopted to perform sharpness enhancement 23 on the interpolated white signal.

Still referring to FIG. 2 , the converted color signals (e.g., red, green and/or blue signals) and white signals are subjected to an interpolation 24 of white-chroma (or white-color) difference (e.g., W-R, W-G, W-B). The white-chroma interpolation 24 may be performed in one of four manners depending on what the current center pixel is: a white pixel (as exemplified in FIG. 6 ), a green pixel (as exemplified in FIG. 7 ), a blue pixel (as exemplified in FIG. 8 ) and a red pixel.

Take the current white center pixel demonstrated in FIG. 6 as an example, FIG. 9 illustrates a flow diagram of the white-chroma interpolation block 24 of FIG. 2 according to one embodiment of the present invention. In step 241 , a white-red difference set dWR_set 10 composed of differences respectively between the converted white signals and neighboring converted red signals is obtained, a white-green difference set dWG_set 16 composed of differences respectively between the converted white signals and neighboring converted green signals is obtained, and a white-blue difference set dWB_set 10 composed of differences respectively between the converted white signals and neighboring converted blue signals is obtained. Obtaining the white-chroma difference sets in step 241 may be carried out according to the following exemplary expressions:

at P 13 : dWR_set 10 =[W 13 -R 14 ;W 9 -R 14 ;W 15 -R 14 ;W 19 -R 14 ;W 7 -R 2 ;W 1 -R 2 ;W 3 -R 2 ;W 17 -R 22 ;W 21 -R 22 ;W 23 -R 22 ]; dWG_set 16 =[W 3 -G 8 ;W 7 -G 8 ;W 13 -G 8 ;W 9 -G 8 ;W 13 -G 18 ;W 17 -G 18 ;W 19 -G 18 ;W 23 -G 18 ; W 7 -G 6 ;W 11 -G 6 ;W 11 -G 16 ;W 17 -G 16 ;W 19 -G 20 ;W 15 -G 20 ;W 15 -G 10 ;W 9 -G 10 ]; dWB_set 10 =[W 7 -B 12 ;W 11 -B 12 ;W 17 -B 12 ;W 13 -B 12 ;W 3 -B 4 ;W 5 -B 4 ;W 9 -B 4 ;W 19 -B 24 ;W 23 -B 24 ;W 25 -B 24 ]

Subsequently, in step 242 , the white-chroma difference set is analyzed to determine its data distribution. In the embodiment, a ratio of (maximum value minus minimum value) to (maximum value plus minimum value) of the white-chroma difference set is used to determine data distribution. If the ratio is greater than a threshold value th, indicating that the white-chroma difference set is not normally distributed, a median of the white-chroma difference set is preferably used as an interpolated white-chroma difference signal (step 243 ), otherwise, a mean of the white-chroma difference set is used as an interpolated white-chroma difference signal (step 244 ). Steps 242 - 244 may be carried out according to the following exemplary expressions:

dWR=median(dWR_set 10 ); if (max(dWR_set 10 )−min(dWR_set 10 ))/(max(dWR_set 10 )+min(dWR_set 10 ))>th; Else dWR=mean(dWR_set 10 ); dWG=median(dWG_set 16 ); if (max(dWG_set 16 )−min(dWG_set 16 ))/(max(dWG_set 16 )+min(dWG_set 16 ))>th; Else dWG=mean(dWG_set 16 ); dWB=median(dWB_set 10 ); if (max(dWB_set 10 )−min(dWB_set 10 ))/(max(dWB_set 10 )+min(dWB_set 10 ))>th; Else dWB=mean(dWB_set 10 )

The interpolation 24 of white-chroma difference with a green current center pixel ( FIG. 7 ) may be carried out according to the following exemplary expressions. It is noted that W, G, R or B in the following expressions may represent the filtered signal if it is not available directly from the RGBW array 20 .

at P 13 : dWR_set 8 =[W 13 -R 14 ;W 9 -R 14 ;W 15 -R 14 ;W 19 -R 14 ;W 7 -R 2 ;W 1 -R 2 ;W 3 -R 2 ;W 17 -R 22 ; W 21 -R 22 ; W 23 -R 22 ]; dWG_set 16 =[W 3 -G 8 ;W 7 -G 8 ;W 13 -G 8 ;W 9 -G 8 ;W 13 -G 18 ;W 17 -G 18 ;W 19 -G 18 ;W 23 -G 18 ; W 7 -G 6 ;W 11 -G 6 ;W 11 -G 16 ;W 17 -G 16 ;W 19 -G 20 ;W 15 -G 20 ;W 15 -G 10 ;W 9 -G 10 ]; dWB_set 8 =[W 7 -B 12 ;W 11 -B 12 ;W 17 -B 12 ;W 13 -B 12 ;W 3 -B 4 ;W 5 -B 4 ;W 9 -B 4 ;W 19 -B 24 ; W 23 -B 24 ; W 25 -B 24 ]; dWR=median(dWR_set 8 ); if (max(dWR_set 8 )−min(dWR_set 8 ))/(max(dWR_set 8 )+min(dWR_set 8 ))>th; Else dWR=mean(dWR_set 8 ); dWG=median(dWG_set 16 ); if (max(dWG_set 16 )−min(dWG_set 16 ))/(max(dWG_set 16 )+min(dWG_set 16 ))>th; Else dWG=mean(dWG_set 16 ); dWB=median(dWB_set 8 ); if (max(dWB_set 8 )−min(dWB_set 8 ))/(max(dWB_set 8 )+min(dWB_set 8 ))>th; Else dWB=mean(dWB_set 8 )

The interpolation 24 of white-chroma difference with a blue current center pixel ( FIG. 8 ) may be carried out according to the following exemplary expressions. It is noted that W, G, R or B in the following expressions may represent the filtered signal if it is not available directly from the RGBW array 20 .

at P 13 : dWR_set 12 =[W 2 -R 3 ;W 4 -R 3 ;W 8 -R 3 ;W 6 -R 11 ;W 16 -R 11 ;W 12 -R 11 ;W 18 -R 23 ;W 22 -R 23 ;W 24 -R 23 ;W 10 -R 15 ;W 14 -R 15 ;W 20 -R 15 ]; dWG_set 16 =[W 2 -G 7 ;W 6 -G 7 ;W 8 -G 7 ;W 12 -G 7 ;W 4 -G 9 ;W 8 -G 9 ;W 10 -G 9 ;W 14 -G 9 ; W 12 -G 17 ;W 16 -G 17 ;W 18 -G 17 ;W 22 -G 17 ;W 18 -G 19 ;W 14 -G 19 ;W 20 -G 19 ;W 24 -G 19 ], dWB_set 12 =[W 8 -B 13 ;W 12 -B 13 ;W 14 -B 13 ;W 18 -B 13 ;W 2 -B 1 ;W 6 -B 1 ;W 4 -B 5 ;W 10 -B 5 ;W 16 -B 21 ; W 22 -B 21 ;W 20 -B 25 ;W 24 -B 25 ]; dWR=median(dWR_set 12 ); if (max(dWR_set 12 )−min(dWR_set 12 ))/(max(dWR_set 12 )+min(dWR_set 12 ))>th; Else dWR=mean(dWR_set 12 ); dWG=median(dWG_set 16 ); if (max(dWG_set 16 )−min(dWG_set 16 ))/(max(dWG_set 16 )+min(dWG_set 16 ))>th; Else dWG=mean(dWG_set 16 ); dWB=median(dWB_set 12 ); if (max(dWB_set 12 )−min(dWB_set 12 ))/(max(dWB_set 12 )+min(dWB_set 12 ))>th; Else dWB=mean(dWB_set 12 )

The interpolation 24 of white-chroma difference with a red current center pixel may be carried out in a manner similar to the interpolation with a blue current center pixel, and details of which are thus omitted for brevity.

Referring back to FIG. 2 , the interpolated white signal from the block 22 (or enhanced one from the block 23 ) and the interpolated white-chroma difference signals from the block 24 are transformed into a color space (e.g., RGB space), therefore generating transformed color signals. Specifically, in the embodiment, a transformed red signal may be generated by subtracting the interpolated white-red difference signal from the interpolated white signal, a transformed green signal may be generated by subtracting the interpolated white-green difference signal from the interpolated white signal, and a transformed blue signal may be generated by subtracting the interpolated white-blue difference signal from the interpolated white signal, according to the following exemplary expressions. It is noted that W in the following expressions represent the interpolated white signal.

{circumflex over (R)}=W−dWR;

Ĝ=W−dWG;

{circumflex over (B)}=W−dWB;

In an alternative embodiment, the transformed color signals may be generated taking into consideration low light (i.e., less than normal lighting) auto-exposure adjustment. Specifically, auto-exposure may generally be determined by integration time (INTG), analog gain (AG) and digital gain (DG) obtained from the circuitry 21 . A parameter α, which acts as weighting on the interpolated white signal, is introduced here for brightness adaptation under different lighting conditions. The transformed color signals, in the embodiment, may be generated according to the following exemplary expressions:

{circumflex over (R)}=W −(1−α)* dWR=α*W −(1−α)* R;

Ĝ=W −(1−α)* dWR=α*W −(1−α)* G;

{circumflex over (B)}=W −(1−α)* dWR=α*W −(1−α)* B;

In normal lighting, the parameter α is equal to 0. As lighting goes down, INTG, AG and DG mentioned above all moves toward maximum. The parameter α starts to increase from 0 to 1 so that the brightness is boosted. Suppose an auto-exposure target is AE target , a current mean brightness value is Y mean , a current mean white value is W mean , then the parameter α may be obtained as:

otherwise α=1

the corresponding low light brightness gain=

Although specific embodiments have been illustrated and described, it will be appreciated by those skilled in the art that various modifications may be made without departing from the scope of the present invention, which is intended to be limited solely by the appended claims.