Image pickup device and method for processing an interpolated color signal

This application is based upon and claims the benefit of priority from the prior Japanese Patent Applications No. 2007-196472, filed on Jul. 27, 2007 and No. 2008-108014, filed on Apr. 17, 2008, the entire contents of which are incorporated herein by reference.

1. Field of the Invention

The present invention relates to an image pickup device provided with pixels in a bayer array.

2. Related Art

Conventionally, in a color camera using a CMOS image sensor, it is general to use a color filter of a so-called bayer array in which filters with primary colors of red (R), green (G) and blue (B), hereinafter collectively called RGB, are arranged in a checkerboard pattern. In the bayer array, G pixels are arranged first in a checkerboard pattern and, in the remaining portions, and then R pixels and B pixels are respectively arranged in checkerboard patterns.

On the other hand, an image sensor in order to improve signal to noise ratio by replacing one of the G pixels with a white (W) pixel to increase light transmission has recently been introduced (see Reference Document 1: A novel Bayer-like WRGB color filter array for CMOS image sensors by Hiroto Honda, et al., Proceedings of SPIE-IS&T Electronic Imaging, SPIE Vol. 6492, pp. 64921J-1 to 64921J-10 (2007); Reference Document 2: High Sensitivity Color CMOS Image Sensor with WRGB Color Filter Array and Color Separation Process using Edge Detection by Hiroto Honda, et al., 2007 International Image Sensor Workshop, Jun. 7-10, 2007; and Reference Document 3: News Release from Eastman Kodak, Jun. 14, 2007).

In Reference Document 1 and Reference Document 2, in order to improve signal to noise ratios of RGB signals, RGB pixel values (RW, GW and BW) at the pixel position of W pixels in this color filter array are generated based on the following Equations 1 to 3.

RW=W×R -average/( R -average+ G -average+ B -average)  Equation 1

GW=W×G -average/( R -average+ G -average+ B -average)  Equation 2

BW=W×B -average/( R -average+ G -average+ B -average)  Equation 3

Here, an R-average is calculated by averaging two R pixels neighboring a W pixel. Similarly, a B-average is calculated by averaging two B pixels neighboring the W pixel and a G-average is calculated by averaging four G pixels neighboring the W pixel.

Since Reference Document 1 to Reference Document 3 assume the use of an image sensor having W pixels, the sensor cost is likely to become high. Further, since an existing sensor with a bayer array can not be used, it is likely to take time to gain popularity, and verification of electrical characteristics of the sensor having W pixels is also required. Particularly, there may be also various problems associated with sensitivity differences between W pixels and other pixels and variations in filter characteristics and such.

The present invention may provide an image pickup device capable of improving signal to noise ratios of color signals and image quality by using an existing sensor in a bayer array.

According to one aspect of the present invention, an image pickup device comprising:

a lens;

first, second and third color filters which are provided by each pixel and arranged in bayer array, the first color filter being arranged in a checkerboard pattern and one of pixels corresponding to the first color filter being used as a reference pixel;

an image sensor which photoelectrically converts lights passing through the first to third filters via the lens to generate a first color signal corresponding to the first color filter, a second color signal corresponding to the second color filter and a third color signal corresponding to the third color filter; and

a signal processing part configured to perform signal process for the first to third color signals,

wherein the signal processing part includes:

a first color generator configured to generate a fourth color signal corresponding to the reference pixel based on a ratio between the second color signal at a pixel located in vicinity of the reference pixel and the first color signal at a pixel located in vicinity of the reference pixel;

a second color generator configured to generate a fifth color signal corresponding to the reference pixel based on a ratio between the third color signal at a pixel located in vicinity of the reference pixel and the first color signal at a pixel located in vicinity of the reference pixel; and

a image quality converter configured to generate color signals by performing a predetermined image process based on the first to fifth color signals.

According to one aspect of the present invention, an image pickup device comprising:

a lens;

first, second and third color filters which are provided by each pixel and arranged in bayer array, the first color filter being arranged in a checkerboard pattern and one of pixels corresponding to the second color filter or the third color filter being used as a reference pixel;

an image sensor which photoelectrically converts lights passing through the first to third filters via the lens to generate a first color signal corresponding to the first color filter, a second color signal corresponding to the second color filter and a third color signal corresponding to the third color filter; and

a signal processing part configured to perform signal process for the first to third color signals,

wherein the signal processing part includes:

a first color generator configured to generate a fourth color signal corresponding to the reference pixel based on a ratio between the second color signal at a pixel located in vicinity of the reference pixel and the third color signal at a pixel located in vicinity of the reference pixel;

a second color generator configured to generate a fifth color signal corresponding to the reference pixel based on a ratio between the third color signal at a pixel located in vicinity of the reference pixel and the second color signal at a pixel located in vicinity of the reference pixel; and

a image quality converter configured to generate color signals by performing a predetermined image process based on the first to fifth color signals.

FIG. 1 is a block diagram schematically illustrating a configuration of an image pickup device of an embodiment of the present invention;

FIG. 2 is a schematic view illustrating a color arrangement of a color filter array 3 of the embodiment;

FIG. 3 is a graph illustrating spectral sensitivity characteristics of each of color signals used in a single chip color camera;

FIGS. 4A to 4C are schematic views, FIG. 4A showing the color filter array 3 in a bayer array, FIG. 4B illustrating a generation of a virtual pixel Bv 22 from a reference pixel G 22 , and FIG. 4C illustrating a generation of a virtual pixel Rv 22 from the reference pixel G 22 ;

FIGS. 5A and 5B are schematic views, FIG. 5A illustrating an array pattern of virtual Bv pixels and FIG. 5B illustrating an array pattern of virtual Rv pixels;

FIG. 6 is a block diagram illustrating an example of an internal structure of a virtual pixel arithmetic circuit 12 ;

FIG. 7 is a block diagram illustrating an example of a detailed structure of a synchronous circuit 11 and the virtual pixel arithmetic circuit 12 shown in FIG. 1 ;

FIG. 8 is a flowchart illustrating an example of a process operation of the virtual pixel arithmetic circuit 12 ;

FIGS. 9A and 9B are schematic views, FIG. 9A illustrating generated virtual Bv pixels, and FIG. 9B illustrating generated virtual Rv pixels;

FIG. 10 is a schematic view illustrating an example of a pixel block composed of 3 by 5 pixels;

FIG. 11 is a block diagram illustrating an example of an internal structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to calculate Equations 13 and 14;

FIGS. 12A to 12C are schematic views to illustrate calculation of virtual pixels only in a horizontal direction;

FIGS. 13A and 13B are schematic views illustrating the result of the processes shown in FIGS. 12A to 12C ;

FIGS. 14A to 14C are schematic views to illustrate calculation of virtual pixels only in a vertical direction;

FIGS. 15A and 15B are schematic views illustrating the result of the processes shown in FIGS. 14A to 14C ;

FIGS. 16A and 16B are schematic views illustrating a pitch of B pixels of a sensor including W pixels and a spatial frequency thereof;

FIGS. 17A and 17B are schematic views illustrating a pitch of B pixels after virtual pixel calculations of the embodiment and a spatial frequency thereof;

FIG. 18 is a schematic view illustrating to define a pixel block of 3 pixels tall by 5 pixels wide neighboring a reference pixel G 33 ;

FIG. 19 is a schematic view illustrating to define a pixel block of 5 pixels tall by 3 pixels wide neighboring the reference pixel G 33 ;

FIG. 20 is a block diagram illustrating an example of internal structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to perform calculations of Equations 17 and 18; and

FIG. 21 is a block diagram illustrating an example of internal structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to perform calculations of Equations 19 and 20.

Embodiments according to the present invention will now be explained with reference to the accompanying drawings.

FIG. 1 is a block diagram schematically illustrating a configuration of an image pickup device of the embodiment. The image pickup device shown in FIG. 1 is provided with a lens 1 , a CMOS image sensor 2 having a photoelectric conversion element for each of pixels, a color filter array 3 disposed over each of the photoelectric conversion elements of the sensor 2 , and a signal processing circuit 14 .

The signal processing circuit 14 has a synchronous circuit 11 which outputs pixel values in parallel by the unit of a pixel block composed of 3 by 3 pixels, a virtual pixel arithmetic circuit (a first color generating unit and a second color generating unit) 12 described in detail later, a property conversion circuit (an image quality converter) 13 which generates color signals.

An optical image of a photographic subject, passing through the lens 1 , is formed on an imaging area of the CMOS image sensor 2 . On the imaging area, the color filter array 3 is provided, and light in wavelengths passing through the color filter array 3 are photoelectrically converted by the photoelectric conversion elements to obtain color information.

The output signals from the CMOS image sensor 2 are of RGB signals corresponding to the color filter array 3 and, typically after converted to digital signals by an A/D converter not shown, fed to the signal processing circuit 14 .

The RGB signals entered to the signal processing circuit 14 are of dot-sequential signals (serial signals). The synchronous circuit 11 outputs the color signals for 3 by 3 pixels which compose a pixel block in parallel at the same timing. The virtual pixel arithmetic circuit 12 newly generates a virtual R signal and a virtual B signal from the R signal, G signal and B signal and feeds to the property conversion circuit 13 .

The property conversion circuit 13 carries out, by using the virtual R signal and the virtual B signal generated by the virtual pixel arithmetic circuit 12 and the color signals fed to the signal processing circuit 14 , non-linear processes of matrix calculation, edge correction, gamma correction and such, and outputs the color signals in a signal format of, for example, YUV or RGB.

Next, the virtual pixel arithmetic circuit 12 will be described in detail. FIG. 2 is a schematic view illustrating a color arrangement of the color filter array 3 of the embodiment and FIG. 3 is a graph illustrating spectral sensitivity characteristics of each of the color signals used in a single chip color camera.

The magnitude of each of the RGB color signals output from the image sensor 2 is to be the value obtained by multiplying the wavelength characteristics of the color filter in FIG. 3 by the spectral sensitivity characteristics of the image sensor 2 and integrating by wavelengths. Presently, the spectral sensitivity characteristics of each of the color signals used in most single chip color cameras are in curves as shown in FIG. 3 .

Here, since dyes for RGB colors are limited mainly in view of reliability, each of the color characteristics is not of a sharp cutoff and gets into domains of other color characteristics. As shown in FIG. 3 , it is particularly significant with a G color filter, other than the intended wavelengths corresponding to G components, R components and B components are substantially infiltrated. Even in a generally used G color filter, the actual wavelength characteristics are of rather broad characteristics and pixels corresponding to the G color filter also contain R and B components.

Meanwhile, since each pixel is provided only with a single color filter and a signal wiring corresponding thereto, from the pixels arranged with G filters, only the G signal is obtained. In a bayer array, since R and B color filters are provided only with half the number of G filters, the amount of signals for the R signal and B signal becomes less than that of the G signal.

Therefore, in the embodiment, when using the color filter 3 in a bayer array, color signals of different colors from the color filter of a reference pixel are defined by analogy in the position of the reference pixel. The pixel analogized as described above is defined as a virtual pixel.

Such processes of the embodiment are based on the facts in that, in general, adjacent pixels have strong correlations in image information and most photographic subjects have broad wavelength characteristics to a certain extent, not formed by the components of a specific wavelength.

FIGS. 4A to 4C are schematic views to illustrate generations of a virtual B signal and a virtual R signal, FIG. 4A showing the color filter array 3 in a bayer array, FIG. 4B illustrating the generation of a virtual pixel Bv 22 from a reference pixel G 22 , and FIG. 4C illustrating the generation of a virtual pixel Rv 22 from the reference pixel G 22 .

First, as shown in FIG. 4B , a method for generating the virtual pixel Bv 22 from the reference pixel G 22 will be described. In the neighborhood of the reference pixel G 22 , there exist four pixels of G 11 , G 13 , G 31 and G 33 and two pixels of B 21 and B 23 . The B signal at the position of the reference pixel G 22 , i.e. the color signal of the virtual pixel Bv 22 , is not far removed from the color signals of the two B pixels B 21 and B 23 of adjacent pixels as mentioned above. However, the magnitude of the color signal is not accurately known.

Therefore, in the embodiment, with the magnitude of the color signal of the reference pixel G 22 as a reference, by using B signals of the two adjacent pixels B 21 and B 23 and G signals of the four neighboring pixels G 11 , G 13 , G 31 and G 33 , the ratio between B components and G components is calculated to generate the B components at the position of the reference pixel.

More specifically, as shown in the following Equation 4, with the average value of the two B signals of the adjacent pixels B 21 and B 23 ((B 21 +B 23 )/2) and the average value of the four G signals of the adjacent pixels G 11 , G 13 , G 31 and G 33 ((G 11 +G 13 +G 31 +G 33 )/4), the ratio therebetween (2(B 21 +B 23 )/(G 11 +G 13 +G 31 +G 33 )) is calculated and then multiplied by the G 22 signal to define a virtual Bv signal.

Bv=G 22×2( B 21 +B 23)/( G 11+ G 13+ G 31+ G 33)  Equation 4

Likewise, as shown in FIG. 4C , a virtual Rv is calculated by using the reference pixel G 22 . In the neighborhood of the pixel G 22 , there exist the four G signals of G 11 , G 13 , G 31 and G 33 and two signals of R 12 and R 32 . Therefore, the R signal at the position of the reference pixel G 22 , i.e. the virtual Rv signal, can be defined, with the magnitude of the G 22 signal as a reference and the calculation of the ratio between the neighboring R components and G components, by replacing the G components with the R components. When this calculation is carried out, the virtual Rv signal at the position of the reference pixel G 22 is obtained by the following Equation 5.

Rv=G 22×2( R 12+ R 32)/( G 11+ G 13+ G 31+ G 33)  Equation 5

By the above Equations 4 and 5, the virtual Bv signal and the virtual Rv signal at the position of the reference pixel G 22 are obtained.

Then, by shifting one pixel by one pixel in vertical and horizontal directions, the virtual Bv signal and virtual Rv signal are calculated at the position of the reference pixel G 33 of a new pixel block. As shown in FIG. 2 , in the neighborhood of the reference pixel G 33 , there exist four pixels of G 22 , G 24 , G 42 and G 44 and two pixels of R 32 and R 34 . Therefore, the R signal at the position of the reference pixel G 33 , i.e. the virtual Rv signal, is obtained by the following Equation 6.

Rv=G 33×2( R 32+ R 34)/( G 22+ G 24+ G 42+ G 44)  Equation 6

Likewise, the virtual Bv signal is calculated at the position of the reference pixel G 33 . In the neighborhood of the reference pixel G 33 , there exist the four pixels of G 22 , G 24 , G 42 and G 44 and two pixels of B 23 and B 43 . Therefore, the B signal at the position of the reference pixel G 33 , i.e. the virtual Bv signal, is obtained by the following Equation 7.

Bv=G 33×2( B 23+ B 43)/( G 22+ G 24+ G 42+ G 44)  Equation 7

By the above Equations 6 and 7, the virtual Rv signal and the virtual Bv signal are obtained at the position of the reference G 33 .

As described above, at the pixel position of a G pixel (reference pixel) where no R pixel or B pixel exists, with the pixel value of the G pixel itself and the pixel values of neighboring R or B pixels, the virtual R signal and the virtual B signal can be calculated.

Consequently, as shown in FIGS. 5A and 5B , in a pixel block composed of 3 by 3 pixels, respective color signals of five virtual Rv pixels and five virtual Bv pixels can be calculated.

According to this embodiment, since the number of R pixels and B pixels in a pixel block is increased from two pixels each to seven pixels each, color resolutions and signal to noise ratios of the R signal and B signal can be significantly improved.

The abovementioned virtual pixels Rv and Bv are substantially different in physical properties from an apparent pixel Ba generated by an ordinary interpolation process. The differences will be described in detail below.

When assigning B pixel information Ba at the position of the reference pixel G 22 shown in FIG. 4A by an ordinary interpolation process, since no B pixel exists at the G 22 pixel position, the interpolation process is carried out by the following Equation 8 using the two pixels B 21 and B 23 adjacent to the G 22 pixel.

Ba= ( B 21+ B 23)/2  Equation 8

The above Equation 8 is simply an averaging process, not adding any new pixel information. The pixel information Ba added by such interpolation process is merely the average of the neighboring B signals of B 21 and B 23 and the noise contained in the B 21 signal and the B 23 signal are also averaged, thus never improving the signal to noise ratio.

On the contrary, in the virtual pixel of this embodiment, as shown in Equations 4 to 7 above, the pixel values are calculated directly using the pixel values of the pixel position of the reference pixels G 22 and G 33 where the virtual Rv signal or the virtual Bv signal are to be generated. Further, not only the average of the neighboring R signals or B signals is calculated, but also the ratio between the average of the neighboring R signals or B signals and the average of the neighboring G signals is calculated.

More specifically, in the embodiment, the virtual pixels Bv and Rv are calculated by multiplying the abovementioned ratio with the pixel value of the reference pixel. Consequently, in the embodiment, rather than obtaining the virtual pixels by averaging the neighboring pixels, the new pixel information is created.

The reasons why the new pixel information can be created by the above procedure are because brightness information is composed of R signal, G signal and B signal and most of images have correlations in horizontal and vertical directions. More specifically, when the brightness of most photographic subjects varies, the R signal, G signal and B signal vary nearly in proportion. Therefore, when one of the RGB signals, for example the G signal, is increased, the remaining R signal and B signal are also increased in most cases, thus it just means that the R signal and the B signal can be generated from the G signal.

Meanwhile, in most of images, the R signal, the G signal and the B signal respectively have, in a two-dimensional space, strong correlations in the horizontal direction and in the vertical direction. Except for special patterns artificially created by computers and such, significant changes of the R signal, G signal and B signal by each of the pixels of the image sensor 2 will never occur. Minute changes express contrast in black and white information and are unlikely to contain color information.

Therefore, it is unlikely that the ratio of the RGB signal components obtained from each of the pixels significantly differs from the ratio of the RGB signal components in the neighborhood thereof. Consequently, the ratios of R to G and B to G are unlikely to change by each of the pixels but gradually change over several pixels.

With the magnitude of the G signal as a reference, by multiplying the ratio between the neighboring B signal and the G signal, while maintaining the brightness information which the G signal contains, the B signal can be created. More specifically, the brightness information is obtained from the G signal and the color information is obtained by calculating the ratio between the neighboring B and G signals and the respective average pixel values. Likewise, with the magnitude of the G signal as a reference, by multiplying the ratio between the neighboring R signal and the G signal, the R signal is created. These calculations are carried out by the virtual pixel arithmetic circuit 12 .

FIG. 6 is a block diagram illustrating an example of an internal structure of the virtual pixel arithmetic circuit 12 . As shown in the drawing, the virtual pixel arithmetic circuit 12 has a neighboring B signal extraction circuit 21 , a neighboring R signal extraction circuit 22 , a neighboring G signal extraction circuit 23 and an arithmetic circuit 24 . In the below descriptions and in FIG. 6 , the reference pixel is defined as G 22 .

The neighboring B signal extraction circuit 21 extracts signals of two pixels B 21 and B 23 in the neighborhood of the reference pixel G 22 which is a target for generating the virtual R signal and the virtual B signal. The neighboring R signal extraction circuit 22 extracts signals of two pixels R 12 and R 32 in the neighborhood of the reference pixel G 22 . The neighboring G signal extraction circuit 23 extracts signals of four pixels G 11 , G 13 , G 31 and G 33 in the neighborhood of the reference pixel G 22 .

The arithmetic circuit 24 generates, for example, the virtual Bv signal in accordance with Equation 4. Likewise, the arithmetic circuit 24 also generates the virtual Rv signal in accordance with Equation 5.

FIG. 7 is a block diagram illustrating an example of a detailed structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 shown in FIG. 1 . As shown in the drawing, the synchronous circuit 11 has a 1H delay circuit 31 , a 2H delay circuit 32 , and 1-pixel delay circuits 33 and 34 . The virtual pixel arithmetic circuit 12 has the neighboring B signal extraction circuit 21 , the neighboring R signal extraction circuit 22 , the neighboring G signal extraction circuit 23 and the arithmetic circuit 24 . The neighboring B signal extraction circuit 21 has a B adder circuit 35 and a halving (½) circuit 36 . The neighboring R signal extraction circuit 22 has an R adder circuit 37 and a ½ circuit 38 . The neighboring G signal extraction circuit 23 has a G adder circuit 39 and a quartering (¼) circuit 40 . The arithmetic circuit 24 has a B to G arithmetic circuit 41 , an R to G arithmetic circuit 42 , a Bv calculation circuit 43 and an Rv calculation circuit 44 .

The synchronous circuit 11 makes output signals from the image sensor 2 , depending on the pixel position of the pixel block, delay either 1 line or 2 lines by the 1H delay circuit 31 or the 2H delay circuit 32 and thereafter further delay for one pixel or two pixels by the 1-pixel delay circuits 33 and 34 so as to be processed in parallel by the unit of a pixel block composed of 3 by 3 pixels. Consequently, as shown in FIG. 4 , the signals of eight neighboring pixels centered on the reference pixel G 22 can be processed simultaneously.

The neighboring B signal extraction circuit 21 in the virtual pixel arithmetic circuit 12 adds the B signals of two pixels neighboring the reference pixel in the B adder circuit 35 and averages the B signals of these two pixels in the ½ circuit 36 . Likewise, the neighboring R signal extraction circuit 22 adds the R signals of two pixels neighboring the reference pixel in the R adder circuit 37 and averages the R signals of these two pixels in the ½ circuit 38 . Similarly, the neighboring G signal extraction circuit 23 adds the G signals of four pixels neighboring the reference pixel in the G adder circuit 39 and averages the G signals of these four pixels in the ¼ circuit 40 .

The B to G arithmetic circuit 41 in the virtual pixel arithmetic circuit 12 calculates the ratio between the average value of the B signals of the two pixels neighboring the reference pixel and the average value of the G signals of the neighboring four pixels. The R to G arithmetic circuit 42 calculates the ratio between the average value of the R signals of the two pixels neighboring the reference pixel and the average value of the G signals of the neighboring four pixels. The Bv calculation circuit 43 outputs the value of the ratio calculated by the B to G arithmetic circuit 41 multiplied by the G signal of the reference pixel G 22 as the virtual Bv signal. The Rv calculation circuit 44 outputs the value of the ratio calculated by the R to G arithmetic circuit 42 multiplied by the G signal of the reference pixel G 22 as the virtual Rv signal.

The virtual Bv and Rv signals are fed to the property conversion circuit 13 together with the original RGB signals and color signals of YUV or RGB are generated.

When the process for one pixel block is completed, by shifting two pixels in a vertical direction or horizontal direction, with a pixel block of new 3 by 3 pixels as a unit, the virtual Bv and Rv signals corresponding to the reference pixel in the center thereof are calculated.

FIG. 8 is a flowchart illustrating an example of a process operation of the virtual pixel arithmetic circuit 12 . First, the RGB signals of the pixel block of 3 by 3 pixels including the reference pixel G 22 in the center are extracted (step S 1 ). Next, the processes of extracting the signals of two pixels of B 21 and B 23 neighboring the reference pixel (step S 2 ), extracting the signals of two pixels of R 12 and R 32 neighboring the reference pixel (step S 3 ), extracting the G 22 signal of the reference pixel (step S 4 ), and extracting the signals of four pixels of G 11 , G 13 , G 31 and G 33 neighboring the reference pixel (step S 5 ) are carried out in parallel.

Then, the processes of adding B 21 and B 23 signals extracted in the step S 2 (step S 6 ), adding R 12 and R 32 signals extracted in the step S 3 (step S 7 ), and adding G 11 , G 13 , G 31 and G 33 signals extracted in the step S 5 (step S 8 ) are carried out in parallel.

Next, the processes of averaging the added result in the step S 6 by halving (step S 9 ), averaging the added result in the step S 7 by halving (step S 10 ), and averaging the added result in the step S 8 by quartering (step S 11 ) are carried out in parallel.

Then, the processes of calculating the ratio between the B signals of two pixels neighboring the reference pixel and the G signals of four neighboring pixels (step S 12 ) and calculating the ratio between the R signals of two pixels neighboring the reference pixel and the G signals of four neighboring pixels (step S 13 ) are carried out in parallel.

Thereafter, the processes of calculating the virtual Bv signal by multiplying the calculation result in the step S 12 by the G 22 signal (step S 14 ) and calculating the virtual Rv signal by multiplying the calculation result in the step S 13 by the G 22 signal (step S 15 ) are carried out in parallel.

While examples of virtual Rv and Bv pixels are described above, the general expression of the virtual Rv and Bv pixels will be described below.

When the reference pixel Ga,b is located in an even numbered row shown in FIG. 2 , i.e. the reference pixel Ga,b is at G 22 , G 24 , . . . , G 42 , G 44 , and so on, the B pixels are calculated by using Ba,b−1 and Ba,b+1 signals of both horizontally adjacent sides of the reference pixel Ga,b and the R pixels are calculated by using Ra−1,b and Ra+1,b signals of both vertically adjacent sides of the reference pixel Ga,b. Therefore, the virtual Rv and the virtual Bv pixels in even numbered rows are expressed by the following Equations 9 and 10.

Bva,b= 2 Ga,b ( Ba,b −1 +Ba,b+ 1)/( Ga −1 ,b −1 +Ga− 1 ,b+ 1 +Ga+ 1 ,b− 1 +Ga+ 1 ,b+ 1)  Equation 9

Rva,b= 2 Ga,b ( Ra −1 ,b+Ra+ 1 ,b )/( Ga −1 ,b− 1 +Ga− 1 ,b+ 1 +Ga+ 1 ,b− 1 +Ga+ 1 ,b+ 1)  Equation 10

Meanwhile, when the reference pixel Ga,b is located in an odd numbered row shown in FIG. 2 , i.e. the reference pixel Ga,b is at G 11 , G 13 , . . . , G 31 , G 33 , and so on, the B pixels are calculated by using Ba−1,b and Ba+1,b signals of both vertically adjacent sides of the reference pixel Ga,b and the R pixels are calculated by using Ra,b−1 and Ra,b+1 signals of both horizontally adjacent sides of the reference pixel Ga,b. Therefore, the virtual Rv and the virtual Bv pixels in odd numbered rows are expressed by the following Equations 11 and 12.

Bva,b= 2 Ga,b ( Ba− 1 ,b+Ba+ 1 ,b )/( Ga− 1 ,b− 1 +Ga− 1 ,b+ 1 +Ga+ 1 ,b− 1 +Ga+ 1 ,b+ 1)  Equation 11

Rva,b= 2 Ga,b ( Ra,b− 1 +Ra,b+ 1)/( Ga− 1 ,b− 1 +Ga− 1 ,b+ 1 +Ga+ 1 ,b− 1 +Ga+ 1 ,b+ 1)  Equation 12

By generating the virtual Rv and the virtual Bv pixels in accordance with the above Equations 9 to 12, the number of pixels for R and B becomes three times more than that of the original pixels and the signal to noise ratios of the color signals are improved by 9.5 dB.

When generating the virtual Rv and the virtual Bv pixels by the abovementioned method, B pixel components are not assigned, as shown in FIG. 9A , at the R pixels next to the virtual Bv pixels (blank areas shown in the drawing) and, as shown in FIG. 9B , R pixel components are not assigned at the B pixels next to the virtual Rv pixels (blank areas shown in the drawing). In order to assign the R pixel components or the B pixel components to all pixels, the size of the pixel block needs to be expanded larger than 3 by 3 pixels to, for example, 3 by 5 pixels.

For example, with regard to an R 34 pixel which is the center of the pixel block of 3 by 5 pixels shown in solid bold line in FIG. 10 , an example of generating B pixel components Bv 34 will be described hereinafter. In an area of 5 pixels wide and 3 pixels tall centering on the R 34 , two R pixels and four B pixels are included in addition to the R 34 pixel. Therefore, when the ratio is calculated from these six pixels similar to the above equations, the virtual pixel Bv 34 at the pixel position of R 34 is expressed by the following Equation 13.

Bv 34 =R 34( B 23+ B 25+ B 43+ B 45)/2( R 32+ R 36)  Equation 13

In Equation 13, the virtual pixel Bv 34 is generated at the position of the reference pixel by multiplying the ratio between the average value of four B pixels and the average value of two R pixels in the pixel block by the pixel value R 34 of the reference pixel.

Likewise, in a pixel block of 3 pixels tall and 5 pixels wide centering on the B 23 pixel as shown in a broken line in FIG. 10 , two B pixels and four R pixels are included in addition to the B 23 pixel. Therefore, when the ratio is calculated from these six pixels similar to the above equations, the signal of the virtual pixel Rv 23 at the B 23 pixel position is expressed by the following Equation 14.

Rv 23 =B 23( R 12+ R 14+ R 32+ R 34)/2( B 21+ B 25)  Equation 14

While the pixel block of 3 pixels tall by 5 pixels wide is described above, the virtual Rv or the virtual Bv pixels at a reference pixel position may be generated using pixel values in a pixel block of 5 pixels tall by 5 pixels wide centering on the reference pixel shown in FIG. 10 . In this case, since the number of R pixels used to generate the virtual Bv 34 is increased from two pixels to four pixels, the ratio of B to R can be obtained with a higher signal to noise ratio.

When generating the virtual Rv and the virtual Bv pixels using Equations 13 and 14 for the pixel blocks of 3 by 5 pixels or 5 by 5 pixels, calculations are likely to become complex. As an easier method to avoid such complication, it may be possible to interpolate by averaging the neighboring pixels. For example, the virtual Ba 32 at the position of the R 32 shown in FIG. 10 can be interpolated by the following Equations 15 or 16. The same can be applied to the virtual Rv.

Ba 32=( B 21+ B 23+ B 41+ B 43)/4  Equation 15

Ba 32=( G 31+ G 33+ G 22+ G 42)/4  Equation 16

The arithmetic processes of Equations 13 and 14 above are carried out by the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 shown in FIG. 1 . FIG. 11 is a block diagram illustrating an example of an internal structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to perform calculations of Equations 13 and 14 above. The synchronous circuit 11 has three cascade coupled 1H delay circuits 51 to 53 and five cascade coupled 1-pixel delay circuits 54 to 58 .

The virtual pixel arithmetic circuit 12 has an R adder circuit 59 , a B adder circuit 60 , a doubler circuit 61 , an R to B arithmetic circuit 62 and a multiplier circuit 63 to calculate the virtual Rv pixels; a B adder circuit 64 , an R adder circuit 65 , a doubler circuit 66 , a B to R arithmetic circuit 67 and a multiplier circuit 68 to calculate the virtual Bv pixels; and a property conversion circuit 13 .

With an example of generating virtual Rv 23 pixel and virtual Bv 34 pixel shown in FIG. 10 , the operation of circuits shown in FIG. 11 will be described below. The 1H delay circuit 51 outputs a pixel value of B 21 . The 1H delay circuit 52 outputs a pixel value of G 31 . The 1H delay circuit 53 outputs a pixel value of B 41 . The 1-pixel delay circuit 54 outputs respective pixel values of, in an order from the top, R 12 , G 22 , R 32 and G 42 . The 1-pixel delay circuit 55 outputs respective pixel values of G 13 , B 23 , G 33 and B 43 in an order from the top. The 1-pixel delay circuit 56 outputs respective pixel values of R 14 , G 24 , R 34 and G 44 in an order from the top. The 1-pixel delay circuit 57 outputs respective pixel values of B 25 , G 35 and B 45 in an order from the top. The 1-pixel delay circuit 58 outputs a pixel value of R 36 .

The R adder circuit 59 calculates (R 12 +R 14 +R 32 +R 34 ). The B adder circuit 60 calculates (B 21 +B 25 ). The doubler circuit 61 calculates 2(B 21 +B 25 ). The R to B arithmetic circuit 62 calculates (R 12 +R 14 +R 32 +R 34 )/2(B 21 +B 25 ). The multiplier circuit 63 finally calculates Equation 14 to generate the Rv 23 .

The B adder circuit 64 calculates (B 23 +B 25 +B 43 +B 45 ). The R adder circuit 65 calculates (R 32 +R 36 ). The doubler circuit 66 calculates 2(R 32 +R 36 ). The B to R arithmetic circuit 67 calculates (B 23 +B 25 +B 43 +B 45 )/2(R 32 +R 36 ). The multiplier circuit 68 finally calculates Equation 13 to generate the Bv 34 .

The generated virtual Rv 23 signal and virtual Bv 34 signal are fed into the property conversion circuit 13 and color signals of YUV or RGB are generated.

In the above, an example of calculating an average from both horizontal and vertical directions has been described when the virtual R signals and the virtual B signals are in the neighborhood of the reference G pixel. On the contrary, the calculation of the average may be carried out only in a horizontal direction or in a vertical direction.

FIGS. 12A to 12C are schematic views to illustrate calculation of virtual pixels only in a horizontal direction, FIG. 12A showing a bayer array, FIG. 12B illustrating the generation of a virtual Bv pixel, and FIG. 12C illustrating the generation of a virtual Rv pixel. In the case shown in FIG. 12 , at the reference pixel of an each pixel block, only either one of the virtual Rv pixel or the virtual Bv pixel is generated. More specifically, the virtual Rv pixel and the virtual Bv pixel are generated from pixel blocks different from each other. Therefore, as shown in FIG. 13 , at the position of original G pixels, the R signals and B signals are alternately defined in the vertical direction. FIG. 13A is a schematic view to illustrate a pixel array of generated virtual Bv pixels and FIG. 13B is a schematic view to illustrate a pixel array of generated virtual Rv pixels.

Meanwhile, FIGS. 14A to 14C illustrate the calculation of virtual pixels only in a vertical direction. Further, in this case, at the reference pixel of an each pixel block, only either one of the virtual Rv pixel or the virtual Bv pixel is generated. Consequently, as shown in FIG. 15 , the R signals and B signals are alternately defined in the horizontal direction.

In the cases shown in FIGS. 12 to 15 , in a single pixel block, only either one of the virtual Rv pixel or the virtual Bv pixel is generated. More specifically, the pixel block to generate the virtual Rv pixel is different from the pixel block to generate the virtual Bv pixel. In this regard, the abovementioned cases differ from the method which generates both the virtual Rv pixel and the virtual Bv pixel in one pixel block as shown in FIG. 4 .

The methods described in reference with FIGS. 12 to 15 have a feature in that the R signal and B signal are easily obtainable when de-mosaicing in consideration of alias signals. More specifically, when obtaining correlations in the vertical direction and the horizontal direction, since the relations of pixels are clear, the process of de-mosaicing can be simplified and interference signals by the generation of aliasing signals are unlikely to occur.

While it is less likely to happen with natural images, there may be a case of taking an image of a photographic subject close to a single wavelength of red or blue in some situations. When taking the image of such subject, in comparison with R signal and B signal, G signal components become extremely small. In this case, when the R signal and B signal are calculated by the abovementioned virtual pixel calculations, the signals may inversely become more noisy. In this case, the abovementioned calculation process may be halted to prevent the signal to noise ratio from lowering. More specifically, by defining a threshold of signal, when G signal becomes a level smaller than a certain level, the abovementioned calculation process may be halted not to generate the virtual Bv and Rv signals. In this case, before performing the process of step S 1 , a process for determining whether the G signal is equal to or more than a prescribed threshold value is provided. Only when the determination is YES, the process of step S 1 and the subsequent processes are performed, and when the determination is NO, the process of step S 1 and the subsequent processes are omitted. Alternatively, similar to the above, by comparing the Rv signal and Bv signal with the original R signal and B signal, a selector may be provided not to carry out additions when the signal to noise ratios become lower.

In the above calculations, while the virtual Bv signal and the virtual Rv signal are generated with only eight neighboring pixels of the reference G pixel, the virtual Bv signal and virtual Rv signal may be generated using pixel information in a wider area. The wider the area expanded, the lower the influence of noise can be when averaging while signal levels are small.

In the above descriptions, while the average value of pixels neighboring the reference pixel is calculated and then the ratio is obtained based on the calculation result thereof, in place of calculating simple average values, the ratio may be obtained using the calculation result utilizing weighted additions and other various functions.

While the example of using the CMOS sensor 2 as the image sensor 2 is illustrated in the above, the same process can also be applied when a CCD having a color filter of a bayer method is used.

As described above, the embodiment of the present invention has significant characteristics in that, at the pixel position where a color filter of a specific color is not provided, the color information of the specific color thereof can be added. Consequently, the color resolution is improved and the signal to noise ratio of the color signals can be improved.

Further, in the embodiment, since the ordinary image sensor 2 in a bayer array is used, not the special image sensor provided with W pixels, color images of good color characteristics and of a good color reproducibility can be obtained.

Furthermore, while the simple interpolation process in related art merely averages pixels, hence not generating new pixels, in the embodiment, since averaging is not simply carried out, the effect in that the new pixels in other colors can be generated at the position of the reference pixel is obtained.

Now, the effects of the embodiment will be more specifically described.

(1) When the image sensor 2 is of a WRGB array, the level of incoming light is limited by W, thus the signal to noise ratio in standard condition is lowered. On the contrary, in the embodiment, since the RGB array is adopted, the balance of signal outputs becomes better and the signal to noise ratios even in standard condition can be improved to the extent that there are no W pixels.

(2) When using the W pixels, since the R+G+B signals obtained by color filters will not make W signal in a strict sense, even if RGB signals are obtained by calculation, the correct RGB signals are not obtainable. On the contrary, in the embodiment, since R and B signals are calculated based on the correct G signal, the RGB signals in correct characteristics are obtained and the color reproducibility can be improved.

(3) Since the G signal is directly obtained, not by generating virtual G, similar to a bayer method in related art, the G components of good accuracy is obtained and the correct G signal is obtainable. Therefore, the G signal of an accurate spectroscopic characterization is obtained and the color reproducibility is improved. Further, an excellent color reproducibility equivalent to that of a bayer method is maintained.

(4) R and B signals can be equally calculated.

In the case of W pixels, since R signal is obtained by horizontally averaging and B signal is obtained by vertically averaging, while there is a possibility that errors occur when there are correlations in the vertical direction and horizontal direction, the errors can be reduced by respectively calculating virtual Rv signals and virtual Bv signals with the average of adjacent R signals and adjacent B signals in the horizontal direction.

Likewise, the errors can be reduced by respectively calculating virtual Rv signals and virtual Bv signals with the average of adjacent R signals and the average of adjacent B signals in the vertical direction.

FIGS. 16A and 16B are schematic views illustrating a pitch of B pixels of a sensor including W pixels and a spatial frequency thereof and FIGS. 17A and 17B are schematic views illustrating a pitch of B pixels after virtual pixel calculations of the embodiment and a spatial frequency thereof.

In a conventional bayer method, since B pixels are disposed, as shown in FIG. 16A , at a pitch of 2 a horizontally and vertically, the spatial frequency is distributed, as shown in FIG. 16B , over the square range of ½ a both horizontally and vertically.

On the contrary, according to the embodiment, as shown in FIG. 17A , the number of B Pixels are practically increased and, similar to G pixels, diagonal components are increased. Since the pitch of B pixels becomes a, the spatial frequency is, as shown in FIG. 17B , distributed over the square range of 1/a both horizontally and vertically. The broken line shown in FIG. 17B represents the spatial frequency shown in FIG. 16B and it can be found that the spatial frequency is extended. Consequently, according to the embodiment, an improvement of resolution can be achieved.

Furthermore, in an image pickup device of a bayer method in related art, generation of color alias signals has been a major problem. In a bayer array, since G pixels of twice the number of R or B pixels are disposed in a checkerboard pattern, while the spatial frequency of G signal is distributed in a wide range similar to that shown in FIG. 17B , those of R and B signals are in a narrow range as shown in FIG. 16B and in a different square shape. Therefore, characteristics between R and B signals and G signal are different, making it difficult to design an optical LPF. When the design of the optical LPF is tailored to G signals, substantial amount of alias signals of R and B signals are increased and significantly degrade the image quality. When the cut-off frequency of the optical LPF is lowered tailoring to R and B signals, there have been drawbacks in that the overall resolution is lowered and a clear image is not obtainable.

On the contrary, according to the embodiment, since the shape of the spatial frequencies of each of the R, G, and B signals becomes equivalent, the generation of alias signals associated with sampling becomes equivalent. Consequently, the embodiment has significant effects in that the designing of the optical LPF becomes easier, generation of alias signals becomes small as the alias signals can be almost completely removed, and the image quality is significantly improved. Further, since the cut-off frequency of the optical LPF can be set higher, the embodiment has significant features in that the resolution of color image is improved and a clear image is obtainable.

In the abovementioned Equations 4 to 7, while pixel values in the pixel block of 3 by 3 pixels neighboring the center pixel are used when generating virtual Bv signals and virtual Rv signals, by expanding the size of the pixel block to either of 3 by 5 pixels or 5 by 5 pixels, the signal to noise ratios in ratio calculations can be improved. However, while the size of the pixel block is set large, there is a drawback in that errors may increase when image greatly varies. Therefore, it may be configured to select how large the size of the pixel block is to be set according to the contents of an image.

For example, FIG. 18 is a schematic view to illustrate defining a pixel block of 3 pixels tall by 5 pixels wide neighboring the reference pixel G 33 . The virtual pixels Bv 33 and Rv 33 at the position of the reference pixel G 33 are expressed by the following Equations 17 and 18.

Bv 33= G 33( B 21+ B 23+ B 25+ B 41+ B 43+ B 45)/( G 22+ G 24+ G 31+ G 35+ G 42+ G 44)  Equation 17

Rv 33=3 G 33( R 32+ R 34)/( G 22+ G 24+ G 31+ G 35+ G 42+ G 44)  Equation 18

In the above Equation 17, the virtual Bv 33 pixel is generated by multiplying the ratio between the average value of six B pixels and the average value of six G pixels in the pixel block by the reference pixel G 33 .

In the above Equation 18, the virtual Rv 33 pixel is generated by multiplying the ratio between the average value of two R pixels and the average value of six G pixels in the pixel block by the reference pixel G 33 .

In the virtual Bv 33 pixel, since the ratio is calculated by using the average value of B signals for six pixels, the signal to noise ratio of the B to G ratio becomes approximately 6 dB better than that of the virtual Rv 33 pixel which uses the average value of R signals for two pixels.

For the virtual Rv 33 pixel, when pixel values are calculated based on the pixel block shown by a solid bold line in FIG. 19 , the ratio can be calculated by using the average value of R signals for six pixels. Therefore, a similar signal to noise ratio to that of the virtual Bv 33 pixel by Equation 17 can be obtained.

For example, based on the pixel block of 5 pixels tall by 3 pixels wide shown by the solid bold line in FIG. 19 , the virtual pixels Bv 33 and Rv 33 at the position of the reference pixel G 33 are expressed by following Equations 19 and 20.

Bv 33=3 G 33( B 23+ B 43)/( G 13+ G 22+ G 24+ G 42+ G 44+ G 53)  Equation 19

Rv 33 =G 33( R 12+ R 14+ R 32+ R 34+ R 52+ R 54)/( G 13+ G 22+ G 24+ G 42+ G 44+ G 53)  Equation 20

As described above, the pixel block can be either of 3 pixels tall by 5 pixels wide or 5 pixels tall by 3 pixels wide, hence the size of the pixel block may be determined depending on the reference pixel being in an odd row or an even row. It is conceivable that, for example, when the reference pixel is in an odd row, since there are B pixels above and below in the vertical direction, the pixel block of 3 pixels tall by 5 pixels wide is to be used for the calculation of the virtual Bv signals and, for the calculation of the virtual Rv signals, since there are R pixels on left and right in the horizontal direction, the pixel block of 5 pixels tall by 3 pixels wide is to be used. Meanwhile, when the reference pixel is in an even row, since there are R pixels above and below in the vertical direction, the pixel block of 3 pixels tall by 5 pixels wide is to be used for the calculation of the virtual Rv signals and, for the calculation of the virtual Bv signals, since there are B pixels on left and right in the horizontal direction, the pixel block of 5 pixels tall by 3 pixels wide is to be used. Consequently, the virtual Rv signals and virtual Bv signals can be efficiently generated with less number of pixels.

FIG. 20 is a block diagram illustrating an example of an internal structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to calculate the above Equations 17 and 18. The synchronous circuit 11 shown in FIG. 20 has cascade coupled 1H delay circuits 71 to 73 and cascade coupled 1-pixel delay circuits 74 and 77 . The virtual pixel arithmetic circuit 12 has an R adder circuit 78 , a G adder circuit 79 , a B adder circuit 80 , a B to G calculation circuit 81 , a multiplier circuit 82 , a tripler circuit 83 , an R to G calculation circuit 84 and a multiplier circuit 85 . The multiplier circuit 82 outputs the Bv 33 signal of the calculation result of Equation 17 and the multiplier circuit 85 outputs the Rv 33 signal of the calculation result of Equation 18.

FIG. 21 is a block diagram illustrating an example of an internal structure of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to calculate the above Equations 19 and 20. The synchronous circuit 11 shown in FIG. 21 has cascade coupled 1H delay circuits 91 to 94 and cascade coupled 1-pixel delay circuits 95 and 96 . The virtual pixel arithmetic circuit 12 has an R adder circuit 97 , a G adder circuit 98 , a B adder circuit 99 , a B to G calculation circuit 100 , a tripler circuit 101 , a multiplier circuit 102 , an R to G calculation circuit 103 and a multiplier circuit 104 . The multiplier circuit 102 outputs the Bv 33 signal of the calculation result of Equation 19 and the multiplier circuit 104 outputs the Rv 33 signal of the calculation result of Equation 20.

The internal structures of the synchronous circuit 11 and the virtual pixel arithmetic circuit 12 to generate virtual pixels are not limited to those shown in FIGS. 7 , 11 , 20 and 21 .