Method and apparatus for reduction of residual signal in digital images captured by image sensors

This invention relates to the field of digital imaging and, more particularly, to an apparatus and method for reducing the residual signal in digital images captured by a digital camera using an image sensor.

A traditional camera system includes camera body, lens, electronics to control the functionality of the camera and most importantly the image-capturing media—film. In contrast, instead of using film, a digital camera system usually employs an image sensor made on semiconductor substrate. Typically the image sensor is either a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor. There are several types of CCD sensors, such as frame transfer CCD, full frame CCD, interline CCD and linear CCD sensors. For purposes of an illustrative embodiment, the image sensor of a full frame CCD is discussed for illustrating the effects of a residual signal.

A full frame CCD consists of a two-dimensional array of photosensitive elements of X columns by Y rows. The sensor captures light during exposure and converts photons to electrical charge, namely electrons. At the end of exposure, it starts the readout process. First it transfers all the electrons vertically row by row to a serial of horizontal registers. After each row transfer, the charge in the horizontal registers is transferred out to a sensing node, namely a floating diffusion, and converted to a voltage signal. An analog-to-digital converter converts this voltage to a digital count. A single image capture is completed when all the rows of signals are transferred out and converted. Finally the onboard electronics renders all the digital counts into a digital image. At this time, the camera is ready for the next capture.

A digital camera with a full frame CCD captures a single still image in the following manner. When a mechanical shutter is pressed, a flush cycle with a fixed amount of time is executed to clear out the existing charge in the CCD (see Japanese Patent No: JP62152281 for further discussion). This is because the existing charge, namely the dark current, is accumulated during either the camera power off period or camera standby period. This dark current needs to be cleared out. Then the exposure process begins and charge starts to accumulate in the CCD. The close of the shutter ends the exposure and starts the readout process in which the charge is transferred out of the CCD and converted to a digital image.

The digital camera with a full frame CCD usually only captures still images, not videos. The reason for this is that for the high frame rate required by the video mode, a mechanical shutter is not practical to use. As a result, photon-induced charge continues to accumulate even after the exposure period when the accumulated charge is being transferred out of CCD. This additional charge results in an artifact, namely smear, to appear in the images. Therefore, video capture usually is not an option for a digital camera using a full frame CCD sensor. There are ways to reduce smear without a mechanical shutter (see US Patent Application Publication No: US2002/0186308A1).

However, many high-end digital cameras using a full frame CCD sensor have a so-called continuous capture mode, which allows users to continuously capture several images in a row. In order to get high quality images for all images, several obstacles need to be overcome and they are smear, dark current and the residual signal. The smear problem can be solved by using a mechanical shutter if the frame rate is not very fast. For example, the frame rate for an 8 million pixel CCD operating at 30 Mhz pixel rate is about 0.3 seconds, which is long enough for a mechanical shutter to operate smoothly. Removal of dark current can be achieved by subtracting a dark image with the same exposure time captured at the end of continuous capture mode with the shutter closed.

The residual signal, however, cannot be solved by either a mechanical shutter or a dark current subtraction. It is defined that charge related to the previous image is superimposed on the next image when the camera operates in its continuous capture mode. Referring to FIG. 1A and FIG. 1B , there are shown two scene settings for testing residual signal in a full frame CCD. In both cases, there is a candle 1 on a table. In FIG. 1A , the candle 1 is lit and has a flare 2 while in FIG. 1B the candle 1 is put off and the flare is gone. Referring to FIG. 1C and FIG. 1D , two images continuously captured from these two scenes using a camera in its two-image-capture mode are shown. In the FIG. 1D a residual image 3 of the flare 2 is still visible in the image although in the real scene ( FIG. 1B ) there is no such flare 2 .

When light strikes onto a full frame CCD during the first image exposure, some photon-generated charges are trapped in some pixels due to the crystal defects in the silicon. After exposure, these traps start to release trapped charge gradually. If during the second image exposure and readout, these traps still release charge, the total charge in the second image will be the summation of the charge due to second exposure and the charge released from the traps generated during the first exposure. Therefore, the second image will have a residual image of the first image in it. This phenomenon will become especially worse when the first image has a strong illumination scene such as FIG. 1A .

The residual charge is due to the thermal release of charge from trapping sites in a CCD. The number of traps is exponentially proportional to light intensity and exposure time. Referring to FIG. 2 , there is shown the relationship between the residual signal generated at the end of the first exposure and the exposure time for different light intensities. Under light intensity of 0.5 Esat, the residual signal increases from 0 to about 45 electrons for a five-second exposure. Esat is a relative light intensity unit. If an image sensor's output equals to its saturation level when it is illuminated under a light source for a fixed amount of exposure time, that light intensity is defined as 1 Esat. At the same exposure time, the residual signal also increases with the light intensity. At 20 Esat, it reaches about 115 electrons at 5 seconds.

After the first exposure, the traps will release the trapped charge and the process will slow down and eventually diminish with time. Theoretically, by adding infinite flush time between the end of the first image readout and the start of second image exposure, the residual signal will be eliminated in the second image because all the charge in the traps will be released and flushed out during this infinite time before the next exposure begins. However, in the practical use, the flush time cannot be set for too long. Otherwise, a camera user may feel a delay between each continuous capture and that delay may result in missing important scenes.

On the other hand, by setting a predefined maximum-allowable flush time sometimes can cause unnecessary slowdown of the continuous capture process if the illumination of a scene is not quite as strong as to generate a substantial residual signal beyond the camera's tolerance.

Therefore, there is a need for a method to automatically adjust the flush time based on the exposure information of a previous image capture and substantially reduce the residual signal in the next image in a continuous capture mode by a digital camera.

The present invention is directed to overcoming one or more of the problems set forth above. Briefly summarized, according to one aspect of the present invention, the invention resides in a method for reducing residual signal from a first image from appearing in a second image, the method comprising the steps of capturing the first image on an image sensor having a known exposure that includes exposure time and a peak light intensity; estimating a flush time for the second image based on the known exposure in capturing the first image; capturing the second image using the estimated flush time for substantially reducing the residual signal.

It is therefore an object of the present invention to automatically adjust the flush time based on the exposure information of a previous image capture and substantially reduce the residual signal in the next image in a continuous capture mode by a digital camera using an image sensor.

This object is achieved by a method to determine an optimal flush time based on the relationship between the residual signal and the flush time. The residual signal in the next image is estimated based on the previous exposure which is the multiplication of the light intensity measured by a light meter and the exposure time.

FIGS. 1A-1D are illustrations of a residual signal;

FIG. 2 is a chart showing the relationship between the residual signal generated at the end of the exposure and the exposure time for different light intensities;

FIG. 3 is a block diagram for a digital camera system in accordance with the invention;

FIG. 4 is a block diagram of the image sensor timing to reduce the residual signal in the second image for a two-image-capture operation;

FIG. 5 is a chart showing the relationship between the residual signal and the flush time in the second image; and

FIG. 6 is a flow chart of a method of automatically reducing the residual signal for a two-image-capture operation.

Referring to FIG. 3 , there is shown a representative diagram showing a digital camera system 8 in accordance with the present invention. An image sensor 14 (preferably a CCD) is positioned in front of a lens with a mechanical shutter (the combination of the lens and the shutter is illustrated as 10 ). A light meter 12 measures the light intensity of the scene of interest passing through the lens-shutter fixture 10 . This value is saved in an onboard memory 32 for the future use. A microcontrol processor 22 controls the lens fixture 10 and it also controls a flush timing control 28 . The image sensor 14 is driven by clock drivers 20 and a clock-timing generator 30 which is also controlled by the microcontrol processor 22 . The analog-to-digital chain includes an analog gain and correlated double sampling (CDS) unit 16 and an analog-to-digital converter (ADC) 18 . The digital counts related to the scene of interest are processed by a digital signal processor (DSP) circuit 24 and finally transferred to either a viewing device 26 like a display or a memory card like a flash memory 34 . The onboard memory 32 saves exposure information of the last image such as exposure time and light intensity. The microcontrol processor 22 uses the information to calculate the flush time for the next image capture if the camera system is operated in a continuous image capture mode.

Referring to FIG. 4 , there is shown a diagram showing the timing sequence in accordance with the present invention, assuming the digital camera system continuously captures two images in a row. The 1 st flush 102 is a flush cycle which is used to primarily remove the dark current in the image sensor pixels during either power off or standby period. Then the 1 st exposure 104 is initiated. After 1 st exposure 104 , 1 st readout 106 is executed and the first image is outputted. Immediately, based on the 1 st exposure 104 including the exposure time (derived from the camera) and the light intensity, the microcontrol processor 22 calculates the optimal flush time for the 2 nd flush 108 through either the lookup tables or the predefined fit-curves and starts the second image capture process including 2 nd exposure 110 and 2 nd readout 112 . The 2 nd flush 108 will substantially remove the residual signal while in the meantime it keeps the delay between two images captured at minimum.

The process of determination of the optimal flush time is based on the relationship between the residual signal and the flush time for the second capture and this relationship is shown in FIG. 5 . A threshold of the residual signal is defined as the maximum residual signal which a digital camera system can tolerate without sacrificing the image quality it captures. Suppose the estimated residual signal which will appear in the second image if, without any flush, is 90 electrons and the threshold is only is 50 electrons, the necessary flush time will be t 1 -t 0 , i.e., 0.75 seconds to remove the residual signal in the second image to the level which the camera can tolerate.

The onboard memory 32 in the digital camera system can store a plurality of lookup tables of the residual signal versus exposure time for different light intensities ( FIG. 2 ) and another lookup table of the residual signal versus flush time ( FIG. 5 ). Therefore, once the previous light intensity and exposure time are known, the system can immediately calculate the optimal flush time for the next capture by searching these lookup tables.

Another method is to store the parameters of a plurality of fit-curves of the residual signal versus exposure time for different light intensities ( FIG. 2 ) and the parameters of another fit-curve of the residual signal versus flush time ( FIG. 5 ). For example, in a linear curve, the only parameters of the fit-curve that are stored are the slope and the intercept. The flush time will then be interpolated and calculated based on these parameters of the fit-curves.

Referring to FIG. 6 , there is shown a flowchart of a 2-image-capture mode in accordance to the present invention. When this mode 202 starts, a user presses a shutter button halfway to measure light intensity 204 . After the button is fully pressed down 206 , the 1 st fixed flush 208 is executed to remove the background dark current. Then the mechanical shutter is opened to start 1 st exposure 210 . The light intensities and exposure time are saved in the onboard memory 32 for later retrieval. After that, the mechanical shutter is closed to start 1 st readout 212 . Once finishing the 1 st readout 212 , the microcontrol processor 22 first retrieves the exposure information of the measured light intensity and the exposure time from the onboard memory 32 and then estimates the residual signal which will appear in the second image if without any flush based on the relationship between the residual signal and exposure ( FIG. 2 ). Finally it calculates the optimal flush time 214 based on a predefined threshold of residual signal level ( FIG. 5 ) and the relationship between the residual signal and the flush time. A comparison 216 decides if an optimal flush time 218 or a maximum allowable flush time 220 is used. A maximum flush time 220 is the maximum time the camera system allows before a user will not notice a significant amount of delay between two continuously captured images. This only occurs when the scene has an extreme bright illumination which generates a residual signal so large that a maximum flush time is needed to reduce the residual signal. After the flush, the mechanical shutter is opened again and starts the 2 nd exposure 222 , and then the shutter is closed and the 2 nd readout 224 is executed. The 2-image-capture mode 226 ends with two images either stored in memory card 34 or outputted to a viewing device 26 .

This invention has been described in detail with a particular configuration of a two-image continuous capture mode, but it will be understood that it can also apply to N-image continuous capture mode.

1 candle

2 flare

3 residual image

8 digital camera system

10 lens-shutter fixture

12 light meter

14 CCD image sensor

16 analog gain/CDS unit

18 A/D converter

20 clock drivers

22 microcontrol processor

24 DSP circuit

26 viewing device

28 flush timing control

30 clock timing generator

32 onboard memory

34 memory card

102 1 st flush

104 1 st exposure

106 1 st readout

108 2 nd flush

110 2 nd exposure

112 2 nd readout

202 start

204 meter exposure/light intensity

206 start capture

208 1 st fixed flush

210 1 st exposure

212 1 st readout

214 calculate flush time

216 comparison

218 2 nd optimal flush

220 maximum flush

222 2 nd exposure

224 2 nd readout

226 2-frame-capture end