Image synthesizing apparatus, image recording method, and recording medium

This application is based on and claims the benefit of priority from Japanese Patent Application No. 2011-248251, respectively filed on 14 Nov. 2011, the content of which is incorporated herein by reference.

1. Field of the Invention

The present invention relates to an image synthesizing apparatus, an image recording method, and a recording medium.

2. Related Art

Conventionally, a technique has been known, in which additive synthesis is executed on pixels of continuously captured images, thereby generating an image with sufficient brightness for recording, even in a dark photographic environment with a low level of light such as a night view.

Japanese Unexamined Patent Application, Publication No. 2006-148550 discloses an image synthesis technique for correcting blurring in an image capturing field angle, by continuously capturing images and synthesizing the images.

Incidentally, when a stroboscopic flash unexpectedly emits light during the process of exposure and changes the brightness, an image with uneven brightness may be acquired.

If such an image is used for additive synthesis, the change in brightness still remains in a synthetic result, and thus a synthetic image with uneven brightness is eventually generated.

The present invention has been made in view of such problems, and an object of the present invention is to prevent a synthetic result from being affected by an acquired image with uneven brightness.

In order to achieve the above object, an aspect of the present invention is characterized by including: an image acquisition unit that acquires a plurality of images being temporally continuous; a selection unit that selects a single image serving as reference of brightness, from the plurality of images acquired by the image acquisition unit; a detection unit that compares the single image selected by the selection unit with other images acquired by the image acquisition unit, and detects an image with a difference value being higher than a predetermined threshold value as a result of such comparison; a setting unit that sets a synthetic rate, based on the image detected by the detection unit; and an image generation unit that generates an image by executing synthesis of the plurality of images acquired by the image acquisition unit including the image detected by the detection unit, by using the synthetic rate that is set by the setting unit. In order to achieve the above object, another aspect of the present invention is characterized by including: an image acquisition step of acquiring a plurality of images being temporally continuous; a selection step of selecting a single image serving as reference of brightness, from the plurality of images acquired in the image acquisition step; a detection step of comparing the single image selected in the selection step with other images acquired in the image acquisition step, and detecting an image with a difference value being higher than a predetermined threshold value as a result of such comparison; a setting step of setting a synthetic rate, based on the image detected in the detection step; and an image generation step of generating a single image by executing synthesis of the plurality of images acquired in the image acquisition step including the image detected in the detection step, by using the synthetic rate that is set in the setting step. In order to achieve the above object, still another aspect of the present invention is characterized by causing a computer, which controls an image synthesizing apparatus for generating synthetic image data, to execute image synthesis processing including: an image acquisition function of acquiring a plurality of images being temporally continuous; a selection function of selecting a single image serving as reference of brightness, from the plurality of images acquired by the image acquisition function; a detection function of comparing the single image selected by the selection function with other images acquired by the image acquisition function, and detecting an image with a difference value being higher than a predetermined threshold value as a result of such comparison; a setting function of setting a synthetic rate, based on the image detected by the detection function; and an image generation function of generating an image by executing synthesis of the plurality of images acquired by the image acquisition function including the image detected by the detection function, by using the synthetic rate that is set by the setting function.

FIG. 1 is a circuit block diagram of an image capturing apparatus, to which an embodiment of the present invention is applied;

FIG. 2 is a flowchart showing a processing procedure in the image capturing apparatus;

FIG. 3A is a diagram showing an image example in the processing procedure;

FIG. 3B is a diagram showing an image example in the processing procedure;

FIG. 3C is a diagram showing an image example in the processing procedure;

FIG. 3D is a diagram showing an image example in the processing procedure; and

FIG. 3E is a diagram showing an image example in the processing procedure.

An embodiment of the present invention is hereinafter described with reference to the drawings. FIG. 1 is a circuit configuration diagram of an image capturing apparatus including an image processing device according to an embodiment of the present invention. The image capturing apparatus has a function of reading an image signal by a rolling shutter scheme, and includes a drive control unit 2 , a CDS/ADC 3 , a key input unit 4 , a display unit 5 , an image recording unit 6 , program memory 7 , RAM 8 , a control unit 9 , and an image processing unit 10 . These are connected via a bus line. The key input unit 4 includes a shutter key 41 for detecting a recording instruction of a photographer; and the image processing unit 10 includes a synthesis unit 11 .

In FIG. 1 , the image capturing unit 1 has a built-in image sensor such as a CMOS; a built-in RGB color filter provided on the image sensor; and a built-in driver that retains light intensity as charge accumulation for a certain period of time in accordance with control by the drive control unit 2 so as to be output as an analog image capturing signal to the CDS/ADC 3 . A plurality of images (color images) are acquired by detecting image capturing instructions of the photographer, the instructions being routed through the shutter key 41 , the control unit 9 and the drive control unit 2 .

The CDS/ADC 3 is configured by a CDS, a gain adjustment amplifier (AGC), an A/D converter (ADC), etc. An analog image capturing signal of an optical image of a subject is output from the image capturing unit 1 , and is input into the CDS that retains the image capturing signal thus input. The gain adjustment amplifier amplifies the image capturing signal. The A/D converter (ADC) converts the image capturing signal thus amplified into a digital image capturing signal. Control involved with adjustment of the gain adjustment amplifier is also executed based on an instruction from the drive control unit 2 . Therefore, even if a plurality of images are acquired under the same exposure condition (shutter speed or diaphragm stop), the plurality of images can be generated under different conditions by sequentially changing RGB of the gain adjustment amplifier and tints of the images.

Although the drive control unit 2 executes control involved with the gain adjustment amplifier in the present embodiment, the present invention is not limited thereto. For example, the control unit 9 may execute the control.

In addition to the shutter key 41 , the key input unit 4 includes various keys for detecting switchover to a photography mode for acquiring and recording an image according to the present invention, keys for switching display, etc.

The display unit 5 has a function of displaying a synthesized image. After executing synthesis processing according to the present invention, the image recording unit 6 stores image data (image file) coded by a JPEG scheme. The program memory 7 stores a program to be executed by the control unit 9 and the image processing unit 10 ; and the control unit 9 reads the program as necessary. The RAM 8 has a function of temporarily storing data in process, which is generated in each processing. The control unit 9 controls processing operations of the entirety of the image capturing apparatus. In addition to executing encoding/decoding processing of image data, the image processing unit 10 includes the synthesis unit 11 corresponding to a characteristic feature of the present invention.

More specifically, the synthesis unit 11 executes image synthesis of a single reference image selected from continuously captured images, and a synthetic image generated by additive synthesis of the continuously captured images. In addition, the synthesis unit 11 uses a map (to be described below) for the synthetic image as a transparence intensity map (α map) corresponding to a difference area, thereby executing synthesis while changing a synthetic rate for each pixel region, and reducing an influence due to uneven brightness.

Next, descriptions are provided for operations in the present embodiment. When the user operates a mode button provided to the key input unit 4 to set a photography mode according to the present embodiment, the control unit 9 reads a program from the program memory 7 to initiate processing as shown in the flowchart of FIG. 2 .

More specifically, the control unit 9 determines whether depression of the shutter key 41 is detected (Step S 101 ). In a case in which depression of the shutter key 41 is detected, the control unit 9 controls the drive control unit 2 to continuously capture N pieces of images (Step S 102 ). Subsequently, based on image data obtained by the continuous capturing, an image data group of the continuously captured images in the YUV format expressed in color space information of brightness and color difference is generated and stored in the RAM 8 (Step S 103 ).

Therefore, for example, in a case in which the N pieces of continuously captured images are six pieces, continuously captured images P 1 to P 6 are stored in the RAM 8 . It is assumed that blurring of the continuously captured images P 1 to P 6 is corrected by a well-known technique (a field angle correction technique using CCD shift and/or lens shift).

Next, a reference image is selected from the continuously captured images (Step S 104 ). The reference image is selected based on an external operation by the user. For example, the continuously captured images P 1 to P 6 are displayed as a list on the display unit 5 , and the user selects a single image as a reference image from the continuously captured images P 1 to P 6 displayed as the list. In the present embodiment, it is assumed that the continuously captured image P 3 is selected as a reference image.

Subsequently, noise reduction processing is executed on the image data of the selected reference image by using an LPF (Low-Pass-Filter) (Step S 105 ). The image data of the reference image after executing the noise reduction processing is stored in the RAM 8 (Step S 106 ). Therefore, in this case, as a result of the processing in Steps S 105 and S 106 , the noise reduction processing is executed on the image data of the continuously captured image P 3 (reference image), which is then stored into the RAM 8 .

The plurality of continuously captured images are synthesized by executing noise reduction (Step S 107 ). In other words, noise reduction and synthesis can be executed together by averaging the image data of the plurality of continuously captured images for each pixel. Equations 1 to 3 for image data averaging calculation for N pieces of continuously captured images are shown as follows. Y_result, U_result and V_result represent information after executing synthesis of brightness and color difference parameters for each pixel.

Y _result=( Y[ 0 ]+Y[ 1 ]+ . . . +Y[N ])/ N   Equation 1

U _result=( U[ 0 ]+U[ 1 ]+ . . . +U[N ])/ N   Equation 2

V _result=( V[ 0 ]+V[ 1 ]+ . . . +V[N ])/ N   Equation 3

As long as a noise reduction effect is achieved by using a plurality of continuously captured images, the synthetic method is not limited to the averaging, and may be any other processing method. Image data of a synthetic image of all the continuously captured images after executing the noise reduction is stored into the RAM 8 (Step S 108 ).

Therefore, when the processing in Step S 108 is completed, the following image data is stored into the RAM 8 .

(1) Image data of each of the continuously captured images;

(2) Image data of the reference image after executing the noise reduction processing; and

(3) Image data of the synthetic image after executing the noise reduction processing.

Next, an evaluation reference image is generated from the reference image selected in Step S 104 (Step S 109 ). Here, the evaluation reference image is, for example, a reduced image by reducing the reference image to 32×32 pixels. Subsequently, an evaluation object image is generated from the continuously captured images other than the reference image selected in Step S 104 (Step S 110 ). Here, the evaluation object image is, for example, a reduced image by reducing the reference image to 32×32 pixels.

In the following processing in Steps S 111 to S 115 , the evaluation reference image and each of the continuously captured images are compared and evaluated for difference in each pixel. In this case, determination is firstly made as to whether there is an evaluation object image (continuously captured image) that is not compared and evaluated yet (Step S 111 ). If there is an evaluation object image (continuously captured image) that is not compared and evaluated yet, the evaluation object image is compared with the evaluation object image that is not compared and evaluated yet (Step S 112 ).

After such comparison, determination is made as to whether difference between the evaluation object image and the evaluation reference image is equal to or higher than a threshold value (Step S 113 ). More specifically, the control unit 9 obtains absolute value difference between the (2) image data of the reference image after executing the noise reduction processing and the (1) image data of the evaluation object image (each of the continuously captured images composed of reduced images), for each pixel of the (1) image data of each of the continuously captured images, as shown in Equations 4 to 6 below.

More specifically, Equations below assume the following.

Y_Base_Nr, U_Base_Nr, V_Base_Nr: each YUV parameter of the (2) reference image data after executing the noise reduction.

Y[n], U[n], V[n]: each YUV parameter of the Nth image data.

N: the number of synthesized images.

fMax( ): maximum value enumeration function.

fLpf( ): smoothing function.

fEmphasis( ): map enhancement function.

Then, the following Equations are calculated for each pixel of each of the continuously captured images.

Diff — Y[n]=|Y _base — Nr−Y[n]|   Equation 4 (absolute value difference of // Y )

Diff — U[n]=|U _base — Nr−U[n]|   Equation 5 (absolute value difference of // U )

Diff — V[n]=|V _base — Nr−V[n]|   Equation 6 (absolute value difference of // V )

Determination is then made as to whether the absolute value difference is equal to or higher than a predetermined threshold value. In a case in which the difference is not equal to or higher than the threshold value as a result of the determination in Step S 113 , the evaluation object image is determined to be of image data without uneven brightness, then increment processing is executed (Step S 115 ), and the processing from Step S 112 is repeated for the next evaluation object image. In a case in which the difference is equal to or higher than the threshold value as a result of the determination in Step S 113 , it is determined that the brightness is uneven, and that there is a partial image area being bright, and a flag is set to this evaluation object image (Step S 114 ).

In other words, as described above, the image capturing apparatus according to the present embodiment has the function of reading an image signal by the rolling shutter scheme. Therefore, in a case in which brightness temporarily changes in the photographic environment during the process of continuous capturing, not only an image without uneven brightness as shown in FIG. 3A , but also an image with uneven brightness as shown in FIG. 3B may be acquired. The image shown in FIG. 3B is an image with uneven brightness, in which only a partial area B 2 (a hatching area) is brighter than other areas B 1 and B 3 . Therefore, when the image shown in FIG. 3A is a comparison reference image PA, and the image shown in FIG. 3B is a comparison object image PB, since the absolute value difference is equal to or higher than the predetermined threshold value in only the area B 2 , the determination in Step S 113 is YES, and a flag is set to the comparison object image PB.

In this manner, when the processing from Steps S 111 to S 115 is repeated for the number of times the images are continuously captured, there will be no evaluation object image as a comparison object, and the determination in Step S 111 will become NO. Accordingly, the control unit 9 advances the processing from Step S 111 to S 116 , and determines whether there is an evaluation object image (continuously captured image) with a flag being set as a result of executing the processing in Step S 114 . In a case in which there is no evaluation object image with a flag being set, the processing advances to Step S 118 without executing processing in Step S 117 , and an α map is used to execute synthesis.

More specifically, in Equations 4 to 6, an image area with larger difference can be considered to be an image area with uneven brightness, an image area with maximum difference is selected (with maximum difference for each corresponding pixel of the continuously captured images) as shown in Equation 7 below.

Diff[ n]=f Max(Diff — Y[n ],Diff — U[n ],Diff — V[n ])  Equation 7

Next, as shown in Equation 8 below, processing is executed for averaging maximum difference of each of the continuously captured images other than the reference image.

Ave_Diff=(Diff[0]+Diff[1]+ . . . +Diff[ N ])/ N− 1  Equation 8

Furthermore, smoothing processing is executed as shown in Equation 9 below.

Ave_Diff — Lpf=fLpf (Ave_Diff)  Equation 9

A map M is generated by the smoothing processing. The map M is composed of an area where α=0 allowing complete transparence, and an area where α=255 allowing no transparence.

Continuously, enhancement processing is executed to increase the contrast of the map M as shown in Equation 10 below.

Map_Move= f Emphasis(Ave_Diff — Lpf )  Equation 10

The enhancement processing increases the contrast of an area where the image of the moving body overlaps in the map M.

Eventually, the map M ((Map_Move [0˜255]: 0=there is no moving body) is used as the α map to synthesize the

(2) image data of the reference image after executing the noise reduction processing and the

(3) image data of the synthetic image after executing the noise reduction processing (Step S 118 ).

Each YUV parameter (Y_result, U_result, V_result) of each synthesized pixel is expressed in Equations 11 to 13 below.

Y _result=( Y — Nr _Mix×(255−Map_Move)+ Y _Base — Nr× Map_Move)/255  Equation 11

U _result=( U — Nr _Mix×(255−Map_Move)+ U _Base — Nr× Map_Move)/255  Equation 12

V _result=( V — Nr _Mix×(255−Map_Move)+ V _Base — Nr× Map_Move)/255  Equation 13

As a result, image data of a synthetic image is generated.

However, in a case in which there is an evaluation object image with a flag being set as a result of the determination in Step S 116 , an α 0 blend rate is corrected for the image corresponding to the evaluation object image with the flag being set (Step S 117 ), and the processing in Step S 118 is executed thereafter.

More specifically, the comparison object image PB shown in FIG. 3B is an image, in which only the partial area B 2 is brighter than the other areas B 1 and B 3 , and a flag is set to the comparison object image PB in the processing in Step S 114 . Therefore, since the comparison object image PB with the flag being set exists, the processing advances from Step S 116 to S 117 , and the α blend rate is corrected for the image corresponding to the evaluation object image PB (Step S 117 ).

In other words, when an image PC shown in FIG. 3C is described as an example, in the image PC, only a partial area C 2 corresponding to the area B 2 of the evaluation object image PB is brighter than other areas C 1 and C 3 corresponding to the areas B 1 and B 3 , respectively.

Therefore, the continuously captured image PC with only the area C 2 being bright due to the flash effect is synthesized in the synthetic image of all the continuously captured images after executing the noise reduction processing ((3) YUV image data of the synthetic image after executing the noise reduction processing). Therefore, when such a synthetic image is used as it is, and the map M is used as the α map to synthesize

(2) image data of the reference image after executing the noise reduction processing and

(3) image data of the synthetic image after executing the noise reduction processing,

the brightness components of the area C 2 are included in “(3) image data of the synthetic image after executing the noise reduction processing”. Therefore, as shown in FIG. 3D , a synthetic image PD is generated, in which only a partial area D 2 corresponding to the area C 2 of the continuously captured image PC is brighter than other areas D 1 and D 3 corresponding to the areas C 1 and C 3 , respectively.

Accordingly, in the present embodiment, in Step S 117 , the α blend rate is corrected for the image corresponding to the evaluation object image with a flag being set. More specifically, the processing similar to Step S 107 is executed by employing the areas C 1 and C 3 without employing the area C 2 of the image PC, and the plurality of continuously captured images are synthesized by executing the noise reduction processing. As a result, “(3) image data of the synthetic image after executing the noise reduction processing” is generated, in which the area C 2 of the image PC is not employed in the plurality of continuously captured images.

By using “(3) image data of the synthetic image after executing the noise reduction processing” that is newly generated in the processing in Step S 117 , the map M is used as the α map in Step S 118 to synthesize

(2) image data of the reference image after executing the noise reduction processing and

(3) image data of the synthetic image after executing the noise reduction processing.

At this time, components corresponding to the area C 2 of the continuously captured image PC are not included in “(3) image data of the synthetic image after executing the noise reduction processing” that is used for the synthesizing in Step S 118 . Consequently, as shown in FIG. 3E , a synthetic image PE can be obtained, in which brightness in areas E 1 , E 2 and E 3 are the same without any uneven brightness.

Furthermore, as described above, since the smoothing processing is executed in the present embodiment, the synthesis can create a more natural boarder line between a moving body and a background. Moreover, since the processing of increasing the contrast of the map M is executed, residual images (picture ghosting) by blurring a subject in the synthesis processing can also be decreased.

The embodiment of the present invention has been described above, but the present invention should not be limited thereto, and includes the invention according to the claims and the scope equivalent to the claims.