Image processing apparatus, image processing method, and program

The present technology relates to an image processing apparatus, an image processing method, and a program, and more particularly, to an image processing apparatus, an image processing method, and a program capable of detecting a parallax of a stereoscopic image formed by right and left images more accurately.

A parallax obtained from a stereoscopic image formed by right and left images corresponds to a depth from a user who views the stereoscopic image to a subject. Therefore, parallax detection of a stereoscopic image can be applied to various kinds of application software such as parallax adjustment or viewpoint synthesis of a stereoscopic image and a three-dimensional interface. The parallax detection of a stereoscopic image is associated with a problem of detecting corresponding points in right and left images (for example, see Japanese Unexamined Patent Application Publication No. 2009-008539).

However, when there are a plurality of candidates for a reliable corresponding point on one image of a stereoscopic image with respect to a point on the other image of a stereoscopic image, such as when there is a flat or repetitively patterned portion on the image, it is difficult to match the points accurately, and thus the parallax may not be detected with high accuracy.

For example, when the side planes of an object on an image or a plurality of objects overlap each other in a depth direction, the region of a specific subject is present only in one image of the stereoscopic image and the parallax is rarely detected by a method of detecting the corresponding points in the region.

It is desirable to provide an image processing apparatus, an image processing method, and a program capable of detecting a parallax of a stereoscopic image more accurately.

According to an embodiment of the present technology, there is provided an image processing apparatus including: a parallax map generation unit that generates a parallax map indicating a parallax of each pixel of a stereoscopic image based on right and left images forming the stereoscopic image; a reliability calculation unit that generates a reliability map indicating reliability of the parallax map based on the right and left images; a depth information estimation unit that generates a depth information map indicating a depth of a subject on the stereoscopic image based on one of the right and left images; a depth parallax conversion unit that converts the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth; and a parallax synthesis unit that synthesizes the parallax map and the pseudo-parallax map to generate a corrected parallax map based on the reliability map.

The parallax synthesis unit may generate the corrected parallax map by adding the parallax map and the pseudo-parallax map in a weighted manner using a weight determined by the reliability map.

The parallax synthesis unit may include a component synthesis unit that synthesizes a high-pass component of the pseudo-parallax map and a low-pass component of the parallax map to generate a synthesized map; and an adding unit that synthesizes the parallax map and the synthesized map to generate the corrected parallax map based on the reliability map.

The parallax synthesis unit may include a component synthesis unit that synthesizes a low-pass component of the pseudo-parallax map and a high-pass component of the parallax map to generate a synthesized map; and an adding unit that synthesizes the parallax map and the synthesized map to generate the corrected parallax map based on the reliability map.

The image processing apparatus may further include a residual map generation unit that generates a residual map indicating a residual between a region near a predetermined pixel of one of the right and left images and a region near a corresponding pixel of the other image which corresponds to the predetermined pixel. The reliability calculation unit may generate the reliability map based on the residual map.

The parallax map generation unit may generate the parallax map by detecting, as a pixel corresponding to a predetermined pixel of one of the right and left images, a pixel of the other image for which a residual between a region near the predetermined pixel of the one image and a region near a pixel of the other image is minimum. The image processing apparatus may further include a peak sharpness map generation unit that generates a peak sharpness map that indicates sharpness of a peak of the residual of the pixels lined up in a predetermined direction on the other image and is used for generating the reliability map.

The image processing apparatus may further include a matching degree calculation unit that generates a matching degree map indicating a matching degree between a predetermined pixel of one of the right and left images and a first corresponding pixel of the other image corresponding to the predetermined pixel based on a detection result of the first corresponding pixel of the other image and a detection result of a second corresponding pixel of the one image corresponding to the first corresponding pixel of the other image. The reliability calculation unit may generate the reliability map based on the matching degree map.

According to another embodiment of the present technology, there is provided an image processing method or a program that includes generating a parallax map indicating a parallax of each pixel of a stereoscopic image based on right and left images forming the stereoscopic image; generating a reliability map indicating reliability of the parallax map based on the right and left images; generating a depth information map indicating a depth of a subject on the stereoscopic image based on one of the right and left images; and converting the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth and synthesizing the parallax map and the pseudo-parallax map to generate a corrected parallax map based on the reliability map.

According to still another embodiment of the present technology, a parallax map indicating a parallax of each pixel of a stereoscopic image is generated based on right and left images forming the stereoscopic image; a reliability map indicating reliability of the parallax map is generated based on the right and left images; a depth information map indicating a depth of a subject on the stereoscopic image is generated based on one of the right and left images; and the depth information map is converted into a pseudo-parallax map indicating a parallax corresponding to the depth, and the parallax map and the pseudo-parallax map are synthesized to generate a corrected parallax map based on the reliability map.

According to still another embodiment of the present technology, there is provided an image processing apparatus including: a depth information estimation unit that generates a depth information map indicating a depth of a subject on a stereoscopic image based on one of right and left images forming the stereoscopic image; a depth parallax conversion unit that converts the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth; and a parallax map generation unit that generates a parallax map indicating the parallax of each pixel of the stereoscopic image based on the right and left images and the pseudo-parallax map.

According to still another embodiment of the present technology, there is provided an image processing method or a program that includes generating a depth information map indicating a depth of a subject on a stereoscopic image based on one of right and left images forming the stereoscopic image; and converting the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth and generating a parallax map indicating the parallax of each pixel of the stereoscopic image based on the right and left images and the pseudo-parallax map.

According to still another embodiment of the present technology, a depth information map indicating a depth of a subject on a stereoscopic image is generated based on one of right and left images forming the stereoscopic image, the depth information map is converted into a pseudo-parallax map indicating a parallax corresponding to the depth, and a parallax map indicating the parallax of each pixel of the stereoscopic image is generated based on the right and left images and the pseudo-parallax map.

According to the embodiments of the present technology described above, the parallax of a stereoscopic image can be detected more accurately.

FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus according to an embodiment;

FIG. 2 is a diagram illustrating an example of the configuration of a parallax detection unit;

FIG. 3 is a diagram illustrating an example of the configuration of a parallax synthesis unit;

FIG. 4 is a flowchart illustrating a parallax detection process;

FIG. 5 is a flowchart illustrating a parallax map generation process;

FIG. 6 is a diagram illustrating detection of a corresponding pixel;

FIG. 7 is a diagram illustrating a function of calculating a residual;

FIG. 8 is a diagram illustrating the sharpness of a peak of the residual;

FIG. 9 is a diagram illustrating a function of calculating an evaluation value of the depth of a peak;

FIG. 10 is a diagram illustrating a function of calculating an evaluation value of the width of a peak;

FIG. 11 is a diagram illustrating calculation of a matching degree;

FIG. 12 is a diagram illustrating a function of calculating the matching degree;

FIG. 13 is a flowchart illustrating a corrected parallax map generation process;

FIG. 14 is a diagram illustrating another example of the configuration of the parallax synthesis unit;

FIG. 15 is a flowchart illustrating a corrected parallax map generation process;

FIG. 16 is a diagram illustrating still another example of the configuration of the parallax synthesis unit;

FIG. 17 is a flowchart illustrating a corrected parallax map generation process;

FIG. 18 is a diagram illustrating another example of the configuration of the image processing apparatus;

FIG. 19 is a flowchart illustrating a parallax detection process;

FIG. 20 is a diagram illustrating detection of a corresponding pixel; and

FIG. 21 is a diagram illustrating an example of the configuration of a computer.

Hereinafter, preferred embodiments of the present technology will be described in detail with reference to the appended drawings. Note that, in this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.

First Embodiment

FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus according to an embodiment of the present technology.

When a stereoscopic image formed by right and left images is supplied, an image processing apparatus 11 detects a parallax of the stereoscopic image and outputs the detection result. Here, the right image refers to an image that is displayed so as to be viewed with the right eye of a user when the stereoscopic image is displayed. The left image refers to an image that is displayed so as to be viewed with the left eye of the user when the stereoscopic image is displayed.

The image processing apparatus 11 includes a parallax detection unit 21 , a depth information estimation unit 22 R, a depth information estimation unit 22 L, and a unifying unit 23 .

Based on the supplied right and left images, the parallax detection unit 21 generates a parallax map indicating a parallax in each pixel of the stereoscopic image and a reliability map indicating the certainty, that is, the reliability, of the parallax of each pixel expressed by the parallax map, and then supplies the parallax map and the reliability map to the unifying unit 23 .

Hereinafter, a parallax map formed with reference to the right image, that is, a parallax map indicating a parallax between each pixel of the right image and each pixel of the left image which corresponds to each pixel of the right image is referred to as an RL parallax map. A parallax map formed with reference to the left image is referred to as an LR parallax map. Further, a reliability map indicating the reliability of the RL parallax map is referred to as an RL reliability map and a reliability map indicating the reliability of the LR parallax map is referred to as an LR reliability map.

Based on the supplied right image, the depth information estimation unit 22 R generates an R depth information map indicating the depth of each region (subject) on the right image, and then supplies the R depth information map to the unifying unit 23 . Based on the supplied left image, the depth information estimation unit 22 L generates an L depth information map indicating the depth of each region of the left image, and then supplies the L depth information map to the unifying unit 23 .

Here, the depth of a subject on an image refers to the relative depth position of a subject estimated to be perceived by the user viewing the image. When it is not necessary to particularly distinguish between the depth information estimation unit 22 R and the depth information estimation unit 22 L, the depth information estimation unit 22 R and the depth information estimation unit 22 L are simply referred to as the depth information estimation units 22 below. Further, when it is not necessary to particularly distinguish between the R depth information map and the L depth information map, the R depth information map and the L depth information map are simply referred to as the depth information maps below.

Based on the reliability map from the parallax detection unit 21 , the unifying unit 23 unifies the parallax map from the parallax detection unit 21 and the depth information maps from the depth information estimation units 22 to generate a corrected parallax map. In other words, the parallax map is corrected using the depth information maps in accordance with the reliability map.

The unifying unit 23 includes a depth parallax conversion unit 31 R, a depth parallax conversion unit 31 L, a parallax synthesis unit 32 R, and a parallax synthesis unit 32 L.

The depth parallax conversion unit 31 R converts the R depth information map from the depth information estimation unit 22 R into an R pseudo-parallax map having parallax information based on the RL parallax map and the RL reliability map supplied from the parallax detection unit 21 , and then supplies the R pseudo-parallax map to the parallax synthesis unit 32 R. That is, the R depth information map indicating the depth is converted to the R pseudo-parallax map indicating the parallax corresponding to the depth.

The parallax synthesis unit 32 R synthesizes the RL parallax map from the parallax detection unit 21 and the R pseudo-parallax map from the depth parallax conversion unit 31 R based on the RL reliability map supplied from the parallax detection unit 21 , generates a corrected RL parallax map, and then outputs the corrected RL parallax map.

The depth parallax conversion unit 31 L converts the L depth information map from the depth information estimation unit 22 L into an L pseudo-parallax map having parallax information based on the LR parallax map and the LR reliability map supplied from the parallax detection unit 21 , and then supplies the L pseudo-parallax map to the parallax synthesis unit 32 L. The parallax synthesis unit 32 L synthesizes the LR parallax map from the parallax detection unit 21 and the L pseudo-parallax map from the depth parallax conversion unit 31 L based on the LR reliability map supplied from the parallax detection unit 21 , generates a corrected LR parallax map, and then outputs the corrected RL parallax map.

When it is not necessary to particularly distinguish between the depth parallax conversion unit 31 R and the depth parallax conversion unit 31 L, the depth parallax conversion unit 31 R and the depth parallax conversion unit 31 L are simply referred to as the depth parallax conversion units 31 below. Further, when it is not necessary to particularly distinguish between the parallax synthesis unit 32 R and the parallax synthesis unit 32 L, the parallax synthesis unit 32 R and the parallax synthesis unit 32 L are simply referred to as the parallax synthesis units 32 below.

When it is not necessary to particularly distinguish between the R pseudo-parallax map and the L pseudo-parallax map, the R pseudo-parallax map and the L pseudo-parallax map are simply referred to as the pseudo-parallax maps below. Further, when it is not necessary to particularly distinguish between the corrected RL parallax map and the corrected LR parallax map, the corrected RL parallax map and the corrected LR parallax map are simply referred to as the corrected parallax maps below.

FIG. 2 shows the detailed configuration of the parallax detection unit 21 in FIG. 1 .

That is, the parallax detection unit 21 includes a corresponding point detection unit 61 R, a corresponding point detection unit 61 L, a matching degree calculation unit 62 , a reliability calculation unit 63 R, and a reliability calculation unit 63 L.

The corresponding point detection unit 61 R detects the pixels corresponding to the pixels of the right image from the left image and generates the RL parallax map, an RL residual map, and an RL peak sharpness map. The corresponding point detection unit 61 R includes a parallax map generation unit 71 R, a residual map generation unit 72 R, and a peak sharpness map generation unit 73 R.

The parallax map generation unit 71 R generates the RL parallax map based on the supplied right and left images, and then supplies the RL parallax map to the matching degree calculation unit 62 , the depth parallax conversion unit 31 R, and the parallax synthesis unit 32 R. That is, the pixels of the left image which respectively correspond to the pixels of the right image are detected and the RL parallax map is generated based on the positions of these pixels. Hereinafter, the pixels of one of the right and left images which correspond to the pixel of the other image are referred to as corresponding pixels. For example, the corresponding pixels of the left image which correspond to predetermined pixels of the right image are pixels used to show the same subject as that of the predetermined pixel of the right image.

The residual map generation unit 72 R generates an RL residual map, which indicates a residual between a region near the pixels of the right image and a region near the corresponding pixels of the left image which correspond to the pixels of the right image, based on the right and left images and the detection result of the corresponding pixels obtained by the parallax map generation unit 71 R, and then supplies the RL residual map to the reliability calculation unit 63 R.

The peak sharpness map generation unit 73 R generates the sharpness of a peak of a curve indicating the residual between each pixel of the right image and each pixel of the left image, that is, an RL peak sharpness map indicating the degree of dispersion of the residual, based on the supplied right and left images, and then supplies the RL peak sharpness map to the reliability calculation unit 63 R.

The corresponding point detection unit 61 L detects the corresponding pixels of the right image with respect to the pixels of the left image and generates the LR parallax map, an LR residual map, and an LR peak sharpness map. The corresponding point detection unit 61 L includes a parallax map generation unit 71 L, a residual map generation unit 72 L, and a peak sharpness map generation unit 73 L.

The parallax map generation unit 71 L, the residual map generation unit 72 L, and the peak sharpness map generation unit 73 L perform the same processes as the parallax map generation unit 71 R, the residual map generation unit 72 R, and the peak sharpness map generation unit 73 R, respectively, and thus the description thereof will not be repeated. The parallax map generation unit 71 L, the residual map generation unit 72 L, and the peak sharpness map generation unit 73 L of the corresponding point detection unit 61 L generate the LR parallax map, the LR residual map, and the LR peak sharpness map, respectively.

When it is not necessary to particularly distinguish between the corresponding point detection unit 61 R and the corresponding point detection unit 61 L, the corresponding point detection unit 61 R and the corresponding point detection unit 61 L are simply referred to as the corresponding point detection units 61 below. When it is not necessary to particularly distinguish between the parallax map generation unit 71 R and the parallax map generation unit 71 L, the parallax map generation unit 71 R and the parallax map generation unit 71 L are simply referred to as the parallax map generation units 71 below. When it is not necessary to particularly distinguish between the residual map generation unit 72 R and the residual map generation unit 72 L, the residual map generation unit 72 R and the residual map generation unit 72 L are simply referred to as the residual map generation units 72 below. When it is not necessary to particularly distinguish between the peak sharpness map generation unit 73 R and the peak sharpness map generation unit 73 L, the peak sharpness map generation unit 73 R and the peak sharpness map generation unit 73 L are simply referred to as the peak sharpness map generation units 73 below.

When it is not necessary to particularly distinguish between the RL residual map and the LR residual map, the RL residual map and the LR residual map are simply referred to as the residual maps below. When it is not necessary to particularly distinguish between the RL peak sharpness map and the LR peak sharpness map, the RL peak sharpness map and the LR peak sharpness map are simply referred to as the peak sharpness maps below.

The matching degree calculation unit 62 generates matching degree maps indicating the matching degree between the parallax maps based on the RL parallax map from the corresponding point detection unit 61 R and the LR parallax map from the corresponding point detection unit 61 L. That is, the matching degree calculation unit 62 generates an RL matching degree map indicating the matching degree of the RL parallax map to the LR parallax map, and then supplies the RL matching degree map to the reliability calculation unit 63 R. Further, the matching degree calculation unit 62 generates an LR matching degree map indicating the matching degree of the LR parallax map to the RL parallax map, and then supplies the LR matching degree map to the reliability calculation unit 63 L.

When it is not necessary to particularly distinguish between the RL matching degree map and the LR matching degree map, the RL matching degree map and the LR matching degree map are simply referred to as the matching degree maps below.

The reliability calculation unit 63 R generates the RL reliability map based on the RL residual map and the RL peak sharpness map from the corresponding point detection unit 61 R and the RL matching degree map from the matching degree calculation unit 62 , and then supplies the RL reliability map to the depth parallax conversion unit 31 R and the parallax synthesis unit 32 R.

The reliability calculation unit 63 L generates the LR reliability map based on the LR residual map and the LR peak sharpness map from the corresponding point detection unit 61 L and the LR matching degree map from the matching degree calculation unit 62 , and then supplies the LR reliability map to the depth parallax conversion unit 31 L and the parallax synthesis unit 32 L. When it is not necessary to particularly distinguish between the reliability calculation unit 63 R and the reliability calculation unit 63 L, the reliability calculation unit 63 R and the reliability calculation unit 63 L are simply referred to as the reliability calculation units 63 below.

FIG. 3 shows an example of the detailed configuration of the parallax synthesis unit 32 R.

That is, the parallax synthesis unit 32 R includes a multiplying unit 101 R, a multiplying unit 102 R, and an adding unit 103 R. The parallax synthesis unit 32 R supplies the RL reliability map supplied from the reliability calculation unit 63 R as a weight without change to the multiplying unit 102 R and supplies a weight determined in accordance with the RL reliability map to the multiplying unit 101 R.

The multiplying unit 101 R multiplies the R pseudo-parallax map supplied from the depth parallax conversion unit 31 R by the supplied weight and supplies the multiplying result to the adding unit 103 R. The multiplying unit 102 R multiples the RL parallax map supplied from the corresponding point detection unit 61 R by the supplied weight and supplies the multiplying result to the adding unit 103 R.

The adding unit 103 R adds the R pseudo-parallax map from the multiplying unit 101 R to the RL parallax map from the multiplying unit 102 R to generate a corrected RL parallax map, and then outputs the corrected RL parallax map to a rear stage block.

Since the parallax synthesis unit 32 L has the same configuration as the parallax synthesis unit 32 R in FIG. 3 , the configuration of the parallax synthesis unit 32 L is not illustrated and the description thereof will not be repeated. Units of the parallax synthesis unit 32 L corresponding to the multiplying unit 101 R, the multiplying unit 102 R, and the adding unit 103 R are referred to as a multiplying unit 101 L, a multiplying unit 102 L, and an adding unit 103 L, respectively.

When it is not necessary to distinguish between the multiplying units 101 R and 101 L, the multiplying units 101 R and 101 L are simply referred to as the multiplying units 101 below. When it is not necessary to distinguish between the multiplying units 102 R and 102 L, the multiplying units 102 R and 102 L are simply referred to as the multiplying units 102 below. When it is not necessary to distinguish between the adding units 103 R and 103 L, the adding units 103 R and 103 L are simply referred to as the adding units 103 below.

Description of Parallax Detection Process

When the image processing apparatus 11 is supplied with the stereoscopic image formed by the right and left images and receives an instruction to detect the parallax, the image processing apparatus 11 generates the corrected parallax maps by performing a parallax detection process and outputs the corrected parallax maps. Hereinafter, the parallax detection process of the image processing apparatus 11 will be described with reference to the flowchart of FIG. 4 .

In step S 11 , the parallax detection unit 21 generates the parallax maps and the reliability maps by performing a parallax map generation process and supplies the parallax maps and the reliability maps to the unifying unit 23 . The parallax map generation process will be described in detail later. When the parallax detection unit 21 performs the parallax map generation process, the parallax detection unit 21 supplies the generated parallax map and the generated reliability map to the depth parallax conversion unit 31 and the parallax synthesis unit 32 .

In step S 12 , the depth information estimation units 22 generate the depth information maps and supply the depth information maps to the depth parallax conversion units 31 . That is, the depth information estimation unit 22 R generates the R depth information map based on the supplied right image and supplies the R depth information map to the depth parallax conversion unit 31 R. The depth information estimation unit 22 L generates the L depth information map based on the supplied left image and supplies the L depth information map to the depth parallax conversion unit 31 L.

For example, the depth information estimation unit 22 R extracts high-pass components from the right image by a filtering process or the like and divides the right image into a plurality of blocks. Then, the depth information estimation unit 22 R generates the R depth information map by calculating a sum (integrated value) of the absolute values of the values of the high-pass components at respective positions in the block for each block. The depth information estimation unit 22 L generates the L depth information map by performing the same processes.

Since the high-frequency components are generally abundant in a subject viewed (positioned) on the front side in a view from a user, that is, the region of a foreground or a focused region on an image, the value (integrated value of the high-pass components) of the pixels of the R depth information map corresponding to this region. Accordingly, as a region has larger pixel values of the pixels in the R depth information map, the region of the right image located at the same position as the region can be estimated as a region of a subject closer to the user who views the right image. A method of estimating the depth in an image is disclosed in detail in, for example, Japanese Unexamined Patent Application Publication No. 2007-208399.

For example, the depth information estimation unit 22 may generate the depth information map by specifying the colors of the pixels on an image, setting the pixels with colors of the warm color system to the pixels of a subject located on the front side, and setting the pixels with colors of the cold color system to the pixels of a subject located on the rear side. Such a depth information map is generated using the human visual characteristic that the colors of the warm color system are viewed to be located on the front side.

For example, the depth information estimation unit 22 may recognize a scene for an image (a right image or a left image) and output a depth information map determined in advance for the recognized scene.

As a method of generating the depth information map, any method such as a method of using the existing 2D-to-3D conversion technology may be used, as long as depth information is estimated and a map indicating the depth information is generated. For example, the 2D-to-3D conversion technology is described in detail in “JVC, “Real-time 2D-to-3D Conversion at Full HD 1080P Resolution” The 13th IEEE International Symposium on Consumer Electronics (ISCE2009), “Toshiba “2D-to-3D content conversion technology” at 2010 annual conference of Image Information and Television Engineers”, or the like.

In step S 13 , the depth parallax conversion unit 31 generates a conversion equation used to convert the depth information to parallax information based on the depth information maps from the depth information estimation units 22 , the parallax maps from the corresponding point detection units 61 , and the reliability maps from the reliability calculation units 63 .

For example, in an ij coordinate system in which the direction of a parallax of the right and left images is an i direction and a direction perpendicular to the i direction is a j direction, it is assumed that the pixel value of a pixel of the R depth information map located at the same position as a pixel located at coordinates (i, j) on the right image is O R (i, j). Further, it is assumed that a pixel value of a pixel of the R pseudo-parallax map located at the same position as a pixel located at coordinates (i, j) on the right image is PDP R (i, j). Here, the pixel value PDP R (i, j) of the pixel of the R pseudo-parallax map is a value to be calculated hereinafter.

At this time, the depth parallax conversion unit 31 R models a relation between the parallax information and the depth information. That is, the depth parallax conversion unit 31 R supposes that the pixel value O R (i, j) of a pixel of the R depth information map is linearly converted to the pixel value PDP R (i, j) of a pixel of the R pseudo-parallax map by Equation (1) below. In Equation (1), p and q are an integer.

PDP R ( i,j )= p×O R ( i,j )+ q   (1)

The depth information map indicates depth information of each subject on the right image. The characteristics of the depth information are different from those of the parallax information detected as a position deviation between the right and left images. Therefore, to synthesize the depth information map and the parallax map, it is necessary to match the characteristics of these maps.

Accordingly, the depth parallax conversion unit 31 R calculates a linear function “p×O R (i, j)+q” to convert the pixel value O R (i, j), which is the depth information, to the equivalent parallax information by this linear function.

Specifically, the depth parallax conversion unit 31 R calculates the integers p and q, for which the value of Equation (2) below is the minimum, by the least square method on the assumption that the pixel value of a pixel of the RL parallax map located at the same position as a given pixel located at coordinates (i, j) on the right image is DP RL (i, j).

That is, the depth parallax conversion unit 31 R extracts, from the pixels of the RL parallax map, only the pixels having a pixel value equal to or greater than a predetermined threshold value T1 in the RL reliability map and located at the same positions as the pixels of the RL parallax map. Then, the depth parallax conversion unit 31 R calculates “DP RL (i, j)−p×O R (i, j)−q” for each of the extracted pixels based on the pixel value DP RL (i, j) of each of the extracted pixels and the pixel value O R (i, j) of each of the pixels located at the same positions as the extracted pixels. Further, the depth parallax conversion unit 31 R calculates the integers p and q for which a sum F R of the square of the obtained difference is the minimum.

A conversion equation “p×O R (i, j)+q” used to convert the depth information to the parallax information, that is, the right side of Equation (1) described above, can be obtained from the obtained integers p and q. Further, the depth parallax conversion unit 31 L performs the same processes as the depth parallax conversion unit 31 R to calculate a conversion equation used to convert the L depth information map to the L pseudo-parallax map.

Thus, the depth information map with the depth information can be suitably converted to the pseudo-parallax map having the parallax information by calculating the conversion equation used to convert the depth information to the parallax information.

The foregoing description pertains to an example in which the linear function is used as the conversion equation. However, the conversion equation is not limited to the linear function, but may be any function. Since the relation between the parallax information and the depth information is different depending on how the depth information is calculated, the relation may be modeled suitably in accordance with the depth information map and the conversion equation may be calculated.

In step S 14 , the depth parallax conversion units 31 convert the depth information maps to the pseudo-parallax maps using the conversion equation calculated by the process of step S 13 , and then supply the converted pseudo-parallax maps to the multiplying units 101 of the parallax synthesis units 32 .

For example, the depth parallax conversion unit 31 R sets the value, which is obtained by substituting the pixel value O R (i, j) of each pixel of the R depth information map to Equation (1) described above, to the pixel value PDP R (i, j) of each pixel of the R pseudo-parallax map located at the same position as the pixel of the R-depth information map. The depth parallax conversion unit 31 L performs the same process as the depth parallax conversion unit 31 R to generate the L pseudo-parallax map and supplies the L pseudo-parallax map to the multiplying unit 101 L.

In step S 15 , the parallax synthesis units 32 perform a corrected parallax map generation process to generate the corrected parallax maps.

The corrected parallax map generation process will be described in detail later. In the corrected parallax map generation process, the parallax map and the pseudo-parallax map are synthesized (unified) and the synthesized parallax map is considered as the corrected parallax map. For example, in a region where the reliability of the parallax map is high, the parallax map is used without synthesis and is considered as the corrected parallax map. In a region where the reliability of the parallax map is low, the contribution ratio of the pseudo-parallax map is set to be higher when the corrected parallax map is generated.

When the corrected parallax map generation process is performed, the generated corrected parallax maps are output to the rear stage, and then the parallax detection process ends.

In this way, the image processing apparatus 11 generates the parallax maps from the right and left images, generates the depth information map from one of the right and left images, and converts the depth information maps into the pseudo-parallax maps. Then, the image processing apparatus 11 synthesizes the parallax maps and the pseudo-parallax maps in accordance with the reliabilities of the parallax maps and generates the corrected parallax maps as the final parallax detection result.

Thus, the parallax can be detected more accurately and reliably by unifying the parallax maps and the pseudo-parallax maps in accordance with the reliabilities of the parallax maps and generating the corrected parallax maps as the final parallax detection result.

For example, in a region, such as a flat portion or a repeating pattern portion, where it is difficult to match the right and left images, the pixel value of each pixel of the reliability map decreases and the reliability of the parallax map is thus lowered. In this case, however, since the parallax estimated from one of the right and left images, that is, the pseudo-parallax map, is used more, the more accurate parallax of the stereoscopic image can be obtained compared to the case where only the parallax map is used.

Description of Parallax Map Generation Process

Next, a parallax map generation process corresponding to the process of step S 11 in FIG. 4 will be described with reference to the flowchart of FIG. 5 .

In step S 41 , the parallax map generation units 71 generate the parallax maps based on the supplied right and left images and supply the parallax maps to the matching degree calculation unit 62 and the parallax synthesis units 32 .

For example, the parallax map generation unit 71 L detects the corresponding pixels of the right image to the pixels of the left image by block matching or the like.

Specifically, as shown in the upper side of FIG. 6 , the parallax map generation unit 71 L sets one pixel on a left image I L as an interest pixel g L (i, j) and sets a pixel on a right image I R , which is a candidate of the corresponding point to the interest pixel g L (i, j), to a pixel g L ′(i+d, j). Here, the pixel g L ′(i+d, j), which is a candidate of the corresponding point, has the same j coordinate as the interest pixel g L (i, j). The pixel g L ′(i+d, j) is a pixel located at a position distant by d pixels in the direction from the same position of the interest pixel of the right image I R .

In FIG. 6 , the horizontal and vertical directions represent the i and j directions of the ij coordinate system. Further, it is assumed that I L (i, j) is the pixel value of a pixel with coordinates (i, j) on the left image I L and I R (i, j) is the pixel value of a pixel with coordinates (i, j) on the right image I R . For example, a pixel with the pixel value I L (i, j) on the left image I L is a pixel that is located at an i-th position from the left end of the left image I L and at a j-th position from the upper end of the left image I L .

The parallax map generation unit 71 L calculates, as a residual R i, j (d) at the interest pixel, a difference absolute value sum of the pixel value of a pixel inside a predetermined region gb L centering the interest pixel g L (i, j) and the pixel value of a pixel inside a predetermined region gb L ′ centering the pixel g L ′(i+d, j).

That is, the parallax map generation unit 71 L calculates the residual R i, j (d) by Equation (3) below.

The parallax map generation unit 71 L calculates the residual R i, j (d) of the interest pixel for each d, that is, the candidate of each corresponding pixel, while varying the value of d within a preset range (for example, −d rng ≦d≦ rng ).

Here, the residual R i, j (d) is a difference absolute value sum of the pixel value of a pixel inside the region gb L near the interest pixel g L (i, j) and the pixel value of a pixel inside the region gb L ′ near the pixel g L ′(i+d, j) which is the candidate of the interest pixel. Accordingly, the more similar the region gb L and the region gb L ′ are, the smaller the residual R i, j (d) is.

Thus, the parallax map generation unit 71 L sets the pixel g L ′(i+d, j) with the minimum residual R i, j (d) among the pixels of the right image I R as the corresponding pixel of the interest pixel and calculates the pixel value DP LR (i, j) of the pixel of the LR parallax map located at the same position as the interest pixel g L (i, j) by Equation (4) below.

DP LR ( i,j )={ d |MIN[ R i,j ( d )]}  (4)

That is, the value of d when the residual R i, j (d) is the minimum is detected as the parallax at the interest pixel g L (i, j) of the left image I L between the right image I R and the left image I L . The parallax d (pixel value DP LR (i, j)) is a value that is obtained by subtracting the i coordinate of the interest pixel from the i coordinate of the corresponding pixel.

The parallax map generation unit 71 L sequentially sets the pixels of the left image I L as the interest pixels and generates the LR parallax map by calculating the pixel values DP LR (i, j) of the pixels of the LR parallax map located at the same position as the interest pixels. The parallax map generation unit 71 L supplies the generated LR parallax map to the matching degree calculation unit 62 and the multiplying unit 102 L.

Further, the parallax map generation unit 71 R performs the same processes as the parallax map generation unit 71 L. That is, the parallax map generation unit 71 R generates the RL parallax map by calculating the pixel values DP RL (i, j) of the pixels of the RL parallax map located at the same positions as the interest pixels.

That is, for example, as shown in the lower side of FIG. 6 , the parallax map generation unit 71 R sets one pixel of the right image I R to an interest pixel g L (i, j) and sets the pixel of the left image I L which is a candidate of the corresponding point of the interest pixel g L (i, j) to a pixel g R ′(i+d, j).

The parallax map generation unit 71 R calculates, as a residual at the interest pixel, a difference absolute value sum of the pixel value of a pixel inside a predetermined region gb R centering the interest pixel g R (i, j) and the pixel value of a pixel inside a predetermined region gb R ′ centering the pixel g R ′(i+d, j). That is, the same calculation performed by Equation (3) described above is performed.

The parallax map generation unit 71 R sets the value of d obtained at the minimum residual as the parallax at the interest pixel g R (i, j) of the right image I R between the right image I R and the left image I L and sets the value of the parallax to the pixel value DP RL (i, j) of the pixel of the RL parallax map located at the same position as the interest pixel. That is, the pixel value DP RL (i, j) is a value that is obtained by subtracting the i coordinate of the interest pixel from the i coordinate of the corresponding pixel.

The parallax map generation unit 71 R sequentially sets the pixels of the right image I R as the interest pixels and generates the RL parallax map by calculating the pixel values DP RL (i, j) of the pixels of the RL parallax map located at the same position as the interest pixels. The parallax map generation unit 71 R supplies the generated RL parallax map to the matching degree calculation unit 62 and the multiplying unit 102 R.

Referring back to the flowchart of FIG. 5 , in step S 42 , the residual map generation units 72 generate the residual maps based on the parallax maps of the right and left images and supply the residual maps to the reliability calculation unit 63 .

For example, the residual map generation unit 72 L sequentially sets the pixels of the left image and calculates pixel values E LR (i, j) of the pixels of the LR residual map located at the same positions as the interest pixels having the coordinates (i, j) by Equation (5) below.

That is, a difference absolute value sum of the pixel value of a pixel inside a region near the interest pixel of the left image and the pixel value of a pixel inside a region near the corresponding pixel of the right image is substituted into a predetermined function N 1 (x) to obtain the pixel value E LR (i, j) of a pixel of the LR residual map.

Here, for example, the function N 1 (x) is a function shown in FIG. 7 . In FIG. 7 , the vertical axis represents the value of the function N 1 (x) and the horizontal axis represents the value of x substituted into the function N 1 (x), that is, the value of the difference absolute value sum.

In FIG. 7 , a folded line C 11 represents the value of the function N 1 (x) for each value of x. In the range of “0≦x≦T11”, the function N 1 (x) is equal to 0. Further, in the range of “x≧T12”, the function N 1 (x) is equal to 1. Furthermore, in the range of “T11≦x≦T12”, the value of the function N 1 (x) increases in proportion to the value of x.

Accordingly, the more the region near the interest pixel and the region near the corresponding pixel of the interest pixel are similar, that is, the smaller the difference absolute value sum (residual) of the regions is, the smaller the pixel value E LR (i, j) of the pixel of the LR residual map is.

The residual map generation unit 72 L generates the LR residual map by setting the pixels of the left image to the interest pixels and calculating the pixel values E LR (i, j) of the pixels of the LR residual map corresponding to the interest pixels, and then supplies the LR residual map to the reliability calculation unit 63 L. The more the parallax of the pixel of the left image located at the same pixel of the generated LR residual map is reliable, the smaller the pixel value of the pixel of the generated LR residual map is.

Likewise, the residual map generation unit 72 R sequentially sets the pixels of the left image and calculates pixel values E RL (i, j) of the pixels of the RL residual map located at the same positions as the interest pixels having the coordinates (i, j) by Equation (6) below.

That is, a difference absolute value sum of the pixel value of a pixel inside a region near the interest pixel of the right image and the pixel value of a pixel inside a region near the corresponding pixel of the left image is substituted into the function N 1 (x) to obtain the pixel value E RL (i, j) of a pixel of the RL residual map.

Even in the RL residual map, the smaller the difference absolute value sum (residual) of the region near the interest pixel and the region near the corresponding pixel of the interest pixel is, the smaller the pixel value E RL (i, j) of the pixel of the RL residual map is. Accordingly, the more the parallax of the pixel of the right image located at the same pixel of the RL residual map is reliable, the smaller the pixel value of the pixel of the RL residual map is.

The residual map generation unit 72 R generates the RL residual map by setting the pixels of the right image to the interest pixels and calculating the pixel values E RL (i, j) of the pixels of the RL residual map corresponding to the interest pixels, and then supplies the RL residual map to the reliability calculation unit 63 R.

In step S 43 , the peak sharpness map generation units 73 generate the peak sharpness maps based on the supplied right and left images and supply the peak sharpness maps to the reliability calculation units 63 .

For example, the peak sharpness map generation unit 73 L sequentially sets the pixels of the left image to interest pixels. Then, the peak sharpness map generation unit 73 L calculates a pixel value PK LR (i, j) of a pixel of the LR peak sharpness map located at the same position as the interest pixel based on the calculation result of the residual R i, j (d), which is calculated by the parallax map generation unit 71 L, for each parallax d in the interest pixel.

Specifically, for example, a result shown in FIG. 8 is obtained as the calculation result of the residual R i, j (d) for each parallax d (the candidate of the corresponding pixel) in the interest pixel of the left image. In FIG. 8 , the horizontal axis represents the parallax d between the interest pixel and the candidate of the corresponding pixel and the vertical axis represents the value of the residual R i, j (d) for each parallax d.

In the example of FIG. 8 , a curve C 21 represents the residual R i, j (d) for each parallax d. In the curve C 21 , the parallax of the interest pixel is equal to DP LR (i, j) at the downward convex peak position, that is, the position of d at which the residual R i, j (d) is the minimum.

The residual R i, j (d) for each parallax d is a residual calculated for the candidate of each corresponding pixel by Equation (3). Therefore, as the depth of the peak in the curve C 21 is deeper and the width of the peak portion is narrower, that is, the relatively small residual is further focused in the narrow range, the sharpness of the peak is larger. Here, when it is assumed that r max is the maximum value of the residual R i, j (d) and r min is the minimum value of the residual R i, j (d), the depth of the peak of the residual can be defined by, for example, “r max −r min .”

As the larger the peak sharpness is, the smaller the candidates of the corresponding pixel are in number. Therefore, the calculated parallax DP LR (i, j) of the interest pixel is more reliable. In contrast, in a flat portion or repeating pattern portion in which it is difficult to detect the corresponding pixel on an image, the peak of the residual is shallow and the peak width is larger.

The peak sharpness map generation unit 73 L calculates the pixel value PK LR (i, j) of the LR peak sharpness map configured to estimate the reliability of the parallax shown by the LR parallax map by using the peak sharpness of the residual as an index.

That is, the peak sharpness map generation unit 73 L calculates Equation (7) below based on the calculation result of the residual R i, j (d) for each parallax d in the interest pixel having the coordinates (i, j) and calculates an evaluation value PKH LR (i, j) for which the depth of the peak of the residual is used as an index.

PKH LR ( i,j )= N 2 ( r max −r min )  (7)

That is, a difference between the maximum value r max and the minimum value r min of the residual R i, j (d) is substituted into a predetermined function N 2 (x) to obtain the evaluation value PKH LR (i, j).

Here, for example, the function N 2 (x) is a function shown in FIG. 9 . In FIG. 9 , the vertical axis represents the value of the function N 2 (x) and the horizontal axis represents the value of x substituted into the function N 2 (x), that is, the value of the difference absolute value sum.

In FIG. 9 , a folded line C 31 represents the value of the function N 2 (x) for each value of x. In the range of “0≦x≦T21”, the function N 2 (x) is equal to 0. Further, in the range of “x≧T22”, the function N 2 (x) is equal to 1. Furthermore, in the range of “T21≦x≦T22”, the value of the function N 2 (x) increases in proportion to the value of x. Accordingly, the deeper (larger) the depth of the peak of the residual is, the larger the value of the evaluation value PKH LR (i, j) is.

Next, the peak sharpness map generation unit 73 L calculates Equation (8) below based on the calculation result of the residual R i, j (d) for each parallax d in the interest pixel having the coordinates (i, j) and calculates an evaluation value PKW LR (i, j) for which the width of the peak of the residual is used as an index.

That is, an aggregate sum of the differences between the maximum value r max and the residual R i, j (d) is calculated for each parallax d (the candidate of the corresponding pixel) within a range of “−d rng ≦d≦d rng which is a preset range of d. Then, a value obtained by subtracting “(2d rng +1)(r max −r min )” from the calculated aggregate sum of the differences is substituted into a predetermined function N 3 (x) to obtain an evaluation value PKW LR (i, j). Here, the narrower (the smaller) the width of the peak of the residual is, the smaller the term of the function N 3 (x) in the evaluation value PKW LR (i, j) is.

Further, the function N 3 (x) is a function shown in FIG. 10 . In FIG. 10 , the vertical axis represents the value of the function N 3 (x) and the horizontal axis represents the value of x substituted into the function N 3 (x), that is, “Σ(r max −R i, j (d))/((2d rng +1)(r max −r min )).”

In FIG. 10 , a folded line C 41 represents the value of the function N 3 (x) for each value of x. In the range of “0≦x≦T31”, the function N 3 (x) is equal to 1. Further, in the range of “x≦T32”, the function N 3 (x) is equal to 0. Furthermore, in the range of “T31≦x≦T32”, the value of the function N 3 (x) decreases in inverse proportion to the value of x. Accordingly, the narrower the width of the peak of the residual is, the larger the value of the evaluation value PKW LR (i, j) is.

Further, the peak sharpness map generation unit 73 L calculates the pixel value PK LR (i, j) of the pixel of the LR peak sharpness map by substituting the calculated evaluation values PKH LR (i, j) and PKW LR (i, j) into Equation (9) below.

PK LR ( i,j )= PKH LR ( i,j ) PKW LR ( i,j )  (9)

That is, a product of evaluation values PKH LR (i, j) and PKW LR (i, j) calculated for the interest pixel is the pixel value PK LR (i, j) of a pixel of the LR peak sharpness map located at the same position as the interest pixel.

The deeper the peak of the residual is and the narrower the width of the peak is, the larger the pixel value PK LR (i, j) is. In other words, the larger the sharpness of the peak of the residual is and the more the parallax of the interest pixel is, the larger the pixel value PK LR (i, j) is.

The peak sharpness map generation unit 73 L generates the LR peak sharpness map by setting the pixels of the left image as the interest pixels and calculating the pixel value PK LR (i, j) of the pixels of the LR peak sharpness map located at the same positions as the interest pixels. The peak sharpness map generation unit 73 L supplies the generated LR peak sharpness map to the reliability calculation unit 63 L.

The peak sharpness map generation unit 73 R also performs the same processes as the peak sharpness map generation unit 73 L to generate the RL peak sharpness map and supplies the generated RL peak sharpness map to the reliability calculation unit 63 R.

In step S 44 , the matching degree calculation unit 62 generates the matching degree maps based on the RL parallax map from the corresponding point detection unit 61 R and the LR parallax map from the corresponding point detection unit 61 L, and then supplies the matching degree maps to the reliability calculation units 63 .

For example, as shown in FIG. 11 , the coordinates of a corresponding pixel gc′ of the right image I R for a pixel gc located at the coordinates (i, j) on the left image I L can be obtained from the L R parallax map. That is, on the assumption that DP LR (i, j) is the pixel value of a pixel located at the same position as the pixel gc on the LR parallax map, the coordinates of the corresponding pixel gc′ are (i+DP LR (i, j), j).

On the other hand, when the corresponding pixel gc′ is interested on the right image I R , the coordinates of the corresponding pixel of the left image I L with respect to the corresponding pixel gc′ are (i+DP LR (i, j)+DP LR (i+DP LR (i, j), j), j) from the pixel value of the pixel located at the same position as the pixel gc′ of the RL parallax map.

Accordingly, when the corresponding pixel of the left image I L with respect to the corresponding pixel gc′ is the pixel gc, the corresponding pixel gc′ of the pixel gc is appropriately obtained, and thus a correspondence between the pixel gc and the corresponding pixel gc′ is achieved. That is, the pixel gc matches the corresponding pixel gc′ and the same subject is displayed at these pixels.

Here, since the corresponding pixel of the corresponding pixel gc′ is the pixel gc, the coordinates (i+DP LR (i, j)+DP LR (i+DP LR (i, j), j), j) of the corresponding pixel may be identical with the coordinates (i, j) of the pixel gc. Therefore, “DP LR (i, j)+DP LR (i+DP LR (i, j), j)=0” is achieved. Further, when DP LR (i, j)+DP LR (i+DP LR (i, j), j) is not equal to 0 but the absolute value thereof is a small value, the correspondence between the pixels is achieved to some extent.

Therefore, the matching degree calculation unit 62 calculates the matching degree between the pixels of the left image and the corresponding pixels of the pixels of the left image by Equation (10) below and calculates a pixel value C LR (i, j) of each pixel of the LR matching degree map located at the same position as the pixel of the left image.

C LR ( i,j )= N 4 (| DP LR ( i,j )+ DP RL ( i+DP LR ( i,j ), j )|)  (10)

That is, when a predetermined pixel of the left image is set to an interest pixel, the pixel value C LR (i, j) of a pixel of the LR matching degree map located at the same position as the interest pixel can be calculated as follows. First, the absolute value of the sum of the pixel value of the pixel of the LR parallax map located at the same position as the interest pixel and the pixel value of the pixel of the RL parallax map located at the same position as the corresponding pixel of the interest pixel is calculated. Then, the calculated absolute value of the sum is substituted into a predetermined function N 4 (x) to obtain the pixel value C LR (i, j). The more the correspondence between the interest pixel and the corresponding pixel is achieved, the larger the pixel value C LR (i, j) is.

Here, for example, the function N 4 (x) is a function shown in FIG. 12 . In FIG. 12 , the vertical axis represents the value of the function N 4 (x) and the horizontal axis represents the value of x substituted into the function N 4 (x), that is, “|DP LR (i, j)+DP RL (i+DP LR (i, j), j)|.”

In FIG. 12 , a folded line C 51 represents the value of the function N 4 (x) for each value of x. In the range of “0≦x≦T41”, the function N 4 (x) is equal to 1. Further, in the range of “x≧T42”, the function N 4 (x) is equal to 0. Furthermore, in the range of “T41≦x≦T42”, the value of the function N 4 (x) decreases in inverse proportion to the value of x. Accordingly, the higher the matching degree between the pixel of the left image and the corresponding pixel of the pixel of the left image is, the larger the pixel value C LR (i, j) of the pixel of the LR matching degree map is.

The matching degree calculation unit 62 generates the LR matching degree map by calculating the pixel values of the pixels of the LR matching degree map, and then supplies the generated LR matching degree map to the reliability calculation unit 63 L. Further, the matching degree calculation unit 62 calculates Equation (11) below based on the LR parallax map and the RL parallax map to calculate the pixel values C LR (i, j) of the pixels of the RL matching degree map.

C RL ( i,j )= N 4 (| DP RL ( i,j )+ DP LR ( i+DP RL ( i,j ), j )|)  (11)

In Equation (11), DP LR (i, j) and DP RL (i, j) denote the pixel value of a pixel of the LR parallax map and the pixel value of a pixel of the RL parallax map, respectively, and N 4 (x) denotes the same function as the function N 4 (x) of Equation (10).

The matching degree calculation unit 62 generates the RL matching degree map by calculating the pixel values of the pixels of the RL matching degree map, and then supplies the generated RL matching degree map to the reliability calculation unit 63 R.

In step S 45 , the reliability calculation units 63 generate the reliability maps based on the residual maps and the peak sharpness maps from the corresponding point detection units 61 and the matching degree maps from the matching degree calculation units 62 .

For example, the reliability calculation unit 63 L calculates a pixel value R LR (i, j) of the interest pixel by setting each pixel of the LR reliability map to be generated as the interest pixel and calculating Equation (12) below based on the pixel value of each pixel of each of the LR residual map, the LR matching degree map, and the LR peak sharpness map located at the position of the interest pixel.

R LR ( i,j )=α(1 −E LR ( i,j ))+β C LR ( i,j )+γ PK LR ( i,j )  (12)

In Equation (12), E LR (i, j), C LR (i, j), and PK LR (i, j) denote the pixel values of the pixels of the LR residual map, the LR matching degree map, and the LR peak sharpness map, respectively. Further, α, β, and γ are predetermined weights set in advance. The weights are determined to satisfy Equation (13) below.

α+β+γ=1  (13)

Thus, the LR reliability map is calculated by adding the LR residual map, the LR matching degree map, and the LR peak sharpness map in a weighted manner. Further, the smaller the residual between a pixel of the left image and the corresponding pixel of the pixel of the left image is, the larger the peak sharpness of the residual is, and the higher the matching degree between the pixels is, the larger the pixel value of the pixel of the LR reliability map is. Accordingly, the larger the pixel value of the pixel of the LR reliability map is, the higher the reliability of the parallax represented by the LR parallax map is.

The reliability calculation unit 63 L generates the LR reliability map, and then supplies the LR reliability map to the depth parallax conversion unit 31 L and the parallax synthesis unit 32 L.

The reliability calculation unit 63 R calculates Equation (14) using the RL residual map, the RL matching degree map, and the RL peak sharpness map and the weights α, β, and γ determined in Equation (13) above to calculates R RL (i, j) of each pixel of the RL reliability map.

R RL ( i,j )=α(1− E RL ( i,j ))+β C RL ( i,j )+γ PK RL ( i,j )  (14)

In Equation (14), E RL (i, j), C RL (i, j), and PK RL (i, j) denote the pixel values of the pixels of the RL residual map, the RL matching degree map, and the RL peak sharpness map, respectively. The reliability calculation unit 63 R generates the RL reliability map, and then supplies the RL reliability map to the depth parallax conversion unit 31 R and the parallax synthesis unit 32 R.

When the reliability maps are generated, the parallax map generation process ends, and then the process proceeds to step S 12 of FIG. 4 .

Thus, the parallax detection unit 21 generates the parallax maps and the reliability maps from the right and left images. When not only the parallax maps but also the reliability maps indicating the reliability are generated, it is possible to specify how the parallax represented by the parallax maps is reliable for each region of a stereoscopic image. Accordingly, the corrected parallax maps indicating the more accurate parallax can be obtained in the rear stage.

Description of Corrected Parallax Map Generation Process

Next, a corrected parallax map generation process corresponding to the process of step S 15 of FIG. 4 will be described with reference to the flowchart of FIG. 13 .

In step S 71 , the multiplying units 101 multiply the pseudo-parallax maps supplied from the depth parallax conversion units 31 by a weight.

For example, when the RL reliability map is supplied from the reliability calculation unit 63 R, the parallax synthesis unit 32 R supplies the multiplying unit 102 R with the pixel value R RL (i, j) of each pixel of the RL reliability map as the weight of each pixel without change. Further, the parallax synthesis unit 32 R supplies the multiplying unit 101 R with a value of “1−R RL (i, j)” obtained by subtracting the pixel value R RL (i, j) of each pixel of the RL reliability map from 1 as the weight of each pixel.

The multiplying unit 101 R multiplies the pixel value PDP R (i, j) of each pixel of the R pseudo-parallax map by the supplied weight “1−R RL (i, j)” of each pixel, and then supplies the calculation result to the adding unit 103 R. At this time, the pixel value of each pixel of the R pseudo-parallax map is multiplied by the weight obtained from the pixel value of each pixel of the reliability map located at the same position as each pixel of the R pseudo-parallax map.

As in the parallax synthesis unit 32 R, the multiplying unit 101 L of the parallax synthesis unit 32 L multiplies the pixel value PDP L (i, j) of each pixel of the L pseudo-parallax map supplied from the depth parallax conversion unit 31 L by a supplied weight “1−R LR (i, j)”, and then supplies the calculation result to the adding unit 103 L.

In step S 72 , the multiplying units 102 multiply the parallax maps supplied from the corresponding point detection units 61 by the supplied weights, and then supply the calculation result to the adding units 103 .

For example, the multiplying unit 102 R multiplies the pixel value of each pixel of the RL parallax map by the weight R RL (i, j) of each pixel, and then supplies the calculation result to the adding unit 103 R. Likewise, the multiplying unit 102 L multiplies the pixel value of each pixel of the LR parallax map by the weight R LR (i, j) of each pixel, and then supplies the calculation result to the adding unit 103 L.

In step S 73 , the adding units 103 add the pseudo-parallax maps from the multiplying units 101 to the parallax maps from the multiplying units 102 to generate the corrected parallax maps and output the corrected parallax maps.

For example, the adding unit 103 R sets a value, which is obtained by adding the R pseudo-parallax map multiplied by the weight to the pixel value of the pixel located at the same position as the RL parallax map multiplied by the weight, to a pixel value DP RL ′(i, j) of each pixel of the corrected RL parallax map located at the position as the pixel.

That is, Equation (15) is calculated to obtain the pixel value DP RL ′(i, j) of each pixel of the corrected RL parallax map.

DP RL ′( i,j )= R RL ( i,j )× DP RL ( i,j )+(1 −R RL ( i,j ))× PDP R ( i,j )  (15)

Of the pixels of the RL parallax map, the pixels with high reliability are used without conversion as the pixels of the corrected RL parallax map by the calculation of Equation (15). As for the pixels with low reliability, the pixels of the R pseudo-parallax map are used as the pixels of the corrected RL parallax map. Thus, the corrected parallax map indicating the more reliable parallax can be obtained in accordance with the reliability of each pixel of the parallax map. Further, to convert the parallax map and the pseudo-parallax map more smoothly, a smoothing process may be performed on the reliability map before the parallax map and the pseudo-parallax map are synthesized.

Likewise, the adding unit 103 L adds the L pseudo-parallax map multiplied by the weight to the pixel value of each pixel located at the same position as the LR parallax map multiplied by the weight to generate the corrected LR parallax map, and then outputs the corrected LR parallax map.

When the corrected parallax maps are generated and output, the corrected parallax map generation process ends. Then, the process of step S 15 in FIG. 4 also ends and the parallax detection process thus ends.

As described above, the parallax synthesis units 32 synthesize the parallax maps and the pseudo-parallax maps to generate the corrected parallax maps. Thus, more accurate and stable parallax information can be obtained by synthesizing the parallax maps obtained from the right and left images and the pseudo-parallax maps obtained from one image in accordance with the reliabilities of the parallax maps and generating the corrected parallax maps.

Modified Example 1

The foregoing description pertains to an example in which each parallax map and each pseudo-parallax maps are added to each other in the weighted manner in accordance with the reliability and are synthesized. However, only the high-pass component of the pseudo-parallax map may be used for the synthesis.

In this case, the parallax synthesis unit 32 R in FIG. 1 has a configuration shown in, for example, FIG. 14 . In FIG. 14 , the same reference numerals are given to units corresponding to the units shown in FIG. 3 and the description thereof will appropriately not be repeated.

A parallax synthesis unit 32 R in FIG. 14 includes a high-pass component extraction unit 131 R, a low-pass component extraction unit 132 R, a component synthesis unit 133 R, a multiplying unit 101 R, a multiplying unit 102 R, and an adding unit 103 R.

The high-pass component extraction unit 131 R extracts a high-pass component from the R pseudo-parallax map supplied from the depth parallax conversion unit 31 R, and then supplies the high-pass component to the component synthesis unit 133 R. Further, the low-pass component extraction unit 132 R extracts a low-pass component from the RL parallax map supplied from the corresponding point detection unit 61 R, and then supplies the low-pass component to the component synthesis unit 133 R.

The component synthesis unit 133 R synthesizes the high-pass component from the high-pass component extraction unit 131 R and the low-pass component from the low-pass component extraction unit 132 R, and then supplies the synthesized map obtained as the synthesis result to the multiplying unit 101 R. The multiplying unit 101 R multiples the pixel value of each pixel of the synthesized map supplied from the component synthesis unit 133 R by a supplied weight “1−R RL (i, j), and then supplies the calculation result to the adding unit 103 R.

When the parallax synthesis unit 32 R in FIG. 1 has the configuration shown in FIG. 14 , the parallax synthesis unit 32 L also has the same configuration as the parallax synthesis unit 32 R shown in FIG. 14 . Therefore, the parallax synthesis unit 32 L is not illustrated and will not be described. Hereinafter, the units of the parallax synthesis unit 32 L corresponding to the high-pass component extraction unit 131 R, the low-pass component extraction unit 132 R, and the component synthesis unit 133 R are referred to as a high-pass component extraction unit 131 L, a low-pass component extraction unit 132 L, and a component synthesis unit 133 L, respectively.

When it is not necessary to distinguish between the high-pass component extraction units 131 R and 131 L, between the low-pass component extraction units 132 R and 132 L, and between the component synthesis units 133 R and 133 L, respectively, the high-pass component extraction units 131 R and 131 L, the low-pass component extraction units 132 R and 132 L, and the component synthesis units 133 R and 133 L are referred to as the high-pass component extraction units 131 , the low-pass component extraction units 132 , and the component synthesis units 133 , respectively, below.

Description of Corrected Parallax Map Generation Process

When each parallax synthesis unit 32 includes the high-pass component extraction unit 131 , the low-pass component extraction unit 132 , and the component synthesis unit 133 , a corrected parallax map generation process in FIG. 15 is performed in step S 15 of the parallax detection process described above in FIG. 4 .

Hereinafter, the corrected parallax map generation process of the parallax synthesis unit 32 will be described with reference to the flowchart of FIG. 15 .

In step S 101 , the high-pass component extraction units 131 extract the high-pass components from the pseudo-parallax maps by performing a filtering process or the like on the pseudo-parallax maps supplied from the depth parallax conversion units 31 , and then supply the extracted high-pass components to the component synthesis units 133 . In step S 101 , each of the high-pass component extraction units 131 R and 131 L extracts the high-pass component.

In step S 102 , the low-pass component extraction units 132 extract the low-pass components from the parallax maps by performing a filtering process or the like on the parallax maps supplied from the corresponding point detection units 61 , and then supply the extracted low-pass components to the component synthesis units 133 . In step S 102 , each of the low-pass component extraction units 132 R and 132 L also extracts the low-pass component.

In step S 103 the component synthesis units 133 synthesize the high-pass components from the high-pass component extraction units 131 and the low-pass components from the low-pass component extraction units 132 , and then supply the synthesized maps obtained as the synthesis result to the multiplying units 101 .

For example, the pixel values of the pixels located at the same positions as the high-pass components and the low-pass components are added and are considered as the pixel values of the pixels of the synthesized maps located at the pixels. In step S 103 , each of the component synthesis units 133 R and 133 L also generates the synthesized map.

In step S 104 , the adding units 103 add the synthesized maps to the parallax maps to generate the corrected parallax maps.

For example, the multiplying unit 101 R multiples the pixel value of each pixel of the synthesized map supplied from the component synthesis unit 133 R by the weight “1−R RL (i, j), and then supplies the calculation result to the adding unit 103 R. Further, the multiplying unit 102 R multiples the pixel value of each pixel of the RL parallax map from the corresponding point detection 61 R by a weight R RL (i, j), and then supplies the calculation result to the adding unit 103 R. Then, the adding unit 103 R adds the synthesized map from the multiplying unit 101 R to the RL parallax map from the multiplying unit 102 R to generate the corrected RL parallax map, and then the corrected RL parallax map.

When each synthesized map and each parallax map are synthesized, the pixels with high reliability are used without change as the pixels of each corrected parallax map. As for the pixels with low reliability, only the low-pass components of the parallax map are used and the pixels of the pseudo-parallax map are used as the high-pass components. Further, to convert the parallax map and the synthesized map more smoothly, a smoothing process may be performed on the reliability map before the parallax map and the synthesized map are synthesized.

The multiplying unit 101 L, the multiplying unit 102 L, and the adding unit 103 L perform the same processes as the multiplying unit 101 R, the multiplying unit 102 R, and the adding unit 103 R, respectively, to generate the corrected LR parallax map. When the corrected parallax map is generated and output, the corrected parallax map generation process ends. Then, the process of step S 15 in FIG. 4 also ends and the parallax detection process thus ends.

Modified Example 2

In FIG. 14 , The foregoing description pertains to an example in which the high-pass component of the pseudo-parallax map and the low-pass component of the parallax map are synthesized to generate the synthesized map. However, the low-pass component of the pseudo-parallax map and the high-pass component of the parallax map may be synthesized to obtain the synthesized map.

In this case, the parallax synthesis unit 32 R in FIG. 1 has a configuration shown in, for example, FIG. 16 . In FIG. 16 , the same reference numerals are given to units corresponding to the units shown in FIG. 14 and the description thereof will appropriately not be repeated.

A synthesis unit 32 R in FIG. 16 includes a high-pass component extraction unit 131 R, a low-pass component extraction unit 132 R, a component synthesis unit 133 R, a multiplying unit 101 R, a multiplying unit 102 R, and an adding unit 103 R. In the parallax synthesis unit 32 R in FIG. 16 , the RL parallax map is supplied to the high-pass component extraction unit 131 R and the R pseudo-parallax map is supplied to the low-pass component extraction unit 132 R.

The high-pass component extraction unit 131 R extracts a high-pass component from the RL parallax map supplied from the corresponding point detection unit 61 R, and then supplies the high-pass component to the component synthesis unit 133 R. Further, the low-pass component extraction unit 132 R extracts a low-pass component from the R pseudo-parallax map supplied from the depth parallax conversion unit 31 R, and then supplies the low-pass component to the component synthesis unit 133 R.

When the parallax synthesis unit 32 R has the configuration shown in FIG. 16 , the parallax synthesis unit 32 L also has the same configuration as the parallax synthesis unit 32 R shown in FIG. 16 . Therefore, the parallax synthesis unit 32 L is not illustrated and will not be described.

Description of Corrected Parallax Map Generation Process

When the parallax synthesis unit 32 R has the configuration shown in FIG. 16 , a corrected parallax map generation process shown in FIG. 17 is performed in step S 15 of the parallax detection process in FIG. 4 .

Hereinafter, the corrected parallax map generation process of the parallax synthesis unit 32 will be described with reference to the flowchart of FIG. 17 .

In step S 131 , the low-pass component extraction units 132 extract the low-pass components from the pseudo-parallax maps by performing a filtering process or the like on the pseudo-parallax maps supplied from the depth parallax conversion units 31 , and then supply the extracted low-pass components to the component synthesis units 133 . In step S 131 , each of the low-pass component extraction units 132 R and 132 L extracts the low-pass component.

In step S 132 , the high-pass component extraction units 131 extract the high-pass components from the parallax maps by performing a filtering process or the like on the parallax maps supplied from the corresponding point detection units 61 , and then supply the extracted high-pass components to the component synthesis units 133 . In step S 132 , each of the high-pass component extraction units 131 R and 131 L also extracts the high-pass component.

After the process of step S 132 , the processes of step S 133 and step S 134 are performed and the corrected parallax map generation process ends. Since these processes are the same as the processes of step S 103 and step S 104 in FIG. 15 , the description thereof will not be repeated. However, in step S 133 , the low-pass component of the pseudo-parallax map and the high-pass component of the parallax map are synthesized to generate the synthesized map.

When the corrected parallax map generation process ends, step S 15 in FIG. 4 also ends and the parallax detection process thus ends.

In the corrected parallax map generation process in FIG. 17 , the pixels with high reliability are used without change as the pixels of each corrected parallax map. As for the pixels with low reliability, only the high-pass components of the parallax map are used and the pixels of the pseudo-parallax map are used as the low-pass components. Further, to convert the parallax map and the synthesized map more smoothly, a smoothing process may be performed on the reliability map before the parallax map and the synthesized map are synthesized.

Second Embodiment

The foregoing description pertains to the case in which the parallax map and the pseudo-parallax map are synthesized to generate the corrected parallax map. However, the parallax maps may be generated using the right and left images and the pseudo-parallax maps.

In this case, the image processing apparatus has a configuration shown in, for example, FIG. 18 . In FIG. 18 , the same reference numerals are given to units corresponding to the units shown in FIG. 1 and the description thereof will appropriately not be repeated.

An image processing apparatus 171 in FIG. 18 includes a depth information estimation unit 22 R, a depth information estimation unit 22 L, a depth parallax conversion unit 31 R, a depth parallax conversion unit 31 L, and a parallax detection unit 181 .

The depth information estimation unit 22 R generates an R depth information map based on a supplied right image and supplies the R depth information map to the depth parallax conversion unit 31 R. Further, the depth information estimation unit 22 L generates an L depth information map based on a supplied left image and supplies the L depth information map to the depth parallax conversion unit 31 L.

The depth parallax conversion unit 31 R the R depth information map supplied from the depth information estimation unit 22 R into an R pseudo-parallax map, and then supplies the converted R pseudo-parallax map to the parallax detection unit 181 . The depth parallax conversion unit 31 L converts the L depth information map supplied from the depth information estimation unit 22 L into an L pseudo-parallax map, and then supplies the converted L pseudo-parallax map to the parallax detection unit 181 .

The parallax detection unit 181 generates a parallax map based on the supplied right and left images and the pseudo-parallax maps from the depth parallax conversion units 31 and outputs the generated parallax map. The parallax detection unit 181 includes corresponding point detection units 191 R and 191 L.

The corresponding point detection unit 191 R generates an RL parallax map based on the supplied right and left images and the R pseudo-parallax map supplied from the depth parallax conversion unit 31 R, and then outputs the generated RL parallax map. Further, the corresponding point detection unit 191 L generates an LR parallax map based on the supplied right and left images and the L pseudo-parallax map supplied from the depth parallax conversion unit 31 L, and then outputs the generated LR parallax map.

When it is not necessary to distinguish between the corresponding point detection units 191 R and 191 L, the corresponding point detection units 191 R and 191 L are simply referred to as the corresponding point detection units 191 below.

Description of Parallax Detection Process

Next, a parallax detection process of the image processing apparatus 171 will be described with reference to the flowchart of FIG. 19 .

In step S 161 , the depth information estimation units 22 generate the depth information maps based on the supplied right and left images, and then supply the generated depth information maps to the depth parallax conversion units 31 . In step S 161 , the same process as the process of step S 12 in FIG. 14 is performed.

In step S 162 , the depth parallax conversion units 31 convert the depth information maps supplied from the depth information estimation units 22 into the pseudo-parallax maps, and then supply the converted pseudo-parallax maps to the corresponding point detection unit 191 .

For example, a conversion equation used to convert depth information such as “p×O R (i, j)+q)” described above to parallax information is recorded in advance in the depth parallax conversion units 31 . In this case, the depth parallax conversion unit 31 R converts the R depth information map into the R pseudo-parallax map using the recorded conversion equation and supplies the result to the corresponding point detection unit 191 R. That is, the same process as the process of step S 14 in FIG. 4 is performed.

Further, the depth parallax conversion unit 31 L also converts the L depth information map into the L pseudo-parallax map using the recorded conversion equation and supplies the result to the corresponding point detection unit 191 L.

In step S 163 , the corresponding point detection units 191 generate parallax maps based on the supplied right and left images and the pseudo-parallax maps from the depth parallax conversion units 31 .

For example, the corresponding point detection unit 191 L sets one pixel of the left image I L to an interest pixel ge, as shown in FIG. 20 . Here, the coordinates of the interest pixel ge in the ij coordinate system are (i, j). Next, the corresponding point detection unit 191 L specifies a pixel value PDP L (i, j) of a pixel of the L parallax map located at the same position as the interest pixel ge and sets a region, which has a preset size and centers coordinates (i+PDP L (i, j), j) on the right image I R , to a search region SR.

In the L pseudo-parallax map, the corresponding pixel of the interest pixel ge is a pixel ge′ located at the coordinates (i+PDP L (i, j), j) on the right image I R . Therefore, there is a high possibility that the genuine corresponding pixel of the interest pixel ge is present near the pixel ge′.

Accordingly, the corresponding point detection unit 191 L sets each pixel present within the search region SR near the pixel ge′ as a candidate of the corresponding pixel and calculates a residual Ri, j(d) of each candidate by Equation (3) described above. Then, the corresponding point detection unit 191 L calculates the pixel value DP LR (i, j) of each pixel of the LR parallax map located at the same position as the interest pixel ge based on the residual and Equation (4).

The corresponding point detection unit 191 L generates the LR parallax map by sequentially setting the pixels of the left image to the interest pixel and calculating the pixel value DP LR (i, j) of each pixel of the LR parallax map located at the same position as the interest pixel.

Thus, the corresponding pixel can be more accurately and reliably by setting only each pixel present within the search region (search range) of the right image determined by the pseudo-parallax map to the candidate of the corresponding pixel, when the corresponding pixel corresponding to each pixel of the left image. Further, since the search range is restricted by the pseudo-parallax map, the corresponding pixel can be detected more rapidly.

The corresponding point detection unit 191 L outputs the generated LR parallax map. Further, the corresponding point detection unit 191 R performs the same process as the corresponding point detection unit 191 L based on the right and left images and the R pseudo-parallax map to generate the RL parallax map, and then outputs the RL parallax map.

When the parallax maps are generated and output by the corresponding point detection units 191 , the parallax detection process ends.

The image processing apparatus 171 generates the pseudo-parallax maps based on the right and left images and generates the parallax maps based on the right and left images and the pseudo-parallax maps. By appropriately setting the search range of the corresponding pixel based on the pseudo-parallax maps when the parallax maps are generated, the parallax can be reliably detected more rapidly and accurately.

The above-described series of processes may be performed by hardware or software. When the series of processes are performed by software, a program for the software is installed on, for example, a computer in which dedicated hardware is embedded or a general personal computer capable of executing various functions by installing various programs from a program recording medium.

FIG. 21 is a block diagram illustrating an example of a hardware configuration of a computer executing the above-described series of processes by a program.

In the computer, a central processing unit (CPU) 501 , a read-only memory (ROM) 502 , and a random access memory (RAM) 503 are connected to each other via a bus 504 .

An input/output interface 505 is connected to the bus 504 . An input unit 506 configured by a keyboard, a mouse, a microphone, or the like, an output unit 507 configured by a display, a speaker, or the like, a recording unit 508 configured by a hard disk, a non-volatile memory, or the like, a communication unit 509 configured by a network interface or the like, and a drive 510 driving a removable medium 511 such as a magnetic disk, an optical disc, a magneto-optical disk, or a semiconductor memory are connected to the input/output interface 505 .

In the computer with the above-described configuration, the CPU 501 executes the above-described series of processes, for example, by loading and executing the program stored in the recording unit 508 on the RAM 503 via the input/output interface 505 and the bus 504 .

For example, the program executed by the computer (the CPU 501 ) is recorded in the removable medium 511 as a package medium, such as a magnetic disk (including a flexible disk), an optical disc (including a compact disc-read only memory (CD-ROM) and a digital versatile disc (DVD)), a magneto-optical disc, or a semiconductor memory, for supply. Further, the program can be supplied via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcast.

The program can be installed on the recording unit 508 via the input/output interface 505 by mounting the removable medium 511 on the drive 510 . Further, the program can be received by the communication unit 509 via a wired or wireless transmission medium and can be installed on the recording unit 508 . Further, the program can be installed in advance on the ROM 502 or the recording unit 508 .

The program executed by the computer may be a program processed chronologically in the order described in the specification or a program processed in parallel or at a necessary timing when called.

Embodiments of the present technology are not limited to the above-described embodiments, but may be modified in various ways without departing from the spirit and scope of the present technology.

The present technology may be configured as follows.

[1] An image processing apparatus including:

a parallax map generation unit that generates a parallax map indicating a parallax of each pixel of a stereoscopic image based on right and left images forming the stereoscopic image;

a reliability calculation unit that generates a reliability map indicating reliability of the parallax map based on the right and left images;

a depth information estimation unit that generates a depth information map indicating a depth of a subject on the stereoscopic image based on one of the right and left images;

a depth parallax conversion unit that converts the depth information map into a pseudo-parallax map indicating a parallax corresponding to the depth; and

a parallax synthesis unit that synthesizes the parallax map and the pseudo-parallax map to generate a corrected parallax map based on the reliability map.

[2] The image processing apparatus according to [1], wherein the parallax synthesis unit generates the corrected parallax map by adding the parallax map and the pseudo-parallax map in a weighted manner using a weight determined by the reliability map.

[3] The image processing apparatus according to [1], wherein the parallax synthesis unit includes

a component synthesis unit that synthesizes a high-pass component of the pseudo-parallax map and a low-pass component of the parallax map to generate a synthesized map, and

an adding unit that synthesizes the parallax map and the synthesized map to generate the corrected parallax map based on the reliability map.

[4] The image processing apparatus according to [1], wherein the parallax synthesis unit includes

a component synthesis unit that synthesizes a low-pass component of the pseudo-parallax map and a high-pass component of the parallax map to generate a synthesized map, and

an adding unit that synthesizes the parallax map and the synthesized map to generate the corrected parallax map based on the reliability map.

[5] The image processing apparatus according to any one of [1] to [4], further comprising:

a residual map generation unit that generates a residual map indicating a residual between a region near a predetermined pixel of one of the right and left images and a region near a corresponding pixel of the other image which corresponds to the predetermined pixel,

wherein the reliability calculation unit generates the reliability map based on the residual map.

[6] The image processing apparatus according to any one of [1] to [4],

wherein the parallax map generation unit generates the parallax map by detecting, as a pixel corresponding to a predetermined pixel of one of the right and left images, a pixel of the other image for which a residual between a region near the predetermined pixel of the one image and a region near a pixel of the other image is minimum, and

wherein the image processing apparatus further comprises:

a peak sharpness map generation unit that generates a peak sharpness map that indicates sharpness of a peak of the residual of the pixels lined up in a predetermined direction on the other image and is used for generating the reliability map.

[7] The image processing apparatus according to any one of [1] to [4], further including:

a matching degree calculation unit that generates a matching degree map indicating a matching degree between a predetermined pixel of one of the right and left images and a first corresponding pixel of the other image corresponding to the predetermined pixel based on a detection result of the first corresponding pixel of the other image and a detection result of a second corresponding pixel of the one image corresponding to the first corresponding pixel of the other image,

wherein the reliability calculation unit generates the reliability map based on the matching degree map.

The present disclosure contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2011-126403 filed in the Japan Patent Office on Jun. 6, 2011, the entire content of which is hereby incorporated by reference.