Apparatus and method for processing an image

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2012-224520, filed on Oct. 9, 2012; the entire contents of which are incorporated herein by reference.

Embodiments described herein relate generally to an apparatus and a method for controlling an apparent gloss and an apparent brightness of an input image.

Many techniques to change emphasis of feeling of material (especially, “apparent gloss” and “apparent shininess”) of an object in an image by image processing are proposed. As to a first method of conventional technique, a model to link “sensitivity amount of high order” (such as a feeling of metallic material) with the image is trained, and the feeling of material is changed using the model trained. As to a second method of conventional technique, the image is separated into a diffused reflection image (color of the object itself) and a specular reflection image (color reflected from the object by illumination light). By adding the specular reflection image emphasized to the diffuse reflection image, an apparent gloss of the object in the image is emphasized.

However, in above-mentioned two methods, many computer resources are necessary to change the apparent gloss.

FIG. 1 is a block diagram of an image processing apparatus according to the first embodiment.

FIG. 2 is a block diagram of the image processing apparatus according to a modification of the first embodiment.

FIG. 3 is a flow chart of processing of the image processing apparatus according to the first embodiment.

FIG. 4 is a schematic diagram to explain a function to change a response of parameter.

FIG. 5 is a schematic diagram to explain a function to compensate gradation.

FIG. 6 is a block diagram of the image processing apparatus according to the second embodiment.

FIG. 7 is a flow chart of the image processing apparatus according to the second embodiment.

FIG. 8 is a schematic diagram to explain a reference range to calculate an average.

According to one embodiment, an image processing apparatus includes a chroma calculation unit, a lightness calculation unit, a glossiness calculation unit, and a correction unit. The chroma calculation unit is configured to calculate a chroma of each pixel of an image to be processed. The lightness calculation unit is configured to calculate a lightness of each pixel of the image. The glossiness calculation unit is configured to estimate a glossiness of each pixel of the image, based on the chroma and the lightness. The correction unit is configured to correct each pixel of the image so that a correction amount of a pixel is larger when the glossiness of the pixel is higher.

Various embodiments will be described hereinafter with reference to the accompanying drawings.

The First Embodiment

The first embodiment relates to an image processing apparatus to generate an image having an apparent gloss emphasized from an input image (target image). The input image includes a pixel value of each pixel. For example, the pixel value is an image having a luminance signal and a color difference signal based on standard of International Telecommunication Union (Hereinafter, it is called “ITU”). These signals may be based on any of a system to have RGB (three primary colors) components and a system to convert from RGB to the luminance signal and the color difference signal. In the first embodiment, as one example, the pixel value is represented as Y (luminance signal), Cb and Cr (color difference signal) converted by gamut-conversion method based on ITU-RBT.601 standard. Y, Cb and Cr are converted from RGB by following conversion method. Moreover, R is a signal value of luminance of red component, G is a signal value of luminance of green component, and B is a signal value of luminance of blue component.

Y= 219 Y′+ 16

Cb= 224 Pb+ 128

Cr= 224 Pr+ 128

Here, Y′, Pb and Pr are as follows.

Y′= 0.299 R+ 0.587 G+ 0.114 B

Pb= 0.564( B−Y ′)

Pr= 0.713( R−Y ′)

Y, Cb and Cr include discrete values respectively. In the first embodiment, an example that Y, Cb and Cr are represented by eight bits respectively, will by explained. Moreover, the image may be a single image or sequential images.

FIG. 1 is a block diagram of the image processing apparatus according to the first embodiment. The image processing apparatus includes a calculation unit 100 , a correction unit 200 , and a correction amount calculation unit 300 . Furthermore, the calculation unit 100 includes a chroma calculation unit 101 , a lightness calculation unit 102 , a gradient calculation unit 103 , and a glossiness calculation unit 110 .

The calculation unit 100 calculates a glossiness from an input image. The glossiness is an evaluation value to evaluate how many components due to gloss are added to a pixel value of each pixel in the input image. In the first embodiment, if components due to gloss are more added, the glossiness becomes higher. The gloss is a phenomenon that, a surface of an object (subject) is brightened by reflecting a light thereon, or another image is included in the input image.

The chroma calculation unit 101 calculates a chroma from each pixel value of the input image. The chroma is a scale of vividness of color. The chroma is in proportion to a distance from a center axis (achromatic color axis) in a color space. If the distance is longer, a value of the chroma becomes larger.

The lightness calculation unit 102 calculates a lightness from each pixel in the input image. The lightness is a scale of brightness of color. If the pixel value is brighter, a value of the lightness becomes larger.

Hereinafter, an example to calculate the chroma and the lightness in HSV color space will be explained. However, the chroma and the lightness may be calculated in another color space. For example, the chroma and the lightness mat be values calculated using HLS or YCbCr.

The gradient calculation unit 103 calculates a gradient intensity by referring to a target pixel and other pixels (surrounding pixels) around the target pixel in the input image. The gradient intensity is an evaluation value representing an intensity of gradient of the target pixel. If the gradient is stronger, a value of the gradient intensity becomes larger. The gradient of the image represents a degree how much a pixel value of the target pixel is changed from pixel values of the surrounding pixels.

The glossiness calculation unit 110 calculates a glossiness of each pixel from the chroma, the lightness and the gradient intensity. In the first embodiment, the glossiness is calculated based on following assumptions.

(Assumption 1) As to a pixel of which chroma is lower and lightness is higher, a glossiness of the pixel is higher.

(Assumption 2) As to a pixel of which gradient intensity is low degree˜middle degree, a glossiness of the pixel is higher.

A concrete method for calculating the gloss will be explained afterwards. The correction amount calculation unit 300 calculates a correction amount of a pixel value of each pixel according to the glossiness calculated.

Next, an operation of the image processing apparatus of the first embodiment will be explained. FIG. 3 is a flow chart of the image processing apparatus of the first embodiment.

The chroma calculation unit 101 calculates a chroma of each pixel of the image (S 11 ). In the first embodiment, a component to calculate a chroma and a lightness represented by HSV color space is shown as an example. In HSV color space, the chroma S and the lightness V are calculated from signal values of RGB. YCbCr [BT.601] (0˜255) as pixel values of the input image can be converted to pixel values of RGB(0.0˜1.0) format based on an equation (1).

Here, the chroma S(x,y) of a pixel at a coordinate (x,y) is calculated by an equation (2).

S ( x,y )=(MAX (x,y) ( R,G,B )−MIN (x,y) ( R,G,B )/MAX (x,y) ( R,G,B )  (2)

Here, MAX (x,y) (R,G,B) is a maximum among pixel values of respective RGB channels of a pixel at a coordinate (x,y). MIN (x,y) (R,G,B) is a minimum among pixel values of respective RGB channels of a pixel at a coordinate (x,y).

The lightness calculation unit 102 calculates a lightness of each pixel value (S 12 ). A lightness V(x,y) is calculated by an equation (3).

V ( x,y )=MAX (x,y) ( R,G,B )  (3)

The gradient calculation unit 103 calculates a gradient intensity of a target pixel from pixel values of the target pixel and surrounding pixels thereof (S 13 ). In the first embodiment, an example to calculate a gradient using a Roberts filter will be explained. The Roberts filter calculates a gradient dx along x-direction and a gradient dy along y-direction based on equations (4) and (5).

dx ( x,y )= I ( x,y )− I ( x+ 1 ,y+ 1)  (4)

dy ( x,y )= I ( x+ 1 ,y )− I ( x,y+ 1)  (5)

Here, I(x,y) represents a luminance (Y) of a pixel at a coordinate (x,y). In the equations (4) and (5), a gradient is calculated from a difference between a luminance of a target pixel at the coordinate (x,y) and a luminance of another pixel adjacent to the target pixel along x-direction or y-direction. A gradient intensity D (x,y) of the target pixel at the coordinate (x,y) is calculated by an equation (6).

D ( x,y )=| dx ( x,y )|+| dy ( x,y )|  (6)

As a method for calculating the gradient, a Sobel filter or a difference between a maximum and a minimum of pixel values of surrounding pixels may be used.

Furthermore, a gradient of not luminance I(x,y) but lightness V(x,y) may be used. Furthermore, a gradient of a value digitizing (Assumption 1) or a whiteness degree f1(x,y) may be calculated. Furthermore, these gradients may be combined.

A glossiness F(x,y) as an evaluation value to digitize assumption related to gloss of each pixel is calculated (S 14 ). In the first embodiment, the glossiness calculation unit 110 digitizes (Assumption 1) by an equation (7), and the digitized value is represented as f 1 (x,y).

f 1 ( x,y )=(1.0 −S ( x,y ))· V ( x,y )  (7)

The chroma S(x,y) and the lightness V(x,y) are normalized to a segment (0.0˜1.0) respectively. Value range of f 1 (x,y) is (0.0˜1.0). If the chroma S(x,y) is nearer to 0.0 and if the lightness V(x,y) is nearer to 1.0, f 1 (x,y) becomes higher. At a pixel having white approximately, f 1 (x,y) is high. Accordingly, f 1 (x,y) is called a whiteness degree. Here, by expanding the right side of the equation (7), f 1 (x,y) is defined as “f 1 (x,y)=MIN (x,y) (R,G,B)”. By using minimum of RGB, the whiteness degree f 1 (x,y) can be calculated. Accordingly, calculation cost thereof can be reduced.

When f 1 (x,y) is calculated, as shown on an equation (8), a function g 1 for adjustment may be used.

f 1 ( x,y )= g 1 ((1.0 −S ( x,y ))· V ( x,y ))  (8)

FIG. 4 shows an example of the function g 1 . By using the function g 1 having a response shown in FIG. 4 , f 1 (x,y) may be calculated. By changing the response using the function for adjustment, adjustment between the whiteness degree and an emphasis amount can be performed. For example, by selectively emphasizing pixels of which whiteness degree is high, unnecessary change of pixels of which whiteness degree is low can be prevented.

Furthermore, the glossiness calculation unit 110 calculates a glossiness based on (Assumption 2) by an equation (9)

f 2 ( x,y )= g 2 ( D )  (9)

In the equation (9), a gradient intensity D(x,y) is normalized to a segment (0.0˜1.0). Here, if the gradient intensity of a pixel is nearer to a specific value, the function g 2 assigns a larger weight to the pixel. By setting the specific value to a value of low degree˜middle degree, a glossiness f 2 based on (assumption 2) can be calculated. In this case, f 2 (x,y) is noted as an equation (10).

f 2 ( x,y )=exp(−0.5·( D−μ g2 )/σ 2 g2 )  (10)

In the equation (10), μ g2 and σ 2 g2 are parameters for adjustment. Specifically, μ g2 can adjust an average of gradient intensity to be weighted, and σ 2 g2 can adjust range of the weight. If the gradient intensity D of a target pixel is nearer to μ g2 , f 2 (x,y) thereof is larger.

The glossiness calculation unit 110 calculates a glossiness F(x,y) from f 1 (x,y) and f 2 (x,y) by an equation (11).

F ( x,y )= f 1 ( x,y )· f 2 ( x,y )  (11)

The correction amount calculation unit 300 calculates a correction amount of a pixel value of the target pixel based on the glossiness F(x,y) (S 15 ). In the first embodiment, a component to calculate a correction amount based on the glossiness and to emphasize a gloss by adding the correction amount to an input luminance signal (Y in YCbCr) will be explained as an example. The first embodiment is not limited to this component. For example, a component to calculate a correction amount by constantly multiplying the luminance signal based on the glossiness may be used. Furthermore, not only a component to emphasize the gloss but also a component to suppress the gloss may be used. In the first embodiment, the correction amount is defined as “γ×F(x,y)×Y max ”. Here, Y max is a maximum of the luminance signal value. In case of YCbCr [BT.609] (eight bits), Y max is “235”. Furthermore, γ is a gain parameter to adjust the correction amount.

The correction unit 200 corrects pixel values according to the correction amount (S 16 ). A luminance signal value Y′ of which gloss is emphasized based on the correction amount is calculated according to an equation (12).

Y ′( x,y )= Y ( x,y )+γ· F ( x,y )· Y max   (12)

Moreover, in the first embodiment, in order to prevent a gradation of luminance from saturating at a bright part, by calculating Y out (x,y) using a function of order 0.5 shown in FIG. 5 , a gradation of an output image is compensated. By indicating two points (l in , l out ) and (h in h out ) as shown in FIG. 5 , a luminance signal Y out (x,y) of which gradation is compensated is calculated by an equation (12).

Y out ( x,y )= a ·√{square root over ( Y ′( x,y ))}+ b

a =( h out −l out )/√{square root over ( h in )}−√{square root over ( l in )})

b=h out −a ·√{square root over (h in )}  (13)

Above-mentioned compensation method is one example, and different component may be used. By emphasizing a luminance of pixel value having high glossiness, the gradation is often lost, i.e., saturation is occurred at a pixel having high luminance. However, in the first embodiment, the saturation can be corrected. Moreover, a component to compensate the gradation is not always necessary. A component not to compensate the gradation may be used.

In above explanation, an example to correct the luminance signal is already explained. However, a color difference signal may be corrected based on the purpose. Examples of correct equation of CbCr are shown as equations (14) and (15).

Cb ′( x,y )= Cb ( x,y )+( Cb ( x,y )−128)×( Y ′( x,y )− Y ( x,y ))×( Y max −Y ( x,y ))×λ  (15)

Cr ′( x,y )= Cr ( x,y )+( Cr ( x,y )−128)×( Y ′( x,y )− Y ( x,y ))×( Y max −Y ( x,y ))×λ  (16)

The second term of the right side is a correction amount. Furthermore, λ is a parameter to adjust increase/decrease amount of color difference. By increasing/decreasing CbCr value according to equations (14) and (15), the gloss can be emphasized while vividness of pixel having high chroma is maintained in the input image.

By above-mentioned operation, processing of one pixel is completed. Processing of S 11 ˜S 16 is repeated for all pixels in the input image.

FIG. 2 is a block diagram of modification of the calculation unit 100 of the first embodiment. The calculation unit 100 shown in FIG. 1 is replaced with the calculation unit 100 shown in FIG. 2 . In this modification, a glossiness calculation unit 110 calculates the glossiness from the chroma and the lightness. In this case, by using (Assumption 1), “F(x,y)=f 1 (x,y)” is used.

Recently, a hardware for image processing is installed onto digital products such as a television or a digital camera. However, in case of general digital product, computer resources of this hardware are not abundant, and image processing technique to obtain high effect from few calculation loads is necessary. When a gloss of an object in an image is changed by a method disclosed in JP-A (Kokai) 2011-171807, by detecting a region of the object from the image, a change amount of image characteristic such as chroma needs to be calculated based on a model trained. Furthermore, when a gloss of an object in an image is changed by a method disclosed in JP-A (Kokai) H06-236440, in order to separate each component from the image, a wide region in the image needs to be repeatedly referred.

According to the first embodiment, by using image characteristic (such as chroma, lightness, gradient intensity) simply computable, an image of which gloss is controlled can be generated.

The Second Embodiment

FIG. 6 is a block diagram of an image processing apparatus according to the second embodiment. As to the second embodiment, in comparison with the first embodiment, a correction degree calculation unit 400 is further included. The correction degree calculation unit 400 includes a calculation unit 401 , a calculation unit 402 , and a synthesis unit 410 .

The correction degree calculation unit 400 calculates a correction degree to suppress the glossiness. This correction degree is used to adjust appearance of the output image. The calculation unit 401 calculates a correction degree to suppress the correction amount of glossiness by referring to the whiteness degree of entire image. The calculation unit 402 calculates a correction degree to suppress the correction amount of glossiness by detecting pixels having skin-color. The synthesis unit 410 calculates a correction degree by synthesizing respective correction degrees calculated by the calculation unit 401 and the calculation unit 402 .

Next, operation of the image processing apparatus of the second embodiment will be explained. Here, a component having the calculation unit 401 and the calculation unit 402 will be explained. However, the second embodiment is not limited to this component. For example, by targeting maintenance of readability of characters, a correction degree to suppress the correction amount of glossiness of character regions in the image may be calculated.

FIG. 7 is a flow chart of the image processing apparatus of the second embodiment. In comparison with the flow chart of FIG. 3 , a first point that the correction degree calculation unit 400 calculates a correction degree (S 81 ), and a second point that the correction unit 200 performs conversion based on the correction degree (S 82 ) are different. Hereinafter, these different points will be explained.

The calculation unit 410 calculates a correction degree to suppress the correction amount from the whiteness degree of entire image. In general, a gloss region (a region reflected by a light) partially occurs on a surface of the object. Accordingly, when an image including an object having apparent of gloss is inputted, the number of pixels having high whiteness degree is reduced in comparison with the number of all pixels in the image. On the other hand, when many pixels have high whiteness degree, the case such as “the object is very strongly illuminated” or “a white object occupies in entire image” can be supposed. In such case, processing to emphasize gloss is often unnecessary.

In the second embodiment, by using an average of whiteness degree f 1 (x,y) of entire image, the calculation unit 401 calculates a correction degree c 1 to suppress the correction amount by an equation (16).

Here, w is the number of pixels along a lateral direction on the input image, and h is the number of pixels along a vertical direction on the image. Value range of c 1 is (0.0˜1.0). If all pixels have high whiteness degree, c 1 is near to 0.0.

When the average is calculated, a region not to be referred may be set onto the image. As shown in FIG. 8 , assume that respective regions having width Δw or width Δh are not referred. The correction degree c 1 is calculated by an equation (17).

A function to adjust a reference region is especially useful for broadcast wave. In case of the broadcast wave, a black frame (called “letter box” or “side panel”) to adjust aspect ratio is often displayed. In above-mentioned case, by adjusting the reference region, as to existence/non-existence of black frame, correction can be stably performed. As Δw and Δh, suitable values are previously calculated. Alternatively, these values may be estimated from the image.

Here, the case that sequential images are inputted is thought about. By using an image corresponding to time t, a correction degree to correct the correction amount of an image corresponding to time (t+1) (or (t+2) . . . ) is calculated. Similarly, in order to correct the correction amount of the image corresponding to time t, a memory to store information of one image is necessary. If such memory having large capacity is not utilized, a correction degree c 1 is calculated using an image corresponding to time (t−1) (in the past). Furthermore, a filter to smooth a response of c 1 along time direction may be used. In the second embodiment, in case of the sequential images, the correction degree corresponding to time t is calculated by an equation (18).

Here, c t-1 1 is a correction amount corresponding to time (t−1), and f t-1 1 (x,y) is a whiteness degree of an image corresponding to time (t−1). Furthermore, β is a parameter to adjust speed that changes a value of c 1 . If β is nearer to 0.0, the value of c 1 is changed more slowly. On the other hand, if β is equal to 1.0, an image corresponding to time (t−1) is only referred, and the value of c 1 is changed more rapidly.

The calculation unit 402 detects pixels having skin-color, and calculates a correction degree to suppress the correction amount. In case of the object such as a person's face, for example, feeling of oily skin is emphasized, i.e., the gloss emphasis effect is undesired as appearance. Accordingly, a function to suppress the gloss emphasis effect for skin-color region is useful practically. In the second embodiment, “probability that some pixel has skin-color” is defined as a skin-color probability f skin (x,y). A correction degree C 2 (x,y) to correct the correction amount of pixels having high skin-color probability is calculated.

In the second embodiment, signal values of input Cb and Cr are noticed. If Cb and Cr of general skin-color are in conformity with the normal distribution, a probability P skin (x,y) that some pixel has skin-color is calculated by an equation (19).

In the equation (19), Col(x,y) is defined as Col(x,y)=(Cb(x,y),Cr(x,y)) T . Furthermore, μ and ρ are an average vector and a variance-covariance matrix of Cb and Cr of skin-color respectively. Furthermore, Z is a normalized constant. By preparing a large number of image data including skin and by taking the statistics thereof, μ and Σ can be estimated. Furthermore, if knowledge related to parameters is already acquired, this knowledge may be reflected thereto. In this case, in the second embodiment, a skin-color probability f skin (x,y) is defined as an equation (20).

f skin ( x,y )=exp(−0.5·( Col ( x,y )−μ) T Σ −1 ( Col ( x,y )−μ))  (20)

In the equation (20), value range of f skin (x,y) is (0.0˜1.0). If f skin (x,y) is nearer to 1.0, a probability that the pixel has skin-color is higher.

From the above, a correction degree C 2 (x,y) is calculated by an equation (21).

c 2 ( x,y )=1.0 −f skin ( x,y )  (21)

In the second embodiment, as to distribution of skin-color, Cb and Cr are noticed. However, the distribution of skin-color is not limited thereto. For example, distribution including Y-value may be estimated, or another color-expression may be used. Furthermore, in the second embodiment, skin-color probability is calculated using Gaussian distribution. However, the skin-color probability may be calculated by a discriminator such as support-vector machine.

Based on the correction degree calculated, the synthesis unit 310 calculates a correction degree C(x,y) of each pixel (to be used by the correction amount calculation unit 300 ). In the second embodiment, the correction degree C(x,y) is calculated by a following equation.

C ( x,y )= C 1 ( x,y )× C 2 ( x,y )

Furthermore, as shown in an equation (22), a function to increase/degrease a response of each correction degree in a range between 0.0˜1.0 may be utilized.

C ( x,y )= g c1 ( c 1 )* g c2 ( c 2 ( x,y ))  (22)

In the second embodiment, by adding the correction degree C(x,y), the correction unit 200 calculates a luminance signal Y′(x,y) after emphasizing, and calculates a luminance signal Y′ out (x,y) after compensating gradation. Here, following equations are used.

Y ′( x,y )= Y ( x,y )+γ* F ( x,y )* C ( x,y )* Y max   (23)

Y out ( x,y )= C ( x,y )( a ·√{square root over ( Y ′( x,y ))}+ b )+(1.0 −C ( x,y ))· Y ( x,y )  (24)

Operation of the equation will be explained. For example, as to pixels having high skin-color probability, the correction degree C(x,y) is approximated to 0.0. Accordingly, if a correction amount is calculated based on the equation (23), effect to selectively weaken the correction amount of skin region is expected. Furthermore, the equation (24) is noticed. When the correction degree is near 1.0, a correction amount to be added to the luminance signal is large. In this case, the correction amount is compensated by rounding a gradation according to the equation (24). On the other hand, when the correction degree is near 0.0, a correction amount to be added to the luminance signal is small. Accordingly, by outputting a value near the input signal, unnecessary gradation-change is prevented.

According to the second embodiment, by using image characteristic (such as chroma, lightness, gradient intensity) simply computable, an image of which gloss is emphasized can be generated. Furthermore, by suppressing the correction amount based on whiteness degree or skin-color probability, final appearance of the image can be corrected.

While certain embodiments have been described, these embodiments have been presented by way of examples only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.