Detection and segmentation of sweeps in color graphics images

The present invention relates to color graphics image processing. It finds particular application in conjunction with detection and segmentation of sweeps in color graphics images, and will be described with particular reference thereto. However, it is to be appreciated that the present invention is also amenable to other like applications.

Content-based image classification has emerged as an important area in multimedia computing due to the rapid development of digital imaging, storage, and networking technologies. A reproduction system, such as a copier or a printer, strives for a pleasing rendering of color documents. Picture/graphics classifiers have been developed to differentiate between a picture image and a graphics image with high accuracy by analyzing low-level image statistics.

For example, U.S. Pat. No. 5,767,978 to Revankar et al. discloses an adaptable image segmentation system for differentially rendering black and white and/or color images using a plurality of imaging techniques. An image is segmented according to classes of regions that may be rendered according to the same imaging techniques. Image regions may be rendered according to a three-class system (such as traditional text, graphic, and picture systems), or according to more than three image classes. In addition, only two image classes may be required to render high quality draft or final output images. The image characteristics that may be rendered differently from class to class may include half toning, colorization and other image attributes.

A color output device such as a CRT computer monitor, liquid crystal display, inkjet printer, xerographic printer, etc can display a limited range of colors (the gamut of the output device). If the colors in an image do not reside wholly with in an output device gamut, a gamut-mapping algorithm is often applied to map the image colors to colors that the output device can produce. A simple approach is to preserve in-gamut colors without alteration and clip out-of-gamut colors to the closest in-gamut color. More sophisticated techniques can be used. Ideally, the gamut-mapping algorithm used should be tailored to the image type. For example, a region of smoothly varying colors should appear smoothly varying on the output device. Were the colors of a sweep to exceed the gamut of an output device, the aforementioned clipping approach will show disagreeable artifacts. In fact, it may be desirable to sacrifice color fidelity within the gamut to achieve a smooth color transition. Thus knowing that a region is, or contains, a sweep aids in color reproduction. In general, coloring schemes (gamut-mapping algorithms) are tailored for specific types of images to obtain quality reproduction. Once an image has been identified as a graphics image, further identification of image characteristics can be used to fine-tune the coloring schemes for more appealing reproductions. The most prominent characteristics of a graphics image include patches or areas of the image with uniform color and areas with uniformly changing colors. This invention focuses on the identification of the second characteristic.

One example where areas with uniformly changing color can usually be observed is in the gradient backgrounds of color business presentation slides. These areas of uniformly changing color are called sweeps and are constructed in the three-dimensional color space as a line during the construction of the synthetic graphics. A sweep is constructed by a mathematical formula to cause adjacent pixels to change color in a smooth, predictable way. For example, one can use linear interpolation of two colors specified for the sweep and render the original image by plotting pixels of interpolated colors such that neighboring spatial regions are rendered with colors from neighboring color regions. One can contemplate other mathematical descriptions of curves that achieve like effects. If such a document is printed or scanned, the sweeps do not exactly contain the colors on the line due to halftone noise introduced. If a reproduction system can correctly identify and segment the sweep areas in an image, the original sweeps can be reconstructed in the color space and rendered. The sweeps thus rendered will be very smooth and the noise introduced by the halftone will not be reproduced. Secondly, if the extreme colors of the sweep can be automatically identified, the coloring schemes can be tailored to maximize the smoothness as well as contrast and differentiation among colors to render business graphics documents.

Further identification of the properties of the graphics image can be used to fine-tune the coloring scheme to obtain a more appealing reproduction. The detection of sweeps in a graphics image can be used to reconstruct synthetic sweeps that may otherwise be perturbed due to half toning, scanning artifacts, or aging of a document or for other reasons. The extent of the sweeps (i.e., the change from color 1 to color 2 ) may also be used to tailor the coloring scheme to achieve best smoothness, contrast and differentiation among colors in the reconstructed sweeps.

The present invention proposes a new and improved method for detecting and segmenting sweeps in a color graphics image that overcomes the above-referenced problems and others.

In accordance with one aspect of the present invention, sweeps in a graphics image are detected and segmented. The input image is transformed into an appropriate color space (e.g., CIELUV) and sweep segment information from one or more color channel histograms of the image is detected. Then the graphics image is segmented into sweep and non-sweep areas using the sweep segment information.

Still further advantages and benefits of the present invention will become apparent to those of ordinary skill in the art upon reading and understanding the following detailed description of the preferred embodiments.

The invention may take form in various components and arrangements of components, and in various steps and arrangements of steps. The drawings are only for purposes of illustrating preferred embodiments and are not to be construed as limiting the invention.

FIG. 1 is a flowchart of the process to detect and segment sweeps in a graphics image in accordance with the present invention;

FIG. 2 shows an embodiment for sub-process 106 of FIG. 1 ;

FIG. 3 shows an embodiment for sub-process 108 of FIG. 1 ;

FIG. 4 shows an embodiment for sub-process 110 of FIG. 1 ;

FIG. 5 shows an embodiment for sub-process 112 of FIG. 1 ; and

FIG. 6 shows an embodiment for optional sub-process 114 of FIG. 1 .

With reference to FIG. 1 , a flowchart of the process to detect and segment sweeps in a graphics image 100 is provided. The process 100 begins with an input graphics image 102 . Next, the input graphics image is transformed into a three-dimensional histogram in an appropriate color space 104 . In the preferred embodiment, the CIELUV color space is used. This space is chosen because additive mixtures of two colors 1 and 2 lie along a locus that is close to a straight line in three-dimensional CIELUV space. It also provides rough perceptual uniformity, i.e., the Euclidean distance between a pair of colors specified in CIELUV roughly corresponds to perceived color difference. Other color spaces with similar properties such as CIEL*a*b* are contemplated in this invention. Color sweeps appear as lines in the three-dimensional CIELUV color histogram. However, since further processing (e.g., edge detection, line detection, etc.) in the three-dimensional space is difficult and time consuming, the present invention operates on two-dimensional projections of the three-dimensional histogram. Lines project as either lines or points in any two-dimensional projection from a three-dimensional space. Continuing with the flowchart of FIG. 1 , two-dimensional histograms are estimated from the three-dimensional histogram 106 . Next, the two-dimensional histograms are processed to detect and segment sweeps 108 . Sweep segment information from the processing of the two-dimensional histograms is combined 110 . The combined sweep segment information is used to process the input graphics image to identify and segment sweeps 112 . Post-processing may be optionally and selectively used to reject false alarms (i.e., areas falsely identified as sweeps) 114 . The process ends 116 after post-processing 114 or, if post-processing is not used, after the input graphics image is processed 112 .

FIG. 1 depicts an embodiment of the method of the present invention, comprising: 1) converting an input graphics image to a color space 104 ; 2) projecting the image to a number of planes within the color space 106 ; 3) detecting curves in each plane and linking overlapping curves 108 ; 4) associating pixels with colors that project to the detected curves in each plane with evidence that the color belongs to a sweep 108 ; and 5) for each pixel color, combining sweep evidence to determine whether pixels of that color are part of a sweep 110 , 112 . A preferred embodiment of the method of FIG. 1 uses the CIELUV color space 104 , projects the image to three orthogonal planes 106 , detects curves using a Hough transform and edge linking 108 , identifies whether pixels are associated with colors of the detected curves with a logical label (e.g., TRUE/FALSE) 108 , and uses a logical AND operation to combine sweep evidence 110 , 112 .

With reference to FIG. 2 , a preferred embodiment for estimating two-dimensional histograms from the three-dimensional histogram 106 is provided. In the preferred embodiment, the first step of the sub process estimates the two-dimensional histograms (size: GL×GL, GL is typically 256) from the input image in UV, LU, and LV projections, respectively 206 . Next, the histograms are normalized by a scaling scheme to create H_UV, H_LU, and H_LV histograms, in UV, LU, and LV projections respectively 208 . For example, in a scaling scheme the bin values are integers between 0 and GL-1 (typically 255). The normalized histograms are considered as gray-scale images for further processing.

With reference to FIG. 3 , the preferred embodiment for processing the two-dimensional histograms to detect sweep segments 108 is provided. In the preferred embodiment, the first step of the sub process segments the H_UV, H_LU, and H_LV histograms by detecting edges and creating an edge map 308 for each projection. A standard edge detector, such as the commonly known Canny edge detector (sigma=5, mask size=51, lower threshold=0.4, higher threshold=0.9), can be used for detecting edges in the H_UV, H_LU, and H_LV histograms. The parameters identified for the edge detector were determined empirically. By using a larger mask size, thick edges are detected but small edges are missed. By using a smaller mask size, thin edges are detected but thick edges are missed. In experiments, the mask size of 51 obtained the desired balance between detecting thin and thick edges that correspond to sweeps in histogram images. Next, a connectivity analysis of the edges is performed on each of the H_UV, H_LU, and H_LV edge maps 310 . For example, a standard 8-connected component algorithm can be used to perform the connectivity analysis and ignore very small edges (e.g., less than 30 pixels). Note that each sweep appears as a line segment in the histogram images and is detected as two parallel edges in the edge map. The edge maps are binary images with white pixels representing edges and black pixels representing non-edges.

With continuing reference to FIG. 3 , in order to estimate the location and orientation of the sweep line segments, the detected edges in the H_UV, H_LU, and H_LV edge maps are converted to points in a Hough parametric space 312 . For example, a standard Hough Transform algorithm can be used to convert the edges in the spatial domain to the Hough parametric space. Points in the Hough space with a large number of votes (i.e., more than a threshold T) are selected. Experimentally, a satisfactory threshold T can be empirically determined so that most of the lines longer than 20 pixels are detected. The selected points represent straight lines in the H_UV, H_LU, and H_LV histograms. The lines detected in the Hough space are in parametric form and run to infinity at both ends. However, identification of end points of sweeps is important.

For this purpose, each detected line in the Hough space is rendered (e.g., drawn as a 3-pixel wide line) on the edge map using a standard scan-line-drawing algorithm 314 . Typically, the rendering is performed using a particular gray value (e.g., 100) and the overlap between the edges and the line drawn are marked as “overlap.” The extremities of the pixels marked as “overlap” are also noted. These extremities define the line segments in the two-dimensional histogram images. As mentioned before, each sweep in the image appears as parallel line segments in the edge map. Hence, the pairs of parallel line segments in the edge maps are identified 316 and are considered for further processing while other segments are ignored. Each pair of parallel line segments correspond to a single sweep in the original image. Next, the mid segment of each pair of parallel line segments is computed 318 and recognized as a sweep.

The sweeps detected as segments in the H_UV, H_LU, and H_LV edge maps are projections of the original sweeps from the three-dimensional color space. Segment information consists of a data structure that indicates for each pair (u, v), (l, u) and (l, v) whether or not the pair corresponds to a sweep. To segment the input graphics image into sweep regions and non-sweep regions, the sweep segment information from the three projections must be combined. Referring to FIG. 4 , a preferred embodiment for combining sweep segment information from the two-dimensional histograms 110 is provided. In this embodiment, the sweep segment information from the UV, LU, and LV projections are combined 410 . A restrictive combination scheme performs an “and” operation on the information from the three projections. Specifically, the “and” condition states that a pixel with color (l, u, v) is in a sweep if and only if, from the three projections, (u, v) is a sweep AND (l, u) is a sweep AND (l, v) is a SWEEP. Alternatively, a liberal scheme performs an “or” operation on the information from the three projections: a pixel with color (l, u, v) is in a sweep if and only if, from the three projections, (u, v) is a sweep OR (l, u) is a sweep OR (l, v) is a SWEEP. The choice between restrictive and liberal combination schemes depends on the application. This invention contemplates other logical or functional combinations that may give advantage for other applications.

Once the sweep segments in each of the H_UV, H_LU, and H_LV edge maps are identified and noted and the UV, LU, and LV projections are combined, the original graphics image is revisited. Referring to FIG. 5 , a preferred embodiment for processing the input graphic image to identify sweep segments 112 is provided. In this embodiment, each pixel in the input graphics image is labeled either “sweep” or “non-sweep.” The sub process begins with selection of a first pixel 512 . Next, the distance between the sweep segments and the pixel is computed 514 and compared to a particular threshold value 516 . If the distance is less than the threshold, the pixel is labeled “sweep” 518 . Otherwise, the pixel is labeled “non-sweep” 520 . Finally, the sub process determines if it has reached the last pixel 522 . Until the last pixel is reached, the sub process is repeated for the next pixel. When the last pixel is reached, the sub process is complete.

For synthetic graphics this scheme for detecting and segmenting sweeps works well. However, for scanned graphics images, the result from this segmentation scheme may have an unacceptable number of errors. Often, for a liberal scheme, there are far more false alarms than false misses. A post-processing stage may be used to reject several types of false alarms. Referring to FIG. 6 , several potential components of optional post-processing 114 are provided. The optional components may be implemented individually or in any combination. First, the post-processing scheme may use a digital filter to check for connectivity and reject small isolated areas of sweeps and non-sweeps 614 . For example, a median digital filter was used in experiments. Second, the post-processing scheme may compute the gradient information in the image and reject those areas where the gradient in the image is less than a threshold 616 . Third, a more sophisticated type of gradient post-processing could check for consistency of the gradient at several scales 618 . Finally, the post-processing scheme may ignore gray sweeps by rejecting the detected horizontal lines in the H_LU and H_LV edge maps 620 .

Although the Hough transform is used to detect straight line segments in this embodiment, known variants of the Hough methodology can be used to detect parameterized curves, surfaces, and shapes. Other color spaces and sweeps may produce other curves or surfaces in the three projections and these can be detected with methods known in the art and are within the scope of this invention.

The invention has been described with reference to the preferred embodiments. Obviously, modifications and alterations will occur to others upon reading and understanding the preceding detailed description. It is intended that the invention be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.