Method for performing image processing applications using quadratic programming

This invention relates generally to image processing, and more particularly to processing images using quadratic programming (QP).

As known in the art, a non-negative quadratic program (NNQP)

min x ⁢ 1 2 ⁢ x T ⁢ Qx - x T ⁢ h s . t . x ≥ 0 , ( 1 )

with vectors

x,h ε n ,

and a symmetric positive semi-definite matrix

Qε n×n ,

where T is a transpose operator is ubiquitous in image processing applications, where x is a vector of non-negative quantities, such as light intensities, to be estimated under a squared error norm.

For example, Eq. (1) subsumes a non-negative least squares (NNLS) problem encountered in deblurring and super-resolution applications,

min x ⁢  Ax - b  2 2 s . t . x ≥ 0 , ( 2 )

with

Q=A T A and h=A T b.

where A is a blurring or downsampling matrix, and b is a vector of pixel values. Eq. (1) is also the dual of a broader class of convex quadratic programs of the form

min y ⁢ 1 2 ⁢ y T ⁢ Hy - y T ⁢ f s . t . Ay ≥ b , ( 3 )

h are derived from vectors f, b are matrices A, and H via a dual transform.

Such quadratic programs are used in image labeling and image segmentation applications, and image operations that involve solving Poisson equation, e.g., matting. Solutions of Eq. (1) yield solutions of Eq. (3).

Eqs. (1-3) are well know in the prior art. State-of-the-art interior-point, active-set, and primal-dual algorithms offer linear-to-quadratic rates of convergence, but each iteration requires solution of a linear equation at least as large as Qy=z, which can take O(n 3 ) time for a vector y of n values. This is impractical for image processing, where the number n of pixels in a single image can be 10 7 or larger, Obviously this is an extremely complex operation and cannot simply be performed.

Multiplicative update methods are suitable for very large problems. They allow rough estimates of the solution vector to be improved without solving linear equations. Some well-known examples are the Richardson-Lucy deblurring algorithm, the image-space reconstruction algorithm, and Lee-Seung non-negative matrix factorization algorithm. These will turn out to be special cases of the invention presented here.

However, those applications rely on some combination of strictly nonnegative coefficients, positive definiteness, or favorable initialization for convergence, if convergence is provable at all.

Many image processing and computer vision applications can be solved as a non-negative quadratic program.

The embodiments of the invention provide a convergent multiplicative update that has a simple form and is amenable to fine-grained data parallelism.

Conventional methods for deblurring, matrix factorization, and tomography are covered as special cases. The method can be used for image super-resolution, labeling and segmentation applications.

Particularly, a method performs an image processing application by expressing the image processing application as a non-negative quadratic program (NNQP) with a quadratic objective, and nonnegativity constraints, and expressing a Karush-Kuhn-Tucker condition of the NNQP as a fixpoint ratio.

Then, the fixpoint ratio is determined iteratively until a solution is reached with a desired precision.

Specifically, a method performs an image processing application by expressing the image processing application as a non-negative quadratic program (NNQP) with a quadratic objective, and nonnegativity constraints. A Karush-Kuhn-Tucker condition of the NNQP is expressed as a fixpoint ratio. Then, the fixpoint ratio is determined iteratively until a solution to the image processing application is reached with a desired precision.

FIG. 1 is a flow diagram of a general method for performing an image processing application according to embodiments of the invention;

FIG. 2 is a flow diagram of a method for performing an image processing application to generate a high-resolution image from a set of low-resolution images according to embodiments of the invention;

FIG. 3 is a flow diagram of a method for deblurring an image according to embodiments of the invention.

FIG. 4 is a flow diagram of a general method for performing an image processing application according to embodiments of the invention;

We provide a multiplicative update method from Karush-Kuhn-Tucker (KKT) first-order optimality conditions of the Lagrangian function of Eq. (1)

L ( x , λ)=½ x T Qx−h−λx

with

λε n >0.

These are

x∘∇ x L=x ∘( Qx−h −λ)=0

λ∘∇ λ L=λ∘x= 0

with ∘ denoting the Hadamard element-wise product.

The update arises from a variational treatment of the above equation. We define the matrix split Q=Q + −Q− with Q + −max(Q,0)+diag(r) and Q + =max(−Q, 0)+diag(r) for some r. Similarly, h=h + −h−with h + =max(h, 0)+s, h − =max(−h, 0)+s.

Using the split, we can write

x∘∇ x L=x ∘(( Q + x+h − )−( Q − x+h + +λ))=0.

If all x i >0, possibly infinitesimally, then λ i= 0 and, element-wise,

x i ( Q + x+h − ) i=x i ( Q + x+h − ) i .

Fixpoint Ratio

Rearranging terms gives the multiplicative update rule

x i = x i ⁡ [ h i + + ( Q - ⁢ x ) i h i - + ( Q + ⁢ x ) i ] ,

which by construction is stationary at an optimum x*. The right side of the rule can be determined in parallel for all x by a matrix-vector multiply, and a vectorized divide.

Image Processing Applications

Image Super-Resolution

Image super-resolution (SR) refers an image processing application where a set of low-resolution images are combined into a single high-resolution image. Each low-resolution image g k is assumed to be generated from an ideal high-resolution image f via a displacement matrix S k , a decimation D k matrix, and a noise process n k :

gk=D k S k f+n k , for k= 1,2, . . . , K,

We use image registration to estimate the displacement matrices S k . Then, we reconstruct the high-resolution image by iteratively solving the NNQP

This NNQP is converted in the form of Eq (1) and then solved using the above fixpoint ratio.

Image Labeling

In Markov random fields (MRF)-based interactive image segmentation applications, a small subset of pixels is labeled, and the MRF propagates the labels across the image, typically finding high-gradient contours as segmentation boundaries where the labeling changes.

This problem is combinatorial when there are more than two kinds of labels, and usually approximately optimized by network flow process. The problem can also be expressed as a NNQP. Let r be a pixel in the image or the region of interest Ω={r i :i=0, 1, . . . , N}. The class set is denoted by K={1, 2, . . . , K}. Then, the probabilistic segmentation approaches compute a probability measure field x={x k (r):rεΩ,kΩK}, such that

Σ k=1 K x k ( r )=1, x k ( r )≧0

The set of neighbors pixel r is denoted as N r ={sεΩ:|r−s|=1}.

The QP has the form

min x ⁢ = ∑ k = 1 K ⁢ ∑ r ⁢ ( η 2 ⁢ ∑ j ∈ ?? ⁡ ( r ) ⁢ ω rj ⁡ ( x rk - x rk ) 2 + d rk ⁢ x rk ) ,

⁢ s . t . ∑ k = 1 K ⁢ x k ⁡ ( r ) = 1 , x k ⁡ ( r ) ≥ 0 , ∀ k ∈ K , ∀ r ∈ Ω ,

the cost d rk , of assigning label k to pixel r is obtained from a classifier that is trained with a small set of labeled, using e.g., a support vector machine (SVM), which is trainable using a multiplicative update rule described below.

The quadratic term controls the granularity of the regions, i.e., promotes smooth regions. The spatial smoothness is globally controlled by the positive parameter η, and locally controlled by weight ω rs ≈1, when the neighboring pixels r and s are likely to belong to the same class, and ω rs ≈0 otherwise. We set ω rs to the cosine of the angle between the vectors of two pixels in a LAB color space, which approximates the color response of human vision.

This QP is a convex relaxation of a combinatorial {0,1}-labeling problem. We solve this QP by putting it in dual form to obtain an NNQP of the form of Eq. (1), then iteratively applying the fixpoint to find the optimal dual vector. The dual vector is then transformed back into the primal vector x using the standard Lagrange dual transformation for quadratic programs. As an alternate solution method, one can derive multiplicative update rules directly from a Lagrangian function, as shown in the next paragraph.

Here the first-order KKT optimality conditions

η ⁢ ∑ j ∈ ?? ⁡ ( r ) ⁢ ω rj ⁡ ( x rk - x rk ) + d rk - λ r = 0 η ⁢ ⁢ x rk ( ∑ j ∈ ?? ⁡ ( r ) ⁢ ω rj ) - η ⁢ ∑ j ∈ ?? ⁢ ω rj ⁢ x jk + d rk - λ r = 0

yield a two-step update for primal and dual parameters

Depending on the size of the image and the number of classes, the MRF QP has 1-2 million variables and converges in 5-6 seconds on a serial processor.

Analysis

We consider here the behavior of the NNQP fixpoint ratio

x i = x i ⁡ [ h i + + ( Q - ⁢ x ) i h i - + ( Q + ⁢ x ) i ] ,

as a multiplicative update for positive vectors x≠x*.

The bracketed fixpoint ratio is always non-negative, and positive if s i >0 or r i x i >0). Thus, Eq. 1 preserves the feasibility of x, implicitly satisfying the second KKT condition.

As is the case for gradient descent methods, the fixpoint moves x i in the direction opposite the partial derivative ∂ xi L(x, λ).

Indeed, iteration of the rule from any x>0 solves Eq. (1) and, by extension, Eq. (2-3). More precisely, for any Q 0 (positive semi-definite), there is a vector r for which the fixpoint rule converges monotonically and asymptotically from any positive x>0 to the optimum of Eq. (1).

The value r guarantees that the update gives a contraction even in the nullspace of semidefinite Q. There are many satisfactory choices of r, for example,

r i =max(Σ j Q − ij ,Q ii ).

For many strictly positive or strictly convex image processing problems, it suffices to have r=0. Convergence is asymptotic; if x* i =0, we obtain x i→ 0 on an ideal infinite-precision computer). On real computers, which use floating point numbers, we see x i rapidly converge to x*, regardless of its value.

The asymptotic convergence rate is linear. Asymptotically, each iteration of the update rule improves the estimate {circumflex over (x)} by a constant number of bits of precision. For the sparse Q matrices typically encountered in image processing applications, each update takes linear serial time or constant parallel time, and a linear number of updates are needed to achieve any fixed precision result. Thus, the multiplicative update rule combines the simplicity and processing speed of gradient descent with the feasibility and convergence guarantees of modern quadratic programming algorithms.

As such, it is particularly well suited for implementation on low-cost parallel processing arrays, such as single instruction, multiple data (SIMD) processor or a graphic processing unit (GPU) in a cellular telephone, especially because Q typically has low bandwidth in image processing applications, so that the update requires very limited communication between processors. One GPU implementations solves a quadratic programs of 10 5 variables in a fraction of a second.

Image Processing Applications

General

As shown in FIG. 1 , our general method performs an image processing application 101 by expressing 110 the image processing application as a non-negative quadratic program (NNQP) 111 with a quadratic objective, and nonnegativity constraints. A Karush-Kuhn-Tucker condition of the NNQP is expressed as a fixpoint ratio 121 . Then, the fixpoint ratios is determined iteratively until a solution is reached with a desired precision, wherein the steps are performed in a processor.

Super-Resolution Image Processing Application

As shown in FIG. 2 , out method can be used to generate a high-resolution image. In this application, the input to the method is a set of low-resolution images 201 . It is desired to generate a non-negative high-resolution image with minimal squared difference to all input images 210 . In this case, the fixpoint ratio is the NNLS iterator 220 described above. Then, the fixpoint ratio is determined iteratively 230 until all variables, i.e., the high-resolution pixel values, are reached solution with desired precision.

Image Processing Application for Labeling

As shown in FIG. 3 , out method can also be used to label images. In this application, the input to the method is a partially labeled image 301 . It is desired to generate a non-negative labeled image with minimal squared difference between adjacent pixels and minimal label-to-data costs 310 . In this case, the fixpoint ratio is the NNLS iterator 320 for the QP dual, or the iterator derived directly from the KKT conditions. Then, the fixpoint ratio is determined iteratively 230 until all variables, i.e., the label values, stabilize to a desired precision.

Image Processing Application for Deblurring

As shown in FIG. 4 , out method can be used to deblur an image. In this application, the input to the method is a blurred image 401 . It is desired to generate a non-negative deblurred image with minimal squared difference between the deblurred image and the input image 410 . In this case, the fixpoint ratio is the NNLS iterator 420 . Then, the fixpoint ratio is determined iteratively 430 until all variables, i.e., pixel values, are reached solution with desired precision.

Effect of the Invention

By expressing the Karush-Kuhn-Tucker (KKT) first-order optimality conditions as the fixpoint ratio, we obtained a multiplicative fixpoint that can solve very large quadratic programs on low-cost parallel processors.

The update converges from all positive initial guesses. We also proved asymptotic linear convergence rate. Although QP problems are ubiquitous in image processing, QP methods are infrequently used because the QPs themselves are prohibitively large.

Our multiplicative update makes the QP approach viable, with one method solving many problems. Moreover, the method is well suited for single instruction, multiple data (SIMD), and multiple instruction, multiple data stream (MIMD) parallel processors.

Our method can be used for super-resolution, image segmentation, deblurring, matting, denoising, synthesis, and tomography applications.

Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.