Method and system for image classification

The application claims priority to U.S. Provisional Application Ser. No. 61/314,386 filed Mar. 16, 2010, the content of which is incorporated by reference.

The invention relates to method and system for image classification.

Image classification, including object recognition and scene classification, remains to be a major challenge to the computer vision community. Perhaps one of the most significant developments in the last decade is the application of local features to image classification, including the introduction of “bag-of-visual-words” representation.

One conventional approach applies probabilistic generative models with the objective towards understanding the semantic content of images. Typically those models extend topic models on bag-of-word representation by further considering the spatial information of visual words.

Certain existing approaches apply vector quantization (VQ) coding on local image descriptors, for example SIFT features or SURF features, and then average pooling to obtain the so-called “bag-of-visual-words” representation, which is fed into a nonlinear classifier based on SVMs using Chi-square or intersection kernel.

A further extension is to incorporate the spatial information of local descriptors in an image, by partition images into regions in different locations and scales and compute region-based histograms, instead of computing the global histogram for the entire image. These region-based histograms are concatenated to form the feature vector for the image. Then nonlinear SVM is applied for classification. This approach is called “spatial pyramid matching kernel” (SPMK) method. SPMK is regarded the state-of-the-art method for image classification.

It is known that SVMs use pyramid matching kernels, biologically-inspired models, and KNN methods. Over the past years, the nonlinear SVM method using spatial pyramid matching (SPM) kernels seems to be dominant among the top performers in various image classification benchmarks, including Caltech-101, PASCAL, and TRECVID. The recent improvements were often achieved by combining different types of local descriptors, without any fundamental change of the underlying classification method. In addition to the demand for more accurate classifiers, one has to develop more practical methods. Nonlinear SVMs scale at least quadratically to the size of training data, which makes it nontrivial to handle large-scale training data. It is thus necessary to design algorithms that are computationally more efficient.

In one aspect, methods and systems are disclosed for image classification coding an image by nonlinearly mapping an image descriptor to form a high-dimensional sparse vector; spatially pooling each local region to form an image-level feature vector using a probability kernel incorporating a similarity metric of local descriptors; and classifying the image.

In another aspect, a method for image classification includes nonlinearly mapping one or more descriptors of an image to form a high-dimensional sparse vector using Super-Vector nonlinear coding; spatial pooling each local region by aggregating codes of the descriptors in each local region to form a single vector, and concatenating vectors of different regions to form the image-level feature vector using probability kernel incorporating the similarity metric of local descriptors; and image classifying by normalizing image-level feature vector using linear SVMs.

In a related aspect, a system for image classification includes means for coding descriptor of an image by nonlinearly mapping to form a high-dimensional sparse vector using Super-Vector nonlinear coding method; means for spatial pooling each local region by aggregating the codes of all the descriptors in each local region to form a single vector, and concatenating vectors of different regions to form the image-level feature vector using probability kernel incorporating the similarity metric of local descriptors; and means for image classifying by normalizing image-level feature vector using linear SVMs.

In yet another aspect, a method for image classification includes extracting local image descriptors from a grid of locations in an image; nonlinearly coding extracted image descriptors to form a high-dimensional sparse vector; spatial pooling each image by partitioning into regions in different scales and locations, aggregating the codes of the descriptors in each region to form a single vector, and concatenating vectors of different regions to form the image-level feature vector; and linear classifying image-level feature vector.

In another related aspect, the system for image classification includes means for extracting local image descriptors from a grid of locations in an image; means for nonlinearly coding extracted image descriptors to form a high-dimensional sparse vector; means for spatial pooling each image by partitioning into regions in different scales and locations, aggregating the codes of all the descriptors in each region to form a single vector, means for concatenating vectors of different regions to form the image-level feature vector; and means for linear classifying image-level feature vector.

Advantages of the preferred embodiments may include one or more of the following. Image classification can be done using local visual descriptors. The system is more scalable in computation, transparent in classification, and greater accuracy than conventional systems. The overall image classification framework enjoys a linear training complexity, and also a great interpretability that is missing from conventional systems.

FIG. 1 is a flow chart showing image classification method.

FIG. 2 shows an exemplary system to perform image classification.

FIG. 1 is a flow chart showing image classification method. As shown in FIG. 1 , the method receives an input image in 110 . Next, the method performs a descriptor extraction in 120 . This operation extracts local image descriptors, such as SIFT, SURF, or any other local features, from a grid of locations in the image. As a result, the image is represented as a set of descriptor vectors with their 2D location coordinates. Next, the method performs nonlinear coding in 130 . Each descriptor of an image is nonlinearly mapped to form a high-dimensional sparse vector. The invention propose a novel nonlinear coding method called Super-Vector (SV) coding, which enjoys better theoretical properties than Vector Quantization (VQ) coding. Next, in 140 , the method performs spatial pooling where each image is partitioned into regions in different scales and locations. For each region, the codes of all the descriptors in it are aggregated to form a single vector, then vectors of different regions are concatenated to form the image-level feature vector. A probability kernel incorporating the similarity metric of local descriptors can be used in one embodiment as described in detail below. Next, the process performs linear classification in 150 . The image-level feature vector is normalized and fed into a classifier to detect an object such as a cat in 160 . Linear SVMs, which scale linearly to the size of training data, are used in the method. In contrast, the previous state-of-the-art systems used nonlinear SVMs which requires quadratic or higher-order computational complexity for training.

Next, one embodiment of the descriptor coding is described. In this embodiment, the coding method enjoys appealing theoretical properties. It is interested in learning a smooth nonlinear function ƒ(x) defined on a high dimensional space R d . The question is, how to derive a good coding scheme (or nonlinear mapping) φ(x) such that ƒ(x) can be well approximated by a linear function on it, namely w T φ(x). Assumption here is that ƒ(x) should be sufficiently smooth.

In a general unsupervised learning setting, where a set of bases C⊂R d , called codebook or dictionary, is employed to approximate any x, namely,

x ≈ ∑ v ∈ C ⁢ γ v ⁡ ( x ) ⁢ v ,

where γ(x)=[γ v (x)] vεC is the coefficients, and sometimes Σ v γ v (x)=1. By restricting the cardinality of nonzeros of γ(x) to be 1 and γ v (x)≧0, the Vector Quantization (VQ) method is obtained as:

v * ⁡ ( x ) = arg ⁢ ⁢ min v ⁢ ∈ C ⁢  x - v  ,

where P·P is the Euclidean norm (2-norm). The VQ method uses the coding γ v (x)=1 if v=v * (x) and γ v (x)=0 otherwise. ƒ(x) is β Lipschitz derivative smooth if for all x,x′εR d :

 f ⁡ ( x ) - f ⁡ ( x ′ ) - ∇ f ⁡ ( x ′ ) T ⁢ ( x - x ′ )  ≤ β 2 ⁢  x - x ′  2 .

It immediately implies the following simple function approximation bound via VQ coding: for all xεR d :

This bounds simply states that one can approximate ƒ(x) by ƒ(v * (x))+∇ƒ(v * (x)) T (x−v * (x)), and the approximation error is upper bounded by the quality of VQ. It further suggests that the function approximation can be improved by learning the codebook C to minimize this upper bound. One way is the K-means algorithm

C = arg ⁢ ⁢ min C ⁢ { ∑ x ⁢ min v ∈ C ⁢  x - v  2 } .

Eq. (1) also suggests that the approximation to ƒ(x) can be expressed as a linear function on a nonlinear coding scheme

ƒ( x )≈ g ( x )≡ w T φ( x ),

where φ(x) is called the Super-Vector (SV) coding of x, defined by

φ( x )=[ sγ v ( x ),γ v ( x )( x−v ) T ] vεC T   (2)

where s is a nonnegative constant. It is not difficult to see that

w = [ 1 s ⁢ f ⁡ ( v ) , ∇ f ⁡ ( v ) ] v ∈ C ,

which can be regarded as unknown parameters to be estimated. Because γ v (x)=1 if v=v * (x), otherwise γ v (x)=0, the obtained φ(x) a is highly sparse representation, with dimensionality |C|(d+1). For example, if |C|=3 and γ(x)=[0,1,0], then

w T φ(x) provides a piece-wise linear function to approximate a nonlinear function ƒ(x), while with VQ coding φ(x)=[γ v (x)] vεC T , the same formulation w T φ(x) gives a piece-wise constant approximation. SV coding may achieve a lower function approximation error than VQ coding. It should be noted that the popular bag-of-features image classification method essentially employs VQ to obtain histogram representations. The proposed SV coding is a simple extension of VQ, and may lead to a better approach to image classification.

Next, one embodiment of spatial pooling is discussed. Each image can be represented as a set of descriptor vectors x that follows an image-specific distribution, represented as a probability density function p(x) with respect to an image independent back-ground measure dμ(x). Let's first ignore the spacial locations of x, and address the spacial pooling later. A kernel-based method for image classification is based on a kernel on the probability distributions over xεΩ, K:P×P R. A well-known example is the Bhattacharyya kernel:

Here p(x) and q(x) represent two images as distributions over local descriptor vectors, and μ(x) is the image independent background measure. Bhattacharyya kernel is closely associated with Hellinger distance, defined as D h (p,q)=2−K b (p,q), which can be seen as a principled symmetric approximation of the Kullback Leibler (KL) divergence. Despite the popular application of both Bhattacharyya kernel and KL divergence, a significant drawback is the ignorance of the underlying similarity metric of x. In order to avoid this problem, one has to work with very smooth distribution families that are inconvenient to work with in practice. This invention propose a novel formulation that explicitly takes the similarity of x into account:

K s ⁡ ( p , q ) = ⁢ ∫ Ω ⁢ ∫ Ω ⁢ p ⁡ ( x ) 1 2 ⁢ q ⁡ ( x ′ ) 1 2 ⁢ κ ⁡ ( x , x ′ ) ⁢ ⅆ μ ⁡ ( x ) ⁢ ⅆ μ ⁡ ( x ′ ) = ⁢ ∫ Ω ⁢ ∫ Ω ⁢ p ⁡ ( x ) - 1 2 ⁢ q ⁡ ( x ′ ) - 1 2 ⁢ κ ⁡ ( x , x ′ ) ⁢ p ⁡ ( x ) ⁢ q ⁡ ( x ′ ) ⁢ ⅆ μ ⁡ ( x ) ⁢ ⅆ μ ⁡ ( x ′ )

where K(x,x′) is a RKHS kernel on Q that reflects the similarity structure of x. In the extreme case where K(x,x′)=δ(x−x′) is the delta-function with respect to μ(•), then the above kernel reduces to the Bhattacharyya kernel.

The system cannot directly observe p(x) from any image, but a set X of local descriptors. Therefore, based on the empirical approximation to K s (p,q), a kernel between sets of vectors is defined as:

K ( X , X ′ ⁢ ) = 1 NN ′ ⁢ ∑ x ∈ X ⁢ ∑ x ′ ∈ X ′ ⁢ ⁢ p ⁡ ( x ) - 1 2 ⁢ q ⁡ ( x ′ ) - 1 2 ⁢ κ ⁡ ( x , x ′ ) ( 4 )

where N and N′ are the sizes of the descriptor sets from two images.

If κ(x,x′)=(φ(x),φ(x′)), where φ(x) is the SV coding defined in the previous section. It is easy to see that κ(x,x′)=0 if x and x′ fall into different clusters. Then Eq. (4) is presented as follows:

K ⁡ ( X , X ′ ) = 1 NN ′ ⁢ ∑ k = 1  C  ⁢ ∑ x ∈ X k ⁢ ∑ x ′ ∈ X k ′ ⁢ p ⁡ ( x ) - 1 2 ⁢ q ⁡ ( x ′ ) - 1 2 ⁢ κ ⁡ ( x , x ′ )

where X k is the subset of X fallen into the k-th cluster. Furthermore, assume that p(x) remains constant within each cluster partition, i.e., p(x) gives rise to a histogram [p k ] k=1 |C| , then

The above kernel can be re-written as an inner product kernel of the form K(X,X′)=(Φ(X),Φ(X′)), where

Therefore functions in the reproducing kernel Hilbert space for this kernel has a linear representation ƒ(X)=w T Φ(X). In other words, Φ(X) can be used simply as nonlinear feature vector and then a linear classifier is learned using this feature vector. The effect is equivalent to using nonlinear kernel K(X,X′) between image pairs X and X′.

Finally, weighting by histogram p k is equivalent to treating density p(x) as piece-wise constant around each VQ basis, under a specific choice of background measure μ(x) that equalizes different partitions. This representation is not sensitive to the choice of background measure μ(x), which is image independent. In particular, a change of measure μ(•) (still piece-wise constant in each partition) leads to a rescaling of different components in Φ(X). This means that the space of linear classifier ƒ(x)=w T Φ(X) remains the same.

To incorporate the spatial location information of x, the idea of spatial pyramid matching is applied. Let each image be evenly partitioned into 1×1, 2×2, and 3×1 blocks, respectively in 3 different levels. Based on which block each descriptor comes from, the whole set X of an image is then organized into three levels of subsets: X 11 1 , X 11 2 , X 12 2 , X 21 2 , X 22 2 , X 11 3 , X 12 3 , and X 13 3 . Then the pooling operation introduced in the last subsection can be applied to each of the subsets. An image's spatial pyramid representation is then obtained by concatenating the results of local pooling

φ s ( X )=[Φ( X 11 1 ),Φ( X 11 2 ),Φ( X 12 2 ),Φ( X 21 2 ),Φ( X 22 2 ),Φ( X 11 3 ),Φ( X 12 3 ),Φ( X 13 3 )]

Next, one embodiment of image classification is described. Image classification is done by applying classifiers based on the image representations obtained from the pooling step. It is required to find whether a particular category of objects is contained in an image or not, which can be translated into a binary classification problem. This is performed by applying a linear SVM that employs a hinge loss to learn g(X)=w T Φ s (X). It should be noted that the function is nonlinear on X since Φ s (X) is a nonlinear operator.

The image-level classification function is closely connected to a real-valued function on local descriptors. Without loss of generality, let's assume that only global pooling is used, which means Φ s (X)=Φ(X) in this case.

g ⁡ ( X ) = w T ⁢ Φ ⁡ ( X ) = 1 N ⁢ ∑ k = 1  C  ⁢ 1 p k ⁢ ∑ x ∈ X k ⁢ w T ⁢ ϕ ⁡ ( x ) = 1 N ⁢ ∑ k = 1  C  ⁢ 1 p k ⁢ ∑ x ∈ X k ⁢ g ⁡ ( x ) ( 5 )

where g(x)=w T Φ(x). The above equation provides an interesting insight to the classification process: a patch-level pattern matching is operated everywhere in the image, and the responses are then aggregated together to generate the score indicating how likely a particular category of objects is present. This observation is well-aligned with the biologically-inspired vision models, like Convolution Neural Networks and HMAX model, which mostly employ feed-forward pattern matching for object recognition.

This connection stresses the importance of learning a good coding scheme on local descriptors x, because Φ(x) solely defines the function space of g(x)=w T Φ(x), which consequently determines if the unknown classification function can be well learned. The connection also implies that supervised training of Φ(x) could potentially lead to further improvements.

Furthermore, the classification model enjoys the advantages of interpretability and computational scalability. Once the model is trained, Eq. (5) suggests that one can compute a response map based on g(x), which visualizes where the classifier focuses on in the image. Since the proposed method naturally requires a linear classifier, it enjoys a training scalability which is linear to the number of training images, while nonlinear kernel-based methods suffer quadratic or higher complexity.

The classification model is more related to local coordinate coding (LCC), which points out that in some cases a desired sparsity of φ(x) should come from a locality of the coding scheme. Indeed, the proposed SV coding leads to a highly sparse representation φ(x), as defined by Eq. (2), which activates those coordinates associated to the neighborhood of x. As the result, g(x)=w T φ(x) gives rise to a local linear function (i.e., piece-wise linear) to approximate the unknown nonlinear function ƒ(x). But, the computation of SV coding is much simpler than sparse coding approaches.

The method can be further improved by considering a soft assignment of x to bases C. The underlying interpretation of ƒ(x)≈w T φ(x) is the approximation

ƒ( x )≈ƒ( v * ( x ))+∇ƒ( v * ( x )) T ( x−v * ( x ))

which essentially uses the unknown function's Taylor expansion at a nearby location v * (x) to interpolate ƒ(x). One natural idea to improve this is using several neighbors in C instead of the nearest one. Let's consider a soft K-means that computes p k (x), the posterior probability of cluster assignment for x. Then the function approximation can be handled as the expectation

f ⁡ ( x ) ≈ ∑ k = 1  C  ⁢ p k ⁡ ( x ) ⁡ [ f ⁡ ( v k ) + ∇ f ⁡ ( v k ) T ⁢ ( x - v k ) ]

Then the pooling step becomes a computation of the expectation

Φ ⁡ ( X ) = 1 N [ 1 p k ⁢ ∑ x ∈ X ⁢ p k ⁡ ( x ) ⁢ ( x - v k + s ) ] k = 1  C 

where

p k = 1 N ⁢ ∑ x ∈ X ⁢ p k ⁡ ( x ) ,

and s comes from Eq. (2). This approach is different from the image classification using GMM. Basically, those GMM methods consider the distribution kernel, while the inventive method incorporates nonlinear coding into the distribution kernel. Furthermore, the model according to the invention requires the stickiness to VQ—the soft version requires all the components share the same isotropic diagonal covariance. That means a much less number of parameters to estimate, and therefore a significantly higher accuracy can be obtained.

As suggested by Eq. 5, a very unique perspective of this method is the “transparency” of the classification model. Once the image classifier is trained, a real-valued function g(x) is automatically obtained on the local descriptor level. Therefore a response map of g(x) can be visualized on test images.

The invention may be implemented in hardware, firmware or software, or a combination of the three. Preferably the invention is implemented in a computer program executed on a programmable computer having a processor, a data storage system, volatile and non-volatile memory and/or storage elements, at least one input device and at least one output device.

By way of example, a block diagram of a computer to support the system is shown in FIG. 2 . The computer preferably includes a processor, random access memory (RAM), a program memory (preferably a writable read-only memory (ROM) such as a flash ROM) and an input/output (I/O) controller coupled by a CPU bus. The computer may optionally include a hard drive controller which is coupled to a hard disk and CPU bus. Hard disk may be used for storing application programs, such as the present invention, and data. Alternatively, application programs may be stored in RAM or ROM. I/O controller is coupled by means of an I/O bus to an I/O interface. I/O interface receives and transmits data in analog or digital form over communication links such as a serial link, local area network, wireless link, and parallel link. Optionally, a display, a keyboard and a pointing device (mouse) may also be connected to I/O bus. Alternatively, separate connections (separate buses) may be used for I/O interface, display, keyboard and pointing device. Programmable processing system may be preprogrammed or it may be programmed (and reprogrammed) by downloading a program from another source (e.g., a floppy disk, CD-ROM, or another computer).

The system of FIG. 2 receives images to be classified. Each image is represented by a set of local descriptors with their spatial coordinates. The descriptor can be SIFT, or any other local features, computed from image patches at locations on a 2D grid. In one embodiment, the images is processed by a descriptor coding module where each descriptor of an image is nonlinearly mapped to form a high-dimensional sparse vector. A nonlinear coding method called vector machine coding can be used, which is an extension of Vector Quantization (VQ) coding. Next, the descriptor are provided to a spatial pooling module. For each local region, the codes of all the descriptors in it are aggregated to form a single vector, then vectors of different regions are concatenated to form the image-level feature vector. This step is based on a novel probability kernel incorporating the similarity metric of local descriptors. The image-level feature vector is normalized and fed into a classifier. Linear SVMs, which scale linearly to the size of training data, is used in this step.

Each computer program is tangibly stored in a machine-readable storage media or device (e.g., program memory or magnetic disk) readable by a general or special purpose programmable computer, for configuring and controlling operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

The invention has been described herein in considerable detail in order to comply with the patent Statutes and to provide those skilled in the art with the information needed to apply the novel principles and to construct and use such specialized components as are required. However, it is to be understood that the invention can be carried out by specifically different equipment and devices, and that various modifications, both as to the equipment details and operating procedures, can be accomplished without departing from the scope of the invention itself.