Method and apparatus for tracking objects across images

This application claims benefit of U.S. provisional patent application Ser. No. 61/267,578, filed Dec. 8, 2009, which is herein incorporated by reference.

1. Field of the Invention

Embodiments of the present invention generally relate to a method and apparatus for tracking objects across images.

2. Description of the Related Art

Object tracking is a fundamental requirement in many computer vision based applications: it enables the application to assign unique identifiers (or labels) to all detected objects in the scene, and maintains the mapping between objects and identifiers over time as the objects move and interact in the scene. The challenge is to be able to preserve the correct object identifiers in spite of changing illumination in the scene, changes in appearance of the object, occlusions, interactions between objects, etc.

Embodiments of the present invention relate to a method and apparatus for tracking objects across images. The method includes retrieving object location in a current frame, determining the appearance and motion signatures of the object in the current frame, predicting the new location of the object based on object dynamics, searching for a location with similar appearance and motion signatures in a next frame, and utilizing the location with similar appearance and motion signatures to determine the final location of the object in the next frame.

So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.

FIG. 1 is an embodiment depicting an overview of tracking algorithm; and

FIG. 2 is an embodiment depicting feature computation. (a) Detected bounding boxes around object, (b) Motion feature computation, (c) Appearance feature computation showing center-surround kernel weights.

Tracking enables the application to make intelligent decisions. For example, in video surveillance, object tracking enables an application to detect when a person crosses a pre-defined zone, identify if a person has been following a ‘suspicious’ path, etc. In human-device interfaces, an application can employ the object tracker to track the user's hands, and determine what gestures are being performed by the user.

In one embodiment, the object detection module determines the location and size of all objects in the scene. The object tracker is then responsible for assigning and maintaining unique identities for each object as they move in the scene.

An overview of an embodiment of the proposed tracking algorithm is provided in FIG. 1 . FIG. 1 is an embodiment depicting an overview of tracking algorithm.

In this embodiment, the approach uses the Nelder-Mead Simplex to search for the best matching image location in the next frame corresponding to the known object size and location in the current frame. The features that are matched during the simplex search are derived from a combination of motion and appearance histograms. The metric used in this embodiment to quantify the match is the Bhattacharya distance. The method may employ a Kalman filter to initialize the start location of the simplex search in a new frame and/or predict step. The Kalman filter may also be employed to smooth out the estimated location and/or correct step.

Details of the specific blocks of the algorithm are described next, highlighting the advantages or novelty of each step. In one embodiment, for each object O i in frame I t , we extract motion and appearance histograms. Object O i is described as follows O i =[x, y, w, h], where x,y are image coordinates of the center, and w,h are the dimensions of the bounding box that best represents the scale of the object.

FIG. 2 is an embodiment depicting feature computation. (a) Detected bounding boxes around object, (b) Motion feature computation, (c) Appearance feature computation showing center-surround kernel weights. For both the appearance and motion features, we compute histograms in non-overlapping regions and then combine them. In the case of the motion feature, this captures the spatial distribution of the motion pixels, providing a motion signature for each object. Similarly, for the appearance feature, dividing the image region enables the tracker to have a better sense of the spatial layout of the object.

For the motion histogram, we first compute the thresholded image difference I D

I Diff = I t - I t - 1 I D i = { I Diff i  I Diff i  ≥ T 0  I Diff i  < T

where T is a threshold that can be set by the application based on the required sensitivity.

The bounding box for O i is then divided into four equal quadrants as shown in FIG. 2( c ) and the histograms is computed in each of the four quadrants. For a typical 8-bit image, the range of the pixel values in I D is bounded in the range [−255, 255]. Additionally, taking into consideration threshold T, the range may be further divided as [−255,−T] U [T, 255]. This information can be useful in determining the bin edges of the histograms. Denoting the histograms in each of the quadrants as h1, h2, h3, h4, the final motion histogram feature is defined as H m =[h1, h2, h3, h4], where [ ] performs a simple concatenation operation.

For the appearance histogram, H a , we proceed in a similar fashion using the current frame I t . We divide the image region corresponding to O i in I t into 4 quadrants, as shown in FIG. 2 ( b ). We also apply a kernel weighted approach that assigns a higher weight to the center of the object bounding box, and lesser weight to the surrounding background regions. As with the motion histogram, we combine the individual appearance histograms to arrive at the appearance feature H a =[h1, h2, h3, h4].

During the Simplex search, we may employ the Bhattacharya distance to measure the similarity of the motion and appearance features. We then combine the motion and appearance Bhattacharya distance into a single score using a weighted average.

Using both appearance and motion features makes our tracking algorithm robust to a variety of conditions that typically impede the performance of other methods. The motion features are relatively invariant to changes in ambient illumination, and conversely, the appearance features are invariant to the dynamics of the object. Thus this combination provides a powerful complementary feature set for tracking.

The search algorithm used is the Nelder-Mead Simplex, often used in optimization problems for function minimization. We found it ideally suited for real-time tracking in embedded platforms. For object O i =[x,y,w,h], we employed the Simplex search to minimize following function

F ( c,r,s )= K ( B ( H a t+1 ( c,r,s·w,s·h ), H a t ( x,y,w,h ), B ( H m t+1 ( c,r,s·w,s·h ), H m t ( x,y,w,h ))

where, H a and H m are appearance and motion feature vectors, and the superscripts t and t+1 denote current frame and the next frame. B(·) denotes the Bhattacharya distance between two features, and K(·) represents a function combining the Bhattacharya distance scores. The symbols c, r, and s stand for the 2D image coordinates and the scale of the rectangular image region on which the features are computed.

In one embodiment, the solution employs Nelder-Mead Simplex to search for object location in next frame. The NM Simplex is well suited for efficient tracking in embedded platforms. It is a classic optimization method used often to search over a solution space in function minimization problems. Typically, object tracking algorithms utilize search mechanisms such as mean-shift and trust regions.

While these methods might be more optimal than simplex from a strictly function minimization perspective, the simplex method has characteristics that make it attractive for fast and efficient tracking, especially in resource-bound embedded platforms. The simplex method does not require a closed-form expression of the function, nor does it require calculating of the slope (derivative) of the function. Since derivatives need not be computed, the simplex method typically requires fewer evaluations of the function, thus saving time. Further, the Simplex method is a search algorithm that can be completely implemented using fixed-point representation. Fixed-point code typically runs faster than corresponding floating point implementations.

In addition, in one embodiment the solution handles scale changes within the Simplex search procedure. Changes in object scale (or size) are often problematic for 2D object trackers. As an object moves closer or farther from the camera, the object size varies, and if the tracker does not update the scale of its feature computation, the algorithm may drift away from (or lose track of) the object. Instead of using the simplex technique to merely search for the best matching fixed size region in 2D, we include the scale of the object in the parameter list of the function to be minimized. Thus, the simplex searches in spatial as well as scale space for the best match. Increasing the dimensionality of the function has the drawback of potentially increasing the complexity of the search space, making it more time consuming. To limit this problem, we introduce only one extra dimension, a single scale parameter for both width and height. Thus the function changes from F(x,y) to F(x,y,s).

The solution may also use a combination of motion and appearance features. A desired characteristic of features used in tracking is constancy, or robustness to changes in appearance, illumination, etc. We strive to achieve this goal by using a combination of motion and appearance features. The motion features are helpful in disambiguating objects from stationary parts of the scene, and also from other objects with very different motion signatures than the one being tracked. It is also quite robust to ambient illumination fluctuations. The motion features are however not very discriminative between moving objects of the same type, such as two persons walking close to each other and often overlapping in the image. In these scenarios, our appearance features are helpful in separating out objects. Further, the appearance features are helpful in maintaining tracking even when objects become stationary (and the motion features are useless).

The solution may also use spatially separated histograms for motion and appearance. Histograms of intensity values are commonly used features. In tracking they have a limitation in that they do not provide any information about the spatial distribution (or arrangement) of the intensity values. We attempt to mitigate this drawback by computing histograms in 4 separate quadrants of the object's bounding box and concatenating these individual histograms to form our feature vector. Additionally, for the appearance features, we also adopt a kernel weighted histogram computation so that we have more weight assigned to the center of the object and lesser weight to the surrounding background.

Such embodiments present advantages, such as, a) robust to illumination changes by relying on appearance and motion signature, b) provision to be adaptive to scale changes by Incorporating scale dimension in simplex, c) faster processing by utilizing the chosen algorithm components and the optimized implementation, e) robust to small/brief occlusions based on underlying Kalman filter, and the likes.

While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.