Method for segmenting and tracking content in videos using low-dimensional subspaces and sparse vectors

This invention relates generally to computer vision, and more particularly to segmenting and tracking content in videos.

The problem of segmenting and tracking low-dimensional subspaces embedded in high dimensional data arises in many applications such as background segmentation, anomaly detection, motion segmentation, and target localization. For example, a scene acquired by a stationary or moving camera can be partitioned into a low-rank component spanning a subspace that characterizes a relatively stationary background in the scene, and a sparse component corresponding to moving objects in the video scene, usually in the foreground.

The problem is to identify, at every time step t, e.g., each image in a sequence of images, an r-dimensional subspace t in n with r<<n that is spanned by columns of a rank-r matrix U t ε n×r from incomplete and noisy measurements

b t =Ω t ( U t a t +s t ),   (1)

where Ω t is a selection operator that specifies subset of the sets of one or more images at time t, a t ε r are coefficients specifying linear combination of columns of the subspace U t , and s t ε n is a vector of sparse outliers.

When the subspace t is stationary, the subscript t is omitted from U t and the problem reduces to matrix completion or principal component analysis (PCA) where the task is to separate a matrix BεE n×m into a low-rank component UA, and a sparse component S using incomplete measurements

B Ω =Ω( UA+S ).

The columns of the matrices A and S are respectively the vectors a t and s t stacked horizontally for all tε{1 . . . m}, and the selection operator Ω specifies the measured data in the matrix B.

Conventional methods for low-dimensional subspace identification first organize the measured data into a matrix and then determine basis vectors that span the subspace using a variety of techniques, e.g., low-rank matrix factorization.

Extensions of those methods factor the matrix into a low-rank component corresponding to the subspace, and a sparse component that represents noise.

However, when the dimensionality of the data becomes large, as is the case of a video, latency becomes a problem. Hence, it is necessary to provide a method that can segment and track the low-dimensional subspace as the data are acquired or processed in real-time, even when the data are incomplete and corrupted by sparse noise. Another problem is that the low-dimensional subspace (background) can vary over time, in which case the subspace cannot be represented by a low rank matrix when all data are grouped into one matrix. For example, the background in outdoor scene can vary in illumination during the day. Similarly, erstwhile moving objects can be added or removed from the background, where the are stationary, in surveillance videos.

One prior art method, U.S. Pat. No. 7,463,754, “Adaptive probabilistic visual tracking with incremental subspace update,” describes a method for adaptive probabilistic tracking of an object as represented in a motion video. The method identifies an eigenbasis that represents the object being tracked. A maximum a posteriori estimate of the object location is determined using the current estimate of the eigenbasis. The eigenbasis is then updated to account for changes in the appearance of the target object.

Another prior art, US 20030108220, “Robust, on-line, view-based appearance models for visual motion analysis and visual tracking,” describes learning an appearance model that includes both a stable model component, learned over a long time course, and a transient component, learned over a relatively short time course. The model parameters are adapted over time using an online expectation-maximization (EM) algorithm.

U.S. Pat. No. 8,477,998, “Object tracking in video with visual constraints,” describes tracking objects represented in a video by determining tracking states of the object based on a pose model, an alignment confidence score, and an adaptive term value. The tracking state defines a likely position of the object in the frame given the object's likely position in a set of previous frames in the video.

Grassmanian Rank-One Update Subspace Estimation (GROUSE) is one method that can handle real-time subspace estimation from incomplete data, see Balzano et al., “Online identification and tracking of subspaces from highly incomplete information,” 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 704-711, Sep. 2010. GROUSE uses rank-one updates of the subspace on a Grassmannian manifold. However, GROUSE can get trapped at a local minima.

Parallel Subspace Estimation and Tracking by Recursive Least Squares (PETRELS) can also identify a low-dimensional subspace in real-time, see Chi et al., “Petrels: Parallel subspace estimation and tracking by recursive least squares from partial observations,” IEEE Transactions on Signal Processing, vol. 61, no. 23, pp. 5947-5959, 2013. PETRELS minimizing, in parallel, a geometrically discounted sum of projection residuals on the data for each time step using a recursive procedure with discounting for each row of the subspace matrix. Both GROUSE and PETRELS cannot correctly handle corrupted data and data subject to non-Gaussian noise.

Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA) is similar to GROUSE, see Cornell University, arXiv:1109.3827, Sep. September 2011. GRASTA also updates the Grassmannian manifold, but replaces the l 2 cost function of GROUSE with a l 1 -norm cost function. This cost function minimizes a sum of absolute errors, while correctly handling outliers in the data.

Another real-time method is Recursive Projected Compressive Sensing (ReProCS), see Qiu et al. “ReProCS: A missing link between recursive robust PCA and recursive sparse recovery in large but correlated noise,” CoRR, vol. abs/1106.3286, 2011. ReProCS recursively projects data onto a orthogonal complement of the subspace followed by sparse recovery to determine outliers. However, that method requires an accurate initial estimate of the subspace.

The embodiments of the invention provide a method and system for segmenting and tracking time-varying content of videos in real-time. The content can include a dominant component and a sparse components. The dominant component can include, e.g., a largely stationary background represented as a low-dimensional subspace. The sparse component can include moving objects represented by sparse vectors.

The video can be acquired from incomplete measurements and in the presence of noisy outliers. The method minimizes a l 1 -norm cost function between the measurements and a projection of the measurements onto an estimate of the subspace.

Coefficients of the projection, and the sparse outliers are first determined for the current estimate of the subspace using, for example, an alternating direction method of multipliers (ADMM), and the subspace estimate is updated using a proximal point iterative procedure with an adaptive step-size parameter. The proximal point iterative procedure, similar to Newton's method, solves unconstrained smooth minimization problems involving high-dimensional data.

In one embodiment of the invention, the measurements are a sequence of images in a video. A set of one or more images are processed for each time step. For example, the set can be a group of pictures (GoP). The method characterizes the dominant stationary background of the video as the low-dimensional subspace and separates the relatively large background from sparse objects that typically characterize smaller moving objects in the video, usually in the foreground.

In another embodiment of the invention, the measurements are motion vectors extracted from a sequence of images in a compressed video. The motion vectors can represent the optical flow in the video. The method according to this embodiment identifies and tracks a dominant optical flow in the video using the low-dimensional subspace and separates the dominant optical flow from alternate optical flows, for example optical flows that are different than the dominant optical flow.

In another embodient, the measured data corresponds to interest points with feature descriptors that are extracted and tracked in a streaming video. In one example, only a subset of the extracted interest points are tracked. An adjacency matrix is constructed to characterise affinities between the feature descriptors within an image and across received video images. The method according to this embodiment identifies and tracks clusters of interest points that are represented by one or a union of subspaces that occupy a portion of the spectrum of the graph.

FIG. 1 is a block diagram of a method and system for processing a video according to the embodiments of the invention; and

FIG. 2 is a block diagram of pseudocode of the method of FIG. 1 .

The embodiments of our invention provide a method and system for segmenting and tracking objects in a data stream that lie in low dimensional subspaces using measurements of the data stream. For example, consider a sequence of images in a video stream where a large collection of objects in the video have a dominant motion trajectory that is static or changing slowly and where other objects in the video have a different motion trajectory than the dominant trajectory.

In one embodiment, the method described in this invention separates a stationary and relatively large background component of measurements, in the form of images of a video arriving at a processor after the images are acquired, from typically sparse objects that characterize smaller moving objects in the video, usually in the foreground.

In another embodiment of the invention, the measurements are motion vectors extracted from a sequence of images in a compressed video possibly corrupted with non-Gaussian noise. The motion vectors represent the optical flow in the video that tracks the motion of a large collection of objects in the video. The method segments and tracks a dominant optical flow using the low-dimensional subspace or a union of low-dimensional subspaces.

In yet another embodiment, the measurements have missing data points. For example, if the motion vectors are extracted from only a subset of a sequence of images in a compressed video. The method of this invention determines the missing data points that correspond to the dominant optical flow after identifying the low-dimensional subspace or a union of low-dimensional subspaces.

As shown in FIG. 1 , the method and system for processing measured data 101 , e.g., a sequence of images in a video 101 that is acquired of a scene 102 by a camera 103 . The camera can be stationary or moving. The images can be processed in real-time, for example, at the same frame (24 or 60 fps) as the images were acquired. The method operates in a processor 100 connected to memory and input/output interfaces by buses as known in the art. The memory is used to store the measured data representing the images, as well as other data structures, such as vectors, matrices and parameters used by the method.

For first measured data 104 in the stream, e.g., a first image in the video sequence, an intial subspace matrix 131 and an initial step-size parameter 126 are initialized 105 . A sparse component 111 , in the form of a sparse vector, and subspace coefficients 112 are determined 110 from the first measured data using an iterative solver, for example an alternating direction method of multipliers (ADMM). Next, a change in the subspace 121 is determined 120 according to the first measured data 104 , the sparse component 111 , and the subspace coefficients 112 . An adaptive step-size parameter 126 is updated 125 according to the change in subspace 121 is updated 120 . The subspace matrix 131 is then updated 130 using the change in subspace 121 and the updated adaptive step-size parameter 126 . For the second measured data and every subsequent measured data in the stream, the updated subspace matrix 131 and the updated adaptive step-size parameter are used as the initial subspace matrix and the initial step-size parameter. The process is repeated iteratively until all the measured data 101 from the data stream are processed.

After the arrival of every new measured data from the stream, the moving objects 108 as represented by the sparse vector have been separated from the background as represented in the current subspace matrix 121 .

FIG. 2 shows pseudocode for the detailed steps of the method. The variables used by the pseudocode are defined below.

In one embodiment, the measured data correspond to features of interest points in a video sequence. A graph is constructed from the measured data using feature descriptors, such as a Scale Invariant Feature Transform (SIFT), a Fast Retina Keypoint (FREAK), a Binary Robust Invariant Scalable Keypoints (BRISK), etc., corresponding to the interest points in order to assign edges and weights between the interest points. The method then identifies one or a union of a low-dimensional subspaces that occupy a portion of a spectrum of the graph and that characterises the dominant association between the interest points. The method also segments the dominant association from sparse or non-Gaussian distributed associations that exist in the graph spectrum.

Real-time Subspace Estimation

We descibe real-time estimation of the low-dimensional subspace matrix 131 from incomplete streaming measurements 101 , e.g., a compressed video, that may be corrupted with non-Gaussian noise. First, we describe our problem and define the notation used. Then, we describe minimizing a l- 1 -norm cost function between the measurements and their projection onto the subspace to determine the subspace coefficients 112 , and sparse outliers 111 . Then, the subspace is updated 130 using a proximal point iterative procedure, based on using least squares estimation, while updating 125 the adaptive step-size parameter 126 .

As advantages, our method does not restrict the subspace update to a Grassmannian manifold as in the prior art, and uses an adaptive step size. In addition, the method does not require an accurate initial estimate of the subspace, e.g., the subspace is set to a random subspace.

Augmented Lagrangian-Based Proximal Point Iterative Procedure

The method minimizes an augmented Lagrangian with the l 1 -norm cost function, and uses a smoothing term that maintains a proximity of the update to the previous subspace estimate over the variables (U t , s t , a t , y t ). An objective cost can be represented by

ℒ ′ ⁡ ( U t , s t , a t , e t , y t ) =  s t  1 + y t T ⁡ ( b t - ( U t ⁢ a t + s t + e t ) ) + μ 2 ⁢  b t - ( U t ⁢ a t + s t + e t )  2 2 + μ 2 ⁢  U t - U t - 1  2 2 , ( 2 )

where e t is supported on the complement of the selection operator Ω t , denoted Ω t c , such that Ω t (e t )=0 and Ω t c (e t )=−Ω t c (U t a t ).

Equation (2) is non convex in the variables U t and a t . Therefore, we follow the PETRELS and GRASTA approach of alternating the minimization over the variables (s t , a t , y t ) and then the variables U t . By fixing U t , the minimizers of equation (2) are equal, i.e.,

Then, the subspace U t is updated by taking a gradient step to minimize the function

ℱ ⁡ ( U t ) = 1 2 ⁢  b t - ( U t ⁢ a t + s t + e t )  2 2 + 1 2 ⁢  U t - U t - 1  2 2 ( 4 )

using the adaptive step-size μ.

Method

As shown by the pseudocode in FIG. 2 , after prelimenaries, the first stage (steps 4 - 11 ) uses an iterative method, for example the ADMM, to solve equation (3). The variables a t , s t , and y t are determined (steps 7 - 10 ), until a termination condition is satisfied, by iterating the following sequence of updates:

a t k = U t - 1 † ⁡ ( b t - s t k - 1 - e t k - 1 + 1 μ t - 1 ⁢ y t k - 1 ) ,

⁢ e t k = - Ω t c ⁡ ( U t - 1 ⁢ a t k ) ,

⁢ s t k = 1 μ t - 1 ⁢ ( b t - U t - 1 ⁢ a t k - e t k - 1 μ t - 1 ⁢ y t k - 1 ) ,

⁢ and ⁢

⁢ y t k = y t k - 1 + μ t - 1 ⁡ ( b t - U t + 1 ⁢ a t k - s t k - e t k ) ) , ( 5 )

where τ (x)=sign(x)·max{|x|−τ,0} denotes an element-wise soft thresholding operator with threshold τ, k indicates the iteration number, and † represents a Moore-Penrose pseudo-inverse of a matrix.

In the second stage (steps 12 - 17 ), the subspace U t is updated (step 19 ) by minimizing equation (4) using

U t = μ t - 1 μ t ⁢ ( U t - 1 + ( b t - s t - e t ) ⁢ a t T ) ⁢ ( I r + a t ⁢ a t T ) - 1 , ( 6 )

where I r is an r×r identity matrix, and the step size μ t 126 is updated adaptively.

Adaptive Step-Size Parameter

For the adaptive step-size parameter, the method uses a regularizer μ t to control the speed of convergence of the estimation of the subspace 131 . In particular, a smaller value of μ allows for faster adaptability of U t to a changing subspace, i.e., with a larger descent direction, whereas a larger value of μ only allows a slower change in U t .

Consider a descent direction

D t =( U t-1 +( b t −s t −e t ) a t T )( I r +a t a t T ) −1 −U t-1 ,   (7)

and determine its projection onto an orthogonal complement of the previous subspace estimate to obtain the change in subspace 121

G t =( I−U t-1 U t-1 † )D t .  (8)

Then, the adaptive step-size parameter μ t 126 can be updated 125 according to

μ t = C ⁢ ⁢ 2 - l 1 + η t ,

where

η t = η t - 1 + sigmoid ⁡ ( 〈 G t - 1 , G t 〉  G t - 1  F ⁢  G t  F ) ,

and lε{−1,0,1,2} are set according to predetermined thresholds for η t . Here, sigmoid(x)=ƒ+2ƒ/(1+e 10x ) for some predefined ƒ.

Similar to GRASTA, the intuition behind selecting such an update rule comes from the idea that if two consecutive subspace updates G t-1 and G t have the same direction, i.e., (G t-1 , G t )>0, then the target subspace is still far from the current subspace estimate. Consequently, the updated step size μ t should be smaller to allow for fast adaptability, which is achieved by increasing η t . Similarly, when (G t-1 , G t )<0, the subspace update can oscilate around the target subspace and hence a larger μ t is needed. Note that when the product of the norm of the subspace updates (∥G t-1 ∥ F ·∥G t ∥ F ) is too small, e.g., smaller than 10 −6 , we assume that the current subspace estimate is close to the target subspace, and we force η t to decrease by the magnitude of the sigmoid.

Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.