Information processing unit, information processing method, and recording medium therewith

1. Field of the Invention

The present invention relates to information processing units, information processing methods, and recording media therewith. More particularly, the present invention relates to an information processing unit, an information processing method, and a recording medium therewith, for accurately predicting subsequent motion by storing motion information and torque information in advance and predicting the subsequent motion based on the stored motion information and torque information.

2. Description of the Related Art

Hitherto, conventional systems for measuring or estimating motion parameters of an object based on an input of an image or a sensor output fail to accurately estimate these parameters due to noise included in the input data and the inadequacy of data relative to the parameters to be estimated.

To this end, Kalman filters described in “A New Approach to Linear Filtering and Prediction Problems” by R. E. Kalman, Trans. ASME-J. Basic Eng., pp. 35-45 March 1960, or recursive estimation filters derived from the Kalman filters have been used.

In linear systems, Kalman filters are often used as the recursive estimation filters. In nonlinear systems, extended Kalman filters (hereinafter referred to as “EKF”) are the most popular. The EKF is described in “Applied Optimal Estimation” by A. Gelb, The Analytic Science Corp., 1974. Other applications of Kalman filters include unscented filters described in “A New Extension of the Kalman Filter to Nonlinear Systems” by S. J. Julier, Proc. of AeroSense: The 11th International Symposium on Aerospace/Defense Sensing, Simulation and Control, and pseudo-Kalman filters described in “Using Pseudo Kalman Filters in the Presence of Constraints, Application to Sensing Behaviours” by T. Vileville and P. Sander, Technical Report 1669, INRIA-Sophia, Valbonne, France, 1992.

FIG. 11 is a block diagram of a Kalman filter 120 . The Kalman filter 120 employs an efficient least-square method. The Kalman filter 120 estimates states in the past, present, and future. The Kalman filter 120 is capable of estimating state variables based on incomplete information. In addition to the state variables to be estimated, the Kalman filter 120 updates simultaneously error covariance indicating estimation accuracy. Use of the Kalman filter 120 is known as a robust estimation method.

The Kalman filter 120 is one of the recursive estimation filters. The Kalman filter 120 always exclusively uses the most recent data from observed values input in series. The Kalman filter 120 includes a predicting unit 131 for predicting motion and an updating unit 132 for updating the predicted data. Estimation is carried out by alternatively repeating prediction and updating of the data by, respectively, the predicting unit 131 and the updating unit 132 .

Referring to FIG. 12, operation of the Kalman filter 120 is described. A set of state variables to be estimated is defined as a state vector x. When the number of the state variables is n, the state vector x is n-dimensional. A k-th cycle (one cycle corresponds to one observation cycle) state vector x is expressed by a state vector X k . A prediction process for predicting a subsequent state is expressed by an expression (1). In expressions illustrated hereinafter, a vector is expressed with an arrow at the top thereof; a first order time differential of a vector is indicated by a dot ({dot over ( )}) at the top thereof; and a second order time differential is indicated by two dots at the top ({dot over ( )}) thereof:

{right arrow over (x k+1 )}= A k {right arrow over (x k )}+ {right arrow over (w k )}   (1)

In the above expression, a matrix A k is the k-th cycle state transition matrix with n rows and n columns. w k represents prediction process noise generated in the prediction process. It is assumed that w k is normally distributed Gaussian noise.

An observation process is expressed by an expression (2):

{right arrow over (y k )}= H k {right arrow over (x k )}+ {right arrow over (v k )}   (2)

The covariance matrix of the process noise generated in the process shown by the expression (1) is expressed by Q k .

In the expression (2), y k indicates an observation vector representing a set of the k-th cycle observed values, and H k represents the k-th cycle observation matrix. The observation vector y k does not necessary have the same number of dimensions as the state vector. If the observation vector y k is assumed to be m-dimensional, the observation matrix H k has m rows and n columns. v k represents a vector of observation noise generated in the observation process, which is assumed to be Gaussian noise. The covariance matrix of the observation noise is expressed by R k .

The Kalman filter 120 performs estimation through repetitions of predicting and updating. The estimated state vector before the updating is expressed by an expression (3), and the estimated state vector after the updating is expressed by an expression (4):

{right arrow over (X {overscore (k)})}   (3)

{right arrow over (X k )}   (4)

Let P k represent the estimated error covariance after the updating. The estimated error covariance before the updating is expressed by an expression (5):

P − k   (5)

Assuming the above, processing of the predicting unit 131 is described. As shown in FIG. 12, the predicting unit 131 performs process e 1 to predict the next state based on an expression (6):

{right arrow over (X {overscore (k)})}+ 1= A k {right arrow over (X k )}   (6)

The predicting unit 131 performs process e 2 to predict the next estimated error covariance P − k+1 based on an expression (7):

P − k+1 =A k P k A k T +Q k   (7)

In the above expression, A k T represents a transposed matrix of the state transition matrix A k .

Next, processing of the updating unit 132 is described. The updating unit 132 performs process c 1 to compute a Kalman gain K k based on an expression (8):

K k =P − k H k T ( H k P − k H k T +R k ) −1   (8)

In the above expression, H k T represents a transposed matrix of the observation matrix H k , and ( ) −1 represents an inverse matrix of the matrix within the parentheses ( ).

The updating unit 132 performs process c 2 to update the estimated value of the state variable based on the input observed value using an expression (9):

{right arrow over (X k )}= {right arrow over (X {overscore (k)})} + K k ( {right arrow over (y k )}− H k {right arrow over (X {overscore (k)})} )  (9)

The updating unit 132 performs process c 3 to update the estimated error covariance P k based on an expression (10):

P k =( I−K k H k ) P − k   (10)

In the above expression, I represents a unit matrix that has n×n dimensions.

The above-described Kalman filter 120 assumes that both the prediction process and the observation process are linear. In practice, however, the prediction process and the observation process are often nonlinear. To accommodate such nonlinear processes, various types of derived filters including the EKF have been proposed. The EKF, which is most commonly used, is described below.

The EKF expresses a prediction process as an expression (11) (corresponding to the expression (1) of the Kalman filter 120 ), and an observation process as an expression (12) (corresponding to the expression (2) of the Kalman filter 120 ):

{right arrow over (x k+1 )}= f{right arrow over ((x k ))}+ {right arrow over (w k )}   (11)

{right arrow over (y k )}= h{right arrow over ((x k ))}+ {right arrow over (v k )}   (12)

In the expression (11), f represents a nonlinear state transition matrix. In the expression (12), h represents a nonlinear observation matrix.

In the EKF, the actual estimation framework is expressed by an expression (13) which corresponds to process e 1 of the Kalman filter 120 (FIG. 12 ):

{right arrow over (X {overscore (k)})}+ 1= f{right arrow over ((X k ))}   (13)

In the EKF, processing corresponding to process c 2 of the Kalman filter 120 (FIG. 12) is expressed by an expression (14):

{right arrow over (X k )}= {right arrow over (X {overscore (k)})} + K k ( {right arrow over (y k )}− h{right arrow over ((X {overscore (k)})} )  (14)

In the EKF, a state transition matrix A k is expressed by an expression (15), and an observation matrix H k is expressed by an expression (16): A k     ( i , j ) = ∂ f ( i )     ( X k ) → ∂ x k  ( j ) ( 15 ) H k     ( i , j ) = ∂ h ( i )     ( X k ) → ∂ x k  ( j ) → ( 16 )

In the expression (15), A k(i,j) represents a matrix element in the i-th row, j-th column, of the state transition matrix A k , which is computed by partially differentiating the i-component of a function f by the j-component of the state vector x k . In the expression (16), H k(i,j) represents a matrix element in the i-th row, j-th column, of the observation matrix H k , which is computed by partially differentiating the i-component of a function h by the j-component of the state vector x k .

Except the points described above, the EKF operates in the same manner as the Kalman filter 120 .

In the above-described conventional recursive filters, the prediction process (represented by the expression (1) or the expression (2)) employed for prediction assumes constant angular velocity, constant angular acceleration, constant moving velocity, or constant moving angular acceleration.

For example, the angular acceleration is assumed to be constant in publications such as “Measurement of Angular Velocity and Angular Acceleration in Sports Using Extended Kalman Filters” by K. Ohta and K. Kobayashi, Collected Papers of the Society of Instrument and Control Engineers, vol. 31, no. 9. By way of another example, the moving acceleration is assumed to be constant in publications such as “Tracking Hand Dynamics in Unconstrained Environments” by Y. Azoz, IEEE Automatic Face and Gesture Recognition, pp. 274-279, April 1998. These assumptions do not sufficiently model the physical properties of an object. Therefore, the following problems may arise:

1. When inputting an image, ambiguous information on the depth direction may not be recovered.

2. Estimation errors may occur due to inadequacy of information for predicting the motion of a fast-moving object between two successive images or between successively sampled sensor information.

These problems are described below by way of examples. Referring to FIG. 13, it is assumed that an arm 142 of an object (a person) 141 is observed from the p-direction (direction from the right side of the object 141 ) and from the i-direction perpendicular to the p-direction. Joints include a wrist joint 142 - 1 , an elbow joint 142 - 2 , and a shoulder joint 142 - 3 . Referring to FIGS. 14A and 14B and FIGS. 15A to 15 E, examples of the first problem described above are illustrated. FIGS. 14A and 14B are illustrations of the arm 142 of the object 141 observed from the i-direction. Continuous movement of the arm 142 is illustrated by FIGS. 14A and 14B, moving from image i 1 shown in FIG. 14A to image i 2 shown in FIG. 14 B. Image p 1 in FIG. 15 A and image p 2 -a in FIG. 15B illustrate images of the arm 142 of the object 141 captured from the p-direction, when images i 1 and i 2 in FIGS. 14A and 14B are observed. Even if the data of image p 1 in FIG. 15A is accurately estimated for image i 1 in FIG. 14A, when the prediction process is inappropriate, the actual position indicated by the data of image i 2 in FIG. 14B will not be estimated correctly for image p 2 -a in FIG. 15 B. Instead, the actual position may be erroneously estimated as image p 2 -b in FIG. 15C, image p 2 -c in FIG. 15D, or image p 2 -d in FIG. 15 E.

Referring now to FIGS. 16A and 16B, an example of the second problem described above is illustrated. When an image moves from the state shown in FIG. 16A to the state shown in FIG. 16B, the magnitude of movement is great. The conventional method fails to accurately predict the second state (FIG. 16B) based on the first state (FIG. 16A) due to inadequacy of information.

Accordingly, it is an object of the present invention to provide an information processing unit, an information processing method, and a recording medium therewith, to accurately predict motion of an object.

In accordance with a first aspect of the present invention, there is provided an information processing unit including an information storage unit for storing motion information and torque information of an object and a motion estimating unit for estimating motion based on the information stored by the information storage unit.

In accordance with another aspect of the present invention, there is provided an information processing method including an information storing step for storing motion information and torque information of an object and a motion estimating step for estimating motion based on the information stored in the information storage step.

In accordance with another aspect of the present invention, there is provided a recording medium for recording a computer readable program. The program includes an information obtaining step for obtaining motion information and torque information of an object and a motion estimating step for estimating motion based on the information obtained in the information obtaining step.

Preferably, in the information processing unit and the information processing method, the motion information and the torque information of the object are stored, and the motion is estimated based on the stored information.

Preferably, in the program recorded on the recording medium, the motion information and the torque information of the object are obtained, and the motion is estimated based on the obtained information.

FIG. 1 is a block diagram of an articulated motion estimating unit 1 according to an embodiment of the present invention;

FIG. 2 is an illustration of a processing concept of the articulated motion estimating unit shown in FIG. 1 .

FIG. 3 is a flowchart showing operation of the articulated motion estimating unit 1 shown in FIG. 1;

FIG. 4 is a flowchart showing an updating process in step S 4 shown in FIG. 3;

FIG. 5 is a flowchart showing a prediction process in step S 6 shown in FIG. 3;

FIG. 6 is a data flowchart showing the operation of the articulated motion estimating unit 1 shown in FIG. 1;

FIG. 7 is another flowchart showing the operation of the articulated motion estimating unit 1 shown in FIG. 1;

FIG. 8 is a conceptual diagram of an application of the articulated motion estimating unit 1 shown in FIG. 1;

FIG. 9 is a block diagram of a motion capture system 51 ;

FIGS. 10A to 10 C are illustrations of an application of the articulated motion estimating unit 1 shown in FIG. 1;

FIG. 11 is a block diagram of a conventional Kalman filter 120 ;

FIG. 12 is an illustration of operation of the Kalman filter 120 shown in FIG. 11;

FIG. 13 is an illustration of problems of the Kalman filter 120 shown in FIG. 11;

FIGS. 14A and 14B are other illustrations of the problems of the Kalman filter 120 shown in FIG. 11;

FIGS. 15A to 15 E are other illustrations of the problems of the Kalman filter 120 shown in FIG. 11; and

FIGS. 16A and 16B are other illustrations of the problems of the Kalman filter 120 shown in FIG. 11 .

FIG. 1 is a block diagram of an articulated motion estimating unit 1 according to an embodiment of the present invention. A CPU 11 performs various processes in accordance with a program stored in a program memory 12 .

A data memory 13 stores not only angle information, angular velocity information, and torque information for estimating motion, but also observation information, such as a signal from a sensor or information from an image sequence, which is necessary to update estimated values. A frame memory 14 stores the image sequence when directly accessing the image sequence as the observation information, or image information generated to visualize estimated three-dimensional parameters.

An image display unit 15 displays the image information stored in the frame memory 14 . An input unit 16 inputs an image and a sensor signal using a mouse and a keyboard.

An external storage unit 17 stores observed values or estimated information. Preferably, a random access recording medium, such as a hard disk drive (HDD) or an optical disk, is used as the external storage unit 17 . However, a recording medium which is not particularly suited for random access, such as a tape streamer, may also be employed as the external storage unit 17 . An external storage unit of another system connected to a network may be used as the external storage unit 17 . Alternatively, a combination of the above-described various recording media and the external storage unit 17 may used as the external storage unit 17 . Such media may be used for storing the observed values from the image or the sensor when the articulated motion estimating unit 1 performs batch processing.

Referring now to FIG. 2, a processing concept of the articulated motion estimating unit 1 is illustrated. The articulated motion estimating unit 1 pre-stores angle information, angular velocity information, and torque information for each joint of an articulated object. Assuming that torque changes are constant, the articulated motion estimating unit 1 estimates motion by repetitively updating and predicting the input observation data based on the stored angle information, angular velocity information, and torque information.

A k-th cycle state vector x k for each joint of the object to be estimated, which is used in the articulated motion estimating unit 1 , is expressed by an expression (17): x k → = [ q k → q k → . τ k → ] ( 17 )

In the above expression, q k represents the k-th cycle angle for each joint, q k with a dot ({dot over ( )}) represents the k-th cycle angular velocity for each joint, and τ k represents the k-th cycle torque for each joint. When the degree of freedom of articulation is n-degree, the state vector x k is three-n dimensional.

Referring to FIGS. 3 to 6 , operation of the articulated motion estimating unit 1 is described. In step S 1 in FIG. 3, a user inputs an initial estimated value X 0 of the state vector x k shown in the expression (17) and an initial estimated error covariance P 0 of estimated error covariance P k using the input unit 16 . These values may be input directly by the user, or alternatively, may be supplied from other sensor information. When inputting other image data, the user may perform input by performing interactive three-dimensional fitting using the keyboard and the mouse of the input unit 16 while looking at the first frame of the image. The initial values may be output from another motion parameter estimating unit.

Step S 1 in FIG. 3 corresponds to input from INPUT in FIG. 6 . Referring to FIG. 6, elliptical portions indicate data conversion processes, which are referred to as “processes”; arrows indicate transfer of the processed data, which are referred to as “arrows”; rectangular portions indicate data production or data consumption, which are referred to as “actor objects”; and portions sandwiched between two lines represent passive data storage, which are referred to as “data store objects”.

The initial estimated value X 0 of the state vector x k and the initial estimated error covariance P 0 of the estimated error covariance P k input using the input unit 16 are stored via an internal bus 31 into the data memory 13 .

In step S 2 , the CPU 11 determines that k of the initial estimated value X 0 of the state vector x k and the initial estimated error covariance P 0 of the estimated error covariance P k stored in the data memory 13 is k=1, i.e., the first cycle data. This is used as the first cycle internal information.

In step S 3 , the CPU 11 computes and predicts a (k+1)-th cycle state vector X (k+1) and estimated error covariance P (k+1) using the updated k-th cycle vector x k and estimated error covariance P k . Details of the prediction processing are illustrated in FIG. 5 .

In step S 21 in FIG. 5, the CPU 11 computes a k-th cycle angular acceleration q k with two dots ({umlaut over ( )}) using the state vector x k stored in the data memory 13 . Specifically, an expression (18) of a dynamics equation, which is often used in robotics to obtain a relationship among an angle, acceleration, and torque, is modified into an expression (19). Using the expression (19), the CPU 11 computes the angular acceleration q k with two dots by substituting the angle q k , the angular velocity q k with a dot, and the torque τ k :

{right arrow over (τ)}= M{right arrow over ({umlaut over (q)})}+{right arrow over (b)} ( {right arrow over (q)}, {right arrow over ({dot over (q)})} )  (18)

{right arrow over ({umlaut over (q)})}=M −1 ({right arrow over (τ)}− {right arrow over (b)} ( {right arrow over (q)}, {right arrow over ({dot over (q)})} ))  (19)

For a matrix M (mass matrix), it is certain that there is always an inverse matrix M −1 . (This is described in “Introduction to Robotics-Mechanics and Control” by John J. Craig, second edition, Addison-Wesley Publishing Co., 1989.) Thus, the angular acceleration q k with two dots is computed based on the expression (19).

Referring to FIG. 6, step S 21 is illustrated by arrows indicating the state vector having the angle, angular velocity, and torque stored in a data store sd directed to an angular acceleration computing process de in a prediction block, angular acceleration computing processing in the angular acceleration computing process de, and an arrow indicating the angular acceleration computed in the angular acceleration computing process directed to an angle/angular velocity prediction process ode.

In step S 22 , the CPU 11 computes the next (k+1)-th cycle angle q k+1 and angular velocity q k+1 with a dot ({dot over ( )}) based on the state vector x k stored in the data memory 13 and the angular acceleration q k with two dots ({umlaut over ( )}) computed in step S 21 . An ordinary differential equation of an expression (20) is used to compute the next (k+1)-th cycle vector z k+1 based on a vector z k and its differential z k with a dot ({dot over ( )}):

{right arrow over (Z k+1 )}=ODEINT( {right arrow over (z k )}, {right arrow over ({dot over (z)})} k )  (20)

In the above expression, ODEINT includes an ordinary differential equation integrator (solver), which is typically represented by the Runge-Kutta method. Using ODEINT and the angular acceleration q with two dots ({umlaut over ( )}) computed by the expression (19), the CPU 11 computes the next (k+1)-th cycle angle q k+1 and angular velocity q k+1 with a dot ({dot over ( )}) based on an expression (21): [ q k + 1 → q k + 1 → . ] = ODEINT     ( [ q → q → . ] , [ q → . q → ¨ ] ) ( 21 )

Accordingly, the next (k+1)-th cycle state vector x k+1 is accurately computed based on information on the present cycle state vector x k . With continued reference to FIG. 6, step S 22 is illustrated by arrows indicating the angle and angular velocity stored in the data store sd directed to the angle/angular velocity prediction process ode, the angle/angular velocity prediction processing in the angle/angular velocity prediction process ode, and arrows indicating a predicted angle and predicted angular velocity computed in the angle/angular velocity prediction process ode directed to an updating process ud.

The above-described prediction processing is rearranged as follows: [ q k + 1 → q k + 1 → . τ k + 1 → ] = [ ODEINT     ( [ q k → q k → . ] , [ q k → . q k → ¨ ] ) τ k → ] + [ ɛ 1 → ɛ 2 → ω → ] ( 22 )

The articulated motion estimating unit 1 assumes that torque changes are constant and the torque changes are Gaussian noise. In the expression (22), ω represents Gaussian noise. The above-described method performs substantially accurate computation of the next cycle angle and angular velocity. In order to eliminate or minimize numerical errors, noise vectors ε 1 and ε 2 representative of the numerical errors are defined. The noise vectors ε 1 and ε 2 are very small values. For example, the noise vectors ε 1 and ε 2 are small enough with respect to the Gaussian noise ω, satisfying ε 1 , ε 2 <<ω. It is possible to increase the values of the noise vectors ε 1 and ε 2 in a processing system with lower numerical calculation accuracy. When an ordinary differential equation integrator (solver) with lower accuracy, such as the Euler method, is used to minimize computation time, the values of the noise vectors ε 1 and ε 2 may be increased.

Step S 21 , i.e., the expression (19), and step S 22 , i.e., the expression (22), may be concluded as follows: [ q k + 1 → q k + 1 → . τ k + 1 → ] = [ ODEINT     ( [ q k → q k → . ] , [ q k → . M - 1     ( τ k → - b →     ( q k → , q k → ) ) ] ) τ k → ] + [ ɛ 1 → ɛ 2 → ω → ] ( 23 )

In step S 23 , the CPU 11 computes the next (k+1)-th cycle estimated error covariance P k+1 based on the estimated error covariance P k stored in the data memory 13 . The CPU 11 uses the next (k+1)-th cycle angle q k+1 and angular acceleration q k+1 with two dots ({umlaut over ( )}) computed in step S 22 to compute a state transition matrix A k based on an expression (24) (the same as the expression (15)): A k     ( i , j ) = ∂ f ( i )     ( X k ) → ∂ x k     ( j ) → ( 24 )

As described above, A k(i,j) represents a matrix element in the i-th row, j-th column, of the state transition matrix A k , and f (j) represents the j-component of a function f. It is shown that A k(i,j) is computed by partially differentiating f (j) by the j-component of the state vector x k .

Next, the CPU 11 uses the computed state transition matrix A k and the estimated error covariance P k stored in the data memory 13 to compute the next (k+1)-th cycle estimated error covariance P − k+1 based on an expression (25) (the same as the expression (7)):

P − k+1 =A k P k A k T +Q k   (25)

Referring back to FIG. 3, in step S 4 , the CPU 11 obtains the k-th cycle observation data based on the data stored in the data memory 13 .

Instead of the observation data, a sensor signal may be used. In such a case, an interface with the sensor signal is provided in the input unit 16 shown in FIG. 1 . Alternatively, tracking (characteristic point tracking or region tracking) and segmentation (region segmentation) may be performed for an image in each frame of the image sequence, thus obtaining positional information and posture information for each joint, and a combination of these. The resultant information may be used as the observation data. In this case, an interface with a unit for performing tracking and segmentation is provided in the input unit 16 . As described above, when a system for obtaining information from an image is constructed on the same hardware, an interface with an image unit, e.g., a camera or a video, may be provided in the input unit 16 . The above-described system may share the units of the articulated motion estimating unit 1 . Step S 4 corresponds to the input from INPUT shown in FIG. 6 .

In step S 5 , the CPU 11 updates the data stored in the data memory 13 . Details of the updating process are shown in FIG. 4 . Specifically, in step S 11 , the CPU 11 computes a Kalman gain K k based on the obtained k-th cycle observation data y k . First, the CPU 11 computes an observation matrix H k based on an expression (26) (the same as the expression (16)): H k     ( i , j ) = ∂ h ( i )     ( X k ) → ∂ x k     ( j ) → ( 26 )

As described above, H k(i,j) represents an element in the i-th row, j-th column, of the observation matrix H k . It is shown that H k(i,j) is computed by partially differentiating the i-component of a function h by the j-component of the state vector x k .

The CPU 11 computes the k-th cycle Kalman gain K k based on an expression (27) (the same as the expression (8)):

K k =P − k H k T ( H k P − k H k T +R k ) −1   (27)

In the above expression, P − k represents the estimated error covariance before the updating, P k represents the estimated error covariance after the updating, and H k T represents a transposed matrix of H k .

In step S 12 , the CPU 11 updates the estimated value of the state vector x based on the observation data y k . Specifically, the CPU 11 updates the state vector based on the observation data y k using an expression (28) (the same as the expression (14)):

{right arrow over (X k )}= {right arrow over (X {overscore (k)})} + K k ( {right arrow over (y k )}− h {right arrow over ( X {overscore (k)})} ))  (28)

As described above, X − k represents the estimated state vector before the updating, and X k represents the estimated state vector after the updating.

In step S 13 , the CPU 11 updates the estimated error covariance P k based on an expression (29) (the same as the expression (11)):

P k =( I−K k H k ) P − k   (29)

In the above expression, I represents a unit matrix that has the same dimensions as P k .

Step S 12 and step S 13 may be performed simultaneously, or alternatively, may be performed sequentially from step S 12 to step S 13 or from step S 13 to step S 12 .

Referring back to FIG. 3, in step S 6 , the CPU 11 stores the updated k-th cycle state vector X k and estimated error covariance P k in the data memory 13 . In FIG. 6, this process is indicated as follows with corresponding arrows. The state vector having the angle, angular velocity, and torque which are updated in the updating process ud is stored in the data store sd, and the estimated error covariance updated in the updating process ud is transferred to the data store eec.

In step S 7 , the CPU 11 increments the number of cycles to obtain data for the next cycle. Specifically, the CPU 11 increments the number of the cycles from k to (k+1).

In step S 8 , the CPU 11 determines whether the above estimation process is completed. If the determination is negative, the CPU 11 returns to step S 3 and repeats step S 3 to step S 8 using the cycle number (k+1).

If the determination is affirmative, the process is ended. Alternatively, the determination may be performed by input from the user. When the data is image data, the affirmative determination may correspond to completion of the processing of the last frame of the image data. The affirmative determination may also be made due to hardware constraints or software constraints, such as the memory being full.

As described above, the articulated motion estimating unit 1 accurately predicts motion by maintaining each joint torque as a state variable and assuming torque changes are constant.

Elements of the state vector x are angle, angular velocity, and torque for each joint. Alternatively, the length of each joint may be added to the elements of the state vector x, so that accuracy of the motion estimation is improved.

In the above embodiment, the updated state vector x is recorded in step S 5 . However, if there is no need to perform real-time processing, step S 5 may be omitted. As illustrated in FIG. 7, step S 51 and step S 52 may be inserted in between step S 5 ′ and step S 6 ′. In step S 51 , contents of the updated state vector x are visualized, and the frame memory 14 records the visualized state vector x. In step S 52 , the image display unit 15 displays the visualized data. Accordingly, the user may immediately recognize the prediction and updating of the state vector from the image display unit 15 .

In the above embodiment, it is assumed that the observation data has been obtained in advance. However, the articulated motion estimating unit 1 may be utilized for generating observation data. The concept of this is illustrated in FIG. 8 .

Referring to FIG. 8, information on joints is retrieved from an input image by a retrieving unit, e.g., a tracker, for retrieving information on joints. Then, the articulated motion estimating unit 1 predicts an estimated value, which in turn is fed back to the retrieving unit. Accordingly, the observation accuracy is improved.

Referring now to FIG. 9 and FIGS. 10A to 10 C, an embodiment of a system realizing the concept in FIG. 8 is illustrated. FIG. 9 is a block diagram of a motion capture system 51 utilizing the articulated motion estimating unit 1 . A video camera 71 outputs image data, which in turn is input to the articulated motion estimating unit 1 . Then, a state vector is estimated, which is input as three-dimensional motion parameters to a three-dimensional animation creating unit 72 . The three-dimensional animation creating unit 72 creates three-dimensional animation based on the supplied motion parameter three-dimensional animation data.

Instead of the output from the video camera 71 , the input to the articulated motion estimating unit 1 may be input from a videocassette recorder or the like. Image sequence information which is binary-coded in advance may be input to the articulated motion estimating unit 1 via the external storage unit 17 .

FIGS. 10A to 10 C illustrate an example of a system in which the articulated motion estimating unit 1 is used as a user interface unit. Referring to FIG. 10A, the articulated motion estimating unit 1 estimates three-dimensional motion information for joints of a user 141 based on image information (for example, as shown in FIG. 10B, the image information on the user 141 captured by the video camera 71 ) input from the video camera 71 . This is then transformed into a command, which is supplied as a control signal to a controlled unit 91 (for example, as shown in FIG. 10C, a television receiver 121 ). The controlled unit 91 operates in accordance with the control signal.

The system shown in FIGS. 10A to 10 C may be effectively applied to entertainment fields, where the controlled unit 91 includes a game machine or the like.

The system terminology used herein refers to an overall unit formed by a plurality of units.

A series of the above-described processes may be performed by hardware, or alternatively, may be performed by software. In the latter case, the software program is installed from a recording medium into a computer embedded in exclusive hardware, or into a general-purpose personal computer for executing various functions by installing various programs.

The recording media may include, as illustrated in FIG. 1, in addition to the articulated motion estimating unit 1 , a magnetic disk 21 (including a floppy disk), an optical disk 22 (including a compact disk-read only memory (CD-ROM) and a digital versatile disk (DVD)), a magneto-optical disk 23 (including a mini-disk (MD)), and packaged media formed of a semiconductor memory 24 or the like. Such media for recording a program are distributed among users for supplying the program, and are to be mounted on the drive 18 . The recording media may also include the program memory 12 for recording the program and a hard disk contained in the external storage unit 17 , which are supplied to the user as built-in units in the articulated motion estimating unit 1 .

In addition to time-series processing following the described order, the process for writing a program to be recorded on the recording medium may also be performed by parallel processing or serial processing.