Real-time single-view action recognition based on key pose analysis for sports videos

This disclosure relates generally to digital content processing, and more specifically to real-time, single view action recognition based on analysis of key poses of sports videos.

Smart computing devices, such as smart phones and tablet computers, have become increasingly popular. The increased availability and bandwidth of network access (for wired and wireless network) has enabled a variety of mobile applications for digital content processing with improved efficiency to enhance user experience with mobile applications. For example, a user may use a mobile application to record videos of him/her playing golf with his/her smart phone and to save the sports actions performed by the user, e.g., golf swings, such that the recorded sports actions can be compared with sports actions performed by professional athletes later. To compare sports actions by the user with those performed by professional athletes, the mobile application needs to be able to recognize sports actions recorded by the user's smart phone. Recognizing sports actions for a sports video is to find out whether a sports action (e.g., a baseball swing or a golf swing) has happened in the video frames of the sports video.

Various solutions for player action recognition in sports video were proposed based on e.g., machine learning techniques, exemplar-based multi-view analysis. For example, machine learning technologies are used to train feature models based on a large corpus of sports videos. The trained feature models are applied to input sports videos for real-time action recognition. However, conventional solutions based on machine learning techniques for training feature models often rely on manual classification to select video frames showing specific sports actions, which is computationally expensive and challenging to efficiently recognize sports actions in a large corpus of video frames. Additionally, existing solutions are generally not suited for real-time action recognition, especially for videos captured by mobile computing devices with limited computational power, which degrade user experience with sports videos.

Embodiments of the disclosure includes is a system (or a computer implemented method or a non-transitory computer readable medium) for real-time single-view action recognition for sports videos based on analysis of identified key poses of sports videos by a set of trained feature models. A single-view of a sports video refers to a designated view point, from which the sports video is recorded. Key poses of a sports video are associated with the type of the sports video, e.g., golf, baseball and tennis; key poses of a sports action in a sports video of a particular type of sports refer to a set of visually distinctive postures/poses in a specific order when a player performs the sports action. Different types of sports videos, e.g., golf, baseball, have different key poses identified by the disclosed system. For example, a baseball swing is represented by a set of three key poses (e.g., a begin pose, an impact pose, and an end pose), which distinguish from key poses of other sports types such as a golf swing of a golf sports video. Real-time, single-view action recognition based on key pose analysis for a sports video refers to determining whether a sports action for a specific sports type (e.g., a baseball swing for a baseball sports video) has happened in the videos frames of the sports video based on the analysis of identified key poses of the sports action with respect to the timeline of the video frames of the sports video.

An embodiment of the method comprises receiving multiple training videos, each of which is associated with a sports type, e.g., a baseball video, and a distinctive sports action, e.g., a baseball swing. Each sports action associated with a type of sports is represented by a set of distinctive poses. For each sports type, one or more feature models are trained using the training videos. For example, for a specific sports action, e.g., a baseball swing, a set of feature models, e.g., a player detector, a set of key pose identifiers and a meta classifier, are trained using the training videos. When an input sports video of a specific sports type is received for real-time action recognition, the method selects the set of trained feature models for the requested specific sports type, and applies the set of trained feature models to the received input sports video to generate real-time action recognition result. The action recognition result is provided to the user in real-time as the sports video is received from the user and a final report indicating a complete recognition result is also presented to the user.

FIG. 1 is a block diagram of a system environment for action recognition for sports videos based on key pose analysis, according to one embodiment.

FIG. 2A shows a block diagram of a training module for training feature models using videos stored in a training video database illustrated in FIG. 1 , according to one embodiment.

FIG. 2B shows a block diagram of a recognition application module for applying trained feature models to an input video stored in an input video database illustrated in FIG. 1 , according to one embodiment.

FIG. 3A shows an example of three key poses representing a baseball swing action, according to one embodiment.

FIG. 3B shows an example of four key poses representing a golf swing action, according to one embodiment.

FIG. 4 shows an example of training a player detector for a baseball swing action, according to one embodiment.

FIG. 5 shows an example of training a set of key pose identifiers for the baseball swing action illustrated in FIG. 3A , according to one embodiment.

FIG. 6 shows a frame-by-frame analysis of score sequences for the three key poses representing a baseball swing action illustrated in FIG. 3A , according to one embodiment.

FIG. 7A shows training a meta classifier by applying a time window to the score sequences generated by key pose identifiers for a baseball swing action illustrated in FIG. 6 , according to one embodiment.

FIG. 7B shows example training results from training the meta classifier illustrated in FIG. 7A , according to one embodiment.

FIG. 8 shows a flow chart illustrating a process of training feature models illustrated in FIG. 4 - FIG. 7B , according to one embodiment.

FIG. 9 shows a flow chart illustrating a process of real-time action recognition using the trained feature models, according to one embodiment.

The figures and the following description describe certain embodiments by way of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures to indicate similar or like functionality.

FIG. 1 is a block diagram of a system environment 100 for action recognition for sports videos based on key pose analysis, according to one embodiment. In the embodiment of FIG. 1 , the system environment 100 includes one or more client devices 110 A- 110 B, and an action recognition server 200 connected over a network 120 . Only two client devices 110 and one action recognition server 200 are shown in FIG. 1 in order to simplify the description. Other embodiments of the system environment 100 can have multiple client devices 110 and recognition servers 200 through the network 120 . Likewise, the functions performed by the various entities of FIG. 1 may differ in different embodiments.

A client device 110 is an electronic device used by a user to perform functions such as recording a sports video, consuming digital content, executing software applications, browsing websites hosted by web servers on the network 120 , downloading and uploading files (e.g., uploading a sports video and downloading a sports action recognition result), and the like. Example client devices include a smart phone, or tablet, laptop, or desktop computer. The client device 110 includes and/or interfaces with a display device on which the user may view videos and other content. Additionally, the client device 110 provides a user interface (UI), such as physical and/or on-screen buttons, with which the user may interact with the client device 110 to perform functions such as viewing, selecting, and consuming digital content. The client device 110 may also include a digital camera or a component cable of being used by a user of the client device 110 to capture sports videos.

In one embodiment, each of the client devices 110 further includes a software application module 112 executing a software application on the client device 110 for sports action recognition. For example, the software application module 112 A executing a software application on the client device 110 A receives a sports video recorded by the client device 110 A and uploads the recorded sports video for action recognition to the action recognition server 200 . The software application module 112 A receives the action recognition results from the action recognition server 200 and provides the action recognition result for presentation to the user of the client device 110 A.

In another embodiment, the software application module 112 of the client device 110 receives instructions from the action recognition server 200 to execute the software application to recognize the sports actions in a recorded sports video by the client device 110 . The instructions from the action recognition server 200 includes instructions to apply a trained model associated with the type of the recorded sports video (e.g., baseball) to identify the set of key poses associated with the type of the sports videos from the sequence of video frames of the recorded sports video. Based on the analysis of the identified key poses in the recorded sports video, the software application module 120 determines whether one or more sports actions (e.g., baseball swings) have happened and locations in terms of frame numbers or timestamps of the identified sports actions in the recorded sports video.

The network 120 enables communications between the client devices 110 and the recognition server 200 . In one embodiment, the network 120 comprises the Internet and uses standard communications technologies and/or protocols, e.g., clouding computing. In another embodiment, the network 120 can use custom and/or dedicated data communications technologies.

The action recognition server 200 is configured for recognizing sports actions in various types of sports videos. In one embodiment, the action recognition server 200 performs the action recognition in two stages: a training stage and an applications stage. In the training stage, the action recognition server 200 trains various feature models: a player detector, a set of key pose identifiers, and a meta classifier for each type of sports videos. The player detector is trained for detecting the location of the player in the video frames of a sports video. A set of key pose identifiers for each type of sports videos is trained to identify key poses associated with a target sport action associated with the type of sports video. The meta classifier is trained to determine whether a target sports action of a sport video has happened and when the action happened based on analysis of the set of key pose identifiers associated with the sports video. In the application stage, the action recognition server 200 applies the trained player detector, a set of key pose identifiers and the trained meta classifier to recognize one or more sports actions in an input sports video.

The action recognition server 200 illustrated in the embodiment of FIG. 1 includes a video database 210 , a model database 220 , a training module 230 and a recognition application module 240 . Other embodiments of the recognition server 200 may include additional and/or different components. The video database 210 stores large corpuses of sports videos. In one embodiment, the video database 210 is partitioned to a training video database 212 and an input video database 214 . The training video database 212 stores a large corpus of training sports videos that are used to train the feature models for different types of sports videos, e.g., baseball, golf and tennis. The training videos may be received by the training video database 212 from data sources such as the sports websites, online forums or search engines. The input video database 214 stores input sports videos that are uploaded by users or captured by cameras of the client devices 110 for sports action recognition.

The model database 220 stores different sets of trained feature models associated with different types of sports videos. It is noted that a sports action for a type of sports video, e.g., a baseball swing or golf swing, can be represented by a small set of distinctive key poses and an order of the distinctive key poses associated with that particular sports type. For example, a baseball swing action can be represented by three key poses: a pose representing begin swing, a pose representing hitting baseball (or ball impact), and a pose representing end swing. For each specific sports action associated with a sports type (e.g., golf, baseball), a set of trained feature models include a player detector, a set of key pose identifiers and a meta classifier. Training the different feature models by the training module 230 are further described below with reference to FIG. 4 through FIG. 7B .

The recognition application module 240 selects a specific set of feature models trained by the training module 230 for an input sports video based on the type of the input sports video, and applies the set of feature models to input sports video for real-time action recognition in the input sports video. The recognition application module 240 may also present the action recognition result in real-time to the user of the input sports video. The recognition application module 240 may also provide a final report indicating the action recognition result and related statistics to the user of the input sports video.

FIG. 2A shows a block diagram of a training module 230 for training feature models using videos stored in the training video database 212 illustrated in FIG. 1 , according to one embodiment. The training module 230 illustrated in FIG. 2A includes a player detector training module 232 , a key pose identifier training module 234 and a meta classifier training module 236 . For a specific sports action of a sports video of a specific sports type, e.g., a baseball swing for a baseball video, the training module 230 trains a player detector to locate the player performing the specific sports action, and trains a set of key pose identifiers for identifying distinctive key poses representing the specific sports action. The training module 320 also trains a meta classifier for the sports action to determine whether the specific sports action has happened in the sports video based on the analysis of videos frames of the sports video by the key pose identifiers. The training module 230 may train different player detectors, different sets of key pose identifiers and different meta classifiers for different sports actions associated with different sports types. The trained feature models by the training module 230 are stored in the model database 220 .

The player detector training module 232 trains a player detector to locate the player who performs a target sports action, e.g., a baseball swing, in a sports video. The player detector is to detect the location of the player in each video frame of a sports video, where the location of the player is indicated by a player region within which a player is located within a video frame of the sports video. In one embodiment, to train the player detector, the player detector training module 232 extracts low-level visual features representing players in a corpus of training videos. In one embodiment, the training videos include video frames with manually labeled player regions and video frames with random image regions. A manually labeled player region refers to a part of a video frame that contains a player within the part and is considered as a positive sample for the player detector training procedure. An example manually labeled player region can be a rectangular box in a video frame of a sports video within which a player is contained. A random image region of a video frame refers to a part of the video frame that does not contain a player within that part and is considered as a negative sample for the player detector training procedure. In one embodiment, a player detector is trained to distinguish the positive samples and the negative samples using classification techniques such as aggregated channel features (ACF features) and a boosted tree classifier. The ACF features are an aggregation of image features such as pixel intensities, gradient magnitudes and gradient orientations.

Turning now to FIG. 4 , FIG. 4 shows an example of training a player detector for a baseball swing action, according to one embodiment. The video frame 400 illustrated by FIG. 4 is from a video frame of a training video, and the video frame 400 shows a player 410 that is playing baseball. The video frame 400 has one or more background items 430 , such as clouds in the sky 430 A and 430 B and grass 430 C behind the player 410 . In the example of FIG. 4 , the player 410 is performing a baseball swing with a stationary baseball placed on the baseball batting tee 460 in front of the player. The location of the player 410 within the video frame 400 is defined by a rectangular bounding box 450 .

The player detector trained by the player detection training module 232 is used to detect a player within a video frame of a sports video, for example, to detect the player 410 within the video frame 400 . More specifically, the player detector determines the player's location within a video frame, for example, the player 410 is determined to be located within the rectangular bounding box 450 in the video frame 400 . In one embodiment, the player detector training module 232 also determines a series of video frames of a sports video, each of which has a player detected, and the series of video frames can be selected for the key pose identifier training module 234 to train key pose identifiers, and for the meta classifier training module 236 to train meta classifiers.

The key pose identifier training module 234 trains a set of key pose identifiers, each key pose identifier is trained for identifying a corresponding key pose of a target sports action, such as a baseball swing, and the target sport action is represented by the corresponding key poses identified by the set of key pose identifiers. More specifically, for each video frame of a selected series of video frames of a sports video, each of the key pose identifiers is used to identify whether a corresponding key pose has happened and to provide a score indicating the possibility of the corresponding key pose happening within that video frame. For each of the key pose identifiers, a score sequence is generated for the selected series of video frames. The key pose identifier training module 234 then provides the meta classifier training module 236 with a set of score sequences generated by all the key pose identifiers to train a meta classifier.

In one embodiment, the key pose identifier training module 234 identifies a set of distinctive key poses that represent a target action of a sports video of a particular type and collects training data of the same type (e.g., baseball sports videos for training baseball swing key pose identifiers and golf sports videos for training golf swing key pose identifiers). For example, for baseball sports videos, the target action is baseball swing; for golf sports videos, the target video is golf swings. It is noted that a sports action for a type of sports video, e.g., a baseball swing or a golf swing, can be represented by a small set of distinctive key poses and an order of the distinctive key poses associated with that particular type sports. For example, a baseball swing action can be represented by three key poses: a pose representing begin swing, a pose representing hitting a baseball (or ball impact), and a pose representing end swing.

FIG. 3A shows an example of baseball swing action represented by three key poses performed by a player, according to one embodiment. The baseball swing action 310 illustrated in FIG. 3A includes three different key poses, which are a begin pose 312 , an impact pose 314 and an end pose 316 . The example of FIG. 3A also shows a baseball player 321 , a baseball bat 322 held by the player, a baseball 323 that is to be hit by the baseball bat, and a baseball batting tee 324 that is a sports equipment used to support a stationary baseball such that the player can strike the stationary baseball when practicing baseball swings.

The begin pose 312 illustrated in FIG. 3A shows that the player 321 lifts up the baseball bat 322 and is about to swing the baseball bat to strike the baseball 323 placed on the baseball batting tee 324 in front of the player. The impact pose 314 shows the moment that the player 321 strikes the baseball 323 using his baseball bat 322 and the baseball is traveling away from the player as it is hit by the baseball bat. The end pose 316 shows that the player 321 finishes striking the baseball 323 with a body rotation taking place in the arms, hip and spine and with one hand released from the baseball bat 322 .

Other embodiments may show additional and/or different components in FIG. 3A . For example, in an alternative embodiment to describe the baseball swing action, a baseball player may try to strike a baseball that is traveling towards the player instead of a stationary baseball being placed on a baseball batting tee set in front of the player. Additionally, additional or fewer key poses not shown in FIG. 3A may be included in the set of key poses representing a baseball swing action in other embodiments. Likewise, other types of body movement of the player to perform a baseball swing action not shown in FIG. 3A may be included in the set of key poses representing the baseball swing action.

For golf swing action in golf sports videos, a golf swing action can be represented by four key poses: a pose representing begin swing, a pose representing upward movement, a pose representing hitting a golf ball (or ball impact), and a pose representing end swing.

FIG. 3B shows an example of golf swing action represented by four key poses performed by a player, according to one embodiment. The golf swing action 350 illustrated in FIG. 3B includes four different key poses, which are a begin pose 352 , an upswing pose 354 , an impact pose 356 , and an end pose 358 . The example of FIG. 3B also shows a golf player 361 , a golf club shaft 362 that is held by the player with a golf club head 363 , and a golf ball 364 that is to be hit by the golf club shaft.

The begin pose 352 shows that the player 361 is standing in front of the golf ball 364 , gripping the golf club shaft 362 , and the golf club head 363 is placed on the ground behind the golf ball 364 . The upswing pose 354 shows that the player 361 lifts up the golf club shaft 362 with the shoulders turning back fully, and the player is about to have a quick downswing to strike the golf ball 364 . The impact pose 356 shows the moment that the player 361 strikes the golf ball 364 using the golf club shaft 362 after down swinging the golf club shaft from an upswing pose 354 , and the golf ball is travelling away from the player as it is hit by the golf club head 363 . The end pose 358 shows the player 361 finishes striking the golf ball 364 with the arms rotating and lifting back.

Referring back to FIG. 2A , for each target sports action, the key pose identifier training module 234 trains a set of key pose identifiers, each of which is trained for identifying a corresponding key pose of the target sports action. For example, for a baseball swing, the key pose identifier training module 234 trains three key pose identifiers: one for begin pose, one for impact pose and one for end pose. In one embodiment, the key pose identifier training module 234 uses both positive and negative training samples for training each key pose identifier. A positive training sample for training a key pose identifier for a target key pose is a video frame, where a player detected in a player region performs the target key pose; a negative training sample refers to a video frame, where a player detected in a player region performs an action other than the target key pose. Taking baseball swing as an example, to train a key pose identifier for a begin swing pose, a video frame showing a player performing a begin swing is considered as a positive sample, while a video frame showing a player hitting a baseball or performing an end swing is considered a negative sample.

FIG. 5 shows an example of training a set of key pose identifiers for the baseball swing action illustrated in FIG. 3A , according to one embodiment. FIG. 5 shows, for each key pose of a baseball swing action, positive training samples 510 , negative training samples 520 , and scores 530 . The key poses for baseball swing action illustrated in FIG. 5 are: a begin pose 550 , an impact pose 560 and an end pose 570 as described above.

For each of the key poses, a key pose identifier is trained using both positive and negative training samples. In the example shown in FIG. 5 , for the target begin pose 550 , a positive training sample 510 A is a begin pose performed by the player within the detected player region 540 of the video frame, and negative training samples 520 include an impact pose 520 A and an end pose 520 B performed by the player within the detected player regions 540 of the corresponding video frames. Likewise, for the target impact pose 560 , a positive training sample 510 is an impact pose and a begin pose and an end pose are negative training samples 520 . For the target end pose 570 , a positive training sample 510 is an end pose and a begin pose and an impact pose are negative training samples 520 .

After the positive training samples 510 and negative training samples 520 are determined, the key pose identifier training module 234 extracts visual features from feature extraction regions (e.g., histogram values of pixels in the regions) of both positive training samples and negative training samples. A feature extraction region is a selected region in a video frame, from which visual features are extracted to train key pose identifiers. Feature extraction regions can be generated or defined in different ways to achieve different performances of action recognition results by the trained feature models that are applied to input sports videos in real-time. In one embodiment, a feature extraction region can be a detected player region of a video frame that is determined by the trained player detector. In another embodiment, a feature extraction region can be a region manually labeled by human inspectors, for example, a bounding box of a selection region that is manually labeled in a video frame by a human. Both ways of generating feature extraction regions can be jointly used to improve sports action recognition performance. According to testing statistics for one example test, as responsive to both methods of feature extraction regions generation being combined in use to train key pose identifiers, a 2.5% increase of action recognition rate is found compared with using only the detected player region determined by the player detector, and a 4.0% increase of action recognition rate is found compared with using only human-labeled region.

Additionally, to reduce noise from the human labeling method to generate feature extraction regions, a jittering procedure can be applied to produce a better performance of training the key pose identifiers for future real-time action recognition. The jittering procedure is a data augmentation method that is used in machine learning technologies to enlarge training sample set for reducing overfitting. For example, in a jittering procedure, the center of the human labeled bounding box can be moved in a horizontal direction and/or a vertical direction by a small amount. This small amount can be a random amount that is no larger than 2% of the maximum of width of the bounding box if the center is moved in a horizontal way, or smaller than 2% of the maximum of height of the bounding box if the center is moved in a vertical way. In one embodiment, the jittering procedure applies a fixed amount of movement, e.g., 4 pixels in each horizontal and vertical direction.

To train a key pose identifier for a corresponding target pose with video frames of a training video, the key pose identifier training module 234 generates a score, e.g., a real-valued score, for each video frame based on a comparison of visual features extracted from the feature extraction region of the video frame and visual features representing the target pose. The real-valued score associated with a video frame indicates a likelihood that a player in the video frame is performing the target pose. A training video frame that is a positive training sample for the target key pose should receive a relatively higher score compared to a training video frame that is a negative training sample for the target key pose. In the example shown in FIG. 5 , for the target begin pose 550 , the positive sample receives a score higher than 50 and the two negative samples receive scores lower than 0. A threshold value may be determined for each key pose identifier to distinguish between a positive sample from a negative sample when the key pose identifier is applied for future training and real-time sports action recognition. The trained key pose identifier for a target key pose generates a real-valued score for each video frame of an input video, where the score indicates how likely a player in the video frame is performing the target key pose.

FIG. 6 shows a frame-by-frame analysis of score sequences generated by the set of trained key pose identifiers for the three key poses representing a baseball swing action illustrated in FIG. 3A , according to one embodiment. In the embodiment of FIG. 6 , a sequence of continuous video frames 610 of a baseball sports video and three score sequences 650 A, 650 B and 650 C, corresponding to the three key pose identifiers for a baseball swing action are shown. The video frame 610 A represents a video frame showing a pose that best matches a begin pose for baseball swing action because it has the highest score among other video frames shown in the score sequence 650 A for the begin pose, the video frame 610 B represents a video frame showing a pose that best matches an impact pose for baseball swing action because it has the highest score among other video frames shown in the score sequence 650 B for the impact pose, and the video frame 610 C represents a video frame showing a pose that best matches an end pose for baseball swing action because it has the highest score among other video frames shown in the score sequence 650 C for the end pose.

Each of the score sequences 650 A-C further includes a frame timeline 651 as a horizontal axis, and a score axis 652 as a vertical axis. The frame timeline 651 shows, for each frame, a frame label and a relative location in terms of timestamp of the frame among the whole sequence of video frames 610 . The score axis 652 shows, for each frame, a score generated by a trained key pose identifier, which indicates how well the pose shown by that frame matches the key pose identified by the key pose identifier, and which further indicates how likely the key pose has happened within that frame. The example illustrated in FIG. 6 also shows that, in score sequence 650 A for begin pose of a baseball swing action, the video frames 610 that are located on around the first half of the frame timeline 651 have relatively higher scores compared with video frames that are located on the second half of the frame timeline because it is highly probable that the being pose or the poses similar to that has happened before the other two key poses, impact pose and end pose.

Similarly, score sequence 650 B for impact pose shows, the video frames 610 that are located on around middle part of the frame timeline 651 have relatively higher scores compared with video frames that are located on other parts of the frame timeline, because it is highly probable that the impact pose or the poses similar to that happen after the begin pose and before the end pose.

Score sequence 650 C for end pose shows, the video frames 610 located around the latter part of the frame timeline 651 have relatively higher scores compared with video frames that are located on other parts of the frame timeline because the end pose or the poses similar are most likely to happen after both the begin pose and the end pose.

Referring back to FIG. 2A , the meta classifier module 236 trains a meta classifier to analyze the score sequences generated by the trained key pose identifiers to determine whether a target sports action by a player has happened within a selected series of video frames and to locate the sports action on the timeline of the video frames if the target sports action has happened. In one embodiment, the meta classifier module 236 trains a meta classifier for a target sports action by applying a time window of a predefined width over a real-valued vector, that includes the score sequences for all key poses identified for the target sport action, and slides the time window along an axis of the score sequences stacked together. A portion of the stacked score sequences, that includes the highest score for each key pose within the same time window is identified as the portion of the corresponding sports video, where the target sport action occurred.

In one embodiment, the meta classifier being trained is a support vector machine (SVM) machine learning model, which produces a score measuring how likely a real-valued score associated with the key poses of a sports action matches a sports action. The meta classifier module 236 can also use a validation sample set to determine a threshold to further classify the real-valued score result generated by the SVM to a “positive” result or a “negative” result. A positive result indicates a sports action has happened and a negative result indicates a sports action hasn't happened.

FIG. 7A shows training a meta classifier by applying a time window to the score sequences generated by key pose identifiers for a baseball swing action illustrated in FIG. 6 , according to one embodiment. The embodiment illustrated in FIG. 7A also shows that the three score sequences for the three key poses for baseball swing sports action are aligned according to the frame timeline with time windows 710 applied to each of the score sequences. Time windows 710 (e.g., 710 A, 710 B and 710 C) are used to generate training samples for training the meta classifier. In one embodiment, a training sample for a meta classifier is a real-valued vector of sequence scores for all key poses within a same time window, and the real-valued vector reflects a part of the score sequences of each key pose within a same time window. In other words, for the meta classifier, a positive training sample refers to a training sample, where its corresponding time window covers the entire period of the sports action, and the highest scores generated by each of the key pose identifiers are all included in the selected time window. As more fully described below in FIG. 7B , the part of the score sequences for each key pose are placed in the real-valued vector in an order that reflects an order the player performs the whole sports action, for example, in an order of the begin pose, the impact pose and the end pose for a baseball swing action.

FIG. 7B shows example training samples of the meta classifier shown in FIG. 7A , according to one embodiment. The example illustrated in FIG. 7B shows a training sample 720 A that corresponds to the time window 710 A in FIG. 7A , a training sample 720 B corresponds to the time window 710 B in FIG. 7A , and a training sample 720 C corresponds to a time window 710 C in FIG. 7A . For example, training sample 720 B shown in FIG. 7B is a positive training sample, which covers the three highest scores generated by all the three key pose identifiers, and which further indicates that the entire period of the baseball swing action is covered in the selected time window 710 B. In contrast, training sample 720 A and training sample 720 C are negative training samples since not all three highest scores generated by all the three key pose identifiers are covered in the corresponding time windows 710 B and 710 C, which further indicates the entire period of the sports action is nether covered by time window 710 A nor time window 710 C.

FIG. 8 shows a flow chart illustrating a process of training feature models illustrated in FIG. 4 - FIG. 7B , according to one embodiment. Initially, the training module 230 receives 810 training videos for training a player detector, a set of key pose identifiers and a meta classifier for a specific sports action associated with a specific sports type. A training video may include multiple video frames that captures a player's actions associated with the specific sports type, e.g., playing baseball. The training module 230 trains 820 a player detector to detect a player within a video frame and to determine a player region within which the player is located in the video frame. The training module 230 trains 830 a set of key pose identifiers to identify key poses for the specific sports action and to produce scores indicating how well a pose recorded in a video frame matches each of the key poses. The training module 230 trains 840 a meta classifier to analyze the scores produced by the trained key pose identifiers and to produce a final result about how likely a sports action has happened during a video and to locate the sports action with respect to the video frame timeline of the video if the sports action has happened. The trained models, including the player detector, the set of key pose identifiers, and the meta classifier, are stored 850 for future real-time action recognition for input sports videos. In alternative embodiments not shown in FIG. 8 , different sets of feature models can be trained for different sports actions associated with different sports types.

Turning back to FIG. 2B , FIG. 2B shows a block diagram of a recognition application module 240 for applying trained feature models to an input video stored in an input video database illustrated in FIG. 1 . In other embodiments, the recognition application module 240 is located at the client device 110 , and function as the software application module 112 . The input video database 214 provides the recognition application module 240 with input sports videos for real-time action recognition. In one embodiment, the input video database 214 provides the recognition application module 240 with a complete input sports video after all the video frames of the input sports videos are received from a user. In another embodiment, as each video frame of the input sports video is received from the user, the input video database 214 provides the video frame for the recognition application module 240 . For example, while the user is recording a sports video with a client device 110 when the user or another different user is playing a certain sports, the client device 110 uploads each frame to the input video database 214 as each frame of the video is captured by a camera embedded with the client device 110 , and at the same time, each frame received by the input video database is provided to the recognition application module 240 for real-time action recognition. The action recognition module 240 may also provide the user with real-time recognition result as each frame is received and analyzed by the feature models.

The recognition application module 240 illustrated in FIG. 2B includes a player detection module 242 , a key pose identification module 244 , and a meta classification module 246 . After the recognition application module 240 receives an input sports video from the input video database 214 for action recognition for a specific sports action associated with a specific sports type, the recognition application module 240 selects from the model database 220 a set of trained feature models corresponding to the specific sports action associated with the specific sport types and applies the selected feature models to the received input sports video. The selected feature models include a player detector, a set of key pose identifiers, and a meta classifier for the specific sports action.

The player detection module 242 detects, within each video frame of an input video, a player region within which a player is detected using the trained player detector. The player detection module 242 can also select one or more video frames from the input video based on the player detection, where the selected video frames are used by the key pose identification module 244 and the meta classification module 246 .

The key pose identification module 244 uses a set of trained key poses identifiers associated with the specific sports action to extract features from a feature extraction region within each video frame of the input video. The key pose identification module 244 applies the trained key pose identifiers to generate a score sequence for each key pose of the set of key poses representing the specific sport action.

The meta classification module 246 applies the trained meta classifier to the score sequences generated by the key pose identification module 244 to determine whether the specific sports action has occurred and where (in terms of timestamps of the video frames) the action has occurred. The recognition application module 240 generates action recognition result based on the applications of the trained feature models and presents the result to the user of the client device 110 .

In one embodiment, as described above, the recognition application module 240 applies trained feature models and generates action recognition result in an efficient way that as each video frame of the input sports video is continuously received from the input video database 214 , the recognition application module 240 can detect whether a player is within the video frame and generates scores by the different key pose identifiers for that frame, and the action recognition result is also updated by the trained meta classifier. The real-time action recognition result is provided by the recognition application module 240 to the user as video frames of the input sports video are continuously received and analyzed.

FIG. 9 shows a flow chart illustrating a process of real-time action recognition by the recognition application module 240 using the trained feature models shown in FIG. 8 , according to one embodiment. In the embodiment shown in FIG. 9 , the recognition application module 240 receives an input sports video 910 from a user requesting action recognition for the input sports video. A sports type of the input sports video is detected 920 and a set of trained feature models are selected 930 for the received input sports video based on the detected sports type, e.g., a baseball swing action for a baseball sports video. The set of selected feature models include a player detector, a set of key pose identifiers and a meta classifier for the sports type of the input sports video.

The recognition application module 240 applies the set of selected feature models to the input sports video. At first, the recognition application module 240 applies the trained player detector to detect 940 a player recorded in each video frame of the input sports video. The recognition application module 240 detects a player region, where a player is detected in a video frame and from which features of the video frame are extracted for later use by trained key pose identifiers and meta classifier. The recognition application module 240 applies the set of selected trained key pose identifiers to identify 950 the key poses performed by the detected player within the detected player region. More specifically, the features extracted in 940 may be fed into the key pose identifiers to generate score sequences for each key pose of the sports action, indicating how well a pose in a video frame of the input sports video matches each of the key poses of the sports action. After the score sequences are generated by the key pose identifiers, the recognition application module 240 applies the trained meta classifier to determine 960 whether a sports action has happened in the input sports video. The meta classifier also locates the sports action in the input sports video using a time window of a predefined width.

The recognition application module 240 generates 970 an action recognition report showing whether the sports action has happened in the input sports video and location of the sports action. The recognition report is presented 980 to the user to consume.

In one embodiment, the action recognition report is presented to the user in real-time as each video frame of the input sports video is received. As one example, after the input sports video is uploaded to the system, the input sports video with a complete series of video frames may be stored in the system and then analyzed by applying the trained feature models. As another example, as the system receives a new video frame, the trained key pose identifiers produce scores for that video frame, and the trained meta classifier updates the time window selected for the real-valued vector including the real-valued scores for each key pose and updates the recognition result with the newly added scores of this frame. In this case, the user can receive a real-time recognition result from the system as each frame of the sports video is received by the system.

General

The foregoing description of the embodiments of the invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments of the invention may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments of the invention is intended to be illustrative, but not limiting, of the scope of the invention, which is set forth in the following claims.