System and method for parallel video processing in multicore devices

1. Field of the Invention

The present invention relates in general to scalable video processing and, more specifically, to a system and method for efficient parallel processing of video data.

2. Description of the Related Art

The Advanced Video Coding (AVC) standard, Part 10 of MPEG4 (Motion Picture Experts Group), otherwise known as H.264, includes advanced compression techniques that were developed to enable transmission of video signals at a lower bit rate or to enable improved video quality at a given transmission rate. The processing of video signals in accordance with the H.264 standard, or other standards, requires a very large amount of computational resources. For example, to encode a H.264 high-definition (HD) data stream for 720p@30 fps video in real-time, it is usually necessary to use a multi-core processor having clock speeds greater than 1 GHz. One of the most challenging aspects of digital signal processing relates to scenarios for dispatching tasks among multiple digital signal (DSP) processing cores. There is a need, therefore, for a system and method for balancing the processing load among multiple processing cores in a multi-core device.

The present invention may be understood, and its numerous objects, features and advantages obtained, when the following detailed description is considered in conjunction with the following drawings, in which:

FIG. 1 is an illustration of a codec comprising a multi-core processor for processing video signals;

FIG. 2 is an illustration of a plurality of processing cores configured in a master-slave relationship;

FIG. 3 is a flowchart illustration of a process flow for processing video data in accordance with embodiments of the invention;

FIG. 4 is a flowchart illustration of a process flow for determining sequential processing tasks in accordance with embodiments of the invention; and

FIG. 5 is a flowchart illustration of a process flow for updating the state of processing cores in accordance with embodiments of the invention.

Embodiments of systems and methods are described herein for providing load balancing among multiple processing cores to perform parallel processing of video data streams. Various illustrative embodiments of the present invention will now be described in detail with reference to the accompanying figures. While various details are set forth in the following description, it will be appreciated that the present invention may be practiced without these specific details, and that numerous implementation-specific decisions may be made to the invention described herein to achieve the device designer's specific goals, such as compliance with process technology or design-related constraints, which will vary from one implementation to another. While such a development effort might be complex and time-consuming, it would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure. For example, selected aspects are depicted with reference to simplified drawings in order to avoid limiting or obscuring the present invention. Such descriptions and representations are used by those skilled in the art to describe and convey the substance of their work to others skilled in the art. Various illustrative embodiments of the present invention will now be described in detail with reference to the figures.

FIG. 1 is an illustration of a codec 100 that is operable to receive a video data stream from a video capture device 102 and to process the video data for use by a display 104 . Although the embodiment illustrated in FIG. 1 shows data being captured by a video capture device, those of skill in the art will understand that there are numerous other ways of providing video data to a multi-core processor. For example, in infrastructure applications, the video is received compressed over a network, and is decoded and re-encoded in the system. In this case, video may arrive on various interfaces (PCI, SRIO, etc.). An encoder 106 comprises a multi-core processor 108 that further comprises a plurality of data processing cores 110 a , 110 b , . . . , 110 n (hereinafter sometimes referred to as “cores”). A memory controller 112 is operable to process memory requests to access memory 114 , which includes M2 memory 116 and M3 memory 118 . As will be discussed in greater detail hereinbelow, the various cores 110 a , 110 b , . . . , 110 n may be configured in a master-slave relationship wherein the “master” processing core may have access to shared M2 memory 120 and private M2 memory 122 . The “slave” cores may have access to shared memory, their respective private M2 memory, and to their message queue (which is shared with the master core). The codec 100 may be coupled to a variety of other components via a system bus 124 .

When the video processing system is initiated, control software is executed by the multi-core processor 108 to establish the master-slave relationship between the various cores 110 a , 102 b , . . . , 102 n , using techniques known to those of skill in the art. The master core is provided with information regarding the number of slave cores and is provided with handles to message queues.

FIG. 2 is an illustration of a plurality of processing cores configured in a master-slave relationship in accordance with embodiments of the invention. For purposes of discussion, core 110 a is designated as the master, and cores 110 b and 110 c are designated as slave cores. The master core 110 a is allocated its private memory 122 a and all shared memory. The slave cores 110 b and 110 c are allocated their respective private memories 122 b and 122 c.

Most current implementations of H.264 encode images in macroblock scan order, i.e., from left to right, and from top to bottom. After all macroblocks have been encoded, the encoder will perform the loop filter on the entire frame. In general, the encoder encodes the entire frame into one slice. In embodiments of the invention, however, the encoder is instructed to generate multiple slices by limiting the number of bits per slice or the number of macroblocks per slice. To run efficiently, the codec strives to access as much data as possible in M2 memory. Large data that is located in M3 and DDR is DMAed to M2. Since the encoder controls the DMA via the DMA API, the multicore implementation allocates a DMA channel for each core. In embodiments where DMA is used, the codec maintains cache coherency using software cache commands.

The partitioning of a frame into slices is done in a manner to generate as few slices as possible to maintain high quality. In the various embodiments of the invention, all decisions are made only by the master core. Other slave cores receive requests from the master core and carry them out. Slaves cores periodically report their status to the master core, as discussed in greater detail hereinbelow. To partition a frame into slices, the master core sends a message to each slave core with the address of the first macroblock in the slice that the respective cores are asked to encode. The master core is also operable to send a message with the address of the last macroblock to be encoded in that slice; however, this message can be sent later based on regular progress sent by the slave cores. Data partitioning is the method of partitioning or processing data blocks in parallel. Source, reconstructed and reference image data is placed in shared memory. In various embodiments of the invention, communication between the cores is implemented using half-duplex channels. Each slave core has two channels: an ingress port and an egress port. The ports may be implemented using a simple FIFO structure shared in M2 memory. All accesses to the ports are non-cacheable. Therefore, the communication messages are generally short (64 bits).

To minimize the impact on data quality, the slice size is determined dynamically by the master core. The master core monitors the process in progress of the slave cores and sends a message to the relevant slave core with a slice size. Embodiments of the invention, are therefore, operable to provide slice-based load balancing of data processing between the respective processing cores on an intra-frame basis, rather than between multiple frames.

The master core 110 a is operable to dispatch two types of tasks, encoding and deblocking, to slaves (and to itself). The encoding task is to start encoding at a given starting macroblock address (MBAddr). The master does not have to provide an “end MBAddr” when the encoding task starts. If an end MBAddr is provided, the slave will start a new slice at MBAddr and encode until end MBaddr. If ‘end MBAddr’ was not provided by the master, the slave continues until it finishes the frame or receives an end MBAddr by the master. An advantage of postponing the selection of an ‘end MBAddr’ is better load balancing. In an alternate embodiment, even after sending the endMBAddr, the master may decide to update the endMBAddr with a new value. This may be done if one of the cores is ‘slow’ and has a large portion of a slice remaining while the other cores are idle. In this case, the master will update the endMBaddr with a new value and dispatch a new ‘encoding task’ to the idle slaves. This further enhances the load balancing.

The deblocking task is a task to deblock from startMBAddr1 to endMBAddr2. The master core 110 a keeps track of which MBs have been encoded and are ready for deblocking. If there are enough MBs to be deblocked, it will start the ‘deblocking task.’ This allows deblocking to be done as soon as possible. This is very important for load balancing. Slaves send periodic notification on their progress, i.e., which MBs they have finished encoding. The master 110 a uses this information in determining what subsequent task to send to slave cores.

The load balancing does not have to be equal. For example, in an embodiment of the invention comprising three slave cores, two of the cores may 100% utilized, while the remaining core may be used only 50%. The system and method provided by embodiments of the invention is very flexible because all the required data is shared between slaves, i.e. reference frame, deblocking parameters, etc.

FIG. 3 is a flowchart illustration of a process flow for processing video data in accordance with embodiments of the invention. In step 302 , processing of a frame is initiated. In step 304 , the master core 110 a conducts a test to determine whether there are any idle slave cores. If the result of the test performed in step 304 indicates that there are idle slave cores, processing proceeds to step 306 wherein the master core 110 a determines the next task for an idle slave core. In step 308 , the master sends a message to the respective idle slave core indicating the task that is to be performed by that slave core.

If, however, the result of the test conducted in step 304 indicates that no slaves are idle, processing proceeds so step 3 10 , wherein a test is conducted to determine whether any messages have been received. If the result of the test conducted in step 310 indicates that there are incoming messages, processing proceeds to step 312 wherein the processing states of the slaves is updated. If, however, the result of the test conducted in step 310 indicates that no messages were received, processing proceeds to step 3 14 , wherein a test is conducted to determine whether there is a need to update the slaves.

If the result of the test conducted in step 314 indicates that the slaves do need to be updated, processing proceeds to step 316 , wherein the update is sent to the slaves and processing again returns to step 310 ; if, however, the result of the test in step 314 indicates that the slaves do not need to be updated, processing returns immediately to step 3 10 . If the test in step 310 indicates that there are no incoming messages, processing proceeds to step 318 wherein a test is conducted to determine whether all tasks have been completed. If the result of the test in step 318 indicates that all processing tasks have not been completed, processing returns to step 304 and the steps discussed above are repeated. If, however the test in step 318 indicates that all processing steps have been completed, processing proceeds to step 320 wherein the end of frame processing is ended.

FIG. 4 is a flowchart of the processing steps for determining the next task to be processed as indicated in step 306 of FIG. 3 . In step 402 , a test is conducted to determine whether there is additional processing needed to finish deblocking a previous frame. If the result of the test conducted in step 402 indicates that it is necessary to finish deblocking a previous frame, processing proceeds to step 404 wherein the deblocking of the previous frame is completed. Processing then proceeds to step 406 , wherein the status of the cores is updated. If the result of the test conducted in step 402 indicates that it is not necessary to finish deblocking a previous frame, processing proceeds to step 408 wherein a test is conducted to determine whether the respective slave core is able to deblock part of the current frame. If the test conducted in step 408 indicates that the respective slave core is able to deblock part of the current frame, processing proceeds to step 410 , wherein the current frame is deblocked. If, however, the result of the test conducted in step 408 indicates that it is not possible to deblock part of the current frame, processing proceeds to step 412 wherein a test is conducted to determine if there is a free slice to be encoded. If the result of the test conducted in step 412 indicates that there is a slice to be encoded, processing proceeds to step 414 wherein the available slice is encoded. If, however, the test conducted in step 412 indicates that there is no slice available for encoding, processing proceeds to step 416 wherein the encoding task is split between cores, followed by step 406 , wherein the status of the cores is updated in step 406 .

FIG. 5 is a flowchart illustration of a process flow or updating the state of processing cores in accordance with embodiments of the invention. In step 502 , the update state is determined. In step 504 , the core status is updated. In step 506 , a test is conducted to determine whether more than half of a frame has been deblocked. If the result of the test conducted in step 506 indicates that more than half of a frame has been deblocked, processing proceeds to step 508 wherein deblocking of the current frame is terminated. If, however, the test conducted in step 506 indicates that less than half of a frame has been deblocked, processing proceeds to step 510 , wherein processing proceeds as discussed hereinabove with regard to FIG. 3 .

From the foregoing, it will be appreciated that embodiments of the invention provide a method for parallel processing of video signals. Embodiments of the method comprise the following steps: using a multi-core processor to establish a master-slave relationship between a first processing core and a plurality of individual processing cores; using a shared memory to store data and control messages; associating a plurality of individual private memories with each of the individual processing cores; and using control logic to establish a master-slave protocol for using the plurality of individual cores to process video data, wherein one of the individual processing cores is designated as a master and a predetermined number of other individual cores are designated as slave processors and wherein the master processing core is operable to balance the video data processing load among the individual slave processing cores. The individual slaves are operable to generate data processing progress reports and to send the data processing progress reports to the master processing core. In various embodiments, the master core is operable to partition a video data frame and is further operable to establish messaging queues between the master processing core and each of the slave cores. The master processing core is further operable to assign partition slices to predetermined individual slave processing cores and to assign an encoding or a deblocking processing request to the individual slave processing cores. In some embodiments the master processing unit is allocated its private memory and all shared memory, and the individual slave processing cores are allocated their respective individual private memories.

Other embodiments of the invention provide a system for parallel processing of video signals. The system is broadly comprised of: a multicore processor comprising a plurality of individual processing cores; a shared memory; a plurality of individual private memories associated with each of the individual processing cores; and control logic to operable to establish a master-slave protocol for using the plurality of individual cores to process video data, wherein one of the individual processing cores is designated as a master and a predetermined number of other individual cores are designated as slave processors and wherein the master processing core is operable to balance the video data processing load among the individual slave processing cores.

Although the described exemplary embodiments disclosed herein are directed to various examples of an improved system and method for providing load balancing between multiple cores used for parallel processing of video signals, the present invention is not necessarily limited to the example embodiments. Thus, the particular embodiments disclosed above are illustrative only and should not be taken as limitations upon the present invention, as the invention may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. Accordingly, the foregoing description is not intended to limit the invention to the particular form set forth, but on the contrary, is intended to cover such alternatives, modifications and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims so that those skilled in the art should understand that they can make various changes, substitutions and alterations without departing from the spirit and scope of the invention in its broadest form.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or element of any or all the claims. As used herein, the terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.