Buffering apparatus for buffering multi-partition video/image bitstream and related method thereof

This application claims the benefit of U.S. provisional application No. 61/433,272, filed on Jan. 17, 2011 and incorporated herein by reference.

The disclosed embodiments of the present invention relate to video/image processing, and more particularly, to a buffering apparatus for buffering a multi-partition video/image bitstream and related method thereof.

VP8 is an open video compression format released by Google®. Like many modern video compression schemes, VP8 is based on decomposition of frames into square subblocks of pixels, prediction of such subblocks using previously constructed blocks, and adjustment of such predictions (as well as synthesis of unpredicted blocks) using a discrete cosine transform (DCT). In one special case, however, VP8 uses a Walsh-Hadamard transform (WHT) instead of the commonly used DCT.

WebP is an image format developed by Google® according to VP8. Specifically, WebP is based on VP8's intra-frame coding and uses a container based on resource interchange file format (RIFF). Besides, WebP is announced to be a new open standard that provides lossy compression for photographic images. In a large scale study of 900,000 web images, WebP images are found 39.8% smaller than joint Photographic Experts Group (JPEG) images of similar quality. Webmasters, web developers and browser developers therefore can use the WebP format to create smaller, better looking images that can help to improve user's web surfing.

In accordance with the VP8/WebP specification, the input to a VP8/WebP decoder is a sequence of compressed frames whose order matches their order in time. Besides, every compressed frame has multiple partitions included therein. As the VP8/WebP bitstream is configured to transmit compressed frames each having a plurality of partitions included therein, how to efficiently buffer and decode each compressed frame of a multi-partition VP8/WebP bitstream becomes an important issue in this technical field.

In accordance with exemplary embodiments of the present invention, a buffering apparatus for buffering a multi-partition video/image bitstream and related method thereof are proposed to solve the above-mentioned problem.

According to a first aspect of the present invention, a buffering apparatus for buffering a multi-partition video/image bitstream is disclosed. The multi-partition video/image bitstream transmits a plurality of compressed frames each having a plurality of partitions. The buffering apparatus includes a first bitstream buffer and a second bitstream buffer. The first bitstream buffer is arranged to buffer data of a first partition of the partitions of a specific compressed frame. The second bitstream buffer is arranged to buffer data of a second partition of the partitions of the specific compressed frame.

According to a second aspect of the present invention, a method for buffering a multi-partition video/image bitstream is disclosed. The multi-partition video/image bitstream transmits a plurality of compressed frames each having a plurality of partitions. The method includes: utilizing a first bitstream buffer to buffer data of a first partition of the partitions of a specific compressed frame; and utilizing a second bitstream buffer to buffer data of a second partition of the partitions of the specific compressed frame.

According to a third aspect of the present invention, a buffering apparatus for buffering a multi-partition video/image bitstream is disclosed. The multi-partition video/image bitstream transmits a plurality of compressed frames each having a plurality of partitions. The buffering apparatus includes a bitstream buffer and a prefetch circuit. The bitstream buffer is arranged to buffer data of the partitions of a specific compressed frame. The prefetch circuit includes a storage unit and a prefetch unit. The prefetch unit is arranged to prefetch data of a first partition of the partitions of the specific compressed frame from the bitstream buffer and store the prefetched data into the storage unit while a decoding apparatus is decoding a second partition of the partitions of the specific compressed frame, wherein the prefetched data stored in the prefetch unit is read by the decoding apparatus when the decoding apparatus starts decoding the second partition.

These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.

FIG. 1 is a diagram illustrating multiple partitions in a compressed frame to be processed by a proposed buffering apparatus of the present invention.

FIG. 2 is a diagram illustrating how transform coefficients in a compressed frame are packed into four partitions.

FIG. 3 is a diagram illustrating a video/image decoding system according to a first exemplary embodiment of the present invention.

FIG. 4 is a diagram illustrating a video/image decoding system according to a second exemplary embodiment of the present invention.

FIG. 5 is a diagram illustrating an alternative design of a buffering apparatus.

FIG. 6 is a diagram illustrating a video/image decoding system according to a third exemplary embodiment of the present invention.

FIG. 7 is a diagram illustrating a video/image decoding system according to a fourth exemplary embodiment of the present invention.

Certain terms are used throughout the description and following claims to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ”. Also, the term “couple” is intended to mean either an indirect or direct electrical connection. Accordingly, if one device is electrically connected to another device, that connection may be through a direct electrical connection, or through an indirect electrical connection via other devices and connections.

In accordance with the VP8/WebP specification, the input to a VP8/WebP decoder is a sequence of compressed frames each having 2-9 partitions. These partitions begin and end on byte boundaries. The leading partition of a compressed frame (i.e., the partition that is transmitted first) has two subsections: header information that applies to the compressed frame as a whole and per-macroblock prediction information that includes prediction information of each macroblock in the compressed frame. The remaining partitions (1, 2, 4 or 8) contain transform coefficients (e.g., DCT/WHT coefficients) of the residue signal.

Please refer to FIG. 1 , which is a diagram illustrating multiple partitions in a compressed frame to be processed by a proposed buffering apparatus of the present invention. The compressed frame 100 is transmitted via a VP8/WebP bitstream, and therefore contains N partitions 102 _ 1 - 102 _N which are sequentially transmitted. That is, the partition 102 _ 1 is the leading partition of the compressed frame 100 , and the partition 102 _N is the last partition of the compressed frame 100 . The partition 102 _ 1 includes header information applied to the whole frame 100 , and also includes the prediction information for each MB in the same frame 100 . Regarding each of the remaining partitions 102 _ 2 - 102 _N following the partition 102 _ 1 , it includes transforming coefficients of the residue, such as DCT coefficients or WHT coefficients. When there is more than one partition for the transform coefficients, the sizes of the partitions—except the last partition—in bytes are also present in the bitstream right after the above-mentioned leading partition 102 _ 1 . Each of the partition sizes is recorded by a 3-byte data item. For example, a 3-byte partition size PS 2 shown in FIG. 1 indicates the size of the partition 102 _ 2 , and a 3-byte partition size PS 3 shown in FIG. 1 indicates the size of the partition 102 _ 3 . These partition sizes provide the decoding apparatus direct access to all DCT/WHT coefficient partitions, which may enable parallel processing of the coefficients in a decoding apparatus.

However, VP8/WebP packs the DCT/WHT coefficients from macroblock (MB) rows into separate partitions. Please refer to FIG. 2 , which is a diagram illustrating how transform coefficients in a compressed frame are packed into four partitions. As shown in the figure, there are many MB rows MB_ 0 -MB_ 15 in the exemplary compressed frame 200 . The transform coefficients of the MB rows MB_ 0 , MB_ 4 , MB_ 8 , and MB_ 12 are packed in a partition Partition_ 1 , the transform coefficients of the MB rows MB_ 1 , MB_ 5 , MB_ 9 , and MB_ 13 are packed in a partition Partition_ 2 , the transform coefficients of the MB rows MB_ 2 , MB_ 6 , MB_ 10 , and MB_ 14 are packed in a partition Partition_ 3 , and the transform coefficients of the MB rows MB_ 3 , MB_ 7 , MB_ 11 , and MB_ 15 are packed in a partition Partition_ 4 . Therefore, as successive MB rows are not packed in the same partition, decoding of one MB may require data read from different partitions. In a case where the employed bitstream buffer does not have enough storage space for buffering data of the whole compressed frame, certain data requested by the decoder may not be immediately available in the bitstream buffer. As a result, the bitstream buffer may have to release the buffered data of one partition and then load the requested data in another partition.

However, switching between different partitions would lower the decoding speed due to the time period needed for loading the requested data. Thus, to improve the efficiency of decoding each compressed frame of a multi-partition VP8/WebP bitstream, the present invention therefore proposes an innovative buffer maintenance and control mechanism. Further details are described as below.

FIG. 3 is a diagram illustrating a video/image decoding system according to a first exemplary embodiment of the present invention. The video/image decoding system 300 includes a buffering apparatus 302 and a decoding apparatus 304 . The buffering apparatus 302 is for buffering a multi-partition video/image bitstream BS_IN which transmits a plurality of compressed frames each having a plurality of partitions. In this exemplary embodiment, the buffering apparatus 302 includes a plurality of bitstream buffers 312 _ 1 - 312 _N, a buffer controller 314 , and a multiplexer (MUX) 315 . The bitstream buffers 312 _ 1 - 312 _N are arranged to buffer data of the partitions 102 _ 1 - 102 _N shown in FIG. 1 , respectively. The bitstream data is stored into the bitstream buffers 312 _ 1 - 312 _N according to write pointers WPTR_ 1 -WPTR_N, and the bitstream data is read from the bitstream buffers 312 _ 1 - 312 _N according to read pointers RPTR_ 1 -RPTR_N. More specifically, the write pointer WPTR_ 1 controls the write address at which the header information/per-macroblock prediction information is stored into the bitstream buffer 312 _ 1 , and the read pointer RPTR_ 1 controls the read address at which the buffered header information/per-macroblock prediction information of the partition 102 _ 1 is read from the bitstream buffer 312 _ 1 ; the write pointer WPTR_ 2 controls the write address at which the transform coefficient (e.g., a DCT/WHT coefficient) of the partition 102 _ 2 is stored into the bitstream buffer 312 _ 2 , and the read pointer RPTR_ 2 controls the read address at which the buffered transform coefficient is read from the bitstream buffer 312 _ 2 ; and the write pointer WPTR_N controls the write address at which the transform coefficient (e.g., a DCT/WHT coefficient) of the partition 102 _N is stored into the bitstream buffer 312 _N, and the read pointer RPTR_N controls the read address at which the buffered transform coefficient is read from the bitstream buffer 312 _N.

In this exemplary embodiment, the bitstream buffers 312 _ 1 - 312 _N may be continuous/discontinuous ring buffers dedicated to buffering data of the partitions 102 _ 1 - 102 _N, respectively, and data is allowed to be fed into a ring buffer when the ring buffer has free storage space (i.e., the write pointer does not catch up the read pointer yet). In one exemplary design, the buffer controller 314 is arranged to monitor the write pointers WPTR_ 1 -WPTR_N and the read pointers RPTR_ 1 -RPTR_N of all bitstream buffers 312 _ 1 - 312 _N at the same time. Therefore, when the buffer controller 314 detects any bitstream buffer that has free space for receiving more data that is not buffered yet, the buffer controller 314 adjusts the corresponding write pointer and allows data that is not buffered yet to be filled into the bitstream buffer.

In another exemplary design, the buffer controller 314 is arranged to only monitor the write pointer and the read pointer of a currently used bitstream buffer in which the buffered data is being decoded. Therefore, when the buffer controller 314 detects that the currently used bitstream buffer has free space for receiving more data that is not buffered yet, the buffer controller 314 adjusts the corresponding write pointer and allows data that is not buffered yet to be filled into the currently used bitstream buffer.

In above-mentioned exemplary designs, a read pointer and a write pointer of a specific bitstream buffer are used to determine/detect whether the specific bitstream buffer is full or empty or to determine/detect how much free storage space remained in the specific bitstream buffer. However, this is for illustrative purposes only, and is not meant to be a limitation of the present invention. Using other means capable of determining/detecting whether the specific bitstream buffer is full or empty or to determining/detecting how much free storage space remained in the specific bitstream buffer is also feasible.

Regarding the decoding apparatus 304 , it includes a plurality of bitstream direct memory access (DMA) controllers 316 , 317 , and a plurality of barrel shifters 318 , 319 . The bitstream DMA controller 316 is arranged to transmit buffered bitstream data (i.e., header information/per-macroblock prediction information) from the bitstream buffer 312 _ 1 to the barrel shifter 318 via DMA manner, and the barrel shifter 318 is arranged to parse the bitstream data provided by the preceding bitstream DMA controller 316 . The bitstream DMA controller 316 is arranged to transmit buffered bitstream data (i.e., coefficient data) from one of the bitstream buffers 312 _ 2 - 312 _N to the barrel shifter 319 via DMA manner, and the barrel shifter 319 is arranged to parse the bitstream data provided by the preceding bitstream DMA controller 317 . Therefore, the decoding apparatus 304 shown in FIG. 3 is capable of decoding two partitions simultaneously.

As only one of the coefficient partitions (i.e., partitions 102 _ 2 - 102 _N) is allowed to be decoded by the decoding apparatus 302 , the buffering apparatus 302 therefore uses the multiplexer 315 to select one of the bitstream buffers 312 _ 2 - 312 _N as a data source to be accessed by the bitstream DMA controller 317 . For example, when the coefficient data of the partition 102 _ 2 is required to be processed at a first time point, the multiplexer 315 couples the bitstream buffer 312 _ 2 to the bitstream DMA controller 317 . However, when the coefficient data of the partition 102 _ 3 is required to be processed at a second time point, the multiplexer 315 couples the bitstream buffer 312 _ 3 to the bitstream DMA controller 317 . As the requested coefficient data may be guaranteed to be available in the bitstream buffers (e.g., ring buffers) 312 _ 2 - 312 _N if each of the bitstream buffer 312 _ 2 - 312 _N is properly controlled to buffer data to be decoded when there is free storage space, the buffering apparatus 302 is not required to release buffered data of one partition and load requested data in another partition. To put it another way, the decoding performance may be greatly improved due to the buffering mechanism which employs multiple bitstream buffers dedicated to buffering partial data of respective partitions, thus avoiding frequent releasing of buffered data and loading of requested data.

Please note that the circuit configuration shown in FIG. 3 merely serves as one exemplary embodiment of the present invention. Any alternative design that does not depart from the spirit of the present invention falls within the scope of the present invention. For example, the spirit of the present invention is obeyed as long as the buffering apparatus includes multiple bitstream buffers arranged to buffer data of different partitions in the same compressed frame, respectively. For example, in one alternative design, the buffering apparatus 302 is modified to include the bitstream buffer 312 _ 1 used for buffering bitstream data of the partition 102 _ 1 , at least one of the bitstream buffers 312 _ 2 - 312 _N used for buffering at least one of the partitions 102 _ 2 - 102 _N, and a single bitstream buffer used for buffering bitstream data of the rest of the partitions 102 _ 2 - 102 _N. In another alternative design, the buffering apparatus 302 is modified to include at least two of the bitstream buffers 312 _ 2 - 312 _N used for buffering at least two of the partitions 102 _ 2 - 102 _N, and a single bitstream buffer used for buffering bitstream data of the partition 102 _ 1 and bitstream data of the rest of the partitions 102 _ 2 - 102 _N. The objective of improving the decoding performance of the decoding apparatus 304 is also achieved.

The decoding performance of the decoding apparatus 304 may be further improved by utilizing a buffering apparatus with a prefetch mechanism employed therein. Please refer to FIG. 4 , which is a diagram illustrating a video/image decoding system according to a second exemplary embodiment of the present invention. The major difference between the video/image decoding systems 300 and 400 is that the buffering apparatus 402 shown in FIG. 4 has a prefetch circuit 404 included therein. In this exemplary embodiment, the prefetch circuit 404 includes a prefetch unit 406 and a storage unit 408 . The prefetch unit 406 is arranged to prefetch data from a bitstream buffer in which the coefficient data of a next partition to be processed is stored and store the prefetched data into the storage unit 404 while the decoding apparatus 304 is decoding a current partition, wherein the prefetched data stored in the prefetch unit 408 is read by the decoding apparatus 304 when the decoding apparatus 304 starts decoding the next partition. The storage unit 408 may be an internal buffer of the decoding apparatus 304 . Thus, a data access speed of the storage unit 408 could be faster than a data access speed of each of the bitstream buffers 312 _ 2 - 312 _N. For example, the storage unit 408 may be implemented by a register or a static random access memory (SRAM). When the decoding apparatus 304 switches for decoding of the next partition, the time period needed for fetching the coefficient data of the next partition from one of the bitstream buffers 312 _ 2 - 312 _N can be saved/reduced due to the prefetched data available in the storage unit 408 . In other words, the time period needed for fetching the coefficient data of the next partition is covered in the time period during which the current partition is decoded. Thus, the use of the prefetch circuit 404 is capable of speeding up the overall decoding process.

In the exemplary embodiment shown in FIG. 4 , the prefetch mechanism is employed for prefetching next partition's data to be decoded by the following decoding apparatus. However, the same conception may be applied to prefetching next partition's data to be buffered by one of the bitstream buffers. Please refer to FIG. 5 , which is a diagram illustrating an alternative design of the buffering apparatus 302 shown in FIG. 3 . The buffering apparatus 502 includes a prefetch circuit 504 and the aforementioned bitstream buffers 312 _ 1 - 312 _N and multiplexer 315 . The prefetch circuit 504 is arranged to concurrently monitor one of the bitstreams 312 _ 1 - 312 _N that is buffering coefficient data of a current partition and one or more of the bitstreams 312 _ 1 - 312 _N that are used for buffering coefficient data of next partitions, and requests more data from a previous stage (e.g., Internet, middleware, or disk) when one or more of the bitstreams that are used for buffering coefficient data of next partitions have free storage space available for buffering prefetched data. To put it simply, the prefetch circuit 504 is arrange to prefetch data and store the prefetched data into at least a next partition bitstream buffer while a current partition bitstream buffer is buffering the coefficient data of the current partition processed by the following decoding apparatus. Therefore, with the help of the implemented prefetch mechanism, the bitstream buffering efficiency of the buffering apparatus is improved.

In above exemplary embodiments, the buffering apparatus with the prefetch mechanism employed therein has N bitstream buffers dedicated to buffering data of respective partitions, where N may any positive integer greater than 1. However, the proposed prefetch mechanism may also be employed in a buffering apparatus with a single bitstream buffer used for buffering data of a plurality of partitions.

Please refer to FIG. 6 , which is a diagram illustrating a video/image decoding system according to a third exemplary embodiment of the present invention. The video/image decoding system 600 includes a buffering apparatus 602 and a decoding apparatus 604 , wherein the buffering apparatus 602 includes a single bitstream buffer 612 and the aforementioned prefetch circuit 404 , and the decoding apparatus 604 includes a single bitstream DMA controller 616 and a single barrel shifter 618 . In this exemplary embodiment, the single bitstream buffer 612 is not a ring buffer. Besides, the bitstream size of the compressed frame 100 may be large. Thus, in a case where the buffer size of the single bitstream buffer 612 is smaller than the bitstream size of the compressed frame 100 , the single bitstream buffer 612 only buffers partial data of the compressed frame 100 (i.e., data of a current partition and next partition(s) of the compressed frame 100 ). Though the single bitstream buffer 612 may need to switch between partitions for loading requested data from a previous stage (e.g., Internet, middleware, or disk), the use of the prefetch circuit 404 is capable of improving the decoding efficiency of the decoding apparatus 604 by immediately feeding the requested data of the next partition to the decoding apparatus 604 when decoding of the next partition is started.

Moreover, no matter what the buffer size of the single bitstream buffer 612 is (e.g. smaller than/bigger than/equal to the bitstream size of the compressed frame 100 ), the use of the prefetch circuit 404 is still capable of speeding up the overall decoding process. As a person skilled in the art should readily understand operations of the decoding apparatus 604 and the prefetch circuit 404 after reading above paragraphs, further description is omitted here for brevity.

FIG. 7 is a diagram illustrating a video/image decoding system according to a fourth exemplary embodiment of the present invention. The video/image decoding system 700 includes the aforementioned buffering apparatus 602 and decoding apparatus 304 . Specifically, the buffering apparatus 602 includes a single bitstream buffer 612 and a prefetch circuit 404 , and the decoding apparatus 304 includes a plurality of bitstream DMA controllers 316 , 317 and a plurality of barrel shifters 318 , 319 . Compared to the decoding apparatus 604 shown in FIG. 6 , the decoding apparatus 304 shown in FIG. 7 is capable of decoding two partitions simultaneously. The combination of the bitstream DMA controller 316 and barrel shifter 318 is used for processing header information and per-macroblock prediction information contained in the partition 102 _ 1 , and the combination of the bitstream DMA controller 317 and barrel shifter 319 is used for processing coefficient data contained in the partitions 102 _ 2 - 102 _N. Similarly, though the single bitstream buffer 612 may need to switch between partitions for loading requested data from a previous stage (e.g., Internet, middleware, or disk), the use of the prefetch circuit 404 is capable of improving the decoding efficiency of the decoding apparatus 304 by immediately feeding the requested data of the next partition to the decoding apparatus 304 when decoding of the next partition is started.

Moreover, no matter what the buffer size of the single bitstream buffer 612 is (e.g. smaller than/bigger than/equal to the bitstream size of the compressed frame 100 ), the use of the prefetch circuit 404 is still capable of speeding up the overall decoding process. As a person skilled in the art should readily understand operations of the decoding apparatus 604 and the prefetch circuit 404 after reading above paragraphs, further description is omitted here for brevity.

Please note that the above-mentioned exemplary embodiments are directed to buffering and decoding a multi-partition VP8/WebP bitstream. However, this is not meant to be a limitation of the present invention. The proposed buffering mechanism and/or prefetch mechanism may be employed for processing any multi-partition based bitstream.

Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.