Systems, methods, and apparatus for improving display of compressed video data

This application claims priority to “Systems, Methods, and Apparatus for Improving Display of Compressed Video Data”, Provisional Application for U.S. Patent Ser. No. 60/681,269, filed May 16, 2005, which is incorporated herein by reference for all purposes.

[Not Applicable]

[Not Applicable]

Video compression is used to compress video data for storage on devices with limited memory and transmission over communication channels with limited bandwidth. A number of video compression standards, such as MPEG-2, VC-1, and AVC compress the video data by, among other things, removing or deemphasizing higher frequency components. The higher frequency components have been observed to be less perceptible to the human eye.

However, when the compressed video data is decompressed and displayed on a display device, the video data has been observed to produce a different intensity compared to the original video data. The intensity difference is the result of the nonlinear response of system. The eye takes things with high spatial frequency and averages their intensity. When a nonlinear response is involved, this averaging changes the perceived value. For example, if a system has a response of y=x 2 , with the values of x, 1, 2, and 3, the output is 1, 4, and 9. If the input is low pass filtered and changed to 2, 2, and 2, the output is 4, 4, 4. In the original case, the average of the output value is 14/3 In the low pass filtered case, it is 4.

Display devices, such as CRTs, have non-linear characteristics between pixel value and brightness. To compensate for this, the displayed video data is gamma corrected. However, even where the decompressed video is perceptually similar to the original video, its response to gamma correction often produces a different intensity compared to the original video data.

Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with some aspects of the present invention as set forth in the remainder of the present application with reference to the drawings.

Aspects of the present invention may be found in a system, method, and/or apparatus for improving the display of compressed video data, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims.

These and other advantages and novel features of the present invention, as well as illustrated embodiments thereof will be more fully understood from the following description and drawings.

FIG. 1 is a block diagram of an exemplary circuit in accordance with an embodiment of the present invention;

FIG. 2 is a flow diagram for encoding video data in accordance with an embodiment of the present invention;

FIG. 3A is a block diagram describing spatial prediction;

FIG. 3B is a block diagram describing motion prediction;

FIG. 3C is a block diagram describing transformation and quantization;

FIG. 4 is a block diagram of an exemplary circuit in accordance with an embodiment of the present invention;

FIG. 5 is a block diagram of an exemplary circuit in accordance with an embodiment of the present invention; and

FIG. 6 is a flow diagram for encoding video data in accordance with an embodiment of the present invention.

Referring now to FIG. 1 , there is illustrated a block diagram describing an exemplary circuit 100 for encoding video data in accordance with an embodiment of the present invention.

The circuit 100 comprises a first circuit 110 and a second circuit 200 . The first circuit 110 estimates the display intensity for a video source, when the video source is displayed on a predetermined display device and estimates a displayed intensity of the reconstruction of the video source displayed on a predetermined display device.

It is noted that the video source may be compressed prior to transmission over a communication media. Reconstructed video data can be video data that results from compressing and then decompressed the video source.

The display intensity can be estimated in a variety of ways. In certain embodiments of the present invention, the display intensity can be measured by applying a transformation to the video source and the reconstructed video data that models the predetermined display device.

The display intensity of the reconstructed video source can be estimated in a variety of ways. In certain embodiments of the present invention, the video source can be compressed in accordance with an encoding standard and then decompressed. A transformation modeling the display device can be applied to the decompressed video source to estimate the display intensity of the reconstructed pixels.

The second circuit 200 corrects the video source as a function of the estimated displayed intensity of the video source on the predetermined display device and the estimated displayed intensity of the reconstruction of the video source on the predetermined display device.

In certain embodiments of the present invention, the video source can be corrected as a function of the difference between the estimated displayed intensity of the video source on the predetermined display device and the estimated displayed intensity of the reconstruction of the video source on the predetermined display device. In certain embodiments of the present invention, the difference can comprise the average difference. In accordance with certain embodiments of the present invention, the resulting corrected video source can then be compressed and transmitted over a communication media.

Referring now to FIG. 2 , there is illustrated a flow diagram for encoding video data in accordance with an embodiment of the present invention. At 205 , the intensity of the video source displayed on a predetermined display device is estimated. At 210 , the intensity of a reconstruction of the video source displayed on the predetermined display device is measured. At 215 , the video source is corrected as a function of the estimated displayed intensity of the video source on the predetermined display device and the estimated displayed intensity of the reconstruction of the video source on the predetermined display device.

As noted above, there are a variety of different compression standards for compressing video data. Certain embodiments of the present invention can be incorporated in the context of a compression standard. In certain embodiments of the present invention, the video source can be compressed in accordance with a standard. The reconstruction of the compressed video data can be the resulting video data that is generated by decompressing the compressed video data and the display intensity of the reconstructed data can be estimated. The video source can be corrected according to the display intensity of the video source and the reconstructed video data. The corrected video source can then be compressed and transmitted in accordance with the compression standard. An exemplary compression standard will now be described followed by exemplary embodiments of the present invention.

Advanced Video Coding (also known as H.264 and MPEG-4, Part 10) generally provides for the compression of video data by dividing video pictures into fixed size blocks, known as macroblocks 320 . The macroblocks 320 can then be further divided into smaller partitions 430 with varying dimensions.

The partitions 430 can then be encoded, by selecting a method of prediction and then encoding what is known as a prediction error. AVC provides two types of predictors, temporal and spatial. The temporal prediction uses a motion vector to identify a same size block in another picture and the spatial predictor generates a prediction using one of a number of algorithms that transform surrounding pixel values into a prediction. Note that the data coded includes the information needed to specify the type of prediction, for example, which reference frame, partition size, spatial prediction mode etc.

The reference pixels can either comprise pixels from the same picture or a different picture. Where the reference block is from the same picture, the partition 430 is spatially predicted. Where the reference block is from another picture, the partition 430 is temporally predicted.

Spatial Prediction

Referring now to FIG. 3A , there is illustrated a block diagram describing spatially encoded macroblocks 320 . Spatial prediction, also referred to as intra prediction, is used by H.264 and involves prediction of pixels from neighboring pixels. Prediction pixels are generated from the neighboring pixels in any one of a variety of ways.

The difference between the actual pixels of the partition 430 and the prediction pixels P generated from the neighboring pixels is known as the prediction error E. The prediction error E is calculated and encoded.

Temporal Prediction

Referring now to FIG. 3B , there is illustrated a block diagram describing temporally prediction. With temporal prediction, partitions 430 are predicted by finding a partition of the same size and shape in a previously encoded reference frame. Additionally, the predicted pixels P can be interpolated from pixels in the frame or field, with as much as ¼ pixel resolution in each direction. A macroblock 320 is encoded as the combination of data that specifies the derivation of the reference pixels P and the prediction errors E representing its partitions 430 . The process of searching for the similar block of predicted pixels P in pictures is known as motion estimation.

The similar block of pixels is known as the predicted block P. The difference between the block 430 and the predicted block P is known as the prediction error E. The prediction error E is calculated and encoded, along with an identification of the predicted block P. The predicted blocks P are identified by motion vectors MV and the reference frame they came from. Motion vectors MV describe the spatial displacement between the block 430 and the predicted block P.

Transformation, Quantization, and Scanning

Referring now to FIG. 3C , there is illustrated a block diagram describing the encoding of the prediction error E. With both spatial prediction and temporal prediction, the macroblock 320 is represented by a prediction error E. The prediction error E is a two-dimensional grid of pixel values for the luma Y, chroma red Cr, and chroma blue Cb components with the same dimensions as the macroblock 320 , like the macroblock.

In certain embodiments of the present invention, the display intensity of a macroblock 320 of the video source and the display intensity of the corresponding macroblock in the reconstructed video source can be estimated. The difference of the display intensity of the macroblock of the video source and the corresponding macroblock in the reconstructed video source can be added to the prediction error E, thereby resulting in a corrected prediction error.

A transformation transforms the prediction errors E to the frequency domain. In H.264, the blocks can be 4×4, or 8×8. The foregoing results in sets of frequency coefficients f 00 . . . f mn , with the same dimensions as the block size. The sets of frequency coefficients are then quantized, resulting in sets 440 of quantized frequency coefficients, F 00 . . . F mn .

Quantization is a lossy compression technique where the amount of information that is lost depends on the quantization parameters. The information loss is a tradeoff for greater compression. In general, the greater the information loss, the greater the compression, but, also, the greater the likelihood of perceptual differences between the encoded video data, and the original video data.

The pictures 115 are encoded as the portions 120 forming them. The video sequence is encoded as the frames forming it. The encoded video sequence is known as a video elementary stream.

Referring now to FIG. 4 , there is illustrated a block diagram of an exemplary circuit for encoding video data in accordance with an embodiment of the present invention. The circuit comprises a first compressor 405 , a decompressor 410 , a display intensity transformation circuit 415 , a subtractor 420 , an adder 425 , and a second compressor 432 .

The first compressor 405 compresses the video source. The decompressor 410 decompresses the video source, thereby resulting in a reconstructed video source. The display intensity transformation circuit 415 measures the display intensity for the video source and the reconstructed video source. The subtractor 420 measures the difference of the display intensity for the video source and the reconstructed video source. The adder 425 adds the difference of the display intensity to the video source, thereby resulting a corrected video source. The second compressor 430 compresses the corrected video source. In certain embodiments, a low pass filter 440 filters the difference.

Referring now to FIG. 5 , there is illustrated a block diagram of another exemplary circuit in accordance with an embodiment of the present invention. The circuit comprises a predictor circuit 505 , first transformer/quantizer 510 , an inverse quantizer/transformer 515 , a motion compensator 518 , a display intensity transformer 520 , a subtractor 525 , an adder 530 , and a second transformer/quantizer 535 .

The predictor 505 receives actual blocks from the video source for either spatial or motion prediction and outputs a prediction error E and an identification of the reference pixels. It is noted that the reference pixels can be identified by a motion vector (where the case is motion prediction). The transformer/quantizer 510 transforms and quantizes the prediction error E. The inverse quantizer/transformer 515 inverse quantizes and transforms, providing a reconstructed prediction error E′.

Display intensity transformer 520 determines the display intensity for the reconstructed prediction error E′ and the original prediction error E. The subtractor 525 provides the difference between the display intensity for the prediction error E and the reconstructed prediction error E′. Adder 530 adds the difference to the original prediction error E′, thereby resulting in a corrected prediction error. The second transformer/quantizer 535 transforms and inverse quantizes the corrected prediction error.

Referring now to FIG. 6 , there is illustrated a flow diagram for encoding video data in accordance with an embodiment of the present invention. At 605 , a portion of a picture from the video source is predicted, either temporally or spatially, thereby resulting in a prediction error. At 610 , the prediction error is transformed and quantized, thereby resulting in a compressed portion of the video source.

At 615 , the compressed video source is decompressed, thereby resulting in a reconstructed portion of the video source. A transfer function that models the predetermined display device is applied to portion of the video source at 620 and the reconstructed video source at 625 to estimate the display intensity of the portion of the video source and the reconstructed portion of the video source.

At 630 , the difference between the estimated display intensity of the portion of the video source and the reconstructed portion of the video source is taken, and added to the prediction error at 635 , thereby resulting in a corrected prediction error. At 640 , lossy compression is applied to the prediction error.

The embodiments described herein may be implemented as software, a board level product, as a single chip, application specific integrated circuit (ASIC), or with varying levels of the decoder system integrated with other portions of the system as separate components.

The degree of integration of the decoder system will primarily be determined by the speed and cost considerations. Because of the sophisticated nature of modern processor, it is possible to utilize a commercially available processor, which may be implemented external to an ASIC implementation.

If the processor is available as an ASIC core or logic block, then the commercially available processor can be implemented as part of an ASIC device wherein certain functions can be implemented in firmware. Alternatively, the functions can be implemented as hardware accelerator units controlled by the processor.

While the present invention has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present invention.

Additionally, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. For example, although the invention has been described with a particular emphasis on H.264 encoded video data, the invention can be applied to a video data encoded with a wide variety of standards.

Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims.