Bi-pred mode decision in GOP architecture

The application makes reference to, claims priority to, and claims the benefit of U.S. Provisional Application Ser. No. 60/681,698 filed on May 17, 2005.

This application also makes reference to U.S. Provisional Patent Application No. 60/580,950 filed Jun. 17, 2004.

The above stated applications are hereby incorporated herein by reference in their entirety.

This application is a continuation-in-part of application Ser. No. 11/154,032, filed Jun. 16, 2005 by Chin, now U.S. Pat. No. 7,848,428 published as U.S. Patent Application Pub. No. 2005/0281329, and now issued as U.S. Pat. No. 7,848,428, which claims the benefit of and priority to U.S. Provisional Application No. 60/580,950, filed on Jun. 17, 2004.

[Not Applicable]

[Not Applicable]

Video encoders encode a video bit-stream according to a predetermined standard syntax, such as MPEG-2 or Advanced Video Compression (AVC). An encoder generating a compressed video bit-stream makes a number of choices for converting the video stream into a compressed video bit-stream that satisfies the quality of service and bit-rate requirements of a channel and media. Often, video data is transferred over a communication channel, with limited bandwidth, thus requiring compression on the encoder side.

One of the problems associated with compression is that it is a lossy process. Since compression discards some information in the video data, the discarded information is not recoverable. Other problems arise as a result.

To illustrate, consider, for example, an MPEG encoding process where video data comprises a series of frames. Each frame comprises two-dimensional grids of pixels. The two-dimensional grids are divided into smaller blocks of pixels, such as 16×16 blocks of pixels. These 16×16 blocks of pixels form a data structure known as a macroblock. The macroblock includes luminance and chrominance information, as well as additional parameters, including motion vectors. Each macroblock represents image data in a 16×16 block area of the image.

The data in the macroblocks is compressed in accordance with algorithms that take advantage of temporal and spatial redundancies. The macroblocks representing a frame are grouped into different slice groups. The slice group includes the macroblocks, as well as additional parameters describing the slice group. Each of the slice groups forming the frame forms the data portion of a picture structure. The picture includes the slice groups as well as additional parameters that further define the picture.

The pictures are then grouped together as a group of pictures (GOP). The GOP also includes additional parameters further describing the GOP. Groups of pictures are then stored, forming what is known as a video elementary stream (VES). The VES is then packetized to form a packetized elementary sequence. Each packet is then associated with a transport header, forming what are known as transport packets.

The transport packets can be multiplexed with other transport packets carrying other content, such as another video elementary stream or an audio elementary stream. The multiplexed transport packets form what is known as a transport stream. The transport stream is transmitted over a communication medium for decoding and displaying.

A group of pictures comprises reference pictures I and P, and bi-directional pictures B. Pictures coded using temporal redundancy with respect to exclusively earlier pictures of the video sequence are known as predicted pictures (or P-pictures). Pictures coded using temporal redundancy with respect to earlier and/or later pictures of the video sequence are known as bi-directional pictures (or B-pictures). Pictures not coded using temporal redundancy are known as I-pictures. In the MPEG-2 standard, I-pictures and P-pictures are also referred to as reference pictures.

A GOP generally starts with an I picture, followed by B and P pictures, up to the end of the GOP, where another GOP begins with another I pictures. Often, GOPs represent adjacent parts of a video sequence. In each GOP, B or P pictures are typically encoded using the I pictures associated with the GOP. When the next GOP begins, the pictures are encoded using the new I. Problems arise because of the time gap between one I picture and the next I picture. The pictures in one GOP are coded based on the I picture in the beginning of the GOP, and all of a sudden the pictures in the next GOP begins with an I picture that is relatively later in time from the earlier I picture. While in an original video sequence the change from the first I picture to the next I picture has a smooth transition, coding the B pictures based on the preceding I picture causes visible interruption in the transition between consecutive GOPs. This is known as “I-clicking” and may be seen in areas within a picture with a lot of detail such as, for example, a picture with wood texture.

Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with some aspects of the present invention as set forth in the remainder of the present application with reference to the drawings.

A system and/or method is provided for a bi-pred mode decision in GOP architecture, substantially as shown in and/or described in connection with at least one of the figures, as set forth more completely in the claims.

These and other features and advantages of the present invention may be appreciated from a review of the following detailed description of the present invention, along with the accompanying figures in which like reference numerals refer to like parts throughout.

FIG. 1 illustrates an exemplary encoding of a group of pictures, in accordance with an embodiment of the present invention.

FIG. 2 illustrates another exemplary encoding of a group of pictures, in accordance with an embodiment of the present invention.

FIG. 3 illustrates a block diagram of an exemplary video encoder, in accordance with an embodiment of the present invention.

FIG. 4A illustrates a flow diagram of an exemplary method of encoding pictures in a linear GOP, in accordance with an embodiment of the present invention.

FIG. 4B illustrates a block diagram of exemplary coding of a macroblock using bi-prediction mode, in accordance with an embodiment of the present invention.

FIG. 5 illustrates a flow diagram of another exemplary method of encoding pictures in a multi-level GOP, in accordance with an embodiment of the present invention.

FIG. 6 illustrates a block diagram of an exemplary video encoder, in accordance with an embodiment of the present invention.

Aspects of the present invention relate to video processing. More specifically, certain embodiments of the present invention relate to encoding video data using prediction methods. While the following discusses aspects of the present invention in association with video data, it should be understood that embodiments of the present invention may be applicable in systems utilizing compression and/or quantization methods that may result in loss of detail from the processed data.

FIG. 1 illustrates an exemplary encoding of a group of pictures, in accordance with an embodiment of the present invention. In a GOP 100 , a reference picture such as, for example, picture B A2 can be temporally coded using a reference picture from another GOP such as, for example, reference picture I 2 . Other reference pictures at predetermined intervals within the GOP 100 such as, for example, reference pictures B A1 and B A3 can be temporally coded using a reference picture from another GOP, i.e., I 2 . The remaining pictures in the GOP 100 can be temporally encoded using the preceding and/or subsequent reference pictures.

To illustrate, consider the GOP 100 , where anchor B pictures, B A , may be encoded such that the anchor B pictures closer to the subsequent GOP may utilize the I picture of the subsequent GOP. In a linear GOP, 100 , the first anchor B picture, B A1 may be encoded using I 1 and I 2 , where I 1 may be the reference picture of GOP 100 , and I 2 may be the reference picture of the following GOP. B A2 may be encoded using B A1 and I 2 , and B A3 may be encoded using B A2 and I 2 . The remaining B pictures, indicated by b, may then be encoded using the preceding I picture and the anchor B pictures, and as a result, the B pictures may be encoded using reference pictures containing information from both I 1 and I 2 and consequently a smooth transition may result between adjacent GOPs. Both I and anchor B pictures can be considered reference pictures, where I pictures may be considered primary reference pictures and anchor B pictures may be considered secondary reference pictures.

FIG. 2 illustrates another exemplary encoding of a group of pictures, in accordance with an embodiment of the present invention. In a GOP 200 , a reference picture such as, for example, picture B A2 can be temporally coded using a reference picture from another GOP such as, for example, reference picture I 2 . Other reference pictures at predetermined intervals within the GOP 200 such as, for example, reference pictures B A1 and B A3 can be temporally coded using the preceding reference picture and the subsequent reference picture. The remaining pictures in the GOP 200 can be also temporally encoded using the preceding and/or subsequent reference pictures.

To illustrate, consider the GOP 200 , where anchor B pictures, B A , may be encoded such that the anchor B pictures closer to the subsequent GOP may utilize the I picture of the subsequent GOP. In a multi-level GOP, 200 , the first anchor B picture, B A1 may be encoded using a preceding I picture and the second anchor B picture, I 1 and B A2 , respectively. The second anchor B picture, B A2 may be encoded using the preceding and the subsequent I pictures, I 1 and I 2 , respectively, where I 1 may be the reference picture of GOP 200 , and I 2 may be the reference picture of the following GOP. B A3 may be encoded using B A2 and I 2 . The remaining B pictures, indicated by b, may then be encoded using the preceding I picture and the anchor B pictures, and as a result, the B pictures, b, may be encoded using reference pictures containing information from both I 1 and I 2 and consequently a smooth transition may result between adjacent GOPs.

FIG. 3 illustrates a block diagram of an exemplary video encoder, in accordance with an embodiment of the present invention. The video encoder encodes video data comprising a set of frames F 0 . . . F n . The video encoder may comprise motion estimators 305 , motion compensators 310 , spatial predictors 315 , transformation engine 320 , quantizer 325 , scanner 330 , entropy encoders 335 , inverse quantizer 340 , and inverse transformation engine 345 . The foregoing may comprise hardware accelerator units under the control of a CPU.

In an embodiment of the present invention, in temporal prediction mode, the motion estimators 305 and motion compensators 310 form a prediction macroblock from one or more reference frames. Additionally, the motion estimators 305 and motion compensators 310 provide motion vectors identifying the prediction block. The motion estimators 305 , in identifying prediction blocks for a picture in a group of pictures, may check an earlier picture and/or a later picture, where the earlier and the later picture may be a reference, I picture or an anchor B, B A picture.

The motion estimators 305 and motion compensators 310 may utilize, in encoding an anchor picture using prediction block from an earlier and/or a later picture. In one embodiment of the present invention, the motion estimators 305 and motion compensators 310 may use a prediction block from an earlier picture or a later picture based on a decision made as to which may yield a better result. In another embodiment of the present invention, the motion estimators 305 and motion compensators 310 may use a prediction block from an earlier picture and a prediction bloc from a later picture and assign different prediction weights for each of the prediction blocks.

FIG. 4A illustrates a flow diagram of an exemplary method of encoding pictures in a linear GOP, in accordance with an embodiment of the present invention. At 405 it may be determined whether a picture contains content that may be susceptible to “I-clicking.” If the picture contains content susceptible to “I-clicking,” bi-prediction mode may be selected and at 410 anchor pictures may be temporally encoded utilizing the preceding reference picture from the same GOP associated with the anchor picture, and the first reference picture from the GOP following the GOP associated with the anchor picture. Thus, anchor pictures in a GOP may be encoded utilizing a prior reference picture in the GOP, where the prior picture may be an I picture or another anchor picture in the same GOP, and the I picture from the subsequent GOP. In bi-prediction mode, illustrated in FIG. 4B , macroblocks within a picture may be encoded using a macroblock from a previous picture and a macroblock from a subsequent picture. Each of the macroblocks used can be given a different weight corresponding to a combination that yields the most desirable result. For example, a macroblock M 2 in a picture 450 may be temporally encoded using bi-prediction mode, utilizing macroblock M 1 from a preceding picture 450 and a macroblock M 3 from a preceding picture 460 , wherein different weights may be used with each macroblock as follows:

M 2 =w 1 M 1 +w 3 M 3

If the picture does not contain content susceptible to “I-clicking,” regular prediction mode may be selected at 415 and reference pictures may be encoded utilizing other reference pictures within the GOP.

At 420 , the remaining pictures in the GOP may be encoded utilizing the prior and subsequent reference pictures, which may be the I picture or anchor pictures from the same GOP, or the I picture from the subsequent GOP.

FIG. 5 illustrates a flow diagram of another exemplary method of encoding pictures in a multi-level GOP, in accordance with an embodiment of the present invention. At 505 it may be determined whether a picture contains content that may be susceptible to “I-clicking.” If the picture contains content susceptible to “I-clicking,” bi-prediction mode may be selected and at 510 an anchor pictures in the middle of the GOP may be temporally encoded utilizing the I picture from the same GOP and the I picture from the subsequent GOP. At 515 the remaining anchor pictures in the GOP may be temporally encoded utilizing the preceding and subsequent reference pictures, which may be I pictures from the same and the subsequent GOP, or other anchor pictures within the same GOP.

If the picture does not contain content susceptible to “I-clicking,” regular prediction mode may be selected at 520 and reference pictures may be encoded utilizing other reference pictures within the GOP.

At 525 , the remaining pictures in the GOP may be encoded utilizing the prior and subsequent reference pictures, which may be the I picture or anchor pictures from the same GOP, or the I picture from the subsequent GOP.

In an embodiment of the present invention, the methods described by FIG. 4 and FIG. 5 may be utilized at a macroblock level, where the content of a macroblock within the pictures may be classified as susceptible to “I-clicking” and based on the content classification, an prediction mode may be selected for encoding the macroblock.

FIG. 6 illustrates a block diagram of an exemplary video encoder system 600 , in accordance with an embodiment of the present invention. The system 600 comprises a picture rate controller 605 , a macroblock rate controller 610 , a pre-encoder 615 , hardware accelerator 620 , spatial from original comparator 625 , an activity metric calculator 630 , a motion estimator 635 , a mode decision and transform engine 640 , an arithmetic encoder 650 , and a CABAC encoder 655 .

The picture rate controller 605 may comprise software or firmware residing on a CPU. The macroblock rate controller 610 , pre-encoder 615 , spatial from original comparator 625 , mode decision and transform engine 640 , spatial predictor 645 , arithmetic encoder 650 , and CABAC encoder 655 may comprise software or firmware residing on the CPU. The pre-encoder 615 includes a complexity engine 660 and a classification engine 665 .

In an embodiment of the present invention, the classification engine 665 may classify pictures according to their content. As a result, the classification engine 665 may indicate when a picture or a video sequence contains content susceptible to “I-clicking.” The classification determined by the classification engine 665 may then be indicated to the mode decision and transform engine 640 , which mainly determines the best mode for encoding a picture based on the input from the motion estimator 635 , the spatial encoder 645 , and the pre-encoder 615 . Based on the classification of the content associated with a picture, the mode decision and transform engine 640 may determine whether a picture is to be encoded as an anchor picture.

Accordingly, the present invention may be realized in hardware, software, or a combination thereof. The present invention may be realized in a centralized fashion in at least one computer system, or in a distributed fashion where different elements may be spread across several interconnected computer systems. Any kind of computer system or other apparatus adapted for carrying out the methods described herein may be suited. A typical combination of hardware and software may be a general-purpose computer system with a computer program that, when being loaded and executed, may control the computer system such that it carries out the methods described herein.

The present invention may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which when loaded in a computer system is able to carry out these methods. Computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form.

While the present invention has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims.