Temporal domain rate distortion optimization considering coding-mode adaptive distortion propagation

This application is a continuation application of International Application No. PCT/CN2020/132812, filed on Nov. 30, 2021, which is based upon and claims priority to Chinese Patent Application No. 202010241861.4, filed on Mar. 31, 2020, the entire contents of which are incorporated herein by reference.

The present invention belongs to the technical field of video coding and decoding, and in particular, relates to temporal domain rate distortion optimization considering coding-mode adaptive distortion propagation.

Rate distortion theory is the basic theory of lossy coding. The rate distortion optimization (RDO) technology developed based on this theory is one of the important tools to improve the coding efficiency, and has been widely applied in the field of video coding.

The performance of the video coding needs to be measured by coding bit and reconstruction distortion. On one hand, when we want higher video quality, the coding bit of the video will be increased; and on the other hand, at the lower coding bit level, the distortion of the video will be greatly increased, so there is a mutually contradictory and mutually constrained relationship between the coding bit and the reconstruction distortion. The rate distortion optimization technology is to make the encoder to select one group of optimal coding parameter set, so that the coding distortion is minimum on the premise that the coding bit is less than a target bit, and the mathematical expression is shown in a formula (1.1):

wherein D i and R i represent the distortion and bit number of the coding unit, N is a total number of the coding units, and R c represents the target bit number.

In order to solve the above restrictive rate distortion optimization problem, the global Lagrange multiplier λ g may be introduced to transform the constrained problem into an unconstrained problem of a formula (1.2), wherein J is called a rate distortion cost function.

FIG. 1 shows a classic R-D curve. When the video is coded by one group of coding parameters, distortion and code rate under the condition of the coding parameter may be obtained. We draw (R,D) combinations under different coding parameters in the form of points, and these points are called actual rate distortion operable points. We may observed that for the given rate Rx, an operable point with the smallest D can be always found, these points are called the optimal operable points which may be actually achieved, these optimal operable points are connected, and an actually operable R-D curve may be obtained.

Under the condition of independent rate distortion optimization, that is, the rate distortion performance between different coding units is mutually independent, the formula (1.2) is derived with respect to R to obtain λ g =∂D v /∂R i . It can be seen that λ g is a negative slope of a certain point on a rate distortion curve, the larger λ g corresponds to an operation point with a smaller code rate and larger distortion, the smaller λ g corresponds to a larger code rate, and the operable point with smaller distortion is the most important determining factor affecting the rate distortion performance, therefore, it is very important to select the Lagrange multiplier λ g . The size of λ g in a current VVC is mainly determined by a preset quantization parameter (QP) and is irrelevant to an input video sequence.

However, since intra-frame/inter-frame will introduce dependency among different coding units, and using the independent rate distortion optimization technology for each coding unit cannot achieve the optimal coding performance. Therefore, a global rate distortion optimization method with acceptable complexity is required to further improve the coding efficiency.

A temporal domain rate distortion optimization algorithm under an LD coding structure is studied in the literature temporally dependent rate-distortion optimization for low-delay hierarchical video coding. According to the time dependent relationship under the LD configuration, a temporal domain distortion propagation chain under multiple reference frames is established, the distortion propagation degree is estimated, and the propagation factor is calculated, so that the global Lagrange multiplier is adjusted according to the aggregation propagation factor, thereby realizing temporal domain rate distortion optimization and solving the problem about the global rate distortion optimization.

FIG. 2 shows a method for constructing a temporal domain distortion propagation chain under an LD coding structure. Due to the adoption of a multi-reference frame coding method, one coding block in a key frame may directly affect a plurality of blocks in the subsequent different frames and continue to extend forward indirectly. Therefore, a large number of experiments are required to figure out the utilization rate of each reference frame in the GOP, and a plurality of possible influence blocks are subjected to weighing average to calculate the expected distortion of the subsequent coding block. The affected coding block in the subsequent frame may be determined on the basis of the temporal domain dependent relationship under the LD coding structure in FIG. 3 and by adopting forward motion search.

When the temporal domain rate distortion optimization of the coding unit B i in the key frame f i is considered under the LD coding structure, the expected distortion of the affected coding unit B i+1 in the coding frame f i+1 is:

assuming that P i,j is the probability that the coding frame f i is referenced by the coding frame f i , o i is the coding parameter of B i . The last three terms are irrelevant to the coding parameter o i of B i , so the formula (1.3) may be simplified as:

E ( D i+1 )= P i,i+1 ·D i+1 ( o i ,o i+1 1 )+ a i+1   (1.4)

In the same way, the expected distortion of the coding unit B i+2 may be written as:

E ( D i+2 )= P i,j+2 ·D i+2 ( o i ,o i+2 2 )+ P i+1,i+2 ·D i+2 ( o i ,o i+1 *,o i+2 1 )+ a i+2   (1.5)

wherein a i+2 =P i−4,i+2 ·D i+2 (o i−4 ,o i+2 3 )+P i−8,i+2 ·D i+2 (o i−8 ,o i+2 4 ) is irrelevant to the coding parameter o i of B i , and the expected distortion of the coding unit which will affect the subsequent coding unit may be obtained by the similar method.

Based on the concept of the expected distortion, the rate distortion problem of the formula (1.2) may be represented again as:

The algorithm is relative rough to the expected distortion estimated by the current coding unit and the subsequent coding unit, so it is difficult for the propagation factor to accurately measure the influence on the subsequent coding distortion by the distortion of the current coding unit, and a loss is generated in the new generation video coding standard VCC; and meanwhile, the algorithm does not perform temporal domain rate distortion optimization on the I frame, and the coding performance of the I frame is very important in the LD coding structure.

For the above problem, in order to further optimize the temporal domain rate distortion optimization solution under the LD coding structure, the problem of dependency rate distortion optimization based on temporal domain distortion propagation is induced again according to a temporal domain dependency relationship under an LD structure and distortion propagation analysis under the skip mode and the inter mode; and the aggregation distortion of a current coding unit and an affected future coding unit are estimated and a propagation factor of a coding unit in a temporal domain distortion propagation model is calculated by constructing a temporal domain distortion propagation chain, so that a Lagrange multiplier is adjusted through a more accurate propagation factor to realize temporal domain dependency rate distortion optimization, and an I frame is subjected to a secondary coding technology to realize temporal domain dependency rate distortion optimization of the I frame.

The present invention adopts the following technical solutions:

The reconstruction distortion of a coding unit B i is assumed to be D i . Due to the presence of a skip mode in inter-frame prediction, it is unnecessary to transmit residual error in this mode, an inter-frame prediction value is directly used as a reconstruction value, and it is necessary to transmit residual error in another mode which is called an inter mode; therefore, the distortion of the current coding unit may consist of distortions brought by the skip mode and the inter mode:

D i =p inter ·D i inter +p skip ·D i skip =d inter +d skip   (1.7)

Only the partial distortion d inter of the current coding unit in the inter mode will affect the subsequent coding unit, because it is unnecessary to transmit a predicted residual error when the coded reference unit serves as a prediction block in the skip mode. The distortion of the current coding unit is determined by the distortion of the previously coded reference unit, so the influence on the subsequent coding unit is determined by the previously coded unit, and the distortion in the skip mode should be eliminated when the influence on the subsequent coding unit by the current coding unit is considered. Assuming that D i inter and D i skip are coding distortions of the current coding unit selecting the inter mode and the skip mode respectively, p inter and p skip are the probabilities that the current coding unit selects the inter mode and the slip mode respectively, and the sum of the two is 1. The larger error between the current coding unit and the prediction unit will cause larger probability that an encoder selects the inter mode, and the larger quantification step size will increase the probability that the encoder selects the skip mode. Therefore, p inter is defined as:

wherein D i OMCP =∥F i −F i−1 ∥ 2 is an original motion compensation error obtained by B i in an original frame through motion search, F i and F i−1 represents original pixels of a coding unit B i and a reference unit B i−1 respectively, and Δ is quantification step size.

when B i is coded, a partial derivative of a formula (1.6) with respect to Ri is evaluated to obtain a global Lagrange multiplier λ g :

A ∂R i /∂D i is multiplied at both ends of the formula (1.9) and assuming that ∂D i /∂R i =λ i , it may be obtained as follows:

wherein is a Lagrange multiplier of the coding unit B i under the global rate distortion performance. In addition, κ i represents the influence on the subsequent video sequence coding distortion by the coding unit B i , which is called a propagation factor of the coding unit B i .

The distortion function under the inter mode with high code rate may be represented as D i+1 inter =e −bR i+1 ·D i+1 MCP , the distortion function under the skip mode may be represented as D i+1 inter =D i+1 MCP , R i+1 is a code rate, b is a constant relevant to information source distribution, and D i+1 MCP is a motion compensation prediction error of B i+1 .

F i represents an original pixel of the coding unit Bi, {circumflex over (F)} i represents a reconstruction pixel of the coding unit B i and F i+1 represents an original pixel of the coding unit B i+1 .

According to the experimental observation, a is about equal to a constant, and at this time, the distortion of the coding unit B i+1 may be represented as:

D i+1 ≈p i,i+1 inter ·e −bR i+1 ·α·( D i+1 OMCP +D i )+ p i,i+1 skip ·α·( D i+1 OMCP +D i )  (1-12)

wherein P i,i+1 inter and P i,i+1 skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+1 is referenced to the coding unit B i , and D i+1 OMCP represents an original motion compensation error of the coding unit B i+1 .

At this time, the expected distortion of the coding unit B i+1 affected by the coding unit B i in the coding frame f i+1 may be obtained by a formula (1.4) and a formula (1.7):

Wherein γ i,i+1 =α·(p i,i+1 inter ·e −bR i+1 +p i,i+1 skip ), e −bR i+1 is only relevant to the code rate R i+1 of the coding unit B i+1 and is irrelevant to the coding parameter o i of the coding unit B i , c i+1 =P i,i+1 ·(p i,i+1 inter ·e −bR i+1 ·α·(D i+1 OMCP +D i skip )+p i,i+1 skip ·α·(D i+1 OMCP +D i skip )+a i+1 is also irrelevant to the coding parameter o i of B i , and only the probability P i,i+1 that the coding frame f i is referenced by the coding frame f i+1 , the coding distortion D i inter of the coding unit B i in the inter mode and the parameter γ i,i+1 are relevant to the coding parameter o i .

In the same way, the expected distortion of the coding unit B i+2 affected by B i in the coding frame f i+2 is:

E ( D i+2 )=( P i+1,i+2 ·γ i+1,i+2 ·P i,i+1 ·γ i,i+1 +P i,i+2 ·γ i,i+2 )· D i inter +c i+2   (1-14)

wherein γ i+1,i+2 =α··(p i+1,i+2 inter ·e −bR i+2 +p i+1,i+2 skip ), wherein p i+1,i+2 inter and p i+1,i+2 skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+2 is referenced to the coding unit B i+1 respectively, γ i,i+2 =α··(p i,i+2 inter ·e −bR i+2 +p i,i+2 skip ), wherein p i,i+2 inter and p i,i+2 skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+2 is referenced to the coding unit B i respectively, R i+2 represents the code rate of the coding unit B i+2 ,P i+1,i+2 and P i,i+2 represent the probabilities that the coding frames f i+1 and f i are referenced by the coding frame f i+2 respectively. c i+2 is an irrelevant term irrelevant to the coding parameter o i of the coding unit B i .

In the same way, the expected distortion of the coding unit B i+3 affected by B i in the coding frame f i+3 is:

wherein γ i+2,i+3 =α··(p i+2,i+3 inter ·e −bR i+3 +p i+2,i+3 skip ), wherein P i+2,i+3 inter and p i+2,i+3 skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+3 is referenced to the coding unit B i+2 respectively, γ i,i+3 =α··(p i,i+3 inter ·e −bR i+3 +p i,i+3 skip ), wherein p i,i+3 inter and p i,i+3 skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+3 is referenced to the coding unit B i respectively, R i+3 represents the code rate of the coding unit B i+3 , P i+2,i+3 and P i,i+3 represent the probabilities that the coding frames f i+2 and f i are referenced by the coding frame f i+3 respectively. c i+3 is an irrelevant term irrelevant to the coding parameter o i of the coding unit B i .

Therefore, the aggregation distortion of all the coding units influenced by the coding unit B in four coding frames in the current GOP is:

wherein γ i,i+k+1−t =α··(p i,i+k+1−t inter ·e −bR i,i+k+1−t +p i,i+k+1−t skip ), wherein p i,i+k+1−1 inter and p i,i+k+1−t skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+k+1−t , is referenced to the coding unit Bi respectively, γ j,j+1 =α··(p j,j+1 inter ·e −bR j+1 +p j,j+1 skip ), wherein p j,j+1 inter and p j,j+1 skip represent the probabilities of using the inter mode and the skip mode when the coding unit B j+1 is referenced to the coding unit B j respectively, P i,i+k+1−t represents a probability that the coding frame f i is referenced by the coding frame f i+k+1−t , and P i,i+1 represents a probability that the coding frame f j is referenced by the coding unit f j+1 ,

L i = ∑ k = 0 3 ⁢ c i + k - 1

being irrelevant to the coding parameter o i of the coding unit B i .

In the same way, the aggregation distortion of all the coding units influenced by the coding unit B i in four coding frames in the m-th GOP is:

γ i+4m,j+4m+k+1−t =α··(p i+4m,i+4m+k+1−t inter ·e −bR i+4m+k+1−t +p i+4m,i+4m+k+1−t skip ), wherein p i+4m,i+4m+k+1−t inter and p i+4m,i+4m+k+1−t skip represent the probabilities of using the inter mode and the skip mode when the coding unit B i+4m+k+1−t is referenced to the coding unit B i+4m respectively, P i+4m,i+4m+k+1−t represents a probability that the coding frame f i+4m is referenced by the coding frame f i+4m+k+1−1 , and P j,j+1 represents a probability that the coding frame f j is referenced by the coding frame f j+1 ,

L 4 ⁢ m = ∑ k = 0 3 ⁢ c i + 4 ⁢ m + k + 1

being irrelevant to the coding parameter o i of the coding unit B i .

The aggregation distortion of the coding units affected by B i in all the subsequent coding frames from the coding frame f i+1 to the last coding frame f N is:

wherein M is a total number of the GOP from the coding frame f i+1 to the last coding frame f N , and L represents an item irrelevant to o i .

It may be seen from a formula (1.8) that a relationship between the coding distortion D i inter of the current coding unit B i using the inter mode and the actual coding distortion D i is as follows:

making

e - bR i 1 + ( e - bR i - 1 ) ⁢ p i inter = η i , p i inter

being the probability of the coding unit B i selecting the inter mode, and a formula (1.19) may be represented as: D i inter =η i D i .

According to a formula (1.10), the calculation formula of the propagation factor κ i is:

the CTU-level global Lagrange multiplier λ g may be adaptively adjusted by using the propagation factor κ i , the CTU-level QP is further adjusted, and the frame level QP of all the B frames is adjusted by using a frame level average propagation factor.

Since the I frame is particularly important under the LD coding structure, and the subsequent coding frames need to be referenced to the I frame. At present, the QP of the I frame is uniformly lowered by 1 in the VTM, but the importance of the I frame is different in different sequences, so the I frame may be coded twice, the distortion propagation chain is established by the coding distortion obtained by the first coding, the propagation factor of each 16×16 block in the I frame is calculated, and the QP of the I frame is adjusted by the frame level average propagation factor, so that the QP of the I frame may be adjusted according to the influence on the subsequent coding frame by the I frame and the adjustment value is not limited to −1.

The present invention has the following beneficial effects: the problem in the traditional method that the I frame is not subjected to temporal domain rate distortion optimization is solved, so that the global rate distortion performance of the I frame is optimal, the problem of dependent rate distortion optimization based on temporal domain distortion propagation is induced again according to the temporal domain dependent relationship under the LD coding structure and the distortion propagation analysis in the skip mode and the inter mode, and the rate distortion optimization performance under the LD coding structure is improved.

FIG. 1 is an operable rate distortion curve.

FIG. 2 is a construction schematic diagram of a temporal domain distortion propagation chain under an LD coding structure.

FIG. 3 is a schematic diagram of an LD coding structure.

FIG. 4 is a rate distortion curve diagram of a Fourpeople sequence.

The present invention is described in detail below with reference to the embodiments:

in order to simplify the implementation method of a global rate distortion algorithm, a global Lagrange multiplier λ g may be directly modified in VTM through a propagation factor κ i . The subsequent coding unit is not really coded when deducing a propagation factor κ i , so it is necessary to estimate the distortion of the subsequent coding unit.

Under the condition of high code rate, the large probability of the coding distortion of the subsequent coding unit is inter distortion, and at this time, D i+1 =e −bR +1 ·D i+1 MCP is present. Since B i+1 is not coded, R i+1 cannot be obtained and D i+1 cannot use (1.21) calculation, but the coding distortion of B i+1 may be represented as D i+1 =D i+1 MCP ·F(θ) in case that the quantification step size is Q step ,

F (θ)= D i+1 /D i+1 MCP =e −bR i+1   (1-21)

wherein θ=√{square root over (2)}Q step /√{square root over (D MCP )}, a F(θ) curve may be fit based on a large amount of experiments with different quantification step sizes and coding units, a query table is established according to the curve, and the value of F(θ) is queried by calculating θ, so that the inter distortion of the coding block is estimated; meanwhile, in the present invention, α is set as 0.94.

According to the formula (1-9), a global Lagrange multiplier may be obtained:

Meanwhile, the Lagrange multiplier λ VTM =−∂D i VTM /∂R i VTM =bD i VTM of VTM. Therefore, Δ g and λ VTM have the following relationship:

D i ·λ g =(1+κ i ) D i VTM ·λ VTM   (1-23)

For all the coding units, there is:

The global Lagrange multiplier λ g may be evaluated by a formula (1-24), wherein N is the number of all the coding units, the distortion of all the coding units cannot be obtained in the coding process, and λ g is updated by the weighted sum of the distortion at this time, the distortion of the coded frame and the distortion of the coding frame which is just completed. Since D i VTM cannot be obtained in an encoder which integrates with the rate distortion algorithm proposed in this section, D i is used for replacing.

When the distortion propagation chain is established, motion search is done based on a 16×16 block, and a propagation factor of each block is calculated. The CTU of 128-128 is independently divided and coded in the VTM, so the average value of the propagation factors in all the 16×16 blocks in the CTU is taken as the propagation factor of the CTU, and the CTU-level Lagrange multiplier and the QP are adjusted; meanwhile, the QP of the frame level is adjusted by using the average propagation factor of a whole image.

The I frame is coded for twice to optimize and adjust the QP of the I frame. In order to reduce the coding complexity, the first coding process of the I frame is optimized, binary tree and ternary tree division modes are skipped, the CTU is divided only by a quad tree division mode, the minimum dividing size of the coding unit is set as 16×16 without smaller size division, and the distortion obtained based on the first coding of the I frame may estimate the influence on the subsequent coding unit by the distortion of the coding unit at the I frame, thereby realizing adaptive adjustment of the QP of the I frame.

According to the present invention, VVC reference software VTM5.0 serves as an experimental platform, the experimental environment is configured according to the common test conditions (CTC) specified by JVET and the reference software, the experiment is only performed under an LDB coding structure, the experiment test sequences are 16 video sequences such as Class B, C, D and E suggested by CTC, and each test sequence uses four QP points (22, 27, 32 and 37) for coding.

The coding experimental result is shown in Table 1. The table shows the Y component of the test sequence under the LDB coding structure achieves 2.57% coding performance. For most test sequences, the performance of the present invention is obviously improved, especially for Class E, the performance is obviously improved, and 10.13% code rate is saved under the Y component. The main reason is that Class E is a video sequence with a relatively fixed scene, each video frame has high similarity and high temporal domain dependence, and the present invention can achieve a better effect for the sequence. Then, some sequences are selected, a curve comparison diagram is optimized on the basis of the rate distortion, and the improvement condition of the coding performance of the sequences is observed. As shown in FIG. 4 which is a rate distortion curve diagram of a Fourpeople sequence, wherein the x-coordinate is the code rate, the y-coordinate is reconstructed peak signal to noise ratio (PSNR), the circular marking curve is a rate distortion curve of the global rate distortion optimization algorithm, and the square marking curve is a rate distortion curve of the original VTM5.0. It may be seen that for the sequence with strong temporal domain dependence, the coding efficiency of the algorithm is obviously improved.

Similarly, in the aspect of the coding complexity, the coding complexity of the temporal domain rate distortion optimization algorithm under the LDB coding structure is averagely increased by 15%, which is mainly due to that it takes a certain amount of time to do motion search on each 16×16 block to find the affected coding block so as to establish the distortion propagation chain; meanwhile, the I frame is optimized through 2-pass coding. Although the first coding process of the I frame is simplified, a small amount of coding complexity is increased.