Zero-delay panoramic video bit rate control method considering temporal distortion propagation

This application is the national phase entry of International Application No. PCT/CN2023/087513, filed on Apr. 11, 2023, which is based upon and claims priority to Chinese Patent Application No. 202211273536.1, filed on Oct. 18, 2022, the entire contents of which are incorporated herein by reference.

The present application belongs to the technical field of image processing, and in particular, relates to a zero-delay panoramic video bit rate control method considering distortion temporal propagation influence.

A panoramic video is a video shot by a camera array or a 3D video camera in all-round 360 degrees. A user can adjust the up, down, left and right directions of the video at will for watching when watching the video. The panoramic video has the characteristics of high frame rate, high resolution (at least 4K) and wide field of vision, so saving and transmitting the panoramic video will consume a lot of resources. Bit rate control is to optimize the bit allocation layer by layer according to the target bit rate and dynamically adjust the coding parameter of the encoder according to a certain coding strategy, so that the encoder outputs a code stream to meet the bandwidth limitation of a transmission channel and the requirement of a storage space of a storage device, and the actual output bit rate is consistent with the target bit rate as much as possible. The bit rate control is widely applied to a real-time video communication system. The traditional panoramic video has the following problems: first, the pixel redundancy will be generated in the projection process to affect the coding performance; meanwhile, the traditional encoder does not use the temporal correlation of distortion between interframe coding units for coding, and the coding performance still has a great room for improvement; and the main reason why the traditional encoder does not consider the interframe influence factor of distortion is that the calculation complexity is too high, which is not conducive to real-time communication. In view of the above problems, the present application provides a zero-delay panoramic video bit rate control method considering distortion temporal propagation influence. According to the present application, first, the coding parameter is adjusted according to a ratio of the area of a panoramic video spherical surface and the projection area so as to reduce the loss of the coding performance caused by the pixel redundancy; meanwhile, whether an image scenario is changed is determined by calculating the change quantity of the pixel mean value and variance value of the coding frame; and when the scenario change does not occur, the distortion temporal propagation influence factor is calculated based on that the video image has the characteristic of continuity and by using the reconstruction error and motion compensation prediction error of the previous coding frame in the time domain, and the current frame is coded by the factor. According to the present application, it is unnecessary to pre-store an uncoded frame, the distortion temporal propagation influence factor is calculated by coded frame information, the calculation quantity is small, the coding complexity is extremely low, and the rate-distortion performance can be greatly improved, so the present application is easy to popularize.

The current bit rate control method includes:

an H.266/VVC bit rate control method mainly includes the following steps: Step 1: bit allocation

VTM adopts a layered bit allocation structure to perform bit allocation respectively on a Group of Pictures (GOP) layer, a frame layer and a CTU layer. The further down, the more accurate the bit rate allocation, and the better it is to correct the error. VTM allocates the target bit rate on averagely at all levels. To gradually eliminate the bit rate control error generated by the coded unit, bit rate fluctuation is smoothed by a sliding window, so that the influence on the video quality caused by the great bit rate fluctuation. Layered bit rate allocation is implemented by the following formulas, that is,

ω pic and

∑ n ⁢ 0 ⁢ tcoded ctu pic

ω ctu are respectively the number of the coded frames, the bits consumed by the coded video sequence, a GOP frame number, the number of coded frames in the GOP, the weight of the coding frame, the weight of the coding unit, the sum of the weights of the uncoded frames and the sum of the uncoded CTU in the frame.

The Lagrange multiplier λ i is the slope of the rate-distortion curve, and D i and R i are respectively the distortion and coding bit of the i th coding unit and the relationship between the Lagrange multiplier λ i and the bit rate R i .

The parameters in the above formula will be updated automatically after one frame or one coding tree unit is coded. The parameter updating formula is:

In the formula, D i and R i can be determined after one frame or one CTU is coded, the frame-level λ i is replaced with the Lagrange multiplier of the most adjacent coded frame on the same level in the same GOP, and CTU-level λ i is replaced with the Lagrange multiplier of the CTU corresponding to the most adjacent coded frame on the same level with the coding frame, so as to update the parameters c i and k i .

QP i =4.2005·ln λ i +13.7122

It can be seen that the quantization parameter in the bit rate control algorithm and the logarithm of the Lagrange multiplier can be fitted by a linear function relationship.

The traditional bit rate control method has the following problem that an R-Lambda bit rate control model in VVC can reach a high control accuracy, but does not consider the problem of pixel redundancy of the panoramic video caused by projection. Meanwhile, the interframe correlation of distortion is not used for rate-distortion optimization coding, so the coding quality has a great room for improvement.

In view of the above problem, the present application provides a zero-delay panoramic video bit rate control method considering time-domain distortion propagation. The method according to the present application is mainly for optimizing target bit allocation and global rate-distortion optimization, and includes coding tree unit (CTU)-level bit rate control and temporal global rate-distortion optimization. The CTU-level bit rate control mainly includes optimizing target bit allocation and updating bit rate control parameters. The temporal global rate-distortion optimization primarily quantifies the temporal correlation between the current coding frame and the nearest previously coded frame by calculating the proportional relationship of the reconstruction distortion and motion compensation prediction distortion for each CTU (Coding Tree Unit). It then adjusts the coding parameters based on the magnitude of this temporal correlation. Additionally, the coding parameters are further adjusted according to the area stretching ratio during the process of projecting the panoramic video from a 3D spherical surface to a 2D plane, taking into account the adverse effects of interpolated redundant pixels on the coding process.

The technical solution of the present application is as follows:

a zero-delay panoramic video bit rate control method, which considers the influence of distortion temporal propagation and redundant pixels, includes the following steps: S1: inputting a target bit rate and a sequence to be coded to an encoder; S2: determining whether a current frame is a first frame, and if so, performing the following operations: calculating a latitude value corresponding to a CTU line and calculating an area stretching ratio of a spherical surface and a projection plane thereof, initializing the parameters of a bit rate control unit, calculating the number of target bits of the current frame, calculating a frame-level Lagrange multiplier of the current frame according to the number of the target bits, calculating a frame-level QP according to the frame-level Lagrange multiplier and performing intraframe coding, repeating step S2 after coding the current frame, otherwise, entering step S3; S3: calculating pixel mean and variance values of a frame; S4: determining whether the current frame number is greater than twice the GOP size, if so, entering S5, otherwise, entering S6; S5: calculating a global Lagrange multiplier according to an iterative algorithm, optimizing bit allocation of each CTU in the frame, and entering step S7; S6: using the bit allocation algorithm of the encoder to allocate bits for each CTU in the frame; S7: calculating the frame-level Lagrange multiplier and QP; S8: coding the CTUs in sequence; S9: determining whether a difference value between the pixel variance values of the current frame and the previous frame is greater than a threshold, if so, entering step S10, otherwise, entering step S1; S10: adjusting a Lagrange multiplier of the current CTU according to the area stretching ratio obtained in step S2, and entering step S13; S11: determining the current frame number is greater than 3, if so, entering S13, otherwise, entering S10; S12: adjusting the Lagrange multiplier of each CTU according to the temporal distortion influence factor of the previous frame and the area stretching ratio obtained in S2; S13: calculating the quantization parameter (QP) of the current CTU according to the Lagrange multiplier of the current CTU calculated in S10 or S12, and coding the CTU; S14: determining whether the current CTU is a last CTU, if so, coding the current frame and entering step S15, otherwise, returning to step S8; S15: updating bit rate control parameters and the number of actually used bits of the current frame and each CTU; S16: acquiring a reconstruction error and a motion compensation prediction error of each CTU saved in the current frame to calculate the temporal distortion influence factor of each CTU in the current frame; S17: determining whether the current frame is the last frame, if so, entering S18, otherwise, entering S2; S18: completing coding of a current panoramic video sequence.

Further, in step S2, if a radius of a panoramic video sphere is defined as r, the area of a spherical girdle region with the latitude θ is:

S s (θ)=2π· r 2 ·cos θ·sin dθ

after being projected onto a 2D plane, the area of a stretching region is obtained as follows:

Further, step S3 is specifically as follows:

Further, step S5 is specifically as follows:

λ i is a slope of a rate-distortion curve, D i and R i are respectively distortion and coding bit of an i th coding unit, and M is the number of CTUs.

Further, in the step 10, a method for adjusting the Lagrange multiplier is as follows:

Further, in the step 12, a method for adjusting the Lagrange multiplier is as follows:

The present application has the following beneficial effects: according to the present application, on the premise of reducing the coding complexity and not caching the subsequent frame, the bit rate control error can be very small, the coding time can be effectively shortened, and the video coding quality can be improved under the condition of the given bandwidth.

FIGURE is a flowchart of a method according to the present application.

The present application is described in detail below with reference to the accompanying drawing.

In the video bit rate control, an encoder selects a group of optimal coding parameters and a coding mode with a minimum rate-distortion cost for an input video through a rate-distortion optimization (RDO) technology. The objective is to reduce the coding distortion as much as possible under a certain bit rate limited condition or reduce a coding bit as much as possible under a certain coding distortion restriction condition. A zero-delay method provided by the present application means that an encoder is not allowed to obtain information of a subsequent frame in advance, that is, data is not cached and analyzed after a frame to be coded is obtained and is necessarily coded at once. The specific implementation manner is shown in FIGURE.

For an input first frame image, a time-domain distortion influence factor has not been obtained inside the encoder, so it is necessary to initialize the internal parameters of the encoder based on the given target bit rate first. The initialization mode is consistent with that in a VVC bit rate control method. The number of target bits of each coding level is:

ω pic and

∑ n ⁢ 0 ⁢ tcoded ctu pic

ω ctu are respectively the number of the coded frames, the bits consumed by the coded video sequence, a GOP frame number, the number of coded frames in the GOP, the weight of the coding frame, the weight of the coding unit, the sum of the weights of the uncoded frames and the sum of the uncoded CTU in the frame.

Different from the traditional method, in the method provided by the present application, it is also necessary to calculate a latitude value corresponding to a CTU line and calculate an area stretching ratio of a spherical surface and a projection plane thereof. The specific method is as follows:

r is defined as a radius of a sphere, and the area of a spherical surface girdle region with the latitude θ can be calculated as:

S s (θ)=2π· r 2 ·cos θ·sin dθ

Corresponding to the projection of a 2D plane, the area of the stretching area is:

An area ratio before and after projection is:

It can be seen that the ratio decreases with the increase of the latitude, in the equatorial region, that is, at the latitude θ of 0, the area ratio is 1, without stretching. Assuming that the entropy of the equatorial region has not changed before and after the projection, and the closer to the pole, the greater the entropy change.

Starting from a second image group GOP, the optimal Lagrange multiplier in a bit allocation formula is approximated by an iterative algorithm. The bits are allocated by the optimal Lagrange multiplier, specifically as follows:

In the formula, R and r are respectively frame-level and CTU-level target bits, λ g is the global Lagrange multiplier, a i =c i ·k i ,

β i = 1 k i + 1 ,

and one frame has M CTUs.

For the second frame and the subsequent images, the Lagrange multiplier can be adjusted by selecting the area-based stretching ratio. For the third frame and the subsequent images, since the time-domain distortion influence factor has been obtained, the Lagrange multiplier is adjusted based on the time-domain distortion influence factor and the area stretching ratio, specifically as follows: a new Lagrange multiplier λ n is obtained by dividing the original Lagrange multiplier by an adjusting weight, so that time-domain rate-distortion optimization is achieved.

The condition of selecting a Lagrange multiplier adjusting mode is to determine whether a difference value between the pixel variance value of the current frame and the previous frame is greater than a threshold of 50. Since the premise of the time-domain distortion propagation is the continuity of a video image, if image scenario switching occurs, a propagation chain will be disconnected, whether the scenario is switched is determined through the transformation quantity of the image variance value.

The calculation mode of the Lagrange multiplier and QP is as follows:

first, the average number (T bpp ) of target bits per pixel of the current frame image is calculated. The calculation formula is:

T bpp =T pic /N pixels

N pixels refers to the number of pixels of a picture. Then the frame-level Lagrange parameter of the current image is calculated.

For a frame I: a bit rate control method based on an intraframe complexity is adopted. According to the experiment, the intraframe complexity is defined as

MAD BPP ,

then

and the Lagrange multiplier have the following relationship:

For a frame P: the calculation mode is λ=αR β , the initial value of α is set to 1058, and the value of β is set to −1.327.

The corresponding frame-level QP is calculated by using the following relational expression:

QP i =4.2005·ln λ i +13.7122

after coding of one frame image is completed, the bit rate control parameters of the current frame and each CTU as well as the number of actually used bits are updated, a reconstruction error and a motion compensation prediction error of each CTU saved in the current frame are obtained to calculate a time-domain distortion influence factor of each CTU in the current frame. Specifically, the time-domain dependence in video coding is measured by using the reconstruction distortion and the motion compensation prediction error distortion of the coding tree unit, that is,

of the coding block. For the current coding block, the distortion thereof cannot be obtained before the real coding. To achieve zero delay, it is considered to approximately replace the time-domain dependence of the current block with the coding tree unit information at the corresponding position of the previous frame. This is because the image characteristics of the adjacent two frames are generally similar. Meanwhile, whether scenario switching occurs in the image is determined according to the difference between the pixel mean value and variance value of the current frame and the previous frame so as to determine whether to use the distortion influence factor.

Automatic update will be performed after one frame or one coding tree unit (CTU) is coded. The parameter updating formula is:

In the formula, the distortion D i and the bit rate R i can be determined after one frame or one CTU is coded, the frame-level λ i is replaced with the Lagrange multiplier of the most adjacent coded frame on the same level in the same GOP, and CTU-level λ i is replaced with the Lagrange multiplier of the CTU corresponding to the most adjacent coded frame on the same level with the coding frame, so as to update the parameters c i and k i .

The effectiveness of the scheme of the present application is described below by using the experimental results. A panoramic video bit rate control algorithm is integrated into H.266/VVC reference software VTM14.0 based on 360lib, the encoder is configured as Lowdelay-P, the experimental conditions comply with the standard panoramic video sequence recommended by International Coding Organization for Standardization JCT-VC as a test video, including all 14 video sequences in three categories of 8K, 6K and 4K, and the comparison indexes are BD-Rate, bit rate control accuracy and coding time. When the BD-Rate is negative, it means that the bit rate decreases under the same reconstruction quality and the technology has gain; and when the BD-Rate is positive, it means that the bit rate increases under the same reconstruction quality and the technology has a loss. Here, the coding quality of the VVC encoder and the bit rate control algorithm of VVC when QP is given are taken as a comparative basis.

Table 1 shows the performance of this scheme and the bit rate control method of VTM14.0 and 360lib on BD-Rate. Compared with the bit rate control method of vtm, the bit rate is greatly saved, and the bit rate is saved by 8.7% in this method.

Table 2 shows a comparison between this scheme and the rate-distortion optimization algorithm of VTM14.0 on BD-rate, and the bit rate is saved averagely by about 4.3%.

Table 3 shows the size of the coding bit rate control errors of this scheme, VTM14.0 and 360lib The calculation method of the bit error rate is: the absolute error between the actual coding output bit rate and the set target bit rate is divided by the target bit rate. The larger the deviation, the higher the bit error rate, and the lower the bit rate control accuracy. This scheme ensures an extremely low bit rate control error of 0.0891%.

Table 4 shows the performance of this scheme on coding time. The total coding time of all the sequences is reduced from 3972 hours to 3963 hours.