System and method for time-constrained transmission of video in a communication system

The present invention relates to transmission of video signals, and in particular to time-constrained transmission and retransmission of video packets in a communication network.

In packet transmission between a sender and a receiver in a communication network, data packets can be lost or suffer bit errors during transmission on an un-reliable link such as communication channel in wireless local area network (WLAN) environments. In such circumstances, retransmission of lost or damaged packets from the sender to the receiver is required.

Conventionally, there are mainly two approaches for retransmission of data packets that are lost or not received at a receiver correctly. One approach involves retransmitting the data packet from the sender repeatedly until the maximum retry limit is reached. Another retransmission approach involves setting a fixed time deadline for retries such that any number of retransmissions by the sender can only be performed before that time deadline. After the time deadline, the sender will not retransmit a data packet, even if the receiver has not received the data packet correctly. A combination of the above two approaches is possible, wherein retransmission stops whenever either a maximum retry limit, or the time deadline, is reached.

However, since video transmission has inherent time constraints in terms of playback at the receiver, the above retransmission approaches prove problematic. Repeated retransmission of a data packet for video traffic until reaching the maximum retry limit is not effective if the retransmitted packet misses the receiver's time limit for decoding and playback. Such useless retransmission wastes network bandwidth.

Further, setting a fixed deadline for retries is inadequate for transmission of compressed video such as MPEG2. This is because picture frames need to be displayed constantly (25 or 30 frames/second as an example) after being decoded. However, the frame sizes of the compressed video sequence are variable, which means that variable transmission times are needed for different frames. For example, I-frames of MPEG2 can be 10 to 100 times larger than P-frames or B-frames. I-frames need much more time to be transmitted than B-frames or P-frames. Moreover, it is common that one Media Access Control (MAC) packet cannot carry a whole MPEG2 frame. If a MPEG2 frame is segmented into different MAC packets, their allowed retransmission duration may be quite different because the segments have the same decoding and playback deadline, but were transmitted at different times.

If the fixed retransmission deadline is longer than the actual allowed retransmission duration, network bandwidth is wasted due to useless retransmission. If the fixed retransmission deadline is shorter than the actual allowed retransmission duration, video transmission quality may be degraded since more retransmissions are needed to help recover packet error after the retransmission deadline, but before the decoding and playback deadline. There is, therefore, a need for a method and system for time-constrained transmission and retransmission of video packets in a communication network.

The present invention provides a retransmission method and system for video packet transmission between a sender and a receiver, which individually adjusts the retransmission deadline of each packet according to the actual decoding time deadline at the receiver decoder.

Variable retransmission deadlines are set corresponding to the decoding/playback time for each data packet which carries compressed video information. As such, both network efficiency and video transmission quality are improved compared to the conventional retransmission approaches.

These and other features, aspects and advantages of the present invention will become understood with reference to the following description, appended claims and accompanying figures.

FIG. 1 shows a functional block diagram for construction of a MPEG2 transport stream.

FIG. 2 shows a diagrammatical example of constructing MPEG2 transport stream (TS) packets.

FIG. 3 shows a functional block diagram of a system for time-constrained retransmission for MPEG2 high definition (HD) video content, according to an embodiment of the present invention.

FIG. 4 shows an example flowchart of the steps of a communication process implemented by the sender in the system FIG. 3 , according to an embodiment of the present invention.

FIG. 5 shows an example flowchart of the steps of a communication process implemented by the sender in the system FIG. 3 , according to an embodiment of the present invention.

The present invention provides a method and system implementing a process for communication of time-constrained video packets between a sender and a receiver in a communication network. In one embodiment, the communication process individually adjusts the retransmission deadline of each packet (e.g., a MAC packet including video information payload), according to the actual decoding time deadline at the receiver decoder. Accordingly, variable retransmission deadlines are set corresponding to the decoding/playback time for each data packet which carries video information including compressed video. As such, both network efficiency and video transmission quality are improved compared to the conventional retransmission approaches.

An example in implementation of the present invention for a WLAN utilizing an IEEE 802.11n communication protocol is now described. The IEEE 802.11n protocol is described in IEEE P802.11n/D1.0 Draft Amendment to STANDARD [FOR] Information Technology-Telecommunications and information exchange between systems-Local and Metropolitan networks-Specific requirements-Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) specifications: Enhancements for Higher Throughput, March 2006, incorporated herein by reference.

The IEEE 802.11n MAC packet and aggregation format can be used as is. The sender and the receiver implement a frame structure for data transmission therebetween, using packet transmission in a MAC layer and a physical (PHY) layer. The MAC layer of the sender receives a data packet including payload data, and attaches a MAC header thereto, in order to construct a MAC Protocol Data Unit (MPDU). The MAC header includes information such as a source address (SA) and a destination address (DA). The MPDU is a part of a PHY Service Data Unit (PSDU) and is transferred to a PHY layer in the sender (e.g., access point (AP)) to attach a PHY header (i.e., PHY preamble) thereto to construct a PHY Protocol Data Unit (PPDU). The PHY header includes parameters for determining a transmission scheme including a coding/modulation scheme. Typically, the most reliable coding/modulation scheme is applied to a PHY signal field in the PHY header, and an additional cyclic redundancy check (CRC) is added to ensure this information is received correctly at the receiver. The MAC header and payload data are usually treated equally and transmitted using the same coding/modulation scheme, which is less robust than that for the PHY signal field of the PHY header. Further, before transmission as a packet, a preamble is attached to the PPDU, which can include channel estimation and synchronization information.

No timestamp information is required in the MAC header or in the aggregation delimiters of an aggregated MSDU (A-MSDU) or an aggregated MPDU (A-MPDU). The example implementation is described in conjunction with HD video signals using MPEG2 encoding. Specifically, first an overview of the MPEG2 transport stream construction is provided, and then an adaptive retransmission process for time-constrained data packets in conjunction with HD video signals using MPEG2 encoding, according to an embodiment of the present invention, is described.

MPEG2 Transport Stream Construction

HD television (HDTV) broadcasting uses the Advanced Television Systems Committee (ATSC) standard. ATSC HDTV video employs the MPEG2 standard with some constraints. Specifically, ATSC HDTV uses a transport stream (TS) format specified in the MPEG2 standard. As shown in the functional block diagram of a conventional sender (transmitter) 100 in FIG. 1 , according to the MPEG2 specification, a MPEG2 video encoder 102 receives analog video data and generates a compressed video stream, termed elemental stream (ES). FIG. 2 shows a diagram 200 of an example conventional TS packet construction. A video ES comprises all of the video data for a sequence, including the sequence header and all of the subparts of a sequence. An ES carries only one type of data (video or audio) from a single video or audio encoder.

After the encoder 102 ( FIG. 1 ), the ES is packetized by a packetizer 104 , generating a packetized elemental stream (PES). The PES comprises a single ES that has been converted into packets, each packet starting with an added packet header. A PES stream includes only one type of data from one source, e.g., from one video or audio encoder.

PES packets have variable lengths which do not correspond to the fixed packet lengths of transport packets. Indeed, each PES packet may be much longer than a transport packet. When transport packets are formed from a PES packet, the PES packet header is always placed at the beginning of a transport packet payload, immediately following the transport packet header. The remaining PES packet content fills the payloads of successive transport packets until the PES packet is full. The final transport packet is filled to a fixed length by padding it with bytes=0xFF (all ones), as necessary. The different transport packets are then multiplexed by a TS multiplexer 106 .

Each PES packet header includes an 8-bit stream ID identifying the source of the payload. The PES packet header may further include timing references, such as: a Presentation Time Stamp (PTS) which indicates the sender-selected time at which a decoded audio or video access unit is to be presented by the decoder; a Decoding Time Stamp (DTS) which indicates the time at which an access unit is decoded by the decoder; an Elemental Stream Clock Reference (ESCR); etc.

The ATSC standard further constrains PES packets for video. Every video PES packet starts with the beginning of a video access unit. For terrestrial broadcast, each PES packet includes no more than one coded video frame (and may contain one or two coded fields or one frame). Every PES header includes a PTS. A PES packet devoid of video information must include a discontinuity_indicator to signal that a continuity_counter may be discontinuous.

The MPEG protocol packages all data into fixed-size 188-byte TS packets for transport. Video or audio payload data may be placed in a PES packet before the PES packet is segmented into fixed length transport packet payloads. Each transport packet starts with a sync byte=0x47. In the ATSC US terrestrial DTV VSB transmission system, the sync byte is not processed, but is replaced by a different sync symbol especially suited to radio frequency (RF) transmission. The transport packet header includes a 13-bit PID (packet ID), which corresponds to a particular ES of video, audio, or other program element.

Adaptive Retransmission Scheme for Time-Constrained Data Packets

In a system for communication of compressed video such as MPEG2 video, picture/image frames should be played back at the receiver continuously (e.g., 25 or 30 frames/second) after being decoded. However, as described, since the frame sizes of compressed video sequences are variable, variable transmission times for different frames are utilized. It is common that one MAC packet cannot carry a whole MPEG2 frame. If a MPEG2 frame is segmented to different MAC packets, the allowed retransmission duration of each MAC packet may be quite different. This is because the segments have the same decoding and playback deadline but were transmitted at different times.

In a communication process according to the present invention, if retransmitting a packet will not result in arrival of the packet at the receiver by the playback (i.e., decoding/presentation) deadline of the receiver decoder, then that packet is not retransmitted by the sender. For each MAC packet carrying MPEG2 video information, the MAC packet retransmission deadline is set to the required decoding and presentation time (Td) of the video frame to which the MAC packet belongs, minus the transmission time (Tp) of the MAC packet. The Tp of a MAC packet (header and payload) can be determined as: the sum of a PHY header transmission time and the ratio of a MAC packet length to data rate, such that Tp=the PHY header transmission time+(MAC packet length divided by the data rate). The data rate is the selected transmission data rate for the MAC packet.

The time Td for the video frame to which the MAC packet belongs, is determined as follows. For MPEG2 standardization, the receiver follows the playback deadline set by the sender (i.e., PTS time) in the PES header. As such, Td is a function of PTS. Based on the ATSC HDTV standard, every coded video frame should be carried in one or several separated PES packets. Further, PTS information must be carried in every PES header. Therefore, the sender can obtain the PTS information for every video frame if it can parse the PES header, such that: Td=PTS+Tos, wherein Tos is the time offset between the starting time when the beginning of the video sequence is encoded, and the starting time when the beginning of the compressed video sequence is passed to the MAC layer of the communication component of the sender. Timestamps at the first PES packet and/or TS packet carry information indicating the starting time when the beginning of the video sequence is encoded.

Tos may be set to zero in case of live video encoding and transmission. If a video sequence is pre-encoded and pre-packaged into PES and TS streams, Tos is a fixed value reflecting the time difference between encoding and transmission. If the communication component can directly obtain a PES stream from applications and packetize the PES stream into MAC packets, the PTS information can be obtained by parsing the PES header. However, typically the communication component of the sender can only obtain a TS stream from applications. The PES header is encapsulated into the payload field of the TS packet and cannot be directly accessed by the communication component by parsing the TS header.

A bit in the TS header (i.e., payload start indicator bit) indicates the start of a PES packet for the current TS packet. Further, the PES header is always placed in the TS packet with a payload start indicator set to “1”. Therefore, the sender can parse the payload information of the TS packets with a payload start indicator setting to “1” to obtain the PTS of the video frame to which the TS packet belongs. Multiple TS packets belonging to the same PES can be aggregated into one MAC packet. The communication component can set the retransmission deadline of the MAC packet according to the PTS information of the corresponding PES packet.

Further, in some video transmission implementations, an amount of buffing for video pre-loading is allowed to compensate delay jitter and packet corruption caused by unreliable transmission channel. The required decoding and presentation time, of the video frame to which the MAC packet belongs, is denoted as Td. The time Td for a video frame may be later than the nominal decoding and PTS. The gap between these two times (Td−PTS), is denoted as Δb.

At the receiver, Δb is calculated by subtracting PTS of the most current video frame from the actual starting time of decoding and presentation time, Td, of the same video frame. The value of Δb is fed back to the sender regularly to adjust the retransmission deadline of the packets belonging to the succeeding video frames. The sender uses Δb to estimate the maximum allowed receiver buffering time of a MAC packet before being decoded. With this estimation, the sender can set the retransmission deadline of a MAC packet more accurately.

Process for Transmission and Retransmission of MPEG2 TS Streams

An example implementation of a process for transmission and retransmission of MPEG2 TS streams between a sender and a receiver, according to the present invention, is described below. It is assumed that system clocks between the sender and the receiver are time synchronized with one another.

FIG. 3 shows a functional block diagram of an example system 300 implementing an embodiment of the present invention, including a sender 302 and a receiver 304 that transmit packets via a communication channel 306 (e.g., wireless channel implementing the IEEE 802.11n communication protocol). The sender 302 includes: a MPEG2 encoder 308 , a packetizer 310 , a TS multiplexer 312 , a MAC packetizer 330 , a retransmission control 314 and a packet buffer 316 . The receiver 304 includes: a MAC de-packetizer 332 , a TS demultiplexer 318 , a de-packetizer 320 , a MPEG2 decoder 322 , and a Δb calculator 324 . In this example, the sender 302 places the MPEG2 TS streams into the MAC frames and sends the MAC frames to the receiver 304 .

The sender 302 and the receiver 304 implement a method of time-constrained retransmission for MPEG2 HD content according to the processes 400 and 500 represented by the flowcharts in FIGS. 4 and 5 , respectively, as described below. Initially, the MPEG2 encoder 308 encodes the incoming analog video data into ES data, the packetizer 310 converts the ES data payload into PES packets, and the TS multiplexer 312 generates MPEG2 TS packets. Referring to the process 400 in FIG. 4 , the sender 302 implements the following further steps:

Step 402 : The MAC packetizer 330 parses the payload of TS packets with payload start indicator setting to “1”, and obtains the PTS of the corresponding PES. For the first PES/TS of the video sequence, in addition to PTS information, the sender also obtains System Clock Reference (SCR) and/or Program Clock Reference (PCR) information. Tos can then be calculated as the difference between SCR and the starting transmission time of the first TS packet. If pre-loading is allowed, the initial Δb is set to the pre-loading time plus Tos. Usually the pre-loading duration is chosen between 100 ms and 10 seconds decided by the video transmission application. Step 404 : The MAC packetizer 330 assembles one or multiple TS packets into a MAC packet and transmits each MAC packet to the receiver via the communication channel 306 . Step 406 : The sender 302 maintains a copy of the MAC packet at its sending buffer 316 before the sender 302 receives a positive acknowledgement indicating that the MAC packet arrived at the receiver 304 correctly. The sender sets the retransmission deadline of a MAC packet to: PTS+Δb−Tp. Different MAC packets belonging to the same video frame have the same retransmission deadline (i.e., PTS+Δb−Tp), even though their starting transmission times are different. The retransmission deadline of all MAC packets in the same video frame is dynamically updated when the sender receives a new value of Δb fed back from the receiver. Since for MPEG2 the decoding of a current P-frame, or B-frame, refers to its pervious I-frame, the data of a compressed I-frame is useful for the following B-frame or P-frame even if the current I-frame data already missed the decoding deadline upon arrival at the receiver. When the data misses the decoding deadline of the receiver, the deadline of a MAC packet is set to: PTS+Δb−Tp+1/f, wherein f is the frame rate for the video playback. As such, in that case, the retransmission deadline is extended by 1/f (one frame time) to allow processing of following B-frame or P-frame. Step 408 : If the sender 302 receives a negative acknowledgement (ACK) indicating that the receiver 304 did not receive the MAC packet correctly, or the retransmission timeout is triggered before receiving any acknowledgement (e.g., receiver did not send any ACK), the retransmission control 314 checks the retransmission deadline of the MAC frame to determine if there is time left to retransmit the MAC packet. If the retransmission deadline cannot be met, then the sender ceases attempts to retransmit the MAC packet. As far as buffering status at the sender is concerned, the sender retransmits a copy of the MAC packet to the receiver if the sender determines that it can perform the retransmission before the retransmission deadline. Otherwise, the sender removes the copy of the MAC packet from the buffer 316 and ceases retransmission attempts of that MAC packet. Step 410 : After the sender 302 obtains an updated Δb feedback from the receiver 304 , the retransmission control 314 updates the Δb value for future MAC packet transmission and retransmissions.

Referring to the flowchart 500 in FIG. 5 , the sender 302 implements the following steps:

Step 502 : When the receiver 304 receives a MAC packet from the sender 302 via the channel 306 , the MAC de-packetizer 332 performs error detection (e.g., CRC verification) for the received MAC packet. If no errors are detected, the receiver sends back a positive acknowledgement (PACK) to the sender. Otherwise, the receiver sends back a negative acknowledgement (NACK), or no acknowledgement, to the sender, depending on the acknowledgement mechanism being used. A received MAC packet with an incorrect CRC check may or may not be used. Step 504 : After obtaining the received MAC packet payload with a correct CRC check, the MAC de-packetizer 332 segments the payload to individual TS packets. Then, the TS demultiplexer 318 can re-assemble the PES stream from payload of TS packets. Step 506 : When a PES packet is generated from TS packets by the TS demultiplexer 318 , and the PES packet is ready to be passed to the video decoder 322 at the receiver, the PTS of the PES packet is recorded. The de-packetizer 320 is used to recover ES data from PES packets, wherein the ES data stream is provided to the decoder 322 . Further, the start time (Ta) of actual decoding of the video frame is recorded. The recorded PTS and the recorded start time (Ta) of actual decoding are used by the Δb calculator 324 to calculate a new Δb as: Δb=Ta−PTS. The values Td and PTS are indicative of the occupation level of a receive buffer at the receiver. Step 508 : If the Δb of the currently decoded video frame is different from Δb of the previous video frame, the receiver sends back an updated value of Δb to the sender, for example piggybacked with other MAC packets such as ACK packets.

In the above example, Δb is calculated at the receiver by subtracting PTS of the most current video frame from the actual starting decoding and presentation time Td of the same video frame. The sender then sets the retransmission deadline of its MAC packet to: PTS+Δb−Tp. An alternative scheme for the sender is to set the retransmission deadline of its MAC packet to: PTS+Δb′−Tp, wherein Δb′ is the average of Δbs of the recent video frames decoded.

MPEG Header Tables

In Tables 1 and 2 below, MPEG header tables are provided for reference.

The retransmission method and system for time-constrained video packet transmission between a sender and a receiver, provides that the retransmission deadline of each packet is individually adjusted based on the actual decoding time deadline at the receiver decoder. Accordingly, variable retransmission deadlines are set corresponding to the decoding and playback time for each data packet which carries compressed video information. As such, both network efficiency and video transmission quality are improved compared to the conventional retransmission approaches.

As is known to those skilled in the art, the aforementioned example architectures described above, according to the present invention, can be implemented in many ways, such as program instructions for execution by a processor, as logic circuits, as an application specific integrated circuit, as firmware, etc.

The present invention has been described in considerable detail with reference to certain preferred versions thereof; however, other versions are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the preferred versions contained herein.