]> ByteDance

lideping.byter@bytedance.com

ART avtcore RTP,Webrtc,BFrame,Video This document describes an RTP header extension used to convey decoding time information about video when Bi-directional predicted frames exist.It adds CompositionTime(CTS) as value so that receiver can decode video with correct sequence.

Introduction As video codec, H264/HEVC is widely used in RTP base system. Those codec support I-Frame, B-Frame, and P-frame . Most RTP systems do not support B-Frame, while B-Frame is widely used in streaming systems, with the rapid deploy of Real Time Communication(RTC) in low latency streaming scenario, support for Bi-directional predicted frames in RTP base system are necessary. Video streams contain a lot of details, including timestamps, so a decoder knows how to handle the content properly. The DTS(DecodingTimeStamp) decides when a frame has to be decoded, while the PTS(PresentationTimeStamp) describes when a frame has to be presented.This difference becomes important when using B-frames, which are frames that can have references to frames in the past, but also to frames in the future. Given that, there will be frames in the future, which a decoder needs to decode first in order to use them as reference. Therefore, decoder needs DTS when B-frames exist, while, the RTP timestamp reflects the presentation time(PTS) only. This document specifies an RTP extension header that allows video rtp senders deliver CTS(CompositionTime) to rtp receiver . The CTS value is PTS minus DTS. Therefore , the rtp receiver gets DTS value via RTP timestamp adding CTS value. This new header extension uses the general mechanism for RTP header extensions as described in ()]. Rtp sender only needs to add CTS to the first rtp packet when the video frame contains several packets, which reduces overhead.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

• RTP: Real-time Transport Protocol (RFC 3550)
• RTCP: RTP Control Protocol (RFC 3550)
• RTCP RR: RTCP Receiver Report
• RTCP SR: RTCP Sender Report
• SDP: Session Description Protocol (RFC 4566)
• Clock Rate: The multiplier used to convert from a wallclock value in seconds to an equivalent RTP timestamp value (without the fixed random offset). Note that RFC 3550 uses various terms like "clock frequency", "media clock rate", "timestamp unit", "timestamp frequency", and "RTP timestamp clock rate" as synonymous to clock rate.
• RTP Sender: A logical network element that sends RTP packets, sends RTCP SR packets, and receives RTCP reception report blocks.
• RTP Receiver: A logical network element that receives RTP packets, receives RTCP SR packets, and sends RTCP reception report blocks.
• RTC: Real Time Communication
• PTS: Video Presentation TimeStamp
• DTS: Video Decoding TimeStamp
• CTS: Video CompositionTime

RTP header extension format The general RTP payload format follows the RTP header format () and generic RTP header extensions (), RTP header extension MAY encoded using the one-byte header or two-byte header as described in (). The two-byte header format is used as an example in this memo. The following RTP header extension is RECOMMENDED. The ID is assigned per (), and format is shown below.

extension format

ID: extension id. cts: PTS minus DTS and divide by 90 (Video Clock Rate)

Video rtp sender The video sender here MAY be video client or middle box perform RTP switch. Video client MAY encode video with B-frame, it SHOULD add this rtp header extension in the rtp packetization module . Only adding in the first rtp packet is RECOMMENDED when the video frame contains multi rtp packets, which will reduce overhead. The middle box MAY perform RTMP or other streaming video protocols translate to rtp streams work, it SHOULD add this header extension when streaming video contains B-frame.

Video rtp receiver The video rtp receiver here is a client which decodes video . It SHOULD extract cts value when this extension exists , and calculate DTS value with rtp timestamp(PTS) and CTS. 90 is video clock rate, Video receiver construction frame and put to jitter buffer, decoder MUST decode frame by DTS sequence, and video render module MUST render the decoded frame with PTS sequence, which come from rtp timestamp.

Usage considerations In practice, when receiver that decode video does not support B-frame, In order to successfully decode an incoming video stream, it is RECOMMENDED An RTP middle box discard B-frame when video rtp sender contains B-frame, the decoder at the Endpoint SHOULD add whether it support video B-frame capability in SDP payload format specific paramaters(a=fmtp), and follow the Offer/Answer procedure describe in ().

Session Description Protocol (SDP) Signaling The URI for declaring this header extension in an extmap attribute is "urn:ietf:params:rtp-hdrext:CompositionTime". It does not contain any extension attributes, It follows the standard mechanism described in () An example attribute line in SDP:

Security Considerations The security considerations of the RTP specification () and the general mechanism for RTP header extensions () apply. and all the security considerations of typologies () () for these two types of RTP intermediaries are applicable to this header extension. Security considerations for SDP are described in the corresponding section in (), In the Secure Real-time Transport Protocol (SRTP) (), RTP header extensions are authenticated but not encrypted. When this header extension is used, cts are therefore visible on a frame-by-frame basis to an attacker passively observing the video stream, In scenarios where this is a concern, additional mechanisms MUST be used to protect the confidentiality of the header extension. This mechanism could be header extension encryption (), or a lower-level security and authentication mechanism such as IPsec ().

IANA Considerations IANA has registered the following entry in the "RTP Compact Header Extensions" registry: Extension URI: uri:ietf:rtc:rtp-hdrext:video:CompositionTime Description: video B frame compositionTime Contact: lideping.byter@bytedance.com

Acknowledgements

Normative References This memorandum describes RTP, the real-time transport protocol. RTP provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. RTP does not address resource reservation and does not guarantee quality-of- service for real-time services. The data transport is augmented by a control protocol (RTCP) to allow monitoring of the data delivery in a manner scalable to large multicast networks, and to provide minimal control and identification functionality. RTP and RTCP are designed to be independent of the underlying transport and network layers. The protocol supports the use of RTP-level translators and mixers. Most of the text in this memorandum is identical to RFC 1889 which it obsoletes. There are no changes in the packet formats on the wire, only changes to the rules and algorithms governing how the protocol is used. The biggest change is an enhancement to the scalable timer algorithm for calculating when to send RTCP packets in order to minimize transmission in excess of the intended rate when many participants join a session simultaneously. [STANDARDS-TRACK] This document provides a general mechanism to use the header extension feature of RTP (the Real-time Transport Protocol). It provides the option to use a small number of small extensions in each RTP packet, where the universe of possible extensions is large and registration is decentralized. The actual extensions in use in a session are signaled in the setup information for that session. This document obsoletes RFC 5285. This document provides a general mechanism to use the header extension feature of RTP (the Real-Time Transport Protocol). It provides the option to use a small number of small extensions in each RTP packet, where the universe of possible extensions is large and registration is de-centralized. The actual extensions in use in a session are signaled in the setup information for that session. [STANDARDS-TRACK] In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This document discusses point-to-point and multi-endpoint topologies used in environments based on the Real-time Transport Protocol (RTP). In particular, centralized topologies commonly employed in the video conferencing industry are mapped to the RTP terminology. The Real-time Transport Protocol (RTP) is used in a large number of different application domains and environments. This heterogeneity implies that different security mechanisms are needed to provide services such as confidentiality, integrity, and source authentication of RTP and RTP Control Protocol (RTCP) packets suitable for the various environments. The range of solutions makes it difficult for RTP-based application developers to pick the most suitable mechanism. This document provides an overview of a number of security solutions for RTP and gives guidance for developers on how to choose the appropriate security mechanism. This memo defines the Session Description Protocol (SDP). SDP is intended for describing multimedia sessions for the purposes of session announcement, session invitation, and other forms of multimedia session initiation. This document obsoletes RFC 4566. This document describes the Secure Real-time Transport Protocol (SRTP), a profile of the Real-time Transport Protocol (RTP), which can provide confidentiality, message authentication, and replay protection to the RTP traffic and to the control traffic for RTP, the Real-time Transport Control Protocol (RTCP). [STANDARDS-TRACK] The Secure Real-time Transport Protocol (SRTP) provides authentication, but not encryption, of the headers of Real-time Transport Protocol (RTP) packets. However, RTP header extensions may carry sensitive information for which participants in multimedia sessions want confidentiality. This document provides a mechanism, extending the mechanisms of SRTP, to selectively encrypt RTP header extensions in SRTP. This document updates RFC 3711, the Secure Real-time Transport Protocol specification, to require that all future SRTP encryption transforms specify how RTP header extensions are to be encrypted. This document describes an updated version of the "Security Architecture for IP", which is designed to provide security services for traffic at the IP layer. This document obsoletes RFC 2401 (November 1998). [STANDARDS-TRACK]