MoQ Media Interop

Protocol Operation

Track Names The publisher selects a namespace of their choosing, and sends an ANNOUNCE message for this namespace. Within the publisher namespace the publisher will offer media tracks named as videoX and audioX where X will be an integer starting at 0. So in case the publisher issues 2 audio tracks and 1 video track, the track names available will be video0, audio0, and audio1. The subscriber will consider all of those tracks belonging to the same namespace as part of the same synchronization group (timestamps aligned to the same timeline).

Mapping Tracks to MoQT Object Model For the video track, the publisher begins a new group at the start of each IDR (so object 0 will be always an IDR Keyframe), and each group contains a single subgroup. Each object has the format described in . For the audio track, the publisher begins a new group with each audio object, and each group contains a single subgroup. Each object has the format described in . TODO: Datagram forwarding preference could be used, but has problems if audio frame does not fit in a single UDP payload.

Timestamps To avoid using fractional numbers and having to deal with rounding errors, timestamps will be expressed with two integers: - timestamp numerator (ex: PTS, DTS, duration) - timebase To convert a timestamp into seconds you just need to: timestamp(s) = timestamp numerator / timebase Example: PTS = 11, timebase = 30 PTS(s) = 11/30 = 0.366666s

Object Format MoQ-MI uses MOQT extension headers to provide metadata that identifies and augemts the media information found in the object payload.

Media type header extension (header extension type = 0x0A) It defines the media type inside object payload (see section IANA in MOQ TODO), and it MUST be present in all objects

Value	Media type
0x0	Video H264 in AVCC
0x1	Audio Opus bitsream
0x2	UTF-8 text
0x3	Audio AAC-LC in MPEG4

Video H264 in AVCC metadata header extension (header extension type = 0x15) It provides video metadata useful to consume the video carried in the payload of the object. The following table specifies the data inside this extesion header.

MOQT Media video h264data header extension It MUST be present in all objects where "media type header extension" is equal to "Video H264 in AVCC"(0x0)

Seq ID Monotonically increasing counter for this media track

PTS Timestamp Indicates PTS in timebase TODO: Varint does NOT accept easily negative, so it could be challenging to encode at start (priming)

DTS Timestamp Not needed if B frames are NOT used, in that case should be same value as PTS. TODO: Varint does NOT accept easily negative, so it could be challenging to encode at start (priming)

Timebase Units used in PTS, DTS, and duration.

Duration Duration in timebase. It will be 0 if not set

Wall Clock EPOCH time in ms when this frame started being captured. It will be 0 if not set

Video H264 in AVCC extradata header extension (Header extension type = 0x0D) Provides extradata needed to start decoding the video stream It MUST be present in all object 0 (start of group) where "media type header extension" is equal to "Video H264 in AVCC"(0x0) AND there has been an update on the encoding paramets (or very start of the stream)

MOQT Media video h264 extradata header extension

Extradata This will be AVCDecoderConfigurationRecord as described in [ISO14496-15:2019] section 5.3.3.1, with field lengthSizeMinusOne = 3 (So length = 4). If any other size length is indicated (in AVCDecoderConfigurationRecord) we should error with “Protocol violation”

Audio Opus bitsream data header extension (Header extension type = 0x0F) It provides audio metadata useful to consume the audio carried in the payload of the object. Following table specifies the data inside this extesion header. It MUST be present in all objects where "media type header extension" is equal to "Audio Opus bitsream"(0x1)

MOQT Media data audio Opus header extension

Seq Id Monotonically increasing counter for this media track

PTS Timestamp Indicates PTS in timebase TODO: Varint does NOT accept easily negative, so it could be challenging to encode at start (priming)

Timebase Units used in PTS, DTS, and duration

Sample Freq Sample frequency used in the original signal (before encoding)

Num Channels Number of channels in the original signal (before encoding)

Duration Duration in timebase. It will be 0 if not set

Wallclock EPOCH time in ms when this frame started being captured. It will be 0 if not set

UTF-8 Text header extension (Header extension type = 0x11)

MOQT UTF-8 Text header extension

Seq Id Monotonically increasing counter for this track

Audio AAC-LC in MPEG4 bitstream data header extension (Header extension type = 0x13)

MOQT Media audio AAC-LC MPEG4 object header

Seq Id Monotonically increasing counter for this media track

PTS Timestamp Indicates PTS in timebase TODO: Varint does NOT accept easily negative, so it could be challenging to encode at start (priming)

Timebase Units used in PTS, DTS, and duration

Sample Freq Sample frequency used in the original signal (before encoding)

Num Channels Number of channels in the original signal (before encoding)

Duration Duration in timebase. It will be 0 if not set

Wallclock EPOCH time in ms when this frame started being captured. It will be 0 if not set