]> Akamai

wilaw@akamai.com

Twitch

kixelated@gmail.com

Google

vasilvv@google.com

Cisco

snandaku@cisco.com

Meta

ikir@meta.com

Applications and Real-Time Media Over QUIC MoQ MoQTransport WARP This document specifies the WARP Streaming Format, designed to operate on Media Over QUIC Transport. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Media Over QUIC Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction WARP Streaming Format (WARP) is a media format designed to deliver LOC compliant media content over Media Over QUIC Transport (MOQT) . WARP works by fragmenting the bitstream into objects that can be independently transmitted. WARP leverages a catalog format to describe the output of the original publisher. WARP specifies how content should be packaged and signaled, defines how the catalog communicates the content, specifies prioritization strategies for real-time and workflows for beginning and terminating broadcasts. WARP also details how end-subscribers may perform adaptive bitrate switching. WARP is targeted at real-time and interactive levels of live latency, as well as VOD content. This document describes version 1 of the streaming format.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document uses the conventions detailed in Section 1.3 of when describing the binary encoding.

Scope The purpose of WARP is to provide an interoperable media streaming format operating over . Interoperability implies that:

• An original publisher can package incoming media content into tracks, prepare a catalog and annouce the availability of the content to a MOQT relay. Media content refers to audio and video data, as well as ancillary data such as captions, subtitles, accessibility and other timed-text data.
• A MOQT relay can process the annoucement as well as cache and propagate the tracks, both to other relays or to the final subscriber.
• A final subscriber can parse the catalog, request tracks, decode and render the received media data.

WARP is intended to provide a format for delivering commercial media content. To that end, the following features are within scope:

• Video codecs - all codecs supported by
• Audio codecs - all audio codecs supported by
• Catalog track - describes the availability and characteristics of content produced by the original publisher.
• Timeline track - describes the relationship between MOQT Group and Object IDs to media time.
• Token-based authorization and access control
• Captions + Subtitles - support for and transmission
• Latency support across multiple regimes (thresholds are informative only and describe the delay between the original publisher placing the content on the wire and the final subscriber rendering it)
• Real-time - less than 500ms
• Interactive - between 500ms and 2500ms
• Standard - above 2500ms
• VOD latency - content that was previously produced, is no longer live and is available indefinitely.
• Content encryption
• ABR between time-synced tracks - subscribers may switch between between tracks at different quality levels in order to maximize visual or audio quality under fconditions of throughput variability.
• Capable of delivering interstitial advertising.
• Logs and analytics management - support for the reporting of client-side QoE and relay delivery actions.

Initial verisons of WARP will prioritize basic features necessary to exercise interoperability across delivery systems. Later versions will add commercially necessary features.

Media packaging WARP delivers LOC packaged media bitstreams.

LOC packaging This specification references Low Overhead Container (LOC) to define how audio and video content is packaged. With this packaging mode, each EncodedAudioChunk or EncodedVideoChunk sample is placed in a separate MOQT Object. Samples that belong to the same Group of Pictures (GOP) MUST be placed within the same MOQT Group. When LOC packaging is used for a track, the catalog packaging attribute () MUST be present and it MUST be populated with a value of "loc".

Time-alignment WARP Tracks MAY be time-aligned. Those that are, are subject to the following requirements:

• Time-aligned tracks MUST be advertised in the catalog as belonging to a common render group.
• The render duration of the first media object of each equally numbered MOQT Group, after decoding, MUST have overlapping presentation time.

A consequence of this restriction is that a WARP receiver SHOULD be able to cleanly switch between time-aligned media tracks at group boundaries.

Content protection and encryption ToDo - content protection for LOC-packaged content.

Catalog A Catalog is a MOQT Track that provides information about the other tracks being produced by a WARP publisher. A Catalog is used by WARP publishers for advertising their output and for subscribers in consuming that output. The payload of the Catalog object is opaque to Relays and can be end-to-end encrypted. The Catalog provides the names and namespaces of the tracks being produced, along with therelationship between tracks, properties of the tracks that consumers may use for selection and any relevant initialization data. The catalog track MUST have a case-sensitive Track Name of "catalog". A catalog object MAY be independent of other catalog objects or it MAY represent a delta update of a prior catalog object. The first catalog object published within a new group MUST be independent. A catalog object SHOULD only be published only when the availability of tracks changes. Each catalog update MUST be mapped to a discreet MOQT Object.

Catalog Fields A catalog is a JSON document, comprised of a series of mandatory and optional fields. At a minimum, a catalog MUST provide all mandatory fields. A producer MAY add additional fields to the ones described in this draft. Custom field names MUST NOT collide with field names described in this draft. The order of field names within the JSON document is not important. A parser MUST ignore fields it does not understand. Table 1 provides an overview of all fields defined by this document.

Field	Name	Definition
WARP version	version
Delta update	deltaUpdate
Add tracks	addTracks
Remove tracks	removeTracks
Clone tracks	cloneTracks
Generated at	generatedAt
Tracks	tracks
Track namespace	namespace
Track name	name
Packaging	packaging
Is Live	isLive
Track role	role
Track label	label
Render group	renderGroup
Alternate group	altGroup
Initialization data	initData
Dependencies	depends
Temporal ID	temporalId
Spatial ID	spatialId
Codec	codec
Mime type	mimeType
Framerate	framerate
Timescale	timescale
Bitrate	bitrate
Width	width
Height	height
Audio sample rate	samplerate
Channel configuration	channelConfig
Display width	displayWidth
Display height	displayHeight
Language	lang
Parent name	parentName
Track duration	trackDuration

Table 2 defines the allowed locations for these fields within the document

Location	Allowed locations for the field
R	The Root of the JSON object
T	Track object

WARP version Location: R Required: Yes JSON Type: Number Specifies the version of WARP referenced by this catalog. There is no guarantee that future catalog versions are backwards compatible and field definitions and interpretation may change between versions. A subscriber MUST NOT attempt to parse a catalog version which it does not understand.

Delta update Location: R Required: Optional JSON Type: Boolean A Boolean that if true indicates that this catalog object represents a delta (or partial) update. A delta update has a restricted set of fields and special processing rules - see . This value SHOULD NOT be added to a catalog if it is false.

Add tracks Location: R Required: Optional JSON Type: Array Indicates a delta processing instruction to add new tracks. The value of this field is an Array of track objects .

Remove tracks Location: R Required: Optional JSON Type: Array Indicates a delta processing instruction to remove new tracks. The value of this field is an Array of track objects . Each track object MUST include a Track Name field, MAY include a Track Namespace field and MUST NOT hold any other fields.

Clone tracks Location: R Required: Optional JSON Type: Array Indicates a delta processing instruction to clone new tracks from previously declared tracks. The value of this field is an Array of track objects . Each track object MUST include a Parent Name field.

Generated at Location: R Required: Optional JSON Type: Number The wallclock time at which this catalog instance was generated, expressed as the number of milliseconds that have elapsed since January 1, 1970 (midnight UTC/GMT). This field SHOULD NOT be included if the isLive field is false.

Tracks Location: R Required: Yes JSON Type: Array An array of track objects .

Tracks object A track object is JSON Object containing a collection of fields whose location is specified 'T' in Table 2.

Track namespace Location: T Required: Optional JSON Type: String The name space under which the track name is defined. See section 2.3 of . The track namespace is optional. If it is not declared within a track, then each track MUST inherit the namespace of the catalog track. A namespace declared in a track object overwrites any inherited name space.

Track name Location: T Required: Yes JSON Type: String A string defining the name of the track. See section 2.3 of . Within the catalog, track names MUST be unique per namespace.

Packaging Location: T Required: Yes JSON Type: String A string defining the type of payload encapsulation. Allowed values are strings as defined in Table 3. Table 3: Allowed packaging values

Name	Value	Reference
LOC	loc	See RFC XXXX
Timeline	timeline	See

Track role Location: T Required: Optional JSON Type: String A string defining the role of content carried by the track. Reserved roles are described in Table 4. These role values are case-sensitive. This role field MAY be used in conjunction with the Mimetype to fully describe the content of the track. Table 4: Reserved track roles

Role	Description
audiodescription	An audio description for visually impaired users
video	Visual content
audio	Audio content
timeline	A WARP timeline
caption	A textual representation of the audio track
subtitle	A transcription of the spoken dialogue
signlanguage	A visual track for hearing impaired users.

Custom roles MAY be used as long as they do not collide with the reserved roles.

Is Live Location: T Required: Required JSON Type: Boolean True if new Objects will be added to the track. False if no new Objects will be added to the track. This is sent under two possible conditions: * the publisher of a previously live track has ended the track. * the track is Video-On-Demand (VOD) and was never live.

Track label Location: TF Required: Optional JSON Type: String A string defining a human-readable label for the track. Examples might be "Overhead camera view" or "Deutscher Kommentar". Note that the spec requires UTF-8 support by decoders.

Render group Location: T Required: Optional JSON Type: Number An integer specifying a group of tracks which are designed to be rendered together. Tracks with the same group number SHOULD be rendered simultaneously, are usually time-aligned and are designed to accompany one another. A common example would be tying together audio and video tracks.

Alternate group Location: T Required: Optional JSON Type: Number An integer specifying a group of tracks which are alternate versions of one-another. Alternate tracks represent the same media content, but differ in their selection properties. Alternate tracks SHOULD have matching framerate and media time sequences. A subscriber typically subscribes to one track from a set of tracks specifying the same alternate group number. A common example would be a set video tracks of the same content offered in alternate bitrates.

Initialization data Location: T Required: Optional JSON Type: String A string holding Base64 encoded initialization data for the track.

Dependencies Location: T Required: Optional JSON Type: Array Certain tracks may depend on other tracks for decoding. Dependencies holds an array of track names on which the current track is dependent. Since only the track name is signaled, the namespace of the dependencies is assumed to match that of the track declaring the dependencies.

Temporal ID Location: T Required: Optional JSON Type: Number A number identifying the temporal layer/sub-layer encoding of the track, starting with 0 for the base layer, and increasing with higher temporal fidelity.

Spatial ID Location: T Required: Optional JSON Type: Number A number identifying the spatial layer encoding of the track, starting with 0 for the base layer, and increasing with higher fidelity.

Codec Location: T Required: Optional JSON Type: String A string defining the codec used to encode the track. For LOC packaged content, the string codec registrations are defined in Sect 3 and Section 4 of .

Mimetype Location: T Required: Optional JSON Type: String A string defining the mime type of the track.

Framerate Location: T Required: Optional JSON Type: Number A number defining the video framerate of the track, expressed as frames per second.

Timescale Location: T Required: Optional JSON Type: Number The number of time units that pass per second.

Bitrate Location: T Required: Optional JSON Type: Number A number defining the bitrate of track, expressed in bits per second.

Width Location: T Required: Optional JSON Type: Number A number expressing the encoded width of the video frames in pixels.

Height Location: T Required: Optional JSON Type: Number A number expressing the encoded height of the video frames in pixels.

Audio sample rate Location: T Required: Optional JSON Type: Number The number of audio frame samples per second. This property SHOULD only accompany audio codecs.

Channel configuration Location: T Required: Optional JSON Type: String A string specifying the audio channel configuration. This property SHOULD only accompany audio codecs. A string is used in order to provide the flexibility to describe complex channel configurations for multi-channel and Next Generation Audio schemas.

Display width Location: T Required: Optional JSON Type: Number A number expressing the intended display width of the track content in pixels.

Display height Location: T Required: Optional JSON Type: Number A number expressing the intended display height of the track content in pixels.

Language Location: T Required: Optional JSON Type: String A string defining the dominant language of the track. The string MUST be one of the standard Tags for Identifying Languages as defined by .

Parent name Location: T Required: Optional JSON Type: String A string defining the parent track name to be cloned. This field MUST only be included inside a Clone tracks object.

Track duration Location: T Required: Optional JSON Type: Number The duration of the track expressed in integer milliseconds. This field MUST NOT be included if the isLive field value is false.

Delta updates A catalog update might contain incremental changes. This is a useful property if many tracks may be initially declared but then there are small changes to a subset of tracks. The producer can issue a delta update to describe these changes. Changes are described incrementally, meaning that a delta update can itself modify a prior delta update. A restricted set of operations are allowed with each delta update: * Add a new track that has not previously been declared. * Add a new track by cloning a previously declared track. * Remove a track that has been previously declared. The following rules are to be followed in constructing and processing delta updates:

• A delta update MUST include the Delta Update field set to true.
• A delta update catalog MUST contain at least one instance of Add tracks , Remove tracks or Clone Tracks fields and MAY contain more. It MUST NOT contain an instance of a Tracks field or a WARP version field.
• The Add, Delete and Clone operations are applied sequentially in the order they are declared in the document. Each operation in the sequence is applied to the target document; the resulting document becomes the target of the next operation. Evaluation continues until all operations are successfully applied.
• A Cloned track inherits all the attributes of the track defined by the Parent Name , except the Track Name which MUST be new. Attributes redefined in the cloning Object overwrite inherited values.
• The tuple of Track Namespace and Track Name defines a fixed set of Track attributes which MUST NOT be modified after being declared. To modify any attribute, a new track with a different Namespace|Name tuple is created by Adding or Cloning and then the old track is removed.

Catalog Examples The following section provides non-normative JSON examples of various catalogs compliant with this draft.

Time-aligned Audio/Video Tracks with single quality This example shows a catalog for a media producer capable of sending LOC packaged, time-aligned audio and video tracks.

Simulcast video tracks - 3 alternate qualities along with audio This example shows catalog for a media producer capable of sending 3 time-aligned video tracks for high definition, low definition and medium definition video qualities, along with an audio track. In this example the namespace is absent, which infers that each track must inherit the namespace of the catalog.

SVC video tracks with 2 spatial and 2 temporal qualities This example shows catalog for a media producer capable of sending scalable video codec with 2 spatial and 2 temporal layers with a dependency relation as shown below: | S1T1 | | | 1080p30 | | +----------+ | ^ | | +----------+ | | S1TO | | | 1080p15 | | +----------+ +-----+----+ ^ | SOT1 | | | 480p30 | | +----------+ | ^ +----------+ | | SOTO | | | 480p15 |---------+ +----------+ ]]> The corresponding catalog uses "depends" attribute to express the track relationships.

Delta update - adding two tracks This example shows the catalog delta update for a media producer adding two tracks to an established video conference. One track is newly declared, the other is cloned from a previous track.

Delta update removing tracks This example shows a delta update for a media producer removing two tracks from an established video conference.

Time-aligned Audio/Video Tracks with custom field values This example shows catalog for a media producer capable of sending LOC packaged, time-aligned audio and video tracks along with custom fields in each track description.

Time-aligned VOD Audio/Video Tracks This example shows catalog for a media producer offering VOD (video on-demand) non-live content. The content is LOC packaged, and includes time-aligned audio and video tracks.

Media transmission The MOQT Groups and MOQT Objects need to be mapped to MOQT Streams. Irrespective of the in place, each MOQT Object MUST be mapped to a new MOQT Stream.

Group numbering The Group ID of the first Group published in a track at application startup MUST be a unique integer that will not repeat in the future. One approach to achieve this is to set the initial Group ID to the creation time of the first Object in the group, represented as the number of milliseconds since the Unix epoch, rounded to the nearest millisecond. This ensures that republishing the same track in the future, such as after a loss of connectivity or an encoder restart, will not result in smaller or duplicate Group IDs for the same track name. Note that this method does not prevent duplication if more than 1000 groups are published per second. Each subsequent Group ID MUST increase by 1. If a publisher is able to maintain state across a republish, it MUST signal the gap in Group IDs using the MOQT Prior Group ID Gap Extension header.

Timeline track The timeline track provides data about the previously published groups and their relationship to wallclock time, media time and associated timed-metadata. Timeline tracks allow players to seek to precise points behind the live head in a live broadcast, or for random access in a VOD asset. A timeline track might also be used to insert events at media times which do not correlate with Object boundaries. Timeline tracks are optional. Multiple timeline tracks can exist inside a catalog.

Timeline track payload The payload of a timeline track is a UTF-8 encoded CSV text file. This payload is formatted according to RFC4180 "Common Format and MIME Type for Comma-Separated Values (CSV)" Files . The separator is a comma and each line is separated by a carriage return. The mime-type of a timeline track MUST be specified as "text/csv" in the catalog. Each timeline track begins with a header row of MEDIA_PTS,GROUP_ID,OBJECT_ID, WALLCLOCK,METADATA. This row defines the 5 columns of data within each record.

• MEDIA_PTS: a media timestamp rounded to the nearest millisecond. This entry MUST NOT be empty. If the Object ID entry is present, then this value MUST match the media presentation timestamp of the first media sample in the referenced Object.
• GROUP_ID: the MOQT Group ID. This entry MAY be empty.
• OBJECT_ID: the MOQT Object ID. This entry MAY be empty.
• WALLCLOCK: the wallclock time at which the media was encoded, expressed as the number of milliseconds that have elapsed since January 1, 1970 (midnight UTC/GMT). For VOD assets, or if the wallclock time is not known, the value SHOULD be 0.
• METADATA: a flexible field holding arbitrary string metadata. This field may be empty. If not empty, it MUST be enclosed in double quotes. A double-quote appearing inside this field MUST be escaped by preceding it with another double quote.

Timeline Catalog requirements A timeline track MUST carry a 'type' identifier in the Catalog with a value of "timeline". A timeline track MUST carry a 'dependes' attribute which contains an array of all track names to which the timeline track applies.

Timeline track updating. The publisher MUST publish a complete timeline in the first MOQT Object of each MOQT Group of a timeline track. The publisher MAY publish incremental updates in the second and subsequent Objects within each GROUP. Incremental updates only contain timeline events since the last timeline Object. Group duration SHOULD not exceed 30 seconds inside a timeline track.

Workflow A WARP publisher MUST publish a catalog track object before publishing any media track objects. At the completion of a session, a publisher MUST publish a catalog update that removes all currently active tracks. This action SHOULD be interpreted by receivers to mean that the publish session is complete.

Security Considerations ToDo

IANA Considerations This document creates a new entry in the "MoQ Streaming Format" Registry (see Sect 8). The type value is 0x001, the name is "WARP Streaming Format" and the RFC is XXX.

Normative References Cisco Google Google Meta This document defines the core behavior for Media over QUIC Transport (MOQT), a media transport protocol designed to operate over QUIC and WebTransport, which have similar functionality. MOQT allows a producer of media to publish data and have it consumed via subscription by a multiplicity of endpoints. It supports intermediate content distribution networks and is designed for high scale and low latency distribution. Cisco Cisco Microsoft This specification describes a media container format for encoded and encrypted audio and video media data to be used primarily for interactive Media over QUIC Transport (MOQT) [MoQTransport], with the goal of it being a low-overhead format. It further defines the LOC Streaming Format for the MOQ Common Catalog format [MoQCatalog] for publishers to annouce and describe their LOC tracks and for subscribers to consume them. The specification also provides examples to aid application developers for building media applications over MOQT and intending to use LOC as the streaming format. This document describes the commonly used base 64, base 32, and base 16 encoding schemes. It also discusses the use of line-feeds in encoded data, use of padding in encoded data, use of non-alphabet characters in encoded data, use of different encoding alphabets, and canonical encodings. [STANDARDS-TRACK] JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format. It was derived from the ECMAScript Programming Language Standard. JSON defines a small set of formatting rules for the portable representation of structured data. This document removes inconsistencies with other specifications of JSON, repairs specification errors, and offers experience-based interoperability guidance. This document describes the structure, content, construction, and semantics of language tags for use in cases where it is desirable to indicate the language used in an information object. It also describes how to register values for use in language tags and the creation of user-defined extensions for private interchange. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This document defines procedures for the specification and registration of media types for use in HTTP, MIME, and other Internet protocols. This memo documents an Internet Best Current Practice. This document defines the core of the QUIC transport protocol. QUIC provides applications with flow-controlled streams for structured communication, low-latency connection establishment, and network path migration. QUIC includes security measures that ensure confidentiality, integrity, and availability in a range of deployment circumstances. Accompanying documents describe the integration of TLS for key negotiation, loss detection, and an exemplary congestion control algorithm. This RFC documents the format used for Comma-Separated Values (CSV) files and registers the associated MIME type "text/csv". This memo provides information for the Internet community. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.

Acknowledgments

• the MoQ Workgroup and mailing lists.